2023-10-30T13:06:50.807 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-30T13:06:50.808 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045 --name leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi --owner scheduled_leonidus@teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --verbose -- /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045/orig.config.yaml 2023-10-30T13:06:50.860 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T13:06:51.069 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045 branch: wip-lusov-balance-after-stop description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: null first_in_suite: false job_id: '7441045' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-lusov-balance-after-stop ceph: cephfs: max_mds: 3 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-balance-after-stop sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 owner: scheduled_leonidus@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 8397 sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 sleep_before_teardown: 0 subset: 464/1024 suite: fs suite_branch: wip-lusov-balance-after-stop suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE8KRBqKi4J4ac52uwYhjG88eMhxcjlX+sg7m3/vyM3wFY/30AJq3vthxKneW/V1m+99tGjKFak5Zb3TtmS+yWc= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBQUkOw5PZhB5D+tQQfIqdQCcoWCwOAKnTkiMvcCnO+2w8k4FWZdHEkcrgVnAq8Uovev07ctOSFh0WtmGK+Dgo= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-30_06:34:16 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-30T13:06:51.070 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa; will attempt to use it 2023-10-30T13:06:51.070 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks 2023-10-30T13:06:51.071 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-30T13:06:51.073 INFO:teuthology.task.internal:Checking packages... 2023-10-30T13:06:51.095 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '00c1c13cea21a686b046584596fbeac5315a7d71' 2023-10-30T13:06:51.095 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-30T13:06:51.096 INFO:teuthology.packaging:ref: None 2023-10-30T13:06:51.096 INFO:teuthology.packaging:tag: None 2023-10-30T13:06:51.096 INFO:teuthology.packaging:branch: wip-lusov-balance-after-stop 2023-10-30T13:06:51.096 INFO:teuthology.packaging:sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:06:51.097 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-lusov-balance-after-stop 2023-10-30T13:06:51.345 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:06:51.345 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-30T13:06:51.353 INFO:teuthology.task.internal:no buildpackages task found 2023-10-30T13:06:51.353 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-30T13:06:51.369 INFO:teuthology.task.internal:Saving configuration 2023-10-30T13:06:51.387 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-30T13:06:51.395 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T13:06:51.421 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 13:01:04.544015', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMZBMO8UPnR55aa18+oqkfDxSe0byIjkfnowjINYeY0q'} 2023-10-30T13:06:51.445 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 13:01:04.542408', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICRKcjd/MifI4sVQedSU5g381cutj+IK5EeHSfpNvgj4'} 2023-10-30T13:06:51.445 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-30T13:06:51.453 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-30T13:06:51.453 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-10-30T13:06:51.453 INFO:teuthology.run_tasks:Running task console_log... 2023-10-30T13:06:51.557 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6ce4e0ac10>, signals=[15]) 2023-10-30T13:06:51.558 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-30T13:06:51.570 INFO:teuthology.task.internal:Opening connections... 2023-10-30T13:06:51.571 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2023-10-30T13:06:51.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:06:51.652 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2023-10-30T13:06:51.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:06:51.728 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-30T13:06:51.740 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2023-10-30T13:06:51.746 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2023-10-30T13:06:51.746 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2023-10-30T13:06:51.794 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-30T13:06:51.794 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="Ubuntu" 2023-10-30T13:06:51.795 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="22.04" 2023-10-30T13:06:51.795 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-30T13:06:51.795 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_CODENAME=jammy 2023-10-30T13:06:51.795 INFO:teuthology.orchestra.run.smithi046.stdout:ID=ubuntu 2023-10-30T13:06:51.796 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE=debian 2023-10-30T13:06:51.796 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-30T13:06:51.796 INFO:teuthology.orchestra.run.smithi046.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-30T13:06:51.796 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-30T13:06:51.796 INFO:teuthology.orchestra.run.smithi046.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-30T13:06:51.797 INFO:teuthology.orchestra.run.smithi046.stdout:UBUNTU_CODENAME=jammy 2023-10-30T13:06:51.797 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2023-10-30T13:06:51.822 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2023-10-30T13:06:51.827 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2023-10-30T13:06:51.828 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2023-10-30T13:06:51.876 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-30T13:06:51.876 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="Ubuntu" 2023-10-30T13:06:51.877 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="22.04" 2023-10-30T13:06:51.877 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-30T13:06:51.877 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_CODENAME=jammy 2023-10-30T13:06:51.877 INFO:teuthology.orchestra.run.smithi154.stdout:ID=ubuntu 2023-10-30T13:06:51.877 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE=debian 2023-10-30T13:06:51.878 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-30T13:06:51.878 INFO:teuthology.orchestra.run.smithi154.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-30T13:06:51.878 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-30T13:06:51.878 INFO:teuthology.orchestra.run.smithi154.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-30T13:06:51.878 INFO:teuthology.orchestra.run.smithi154.stdout:UBUNTU_CODENAME=jammy 2023-10-30T13:06:51.879 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2023-10-30T13:06:51.897 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-30T13:06:51.912 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-30T13:06:51.925 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-30T13:06:51.926 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2023-10-30T13:06:51.929 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2023-10-30T13:06:51.933 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-30T13:06:51.945 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-30T13:06:51.945 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2023-10-30T13:06:51.975 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2023-10-30T13:06:51.990 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-30T13:06:52.053 INFO:teuthology.run_tasks:Running task kernel... 2023-10-30T13:06:52.073 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-10-30T13:06:52.074 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-10-30T13:06:52.074 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-10-30T13:06:52.075 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T13:06:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:06:52.080 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2023-10-30T13:06:52.247 DEBUG:teuthology.task.kernel:sha1 for client.0 is eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T13:06:52.247 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "eacf00cc51e204f401b4d5d4f84825a68f373c74"... 2023-10-30T13:06:52.247 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2023-10-30T13:06:52.253 INFO:teuthology.orchestra.run.smithi046.stdout:5.15.0-56-generic 2023-10-30T13:06:52.253 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T13:06:52.254 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-30T13:06:52.254 INFO:teuthology.task.kernel:Installing linux-firmware on client.0... 2023-10-30T13:06:52.254 DEBUG:teuthology.orchestra.run.smithi046:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-10-30T13:06:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2023-10-30T13:06:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2023-10-30T13:06:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:hint: is subject to change. To configure the initial branch name to use in all 2023-10-30T13:06:52.430 INFO:teuthology.orchestra.run.smithi046.stderr:hint: of your new repositories, which will suppress this warning, call: 2023-10-30T13:06:52.430 INFO:teuthology.orchestra.run.smithi046.stderr:hint: 2023-10-30T13:06:52.430 INFO:teuthology.orchestra.run.smithi046.stderr:hint: git config --global init.defaultBranch 2023-10-30T13:06:52.430 INFO:teuthology.orchestra.run.smithi046.stderr:hint: 2023-10-30T13:06:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-10-30T13:06:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2023-10-30T13:06:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:hint: 2023-10-30T13:06:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:hint: git branch -m 2023-10-30T13:06:52.432 INFO:teuthology.orchestra.run.smithi046.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-10-30T13:06:52.433 DEBUG:teuthology.orchestra.run.smithi046:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-10-30T13:06:52.497 DEBUG:teuthology.orchestra.run.smithi046:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-10-30T13:06:52.551 DEBUG:teuthology.orchestra.run.smithi046:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-10-30T13:07:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-10-30T13:07:50.820 INFO:teuthology.orchestra.run.smithi046.stderr: * [new branch] main -> origin/main 2023-10-30T13:07:50.851 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20190312 -> 20190312 2023-10-30T13:07:50.851 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20190416 -> 20190416 2023-10-30T13:07:50.851 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20190514 -> 20190514 2023-10-30T13:07:50.852 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20190618 -> 20190618 2023-10-30T13:07:50.852 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20190717 -> 20190717 2023-10-30T13:07:50.854 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20190815 -> 20190815 2023-10-30T13:07:50.855 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20190923 -> 20190923 2023-10-30T13:07:50.855 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20191022 -> 20191022 2023-10-30T13:07:50.855 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20191215 -> 20191215 2023-10-30T13:07:50.855 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20200122 -> 20200122 2023-10-30T13:07:50.855 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20200316 -> 20200316 2023-10-30T13:07:50.856 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20200421 -> 20200421 2023-10-30T13:07:50.856 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20200519 -> 20200519 2023-10-30T13:07:50.856 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20200619 -> 20200619 2023-10-30T13:07:50.856 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20200721 -> 20200721 2023-10-30T13:07:50.856 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20200817 -> 20200817 2023-10-30T13:07:50.857 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20200918 -> 20200918 2023-10-30T13:07:50.857 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20201022 -> 20201022 2023-10-30T13:07:50.857 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20201118 -> 20201118 2023-10-30T13:07:50.857 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20201218 -> 20201218 2023-10-30T13:07:50.857 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20210208 -> 20210208 2023-10-30T13:07:50.858 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20210315 -> 20210315 2023-10-30T13:07:50.858 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20210511 -> 20210511 2023-10-30T13:07:50.858 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20210716 -> 20210716 2023-10-30T13:07:50.858 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20210818 -> 20210818 2023-10-30T13:07:50.858 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20210919 -> 20210919 2023-10-30T13:07:50.859 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20211027 -> 20211027 2023-10-30T13:07:50.859 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20211216 -> 20211216 2023-10-30T13:07:50.861 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20220209 -> 20220209 2023-10-30T13:07:50.861 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20220310 -> 20220310 2023-10-30T13:07:50.861 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20220411 -> 20220411 2023-10-30T13:07:50.861 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20220509 -> 20220509 2023-10-30T13:07:50.862 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20220610 -> 20220610 2023-10-30T13:07:50.862 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20220708 -> 20220708 2023-10-30T13:07:50.862 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20220815 -> 20220815 2023-10-30T13:07:50.862 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20220913 -> 20220913 2023-10-30T13:07:50.862 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20221012 -> 20221012 2023-10-30T13:07:50.862 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20221109 -> 20221109 2023-10-30T13:07:50.863 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20221214 -> 20221214 2023-10-30T13:07:50.863 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20230117 -> 20230117 2023-10-30T13:07:50.863 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20230210 -> 20230210 2023-10-30T13:07:50.863 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20230310 -> 20230310 2023-10-30T13:07:50.863 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20230404 -> 20230404 2023-10-30T13:07:50.864 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20230515 -> 20230515 2023-10-30T13:07:50.864 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20230625 -> 20230625 2023-10-30T13:07:50.864 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20230804 -> 20230804 2023-10-30T13:07:50.864 INFO:teuthology.orchestra.run.smithi046.stderr: * [new tag] 20230919 -> 20230919 2023-10-30T13:07:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:Updating files: 49% (1572/3168) Updating files: 50% (1584/3168) Updating files: 51% (1616/3168) Updating files: 52% (1648/3168) Updating files: 53% (1680/3168) Updating files: 54% (1711/3168) Updating files: 55% (1743/3168) Updating files: 56% (1775/3168) Updating files: 57% (1806/3168) Updating files: 58% (1838/3168) Updating files: 59% (1870/3168) Updating files: 60% (1901/3168) Updating files: 61% (1933/3168) Updating files: 62% (1965/3168) Updating files: 63% (1996/3168) Updating files: 64% (2028/3168) Updating files: 64% (2041/3168) Updating files: 65% (2060/3168) Updating files: 66% (2091/3168) Updating files: 67% (2123/3168) Updating files: 68% (2155/3168) Updating files: 69% (2186/3168) Updating files: 70% (2218/3168) Updating files: 71% (2250/3168) Updating files: 72% (2281/3168) Updating files: 73% (2313/3168) Updating files: 74% (2345/3168) Updating files: 75% (2376/3168) Updating files: 76% (2408/3168) Updating files: 77% (2440/3168) Updating files: 78% (2472/3168) Updating files: 79% (2503/3168) Updating files: 80% (2535/3168) Updating files: 81% (2567/3168) Updating files: 81% (2582/3168) Updating files: 82% (2598/3168) Updating files: 83% (2630/3168) Updating files: 84% (2662/3168) Updating files: 85% (2693/3168) Updating files: 86% (2725/3168) Updating files: 87% (2757/3168) Updating files: 88% (2788/3168) Updating files: 89% (2820/3168) Updating files: 90% (2852/3168) Updating files: 91% (2883/3168) Updating files: 92% (2915/3168) Updating files: 93% (2947/3168) Updating files: 94% (2978/3168) Updating files: 95% (3010/3168) Updating files: 96% (3042/3168) Updating files: 97% (3073/3168) Updating files: 98% (3105/3168) Updating files: 99% (3137/3168) Updating files: 100% (3168/3168) Updating files: 100% (3168/3168), done. 2023-10-30T13:07:54.766 INFO:teuthology.orchestra.run.smithi046.stdout:HEAD is now at 2b304bfe Merge branch 'main' into 'main' 2023-10-30T13:07:54.787 INFO:teuthology.task.kernel:Downloading kernel eacf00cc51e204f401b4d5d4f84825a68f373c74 on client.0... 2023-10-30T13:07:54.788 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T13:07:55.014 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/ 2023-10-30T13:07:55.015 DEBUG:teuthology.orchestra.run.smithi046:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-rc5-geacf00cc51e2_6.6.0-rc5-geacf00cc51e2-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/ --input-file=- 2023-10-30T13:07:55.017 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi046.front.sepia.ceph.com to complete... 2023-10-30T13:07:56.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30 13:07:56 URL:https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/linux-image-6.6.0-rc5-geacf00cc51e2_6.6.0-rc5-geacf00cc51e2-1_amd64.deb [38909658/38909658] -> "/tmp/linux-image.deb" [1] 2023-10-30T13:07:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:FINISHED --2023-10-30 13:07:56-- 2023-10-30T13:07:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:Total wall clock time: 1.7s 2023-10-30T13:07:56.775 INFO:teuthology.orchestra.run.smithi046.stderr:Downloaded: 1 files, 37M in 1.5s (24.1 MB/s) 2023-10-30T13:07:56.778 INFO:teuthology.task.kernel:Installing kernel eacf00cc51e204f401b4d5d4f84825a68f373c74 on client.0... 2023-10-30T13:07:56.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo dpkg -i /tmp/linux-image.deb 2023-10-30T13:07:57.022 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package linux-image-6.6.0-rc5-geacf00cc51e2. 2023-10-30T13:07:58.940 INFO:teuthology.orchestra.run.smithi046.stdout:(Reading database ... 129542 files and directories currently installed.) 2023-10-30T13:07:58.944 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-10-30T13:07:59.039 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking linux-image-6.6.0-rc5-geacf00cc51e2 (6.6.0-rc5-geacf00cc51e2-1) ... 2023-10-30T13:08:02.571 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up linux-image-6.6.0-rc5-geacf00cc51e2 (6.6.0-rc5-geacf00cc51e2-1) ... 2023-10-30T13:08:02.676 INFO:teuthology.orchestra.run.smithi046.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:08:22.751 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub' 2023-10-30T13:08:22.768 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-30T13:08:22.828 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2023-10-30T13:08:23.174 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:08:23.196 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:08:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-30T13:08:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-30T13:08:23.637 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-30T13:08:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-30T13:08:23.786 INFO:teuthology.orchestra.run.smithi046.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-30T13:08:23.786 INFO:teuthology.orchestra.run.smithi046.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-30T13:08:23.786 INFO:teuthology.orchestra.run.smithi046.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-30T13:08:23.833 INFO:teuthology.orchestra.run.smithi046.stderr:done 2023-10-30T13:08:24.002 DEBUG:teuthology.orchestra.run.smithi046:> dpkg-deb -c /tmp/linux-image.deb 2023-10-30T13:08:24.010 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./ 2023-10-30T13:08:24.011 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./boot/ 2023-10-30T13:08:24.011 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 4415097 2023-10-30 03:46 ./boot/System.map-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:08:24.023 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 169233 2023-10-30 03:46 ./boot/config-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:08:24.024 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11352928 2023-10-30 03:46 ./boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:08:24.043 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/ 2023-10-30T13:08:24.044 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/ 2023-10-30T13:08:24.044 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/postinst.d/ 2023-10-30T13:08:24.044 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/postrm.d/ 2023-10-30T13:08:24.044 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/preinst.d/ 2023-10-30T13:08:24.044 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/prerm.d/ 2023-10-30T13:08:24.045 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/ 2023-10-30T13:08:24.045 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/ 2023-10-30T13:08:24.045 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/ 2023-10-30T13:08:24.045 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/ 2023-10-30T13:08:24.045 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/ 2023-10-30T13:08:24.045 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/ 2023-10-30T13:08:24.046 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/ 2023-10-30T13:08:24.046 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 490233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/aesni-intel.ko 2023-10-30T13:08:24.049 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-30T13:08:24.049 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-30T13:08:24.049 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-30T13:08:24.050 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 93113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-30T13:08:24.050 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 106729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-30T13:08:24.050 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 152241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-30T13:08:24.050 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-30T13:08:24.051 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-30T13:08:24.051 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 83889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-30T13:08:24.051 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-30T13:08:24.051 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-30T13:08:24.051 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-30T13:08:24.052 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-30T13:08:24.052 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-30T13:08:24.053 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 40529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-30T13:08:24.053 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 58585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-30T13:08:24.053 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 50497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-30T13:08:24.053 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-30T13:08:24.054 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-30T13:08:24.054 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/ 2023-10-30T13:08:24.054 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/ 2023-10-30T13:08:24.054 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/mce/ 2023-10-30T13:08:24.055 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 40113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-30T13:08:24.055 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpuid.ko 2023-10-30T13:08:24.055 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/msr.ko 2023-10-30T13:08:24.056 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/ 2023-10-30T13:08:24.056 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 353761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-30T13:08:24.056 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 881713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-30T13:08:24.056 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 2276817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm.ko 2023-10-30T13:08:24.127 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ 2023-10-30T13:08:24.128 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 53761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/af_alg.ko 2023-10-30T13:08:24.128 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/algif_hash.ko 2023-10-30T13:08:24.128 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/algif_skcipher.ko 2023-10-30T13:08:24.128 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ansi_cprng.ko 2023-10-30T13:08:24.129 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/anubis.ko 2023-10-30T13:08:24.129 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/arc4.ko 2023-10-30T13:08:24.129 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/asymmetric_keys/ 2023-10-30T13:08:24.129 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-10-30T13:08:24.130 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/ 2023-10-30T13:08:24.130 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_memcpy.ko 2023-10-30T13:08:24.130 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_pq.ko 2023-10-30T13:08:24.130 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-30T13:08:24.130 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_tx.ko 2023-10-30T13:08:24.131 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_xor.ko 2023-10-30T13:08:24.131 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/raid6test.ko 2023-10-30T13:08:24.131 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/authenc.ko 2023-10-30T13:08:24.131 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/authencesn.ko 2023-10-30T13:08:24.131 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blake2b_generic.ko 2023-10-30T13:08:24.132 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blowfish_common.ko 2023-10-30T13:08:24.132 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blowfish_generic.ko 2023-10-30T13:08:24.132 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/camellia_generic.ko 2023-10-30T13:08:24.132 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast5_generic.ko 2023-10-30T13:08:24.132 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast6_generic.ko 2023-10-30T13:08:24.133 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast_common.ko 2023-10-30T13:08:24.133 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ccm.ko 2023-10-30T13:08:24.133 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cmac.ko 2023-10-30T13:08:24.133 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crc32_generic.ko 2023-10-30T13:08:24.133 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cryptd.ko 2023-10-30T13:08:24.134 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_engine.ko 2023-10-30T13:08:24.134 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_simd.ko 2023-10-30T13:08:24.134 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_user.ko 2023-10-30T13:08:24.134 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cts.ko 2023-10-30T13:08:24.134 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/des_generic.ko 2023-10-30T13:08:24.135 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/dh_generic.ko 2023-10-30T13:08:24.135 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/echainiv.ko 2023-10-30T13:08:24.135 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/essiv.ko 2023-10-30T13:08:24.135 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/fcrypt.ko 2023-10-30T13:08:24.135 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/khazad.ko 2023-10-30T13:08:24.136 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lrw.ko 2023-10-30T13:08:24.136 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lz4.ko 2023-10-30T13:08:24.136 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lz4hc.ko 2023-10-30T13:08:24.136 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/md4.ko 2023-10-30T13:08:24.136 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/michael_mic.ko 2023-10-30T13:08:24.137 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/pcbc.ko 2023-10-30T13:08:24.137 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/pcrypt.ko 2023-10-30T13:08:24.137 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/rmd160.ko 2023-10-30T13:08:24.137 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/seed.ko 2023-10-30T13:08:24.137 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/serpent_generic.ko 2023-10-30T13:08:24.138 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 149193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/tcrypt.ko 2023-10-30T13:08:24.138 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/tea.ko 2023-10-30T13:08:24.138 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/twofish_common.ko 2023-10-30T13:08:24.138 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/twofish_generic.ko 2023-10-30T13:08:24.138 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/vmac.ko 2023-10-30T13:08:24.139 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/wp512.ko 2023-10-30T13:08:24.139 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xcbc.ko 2023-10-30T13:08:24.139 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xor.ko 2023-10-30T13:08:24.139 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xts.ko 2023-10-30T13:08:24.139 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xxhash_generic.ko 2023-10-30T13:08:24.140 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ 2023-10-30T13:08:24.140 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/ 2023-10-30T13:08:24.140 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_extlog.ko 2023-10-30T13:08:24.140 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-30T13:08:24.140 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_pad.ko 2023-10-30T13:08:24.141 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/apei/ 2023-10-30T13:08:24.141 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/apei/einj.ko 2023-10-30T13:08:24.141 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/ec_sys.ko 2023-10-30T13:08:24.141 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/platform_profile.ko 2023-10-30T13:08:24.141 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/sbs.ko 2023-10-30T13:08:24.142 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/sbshc.ko 2023-10-30T13:08:24.142 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 103025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/video.ko 2023-10-30T13:08:24.142 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ 2023-10-30T13:08:24.142 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/acard-ahci.ko 2023-10-30T13:08:24.142 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 88865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ahci.ko 2023-10-30T13:08:24.143 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ahci_platform.ko 2023-10-30T13:08:24.143 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 88289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/libahci.ko 2023-10-30T13:08:24.143 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/libahci_platform.ko 2023-10-30T13:08:24.143 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_acpi.ko 2023-10-30T13:08:24.143 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ali.ko 2023-10-30T13:08:24.144 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_amd.ko 2023-10-30T13:08:24.144 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_artop.ko 2023-10-30T13:08:24.144 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_atiixp.ko 2023-10-30T13:08:24.144 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_atp867x.ko 2023-10-30T13:08:24.144 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cmd640.ko 2023-10-30T13:08:24.145 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cmd64x.ko 2023-10-30T13:08:24.145 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cypress.ko 2023-10-30T13:08:24.145 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_efar.ko 2023-10-30T13:08:24.145 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt366.ko 2023-10-30T13:08:24.145 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt37x.ko 2023-10-30T13:08:24.146 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-30T13:08:24.146 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-30T13:08:24.146 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_it8213.ko 2023-10-30T13:08:24.146 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_it821x.ko 2023-10-30T13:08:24.146 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_jmicron.ko 2023-10-30T13:08:24.147 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_legacy.ko 2023-10-30T13:08:24.147 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_marvell.ko 2023-10-30T13:08:24.147 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_mpiix.ko 2023-10-30T13:08:24.147 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_netcell.ko 2023-10-30T13:08:24.147 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ninja32.ko 2023-10-30T13:08:24.148 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ns87410.ko 2023-10-30T13:08:24.148 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ns87415.ko 2023-10-30T13:08:24.148 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_oldpiix.ko 2023-10-30T13:08:24.148 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_opti.ko 2023-10-30T13:08:24.148 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_optidma.ko 2023-10-30T13:08:24.149 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pcmcia.ko 2023-10-30T13:08:24.149 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-30T13:08:24.149 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-30T13:08:24.149 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_piccolo.ko 2023-10-30T13:08:24.149 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_radisys.ko 2023-10-30T13:08:24.150 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_rdc.ko 2023-10-30T13:08:24.150 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_rz1000.ko 2023-10-30T13:08:24.150 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sch.ko 2023-10-30T13:08:24.150 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_serverworks.ko 2023-10-30T13:08:24.150 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sil680.ko 2023-10-30T13:08:24.151 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sl82c105.ko 2023-10-30T13:08:24.151 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_triflex.ko 2023-10-30T13:08:24.151 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_via.ko 2023-10-30T13:08:24.151 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pdc_adma.ko 2023-10-30T13:08:24.151 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_inic162x.ko 2023-10-30T13:08:24.152 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 83345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_mv.ko 2023-10-30T13:08:24.152 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 70393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_nv.ko 2023-10-30T13:08:24.152 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_promise.ko 2023-10-30T13:08:24.152 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_qstor.ko 2023-10-30T13:08:24.152 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sil.ko 2023-10-30T13:08:24.153 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sil24.ko 2023-10-30T13:08:24.153 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sis.ko 2023-10-30T13:08:24.153 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_svw.ko 2023-10-30T13:08:24.153 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 44793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sx4.ko 2023-10-30T13:08:24.153 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_uli.ko 2023-10-30T13:08:24.154 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_via.ko 2023-10-30T13:08:24.154 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_vsc.ko 2023-10-30T13:08:24.154 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/auxdisplay/ 2023-10-30T13:08:24.154 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/auxdisplay/ks0108.ko 2023-10-30T13:08:24.154 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/bcma/ 2023-10-30T13:08:24.155 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 124473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/bcma/bcma.ko 2023-10-30T13:08:24.155 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/ 2023-10-30T13:08:24.155 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/aoe/ 2023-10-30T13:08:24.155 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 113665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/aoe/aoe.ko 2023-10-30T13:08:24.155 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/drbd/ 2023-10-30T13:08:24.155 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 965585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/drbd/drbd.ko 2023-10-30T13:08:24.156 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 173225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/floppy.ko 2023-10-30T13:08:24.156 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/mtip32xx/ 2023-10-30T13:08:24.156 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 94297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-30T13:08:24.156 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 115793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/nbd.ko 2023-10-30T13:08:24.156 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/null_blk/ 2023-10-30T13:08:24.157 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 110449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/null_blk/null_blk.ko 2023-10-30T13:08:24.157 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 98193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/pktcdvd.ko 2023-10-30T13:08:24.157 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 263769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/rbd.ko 2023-10-30T13:08:24.157 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/xen-blkback/ 2023-10-30T13:08:24.157 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 98145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-30T13:08:24.158 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/zram/ 2023-10-30T13:08:24.158 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 60217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/zram/zram.ko 2023-10-30T13:08:24.158 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ 2023-10-30T13:08:24.158 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/applicom.ko 2023-10-30T13:08:24.158 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hangcheck-timer.ko 2023-10-30T13:08:24.159 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/ 2023-10-30T13:08:24.159 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-30T13:08:24.159 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-30T13:08:24.159 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-30T13:08:24.159 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/via-rng.ko 2023-10-30T13:08:24.160 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-30T13:08:24.160 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ 2023-10-30T13:08:24.160 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-30T13:08:24.160 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 148601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-30T13:08:24.160 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-30T13:08:24.161 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 139313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-30T13:08:24.161 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 70921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-30T13:08:24.161 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 53513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-30T13:08:24.161 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/lp.ko 2023-10-30T13:08:24.161 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/mwave/ 2023-10-30T13:08:24.161 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 84097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/mwave/mwave.ko 2023-10-30T13:08:24.162 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/nvram.ko 2023-10-30T13:08:24.162 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ppdev.ko 2023-10-30T13:08:24.162 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tlclk.ko 2023-10-30T13:08:24.162 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/ 2023-10-30T13:08:24.162 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-30T13:08:24.163 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-10-30T13:08:24.163 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-10-30T13:08:24.163 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-10-30T13:08:24.163 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-30T13:08:24.163 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-30T13:08:24.164 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-10-30T13:08:24.164 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/ 2023-10-30T13:08:24.164 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_class.ko 2023-10-30T13:08:24.164 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 50945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_core.ko 2023-10-30T13:08:24.164 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-10-30T13:08:24.165 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/ 2023-10-30T13:08:24.165 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-palmas.ko 2023-10-30T13:08:24.165 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-si5351.ko 2023-10-30T13:08:24.165 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-twl6040.ko 2023-10-30T13:08:24.165 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-wm831x.ko 2023-10-30T13:08:24.166 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/ 2023-10-30T13:08:24.166 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-30T13:08:24.166 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-30T13:08:24.166 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-30T13:08:24.167 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ 2023-10-30T13:08:24.167 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ 2023-10-30T13:08:24.167 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 70801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-30T13:08:24.167 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 243025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ccp.ko 2023-10-30T13:08:24.167 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/ 2023-10-30T13:08:24.167 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/ 2023-10-30T13:08:24.168 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_common/ 2023-10-30T13:08:24.168 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 438761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-30T13:08:24.168 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-10-30T13:08:24.168 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-30T13:08:24.168 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/padlock-aes.ko 2023-10-30T13:08:24.169 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/padlock-sha.ko 2023-10-30T13:08:24.169 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/virtio/ 2023-10-30T13:08:24.169 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 68857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-30T13:08:24.169 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dca/ 2023-10-30T13:08:24.169 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dca/dca.ko 2023-10-30T13:08:24.170 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ 2023-10-30T13:08:24.170 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/dw/ 2023-10-30T13:08:24.170 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-30T13:08:24.170 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ioat/ 2023-10-30T13:08:24.170 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 122569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-30T13:08:24.171 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/ 2023-10-30T13:08:24.171 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 79953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/amd64_edac.ko 2023-10-30T13:08:24.171 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/e752x_edac.ko 2023-10-30T13:08:24.171 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 63617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/edac_mce_amd.ko 2023-10-30T13:08:24.171 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i3000_edac.ko 2023-10-30T13:08:24.172 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i3200_edac.ko 2023-10-30T13:08:24.172 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i5100_edac.ko 2023-10-30T13:08:24.172 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i5400_edac.ko 2023-10-30T13:08:24.172 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i7300_edac.ko 2023-10-30T13:08:24.172 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 52041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i7core_edac.ko 2023-10-30T13:08:24.172 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i82975x_edac.ko 2023-10-30T13:08:24.173 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/ie31200_edac.ko 2023-10-30T13:08:24.173 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 56825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/sb_edac.ko 2023-10-30T13:08:24.173 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/x38_edac.ko 2023-10-30T13:08:24.173 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/ 2023-10-30T13:08:24.173 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-gpio.ko 2023-10-30T13:08:24.174 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max14577.ko 2023-10-30T13:08:24.174 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max77693.ko 2023-10-30T13:08:24.174 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max8997.ko 2023-10-30T13:08:24.174 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-palmas.ko 2023-10-30T13:08:24.174 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-rt8973a.ko 2023-10-30T13:08:24.175 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-sm5502.ko 2023-10-30T13:08:24.175 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/ 2023-10-30T13:08:24.175 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/dmi-sysfs.ko 2023-10-30T13:08:24.175 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/efi/ 2023-10-30T13:08:24.175 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/efi/efi-pstore.ko 2023-10-30T13:08:24.176 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-30T13:08:24.176 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/ 2023-10-30T13:08:24.176 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-adp5520.ko 2023-10-30T13:08:24.176 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-amd8111.ko 2023-10-30T13:08:24.176 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-arizona.ko 2023-10-30T13:08:24.177 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-da9052.ko 2023-10-30T13:08:24.177 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-da9055.ko 2023-10-30T13:08:24.177 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-dln2.ko 2023-10-30T13:08:24.177 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 77849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-f7188x.ko 2023-10-30T13:08:24.177 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-generic.ko 2023-10-30T13:08:24.178 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-ich.ko 2023-10-30T13:08:24.178 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-10-30T13:08:24.178 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-kempld.ko 2023-10-30T13:08:24.178 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-lp3943.ko 2023-10-30T13:08:24.178 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max7300.ko 2023-10-30T13:08:24.178 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max730x.ko 2023-10-30T13:08:24.179 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max732x.ko 2023-10-30T13:08:24.179 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-10-30T13:08:24.179 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-pca953x.ko 2023-10-30T13:08:24.179 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-pcf857x.ko 2023-10-30T13:08:24.179 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-rdc321x.ko 2023-10-30T13:08:24.180 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-sch.ko 2023-10-30T13:08:24.180 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-sch311x.ko 2023-10-30T13:08:24.180 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-tps65912.ko 2023-10-30T13:08:24.181 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-twl4030.ko 2023-10-30T13:08:24.181 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-twl6040.ko 2023-10-30T13:08:24.181 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-30T13:08:24.182 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-vx855.ko 2023-10-30T13:08:24.182 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm831x.ko 2023-10-30T13:08:24.182 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm8350.ko 2023-10-30T13:08:24.182 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm8994.ko 2023-10-30T13:08:24.182 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/ 2023-10-30T13:08:24.183 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-a4tech.ko 2023-10-30T13:08:24.183 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-apple.ko 2023-10-30T13:08:24.183 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-appleir.ko 2023-10-30T13:08:24.183 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-aureal.ko 2023-10-30T13:08:24.183 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-axff.ko 2023-10-30T13:08:24.184 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-belkin.ko 2023-10-30T13:08:24.184 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cherry.ko 2023-10-30T13:08:24.184 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-chicony.ko 2023-10-30T13:08:24.184 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 60649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cp2112.ko 2023-10-30T13:08:24.184 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cypress.ko 2023-10-30T13:08:24.185 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-dr.ko 2023-10-30T13:08:24.185 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-elecom.ko 2023-10-30T13:08:24.185 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-elo.ko 2023-10-30T13:08:24.185 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-emsff.ko 2023-10-30T13:08:24.185 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ezkey.ko 2023-10-30T13:08:24.186 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gaff.ko 2023-10-30T13:08:24.186 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-generic.ko 2023-10-30T13:08:24.186 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gt683r.ko 2023-10-30T13:08:24.186 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gyration.ko 2023-10-30T13:08:24.186 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-30T13:08:24.187 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-30T13:08:24.187 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtekff.ko 2023-10-30T13:08:24.187 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-hyperv.ko 2023-10-30T13:08:24.187 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-icade.ko 2023-10-30T13:08:24.187 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-kensington.ko 2023-10-30T13:08:24.188 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-keytouch.ko 2023-10-30T13:08:24.188 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-kye.ko 2023-10-30T13:08:24.188 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lcpower.ko 2023-10-30T13:08:24.188 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-led.ko 2023-10-30T13:08:24.188 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 42361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lenovo.ko 2023-10-30T13:08:24.189 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lg-g15.ko 2023-10-30T13:08:24.189 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 58185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-30T13:08:24.189 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 101369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-30T13:08:24.190 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 80993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech.ko 2023-10-30T13:08:24.190 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-magicmouse.ko 2023-10-30T13:08:24.191 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-microsoft.ko 2023-10-30T13:08:24.191 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-monterey.ko 2023-10-30T13:08:24.191 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 50161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-multitouch.ko 2023-10-30T13:08:24.192 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ntrig.ko 2023-10-30T13:08:24.192 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ortek.ko 2023-10-30T13:08:24.192 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-penmount.ko 2023-10-30T13:08:24.192 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-petalynx.ko 2023-10-30T13:08:24.193 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 72937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-picolcd.ko 2023-10-30T13:08:24.193 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-pl.ko 2023-10-30T13:08:24.193 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-plantronics.ko 2023-10-30T13:08:24.194 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-primax.ko 2023-10-30T13:08:24.194 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-rmi.ko 2023-10-30T13:08:24.194 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-30T13:08:24.194 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-common.ko 2023-10-30T13:08:24.195 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-30T13:08:24.195 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-30T13:08:24.195 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-30T13:08:24.195 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-30T13:08:24.196 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-30T13:08:24.196 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-30T13:08:24.196 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-30T13:08:24.196 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-30T13:08:24.196 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-30T13:08:24.196 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat.ko 2023-10-30T13:08:24.197 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-saitek.ko 2023-10-30T13:08:24.197 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-samsung.ko 2023-10-30T13:08:24.197 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-30T13:08:24.197 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sjoy.ko 2023-10-30T13:08:24.197 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sony.ko 2023-10-30T13:08:24.198 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-speedlink.ko 2023-10-30T13:08:24.198 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-steelseries.ko 2023-10-30T13:08:24.198 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sunplus.ko 2023-10-30T13:08:24.198 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-30T13:08:24.198 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-tivo.ko 2023-10-30T13:08:24.199 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-tmff.ko 2023-10-30T13:08:24.199 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-topseed.ko 2023-10-30T13:08:24.199 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-twinhan.ko 2023-10-30T13:08:24.199 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 71073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-uclogic.ko 2023-10-30T13:08:24.199 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-waltop.ko 2023-10-30T13:08:24.200 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 143473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-wiimote.ko 2023-10-30T13:08:24.200 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-xinmo.ko 2023-10-30T13:08:24.200 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-zpff.ko 2023-10-30T13:08:24.200 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-zydacron.ko 2023-10-30T13:08:24.200 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 267825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid.ko 2023-10-30T13:08:24.201 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/uhid.ko 2023-10-30T13:08:24.201 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/ 2023-10-30T13:08:24.201 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 125721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbhid.ko 2023-10-30T13:08:24.201 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbkbd.ko 2023-10-30T13:08:24.201 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbmouse.ko 2023-10-30T13:08:24.201 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 217873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/wacom.ko 2023-10-30T13:08:24.202 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/ 2023-10-30T13:08:24.202 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 64201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_balloon.ko 2023-10-30T13:08:24.202 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 89625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_utils.ko 2023-10-30T13:08:24.202 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 305465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_vmbus.ko 2023-10-30T13:08:24.202 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ 2023-10-30T13:08:24.203 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 66521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/abituguru.ko 2023-10-30T13:08:24.203 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 61393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/abituguru3.ko 2023-10-30T13:08:24.203 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-30T13:08:24.203 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ad7414.ko 2023-10-30T13:08:24.203 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ad7418.ko 2023-10-30T13:08:24.204 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adc128d818.ko 2023-10-30T13:08:24.204 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1025.ko 2023-10-30T13:08:24.204 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 97977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1026.ko 2023-10-30T13:08:24.204 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1029.ko 2023-10-30T13:08:24.204 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 45553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1031.ko 2023-10-30T13:08:24.205 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm9240.ko 2023-10-30T13:08:24.205 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ads7828.ko 2023-10-30T13:08:24.205 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7410.ko 2023-10-30T13:08:24.205 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7411.ko 2023-10-30T13:08:24.205 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 74433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7462.ko 2023-10-30T13:08:24.206 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7470.ko 2023-10-30T13:08:24.206 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 67929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7475.ko 2023-10-30T13:08:24.206 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7x10.ko 2023-10-30T13:08:24.206 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/amc6821.ko 2023-10-30T13:08:24.206 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 55265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/applesmc.ko 2023-10-30T13:08:24.207 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 49913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asb100.ko 2023-10-30T13:08:24.207 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 58617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asc7621.ko 2023-10-30T13:08:24.207 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 40889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-30T13:08:24.207 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/atxp1.ko 2023-10-30T13:08:24.207 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/coretemp.ko 2023-10-30T13:08:24.208 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/da9052-hwmon.ko 2023-10-30T13:08:24.208 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/da9055-hwmon.ko 2023-10-30T13:08:24.208 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 44465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-30T13:08:24.208 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 99601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/dme1737.ko 2023-10-30T13:08:24.208 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ds1621.ko 2023-10-30T13:08:24.209 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ds620.ko 2023-10-30T13:08:24.209 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc1403.ko 2023-10-30T13:08:24.209 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc2103.ko 2023-10-30T13:08:24.209 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc6w201.ko 2023-10-30T13:08:24.209 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 66985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f71805f.ko 2023-10-30T13:08:24.210 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 94457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f71882fg.ko 2023-10-30T13:08:24.210 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f75375s.ko 2023-10-30T13:08:24.210 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/fam15h_power.ko 2023-10-30T13:08:24.210 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 55449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/fschmd.ko 2023-10-30T13:08:24.211 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/g760a.ko 2023-10-30T13:08:24.211 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/g762.ko 2023-10-30T13:08:24.211 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 46801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/gl518sm.ko 2023-10-30T13:08:24.212 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 45529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/gl520sm.ko 2023-10-30T13:08:24.212 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/hih6130.ko 2023-10-30T13:08:24.212 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-30T13:08:24.212 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/i5500_temp.ko 2023-10-30T13:08:24.213 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/i5k_amb.ko 2023-10-30T13:08:24.213 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ibmaem.ko 2023-10-30T13:08:24.213 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ibmpex.ko 2023-10-30T13:08:24.213 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ina209.ko 2023-10-30T13:08:24.213 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ina2xx.ko 2023-10-30T13:08:24.214 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 130449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/it87.ko 2023-10-30T13:08:24.214 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/jc42.ko 2023-10-30T13:08:24.214 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/k10temp.ko 2023-10-30T13:08:24.214 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/k8temp.ko 2023-10-30T13:08:24.214 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lineage-pem.ko 2023-10-30T13:08:24.215 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 48601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm63.ko 2023-10-30T13:08:24.215 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm73.ko 2023-10-30T13:08:24.215 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm75.ko 2023-10-30T13:08:24.215 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm77.ko 2023-10-30T13:08:24.215 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm78.ko 2023-10-30T13:08:24.216 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm80.ko 2023-10-30T13:08:24.216 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm83.ko 2023-10-30T13:08:24.216 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 72833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm85.ko 2023-10-30T13:08:24.216 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 46153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm87.ko 2023-10-30T13:08:24.216 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm90.ko 2023-10-30T13:08:24.217 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm92.ko 2023-10-30T13:08:24.217 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 108841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm93.ko 2023-10-30T13:08:24.217 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95234.ko 2023-10-30T13:08:24.217 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95241.ko 2023-10-30T13:08:24.217 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95245.ko 2023-10-30T13:08:24.218 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc2945.ko 2023-10-30T13:08:24.218 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4151.ko 2023-10-30T13:08:24.218 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4215.ko 2023-10-30T13:08:24.218 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4222.ko 2023-10-30T13:08:24.218 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4245.ko 2023-10-30T13:08:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4260.ko 2023-10-30T13:08:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4261.ko 2023-10-30T13:08:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max16065.ko 2023-10-30T13:08:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max1619.ko 2023-10-30T13:08:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max1668.ko 2023-10-30T13:08:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max197.ko 2023-10-30T13:08:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6639.ko 2023-10-30T13:08:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6650.ko 2023-10-30T13:08:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6697.ko 2023-10-30T13:08:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/mc13783-adc.ko 2023-10-30T13:08:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/mcp3021.ko 2023-10-30T13:08:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-10-30T13:08:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 44129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6683.ko 2023-10-30T13:08:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 119817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6775-core.ko 2023-10-30T13:08:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 60705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6775.ko 2023-10-30T13:08:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 59737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct7802.ko 2023-10-30T13:08:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 82337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pc87360.ko 2023-10-30T13:08:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 54433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pc87427.ko 2023-10-30T13:08:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pcf8591.ko 2023-10-30T13:08:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ 2023-10-30T13:08:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-30T13:08:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-30T13:08:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-30T13:08:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-30T13:08:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-30T13:08:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-30T13:08:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-30T13:08:24.224 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 82577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-30T13:08:24.224 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-30T13:08:24.224 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-30T13:08:24.224 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-30T13:08:24.224 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-30T13:08:24.225 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/powr1220.ko 2023-10-30T13:08:24.225 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch5627.ko 2023-10-30T13:08:24.225 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch5636.ko 2023-10-30T13:08:24.225 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-30T13:08:24.225 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sht15.ko 2023-10-30T13:08:24.225 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sht21.ko 2023-10-30T13:08:24.226 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/shtc1.ko 2023-10-30T13:08:24.226 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 40105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sis5595.ko 2023-10-30T13:08:24.226 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47b397.ko 2023-10-30T13:08:24.226 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47m1.ko 2023-10-30T13:08:24.226 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47m192.ko 2023-10-30T13:08:24.227 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/thmc50.ko 2023-10-30T13:08:24.227 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp102.ko 2023-10-30T13:08:24.227 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp103.ko 2023-10-30T13:08:24.227 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp401.ko 2023-10-30T13:08:24.227 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp421.ko 2023-10-30T13:08:24.228 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/via-cputemp.ko 2023-10-30T13:08:24.228 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/via686a.ko 2023-10-30T13:08:24.228 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 52457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/vt1211.ko 2023-10-30T13:08:24.228 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/vt8231.ko 2023-10-30T13:08:24.228 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 57513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83627ehf.ko 2023-10-30T13:08:24.229 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 78873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83627hf.ko 2023-10-30T13:08:24.229 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 71345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83781d.ko 2023-10-30T13:08:24.229 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 63289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83791d.ko 2023-10-30T13:08:24.229 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 71321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83792d.ko 2023-10-30T13:08:24.229 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 101473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83793.ko 2023-10-30T13:08:24.230 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 119505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83795.ko 2023-10-30T13:08:24.230 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83l785ts.ko 2023-10-30T13:08:24.230 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83l786ng.ko 2023-10-30T13:08:24.230 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-10-30T13:08:24.230 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-10-30T13:08:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/ 2023-10-30T13:08:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/ 2023-10-30T13:08:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-30T13:08:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-30T13:08:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/ 2023-10-30T13:08:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-10-30T13:08:24.232 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-10-30T13:08:24.232 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-10-30T13:08:24.232 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-30T13:08:24.232 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-30T13:08:24.232 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-30T13:08:24.233 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-10-30T13:08:24.233 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 6929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-30T13:08:24.233 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-10-30T13:08:24.233 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 50201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-30T13:08:24.233 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-30T13:08:24.234 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-30T13:08:24.234 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-30T13:08:24.234 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-10-30T13:08:24.234 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-10-30T13:08:24.234 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 62817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-30T13:08:24.235 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-30T13:08:24.235 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-30T13:08:24.235 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-10-30T13:08:24.235 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-30T13:08:24.235 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-30T13:08:24.236 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-10-30T13:08:24.236 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-30T13:08:24.236 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-30T13:08:24.236 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-30T13:08:24.236 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-10-30T13:08:24.236 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-30T13:08:24.237 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-30T13:08:24.237 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-10-30T13:08:24.237 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-10-30T13:08:24.238 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-30T13:08:24.238 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-10-30T13:08:24.238 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-30T13:08:24.239 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-30T13:08:24.239 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-30T13:08:24.239 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-30T13:08:24.239 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-10-30T13:08:24.240 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-mux.ko 2023-10-30T13:08:24.240 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-smbus.ko 2023-10-30T13:08:24.240 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-stub.ko 2023-10-30T13:08:24.241 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/ 2023-10-30T13:08:24.241 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-10-30T13:08:24.241 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-10-30T13:08:24.241 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-10-30T13:08:24.242 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ 2023-10-30T13:08:24.242 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ 2023-10-30T13:08:24.242 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 273609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-30T13:08:24.242 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 879969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_core.ko 2023-10-30T13:08:24.242 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 69337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-30T13:08:24.243 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 324217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-30T13:08:24.243 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 97985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-30T13:08:24.243 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 282273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-30T13:08:24.243 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 66201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-30T13:08:24.243 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ 2023-10-30T13:08:24.244 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/cxgb4/ 2023-10-30T13:08:24.244 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 493073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-30T13:08:24.244 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx4/ 2023-10-30T13:08:24.244 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 419737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-30T13:08:24.244 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx5/ 2023-10-30T13:08:24.244 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 739025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-30T13:08:24.245 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mthca/ 2023-10-30T13:08:24.245 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 296625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-30T13:08:24.245 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ocrdma/ 2023-10-30T13:08:24.245 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 153697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-10-30T13:08:24.245 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/usnic/ 2023-10-30T13:08:24.246 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 168969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-10-30T13:08:24.246 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ 2023-10-30T13:08:24.246 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ipoib/ 2023-10-30T13:08:24.246 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 263081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-30T13:08:24.246 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/iser/ 2023-10-30T13:08:24.247 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 100241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-30T13:08:24.247 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/isert/ 2023-10-30T13:08:24.247 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 111425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-30T13:08:24.247 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srp/ 2023-10-30T13:08:24.247 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 128753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-30T13:08:24.248 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srpt/ 2023-10-30T13:08:24.248 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 129529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-30T13:08:24.248 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/ 2023-10-30T13:08:24.248 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/evbug.ko 2023-10-30T13:08:24.248 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/ff-memless.ko 2023-10-30T13:08:24.248 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/ 2023-10-30T13:08:24.249 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-30T13:08:24.249 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-30T13:08:24.249 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/gameport.ko 2023-10-30T13:08:24.249 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/joydev.ko 2023-10-30T13:08:24.249 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/ 2023-10-30T13:08:24.250 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-10-30T13:08:24.250 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-10-30T13:08:24.250 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-10-30T13:08:24.250 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-10-30T13:08:24.250 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-30T13:08:24.251 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-10-30T13:08:24.251 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lkkbd.ko 2023-10-30T13:08:24.251 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lm8323.ko 2023-10-30T13:08:24.251 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lm8333.ko 2023-10-30T13:08:24.251 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-10-30T13:08:24.252 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-10-30T13:08:24.252 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-10-30T13:08:24.252 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-10-30T13:08:24.252 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/newtonkbd.ko 2023-10-30T13:08:24.252 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-10-30T13:08:24.253 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/qt1070.ko 2023-10-30T13:08:24.253 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/qt2160.ko 2023-10-30T13:08:24.253 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-10-30T13:08:24.253 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/stowaway.ko 2023-10-30T13:08:24.253 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/sunkbd.ko 2023-10-30T13:08:24.254 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-10-30T13:08:24.254 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-10-30T13:08:24.254 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-10-30T13:08:24.254 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/xtkbd.ko 2023-10-30T13:08:24.254 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/matrix-keymap.ko 2023-10-30T13:08:24.255 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/ 2023-10-30T13:08:24.255 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/appletouch.ko 2023-10-30T13:08:24.255 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/bcm5974.ko 2023-10-30T13:08:24.255 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 123681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/cyapatp.ko 2023-10-30T13:08:24.255 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 89449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-30T13:08:24.256 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/gpio_mouse.ko 2023-10-30T13:08:24.256 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 335977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/psmouse.ko 2023-10-30T13:08:24.256 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/sermouse.ko 2023-10-30T13:08:24.256 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-30T13:08:24.256 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-30T13:08:24.256 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-30T13:08:24.257 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/rmi4/ 2023-10-30T13:08:24.257 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 109873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-30T13:08:24.257 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ 2023-10-30T13:08:24.257 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/altera_ps2.ko 2023-10-30T13:08:24.257 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/arc_ps2.ko 2023-10-30T13:08:24.258 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ct82c710.ko 2023-10-30T13:08:24.258 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-30T13:08:24.258 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/parkbd.ko 2023-10-30T13:08:24.258 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/pcips2.ko 2023-10-30T13:08:24.258 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ps2mult.ko 2023-10-30T13:08:24.259 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/serio_raw.ko 2023-10-30T13:08:24.259 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/serport.ko 2023-10-30T13:08:24.259 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/sparse-keymap.ko 2023-10-30T13:08:24.259 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/ 2023-10-30T13:08:24.259 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/amd/ 2023-10-30T13:08:24.259 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-30T13:08:24.260 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/ 2023-10-30T13:08:24.260 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-88pm860x.ko 2023-10-30T13:08:24.260 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-adp5520.ko 2023-10-30T13:08:24.260 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 59753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-bd2802.ko 2023-10-30T13:08:24.260 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-blinkm.ko 2023-10-30T13:08:24.261 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-da903x.ko 2023-10-30T13:08:24.261 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-da9052.ko 2023-10-30T13:08:24.261 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-gpio.ko 2023-10-30T13:08:24.261 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3530.ko 2023-10-30T13:08:24.261 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3533.ko 2023-10-30T13:08:24.262 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm355x.ko 2023-10-30T13:08:24.262 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3642.ko 2023-10-30T13:08:24.262 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lp3944.ko 2023-10-30T13:08:24.262 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lp8788.ko 2023-10-30T13:08:24.262 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-max8997.ko 2023-10-30T13:08:24.263 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-mc13783.ko 2023-10-30T13:08:24.263 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-menf21bmc.ko 2023-10-30T13:08:24.263 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca9532.ko 2023-10-30T13:08:24.263 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca955x.ko 2023-10-30T13:08:24.263 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca963x.ko 2023-10-30T13:08:24.264 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-ss4200.ko 2023-10-30T13:08:24.264 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-tca6507.ko 2023-10-30T13:08:24.264 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-wm831x-status.ko 2023-10-30T13:08:24.264 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-wm8350.ko 2023-10-30T13:08:24.264 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ 2023-10-30T13:08:24.264 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-30T13:08:24.265 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-30T13:08:24.265 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-30T13:08:24.265 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-30T13:08:24.265 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-30T13:08:24.265 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-30T13:08:24.266 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-30T13:08:24.266 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-30T13:08:24.266 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/ 2023-10-30T13:08:24.266 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/bcache/ 2023-10-30T13:08:24.267 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 654969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/bcache/bcache.ko 2023-10-30T13:08:24.267 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 40577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-bio-prison.ko 2023-10-30T13:08:24.267 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 89601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-bufio.ko 2023-10-30T13:08:24.267 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-cache-smq.ko 2023-10-30T13:08:24.268 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 156009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-cache.ko 2023-10-30T13:08:24.268 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 106897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-crypt.ko 2023-10-30T13:08:24.268 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-delay.ko 2023-10-30T13:08:24.268 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 53857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-era.ko 2023-10-30T13:08:24.268 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-flakey.ko 2023-10-30T13:08:24.269 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-log-userspace.ko 2023-10-30T13:08:24.269 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-log.ko 2023-10-30T13:08:24.269 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 49545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-mirror.ko 2023-10-30T13:08:24.269 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 78033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-multipath.ko 2023-10-30T13:08:24.269 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-queue-length.ko 2023-10-30T13:08:24.269 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 70825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-raid.ko 2023-10-30T13:08:24.270 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-region-hash.ko 2023-10-30T13:08:24.270 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-round-robin.ko 2023-10-30T13:08:24.270 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-service-time.ko 2023-10-30T13:08:24.270 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 109593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-snapshot.ko 2023-10-30T13:08:24.270 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-switch.ko 2023-10-30T13:08:24.271 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 181865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-thin-pool.ko 2023-10-30T13:08:24.271 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-verity.ko 2023-10-30T13:08:24.271 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-zero.ko 2023-10-30T13:08:24.271 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/faulty.ko 2023-10-30T13:08:24.271 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/linear.ko 2023-10-30T13:08:24.272 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/multipath.ko 2023-10-30T13:08:24.272 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/persistent-data/ 2023-10-30T13:08:24.272 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 199937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-30T13:08:24.272 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid0.ko 2023-10-30T13:08:24.272 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 113073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid1.ko 2023-10-30T13:08:24.273 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 158681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid10.ko 2023-10-30T13:08:24.273 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 395721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid456.ko 2023-10-30T13:08:24.273 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/ 2023-10-30T13:08:24.273 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/ 2023-10-30T13:08:24.273 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 178137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptbase.ko 2023-10-30T13:08:24.273 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 63385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptctl.ko 2023-10-30T13:08:24.274 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 44665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptfc.ko 2023-10-30T13:08:24.274 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptlan.ko 2023-10-30T13:08:24.274 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 123321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptsas.ko 2023-10-30T13:08:24.274 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 79041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptscsih.ko 2023-10-30T13:08:24.274 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 53081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptspi.ko 2023-10-30T13:08:24.275 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/ 2023-10-30T13:08:24.275 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm800.ko 2023-10-30T13:08:24.275 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm805.ko 2023-10-30T13:08:24.275 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm80x.ko 2023-10-30T13:08:24.275 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/arizona-i2c.ko 2023-10-30T13:08:24.276 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 123721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/arizona.ko 2023-10-30T13:08:24.276 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/bcm590xx.ko 2023-10-30T13:08:24.276 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/cros_ec_dev.ko 2023-10-30T13:08:24.276 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/dln2.ko 2023-10-30T13:08:24.276 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/janz-cmodio.ko 2023-10-30T13:08:24.277 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 53585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/kempld-core.ko 2023-10-30T13:08:24.277 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lm3533-core.ko 2023-10-30T13:08:24.277 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-10-30T13:08:24.277 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lp3943.ko 2023-10-30T13:08:24.277 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 54569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lpc_ich.ko 2023-10-30T13:08:24.277 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lpc_sch.ko 2023-10-30T13:08:24.278 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/max8907.ko 2023-10-30T13:08:24.278 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/mc13xxx-core.ko 2023-10-30T13:08:24.278 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-10-30T13:08:24.278 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/menf21bmc.ko 2023-10-30T13:08:24.278 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633-adc.ko 2023-10-30T13:08:24.279 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633-gpio.ko 2023-10-30T13:08:24.279 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633.ko 2023-10-30T13:08:24.279 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-10-30T13:08:24.279 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/retu-mfd.ko 2023-10-30T13:08:24.279 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 80425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/si476x-core.ko 2023-10-30T13:08:24.280 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 48457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/sm501.ko 2023-10-30T13:08:24.280 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-10-30T13:08:24.280 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/tps65010.ko 2023-10-30T13:08:24.280 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/tps6507x.ko 2023-10-30T13:08:24.280 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/viperboard.ko 2023-10-30T13:08:24.281 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/vx855.ko 2023-10-30T13:08:24.281 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/wl1273-core.ko 2023-10-30T13:08:24.281 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 44345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/wm8994.ko 2023-10-30T13:08:24.281 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ 2023-10-30T13:08:24.281 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-10-30T13:08:24.282 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 48489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ad525x_dpot.ko 2023-10-30T13:08:24.282 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/altera-stapl/ 2023-10-30T13:08:24.282 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 46585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-30T13:08:24.282 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/apds9802als.ko 2023-10-30T13:08:24.282 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/apds990x.ko 2023-10-30T13:08:24.283 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 45593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/bh1770glc.ko 2023-10-30T13:08:24.283 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/ 2023-10-30T13:08:24.283 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-10-30T13:08:24.283 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/core.ko 2023-10-30T13:08:24.283 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/cb710/ 2023-10-30T13:08:24.283 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/cb710/cb710.ko 2023-10-30T13:08:24.284 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ds1682.ko 2023-10-30T13:08:24.284 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/dummy-irq.ko 2023-10-30T13:08:24.284 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/echo/ 2023-10-30T13:08:24.284 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/echo/echo.ko 2023-10-30T13:08:24.284 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/ 2023-10-30T13:08:24.285 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/at24.ko 2023-10-30T13:08:24.285 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-30T13:08:24.285 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-30T13:08:24.285 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/max6875.ko 2023-10-30T13:08:24.285 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/enclosure.ko 2023-10-30T13:08:24.286 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/genwqe/ 2023-10-30T13:08:24.286 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 152201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-10-30T13:08:24.286 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/hmc6352.ko 2023-10-30T13:08:24.286 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/hpilo.ko 2023-10-30T13:08:24.286 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ibmasm/ 2023-10-30T13:08:24.287 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 72129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-10-30T13:08:24.287 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ics932s401.ko 2023-10-30T13:08:24.287 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/isl29003.ko 2023-10-30T13:08:24.287 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/isl29020.ko 2023-10-30T13:08:24.287 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/ 2023-10-30T13:08:24.288 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 42513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-30T13:08:24.288 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-30T13:08:24.288 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/ 2023-10-30T13:08:24.288 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 125209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei-me.ko 2023-10-30T13:08:24.288 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 56521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei-txe.ko 2023-10-30T13:08:24.289 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 310113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei.ko 2023-10-30T13:08:24.289 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/phantom.ko 2023-10-30T13:08:24.289 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ti-st/ 2023-10-30T13:08:24.289 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 74265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ti-st/st_drv.ko 2023-10-30T13:08:24.289 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tifm_7xx1.ko 2023-10-30T13:08:24.289 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tifm_core.ko 2023-10-30T13:08:24.290 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tsl2550.ko 2023-10-30T13:08:24.290 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 48697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_balloon.ko 2023-10-30T13:08:24.290 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_vmci/ 2023-10-30T13:08:24.290 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 183609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-30T13:08:24.290 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ 2023-10-30T13:08:24.291 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/bonding/ 2023-10-30T13:08:24.291 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 490129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/bonding/bonding.ko 2023-10-30T13:08:24.291 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/ 2023-10-30T13:08:24.291 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/b53/ 2023-10-30T13:08:24.291 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 106033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-30T13:08:24.437 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 40297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-30T13:08:24.438 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-30T13:08:24.438 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6xxx/ 2023-10-30T13:08:24.438 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 453737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-30T13:08:24.439 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dummy.ko 2023-10-30T13:08:24.439 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/eql.ko 2023-10-30T13:08:24.439 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ 2023-10-30T13:08:24.439 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/ 2023-10-30T13:08:24.439 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 45785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-30T13:08:24.439 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 42105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-30T13:08:24.440 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 130553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-30T13:08:24.440 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 68849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-30T13:08:24.440 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/ 2023-10-30T13:08:24.440 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-30T13:08:24.440 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 59689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-30T13:08:24.441 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-30T13:08:24.441 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 99113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-30T13:08:24.441 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/adaptec/ 2023-10-30T13:08:24.441 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 54521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-30T13:08:24.441 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/agere/ 2023-10-30T13:08:24.442 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 69201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-30T13:08:24.442 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/alteon/ 2023-10-30T13:08:24.442 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 59585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-30T13:08:24.442 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/altera/ 2023-10-30T13:08:24.443 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-30T13:08:24.443 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/ 2023-10-30T13:08:24.443 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 54609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-30T13:08:24.443 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 49289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-30T13:08:24.443 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 87297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-30T13:08:24.443 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/ 2023-10-30T13:08:24.444 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/alx/ 2023-10-30T13:08:24.444 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 97337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-30T13:08:24.444 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-10-30T13:08:24.444 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 104673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-30T13:08:24.444 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-10-30T13:08:24.445 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 90257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-30T13:08:24.445 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/ 2023-10-30T13:08:24.445 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 81569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-30T13:08:24.445 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 69729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-30T13:08:24.445 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/ 2023-10-30T13:08:24.445 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 78401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-30T13:08:24.446 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 176761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-30T13:08:24.446 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-10-30T13:08:24.446 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1421497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-30T13:08:24.446 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 128569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-30T13:08:24.446 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/genet/ 2023-10-30T13:08:24.447 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 132089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-30T13:08:24.447 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 312081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-30T13:08:24.447 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/ 2023-10-30T13:08:24.447 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/bna/ 2023-10-30T13:08:24.447 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 323513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-30T13:08:24.448 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/ 2023-10-30T13:08:24.448 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-10-30T13:08:24.448 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 193497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-30T13:08:24.448 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-10-30T13:08:24.448 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 347417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-30T13:08:24.449 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-10-30T13:08:24.449 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1044433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-30T13:08:24.449 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-10-30T13:08:24.449 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 166193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-30T13:08:24.449 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-10-30T13:08:24.449 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-30T13:08:24.450 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/ 2023-10-30T13:08:24.450 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/enic/ 2023-10-30T13:08:24.450 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 204345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-30T13:08:24.450 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/ 2023-10-30T13:08:24.450 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/ 2023-10-30T13:08:24.451 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 69449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-30T13:08:24.451 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 61489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-30T13:08:24.451 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 141225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-30T13:08:24.451 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-30T13:08:24.451 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 62817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-30T13:08:24.452 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-30T13:08:24.452 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/ 2023-10-30T13:08:24.452 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 60873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-30T13:08:24.452 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 62833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-30T13:08:24.452 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 45721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dnet.ko 2023-10-30T13:08:24.453 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ec_bhf.ko 2023-10-30T13:08:24.453 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/ 2023-10-30T13:08:24.453 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/benet/ 2023-10-30T13:08:24.453 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 281137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-30T13:08:24.453 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 49457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ethoc.ko 2023-10-30T13:08:24.453 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fealnx.ko 2023-10-30T13:08:24.454 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fujitsu/ 2023-10-30T13:08:24.454 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 45345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-10-30T13:08:24.454 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ 2023-10-30T13:08:24.454 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 90465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-30T13:08:24.454 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000/ 2023-10-30T13:08:24.455 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 295577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-30T13:08:24.455 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000e/ 2023-10-30T13:08:24.455 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 584497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-30T13:08:24.455 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/fm10k/ 2023-10-30T13:08:24.455 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 246209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-30T13:08:24.455 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/i40e/ 2023-10-30T13:08:24.456 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 925697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-30T13:08:24.456 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/iavf/ 2023-10-30T13:08:24.456 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 302849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-30T13:08:24.456 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igb/ 2023-10-30T13:08:24.457 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 514681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-30T13:08:24.457 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igbvf/ 2023-10-30T13:08:24.457 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 93593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-30T13:08:24.457 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-10-30T13:08:24.457 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 734305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-30T13:08:24.457 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-10-30T13:08:24.458 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 151769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-30T13:08:24.458 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 86921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/jme.ko 2023-10-30T13:08:24.458 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/ 2023-10-30T13:08:24.458 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-30T13:08:24.458 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 102289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-30T13:08:24.459 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 122649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-30T13:08:24.459 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/ 2023-10-30T13:08:24.459 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-10-30T13:08:24.459 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 727601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-30T13:08:24.459 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 278937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-30T13:08:24.459 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-10-30T13:08:24.460 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-10-30T13:08:24.460 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1113993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-30T13:08:24.460 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ 2023-10-30T13:08:24.460 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 53105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-10-30T13:08:24.460 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 40993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-10-30T13:08:24.460 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-10-30T13:08:24.461 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 95665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-30T13:08:24.461 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/ 2023-10-30T13:08:24.461 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-10-30T13:08:24.461 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 108649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-30T13:08:24.461 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/ 2023-10-30T13:08:24.461 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 73441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-30T13:08:24.462 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-30T13:08:24.462 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/neterion/ 2023-10-30T13:08:24.462 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 150905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-30T13:08:24.462 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/nvidia/ 2023-10-30T13:08:24.462 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 136721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-30T13:08:24.463 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/ 2023-10-30T13:08:24.463 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 50481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-30T13:08:24.463 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 43145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-30T13:08:24.463 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/ 2023-10-30T13:08:24.463 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-10-30T13:08:24.463 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 227897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-30T13:08:24.464 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 87081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-30T13:08:24.464 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-10-30T13:08:24.464 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 672049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-30T13:08:24.464 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/rdc/ 2023-10-30T13:08:24.464 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 43017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-30T13:08:24.465 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/ 2023-10-30T13:08:24.465 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 67385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-30T13:08:24.465 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 83985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-30T13:08:24.465 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-30T13:08:24.465 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 189801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-30T13:08:24.465 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/ 2023-10-30T13:08:24.466 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-10-30T13:08:24.466 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 104241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-10-30T13:08:24.466 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sfc/ 2023-10-30T13:08:24.466 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1053921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-30T13:08:24.466 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/silan/ 2023-10-30T13:08:24.467 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-30T13:08:24.467 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/ 2023-10-30T13:08:24.467 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 54817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-30T13:08:24.467 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 63337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-30T13:08:24.467 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/ 2023-10-30T13:08:24.467 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 61601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-30T13:08:24.468 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 67193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-30T13:08:24.468 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 73409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-30T13:08:24.468 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 68177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-30T13:08:24.468 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/ 2023-10-30T13:08:24.468 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-10-30T13:08:24.468 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-10-30T13:08:24.469 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-30T13:08:24.469 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-10-30T13:08:24.469 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 486961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-30T13:08:24.469 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/ 2023-10-30T13:08:24.469 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 103337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-30T13:08:24.469 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 166161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-30T13:08:24.470 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 66329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-30T13:08:24.470 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 78801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-30T13:08:24.470 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/tehuti/ 2023-10-30T13:08:24.470 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 50057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-30T13:08:24.470 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ti/ 2023-10-30T13:08:24.471 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 86529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-30T13:08:24.471 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/ 2023-10-30T13:08:24.471 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 80273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-30T13:08:24.471 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 83633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-30T13:08:24.471 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/ 2023-10-30T13:08:24.471 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 46241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-30T13:08:24.472 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-30T13:08:24.472 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/xircom/ 2023-10-30T13:08:24.472 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 60057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-30T13:08:24.472 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/ 2023-10-30T13:08:24.472 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/defxx.ko 2023-10-30T13:08:24.472 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/skfp/ 2023-10-30T13:08:24.473 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 216985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/skfp/skfp.ko 2023-10-30T13:08:24.473 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 77345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/geneve.ko 2023-10-30T13:08:24.473 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/hyperv/ 2023-10-30T13:08:24.473 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 235521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-30T13:08:24.473 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ieee802154/ 2023-10-30T13:08:24.474 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-30T13:08:24.474 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ifb.ko 2023-10-30T13:08:24.474 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ipvlan/ 2023-10-30T13:08:24.474 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 83769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-30T13:08:24.474 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 62209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/macvlan.ko 2023-10-30T13:08:24.474 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/macvtap.ko 2023-10-30T13:08:24.475 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/ 2023-10-30T13:08:24.475 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-30T13:08:24.475 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-30T13:08:24.475 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-gpio.ko 2023-10-30T13:08:24.475 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-30T13:08:24.476 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio.ko 2023-10-30T13:08:24.476 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mii.ko 2023-10-30T13:08:24.476 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 43953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/netconsole.ko 2023-10-30T13:08:24.476 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/nlmon.ko 2023-10-30T13:08:24.476 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/ 2023-10-30T13:08:24.477 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-30T13:08:24.477 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-30T13:08:24.477 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/ 2023-10-30T13:08:24.477 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/amd.ko 2023-10-30T13:08:24.477 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 50097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-30T13:08:24.478 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 44505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-30T13:08:24.478 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm87xx.ko 2023-10-30T13:08:24.478 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 42241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/broadcom.ko 2023-10-30T13:08:24.478 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/cicada.ko 2023-10-30T13:08:24.478 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/davicom.ko 2023-10-30T13:08:24.479 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/et1011c.ko 2023-10-30T13:08:24.479 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/icplus.ko 2023-10-30T13:08:24.479 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/lxt.ko 2023-10-30T13:08:24.479 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 82049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/marvell.ko 2023-10-30T13:08:24.479 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 76177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/micrel.ko 2023-10-30T13:08:24.479 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/national.ko 2023-10-30T13:08:24.480 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 97857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/phylink.ko 2023-10-30T13:08:24.480 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/qsemi.ko 2023-10-30T13:08:24.480 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 50065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/realtek.ko 2023-10-30T13:08:24.480 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/smsc.ko 2023-10-30T13:08:24.480 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/ste10Xp.ko 2023-10-30T13:08:24.481 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/vitesse.ko 2023-10-30T13:08:24.481 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/plip/ 2023-10-30T13:08:24.481 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 43569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/plip/plip.ko 2023-10-30T13:08:24.481 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ 2023-10-30T13:08:24.481 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-30T13:08:24.482 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_async.ko 2023-10-30T13:08:24.482 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-30T13:08:24.482 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-30T13:08:24.482 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-30T13:08:24.482 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 42545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pppoe.ko 2023-10-30T13:08:24.483 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pppox.ko 2023-10-30T13:08:24.483 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pptp.ko 2023-10-30T13:08:24.483 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/sb1000.ko 2023-10-30T13:08:24.483 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/slip/ 2023-10-30T13:08:24.483 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 42465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/slip/slip.ko 2023-10-30T13:08:24.484 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/sungem_phy.ko 2023-10-30T13:08:24.484 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 64385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/tap.ko 2023-10-30T13:08:24.484 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/ 2023-10-30T13:08:24.484 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 106169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team.ko 2023-10-30T13:08:24.484 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-30T13:08:24.485 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-30T13:08:24.485 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-30T13:08:24.485 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_random.ko 2023-10-30T13:08:24.485 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-30T13:08:24.485 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 87017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/veth.ko 2023-10-30T13:08:24.485 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vmxnet3/ 2023-10-30T13:08:24.486 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 144297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-30T13:08:24.486 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vxlan/ 2023-10-30T13:08:24.486 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 297329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vxlan/vxlan.ko 2023-10-30T13:08:24.486 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/xen-netback/ 2023-10-30T13:08:24.486 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 144649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-30T13:08:24.487 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ntb/ 2023-10-30T13:08:24.487 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ntb/ntb.ko 2023-10-30T13:08:24.487 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/ 2023-10-30T13:08:24.487 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/ 2023-10-30T13:08:24.487 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 264937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/nvme-core.ko 2023-10-30T13:08:24.488 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 102209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/nvme.ko 2023-10-30T13:08:24.488 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/ 2023-10-30T13:08:24.488 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 133841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport.ko 2023-10-30T13:08:24.488 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_cs.ko 2023-10-30T13:08:24.488 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 84465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_pc.ko 2023-10-30T13:08:24.489 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_serial.ko 2023-10-30T13:08:24.489 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/ 2023-10-30T13:08:24.489 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/ 2023-10-30T13:08:24.489 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-30T13:08:24.489 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-10-30T13:08:24.490 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-10-30T13:08:24.490 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/pci-stub.ko 2023-10-30T13:08:24.490 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 46529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/xen-pcifront.ko 2023-10-30T13:08:24.490 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/ 2023-10-30T13:08:24.490 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/i82092.ko 2023-10-30T13:08:24.491 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 145809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia.ko 2023-10-30T13:08:24.491 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 62137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia_core.ko 2023-10-30T13:08:24.491 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-10-30T13:08:24.491 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pd6729.ko 2023-10-30T13:08:24.491 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 112889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-30T13:08:24.491 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/ 2023-10-30T13:08:24.492 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/broadcom/ 2023-10-30T13:08:24.492 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-10-30T13:08:24.492 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/samsung/ 2023-10-30T13:08:24.492 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-10-30T13:08:24.492 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/ 2023-10-30T13:08:24.493 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/intel/ 2023-10-30T13:08:24.493 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 70153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-30T13:08:24.493 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/ 2023-10-30T13:08:24.493 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/ 2023-10-30T13:08:24.493 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 50889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-10-30T13:08:24.494 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec.ko 2023-10-30T13:08:24.494 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-10-30T13:08:24.494 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-10-30T13:08:24.494 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-10-30T13:08:24.494 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-10-30T13:08:24.494 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-10-30T13:08:24.495 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/ 2023-10-30T13:08:24.495 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 67097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-30T13:08:24.495 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/acerhdf.ko 2023-10-30T13:08:24.495 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-30T13:08:24.495 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 59665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-30T13:08:24.496 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-30T13:08:24.496 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 127569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-30T13:08:24.496 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-30T13:08:24.496 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 49121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-30T13:08:24.496 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-30T13:08:24.496 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 40265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-30T13:08:24.497 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-30T13:08:24.497 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/hdaps.ko 2023-10-30T13:08:24.497 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/ibm_rtl.ko 2023-10-30T13:08:24.497 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/ 2023-10-30T13:08:24.497 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-30T13:08:24.498 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-10-30T13:08:24.498 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 43497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel_ips.ko 2023-10-30T13:08:24.498 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-30T13:08:24.498 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-30T13:08:24.498 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-30T13:08:24.499 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 42329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-30T13:08:24.499 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-30T13:08:24.499 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-30T13:08:24.499 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-30T13:08:24.499 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-30T13:08:24.499 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-30T13:08:24.500 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 57569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/wmi.ko 2023-10-30T13:08:24.500 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/ 2023-10-30T13:08:24.500 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ 2023-10-30T13:08:24.500 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/88pm860x_battery.ko 2023-10-30T13:08:24.500 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/88pm860x_charger.ko 2023-10-30T13:08:24.500 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 55057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq2415x_charger.ko 2023-10-30T13:08:24.501 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 54081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq24190_charger.ko 2023-10-30T13:08:24.501 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq24735-charger.ko 2023-10-30T13:08:24.501 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-10-30T13:08:24.501 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/da9030_battery.ko 2023-10-30T13:08:24.501 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/da9052-battery.ko 2023-10-30T13:08:24.502 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2760_battery.ko 2023-10-30T13:08:24.502 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2780_battery.ko 2023-10-30T13:08:24.502 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2781_battery.ko 2023-10-30T13:08:24.502 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2782_battery.ko 2023-10-30T13:08:24.502 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/gpio-charger.ko 2023-10-30T13:08:24.503 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/isp1704_charger.ko 2023-10-30T13:08:24.503 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/lp8727_charger.ko 2023-10-30T13:08:24.503 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max14577_charger.ko 2023-10-30T13:08:24.503 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max17040_battery.ko 2023-10-30T13:08:24.503 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max17042_battery.ko 2023-10-30T13:08:24.503 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max8903_charger.ko 2023-10-30T13:08:24.504 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max8925_power.ko 2023-10-30T13:08:24.504 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/pcf50633-charger.ko 2023-10-30T13:08:24.504 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/sbs-battery.ko 2023-10-30T13:08:24.504 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/test_power.ko 2023-10-30T13:08:24.504 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/tps65090-charger.ko 2023-10-30T13:08:24.504 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm831x_backup.ko 2023-10-30T13:08:24.505 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm831x_power.ko 2023-10-30T13:08:24.505 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm8350_power.ko 2023-10-30T13:08:24.505 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/ 2023-10-30T13:08:24.505 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 57729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-30T13:08:24.505 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-30T13:08:24.505 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/ 2023-10-30T13:08:24.506 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/ 2023-10-30T13:08:24.506 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-30T13:08:24.506 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-30T13:08:24.506 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps_parport.ko 2023-10-30T13:08:24.506 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/pps_core.ko 2023-10-30T13:08:24.507 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ 2023-10-30T13:08:24.507 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 66385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ptp.ko 2023-10-30T13:08:24.507 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ptp_kvm.ko 2023-10-30T13:08:24.507 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/ 2023-10-30T13:08:24.507 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-88pm80x.ko 2023-10-30T13:08:24.508 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-88pm860x.ko 2023-10-30T13:08:24.508 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-30T13:08:24.508 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9052.ko 2023-10-30T13:08:24.508 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9055.ko 2023-10-30T13:08:24.508 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9063.ko 2023-10-30T13:08:24.508 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-30T13:08:24.509 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 68577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-30T13:08:24.509 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-30T13:08:24.509 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-30T13:08:24.509 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-30T13:08:24.509 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-30T13:08:24.509 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-30T13:08:24.510 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-30T13:08:24.510 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-30T13:08:24.510 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-em3027.ko 2023-10-30T13:08:24.510 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-30T13:08:24.510 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-30T13:08:24.511 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-30T13:08:24.511 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-lp8788.ko 2023-10-30T13:08:24.511 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-30T13:08:24.511 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-30T13:08:24.511 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-30T13:08:24.512 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t86.ko 2023-10-30T13:08:24.512 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max6900.ko 2023-10-30T13:08:24.512 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8907.ko 2023-10-30T13:08:24.512 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8925.ko 2023-10-30T13:08:24.512 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8997.ko 2023-10-30T13:08:24.512 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8998.ko 2023-10-30T13:08:24.513 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-10-30T13:08:24.513 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-30T13:08:24.513 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-palmas.ko 2023-10-30T13:08:24.513 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 46633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-30T13:08:24.513 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf50633.ko 2023-10-30T13:08:24.514 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-30T13:08:24.514 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-30T13:08:24.514 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-30T13:08:24.514 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-30T13:08:24.514 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rc5t583.ko 2023-10-30T13:08:24.514 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-30T13:08:24.515 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-30T13:08:24.515 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-30T13:08:24.515 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-30T13:08:24.515 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-30T13:08:24.515 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-s35390a.ko 2023-10-30T13:08:24.516 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-30T13:08:24.516 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-tps6586x.ko 2023-10-30T13:08:24.516 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-tps65910.ko 2023-10-30T13:08:24.516 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-wm831x.ko 2023-10-30T13:08:24.516 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-wm8350.ko 2023-10-30T13:08:24.516 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-x1205.ko 2023-10-30T13:08:24.517 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ 2023-10-30T13:08:24.517 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 77617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-9xxx.ko 2023-10-30T13:08:24.517 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 57665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-sas.ko 2023-10-30T13:08:24.517 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 55169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-xxxx.ko 2023-10-30T13:08:24.517 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 135185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/BusLogic.ko 2023-10-30T13:08:24.517 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/a100u2w.ko 2023-10-30T13:08:24.518 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aacraid/ 2023-10-30T13:08:24.518 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 272305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-30T13:08:24.518 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 100617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/advansys.ko 2023-10-30T13:08:24.518 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/ 2023-10-30T13:08:24.518 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 356265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-30T13:08:24.519 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 295801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-30T13:08:24.519 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic94xx/ 2023-10-30T13:08:24.519 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 189185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-10-30T13:08:24.519 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/am53c974.ko 2023-10-30T13:08:24.519 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/arcmsr/ 2023-10-30T13:08:24.519 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 106881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-30T13:08:24.520 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/atp870u.ko 2023-10-30T13:08:24.520 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/be2iscsi/ 2023-10-30T13:08:24.520 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 246985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-30T13:08:24.520 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bfa/ 2023-10-30T13:08:24.520 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1171945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bfa/bfa.ko 2023-10-30T13:08:24.520 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2fc/ 2023-10-30T13:08:24.521 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 237273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-30T13:08:24.521 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2i/ 2023-10-30T13:08:24.521 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 115369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-30T13:08:24.521 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 40585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ch.ko 2023-10-30T13:08:24.521 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/csiostor/ 2023-10-30T13:08:24.521 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 308921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-30T13:08:24.522 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/ 2023-10-30T13:08:24.522 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-10-30T13:08:24.522 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 76145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-30T13:08:24.522 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-10-30T13:08:24.522 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 101449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-30T13:08:24.523 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 146689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-30T13:08:24.523 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 75553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/dc395x.ko 2023-10-30T13:08:24.523 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/dmx3191d.ko 2023-10-30T13:08:24.523 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esas2r/ 2023-10-30T13:08:24.523 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 212425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-30T13:08:24.524 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 60505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esp_scsi.ko 2023-10-30T13:08:24.524 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/ 2023-10-30T13:08:24.524 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 78209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-30T13:08:24.524 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 158505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-30T13:08:24.524 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fdomain.ko 2023-10-30T13:08:24.524 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fnic/ 2023-10-30T13:08:24.525 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 263489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fnic/fnic.ko 2023-10-30T13:08:24.525 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 201817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hpsa.ko 2023-10-30T13:08:24.525 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hptiop.ko 2023-10-30T13:08:24.525 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 49041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hv_storvsc.ko 2023-10-30T13:08:24.525 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/imm.ko 2023-10-30T13:08:24.525 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 44057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/initio.ko 2023-10-30T13:08:24.526 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 266537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ipr.ko 2023-10-30T13:08:24.526 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 101881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ips.ko 2023-10-30T13:08:24.526 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/isci/ 2023-10-30T13:08:24.526 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 317153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/isci/isci.ko 2023-10-30T13:08:24.526 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-30T13:08:24.527 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 52961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-30T13:08:24.527 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libfc/ 2023-10-30T13:08:24.527 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 314865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libfc/libfc.ko 2023-10-30T13:08:24.527 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 145585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libiscsi.ko 2023-10-30T13:08:24.527 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 53521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-30T13:08:24.527 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libsas/ 2023-10-30T13:08:24.528 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 216665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libsas/libsas.ko 2023-10-30T13:08:24.528 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/lpfc/ 2023-10-30T13:08:24.528 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 2197249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-30T13:08:24.528 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/ 2023-10-30T13:08:24.528 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 73569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-30T13:08:24.528 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-30T13:08:24.529 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 310617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-30T13:08:24.529 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 84721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid.ko 2023-10-30T13:08:24.529 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mpt3sas/ 2023-10-30T13:08:24.529 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 659777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-30T13:08:24.529 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvsas/ 2023-10-30T13:08:24.530 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 119561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-30T13:08:24.530 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 78129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvumi.ko 2023-10-30T13:08:24.530 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/ 2023-10-30T13:08:24.530 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 81625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-10-30T13:08:24.530 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-10-30T13:08:24.530 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-10-30T13:08:24.531 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-10-30T13:08:24.531 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pm8001/ 2023-10-30T13:08:24.531 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 364105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-30T13:08:24.531 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 123617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pmcraid.ko 2023-10-30T13:08:24.531 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ppa.ko 2023-10-30T13:08:24.532 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 55361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla1280.ko 2023-10-30T13:08:24.532 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/ 2023-10-30T13:08:24.532 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1876625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-30T13:08:24.532 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 85289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-30T13:08:24.532 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla4xxx/ 2023-10-30T13:08:24.532 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 559033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-30T13:08:24.533 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qlogicfas408.ko 2023-10-30T13:08:24.533 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/raid_class.ko 2023-10-30T13:08:24.533 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 238825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_debug.ko 2023-10-30T13:08:24.533 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 173937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-30T13:08:24.533 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 279313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-30T13:08:24.534 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 97729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-30T13:08:24.534 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 73881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-30T13:08:24.534 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 44993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-30T13:08:24.534 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ses.ko 2023-10-30T13:08:24.534 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 115889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/st.ko 2023-10-30T13:08:24.534 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 46825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/stex.ko 2023-10-30T13:08:24.535 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/sym53c8xx_2/ 2023-10-30T13:08:24.535 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 162121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-30T13:08:24.535 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 42297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/virtio_scsi.ko 2023-10-30T13:08:24.535 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 50321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-30T13:08:24.535 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 40025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/wd719x.ko 2023-10-30T13:08:24.535 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 50025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/xen-scsifront.ko 2023-10-30T13:08:24.536 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ssb/ 2023-10-30T13:08:24.536 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 129441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ssb/ssb.ko 2023-10-30T13:08:24.536 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/ 2023-10-30T13:08:24.536 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/iscsi/ 2023-10-30T13:08:24.536 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 736601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-30T13:08:24.537 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/loopback/ 2023-10-30T13:08:24.537 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 49985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-30T13:08:24.537 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_file.ko 2023-10-30T13:08:24.537 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 43025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_iblock.ko 2023-10-30T13:08:24.537 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 930321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_mod.ko 2023-10-30T13:08:24.537 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 44305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_pscsi.ko 2023-10-30T13:08:24.538 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 122521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_user.ko 2023-10-30T13:08:24.538 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/tcm_fc/ 2023-10-30T13:08:24.538 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 75593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-30T13:08:24.538 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/ 2023-10-30T13:08:24.538 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/ 2023-10-30T13:08:24.539 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/ 2023-10-30T13:08:24.539 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-30T13:08:24.539 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-30T13:08:24.539 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-30T13:08:24.539 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-30T13:08:24.539 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-30T13:08:24.540 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-30T13:08:24.540 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-30T13:08:24.540 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-30T13:08:24.540 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-30T13:08:24.540 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-30T13:08:24.541 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-30T13:08:24.541 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-30T13:08:24.541 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-10-30T13:08:24.541 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-30T13:08:24.541 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-30T13:08:24.541 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-30T13:08:24.542 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ 2023-10-30T13:08:24.542 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ipwireless/ 2023-10-30T13:08:24.542 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 78425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-30T13:08:24.542 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 57865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/moxa.ko 2023-10-30T13:08:24.542 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 50953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/mxser.ko 2023-10-30T13:08:24.543 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 108601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/n_gsm.ko 2023-10-30T13:08:24.543 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/n_hdlc.ko 2023-10-30T13:08:24.543 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 67929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/nozomi.ko 2023-10-30T13:08:24.543 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/ 2023-10-30T13:08:24.543 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/ 2023-10-30T13:08:24.544 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-10-30T13:08:24.544 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 59497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-30T13:08:24.544 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-10-30T13:08:24.544 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/altera_uart.ko 2023-10-30T13:08:24.544 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/arc_uart.ko 2023-10-30T13:08:24.545 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 72425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-10-30T13:08:24.545 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/jsm/ 2023-10-30T13:08:24.545 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 103073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-30T13:08:24.545 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/rp2.ko 2023-10-30T13:08:24.545 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 52817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/sc16is7xx.ko 2023-10-30T13:08:24.545 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/ 2023-10-30T13:08:24.546 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/core/ 2023-10-30T13:08:24.546 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 531265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-30T13:08:24.546 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ 2023-10-30T13:08:24.546 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-30T13:08:24.546 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-10-30T13:08:24.547 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/ 2023-10-30T13:08:24.547 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 44513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio.ko 2023-10-30T13:08:24.547 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_aec.ko 2023-10-30T13:08:24.547 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_cif.ko 2023-10-30T13:08:24.547 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_dmem_genirq.ko 2023-10-30T13:08:24.548 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_mf624.ko 2023-10-30T13:08:24.548 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_netx.ko 2023-10-30T13:08:24.548 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_pci_generic.ko 2023-10-30T13:08:24.548 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-10-30T13:08:24.548 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_sercos3.ko 2023-10-30T13:08:24.548 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/ 2023-10-30T13:08:24.549 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/c67x00/ 2023-10-30T13:08:24.549 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 73297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/c67x00/c67x00.ko 2023-10-30T13:08:24.549 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ 2023-10-30T13:08:24.549 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 144377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-10-30T13:08:24.549 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-10-30T13:08:24.550 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-10-30T13:08:24.550 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-10-30T13:08:24.550 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/ 2023-10-30T13:08:24.550 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 87089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/cdc-acm.ko 2023-10-30T13:08:24.550 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-30T13:08:24.550 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/usblp.ko 2023-10-30T13:08:24.551 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 89857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/usbtmc.ko 2023-10-30T13:08:24.551 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/common/ 2023-10-30T13:08:24.551 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/common/ulpi.ko 2023-10-30T13:08:24.551 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc2/ 2023-10-30T13:08:24.551 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-10-30T13:08:24.552 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/ 2023-10-30T13:08:24.552 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-10-30T13:08:24.552 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-10-30T13:08:24.552 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 444553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3.ko 2023-10-30T13:08:24.552 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/ 2023-10-30T13:08:24.552 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/ 2023-10-30T13:08:24.553 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 60177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/u_ether.ko 2023-10-30T13:08:24.553 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/u_serial.ko 2023-10-30T13:08:24.553 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-10-30T13:08:24.553 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 42449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-10-30T13:08:24.553 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-10-30T13:08:24.554 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-10-30T13:08:24.554 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 92489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-10-30T13:08:24.554 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-10-30T13:08:24.554 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 122081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-10-30T13:08:24.554 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 57617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-10-30T13:08:24.554 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-10-30T13:08:24.555 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-10-30T13:08:24.555 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 57729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-10-30T13:08:24.555 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 85609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-10-30T13:08:24.555 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-10-30T13:08:24.555 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 58113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-10-30T13:08:24.556 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 67337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-10-30T13:08:24.556 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/ 2023-10-30T13:08:24.556 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-10-30T13:08:24.556 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-10-30T13:08:24.556 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-10-30T13:08:24.557 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-10-30T13:08:24.557 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-10-30T13:08:24.557 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-10-30T13:08:24.557 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-10-30T13:08:24.557 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-10-30T13:08:24.557 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-10-30T13:08:24.558 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-10-30T13:08:24.558 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-10-30T13:08:24.558 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-10-30T13:08:24.558 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 62185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-10-30T13:08:24.558 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-10-30T13:08:24.558 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 161225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/libcomposite.ko 2023-10-30T13:08:24.559 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/ 2023-10-30T13:08:24.559 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-10-30T13:08:24.559 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/bdc/ 2023-10-30T13:08:24.559 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 157681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-10-30T13:08:24.559 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-10-30T13:08:24.560 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 52721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-10-30T13:08:24.560 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 69705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-10-30T13:08:24.560 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 56033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-10-30T13:08:24.560 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 76057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/net2272.ko 2023-10-30T13:08:24.560 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 75073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/net2280.ko 2023-10-30T13:08:24.560 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 73169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-10-30T13:08:24.561 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 65809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-10-30T13:08:24.561 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 66097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-10-30T13:08:24.561 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 81505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-10-30T13:08:24.561 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 156073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-10-30T13:08:24.561 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/ 2023-10-30T13:08:24.562 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/bcma-hcd.ko 2023-10-30T13:08:24.562 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 80113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/isp116x-hcd.ko 2023-10-30T13:08:24.562 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 87353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-10-30T13:08:24.562 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 78873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-10-30T13:08:24.562 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-30T13:08:24.562 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/sl811_cs.ko 2023-10-30T13:08:24.563 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/ssb-hcd.ko 2023-10-30T13:08:24.563 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-10-30T13:08:24.563 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/ 2023-10-30T13:08:24.563 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/mdc800.ko 2023-10-30T13:08:24.563 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/microtek.ko 2023-10-30T13:08:24.564 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ 2023-10-30T13:08:24.564 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/adutux.ko 2023-10-30T13:08:24.564 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/appledisplay.ko 2023-10-30T13:08:24.564 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-10-30T13:08:24.564 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/cytherm.ko 2023-10-30T13:08:24.565 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ehset.ko 2023-10-30T13:08:24.565 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/emi26.ko 2023-10-30T13:08:24.565 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/emi62.ko 2023-10-30T13:08:24.565 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ezusb.ko 2023-10-30T13:08:24.565 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/idmouse.ko 2023-10-30T13:08:24.565 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/iowarrior.ko 2023-10-30T13:08:24.566 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-30T13:08:24.566 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ldusb.ko 2023-10-30T13:08:24.566 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/legousbtower.ko 2023-10-30T13:08:24.566 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/lvstest.ko 2023-10-30T13:08:24.566 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/sisusbvga/ 2023-10-30T13:08:24.567 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 64841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-30T13:08:24.567 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-30T13:08:24.567 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usb3503.ko 2023-10-30T13:08:24.567 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usblcd.ko 2023-10-30T13:08:24.567 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-30T13:08:24.567 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 70873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usbtest.ko 2023-10-30T13:08:24.568 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/uss720.ko 2023-10-30T13:08:24.568 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/yurex.ko 2023-10-30T13:08:24.568 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/mon/ 2023-10-30T13:08:24.568 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 72953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/mon/usbmon.ko 2023-10-30T13:08:24.568 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/musb/ 2023-10-30T13:08:24.568 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 301545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/musb/musb_hdrc.ko 2023-10-30T13:08:24.569 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/ 2023-10-30T13:08:24.569 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-generic.ko 2023-10-30T13:08:24.569 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-10-30T13:08:24.569 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-isp1301.ko 2023-10-30T13:08:24.569 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-tahvo.ko 2023-10-30T13:08:24.570 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ 2023-10-30T13:08:24.570 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/aircable.ko 2023-10-30T13:08:24.570 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ark3116.ko 2023-10-30T13:08:24.570 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-30T13:08:24.570 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ch341.ko 2023-10-30T13:08:24.570 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 70257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cp210x.ko 2023-10-30T13:08:24.571 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cyberjack.ko 2023-10-30T13:08:24.571 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 58217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-30T13:08:24.571 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 49817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-30T13:08:24.571 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/empeg.ko 2023-10-30T13:08:24.572 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 44337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/f81232.ko 2023-10-30T13:08:24.572 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 175137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-30T13:08:24.572 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 53113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-30T13:08:24.572 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 137081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-30T13:08:24.572 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 124833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/io_ti.ko 2023-10-30T13:08:24.572 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 67769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ipaq.ko 2023-10-30T13:08:24.573 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ipw.ko 2023-10-30T13:08:24.573 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ir-usb.ko 2023-10-30T13:08:24.573 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 58233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-30T13:08:24.573 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 107521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/keyspan.ko 2023-10-30T13:08:24.573 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-30T13:08:24.573 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-30T13:08:24.574 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-30T13:08:24.574 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mct_u232.ko 2023-10-30T13:08:24.574 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/metro-usb.ko 2023-10-30T13:08:24.574 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 66449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mos7720.ko 2023-10-30T13:08:24.575 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 78217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mos7840.ko 2023-10-30T13:08:24.575 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 56337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mxuport.ko 2023-10-30T13:08:24.575 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/navman.ko 2023-10-30T13:08:24.575 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/omninet.ko 2023-10-30T13:08:24.575 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/opticon.ko 2023-10-30T13:08:24.575 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 190633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/option.ko 2023-10-30T13:08:24.576 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/oti6858.ko 2023-10-30T13:08:24.576 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 54353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/pl2303.ko 2023-10-30T13:08:24.576 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/qcaux.ko 2023-10-30T13:08:24.576 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/qcserial.ko 2023-10-30T13:08:24.576 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/quatech2.ko 2023-10-30T13:08:24.577 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/safe_serial.ko 2023-10-30T13:08:24.577 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 51625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/sierra.ko 2023-10-30T13:08:24.577 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-30T13:08:24.577 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ssu100.ko 2023-10-30T13:08:24.577 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/symbolserial.ko 2023-10-30T13:08:24.577 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 61489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-30T13:08:24.578 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-30T13:08:24.578 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb_debug.ko 2023-10-30T13:08:24.578 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-30T13:08:24.578 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 125609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usbserial.ko 2023-10-30T13:08:24.578 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/visor.ko 2023-10-30T13:08:24.579 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/whiteheat.ko 2023-10-30T13:08:24.579 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/wishbone-serial.ko 2023-10-30T13:08:24.579 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-30T13:08:24.579 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ 2023-10-30T13:08:24.579 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 52913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/uas.ko 2023-10-30T13:08:24.579 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-30T13:08:24.580 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-30T13:08:24.580 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-30T13:08:24.580 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-30T13:08:24.580 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-30T13:08:24.580 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-30T13:08:24.580 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-30T13:08:24.581 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-karma.ko 2023-10-30T13:08:24.581 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-30T13:08:24.581 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-30T13:08:24.581 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-30T13:08:24.581 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-30T13:08:24.582 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-30T13:08:24.582 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 179969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/usb-storage.ko 2023-10-30T13:08:24.582 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/ 2023-10-30T13:08:24.582 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 67481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-30T13:08:24.582 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 72177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-30T13:08:24.583 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 107001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-30T13:08:24.583 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/ 2023-10-30T13:08:24.583 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/ 2023-10-30T13:08:24.583 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 154297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-30T13:08:24.583 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-30T13:08:24.584 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 108737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/vfio.ko 2023-10-30T13:08:24.584 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 77609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-30T13:08:24.584 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/ 2023-10-30T13:08:24.584 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 113873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost.ko 2023-10-30T13:08:24.584 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-30T13:08:24.584 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 58089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_net.ko 2023-10-30T13:08:24.585 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 89249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_scsi.ko 2023-10-30T13:08:24.585 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/ 2023-10-30T13:08:24.585 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/ 2023-10-30T13:08:24.585 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-10-30T13:08:24.585 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/aat2870_bl.ko 2023-10-30T13:08:24.586 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp5520_bl.ko 2023-10-30T13:08:24.586 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp8860_bl.ko 2023-10-30T13:08:24.586 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp8870_bl.ko 2023-10-30T13:08:24.586 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/apple_bl.ko 2023-10-30T13:08:24.586 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/as3711_bl.ko 2023-10-30T13:08:24.587 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/bd6107.ko 2023-10-30T13:08:24.587 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/da903x_bl.ko 2023-10-30T13:08:24.587 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/da9052_bl.ko 2023-10-30T13:08:24.587 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/gpio_backlight.ko 2023-10-30T13:08:24.587 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/kb3886_bl.ko 2023-10-30T13:08:24.587 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lcd.ko 2023-10-30T13:08:24.588 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lm3533_bl.ko 2023-10-30T13:08:24.588 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lm3639_bl.ko 2023-10-30T13:08:24.588 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lv5207lp.ko 2023-10-30T13:08:24.588 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/max8925_bl.ko 2023-10-30T13:08:24.588 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/pandora_bl.ko 2023-10-30T13:08:24.588 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-10-30T13:08:24.589 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-30T13:08:24.589 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/wm831x_bl.ko 2023-10-30T13:08:24.589 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/ 2023-10-30T13:08:24.589 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ 2023-10-30T13:08:24.589 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ds2482.ko 2023-10-30T13:08:24.589 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ds2490.ko 2023-10-30T13:08:24.590 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/matrox_w1.ko 2023-10-30T13:08:24.590 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/w1-gpio.ko 2023-10-30T13:08:24.590 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/ 2023-10-30T13:08:24.590 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-30T13:08:24.590 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-30T13:08:24.591 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-30T13:08:24.591 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-30T13:08:24.591 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-30T13:08:24.591 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-30T13:08:24.591 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-30T13:08:24.591 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-30T13:08:24.592 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-30T13:08:24.592 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-30T13:08:24.592 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 56425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-30T13:08:24.592 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 102673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/wire.ko 2023-10-30T13:08:24.592 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ 2023-10-30T13:08:24.592 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/acquirewdt.ko 2023-10-30T13:08:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/advantechwdt.ko 2023-10-30T13:08:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-30T13:08:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-30T13:08:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/cpu5wdt.ko 2023-10-30T13:08:24.593 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9052_wdt.ko 2023-10-30T13:08:24.594 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9055_wdt.ko 2023-10-30T13:08:24.594 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9063_wdt.ko 2023-10-30T13:08:24.594 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/dw_wdt.ko 2023-10-30T13:08:24.594 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/eurotechwdt.ko 2023-10-30T13:08:24.594 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-30T13:08:24.595 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/hpwdt.ko 2023-10-30T13:08:24.595 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/i6300esb.ko 2023-10-30T13:08:24.595 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-30T13:08:24.595 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-30T13:08:24.595 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ib700wdt.ko 2023-10-30T13:08:24.595 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ibmasr.ko 2023-10-30T13:08:24.596 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-30T13:08:24.596 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-30T13:08:24.596 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/it87_wdt.ko 2023-10-30T13:08:24.596 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/kempld_wdt.ko 2023-10-30T13:08:24.596 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/machzwd.ko 2023-10-30T13:08:24.597 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/mena21_wdt.ko 2023-10-30T13:08:24.597 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-10-30T13:08:24.597 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/nv_tco.ko 2023-10-30T13:08:24.597 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-10-30T13:08:24.597 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pc87413_wdt.ko 2023-10-30T13:08:24.597 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-30T13:08:24.598 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-30T13:08:24.598 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/retu_wdt.ko 2023-10-30T13:08:24.598 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-10-30T13:08:24.598 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-10-30T13:08:24.598 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-30T13:08:24.599 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sc1200wdt.ko 2023-10-30T13:08:24.599 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-30T13:08:24.599 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-10-30T13:08:24.599 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/softdog.ko 2023-10-30T13:08:24.599 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-30T13:08:24.599 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/twl4030_wdt.ko 2023-10-30T13:08:24.600 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/via_wdt.ko 2023-10-30T13:08:24.600 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-30T13:08:24.600 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-30T13:08:24.600 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-30T13:08:24.600 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wafer5823wdt.ko 2023-10-30T13:08:24.601 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wdt_pci.ko 2023-10-30T13:08:24.601 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wm831x_wdt.ko 2023-10-30T13:08:24.601 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wm8350_wdt.ko 2023-10-30T13:08:24.601 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/xen_wdt.ko 2023-10-30T13:08:24.601 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/ 2023-10-30T13:08:24.602 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-evtchn.ko 2023-10-30T13:08:24.602 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-gntalloc.ko 2023-10-30T13:08:24.602 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-gntdev.ko 2023-10-30T13:08:24.602 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-pciback/ 2023-10-30T13:08:24.602 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 176441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-30T13:08:24.603 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 40121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-privcmd.ko 2023-10-30T13:08:24.603 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 71249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-scsiback.ko 2023-10-30T13:08:24.603 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xenfs/ 2023-10-30T13:08:24.603 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-30T13:08:24.603 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ 2023-10-30T13:08:24.603 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/9p/ 2023-10-30T13:08:24.604 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 166425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/9p/9p.ko 2023-10-30T13:08:24.604 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/adfs/ 2023-10-30T13:08:24.604 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 61073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/adfs/adfs.ko 2023-10-30T13:08:24.604 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/affs/ 2023-10-30T13:08:24.604 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 163593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/affs/affs.ko 2023-10-30T13:08:24.605 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/afs/ 2023-10-30T13:08:24.605 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 932817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/afs/kafs.ko 2023-10-30T13:08:24.605 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/autofs/ 2023-10-30T13:08:24.605 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 101057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/autofs/autofs4.ko 2023-10-30T13:08:24.605 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/befs/ 2023-10-30T13:08:24.605 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 64545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/befs/befs.ko 2023-10-30T13:08:24.606 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/bfs/ 2023-10-30T13:08:24.606 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/bfs/bfs.ko 2023-10-30T13:08:24.606 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/binfmt_misc.ko 2023-10-30T13:08:24.606 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/btrfs/ 2023-10-30T13:08:24.606 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 3402585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/btrfs/btrfs.ko 2023-10-30T13:08:24.607 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cachefiles/ 2023-10-30T13:08:24.607 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 306073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cachefiles/cachefiles.ko 2023-10-30T13:08:24.607 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ceph/ 2023-10-30T13:08:24.607 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1323945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ceph/ceph.ko 2023-10-30T13:08:24.607 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/coda/ 2023-10-30T13:08:24.608 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 100801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/coda/coda.ko 2023-10-30T13:08:24.608 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/configfs/ 2023-10-30T13:08:24.608 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 108393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/configfs/configfs.ko 2023-10-30T13:08:24.608 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cramfs/ 2023-10-30T13:08:24.608 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 29625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cramfs/cramfs.ko 2023-10-30T13:08:24.609 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/dlm/ 2023-10-30T13:08:24.609 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 642577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/dlm/dlm.ko 2023-10-30T13:08:24.609 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/efs/ 2023-10-30T13:08:24.609 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/efs/efs.ko 2023-10-30T13:08:24.609 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/f2fs/ 2023-10-30T13:08:24.610 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1505025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/f2fs/f2fs.ko 2023-10-30T13:08:24.610 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fat/ 2023-10-30T13:08:24.610 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fat/msdos.ko 2023-10-30T13:08:24.610 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/freevxfs/ 2023-10-30T13:08:24.610 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 42545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/freevxfs/freevxfs.ko 2023-10-30T13:08:24.610 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fscache/ 2023-10-30T13:08:24.611 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 226481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fscache/fscache.ko 2023-10-30T13:08:24.611 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fuse/ 2023-10-30T13:08:24.611 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fuse/cuse.ko 2023-10-30T13:08:24.611 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/gfs2/ 2023-10-30T13:08:24.611 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1231529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/gfs2/gfs2.ko 2023-10-30T13:08:24.612 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfs/ 2023-10-30T13:08:24.612 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 141209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfs/hfs.ko 2023-10-30T13:08:24.612 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfsplus/ 2023-10-30T13:08:24.612 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 222817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfsplus/hfsplus.ko 2023-10-30T13:08:24.612 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hpfs/ 2023-10-30T13:08:24.612 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 192329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hpfs/hpfs.ko 2023-10-30T13:08:24.613 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/isofs/ 2023-10-30T13:08:24.613 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 74065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/isofs/isofs.ko 2023-10-30T13:08:24.613 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/jfs/ 2023-10-30T13:08:24.613 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 419497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/jfs/jfs.ko 2023-10-30T13:08:24.613 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/lockd/ 2023-10-30T13:08:24.613 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 280313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/lockd/lockd.ko 2023-10-30T13:08:24.614 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/minix/ 2023-10-30T13:08:24.614 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 89193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/minix/minix.ko 2023-10-30T13:08:24.614 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/netfs/ 2023-10-30T13:08:24.614 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 124649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/netfs/netfs.ko 2023-10-30T13:08:24.614 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/ 2023-10-30T13:08:24.615 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/blocklayout/ 2023-10-30T13:08:24.615 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 81449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-30T13:08:24.615 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/filelayout/ 2023-10-30T13:08:24.615 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 59449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-30T13:08:24.615 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/flexfilelayout/ 2023-10-30T13:08:24.616 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 117513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-30T13:08:24.616 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1055649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfs.ko 2023-10-30T13:08:24.616 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 66441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv2.ko 2023-10-30T13:08:24.616 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 120177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv3.ko 2023-10-30T13:08:24.616 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 2041689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv4.ko 2023-10-30T13:08:24.616 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/ 2023-10-30T13:08:24.616 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/grace.ko 2023-10-30T13:08:24.617 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/nfs_acl.ko 2023-10-30T13:08:24.617 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfsd/ 2023-10-30T13:08:24.617 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1516809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfsd/nfsd.ko 2023-10-30T13:08:24.617 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nilfs2/ 2023-10-30T13:08:24.617 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 592833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nilfs2/nilfs2.ko 2023-10-30T13:08:24.617 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/ 2023-10-30T13:08:24.618 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-celtic.ko 2023-10-30T13:08:24.618 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-centeuro.ko 2023-10-30T13:08:24.618 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-croatian.ko 2023-10-30T13:08:24.618 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-cyrillic.ko 2023-10-30T13:08:24.618 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-gaelic.ko 2023-10-30T13:08:24.619 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-greek.ko 2023-10-30T13:08:24.619 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-iceland.ko 2023-10-30T13:08:24.619 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-inuit.ko 2023-10-30T13:08:24.619 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-roman.ko 2023-10-30T13:08:24.619 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-romanian.ko 2023-10-30T13:08:24.620 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-turkish.ko 2023-10-30T13:08:24.620 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_ascii.ko 2023-10-30T13:08:24.620 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1250.ko 2023-10-30T13:08:24.620 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1251.ko 2023-10-30T13:08:24.620 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1255.ko 2023-10-30T13:08:24.620 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp737.ko 2023-10-30T13:08:24.621 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp775.ko 2023-10-30T13:08:24.621 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp850.ko 2023-10-30T13:08:24.621 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp852.ko 2023-10-30T13:08:24.621 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp855.ko 2023-10-30T13:08:24.621 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp857.ko 2023-10-30T13:08:24.621 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp860.ko 2023-10-30T13:08:24.622 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp861.ko 2023-10-30T13:08:24.622 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp862.ko 2023-10-30T13:08:24.622 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp863.ko 2023-10-30T13:08:24.622 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp864.ko 2023-10-30T13:08:24.623 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp865.ko 2023-10-30T13:08:24.623 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp866.ko 2023-10-30T13:08:24.623 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp869.ko 2023-10-30T13:08:24.623 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp874.ko 2023-10-30T13:08:24.623 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 94137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp932.ko 2023-10-30T13:08:24.624 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 142673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp936.ko 2023-10-30T13:08:24.624 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 165745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp949.ko 2023-10-30T13:08:24.624 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 118121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp950.ko 2023-10-30T13:08:24.624 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_euc-jp.ko 2023-10-30T13:08:24.624 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-1.ko 2023-10-30T13:08:24.624 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-13.ko 2023-10-30T13:08:24.625 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-14.ko 2023-10-30T13:08:24.625 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-15.ko 2023-10-30T13:08:24.625 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-2.ko 2023-10-30T13:08:24.625 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-3.ko 2023-10-30T13:08:24.625 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-4.ko 2023-10-30T13:08:24.626 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-5.ko 2023-10-30T13:08:24.626 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-6.ko 2023-10-30T13:08:24.626 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-7.ko 2023-10-30T13:08:24.626 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-9.ko 2023-10-30T13:08:24.626 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-r.ko 2023-10-30T13:08:24.627 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-ru.ko 2023-10-30T13:08:24.627 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-u.ko 2023-10-30T13:08:24.627 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-30T13:08:24.627 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_utf8.ko 2023-10-30T13:08:24.627 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ntfs/ 2023-10-30T13:08:24.627 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 219417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ntfs/ntfs.ko 2023-10-30T13:08:24.628 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ 2023-10-30T13:08:24.628 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/cluster/ 2023-10-30T13:08:24.628 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 244409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-30T13:08:24.628 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlm/ 2023-10-30T13:08:24.628 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 440737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-30T13:08:24.628 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlmfs/ 2023-10-30T13:08:24.629 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 54049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-30T13:08:24.629 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 2592433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2.ko 2023-10-30T13:08:24.629 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-30T13:08:24.629 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-30T13:08:24.629 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-30T13:08:24.630 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/omfs/ 2023-10-30T13:08:24.630 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 53433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/omfs/omfs.ko 2023-10-30T13:08:24.630 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/overlayfs/ 2023-10-30T13:08:24.630 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 352153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/overlayfs/overlay.ko 2023-10-30T13:08:24.630 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/pstore/ 2023-10-30T13:08:24.631 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 54809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/pstore/ramoops.ko 2023-10-30T13:08:24.631 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx4/ 2023-10-30T13:08:24.631 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx4/qnx4.ko 2023-10-30T13:08:24.631 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx6/ 2023-10-30T13:08:24.631 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 46689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx6/qnx6.ko 2023-10-30T13:08:24.631 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/ 2023-10-30T13:08:24.632 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_tree.ko 2023-10-30T13:08:24.632 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_v1.ko 2023-10-30T13:08:24.632 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_v2.ko 2023-10-30T13:08:24.632 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/reiserfs/ 2023-10-30T13:08:24.632 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 532673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/reiserfs/reiserfs.ko 2023-10-30T13:08:24.632 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/romfs/ 2023-10-30T13:08:24.633 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/romfs/romfs.ko 2023-10-30T13:08:24.633 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/ 2023-10-30T13:08:24.633 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/client/ 2023-10-30T13:08:24.634 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 2668121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/client/cifs.ko 2023-10-30T13:08:24.634 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/ 2023-10-30T13:08:24.634 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/cifs_arc4.ko 2023-10-30T13:08:24.635 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/cifs_md4.ko 2023-10-30T13:08:24.635 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/squashfs/ 2023-10-30T13:08:24.635 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 134641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/squashfs/squashfs.ko 2023-10-30T13:08:24.635 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/sysv/ 2023-10-30T13:08:24.636 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 90721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/sysv/sysv.ko 2023-10-30T13:08:24.636 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/udf/ 2023-10-30T13:08:24.636 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 247737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/udf/udf.ko 2023-10-30T13:08:24.636 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ufs/ 2023-10-30T13:08:24.636 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 162497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ufs/ufs.ko 2023-10-30T13:08:24.637 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/xfs/ 2023-10-30T13:08:24.637 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 4010729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/xfs/xfs.ko 2023-10-30T13:08:24.637 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/ 2023-10-30T13:08:24.637 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/locking/ 2023-10-30T13:08:24.638 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 75489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/locking/locktorture.ko 2023-10-30T13:08:24.638 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/time/ 2023-10-30T13:08:24.638 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/time/test_udelay.ko 2023-10-30T13:08:24.638 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 60553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/torture.ko 2023-10-30T13:08:24.639 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ 2023-10-30T13:08:24.639 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 6833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc-itu-t.ko 2023-10-30T13:08:24.639 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 6529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc7.ko 2023-10-30T13:08:24.639 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc8.ko 2023-10-30T13:08:24.639 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/ 2023-10-30T13:08:24.640 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/libarc4.ko 2023-10-30T13:08:24.640 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/libdes.ko 2023-10-30T13:08:24.640 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/interval_tree_test.ko 2023-10-30T13:08:24.640 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/libcrc32c.ko 2023-10-30T13:08:24.640 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lru_cache.ko 2023-10-30T13:08:24.641 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/ 2023-10-30T13:08:24.641 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/lz4_compress.ko 2023-10-30T13:08:24.641 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/lz4hc_compress.ko 2023-10-30T13:08:24.641 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/math/ 2023-10-30T13:08:24.641 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 6617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/math/cordic.ko 2023-10-30T13:08:24.642 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/memory-notifier-error-inject.ko 2023-10-30T13:08:24.642 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/notifier-error-inject.ko 2023-10-30T13:08:24.642 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/percpu_test.ko 2023-10-30T13:08:24.642 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/pm-notifier-error-inject.ko 2023-10-30T13:08:24.642 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/raid6/ 2023-10-30T13:08:24.643 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 143433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/raid6/raid6_pq.ko 2023-10-30T13:08:24.643 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/rbtree_test.ko 2023-10-30T13:08:24.643 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/reed_solomon/ 2023-10-30T13:08:24.643 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-30T13:08:24.644 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test-kstrtox.ko 2023-10-30T13:08:24.644 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test-string_helpers.ko 2023-10-30T13:08:24.644 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 4760425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_bpf.ko 2023-10-30T13:08:24.644 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 49089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_firmware.ko 2023-10-30T13:08:24.644 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 6369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_module.ko 2023-10-30T13:08:24.645 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_user_copy.ko 2023-10-30T13:08:24.645 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_bm.ko 2023-10-30T13:08:24.645 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_fsm.ko 2023-10-30T13:08:24.645 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_kmp.ko 2023-10-30T13:08:24.645 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/xz/ 2023-10-30T13:08:24.646 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/xz/xz_dec_test.ko 2023-10-30T13:08:24.646 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/mm/ 2023-10-30T13:08:24.646 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/mm/hwpoison-inject.ko 2023-10-30T13:08:24.646 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ 2023-10-30T13:08:24.647 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/ 2023-10-30T13:08:24.647 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 55737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/6lowpan.ko 2023-10-30T13:08:24.647 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_dest.ko 2023-10-30T13:08:24.647 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_fragment.ko 2023-10-30T13:08:24.647 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_hop.ko 2023-10-30T13:08:24.648 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 6985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-30T13:08:24.648 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_mobility.ko 2023-10-30T13:08:24.648 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_routing.ko 2023-10-30T13:08:24.648 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_udp.ko 2023-10-30T13:08:24.648 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/ 2023-10-30T13:08:24.648 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/garp.ko 2023-10-30T13:08:24.649 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/mrp.ko 2023-10-30T13:08:24.649 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/p8022.ko 2023-10-30T13:08:24.649 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/psnap.ko 2023-10-30T13:08:24.649 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/stp.ko 2023-10-30T13:08:24.649 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/8021q/ 2023-10-30T13:08:24.650 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 81737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/8021q/8021q.ko 2023-10-30T13:08:24.650 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/ 2023-10-30T13:08:24.650 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 198337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet.ko 2023-10-30T13:08:24.650 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_fd.ko 2023-10-30T13:08:24.651 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_rdma.ko 2023-10-30T13:08:24.651 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_virtio.ko 2023-10-30T13:08:24.651 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/batman-adv/ 2023-10-30T13:08:24.651 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 649297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/batman-adv/batman-adv.ko 2023-10-30T13:08:24.651 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/ 2023-10-30T13:08:24.651 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 63809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/br_netfilter.ko 2023-10-30T13:08:24.652 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 777689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/bridge.ko 2023-10-30T13:08:24.652 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ 2023-10-30T13:08:24.652 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-30T13:08:24.652 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-30T13:08:24.652 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-30T13:08:24.653 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-30T13:08:24.653 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-30T13:08:24.653 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-30T13:08:24.653 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-30T13:08:24.653 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-30T13:08:24.654 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-30T13:08:24.654 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-30T13:08:24.654 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-30T13:08:24.654 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-30T13:08:24.655 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-30T13:08:24.655 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-30T13:08:24.655 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-30T13:08:24.655 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-30T13:08:24.655 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-30T13:08:24.655 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-30T13:08:24.656 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-30T13:08:24.656 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-30T13:08:24.656 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 55929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtables.ko 2023-10-30T13:08:24.656 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-30T13:08:24.657 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-30T13:08:24.657 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ceph/ 2023-10-30T13:08:24.657 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1047497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ceph/libceph.ko 2023-10-30T13:08:24.657 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/core/ 2023-10-30T13:08:24.657 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 125697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/core/pktgen.ko 2023-10-30T13:08:24.658 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/ 2023-10-30T13:08:24.658 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 194209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp.ko 2023-10-30T13:08:24.658 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_diag.ko 2023-10-30T13:08:24.658 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 57881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_ipv4.ko 2023-10-30T13:08:24.658 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 60337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_ipv6.ko 2023-10-30T13:08:24.659 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/ 2023-10-30T13:08:24.659 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 390473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/dsa_core.ko 2023-10-30T13:08:24.659 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_brcm.ko 2023-10-30T13:08:24.659 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_dsa.ko 2023-10-30T13:08:24.659 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_none.ko 2023-10-30T13:08:24.660 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_trailer.ko 2023-10-30T13:08:24.660 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/hsr/ 2023-10-30T13:08:24.660 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 99729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/hsr/hsr.ko 2023-10-30T13:08:24.660 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ 2023-10-30T13:08:24.660 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/6lowpan/ 2023-10-30T13:08:24.661 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 45233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-30T13:08:24.661 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 254625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ieee802154.ko 2023-10-30T13:08:24.661 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-30T13:08:24.661 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ 2023-10-30T13:08:24.661 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ah4.ko 2023-10-30T13:08:24.662 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/esp4.ko 2023-10-30T13:08:24.662 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 48401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/fou.ko 2023-10-30T13:08:24.662 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/gre.ko 2023-10-30T13:08:24.662 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 48657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/inet_diag.ko 2023-10-30T13:08:24.663 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 60633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_gre.ko 2023-10-30T13:08:24.663 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 61225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_tunnel.ko 2023-10-30T13:08:24.663 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_vti.ko 2023-10-30T13:08:24.663 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ipcomp.ko 2023-10-30T13:08:24.663 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ipip.ko 2023-10-30T13:08:24.663 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ 2023-10-30T13:08:24.664 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-30T13:08:24.664 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-30T13:08:24.664 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-30T13:08:24.664 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ip_tables.ko 2023-10-30T13:08:24.664 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-30T13:08:24.665 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-10-30T13:08:24.665 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-30T13:08:24.665 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-30T13:08:24.665 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-30T13:08:24.665 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-10-30T13:08:24.666 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-30T13:08:24.666 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-30T13:08:24.666 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-30T13:08:24.666 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-30T13:08:24.667 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-30T13:08:24.667 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-30T13:08:24.667 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-30T13:08:24.667 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-30T13:08:24.667 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-30T13:08:24.668 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-10-30T13:08:24.668 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-30T13:08:24.668 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-30T13:08:24.668 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-30T13:08:24.669 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-30T13:08:24.669 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-30T13:08:24.669 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_bic.ko 2023-10-30T13:08:24.669 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_dctcp.ko 2023-10-30T13:08:24.669 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_diag.ko 2023-10-30T13:08:24.670 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_highspeed.ko 2023-10-30T13:08:24.670 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_htcp.ko 2023-10-30T13:08:24.670 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_hybla.ko 2023-10-30T13:08:24.670 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_illinois.ko 2023-10-30T13:08:24.670 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_lp.ko 2023-10-30T13:08:24.671 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_scalable.ko 2023-10-30T13:08:24.671 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_vegas.ko 2023-10-30T13:08:24.671 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_veno.ko 2023-10-30T13:08:24.671 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_westwood.ko 2023-10-30T13:08:24.671 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_yeah.ko 2023-10-30T13:08:24.672 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tunnel4.ko 2023-10-30T13:08:24.672 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/udp_diag.ko 2023-10-30T13:08:24.672 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 41665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/udp_tunnel.ko 2023-10-30T13:08:24.672 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-30T13:08:24.672 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ 2023-10-30T13:08:24.673 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ah6.ko 2023-10-30T13:08:24.673 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/esp6.ko 2023-10-30T13:08:24.673 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/fou6.ko 2023-10-30T13:08:24.673 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 80897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_gre.ko 2023-10-30T13:08:24.673 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 93577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_tunnel.ko 2023-10-30T13:08:24.674 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-30T13:08:24.674 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 53273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_vti.ko 2023-10-30T13:08:24.674 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ipcomp6.ko 2023-10-30T13:08:24.674 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/mip6.ko 2023-10-30T13:08:24.674 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ 2023-10-30T13:08:24.675 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 42169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-30T13:08:24.675 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-10-30T13:08:24.675 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-30T13:08:24.675 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-30T13:08:24.675 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-30T13:08:24.676 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-30T13:08:24.676 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-30T13:08:24.676 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-30T13:08:24.676 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-30T13:08:24.676 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-30T13:08:24.677 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-30T13:08:24.677 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-30T13:08:24.677 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-30T13:08:24.677 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-30T13:08:24.677 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-30T13:08:24.678 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-30T13:08:24.678 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-30T13:08:24.678 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-30T13:08:24.678 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-30T13:08:24.678 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-30T13:08:24.679 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-30T13:08:24.679 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-30T13:08:24.679 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-30T13:08:24.679 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-30T13:08:24.680 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-30T13:08:24.680 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 74073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/sit.ko 2023-10-30T13:08:24.680 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/tunnel6.ko 2023-10-30T13:08:24.680 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-30T13:08:24.680 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/key/ 2023-10-30T13:08:24.681 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 82833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/key/af_key.ko 2023-10-30T13:08:24.681 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/ 2023-10-30T13:08:24.681 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 125281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_core.ko 2023-10-30T13:08:24.681 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-30T13:08:24.681 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 25953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_eth.ko 2023-10-30T13:08:24.682 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ip.ko 2023-10-30T13:08:24.682 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ip6.ko 2023-10-30T13:08:24.682 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_netlink.ko 2023-10-30T13:08:24.682 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 53241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ppp.ko 2023-10-30T13:08:24.683 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/lapb/ 2023-10-30T13:08:24.683 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 47977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/lapb/lapb.ko 2023-10-30T13:08:24.683 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/ 2023-10-30T13:08:24.683 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/llc.ko 2023-10-30T13:08:24.684 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 267777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/llc2.ko 2023-10-30T13:08:24.684 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mac802154/ 2023-10-30T13:08:24.684 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 290417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mac802154/mac802154.ko 2023-10-30T13:08:24.684 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mpls/ 2023-10-30T13:08:24.684 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mpls/mpls_gso.ko 2023-10-30T13:08:24.685 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ 2023-10-30T13:08:24.685 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ 2023-10-30T13:08:24.685 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 119585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set.ko 2023-10-30T13:08:24.685 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-30T13:08:24.685 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-30T13:08:24.686 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-30T13:08:24.686 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 100297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-30T13:08:24.686 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 101441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-30T13:08:24.686 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 102817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-30T13:08:24.686 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 103481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-30T13:08:24.687 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 112849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-30T13:08:24.687 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 60329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-30T13:08:24.687 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 109593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-30T13:08:24.687 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 113129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-30T13:08:24.687 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 112585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-30T13:08:24.687 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 111585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-30T13:08:24.688 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 114465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-30T13:08:24.688 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-30T13:08:24.688 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ 2023-10-30T13:08:24.688 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 411729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-30T13:08:24.688 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-30T13:08:24.688 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-30T13:08:24.689 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-30T13:08:24.689 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-30T13:08:24.689 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-30T13:08:24.689 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-30T13:08:24.689 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-30T13:08:24.689 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-30T13:08:24.690 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-30T13:08:24.690 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-30T13:08:24.690 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-30T13:08:24.690 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-30T13:08:24.690 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-30T13:08:24.690 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 49257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conncount.ko 2023-10-30T13:08:24.691 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 408257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack.ko 2023-10-30T13:08:24.691 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-30T13:08:24.691 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-30T13:08:24.691 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-30T13:08:24.691 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 118145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-30T13:08:24.691 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-30T13:08:24.692 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-30T13:08:24.692 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 107297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-30T13:08:24.692 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-30T13:08:24.692 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-30T13:08:24.692 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 66641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-30T13:08:24.692 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-30T13:08:24.693 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-30T13:08:24.693 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-30T13:08:24.693 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 89473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_flow_table.ko 2023-10-30T13:08:24.693 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-30T13:08:24.693 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 37129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_log_syslog.ko 2023-10-30T13:08:24.693 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 95417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat.ko 2023-10-30T13:08:24.694 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-30T13:08:24.694 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-30T13:08:24.694 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_irc.ko 2023-10-30T13:08:24.694 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_sip.ko 2023-10-30T13:08:24.694 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-30T13:08:24.694 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 44065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-30T13:08:24.695 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 615313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_tables.ko 2023-10-30T13:08:24.695 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 42561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink.ko 2023-10-30T13:08:24.695 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-30T13:08:24.695 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-10-30T13:08:24.695 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 39785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_log.ko 2023-10-30T13:08:24.695 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-30T13:08:24.696 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 49377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-30T13:08:24.696 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_chain_nat.ko 2023-10-30T13:08:24.696 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_compat.ko 2023-10-30T13:08:24.696 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 40721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_ct.ko 2023-10-30T13:08:24.697 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-30T13:08:24.697 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_fib.ko 2023-10-30T13:08:24.697 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-30T13:08:24.697 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_hash.ko 2023-10-30T13:08:24.697 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_limit.ko 2023-10-30T13:08:24.697 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_log.ko 2023-10-30T13:08:24.698 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_masq.ko 2023-10-30T13:08:24.698 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_nat.ko 2023-10-30T13:08:24.698 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_numgen.ko 2023-10-30T13:08:24.698 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_osf.ko 2023-10-30T13:08:24.698 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_queue.ko 2023-10-30T13:08:24.699 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_redir.ko 2023-10-30T13:08:24.699 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_reject.ko 2023-10-30T13:08:24.699 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_reject_inet.ko 2023-10-30T13:08:24.699 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_socket.ko 2023-10-30T13:08:24.699 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_synproxy.ko 2023-10-30T13:08:24.699 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_tproxy.ko 2023-10-30T13:08:24.700 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 89313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/x_tables.ko 2023-10-30T13:08:24.700 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_AUDIT.ko 2023-10-30T13:08:24.700 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-30T13:08:24.700 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-30T13:08:24.700 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-30T13:08:24.700 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 18169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CT.ko 2023-10-30T13:08:24.701 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_DSCP.ko 2023-10-30T13:08:24.701 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_HL.ko 2023-10-30T13:08:24.701 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_HMARK.ko 2023-10-30T13:08:24.701 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-30T13:08:24.701 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_LED.ko 2023-10-30T13:08:24.701 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_LOG.ko 2023-10-30T13:08:24.702 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-30T13:08:24.702 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NETMAP.ko 2023-10-30T13:08:24.702 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NFLOG.ko 2023-10-30T13:08:24.702 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-30T13:08:24.702 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_RATEEST.ko 2023-10-30T13:08:24.703 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-30T13:08:24.703 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 14337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_SECMARK.ko 2023-10-30T13:08:24.703 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-30T13:08:24.703 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-30T13:08:24.703 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TEE.ko 2023-10-30T13:08:24.704 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TPROXY.ko 2023-10-30T13:08:24.704 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TRACE.ko 2023-10-30T13:08:24.704 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_addrtype.ko 2023-10-30T13:08:24.704 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_bpf.ko 2023-10-30T13:08:24.704 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cgroup.ko 2023-10-30T13:08:24.704 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cluster.ko 2023-10-30T13:08:24.705 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_comment.ko 2023-10-30T13:08:24.705 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connbytes.ko 2023-10-30T13:08:24.705 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connlabel.ko 2023-10-30T13:08:24.705 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connlimit.ko 2023-10-30T13:08:24.705 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connmark.ko 2023-10-30T13:08:24.705 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_conntrack.ko 2023-10-30T13:08:24.706 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cpu.ko 2023-10-30T13:08:24.706 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_dccp.ko 2023-10-30T13:08:24.706 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_devgroup.ko 2023-10-30T13:08:24.706 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_dscp.ko 2023-10-30T13:08:24.707 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ecn.ko 2023-10-30T13:08:24.707 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_esp.ko 2023-10-30T13:08:24.707 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 43289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_hashlimit.ko 2023-10-30T13:08:24.707 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_helper.ko 2023-10-30T13:08:24.707 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_hl.ko 2023-10-30T13:08:24.708 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ipcomp.ko 2023-10-30T13:08:24.708 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_iprange.ko 2023-10-30T13:08:24.708 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ipvs.ko 2023-10-30T13:08:24.708 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_l2tp.ko 2023-10-30T13:08:24.708 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_length.ko 2023-10-30T13:08:24.709 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_limit.ko 2023-10-30T13:08:24.709 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_mac.ko 2023-10-30T13:08:24.709 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_mark.ko 2023-10-30T13:08:24.709 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_multiport.ko 2023-10-30T13:08:24.709 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_nat.ko 2023-10-30T13:08:24.710 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_nfacct.ko 2023-10-30T13:08:24.710 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_osf.ko 2023-10-30T13:08:24.710 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_owner.ko 2023-10-30T13:08:24.710 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_physdev.ko 2023-10-30T13:08:24.710 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_pkttype.ko 2023-10-30T13:08:24.711 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_policy.ko 2023-10-30T13:08:24.711 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_quota.ko 2023-10-30T13:08:24.711 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_rateest.ko 2023-10-30T13:08:24.711 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_realm.ko 2023-10-30T13:08:24.711 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_recent.ko 2023-10-30T13:08:24.712 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_sctp.ko 2023-10-30T13:08:24.712 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_set.ko 2023-10-30T13:08:24.712 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 16089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_socket.ko 2023-10-30T13:08:24.712 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_state.ko 2023-10-30T13:08:24.712 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_statistic.ko 2023-10-30T13:08:24.712 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_string.ko 2023-10-30T13:08:24.713 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 8553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_tcpmss.ko 2023-10-30T13:08:24.713 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_tcpudp.ko 2023-10-30T13:08:24.713 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_time.ko 2023-10-30T13:08:24.713 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_u32.ko 2023-10-30T13:08:24.713 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netlink/ 2023-10-30T13:08:24.714 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netlink/netlink_diag.ko 2023-10-30T13:08:24.714 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/nsh/ 2023-10-30T13:08:24.714 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 12425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/nsh/nsh.ko 2023-10-30T13:08:24.714 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/ 2023-10-30T13:08:24.714 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 380033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/openvswitch.ko 2023-10-30T13:08:24.715 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-geneve.ko 2023-10-30T13:08:24.715 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-gre.ko 2023-10-30T13:08:24.715 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 11793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-vxlan.ko 2023-10-30T13:08:24.715 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/packet/ 2023-10-30T13:08:24.715 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/packet/af_packet_diag.ko 2023-10-30T13:08:24.715 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/ 2023-10-30T13:08:24.716 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 110201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/phonet.ko 2023-10-30T13:08:24.716 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 54697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/pn_pep.ko 2023-10-30T13:08:24.716 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/ 2023-10-30T13:08:24.716 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 251481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds.ko 2023-10-30T13:08:24.716 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 164937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds_rdma.ko 2023-10-30T13:08:24.717 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 67377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds_tcp.ko 2023-10-30T13:08:24.717 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rxrpc/ 2023-10-30T13:08:24.717 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 819657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rxrpc/rxrpc.ko 2023-10-30T13:08:24.717 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/ 2023-10-30T13:08:24.717 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_csum.ko 2023-10-30T13:08:24.718 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_gact.ko 2023-10-30T13:08:24.718 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_ipt.ko 2023-10-30T13:08:24.718 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_mirred.ko 2023-10-30T13:08:24.718 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_nat.ko 2023-10-30T13:08:24.718 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 43449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_pedit.ko 2023-10-30T13:08:24.719 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_police.ko 2023-10-30T13:08:24.719 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_simple.ko 2023-10-30T13:08:24.719 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_skbedit.ko 2023-10-30T13:08:24.719 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_vlan.ko 2023-10-30T13:08:24.719 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_basic.ko 2023-10-30T13:08:24.719 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_bpf.ko 2023-10-30T13:08:24.720 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_cgroup.ko 2023-10-30T13:08:24.720 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_flow.ko 2023-10-30T13:08:24.720 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_fw.ko 2023-10-30T13:08:24.720 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 38489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_route.ko 2023-10-30T13:08:24.720 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 61409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_u32.ko 2023-10-30T13:08:24.720 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_cmp.ko 2023-10-30T13:08:24.721 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 13409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_ipset.ko 2023-10-30T13:08:24.721 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_meta.ko 2023-10-30T13:08:24.721 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_nbyte.ko 2023-10-30T13:08:24.721 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_text.ko 2023-10-30T13:08:24.721 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_u32.ko 2023-10-30T13:08:24.722 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 21649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_choke.ko 2023-10-30T13:08:24.722 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_codel.ko 2023-10-30T13:08:24.722 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 34633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_drr.ko 2023-10-30T13:08:24.722 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 33609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_fq.ko 2023-10-30T13:08:24.722 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_fq_codel.ko 2023-10-30T13:08:24.723 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 35433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_gred.ko 2023-10-30T13:08:24.723 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 49601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_hfsc.ko 2023-10-30T13:08:24.723 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_hhf.ko 2023-10-30T13:08:24.723 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 66393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_htb.ko 2023-10-30T13:08:24.723 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_ingress.ko 2023-10-30T13:08:24.724 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 30273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_mqprio.ko 2023-10-30T13:08:24.724 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_mqprio_lib.ko 2023-10-30T13:08:24.724 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_multiq.ko 2023-10-30T13:08:24.724 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 36137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_netem.ko 2023-10-30T13:08:24.724 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 27497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_pie.ko 2023-10-30T13:08:24.725 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_plug.ko 2023-10-30T13:08:24.725 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_prio.ko 2023-10-30T13:08:24.725 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 48793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_qfq.ko 2023-10-30T13:08:24.725 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 32905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_red.ko 2023-10-30T13:08:24.725 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 26881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_sfb.ko 2023-10-30T13:08:24.726 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 31353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_sfq.ko 2023-10-30T13:08:24.726 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_tbf.ko 2023-10-30T13:08:24.726 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 28105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_teql.ko 2023-10-30T13:08:24.726 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/ 2023-10-30T13:08:24.726 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 929745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/sctp.ko 2023-10-30T13:08:24.727 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 19033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/sctp_diag.ko 2023-10-30T13:08:24.727 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/ 2023-10-30T13:08:24.727 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/ 2023-10-30T13:08:24.727 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 336529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-30T13:08:24.727 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 54737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-30T13:08:24.728 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 1520161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/sunrpc.ko 2023-10-30T13:08:24.728 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/xprtrdma/ 2023-10-30T13:08:24.728 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 707721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-30T13:08:24.728 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/ 2023-10-30T13:08:24.728 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/diag.ko 2023-10-30T13:08:24.729 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 862641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/tipc.ko 2023-10-30T13:08:24.729 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/unix/ 2023-10-30T13:08:24.729 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/unix/unix_diag.ko 2023-10-30T13:08:24.729 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/ 2023-10-30T13:08:24.729 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 85337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-30T13:08:24.730 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 68049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-30T13:08:24.730 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 96193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock.ko 2023-10-30T13:08:24.730 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 10705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-30T13:08:24.730 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 17521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-30T13:08:24.730 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/x25/ 2023-10-30T13:08:24.731 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 125177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/x25/x25.ko 2023-10-30T13:08:24.731 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/ 2023-10-30T13:08:24.731 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 22657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_algo.ko 2023-10-30T13:08:24.731 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 20649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-30T13:08:24.731 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 102257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_user.ko 2023-10-30T13:08:24.732 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/ 2023-10-30T13:08:24.732 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/lib/ 2023-10-30T13:08:24.732 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 15537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/lib/irqbypass.ko 2023-10-30T13:08:24.732 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 639863 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.alias 2023-10-30T13:08:24.732 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 623919 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.alias.bin 2023-10-30T13:08:24.733 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 7663 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin 2023-10-30T13:08:24.733 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 23428 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.alias.bin 2023-10-30T13:08:24.733 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 9821 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.bin 2023-10-30T13:08:24.733 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 70004 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.modinfo 2023-10-30T13:08:24.734 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 146313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.dep 2023-10-30T13:08:24.734 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 232548 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.dep.bin 2023-10-30T13:08:24.734 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 210 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.devname 2023-10-30T13:08:24.734 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 72606 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.order 2023-10-30T13:08:24.734 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 879 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.softdep 2023-10-30T13:08:24.734 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 254438 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.symbols 2023-10-30T13:08:24.735 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 309363 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.symbols.bin 2023-10-30T13:08:24.735 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/ 2023-10-30T13:08:24.735 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/ 2023-10-30T13:08:24.735 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/doc/ 2023-10-30T13:08:24.735 INFO:teuthology.orchestra.run.smithi046.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/ 2023-10-30T13:08:24.735 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 173 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/changelog.Debian.gz 2023-10-30T13:08:24.736 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r-- root/root 691 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/copyright 2023-10-30T13:08:24.737 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-geacf00cc51e2 2023-10-30T13:08:24.737 INFO:teuthology.task.kernel:searching for kernel 6.6.0-rc5-geacf00cc51e2 2023-10-30T13:08:24.737 DEBUG:teuthology.orchestra.run.smithi046:> egrep '(submenu|menuentry.*6.6.0-rc5-geacf00cc51e2).*{' /boot/grub/grub.cfg 2023-10-30T13:08:24.745 INFO:teuthology.orchestra.run.smithi046.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T13:08:24.745 INFO:teuthology.orchestra.run.smithi046.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-geacf00cc51e2-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T13:08:24.745 INFO:teuthology.orchestra.run.smithi046.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-geacf00cc51e2-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T13:08:24.746 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-10-30T13:08:24.746 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2 2023-10-30T13:08:24.747 DEBUG:teuthology.orchestra.run.smithi046:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-10-30T13:08:24.788 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi046.front.sepia.ceph.com to complete... 2023-10-30T13:08:25.225 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub' 2023-10-30T13:08:25.226 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-30T13:08:25.251 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2023-10-30T13:08:25.481 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:08:25.488 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T13:08:25.795 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-30T13:08:25.801 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-30T13:08:25.847 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-30T13:08:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-30T13:08:25.972 INFO:teuthology.orchestra.run.smithi046.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-30T13:08:25.972 INFO:teuthology.orchestra.run.smithi046.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-30T13:08:25.972 INFO:teuthology.orchestra.run.smithi046.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-30T13:08:25.993 INFO:teuthology.orchestra.run.smithi046.stderr:done 2023-10-30T13:08:57.130 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-30T13:08:57.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-10-30T13:08:57.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:09:31.642 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2023-10-30T13:09:40.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-10-30T13:09:40.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:10:40.647 DEBUG:teuthology.orchestra.remote:timed out 2023-10-30T13:10:52.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-10-30T13:10:52.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:10:52.931 DEBUG:teuthology.orchestra.run.smithi046:> true 2023-10-30T13:10:53.637 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-10-30T13:10:53.637 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-geacf00cc51e2"... 2023-10-30T13:10:53.637 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2023-10-30T13:10:53.689 INFO:teuthology.orchestra.run.smithi046.stdout:6.6.0-rc5-geacf00cc51e2 2023-10-30T13:10:53.689 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-geacf00cc51e2 vs 6.6.0-rc5-geacf00cc51e2 2023-10-30T13:10:53.689 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T13:10:53.689 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-30T13:10:54.691 DEBUG:teuthology.parallel:result is None 2023-10-30T13:10:54.691 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-30T13:10:54.699 INFO:teuthology.task.internal:Creating test directory... 2023-10-30T13:10:54.699 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T13:10:54.703 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T13:10:54.710 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-30T13:10:54.717 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-30T13:10:54.728 INFO:teuthology.task.internal:Creating archive directory... 2023-10-30T13:10:54.728 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T13:10:54.756 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T13:10:54.796 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-30T13:10:54.803 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-30T13:10:54.803 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T13:10:54.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:10:54.815 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T13:10:54.836 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T13:10:54.877 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T13:10:54.888 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T13:10:54.935 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T13:10:54.977 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T13:10:54.979 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-30T13:10:54.986 INFO:teuthology.task.internal:Configuring sudo... 2023-10-30T13:10:54.987 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T13:10:54.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T13:10:55.010 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-30T13:10:55.019 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-30T13:10:55.020 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T13:10:55.056 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T13:10:55.062 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T13:10:55.114 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T13:10:55.170 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:10:55.171 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T13:10:55.248 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T13:10:55.255 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T13:10:55.304 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:10:55.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T13:10:55.375 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2023-10-30T13:10:55.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2023-10-30T13:10:55.483 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-30T13:10:55.491 INFO:teuthology.task.internal:Starting timer... 2023-10-30T13:10:55.491 INFO:teuthology.run_tasks:Running task pcp... 2023-10-30T13:10:55.500 INFO:teuthology.run_tasks:Running task selinux... 2023-10-30T13:10:55.509 DEBUG:teuthology.task.selinux:Excluding smithi046: OS 'ubuntu' does not support SELinux 2023-10-30T13:10:55.509 DEBUG:teuthology.task.selinux:Excluding smithi154: OS 'ubuntu' does not support SELinux 2023-10-30T13:10:55.510 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-30T13:10:55.510 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-30T13:10:55.510 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-30T13:10:55.510 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-30T13:10:55.519 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-30T13:10:55.519 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-30T13:10:55.534 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-30T13:10:55.536 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-30T13:19:24.667 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2023-10-30T13:19:24.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-10-30T13:19:24.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:19:24.749 DEBUG:teuthology.orchestra.run.smithi046:> true 2023-10-30T13:19:24.885 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-10-30T13:19:24.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-10-30T13:19:24.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:19:24.962 DEBUG:teuthology.orchestra.run.smithi154:> true 2023-10-30T13:19:25.070 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-10-30T13:19:25.070 INFO:teuthology.run_tasks:Running task clock... 2023-10-30T13:19:25.082 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-30T13:19:25.083 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T13:19:25.083 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T13:19:25.087 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T13:19:25.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T13:19:25.124 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-30T13:19:25.124 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: Command line: ntpd -gq 2023-10-30T13:19:25.124 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: ---------------------------------------------------- 2023-10-30T13:19:25.124 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: ntp-4 is maintained by Network Time Foundation, 2023-10-30T13:19:25.124 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-30T13:19:25.125 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: corporation. Support and training for ntp-4 are 2023-10-30T13:19:25.125 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: available at https://www.nwtime.org/support 2023-10-30T13:19:25.125 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: ---------------------------------------------------- 2023-10-30T13:19:25.126 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: proto: precision = 0.043 usec (-24) 2023-10-30T13:19:25.126 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: basedate set to 2022-02-04 2023-10-30T13:19:25.126 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: gps base set to 2022-02-06 (week 2196) 2023-10-30T13:19:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T13:19:25.127 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T13:19:25.127 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: restrict ::: KOD does nothing without LIMITED. 2023-10-30T13:19:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-30T13:19:25.128 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: Listen and drop on 0 v6wildcard [::]:123 2023-10-30T13:19:25.129 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-30T13:19:25.129 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: Listen normally on 2 lo 127.0.0.1:123 2023-10-30T13:19:25.129 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: Listen normally on 3 ens1f0 172.21.15.46:123 2023-10-30T13:19:25.129 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: Listen normally on 4 lo [::1]:123 2023-10-30T13:19:25.129 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15f8%4]:123 2023-10-30T13:19:25.129 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:25 ntpd[27910]: Listening on routing socket on fd #22 for interface updates 2023-10-30T13:19:25.136 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-30T13:19:25.137 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: Command line: ntpd -gq 2023-10-30T13:19:25.137 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: ---------------------------------------------------- 2023-10-30T13:19:25.137 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: ntp-4 is maintained by Network Time Foundation, 2023-10-30T13:19:25.137 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-30T13:19:25.137 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: corporation. Support and training for ntp-4 are 2023-10-30T13:19:25.138 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: available at https://www.nwtime.org/support 2023-10-30T13:19:25.138 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: ---------------------------------------------------- 2023-10-30T13:19:25.138 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: proto: precision = 0.043 usec (-24) 2023-10-30T13:19:25.138 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: basedate set to 2022-02-04 2023-10-30T13:19:25.138 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: gps base set to 2022-02-06 (week 2196) 2023-10-30T13:19:25.139 INFO:teuthology.orchestra.run.smithi154.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T13:19:25.139 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T13:19:25.139 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: restrict ::: KOD does nothing without LIMITED. 2023-10-30T13:19:25.140 INFO:teuthology.orchestra.run.smithi154.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-30T13:19:25.141 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: Listen and drop on 0 v6wildcard [::]:123 2023-10-30T13:19:25.141 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-30T13:19:25.141 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: Listen normally on 2 lo 127.0.0.1:123 2023-10-30T13:19:25.141 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: Listen normally on 3 enp3s0f1 172.21.15.154:123 2023-10-30T13:19:25.141 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: Listen normally on 4 lo [::1]:123 2023-10-30T13:19:25.141 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7761%5]:123 2023-10-30T13:19:25.142 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:25 ntpd[28447]: Listening on routing socket on fd #22 for interface updates 2023-10-30T13:19:26.124 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:26 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:26.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:26 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:26.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:26 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:26.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:26 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:26 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:26 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:26 ntpd[27910]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:26 ntpd[27910]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:26 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:26 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:26.128 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:26 ntpd[27910]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:26.128 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:26 ntpd[27910]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:26.137 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:26.137 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:26.137 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:26.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:26.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:26.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:26.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:26.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:26.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:26.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:26.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:26.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:26.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:26.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:26 ntpd[28447]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:28.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:28 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:28 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:28 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:28 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:28 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:28 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:28.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:28 ntpd[27910]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:28.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:28 ntpd[27910]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:28.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:28 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:28.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:28 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:28.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:28 ntpd[27910]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:28.128 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:28 ntpd[27910]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:28.137 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:28.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:28.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:28.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:28.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:28.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:28.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:28.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:28.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:28.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:28.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:28.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:28.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:28.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:28.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:28.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:28.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:28.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:28 ntpd[28447]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:29.124 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:29 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:29.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:29 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:29.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:29 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:29.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:29 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:29.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:29 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:29.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:29 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:29.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:29 ntpd[27910]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:29.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:29 ntpd[27910]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:30.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:30 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:30.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:30 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:30.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:30 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:30.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:30 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:30.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:30 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:30.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:30 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:30.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:30 ntpd[27910]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:30.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:30 ntpd[27910]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:30.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:30 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:30.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:30 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:30.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:30 ntpd[27910]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:30.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:30 ntpd[27910]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:30.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:30.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:30.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:30.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:30.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:30.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:30.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:30.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:30.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:30.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:30.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:30.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:30.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:30.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:30.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:30.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:30.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:30.141 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:30 ntpd[28447]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:31.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:31 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:31.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:31 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:31.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:31 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:31.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:31 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:31.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:31 ntpd[27910]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:31.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:31 ntpd[27910]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:32.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:32 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:32.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:32 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:32.125 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:32 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:32.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:32 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:32.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:32 ntpd[27910]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:32.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:32 ntpd[27910]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:32.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:32 ntpd[27910]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:32.126 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:32 ntpd[27910]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:32.127 INFO:teuthology.orchestra.run.smithi046.stdout:30 Oct 13:19:32 ntpd[27910]: ntpd: time slew -0.000525 s 2023-10-30T13:19:32.127 INFO:teuthology.orchestra.run.smithi046.stdout:ntpd: time slew -0.000525s 2023-10-30T13:19:32.127 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:32 ntpd[27910]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-30T13:19:32.128 INFO:teuthology.orchestra.run.smithi046.stderr:30 Oct 13:19:32 ntpd[27910]: can't open /var/log/ntpstats/loopstats.20231030: Permission denied 2023-10-30T13:19:32.137 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:32.137 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:32.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:32.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:32.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:32.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:32.138 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:32.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:32.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:32.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:32.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:19:32.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:19:32.139 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:19:32.140 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:19:32.140 INFO:teuthology.orchestra.run.smithi154.stdout:30 Oct 13:19:32 ntpd[28447]: ntpd: time slew -0.005669 s 2023-10-30T13:19:32.140 INFO:teuthology.orchestra.run.smithi154.stdout:ntpd: time slew -0.005669s 2023-10-30T13:19:32.141 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-30T13:19:32.141 INFO:teuthology.orchestra.run.smithi154.stderr:30 Oct 13:19:32 ntpd[28447]: can't open /var/log/ntpstats/loopstats.20231030: Permission denied 2023-10-30T13:19:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T13:19:32.212 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================== 2023-10-30T13:19:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:19:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:19:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:19:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:19:32.216 INFO:teuthology.orchestra.run.smithi046.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T13:19:32.216 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================== 2023-10-30T13:19:32.216 INFO:teuthology.orchestra.run.smithi046.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:19:32.217 INFO:teuthology.orchestra.run.smithi046.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:19:32.217 INFO:teuthology.orchestra.run.smithi046.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:19:32.217 INFO:teuthology.orchestra.run.smithi046.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:19:32.218 INFO:teuthology.run_tasks:Running task install... 2023-10-30T13:19:32.227 DEBUG:teuthology.task.install:project ceph 2023-10-30T13:19:32.227 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}} 2023-10-30T13:19:32.227 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'} 2023-10-30T13:19:32.228 INFO:teuthology.task.install:Using flavor: default 2023-10-30T13:19:32.233 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-30T13:19:32.233 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-30T13:19:32.234 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-key list | grep Ceph 2023-10-30T13:19:32.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-key list | grep Ceph 2023-10-30T13:19:32.316 INFO:teuthology.orchestra.run.smithi154.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-30T13:19:32.344 INFO:teuthology.orchestra.run.smithi154.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-30T13:19:32.344 INFO:teuthology.orchestra.run.smithi154.stdout:uid [ unknown] Ceph.com (release key) 2023-10-30T13:19:32.345 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-30T13:19:32.345 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-30T13:19:32.346 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:19:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-30T13:19:32.428 INFO:teuthology.orchestra.run.smithi046.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-30T13:19:32.428 INFO:teuthology.orchestra.run.smithi046.stdout:uid [ unknown] Ceph.com (release key) 2023-10-30T13:19:32.428 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-30T13:19:32.428 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-30T13:19:32.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:19:32.578 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default/ 2023-10-30T13:19:32.578 INFO:teuthology.task.install.deb:Package version is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:19:32.646 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default/ 2023-10-30T13:19:32.646 INFO:teuthology.task.install.deb:Package version is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:19:32.692 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:19:32.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-30T13:19:32.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get update 2023-10-30T13:19:32.811 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:19:32.812 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-30T13:19:32.833 DEBUG:teuthology.orchestra.run.smithi046:> sudo apt-get update 2023-10-30T13:19:32.898 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-30T13:19:32.907 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-30T13:19:32.924 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-30T13:19:32.932 INFO:teuthology.orchestra.run.smithi154.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy InRelease 2023-10-30T13:19:32.941 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-30T13:19:32.954 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-30T13:19:32.976 INFO:teuthology.orchestra.run.smithi154.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-30T13:19:32.997 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-30T13:19:33.025 INFO:teuthology.orchestra.run.smithi154.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-30T13:19:33.138 INFO:teuthology.orchestra.run.smithi046.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy InRelease 2023-10-30T13:19:33.156 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-30T13:19:33.156 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-30T13:19:33.175 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-30T13:19:33.212 INFO:teuthology.orchestra.run.smithi046.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-30T13:19:33.227 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-30T13:19:33.250 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-30T13:19:33.260 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 28.5 kB in 0s (76.1 kB/s) 2023-10-30T13:19:33.298 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-30T13:19:33.309 INFO:teuthology.orchestra.run.smithi046.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-30T13:19:33.784 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 139 kB in 1s (193 kB/s) 2023-10-30T13:19:34.101 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2023-10-30T13:19:34.116 DEBUG:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6835-g00c1c13c-1jammy cephadm=18.0.0-6835-g00c1c13c-1jammy ceph-mds=18.0.0-6835-g00c1c13c-1jammy ceph-mgr=18.0.0-6835-g00c1c13c-1jammy ceph-common=18.0.0-6835-g00c1c13c-1jammy ceph-fuse=18.0.0-6835-g00c1c13c-1jammy ceph-test=18.0.0-6835-g00c1c13c-1jammy ceph-volume=18.0.0-6835-g00c1c13c-1jammy radosgw=18.0.0-6835-g00c1c13c-1jammy python3-rados=18.0.0-6835-g00c1c13c-1jammy python3-rgw=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy python3-rbd=18.0.0-6835-g00c1c13c-1jammy libcephfs2=18.0.0-6835-g00c1c13c-1jammy libcephfs-dev=18.0.0-6835-g00c1c13c-1jammy librados2=18.0.0-6835-g00c1c13c-1jammy librbd1=18.0.0-6835-g00c1c13c-1jammy rbd-fuse=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy cephfs-shell=18.0.0-6835-g00c1c13c-1jammy cephfs-top=18.0.0-6835-g00c1c13c-1jammy cephfs-mirror=18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:19:34.175 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2023-10-30T13:19:34.364 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2023-10-30T13:19:34.365 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2023-10-30T13:19:34.531 INFO:teuthology.orchestra.run.smithi154.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T13:19:34.532 INFO:teuthology.orchestra.run.smithi154.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T13:19:34.532 INFO:teuthology.orchestra.run.smithi154.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T13:19:34.533 INFO:teuthology.orchestra.run.smithi154.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T13:19:34.534 INFO:teuthology.orchestra.run.smithi154.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-30T13:19:34.534 INFO:teuthology.orchestra.run.smithi154.stdout: python2.7-minimal 2023-10-30T13:19:34.534 INFO:teuthology.orchestra.run.smithi154.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T13:19:34.534 INFO:teuthology.orchestra.run.smithi154.stdout:The following additional packages will be installed: 2023-10-30T13:19:34.535 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-30T13:19:34.535 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-30T13:19:34.535 INFO:teuthology.orchestra.run.smithi154.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T13:19:34.535 INFO:teuthology.orchestra.run.smithi154.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-30T13:19:34.536 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-30T13:19:34.536 INFO:teuthology.orchestra.run.smithi154.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-30T13:19:34.536 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-30T13:19:34.536 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-30T13:19:34.536 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-30T13:19:34.537 INFO:teuthology.orchestra.run.smithi154.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-30T13:19:34.537 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-30T13:19:34.537 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-30T13:19:34.537 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-30T13:19:34.537 INFO:teuthology.orchestra.run.smithi154.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-30T13:19:34.538 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-30T13:19:34.538 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-30T13:19:34.538 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-30T13:19:34.538 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-30T13:19:34.539 INFO:teuthology.orchestra.run.smithi154.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-30T13:19:34.539 INFO:teuthology.orchestra.run.smithi154.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-30T13:19:34.539 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-30T13:19:34.539 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-30T13:19:34.539 INFO:teuthology.orchestra.run.smithi154.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-30T13:19:34.540 INFO:teuthology.orchestra.run.smithi154.stdout:Suggested packages: 2023-10-30T13:19:34.540 INFO:teuthology.orchestra.run.smithi154.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-30T13:19:34.540 INFO:teuthology.orchestra.run.smithi154.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-30T13:19:34.540 INFO:teuthology.orchestra.run.smithi154.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-30T13:19:34.541 INFO:teuthology.orchestra.run.smithi154.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-30T13:19:34.541 INFO:teuthology.orchestra.run.smithi154.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-30T13:19:34.541 INFO:teuthology.orchestra.run.smithi154.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-30T13:19:34.541 INFO:teuthology.orchestra.run.smithi154.stdout: python3-watchdog 2023-10-30T13:19:34.541 INFO:teuthology.orchestra.run.smithi154.stdout:Recommended packages: 2023-10-30T13:19:34.542 INFO:teuthology.orchestra.run.smithi154.stdout: btrfs-tools 2023-10-30T13:19:34.636 INFO:teuthology.orchestra.run.smithi154.stdout:The following NEW packages will be installed: 2023-10-30T13:19:34.636 INFO:teuthology.orchestra.run.smithi154.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-30T13:19:34.637 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-30T13:19:34.637 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-30T13:19:34.637 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-30T13:19:34.637 INFO:teuthology.orchestra.run.smithi154.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T13:19:34.637 INFO:teuthology.orchestra.run.smithi154.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-30T13:19:34.638 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-30T13:19:34.638 INFO:teuthology.orchestra.run.smithi154.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-30T13:19:34.638 INFO:teuthology.orchestra.run.smithi154.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-30T13:19:34.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-30T13:19:34.638 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-30T13:19:34.639 INFO:teuthology.orchestra.run.smithi154.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-30T13:19:34.639 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-30T13:19:34.639 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-30T13:19:34.639 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-30T13:19:34.639 INFO:teuthology.orchestra.run.smithi154.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-30T13:19:34.640 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-30T13:19:34.640 INFO:teuthology.orchestra.run.smithi154.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-30T13:19:34.640 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-30T13:19:34.640 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-30T13:19:34.640 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-30T13:19:34.641 INFO:teuthology.orchestra.run.smithi154.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-30T13:19:34.641 INFO:teuthology.orchestra.run.smithi154.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-30T13:19:34.641 INFO:teuthology.orchestra.run.smithi154.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-30T13:19:34.641 INFO:teuthology.orchestra.run.smithi154.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-30T13:19:34.641 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-30T13:19:34.642 INFO:teuthology.orchestra.run.smithi154.stdout: socat xmlstarlet 2023-10-30T13:19:34.692 INFO:teuthology.orchestra.run.smithi154.stdout:0 upgraded, 115 newly installed, 0 to remove and 260 not upgraded. 2023-10-30T13:19:34.692 INFO:teuthology.orchestra.run.smithi154.stdout:Need to get 184 MB of archives. 2023-10-30T13:19:34.693 INFO:teuthology.orchestra.run.smithi154.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-30T13:19:34.693 INFO:teuthology.orchestra.run.smithi154.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-30T13:19:34.710 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-30T13:19:34.720 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-30T13:19:34.722 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2023-10-30T13:19:34.737 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6835-g00c1c13c-1jammy cephadm=18.0.0-6835-g00c1c13c-1jammy ceph-mds=18.0.0-6835-g00c1c13c-1jammy ceph-mgr=18.0.0-6835-g00c1c13c-1jammy ceph-common=18.0.0-6835-g00c1c13c-1jammy ceph-fuse=18.0.0-6835-g00c1c13c-1jammy ceph-test=18.0.0-6835-g00c1c13c-1jammy ceph-volume=18.0.0-6835-g00c1c13c-1jammy radosgw=18.0.0-6835-g00c1c13c-1jammy python3-rados=18.0.0-6835-g00c1c13c-1jammy python3-rgw=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy python3-rbd=18.0.0-6835-g00c1c13c-1jammy libcephfs2=18.0.0-6835-g00c1c13c-1jammy libcephfs-dev=18.0.0-6835-g00c1c13c-1jammy librados2=18.0.0-6835-g00c1c13c-1jammy librbd1=18.0.0-6835-g00c1c13c-1jammy rbd-fuse=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy cephfs-shell=18.0.0-6835-g00c1c13c-1jammy cephfs-top=18.0.0-6835-g00c1c13c-1jammy cephfs-mirror=18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:19:34.759 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-30T13:19:34.768 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-30T13:19:34.778 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6835-g00c1c13c-1jammy [3,450 kB] 2023-10-30T13:19:34.784 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-30T13:19:34.786 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-30T13:19:34.795 INFO:teuthology.orchestra.run.smithi154.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-30T13:19:34.815 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2023-10-30T13:19:34.841 INFO:teuthology.orchestra.run.smithi154.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-30T13:19:34.845 INFO:teuthology.orchestra.run.smithi154.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-30T13:19:34.856 INFO:teuthology.orchestra.run.smithi154.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-30T13:19:34.860 INFO:teuthology.orchestra.run.smithi154.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-30T13:19:34.860 INFO:teuthology.orchestra.run.smithi154.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-30T13:19:34.869 INFO:teuthology.orchestra.run.smithi154.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-30T13:19:34.869 INFO:teuthology.orchestra.run.smithi154.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-30T13:19:34.870 INFO:teuthology.orchestra.run.smithi154.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-30T13:19:34.870 INFO:teuthology.orchestra.run.smithi154.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-30T13:19:34.871 INFO:teuthology.orchestra.run.smithi154.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-30T13:19:34.878 INFO:teuthology.orchestra.run.smithi154.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-30T13:19:34.879 INFO:teuthology.orchestra.run.smithi154.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-30T13:19:34.895 INFO:teuthology.orchestra.run.smithi154.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-30T13:19:34.896 INFO:teuthology.orchestra.run.smithi154.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-30T13:19:34.896 INFO:teuthology.orchestra.run.smithi154.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-30T13:19:34.898 INFO:teuthology.orchestra.run.smithi154.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-30T13:19:34.899 INFO:teuthology.orchestra.run.smithi154.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-30T13:19:34.900 INFO:teuthology.orchestra.run.smithi154.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-30T13:19:34.900 INFO:teuthology.orchestra.run.smithi154.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-30T13:19:34.901 INFO:teuthology.orchestra.run.smithi154.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-30T13:19:34.912 INFO:teuthology.orchestra.run.smithi154.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-30T13:19:34.912 INFO:teuthology.orchestra.run.smithi154.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-30T13:19:34.929 INFO:teuthology.orchestra.run.smithi154.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-30T13:19:34.930 INFO:teuthology.orchestra.run.smithi154.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-30T13:19:34.930 INFO:teuthology.orchestra.run.smithi154.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-30T13:19:34.935 INFO:teuthology.orchestra.run.smithi154.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-30T13:19:34.935 INFO:teuthology.orchestra.run.smithi154.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-30T13:19:34.936 INFO:teuthology.orchestra.run.smithi154.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-30T13:19:34.936 INFO:teuthology.orchestra.run.smithi154.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-30T13:19:34.938 INFO:teuthology.orchestra.run.smithi154.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-30T13:19:34.946 INFO:teuthology.orchestra.run.smithi154.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-30T13:19:34.946 INFO:teuthology.orchestra.run.smithi154.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-30T13:19:34.957 INFO:teuthology.orchestra.run.smithi154.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-30T13:19:34.958 INFO:teuthology.orchestra.run.smithi154.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-30T13:19:34.960 INFO:teuthology.orchestra.run.smithi154.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-30T13:19:34.960 INFO:teuthology.orchestra.run.smithi154.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-30T13:19:34.963 INFO:teuthology.orchestra.run.smithi154.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-30T13:19:34.989 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2023-10-30T13:19:34.990 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2023-10-30T13:19:35.014 INFO:teuthology.orchestra.run.smithi154.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-30T13:19:35.015 INFO:teuthology.orchestra.run.smithi154.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-30T13:19:35.015 INFO:teuthology.orchestra.run.smithi154.stdout:Get:49 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6835-g00c1c13c-1jammy [3,131 kB] 2023-10-30T13:19:35.055 INFO:teuthology.orchestra.run.smithi154.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-30T13:19:35.056 INFO:teuthology.orchestra.run.smithi154.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-30T13:19:35.056 INFO:teuthology.orchestra.run.smithi154.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-30T13:19:35.057 INFO:teuthology.orchestra.run.smithi154.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-30T13:19:35.077 INFO:teuthology.orchestra.run.smithi154.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-30T13:19:35.079 INFO:teuthology.orchestra.run.smithi154.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-30T13:19:35.080 INFO:teuthology.orchestra.run.smithi154.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-30T13:19:35.097 INFO:teuthology.orchestra.run.smithi154.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-30T13:19:35.097 INFO:teuthology.orchestra.run.smithi154.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-30T13:19:35.098 INFO:teuthology.orchestra.run.smithi154.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-30T13:19:35.098 INFO:teuthology.orchestra.run.smithi154.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-30T13:19:35.099 INFO:teuthology.orchestra.run.smithi154.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-30T13:19:35.099 INFO:teuthology.orchestra.run.smithi154.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-30T13:19:35.106 INFO:teuthology.orchestra.run.smithi154.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-30T13:19:35.107 INFO:teuthology.orchestra.run.smithi154.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-30T13:19:35.110 INFO:teuthology.orchestra.run.smithi154.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-30T13:19:35.113 INFO:teuthology.orchestra.run.smithi154.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6835-g00c1c13c-1jammy [689 kB] 2023-10-30T13:19:35.123 INFO:teuthology.orchestra.run.smithi154.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-30T13:19:35.125 INFO:teuthology.orchestra.run.smithi154.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-30T13:19:35.125 INFO:teuthology.orchestra.run.smithi154.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-30T13:19:35.129 INFO:teuthology.orchestra.run.smithi154.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-30T13:19:35.131 INFO:teuthology.orchestra.run.smithi154.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-30T13:19:35.132 INFO:teuthology.orchestra.run.smithi154.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-30T13:19:35.135 INFO:teuthology.orchestra.run.smithi154.stdout:Get:73 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6835-g00c1c13c-1jammy [356 kB] 2023-10-30T13:19:35.136 INFO:teuthology.orchestra.run.smithi154.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-30T13:19:35.140 INFO:teuthology.orchestra.run.smithi154.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-30T13:19:35.140 INFO:teuthology.orchestra.run.smithi154.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-30T13:19:35.141 INFO:teuthology.orchestra.run.smithi154.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-30T13:19:35.147 INFO:teuthology.orchestra.run.smithi154.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6835-g00c1c13c-1jammy [32.8 kB] 2023-10-30T13:19:35.148 INFO:teuthology.orchestra.run.smithi154.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6835-g00c1c13c-1jammy [180 kB] 2023-10-30T13:19:35.152 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T13:19:35.152 INFO:teuthology.orchestra.run.smithi046.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T13:19:35.152 INFO:teuthology.orchestra.run.smithi046.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T13:19:35.152 INFO:teuthology.orchestra.run.smithi046.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T13:19:35.153 INFO:teuthology.orchestra.run.smithi046.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-30T13:19:35.153 INFO:teuthology.orchestra.run.smithi046.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-30T13:19:35.153 INFO:teuthology.orchestra.run.smithi046.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T13:19:35.154 INFO:teuthology.orchestra.run.smithi046.stdout:The following additional packages will be installed: 2023-10-30T13:19:35.154 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-30T13:19:35.155 INFO:teuthology.orchestra.run.smithi154.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6835-g00c1c13c-1jammy [62.4 kB] 2023-10-30T13:19:35.156 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-30T13:19:35.156 INFO:teuthology.orchestra.run.smithi046.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T13:19:35.156 INFO:teuthology.orchestra.run.smithi046.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-30T13:19:35.156 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-30T13:19:35.156 INFO:teuthology.orchestra.run.smithi046.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-30T13:19:35.157 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-30T13:19:35.157 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-30T13:19:35.157 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-30T13:19:35.157 INFO:teuthology.orchestra.run.smithi046.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-30T13:19:35.157 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-30T13:19:35.158 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-30T13:19:35.158 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-30T13:19:35.158 INFO:teuthology.orchestra.run.smithi046.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-30T13:19:35.158 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-30T13:19:35.158 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-30T13:19:35.159 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-30T13:19:35.159 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-30T13:19:35.159 INFO:teuthology.orchestra.run.smithi046.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-30T13:19:35.159 INFO:teuthology.orchestra.run.smithi046.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-30T13:19:35.159 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-30T13:19:35.159 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-30T13:19:35.160 INFO:teuthology.orchestra.run.smithi046.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-30T13:19:35.160 INFO:teuthology.orchestra.run.smithi046.stdout:Suggested packages: 2023-10-30T13:19:35.160 INFO:teuthology.orchestra.run.smithi046.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-30T13:19:35.160 INFO:teuthology.orchestra.run.smithi046.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-30T13:19:35.160 INFO:teuthology.orchestra.run.smithi046.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-30T13:19:35.160 INFO:teuthology.orchestra.run.smithi046.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-30T13:19:35.161 INFO:teuthology.orchestra.run.smithi046.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-30T13:19:35.161 INFO:teuthology.orchestra.run.smithi046.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-30T13:19:35.161 INFO:teuthology.orchestra.run.smithi046.stdout: python3-watchdog 2023-10-30T13:19:35.161 INFO:teuthology.orchestra.run.smithi046.stdout:Recommended packages: 2023-10-30T13:19:35.161 INFO:teuthology.orchestra.run.smithi046.stdout: btrfs-tools 2023-10-30T13:19:35.162 INFO:teuthology.orchestra.run.smithi154.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6835-g00c1c13c-1jammy [330 kB] 2023-10-30T13:19:35.162 INFO:teuthology.orchestra.run.smithi154.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-30T13:19:35.162 INFO:teuthology.orchestra.run.smithi154.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-30T13:19:35.163 INFO:teuthology.orchestra.run.smithi154.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-30T13:19:35.165 INFO:teuthology.orchestra.run.smithi154.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-30T13:19:35.166 INFO:teuthology.orchestra.run.smithi154.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-30T13:19:35.167 INFO:teuthology.orchestra.run.smithi154.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-30T13:19:35.167 INFO:teuthology.orchestra.run.smithi154.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6835-g00c1c13c-1jammy [6,226 kB] 2023-10-30T13:19:35.168 INFO:teuthology.orchestra.run.smithi154.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-30T13:19:35.173 INFO:teuthology.orchestra.run.smithi154.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-30T13:19:35.304 INFO:teuthology.orchestra.run.smithi046.stdout:The following NEW packages will be installed: 2023-10-30T13:19:35.305 INFO:teuthology.orchestra.run.smithi046.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-30T13:19:35.305 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-30T13:19:35.305 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-30T13:19:35.305 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-30T13:19:35.307 INFO:teuthology.orchestra.run.smithi046.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T13:19:35.307 INFO:teuthology.orchestra.run.smithi046.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-30T13:19:35.307 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-30T13:19:35.307 INFO:teuthology.orchestra.run.smithi046.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-30T13:19:35.307 INFO:teuthology.orchestra.run.smithi046.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-30T13:19:35.308 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-30T13:19:35.308 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-30T13:19:35.308 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-30T13:19:35.308 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-30T13:19:35.308 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-30T13:19:35.308 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-30T13:19:35.309 INFO:teuthology.orchestra.run.smithi046.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-30T13:19:35.309 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-30T13:19:35.309 INFO:teuthology.orchestra.run.smithi046.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-30T13:19:35.309 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-30T13:19:35.309 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-30T13:19:35.310 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-30T13:19:35.310 INFO:teuthology.orchestra.run.smithi046.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-30T13:19:35.310 INFO:teuthology.orchestra.run.smithi046.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-30T13:19:35.310 INFO:teuthology.orchestra.run.smithi046.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-30T13:19:35.310 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-30T13:19:35.310 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-30T13:19:35.311 INFO:teuthology.orchestra.run.smithi046.stdout: socat xmlstarlet 2023-10-30T13:19:35.372 INFO:teuthology.orchestra.run.smithi154.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6835-g00c1c13c-1jammy [112 kB] 2023-10-30T13:19:35.375 INFO:teuthology.orchestra.run.smithi154.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6835-g00c1c13c-1jammy [439 kB] 2023-10-30T13:19:35.379 INFO:teuthology.orchestra.run.smithi046.stdout:0 upgraded, 115 newly installed, 0 to remove and 260 not upgraded. 2023-10-30T13:19:35.380 INFO:teuthology.orchestra.run.smithi046.stdout:Need to get 184 MB of archives. 2023-10-30T13:19:35.380 INFO:teuthology.orchestra.run.smithi046.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-30T13:19:35.380 INFO:teuthology.orchestra.run.smithi046.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-30T13:19:35.389 INFO:teuthology.orchestra.run.smithi154.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6835-g00c1c13c-1jammy [22.6 MB] 2023-10-30T13:19:35.401 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-30T13:19:35.417 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-30T13:19:35.463 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-30T13:19:35.473 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-30T13:19:35.492 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-30T13:19:35.496 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-30T13:19:35.507 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-30T13:19:35.548 INFO:teuthology.orchestra.run.smithi046.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6835-g00c1c13c-1jammy [3,450 kB] 2023-10-30T13:19:35.553 INFO:teuthology.orchestra.run.smithi046.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-30T13:19:35.555 INFO:teuthology.orchestra.run.smithi046.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-30T13:19:35.566 INFO:teuthology.orchestra.run.smithi046.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-30T13:19:35.569 INFO:teuthology.orchestra.run.smithi046.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-30T13:19:35.570 INFO:teuthology.orchestra.run.smithi046.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-30T13:19:35.578 INFO:teuthology.orchestra.run.smithi046.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-30T13:19:35.578 INFO:teuthology.orchestra.run.smithi046.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-30T13:19:35.579 INFO:teuthology.orchestra.run.smithi046.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-30T13:19:35.580 INFO:teuthology.orchestra.run.smithi046.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-30T13:19:35.583 INFO:teuthology.orchestra.run.smithi046.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-30T13:19:35.600 INFO:teuthology.orchestra.run.smithi046.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-30T13:19:35.600 INFO:teuthology.orchestra.run.smithi046.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-30T13:19:35.600 INFO:teuthology.orchestra.run.smithi046.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-30T13:19:35.601 INFO:teuthology.orchestra.run.smithi046.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-30T13:19:35.601 INFO:teuthology.orchestra.run.smithi046.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-30T13:19:35.603 INFO:teuthology.orchestra.run.smithi046.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-30T13:19:35.604 INFO:teuthology.orchestra.run.smithi046.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-30T13:19:35.607 INFO:teuthology.orchestra.run.smithi046.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-30T13:19:35.607 INFO:teuthology.orchestra.run.smithi046.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-30T13:19:35.628 INFO:teuthology.orchestra.run.smithi046.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-30T13:19:35.629 INFO:teuthology.orchestra.run.smithi046.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-30T13:19:35.629 INFO:teuthology.orchestra.run.smithi046.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-30T13:19:35.630 INFO:teuthology.orchestra.run.smithi046.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-30T13:19:35.631 INFO:teuthology.orchestra.run.smithi046.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-30T13:19:35.631 INFO:teuthology.orchestra.run.smithi046.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-30T13:19:35.637 INFO:teuthology.orchestra.run.smithi046.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-30T13:19:35.638 INFO:teuthology.orchestra.run.smithi046.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-30T13:19:35.638 INFO:teuthology.orchestra.run.smithi046.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-30T13:19:35.648 INFO:teuthology.orchestra.run.smithi046.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-30T13:19:35.659 INFO:teuthology.orchestra.run.smithi046.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-30T13:19:35.661 INFO:teuthology.orchestra.run.smithi046.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-30T13:19:35.661 INFO:teuthology.orchestra.run.smithi046.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-30T13:19:35.663 INFO:teuthology.orchestra.run.smithi046.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-30T13:19:35.663 INFO:teuthology.orchestra.run.smithi046.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-30T13:19:35.666 INFO:teuthology.orchestra.run.smithi046.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-30T13:19:35.667 INFO:teuthology.orchestra.run.smithi046.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-30T13:19:35.671 INFO:teuthology.orchestra.run.smithi046.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-30T13:19:35.740 INFO:teuthology.orchestra.run.smithi046.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-30T13:19:35.741 INFO:teuthology.orchestra.run.smithi046.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-30T13:19:35.742 INFO:teuthology.orchestra.run.smithi046.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-30T13:19:35.743 INFO:teuthology.orchestra.run.smithi046.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-30T13:19:35.743 INFO:teuthology.orchestra.run.smithi046.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-30T13:19:35.745 INFO:teuthology.orchestra.run.smithi046.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-30T13:19:35.772 INFO:teuthology.orchestra.run.smithi046.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-30T13:19:35.774 INFO:teuthology.orchestra.run.smithi046.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-30T13:19:35.775 INFO:teuthology.orchestra.run.smithi046.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-30T13:19:35.807 INFO:teuthology.orchestra.run.smithi046.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-30T13:19:35.813 INFO:teuthology.orchestra.run.smithi046.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-30T13:19:35.814 INFO:teuthology.orchestra.run.smithi046.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-30T13:19:35.814 INFO:teuthology.orchestra.run.smithi046.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-30T13:19:35.815 INFO:teuthology.orchestra.run.smithi046.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-30T13:19:35.815 INFO:teuthology.orchestra.run.smithi046.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-30T13:19:35.816 INFO:teuthology.orchestra.run.smithi046.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-30T13:19:35.816 INFO:teuthology.orchestra.run.smithi046.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-30T13:19:35.828 INFO:teuthology.orchestra.run.smithi046.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-30T13:19:35.842 INFO:teuthology.orchestra.run.smithi046.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-30T13:19:35.844 INFO:teuthology.orchestra.run.smithi046.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-30T13:19:35.845 INFO:teuthology.orchestra.run.smithi046.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-30T13:19:35.849 INFO:teuthology.orchestra.run.smithi046.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-30T13:19:35.851 INFO:teuthology.orchestra.run.smithi046.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-30T13:19:35.852 INFO:teuthology.orchestra.run.smithi046.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-30T13:19:35.856 INFO:teuthology.orchestra.run.smithi046.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-30T13:19:35.860 INFO:teuthology.orchestra.run.smithi046.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-30T13:19:35.860 INFO:teuthology.orchestra.run.smithi046.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-30T13:19:35.879 INFO:teuthology.orchestra.run.smithi046.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-30T13:19:35.879 INFO:teuthology.orchestra.run.smithi046.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-30T13:19:35.881 INFO:teuthology.orchestra.run.smithi046.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-30T13:19:35.882 INFO:teuthology.orchestra.run.smithi046.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-30T13:19:35.889 INFO:teuthology.orchestra.run.smithi046.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-30T13:19:35.889 INFO:teuthology.orchestra.run.smithi046.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-30T13:19:35.890 INFO:teuthology.orchestra.run.smithi046.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-30T13:19:35.898 INFO:teuthology.orchestra.run.smithi046.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-30T13:19:35.899 INFO:teuthology.orchestra.run.smithi046.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-30T13:19:36.251 INFO:teuthology.orchestra.run.smithi154.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6835-g00c1c13c-1jammy [6,858 kB] 2023-10-30T13:19:36.394 INFO:teuthology.orchestra.run.smithi154.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6835-g00c1c13c-1jammy [238 kB] 2023-10-30T13:19:36.477 INFO:teuthology.orchestra.run.smithi154.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6835-g00c1c13c-1jammy [128 kB] 2023-10-30T13:19:36.479 INFO:teuthology.orchestra.run.smithi154.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6835-g00c1c13c-1jammy [1,496 kB] 2023-10-30T13:19:36.508 INFO:teuthology.orchestra.run.smithi154.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6835-g00c1c13c-1jammy [9,892 kB] 2023-10-30T13:19:36.804 INFO:teuthology.orchestra.run.smithi154.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6835-g00c1c13c-1jammy [31.7 MB] 2023-10-30T13:19:37.646 INFO:teuthology.orchestra.run.smithi046.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6835-g00c1c13c-1jammy [3,131 kB] 2023-10-30T13:19:38.079 INFO:teuthology.orchestra.run.smithi154.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6835-g00c1c13c-1jammy [14.1 kB] 2023-10-30T13:19:38.080 INFO:teuthology.orchestra.run.smithi154.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [839 kB] 2023-10-30T13:19:38.096 INFO:teuthology.orchestra.run.smithi154.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6835-g00c1c13c-1jammy [2,148 kB] 2023-10-30T13:19:38.135 INFO:teuthology.orchestra.run.smithi154.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6835-g00c1c13c-1jammy [284 kB] 2023-10-30T13:19:38.139 INFO:teuthology.orchestra.run.smithi154.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6835-g00c1c13c-1jammy [138 kB] 2023-10-30T13:19:38.142 INFO:teuthology.orchestra.run.smithi154.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6835-g00c1c13c-1jammy [1,907 kB] 2023-10-30T13:19:38.163 INFO:teuthology.orchestra.run.smithi154.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6835-g00c1c13c-1jammy [8,625 kB] 2023-10-30T13:19:38.457 INFO:teuthology.orchestra.run.smithi154.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6835-g00c1c13c-1jammy [14.2 kB] 2023-10-30T13:19:38.457 INFO:teuthology.orchestra.run.smithi154.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6835-g00c1c13c-1jammy [47.8 MB] 2023-10-30T13:19:39.058 INFO:teuthology.orchestra.run.smithi046.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6835-g00c1c13c-1jammy [689 kB] 2023-10-30T13:19:39.442 INFO:teuthology.orchestra.run.smithi046.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6835-g00c1c13c-1jammy [356 kB] 2023-10-30T13:19:39.639 INFO:teuthology.orchestra.run.smithi046.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6835-g00c1c13c-1jammy [32.8 kB] 2023-10-30T13:19:39.648 INFO:teuthology.orchestra.run.smithi046.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6835-g00c1c13c-1jammy [180 kB] 2023-10-30T13:19:39.726 INFO:teuthology.orchestra.run.smithi046.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6835-g00c1c13c-1jammy [62.4 kB] 2023-10-30T13:19:39.782 INFO:teuthology.orchestra.run.smithi046.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6835-g00c1c13c-1jammy [330 kB] 2023-10-30T13:19:39.933 INFO:teuthology.orchestra.run.smithi046.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6835-g00c1c13c-1jammy [6,226 kB] 2023-10-30T13:19:40.376 INFO:teuthology.orchestra.run.smithi154.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6835-g00c1c13c-1jammy [120 kB] 2023-10-30T13:19:40.379 INFO:teuthology.orchestra.run.smithi154.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6835-g00c1c13c-1jammy [227 kB] 2023-10-30T13:19:40.383 INFO:teuthology.orchestra.run.smithi154.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6835-g00c1c13c-1jammy [34.0 kB] 2023-10-30T13:19:40.384 INFO:teuthology.orchestra.run.smithi154.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6835-g00c1c13c-1jammy [23.4 kB] 2023-10-30T13:19:40.385 INFO:teuthology.orchestra.run.smithi154.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6835-g00c1c13c-1jammy [38.3 kB] 2023-10-30T13:19:40.386 INFO:teuthology.orchestra.run.smithi154.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6835-g00c1c13c-1jammy [12.3 MB] 2023-10-30T13:19:40.700 INFO:teuthology.orchestra.run.smithi154.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [91.6 kB] 2023-10-30T13:19:41.248 INFO:teuthology.orchestra.run.smithi046.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6835-g00c1c13c-1jammy [112 kB] 2023-10-30T13:19:41.253 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 184 MB in 6s (30.4 MB/s) 2023-10-30T13:19:41.266 INFO:teuthology.orchestra.run.smithi046.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6835-g00c1c13c-1jammy [439 kB] 2023-10-30T13:19:41.310 INFO:teuthology.orchestra.run.smithi046.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6835-g00c1c13c-1jammy [22.6 MB] 2023-10-30T13:19:41.390 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-30T13:19:41.461 INFO:teuthology.orchestra.run.smithi154.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 130506 files and directories currently installed.) 2023-10-30T13:19:41.465 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-30T13:19:41.505 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-30T13:19:41.783 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-30T13:19:41.803 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-30T13:19:41.841 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T13:19:42.179 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cmd2. 2023-10-30T13:19:42.200 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-30T13:19:42.237 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-30T13:19:42.573 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-30T13:19:42.594 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-30T13:19:42.674 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T13:19:42.960 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-30T13:19:42.981 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-30T13:19:43.010 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T13:19:43.347 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-30T13:19:43.367 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-30T13:19:43.405 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T13:19:43.662 INFO:teuthology.orchestra.run.smithi046.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6835-g00c1c13c-1jammy [6,858 kB] 2023-10-30T13:19:43.741 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-30T13:19:43.762 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-30T13:19:43.800 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T13:19:44.211 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-30T13:19:44.230 INFO:teuthology.orchestra.run.smithi046.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6835-g00c1c13c-1jammy [238 kB] 2023-10-30T13:19:44.232 INFO:teuthology.orchestra.run.smithi046.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6835-g00c1c13c-1jammy [128 kB] 2023-10-30T13:19:44.232 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:19:44.252 INFO:teuthology.orchestra.run.smithi046.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6835-g00c1c13c-1jammy [1,496 kB] 2023-10-30T13:19:44.278 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:19:44.378 INFO:teuthology.orchestra.run.smithi046.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6835-g00c1c13c-1jammy [9,892 kB] 2023-10-30T13:19:44.681 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-30T13:19:44.702 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:19:44.739 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:19:45.093 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-30T13:19:45.114 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:19:45.116 INFO:teuthology.orchestra.run.smithi046.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6835-g00c1c13c-1jammy [31.7 MB] 2023-10-30T13:19:45.151 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:19:45.512 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-30T13:19:45.533 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-30T13:19:45.562 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T13:19:45.899 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librados2. 2023-10-30T13:19:45.919 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../011-librados2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:45.957 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:46.418 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librbd1. 2023-10-30T13:19:46.438 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../012-librbd1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:46.476 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:46.888 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libcephfs2. 2023-10-30T13:19:46.899 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:46.937 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:47.178 INFO:teuthology.orchestra.run.smithi046.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6835-g00c1c13c-1jammy [14.1 kB] 2023-10-30T13:19:47.179 INFO:teuthology.orchestra.run.smithi046.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [839 kB] 2023-10-30T13:19:47.206 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rados. 2023-10-30T13:19:47.218 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:47.228 INFO:teuthology.orchestra.run.smithi046.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6835-g00c1c13c-1jammy [2,148 kB] 2023-10-30T13:19:47.256 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:47.354 INFO:teuthology.orchestra.run.smithi046.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6835-g00c1c13c-1jammy [284 kB] 2023-10-30T13:19:47.368 INFO:teuthology.orchestra.run.smithi046.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6835-g00c1c13c-1jammy [138 kB] 2023-10-30T13:19:47.375 INFO:teuthology.orchestra.run.smithi046.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6835-g00c1c13c-1jammy [1,907 kB] 2023-10-30T13:19:47.481 INFO:teuthology.orchestra.run.smithi046.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6835-g00c1c13c-1jammy [8,625 kB] 2023-10-30T13:19:47.584 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-30T13:19:47.605 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:19:47.642 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:47.887 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cephfs. 2023-10-30T13:19:47.907 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:47.953 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:48.017 INFO:teuthology.orchestra.run.smithi046.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6835-g00c1c13c-1jammy [14.2 kB] 2023-10-30T13:19:48.022 INFO:teuthology.orchestra.run.smithi046.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6835-g00c1c13c-1jammy [47.8 MB] 2023-10-30T13:19:48.281 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-30T13:19:48.302 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:19:48.340 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:48.668 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-prettytable. 2023-10-30T13:19:48.688 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-30T13:19:48.726 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-30T13:19:48.995 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rbd. 2023-10-30T13:19:49.007 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:49.045 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:49.348 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-30T13:19:49.360 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-30T13:19:49.398 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T13:19:49.709 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librgw2. 2023-10-30T13:19:49.730 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../021-librgw2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:49.767 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:50.168 INFO:teuthology.orchestra.run.smithi046.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6835-g00c1c13c-1jammy [120 kB] 2023-10-30T13:19:50.169 INFO:teuthology.orchestra.run.smithi046.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6835-g00c1c13c-1jammy [227 kB] 2023-10-30T13:19:50.171 INFO:teuthology.orchestra.run.smithi046.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6835-g00c1c13c-1jammy [34.0 kB] 2023-10-30T13:19:50.172 INFO:teuthology.orchestra.run.smithi046.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6835-g00c1c13c-1jammy [23.4 kB] 2023-10-30T13:19:50.172 INFO:teuthology.orchestra.run.smithi046.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6835-g00c1c13c-1jammy [38.3 kB] 2023-10-30T13:19:50.173 INFO:teuthology.orchestra.run.smithi046.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6835-g00c1c13c-1jammy [12.3 MB] 2023-10-30T13:19:50.412 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rgw. 2023-10-30T13:19:50.433 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:50.470 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:50.595 INFO:teuthology.orchestra.run.smithi046.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [91.6 kB] 2023-10-30T13:19:50.815 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-30T13:19:50.835 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-30T13:19:50.865 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T13:19:51.184 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libradosstriper1. 2023-10-30T13:19:51.189 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 184 MB in 15s (12.1 MB/s) 2023-10-30T13:19:51.205 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:51.242 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:51.333 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-30T13:19:51.404 INFO:teuthology.orchestra.run.smithi046.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132932 files and directories currently installed.) 2023-10-30T13:19:51.407 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-30T13:19:51.449 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-30T13:19:51.679 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-common. 2023-10-30T13:19:51.700 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:51.701 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-30T13:19:51.715 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-30T13:19:51.737 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:51.752 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T13:19:52.079 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cmd2. 2023-10-30T13:19:52.093 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-30T13:19:52.138 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-30T13:19:52.458 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-30T13:19:52.472 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-30T13:19:52.574 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T13:19:52.877 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-30T13:19:52.892 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-30T13:19:52.927 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T13:19:53.001 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-base. 2023-10-30T13:19:53.022 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:53.069 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:53.281 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-30T13:19:53.296 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-30T13:19:53.339 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T13:19:53.667 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-30T13:19:53.682 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-30T13:19:53.725 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T13:19:53.806 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-30T13:19:53.827 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-30T13:19:53.864 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-30T13:19:54.154 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-30T13:19:54.167 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cheroot. 2023-10-30T13:19:54.169 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:19:54.188 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-30T13:19:54.213 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:19:54.225 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T13:19:54.570 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-30T13:19:54.590 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-30T13:19:54.591 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-30T13:19:54.606 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:19:54.628 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-30T13:19:54.649 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:19:54.914 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-30T13:19:54.935 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-30T13:19:54.972 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-30T13:19:54.993 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-30T13:19:55.008 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:19:55.051 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:19:55.309 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-30T13:19:55.330 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-30T13:19:55.367 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-30T13:19:55.379 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-30T13:19:55.395 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-30T13:19:55.429 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T13:19:55.670 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-tempora. 2023-10-30T13:19:55.691 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-30T13:19:55.732 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librados2. 2023-10-30T13:19:55.737 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-30T13:19:55.748 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../011-librados2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:55.790 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:56.065 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-portend. 2023-10-30T13:19:56.086 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-30T13:19:56.123 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-30T13:19:56.394 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librbd1. 2023-10-30T13:19:56.400 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-30T13:19:56.409 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../012-librbd1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:56.422 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-30T13:19:56.451 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:56.459 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-30T13:19:56.762 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-30T13:19:56.783 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-30T13:19:56.828 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-30T13:19:56.846 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libcephfs2. 2023-10-30T13:19:56.862 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:56.896 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:57.107 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rados. 2023-10-30T13:19:57.122 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:57.149 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-natsort. 2023-10-30T13:19:57.165 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:57.170 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-30T13:19:57.207 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-30T13:19:57.510 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-30T13:19:57.510 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-logutils. 2023-10-30T13:19:57.525 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:19:57.530 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-30T13:19:57.568 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:57.568 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-30T13:19:57.812 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cephfs. 2023-10-30T13:19:57.827 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:57.854 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-30T13:19:57.870 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:57.876 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-30T13:19:57.913 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-30T13:19:58.181 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-30T13:19:58.197 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:19:58.199 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-mako. 2023-10-30T13:19:58.220 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-30T13:19:58.232 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:58.257 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T13:19:58.543 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-prettytable. 2023-10-30T13:19:58.558 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-30T13:19:58.560 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-30T13:19:58.572 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-30T13:19:58.593 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-30T13:19:58.602 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-30T13:19:58.846 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rbd. 2023-10-30T13:19:58.861 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:58.904 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:58.913 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-30T13:19:58.934 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-30T13:19:58.971 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-30T13:19:59.248 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-30T13:19:59.263 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-30T13:19:59.274 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-webob. 2023-10-30T13:19:59.295 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-30T13:19:59.306 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T13:19:59.332 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-30T13:19:59.651 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package librgw2. 2023-10-30T13:19:59.667 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../021-librgw2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:19:59.694 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-waitress. 2023-10-30T13:19:59.709 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:19:59.714 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-30T13:19:59.755 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T13:20:00.097 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-tempita. 2023-10-30T13:20:00.118 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-30T13:20:00.147 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T13:20:00.155 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rgw. 2023-10-30T13:20:00.171 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:00.205 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:00.425 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-paste. 2023-10-30T13:20:00.445 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-30T13:20:00.483 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T13:20:00.533 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-30T13:20:00.548 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-30T13:20:00.616 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T13:20:00.811 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-30T13:20:00.823 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-30T13:20:00.861 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T13:20:00.944 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libradosstriper1. 2023-10-30T13:20:00.960 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:01.003 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:01.156 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-30T13:20:01.176 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-30T13:20:01.214 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-30T13:20:01.431 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-common. 2023-10-30T13:20:01.447 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:01.489 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:01.517 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-webtest. 2023-10-30T13:20:01.537 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-30T13:20:01.575 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-30T13:20:01.861 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pecan. 2023-10-30T13:20:01.873 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-30T13:20:01.903 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T13:20:02.274 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-certifi. 2023-10-30T13:20:02.295 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-30T13:20:02.332 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-30T13:20:02.578 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-base. 2023-10-30T13:20:02.594 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:02.646 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:02.660 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-idna. 2023-10-30T13:20:02.681 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-30T13:20:02.718 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-30T13:20:03.064 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-urllib3. 2023-10-30T13:20:03.084 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-30T13:20:03.122 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T13:20:03.291 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-30T13:20:03.306 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-30T13:20:03.349 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-30T13:20:03.450 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-requests. 2023-10-30T13:20:03.470 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-30T13:20:03.525 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T13:20:03.652 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cheroot. 2023-10-30T13:20:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-30T13:20:03.710 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T13:20:03.786 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-30T13:20:03.807 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-30T13:20:03.844 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T13:20:04.088 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-30T13:20:04.104 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-30T13:20:04.138 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-30T13:20:04.156 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-30T13:20:04.168 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:04.206 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:04.383 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-30T13:20:04.398 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-30T13:20:04.441 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-30T13:20:04.602 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-30T13:20:04.623 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:04.660 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:04.744 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-30T13:20:04.760 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-30T13:20:04.802 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-30T13:20:05.005 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-openssl. 2023-10-30T13:20:05.026 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-30T13:20:05.063 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-30T13:20:05.130 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-tempora. 2023-10-30T13:20:05.146 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-30T13:20:05.189 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-30T13:20:05.383 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr. 2023-10-30T13:20:05.404 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:05.433 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-portend. 2023-10-30T13:20:05.441 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:05.448 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-30T13:20:05.483 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-30T13:20:05.769 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-30T13:20:05.785 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-30T13:20:05.827 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-30T13:20:05.828 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mon. 2023-10-30T13:20:05.841 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:05.886 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:06.172 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-30T13:20:06.188 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-30T13:20:06.230 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-30T13:20:06.815 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-osd. 2023-10-30T13:20:06.825 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-natsort. 2023-10-30T13:20:06.835 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:06.841 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-30T13:20:06.873 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:06.884 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-30T13:20:07.228 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-logutils. 2023-10-30T13:20:07.244 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-30T13:20:07.286 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-30T13:20:07.589 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-30T13:20:07.605 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-30T13:20:07.648 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-30T13:20:07.892 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-mako. 2023-10-30T13:20:07.908 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-30T13:20:08.076 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T13:20:08.846 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-30T13:20:08.861 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-30T13:20:08.904 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-30T13:20:09.215 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-30T13:20:09.231 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-30T13:20:09.273 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-30T13:20:09.326 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph. 2023-10-30T13:20:09.347 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../061-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:09.393 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:09.551 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-webob. 2023-10-30T13:20:09.567 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-30T13:20:09.618 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-30T13:20:09.787 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-fuse. 2023-10-30T13:20:09.808 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:09.862 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:09.912 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-waitress. 2023-10-30T13:20:09.928 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-30T13:20:09.965 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T13:20:10.274 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-tempita. 2023-10-30T13:20:10.290 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-30T13:20:10.307 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mds. 2023-10-30T13:20:10.329 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:10.332 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T13:20:10.374 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-paste. 2023-10-30T13:20:10.601 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-30T13:20:10.668 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T13:20:10.980 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-30T13:20:10.995 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-30T13:20:11.037 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T13:20:11.210 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package cephadm. 2023-10-30T13:20:11.232 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../064-cephadm_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:11.268 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:11.332 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-30T13:20:11.348 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-30T13:20:11.382 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-30T13:20:11.622 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-30T13:20:11.643 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-30T13:20:11.652 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-webtest. 2023-10-30T13:20:11.668 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-30T13:20:11.688 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-30T13:20:11.743 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-30T13:20:12.032 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-30T13:20:12.045 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T13:20:12.079 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pecan. 2023-10-30T13:20:12.082 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T13:20:12.095 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-30T13:20:12.138 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T13:20:12.493 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-certifi. 2023-10-30T13:20:12.509 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-30T13:20:12.551 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-30T13:20:12.879 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-idna. 2023-10-30T13:20:12.895 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-30T13:20:12.937 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-30T13:20:13.146 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-babel. 2023-10-30T13:20:13.167 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T13:20:13.204 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T13:20:13.215 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-urllib3. 2023-10-30T13:20:13.231 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-30T13:20:13.265 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T13:20:13.535 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-requests. 2023-10-30T13:20:13.549 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jinja2. 2023-10-30T13:20:13.551 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-30T13:20:13.570 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-30T13:20:13.585 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T13:20:13.607 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-30T13:20:13.905 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-30T13:20:13.920 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-30T13:20:13.952 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-30T13:20:13.963 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T13:20:13.973 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:14.010 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:14.224 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-30T13:20:14.240 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:14.274 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:14.398 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-30T13:20:14.419 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-30T13:20:14.456 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-30T13:20:14.654 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-30T13:20:14.669 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:14.712 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:14.767 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-30T13:20:14.787 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-30T13:20:14.817 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-30T13:20:14.998 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-openssl. 2023-10-30T13:20:15.014 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-30T13:20:15.048 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-30T13:20:15.120 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-routes. 2023-10-30T13:20:15.141 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-30T13:20:15.178 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T13:20:15.384 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr. 2023-10-30T13:20:15.400 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:15.442 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:15.507 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-30T13:20:15.527 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:15.565 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:15.787 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mon. 2023-10-30T13:20:15.802 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:15.837 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:16.018 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-30T13:20:16.030 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-30T13:20:16.068 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T13:20:16.597 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-joblib. 2023-10-30T13:20:16.618 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-30T13:20:16.633 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-osd. 2023-10-30T13:20:16.648 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:16.663 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T13:20:16.691 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:17.017 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-30T13:20:17.039 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-30T13:20:17.083 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-30T13:20:17.386 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-sklearn. 2023-10-30T13:20:17.408 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-30T13:20:17.453 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T13:20:18.003 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph. 2023-10-30T13:20:18.018 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../061-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:18.061 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:18.120 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-30T13:20:18.142 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:18.186 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:18.347 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-fuse. 2023-10-30T13:20:18.362 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:18.397 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:18.750 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mds. 2023-10-30T13:20:18.766 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:18.808 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:19.178 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package cephadm. 2023-10-30T13:20:19.182 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cachetools. 2023-10-30T13:20:19.193 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../064-cephadm_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:19.203 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-30T13:20:19.236 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:19.248 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-30T13:20:19.514 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-30T13:20:19.530 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-30T13:20:19.564 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-30T13:20:19.576 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-30T13:20:19.598 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-30T13:20:19.642 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-30T13:20:19.850 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-30T13:20:19.866 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T13:20:19.901 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T13:20:19.971 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-30T13:20:19.983 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-30T13:20:20.029 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T13:20:20.349 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rsa. 2023-10-30T13:20:20.370 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-30T13:20:20.407 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-30T13:20:20.812 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-google-auth. 2023-10-30T13:20:20.833 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-30T13:20:20.870 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-30T13:20:20.914 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-babel. 2023-10-30T13:20:20.928 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T13:20:20.972 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T13:20:21.317 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-jinja2. 2023-10-30T13:20:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-30T13:20:21.375 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-30T13:20:21.434 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-30T13:20:21.456 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-30T13:20:21.500 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T13:20:21.745 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-30T13:20:21.760 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:21.803 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:21.812 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-websocket. 2023-10-30T13:20:21.825 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-30T13:20:21.861 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-30T13:20:22.140 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-30T13:20:22.149 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-30T13:20:22.156 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-30T13:20:22.170 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-30T13:20:22.198 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-30T13:20:22.215 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T13:20:22.460 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-30T13:20:22.475 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-30T13:20:22.509 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-30T13:20:22.804 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-routes. 2023-10-30T13:20:22.819 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-30T13:20:22.862 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T13:20:22.904 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-30T13:20:22.917 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:22.953 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:23.158 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-30T13:20:23.173 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:23.216 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:23.273 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-30T13:20:23.295 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-30T13:20:23.348 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T13:20:23.586 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-30T13:20:23.600 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-30T13:20:23.636 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T13:20:23.710 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-30T13:20:23.732 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T13:20:23.776 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T13:20:24.054 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package jq. 2023-10-30T13:20:24.075 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T13:20:24.112 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-30T13:20:24.148 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-joblib. 2023-10-30T13:20:24.163 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-30T13:20:24.206 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T13:20:24.407 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package socat. 2023-10-30T13:20:24.419 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-30T13:20:24.465 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-30T13:20:24.610 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-30T13:20:24.626 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-30T13:20:24.676 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-30T13:20:24.793 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package xmlstarlet. 2023-10-30T13:20:24.805 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-30T13:20:24.851 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-30T13:20:25.063 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-sklearn. 2023-10-30T13:20:25.079 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-30T13:20:25.129 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T13:20:25.255 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-test. 2023-10-30T13:20:25.277 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:25.330 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:25.805 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-30T13:20:25.819 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:25.879 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:26.816 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-cachetools. 2023-10-30T13:20:26.832 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-30T13:20:26.874 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-30T13:20:27.144 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-30T13:20:27.160 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-30T13:20:27.193 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-30T13:20:27.514 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-30T13:20:27.529 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-30T13:20:27.572 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T13:20:27.643 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-volume. 2023-10-30T13:20:27.664 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:27.701 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:27.858 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-rsa. 2023-10-30T13:20:27.874 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-30T13:20:27.916 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-30T13:20:28.055 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-30T13:20:28.076 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:28.121 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:28.441 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-colorama. 2023-10-30T13:20:28.463 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-30T13:20:28.508 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-30T13:20:28.530 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-google-auth. 2023-10-30T13:20:28.546 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-30T13:20:28.588 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-30T13:20:28.752 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package cephfs-shell. 2023-10-30T13:20:28.764 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:28.802 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:28.900 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-30T13:20:28.916 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-30T13:20:29.058 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T13:20:29.080 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package cephfs-top. 2023-10-30T13:20:29.092 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:29.130 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:29.336 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-websocket. 2023-10-30T13:20:29.352 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-30T13:20:29.375 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-30T13:20:29.394 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-30T13:20:29.396 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:29.441 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:29.689 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-30T13:20:29.705 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-30T13:20:29.747 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T13:20:29.944 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package nvme-cli. 2023-10-30T13:20:29.965 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-30T13:20:30.010 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T13:20:30.366 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-30T13:20:30.387 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-30T13:20:30.432 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-30T13:20:30.453 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-30T13:20:30.467 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:30.511 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:30.797 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-30T13:20:30.812 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-30T13:20:30.847 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T13:20:30.861 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-30T13:20:30.882 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-30T13:20:30.927 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-30T13:20:31.199 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-30T13:20:31.214 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T13:20:31.230 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pastescript. 2023-10-30T13:20:31.251 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-30T13:20:31.258 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T13:20:31.296 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-30T13:20:31.520 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package jq. 2023-10-30T13:20:31.535 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T13:20:31.577 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-30T13:20:31.616 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pluggy. 2023-10-30T13:20:31.638 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-30T13:20:31.674 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-30T13:20:31.830 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package socat. 2023-10-30T13:20:31.846 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-30T13:20:31.889 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-30T13:20:31.986 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-psutil. 2023-10-30T13:20:32.007 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-30T13:20:32.052 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-30T13:20:32.175 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package xmlstarlet. 2023-10-30T13:20:32.191 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-30T13:20:32.225 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-30T13:20:32.380 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-py. 2023-10-30T13:20:32.402 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-30T13:20:32.446 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-30T13:20:32.596 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-test. 2023-10-30T13:20:32.611 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:32.654 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:32.758 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pygments. 2023-10-30T13:20:32.780 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-30T13:20:32.825 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T13:20:33.287 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-30T13:20:33.309 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-30T13:20:33.354 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-30T13:20:33.673 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-toml. 2023-10-30T13:20:33.694 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-30T13:20:33.740 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-30T13:20:34.076 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pytest. 2023-10-30T13:20:34.089 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-30T13:20:34.126 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T13:20:34.471 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-simplejson. 2023-10-30T13:20:34.493 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-30T13:20:34.533 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ceph-volume. 2023-10-30T13:20:34.537 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-30T13:20:34.549 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:34.583 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:34.807 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-30T13:20:34.820 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-30T13:20:34.857 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-30T13:20:34.921 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-30T13:20:34.937 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:34.979 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:35.299 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-colorama. 2023-10-30T13:20:35.315 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-30T13:20:35.357 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-30T13:20:35.643 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package cephfs-shell. 2023-10-30T13:20:35.659 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:35.686 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package radosgw. 2023-10-30T13:20:35.701 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:35.708 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../113-radosgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:35.761 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:35.929 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package cephfs-top. 2023-10-30T13:20:35.945 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:20:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:36.207 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-30T13:20:36.223 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:36.265 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:36.597 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package rbd-fuse. 2023-10-30T13:20:36.619 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:36.672 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:36.689 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package nvme-cli. 2023-10-30T13:20:36.705 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-30T13:20:36.751 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T13:20:37.090 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-30T13:20:37.105 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-30T13:20:37.148 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-30T13:20:37.349 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-30T13:20:37.586 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-30T13:20:37.602 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T13:20:37.603 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-30T13:20:37.644 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-30T13:20:37.728 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T13:20:37.972 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pastescript. 2023-10-30T13:20:37.989 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-30T13:20:37.989 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-30T13:20:38.030 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-30T13:20:38.317 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pluggy. 2023-10-30T13:20:38.333 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-30T13:20:38.366 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-30T13:20:38.372 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-30T13:20:38.653 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-psutil. 2023-10-30T13:20:38.669 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-30T13:20:38.711 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-30T13:20:38.889 INFO:teuthology.orchestra.run.smithi154.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-30T13:20:39.005 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:39.089 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-py. 2023-10-30T13:20:39.105 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-30T13:20:39.147 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-30T13:20:39.475 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pygments. 2023-10-30T13:20:39.491 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-30T13:20:39.533 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T13:20:39.607 INFO:teuthology.orchestra.run.smithi154.stdout:Adding system user cephadm....done 2023-10-30T13:20:39.770 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T13:20:40.046 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-30T13:20:40.054 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-30T13:20:40.061 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-30T13:20:40.104 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-30T13:20:40.306 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-30T13:20:40.423 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-toml. 2023-10-30T13:20:40.432 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-30T13:20:40.439 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-30T13:20:40.481 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-30T13:20:40.683 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-30T13:20:40.751 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-pytest. 2023-10-30T13:20:40.767 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-30T13:20:40.801 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T13:20:40.934 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T13:20:41.060 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-30T13:20:41.171 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package python3-simplejson. 2023-10-30T13:20:41.187 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-30T13:20:41.229 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-30T13:20:41.370 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T13:20:41.498 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-30T13:20:41.514 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-30T13:20:41.557 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-30T13:20:41.730 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-30T13:20:41.982 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-30T13:20:42.241 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-30T13:20:42.277 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package radosgw. 2023-10-30T13:20:42.292 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../113-radosgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:42.335 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:42.501 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-30T13:20:42.743 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-30T13:20:43.003 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-30T13:20:43.029 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package rbd-fuse. 2023-10-30T13:20:43.046 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:20:43.088 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:43.263 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:43.407 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-30T13:20:43.522 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T13:20:43.658 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-30T13:20:43.768 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T13:20:43.886 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T13:20:43.916 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-30T13:20:44.164 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-30T13:20:44.169 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T13:20:44.293 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-30T13:20:44.621 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-30T13:20:44.661 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T13:20:44.787 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-30T13:20:45.063 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T13:20:45.251 INFO:teuthology.orchestra.run.smithi046.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-30T13:20:45.306 INFO:teuthology.orchestra.run.smithi046.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2023-10-30T13:20:45.314 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-30T13:20:45.353 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:45.624 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T13:20:46.024 INFO:teuthology.orchestra.run.smithi046.stdout:Adding system user cephadm....done 2023-10-30T13:20:46.184 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T13:20:46.343 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T13:20:46.469 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-30T13:20:46.602 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T13:20:46.728 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-30T13:20:46.728 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-30T13:20:46.854 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T13:20:46.855 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-30T13:20:47.155 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-30T13:20:47.189 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-30T13:20:47.473 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T13:20:47.540 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-30T13:20:47.599 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-30T13:20:47.791 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T13:20:47.917 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T13:20:48.143 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T13:20:48.295 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-30T13:20:48.427 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-30T13:20:48.579 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-30T13:20:48.770 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T13:20:48.830 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-30T13:20:49.106 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-30T13:20:49.113 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-30T13:20:49.364 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-30T13:20:49.374 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-30T13:20:49.491 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-30T13:20:49.783 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-idna (3.3-1) ... 2023-10-30T13:20:50.001 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:50.126 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T13:20:50.260 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T13:20:50.378 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-30T13:20:50.646 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-30T13:20:50.930 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T13:20:51.031 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-30T13:20:51.340 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:20:51.391 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T13:20:51.458 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-30T13:20:51.516 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-30T13:20:51.726 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T13:20:51.815 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T13:20:51.852 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-30T13:20:51.978 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T13:20:52.077 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-30T13:20:52.288 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-30T13:20:52.395 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T13:20:52.739 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-30T13:20:52.998 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:20:53.114 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T13:20:53.116 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T13:20:53.234 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-30T13:20:53.381 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T13:20:53.491 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-30T13:20:53.535 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-30T13:20:53.616 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T13:20:53.778 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-30T13:20:53.976 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-30T13:20:54.029 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-30T13:20:54.280 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-30T13:20:54.369 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-30T13:20:54.540 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-30T13:20:54.637 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T13:20:54.824 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T13:20:54.925 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-30T13:20:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T13:20:55.235 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T13:20:55.353 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T13:20:55.356 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-30T13:20:55.654 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-30T13:20:55.699 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T13:20:55.955 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-30T13:20:56.084 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-30T13:20:56.206 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T13:20:56.351 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-30T13:20:56.477 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:20:56.696 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-30T13:20:56.787 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-idna (3.3-1) ... 2023-10-30T13:20:56.959 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T13:20:57.086 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T13:20:57.154 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T13:20:57.461 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-30T13:20:57.713 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-30T13:20:57.964 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:20:58.098 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-30T13:20:58.419 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:20:58.424 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T13:20:58.544 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-30T13:20:58.726 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T13:20:58.829 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T13:20:58.930 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-30T13:20:59.010 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-30T13:20:59.031 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T13:20:59.136 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-30T13:20:59.332 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-30T13:20:59.404 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-30T13:20:59.608 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-30T13:20:59.672 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T13:20:59.859 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:20:59.948 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-30T13:20:59.960 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T13:21:00.069 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-30T13:21:00.316 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-30T13:21:00.388 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-30T13:21:00.592 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T13:21:00.639 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-30T13:21:00.906 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-30T13:21:00.944 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T13:21:01.070 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:01.174 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-30T13:21:01.213 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:01.425 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:01.442 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-30T13:21:01.551 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:01.677 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:01.718 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T13:21:01.803 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-30T13:21:01.827 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-30T13:21:02.080 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:02.112 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T13:21:02.206 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:02.238 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T13:21:02.340 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:02.539 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-30T13:21:02.570 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T13:21:02.570 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T13:21:02.831 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-30T13:21:03.083 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:03.108 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T13:21:03.235 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:03.344 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:03.453 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T13:21:03.510 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-30T13:21:03.827 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T13:21:03.945 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T13:21:04.355 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-30T13:21:04.614 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-30T13:21:04.691 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:04.800 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:04.882 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:21:04.926 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:05.008 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-30T13:21:05.052 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:05.178 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:05.304 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:05.350 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T13:21:05.685 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T13:21:05.873 INFO:teuthology.orchestra.run.smithi154.stdout:Adding group ceph....done 2023-10-30T13:21:05.978 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-30T13:21:06.104 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-30T13:21:06.349 INFO:teuthology.orchestra.run.smithi154.stdout:Adding system user ceph....done 2023-10-30T13:21:06.389 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-30T13:21:06.535 INFO:teuthology.orchestra.run.smithi154.stdout:Setting system user ceph properties....done 2023-10-30T13:21:06.553 INFO:teuthology.orchestra.run.smithi154.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-30T13:21:06.664 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T13:21:06.691 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-30T13:21:06.966 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-30T13:21:07.105 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-30T13:21:07.359 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-30T13:21:07.660 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T13:21:07.690 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:08.035 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T13:21:08.035 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T13:21:08.037 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T13:21:08.163 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:08.289 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:08.500 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:08.593 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:08.601 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:08.711 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:08.745 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:08.829 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-30T13:21:09.147 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:09.248 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:09.260 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T13:21:09.260 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T13:21:09.357 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:09.600 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T13:21:09.601 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T13:21:09.788 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:10.130 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-30T13:21:10.492 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:10.593 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:10.623 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:10.703 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:10.829 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T13:21:10.864 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T13:21:10.865 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T13:21:11.442 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:11.658 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T13:21:11.658 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T13:21:12.175 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:12.226 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:12.300 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:12.426 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:12.489 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T13:21:12.489 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T13:21:12.553 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:12.695 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:12.804 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:13.053 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:13.195 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:13.317 INFO:teuthology.orchestra.run.smithi046.stdout:Adding group ceph....done 2023-10-30T13:21:13.393 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:13.668 INFO:teuthology.orchestra.run.smithi046.stdout:Adding system user ceph....done 2023-10-30T13:21:13.804 INFO:teuthology.orchestra.run.smithi046.stdout:Setting system user ceph properties....done 2023-10-30T13:21:13.825 INFO:teuthology.orchestra.run.smithi046.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-30T13:21:13.830 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T13:21:13.830 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T13:21:13.973 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-30T13:21:14.362 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:14.395 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-30T13:21:14.513 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:14.639 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:14.790 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:15.073 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:15.143 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T13:21:15.288 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T13:21:15.288 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T13:21:15.916 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:16.042 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:16.235 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T13:21:16.588 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T13:21:16.589 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T13:21:16.682 INFO:teuthology.orchestra.run.smithi154.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T13:21:16.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-30T13:21:16.744 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2023-10-30T13:21:16.931 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2023-10-30T13:21:16.932 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2023-10-30T13:21:17.097 INFO:teuthology.orchestra.run.smithi154.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-30T13:21:17.097 INFO:teuthology.orchestra.run.smithi154.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-30T13:21:17.097 INFO:teuthology.orchestra.run.smithi154.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-30T13:21:17.097 INFO:teuthology.orchestra.run.smithi154.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-30T13:21:17.098 INFO:teuthology.orchestra.run.smithi154.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-30T13:21:17.098 INFO:teuthology.orchestra.run.smithi154.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T13:21:17.098 INFO:teuthology.orchestra.run.smithi154.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T13:21:17.098 INFO:teuthology.orchestra.run.smithi154.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T13:21:17.099 INFO:teuthology.orchestra.run.smithi154.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T13:21:17.099 INFO:teuthology.orchestra.run.smithi154.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-30T13:21:17.099 INFO:teuthology.orchestra.run.smithi154.stdout: python2.7-minimal 2023-10-30T13:21:17.099 INFO:teuthology.orchestra.run.smithi154.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T13:21:17.100 INFO:teuthology.orchestra.run.smithi154.stdout:The following additional packages will be installed: 2023-10-30T13:21:17.100 INFO:teuthology.orchestra.run.smithi154.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-30T13:21:17.100 INFO:teuthology.orchestra.run.smithi154.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-30T13:21:17.101 INFO:teuthology.orchestra.run.smithi154.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-30T13:21:17.101 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-30T13:21:17.101 INFO:teuthology.orchestra.run.smithi154.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T13:21:17.101 INFO:teuthology.orchestra.run.smithi154.stdout:Suggested packages: 2023-10-30T13:21:17.101 INFO:teuthology.orchestra.run.smithi154.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-30T13:21:17.102 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-30T13:21:17.102 INFO:teuthology.orchestra.run.smithi154.stdout: libengine-pkcs11-openssl 2023-10-30T13:21:17.136 INFO:teuthology.orchestra.run.smithi154.stdout:The following NEW packages will be installed: 2023-10-30T13:21:17.137 INFO:teuthology.orchestra.run.smithi154.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-30T13:21:17.137 INFO:teuthology.orchestra.run.smithi154.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-30T13:21:17.138 INFO:teuthology.orchestra.run.smithi154.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-30T13:21:17.138 INFO:teuthology.orchestra.run.smithi154.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-30T13:21:17.138 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-30T13:21:17.138 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-30T13:21:17.139 INFO:teuthology.orchestra.run.smithi154.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T13:21:17.139 INFO:teuthology.orchestra.run.smithi154.stdout:The following packages will be upgraded: 2023-10-30T13:21:17.139 INFO:teuthology.orchestra.run.smithi154.stdout: libssl3 2023-10-30T13:21:17.210 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:17.319 INFO:teuthology.orchestra.run.smithi154.stdout:1 upgraded, 39 newly installed, 0 to remove and 259 not upgraded. 2023-10-30T13:21:17.319 INFO:teuthology.orchestra.run.smithi154.stdout:Need to get 54.6 MB of archives. 2023-10-30T13:21:17.319 INFO:teuthology.orchestra.run.smithi154.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-30T13:21:17.319 INFO:teuthology.orchestra.run.smithi154.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-30T13:21:17.607 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-30T13:21:17.799 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-30T13:21:18.055 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-30T13:21:18.090 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-30T13:21:18.092 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-30T13:21:18.097 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-30T13:21:18.100 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-30T13:21:18.101 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-30T13:21:18.104 INFO:teuthology.orchestra.run.smithi154.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-30T13:21:18.106 INFO:teuthology.orchestra.run.smithi154.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-30T13:21:18.270 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:18.272 INFO:teuthology.orchestra.run.smithi154.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-30T13:21:18.298 INFO:teuthology.orchestra.run.smithi154.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-30T13:21:18.490 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T13:21:18.491 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T13:21:18.515 INFO:teuthology.orchestra.run.smithi154.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-30T13:21:19.147 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:19.367 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T13:21:19.367 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T13:21:19.659 INFO:teuthology.orchestra.run.smithi154.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-30T13:21:19.664 INFO:teuthology.orchestra.run.smithi154.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-30T13:21:19.664 INFO:teuthology.orchestra.run.smithi154.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-30T13:21:19.670 INFO:teuthology.orchestra.run.smithi154.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-30T13:21:19.671 INFO:teuthology.orchestra.run.smithi154.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-30T13:21:19.681 INFO:teuthology.orchestra.run.smithi154.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-30T13:21:19.684 INFO:teuthology.orchestra.run.smithi154.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-30T13:21:19.700 INFO:teuthology.orchestra.run.smithi154.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-30T13:21:19.865 INFO:teuthology.orchestra.run.smithi154.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-30T13:21:20.074 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:20.474 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T13:21:20.474 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T13:21:20.586 INFO:teuthology.orchestra.run.smithi154.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-30T13:21:20.587 INFO:teuthology.orchestra.run.smithi154.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-30T13:21:20.613 INFO:teuthology.orchestra.run.smithi154.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-30T13:21:20.691 INFO:teuthology.orchestra.run.smithi154.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-30T13:21:20.699 INFO:teuthology.orchestra.run.smithi154.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-30T13:21:20.701 INFO:teuthology.orchestra.run.smithi154.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-30T13:21:20.702 INFO:teuthology.orchestra.run.smithi154.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-30T13:21:20.703 INFO:teuthology.orchestra.run.smithi154.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-30T13:21:20.729 INFO:teuthology.orchestra.run.smithi154.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-30T13:21:20.732 INFO:teuthology.orchestra.run.smithi154.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-30T13:21:21.170 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:21.363 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:21.382 INFO:teuthology.orchestra.run.smithi154.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-30T13:21:21.466 INFO:teuthology.orchestra.run.smithi154.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-30T13:21:21.467 INFO:teuthology.orchestra.run.smithi154.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-30T13:21:21.468 INFO:teuthology.orchestra.run.smithi154.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-30T13:21:21.555 INFO:teuthology.orchestra.run.smithi154.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-30T13:21:21.561 INFO:teuthology.orchestra.run.smithi154.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-30T13:21:21.591 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:21.901 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T13:21:21.901 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T13:21:22.017 INFO:teuthology.orchestra.run.smithi154.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-30T13:21:22.019 INFO:teuthology.orchestra.run.smithi154.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-30T13:21:22.415 INFO:teuthology.orchestra.run.smithi154.stdout:Preconfiguring packages ... 2023-10-30T13:21:22.585 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 54.6 MB in 5s (11.2 MB/s) 2023-10-30T13:21:22.587 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:22.742 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package flex. 2023-10-30T13:21:22.747 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:22.832 INFO:teuthology.orchestra.run.smithi154.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137425 files and directories currently installed.) 2023-10-30T13:21:22.836 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-30T13:21:22.864 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:22.874 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-30T13:21:23.015 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:21:23.376 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T13:21:23.460 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T13:21:23.696 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-30T13:21:24.128 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:21:24.151 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T13:21:24.631 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package bison. 2023-10-30T13:21:24.650 INFO:teuthology.orchestra.run.smithi046.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T13:21:24.655 DEBUG:teuthology.orchestra.run.smithi046:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-30T13:21:24.720 INFO:teuthology.orchestra.run.smithi154.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137498 files and directories currently installed.) 2023-10-30T13:21:24.724 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-30T13:21:24.733 INFO:teuthology.orchestra.run.smithi046.stdout:Reading package lists... 2023-10-30T13:21:24.774 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T13:21:24.902 INFO:teuthology.orchestra.run.smithi046.stdout:Building dependency tree... 2023-10-30T13:21:24.903 INFO:teuthology.orchestra.run.smithi046.stdout:Reading state information... 2023-10-30T13:21:25.056 INFO:teuthology.orchestra.run.smithi046.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-30T13:21:25.057 INFO:teuthology.orchestra.run.smithi046.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-30T13:21:25.057 INFO:teuthology.orchestra.run.smithi046.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-30T13:21:25.057 INFO:teuthology.orchestra.run.smithi046.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-30T13:21:25.057 INFO:teuthology.orchestra.run.smithi046.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-30T13:21:25.057 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T13:21:25.058 INFO:teuthology.orchestra.run.smithi046.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T13:21:25.058 INFO:teuthology.orchestra.run.smithi046.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T13:21:25.058 INFO:teuthology.orchestra.run.smithi046.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T13:21:25.059 INFO:teuthology.orchestra.run.smithi046.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-30T13:21:25.059 INFO:teuthology.orchestra.run.smithi046.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-30T13:21:25.059 INFO:teuthology.orchestra.run.smithi046.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T13:21:25.060 INFO:teuthology.orchestra.run.smithi046.stdout:The following additional packages will be installed: 2023-10-30T13:21:25.060 INFO:teuthology.orchestra.run.smithi046.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-30T13:21:25.060 INFO:teuthology.orchestra.run.smithi046.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-30T13:21:25.060 INFO:teuthology.orchestra.run.smithi046.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-30T13:21:25.060 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-30T13:21:25.060 INFO:teuthology.orchestra.run.smithi046.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T13:21:25.061 INFO:teuthology.orchestra.run.smithi046.stdout:Suggested packages: 2023-10-30T13:21:25.061 INFO:teuthology.orchestra.run.smithi046.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-30T13:21:25.061 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-30T13:21:25.061 INFO:teuthology.orchestra.run.smithi046.stdout: libengine-pkcs11-openssl 2023-10-30T13:21:25.110 INFO:teuthology.orchestra.run.smithi046.stdout:The following NEW packages will be installed: 2023-10-30T13:21:25.111 INFO:teuthology.orchestra.run.smithi046.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-30T13:21:25.111 INFO:teuthology.orchestra.run.smithi046.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-30T13:21:25.112 INFO:teuthology.orchestra.run.smithi046.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-30T13:21:25.112 INFO:teuthology.orchestra.run.smithi046.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-30T13:21:25.112 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-30T13:21:25.112 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-30T13:21:25.112 INFO:teuthology.orchestra.run.smithi046.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T13:21:25.113 INFO:teuthology.orchestra.run.smithi046.stdout:The following packages will be upgraded: 2023-10-30T13:21:25.113 INFO:teuthology.orchestra.run.smithi046.stdout: libssl3 2023-10-30T13:21:25.225 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-30T13:21:25.247 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-30T13:21:25.292 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T13:21:25.297 INFO:teuthology.orchestra.run.smithi046.stdout:1 upgraded, 39 newly installed, 0 to remove and 259 not upgraded. 2023-10-30T13:21:25.297 INFO:teuthology.orchestra.run.smithi046.stdout:Need to get 54.6 MB of archives. 2023-10-30T13:21:25.297 INFO:teuthology.orchestra.run.smithi046.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-30T13:21:25.298 INFO:teuthology.orchestra.run.smithi046.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-30T13:21:25.637 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package dump. 2023-10-30T13:21:25.658 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-30T13:21:25.703 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking dump (0.4b47-3) ... 2023-10-30T13:21:25.790 INFO:teuthology.orchestra.run.smithi046.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-30T13:21:25.981 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package indent. 2023-10-30T13:21:25.994 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:21:26.031 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T13:21:26.047 INFO:teuthology.orchestra.run.smithi046.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-30T13:21:26.082 INFO:teuthology.orchestra.run.smithi046.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-30T13:21:26.083 INFO:teuthology.orchestra.run.smithi046.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-30T13:21:26.088 INFO:teuthology.orchestra.run.smithi046.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-30T13:21:26.092 INFO:teuthology.orchestra.run.smithi046.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-30T13:21:26.093 INFO:teuthology.orchestra.run.smithi046.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-30T13:21:26.096 INFO:teuthology.orchestra.run.smithi046.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-30T13:21:26.098 INFO:teuthology.orchestra.run.smithi046.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-30T13:21:26.263 INFO:teuthology.orchestra.run.smithi046.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-30T13:21:26.263 INFO:teuthology.orchestra.run.smithi046.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-30T13:21:26.300 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-30T13:21:26.313 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-30T13:21:26.350 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T13:21:26.513 INFO:teuthology.orchestra.run.smithi046.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-30T13:21:26.679 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-30T13:21:26.701 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-30T13:21:26.746 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T13:21:27.015 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-30T13:21:27.037 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-30T13:21:27.081 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T13:21:27.441 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-30T13:21:27.462 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-30T13:21:27.501 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T13:21:27.517 INFO:teuthology.orchestra.run.smithi046.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-30T13:21:27.518 INFO:teuthology.orchestra.run.smithi046.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-30T13:21:27.519 INFO:teuthology.orchestra.run.smithi046.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-30T13:21:27.522 INFO:teuthology.orchestra.run.smithi046.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-30T13:21:27.522 INFO:teuthology.orchestra.run.smithi046.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-30T13:21:27.549 INFO:teuthology.orchestra.run.smithi046.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-30T13:21:27.550 INFO:teuthology.orchestra.run.smithi046.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-30T13:21:27.558 INFO:teuthology.orchestra.run.smithi046.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-30T13:21:27.723 INFO:teuthology.orchestra.run.smithi046.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-30T13:21:27.754 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-30T13:21:27.776 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-30T13:21:27.820 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T13:21:28.073 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libjson-perl. 2023-10-30T13:21:28.086 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-30T13:21:28.123 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-30T13:21:28.418 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-30T13:21:28.440 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-30T13:21:28.466 INFO:teuthology.orchestra.run.smithi046.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-30T13:21:28.466 INFO:teuthology.orchestra.run.smithi046.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-30T13:21:28.476 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T13:21:28.493 INFO:teuthology.orchestra.run.smithi046.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-30T13:21:28.570 INFO:teuthology.orchestra.run.smithi046.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-30T13:21:28.579 INFO:teuthology.orchestra.run.smithi046.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-30T13:21:28.579 INFO:teuthology.orchestra.run.smithi046.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-30T13:21:28.580 INFO:teuthology.orchestra.run.smithi046.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-30T13:21:28.581 INFO:teuthology.orchestra.run.smithi046.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-30T13:21:28.609 INFO:teuthology.orchestra.run.smithi046.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-30T13:21:28.612 INFO:teuthology.orchestra.run.smithi046.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-30T13:21:29.238 INFO:teuthology.orchestra.run.smithi046.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-30T13:21:29.320 INFO:teuthology.orchestra.run.smithi046.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-30T13:21:29.321 INFO:teuthology.orchestra.run.smithi046.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-30T13:21:29.321 INFO:teuthology.orchestra.run.smithi046.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-30T13:21:29.410 INFO:teuthology.orchestra.run.smithi046.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-30T13:21:29.415 INFO:teuthology.orchestra.run.smithi046.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-30T13:21:29.879 INFO:teuthology.orchestra.run.smithi046.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-30T13:21:29.880 INFO:teuthology.orchestra.run.smithi046.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-30T13:21:30.195 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-30T13:21:30.217 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T13:21:30.262 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:21:30.295 INFO:teuthology.orchestra.run.smithi046.stdout:Preconfiguring packages ... 2023-10-30T13:21:30.435 INFO:teuthology.orchestra.run.smithi046.stdout:Fetched 54.6 MB in 5s (11.5 MB/s) 2023-10-30T13:21:30.539 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-30T13:21:30.561 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T13:21:30.585 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package flex. 2023-10-30T13:21:30.606 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:21:30.660 INFO:teuthology.orchestra.run.smithi046.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139851 files and directories currently installed.) 2023-10-30T13:21:30.664 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-30T13:21:30.708 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-30T13:21:30.943 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-30T13:21:30.964 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T13:21:31.009 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:21:31.220 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T13:21:31.328 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-30T13:21:31.350 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-30T13:21:31.364 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-30T13:21:31.395 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T13:21:31.731 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-30T13:21:31.753 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T13:21:31.762 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:21:31.797 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T13:21:32.176 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-30T13:21:32.197 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:21:32.242 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:21:32.314 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package bison. 2023-10-30T13:21:32.388 INFO:teuthology.orchestra.run.smithi046.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139924 files and directories currently installed.) 2023-10-30T13:21:32.391 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-30T13:21:32.433 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T13:21:32.586 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-30T13:21:32.608 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:21:32.644 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:21:32.875 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-30T13:21:32.890 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-30T13:21:32.934 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T13:21:32.989 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-30T13:21:33.001 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:21:33.039 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:21:33.237 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package dump. 2023-10-30T13:21:33.252 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-30T13:21:33.291 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-30T13:21:33.295 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking dump (0.4b47-3) ... 2023-10-30T13:21:33.313 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T13:21:33.366 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:21:33.591 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package indent. 2023-10-30T13:21:33.606 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:21:33.640 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T13:21:33.845 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-30T13:21:33.867 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-30T13:21:33.867 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-30T13:21:33.883 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-30T13:21:33.903 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T13:21:33.925 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T13:21:34.231 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package modemmanager. 2023-10-30T13:21:34.252 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T13:21:34.262 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-30T13:21:34.278 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-30T13:21:34.297 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:21:34.321 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T13:21:34.590 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-30T13:21:34.606 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-30T13:21:34.640 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T13:21:34.929 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package network-manager. 2023-10-30T13:21:34.950 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T13:21:34.993 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-30T13:21:35.003 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-30T13:21:35.010 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-30T13:21:35.052 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T13:21:35.315 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-30T13:21:35.331 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-30T13:21:35.373 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T13:21:35.618 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libjson-perl. 2023-10-30T13:21:35.633 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-30T13:21:35.661 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ppp. 2023-10-30T13:21:35.667 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-30T13:21:35.683 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-30T13:21:35.945 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-30T13:21:35.961 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-30T13:21:35.975 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T13:21:36.003 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T13:21:36.291 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package pptp-linux. 2023-10-30T13:21:36.313 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-30T13:21:36.349 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-30T13:21:36.593 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-30T13:21:36.616 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-30T13:21:36.660 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-30T13:21:36.963 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-30T13:21:36.985 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-30T13:21:37.038 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking postgresql-client-common (238) ... 2023-10-30T13:21:37.364 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-30T13:21:37.375 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-30T13:21:37.380 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T13:21:37.397 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:21:37.422 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:21:37.441 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:21:37.675 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-30T13:21:37.691 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T13:21:37.733 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:21:38.052 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-30T13:21:38.063 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package postgresql-common. 2023-10-30T13:21:38.068 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T13:21:38.085 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-30T13:21:38.138 INFO:teuthology.orchestra.run.smithi154.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-30T13:21:38.143 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:21:38.160 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking postgresql-common (238) ... 2023-10-30T13:21:38.430 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-30T13:21:38.446 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-30T13:21:38.504 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T13:21:38.629 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package postgresql-14. 2023-10-30T13:21:38.650 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:21:38.698 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:21:38.832 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-30T13:21:38.848 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T13:21:38.890 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T13:21:39.243 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-30T13:21:39.259 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:21:39.301 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:21:39.612 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-30T13:21:39.628 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:21:39.662 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:21:39.990 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-30T13:21:40.006 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:21:40.048 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:21:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-30T13:21:40.350 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T13:21:40.400 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:21:40.723 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package postgresql. 2023-10-30T13:21:40.736 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-30T13:21:40.773 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking postgresql (14+238) ... 2023-10-30T13:21:40.871 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-30T13:21:40.886 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-30T13:21:40.929 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T13:21:41.000 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package postgresql-client. 2023-10-30T13:21:41.013 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-30T13:21:41.050 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking postgresql-client (14+238) ... 2023-10-30T13:21:41.282 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package modemmanager. 2023-10-30T13:21:41.294 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-30T13:21:41.298 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T13:21:41.307 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-30T13:21:41.344 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-30T13:21:41.365 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:21:41.680 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package tcl8.6. 2023-10-30T13:21:41.702 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-30T13:21:41.746 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T13:21:41.937 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package network-manager. 2023-10-30T13:21:41.954 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T13:21:41.996 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-30T13:21:42.058 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package tcl. 2023-10-30T13:21:42.080 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-30T13:21:42.129 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-30T13:21:42.461 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package wpasupplicant. 2023-10-30T13:21:42.483 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-30T13:21:42.527 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T13:21:42.669 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package ppp. 2023-10-30T13:21:42.686 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-30T13:21:42.890 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-30T13:21:42.902 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-30T13:21:42.939 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-30T13:21:43.051 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T13:21:43.313 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-30T13:21:43.335 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-30T13:21:43.374 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package pptp-linux. 2023-10-30T13:21:43.385 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T13:21:43.388 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-30T13:21:43.432 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-30T13:21:43.677 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up postgresql-client-common (238) ... 2023-10-30T13:21:43.693 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-30T13:21:43.708 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-30T13:21:43.751 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-30T13:21:43.924 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-30T13:21:44.054 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-30T13:21:44.069 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-30T13:21:44.075 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T13:21:44.104 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking postgresql-client-common (238) ... 2023-10-30T13:21:44.226 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:21:44.374 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-30T13:21:44.377 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T13:21:44.390 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:21:44.433 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:21:44.528 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T13:21:44.680 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:21:45.172 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package postgresql-common. 2023-10-30T13:21:45.188 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-30T13:21:45.209 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-30T13:21:45.258 INFO:teuthology.orchestra.run.smithi046.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-30T13:21:45.288 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking postgresql-common (238) ... 2023-10-30T13:21:45.318 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T13:21:45.755 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package postgresql-14. 2023-10-30T13:21:45.759 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T13:21:45.759 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T13:21:45.771 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:21:45.815 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:21:46.313 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T13:21:46.439 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:21:46.582 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-30T13:21:46.733 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T13:21:46.909 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T13:21:46.973 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package postgresql. 2023-10-30T13:21:46.989 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-30T13:21:47.032 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking postgresql (14+238) ... 2023-10-30T13:21:47.060 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T13:21:47.214 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T13:21:47.275 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package postgresql-client. 2023-10-30T13:21:47.291 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-30T13:21:47.334 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking postgresql-client (14+238) ... 2023-10-30T13:21:47.577 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-30T13:21:47.593 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-30T13:21:47.627 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-30T13:21:47.913 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package tcl8.6. 2023-10-30T13:21:47.930 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-30T13:21:47.971 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T13:21:48.042 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T13:21:48.201 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up dump (0.4b47-3) ... 2023-10-30T13:21:48.266 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package tcl. 2023-10-30T13:21:48.282 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-30T13:21:48.312 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-30T13:21:48.329 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-30T13:21:48.395 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:21:48.677 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package wpasupplicant. 2023-10-30T13:21:48.693 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-30T13:21:48.735 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T13:21:49.030 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T13:21:49.089 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-30T13:21:49.106 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-30T13:21:49.139 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-30T13:21:49.164 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-30T13:21:49.290 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-30T13:21:49.441 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T13:21:49.509 INFO:teuthology.orchestra.run.smithi046.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-30T13:21:49.525 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-30T13:21:49.548 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-30T13:21:49.576 INFO:teuthology.orchestra.run.smithi046.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T13:21:49.626 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T13:21:49.777 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T13:21:49.852 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up postgresql-client-common (238) ... 2023-10-30T13:21:49.987 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T13:21:50.127 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-30T13:21:50.138 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:21:50.272 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-30T13:21:50.441 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up postgresql-client (14+238) ... 2023-10-30T13:21:50.549 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T13:21:50.592 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-30T13:21:50.650 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:21:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T13:21:50.901 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T13:21:51.027 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:21:51.082 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:21:51.082 INFO:teuthology.orchestra.run.smithi154.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-30T13:21:51.082 INFO:teuthology.orchestra.run.smithi154.stdout:which means they are currently configured by ifupdown: 2023-10-30T13:21:51.082 INFO:teuthology.orchestra.run.smithi154.stdout:- enp3s0f1 2023-10-30T13:21:51.083 INFO:teuthology.orchestra.run.smithi154.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-30T13:21:51.083 INFO:teuthology.orchestra.run.smithi154.stdout:remove their configuration from /etc/network/interfaces. 2023-10-30T13:21:51.083 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:21:51.191 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-30T13:21:51.541 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-30T13:21:51.906 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-30T13:21:52.086 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-30T13:21:52.215 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T13:21:52.461 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:21:52.593 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T13:21:52.593 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T13:21:52.629 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T13:21:52.772 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-30T13:21:52.898 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up postgresql-common (238) ... 2023-10-30T13:21:53.318 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T13:21:53.435 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:21:53.561 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-30T13:21:53.687 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T13:21:53.813 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T13:21:53.831 INFO:teuthology.orchestra.run.smithi154.stdout:Adding user postgres to group ssl-cert 2023-10-30T13:21:53.989 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T13:21:54.051 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:21:54.051 INFO:teuthology.orchestra.run.smithi154.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-30T13:21:54.115 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T13:21:54.223 INFO:teuthology.orchestra.run.smithi154.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-30T13:21:54.224 INFO:teuthology.orchestra.run.smithi154.stdout:Removing obsolete dictionary files: 2023-10-30T13:21:54.779 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T13:21:54.799 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-30T13:21:54.904 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up dump (0.4b47-3) ... 2023-10-30T13:21:54.999 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-30T13:21:55.073 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:21:55.174 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T13:21:55.225 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:21:55.292 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-30T13:21:55.451 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-30T13:21:55.543 INFO:teuthology.orchestra.run.smithi154.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-30T13:21:55.561 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-30T13:21:55.617 INFO:teuthology.orchestra.run.smithi154.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-30T13:21:55.617 INFO:teuthology.orchestra.run.smithi154.stdout:This user must also own the server process. 2023-10-30T13:21:55.617 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:21:55.617 INFO:teuthology.orchestra.run.smithi154.stdout:The database cluster will be initialized with locale "en_US". 2023-10-30T13:21:55.618 INFO:teuthology.orchestra.run.smithi154.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-30T13:21:55.618 INFO:teuthology.orchestra.run.smithi154.stdout:The default text search configuration will be set to "english". 2023-10-30T13:21:55.618 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:21:55.618 INFO:teuthology.orchestra.run.smithi154.stdout:Data page checksums are disabled. 2023-10-30T13:21:55.619 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:21:55.619 INFO:teuthology.orchestra.run.smithi154.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-30T13:21:55.619 INFO:teuthology.orchestra.run.smithi154.stdout:creating subdirectories ... ok 2023-10-30T13:21:55.619 INFO:teuthology.orchestra.run.smithi154.stdout:selecting dynamic shared memory implementation ... posix 2023-10-30T13:21:55.630 INFO:teuthology.orchestra.run.smithi154.stdout:selecting default max_connections ... 100 2023-10-30T13:21:55.652 INFO:teuthology.orchestra.run.smithi154.stdout:selecting default shared_buffers ... 128MB 2023-10-30T13:21:55.663 INFO:teuthology.orchestra.run.smithi154.stdout:selecting default time zone ... Etc/UTC 2023-10-30T13:21:55.663 INFO:teuthology.orchestra.run.smithi154.stdout:creating configuration files ... ok 2023-10-30T13:21:55.775 INFO:teuthology.orchestra.run.smithi154.stdout:running bootstrap script ... ok 2023-10-30T13:21:55.894 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T13:21:55.984 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-30T13:21:56.045 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T13:21:56.171 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T13:21:56.347 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T13:21:56.465 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:21:56.476 INFO:teuthology.orchestra.run.smithi154.stdout:performing post-bootstrap initialization ... ok 2023-10-30T13:21:56.616 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-30T13:21:56.784 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up postgresql-client (14+238) ... 2023-10-30T13:21:56.910 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-30T13:21:57.221 INFO:teuthology.orchestra.run.smithi154.stdout:syncing data to disk ... ok 2023-10-30T13:21:57.505 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:21:57.506 INFO:teuthology.orchestra.run.smithi046.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-30T13:21:57.506 INFO:teuthology.orchestra.run.smithi046.stdout:which means they are currently configured by ifupdown: 2023-10-30T13:21:57.506 INFO:teuthology.orchestra.run.smithi046.stdout:- ens1f0 2023-10-30T13:21:57.506 INFO:teuthology.orchestra.run.smithi046.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-30T13:21:57.507 INFO:teuthology.orchestra.run.smithi046.stdout:remove their configuration from /etc/network/interfaces. 2023-10-30T13:21:57.507 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:21:57.636 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-30T13:21:57.645 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-30T13:21:58.078 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-30T13:21:58.533 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-30T13:21:59.246 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:21:59.397 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T13:21:59.524 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-30T13:21:59.650 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up postgresql-common (238) ... 2023-10-30T13:22:00.566 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:22:00.603 INFO:teuthology.orchestra.run.smithi046.stdout:Adding user postgres to group ssl-cert 2023-10-30T13:22:00.725 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-30T13:22:00.874 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:00.874 INFO:teuthology.orchestra.run.smithi046.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-30T13:22:00.876 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up postgresql (14+238) ... 2023-10-30T13:22:01.010 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:22:01.136 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:22:01.140 INFO:teuthology.orchestra.run.smithi046.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-30T13:22:01.141 INFO:teuthology.orchestra.run.smithi046.stdout:Removing obsolete dictionary files: 2023-10-30T13:22:01.389 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T13:22:01.389 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T13:22:01.900 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-30T13:22:02.123 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-30T13:22:02.351 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:22:02.433 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T13:22:02.584 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T13:22:02.774 INFO:teuthology.orchestra.run.smithi046.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-30T13:22:02.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-30T13:22:02.853 INFO:teuthology.orchestra.run.smithi046.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-30T13:22:02.854 INFO:teuthology.orchestra.run.smithi046.stdout:This user must also own the server process. 2023-10-30T13:22:02.854 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:02.854 INFO:teuthology.orchestra.run.smithi046.stdout:The database cluster will be initialized with locale "en_US". 2023-10-30T13:22:02.854 INFO:teuthology.orchestra.run.smithi046.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-30T13:22:02.854 INFO:teuthology.orchestra.run.smithi046.stdout:The default text search configuration will be set to "english". 2023-10-30T13:22:02.855 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:02.855 INFO:teuthology.orchestra.run.smithi046.stdout:Data page checksums are disabled. 2023-10-30T13:22:02.855 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:02.855 INFO:teuthology.orchestra.run.smithi046.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-30T13:22:02.856 INFO:teuthology.orchestra.run.smithi046.stdout:creating subdirectories ... ok 2023-10-30T13:22:02.856 INFO:teuthology.orchestra.run.smithi046.stdout:selecting dynamic shared memory implementation ... posix 2023-10-30T13:22:02.880 INFO:teuthology.orchestra.run.smithi046.stdout:selecting default max_connections ... 100 2023-10-30T13:22:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:selecting default shared_buffers ... 128MB 2023-10-30T13:22:02.932 INFO:teuthology.orchestra.run.smithi046.stdout:selecting default time zone ... Etc/UTC 2023-10-30T13:22:02.933 INFO:teuthology.orchestra.run.smithi046.stdout:creating configuration files ... ok 2023-10-30T13:22:03.110 INFO:teuthology.orchestra.run.smithi046.stdout:running bootstrap script ... ok 2023-10-30T13:22:03.736 INFO:teuthology.orchestra.run.smithi046.stdout:performing post-bootstrap initialization ... ok 2023-10-30T13:22:04.104 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-30T13:22:04.204 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-30T13:22:04.330 INFO:teuthology.orchestra.run.smithi046.stdout:syncing data to disk ... ok 2023-10-30T13:22:04.838 INFO:teuthology.orchestra.run.smithi154.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T13:22:04.841 DEBUG:teuthology.parallel:result is None 2023-10-30T13:22:04.882 INFO:teuthology.orchestra.run.smithi046.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-30T13:22:07.833 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:22:07.934 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-30T13:22:08.060 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up postgresql (14+238) ... 2023-10-30T13:22:08.186 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:22:08.312 INFO:teuthology.orchestra.run.smithi046.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:22:08.601 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T13:22:08.602 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T13:22:09.348 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-30T13:22:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T13:22:09.818 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T13:22:11.312 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-30T13:22:11.412 INFO:teuthology.orchestra.run.smithi046.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-30T13:22:12.154 INFO:teuthology.orchestra.run.smithi046.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T13:22:12.159 DEBUG:teuthology.parallel:result is None 2023-10-30T13:22:12.160 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:22:12.322 DEBUG:teuthology.orchestra.run.smithi046:> dpkg-query -W -f '${Version}' ceph 2023-10-30T13:22:12.346 INFO:teuthology.orchestra.run.smithi046.stdout:18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:22:12.346 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:22:12.347 INFO:teuthology.task.install:The correct ceph version 18.0.0-6835-g00c1c13c-1jammy is installed. 2023-10-30T13:22:12.347 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:22:12.566 DEBUG:teuthology.orchestra.run.smithi154:> dpkg-query -W -f '${Version}' ceph 2023-10-30T13:22:12.586 INFO:teuthology.orchestra.run.smithi154.stdout:18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:22:12.586 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:22:12.586 INFO:teuthology.task.install:The correct ceph version 18.0.0-6835-g00c1c13c-1jammy is installed. 2023-10-30T13:22:12.587 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-30T13:22:12.587 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:12.587 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T13:22:12.609 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:12.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T13:22:12.643 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-30T13:22:12.643 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:12.644 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T13:22:12.672 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T13:22:12.736 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:12.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T13:22:12.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T13:22:12.805 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-30T13:22:12.805 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:12.806 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T13:22:12.827 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T13:22:12.892 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:12.893 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T13:22:12.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T13:22:12.957 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-30T13:22:12.958 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:12.958 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T13:22:12.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T13:22:13.040 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:13.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T13:22:13.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T13:22:13.109 INFO:teuthology.run_tasks:Running task ceph... 2023-10-30T13:22:13.169 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-30T13:22:13.170 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /var/log/ceph 2023-10-30T13:22:13.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /var/log/ceph 2023-10-30T13:22:13.191 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-30T13:22:13.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T13:22:13.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T13:22:13.252 INFO:tasks.ceph:Creating extra log directories... 2023-10-30T13:22:13.252 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T13:22:13.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T13:22:13.323 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-30T13:22:13.323 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-30T13:22:13.324 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-30T13:22:13.324 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:13.324 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-30T13:22:13.369 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:13.462 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:13.463 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-30T13:22:13.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:13.548 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-30T13:22:13.548 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-30T13:22:13.548 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045', 'branch': 'wip-lusov-balance-after-stop', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': None, 'first_in_suite': False, 'job_id': '7441045', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-lusov-balance-after-stop'}, 'ceph': {'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-balance-after-stop', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 8397, 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'sleep_before_teardown': 0, 'subset': '464/1024', 'suite': 'fs', 'suite_branch': 'wip-lusov-balance-after-stop', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi046.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE8KRBqKi4J4ac52uwYhjG88eMhxcjlX+sg7m3/vyM3wFY/30AJq3vthxKneW/V1m+99tGjKFak5Zb3TtmS+yWc=', 'smithi154.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBQUkOw5PZhB5D+tQQfIqdQCcoWCwOAKnTkiMvcCnO+2w8k4FWZdHEkcrgVnAq8Uovev07ctOSFh0WtmGK+Dgo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'cluster': 'ceph'}}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-30_06:34:16', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-30T13:22:13.549 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T13:22:13.553 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T13:22:13.591 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T13:22:13.607 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T13:22:13.646 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:13.647 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T13:22:13.681 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T13:22:13.681 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2023-10-30T13:22:13.733 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T13:22:13.733 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:22:13.733 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 901 Links: 1 2023-10-30T13:22:13.734 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:22:13.734 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-10-30 13:15:49.741513442 +0000 2023-10-30T13:22:13.734 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-10-30 13:15:49.377516693 +0000 2023-10-30T13:22:13.734 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-10-30 13:15:49.377516693 +0000 2023-10-30T13:22:13.734 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2023-10-30 13:15:49.377516693 +0000 2023-10-30T13:22:13.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T13:22:13.795 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-10-30T13:22:13.796 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-10-30T13:22:13.796 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000259745 s, 2.0 MB/s 2023-10-30T13:22:13.797 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T13:22:13.851 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2023-10-30T13:22:13.905 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T13:22:13.906 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:22:13.906 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 920 Links: 1 2023-10-30T13:22:13.906 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:22:13.906 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-10-30 13:15:49.997511155 +0000 2023-10-30T13:22:13.906 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-10-30 13:15:49.993511191 +0000 2023-10-30T13:22:13.907 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-10-30 13:15:49.993511191 +0000 2023-10-30T13:22:13.907 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2023-10-30 13:15:49.993511191 +0000 2023-10-30T13:22:13.907 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T13:22:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-10-30T13:22:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-10-30T13:22:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000269808 s, 1.9 MB/s 2023-10-30T13:22:13.970 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T13:22:14.027 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2023-10-30T13:22:14.078 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T13:22:14.078 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:22:14.078 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 936 Links: 1 2023-10-30T13:22:14.078 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:22:14.078 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-10-30 13:15:51.097501330 +0000 2023-10-30T13:22:14.079 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-10-30 13:15:50.689504975 +0000 2023-10-30T13:22:14.079 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-10-30 13:15:50.689504975 +0000 2023-10-30T13:22:14.079 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2023-10-30 13:15:50.689504975 +0000 2023-10-30T13:22:14.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T13:22:14.144 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-10-30T13:22:14.144 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-10-30T13:22:14.144 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000263298 s, 1.9 MB/s 2023-10-30T13:22:14.146 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T13:22:14.199 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2023-10-30T13:22:14.253 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T13:22:14.254 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:22:14.254 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 952 Links: 1 2023-10-30T13:22:14.254 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:22:14.254 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-10-30 13:15:51.761495399 +0000 2023-10-30T13:22:14.254 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-10-30 13:15:51.369498901 +0000 2023-10-30T13:22:14.254 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-10-30 13:15:51.369498901 +0000 2023-10-30T13:22:14.255 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2023-10-30 13:15:51.369498901 +0000 2023-10-30T13:22:14.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T13:22:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-10-30T13:22:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-10-30T13:22:14.317 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000264774 s, 1.9 MB/s 2023-10-30T13:22:14.318 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T13:22:14.372 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:22:14.372 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:14.372 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T13:22:14.378 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T13:22:14.379 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2023-10-30T13:22:14.427 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T13:22:14.427 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:22:14.427 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 898 Links: 1 2023-10-30T13:22:14.427 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:22:14.428 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2023-10-30 13:15:22.214881699 +0000 2023-10-30T13:22:14.428 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2023-10-30 13:15:21.878884701 +0000 2023-10-30T13:22:14.428 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2023-10-30 13:15:21.878884701 +0000 2023-10-30T13:22:14.428 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2023-10-30T13:22:14.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T13:22:14.481 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2023-10-30T13:22:14.481 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2023-10-30T13:22:14.481 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000504466 s, 1.0 MB/s 2023-10-30T13:22:14.482 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T13:22:14.531 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2023-10-30T13:22:14.578 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T13:22:14.579 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:22:14.579 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 914 Links: 1 2023-10-30T13:22:14.579 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:22:14.579 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2023-10-30 13:15:22.826876233 +0000 2023-10-30T13:22:14.579 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2023-10-30 13:15:22.482879306 +0000 2023-10-30T13:22:14.580 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2023-10-30 13:15:22.482879306 +0000 2023-10-30T13:22:14.580 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2023-10-30T13:22:14.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T13:22:14.633 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2023-10-30T13:22:14.633 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2023-10-30T13:22:14.633 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000505576 s, 1.0 MB/s 2023-10-30T13:22:14.634 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T13:22:14.683 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2023-10-30T13:22:14.730 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T13:22:14.731 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:22:14.731 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 933 Links: 1 2023-10-30T13:22:14.731 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:22:14.731 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2023-10-30 13:15:23.418870946 +0000 2023-10-30T13:22:14.731 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2023-10-30 13:15:23.066874089 +0000 2023-10-30T13:22:14.732 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2023-10-30 13:15:23.066874089 +0000 2023-10-30T13:22:14.732 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2023-10-30T13:22:14.732 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T13:22:14.784 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2023-10-30T13:22:14.785 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2023-10-30T13:22:14.785 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000535062 s, 957 kB/s 2023-10-30T13:22:14.786 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T13:22:14.836 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2023-10-30T13:22:14.882 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T13:22:14.882 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:22:14.883 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 942 Links: 1 2023-10-30T13:22:14.883 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:22:14.883 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2023-10-30 13:15:24.022865551 +0000 2023-10-30T13:22:14.883 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2023-10-30 13:15:23.658868802 +0000 2023-10-30T13:22:14.883 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2023-10-30 13:15:23.658868802 +0000 2023-10-30T13:22:14.884 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2023-10-30T13:22:14.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T13:22:14.936 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2023-10-30T13:22:14.937 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2023-10-30T13:22:14.937 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000471032 s, 1.1 MB/s 2023-10-30T13:22:14.938 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T13:22:14.987 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:22:14.987 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T13:22:14.988 INFO:tasks.ceph:Generating config... 2023-10-30T13:22:14.989 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-30T13:22:14.989 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2023-10-30T13:22:14.989 INFO:tasks.ceph:[client] debug client = 20 2023-10-30T13:22:14.990 INFO:tasks.ceph:[client] debug ms = 1 2023-10-30T13:22:14.990 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-30T13:22:14.990 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-30T13:22:14.990 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-30T13:22:14.991 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-30T13:22:14.991 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-30T13:22:14.991 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-30T13:22:14.991 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-10-30T13:22:14.992 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-10-30T13:22:14.992 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-10-30T13:22:14.992 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-10-30T13:22:14.992 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-30T13:22:14.992 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-30T13:22:14.993 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2023-10-30T13:22:14.993 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-30T13:22:14.993 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-30T13:22:14.993 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2023-10-30T13:22:14.994 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-10-30T13:22:14.994 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-10-30T13:22:14.994 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-10-30T13:22:14.994 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-10-30T13:22:14.995 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-10-30T13:22:14.995 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-10-30T13:22:14.995 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-30T13:22:14.995 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-30T13:22:14.996 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-30T13:22:14.996 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-30T13:22:14.996 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-30T13:22:14.996 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-30T13:22:14.997 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-30T13:22:14.997 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-30T13:22:14.997 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-30T13:22:14.997 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-30T13:22:14.998 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-30T13:22:14.998 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-30T13:22:14.998 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-30T13:22:14.998 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-30T13:22:14.999 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-30T13:22:14.999 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-30T13:22:14.999 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-30T13:22:15.000 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-30T13:22:15.000 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-30T13:22:15.000 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-10-30T13:22:15.000 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-30T13:22:15.001 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-30T13:22:15.001 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-30T13:22:15.001 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-10-30T13:22:15.001 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-10-30T13:22:15.002 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-10-30T13:22:15.002 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-10-30T13:22:15.002 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-10-30T13:22:15.002 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-30T13:22:15.003 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-10-30T13:22:15.003 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-30T13:22:15.003 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-30T13:22:15.003 INFO:tasks.ceph:Setting up mon.a... 2023-10-30T13:22:15.004 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-30T13:22:15.052 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.keyring 2023-10-30T13:22:15.068 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-30T13:22:15.119 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T13:22:15.140 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.46'), ('mon.b', '172.21.15.154'), ('mon.c', '[v2:172.21.15.154:3301,v1:172.21.15.154:6790]')] 2023-10-30T13:22:15.141 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.46,172.21.15.154,[v2:172.21.15.154:3301,v1:172.21.15.154:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-30T13:22:15.142 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:15.142 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T13:22:15.189 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.46 --add b 172.21.15.154 --addv c '[v2:172.21.15.154:3301,v1:172.21.15.154:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:15.280 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:15.281 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: generated fsid 6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2 2023-10-30T13:22:15.281 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = pacific 2023-10-30T13:22:15.281 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 0 2023-10-30T13:22:15.281 INFO:teuthology.orchestra.run.smithi046.stdout:fsid 6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2 2023-10-30T13:22:15.281 INFO:teuthology.orchestra.run.smithi046.stdout:last_changed 2023-10-30T13:22:15.279699+0000 2023-10-30T13:22:15.282 INFO:teuthology.orchestra.run.smithi046.stdout:created 2023-10-30T13:22:15.279699+0000 2023-10-30T13:22:15.282 INFO:teuthology.orchestra.run.smithi046.stdout:min_mon_release 16 (pacific) 2023-10-30T13:22:15.282 INFO:teuthology.orchestra.run.smithi046.stdout:election_strategy: 1 2023-10-30T13:22:15.282 INFO:teuthology.orchestra.run.smithi046.stdout:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2023-10-30T13:22:15.282 INFO:teuthology.orchestra.run.smithi046.stdout:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2023-10-30T13:22:15.283 INFO:teuthology.orchestra.run.smithi046.stdout:2: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.c 2023-10-30T13:22:15.283 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-30T13:22:15.284 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T13:22:15.292 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2... 2023-10-30T13:22:15.293 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T13:22:15.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T13:22:15.370 INFO:teuthology.orchestra.run.smithi154.stdout:[global] 2023-10-30T13:22:15.371 INFO:teuthology.orchestra.run.smithi154.stdout: chdir = "" 2023-10-30T13:22:15.371 INFO:teuthology.orchestra.run.smithi154.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T13:22:15.371 INFO:teuthology.orchestra.run.smithi154.stdout: auth supported = cephx 2023-10-30T13:22:15.371 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.371 INFO:teuthology.orchestra.run.smithi154.stdout: filestore xattr use omap = true 2023-10-30T13:22:15.372 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.372 INFO:teuthology.orchestra.run.smithi154.stdout: mon clock drift allowed = 1.000 2023-10-30T13:22:15.372 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.372 INFO:teuthology.orchestra.run.smithi154.stdout: osd crush chooseleaf type = 0 2023-10-30T13:22:15.372 INFO:teuthology.orchestra.run.smithi154.stdout: auth debug = true 2023-10-30T13:22:15.373 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.373 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on old message = true 2023-10-30T13:22:15.373 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on bug = true 2023-10-30T13:22:15.373 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.373 INFO:teuthology.orchestra.run.smithi154.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T13:22:15.374 INFO:teuthology.orchestra.run.smithi154.stdout: mon pg warn max object skew = 0 2023-10-30T13:22:15.374 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.374 INFO:teuthology.orchestra.run.smithi154.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T13:22:15.374 INFO:teuthology.orchestra.run.smithi154.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T13:22:15.374 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.374 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default size = 2 2023-10-30T13:22:15.375 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.375 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow primary affinity = true 2023-10-30T13:22:15.375 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow pg remap = true 2023-10-30T13:22:15.375 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on legacy crush tunables = false 2023-10-30T13:22:15.375 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on crush straw calc version zero = false 2023-10-30T13:22:15.376 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on no sortbitwise = false 2023-10-30T13:22:15.376 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on osd down out interval zero = false 2023-10-30T13:22:15.376 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on too few osds = false 2023-10-30T13:22:15.376 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T13:22:15.376 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T13:22:15.377 INFO:teuthology.orchestra.run.smithi154.stdout: mon_allow_pool_size_one = true 2023-10-30T13:22:15.377 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.377 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T13:22:15.377 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.377 INFO:teuthology.orchestra.run.smithi154.stdout: osd default data pool replay window = 5 2023-10-30T13:22:15.377 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.378 INFO:teuthology.orchestra.run.smithi154.stdout: mon allow pool delete = true 2023-10-30T13:22:15.378 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.378 INFO:teuthology.orchestra.run.smithi154.stdout: mon cluster log file level = debug 2023-10-30T13:22:15.378 INFO:teuthology.orchestra.run.smithi154.stdout: debug asserts on shutdown = true 2023-10-30T13:22:15.378 INFO:teuthology.orchestra.run.smithi154.stdout: mon health detail to clog = false 2023-10-30T13:22:15.379 INFO:teuthology.orchestra.run.smithi154.stdout: mon host = "172.21.15.46,172.21.15.154,[v2:172.21.15.154:3301,v1:172.21.15.154:6790]" 2023-10-30T13:22:15.379 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on skipped message = False 2023-10-30T13:22:15.379 INFO:teuthology.orchestra.run.smithi154.stdout: fsid = 6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2 2023-10-30T13:22:15.379 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.379 INFO:teuthology.orchestra.run.smithi154.stdout:[osd] 2023-10-30T13:22:15.379 INFO:teuthology.orchestra.run.smithi154.stdout: osd journal size = 100 2023-10-30T13:22:15.380 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.380 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub load threshold = 5.0 2023-10-30T13:22:15.380 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub max interval = 600 2023-10-30T13:22:15.380 INFO:teuthology.orchestra.run.smithi154.stdout: osd mclock profile = high_recovery_ops 2023-10-30T13:22:15.380 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.381 INFO:teuthology.orchestra.run.smithi154.stdout: osd recover clone overlap = true 2023-10-30T13:22:15.381 INFO:teuthology.orchestra.run.smithi154.stdout: osd recovery max chunk = 1048576 2023-10-30T13:22:15.381 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.381 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug shutdown = true 2023-10-30T13:22:15.381 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug op order = true 2023-10-30T13:22:15.382 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify stray on activate = true 2023-10-30T13:22:15.382 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.382 INFO:teuthology.orchestra.run.smithi154.stdout: osd open classes on start = true 2023-10-30T13:22:15.382 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug pg log writeout = true 2023-10-30T13:22:15.382 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.382 INFO:teuthology.orchestra.run.smithi154.stdout: osd deep scrub update digest min age = 30 2023-10-30T13:22:15.383 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.383 INFO:teuthology.orchestra.run.smithi154.stdout: osd map max advance = 10 2023-10-30T13:22:15.383 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.383 INFO:teuthology.orchestra.run.smithi154.stdout: journal zero on create = true 2023-10-30T13:22:15.383 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.384 INFO:teuthology.orchestra.run.smithi154.stdout: filestore ondisk finisher threads = 3 2023-10-30T13:22:15.384 INFO:teuthology.orchestra.run.smithi154.stdout: filestore apply finisher threads = 3 2023-10-30T13:22:15.384 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.384 INFO:teuthology.orchestra.run.smithi154.stdout: bdev debug aio = true 2023-10-30T13:22:15.384 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug misdirected ops = true 2023-10-30T13:22:15.384 INFO:teuthology.orchestra.run.smithi154.stdout: bdev async discard = True 2023-10-30T13:22:15.385 INFO:teuthology.orchestra.run.smithi154.stdout: bdev enable discard = True 2023-10-30T13:22:15.385 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore allocator = bitmap 2023-10-30T13:22:15.385 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore block size = 96636764160 2023-10-30T13:22:15.385 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore fsck on mount = True 2023-10-30T13:22:15.385 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluefs = 1/20 2023-10-30T13:22:15.386 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluestore = 1/20 2023-10-30T13:22:15.386 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2023-10-30T13:22:15.386 INFO:teuthology.orchestra.run.smithi154.stdout: debug osd = 20 2023-10-30T13:22:15.386 INFO:teuthology.orchestra.run.smithi154.stdout: debug rocksdb = 4/10 2023-10-30T13:22:15.386 INFO:teuthology.orchestra.run.smithi154.stdout: mon client directed command retry = 5 2023-10-30T13:22:15.387 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T13:22:15.387 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd full ratio = 0.9 2023-10-30T13:22:15.387 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd nearfull ratio = 0.8 2023-10-30T13:22:15.387 INFO:teuthology.orchestra.run.smithi154.stdout: ms inject delay max = 1 2023-10-30T13:22:15.387 INFO:teuthology.orchestra.run.smithi154.stdout: ms inject delay probability = 0.005 2023-10-30T13:22:15.387 INFO:teuthology.orchestra.run.smithi154.stdout: ms inject delay type = client mds 2023-10-30T13:22:15.388 INFO:teuthology.orchestra.run.smithi154.stdout: ms inject socket failures = 2500 2023-10-30T13:22:15.388 INFO:teuthology.orchestra.run.smithi154.stdout: osd blocked scrub grace period = 3600 2023-10-30T13:22:15.388 INFO:teuthology.orchestra.run.smithi154.stdout: osd failsafe full ratio = 0.95 2023-10-30T13:22:15.388 INFO:teuthology.orchestra.run.smithi154.stdout: osd max markdown count = 1000 2023-10-30T13:22:15.388 INFO:teuthology.orchestra.run.smithi154.stdout: osd objectstore = bluestore 2023-10-30T13:22:15.389 INFO:teuthology.orchestra.run.smithi154.stdout: osd op complaint time = 180 2023-10-30T13:22:15.389 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.389 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr] 2023-10-30T13:22:15.389 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2023-10-30T13:22:15.389 INFO:teuthology.orchestra.run.smithi154.stdout: debug mgr = 20 2023-10-30T13:22:15.389 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2023-10-30T13:22:15.390 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2023-10-30T13:22:15.390 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2023-10-30T13:22:15.390 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2023-10-30T13:22:15.390 INFO:teuthology.orchestra.run.smithi154.stdout: mgr/telemetry/nag = false 2023-10-30T13:22:15.390 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.391 INFO:teuthology.orchestra.run.smithi154.stdout:[mon] 2023-10-30T13:22:15.391 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2023-10-30T13:22:15.391 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2023-10-30T13:22:15.391 INFO:teuthology.orchestra.run.smithi154.stdout: debug paxos = 20 2023-10-30T13:22:15.391 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2023-10-30T13:22:15.392 INFO:teuthology.orchestra.run.smithi154.stdout: mon data avail warn = 5 2023-10-30T13:22:15.392 INFO:teuthology.orchestra.run.smithi154.stdout: mon mgr mkfs grace = 240 2023-10-30T13:22:15.392 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2023-10-30T13:22:15.392 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd reporter subtree level = osd 2023-10-30T13:22:15.392 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd prime pg temp = true 2023-10-30T13:22:15.392 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2023-10-30T13:22:15.393 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.393 INFO:teuthology.orchestra.run.smithi154.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T13:22:15.393 INFO:teuthology.orchestra.run.smithi154.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T13:22:15.393 INFO:teuthology.orchestra.run.smithi154.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T13:22:15.393 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.394 INFO:teuthology.orchestra.run.smithi154.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T13:22:15.394 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T13:22:15.394 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T13:22:15.394 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.394 INFO:teuthology.orchestra.run.smithi154.stdout: # 1m isn't quite enough 2023-10-30T13:22:15.395 INFO:teuthology.orchestra.run.smithi154.stdout: mon_down_mkfs_grace = 2m 2023-10-30T13:22:15.395 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.395 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_filestore_osds = false 2023-10-30T13:22:15.395 INFO:teuthology.orchestra.run.smithi154.stdout: mon op complaint time = 120 2023-10-30T13:22:15.395 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-10-30T13:22:15.395 INFO:teuthology.orchestra.run.smithi154.stdout:[client] 2023-10-30T13:22:15.396 INFO:teuthology.orchestra.run.smithi154.stdout: rgw cache enabled = true 2023-10-30T13:22:15.396 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable ops log = true 2023-10-30T13:22:15.396 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable usage log = true 2023-10-30T13:22:15.396 INFO:teuthology.orchestra.run.smithi154.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T13:22:15.396 INFO:teuthology.orchestra.run.smithi154.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T13:22:15.397 INFO:teuthology.orchestra.run.smithi154.stdout: client mount timeout = 600 2023-10-30T13:22:15.397 INFO:teuthology.orchestra.run.smithi154.stdout: client_shutdown_timeout = 120 2023-10-30T13:22:15.397 INFO:teuthology.orchestra.run.smithi154.stdout: debug client = 20 2023-10-30T13:22:15.397 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2023-10-30T13:22:15.397 INFO:teuthology.orchestra.run.smithi154.stdout: rados mon op timeout = 900 2023-10-30T13:22:15.398 INFO:teuthology.orchestra.run.smithi154.stdout: rados osd op timeout = 900 2023-10-30T13:22:15.398 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.a] 2023-10-30T13:22:15.398 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.b] 2023-10-30T13:22:15.398 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.c] 2023-10-30T13:22:15.398 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.a] 2023-10-30T13:22:15.398 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.c] 2023-10-30T13:22:15.399 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.e] 2023-10-30T13:22:15.399 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.b] 2023-10-30T13:22:15.399 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.d] 2023-10-30T13:22:15.399 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.f] 2023-10-30T13:22:15.399 INFO:teuthology.orchestra.run.smithi154.stdout:[mds] 2023-10-30T13:22:15.400 INFO:teuthology.orchestra.run.smithi154.stdout: debug mds = 20 2023-10-30T13:22:15.400 INFO:teuthology.orchestra.run.smithi154.stdout: debug mds balancer = 20 2023-10-30T13:22:15.400 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2023-10-30T13:22:15.400 INFO:teuthology.orchestra.run.smithi154.stdout: mds bal fragment size max = 10000 2023-10-30T13:22:15.400 INFO:teuthology.orchestra.run.smithi154.stdout: mds bal merge size = 5 2023-10-30T13:22:15.400 INFO:teuthology.orchestra.run.smithi154.stdout: mds bal split bits = 3 2023-10-30T13:22:15.401 INFO:teuthology.orchestra.run.smithi154.stdout: mds bal split size = 100 2023-10-30T13:22:15.401 INFO:teuthology.orchestra.run.smithi154.stdout: mds debug frag = True 2023-10-30T13:22:15.401 INFO:teuthology.orchestra.run.smithi154.stdout: mds debug scatterstat = True 2023-10-30T13:22:15.401 INFO:teuthology.orchestra.run.smithi154.stdout: mds oft prefetch dirfrags = True 2023-10-30T13:22:15.401 INFO:teuthology.orchestra.run.smithi154.stdout: mds op complaint time = 180 2023-10-30T13:22:15.402 INFO:teuthology.orchestra.run.smithi154.stdout: mds verify scatter = True 2023-10-30T13:22:15.402 INFO:teuthology.orchestra.run.smithi154.stdout: mds_dir_prefetch = False 2023-10-30T13:22:15.402 INFO:teuthology.orchestra.run.smithi154.stdout: mds_sleep_rank_change = 5000000.0 2023-10-30T13:22:15.402 INFO:teuthology.orchestra.run.smithi154.stdout: mon client directed command retry = 5 2023-10-30T13:22:15.402 INFO:teuthology.orchestra.run.smithi154.stdout: ms inject delay max = 1 2023-10-30T13:22:15.402 INFO:teuthology.orchestra.run.smithi154.stdout: ms inject delay probability = 0.005 2023-10-30T13:22:15.403 INFO:teuthology.orchestra.run.smithi154.stdout: ms inject delay type = client mds osd 2023-10-30T13:22:15.403 INFO:teuthology.orchestra.run.smithi154.stdout: ms inject socket failures = 2500 2023-10-30T13:22:15.403 INFO:teuthology.orchestra.run.smithi154.stdout: osd op complaint time = 180 2023-10-30T13:22:15.403 INFO:teuthology.orchestra.run.smithi154.stdout: rados mon op timeout = 900 2023-10-30T13:22:15.403 INFO:teuthology.orchestra.run.smithi154.stdout: rados osd op timeout = 900 2023-10-30T13:22:15.405 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2023-10-30T13:22:15.405 INFO:teuthology.orchestra.run.smithi046.stdout: chdir = "" 2023-10-30T13:22:15.405 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T13:22:15.406 INFO:teuthology.orchestra.run.smithi046.stdout: auth supported = cephx 2023-10-30T13:22:15.406 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.406 INFO:teuthology.orchestra.run.smithi046.stdout: filestore xattr use omap = true 2023-10-30T13:22:15.406 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.406 INFO:teuthology.orchestra.run.smithi046.stdout: mon clock drift allowed = 1.000 2023-10-30T13:22:15.407 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.407 INFO:teuthology.orchestra.run.smithi046.stdout: osd crush chooseleaf type = 0 2023-10-30T13:22:15.407 INFO:teuthology.orchestra.run.smithi046.stdout: auth debug = true 2023-10-30T13:22:15.407 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.407 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on old message = true 2023-10-30T13:22:15.407 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on bug = true 2023-10-30T13:22:15.408 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.408 INFO:teuthology.orchestra.run.smithi046.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T13:22:15.408 INFO:teuthology.orchestra.run.smithi046.stdout: mon pg warn max object skew = 0 2023-10-30T13:22:15.408 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.408 INFO:teuthology.orchestra.run.smithi046.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T13:22:15.409 INFO:teuthology.orchestra.run.smithi046.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T13:22:15.409 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.409 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default size = 2 2023-10-30T13:22:15.409 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.409 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow primary affinity = true 2023-10-30T13:22:15.410 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow pg remap = true 2023-10-30T13:22:15.410 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on legacy crush tunables = false 2023-10-30T13:22:15.410 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on crush straw calc version zero = false 2023-10-30T13:22:15.410 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on no sortbitwise = false 2023-10-30T13:22:15.410 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on osd down out interval zero = false 2023-10-30T13:22:15.411 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on too few osds = false 2023-10-30T13:22:15.411 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T13:22:15.411 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T13:22:15.411 INFO:teuthology.orchestra.run.smithi046.stdout: mon_allow_pool_size_one = true 2023-10-30T13:22:15.411 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.411 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T13:22:15.412 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.412 INFO:teuthology.orchestra.run.smithi046.stdout: osd default data pool replay window = 5 2023-10-30T13:22:15.412 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.412 INFO:teuthology.orchestra.run.smithi046.stdout: mon allow pool delete = true 2023-10-30T13:22:15.412 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.413 INFO:teuthology.orchestra.run.smithi046.stdout: mon cluster log file level = debug 2023-10-30T13:22:15.413 INFO:teuthology.orchestra.run.smithi046.stdout: debug asserts on shutdown = true 2023-10-30T13:22:15.413 INFO:teuthology.orchestra.run.smithi046.stdout: mon health detail to clog = false 2023-10-30T13:22:15.413 INFO:teuthology.orchestra.run.smithi046.stdout: mon host = "172.21.15.46,172.21.15.154,[v2:172.21.15.154:3301,v1:172.21.15.154:6790]" 2023-10-30T13:22:15.413 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on skipped message = False 2023-10-30T13:22:15.413 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = 6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2 2023-10-30T13:22:15.414 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.414 INFO:teuthology.orchestra.run.smithi046.stdout:[osd] 2023-10-30T13:22:15.414 INFO:teuthology.orchestra.run.smithi046.stdout: osd journal size = 100 2023-10-30T13:22:15.414 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.414 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub load threshold = 5.0 2023-10-30T13:22:15.415 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub max interval = 600 2023-10-30T13:22:15.415 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock profile = high_recovery_ops 2023-10-30T13:22:15.415 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.415 INFO:teuthology.orchestra.run.smithi046.stdout: osd recover clone overlap = true 2023-10-30T13:22:15.415 INFO:teuthology.orchestra.run.smithi046.stdout: osd recovery max chunk = 1048576 2023-10-30T13:22:15.416 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.416 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug shutdown = true 2023-10-30T13:22:15.416 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug op order = true 2023-10-30T13:22:15.416 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify stray on activate = true 2023-10-30T13:22:15.416 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.417 INFO:teuthology.orchestra.run.smithi046.stdout: osd open classes on start = true 2023-10-30T13:22:15.417 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug pg log writeout = true 2023-10-30T13:22:15.417 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.417 INFO:teuthology.orchestra.run.smithi046.stdout: osd deep scrub update digest min age = 30 2023-10-30T13:22:15.417 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.417 INFO:teuthology.orchestra.run.smithi046.stdout: osd map max advance = 10 2023-10-30T13:22:15.418 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.418 INFO:teuthology.orchestra.run.smithi046.stdout: journal zero on create = true 2023-10-30T13:22:15.418 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.418 INFO:teuthology.orchestra.run.smithi046.stdout: filestore ondisk finisher threads = 3 2023-10-30T13:22:15.418 INFO:teuthology.orchestra.run.smithi046.stdout: filestore apply finisher threads = 3 2023-10-30T13:22:15.419 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.419 INFO:teuthology.orchestra.run.smithi046.stdout: bdev debug aio = true 2023-10-30T13:22:15.419 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug misdirected ops = true 2023-10-30T13:22:15.419 INFO:teuthology.orchestra.run.smithi046.stdout: bdev async discard = True 2023-10-30T13:22:15.419 INFO:teuthology.orchestra.run.smithi046.stdout: bdev enable discard = True 2023-10-30T13:22:15.420 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore allocator = bitmap 2023-10-30T13:22:15.420 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore block size = 96636764160 2023-10-30T13:22:15.420 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore fsck on mount = True 2023-10-30T13:22:15.420 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluefs = 1/20 2023-10-30T13:22:15.420 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluestore = 1/20 2023-10-30T13:22:15.421 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-10-30T13:22:15.421 INFO:teuthology.orchestra.run.smithi046.stdout: debug osd = 20 2023-10-30T13:22:15.421 INFO:teuthology.orchestra.run.smithi046.stdout: debug rocksdb = 4/10 2023-10-30T13:22:15.421 INFO:teuthology.orchestra.run.smithi046.stdout: mon client directed command retry = 5 2023-10-30T13:22:15.421 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T13:22:15.421 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd full ratio = 0.9 2023-10-30T13:22:15.422 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd nearfull ratio = 0.8 2023-10-30T13:22:15.422 INFO:teuthology.orchestra.run.smithi046.stdout: ms inject delay max = 1 2023-10-30T13:22:15.422 INFO:teuthology.orchestra.run.smithi046.stdout: ms inject delay probability = 0.005 2023-10-30T13:22:15.422 INFO:teuthology.orchestra.run.smithi046.stdout: ms inject delay type = client mds 2023-10-30T13:22:15.422 INFO:teuthology.orchestra.run.smithi046.stdout: ms inject socket failures = 2500 2023-10-30T13:22:15.423 INFO:teuthology.orchestra.run.smithi046.stdout: osd blocked scrub grace period = 3600 2023-10-30T13:22:15.423 INFO:teuthology.orchestra.run.smithi046.stdout: osd failsafe full ratio = 0.95 2023-10-30T13:22:15.423 INFO:teuthology.orchestra.run.smithi046.stdout: osd max markdown count = 1000 2023-10-30T13:22:15.423 INFO:teuthology.orchestra.run.smithi046.stdout: osd objectstore = bluestore 2023-10-30T13:22:15.423 INFO:teuthology.orchestra.run.smithi046.stdout: osd op complaint time = 180 2023-10-30T13:22:15.424 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.424 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr] 2023-10-30T13:22:15.424 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-10-30T13:22:15.424 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgr = 20 2023-10-30T13:22:15.424 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2023-10-30T13:22:15.424 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2023-10-30T13:22:15.425 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2023-10-30T13:22:15.425 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2023-10-30T13:22:15.425 INFO:teuthology.orchestra.run.smithi046.stdout: mgr/telemetry/nag = false 2023-10-30T13:22:15.425 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.425 INFO:teuthology.orchestra.run.smithi046.stdout:[mon] 2023-10-30T13:22:15.426 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-10-30T13:22:15.426 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2023-10-30T13:22:15.426 INFO:teuthology.orchestra.run.smithi046.stdout: debug paxos = 20 2023-10-30T13:22:15.426 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2023-10-30T13:22:15.426 INFO:teuthology.orchestra.run.smithi046.stdout: mon data avail warn = 5 2023-10-30T13:22:15.426 INFO:teuthology.orchestra.run.smithi046.stdout: mon mgr mkfs grace = 240 2023-10-30T13:22:15.427 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2023-10-30T13:22:15.427 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd reporter subtree level = osd 2023-10-30T13:22:15.427 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd prime pg temp = true 2023-10-30T13:22:15.427 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2023-10-30T13:22:15.427 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.428 INFO:teuthology.orchestra.run.smithi046.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T13:22:15.428 INFO:teuthology.orchestra.run.smithi046.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T13:22:15.428 INFO:teuthology.orchestra.run.smithi046.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T13:22:15.428 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.428 INFO:teuthology.orchestra.run.smithi046.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T13:22:15.428 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T13:22:15.429 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T13:22:15.429 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.429 INFO:teuthology.orchestra.run.smithi046.stdout: # 1m isn't quite enough 2023-10-30T13:22:15.429 INFO:teuthology.orchestra.run.smithi046.stdout: mon_down_mkfs_grace = 2m 2023-10-30T13:22:15.429 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.430 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_filestore_osds = false 2023-10-30T13:22:15.430 INFO:teuthology.orchestra.run.smithi046.stdout: mon op complaint time = 120 2023-10-30T13:22:15.430 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:15.430 INFO:teuthology.orchestra.run.smithi046.stdout:[client] 2023-10-30T13:22:15.430 INFO:teuthology.orchestra.run.smithi046.stdout: rgw cache enabled = true 2023-10-30T13:22:15.430 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable ops log = true 2023-10-30T13:22:15.431 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable usage log = true 2023-10-30T13:22:15.431 INFO:teuthology.orchestra.run.smithi046.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T13:22:15.431 INFO:teuthology.orchestra.run.smithi046.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T13:22:15.431 INFO:teuthology.orchestra.run.smithi046.stdout: client mount timeout = 600 2023-10-30T13:22:15.431 INFO:teuthology.orchestra.run.smithi046.stdout: client_shutdown_timeout = 120 2023-10-30T13:22:15.432 INFO:teuthology.orchestra.run.smithi046.stdout: debug client = 20 2023-10-30T13:22:15.432 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-10-30T13:22:15.432 INFO:teuthology.orchestra.run.smithi046.stdout: rados mon op timeout = 900 2023-10-30T13:22:15.432 INFO:teuthology.orchestra.run.smithi046.stdout: rados osd op timeout = 900 2023-10-30T13:22:15.432 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.a] 2023-10-30T13:22:15.432 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.b] 2023-10-30T13:22:15.433 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.c] 2023-10-30T13:22:15.433 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a] 2023-10-30T13:22:15.433 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c] 2023-10-30T13:22:15.433 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.e] 2023-10-30T13:22:15.433 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b] 2023-10-30T13:22:15.434 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d] 2023-10-30T13:22:15.434 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.f] 2023-10-30T13:22:15.434 INFO:teuthology.orchestra.run.smithi046.stdout:[mds] 2023-10-30T13:22:15.434 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds = 20 2023-10-30T13:22:15.434 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds balancer = 20 2023-10-30T13:22:15.434 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-10-30T13:22:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: mds bal fragment size max = 10000 2023-10-30T13:22:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: mds bal merge size = 5 2023-10-30T13:22:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: mds bal split bits = 3 2023-10-30T13:22:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: mds bal split size = 100 2023-10-30T13:22:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: mds debug frag = True 2023-10-30T13:22:15.436 INFO:teuthology.orchestra.run.smithi046.stdout: mds debug scatterstat = True 2023-10-30T13:22:15.436 INFO:teuthology.orchestra.run.smithi046.stdout: mds oft prefetch dirfrags = True 2023-10-30T13:22:15.436 INFO:teuthology.orchestra.run.smithi046.stdout: mds op complaint time = 180 2023-10-30T13:22:15.436 INFO:teuthology.orchestra.run.smithi046.stdout: mds verify scatter = True 2023-10-30T13:22:15.436 INFO:teuthology.orchestra.run.smithi046.stdout: mds_dir_prefetch = False 2023-10-30T13:22:15.436 INFO:teuthology.orchestra.run.smithi046.stdout: mds_sleep_rank_change = 5000000.0 2023-10-30T13:22:15.437 INFO:teuthology.orchestra.run.smithi046.stdout: mon client directed command retry = 5 2023-10-30T13:22:15.437 INFO:teuthology.orchestra.run.smithi046.stdout: ms inject delay max = 1 2023-10-30T13:22:15.437 INFO:teuthology.orchestra.run.smithi046.stdout: ms inject delay probability = 0.005 2023-10-30T13:22:15.437 INFO:teuthology.orchestra.run.smithi046.stdout: ms inject delay type = client mds osd 2023-10-30T13:22:15.437 INFO:teuthology.orchestra.run.smithi046.stdout: ms inject socket failures = 2500 2023-10-30T13:22:15.438 INFO:teuthology.orchestra.run.smithi046.stdout: osd op complaint time = 180 2023-10-30T13:22:15.438 INFO:teuthology.orchestra.run.smithi046.stdout: rados mon op timeout = 900 2023-10-30T13:22:15.438 INFO:teuthology.orchestra.run.smithi046.stdout: rados osd op timeout = 900 2023-10-30T13:22:15.438 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-30T13:22:15.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-30T13:22:15.507 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-30T13:22:15.507 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:15.507 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-30T13:22:15.517 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:15.517 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-30T13:22:15.572 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi046.front.sepia.ceph.com 2023-10-30T13:22:15.572 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:15.573 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T13:22:15.573 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T13:22:15.645 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:15.645 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:15.700 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi154.front.sepia.ceph.com 2023-10-30T13:22:15.700 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:15.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T13:22:15.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T13:22:15.724 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:15.724 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:15.770 INFO:tasks.ceph:Setting up mon nodes... 2023-10-30T13:22:15.770 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-30T13:22:15.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T13:22:15.828 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T13:22:15.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T13:22:15.875 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T13:22:15.888 INFO:tasks.ceph:Setting up mds nodes... 2023-10-30T13:22:15.889 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-30T13:22:15.947 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-30T13:22:15.951 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-30T13:22:15.972 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-30T13:22:16.077 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-30T13:22:16.081 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-30T13:22:16.102 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-10-30T13:22:16.201 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-10-30T13:22:16.205 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-10-30T13:22:16.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-30T13:22:16.270 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-30T13:22:16.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-30T13:22:16.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-30T13:22:16.369 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-30T13:22:16.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-30T13:22:16.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-10-30T13:22:16.502 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-10-30T13:22:16.505 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-10-30T13:22:16.561 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-30T13:22:16.562 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-30T13:22:16.606 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-30T13:22:16.624 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-30T13:22:16.624 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T13:22:16.625 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-30T13:22:16.684 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:22:16.684 INFO:tasks.ceph:role: osd.0 2023-10-30T13:22:16.685 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com 2023-10-30T13:22:16.685 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T13:22:16.752 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:22:16.753 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:22:16.753 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:22:16.753 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:22:16.753 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:22:16.754 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:22:16.754 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:22:16.754 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:22:16.754 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:22:16.754 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:22:16.780 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2023-10-30T13:22:16.820 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2023-10-30T13:22:16.820 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-30T13:22:16.886 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-30T13:22:16.945 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:22:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:22:16.946 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-30T13:22:17.008 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:22:17.008 INFO:tasks.ceph:role: osd.1 2023-10-30T13:22:17.009 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com 2023-10-30T13:22:17.009 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T13:22:17.076 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:22:17.076 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:22:17.077 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:22:17.077 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:22:17.077 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:22:17.077 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:22:17.077 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:22:17.078 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:22:17.078 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:22:17.078 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:22:17.103 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2023-10-30T13:22:17.106 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2023-10-30T13:22:17.107 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-30T13:22:17.177 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-30T13:22:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:22:17.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:22:17.238 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-30T13:22:17.300 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:22:17.300 INFO:tasks.ceph:role: osd.2 2023-10-30T13:22:17.300 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com 2023-10-30T13:22:17.300 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T13:22:17.364 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:22:17.364 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:22:17.364 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:22:17.364 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:22:17.365 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:22:17.365 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:22:17.365 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:22:17.365 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:22:17.365 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:22:17.366 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:22:17.390 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2023-10-30T13:22:17.394 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2023-10-30T13:22:17.394 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-30T13:22:17.461 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-30T13:22:17.525 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:22:17.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:22:17.526 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-30T13:22:17.592 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:22:17.593 INFO:tasks.ceph:role: osd.3 2023-10-30T13:22:17.593 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi046.front.sepia.ceph.com 2023-10-30T13:22:17.593 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T13:22:17.656 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:22:17.656 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:22:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:22:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:22:17.657 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:22:17.657 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:22:17.657 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:22:17.658 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:22:17.658 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:22:17.658 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:22:17.683 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2023-10-30T13:22:17.686 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2023-10-30T13:22:17.686 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-30T13:22:17.757 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-30T13:22:17.816 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:22:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:22:17.817 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:17.949+0000 7f71137da640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-30T13:22:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:17.949+0000 7f71137da640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-30T13:22:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:17.949+0000 7f71137da640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T13:22:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:17.949+0000 7f71137da640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T13:22:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:17.949+0000 7f71137da640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T13:22:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:17.949+0000 7f71137da640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-30T13:22:20.945 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-30T13:22:20.967 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:21.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:21.061+0000 7f074d765640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-30T13:22:21.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:21.061+0000 7f074d765640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-30T13:22:21.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:21.065+0000 7f074d765640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T13:22:21.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:21.065+0000 7f074d765640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T13:22:21.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:21.065+0000 7f074d765640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T13:22:21.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:21.065+0000 7f074d765640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-30T13:22:24.048 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-30T13:22:24.070 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:24.165+0000 7ff20f856640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-30T13:22:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:24.165+0000 7ff20f856640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-30T13:22:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:24.165+0000 7ff20f856640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T13:22:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:24.165+0000 7ff20f856640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T13:22:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:24.165+0000 7ff20f856640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T13:22:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:24.169+0000 7ff20f856640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-30T13:22:27.157 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-30T13:22:27.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:27.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:27.277+0000 7f21304a7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-30T13:22:27.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:27.277+0000 7f21304a7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-30T13:22:27.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:27.277+0000 7f21304a7640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T13:22:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:27.277+0000 7f21304a7640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T13:22:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:27.277+0000 7f21304a7640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T13:22:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:27.277+0000 7f21304a7640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-30T13:22:30.335 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-30T13:22:30.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-30T13:22:30.371 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:22:30.372 INFO:tasks.ceph:role: osd.4 2023-10-30T13:22:30.372 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com 2023-10-30T13:22:30.372 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T13:22:30.433 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:22:30.433 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:22:30.433 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:22:30.434 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:22:30.434 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:22:30.434 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:22:30.434 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:22:30.434 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:22:30.435 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:22:30.435 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:22:30.917 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2023-10-30T13:22:30.919 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2023-10-30T13:22:30.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-30T13:22:31.063 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-30T13:22:31.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:22:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:22:31.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-30T13:22:31.129 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:22:31.129 INFO:tasks.ceph:role: osd.5 2023-10-30T13:22:31.130 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com 2023-10-30T13:22:31.130 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T13:22:31.192 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:22:31.192 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:22:31.193 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:22:31.193 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:22:31.193 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:22:31.193 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:22:31.193 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:22:31.194 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:22:31.194 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:22:31.194 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:22:31.641 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2023-10-30T13:22:31.643 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2023-10-30T13:22:31.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-30T13:22:31.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-30T13:22:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:22:31.755 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:22:31.755 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-30T13:22:31.809 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:22:31.810 INFO:tasks.ceph:role: osd.6 2023-10-30T13:22:31.810 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com 2023-10-30T13:22:31.811 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T13:22:31.872 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:22:31.872 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:22:31.873 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:22:31.873 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:22:31.873 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:22:31.873 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:22:31.873 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:22:31.874 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:22:31.874 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:22:31.874 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:22:32.330 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2023-10-30T13:22:32.333 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2023-10-30T13:22:32.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-30T13:22:32.456 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-30T13:22:32.468 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:22:32.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:22:32.469 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-30T13:22:32.529 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:22:32.529 INFO:tasks.ceph:role: osd.7 2023-10-30T13:22:32.530 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi154.front.sepia.ceph.com 2023-10-30T13:22:32.530 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T13:22:32.593 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:22:32.593 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:22:32.594 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:22:32.594 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:22:32.594 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:22:32.594 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:22:32.594 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:22:32.595 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:22:32.595 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:22:32.595 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:22:33.101 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2023-10-30T13:22:33.103 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2023-10-30T13:22:33.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-30T13:22:33.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-30T13:22:33.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:22:33.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:22:33.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:33.372 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:33.368+0000 7f272780f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-30T13:22:33.376 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:33.372+0000 7f272780f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-30T13:22:33.376 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:33.372+0000 7f272780f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T13:22:33.377 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:33.372+0000 7f272780f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T13:22:33.377 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:33.372+0000 7f272780f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T13:22:33.377 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:33.372+0000 7f272780f640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-30T13:22:36.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-30T13:22:36.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:36.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:36.508+0000 7f12d0e02640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-30T13:22:36.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:36.508+0000 7f12d0e02640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-30T13:22:36.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:36.508+0000 7f12d0e02640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T13:22:36.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:36.508+0000 7f12d0e02640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T13:22:36.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:36.508+0000 7f12d0e02640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T13:22:36.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:36.508+0000 7f12d0e02640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-30T13:22:39.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-30T13:22:39.577 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:39.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:39.660+0000 7fded257f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-30T13:22:39.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:39.660+0000 7fded257f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-30T13:22:39.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:39.660+0000 7fded257f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T13:22:39.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:39.660+0000 7fded257f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T13:22:39.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:39.660+0000 7fded257f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T13:22:39.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:39.660+0000 7fded257f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-30T13:22:42.664 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-30T13:22:42.719 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:42.801 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:42.796+0000 7f4a7abdd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-30T13:22:42.802 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:42.800+0000 7f4a7abdd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-30T13:22:42.802 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:42.800+0000 7f4a7abdd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T13:22:42.802 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:42.800+0000 7f4a7abdd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T13:22:42.803 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:42.800+0000 7f4a7abdd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T13:22:42.803 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:42.800+0000 7f4a7abdd640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-30T13:22:43.550 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:43.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:45.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-30T13:22:45.826 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-30T13:22:45.826 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:45.827 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-30T13:22:45.850 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:45.850 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-30T13:22:45.918 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:45.918 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-30T13:22:45.982 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:45.983 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-10-30T13:22:46.047 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:46.047 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-30T13:22:46.114 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:46.114 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-30T13:22:46.178 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:46.178 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-30T13:22:46.242 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:46.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-30T13:22:46.305 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:46.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-30T13:22:46.320 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:46.320 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-30T13:22:46.373 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:46.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-30T13:22:46.428 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:46.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-10-30T13:22:46.481 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:46.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-30T13:22:46.537 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:46.537 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-30T13:22:46.592 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:46.592 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-30T13:22:46.644 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:46.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-30T13:22:46.696 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:46.696 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-30T13:22:46.706 INFO:tasks.ceph:Adding keys to all mons... 2023-10-30T13:22:46.706 DEBUG:teuthology.orchestra.run.smithi046:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T13:22:46.756 DEBUG:teuthology.orchestra.run.smithi154:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T13:22:46.767 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.y] 2023-10-30T13:22:46.767 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAHrj9lvJxSMRAAdfh3lYeLa4gfLK2FnGi7Bw== 2023-10-30T13:22:46.767 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.a] 2023-10-30T13:22:46.768 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAHrj9l6O5mOBAAFXGQOO0UDjDXYB/mtTYicA== 2023-10-30T13:22:46.768 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.c] 2023-10-30T13:22:46.768 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAIrj9lOG6MBBAAqO0GFn99pYPZ4m4oBVh5PA== 2023-10-30T13:22:46.768 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.e] 2023-10-30T13:22:46.769 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAIrj9laLjtCxAA6160z8lyeG9/EIUBx7alnw== 2023-10-30T13:22:46.769 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.0] 2023-10-30T13:22:46.769 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAJrj9lG6bFOBAA2uLFuA2CDPFY+ADIkQa2Tw== 2023-10-30T13:22:46.769 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.1] 2023-10-30T13:22:46.769 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQANrj9lEDUHBBAAVA11KrOuW7LPkaKOVGv2lw== 2023-10-30T13:22:46.769 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.2] 2023-10-30T13:22:46.770 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAQrj9l620yChAA0jbJePTmAUY/Y9C/a6TiCQ== 2023-10-30T13:22:46.770 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.3] 2023-10-30T13:22:46.770 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQATrj9lfG29EBAAoya5WbUUXHGKSJse+vLEOA== 2023-10-30T13:22:46.770 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.x] 2023-10-30T13:22:46.771 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAHrj9lAhAtNBAA/uW6CGXWZTZSitZI9If3lw== 2023-10-30T13:22:46.771 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.b] 2023-10-30T13:22:46.771 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAIrj9ldc4hEBAAezlsMgVfU87+Fp7yWBssVQ== 2023-10-30T13:22:46.771 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.d] 2023-10-30T13:22:46.771 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAIrj9l0zX6FRAAgS4CboAYF9soVIikFoK3RQ== 2023-10-30T13:22:46.772 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.f] 2023-10-30T13:22:46.772 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAIrj9l2w/0HRAAyB1jpRJWXDsMRbjYI5mj2g== 2023-10-30T13:22:46.772 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.4] 2023-10-30T13:22:46.772 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAZrj9lriwiFhAA/WzCvARLqwSjAFY3AbIF1A== 2023-10-30T13:22:46.772 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.5] 2023-10-30T13:22:46.772 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAcrj9l/WZ4HhAAVBJkTOBNQgVYAqXsTnWucg== 2023-10-30T13:22:46.773 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.6] 2023-10-30T13:22:46.773 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAfrj9l3axzJxAAtKfy3mGaeVT/FMBoazMrXw== 2023-10-30T13:22:46.773 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.7] 2023-10-30T13:22:46.773 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAirj9lJT/GLxAAC7POyLPqp7vk+vVAw0aKOA== 2023-10-30T13:22:46.773 INFO:teuthology.orchestra.run.smithi154.stdout:[client.0] 2023-10-30T13:22:46.774 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAIrj9lIaocJBAArrx1Lw4RCTpbBQ50ht2cKA== 2023-10-30T13:22:46.775 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.y] 2023-10-30T13:22:46.775 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAHrj9lvJxSMRAAdfh3lYeLa4gfLK2FnGi7Bw== 2023-10-30T13:22:46.775 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a] 2023-10-30T13:22:46.776 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAHrj9l6O5mOBAAFXGQOO0UDjDXYB/mtTYicA== 2023-10-30T13:22:46.776 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c] 2023-10-30T13:22:46.776 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAIrj9lOG6MBBAAqO0GFn99pYPZ4m4oBVh5PA== 2023-10-30T13:22:46.776 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.e] 2023-10-30T13:22:46.776 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAIrj9laLjtCxAA6160z8lyeG9/EIUBx7alnw== 2023-10-30T13:22:46.777 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.0] 2023-10-30T13:22:46.777 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAJrj9lG6bFOBAA2uLFuA2CDPFY+ADIkQa2Tw== 2023-10-30T13:22:46.777 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.1] 2023-10-30T13:22:46.777 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQANrj9lEDUHBBAAVA11KrOuW7LPkaKOVGv2lw== 2023-10-30T13:22:46.777 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.2] 2023-10-30T13:22:46.777 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAQrj9l620yChAA0jbJePTmAUY/Y9C/a6TiCQ== 2023-10-30T13:22:46.778 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.3] 2023-10-30T13:22:46.778 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQATrj9lfG29EBAAoya5WbUUXHGKSJse+vLEOA== 2023-10-30T13:22:46.778 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.x] 2023-10-30T13:22:46.778 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAHrj9lAhAtNBAA/uW6CGXWZTZSitZI9If3lw== 2023-10-30T13:22:46.778 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b] 2023-10-30T13:22:46.779 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAIrj9ldc4hEBAAezlsMgVfU87+Fp7yWBssVQ== 2023-10-30T13:22:46.779 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d] 2023-10-30T13:22:46.779 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAIrj9l0zX6FRAAgS4CboAYF9soVIikFoK3RQ== 2023-10-30T13:22:46.779 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.f] 2023-10-30T13:22:46.779 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAIrj9l2w/0HRAAyB1jpRJWXDsMRbjYI5mj2g== 2023-10-30T13:22:46.780 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.4] 2023-10-30T13:22:46.780 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAZrj9lriwiFhAA/WzCvARLqwSjAFY3AbIF1A== 2023-10-30T13:22:46.780 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.5] 2023-10-30T13:22:46.780 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAcrj9l/WZ4HhAAVBJkTOBNQgVYAqXsTnWucg== 2023-10-30T13:22:46.781 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.6] 2023-10-30T13:22:46.781 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAfrj9l3axzJxAAtKfy3mGaeVT/FMBoazMrXw== 2023-10-30T13:22:46.781 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.7] 2023-10-30T13:22:46.781 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAirj9lJT/GLxAAC7POyLPqp7vk+vVAw0aKOA== 2023-10-30T13:22:46.781 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2023-10-30T13:22:46.782 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAIrj9lIaocJBAArrx1Lw4RCTpbBQ50ht2cKA== 2023-10-30T13:22:46.782 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T13:22:46.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T13:22:46.877 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:22:46.881 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:22:46.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:22:46.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:22:46.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:22:47.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:22:47.070 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.157 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.161 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.213 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.279 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.362 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T13:22:47.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T13:22:47.454 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:22:47.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:22:47.544 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:22:47.547 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:22:47.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:22:47.607 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:22:47.681 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.737 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.803 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.888 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.893 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:22:47.978 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T13:22:47.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T13:22:48.037 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-30T13:22:48.037 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-30T13:22:48.058 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T13:22:48.178 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-30T13:22:48.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-30T13:22:48.213 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T13:22:48.313 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-30T13:22:48.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-30T13:22:48.379 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T13:22:48.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-30T13:22:48.490 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:48.494 DEBUG:teuthology.orchestra.run.smithi154:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:22:48.536 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-30T13:22:48.537 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T13:22:48.538 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T13:22:48.547 INFO:tasks.ceph.mon.a:Started 2023-10-30T13:22:48.547 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T13:22:48.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T13:22:48.583 INFO:tasks.ceph.mon.b:Started 2023-10-30T13:22:48.583 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T13:22:48.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-30T13:22:48.585 INFO:tasks.ceph.mon.c:Started 2023-10-30T13:22:48.586 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-30T13:22:48.586 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-30T13:22:48.586 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-30T13:22:48.590 INFO:tasks.ceph.mgr.y:Started 2023-10-30T13:22:48.591 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-30T13:22:48.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-30T13:22:48.593 INFO:tasks.ceph.mgr.x:Started 2023-10-30T13:22:48.594 INFO:tasks.ceph:Setting crush tunables to default 2023-10-30T13:22:48.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd crush tunables default 2023-10-30T13:22:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.737+0000 7f7ccc816640 1 Processor -- start 2023-10-30T13:22:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.737+0000 7f7ccc816640 1 -- start start 2023-10-30T13:22:48.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.741+0000 7f7ccc816640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:48.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.741+0000 7f7ccc816640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc815a6b0 con 0x7f7cc8153c20 2023-10-30T13:22:48.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.741+0000 7f7ccc816640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc815d730 con 0x7f7cc80cb4c0 2023-10-30T13:22:48.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.741+0000 7f7ccc816640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc815dbc0 con 0x7f7cc8153840 2023-10-30T13:22:48.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.741+0000 7f7cc6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:48.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.741+0000 7f7cc6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35370/0 (socket says 172.21.15.46:35370) 2023-10-30T13:22:48.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.741+0000 7f7cc659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7cc8153c20 0x7f7cc80c3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57540/0 (socket says 172.21.15.46:57540) 2023-10-30T13:22:48.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.741+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 learned_addr learned my addr 172.21.15.46:0/4020342878 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:22:48.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.741+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 msgr2=0x7f7cc815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:22:48.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.741+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 msgr2=0x7f7cc815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:22:48.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.741+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:22:48.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.741+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:22:48.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.945+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:48.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.945+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 msgr2=0x7f7cc815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:22:48.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.945+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 msgr2=0x7f7cc815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:22:48.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.945+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:22:48.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:48.945+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:22:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:49.349+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:49.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:49.349+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 msgr2=0x7f7cc815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:22:49.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:49.349+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 msgr2=0x7f7cc815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:22:49.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:49.349+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:22:49.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:49.349+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:22:50.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:50.153+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:50.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:50.153+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 msgr2=0x7f7cc815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:22:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:50.153+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 msgr2=0x7f7cc815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:22:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:50.153+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:22:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:50.153+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T13:22:51.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc4d99640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 msgr2=0x7f7cc815a170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:22:51.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc4d99640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cc815a170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:51.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc4d99640 1 -- 172.21.15.46:0/4020342878 >> v1:172.21.15.154:6789/0 conn(0x7f7cc80cb4c0 legacy=0x7f7cc80cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:51.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc4d99640 1 -- 172.21.15.46:0/4020342878 >> v1:172.21.15.46:6789/0 conn(0x7f7cc8153c20 legacy=0x7f7cc80c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:51.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc4d99640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:51.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc4d99640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:51.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc4d99640 1 -- 172.21.15.46:0/4020342878 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc815dbc0 con 0x7f7cb40036f0 2023-10-30T13:22:51.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc4d99640 1 -- 172.21.15.46:0/4020342878 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb4008e00 con 0x7f7cb4006220 2023-10-30T13:22:51.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:51.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:51.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 msgr2=0x7f7cb4005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:22:51.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 msgr2=0x7f7cb4005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:22:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:22:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc659c640 1 -- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 msgr2=0x7f7cb40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:22:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc659c640 1 -- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 msgr2=0x7f7cb40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:22:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:22:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:22:51.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.741+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:22:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.945+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.945+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.945+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 msgr2=0x7f7cb4005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:22:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.945+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 msgr2=0x7f7cb4005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:22:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.945+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:22:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.945+0000 7f7cc659c640 1 -- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 msgr2=0x7f7cb40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:22:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.945+0000 7f7cc659c640 1 -- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 msgr2=0x7f7cb40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:22:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.945+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:22:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.945+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:22:51.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:51.945+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:22:52.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:52.349+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:52.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:52.349+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:52.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:52.349+0000 7f7cc659c640 1 -- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 msgr2=0x7f7cb40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:22:52.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:52.349+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 msgr2=0x7f7cb4005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:22:52.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:52.349+0000 7f7cc659c640 1 -- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 msgr2=0x7f7cb40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:22:52.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:52.349+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 msgr2=0x7f7cb4005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:22:52.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:52.349+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:22:52.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:52.349+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:22:52.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:52.349+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:22:52.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:52.349+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:22:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:53.153+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:53.153+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:53.153+0000 7f7cc659c640 1 -- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 msgr2=0x7f7cb40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:22:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:53.153+0000 7f7cc659c640 1 -- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 msgr2=0x7f7cb40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:22:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:53.153+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:22:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:53.153+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 msgr2=0x7f7cb4005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:22:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:53.153+0000 7f7cc6d9d640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 msgr2=0x7f7cb4005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:22:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:53.153+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:22:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:53.153+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T13:22:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:53.153+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T13:22:54.220 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:54.216+0000 7f11d552b180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T13:22:54.332 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:54.324+0000 7f8d267d9180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T13:22:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc4d99640 1 -- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 msgr2=0x7f7cb40086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:22:54.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc4d99640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc4d99640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 msgr2=0x7f7cb4005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:22:54.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc4d99640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb40036f0 0x7f7cb4005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc4d99640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb4009200 0x7f7cb400b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc759e640 1 -- 172.21.15.46:0/4020342878 reap_dead start 2023-10-30T13:22:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc4d99640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc4d99640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cb400bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:54.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc4d99640 1 -- 172.21.15.46:0/4020342878 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc815dbc0 con 0x7f7cc8153840 2023-10-30T13:22:54.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc4d99640 1 -- 172.21.15.46:0/4020342878 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb4008e00 con 0x7f7cb4009200 2023-10-30T13:22:54.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc4d99640 1 -- 172.21.15.46:0/4020342878 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb4005c50 con 0x7f7cb4006220 2023-10-30T13:22:54.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc759e640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.46:3300/0 conn(0x7f7cc8153840 0x7f7cb400bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:54.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc6d9d640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb4009200 0x7f7cb400b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:54.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:54.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc759e640 1 -- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 msgr2=0x7f7cb40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:54.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc759e640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc759e640 1 -- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb4009200 msgr2=0x7f7cb400b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:54.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc759e640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb4009200 0x7f7cb400b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc759e640 1 -- 172.21.15.46:0/4020342878 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc80ca8f0 con 0x7f7cc8153840 2023-10-30T13:22:54.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.740+0000 7f7cc659c640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7cc759e640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8153840 0x7f7cb400bb30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7cc80cfb80 tx=0x7f7cc001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7939fee1b85ce4a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7cc5d9b640 1 -- 172.21.15.46:0/4020342878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0005d50 con 0x7f7cc8153840 2023-10-30T13:22:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7cc5d9b640 1 -- 172.21.15.46:0/4020342878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc0003980 con 0x7f7cc8153840 2023-10-30T13:22:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7cc5d9b640 1 -- 172.21.15.46:0/4020342878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0003d70 con 0x7f7cc8153840 2023-10-30T13:22:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7ccc816640 1 -- 172.21.15.46:0/4020342878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8153840 msgr2=0x7f7cb400bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7ccc816640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8153840 0x7f7cb400bb30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7cc80cfb80 tx=0x7f7cc001c420 comp rx=0 tx=0).stop 2023-10-30T13:22:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7ccc816640 1 -- 172.21.15.46:0/4020342878 shutdown_connections 2023-10-30T13:22:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7ccc816640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8153840 0x7f7cb400bb30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7ccc816640 1 --2- 172.21.15.46:0/4020342878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb4006220 0x7f7cb40086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7ccc816640 1 --2- 172.21.15.46:0/4020342878 >> v2:172.21.15.154:3300/0 conn(0x7f7cb4009200 0x7f7cb400b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7ccc816640 1 -- 172.21.15.46:0/4020342878 >> 172.21.15.46:0/4020342878 conn(0x7f7cc806be40 msgr2=0x7f7cc806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7ccc816640 1 -- 172.21.15.46:0/4020342878 shutdown_connections 2023-10-30T13:22:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.744+0000 7f7ccc816640 1 -- 172.21.15.46:0/4020342878 wait complete. 2023-10-30T13:22:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7ccc816640 1 Processor -- start 2023-10-30T13:22:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7ccc816640 1 -- start start 2023-10-30T13:22:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7ccc816640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb4006220 0x7f7cc80c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7ccc816640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cb4009200 0x7f7cc815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7ccc816640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cc8153840 0x7f7cc81f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7ccc816640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc80c76f0 con 0x7f7cb4009200 2023-10-30T13:22:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7ccc816640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc80c60e0 con 0x7f7cb4006220 2023-10-30T13:22:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7ccc816640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc81494a0 con 0x7f7cc8153840 2023-10-30T13:22:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7cc659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cb4009200 0x7f7cc815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7cc659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cb4009200 0x7f7cc815c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35504/0 (socket says 172.21.15.46:35504) 2023-10-30T13:22:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7cc6d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb4006220 0x7f7cc80c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7cc659c640 1 -- 172.21.15.46:0/1333275072 learned_addr learned my addr 172.21.15.46:0/1333275072 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:22:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7cc759e640 1 --2- 172.21.15.46:0/1333275072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cc8153840 0x7f7cc81f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7cc659c640 1 -- 172.21.15.46:0/1333275072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cc8153840 msgr2=0x7f7cc81f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7cc659c640 1 --2- 172.21.15.46:0/1333275072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cc8153840 0x7f7cc81f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7cc659c640 1 -- 172.21.15.46:0/1333275072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb4006220 msgr2=0x7f7cc80c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7cc659c640 1 --2- 172.21.15.46:0/1333275072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb4006220 0x7f7cc80c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7cc659c640 1 -- 172.21.15.46:0/1333275072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc815d730 con 0x7f7cb4009200 2023-10-30T13:22:54.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7cc659c640 1 --2- 172.21.15.46:0/1333275072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cb4009200 0x7f7cc815c2b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc0076f0 tx=0x7f7cbc001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:54.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7caf7fe640 1 -- 172.21.15.46:0/1333275072 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc017840 con 0x7f7cb4009200 2023-10-30T13:22:54.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7caf7fe640 1 -- 172.21.15.46:0/1333275072 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cbc0179a0 con 0x7f7cb4009200 2023-10-30T13:22:54.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7caf7fe640 1 -- 172.21.15.46:0/1333275072 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc017d10 con 0x7f7cb4009200 2023-10-30T13:22:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7ccc816640 1 -- 172.21.15.46:0/1333275072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc80c67e0 con 0x7f7cb4009200 2023-10-30T13:22:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7ccc816640 1 -- 172.21.15.46:0/1333275072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc80c6a30 con 0x7f7cb4009200 2023-10-30T13:22:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.748+0000 7f7caf7fe640 1 -- 172.21.15.46:0/1333275072 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7cbc0163c0 con 0x7f7cb4009200 2023-10-30T13:22:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.752+0000 7f7caf7fe640 1 -- 172.21.15.46:0/1333275072 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7cbc015700 con 0x7f7cb4009200 2023-10-30T13:22:54.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.752+0000 7f7ccc816640 1 -- 172.21.15.46:0/1333275072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc815d090 con 0x7f7cb4009200 2023-10-30T13:22:54.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.764+0000 7f7caf7fe640 1 -- 172.21.15.46:0/1333275072 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f7cbc040b40 con 0x7f7cb4009200 2023-10-30T13:22:54.793 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:54.788+0000 7f11d552b180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T13:22:54.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.900+0000 7f7ccc816640 1 -- 172.21.15.46:0/1333275072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7cc804e9c0 con 0x7f7cb4009200 2023-10-30T13:22:54.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.900+0000 7f7caf7fe640 1 -- 172.21.15.46:0/1333275072 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7cbc02f070 con 0x7f7cb4009200 2023-10-30T13:22:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2023-10-30T13:22:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.904+0000 7f7ccc816640 1 -- 172.21.15.46:0/1333275072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cb4009200 msgr2=0x7f7cc815c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.904+0000 7f7ccc816640 1 --2- 172.21.15.46:0/1333275072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cb4009200 0x7f7cc815c2b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc0076f0 tx=0x7f7cbc001640 comp rx=0 tx=0).stop 2023-10-30T13:22:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.904+0000 7f7ccc816640 1 -- 172.21.15.46:0/1333275072 shutdown_connections 2023-10-30T13:22:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.904+0000 7f7ccc816640 1 --2- 172.21.15.46:0/1333275072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cc8153840 0x7f7cc81f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.904+0000 7f7ccc816640 1 --2- 172.21.15.46:0/1333275072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cb4009200 0x7f7cc815c2b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.904+0000 7f7ccc816640 1 --2- 172.21.15.46:0/1333275072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb4006220 0x7f7cc80c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.904+0000 7f7ccc816640 1 -- 172.21.15.46:0/1333275072 >> 172.21.15.46:0/1333275072 conn(0x7f7cc806be40 msgr2=0x7f7cc814f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.904+0000 7f7ccc816640 1 -- 172.21.15.46:0/1333275072 shutdown_connections 2023-10-30T13:22:54.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:54.904+0000 7f7ccc816640 1 -- 172.21.15.46:0/1333275072 wait complete. 2023-10-30T13:22:54.929 INFO:tasks.ceph:check_enable_crimson: False 2023-10-30T13:22:54.929 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-30T13:22:54.930 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:54.930 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-30T13:22:54.952 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:54.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-30T13:22:55.006 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:55.004+0000 7f11d552b180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T13:22:55.013 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:55.013 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-30T13:22:55.016 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:55.008+0000 7f8d267d9180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T13:22:55.037 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:22:55.037 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-30T13:22:55.101 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:55.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-30T13:22:55.115 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:55.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-30T13:22:55.168 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:55.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-30T13:22:55.221 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-10-30T13:22:55.222 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-30T13:22:55.274 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:55.268+0000 7f8d267d9180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T13:22:55.277 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new de627c1f-8bc7-4f98-ae7d-2be818fb4024 0 2023-10-30T13:22:55.444 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.440+0000 7f628a2fc640 1 Processor -- start 2023-10-30T13:22:55.444 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.440+0000 7f628a2fc640 1 -- start start 2023-10-30T13:22:55.445 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.440+0000 7f628a2fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f628406e570 0x7f62840c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.445 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.440+0000 7f628a2fc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f62840c79a0 0x7f628406f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.446 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.440+0000 7f628a2fc640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628406e360 con 0x7f6284151900 2023-10-30T13:22:55.446 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.440+0000 7f628a2fc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628406eab0 con 0x7f628406e570 2023-10-30T13:22:55.446 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.440+0000 7f628a2fc640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62840c7480 con 0x7f62840c79a0 2023-10-30T13:22:55.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.440+0000 7f6288899640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f62840c79a0 0x7f628406f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.440+0000 7f6283fff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f6284151900 0x7f628406d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:40108/0 (socket says 172.21.15.154:40108) 2023-10-30T13:22:55.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.440+0000 7f6283fff640 1 -- 172.21.15.154:0/1075255067 learned_addr learned my addr 172.21.15.154:0/1075255067 (peer_addr_for_me v1:172.21.15.154:0/0) 2023-10-30T13:22:55.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f62837fe640 1 --2- 172.21.15.154:0/1075255067 >> v2:172.21.15.46:3300/0 conn(0x7f628406e570 0x7f62840c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f6288899640 1 -- 172.21.15.154:0/1075255067 >> v2:172.21.15.46:3300/0 conn(0x7f628406e570 msgr2=0x7f62840c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:55.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f6288899640 1 --2- 172.21.15.154:0/1075255067 >> v2:172.21.15.46:3300/0 conn(0x7f628406e570 0x7f62840c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f6288899640 1 -- 172.21.15.154:0/1075255067 >> v1:172.21.15.154:6789/0 conn(0x7f6284151900 legacy=0x7f628406d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f6288899640 1 -- 172.21.15.154:0/1075255067 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6284158320 con 0x7f62840c79a0 2023-10-30T13:22:55.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f6288899640 1 --2- 172.21.15.154:0/1075255067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62840c79a0 0x7f628406f1c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f628406e9c0 tx=0x7f627401c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3397735669cd52b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:55.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f6282ffd640 1 -- 172.21.15.154:0/1075255067 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62740ab600 con 0x7f62840c79a0 2023-10-30T13:22:55.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f6282ffd640 1 -- 172.21.15.154:0/1075255067 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62740a7050 con 0x7f62840c79a0 2023-10-30T13:22:55.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f6282ffd640 1 -- 172.21.15.154:0/1075255067 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62740ab980 con 0x7f62840c79a0 2023-10-30T13:22:55.451 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f628a2fc640 1 -- 172.21.15.154:0/1075255067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62840c79a0 msgr2=0x7f628406f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.451 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f628a2fc640 1 --2- 172.21.15.154:0/1075255067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62840c79a0 0x7f628406f1c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f628406e9c0 tx=0x7f627401c430 comp rx=0 tx=0).stop 2023-10-30T13:22:55.451 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f628a2fc640 1 -- 172.21.15.154:0/1075255067 shutdown_connections 2023-10-30T13:22:55.451 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f628a2fc640 1 --2- 172.21.15.154:0/1075255067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62840c79a0 0x7f628406f1c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.451 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f628a2fc640 1 --2- 172.21.15.154:0/1075255067 >> v2:172.21.15.46:3300/0 conn(0x7f628406e570 0x7f62840c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.452 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f628a2fc640 1 -- 172.21.15.154:0/1075255067 >> 172.21.15.154:0/1075255067 conn(0x7f62840ccfe0 msgr2=0x7f62840cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:55.452 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f628a2fc640 1 -- 172.21.15.154:0/1075255067 shutdown_connections 2023-10-30T13:22:55.452 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.444+0000 7f628a2fc640 1 -- 172.21.15.154:0/1075255067 wait complete. 2023-10-30T13:22:55.452 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f628a2fc640 1 Processor -- start 2023-10-30T13:22:55.452 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f628a2fc640 1 -- start start 2023-10-30T13:22:55.454 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f628a2fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f628406e570 0x7f628414c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.454 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f628a2fc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62840c79a0 0x7f62841f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.455 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f628a2fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6284151900 0x7f628414d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.455 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f628a2fc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628414c3b0 con 0x7f628406e570 2023-10-30T13:22:55.455 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f628a2fc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628414bd80 con 0x7f6284151900 2023-10-30T13:22:55.455 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f628a2fc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62841613b0 con 0x7f62840c79a0 2023-10-30T13:22:55.455 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6288899640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6284151900 0x7f628414d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.456 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6288899640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6284151900 0x7f628414d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:35050/0 (socket says 172.21.15.154:35050) 2023-10-30T13:22:55.456 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6288899640 1 -- 172.21.15.154:0/1688617551 learned_addr learned my addr 172.21.15.154:0/1688617551 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-10-30T13:22:55.456 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f62837fe640 1 --2- 172.21.15.154:0/1688617551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62840c79a0 0x7f62841f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.456 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6283fff640 1 --2- 172.21.15.154:0/1688617551 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f628406e570 0x7f628414c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6288899640 1 -- 172.21.15.154:0/1688617551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62840c79a0 msgr2=0x7f62841f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6288899640 1 --2- 172.21.15.154:0/1688617551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62840c79a0 0x7f62841f2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6288899640 1 -- 172.21.15.154:0/1688617551 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f628406e570 msgr2=0x7f628414c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6288899640 1 --2- 172.21.15.154:0/1688617551 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f628406e570 0x7f628414c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6288899640 1 -- 172.21.15.154:0/1688617551 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6274009420 con 0x7f6284151900 2023-10-30T13:22:55.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f62837fe640 1 --2- 172.21.15.154:0/1688617551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62840c79a0 0x7f62841f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:55.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6288899640 1 --2- 172.21.15.154:0/1688617551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6284151900 0x7f628414d890 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6274019040 tx=0x7f627401ac70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:55.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6280ff9640 1 -- 172.21.15.154:0/1688617551 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62740b4420 con 0x7f6284151900 2023-10-30T13:22:55.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6280ff9640 1 -- 172.21.15.154:0/1688617551 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62740b4580 con 0x7f6284151900 2023-10-30T13:22:55.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6280ff9640 1 -- 172.21.15.154:0/1688617551 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62740b4910 con 0x7f6284151900 2023-10-30T13:22:55.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f628a2fc640 1 -- 172.21.15.154:0/1688617551 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6284151e60 con 0x7f6284151900 2023-10-30T13:22:55.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f628a2fc640 1 -- 172.21.15.154:0/1688617551 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628414c040 con 0x7f6284151900 2023-10-30T13:22:55.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6280ff9640 1 -- 172.21.15.154:0/1688617551 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f62740b2070 con 0x7f6284151900 2023-10-30T13:22:55.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f6280ff9640 1 -- 172.21.15.154:0/1688617551 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f62740bdb70 con 0x7f6284151900 2023-10-30T13:22:55.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.448+0000 7f628a2fc640 1 -- 172.21.15.154:0/1688617551 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62840cb4f0 con 0x7f6284151900 2023-10-30T13:22:55.460 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.452+0000 7f6280ff9640 1 -- 172.21.15.154:0/1688617551 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f62740b9070 con 0x7f6284151900 2023-10-30T13:22:55.523 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.520+0000 7f628a2fc640 1 -- 172.21.15.154:0/1688617551 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "de627c1f-8bc7-4f98-ae7d-2be818fb4024", "id": 0} v 0) v1 -- 0x7f628406c050 con 0x7f6284151900 2023-10-30T13:22:55.532 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.528+0000 7f6280ff9640 1 -- 172.21.15.154:0/1688617551 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "de627c1f-8bc7-4f98-ae7d-2be818fb4024", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f62740b0070 con 0x7f6284151900 2023-10-30T13:22:55.532 INFO:teuthology.orchestra.run.smithi154.stdout:0 2023-10-30T13:22:55.533 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.528+0000 7f628a2fc640 1 -- 172.21.15.154:0/1688617551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6284151900 msgr2=0x7f628414d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.533 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.528+0000 7f628a2fc640 1 --2- 172.21.15.154:0/1688617551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6284151900 0x7f628414d890 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6274019040 tx=0x7f627401ac70 comp rx=0 tx=0).stop 2023-10-30T13:22:55.534 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.528+0000 7f628a2fc640 1 -- 172.21.15.154:0/1688617551 shutdown_connections 2023-10-30T13:22:55.534 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.528+0000 7f628a2fc640 1 --2- 172.21.15.154:0/1688617551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6284151900 0x7f628414d890 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.534 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.528+0000 7f628a2fc640 1 --2- 172.21.15.154:0/1688617551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62840c79a0 0x7f62841f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.534 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.528+0000 7f628a2fc640 1 --2- 172.21.15.154:0/1688617551 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f628406e570 0x7f628414c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.534 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.528+0000 7f628a2fc640 1 -- 172.21.15.154:0/1688617551 >> 172.21.15.154:0/1688617551 conn(0x7f62840ccfe0 msgr2=0x7f62840cade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:55.535 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.528+0000 7f628a2fc640 1 -- 172.21.15.154:0/1688617551 shutdown_connections 2023-10-30T13:22:55.535 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.528+0000 7f628a2fc640 1 -- 172.21.15.154:0/1688617551 wait complete. 2023-10-30T13:22:55.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new e21e18dd-5538-4419-9f05-8ef2a6ce40d2 1 2023-10-30T13:22:55.619 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:55.616+0000 7f11d552b180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T13:22:55.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.656+0000 7fe96cb4d640 1 Processor -- start 2023-10-30T13:22:55.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.656+0000 7fe96cb4d640 1 -- start start 2023-10-30T13:22:55.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96cb4d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe96815a780 0x7fe9680c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96cb4d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe968159be0 con 0x7fe968150800 2023-10-30T13:22:55.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96cb4d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe968156850 con 0x7fe9680c2e20 2023-10-30T13:22:55.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96cb4d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9680c7020 con 0x7fe96815a780 2023-10-30T13:22:55.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe965d9b640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fe9680c2e20 0x7fe96815a070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:40118/0 (socket says 172.21.15.154:40118) 2023-10-30T13:22:55.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe965d9b640 1 -- 172.21.15.154:0/392861985 learned_addr learned my addr 172.21.15.154:0/392861985 (peer_addr_for_me v1:172.21.15.154:0/0) 2023-10-30T13:22:55.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe966d9d640 1 --2- 172.21.15.154:0/392861985 >> v2:172.21.15.46:3300/0 conn(0x7fe96815a780 0x7fe9680c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081877141 0 0) 0x7fe968156850 con 0x7fe9680c2e20 2023-10-30T13:22:55.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe950005740 con 0x7fe9680c2e20 2023-10-30T13:22:55.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1526533518 0 0) 0x7fe950005740 con 0x7fe9680c2e20 2023-10-30T13:22:55.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 >> v2:172.21.15.46:3300/0 conn(0x7fe96815a780 msgr2=0x7fe9680c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:55.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 --2- 172.21.15.154:0/392861985 >> v2:172.21.15.46:3300/0 conn(0x7fe96815a780 0x7fe9680c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 >> v1:172.21.15.46:6789/0 conn(0x7fe968150800 legacy=0x7fe9680cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe968156b30 con 0x7fe9680c2e20 2023-10-30T13:22:55.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fe95c001680 con 0x7fe9680c2e20 2023-10-30T13:22:55.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 >> v1:172.21.15.154:6789/0 conn(0x7fe9680c2e20 legacy=0x7fe96815a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 --2- 172.21.15.154:0/392861985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe950005b90 0x7fe950005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 --2- 172.21.15.154:0/392861985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 0x7fe950006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 --2- 172.21.15.154:0/392861985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9500073a0 0x7fe95000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95000e220 con 0x7fe950005b90 2023-10-30T13:22:55.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95000e390 con 0x7fe9500069e0 2023-10-30T13:22:55.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95000e550 con 0x7fe9500073a0 2023-10-30T13:22:55.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe965d9b640 1 --2- 172.21.15.154:0/392861985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 0x7fe950006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe966d9d640 1 --2- 172.21.15.154:0/392861985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9500073a0 0x7fe95000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe96659c640 1 --2- 172.21.15.154:0/392861985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe950005b90 0x7fe950005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe965d9b640 1 -- 172.21.15.154:0/392861985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9500073a0 msgr2=0x7fe95000dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:55.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe965d9b640 1 --2- 172.21.15.154:0/392861985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9500073a0 0x7fe95000dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe965d9b640 1 -- 172.21.15.154:0/392861985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe950005b90 msgr2=0x7fe950005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:55.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe965d9b640 1 --2- 172.21.15.154:0/392861985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe950005b90 0x7fe950005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe965d9b640 1 -- 172.21.15.154:0/392861985 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe950005740 con 0x7fe9500069e0 2023-10-30T13:22:55.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.660+0000 7fe966d9d640 1 -- 172.21.15.154:0/392861985 reap_dead start 2023-10-30T13:22:55.672 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe965d9b640 1 --2- 172.21.15.154:0/392861985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 0x7fe950006e60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe95c00c240 tx=0x7fe95c01f330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e95fcd077bc2392 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:55.672 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c005ae0 con 0x7fe9500069e0 2023-10-30T13:22:55.673 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95c0ab070 con 0x7fe9500069e0 2023-10-30T13:22:55.673 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe96559a640 1 -- 172.21.15.154:0/392861985 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c007070 con 0x7fe9500069e0 2023-10-30T13:22:55.673 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe96cb4d640 1 -- 172.21.15.154:0/392861985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 msgr2=0x7fe950006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.673 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe96cb4d640 1 --2- 172.21.15.154:0/392861985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 0x7fe950006e60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe95c00c240 tx=0x7fe95c01f330 comp rx=0 tx=0).stop 2023-10-30T13:22:55.673 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe96cb4d640 1 -- 172.21.15.154:0/392861985 shutdown_connections 2023-10-30T13:22:55.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe96cb4d640 1 --2- 172.21.15.154:0/392861985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 0x7fe950006e60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe96cb4d640 1 -- 172.21.15.154:0/392861985 >> 172.21.15.154:0/392861985 conn(0x7fe96806c850 msgr2=0x7fe96806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:55.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe96cb4d640 1 -- 172.21.15.154:0/392861985 shutdown_connections 2023-10-30T13:22:55.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe96cb4d640 1 -- 172.21.15.154:0/392861985 wait complete. 2023-10-30T13:22:55.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe96cb4d640 1 Processor -- start 2023-10-30T13:22:55.675 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.664+0000 7fe96cb4d640 1 -- start start 2023-10-30T13:22:55.675 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe96cb4d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 0x7fe9681588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.675 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe96cb4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe968155a70 0x7fe968155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.675 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe96cb4d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe96814e0c0 0x7fe96814e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.676 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe96cb4d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe968158e00 con 0x7fe968155a70 2023-10-30T13:22:55.676 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe96cb4d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe968154ce0 con 0x7fe9500069e0 2023-10-30T13:22:55.676 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe96cb4d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96814ea80 con 0x7fe96814e0c0 2023-10-30T13:22:55.676 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe96659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 0x7fe9681588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.676 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe966d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe96814e0c0 0x7fe96814e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe966d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe96814e0c0 0x7fe96814e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:41730/0 (socket says 172.21.15.154:41730) 2023-10-30T13:22:55.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe966d9d640 1 -- 172.21.15.154:0/2096113048 learned_addr learned my addr 172.21.15.154:0/2096113048 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-10-30T13:22:55.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe96659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 0x7fe9681588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:35076/0 (socket says 172.21.15.154:35076) 2023-10-30T13:22:55.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe965d9b640 1 --2- 172.21.15.154:0/2096113048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe968155a70 0x7fe968155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe966d9d640 1 -- 172.21.15.154:0/2096113048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 msgr2=0x7fe9681588c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe966d9d640 1 --2- 172.21.15.154:0/2096113048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 0x7fe9681588c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.678 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe966d9d640 1 -- 172.21.15.154:0/2096113048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe968155a70 msgr2=0x7fe968155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.678 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe966d9d640 1 --2- 172.21.15.154:0/2096113048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe968155a70 0x7fe968155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.678 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe966d9d640 1 -- 172.21.15.154:0/2096113048 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9681579c0 con 0x7fe96814e0c0 2023-10-30T13:22:55.678 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe965d9b640 1 --2- 172.21.15.154:0/2096113048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe968155a70 0x7fe968155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:55.678 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe96659c640 1 --2- 172.21.15.154:0/2096113048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 0x7fe9681588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:55.679 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe966d9d640 1 --2- 172.21.15.154:0/2096113048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe96814e0c0 0x7fe96814e540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe9680c9a40 tx=0x7fe96000f580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:55.679 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe956ffd640 1 -- 172.21.15.154:0/2096113048 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96000e040 con 0x7fe96814e0c0 2023-10-30T13:22:55.679 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe956ffd640 1 -- 172.21.15.154:0/2096113048 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96000f040 con 0x7fe96814e0c0 2023-10-30T13:22:55.679 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe96cb4d640 1 -- 172.21.15.154:0/2096113048 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe968154760 con 0x7fe96814e0c0 2023-10-30T13:22:55.679 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe956ffd640 1 -- 172.21.15.154:0/2096113048 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe960015430 con 0x7fe96814e0c0 2023-10-30T13:22:55.680 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe96cb4d640 1 -- 172.21.15.154:0/2096113048 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe968159380 con 0x7fe96814e0c0 2023-10-30T13:22:55.680 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe956ffd640 1 -- 172.21.15.154:0/2096113048 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe960015890 con 0x7fe96814e0c0 2023-10-30T13:22:55.680 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe956ffd640 1 -- 172.21.15.154:0/2096113048 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fe96001dd80 con 0x7fe96814e0c0 2023-10-30T13:22:55.681 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.668+0000 7fe96cb4d640 1 -- 172.21.15.154:0/2096113048 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9681591c0 con 0x7fe96814e0c0 2023-10-30T13:22:55.681 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:55.672+0000 7f11d552b180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T13:22:55.682 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.676+0000 7fe956ffd640 1 -- 172.21.15.154:0/2096113048 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fe960022020 con 0x7fe96814e0c0 2023-10-30T13:22:55.730 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:55.728+0000 7f11d552b180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T13:22:55.748 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.744+0000 7fe96cb4d640 1 -- 172.21.15.154:0/2096113048 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e21e18dd-5538-4419-9f05-8ef2a6ce40d2", "id": 1} v 0) v1 -- 0x7fe96804e170 con 0x7fe96814e0c0 2023-10-30T13:22:55.761 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.756+0000 7fe956ffd640 1 -- 172.21.15.154:0/2096113048 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e21e18dd-5538-4419-9f05-8ef2a6ce40d2", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe96001d400 con 0x7fe96814e0c0 2023-10-30T13:22:55.761 INFO:teuthology.orchestra.run.smithi154.stdout:1 2023-10-30T13:22:55.762 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.760+0000 7fe96cb4d640 1 -- 172.21.15.154:0/2096113048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe96814e0c0 msgr2=0x7fe96814e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.763 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.760+0000 7fe96cb4d640 1 --2- 172.21.15.154:0/2096113048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe96814e0c0 0x7fe96814e540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe9680c9a40 tx=0x7fe96000f580 comp rx=0 tx=0).stop 2023-10-30T13:22:55.763 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.760+0000 7fe96cb4d640 1 -- 172.21.15.154:0/2096113048 shutdown_connections 2023-10-30T13:22:55.763 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.760+0000 7fe96cb4d640 1 --2- 172.21.15.154:0/2096113048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe96814e0c0 0x7fe96814e540 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.763 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.760+0000 7fe96cb4d640 1 --2- 172.21.15.154:0/2096113048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe968155a70 0x7fe968155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.763 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.760+0000 7fe96cb4d640 1 --2- 172.21.15.154:0/2096113048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9500069e0 0x7fe9681588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.764 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.760+0000 7fe96cb4d640 1 -- 172.21.15.154:0/2096113048 >> 172.21.15.154:0/2096113048 conn(0x7fe96806c850 msgr2=0x7fe9680c2380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:55.764 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.760+0000 7fe96cb4d640 1 -- 172.21.15.154:0/2096113048 shutdown_connections 2023-10-30T13:22:55.764 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.760+0000 7fe96cb4d640 1 -- 172.21.15.154:0/2096113048 wait complete. 2023-10-30T13:22:55.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 281c484d-79d3-4873-85c5-fb4fd3369397 2 2023-10-30T13:22:55.811 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:55.808+0000 7f11d552b180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T13:22:55.867 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:55.864+0000 7f11d552b180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T13:22:55.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.892+0000 7fb9f22e5640 1 Processor -- start 2023-10-30T13:22:55.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.892+0000 7fb9f22e5640 1 -- start start 2023-10-30T13:22:55.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9f22e5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c2e20 0x7fb9ec0c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9f22e5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb9ec15a070 0x7fb9ec15a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9f22e5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ec159be0 con 0x7fb9ec150800 2023-10-30T13:22:55.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9f22e5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9ec0c7020 con 0x7fb9ec15a070 2023-10-30T13:22:55.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9f22e5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec15a9d0 con 0x7fb9ec0c2e20 2023-10-30T13:22:55.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9f0882640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb9ec15a070 0x7fb9ec15a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9ebfff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fb9ec150800 0x7fb9ec0cb170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:40120/0 (socket says 172.21.15.154:40120) 2023-10-30T13:22:55.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9ebfff640 1 -- 172.21.15.154:0/3566729576 learned_addr learned my addr 172.21.15.154:0/3566729576 (peer_addr_for_me v1:172.21.15.154:0/0) 2023-10-30T13:22:55.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9eb7fe640 1 --2- 172.21.15.154:0/3566729576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c2e20 0x7fb9ec0c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9f0882640 1 -- 172.21.15.154:0/3566729576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c2e20 msgr2=0x7fb9ec0c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:55.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9f0882640 1 --2- 172.21.15.154:0/3566729576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c2e20 0x7fb9ec0c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9f0882640 1 -- 172.21.15.154:0/3566729576 >> v1:172.21.15.154:6789/0 conn(0x7fb9ec150800 legacy=0x7fb9ec0cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9eaffd640 1 -- 172.21.15.154:0/3566729576 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703799306 0 0) 0x7fb9ec159be0 con 0x7fb9ec150800 2023-10-30T13:22:55.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9f0882640 1 -- 172.21.15.154:0/3566729576 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec156850 con 0x7fb9ec15a070 2023-10-30T13:22:55.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9eb7fe640 1 --2- 172.21.15.154:0/3566729576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c2e20 0x7fb9ec0c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:55.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9f0882640 1 --2- 172.21.15.154:0/3566729576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec15a070 0x7fb9ec15a490 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb9e40097c0 tx=0x7fb9e401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40a62a8fb7d45a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:55.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9eaffd640 1 -- 172.21.15.154:0/3566729576 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e40a8970 con 0x7fb9ec15a070 2023-10-30T13:22:55.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9eaffd640 1 -- 172.21.15.154:0/3566729576 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e40a8ad0 con 0x7fb9ec15a070 2023-10-30T13:22:55.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.896+0000 7fb9eaffd640 1 -- 172.21.15.154:0/3566729576 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e40a8e60 con 0x7fb9ec15a070 2023-10-30T13:22:55.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 -- 172.21.15.154:0/3566729576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec15a070 msgr2=0x7fb9ec15a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 --2- 172.21.15.154:0/3566729576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec15a070 0x7fb9ec15a490 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb9e40097c0 tx=0x7fb9e401ec70 comp rx=0 tx=0).stop 2023-10-30T13:22:55.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 -- 172.21.15.154:0/3566729576 shutdown_connections 2023-10-30T13:22:55.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 --2- 172.21.15.154:0/3566729576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec15a070 0x7fb9ec15a490 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 --2- 172.21.15.154:0/3566729576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c2e20 0x7fb9ec0c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 -- 172.21.15.154:0/3566729576 >> 172.21.15.154:0/3566729576 conn(0x7fb9ec06c850 msgr2=0x7fb9ec06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:55.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 -- 172.21.15.154:0/3566729576 shutdown_connections 2023-10-30T13:22:55.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 -- 172.21.15.154:0/3566729576 wait complete. 2023-10-30T13:22:55.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 Processor -- start 2023-10-30T13:22:55.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 -- start start 2023-10-30T13:22:55.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec0c2e20 0x7fb9ec158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec150800 0x7fb9ec1e6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec15a070 0x7fb9ec14d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:55.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec1ef0f0 con 0x7fb9ec0c2e20 2023-10-30T13:22:55.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec159250 con 0x7fb9ec150800 2023-10-30T13:22:55.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec1ef390 con 0x7fb9ec15a070 2023-10-30T13:22:55.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f0882640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec15a070 0x7fb9ec14d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9eb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec150800 0x7fb9ec1e6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9eb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec150800 0x7fb9ec1e6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:58066/0 (socket says 172.21.15.154:58066) 2023-10-30T13:22:55.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9eb7fe640 1 -- 172.21.15.154:0/2457528132 learned_addr learned my addr 172.21.15.154:0/2457528132 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-10-30T13:22:55.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f0882640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec15a070 0x7fb9ec14d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:54106/0 (socket says 172.21.15.154:54106) 2023-10-30T13:22:55.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9ebfff640 1 --2- 172.21.15.154:0/2457528132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec0c2e20 0x7fb9ec158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:55.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9eb7fe640 1 -- 172.21.15.154:0/2457528132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec15a070 msgr2=0x7fb9ec14d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9eb7fe640 1 --2- 172.21.15.154:0/2457528132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec15a070 0x7fb9ec14d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9eb7fe640 1 -- 172.21.15.154:0/2457528132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec0c2e20 msgr2=0x7fb9ec158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9eb7fe640 1 --2- 172.21.15.154:0/2457528132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec0c2e20 0x7fb9ec158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9eb7fe640 1 -- 172.21.15.154:0/2457528132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d4009420 con 0x7fb9ec150800 2023-10-30T13:22:55.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9ebfff640 1 --2- 172.21.15.154:0/2457528132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec0c2e20 0x7fb9ec158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:22:55.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9eb7fe640 1 --2- 172.21.15.154:0/2457528132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec150800 0x7fb9ec1e6260 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400ebe0 tx=0x7fb9d400c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:55.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9e8ff9640 1 -- 172.21.15.154:0/2457528132 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d400eee0 con 0x7fb9ec150800 2023-10-30T13:22:55.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9e8ff9640 1 -- 172.21.15.154:0/2457528132 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d400cc30 con 0x7fb9ec150800 2023-10-30T13:22:55.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 -- 172.21.15.154:0/2457528132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e4009420 con 0x7fb9ec150800 2023-10-30T13:22:55.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9e8ff9640 1 -- 172.21.15.154:0/2457528132 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4013470 con 0x7fb9ec150800 2023-10-30T13:22:55.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9f22e5640 1 -- 172.21.15.154:0/2457528132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ec1ef950 con 0x7fb9ec150800 2023-10-30T13:22:55.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9e8ff9640 1 -- 172.21.15.154:0/2457528132 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb9d4005ca0 con 0x7fb9ec150800 2023-10-30T13:22:55.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.900+0000 7fb9e8ff9640 1 -- 172.21.15.154:0/2457528132 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fb9d40139f0 con 0x7fb9ec150800 2023-10-30T13:22:55.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.904+0000 7fb9f22e5640 1 -- 172.21.15.154:0/2457528132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ec04e170 con 0x7fb9ec150800 2023-10-30T13:22:55.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.904+0000 7fb9e8ff9640 1 -- 172.21.15.154:0/2457528132 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fb9d4021060 con 0x7fb9ec150800 2023-10-30T13:22:55.954 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:55.952+0000 7f11d552b180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T13:22:55.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.972+0000 7fb9f22e5640 1 -- 172.21.15.154:0/2457528132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "281c484d-79d3-4873-85c5-fb4fd3369397", "id": 2} v 0) v1 -- 0x7fb9ec073e10 con 0x7fb9ec150800 2023-10-30T13:22:55.981 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.976+0000 7fb9e8ff9640 1 -- 172.21.15.154:0/2457528132 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "281c484d-79d3-4873-85c5-fb4fd3369397", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb9d4017070 con 0x7fb9ec150800 2023-10-30T13:22:55.981 INFO:teuthology.orchestra.run.smithi154.stdout:2 2023-10-30T13:22:55.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.980+0000 7fb9f22e5640 1 -- 172.21.15.154:0/2457528132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec150800 msgr2=0x7fb9ec1e6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:55.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.980+0000 7fb9f22e5640 1 --2- 172.21.15.154:0/2457528132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec150800 0x7fb9ec1e6260 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400ebe0 tx=0x7fb9d400c520 comp rx=0 tx=0).stop 2023-10-30T13:22:55.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.980+0000 7fb9f22e5640 1 -- 172.21.15.154:0/2457528132 shutdown_connections 2023-10-30T13:22:55.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.980+0000 7fb9f22e5640 1 --2- 172.21.15.154:0/2457528132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec15a070 0x7fb9ec14d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.980+0000 7fb9f22e5640 1 --2- 172.21.15.154:0/2457528132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec150800 0x7fb9ec1e6260 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.980+0000 7fb9f22e5640 1 --2- 172.21.15.154:0/2457528132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec0c2e20 0x7fb9ec158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:55.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.980+0000 7fb9f22e5640 1 -- 172.21.15.154:0/2457528132 >> 172.21.15.154:0/2457528132 conn(0x7fb9ec06c850 msgr2=0x7fb9ec06cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:55.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.980+0000 7fb9f22e5640 1 -- 172.21.15.154:0/2457528132 shutdown_connections 2023-10-30T13:22:55.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:55.980+0000 7fb9f22e5640 1 -- 172.21.15.154:0/2457528132 wait complete. 2023-10-30T13:22:55.998 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 58d4f3f8-7966-4175-a940-907e8561511d 3 2023-10-30T13:22:56.013 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:56.008+0000 7f11d552b180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T13:22:56.020 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:56.012+0000 7f8d267d9180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T13:22:56.085 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:56.076+0000 7f8d267d9180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T13:22:56.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc97c051640 1 Processor -- start 2023-10-30T13:22:56.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc97c051640 1 -- start start 2023-10-30T13:22:56.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc97c051640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc974153980 0x7fc9740caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc97c051640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9740cb1e0 0x7fc97415d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.130 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc97c051640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97415a920 con 0x7fc974159930 2023-10-30T13:22:56.130 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc97c051640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9740cb930 con 0x7fc9740cb1e0 2023-10-30T13:22:56.130 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc97c051640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc974159dc0 con 0x7fc974153980 2023-10-30T13:22:56.130 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc9795ec640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fc974159930 0x7fc9740cf1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:46856/0 (socket says 172.21.15.154:46856) 2023-10-30T13:22:56.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc9795ec640 1 -- 172.21.15.154:0/1630186032 learned_addr learned my addr 172.21.15.154:0/1630186032 (peer_addr_for_me v1:172.21.15.154:0/0) 2023-10-30T13:22:56.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc979ded640 1 --2- 172.21.15.154:0/1630186032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc974153980 0x7fc9740caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc97a5ee640 1 --2- 172.21.15.154:0/1630186032 >> v2:172.21.15.154:3300/0 conn(0x7fc9740cb1e0 0x7fc97415d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc978deb640 1 -- 172.21.15.154:0/1630186032 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596131740 0 0) 0x7fc97415a920 con 0x7fc974159930 2023-10-30T13:22:56.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc978deb640 1 -- 172.21.15.154:0/1630186032 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc960003530 con 0x7fc974159930 2023-10-30T13:22:56.132 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc979ded640 1 -- 172.21.15.154:0/1630186032 >> v2:172.21.15.154:3300/0 conn(0x7fc9740cb1e0 msgr2=0x7fc97415d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:56.132 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc979ded640 1 --2- 172.21.15.154:0/1630186032 >> v2:172.21.15.154:3300/0 conn(0x7fc9740cb1e0 0x7fc97415d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.132 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.124+0000 7fc979ded640 1 -- 172.21.15.154:0/1630186032 >> v1:172.21.15.154:6789/0 conn(0x7fc974159930 legacy=0x7fc9740cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.132 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc979ded640 1 -- 172.21.15.154:0/1630186032 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97415a080 con 0x7fc974153980 2023-10-30T13:22:56.132 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc97a5ee640 1 --2- 172.21.15.154:0/1630186032 >> v2:172.21.15.154:3300/0 conn(0x7fc9740cb1e0 0x7fc97415d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:56.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc979ded640 1 --2- 172.21.15.154:0/1630186032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc974153980 0x7fc9740caca0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009d90 tx=0x7fc96c01c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f26d1f9deae0f0f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:56.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc978deb640 1 -- 172.21.15.154:0/1630186032 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0a58f0 con 0x7fc974153980 2023-10-30T13:22:56.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc978deb640 1 -- 172.21.15.154:0/1630186032 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96c0a5a50 con 0x7fc974153980 2023-10-30T13:22:56.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc978deb640 1 -- 172.21.15.154:0/1630186032 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0a5de0 con 0x7fc974153980 2023-10-30T13:22:56.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc97c051640 1 -- 172.21.15.154:0/1630186032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc974153980 msgr2=0x7fc9740caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.137 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc97c051640 1 --2- 172.21.15.154:0/1630186032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc974153980 0x7fc9740caca0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009d90 tx=0x7fc96c01c920 comp rx=0 tx=0).stop 2023-10-30T13:22:56.137 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc97c051640 1 -- 172.21.15.154:0/1630186032 shutdown_connections 2023-10-30T13:22:56.137 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc97c051640 1 --2- 172.21.15.154:0/1630186032 >> v2:172.21.15.154:3300/0 conn(0x7fc9740cb1e0 0x7fc97415d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.137 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc97c051640 1 --2- 172.21.15.154:0/1630186032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc974153980 0x7fc9740caca0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.137 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc97c051640 1 -- 172.21.15.154:0/1630186032 >> 172.21.15.154:0/1630186032 conn(0x7fc97406c0f0 msgr2=0x7fc97406c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:56.138 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc97c051640 1 -- 172.21.15.154:0/1630186032 shutdown_connections 2023-10-30T13:22:56.138 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.128+0000 7fc97c051640 1 -- 172.21.15.154:0/1630186032 wait complete. 2023-10-30T13:22:56.138 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97c051640 1 Processor -- start 2023-10-30T13:22:56.138 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97c051640 1 -- start start 2023-10-30T13:22:56.138 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97c051640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9740cb1e0 0x7fc97414b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.138 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97c051640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc974153980 0x7fc9740b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.139 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97c051640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9740ce760 0x7fc97414bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.139 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97c051640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9740ce5d0 con 0x7fc9740ce760 2023-10-30T13:22:56.139 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97c051640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc974154b50 con 0x7fc974153980 2023-10-30T13:22:56.139 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97c051640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97414b900 con 0x7fc9740cb1e0 2023-10-30T13:22:56.139 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc979ded640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9740cb1e0 0x7fc97414b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc9795ec640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc974153980 0x7fc9740b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc979ded640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9740cb1e0 0x7fc97414b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:54112/0 (socket says 172.21.15.154:54112) 2023-10-30T13:22:56.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc9795ec640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc974153980 0x7fc9740b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:58072/0 (socket says 172.21.15.154:58072) 2023-10-30T13:22:56.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc979ded640 1 -- 172.21.15.154:0/3083769571 learned_addr learned my addr 172.21.15.154:0/3083769571 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-10-30T13:22:56.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97a5ee640 1 --2- 172.21.15.154:0/3083769571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9740ce760 0x7fc97414bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc9795ec640 1 -- 172.21.15.154:0/3083769571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9740cb1e0 msgr2=0x7fc97414b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc9795ec640 1 --2- 172.21.15.154:0/3083769571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9740cb1e0 0x7fc97414b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc9795ec640 1 -- 172.21.15.154:0/3083769571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9740ce760 msgr2=0x7fc97414bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc9795ec640 1 --2- 172.21.15.154:0/3083769571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9740ce760 0x7fc97414bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc9795ec640 1 -- 172.21.15.154:0/3083769571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc970009420 con 0x7fc974153980 2023-10-30T13:22:56.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc979ded640 1 --2- 172.21.15.154:0/3083769571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9740cb1e0 0x7fc97414b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:56.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97a5ee640 1 --2- 172.21.15.154:0/3083769571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9740ce760 0x7fc97414bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:22:56.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc9795ec640 1 --2- 172.21.15.154:0/3083769571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc974153980 0x7fc9740b4330 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc964001ec0 tx=0x7fc96400c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:56.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc96a7fc640 1 -- 172.21.15.154:0/3083769571 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964005cf0 con 0x7fc974153980 2023-10-30T13:22:56.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc96a7fc640 1 -- 172.21.15.154:0/3083769571 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc964005e50 con 0x7fc974153980 2023-10-30T13:22:56.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc96a7fc640 1 -- 172.21.15.154:0/3083769571 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9640072d0 con 0x7fc974153980 2023-10-30T13:22:56.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97c051640 1 -- 172.21.15.154:0/3083769571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc96c009a70 con 0x7fc974153980 2023-10-30T13:22:56.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97c051640 1 -- 172.21.15.154:0/3083769571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9740cf080 con 0x7fc974153980 2023-10-30T13:22:56.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc96a7fc640 1 -- 172.21.15.154:0/3083769571 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc96401c440 con 0x7fc974153980 2023-10-30T13:22:56.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc96a7fc640 1 -- 172.21.15.154:0/3083769571 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fc964012890 con 0x7fc974153980 2023-10-30T13:22:56.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.132+0000 7fc97c051640 1 -- 172.21.15.154:0/3083769571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9740cdd70 con 0x7fc974153980 2023-10-30T13:22:56.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.136+0000 7fc96a7fc640 1 -- 172.21.15.154:0/3083769571 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fc9740cdd70 con 0x7fc974153980 2023-10-30T13:22:56.148 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:56.140+0000 7f8d267d9180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T13:22:56.160 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:56.156+0000 7f11d552b180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T13:22:56.205 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.200+0000 7fc97c051640 1 -- 172.21.15.154:0/3083769571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "58d4f3f8-7966-4175-a940-907e8561511d", "id": 3} v 0) v1 -- 0x7fc97415d050 con 0x7fc974153980 2023-10-30T13:22:56.211 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.208+0000 7fc96a7fc640 1 -- 172.21.15.154:0/3083769571 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "58d4f3f8-7966-4175-a940-907e8561511d", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc964021060 con 0x7fc974153980 2023-10-30T13:22:56.212 INFO:teuthology.orchestra.run.smithi154.stdout:3 2023-10-30T13:22:56.213 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.208+0000 7fc97c051640 1 -- 172.21.15.154:0/3083769571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc974153980 msgr2=0x7fc9740b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.213 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.208+0000 7fc97c051640 1 --2- 172.21.15.154:0/3083769571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc974153980 0x7fc9740b4330 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc964001ec0 tx=0x7fc96400c820 comp rx=0 tx=0).stop 2023-10-30T13:22:56.213 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.208+0000 7fc97c051640 1 -- 172.21.15.154:0/3083769571 shutdown_connections 2023-10-30T13:22:56.214 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.208+0000 7fc97c051640 1 --2- 172.21.15.154:0/3083769571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9740ce760 0x7fc97414bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.214 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.208+0000 7fc97c051640 1 --2- 172.21.15.154:0/3083769571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc974153980 0x7fc9740b4330 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.214 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.208+0000 7fc97c051640 1 --2- 172.21.15.154:0/3083769571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9740cb1e0 0x7fc97414b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.214 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.208+0000 7fc97c051640 1 -- 172.21.15.154:0/3083769571 >> 172.21.15.154:0/3083769571 conn(0x7fc97406c0f0 msgr2=0x7fc97415c320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:56.214 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.208+0000 7fc97c051640 1 -- 172.21.15.154:0/3083769571 shutdown_connections 2023-10-30T13:22:56.215 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.208+0000 7fc97c051640 1 -- 172.21.15.154:0/3083769571 wait complete. 2023-10-30T13:22:56.222 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:56.220+0000 7f11d552b180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T13:22:56.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 66d8f937-0303-4a2c-a493-54991edb7f32 4 2023-10-30T13:22:56.240 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:56.232+0000 7f8d267d9180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T13:22:56.303 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:56.296+0000 7f8d267d9180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T13:22:56.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.340+0000 7f66788b1640 1 Processor -- start 2023-10-30T13:22:56.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.340+0000 7f66788b1640 1 -- start start 2023-10-30T13:22:56.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f66741519c0 0x7f667406d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c8470 0x7f66740c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667415ae60 con 0x7f66740c7300 2023-10-30T13:22:56.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667406a030 con 0x7f66741519c0 2023-10-30T13:22:56.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66740c6a60 con 0x7f66740c8470 2023-10-30T13:22:56.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f6671d9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c8470 0x7f66740c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.348 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f6672d9d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f66740c7300 0x7f667406a450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:46872/0 (socket says 172.21.15.154:46872) 2023-10-30T13:22:56.348 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f6672d9d640 1 -- 172.21.15.154:0/731772726 learned_addr learned my addr 172.21.15.154:0/731772726 (peer_addr_for_me v1:172.21.15.154:0/0) 2023-10-30T13:22:56.348 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f667259c640 1 --2- 172.21.15.154:0/731772726 >> v2:172.21.15.46:3300/0 conn(0x7f66741519c0 0x7f667406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.348 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f6671d9b640 1 -- 172.21.15.154:0/731772726 >> v2:172.21.15.46:3300/0 conn(0x7f66741519c0 msgr2=0x7f667406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:56.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f6671d9b640 1 --2- 172.21.15.154:0/731772726 >> v2:172.21.15.46:3300/0 conn(0x7f66741519c0 0x7f667406d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f6671d9b640 1 -- 172.21.15.154:0/731772726 >> v1:172.21.15.154:6789/0 conn(0x7f66740c7300 legacy=0x7f667406a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f667159a640 1 -- 172.21.15.154:0/731772726 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895339407 0 0) 0x7f667415ae60 con 0x7f66740c7300 2023-10-30T13:22:56.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f6671d9b640 1 -- 172.21.15.154:0/731772726 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667415b140 con 0x7f66740c8470 2023-10-30T13:22:56.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f6671d9b640 1 --2- 172.21.15.154:0/731772726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c8470 0x7f66740c6dc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f66680097c0 tx=0x7f666801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc56c110d5e2504b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:56.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f667159a640 1 -- 172.21.15.154:0/731772726 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66680ad640 con 0x7f66740c8470 2023-10-30T13:22:56.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f667159a640 1 -- 172.21.15.154:0/731772726 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66680ad7a0 con 0x7f66740c8470 2023-10-30T13:22:56.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f667159a640 1 -- 172.21.15.154:0/731772726 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66680adb30 con 0x7f66740c8470 2023-10-30T13:22:56.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 -- 172.21.15.154:0/731772726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c8470 msgr2=0x7f66740c6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 --2- 172.21.15.154:0/731772726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c8470 0x7f66740c6dc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f66680097c0 tx=0x7f666801c480 comp rx=0 tx=0).stop 2023-10-30T13:22:56.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 -- 172.21.15.154:0/731772726 shutdown_connections 2023-10-30T13:22:56.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 --2- 172.21.15.154:0/731772726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c8470 0x7f66740c6dc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 --2- 172.21.15.154:0/731772726 >> v2:172.21.15.46:3300/0 conn(0x7f66741519c0 0x7f667406d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 -- 172.21.15.154:0/731772726 >> 172.21.15.154:0/731772726 conn(0x7f667406c850 msgr2=0x7f667406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:56.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 -- 172.21.15.154:0/731772726 shutdown_connections 2023-10-30T13:22:56.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.344+0000 7f66788b1640 1 -- 172.21.15.154:0/731772726 wait complete. 2023-10-30T13:22:56.356 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f66788b1640 1 Processor -- start 2023-10-30T13:22:56.356 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f66788b1640 1 -- start start 2023-10-30T13:22:56.356 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f66788b1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c7300 0x7f66740c1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.356 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f66788b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66740c8470 0x7f66741613f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.356 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f66788b1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66741519c0 0x7f667414d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f66788b1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667415a9c0 con 0x7f66740c8470 2023-10-30T13:22:56.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f66788b1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66741f2d70 con 0x7f66741519c0 2023-10-30T13:22:56.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f66788b1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6674161ca0 con 0x7f66740c7300 2023-10-30T13:22:56.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f667259c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c7300 0x7f66740c1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f6672d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66741519c0 0x7f667414d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f667259c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c7300 0x7f66740c1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:54132/0 (socket says 172.21.15.154:54132) 2023-10-30T13:22:56.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f667259c640 1 -- 172.21.15.154:0/1245322718 learned_addr learned my addr 172.21.15.154:0/1245322718 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-10-30T13:22:56.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f6671d9b640 1 --2- 172.21.15.154:0/1245322718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66740c8470 0x7f66741613f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f667259c640 1 -- 172.21.15.154:0/1245322718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66741519c0 msgr2=0x7f667414d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f667259c640 1 --2- 172.21.15.154:0/1245322718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66741519c0 0x7f667414d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f667259c640 1 -- 172.21.15.154:0/1245322718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66740c8470 msgr2=0x7f66741613f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f667259c640 1 --2- 172.21.15.154:0/1245322718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66740c8470 0x7f66741613f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f667259c640 1 -- 172.21.15.154:0/1245322718 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6668009420 con 0x7f66740c7300 2023-10-30T13:22:56.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f6672d9d640 1 --2- 172.21.15.154:0/1245322718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66741519c0 0x7f667414d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:56.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f6671d9b640 1 --2- 172.21.15.154:0/1245322718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66740c8470 0x7f66741613f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:56.360 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f667259c640 1 --2- 172.21.15.154:0/1245322718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c7300 0x7f66740c1800 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f666400d5f0 tx=0x7f666400dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:56.360 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f6662ffd640 1 -- 172.21.15.154:0/1245322718 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66640125f0 con 0x7f66740c7300 2023-10-30T13:22:56.360 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f6662ffd640 1 -- 172.21.15.154:0/1245322718 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6664012750 con 0x7f66740c7300 2023-10-30T13:22:56.360 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f66788b1640 1 -- 172.21.15.154:0/1245322718 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66741f20f0 con 0x7f66740c7300 2023-10-30T13:22:56.360 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f6662ffd640 1 -- 172.21.15.154:0/1245322718 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6664012ae0 con 0x7f66740c7300 2023-10-30T13:22:56.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f66788b1640 1 -- 172.21.15.154:0/1245322718 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66741f26b0 con 0x7f66740c7300 2023-10-30T13:22:56.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f6662ffd640 1 -- 172.21.15.154:0/1245322718 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f666401e460 con 0x7f66740c7300 2023-10-30T13:22:56.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f6662ffd640 1 -- 172.21.15.154:0/1245322718 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f6664014990 con 0x7f66740c7300 2023-10-30T13:22:56.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.348+0000 7f66788b1640 1 -- 172.21.15.154:0/1245322718 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667404e170 con 0x7f66740c7300 2023-10-30T13:22:56.362 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.352+0000 7f6662ffd640 1 -- 172.21.15.154:0/1245322718 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f6664014c70 con 0x7f66740c7300 2023-10-30T13:22:56.395 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:56.388+0000 7f8d267d9180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T13:22:56.421 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.416+0000 7f66788b1640 1 -- 172.21.15.154:0/1245322718 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd new", "uuid": "66d8f937-0303-4a2c-a493-54991edb7f32", "id": 4} v 0) v1 -- 0x7f66740b1a00 con 0x7f66740c7300 2023-10-30T13:22:56.425 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.420+0000 7f6662ffd640 1 -- 172.21.15.154:0/1245322718 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "66d8f937-0303-4a2c-a493-54991edb7f32", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6664014480 con 0x7f66740c7300 2023-10-30T13:22:56.425 INFO:teuthology.orchestra.run.smithi154.stdout:4 2023-10-30T13:22:56.427 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.424+0000 7f66788b1640 1 -- 172.21.15.154:0/1245322718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c7300 msgr2=0x7f66740c1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.427 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.424+0000 7f66788b1640 1 --2- 172.21.15.154:0/1245322718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c7300 0x7f66740c1800 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f666400d5f0 tx=0x7f666400dab0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.427 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.424+0000 7f66788b1640 1 -- 172.21.15.154:0/1245322718 shutdown_connections 2023-10-30T13:22:56.427 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.424+0000 7f66788b1640 1 --2- 172.21.15.154:0/1245322718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66741519c0 0x7f667414d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.427 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.424+0000 7f66788b1640 1 --2- 172.21.15.154:0/1245322718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66740c8470 0x7f66741613f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.428 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.424+0000 7f66788b1640 1 --2- 172.21.15.154:0/1245322718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66740c7300 0x7f66740c1800 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.428 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.424+0000 7f66788b1640 1 -- 172.21.15.154:0/1245322718 >> 172.21.15.154:0/1245322718 conn(0x7f667406c850 msgr2=0x7f6674149f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:56.428 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.424+0000 7f66788b1640 1 -- 172.21.15.154:0/1245322718 shutdown_connections 2023-10-30T13:22:56.428 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.424+0000 7f66788b1640 1 -- 172.21.15.154:0/1245322718 wait complete. 2023-10-30T13:22:56.441 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 56cc0fc0-d1d1-4326-9b45-b806a903dd6d 5 2023-10-30T13:22:56.457 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:56.452+0000 7f8d267d9180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T13:22:56.591 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497fa9f640 1 Processor -- start 2023-10-30T13:22:56.592 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497fa9f640 1 -- start start 2023-10-30T13:22:56.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497fa9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49780c79a0 0x7f49780c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497fa9f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497806e360 con 0x7f497806e570 2023-10-30T13:22:56.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497fa9f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49780c7480 con 0x7f4978151900 2023-10-30T13:22:56.594 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497fa9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497806eab0 con 0x7f49780c79a0 2023-10-30T13:22:56.594 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497d83b640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f4978151900 0x7f497806d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:46874/0 (socket says 172.21.15.154:46874) 2023-10-30T13:22:56.594 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497d83b640 1 -- 172.21.15.154:0/2371894076 learned_addr learned my addr 172.21.15.154:0/2371894076 (peer_addr_for_me v1:172.21.15.154:0/0) 2023-10-30T13:22:56.594 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497e03c640 1 --2- 172.21.15.154:0/2371894076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49780c79a0 0x7f49780c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.594 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497c839640 1 -- 172.21.15.154:0/2371894076 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884640632 0 0) 0x7f49780c7480 con 0x7f4978151900 2023-10-30T13:22:56.595 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497e03c640 1 -- 172.21.15.154:0/2371894076 >> v1:172.21.15.154:6789/0 conn(0x7f4978151900 legacy=0x7f497806d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.595 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497e03c640 1 -- 172.21.15.154:0/2371894076 >> v1:172.21.15.46:6789/0 conn(0x7f497806e570 legacy=0x7f497806f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:56.595 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.588+0000 7f497e03c640 1 -- 172.21.15.154:0/2371894076 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4978158320 con 0x7f49780c79a0 2023-10-30T13:22:56.597 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497e03c640 1 --2- 172.21.15.154:0/2371894076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49780c79a0 0x7f49780c22f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4974009dc0 tx=0x7f497401c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bc8a720f587f9c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:56.597 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497c839640 1 -- 172.21.15.154:0/2371894076 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49740a5890 con 0x7f49780c79a0 2023-10-30T13:22:56.597 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497c839640 1 -- 172.21.15.154:0/2371894076 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49740a59f0 con 0x7f49780c79a0 2023-10-30T13:22:56.597 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497c839640 1 -- 172.21.15.154:0/2371894076 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49740a5d80 con 0x7f49780c79a0 2023-10-30T13:22:56.598 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497fa9f640 1 -- 172.21.15.154:0/2371894076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49780c79a0 msgr2=0x7f49780c22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.598 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497fa9f640 1 --2- 172.21.15.154:0/2371894076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49780c79a0 0x7f49780c22f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4974009dc0 tx=0x7f497401c900 comp rx=0 tx=0).stop 2023-10-30T13:22:56.598 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497fa9f640 1 -- 172.21.15.154:0/2371894076 shutdown_connections 2023-10-30T13:22:56.598 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497fa9f640 1 --2- 172.21.15.154:0/2371894076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49780c79a0 0x7f49780c22f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.598 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497fa9f640 1 -- 172.21.15.154:0/2371894076 >> 172.21.15.154:0/2371894076 conn(0x7f49780ccfe0 msgr2=0x7f49780cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:56.599 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497fa9f640 1 -- 172.21.15.154:0/2371894076 shutdown_connections 2023-10-30T13:22:56.599 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497fa9f640 1 -- 172.21.15.154:0/2371894076 wait complete. 2023-10-30T13:22:56.599 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497fa9f640 1 Processor -- start 2023-10-30T13:22:56.599 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.592+0000 7f497fa9f640 1 -- start start 2023-10-30T13:22:56.602 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497fa9f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f497806e570 0x7f497814cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.602 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497fa9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49780c79a0 0x7f49781576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.602 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497fa9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4978151900 0x7f497814e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.602 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497fa9f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49781502a0 con 0x7f49780c79a0 2023-10-30T13:22:56.603 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497fa9f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497814e0e0 con 0x7f497806e570 2023-10-30T13:22:56.603 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497fa9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497806b9b0 con 0x7f4978151900 2023-10-30T13:22:56.603 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497d83b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f497806e570 0x7f497814cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.603 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497e03c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4978151900 0x7f497814e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.603 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497d83b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f497806e570 0x7f497814cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:58092/0 (socket says 172.21.15.154:58092) 2023-10-30T13:22:56.604 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497d83b640 1 -- 172.21.15.154:0/2666433088 learned_addr learned my addr 172.21.15.154:0/2666433088 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-10-30T13:22:56.604 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497d03a640 1 --2- 172.21.15.154:0/2666433088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49780c79a0 0x7f49781576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.604 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497d83b640 1 -- 172.21.15.154:0/2666433088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4978151900 msgr2=0x7f497814e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.604 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497d83b640 1 --2- 172.21.15.154:0/2666433088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4978151900 0x7f497814e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.604 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497d83b640 1 -- 172.21.15.154:0/2666433088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49780c79a0 msgr2=0x7f49781576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.605 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497d83b640 1 --2- 172.21.15.154:0/2666433088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49780c79a0 0x7f49781576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.605 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497d83b640 1 -- 172.21.15.154:0/2666433088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4974009a70 con 0x7f497806e570 2023-10-30T13:22:56.605 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497e03c640 1 --2- 172.21.15.154:0/2666433088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4978151900 0x7f497814e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:56.605 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497d03a640 1 --2- 172.21.15.154:0/2666433088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49780c79a0 0x7f49781576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:56.605 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497d83b640 1 --2- 172.21.15.154:0/2666433088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f497806e570 0x7f497814cd50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f497000c4b0 tx=0x7f497000c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:56.606 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f496e7fc640 1 -- 172.21.15.154:0/2666433088 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49700074e0 con 0x7f497806e570 2023-10-30T13:22:56.606 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f496e7fc640 1 -- 172.21.15.154:0/2666433088 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4970007660 con 0x7f497806e570 2023-10-30T13:22:56.606 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f496e7fc640 1 -- 172.21.15.154:0/2666433088 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4970004a30 con 0x7f497806e570 2023-10-30T13:22:56.606 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497fa9f640 1 -- 172.21.15.154:0/2666433088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4978151e60 con 0x7f497806e570 2023-10-30T13:22:56.606 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497fa9f640 1 -- 172.21.15.154:0/2666433088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497806c250 con 0x7f497806e570 2023-10-30T13:22:56.607 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f496e7fc640 1 -- 172.21.15.154:0/2666433088 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4970012400 con 0x7f497806e570 2023-10-30T13:22:56.607 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f496e7fc640 1 -- 172.21.15.154:0/2666433088 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f497001b8e0 con 0x7f497806e570 2023-10-30T13:22:56.607 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.596+0000 7f497fa9f640 1 -- 172.21.15.154:0/2666433088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497804e170 con 0x7f497806e570 2023-10-30T13:22:56.607 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.600+0000 7f496e7fc640 1 -- 172.21.15.154:0/2666433088 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f4970018070 con 0x7f497806e570 2023-10-30T13:22:56.621 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:56.612+0000 7f8d267d9180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T13:22:56.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.664+0000 7f497fa9f640 1 -- 172.21.15.154:0/2666433088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "56cc0fc0-d1d1-4326-9b45-b806a903dd6d", "id": 5} v 0) v1 -- 0x7f49780cc1b0 con 0x7f497806e570 2023-10-30T13:22:56.675 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.672+0000 7f496e7fc640 1 -- 172.21.15.154:0/2666433088 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "56cc0fc0-d1d1-4326-9b45-b806a903dd6d", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f497001bc50 con 0x7f497806e570 2023-10-30T13:22:56.675 INFO:teuthology.orchestra.run.smithi154.stdout:5 2023-10-30T13:22:56.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.672+0000 7f497fa9f640 1 -- 172.21.15.154:0/2666433088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f497806e570 msgr2=0x7f497814cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.672+0000 7f497fa9f640 1 --2- 172.21.15.154:0/2666433088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f497806e570 0x7f497814cd50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f497000c4b0 tx=0x7f497000c970 comp rx=0 tx=0).stop 2023-10-30T13:22:56.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.672+0000 7f497fa9f640 1 -- 172.21.15.154:0/2666433088 shutdown_connections 2023-10-30T13:22:56.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.672+0000 7f497fa9f640 1 --2- 172.21.15.154:0/2666433088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4978151900 0x7f497814e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.678 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.672+0000 7f497fa9f640 1 --2- 172.21.15.154:0/2666433088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49780c79a0 0x7f49781576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.678 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.672+0000 7f497fa9f640 1 --2- 172.21.15.154:0/2666433088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f497806e570 0x7f497814cd50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.678 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.672+0000 7f497fa9f640 1 -- 172.21.15.154:0/2666433088 >> 172.21.15.154:0/2666433088 conn(0x7f49780ccfe0 msgr2=0x7f49780c8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:56.678 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.672+0000 7f497fa9f640 1 -- 172.21.15.154:0/2666433088 shutdown_connections 2023-10-30T13:22:56.678 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.672+0000 7f497fa9f640 1 -- 172.21.15.154:0/2666433088 wait complete. 2023-10-30T13:22:56.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new b8f618d5-d080-47ef-b94c-2dbdaba5dfd1 6 2023-10-30T13:22:56.692 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:56.684+0000 7f8d267d9180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T13:22:56.804 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.800+0000 7f6f8b811640 1 Processor -- start 2023-10-30T13:22:56.805 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.800+0000 7f6f8b811640 1 -- start start 2023-10-30T13:22:56.806 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.800+0000 7f6f8b811640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 0x7f6f840c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.806 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.800+0000 7f6f8b811640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f84158560 con 0x7f6f84065ea0 2023-10-30T13:22:56.807 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f8b811640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f840c36c0 con 0x7f6f84151800 2023-10-30T13:22:56.807 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f8b811640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f840c6ae0 con 0x7f6f840c2d30 2023-10-30T13:22:56.807 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f89dae640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 0x7f6f840c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.807 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f895ad640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f6f84151800 0x7f6f840c5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:46876/0 (socket says 172.21.15.154:46876) 2023-10-30T13:22:56.808 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f895ad640 1 -- 172.21.15.154:0/2601396366 learned_addr learned my addr 172.21.15.154:0/2601396366 (peer_addr_for_me v1:172.21.15.154:0/0) 2023-10-30T13:22:56.808 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f7bfff640 1 -- 172.21.15.154:0/2601396366 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268540955 0 0) 0x7f6f840c36c0 con 0x7f6f84151800 2023-10-30T13:22:56.808 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f89dae640 1 -- 172.21.15.154:0/2601396366 >> v1:172.21.15.154:6789/0 conn(0x7f6f84151800 legacy=0x7f6f840c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.808 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f89dae640 1 -- 172.21.15.154:0/2601396366 >> v1:172.21.15.46:6789/0 conn(0x7f6f84065ea0 legacy=0x7f6f840cb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:56.809 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f89dae640 1 -- 172.21.15.154:0/2601396366 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f84157670 con 0x7f6f840c2d30 2023-10-30T13:22:56.810 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f89dae640 1 --2- 172.21.15.154:0/2601396366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 0x7f6f840c3180 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009dc0 tx=0x7f6f7c01f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c32c4e7116f5d85a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:56.810 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f7bfff640 1 -- 172.21.15.154:0/2601396366 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c0af600 con 0x7f6f840c2d30 2023-10-30T13:22:56.811 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f7bfff640 1 -- 172.21.15.154:0/2601396366 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c0ab050 con 0x7f6f840c2d30 2023-10-30T13:22:56.811 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f7bfff640 1 -- 172.21.15.154:0/2601396366 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c0af980 con 0x7f6f840c2d30 2023-10-30T13:22:56.811 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f8b811640 1 -- 172.21.15.154:0/2601396366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 msgr2=0x7f6f840c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.811 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f8b811640 1 --2- 172.21.15.154:0/2601396366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 0x7f6f840c3180 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009dc0 tx=0x7f6f7c01f150 comp rx=0 tx=0).stop 2023-10-30T13:22:56.812 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f8b811640 1 -- 172.21.15.154:0/2601396366 shutdown_connections 2023-10-30T13:22:56.812 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f8b811640 1 --2- 172.21.15.154:0/2601396366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 0x7f6f840c3180 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.812 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f8b811640 1 -- 172.21.15.154:0/2601396366 >> 172.21.15.154:0/2601396366 conn(0x7f6f8406c620 msgr2=0x7f6f84069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:56.812 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f8b811640 1 -- 172.21.15.154:0/2601396366 shutdown_connections 2023-10-30T13:22:56.812 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.804+0000 7f6f8b811640 1 -- 172.21.15.154:0/2601396366 wait complete. 2023-10-30T13:22:56.813 INFO:tasks.ceph.mgr.x.smithi154.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-30T13:22:56.813 INFO:tasks.ceph.mgr.x.smithi154.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-30T13:22:56.813 INFO:tasks.ceph.mgr.x.smithi154.stderr: from numpy import show_config as show_numpy_config 2023-10-30T13:22:56.816 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f8b811640 1 Processor -- start 2023-10-30T13:22:56.816 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f8b811640 1 -- start start 2023-10-30T13:22:56.817 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f8b811640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f84065ea0 0x7f6f84158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.817 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f8b811640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 0x7f6f8414a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.817 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f8b811640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f84151800 0x7f6f8414d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.817 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f8b811640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f841efd60 con 0x7f6f84065ea0 2023-10-30T13:22:56.817 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f8b811640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f841521b0 con 0x7f6f84151800 2023-10-30T13:22:56.818 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f8b811640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f84152490 con 0x7f6f840c2d30 2023-10-30T13:22:56.818 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f88dac640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 0x7f6f8414a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.818 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f89dae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f84151800 0x7f6f8414d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.818 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f88dac640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 0x7f6f8414a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:54162/0 (socket says 172.21.15.154:54162) 2023-10-30T13:22:56.819 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f88dac640 1 -- 172.21.15.154:0/3672076935 learned_addr learned my addr 172.21.15.154:0/3672076935 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-10-30T13:22:56.819 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f89dae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f84151800 0x7f6f8414d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:58104/0 (socket says 172.21.15.154:58104) 2023-10-30T13:22:56.819 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f895ad640 1 --2- 172.21.15.154:0/3672076935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f84065ea0 0x7f6f84158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.819 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f88dac640 1 -- 172.21.15.154:0/3672076935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f84151800 msgr2=0x7f6f8414d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.820 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f88dac640 1 --2- 172.21.15.154:0/3672076935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f84151800 0x7f6f8414d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.820 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f88dac640 1 -- 172.21.15.154:0/3672076935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f84065ea0 msgr2=0x7f6f84158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.820 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f88dac640 1 --2- 172.21.15.154:0/3672076935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f84065ea0 0x7f6f84158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.820 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f88dac640 1 -- 172.21.15.154:0/3672076935 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c009a70 con 0x7f6f840c2d30 2023-10-30T13:22:56.820 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f88dac640 1 --2- 172.21.15.154:0/3672076935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 0x7f6f8414a260 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6f7400c3c0 tx=0x7f6f7400c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:56.821 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f79ffb640 1 -- 172.21.15.154:0/3672076935 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f740052d0 con 0x7f6f840c2d30 2023-10-30T13:22:56.821 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f79ffb640 1 -- 172.21.15.154:0/3672076935 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f74005430 con 0x7f6f840c2d30 2023-10-30T13:22:56.821 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f8b811640 1 -- 172.21.15.154:0/3672076935 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f841effb0 con 0x7f6f840c2d30 2023-10-30T13:22:56.821 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f79ffb640 1 -- 172.21.15.154:0/3672076935 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f74003840 con 0x7f6f840c2d30 2023-10-30T13:22:56.822 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f8b811640 1 -- 172.21.15.154:0/3672076935 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f841f0570 con 0x7f6f840c2d30 2023-10-30T13:22:56.822 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f79ffb640 1 -- 172.21.15.154:0/3672076935 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6f74003ca0 con 0x7f6f840c2d30 2023-10-30T13:22:56.822 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f79ffb640 1 -- 172.21.15.154:0/3672076935 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f6f74011ea0 con 0x7f6f840c2d30 2023-10-30T13:22:56.822 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.808+0000 7f6f8b811640 1 -- 172.21.15.154:0/3672076935 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8404e170 con 0x7f6f840c2d30 2023-10-30T13:22:56.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.812+0000 7f6f79ffb640 1 -- 172.21.15.154:0/3672076935 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f6f74017070 con 0x7f6f840c2d30 2023-10-30T13:22:56.858 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:56.856+0000 7f11d552b180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T13:22:56.876 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.872+0000 7f6f8b811640 1 -- 172.21.15.154:0/3672076935 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b8f618d5-d080-47ef-b94c-2dbdaba5dfd1", "id": 6} v 0) v1 -- 0x7f6f840c6730 con 0x7f6f840c2d30 2023-10-30T13:22:56.881 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.876+0000 7f6f79ffb640 1 -- 172.21.15.154:0/3672076935 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b8f618d5-d080-47ef-b94c-2dbdaba5dfd1", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6f84158560 con 0x7f6f840c2d30 2023-10-30T13:22:56.881 INFO:teuthology.orchestra.run.smithi154.stdout:6 2023-10-30T13:22:56.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.880+0000 7f6f8b811640 1 -- 172.21.15.154:0/3672076935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 msgr2=0x7f6f8414a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.880+0000 7f6f8b811640 1 --2- 172.21.15.154:0/3672076935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 0x7f6f8414a260 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6f7400c3c0 tx=0x7f6f7400c880 comp rx=0 tx=0).stop 2023-10-30T13:22:56.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.880+0000 7f6f8b811640 1 -- 172.21.15.154:0/3672076935 shutdown_connections 2023-10-30T13:22:56.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.880+0000 7f6f8b811640 1 --2- 172.21.15.154:0/3672076935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f84151800 0x7f6f8414d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.880+0000 7f6f8b811640 1 --2- 172.21.15.154:0/3672076935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f840c2d30 0x7f6f8414a260 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.880+0000 7f6f8b811640 1 --2- 172.21.15.154:0/3672076935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f84065ea0 0x7f6f84158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.880+0000 7f6f8b811640 1 -- 172.21.15.154:0/3672076935 >> 172.21.15.154:0/3672076935 conn(0x7f6f8406c620 msgr2=0x7f6f840cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:56.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.880+0000 7f6f8b811640 1 -- 172.21.15.154:0/3672076935 shutdown_connections 2023-10-30T13:22:56.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:56.880+0000 7f6f8b811640 1 -- 172.21.15.154:0/3672076935 wait complete. 2023-10-30T13:22:56.896 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new dfcbd9f0-0ca4-4068-b367-021944895fc2 7 2023-10-30T13:22:56.985 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:56.984+0000 7f11d552b180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T13:22:57.008 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.004+0000 7f0bbd609640 1 Processor -- start 2023-10-30T13:22:57.008 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.004+0000 7f0bbd609640 1 -- start start 2023-10-30T13:22:57.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.004+0000 7f0bbd609640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0bb80cb770 0x7f0bb80c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:57.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.004+0000 7f0bbd609640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0bb806fe90 0x7f0bb8153390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:57.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.004+0000 7f0bbd609640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb8159940 con 0x7f0bb80cbb50 2023-10-30T13:22:57.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.004+0000 7f0bbd609640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb8159750 con 0x7f0bb806fe90 2023-10-30T13:22:57.011 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.004+0000 7f0bbd609640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb8159c20 con 0x7f0bb80cb770 2023-10-30T13:22:57.011 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.004+0000 7f0bb6ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0bb80cb770 0x7f0bb80c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:57.011 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.004+0000 7f0bb6ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0bb80cb770 0x7f0bb80c5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:58112/0 (socket says 172.21.15.154:58112) 2023-10-30T13:22:57.011 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.004+0000 7f0baffff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f0bb80cbb50 0x7f0bb815ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:46886/0 (socket says 172.21.15.154:46886) 2023-10-30T13:22:57.012 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.004+0000 7f0bb6ffd640 1 -- 172.21.15.154:0/3081054797 learned_addr learned my addr 172.21.15.154:0/3081054797 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-10-30T13:22:57.012 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.004+0000 7f0bb77fe640 1 --2- 172.21.15.154:0/3081054797 >> v2:172.21.15.46:3300/0 conn(0x7f0bb806fe90 0x7f0bb8153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:57.012 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bb6ffd640 1 -- 172.21.15.154:0/3081054797 >> v2:172.21.15.46:3300/0 conn(0x7f0bb806fe90 msgr2=0x7f0bb8153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:57.012 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bb6ffd640 1 --2- 172.21.15.154:0/3081054797 >> v2:172.21.15.46:3300/0 conn(0x7f0bb806fe90 0x7f0bb8153390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.012 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bb6ffd640 1 -- 172.21.15.154:0/3081054797 >> v1:172.21.15.154:6789/0 conn(0x7f0bb80cbb50 legacy=0x7f0bb815ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:57.013 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bb6ffd640 1 -- 172.21.15.154:0/3081054797 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb8159f00 con 0x7f0bb80cb770 2023-10-30T13:22:57.013 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bb67fc640 1 -- 172.21.15.154:0/3081054797 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481076621 0 0) 0x7f0bb8159940 con 0x7f0bb80cbb50 2023-10-30T13:22:57.016 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bb6ffd640 1 --2- 172.21.15.154:0/3081054797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bb80cb770 0x7f0bb80c5f40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80c7fa0 tx=0x7f0ba801ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3db96a6e3910120e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:57.017 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bb67fc640 1 -- 172.21.15.154:0/3081054797 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80a89f0 con 0x7f0bb80cb770 2023-10-30T13:22:57.017 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bb67fc640 1 -- 172.21.15.154:0/3081054797 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba80a8b50 con 0x7f0bb80cb770 2023-10-30T13:22:57.017 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bb67fc640 1 -- 172.21.15.154:0/3081054797 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80b0440 con 0x7f0bb80cb770 2023-10-30T13:22:57.017 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bbd609640 1 -- 172.21.15.154:0/3081054797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bb80cb770 msgr2=0x7f0bb80c5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:57.017 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bbd609640 1 --2- 172.21.15.154:0/3081054797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bb80cb770 0x7f0bb80c5f40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80c7fa0 tx=0x7f0ba801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.018 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bbd609640 1 -- 172.21.15.154:0/3081054797 shutdown_connections 2023-10-30T13:22:57.018 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bbd609640 1 --2- 172.21.15.154:0/3081054797 >> v2:172.21.15.46:3300/0 conn(0x7f0bb806fe90 0x7f0bb8153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.018 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bbd609640 1 --2- 172.21.15.154:0/3081054797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bb80cb770 0x7f0bb80c5f40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.019 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bbd609640 1 -- 172.21.15.154:0/3081054797 >> 172.21.15.154:0/3081054797 conn(0x7f0bb806c620 msgr2=0x7f0bb8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:57.019 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bbd609640 1 -- 172.21.15.154:0/3081054797 shutdown_connections 2023-10-30T13:22:57.019 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.008+0000 7f0bbd609640 1 -- 172.21.15.154:0/3081054797 wait complete. 2023-10-30T13:22:57.019 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bbd609640 1 Processor -- start 2023-10-30T13:22:57.019 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bbd609640 1 -- start start 2023-10-30T13:22:57.020 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bbd609640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb806fe90 0x7f0bb81563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:57.020 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bbd609640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb80cb770 0x7f0bb80c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:57.020 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bbd609640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bb80cbb50 0x7f0bb80c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:57.020 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bbd609640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb80c3830 con 0x7f0bb806fe90 2023-10-30T13:22:57.020 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bbd609640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb80c5560 con 0x7f0bb80cbb50 2023-10-30T13:22:57.021 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bbd609640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb80c32b0 con 0x7f0bb80cb770 2023-10-30T13:22:57.021 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bb77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bb80cbb50 0x7f0bb80c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:57.021 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0baffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb80cb770 0x7f0bb80c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:57.021 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bb77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bb80cbb50 0x7f0bb80c4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:58114/0 (socket says 172.21.15.154:58114) 2023-10-30T13:22:57.021 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bb77fe640 1 -- 172.21.15.154:0/116191251 learned_addr learned my addr 172.21.15.154:0/116191251 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-10-30T13:22:57.022 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bb6ffd640 1 --2- 172.21.15.154:0/116191251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb806fe90 0x7f0bb81563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:57.022 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bb77fe640 1 -- 172.21.15.154:0/116191251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb80cb770 msgr2=0x7f0bb80c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:57.022 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bb77fe640 1 --2- 172.21.15.154:0/116191251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb80cb770 0x7f0bb80c46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.022 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bb77fe640 1 -- 172.21.15.154:0/116191251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb806fe90 msgr2=0x7f0bb81563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:57.022 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bb77fe640 1 --2- 172.21.15.154:0/116191251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb806fe90 0x7f0bb81563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.023 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bb77fe640 1 -- 172.21.15.154:0/116191251 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba80094d0 con 0x7f0bb80cbb50 2023-10-30T13:22:57.023 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bb6ffd640 1 --2- 172.21.15.154:0/116191251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb806fe90 0x7f0bb81563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:57.023 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bb77fe640 1 --2- 172.21.15.154:0/116191251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bb80cbb50 0x7f0bb80c4c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0007e70 tx=0x7f0bb000e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:57.023 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0baf7fe640 1 -- 172.21.15.154:0/116191251 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0014620 con 0x7f0bb80cbb50 2023-10-30T13:22:57.023 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0baf7fe640 1 -- 172.21.15.154:0/116191251 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb00147a0 con 0x7f0bb80cbb50 2023-10-30T13:22:57.023 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0baf7fe640 1 -- 172.21.15.154:0/116191251 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0014b50 con 0x7f0bb80cbb50 2023-10-30T13:22:57.024 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bbd609640 1 -- 172.21.15.154:0/116191251 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb80c3550 con 0x7f0bb80cbb50 2023-10-30T13:22:57.024 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bbd609640 1 -- 172.21.15.154:0/116191251 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb81544a0 con 0x7f0bb80cbb50 2023-10-30T13:22:57.024 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0baf7fe640 1 -- 172.21.15.154:0/116191251 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0bb001e460 con 0x7f0bb80cbb50 2023-10-30T13:22:57.024 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0baf7fe640 1 -- 172.21.15.154:0/116191251 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f0bb001da20 con 0x7f0bb80cbb50 2023-10-30T13:22:57.025 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.012+0000 7f0bbd609640 1 -- 172.21.15.154:0/116191251 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb804e170 con 0x7f0bb80cbb50 2023-10-30T13:22:57.025 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.016+0000 7f0baf7fe640 1 -- 172.21.15.154:0/116191251 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f0bb001dd20 con 0x7f0bb80cbb50 2023-10-30T13:22:57.134 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.132+0000 7f0bbd609640 1 -- 172.21.15.154:0/116191251 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dfcbd9f0-0ca4-4068-b367-021944895fc2", "id": 7} v 0) v1 -- 0x7f0bb8150bd0 con 0x7f0bb80cbb50 2023-10-30T13:22:57.138 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.136+0000 7f0baf7fe640 1 -- 172.21.15.154:0/116191251 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dfcbd9f0-0ca4-4068-b367-021944895fc2", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0bb002d0b0 con 0x7f0bb80cbb50 2023-10-30T13:22:57.139 INFO:teuthology.orchestra.run.smithi154.stdout:7 2023-10-30T13:22:57.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.136+0000 7f0bbd609640 1 -- 172.21.15.154:0/116191251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bb80cbb50 msgr2=0x7f0bb80c4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:57.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.136+0000 7f0bbd609640 1 --2- 172.21.15.154:0/116191251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bb80cbb50 0x7f0bb80c4c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0007e70 tx=0x7f0bb000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.136+0000 7f0bbd609640 1 -- 172.21.15.154:0/116191251 shutdown_connections 2023-10-30T13:22:57.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.136+0000 7f0bbd609640 1 --2- 172.21.15.154:0/116191251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bb80cbb50 0x7f0bb80c4c30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.136+0000 7f0bbd609640 1 --2- 172.21.15.154:0/116191251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb80cb770 0x7f0bb80c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.136+0000 7f0bbd609640 1 --2- 172.21.15.154:0/116191251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb806fe90 0x7f0bb81563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.136+0000 7f0bbd609640 1 -- 172.21.15.154:0/116191251 >> 172.21.15.154:0/116191251 conn(0x7f0bb806c620 msgr2=0x7f0bb80ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:57.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.136+0000 7f0bbd609640 1 -- 172.21.15.154:0/116191251 shutdown_connections 2023-10-30T13:22:57.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-10-30T13:22:57.136+0000 7f0bbd609640 1 -- 172.21.15.154:0/116191251 wait complete. 2023-10-30T13:22:57.155 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-30T13:22:57.155 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-30T13:22:57.159 INFO:tasks.ceph.osd.0:Started 2023-10-30T13:22:57.159 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-30T13:22:57.159 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-30T13:22:57.163 INFO:tasks.ceph.osd.1:Started 2023-10-30T13:22:57.163 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-30T13:22:57.163 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-30T13:22:57.167 INFO:tasks.ceph.osd.2:Started 2023-10-30T13:22:57.167 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-30T13:22:57.168 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-30T13:22:57.171 INFO:tasks.ceph.osd.3:Started 2023-10-30T13:22:57.171 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-30T13:22:57.172 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-30T13:22:57.174 INFO:tasks.ceph.osd.4:Started 2023-10-30T13:22:57.174 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-30T13:22:57.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-30T13:22:57.175 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:57.172+0000 7f11d552b180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T13:22:57.179 INFO:tasks.ceph.osd.5:Started 2023-10-30T13:22:57.179 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-30T13:22:57.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-30T13:22:57.182 INFO:tasks.ceph.osd.6:Started 2023-10-30T13:22:57.182 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-30T13:22:57.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-30T13:22:57.185 INFO:tasks.ceph.osd.7:Started 2023-10-30T13:22:57.185 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:22:57.264 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:57.260+0000 7f11d552b180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T13:22:57.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.368+0000 7fb20a42f640 1 Processor -- start 2023-10-30T13:22:57.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.368+0000 7fb20a42f640 1 -- start start 2023-10-30T13:22:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.368+0000 7fb20a42f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2041538f0 0x7fb204159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.368+0000 7fb20a42f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb20415cd50 0x7fb2040c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.368+0000 7fb20a42f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20415d300 con 0x7fb2040cefa0 2023-10-30T13:22:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.368+0000 7fb20a42f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb20415d5e0 con 0x7fb2041538f0 2023-10-30T13:22:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.368+0000 7fb20a42f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20415d8c0 con 0x7fb20415cd50 2023-10-30T13:22:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.368+0000 7fb20942d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2041538f0 0x7fb204159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.368+0000 7fb20942d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2041538f0 0x7fb204159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57722/0 (socket says 172.21.15.46:57722) 2023-10-30T13:22:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.372+0000 7fb20942d640 1 -- 172.21.15.46:0/2545955288 learned_addr learned my addr 172.21.15.46:0/2545955288 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:22:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.372+0000 7fb208c2c640 1 --2- 172.21.15.46:0/2545955288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb20415cd50 0x7fb2040c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.372+0000 7fb20942d640 1 -- 172.21.15.46:0/2545955288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb20415cd50 msgr2=0x7fb2040c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:57.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.372+0000 7fb20942d640 1 --2- 172.21.15.46:0/2545955288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb20415cd50 0x7fb2040c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.372+0000 7fb20942d640 1 -- 172.21.15.46:0/2545955288 >> v1:172.21.15.154:6789/0 conn(0x7fb2040cefa0 legacy=0x7fb2040cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:57.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.372+0000 7fb20942d640 1 -- 172.21.15.46:0/2545955288 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20415dba0 con 0x7fb2041538f0 2023-10-30T13:22:57.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.372+0000 7fb20942d640 1 --2- 172.21.15.46:0/2545955288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2041538f0 0x7fb204159fb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb200009870 tx=0x7fb20001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0775c04b6a9e15 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:57.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.372+0000 7fb1fbfff640 1 -- 172.21.15.46:0/2545955288 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2000a8a30 con 0x7fb2041538f0 2023-10-30T13:22:57.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.372+0000 7fb1fbfff640 1 -- 172.21.15.46:0/2545955288 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2000a8b90 con 0x7fb2041538f0 2023-10-30T13:22:57.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.372+0000 7fb1fbfff640 1 -- 172.21.15.46:0/2545955288 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2000b0440 con 0x7fb2041538f0 2023-10-30T13:22:57.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 -- 172.21.15.46:0/2545955288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2041538f0 msgr2=0x7fb204159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 --2- 172.21.15.46:0/2545955288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2041538f0 0x7fb204159fb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb200009870 tx=0x7fb20001ed30 comp rx=0 tx=0).stop 2023-10-30T13:22:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 -- 172.21.15.46:0/2545955288 shutdown_connections 2023-10-30T13:22:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 --2- 172.21.15.46:0/2545955288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb20415cd50 0x7fb2040c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 --2- 172.21.15.46:0/2545955288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2041538f0 0x7fb204159fb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 -- 172.21.15.46:0/2545955288 >> 172.21.15.46:0/2545955288 conn(0x7fb20406a640 msgr2=0x7fb204065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 -- 172.21.15.46:0/2545955288 shutdown_connections 2023-10-30T13:22:57.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 -- 172.21.15.46:0/2545955288 wait complete. 2023-10-30T13:22:57.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 Processor -- start 2023-10-30T13:22:57.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 -- start start 2023-10-30T13:22:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2040cefa0 0x7fb2040c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2041538f0 0x7fb20415ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb20415cd50 0x7fb20415b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb204146120 con 0x7fb2041538f0 2023-10-30T13:22:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb204146540 con 0x7fb20415cd50 2023-10-30T13:22:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20a42f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb204145a60 con 0x7fb2040cefa0 2023-10-30T13:22:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb208c2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2041538f0 0x7fb20415ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb20942d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2040cefa0 0x7fb2040c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:57.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb208c2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2041538f0 0x7fb20415ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57736/0 (socket says 172.21.15.46:57736) 2023-10-30T13:22:57.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb208c2c640 1 -- 172.21.15.46:0/369749371 learned_addr learned my addr 172.21.15.46:0/369749371 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:22:57.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.376+0000 7fb209c2e640 1 --2- 172.21.15.46:0/369749371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb20415cd50 0x7fb20415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:57.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb208c2c640 1 -- 172.21.15.46:0/369749371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2040cefa0 msgr2=0x7fb2040c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:57.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb208c2c640 1 --2- 172.21.15.46:0/369749371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2040cefa0 0x7fb2040c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb208c2c640 1 -- 172.21.15.46:0/369749371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb20415cd50 msgr2=0x7fb20415b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:57.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb208c2c640 1 --2- 172.21.15.46:0/369749371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb20415cd50 0x7fb20415b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb208c2c640 1 -- 172.21.15.46:0/369749371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2000094d0 con 0x7fb2041538f0 2023-10-30T13:22:57.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb208c2c640 1 --2- 172.21.15.46:0/369749371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2041538f0 0x7fb20415ae00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400e870 tx=0x7fb1f400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:57.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb1f9ffb640 1 -- 172.21.15.46:0/369749371 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4009510 con 0x7fb2041538f0 2023-10-30T13:22:57.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb1f9ffb640 1 -- 172.21.15.46:0/369749371 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f400ee60 con 0x7fb2041538f0 2023-10-30T13:22:57.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb1f9ffb640 1 -- 172.21.15.46:0/369749371 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f400cae0 con 0x7fb2041538f0 2023-10-30T13:22:57.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb20a42f640 1 -- 172.21.15.46:0/369749371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20415c390 con 0x7fb2041538f0 2023-10-30T13:22:57.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb1f9ffb640 1 -- 172.21.15.46:0/369749371 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb1f4013470 con 0x7fb2041538f0 2023-10-30T13:22:57.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb20a42f640 1 -- 172.21.15.46:0/369749371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20415c5e0 con 0x7fb2041538f0 2023-10-30T13:22:57.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb1f9ffb640 1 -- 172.21.15.46:0/369749371 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb1f401c6d0 con 0x7fb2041538f0 2023-10-30T13:22:57.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.380+0000 7fb20a42f640 1 -- 172.21.15.46:0/369749371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20404e170 con 0x7fb2041538f0 2023-10-30T13:22:57.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.384+0000 7fb1f9ffb640 1 -- 172.21.15.46:0/369749371 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fb1f403bb40 con 0x7fb2041538f0 2023-10-30T13:22:57.398 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:57.384+0000 7f11d552b180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T13:22:57.414 INFO:tasks.ceph.mgr.y.smithi046.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-30T13:22:57.415 INFO:tasks.ceph.mgr.y.smithi046.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-30T13:22:57.415 INFO:tasks.ceph.mgr.y.smithi046.stderr: from numpy import show_config as show_numpy_config 2023-10-30T13:22:57.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.440+0000 7fb20a42f640 1 -- 172.21.15.46:0/369749371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2040b2290 con 0x7fb2041538f0 2023-10-30T13:22:57.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.440+0000 7fb1f9ffb640 1 -- 172.21.15.46:0/369749371 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fb1f402a070 con 0x7fb2041538f0 2023-10-30T13:22:57.448 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:22:57.449 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":10,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:22:57.133452+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:22:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.440+0000 7fb20a42f640 1 -- 172.21.15.46:0/369749371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2041538f0 msgr2=0x7fb20415ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.440+0000 7fb20a42f640 1 --2- 172.21.15.46:0/369749371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2041538f0 0x7fb20415ae00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400e870 tx=0x7fb1f400ec70 comp rx=0 tx=0).stop 2023-10-30T13:22:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.440+0000 7fb20a42f640 1 -- 172.21.15.46:0/369749371 shutdown_connections 2023-10-30T13:22:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.440+0000 7fb20a42f640 1 --2- 172.21.15.46:0/369749371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb20415cd50 0x7fb20415b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.440+0000 7fb20a42f640 1 --2- 172.21.15.46:0/369749371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2041538f0 0x7fb20415ae00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.440+0000 7fb20a42f640 1 --2- 172.21.15.46:0/369749371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2040cefa0 0x7fb2040c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:57.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.440+0000 7fb20a42f640 1 -- 172.21.15.46:0/369749371 >> 172.21.15.46:0/369749371 conn(0x7fb20406a640 msgr2=0x7fb204065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:57.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.440+0000 7fb20a42f640 1 -- 172.21.15.46:0/369749371 shutdown_connections 2023-10-30T13:22:57.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:22:57.444+0000 7fb20a42f640 1 -- 172.21.15.46:0/369749371 wait complete. 2023-10-30T13:22:57.468 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-30T13:22:57.469 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-30T13:22:57.474 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:57.468+0000 7f8d267d9180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T13:22:57.616 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:57.608+0000 7f8d267d9180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T13:22:57.787 INFO:tasks.ceph.osd.4.smithi154.stderr:2023-10-30T13:22:57.784+0000 7f026629a640 -1 Falling back to public interface 2023-10-30T13:22:57.788 INFO:tasks.ceph.osd.5.smithi154.stderr:2023-10-30T13:22:57.784+0000 7fd25fa9f640 -1 Falling back to public interface 2023-10-30T13:22:57.788 INFO:tasks.ceph.osd.5.smithi154.stderr:2023-10-30T13:22:57.784+0000 7fd25ea60640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2023-10-30T13:22:57.802 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-10-30T13:22:57.800+0000 7f3aba514640 -1 Falling back to public interface 2023-10-30T13:22:57.810 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:57.804+0000 7f8d267d9180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T13:22:57.811 INFO:tasks.ceph.osd.6.smithi154.stderr:2023-10-30T13:22:57.808+0000 7f42470f8640 -1 Falling back to public interface 2023-10-30T13:22:57.826 INFO:tasks.ceph.osd.1.smithi046.stderr:2023-10-30T13:22:57.820+0000 7f9e47e95640 -1 Falling back to public interface 2023-10-30T13:22:57.835 INFO:tasks.ceph.osd.2.smithi046.stderr:2023-10-30T13:22:57.828+0000 7f1329887640 -1 Falling back to public interface 2023-10-30T13:22:57.835 INFO:tasks.ceph.osd.0.smithi046.stderr:2023-10-30T13:22:57.828+0000 7fef80a2b640 -1 Falling back to public interface 2023-10-30T13:22:57.858 INFO:tasks.ceph.osd.3.smithi046.stderr:2023-10-30T13:22:57.852+0000 7f0c9c84f640 -1 Falling back to public interface 2023-10-30T13:22:57.894 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:57.888+0000 7f8d267d9180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T13:22:58.001 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:58.000+0000 7f11d552b180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T13:22:58.039 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:58.032+0000 7f8d267d9180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T13:22:58.077 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:58.072+0000 7f11d552b180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T13:22:58.248 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:58.243+0000 7f11d552b180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T13:22:58.275 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T13:22:58.321 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:58.315+0000 7f11d552b180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T13:22:58.409 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54ba1a1640 1 Processor -- start 2023-10-30T13:22:58.410 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54ba1a1640 1 -- start start 2023-10-30T13:22:58.411 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54ba1a1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f54b41539f0 0x7f54b40c5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.411 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54ba1a1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54b40cf080 0x7f54b40cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.412 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54ba1a1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b40c5390 con 0x7f54b40c9f90 2023-10-30T13:22:58.412 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54ba1a1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b40ca980 con 0x7f54b40cf080 2023-10-30T13:22:58.412 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54ba1a1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b415a240 con 0x7f54b41539f0 2023-10-30T13:22:58.412 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54b3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54b40cf080 0x7f54b40cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.412 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54b3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f54b40cf080 0x7f54b40cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57750/0 (socket says 172.21.15.46:57750) 2023-10-30T13:22:58.413 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54b3fff640 1 -- 172.21.15.46:0/978970160 learned_addr learned my addr 172.21.15.46:0/978970160 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:22:58.413 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54b37fe640 1 --2- 172.21.15.46:0/978970160 >> v2:172.21.15.154:3300/0 conn(0x7f54b41539f0 0x7f54b40c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.413 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54b3fff640 1 -- 172.21.15.46:0/978970160 >> v2:172.21.15.154:3300/0 conn(0x7f54b41539f0 msgr2=0x7f54b40c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:58.414 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54b3fff640 1 --2- 172.21.15.46:0/978970160 >> v2:172.21.15.154:3300/0 conn(0x7f54b41539f0 0x7f54b40c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.414 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54b3fff640 1 -- 172.21.15.46:0/978970160 >> v1:172.21.15.154:6789/0 conn(0x7f54b40c9f90 legacy=0x7f54b4159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:58.414 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.404+0000 7f54b3fff640 1 -- 172.21.15.46:0/978970160 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b415a520 con 0x7f54b40cf080 2023-10-30T13:22:58.416 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.408+0000 7f54b3fff640 1 --2- 172.21.15.46:0/978970160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b40cf080 0x7f54b40cf510 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f54b4066a30 tx=0x7f54ac01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=339fde323aeaca5a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:58.417 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.408+0000 7f54b27fc640 1 -- 172.21.15.46:0/978970160 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac0a5780 con 0x7f54b40cf080 2023-10-30T13:22:58.417 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.408+0000 7f54b27fc640 1 -- 172.21.15.46:0/978970160 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ac0a58e0 con 0x7f54b40cf080 2023-10-30T13:22:58.417 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.408+0000 7f54b27fc640 1 -- 172.21.15.46:0/978970160 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac0a5c70 con 0x7f54b40cf080 2023-10-30T13:22:58.418 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 -- 172.21.15.46:0/978970160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b40cf080 msgr2=0x7f54b40cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:58.419 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 --2- 172.21.15.46:0/978970160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b40cf080 0x7f54b40cf510 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f54b4066a30 tx=0x7f54ac01c9d0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.419 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 -- 172.21.15.46:0/978970160 shutdown_connections 2023-10-30T13:22:58.419 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 --2- 172.21.15.46:0/978970160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b40cf080 0x7f54b40cf510 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.419 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 --2- 172.21.15.46:0/978970160 >> v2:172.21.15.154:3300/0 conn(0x7f54b41539f0 0x7f54b40c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.419 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 -- 172.21.15.46:0/978970160 >> 172.21.15.46:0/978970160 conn(0x7f54b4065bf0 msgr2=0x7f54b4066000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:58.420 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 -- 172.21.15.46:0/978970160 shutdown_connections 2023-10-30T13:22:58.420 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 -- 172.21.15.46:0/978970160 wait complete. 2023-10-30T13:22:58.422 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 Processor -- start 2023-10-30T13:22:58.423 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 -- start start 2023-10-30T13:22:58.423 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b40c9f90 0x7f54b40cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.423 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b40cf080 0x7f54b41f5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.423 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54b41539f0 0x7f54b413fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.423 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b4157380 con 0x7f54b40c9f90 2023-10-30T13:22:58.424 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b4158070 con 0x7f54b40cf080 2023-10-30T13:22:58.424 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54ba1a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41458c0 con 0x7f54b41539f0 2023-10-30T13:22:58.424 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54b37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b40c9f90 0x7f54b40cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.424 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54b37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b40c9f90 0x7f54b40cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57752/0 (socket says 172.21.15.46:57752) 2023-10-30T13:22:58.424 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54b37fe640 1 -- 172.21.15.46:0/3125760954 learned_addr learned my addr 172.21.15.46:0/3125760954 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:22:58.425 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54b2ffd640 1 --2- 172.21.15.46:0/3125760954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b40cf080 0x7f54b41f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.425 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54b3fff640 1 --2- 172.21.15.46:0/3125760954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54b41539f0 0x7f54b413fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.425 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54b37fe640 1 -- 172.21.15.46:0/3125760954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54b41539f0 msgr2=0x7f54b413fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:58.425 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.412+0000 7f54b37fe640 1 --2- 172.21.15.46:0/3125760954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54b41539f0 0x7f54b413fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.425 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f54b37fe640 1 -- 172.21.15.46:0/3125760954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b40cf080 msgr2=0x7f54b41f5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:58.426 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f54b37fe640 1 --2- 172.21.15.46:0/3125760954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b40cf080 0x7f54b41f5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.426 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f54b37fe640 1 -- 172.21.15.46:0/3125760954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac009b30 con 0x7f54b40c9f90 2023-10-30T13:22:58.426 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f54b2ffd640 1 --2- 172.21.15.46:0/3125760954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b40cf080 0x7f54b41f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:58.426 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f54b37fe640 1 --2- 172.21.15.46:0/3125760954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b40c9f90 0x7f54b40cba20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f54a400e870 tx=0x7f54a400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:58.426 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f549bfff640 1 -- 172.21.15.46:0/3125760954 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4009510 con 0x7f54b40c9f90 2023-10-30T13:22:58.427 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f549bfff640 1 -- 172.21.15.46:0/3125760954 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a400eec0 con 0x7f54b40c9f90 2023-10-30T13:22:58.427 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f54ba1a1640 1 -- 172.21.15.46:0/3125760954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b4145ba0 con 0x7f54b40c9f90 2023-10-30T13:22:58.427 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f549bfff640 1 -- 172.21.15.46:0/3125760954 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a400cb40 con 0x7f54b40c9f90 2023-10-30T13:22:58.427 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f54ba1a1640 1 -- 172.21.15.46:0/3125760954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b4157870 con 0x7f54b40c9f90 2023-10-30T13:22:58.427 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f549bfff640 1 -- 172.21.15.46:0/3125760954 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f54a40059a0 con 0x7f54b40c9f90 2023-10-30T13:22:58.428 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f549bfff640 1 -- 172.21.15.46:0/3125760954 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f54a40067f0 con 0x7f54b40c9f90 2023-10-30T13:22:58.428 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.416+0000 7f54ba1a1640 1 -- 172.21.15.46:0/3125760954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b404e170 con 0x7f54b40c9f90 2023-10-30T13:22:58.429 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.420+0000 7f549bfff640 1 -- 172.21.15.46:0/3125760954 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f54a4021060 con 0x7f54b40c9f90 2023-10-30T13:22:58.483 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.476+0000 7f54ba1a1640 1 -- 172.21.15.46:0/3125760954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54b414d860 con 0x7f54b40c9f90 2023-10-30T13:22:58.484 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.476+0000 7f549bfff640 1 -- 172.21.15.46:0/3125760954 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f54a4017070 con 0x7f54b40c9f90 2023-10-30T13:22:58.484 INFO:teuthology.misc.health.smithi046.stdout: 2023-10-30T13:22:58.485 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:22:57.133452+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:22:58.486 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.476+0000 7f54ba1a1640 1 -- 172.21.15.46:0/3125760954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b40c9f90 msgr2=0x7f54b40cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:58.486 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.476+0000 7f54ba1a1640 1 --2- 172.21.15.46:0/3125760954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b40c9f90 0x7f54b40cba20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f54a400e870 tx=0x7f54a400ec70 comp rx=0 tx=0).stop 2023-10-30T13:22:58.486 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.476+0000 7f54ba1a1640 1 -- 172.21.15.46:0/3125760954 shutdown_connections 2023-10-30T13:22:58.486 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.476+0000 7f54ba1a1640 1 --2- 172.21.15.46:0/3125760954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54b41539f0 0x7f54b413fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.486 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.476+0000 7f54ba1a1640 1 --2- 172.21.15.46:0/3125760954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b40cf080 0x7f54b41f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.487 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.476+0000 7f54ba1a1640 1 --2- 172.21.15.46:0/3125760954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b40c9f90 0x7f54b40cba20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.487 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.476+0000 7f54ba1a1640 1 -- 172.21.15.46:0/3125760954 >> 172.21.15.46:0/3125760954 conn(0x7f54b4065bf0 msgr2=0x7f54b41527f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:58.487 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.480+0000 7f54ba1a1640 1 -- 172.21.15.46:0/3125760954 shutdown_connections 2023-10-30T13:22:58.488 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:22:58.480+0000 7f54ba1a1640 1 -- 172.21.15.46:0/3125760954 wait complete. 2023-10-30T13:22:58.502 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T13:22:58.610 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:58.607+0000 7f11d552b180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T13:22:58.671 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:58.667+0000 7f11d552b180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T13:22:58.768 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:58.760+0000 7f8d267d9180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T13:22:58.859 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:58.852+0000 7f8d267d9180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T13:22:59.060 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:59.052+0000 7f8d267d9180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T13:22:59.150 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:59.144+0000 7f8d267d9180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T13:22:59.496 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:59.488+0000 7f8d267d9180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T13:22:59.567 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-10-30T13:22:59.560+0000 7f8d267d9180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T13:22:59.722 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:59.719+0000 7f117cca1640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-30T13:22:59.722 INFO:tasks.ceph.mgr.x.smithi154.stderr:2023-10-30T13:22:59.719+0000 7f117cca1640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-30T13:23:01.620 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-10-30T13:23:01.615+0000 7f3aba514640 -1 osd.7 0 log_to_monitors true 2023-10-30T13:23:01.643 INFO:tasks.ceph.osd.6.smithi154.stderr:2023-10-30T13:23:01.639+0000 7f42470f8640 -1 osd.6 0 log_to_monitors true 2023-10-30T13:23:01.674 INFO:tasks.ceph.osd.2.smithi046.stderr:2023-10-30T13:23:01.668+0000 7f1329887640 -1 osd.2 0 log_to_monitors true 2023-10-30T13:23:01.691 INFO:tasks.ceph.osd.4.smithi154.stderr:2023-10-30T13:23:01.687+0000 7f026629a640 -1 osd.4 0 log_to_monitors true 2023-10-30T13:23:01.692 INFO:tasks.ceph.osd.5.smithi154.stderr:2023-10-30T13:23:01.687+0000 7fd25fa9f640 -1 osd.5 0 log_to_monitors true 2023-10-30T13:23:01.798 INFO:tasks.ceph.osd.3.smithi046.stderr:2023-10-30T13:23:01.792+0000 7f0c9c84f640 -1 osd.3 0 log_to_monitors true 2023-10-30T13:23:01.800 INFO:tasks.ceph.osd.0.smithi046.stderr:2023-10-30T13:23:01.792+0000 7fef80a2b640 -1 osd.0 0 log_to_monitors true 2023-10-30T13:23:01.836 INFO:tasks.ceph.osd.1.smithi046.stderr:2023-10-30T13:23:01.828+0000 7f9e47e95640 -1 osd.1 0 log_to_monitors true 2023-10-30T13:23:04.813 INFO:tasks.ceph.osd.1.smithi046.stderr:2023-10-30T13:23:04.808+0000 7f9e4361c640 -1 osd.1 0 waiting for initial osdmap 2023-10-30T13:23:04.851 INFO:tasks.ceph.osd.1.smithi046.stderr:2023-10-30T13:23:04.844+0000 7f9e3bc63640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:23:04.855 INFO:tasks.ceph.osd.2.smithi046.stderr:2023-10-30T13:23:04.848+0000 7f1326846640 -1 osd.2 0 waiting for initial osdmap 2023-10-30T13:23:04.888 INFO:tasks.ceph.osd.0.smithi046.stderr:2023-10-30T13:23:04.880+0000 7fef7b76c640 -1 osd.0 0 waiting for initial osdmap 2023-10-30T13:23:04.907 INFO:tasks.ceph.osd.2.smithi046.stderr:2023-10-30T13:23:04.900+0000 7f131d655640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:23:04.926 INFO:tasks.ceph.osd.0.smithi046.stderr:2023-10-30T13:23:04.920+0000 7fef73db3640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:23:04.944 INFO:tasks.ceph.osd.3.smithi046.stderr:2023-10-30T13:23:04.936+0000 7f0c97fd4640 -1 osd.3 0 waiting for initial osdmap 2023-10-30T13:23:04.970 INFO:tasks.ceph.osd.3.smithi046.stderr:2023-10-30T13:23:04.964+0000 7f0c9061b640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:23:05.175 INFO:tasks.ceph.osd.5.smithi154.stderr:2023-10-30T13:23:05.171+0000 7fd25ca5c640 -1 osd.5 0 waiting for initial osdmap 2023-10-30T13:23:05.206 INFO:tasks.ceph.osd.5.smithi154.stderr:2023-10-30T13:23:05.203+0000 7fd25386b640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:23:05.215 INFO:tasks.ceph.osd.6.smithi154.stderr:2023-10-30T13:23:05.211+0000 7f424287d640 -1 osd.6 0 waiting for initial osdmap 2023-10-30T13:23:05.246 INFO:tasks.ceph.osd.6.smithi154.stderr:2023-10-30T13:23:05.243+0000 7f423aec4640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:23:05.306 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:05.309 INFO:tasks.ceph.osd.4.smithi154.stderr:2023-10-30T13:23:05.303+0000 7f0263257640 -1 osd.4 0 waiting for initial osdmap 2023-10-30T13:23:05.311 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-10-30T13:23:05.307+0000 7f3ab5c99640 -1 osd.7 0 waiting for initial osdmap 2023-10-30T13:23:05.331 INFO:tasks.ceph.osd.4.smithi154.stderr:2023-10-30T13:23:05.327+0000 7f025a066640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:23:05.334 INFO:tasks.ceph.osd.7.smithi154.stderr:2023-10-30T13:23:05.331+0000 7f3aae2e0640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:23:05.440 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.432+0000 7fe68cfb8640 1 Processor -- start 2023-10-30T13:23:05.440 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.432+0000 7fe68cfb8640 1 -- start start 2023-10-30T13:23:05.442 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.432+0000 7fe68cfb8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe688153930 0x7fe6880c7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:05.443 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.432+0000 7fe68cfb8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68806e910 con 0x7fe68806e4a0 2023-10-30T13:23:05.443 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.432+0000 7fe68cfb8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68806c980 con 0x7fe68815a100 2023-10-30T13:23:05.444 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.432+0000 7fe68cfb8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe68815a520 con 0x7fe688153930 2023-10-30T13:23:05.444 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.436+0000 7fe68659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe688153930 0x7fe6880c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:05.444 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.436+0000 7fe68659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe688153930 0x7fe6880c7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57804/0 (socket says 172.21.15.46:57804) 2023-10-30T13:23:05.444 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.436+0000 7fe68659c640 1 -- 172.21.15.46:0/4174198242 learned_addr learned my addr 172.21.15.46:0/4174198242 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:05.445 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.436+0000 7fe68659c640 1 -- 172.21.15.46:0/4174198242 >> v1:172.21.15.154:6789/0 conn(0x7fe68815a100 legacy=0x7fe688164030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.445 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.436+0000 7fe68659c640 1 -- 172.21.15.46:0/4174198242 >> v1:172.21.15.46:6789/0 conn(0x7fe68806e4a0 legacy=0x7fe68806d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.445 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.436+0000 7fe685d9b640 1 -- 172.21.15.46:0/4174198242 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709298650 0 0) 0x7fe68806e910 con 0x7fe68806e4a0 2023-10-30T13:23:05.446 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.436+0000 7fe68659c640 1 -- 172.21.15.46:0/4174198242 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68806cbf0 con 0x7fe688153930 2023-10-30T13:23:05.447 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.436+0000 7fe68659c640 1 --2- 172.21.15.46:0/4174198242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe688153930 0x7fe6880c7930 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe678009870 tx=0x7fe67801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ea57976a8fd72e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:05.447 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.436+0000 7fe685d9b640 1 -- 172.21.15.46:0/4174198242 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6780ab650 con 0x7fe688153930 2023-10-30T13:23:05.447 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.436+0000 7fe685d9b640 1 -- 172.21.15.46:0/4174198242 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6780ab7b0 con 0x7fe688153930 2023-10-30T13:23:05.448 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.436+0000 7fe685d9b640 1 -- 172.21.15.46:0/4174198242 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6780abb40 con 0x7fe688153930 2023-10-30T13:23:05.448 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 -- 172.21.15.46:0/4174198242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe688153930 msgr2=0x7fe6880c7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.448 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 --2- 172.21.15.46:0/4174198242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe688153930 0x7fe6880c7930 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe678009870 tx=0x7fe67801c540 comp rx=0 tx=0).stop 2023-10-30T13:23:05.448 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 -- 172.21.15.46:0/4174198242 shutdown_connections 2023-10-30T13:23:05.449 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 --2- 172.21.15.46:0/4174198242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe688153930 0x7fe6880c7930 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.449 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 -- 172.21.15.46:0/4174198242 >> 172.21.15.46:0/4174198242 conn(0x7fe688071de0 msgr2=0x7fe688065bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:05.449 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 -- 172.21.15.46:0/4174198242 shutdown_connections 2023-10-30T13:23:05.449 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 -- 172.21.15.46:0/4174198242 wait complete. 2023-10-30T13:23:05.449 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 Processor -- start 2023-10-30T13:23:05.450 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 -- start start 2023-10-30T13:23:05.452 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe68806e4a0 0x7fe6880c3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:05.452 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe688153930 0x7fe688150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:05.452 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe68815a100 0x7fe688150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:05.452 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6880c6470 con 0x7fe68815a100 2023-10-30T13:23:05.452 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe688158800 con 0x7fe688153930 2023-10-30T13:23:05.453 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68cfb8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6880c4d90 con 0x7fe68806e4a0 2023-10-30T13:23:05.453 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe686d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe68815a100 0x7fe688150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:05.453 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe68659c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe68806e4a0 0x7fe6880c3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:05.453 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe686d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe68815a100 0x7fe688150cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57810/0 (socket says 172.21.15.46:57810) 2023-10-30T13:23:05.453 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe686d9d640 1 -- 172.21.15.46:0/2664367373 learned_addr learned my addr 172.21.15.46:0/2664367373 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:05.454 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe67ffff640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe688153930 0x7fe688150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:05.454 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe686d9d640 1 -- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe68806e4a0 msgr2=0x7fe6880c3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.454 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe686d9d640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe68806e4a0 0x7fe6880c3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.454 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe686d9d640 1 -- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe688153930 msgr2=0x7fe688150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.455 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe686d9d640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe688153930 0x7fe688150780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.455 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.440+0000 7fe686d9d640 1 -- 172.21.15.46:0/2664367373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6780094d0 con 0x7fe68815a100 2023-10-30T13:23:05.455 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.444+0000 7fe686d9d640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe68815a100 0x7fe688150cc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe68000c6b0 tx=0x7fe68000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:05.455 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.444+0000 7fe67effd640 1 -- 172.21.15.46:0/2664367373 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680003bd0 con 0x7fe68815a100 2023-10-30T13:23:05.455 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.444+0000 7fe67effd640 1 -- 172.21.15.46:0/2664367373 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe680003d50 con 0x7fe68815a100 2023-10-30T13:23:05.456 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.444+0000 7fe67effd640 1 -- 172.21.15.46:0/2664367373 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680005310 con 0x7fe68815a100 2023-10-30T13:23:05.456 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.444+0000 7fe68cfb8640 1 -- 172.21.15.46:0/2664367373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6880c5220 con 0x7fe68815a100 2023-10-30T13:23:05.456 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.444+0000 7fe68cfb8640 1 -- 172.21.15.46:0/2664367373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6880c4460 con 0x7fe68815a100 2023-10-30T13:23:05.456 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.444+0000 7fe68cfb8640 1 -- 172.21.15.46:0/2664367373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6880c4f00 con 0x7fe68815a100 2023-10-30T13:23:05.456 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.444+0000 7fe67effd640 1 -- 172.21.15.46:0/2664367373 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 85794+0+0 (secure 0 0 0) 0x7fe680005470 con 0x7fe68815a100 2023-10-30T13:23:05.457 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.444+0000 7fe67effd640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe668066410 0x7fe6680688d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:05.457 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.444+0000 7fe67effd640 1 -- 172.21.15.46:0/2664367373 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fe680082d80 con 0x7fe68815a100 2023-10-30T13:23:05.458 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.444+0000 7fe68659c640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe668066410 0x7fe6680688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:05.458 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.448+0000 7fe68659c640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe668066410 0x7fe6680688d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe6780af000 tx=0x7fe6780b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:05.458 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.448+0000 7fe67effd640 1 -- 172.21.15.46:0/2664367373 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe680051000 con 0x7fe68815a100 2023-10-30T13:23:05.614 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 -- 172.21.15.46:0/2664367373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe68804e170 con 0x7fe68815a100 2023-10-30T13:23:05.615 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe67effd640 1 -- 172.21.15.46:0/2664367373 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe680054620 con 0x7fe68815a100 2023-10-30T13:23:05.615 INFO:teuthology.misc.health.smithi046.stdout: 2023-10-30T13:23:05.616 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":12,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:02.965775+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:05.617 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 -- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe668066410 msgr2=0x7fe6680688d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.617 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe668066410 0x7fe6680688d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe6780af000 tx=0x7fe6780b6040 comp rx=0 tx=0).stop 2023-10-30T13:23:05.617 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 -- 172.21.15.46:0/2664367373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe68815a100 msgr2=0x7fe688150cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.618 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe68815a100 0x7fe688150cc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe68000c6b0 tx=0x7fe68000cb70 comp rx=0 tx=0).stop 2023-10-30T13:23:05.618 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 -- 172.21.15.46:0/2664367373 shutdown_connections 2023-10-30T13:23:05.618 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe668066410 0x7fe6680688d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.618 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe68815a100 0x7fe688150cc0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.618 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe688153930 0x7fe688150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.619 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 --2- 172.21.15.46:0/2664367373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe68806e4a0 0x7fe6880c3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.619 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 -- 172.21.15.46:0/2664367373 >> 172.21.15.46:0/2664367373 conn(0x7fe688071de0 msgr2=0x7fe6880c9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:05.619 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 -- 172.21.15.46:0/2664367373 shutdown_connections 2023-10-30T13:23:05.619 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:05.608+0000 7fe68cfb8640 1 -- 172.21.15.46:0/2664367373 wait complete. 2023-10-30T13:23:05.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T13:23:12.441 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:12.584 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.576+0000 7fb9fb9aa640 1 Processor -- start 2023-10-30T13:23:12.584 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.576+0000 7fb9fb9aa640 1 -- start start 2023-10-30T13:23:12.586 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9fb9aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9f406fb80 0x7fb9f40c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.586 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9fb9aa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f4070070 con 0x7fb9f41515d0 2023-10-30T13:23:12.587 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9fb9aa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f40b0a10 con 0x7fb9f40708e0 2023-10-30T13:23:12.587 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9fb9aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f4070350 con 0x7fb9f406fb80 2023-10-30T13:23:12.587 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9f9746640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb9f41515d0 0x7fb9f40c4420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45948/0 (socket says 172.21.15.46:45948) 2023-10-30T13:23:12.587 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9f9746640 1 -- 172.21.15.46:0/3664037621 learned_addr learned my addr 172.21.15.46:0/3664037621 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:12.587 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9f8f45640 1 --2- 172.21.15.46:0/3664037621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9f406fb80 0x7fb9f40c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.588 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34530177 0 0) 0x7fb9f4070070 con 0x7fb9f41515d0 2023-10-30T13:23:12.588 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e0003530 con 0x7fb9f41515d0 2023-10-30T13:23:12.590 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499371735 0 0) 0x7fb9f40b0a10 con 0x7fb9f40708e0 2023-10-30T13:23:12.590 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f4070070 con 0x7fb9f40708e0 2023-10-30T13:23:12.590 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 621173749 0 0) 0x7fb9e0003530 con 0x7fb9f41515d0 2023-10-30T13:23:12.590 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9f406fb80 msgr2=0x7fb9f40c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:12.591 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 --2- 172.21.15.46:0/3664037621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9f406fb80 0x7fb9f40c2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.591 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 >> v1:172.21.15.154:6789/0 conn(0x7fb9f40708e0 legacy=0x7fb9f4157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.591 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f40b0cf0 con 0x7fb9f41515d0 2023-10-30T13:23:12.591 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fb9f0001680 con 0x7fb9f41515d0 2023-10-30T13:23:12.591 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 >> v1:172.21.15.46:6789/0 conn(0x7fb9f41515d0 legacy=0x7fb9f40c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.592 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 --2- 172.21.15.46:0/3664037621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9f406fb80 0x7fb9e0003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.592 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 --2- 172.21.15.46:0/3664037621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e00040f0 0x7fb9e00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.592 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 --2- 172.21.15.46:0/3664037621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9e0004d10 0x7fb9e0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.592 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f4070350 con 0x7fb9e0004d10 2023-10-30T13:23:12.592 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e000bad0 con 0x7fb9e00040f0 2023-10-30T13:23:12.593 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e00100a0 con 0x7fb9f406fb80 2023-10-30T13:23:12.593 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9f9f47640 1 --2- 172.21.15.46:0/3664037621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9e0004d10 0x7fb9e0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.593 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9f8f45640 1 --2- 172.21.15.46:0/3664037621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e00040f0 0x7fb9e00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.593 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9f9746640 1 --2- 172.21.15.46:0/3664037621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9f406fb80 0x7fb9e0003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.593 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9f9f47640 1 -- 172.21.15.46:0/3664037621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9f406fb80 msgr2=0x7fb9e0003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:12.594 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9f9f47640 1 --2- 172.21.15.46:0/3664037621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9f406fb80 0x7fb9e0003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.594 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9f9f47640 1 -- 172.21.15.46:0/3664037621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e00040f0 msgr2=0x7fb9e00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:12.594 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9f9f47640 1 --2- 172.21.15.46:0/3664037621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e00040f0 0x7fb9e00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.594 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.580+0000 7fb9f9f47640 1 -- 172.21.15.46:0/3664037621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e0003530 con 0x7fb9e0004d10 2023-10-30T13:23:12.597 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.584+0000 7fb9f9f47640 1 --2- 172.21.15.46:0/3664037621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9e0004d10 0x7fb9e0005190 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec001fd0 tx=0x7fb9ec01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b505b3364207c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:12.597 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.584+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec003b80 con 0x7fb9e0004d10 2023-10-30T13:23:12.597 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.584+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ec003d00 con 0x7fb9e0004d10 2023-10-30T13:23:12.598 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.584+0000 7fb9ebfff640 1 -- 172.21.15.46:0/3664037621 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec005310 con 0x7fb9e0004d10 2023-10-30T13:23:12.598 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.584+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/3664037621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9e0004d10 msgr2=0x7fb9e0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.598 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.584+0000 7fb9fb9aa640 1 --2- 172.21.15.46:0/3664037621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9e0004d10 0x7fb9e0005190 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec001fd0 tx=0x7fb9ec01c8e0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.598 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.584+0000 7fb9f9f47640 1 -- 172.21.15.46:0/3664037621 reap_dead start 2023-10-30T13:23:12.598 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.584+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/3664037621 shutdown_connections 2023-10-30T13:23:12.599 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.584+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/3664037621 >> 172.21.15.46:0/3664037621 conn(0x7fb9f40cb0b0 msgr2=0x7fb9f40cb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:12.599 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.584+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/3664037621 shutdown_connections 2023-10-30T13:23:12.599 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.584+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/3664037621 wait complete. 2023-10-30T13:23:12.599 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9fb9aa640 1 Processor -- start 2023-10-30T13:23:12.600 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9fb9aa640 1 -- start start 2023-10-30T13:23:12.600 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9fb9aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e0004d10 0x7fb9f4156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.600 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9fb9aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f414ee20 0x7fb9f4156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.600 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9fb9aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9f406a3b0 0x7fb9f406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.600 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9fb9aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f4069f90 con 0x7fb9f406a3b0 2023-10-30T13:23:12.600 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9fb9aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f41ee060 con 0x7fb9f414ee20 2023-10-30T13:23:12.601 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9fb9aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f406ad70 con 0x7fb9e0004d10 2023-10-30T13:23:12.601 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9f9f47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9f406a3b0 0x7fb9f406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.601 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9f9f47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9f406a3b0 0x7fb9f406a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55166/0 (socket says 172.21.15.46:55166) 2023-10-30T13:23:12.601 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9f9f47640 1 -- 172.21.15.46:0/2326077364 learned_addr learned my addr 172.21.15.46:0/2326077364 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:12.601 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9f9746640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e0004d10 0x7fb9f4156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.602 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9f8f45640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f414ee20 0x7fb9f4156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.602 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9f9f47640 1 -- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e0004d10 msgr2=0x7fb9f4156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.602 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9f9f47640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e0004d10 0x7fb9f4156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.602 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9f9f47640 1 -- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f414ee20 msgr2=0x7fb9f4156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.602 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9f9f47640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f414ee20 0x7fb9f4156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.603 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9f9f47640 1 -- 172.21.15.46:0/2326077364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f414fd80 con 0x7fb9f406a3b0 2023-10-30T13:23:12.603 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9f9746640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e0004d10 0x7fb9f4156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:12.603 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9f9f47640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9f406a3b0 0x7fb9f406a830 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec01cdc0 tx=0x7fb9e0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:12.603 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9e9ffb640 1 -- 172.21.15.46:0/2326077364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec003e70 con 0x7fb9f406a3b0 2023-10-30T13:23:12.604 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9e9ffb640 1 -- 172.21.15.46:0/2326077364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ec003980 con 0x7fb9f406a3b0 2023-10-30T13:23:12.604 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9e9ffb640 1 -- 172.21.15.46:0/2326077364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec006b60 con 0x7fb9f406a3b0 2023-10-30T13:23:12.604 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/2326077364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f41ee570 con 0x7fb9f406a3b0 2023-10-30T13:23:12.604 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/2326077364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f406b7a0 con 0x7fb9f406a3b0 2023-10-30T13:23:12.604 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.588+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/2326077364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f406a0d0 con 0x7fb9f406a3b0 2023-10-30T13:23:12.605 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.592+0000 7fb9e9ffb640 1 -- 172.21.15.46:0/2326077364 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9ec0a55d0 con 0x7fb9f406a3b0 2023-10-30T13:23:12.606 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.592+0000 7fb9e9ffb640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9d80664a0 0x7fb9d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.606 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.592+0000 7fb9e9ffb640 1 -- 172.21.15.46:0/2326077364 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb9ec11d660 con 0x7fb9f406a3b0 2023-10-30T13:23:12.606 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.592+0000 7fb9f9746640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9d80664a0 0x7fb9d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.606 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.592+0000 7fb9f9746640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9d80664a0 0x7fb9d8068960 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0000c10 tx=0x7fb9f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:12.606 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.596+0000 7fb9e9ffb640 1 -- 172.21.15.46:0/2326077364 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9f406a0d0 con 0x7fb9f406a3b0 2023-10-30T13:23:12.761 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.752+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/2326077364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9f406a0d0 con 0x7fb9f406a3b0 2023-10-30T13:23:12.763 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9e9ffb640 1 -- 172.21.15.46:0/2326077364 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12850 (secure 0 0 0) 0x7fb9f406a0d0 con 0x7fb9f406a3b0 2023-10-30T13:23:12.763 INFO:teuthology.misc.health.smithi046.stdout: 2023-10-30T13:23:12.763 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":16,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:09.200520+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:12.765 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9d80664a0 msgr2=0x7fb9d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.765 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9fb9aa640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9d80664a0 0x7fb9d8068960 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0000c10 tx=0x7fb9f0009040 comp rx=0 tx=0).stop 2023-10-30T13:23:12.765 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/2326077364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9f406a3b0 msgr2=0x7fb9f406a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.766 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9fb9aa640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9f406a3b0 0x7fb9f406a830 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec01cdc0 tx=0x7fb9e0005800 comp rx=0 tx=0).stop 2023-10-30T13:23:12.766 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/2326077364 shutdown_connections 2023-10-30T13:23:12.766 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9fb9aa640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9d80664a0 0x7fb9d8068960 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.766 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9fb9aa640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9f406a3b0 0x7fb9f406a830 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.767 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9fb9aa640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9f414ee20 0x7fb9f4156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.767 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9fb9aa640 1 --2- 172.21.15.46:0/2326077364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e0004d10 0x7fb9f4156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.767 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/2326077364 >> 172.21.15.46:0/2326077364 conn(0x7fb9f40cb0b0 msgr2=0x7fb9f40c6270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:12.767 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/2326077364 shutdown_connections 2023-10-30T13:23:12.767 INFO:teuthology.misc.health.smithi046.stderr:2023-10-30T13:23:12.756+0000 7fb9fb9aa640 1 -- 172.21.15.46:0/2326077364 wait complete. 2023-10-30T13:23:12.783 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-30T13:23:12.783 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-30T13:23:12.784 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:23:12.784 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:23:12.788 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:23:12.788 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:23:12.789 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:23:12.793 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:23:12.793 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:23:12.793 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:23:12.797 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:23:12.797 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T13:23:12.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T13:23:12.801 INFO:tasks.ceph.mds.b:Started 2023-10-30T13:23:12.801 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:23:12.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:23:12.804 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:23:12.804 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T13:23:12.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T13:23:12.807 INFO:tasks.ceph.mds.f:Started 2023-10-30T13:23:12.807 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-30T13:23:12.808 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:12.885 INFO:tasks.ceph.mds.b.smithi154.stdout:starting mds.b at 2023-10-30T13:23:12.886 INFO:tasks.ceph.mds.f.smithi154.stdout:starting mds.f at 2023-10-30T13:23:12.887 INFO:tasks.ceph.mds.d.smithi154.stdout:starting mds.d at 2023-10-30T13:23:12.893 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-10-30T13:23:12.905 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-10-30T13:23:12.905 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T13:23:12.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab7bb8640 1 Processor -- start 2023-10-30T13:23:12.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab7bb8640 1 -- start start 2023-10-30T13:23:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab7bb8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab01538a0 0x7fbab015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab7bb8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbab00ca0e0 0x7fbab015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab7bb8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab00cb800 con 0x7fbab00cf160 2023-10-30T13:23:12.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab7bb8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab015dc30 con 0x7fbab00ca0e0 2023-10-30T13:23:12.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab7bb8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab00cb330 con 0x7fbab01538a0 2023-10-30T13:23:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab5153640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbab00ca0e0 0x7fbab015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab5153640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbab00ca0e0 0x7fbab015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55214/0 (socket says 172.21.15.46:55214) 2023-10-30T13:23:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab5954640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab01538a0 0x7fbab015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab6155640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbab00cf160 0x7fbab00cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45964/0 (socket says 172.21.15.46:45964) 2023-10-30T13:23:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab5153640 1 -- 172.21.15.46:0/2174126533 learned_addr learned my addr 172.21.15.46:0/2174126533 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab5153640 1 -- 172.21.15.46:0/2174126533 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab01538a0 msgr2=0x7fbab015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab5153640 1 --2- 172.21.15.46:0/2174126533 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab01538a0 0x7fbab015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab5153640 1 -- 172.21.15.46:0/2174126533 >> v1:172.21.15.46:6789/0 conn(0x7fbab00cf160 legacy=0x7fbab00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab5153640 1 -- 172.21.15.46:0/2174126533 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab00cad70 con 0x7fbab00ca0e0 2023-10-30T13:23:12.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.956+0000 7fbab4952640 1 -- 172.21.15.46:0/2174126533 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415968565 0 0) 0x7fbab00cb800 con 0x7fbab00cf160 2023-10-30T13:23:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.960+0000 7fbab5153640 1 --2- 172.21.15.46:0/2174126533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab00ca0e0 0x7fbab015a090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0009870 tx=0x7fbaa001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db9012c6315d528 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.960+0000 7fbab4952640 1 -- 172.21.15.46:0/2174126533 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa00a8a30 con 0x7fbab00ca0e0 2023-10-30T13:23:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.960+0000 7fbab4952640 1 -- 172.21.15.46:0/2174126533 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa00a8b90 con 0x7fbab00ca0e0 2023-10-30T13:23:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.960+0000 7fbab4952640 1 -- 172.21.15.46:0/2174126533 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa00b0440 con 0x7fbab00ca0e0 2023-10-30T13:23:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.960+0000 7fbab7bb8640 1 -- 172.21.15.46:0/2174126533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab00ca0e0 msgr2=0x7fbab015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.960+0000 7fbab7bb8640 1 --2- 172.21.15.46:0/2174126533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab00ca0e0 0x7fbab015a090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0009870 tx=0x7fbaa001ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.960+0000 7fbab7bb8640 1 -- 172.21.15.46:0/2174126533 shutdown_connections 2023-10-30T13:23:12.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.960+0000 7fbab7bb8640 1 --2- 172.21.15.46:0/2174126533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab00ca0e0 0x7fbab015a090 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.960+0000 7fbab7bb8640 1 --2- 172.21.15.46:0/2174126533 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab01538a0 0x7fbab015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.960+0000 7fbab7bb8640 1 -- 172.21.15.46:0/2174126533 >> 172.21.15.46:0/2174126533 conn(0x7fbab006bfe0 msgr2=0x7fbab006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:12.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.960+0000 7fbab7bb8640 1 -- 172.21.15.46:0/2174126533 shutdown_connections 2023-10-30T13:23:12.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.960+0000 7fbab7bb8640 1 -- 172.21.15.46:0/2174126533 wait complete. 2023-10-30T13:23:12.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab7bb8640 1 Processor -- start 2023-10-30T13:23:12.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab7bb8640 1 -- start start 2023-10-30T13:23:12.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab7bb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab00ca0e0 0x7fbab0157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab7bb8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbab00cf160 0x7fbab015cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab7bb8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab01538a0 0x7fbab01f0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab7bb8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab0142700 con 0x7fbab00ca0e0 2023-10-30T13:23:12.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab7bb8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab00ce9c0 con 0x7fbab00cf160 2023-10-30T13:23:12.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab7bb8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab00cec60 con 0x7fbab01538a0 2023-10-30T13:23:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab5954640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab00ca0e0 0x7fbab0157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab5954640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab00ca0e0 0x7fbab0157f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55230/0 (socket says 172.21.15.46:55230) 2023-10-30T13:23:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab5954640 1 -- 172.21.15.46:0/319066675 learned_addr learned my addr 172.21.15.46:0/319066675 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab6155640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab01538a0 0x7fbab01f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab5153640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbab00cf160 0x7fbab015cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab5954640 1 -- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab01538a0 msgr2=0x7fbab01f0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab5954640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab01538a0 0x7fbab01f0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab5954640 1 -- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbab00cf160 msgr2=0x7fbab015cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab5954640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbab00cf160 0x7fbab015cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab5954640 1 -- 172.21.15.46:0/319066675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00094d0 con 0x7fbab00ca0e0 2023-10-30T13:23:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab5153640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbab00cf160 0x7fbab015cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab5954640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab00ca0e0 0x7fbab0157f30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00e870 tx=0x7fbaac00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbaa67fc640 1 -- 172.21.15.46:0/319066675 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac009530 con 0x7fbab00ca0e0 2023-10-30T13:23:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbaa67fc640 1 -- 172.21.15.46:0/319066675 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac00c940 con 0x7fbab00ca0e0 2023-10-30T13:23:12.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab7bb8640 1 -- 172.21.15.46:0/319066675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab00cef20 con 0x7fbab00ca0e0 2023-10-30T13:23:12.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbaa67fc640 1 -- 172.21.15.46:0/319066675 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac00ccd0 con 0x7fbab00ca0e0 2023-10-30T13:23:12.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab7bb8640 1 -- 172.21.15.46:0/319066675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab01f28f0 con 0x7fbab00ca0e0 2023-10-30T13:23:12.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.964+0000 7fbab7bb8640 1 -- 172.21.15.46:0/319066675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab00b2a50 con 0x7fbab00ca0e0 2023-10-30T13:23:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.968+0000 7fbaa67fc640 1 -- 172.21.15.46:0/319066675 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbaac013470 con 0x7fbab00ca0e0 2023-10-30T13:23:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.968+0000 7fbaa67fc640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fba940664a0 0x7fba94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.968+0000 7fbaa67fc640 1 -- 172.21.15.46:0/319066675 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbaac083c50 con 0x7fbab00ca0e0 2023-10-30T13:23:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.968+0000 7fbab5153640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fba940664a0 0x7fba94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.968+0000 7fbab5153640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fba940664a0 0x7fba94068960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00b2000 tx=0x7fbaa00aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:12.972+0000 7fbaa67fc640 1 -- 172.21.15.46:0/319066675 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbaac089020 con 0x7fbab00ca0e0 2023-10-30T13:23:13.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 -- 172.21.15.46:0/319066675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbab0159820 con 0x7fbab00ca0e0 2023-10-30T13:23:13.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbaa67fc640 1 -- 172.21.15.46:0/319066675 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12850 (secure 0 0 0) 0x7fbaac0514b0 con 0x7fbab00ca0e0 2023-10-30T13:23:13.138 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:13.139 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":16,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:09.200520+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:13.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 -- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fba940664a0 msgr2=0x7fba94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fba940664a0 0x7fba94068960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00b2000 tx=0x7fbaa00aa070 comp rx=0 tx=0).stop 2023-10-30T13:23:13.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 -- 172.21.15.46:0/319066675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab00ca0e0 msgr2=0x7fbab0157f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab00ca0e0 0x7fbab0157f30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00e870 tx=0x7fbaac00ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:13.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 -- 172.21.15.46:0/319066675 shutdown_connections 2023-10-30T13:23:13.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fba940664a0 0x7fba94068960 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab01538a0 0x7fbab01f0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbab00cf160 0x7fbab015cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 --2- 172.21.15.46:0/319066675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab00ca0e0 0x7fbab0157f30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 -- 172.21.15.46:0/319066675 >> 172.21.15.46:0/319066675 conn(0x7fbab006bfe0 msgr2=0x7fbab01524c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:13.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 -- 172.21.15.46:0/319066675 shutdown_connections 2023-10-30T13:23:13.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.132+0000 7fbab7bb8640 1 -- 172.21.15.46:0/319066675 wait complete. 2023-10-30T13:23:13.159 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:23:06.788949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:23:13.160 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:23:13.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe3642d6640 1 Processor -- start 2023-10-30T13:23:13.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe3642d6640 1 -- start start 2023-10-30T13:23:13.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe3642d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35c153980 0x7fe35c0caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe3642d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c15a920 con 0x7fe35c0cb1e0 2023-10-30T13:23:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe3642d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c0cb930 con 0x7fe35c159930 2023-10-30T13:23:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe3642d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c159dc0 con 0x7fe35c153980 2023-10-30T13:23:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe362873640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe35c0cb1e0 0x7fe35c15d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45974/0 (socket says 172.21.15.46:45974) 2023-10-30T13:23:13.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe362873640 1 -- 172.21.15.46:0/2532401289 learned_addr learned my addr 172.21.15.46:0/2532401289 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe362072640 1 --2- 172.21.15.46:0/2532401289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35c153980 0x7fe35c0caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe361070640 1 -- 172.21.15.46:0/2532401289 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707040923 0 0) 0x7fe35c15a920 con 0x7fe35c0cb1e0 2023-10-30T13:23:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe361070640 1 -- 172.21.15.46:0/2532401289 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe348003530 con 0x7fe35c0cb1e0 2023-10-30T13:23:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe361070640 1 -- 172.21.15.46:0/2532401289 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177799843 0 0) 0x7fe35c0cb930 con 0x7fe35c159930 2023-10-30T13:23:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe361070640 1 -- 172.21.15.46:0/2532401289 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe35c15a920 con 0x7fe35c159930 2023-10-30T13:23:13.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe362072640 1 -- 172.21.15.46:0/2532401289 >> v1:172.21.15.154:6789/0 conn(0x7fe35c159930 legacy=0x7fe35c0cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe361070640 1 -- 172.21.15.46:0/2532401289 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2675852029 0 0) 0x7fe348003530 con 0x7fe35c0cb1e0 2023-10-30T13:23:13.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe362072640 1 -- 172.21.15.46:0/2532401289 >> v1:172.21.15.46:6789/0 conn(0x7fe35c0cb1e0 legacy=0x7fe35c15d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.300+0000 7fe362072640 1 -- 172.21.15.46:0/2532401289 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c15a080 con 0x7fe35c153980 2023-10-30T13:23:13.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.304+0000 7fe362072640 1 --2- 172.21.15.46:0/2532401289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35c153980 0x7fe35c0caca0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3580097f0 tx=0x7fe35801c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f01b31845236e63a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:13.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.304+0000 7fe361070640 1 -- 172.21.15.46:0/2532401289 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354001a00 con 0x7fe35c153980 2023-10-30T13:23:13.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.304+0000 7fe361070640 1 -- 172.21.15.46:0/2532401289 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3580a7050 con 0x7fe35c153980 2023-10-30T13:23:13.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.304+0000 7fe361070640 1 -- 172.21.15.46:0/2532401289 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3580ab600 con 0x7fe35c153980 2023-10-30T13:23:13.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.304+0000 7fe3642d6640 1 -- 172.21.15.46:0/2532401289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35c153980 msgr2=0x7fe35c0caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.304+0000 7fe3642d6640 1 --2- 172.21.15.46:0/2532401289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35c153980 0x7fe35c0caca0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3580097f0 tx=0x7fe35801c3b0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.304+0000 7fe3642d6640 1 -- 172.21.15.46:0/2532401289 shutdown_connections 2023-10-30T13:23:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.304+0000 7fe3642d6640 1 --2- 172.21.15.46:0/2532401289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35c153980 0x7fe35c0caca0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.304+0000 7fe3642d6640 1 -- 172.21.15.46:0/2532401289 >> 172.21.15.46:0/2532401289 conn(0x7fe35c06c0f0 msgr2=0x7fe35c06c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:13.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.304+0000 7fe3642d6640 1 -- 172.21.15.46:0/2532401289 shutdown_connections 2023-10-30T13:23:13.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.304+0000 7fe3642d6640 1 -- 172.21.15.46:0/2532401289 wait complete. 2023-10-30T13:23:13.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe3642d6640 1 Processor -- start 2023-10-30T13:23:13.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe3642d6640 1 -- start start 2023-10-30T13:23:13.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe3642d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35c0cb1e0 0x7fe35c0c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe3642d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe35c153980 0x7fe35c148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe3642d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0ce150 0x7fe35c147a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe3642d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c1f2aa0 con 0x7fe35c0ce150 2023-10-30T13:23:13.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe3642d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c1f1f20 con 0x7fe35c153980 2023-10-30T13:23:13.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe3642d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c1f2200 con 0x7fe35c0cb1e0 2023-10-30T13:23:13.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe362873640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0ce150 0x7fe35c147a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe362873640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0ce150 0x7fe35c147a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55240/0 (socket says 172.21.15.46:55240) 2023-10-30T13:23:13.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe362873640 1 -- 172.21.15.46:0/64801828 learned_addr learned my addr 172.21.15.46:0/64801828 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:13.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe362072640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35c0cb1e0 0x7fe35c0c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe361871640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe35c153980 0x7fe35c148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe362873640 1 -- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35c0cb1e0 msgr2=0x7fe35c0c6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe362873640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35c0cb1e0 0x7fe35c0c6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe362873640 1 -- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe35c153980 msgr2=0x7fe35c148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe362873640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe35c153980 0x7fe35c148d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe362873640 1 -- 172.21.15.46:0/64801828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe358009420 con 0x7fe35c0ce150 2023-10-30T13:23:13.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe362072640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35c0cb1e0 0x7fe35c0c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:13.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe362873640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0ce150 0x7fe35c147a50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe35400cc20 tx=0x7fe35400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:13.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe352ffd640 1 -- 172.21.15.46:0/64801828 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400ce30 con 0x7fe35c0ce150 2023-10-30T13:23:13.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe3642d6640 1 -- 172.21.15.46:0/64801828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35c1f24e0 con 0x7fe35c0ce150 2023-10-30T13:23:13.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe352ffd640 1 -- 172.21.15.46:0/64801828 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe354012400 con 0x7fe35c0ce150 2023-10-30T13:23:13.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe352ffd640 1 -- 172.21.15.46:0/64801828 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354012790 con 0x7fe35c0ce150 2023-10-30T13:23:13.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.308+0000 7fe3642d6640 1 -- 172.21.15.46:0/64801828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35c147530 con 0x7fe35c0ce150 2023-10-30T13:23:13.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.312+0000 7fe3642d6640 1 -- 172.21.15.46:0/64801828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35c147750 con 0x7fe35c0ce150 2023-10-30T13:23:13.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.312+0000 7fe352ffd640 1 -- 172.21.15.46:0/64801828 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe3540128f0 con 0x7fe35c0ce150 2023-10-30T13:23:13.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.312+0000 7fe352ffd640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3400664d0 0x7fe340068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.312+0000 7fe352ffd640 1 -- 172.21.15.46:0/64801828 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe3540842a0 con 0x7fe35c0ce150 2023-10-30T13:23:13.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.312+0000 7fe362072640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3400664d0 0x7fe340068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.312+0000 7fe362072640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3400664d0 0x7fe340068990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe3580ad000 tx=0x7fe3580a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:13.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.316+0000 7fe352ffd640 1 -- 172.21.15.46:0/64801828 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe354051950 con 0x7fe35c0ce150 2023-10-30T13:23:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 -- 172.21.15.46:0/64801828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe35c153d60 con 0x7fe35c0ce150 2023-10-30T13:23:13.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe352ffd640 1 -- 172.21.15.46:0/64801828 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe354054f70 con 0x7fe35c0ce150 2023-10-30T13:23:13.478 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-10-30T13:23:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 -- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3400664d0 msgr2=0x7fe340068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3400664d0 0x7fe340068990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe3580ad000 tx=0x7fe3580a6000 comp rx=0 tx=0).stop 2023-10-30T13:23:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 -- 172.21.15.46:0/64801828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0ce150 msgr2=0x7fe35c147a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0ce150 0x7fe35c147a50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe35400cc20 tx=0x7fe35400a430 comp rx=0 tx=0).stop 2023-10-30T13:23:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 -- 172.21.15.46:0/64801828 shutdown_connections 2023-10-30T13:23:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3400664d0 0x7fe340068990 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0ce150 0x7fe35c147a50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe35c153980 0x7fe35c148d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 --2- 172.21.15.46:0/64801828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35c0cb1e0 0x7fe35c0c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 -- 172.21.15.46:0/64801828 >> 172.21.15.46:0/64801828 conn(0x7fe35c06c0f0 msgr2=0x7fe35c158970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 -- 172.21.15.46:0/64801828 shutdown_connections 2023-10-30T13:23:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.472+0000 7fe3642d6640 1 -- 172.21.15.46:0/64801828 wait complete. 2023-10-30T13:23:13.499 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:13.625 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:23:13.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:23:13.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81cdd9d640 1 Processor -- start 2023-10-30T13:23:13.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81cdd9d640 1 -- start start 2023-10-30T13:23:13.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81cdd9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f81c806f7a0 0x7f81c806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81cdd9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8071110 0x7f81c8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81cdd9d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c8066cf0 con 0x7f81c80c6bd0 2023-10-30T13:23:13.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81cdd9d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c80700c0 con 0x7f81c806f7a0 2023-10-30T13:23:13.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81cdd9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c80c70d0 con 0x7f81c8071110 2023-10-30T13:23:13.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81c6ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f81c80c6bd0 0x7f81c8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45976/0 (socket says 172.21.15.46:45976) 2023-10-30T13:23:13.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81c6ffd640 1 -- 172.21.15.46:0/2888264600 learned_addr learned my addr 172.21.15.46:0/2888264600 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:13.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81c7fff640 1 --2- 172.21.15.46:0/2888264600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8071110 0x7f81c8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81c77fe640 1 --2- 172.21.15.46:0/2888264600 >> v2:172.21.15.154:3300/0 conn(0x7f81c806f7a0 0x7f81c806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81c67fc640 1 -- 172.21.15.46:0/2888264600 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775777307 0 0) 0x7f81c8066cf0 con 0x7f81c80c6bd0 2023-10-30T13:23:13.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81c67fc640 1 -- 172.21.15.46:0/2888264600 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b0003530 con 0x7f81c80c6bd0 2023-10-30T13:23:13.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81c77fe640 1 -- 172.21.15.46:0/2888264600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8071110 msgr2=0x7f81c8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:13.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81c77fe640 1 --2- 172.21.15.46:0/2888264600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8071110 0x7f81c8158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81c77fe640 1 -- 172.21.15.46:0/2888264600 >> v1:172.21.15.46:6789/0 conn(0x7f81c80c6bd0 legacy=0x7f81c8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.636+0000 7f81c77fe640 1 -- 172.21.15.46:0/2888264600 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c80c7390 con 0x7f81c806f7a0 2023-10-30T13:23:13.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.640+0000 7f81c77fe640 1 --2- 172.21.15.46:0/2888264600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c806f7a0 0x7f81c806fb80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81b80097c0 tx=0x7f81b801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfe414882e729afb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:13.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.640+0000 7f81c67fc640 1 -- 172.21.15.46:0/2888264600 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b80a8970 con 0x7f81c806f7a0 2023-10-30T13:23:13.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.640+0000 7f81c67fc640 1 -- 172.21.15.46:0/2888264600 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b80a8ad0 con 0x7f81c806f7a0 2023-10-30T13:23:13.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.640+0000 7f81c67fc640 1 -- 172.21.15.46:0/2888264600 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b80a8e40 con 0x7f81c806f7a0 2023-10-30T13:23:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.640+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2888264600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c806f7a0 msgr2=0x7f81c806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.640+0000 7f81cdd9d640 1 --2- 172.21.15.46:0/2888264600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c806f7a0 0x7f81c806fb80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81b80097c0 tx=0x7f81b801ec70 comp rx=0 tx=0).stop 2023-10-30T13:23:13.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.640+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2888264600 shutdown_connections 2023-10-30T13:23:13.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.640+0000 7f81cdd9d640 1 --2- 172.21.15.46:0/2888264600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8071110 0x7f81c8158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.640+0000 7f81cdd9d640 1 --2- 172.21.15.46:0/2888264600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c806f7a0 0x7f81c806fb80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.640+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2888264600 >> 172.21.15.46:0/2888264600 conn(0x7f81c80c7910 msgr2=0x7f81c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:13.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.640+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2888264600 shutdown_connections 2023-10-30T13:23:13.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.640+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2888264600 wait complete. 2023-10-30T13:23:13.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81cdd9d640 1 Processor -- start 2023-10-30T13:23:13.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81cdd9d640 1 -- start start 2023-10-30T13:23:13.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81cdd9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c806f7a0 0x7f81c8154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81cdd9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8071110 0x7f81c80aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81cdd9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c80c6bd0 0x7f81c8152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81cdd9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8155110 con 0x7f81c806f7a0 2023-10-30T13:23:13.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81cdd9d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c81546f0 con 0x7f81c80c6bd0 2023-10-30T13:23:13.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81cdd9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8157910 con 0x7f81c8071110 2023-10-30T13:23:13.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c806f7a0 0x7f81c8154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c806f7a0 0x7f81c8154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55242/0 (socket says 172.21.15.46:55242) 2023-10-30T13:23:13.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81c77fe640 1 -- 172.21.15.46:0/2911666183 learned_addr learned my addr 172.21.15.46:0/2911666183 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:13.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81c7fff640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c80c6bd0 0x7f81c8152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81c6ffd640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8071110 0x7f81c80aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81c77fe640 1 -- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8071110 msgr2=0x7f81c80aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81c77fe640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8071110 0x7f81c80aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81c77fe640 1 -- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c80c6bd0 msgr2=0x7f81c8152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81c77fe640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c80c6bd0 0x7f81c8152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81c77fe640 1 -- 172.21.15.46:0/2911666183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b8009420 con 0x7f81c806f7a0 2023-10-30T13:23:13.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81c77fe640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c806f7a0 0x7f81c8154a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f81b801c8a0 tx=0x7f81b801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:13.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81affff640 1 -- 172.21.15.46:0/2911666183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8004450 con 0x7f81c806f7a0 2023-10-30T13:23:13.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81affff640 1 -- 172.21.15.46:0/2911666183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b8003620 con 0x7f81c806f7a0 2023-10-30T13:23:13.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81affff640 1 -- 172.21.15.46:0/2911666183 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8003990 con 0x7f81c806f7a0 2023-10-30T13:23:13.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2911666183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c8157bf0 con 0x7f81c806f7a0 2023-10-30T13:23:13.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.644+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2911666183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c81581b0 con 0x7f81c806f7a0 2023-10-30T13:23:13.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.648+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2911666183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c8070330 con 0x7f81c806f7a0 2023-10-30T13:23:13.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.648+0000 7f81affff640 1 -- 172.21.15.46:0/2911666183 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f81b80b7040 con 0x7f81c806f7a0 2023-10-30T13:23:13.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.648+0000 7f81affff640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81b40664a0 0x7f81b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.648+0000 7f81affff640 1 -- 172.21.15.46:0/2911666183 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f81b81211d0 con 0x7f81c806f7a0 2023-10-30T13:23:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.648+0000 7f81c6ffd640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81b40664a0 0x7f81b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.648+0000 7f81c6ffd640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81b40664a0 0x7f81b4068960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f81c814feb0 tx=0x7f81bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:13.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.652+0000 7f81affff640 1 -- 172.21.15.46:0/2911666183 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f81c8070330 con 0x7f81c806f7a0 2023-10-30T13:23:13.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2911666183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81c8070330 con 0x7f81c806f7a0 2023-10-30T13:23:13.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81affff640 1 -- 172.21.15.46:0/2911666183 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f81c8070330 con 0x7f81c806f7a0 2023-10-30T13:23:13.851 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:13.851 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-30T13:23:13.852 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 2 2023-10-30T13:23:13.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81b40664a0 msgr2=0x7f81b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81b40664a0 0x7f81b4068960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f81c814feb0 tx=0x7f81bc009040 comp rx=0 tx=0).stop 2023-10-30T13:23:13.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2911666183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c806f7a0 msgr2=0x7f81c8154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c806f7a0 0x7f81c8154a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f81b801c8a0 tx=0x7f81b801ba10 comp rx=0 tx=0).stop 2023-10-30T13:23:13.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2911666183 shutdown_connections 2023-10-30T13:23:13.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81b40664a0 0x7f81b4068960 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c80c6bd0 0x7f81c8152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8071110 0x7f81c80aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 --2- 172.21.15.46:0/2911666183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c806f7a0 0x7f81c8154a60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2911666183 >> 172.21.15.46:0/2911666183 conn(0x7f81c80c7910 msgr2=0x7f81c80c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:13.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2911666183 shutdown_connections 2023-10-30T13:23:13.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:13.844+0000 7f81cdd9d640 1 -- 172.21.15.46:0/2911666183 wait complete. 2023-10-30T13:23:15.873 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:16.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.020+0000 7fb401bd3640 1 Processor -- start 2023-10-30T13:23:16.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.020+0000 7fb401bd3640 1 -- start start 2023-10-30T13:23:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb401bd3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3fc0c05d0 0x7fb3fc13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb401bd3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb3fc0c6f60 0x7fb3fc13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb401bd3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc0c0a50 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb401bd3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3fc06f6e0 con 0x7fb3fc0c05d0 2023-10-30T13:23:16.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb401bd3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3fc06f9c0 con 0x7fb3fc0c6f60 2023-10-30T13:23:16.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb3faffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3fc0c05d0 0x7fb3fc13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb3faffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3fc0c05d0 0x7fb3fc13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55248/0 (socket says 172.21.15.46:55248) 2023-10-30T13:23:16.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb3faffd640 1 -- 172.21.15.46:0/1510130649 learned_addr learned my addr 172.21.15.46:0/1510130649 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:16.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb3fbfff640 1 --2- 172.21.15.46:0/1510130649 >> v2:172.21.15.154:3300/0 conn(0x7fb3fc0c6f60 0x7fb3fc13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb3faffd640 1 -- 172.21.15.46:0/1510130649 >> v2:172.21.15.154:3300/0 conn(0x7fb3fc0c6f60 msgr2=0x7fb3fc13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:16.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb3faffd640 1 --2- 172.21.15.46:0/1510130649 >> v2:172.21.15.154:3300/0 conn(0x7fb3fc0c6f60 0x7fb3fc13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb3faffd640 1 -- 172.21.15.46:0/1510130649 >> v1:172.21.15.154:6789/0 conn(0x7fb3fc0c6b80 legacy=0x7fb3fc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.024+0000 7fb3faffd640 1 -- 172.21.15.46:0/1510130649 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc06fca0 con 0x7fb3fc0c05d0 2023-10-30T13:23:16.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb3faffd640 1 --2- 172.21.15.46:0/1510130649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3fc0c05d0 0x7fb3fc13ed90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0c89b0 tx=0x7fb3f001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe176a736e475af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:16.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb3fa7fc640 1 -- 172.21.15.46:0/1510130649 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f00ab610 con 0x7fb3fc0c05d0 2023-10-30T13:23:16.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb3fa7fc640 1 -- 172.21.15.46:0/1510130649 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f00ab770 con 0x7fb3fc0c05d0 2023-10-30T13:23:16.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb3fa7fc640 1 -- 172.21.15.46:0/1510130649 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f00abb00 con 0x7fb3fc0c05d0 2023-10-30T13:23:16.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb401bd3640 1 -- 172.21.15.46:0/1510130649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3fc0c05d0 msgr2=0x7fb3fc13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb401bd3640 1 --2- 172.21.15.46:0/1510130649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3fc0c05d0 0x7fb3fc13ed90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0c89b0 tx=0x7fb3f001c500 comp rx=0 tx=0).stop 2023-10-30T13:23:16.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb401bd3640 1 -- 172.21.15.46:0/1510130649 shutdown_connections 2023-10-30T13:23:16.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb401bd3640 1 --2- 172.21.15.46:0/1510130649 >> v2:172.21.15.154:3300/0 conn(0x7fb3fc0c6f60 0x7fb3fc13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb401bd3640 1 --2- 172.21.15.46:0/1510130649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3fc0c05d0 0x7fb3fc13ed90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb401bd3640 1 -- 172.21.15.46:0/1510130649 >> 172.21.15.46:0/1510130649 conn(0x7fb3fc0c7910 msgr2=0x7fb3fc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:16.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb401bd3640 1 -- 172.21.15.46:0/1510130649 shutdown_connections 2023-10-30T13:23:16.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb401bd3640 1 -- 172.21.15.46:0/1510130649 wait complete. 2023-10-30T13:23:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb401bd3640 1 Processor -- start 2023-10-30T13:23:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb401bd3640 1 -- start start 2023-10-30T13:23:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.028+0000 7fb401bd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3fc0c05d0 0x7fb3fc155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb401bd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3fc0c6b80 0x7fb3fc0c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb401bd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3fc0c6f60 0x7fb3fc0c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb401bd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc06a9c0 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb401bd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc06a6c0 con 0x7fb3fc0c6f60 2023-10-30T13:23:16.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb401bd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc1f6160 con 0x7fb3fc0c05d0 2023-10-30T13:23:16.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3faffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3fc0c6b80 0x7fb3fc0c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3faffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3fc0c6b80 0x7fb3fc0c8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55254/0 (socket says 172.21.15.46:55254) 2023-10-30T13:23:16.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3faffd640 1 -- 172.21.15.46:0/2325787042 learned_addr learned my addr 172.21.15.46:0/2325787042 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:16.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3fb7fe640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3fc0c05d0 0x7fb3fc155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3fbfff640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3fc0c6f60 0x7fb3fc0c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3faffd640 1 -- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3fc0c05d0 msgr2=0x7fb3fc155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3faffd640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3fc0c05d0 0x7fb3fc155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3faffd640 1 -- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3fc0c6f60 msgr2=0x7fb3fc0c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3faffd640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3fc0c6f60 0x7fb3fc0c91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3faffd640 1 -- 172.21.15.46:0/2325787042 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f00094d0 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3faffd640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3fc0c6b80 0x7fb3fc0c8ca0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb3f001a130 tx=0x7fb3f001ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:16.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3e3fff640 1 -- 172.21.15.46:0/2325787042 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f00b4800 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3e3fff640 1 -- 172.21.15.46:0/2325787042 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f00b4980 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3e3fff640 1 -- 172.21.15.46:0/2325787042 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f00b4d30 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb401bd3640 1 -- 172.21.15.46:0/2325787042 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc135210 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb401bd3640 1 -- 172.21.15.46:0/2325787042 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc1354f0 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb401bd3640 1 -- 172.21.15.46:0/2325787042 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc0c7710 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.032+0000 7fb3e3fff640 1 -- 172.21.15.46:0/2325787042 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb3f00c7440 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.036+0000 7fb3e3fff640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3e80664a0 0x7fb3e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.036+0000 7fb3e3fff640 1 -- 172.21.15.46:0/2325787042 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb3f011e820 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.036+0000 7fb3fb7fe640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3e80664a0 0x7fb3e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.036+0000 7fb3fb7fe640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3e80664a0 0x7fb3e8068960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec001810 tx=0x7fb3ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:16.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.040+0000 7fb3e3fff640 1 -- 172.21.15.46:0/2325787042 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb3f00ec090 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 -- 172.21.15.46:0/2325787042 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3fc04e170 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb3e3fff640 1 -- 172.21.15.46:0/2325787042 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5271 (secure 0 0 0) 0x7fb3f00ef6b0 con 0x7fb3fc0c6b80 2023-10-30T13:23:16.235 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:16.235 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4271,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-10-30T13:23:16.236 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 3 2023-10-30T13:23:16.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 -- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3e80664a0 msgr2=0x7fb3e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3e80664a0 0x7fb3e8068960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec001810 tx=0x7fb3ec009040 comp rx=0 tx=0).stop 2023-10-30T13:23:16.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 -- 172.21.15.46:0/2325787042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3fc0c6b80 msgr2=0x7fb3fc0c8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3fc0c6b80 0x7fb3fc0c8ca0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb3f001a130 tx=0x7fb3f001ad40 comp rx=0 tx=0).stop 2023-10-30T13:23:16.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 -- 172.21.15.46:0/2325787042 shutdown_connections 2023-10-30T13:23:16.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3e80664a0 0x7fb3e8068960 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3fc0c6f60 0x7fb3fc0c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3fc0c6b80 0x7fb3fc0c8ca0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 --2- 172.21.15.46:0/2325787042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3fc0c05d0 0x7fb3fc155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 -- 172.21.15.46:0/2325787042 >> 172.21.15.46:0/2325787042 conn(0x7fb3fc0c7910 msgr2=0x7fb3fc1392c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:16.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 -- 172.21.15.46:0/2325787042 shutdown_connections 2023-10-30T13:23:16.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.228+0000 7fb401bd3640 1 -- 172.21.15.46:0/2325787042 wait complete. 2023-10-30T13:23:16.256 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:16.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa98b4c9640 1 Processor -- start 2023-10-30T13:23:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa98b4c9640 1 -- start start 2023-10-30T13:23:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa98b4c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9841538a0 0x7fa98415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa98b4c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa9840ca0e0 0x7fa98415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa98b4c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9840cb800 con 0x7fa9840cf160 2023-10-30T13:23:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa98b4c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98415dc30 con 0x7fa9840ca0e0 2023-10-30T13:23:16.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa98b4c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840cb330 con 0x7fa9841538a0 2023-10-30T13:23:16.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa989a66640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa9840cf160 0x7fa9840cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45980/0 (socket says 172.21.15.46:45980) 2023-10-30T13:23:16.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa989a66640 1 -- 172.21.15.46:0/1099085602 learned_addr learned my addr 172.21.15.46:0/1099085602 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:16.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa989265640 1 --2- 172.21.15.46:0/1099085602 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9841538a0 0x7fa98415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa988a64640 1 --2- 172.21.15.46:0/1099085602 >> v2:172.21.15.154:3300/0 conn(0x7fa9840ca0e0 0x7fa98415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa97bfff640 1 -- 172.21.15.46:0/1099085602 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423921982 0 0) 0x7fa9840cb800 con 0x7fa9840cf160 2023-10-30T13:23:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa97bfff640 1 -- 172.21.15.46:0/1099085602 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa970003530 con 0x7fa9840cf160 2023-10-30T13:23:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa988a64640 1 -- 172.21.15.46:0/1099085602 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9841538a0 msgr2=0x7fa98415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa97bfff640 1 -- 172.21.15.46:0/1099085602 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3728503944 0 0) 0x7fa970003530 con 0x7fa9840cf160 2023-10-30T13:23:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa988a64640 1 --2- 172.21.15.46:0/1099085602 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9841538a0 0x7fa98415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa988a64640 1 -- 172.21.15.46:0/1099085602 >> v1:172.21.15.46:6789/0 conn(0x7fa9840cf160 legacy=0x7fa9840cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa988a64640 1 -- 172.21.15.46:0/1099085602 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9840cad70 con 0x7fa9840ca0e0 2023-10-30T13:23:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.392+0000 7fa989265640 1 --2- 172.21.15.46:0/1099085602 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9841538a0 0x7fa98415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.396+0000 7fa988a64640 1 --2- 172.21.15.46:0/1099085602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9840ca0e0 0x7fa98415a090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa9740097c0 tx=0x7fa97401eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a67b294f25c9d47 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.396+0000 7fa97bfff640 1 -- 172.21.15.46:0/1099085602 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c001a00 con 0x7fa9840ca0e0 2023-10-30T13:23:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.396+0000 7fa97bfff640 1 -- 172.21.15.46:0/1099085602 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9740a85d0 con 0x7fa9840ca0e0 2023-10-30T13:23:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.396+0000 7fa97bfff640 1 -- 172.21.15.46:0/1099085602 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9740a8940 con 0x7fa9840ca0e0 2023-10-30T13:23:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.396+0000 7fa98b4c9640 1 -- 172.21.15.46:0/1099085602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9840ca0e0 msgr2=0x7fa98415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.396+0000 7fa98b4c9640 1 --2- 172.21.15.46:0/1099085602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9840ca0e0 0x7fa98415a090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa9740097c0 tx=0x7fa97401eb90 comp rx=0 tx=0).stop 2023-10-30T13:23:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.396+0000 7fa98b4c9640 1 -- 172.21.15.46:0/1099085602 shutdown_connections 2023-10-30T13:23:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.396+0000 7fa98b4c9640 1 --2- 172.21.15.46:0/1099085602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9840ca0e0 0x7fa98415a090 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.396+0000 7fa98b4c9640 1 --2- 172.21.15.46:0/1099085602 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9841538a0 0x7fa98415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.396+0000 7fa98b4c9640 1 -- 172.21.15.46:0/1099085602 >> 172.21.15.46:0/1099085602 conn(0x7fa98406bfe0 msgr2=0x7fa98406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa98b4c9640 1 -- 172.21.15.46:0/1099085602 shutdown_connections 2023-10-30T13:23:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa98b4c9640 1 -- 172.21.15.46:0/1099085602 wait complete. 2023-10-30T13:23:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa98b4c9640 1 Processor -- start 2023-10-30T13:23:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa98b4c9640 1 -- start start 2023-10-30T13:23:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa98b4c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9840ca0e0 0x7fa9840c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa98b4c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9840cf160 0x7fa9840c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa98b4c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9841538a0 0x7fa9841e4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa98b4c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840c7270 con 0x7fa9841538a0 2023-10-30T13:23:16.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa98b4c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840c74c0 con 0x7fa9840cf160 2023-10-30T13:23:16.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa98b4c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840c77a0 con 0x7fa9840ca0e0 2023-10-30T13:23:16.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa989a66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9841538a0 0x7fa9841e4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa989a66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9841538a0 0x7fa9841e4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55260/0 (socket says 172.21.15.46:55260) 2023-10-30T13:23:16.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa989a66640 1 -- 172.21.15.46:0/3952781994 learned_addr learned my addr 172.21.15.46:0/3952781994 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:16.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa988a64640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9840cf160 0x7fa9840c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa989265640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9840ca0e0 0x7fa9840c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa989a66640 1 -- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9840ca0e0 msgr2=0x7fa9840c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa989a66640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9840ca0e0 0x7fa9840c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa989a66640 1 -- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9840cf160 msgr2=0x7fa9840c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa989a66640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9840cf160 0x7fa9840c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa989a66640 1 -- 172.21.15.46:0/3952781994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa980009420 con 0x7fa9841538a0 2023-10-30T13:23:16.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa989a66640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9841538a0 0x7fa9841e4bf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa97c00cbf0 tx=0x7fa97c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:16.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa979ffb640 1 -- 172.21.15.46:0/3952781994 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c00ad50 con 0x7fa9841538a0 2023-10-30T13:23:16.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.400+0000 7fa979ffb640 1 -- 172.21.15.46:0/3952781994 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c012400 con 0x7fa9841538a0 2023-10-30T13:23:16.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.404+0000 7fa98b4c9640 1 -- 172.21.15.46:0/3952781994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa974009420 con 0x7fa9841538a0 2023-10-30T13:23:16.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.404+0000 7fa979ffb640 1 -- 172.21.15.46:0/3952781994 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c0127f0 con 0x7fa9841538a0 2023-10-30T13:23:16.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.404+0000 7fa98b4c9640 1 -- 172.21.15.46:0/3952781994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9841e43f0 con 0x7fa9841538a0 2023-10-30T13:23:16.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.404+0000 7fa98b4c9640 1 -- 172.21.15.46:0/3952781994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9840b2a50 con 0x7fa9841538a0 2023-10-30T13:23:16.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.404+0000 7fa979ffb640 1 -- 172.21.15.46:0/3952781994 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa97c012950 con 0x7fa9841538a0 2023-10-30T13:23:16.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.404+0000 7fa979ffb640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9680664d0 0x7fa968068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.404+0000 7fa979ffb640 1 -- 172.21.15.46:0/3952781994 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa97c084020 con 0x7fa9841538a0 2023-10-30T13:23:16.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.404+0000 7fa989265640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9680664d0 0x7fa968068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.408+0000 7fa989265640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9680664d0 0x7fa968068990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa980009cf0 tx=0x7fa980009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:16.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.408+0000 7fa979ffb640 1 -- 172.21.15.46:0/3952781994 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa97c051930 con 0x7fa9841538a0 2023-10-30T13:23:16.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.568+0000 7fa98b4c9640 1 -- 172.21.15.46:0/3952781994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa98404e170 con 0x7fa9841538a0 2023-10-30T13:23:16.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.568+0000 7fa979ffb640 1 -- 172.21.15.46:0/3952781994 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12850 (secure 0 0 0) 0x7fa97c054f50 con 0x7fa9841538a0 2023-10-30T13:23:16.576 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:16.576 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":16,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:09.200520+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.572+0000 7fa98b4c9640 1 -- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9680664d0 msgr2=0x7fa968068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.572+0000 7fa98b4c9640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9680664d0 0x7fa968068990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa980009cf0 tx=0x7fa980009190 comp rx=0 tx=0).stop 2023-10-30T13:23:16.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.572+0000 7fa98b4c9640 1 -- 172.21.15.46:0/3952781994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9841538a0 msgr2=0x7fa9841e4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.572+0000 7fa98b4c9640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9841538a0 0x7fa9841e4bf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa97c00cbf0 tx=0x7fa97c00a430 comp rx=0 tx=0).stop 2023-10-30T13:23:16.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.572+0000 7fa98b4c9640 1 -- 172.21.15.46:0/3952781994 shutdown_connections 2023-10-30T13:23:16.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.572+0000 7fa98b4c9640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9680664d0 0x7fa968068990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.572+0000 7fa98b4c9640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9841538a0 0x7fa9841e4bf0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.572+0000 7fa98b4c9640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9840cf160 0x7fa9840c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.572+0000 7fa98b4c9640 1 --2- 172.21.15.46:0/3952781994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9840ca0e0 0x7fa9840c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.572+0000 7fa98b4c9640 1 -- 172.21.15.46:0/3952781994 >> 172.21.15.46:0/3952781994 conn(0x7fa98406bfe0 msgr2=0x7fa9841515b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:16.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.572+0000 7fa98b4c9640 1 -- 172.21.15.46:0/3952781994 shutdown_connections 2023-10-30T13:23:16.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.572+0000 7fa98b4c9640 1 -- 172.21.15.46:0/3952781994 wait complete. 2023-10-30T13:23:16.598 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:23:06.788949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:23:16.598 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:23:16.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.736+0000 7f1c2c1d9640 1 Processor -- start 2023-10-30T13:23:16.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.736+0000 7f1c2c1d9640 1 -- start start 2023-10-30T13:23:16.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.740+0000 7f1c2c1d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c240cd6b0 0x7f1c2406afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.740+0000 7f1c2c1d9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2406b4e0 con 0x7f1c241570d0 2023-10-30T13:23:16.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.740+0000 7f1c2c1d9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c241599b0 con 0x7f1c2406c9e0 2023-10-30T13:23:16.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.740+0000 7f1c2c1d9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2415a630 con 0x7f1c240cd6b0 2023-10-30T13:23:16.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.740+0000 7f1c29f75640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1c241570d0 0x7f1c24071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53090/0 (socket says 172.21.15.46:53090) 2023-10-30T13:23:16.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.740+0000 7f1c29f75640 1 -- 172.21.15.46:0/1528214270 learned_addr learned my addr 172.21.15.46:0/1528214270 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:16.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.740+0000 7f1c2a776640 1 --2- 172.21.15.46:0/1528214270 >> v2:172.21.15.46:3300/0 conn(0x7f1c240cd6b0 0x7f1c2406afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.740+0000 7f1c2a776640 1 -- 172.21.15.46:0/1528214270 >> v1:172.21.15.154:6789/0 conn(0x7f1c2406c9e0 legacy=0x7f1c240ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.740+0000 7f1c2a776640 1 -- 172.21.15.46:0/1528214270 >> v1:172.21.15.46:6789/0 conn(0x7f1c241570d0 legacy=0x7f1c24071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.740+0000 7f1c2a776640 1 -- 172.21.15.46:0/1528214270 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c24159c90 con 0x7f1c240cd6b0 2023-10-30T13:23:16.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.740+0000 7f1c28f73640 1 -- 172.21.15.46:0/1528214270 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978216360 0 0) 0x7f1c20001120 con 0x7f1c241570d0 2023-10-30T13:23:16.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.740+0000 7f1c2a776640 1 --2- 172.21.15.46:0/1528214270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c240cd6b0 0x7f1c2406afa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c009d90 tx=0x7f1c1c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5758c51dac88b4de server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:16.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c28f73640 1 -- 172.21.15.46:0/1528214270 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c0af600 con 0x7f1c240cd6b0 2023-10-30T13:23:16.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c28f73640 1 -- 172.21.15.46:0/1528214270 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1c0ab050 con 0x7f1c240cd6b0 2023-10-30T13:23:16.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c28f73640 1 -- 172.21.15.46:0/1528214270 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c0af980 con 0x7f1c240cd6b0 2023-10-30T13:23:16.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/1528214270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c240cd6b0 msgr2=0x7f1c2406afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 --2- 172.21.15.46:0/1528214270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c240cd6b0 0x7f1c2406afa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c009d90 tx=0x7f1c1c01f130 comp rx=0 tx=0).stop 2023-10-30T13:23:16.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/1528214270 shutdown_connections 2023-10-30T13:23:16.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 --2- 172.21.15.46:0/1528214270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c240cd6b0 0x7f1c2406afa0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/1528214270 >> 172.21.15.46:0/1528214270 conn(0x7f1c24069e90 msgr2=0x7f1c2406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:16.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/1528214270 shutdown_connections 2023-10-30T13:23:16.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/1528214270 wait complete. 2023-10-30T13:23:16.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 Processor -- start 2023-10-30T13:23:16.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 -- start start 2023-10-30T13:23:16.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c2406c9e0 0x7f1c241e5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c240cd6b0 0x7f1c2414c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c241570d0 0x7f1c2414c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c24153220 con 0x7f1c241570d0 2023-10-30T13:23:16.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c241f3310 con 0x7f1c240cd6b0 2023-10-30T13:23:16.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2c1d9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c241f35f0 con 0x7f1c2406c9e0 2023-10-30T13:23:16.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.744+0000 7f1c2a776640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c241570d0 0x7f1c2414c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c2a776640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c241570d0 0x7f1c2414c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51748/0 (socket says 172.21.15.46:51748) 2023-10-30T13:23:16.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c2a776640 1 -- 172.21.15.46:0/2337258695 learned_addr learned my addr 172.21.15.46:0/2337258695 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:16.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c29f75640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c2406c9e0 0x7f1c241e5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c29774640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c240cd6b0 0x7f1c2414c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c2a776640 1 -- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c2406c9e0 msgr2=0x7f1c241e5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c2a776640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c2406c9e0 0x7f1c241e5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c2a776640 1 -- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c240cd6b0 msgr2=0x7f1c2414c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c2a776640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c240cd6b0 0x7f1c2414c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c2a776640 1 -- 172.21.15.46:0/2337258695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1c009a70 con 0x7f1c241570d0 2023-10-30T13:23:16.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c2a776640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c241570d0 0x7f1c2414c590 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c01cd60 tx=0x7f1c1c01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:16.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c1a7fc640 1 -- 172.21.15.46:0/2337258695 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c0b9610 con 0x7f1c241570d0 2023-10-30T13:23:16.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c1a7fc640 1 -- 172.21.15.46:0/2337258695 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1c0b9770 con 0x7f1c241570d0 2023-10-30T13:23:16.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/2337258695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c241f38d0 con 0x7f1c241570d0 2023-10-30T13:23:16.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c1a7fc640 1 -- 172.21.15.46:0/2337258695 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c0b9b00 con 0x7f1c241570d0 2023-10-30T13:23:16.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/2337258695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c241f3e90 con 0x7f1c241570d0 2023-10-30T13:23:16.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/2337258695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2404e170 con 0x7f1c241570d0 2023-10-30T13:23:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c1a7fc640 1 -- 172.21.15.46:0/2337258695 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1c1c0b6050 con 0x7f1c241570d0 2023-10-30T13:23:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c1a7fc640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c080664d0 0x7f1c08068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.748+0000 7f1c1a7fc640 1 -- 172.21.15.46:0/2337258695 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1c1c1221f0 con 0x7f1c241570d0 2023-10-30T13:23:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.752+0000 7f1c29f75640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c080664d0 0x7f1c08068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.752+0000 7f1c29f75640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c080664d0 0x7f1c08068990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1c20007fd0 tx=0x7f1c20007a20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:16.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.752+0000 7f1c1a7fc640 1 -- 172.21.15.46:0/2337258695 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1c1c0efc30 con 0x7f1c241570d0 2023-10-30T13:23:16.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.912+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/2337258695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1c240b1910 con 0x7f1c241570d0 2023-10-30T13:23:16.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.912+0000 7f1c1a7fc640 1 -- 172.21.15.46:0/2337258695 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1c1c0f3250 con 0x7f1c241570d0 2023-10-30T13:23:16.921 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-10-30T13:23:16.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.916+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c080664d0 msgr2=0x7f1c08068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.916+0000 7f1c2c1d9640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c080664d0 0x7f1c08068990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1c20007fd0 tx=0x7f1c20007a20 comp rx=0 tx=0).stop 2023-10-30T13:23:16.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.916+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/2337258695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c241570d0 msgr2=0x7f1c2414c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:16.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.916+0000 7f1c2c1d9640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c241570d0 0x7f1c2414c590 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c01cd60 tx=0x7f1c1c01bed0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.916+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/2337258695 shutdown_connections 2023-10-30T13:23:16.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.916+0000 7f1c2c1d9640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c080664d0 0x7f1c08068990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.916+0000 7f1c2c1d9640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c241570d0 0x7f1c2414c590 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.916+0000 7f1c2c1d9640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c240cd6b0 0x7f1c2414c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.916+0000 7f1c2c1d9640 1 --2- 172.21.15.46:0/2337258695 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c2406c9e0 0x7f1c241e5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:16.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.916+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/2337258695 >> 172.21.15.46:0/2337258695 conn(0x7f1c24069e90 msgr2=0x7f1c24071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:16.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.916+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/2337258695 shutdown_connections 2023-10-30T13:23:16.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:16.916+0000 7f1c2c1d9640 1 -- 172.21.15.46:0/2337258695 wait complete. 2023-10-30T13:23:16.942 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-30T13:23:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.076+0000 7f43cdb94640 1 Processor -- start 2023-10-30T13:23:17.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.076+0000 7f43cdb94640 1 -- start start 2023-10-30T13:23:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43cdb94640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43c81538a0 0x7f43c815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43cdb94640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80ca0e0 0x7f43c815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43cdb94640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c80cb800 con 0x7f43c80cf160 2023-10-30T13:23:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43cdb94640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c815dc30 con 0x7f43c81538a0 2023-10-30T13:23:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43cdb94640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c80cb330 con 0x7f43c80ca0e0 2023-10-30T13:23:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43c77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43c81538a0 0x7f43c815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43c77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43c81538a0 0x7f43c815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51762/0 (socket says 172.21.15.46:51762) 2023-10-30T13:23:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43c77fe640 1 -- 172.21.15.46:0/3930035992 learned_addr learned my addr 172.21.15.46:0/3930035992 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43c6ffd640 1 --2- 172.21.15.46:0/3930035992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80ca0e0 0x7f43c815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43c77fe640 1 -- 172.21.15.46:0/3930035992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80ca0e0 msgr2=0x7f43c815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43c77fe640 1 --2- 172.21.15.46:0/3930035992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80ca0e0 0x7f43c815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43c77fe640 1 -- 172.21.15.46:0/3930035992 >> v1:172.21.15.154:6789/0 conn(0x7f43c80cf160 legacy=0x7f43c80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.080+0000 7f43c77fe640 1 -- 172.21.15.46:0/3930035992 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c80cad70 con 0x7f43c81538a0 2023-10-30T13:23:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43c77fe640 1 --2- 172.21.15.46:0/3930035992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43c81538a0 0x7f43c815d6f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009870 tx=0x7f43b801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac1f54cfc98b2f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43c67fc640 1 -- 172.21.15.46:0/3930035992 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b80a8a30 con 0x7f43c81538a0 2023-10-30T13:23:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43c67fc640 1 -- 172.21.15.46:0/3930035992 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b80a8b90 con 0x7f43c81538a0 2023-10-30T13:23:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43c67fc640 1 -- 172.21.15.46:0/3930035992 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b80b0440 con 0x7f43c81538a0 2023-10-30T13:23:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43cdb94640 1 -- 172.21.15.46:0/3930035992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43c81538a0 msgr2=0x7f43c815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43cdb94640 1 --2- 172.21.15.46:0/3930035992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43c81538a0 0x7f43c815d6f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009870 tx=0x7f43b801ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43cdb94640 1 -- 172.21.15.46:0/3930035992 shutdown_connections 2023-10-30T13:23:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43cdb94640 1 --2- 172.21.15.46:0/3930035992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80ca0e0 0x7f43c815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43cdb94640 1 --2- 172.21.15.46:0/3930035992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43c81538a0 0x7f43c815d6f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43cdb94640 1 -- 172.21.15.46:0/3930035992 >> 172.21.15.46:0/3930035992 conn(0x7f43c806bfe0 msgr2=0x7f43c806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43cdb94640 1 -- 172.21.15.46:0/3930035992 shutdown_connections 2023-10-30T13:23:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43cdb94640 1 -- 172.21.15.46:0/3930035992 wait complete. 2023-10-30T13:23:17.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43cdb94640 1 Processor -- start 2023-10-30T13:23:17.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43cdb94640 1 -- start start 2023-10-30T13:23:17.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.084+0000 7f43cdb94640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43c80ca0e0 0x7f43c80ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43cdb94640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80cf160 0x7f43c81e3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43cdb94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43c81538a0 0x7f43c81e4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43cdb94640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c8159340 con 0x7f43c81538a0 2023-10-30T13:23:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43cdb94640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c8159620 con 0x7f43c80ca0e0 2023-10-30T13:23:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43cdb94640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c81e2b80 con 0x7f43c80cf160 2023-10-30T13:23:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43c81538a0 0x7f43c81e4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43c81538a0 0x7f43c81e4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51768/0 (socket says 172.21.15.46:51768) 2023-10-30T13:23:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43c7fff640 1 -- 172.21.15.46:0/4284288137 learned_addr learned my addr 172.21.15.46:0/4284288137 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43c6ffd640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80cf160 0x7f43c81e3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43c77fe640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43c80ca0e0 0x7f43c80ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43c7fff640 1 -- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80cf160 msgr2=0x7f43c81e3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43c7fff640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80cf160 0x7f43c81e3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43c7fff640 1 -- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43c80ca0e0 msgr2=0x7f43c80ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43c7fff640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43c80ca0e0 0x7f43c80ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43c7fff640 1 -- 172.21.15.46:0/4284288137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b80094d0 con 0x7f43c81538a0 2023-10-30T13:23:17.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43c7fff640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43c81538a0 0x7f43c81e4350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f43c000c6c0 tx=0x7f43c000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:17.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43affff640 1 -- 172.21.15.46:0/4284288137 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c0003b80 con 0x7f43c81538a0 2023-10-30T13:23:17.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43affff640 1 -- 172.21.15.46:0/4284288137 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c0003ce0 con 0x7f43c81538a0 2023-10-30T13:23:17.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43affff640 1 -- 172.21.15.46:0/4284288137 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c00052f0 con 0x7f43c81538a0 2023-10-30T13:23:17.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43cdb94640 1 -- 172.21.15.46:0/4284288137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c81e2e60 con 0x7f43c81538a0 2023-10-30T13:23:17.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43cdb94640 1 -- 172.21.15.46:0/4284288137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c81e38f0 con 0x7f43c81538a0 2023-10-30T13:23:17.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43cdb94640 1 -- 172.21.15.46:0/4284288137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c80b2a50 con 0x7f43c81538a0 2023-10-30T13:23:17.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.088+0000 7f43affff640 1 -- 172.21.15.46:0/4284288137 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f43c0005450 con 0x7f43c81538a0 2023-10-30T13:23:17.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.092+0000 7f43affff640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43b40664a0 0x7f43b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.092+0000 7f43affff640 1 -- 172.21.15.46:0/4284288137 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f43c0083a10 con 0x7f43c81538a0 2023-10-30T13:23:17.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.092+0000 7f43c77fe640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43b40664a0 0x7f43b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.092+0000 7f43c77fe640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43b40664a0 0x7f43b4068960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f43b80b2000 tx=0x7f43b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:17.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.096+0000 7f43affff640 1 -- 172.21.15.46:0/4284288137 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f43c00510b0 con 0x7f43c81538a0 2023-10-30T13:23:17.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.252+0000 7f43cdb94640 1 -- 172.21.15.46:0/4284288137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f43c804e170 con 0x7f43c81538a0 2023-10-30T13:23:17.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.252+0000 7f43affff640 1 -- 172.21.15.46:0/4284288137 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f43c00546d0 con 0x7f43c81538a0 2023-10-30T13:23:17.261 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:17.261 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2023-10-30T13:23:17.261 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-10-30T13:23:17.261 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2023-10-30T13:23:17.262 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2023-10-30T13:23:17.262 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-10-30T13:23:17.262 INFO:teuthology.orchestra.run.smithi046.stdout:] 2023-10-30T13:23:17.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.256+0000 7f43cdb94640 1 -- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43b40664a0 msgr2=0x7f43b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.256+0000 7f43cdb94640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43b40664a0 0x7f43b4068960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f43b80b2000 tx=0x7f43b8008040 comp rx=0 tx=0).stop 2023-10-30T13:23:17.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.256+0000 7f43cdb94640 1 -- 172.21.15.46:0/4284288137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43c81538a0 msgr2=0x7f43c81e4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.256+0000 7f43cdb94640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43c81538a0 0x7f43c81e4350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f43c000c6c0 tx=0x7f43c000cb80 comp rx=0 tx=0).stop 2023-10-30T13:23:17.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.256+0000 7f43cdb94640 1 -- 172.21.15.46:0/4284288137 shutdown_connections 2023-10-30T13:23:17.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.256+0000 7f43cdb94640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43b40664a0 0x7f43b4068960 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.256+0000 7f43cdb94640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43c81538a0 0x7f43c81e4350 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.256+0000 7f43cdb94640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80cf160 0x7f43c81e3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.256+0000 7f43cdb94640 1 --2- 172.21.15.46:0/4284288137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43c80ca0e0 0x7f43c80ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.256+0000 7f43cdb94640 1 -- 172.21.15.46:0/4284288137 >> 172.21.15.46:0/4284288137 conn(0x7f43c806bfe0 msgr2=0x7f43c8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:17.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.256+0000 7f43cdb94640 1 -- 172.21.15.46:0/4284288137 shutdown_connections 2023-10-30T13:23:17.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.256+0000 7f43cdb94640 1 -- 172.21.15.46:0/4284288137 wait complete. 2023-10-30T13:23:17.281 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-30T13:23:17.282 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-30T13:23:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.420+0000 7f2536e54640 1 Processor -- start 2023-10-30T13:23:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.420+0000 7f2536e54640 1 -- start start 2023-10-30T13:23:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f2536e54640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f253014cf40 0x7f2530069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f2536e54640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253015a270 con 0x7f253015a700 2023-10-30T13:23:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f2536e54640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253006c7c0 con 0x7f25300c8430 2023-10-30T13:23:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f2536e54640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253006a3d0 con 0x7f253014cf40 2023-10-30T13:23:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f253015a700 0x7f2530156490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53096/0 (socket says 172.21.15.46:53096) 2023-10-30T13:23:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252ffff640 1 -- 172.21.15.46:0/3657923934 learned_addr learned my addr 172.21.15.46:0/3657923934 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f2534bf0640 1 --2- 172.21.15.46:0/3657923934 >> v2:172.21.15.154:3300/0 conn(0x7f253014cf40 0x7f2530069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353347388 0 0) 0x7f253015a270 con 0x7f253015a700 2023-10-30T13:23:17.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2520003530 con 0x7f253015a700 2023-10-30T13:23:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1994943757 0 0) 0x7f2520003530 con 0x7f253015a700 2023-10-30T13:23:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 >> v2:172.21.15.154:3300/0 conn(0x7f253014cf40 msgr2=0x7f2530069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 --2- 172.21.15.46:0/3657923934 >> v2:172.21.15.154:3300/0 conn(0x7f253014cf40 0x7f2530069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 >> v1:172.21.15.154:6789/0 conn(0x7f25300c8430 legacy=0x7f25300c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2530156ed0 con 0x7f253015a700 2023-10-30T13:23:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f251c001680 con 0x7f253015a700 2023-10-30T13:23:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 >> v1:172.21.15.46:6789/0 conn(0x7f253015a700 legacy=0x7f2530156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 --2- 172.21.15.46:0/3657923934 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2520003980 0x7f2520003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 --2- 172.21.15.46:0/3657923934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25200047d0 0x7f2520004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 --2- 172.21.15.46:0/3657923934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2520005190 0x7f252000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253006a3d0 con 0x7f2520005190 2023-10-30T13:23:17.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252000c030 con 0x7f25200047d0 2023-10-30T13:23:17.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252000c1f0 con 0x7f2520003980 2023-10-30T13:23:17.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f25353f1640 1 --2- 172.21.15.46:0/3657923934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2520005190 0x7f252000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f2534bf0640 1 --2- 172.21.15.46:0/3657923934 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2520003980 0x7f2520003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252ffff640 1 --2- 172.21.15.46:0/3657923934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25200047d0 0x7f2520004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f25353f1640 1 -- 172.21.15.46:0/3657923934 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2520003980 msgr2=0x7f2520003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f25353f1640 1 --2- 172.21.15.46:0/3657923934 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2520003980 0x7f2520003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f25353f1640 1 -- 172.21.15.46:0/3657923934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25200047d0 msgr2=0x7f2520004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f25353f1640 1 --2- 172.21.15.46:0/3657923934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25200047d0 0x7f2520004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f25353f1640 1 -- 172.21.15.46:0/3657923934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2520003530 con 0x7f2520005190 2023-10-30T13:23:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.424+0000 7f252ffff640 1 --2- 172.21.15.46:0/3657923934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25200047d0 0x7f2520004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:17.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.428+0000 7f25353f1640 1 -- 172.21.15.46:0/3657923934 reap_dead start 2023-10-30T13:23:17.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.428+0000 7f25353f1640 1 --2- 172.21.15.46:0/3657923934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2520005190 0x7f252000b990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2528001e40 tx=0x7f252801c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc538e4e72304c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:17.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.428+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528003b80 con 0x7f2520005190 2023-10-30T13:23:17.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.428+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2528003ce0 con 0x7f2520005190 2023-10-30T13:23:17.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.428+0000 7f252f7fe640 1 -- 172.21.15.46:0/3657923934 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528005260 con 0x7f2520005190 2023-10-30T13:23:17.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.428+0000 7f2536e54640 1 -- 172.21.15.46:0/3657923934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2520005190 msgr2=0x7f252000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.428+0000 7f2536e54640 1 --2- 172.21.15.46:0/3657923934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2520005190 0x7f252000b990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2528001e40 tx=0x7f252801c740 comp rx=0 tx=0).stop 2023-10-30T13:23:17.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.428+0000 7f2536e54640 1 -- 172.21.15.46:0/3657923934 shutdown_connections 2023-10-30T13:23:17.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.428+0000 7f2536e54640 1 --2- 172.21.15.46:0/3657923934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2520005190 0x7f252000b990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.428+0000 7f2536e54640 1 -- 172.21.15.46:0/3657923934 >> 172.21.15.46:0/3657923934 conn(0x7f25300c7910 msgr2=0x7f25300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:17.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.428+0000 7f2536e54640 1 -- 172.21.15.46:0/3657923934 shutdown_connections 2023-10-30T13:23:17.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.428+0000 7f2536e54640 1 -- 172.21.15.46:0/3657923934 wait complete. 2023-10-30T13:23:17.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f2536e54640 1 Processor -- start 2023-10-30T13:23:17.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f2536e54640 1 -- start start 2023-10-30T13:23:17.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f2536e54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2520005190 0x7f25301eae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f2536e54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25301eb340 0x7f25300c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f2536e54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25300ca2d0 0x7f25301588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f2536e54640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25301ea660 con 0x7f25300ca2d0 2023-10-30T13:23:17.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f2536e54640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25300c2fd0 con 0x7f25301eb340 2023-10-30T13:23:17.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f2536e54640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25300c55a0 con 0x7f2520005190 2023-10-30T13:23:17.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f25353f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25300ca2d0 0x7f25301588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f25353f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25300ca2d0 0x7f25301588b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51794/0 (socket says 172.21.15.46:51794) 2023-10-30T13:23:17.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f25353f1640 1 -- 172.21.15.46:0/3586621707 learned_addr learned my addr 172.21.15.46:0/3586621707 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:17.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f252ffff640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25301eb340 0x7f25300c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f2534bf0640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2520005190 0x7f25301eae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f25353f1640 1 -- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2520005190 msgr2=0x7f25301eae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f25353f1640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2520005190 0x7f25301eae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f25353f1640 1 -- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25301eb340 msgr2=0x7f25300c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f25353f1640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25301eb340 0x7f25300c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f25353f1640 1 -- 172.21.15.46:0/3586621707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25301ee780 con 0x7f25300ca2d0 2023-10-30T13:23:17.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f25353f1640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25300ca2d0 0x7f25301588b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f252801c570 tx=0x7f2520005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:17.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f252d7fa640 1 -- 172.21.15.46:0/3586621707 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25280b4070 con 0x7f25300ca2d0 2023-10-30T13:23:17.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f252d7fa640 1 -- 172.21.15.46:0/3586621707 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2528005230 con 0x7f25300ca2d0 2023-10-30T13:23:17.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f252d7fa640 1 -- 172.21.15.46:0/3586621707 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25280055e0 con 0x7f25300ca2d0 2023-10-30T13:23:17.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f2536e54640 1 -- 172.21.15.46:0/3586621707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25300c5840 con 0x7f25300ca2d0 2023-10-30T13:23:17.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.432+0000 7f2536e54640 1 -- 172.21.15.46:0/3586621707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25301eea60 con 0x7f25300ca2d0 2023-10-30T13:23:17.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.436+0000 7f2536e54640 1 -- 172.21.15.46:0/3586621707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25301516c0 con 0x7f25300ca2d0 2023-10-30T13:23:17.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.436+0000 7f252d7fa640 1 -- 172.21.15.46:0/3586621707 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25280a6440 con 0x7f25300ca2d0 2023-10-30T13:23:17.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.436+0000 7f252d7fa640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25140664a0 0x7f2514068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:17.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.436+0000 7f252d7fa640 1 -- 172.21.15.46:0/3586621707 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f252811c9c0 con 0x7f25300ca2d0 2023-10-30T13:23:17.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.436+0000 7f2534bf0640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25140664a0 0x7f2514068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:17.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.436+0000 7f2534bf0640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25140664a0 0x7f2514068960 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2524009aa0 tx=0x7f2524008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:17.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.440+0000 7f252d7fa640 1 -- 172.21.15.46:0/3586621707 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25280ea0f0 con 0x7f25300ca2d0 2023-10-30T13:23:17.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.596+0000 7f2536e54640 1 -- 172.21.15.46:0/3586621707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f25300c5a30 con 0x7f25300ca2d0 2023-10-30T13:23:17.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.920+0000 7f252d7fa640 1 -- 172.21.15.46:0/3586621707 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f25280ed710 con 0x7f25300ca2d0 2023-10-30T13:23:17.929 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2023-10-30T13:23:17.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.924+0000 7f2536e54640 1 -- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25140664a0 msgr2=0x7f2514068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.924+0000 7f2536e54640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25140664a0 0x7f2514068960 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2524009aa0 tx=0x7f2524008040 comp rx=0 tx=0).stop 2023-10-30T13:23:17.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.924+0000 7f2536e54640 1 -- 172.21.15.46:0/3586621707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25300ca2d0 msgr2=0x7f25301588b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:17.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.924+0000 7f2536e54640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25300ca2d0 0x7f25301588b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f252801c570 tx=0x7f2520005800 comp rx=0 tx=0).stop 2023-10-30T13:23:17.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.924+0000 7f2536e54640 1 -- 172.21.15.46:0/3586621707 shutdown_connections 2023-10-30T13:23:17.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.924+0000 7f2536e54640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25140664a0 0x7f2514068960 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.924+0000 7f2536e54640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25300ca2d0 0x7f25301588b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.924+0000 7f2536e54640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25301eb340 0x7f25300c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.924+0000 7f2536e54640 1 --2- 172.21.15.46:0/3586621707 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2520005190 0x7f25301eae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:17.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.924+0000 7f2536e54640 1 -- 172.21.15.46:0/3586621707 >> 172.21.15.46:0/3586621707 conn(0x7f25300c7910 msgr2=0x7f253006a510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:17.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.924+0000 7f2536e54640 1 -- 172.21.15.46:0/3586621707 shutdown_connections 2023-10-30T13:23:17.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:17.924+0000 7f2536e54640 1 -- 172.21.15.46:0/3586621707 wait complete. 2023-10-30T13:23:17.957 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-30T13:23:18.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.100+0000 7f1c3b409640 1 Processor -- start 2023-10-30T13:23:18.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.100+0000 7f1c3b409640 1 -- start start 2023-10-30T13:23:18.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.100+0000 7f1c3b409640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c34151650 0x7f1c340c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.100+0000 7f1c3b409640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c3406fe40 0x7f1c34157670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c3b409640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c340c9200 con 0x7f1c340c1f50 2023-10-30T13:23:18.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c3b409640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c340c8ed0 con 0x7f1c34151650 2023-10-30T13:23:18.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c3b409640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c340c3b10 con 0x7f1c3406fe40 2023-10-30T13:23:18.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c391a5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c34151650 0x7f1c340c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c391a5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c34151650 0x7f1c340c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51804/0 (socket says 172.21.15.46:51804) 2023-10-30T13:23:18.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c391a5640 1 -- 172.21.15.46:0/3232985000 learned_addr learned my addr 172.21.15.46:0/3232985000 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:18.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c389a4640 1 --2- 172.21.15.46:0/3232985000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c3406fe40 0x7f1c34157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c391a5640 1 -- 172.21.15.46:0/3232985000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c3406fe40 msgr2=0x7f1c34157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:18.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c391a5640 1 --2- 172.21.15.46:0/3232985000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c3406fe40 0x7f1c34157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c391a5640 1 -- 172.21.15.46:0/3232985000 >> v1:172.21.15.154:6789/0 conn(0x7f1c340c1f50 legacy=0x7f1c3406f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:18.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c391a5640 1 -- 172.21.15.46:0/3232985000 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c340c3df0 con 0x7f1c34151650 2023-10-30T13:23:18.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c391a5640 1 --2- 172.21.15.46:0/3232985000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c34151650 0x7f1c340c7990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1c30009870 tx=0x7f1c3001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82972e851c83d2f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:18.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c2bfff640 1 -- 172.21.15.46:0/3232985000 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c300a89b0 con 0x7f1c34151650 2023-10-30T13:23:18.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c2bfff640 1 -- 172.21.15.46:0/3232985000 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c300a8b10 con 0x7f1c34151650 2023-10-30T13:23:18.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.104+0000 7f1c2bfff640 1 -- 172.21.15.46:0/3232985000 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c300a8ea0 con 0x7f1c34151650 2023-10-30T13:23:18.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 -- 172.21.15.46:0/3232985000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c34151650 msgr2=0x7f1c340c7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:18.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 --2- 172.21.15.46:0/3232985000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c34151650 0x7f1c340c7990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1c30009870 tx=0x7f1c3001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 -- 172.21.15.46:0/3232985000 shutdown_connections 2023-10-30T13:23:18.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 --2- 172.21.15.46:0/3232985000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c3406fe40 0x7f1c34157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 --2- 172.21.15.46:0/3232985000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c34151650 0x7f1c340c7990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 -- 172.21.15.46:0/3232985000 >> 172.21.15.46:0/3232985000 conn(0x7f1c34069e90 msgr2=0x7f1c3406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:18.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 -- 172.21.15.46:0/3232985000 shutdown_connections 2023-10-30T13:23:18.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 -- 172.21.15.46:0/3232985000 wait complete. 2023-10-30T13:23:18.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 Processor -- start 2023-10-30T13:23:18.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 -- start start 2023-10-30T13:23:18.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c3406fe40 0x7f1c341f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c340c1f50 0x7f1c34144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c34151650 0x7f1c340cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c341e34c0 con 0x7f1c340c1f50 2023-10-30T13:23:18.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c341f2850 con 0x7f1c3406fe40 2023-10-30T13:23:18.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c3b409640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3414f150 con 0x7f1c34151650 2023-10-30T13:23:18.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c389a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c340c1f50 0x7f1c34144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c389a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c340c1f50 0x7f1c34144650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51806/0 (socket says 172.21.15.46:51806) 2023-10-30T13:23:18.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c389a4640 1 -- 172.21.15.46:0/1094026168 learned_addr learned my addr 172.21.15.46:0/1094026168 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:18.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c391a5640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c3406fe40 0x7f1c341f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c399a6640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c34151650 0x7f1c340cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c389a4640 1 -- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c34151650 msgr2=0x7f1c340cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:18.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c389a4640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c34151650 0x7f1c340cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c389a4640 1 -- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c3406fe40 msgr2=0x7f1c341f2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:18.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.108+0000 7f1c389a4640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c3406fe40 0x7f1c341f2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.112+0000 7f1c389a4640 1 -- 172.21.15.46:0/1094026168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c300094d0 con 0x7f1c340c1f50 2023-10-30T13:23:18.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.112+0000 7f1c389a4640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c340c1f50 0x7f1c34144650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400d7d0 tx=0x7f1c2400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:18.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.112+0000 7f1c29ffb640 1 -- 172.21.15.46:0/1094026168 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c24018070 con 0x7f1c340c1f50 2023-10-30T13:23:18.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.112+0000 7f1c29ffb640 1 -- 172.21.15.46:0/1094026168 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c24004020 con 0x7f1c340c1f50 2023-10-30T13:23:18.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.112+0000 7f1c29ffb640 1 -- 172.21.15.46:0/1094026168 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c240041d0 con 0x7f1c340c1f50 2023-10-30T13:23:18.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.112+0000 7f1c3b409640 1 -- 172.21.15.46:0/1094026168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3414e8d0 con 0x7f1c340c1f50 2023-10-30T13:23:18.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.112+0000 7f1c3b409640 1 -- 172.21.15.46:0/1094026168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3414ee70 con 0x7f1c340c1f50 2023-10-30T13:23:18.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.112+0000 7f1c3b409640 1 -- 172.21.15.46:0/1094026168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3404e170 con 0x7f1c340c1f50 2023-10-30T13:23:18.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.112+0000 7f1c29ffb640 1 -- 172.21.15.46:0/1094026168 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1c24004330 con 0x7f1c340c1f50 2023-10-30T13:23:18.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.112+0000 7f1c29ffb640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c180664a0 0x7f1c18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.112+0000 7f1c29ffb640 1 -- 172.21.15.46:0/1094026168 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f1c24083b80 con 0x7f1c340c1f50 2023-10-30T13:23:18.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.112+0000 7f1c391a5640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c180664a0 0x7f1c18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.116+0000 7f1c391a5640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c180664a0 0x7f1c18068960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1c300b2000 tx=0x7f1c30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:18.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.116+0000 7f1c29ffb640 1 -- 172.21.15.46:0/1094026168 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1c240511e0 con 0x7f1c340c1f50 2023-10-30T13:23:18.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.276+0000 7f1c3b409640 1 -- 172.21.15.46:0/1094026168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1c340724d0 con 0x7f1c340c1f50 2023-10-30T13:23:18.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.924+0000 7f1c29ffb640 1 -- 172.21.15.46:0/1094026168 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1c24054800 con 0x7f1c340c1f50 2023-10-30T13:23:18.932 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2023-10-30T13:23:18.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.928+0000 7f1c3b409640 1 -- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c180664a0 msgr2=0x7f1c18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:18.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.928+0000 7f1c3b409640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c180664a0 0x7f1c18068960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1c300b2000 tx=0x7f1c30008040 comp rx=0 tx=0).stop 2023-10-30T13:23:18.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.928+0000 7f1c3b409640 1 -- 172.21.15.46:0/1094026168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c340c1f50 msgr2=0x7f1c34144650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:18.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.928+0000 7f1c3b409640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c340c1f50 0x7f1c34144650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400d7d0 tx=0x7f1c2400dc90 comp rx=0 tx=0).stop 2023-10-30T13:23:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.928+0000 7f1c3b409640 1 -- 172.21.15.46:0/1094026168 shutdown_connections 2023-10-30T13:23:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.928+0000 7f1c3b409640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c180664a0 0x7f1c18068960 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.928+0000 7f1c3b409640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c34151650 0x7f1c340cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.928+0000 7f1c3b409640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c340c1f50 0x7f1c34144650 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.928+0000 7f1c3b409640 1 --2- 172.21.15.46:0/1094026168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c3406fe40 0x7f1c341f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.928+0000 7f1c3b409640 1 -- 172.21.15.46:0/1094026168 >> 172.21.15.46:0/1094026168 conn(0x7f1c34069e90 msgr2=0x7f1c34065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.928+0000 7f1c3b409640 1 -- 172.21.15.46:0/1094026168 shutdown_connections 2023-10-30T13:23:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:18.928+0000 7f1c3b409640 1 -- 172.21.15.46:0/1094026168 wait complete. 2023-10-30T13:23:18.978 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-30T13:23:19.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecf0d6640 1 Processor -- start 2023-10-30T13:23:19.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecf0d6640 1 -- start start 2023-10-30T13:23:19.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecf0d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b2300 0x7f7ec806afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:19.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecf0d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7ec806fdf0 0x7f7ec81598a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:19.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecf0d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec806f1d0 con 0x7f7ec8156db0 2023-10-30T13:23:19.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecf0d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec806b8a0 con 0x7f7ec806fdf0 2023-10-30T13:23:19.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecf0d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec815d9a0 con 0x7f7ec80b2300 2023-10-30T13:23:19.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecce72640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7ec8156db0 0x7f7ec8071b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53110/0 (socket says 172.21.15.46:53110) 2023-10-30T13:23:19.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecce72640 1 -- 172.21.15.46:0/3640852739 learned_addr learned my addr 172.21.15.46:0/3640852739 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:19.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec7fff640 1 --2- 172.21.15.46:0/3640852739 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b2300 0x7f7ec806afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:19.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecd673640 1 --2- 172.21.15.46:0/3640852739 >> v2:172.21.15.154:3300/0 conn(0x7f7ec806fdf0 0x7f7ec81598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:19.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130064675 0 0) 0x7f7ec806f1d0 con 0x7f7ec8156db0 2023-10-30T13:23:19.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb8003530 con 0x7f7ec8156db0 2023-10-30T13:23:19.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3255438536 0 0) 0x7f7eb8003530 con 0x7f7ec8156db0 2023-10-30T13:23:19.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b2300 msgr2=0x7f7ec806afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:19.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 --2- 172.21.15.46:0/3640852739 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b2300 0x7f7ec806afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:19.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 >> v2:172.21.15.154:3300/0 conn(0x7f7ec806fdf0 msgr2=0x7f7ec81598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:19.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 --2- 172.21.15.46:0/3640852739 >> v2:172.21.15.154:3300/0 conn(0x7f7ec806fdf0 0x7f7ec81598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:19.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec806f470 con 0x7f7ec8156db0 2023-10-30T13:23:19.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f7ebc001680 con 0x7f7ec8156db0 2023-10-30T13:23:19.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 >> v1:172.21.15.46:6789/0 conn(0x7f7ec8156db0 legacy=0x7f7ec8071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:19.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 --2- 172.21.15.46:0/3640852739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8003980 0x7f7eb8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:19.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 --2- 172.21.15.46:0/3640852739 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b2300 0x7f7eb80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:19.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 --2- 172.21.15.46:0/3640852739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb8004d10 0x7f7eb8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:19.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec806b8a0 con 0x7f7eb8004d10 2023-10-30T13:23:19.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec815d9a0 con 0x7f7eb8003980 2023-10-30T13:23:19.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb800b990 con 0x7f7ec80b2300 2023-10-30T13:23:19.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecd673640 1 --2- 172.21.15.46:0/3640852739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb8004d10 0x7f7eb8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:19.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecce72640 1 --2- 172.21.15.46:0/3640852739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8003980 0x7f7eb8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:19.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ec7fff640 1 --2- 172.21.15.46:0/3640852739 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b2300 0x7f7eb80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:19.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecd673640 1 -- 172.21.15.46:0/3640852739 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b2300 msgr2=0x7f7eb80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:19.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecd673640 1 --2- 172.21.15.46:0/3640852739 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b2300 0x7f7eb80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:19.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecd673640 1 -- 172.21.15.46:0/3640852739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8003980 msgr2=0x7f7eb8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:19.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecd673640 1 --2- 172.21.15.46:0/3640852739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8003980 0x7f7eb8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:19.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.124+0000 7f7ecd673640 1 -- 172.21.15.46:0/3640852739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb8003530 con 0x7f7eb8004d10 2023-10-30T13:23:19.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.128+0000 7f7ecd673640 1 --2- 172.21.15.46:0/3640852739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb8004d10 0x7f7eb8005190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009f90 tx=0x7f7ec001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9370cd4bd7ab9ec0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:19.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.128+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec00a5c70 con 0x7f7eb8004d10 2023-10-30T13:23:19.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.128+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec000f040 con 0x7f7eb8004d10 2023-10-30T13:23:19.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.128+0000 7f7ec77fe640 1 -- 172.21.15.46:0/3640852739 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec00ad440 con 0x7f7eb8004d10 2023-10-30T13:23:19.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.128+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/3640852739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb8004d10 msgr2=0x7f7eb8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:19.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.128+0000 7f7ecf0d6640 1 --2- 172.21.15.46:0/3640852739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb8004d10 0x7f7eb8005190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009f90 tx=0x7f7ec001cc60 comp rx=0 tx=0).stop 2023-10-30T13:23:19.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.128+0000 7f7ecd673640 1 -- 172.21.15.46:0/3640852739 reap_dead start 2023-10-30T13:23:19.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.128+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/3640852739 shutdown_connections 2023-10-30T13:23:19.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.128+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/3640852739 >> 172.21.15.46:0/3640852739 conn(0x7f7ec8069e90 msgr2=0x7f7ec806a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:19.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.128+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/3640852739 shutdown_connections 2023-10-30T13:23:19.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.128+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/3640852739 wait complete. 2023-10-30T13:23:19.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ecf0d6640 1 Processor -- start 2023-10-30T13:23:19.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ecf0d6640 1 -- start start 2023-10-30T13:23:19.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ecf0d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8004d10 0x7f7ec80cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:19.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ecf0d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8156350 0x7f7ec814cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:19.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ecf0d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b1e20 0x7f7ec8150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:19.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ecf0d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec81ee660 con 0x7f7ec8156350 2023-10-30T13:23:19.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ecf0d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80cdfe0 con 0x7f7eb8004d10 2023-10-30T13:23:19.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ecf0d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec81ee870 con 0x7f7ec80b1e20 2023-10-30T13:23:19.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ec7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8156350 0x7f7ec814cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:19.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ec7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8156350 0x7f7ec814cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51816/0 (socket says 172.21.15.46:51816) 2023-10-30T13:23:19.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ec7fff640 1 -- 172.21.15.46:0/1824288118 learned_addr learned my addr 172.21.15.46:0/1824288118 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:19.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ecd673640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b1e20 0x7f7ec8150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:19.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ecce72640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8004d10 0x7f7ec80cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:19.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ec7fff640 1 -- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b1e20 msgr2=0x7f7ec8150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:19.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ec7fff640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b1e20 0x7f7ec8150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:19.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ec7fff640 1 -- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8004d10 msgr2=0x7f7ec80cd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ec7fff640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8004d10 0x7f7ec80cd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ec7fff640 1 -- 172.21.15.46:0/1824288118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec81eeb10 con 0x7f7ec8156350 2023-10-30T13:23:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ec7fff640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8156350 0x7f7ec814cc10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4011a00 tx=0x7f7eb4011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ec57fa640 1 -- 172.21.15.46:0/1824288118 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb400fb50 con 0x7f7ec8156350 2023-10-30T13:23:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ec57fa640 1 -- 172.21.15.46:0/1824288118 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb400fcb0 con 0x7f7ec8156350 2023-10-30T13:23:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/1824288118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec81eedf0 con 0x7f7ec8156350 2023-10-30T13:23:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ec57fa640 1 -- 172.21.15.46:0/1824288118 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb4014440 con 0x7f7ec8156350 2023-10-30T13:23:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.132+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/1824288118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec81ef3b0 con 0x7f7ec8156350 2023-10-30T13:23:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.136+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/1824288118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec806cbb0 con 0x7f7ec8156350 2023-10-30T13:23:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.136+0000 7f7ec57fa640 1 -- 172.21.15.46:0/1824288118 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7eb40145a0 con 0x7f7ec8156350 2023-10-30T13:23:19.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.136+0000 7f7ec57fa640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7eac0664a0 0x7f7eac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:19.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.136+0000 7f7ec57fa640 1 -- 172.21.15.46:0/1824288118 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f7eb4084850 con 0x7f7ec8156350 2023-10-30T13:23:19.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.136+0000 7f7ecce72640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7eac0664a0 0x7f7eac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:19.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.136+0000 7f7ecce72640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7eac0664a0 0x7f7eac068960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc000c10 tx=0x7f7ebc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:19.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.140+0000 7f7ec57fa640 1 -- 172.21.15.46:0/1824288118 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7ec806cbb0 con 0x7f7ec8156350 2023-10-30T13:23:19.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.332+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/1824288118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7ec806cbb0 con 0x7f7ec8156350 2023-10-30T13:23:19.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.940+0000 7f7ec57fa640 1 -- 172.21.15.46:0/1824288118 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7ec806cbb0 con 0x7f7ec8156350 2023-10-30T13:23:19.949 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 2 and data pool 3 2023-10-30T13:23:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.944+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7eac0664a0 msgr2=0x7f7eac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.944+0000 7f7ecf0d6640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7eac0664a0 0x7f7eac068960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc000c10 tx=0x7f7ebc009040 comp rx=0 tx=0).stop 2023-10-30T13:23:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.944+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/1824288118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8156350 msgr2=0x7f7ec814cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:19.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.944+0000 7f7ecf0d6640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8156350 0x7f7ec814cc10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4011a00 tx=0x7f7eb4011ec0 comp rx=0 tx=0).stop 2023-10-30T13:23:19.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.944+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/1824288118 shutdown_connections 2023-10-30T13:23:19.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.944+0000 7f7ecf0d6640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7eac0664a0 0x7f7eac068960 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:19.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.944+0000 7f7ecf0d6640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ec80b1e20 0x7f7ec8150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:19.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.944+0000 7f7ecf0d6640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8156350 0x7f7ec814cc10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:19.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.944+0000 7f7ecf0d6640 1 --2- 172.21.15.46:0/1824288118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8004d10 0x7f7ec80cd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:19.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.944+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/1824288118 >> 172.21.15.46:0/1824288118 conn(0x7f7ec8069e90 msgr2=0x7f7ec81527d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:19.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.944+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/1824288118 shutdown_connections 2023-10-30T13:23:19.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:19.944+0000 7f7ecf0d6640 1 -- 172.21.15.46:0/1824288118 wait complete. 2023-10-30T13:23:19.973 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.108+0000 7ff498054640 1 Processor -- start 2023-10-30T13:23:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.108+0000 7ff498054640 1 -- start start 2023-10-30T13:23:20.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff498054640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff490070740 0x7ff4900c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff498054640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4900c5810 con 0x7ff49015e700 2023-10-30T13:23:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff498054640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49015eb70 con 0x7ff4901580e0 2023-10-30T13:23:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff498054640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49006acb0 con 0x7ff490070740 2023-10-30T13:23:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff4955ef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff490070740 0x7ff4900c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff4955ef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff490070740 0x7ff4900c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51830/0 (socket says 172.21.15.46:51830) 2023-10-30T13:23:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff4955ef640 1 -- 172.21.15.46:0/2638370838 learned_addr learned my addr 172.21.15.46:0/2638370838 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff4965f1640 1 --1- 172.21.15.46:0/2638370838 >> v1:172.21.15.46:6789/0 conn(0x7ff49015e700 0x7ff49006a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53118/0 (socket says 172.21.15.46:53118) 2023-10-30T13:23:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff4955ef640 1 -- 172.21.15.46:0/2638370838 >> v1:172.21.15.154:6789/0 conn(0x7ff4901580e0 legacy=0x7ff490069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff4955ef640 1 -- 172.21.15.46:0/2638370838 >> v1:172.21.15.46:6789/0 conn(0x7ff49015e700 legacy=0x7ff49006a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff4955ef640 1 -- 172.21.15.46:0/2638370838 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4900c6bb0 con 0x7ff490070740 2023-10-30T13:23:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff494dee640 1 -- 172.21.15.46:0/2638370838 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089224176 0 0) 0x7ff4900c5810 con 0x7ff49015e700 2023-10-30T13:23:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.112+0000 7ff4955ef640 1 --2- 172.21.15.46:0/2638370838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff490070740 0x7ff4900c6540 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff480009870 tx=0x7ff48001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f1d36dd719226e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff494dee640 1 -- 172.21.15.46:0/2638370838 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4800a7070 con 0x7ff490070740 2023-10-30T13:23:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff494dee640 1 -- 172.21.15.46:0/2638370838 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4800ab430 con 0x7ff490070740 2023-10-30T13:23:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff494dee640 1 -- 172.21.15.46:0/2638370838 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4800ab7c0 con 0x7ff490070740 2023-10-30T13:23:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 -- 172.21.15.46:0/2638370838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff490070740 msgr2=0x7ff4900c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 --2- 172.21.15.46:0/2638370838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff490070740 0x7ff4900c6540 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff480009870 tx=0x7ff48001c540 comp rx=0 tx=0).stop 2023-10-30T13:23:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 -- 172.21.15.46:0/2638370838 shutdown_connections 2023-10-30T13:23:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 --2- 172.21.15.46:0/2638370838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff490070740 0x7ff4900c6540 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 -- 172.21.15.46:0/2638370838 >> 172.21.15.46:0/2638370838 conn(0x7ff4900c7910 msgr2=0x7ff4900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:20.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 -- 172.21.15.46:0/2638370838 shutdown_connections 2023-10-30T13:23:20.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 -- 172.21.15.46:0/2638370838 wait complete. 2023-10-30T13:23:20.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 Processor -- start 2023-10-30T13:23:20.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 -- start start 2023-10-30T13:23:20.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff490070740 0x7ff4900bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4901580e0 0x7ff4901fa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff49015e700 0x7ff490142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff490147fc0 con 0x7ff4901580e0 2023-10-30T13:23:20.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4901fab10 con 0x7ff490070740 2023-10-30T13:23:20.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff498054640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4901490d0 con 0x7ff49015e700 2023-10-30T13:23:20.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff4955ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4901580e0 0x7ff4901fa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff4955ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4901580e0 0x7ff4901fa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51836/0 (socket says 172.21.15.46:51836) 2023-10-30T13:23:20.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff4955ef640 1 -- 172.21.15.46:0/3925025787 learned_addr learned my addr 172.21.15.46:0/3925025787 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:20.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff495df0640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff490070740 0x7ff4900bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.116+0000 7ff4965f1640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff49015e700 0x7ff490142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4955ef640 1 -- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff49015e700 msgr2=0x7ff490142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4955ef640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff49015e700 0x7ff490142dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4955ef640 1 -- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff490070740 msgr2=0x7ff4900bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4955ef640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff490070740 0x7ff4900bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4955ef640 1 -- 172.21.15.46:0/3925025787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4800094d0 con 0x7ff4901580e0 2023-10-30T13:23:20.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4965f1640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff49015e700 0x7ff490142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:20.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff495df0640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff490070740 0x7ff4900bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:20.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4955ef640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4901580e0 0x7ff4901fa0b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff4800ad040 tx=0x7ff48001ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:20.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4867fc640 1 -- 172.21.15.46:0/3925025787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4800b4800 con 0x7ff4901580e0 2023-10-30T13:23:20.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4867fc640 1 -- 172.21.15.46:0/3925025787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4800b4980 con 0x7ff4901580e0 2023-10-30T13:23:20.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff498054640 1 -- 172.21.15.46:0/3925025787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff490148260 con 0x7ff4901580e0 2023-10-30T13:23:20.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4867fc640 1 -- 172.21.15.46:0/3925025787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4800b4d30 con 0x7ff4901580e0 2023-10-30T13:23:20.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff498054640 1 -- 172.21.15.46:0/3925025787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4901493b0 con 0x7ff4901580e0 2023-10-30T13:23:20.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff498054640 1 -- 172.21.15.46:0/3925025787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4901470c0 con 0x7ff4901580e0 2023-10-30T13:23:20.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4867fc640 1 -- 172.21.15.46:0/3925025787 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4800b3380 con 0x7ff4901580e0 2023-10-30T13:23:20.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4867fc640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff4740664d0 0x7ff474068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.120+0000 7ff4867fc640 1 -- 172.21.15.46:0/3925025787 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff48011e2e0 con 0x7ff4901580e0 2023-10-30T13:23:20.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.124+0000 7ff495df0640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff4740664d0 0x7ff474068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.124+0000 7ff495df0640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff4740664d0 0x7ff474068990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff48c000f70 tx=0x7ff48c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:20.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.124+0000 7ff4867fc640 1 -- 172.21.15.46:0/3925025787 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4800eb630 con 0x7ff4901580e0 2023-10-30T13:23:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.284+0000 7ff498054640 1 -- 172.21.15.46:0/3925025787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff49004e170 con 0x7ff4901580e0 2023-10-30T13:23:20.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.284+0000 7ff4867fc640 1 -- 172.21.15.46:0/3925025787 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16518 (secure 0 0 0) 0x7ff4800eec50 con 0x7ff4901580e0 2023-10-30T13:23:20.293 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:20.293 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":19,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:19.337021+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.288+0000 7ff498054640 1 -- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff4740664d0 msgr2=0x7ff474068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.288+0000 7ff498054640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff4740664d0 0x7ff474068990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff48c000f70 tx=0x7ff48c009040 comp rx=0 tx=0).stop 2023-10-30T13:23:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.288+0000 7ff498054640 1 -- 172.21.15.46:0/3925025787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4901580e0 msgr2=0x7ff4901fa0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.288+0000 7ff498054640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4901580e0 0x7ff4901fa0b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff4800ad040 tx=0x7ff48001ad80 comp rx=0 tx=0).stop 2023-10-30T13:23:20.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.288+0000 7ff498054640 1 -- 172.21.15.46:0/3925025787 shutdown_connections 2023-10-30T13:23:20.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.288+0000 7ff498054640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff4740664d0 0x7ff474068990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.288+0000 7ff498054640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff49015e700 0x7ff490142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.288+0000 7ff498054640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4901580e0 0x7ff4901fa0b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.288+0000 7ff498054640 1 --2- 172.21.15.46:0/3925025787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff490070740 0x7ff4900bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.288+0000 7ff498054640 1 -- 172.21.15.46:0/3925025787 >> 172.21.15.46:0/3925025787 conn(0x7ff4900c7910 msgr2=0x7ff4901469b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:20.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.288+0000 7ff498054640 1 -- 172.21.15.46:0/3925025787 shutdown_connections 2023-10-30T13:23:20.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.288+0000 7ff498054640 1 -- 172.21.15.46:0/3925025787 wait complete. 2023-10-30T13:23:20.314 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:20.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.448+0000 7f0485780640 1 Processor -- start 2023-10-30T13:23:20.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.448+0000 7f0485780640 1 -- start start 2023-10-30T13:23:20.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f0485780640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f048015d6f0 0x7f048015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f0485780640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04800cf160 0x7f04800cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f0485780640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04800cb800 con 0x7f04801538a0 2023-10-30T13:23:20.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f0485780640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04800cb330 con 0x7f048015d6f0 2023-10-30T13:23:20.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f0485780640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04800cad70 con 0x7f04800cf160 2023-10-30T13:23:20.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f047e7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f048015d6f0 0x7f048015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f047e7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f048015d6f0 0x7f048015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51848/0 (socket says 172.21.15.46:51848) 2023-10-30T13:23:20.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f047e7fc640 1 -- 172.21.15.46:0/22558625 learned_addr learned my addr 172.21.15.46:0/22558625 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:20.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f047f7fe640 1 --2- 172.21.15.46:0/22558625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04800cf160 0x7f04800cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f047e7fc640 1 -- 172.21.15.46:0/22558625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04800cf160 msgr2=0x7f04800cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:20.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f047e7fc640 1 --2- 172.21.15.46:0/22558625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04800cf160 0x7f04800cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f047e7fc640 1 -- 172.21.15.46:0/22558625 >> v1:172.21.15.154:6789/0 conn(0x7f04801538a0 legacy=0x7f04800ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.452+0000 7f047e7fc640 1 -- 172.21.15.46:0/22558625 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04800cb050 con 0x7f048015d6f0 2023-10-30T13:23:20.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f047e7fc640 1 --2- 172.21.15.46:0/22558625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048015d6f0 0x7f048015a090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0474009870 tx=0x7f047401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8609003659b806a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:20.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f047dffb640 1 -- 172.21.15.46:0/22558625 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04740a8a30 con 0x7f048015d6f0 2023-10-30T13:23:20.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f047dffb640 1 -- 172.21.15.46:0/22558625 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04740a8b90 con 0x7f048015d6f0 2023-10-30T13:23:20.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f047dffb640 1 -- 172.21.15.46:0/22558625 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04740b0440 con 0x7f048015d6f0 2023-10-30T13:23:20.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 -- 172.21.15.46:0/22558625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048015d6f0 msgr2=0x7f048015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 --2- 172.21.15.46:0/22558625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048015d6f0 0x7f048015a090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0474009870 tx=0x7f047401ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:20.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 -- 172.21.15.46:0/22558625 shutdown_connections 2023-10-30T13:23:20.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 --2- 172.21.15.46:0/22558625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04800cf160 0x7f04800cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 --2- 172.21.15.46:0/22558625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048015d6f0 0x7f048015a090 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 -- 172.21.15.46:0/22558625 >> 172.21.15.46:0/22558625 conn(0x7f048006bfe0 msgr2=0x7f048006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:20.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 -- 172.21.15.46:0/22558625 shutdown_connections 2023-10-30T13:23:20.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 -- 172.21.15.46:0/22558625 wait complete. 2023-10-30T13:23:20.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 Processor -- start 2023-10-30T13:23:20.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 -- start start 2023-10-30T13:23:20.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04800cf160 0x7f0480159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04801538a0 0x7f04800c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048015d6f0 0x7f04801e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04801e3bf0 con 0x7f048015d6f0 2023-10-30T13:23:20.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04801580f0 con 0x7f04800cf160 2023-10-30T13:23:20.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.456+0000 7f0485780640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04800c59a0 con 0x7f04801538a0 2023-10-30T13:23:20.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f047f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048015d6f0 0x7f04801e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f047f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048015d6f0 0x7f04801e2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51858/0 (socket says 172.21.15.46:51858) 2023-10-30T13:23:20.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f047f7fe640 1 -- 172.21.15.46:0/619619353 learned_addr learned my addr 172.21.15.46:0/619619353 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:20.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f047e7fc640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04801538a0 0x7f04800c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f047effd640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04800cf160 0x7f0480159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f047f7fe640 1 -- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04801538a0 msgr2=0x7f04800c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f047f7fe640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04801538a0 0x7f04800c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f047f7fe640 1 -- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04800cf160 msgr2=0x7f0480159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f047f7fe640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04800cf160 0x7f0480159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f047f7fe640 1 -- 172.21.15.46:0/619619353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04740094d0 con 0x7f048015d6f0 2023-10-30T13:23:20.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f047f7fe640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048015d6f0 0x7f04801e2a40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f047800efc0 tx=0x7f047800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:20.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f04677fe640 1 -- 172.21.15.46:0/619619353 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0478009b10 con 0x7f048015d6f0 2023-10-30T13:23:20.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f04677fe640 1 -- 172.21.15.46:0/619619353 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047800f040 con 0x7f048015d6f0 2023-10-30T13:23:20.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f04677fe640 1 -- 172.21.15.46:0/619619353 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04780146c0 con 0x7f048015d6f0 2023-10-30T13:23:20.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f0485780640 1 -- 172.21.15.46:0/619619353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04801584d0 con 0x7f048015d6f0 2023-10-30T13:23:20.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f0485780640 1 -- 172.21.15.46:0/619619353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04800c4f60 con 0x7f048015d6f0 2023-10-30T13:23:20.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f0485780640 1 -- 172.21.15.46:0/619619353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04800b2a50 con 0x7f048015d6f0 2023-10-30T13:23:20.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.460+0000 7f04677fe640 1 -- 172.21.15.46:0/619619353 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0478014820 con 0x7f048015d6f0 2023-10-30T13:23:20.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.464+0000 7f04677fe640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f046c0664d0 0x7f046c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.464+0000 7f04677fe640 1 -- 172.21.15.46:0/619619353 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0478084910 con 0x7f048015d6f0 2023-10-30T13:23:20.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.464+0000 7f047effd640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f046c0664d0 0x7f046c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.464+0000 7f047effd640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f046c0664d0 0x7f046c068990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0470001810 tx=0x7f0470009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:20.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.464+0000 7f04677fe640 1 -- 172.21.15.46:0/619619353 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0478051d10 con 0x7f048015d6f0 2023-10-30T13:23:20.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.624+0000 7f0485780640 1 -- 172.21.15.46:0/619619353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f048004e170 con 0x7f048015d6f0 2023-10-30T13:23:20.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.624+0000 7f04677fe640 1 -- 172.21.15.46:0/619619353 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16518 (secure 0 0 0) 0x7f0478018020 con 0x7f048015d6f0 2023-10-30T13:23:20.632 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:20.632 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":19,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:19.337021+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.628+0000 7f0485780640 1 -- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f046c0664d0 msgr2=0x7f046c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.628+0000 7f0485780640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f046c0664d0 0x7f046c068990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0470001810 tx=0x7f0470009040 comp rx=0 tx=0).stop 2023-10-30T13:23:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.628+0000 7f0485780640 1 -- 172.21.15.46:0/619619353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048015d6f0 msgr2=0x7f04801e2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.628+0000 7f0485780640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048015d6f0 0x7f04801e2a40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f047800efc0 tx=0x7f047800c490 comp rx=0 tx=0).stop 2023-10-30T13:23:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.628+0000 7f0485780640 1 -- 172.21.15.46:0/619619353 shutdown_connections 2023-10-30T13:23:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.628+0000 7f0485780640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f046c0664d0 0x7f046c068990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.628+0000 7f0485780640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048015d6f0 0x7f04801e2a40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.628+0000 7f0485780640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04801538a0 0x7f04800c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.628+0000 7f0485780640 1 --2- 172.21.15.46:0/619619353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04800cf160 0x7f0480159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.628+0000 7f0485780640 1 -- 172.21.15.46:0/619619353 >> 172.21.15.46:0/619619353 conn(0x7f048006bfe0 msgr2=0x7f048014ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.628+0000 7f0485780640 1 -- 172.21.15.46:0/619619353 shutdown_connections 2023-10-30T13:23:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.628+0000 7f0485780640 1 -- 172.21.15.46:0/619619353 wait complete. 2023-10-30T13:23:20.653 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-30T13:23:20.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.792+0000 7f573e046640 1 Processor -- start 2023-10-30T13:23:20.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.792+0000 7f573e046640 1 -- start start 2023-10-30T13:23:20.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f573e046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5738153980 0x7f57380caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f573e046640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5738159930 0x7f57380cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f573e046640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573815a920 con 0x7f57380cf1a0 2023-10-30T13:23:20.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f573e046640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57380cb930 con 0x7f5738159930 2023-10-30T13:23:20.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f573e046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5738159dc0 con 0x7f5738153980 2023-10-30T13:23:20.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f5736ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5738159930 0x7f57380cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f5736ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5738159930 0x7f57380cb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51870/0 (socket says 172.21.15.46:51870) 2023-10-30T13:23:20.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f5736ffd640 1 -- 172.21.15.46:0/3548337988 learned_addr learned my addr 172.21.15.46:0/3548337988 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:20.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f57377fe640 1 --2- 172.21.15.46:0/3548337988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5738153980 0x7f57380caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f5736ffd640 1 -- 172.21.15.46:0/3548337988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5738153980 msgr2=0x7f57380caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:20.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f5736ffd640 1 --2- 172.21.15.46:0/3548337988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5738153980 0x7f57380caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f5736ffd640 1 -- 172.21.15.46:0/3548337988 >> v1:172.21.15.154:6789/0 conn(0x7f57380cf1a0 legacy=0x7f573815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f5736ffd640 1 -- 172.21.15.46:0/3548337988 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573815a080 con 0x7f5738159930 2023-10-30T13:23:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.796+0000 7f5736ffd640 1 --2- 172.21.15.46:0/3548337988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5738159930 0x7f57380cb1e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f572c009870 tx=0x7f572c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c481a36f246a4e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f57367fc640 1 -- 172.21.15.46:0/3548337988 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c0aa670 con 0x7f5738159930 2023-10-30T13:23:20.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f57367fc640 1 -- 172.21.15.46:0/3548337988 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c0aa7d0 con 0x7f5738159930 2023-10-30T13:23:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f57367fc640 1 -- 172.21.15.46:0/3548337988 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c0aab60 con 0x7f5738159930 2023-10-30T13:23:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 -- 172.21.15.46:0/3548337988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5738159930 msgr2=0x7f57380cb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 --2- 172.21.15.46:0/3548337988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5738159930 0x7f57380cb1e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f572c009870 tx=0x7f572c01c510 comp rx=0 tx=0).stop 2023-10-30T13:23:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 -- 172.21.15.46:0/3548337988 shutdown_connections 2023-10-30T13:23:20.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 --2- 172.21.15.46:0/3548337988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5738159930 0x7f57380cb1e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 --2- 172.21.15.46:0/3548337988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5738153980 0x7f57380caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 -- 172.21.15.46:0/3548337988 >> 172.21.15.46:0/3548337988 conn(0x7f573806c0f0 msgr2=0x7f573806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 -- 172.21.15.46:0/3548337988 shutdown_connections 2023-10-30T13:23:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 -- 172.21.15.46:0/3548337988 wait complete. 2023-10-30T13:23:20.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 Processor -- start 2023-10-30T13:23:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 -- start start 2023-10-30T13:23:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57380cf1a0 0x7f573814b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5738153980 0x7f57380b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57380b4050 0x7f57381435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573814c040 con 0x7f57380b4050 2023-10-30T13:23:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57381523c0 con 0x7f57380cf1a0 2023-10-30T13:23:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f573e046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5738152850 con 0x7f5738153980 2023-10-30T13:23:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f5737fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57380b4050 0x7f57381435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f57377fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57380cf1a0 0x7f573814b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f5737fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57380b4050 0x7f57381435a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51878/0 (socket says 172.21.15.46:51878) 2023-10-30T13:23:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f5737fff640 1 -- 172.21.15.46:0/886481217 learned_addr learned my addr 172.21.15.46:0/886481217 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:20.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.800+0000 7f5736ffd640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5738153980 0x7f57380b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f5737fff640 1 -- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5738153980 msgr2=0x7f57380b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f5737fff640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5738153980 0x7f57380b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f5737fff640 1 -- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57380cf1a0 msgr2=0x7f573814b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f5737fff640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57380cf1a0 0x7f573814b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f5737fff640 1 -- 172.21.15.46:0/886481217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c0094d0 con 0x7f57380b4050 2023-10-30T13:23:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f5737fff640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57380b4050 0x7f57381435a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f573000c770 tx=0x7f573000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f571ffff640 1 -- 172.21.15.46:0/886481217 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730003b80 con 0x7f57380b4050 2023-10-30T13:23:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f571ffff640 1 -- 172.21.15.46:0/886481217 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5730003ce0 con 0x7f57380b4050 2023-10-30T13:23:20.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f571ffff640 1 -- 172.21.15.46:0/886481217 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573000aa90 con 0x7f57380b4050 2023-10-30T13:23:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f573e046640 1 -- 172.21.15.46:0/886481217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573814d520 con 0x7f57380b4050 2023-10-30T13:23:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f573e046640 1 -- 172.21.15.46:0/886481217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573814ba80 con 0x7f57380b4050 2023-10-30T13:23:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f573e046640 1 -- 172.21.15.46:0/886481217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573804e170 con 0x7f57380b4050 2023-10-30T13:23:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f571ffff640 1 -- 172.21.15.46:0/886481217 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f573000abf0 con 0x7f57380b4050 2023-10-30T13:23:20.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.804+0000 7f571ffff640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f57240664d0 0x7f5724068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.808+0000 7f571ffff640 1 -- 172.21.15.46:0/886481217 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5730083c30 con 0x7f57380b4050 2023-10-30T13:23:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.808+0000 7f57377fe640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f57240664d0 0x7f5724068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.808+0000 7f57377fe640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f57240664d0 0x7f5724068990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5728007a60 tx=0x7f5728008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:20.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:20.808+0000 7f571ffff640 1 -- 172.21.15.46:0/886481217 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f57300510c0 con 0x7f57380b4050 2023-10-30T13:23:21.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.004+0000 7f573e046640 1 -- 172.21.15.46:0/886481217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f57380caa80 con 0x7f57380b4050 2023-10-30T13:23:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.964+0000 7f571ffff640 1 -- 172.21.15.46:0/886481217 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7f57300546e0 con 0x7f57380b4050 2023-10-30T13:23:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.968+0000 7f573e046640 1 -- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f57240664d0 msgr2=0x7f5724068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.968+0000 7f573e046640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f57240664d0 0x7f5724068990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5728007a60 tx=0x7f5728008040 comp rx=0 tx=0).stop 2023-10-30T13:23:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.968+0000 7f573e046640 1 -- 172.21.15.46:0/886481217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57380b4050 msgr2=0x7f57381435a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.968+0000 7f573e046640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57380b4050 0x7f57381435a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f573000c770 tx=0x7f573000cc30 comp rx=0 tx=0).stop 2023-10-30T13:23:21.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.968+0000 7f573e046640 1 -- 172.21.15.46:0/886481217 shutdown_connections 2023-10-30T13:23:21.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.968+0000 7f573e046640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f57240664d0 0x7f5724068990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:21.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.968+0000 7f573e046640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57380b4050 0x7f57381435a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:21.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.968+0000 7f573e046640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5738153980 0x7f57380b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:21.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.968+0000 7f573e046640 1 --2- 172.21.15.46:0/886481217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57380cf1a0 0x7f573814b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:21.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.968+0000 7f573e046640 1 -- 172.21.15.46:0/886481217 >> 172.21.15.46:0/886481217 conn(0x7f573806c0f0 msgr2=0x7f5738149770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:21.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.968+0000 7f573e046640 1 -- 172.21.15.46:0/886481217 shutdown_connections 2023-10-30T13:23:21.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:21.968+0000 7f573e046640 1 -- 172.21.15.46:0/886481217 wait complete. 2023-10-30T13:23:21.998 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300} 2023-10-30T13:23:21.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-10-30T13:23:22.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.140+0000 7f12d891d640 1 Processor -- start 2023-10-30T13:23:22.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.140+0000 7f12d891d640 1 -- start start 2023-10-30T13:23:22.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d891d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d4156db0 0x7f12d406afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d891d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f12d40b2300 0x7f12d4071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d891d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12d406fdf0 0x7f12d40720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d891d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d406b8a0 con 0x7f12d406fdf0 2023-10-30T13:23:22.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d891d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d406f1d0 con 0x7f12d40b2300 2023-10-30T13:23:22.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d891d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d415d9a0 con 0x7f12d4156db0 2023-10-30T13:23:22.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12d406fdf0 0x7f12d40720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12d406fdf0 0x7f12d40720d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51884/0 (socket says 172.21.15.46:51884) 2023-10-30T13:23:22.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d2d9d640 1 -- 172.21.15.46:0/1596263856 learned_addr learned my addr 172.21.15.46:0/1596263856 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:22.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d259c640 1 --2- 172.21.15.46:0/1596263856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d4156db0 0x7f12d406afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12cbfff640 1 --2- 172.21.15.46:0/1596263856 >> v2:172.21.15.154:3300/0 conn(0x7f12d40b2300 0x7f12d4071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d2d9d640 1 -- 172.21.15.46:0/1596263856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d4156db0 msgr2=0x7f12d406afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:22.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d2d9d640 1 --2- 172.21.15.46:0/1596263856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d4156db0 0x7f12d406afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d2d9d640 1 -- 172.21.15.46:0/1596263856 >> v2:172.21.15.154:3300/0 conn(0x7f12d40b2300 msgr2=0x7f12d4071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:22.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d2d9d640 1 --2- 172.21.15.46:0/1596263856 >> v2:172.21.15.154:3300/0 conn(0x7f12d40b2300 0x7f12d4071b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.144+0000 7f12d2d9d640 1 -- 172.21.15.46:0/1596263856 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d406f470 con 0x7f12d406fdf0 2023-10-30T13:23:22.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d2d9d640 1 --2- 172.21.15.46:0/1596263856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d406fdf0 0x7f12d40720d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f12d4072620 tx=0x7f12cc01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc61cbb3bfb108d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:22.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d1d9b640 1 -- 172.21.15.46:0/1596263856 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc0af400 con 0x7f12d406fdf0 2023-10-30T13:23:22.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d1d9b640 1 -- 172.21.15.46:0/1596263856 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12cc0ab050 con 0x7f12d406fdf0 2023-10-30T13:23:22.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d1d9b640 1 -- 172.21.15.46:0/1596263856 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc0af780 con 0x7f12d406fdf0 2023-10-30T13:23:22.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d891d640 1 -- 172.21.15.46:0/1596263856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d406fdf0 msgr2=0x7f12d40720d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:22.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d891d640 1 --2- 172.21.15.46:0/1596263856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d406fdf0 0x7f12d40720d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f12d4072620 tx=0x7f12cc01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d891d640 1 -- 172.21.15.46:0/1596263856 shutdown_connections 2023-10-30T13:23:22.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d891d640 1 --2- 172.21.15.46:0/1596263856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d406fdf0 0x7f12d40720d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d891d640 1 --2- 172.21.15.46:0/1596263856 >> v2:172.21.15.154:3300/0 conn(0x7f12d40b2300 0x7f12d4071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d891d640 1 --2- 172.21.15.46:0/1596263856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d4156db0 0x7f12d406afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d891d640 1 -- 172.21.15.46:0/1596263856 >> 172.21.15.46:0/1596263856 conn(0x7f12d4069e90 msgr2=0x7f12d406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:22.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d891d640 1 -- 172.21.15.46:0/1596263856 shutdown_connections 2023-10-30T13:23:22.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d891d640 1 -- 172.21.15.46:0/1596263856 wait complete. 2023-10-30T13:23:22.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d891d640 1 Processor -- start 2023-10-30T13:23:22.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.148+0000 7f12d891d640 1 -- start start 2023-10-30T13:23:22.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d891d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d406fdf0 0x7f12d415bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d891d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12d40b2300 0x7f12d41f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d891d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d4156db0 0x7f12d41f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d891d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d41592b0 con 0x7f12d406fdf0 2023-10-30T13:23:22.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d891d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d4157db0 con 0x7f12d40b2300 2023-10-30T13:23:22.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d891d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d40c36d0 con 0x7f12d4156db0 2023-10-30T13:23:22.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d406fdf0 0x7f12d415bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d406fdf0 0x7f12d415bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51886/0 (socket says 172.21.15.46:51886) 2023-10-30T13:23:22.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d259c640 1 -- 172.21.15.46:0/1953152655 learned_addr learned my addr 172.21.15.46:0/1953152655 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:22.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d2d9d640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d4156db0 0x7f12d41f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12cbfff640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12d40b2300 0x7f12d41f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d259c640 1 -- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d4156db0 msgr2=0x7f12d41f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:22.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d259c640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d4156db0 0x7f12d41f2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d259c640 1 -- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12d40b2300 msgr2=0x7f12d41f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:22.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d259c640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12d40b2300 0x7f12d41f22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d259c640 1 -- 172.21.15.46:0/1953152655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12cc009b30 con 0x7f12d406fdf0 2023-10-30T13:23:22.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d259c640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d406fdf0 0x7f12d415bf00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f12c400d5f0 tx=0x7f12c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:22.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12caffd640 1 -- 172.21.15.46:0/1953152655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c40125f0 con 0x7f12d406fdf0 2023-10-30T13:23:22.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12caffd640 1 -- 172.21.15.46:0/1953152655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c4012750 con 0x7f12d406fdf0 2023-10-30T13:23:22.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d891d640 1 -- 172.21.15.46:0/1953152655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d40c3970 con 0x7f12d406fdf0 2023-10-30T13:23:22.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12caffd640 1 -- 172.21.15.46:0/1953152655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4012ae0 con 0x7f12d406fdf0 2023-10-30T13:23:22.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d891d640 1 -- 172.21.15.46:0/1953152655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d4157760 con 0x7f12d406fdf0 2023-10-30T13:23:22.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12d891d640 1 -- 172.21.15.46:0/1953152655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d404e170 con 0x7f12d406fdf0 2023-10-30T13:23:22.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.152+0000 7f12caffd640 1 -- 172.21.15.46:0/1953152655 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f12c4012c40 con 0x7f12d406fdf0 2023-10-30T13:23:22.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.156+0000 7f12caffd640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12b40664a0 0x7f12b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.156+0000 7f12caffd640 1 -- 172.21.15.46:0/1953152655 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12c4084190 con 0x7f12d406fdf0 2023-10-30T13:23:22.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.156+0000 7f12cbfff640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12b40664a0 0x7f12b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.156+0000 7f12cbfff640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12b40664a0 0x7f12b4068960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f12d40c7480 tx=0x7f12bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:22.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.160+0000 7f12caffd640 1 -- 172.21.15.46:0/1953152655 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f12c4051600 con 0x7f12d406fdf0 2023-10-30T13:23:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.348+0000 7f12d891d640 1 -- 172.21.15.46:0/1953152655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f12d406a400 con 0x7f12d406fdf0 2023-10-30T13:23:22.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12caffd640 1 -- 172.21.15.46:0/1953152655 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f12c4054c20 con 0x7f12d406fdf0 2023-10-30T13:23:22.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12d891d640 1 -- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12b40664a0 msgr2=0x7f12b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:22.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12d891d640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12b40664a0 0x7f12b4068960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f12d40c7480 tx=0x7f12bc008040 comp rx=0 tx=0).stop 2023-10-30T13:23:22.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12d891d640 1 -- 172.21.15.46:0/1953152655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d406fdf0 msgr2=0x7f12d415bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:22.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12d891d640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d406fdf0 0x7f12d415bf00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f12c400d5f0 tx=0x7f12c400dab0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12d891d640 1 -- 172.21.15.46:0/1953152655 shutdown_connections 2023-10-30T13:23:22.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12d891d640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12b40664a0 0x7f12b4068960 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12d891d640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d4156db0 0x7f12d41f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12d891d640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12d40b2300 0x7f12d41f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12d891d640 1 --2- 172.21.15.46:0/1953152655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d406fdf0 0x7f12d415bf00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12d891d640 1 -- 172.21.15.46:0/1953152655 >> 172.21.15.46:0/1953152655 conn(0x7f12d4069e90 msgr2=0x7f12d4071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12d891d640 1 -- 172.21.15.46:0/1953152655 shutdown_connections 2023-10-30T13:23:22.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:22.976+0000 7f12d891d640 1 -- 172.21.15.46:0/1953152655 wait complete. 2023-10-30T13:23:23.009 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-30T13:23:23.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.152+0000 7efd676ea640 1 Processor -- start 2023-10-30T13:23:23.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.152+0000 7efd676ea640 1 -- start start 2023-10-30T13:23:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd676ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efd60159930 0x7efd600cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd676ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efd600cf6e0 0x7efd6015d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd676ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6015a920 con 0x7efd60153980 2023-10-30T13:23:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd676ea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd600cb930 con 0x7efd600cf6e0 2023-10-30T13:23:23.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd676ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd60159dc0 con 0x7efd60159930 2023-10-30T13:23:23.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd65486640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efd60153980 0x7efd600caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53132/0 (socket says 172.21.15.46:53132) 2023-10-30T13:23:23.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd65486640 1 -- 172.21.15.46:0/305821298 learned_addr learned my addr 172.21.15.46:0/305821298 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:23.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd65c87640 1 --2- 172.21.15.46:0/305821298 >> v2:172.21.15.46:3300/0 conn(0x7efd600cf6e0 0x7efd6015d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:23.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd64c85640 1 --2- 172.21.15.46:0/305821298 >> v2:172.21.15.154:3300/0 conn(0x7efd60159930 0x7efd600cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd57fff640 1 -- 172.21.15.46:0/305821298 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787969268 0 0) 0x7efd6015a920 con 0x7efd60153980 2023-10-30T13:23:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd57fff640 1 -- 172.21.15.46:0/305821298 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd4c003530 con 0x7efd60153980 2023-10-30T13:23:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd65c87640 1 -- 172.21.15.46:0/305821298 >> v2:172.21.15.154:3300/0 conn(0x7efd60159930 msgr2=0x7efd600cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd65c87640 1 --2- 172.21.15.46:0/305821298 >> v2:172.21.15.154:3300/0 conn(0x7efd60159930 0x7efd600cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:23.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd65c87640 1 -- 172.21.15.46:0/305821298 >> v1:172.21.15.46:6789/0 conn(0x7efd60153980 legacy=0x7efd600caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:23.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.156+0000 7efd65c87640 1 -- 172.21.15.46:0/305821298 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6015a080 con 0x7efd600cf6e0 2023-10-30T13:23:23.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd65c87640 1 --2- 172.21.15.46:0/305821298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600cf6e0 0x7efd6015d570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efd600742c0 tx=0x7efd5801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2015a2965b794e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:23.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd57fff640 1 -- 172.21.15.46:0/305821298 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd580a58b0 con 0x7efd600cf6e0 2023-10-30T13:23:23.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd57fff640 1 -- 172.21.15.46:0/305821298 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd580a5a10 con 0x7efd600cf6e0 2023-10-30T13:23:23.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd57fff640 1 -- 172.21.15.46:0/305821298 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd580a5da0 con 0x7efd600cf6e0 2023-10-30T13:23:23.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd676ea640 1 -- 172.21.15.46:0/305821298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600cf6e0 msgr2=0x7efd6015d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:23.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd676ea640 1 --2- 172.21.15.46:0/305821298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600cf6e0 0x7efd6015d570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efd600742c0 tx=0x7efd5801c8e0 comp rx=0 tx=0).stop 2023-10-30T13:23:23.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd676ea640 1 -- 172.21.15.46:0/305821298 shutdown_connections 2023-10-30T13:23:23.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd676ea640 1 --2- 172.21.15.46:0/305821298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600cf6e0 0x7efd6015d570 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:23.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd676ea640 1 --2- 172.21.15.46:0/305821298 >> v2:172.21.15.154:3300/0 conn(0x7efd60159930 0x7efd600cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:23.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd676ea640 1 -- 172.21.15.46:0/305821298 >> 172.21.15.46:0/305821298 conn(0x7efd6006c0f0 msgr2=0x7efd6006c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:23.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd676ea640 1 -- 172.21.15.46:0/305821298 shutdown_connections 2023-10-30T13:23:23.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd676ea640 1 -- 172.21.15.46:0/305821298 wait complete. 2023-10-30T13:23:23.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd676ea640 1 Processor -- start 2023-10-30T13:23:23.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.160+0000 7efd676ea640 1 -- start start 2023-10-30T13:23:23.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd676ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600cf6e0 0x7efd60147070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:23.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd676ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd60153980 0x7efd600b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:23.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd676ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd600b3ba0 0x7efd600b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:23.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd676ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6014f2e0 con 0x7efd600cf6e0 2023-10-30T13:23:23.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd676ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd60153250 con 0x7efd600b3ba0 2023-10-30T13:23:23.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd676ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd601513a0 con 0x7efd60153980 2023-10-30T13:23:23.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd65486640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600cf6e0 0x7efd60147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:23.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd65486640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600cf6e0 0x7efd60147070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51904/0 (socket says 172.21.15.46:51904) 2023-10-30T13:23:23.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd65486640 1 -- 172.21.15.46:0/3455183883 learned_addr learned my addr 172.21.15.46:0/3455183883 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:23.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd64c85640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd60153980 0x7efd600b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:23.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd65c87640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd600b3ba0 0x7efd600b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:23.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd65486640 1 -- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd60153980 msgr2=0x7efd600b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:23.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd65486640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd60153980 0x7efd600b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:23.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd65486640 1 -- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd600b3ba0 msgr2=0x7efd600b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:23.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd65486640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd600b3ba0 0x7efd600b4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:23.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd65486640 1 -- 172.21.15.46:0/3455183883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd58009a70 con 0x7efd600cf6e0 2023-10-30T13:23:23.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd64c85640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd60153980 0x7efd600b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:23.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd65486640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600cf6e0 0x7efd60147070 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00c500 tx=0x7efd5c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:23.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd55ffb640 1 -- 172.21.15.46:0/3455183883 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c005cf0 con 0x7efd600cf6e0 2023-10-30T13:23:23.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd55ffb640 1 -- 172.21.15.46:0/3455183883 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5c005e50 con 0x7efd600cf6e0 2023-10-30T13:23:23.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd676ea640 1 -- 172.21.15.46:0/3455183883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd60147920 con 0x7efd600cf6e0 2023-10-30T13:23:23.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd55ffb640 1 -- 172.21.15.46:0/3455183883 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0072f0 con 0x7efd600cf6e0 2023-10-30T13:23:23.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd676ea640 1 -- 172.21.15.46:0/3455183883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6014e7e0 con 0x7efd600cf6e0 2023-10-30T13:23:23.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd676ea640 1 -- 172.21.15.46:0/3455183883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd600b49f0 con 0x7efd600cf6e0 2023-10-30T13:23:23.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.164+0000 7efd55ffb640 1 -- 172.21.15.46:0/3455183883 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efd5c007450 con 0x7efd600cf6e0 2023-10-30T13:23:23.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.168+0000 7efd55ffb640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efd440664d0 0x7efd44068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:23.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.168+0000 7efd55ffb640 1 -- 172.21.15.46:0/3455183883 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd5c0832f0 con 0x7efd600cf6e0 2023-10-30T13:23:23.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.168+0000 7efd64c85640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efd440664d0 0x7efd44068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:23.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.168+0000 7efd64c85640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efd440664d0 0x7efd44068990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efd601475b0 tx=0x7efd50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:23.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.172+0000 7efd55ffb640 1 -- 172.21.15.46:0/3455183883 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efd600b49f0 con 0x7efd600cf6e0 2023-10-30T13:23:23.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.360+0000 7efd676ea640 1 -- 172.21.15.46:0/3455183883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7efd600b49f0 con 0x7efd600cf6e0 2023-10-30T13:23:23.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.976+0000 7efd55ffb640 1 -- 172.21.15.46:0/3455183883 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7efd600b49f0 con 0x7efd600cf6e0 2023-10-30T13:23:23.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.980+0000 7efd676ea640 1 -- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efd440664d0 msgr2=0x7efd44068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:23.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.980+0000 7efd676ea640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efd440664d0 0x7efd44068990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efd601475b0 tx=0x7efd50008040 comp rx=0 tx=0).stop 2023-10-30T13:23:23.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.980+0000 7efd676ea640 1 -- 172.21.15.46:0/3455183883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600cf6e0 msgr2=0x7efd60147070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:23.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.980+0000 7efd676ea640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600cf6e0 0x7efd60147070 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00c500 tx=0x7efd5c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:23:23.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.980+0000 7efd676ea640 1 -- 172.21.15.46:0/3455183883 shutdown_connections 2023-10-30T13:23:23.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.980+0000 7efd676ea640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efd440664d0 0x7efd44068990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:23.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.980+0000 7efd676ea640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd600b3ba0 0x7efd600b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:23.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.980+0000 7efd676ea640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd60153980 0x7efd600b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.980+0000 7efd676ea640 1 --2- 172.21.15.46:0/3455183883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd600cf6e0 0x7efd60147070 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.980+0000 7efd676ea640 1 -- 172.21.15.46:0/3455183883 >> 172.21.15.46:0/3455183883 conn(0x7efd6006c0f0 msgr2=0x7efd6006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.980+0000 7efd676ea640 1 -- 172.21.15.46:0/3455183883 shutdown_connections 2023-10-30T13:23:23.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:23.980+0000 7efd676ea640 1 -- 172.21.15.46:0/3455183883 wait complete. 2023-10-30T13:23:24.011 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-10-30T13:23:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff712772640 1 Processor -- start 2023-10-30T13:23:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff712772640 1 -- start start 2023-10-30T13:23:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff712772640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff70c06c9e0 0x7ff70c06afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff712772640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff70c06fdd0 0x7ff70c0ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff712772640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70c06b4e0 con 0x7ff70c1570d0 2023-10-30T13:23:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff712772640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70c15a630 con 0x7ff70c06fdd0 2023-10-30T13:23:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff712772640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c1599b0 con 0x7ff70c06c9e0 2023-10-30T13:23:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff710d0f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff70c06fdd0 0x7ff70c0ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff70bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff70c1570d0 0x7ff70c071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53142/0 (socket says 172.21.15.46:53142) 2023-10-30T13:23:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff710d0f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff70c06fdd0 0x7ff70c0ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51906/0 (socket says 172.21.15.46:51906) 2023-10-30T13:23:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff70bfff640 1 -- 172.21.15.46:0/3890405217 learned_addr learned my addr 172.21.15.46:0/3890405217 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff70b7fe640 1 --2- 172.21.15.46:0/3890405217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff70c06c9e0 0x7ff70c06afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff710d0f640 1 -- 172.21.15.46:0/3890405217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff70c06c9e0 msgr2=0x7ff70c06afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff710d0f640 1 --2- 172.21.15.46:0/3890405217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff70c06c9e0 0x7ff70c06afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff710d0f640 1 -- 172.21.15.46:0/3890405217 >> v1:172.21.15.46:6789/0 conn(0x7ff70c1570d0 legacy=0x7ff70c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff710d0f640 1 -- 172.21.15.46:0/3890405217 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70c159c90 con 0x7ff70c06fdd0 2023-10-30T13:23:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.156+0000 7ff70affd640 1 -- 172.21.15.46:0/3890405217 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420957922 0 0) 0x7ff6fc001120 con 0x7ff70c1570d0 2023-10-30T13:23:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.160+0000 7ff710d0f640 1 --2- 172.21.15.46:0/3890405217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70c06fdd0 0x7ff70c0ccfa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff704009d90 tx=0x7ff70401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422d4358ca8d0b8f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.160+0000 7ff70affd640 1 -- 172.21.15.46:0/3890405217 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7040af600 con 0x7ff70c06fdd0 2023-10-30T13:23:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.160+0000 7ff70affd640 1 -- 172.21.15.46:0/3890405217 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7040ab050 con 0x7ff70c06fdd0 2023-10-30T13:23:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.160+0000 7ff70affd640 1 -- 172.21.15.46:0/3890405217 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7040af980 con 0x7ff70c06fdd0 2023-10-30T13:23:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.160+0000 7ff712772640 1 -- 172.21.15.46:0/3890405217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70c06fdd0 msgr2=0x7ff70c0ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.160+0000 7ff712772640 1 --2- 172.21.15.46:0/3890405217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70c06fdd0 0x7ff70c0ccfa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff704009d90 tx=0x7ff70401f130 comp rx=0 tx=0).stop 2023-10-30T13:23:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.160+0000 7ff712772640 1 -- 172.21.15.46:0/3890405217 shutdown_connections 2023-10-30T13:23:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.160+0000 7ff712772640 1 --2- 172.21.15.46:0/3890405217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70c06fdd0 0x7ff70c0ccfa0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.160+0000 7ff712772640 1 --2- 172.21.15.46:0/3890405217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff70c06c9e0 0x7ff70c06afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.160+0000 7ff712772640 1 -- 172.21.15.46:0/3890405217 >> 172.21.15.46:0/3890405217 conn(0x7ff70c069e90 msgr2=0x7ff70c06a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.160+0000 7ff712772640 1 -- 172.21.15.46:0/3890405217 shutdown_connections 2023-10-30T13:23:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.160+0000 7ff712772640 1 -- 172.21.15.46:0/3890405217 wait complete. 2023-10-30T13:23:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff712772640 1 Processor -- start 2023-10-30T13:23:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff712772640 1 -- start start 2023-10-30T13:23:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff712772640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70c06c9e0 0x7ff70c1e4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff712772640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff70c06fdd0 0x7ff70c1e2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff712772640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff70c1570d0 0x7ff70c1f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff712772640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c1e4090 con 0x7ff70c06c9e0 2023-10-30T13:23:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff712772640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c1e4680 con 0x7ff70c1570d0 2023-10-30T13:23:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff712772640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c158260 con 0x7ff70c06fdd0 2023-10-30T13:23:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff70bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70c06c9e0 0x7ff70c1e4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff70bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70c06c9e0 0x7ff70c1e4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51922/0 (socket says 172.21.15.46:51922) 2023-10-30T13:23:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff70bfff640 1 -- 172.21.15.46:0/271534988 learned_addr learned my addr 172.21.15.46:0/271534988 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff70b7fe640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff70c06fdd0 0x7ff70c1e2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff710d0f640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff70c1570d0 0x7ff70c1f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff70bfff640 1 -- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff70c06fdd0 msgr2=0x7ff70c1e2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff70bfff640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff70c06fdd0 0x7ff70c1e2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff70bfff640 1 -- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff70c1570d0 msgr2=0x7ff70c1f02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff70bfff640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff70c1570d0 0x7ff70c1f02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff70bfff640 1 -- 172.21.15.46:0/271534988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff704009a70 con 0x7ff70c06c9e0 2023-10-30T13:23:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff70b7fe640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff70c06fdd0 0x7ff70c1e2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff70bfff640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70c06c9e0 0x7ff70c1e4bc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00c480 tx=0x7ff6fc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff708ff9640 1 -- 172.21.15.46:0/271534988 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc0074e0 con 0x7ff70c06c9e0 2023-10-30T13:23:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff708ff9640 1 -- 172.21.15.46:0/271534988 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6fc007660 con 0x7ff70c06c9e0 2023-10-30T13:23:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff708ff9640 1 -- 172.21.15.46:0/271534988 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc004a30 con 0x7ff70c06c9e0 2023-10-30T13:23:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff712772640 1 -- 172.21.15.46:0/271534988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70c158540 con 0x7ff70c06c9e0 2023-10-30T13:23:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff712772640 1 -- 172.21.15.46:0/271534988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70c158b00 con 0x7ff70c06c9e0 2023-10-30T13:23:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.164+0000 7ff712772640 1 -- 172.21.15.46:0/271534988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff70c04e170 con 0x7ff70c06c9e0 2023-10-30T13:23:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.168+0000 7ff708ff9640 1 -- 172.21.15.46:0/271534988 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff6fc004b90 con 0x7ff70c06c9e0 2023-10-30T13:23:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.168+0000 7ff708ff9640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6f80664a0 0x7ff6f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.168+0000 7ff708ff9640 1 -- 172.21.15.46:0/271534988 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6fc082e10 con 0x7ff70c06c9e0 2023-10-30T13:23:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.168+0000 7ff70b7fe640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6f80664a0 0x7ff6f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.168+0000 7ff70b7fe640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6f80664a0 0x7ff6f8068960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff70c0c3c00 tx=0x7ff700008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.172+0000 7ff708ff9640 1 -- 172.21.15.46:0/271534988 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6fc086030 con 0x7ff70c06c9e0 2023-10-30T13:23:24.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.364+0000 7ff712772640 1 -- 172.21.15.46:0/271534988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7ff70c06b4e0 con 0x7ff70c06c9e0 2023-10-30T13:23:25.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.992+0000 7ff708ff9640 1 -- 172.21.15.46:0/271534988 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 115+0+0 (secure 0 0 0) 0x7ff70c06b4e0 con 0x7ff70c06c9e0 2023-10-30T13:23:25.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.996+0000 7ff712772640 1 -- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6f80664a0 msgr2=0x7ff6f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.996+0000 7ff712772640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6f80664a0 0x7ff6f8068960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff70c0c3c00 tx=0x7ff700008040 comp rx=0 tx=0).stop 2023-10-30T13:23:25.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.996+0000 7ff712772640 1 -- 172.21.15.46:0/271534988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70c06c9e0 msgr2=0x7ff70c1e4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.996+0000 7ff712772640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70c06c9e0 0x7ff70c1e4bc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00c480 tx=0x7ff6fc00c940 comp rx=0 tx=0).stop 2023-10-30T13:23:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.996+0000 7ff712772640 1 -- 172.21.15.46:0/271534988 shutdown_connections 2023-10-30T13:23:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.996+0000 7ff712772640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6f80664a0 0x7ff6f8068960 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.996+0000 7ff712772640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff70c1570d0 0x7ff70c1f02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.996+0000 7ff712772640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff70c06fdd0 0x7ff70c1e2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.996+0000 7ff712772640 1 --2- 172.21.15.46:0/271534988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff70c06c9e0 0x7ff70c1e4bc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.996+0000 7ff712772640 1 -- 172.21.15.46:0/271534988 >> 172.21.15.46:0/271534988 conn(0x7ff70c069e90 msgr2=0x7ff70c0c3480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.996+0000 7ff712772640 1 -- 172.21.15.46:0/271534988 shutdown_connections 2023-10-30T13:23:25.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:24.996+0000 7ff712772640 1 -- 172.21.15.46:0/271534988 wait complete. 2023-10-30T13:23:25.027 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-30T13:23:25.027 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:25.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.164+0000 7f78c6825640 1 Processor -- start 2023-10-30T13:23:25.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.164+0000 7f78c6825640 1 -- start start 2023-10-30T13:23:25.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.168+0000 7f78c6825640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f78c00c6b80 0x7f78c00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.168+0000 7f78c6825640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c00c0a50 con 0x7f78c00c6f60 2023-10-30T13:23:25.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.168+0000 7f78c6825640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c006f6e0 con 0x7f78c013f4a0 2023-10-30T13:23:25.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.168+0000 7f78c6825640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c006f9c0 con 0x7f78c00c6b80 2023-10-30T13:23:25.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.168+0000 7f78bffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f78c00c6b80 0x7f78c00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.168+0000 7f78bffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f78c00c6b80 0x7f78c00bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51932/0 (socket says 172.21.15.46:51932) 2023-10-30T13:23:25.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.168+0000 7f78bffff640 1 -- 172.21.15.46:0/903359615 learned_addr learned my addr 172.21.15.46:0/903359615 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:25.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.168+0000 7f78bf7fe640 1 --1- 172.21.15.46:0/903359615 >> v1:172.21.15.46:6789/0 conn(0x7f78c00c6f60 0x7f78c013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53150/0 (socket says 172.21.15.46:53150) 2023-10-30T13:23:25.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.168+0000 7f78bffff640 1 -- 172.21.15.46:0/903359615 >> v1:172.21.15.154:6789/0 conn(0x7f78c013f4a0 legacy=0x7f78c01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.168+0000 7f78bffff640 1 -- 172.21.15.46:0/903359615 >> v1:172.21.15.46:6789/0 conn(0x7f78c00c6f60 legacy=0x7f78c013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.168+0000 7f78bffff640 1 -- 172.21.15.46:0/903359615 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c006fca0 con 0x7f78c00c6b80 2023-10-30T13:23:25.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78bffff640 1 --2- 172.21.15.46:0/903359615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78c00c6b80 0x7f78c00bfec0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f78b000bb10 tx=0x7f78b001ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ef210e890dca25 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:25.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78beffd640 1 -- 172.21.15.46:0/903359615 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b00a7af0 con 0x7f78c00c6b80 2023-10-30T13:23:25.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78beffd640 1 -- 172.21.15.46:0/903359615 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b00a7c50 con 0x7f78c00c6b80 2023-10-30T13:23:25.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78beffd640 1 -- 172.21.15.46:0/903359615 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b00af440 con 0x7f78c00c6b80 2023-10-30T13:23:25.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78c6825640 1 -- 172.21.15.46:0/903359615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78c00c6b80 msgr2=0x7f78c00bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78c6825640 1 --2- 172.21.15.46:0/903359615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78c00c6b80 0x7f78c00bfec0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f78b000bb10 tx=0x7f78b001ec10 comp rx=0 tx=0).stop 2023-10-30T13:23:25.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78c6825640 1 -- 172.21.15.46:0/903359615 shutdown_connections 2023-10-30T13:23:25.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78c6825640 1 --2- 172.21.15.46:0/903359615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78c00c6b80 0x7f78c00bfec0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78c6825640 1 -- 172.21.15.46:0/903359615 >> 172.21.15.46:0/903359615 conn(0x7f78c00c7910 msgr2=0x7f78c00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:25.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78c6825640 1 -- 172.21.15.46:0/903359615 shutdown_connections 2023-10-30T13:23:25.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78c6825640 1 -- 172.21.15.46:0/903359615 wait complete. 2023-10-30T13:23:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78c6825640 1 Processor -- start 2023-10-30T13:23:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.172+0000 7f78c6825640 1 -- start start 2023-10-30T13:23:25.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c6825640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78c00c6b80 0x7f78c00c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c6825640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78c00c6f60 0x7f78c01f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c6825640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78c013f4a0 0x7f78c01544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c6825640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c01f22b0 con 0x7f78c013f4a0 2023-10-30T13:23:25.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c6825640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c01f3210 con 0x7f78c00c6b80 2023-10-30T13:23:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c6825640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c00c4a90 con 0x7f78c00c6f60 2023-10-30T13:23:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c4dc2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78c013f4a0 0x7f78c01544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c4dc2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78c013f4a0 0x7f78c01544f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51942/0 (socket says 172.21.15.46:51942) 2023-10-30T13:23:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c4dc2640 1 -- 172.21.15.46:0/2516914328 learned_addr learned my addr 172.21.15.46:0/2516914328 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78bffff640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78c00c6b80 0x7f78c00c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78bf7fe640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78c00c6f60 0x7f78c01f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c4dc2640 1 -- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78c00c6f60 msgr2=0x7f78c01f2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c4dc2640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78c00c6f60 0x7f78c01f2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c4dc2640 1 -- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78c00c6b80 msgr2=0x7f78c00c2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c4dc2640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78c00c6b80 0x7f78c00c2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c4dc2640 1 -- 172.21.15.46:0/2516914328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b000b770 con 0x7f78c013f4a0 2023-10-30T13:23:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c4dc2640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78c013f4a0 0x7f78c01544f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c730 tx=0x7f78b800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78bcff9640 1 -- 172.21.15.46:0/2516914328 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b8003b80 con 0x7f78c013f4a0 2023-10-30T13:23:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78bcff9640 1 -- 172.21.15.46:0/2516914328 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b8003ce0 con 0x7f78c013f4a0 2023-10-30T13:23:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c6825640 1 -- 172.21.15.46:0/2516914328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c00c4d70 con 0x7f78c013f4a0 2023-10-30T13:23:25.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78bcff9640 1 -- 172.21.15.46:0/2516914328 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b800aa30 con 0x7f78c013f4a0 2023-10-30T13:23:25.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c6825640 1 -- 172.21.15.46:0/2516914328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c01f24d0 con 0x7f78c013f4a0 2023-10-30T13:23:25.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78c6825640 1 -- 172.21.15.46:0/2516914328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c00c5290 con 0x7f78c013f4a0 2023-10-30T13:23:25.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.176+0000 7f78bcff9640 1 -- 172.21.15.46:0/2516914328 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f78b800ab90 con 0x7f78c013f4a0 2023-10-30T13:23:25.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.180+0000 7f78bcff9640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f78ac0664a0 0x7f78ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.180+0000 7f78bcff9640 1 -- 172.21.15.46:0/2516914328 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78b8083cb0 con 0x7f78c013f4a0 2023-10-30T13:23:25.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.180+0000 7f78bffff640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f78ac0664a0 0x7f78ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.180+0000 7f78bffff640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f78ac0664a0 0x7f78ac068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f78b00b1000 tx=0x7f78b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:25.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.184+0000 7f78bcff9640 1 -- 172.21.15.46:0/2516914328 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f78b80510a0 con 0x7f78c013f4a0 2023-10-30T13:23:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.372+0000 7f78c6825640 1 -- 172.21.15.46:0/2516914328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78c004e170 con 0x7f78c013f4a0 2023-10-30T13:23:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.372+0000 7f78bcff9640 1 -- 172.21.15.46:0/2516914328 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6501 (secure 0 0 0) 0x7f78b80546c0 con 0x7f78c013f4a0 2023-10-30T13:23:25.380 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:25.380 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4271,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:25.002360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":4306,"mds_1":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:creating","state_seq":1,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2023-10-30T13:23:25.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.372+0000 7f78c6825640 1 -- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f78ac0664a0 msgr2=0x7f78ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.372+0000 7f78c6825640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f78ac0664a0 0x7f78ac068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f78b00b1000 tx=0x7f78b0008040 comp rx=0 tx=0).stop 2023-10-30T13:23:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.372+0000 7f78c6825640 1 -- 172.21.15.46:0/2516914328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78c013f4a0 msgr2=0x7f78c01544f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.372+0000 7f78c6825640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78c013f4a0 0x7f78c01544f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c730 tx=0x7f78b800cbf0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.376+0000 7f78c6825640 1 -- 172.21.15.46:0/2516914328 shutdown_connections 2023-10-30T13:23:25.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.376+0000 7f78c6825640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f78ac0664a0 0x7f78ac068960 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.376+0000 7f78c6825640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78c013f4a0 0x7f78c01544f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.376+0000 7f78c6825640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78c00c6f60 0x7f78c01f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.376+0000 7f78c6825640 1 --2- 172.21.15.46:0/2516914328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78c00c6b80 0x7f78c00c2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.376+0000 7f78c6825640 1 -- 172.21.15.46:0/2516914328 >> 172.21.15.46:0/2516914328 conn(0x7f78c00c7910 msgr2=0x7f78c0138ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.376+0000 7f78c6825640 1 -- 172.21.15.46:0/2516914328 shutdown_connections 2023-10-30T13:23:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.376+0000 7f78c6825640 1 -- 172.21.15.46:0/2516914328 wait complete. 2023-10-30T13:23:25.403 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:25.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.536+0000 7ff8b359e640 1 Processor -- start 2023-10-30T13:23:25.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b359e640 1 -- start start 2023-10-30T13:23:25.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b359e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8b415cd50 0x7ff8b40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b359e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b415a6c0 con 0x7ff8b40cefa0 2023-10-30T13:23:25.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b359e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b415d300 con 0x7ff8b41538f0 2023-10-30T13:23:25.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b359e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b415d5e0 con 0x7ff8b415cd50 2023-10-30T13:23:25.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b2d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff8b40cefa0 0x7ff8b41640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53156/0 (socket says 172.21.15.46:53156) 2023-10-30T13:23:25.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b2d9d640 1 -- 172.21.15.46:0/1575490904 learned_addr learned my addr 172.21.15.46:0/1575490904 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:25.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b1d9b640 1 --2- 172.21.15.46:0/1575490904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8b415cd50 0x7ff8b40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b159a640 1 -- 172.21.15.46:0/1575490904 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355601308 0 0) 0x7ff8b415a6c0 con 0x7ff8b40cefa0 2023-10-30T13:23:25.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b159a640 1 -- 172.21.15.46:0/1575490904 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff89c003530 con 0x7ff8b40cefa0 2023-10-30T13:23:25.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b159a640 1 -- 172.21.15.46:0/1575490904 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213170423 0 0) 0x7ff8b415d300 con 0x7ff8b41538f0 2023-10-30T13:23:25.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b159a640 1 -- 172.21.15.46:0/1575490904 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b415a6c0 con 0x7ff8b41538f0 2023-10-30T13:23:25.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b159a640 1 -- 172.21.15.46:0/1575490904 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3312447697 0 0) 0x7ff89c003530 con 0x7ff8b40cefa0 2023-10-30T13:23:25.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b1d9b640 1 -- 172.21.15.46:0/1575490904 >> v1:172.21.15.154:6789/0 conn(0x7ff8b41538f0 legacy=0x7ff8b4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b1d9b640 1 -- 172.21.15.46:0/1575490904 >> v1:172.21.15.46:6789/0 conn(0x7ff8b40cefa0 legacy=0x7ff8b41640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.540+0000 7ff8b1d9b640 1 -- 172.21.15.46:0/1575490904 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b415d8c0 con 0x7ff8b415cd50 2023-10-30T13:23:25.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.544+0000 7ff8b1d9b640 1 --2- 172.21.15.46:0/1575490904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8b415cd50 0x7ff8b40c9f90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80097f0 tx=0x7ff8a801ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25b06d1b2cd504c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:25.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.544+0000 7ff8b159a640 1 -- 172.21.15.46:0/1575490904 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac001a00 con 0x7ff8b415cd50 2023-10-30T13:23:25.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.544+0000 7ff8b159a640 1 -- 172.21.15.46:0/1575490904 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a80a8650 con 0x7ff8b415cd50 2023-10-30T13:23:25.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.544+0000 7ff8b159a640 1 -- 172.21.15.46:0/1575490904 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80a89e0 con 0x7ff8b415cd50 2023-10-30T13:23:25.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.544+0000 7ff8b359e640 1 -- 172.21.15.46:0/1575490904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8b415cd50 msgr2=0x7ff8b40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.544+0000 7ff8b359e640 1 --2- 172.21.15.46:0/1575490904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8b415cd50 0x7ff8b40c9f90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80097f0 tx=0x7ff8a801ebd0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.544+0000 7ff8b359e640 1 -- 172.21.15.46:0/1575490904 shutdown_connections 2023-10-30T13:23:25.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.544+0000 7ff8b359e640 1 --2- 172.21.15.46:0/1575490904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8b415cd50 0x7ff8b40c9f90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.544+0000 7ff8b359e640 1 -- 172.21.15.46:0/1575490904 >> 172.21.15.46:0/1575490904 conn(0x7ff8b406a640 msgr2=0x7ff8b4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.544+0000 7ff8b359e640 1 -- 172.21.15.46:0/1575490904 shutdown_connections 2023-10-30T13:23:25.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.544+0000 7ff8b359e640 1 -- 172.21.15.46:0/1575490904 wait complete. 2023-10-30T13:23:25.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b359e640 1 Processor -- start 2023-10-30T13:23:25.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b359e640 1 -- start start 2023-10-30T13:23:25.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b40cefa0 0x7ff8b41ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b359e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b41538f0 0x7ff8b415a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b359e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8b415cd50 0x7ff8b4144ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b359e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b41ef5b0 con 0x7ff8b40cefa0 2023-10-30T13:23:25.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b359e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4158950 con 0x7ff8b41538f0 2023-10-30T13:23:25.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b359e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b41f2970 con 0x7ff8b415cd50 2023-10-30T13:23:25.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b40cefa0 0x7ff8b41ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b40cefa0 0x7ff8b41ef9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51952/0 (socket says 172.21.15.46:51952) 2023-10-30T13:23:25.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b259c640 1 -- 172.21.15.46:0/3433036587 learned_addr learned my addr 172.21.15.46:0/3433036587 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:25.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b2d9d640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8b415cd50 0x7ff8b4144ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b1d9b640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b41538f0 0x7ff8b415a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b259c640 1 -- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8b415cd50 msgr2=0x7ff8b4144ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b259c640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8b415cd50 0x7ff8b4144ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b259c640 1 -- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b41538f0 msgr2=0x7ff8b415a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b259c640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b41538f0 0x7ff8b415a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b259c640 1 -- 172.21.15.46:0/3433036587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a8009420 con 0x7ff8b40cefa0 2023-10-30T13:23:25.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b2d9d640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8b415cd50 0x7ff8b4144ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:25.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b259c640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b40cefa0 0x7ff8b41ef9d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff8a400c490 tx=0x7ff8a400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:25.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8a2ffd640 1 -- 172.21.15.46:0/3433036587 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a40052d0 con 0x7ff8b40cefa0 2023-10-30T13:23:25.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b359e640 1 -- 172.21.15.46:0/3433036587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b41f2c30 con 0x7ff8b40cefa0 2023-10-30T13:23:25.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8a2ffd640 1 -- 172.21.15.46:0/3433036587 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a4005430 con 0x7ff8b40cefa0 2023-10-30T13:23:25.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8a2ffd640 1 -- 172.21.15.46:0/3433036587 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4012870 con 0x7ff8b40cefa0 2023-10-30T13:23:25.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b359e640 1 -- 172.21.15.46:0/3433036587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b41f31f0 con 0x7ff8b40cefa0 2023-10-30T13:23:25.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.548+0000 7ff8b359e640 1 -- 172.21.15.46:0/3433036587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b41ef6f0 con 0x7ff8b40cefa0 2023-10-30T13:23:25.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.552+0000 7ff8a2ffd640 1 -- 172.21.15.46:0/3433036587 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff8a40129d0 con 0x7ff8b40cefa0 2023-10-30T13:23:25.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.552+0000 7ff8a2ffd640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8940664a0 0x7ff894068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.552+0000 7ff8a2ffd640 1 -- 172.21.15.46:0/3433036587 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8a4083b80 con 0x7ff8b40cefa0 2023-10-30T13:23:25.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.552+0000 7ff8b1d9b640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8940664a0 0x7ff894068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.552+0000 7ff8b1d9b640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8940664a0 0x7ff894068960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80b2000 tx=0x7ff8a80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:25.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.556+0000 7ff8a2ffd640 1 -- 172.21.15.46:0/3433036587 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff8b41ef6f0 con 0x7ff8b40cefa0 2023-10-30T13:23:25.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.712+0000 7ff8b359e640 1 -- 172.21.15.46:0/3433036587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8b41ef6f0 con 0x7ff8b40cefa0 2023-10-30T13:23:25.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.712+0000 7ff8a2ffd640 1 -- 172.21.15.46:0/3433036587 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7ff8b41ef6f0 con 0x7ff8b40cefa0 2023-10-30T13:23:25.721 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:25.721 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:20.665723+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:25.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.716+0000 7ff8b359e640 1 -- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8940664a0 msgr2=0x7ff894068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.716+0000 7ff8b359e640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8940664a0 0x7ff894068960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80b2000 tx=0x7ff8a80aa070 comp rx=0 tx=0).stop 2023-10-30T13:23:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.716+0000 7ff8b359e640 1 -- 172.21.15.46:0/3433036587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b40cefa0 msgr2=0x7ff8b41ef9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.716+0000 7ff8b359e640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b40cefa0 0x7ff8b41ef9d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff8a400c490 tx=0x7ff8a400c950 comp rx=0 tx=0).stop 2023-10-30T13:23:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.716+0000 7ff8b359e640 1 -- 172.21.15.46:0/3433036587 shutdown_connections 2023-10-30T13:23:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.716+0000 7ff8b359e640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8940664a0 0x7ff894068960 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.716+0000 7ff8b359e640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8b415cd50 0x7ff8b4144ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.716+0000 7ff8b359e640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b41538f0 0x7ff8b415a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.716+0000 7ff8b359e640 1 --2- 172.21.15.46:0/3433036587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b40cefa0 0x7ff8b41ef9d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.716+0000 7ff8b359e640 1 -- 172.21.15.46:0/3433036587 >> 172.21.15.46:0/3433036587 conn(0x7ff8b406a640 msgr2=0x7ff8b40c9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.716+0000 7ff8b359e640 1 -- 172.21.15.46:0/3433036587 shutdown_connections 2023-10-30T13:23:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.716+0000 7ff8b359e640 1 -- 172.21.15.46:0/3433036587 wait complete. 2023-10-30T13:23:25.742 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-30T13:23:25.743 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:23:25.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f7622b80640 1 Processor -- start 2023-10-30T13:23:25.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f7622b80640 1 -- start start 2023-10-30T13:23:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f7622b80640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f761c1538a0 0x7f761c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f7622b80640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f761c0cf160 0x7f761c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f7622b80640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c0cb800 con 0x7f761c0ca0e0 2023-10-30T13:23:25.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f7622b80640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761c15dc30 con 0x7f761c1538a0 2023-10-30T13:23:25.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f7622b80640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761c0cb330 con 0x7f761c0cf160 2023-10-30T13:23:25.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f762091c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f761c1538a0 0x7f761c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f762091c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f761c1538a0 0x7f761c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51954/0 (socket says 172.21.15.46:51954) 2023-10-30T13:23:25.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f762091c640 1 -- 172.21.15.46:0/4238896764 learned_addr learned my addr 172.21.15.46:0/4238896764 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f762111d640 1 --2- 172.21.15.46:0/4238896764 >> v2:172.21.15.154:3300/0 conn(0x7f761c0cf160 0x7f761c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f762091c640 1 -- 172.21.15.46:0/4238896764 >> v2:172.21.15.154:3300/0 conn(0x7f761c0cf160 msgr2=0x7f761c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f762091c640 1 --2- 172.21.15.46:0/4238896764 >> v2:172.21.15.154:3300/0 conn(0x7f761c0cf160 0x7f761c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f762091c640 1 -- 172.21.15.46:0/4238896764 >> v1:172.21.15.154:6789/0 conn(0x7f761c0ca0e0 legacy=0x7f761c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.880+0000 7f762091c640 1 -- 172.21.15.46:0/4238896764 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c0cad70 con 0x7f761c1538a0 2023-10-30T13:23:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f762091c640 1 --2- 172.21.15.46:0/4238896764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f761c1538a0 0x7f761c15d6f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f761c0b5330 tx=0x7f761001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4a856c1bd22805 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f761b7fe640 1 -- 172.21.15.46:0/4238896764 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76100a89f0 con 0x7f761c1538a0 2023-10-30T13:23:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f761b7fe640 1 -- 172.21.15.46:0/4238896764 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76100a8b50 con 0x7f761c1538a0 2023-10-30T13:23:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f761b7fe640 1 -- 172.21.15.46:0/4238896764 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76100b0440 con 0x7f761c1538a0 2023-10-30T13:23:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f7622b80640 1 -- 172.21.15.46:0/4238896764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f761c1538a0 msgr2=0x7f761c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f7622b80640 1 --2- 172.21.15.46:0/4238896764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f761c1538a0 0x7f761c15d6f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f761c0b5330 tx=0x7f761001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f7622b80640 1 -- 172.21.15.46:0/4238896764 shutdown_connections 2023-10-30T13:23:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f7622b80640 1 --2- 172.21.15.46:0/4238896764 >> v2:172.21.15.154:3300/0 conn(0x7f761c0cf160 0x7f761c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f7622b80640 1 --2- 172.21.15.46:0/4238896764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f761c1538a0 0x7f761c15d6f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f7622b80640 1 -- 172.21.15.46:0/4238896764 >> 172.21.15.46:0/4238896764 conn(0x7f761c06bfe0 msgr2=0x7f761c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f7622b80640 1 -- 172.21.15.46:0/4238896764 shutdown_connections 2023-10-30T13:23:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f7622b80640 1 -- 172.21.15.46:0/4238896764 wait complete. 2023-10-30T13:23:25.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.884+0000 7f7622b80640 1 Processor -- start 2023-10-30T13:23:25.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f7622b80640 1 -- start start 2023-10-30T13:23:25.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f7622b80640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f761c0ca0e0 0x7f761c158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f7622b80640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f761c0cf160 0x7f761c1e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f7622b80640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f761c1538a0 0x7f761c1e4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f7622b80640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c15d9f0 con 0x7f761c1538a0 2023-10-30T13:23:25.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f7622b80640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c15d710 con 0x7f761c0ca0e0 2023-10-30T13:23:25.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f7622b80640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761c1e3600 con 0x7f761c0cf160 2023-10-30T13:23:25.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f762111d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f761c1538a0 0x7f761c1e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f762111d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f761c1538a0 0x7f761c1e4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51958/0 (socket says 172.21.15.46:51958) 2023-10-30T13:23:25.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f762111d640 1 -- 172.21.15.46:0/3869862565 learned_addr learned my addr 172.21.15.46:0/3869862565 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:25.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f762091c640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f761c0ca0e0 0x7f761c158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f761bfff640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f761c0cf160 0x7f761c1e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f762111d640 1 -- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f761c0cf160 msgr2=0x7f761c1e40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f762111d640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f761c0cf160 0x7f761c1e40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f762111d640 1 -- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f761c0ca0e0 msgr2=0x7f761c158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f762111d640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f761c0ca0e0 0x7f761c158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f762111d640 1 -- 172.21.15.46:0/3869862565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76100094d0 con 0x7f761c1538a0 2023-10-30T13:23:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f762111d640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f761c1538a0 0x7f761c1e4620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7614007d90 tx=0x7f761400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f76197fa640 1 -- 172.21.15.46:0/3869862565 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7614014420 con 0x7f761c1538a0 2023-10-30T13:23:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f76197fa640 1 -- 172.21.15.46:0/3869862565 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76140145a0 con 0x7f761c1538a0 2023-10-30T13:23:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f7622b80640 1 -- 172.21.15.46:0/3869862565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c1e38e0 con 0x7f761c1538a0 2023-10-30T13:23:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f76197fa640 1 -- 172.21.15.46:0/3869862565 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7614014950 con 0x7f761c1538a0 2023-10-30T13:23:25.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f7622b80640 1 -- 172.21.15.46:0/3869862565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c1595d0 con 0x7f761c1538a0 2023-10-30T13:23:25.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.888+0000 7f7622b80640 1 -- 172.21.15.46:0/3869862565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761c0b2a50 con 0x7f761c1538a0 2023-10-30T13:23:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.892+0000 7f76197fa640 1 -- 172.21.15.46:0/3869862565 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7614014ab0 con 0x7f761c1538a0 2023-10-30T13:23:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.892+0000 7f76197fa640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76000664a0 0x7f7600068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.892+0000 7f76197fa640 1 -- 172.21.15.46:0/3869862565 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7614084720 con 0x7f761c1538a0 2023-10-30T13:23:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.892+0000 7f762091c640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76000664a0 0x7f7600068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.892+0000 7f762091c640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76000664a0 0x7f7600068960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f76100b2000 tx=0x7f7610009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:25.896+0000 7f76197fa640 1 -- 172.21.15.46:0/3869862565 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7614051bb0 con 0x7f761c1538a0 2023-10-30T13:23:26.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.060+0000 7f7622b80640 1 -- 172.21.15.46:0/3869862565 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f761c159890 con 0x7f76000664a0 2023-10-30T13:23:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.084+0000 7f76197fa640 1 -- 172.21.15.46:0/3869862565 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264543 (secure 0 0 0) 0x7f76100b0540 con 0x7f76000664a0 2023-10-30T13:23:26.092 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:26.094 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T13:23:26.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.088+0000 7f7622b80640 1 -- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76000664a0 msgr2=0x7f7600068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.088+0000 7f7622b80640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76000664a0 0x7f7600068960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f76100b2000 tx=0x7f7610009040 comp rx=0 tx=0).stop 2023-10-30T13:23:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.088+0000 7f7622b80640 1 -- 172.21.15.46:0/3869862565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f761c1538a0 msgr2=0x7f761c1e4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.088+0000 7f7622b80640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f761c1538a0 0x7f761c1e4620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7614007d90 tx=0x7f761400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.088+0000 7f7622b80640 1 -- 172.21.15.46:0/3869862565 shutdown_connections 2023-10-30T13:23:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.088+0000 7f7622b80640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76000664a0 0x7f7600068960 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.088+0000 7f7622b80640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f761c1538a0 0x7f761c1e4620 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.088+0000 7f7622b80640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f761c0cf160 0x7f761c1e40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.088+0000 7f7622b80640 1 --2- 172.21.15.46:0/3869862565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f761c0ca0e0 0x7f761c158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.088+0000 7f7622b80640 1 -- 172.21.15.46:0/3869862565 >> 172.21.15.46:0/3869862565 conn(0x7f761c06bfe0 msgr2=0x7f761c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.088+0000 7f7622b80640 1 -- 172.21.15.46:0/3869862565 shutdown_connections 2023-10-30T13:23:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.088+0000 7f7622b80640 1 -- 172.21.15.46:0/3869862565 wait complete. 2023-10-30T13:23:26.113 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-30T13:23:24.710335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235516,"kb_used_data":2676,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739204,"statfs":{"total":773094113280,"available":772852944896,"internally_reserved":0,"allocated":2740224,"data_stored":1707063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":56,"apply_latency_ms":56,"commit_latency_ns":56000000,"apply_latency_ns":56000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002077"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740157+0000","last_change":"2023-10-30T13:23:19.719271+0000","last_active":"2023-10-30T13:23:19.740157+0000","last_peered":"2023-10-30T13:23:19.740157+0000","last_clean":"2023-10-30T13:23:19.740157+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T13:23:19.740157+0000","last_undegraded":"2023-10-30T13:23:19.740157+0000","last_fullsized":"2023-10-30T13:23:19.740157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:53.868003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670716+0000","last_change":"2023-10-30T13:23:19.687152+0000","last_active":"2023-10-30T13:23:20.670716+0000","last_peered":"2023-10-30T13:23:20.670716+0000","last_clean":"2023-10-30T13:23:20.670716+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T13:23:20.670716+0000","last_undegraded":"2023-10-30T13:23:20.670716+0000","last_fullsized":"2023-10-30T13:23:20.670716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.692768+0000","last_change":"2023-10-30T13:23:19.689554+0000","last_active":"2023-10-30T13:23:19.692768+0000","last_peered":"2023-10-30T13:23:19.692768+0000","last_clean":"2023-10-30T13:23:19.692768+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T13:23:19.692768+0000","last_undegraded":"2023-10-30T13:23:19.692768+0000","last_fullsized":"2023-10-30T13:23:19.692768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:59:48.180076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.729256+0000","last_change":"2023-10-30T13:23:19.714018+0000","last_active":"2023-10-30T13:23:19.729256+0000","last_peered":"2023-10-30T13:23:19.729256+0000","last_clean":"2023-10-30T13:23:19.729256+0000","last_became_active":"2023-10-30T13:23:19.711673+0000","last_became_peered":"2023-10-30T13:23:19.711673+0000","last_unstale":"2023-10-30T13:23:19.729256+0000","last_undegraded":"2023-10-30T13:23:19.729256+0000","last_fullsized":"2023-10-30T13:23:19.729256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:31:43.873032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740065+0000","last_change":"2023-10-30T13:23:19.719999+0000","last_active":"2023-10-30T13:23:19.740065+0000","last_peered":"2023-10-30T13:23:19.740065+0000","last_clean":"2023-10-30T13:23:19.740065+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T13:23:19.740065+0000","last_undegraded":"2023-10-30T13:23:19.740065+0000","last_fullsized":"2023-10-30T13:23:19.740065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:07:00.548753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.717402+0000","last_change":"2023-10-30T13:23:19.703474+0000","last_active":"2023-10-30T13:23:19.717402+0000","last_peered":"2023-10-30T13:23:19.717402+0000","last_clean":"2023-10-30T13:23:19.717402+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T13:23:19.717402+0000","last_undegraded":"2023-10-30T13:23:19.717402+0000","last_fullsized":"2023-10-30T13:23:19.717402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:38.408929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740452+0000","last_change":"2023-10-30T13:23:19.726985+0000","last_active":"2023-10-30T13:23:19.740452+0000","last_peered":"2023-10-30T13:23:19.740452+0000","last_clean":"2023-10-30T13:23:19.740452+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T13:23:19.740452+0000","last_undegraded":"2023-10-30T13:23:19.740452+0000","last_fullsized":"2023-10-30T13:23:19.740452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:16.978917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.715118+0000","last_change":"2023-10-30T13:23:19.714153+0000","last_active":"2023-10-30T13:23:19.715118+0000","last_peered":"2023-10-30T13:23:19.715118+0000","last_clean":"2023-10-30T13:23:19.715118+0000","last_became_active":"2023-10-30T13:23:19.713876+0000","last_became_peered":"2023-10-30T13:23:19.713876+0000","last_unstale":"2023-10-30T13:23:19.715118+0000","last_undegraded":"2023-10-30T13:23:19.715118+0000","last_fullsized":"2023-10-30T13:23:19.715118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:17:06.765922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740388+0000","last_change":"2023-10-30T13:23:19.724583+0000","last_active":"2023-10-30T13:23:19.740388+0000","last_peered":"2023-10-30T13:23:19.740388+0000","last_clean":"2023-10-30T13:23:19.740388+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T13:23:19.740388+0000","last_undegraded":"2023-10-30T13:23:19.740388+0000","last_fullsized":"2023-10-30T13:23:19.740388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:01:19.373370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677145+0000","last_change":"2023-10-30T13:23:19.704399+0000","last_active":"2023-10-30T13:23:20.677145+0000","last_peered":"2023-10-30T13:23:20.677145+0000","last_clean":"2023-10-30T13:23:20.677145+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T13:23:20.677145+0000","last_undegraded":"2023-10-30T13:23:20.677145+0000","last_fullsized":"2023-10-30T13:23:20.677145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676522+0000","last_change":"2023-10-30T13:23:19.726077+0000","last_active":"2023-10-30T13:23:20.676522+0000","last_peered":"2023-10-30T13:23:20.676522+0000","last_clean":"2023-10-30T13:23:20.676522+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T13:23:20.676522+0000","last_undegraded":"2023-10-30T13:23:20.676522+0000","last_fullsized":"2023-10-30T13:23:20.676522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675959+0000","last_change":"2023-10-30T13:23:19.720995+0000","last_active":"2023-10-30T13:23:20.675959+0000","last_peered":"2023-10-30T13:23:20.675959+0000","last_clean":"2023-10-30T13:23:20.675959+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T13:23:20.675959+0000","last_undegraded":"2023-10-30T13:23:20.675959+0000","last_fullsized":"2023-10-30T13:23:20.675959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:44.490568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.695027+0000","last_change":"2023-10-30T13:23:19.692509+0000","last_active":"2023-10-30T13:23:19.695027+0000","last_peered":"2023-10-30T13:23:19.695027+0000","last_clean":"2023-10-30T13:23:19.695027+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T13:23:19.695027+0000","last_undegraded":"2023-10-30T13:23:19.695027+0000","last_fullsized":"2023-10-30T13:23:19.695027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:36.485777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.688101+0000","last_change":"2023-10-30T13:23:19.685435+0000","last_active":"2023-10-30T13:23:19.688101+0000","last_peered":"2023-10-30T13:23:19.688101+0000","last_clean":"2023-10-30T13:23:19.688101+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T13:23:19.688101+0000","last_undegraded":"2023-10-30T13:23:19.688101+0000","last_fullsized":"2023-10-30T13:23:19.688101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:10:55.258514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670272+0000","last_change":"2023-10-30T13:23:19.706721+0000","last_active":"2023-10-30T13:23:20.670272+0000","last_peered":"2023-10-30T13:23:20.670272+0000","last_clean":"2023-10-30T13:23:20.670272+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T13:23:20.670272+0000","last_undegraded":"2023-10-30T13:23:20.670272+0000","last_fullsized":"2023-10-30T13:23:20.670272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.719618+0000","last_change":"2023-10-30T13:23:19.710432+0000","last_active":"2023-10-30T13:23:19.719618+0000","last_peered":"2023-10-30T13:23:19.719618+0000","last_clean":"2023-10-30T13:23:19.719618+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T13:23:19.719618+0000","last_undegraded":"2023-10-30T13:23:19.719618+0000","last_fullsized":"2023-10-30T13:23:19.719618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:13:14.565885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.747350+0000","last_change":"2023-10-30T13:23:19.741225+0000","last_active":"2023-10-30T13:23:19.747350+0000","last_peered":"2023-10-30T13:23:19.747350+0000","last_clean":"2023-10-30T13:23:19.747350+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T13:23:19.747350+0000","last_undegraded":"2023-10-30T13:23:19.747350+0000","last_fullsized":"2023-10-30T13:23:19.747350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:22:59.183745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.701718+0000","last_change":"2023-10-30T13:23:19.701024+0000","last_active":"2023-10-30T13:23:19.701718+0000","last_peered":"2023-10-30T13:23:19.701718+0000","last_clean":"2023-10-30T13:23:19.701718+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T13:23:19.701718+0000","last_undegraded":"2023-10-30T13:23:19.701718+0000","last_fullsized":"2023-10-30T13:23:19.701718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:48.598355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740337+0000","last_change":"2023-10-30T13:23:19.723929+0000","last_active":"2023-10-30T13:23:19.740337+0000","last_peered":"2023-10-30T13:23:19.740337+0000","last_clean":"2023-10-30T13:23:19.740337+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T13:23:19.740337+0000","last_undegraded":"2023-10-30T13:23:19.740337+0000","last_fullsized":"2023-10-30T13:23:19.740337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:52.219146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674806+0000","last_change":"2023-10-30T13:23:19.725101+0000","last_active":"2023-10-30T13:23:20.674806+0000","last_peered":"2023-10-30T13:23:20.674806+0000","last_clean":"2023-10-30T13:23:20.674806+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T13:23:20.674806+0000","last_undegraded":"2023-10-30T13:23:20.674806+0000","last_fullsized":"2023-10-30T13:23:20.674806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.722967+0000","last_change":"2023-10-30T13:23:19.721813+0000","last_active":"2023-10-30T13:23:19.722967+0000","last_peered":"2023-10-30T13:23:19.722967+0000","last_clean":"2023-10-30T13:23:19.722967+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T13:23:19.722967+0000","last_undegraded":"2023-10-30T13:23:19.722967+0000","last_fullsized":"2023-10-30T13:23:19.722967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:11:02.070206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.734522+0000","last_change":"2023-10-30T13:23:19.724763+0000","last_active":"2023-10-30T13:23:19.734522+0000","last_peered":"2023-10-30T13:23:19.734522+0000","last_clean":"2023-10-30T13:23:19.734522+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T13:23:19.734522+0000","last_undegraded":"2023-10-30T13:23:19.734522+0000","last_fullsized":"2023-10-30T13:23:19.734522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:09:14.832707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.717905+0000","last_change":"2023-10-30T13:23:19.716706+0000","last_active":"2023-10-30T13:23:19.717905+0000","last_peered":"2023-10-30T13:23:19.717905+0000","last_clean":"2023-10-30T13:23:19.717905+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T13:23:19.717905+0000","last_undegraded":"2023-10-30T13:23:19.717905+0000","last_fullsized":"2023-10-30T13:23:19.717905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:16.549139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.718012+0000","last_change":"2023-10-30T13:23:19.704791+0000","last_active":"2023-10-30T13:23:19.718012+0000","last_peered":"2023-10-30T13:23:19.718012+0000","last_clean":"2023-10-30T13:23:19.718012+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T13:23:19.718012+0000","last_undegraded":"2023-10-30T13:23:19.718012+0000","last_fullsized":"2023-10-30T13:23:19.718012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:02.097204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.718244+0000","last_change":"2023-10-30T13:23:19.717027+0000","last_active":"2023-10-30T13:23:19.718244+0000","last_peered":"2023-10-30T13:23:19.718244+0000","last_clean":"2023-10-30T13:23:19.718244+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T13:23:19.718244+0000","last_undegraded":"2023-10-30T13:23:19.718244+0000","last_fullsized":"2023-10-30T13:23:19.718244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:01.859148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677247+0000","last_change":"2023-10-30T13:23:19.706002+0000","last_active":"2023-10-30T13:23:20.677247+0000","last_peered":"2023-10-30T13:23:20.677247+0000","last_clean":"2023-10-30T13:23:20.677247+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T13:23:20.677247+0000","last_undegraded":"2023-10-30T13:23:20.677247+0000","last_fullsized":"2023-10-30T13:23:20.677247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:46.882474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670564+0000","last_change":"2023-10-30T13:23:19.678472+0000","last_active":"2023-10-30T13:23:20.670564+0000","last_peered":"2023-10-30T13:23:20.670564+0000","last_clean":"2023-10-30T13:23:20.670564+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T13:23:20.670564+0000","last_undegraded":"2023-10-30T13:23:20.670564+0000","last_fullsized":"2023-10-30T13:23:20.670564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.703220+0000","last_change":"2023-10-30T13:23:19.702155+0000","last_active":"2023-10-30T13:23:19.703220+0000","last_peered":"2023-10-30T13:23:19.703220+0000","last_clean":"2023-10-30T13:23:19.703220+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T13:23:19.703220+0000","last_undegraded":"2023-10-30T13:23:19.703220+0000","last_fullsized":"2023-10-30T13:23:19.703220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:05:38.564603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.689534+0000","last_change":"2023-10-30T13:23:19.687820+0000","last_active":"2023-10-30T13:23:19.689534+0000","last_peered":"2023-10-30T13:23:19.689534+0000","last_clean":"2023-10-30T13:23:19.689534+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T13:23:19.689534+0000","last_undegraded":"2023-10-30T13:23:19.689534+0000","last_fullsized":"2023-10-30T13:23:19.689534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:27:34.932720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.717351+0000","last_change":"2023-10-30T13:23:19.702442+0000","last_active":"2023-10-30T13:23:19.717351+0000","last_peered":"2023-10-30T13:23:19.717351+0000","last_clean":"2023-10-30T13:23:19.717351+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T13:23:19.717351+0000","last_undegraded":"2023-10-30T13:23:19.717351+0000","last_fullsized":"2023-10-30T13:23:19.717351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:51:00.809353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.712263+0000","last_change":"2023-10-30T13:23:19.711364+0000","last_active":"2023-10-30T13:23:19.712263+0000","last_peered":"2023-10-30T13:23:19.712263+0000","last_clean":"2023-10-30T13:23:19.712263+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T13:23:19.712263+0000","last_undegraded":"2023-10-30T13:23:19.712263+0000","last_fullsized":"2023-10-30T13:23:19.712263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:54.029087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674450+0000","last_change":"2023-10-30T13:23:19.698485+0000","last_active":"2023-10-30T13:23:20.674450+0000","last_peered":"2023-10-30T13:23:20.674450+0000","last_clean":"2023-10-30T13:23:20.674450+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T13:23:20.674450+0000","last_undegraded":"2023-10-30T13:23:20.674450+0000","last_fullsized":"2023-10-30T13:23:20.674450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675615+0000","last_change":"2023-10-30T13:23:19.703223+0000","last_active":"2023-10-30T13:23:20.675615+0000","last_peered":"2023-10-30T13:23:20.675615+0000","last_clean":"2023-10-30T13:23:20.675615+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T13:23:20.675615+0000","last_undegraded":"2023-10-30T13:23:20.675615+0000","last_fullsized":"2023-10-30T13:23:20.675615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:28.065471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.674639+0000","last_change":"2023-10-30T13:23:18.963933+0000","last_active":"2023-10-30T13:23:19.674639+0000","last_peered":"2023-10-30T13:23:19.674639+0000","last_clean":"2023-10-30T13:23:19.674639+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T13:23:19.674639+0000","last_undegraded":"2023-10-30T13:23:19.674639+0000","last_fullsized":"2023-10-30T13:23:19.674639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:48:58.196567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.719070+0000","last_change":"2023-10-30T13:23:19.717819+0000","last_active":"2023-10-30T13:23:19.719070+0000","last_peered":"2023-10-30T13:23:19.719070+0000","last_clean":"2023-10-30T13:23:19.719070+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T13:23:19.719070+0000","last_undegraded":"2023-10-30T13:23:19.719070+0000","last_fullsized":"2023-10-30T13:23:19.719070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:35.652687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674787+0000","last_change":"2023-10-30T13:23:18.976615+0000","last_active":"2023-10-30T13:23:20.674787+0000","last_peered":"2023-10-30T13:23:20.674787+0000","last_clean":"2023-10-30T13:23:20.674787+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T13:23:20.674787+0000","last_undegraded":"2023-10-30T13:23:20.674787+0000","last_fullsized":"2023-10-30T13:23:20.674787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:59.048396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.675429+0000","last_change":"2023-10-30T13:23:18.977357+0000","last_active":"2023-10-30T13:23:19.675429+0000","last_peered":"2023-10-30T13:23:19.675429+0000","last_clean":"2023-10-30T13:23:19.675429+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T13:23:19.675429+0000","last_undegraded":"2023-10-30T13:23:19.675429+0000","last_fullsized":"2023-10-30T13:23:19.675429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:53.502681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.690620+0000","last_change":"2023-10-30T13:23:19.689197+0000","last_active":"2023-10-30T13:23:19.690620+0000","last_peered":"2023-10-30T13:23:19.690620+0000","last_clean":"2023-10-30T13:23:19.690620+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T13:23:19.690620+0000","last_undegraded":"2023-10-30T13:23:19.690620+0000","last_fullsized":"2023-10-30T13:23:19.690620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:33:45.920519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.711652+0000","last_change":"2023-10-30T13:23:18.989299+0000","last_active":"2023-10-30T13:23:19.711652+0000","last_peered":"2023-10-30T13:23:19.711652+0000","last_clean":"2023-10-30T13:23:19.711652+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T13:23:19.711652+0000","last_undegraded":"2023-10-30T13:23:19.711652+0000","last_fullsized":"2023-10-30T13:23:19.711652+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:44.510184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670217+0000","last_change":"2023-10-30T13:23:19.679342+0000","last_active":"2023-10-30T13:23:20.670217+0000","last_peered":"2023-10-30T13:23:20.670217+0000","last_clean":"2023-10-30T13:23:20.670217+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T13:23:20.670217+0000","last_undegraded":"2023-10-30T13:23:20.670217+0000","last_fullsized":"2023-10-30T13:23:20.670217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677093+0000","last_change":"2023-10-30T13:23:18.989515+0000","last_active":"2023-10-30T13:23:20.677093+0000","last_peered":"2023-10-30T13:23:20.677093+0000","last_clean":"2023-10-30T13:23:20.677093+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T13:23:20.677093+0000","last_undegraded":"2023-10-30T13:23:20.677093+0000","last_fullsized":"2023-10-30T13:23:20.677093+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:35.133762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.739505+0000","last_change":"2023-10-30T13:23:19.732859+0000","last_active":"2023-10-30T13:23:19.739505+0000","last_peered":"2023-10-30T13:23:19.739505+0000","last_clean":"2023-10-30T13:23:19.739505+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T13:23:19.739505+0000","last_undegraded":"2023-10-30T13:23:19.739505+0000","last_fullsized":"2023-10-30T13:23:19.739505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:44.581517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.712355+0000","last_change":"2023-10-30T13:23:18.991454+0000","last_active":"2023-10-30T13:23:19.712355+0000","last_peered":"2023-10-30T13:23:19.712355+0000","last_clean":"2023-10-30T13:23:19.712355+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T13:23:19.712355+0000","last_undegraded":"2023-10-30T13:23:19.712355+0000","last_fullsized":"2023-10-30T13:23:19.712355+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:13.450999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.723415+0000","last_change":"2023-10-30T13:23:19.722347+0000","last_active":"2023-10-30T13:23:19.723415+0000","last_peered":"2023-10-30T13:23:19.723415+0000","last_clean":"2023-10-30T13:23:19.723415+0000","last_became_active":"2023-10-30T13:23:19.722077+0000","last_became_peered":"2023-10-30T13:23:19.722077+0000","last_unstale":"2023-10-30T13:23:19.723415+0000","last_undegraded":"2023-10-30T13:23:19.723415+0000","last_fullsized":"2023-10-30T13:23:19.723415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:26:46.361249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670533+0000","last_change":"2023-10-30T13:23:18.982337+0000","last_active":"2023-10-30T13:23:20.670533+0000","last_peered":"2023-10-30T13:23:20.670533+0000","last_clean":"2023-10-30T13:23:20.670533+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T13:23:20.670533+0000","last_undegraded":"2023-10-30T13:23:20.670533+0000","last_fullsized":"2023-10-30T13:23:20.670533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:09:39.996740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676299+0000","last_change":"2023-10-30T13:23:19.706643+0000","last_active":"2023-10-30T13:23:20.676299+0000","last_peered":"2023-10-30T13:23:20.676299+0000","last_clean":"2023-10-30T13:23:20.676299+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T13:23:20.676299+0000","last_undegraded":"2023-10-30T13:23:20.676299+0000","last_fullsized":"2023-10-30T13:23:20.676299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:29.238430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676290+0000","last_change":"2023-10-30T13:23:18.969882+0000","last_active":"2023-10-30T13:23:20.676290+0000","last_peered":"2023-10-30T13:23:20.676290+0000","last_clean":"2023-10-30T13:23:20.676290+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T13:23:20.676290+0000","last_undegraded":"2023-10-30T13:23:20.676290+0000","last_fullsized":"2023-10-30T13:23:20.676290+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:17.093448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740535+0000","last_change":"2023-10-30T13:23:19.728295+0000","last_active":"2023-10-30T13:23:19.740535+0000","last_peered":"2023-10-30T13:23:19.740535+0000","last_clean":"2023-10-30T13:23:19.740535+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T13:23:19.740535+0000","last_undegraded":"2023-10-30T13:23:19.740535+0000","last_fullsized":"2023-10-30T13:23:19.740535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:50:04.323145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.687496+0000","last_change":"2023-10-30T13:23:18.971632+0000","last_active":"2023-10-30T13:23:19.687496+0000","last_peered":"2023-10-30T13:23:19.687496+0000","last_clean":"2023-10-30T13:23:19.687496+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T13:23:19.687496+0000","last_undegraded":"2023-10-30T13:23:19.687496+0000","last_fullsized":"2023-10-30T13:23:19.687496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:54.428499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.719439+0000","last_change":"2023-10-30T13:23:19.718168+0000","last_active":"2023-10-30T13:23:19.719439+0000","last_peered":"2023-10-30T13:23:19.719439+0000","last_clean":"2023-10-30T13:23:19.719439+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T13:23:19.719439+0000","last_undegraded":"2023-10-30T13:23:19.719439+0000","last_fullsized":"2023-10-30T13:23:19.719439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:55:53.554238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.713067+0000","last_change":"2023-10-30T13:23:18.978420+0000","last_active":"2023-10-30T13:23:19.713067+0000","last_peered":"2023-10-30T13:23:19.713067+0000","last_clean":"2023-10-30T13:23:19.713067+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T13:23:19.713067+0000","last_undegraded":"2023-10-30T13:23:19.713067+0000","last_fullsized":"2023-10-30T13:23:19.713067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:10:44.511282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740233+0000","last_change":"2023-10-30T13:23:19.721434+0000","last_active":"2023-10-30T13:23:19.740233+0000","last_peered":"2023-10-30T13:23:19.740233+0000","last_clean":"2023-10-30T13:23:19.740233+0000","last_became_active":"2023-10-30T13:23:19.720704+0000","last_became_peered":"2023-10-30T13:23:19.720704+0000","last_unstale":"2023-10-30T13:23:19.740233+0000","last_undegraded":"2023-10-30T13:23:19.740233+0000","last_fullsized":"2023-10-30T13:23:19.740233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:15:16.602339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674378+0000","last_change":"2023-10-30T13:23:18.972707+0000","last_active":"2023-10-30T13:23:20.674378+0000","last_peered":"2023-10-30T13:23:20.674378+0000","last_clean":"2023-10-30T13:23:20.674378+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T13:23:20.674378+0000","last_undegraded":"2023-10-30T13:23:20.674378+0000","last_fullsized":"2023-10-30T13:23:20.674378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674810+0000","last_change":"2023-10-30T13:23:19.698274+0000","last_active":"2023-10-30T13:23:20.674810+0000","last_peered":"2023-10-30T13:23:20.674810+0000","last_clean":"2023-10-30T13:23:20.674810+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T13:23:20.674810+0000","last_undegraded":"2023-10-30T13:23:20.674810+0000","last_fullsized":"2023-10-30T13:23:20.674810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.676825+0000","last_change":"2023-10-30T13:23:18.948296+0000","last_active":"2023-10-30T13:23:19.676825+0000","last_peered":"2023-10-30T13:23:19.676825+0000","last_clean":"2023-10-30T13:23:19.676825+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T13:23:19.676825+0000","last_undegraded":"2023-10-30T13:23:19.676825+0000","last_fullsized":"2023-10-30T13:23:19.676825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:55.308329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740517+0000","last_change":"2023-10-30T13:23:19.728647+0000","last_active":"2023-10-30T13:23:19.740517+0000","last_peered":"2023-10-30T13:23:19.740517+0000","last_clean":"2023-10-30T13:23:19.740517+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T13:23:19.740517+0000","last_undegraded":"2023-10-30T13:23:19.740517+0000","last_fullsized":"2023-10-30T13:23:19.740517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:42.328530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675830+0000","last_change":"2023-10-30T13:23:18.962343+0000","last_active":"2023-10-30T13:23:20.675830+0000","last_peered":"2023-10-30T13:23:20.675830+0000","last_clean":"2023-10-30T13:23:20.675830+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T13:23:20.675830+0000","last_undegraded":"2023-10-30T13:23:20.675830+0000","last_fullsized":"2023-10-30T13:23:20.675830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:36.082164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740109+0000","last_change":"2023-10-30T13:23:19.720770+0000","last_active":"2023-10-30T13:23:19.740109+0000","last_peered":"2023-10-30T13:23:19.740109+0000","last_clean":"2023-10-30T13:23:19.740109+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T13:23:19.740109+0000","last_undegraded":"2023-10-30T13:23:19.740109+0000","last_fullsized":"2023-10-30T13:23:19.740109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:32.308859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.677370+0000","last_change":"2023-10-30T13:23:18.949081+0000","last_active":"2023-10-30T13:23:19.677370+0000","last_peered":"2023-10-30T13:23:19.677370+0000","last_clean":"2023-10-30T13:23:19.677370+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T13:23:19.677370+0000","last_undegraded":"2023-10-30T13:23:19.677370+0000","last_fullsized":"2023-10-30T13:23:19.677370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:00.356885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740342+0000","last_change":"2023-10-30T13:23:19.724180+0000","last_active":"2023-10-30T13:23:19.740342+0000","last_peered":"2023-10-30T13:23:19.740342+0000","last_clean":"2023-10-30T13:23:19.740342+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T13:23:19.740342+0000","last_undegraded":"2023-10-30T13:23:19.740342+0000","last_fullsized":"2023-10-30T13:23:19.740342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:11:20.637012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670614+0000","last_change":"2023-10-30T13:23:18.970780+0000","last_active":"2023-10-30T13:23:20.670614+0000","last_peered":"2023-10-30T13:23:20.670614+0000","last_clean":"2023-10-30T13:23:20.670614+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T13:23:20.670614+0000","last_undegraded":"2023-10-30T13:23:20.670614+0000","last_fullsized":"2023-10-30T13:23:20.670614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:38:51.113629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.718082+0000","last_change":"2023-10-30T13:23:19.704834+0000","last_active":"2023-10-30T13:23:19.718082+0000","last_peered":"2023-10-30T13:23:19.718082+0000","last_clean":"2023-10-30T13:23:19.718082+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T13:23:19.718082+0000","last_undegraded":"2023-10-30T13:23:19.718082+0000","last_fullsized":"2023-10-30T13:23:19.718082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:12:23.420417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.677209+0000","last_change":"2023-10-30T13:23:18.977920+0000","last_active":"2023-10-30T13:23:19.677209+0000","last_peered":"2023-10-30T13:23:19.677209+0000","last_clean":"2023-10-30T13:23:19.677209+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T13:23:19.677209+0000","last_undegraded":"2023-10-30T13:23:19.677209+0000","last_fullsized":"2023-10-30T13:23:19.677209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:07:59.139948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740376+0000","last_change":"2023-10-30T13:23:19.724497+0000","last_active":"2023-10-30T13:23:19.740376+0000","last_peered":"2023-10-30T13:23:19.740376+0000","last_clean":"2023-10-30T13:23:19.740376+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T13:23:19.740376+0000","last_undegraded":"2023-10-30T13:23:19.740376+0000","last_fullsized":"2023-10-30T13:23:19.740376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:17.664407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-30T13:23:18.964316+0000","last_change":"2023-10-30T13:23:08.208031+0000","last_active":"2023-10-30T13:23:18.964316+0000","last_peered":"2023-10-30T13:23:18.964316+0000","last_clean":"2023-10-30T13:23:18.964316+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T13:23:18.964316+0000","last_undegraded":"2023-10-30T13:23:18.964316+0000","last_fullsized":"2023-10-30T13:23:18.964316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_clean_scrub_stamp":"2023-10-30T13:23:07.179866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:22.691367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.711477+0000","last_change":"2023-10-30T13:23:18.979328+0000","last_active":"2023-10-30T13:23:19.711477+0000","last_peered":"2023-10-30T13:23:19.711477+0000","last_clean":"2023-10-30T13:23:19.711477+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T13:23:19.711477+0000","last_undegraded":"2023-10-30T13:23:19.711477+0000","last_fullsized":"2023-10-30T13:23:19.711477+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:04.942593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.712856+0000","last_change":"2023-10-30T13:23:19.711222+0000","last_active":"2023-10-30T13:23:19.712856+0000","last_peered":"2023-10-30T13:23:19.712856+0000","last_clean":"2023-10-30T13:23:19.712856+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T13:23:19.712856+0000","last_undegraded":"2023-10-30T13:23:19.712856+0000","last_fullsized":"2023-10-30T13:23:19.712856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:48:09.825585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.686363+0000","last_change":"2023-10-30T13:23:18.985318+0000","last_active":"2023-10-30T13:23:19.686363+0000","last_peered":"2023-10-30T13:23:19.686363+0000","last_clean":"2023-10-30T13:23:19.686363+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T13:23:19.686363+0000","last_undegraded":"2023-10-30T13:23:19.686363+0000","last_fullsized":"2023-10-30T13:23:19.686363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:16.754084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.719090+0000","last_change":"2023-10-30T13:23:19.717938+0000","last_active":"2023-10-30T13:23:19.719090+0000","last_peered":"2023-10-30T13:23:19.719090+0000","last_clean":"2023-10-30T13:23:19.719090+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T13:23:19.719090+0000","last_undegraded":"2023-10-30T13:23:19.719090+0000","last_fullsized":"2023-10-30T13:23:19.719090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:39:10.975966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.677163+0000","last_change":"2023-10-30T13:23:18.949565+0000","last_active":"2023-10-30T13:23:19.677163+0000","last_peered":"2023-10-30T13:23:19.677163+0000","last_clean":"2023-10-30T13:23:19.677163+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T13:23:19.677163+0000","last_undegraded":"2023-10-30T13:23:19.677163+0000","last_fullsized":"2023-10-30T13:23:19.677163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:55:46.029368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670291+0000","last_change":"2023-10-30T13:23:19.678037+0000","last_active":"2023-10-30T13:23:20.670291+0000","last_peered":"2023-10-30T13:23:20.670291+0000","last_clean":"2023-10-30T13:23:20.670291+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T13:23:20.670291+0000","last_undegraded":"2023-10-30T13:23:20.670291+0000","last_fullsized":"2023-10-30T13:23:20.670291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.712524+0000","last_change":"2023-10-30T13:23:18.984535+0000","last_active":"2023-10-30T13:23:19.712524+0000","last_peered":"2023-10-30T13:23:19.712524+0000","last_clean":"2023-10-30T13:23:19.712524+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T13:23:19.712524+0000","last_undegraded":"2023-10-30T13:23:19.712524+0000","last_fullsized":"2023-10-30T13:23:19.712524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:28.405405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675980+0000","last_change":"2023-10-30T13:23:19.691867+0000","last_active":"2023-10-30T13:23:20.675980+0000","last_peered":"2023-10-30T13:23:20.675980+0000","last_clean":"2023-10-30T13:23:20.675980+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T13:23:20.675980+0000","last_undegraded":"2023-10-30T13:23:20.675980+0000","last_fullsized":"2023-10-30T13:23:20.675980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:02.276042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670494+0000","last_change":"2023-10-30T13:23:18.958210+0000","last_active":"2023-10-30T13:23:20.670494+0000","last_peered":"2023-10-30T13:23:20.670494+0000","last_clean":"2023-10-30T13:23:20.670494+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T13:23:20.670494+0000","last_undegraded":"2023-10-30T13:23:20.670494+0000","last_fullsized":"2023-10-30T13:23:20.670494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:34.653005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.727698+0000","last_change":"2023-10-30T13:23:19.713174+0000","last_active":"2023-10-30T13:23:19.727698+0000","last_peered":"2023-10-30T13:23:19.727698+0000","last_clean":"2023-10-30T13:23:19.727698+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T13:23:19.727698+0000","last_undegraded":"2023-10-30T13:23:19.727698+0000","last_fullsized":"2023-10-30T13:23:19.727698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:31:10.138482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.691426+0000","last_change":"2023-10-30T13:23:18.985185+0000","last_active":"2023-10-30T13:23:19.691426+0000","last_peered":"2023-10-30T13:23:19.691426+0000","last_clean":"2023-10-30T13:23:19.691426+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T13:23:19.691426+0000","last_undegraded":"2023-10-30T13:23:19.691426+0000","last_fullsized":"2023-10-30T13:23:19.691426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:36.230072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670523+0000","last_change":"2023-10-30T13:23:19.680185+0000","last_active":"2023-10-30T13:23:20.670523+0000","last_peered":"2023-10-30T13:23:20.670523+0000","last_clean":"2023-10-30T13:23:20.670523+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T13:23:20.670523+0000","last_undegraded":"2023-10-30T13:23:20.670523+0000","last_fullsized":"2023-10-30T13:23:20.670523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670731+0000","last_change":"2023-10-30T13:23:18.978085+0000","last_active":"2023-10-30T13:23:20.670731+0000","last_peered":"2023-10-30T13:23:20.670731+0000","last_clean":"2023-10-30T13:23:20.670731+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T13:23:20.670731+0000","last_undegraded":"2023-10-30T13:23:20.670731+0000","last_fullsized":"2023-10-30T13:23:20.670731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:47.588972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.704966+0000","last_change":"2023-10-30T13:23:19.704291+0000","last_active":"2023-10-30T13:23:19.704966+0000","last_peered":"2023-10-30T13:23:19.704966+0000","last_clean":"2023-10-30T13:23:19.704966+0000","last_became_active":"2023-10-30T13:23:19.704066+0000","last_became_peered":"2023-10-30T13:23:19.704066+0000","last_unstale":"2023-10-30T13:23:19.704966+0000","last_undegraded":"2023-10-30T13:23:19.704966+0000","last_fullsized":"2023-10-30T13:23:19.704966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:36.143146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677207+0000","last_change":"2023-10-30T13:23:18.970566+0000","last_active":"2023-10-30T13:23:20.677207+0000","last_peered":"2023-10-30T13:23:20.677207+0000","last_clean":"2023-10-30T13:23:20.677207+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T13:23:20.677207+0000","last_undegraded":"2023-10-30T13:23:20.677207+0000","last_fullsized":"2023-10-30T13:23:20.677207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:03.851889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.688164+0000","last_change":"2023-10-30T13:23:19.685481+0000","last_active":"2023-10-30T13:23:19.688164+0000","last_peered":"2023-10-30T13:23:19.688164+0000","last_clean":"2023-10-30T13:23:19.688164+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T13:23:19.688164+0000","last_undegraded":"2023-10-30T13:23:19.688164+0000","last_fullsized":"2023-10-30T13:23:19.688164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:48:45.754147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.686031+0000","last_change":"2023-10-30T13:23:18.986394+0000","last_active":"2023-10-30T13:23:19.686031+0000","last_peered":"2023-10-30T13:23:19.686031+0000","last_clean":"2023-10-30T13:23:19.686031+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T13:23:19.686031+0000","last_undegraded":"2023-10-30T13:23:19.686031+0000","last_fullsized":"2023-10-30T13:23:19.686031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:59.986981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675944+0000","last_change":"2023-10-30T13:23:19.681239+0000","last_active":"2023-10-30T13:23:20.675944+0000","last_peered":"2023-10-30T13:23:20.675944+0000","last_clean":"2023-10-30T13:23:20.675944+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T13:23:20.675944+0000","last_undegraded":"2023-10-30T13:23:20.675944+0000","last_fullsized":"2023-10-30T13:23:20.675944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:00.121240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670308+0000","last_change":"2023-10-30T13:23:18.963905+0000","last_active":"2023-10-30T13:23:20.670308+0000","last_peered":"2023-10-30T13:23:20.670308+0000","last_clean":"2023-10-30T13:23:20.670308+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T13:23:20.670308+0000","last_undegraded":"2023-10-30T13:23:20.670308+0000","last_fullsized":"2023-10-30T13:23:20.670308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.681776+0000","last_change":"2023-10-30T13:23:19.678697+0000","last_active":"2023-10-30T13:23:19.681776+0000","last_peered":"2023-10-30T13:23:19.681776+0000","last_clean":"2023-10-30T13:23:19.681776+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T13:23:19.681776+0000","last_undegraded":"2023-10-30T13:23:19.681776+0000","last_fullsized":"2023-10-30T13:23:19.681776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:53.604002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670474+0000","last_change":"2023-10-30T13:23:18.952302+0000","last_active":"2023-10-30T13:23:20.670474+0000","last_peered":"2023-10-30T13:23:20.670474+0000","last_clean":"2023-10-30T13:23:20.670474+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T13:23:20.670474+0000","last_undegraded":"2023-10-30T13:23:20.670474+0000","last_fullsized":"2023-10-30T13:23:20.670474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:12.827417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.690206+0000","last_change":"2023-10-30T13:23:19.688716+0000","last_active":"2023-10-30T13:23:19.690206+0000","last_peered":"2023-10-30T13:23:19.690206+0000","last_clean":"2023-10-30T13:23:19.690206+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T13:23:19.690206+0000","last_undegraded":"2023-10-30T13:23:19.690206+0000","last_fullsized":"2023-10-30T13:23:19.690206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:23.730385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.695931+0000","last_change":"2023-10-30T13:23:18.984912+0000","last_active":"2023-10-30T13:23:19.695931+0000","last_peered":"2023-10-30T13:23:19.695931+0000","last_clean":"2023-10-30T13:23:19.695931+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T13:23:19.695931+0000","last_undegraded":"2023-10-30T13:23:19.695931+0000","last_fullsized":"2023-10-30T13:23:19.695931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:27:16.205042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674560+0000","last_change":"2023-10-30T13:23:19.718922+0000","last_active":"2023-10-30T13:23:20.674560+0000","last_peered":"2023-10-30T13:23:20.674560+0000","last_clean":"2023-10-30T13:23:20.674560+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T13:23:20.674560+0000","last_undegraded":"2023-10-30T13:23:20.674560+0000","last_fullsized":"2023-10-30T13:23:20.674560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.739431+0000","last_change":"2023-10-30T13:23:19.732322+0000","last_active":"2023-10-30T13:23:19.739431+0000","last_peered":"2023-10-30T13:23:19.739431+0000","last_clean":"2023-10-30T13:23:19.739431+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T13:23:19.739431+0000","last_undegraded":"2023-10-30T13:23:19.739431+0000","last_fullsized":"2023-10-30T13:23:19.739431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:32.096373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.690173+0000","last_change":"2023-10-30T13:23:18.993372+0000","last_active":"2023-10-30T13:23:19.690173+0000","last_peered":"2023-10-30T13:23:19.690173+0000","last_clean":"2023-10-30T13:23:19.690173+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T13:23:19.690173+0000","last_undegraded":"2023-10-30T13:23:19.690173+0000","last_fullsized":"2023-10-30T13:23:19.690173+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:41.370678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.739485+0000","last_change":"2023-10-30T13:23:19.732769+0000","last_active":"2023-10-30T13:23:19.739485+0000","last_peered":"2023-10-30T13:23:19.739485+0000","last_clean":"2023-10-30T13:23:19.739485+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T13:23:19.739485+0000","last_undegraded":"2023-10-30T13:23:19.739485+0000","last_fullsized":"2023-10-30T13:23:19.739485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:48.814488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.689269+0000","last_change":"2023-10-30T13:23:18.987633+0000","last_active":"2023-10-30T13:23:19.689269+0000","last_peered":"2023-10-30T13:23:19.689269+0000","last_clean":"2023-10-30T13:23:19.689269+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T13:23:19.689269+0000","last_undegraded":"2023-10-30T13:23:19.689269+0000","last_fullsized":"2023-10-30T13:23:19.689269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:09:33.324898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.723406+0000","last_change":"2023-10-30T13:23:19.722311+0000","last_active":"2023-10-30T13:23:19.723406+0000","last_peered":"2023-10-30T13:23:19.723406+0000","last_clean":"2023-10-30T13:23:19.723406+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T13:23:19.723406+0000","last_undegraded":"2023-10-30T13:23:19.723406+0000","last_fullsized":"2023-10-30T13:23:19.723406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:18.341639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.674428+0000","last_change":"2023-10-30T13:23:18.958470+0000","last_active":"2023-10-30T13:23:19.674428+0000","last_peered":"2023-10-30T13:23:19.674428+0000","last_clean":"2023-10-30T13:23:19.674428+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T13:23:19.674428+0000","last_undegraded":"2023-10-30T13:23:19.674428+0000","last_fullsized":"2023-10-30T13:23:19.674428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:21.507744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.700686+0000","last_change":"2023-10-30T13:23:19.699742+0000","last_active":"2023-10-30T13:23:19.700686+0000","last_peered":"2023-10-30T13:23:19.700686+0000","last_clean":"2023-10-30T13:23:19.700686+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T13:23:19.700686+0000","last_undegraded":"2023-10-30T13:23:19.700686+0000","last_fullsized":"2023-10-30T13:23:19.700686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:49.366134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.687576+0000","last_change":"2023-10-30T13:23:18.980952+0000","last_active":"2023-10-30T13:23:19.687576+0000","last_peered":"2023-10-30T13:23:19.687576+0000","last_clean":"2023-10-30T13:23:19.687576+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T13:23:19.687576+0000","last_undegraded":"2023-10-30T13:23:19.687576+0000","last_fullsized":"2023-10-30T13:23:19.687576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:44.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":786432,"data_stored":654966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.254}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2050000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6619999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4630000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9430000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":63579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":192512,"data_stored":64598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":188416,"data_stored":63579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":192512,"data_stored":64598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":782336,"data_stored":653947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:23:26.119 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:23:26.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f992ad96640 1 Processor -- start 2023-10-30T13:23:26.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f992ad96640 1 -- start start 2023-10-30T13:23:26.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f992ad96640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f992415cd50 0x7f99240c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:26.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f992ad96640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f99240cefa0 0x7f99240cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:26.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f992ad96640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992415a6c0 con 0x7f99241538f0 2023-10-30T13:23:26.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f992ad96640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99240cf8f0 con 0x7f992415cd50 2023-10-30T13:23:26.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f992ad96640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992415d300 con 0x7f99240cefa0 2023-10-30T13:23:26.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f9929d94640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f99241538f0 0x7f9924159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53164/0 (socket says 172.21.15.46:53164) 2023-10-30T13:23:26.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f9929d94640 1 -- 172.21.15.46:0/3111181088 learned_addr learned my addr 172.21.15.46:0/3111181088 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:26.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f9929593640 1 --2- 172.21.15.46:0/3111181088 >> v2:172.21.15.46:3300/0 conn(0x7f992415cd50 0x7f99240c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:26.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f992a595640 1 --2- 172.21.15.46:0/3111181088 >> v2:172.21.15.154:3300/0 conn(0x7f99240cefa0 0x7f99240cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:26.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f9928d92640 1 -- 172.21.15.46:0/3111181088 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443940688 0 0) 0x7f992415a6c0 con 0x7f99241538f0 2023-10-30T13:23:26.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f9929593640 1 -- 172.21.15.46:0/3111181088 >> v2:172.21.15.154:3300/0 conn(0x7f99240cefa0 msgr2=0x7f99240cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:26.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f9929593640 1 --2- 172.21.15.46:0/3111181088 >> v2:172.21.15.154:3300/0 conn(0x7f99240cefa0 0x7f99240cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f9929593640 1 -- 172.21.15.46:0/3111181088 >> v1:172.21.15.46:6789/0 conn(0x7f99241538f0 legacy=0x7f9924159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.292+0000 7f9929593640 1 -- 172.21.15.46:0/3111181088 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992415d5e0 con 0x7f992415cd50 2023-10-30T13:23:26.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.296+0000 7f9929593640 1 --2- 172.21.15.46:0/3111181088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992415cd50 0x7f99240c9f90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9924066a40 tx=0x7f991401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5bae523cb7dcb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:26.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.296+0000 7f9928d92640 1 -- 172.21.15.46:0/3111181088 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99140a8950 con 0x7f992415cd50 2023-10-30T13:23:26.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.296+0000 7f9928d92640 1 -- 172.21.15.46:0/3111181088 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99140a8ab0 con 0x7f992415cd50 2023-10-30T13:23:26.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.296+0000 7f9928d92640 1 -- 172.21.15.46:0/3111181088 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99140a8e40 con 0x7f992415cd50 2023-10-30T13:23:26.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.296+0000 7f992ad96640 1 -- 172.21.15.46:0/3111181088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992415cd50 msgr2=0x7f99240c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.296+0000 7f992ad96640 1 --2- 172.21.15.46:0/3111181088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992415cd50 0x7f99240c9f90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9924066a40 tx=0x7f991401ec50 comp rx=0 tx=0).stop 2023-10-30T13:23:26.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.296+0000 7f992ad96640 1 -- 172.21.15.46:0/3111181088 shutdown_connections 2023-10-30T13:23:26.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.296+0000 7f992ad96640 1 --2- 172.21.15.46:0/3111181088 >> v2:172.21.15.154:3300/0 conn(0x7f99240cefa0 0x7f99240cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.296+0000 7f992ad96640 1 --2- 172.21.15.46:0/3111181088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992415cd50 0x7f99240c9f90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.296+0000 7f992ad96640 1 -- 172.21.15.46:0/3111181088 >> 172.21.15.46:0/3111181088 conn(0x7f992406a640 msgr2=0x7f9924065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:26.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.296+0000 7f992ad96640 1 -- 172.21.15.46:0/3111181088 shutdown_connections 2023-10-30T13:23:26.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.296+0000 7f992ad96640 1 -- 172.21.15.46:0/3111181088 wait complete. 2023-10-30T13:23:26.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f992ad96640 1 Processor -- start 2023-10-30T13:23:26.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f992ad96640 1 -- start start 2023-10-30T13:23:26.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f992ad96640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99240cefa0 0x7f992415c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:26.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f992ad96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99241538f0 0x7f99240b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:26.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f992ad96640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f992415cd50 0x7f99240b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:26.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f992ad96640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99240cf4d0 con 0x7f99241538f0 2023-10-30T13:23:26.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f992ad96640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99241efb30 con 0x7f992415cd50 2023-10-30T13:23:26.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f992ad96640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99241573a0 con 0x7f99240cefa0 2023-10-30T13:23:26.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f9929593640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99241538f0 0x7f99240b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:26.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f9929593640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99241538f0 0x7f99240b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51978/0 (socket says 172.21.15.46:51978) 2023-10-30T13:23:26.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f9929593640 1 -- 172.21.15.46:0/2300500859 learned_addr learned my addr 172.21.15.46:0/2300500859 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:26.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f9929d94640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99240cefa0 0x7f992415c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:26.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f992a595640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f992415cd50 0x7f99240b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:26.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f9929593640 1 -- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99240cefa0 msgr2=0x7f992415c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f9929593640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99240cefa0 0x7f992415c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f9929593640 1 -- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f992415cd50 msgr2=0x7f99240b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f9929593640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f992415cd50 0x7f99240b33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f9929593640 1 -- 172.21.15.46:0/2300500859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9914009420 con 0x7f99241538f0 2023-10-30T13:23:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f9929593640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99241538f0 0x7f99240b2ea0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f99140b3040 tx=0x7f991401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f991a7fc640 1 -- 172.21.15.46:0/2300500859 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99140b9630 con 0x7f99241538f0 2023-10-30T13:23:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f991a7fc640 1 -- 172.21.15.46:0/2300500859 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99140b97b0 con 0x7f99241538f0 2023-10-30T13:23:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f991a7fc640 1 -- 172.21.15.46:0/2300500859 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99140b9b60 con 0x7f99241538f0 2023-10-30T13:23:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f992ad96640 1 -- 172.21.15.46:0/2300500859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9924157680 con 0x7f99241538f0 2023-10-30T13:23:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f992ad96640 1 -- 172.21.15.46:0/2300500859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9924157c40 con 0x7f99241538f0 2023-10-30T13:23:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.300+0000 7f992ad96640 1 -- 172.21.15.46:0/2300500859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992404e170 con 0x7f99241538f0 2023-10-30T13:23:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.304+0000 7f991a7fc640 1 -- 172.21.15.46:0/2300500859 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f99140b7040 con 0x7f99241538f0 2023-10-30T13:23:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.304+0000 7f991a7fc640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99080664a0 0x7f9908068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.304+0000 7f991a7fc640 1 -- 172.21.15.46:0/2300500859 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9914121ef0 con 0x7f99241538f0 2023-10-30T13:23:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.304+0000 7f9929d94640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99080664a0 0x7f9908068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:26.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.304+0000 7f9929d94640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99080664a0 0x7f9908068960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9920000f70 tx=0x7f9920007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.308+0000 7f991a7fc640 1 -- 172.21.15.46:0/2300500859 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f99140ef240 con 0x7f99241538f0 2023-10-30T13:23:26.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.464+0000 7f992ad96640 1 -- 172.21.15.46:0/2300500859 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f992415dba0 con 0x7f99080664a0 2023-10-30T13:23:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f991a7fc640 1 -- 172.21.15.46:0/2300500859 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264543 (secure 0 0 0) 0x7f99200071f0 con 0x7f99080664a0 2023-10-30T13:23:26.492 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:26.493 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T13:23:26.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f992ad96640 1 -- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99080664a0 msgr2=0x7f9908068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f992ad96640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99080664a0 0x7f9908068960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9920000f70 tx=0x7f9920007b60 comp rx=0 tx=0).stop 2023-10-30T13:23:26.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f992ad96640 1 -- 172.21.15.46:0/2300500859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99241538f0 msgr2=0x7f99240b2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f992ad96640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99241538f0 0x7f99240b2ea0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f99140b3040 tx=0x7f991401b9f0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f992ad96640 1 -- 172.21.15.46:0/2300500859 shutdown_connections 2023-10-30T13:23:26.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f992ad96640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99080664a0 0x7f9908068960 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f992ad96640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f992415cd50 0x7f99240b33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f992ad96640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99241538f0 0x7f99240b2ea0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f992ad96640 1 --2- 172.21.15.46:0/2300500859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99240cefa0 0x7f992415c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f992ad96640 1 -- 172.21.15.46:0/2300500859 >> 172.21.15.46:0/2300500859 conn(0x7f992406a640 msgr2=0x7f99240c3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:26.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f992ad96640 1 -- 172.21.15.46:0/2300500859 shutdown_connections 2023-10-30T13:23:26.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.484+0000 7f992ad96640 1 -- 172.21.15.46:0/2300500859 wait complete. 2023-10-30T13:23:26.512 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-30T13:23:24.710335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235516,"kb_used_data":2676,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739204,"statfs":{"total":773094113280,"available":772852944896,"internally_reserved":0,"allocated":2740224,"data_stored":1707063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":56,"apply_latency_ms":56,"commit_latency_ns":56000000,"apply_latency_ns":56000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002077"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740157+0000","last_change":"2023-10-30T13:23:19.719271+0000","last_active":"2023-10-30T13:23:19.740157+0000","last_peered":"2023-10-30T13:23:19.740157+0000","last_clean":"2023-10-30T13:23:19.740157+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T13:23:19.740157+0000","last_undegraded":"2023-10-30T13:23:19.740157+0000","last_fullsized":"2023-10-30T13:23:19.740157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:08:53.868003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670716+0000","last_change":"2023-10-30T13:23:19.687152+0000","last_active":"2023-10-30T13:23:20.670716+0000","last_peered":"2023-10-30T13:23:20.670716+0000","last_clean":"2023-10-30T13:23:20.670716+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T13:23:20.670716+0000","last_undegraded":"2023-10-30T13:23:20.670716+0000","last_fullsized":"2023-10-30T13:23:20.670716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.692768+0000","last_change":"2023-10-30T13:23:19.689554+0000","last_active":"2023-10-30T13:23:19.692768+0000","last_peered":"2023-10-30T13:23:19.692768+0000","last_clean":"2023-10-30T13:23:19.692768+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T13:23:19.692768+0000","last_undegraded":"2023-10-30T13:23:19.692768+0000","last_fullsized":"2023-10-30T13:23:19.692768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:59:48.180076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.729256+0000","last_change":"2023-10-30T13:23:19.714018+0000","last_active":"2023-10-30T13:23:19.729256+0000","last_peered":"2023-10-30T13:23:19.729256+0000","last_clean":"2023-10-30T13:23:19.729256+0000","last_became_active":"2023-10-30T13:23:19.711673+0000","last_became_peered":"2023-10-30T13:23:19.711673+0000","last_unstale":"2023-10-30T13:23:19.729256+0000","last_undegraded":"2023-10-30T13:23:19.729256+0000","last_fullsized":"2023-10-30T13:23:19.729256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:31:43.873032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740065+0000","last_change":"2023-10-30T13:23:19.719999+0000","last_active":"2023-10-30T13:23:19.740065+0000","last_peered":"2023-10-30T13:23:19.740065+0000","last_clean":"2023-10-30T13:23:19.740065+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T13:23:19.740065+0000","last_undegraded":"2023-10-30T13:23:19.740065+0000","last_fullsized":"2023-10-30T13:23:19.740065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:07:00.548753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.717402+0000","last_change":"2023-10-30T13:23:19.703474+0000","last_active":"2023-10-30T13:23:19.717402+0000","last_peered":"2023-10-30T13:23:19.717402+0000","last_clean":"2023-10-30T13:23:19.717402+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T13:23:19.717402+0000","last_undegraded":"2023-10-30T13:23:19.717402+0000","last_fullsized":"2023-10-30T13:23:19.717402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:38.408929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740452+0000","last_change":"2023-10-30T13:23:19.726985+0000","last_active":"2023-10-30T13:23:19.740452+0000","last_peered":"2023-10-30T13:23:19.740452+0000","last_clean":"2023-10-30T13:23:19.740452+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T13:23:19.740452+0000","last_undegraded":"2023-10-30T13:23:19.740452+0000","last_fullsized":"2023-10-30T13:23:19.740452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:16.978917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.715118+0000","last_change":"2023-10-30T13:23:19.714153+0000","last_active":"2023-10-30T13:23:19.715118+0000","last_peered":"2023-10-30T13:23:19.715118+0000","last_clean":"2023-10-30T13:23:19.715118+0000","last_became_active":"2023-10-30T13:23:19.713876+0000","last_became_peered":"2023-10-30T13:23:19.713876+0000","last_unstale":"2023-10-30T13:23:19.715118+0000","last_undegraded":"2023-10-30T13:23:19.715118+0000","last_fullsized":"2023-10-30T13:23:19.715118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:17:06.765922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740388+0000","last_change":"2023-10-30T13:23:19.724583+0000","last_active":"2023-10-30T13:23:19.740388+0000","last_peered":"2023-10-30T13:23:19.740388+0000","last_clean":"2023-10-30T13:23:19.740388+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T13:23:19.740388+0000","last_undegraded":"2023-10-30T13:23:19.740388+0000","last_fullsized":"2023-10-30T13:23:19.740388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:01:19.373370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677145+0000","last_change":"2023-10-30T13:23:19.704399+0000","last_active":"2023-10-30T13:23:20.677145+0000","last_peered":"2023-10-30T13:23:20.677145+0000","last_clean":"2023-10-30T13:23:20.677145+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T13:23:20.677145+0000","last_undegraded":"2023-10-30T13:23:20.677145+0000","last_fullsized":"2023-10-30T13:23:20.677145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676522+0000","last_change":"2023-10-30T13:23:19.726077+0000","last_active":"2023-10-30T13:23:20.676522+0000","last_peered":"2023-10-30T13:23:20.676522+0000","last_clean":"2023-10-30T13:23:20.676522+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T13:23:20.676522+0000","last_undegraded":"2023-10-30T13:23:20.676522+0000","last_fullsized":"2023-10-30T13:23:20.676522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675959+0000","last_change":"2023-10-30T13:23:19.720995+0000","last_active":"2023-10-30T13:23:20.675959+0000","last_peered":"2023-10-30T13:23:20.675959+0000","last_clean":"2023-10-30T13:23:20.675959+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T13:23:20.675959+0000","last_undegraded":"2023-10-30T13:23:20.675959+0000","last_fullsized":"2023-10-30T13:23:20.675959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:44.490568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.695027+0000","last_change":"2023-10-30T13:23:19.692509+0000","last_active":"2023-10-30T13:23:19.695027+0000","last_peered":"2023-10-30T13:23:19.695027+0000","last_clean":"2023-10-30T13:23:19.695027+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T13:23:19.695027+0000","last_undegraded":"2023-10-30T13:23:19.695027+0000","last_fullsized":"2023-10-30T13:23:19.695027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:36.485777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.688101+0000","last_change":"2023-10-30T13:23:19.685435+0000","last_active":"2023-10-30T13:23:19.688101+0000","last_peered":"2023-10-30T13:23:19.688101+0000","last_clean":"2023-10-30T13:23:19.688101+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T13:23:19.688101+0000","last_undegraded":"2023-10-30T13:23:19.688101+0000","last_fullsized":"2023-10-30T13:23:19.688101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:10:55.258514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670272+0000","last_change":"2023-10-30T13:23:19.706721+0000","last_active":"2023-10-30T13:23:20.670272+0000","last_peered":"2023-10-30T13:23:20.670272+0000","last_clean":"2023-10-30T13:23:20.670272+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T13:23:20.670272+0000","last_undegraded":"2023-10-30T13:23:20.670272+0000","last_fullsized":"2023-10-30T13:23:20.670272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.719618+0000","last_change":"2023-10-30T13:23:19.710432+0000","last_active":"2023-10-30T13:23:19.719618+0000","last_peered":"2023-10-30T13:23:19.719618+0000","last_clean":"2023-10-30T13:23:19.719618+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T13:23:19.719618+0000","last_undegraded":"2023-10-30T13:23:19.719618+0000","last_fullsized":"2023-10-30T13:23:19.719618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:13:14.565885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.747350+0000","last_change":"2023-10-30T13:23:19.741225+0000","last_active":"2023-10-30T13:23:19.747350+0000","last_peered":"2023-10-30T13:23:19.747350+0000","last_clean":"2023-10-30T13:23:19.747350+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T13:23:19.747350+0000","last_undegraded":"2023-10-30T13:23:19.747350+0000","last_fullsized":"2023-10-30T13:23:19.747350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:22:59.183745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.701718+0000","last_change":"2023-10-30T13:23:19.701024+0000","last_active":"2023-10-30T13:23:19.701718+0000","last_peered":"2023-10-30T13:23:19.701718+0000","last_clean":"2023-10-30T13:23:19.701718+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T13:23:19.701718+0000","last_undegraded":"2023-10-30T13:23:19.701718+0000","last_fullsized":"2023-10-30T13:23:19.701718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:48.598355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740337+0000","last_change":"2023-10-30T13:23:19.723929+0000","last_active":"2023-10-30T13:23:19.740337+0000","last_peered":"2023-10-30T13:23:19.740337+0000","last_clean":"2023-10-30T13:23:19.740337+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T13:23:19.740337+0000","last_undegraded":"2023-10-30T13:23:19.740337+0000","last_fullsized":"2023-10-30T13:23:19.740337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:52.219146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674806+0000","last_change":"2023-10-30T13:23:19.725101+0000","last_active":"2023-10-30T13:23:20.674806+0000","last_peered":"2023-10-30T13:23:20.674806+0000","last_clean":"2023-10-30T13:23:20.674806+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T13:23:20.674806+0000","last_undegraded":"2023-10-30T13:23:20.674806+0000","last_fullsized":"2023-10-30T13:23:20.674806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.722967+0000","last_change":"2023-10-30T13:23:19.721813+0000","last_active":"2023-10-30T13:23:19.722967+0000","last_peered":"2023-10-30T13:23:19.722967+0000","last_clean":"2023-10-30T13:23:19.722967+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T13:23:19.722967+0000","last_undegraded":"2023-10-30T13:23:19.722967+0000","last_fullsized":"2023-10-30T13:23:19.722967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:11:02.070206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.734522+0000","last_change":"2023-10-30T13:23:19.724763+0000","last_active":"2023-10-30T13:23:19.734522+0000","last_peered":"2023-10-30T13:23:19.734522+0000","last_clean":"2023-10-30T13:23:19.734522+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T13:23:19.734522+0000","last_undegraded":"2023-10-30T13:23:19.734522+0000","last_fullsized":"2023-10-30T13:23:19.734522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:09:14.832707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.717905+0000","last_change":"2023-10-30T13:23:19.716706+0000","last_active":"2023-10-30T13:23:19.717905+0000","last_peered":"2023-10-30T13:23:19.717905+0000","last_clean":"2023-10-30T13:23:19.717905+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T13:23:19.717905+0000","last_undegraded":"2023-10-30T13:23:19.717905+0000","last_fullsized":"2023-10-30T13:23:19.717905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:16.549139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.718012+0000","last_change":"2023-10-30T13:23:19.704791+0000","last_active":"2023-10-30T13:23:19.718012+0000","last_peered":"2023-10-30T13:23:19.718012+0000","last_clean":"2023-10-30T13:23:19.718012+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T13:23:19.718012+0000","last_undegraded":"2023-10-30T13:23:19.718012+0000","last_fullsized":"2023-10-30T13:23:19.718012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:02.097204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.718244+0000","last_change":"2023-10-30T13:23:19.717027+0000","last_active":"2023-10-30T13:23:19.718244+0000","last_peered":"2023-10-30T13:23:19.718244+0000","last_clean":"2023-10-30T13:23:19.718244+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T13:23:19.718244+0000","last_undegraded":"2023-10-30T13:23:19.718244+0000","last_fullsized":"2023-10-30T13:23:19.718244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:01.859148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677247+0000","last_change":"2023-10-30T13:23:19.706002+0000","last_active":"2023-10-30T13:23:20.677247+0000","last_peered":"2023-10-30T13:23:20.677247+0000","last_clean":"2023-10-30T13:23:20.677247+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T13:23:20.677247+0000","last_undegraded":"2023-10-30T13:23:20.677247+0000","last_fullsized":"2023-10-30T13:23:20.677247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:46.882474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670564+0000","last_change":"2023-10-30T13:23:19.678472+0000","last_active":"2023-10-30T13:23:20.670564+0000","last_peered":"2023-10-30T13:23:20.670564+0000","last_clean":"2023-10-30T13:23:20.670564+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T13:23:20.670564+0000","last_undegraded":"2023-10-30T13:23:20.670564+0000","last_fullsized":"2023-10-30T13:23:20.670564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.703220+0000","last_change":"2023-10-30T13:23:19.702155+0000","last_active":"2023-10-30T13:23:19.703220+0000","last_peered":"2023-10-30T13:23:19.703220+0000","last_clean":"2023-10-30T13:23:19.703220+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T13:23:19.703220+0000","last_undegraded":"2023-10-30T13:23:19.703220+0000","last_fullsized":"2023-10-30T13:23:19.703220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:05:38.564603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.689534+0000","last_change":"2023-10-30T13:23:19.687820+0000","last_active":"2023-10-30T13:23:19.689534+0000","last_peered":"2023-10-30T13:23:19.689534+0000","last_clean":"2023-10-30T13:23:19.689534+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T13:23:19.689534+0000","last_undegraded":"2023-10-30T13:23:19.689534+0000","last_fullsized":"2023-10-30T13:23:19.689534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:27:34.932720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.717351+0000","last_change":"2023-10-30T13:23:19.702442+0000","last_active":"2023-10-30T13:23:19.717351+0000","last_peered":"2023-10-30T13:23:19.717351+0000","last_clean":"2023-10-30T13:23:19.717351+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T13:23:19.717351+0000","last_undegraded":"2023-10-30T13:23:19.717351+0000","last_fullsized":"2023-10-30T13:23:19.717351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:51:00.809353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.712263+0000","last_change":"2023-10-30T13:23:19.711364+0000","last_active":"2023-10-30T13:23:19.712263+0000","last_peered":"2023-10-30T13:23:19.712263+0000","last_clean":"2023-10-30T13:23:19.712263+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T13:23:19.712263+0000","last_undegraded":"2023-10-30T13:23:19.712263+0000","last_fullsized":"2023-10-30T13:23:19.712263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:54.029087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674450+0000","last_change":"2023-10-30T13:23:19.698485+0000","last_active":"2023-10-30T13:23:20.674450+0000","last_peered":"2023-10-30T13:23:20.674450+0000","last_clean":"2023-10-30T13:23:20.674450+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T13:23:20.674450+0000","last_undegraded":"2023-10-30T13:23:20.674450+0000","last_fullsized":"2023-10-30T13:23:20.674450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675615+0000","last_change":"2023-10-30T13:23:19.703223+0000","last_active":"2023-10-30T13:23:20.675615+0000","last_peered":"2023-10-30T13:23:20.675615+0000","last_clean":"2023-10-30T13:23:20.675615+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T13:23:20.675615+0000","last_undegraded":"2023-10-30T13:23:20.675615+0000","last_fullsized":"2023-10-30T13:23:20.675615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:28.065471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.674639+0000","last_change":"2023-10-30T13:23:18.963933+0000","last_active":"2023-10-30T13:23:19.674639+0000","last_peered":"2023-10-30T13:23:19.674639+0000","last_clean":"2023-10-30T13:23:19.674639+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T13:23:19.674639+0000","last_undegraded":"2023-10-30T13:23:19.674639+0000","last_fullsized":"2023-10-30T13:23:19.674639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:48:58.196567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.719070+0000","last_change":"2023-10-30T13:23:19.717819+0000","last_active":"2023-10-30T13:23:19.719070+0000","last_peered":"2023-10-30T13:23:19.719070+0000","last_clean":"2023-10-30T13:23:19.719070+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T13:23:19.719070+0000","last_undegraded":"2023-10-30T13:23:19.719070+0000","last_fullsized":"2023-10-30T13:23:19.719070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:35.652687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674787+0000","last_change":"2023-10-30T13:23:18.976615+0000","last_active":"2023-10-30T13:23:20.674787+0000","last_peered":"2023-10-30T13:23:20.674787+0000","last_clean":"2023-10-30T13:23:20.674787+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T13:23:20.674787+0000","last_undegraded":"2023-10-30T13:23:20.674787+0000","last_fullsized":"2023-10-30T13:23:20.674787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:59.048396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.675429+0000","last_change":"2023-10-30T13:23:18.977357+0000","last_active":"2023-10-30T13:23:19.675429+0000","last_peered":"2023-10-30T13:23:19.675429+0000","last_clean":"2023-10-30T13:23:19.675429+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T13:23:19.675429+0000","last_undegraded":"2023-10-30T13:23:19.675429+0000","last_fullsized":"2023-10-30T13:23:19.675429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:53.502681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.690620+0000","last_change":"2023-10-30T13:23:19.689197+0000","last_active":"2023-10-30T13:23:19.690620+0000","last_peered":"2023-10-30T13:23:19.690620+0000","last_clean":"2023-10-30T13:23:19.690620+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T13:23:19.690620+0000","last_undegraded":"2023-10-30T13:23:19.690620+0000","last_fullsized":"2023-10-30T13:23:19.690620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:33:45.920519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.711652+0000","last_change":"2023-10-30T13:23:18.989299+0000","last_active":"2023-10-30T13:23:19.711652+0000","last_peered":"2023-10-30T13:23:19.711652+0000","last_clean":"2023-10-30T13:23:19.711652+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T13:23:19.711652+0000","last_undegraded":"2023-10-30T13:23:19.711652+0000","last_fullsized":"2023-10-30T13:23:19.711652+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:44.510184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670217+0000","last_change":"2023-10-30T13:23:19.679342+0000","last_active":"2023-10-30T13:23:20.670217+0000","last_peered":"2023-10-30T13:23:20.670217+0000","last_clean":"2023-10-30T13:23:20.670217+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T13:23:20.670217+0000","last_undegraded":"2023-10-30T13:23:20.670217+0000","last_fullsized":"2023-10-30T13:23:20.670217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677093+0000","last_change":"2023-10-30T13:23:18.989515+0000","last_active":"2023-10-30T13:23:20.677093+0000","last_peered":"2023-10-30T13:23:20.677093+0000","last_clean":"2023-10-30T13:23:20.677093+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T13:23:20.677093+0000","last_undegraded":"2023-10-30T13:23:20.677093+0000","last_fullsized":"2023-10-30T13:23:20.677093+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:35.133762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.739505+0000","last_change":"2023-10-30T13:23:19.732859+0000","last_active":"2023-10-30T13:23:19.739505+0000","last_peered":"2023-10-30T13:23:19.739505+0000","last_clean":"2023-10-30T13:23:19.739505+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T13:23:19.739505+0000","last_undegraded":"2023-10-30T13:23:19.739505+0000","last_fullsized":"2023-10-30T13:23:19.739505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:44.581517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.712355+0000","last_change":"2023-10-30T13:23:18.991454+0000","last_active":"2023-10-30T13:23:19.712355+0000","last_peered":"2023-10-30T13:23:19.712355+0000","last_clean":"2023-10-30T13:23:19.712355+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T13:23:19.712355+0000","last_undegraded":"2023-10-30T13:23:19.712355+0000","last_fullsized":"2023-10-30T13:23:19.712355+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:13.450999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.723415+0000","last_change":"2023-10-30T13:23:19.722347+0000","last_active":"2023-10-30T13:23:19.723415+0000","last_peered":"2023-10-30T13:23:19.723415+0000","last_clean":"2023-10-30T13:23:19.723415+0000","last_became_active":"2023-10-30T13:23:19.722077+0000","last_became_peered":"2023-10-30T13:23:19.722077+0000","last_unstale":"2023-10-30T13:23:19.723415+0000","last_undegraded":"2023-10-30T13:23:19.723415+0000","last_fullsized":"2023-10-30T13:23:19.723415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:26:46.361249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670533+0000","last_change":"2023-10-30T13:23:18.982337+0000","last_active":"2023-10-30T13:23:20.670533+0000","last_peered":"2023-10-30T13:23:20.670533+0000","last_clean":"2023-10-30T13:23:20.670533+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T13:23:20.670533+0000","last_undegraded":"2023-10-30T13:23:20.670533+0000","last_fullsized":"2023-10-30T13:23:20.670533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:09:39.996740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676299+0000","last_change":"2023-10-30T13:23:19.706643+0000","last_active":"2023-10-30T13:23:20.676299+0000","last_peered":"2023-10-30T13:23:20.676299+0000","last_clean":"2023-10-30T13:23:20.676299+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T13:23:20.676299+0000","last_undegraded":"2023-10-30T13:23:20.676299+0000","last_fullsized":"2023-10-30T13:23:20.676299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:29.238430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676290+0000","last_change":"2023-10-30T13:23:18.969882+0000","last_active":"2023-10-30T13:23:20.676290+0000","last_peered":"2023-10-30T13:23:20.676290+0000","last_clean":"2023-10-30T13:23:20.676290+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T13:23:20.676290+0000","last_undegraded":"2023-10-30T13:23:20.676290+0000","last_fullsized":"2023-10-30T13:23:20.676290+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:17.093448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740535+0000","last_change":"2023-10-30T13:23:19.728295+0000","last_active":"2023-10-30T13:23:19.740535+0000","last_peered":"2023-10-30T13:23:19.740535+0000","last_clean":"2023-10-30T13:23:19.740535+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T13:23:19.740535+0000","last_undegraded":"2023-10-30T13:23:19.740535+0000","last_fullsized":"2023-10-30T13:23:19.740535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:50:04.323145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.687496+0000","last_change":"2023-10-30T13:23:18.971632+0000","last_active":"2023-10-30T13:23:19.687496+0000","last_peered":"2023-10-30T13:23:19.687496+0000","last_clean":"2023-10-30T13:23:19.687496+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T13:23:19.687496+0000","last_undegraded":"2023-10-30T13:23:19.687496+0000","last_fullsized":"2023-10-30T13:23:19.687496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:54.428499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.719439+0000","last_change":"2023-10-30T13:23:19.718168+0000","last_active":"2023-10-30T13:23:19.719439+0000","last_peered":"2023-10-30T13:23:19.719439+0000","last_clean":"2023-10-30T13:23:19.719439+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T13:23:19.719439+0000","last_undegraded":"2023-10-30T13:23:19.719439+0000","last_fullsized":"2023-10-30T13:23:19.719439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:55:53.554238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.713067+0000","last_change":"2023-10-30T13:23:18.978420+0000","last_active":"2023-10-30T13:23:19.713067+0000","last_peered":"2023-10-30T13:23:19.713067+0000","last_clean":"2023-10-30T13:23:19.713067+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T13:23:19.713067+0000","last_undegraded":"2023-10-30T13:23:19.713067+0000","last_fullsized":"2023-10-30T13:23:19.713067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:10:44.511282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740233+0000","last_change":"2023-10-30T13:23:19.721434+0000","last_active":"2023-10-30T13:23:19.740233+0000","last_peered":"2023-10-30T13:23:19.740233+0000","last_clean":"2023-10-30T13:23:19.740233+0000","last_became_active":"2023-10-30T13:23:19.720704+0000","last_became_peered":"2023-10-30T13:23:19.720704+0000","last_unstale":"2023-10-30T13:23:19.740233+0000","last_undegraded":"2023-10-30T13:23:19.740233+0000","last_fullsized":"2023-10-30T13:23:19.740233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:15:16.602339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674378+0000","last_change":"2023-10-30T13:23:18.972707+0000","last_active":"2023-10-30T13:23:20.674378+0000","last_peered":"2023-10-30T13:23:20.674378+0000","last_clean":"2023-10-30T13:23:20.674378+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T13:23:20.674378+0000","last_undegraded":"2023-10-30T13:23:20.674378+0000","last_fullsized":"2023-10-30T13:23:20.674378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674810+0000","last_change":"2023-10-30T13:23:19.698274+0000","last_active":"2023-10-30T13:23:20.674810+0000","last_peered":"2023-10-30T13:23:20.674810+0000","last_clean":"2023-10-30T13:23:20.674810+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T13:23:20.674810+0000","last_undegraded":"2023-10-30T13:23:20.674810+0000","last_fullsized":"2023-10-30T13:23:20.674810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.676825+0000","last_change":"2023-10-30T13:23:18.948296+0000","last_active":"2023-10-30T13:23:19.676825+0000","last_peered":"2023-10-30T13:23:19.676825+0000","last_clean":"2023-10-30T13:23:19.676825+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T13:23:19.676825+0000","last_undegraded":"2023-10-30T13:23:19.676825+0000","last_fullsized":"2023-10-30T13:23:19.676825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:55.308329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740517+0000","last_change":"2023-10-30T13:23:19.728647+0000","last_active":"2023-10-30T13:23:19.740517+0000","last_peered":"2023-10-30T13:23:19.740517+0000","last_clean":"2023-10-30T13:23:19.740517+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T13:23:19.740517+0000","last_undegraded":"2023-10-30T13:23:19.740517+0000","last_fullsized":"2023-10-30T13:23:19.740517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:42.328530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675830+0000","last_change":"2023-10-30T13:23:18.962343+0000","last_active":"2023-10-30T13:23:20.675830+0000","last_peered":"2023-10-30T13:23:20.675830+0000","last_clean":"2023-10-30T13:23:20.675830+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T13:23:20.675830+0000","last_undegraded":"2023-10-30T13:23:20.675830+0000","last_fullsized":"2023-10-30T13:23:20.675830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:36.082164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740109+0000","last_change":"2023-10-30T13:23:19.720770+0000","last_active":"2023-10-30T13:23:19.740109+0000","last_peered":"2023-10-30T13:23:19.740109+0000","last_clean":"2023-10-30T13:23:19.740109+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T13:23:19.740109+0000","last_undegraded":"2023-10-30T13:23:19.740109+0000","last_fullsized":"2023-10-30T13:23:19.740109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:32.308859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.677370+0000","last_change":"2023-10-30T13:23:18.949081+0000","last_active":"2023-10-30T13:23:19.677370+0000","last_peered":"2023-10-30T13:23:19.677370+0000","last_clean":"2023-10-30T13:23:19.677370+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T13:23:19.677370+0000","last_undegraded":"2023-10-30T13:23:19.677370+0000","last_fullsized":"2023-10-30T13:23:19.677370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:00.356885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740342+0000","last_change":"2023-10-30T13:23:19.724180+0000","last_active":"2023-10-30T13:23:19.740342+0000","last_peered":"2023-10-30T13:23:19.740342+0000","last_clean":"2023-10-30T13:23:19.740342+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T13:23:19.740342+0000","last_undegraded":"2023-10-30T13:23:19.740342+0000","last_fullsized":"2023-10-30T13:23:19.740342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:11:20.637012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670614+0000","last_change":"2023-10-30T13:23:18.970780+0000","last_active":"2023-10-30T13:23:20.670614+0000","last_peered":"2023-10-30T13:23:20.670614+0000","last_clean":"2023-10-30T13:23:20.670614+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T13:23:20.670614+0000","last_undegraded":"2023-10-30T13:23:20.670614+0000","last_fullsized":"2023-10-30T13:23:20.670614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:38:51.113629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.718082+0000","last_change":"2023-10-30T13:23:19.704834+0000","last_active":"2023-10-30T13:23:19.718082+0000","last_peered":"2023-10-30T13:23:19.718082+0000","last_clean":"2023-10-30T13:23:19.718082+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T13:23:19.718082+0000","last_undegraded":"2023-10-30T13:23:19.718082+0000","last_fullsized":"2023-10-30T13:23:19.718082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:12:23.420417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.677209+0000","last_change":"2023-10-30T13:23:18.977920+0000","last_active":"2023-10-30T13:23:19.677209+0000","last_peered":"2023-10-30T13:23:19.677209+0000","last_clean":"2023-10-30T13:23:19.677209+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T13:23:19.677209+0000","last_undegraded":"2023-10-30T13:23:19.677209+0000","last_fullsized":"2023-10-30T13:23:19.677209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:07:59.139948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.740376+0000","last_change":"2023-10-30T13:23:19.724497+0000","last_active":"2023-10-30T13:23:19.740376+0000","last_peered":"2023-10-30T13:23:19.740376+0000","last_clean":"2023-10-30T13:23:19.740376+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T13:23:19.740376+0000","last_undegraded":"2023-10-30T13:23:19.740376+0000","last_fullsized":"2023-10-30T13:23:19.740376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:17.664407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-30T13:23:18.964316+0000","last_change":"2023-10-30T13:23:08.208031+0000","last_active":"2023-10-30T13:23:18.964316+0000","last_peered":"2023-10-30T13:23:18.964316+0000","last_clean":"2023-10-30T13:23:18.964316+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T13:23:18.964316+0000","last_undegraded":"2023-10-30T13:23:18.964316+0000","last_fullsized":"2023-10-30T13:23:18.964316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_clean_scrub_stamp":"2023-10-30T13:23:07.179866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:22.691367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.711477+0000","last_change":"2023-10-30T13:23:18.979328+0000","last_active":"2023-10-30T13:23:19.711477+0000","last_peered":"2023-10-30T13:23:19.711477+0000","last_clean":"2023-10-30T13:23:19.711477+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T13:23:19.711477+0000","last_undegraded":"2023-10-30T13:23:19.711477+0000","last_fullsized":"2023-10-30T13:23:19.711477+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:04.942593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.712856+0000","last_change":"2023-10-30T13:23:19.711222+0000","last_active":"2023-10-30T13:23:19.712856+0000","last_peered":"2023-10-30T13:23:19.712856+0000","last_clean":"2023-10-30T13:23:19.712856+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T13:23:19.712856+0000","last_undegraded":"2023-10-30T13:23:19.712856+0000","last_fullsized":"2023-10-30T13:23:19.712856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:48:09.825585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.686363+0000","last_change":"2023-10-30T13:23:18.985318+0000","last_active":"2023-10-30T13:23:19.686363+0000","last_peered":"2023-10-30T13:23:19.686363+0000","last_clean":"2023-10-30T13:23:19.686363+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T13:23:19.686363+0000","last_undegraded":"2023-10-30T13:23:19.686363+0000","last_fullsized":"2023-10-30T13:23:19.686363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:16.754084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.719090+0000","last_change":"2023-10-30T13:23:19.717938+0000","last_active":"2023-10-30T13:23:19.719090+0000","last_peered":"2023-10-30T13:23:19.719090+0000","last_clean":"2023-10-30T13:23:19.719090+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T13:23:19.719090+0000","last_undegraded":"2023-10-30T13:23:19.719090+0000","last_fullsized":"2023-10-30T13:23:19.719090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:39:10.975966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.677163+0000","last_change":"2023-10-30T13:23:18.949565+0000","last_active":"2023-10-30T13:23:19.677163+0000","last_peered":"2023-10-30T13:23:19.677163+0000","last_clean":"2023-10-30T13:23:19.677163+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T13:23:19.677163+0000","last_undegraded":"2023-10-30T13:23:19.677163+0000","last_fullsized":"2023-10-30T13:23:19.677163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:55:46.029368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670291+0000","last_change":"2023-10-30T13:23:19.678037+0000","last_active":"2023-10-30T13:23:20.670291+0000","last_peered":"2023-10-30T13:23:20.670291+0000","last_clean":"2023-10-30T13:23:20.670291+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T13:23:20.670291+0000","last_undegraded":"2023-10-30T13:23:20.670291+0000","last_fullsized":"2023-10-30T13:23:20.670291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.712524+0000","last_change":"2023-10-30T13:23:18.984535+0000","last_active":"2023-10-30T13:23:19.712524+0000","last_peered":"2023-10-30T13:23:19.712524+0000","last_clean":"2023-10-30T13:23:19.712524+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T13:23:19.712524+0000","last_undegraded":"2023-10-30T13:23:19.712524+0000","last_fullsized":"2023-10-30T13:23:19.712524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:28.405405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675980+0000","last_change":"2023-10-30T13:23:19.691867+0000","last_active":"2023-10-30T13:23:20.675980+0000","last_peered":"2023-10-30T13:23:20.675980+0000","last_clean":"2023-10-30T13:23:20.675980+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T13:23:20.675980+0000","last_undegraded":"2023-10-30T13:23:20.675980+0000","last_fullsized":"2023-10-30T13:23:20.675980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:02.276042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670494+0000","last_change":"2023-10-30T13:23:18.958210+0000","last_active":"2023-10-30T13:23:20.670494+0000","last_peered":"2023-10-30T13:23:20.670494+0000","last_clean":"2023-10-30T13:23:20.670494+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T13:23:20.670494+0000","last_undegraded":"2023-10-30T13:23:20.670494+0000","last_fullsized":"2023-10-30T13:23:20.670494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:34.653005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.727698+0000","last_change":"2023-10-30T13:23:19.713174+0000","last_active":"2023-10-30T13:23:19.727698+0000","last_peered":"2023-10-30T13:23:19.727698+0000","last_clean":"2023-10-30T13:23:19.727698+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T13:23:19.727698+0000","last_undegraded":"2023-10-30T13:23:19.727698+0000","last_fullsized":"2023-10-30T13:23:19.727698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:31:10.138482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.691426+0000","last_change":"2023-10-30T13:23:18.985185+0000","last_active":"2023-10-30T13:23:19.691426+0000","last_peered":"2023-10-30T13:23:19.691426+0000","last_clean":"2023-10-30T13:23:19.691426+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T13:23:19.691426+0000","last_undegraded":"2023-10-30T13:23:19.691426+0000","last_fullsized":"2023-10-30T13:23:19.691426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:36.230072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670523+0000","last_change":"2023-10-30T13:23:19.680185+0000","last_active":"2023-10-30T13:23:20.670523+0000","last_peered":"2023-10-30T13:23:20.670523+0000","last_clean":"2023-10-30T13:23:20.670523+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T13:23:20.670523+0000","last_undegraded":"2023-10-30T13:23:20.670523+0000","last_fullsized":"2023-10-30T13:23:20.670523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670731+0000","last_change":"2023-10-30T13:23:18.978085+0000","last_active":"2023-10-30T13:23:20.670731+0000","last_peered":"2023-10-30T13:23:20.670731+0000","last_clean":"2023-10-30T13:23:20.670731+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T13:23:20.670731+0000","last_undegraded":"2023-10-30T13:23:20.670731+0000","last_fullsized":"2023-10-30T13:23:20.670731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:47.588972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.704966+0000","last_change":"2023-10-30T13:23:19.704291+0000","last_active":"2023-10-30T13:23:19.704966+0000","last_peered":"2023-10-30T13:23:19.704966+0000","last_clean":"2023-10-30T13:23:19.704966+0000","last_became_active":"2023-10-30T13:23:19.704066+0000","last_became_peered":"2023-10-30T13:23:19.704066+0000","last_unstale":"2023-10-30T13:23:19.704966+0000","last_undegraded":"2023-10-30T13:23:19.704966+0000","last_fullsized":"2023-10-30T13:23:19.704966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:36.143146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677207+0000","last_change":"2023-10-30T13:23:18.970566+0000","last_active":"2023-10-30T13:23:20.677207+0000","last_peered":"2023-10-30T13:23:20.677207+0000","last_clean":"2023-10-30T13:23:20.677207+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T13:23:20.677207+0000","last_undegraded":"2023-10-30T13:23:20.677207+0000","last_fullsized":"2023-10-30T13:23:20.677207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:03.851889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.688164+0000","last_change":"2023-10-30T13:23:19.685481+0000","last_active":"2023-10-30T13:23:19.688164+0000","last_peered":"2023-10-30T13:23:19.688164+0000","last_clean":"2023-10-30T13:23:19.688164+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T13:23:19.688164+0000","last_undegraded":"2023-10-30T13:23:19.688164+0000","last_fullsized":"2023-10-30T13:23:19.688164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:48:45.754147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.686031+0000","last_change":"2023-10-30T13:23:18.986394+0000","last_active":"2023-10-30T13:23:19.686031+0000","last_peered":"2023-10-30T13:23:19.686031+0000","last_clean":"2023-10-30T13:23:19.686031+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T13:23:19.686031+0000","last_undegraded":"2023-10-30T13:23:19.686031+0000","last_fullsized":"2023-10-30T13:23:19.686031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:59.986981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675944+0000","last_change":"2023-10-30T13:23:19.681239+0000","last_active":"2023-10-30T13:23:20.675944+0000","last_peered":"2023-10-30T13:23:20.675944+0000","last_clean":"2023-10-30T13:23:20.675944+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T13:23:20.675944+0000","last_undegraded":"2023-10-30T13:23:20.675944+0000","last_fullsized":"2023-10-30T13:23:20.675944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:00.121240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670308+0000","last_change":"2023-10-30T13:23:18.963905+0000","last_active":"2023-10-30T13:23:20.670308+0000","last_peered":"2023-10-30T13:23:20.670308+0000","last_clean":"2023-10-30T13:23:20.670308+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T13:23:20.670308+0000","last_undegraded":"2023-10-30T13:23:20.670308+0000","last_fullsized":"2023-10-30T13:23:20.670308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.681776+0000","last_change":"2023-10-30T13:23:19.678697+0000","last_active":"2023-10-30T13:23:19.681776+0000","last_peered":"2023-10-30T13:23:19.681776+0000","last_clean":"2023-10-30T13:23:19.681776+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T13:23:19.681776+0000","last_undegraded":"2023-10-30T13:23:19.681776+0000","last_fullsized":"2023-10-30T13:23:19.681776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:53.604002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670474+0000","last_change":"2023-10-30T13:23:18.952302+0000","last_active":"2023-10-30T13:23:20.670474+0000","last_peered":"2023-10-30T13:23:20.670474+0000","last_clean":"2023-10-30T13:23:20.670474+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T13:23:20.670474+0000","last_undegraded":"2023-10-30T13:23:20.670474+0000","last_fullsized":"2023-10-30T13:23:20.670474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:12.827417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.690206+0000","last_change":"2023-10-30T13:23:19.688716+0000","last_active":"2023-10-30T13:23:19.690206+0000","last_peered":"2023-10-30T13:23:19.690206+0000","last_clean":"2023-10-30T13:23:19.690206+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T13:23:19.690206+0000","last_undegraded":"2023-10-30T13:23:19.690206+0000","last_fullsized":"2023-10-30T13:23:19.690206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:02:23.730385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.695931+0000","last_change":"2023-10-30T13:23:18.984912+0000","last_active":"2023-10-30T13:23:19.695931+0000","last_peered":"2023-10-30T13:23:19.695931+0000","last_clean":"2023-10-30T13:23:19.695931+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T13:23:19.695931+0000","last_undegraded":"2023-10-30T13:23:19.695931+0000","last_fullsized":"2023-10-30T13:23:19.695931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:27:16.205042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674560+0000","last_change":"2023-10-30T13:23:19.718922+0000","last_active":"2023-10-30T13:23:20.674560+0000","last_peered":"2023-10-30T13:23:20.674560+0000","last_clean":"2023-10-30T13:23:20.674560+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T13:23:20.674560+0000","last_undegraded":"2023-10-30T13:23:20.674560+0000","last_fullsized":"2023-10-30T13:23:20.674560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.739431+0000","last_change":"2023-10-30T13:23:19.732322+0000","last_active":"2023-10-30T13:23:19.739431+0000","last_peered":"2023-10-30T13:23:19.739431+0000","last_clean":"2023-10-30T13:23:19.739431+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T13:23:19.739431+0000","last_undegraded":"2023-10-30T13:23:19.739431+0000","last_fullsized":"2023-10-30T13:23:19.739431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:32.096373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.690173+0000","last_change":"2023-10-30T13:23:18.993372+0000","last_active":"2023-10-30T13:23:19.690173+0000","last_peered":"2023-10-30T13:23:19.690173+0000","last_clean":"2023-10-30T13:23:19.690173+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T13:23:19.690173+0000","last_undegraded":"2023-10-30T13:23:19.690173+0000","last_fullsized":"2023-10-30T13:23:19.690173+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:41.370678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.739485+0000","last_change":"2023-10-30T13:23:19.732769+0000","last_active":"2023-10-30T13:23:19.739485+0000","last_peered":"2023-10-30T13:23:19.739485+0000","last_clean":"2023-10-30T13:23:19.739485+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T13:23:19.739485+0000","last_undegraded":"2023-10-30T13:23:19.739485+0000","last_fullsized":"2023-10-30T13:23:19.739485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:48.814488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.689269+0000","last_change":"2023-10-30T13:23:18.987633+0000","last_active":"2023-10-30T13:23:19.689269+0000","last_peered":"2023-10-30T13:23:19.689269+0000","last_clean":"2023-10-30T13:23:19.689269+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T13:23:19.689269+0000","last_undegraded":"2023-10-30T13:23:19.689269+0000","last_fullsized":"2023-10-30T13:23:19.689269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:09:33.324898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.723406+0000","last_change":"2023-10-30T13:23:19.722311+0000","last_active":"2023-10-30T13:23:19.723406+0000","last_peered":"2023-10-30T13:23:19.723406+0000","last_clean":"2023-10-30T13:23:19.723406+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T13:23:19.723406+0000","last_undegraded":"2023-10-30T13:23:19.723406+0000","last_fullsized":"2023-10-30T13:23:19.723406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:18.341639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.674428+0000","last_change":"2023-10-30T13:23:18.958470+0000","last_active":"2023-10-30T13:23:19.674428+0000","last_peered":"2023-10-30T13:23:19.674428+0000","last_clean":"2023-10-30T13:23:19.674428+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T13:23:19.674428+0000","last_undegraded":"2023-10-30T13:23:19.674428+0000","last_fullsized":"2023-10-30T13:23:19.674428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:21.507744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.700686+0000","last_change":"2023-10-30T13:23:19.699742+0000","last_active":"2023-10-30T13:23:19.700686+0000","last_peered":"2023-10-30T13:23:19.700686+0000","last_clean":"2023-10-30T13:23:19.700686+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T13:23:19.700686+0000","last_undegraded":"2023-10-30T13:23:19.700686+0000","last_fullsized":"2023-10-30T13:23:19.700686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:49.366134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.687576+0000","last_change":"2023-10-30T13:23:18.980952+0000","last_active":"2023-10-30T13:23:19.687576+0000","last_peered":"2023-10-30T13:23:19.687576+0000","last_clean":"2023-10-30T13:23:19.687576+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T13:23:19.687576+0000","last_undegraded":"2023-10-30T13:23:19.687576+0000","last_fullsized":"2023-10-30T13:23:19.687576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:44.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":786432,"data_stored":654966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.254}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2050000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6619999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4630000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9430000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":63579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":192512,"data_stored":64598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":188416,"data_stored":63579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":192512,"data_stored":64598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":782336,"data_stored":653947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:23:26.517 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-30T13:23:26.518 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-30T13:23:26.518 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-30T13:23:26.519 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-30T13:23:26.519 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-30T13:23:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.692+0000 7f073a69a640 1 Processor -- start 2023-10-30T13:23:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.692+0000 7f073a69a640 1 -- start start 2023-10-30T13:23:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.692+0000 7f073a69a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0734070740 0x7f07340c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.692+0000 7f073a69a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f073415e700 0x7f073406a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.692+0000 7f073a69a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07340c5810 con 0x7f07341580e0 2023-10-30T13:23:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.692+0000 7f073a69a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073415eb70 con 0x7f0734070740 2023-10-30T13:23:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.692+0000 7f073a69a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07340c6bb0 con 0x7f073415e700 2023-10-30T13:23:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f07337fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0734070740 0x7f07340c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:26.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f07337fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0734070740 0x7f07340c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39390/0 (socket says 172.21.15.46:39390) 2023-10-30T13:23:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f07337fe640 1 -- 172.21.15.46:0/2215109373 learned_addr learned my addr 172.21.15.46:0/2215109373 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f0733fff640 1 --1- 172.21.15.46:0/2215109373 >> v1:172.21.15.46:6789/0 conn(0x7f07341580e0 0x7f0734069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48436/0 (socket says 172.21.15.46:48436) 2023-10-30T13:23:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f0738c37640 1 --2- 172.21.15.46:0/2215109373 >> v2:172.21.15.154:3300/0 conn(0x7f073415e700 0x7f073406a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f07337fe640 1 -- 172.21.15.46:0/2215109373 >> v2:172.21.15.154:3300/0 conn(0x7f073415e700 msgr2=0x7f073406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f07337fe640 1 --2- 172.21.15.46:0/2215109373 >> v2:172.21.15.154:3300/0 conn(0x7f073415e700 0x7f073406a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f07337fe640 1 -- 172.21.15.46:0/2215109373 >> v1:172.21.15.46:6789/0 conn(0x7f07341580e0 legacy=0x7f0734069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f07337fe640 1 -- 172.21.15.46:0/2215109373 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07340c6e70 con 0x7f0734070740 2023-10-30T13:23:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f0732ffd640 1 -- 172.21.15.46:0/2215109373 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144301045 0 0) 0x7f07340c5810 con 0x7f07341580e0 2023-10-30T13:23:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f07337fe640 1 --2- 172.21.15.46:0/2215109373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0734070740 0x7f07340c6540 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f07340cbeb0 tx=0x7f072801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599e058ec4872adc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f0732ffd640 1 -- 172.21.15.46:0/2215109373 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07280ab5f0 con 0x7f0734070740 2023-10-30T13:23:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f0732ffd640 1 -- 172.21.15.46:0/2215109373 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07280ab750 con 0x7f0734070740 2023-10-30T13:23:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.696+0000 7f0732ffd640 1 -- 172.21.15.46:0/2215109373 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07280abae0 con 0x7f0734070740 2023-10-30T13:23:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 -- 172.21.15.46:0/2215109373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0734070740 msgr2=0x7f07340c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 --2- 172.21.15.46:0/2215109373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0734070740 0x7f07340c6540 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f07340cbeb0 tx=0x7f072801c500 comp rx=0 tx=0).stop 2023-10-30T13:23:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 -- 172.21.15.46:0/2215109373 shutdown_connections 2023-10-30T13:23:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 --2- 172.21.15.46:0/2215109373 >> v2:172.21.15.154:3300/0 conn(0x7f073415e700 0x7f073406a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 --2- 172.21.15.46:0/2215109373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0734070740 0x7f07340c6540 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 -- 172.21.15.46:0/2215109373 >> 172.21.15.46:0/2215109373 conn(0x7f07340c7910 msgr2=0x7f07340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 -- 172.21.15.46:0/2215109373 shutdown_connections 2023-10-30T13:23:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 -- 172.21.15.46:0/2215109373 wait complete. 2023-10-30T13:23:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 Processor -- start 2023-10-30T13:23:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 -- start start 2023-10-30T13:23:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0734070740 0x7f073413baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07341580e0 0x7f07341411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f073415e700 0x7f07340c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073413ec40 con 0x7f073415e700 2023-10-30T13:23:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07340c8f60 con 0x7f0734070740 2023-10-30T13:23:26.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f073a69a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073414a3b0 con 0x7f07341580e0 2023-10-30T13:23:26.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f0738c37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f073415e700 0x7f07340c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:26.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f0733fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0734070740 0x7f073413baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:26.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f0738c37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f073415e700 0x7f07340c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39398/0 (socket says 172.21.15.46:39398) 2023-10-30T13:23:26.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f0733fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0734070740 0x7f073413baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39110/0 (socket says 172.21.15.46:39110) 2023-10-30T13:23:26.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f0738c37640 1 -- 172.21.15.46:0/415894101 learned_addr learned my addr 172.21.15.46:0/415894101 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:26.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f07337fe640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07341580e0 0x7f07341411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:26.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f0738c37640 1 -- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07341580e0 msgr2=0x7f07341411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f0738c37640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07341580e0 0x7f07341411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f0738c37640 1 -- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0734070740 msgr2=0x7f073413baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f0738c37640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0734070740 0x7f073413baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.700+0000 7f0738c37640 1 -- 172.21.15.46:0/415894101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07280094d0 con 0x7f073415e700 2023-10-30T13:23:26.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.704+0000 7f0738c37640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f073415e700 0x7f07340c9f90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f072c007e90 tx=0x7f072c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:26.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.704+0000 7f0730ff9640 1 -- 172.21.15.46:0/415894101 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c014420 con 0x7f073415e700 2023-10-30T13:23:26.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.704+0000 7f0730ff9640 1 -- 172.21.15.46:0/415894101 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072c014580 con 0x7f073415e700 2023-10-30T13:23:26.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.704+0000 7f073a69a640 1 -- 172.21.15.46:0/415894101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07341f9fe0 con 0x7f073415e700 2023-10-30T13:23:26.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.704+0000 7f0730ff9640 1 -- 172.21.15.46:0/415894101 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c014930 con 0x7f073415e700 2023-10-30T13:23:26.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.704+0000 7f073a69a640 1 -- 172.21.15.46:0/415894101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07340c94b0 con 0x7f073415e700 2023-10-30T13:23:26.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.704+0000 7f073a69a640 1 -- 172.21.15.46:0/415894101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073404e170 con 0x7f073415e700 2023-10-30T13:23:26.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.704+0000 7f0730ff9640 1 -- 172.21.15.46:0/415894101 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f072c014a90 con 0x7f073415e700 2023-10-30T13:23:26.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.704+0000 7f0730ff9640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07200664a0 0x7f0720068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:26.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.704+0000 7f0730ff9640 1 -- 172.21.15.46:0/415894101 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f072c084690 con 0x7f073415e700 2023-10-30T13:23:26.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.708+0000 7f0733fff640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07200664a0 0x7f0720068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:26.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.708+0000 7f0733fff640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07200664a0 0x7f0720068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0724007fd0 tx=0x7f0724007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:26.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.708+0000 7f0730ff9640 1 -- 172.21.15.46:0/415894101 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f072c051b30 con 0x7f073415e700 2023-10-30T13:23:26.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.904+0000 7f073a69a640 1 -- 172.21.15.46:0/415894101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0734141860 con 0x7f073415e700 2023-10-30T13:23:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.908+0000 7f0730ff9640 1 -- 172.21.15.46:0/415894101 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164433 (secure 0 0 0) 0x7f072c055150 con 0x7f073415e700 2023-10-30T13:23:26.917 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:26.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.912+0000 7f073a69a640 1 -- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07200664a0 msgr2=0x7f0720068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.912+0000 7f073a69a640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07200664a0 0x7f0720068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0724007fd0 tx=0x7f0724007d60 comp rx=0 tx=0).stop 2023-10-30T13:23:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.912+0000 7f073a69a640 1 -- 172.21.15.46:0/415894101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f073415e700 msgr2=0x7f07340c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.912+0000 7f073a69a640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f073415e700 0x7f07340c9f90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f072c007e90 tx=0x7f072c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.912+0000 7f073a69a640 1 -- 172.21.15.46:0/415894101 shutdown_connections 2023-10-30T13:23:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.912+0000 7f073a69a640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07200664a0 0x7f0720068960 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.912+0000 7f073a69a640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f073415e700 0x7f07340c9f90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.912+0000 7f073a69a640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07341580e0 0x7f07341411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.912+0000 7f073a69a640 1 --2- 172.21.15.46:0/415894101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0734070740 0x7f073413baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.912+0000 7f073a69a640 1 -- 172.21.15.46:0/415894101 >> 172.21.15.46:0/415894101 conn(0x7f07340c7910 msgr2=0x7f0734156c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.912+0000 7f073a69a640 1 -- 172.21.15.46:0/415894101 shutdown_connections 2023-10-30T13:23:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:26.912+0000 7f073a69a640 1 -- 172.21.15.46:0/415894101 wait complete. 2023-10-30T13:23:26.937 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":313953570},{"type":"v1","addr":"172.21.15.154:6833","nonce":313953570}]},"active_addr":"172.21.15.154:6833/313953570","active_change":"2023-10-30T13:22:58.691231+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":2937950077}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":1971914066}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":676668762}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":1491828548}]}]} 2023-10-30T13:23:26.941 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-30T13:23:26.941 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-30T13:23:26.942 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:27.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05e2d1d640 1 Processor -- start 2023-10-30T13:23:27.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05e2d1d640 1 -- start start 2023-10-30T13:23:27.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05e2d1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc1538a0 0x7f05dc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05e2d1d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f05dc0cf160 0x7f05dc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05e2d1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05dc0cb800 con 0x7f05dc0ca0e0 2023-10-30T13:23:27.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05e2d1d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05dc15dc30 con 0x7f05dc0cf160 2023-10-30T13:23:27.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05e2d1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc0cb330 con 0x7f05dc1538a0 2023-10-30T13:23:27.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05dbfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f05dc0ca0e0 0x7f05dc15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48446/0 (socket says 172.21.15.46:48446) 2023-10-30T13:23:27.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05dbfff640 1 -- 172.21.15.46:0/20613117 learned_addr learned my addr 172.21.15.46:0/20613117 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:27.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05e0ab9640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc1538a0 0x7f05dc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05e12ba640 1 --2- 172.21.15.46:0/20613117 >> v2:172.21.15.154:3300/0 conn(0x7f05dc0cf160 0x7f05dc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979415194 0 0) 0x7f05dc0cb800 con 0x7f05dc0ca0e0 2023-10-30T13:23:27.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05cc003530 con 0x7f05dc0ca0e0 2023-10-30T13:23:27.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 805399791 0 0) 0x7f05cc003530 con 0x7f05dc0ca0e0 2023-10-30T13:23:27.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc1538a0 msgr2=0x7f05dc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc1538a0 0x7f05dc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 >> v2:172.21.15.154:3300/0 conn(0x7f05dc0cf160 msgr2=0x7f05dc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 --2- 172.21.15.46:0/20613117 >> v2:172.21.15.154:3300/0 conn(0x7f05dc0cf160 0x7f05dc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05dc0cad70 con 0x7f05dc0ca0e0 2023-10-30T13:23:27.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f05c8001680 con 0x7f05dc0ca0e0 2023-10-30T13:23:27.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 >> v1:172.21.15.46:6789/0 conn(0x7f05dc0ca0e0 legacy=0x7f05dc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05cc003980 0x7f05cc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05cc0047d0 0x7f05cc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc1538a0 0x7f05cc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc15dc30 con 0x7f05cc0047d0 2023-10-30T13:23:27.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc0cb330 con 0x7f05cc003980 2023-10-30T13:23:27.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05cc00b990 con 0x7f05dc1538a0 2023-10-30T13:23:27.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05dbfff640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05cc0047d0 0x7f05cc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05e12ba640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc1538a0 0x7f05cc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.080+0000 7f05e0ab9640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05cc003980 0x7f05cc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05dbfff640 1 -- 172.21.15.46:0/20613117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc1538a0 msgr2=0x7f05cc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05dbfff640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc1538a0 0x7f05cc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05dbfff640 1 -- 172.21.15.46:0/20613117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05cc003980 msgr2=0x7f05cc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05dbfff640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05cc003980 0x7f05cc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05dbfff640 1 -- 172.21.15.46:0/20613117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc003530 con 0x7f05cc0047d0 2023-10-30T13:23:27.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05e0ab9640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05cc003980 0x7f05cc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:27.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05dbfff640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05cc0047d0 0x7f05cc004c50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f05c800c240 tx=0x7f05c801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5481504799772872 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c80058e0 con 0x7f05cc0047d0 2023-10-30T13:23:27.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c8005a40 con 0x7f05cc0047d0 2023-10-30T13:23:27.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05db7fe640 1 -- 172.21.15.46:0/20613117 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c8005e10 con 0x7f05cc0047d0 2023-10-30T13:23:27.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05e2d1d640 1 -- 172.21.15.46:0/20613117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05cc0047d0 msgr2=0x7f05cc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05e2d1d640 1 --2- 172.21.15.46:0/20613117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05cc0047d0 0x7f05cc004c50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f05c800c240 tx=0x7f05c801cc60 comp rx=0 tx=0).stop 2023-10-30T13:23:27.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05e12ba640 1 -- 172.21.15.46:0/20613117 reap_dead start 2023-10-30T13:23:27.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05e2d1d640 1 -- 172.21.15.46:0/20613117 shutdown_connections 2023-10-30T13:23:27.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05e2d1d640 1 -- 172.21.15.46:0/20613117 >> 172.21.15.46:0/20613117 conn(0x7f05dc06bfe0 msgr2=0x7f05dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:27.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05e2d1d640 1 -- 172.21.15.46:0/20613117 shutdown_connections 2023-10-30T13:23:27.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.084+0000 7f05e2d1d640 1 -- 172.21.15.46:0/20613117 wait complete. 2023-10-30T13:23:27.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e2d1d640 1 Processor -- start 2023-10-30T13:23:27.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e2d1d640 1 -- start start 2023-10-30T13:23:27.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e2d1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05cc0047d0 0x7f05dc1ef0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e2d1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc152900 0x7f05dc15ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e2d1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05dc15b1d0 0x7f05dc14c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e2d1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc15b680 con 0x7f05cc0047d0 2023-10-30T13:23:27.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e2d1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc15d7b0 con 0x7f05dc15b1d0 2023-10-30T13:23:27.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e2d1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc15c180 con 0x7f05dc152900 2023-10-30T13:23:27.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e0ab9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05cc0047d0 0x7f05dc1ef0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e0ab9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05cc0047d0 0x7f05dc1ef0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39424/0 (socket says 172.21.15.46:39424) 2023-10-30T13:23:27.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e0ab9640 1 -- 172.21.15.46:0/191898561 learned_addr learned my addr 172.21.15.46:0/191898561 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:27.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05dbfff640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc152900 0x7f05dc15ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e12ba640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05dc15b1d0 0x7f05dc14c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e0ab9640 1 -- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc152900 msgr2=0x7f05dc15ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e0ab9640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc152900 0x7f05dc15ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e0ab9640 1 -- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05dc15b1d0 msgr2=0x7f05dc14c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e0ab9640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05dc15b1d0 0x7f05dc14c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e0ab9640 1 -- 172.21.15.46:0/191898561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05dc15c460 con 0x7f05cc0047d0 2023-10-30T13:23:27.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.088+0000 7f05e0ab9640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05cc0047d0 0x7f05dc1ef0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009160 tx=0x7f05d0011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.092+0000 7f05d97fa640 1 -- 172.21.15.46:0/191898561 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0004220 con 0x7f05cc0047d0 2023-10-30T13:23:27.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.092+0000 7f05d97fa640 1 -- 172.21.15.46:0/191898561 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d0004380 con 0x7f05cc0047d0 2023-10-30T13:23:27.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.092+0000 7f05e2d1d640 1 -- 172.21.15.46:0/191898561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05dc15c740 con 0x7f05cc0047d0 2023-10-30T13:23:27.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.092+0000 7f05d97fa640 1 -- 172.21.15.46:0/191898561 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0015890 con 0x7f05cc0047d0 2023-10-30T13:23:27.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.092+0000 7f05e2d1d640 1 -- 172.21.15.46:0/191898561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05dc15cd00 con 0x7f05cc0047d0 2023-10-30T13:23:27.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.092+0000 7f05e2d1d640 1 -- 172.21.15.46:0/191898561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05dc04e170 con 0x7f05cc0047d0 2023-10-30T13:23:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.092+0000 7f05d97fa640 1 -- 172.21.15.46:0/191898561 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f05d00159f0 con 0x7f05cc0047d0 2023-10-30T13:23:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.092+0000 7f05d97fa640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f05c00664d0 0x7f05c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.092+0000 7f05d97fa640 1 -- 172.21.15.46:0/191898561 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05d00843b0 con 0x7f05cc0047d0 2023-10-30T13:23:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.092+0000 7f05dbfff640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f05c00664d0 0x7f05c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.096+0000 7f05dbfff640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f05c00664d0 0x7f05c0068990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f05c80aa000 tx=0x7f05c8001a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.096+0000 7f05d97fa640 1 -- 172.21.15.46:0/191898561 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f05d0051700 con 0x7f05cc0047d0 2023-10-30T13:23:27.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.256+0000 7f05e2d1d640 1 -- 172.21.15.46:0/191898561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05dc0b1dd0 con 0x7f05cc0047d0 2023-10-30T13:23:27.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.256+0000 7f05d97fa640 1 -- 172.21.15.46:0/191898561 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f05d0009280 con 0x7f05cc0047d0 2023-10-30T13:23:27.265 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:27.265 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:20.665723+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.260+0000 7f05e2d1d640 1 -- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f05c00664d0 msgr2=0x7f05c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.260+0000 7f05e2d1d640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f05c00664d0 0x7f05c0068990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f05c80aa000 tx=0x7f05c8001a80 comp rx=0 tx=0).stop 2023-10-30T13:23:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.260+0000 7f05e2d1d640 1 -- 172.21.15.46:0/191898561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05cc0047d0 msgr2=0x7f05dc1ef0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.260+0000 7f05e2d1d640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05cc0047d0 0x7f05dc1ef0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009160 tx=0x7f05d0011a60 comp rx=0 tx=0).stop 2023-10-30T13:23:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.260+0000 7f05e2d1d640 1 -- 172.21.15.46:0/191898561 shutdown_connections 2023-10-30T13:23:27.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.260+0000 7f05e2d1d640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f05c00664d0 0x7f05c0068990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.260+0000 7f05e2d1d640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05dc15b1d0 0x7f05dc14c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.260+0000 7f05e2d1d640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05dc152900 0x7f05dc15ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.260+0000 7f05e2d1d640 1 --2- 172.21.15.46:0/191898561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05cc0047d0 0x7f05dc1ef0c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.260+0000 7f05e2d1d640 1 -- 172.21.15.46:0/191898561 >> 172.21.15.46:0/191898561 conn(0x7f05dc06bfe0 msgr2=0x7f05dc0c4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:27.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.260+0000 7f05e2d1d640 1 -- 172.21.15.46:0/191898561 shutdown_connections 2023-10-30T13:23:27.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.260+0000 7f05e2d1d640 1 -- 172.21.15.46:0/191898561 wait complete. 2023-10-30T13:23:27.286 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-30T13:23:27.287 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:27.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd274794640 1 Processor -- start 2023-10-30T13:23:27.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd274794640 1 -- start start 2023-10-30T13:23:27.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd274794640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd26c1538a0 0x7fd26c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd274794640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd26c0cf160 0x7fd26c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd274794640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26c0cb800 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd274794640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd26c15dc30 con 0x7fd26c0cf160 2023-10-30T13:23:27.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd274794640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c0cb330 con 0x7fd26c1538a0 2023-10-30T13:23:27.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd271d2f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd26c0ca0e0 0x7fd26c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48448/0 (socket says 172.21.15.46:48448) 2023-10-30T13:23:27.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd271d2f640 1 -- 172.21.15.46:0/823619802 learned_addr learned my addr 172.21.15.46:0/823619802 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:27.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd272530640 1 --2- 172.21.15.46:0/823619802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd26c1538a0 0x7fd26c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd272d31640 1 --2- 172.21.15.46:0/823619802 >> v2:172.21.15.154:3300/0 conn(0x7fd26c0cf160 0x7fd26c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd27152e640 1 -- 172.21.15.46:0/823619802 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740271263 0 0) 0x7fd26c0cb800 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd27152e640 1 -- 172.21.15.46:0/823619802 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd258003530 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd272530640 1 -- 172.21.15.46:0/823619802 >> v2:172.21.15.154:3300/0 conn(0x7fd26c0cf160 msgr2=0x7fd26c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd272530640 1 --2- 172.21.15.46:0/823619802 >> v2:172.21.15.154:3300/0 conn(0x7fd26c0cf160 0x7fd26c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd27152e640 1 -- 172.21.15.46:0/823619802 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4163554474 0 0) 0x7fd258003530 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd272530640 1 -- 172.21.15.46:0/823619802 >> v1:172.21.15.46:6789/0 conn(0x7fd26c0ca0e0 legacy=0x7fd26c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.424+0000 7fd272530640 1 -- 172.21.15.46:0/823619802 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c0cad70 con 0x7fd26c1538a0 2023-10-30T13:23:27.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.428+0000 7fd272530640 1 --2- 172.21.15.46:0/823619802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd26c1538a0 0x7fd26c15d6f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd2680097c0 tx=0x7fd26801eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=245e7b12dc4ab373 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.428+0000 7fd27152e640 1 -- 172.21.15.46:0/823619802 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c001680 con 0x7fd26c1538a0 2023-10-30T13:23:27.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.428+0000 7fd27152e640 1 -- 172.21.15.46:0/823619802 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2680a85d0 con 0x7fd26c1538a0 2023-10-30T13:23:27.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.428+0000 7fd27152e640 1 -- 172.21.15.46:0/823619802 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2680a8940 con 0x7fd26c1538a0 2023-10-30T13:23:27.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.428+0000 7fd274794640 1 -- 172.21.15.46:0/823619802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd26c1538a0 msgr2=0x7fd26c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.428+0000 7fd274794640 1 --2- 172.21.15.46:0/823619802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd26c1538a0 0x7fd26c15d6f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd2680097c0 tx=0x7fd26801eb90 comp rx=0 tx=0).stop 2023-10-30T13:23:27.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.428+0000 7fd274794640 1 -- 172.21.15.46:0/823619802 shutdown_connections 2023-10-30T13:23:27.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.428+0000 7fd274794640 1 --2- 172.21.15.46:0/823619802 >> v2:172.21.15.154:3300/0 conn(0x7fd26c0cf160 0x7fd26c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.428+0000 7fd274794640 1 --2- 172.21.15.46:0/823619802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd26c1538a0 0x7fd26c15d6f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.428+0000 7fd274794640 1 -- 172.21.15.46:0/823619802 >> 172.21.15.46:0/823619802 conn(0x7fd26c06bfe0 msgr2=0x7fd26c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:27.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.428+0000 7fd274794640 1 -- 172.21.15.46:0/823619802 shutdown_connections 2023-10-30T13:23:27.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.428+0000 7fd274794640 1 -- 172.21.15.46:0/823619802 wait complete. 2023-10-30T13:23:27.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd274794640 1 Processor -- start 2023-10-30T13:23:27.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd274794640 1 -- start start 2023-10-30T13:23:27.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd274794640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd26c0ca0e0 0x7fd26c15b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd274794640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd26c0cf160 0x7fd26c1f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd274794640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd26c1538a0 0x7fd26c1e6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd274794640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c1f32e0 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd274794640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c15a8b0 con 0x7fd26c1538a0 2023-10-30T13:23:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd274794640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c15d580 con 0x7fd26c0cf160 2023-10-30T13:23:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd272530640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd26c0ca0e0 0x7fd26c15b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd272530640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd26c0ca0e0 0x7fd26c15b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39434/0 (socket says 172.21.15.46:39434) 2023-10-30T13:23:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd272530640 1 -- 172.21.15.46:0/2888813973 learned_addr learned my addr 172.21.15.46:0/2888813973 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd271d2f640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd26c0cf160 0x7fd26c1f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd272d31640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd26c1538a0 0x7fd26c1e6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd272530640 1 -- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd26c0cf160 msgr2=0x7fd26c1f2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd272530640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd26c0cf160 0x7fd26c1f2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd272530640 1 -- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd26c1538a0 msgr2=0x7fd26c1e6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd272530640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd26c1538a0 0x7fd26c1e6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd272530640 1 -- 172.21.15.46:0/2888813973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd268009420 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd271d2f640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd26c0cf160 0x7fd26c1f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd272530640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd26c0ca0e0 0x7fd26c15b5e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd26801f430 tx=0x7fd26801b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd262ffd640 1 -- 172.21.15.46:0/2888813973 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2680bf070 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd262ffd640 1 -- 172.21.15.46:0/2888813973 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2680a8950 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd262ffd640 1 -- 172.21.15.46:0/2888813973 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2680a8cc0 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd274794640 1 -- 172.21.15.46:0/2888813973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26c15d860 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.432+0000 7fd274794640 1 -- 172.21.15.46:0/2888813973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26c15ade0 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.436+0000 7fd274794640 1 -- 172.21.15.46:0/2888813973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26c0b2a50 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.436+0000 7fd262ffd640 1 -- 172.21.15.46:0/2888813973 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd268003620 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.436+0000 7fd262ffd640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd2500664d0 0x7fd250068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.436+0000 7fd262ffd640 1 -- 172.21.15.46:0/2888813973 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd268121340 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.436+0000 7fd271d2f640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd2500664d0 0x7fd250068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.436+0000 7fd271d2f640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd2500664d0 0x7fd250068990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd26c150d00 tx=0x7fd25c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.440+0000 7fd262ffd640 1 -- 172.21.15.46:0/2888813973 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd2680ee8f0 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.596+0000 7fd274794640 1 -- 172.21.15.46:0/2888813973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd26c04e170 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.596+0000 7fd262ffd640 1 -- 172.21.15.46:0/2888813973 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fd2680f1f10 con 0x7fd26c0ca0e0 2023-10-30T13:23:27.605 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:27.605 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:20.665723+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:27.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.600+0000 7fd274794640 1 -- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd2500664d0 msgr2=0x7fd250068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.600+0000 7fd274794640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd2500664d0 0x7fd250068990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd26c150d00 tx=0x7fd25c009040 comp rx=0 tx=0).stop 2023-10-30T13:23:27.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.600+0000 7fd274794640 1 -- 172.21.15.46:0/2888813973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd26c0ca0e0 msgr2=0x7fd26c15b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.600+0000 7fd274794640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd26c0ca0e0 0x7fd26c15b5e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd26801f430 tx=0x7fd26801b930 comp rx=0 tx=0).stop 2023-10-30T13:23:27.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.600+0000 7fd274794640 1 -- 172.21.15.46:0/2888813973 shutdown_connections 2023-10-30T13:23:27.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.600+0000 7fd274794640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd2500664d0 0x7fd250068990 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.600+0000 7fd274794640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd26c1538a0 0x7fd26c1e6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.600+0000 7fd274794640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd26c0cf160 0x7fd26c1f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.600+0000 7fd274794640 1 --2- 172.21.15.46:0/2888813973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd26c0ca0e0 0x7fd26c15b5e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.600+0000 7fd274794640 1 -- 172.21.15.46:0/2888813973 >> 172.21.15.46:0/2888813973 conn(0x7fd26c06bfe0 msgr2=0x7fd26c151420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.600+0000 7fd274794640 1 -- 172.21.15.46:0/2888813973 shutdown_connections 2023-10-30T13:23:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.600+0000 7fd274794640 1 -- 172.21.15.46:0/2888813973 wait complete. 2023-10-30T13:23:27.627 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-30T13:23:27.628 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-30T13:23:27.628 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-30T13:23:27.629 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-30T13:23:27.629 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-30T13:23:27.630 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-30T13:23:27.630 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-30T13:23:27.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-30T13:23:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.872+0000 7fbb1a8dc640 1 Processor -- start 2023-10-30T13:23:27.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.872+0000 7fbb1a8dc640 1 -- start start 2023-10-30T13:23:27.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.876+0000 7fbb1a8dc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb140c1430 0x7fbb14069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.876+0000 7fbb1a8dc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbb140c1810 0x7fbb1406f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.876+0000 7fbb198da640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb140c1430 0x7fbb14069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.876+0000 7fbb198da640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb140c1430 0x7fbb14069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39444/0 (socket says 172.21.15.46:39444) 2023-10-30T13:23:27.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.876+0000 7fbb198da640 1 -- 172.21.15.46:0/2844694449 learned_addr learned my addr 172.21.15.46:0/2844694449 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:27.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.876+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/2844694449 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb1406a3d0 con 0x7fbb140c80f0 2023-10-30T13:23:27.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.876+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/2844694449 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb14156760 con 0x7fbb140c1430 2023-10-30T13:23:27.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.876+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/2844694449 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb1406a660 con 0x7fbb140c1810 2023-10-30T13:23:27.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fbb190d9640 1 --2- 172.21.15.46:0/2844694449 >> v2:172.21.15.154:3300/0 conn(0x7fbb140c1810 0x7fbb1406f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fbb198da640 1 -- 172.21.15.46:0/2844694449 >> v2:172.21.15.154:3300/0 conn(0x7fbb140c1810 msgr2=0x7fbb1406f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fbb198da640 1 --2- 172.21.15.46:0/2844694449 >> v2:172.21.15.154:3300/0 conn(0x7fbb140c1810 0x7fbb1406f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fbb198da640 1 -- 172.21.15.46:0/2844694449 >> v1:172.21.15.154:6789/0 conn(0x7fbb140c80f0 legacy=0x7fbb140707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fbb198da640 1 -- 172.21.15.46:0/2844694449 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1406a940 con 0x7fbb140c1430 2023-10-30T13:23:27.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fbb190d9640 1 --2- 172.21.15.46:0/2844694449 >> v2:172.21.15.154:3300/0 conn(0x7fbb140c1810 0x7fbb1406f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:27.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fb612ed6640 1 Processor -- start 2023-10-30T13:23:27.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fb612ed6640 1 -- start start 2023-10-30T13:23:27.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fb612ed6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb60c157f70 0x7fb60c161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fb612ed6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb60c0cf160 0x7fb60c0cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fb612ed6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c1620d0 con 0x7fb60c0cad80 2023-10-30T13:23:27.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fb612ed6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c0ca8b0 con 0x7fb60c0cf160 2023-10-30T13:23:27.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.880+0000 7fb612ed6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c158350 con 0x7fb60c157f70 2023-10-30T13:23:27.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.884+0000 7fb610c72640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb60c157f70 0x7fb60c161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.884+0000 7fb610c72640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb60c157f70 0x7fb60c161b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59358/0 (socket says 172.21.15.46:59358) 2023-10-30T13:23:27.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.884+0000 7fb610c72640 1 -- 172.21.15.46:0/3564390612 learned_addr learned my addr 172.21.15.46:0/3564390612 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:27.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.884+0000 7fb611473640 1 --2- 172.21.15.46:0/3564390612 >> v2:172.21.15.154:3300/0 conn(0x7fb60c0cf160 0x7fb60c0cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.884+0000 7fb610c72640 1 -- 172.21.15.46:0/3564390612 >> v2:172.21.15.154:3300/0 conn(0x7fb60c0cf160 msgr2=0x7fb60c0cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.884+0000 7fb610c72640 1 --2- 172.21.15.46:0/3564390612 >> v2:172.21.15.154:3300/0 conn(0x7fb60c0cf160 0x7fb60c0cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.884+0000 7fb610c72640 1 -- 172.21.15.46:0/3564390612 >> v1:172.21.15.154:6789/0 conn(0x7fb60c0cad80 legacy=0x7fb60c0cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.884+0000 7fb610c72640 1 -- 172.21.15.46:0/3564390612 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c0ca2f0 con 0x7fb60c157f70 2023-10-30T13:23:27.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.884+0000 7fe3c4a6b640 1 Processor -- start 2023-10-30T13:23:27.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.884+0000 7fe3c4a6b640 1 -- start start 2023-10-30T13:23:27.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.884+0000 7fbb198da640 1 --2- 172.21.15.46:0/2844694449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb140c1430 0x7fbb14069e90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbb1406b380 tx=0x7fbb0801ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc1d9a6406a3464 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.884+0000 7fb9d413e640 1 Processor -- start 2023-10-30T13:23:27.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fe3c4a6b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe3c00c6f60 0x7fe3c013f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fe3c4a6b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3c00c08d0 0x7fe3c013fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fe3c4a6b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c0072030 con 0x7fe3c00c6b80 2023-10-30T13:23:27.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fe3c4a6b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c0070010 con 0x7fe3c00c08d0 2023-10-30T13:23:27.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fe3c4a6b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c0072260 con 0x7fe3c00c6f60 2023-10-30T13:23:27.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb9d413e640 1 -- start start 2023-10-30T13:23:27.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb610c72640 1 --2- 172.21.15.46:0/3564390612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb60c157f70 0x7fb60c161b90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc009e50 tx=0x7fb5fc01f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f78bdd3420a2983 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fbb188d8640 1 -- 172.21.15.46:0/2844694449 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb080a8c30 con 0x7fbb140c1430 2023-10-30T13:23:27.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fbb188d8640 1 -- 172.21.15.46:0/2844694449 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb080a8d90 con 0x7fbb140c1430 2023-10-30T13:23:27.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fbb188d8640 1 -- 172.21.15.46:0/2844694449 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08008c00 con 0x7fbb140c1430 2023-10-30T13:23:27.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fe3bed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3c00c08d0 0x7fe3c013fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fe3bed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3c00c08d0 0x7fe3c013fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39452/0 (socket says 172.21.15.46:39452) 2023-10-30T13:23:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb60bfff640 1 -- 172.21.15.46:0/3564390612 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0af400 con 0x7fb60c157f70 2023-10-30T13:23:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb60bfff640 1 -- 172.21.15.46:0/3564390612 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc0ab050 con 0x7fb60c157f70 2023-10-30T13:23:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb60bfff640 1 -- 172.21.15.46:0/3564390612 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0af780 con 0x7fb60c157f70 2023-10-30T13:23:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fe3bed9d640 1 -- 172.21.15.46:0/1081447603 learned_addr learned my addr 172.21.15.46:0/1081447603 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fe3bed9d640 1 -- 172.21.15.46:0/1081447603 >> v2:172.21.15.154:3300/0 conn(0x7fe3c00c6f60 msgr2=0x7fe3c013f600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T13:23:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fe3bed9d640 1 --2- 172.21.15.46:0/1081447603 >> v2:172.21.15.154:3300/0 conn(0x7fe3c00c6f60 0x7fe3c013f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fe3bed9d640 1 -- 172.21.15.46:0/1081447603 >> v1:172.21.15.46:6789/0 conn(0x7fe3c00c6b80 legacy=0x7fe3c0071920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:23:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/2844694449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb140c1430 msgr2=0x7fbb14069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fbb1a8dc640 1 --2- 172.21.15.46:0/2844694449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb140c1430 0x7fbb14069e90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbb1406b380 tx=0x7fbb0801ee20 comp rx=0 tx=0).stop 2023-10-30T13:23:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb9d413e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb9cc153c60 0x7fb9cc0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb9d413e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc0cb780 con 0x7fb9cc159830 2023-10-30T13:23:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb9d413e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc15a2e0 con 0x7fb9cc0c4430 2023-10-30T13:23:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fe3bed9d640 1 -- 172.21.15.46:0/1081447603 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c006f6e0 con 0x7fe3c00c08d0 2023-10-30T13:23:27.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb9d413e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9cc154040 con 0x7fb9cc153c60 2023-10-30T13:23:27.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb612ed6640 1 -- 172.21.15.46:0/3564390612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb60c157f70 msgr2=0x7fb60c161b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb9d26db640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb9cc159830 0x7fb9cc0cac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48468/0 (socket says 172.21.15.46:48468) 2023-10-30T13:23:27.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb9d26db640 1 -- 172.21.15.46:0/925298527 learned_addr learned my addr 172.21.15.46:0/925298527 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:27.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb612ed6640 1 --2- 172.21.15.46:0/3564390612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb60c157f70 0x7fb60c161b90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc009e50 tx=0x7fb5fc01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb612ed6640 1 -- 172.21.15.46:0/3564390612 shutdown_connections 2023-10-30T13:23:27.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb612ed6640 1 --2- 172.21.15.46:0/3564390612 >> v2:172.21.15.154:3300/0 conn(0x7fb60c0cf160 0x7fb60c0cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb612ed6640 1 --2- 172.21.15.46:0/3564390612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb60c157f70 0x7fb60c161b90 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc009e50 tx=0x7fb5fc01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.888+0000 7fb612ed6640 1 -- 172.21.15.46:0/3564390612 >> 172.21.15.46:0/3564390612 conn(0x7fb60c06bfe0 msgr2=0x7fb60c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261439896 0 0) 0x7fb9cc0cb780 con 0x7fb9cc159830 2023-10-30T13:23:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9bc003530 con 0x7fb9cc159830 2023-10-30T13:23:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590043350 0 0) 0x7fb9cc15a2e0 con 0x7fb9cc0c4430 2023-10-30T13:23:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc0cb780 con 0x7fb9cc0c4430 2023-10-30T13:23:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d1eda640 1 --2- 172.21.15.46:0/925298527 >> v2:172.21.15.154:3300/0 conn(0x7fb9cc153c60 0x7fb9cc0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3087636390 0 0) 0x7fb9bc003530 con 0x7fb9cc159830 2023-10-30T13:23:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb612ed6640 1 -- 172.21.15.46:0/3564390612 shutdown_connections 2023-10-30T13:23:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 >> v2:172.21.15.154:3300/0 conn(0x7fb9cc153c60 msgr2=0x7fb9cc0ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 --2- 172.21.15.46:0/925298527 >> v2:172.21.15.154:3300/0 conn(0x7fb9cc153c60 0x7fb9cc0ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 >> v1:172.21.15.154:6789/0 conn(0x7fb9cc0c4430 legacy=0x7fb9cc0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc15a5c0 con 0x7fb9cc159830 2023-10-30T13:23:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fb9c4001680 con 0x7fb9cc159830 2023-10-30T13:23:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 >> v1:172.21.15.46:6789/0 conn(0x7fb9cc159830 legacy=0x7fb9cc0cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 --2- 172.21.15.46:0/925298527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc004ba0 0x7fb9bc004f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 --2- 172.21.15.46:0/925298527 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9bc003770 0x7fb9bc003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 --2- 172.21.15.46:0/925298527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9bc004130 0x7fb9bc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b80010f0 con 0x7fb9bc004130 2023-10-30T13:23:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc154040 con 0x7fb9bc004ba0 2023-10-30T13:23:27.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc00c230 con 0x7fb9bc003770 2023-10-30T13:23:27.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d1eda640 1 --2- 172.21.15.46:0/925298527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc004ba0 0x7fb9bc004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d26db640 1 --2- 172.21.15.46:0/925298527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9bc004130 0x7fb9bc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d16d9640 1 --2- 172.21.15.46:0/925298527 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9bc003770 0x7fb9bc003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb612ed6640 1 -- 172.21.15.46:0/3564390612 wait complete. 2023-10-30T13:23:27.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d1eda640 1 -- 172.21.15.46:0/925298527 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9bc003770 msgr2=0x7fb9bc003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d1eda640 1 --2- 172.21.15.46:0/925298527 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9bc003770 0x7fb9bc003bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d1eda640 1 -- 172.21.15.46:0/925298527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9bc004130 msgr2=0x7fb9bc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d1eda640 1 --2- 172.21.15.46:0/925298527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9bc004130 0x7fb9bc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb9d1eda640 1 -- 172.21.15.46:0/925298527 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc003530 con 0x7fb9bc004ba0 2023-10-30T13:23:27.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fe3bed9d640 1 --2- 172.21.15.46:0/1081447603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c00c08d0 0x7fe3c013fb40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00b0850 tx=0x7fe3b001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6b79ed62768660 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb612ed6640 1 Processor -- start 2023-10-30T13:23:27.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fb612ed6640 1 -- start start 2023-10-30T13:23:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/2844694449 shutdown_connections 2023-10-30T13:23:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fbb1a8dc640 1 --2- 172.21.15.46:0/2844694449 >> v2:172.21.15.154:3300/0 conn(0x7fbb140c1810 0x7fbb1406f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fbb1a8dc640 1 --2- 172.21.15.46:0/2844694449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb140c1430 0x7fbb14069e90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/2844694449 >> 172.21.15.46:0/2844694449 conn(0x7fbb140c75d0 msgr2=0x7fbb140c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:27.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.892+0000 7fe3bd59a640 1 -- 172.21.15.46:0/1081447603 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b00aa610 con 0x7fe3c00c08d0 2023-10-30T13:23:27.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fe3bd59a640 1 -- 172.21.15.46:0/1081447603 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b00a6040 con 0x7fe3c00c08d0 2023-10-30T13:23:27.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fe3bd59a640 1 -- 172.21.15.46:0/1081447603 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b00aa990 con 0x7fe3c00c08d0 2023-10-30T13:23:27.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb612ed6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb60c0cad80 0x7fb60c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb612ed6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb60c0cf160 0x7fb60c140310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb9d26db640 1 -- 172.21.15.46:0/925298527 reap_dead start 2023-10-30T13:23:27.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb612ed6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb60c141fa0 0x7fb60c140850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb612ed6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c143890 con 0x7fb60c0cf160 2023-10-30T13:23:27.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb612ed6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c1372c0 con 0x7fb60c141fa0 2023-10-30T13:23:27.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb612ed6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0cdc30 con 0x7fb60c0cad80 2023-10-30T13:23:27.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb610c72640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb60c0cad80 0x7fb60c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb610c72640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb60c0cad80 0x7fb60c0ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59366/0 (socket says 172.21.15.46:59366) 2023-10-30T13:23:27.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb610c72640 1 -- 172.21.15.46:0/3837103134 learned_addr learned my addr 172.21.15.46:0/3837103134 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:27.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb611473640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb60c141fa0 0x7fb60c140850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/2844694449 shutdown_connections 2023-10-30T13:23:27.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/2844694449 wait complete. 2023-10-30T13:23:27.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7f7568254640 1 Processor -- start 2023-10-30T13:23:27.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fe3c4a6b640 1 -- 172.21.15.46:0/1081447603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c00c08d0 msgr2=0x7fe3c013fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fe3c4a6b640 1 --2- 172.21.15.46:0/1081447603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c00c08d0 0x7fe3c013fb40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00b0850 tx=0x7fe3b001c4b0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fe3c4a6b640 1 -- 172.21.15.46:0/1081447603 shutdown_connections 2023-10-30T13:23:27.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fe3c4a6b640 1 --2- 172.21.15.46:0/1081447603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c00c08d0 0x7fe3c013fb40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fe3c4a6b640 1 --2- 172.21.15.46:0/1081447603 >> v2:172.21.15.154:3300/0 conn(0x7fe3c00c6f60 0x7fe3c013f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fe3c4a6b640 1 -- 172.21.15.46:0/1081447603 >> 172.21.15.46:0/1081447603 conn(0x7fe3c00c7910 msgr2=0x7fe3c00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:27.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fe3c4a6b640 1 -- 172.21.15.46:0/1081447603 shutdown_connections 2023-10-30T13:23:27.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7f7568254640 1 -- start start 2023-10-30T13:23:27.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fe3c4a6b640 1 -- 172.21.15.46:0/1081447603 wait complete. 2023-10-30T13:23:27.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb611473640 1 -- 172.21.15.46:0/3837103134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb60c0cad80 msgr2=0x7fb60c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb611473640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb60c0cad80 0x7fb60c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb611473640 1 -- 172.21.15.46:0/3837103134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb60c0cf160 msgr2=0x7fb60c140310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb611473640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb60c0cf160 0x7fb60c140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb611473640 1 -- 172.21.15.46:0/3837103134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc009b30 con 0x7fb60c141fa0 2023-10-30T13:23:27.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fa36682f640 1 Processor -- start 2023-10-30T13:23:27.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb611473640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb60c141fa0 0x7fb60c140850 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb60400d4b0 tx=0x7fb60400d970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fa36682f640 1 -- start start 2023-10-30T13:23:27.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb609ffb640 1 -- 172.21.15.46:0/3837103134 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb604018070 con 0x7fb60c141fa0 2023-10-30T13:23:27.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb609ffb640 1 -- 172.21.15.46:0/3837103134 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb604004020 con 0x7fb60c141fa0 2023-10-30T13:23:27.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb609ffb640 1 -- 172.21.15.46:0/3837103134 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6040041b0 con 0x7fb60c141fa0 2023-10-30T13:23:27.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb612ed6640 1 -- 172.21.15.46:0/3837103134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c0cdf10 con 0x7fb60c141fa0 2023-10-30T13:23:27.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.896+0000 7fb612ed6640 1 -- 172.21.15.46:0/3837103134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c0ce4d0 con 0x7fb60c141fa0 2023-10-30T13:23:27.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fbb1a8dc640 1 Processor -- start 2023-10-30T13:23:27.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f8df6c46640 1 Processor -- start 2023-10-30T13:23:27.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7568254640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f756006a970 0x7f75600c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7568254640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f756006ade0 0x7f75600c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7568254640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756006a4e0 con 0x7f7560151910 2023-10-30T13:23:27.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7568254640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75600c6c50 con 0x7f756006a970 2023-10-30T13:23:27.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7568254640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75600c6f30 con 0x7f756006ade0 2023-10-30T13:23:27.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fbb1a8dc640 1 -- start start 2023-10-30T13:23:27.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fe3c4a6b640 1 Processor -- start 2023-10-30T13:23:27.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36682f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa360157c80 0x7fa360161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36682f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa36006fef0 0x7fa36006f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36682f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa360161f60 con 0x7fa36006ab20 2023-10-30T13:23:27.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36682f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa36015ef30 con 0x7fa360157c80 2023-10-30T13:23:27.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36682f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa360160e50 con 0x7fa36006fef0 2023-10-30T13:23:27.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fe3c4a6b640 1 -- start start 2023-10-30T13:23:27.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36502c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa36006fef0 0x7fa36006f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36502c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa36006fef0 0x7fa36006f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59388/0 (socket says 172.21.15.46:59388) 2023-10-30T13:23:27.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36502c640 1 -- 172.21.15.46:0/2178363553 learned_addr learned my addr 172.21.15.46:0/2178363553 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:27.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fb612ed6640 1 -- 172.21.15.46:0/3837103134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb5f0000fd0 con 0x7fb60c141fa0 2023-10-30T13:23:27.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f75657ef640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f756006a970 0x7f75600c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f75657ef640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f756006a970 0x7f75600c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39226/0 (socket says 172.21.15.46:39226) 2023-10-30T13:23:27.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f75657ef640 1 -- 172.21.15.46:0/51071157 learned_addr learned my addr 172.21.15.46:0/51071157 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:27.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f75667f1640 1 --2- 172.21.15.46:0/51071157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f756006ade0 0x7f75600c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f8df6c46640 1 -- start start 2023-10-30T13:23:27.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115967347 0 0) 0x7f756006a4e0 con 0x7f7560151910 2023-10-30T13:23:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f754c003530 con 0x7f7560151910 2023-10-30T13:23:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fb9d1eda640 1 --2- 172.21.15.46:0/925298527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc004ba0 0x7fb9bc004f80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0007360 tx=0x7fb9c001ce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=418b7fff54d3c68a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36582d640 1 --2- 172.21.15.46:0/2178363553 >> v2:172.21.15.154:3300/0 conn(0x7fa360157c80 0x7fa360161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c00a5d20 con 0x7fb9bc004ba0 2023-10-30T13:23:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c00a5e80 con 0x7fb9bc004ba0 2023-10-30T13:23:27.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1209358075 0 0) 0x7f754c003530 con 0x7f7560151910 2023-10-30T13:23:27.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fb9d0ed8640 1 -- 172.21.15.46:0/925298527 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c00ad660 con 0x7fb9bc004ba0 2023-10-30T13:23:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36502c640 1 -- 172.21.15.46:0/2178363553 >> v2:172.21.15.154:3300/0 conn(0x7fa360157c80 msgr2=0x7fa360161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36502c640 1 --2- 172.21.15.46:0/2178363553 >> v2:172.21.15.154:3300/0 conn(0x7fa360157c80 0x7fa360161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36502c640 1 -- 172.21.15.46:0/2178363553 >> v1:172.21.15.154:6789/0 conn(0x7fa36006ab20 legacy=0x7fa360070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fa36502c640 1 -- 172.21.15.46:0/2178363553 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3601616c0 con 0x7fa36006fef0 2023-10-30T13:23:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fe3c4a6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3c00c08d0 0x7fe3c01efea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f756006ade0 msgr2=0x7f75600c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7564fee640 1 --2- 172.21.15.46:0/51071157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f756006ade0 0x7f75600c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 >> v2:172.21.15.154:3300/0 conn(0x7f756006a970 msgr2=0x7f75600c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7564fee640 1 --2- 172.21.15.46:0/51071157 >> v2:172.21.15.154:3300/0 conn(0x7f756006a970 0x7f75600c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75600c7210 con 0x7f7560151910 2023-10-30T13:23:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f75580019e0 con 0x7f7560151910 2023-10-30T13:23:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 >> v1:172.21.15.154:6789/0 conn(0x7f7560151910 legacy=0x7f7560158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f7564fee640 1 --2- 172.21.15.46:0/51071157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f754c003980 0x7f754c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f8df49e2640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f8df0158090 0x7f8df0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:53106/0 (socket says 172.21.15.46:53106) 2023-10-30T13:23:27.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fbb1a8dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb140c1430 0x7fbb140c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fe3c4a6b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3c00c6b80 0x7fe3c0157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f8df6c46640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8df00c6b40 0x7f8df0161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f8df49e2640 1 -- 172.21.15.46:0/1271299933 learned_addr learned my addr 172.21.15.46:0/1271299933 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:27.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f8df6c46640 1 --2- 172.21.15.46:0/1271299933 >> v2:172.21.15.46:3300/0 conn(0x7f8df00c0ed0 0x7f8df006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb609ffb640 1 -- 172.21.15.46:0/3837103134 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb604004310 con 0x7fb60c141fa0 2023-10-30T13:23:27.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb198da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb140c1430 0x7fbb140c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb198da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb140c1430 0x7fbb140c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39480/0 (socket says 172.21.15.46:39480) 2023-10-30T13:23:27.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb198da640 1 -- 172.21.15.46:0/893893407 learned_addr learned my addr 172.21.15.46:0/893893407 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:27.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb1a8dc640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb140c1810 0x7fbb140c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb1a8dc640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb140c80f0 0x7fbb141ed820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb14071230 con 0x7fbb140c1430 2023-10-30T13:23:27.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb14071510 con 0x7fbb140c1810 2023-10-30T13:23:27.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1406f4c0 con 0x7fbb140c80f0 2023-10-30T13:23:27.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb190d9640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb140c1810 0x7fbb140c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb1a0db640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb140c80f0 0x7fbb141ed820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb9d413e640 1 -- 172.21.15.46:0/925298527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc004ba0 msgr2=0x7fb9bc004f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb9d413e640 1 --2- 172.21.15.46:0/925298527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc004ba0 0x7fb9bc004f80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0007360 tx=0x7fb9c001ce60 comp rx=0 tx=0).stop 2023-10-30T13:23:27.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f7565ff0640 1 --2- 172.21.15.46:0/51071157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f754c003980 0x7f754c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f7564fee640 1 --2- 172.21.15.46:0/51071157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f754c0047d0 0x7f754c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f7564fee640 1 --2- 172.21.15.46:0/51071157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c005190 0x7f754c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600c6c50 con 0x7f754c005190 2023-10-30T13:23:27.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600c6f30 con 0x7f754c003980 2023-10-30T13:23:27.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754c00bed0 con 0x7f754c0047d0 2023-10-30T13:23:27.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fe3c4a6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c00c6f60 0x7fe3c00c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3c4a6b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c01591b0 con 0x7fe3c00c6f60 2023-10-30T13:23:27.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3c4a6b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c0138f00 con 0x7fe3c00c6b80 2023-10-30T13:23:27.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3c4a6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c0158810 con 0x7fe3c00c08d0 2023-10-30T13:23:27.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7fe3be59c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3c00c08d0 0x7fe3c01efea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3be59c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3c00c08d0 0x7fe3c01efea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59402/0 (socket says 172.21.15.46:59402) 2023-10-30T13:23:27.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3be59c640 1 -- 172.21.15.46:0/3701906984 learned_addr learned my addr 172.21.15.46:0/3701906984 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:27.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3bed9d640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c00c6f60 0x7fe3c00c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3bdd9b640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3c00c6b80 0x7fe3c0157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.900+0000 7f8df6c46640 1 -- 172.21.15.46:0/1271299933 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df00c72b0 con 0x7f8df0158090 2023-10-30T13:23:27.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8df6c46640 1 -- 172.21.15.46:0/1271299933 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df0162120 con 0x7f8df00c0ed0 2023-10-30T13:23:27.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8deffff640 1 --2- 172.21.15.46:0/1271299933 >> v2:172.21.15.154:3300/0 conn(0x7f8df00c6b40 0x7f8df0161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258305476 0 0) 0x7f8df00c72b0 con 0x7f8df0158090 2023-10-30T13:23:27.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8df6c46640 1 -- 172.21.15.46:0/1271299933 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df00c6fd0 con 0x7f8df00c6b40 2023-10-30T13:23:27.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8df51e3640 1 --2- 172.21.15.46:0/1271299933 >> v2:172.21.15.46:3300/0 conn(0x7f8df00c0ed0 0x7f8df006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd8003530 con 0x7f8df0158090 2023-10-30T13:23:27.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb609ffb640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5ec0665a0 0x7fb5ec068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb190d9640 1 -- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb140c80f0 msgr2=0x7fbb141ed820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb190d9640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb140c80f0 0x7fbb141ed820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb190d9640 1 -- 172.21.15.46:0/893893407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb140c1430 msgr2=0x7fbb140c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb190d9640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb140c1430 0x7fbb140c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb190d9640 1 -- 172.21.15.46:0/893893407 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb08008050 con 0x7fbb140c1810 2023-10-30T13:23:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f75667f1640 1 --2- 172.21.15.46:0/51071157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c005190 0x7f754c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f75657ef640 1 --2- 172.21.15.46:0/51071157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f754c0047d0 0x7f754c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb609ffb640 1 -- 172.21.15.46:0/3837103134 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6040845f0 con 0x7fb60c141fa0 2023-10-30T13:23:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb190d9640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb140c1810 0x7fbb140c5e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000acf0 tx=0x7fbb10009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb9d413e640 1 -- 172.21.15.46:0/925298527 shutdown_connections 2023-10-30T13:23:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb9d413e640 1 --2- 172.21.15.46:0/925298527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc004ba0 0x7fb9bc004f80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb9d413e640 1 -- 172.21.15.46:0/925298527 >> 172.21.15.46:0/925298527 conn(0x7fb9cc06bfe0 msgr2=0x7fb9cc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:27.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2479205056 0 0) 0x7f8dd8003530 con 0x7f8df0158090 2023-10-30T13:23:27.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb609ffb640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] conn(0x7fb5ec06e7c0 0x7fb5ec070be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb609ffb640 1 -- 172.21.15.46:0/3837103134 --> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5ec071290 con 0x7fb5ec06e7c0 2023-10-30T13:23:27.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb609ffb640 1 -- 172.21.15.46:0/3837103134 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6040849a0 con 0x7fb60c141fa0 2023-10-30T13:23:27.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb610c72640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5ec0665a0 0x7fb5ec068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f7565ff0640 1 -- 172.21.15.46:0/51071157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f754c0047d0 msgr2=0x7f754c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f7565ff0640 1 --2- 172.21.15.46:0/51071157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f754c0047d0 0x7f754c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f7565ff0640 1 -- 172.21.15.46:0/51071157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c005190 msgr2=0x7f754c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f7565ff0640 1 --2- 172.21.15.46:0/51071157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c005190 0x7f754c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f7565ff0640 1 -- 172.21.15.46:0/51071157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75600c7210 con 0x7f754c003980 2023-10-30T13:23:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 >> v2:172.21.15.154:3300/0 conn(0x7f8df00c6b40 msgr2=0x7f8df0161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 --2- 172.21.15.46:0/1271299933 >> v2:172.21.15.154:3300/0 conn(0x7f8df00c6b40 0x7f8df0161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 >> v2:172.21.15.46:3300/0 conn(0x7f8df00c0ed0 msgr2=0x7f8df006a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 --2- 172.21.15.46:0/1271299933 >> v2:172.21.15.46:3300/0 conn(0x7f8df00c0ed0 0x7f8df006a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df015dd30 con 0x7f8df0158090 2023-10-30T13:23:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3bdd9b640 1 -- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3c00c08d0 msgr2=0x7fe3c01efea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3bdd9b640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3c00c08d0 0x7fe3c01efea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3bdd9b640 1 -- 172.21.15.46:0/3701906984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c00c6f60 msgr2=0x7fe3c00c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3bdd9b640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c00c6f60 0x7fe3c00c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3bdd9b640 1 -- 172.21.15.46:0/3701906984 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b00094d0 con 0x7fe3c00c6b80 2023-10-30T13:23:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3be59c640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3c00c08d0 0x7fe3c01efea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3bed9d640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c00c6f60 0x7fe3c00c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:27.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb9d413e640 1 -- 172.21.15.46:0/925298527 shutdown_connections 2023-10-30T13:23:27.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f8de4001680 con 0x7f8df0158090 2023-10-30T13:23:27.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb9d413e640 1 -- 172.21.15.46:0/925298527 wait complete. 2023-10-30T13:23:27.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb603fff640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] conn(0x7fb5ec06e7c0 0x7fb5ec070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fb610c72640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5ec0665a0 0x7fb5ec068a60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc01f230 tx=0x7fb5fc0aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb067fc640 1 -- 172.21.15.46:0/893893407 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb10009e70 con 0x7fbb140c1810 2023-10-30T13:23:27.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb067fc640 1 -- 172.21.15.46:0/893893407 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb10013a20 con 0x7fbb140c1810 2023-10-30T13:23:27.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb067fc640 1 -- 172.21.15.46:0/893893407 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb10013dd0 con 0x7fbb140c1810 2023-10-30T13:23:27.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1406f7a0 con 0x7fbb140c1810 2023-10-30T13:23:27.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb141ea2b0 con 0x7fbb140c1810 2023-10-30T13:23:27.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 >> v1:172.21.15.154:6789/0 conn(0x7f8df0158090 legacy=0x7f8df0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 --2- 172.21.15.46:0/1271299933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd8003980 0x7f8dd8003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 --2- 172.21.15.46:0/1271299933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd8004380 0x7f8dd8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7f8def7fe640 1 --2- 172.21.15.46:0/1271299933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd8004f70 0x7f8dd800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df0162120 con 0x7f8dd8004f70 2023-10-30T13:23:27.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df00c6fd0 con 0x7f8dd8003980 2023-10-30T13:23:27.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd800bf20 con 0x7f8dd8004380 2023-10-30T13:23:27.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.904+0000 7fe3bdd9b640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3c00c6b80 0x7fe3c0157960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe3b400b3e0 tx=0x7fe3b400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fb603fff640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] conn(0x7fb5ec06e7c0 0x7fb5ec070be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fe3aeffd640 1 -- 172.21.15.46:0/3701906984 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4011830 con 0x7fe3c00c6b80 2023-10-30T13:23:27.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fe3aeffd640 1 -- 172.21.15.46:0/3701906984 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b4011990 con 0x7fe3c00c6b80 2023-10-30T13:23:27.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fe3aeffd640 1 -- 172.21.15.46:0/3701906984 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4011d60 con 0x7fe3c00c6b80 2023-10-30T13:23:27.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fe3c4a6b640 1 -- 172.21.15.46:0/3701906984 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c0158af0 con 0x7fe3c00c6b80 2023-10-30T13:23:27.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fe3c4a6b640 1 -- 172.21.15.46:0/3701906984 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c0157ea0 con 0x7fe3c00c6b80 2023-10-30T13:23:27.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7f8df49e2640 1 --2- 172.21.15.46:0/1271299933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd8003980 0x7f8dd8003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7f8deffff640 1 --2- 172.21.15.46:0/1271299933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd8004380 0x7f8dd8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fa36502c640 1 --2- 172.21.15.46:0/2178363553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa36006fef0 0x7fa36006f340 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa350009870 tx=0x7fa35001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24b32f5ccd08af2e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fb9d413e640 1 Processor -- start 2023-10-30T13:23:27.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7f8df51e3640 1 --2- 172.21.15.46:0/1271299933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd8004f70 0x7f8dd800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbb1406f340 con 0x7fbb140c1810 2023-10-30T13:23:27.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fa36482b640 1 -- 172.21.15.46:0/2178363553 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3500a8a30 con 0x7fa36006fef0 2023-10-30T13:23:27.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fa36482b640 1 -- 172.21.15.46:0/2178363553 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3500a8b90 con 0x7fa36006fef0 2023-10-30T13:23:27.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7f8df49e2640 1 -- 172.21.15.46:0/1271299933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd8004380 msgr2=0x7f8dd8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7f8df49e2640 1 --2- 172.21.15.46:0/1271299933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd8004380 0x7f8dd8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7f8df49e2640 1 -- 172.21.15.46:0/1271299933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd8004f70 msgr2=0x7f8dd800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fe3c4a6b640 1 -- 172.21.15.46:0/3701906984 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe3c00c8710 con 0x7fe3c00c6b80 2023-10-30T13:23:27.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fb9d413e640 1 -- start start 2023-10-30T13:23:27.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fb609ffb640 1 -- 172.21.15.46:0/3837103134 <== osd.0 v2:172.21.15.46:6809/1936524547 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb5ec071290 con 0x7fb5ec06e7c0 2023-10-30T13:23:27.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fa36682f640 1 -- 172.21.15.46:0/2178363553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa36006fef0 msgr2=0x7fa36006f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fa36682f640 1 --2- 172.21.15.46:0/2178363553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa36006fef0 0x7fa36006f340 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa350009870 tx=0x7fa35001ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:27.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fa36682f640 1 -- 172.21.15.46:0/2178363553 shutdown_connections 2023-10-30T13:23:27.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fa36682f640 1 --2- 172.21.15.46:0/2178363553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa36006fef0 0x7fa36006f340 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fa36682f640 1 --2- 172.21.15.46:0/2178363553 >> v2:172.21.15.154:3300/0 conn(0x7fa360157c80 0x7fa360161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fa36682f640 1 -- 172.21.15.46:0/2178363553 >> 172.21.15.46:0/2178363553 conn(0x7fa3600c75d0 msgr2=0x7fa3600c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:27.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fb9d413e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9bc004ba0 0x7fb9cc156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fb9d413e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9cc0c5a40 0x7fb9cc0ce400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7fb9d413e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc0ce940 0x7fb9cc148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d413e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc0b4210 con 0x7fb9bc004ba0 2023-10-30T13:23:27.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d413e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc0c6680 con 0x7fb9cc0c5a40 2023-10-30T13:23:27.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d413e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc0b5080 con 0x7fb9cc0ce940 2023-10-30T13:23:27.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fa36682f640 1 -- 172.21.15.46:0/2178363553 shutdown_connections 2023-10-30T13:23:27.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fa36682f640 1 -- 172.21.15.46:0/2178363553 wait complete. 2023-10-30T13:23:27.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7f8df49e2640 1 --2- 172.21.15.46:0/1271299933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd8004f70 0x7f8dd800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7f8df49e2640 1 -- 172.21.15.46:0/1271299933 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd8003530 con 0x7f8dd8003980 2023-10-30T13:23:27.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7f8df51e3640 1 -- 172.21.15.46:0/1271299933 reap_dead start 2023-10-30T13:23:27.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d1eda640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9bc004ba0 0x7fb9cc156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d1eda640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9bc004ba0 0x7fb9cc156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39532/0 (socket says 172.21.15.46:39532) 2023-10-30T13:23:27.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d1eda640 1 -- 172.21.15.46:0/1857185378 learned_addr learned my addr 172.21.15.46:0/1857185378 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:27.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d16d9640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9cc0c5a40 0x7fb9cc0ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d26db640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc0ce940 0x7fb9cc148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.908+0000 7f7565ff0640 1 --2- 172.21.15.46:0/51071157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f754c003980 0x7f754c003dc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f755800c4e0 tx=0x7f755801cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0faa27fe99e4252 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75580a5ba0 con 0x7f754c003980 2023-10-30T13:23:27.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75580a5d20 con 0x7f754c003980 2023-10-30T13:23:27.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fe3aeffd640 1 -- 172.21.15.46:0/3701906984 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe3b401a400 con 0x7fe3c00c6b80 2023-10-30T13:23:27.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7f7564fee640 1 -- 172.21.15.46:0/51071157 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558003bc0 con 0x7f754c003980 2023-10-30T13:23:27.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fbb067fc640 1 -- 172.21.15.46:0/893893407 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb10004030 con 0x7fbb140c1810 2023-10-30T13:23:27.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7f7568254640 1 -- 172.21.15.46:0/51071157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f754c003980 msgr2=0x7f754c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d1eda640 1 -- 172.21.15.46:0/1857185378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc0ce940 msgr2=0x7fb9cc148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d1eda640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc0ce940 0x7fb9cc148640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d1eda640 1 -- 172.21.15.46:0/1857185378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9cc0c5a40 msgr2=0x7fb9cc0ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d1eda640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9cc0c5a40 0x7fb9cc0ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d1eda640 1 -- 172.21.15.46:0/1857185378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc0b5360 con 0x7fb9bc004ba0 2023-10-30T13:23:27.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7f8df49e2640 1 --2- 172.21.15.46:0/1271299933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd8003980 0x7f8dd8003d60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8de4005040 tx=0x7f8de401d2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b3f88c003587729 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fa36682f640 1 Processor -- start 2023-10-30T13:23:27.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7f7568254640 1 --2- 172.21.15.46:0/51071157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f754c003980 0x7f754c003dc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f755800c4e0 tx=0x7f755801cea0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9d1eda640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9bc004ba0 0x7fb9cc156c40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00b0040 tx=0x7fb9c0019c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fa36682f640 1 -- start start 2023-10-30T13:23:27.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7f75667f1640 1 -- 172.21.15.46:0/51071157 reap_dead start 2023-10-30T13:23:27.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fbb067fc640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbaf8066570 0x7fbaf8068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9ca7fc640 1 -- 172.21.15.46:0/1857185378 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c00adac0 con 0x7fb9bc004ba0 2023-10-30T13:23:27.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fbb067fc640 1 -- 172.21.15.46:0/893893407 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb1000ae40 con 0x7fbb140c1810 2023-10-30T13:23:27.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9ca7fc640 1 -- 172.21.15.46:0/1857185378 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c00adc40 con 0x7fb9bc004ba0 2023-10-30T13:23:27.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fe3aeffd640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3a006a8b0 0x7fe3a006cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fe3aeffd640 1 -- 172.21.15.46:0/3701906984 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3b4084510 con 0x7fe3c00c6b80 2023-10-30T13:23:27.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fe3aeffd640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] conn(0x7fe3a0072ad0 0x7fe3a0074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fe3aeffd640 1 -- 172.21.15.46:0/3701906984 --> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3a00755a0 con 0x7fe3a0072ad0 2023-10-30T13:23:27.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fe3aeffd640 1 -- 172.21.15.46:0/3701906984 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3b40848c0 con 0x7fe3c00c6b80 2023-10-30T13:23:27.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fbb198da640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbaf8066570 0x7fbaf8068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7fb9ca7fc640 1 -- 172.21.15.46:0/1857185378 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c00039a0 con 0x7fb9bc004ba0 2023-10-30T13:23:27.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fb9d413e640 1 -- 172.21.15.46:0/1857185378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9cc0b3bc0 con 0x7fb9bc004ba0 2023-10-30T13:23:27.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36682f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa36006ab20 0x7fa3601ff0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36682f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa36006fef0 0x7fa3601583d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36682f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa360157c80 0x7fa3600c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36682f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa360159c70 con 0x7fa36006ab20 2023-10-30T13:23:27.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36682f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3601559a0 con 0x7fa36006fef0 2023-10-30T13:23:27.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36682f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa360155120 con 0x7fa360157c80 2023-10-30T13:23:27.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f7568254640 1 -- 172.21.15.46:0/51071157 shutdown_connections 2023-10-30T13:23:27.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f7568254640 1 -- 172.21.15.46:0/51071157 >> 172.21.15.46:0/51071157 conn(0x7f75600c7910 msgr2=0x7f75600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:27.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fe3be59c640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3a006a8b0 0x7fe3a006cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7f0e3849c640 1 Processor -- start 2023-10-30T13:23:27.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36502c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa36006fef0 0x7fa3601583d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7f0e3849c640 1 -- start start 2023-10-30T13:23:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36502c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa36006fef0 0x7fa3601583d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39310/0 (socket says 172.21.15.46:39310) 2023-10-30T13:23:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36502c640 1 -- 172.21.15.46:0/2267026276 learned_addr learned my addr 172.21.15.46:0/2267026276 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36602e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa360157c80 0x7fa3600c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fbb067fc640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] conn(0x7fbaf806e790 0x7fbaf8070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fbb067fc640 1 -- 172.21.15.46:0/893893407 --> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbaf8071260 con 0x7fbaf806e790 2023-10-30T13:23:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fbb067fc640 1 -- 172.21.15.46:0/893893407 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb1008de10 con 0x7fbb140c1810 2023-10-30T13:23:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.912+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4005ba0 con 0x7f8dd8003980 2023-10-30T13:23:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de40a8070 con 0x7f8dd8003980 2023-10-30T13:23:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f8def7fe640 1 -- 172.21.15.46:0/1271299933 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4008110 con 0x7f8dd8003980 2023-10-30T13:23:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f8df6c46640 1 -- 172.21.15.46:0/1271299933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd8003980 msgr2=0x7f8dd8003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f8df6c46640 1 --2- 172.21.15.46:0/1271299933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd8003980 0x7f8dd8003d60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8de4005040 tx=0x7f8de401d2a0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f8df6c46640 1 -- 172.21.15.46:0/1271299933 shutdown_connections 2023-10-30T13:23:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f8df6c46640 1 --2- 172.21.15.46:0/1271299933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd8003980 0x7f8dd8003d60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f8df6c46640 1 -- 172.21.15.46:0/1271299933 >> 172.21.15.46:0/1271299933 conn(0x7f8df00c7910 msgr2=0x7f8df00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fe3bed9d640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] conn(0x7fe3a0072ad0 0x7fe3a0074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fbb198da640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbaf8066570 0x7fbaf8068a30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbb08000c10 tx=0x7fbb080aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f7568254640 1 -- 172.21.15.46:0/51071157 shutdown_connections 2023-10-30T13:23:28.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fb9d413e640 1 -- 172.21.15.46:0/1857185378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9cc0b4470 con 0x7fb9bc004ba0 2023-10-30T13:23:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e3849c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e300c6b40 0x7f0e30161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e3849c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e300c0ed0 0x7f0e3006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e3849c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e300c72b0 con 0x7f0e30158090 2023-10-30T13:23:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e3849c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e30162120 con 0x7f0e300c6b40 2023-10-30T13:23:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e3849c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e300c6fd0 con 0x7f0e300c0ed0 2023-10-30T13:23:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f8df6c46640 1 -- 172.21.15.46:0/1271299933 shutdown_connections 2023-10-30T13:23:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f8df6c46640 1 -- 172.21.15.46:0/1271299933 wait complete. 2023-10-30T13:23:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fbb1a0db640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] conn(0x7fbaf806e790 0x7fbaf8070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fb9d413e640 1 -- 172.21.15.46:0/1857185378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb9cc0c9820 con 0x7fb9bc004ba0 2023-10-30T13:23:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36502c640 1 -- 172.21.15.46:0/2267026276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa360157c80 msgr2=0x7fa3600c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36502c640 1 --2- 172.21.15.46:0/2267026276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa360157c80 0x7fa3600c8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36502c640 1 -- 172.21.15.46:0/2267026276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa36006ab20 msgr2=0x7fa3601ff0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36502c640 1 --2- 172.21.15.46:0/2267026276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa36006ab20 0x7fa3601ff0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36502c640 1 -- 172.21.15.46:0/2267026276 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3500094d0 con 0x7fa36006fef0 2023-10-30T13:23:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36502c640 1 --2- 172.21.15.46:0/2267026276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa36006fef0 0x7fa3601583d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa35001c960 tx=0x7fa35001bad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f7568254640 1 -- 172.21.15.46:0/51071157 wait complete. 2023-10-30T13:23:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e36238640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f0e30158090 0x7f0e30069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:53112/0 (socket says 172.21.15.46:53112) 2023-10-30T13:23:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e36238640 1 -- 172.21.15.46:0/1163344022 learned_addr learned my addr 172.21.15.46:0/1163344022 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e36a39640 1 --2- 172.21.15.46:0/1163344022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e300c0ed0 0x7f0e3006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e35a37640 1 --2- 172.21.15.46:0/1163344022 >> v2:172.21.15.46:3300/0 conn(0x7f0e300c6b40 0x7f0e30161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fe3be59c640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3a006a8b0 0x7fe3a006cd70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800c260 tx=0x7fe3b800c4f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fe3bed9d640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] conn(0x7fe3a0072ad0 0x7fe3a0074ef0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa3567fc640 1 -- 172.21.15.46:0/2267026276 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3500bd070 con 0x7fa36006fef0 2023-10-30T13:23:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa3567fc640 1 -- 172.21.15.46:0/2267026276 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa350003830 con 0x7fa36006fef0 2023-10-30T13:23:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa3567fc640 1 -- 172.21.15.46:0/2267026276 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa350003bc0 con 0x7fa36006fef0 2023-10-30T13:23:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e35236640 1 -- 172.21.15.46:0/1163344022 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946251488 0 0) 0x7f0e300c72b0 con 0x7f0e30158090 2023-10-30T13:23:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e35236640 1 -- 172.21.15.46:0/1163344022 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e1c003530 con 0x7f0e30158090 2023-10-30T13:23:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36682f640 1 -- 172.21.15.46:0/2267026276 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3601553e0 con 0x7fa36006fef0 2023-10-30T13:23:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7fa36682f640 1 -- 172.21.15.46:0/2267026276 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3601fe8f0 con 0x7fa36006fef0 2023-10-30T13:23:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7fbb1a0db640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] conn(0x7fbaf806e790 0x7fbaf8070bb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8df6c46640 1 Processor -- start 2023-10-30T13:23:28.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e35a37640 1 -- 172.21.15.46:0/1163344022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e300c0ed0 msgr2=0x7f0e3006a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:28.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e35a37640 1 --2- 172.21.15.46:0/1163344022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e300c0ed0 0x7f0e3006a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e35a37640 1 -- 172.21.15.46:0/1163344022 >> v1:172.21.15.154:6789/0 conn(0x7f0e30158090 legacy=0x7f0e30069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.916+0000 7f0e35a37640 1 -- 172.21.15.46:0/1163344022 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3015dd30 con 0x7f0e300c6b40 2023-10-30T13:23:28.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f7568254640 1 Processor -- start 2023-10-30T13:23:28.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7fa36682f640 1 -- 172.21.15.46:0/2267026276 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa360161c80 con 0x7fa36006fef0 2023-10-30T13:23:28.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7fe3aeffd640 1 -- 172.21.15.46:0/3701906984 <== osd.5 v2:172.21.15.154:6801/956300018 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fe3a00755a0 con 0x7fe3a0072ad0 2023-10-30T13:23:28.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8df6c46640 1 -- start start 2023-10-30T13:23:28.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f7568254640 1 -- start start 2023-10-30T13:23:28.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7fbb067fc640 1 -- 172.21.15.46:0/893893407 <== osd.1 v2:172.21.15.46:6800/1107843451 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fbaf8071260 con 0x7fbaf806e790 2023-10-30T13:23:28.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8df6c46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd8003980 0x7f8df015ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8df6c46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df01597a0 0x7f8df014c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8df6c46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8df015cd40 0x7f8df015d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8df6c46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df0159240 con 0x7f8df01597a0 2023-10-30T13:23:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8df6c46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df0161b80 con 0x7f8dd8003980 2023-10-30T13:23:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8df6c46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df0159c70 con 0x7f8df015cd40 2023-10-30T13:23:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8df49e2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd8003980 0x7f8df015ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8df51e3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8df015cd40 0x7f8df015d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f7568254640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f754c003980 0x7f7560151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8df51e3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8df015cd40 0x7f8df015d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59454/0 (socket says 172.21.15.46:59454) 2023-10-30T13:23:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fa3567fc640 1 -- 172.21.15.46:0/2267026276 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3500b7070 con 0x7fa36006fef0 2023-10-30T13:23:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f7568254640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75600c0e20 0x7f7560070920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8df51e3640 1 -- 172.21.15.46:0/2511118376 learned_addr learned my addr 172.21.15.46:0/2511118376 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.920+0000 7f8deffff640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df01597a0 0x7f8df014c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f7568254640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75600c2f70 0x7f75601fe5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8df51e3640 1 -- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd8003980 msgr2=0x7f8df015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8df51e3640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd8003980 0x7f8df015ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8df51e3640 1 -- 172.21.15.46:0/2511118376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df01597a0 msgr2=0x7f8df014c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8df51e3640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df01597a0 0x7f8df014c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8df51e3640 1 -- 172.21.15.46:0/2511118376 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df0158630 con 0x7f8df015cd40 2023-10-30T13:23:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8deffff640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df01597a0 0x7f8df014c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fb9ca7fc640 1 -- 172.21.15.46:0/1857185378 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9c00b4070 con 0x7fb9bc004ba0 2023-10-30T13:23:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fa3567fc640 1 --2- 172.21.15.46:0/2267026276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa344066570 0x7fa344068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fa3567fc640 1 -- 172.21.15.46:0/2267026276 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa350121b30 con 0x7fa36006fef0 2023-10-30T13:23:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fa3567fc640 1 --2- 172.21.15.46:0/2267026276 >> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] conn(0x7fa34406e790 0x7fa344070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fa3567fc640 1 -- 172.21.15.46:0/2267026276 --> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa344071260 con 0x7fa34406e790 2023-10-30T13:23:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fa3567fc640 1 -- 172.21.15.46:0/2267026276 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa350121ee0 con 0x7fa36006fef0 2023-10-30T13:23:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f75657ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75600c0e20 0x7f7560070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f75657ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75600c0e20 0x7f7560070920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39582/0 (socket says 172.21.15.46:39582) 2023-10-30T13:23:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f7565ff0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f754c003980 0x7f7560151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f7565ff0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f754c003980 0x7f7560151fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59464/0 (socket says 172.21.15.46:59464) 2023-10-30T13:23:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f7565ff0640 1 -- 172.21.15.46:0/990813772 learned_addr learned my addr 172.21.15.46:0/990813772 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f75667f1640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75600c2f70 0x7f75601fe5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f0e35a37640 1 --2- 172.21.15.46:0/1163344022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300c6b40 0x7f0e30161be0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0e200097c0 tx=0x7f0e2001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8744b4ef4274f061 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f0e35236640 1 -- 172.21.15.46:0/1163344022 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e200a8970 con 0x7f0e300c6b40 2023-10-30T13:23:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8df51e3640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8df015cd40 0x7f8df015d1c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8df00c8630 tx=0x7f8de000f8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8ded7fa640 1 -- 172.21.15.46:0/2511118376 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0014870 con 0x7f8df015cd40 2023-10-30T13:23:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8ded7fa640 1 -- 172.21.15.46:0/2511118376 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de00149d0 con 0x7f8df015cd40 2023-10-30T13:23:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8ded7fa640 1 -- 172.21.15.46:0/2511118376 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0014d80 con 0x7f8df015cd40 2023-10-30T13:23:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8df6c46640 1 -- 172.21.15.46:0/2511118376 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df0158910 con 0x7f8df015cd40 2023-10-30T13:23:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8df6c46640 1 -- 172.21.15.46:0/2511118376 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df0158ed0 con 0x7f8df015cd40 2023-10-30T13:23:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fa36582d640 1 --2- 172.21.15.46:0/2267026276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa344066570 0x7fa344068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f0e35236640 1 -- 172.21.15.46:0/1163344022 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e200a8ad0 con 0x7f0e300c6b40 2023-10-30T13:23:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f0e35236640 1 -- 172.21.15.46:0/1163344022 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e200a8e60 con 0x7f0e300c6b40 2023-10-30T13:23:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f8df6c46640 1 -- 172.21.15.46:0/2511118376 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8df015e2f0 con 0x7f8df015cd40 2023-10-30T13:23:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fb9ca7fc640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9b00664a0 0x7fb9b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fb9ca7fc640 1 -- 172.21.15.46:0/1857185378 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9c011f410 con 0x7fb9bc004ba0 2023-10-30T13:23:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fb9ca7fc640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] conn(0x7fb9b006e6c0 0x7fb9b0070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fb9ca7fc640 1 -- 172.21.15.46:0/1857185378 --> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9b0071190 con 0x7fb9b006e6c0 2023-10-30T13:23:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7fb9ca7fc640 1 -- 172.21.15.46:0/1857185378 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9c011f7c0 con 0x7fb9bc004ba0 2023-10-30T13:23:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600c4160 con 0x7f75600c0e20 2023-10-30T13:23:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600c4690 con 0x7f75600c2f70 2023-10-30T13:23:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.924+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7560152510 con 0x7f754c003980 2023-10-30T13:23:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7fb9d26db640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] conn(0x7fb9b006e6c0 0x7fb9b0070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f0e3849c640 1 -- 172.21.15.46:0/1163344022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300c6b40 msgr2=0x7f0e30161be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f0e3849c640 1 --2- 172.21.15.46:0/1163344022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300c6b40 0x7f0e30161be0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0e200097c0 tx=0x7f0e2001ecb0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7fb9d16d9640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9b00664a0 0x7fb9b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7fb9d26db640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] conn(0x7fb9b006e6c0 0x7fb9b0070ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7fa36582d640 1 --2- 172.21.15.46:0/2267026276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa344066570 0x7fa344068a30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa3580070b0 tx=0x7fa358007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7fa36602e640 1 --2- 172.21.15.46:0/2267026276 >> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] conn(0x7fa34406e790 0x7fa344070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f0e3849c640 1 -- 172.21.15.46:0/1163344022 shutdown_connections 2023-10-30T13:23:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f0e3849c640 1 --2- 172.21.15.46:0/1163344022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e300c0ed0 0x7f0e3006a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f0e3849c640 1 --2- 172.21.15.46:0/1163344022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300c6b40 0x7f0e30161be0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f0e3849c640 1 -- 172.21.15.46:0/1163344022 >> 172.21.15.46:0/1163344022 conn(0x7f0e300c7910 msgr2=0x7f0e300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f0e3849c640 1 -- 172.21.15.46:0/1163344022 shutdown_connections 2023-10-30T13:23:28.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f0e3849c640 1 -- 172.21.15.46:0/1163344022 wait complete. 2023-10-30T13:23:28.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f75667f1640 1 -- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f754c003980 msgr2=0x7f7560151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f75667f1640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f754c003980 0x7f7560151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f75667f1640 1 -- 172.21.15.46:0/990813772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75600c0e20 msgr2=0x7f7560070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f75667f1640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75600c0e20 0x7f7560070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f75667f1640 1 -- 172.21.15.46:0/990813772 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75600c43b0 con 0x7f75600c2f70 2023-10-30T13:23:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f75667f1640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75600c2f70 0x7f75601fe5a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f756006f0d0 tx=0x7f7550011920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7fa36602e640 1 --2- 172.21.15.46:0/2267026276 >> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] conn(0x7fa34406e790 0x7fa344070bb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f8ded7fa640 1 -- 172.21.15.46:0/2511118376 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8de001f460 con 0x7f8df015cd40 2023-10-30T13:23:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f75567fc640 1 -- 172.21.15.46:0/990813772 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550015870 con 0x7f75600c2f70 2023-10-30T13:23:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f75567fc640 1 -- 172.21.15.46:0/990813772 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75500159d0 con 0x7f75600c2f70 2023-10-30T13:23:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f75567fc640 1 -- 172.21.15.46:0/990813772 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550015d60 con 0x7f75600c2f70 2023-10-30T13:23:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7fb9ca7fc640 1 -- 172.21.15.46:0/1857185378 <== osd.2 v2:172.21.15.46:6808/3816077541 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb9b0071190 con 0x7fb9b006e6c0 2023-10-30T13:23:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75601ff420 con 0x7f75600c2f70 2023-10-30T13:23:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7fb9d16d9640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9b00664a0 0x7fb9b0068960 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80015e0 tx=0x7fb9b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7fa3567fc640 1 -- 172.21.15.46:0/2267026276 <== osd.3 v2:172.21.15.46:6824/2013670008 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fa344071260 con 0x7fa34406e790 2023-10-30T13:23:28.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75601ff9e0 con 0x7f75600c2f70 2023-10-30T13:23:28.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f0e3849c640 1 Processor -- start 2023-10-30T13:23:28.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f0e3849c640 1 -- start start 2023-10-30T13:23:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f8ded7fa640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ddc0664a0 0x7f8ddc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.928+0000 7f8ded7fa640 1 -- 172.21.15.46:0/2511118376 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8de0083e60 con 0x7f8df015cd40 2023-10-30T13:23:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f8ded7fa640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] conn(0x7f8ddc06e6c0 0x7f8ddc070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f8ded7fa640 1 -- 172.21.15.46:0/2511118376 --> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ddc071190 con 0x7f8ddc06e6c0 2023-10-30T13:23:28.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f8ded7fa640 1 -- 172.21.15.46:0/2511118376 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8de0086480 con 0x7f8df015cd40 2023-10-30T13:23:28.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f8df49e2640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ddc0664a0 0x7f8ddc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f756006a170 con 0x7f75600c2f70 2023-10-30T13:23:28.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e3849c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300c0ed0 0x7f0e30160400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e3849c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e30158090 0x7f0e30160940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e3849c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3006c170 0x7f0e3015f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e3849c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e30161780 con 0x7f0e300c0ed0 2023-10-30T13:23:28.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e3849c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3015ffe0 con 0x7f0e3006c170 2023-10-30T13:23:28.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e3849c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e30155cb0 con 0x7f0e30158090 2023-10-30T13:23:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e36a39640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3006c170 0x7f0e3015f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e36a39640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3006c170 0x7f0e3015f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39344/0 (socket says 172.21.15.46:39344) 2023-10-30T13:23:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e36a39640 1 -- 172.21.15.46:0/4270274065 learned_addr learned my addr 172.21.15.46:0/4270274065 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e36238640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300c0ed0 0x7f0e30160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e36a39640 1 -- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e30158090 msgr2=0x7f0e30160940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-30T13:23:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e36a39640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e30158090 0x7f0e30160940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e36a39640 1 -- 172.21.15.46:0/4270274065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300c0ed0 msgr2=0x7f0e30160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e36a39640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300c0ed0 0x7f0e30160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e36a39640 1 -- 172.21.15.46:0/4270274065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e20009420 con 0x7f0e3006c170 2023-10-30T13:23:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e36a39640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3006c170 0x7f0e3015f000 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800efc0 tx=0x7f0e2800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e26ffd640 1 -- 172.21.15.46:0/4270274065 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28014480 con 0x7f0e3006c170 2023-10-30T13:23:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f75567fc640 1 -- 172.21.15.46:0/990813772 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f75500059a0 con 0x7f75600c2f70 2023-10-30T13:23:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e26ffd640 1 -- 172.21.15.46:0/4270274065 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2800f040 con 0x7f0e3006c170 2023-10-30T13:23:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e26ffd640 1 -- 172.21.15.46:0/4270274065 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28014820 con 0x7f0e3006c170 2023-10-30T13:23:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.932+0000 7f0e3849c640 1 -- 172.21.15.46:0/4270274065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e30155730 con 0x7f0e3006c170 2023-10-30T13:23:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f0e3849c640 1 -- 172.21.15.46:0/4270274065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e301559d0 con 0x7f0e3006c170 2023-10-30T13:23:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f8deffff640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] conn(0x7f8ddc06e6c0 0x7f8ddc070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f8deffff640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] conn(0x7f8ddc06e6c0 0x7f8ddc070ae0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f8df49e2640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ddc0664a0 0x7f8ddc068960 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8de4005010 tx=0x7f8de4007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7fb612ed6640 1 -- 172.21.15.46:0/3837103134 --> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb5f0002cc0 con 0x7fb5ec06e7c0 2023-10-30T13:23:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f75567fc640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75440665a0 0x7f7544068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f0e3849c640 1 -- 172.21.15.46:0/4270274065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0e3015e2f0 con 0x7f0e3006c170 2023-10-30T13:23:28.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f75567fc640 1 -- 172.21.15.46:0/990813772 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75500847a0 con 0x7f75600c2f70 2023-10-30T13:23:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f75567fc640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] conn(0x7f754406e7c0 0x7f7544070be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f7565ff0640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75440665a0 0x7f7544068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f75567fc640 1 -- 172.21.15.46:0/990813772 --> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7544071290 con 0x7f754406e7c0 2023-10-30T13:23:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f75657ef640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] conn(0x7f754406e7c0 0x7f7544070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f75567fc640 1 -- 172.21.15.46:0/990813772 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7550084b50 con 0x7f75600c2f70 2023-10-30T13:23:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.936+0000 7f7565ff0640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75440665a0 0x7f7544068a60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7558000f90 tx=0x7f75580b0040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7f8ded7fa640 1 -- 172.21.15.46:0/2511118376 <== osd.7 v2:172.21.15.154:6816/1343138186 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f8ddc071190 con 0x7f8ddc06e6c0 2023-10-30T13:23:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7f0e26ffd640 1 -- 172.21.15.46:0/4270274065 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e28014980 con 0x7f0e3006c170 2023-10-30T13:23:28.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7fb609ffb640 1 -- 172.21.15.46:0/3837103134 <== osd.0 v2:172.21.15.46:6809/1936524547 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb5f0002cc0 con 0x7fb5ec06e7c0 2023-10-30T13:23:28.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7f75657ef640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] conn(0x7f754406e7c0 0x7f7544070be0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7f75567fc640 1 -- 172.21.15.46:0/990813772 <== osd.4 v2:172.21.15.154:6800/1011303130 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f7544071290 con 0x7f754406e7c0 2023-10-30T13:23:28.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7f0e26ffd640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e140664a0 0x7f0e14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7f0e26ffd640 1 -- 172.21.15.46:0/4270274065 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e28085780 con 0x7f0e3006c170 2023-10-30T13:23:28.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7fb612ed6640 1 -- 172.21.15.46:0/3837103134 >> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] conn(0x7fb5ec06e7c0 msgr2=0x7fb5ec070be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7fb612ed6640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] conn(0x7fb5ec06e7c0 0x7fb5ec070be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7f0e36238640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e140664a0 0x7f0e14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7fb612ed6640 1 -- 172.21.15.46:0/3837103134 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5ec0665a0 msgr2=0x7fb5ec068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7fb612ed6640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5ec0665a0 0x7fb5ec068a60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc01f230 tx=0x7fb5fc0aa000 comp rx=0 tx=0).stop 2023-10-30T13:23:28.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7f0e26ffd640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] conn(0x7f0e1406e6c0 0x7f0e14070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7f0e26ffd640 1 -- 172.21.15.46:0/4270274065 --> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e14071190 con 0x7f0e1406e6c0 2023-10-30T13:23:28.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7f0e26ffd640 1 -- 172.21.15.46:0/4270274065 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e28085b30 con 0x7f0e3006c170 2023-10-30T13:23:28.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7fb612ed6640 1 -- 172.21.15.46:0/3837103134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb60c141fa0 msgr2=0x7fb60c140850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7fb612ed6640 1 --2- 172.21.15.46:0/3837103134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb60c141fa0 0x7fb60c140850 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb60400d4b0 tx=0x7fb60400d970 comp rx=0 tx=0).stop 2023-10-30T13:23:28.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7fb611473640 1 -- 172.21.15.46:0/3837103134 reap_dead start 2023-10-30T13:23:28.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7fb612ed6640 1 -- 172.21.15.46:0/3837103134 shutdown_connections 2023-10-30T13:23:28.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7fb612ed6640 1 -- 172.21.15.46:0/3837103134 >> 172.21.15.46:0/3837103134 conn(0x7fb60c06bfe0 msgr2=0x7fb60c0c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.940+0000 7f0e35a37640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] conn(0x7f0e1406e6c0 0x7f0e14070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.944+0000 7fb612ed6640 1 -- 172.21.15.46:0/3837103134 shutdown_connections 2023-10-30T13:23:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.944+0000 7fb612ed6640 1 -- 172.21.15.46:0/3837103134 wait complete. 2023-10-30T13:23:28.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.944+0000 7f0e36238640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e140664a0 0x7f0e14068960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0011f0 tx=0x7f0e2c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.944+0000 7f0e35a37640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] conn(0x7f0e1406e6c0 0x7f0e14070ae0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.944+0000 7f0e26ffd640 1 -- 172.21.15.46:0/4270274065 <== osd.6 v2:172.21.15.154:6824/1247354054 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f0e14071190 con 0x7f0e1406e6c0 2023-10-30T13:23:28.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.948+0000 7fe3c4a6b640 1 -- 172.21.15.46:0/3701906984 --> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe3c01592f0 con 0x7fe3a0072ad0 2023-10-30T13:23:28.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.948+0000 7fe3aeffd640 1 -- 172.21.15.46:0/3701906984 <== osd.5 v2:172.21.15.154:6801/956300018 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe3c01592f0 con 0x7fe3a0072ad0 2023-10-30T13:23:28.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fe3acff9640 1 -- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] conn(0x7fe3a0072ad0 msgr2=0x7fe3a0074ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fe3acff9640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] conn(0x7fe3a0072ad0 0x7fe3a0074ef0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 --> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbb14155c80 con 0x7fbaf806e790 2023-10-30T13:23:28.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fe3acff9640 1 -- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3a006a8b0 msgr2=0x7fe3a006cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fe3acff9640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3a006a8b0 0x7fe3a006cd70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800c260 tx=0x7fe3b800c4f0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fe3acff9640 1 -- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3c00c6b80 msgr2=0x7fe3c0157960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fe3acff9640 1 --2- 172.21.15.46:0/3701906984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3c00c6b80 0x7fe3c0157960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe3b400b3e0 tx=0x7fe3b400b8a0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fe3bed9d640 1 -- 172.21.15.46:0/3701906984 reap_dead start 2023-10-30T13:23:28.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fe3acff9640 1 -- 172.21.15.46:0/3701906984 shutdown_connections 2023-10-30T13:23:28.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fe3acff9640 1 -- 172.21.15.46:0/3701906984 >> 172.21.15.46:0/3701906984 conn(0x7fe3c00c7910 msgr2=0x7fe3c006aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fe3acff9640 1 -- 172.21.15.46:0/3701906984 shutdown_connections 2023-10-30T13:23:28.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fe3acff9640 1 -- 172.21.15.46:0/3701906984 wait complete. 2023-10-30T13:23:28.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fa36682f640 1 -- 172.21.15.46:0/2267026276 --> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa3601619a0 con 0x7fa34406e790 2023-10-30T13:23:28.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fbb067fc640 1 -- 172.21.15.46:0/893893407 <== osd.1 v2:172.21.15.46:6800/1107843451 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbb0c009bb0 con 0x7fbaf806e790 2023-10-30T13:23:28.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.956+0000 7fb9d413e640 1 -- 172.21.15.46:0/1857185378 --> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb9cc0cc1d0 con 0x7fb9b006e6c0 2023-10-30T13:23:28.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fa3567fc640 1 -- 172.21.15.46:0/2267026276 <== osd.3 v2:172.21.15.46:6824/2013670008 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa35c001470 con 0x7fa34406e790 2023-10-30T13:23:28.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 >> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] conn(0x7fbaf806e790 msgr2=0x7fbaf8070bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fbb1a8dc640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] conn(0x7fbaf806e790 0x7fbaf8070bb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbaf8066570 msgr2=0x7fbaf8068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fbb1a8dc640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbaf8066570 0x7fbaf8068a30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbb08000c10 tx=0x7fbb080aa070 comp rx=0 tx=0).stop 2023-10-30T13:23:28.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb140c1810 msgr2=0x7fbb140c5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fbb1a8dc640 1 --2- 172.21.15.46:0/893893407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb140c1810 0x7fbb140c5e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000acf0 tx=0x7fbb10009430 comp rx=0 tx=0).stop 2023-10-30T13:23:28.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fbb1a0db640 1 -- 172.21.15.46:0/893893407 reap_dead start 2023-10-30T13:23:28.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fa36682f640 1 -- 172.21.15.46:0/2267026276 >> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] conn(0x7fa34406e790 msgr2=0x7fa344070bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fa36682f640 1 --2- 172.21.15.46:0/2267026276 >> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] conn(0x7fa34406e790 0x7fa344070bb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fa36682f640 1 -- 172.21.15.46:0/2267026276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa344066570 msgr2=0x7fa344068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fa36682f640 1 --2- 172.21.15.46:0/2267026276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa344066570 0x7fa344068a30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa3580070b0 tx=0x7fa358007040 comp rx=0 tx=0).stop 2023-10-30T13:23:28.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fa36682f640 1 -- 172.21.15.46:0/2267026276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa36006fef0 msgr2=0x7fa3601583d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fa36682f640 1 --2- 172.21.15.46:0/2267026276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa36006fef0 0x7fa3601583d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa35001c960 tx=0x7fa35001bad0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 shutdown_connections 2023-10-30T13:23:28.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 >> 172.21.15.46:0/893893407 conn(0x7fbb140c75d0 msgr2=0x7fbb140c8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fa36602e640 1 -- 172.21.15.46:0/2267026276 reap_dead start 2023-10-30T13:23:28.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fa36682f640 1 -- 172.21.15.46:0/2267026276 shutdown_connections 2023-10-30T13:23:28.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fa36682f640 1 -- 172.21.15.46:0/2267026276 >> 172.21.15.46:0/2267026276 conn(0x7fa3600c75d0 msgr2=0x7fa3600c51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 shutdown_connections 2023-10-30T13:23:28.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fb9ca7fc640 1 -- 172.21.15.46:0/1857185378 <== osd.2 v2:172.21.15.46:6808/3816077541 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb9cc0cc1d0 con 0x7fb9b006e6c0 2023-10-30T13:23:28.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fbb1a8dc640 1 -- 172.21.15.46:0/893893407 wait complete. 2023-10-30T13:23:28.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fa36682f640 1 -- 172.21.15.46:0/2267026276 shutdown_connections 2023-10-30T13:23:28.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.960+0000 7fa36682f640 1 -- 172.21.15.46:0/2267026276 wait complete. 2023-10-30T13:23:28.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7fb9d413e640 1 -- 172.21.15.46:0/1857185378 >> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] conn(0x7fb9b006e6c0 msgr2=0x7fb9b0070ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7fb9d413e640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] conn(0x7fb9b006e6c0 0x7fb9b0070ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7fb9d413e640 1 -- 172.21.15.46:0/1857185378 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9b00664a0 msgr2=0x7fb9b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7fb9d413e640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9b00664a0 0x7fb9b0068960 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80015e0 tx=0x7fb9b8009040 comp rx=0 tx=0).stop 2023-10-30T13:23:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7fb9d413e640 1 -- 172.21.15.46:0/1857185378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9bc004ba0 msgr2=0x7fb9cc156c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7fb9d413e640 1 --2- 172.21.15.46:0/1857185378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9bc004ba0 0x7fb9cc156c40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00b0040 tx=0x7fb9c0019c00 comp rx=0 tx=0).stop 2023-10-30T13:23:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7fb9d26db640 1 -- 172.21.15.46:0/1857185378 reap_dead start 2023-10-30T13:23:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7fb9d413e640 1 -- 172.21.15.46:0/1857185378 shutdown_connections 2023-10-30T13:23:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7fb9d413e640 1 -- 172.21.15.46:0/1857185378 >> 172.21.15.46:0/1857185378 conn(0x7fb9cc06bfe0 msgr2=0x7fb9cc0cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7fb9d413e640 1 -- 172.21.15.46:0/1857185378 shutdown_connections 2023-10-30T13:23:28.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7fb9d413e640 1 -- 172.21.15.46:0/1857185378 wait complete. 2023-10-30T13:23:28.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 --> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7560069e90 con 0x7f754406e7c0 2023-10-30T13:23:28.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7f75567fc640 1 -- 172.21.15.46:0/990813772 <== osd.4 v2:172.21.15.154:6800/1011303130 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f755c006ca0 con 0x7f754406e7c0 2023-10-30T13:23:28.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] conn(0x7f754406e7c0 msgr2=0x7f7544070be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.964+0000 7f7568254640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] conn(0x7f754406e7c0 0x7f7544070be0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.968+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75440665a0 msgr2=0x7f7544068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.968+0000 7f7568254640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75440665a0 0x7f7544068a60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7558000f90 tx=0x7f75580b0040 comp rx=0 tx=0).stop 2023-10-30T13:23:28.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.968+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75600c2f70 msgr2=0x7f75601fe5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.968+0000 7f7568254640 1 --2- 172.21.15.46:0/990813772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75600c2f70 0x7f75601fe5a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f756006f0d0 tx=0x7f7550011920 comp rx=0 tx=0).stop 2023-10-30T13:23:28.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.968+0000 7f75667f1640 1 -- 172.21.15.46:0/990813772 reap_dead start 2023-10-30T13:23:28.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.968+0000 7f8df6c46640 1 -- 172.21.15.46:0/2511118376 --> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8df015e010 con 0x7f8ddc06e6c0 2023-10-30T13:23:28.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.968+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 shutdown_connections 2023-10-30T13:23:28.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.968+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 >> 172.21.15.46:0/990813772 conn(0x7f75600c7910 msgr2=0x7f75600c9820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 shutdown_connections 2023-10-30T13:23:28.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f7568254640 1 -- 172.21.15.46:0/990813772 wait complete. 2023-10-30T13:23:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f8ded7fa640 1 -- 172.21.15.46:0/2511118376 <== osd.7 v2:172.21.15.154:6816/1343138186 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8de800f040 con 0x7f8ddc06e6c0 2023-10-30T13:23:28.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f0e3849c640 1 -- 172.21.15.46:0/4270274065 --> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0e3015e010 con 0x7f0e1406e6c0 2023-10-30T13:23:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f8dd77fe640 1 -- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] conn(0x7f8ddc06e6c0 msgr2=0x7f8ddc070ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f8dd77fe640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] conn(0x7f8ddc06e6c0 0x7f8ddc070ae0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.151 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-10-30T13:23:28.151 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T13:23:28.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f8dd77fe640 1 -- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ddc0664a0 msgr2=0x7f8ddc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f8dd77fe640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ddc0664a0 0x7f8ddc068960 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8de4005010 tx=0x7f8de4007040 comp rx=0 tx=0).stop 2023-10-30T13:23:28.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f8dd77fe640 1 -- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8df015cd40 msgr2=0x7f8df015d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f8dd77fe640 1 --2- 172.21.15.46:0/2511118376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8df015cd40 0x7f8df015d1c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8df00c8630 tx=0x7f8de000f8a0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f8df51e3640 1 -- 172.21.15.46:0/2511118376 reap_dead start 2023-10-30T13:23:28.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f8dd77fe640 1 -- 172.21.15.46:0/2511118376 shutdown_connections 2023-10-30T13:23:28.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f8dd77fe640 1 -- 172.21.15.46:0/2511118376 >> 172.21.15.46:0/2511118376 conn(0x7f8df00c7910 msgr2=0x7f8df0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f8dd77fe640 1 -- 172.21.15.46:0/2511118376 shutdown_connections 2023-10-30T13:23:28.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.972+0000 7f8dd77fe640 1 -- 172.21.15.46:0/2511118376 wait complete. 2023-10-30T13:23:28.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.980+0000 7f0e26ffd640 1 -- 172.21.15.46:0/4270274065 <== osd.6 v2:172.21.15.154:6824/1247354054 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0e200b0440 con 0x7f0e1406e6c0 2023-10-30T13:23:28.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.980+0000 7f0e24ff9640 1 -- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] conn(0x7f0e1406e6c0 msgr2=0x7f0e14070ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.980+0000 7f0e24ff9640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] conn(0x7f0e1406e6c0 0x7f0e14070ae0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.980+0000 7f0e24ff9640 1 -- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e140664a0 msgr2=0x7f0e14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.980+0000 7f0e24ff9640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e140664a0 0x7f0e14068960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0011f0 tx=0x7f0e2c009040 comp rx=0 tx=0).stop 2023-10-30T13:23:28.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.980+0000 7f0e24ff9640 1 -- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3006c170 msgr2=0x7f0e3015f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.980+0000 7f0e24ff9640 1 --2- 172.21.15.46:0/4270274065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3006c170 0x7f0e3015f000 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800efc0 tx=0x7f0e2800c490 comp rx=0 tx=0).stop 2023-10-30T13:23:28.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.980+0000 7f0e36a39640 1 -- 172.21.15.46:0/4270274065 reap_dead start 2023-10-30T13:23:28.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.980+0000 7f0e24ff9640 1 -- 172.21.15.46:0/4270274065 shutdown_connections 2023-10-30T13:23:28.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.980+0000 7f0e24ff9640 1 -- 172.21.15.46:0/4270274065 >> 172.21.15.46:0/4270274065 conn(0x7f0e300c7910 msgr2=0x7f0e30156800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.984+0000 7f0e24ff9640 1 -- 172.21.15.46:0/4270274065 shutdown_connections 2023-10-30T13:23:28.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:27.984+0000 7f0e24ff9640 1 -- 172.21.15.46:0/4270274065 wait complete. 2023-10-30T13:23:28.158 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-10-30T13:23:28.158 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T13:23:28.159 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-10-30T13:23:28.160 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T13:23:28.161 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-10-30T13:23:28.161 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T13:23:28.162 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-10-30T13:23:28.162 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T13:23:28.163 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-10-30T13:23:28.164 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T13:23:28.164 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-10-30T13:23:28.165 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T13:23:28.166 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-10-30T13:23:28.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T13:23:28.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.436+0000 7fc7edc9c640 1 Processor -- start 2023-10-30T13:23:28.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.436+0000 7fc7edc9c640 1 -- start start 2023-10-30T13:23:28.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.440+0000 7fc7edc9c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc7e80ca360 0x7fc7e80cac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.440+0000 7fc7edc9c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e80cb780 con 0x7fc7e80c4430 2023-10-30T13:23:28.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.440+0000 7fc7edc9c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e80ca7f0 con 0x7fc7e8153c60 2023-10-30T13:23:28.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.440+0000 7fc7edc9c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e8154040 con 0x7fc7e80ca360 2023-10-30T13:23:28.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.440+0000 7fc7e6ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc7e80c4430 0x7fc7e8159830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48470/0 (socket says 172.21.15.46:48470) 2023-10-30T13:23:28.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.440+0000 7fc7e6ffd640 1 -- 172.21.15.46:0/757334526 learned_addr learned my addr 172.21.15.46:0/757334526 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:28.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.440+0000 7fc7e7fff640 1 --2- 172.21.15.46:0/757334526 >> v2:172.21.15.154:3300/0 conn(0x7fc7e80ca360 0x7fc7e80cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.440+0000 7fa13e0d6640 1 Processor -- start 2023-10-30T13:23:28.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.440+0000 7fa13e0d6640 1 -- start start 2023-10-30T13:23:28.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fc7e67fc640 1 -- 172.21.15.46:0/757334526 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073072464 0 0) 0x7fc7e80ca7f0 con 0x7fc7e8153c60 2023-10-30T13:23:28.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fc7e67fc640 1 -- 172.21.15.46:0/757334526 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7cc003530 con 0x7fc7e8153c60 2023-10-30T13:23:28.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fc7e7fff640 1 -- 172.21.15.46:0/757334526 >> v1:172.21.15.154:6789/0 conn(0x7fc7e8153c60 legacy=0x7fc7e80cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fc7e7fff640 1 -- 172.21.15.46:0/757334526 >> v1:172.21.15.46:6789/0 conn(0x7fc7e80c4430 legacy=0x7fc7e8159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fc7e7fff640 1 -- 172.21.15.46:0/757334526 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e815a2e0 con 0x7fc7e80ca360 2023-10-30T13:23:28.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fa13e0d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1380c6b40 0x7fa138161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fa13e0d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1380c72b0 con 0x7fa138158090 2023-10-30T13:23:28.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fa13e0d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13806acb0 con 0x7fa1380c0ed0 2023-10-30T13:23:28.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fa13e0d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa138162120 con 0x7fa1380c6b40 2023-10-30T13:23:28.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fa12effd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1380c6b40 0x7fa138161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fa12effd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1380c6b40 0x7fa138161be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59476/0 (socket says 172.21.15.46:59476) 2023-10-30T13:23:28.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fa12effd640 1 -- 172.21.15.46:0/3537073808 learned_addr learned my addr 172.21.15.46:0/3537073808 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fa12effd640 1 -- 172.21.15.46:0/3537073808 >> v1:172.21.15.154:6789/0 conn(0x7fa1380c0ed0 legacy=0x7fa13806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:28.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fa12effd640 1 -- 172.21.15.46:0/3537073808 >> v1:172.21.15.46:6789/0 conn(0x7fa138158090 legacy=0x7fa138069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.444+0000 7fa12effd640 1 -- 172.21.15.46:0/3537073808 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1380c6fd0 con 0x7fa1380c6b40 2023-10-30T13:23:28.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.448+0000 7fc7e7fff640 1 --2- 172.21.15.46:0/757334526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e80ca360 0x7fc7e80cac40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009d90 tx=0x7fc7e001c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dc529e8bf711b03 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.448+0000 7fc7e67fc640 1 -- 172.21.15.46:0/757334526 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e00a58b0 con 0x7fc7e80ca360 2023-10-30T13:23:28.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.448+0000 7fc7e67fc640 1 -- 172.21.15.46:0/757334526 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e00a5a10 con 0x7fc7e80ca360 2023-10-30T13:23:28.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.448+0000 7fc7e67fc640 1 -- 172.21.15.46:0/757334526 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e00a5d80 con 0x7fc7e80ca360 2023-10-30T13:23:28.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.448+0000 7f6ba010f640 1 Processor -- start 2023-10-30T13:23:28.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.448+0000 7fc7edc9c640 1 -- 172.21.15.46:0/757334526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e80ca360 msgr2=0x7fc7e80cac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.448+0000 7fc7edc9c640 1 --2- 172.21.15.46:0/757334526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e80ca360 0x7fc7e80cac40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009d90 tx=0x7fc7e001c8e0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.448+0000 7fa12effd640 1 --2- 172.21.15.46:0/3537073808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1380c6b40 0x7fa138161be0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa1280098a0 tx=0x7fa12801ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=324856f78d6a80eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6ba010f640 1 -- start start 2023-10-30T13:23:28.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fa136ffd640 1 -- 172.21.15.46:0/3537073808 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1280a8a10 con 0x7fa1380c6b40 2023-10-30T13:23:28.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fa136ffd640 1 -- 172.21.15.46:0/3537073808 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1280a8b70 con 0x7fa1380c6b40 2023-10-30T13:23:28.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fa136ffd640 1 -- 172.21.15.46:0/3537073808 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1280b0440 con 0x7fa1380c6b40 2023-10-30T13:23:28.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fc7edc9c640 1 -- 172.21.15.46:0/757334526 shutdown_connections 2023-10-30T13:23:28.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fc7edc9c640 1 --2- 172.21.15.46:0/757334526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e80ca360 0x7fc7e80cac40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fc7edc9c640 1 -- 172.21.15.46:0/757334526 >> 172.21.15.46:0/757334526 conn(0x7fc7e806bfe0 msgr2=0x7fc7e806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fc7edc9c640 1 -- 172.21.15.46:0/757334526 shutdown_connections 2023-10-30T13:23:28.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fc7edc9c640 1 -- 172.21.15.46:0/757334526 wait complete. 2023-10-30T13:23:28.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fa13e0d6640 1 -- 172.21.15.46:0/3537073808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1380c6b40 msgr2=0x7fa138161be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fa13e0d6640 1 --2- 172.21.15.46:0/3537073808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1380c6b40 0x7fa138161be0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa1280098a0 tx=0x7fa12801ed10 comp rx=0 tx=0).stop 2023-10-30T13:23:28.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6ba010f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b98153c60 0x7f6b980ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fa13e0d6640 1 -- 172.21.15.46:0/3537073808 shutdown_connections 2023-10-30T13:23:28.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fa13e0d6640 1 --2- 172.21.15.46:0/3537073808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1380c6b40 0x7fa138161be0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fa13e0d6640 1 -- 172.21.15.46:0/3537073808 >> 172.21.15.46:0/3537073808 conn(0x7fa1380c7910 msgr2=0x7fa1380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6ba010f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6b980c4430 0x7f6b980cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6ba010f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b980cb780 con 0x7f6b98159830 2023-10-30T13:23:28.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6ba010f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b98154040 con 0x7f6b980c4430 2023-10-30T13:23:28.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6ba010f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b980cc9f0 con 0x7f6b98153c60 2023-10-30T13:23:28.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6b9d6aa640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6b980c4430 0x7f6b980cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6b9e6ac640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6b98159830 0x7f6b980cac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48492/0 (socket says 172.21.15.46:48492) 2023-10-30T13:23:28.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6b9e6ac640 1 -- 172.21.15.46:0/3103299896 learned_addr learned my addr 172.21.15.46:0/3103299896 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:28.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6b9d6aa640 1 -- 172.21.15.46:0/3103299896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b98153c60 msgr2=0x7f6b980ca360 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T13:23:28.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6b9d6aa640 1 --2- 172.21.15.46:0/3103299896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b98153c60 0x7f6b980ca360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6b9d6aa640 1 -- 172.21.15.46:0/3103299896 >> v1:172.21.15.46:6789/0 conn(0x7f6b98159830 legacy=0x7f6b980cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7f6b9d6aa640 1 -- 172.21.15.46:0/3103299896 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9815a2e0 con 0x7f6b980c4430 2023-10-30T13:23:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fa13e0d6640 1 -- 172.21.15.46:0/3537073808 shutdown_connections 2023-10-30T13:23:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fa13e0d6640 1 -- 172.21.15.46:0/3537073808 wait complete. 2023-10-30T13:23:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.452+0000 7fc7edc9c640 1 Processor -- start 2023-10-30T13:23:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7edc9c640 1 -- start start 2023-10-30T13:23:28.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7f21f1225640 1 Processor -- start 2023-10-30T13:23:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7edc9c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e80c4430 0x7fc7e80ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7edc9c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e80ca360 0x7fc7e80b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7edc9c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e8153c60 0x7fc7e80c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7edc9c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e80b33b0 con 0x7fc7e8153c60 2023-10-30T13:23:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7edc9c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e80c6940 con 0x7fc7e80ca360 2023-10-30T13:23:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7edc9c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e81485a0 con 0x7fc7e80c4430 2023-10-30T13:23:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e8153c60 0x7fc7e80c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e8153c60 0x7fc7e80c8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39612/0 (socket says 172.21.15.46:39612) 2023-10-30T13:23:28.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7e7fff640 1 -- 172.21.15.46:0/2611327727 learned_addr learned my addr 172.21.15.46:0/2611327727 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7e6ffd640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e80ca360 0x7fc7e80b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7e77fe640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e80c4430 0x7fc7e80ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7e7fff640 1 -- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e80c4430 msgr2=0x7fc7e80ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7e7fff640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e80c4430 0x7fc7e80ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7e7fff640 1 -- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e80ca360 msgr2=0x7fc7e80b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7e7fff640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e80ca360 0x7fc7e80b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7e7fff640 1 -- 172.21.15.46:0/2611327727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e0009a70 con 0x7fc7e8153c60 2023-10-30T13:23:28.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fc7e7fff640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e8153c60 0x7fc7e80c8cb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009d90 tx=0x7fc7e0019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fa13e0d6640 1 Processor -- start 2023-10-30T13:23:28.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7fa13e0d6640 1 -- start start 2023-10-30T13:23:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7f02ca054640 1 Processor -- start 2023-10-30T13:23:28.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7f02ca054640 1 -- start start 2023-10-30T13:23:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.456+0000 7f21f1225640 1 -- start start 2023-10-30T13:23:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21f1225640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21ec158090 0x7f21ec161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21f1225640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f21ec0c6b40 0x7f21ec069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21f1225640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ec0c72b0 con 0x7f21ec06a3d0 2023-10-30T13:23:28.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21f1225640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ec162120 con 0x7f21ec0c6b40 2023-10-30T13:23:28.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21f1225640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec0c6fd0 con 0x7f21ec158090 2023-10-30T13:23:28.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fc7dffff640 1 -- 172.21.15.46:0/2611327727 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e00a58b0 con 0x7fc7e8153c60 2023-10-30T13:23:28.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fc7dffff640 1 -- 172.21.15.46:0/2611327727 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e0003db0 con 0x7fc7e8153c60 2023-10-30T13:23:28.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fc7dffff640 1 -- 172.21.15.46:0/2611327727 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0004240 con 0x7fc7e8153c60 2023-10-30T13:23:28.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fc7edc9c640 1 -- 172.21.15.46:0/2611327727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e80b3060 con 0x7fc7e8153c60 2023-10-30T13:23:28.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21ead9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21ec158090 0x7f21ec161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21eb59e640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f21ec06a3d0 0x7f21ec0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:53140/0 (socket says 172.21.15.46:53140) 2023-10-30T13:23:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21eb59e640 1 -- 172.21.15.46:0/3628627623 learned_addr learned my addr 172.21.15.46:0/3628627623 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21ea59c640 1 --2- 172.21.15.46:0/3628627623 >> v2:172.21.15.154:3300/0 conn(0x7f21ec0c6b40 0x7f21ec069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa13e0d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1380c0ed0 0x7fa13814c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa13e0d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa138158090 0x7fa13814b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa13e0d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa138155ca0 0x7fa13814bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f02ca054640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02c40c4430 0x7f02c40ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fc7edc9c640 1 -- 172.21.15.46:0/2611327727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e81f6080 con 0x7fc7e8153c60 2023-10-30T13:23:28.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f02bb7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f02c4153c60 0x7f02c40cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48506/0 (socket says 172.21.15.46:48506) 2023-10-30T13:23:28.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa12effd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa138158090 0x7fa13814b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa12effd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa138158090 0x7fa13814b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59520/0 (socket says 172.21.15.46:59520) 2023-10-30T13:23:28.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f02baffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02c40c4430 0x7f02c40ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f02baffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02c40c4430 0x7f02c40ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39632/0 (socket says 172.21.15.46:39632) 2023-10-30T13:23:28.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21ea59c640 1 -- 172.21.15.46:0/3628627623 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21ec158090 msgr2=0x7f21ec161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:28.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21ea59c640 1 --2- 172.21.15.46:0/3628627623 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21ec158090 0x7f21ec161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21ea59c640 1 -- 172.21.15.46:0/3628627623 >> v1:172.21.15.154:6789/0 conn(0x7f21ec06a3d0 legacy=0x7f21ec0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f02baffd640 1 -- 172.21.15.46:0/4117438794 learned_addr learned my addr 172.21.15.46:0/4117438794 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fc7ddffb640 1 -- 172.21.15.46:0/2611327727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e80b34f0 con 0x7fc7e8153c60 2023-10-30T13:23:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa137fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa138155ca0 0x7fa13814bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa137fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa138155ca0 0x7fa13814bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39382/0 (socket says 172.21.15.46:39382) 2023-10-30T13:23:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa137fff640 1 -- 172.21.15.46:0/2168517300 learned_addr learned my addr 172.21.15.46:0/2168517300 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13814b0e0 con 0x7fa1380c0ed0 2023-10-30T13:23:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1380c8c10 con 0x7fa138155ca0 2023-10-30T13:23:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13814cae0 con 0x7fa138158090 2023-10-30T13:23:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa89bc54640 1 Processor -- start 2023-10-30T13:23:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa89bc54640 1 -- start start 2023-10-30T13:23:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa89bc54640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8940cad80 0x7fa8940cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa89bc54640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa8940cf160 0x7fa8940cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa89bc54640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa894158350 con 0x7fa894157f70 2023-10-30T13:23:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa89bc54640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8940ca8b0 con 0x7fa8940cad80 2023-10-30T13:23:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa89bc54640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8940ca2f0 con 0x7fa8940cf160 2023-10-30T13:23:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa1377fe640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1380c0ed0 0x7fa13814c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f6b9d6aa640 1 --2- 172.21.15.46:0/3103299896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b980c4430 0x7f6b980cc4b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009d90 tx=0x7f6b9001c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7a0ae77adbc4e2d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f02ca054640 1 -- 172.21.15.46:0/4117438794 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c40cb780 con 0x7f02c4153c60 2023-10-30T13:23:28.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f02ca054640 1 -- 172.21.15.46:0/4117438794 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c415a2e0 con 0x7f02c4159830 2023-10-30T13:23:28.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa8991ef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8940cad80 0x7fa8940cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa8991ef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8940cad80 0x7fa8940cc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39648/0 (socket says 172.21.15.46:39648) 2023-10-30T13:23:28.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7fa8991ef640 1 -- 172.21.15.46:0/3992255075 learned_addr learned my addr 172.21.15.46:0/3992255075 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f02ca054640 1 -- 172.21.15.46:0/4117438794 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c4154040 con 0x7f02c40c4430 2023-10-30T13:23:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa12effd640 1 -- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa138155ca0 msgr2=0x7fa13814bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa12effd640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa138155ca0 0x7fa13814bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa12effd640 1 -- 172.21.15.46:0/2168517300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1380c0ed0 msgr2=0x7fa13814c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa12effd640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1380c0ed0 0x7fa13814c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa12effd640 1 -- 172.21.15.46:0/2168517300 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1280094d0 con 0x7fa138158090 2023-10-30T13:23:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f6b9cea9640 1 -- 172.21.15.46:0/3103299896 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b900a5820 con 0x7f6b980c4430 2023-10-30T13:23:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f6b9cea9640 1 -- 172.21.15.46:0/3103299896 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b900a5980 con 0x7f6b980c4430 2023-10-30T13:23:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f6b9cea9640 1 -- 172.21.15.46:0/3103299896 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b900a5820 con 0x7f6b980c4430 2023-10-30T13:23:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa8991ef640 1 -- 172.21.15.46:0/3992255075 >> v2:172.21.15.154:3300/0 conn(0x7fa8940cf160 msgr2=0x7fa8940cf5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T13:23:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa8991ef640 1 --2- 172.21.15.46:0/3992255075 >> v2:172.21.15.154:3300/0 conn(0x7fa8940cf160 0x7fa8940cf5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa8991ef640 1 -- 172.21.15.46:0/3992255075 >> v1:172.21.15.46:6789/0 conn(0x7fa894157f70 legacy=0x7fa894161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa8991ef640 1 -- 172.21.15.46:0/3992255075 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8940ca5d0 con 0x7fa8940cad80 2023-10-30T13:23:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa12effd640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa138158090 0x7fa13814b800 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa1280af000 tx=0x7fa12801bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936212255 0 0) 0x7f02c415a2e0 con 0x7f02c4159830 2023-10-30T13:23:28.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b0003530 con 0x7f02c4159830 2023-10-30T13:23:28.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa134ff9640 1 -- 172.21.15.46:0/2168517300 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1280b1e10 con 0x7fa138158090 2023-10-30T13:23:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa134ff9640 1 -- 172.21.15.46:0/2168517300 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1280b9440 con 0x7fa138158090 2023-10-30T13:23:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa134ff9640 1 -- 172.21.15.46:0/2168517300 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1280b97d0 con 0x7fa138158090 2023-10-30T13:23:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa13814b330 con 0x7fa138158090 2023-10-30T13:23:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1380caab0 con 0x7fa138158090 2023-10-30T13:23:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f6ba010f640 1 -- 172.21.15.46:0/3103299896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b980c4430 msgr2=0x7f6b980cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f6ba010f640 1 --2- 172.21.15.46:0/3103299896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b980c4430 0x7f6b980cc4b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009d90 tx=0x7f6b9001c800 comp rx=0 tx=0).stop 2023-10-30T13:23:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.460+0000 7f21ea59c640 1 -- 172.21.15.46:0/3628627623 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ec15dd30 con 0x7f21ec0c6b40 2023-10-30T13:23:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2626449325 0 0) 0x7f02b0003530 con 0x7f02c4159830 2023-10-30T13:23:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 >> v2:172.21.15.46:3300/0 conn(0x7f02c40c4430 msgr2=0x7f02c40ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 --2- 172.21.15.46:0/4117438794 >> v2:172.21.15.46:3300/0 conn(0x7f02c40c4430 0x7f02c40ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 >> v1:172.21.15.46:6789/0 conn(0x7f02c4153c60 legacy=0x7f02c40cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c415a5c0 con 0x7f02c4159830 2023-10-30T13:23:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f02bc001a00 con 0x7f02c4159830 2023-10-30T13:23:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 >> v1:172.21.15.154:6789/0 conn(0x7f02c4159830 legacy=0x7f02c40cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 --2- 172.21.15.46:0/4117438794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02b0003980 0x7f02b0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 --2- 172.21.15.46:0/4117438794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 0x7f02b0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 --2- 172.21.15.46:0/4117438794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b0005190 0x7f02b00141a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c4154040 con 0x7f02b00047d0 2023-10-30T13:23:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0014840 con 0x7f02b0003980 2023-10-30T13:23:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0014a00 con 0x7f02b0005190 2023-10-30T13:23:28.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fc7dffff640 1 -- 172.21.15.46:0/2611327727 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7e00b4070 con 0x7fc7e8153c60 2023-10-30T13:23:28.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f6ba010f640 1 -- 172.21.15.46:0/3103299896 shutdown_connections 2023-10-30T13:23:28.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f6ba010f640 1 --2- 172.21.15.46:0/3103299896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b980c4430 0x7f6b980cc4b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f6ba010f640 1 --2- 172.21.15.46:0/3103299896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b98153c60 0x7f6b980ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f6ba010f640 1 -- 172.21.15.46:0/3103299896 >> 172.21.15.46:0/3103299896 conn(0x7f6b9806bfe0 msgr2=0x7f6b9806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f6ba010f640 1 -- 172.21.15.46:0/3103299896 shutdown_connections 2023-10-30T13:23:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f6ba010f640 1 -- 172.21.15.46:0/3103299896 wait complete. 2023-10-30T13:23:28.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f21ea59c640 1 --2- 172.21.15.46:0/3628627623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21ec0c6b40 0x7f21ec069e90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f21e00097c0 tx=0x7f21e001c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac1a2551a0aa3bc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f3c3ff9d640 1 Processor -- start 2023-10-30T13:23:28.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fc7dffff640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d0066570 0x7fc7d0068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7fc7dffff640 1 -- 172.21.15.46:0/2611327727 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7e011f420 con 0x7fc7e8153c60 2023-10-30T13:23:28.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11c005260 con 0x7fa138158090 2023-10-30T13:23:28.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.464+0000 7f02c3fff640 1 --2- 172.21.15.46:0/4117438794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b0005190 0x7f02b00141a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f02baffd640 1 --2- 172.21.15.46:0/4117438794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 0x7f02b0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7fc7e77fe640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d0066570 0x7fc7d0068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f02bb7fe640 1 --2- 172.21.15.46:0/4117438794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02b0003980 0x7f02b0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f3c3ff9d640 1 -- start start 2023-10-30T13:23:28.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f21e9d9b640 1 -- 172.21.15.46:0/3628627623 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00ab600 con 0x7f21ec0c6b40 2023-10-30T13:23:28.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f21e9d9b640 1 -- 172.21.15.46:0/3628627623 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e00a7050 con 0x7f21ec0c6b40 2023-10-30T13:23:28.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f21e9d9b640 1 -- 172.21.15.46:0/3628627623 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00ab980 con 0x7f21ec0c6b40 2023-10-30T13:23:28.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f02baffd640 1 -- 172.21.15.46:0/4117438794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b0005190 msgr2=0x7f02b00141a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:28.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f02baffd640 1 --2- 172.21.15.46:0/4117438794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b0005190 0x7f02b00141a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f02baffd640 1 -- 172.21.15.46:0/4117438794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02b0003980 msgr2=0x7f02b0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:28.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f02baffd640 1 --2- 172.21.15.46:0/4117438794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02b0003980 0x7f02b0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f02baffd640 1 -- 172.21.15.46:0/4117438794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c415a5c0 con 0x7f02b00047d0 2023-10-30T13:23:28.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7fc7e77fe640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d0066570 0x7fc7d0068a30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8007fd0 tx=0x7fc7d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f02c3fff640 1 --2- 172.21.15.46:0/4117438794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b0005190 0x7f02b00141a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:28.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f02c3fff640 1 -- 172.21.15.46:0/4117438794 reap_dead start 2023-10-30T13:23:28.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f3c3ff9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c38158090 0x7f3c38161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f6ba010f640 1 Processor -- start 2023-10-30T13:23:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f3c3ff9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c380c6b40 0x7f3c38069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f6ba010f640 1 -- start start 2023-10-30T13:23:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f3c3ff9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3c3806a3d0 0x7f3c3806a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f21f1225640 1 -- 172.21.15.46:0/3628627623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21ec0c6b40 msgr2=0x7f21ec069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f21f1225640 1 --2- 172.21.15.46:0/3628627623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21ec0c6b40 0x7f21ec069e90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f21e00097c0 tx=0x7f21e001c3a0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f21f1225640 1 -- 172.21.15.46:0/3628627623 shutdown_connections 2023-10-30T13:23:28.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f21f1225640 1 --2- 172.21.15.46:0/3628627623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21ec0c6b40 0x7f21ec069e90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f21f1225640 1 --2- 172.21.15.46:0/3628627623 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21ec158090 0x7f21ec161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f21f1225640 1 -- 172.21.15.46:0/3628627623 >> 172.21.15.46:0/3628627623 conn(0x7f21ec0c7910 msgr2=0x7f21ec0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f21f1225640 1 -- 172.21.15.46:0/3628627623 shutdown_connections 2023-10-30T13:23:28.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f21f1225640 1 -- 172.21.15.46:0/3628627623 wait complete. 2023-10-30T13:23:28.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa8991ef640 1 --2- 172.21.15.46:0/3992255075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8940cad80 0x7fa8940cc400 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa8940cacd0 tx=0x7fa88801f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10785e385b027861 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f3c3dd39640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c38158090 0x7f3c38161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7f3c3dd39640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c38158090 0x7f3c38161be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39674/0 (socket says 172.21.15.46:39674) 2023-10-30T13:23:28.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f3c3ff9d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c38162120 con 0x7f3c38158090 2023-10-30T13:23:28.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f3c3ff9d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c3806ad60 con 0x7f3c3806a3d0 2023-10-30T13:23:28.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f3c3dd39640 1 -- 172.21.15.46:0/1879992865 learned_addr learned my addr 172.21.15.46:0/1879992865 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f3c3d538640 1 --2- 172.21.15.46:0/1879992865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c380c6b40 0x7f3c38069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f3c3ff9d640 1 -- 172.21.15.46:0/1879992865 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380c72b0 con 0x7f3c380c6b40 2023-10-30T13:23:28.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f3c3e53a640 1 --2- 172.21.15.46:0/1879992865 >> v2:172.21.15.154:3300/0 conn(0x7f3c3806a3d0 0x7f3c3806a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.468+0000 7fa134ff9640 1 -- 172.21.15.46:0/2168517300 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1280b7070 con 0x7fa138158090 2023-10-30T13:23:28.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa134ff9640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa118066570 0x7fa118068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa8989ee640 1 -- 172.21.15.46:0/3992255075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8880af400 con 0x7fa8940cad80 2023-10-30T13:23:28.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6ba010f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b980c4430 0x7f6b981541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6ba010f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b98153c60 0x7f6b981504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6ba010f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b98159830 0x7f6b981f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6ba010f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b98150a30 con 0x7f6b980c4430 2023-10-30T13:23:28.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6ba010f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b98157b10 con 0x7f6b98159830 2023-10-30T13:23:28.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6ba010f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b981f7510 con 0x7f6b98153c60 2023-10-30T13:23:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6b9d6aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b98153c60 0x7f6b981504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f3c3e53a640 1 -- 172.21.15.46:0/1879992865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c380c6b40 msgr2=0x7f3c38069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:28.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f3c3e53a640 1 --2- 172.21.15.46:0/1879992865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c380c6b40 0x7f3c38069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fc7dffff640 1 -- 172.21.15.46:0/2611327727 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7e00ec8a0 con 0x7fc7e8153c60 2023-10-30T13:23:28.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa134ff9640 1 -- 172.21.15.46:0/2168517300 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1281217f0 con 0x7fa138158090 2023-10-30T13:23:28.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa1377fe640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa118066570 0x7fa118068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6b9e6ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b98159830 0x7f6b981f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6b9d6aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b98153c60 0x7f6b981504f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59554/0 (socket says 172.21.15.46:59554) 2023-10-30T13:23:28.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6b9d6aa640 1 -- 172.21.15.46:0/3784879407 learned_addr learned my addr 172.21.15.46:0/3784879407 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6b9deab640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b980c4430 0x7f6b981541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f7f02bf8640 1 Processor -- start 2023-10-30T13:23:28.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f3c3e53a640 1 -- 172.21.15.46:0/1879992865 >> v2:172.21.15.46:3300/0 conn(0x7f3c38158090 msgr2=0x7f3c38161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:28.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f3c3e53a640 1 --2- 172.21.15.46:0/1879992865 >> v2:172.21.15.46:3300/0 conn(0x7f3c38158090 0x7f3c38161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f3c3e53a640 1 -- 172.21.15.46:0/1879992865 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c380c6fd0 con 0x7f3c3806a3d0 2023-10-30T13:23:28.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa8989ee640 1 -- 172.21.15.46:0/3992255075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8880ab050 con 0x7fa8940cad80 2023-10-30T13:23:28.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f7f02bf8640 1 -- start start 2023-10-30T13:23:28.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa8989ee640 1 -- 172.21.15.46:0/3992255075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8880af780 con 0x7fa8940cad80 2023-10-30T13:23:28.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6b9d6aa640 1 -- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b98159830 msgr2=0x7f6b981f6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6b9d6aa640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b98159830 0x7f6b981f6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6b9d6aa640 1 -- 172.21.15.46:0/3784879407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b980c4430 msgr2=0x7f6b981541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f02baffd640 1 --2- 172.21.15.46:0/4117438794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 0x7f02b0004c50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f02ac009b60 tx=0x7f02ac01cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b62f99868554fc72 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa89bc54640 1 -- 172.21.15.46:0/3992255075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8940cad80 msgr2=0x7fa8940cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa89bc54640 1 --2- 172.21.15.46:0/3992255075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8940cad80 0x7fa8940cc400 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa8940cacd0 tx=0x7fa88801f1a0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6b9d6aa640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b980c4430 0x7f6b981541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6b9d6aa640 1 -- 172.21.15.46:0/3784879407 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b90009a70 con 0x7f6b98153c60 2023-10-30T13:23:28.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6b9deab640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b980c4430 0x7f6b981541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:28.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f6b9d6aa640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b98153c60 0x7f6b981504f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6b9001aeb0 tx=0x7f6b900195a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f3c3dd39640 1 --2- 172.21.15.46:0/1879992865 >> v2:172.21.15.46:3300/0 conn(0x7f3c38158090 0x7f3c38161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f21f1225640 1 Processor -- start 2023-10-30T13:23:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa89bc54640 1 -- 172.21.15.46:0/3992255075 shutdown_connections 2023-10-30T13:23:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa89bc54640 1 --2- 172.21.15.46:0/3992255075 >> v2:172.21.15.154:3300/0 conn(0x7fa8940cf160 0x7fa8940cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa89bc54640 1 --2- 172.21.15.46:0/3992255075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8940cad80 0x7fa8940cc400 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa89bc54640 1 -- 172.21.15.46:0/3992255075 >> 172.21.15.46:0/3992255075 conn(0x7fa89406bfe0 msgr2=0x7fa89406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa1377fe640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa118066570 0x7fa118068a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa120000f70 tx=0x7fa120009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f21f1225640 1 -- start start 2023-10-30T13:23:28.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa89bc54640 1 -- 172.21.15.46:0/3992255075 shutdown_connections 2023-10-30T13:23:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac0a56d0 con 0x7f02b00047d0 2023-10-30T13:23:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ac0a5830 con 0x7f02b00047d0 2023-10-30T13:23:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7f02bf8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc158090 0x7f7efc161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7f02bf8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7efc0c6b40 0x7f7efc069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7f02bf8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc0c72b0 con 0x7f7efc06a3d0 2023-10-30T13:23:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7f02bf8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7efc162120 con 0x7f7efc0c6b40 2023-10-30T13:23:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7f02bf8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc0c6fd0 con 0x7f7efc158090 2023-10-30T13:23:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7f01195640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7efc06a3d0 0x7f7efc0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48516/0 (socket says 172.21.15.46:48516) 2023-10-30T13:23:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7f01195640 1 -- 172.21.15.46:0/1927658867 learned_addr learned my addr 172.21.15.46:0/1927658867 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7efbfff640 1 --2- 172.21.15.46:0/1927658867 >> v2:172.21.15.154:3300/0 conn(0x7f7efc0c6b40 0x7f7efc069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f02c37fe640 1 -- 172.21.15.46:0/4117438794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac0a5be0 con 0x7f02b00047d0 2023-10-30T13:23:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f02ca054640 1 -- 172.21.15.46:0/4117438794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 msgr2=0x7f02b0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f02ca054640 1 --2- 172.21.15.46:0/4117438794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 0x7f02b0004c50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f02ac009b60 tx=0x7f02ac01cb30 comp rx=0 tx=0).stop 2023-10-30T13:23:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7f00994640 1 --2- 172.21.15.46:0/1927658867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc158090 0x7f7efc161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21f1225640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21ec06a3d0 0x7f21ec1548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21f1225640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21ec158090 0x7f21ec14fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21f1225640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21ec150310 0x7f21ec156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21f1225640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec154e20 con 0x7f21ec06a3d0 2023-10-30T13:23:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21f1225640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec0c6410 con 0x7f21ec150310 2023-10-30T13:23:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21f1225640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec0c6940 con 0x7f21ec158090 2023-10-30T13:23:28.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7efbfff640 1 -- 172.21.15.46:0/1927658867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc158090 msgr2=0x7f7efc161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:28.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7efbfff640 1 --2- 172.21.15.46:0/1927658867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc158090 0x7f7efc161be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7efbfff640 1 -- 172.21.15.46:0/1927658867 >> v1:172.21.15.46:6789/0 conn(0x7f7efc06a3d0 legacy=0x7f7efc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f7efbfff640 1 -- 172.21.15.46:0/1927658867 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc15dd30 con 0x7f7efc0c6b40 2023-10-30T13:23:28.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21eb59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21ec150310 0x7f21ec156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21eb59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21ec150310 0x7f21ec156550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39424/0 (socket says 172.21.15.46:39424) 2023-10-30T13:23:28.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21eb59e640 1 -- 172.21.15.46:0/290873872 learned_addr learned my addr 172.21.15.46:0/290873872 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21ead9d640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21ec06a3d0 0x7f21ec1548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.472+0000 7fa89bc54640 1 -- 172.21.15.46:0/3992255075 wait complete. 2023-10-30T13:23:28.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7fa89bc54640 1 Processor -- start 2023-10-30T13:23:28.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21ea59c640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21ec158090 0x7f21ec14fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f02ca054640 1 -- 172.21.15.46:0/4117438794 shutdown_connections 2023-10-30T13:23:28.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f02ca054640 1 --2- 172.21.15.46:0/4117438794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 0x7f02b0004c50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f02ca054640 1 -- 172.21.15.46:0/4117438794 >> 172.21.15.46:0/4117438794 conn(0x7f02c406bfe0 msgr2=0x7f02c406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21eb59e640 1 -- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21ec158090 msgr2=0x7f21ec14fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21eb59e640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21ec158090 0x7f21ec14fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21eb59e640 1 -- 172.21.15.46:0/290873872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21ec06a3d0 msgr2=0x7f21ec1548e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21eb59e640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21ec06a3d0 0x7f21ec1548e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21eb59e640 1 -- 172.21.15.46:0/290873872 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e0009420 con 0x7f21ec150310 2023-10-30T13:23:28.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7fa89bc54640 1 -- start start 2023-10-30T13:23:28.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21eb59e640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21ec150310 0x7f21ec156550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009e10 tx=0x7f21dc00c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f02ca054640 1 -- 172.21.15.46:0/4117438794 shutdown_connections 2023-10-30T13:23:28.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f02ca054640 1 -- 172.21.15.46:0/4117438794 wait complete. 2023-10-30T13:23:28.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21d37fe640 1 -- 172.21.15.46:0/290873872 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc007af0 con 0x7f21ec150310 2023-10-30T13:23:28.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21d37fe640 1 -- 172.21.15.46:0/290873872 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc007c50 con 0x7f21ec150310 2023-10-30T13:23:28.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21d37fe640 1 -- 172.21.15.46:0/290873872 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00ce10 con 0x7f21ec150310 2023-10-30T13:23:28.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21f1225640 1 -- 172.21.15.46:0/290873872 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ec0c8c10 con 0x7f21ec150310 2023-10-30T13:23:28.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7f21f1225640 1 -- 172.21.15.46:0/290873872 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ec14f010 con 0x7f21ec150310 2023-10-30T13:23:28.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.476+0000 7fa89bc54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8940cad80 0x7fa8940b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89bc54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8940cf160 0x7fa8940b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89bc54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa894157f70 0x7fa8940c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89bc54640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940b3a30 con 0x7fa8940cad80 2023-10-30T13:23:28.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89bc54640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89413c920 con 0x7fa8940cf160 2023-10-30T13:23:28.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89bc54640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89413a3e0 con 0x7fa894157f70 2023-10-30T13:23:28.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89a1f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa894157f70 0x7fa8940c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89a1f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa894157f70 0x7fa8940c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59582/0 (socket says 172.21.15.46:59582) 2023-10-30T13:23:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89a1f1640 1 -- 172.21.15.46:0/397905021 learned_addr learned my addr 172.21.15.46:0/397905021 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f3c3e53a640 1 --2- 172.21.15.46:0/1879992865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c3806a3d0 0x7f3c3806a820 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3c380c5480 tx=0x7f3c3001f110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bda2daa0a6f66540 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f21f1225640 1 -- 172.21.15.46:0/290873872 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ec04e170 con 0x7f21ec150310 2023-10-30T13:23:28.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa8999f0640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8940cad80 0x7fa8940b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa8991ef640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8940cf160 0x7fa8940b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89a1f1640 1 -- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8940cf160 msgr2=0x7fa8940b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89a1f1640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8940cf160 0x7fa8940b3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89a1f1640 1 -- 172.21.15.46:0/397905021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8940cad80 msgr2=0x7fa8940b34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89a1f1640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8940cad80 0x7fa8940b34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89a1f1640 1 -- 172.21.15.46:0/397905021 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa888009b30 con 0x7fa894157f70 2023-10-30T13:23:28.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa8999f0640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8940cad80 0x7fa8940b34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:28.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa8991ef640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8940cf160 0x7fa8940b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:28.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f02ca054640 1 Processor -- start 2023-10-30T13:23:28.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f02ca054640 1 -- start start 2023-10-30T13:23:28.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f3c3cd37640 1 -- 172.21.15.46:0/1879992865 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c300b0440 con 0x7f3c3806a3d0 2023-10-30T13:23:28.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f3c3cd37640 1 -- 172.21.15.46:0/1879992865 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c300b05a0 con 0x7f3c3806a3d0 2023-10-30T13:23:28.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f3c3cd37640 1 -- 172.21.15.46:0/1879992865 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c300b0930 con 0x7f3c3806a3d0 2023-10-30T13:23:28.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7fa89a1f1640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa894157f70 0x7fa8940c8760 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa89000b470 tx=0x7fa89000b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f7efbfff640 1 --2- 172.21.15.46:0/1927658867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc0c6b40 0x7f7efc069e90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009d90 tx=0x7f7eec01f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2ae5fa3e15b55e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f3c3ff9d640 1 -- 172.21.15.46:0/1879992865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c3806a3d0 msgr2=0x7f3c3806a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f3c3ff9d640 1 --2- 172.21.15.46:0/1879992865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c3806a3d0 0x7f3c3806a820 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3c380c5480 tx=0x7f3c3001f110 comp rx=0 tx=0).stop 2023-10-30T13:23:28.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f3c3ff9d640 1 -- 172.21.15.46:0/1879992865 shutdown_connections 2023-10-30T13:23:28.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f3c3ff9d640 1 --2- 172.21.15.46:0/1879992865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c3806a3d0 0x7f3c3806a820 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f3c3ff9d640 1 --2- 172.21.15.46:0/1879992865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c380c6b40 0x7f3c38069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f3c3ff9d640 1 --2- 172.21.15.46:0/1879992865 >> v2:172.21.15.46:3300/0 conn(0x7f3c38158090 0x7f3c38161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f7efb7fe640 1 -- 172.21.15.46:0/1927658867 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec0af600 con 0x7f7efc0c6b40 2023-10-30T13:23:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f3c3ff9d640 1 -- 172.21.15.46:0/1879992865 >> 172.21.15.46:0/1879992865 conn(0x7f3c380c7910 msgr2=0x7f3c380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f7efb7fe640 1 -- 172.21.15.46:0/1927658867 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eec0ab050 con 0x7f7efc0c6b40 2023-10-30T13:23:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f7efb7fe640 1 -- 172.21.15.46:0/1927658867 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec0af980 con 0x7f7efc0c6b40 2023-10-30T13:23:28.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f21d37fe640 1 -- 172.21.15.46:0/290873872 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f21dc009070 con 0x7f21ec150310 2023-10-30T13:23:28.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f02ca054640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 0x7f02c41509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f02ca054640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02c41556a0 0x7f02c41497f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f02ca054640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c40c5840 0x7f02c4149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f02ca054640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c414e5e0 con 0x7f02b00047d0 2023-10-30T13:23:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f02ca054640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c4150f00 con 0x7f02c40c5840 2023-10-30T13:23:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f02ca054640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c414a270 con 0x7f02c41556a0 2023-10-30T13:23:28.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f7f02bf8640 1 -- 172.21.15.46:0/1927658867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc0c6b40 msgr2=0x7f7efc069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f7f02bf8640 1 --2- 172.21.15.46:0/1927658867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc0c6b40 0x7f7efc069e90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009d90 tx=0x7f7eec01f130 comp rx=0 tx=0).stop 2023-10-30T13:23:28.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f3c3ff9d640 1 -- 172.21.15.46:0/1879992865 shutdown_connections 2023-10-30T13:23:28.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f02c3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c40c5840 0x7f02c4149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f02bb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 0x7f02c41509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f02bb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 0x7f02c41509c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39708/0 (socket says 172.21.15.46:39708) 2023-10-30T13:23:28.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.480+0000 7f02bb7fe640 1 -- 172.21.15.46:0/2774297797 learned_addr learned my addr 172.21.15.46:0/2774297797 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f7f02bf8640 1 -- 172.21.15.46:0/1927658867 shutdown_connections 2023-10-30T13:23:28.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f7f02bf8640 1 --2- 172.21.15.46:0/1927658867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc0c6b40 0x7f7efc069e90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f7f02bf8640 1 --2- 172.21.15.46:0/1927658867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc158090 0x7f7efc161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f7f02bf8640 1 -- 172.21.15.46:0/1927658867 >> 172.21.15.46:0/1927658867 conn(0x7f7efc0c7910 msgr2=0x7f7efc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02baffd640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02c41556a0 0x7f02c41497f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f7f02bf8640 1 -- 172.21.15.46:0/1927658867 shutdown_connections 2023-10-30T13:23:28.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f21d37fe640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21d80664d0 0x7f21d8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02bb7fe640 1 -- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02c41556a0 msgr2=0x7f02c41497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f21d37fe640 1 -- 172.21.15.46:0/290873872 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21dc085e40 con 0x7f21ec150310 2023-10-30T13:23:28.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f7f02bf8640 1 -- 172.21.15.46:0/1927658867 wait complete. 2023-10-30T13:23:28.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02bb7fe640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02c41556a0 0x7f02c41497f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02bb7fe640 1 -- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c40c5840 msgr2=0x7f02c4149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02bb7fe640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c40c5840 0x7f02c4149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02bb7fe640 1 -- 172.21.15.46:0/2774297797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c414e020 con 0x7f02b00047d0 2023-10-30T13:23:28.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f3c3ff9d640 1 -- 172.21.15.46:0/1879992865 wait complete. 2023-10-30T13:23:28.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f21ead9d640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21d80664d0 0x7f21d8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02bb7fe640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 0x7f02c41509c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f02b4001580 tx=0x7f02b4001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02c17fa640 1 -- 172.21.15.46:0/2774297797 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b4001d80 con 0x7f02b00047d0 2023-10-30T13:23:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02c17fa640 1 -- 172.21.15.46:0/2774297797 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b40047f0 con 0x7f02b00047d0 2023-10-30T13:23:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02ca054640 1 -- 172.21.15.46:0/2774297797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c414e300 con 0x7f02b00047d0 2023-10-30T13:23:28.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02c17fa640 1 -- 172.21.15.46:0/2774297797 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b4004b80 con 0x7f02b00047d0 2023-10-30T13:23:28.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02ca054640 1 -- 172.21.15.46:0/2774297797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c41501c0 con 0x7f02b00047d0 2023-10-30T13:23:28.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f21ead9d640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21d80664d0 0x7f21d8068990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f21e4011340 tx=0x7f21e400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f3c3ff9d640 1 Processor -- start 2023-10-30T13:23:28.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f3c3ff9d640 1 -- start start 2023-10-30T13:23:28.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f7f02bf8640 1 Processor -- start 2023-10-30T13:23:28.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f02ca054640 1 -- 172.21.15.46:0/2774297797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a8005260 con 0x7f02b00047d0 2023-10-30T13:23:28.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f7f02bf8640 1 -- start start 2023-10-30T13:23:28.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f6b8e7fc640 1 -- 172.21.15.46:0/3784879407 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b90004450 con 0x7f6b98153c60 2023-10-30T13:23:28.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f3c3ff9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c3806a3d0 0x7f3c380c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f3c3ff9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c38158090 0x7f3c3814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f3c3ff9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c3815ad30 0x7f3c3814c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f3c3ff9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c38158590 con 0x7f3c38158090 2023-10-30T13:23:28.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f3c3ff9d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c38160570 con 0x7f3c3815ad30 2023-10-30T13:23:28.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.484+0000 7f3c3ff9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3815ef20 con 0x7f3c3806a3d0 2023-10-30T13:23:28.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7fa134ff9640 1 -- 172.21.15.46:0/2168517300 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa1280eebc0 con 0x7fa138158090 2023-10-30T13:23:28.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f6b8e7fc640 1 -- 172.21.15.46:0/3784879407 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b900045b0 con 0x7f6b98153c60 2023-10-30T13:23:28.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7fa8867fc640 1 -- 172.21.15.46:0/397905021 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890011890 con 0x7fa894157f70 2023-10-30T13:23:28.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7fa8867fc640 1 -- 172.21.15.46:0/397905021 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8900119f0 con 0x7fa894157f70 2023-10-30T13:23:28.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f6b8e7fc640 1 -- 172.21.15.46:0/3784879407 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b90004450 con 0x7f6b98153c60 2023-10-30T13:23:28.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7fa8867fc640 1 -- 172.21.15.46:0/397905021 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890011da0 con 0x7fa894157f70 2023-10-30T13:23:28.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7f02bf8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc06a3d0 0x7f7efc0c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7f02bf8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7efc158090 0x7f7efc0ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7f02bf8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc1eac30 0x7f7efc155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7f02bf8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc0c9bf0 con 0x7f7efc158090 2023-10-30T13:23:28.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7f02bf8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc154dc0 con 0x7f7efc1eac30 2023-10-30T13:23:28.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7f02bf8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc0c08b0 con 0x7f7efc06a3d0 2023-10-30T13:23:28.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f3c3d538640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c38158090 0x7f3c3814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f3c3d538640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c38158090 0x7f3c3814c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39710/0 (socket says 172.21.15.46:39710) 2023-10-30T13:23:28.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f3c3d538640 1 -- 172.21.15.46:0/2094021460 learned_addr learned my addr 172.21.15.46:0/2094021460 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f3c3e53a640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c3815ad30 0x7f3c3814c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f6ba010f640 1 -- 172.21.15.46:0/3784879407 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9814fa10 con 0x7f6b98153c60 2023-10-30T13:23:28.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7f01195640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc1eac30 0x7f7efc155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7f01195640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc1eac30 0x7f7efc155250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39460/0 (socket says 172.21.15.46:39460) 2023-10-30T13:23:28.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7f01195640 1 -- 172.21.15.46:0/2034886513 learned_addr learned my addr 172.21.15.46:0/2034886513 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:28.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f3c3e53a640 1 -- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c3806a3d0 msgr2=0x7f3c380c5760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T13:23:28.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f3c3e53a640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c3806a3d0 0x7f3c380c5760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f3c3e53a640 1 -- 172.21.15.46:0/2094021460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c38158090 msgr2=0x7f3c3814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f3c3e53a640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c38158090 0x7f3c3814c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f3c3e53a640 1 -- 172.21.15.46:0/2094021460 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c340094d0 con 0x7f3c3815ad30 2023-10-30T13:23:28.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7efbfff640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7efc158090 0x7f7efc0ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7f00994640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc06a3d0 0x7f7efc0c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f6ba010f640 1 -- 172.21.15.46:0/3784879407 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9814ffd0 con 0x7f6b98153c60 2023-10-30T13:23:28.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f3c3e53a640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c3815ad30 0x7f3c3814c800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3c300af000 tx=0x7f3c3001beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7fa89bc54640 1 -- 172.21.15.46:0/397905021 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89413a870 con 0x7fa894157f70 2023-10-30T13:23:28.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7fa89bc54640 1 -- 172.21.15.46:0/397905021 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa894139ab0 con 0x7fa894157f70 2023-10-30T13:23:28.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7efbfff640 1 -- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc06a3d0 msgr2=0x7f7efc0c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7efbfff640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc06a3d0 0x7f7efc0c96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7efbfff640 1 -- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc1eac30 msgr2=0x7f7efc155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7efbfff640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc1eac30 0x7f7efc155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7efbfff640 1 -- 172.21.15.46:0/2034886513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec009a70 con 0x7f7efc158090 2023-10-30T13:23:28.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7f01195640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc1eac30 0x7f7efc155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:28.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f7efbfff640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7efc158090 0x7f7efc0ca4b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0af000 tx=0x7f7eec01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f6ba010f640 1 -- 172.21.15.46:0/3784879407 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b9804e170 con 0x7f6b98153c60 2023-10-30T13:23:28.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7fa89bc54640 1 -- 172.21.15.46:0/397905021 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89404e170 con 0x7fa894157f70 2023-10-30T13:23:28.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.488+0000 7f02c17fa640 1 -- 172.21.15.46:0/2774297797 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02b400e740 con 0x7f02b00047d0 2023-10-30T13:23:28.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f02c17fa640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02a4066570 0x7f02a4068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f02c17fa640 1 -- 172.21.15.46:0/2774297797 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02b4083a80 con 0x7f02b00047d0 2023-10-30T13:23:28.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f02baffd640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02a4066570 0x7f02a4068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f6b8e7fc640 1 -- 172.21.15.46:0/3784879407 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b90004720 con 0x7f6b98153c60 2023-10-30T13:23:28.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f02baffd640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02a4066570 0x7f02a4068a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0af000 tx=0x7f02ac0046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f6b8e7fc640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b7c066590 0x7f6b7c068a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f6b8e7fc640 1 -- 172.21.15.46:0/3784879407 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b9011fd50 con 0x7f6b98153c60 2023-10-30T13:23:28.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f6b9deab640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b7c066590 0x7f6b7c068a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f7ef97fa640 1 -- 172.21.15.46:0/2034886513 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec0c1070 con 0x7f7efc158090 2023-10-30T13:23:28.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f7ef97fa640 1 -- 172.21.15.46:0/2034886513 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eec0b9410 con 0x7f7efc158090 2023-10-30T13:23:28.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f7ef97fa640 1 -- 172.21.15.46:0/2034886513 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec0b97a0 con 0x7f7efc158090 2023-10-30T13:23:28.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f7f02bf8640 1 -- 172.21.15.46:0/2034886513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7efc0c0b50 con 0x7f7efc158090 2023-10-30T13:23:28.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.492+0000 7f02c17fa640 1 -- 172.21.15.46:0/2774297797 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f02b4050f90 con 0x7f02b00047d0 2023-10-30T13:23:28.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.496+0000 7fa8867fc640 1 -- 172.21.15.46:0/397905021 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa89001a400 con 0x7fa894157f70 2023-10-30T13:23:28.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.496+0000 7f7f02bf8640 1 -- 172.21.15.46:0/2034886513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7efc154770 con 0x7f7efc158090 2023-10-30T13:23:28.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.496+0000 7fa8867fc640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa878066570 0x7fa878068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.496+0000 7fa8867fc640 1 -- 172.21.15.46:0/397905021 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa890084780 con 0x7fa894157f70 2023-10-30T13:23:28.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.496+0000 7f7f02bf8640 1 -- 172.21.15.46:0/2034886513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7efc04e170 con 0x7f7efc158090 2023-10-30T13:23:28.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.496+0000 7fa8999f0640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa878066570 0x7fa878068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.496+0000 7f6b9deab640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b7c066590 0x7f6b7c068a50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009d50 tx=0x7f6b880090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.496+0000 7f7ef97fa640 1 -- 172.21.15.46:0/2034886513 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7eec0b6050 con 0x7f7efc158090 2023-10-30T13:23:28.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.496+0000 7fa8999f0640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa878066570 0x7fa878068a30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa88c0098b0 tx=0x7fa88c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.496+0000 7f7ef97fa640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7edc0664a0 0x7f7edc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.496+0000 7f7ef97fa640 1 -- 172.21.15.46:0/2034886513 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7eec121c00 con 0x7f7efc158090 2023-10-30T13:23:28.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.496+0000 7f3c2e7fc640 1 -- 172.21.15.46:0/2094021460 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c300c2070 con 0x7f3c3815ad30 2023-10-30T13:23:28.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.500+0000 7f7f00994640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7edc0664a0 0x7f7edc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.500+0000 7f3c2e7fc640 1 -- 172.21.15.46:0/2094021460 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c300ba410 con 0x7f3c3815ad30 2023-10-30T13:23:28.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.500+0000 7f3c3ff9d640 1 -- 172.21.15.46:0/2094021460 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c30009a70 con 0x7f3c3815ad30 2023-10-30T13:23:28.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.500+0000 7f21d37fe640 1 -- 172.21.15.46:0/290873872 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f21dc053240 con 0x7f21ec150310 2023-10-30T13:23:28.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.500+0000 7f3c2e7fc640 1 -- 172.21.15.46:0/2094021460 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c300b0440 con 0x7f3c3815ad30 2023-10-30T13:23:28.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.500+0000 7f3c3ff9d640 1 -- 172.21.15.46:0/2094021460 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c3815f4e0 con 0x7f3c3815ad30 2023-10-30T13:23:28.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.500+0000 7f3c3ff9d640 1 -- 172.21.15.46:0/2094021460 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3804e170 con 0x7f3c3815ad30 2023-10-30T13:23:28.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.500+0000 7f7f00994640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7edc0664a0 0x7f7edc068960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4006fd0 tx=0x7f7ee4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.500+0000 7f3c2e7fc640 1 -- 172.21.15.46:0/2094021460 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3c300b7050 con 0x7f3c3815ad30 2023-10-30T13:23:28.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.504+0000 7f7ef97fa640 1 -- 172.21.15.46:0/2034886513 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7eec0ef0c0 con 0x7f7efc158090 2023-10-30T13:23:28.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.504+0000 7f3c2e7fc640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3c1c0664d0 0x7f3c1c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:28.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.504+0000 7f3c2e7fc640 1 -- 172.21.15.46:0/2094021460 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c30121eb0 con 0x7f3c3815ad30 2023-10-30T13:23:28.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.504+0000 7f3c3dd39640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3c1c0664d0 0x7f3c1c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:28.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.504+0000 7f6b8e7fc640 1 -- 172.21.15.46:0/3784879407 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6b900ed2a0 con 0x7f6b98153c60 2023-10-30T13:23:28.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.504+0000 7f3c3dd39640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3c1c0664d0 0x7f3c1c068990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009af0 tx=0x7f3c34009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:28.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.516+0000 7f3c2e7fc640 1 -- 172.21.15.46:0/2094021460 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3c300ef440 con 0x7f3c3815ad30 2023-10-30T13:23:28.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.520+0000 7fa8867fc640 1 -- 172.21.15.46:0/397905021 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa890051c70 con 0x7fa894157f70 2023-10-30T13:23:28.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.812+0000 7fc7ddffb640 1 -- 172.21.15.46:0/2611327727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc7e80b34f0 con 0x7fc7e8153c60 2023-10-30T13:23:28.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.820+0000 7fc7dffff640 1 -- 172.21.15.46:0/2611327727 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc7e80b34f0 con 0x7fc7e8153c60 2023-10-30T13:23:28.828 INFO:teuthology.orchestra.run.smithi046.stdout:55834574853 2023-10-30T13:23:28.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.820+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa11c005510 con 0x7fa138158090 2023-10-30T13:23:28.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.824+0000 7fc7ddffb640 1 -- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d0066570 msgr2=0x7fc7d0068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.824+0000 7fc7ddffb640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d0066570 0x7fc7d0068a30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8007fd0 tx=0x7fc7d8009040 comp rx=0 tx=0).stop 2023-10-30T13:23:28.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.824+0000 7fc7ddffb640 1 -- 172.21.15.46:0/2611327727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e8153c60 msgr2=0x7fc7e80c8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.824+0000 7fc7ddffb640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e8153c60 0x7fc7e80c8cb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009d90 tx=0x7fc7e0019680 comp rx=0 tx=0).stop 2023-10-30T13:23:28.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.828+0000 7fa134ff9640 1 -- 172.21.15.46:0/2168517300 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa1280b5020 con 0x7fa138158090 2023-10-30T13:23:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.828+0000 7fc7ddffb640 1 -- 172.21.15.46:0/2611327727 shutdown_connections 2023-10-30T13:23:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.828+0000 7fc7ddffb640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d0066570 0x7fc7d0068a30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.828+0000 7fc7ddffb640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e8153c60 0x7fc7e80c8cb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.828+0000 7fc7ddffb640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e80ca360 0x7fc7e80b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.828+0000 7fc7ddffb640 1 --2- 172.21.15.46:0/2611327727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e80c4430 0x7fc7e80ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.828+0000 7fc7ddffb640 1 -- 172.21.15.46:0/2611327727 >> 172.21.15.46:0/2611327727 conn(0x7fc7e806bfe0 msgr2=0x7fc7e80c4880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.828+0000 7fc7ddffb640 1 -- 172.21.15.46:0/2611327727 shutdown_connections 2023-10-30T13:23:28.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.828+0000 7fc7ddffb640 1 -- 172.21.15.46:0/2611327727 wait complete. 2023-10-30T13:23:28.836 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-10-30T13:23:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.832+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa118066570 msgr2=0x7fa118068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.832+0000 7fa13e0d6640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa118066570 0x7fa118068a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa120000f70 tx=0x7fa120009040 comp rx=0 tx=0).stop 2023-10-30T13:23:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.832+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa138158090 msgr2=0x7fa13814b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.832+0000 7fa13e0d6640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa138158090 0x7fa13814b800 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa1280af000 tx=0x7fa12801bab0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.832+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 shutdown_connections 2023-10-30T13:23:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.832+0000 7fa13e0d6640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa118066570 0x7fa118068a30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.832+0000 7fa13e0d6640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa138155ca0 0x7fa13814bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.832+0000 7fa13e0d6640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa138158090 0x7fa13814b800 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.832+0000 7fa13e0d6640 1 --2- 172.21.15.46:0/2168517300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1380c0ed0 0x7fa13814c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.832+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 >> 172.21.15.46:0/2168517300 conn(0x7fa1380c7910 msgr2=0x7fa13815c930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.832+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 shutdown_connections 2023-10-30T13:23:28.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.832+0000 7fa13e0d6640 1 -- 172.21.15.46:0/2168517300 wait complete. 2023-10-30T13:23:28.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.836+0000 7f02ca054640 1 -- 172.21.15.46:0/2774297797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f02a80050d0 con 0x7f02b00047d0 2023-10-30T13:23:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.836+0000 7f02c17fa640 1 -- 172.21.15.46:0/2774297797 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f02b4019070 con 0x7f02b00047d0 2023-10-30T13:23:28.842 INFO:teuthology.orchestra.run.smithi046.stdout:55834574853 2023-10-30T13:23:28.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.836+0000 7f6ba010f640 1 -- 172.21.15.46:0/3784879407 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6b9804e170 con 0x7f6b98153c60 2023-10-30T13:23:28.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.836+0000 7f7f02bf8640 1 -- 172.21.15.46:0/2034886513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7efc0b0860 con 0x7f7efc158090 2023-10-30T13:23:28.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f02ca054640 1 -- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02a4066570 msgr2=0x7f02a4068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f02ca054640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02a4066570 0x7f02a4068a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0af000 tx=0x7f02ac0046d0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f02ca054640 1 -- 172.21.15.46:0/2774297797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 msgr2=0x7f02c41509c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f02ca054640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 0x7f02c41509c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f02b4001580 tx=0x7f02b4001a40 comp rx=0 tx=0).stop 2023-10-30T13:23:28.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f6b8e7fc640 1 -- 172.21.15.46:0/3784879407 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6b900b2020 con 0x7f6b98153c60 2023-10-30T13:23:28.847 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-10-30T13:23:28.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f7ef97fa640 1 -- 172.21.15.46:0/2034886513 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7eec0b4070 con 0x7f7efc158090 2023-10-30T13:23:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f02ca054640 1 -- 172.21.15.46:0/2774297797 shutdown_connections 2023-10-30T13:23:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f02ca054640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02a4066570 0x7f02a4068a30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f02ca054640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c40c5840 0x7f02c4149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f02ca054640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02c41556a0 0x7f02c41497f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f02ca054640 1 --2- 172.21.15.46:0/2774297797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b00047d0 0x7f02c41509c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f02ca054640 1 -- 172.21.15.46:0/2774297797 >> 172.21.15.46:0/2774297797 conn(0x7f02c406bfe0 msgr2=0x7f02c4158760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f21f1225640 1 -- 172.21.15.46:0/290873872 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f21ec157000 con 0x7f21ec150310 2023-10-30T13:23:28.851 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-10-30T13:23:28.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f21d37fe640 1 -- 172.21.15.46:0/290873872 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f21dc019070 con 0x7f21ec150310 2023-10-30T13:23:28.852 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-10-30T13:23:28.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f02ca054640 1 -- 172.21.15.46:0/2774297797 shutdown_connections 2023-10-30T13:23:28.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.840+0000 7f02ca054640 1 -- 172.21.15.46:0/2774297797 wait complete. 2023-10-30T13:23:28.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f6ba010f640 1 -- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b7c066590 msgr2=0x7f6b7c068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f6ba010f640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b7c066590 0x7f6b7c068a50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009d50 tx=0x7f6b880090e0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f6ba010f640 1 -- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b98153c60 msgr2=0x7f6b981504f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f6ba010f640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b98153c60 0x7f6b981504f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6b9001aeb0 tx=0x7f6b900195a0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f7f02bf8640 1 -- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7edc0664a0 msgr2=0x7f7edc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f7f02bf8640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7edc0664a0 0x7f7edc068960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4006fd0 tx=0x7f7ee4008040 comp rx=0 tx=0).stop 2023-10-30T13:23:28.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f6ba010f640 1 -- 172.21.15.46:0/3784879407 shutdown_connections 2023-10-30T13:23:28.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f6ba010f640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b7c066590 0x7f6b7c068a50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f6ba010f640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b98159830 0x7f6b981f6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f6ba010f640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b98153c60 0x7f6b981504f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f6ba010f640 1 --2- 172.21.15.46:0/3784879407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b980c4430 0x7f6b981541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f6ba010f640 1 -- 172.21.15.46:0/3784879407 >> 172.21.15.46:0/3784879407 conn(0x7f6b9806bfe0 msgr2=0x7f6b980c56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f6ba010f640 1 -- 172.21.15.46:0/3784879407 shutdown_connections 2023-10-30T13:23:28.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f6ba010f640 1 -- 172.21.15.46:0/3784879407 wait complete. 2023-10-30T13:23:28.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f7f02bf8640 1 -- 172.21.15.46:0/2034886513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7efc158090 msgr2=0x7f7efc0ca4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f7f02bf8640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7efc158090 0x7f7efc0ca4b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0af000 tx=0x7f7eec01bed0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f7f02bf8640 1 -- 172.21.15.46:0/2034886513 shutdown_connections 2023-10-30T13:23:28.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f7f02bf8640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7edc0664a0 0x7f7edc068960 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f7f02bf8640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc1eac30 0x7f7efc155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f7f02bf8640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7efc158090 0x7f7efc0ca4b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f7f02bf8640 1 --2- 172.21.15.46:0/2034886513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc06a3d0 0x7f7efc0c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f7f02bf8640 1 -- 172.21.15.46:0/2034886513 >> 172.21.15.46:0/2034886513 conn(0x7f7efc0c7910 msgr2=0x7f7efc155f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f7f02bf8640 1 -- 172.21.15.46:0/2034886513 shutdown_connections 2023-10-30T13:23:28.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f7f02bf8640 1 -- 172.21.15.46:0/2034886513 wait complete. 2023-10-30T13:23:28.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f21f1225640 1 -- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21d80664d0 msgr2=0x7f21d8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f21f1225640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21d80664d0 0x7f21d8068990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f21e4011340 tx=0x7f21e400b040 comp rx=0 tx=0).stop 2023-10-30T13:23:28.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f21f1225640 1 -- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21ec150310 msgr2=0x7f21ec156550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f21f1225640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21ec150310 0x7f21ec156550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009e10 tx=0x7f21dc00c430 comp rx=0 tx=0).stop 2023-10-30T13:23:28.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f21f1225640 1 -- 172.21.15.46:0/290873872 shutdown_connections 2023-10-30T13:23:28.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f21f1225640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21d80664d0 0x7f21d8068990 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f21f1225640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21ec150310 0x7f21ec156550 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f21f1225640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21ec158090 0x7f21ec14fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f21f1225640 1 --2- 172.21.15.46:0/290873872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21ec06a3d0 0x7f21ec1548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f21f1225640 1 -- 172.21.15.46:0/290873872 >> 172.21.15.46:0/290873872 conn(0x7f21ec0c7910 msgr2=0x7f21ec0c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f21f1225640 1 -- 172.21.15.46:0/290873872 shutdown_connections 2023-10-30T13:23:28.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.844+0000 7f21f1225640 1 -- 172.21.15.46:0/290873872 wait complete. 2023-10-30T13:23:28.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.848+0000 7f3c3ff9d640 1 -- 172.21.15.46:0/2094021460 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3c3815dd00 con 0x7f3c3815ad30 2023-10-30T13:23:28.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.852+0000 7f3c2e7fc640 1 -- 172.21.15.46:0/2094021460 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3c300b5020 con 0x7f3c3815ad30 2023-10-30T13:23:28.863 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-10-30T13:23:28.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7f3c1bfff640 1 -- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3c1c0664d0 msgr2=0x7f3c1c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7f3c1bfff640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3c1c0664d0 0x7f3c1c068990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009af0 tx=0x7f3c34009240 comp rx=0 tx=0).stop 2023-10-30T13:23:28.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7f3c1bfff640 1 -- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c3815ad30 msgr2=0x7f3c3814c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7f3c1bfff640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c3815ad30 0x7f3c3814c800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3c300af000 tx=0x7f3c3001beb0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7f3c1bfff640 1 -- 172.21.15.46:0/2094021460 shutdown_connections 2023-10-30T13:23:28.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7f3c1bfff640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3c1c0664d0 0x7f3c1c068990 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009af0 tx=0x7f3c34009240 comp rx=0 tx=0).stop 2023-10-30T13:23:28.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7f3c1bfff640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c3815ad30 0x7f3c3814c800 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7f3c1bfff640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c38158090 0x7f3c3814c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7f3c1bfff640 1 --2- 172.21.15.46:0/2094021460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c3806a3d0 0x7f3c380c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7f3c1bfff640 1 -- 172.21.15.46:0/2094021460 >> 172.21.15.46:0/2094021460 conn(0x7f3c380c7910 msgr2=0x7f3c380bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7fa89bc54640 1 -- 172.21.15.46:0/397905021 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa894156aa0 con 0x7fa894157f70 2023-10-30T13:23:28.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7f3c1bfff640 1 -- 172.21.15.46:0/2094021460 shutdown_connections 2023-10-30T13:23:28.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7f3c1bfff640 1 -- 172.21.15.46:0/2094021460 wait complete. 2023-10-30T13:23:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.856+0000 7fa8867fc640 1 -- 172.21.15.46:0/397905021 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa890055290 con 0x7fa894157f70 2023-10-30T13:23:28.868 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-10-30T13:23:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.860+0000 7fa89bc54640 1 -- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa878066570 msgr2=0x7fa878068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.860+0000 7fa89bc54640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa878066570 0x7fa878068a30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa88c0098b0 tx=0x7fa88c008040 comp rx=0 tx=0).stop 2023-10-30T13:23:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.860+0000 7fa89bc54640 1 -- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa894157f70 msgr2=0x7fa8940c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.860+0000 7fa89bc54640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa894157f70 0x7fa8940c8760 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa89000b470 tx=0x7fa89000b930 comp rx=0 tx=0).stop 2023-10-30T13:23:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.860+0000 7fa89bc54640 1 -- 172.21.15.46:0/397905021 shutdown_connections 2023-10-30T13:23:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.860+0000 7fa89bc54640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa878066570 0x7fa878068a30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.860+0000 7fa89bc54640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa894157f70 0x7fa8940c8760 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.860+0000 7fa89bc54640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8940cf160 0x7fa8940b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.860+0000 7fa89bc54640 1 --2- 172.21.15.46:0/397905021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8940cad80 0x7fa8940b34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:28.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.860+0000 7fa89bc54640 1 -- 172.21.15.46:0/397905021 >> 172.21.15.46:0/397905021 conn(0x7fa89406bfe0 msgr2=0x7fa8940cf5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:28.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.860+0000 7fa89bc54640 1 -- 172.21.15.46:0/397905021 shutdown_connections 2023-10-30T13:23:28.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:28.860+0000 7fa89bc54640 1 -- 172.21.15.46:0/397905021 wait complete. 2023-10-30T13:23:28.872 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2023-10-30T13:23:28.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2023-10-30T13:23:28.898 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2023-10-30T13:23:28.899 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2023-10-30T13:23:28.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.7 2023-10-30T13:23:28.901 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2023-10-30T13:23:28.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2023-10-30T13:23:28.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2023-10-30T13:23:29.874 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T13:23:29.881 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T13:23:29.899 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T13:23:29.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T13:23:29.901 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T13:23:29.902 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T13:23:29.902 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T13:23:29.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T13:23:30.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.116+0000 7f05bf7aa640 1 Processor -- start 2023-10-30T13:23:30.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.116+0000 7f05bf7aa640 1 -- start start 2023-10-30T13:23:30.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.120+0000 7f05bf7aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05b8156020 0x7f05b806d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.120+0000 7f05bf7aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b806d620 0x7f05b815ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.120+0000 7f05bf7aa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b80c6090 con 0x7f05b814c140 2023-10-30T13:23:30.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.120+0000 7f05bf7aa640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b80c6370 con 0x7f05b8156020 2023-10-30T13:23:30.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.120+0000 7f05bf7aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b806da30 con 0x7f05b806d620 2023-10-30T13:23:30.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.120+0000 7f05bdd47640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b806d620 0x7f05b815ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.120+0000 7f05bdd47640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b806d620 0x7f05b815ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59602/0 (socket says 172.21.15.46:59602) 2023-10-30T13:23:30.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.120+0000 7f05bdd47640 1 -- 172.21.15.46:0/1260348460 learned_addr learned my addr 172.21.15.46:0/1260348460 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.120+0000 7f05bcd45640 1 --2- 172.21.15.46:0/1260348460 >> v2:172.21.15.46:3300/0 conn(0x7f05b8156020 0x7f05b806d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.124+0000 7f05bdd47640 1 -- 172.21.15.46:0/1260348460 >> v2:172.21.15.46:3300/0 conn(0x7f05b8156020 msgr2=0x7f05b806d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:30.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.124+0000 7f05bdd47640 1 --2- 172.21.15.46:0/1260348460 >> v2:172.21.15.46:3300/0 conn(0x7f05b8156020 0x7f05b806d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.124+0000 7f05bdd47640 1 -- 172.21.15.46:0/1260348460 >> v1:172.21.15.46:6789/0 conn(0x7f05b814c140 legacy=0x7f05b80c2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.124+0000 7f05bdd47640 1 -- 172.21.15.46:0/1260348460 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b8158d30 con 0x7f05b806d620 2023-10-30T13:23:30.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.128+0000 7f05bdd47640 1 --2- 172.21.15.46:0/1260348460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b806d620 0x7f05b815ff20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009e50 tx=0x7f05b0023600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ec8fdb6ffefc631 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.128+0000 7f05affff640 1 -- 172.21.15.46:0/1260348460 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00b3430 con 0x7f05b806d620 2023-10-30T13:23:30.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.128+0000 7f05affff640 1 -- 172.21.15.46:0/1260348460 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b00b3590 con 0x7f05b806d620 2023-10-30T13:23:30.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.128+0000 7f05bf7aa640 1 -- 172.21.15.46:0/1260348460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b806d620 msgr2=0x7f05b815ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.128+0000 7f05affff640 1 -- 172.21.15.46:0/1260348460 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00b3430 con 0x7f05b806d620 2023-10-30T13:23:30.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.128+0000 7f05bf7aa640 1 --2- 172.21.15.46:0/1260348460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b806d620 0x7f05b815ff20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009e50 tx=0x7f05b0023600 comp rx=0 tx=0).stop 2023-10-30T13:23:30.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.132+0000 7f05bf7aa640 1 -- 172.21.15.46:0/1260348460 shutdown_connections 2023-10-30T13:23:30.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.132+0000 7f05bf7aa640 1 --2- 172.21.15.46:0/1260348460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b806d620 0x7f05b815ff20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.132+0000 7f05bf7aa640 1 --2- 172.21.15.46:0/1260348460 >> v2:172.21.15.46:3300/0 conn(0x7f05b8156020 0x7f05b806d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.132+0000 7f05bf7aa640 1 -- 172.21.15.46:0/1260348460 >> 172.21.15.46:0/1260348460 conn(0x7f05b80cac50 msgr2=0x7f05b80cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.132+0000 7f05bf7aa640 1 -- 172.21.15.46:0/1260348460 shutdown_connections 2023-10-30T13:23:30.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.132+0000 7f05bf7aa640 1 -- 172.21.15.46:0/1260348460 wait complete. 2023-10-30T13:23:30.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.136+0000 7f05bf7aa640 1 Processor -- start 2023-10-30T13:23:30.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.136+0000 7f05bf7aa640 1 -- start start 2023-10-30T13:23:30.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.136+0000 7f05bf7aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05b806d620 0x7f05b81fc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.136+0000 7f05bf7aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b814c140 0x7f05b80bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.136+0000 7f05bd546640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05b806d620 0x7f05b81fc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bf7aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8156020 0x7f05b814e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bd546640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05b806d620 0x7f05b81fc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39734/0 (socket says 172.21.15.46:39734) 2023-10-30T13:23:30.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bcd45640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b814c140 0x7f05b80bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bf7aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b81fc7e0 con 0x7f05b806d620 2023-10-30T13:23:30.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bf7aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b80bfb00 con 0x7f05b8156020 2023-10-30T13:23:30.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bf7aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b8152910 con 0x7f05b814c140 2023-10-30T13:23:30.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bcd45640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b814c140 0x7f05b80bff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59606/0 (socket says 172.21.15.46:59606) 2023-10-30T13:23:30.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bcd45640 1 -- 172.21.15.46:0/3570582451 learned_addr learned my addr 172.21.15.46:0/3570582451 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bdd47640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8156020 0x7f05b814e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bd546640 1 -- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b814c140 msgr2=0x7f05b80bff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bd546640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b814c140 0x7f05b80bff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bd546640 1 -- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8156020 msgr2=0x7f05b814e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bd546640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8156020 0x7f05b814e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bd546640 1 -- 172.21.15.46:0/3570582451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b0009b30 con 0x7f05b806d620 2023-10-30T13:23:30.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bd546640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05b806d620 0x7f05b81fc2a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f05b4000f70 tx=0x7f05b400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bdd47640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8156020 0x7f05b814e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:30.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.140+0000 7f05bcd45640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b814c140 0x7f05b80bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:30.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.144+0000 7f05adffb640 1 -- 172.21.15.46:0/3570582451 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b40052d0 con 0x7f05b806d620 2023-10-30T13:23:30.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.144+0000 7f05adffb640 1 -- 172.21.15.46:0/3570582451 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b4005430 con 0x7f05b806d620 2023-10-30T13:23:30.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.144+0000 7f05adffb640 1 -- 172.21.15.46:0/3570582451 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4003840 con 0x7f05b806d620 2023-10-30T13:23:30.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.144+0000 7f05bf7aa640 1 -- 172.21.15.46:0/3570582451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b80c04d0 con 0x7f05b806d620 2023-10-30T13:23:30.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.144+0000 7f05bf7aa640 1 -- 172.21.15.46:0/3570582451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b8152bb0 con 0x7f05b806d620 2023-10-30T13:23:30.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.144+0000 7f05bf7aa640 1 -- 172.21.15.46:0/3570582451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b804e170 con 0x7f05b806d620 2023-10-30T13:23:30.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.156+0000 7f05adffb640 1 -- 172.21.15.46:0/3570582451 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f05b40039a0 con 0x7f05b806d620 2023-10-30T13:23:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.156+0000 7f05adffb640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f059c0664d0 0x7f059c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.156+0000 7f05adffb640 1 -- 172.21.15.46:0/3570582451 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05b4083a40 con 0x7f05b806d620 2023-10-30T13:23:30.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.156+0000 7f05adffb640 1 -- 172.21.15.46:0/3570582451 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f05b4051050 con 0x7f05b806d620 2023-10-30T13:23:30.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.160+0000 7f05bcd45640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f059c0664d0 0x7f059c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.164+0000 7f05bcd45640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f059c0664d0 0x7f059c068990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f05b80c9ad0 tx=0x7f05a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.180+0000 7fa348b3e640 1 Processor -- start 2023-10-30T13:23:30.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.180+0000 7fa348b3e640 1 -- start start 2023-10-30T13:23:30.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.180+0000 7fa208609640 1 Processor -- start 2023-10-30T13:23:30.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.180+0000 7fa208609640 1 -- start start 2023-10-30T13:23:30.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa348b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa34406ad90 0x7fa344156280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa348b3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa344152b80 0x7fa344069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa348b3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3441567c0 con 0x7fa3440c8430 2023-10-30T13:23:30.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa348b3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa344070820 con 0x7fa344152b80 2023-10-30T13:23:30.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa348b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440c6d80 con 0x7fa34406ad90 2023-10-30T13:23:30.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa341d9b640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa3440c8430 0x7fa3440c6670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:53148/0 (socket says 172.21.15.46:53148) 2023-10-30T13:23:30.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa341d9b640 1 -- 172.21.15.46:0/2767866051 learned_addr learned my addr 172.21.15.46:0/2767866051 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:30.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa208609640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa200161be0 0x7fa20006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa208609640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2000c72b0 con 0x7fa200158090 2023-10-30T13:23:30.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa208609640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2000c6fd0 con 0x7fa2000c6b40 2023-10-30T13:23:30.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa208609640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa20015dd30 con 0x7fa200161be0 2023-10-30T13:23:30.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa342d9d640 1 --2- 172.21.15.46:0/2767866051 >> v2:172.21.15.154:3300/0 conn(0x7fa344152b80 0x7fa344069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa34159a640 1 -- 172.21.15.46:0/2767866051 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294605531 0 0) 0x7fa3441567c0 con 0x7fa3440c8430 2023-10-30T13:23:30.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa34159a640 1 -- 172.21.15.46:0/2767866051 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa330003530 con 0x7fa3440c8430 2023-10-30T13:23:30.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa205ba4640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa2000c6b40 0x7fa2000c0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:53160/0 (socket says 172.21.15.46:53160) 2023-10-30T13:23:30.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa206ba6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa200161be0 0x7fa20006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa206ba6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa200161be0 0x7fa20006a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39502/0 (socket says 172.21.15.46:39502) 2023-10-30T13:23:30.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa206ba6640 1 -- 172.21.15.46:0/2186903393 learned_addr learned my addr 172.21.15.46:0/2186903393 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa2063a5640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa200158090 0x7fa200069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48534/0 (socket says 172.21.15.46:48534) 2023-10-30T13:23:30.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa342d9d640 1 -- 172.21.15.46:0/2767866051 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa34406ad90 msgr2=0x7fa344156280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T13:23:30.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa342d9d640 1 --2- 172.21.15.46:0/2767866051 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa34406ad90 0x7fa344156280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa342d9d640 1 -- 172.21.15.46:0/2767866051 >> v1:172.21.15.154:6789/0 conn(0x7fa3440c8430 legacy=0x7fa3440c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa342d9d640 1 -- 172.21.15.46:0/2767866051 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3440c76c0 con 0x7fa344152b80 2023-10-30T13:23:30.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa206ba6640 1 -- 172.21.15.46:0/2186903393 >> v1:172.21.15.154:6789/0 conn(0x7fa2000c6b40 legacy=0x7fa2000c0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa206ba6640 1 -- 172.21.15.46:0/2186903393 >> v1:172.21.15.46:6789/0 conn(0x7fa200158090 legacy=0x7fa200069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.184+0000 7fa206ba6640 1 -- 172.21.15.46:0/2186903393 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20015e010 con 0x7fa200161be0 2023-10-30T13:23:30.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.188+0000 7fa206ba6640 1 --2- 172.21.15.46:0/2186903393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa200161be0 0x7fa20006a5a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009e50 tx=0x7fa1f801f1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b1503a4b4f0ea8a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7f1721d94640 1 Processor -- start 2023-10-30T13:23:30.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa342d9d640 1 --2- 172.21.15.46:0/2767866051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa344152b80 0x7fa344069e90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa334009d90 tx=0x7fa33401c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98bde3245221be45 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa2053a3640 1 -- 172.21.15.46:0/2186903393 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80af400 con 0x7fa200161be0 2023-10-30T13:23:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa2053a3640 1 -- 172.21.15.46:0/2186903393 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f80ab050 con 0x7fa200161be0 2023-10-30T13:23:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa2053a3640 1 -- 172.21.15.46:0/2186903393 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80af780 con 0x7fa200161be0 2023-10-30T13:23:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7f1721d94640 1 -- start start 2023-10-30T13:23:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa34159a640 1 -- 172.21.15.46:0/2767866051 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3340a5890 con 0x7fa344152b80 2023-10-30T13:23:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa34159a640 1 -- 172.21.15.46:0/2767866051 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3340a59f0 con 0x7fa344152b80 2023-10-30T13:23:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa34159a640 1 -- 172.21.15.46:0/2767866051 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3340a5890 con 0x7fa344152b80 2023-10-30T13:23:30.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa208609640 1 -- 172.21.15.46:0/2186903393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa200161be0 msgr2=0x7fa20006a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa208609640 1 --2- 172.21.15.46:0/2186903393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa200161be0 0x7fa20006a5a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009e50 tx=0x7fa1f801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7f94f5b3e640 1 Processor -- start 2023-10-30T13:23:30.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7f94f5b3e640 1 -- start start 2023-10-30T13:23:30.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa208609640 1 -- 172.21.15.46:0/2186903393 shutdown_connections 2023-10-30T13:23:30.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa208609640 1 --2- 172.21.15.46:0/2186903393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa200161be0 0x7fa20006a5a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa208609640 1 -- 172.21.15.46:0/2186903393 >> 172.21.15.46:0/2186903393 conn(0x7fa2000c7910 msgr2=0x7fa2000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa208609640 1 -- 172.21.15.46:0/2186903393 shutdown_connections 2023-10-30T13:23:30.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa208609640 1 -- 172.21.15.46:0/2186903393 wait complete. 2023-10-30T13:23:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa348b3e640 1 -- 172.21.15.46:0/2767866051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa344152b80 msgr2=0x7fa344069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa348b3e640 1 --2- 172.21.15.46:0/2767866051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa344152b80 0x7fa344069e90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa334009d90 tx=0x7fa33401c900 comp rx=0 tx=0).stop 2023-10-30T13:23:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa348b3e640 1 -- 172.21.15.46:0/2767866051 shutdown_connections 2023-10-30T13:23:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa348b3e640 1 --2- 172.21.15.46:0/2767866051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa344152b80 0x7fa344069e90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa348b3e640 1 --2- 172.21.15.46:0/2767866051 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa34406ad90 0x7fa344156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa348b3e640 1 -- 172.21.15.46:0/2767866051 >> 172.21.15.46:0/2767866051 conn(0x7fa3440c7910 msgr2=0x7fa3440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa348b3e640 1 -- 172.21.15.46:0/2767866051 shutdown_connections 2023-10-30T13:23:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7fa348b3e640 1 -- 172.21.15.46:0/2767866051 wait complete. 2023-10-30T13:23:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7f94f5b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94e80452b0 0x7f94e80074d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7f94f5b3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e8007040 con 0x7f94e80dd3b0 2023-10-30T13:23:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7f94f5b3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e8006150 con 0x7f94e804c4b0 2023-10-30T13:23:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7f94f5b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8033650 con 0x7f94e80452b0 2023-10-30T13:23:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7f94ef7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f94e80dd3b0 0x7f94e80e3c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48544/0 (socket says 172.21.15.46:48544) 2023-10-30T13:23:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.192+0000 7f94ef7fe640 1 -- 172.21.15.46:0/563519165 learned_addr learned my addr 172.21.15.46:0/563519165 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f1721d94640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f171c153c60 0x7f171c0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f1721d94640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f171c159830 0x7f171c159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f1721d94640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171c0cb780 con 0x7f171c0c4430 2023-10-30T13:23:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f1721d94640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171c154040 con 0x7f171c153c60 2023-10-30T13:23:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f1721d94640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171c15a2e0 con 0x7f171c159830 2023-10-30T13:23:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f94eeffd640 1 --2- 172.21.15.46:0/563519165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94e80452b0 0x7f94e80074d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7fa208609640 1 Processor -- start 2023-10-30T13:23:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f94ee7fc640 1 -- 172.21.15.46:0/563519165 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3940064489 0 0) 0x7f94e8007040 con 0x7f94e80dd3b0 2023-10-30T13:23:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f94ee7fc640 1 -- 172.21.15.46:0/563519165 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e0003530 con 0x7f94e80dd3b0 2023-10-30T13:23:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f94eeffd640 1 -- 172.21.15.46:0/563519165 >> v1:172.21.15.154:6789/0 conn(0x7f94e804c4b0 legacy=0x7f94e804b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f94eeffd640 1 -- 172.21.15.46:0/563519165 >> v1:172.21.15.46:6789/0 conn(0x7f94e80dd3b0 legacy=0x7f94e80e3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f171b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f171c153c60 0x7f171c0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f1712ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f171c0c4430 0x7f171c0cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48552/0 (socket says 172.21.15.46:48552) 2023-10-30T13:23:30.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f1712ffd640 1 -- 172.21.15.46:0/1612025252 learned_addr learned my addr 172.21.15.46:0/1612025252 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:30.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7fa208609640 1 -- start start 2023-10-30T13:23:30.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f94eeffd640 1 -- 172.21.15.46:0/563519165 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e8006430 con 0x7f94e80452b0 2023-10-30T13:23:30.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7fa08a2d6640 1 Processor -- start 2023-10-30T13:23:30.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f171b7fe640 1 -- 172.21.15.46:0/1612025252 >> v2:172.21.15.154:3300/0 conn(0x7f171c159830 msgr2=0x7f171c159cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T13:23:30.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f171b7fe640 1 --2- 172.21.15.46:0/1612025252 >> v2:172.21.15.154:3300/0 conn(0x7f171c159830 0x7f171c159cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f171b7fe640 1 -- 172.21.15.46:0/1612025252 >> v1:172.21.15.46:6789/0 conn(0x7f171c0c4430 legacy=0x7f171c0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7f171b7fe640 1 -- 172.21.15.46:0/1612025252 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171c15a5c0 con 0x7f171c153c60 2023-10-30T13:23:30.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7fa348b3e640 1 Processor -- start 2023-10-30T13:23:30.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7fa348b3e640 1 -- start start 2023-10-30T13:23:30.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7fa08a2d6640 1 -- start start 2023-10-30T13:23:30.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.196+0000 7fd692491640 1 Processor -- start 2023-10-30T13:23:30.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fd692491640 1 -- start start 2023-10-30T13:23:30.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa08a2d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa084158090 0x7fa084161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa08a2d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa0840c6b40 0x7fa084069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa08a2d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa08406a3d0 0x7fa08406a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa08a2d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa084162120 con 0x7fa08406a3d0 2023-10-30T13:23:30.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa08a2d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa08406ad60 con 0x7fa0840c6b40 2023-10-30T13:23:30.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa08a2d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0840c72b0 con 0x7fa084158090 2023-10-30T13:23:30.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa348b3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa34406ad90 0x7fa3440ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa208609640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa200158090 0x7fa2000c4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa208609640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa200161be0 0x7fa200070930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa208609640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2000c4ee0 0x7fa2001e7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa208609640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2001e9d40 con 0x7fa2000c4ee0 2023-10-30T13:23:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa208609640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2001e8eb0 con 0x7fa200158090 2023-10-30T13:23:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa208609640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2000c4c20 con 0x7fa200161be0 2023-10-30T13:23:30.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa34259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa34406ad90 0x7fa3440ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa34259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa34406ad90 0x7fa3440ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39752/0 (socket says 172.21.15.46:39752) 2023-10-30T13:23:30.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fd692491640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd68c151910 0x7fd68c158120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa088873640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa08406a3d0 0x7fa08406a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa083fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa084158090 0x7fa084161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa348b3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3440c8430 0x7fa3440c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa088873640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa08406a3d0 0x7fa08406a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39750/0 (socket says 172.21.15.46:39750) 2023-10-30T13:23:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa088873640 1 -- 172.21.15.46:0/10628636 learned_addr learned my addr 172.21.15.46:0/10628636 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fd692491640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68c06a4e0 con 0x7fd68c06a970 2023-10-30T13:23:30.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fd692491640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68c158660 con 0x7fd68c06ade0 2023-10-30T13:23:30.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fd692491640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd68c0c6630 con 0x7fd68c151910 2023-10-30T13:23:30.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa348b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa344152b80 0x7fa34414e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa0837fe640 1 --2- 172.21.15.46:0/10628636 >> v2:172.21.15.154:3300/0 conn(0x7fa0840c6b40 0x7fa084069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fd68b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd68c06a970 0x7fd68c0c5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48558/0 (socket says 172.21.15.46:48558) 2023-10-30T13:23:30.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fd68b7fe640 1 -- 172.21.15.46:0/98079364 learned_addr learned my addr 172.21.15.46:0/98079364 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:30.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7f94eeffd640 1 --2- 172.21.15.46:0/563519165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94e80452b0 0x7f94e80074d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0097f0 tx=0x7f94dc01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7e6880a8044728b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fd68bfff640 1 --2- 172.21.15.46:0/98079364 >> v2:172.21.15.154:3300/0 conn(0x7fd68c151910 0x7fd68c158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa206ba6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2000c4ee0 0x7fa2001e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa206ba6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2000c4ee0 0x7fa2001e7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39764/0 (socket says 172.21.15.46:39764) 2023-10-30T13:23:30.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa206ba6640 1 -- 172.21.15.46:0/1166290944 learned_addr learned my addr 172.21.15.46:0/1166290944 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa0837fe640 1 -- 172.21.15.46:0/10628636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa084158090 msgr2=0x7fa084161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:30.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa0837fe640 1 --2- 172.21.15.46:0/10628636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa084158090 0x7fa084161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa0837fe640 1 -- 172.21.15.46:0/10628636 >> v2:172.21.15.46:3300/0 conn(0x7fa08406a3d0 msgr2=0x7fa08406a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:30.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa0837fe640 1 --2- 172.21.15.46:0/10628636 >> v2:172.21.15.46:3300/0 conn(0x7fa08406a3d0 0x7fa08406a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa0837fe640 1 -- 172.21.15.46:0/10628636 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0840c6fd0 con 0x7fa0840c6b40 2023-10-30T13:23:30.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa083fff640 1 --2- 172.21.15.46:0/10628636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa084158090 0x7fa084161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:30.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7f94ee7fc640 1 -- 172.21.15.46:0/563519165 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0ad640 con 0x7f94e80452b0 2023-10-30T13:23:30.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa205ba4640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa200161be0 0x7fa200070930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7f94ee7fc640 1 -- 172.21.15.46:0/563519165 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc0ad7a0 con 0x7f94e80452b0 2023-10-30T13:23:30.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7f94ee7fc640 1 -- 172.21.15.46:0/563519165 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0adb10 con 0x7f94e80452b0 2023-10-30T13:23:30.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa206ba6640 1 -- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa200161be0 msgr2=0x7fa200070930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa206ba6640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa200161be0 0x7fa200070930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa206ba6640 1 -- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa200158090 msgr2=0x7fa2000c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa348b3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34413ae50 con 0x7fa34406ad90 2023-10-30T13:23:30.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa348b3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440ca1a0 con 0x7fa3440c8430 2023-10-30T13:23:30.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa348b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34413a840 con 0x7fa344152b80 2023-10-30T13:23:30.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa34259c640 1 -- 172.21.15.46:0/1131061868 learned_addr learned my addr 172.21.15.46:0/1131061868 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.200+0000 7fa342d9d640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa344152b80 0x7fa34414e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa341d9b640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3440c8430 0x7fa3440c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fd68affd640 1 -- 172.21.15.46:0/98079364 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736362146 0 0) 0x7fd68c06a4e0 con 0x7fd68c06a970 2023-10-30T13:23:30.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fd68affd640 1 -- 172.21.15.46:0/98079364 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd674003530 con 0x7fd68c06a970 2023-10-30T13:23:30.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f171b7fe640 1 --2- 172.21.15.46:0/1612025252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171c153c60 0x7f171c0ca360 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f171c073da0 tx=0x7f170401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9499324159026f67 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fd68affd640 1 -- 172.21.15.46:0/98079364 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743651165 0 0) 0x7fd68c158660 con 0x7fd68c06ade0 2023-10-30T13:23:30.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa342d9d640 1 -- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3440c8430 msgr2=0x7fa3440c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa342d9d640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3440c8430 0x7fa3440c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa342d9d640 1 -- 172.21.15.46:0/1131061868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa34406ad90 msgr2=0x7fa3440ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa342d9d640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa34406ad90 0x7fa3440ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa342d9d640 1 -- 172.21.15.46:0/1131061868 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa334009a70 con 0x7fa344152b80 2023-10-30T13:23:30.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f171affd640 1 -- 172.21.15.46:0/1612025252 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17040a58e0 con 0x7f171c153c60 2023-10-30T13:23:30.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f94f5b3e640 1 -- 172.21.15.46:0/563519165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94e80452b0 msgr2=0x7f94e80074d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f94f5b3e640 1 --2- 172.21.15.46:0/563519165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94e80452b0 0x7f94e80074d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0097f0 tx=0x7f94dc01c480 comp rx=0 tx=0).stop 2023-10-30T13:23:30.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fd68bfff640 1 -- 172.21.15.46:0/98079364 >> v1:172.21.15.154:6789/0 conn(0x7fd68c06ade0 legacy=0x7fd68c0c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fd68bfff640 1 -- 172.21.15.46:0/98079364 >> v1:172.21.15.46:6789/0 conn(0x7fd68c06a970 legacy=0x7fd68c0c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fd68bfff640 1 -- 172.21.15.46:0/98079364 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c0c6c50 con 0x7fd68c151910 2023-10-30T13:23:30.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f171affd640 1 -- 172.21.15.46:0/1612025252 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17040a5a40 con 0x7f171c153c60 2023-10-30T13:23:30.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f171affd640 1 -- 172.21.15.46:0/1612025252 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17040a5dd0 con 0x7f171c153c60 2023-10-30T13:23:30.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa342d9d640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa344152b80 0x7fa34414e330 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa3340ac000 tx=0x7fa3340196a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa2063a5640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa200158090 0x7fa2000c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa206ba6640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa200158090 0x7fa2000c4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa206ba6640 1 -- 172.21.15.46:0/1166290944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f8009b30 con 0x7fa2000c4ee0 2023-10-30T13:23:30.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f2fd34c9640 1 Processor -- start 2023-10-30T13:23:30.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f94f5b3e640 1 -- 172.21.15.46:0/563519165 shutdown_connections 2023-10-30T13:23:30.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f94f5b3e640 1 --2- 172.21.15.46:0/563519165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94e80452b0 0x7f94e80074d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f94f5b3e640 1 -- 172.21.15.46:0/563519165 >> 172.21.15.46:0/563519165 conn(0x7f94e8050310 msgr2=0x7f94e8050720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa33affd640 1 -- 172.21.15.46:0/1131061868 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334003db0 con 0x7fa344152b80 2023-10-30T13:23:30.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f94f5b3e640 1 -- 172.21.15.46:0/563519165 shutdown_connections 2023-10-30T13:23:30.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f2fd34c9640 1 -- start start 2023-10-30T13:23:30.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa33affd640 1 -- 172.21.15.46:0/1131061868 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3340a5850 con 0x7fa344152b80 2023-10-30T13:23:30.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa33affd640 1 -- 172.21.15.46:0/1131061868 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334004240 con 0x7fa344152b80 2023-10-30T13:23:30.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa348b3e640 1 -- 172.21.15.46:0/1131061868 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa344155580 con 0x7fa344152b80 2023-10-30T13:23:30.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f94f5b3e640 1 -- 172.21.15.46:0/563519165 wait complete. 2023-10-30T13:23:30.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa348b3e640 1 -- 172.21.15.46:0/1131061868 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa344155b40 con 0x7fa344152b80 2023-10-30T13:23:30.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f1721d94640 1 -- 172.21.15.46:0/1612025252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171c153c60 msgr2=0x7f171c0ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7f1721d94640 1 --2- 172.21.15.46:0/1612025252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171c153c60 0x7f171c0ca360 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f171c073da0 tx=0x7f170401c930 comp rx=0 tx=0).stop 2023-10-30T13:23:30.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.204+0000 7fa206ba6640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2000c4ee0 0x7fa2001e7080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80b2040 tx=0x7fa1f801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f1721d94640 1 -- 172.21.15.46:0/1612025252 shutdown_connections 2023-10-30T13:23:30.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f1721d94640 1 --2- 172.21.15.46:0/1612025252 >> v2:172.21.15.154:3300/0 conn(0x7f171c159830 0x7f171c159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f1721d94640 1 --2- 172.21.15.46:0/1612025252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171c153c60 0x7f171c0ca360 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f1721d94640 1 -- 172.21.15.46:0/1612025252 >> 172.21.15.46:0/1612025252 conn(0x7f171c06bfe0 msgr2=0x7f171c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa348b3e640 1 -- 172.21.15.46:0/1131061868 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34413afc0 con 0x7fa344152b80 2023-10-30T13:23:30.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa0837fe640 1 --2- 172.21.15.46:0/10628636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0840c6b40 0x7fa084069e90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa08406f170 tx=0x7fa07401ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16aef614167fb6b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f1721d94640 1 -- 172.21.15.46:0/1612025252 shutdown_connections 2023-10-30T13:23:30.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa1f6ffd640 1 -- 172.21.15.46:0/1166290944 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80afde0 con 0x7fa2000c4ee0 2023-10-30T13:23:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa1f6ffd640 1 -- 172.21.15.46:0/1166290944 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f8009510 con 0x7fa2000c4ee0 2023-10-30T13:23:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa1f6ffd640 1 -- 172.21.15.46:0/1166290944 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80ba660 con 0x7fa2000c4ee0 2023-10-30T13:23:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa208609640 1 -- 172.21.15.46:0/1166290944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2001e90c0 con 0x7fa2000c4ee0 2023-10-30T13:23:30.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa082ffd640 1 -- 172.21.15.46:0/10628636 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0740a8910 con 0x7fa0840c6b40 2023-10-30T13:23:30.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa208609640 1 -- 172.21.15.46:0/1166290944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2001e9680 con 0x7fa2000c4ee0 2023-10-30T13:23:30.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa082ffd640 1 -- 172.21.15.46:0/10628636 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0740a8a70 con 0x7fa0840c6b40 2023-10-30T13:23:30.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa082ffd640 1 -- 172.21.15.46:0/10628636 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0740a8e00 con 0x7fa0840c6b40 2023-10-30T13:23:30.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd34c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2fcc152ee0 0x7f2fcc1599d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd34c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2fcc0bfec0 0x7f2fcc0c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd34c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2fcc156290 0x7f2fcc0c6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd1265640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2fcc152ee0 0x7f2fcc1599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd1265640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2fcc152ee0 0x7f2fcc1599d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39548/0 (socket says 172.21.15.46:39548) 2023-10-30T13:23:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f1721d94640 1 -- 172.21.15.46:0/1612025252 wait complete. 2023-10-30T13:23:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f1721d94640 1 Processor -- start 2023-10-30T13:23:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd1265640 1 -- 172.21.15.46:0/3166255855 learned_addr learned my addr 172.21.15.46:0/3166255855 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd34c9640 1 -- 172.21.15.46:0/3166255855 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc1595b0 con 0x7f2fcc0bfec0 2023-10-30T13:23:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd34c9640 1 -- 172.21.15.46:0/3166255855 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc0c0aa0 con 0x7f2fcc152ee0 2023-10-30T13:23:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd34c9640 1 -- 172.21.15.46:0/3166255855 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0c04e0 con 0x7f2fcc156290 2023-10-30T13:23:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd1a66640 1 --2- 172.21.15.46:0/3166255855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2fcc156290 0x7f2fcc0c6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa33affd640 1 -- 172.21.15.46:0/1131061868 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3340b4050 con 0x7fa344152b80 2023-10-30T13:23:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa33affd640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3240664d0 0x7fa324068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7fa33affd640 1 -- 172.21.15.46:0/1131061868 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa33411efe0 con 0x7fa344152b80 2023-10-30T13:23:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa08a2d6640 1 -- 172.21.15.46:0/10628636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0840c6b40 msgr2=0x7fa084069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa08a2d6640 1 --2- 172.21.15.46:0/10628636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0840c6b40 0x7fa084069e90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa08406f170 tx=0x7fa07401ec50 comp rx=0 tx=0).stop 2023-10-30T13:23:30.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd1265640 1 -- 172.21.15.46:0/3166255855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2fcc156290 msgr2=0x7f2fcc0c6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:30.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd1265640 1 --2- 172.21.15.46:0/3166255855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2fcc156290 0x7f2fcc0c6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd1265640 1 -- 172.21.15.46:0/3166255855 >> v2:172.21.15.46:3300/0 conn(0x7f2fcc0bfec0 msgr2=0x7f2fcc0c0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd1265640 1 --2- 172.21.15.46:0/3166255855 >> v2:172.21.15.46:3300/0 conn(0x7f2fcc0bfec0 0x7f2fcc0c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f2fd1265640 1 -- 172.21.15.46:0/3166255855 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc0c07c0 con 0x7f2fcc152ee0 2023-10-30T13:23:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f94f5b3e640 1 Processor -- start 2023-10-30T13:23:30.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f94f5b3e640 1 -- start start 2023-10-30T13:23:30.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f94f5b3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e80452b0 0x7f94e817fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.208+0000 7f94f5b3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e804c4b0 0x7f94e80e30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94ef7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e80452b0 0x7f94e817fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94ef7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e80452b0 0x7f94e817fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39564/0 (socket says 172.21.15.46:39564) 2023-10-30T13:23:30.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94ef7fe640 1 -- 172.21.15.46:0/1501669083 learned_addr learned my addr 172.21.15.46:0/1501669083 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94f5b3e640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94e80dd3b0 0x7f94e817c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e817bf40 con 0x7f94e804c4b0 2023-10-30T13:23:30.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8180060 con 0x7f94e80452b0 2023-10-30T13:23:30.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e80e01a0 con 0x7f94e80dd3b0 2023-10-30T13:23:30.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa208609640 1 -- 172.21.15.46:0/1166290944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2001e9e80 con 0x7fa2000c4ee0 2023-10-30T13:23:30.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94effff640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94e80dd3b0 0x7f94e817c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa08a2d6640 1 -- 172.21.15.46:0/10628636 shutdown_connections 2023-10-30T13:23:30.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa08a2d6640 1 --2- 172.21.15.46:0/10628636 >> v2:172.21.15.46:3300/0 conn(0x7fa08406a3d0 0x7fa08406a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa08a2d6640 1 --2- 172.21.15.46:0/10628636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0840c6b40 0x7fa084069e90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa08a2d6640 1 --2- 172.21.15.46:0/10628636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa084158090 0x7fa084161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa08a2d6640 1 -- 172.21.15.46:0/10628636 >> 172.21.15.46:0/10628636 conn(0x7fa0840c7910 msgr2=0x7fa0840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa34259c640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3240664d0 0x7fa324068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa08a2d6640 1 -- 172.21.15.46:0/10628636 shutdown_connections 2023-10-30T13:23:30.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fd68bfff640 1 --2- 172.21.15.46:0/98079364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c151910 0x7fd68c158120 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0097c0 tx=0x7fd67c01eab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f83c789b33364037 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94eeffd640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e804c4b0 0x7f94e80e30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa08a2d6640 1 -- 172.21.15.46:0/10628636 wait complete. 2023-10-30T13:23:30.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fd68affd640 1 -- 172.21.15.46:0/98079364 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680001600 con 0x7fd68c151910 2023-10-30T13:23:30.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fd68affd640 1 -- 172.21.15.46:0/98079364 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67c0a8570 con 0x7fd68c151910 2023-10-30T13:23:30.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fd68affd640 1 -- 172.21.15.46:0/98079364 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c0a88e0 con 0x7fd68c151910 2023-10-30T13:23:30.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f1721d94640 1 -- start start 2023-10-30T13:23:30.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f1721d94640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f171c0c4430 0x7f171c147ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f1721d94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171c153c60 0x7f171c1541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f1721d94640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f171c159830 0x7f171c1546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f171b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f171c0c4430 0x7f171c147ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fd692491640 1 -- 172.21.15.46:0/98079364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c151910 msgr2=0x7fd68c158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fd692491640 1 --2- 172.21.15.46:0/98079364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c151910 0x7fd68c158120 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0097c0 tx=0x7fd67c01eab0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f171b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f171c0c4430 0x7f171c147ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39568/0 (socket says 172.21.15.46:39568) 2023-10-30T13:23:30.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f171b7fe640 1 -- 172.21.15.46:0/838286783 learned_addr learned my addr 172.21.15.46:0/838286783 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f171bfff640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f171c159830 0x7f171c1546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa34259c640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3240664d0 0x7fa324068990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0098b0 tx=0x7fa32c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fd692491640 1 -- 172.21.15.46:0/98079364 shutdown_connections 2023-10-30T13:23:30.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fd692491640 1 --2- 172.21.15.46:0/98079364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c151910 0x7fd68c158120 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0097c0 tx=0x7fd67c01eab0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fd692491640 1 -- 172.21.15.46:0/98079364 >> 172.21.15.46:0/98079364 conn(0x7fd68c0c7910 msgr2=0x7fd68c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f2fd1265640 1 --2- 172.21.15.46:0/3166255855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fcc152ee0 0x7f2fcc1599d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0c54d0 tx=0x7f2fc8020910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a6ba1ef36ff169 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94ef7fe640 1 -- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94e80dd3b0 msgr2=0x7f94e817c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94ef7fe640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94e80dd3b0 0x7f94e817c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94ef7fe640 1 -- 172.21.15.46:0/1501669083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e804c4b0 msgr2=0x7f94e80e30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94ef7fe640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e804c4b0 0x7f94e80e30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f94ef7fe640 1 -- 172.21.15.46:0/1501669083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94dc009420 con 0x7f94e80452b0 2023-10-30T13:23:30.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7f2fc3fff640 1 -- 172.21.15.46:0/3166255855 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80a98b0 con 0x7f2fcc152ee0 2023-10-30T13:23:30.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f2fc3fff640 1 -- 172.21.15.46:0/3166255855 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc80a9a10 con 0x7f2fcc152ee0 2023-10-30T13:23:30.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f2fc3fff640 1 -- 172.21.15.46:0/3166255855 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80a9da0 con 0x7f2fcc152ee0 2023-10-30T13:23:30.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fd692491640 1 -- 172.21.15.46:0/98079364 shutdown_connections 2023-10-30T13:23:30.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f94ef7fe640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e80452b0 0x7f94e817fb20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f94e400c420 tx=0x7f94e400c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f1721d94640 1 -- 172.21.15.46:0/838286783 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c0c56a0 con 0x7f171c153c60 2023-10-30T13:23:30.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f1721d94640 1 -- 172.21.15.46:0/838286783 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c1564f0 con 0x7f171c0c4430 2023-10-30T13:23:30.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f1721d94640 1 -- 172.21.15.46:0/838286783 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171c14b200 con 0x7f171c159830 2023-10-30T13:23:30.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa08a2d6640 1 Processor -- start 2023-10-30T13:23:30.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.212+0000 7fa1f6ffd640 1 -- 172.21.15.46:0/1166290944 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1f80b6040 con 0x7fa2000c4ee0 2023-10-30T13:23:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa1f6ffd640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1e40665a0 0x7fa1e4068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa1f6ffd640 1 -- 172.21.15.46:0/1166290944 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1f8122450 con 0x7fa2000c4ee0 2023-10-30T13:23:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f94dbfff640 1 -- 172.21.15.46:0/1501669083 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e4005cf0 con 0x7f94e80452b0 2023-10-30T13:23:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f94dbfff640 1 -- 172.21.15.46:0/1501669083 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e4005e50 con 0x7f94e80452b0 2023-10-30T13:23:30.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f94dbfff640 1 -- 172.21.15.46:0/1501669083 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e40072d0 con 0x7f94e80452b0 2023-10-30T13:23:30.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e80ddae0 con 0x7f94e80452b0 2023-10-30T13:23:30.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e80e0440 con 0x7f94e80452b0 2023-10-30T13:23:30.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f1712ffd640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171c153c60 0x7f171c1541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f171bfff640 1 -- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f171c0c4430 msgr2=0x7f171c147ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f171bfff640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f171c0c4430 0x7f171c147ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f171bfff640 1 -- 172.21.15.46:0/838286783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171c153c60 msgr2=0x7f171c1541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f171bfff640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171c153c60 0x7f171c1541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f171bfff640 1 -- 172.21.15.46:0/838286783 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1704009a70 con 0x7f171c159830 2023-10-30T13:23:30.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa2063a5640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1e40665a0 0x7fa1e4068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa08a2d6640 1 -- start start 2023-10-30T13:23:30.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f171bfff640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f171c159830 0x7f171c1546e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f171400b410 tx=0x7f171400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fd692491640 1 -- 172.21.15.46:0/98079364 wait complete. 2023-10-30T13:23:30.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f2fd34c9640 1 -- 172.21.15.46:0/3166255855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fcc152ee0 msgr2=0x7f2fcc1599d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f2fd34c9640 1 --2- 172.21.15.46:0/3166255855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fcc152ee0 0x7f2fcc1599d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0c54d0 tx=0x7f2fc8020910 comp rx=0 tx=0).stop 2023-10-30T13:23:30.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa08a2d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa08406a3d0 0x7fa084154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa08a2d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa084158090 0x7fa0840c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa08a2d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa08414cbd0 0x7fa08414d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa08a2d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0841541a0 con 0x7fa08406a3d0 2023-10-30T13:23:30.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa08a2d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa084161e50 con 0x7fa084158090 2023-10-30T13:23:30.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa08a2d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa084153e00 con 0x7fa08414cbd0 2023-10-30T13:23:30.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e80054d0 con 0x7f94e80452b0 2023-10-30T13:23:30.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa2063a5640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1e40665a0 0x7fa1e4068a60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc001630 tx=0x7fa1fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fd692491640 1 Processor -- start 2023-10-30T13:23:30.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa0837fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa084158090 0x7fa0840c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f2fd34c9640 1 -- 172.21.15.46:0/3166255855 shutdown_connections 2023-10-30T13:23:30.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f2fd34c9640 1 --2- 172.21.15.46:0/3166255855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2fcc156290 0x7f2fcc0c6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f2fd34c9640 1 --2- 172.21.15.46:0/3166255855 >> v2:172.21.15.46:3300/0 conn(0x7f2fcc0bfec0 0x7f2fcc0c0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f2fd34c9640 1 --2- 172.21.15.46:0/3166255855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fcc152ee0 0x7f2fcc1599d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f2fd34c9640 1 -- 172.21.15.46:0/3166255855 >> 172.21.15.46:0/3166255855 conn(0x7f2fcc0c7910 msgr2=0x7f2fcc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa083fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa08406a3d0 0x7fa084154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa0837fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa084158090 0x7fa0840c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39572/0 (socket says 172.21.15.46:39572) 2023-10-30T13:23:30.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa0837fe640 1 -- 172.21.15.46:0/324800723 learned_addr learned my addr 172.21.15.46:0/324800723 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa088873640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa08414cbd0 0x7fa08414d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f2fd34c9640 1 -- 172.21.15.46:0/3166255855 shutdown_connections 2023-10-30T13:23:30.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7f2fd34c9640 1 -- 172.21.15.46:0/3166255855 wait complete. 2023-10-30T13:23:30.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fd692491640 1 -- start start 2023-10-30T13:23:30.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa0837fe640 1 -- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa08414cbd0 msgr2=0x7fa08414d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa0837fe640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa08414cbd0 0x7fa08414d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa0837fe640 1 -- 172.21.15.46:0/324800723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa08406a3d0 msgr2=0x7fa084154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa0837fe640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa08406a3d0 0x7fa084154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa0837fe640 1 -- 172.21.15.46:0/324800723 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa078009420 con 0x7fa084158090 2023-10-30T13:23:30.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.216+0000 7fa083fff640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa08406a3d0 0x7fa084154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:30.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fa088873640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa08414cbd0 0x7fa08414d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:30.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fa0837fe640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa084158090 0x7fa0840c5760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa07401c880 tx=0x7fa07401b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f94dbfff640 1 -- 172.21.15.46:0/1501669083 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94e4007430 con 0x7f94e80452b0 2023-10-30T13:23:30.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd34c9640 1 Processor -- start 2023-10-30T13:23:30.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd34c9640 1 -- start start 2023-10-30T13:23:30.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fd692491640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd68c06a970 0x7fd68c1576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f94dbfff640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94d40664f0 0x7f94d40689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f94dbfff640 1 -- 172.21.15.46:0/1501669083 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94e40837f0 con 0x7f94e80452b0 2023-10-30T13:23:30.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fd692491640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68c156140 0x7fd68c1565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fd692491640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0c2d90 0x7fd68c0c3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd34c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2fcc0bfec0 0x7f2fcc144ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd34c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fcc152ee0 0x7f2fcc149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f94eeffd640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94d40664f0 0x7f94d40689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd34c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fcc156290 0x7f2fcc142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd34c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1f0cc0 con 0x7f2fcc152ee0 2023-10-30T13:23:30.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd34c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc145810 con 0x7f2fcc156290 2023-10-30T13:23:30.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd34c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1efdd0 con 0x7f2fcc0bfec0 2023-10-30T13:23:30.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fd692491640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c1580f0 con 0x7fd68c06a970 2023-10-30T13:23:30.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd68c06a970 0x7fd68c1576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fd68b7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68c156140 0x7fd68c1565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fd68b7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68c156140 0x7fd68c1565c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59694/0 (socket says 172.21.15.46:59694) 2023-10-30T13:23:30.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fd68b7fe640 1 -- 172.21.15.46:0/1937614388 learned_addr learned my addr 172.21.15.46:0/1937614388 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd68c06a970 0x7fd68c1576c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39798/0 (socket says 172.21.15.46:39798) 2023-10-30T13:23:30.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fd690a2e640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0c2d90 0x7fd68c0c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd0a64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fcc152ee0 0x7f2fcc149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd0a64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fcc152ee0 0x7f2fcc149f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39814/0 (socket says 172.21.15.46:39814) 2023-10-30T13:23:30.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd0a64640 1 -- 172.21.15.46:0/4108420095 learned_addr learned my addr 172.21.15.46:0/4108420095 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd1265640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2fcc0bfec0 0x7f2fcc144ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fd692491640 1 -- 172.21.15.46:0/1937614388 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c158600 con 0x7fd68c0c2d90 2023-10-30T13:23:30.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7fd692491640 1 -- 172.21.15.46:0/1937614388 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c156b00 con 0x7fd68c156140 2023-10-30T13:23:30.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd1a66640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fcc156290 0x7f2fcc142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd0a64640 1 -- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2fcc0bfec0 msgr2=0x7f2fcc144ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd0a64640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2fcc0bfec0 0x7f2fcc144ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd0a64640 1 -- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fcc156290 msgr2=0x7f2fcc142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd0a64640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fcc156290 0x7f2fcc142370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd0a64640 1 -- 172.21.15.46:0/4108420095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80094d0 con 0x7f2fcc152ee0 2023-10-30T13:23:30.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd1a66640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fcc156290 0x7f2fcc142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f94eeffd640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94d40664f0 0x7f94d40689b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0af000 tx=0x7f94dc0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.220+0000 7f2fd0a64640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fcc152ee0 0x7f2fcc149f90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00b410 tx=0x7f2fbc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7fd68bfff640 1 -- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68c156140 msgr2=0x7fd68c1565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7fd68bfff640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68c156140 0x7fd68c1565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7fd68bfff640 1 -- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0c2d90 msgr2=0x7fd68c0c3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7fd68bfff640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0c2d90 0x7fd68c0c3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7fd68bfff640 1 -- 172.21.15.46:0/1937614388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c009420 con 0x7fd68c06a970 2023-10-30T13:23:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7fd68bfff640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd68c06a970 0x7fd68c1576c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0af000 tx=0x7fd67c01b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7f1718ff9640 1 -- 172.21.15.46:0/838286783 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714011830 con 0x7f171c159830 2023-10-30T13:23:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7f1718ff9640 1 -- 172.21.15.46:0/838286783 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1714011990 con 0x7f171c159830 2023-10-30T13:23:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7f1721d94640 1 -- 172.21.15.46:0/838286783 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171c14c420 con 0x7f171c159830 2023-10-30T13:23:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7f1718ff9640 1 -- 172.21.15.46:0/838286783 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714011d40 con 0x7f171c159830 2023-10-30T13:23:30.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7f1721d94640 1 -- 172.21.15.46:0/838286783 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171c1554e0 con 0x7f171c159830 2023-10-30T13:23:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7fa33affd640 1 -- 172.21.15.46:0/1131061868 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3340ec470 con 0x7fa344152b80 2023-10-30T13:23:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7f1711ffb640 1 -- 172.21.15.46:0/838286783 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f4005260 con 0x7f171c159830 2023-10-30T13:23:30.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7f2fc1ffb640 1 -- 172.21.15.46:0/4108420095 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc011830 con 0x7f2fcc152ee0 2023-10-30T13:23:30.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7fa1f6ffd640 1 -- 172.21.15.46:0/1166290944 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa1f80ef9f0 con 0x7fa2000c4ee0 2023-10-30T13:23:30.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7f2fd34c9640 1 -- 172.21.15.46:0/4108420095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc1f00b0 con 0x7f2fcc152ee0 2023-10-30T13:23:30.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7f2fc1ffb640 1 -- 172.21.15.46:0/4108420095 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc011990 con 0x7f2fcc152ee0 2023-10-30T13:23:30.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7f2fc1ffb640 1 -- 172.21.15.46:0/4108420095 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc011d00 con 0x7f2fcc152ee0 2023-10-30T13:23:30.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.224+0000 7f2fd34c9640 1 -- 172.21.15.46:0/4108420095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc1f0670 con 0x7f2fcc152ee0 2023-10-30T13:23:30.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.228+0000 7fd688ff9640 1 -- 172.21.15.46:0/1937614388 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c0c0070 con 0x7fd68c06a970 2023-10-30T13:23:30.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.228+0000 7fd688ff9640 1 -- 172.21.15.46:0/1937614388 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67c0a9a00 con 0x7fd68c06a970 2023-10-30T13:23:30.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.228+0000 7fd688ff9640 1 -- 172.21.15.46:0/1937614388 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c0a9d70 con 0x7fd68c06a970 2023-10-30T13:23:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.228+0000 7f2fd34c9640 1 -- 172.21.15.46:0/4108420095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc04e170 con 0x7f2fcc152ee0 2023-10-30T13:23:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.228+0000 7f2fc1ffb640 1 -- 172.21.15.46:0/4108420095 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2fbc01a400 con 0x7f2fcc152ee0 2023-10-30T13:23:30.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.228+0000 7f1718ff9640 1 -- 172.21.15.46:0/838286783 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f171401a400 con 0x7f171c159830 2023-10-30T13:23:30.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.228+0000 7fd692491640 1 -- 172.21.15.46:0/1937614388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68c0c3750 con 0x7fd68c06a970 2023-10-30T13:23:30.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.228+0000 7fd692491640 1 -- 172.21.15.46:0/1937614388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c158320 con 0x7fd68c06a970 2023-10-30T13:23:30.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.228+0000 7f2fc1ffb640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2fb00664d0 0x7f2fb0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.228+0000 7f2fc1ffb640 1 -- 172.21.15.46:0/4108420095 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fbc083d70 con 0x7f2fcc152ee0 2023-10-30T13:23:30.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7f1718ff9640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16fc0664a0 0x7f16fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7f1718ff9640 1 -- 172.21.15.46:0/838286783 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17140842b0 con 0x7f171c159830 2023-10-30T13:23:30.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7fd6727fc640 1 -- 172.21.15.46:0/1937614388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd664005260 con 0x7fd68c06a970 2023-10-30T13:23:30.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7f2fd1265640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2fb00664d0 0x7f2fb0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7f171b7fe640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16fc0664a0 0x7f16fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7f171b7fe640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16fc0664a0 0x7f16fc068960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1704019540 tx=0x7f17040a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7fa080ff9640 1 -- 172.21.15.46:0/324800723 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0740b0850 con 0x7fa084158090 2023-10-30T13:23:30.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7fa080ff9640 1 -- 172.21.15.46:0/324800723 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0740b09d0 con 0x7fa084158090 2023-10-30T13:23:30.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7fa080ff9640 1 -- 172.21.15.46:0/324800723 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0740b0d80 con 0x7fa084158090 2023-10-30T13:23:30.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7fa08a2d6640 1 -- 172.21.15.46:0/324800723 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa074009420 con 0x7fa084158090 2023-10-30T13:23:30.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7fa08a2d6640 1 -- 172.21.15.46:0/324800723 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa084153010 con 0x7fa084158090 2023-10-30T13:23:30.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7f94dbfff640 1 -- 172.21.15.46:0/1501669083 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94e4050d90 con 0x7f94e80452b0 2023-10-30T13:23:30.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7f2fd1265640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2fb00664d0 0x7f2fb0068990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80b3000 tx=0x7f2fc80ab070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.232+0000 7fa08a2d6640 1 -- 172.21.15.46:0/324800723 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08404e170 con 0x7fa084158090 2023-10-30T13:23:30.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.236+0000 7fa080ff9640 1 -- 172.21.15.46:0/324800723 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa0740b7040 con 0x7fa084158090 2023-10-30T13:23:30.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.240+0000 7fa080ff9640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0700664a0 0x7fa070068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.240+0000 7f2fc1ffb640 1 -- 172.21.15.46:0/4108420095 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2fbc051280 con 0x7f2fcc152ee0 2023-10-30T13:23:30.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.240+0000 7fa080ff9640 1 -- 172.21.15.46:0/324800723 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa074121e40 con 0x7fa084158090 2023-10-30T13:23:30.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.240+0000 7f1718ff9640 1 -- 172.21.15.46:0/838286783 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1714051720 con 0x7f171c159830 2023-10-30T13:23:30.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.240+0000 7fa083fff640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0700664a0 0x7fa070068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.240+0000 7fd688ff9640 1 -- 172.21.15.46:0/1937614388 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd67c0b7040 con 0x7fd68c06a970 2023-10-30T13:23:30.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.240+0000 7fa083fff640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0700664a0 0x7fa070068960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa078011310 tx=0x7fa078009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.244+0000 7fd688ff9640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6780664d0 0x7fd678068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.244+0000 7fd688ff9640 1 -- 172.21.15.46:0/1937614388 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd67c1217a0 con 0x7fd68c06a970 2023-10-30T13:23:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.244+0000 7fd688ff9640 1 -- 172.21.15.46:0/1937614388 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd67c121b80 con 0x7fd68c06a970 2023-10-30T13:23:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.248+0000 7fd68b7fe640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6780664d0 0x7fd678068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.248+0000 7fa080ff9640 1 -- 172.21.15.46:0/324800723 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa0740ef360 con 0x7fa084158090 2023-10-30T13:23:30.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.248+0000 7fd68b7fe640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6780664d0 0x7fd678068990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd680000fc0 tx=0x7fd680001cc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.480+0000 7f05bf7aa640 1 -- 172.21.15.46:0/3570582451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f05b80c68b0 con 0x7f05b806d620 2023-10-30T13:23:30.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.484+0000 7f05adffb640 1 -- 172.21.15.46:0/3570582451 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f05b4021090 con 0x7f05b806d620 2023-10-30T13:23:30.490 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-10-30T13:23:30.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.484+0000 7f05bf7aa640 1 -- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f059c0664d0 msgr2=0x7f059c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.484+0000 7f05bf7aa640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f059c0664d0 0x7f059c068990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f05b80c9ad0 tx=0x7f05a8008040 comp rx=0 tx=0).stop 2023-10-30T13:23:30.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.484+0000 7f05bf7aa640 1 -- 172.21.15.46:0/3570582451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05b806d620 msgr2=0x7f05b81fc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.484+0000 7f05bf7aa640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05b806d620 0x7f05b81fc2a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f05b4000f70 tx=0x7f05b400c890 comp rx=0 tx=0).stop 2023-10-30T13:23:30.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.488+0000 7f05bf7aa640 1 -- 172.21.15.46:0/3570582451 shutdown_connections 2023-10-30T13:23:30.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.488+0000 7f05bf7aa640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f059c0664d0 0x7f059c068990 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.488+0000 7f05bf7aa640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8156020 0x7f05b814e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.488+0000 7f05bf7aa640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b814c140 0x7f05b80bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.488+0000 7f05bf7aa640 1 --2- 172.21.15.46:0/3570582451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05b806d620 0x7f05b81fc2a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.488+0000 7f05bf7aa640 1 -- 172.21.15.46:0/3570582451 >> 172.21.15.46:0/3570582451 conn(0x7f05b80cac50 msgr2=0x7f05b80c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.488+0000 7f05bf7aa640 1 -- 172.21.15.46:0/3570582451 shutdown_connections 2023-10-30T13:23:30.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.488+0000 7f05bf7aa640 1 -- 172.21.15.46:0/3570582451 wait complete. 2023-10-30T13:23:30.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2023-10-30T13:23:30.525 DEBUG:teuthology.parallel:result is None 2023-10-30T13:23:30.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.552+0000 7f1711ffb640 1 -- 172.21.15.46:0/838286783 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f16f40057e0 con 0x7f171c159830 2023-10-30T13:23:30.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.552+0000 7f1718ff9640 1 -- 172.21.15.46:0/838286783 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1714015020 con 0x7f171c159830 2023-10-30T13:23:30.561 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-10-30T13:23:30.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7fa348b3e640 1 -- 172.21.15.46:0/1131061868 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa3440b0af0 con 0x7fa344152b80 2023-10-30T13:23:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7fa33affd640 1 -- 172.21.15.46:0/1131061868 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa3340c7020 con 0x7fa344152b80 2023-10-30T13:23:30.563 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-10-30T13:23:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7f1711ffb640 1 -- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16fc0664a0 msgr2=0x7f16fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7f1711ffb640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16fc0664a0 0x7f16fc068960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1704019540 tx=0x7f17040a7070 comp rx=0 tx=0).stop 2023-10-30T13:23:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7f1711ffb640 1 -- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f171c159830 msgr2=0x7f171c1546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7f1711ffb640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f171c159830 0x7f171c1546e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f171400b410 tx=0x7f171400b8d0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7f1711ffb640 1 -- 172.21.15.46:0/838286783 shutdown_connections 2023-10-30T13:23:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7f1711ffb640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16fc0664a0 0x7f16fc068960 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7f1711ffb640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f171c159830 0x7f171c1546e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7f1711ffb640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171c153c60 0x7f171c1541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7f1711ffb640 1 --2- 172.21.15.46:0/838286783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f171c0c4430 0x7f171c147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7f1711ffb640 1 -- 172.21.15.46:0/838286783 >> 172.21.15.46:0/838286783 conn(0x7f171c06bfe0 msgr2=0x7f171c0c4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7f1711ffb640 1 -- 172.21.15.46:0/838286783 shutdown_connections 2023-10-30T13:23:30.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.556+0000 7f1711ffb640 1 -- 172.21.15.46:0/838286783 wait complete. 2023-10-30T13:23:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.560+0000 7fa348b3e640 1 -- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3240664d0 msgr2=0x7fa324068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.560+0000 7fa348b3e640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3240664d0 0x7fa324068990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0098b0 tx=0x7fa32c008040 comp rx=0 tx=0).stop 2023-10-30T13:23:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.560+0000 7fa348b3e640 1 -- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa344152b80 msgr2=0x7fa34414e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.560+0000 7fa348b3e640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa344152b80 0x7fa34414e330 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa3340ac000 tx=0x7fa3340196a0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.560+0000 7fa348b3e640 1 -- 172.21.15.46:0/1131061868 shutdown_connections 2023-10-30T13:23:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.560+0000 7fa348b3e640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3240664d0 0x7fa324068990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.560+0000 7fa348b3e640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa344152b80 0x7fa34414e330 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.560+0000 7fa348b3e640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3440c8430 0x7fa3440c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.560+0000 7fa348b3e640 1 --2- 172.21.15.46:0/1131061868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa34406ad90 0x7fa3440ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.560+0000 7fa348b3e640 1 -- 172.21.15.46:0/1131061868 >> 172.21.15.46:0/1131061868 conn(0x7fa3440c7910 msgr2=0x7fa344150610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.560+0000 7fa348b3e640 1 -- 172.21.15.46:0/1131061868 shutdown_connections 2023-10-30T13:23:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.560+0000 7fa348b3e640 1 -- 172.21.15.46:0/1131061868 wait complete. 2023-10-30T13:23:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.564+0000 7fa208609640 1 -- 172.21.15.46:0/1166290944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa20004e170 con 0x7fa2000c4ee0 2023-10-30T13:23:30.573 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-10-30T13:23:30.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.564+0000 7fa1f6ffd640 1 -- 172.21.15.46:0/1166290944 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa1f80f3010 con 0x7fa2000c4ee0 2023-10-30T13:23:30.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f94e8006ca0 con 0x7f94e80452b0 2023-10-30T13:23:30.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7f2fd34c9640 1 -- 172.21.15.46:0/4108420095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2fcc06aed0 con 0x7f2fcc152ee0 2023-10-30T13:23:30.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7fa208609640 1 -- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1e40665a0 msgr2=0x7fa1e4068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7fa208609640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1e40665a0 0x7fa1e4068a60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc001630 tx=0x7fa1fc009040 comp rx=0 tx=0).stop 2023-10-30T13:23:30.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7fa208609640 1 -- 172.21.15.46:0/1166290944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2000c4ee0 msgr2=0x7fa2001e7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7fa208609640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2000c4ee0 0x7fa2001e7080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80b2040 tx=0x7fa1f801bf90 comp rx=0 tx=0).stop 2023-10-30T13:23:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7fa208609640 1 -- 172.21.15.46:0/1166290944 shutdown_connections 2023-10-30T13:23:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7fa208609640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1e40665a0 0x7fa1e4068a60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7fa208609640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2000c4ee0 0x7fa2001e7080 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7fa208609640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa200161be0 0x7fa200070930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7f94dbfff640 1 -- 172.21.15.46:0/1501669083 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f94e4021090 con 0x7f94e80452b0 2023-10-30T13:23:30.577 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-10-30T13:23:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7f2fc1ffb640 1 -- 172.21.15.46:0/4108420095 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2fbc011e60 con 0x7f2fcc152ee0 2023-10-30T13:23:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7fa208609640 1 --2- 172.21.15.46:0/1166290944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa200158090 0x7fa2000c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7fa208609640 1 -- 172.21.15.46:0/1166290944 >> 172.21.15.46:0/1166290944 conn(0x7fa2000c7910 msgr2=0x7fa200160fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7fa208609640 1 -- 172.21.15.46:0/1166290944 shutdown_connections 2023-10-30T13:23:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7fa208609640 1 -- 172.21.15.46:0/1166290944 wait complete. 2023-10-30T13:23:30.579 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-10-30T13:23:30.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94d40664f0 msgr2=0x7f94d40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7f94f5b3e640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94d40664f0 0x7f94d40689b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0af000 tx=0x7f94dc0a7070 comp rx=0 tx=0).stop 2023-10-30T13:23:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e80452b0 msgr2=0x7f94e817fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.568+0000 7f94f5b3e640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e80452b0 0x7f94e817fb20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f94e400c420 tx=0x7f94e400c8e0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 shutdown_connections 2023-10-30T13:23:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f94f5b3e640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94d40664f0 0x7f94d40689b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f94f5b3e640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94e80dd3b0 0x7f94e817c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f94f5b3e640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e804c4b0 0x7f94e80e30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f94f5b3e640 1 --2- 172.21.15.46:0/1501669083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e80452b0 0x7f94e817fb20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 >> 172.21.15.46:0/1501669083 conn(0x7f94e8050310 msgr2=0x7f94e80d88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f2fd34c9640 1 -- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2fb00664d0 msgr2=0x7f2fb0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f2fd34c9640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2fb00664d0 0x7f2fb0068990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80b3000 tx=0x7f2fc80ab070 comp rx=0 tx=0).stop 2023-10-30T13:23:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 shutdown_connections 2023-10-30T13:23:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f94f5b3e640 1 -- 172.21.15.46:0/1501669083 wait complete. 2023-10-30T13:23:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f2fd34c9640 1 -- 172.21.15.46:0/4108420095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fcc152ee0 msgr2=0x7f2fcc149f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f2fd34c9640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fcc152ee0 0x7f2fcc149f90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00b410 tx=0x7f2fbc00b8d0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f2fd34c9640 1 -- 172.21.15.46:0/4108420095 shutdown_connections 2023-10-30T13:23:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f2fd34c9640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2fb00664d0 0x7f2fb0068990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f2fd34c9640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fcc156290 0x7f2fcc142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f2fd34c9640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fcc152ee0 0x7f2fcc149f90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f2fd34c9640 1 --2- 172.21.15.46:0/4108420095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2fcc0bfec0 0x7f2fcc144ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f2fd34c9640 1 -- 172.21.15.46:0/4108420095 >> 172.21.15.46:0/4108420095 conn(0x7f2fcc0c7910 msgr2=0x7f2fcc1550d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f2fd34c9640 1 -- 172.21.15.46:0/4108420095 shutdown_connections 2023-10-30T13:23:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.572+0000 7f2fd34c9640 1 -- 172.21.15.46:0/4108420095 wait complete. 2023-10-30T13:23:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.580+0000 7fd6727fc640 1 -- 172.21.15.46:0/1937614388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd6640057e0 con 0x7fd68c06a970 2023-10-30T13:23:30.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.584+0000 7fd688ff9640 1 -- 172.21.15.46:0/1937614388 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd67c0b5090 con 0x7fd68c06a970 2023-10-30T13:23:30.590 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-10-30T13:23:30.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.588+0000 7fd6727fc640 1 -- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6780664d0 msgr2=0x7fd678068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.588+0000 7fd6727fc640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6780664d0 0x7fd678068990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd680000fc0 tx=0x7fd680001cc0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.588+0000 7fd6727fc640 1 -- 172.21.15.46:0/1937614388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd68c06a970 msgr2=0x7fd68c1576c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.588+0000 7fd6727fc640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd68c06a970 0x7fd68c1576c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0af000 tx=0x7fd67c01b850 comp rx=0 tx=0).stop 2023-10-30T13:23:30.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.588+0000 7fd6727fc640 1 -- 172.21.15.46:0/1937614388 shutdown_connections 2023-10-30T13:23:30.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.588+0000 7fd6727fc640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6780664d0 0x7fd678068990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.588+0000 7fd6727fc640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0c2d90 0x7fd68c0c3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.588+0000 7fd6727fc640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68c156140 0x7fd68c1565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.588+0000 7fd6727fc640 1 --2- 172.21.15.46:0/1937614388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd68c06a970 0x7fd68c1576c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.588+0000 7fd6727fc640 1 -- 172.21.15.46:0/1937614388 >> 172.21.15.46:0/1937614388 conn(0x7fd68c0c7910 msgr2=0x7fd68c1f2580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.588+0000 7fd6727fc640 1 -- 172.21.15.46:0/1937614388 shutdown_connections 2023-10-30T13:23:30.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.588+0000 7fd6727fc640 1 -- 172.21.15.46:0/1937614388 wait complete. 2023-10-30T13:23:30.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.592+0000 7fa08a2d6640 1 -- 172.21.15.46:0/324800723 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa08415dfe0 con 0x7fa084158090 2023-10-30T13:23:30.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.592+0000 7fa080ff9640 1 -- 172.21.15.46:0/324800723 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa0740b5090 con 0x7fa084158090 2023-10-30T13:23:30.600 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-10-30T13:23:30.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.596+0000 7fa08a2d6640 1 -- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0700664a0 msgr2=0x7fa070068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.596+0000 7fa08a2d6640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0700664a0 0x7fa070068960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa078011310 tx=0x7fa078009190 comp rx=0 tx=0).stop 2023-10-30T13:23:30.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.596+0000 7fa08a2d6640 1 -- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa084158090 msgr2=0x7fa0840c5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.596+0000 7fa08a2d6640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa084158090 0x7fa0840c5760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa07401c880 tx=0x7fa07401b9f0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.606 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2023-10-30T13:23:30.607 DEBUG:teuthology.parallel:result is None 2023-10-30T13:23:30.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.600+0000 7fa08a2d6640 1 -- 172.21.15.46:0/324800723 shutdown_connections 2023-10-30T13:23:30.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.600+0000 7fa08a2d6640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0700664a0 0x7fa070068960 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.600+0000 7fa08a2d6640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa08414cbd0 0x7fa08414d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.600+0000 7fa08a2d6640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa084158090 0x7fa0840c5760 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.600+0000 7fa08a2d6640 1 --2- 172.21.15.46:0/324800723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa08406a3d0 0x7fa084154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.600+0000 7fa08a2d6640 1 -- 172.21.15.46:0/324800723 >> 172.21.15.46:0/324800723 conn(0x7fa0840c7910 msgr2=0x7fa0840c1200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.600+0000 7fa08a2d6640 1 -- 172.21.15.46:0/324800723 shutdown_connections 2023-10-30T13:23:30.612 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2023-10-30T13:23:30.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.600+0000 7fa08a2d6640 1 -- 172.21.15.46:0/324800723 wait complete. 2023-10-30T13:23:30.612 DEBUG:teuthology.parallel:result is None 2023-10-30T13:23:30.614 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2023-10-30T13:23:30.614 DEBUG:teuthology.parallel:result is None 2023-10-30T13:23:30.615 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2023-10-30T13:23:30.615 DEBUG:teuthology.parallel:result is None 2023-10-30T13:23:30.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2023-10-30T13:23:30.618 DEBUG:teuthology.parallel:result is None 2023-10-30T13:23:30.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2023-10-30T13:23:30.627 DEBUG:teuthology.parallel:result is None 2023-10-30T13:23:30.631 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2023-10-30T13:23:30.631 DEBUG:teuthology.parallel:result is None 2023-10-30T13:23:30.631 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T13:23:30.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:23:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.808+0000 7f635aa09640 1 Processor -- start 2023-10-30T13:23:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.808+0000 7f635aa09640 1 -- start start 2023-10-30T13:23:30.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.808+0000 7f635aa09640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f63540c05d0 0x7f635413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.808+0000 7f635aa09640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f63540c6f60 0x7f635413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f635aa09640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63540c0a50 con 0x7f63540c6b80 2023-10-30T13:23:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f635aa09640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f635406f6e0 con 0x7f63540c05d0 2023-10-30T13:23:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f635aa09640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f635406f9c0 con 0x7f63540c6f60 2023-10-30T13:23:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f63537fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f63540c05d0 0x7f635413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f63537fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f63540c05d0 0x7f635413ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39822/0 (socket says 172.21.15.46:39822) 2023-10-30T13:23:30.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f63537fe640 1 -- 172.21.15.46:0/1547116185 learned_addr learned my addr 172.21.15.46:0/1547116185 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f6358fa6640 1 --2- 172.21.15.46:0/1547116185 >> v2:172.21.15.154:3300/0 conn(0x7f63540c6f60 0x7f635413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f63537fe640 1 -- 172.21.15.46:0/1547116185 >> v2:172.21.15.154:3300/0 conn(0x7f63540c6f60 msgr2=0x7f635413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f63537fe640 1 --2- 172.21.15.46:0/1547116185 >> v2:172.21.15.154:3300/0 conn(0x7f63540c6f60 0x7f635413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f63537fe640 1 -- 172.21.15.46:0/1547116185 >> v1:172.21.15.154:6789/0 conn(0x7f63540c6b80 legacy=0x7f63540bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f63537fe640 1 -- 172.21.15.46:0/1547116185 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635406fca0 con 0x7f63540c05d0 2023-10-30T13:23:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f63537fe640 1 --2- 172.21.15.46:0/1547116185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c05d0 0x7f635413ed90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f63540c89b0 tx=0x7f634801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94b5ac35f3db664 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f6352ffd640 1 -- 172.21.15.46:0/1547116185 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63480ab610 con 0x7f63540c05d0 2023-10-30T13:23:30.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f6352ffd640 1 -- 172.21.15.46:0/1547116185 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63480ab770 con 0x7f63540c05d0 2023-10-30T13:23:30.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.812+0000 7f6352ffd640 1 -- 172.21.15.46:0/1547116185 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63480abb00 con 0x7f63540c05d0 2023-10-30T13:23:30.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 -- 172.21.15.46:0/1547116185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c05d0 msgr2=0x7f635413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 --2- 172.21.15.46:0/1547116185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c05d0 0x7f635413ed90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f63540c89b0 tx=0x7f634801c500 comp rx=0 tx=0).stop 2023-10-30T13:23:30.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 -- 172.21.15.46:0/1547116185 shutdown_connections 2023-10-30T13:23:30.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 --2- 172.21.15.46:0/1547116185 >> v2:172.21.15.154:3300/0 conn(0x7f63540c6f60 0x7f635413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 --2- 172.21.15.46:0/1547116185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c05d0 0x7f635413ed90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 -- 172.21.15.46:0/1547116185 >> 172.21.15.46:0/1547116185 conn(0x7f63540c7910 msgr2=0x7f63540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 -- 172.21.15.46:0/1547116185 shutdown_connections 2023-10-30T13:23:30.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 -- 172.21.15.46:0/1547116185 wait complete. 2023-10-30T13:23:30.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 Processor -- start 2023-10-30T13:23:30.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 -- start start 2023-10-30T13:23:30.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63540c05d0 0x7f6354152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63540c6b80 0x7f6354154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c6f60 0x7f6354154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6354163990 con 0x7f63540c6f60 2023-10-30T13:23:30.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63540caaf0 con 0x7f63540c05d0 2023-10-30T13:23:30.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f635aa09640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63540c1760 con 0x7f63540c6b80 2023-10-30T13:23:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f6358fa6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c6f60 0x7f6354154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f6358fa6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c6f60 0x7f6354154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39834/0 (socket says 172.21.15.46:39834) 2023-10-30T13:23:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f6358fa6640 1 -- 172.21.15.46:0/1946573028 learned_addr learned my addr 172.21.15.46:0/1946573028 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f63537fe640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63540c6b80 0x7f6354154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.816+0000 7f6353fff640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63540c05d0 0x7f6354152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6358fa6640 1 -- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63540c6b80 msgr2=0x7f6354154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6358fa6640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63540c6b80 0x7f6354154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6358fa6640 1 -- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63540c05d0 msgr2=0x7f6354152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6358fa6640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63540c05d0 0x7f6354152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6358fa6640 1 -- 172.21.15.46:0/1946573028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63480094d0 con 0x7f63540c6f60 2023-10-30T13:23:30.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6358fa6640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c6f60 0x7f6354154c60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f634c007e90 tx=0x7f634c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6350ff9640 1 -- 172.21.15.46:0/1946573028 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c014420 con 0x7f63540c6f60 2023-10-30T13:23:30.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6350ff9640 1 -- 172.21.15.46:0/1946573028 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634c014580 con 0x7f63540c6f60 2023-10-30T13:23:30.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f635aa09640 1 -- 172.21.15.46:0/1946573028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6354162aa0 con 0x7f63540c6f60 2023-10-30T13:23:30.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6350ff9640 1 -- 172.21.15.46:0/1946573028 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c014930 con 0x7f63540c6f60 2023-10-30T13:23:30.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f635aa09640 1 -- 172.21.15.46:0/1946573028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6354163060 con 0x7f63540c6f60 2023-10-30T13:23:30.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f635aa09640 1 -- 172.21.15.46:0/1946573028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6354163b40 con 0x7f63540c6f60 2023-10-30T13:23:30.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6350ff9640 1 -- 172.21.15.46:0/1946573028 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f634c014a90 con 0x7f63540c6f60 2023-10-30T13:23:30.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6350ff9640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63400664a0 0x7f6340068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6350ff9640 1 -- 172.21.15.46:0/1946573028 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f634c084750 con 0x7f63540c6f60 2023-10-30T13:23:30.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.820+0000 7f6353fff640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63400664a0 0x7f6340068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.824+0000 7f6353fff640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63400664a0 0x7f6340068960 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6344007fd0 tx=0x7f6344009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.824+0000 7f6350ff9640 1 -- 172.21.15.46:0/1946573028 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6354163b40 con 0x7f63540c6f60 2023-10-30T13:23:30.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:30.980+0000 7f635aa09640 1 -- 172.21.15.46:0/1946573028 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6354163b40 con 0x7f63400664a0 2023-10-30T13:23:31.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.000+0000 7f6350ff9640 1 -- 172.21.15.46:0/1946573028 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264486 (secure 0 0 0) 0x7f6354163b40 con 0x7f63400664a0 2023-10-30T13:23:31.009 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:31.010 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T13:23:31.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.004+0000 7f635aa09640 1 -- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63400664a0 msgr2=0x7f6340068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.004+0000 7f635aa09640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63400664a0 0x7f6340068960 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6344007fd0 tx=0x7f6344009040 comp rx=0 tx=0).stop 2023-10-30T13:23:31.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.004+0000 7f635aa09640 1 -- 172.21.15.46:0/1946573028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c6f60 msgr2=0x7f6354154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.004+0000 7f635aa09640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c6f60 0x7f6354154c60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f634c007e90 tx=0x7f634c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.004+0000 7f635aa09640 1 -- 172.21.15.46:0/1946573028 shutdown_connections 2023-10-30T13:23:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.004+0000 7f635aa09640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63400664a0 0x7f6340068960 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.004+0000 7f635aa09640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63540c6f60 0x7f6354154c60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.004+0000 7f635aa09640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63540c6b80 0x7f6354154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.004+0000 7f635aa09640 1 --2- 172.21.15.46:0/1946573028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63540c05d0 0x7f6354152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.004+0000 7f635aa09640 1 -- 172.21.15.46:0/1946573028 >> 172.21.15.46:0/1946573028 conn(0x7f63540c7910 msgr2=0x7f635406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:31.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.004+0000 7f635aa09640 1 -- 172.21.15.46:0/1946573028 shutdown_connections 2023-10-30T13:23:31.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.004+0000 7f635aa09640 1 -- 172.21.15.46:0/1946573028 wait complete. 2023-10-30T13:23:31.030 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-30T13:23:30.712717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235580,"kb_used_data":2812,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739140,"statfs":{"total":773094113280,"available":772852879360,"internally_reserved":0,"allocated":2879488,"data_stored":1749683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.043378"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612886+0000","last_change":"2023-10-30T13:23:19.719271+0000","last_active":"2023-10-30T13:23:21.612886+0000","last_peered":"2023-10-30T13:23:21.612886+0000","last_clean":"2023-10-30T13:23:21.612886+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T13:23:21.612886+0000","last_undegraded":"2023-10-30T13:23:21.612886+0000","last_fullsized":"2023-10-30T13:23:21.612886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:41:37.614073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670716+0000","last_change":"2023-10-30T13:23:19.687152+0000","last_active":"2023-10-30T13:23:20.670716+0000","last_peered":"2023-10-30T13:23:20.670716+0000","last_clean":"2023-10-30T13:23:20.670716+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T13:23:20.670716+0000","last_undegraded":"2023-10-30T13:23:20.670716+0000","last_fullsized":"2023-10-30T13:23:20.670716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107815+0000","last_change":"2023-10-30T13:23:19.689554+0000","last_active":"2023-10-30T13:23:21.107815+0000","last_peered":"2023-10-30T13:23:21.107815+0000","last_clean":"2023-10-30T13:23:21.107815+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T13:23:21.107815+0000","last_undegraded":"2023-10-30T13:23:21.107815+0000","last_fullsized":"2023-10-30T13:23:21.107815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:56:13.795044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611622+0000","last_change":"2023-10-30T13:23:19.714018+0000","last_active":"2023-10-30T13:23:21.611622+0000","last_peered":"2023-10-30T13:23:21.611622+0000","last_clean":"2023-10-30T13:23:21.611622+0000","last_became_active":"2023-10-30T13:23:19.711673+0000","last_became_peered":"2023-10-30T13:23:19.711673+0000","last_unstale":"2023-10-30T13:23:21.611622+0000","last_undegraded":"2023-10-30T13:23:21.611622+0000","last_fullsized":"2023-10-30T13:23:21.611622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:28:16.658840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611546+0000","last_change":"2023-10-30T13:23:19.719999+0000","last_active":"2023-10-30T13:23:21.611546+0000","last_peered":"2023-10-30T13:23:21.611546+0000","last_clean":"2023-10-30T13:23:21.611546+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T13:23:21.611546+0000","last_undegraded":"2023-10-30T13:23:21.611546+0000","last_fullsized":"2023-10-30T13:23:21.611546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:46:31.287013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612300+0000","last_change":"2023-10-30T13:23:19.703474+0000","last_active":"2023-10-30T13:23:21.612300+0000","last_peered":"2023-10-30T13:23:21.612300+0000","last_clean":"2023-10-30T13:23:21.612300+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T13:23:21.612300+0000","last_undegraded":"2023-10-30T13:23:21.612300+0000","last_fullsized":"2023-10-30T13:23:21.612300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:53:58.118821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613234+0000","last_change":"2023-10-30T13:23:19.726985+0000","last_active":"2023-10-30T13:23:21.613234+0000","last_peered":"2023-10-30T13:23:21.613234+0000","last_clean":"2023-10-30T13:23:21.613234+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T13:23:21.613234+0000","last_undegraded":"2023-10-30T13:23:21.613234+0000","last_fullsized":"2023-10-30T13:23:21.613234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:51:13.152339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.004933+0000","last_change":"2023-10-30T13:23:19.714153+0000","last_active":"2023-10-30T13:23:21.004933+0000","last_peered":"2023-10-30T13:23:21.004933+0000","last_clean":"2023-10-30T13:23:21.004933+0000","last_became_active":"2023-10-30T13:23:19.713876+0000","last_became_peered":"2023-10-30T13:23:19.713876+0000","last_unstale":"2023-10-30T13:23:21.004933+0000","last_undegraded":"2023-10-30T13:23:21.004933+0000","last_fullsized":"2023-10-30T13:23:21.004933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:05:13.588780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613220+0000","last_change":"2023-10-30T13:23:19.724583+0000","last_active":"2023-10-30T13:23:21.613220+0000","last_peered":"2023-10-30T13:23:21.613220+0000","last_clean":"2023-10-30T13:23:21.613220+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T13:23:21.613220+0000","last_undegraded":"2023-10-30T13:23:21.613220+0000","last_fullsized":"2023-10-30T13:23:21.613220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:47.936982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677145+0000","last_change":"2023-10-30T13:23:19.704399+0000","last_active":"2023-10-30T13:23:20.677145+0000","last_peered":"2023-10-30T13:23:20.677145+0000","last_clean":"2023-10-30T13:23:20.677145+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T13:23:20.677145+0000","last_undegraded":"2023-10-30T13:23:20.677145+0000","last_fullsized":"2023-10-30T13:23:20.677145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676522+0000","last_change":"2023-10-30T13:23:19.726077+0000","last_active":"2023-10-30T13:23:20.676522+0000","last_peered":"2023-10-30T13:23:20.676522+0000","last_clean":"2023-10-30T13:23:20.676522+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T13:23:20.676522+0000","last_undegraded":"2023-10-30T13:23:20.676522+0000","last_fullsized":"2023-10-30T13:23:20.676522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675959+0000","last_change":"2023-10-30T13:23:19.720995+0000","last_active":"2023-10-30T13:23:20.675959+0000","last_peered":"2023-10-30T13:23:20.675959+0000","last_clean":"2023-10-30T13:23:20.675959+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T13:23:20.675959+0000","last_undegraded":"2023-10-30T13:23:20.675959+0000","last_fullsized":"2023-10-30T13:23:20.675959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:44.490568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005253+0000","last_change":"2023-10-30T13:23:19.692509+0000","last_active":"2023-10-30T13:23:21.005253+0000","last_peered":"2023-10-30T13:23:21.005253+0000","last_clean":"2023-10-30T13:23:21.005253+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T13:23:21.005253+0000","last_undegraded":"2023-10-30T13:23:21.005253+0000","last_fullsized":"2023-10-30T13:23:21.005253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:15:17.956816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107580+0000","last_change":"2023-10-30T13:23:19.685435+0000","last_active":"2023-10-30T13:23:21.107580+0000","last_peered":"2023-10-30T13:23:21.107580+0000","last_clean":"2023-10-30T13:23:21.107580+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T13:23:21.107580+0000","last_undegraded":"2023-10-30T13:23:21.107580+0000","last_fullsized":"2023-10-30T13:23:21.107580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:39.035298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670272+0000","last_change":"2023-10-30T13:23:19.706721+0000","last_active":"2023-10-30T13:23:20.670272+0000","last_peered":"2023-10-30T13:23:20.670272+0000","last_clean":"2023-10-30T13:23:20.670272+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T13:23:20.670272+0000","last_undegraded":"2023-10-30T13:23:20.670272+0000","last_fullsized":"2023-10-30T13:23:20.670272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.610717+0000","last_change":"2023-10-30T13:23:19.710432+0000","last_active":"2023-10-30T13:23:21.610717+0000","last_peered":"2023-10-30T13:23:21.610717+0000","last_clean":"2023-10-30T13:23:21.610717+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T13:23:21.610717+0000","last_undegraded":"2023-10-30T13:23:21.610717+0000","last_fullsized":"2023-10-30T13:23:21.610717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:48:51.992826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612857+0000","last_change":"2023-10-30T13:23:19.741225+0000","last_active":"2023-10-30T13:23:21.612857+0000","last_peered":"2023-10-30T13:23:21.612857+0000","last_clean":"2023-10-30T13:23:21.612857+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T13:23:21.612857+0000","last_undegraded":"2023-10-30T13:23:21.612857+0000","last_fullsized":"2023-10-30T13:23:21.612857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:27.310070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.608570+0000","last_change":"2023-10-30T13:23:19.701024+0000","last_active":"2023-10-30T13:23:21.608570+0000","last_peered":"2023-10-30T13:23:21.608570+0000","last_clean":"2023-10-30T13:23:21.608570+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T13:23:21.608570+0000","last_undegraded":"2023-10-30T13:23:21.608570+0000","last_fullsized":"2023-10-30T13:23:21.608570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:14.460383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612314+0000","last_change":"2023-10-30T13:23:19.723929+0000","last_active":"2023-10-30T13:23:21.612314+0000","last_peered":"2023-10-30T13:23:21.612314+0000","last_clean":"2023-10-30T13:23:21.612314+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T13:23:21.612314+0000","last_undegraded":"2023-10-30T13:23:21.612314+0000","last_fullsized":"2023-10-30T13:23:21.612314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:41.023731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674806+0000","last_change":"2023-10-30T13:23:19.725101+0000","last_active":"2023-10-30T13:23:20.674806+0000","last_peered":"2023-10-30T13:23:20.674806+0000","last_clean":"2023-10-30T13:23:20.674806+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T13:23:20.674806+0000","last_undegraded":"2023-10-30T13:23:20.674806+0000","last_fullsized":"2023-10-30T13:23:20.674806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107754+0000","last_change":"2023-10-30T13:23:19.721813+0000","last_active":"2023-10-30T13:23:21.107754+0000","last_peered":"2023-10-30T13:23:21.107754+0000","last_clean":"2023-10-30T13:23:21.107754+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T13:23:21.107754+0000","last_undegraded":"2023-10-30T13:23:21.107754+0000","last_fullsized":"2023-10-30T13:23:21.107754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:34:17.212695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612670+0000","last_change":"2023-10-30T13:23:19.724763+0000","last_active":"2023-10-30T13:23:21.612670+0000","last_peered":"2023-10-30T13:23:21.612670+0000","last_clean":"2023-10-30T13:23:21.612670+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T13:23:21.612670+0000","last_undegraded":"2023-10-30T13:23:21.612670+0000","last_fullsized":"2023-10-30T13:23:21.612670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:28:53.428578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005009+0000","last_change":"2023-10-30T13:23:19.716706+0000","last_active":"2023-10-30T13:23:21.005009+0000","last_peered":"2023-10-30T13:23:21.005009+0000","last_clean":"2023-10-30T13:23:21.005009+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T13:23:21.005009+0000","last_undegraded":"2023-10-30T13:23:21.005009+0000","last_fullsized":"2023-10-30T13:23:21.005009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:07.718271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.610920+0000","last_change":"2023-10-30T13:23:19.704791+0000","last_active":"2023-10-30T13:23:21.610920+0000","last_peered":"2023-10-30T13:23:21.610920+0000","last_clean":"2023-10-30T13:23:21.610920+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T13:23:21.610920+0000","last_undegraded":"2023-10-30T13:23:21.610920+0000","last_fullsized":"2023-10-30T13:23:21.610920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:54.750514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005422+0000","last_change":"2023-10-30T13:23:19.717027+0000","last_active":"2023-10-30T13:23:21.005422+0000","last_peered":"2023-10-30T13:23:21.005422+0000","last_clean":"2023-10-30T13:23:21.005422+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T13:23:21.005422+0000","last_undegraded":"2023-10-30T13:23:21.005422+0000","last_fullsized":"2023-10-30T13:23:21.005422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:38.788372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677247+0000","last_change":"2023-10-30T13:23:19.706002+0000","last_active":"2023-10-30T13:23:20.677247+0000","last_peered":"2023-10-30T13:23:20.677247+0000","last_clean":"2023-10-30T13:23:20.677247+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T13:23:20.677247+0000","last_undegraded":"2023-10-30T13:23:20.677247+0000","last_fullsized":"2023-10-30T13:23:20.677247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:46.882474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670564+0000","last_change":"2023-10-30T13:23:19.678472+0000","last_active":"2023-10-30T13:23:20.670564+0000","last_peered":"2023-10-30T13:23:20.670564+0000","last_clean":"2023-10-30T13:23:20.670564+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T13:23:20.670564+0000","last_undegraded":"2023-10-30T13:23:20.670564+0000","last_fullsized":"2023-10-30T13:23:20.670564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005470+0000","last_change":"2023-10-30T13:23:19.702155+0000","last_active":"2023-10-30T13:23:21.005470+0000","last_peered":"2023-10-30T13:23:21.005470+0000","last_clean":"2023-10-30T13:23:21.005470+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T13:23:21.005470+0000","last_undegraded":"2023-10-30T13:23:21.005470+0000","last_fullsized":"2023-10-30T13:23:21.005470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:30.131186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107903+0000","last_change":"2023-10-30T13:23:19.687820+0000","last_active":"2023-10-30T13:23:21.107903+0000","last_peered":"2023-10-30T13:23:21.107903+0000","last_clean":"2023-10-30T13:23:21.107903+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T13:23:21.107903+0000","last_undegraded":"2023-10-30T13:23:21.107903+0000","last_fullsized":"2023-10-30T13:23:21.107903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:46.088882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612897+0000","last_change":"2023-10-30T13:23:19.702442+0000","last_active":"2023-10-30T13:23:21.612897+0000","last_peered":"2023-10-30T13:23:21.612897+0000","last_clean":"2023-10-30T13:23:21.612897+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T13:23:21.612897+0000","last_undegraded":"2023-10-30T13:23:21.612897+0000","last_fullsized":"2023-10-30T13:23:21.612897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:58:50.083326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.608295+0000","last_change":"2023-10-30T13:23:19.711364+0000","last_active":"2023-10-30T13:23:21.608295+0000","last_peered":"2023-10-30T13:23:21.608295+0000","last_clean":"2023-10-30T13:23:21.608295+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T13:23:21.608295+0000","last_undegraded":"2023-10-30T13:23:21.608295+0000","last_fullsized":"2023-10-30T13:23:21.608295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:15.250397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674450+0000","last_change":"2023-10-30T13:23:19.698485+0000","last_active":"2023-10-30T13:23:20.674450+0000","last_peered":"2023-10-30T13:23:20.674450+0000","last_clean":"2023-10-30T13:23:20.674450+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T13:23:20.674450+0000","last_undegraded":"2023-10-30T13:23:20.674450+0000","last_fullsized":"2023-10-30T13:23:20.674450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675615+0000","last_change":"2023-10-30T13:23:19.703223+0000","last_active":"2023-10-30T13:23:20.675615+0000","last_peered":"2023-10-30T13:23:20.675615+0000","last_clean":"2023-10-30T13:23:20.675615+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T13:23:20.675615+0000","last_undegraded":"2023-10-30T13:23:20.675615+0000","last_fullsized":"2023-10-30T13:23:20.675615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:28.065471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005464+0000","last_change":"2023-10-30T13:23:18.963933+0000","last_active":"2023-10-30T13:23:21.005464+0000","last_peered":"2023-10-30T13:23:21.005464+0000","last_clean":"2023-10-30T13:23:21.005464+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T13:23:21.005464+0000","last_undegraded":"2023-10-30T13:23:21.005464+0000","last_fullsized":"2023-10-30T13:23:21.005464+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:48:58.196567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107913+0000","last_change":"2023-10-30T13:23:19.717819+0000","last_active":"2023-10-30T13:23:21.107913+0000","last_peered":"2023-10-30T13:23:21.107913+0000","last_clean":"2023-10-30T13:23:21.107913+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T13:23:21.107913+0000","last_undegraded":"2023-10-30T13:23:21.107913+0000","last_fullsized":"2023-10-30T13:23:21.107913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:15.403400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.990350+0000","last_change":"2023-10-30T13:23:18.976615+0000","last_active":"2023-10-30T13:23:20.990350+0000","last_peered":"2023-10-30T13:23:20.990350+0000","last_clean":"2023-10-30T13:23:20.990350+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T13:23:20.990350+0000","last_undegraded":"2023-10-30T13:23:20.990350+0000","last_fullsized":"2023-10-30T13:23:20.990350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:59.048396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.981386+0000","last_change":"2023-10-30T13:23:18.977357+0000","last_active":"2023-10-30T13:23:20.981386+0000","last_peered":"2023-10-30T13:23:20.981386+0000","last_clean":"2023-10-30T13:23:20.981386+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T13:23:20.981386+0000","last_undegraded":"2023-10-30T13:23:20.981386+0000","last_fullsized":"2023-10-30T13:23:20.981386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:53.502681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005000+0000","last_change":"2023-10-30T13:23:19.689197+0000","last_active":"2023-10-30T13:23:21.005000+0000","last_peered":"2023-10-30T13:23:21.005000+0000","last_clean":"2023-10-30T13:23:21.005000+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T13:23:21.005000+0000","last_undegraded":"2023-10-30T13:23:21.005000+0000","last_fullsized":"2023-10-30T13:23:21.005000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:59:47.946429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.989002+0000","last_change":"2023-10-30T13:23:18.989299+0000","last_active":"2023-10-30T13:23:20.989002+0000","last_peered":"2023-10-30T13:23:20.989002+0000","last_clean":"2023-10-30T13:23:20.989002+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T13:23:20.989002+0000","last_undegraded":"2023-10-30T13:23:20.989002+0000","last_fullsized":"2023-10-30T13:23:20.989002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:44.510184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670217+0000","last_change":"2023-10-30T13:23:19.679342+0000","last_active":"2023-10-30T13:23:20.670217+0000","last_peered":"2023-10-30T13:23:20.670217+0000","last_clean":"2023-10-30T13:23:20.670217+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T13:23:20.670217+0000","last_undegraded":"2023-10-30T13:23:20.670217+0000","last_fullsized":"2023-10-30T13:23:20.670217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677093+0000","last_change":"2023-10-30T13:23:18.989515+0000","last_active":"2023-10-30T13:23:20.677093+0000","last_peered":"2023-10-30T13:23:20.677093+0000","last_clean":"2023-10-30T13:23:20.677093+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T13:23:20.677093+0000","last_undegraded":"2023-10-30T13:23:20.677093+0000","last_fullsized":"2023-10-30T13:23:20.677093+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:35.133762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612308+0000","last_change":"2023-10-30T13:23:19.732859+0000","last_active":"2023-10-30T13:23:21.612308+0000","last_peered":"2023-10-30T13:23:21.612308+0000","last_clean":"2023-10-30T13:23:21.612308+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T13:23:21.612308+0000","last_undegraded":"2023-10-30T13:23:21.612308+0000","last_fullsized":"2023-10-30T13:23:21.612308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:27.629521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.989659+0000","last_change":"2023-10-30T13:23:18.991454+0000","last_active":"2023-10-30T13:23:20.989659+0000","last_peered":"2023-10-30T13:23:20.989659+0000","last_clean":"2023-10-30T13:23:20.989659+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T13:23:20.989659+0000","last_undegraded":"2023-10-30T13:23:20.989659+0000","last_fullsized":"2023-10-30T13:23:20.989659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:13.450999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107915+0000","last_change":"2023-10-30T13:23:19.722347+0000","last_active":"2023-10-30T13:23:21.107915+0000","last_peered":"2023-10-30T13:23:21.107915+0000","last_clean":"2023-10-30T13:23:21.107915+0000","last_became_active":"2023-10-30T13:23:19.722077+0000","last_became_peered":"2023-10-30T13:23:19.722077+0000","last_unstale":"2023-10-30T13:23:21.107915+0000","last_undegraded":"2023-10-30T13:23:21.107915+0000","last_fullsized":"2023-10-30T13:23:21.107915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:23:21.544691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.965367+0000","last_change":"2023-10-30T13:23:18.982337+0000","last_active":"2023-10-30T13:23:24.965367+0000","last_peered":"2023-10-30T13:23:24.965367+0000","last_clean":"2023-10-30T13:23:24.965367+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T13:23:24.965367+0000","last_undegraded":"2023-10-30T13:23:24.965367+0000","last_fullsized":"2023-10-30T13:23:24.965367+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:09:39.996740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676299+0000","last_change":"2023-10-30T13:23:19.706643+0000","last_active":"2023-10-30T13:23:20.676299+0000","last_peered":"2023-10-30T13:23:20.676299+0000","last_clean":"2023-10-30T13:23:20.676299+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T13:23:20.676299+0000","last_undegraded":"2023-10-30T13:23:20.676299+0000","last_fullsized":"2023-10-30T13:23:20.676299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:29.238430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.968446+0000","last_change":"2023-10-30T13:23:18.969882+0000","last_active":"2023-10-30T13:23:24.968446+0000","last_peered":"2023-10-30T13:23:24.968446+0000","last_clean":"2023-10-30T13:23:24.968446+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T13:23:24.968446+0000","last_undegraded":"2023-10-30T13:23:24.968446+0000","last_fullsized":"2023-10-30T13:23:24.968446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:17.093448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611588+0000","last_change":"2023-10-30T13:23:19.728295+0000","last_active":"2023-10-30T13:23:21.611588+0000","last_peered":"2023-10-30T13:23:21.611588+0000","last_clean":"2023-10-30T13:23:21.611588+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T13:23:21.611588+0000","last_undegraded":"2023-10-30T13:23:21.611588+0000","last_fullsized":"2023-10-30T13:23:21.611588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:37:54.545985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.966553+0000","last_change":"2023-10-30T13:23:18.971632+0000","last_active":"2023-10-30T13:23:24.966553+0000","last_peered":"2023-10-30T13:23:24.966553+0000","last_clean":"2023-10-30T13:23:24.966553+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T13:23:24.966553+0000","last_undegraded":"2023-10-30T13:23:24.966553+0000","last_fullsized":"2023-10-30T13:23:24.966553+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:54.428499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107653+0000","last_change":"2023-10-30T13:23:19.718168+0000","last_active":"2023-10-30T13:23:21.107653+0000","last_peered":"2023-10-30T13:23:21.107653+0000","last_clean":"2023-10-30T13:23:21.107653+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T13:23:21.107653+0000","last_undegraded":"2023-10-30T13:23:21.107653+0000","last_fullsized":"2023-10-30T13:23:21.107653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:54.832358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.989192+0000","last_change":"2023-10-30T13:23:18.978420+0000","last_active":"2023-10-30T13:23:20.989192+0000","last_peered":"2023-10-30T13:23:20.989192+0000","last_clean":"2023-10-30T13:23:20.989192+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T13:23:20.989192+0000","last_undegraded":"2023-10-30T13:23:20.989192+0000","last_fullsized":"2023-10-30T13:23:20.989192+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:10:44.511282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611634+0000","last_change":"2023-10-30T13:23:19.721434+0000","last_active":"2023-10-30T13:23:21.611634+0000","last_peered":"2023-10-30T13:23:21.611634+0000","last_clean":"2023-10-30T13:23:21.611634+0000","last_became_active":"2023-10-30T13:23:19.720704+0000","last_became_peered":"2023-10-30T13:23:19.720704+0000","last_unstale":"2023-10-30T13:23:21.611634+0000","last_undegraded":"2023-10-30T13:23:21.611634+0000","last_fullsized":"2023-10-30T13:23:21.611634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:45.728513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674378+0000","last_change":"2023-10-30T13:23:18.972707+0000","last_active":"2023-10-30T13:23:20.674378+0000","last_peered":"2023-10-30T13:23:20.674378+0000","last_clean":"2023-10-30T13:23:20.674378+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T13:23:20.674378+0000","last_undegraded":"2023-10-30T13:23:20.674378+0000","last_fullsized":"2023-10-30T13:23:20.674378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674810+0000","last_change":"2023-10-30T13:23:19.698274+0000","last_active":"2023-10-30T13:23:20.674810+0000","last_peered":"2023-10-30T13:23:20.674810+0000","last_clean":"2023-10-30T13:23:20.674810+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T13:23:20.674810+0000","last_undegraded":"2023-10-30T13:23:20.674810+0000","last_fullsized":"2023-10-30T13:23:20.674810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.965136+0000","last_change":"2023-10-30T13:23:18.948296+0000","last_active":"2023-10-30T13:23:24.965136+0000","last_peered":"2023-10-30T13:23:24.965136+0000","last_clean":"2023-10-30T13:23:24.965136+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T13:23:24.965136+0000","last_undegraded":"2023-10-30T13:23:24.965136+0000","last_fullsized":"2023-10-30T13:23:24.965136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:55.308329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611416+0000","last_change":"2023-10-30T13:23:19.728647+0000","last_active":"2023-10-30T13:23:21.611416+0000","last_peered":"2023-10-30T13:23:21.611416+0000","last_clean":"2023-10-30T13:23:21.611416+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T13:23:21.611416+0000","last_undegraded":"2023-10-30T13:23:21.611416+0000","last_fullsized":"2023-10-30T13:23:21.611416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:09.123809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.986186+0000","last_change":"2023-10-30T13:23:18.962343+0000","last_active":"2023-10-30T13:23:20.986186+0000","last_peered":"2023-10-30T13:23:20.986186+0000","last_clean":"2023-10-30T13:23:20.986186+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T13:23:20.986186+0000","last_undegraded":"2023-10-30T13:23:20.986186+0000","last_fullsized":"2023-10-30T13:23:20.986186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:36.082164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611928+0000","last_change":"2023-10-30T13:23:19.720770+0000","last_active":"2023-10-30T13:23:21.611928+0000","last_peered":"2023-10-30T13:23:21.611928+0000","last_clean":"2023-10-30T13:23:21.611928+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T13:23:21.611928+0000","last_undegraded":"2023-10-30T13:23:21.611928+0000","last_fullsized":"2023-10-30T13:23:21.611928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:37.378858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.962456+0000","last_change":"2023-10-30T13:23:18.949081+0000","last_active":"2023-10-30T13:23:24.962456+0000","last_peered":"2023-10-30T13:23:24.962456+0000","last_clean":"2023-10-30T13:23:24.962456+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T13:23:24.962456+0000","last_undegraded":"2023-10-30T13:23:24.962456+0000","last_fullsized":"2023-10-30T13:23:24.962456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:00.356885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613552+0000","last_change":"2023-10-30T13:23:19.724180+0000","last_active":"2023-10-30T13:23:21.613552+0000","last_peered":"2023-10-30T13:23:21.613552+0000","last_clean":"2023-10-30T13:23:21.613552+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T13:23:21.613552+0000","last_undegraded":"2023-10-30T13:23:21.613552+0000","last_fullsized":"2023-10-30T13:23:21.613552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:14.049454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.981649+0000","last_change":"2023-10-30T13:23:18.970780+0000","last_active":"2023-10-30T13:23:20.981649+0000","last_peered":"2023-10-30T13:23:20.981649+0000","last_clean":"2023-10-30T13:23:20.981649+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T13:23:20.981649+0000","last_undegraded":"2023-10-30T13:23:20.981649+0000","last_fullsized":"2023-10-30T13:23:20.981649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:38:51.113629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611979+0000","last_change":"2023-10-30T13:23:19.704834+0000","last_active":"2023-10-30T13:23:21.611979+0000","last_peered":"2023-10-30T13:23:21.611979+0000","last_clean":"2023-10-30T13:23:21.611979+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T13:23:21.611979+0000","last_undegraded":"2023-10-30T13:23:21.611979+0000","last_fullsized":"2023-10-30T13:23:21.611979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:11.266083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.983909+0000","last_change":"2023-10-30T13:23:18.977920+0000","last_active":"2023-10-30T13:23:20.983909+0000","last_peered":"2023-10-30T13:23:20.983909+0000","last_clean":"2023-10-30T13:23:20.983909+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T13:23:20.983909+0000","last_undegraded":"2023-10-30T13:23:20.983909+0000","last_fullsized":"2023-10-30T13:23:20.983909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:07:59.139948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611883+0000","last_change":"2023-10-30T13:23:19.724497+0000","last_active":"2023-10-30T13:23:21.611883+0000","last_peered":"2023-10-30T13:23:21.611883+0000","last_clean":"2023-10-30T13:23:21.611883+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T13:23:21.611883+0000","last_undegraded":"2023-10-30T13:23:21.611883+0000","last_fullsized":"2023-10-30T13:23:21.611883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:23.836868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.686506+0000","last_change":"2023-10-30T13:23:08.208031+0000","last_active":"2023-10-30T13:23:19.686506+0000","last_peered":"2023-10-30T13:23:19.686506+0000","last_clean":"2023-10-30T13:23:19.686506+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T13:23:19.686506+0000","last_undegraded":"2023-10-30T13:23:19.686506+0000","last_fullsized":"2023-10-30T13:23:19.686506+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_clean_scrub_stamp":"2023-10-30T13:23:07.179866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:22.691367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.989305+0000","last_change":"2023-10-30T13:23:18.979328+0000","last_active":"2023-10-30T13:23:20.989305+0000","last_peered":"2023-10-30T13:23:20.989305+0000","last_clean":"2023-10-30T13:23:20.989305+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T13:23:20.989305+0000","last_undegraded":"2023-10-30T13:23:20.989305+0000","last_fullsized":"2023-10-30T13:23:20.989305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:04.942593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005198+0000","last_change":"2023-10-30T13:23:19.711222+0000","last_active":"2023-10-30T13:23:21.005198+0000","last_peered":"2023-10-30T13:23:21.005198+0000","last_clean":"2023-10-30T13:23:21.005198+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T13:23:21.005198+0000","last_undegraded":"2023-10-30T13:23:21.005198+0000","last_fullsized":"2023-10-30T13:23:21.005198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:07.213498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.988400+0000","last_change":"2023-10-30T13:23:18.985318+0000","last_active":"2023-10-30T13:23:20.988400+0000","last_peered":"2023-10-30T13:23:20.988400+0000","last_clean":"2023-10-30T13:23:20.988400+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T13:23:20.988400+0000","last_undegraded":"2023-10-30T13:23:20.988400+0000","last_fullsized":"2023-10-30T13:23:20.988400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:16.754084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107580+0000","last_change":"2023-10-30T13:23:19.717938+0000","last_active":"2023-10-30T13:23:21.107580+0000","last_peered":"2023-10-30T13:23:21.107580+0000","last_clean":"2023-10-30T13:23:21.107580+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T13:23:21.107580+0000","last_undegraded":"2023-10-30T13:23:21.107580+0000","last_fullsized":"2023-10-30T13:23:21.107580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:56:01.624954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.983889+0000","last_change":"2023-10-30T13:23:18.949565+0000","last_active":"2023-10-30T13:23:20.983889+0000","last_peered":"2023-10-30T13:23:20.983889+0000","last_clean":"2023-10-30T13:23:20.983889+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T13:23:20.983889+0000","last_undegraded":"2023-10-30T13:23:20.983889+0000","last_fullsized":"2023-10-30T13:23:20.983889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:55:46.029368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670291+0000","last_change":"2023-10-30T13:23:19.678037+0000","last_active":"2023-10-30T13:23:20.670291+0000","last_peered":"2023-10-30T13:23:20.670291+0000","last_clean":"2023-10-30T13:23:20.670291+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T13:23:20.670291+0000","last_undegraded":"2023-10-30T13:23:20.670291+0000","last_fullsized":"2023-10-30T13:23:20.670291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.989242+0000","last_change":"2023-10-30T13:23:18.984535+0000","last_active":"2023-10-30T13:23:20.989242+0000","last_peered":"2023-10-30T13:23:20.989242+0000","last_clean":"2023-10-30T13:23:20.989242+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T13:23:20.989242+0000","last_undegraded":"2023-10-30T13:23:20.989242+0000","last_fullsized":"2023-10-30T13:23:20.989242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:28.405405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675980+0000","last_change":"2023-10-30T13:23:19.691867+0000","last_active":"2023-10-30T13:23:20.675980+0000","last_peered":"2023-10-30T13:23:20.675980+0000","last_clean":"2023-10-30T13:23:20.675980+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T13:23:20.675980+0000","last_undegraded":"2023-10-30T13:23:20.675980+0000","last_fullsized":"2023-10-30T13:23:20.675980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:02.276042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670494+0000","last_change":"2023-10-30T13:23:18.958210+0000","last_active":"2023-10-30T13:23:20.670494+0000","last_peered":"2023-10-30T13:23:20.670494+0000","last_clean":"2023-10-30T13:23:20.670494+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T13:23:20.670494+0000","last_undegraded":"2023-10-30T13:23:20.670494+0000","last_fullsized":"2023-10-30T13:23:20.670494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:34.653005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613343+0000","last_change":"2023-10-30T13:23:19.713174+0000","last_active":"2023-10-30T13:23:21.613343+0000","last_peered":"2023-10-30T13:23:21.613343+0000","last_clean":"2023-10-30T13:23:21.613343+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T13:23:21.613343+0000","last_undegraded":"2023-10-30T13:23:21.613343+0000","last_fullsized":"2023-10-30T13:23:21.613343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:43.400223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.988980+0000","last_change":"2023-10-30T13:23:18.985185+0000","last_active":"2023-10-30T13:23:20.988980+0000","last_peered":"2023-10-30T13:23:20.988980+0000","last_clean":"2023-10-30T13:23:20.988980+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T13:23:20.988980+0000","last_undegraded":"2023-10-30T13:23:20.988980+0000","last_fullsized":"2023-10-30T13:23:20.988980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:36.230072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670523+0000","last_change":"2023-10-30T13:23:19.680185+0000","last_active":"2023-10-30T13:23:20.670523+0000","last_peered":"2023-10-30T13:23:20.670523+0000","last_clean":"2023-10-30T13:23:20.670523+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T13:23:20.670523+0000","last_undegraded":"2023-10-30T13:23:20.670523+0000","last_fullsized":"2023-10-30T13:23:20.670523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670731+0000","last_change":"2023-10-30T13:23:18.978085+0000","last_active":"2023-10-30T13:23:20.670731+0000","last_peered":"2023-10-30T13:23:20.670731+0000","last_clean":"2023-10-30T13:23:20.670731+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T13:23:20.670731+0000","last_undegraded":"2023-10-30T13:23:20.670731+0000","last_fullsized":"2023-10-30T13:23:20.670731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:47.588972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005467+0000","last_change":"2023-10-30T13:23:19.704291+0000","last_active":"2023-10-30T13:23:21.005467+0000","last_peered":"2023-10-30T13:23:21.005467+0000","last_clean":"2023-10-30T13:23:21.005467+0000","last_became_active":"2023-10-30T13:23:19.704066+0000","last_became_peered":"2023-10-30T13:23:19.704066+0000","last_unstale":"2023-10-30T13:23:21.005467+0000","last_undegraded":"2023-10-30T13:23:21.005467+0000","last_fullsized":"2023-10-30T13:23:21.005467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:13.754194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.968504+0000","last_change":"2023-10-30T13:23:18.970566+0000","last_active":"2023-10-30T13:23:24.968504+0000","last_peered":"2023-10-30T13:23:24.968504+0000","last_clean":"2023-10-30T13:23:24.968504+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T13:23:24.968504+0000","last_undegraded":"2023-10-30T13:23:24.968504+0000","last_fullsized":"2023-10-30T13:23:24.968504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:03.851889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107903+0000","last_change":"2023-10-30T13:23:19.685481+0000","last_active":"2023-10-30T13:23:21.107903+0000","last_peered":"2023-10-30T13:23:21.107903+0000","last_clean":"2023-10-30T13:23:21.107903+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T13:23:21.107903+0000","last_undegraded":"2023-10-30T13:23:21.107903+0000","last_fullsized":"2023-10-30T13:23:21.107903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:33:50.672215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.988346+0000","last_change":"2023-10-30T13:23:18.986394+0000","last_active":"2023-10-30T13:23:20.988346+0000","last_peered":"2023-10-30T13:23:20.988346+0000","last_clean":"2023-10-30T13:23:20.988346+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T13:23:20.988346+0000","last_undegraded":"2023-10-30T13:23:20.988346+0000","last_fullsized":"2023-10-30T13:23:20.988346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:59.986981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675944+0000","last_change":"2023-10-30T13:23:19.681239+0000","last_active":"2023-10-30T13:23:20.675944+0000","last_peered":"2023-10-30T13:23:20.675944+0000","last_clean":"2023-10-30T13:23:20.675944+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T13:23:20.675944+0000","last_undegraded":"2023-10-30T13:23:20.675944+0000","last_fullsized":"2023-10-30T13:23:20.675944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:00.121240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670308+0000","last_change":"2023-10-30T13:23:18.963905+0000","last_active":"2023-10-30T13:23:20.670308+0000","last_peered":"2023-10-30T13:23:20.670308+0000","last_clean":"2023-10-30T13:23:20.670308+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T13:23:20.670308+0000","last_undegraded":"2023-10-30T13:23:20.670308+0000","last_fullsized":"2023-10-30T13:23:20.670308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005091+0000","last_change":"2023-10-30T13:23:19.678697+0000","last_active":"2023-10-30T13:23:21.005091+0000","last_peered":"2023-10-30T13:23:21.005091+0000","last_clean":"2023-10-30T13:23:21.005091+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T13:23:21.005091+0000","last_undegraded":"2023-10-30T13:23:21.005091+0000","last_fullsized":"2023-10-30T13:23:21.005091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:57.442011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670474+0000","last_change":"2023-10-30T13:23:18.952302+0000","last_active":"2023-10-30T13:23:20.670474+0000","last_peered":"2023-10-30T13:23:20.670474+0000","last_clean":"2023-10-30T13:23:20.670474+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T13:23:20.670474+0000","last_undegraded":"2023-10-30T13:23:20.670474+0000","last_fullsized":"2023-10-30T13:23:20.670474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:12.827417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005323+0000","last_change":"2023-10-30T13:23:19.688716+0000","last_active":"2023-10-30T13:23:21.005323+0000","last_peered":"2023-10-30T13:23:21.005323+0000","last_clean":"2023-10-30T13:23:21.005323+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T13:23:21.005323+0000","last_undegraded":"2023-10-30T13:23:21.005323+0000","last_fullsized":"2023-10-30T13:23:21.005323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:26:04.209304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.968535+0000","last_change":"2023-10-30T13:23:18.984912+0000","last_active":"2023-10-30T13:23:24.968535+0000","last_peered":"2023-10-30T13:23:24.968535+0000","last_clean":"2023-10-30T13:23:24.968535+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T13:23:24.968535+0000","last_undegraded":"2023-10-30T13:23:24.968535+0000","last_fullsized":"2023-10-30T13:23:24.968535+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:27:16.205042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674560+0000","last_change":"2023-10-30T13:23:19.718922+0000","last_active":"2023-10-30T13:23:20.674560+0000","last_peered":"2023-10-30T13:23:20.674560+0000","last_clean":"2023-10-30T13:23:20.674560+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T13:23:20.674560+0000","last_undegraded":"2023-10-30T13:23:20.674560+0000","last_fullsized":"2023-10-30T13:23:20.674560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612299+0000","last_change":"2023-10-30T13:23:19.732322+0000","last_active":"2023-10-30T13:23:21.612299+0000","last_peered":"2023-10-30T13:23:21.612299+0000","last_clean":"2023-10-30T13:23:21.612299+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T13:23:21.612299+0000","last_undegraded":"2023-10-30T13:23:21.612299+0000","last_fullsized":"2023-10-30T13:23:21.612299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:21.479138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.968310+0000","last_change":"2023-10-30T13:23:18.993372+0000","last_active":"2023-10-30T13:23:24.968310+0000","last_peered":"2023-10-30T13:23:24.968310+0000","last_clean":"2023-10-30T13:23:24.968310+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T13:23:24.968310+0000","last_undegraded":"2023-10-30T13:23:24.968310+0000","last_fullsized":"2023-10-30T13:23:24.968310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:41.370678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611741+0000","last_change":"2023-10-30T13:23:19.732769+0000","last_active":"2023-10-30T13:23:21.611741+0000","last_peered":"2023-10-30T13:23:21.611741+0000","last_clean":"2023-10-30T13:23:21.611741+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T13:23:21.611741+0000","last_undegraded":"2023-10-30T13:23:21.611741+0000","last_fullsized":"2023-10-30T13:23:21.611741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:56.378098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.967130+0000","last_change":"2023-10-30T13:23:18.987633+0000","last_active":"2023-10-30T13:23:24.967130+0000","last_peered":"2023-10-30T13:23:24.967130+0000","last_clean":"2023-10-30T13:23:24.967130+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T13:23:24.967130+0000","last_undegraded":"2023-10-30T13:23:24.967130+0000","last_fullsized":"2023-10-30T13:23:24.967130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:09:33.324898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107906+0000","last_change":"2023-10-30T13:23:19.722311+0000","last_active":"2023-10-30T13:23:21.107906+0000","last_peered":"2023-10-30T13:23:21.107906+0000","last_clean":"2023-10-30T13:23:21.107906+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T13:23:21.107906+0000","last_undegraded":"2023-10-30T13:23:21.107906+0000","last_fullsized":"2023-10-30T13:23:21.107906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:44.204074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005592+0000","last_change":"2023-10-30T13:23:18.958470+0000","last_active":"2023-10-30T13:23:21.005592+0000","last_peered":"2023-10-30T13:23:21.005592+0000","last_clean":"2023-10-30T13:23:21.005592+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T13:23:21.005592+0000","last_undegraded":"2023-10-30T13:23:21.005592+0000","last_fullsized":"2023-10-30T13:23:21.005592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:21.507744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005410+0000","last_change":"2023-10-30T13:23:19.699742+0000","last_active":"2023-10-30T13:23:21.005410+0000","last_peered":"2023-10-30T13:23:21.005410+0000","last_clean":"2023-10-30T13:23:21.005410+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T13:23:21.005410+0000","last_undegraded":"2023-10-30T13:23:21.005410+0000","last_fullsized":"2023-10-30T13:23:21.005410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:13.504378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.966390+0000","last_change":"2023-10-30T13:23:18.980952+0000","last_active":"2023-10-30T13:23:24.966390+0000","last_peered":"2023-10-30T13:23:24.966390+0000","last_clean":"2023-10-30T13:23:24.966390+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T13:23:24.966390+0000","last_undegraded":"2023-10-30T13:23:24.966390+0000","last_fullsized":"2023-10-30T13:23:24.966390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:44.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":69969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":200704,"data_stored":69879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":806912,"data_stored":660885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:23:31.035 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:23:31.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01b5794640 1 Processor -- start 2023-10-30T13:23:31.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01b5794640 1 -- start start 2023-10-30T13:23:31.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01b5794640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f01b01538a0 0x7f01b015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01b5794640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f01b00ca0e0 0x7f01b015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01b5794640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b00cb800 con 0x7f01b00cf160 2023-10-30T13:23:31.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01b5794640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b015dc30 con 0x7f01b00ca0e0 2023-10-30T13:23:31.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01b5794640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b00cb330 con 0x7f01b01538a0 2023-10-30T13:23:31.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01a7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f01b00ca0e0 0x7f01b015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01a7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f01b00ca0e0 0x7f01b015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39848/0 (socket says 172.21.15.46:39848) 2023-10-30T13:23:31.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01a7fff640 1 -- 172.21.15.46:0/458554154 learned_addr learned my addr 172.21.15.46:0/458554154 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:31.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01aeffd640 1 --2- 172.21.15.46:0/458554154 >> v2:172.21.15.154:3300/0 conn(0x7f01b01538a0 0x7f01b015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01a7fff640 1 -- 172.21.15.46:0/458554154 >> v2:172.21.15.154:3300/0 conn(0x7f01b01538a0 msgr2=0x7f01b015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:31.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01a7fff640 1 --2- 172.21.15.46:0/458554154 >> v2:172.21.15.154:3300/0 conn(0x7f01b01538a0 0x7f01b015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01a7fff640 1 -- 172.21.15.46:0/458554154 >> v1:172.21.15.154:6789/0 conn(0x7f01b00cf160 legacy=0x7f01b00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:31.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.208+0000 7f01a7fff640 1 -- 172.21.15.46:0/458554154 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b00cad70 con 0x7f01b00ca0e0 2023-10-30T13:23:31.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.212+0000 7f01a7fff640 1 --2- 172.21.15.46:0/458554154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00ca0e0 0x7f01b015a090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f01b00cfd10 tx=0x7f019801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692c23c17d100e8f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:31.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.212+0000 7f01ae7fc640 1 -- 172.21.15.46:0/458554154 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01980a89f0 con 0x7f01b00ca0e0 2023-10-30T13:23:31.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.212+0000 7f01ae7fc640 1 -- 172.21.15.46:0/458554154 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01980a8b50 con 0x7f01b00ca0e0 2023-10-30T13:23:31.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.212+0000 7f01ae7fc640 1 -- 172.21.15.46:0/458554154 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01980b0440 con 0x7f01b00ca0e0 2023-10-30T13:23:31.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.212+0000 7f01b5794640 1 -- 172.21.15.46:0/458554154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00ca0e0 msgr2=0x7f01b015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.212+0000 7f01b5794640 1 --2- 172.21.15.46:0/458554154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00ca0e0 0x7f01b015a090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f01b00cfd10 tx=0x7f019801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.212+0000 7f01b5794640 1 -- 172.21.15.46:0/458554154 shutdown_connections 2023-10-30T13:23:31.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.212+0000 7f01b5794640 1 --2- 172.21.15.46:0/458554154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b00ca0e0 0x7f01b015a090 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.212+0000 7f01b5794640 1 --2- 172.21.15.46:0/458554154 >> v2:172.21.15.154:3300/0 conn(0x7f01b01538a0 0x7f01b015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.212+0000 7f01b5794640 1 -- 172.21.15.46:0/458554154 >> 172.21.15.46:0/458554154 conn(0x7f01b006bfe0 msgr2=0x7f01b006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:31.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.212+0000 7f01b5794640 1 -- 172.21.15.46:0/458554154 shutdown_connections 2023-10-30T13:23:31.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.212+0000 7f01b5794640 1 -- 172.21.15.46:0/458554154 wait complete. 2023-10-30T13:23:31.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01b5794640 1 Processor -- start 2023-10-30T13:23:31.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01b5794640 1 -- start start 2023-10-30T13:23:31.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01b5794640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01b00ca0e0 0x7f01b0140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01b5794640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01b00cf160 0x7f01b00ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01b5794640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b01538a0 0x7f01b00cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01b5794640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b01e4380 con 0x7f01b01538a0 2023-10-30T13:23:31.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01b5794640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0158620 con 0x7f01b00ca0e0 2023-10-30T13:23:31.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01b5794640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0159690 con 0x7f01b00cf160 2023-10-30T13:23:31.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01af7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b01538a0 0x7f01b00cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01af7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b01538a0 0x7f01b00cd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39864/0 (socket says 172.21.15.46:39864) 2023-10-30T13:23:31.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01af7fe640 1 -- 172.21.15.46:0/1071625006 learned_addr learned my addr 172.21.15.46:0/1071625006 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:31.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01aeffd640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01b00ca0e0 0x7f01b0140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01a7fff640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01b00cf160 0x7f01b00ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01af7fe640 1 -- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01b00cf160 msgr2=0x7f01b00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01af7fe640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01b00cf160 0x7f01b00ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01af7fe640 1 -- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01b00ca0e0 msgr2=0x7f01b0140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01af7fe640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01b00ca0e0 0x7f01b0140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01af7fe640 1 -- 172.21.15.46:0/1071625006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01980094d0 con 0x7f01b01538a0 2023-10-30T13:23:31.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01af7fe640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b01538a0 0x7f01b00cd4f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f01a800c6f0 tx=0x7f01a800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:31.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01a77fe640 1 -- 172.21.15.46:0/1071625006 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a8003980 con 0x7f01b01538a0 2023-10-30T13:23:31.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01a77fe640 1 -- 172.21.15.46:0/1071625006 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a8003ae0 con 0x7f01b01538a0 2023-10-30T13:23:31.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01a77fe640 1 -- 172.21.15.46:0/1071625006 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a8003e90 con 0x7f01b01538a0 2023-10-30T13:23:31.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01b5794640 1 -- 172.21.15.46:0/1071625006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b01e51e0 con 0x7f01b01538a0 2023-10-30T13:23:31.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.216+0000 7f01b5794640 1 -- 172.21.15.46:0/1071625006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b01e5610 con 0x7f01b01538a0 2023-10-30T13:23:31.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.220+0000 7f01b5794640 1 -- 172.21.15.46:0/1071625006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b00b2a50 con 0x7f01b01538a0 2023-10-30T13:23:31.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.220+0000 7f01a77fe640 1 -- 172.21.15.46:0/1071625006 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f01a800a850 con 0x7f01b01538a0 2023-10-30T13:23:31.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.220+0000 7f01a77fe640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01900664d0 0x7f0190068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.220+0000 7f01a77fe640 1 -- 172.21.15.46:0/1071625006 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01a8083ac0 con 0x7f01b01538a0 2023-10-30T13:23:31.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.220+0000 7f01aeffd640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01900664d0 0x7f0190068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.220+0000 7f01aeffd640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01900664d0 0x7f0190068990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f01b00cf070 tx=0x7f01a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:31.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.224+0000 7f01a77fe640 1 -- 172.21.15.46:0/1071625006 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f01a8050fe0 con 0x7f01b01538a0 2023-10-30T13:23:31.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.376+0000 7f01b5794640 1 -- 172.21.15.46:0/1071625006 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01b0159890 con 0x7f01900664d0 2023-10-30T13:23:31.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01a77fe640 1 -- 172.21.15.46:0/1071625006 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264486 (secure 0 0 0) 0x7f01a00108d0 con 0x7f01900664d0 2023-10-30T13:23:31.408 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:31.409 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T13:23:31.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01b5794640 1 -- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01900664d0 msgr2=0x7f0190068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01b5794640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01900664d0 0x7f0190068990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f01b00cf070 tx=0x7f01a0008040 comp rx=0 tx=0).stop 2023-10-30T13:23:31.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01b5794640 1 -- 172.21.15.46:0/1071625006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b01538a0 msgr2=0x7f01b00cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01b5794640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b01538a0 0x7f01b00cd4f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f01a800c6f0 tx=0x7f01a800cbb0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01b5794640 1 -- 172.21.15.46:0/1071625006 shutdown_connections 2023-10-30T13:23:31.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01b5794640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01900664d0 0x7f0190068990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01b5794640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01b01538a0 0x7f01b00cd4f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01b5794640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01b00cf160 0x7f01b00ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01b5794640 1 --2- 172.21.15.46:0/1071625006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01b00ca0e0 0x7f01b0140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01b5794640 1 -- 172.21.15.46:0/1071625006 >> 172.21.15.46:0/1071625006 conn(0x7f01b006bfe0 msgr2=0x7f01b00c5210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01b5794640 1 -- 172.21.15.46:0/1071625006 shutdown_connections 2023-10-30T13:23:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.400+0000 7f01b5794640 1 -- 172.21.15.46:0/1071625006 wait complete. 2023-10-30T13:23:31.426 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-30T13:23:30.712717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235580,"kb_used_data":2812,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739140,"statfs":{"total":773094113280,"available":772852879360,"internally_reserved":0,"allocated":2879488,"data_stored":1749683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.043378"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612886+0000","last_change":"2023-10-30T13:23:19.719271+0000","last_active":"2023-10-30T13:23:21.612886+0000","last_peered":"2023-10-30T13:23:21.612886+0000","last_clean":"2023-10-30T13:23:21.612886+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T13:23:21.612886+0000","last_undegraded":"2023-10-30T13:23:21.612886+0000","last_fullsized":"2023-10-30T13:23:21.612886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:41:37.614073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670716+0000","last_change":"2023-10-30T13:23:19.687152+0000","last_active":"2023-10-30T13:23:20.670716+0000","last_peered":"2023-10-30T13:23:20.670716+0000","last_clean":"2023-10-30T13:23:20.670716+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T13:23:20.670716+0000","last_undegraded":"2023-10-30T13:23:20.670716+0000","last_fullsized":"2023-10-30T13:23:20.670716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107815+0000","last_change":"2023-10-30T13:23:19.689554+0000","last_active":"2023-10-30T13:23:21.107815+0000","last_peered":"2023-10-30T13:23:21.107815+0000","last_clean":"2023-10-30T13:23:21.107815+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T13:23:21.107815+0000","last_undegraded":"2023-10-30T13:23:21.107815+0000","last_fullsized":"2023-10-30T13:23:21.107815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:56:13.795044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611622+0000","last_change":"2023-10-30T13:23:19.714018+0000","last_active":"2023-10-30T13:23:21.611622+0000","last_peered":"2023-10-30T13:23:21.611622+0000","last_clean":"2023-10-30T13:23:21.611622+0000","last_became_active":"2023-10-30T13:23:19.711673+0000","last_became_peered":"2023-10-30T13:23:19.711673+0000","last_unstale":"2023-10-30T13:23:21.611622+0000","last_undegraded":"2023-10-30T13:23:21.611622+0000","last_fullsized":"2023-10-30T13:23:21.611622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:28:16.658840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611546+0000","last_change":"2023-10-30T13:23:19.719999+0000","last_active":"2023-10-30T13:23:21.611546+0000","last_peered":"2023-10-30T13:23:21.611546+0000","last_clean":"2023-10-30T13:23:21.611546+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T13:23:21.611546+0000","last_undegraded":"2023-10-30T13:23:21.611546+0000","last_fullsized":"2023-10-30T13:23:21.611546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:46:31.287013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612300+0000","last_change":"2023-10-30T13:23:19.703474+0000","last_active":"2023-10-30T13:23:21.612300+0000","last_peered":"2023-10-30T13:23:21.612300+0000","last_clean":"2023-10-30T13:23:21.612300+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T13:23:21.612300+0000","last_undegraded":"2023-10-30T13:23:21.612300+0000","last_fullsized":"2023-10-30T13:23:21.612300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:53:58.118821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613234+0000","last_change":"2023-10-30T13:23:19.726985+0000","last_active":"2023-10-30T13:23:21.613234+0000","last_peered":"2023-10-30T13:23:21.613234+0000","last_clean":"2023-10-30T13:23:21.613234+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T13:23:21.613234+0000","last_undegraded":"2023-10-30T13:23:21.613234+0000","last_fullsized":"2023-10-30T13:23:21.613234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:51:13.152339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.004933+0000","last_change":"2023-10-30T13:23:19.714153+0000","last_active":"2023-10-30T13:23:21.004933+0000","last_peered":"2023-10-30T13:23:21.004933+0000","last_clean":"2023-10-30T13:23:21.004933+0000","last_became_active":"2023-10-30T13:23:19.713876+0000","last_became_peered":"2023-10-30T13:23:19.713876+0000","last_unstale":"2023-10-30T13:23:21.004933+0000","last_undegraded":"2023-10-30T13:23:21.004933+0000","last_fullsized":"2023-10-30T13:23:21.004933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:05:13.588780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613220+0000","last_change":"2023-10-30T13:23:19.724583+0000","last_active":"2023-10-30T13:23:21.613220+0000","last_peered":"2023-10-30T13:23:21.613220+0000","last_clean":"2023-10-30T13:23:21.613220+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T13:23:21.613220+0000","last_undegraded":"2023-10-30T13:23:21.613220+0000","last_fullsized":"2023-10-30T13:23:21.613220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:47.936982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677145+0000","last_change":"2023-10-30T13:23:19.704399+0000","last_active":"2023-10-30T13:23:20.677145+0000","last_peered":"2023-10-30T13:23:20.677145+0000","last_clean":"2023-10-30T13:23:20.677145+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T13:23:20.677145+0000","last_undegraded":"2023-10-30T13:23:20.677145+0000","last_fullsized":"2023-10-30T13:23:20.677145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676522+0000","last_change":"2023-10-30T13:23:19.726077+0000","last_active":"2023-10-30T13:23:20.676522+0000","last_peered":"2023-10-30T13:23:20.676522+0000","last_clean":"2023-10-30T13:23:20.676522+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T13:23:20.676522+0000","last_undegraded":"2023-10-30T13:23:20.676522+0000","last_fullsized":"2023-10-30T13:23:20.676522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675959+0000","last_change":"2023-10-30T13:23:19.720995+0000","last_active":"2023-10-30T13:23:20.675959+0000","last_peered":"2023-10-30T13:23:20.675959+0000","last_clean":"2023-10-30T13:23:20.675959+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T13:23:20.675959+0000","last_undegraded":"2023-10-30T13:23:20.675959+0000","last_fullsized":"2023-10-30T13:23:20.675959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:44.490568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005253+0000","last_change":"2023-10-30T13:23:19.692509+0000","last_active":"2023-10-30T13:23:21.005253+0000","last_peered":"2023-10-30T13:23:21.005253+0000","last_clean":"2023-10-30T13:23:21.005253+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T13:23:21.005253+0000","last_undegraded":"2023-10-30T13:23:21.005253+0000","last_fullsized":"2023-10-30T13:23:21.005253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:15:17.956816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107580+0000","last_change":"2023-10-30T13:23:19.685435+0000","last_active":"2023-10-30T13:23:21.107580+0000","last_peered":"2023-10-30T13:23:21.107580+0000","last_clean":"2023-10-30T13:23:21.107580+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T13:23:21.107580+0000","last_undegraded":"2023-10-30T13:23:21.107580+0000","last_fullsized":"2023-10-30T13:23:21.107580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:39.035298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670272+0000","last_change":"2023-10-30T13:23:19.706721+0000","last_active":"2023-10-30T13:23:20.670272+0000","last_peered":"2023-10-30T13:23:20.670272+0000","last_clean":"2023-10-30T13:23:20.670272+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T13:23:20.670272+0000","last_undegraded":"2023-10-30T13:23:20.670272+0000","last_fullsized":"2023-10-30T13:23:20.670272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.610717+0000","last_change":"2023-10-30T13:23:19.710432+0000","last_active":"2023-10-30T13:23:21.610717+0000","last_peered":"2023-10-30T13:23:21.610717+0000","last_clean":"2023-10-30T13:23:21.610717+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T13:23:21.610717+0000","last_undegraded":"2023-10-30T13:23:21.610717+0000","last_fullsized":"2023-10-30T13:23:21.610717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:48:51.992826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612857+0000","last_change":"2023-10-30T13:23:19.741225+0000","last_active":"2023-10-30T13:23:21.612857+0000","last_peered":"2023-10-30T13:23:21.612857+0000","last_clean":"2023-10-30T13:23:21.612857+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T13:23:21.612857+0000","last_undegraded":"2023-10-30T13:23:21.612857+0000","last_fullsized":"2023-10-30T13:23:21.612857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:27.310070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.608570+0000","last_change":"2023-10-30T13:23:19.701024+0000","last_active":"2023-10-30T13:23:21.608570+0000","last_peered":"2023-10-30T13:23:21.608570+0000","last_clean":"2023-10-30T13:23:21.608570+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T13:23:21.608570+0000","last_undegraded":"2023-10-30T13:23:21.608570+0000","last_fullsized":"2023-10-30T13:23:21.608570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:14.460383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612314+0000","last_change":"2023-10-30T13:23:19.723929+0000","last_active":"2023-10-30T13:23:21.612314+0000","last_peered":"2023-10-30T13:23:21.612314+0000","last_clean":"2023-10-30T13:23:21.612314+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T13:23:21.612314+0000","last_undegraded":"2023-10-30T13:23:21.612314+0000","last_fullsized":"2023-10-30T13:23:21.612314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:41.023731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674806+0000","last_change":"2023-10-30T13:23:19.725101+0000","last_active":"2023-10-30T13:23:20.674806+0000","last_peered":"2023-10-30T13:23:20.674806+0000","last_clean":"2023-10-30T13:23:20.674806+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T13:23:20.674806+0000","last_undegraded":"2023-10-30T13:23:20.674806+0000","last_fullsized":"2023-10-30T13:23:20.674806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107754+0000","last_change":"2023-10-30T13:23:19.721813+0000","last_active":"2023-10-30T13:23:21.107754+0000","last_peered":"2023-10-30T13:23:21.107754+0000","last_clean":"2023-10-30T13:23:21.107754+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T13:23:21.107754+0000","last_undegraded":"2023-10-30T13:23:21.107754+0000","last_fullsized":"2023-10-30T13:23:21.107754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:34:17.212695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612670+0000","last_change":"2023-10-30T13:23:19.724763+0000","last_active":"2023-10-30T13:23:21.612670+0000","last_peered":"2023-10-30T13:23:21.612670+0000","last_clean":"2023-10-30T13:23:21.612670+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T13:23:21.612670+0000","last_undegraded":"2023-10-30T13:23:21.612670+0000","last_fullsized":"2023-10-30T13:23:21.612670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:28:53.428578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005009+0000","last_change":"2023-10-30T13:23:19.716706+0000","last_active":"2023-10-30T13:23:21.005009+0000","last_peered":"2023-10-30T13:23:21.005009+0000","last_clean":"2023-10-30T13:23:21.005009+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T13:23:21.005009+0000","last_undegraded":"2023-10-30T13:23:21.005009+0000","last_fullsized":"2023-10-30T13:23:21.005009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:07.718271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.610920+0000","last_change":"2023-10-30T13:23:19.704791+0000","last_active":"2023-10-30T13:23:21.610920+0000","last_peered":"2023-10-30T13:23:21.610920+0000","last_clean":"2023-10-30T13:23:21.610920+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T13:23:21.610920+0000","last_undegraded":"2023-10-30T13:23:21.610920+0000","last_fullsized":"2023-10-30T13:23:21.610920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:54.750514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005422+0000","last_change":"2023-10-30T13:23:19.717027+0000","last_active":"2023-10-30T13:23:21.005422+0000","last_peered":"2023-10-30T13:23:21.005422+0000","last_clean":"2023-10-30T13:23:21.005422+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T13:23:21.005422+0000","last_undegraded":"2023-10-30T13:23:21.005422+0000","last_fullsized":"2023-10-30T13:23:21.005422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:38.788372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677247+0000","last_change":"2023-10-30T13:23:19.706002+0000","last_active":"2023-10-30T13:23:20.677247+0000","last_peered":"2023-10-30T13:23:20.677247+0000","last_clean":"2023-10-30T13:23:20.677247+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T13:23:20.677247+0000","last_undegraded":"2023-10-30T13:23:20.677247+0000","last_fullsized":"2023-10-30T13:23:20.677247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:46.882474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670564+0000","last_change":"2023-10-30T13:23:19.678472+0000","last_active":"2023-10-30T13:23:20.670564+0000","last_peered":"2023-10-30T13:23:20.670564+0000","last_clean":"2023-10-30T13:23:20.670564+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T13:23:20.670564+0000","last_undegraded":"2023-10-30T13:23:20.670564+0000","last_fullsized":"2023-10-30T13:23:20.670564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005470+0000","last_change":"2023-10-30T13:23:19.702155+0000","last_active":"2023-10-30T13:23:21.005470+0000","last_peered":"2023-10-30T13:23:21.005470+0000","last_clean":"2023-10-30T13:23:21.005470+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T13:23:21.005470+0000","last_undegraded":"2023-10-30T13:23:21.005470+0000","last_fullsized":"2023-10-30T13:23:21.005470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:30.131186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107903+0000","last_change":"2023-10-30T13:23:19.687820+0000","last_active":"2023-10-30T13:23:21.107903+0000","last_peered":"2023-10-30T13:23:21.107903+0000","last_clean":"2023-10-30T13:23:21.107903+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T13:23:21.107903+0000","last_undegraded":"2023-10-30T13:23:21.107903+0000","last_fullsized":"2023-10-30T13:23:21.107903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:46.088882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612897+0000","last_change":"2023-10-30T13:23:19.702442+0000","last_active":"2023-10-30T13:23:21.612897+0000","last_peered":"2023-10-30T13:23:21.612897+0000","last_clean":"2023-10-30T13:23:21.612897+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T13:23:21.612897+0000","last_undegraded":"2023-10-30T13:23:21.612897+0000","last_fullsized":"2023-10-30T13:23:21.612897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:58:50.083326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.608295+0000","last_change":"2023-10-30T13:23:19.711364+0000","last_active":"2023-10-30T13:23:21.608295+0000","last_peered":"2023-10-30T13:23:21.608295+0000","last_clean":"2023-10-30T13:23:21.608295+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T13:23:21.608295+0000","last_undegraded":"2023-10-30T13:23:21.608295+0000","last_fullsized":"2023-10-30T13:23:21.608295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:15.250397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674450+0000","last_change":"2023-10-30T13:23:19.698485+0000","last_active":"2023-10-30T13:23:20.674450+0000","last_peered":"2023-10-30T13:23:20.674450+0000","last_clean":"2023-10-30T13:23:20.674450+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T13:23:20.674450+0000","last_undegraded":"2023-10-30T13:23:20.674450+0000","last_fullsized":"2023-10-30T13:23:20.674450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675615+0000","last_change":"2023-10-30T13:23:19.703223+0000","last_active":"2023-10-30T13:23:20.675615+0000","last_peered":"2023-10-30T13:23:20.675615+0000","last_clean":"2023-10-30T13:23:20.675615+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T13:23:20.675615+0000","last_undegraded":"2023-10-30T13:23:20.675615+0000","last_fullsized":"2023-10-30T13:23:20.675615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:28.065471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005464+0000","last_change":"2023-10-30T13:23:18.963933+0000","last_active":"2023-10-30T13:23:21.005464+0000","last_peered":"2023-10-30T13:23:21.005464+0000","last_clean":"2023-10-30T13:23:21.005464+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T13:23:21.005464+0000","last_undegraded":"2023-10-30T13:23:21.005464+0000","last_fullsized":"2023-10-30T13:23:21.005464+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:48:58.196567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107913+0000","last_change":"2023-10-30T13:23:19.717819+0000","last_active":"2023-10-30T13:23:21.107913+0000","last_peered":"2023-10-30T13:23:21.107913+0000","last_clean":"2023-10-30T13:23:21.107913+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T13:23:21.107913+0000","last_undegraded":"2023-10-30T13:23:21.107913+0000","last_fullsized":"2023-10-30T13:23:21.107913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:15.403400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.990350+0000","last_change":"2023-10-30T13:23:18.976615+0000","last_active":"2023-10-30T13:23:20.990350+0000","last_peered":"2023-10-30T13:23:20.990350+0000","last_clean":"2023-10-30T13:23:20.990350+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T13:23:20.990350+0000","last_undegraded":"2023-10-30T13:23:20.990350+0000","last_fullsized":"2023-10-30T13:23:20.990350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:59.048396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.981386+0000","last_change":"2023-10-30T13:23:18.977357+0000","last_active":"2023-10-30T13:23:20.981386+0000","last_peered":"2023-10-30T13:23:20.981386+0000","last_clean":"2023-10-30T13:23:20.981386+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T13:23:20.981386+0000","last_undegraded":"2023-10-30T13:23:20.981386+0000","last_fullsized":"2023-10-30T13:23:20.981386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:53.502681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005000+0000","last_change":"2023-10-30T13:23:19.689197+0000","last_active":"2023-10-30T13:23:21.005000+0000","last_peered":"2023-10-30T13:23:21.005000+0000","last_clean":"2023-10-30T13:23:21.005000+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T13:23:21.005000+0000","last_undegraded":"2023-10-30T13:23:21.005000+0000","last_fullsized":"2023-10-30T13:23:21.005000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:59:47.946429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.989002+0000","last_change":"2023-10-30T13:23:18.989299+0000","last_active":"2023-10-30T13:23:20.989002+0000","last_peered":"2023-10-30T13:23:20.989002+0000","last_clean":"2023-10-30T13:23:20.989002+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T13:23:20.989002+0000","last_undegraded":"2023-10-30T13:23:20.989002+0000","last_fullsized":"2023-10-30T13:23:20.989002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:44.510184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670217+0000","last_change":"2023-10-30T13:23:19.679342+0000","last_active":"2023-10-30T13:23:20.670217+0000","last_peered":"2023-10-30T13:23:20.670217+0000","last_clean":"2023-10-30T13:23:20.670217+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T13:23:20.670217+0000","last_undegraded":"2023-10-30T13:23:20.670217+0000","last_fullsized":"2023-10-30T13:23:20.670217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677093+0000","last_change":"2023-10-30T13:23:18.989515+0000","last_active":"2023-10-30T13:23:20.677093+0000","last_peered":"2023-10-30T13:23:20.677093+0000","last_clean":"2023-10-30T13:23:20.677093+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T13:23:20.677093+0000","last_undegraded":"2023-10-30T13:23:20.677093+0000","last_fullsized":"2023-10-30T13:23:20.677093+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:35.133762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612308+0000","last_change":"2023-10-30T13:23:19.732859+0000","last_active":"2023-10-30T13:23:21.612308+0000","last_peered":"2023-10-30T13:23:21.612308+0000","last_clean":"2023-10-30T13:23:21.612308+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T13:23:21.612308+0000","last_undegraded":"2023-10-30T13:23:21.612308+0000","last_fullsized":"2023-10-30T13:23:21.612308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:27.629521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.989659+0000","last_change":"2023-10-30T13:23:18.991454+0000","last_active":"2023-10-30T13:23:20.989659+0000","last_peered":"2023-10-30T13:23:20.989659+0000","last_clean":"2023-10-30T13:23:20.989659+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T13:23:20.989659+0000","last_undegraded":"2023-10-30T13:23:20.989659+0000","last_fullsized":"2023-10-30T13:23:20.989659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:13.450999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107915+0000","last_change":"2023-10-30T13:23:19.722347+0000","last_active":"2023-10-30T13:23:21.107915+0000","last_peered":"2023-10-30T13:23:21.107915+0000","last_clean":"2023-10-30T13:23:21.107915+0000","last_became_active":"2023-10-30T13:23:19.722077+0000","last_became_peered":"2023-10-30T13:23:19.722077+0000","last_unstale":"2023-10-30T13:23:21.107915+0000","last_undegraded":"2023-10-30T13:23:21.107915+0000","last_fullsized":"2023-10-30T13:23:21.107915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:23:21.544691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.965367+0000","last_change":"2023-10-30T13:23:18.982337+0000","last_active":"2023-10-30T13:23:24.965367+0000","last_peered":"2023-10-30T13:23:24.965367+0000","last_clean":"2023-10-30T13:23:24.965367+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T13:23:24.965367+0000","last_undegraded":"2023-10-30T13:23:24.965367+0000","last_fullsized":"2023-10-30T13:23:24.965367+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:09:39.996740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676299+0000","last_change":"2023-10-30T13:23:19.706643+0000","last_active":"2023-10-30T13:23:20.676299+0000","last_peered":"2023-10-30T13:23:20.676299+0000","last_clean":"2023-10-30T13:23:20.676299+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T13:23:20.676299+0000","last_undegraded":"2023-10-30T13:23:20.676299+0000","last_fullsized":"2023-10-30T13:23:20.676299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:29.238430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.968446+0000","last_change":"2023-10-30T13:23:18.969882+0000","last_active":"2023-10-30T13:23:24.968446+0000","last_peered":"2023-10-30T13:23:24.968446+0000","last_clean":"2023-10-30T13:23:24.968446+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T13:23:24.968446+0000","last_undegraded":"2023-10-30T13:23:24.968446+0000","last_fullsized":"2023-10-30T13:23:24.968446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:17.093448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611588+0000","last_change":"2023-10-30T13:23:19.728295+0000","last_active":"2023-10-30T13:23:21.611588+0000","last_peered":"2023-10-30T13:23:21.611588+0000","last_clean":"2023-10-30T13:23:21.611588+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T13:23:21.611588+0000","last_undegraded":"2023-10-30T13:23:21.611588+0000","last_fullsized":"2023-10-30T13:23:21.611588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:37:54.545985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.966553+0000","last_change":"2023-10-30T13:23:18.971632+0000","last_active":"2023-10-30T13:23:24.966553+0000","last_peered":"2023-10-30T13:23:24.966553+0000","last_clean":"2023-10-30T13:23:24.966553+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T13:23:24.966553+0000","last_undegraded":"2023-10-30T13:23:24.966553+0000","last_fullsized":"2023-10-30T13:23:24.966553+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:54.428499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107653+0000","last_change":"2023-10-30T13:23:19.718168+0000","last_active":"2023-10-30T13:23:21.107653+0000","last_peered":"2023-10-30T13:23:21.107653+0000","last_clean":"2023-10-30T13:23:21.107653+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T13:23:21.107653+0000","last_undegraded":"2023-10-30T13:23:21.107653+0000","last_fullsized":"2023-10-30T13:23:21.107653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:54.832358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.989192+0000","last_change":"2023-10-30T13:23:18.978420+0000","last_active":"2023-10-30T13:23:20.989192+0000","last_peered":"2023-10-30T13:23:20.989192+0000","last_clean":"2023-10-30T13:23:20.989192+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T13:23:20.989192+0000","last_undegraded":"2023-10-30T13:23:20.989192+0000","last_fullsized":"2023-10-30T13:23:20.989192+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:10:44.511282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611634+0000","last_change":"2023-10-30T13:23:19.721434+0000","last_active":"2023-10-30T13:23:21.611634+0000","last_peered":"2023-10-30T13:23:21.611634+0000","last_clean":"2023-10-30T13:23:21.611634+0000","last_became_active":"2023-10-30T13:23:19.720704+0000","last_became_peered":"2023-10-30T13:23:19.720704+0000","last_unstale":"2023-10-30T13:23:21.611634+0000","last_undegraded":"2023-10-30T13:23:21.611634+0000","last_fullsized":"2023-10-30T13:23:21.611634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:45.728513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674378+0000","last_change":"2023-10-30T13:23:18.972707+0000","last_active":"2023-10-30T13:23:20.674378+0000","last_peered":"2023-10-30T13:23:20.674378+0000","last_clean":"2023-10-30T13:23:20.674378+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T13:23:20.674378+0000","last_undegraded":"2023-10-30T13:23:20.674378+0000","last_fullsized":"2023-10-30T13:23:20.674378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674810+0000","last_change":"2023-10-30T13:23:19.698274+0000","last_active":"2023-10-30T13:23:20.674810+0000","last_peered":"2023-10-30T13:23:20.674810+0000","last_clean":"2023-10-30T13:23:20.674810+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T13:23:20.674810+0000","last_undegraded":"2023-10-30T13:23:20.674810+0000","last_fullsized":"2023-10-30T13:23:20.674810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.965136+0000","last_change":"2023-10-30T13:23:18.948296+0000","last_active":"2023-10-30T13:23:24.965136+0000","last_peered":"2023-10-30T13:23:24.965136+0000","last_clean":"2023-10-30T13:23:24.965136+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T13:23:24.965136+0000","last_undegraded":"2023-10-30T13:23:24.965136+0000","last_fullsized":"2023-10-30T13:23:24.965136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:55.308329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611416+0000","last_change":"2023-10-30T13:23:19.728647+0000","last_active":"2023-10-30T13:23:21.611416+0000","last_peered":"2023-10-30T13:23:21.611416+0000","last_clean":"2023-10-30T13:23:21.611416+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T13:23:21.611416+0000","last_undegraded":"2023-10-30T13:23:21.611416+0000","last_fullsized":"2023-10-30T13:23:21.611416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:09.123809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.986186+0000","last_change":"2023-10-30T13:23:18.962343+0000","last_active":"2023-10-30T13:23:20.986186+0000","last_peered":"2023-10-30T13:23:20.986186+0000","last_clean":"2023-10-30T13:23:20.986186+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T13:23:20.986186+0000","last_undegraded":"2023-10-30T13:23:20.986186+0000","last_fullsized":"2023-10-30T13:23:20.986186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:36.082164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611928+0000","last_change":"2023-10-30T13:23:19.720770+0000","last_active":"2023-10-30T13:23:21.611928+0000","last_peered":"2023-10-30T13:23:21.611928+0000","last_clean":"2023-10-30T13:23:21.611928+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T13:23:21.611928+0000","last_undegraded":"2023-10-30T13:23:21.611928+0000","last_fullsized":"2023-10-30T13:23:21.611928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:37.378858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.962456+0000","last_change":"2023-10-30T13:23:18.949081+0000","last_active":"2023-10-30T13:23:24.962456+0000","last_peered":"2023-10-30T13:23:24.962456+0000","last_clean":"2023-10-30T13:23:24.962456+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T13:23:24.962456+0000","last_undegraded":"2023-10-30T13:23:24.962456+0000","last_fullsized":"2023-10-30T13:23:24.962456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:00.356885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613552+0000","last_change":"2023-10-30T13:23:19.724180+0000","last_active":"2023-10-30T13:23:21.613552+0000","last_peered":"2023-10-30T13:23:21.613552+0000","last_clean":"2023-10-30T13:23:21.613552+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T13:23:21.613552+0000","last_undegraded":"2023-10-30T13:23:21.613552+0000","last_fullsized":"2023-10-30T13:23:21.613552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:14.049454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.981649+0000","last_change":"2023-10-30T13:23:18.970780+0000","last_active":"2023-10-30T13:23:20.981649+0000","last_peered":"2023-10-30T13:23:20.981649+0000","last_clean":"2023-10-30T13:23:20.981649+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T13:23:20.981649+0000","last_undegraded":"2023-10-30T13:23:20.981649+0000","last_fullsized":"2023-10-30T13:23:20.981649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:38:51.113629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611979+0000","last_change":"2023-10-30T13:23:19.704834+0000","last_active":"2023-10-30T13:23:21.611979+0000","last_peered":"2023-10-30T13:23:21.611979+0000","last_clean":"2023-10-30T13:23:21.611979+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T13:23:21.611979+0000","last_undegraded":"2023-10-30T13:23:21.611979+0000","last_fullsized":"2023-10-30T13:23:21.611979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:11.266083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.983909+0000","last_change":"2023-10-30T13:23:18.977920+0000","last_active":"2023-10-30T13:23:20.983909+0000","last_peered":"2023-10-30T13:23:20.983909+0000","last_clean":"2023-10-30T13:23:20.983909+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T13:23:20.983909+0000","last_undegraded":"2023-10-30T13:23:20.983909+0000","last_fullsized":"2023-10-30T13:23:20.983909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:07:59.139948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611883+0000","last_change":"2023-10-30T13:23:19.724497+0000","last_active":"2023-10-30T13:23:21.611883+0000","last_peered":"2023-10-30T13:23:21.611883+0000","last_clean":"2023-10-30T13:23:21.611883+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T13:23:21.611883+0000","last_undegraded":"2023-10-30T13:23:21.611883+0000","last_fullsized":"2023-10-30T13:23:21.611883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:23.836868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.686506+0000","last_change":"2023-10-30T13:23:08.208031+0000","last_active":"2023-10-30T13:23:19.686506+0000","last_peered":"2023-10-30T13:23:19.686506+0000","last_clean":"2023-10-30T13:23:19.686506+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T13:23:19.686506+0000","last_undegraded":"2023-10-30T13:23:19.686506+0000","last_fullsized":"2023-10-30T13:23:19.686506+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_clean_scrub_stamp":"2023-10-30T13:23:07.179866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:22.691367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.989305+0000","last_change":"2023-10-30T13:23:18.979328+0000","last_active":"2023-10-30T13:23:20.989305+0000","last_peered":"2023-10-30T13:23:20.989305+0000","last_clean":"2023-10-30T13:23:20.989305+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T13:23:20.989305+0000","last_undegraded":"2023-10-30T13:23:20.989305+0000","last_fullsized":"2023-10-30T13:23:20.989305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:04.942593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005198+0000","last_change":"2023-10-30T13:23:19.711222+0000","last_active":"2023-10-30T13:23:21.005198+0000","last_peered":"2023-10-30T13:23:21.005198+0000","last_clean":"2023-10-30T13:23:21.005198+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T13:23:21.005198+0000","last_undegraded":"2023-10-30T13:23:21.005198+0000","last_fullsized":"2023-10-30T13:23:21.005198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:07.213498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.988400+0000","last_change":"2023-10-30T13:23:18.985318+0000","last_active":"2023-10-30T13:23:20.988400+0000","last_peered":"2023-10-30T13:23:20.988400+0000","last_clean":"2023-10-30T13:23:20.988400+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T13:23:20.988400+0000","last_undegraded":"2023-10-30T13:23:20.988400+0000","last_fullsized":"2023-10-30T13:23:20.988400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:16.754084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107580+0000","last_change":"2023-10-30T13:23:19.717938+0000","last_active":"2023-10-30T13:23:21.107580+0000","last_peered":"2023-10-30T13:23:21.107580+0000","last_clean":"2023-10-30T13:23:21.107580+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T13:23:21.107580+0000","last_undegraded":"2023-10-30T13:23:21.107580+0000","last_fullsized":"2023-10-30T13:23:21.107580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:56:01.624954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.983889+0000","last_change":"2023-10-30T13:23:18.949565+0000","last_active":"2023-10-30T13:23:20.983889+0000","last_peered":"2023-10-30T13:23:20.983889+0000","last_clean":"2023-10-30T13:23:20.983889+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T13:23:20.983889+0000","last_undegraded":"2023-10-30T13:23:20.983889+0000","last_fullsized":"2023-10-30T13:23:20.983889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:55:46.029368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670291+0000","last_change":"2023-10-30T13:23:19.678037+0000","last_active":"2023-10-30T13:23:20.670291+0000","last_peered":"2023-10-30T13:23:20.670291+0000","last_clean":"2023-10-30T13:23:20.670291+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T13:23:20.670291+0000","last_undegraded":"2023-10-30T13:23:20.670291+0000","last_fullsized":"2023-10-30T13:23:20.670291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.989242+0000","last_change":"2023-10-30T13:23:18.984535+0000","last_active":"2023-10-30T13:23:20.989242+0000","last_peered":"2023-10-30T13:23:20.989242+0000","last_clean":"2023-10-30T13:23:20.989242+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T13:23:20.989242+0000","last_undegraded":"2023-10-30T13:23:20.989242+0000","last_fullsized":"2023-10-30T13:23:20.989242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:28.405405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675980+0000","last_change":"2023-10-30T13:23:19.691867+0000","last_active":"2023-10-30T13:23:20.675980+0000","last_peered":"2023-10-30T13:23:20.675980+0000","last_clean":"2023-10-30T13:23:20.675980+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T13:23:20.675980+0000","last_undegraded":"2023-10-30T13:23:20.675980+0000","last_fullsized":"2023-10-30T13:23:20.675980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:02.276042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670494+0000","last_change":"2023-10-30T13:23:18.958210+0000","last_active":"2023-10-30T13:23:20.670494+0000","last_peered":"2023-10-30T13:23:20.670494+0000","last_clean":"2023-10-30T13:23:20.670494+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T13:23:20.670494+0000","last_undegraded":"2023-10-30T13:23:20.670494+0000","last_fullsized":"2023-10-30T13:23:20.670494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:34.653005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613343+0000","last_change":"2023-10-30T13:23:19.713174+0000","last_active":"2023-10-30T13:23:21.613343+0000","last_peered":"2023-10-30T13:23:21.613343+0000","last_clean":"2023-10-30T13:23:21.613343+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T13:23:21.613343+0000","last_undegraded":"2023-10-30T13:23:21.613343+0000","last_fullsized":"2023-10-30T13:23:21.613343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:43.400223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.988980+0000","last_change":"2023-10-30T13:23:18.985185+0000","last_active":"2023-10-30T13:23:20.988980+0000","last_peered":"2023-10-30T13:23:20.988980+0000","last_clean":"2023-10-30T13:23:20.988980+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T13:23:20.988980+0000","last_undegraded":"2023-10-30T13:23:20.988980+0000","last_fullsized":"2023-10-30T13:23:20.988980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:36.230072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670523+0000","last_change":"2023-10-30T13:23:19.680185+0000","last_active":"2023-10-30T13:23:20.670523+0000","last_peered":"2023-10-30T13:23:20.670523+0000","last_clean":"2023-10-30T13:23:20.670523+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T13:23:20.670523+0000","last_undegraded":"2023-10-30T13:23:20.670523+0000","last_fullsized":"2023-10-30T13:23:20.670523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670731+0000","last_change":"2023-10-30T13:23:18.978085+0000","last_active":"2023-10-30T13:23:20.670731+0000","last_peered":"2023-10-30T13:23:20.670731+0000","last_clean":"2023-10-30T13:23:20.670731+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T13:23:20.670731+0000","last_undegraded":"2023-10-30T13:23:20.670731+0000","last_fullsized":"2023-10-30T13:23:20.670731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:47.588972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005467+0000","last_change":"2023-10-30T13:23:19.704291+0000","last_active":"2023-10-30T13:23:21.005467+0000","last_peered":"2023-10-30T13:23:21.005467+0000","last_clean":"2023-10-30T13:23:21.005467+0000","last_became_active":"2023-10-30T13:23:19.704066+0000","last_became_peered":"2023-10-30T13:23:19.704066+0000","last_unstale":"2023-10-30T13:23:21.005467+0000","last_undegraded":"2023-10-30T13:23:21.005467+0000","last_fullsized":"2023-10-30T13:23:21.005467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:13.754194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.968504+0000","last_change":"2023-10-30T13:23:18.970566+0000","last_active":"2023-10-30T13:23:24.968504+0000","last_peered":"2023-10-30T13:23:24.968504+0000","last_clean":"2023-10-30T13:23:24.968504+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T13:23:24.968504+0000","last_undegraded":"2023-10-30T13:23:24.968504+0000","last_fullsized":"2023-10-30T13:23:24.968504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:03.851889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107903+0000","last_change":"2023-10-30T13:23:19.685481+0000","last_active":"2023-10-30T13:23:21.107903+0000","last_peered":"2023-10-30T13:23:21.107903+0000","last_clean":"2023-10-30T13:23:21.107903+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T13:23:21.107903+0000","last_undegraded":"2023-10-30T13:23:21.107903+0000","last_fullsized":"2023-10-30T13:23:21.107903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:33:50.672215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.988346+0000","last_change":"2023-10-30T13:23:18.986394+0000","last_active":"2023-10-30T13:23:20.988346+0000","last_peered":"2023-10-30T13:23:20.988346+0000","last_clean":"2023-10-30T13:23:20.988346+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T13:23:20.988346+0000","last_undegraded":"2023-10-30T13:23:20.988346+0000","last_fullsized":"2023-10-30T13:23:20.988346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:59.986981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675944+0000","last_change":"2023-10-30T13:23:19.681239+0000","last_active":"2023-10-30T13:23:20.675944+0000","last_peered":"2023-10-30T13:23:20.675944+0000","last_clean":"2023-10-30T13:23:20.675944+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T13:23:20.675944+0000","last_undegraded":"2023-10-30T13:23:20.675944+0000","last_fullsized":"2023-10-30T13:23:20.675944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:00.121240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670308+0000","last_change":"2023-10-30T13:23:18.963905+0000","last_active":"2023-10-30T13:23:20.670308+0000","last_peered":"2023-10-30T13:23:20.670308+0000","last_clean":"2023-10-30T13:23:20.670308+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T13:23:20.670308+0000","last_undegraded":"2023-10-30T13:23:20.670308+0000","last_fullsized":"2023-10-30T13:23:20.670308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005091+0000","last_change":"2023-10-30T13:23:19.678697+0000","last_active":"2023-10-30T13:23:21.005091+0000","last_peered":"2023-10-30T13:23:21.005091+0000","last_clean":"2023-10-30T13:23:21.005091+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T13:23:21.005091+0000","last_undegraded":"2023-10-30T13:23:21.005091+0000","last_fullsized":"2023-10-30T13:23:21.005091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:57.442011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670474+0000","last_change":"2023-10-30T13:23:18.952302+0000","last_active":"2023-10-30T13:23:20.670474+0000","last_peered":"2023-10-30T13:23:20.670474+0000","last_clean":"2023-10-30T13:23:20.670474+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T13:23:20.670474+0000","last_undegraded":"2023-10-30T13:23:20.670474+0000","last_fullsized":"2023-10-30T13:23:20.670474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:12.827417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005323+0000","last_change":"2023-10-30T13:23:19.688716+0000","last_active":"2023-10-30T13:23:21.005323+0000","last_peered":"2023-10-30T13:23:21.005323+0000","last_clean":"2023-10-30T13:23:21.005323+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T13:23:21.005323+0000","last_undegraded":"2023-10-30T13:23:21.005323+0000","last_fullsized":"2023-10-30T13:23:21.005323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:26:04.209304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.968535+0000","last_change":"2023-10-30T13:23:18.984912+0000","last_active":"2023-10-30T13:23:24.968535+0000","last_peered":"2023-10-30T13:23:24.968535+0000","last_clean":"2023-10-30T13:23:24.968535+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T13:23:24.968535+0000","last_undegraded":"2023-10-30T13:23:24.968535+0000","last_fullsized":"2023-10-30T13:23:24.968535+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:27:16.205042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674560+0000","last_change":"2023-10-30T13:23:19.718922+0000","last_active":"2023-10-30T13:23:20.674560+0000","last_peered":"2023-10-30T13:23:20.674560+0000","last_clean":"2023-10-30T13:23:20.674560+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T13:23:20.674560+0000","last_undegraded":"2023-10-30T13:23:20.674560+0000","last_fullsized":"2023-10-30T13:23:20.674560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612299+0000","last_change":"2023-10-30T13:23:19.732322+0000","last_active":"2023-10-30T13:23:21.612299+0000","last_peered":"2023-10-30T13:23:21.612299+0000","last_clean":"2023-10-30T13:23:21.612299+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T13:23:21.612299+0000","last_undegraded":"2023-10-30T13:23:21.612299+0000","last_fullsized":"2023-10-30T13:23:21.612299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:21.479138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.968310+0000","last_change":"2023-10-30T13:23:18.993372+0000","last_active":"2023-10-30T13:23:24.968310+0000","last_peered":"2023-10-30T13:23:24.968310+0000","last_clean":"2023-10-30T13:23:24.968310+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T13:23:24.968310+0000","last_undegraded":"2023-10-30T13:23:24.968310+0000","last_fullsized":"2023-10-30T13:23:24.968310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:41.370678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611741+0000","last_change":"2023-10-30T13:23:19.732769+0000","last_active":"2023-10-30T13:23:21.611741+0000","last_peered":"2023-10-30T13:23:21.611741+0000","last_clean":"2023-10-30T13:23:21.611741+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T13:23:21.611741+0000","last_undegraded":"2023-10-30T13:23:21.611741+0000","last_fullsized":"2023-10-30T13:23:21.611741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:56.378098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.967130+0000","last_change":"2023-10-30T13:23:18.987633+0000","last_active":"2023-10-30T13:23:24.967130+0000","last_peered":"2023-10-30T13:23:24.967130+0000","last_clean":"2023-10-30T13:23:24.967130+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T13:23:24.967130+0000","last_undegraded":"2023-10-30T13:23:24.967130+0000","last_fullsized":"2023-10-30T13:23:24.967130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:09:33.324898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107906+0000","last_change":"2023-10-30T13:23:19.722311+0000","last_active":"2023-10-30T13:23:21.107906+0000","last_peered":"2023-10-30T13:23:21.107906+0000","last_clean":"2023-10-30T13:23:21.107906+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T13:23:21.107906+0000","last_undegraded":"2023-10-30T13:23:21.107906+0000","last_fullsized":"2023-10-30T13:23:21.107906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:44.204074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005592+0000","last_change":"2023-10-30T13:23:18.958470+0000","last_active":"2023-10-30T13:23:21.005592+0000","last_peered":"2023-10-30T13:23:21.005592+0000","last_clean":"2023-10-30T13:23:21.005592+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T13:23:21.005592+0000","last_undegraded":"2023-10-30T13:23:21.005592+0000","last_fullsized":"2023-10-30T13:23:21.005592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:21.507744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005410+0000","last_change":"2023-10-30T13:23:19.699742+0000","last_active":"2023-10-30T13:23:21.005410+0000","last_peered":"2023-10-30T13:23:21.005410+0000","last_clean":"2023-10-30T13:23:21.005410+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T13:23:21.005410+0000","last_undegraded":"2023-10-30T13:23:21.005410+0000","last_fullsized":"2023-10-30T13:23:21.005410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:13.504378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.966390+0000","last_change":"2023-10-30T13:23:18.980952+0000","last_active":"2023-10-30T13:23:24.966390+0000","last_peered":"2023-10-30T13:23:24.966390+0000","last_clean":"2023-10-30T13:23:24.966390+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T13:23:24.966390+0000","last_undegraded":"2023-10-30T13:23:24.966390+0000","last_fullsized":"2023-10-30T13:23:24.966390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:44.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":69969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":200704,"data_stored":69879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":806912,"data_stored":660885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:23:31.432 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T13:23:31.432 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-30T13:23:31.432 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-30T13:23:31.433 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-30T13:23:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9ff2a17640 1 Processor -- start 2023-10-30T13:23:31.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9ff2a17640 1 -- start start 2023-10-30T13:23:31.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9ff2a17640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9fec153c60 0x7f9fec0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9ff2a17640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9fec159830 0x7f9fec159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9ff2a17640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0cb780 con 0x7f9fec0c4430 2023-10-30T13:23:31.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9ff2a17640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fec154040 con 0x7f9fec153c60 2023-10-30T13:23:31.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9ff2a17640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fec15a2e0 con 0x7f9fec159830 2023-10-30T13:23:31.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9febfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9fec153c60 0x7f9fec0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9febfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9fec153c60 0x7f9fec0ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39874/0 (socket says 172.21.15.46:39874) 2023-10-30T13:23:31.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9febfff640 1 -- 172.21.15.46:0/2060289199 learned_addr learned my addr 172.21.15.46:0/2060289199 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:31.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9feb7fe640 1 --1- 172.21.15.46:0/2060289199 >> v1:172.21.15.46:6789/0 conn(0x7f9fec0c4430 0x7f9fec0cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48568/0 (socket says 172.21.15.46:48568) 2023-10-30T13:23:31.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9ff0fb4640 1 --2- 172.21.15.46:0/2060289199 >> v2:172.21.15.154:3300/0 conn(0x7f9fec159830 0x7f9fec159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9febfff640 1 -- 172.21.15.46:0/2060289199 >> v2:172.21.15.154:3300/0 conn(0x7f9fec159830 msgr2=0x7f9fec159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:31.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9febfff640 1 --2- 172.21.15.46:0/2060289199 >> v2:172.21.15.154:3300/0 conn(0x7f9fec159830 0x7f9fec159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9febfff640 1 -- 172.21.15.46:0/2060289199 >> v1:172.21.15.46:6789/0 conn(0x7f9fec0c4430 legacy=0x7f9fec0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.608+0000 7f9febfff640 1 -- 172.21.15.46:0/2060289199 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec15a5c0 con 0x7f9fec153c60 2023-10-30T13:23:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.612+0000 7f9febfff640 1 --2- 172.21.15.46:0/2060289199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fec153c60 0x7f9fec0ca360 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9fec073da0 tx=0x7f9fdc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33473f1f47e61581 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.612+0000 7f9feaffd640 1 -- 172.21.15.46:0/2060289199 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc0ab5f0 con 0x7f9fec153c60 2023-10-30T13:23:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.612+0000 7f9feaffd640 1 -- 172.21.15.46:0/2060289199 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fdc0ab750 con 0x7f9fec153c60 2023-10-30T13:23:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.612+0000 7f9feaffd640 1 -- 172.21.15.46:0/2060289199 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc0abae0 con 0x7f9fec153c60 2023-10-30T13:23:31.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.612+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2060289199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fec153c60 msgr2=0x7f9fec0ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.612+0000 7f9ff2a17640 1 --2- 172.21.15.46:0/2060289199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fec153c60 0x7f9fec0ca360 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9fec073da0 tx=0x7f9fdc01c500 comp rx=0 tx=0).stop 2023-10-30T13:23:31.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.612+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2060289199 shutdown_connections 2023-10-30T13:23:31.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.612+0000 7f9ff2a17640 1 --2- 172.21.15.46:0/2060289199 >> v2:172.21.15.154:3300/0 conn(0x7f9fec159830 0x7f9fec159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.612+0000 7f9ff2a17640 1 --2- 172.21.15.46:0/2060289199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fec153c60 0x7f9fec0ca360 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.612+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2060289199 >> 172.21.15.46:0/2060289199 conn(0x7f9fec06bfe0 msgr2=0x7f9fec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:31.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.612+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2060289199 shutdown_connections 2023-10-30T13:23:31.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.612+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2060289199 wait complete. 2023-10-30T13:23:31.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff2a17640 1 Processor -- start 2023-10-30T13:23:31.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff2a17640 1 -- start start 2023-10-30T13:23:31.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff2a17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fec0c4430 0x7f9fec0c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff2a17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fec153c60 0x7f9fec1f5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff2a17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fec159830 0x7f9fec145e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff2a17640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec142530 con 0x7f9fec159830 2023-10-30T13:23:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff2a17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec144940 con 0x7f9fec0c4430 2023-10-30T13:23:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff2a17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec1439e0 con 0x7f9fec153c60 2023-10-30T13:23:31.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff0fb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fec159830 0x7f9fec145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff0fb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fec159830 0x7f9fec145e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39882/0 (socket says 172.21.15.46:39882) 2023-10-30T13:23:31.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff0fb4640 1 -- 172.21.15.46:0/2924666448 learned_addr learned my addr 172.21.15.46:0/2924666448 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:31.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9feb7fe640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fec153c60 0x7f9fec1f5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9febfff640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fec0c4430 0x7f9fec0c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff0fb4640 1 -- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fec153c60 msgr2=0x7f9fec1f5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff0fb4640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fec153c60 0x7f9fec1f5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff0fb4640 1 -- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fec0c4430 msgr2=0x7f9fec0c7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff0fb4640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fec0c4430 0x7f9fec0c7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff0fb4640 1 -- 172.21.15.46:0/2924666448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc0094d0 con 0x7f9fec159830 2023-10-30T13:23:31.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9febfff640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fec0c4430 0x7f9fec0c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:31.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff0fb4640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fec159830 0x7f9fec145e20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4007ec0 tx=0x7f9fe400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:31.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9fe8ff9640 1 -- 172.21.15.46:0/2924666448 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4014620 con 0x7f9fec159830 2023-10-30T13:23:31.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9fe8ff9640 1 -- 172.21.15.46:0/2924666448 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe40147a0 con 0x7f9fec159830 2023-10-30T13:23:31.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9fe8ff9640 1 -- 172.21.15.46:0/2924666448 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4014b50 con 0x7f9fec159830 2023-10-30T13:23:31.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2924666448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec1435f0 con 0x7f9fec159830 2023-10-30T13:23:31.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.616+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2924666448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec14fd90 con 0x7f9fec159830 2023-10-30T13:23:31.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.620+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2924666448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fec04e170 con 0x7f9fec159830 2023-10-30T13:23:31.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.620+0000 7f9fe8ff9640 1 -- 172.21.15.46:0/2924666448 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9fe4014cb0 con 0x7f9fec159830 2023-10-30T13:23:31.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.620+0000 7f9fe8ff9640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9fd80664d0 0x7f9fd8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.620+0000 7f9fe8ff9640 1 -- 172.21.15.46:0/2924666448 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9fe4084800 con 0x7f9fec159830 2023-10-30T13:23:31.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.620+0000 7f9febfff640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9fd80664d0 0x7f9fd8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.620+0000 7f9febfff640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9fd80664d0 0x7f9fd8068990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0af000 tx=0x7f9fdc0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:31.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.624+0000 7f9fe8ff9640 1 -- 172.21.15.46:0/2924666448 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9fe4051be0 con 0x7f9fec159830 2023-10-30T13:23:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.816+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2924666448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9fec143b20 con 0x7f9fec159830 2023-10-30T13:23:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.816+0000 7f9fe8ff9640 1 -- 172.21.15.46:0/2924666448 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f9fe4055200 con 0x7f9fec159830 2023-10-30T13:23:31.825 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:31.825 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-30T13:23:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.820+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9fd80664d0 msgr2=0x7f9fd8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.820+0000 7f9ff2a17640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9fd80664d0 0x7f9fd8068990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0af000 tx=0x7f9fdc0a7000 comp rx=0 tx=0).stop 2023-10-30T13:23:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.820+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2924666448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fec159830 msgr2=0x7f9fec145e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.820+0000 7f9ff2a17640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fec159830 0x7f9fec145e20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4007ec0 tx=0x7f9fe400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.820+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2924666448 shutdown_connections 2023-10-30T13:23:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.820+0000 7f9ff2a17640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9fd80664d0 0x7f9fd8068990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.820+0000 7f9ff2a17640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fec159830 0x7f9fec145e20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.820+0000 7f9ff2a17640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fec153c60 0x7f9fec1f5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.820+0000 7f9ff2a17640 1 --2- 172.21.15.46:0/2924666448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fec0c4430 0x7f9fec0c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.820+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2924666448 >> 172.21.15.46:0/2924666448 conn(0x7f9fec06bfe0 msgr2=0x7f9fec0c56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:31.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.820+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2924666448 shutdown_connections 2023-10-30T13:23:31.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.820+0000 7f9ff2a17640 1 -- 172.21.15.46:0/2924666448 wait complete. 2023-10-30T13:23:31.845 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-30T13:23:31.846 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:31.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.980+0000 7f32bd13e640 1 Processor -- start 2023-10-30T13:23:31.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.980+0000 7f32bd13e640 1 -- start start 2023-10-30T13:23:31.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32bd13e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32b81538a0 0x7f32b815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32bd13e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32b80cf160 0x7f32b80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32bd13e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b80cb800 con 0x7f32b80ca0e0 2023-10-30T13:23:31.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32bd13e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b815dc30 con 0x7f32b80cf160 2023-10-30T13:23:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32bd13e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b80cb330 con 0x7f32b81538a0 2023-10-30T13:23:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32b759e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32b80cf160 0x7f32b80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32b759e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32b80cf160 0x7f32b80cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39890/0 (socket says 172.21.15.46:39890) 2023-10-30T13:23:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32b659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f32b80ca0e0 0x7f32b815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48580/0 (socket says 172.21.15.46:48580) 2023-10-30T13:23:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32b759e640 1 -- 172.21.15.46:0/1803026341 learned_addr learned my addr 172.21.15.46:0/1803026341 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:31.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32b6d9d640 1 --2- 172.21.15.46:0/1803026341 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32b81538a0 0x7f32b815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:31.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32b759e640 1 -- 172.21.15.46:0/1803026341 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32b81538a0 msgr2=0x7f32b815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:31.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32b759e640 1 --2- 172.21.15.46:0/1803026341 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32b81538a0 0x7f32b815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32b759e640 1 -- 172.21.15.46:0/1803026341 >> v1:172.21.15.46:6789/0 conn(0x7f32b80ca0e0 legacy=0x7f32b815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.984+0000 7f32b759e640 1 -- 172.21.15.46:0/1803026341 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b80cad70 con 0x7f32b80cf160 2023-10-30T13:23:31.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32b759e640 1 --2- 172.21.15.46:0/1803026341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32b80cf160 0x7f32b80cf590 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009e50 tx=0x7f32b001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817ec4d34d5f2e5d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:31.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32b5d9b640 1 -- 172.21.15.46:0/1803026341 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b00af400 con 0x7f32b80cf160 2023-10-30T13:23:31.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32b5d9b640 1 -- 172.21.15.46:0/1803026341 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b00ab050 con 0x7f32b80cf160 2023-10-30T13:23:31.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32b5d9b640 1 -- 172.21.15.46:0/1803026341 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b00af780 con 0x7f32b80cf160 2023-10-30T13:23:31.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 -- 172.21.15.46:0/1803026341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32b80cf160 msgr2=0x7f32b80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:31.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 --2- 172.21.15.46:0/1803026341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32b80cf160 0x7f32b80cf590 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009e50 tx=0x7f32b001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 -- 172.21.15.46:0/1803026341 shutdown_connections 2023-10-30T13:23:31.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 --2- 172.21.15.46:0/1803026341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32b80cf160 0x7f32b80cf590 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 --2- 172.21.15.46:0/1803026341 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32b81538a0 0x7f32b815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:31.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 -- 172.21.15.46:0/1803026341 >> 172.21.15.46:0/1803026341 conn(0x7f32b806bfe0 msgr2=0x7f32b806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:31.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 -- 172.21.15.46:0/1803026341 shutdown_connections 2023-10-30T13:23:31.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 -- 172.21.15.46:0/1803026341 wait complete. 2023-10-30T13:23:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 Processor -- start 2023-10-30T13:23:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 -- start start 2023-10-30T13:23:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b80ca0e0 0x7f32b815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32b80cf160 0x7f32b815b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32b81538a0 0x7f32b815b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b81e3960 con 0x7f32b80cf160 2023-10-30T13:23:32.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b81e4750 con 0x7f32b80ca0e0 2023-10-30T13:23:32.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.988+0000 7f32bd13e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b81e3ba0 con 0x7f32b81538a0 2023-10-30T13:23:32.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32b659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32b80cf160 0x7f32b815b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32b659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32b80cf160 0x7f32b815b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39894/0 (socket says 172.21.15.46:39894) 2023-10-30T13:23:32.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32b659c640 1 -- 172.21.15.46:0/1469809157 learned_addr learned my addr 172.21.15.46:0/1469809157 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:32.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32b759e640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32b81538a0 0x7f32b815b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32b6d9d640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b80ca0e0 0x7f32b815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32b659c640 1 -- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32b81538a0 msgr2=0x7f32b815b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32b659c640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32b81538a0 0x7f32b815b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32b659c640 1 -- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b80ca0e0 msgr2=0x7f32b815a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32b659c640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b80ca0e0 0x7f32b815a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32b659c640 1 -- 172.21.15.46:0/1469809157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b0009b30 con 0x7f32b80cf160 2023-10-30T13:23:32.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32b659c640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32b80cf160 0x7f32b815b300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00c490 tx=0x7f32ac00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:32.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f329f7fe640 1 -- 172.21.15.46:0/1469809157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac007480 con 0x7f32b80cf160 2023-10-30T13:23:32.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f329f7fe640 1 -- 172.21.15.46:0/1469809157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ac0075e0 con 0x7f32b80cf160 2023-10-30T13:23:32.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32bd13e640 1 -- 172.21.15.46:0/1469809157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b81e3430 con 0x7f32b80cf160 2023-10-30T13:23:32.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f329f7fe640 1 -- 172.21.15.46:0/1469809157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac004a10 con 0x7f32b80cf160 2023-10-30T13:23:32.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32bd13e640 1 -- 172.21.15.46:0/1469809157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b81e3db0 con 0x7f32b80cf160 2023-10-30T13:23:32.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f32bd13e640 1 -- 172.21.15.46:0/1469809157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b80b2a50 con 0x7f32b80cf160 2023-10-30T13:23:32.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.992+0000 7f329f7fe640 1 -- 172.21.15.46:0/1469809157 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f32ac004b70 con 0x7f32b80cf160 2023-10-30T13:23:32.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.996+0000 7f329f7fe640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32a40664a0 0x7f32a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.996+0000 7f329f7fe640 1 -- 172.21.15.46:0/1469809157 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32ac083360 con 0x7f32b80cf160 2023-10-30T13:23:32.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.996+0000 7f32b6d9d640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32a40664a0 0x7f32a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:31.996+0000 7f32b6d9d640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32a40664a0 0x7f32a4068960 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f32a800b930 tx=0x7f32a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:32.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.000+0000 7f329f7fe640 1 -- 172.21.15.46:0/1469809157 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f32ac0507f0 con 0x7f32b80cf160 2023-10-30T13:23:32.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.160+0000 7f32bd13e640 1 -- 172.21.15.46:0/1469809157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32b804e170 con 0x7f32b80cf160 2023-10-30T13:23:32.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.160+0000 7f329f7fe640 1 -- 172.21.15.46:0/1469809157 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f32ac053e10 con 0x7f32b80cf160 2023-10-30T13:23:32.168 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:32.168 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:20.665723+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:32.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.164+0000 7f32bd13e640 1 -- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32a40664a0 msgr2=0x7f32a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.164+0000 7f32bd13e640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32a40664a0 0x7f32a4068960 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f32a800b930 tx=0x7f32a8008040 comp rx=0 tx=0).stop 2023-10-30T13:23:32.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.164+0000 7f32bd13e640 1 -- 172.21.15.46:0/1469809157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32b80cf160 msgr2=0x7f32b815b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.164+0000 7f32bd13e640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32b80cf160 0x7f32b815b300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00c490 tx=0x7f32ac00c950 comp rx=0 tx=0).stop 2023-10-30T13:23:32.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.164+0000 7f32bd13e640 1 -- 172.21.15.46:0/1469809157 shutdown_connections 2023-10-30T13:23:32.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.164+0000 7f32bd13e640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32a40664a0 0x7f32a4068960 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.164+0000 7f32bd13e640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32b81538a0 0x7f32b815b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.164+0000 7f32bd13e640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32b80cf160 0x7f32b815b300 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.164+0000 7f32bd13e640 1 --2- 172.21.15.46:0/1469809157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32b80ca0e0 0x7f32b815a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.164+0000 7f32bd13e640 1 -- 172.21.15.46:0/1469809157 >> 172.21.15.46:0/1469809157 conn(0x7f32b806bfe0 msgr2=0x7f32b80c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:32.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.164+0000 7f32bd13e640 1 -- 172.21.15.46:0/1469809157 shutdown_connections 2023-10-30T13:23:32.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.164+0000 7f32bd13e640 1 -- 172.21.15.46:0/1469809157 wait complete. 2023-10-30T13:23:32.190 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:23:06.788949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:23:17.648827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:23:18.326967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:23:32.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:23:32.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.328+0000 7f7cdcf56640 1 Processor -- start 2023-10-30T13:23:32.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.328+0000 7f7cdcf56640 1 -- start start 2023-10-30T13:23:32.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cdcf56640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7cd8153c20 0x7f7cd815a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cdcf56640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cd80cb4c0 0x7f7cd80cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cdcf56640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd815a6b0 con 0x7f7cd8153840 2023-10-30T13:23:32.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cdcf56640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd815dbc0 con 0x7f7cd8153c20 2023-10-30T13:23:32.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cdcf56640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd815d730 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cd6ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7cd8153c20 0x7f7cd815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cd7fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cd80cb4c0 0x7f7cd80cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cd77fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f7cd8153840 0x7f7cd80c3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:53216/0 (socket says 172.21.15.46:53216) 2023-10-30T13:23:32.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cd6ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7cd8153c20 0x7f7cd815a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39660/0 (socket says 172.21.15.46:39660) 2023-10-30T13:23:32.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cd77fe640 1 -- 172.21.15.46:0/299529540 learned_addr learned my addr 172.21.15.46:0/299529540 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:32.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cd7fff640 1 -- 172.21.15.46:0/299529540 >> v2:172.21.15.154:3300/0 conn(0x7f7cd8153c20 msgr2=0x7f7cd815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:32.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cd7fff640 1 --2- 172.21.15.46:0/299529540 >> v2:172.21.15.154:3300/0 conn(0x7f7cd8153c20 0x7f7cd815a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cd7fff640 1 -- 172.21.15.46:0/299529540 >> v1:172.21.15.154:6789/0 conn(0x7f7cd8153840 legacy=0x7f7cd80c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.332+0000 7f7cd7fff640 1 -- 172.21.15.46:0/299529540 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd80ca8f0 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cd7fff640 1 --2- 172.21.15.46:0/299529540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cd80cb4c0 0x7f7cd80cef40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009d90 tx=0x7f7cd001f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d5a6956a00c130a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:32.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cd67fc640 1 -- 172.21.15.46:0/299529540 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd00af600 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cd67fc640 1 -- 172.21.15.46:0/299529540 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd00ab050 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cd67fc640 1 -- 172.21.15.46:0/299529540 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd00af980 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cdcf56640 1 -- 172.21.15.46:0/299529540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cd80cb4c0 msgr2=0x7f7cd80cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cdcf56640 1 --2- 172.21.15.46:0/299529540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cd80cb4c0 0x7f7cd80cef40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009d90 tx=0x7f7cd001f130 comp rx=0 tx=0).stop 2023-10-30T13:23:32.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cdcf56640 1 -- 172.21.15.46:0/299529540 shutdown_connections 2023-10-30T13:23:32.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cdcf56640 1 --2- 172.21.15.46:0/299529540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cd80cb4c0 0x7f7cd80cef40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cdcf56640 1 --2- 172.21.15.46:0/299529540 >> v2:172.21.15.154:3300/0 conn(0x7f7cd8153c20 0x7f7cd815a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cdcf56640 1 -- 172.21.15.46:0/299529540 >> 172.21.15.46:0/299529540 conn(0x7f7cd806be40 msgr2=0x7f7cd806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:32.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cdcf56640 1 -- 172.21.15.46:0/299529540 shutdown_connections 2023-10-30T13:23:32.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cdcf56640 1 -- 172.21.15.46:0/299529540 wait complete. 2023-10-30T13:23:32.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cdcf56640 1 Processor -- start 2023-10-30T13:23:32.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.336+0000 7f7cdcf56640 1 -- start start 2023-10-30T13:23:32.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cdcf56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd80cb4c0 0x7f7cd814ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cdcf56640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cd8153840 0x7f7cd81ef380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cdcf56640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cd8153c20 0x7f7cd81ef8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cdcf56640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd81f3250 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cdcf56640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd81eeea0 con 0x7f7cd8153840 2023-10-30T13:23:32.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cdcf56640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd815ba30 con 0x7f7cd8153c20 2023-10-30T13:23:32.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cd77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd80cb4c0 0x7f7cd814ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cd77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd80cb4c0 0x7f7cd814ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39902/0 (socket says 172.21.15.46:39902) 2023-10-30T13:23:32.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cd77fe640 1 -- 172.21.15.46:0/2195967110 learned_addr learned my addr 172.21.15.46:0/2195967110 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:32.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cd6ffd640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cd8153840 0x7f7cd81ef380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cd7fff640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cd8153c20 0x7f7cd81ef8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cd77fe640 1 -- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cd8153c20 msgr2=0x7f7cd81ef8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cd77fe640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cd8153c20 0x7f7cd81ef8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cd77fe640 1 -- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cd8153840 msgr2=0x7f7cd81ef380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cd77fe640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cd8153840 0x7f7cd81ef380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cd77fe640 1 -- 172.21.15.46:0/2195967110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd0009a70 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cd6ffd640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cd8153840 0x7f7cd81ef380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:32.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cd77fe640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd80cb4c0 0x7f7cd814ed90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800c500 tx=0x7f7cc800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:32.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cbffff640 1 -- 172.21.15.46:0/2195967110 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8005cf0 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cbffff640 1 -- 172.21.15.46:0/2195967110 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc8005e50 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cdcf56640 1 -- 172.21.15.46:0/2195967110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd815bd10 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cbffff640 1 -- 172.21.15.46:0/2195967110 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc80072f0 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cdcf56640 1 -- 172.21.15.46:0/2195967110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd815b690 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.340+0000 7f7cdcf56640 1 -- 172.21.15.46:0/2195967110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd80c3eb0 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.344+0000 7f7cbffff640 1 -- 172.21.15.46:0/2195967110 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7cc8007450 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.344+0000 7f7cbffff640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7cc40664a0 0x7f7cc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.344+0000 7f7cbffff640 1 -- 172.21.15.46:0/2195967110 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7cc80831f0 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.344+0000 7f7cd6ffd640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7cc40664a0 0x7f7cc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.344+0000 7f7cd6ffd640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7cc40664a0 0x7f7cc4068960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7cd814d770 tx=0x7f7ccc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:32.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.348+0000 7f7cbffff640 1 -- 172.21.15.46:0/2195967110 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7cc8050710 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 -- 172.21.15.46:0/2195967110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7cd804e9c0 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cbffff640 1 -- 172.21.15.46:0/2195967110 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7cc8017020 con 0x7f7cd80cb4c0 2023-10-30T13:23:32.514 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-10-30T13:23:32.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 -- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7cc40664a0 msgr2=0x7f7cc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7cc40664a0 0x7f7cc4068960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7cd814d770 tx=0x7f7ccc008040 comp rx=0 tx=0).stop 2023-10-30T13:23:32.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 -- 172.21.15.46:0/2195967110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd80cb4c0 msgr2=0x7f7cd814ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd80cb4c0 0x7f7cd814ed90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800c500 tx=0x7f7cc800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 -- 172.21.15.46:0/2195967110 shutdown_connections 2023-10-30T13:23:32.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7cc40664a0 0x7f7cc4068960 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cd8153c20 0x7f7cd81ef8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cd8153840 0x7f7cd81ef380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 --2- 172.21.15.46:0/2195967110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd80cb4c0 0x7f7cd814ed90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 -- 172.21.15.46:0/2195967110 >> 172.21.15.46:0/2195967110 conn(0x7f7cd806be40 msgr2=0x7f7cd80cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:32.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 -- 172.21.15.46:0/2195967110 shutdown_connections 2023-10-30T13:23:32.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.508+0000 7f7cdcf56640 1 -- 172.21.15.46:0/2195967110 wait complete. 2023-10-30T13:23:32.536 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:23:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd1a1c4640 1 Processor -- start 2023-10-30T13:23:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd1a1c4640 1 -- start start 2023-10-30T13:23:32.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd1a1c4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcd1415d040 0x7fcd1415d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd1a1c4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd140cb010 con 0x7fcd140cb940 2023-10-30T13:23:32.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd1a1c4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1415d9f0 con 0x7fcd140b4d50 2023-10-30T13:23:32.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd1a1c4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1415a890 con 0x7fcd1415d040 2023-10-30T13:23:32.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd137fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcd140cb940 0x7fcd140cbd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48584/0 (socket says 172.21.15.46:48584) 2023-10-30T13:23:32.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd137fe640 1 -- 172.21.15.46:0/4292752323 learned_addr learned my addr 172.21.15.46:0/4292752323 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:32.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968628873 0 0) 0x7fcd140cb010 con 0x7fcd140cb940 2023-10-30T13:23:32.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccfc003530 con 0x7fcd140cb940 2023-10-30T13:23:32.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd13fff640 1 --2- 172.21.15.46:0/4292752323 >> v2:172.21.15.154:3300/0 conn(0x7fcd1415d040 0x7fcd1415d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 963299213 0 0) 0x7fccfc003530 con 0x7fcd140cb940 2023-10-30T13:23:32.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 >> v2:172.21.15.154:3300/0 conn(0x7fcd1415d040 msgr2=0x7fcd1415d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:32.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 --2- 172.21.15.46:0/4292752323 >> v2:172.21.15.154:3300/0 conn(0x7fcd1415d040 0x7fcd1415d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 >> v1:172.21.15.154:6789/0 conn(0x7fcd140b4d50 legacy=0x7fcd140c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd140ca9b0 con 0x7fcd140cb940 2023-10-30T13:23:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fcd04001680 con 0x7fcd140cb940 2023-10-30T13:23:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 >> v1:172.21.15.46:6789/0 conn(0x7fcd140cb940 legacy=0x7fcd140cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 --2- 172.21.15.46:0/4292752323 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccfc003980 0x7fccfc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 --2- 172.21.15.46:0/4292752323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccfc0047d0 0x7fccfc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 --2- 172.21.15.46:0/4292752323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccfc005190 0x7fccfc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1415a890 con 0x7fccfc0047d0 2023-10-30T13:23:32.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc00c0e0 con 0x7fccfc005190 2023-10-30T13:23:32.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc00c2a0 con 0x7fccfc003980 2023-10-30T13:23:32.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd12ffd640 1 --2- 172.21.15.46:0/4292752323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccfc0047d0 0x7fccfc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd137fe640 1 --2- 172.21.15.46:0/4292752323 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccfc003980 0x7fccfc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd13fff640 1 --2- 172.21.15.46:0/4292752323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccfc005190 0x7fccfc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd12ffd640 1 -- 172.21.15.46:0/4292752323 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccfc003980 msgr2=0x7fccfc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd12ffd640 1 --2- 172.21.15.46:0/4292752323 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccfc003980 0x7fccfc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd12ffd640 1 -- 172.21.15.46:0/4292752323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccfc005190 msgr2=0x7fccfc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd12ffd640 1 --2- 172.21.15.46:0/4292752323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccfc005190 0x7fccfc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd12ffd640 1 -- 172.21.15.46:0/4292752323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc003530 con 0x7fccfc0047d0 2023-10-30T13:23:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.676+0000 7fcd13fff640 1 -- 172.21.15.46:0/4292752323 reap_dead start 2023-10-30T13:23:32.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.680+0000 7fcd12ffd640 1 --2- 172.21.15.46:0/4292752323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccfc0047d0 0x7fccfc004c50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcd08001780 tx=0x7fcd0801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ead91998fd21c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:32.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.680+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd080052d0 con 0x7fccfc0047d0 2023-10-30T13:23:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.680+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd080a7050 con 0x7fccfc0047d0 2023-10-30T13:23:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.680+0000 7fcd127fc640 1 -- 172.21.15.46:0/4292752323 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd08003600 con 0x7fccfc0047d0 2023-10-30T13:23:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.680+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/4292752323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccfc0047d0 msgr2=0x7fccfc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.680+0000 7fcd1a1c4640 1 --2- 172.21.15.46:0/4292752323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccfc0047d0 0x7fccfc004c50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcd08001780 tx=0x7fcd0801c3e0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.680+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/4292752323 shutdown_connections 2023-10-30T13:23:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.680+0000 7fcd1a1c4640 1 --2- 172.21.15.46:0/4292752323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccfc0047d0 0x7fccfc004c50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.680+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/4292752323 >> 172.21.15.46:0/4292752323 conn(0x7fcd14065a80 msgr2=0x7fcd14065e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.680+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/4292752323 shutdown_connections 2023-10-30T13:23:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.680+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/4292752323 wait complete. 2023-10-30T13:23:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd1a1c4640 1 Processor -- start 2023-10-30T13:23:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd1a1c4640 1 -- start start 2023-10-30T13:23:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd1a1c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccfc0047d0 0x7fcd141524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd1a1c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1415ca00 0x7fcd1415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd1a1c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd140c5800 0x7fcd1414b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd1a1c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd140ca350 con 0x7fcd140c5800 2023-10-30T13:23:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd1a1c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1414cff0 con 0x7fccfc0047d0 2023-10-30T13:23:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd1a1c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd140c3ec0 con 0x7fcd1415ca00 2023-10-30T13:23:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd13fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd140c5800 0x7fcd1414b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd13fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd140c5800 0x7fcd1414b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39914/0 (socket says 172.21.15.46:39914) 2023-10-30T13:23:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd13fff640 1 -- 172.21.15.46:0/622794462 learned_addr learned my addr 172.21.15.46:0/622794462 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd137fe640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccfc0047d0 0x7fcd141524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd12ffd640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1415ca00 0x7fcd1415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd13fff640 1 -- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1415ca00 msgr2=0x7fcd1415a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd13fff640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1415ca00 0x7fcd1415a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd13fff640 1 -- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccfc0047d0 msgr2=0x7fcd141524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd13fff640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccfc0047d0 0x7fcd141524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd13fff640 1 -- 172.21.15.46:0/622794462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1414dc20 con 0x7fcd140c5800 2023-10-30T13:23:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd13fff640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd140c5800 0x7fcd1414b370 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcd14159850 tx=0x7fcd0c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fccfbfff640 1 -- 172.21.15.46:0/622794462 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c007c80 con 0x7fcd140c5800 2023-10-30T13:23:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fccfbfff640 1 -- 172.21.15.46:0/622794462 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0c007e00 con 0x7fcd140c5800 2023-10-30T13:23:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/622794462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd140c29b0 con 0x7fcd140c5800 2023-10-30T13:23:32.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fccfbfff640 1 -- 172.21.15.46:0/622794462 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c015660 con 0x7fcd140c5800 2023-10-30T13:23:32.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.684+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/622794462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd140c3900 con 0x7fcd140c5800 2023-10-30T13:23:32.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.688+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/622794462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1404e170 con 0x7fcd140c5800 2023-10-30T13:23:32.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.688+0000 7fccfbfff640 1 -- 172.21.15.46:0/622794462 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd0c0157c0 con 0x7fcd140c5800 2023-10-30T13:23:32.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.688+0000 7fccfbfff640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd000664d0 0x7fcd00068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.688+0000 7fccfbfff640 1 -- 172.21.15.46:0/622794462 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd0c0842a0 con 0x7fcd140c5800 2023-10-30T13:23:32.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.688+0000 7fcd137fe640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd000664d0 0x7fcd00068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.688+0000 7fcd137fe640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd000664d0 0x7fcd00068990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcd04007cc0 tx=0x7fcd04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:32.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.692+0000 7fccfbfff640 1 -- 172.21.15.46:0/622794462 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd0c051710 con 0x7fcd140c5800 2023-10-30T13:23:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/622794462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcd1414dc20 con 0x7fcd140c5800 2023-10-30T13:23:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fccfbfff640 1 -- 172.21.15.46:0/622794462 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcd1414dc20 con 0x7fcd140c5800 2023-10-30T13:23:32.858 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-10-30T13:23:32.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd000664d0 msgr2=0x7fcd00068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd000664d0 0x7fcd00068990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcd04007cc0 tx=0x7fcd04009040 comp rx=0 tx=0).stop 2023-10-30T13:23:32.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/622794462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd140c5800 msgr2=0x7fcd1414b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd140c5800 0x7fcd1414b370 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcd14159850 tx=0x7fcd0c007700 comp rx=0 tx=0).stop 2023-10-30T13:23:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/622794462 shutdown_connections 2023-10-30T13:23:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd000664d0 0x7fcd00068990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd140c5800 0x7fcd1414b370 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1415ca00 0x7fcd1415a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 --2- 172.21.15.46:0/622794462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccfc0047d0 0x7fcd141524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/622794462 >> 172.21.15.46:0/622794462 conn(0x7fcd14065a80 msgr2=0x7fcd140c86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/622794462 shutdown_connections 2023-10-30T13:23:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:32.852+0000 7fcd1a1c4640 1 -- 172.21.15.46:0/622794462 wait complete. 2023-10-30T13:23:32.879 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:23:33.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69ed50f640 1 Processor -- start 2023-10-30T13:23:33.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69ed50f640 1 -- start start 2023-10-30T13:23:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69ed50f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f69e8159930 0x7f69e80cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69ed50f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69e80cf6e0 0x7f69e815d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69ed50f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e815a920 con 0x7f69e8153980 2023-10-30T13:23:33.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69ed50f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e80cb930 con 0x7f69e80cf6e0 2023-10-30T13:23:33.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69ed50f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e8159dc0 con 0x7f69e8159930 2023-10-30T13:23:33.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69e77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69e80cf6e0 0x7f69e815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69e77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69e80cf6e0 0x7f69e815d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39928/0 (socket says 172.21.15.46:39928) 2023-10-30T13:23:33.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69e77fe640 1 -- 172.21.15.46:0/1916791046 learned_addr learned my addr 172.21.15.46:0/1916791046 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:33.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69dffff640 1 --2- 172.21.15.46:0/1916791046 >> v2:172.21.15.154:3300/0 conn(0x7f69e8159930 0x7f69e80cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69e77fe640 1 -- 172.21.15.46:0/1916791046 >> v2:172.21.15.154:3300/0 conn(0x7f69e8159930 msgr2=0x7f69e80cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:33.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69e77fe640 1 --2- 172.21.15.46:0/1916791046 >> v2:172.21.15.154:3300/0 conn(0x7f69e8159930 0x7f69e80cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69e77fe640 1 -- 172.21.15.46:0/1916791046 >> v1:172.21.15.154:6789/0 conn(0x7f69e8153980 legacy=0x7f69e80caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.020+0000 7f69e77fe640 1 -- 172.21.15.46:0/1916791046 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e815a080 con 0x7f69e80cf6e0 2023-10-30T13:23:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.024+0000 7f69e77fe640 1 --2- 172.21.15.46:0/1916791046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e80cf6e0 0x7f69e815d570 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f69e80742c0 tx=0x7f69e001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08327ac633a23bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.024+0000 7f69e67fc640 1 -- 172.21.15.46:0/1916791046 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e00a5770 con 0x7f69e80cf6e0 2023-10-30T13:23:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.024+0000 7f69e67fc640 1 -- 172.21.15.46:0/1916791046 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e00a58d0 con 0x7f69e80cf6e0 2023-10-30T13:23:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.024+0000 7f69e67fc640 1 -- 172.21.15.46:0/1916791046 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e00a5c60 con 0x7f69e80cf6e0 2023-10-30T13:23:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.024+0000 7f69ed50f640 1 -- 172.21.15.46:0/1916791046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e80cf6e0 msgr2=0x7f69e815d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.024+0000 7f69ed50f640 1 --2- 172.21.15.46:0/1916791046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e80cf6e0 0x7f69e815d570 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f69e80742c0 tx=0x7f69e001c9a0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.024+0000 7f69ed50f640 1 -- 172.21.15.46:0/1916791046 shutdown_connections 2023-10-30T13:23:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.024+0000 7f69ed50f640 1 --2- 172.21.15.46:0/1916791046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e80cf6e0 0x7f69e815d570 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.024+0000 7f69ed50f640 1 --2- 172.21.15.46:0/1916791046 >> v2:172.21.15.154:3300/0 conn(0x7f69e8159930 0x7f69e80cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.024+0000 7f69ed50f640 1 -- 172.21.15.46:0/1916791046 >> 172.21.15.46:0/1916791046 conn(0x7f69e806c0f0 msgr2=0x7f69e806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.024+0000 7f69ed50f640 1 -- 172.21.15.46:0/1916791046 shutdown_connections 2023-10-30T13:23:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.024+0000 7f69ed50f640 1 -- 172.21.15.46:0/1916791046 wait complete. 2023-10-30T13:23:33.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69ed50f640 1 Processor -- start 2023-10-30T13:23:33.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69ed50f640 1 -- start start 2023-10-30T13:23:33.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69ed50f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e80cf6e0 0x7f69e81f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69ed50f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e8153980 0x7f69e81f1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69ed50f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e81f24e0 0x7f69e814cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69ed50f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e81f4db0 con 0x7f69e81f24e0 2023-10-30T13:23:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69ed50f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e8145c00 con 0x7f69e8153980 2023-10-30T13:23:33.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69ed50f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e80c9670 con 0x7f69e80cf6e0 2023-10-30T13:23:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69e77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e81f24e0 0x7f69e814cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69e77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e81f24e0 0x7f69e814cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39942/0 (socket says 172.21.15.46:39942) 2023-10-30T13:23:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69e77fe640 1 -- 172.21.15.46:0/894994273 learned_addr learned my addr 172.21.15.46:0/894994273 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69dffff640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e8153980 0x7f69e81f1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69e6ffd640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e80cf6e0 0x7f69e81f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69e77fe640 1 -- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e80cf6e0 msgr2=0x7f69e81f57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69e77fe640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e80cf6e0 0x7f69e81f57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69e77fe640 1 -- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e8153980 msgr2=0x7f69e81f1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69e77fe640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e8153980 0x7f69e81f1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69e77fe640 1 -- 172.21.15.46:0/894994273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e0009b30 con 0x7f69e81f24e0 2023-10-30T13:23:33.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69e77fe640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e81f24e0 0x7f69e814cd00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f69e001a5d0 tx=0x7f69e0019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:33.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69df7fe640 1 -- 172.21.15.46:0/894994273 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e00bb070 con 0x7f69e81f24e0 2023-10-30T13:23:33.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69df7fe640 1 -- 172.21.15.46:0/894994273 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e00b6480 con 0x7f69e81f24e0 2023-10-30T13:23:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69ed50f640 1 -- 172.21.15.46:0/894994273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e81f5d00 con 0x7f69e81f24e0 2023-10-30T13:23:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69df7fe640 1 -- 172.21.15.46:0/894994273 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e00b6810 con 0x7f69e81f24e0 2023-10-30T13:23:33.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.028+0000 7f69ed50f640 1 -- 172.21.15.46:0/894994273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e80c9390 con 0x7f69e81f24e0 2023-10-30T13:23:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.032+0000 7f69ed50f640 1 -- 172.21.15.46:0/894994273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e804e170 con 0x7f69e81f24e0 2023-10-30T13:23:33.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.032+0000 7f69df7fe640 1 -- 172.21.15.46:0/894994273 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f69e00b6970 con 0x7f69e81f24e0 2023-10-30T13:23:33.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.032+0000 7f69df7fe640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69c80664a0 0x7f69c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.032+0000 7f69df7fe640 1 -- 172.21.15.46:0/894994273 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69e011e6d0 con 0x7f69e81f24e0 2023-10-30T13:23:33.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.032+0000 7f69e6ffd640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69c80664a0 0x7f69c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.032+0000 7f69e6ffd640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69c80664a0 0x7f69c8068960 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f69d8007fd0 tx=0x7f69d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:33.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.036+0000 7f69df7fe640 1 -- 172.21.15.46:0/894994273 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f69e00ebae0 con 0x7f69e81f24e0 2023-10-30T13:23:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.196+0000 7f69ed50f640 1 -- 172.21.15.46:0/894994273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f69e81f5d00 con 0x7f69e81f24e0 2023-10-30T13:23:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69df7fe640 1 -- 172.21.15.46:0/894994273 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f69e81f5d00 con 0x7f69e81f24e0 2023-10-30T13:23:33.206 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-10-30T13:23:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69ed50f640 1 -- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69c80664a0 msgr2=0x7f69c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69ed50f640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69c80664a0 0x7f69c8068960 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f69d8007fd0 tx=0x7f69d8009040 comp rx=0 tx=0).stop 2023-10-30T13:23:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69ed50f640 1 -- 172.21.15.46:0/894994273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e81f24e0 msgr2=0x7f69e814cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69ed50f640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e81f24e0 0x7f69e814cd00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f69e001a5d0 tx=0x7f69e0019740 comp rx=0 tx=0).stop 2023-10-30T13:23:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69ed50f640 1 -- 172.21.15.46:0/894994273 shutdown_connections 2023-10-30T13:23:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69ed50f640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69c80664a0 0x7f69c8068960 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69ed50f640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e81f24e0 0x7f69e814cd00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69ed50f640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e8153980 0x7f69e81f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69ed50f640 1 --2- 172.21.15.46:0/894994273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e80cf6e0 0x7f69e81f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69ed50f640 1 -- 172.21.15.46:0/894994273 >> 172.21.15.46:0/894994273 conn(0x7f69e806c0f0 msgr2=0x7f69e8159210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69ed50f640 1 -- 172.21.15.46:0/894994273 shutdown_connections 2023-10-30T13:23:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.200+0000 7f69ed50f640 1 -- 172.21.15.46:0/894994273 wait complete. 2023-10-30T13:23:33.227 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b446b640 1 Processor -- start 2023-10-30T13:23:33.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b446b640 1 -- start start 2023-10-30T13:23:33.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b446b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02ac069e90 0x7f02ac15a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b446b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f02ac06cb80 0x7f02ac06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b446b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac06d530 con 0x7f02ac06a270 2023-10-30T13:23:33.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b446b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ac0c8570 con 0x7f02ac069e90 2023-10-30T13:23:33.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b446b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ac0c6c10 con 0x7f02ac06cb80 2023-10-30T13:23:33.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b2207640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02ac069e90 0x7f02ac15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b2207640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02ac069e90 0x7f02ac15a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39954/0 (socket says 172.21.15.46:39954) 2023-10-30T13:23:33.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b2207640 1 -- 172.21.15.46:0/3830727748 learned_addr learned my addr 172.21.15.46:0/3830727748 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:33.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b2a08640 1 --2- 172.21.15.46:0/3830727748 >> v2:172.21.15.154:3300/0 conn(0x7f02ac06cb80 0x7f02ac06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b2207640 1 -- 172.21.15.46:0/3830727748 >> v2:172.21.15.154:3300/0 conn(0x7f02ac06cb80 msgr2=0x7f02ac06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:33.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b2207640 1 --2- 172.21.15.46:0/3830727748 >> v2:172.21.15.154:3300/0 conn(0x7f02ac06cb80 0x7f02ac06cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b2207640 1 -- 172.21.15.46:0/3830727748 >> v1:172.21.15.46:6789/0 conn(0x7f02ac06a270 legacy=0x7f02ac0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b1205640 1 -- 172.21.15.46:0/3830727748 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410667274 0 0) 0x7f02ac06d530 con 0x7f02ac06a270 2023-10-30T13:23:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.364+0000 7f02b2207640 1 -- 172.21.15.46:0/3830727748 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac0c6ef0 con 0x7f02ac069e90 2023-10-30T13:23:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.368+0000 7f02b2207640 1 --2- 172.21.15.46:0/3830727748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac069e90 0x7f02ac15a410 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0ca5c0 tx=0x7f02a801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b0942de8e013f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.368+0000 7f02b1205640 1 -- 172.21.15.46:0/3830727748 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a80ab5f0 con 0x7f02ac069e90 2023-10-30T13:23:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.368+0000 7f02b1205640 1 -- 172.21.15.46:0/3830727748 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a80ab750 con 0x7f02ac069e90 2023-10-30T13:23:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.368+0000 7f02b1205640 1 -- 172.21.15.46:0/3830727748 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a80abae0 con 0x7f02ac069e90 2023-10-30T13:23:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.368+0000 7f02b446b640 1 -- 172.21.15.46:0/3830727748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac069e90 msgr2=0x7f02ac15a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.368+0000 7f02b446b640 1 --2- 172.21.15.46:0/3830727748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac069e90 0x7f02ac15a410 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0ca5c0 tx=0x7f02a801c500 comp rx=0 tx=0).stop 2023-10-30T13:23:33.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.368+0000 7f02b446b640 1 -- 172.21.15.46:0/3830727748 shutdown_connections 2023-10-30T13:23:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.368+0000 7f02b446b640 1 --2- 172.21.15.46:0/3830727748 >> v2:172.21.15.154:3300/0 conn(0x7f02ac06cb80 0x7f02ac06cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.368+0000 7f02b446b640 1 --2- 172.21.15.46:0/3830727748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac069e90 0x7f02ac15a410 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.368+0000 7f02b446b640 1 -- 172.21.15.46:0/3830727748 >> 172.21.15.46:0/3830727748 conn(0x7f02ac06dfd0 msgr2=0x7f02ac06e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:33.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.368+0000 7f02b446b640 1 -- 172.21.15.46:0/3830727748 shutdown_connections 2023-10-30T13:23:33.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.368+0000 7f02b446b640 1 -- 172.21.15.46:0/3830727748 wait complete. 2023-10-30T13:23:33.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b446b640 1 Processor -- start 2023-10-30T13:23:33.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b446b640 1 -- start start 2023-10-30T13:23:33.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b446b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac069e90 0x7f02ac1484b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b446b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac06a270 0x7f02ac1f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b446b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac06cb80 0x7f02ac1f5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b446b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac1558c0 con 0x7f02ac069e90 2023-10-30T13:23:33.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b446b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac1f56a0 con 0x7f02ac06cb80 2023-10-30T13:23:33.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b446b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac14e530 con 0x7f02ac06a270 2023-10-30T13:23:33.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b2207640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac069e90 0x7f02ac1484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b2207640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac069e90 0x7f02ac1484b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39970/0 (socket says 172.21.15.46:39970) 2023-10-30T13:23:33.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b2207640 1 -- 172.21.15.46:0/1482879428 learned_addr learned my addr 172.21.15.46:0/1482879428 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:33.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b1a06640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac06a270 0x7f02ac1f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b2a08640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac06cb80 0x7f02ac1f5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b2207640 1 -- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac06a270 msgr2=0x7f02ac1f4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b2207640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac06a270 0x7f02ac1f4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b2207640 1 -- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac06cb80 msgr2=0x7f02ac1f5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b2207640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac06cb80 0x7f02ac1f5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b2207640 1 -- 172.21.15.46:0/1482879428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a80094d0 con 0x7f02ac069e90 2023-10-30T13:23:33.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b2207640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac069e90 0x7f02ac1484b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f02a8019110 tx=0x7f02a801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:33.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02a2ffd640 1 -- 172.21.15.46:0/1482879428 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a80b4800 con 0x7f02ac069e90 2023-10-30T13:23:33.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02a2ffd640 1 -- 172.21.15.46:0/1482879428 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a80b4960 con 0x7f02ac069e90 2023-10-30T13:23:33.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02a2ffd640 1 -- 172.21.15.46:0/1482879428 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a80b4cf0 con 0x7f02ac069e90 2023-10-30T13:23:33.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b446b640 1 -- 172.21.15.46:0/1482879428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac1f5890 con 0x7f02ac069e90 2023-10-30T13:23:33.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b446b640 1 -- 172.21.15.46:0/1482879428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ac156ad0 con 0x7f02ac069e90 2023-10-30T13:23:33.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.372+0000 7f02b446b640 1 -- 172.21.15.46:0/1482879428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ac06a650 con 0x7f02ac069e90 2023-10-30T13:23:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.376+0000 7f02a2ffd640 1 -- 172.21.15.46:0/1482879428 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02a80b3380 con 0x7f02ac069e90 2023-10-30T13:23:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.376+0000 7f02a2ffd640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02900664a0 0x7f0290068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.376+0000 7f02a2ffd640 1 -- 172.21.15.46:0/1482879428 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02a80a8050 con 0x7f02ac069e90 2023-10-30T13:23:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.376+0000 7f02b1a06640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02900664a0 0x7f0290068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.376+0000 7f02b1a06640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02900664a0 0x7f0290068960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f02ac149f70 tx=0x7f029c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:33.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.380+0000 7f02a2ffd640 1 -- 172.21.15.46:0/1482879428 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f02ac06a650 con 0x7f02ac069e90 2023-10-30T13:23:33.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.568+0000 7f02b446b640 1 -- 172.21.15.46:0/1482879428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02ac06a650 con 0x7f02ac069e90 2023-10-30T13:23:33.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.568+0000 7f02a2ffd640 1 -- 172.21.15.46:0/1482879428 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+6518 (secure 0 0 0) 0x7f02ac06a650 con 0x7f02ac069e90 2023-10-30T13:23:33.576 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:33.576 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:30.713454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:33.576 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 11 2023-10-30T13:23:33.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.568+0000 7f02b446b640 1 -- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02900664a0 msgr2=0x7f0290068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.568+0000 7f02b446b640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02900664a0 0x7f0290068960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f02ac149f70 tx=0x7f029c009040 comp rx=0 tx=0).stop 2023-10-30T13:23:33.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.568+0000 7f02b446b640 1 -- 172.21.15.46:0/1482879428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac069e90 msgr2=0x7f02ac1484b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.568+0000 7f02b446b640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac069e90 0x7f02ac1484b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f02a8019110 tx=0x7f02a801ad40 comp rx=0 tx=0).stop 2023-10-30T13:23:33.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.568+0000 7f02b446b640 1 -- 172.21.15.46:0/1482879428 shutdown_connections 2023-10-30T13:23:33.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.568+0000 7f02b446b640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02900664a0 0x7f0290068960 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.568+0000 7f02b446b640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac06cb80 0x7f02ac1f5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.568+0000 7f02b446b640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac06a270 0x7f02ac1f4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.568+0000 7f02b446b640 1 --2- 172.21.15.46:0/1482879428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac069e90 0x7f02ac1484b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.568+0000 7f02b446b640 1 -- 172.21.15.46:0/1482879428 >> 172.21.15.46:0/1482879428 conn(0x7f02ac06dfd0 msgr2=0x7f02ac0c9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.572+0000 7f02b446b640 1 -- 172.21.15.46:0/1482879428 shutdown_connections 2023-10-30T13:23:33.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.572+0000 7f02b446b640 1 -- 172.21.15.46:0/1482879428 wait complete. 2023-10-30T13:23:33.597 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:33.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e969e640 1 Processor -- start 2023-10-30T13:23:33.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e969e640 1 -- start start 2023-10-30T13:23:33.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e969e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f97e415d6f0 0x7f97e415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e969e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97e40cf160 0x7f97e40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e969e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e40cb800 con 0x7f97e41538a0 2023-10-30T13:23:33.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e969e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e40cb330 con 0x7f97e40cf160 2023-10-30T13:23:33.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e969e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e40cad70 con 0x7f97e415d6f0 2023-10-30T13:23:33.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97e40cf160 0x7f97e40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97e40cf160 0x7f97e40cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39978/0 (socket says 172.21.15.46:39978) 2023-10-30T13:23:33.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e37fe640 1 -- 172.21.15.46:0/2478135964 learned_addr learned my addr 172.21.15.46:0/2478135964 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:33.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97dbfff640 1 --2- 172.21.15.46:0/2478135964 >> v2:172.21.15.154:3300/0 conn(0x7f97e415d6f0 0x7f97e415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e37fe640 1 -- 172.21.15.46:0/2478135964 >> v2:172.21.15.154:3300/0 conn(0x7f97e415d6f0 msgr2=0x7f97e415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:33.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e37fe640 1 --2- 172.21.15.46:0/2478135964 >> v2:172.21.15.154:3300/0 conn(0x7f97e415d6f0 0x7f97e415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e37fe640 1 -- 172.21.15.46:0/2478135964 >> v1:172.21.15.154:6789/0 conn(0x7f97e41538a0 legacy=0x7f97e40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97e37fe640 1 -- 172.21.15.46:0/2478135964 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e40cb050 con 0x7f97e40cf160 2023-10-30T13:23:33.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.736+0000 7f97dbfff640 1 --2- 172.21.15.46:0/2478135964 >> v2:172.21.15.154:3300/0 conn(0x7f97e415d6f0 0x7f97e415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:33.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.740+0000 7f97e37fe640 1 --2- 172.21.15.46:0/2478135964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e40cf160 0x7f97e40cf590 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f97e40b30a0 tx=0x7f97dc01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93fbe4f6ad320d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:33.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.740+0000 7f97e27fc640 1 -- 172.21.15.46:0/2478135964 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc0af400 con 0x7f97e40cf160 2023-10-30T13:23:33.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.740+0000 7f97e27fc640 1 -- 172.21.15.46:0/2478135964 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc0ab050 con 0x7f97e40cf160 2023-10-30T13:23:33.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.740+0000 7f97e27fc640 1 -- 172.21.15.46:0/2478135964 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc0af780 con 0x7f97e40cf160 2023-10-30T13:23:33.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.740+0000 7f97e969e640 1 -- 172.21.15.46:0/2478135964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e40cf160 msgr2=0x7f97e40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.740+0000 7f97e969e640 1 --2- 172.21.15.46:0/2478135964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e40cf160 0x7f97e40cf590 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f97e40b30a0 tx=0x7f97dc01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.740+0000 7f97e969e640 1 -- 172.21.15.46:0/2478135964 shutdown_connections 2023-10-30T13:23:33.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.740+0000 7f97e969e640 1 --2- 172.21.15.46:0/2478135964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e40cf160 0x7f97e40cf590 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.740+0000 7f97e969e640 1 --2- 172.21.15.46:0/2478135964 >> v2:172.21.15.154:3300/0 conn(0x7f97e415d6f0 0x7f97e415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.740+0000 7f97e969e640 1 -- 172.21.15.46:0/2478135964 >> 172.21.15.46:0/2478135964 conn(0x7f97e406bfe0 msgr2=0x7f97e406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:33.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.740+0000 7f97e969e640 1 -- 172.21.15.46:0/2478135964 shutdown_connections 2023-10-30T13:23:33.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.740+0000 7f97e969e640 1 -- 172.21.15.46:0/2478135964 wait complete. 2023-10-30T13:23:33.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e969e640 1 Processor -- start 2023-10-30T13:23:33.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e969e640 1 -- start start 2023-10-30T13:23:33.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e969e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e40cf160 0x7f97e41e2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e969e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e41538a0 0x7f97e4157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e969e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e415d6f0 0x7f97e40ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e969e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e41e55e0 con 0x7f97e40cf160 2023-10-30T13:23:33.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e969e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e41e5260 con 0x7f97e41538a0 2023-10-30T13:23:33.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e969e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e414e540 con 0x7f97e415d6f0 2023-10-30T13:23:33.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e40cf160 0x7f97e41e2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e40cf160 0x7f97e41e2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39988/0 (socket says 172.21.15.46:39988) 2023-10-30T13:23:33.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e2ffd640 1 -- 172.21.15.46:0/2597726618 learned_addr learned my addr 172.21.15.46:0/2597726618 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:33.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97dbfff640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e41538a0 0x7f97e4157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e37fe640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e415d6f0 0x7f97e40ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e2ffd640 1 -- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e415d6f0 msgr2=0x7f97e40ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e2ffd640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e415d6f0 0x7f97e40ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e2ffd640 1 -- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e41538a0 msgr2=0x7f97e4157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e2ffd640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e41538a0 0x7f97e4157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e2ffd640 1 -- 172.21.15.46:0/2597726618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc009b30 con 0x7f97e40cf160 2023-10-30T13:23:33.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97dbfff640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e41538a0 0x7f97e4157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:33.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e2ffd640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e40cf160 0x7f97e41e2c50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f97d400c530 tx=0x7f97d400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:33.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97db7fe640 1 -- 172.21.15.46:0/2597726618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d4005220 con 0x7f97e40cf160 2023-10-30T13:23:33.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97db7fe640 1 -- 172.21.15.46:0/2597726618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d4005380 con 0x7f97e40cf160 2023-10-30T13:23:33.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e969e640 1 -- 172.21.15.46:0/2597726618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e41e41c0 con 0x7f97e40cf160 2023-10-30T13:23:33.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97db7fe640 1 -- 172.21.15.46:0/2597726618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d4007270 con 0x7f97e40cf160 2023-10-30T13:23:33.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e969e640 1 -- 172.21.15.46:0/2597726618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e41e4550 con 0x7f97e40cf160 2023-10-30T13:23:33.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.744+0000 7f97e969e640 1 -- 172.21.15.46:0/2597726618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e40b2a50 con 0x7f97e40cf160 2023-10-30T13:23:33.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.748+0000 7f97db7fe640 1 -- 172.21.15.46:0/2597726618 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f97d40073d0 con 0x7f97e40cf160 2023-10-30T13:23:33.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.748+0000 7f97db7fe640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97c40664a0 0x7f97c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:33.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.748+0000 7f97db7fe640 1 -- 172.21.15.46:0/2597726618 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97d40837e0 con 0x7f97e40cf160 2023-10-30T13:23:33.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.748+0000 7f97dbfff640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97c40664a0 0x7f97c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:33.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.748+0000 7f97dbfff640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97c40664a0 0x7f97c4068960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f97e40c97b0 tx=0x7f97cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.752+0000 7f97db7fe640 1 -- 172.21.15.46:0/2597726618 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97d4050bc0 con 0x7f97e40cf160 2023-10-30T13:23:33.916 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T13:23:33.908+0000 7fce56864640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:23:33.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97e969e640 1 -- 172.21.15.46:0/2597726618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97e404e170 con 0x7f97e40cf160 2023-10-30T13:23:33.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97db7fe640 1 -- 172.21.15.46:0/2597726618 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f97d40541e0 con 0x7f97e40cf160 2023-10-30T13:23:33.926 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:33.926 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:20.665723+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:33.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97e969e640 1 -- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97c40664a0 msgr2=0x7f97c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97e969e640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97c40664a0 0x7f97c4068960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f97e40c97b0 tx=0x7f97cc008040 comp rx=0 tx=0).stop 2023-10-30T13:23:33.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97e969e640 1 -- 172.21.15.46:0/2597726618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e40cf160 msgr2=0x7f97e41e2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:33.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97e969e640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e40cf160 0x7f97e41e2c50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f97d400c530 tx=0x7f97d400c9f0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97e969e640 1 -- 172.21.15.46:0/2597726618 shutdown_connections 2023-10-30T13:23:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97e969e640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97c40664a0 0x7f97c4068960 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97e969e640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e415d6f0 0x7f97e40ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97e969e640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e41538a0 0x7f97e4157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97e969e640 1 --2- 172.21.15.46:0/2597726618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97e40cf160 0x7f97e41e2c50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97e969e640 1 -- 172.21.15.46:0/2597726618 >> 172.21.15.46:0/2597726618 conn(0x7f97e406bfe0 msgr2=0x7f97e4073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.916+0000 7f97e969e640 1 -- 172.21.15.46:0/2597726618 shutdown_connections 2023-10-30T13:23:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:33.920+0000 7f97e969e640 1 -- 172.21.15.46:0/2597726618 wait complete. 2023-10-30T13:23:33.945 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:23:06.788949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:23:17.648827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:23:18.326967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:23:33.945 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:23:34.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f8412846640 1 Processor -- start 2023-10-30T13:23:34.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f8412846640 1 -- start start 2023-10-30T13:23:34.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f8412846640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f840c159930 0x7f840c0cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f8412846640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f840c0cf6e0 0x7f840c15d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f8412846640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840c15a920 con 0x7f840c153980 2023-10-30T13:23:34.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f8412846640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f840c0cb930 con 0x7f840c0cf6e0 2023-10-30T13:23:34.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f8412846640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840c159dc0 con 0x7f840c159930 2023-10-30T13:23:34.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f840bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f840c153980 0x7f840c0caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48606/0 (socket says 172.21.15.46:48606) 2023-10-30T13:23:34.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f840bfff640 1 -- 172.21.15.46:0/4293363496 learned_addr learned my addr 172.21.15.46:0/4293363496 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:34.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f840b7fe640 1 --2- 172.21.15.46:0/4293363496 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f840c159930 0x7f840c0cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f8410de3640 1 --2- 172.21.15.46:0/4293363496 >> v2:172.21.15.154:3300/0 conn(0x7f840c0cf6e0 0x7f840c15d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f840affd640 1 -- 172.21.15.46:0/4293363496 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518178894 0 0) 0x7f840c15a920 con 0x7f840c153980 2023-10-30T13:23:34.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.084+0000 7f840affd640 1 -- 172.21.15.46:0/4293363496 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f4003530 con 0x7f840c153980 2023-10-30T13:23:34.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.088+0000 7f8410de3640 1 -- 172.21.15.46:0/4293363496 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f840c159930 msgr2=0x7f840c0cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:34.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.088+0000 7f8410de3640 1 --2- 172.21.15.46:0/4293363496 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f840c159930 0x7f840c0cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.088+0000 7f8410de3640 1 -- 172.21.15.46:0/4293363496 >> v1:172.21.15.46:6789/0 conn(0x7f840c153980 legacy=0x7f840c0caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.088+0000 7f8410de3640 1 -- 172.21.15.46:0/4293363496 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c15a080 con 0x7f840c0cf6e0 2023-10-30T13:23:34.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.088+0000 7f840b7fe640 1 --2- 172.21.15.46:0/4293363496 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f840c159930 0x7f840c0cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:34.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.088+0000 7f8410de3640 1 --2- 172.21.15.46:0/4293363496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f840c0cf6e0 0x7f840c15d570 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8404009d90 tx=0x7f840401c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ea3c4a3a8f82b15 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:34.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.088+0000 7f840affd640 1 -- 172.21.15.46:0/4293363496 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040a58f0 con 0x7f840c0cf6e0 2023-10-30T13:23:34.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.088+0000 7f840affd640 1 -- 172.21.15.46:0/4293363496 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84040a5a50 con 0x7f840c0cf6e0 2023-10-30T13:23:34.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.088+0000 7f840affd640 1 -- 172.21.15.46:0/4293363496 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040a5dc0 con 0x7f840c0cf6e0 2023-10-30T13:23:34.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 -- 172.21.15.46:0/4293363496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f840c0cf6e0 msgr2=0x7f840c15d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 --2- 172.21.15.46:0/4293363496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f840c0cf6e0 0x7f840c15d570 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8404009d90 tx=0x7f840401c920 comp rx=0 tx=0).stop 2023-10-30T13:23:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 -- 172.21.15.46:0/4293363496 shutdown_connections 2023-10-30T13:23:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 --2- 172.21.15.46:0/4293363496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f840c0cf6e0 0x7f840c15d570 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 --2- 172.21.15.46:0/4293363496 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f840c159930 0x7f840c0cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 -- 172.21.15.46:0/4293363496 >> 172.21.15.46:0/4293363496 conn(0x7f840c06c0f0 msgr2=0x7f840c06c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 -- 172.21.15.46:0/4293363496 shutdown_connections 2023-10-30T13:23:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 -- 172.21.15.46:0/4293363496 wait complete. 2023-10-30T13:23:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 Processor -- start 2023-10-30T13:23:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 -- start start 2023-10-30T13:23:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f840c0cf6e0 0x7f840c0c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f840c153980 0x7f840c1f5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f840c1f1bc0 0x7f840c152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c0b4eb0 con 0x7f840c1f1bc0 2023-10-30T13:23:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c0c6a10 con 0x7f840c0cf6e0 2023-10-30T13:23:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8412846640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840c1f6010 con 0x7f840c153980 2023-10-30T13:23:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8410de3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f840c1f1bc0 0x7f840c152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8410de3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f840c1f1bc0 0x7f840c152aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40004/0 (socket says 172.21.15.46:40004) 2023-10-30T13:23:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8410de3640 1 -- 172.21.15.46:0/3592853891 learned_addr learned my addr 172.21.15.46:0/3592853891 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f840bfff640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f840c0cf6e0 0x7f840c0c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f840b7fe640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f840c153980 0x7f840c1f5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8410de3640 1 -- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f840c153980 msgr2=0x7f840c1f5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8410de3640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f840c153980 0x7f840c1f5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8410de3640 1 -- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f840c0cf6e0 msgr2=0x7f840c0c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8410de3640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f840c0cf6e0 0x7f840c0c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.092+0000 7f8410de3640 1 -- 172.21.15.46:0/3592853891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8400009420 con 0x7f840c1f1bc0 2023-10-30T13:23:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.096+0000 7f8410de3640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f840c1f1bc0 0x7f840c152aa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f840401afd0 tx=0x7f84040196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.096+0000 7f8408ff9640 1 -- 172.21.15.46:0/3592853891 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040039a0 con 0x7f840c1f1bc0 2023-10-30T13:23:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.096+0000 7f8408ff9640 1 -- 172.21.15.46:0/3592853891 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8404003b00 con 0x7f840c1f1bc0 2023-10-30T13:23:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.096+0000 7f8408ff9640 1 -- 172.21.15.46:0/3592853891 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404004040 con 0x7f840c1f1bc0 2023-10-30T13:23:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.096+0000 7f8412846640 1 -- 172.21.15.46:0/3592853891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8404009a70 con 0x7f840c1f1bc0 2023-10-30T13:23:34.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.096+0000 7f8412846640 1 -- 172.21.15.46:0/3592853891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f840c0b4bd0 con 0x7f840c1f1bc0 2023-10-30T13:23:34.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.096+0000 7f8412846640 1 -- 172.21.15.46:0/3592853891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f840c1f57f0 con 0x7f840c1f1bc0 2023-10-30T13:23:34.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.096+0000 7f8408ff9640 1 -- 172.21.15.46:0/3592853891 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f84040041a0 con 0x7f840c1f1bc0 2023-10-30T13:23:34.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.096+0000 7f8408ff9640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83f80664d0 0x7f83f8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.096+0000 7f8408ff9640 1 -- 172.21.15.46:0/3592853891 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f840411eec0 con 0x7f840c1f1bc0 2023-10-30T13:23:34.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.096+0000 7f840bfff640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83f80664d0 0x7f83f8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.100+0000 7f840bfff640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83f80664d0 0x7f83f8068990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f83fc007fd0 tx=0x7f83fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:34.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.100+0000 7f8408ff9640 1 -- 172.21.15.46:0/3592853891 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f840c1f57f0 con 0x7f840c1f1bc0 2023-10-30T13:23:34.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.256+0000 7f8412846640 1 -- 172.21.15.46:0/3592853891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f840c1f57f0 con 0x7f840c1f1bc0 2023-10-30T13:23:34.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.256+0000 7f8408ff9640 1 -- 172.21.15.46:0/3592853891 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f84040b2020 con 0x7f840c1f1bc0 2023-10-30T13:23:34.265 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-10-30T13:23:34.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.260+0000 7f8412846640 1 -- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83f80664d0 msgr2=0x7f83f8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.260+0000 7f8412846640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83f80664d0 0x7f83f8068990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f83fc007fd0 tx=0x7f83fc009040 comp rx=0 tx=0).stop 2023-10-30T13:23:34.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.260+0000 7f8412846640 1 -- 172.21.15.46:0/3592853891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f840c1f1bc0 msgr2=0x7f840c152aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.260+0000 7f8412846640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f840c1f1bc0 0x7f840c152aa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f840401afd0 tx=0x7f84040196c0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.260+0000 7f8412846640 1 -- 172.21.15.46:0/3592853891 shutdown_connections 2023-10-30T13:23:34.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.260+0000 7f8412846640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83f80664d0 0x7f83f8068990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.260+0000 7f8412846640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f840c1f1bc0 0x7f840c152aa0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.260+0000 7f8412846640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f840c153980 0x7f840c1f5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.260+0000 7f8412846640 1 --2- 172.21.15.46:0/3592853891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f840c0cf6e0 0x7f840c0c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.260+0000 7f8412846640 1 -- 172.21.15.46:0/3592853891 >> 172.21.15.46:0/3592853891 conn(0x7f840c06c0f0 msgr2=0x7f840c158470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:34.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.260+0000 7f8412846640 1 -- 172.21.15.46:0/3592853891 shutdown_connections 2023-10-30T13:23:34.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.260+0000 7f8412846640 1 -- 172.21.15.46:0/3592853891 wait complete. 2023-10-30T13:23:34.287 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:23:34.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbda972640 1 Processor -- start 2023-10-30T13:23:34.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbda972640 1 -- start start 2023-10-30T13:23:34.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbda972640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4153c90 0x7ffbd40be770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbda972640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd40be140 con 0x7ffbd406b540 2023-10-30T13:23:34.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbda972640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd4070a40 con 0x7ffbd415cd10 2023-10-30T13:23:34.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbda972640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd4070850 con 0x7ffbd4153c90 2023-10-30T13:23:34.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffbd406b540 0x7ffbd406b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48610/0 (socket says 172.21.15.46:48610) 2023-10-30T13:23:34.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd37fe640 1 -- 172.21.15.46:0/2919044312 learned_addr learned my addr 172.21.15.46:0/2919044312 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:34.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd3fff640 1 --2- 172.21.15.46:0/2919044312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4153c90 0x7ffbd40be770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761858569 0 0) 0x7ffbd40be140 con 0x7ffbd406b540 2023-10-30T13:23:34.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbbc003530 con 0x7ffbd406b540 2023-10-30T13:23:34.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2944236248 0 0) 0x7ffbbc003530 con 0x7ffbd406b540 2023-10-30T13:23:34.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4153c90 msgr2=0x7ffbd40be770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 --2- 172.21.15.46:0/2919044312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4153c90 0x7ffbd40be770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 >> v1:172.21.15.154:6789/0 conn(0x7ffbd415cd10 legacy=0x7ffbd415d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd406e320 con 0x7ffbd406b540 2023-10-30T13:23:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7ffbc80016a0 con 0x7ffbd406b540 2023-10-30T13:23:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 >> v1:172.21.15.46:6789/0 conn(0x7ffbd406b540 legacy=0x7ffbd406b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 --2- 172.21.15.46:0/2919044312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbbc0039b0 0x7ffbbc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 --2- 172.21.15.46:0/2919044312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbbc0047d0 0x7ffbbc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 --2- 172.21.15.46:0/2919044312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4153c90 0x7ffbbc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc00bb20 con 0x7ffbbc0047d0 2023-10-30T13:23:34.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc00bc90 con 0x7ffbbc0039b0 2023-10-30T13:23:34.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc00be50 con 0x7ffbd4153c90 2023-10-30T13:23:34.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd37fe640 1 --2- 172.21.15.46:0/2919044312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbbc0047d0 0x7ffbbc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd3fff640 1 --2- 172.21.15.46:0/2919044312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbbc0039b0 0x7ffbbc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd8f0f640 1 --2- 172.21.15.46:0/2919044312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4153c90 0x7ffbbc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd37fe640 1 -- 172.21.15.46:0/2919044312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4153c90 msgr2=0x7ffbbc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd37fe640 1 --2- 172.21.15.46:0/2919044312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4153c90 0x7ffbbc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd37fe640 1 -- 172.21.15.46:0/2919044312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbbc0039b0 msgr2=0x7ffbbc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd37fe640 1 --2- 172.21.15.46:0/2919044312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbbc0039b0 0x7ffbbc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.428+0000 7ffbd37fe640 1 -- 172.21.15.46:0/2919044312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc003530 con 0x7ffbbc0047d0 2023-10-30T13:23:34.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.432+0000 7ffbd37fe640 1 --2- 172.21.15.46:0/2919044312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbbc0047d0 0x7ffbbc004c50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800c190 tx=0x7ffbc801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7578fba4ddfcb42 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:34.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.432+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc80a5b00 con 0x7ffbbc0047d0 2023-10-30T13:23:34.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.432+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc80a5c60 con 0x7ffbbc0047d0 2023-10-30T13:23:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.432+0000 7ffbd2ffd640 1 -- 172.21.15.46:0/2919044312 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc80058e0 con 0x7ffbbc0047d0 2023-10-30T13:23:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.432+0000 7ffbda972640 1 -- 172.21.15.46:0/2919044312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbbc0047d0 msgr2=0x7ffbbc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.432+0000 7ffbda972640 1 --2- 172.21.15.46:0/2919044312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbbc0047d0 0x7ffbbc004c50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800c190 tx=0x7ffbc801cc00 comp rx=0 tx=0).stop 2023-10-30T13:23:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.432+0000 7ffbd8f0f640 1 -- 172.21.15.46:0/2919044312 reap_dead start 2023-10-30T13:23:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.432+0000 7ffbda972640 1 -- 172.21.15.46:0/2919044312 shutdown_connections 2023-10-30T13:23:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.432+0000 7ffbda972640 1 -- 172.21.15.46:0/2919044312 >> 172.21.15.46:0/2919044312 conn(0x7ffbd40c7fe0 msgr2=0x7ffbd40c83f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.432+0000 7ffbda972640 1 -- 172.21.15.46:0/2919044312 shutdown_connections 2023-10-30T13:23:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.432+0000 7ffbda972640 1 -- 172.21.15.46:0/2919044312 wait complete. 2023-10-30T13:23:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbda972640 1 Processor -- start 2023-10-30T13:23:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbda972640 1 -- start start 2023-10-30T13:23:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbda972640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbbc0047d0 0x7ffbd40c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbda972640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4151b40 0x7ffbd414c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbda972640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbd4150ab0 0x7ffbd4150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbda972640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd40c3860 con 0x7ffbbc0047d0 2023-10-30T13:23:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbda972640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd4151470 con 0x7ffbd4150ab0 2023-10-30T13:23:34.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbda972640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd40ca7a0 con 0x7ffbd4151b40 2023-10-30T13:23:34.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbbc0047d0 0x7ffbd40c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbbc0047d0 0x7ffbd40c2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40018/0 (socket says 172.21.15.46:40018) 2023-10-30T13:23:34.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd3fff640 1 -- 172.21.15.46:0/3299091656 learned_addr learned my addr 172.21.15.46:0/3299091656 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:34.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd37fe640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4151b40 0x7ffbd414c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd8f0f640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbd4150ab0 0x7ffbd4150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd3fff640 1 -- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4151b40 msgr2=0x7ffbd414c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd3fff640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4151b40 0x7ffbd414c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd3fff640 1 -- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbd4150ab0 msgr2=0x7ffbd4150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd3fff640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbd4150ab0 0x7ffbd4150f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd3fff640 1 -- 172.21.15.46:0/3299091656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd40c3dd0 con 0x7ffbbc0047d0 2023-10-30T13:23:34.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd37fe640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4151b40 0x7ffbd414c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:34.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd8f0f640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbd4150ab0 0x7ffbd4150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:34.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd3fff640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbbc0047d0 0x7ffbd40c2860 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400b7f0 tx=0x7ffbc400bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:34.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd0ff9640 1 -- 172.21.15.46:0/3299091656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4004170 con 0x7ffbbc0047d0 2023-10-30T13:23:34.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd0ff9640 1 -- 172.21.15.46:0/3299091656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc4010830 con 0x7ffbbc0047d0 2023-10-30T13:23:34.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbd0ff9640 1 -- 172.21.15.46:0/3299091656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4010be0 con 0x7ffbbc0047d0 2023-10-30T13:23:34.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbda972640 1 -- 172.21.15.46:0/3299091656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd40caa40 con 0x7ffbbc0047d0 2023-10-30T13:23:34.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.436+0000 7ffbda972640 1 -- 172.21.15.46:0/3299091656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd40cb000 con 0x7ffbbc0047d0 2023-10-30T13:23:34.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.440+0000 7ffbda972640 1 -- 172.21.15.46:0/3299091656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd404e170 con 0x7ffbbc0047d0 2023-10-30T13:23:34.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.440+0000 7ffbd0ff9640 1 -- 172.21.15.46:0/3299091656 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffbc4010d40 con 0x7ffbbc0047d0 2023-10-30T13:23:34.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.440+0000 7ffbd0ff9640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffbc00664d0 0x7ffbc0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.440+0000 7ffbd0ff9640 1 -- 172.21.15.46:0/3299091656 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbc4084400 con 0x7ffbbc0047d0 2023-10-30T13:23:34.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.440+0000 7ffbd37fe640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffbc00664d0 0x7ffbc0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.440+0000 7ffbd37fe640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffbc00664d0 0x7ffbc0068990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80ab000 tx=0x7ffbc80a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:34.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.444+0000 7ffbd0ff9640 1 -- 172.21.15.46:0/3299091656 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffbc40518f0 con 0x7ffbbc0047d0 2023-10-30T13:23:34.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 -- 172.21.15.46:0/3299091656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffbd41506d0 con 0x7ffbbc0047d0 2023-10-30T13:23:34.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbd0ff9640 1 -- 172.21.15.46:0/3299091656 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffbc4054f10 con 0x7ffbbc0047d0 2023-10-30T13:23:34.611 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-10-30T13:23:34.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 -- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffbc00664d0 msgr2=0x7ffbc0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffbc00664d0 0x7ffbc0068990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80ab000 tx=0x7ffbc80a7070 comp rx=0 tx=0).stop 2023-10-30T13:23:34.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 -- 172.21.15.46:0/3299091656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbbc0047d0 msgr2=0x7ffbd40c2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbbc0047d0 0x7ffbd40c2860 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400b7f0 tx=0x7ffbc400bcb0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 -- 172.21.15.46:0/3299091656 shutdown_connections 2023-10-30T13:23:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffbc00664d0 0x7ffbc0068990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbd4150ab0 0x7ffbd4150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffbd4151b40 0x7ffbd414c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 --2- 172.21.15.46:0/3299091656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbbc0047d0 0x7ffbd40c2860 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 -- 172.21.15.46:0/3299091656 >> 172.21.15.46:0/3299091656 conn(0x7ffbd40c7fe0 msgr2=0x7ffbd415b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:34.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 -- 172.21.15.46:0/3299091656 shutdown_connections 2023-10-30T13:23:34.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.604+0000 7ffbda972640 1 -- 172.21.15.46:0/3299091656 wait complete. 2023-10-30T13:23:34.632 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:23:34.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.772+0000 7fc82171d640 1 Processor -- start 2023-10-30T13:23:34.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.772+0000 7fc82171d640 1 -- start start 2023-10-30T13:23:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc82171d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c153b80 0x7fc81c065a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc82171d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c0c63c0 con 0x7fc81c0c6600 2023-10-30T13:23:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc82171d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c0cb0e0 con 0x7fc81c0cbd80 2023-10-30T13:23:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc82171d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c0c8820 con 0x7fc81c153b80 2023-10-30T13:23:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc813fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc81c0c6600 0x7fc81c0cb670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48620/0 (socket says 172.21.15.46:48620) 2023-10-30T13:23:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc813fff640 1 -- 172.21.15.46:0/991643123 learned_addr learned my addr 172.21.15.46:0/991643123 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81affd640 1 --2- 172.21.15.46:0/991643123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c153b80 0x7fc81c065a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222645230 0 0) 0x7fc81c0c63c0 con 0x7fc81c0c6600 2023-10-30T13:23:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc808003530 con 0x7fc81c0c6600 2023-10-30T13:23:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724055782 0 0) 0x7fc81c0cb0e0 con 0x7fc81c0cbd80 2023-10-30T13:23:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc81c0c63c0 con 0x7fc81c0cbd80 2023-10-30T13:23:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1305871519 0 0) 0x7fc808003530 con 0x7fc81c0c6600 2023-10-30T13:23:34.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c153b80 msgr2=0x7fc81c065a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:34.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 --2- 172.21.15.46:0/991643123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c153b80 0x7fc81c065a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 >> v1:172.21.15.154:6789/0 conn(0x7fc81c0cbd80 legacy=0x7fc81c163f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c159a10 con 0x7fc81c0c6600 2023-10-30T13:23:34.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fc8040016a0 con 0x7fc81c0c6600 2023-10-30T13:23:34.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 >> v1:172.21.15.46:6789/0 conn(0x7fc81c0c6600 legacy=0x7fc81c0cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 --2- 172.21.15.46:0/991643123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c153b80 0x7fc808003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 --2- 172.21.15.46:0/991643123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8080040f0 0x7fc8080047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 --2- 172.21.15.46:0/991643123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc808004d10 0x7fc808005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80800baa0 con 0x7fc8080040f0 2023-10-30T13:23:34.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80800bc10 con 0x7fc808004d10 2023-10-30T13:23:34.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc80800bdd0 con 0x7fc81c153b80 2023-10-30T13:23:34.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc813fff640 1 --2- 172.21.15.46:0/991643123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8080040f0 0x7fc8080047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81affd640 1 --2- 172.21.15.46:0/991643123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c153b80 0x7fc808003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc81b7fe640 1 --2- 172.21.15.46:0/991643123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc808004d10 0x7fc808005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc813fff640 1 -- 172.21.15.46:0/991643123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c153b80 msgr2=0x7fc808003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:34.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc813fff640 1 --2- 172.21.15.46:0/991643123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c153b80 0x7fc808003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc813fff640 1 -- 172.21.15.46:0/991643123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc808004d10 msgr2=0x7fc808005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:34.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc813fff640 1 --2- 172.21.15.46:0/991643123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc808004d10 0x7fc808005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.776+0000 7fc813fff640 1 -- 172.21.15.46:0/991643123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc808003530 con 0x7fc8080040f0 2023-10-30T13:23:34.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.780+0000 7fc813fff640 1 --2- 172.21.15.46:0/991643123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8080040f0 0x7fc8080047d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0c6ab0 tx=0x7fc80401ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7607a601082a409a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:34.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.780+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc804005ae0 con 0x7fc8080040f0 2023-10-30T13:23:34.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.780+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc804005c40 con 0x7fc8080040f0 2023-10-30T13:23:34.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.780+0000 7fc81a7fc640 1 -- 172.21.15.46:0/991643123 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc804007070 con 0x7fc8080040f0 2023-10-30T13:23:34.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.780+0000 7fc82171d640 1 -- 172.21.15.46:0/991643123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8080040f0 msgr2=0x7fc8080047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.780+0000 7fc82171d640 1 --2- 172.21.15.46:0/991643123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8080040f0 0x7fc8080047d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0c6ab0 tx=0x7fc80401ca00 comp rx=0 tx=0).stop 2023-10-30T13:23:34.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.780+0000 7fc81b7fe640 1 -- 172.21.15.46:0/991643123 reap_dead start 2023-10-30T13:23:34.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.780+0000 7fc82171d640 1 -- 172.21.15.46:0/991643123 shutdown_connections 2023-10-30T13:23:34.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.780+0000 7fc82171d640 1 -- 172.21.15.46:0/991643123 >> 172.21.15.46:0/991643123 conn(0x7fc81c06bed0 msgr2=0x7fc81c06c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:34.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.780+0000 7fc82171d640 1 -- 172.21.15.46:0/991643123 shutdown_connections 2023-10-30T13:23:34.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.780+0000 7fc82171d640 1 -- 172.21.15.46:0/991643123 wait complete. 2023-10-30T13:23:34.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc82171d640 1 Processor -- start 2023-10-30T13:23:34.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc82171d640 1 -- start start 2023-10-30T13:23:34.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc82171d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8080040f0 0x7fc81c1ef770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc82171d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc81c0cf280 0x7fc81c15af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc82171d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c0cf6e0 0x7fc81c15b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc82171d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c1efcb0 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc82171d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c14cbb0 con 0x7fc81c0cf280 2023-10-30T13:23:34.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc82171d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c15db10 con 0x7fc8080040f0 2023-10-30T13:23:34.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc81b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c0cf6e0 0x7fc81c15b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc81b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c0cf6e0 0x7fc81c15b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40032/0 (socket says 172.21.15.46:40032) 2023-10-30T13:23:34.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc81b7fe640 1 -- 172.21.15.46:0/4586195 learned_addr learned my addr 172.21.15.46:0/4586195 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:34.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc81affd640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8080040f0 0x7fc81c1ef770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc813fff640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc81c0cf280 0x7fc81c15af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc81b7fe640 1 -- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8080040f0 msgr2=0x7fc81c1ef770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc81b7fe640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8080040f0 0x7fc81c1ef770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc81b7fe640 1 -- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc81c0cf280 msgr2=0x7fc81c15af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc81b7fe640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc81c0cf280 0x7fc81c15af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc81b7fe640 1 -- 172.21.15.46:0/4586195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c15ceb0 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc81b7fe640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c0cf6e0 0x7fc81c15b480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc81c066a00 tx=0x7fc81400f170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:34.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc8137fe640 1 -- 172.21.15.46:0/4586195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81400fd20 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc8137fe640 1 -- 172.21.15.46:0/4586195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81400fec0 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc8137fe640 1 -- 172.21.15.46:0/4586195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc814015650 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc82171d640 1 -- 172.21.15.46:0/4586195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81c14ce30 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc82171d640 1 -- 172.21.15.46:0/4586195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81c15d7a0 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.784+0000 7fc82171d640 1 -- 172.21.15.46:0/4586195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81c15ac60 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.788+0000 7fc8137fe640 1 -- 172.21.15.46:0/4586195 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc8140157b0 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.788+0000 7fc8137fe640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7fc0664a0 0x7fc7fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.788+0000 7fc8137fe640 1 -- 172.21.15.46:0/4586195 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8140873f0 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.788+0000 7fc81affd640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7fc0664a0 0x7fc7fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.788+0000 7fc81affd640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7fc0664a0 0x7fc7fc068960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009950 tx=0x7fc80c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:34.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.792+0000 7fc8137fe640 1 -- 172.21.15.46:0/4586195 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc81c15ac60 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.948+0000 7fc82171d640 1 -- 172.21.15.46:0/4586195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc81c15ac60 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.948+0000 7fc8137fe640 1 -- 172.21.15.46:0/4586195 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc81c15ac60 con 0x7fc81c0cf6e0 2023-10-30T13:23:34.959 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-10-30T13:23:34.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.952+0000 7fc82171d640 1 -- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7fc0664a0 msgr2=0x7fc7fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.952+0000 7fc82171d640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7fc0664a0 0x7fc7fc068960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009950 tx=0x7fc80c008040 comp rx=0 tx=0).stop 2023-10-30T13:23:34.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.952+0000 7fc82171d640 1 -- 172.21.15.46:0/4586195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c0cf6e0 msgr2=0x7fc81c15b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.952+0000 7fc82171d640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c0cf6e0 0x7fc81c15b480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc81c066a00 tx=0x7fc81400f170 comp rx=0 tx=0).stop 2023-10-30T13:23:34.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.952+0000 7fc82171d640 1 -- 172.21.15.46:0/4586195 shutdown_connections 2023-10-30T13:23:34.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.952+0000 7fc82171d640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7fc0664a0 0x7fc7fc068960 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.952+0000 7fc82171d640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc81c0cf6e0 0x7fc81c15b480 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.952+0000 7fc82171d640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc81c0cf280 0x7fc81c15af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.952+0000 7fc82171d640 1 --2- 172.21.15.46:0/4586195 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8080040f0 0x7fc81c1ef770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.952+0000 7fc82171d640 1 -- 172.21.15.46:0/4586195 >> 172.21.15.46:0/4586195 conn(0x7fc81c06bed0 msgr2=0x7fc81c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:34.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.952+0000 7fc82171d640 1 -- 172.21.15.46:0/4586195 shutdown_connections 2023-10-30T13:23:34.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:34.952+0000 7fc82171d640 1 -- 172.21.15.46:0/4586195 wait complete. 2023-10-30T13:23:34.978 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f7122446640 1 Processor -- start 2023-10-30T13:23:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f7122446640 1 -- start start 2023-10-30T13:23:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f7122446640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f711c15d6f0 0x7f711c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f7122446640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f711c0cf160 0x7f711c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f7122446640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711c0cb800 con 0x7f711c1538a0 2023-10-30T13:23:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f7122446640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f711c0cb330 con 0x7f711c0cf160 2023-10-30T13:23:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f7122446640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f711c0cad70 con 0x7f711c15d6f0 2023-10-30T13:23:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f71209e3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f711c0cf160 0x7f711c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f71209e3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f711c0cf160 0x7f711c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40048/0 (socket says 172.21.15.46:40048) 2023-10-30T13:23:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f71209e3640 1 -- 172.21.15.46:0/2830564270 learned_addr learned my addr 172.21.15.46:0/2830564270 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:35.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f711bfff640 1 --1- 172.21.15.46:0/2830564270 >> v1:172.21.15.46:6789/0 conn(0x7f711c1538a0 0x7f711c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48626/0 (socket says 172.21.15.46:48626) 2023-10-30T13:23:35.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f711b7fe640 1 --2- 172.21.15.46:0/2830564270 >> v2:172.21.15.154:3300/0 conn(0x7f711c15d6f0 0x7f711c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f71209e3640 1 -- 172.21.15.46:0/2830564270 >> v2:172.21.15.154:3300/0 conn(0x7f711c15d6f0 msgr2=0x7f711c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:35.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f71209e3640 1 --2- 172.21.15.46:0/2830564270 >> v2:172.21.15.154:3300/0 conn(0x7f711c15d6f0 0x7f711c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f71209e3640 1 -- 172.21.15.46:0/2830564270 >> v1:172.21.15.46:6789/0 conn(0x7f711c1538a0 legacy=0x7f711c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f71209e3640 1 -- 172.21.15.46:0/2830564270 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f711c0cb050 con 0x7f711c0cf160 2023-10-30T13:23:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.116+0000 7f711affd640 1 -- 172.21.15.46:0/2830564270 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047772996 0 0) 0x7f711c0cb800 con 0x7f711c1538a0 2023-10-30T13:23:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f71209e3640 1 --2- 172.21.15.46:0/2830564270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c0cf160 0x7f711c0cf590 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f711c0b30a0 tx=0x7f711401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08c4cdc33e6840f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f711affd640 1 -- 172.21.15.46:0/2830564270 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140af400 con 0x7f711c0cf160 2023-10-30T13:23:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f711affd640 1 -- 172.21.15.46:0/2830564270 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71140ab050 con 0x7f711c0cf160 2023-10-30T13:23:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f711affd640 1 -- 172.21.15.46:0/2830564270 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140af780 con 0x7f711c0cf160 2023-10-30T13:23:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f7122446640 1 -- 172.21.15.46:0/2830564270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c0cf160 msgr2=0x7f711c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f7122446640 1 --2- 172.21.15.46:0/2830564270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c0cf160 0x7f711c0cf590 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f711c0b30a0 tx=0x7f711401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f7122446640 1 -- 172.21.15.46:0/2830564270 shutdown_connections 2023-10-30T13:23:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f7122446640 1 --2- 172.21.15.46:0/2830564270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c0cf160 0x7f711c0cf590 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f7122446640 1 --2- 172.21.15.46:0/2830564270 >> v2:172.21.15.154:3300/0 conn(0x7f711c15d6f0 0x7f711c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f7122446640 1 -- 172.21.15.46:0/2830564270 >> 172.21.15.46:0/2830564270 conn(0x7f711c06bfe0 msgr2=0x7f711c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:35.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f7122446640 1 -- 172.21.15.46:0/2830564270 shutdown_connections 2023-10-30T13:23:35.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f7122446640 1 -- 172.21.15.46:0/2830564270 wait complete. 2023-10-30T13:23:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.120+0000 7f7122446640 1 Processor -- start 2023-10-30T13:23:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7122446640 1 -- start start 2023-10-30T13:23:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7122446640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c0cf160 0x7f711c1542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7122446640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711c1538a0 0x7f711c0b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7122446640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c15d6f0 0x7f711c1ef880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7122446640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711c156c20 con 0x7f711c15d6f0 2023-10-30T13:23:35.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7122446640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711c1f3690 con 0x7f711c1538a0 2023-10-30T13:23:35.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7122446640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f711c1f3110 con 0x7f711c0cf160 2023-10-30T13:23:35.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f71209e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c15d6f0 0x7f711c1ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f71209e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c15d6f0 0x7f711c1ef880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40056/0 (socket says 172.21.15.46:40056) 2023-10-30T13:23:35.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f71209e3640 1 -- 172.21.15.46:0/3588543542 learned_addr learned my addr 172.21.15.46:0/3588543542 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:35.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f711b7fe640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711c1538a0 0x7f711c0b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f711bfff640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c0cf160 0x7f711c1542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f71209e3640 1 -- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c0cf160 msgr2=0x7f711c1542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f71209e3640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c0cf160 0x7f711c1542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f71209e3640 1 -- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711c1538a0 msgr2=0x7f711c0b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f71209e3640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711c1538a0 0x7f711c0b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f71209e3640 1 -- 172.21.15.46:0/3588543542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7114009b30 con 0x7f711c15d6f0 2023-10-30T13:23:35.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f711b7fe640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711c1538a0 0x7f711c0b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:35.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f71209e3640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c15d6f0 0x7f711c1ef880 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f71140b2040 tx=0x7f711401bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:35.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7118ff9640 1 -- 172.21.15.46:0/3588543542 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140afd10 con 0x7f711c15d6f0 2023-10-30T13:23:35.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7118ff9640 1 -- 172.21.15.46:0/3588543542 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71140afe70 con 0x7f711c15d6f0 2023-10-30T13:23:35.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7122446640 1 -- 172.21.15.46:0/3588543542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f711c154800 con 0x7f711c15d6f0 2023-10-30T13:23:35.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7118ff9640 1 -- 172.21.15.46:0/3588543542 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140ba640 con 0x7f711c15d6f0 2023-10-30T13:23:35.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7122446640 1 -- 172.21.15.46:0/3588543542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f711c1f33b0 con 0x7f711c15d6f0 2023-10-30T13:23:35.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.124+0000 7f7122446640 1 -- 172.21.15.46:0/3588543542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f711c0b2a50 con 0x7f711c15d6f0 2023-10-30T13:23:35.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.128+0000 7f7118ff9640 1 -- 172.21.15.46:0/3588543542 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f71140b6040 con 0x7f711c15d6f0 2023-10-30T13:23:35.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.128+0000 7f7118ff9640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f71080664a0 0x7f7108068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.128+0000 7f7118ff9640 1 -- 172.21.15.46:0/3588543542 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71141220d0 con 0x7f711c15d6f0 2023-10-30T13:23:35.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.128+0000 7f711bfff640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f71080664a0 0x7f7108068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.128+0000 7f711bfff640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f71080664a0 0x7f7108068960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f710c000f70 tx=0x7f710c007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:35.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.132+0000 7f7118ff9640 1 -- 172.21.15.46:0/3588543542 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f71140ef5d0 con 0x7f711c15d6f0 2023-10-30T13:23:35.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.320+0000 7f7122446640 1 -- 172.21.15.46:0/3588543542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f711c04e170 con 0x7f711c15d6f0 2023-10-30T13:23:35.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.320+0000 7f7118ff9640 1 -- 172.21.15.46:0/3588543542 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+6518 (secure 0 0 0) 0x7f7114009510 con 0x7f711c15d6f0 2023-10-30T13:23:35.328 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:35.329 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:30.713454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:35.329 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 11 2023-10-30T13:23:35.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.324+0000 7f7122446640 1 -- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f71080664a0 msgr2=0x7f7108068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.324+0000 7f7122446640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f71080664a0 0x7f7108068960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f710c000f70 tx=0x7f710c007b60 comp rx=0 tx=0).stop 2023-10-30T13:23:35.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.324+0000 7f7122446640 1 -- 172.21.15.46:0/3588543542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c15d6f0 msgr2=0x7f711c1ef880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.324+0000 7f7122446640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c15d6f0 0x7f711c1ef880 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f71140b2040 tx=0x7f711401bf60 comp rx=0 tx=0).stop 2023-10-30T13:23:35.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.324+0000 7f7122446640 1 -- 172.21.15.46:0/3588543542 shutdown_connections 2023-10-30T13:23:35.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.324+0000 7f7122446640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f71080664a0 0x7f7108068960 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.324+0000 7f7122446640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c15d6f0 0x7f711c1ef880 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.324+0000 7f7122446640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711c1538a0 0x7f711c0b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.324+0000 7f7122446640 1 --2- 172.21.15.46:0/3588543542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c0cf160 0x7f711c1542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.324+0000 7f7122446640 1 -- 172.21.15.46:0/3588543542 >> 172.21.15.46:0/3588543542 conn(0x7f711c06bfe0 msgr2=0x7f711c15aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.324+0000 7f7122446640 1 -- 172.21.15.46:0/3588543542 shutdown_connections 2023-10-30T13:23:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.324+0000 7f7122446640 1 -- 172.21.15.46:0/3588543542 wait complete. 2023-10-30T13:23:35.349 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:35.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.484+0000 7ff5cdcd6640 1 Processor -- start 2023-10-30T13:23:35.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.484+0000 7ff5cdcd6640 1 -- start start 2023-10-30T13:23:35.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5cdcd6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5c80c6bd0 0x7ff5c806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5cdcd6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c8066cf0 con 0x7ff5c8071110 2023-10-30T13:23:35.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5cdcd6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c80c70d0 con 0x7ff5c806f7a0 2023-10-30T13:23:35.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5cdcd6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c80700c0 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5c7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff5c8071110 0x7ff5c8158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48632/0 (socket says 172.21.15.46:48632) 2023-10-30T13:23:35.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5c7fff640 1 -- 172.21.15.46:0/1273343226 learned_addr learned my addr 172.21.15.46:0/1273343226 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:35.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5c6ffd640 1 --2- 172.21.15.46:0/1273343226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5c80c6bd0 0x7ff5c806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5c67fc640 1 -- 172.21.15.46:0/1273343226 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538993803 0 0) 0x7ff5c8066cf0 con 0x7ff5c8071110 2023-10-30T13:23:35.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5c67fc640 1 -- 172.21.15.46:0/1273343226 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b0003530 con 0x7ff5c8071110 2023-10-30T13:23:35.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5c67fc640 1 -- 172.21.15.46:0/1273343226 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440121025 0 0) 0x7ff5c80c70d0 con 0x7ff5c806f7a0 2023-10-30T13:23:35.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5c67fc640 1 -- 172.21.15.46:0/1273343226 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c8066cf0 con 0x7ff5c806f7a0 2023-10-30T13:23:35.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5c67fc640 1 -- 172.21.15.46:0/1273343226 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1220250060 0 0) 0x7ff5b0003530 con 0x7ff5c8071110 2023-10-30T13:23:35.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5c6ffd640 1 -- 172.21.15.46:0/1273343226 >> v1:172.21.15.154:6789/0 conn(0x7ff5c806f7a0 legacy=0x7ff5c8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5c6ffd640 1 -- 172.21.15.46:0/1273343226 >> v1:172.21.15.46:6789/0 conn(0x7ff5c8071110 legacy=0x7ff5c8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.488+0000 7ff5c6ffd640 1 -- 172.21.15.46:0/1273343226 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c80c7390 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.492+0000 7ff5c6ffd640 1 --2- 172.21.15.46:0/1273343226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5c80c6bd0 0x7ff5c806fb80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0097f0 tx=0x7ff5bc01ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aca9d936d346445 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:35.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.492+0000 7ff5c67fc640 1 -- 172.21.15.46:0/1273343226 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0001a00 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.492+0000 7ff5c67fc640 1 -- 172.21.15.46:0/1273343226 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc0a8650 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.492+0000 7ff5c67fc640 1 -- 172.21.15.46:0/1273343226 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0a89e0 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.492+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1273343226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5c80c6bd0 msgr2=0x7ff5c806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.492+0000 7ff5cdcd6640 1 --2- 172.21.15.46:0/1273343226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5c80c6bd0 0x7ff5c806fb80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0097f0 tx=0x7ff5bc01ebd0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.492+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1273343226 shutdown_connections 2023-10-30T13:23:35.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.492+0000 7ff5cdcd6640 1 --2- 172.21.15.46:0/1273343226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5c80c6bd0 0x7ff5c806fb80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.492+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1273343226 >> 172.21.15.46:0/1273343226 conn(0x7ff5c80c7910 msgr2=0x7ff5c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:35.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.492+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1273343226 shutdown_connections 2023-10-30T13:23:35.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.492+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1273343226 wait complete. 2023-10-30T13:23:35.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.492+0000 7ff5cdcd6640 1 Processor -- start 2023-10-30T13:23:35.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5cdcd6640 1 -- start start 2023-10-30T13:23:35.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5cdcd6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5c806f7a0 0x7ff5c80adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5cdcd6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c8071110 0x7ff5c80c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5cdcd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c80c6bd0 0x7ff5c81ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5cdcd6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c8144440 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5cdcd6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c81f2430 con 0x7ff5c8071110 2023-10-30T13:23:35.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5cdcd6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c81f26a0 con 0x7ff5c806f7a0 2023-10-30T13:23:35.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c80c6bd0 0x7ff5c81ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c80c6bd0 0x7ff5c81ef130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40060/0 (socket says 172.21.15.46:40060) 2023-10-30T13:23:35.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5c7fff640 1 -- 172.21.15.46:0/1650483781 learned_addr learned my addr 172.21.15.46:0/1650483781 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:35.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5c77fe640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5c806f7a0 0x7ff5c80adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5c6ffd640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c8071110 0x7ff5c80c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5c7fff640 1 -- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5c806f7a0 msgr2=0x7ff5c80adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5c7fff640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5c806f7a0 0x7ff5c80adc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5c7fff640 1 -- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c8071110 msgr2=0x7ff5c80c3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5c7fff640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c8071110 0x7ff5c80c3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5c7fff640 1 -- 172.21.15.46:0/1650483781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5bc009420 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5c6ffd640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c8071110 0x7ff5c80c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:35.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5c7fff640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c80c6bd0 0x7ff5c81ef130 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000cc50 tx=0x7ff5c000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:35.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5affff640 1 -- 172.21.15.46:0/1650483781 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c000ce60 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5affff640 1 -- 172.21.15.46:0/1650483781 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c0012400 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5affff640 1 -- 172.21.15.46:0/1650483781 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0012790 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1650483781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c81f2980 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1650483781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c81f2f40 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.496+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1650483781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c804e170 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.500+0000 7ff5affff640 1 -- 172.21.15.46:0/1650483781 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5c00128f0 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.500+0000 7ff5affff640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5b40664a0 0x7ff5b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.500+0000 7ff5affff640 1 -- 172.21.15.46:0/1650483781 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5c0084720 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.500+0000 7ff5c77fe640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5b40664a0 0x7ff5b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.500+0000 7ff5c77fe640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5b40664a0 0x7ff5b4068960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff5b80011f0 tx=0x7ff5b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:35.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.504+0000 7ff5affff640 1 -- 172.21.15.46:0/1650483781 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5c00519f0 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.660+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1650483781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5c8070600 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5affff640 1 -- 172.21.15.46:0/1650483781 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7ff5c0055010 con 0x7ff5c80c6bd0 2023-10-30T13:23:35.670 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:35.670 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:20.665723+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:35.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5b40664a0 msgr2=0x7ff5b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5cdcd6640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5b40664a0 0x7ff5b4068960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff5b80011f0 tx=0x7ff5b8009040 comp rx=0 tx=0).stop 2023-10-30T13:23:35.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1650483781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c80c6bd0 msgr2=0x7ff5c81ef130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5cdcd6640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c80c6bd0 0x7ff5c81ef130 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000cc50 tx=0x7ff5c000a430 comp rx=0 tx=0).stop 2023-10-30T13:23:35.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1650483781 shutdown_connections 2023-10-30T13:23:35.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5cdcd6640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5b40664a0 0x7ff5b4068960 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5cdcd6640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c80c6bd0 0x7ff5c81ef130 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5cdcd6640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c8071110 0x7ff5c80c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5cdcd6640 1 --2- 172.21.15.46:0/1650483781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5c806f7a0 0x7ff5c80adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1650483781 >> 172.21.15.46:0/1650483781 conn(0x7ff5c80c7910 msgr2=0x7ff5c80c8220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:35.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1650483781 shutdown_connections 2023-10-30T13:23:35.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.664+0000 7ff5cdcd6640 1 -- 172.21.15.46:0/1650483781 wait complete. 2023-10-30T13:23:35.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:35.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.836+0000 7f2a59565640 1 Processor -- start 2023-10-30T13:23:35.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.836+0000 7f2a59565640 1 -- start start 2023-10-30T13:23:35.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a59565640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2a54153d70 0x7f2a5406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a59565640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a5406e470 0x7f2a540cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a59565640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a540c8490 0x7f2a540cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a59565640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a5406d780 con 0x7f2a5406e470 2023-10-30T13:23:35.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a59565640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a5406afa0 con 0x7f2a54153d70 2023-10-30T13:23:35.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a59565640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54154150 con 0x7f2a540c8490 2023-10-30T13:23:35.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a4bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a5406e470 0x7f2a540cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a4bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a5406e470 0x7f2a540cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40074/0 (socket says 172.21.15.46:40074) 2023-10-30T13:23:35.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a4bfff640 1 -- 172.21.15.46:0/1491066157 learned_addr learned my addr 172.21.15.46:0/1491066157 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:35.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a537fe640 1 --2- 172.21.15.46:0/1491066157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a540c8490 0x7f2a540cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a52ffd640 1 --2- 172.21.15.46:0/1491066157 >> v2:172.21.15.154:3300/0 conn(0x7f2a54153d70 0x7f2a5406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a4bfff640 1 -- 172.21.15.46:0/1491066157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a540c8490 msgr2=0x7f2a540cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:35.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a4bfff640 1 --2- 172.21.15.46:0/1491066157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a540c8490 0x7f2a540cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a4bfff640 1 -- 172.21.15.46:0/1491066157 >> v2:172.21.15.154:3300/0 conn(0x7f2a54153d70 msgr2=0x7f2a5406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:35.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a4bfff640 1 --2- 172.21.15.46:0/1491066157 >> v2:172.21.15.154:3300/0 conn(0x7f2a54153d70 0x7f2a5406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.840+0000 7f2a4bfff640 1 -- 172.21.15.46:0/1491066157 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5406e030 con 0x7f2a5406e470 2023-10-30T13:23:35.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a4bfff640 1 --2- 172.21.15.46:0/1491066157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a5406e470 0x7f2a540cce00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2a5406d180 tx=0x7f2a3c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a246e434af478852 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a527fc640 1 -- 172.21.15.46:0/1491066157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c0aa5f0 con 0x7f2a5406e470 2023-10-30T13:23:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a527fc640 1 -- 172.21.15.46:0/1491066157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c0aa750 con 0x7f2a5406e470 2023-10-30T13:23:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a527fc640 1 -- 172.21.15.46:0/1491066157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c0aaae0 con 0x7f2a5406e470 2023-10-30T13:23:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a59565640 1 -- 172.21.15.46:0/1491066157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a5406e470 msgr2=0x7f2a540cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a59565640 1 --2- 172.21.15.46:0/1491066157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a5406e470 0x7f2a540cce00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2a5406d180 tx=0x7f2a3c01c4d0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a59565640 1 -- 172.21.15.46:0/1491066157 shutdown_connections 2023-10-30T13:23:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a59565640 1 --2- 172.21.15.46:0/1491066157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a540c8490 0x7f2a540cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a59565640 1 --2- 172.21.15.46:0/1491066157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a5406e470 0x7f2a540cce00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a59565640 1 --2- 172.21.15.46:0/1491066157 >> v2:172.21.15.154:3300/0 conn(0x7f2a54153d70 0x7f2a5406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a59565640 1 -- 172.21.15.46:0/1491066157 >> 172.21.15.46:0/1491066157 conn(0x7f2a54071ea0 msgr2=0x7f2a54065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:35.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a59565640 1 -- 172.21.15.46:0/1491066157 shutdown_connections 2023-10-30T13:23:35.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a59565640 1 -- 172.21.15.46:0/1491066157 wait complete. 2023-10-30T13:23:35.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a59565640 1 Processor -- start 2023-10-30T13:23:35.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a59565640 1 -- start start 2023-10-30T13:23:35.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.844+0000 7f2a59565640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a5406e470 0x7f2a54156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a59565640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a540c8490 0x7f2a540c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a59565640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a54153d70 0x7f2a5415c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a59565640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5415bb00 con 0x7f2a540c8490 2023-10-30T13:23:35.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a59565640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5415bde0 con 0x7f2a54153d70 2023-10-30T13:23:35.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a59565640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a541f3450 con 0x7f2a5406e470 2023-10-30T13:23:35.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a540c8490 0x7f2a540c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a540c8490 0x7f2a540c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40086/0 (socket says 172.21.15.46:40086) 2023-10-30T13:23:35.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4bfff640 1 -- 172.21.15.46:0/3570055104 learned_addr learned my addr 172.21.15.46:0/3570055104 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:35.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a52ffd640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a5406e470 0x7f2a54156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a537fe640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a54153d70 0x7f2a5415c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4bfff640 1 -- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a5406e470 msgr2=0x7f2a54156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4bfff640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a5406e470 0x7f2a54156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4bfff640 1 -- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a54153d70 msgr2=0x7f2a5415c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4bfff640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a54153d70 0x7f2a5415c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4bfff640 1 -- 172.21.15.46:0/3570055104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c0094d0 con 0x7f2a540c8490 2023-10-30T13:23:35.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a537fe640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a54153d70 0x7f2a5415c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:35.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4bfff640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a540c8490 0x7f2a540c7780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c097f60 tx=0x7f2a3c01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:35.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4b7fe640 1 -- 172.21.15.46:0/3570055104 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c0b3800 con 0x7f2a540c8490 2023-10-30T13:23:35.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4b7fe640 1 -- 172.21.15.46:0/3570055104 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c0b3980 con 0x7f2a540c8490 2023-10-30T13:23:35.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a59565640 1 -- 172.21.15.46:0/3570055104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a541f2ed0 con 0x7f2a540c8490 2023-10-30T13:23:35.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4b7fe640 1 -- 172.21.15.46:0/3570055104 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c0b3d30 con 0x7f2a540c8490 2023-10-30T13:23:35.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a59565640 1 -- 172.21.15.46:0/3570055104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a540ccb00 con 0x7f2a540c8490 2023-10-30T13:23:35.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a59565640 1 -- 172.21.15.46:0/3570055104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a5404e170 con 0x7f2a540c8490 2023-10-30T13:23:35.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.848+0000 7f2a4b7fe640 1 -- 172.21.15.46:0/3570055104 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2a3c0c6390 con 0x7f2a540c8490 2023-10-30T13:23:35.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.852+0000 7f2a4b7fe640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2a340664a0 0x7f2a34068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:35.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.852+0000 7f2a4b7fe640 1 -- 172.21.15.46:0/3570055104 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a3c0a7070 con 0x7f2a540c8490 2023-10-30T13:23:35.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.852+0000 7f2a52ffd640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2a340664a0 0x7f2a34068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:35.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.852+0000 7f2a52ffd640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2a340664a0 0x7f2a34068960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2a44007940 tx=0x7f2a44008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:35.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:35.856+0000 7f2a4b7fe640 1 -- 172.21.15.46:0/3570055104 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2a3c0ebbf0 con 0x7f2a540c8490 2023-10-30T13:23:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 -- 172.21.15.46:0/3570055104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a5406dd20 con 0x7f2a540c8490 2023-10-30T13:23:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a4b7fe640 1 -- 172.21.15.46:0/3570055104 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f2a3c0aa5b0 con 0x7f2a540c8490 2023-10-30T13:23:36.054 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:36.055 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:35.740725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:36.056 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-10-30T13:23:36.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 -- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2a340664a0 msgr2=0x7f2a34068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:36.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2a340664a0 0x7f2a34068960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2a44007940 tx=0x7f2a44008040 comp rx=0 tx=0).stop 2023-10-30T13:23:36.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 -- 172.21.15.46:0/3570055104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a540c8490 msgr2=0x7f2a540c7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:36.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a540c8490 0x7f2a540c7780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c097f60 tx=0x7f2a3c01ad10 comp rx=0 tx=0).stop 2023-10-30T13:23:36.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 -- 172.21.15.46:0/3570055104 shutdown_connections 2023-10-30T13:23:36.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2a340664a0 0x7f2a34068960 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a54153d70 0x7f2a5415c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a540c8490 0x7f2a540c7780 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 --2- 172.21.15.46:0/3570055104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a5406e470 0x7f2a54156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 -- 172.21.15.46:0/3570055104 >> 172.21.15.46:0/3570055104 conn(0x7f2a54071ea0 msgr2=0x7f2a5406f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 -- 172.21.15.46:0/3570055104 shutdown_connections 2023-10-30T13:23:36.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.048+0000 7f2a59565640 1 -- 172.21.15.46:0/3570055104 wait complete. 2023-10-30T13:23:36.678 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-30T13:23:19.337396+0000', 'modified': '2023-10-30T13:23:35.740725+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4306, 'mds_1': 4288, 'mds_2': 4271}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4271': {'gid': 4271, 'name': 'e', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.46:6837/994489214', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6835', 'nonce': 994489214}, {'type': 'v1', 'addr': '172.21.15.46:6837', 'nonce': 994489214}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4288': {'gid': 4288, 'name': 'b', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.154:6835/3109262047', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 3109262047}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 3109262047}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4306': {'gid': 4306, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.46:6836/275749119', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6834', 'nonce': 275749119}, {'type': 'v1', 'addr': '172.21.15.46:6836', 'nonce': 275749119}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-30T13:23:36.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-10-30T13:23:36.679 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-10-30T13:23:36.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.828+0000 7f667ca17640 1 Processor -- start 2023-10-30T13:23:36.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.828+0000 7f667ca17640 1 -- start start 2023-10-30T13:23:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667ca17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66780c7830 0x7f667806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667ca17640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667806d710 con 0x7f66780cce00 2023-10-30T13:23:36.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667ca17640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667806b560 con 0x7f667806e430 2023-10-30T13:23:36.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667ca17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667806dd30 con 0x7f66780c7830 2023-10-30T13:23:36.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f66780cce00 0x7f6678159850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49810/0 (socket says 172.21.15.46:49810) 2023-10-30T13:23:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667659c640 1 -- 172.21.15.46:0/1692433062 learned_addr learned my addr 172.21.15.46:0/1692433062 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f6675d9b640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66780c7830 0x7f667806ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633456967 0 0) 0x7f667806d710 con 0x7f66780cce00 2023-10-30T13:23:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6660003530 con 0x7f66780cce00 2023-10-30T13:23:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289077465 0 0) 0x7f667806b560 con 0x7f667806e430 2023-10-30T13:23:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f667806d710 con 0x7f667806e430 2023-10-30T13:23:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4110751190 0 0) 0x7f6660003530 con 0x7f66780cce00 2023-10-30T13:23:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66780c7830 msgr2=0x7f667806ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66780c7830 0x7f667806ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 >> v1:172.21.15.154:6789/0 conn(0x7f667806e430 legacy=0x7f6678159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780c3c80 con 0x7f66780cce00 2023-10-30T13:23:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f6668001680 con 0x7f66780cce00 2023-10-30T13:23:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 >> v1:172.21.15.46:6789/0 conn(0x7f66780cce00 legacy=0x7f6678159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:36.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6660003b40 0x7f6660003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66600047d0 0x7f6660004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66780c7830 0x7f6660005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667806dd30 con 0x7f6660003b40 2023-10-30T13:23:36.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666000ba40 con 0x7f66600047d0 2023-10-30T13:23:36.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f666000bbd0 con 0x7f66780c7830 2023-10-30T13:23:36.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667659c640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6660003b40 0x7f6660003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f6675d9b640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66600047d0 0x7f6660004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f6676d9d640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66780c7830 0x7f6660005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667659c640 1 -- 172.21.15.46:0/1692433062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66780c7830 msgr2=0x7f6660005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667659c640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66780c7830 0x7f6660005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667659c640 1 -- 172.21.15.46:0/1692433062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66600047d0 msgr2=0x7f6660004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667659c640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66600047d0 0x7f6660004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f667659c640 1 -- 172.21.15.46:0/1692433062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6660003530 con 0x7f6660003b40 2023-10-30T13:23:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.832+0000 7f6676d9d640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66780c7830 0x7f6660005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.836+0000 7f667659c640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6660003b40 0x7f6660003f40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f666800c190 tx=0x7f666801cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c7182ed4f07a76 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.836+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66680058e0 con 0x7f6660003b40 2023-10-30T13:23:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.836+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6668005a40 con 0x7f6660003b40 2023-10-30T13:23:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.836+0000 7f667559a640 1 -- 172.21.15.46:0/1692433062 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6668005e10 con 0x7f6660003b40 2023-10-30T13:23:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.836+0000 7f667ca17640 1 -- 172.21.15.46:0/1692433062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6660003b40 msgr2=0x7f6660003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:36.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.836+0000 7f667ca17640 1 --2- 172.21.15.46:0/1692433062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6660003b40 0x7f6660003f40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f666800c190 tx=0x7f666801cb80 comp rx=0 tx=0).stop 2023-10-30T13:23:36.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.836+0000 7f6676d9d640 1 -- 172.21.15.46:0/1692433062 reap_dead start 2023-10-30T13:23:36.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.836+0000 7f667ca17640 1 -- 172.21.15.46:0/1692433062 shutdown_connections 2023-10-30T13:23:36.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.836+0000 7f667ca17640 1 -- 172.21.15.46:0/1692433062 >> 172.21.15.46:0/1692433062 conn(0x7f6678071ea0 msgr2=0x7f6678065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:36.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.836+0000 7f667ca17640 1 -- 172.21.15.46:0/1692433062 shutdown_connections 2023-10-30T13:23:36.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.836+0000 7f667ca17640 1 -- 172.21.15.46:0/1692433062 wait complete. 2023-10-30T13:23:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f667ca17640 1 Processor -- start 2023-10-30T13:23:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f667ca17640 1 -- start start 2023-10-30T13:23:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f667ca17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6660003b40 0x7f66781582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f667ca17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66781576c0 0x7f66781f5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f667ca17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66781f5f50 0x7f667815cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f667ca17640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6678158ba0 con 0x7f66781f5f50 2023-10-30T13:23:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f667ca17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6678154450 con 0x7f6660003b40 2023-10-30T13:23:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f667ca17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6678159670 con 0x7f66781576c0 2023-10-30T13:23:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6676d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66781f5f50 0x7f667815cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6676d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66781f5f50 0x7f667815cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51222/0 (socket says 172.21.15.46:51222) 2023-10-30T13:23:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6676d9d640 1 -- 172.21.15.46:0/1132931068 learned_addr learned my addr 172.21.15.46:0/1132931068 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f667659c640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6660003b40 0x7f66781582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6675d9b640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66781576c0 0x7f66781f5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6676d9d640 1 -- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66781576c0 msgr2=0x7f66781f5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6676d9d640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66781576c0 0x7f66781f5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6676d9d640 1 -- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6660003b40 msgr2=0x7f66781582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6676d9d640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6660003b40 0x7f66781582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6676d9d640 1 -- 172.21.15.46:0/1132931068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66781587e0 con 0x7f66781f5f50 2023-10-30T13:23:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6676d9d640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66781f5f50 0x7f667815cc40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f667806c400 tx=0x7f667000dad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6666ffd640 1 -- 172.21.15.46:0/1132931068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670003de0 con 0x7f66781f5f50 2023-10-30T13:23:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6666ffd640 1 -- 172.21.15.46:0/1132931068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6670005070 con 0x7f66781f5f50 2023-10-30T13:23:36.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f6666ffd640 1 -- 172.21.15.46:0/1132931068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670005420 con 0x7f66781f5f50 2023-10-30T13:23:36.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f667ca17640 1 -- 172.21.15.46:0/1132931068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66781541b0 con 0x7f66781f5f50 2023-10-30T13:23:36.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.840+0000 7f667ca17640 1 -- 172.21.15.46:0/1132931068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6678159020 con 0x7f66781f5f50 2023-10-30T13:23:36.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f6666ffd640 1 -- 172.21.15.46:0/1132931068 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6670005580 con 0x7f66781f5f50 2023-10-30T13:23:36.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f6666ffd640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66580664d0 0x7f6658068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f6666ffd640 1 -- 172.21.15.46:0/1132931068 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66700830a0 con 0x7f66781f5f50 2023-10-30T13:23:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f667659c640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66580664d0 0x7f6658068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f667fdac000 1 Processor -- start 2023-10-30T13:23:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f667fdac000 1 -- start start 2023-10-30T13:23:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f667659c640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66580664d0 0x7f6658068990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f66680aa000 tx=0x7f6668001990 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:36.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f667fdac000 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 0x5648034f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f667fdac000 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4130 0x5648034f4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f667fdac000 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4a50 0x5648034feaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f667fdac000 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648032a7cc0 con 0x5648034f4a50 2023-10-30T13:23:36.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f667fdac000 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56480330a920 con 0x5648034f3810 2023-10-30T13:23:36.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f667fdac000 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x564803307e50 con 0x5648034f4130 2023-10-30T13:23:36.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f6676d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4a50 0x5648034feaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f667659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 0x5648034f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f6676d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4a50 0x5648034feaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51226/0 (socket says 172.21.15.46:51226) 2023-10-30T13:23:36.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f6676d9d640 1 -- 172.21.15.46:0/1265511940 learned_addr learned my addr 172.21.15.46:0/1265511940 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:36.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.844+0000 7f6675d9b640 1 --2- 172.21.15.46:0/1265511940 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4130 0x5648034f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f6676d9d640 1 -- 172.21.15.46:0/1265511940 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4130 msgr2=0x5648034f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:36.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f6676d9d640 1 --2- 172.21.15.46:0/1265511940 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4130 0x5648034f4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f6676d9d640 1 -- 172.21.15.46:0/1265511940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 msgr2=0x5648034f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:36.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f6676d9d640 1 --2- 172.21.15.46:0/1265511940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 0x5648034f3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f6676d9d640 1 -- 172.21.15.46:0/1265511940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564802ffc890 con 0x5648034f4a50 2023-10-30T13:23:36.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f6676d9d640 1 --2- 172.21.15.46:0/1265511940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4a50 0x5648034feaa0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6670081600 tx=0x7f6670054320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6895b7bed88a474 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:36.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f6657fff640 1 -- 172.21.15.46:0/1265511940 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667005c570 con 0x5648034f4a50 2023-10-30T13:23:36.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f6657fff640 1 -- 172.21.15.46:0/1265511940 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6670058170 con 0x5648034f4a50 2023-10-30T13:23:36.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f6657fff640 1 -- 172.21.15.46:0/1265511940 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66700526c0 con 0x5648034f4a50 2023-10-30T13:23:36.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 -- 172.21.15.46:0/1265511940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4a50 msgr2=0x5648034feaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:36.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 --2- 172.21.15.46:0/1265511940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4a50 0x5648034feaa0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6670081600 tx=0x7f6670054320 comp rx=0 tx=0).stop 2023-10-30T13:23:36.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 -- 172.21.15.46:0/1265511940 shutdown_connections 2023-10-30T13:23:36.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 --2- 172.21.15.46:0/1265511940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4a50 0x5648034feaa0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 --2- 172.21.15.46:0/1265511940 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4130 0x5648034f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 --2- 172.21.15.46:0/1265511940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 0x5648034f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 -- 172.21.15.46:0/1265511940 >> 172.21.15.46:0/1265511940 conn(0x5648034a9fc0 msgr2=0x5648034f15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:36.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 -- 172.21.15.46:0/1265511940 shutdown_connections 2023-10-30T13:23:36.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 -- 172.21.15.46:0/1265511940 wait complete. 2023-10-30T13:23:36.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 -- start start 2023-10-30T13:23:36.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 Processor -- start 2023-10-30T13:23:36.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 0x5648034edb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x5648034ee0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4a50 0x5648034ee5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564803303740 con 0x5648034f4130 2023-10-30T13:23:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648032c2d10 con 0x5648034f3810 2023-10-30T13:23:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648032c29b0 con 0x5648034f4a50 2023-10-30T13:23:36.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f667fdac000 10 client.0 _ll_register_callbacks cb 0x56480349e390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:23:36.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.848+0000 7f6675d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x5648034ee0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f6675d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x5648034ee0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51240/0 (socket says 172.21.15.46:51240) 2023-10-30T13:23:36.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f6675d9b640 1 -- 172.21.15.46:0/3932647568 learned_addr learned my addr 172.21.15.46:0/3932647568 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:36.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667659c640 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 0x5648034edb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f6676d9d640 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4a50 0x5648034ee5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f6675d9b640 1 -- 172.21.15.46:0/3932647568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4a50 msgr2=0x5648034ee5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:36.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f6675d9b640 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4a50 0x5648034ee5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f6675d9b640 1 -- 172.21.15.46:0/3932647568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 msgr2=0x5648034edb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:36.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f6675d9b640 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 0x5648034edb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f6675d9b640 1 -- 172.21.15.46:0/3932647568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6678159020 con 0x5648034f4130 2023-10-30T13:23:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f6675d9b640 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x5648034ee0a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f666c015a40 tx=0x7f666c015b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e582a753b367d631 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f66557fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2023-10-30T13:23:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f66557fa640 1 -- 172.21.15.46:0/3932647568 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c01a070 con 0x5648034f4130 2023-10-30T13:23:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f66557fa640 1 -- 172.21.15.46:0/3932647568 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f666c004300 con 0x5648034f4130 2023-10-30T13:23:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667fdac000 1 -- 172.21.15.46:0/3932647568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5648032b7370 con 0x5648034f4130 2023-10-30T13:23:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f66557fa640 1 -- 172.21.15.46:0/3932647568 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f666c0044c0 con 0x5648034f4130 2023-10-30T13:23:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667fdac000 10 client.4653 fetch_fsmap learned FSMap version 12 2023-10-30T13:23:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667fdac000 1 -- 172.21.15.46:0/3932647568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x5648032c5460 con 0x5648034f4130 2023-10-30T13:23:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f66557fa640 1 -- 172.21.15.46:0/3932647568 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5648032b7370 con 0x5648034f4130 2023-10-30T13:23:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f66557fa640 1 -- 172.21.15.46:0/3932647568 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f666c011040 con 0x5648034f4130 2023-10-30T13:23:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667fdac000 10 client.4653 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T13:23:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667fdac000 10 client.4653 resolve_mds: resolved name 'e' to daemon mds.e 2023-10-30T13:23:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667fdac000 20 client.4653 populate_metadata read hostname 'smithi046' 2023-10-30T13:23:36.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667fdac000 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803507260 0x564803509650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:36.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667fdac000 4 client.4653 mds_command: new command op to 4271 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T13:23:36.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667fdac000 1 -- 172.21.15.46:0/3932647568 --> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56480329d2d0 con 0x564803507260 2023-10-30T13:23:36.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667659c640 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803507260 0x564803509650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:36.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667659c640 1 -- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803507260 msgr2=0x564803509650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2023-10-30T13:23:36.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667659c640 1 -- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803507260 msgr2=0x564803509650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:23:36.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667659c640 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803507260 0x564803509650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:23:36.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:36.852+0000 7f667659c640 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803507260 0x564803509650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:23:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:37.056+0000 7f667659c640 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803507260 0x564803509650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:37.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:37.056+0000 7f667659c640 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803507260 0x564803509650 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6091e7f893e2e2a2 server_cookie=c3eae490d4df5cb8 in_seq=0 out_seq=0 2023-10-30T13:23:37.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:37.056+0000 7f66557fa640 10 client.4653 ms_handle_connect on v2:172.21.15.46:6835/994489214 2023-10-30T13:23:38.917 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T13:23:38.908+0000 7fce56864640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:23:40.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.748+0000 7f66557fa640 1 -- 172.21.15.46:0/3932647568 <== mds.2 v2:172.21.15.46:6835/994489214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56480329d2d0 con 0x564803507260 2023-10-30T13:23:40.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.748+0000 7f66557fa640 10 client.4653 handle_command_reply: tid=0 2023-10-30T13:23:40.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f667fdac000 1 client.4653 shutdown 2023-10-30T13:23:40.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f667fdac000 1 -- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 msgr2=0x5648034ee0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:40.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f667fdac000 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x5648034ee0a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f666c015a40 tx=0x7f666c015b30 comp rx=0 tx=0).stop 2023-10-30T13:23:40.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f667fdac000 1 -- 172.21.15.46:0/3932647568 shutdown_connections 2023-10-30T13:23:40.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f667fdac000 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803507260 0x564803509650 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f667fdac000 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4a50 0x5648034ee5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f667fdac000 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x5648034ee0a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f667fdac000 1 --2- 172.21.15.46:0/3932647568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 0x5648034edb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f66557fa640 0 client.4653 ms_handle_reset on v2:172.21.15.46:6835/994489214 2023-10-30T13:23:40.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f667fdac000 1 -- 172.21.15.46:0/3932647568 >> 172.21.15.46:0/3932647568 conn(0x5648034a9fc0 msgr2=0x5648034f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:40.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f667fdac000 1 -- 172.21.15.46:0/3932647568 shutdown_connections 2023-10-30T13:23:40.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f667fdac000 1 -- 172.21.15.46:0/3932647568 wait complete. 2023-10-30T13:23:40.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.752+0000 7f667fdac000 20 client.4653 trim_cache size 0 max 16384 2023-10-30T13:23:40.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.768+0000 7f667fdac000 1 Processor -- start 2023-10-30T13:23:40.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.768+0000 7f667fdac000 1 -- start start 2023-10-30T13:23:40.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f3810 0x564803552e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x564803553340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f4a50 0x564803553880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564802fbd940 con 0x5648034f4130 2023-10-30T13:23:40.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648033088b0 con 0x5648034f4a50 2023-10-30T13:23:40.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648032a41a0 con 0x5648034f3810 2023-10-30T13:23:40.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6675d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x564803553340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6675d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x564803553340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51250/0 (socket says 172.21.15.46:51250) 2023-10-30T13:23:40.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6675d9b640 1 -- 172.21.15.46:0/3561829889 learned_addr learned my addr 172.21.15.46:0/3561829889 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:40.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667659c640 1 --2- 172.21.15.46:0/3561829889 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f3810 0x564803552e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6676d9d640 1 --2- 172.21.15.46:0/3561829889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f4a50 0x564803553880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6675d9b640 1 -- 172.21.15.46:0/3561829889 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f3810 msgr2=0x564803552e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:40.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6675d9b640 1 --2- 172.21.15.46:0/3561829889 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f3810 0x564803552e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6675d9b640 1 -- 172.21.15.46:0/3561829889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f4a50 msgr2=0x564803553880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:40.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6675d9b640 1 --2- 172.21.15.46:0/3561829889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f4a50 0x564803553880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6675d9b640 1 -- 172.21.15.46:0/3561829889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564803505f50 con 0x5648034f4130 2023-10-30T13:23:40.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6676d9d640 1 --2- 172.21.15.46:0/3561829889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f4a50 0x564803553880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:40.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6675d9b640 1 --2- 172.21.15.46:0/3561829889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x564803553340 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f666c001ec0 tx=0x7f666c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3efa168c2dd6765 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:40.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6664ff9640 1 -- 172.21.15.46:0/3561829889 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c009110 con 0x5648034f4130 2023-10-30T13:23:40.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6664ff9640 1 -- 172.21.15.46:0/3561829889 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c004300 con 0x5648034f4130 2023-10-30T13:23:40.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f6664ff9640 1 -- 172.21.15.46:0/3561829889 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c0044b0 con 0x5648034f4130 2023-10-30T13:23:40.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 -- 172.21.15.46:0/3561829889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 msgr2=0x564803553340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:40.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 --2- 172.21.15.46:0/3561829889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x564803553340 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f666c001ec0 tx=0x7f666c017000 comp rx=0 tx=0).stop 2023-10-30T13:23:40.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 -- 172.21.15.46:0/3561829889 shutdown_connections 2023-10-30T13:23:40.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 --2- 172.21.15.46:0/3561829889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f4a50 0x564803553880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 --2- 172.21.15.46:0/3561829889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x564803553340 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 --2- 172.21.15.46:0/3561829889 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f3810 0x564803552e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 -- 172.21.15.46:0/3561829889 >> 172.21.15.46:0/3561829889 conn(0x5648034a9fc0 msgr2=0x5648034f9080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:40.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 -- 172.21.15.46:0/3561829889 shutdown_connections 2023-10-30T13:23:40.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 -- 172.21.15.46:0/3561829889 wait complete. 2023-10-30T13:23:40.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.772+0000 7f667fdac000 1 -- start start 2023-10-30T13:23:40.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667fdac000 1 Processor -- start 2023-10-30T13:23:40.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667fdac000 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 0x564803556f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667fdac000 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x564803550990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667fdac000 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4a50 0x564803550ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667fdac000 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648032c24d0 con 0x5648034f4130 2023-10-30T13:23:40.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667fdac000 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564803300350 con 0x5648034f3810 2023-10-30T13:23:40.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667fdac000 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648032d9a10 con 0x5648034f4a50 2023-10-30T13:23:40.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667fdac000 10 client.0 _ll_register_callbacks cb 0x7f6670050660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:23:40.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f6675d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x564803550990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f6675d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x564803550990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51258/0 (socket says 172.21.15.46:51258) 2023-10-30T13:23:40.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f6675d9b640 1 -- 172.21.15.46:0/2846900213 learned_addr learned my addr 172.21.15.46:0/2846900213 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:40.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f6676d9d640 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4a50 0x564803550ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667659c640 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 0x564803556f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f6675d9b640 1 -- 172.21.15.46:0/2846900213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4a50 msgr2=0x564803550ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:40.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f6675d9b640 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4a50 0x564803550ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f6675d9b640 1 -- 172.21.15.46:0/2846900213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 msgr2=0x564803556f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:40.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f6675d9b640 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 0x564803556f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f6675d9b640 1 -- 172.21.15.46:0/2846900213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648032c5460 con 0x5648034f4130 2023-10-30T13:23:40.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f6675d9b640 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x564803550990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f666c006aa0 tx=0x7f666c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8007e2a18b42760e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:40.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f66557fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2023-10-30T13:23:40.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f66557fa640 1 -- 172.21.15.46:0/2846900213 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c003710 con 0x5648034f4130 2023-10-30T13:23:40.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f66557fa640 1 -- 172.21.15.46:0/2846900213 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f666c003870 con 0x5648034f4130 2023-10-30T13:23:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667fdac000 1 -- 172.21.15.46:0/2846900213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564803505950 con 0x5648034f4130 2023-10-30T13:23:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f66557fa640 1 -- 172.21.15.46:0/2846900213 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f666c003a30 con 0x5648034f4130 2023-10-30T13:23:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667fdac000 10 client.4659 fetch_fsmap learned FSMap version 12 2023-10-30T13:23:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667fdac000 1 -- 172.21.15.46:0/2846900213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x5648032ab760 con 0x5648034f4130 2023-10-30T13:23:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f66557fa640 1 -- 172.21.15.46:0/2846900213 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564803505950 con 0x5648034f4130 2023-10-30T13:23:40.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f66557fa640 1 -- 172.21.15.46:0/2846900213 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f666c011040 con 0x5648034f4130 2023-10-30T13:23:40.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.776+0000 7f667fdac000 10 client.4659 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T13:23:40.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 10 client.4659 resolve_mds: resolved name 'e' to daemon mds.e 2023-10-30T13:23:40.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 20 client.4659 populate_metadata read hostname 'smithi046' 2023-10-30T13:23:40.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803554c90 0x564803555070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 4 client.4659 mds_command: new command op to 4271 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T13:23:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 -- 172.21.15.46:0/2846900213 --> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56480329b6f0 con 0x564803554c90 2023-10-30T13:23:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667659c640 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803554c90 0x564803555070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667659c640 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803554c90 0x564803555070 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=16a14c34149efbe5 server_cookie=ee980286413f139b in_seq=0 out_seq=0 2023-10-30T13:23:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f66557fa640 10 client.4659 ms_handle_connect on v2:172.21.15.46:6835/994489214 2023-10-30T13:23:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f66557fa640 1 -- 172.21.15.46:0/2846900213 <== mds.2 v2:172.21.15.46:6835/994489214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56480329b6f0 con 0x564803554c90 2023-10-30T13:23:40.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f66557fa640 10 client.4659 handle_command_reply: tid=0 2023-10-30T13:23:40.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 client.4659 shutdown 2023-10-30T13:23:40.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 -- 172.21.15.46:0/2846900213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 msgr2=0x564803550990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:40.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x564803550990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f666c006aa0 tx=0x7f666c004740 comp rx=0 tx=0).stop 2023-10-30T13:23:40.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 -- 172.21.15.46:0/2846900213 shutdown_connections 2023-10-30T13:23:40.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.46:6835/994489214,v1:172.21.15.46:6837/994489214] conn(0x564803554c90 0x564803555070 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5648034f4a50 0x564803550ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648034f4130 0x564803550990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f66557fa640 0 client.4659 ms_handle_reset on v2:172.21.15.46:6835/994489214 2023-10-30T13:23:40.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 --2- 172.21.15.46:0/2846900213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648034f3810 0x564803556f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 -- 172.21.15.46:0/2846900213 >> 172.21.15.46:0/2846900213 conn(0x5648034a9fc0 msgr2=0x5648034f6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 -- 172.21.15.46:0/2846900213 shutdown_connections 2023-10-30T13:23:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 1 -- 172.21.15.46:0/2846900213 wait complete. 2023-10-30T13:23:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667fdac000 20 client.4659 trim_cache size 0 max 16384 2023-10-30T13:23:40.799 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-10-30T13:23:40.800 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2", 2023-10-30T13:23:40.800 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 2, 2023-10-30T13:23:40.800 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4271, 2023-10-30T13:23:40.800 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2023-10-30T13:23:40.800 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2023-10-30T13:23:40.801 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2023-10-30T13:23:40.801 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 10.064582496, 2023-10-30T13:23:40.801 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 12, 2023-10-30T13:23:40.801 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 20, 2023-10-30T13:23:40.801 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 20, 2023-10-30T13:23:40.801 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 27.875001865000002 2023-10-30T13:23:40.802 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-10-30T13:23:40.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667ca17640 1 -- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66580664d0 msgr2=0x7f6658068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:40.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667ca17640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66580664d0 0x7f6658068990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f66680aa000 tx=0x7f6668001990 comp rx=0 tx=0).stop 2023-10-30T13:23:40.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667ca17640 1 -- 172.21.15.46:0/1132931068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66781f5f50 msgr2=0x7f667815cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:40.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667ca17640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66781f5f50 0x7f667815cc40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f667806c400 tx=0x7f667000dad0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667ca17640 1 -- 172.21.15.46:0/1132931068 shutdown_connections 2023-10-30T13:23:40.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667ca17640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66580664d0 0x7f6658068990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667ca17640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66781f5f50 0x7f667815cc40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667ca17640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66781576c0 0x7f66781f5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667ca17640 1 --2- 172.21.15.46:0/1132931068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6660003b40 0x7f66781582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667ca17640 1 -- 172.21.15.46:0/1132931068 >> 172.21.15.46:0/1132931068 conn(0x7f6678071ea0 msgr2=0x7f66780cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:40.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667ca17640 1 -- 172.21.15.46:0/1132931068 shutdown_connections 2023-10-30T13:23:40.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.780+0000 7f667ca17640 1 -- 172.21.15.46:0/1132931068 wait complete. 2023-10-30T13:23:40.811 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-30T13:23:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.952+0000 7f4ff1b65640 1 Processor -- start 2023-10-30T13:23:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.952+0000 7f4ff1b65640 1 -- start start 2023-10-30T13:23:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4ff1b65640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4fec151910 0x7f4fec158120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4ff1b65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4fec06a970 0x7f4fec0c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4ff1b65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec06a4e0 con 0x7f4fec06ade0 2023-10-30T13:23:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4ff1b65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fec158660 con 0x7f4fec06a970 2023-10-30T13:23:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4ff1b65640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fec0c6c50 con 0x7f4fec151910 2023-10-30T13:23:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4feaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4fec06a970 0x7f4fec0c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4feaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4fec06a970 0x7f4fec0c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51274/0 (socket says 172.21.15.46:51274) 2023-10-30T13:23:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4feaffd640 1 -- 172.21.15.46:0/3700545989 learned_addr learned my addr 172.21.15.46:0/3700545989 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/3700545989 >> v2:172.21.15.154:3300/0 conn(0x7f4fec151910 0x7f4fec158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4feaffd640 1 -- 172.21.15.46:0/3700545989 >> v2:172.21.15.154:3300/0 conn(0x7f4fec151910 msgr2=0x7f4fec158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:40.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4feaffd640 1 --2- 172.21.15.46:0/3700545989 >> v2:172.21.15.154:3300/0 conn(0x7f4fec151910 0x7f4fec158120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4feaffd640 1 -- 172.21.15.46:0/3700545989 >> v1:172.21.15.154:6789/0 conn(0x7f4fec06ade0 legacy=0x7f4fec0c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:40.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.956+0000 7f4feaffd640 1 -- 172.21.15.46:0/3700545989 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec0c6f30 con 0x7f4fec06a970 2023-10-30T13:23:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4feaffd640 1 --2- 172.21.15.46:0/3700545989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fec06a970 0x7f4fec0c5810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0b1020 tx=0x7f4fe001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20cc052a619ea114 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4fea7fc640 1 -- 172.21.15.46:0/3700545989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe00a89f0 con 0x7f4fec06a970 2023-10-30T13:23:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4fea7fc640 1 -- 172.21.15.46:0/3700545989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe00a8b50 con 0x7f4fec06a970 2023-10-30T13:23:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4fea7fc640 1 -- 172.21.15.46:0/3700545989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe00b0440 con 0x7f4fec06a970 2023-10-30T13:23:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4ff1b65640 1 -- 172.21.15.46:0/3700545989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fec06a970 msgr2=0x7f4fec0c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4ff1b65640 1 --2- 172.21.15.46:0/3700545989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fec06a970 0x7f4fec0c5810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0b1020 tx=0x7f4fe001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4ff1b65640 1 -- 172.21.15.46:0/3700545989 shutdown_connections 2023-10-30T13:23:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4ff1b65640 1 --2- 172.21.15.46:0/3700545989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fec06a970 0x7f4fec0c5810 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4ff1b65640 1 --2- 172.21.15.46:0/3700545989 >> v2:172.21.15.154:3300/0 conn(0x7f4fec151910 0x7f4fec158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4ff1b65640 1 -- 172.21.15.46:0/3700545989 >> 172.21.15.46:0/3700545989 conn(0x7f4fec0c7910 msgr2=0x7f4fec0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4ff1b65640 1 -- 172.21.15.46:0/3700545989 shutdown_connections 2023-10-30T13:23:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4ff1b65640 1 -- 172.21.15.46:0/3700545989 wait complete. 2023-10-30T13:23:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4ff1b65640 1 Processor -- start 2023-10-30T13:23:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.960+0000 7f4ff1b65640 1 -- start start 2023-10-30T13:23:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4ff1b65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec06a970 0x7f4fec14d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4ff1b65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fec151910 0x7f4fec06ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4ff1b65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fec14c6b0 0x7f4fec0c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4ff1b65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0ca6c0 con 0x7f4fec14c6b0 2023-10-30T13:23:40.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4ff1b65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0ad970 con 0x7f4fec151910 2023-10-30T13:23:40.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4ff1b65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0adc10 con 0x7f4fec06a970 2023-10-30T13:23:40.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4febfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fec14c6b0 0x7f4fec0c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4febfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fec14c6b0 0x7f4fec0c2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51282/0 (socket says 172.21.15.46:51282) 2023-10-30T13:23:40.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4febfff640 1 -- 172.21.15.46:0/2108406013 learned_addr learned my addr 172.21.15.46:0/2108406013 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:40.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec06a970 0x7f4fec14d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4feaffd640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fec151910 0x7f4fec06ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4febfff640 1 -- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec06a970 msgr2=0x7f4fec14d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:40.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4febfff640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec06a970 0x7f4fec14d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4febfff640 1 -- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fec151910 msgr2=0x7f4fec06ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:40.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4febfff640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fec151910 0x7f4fec06ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4febfff640 1 -- 172.21.15.46:0/2108406013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe00094d0 con 0x7f4fec14c6b0 2023-10-30T13:23:40.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4febfff640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fec14c6b0 0x7f4fec0c2df0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400c7a0 tx=0x7f4fe400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:40.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4fd3fff640 1 -- 172.21.15.46:0/2108406013 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4003b80 con 0x7f4fec14c6b0 2023-10-30T13:23:40.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4fd3fff640 1 -- 172.21.15.46:0/2108406013 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe4003ce0 con 0x7f4fec14c6b0 2023-10-30T13:23:40.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4fd3fff640 1 -- 172.21.15.46:0/2108406013 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe400aa30 con 0x7f4fec14c6b0 2023-10-30T13:23:40.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4ff1b65640 1 -- 172.21.15.46:0/2108406013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec0adef0 con 0x7f4fec14c6b0 2023-10-30T13:23:40.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.964+0000 7f4ff1b65640 1 -- 172.21.15.46:0/2108406013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec0c28d0 con 0x7f4fec14c6b0 2023-10-30T13:23:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4fd3fff640 1 -- 172.21.15.46:0/2108406013 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4fe400ab90 con 0x7f4fec14c6b0 2023-10-30T13:23:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4fd3fff640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4fd80664d0 0x7f4fd8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4fd3fff640 1 -- 172.21.15.46:0/2108406013 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fe4083bd0 con 0x7f4fec14c6b0 2023-10-30T13:23:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4ff4ef3000 1 Processor -- start 2023-10-30T13:23:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4fd80664d0 0x7f4fd8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4ff4ef3000 1 -- start start 2023-10-30T13:23:40.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4ff4ef3000 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe1810 0x55bc0dfe1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4ff4ef3000 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dfe2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4ff4ef3000 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe2a50 0x55bc0dfecaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4ff4ef3000 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0dd95cc0 con 0x55bc0dfe2130 2023-10-30T13:23:40.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4ff4ef3000 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ddf8920 con 0x55bc0dfe1810 2023-10-30T13:23:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4ff4ef3000 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ddf5e50 con 0x55bc0dfe2a50 2023-10-30T13:23:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4fd80664d0 0x7f4fd8068990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc007a60 tx=0x7f4fdc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4feaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dfe2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4feaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dfe2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51298/0 (socket says 172.21.15.46:51298) 2023-10-30T13:23:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4feaffd640 1 -- 172.21.15.46:0/159420142 learned_addr learned my addr 172.21.15.46:0/159420142 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/159420142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe1810 0x55bc0dfe1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4febfff640 1 --2- 172.21.15.46:0/159420142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe2a50 0x55bc0dfecaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4feaffd640 1 -- 172.21.15.46:0/159420142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe2a50 msgr2=0x55bc0dfecaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4feaffd640 1 --2- 172.21.15.46:0/159420142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe2a50 0x55bc0dfecaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4feaffd640 1 -- 172.21.15.46:0/159420142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe1810 msgr2=0x55bc0dfe1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4feaffd640 1 --2- 172.21.15.46:0/159420142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe1810 0x55bc0dfe1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4feaffd640 1 -- 172.21.15.46:0/159420142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc0daea890 con 0x55bc0dfe2130 2023-10-30T13:23:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.968+0000 7f4feaffd640 1 --2- 172.21.15.46:0/159420142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dfe2510 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00b3040 tx=0x7f4fe00aa070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dcd167bcfcbbd21 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4fd17fa640 1 -- 172.21.15.46:0/159420142 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe00b0700 con 0x55bc0dfe2130 2023-10-30T13:23:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4fd17fa640 1 -- 172.21.15.46:0/159420142 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe00b0860 con 0x55bc0dfe2130 2023-10-30T13:23:40.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4fd17fa640 1 -- 172.21.15.46:0/159420142 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe00b0c10 con 0x55bc0dfe2130 2023-10-30T13:23:40.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/159420142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 msgr2=0x55bc0dfe2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:40.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/159420142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dfe2510 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00b3040 tx=0x7f4fe00aa070 comp rx=0 tx=0).stop 2023-10-30T13:23:40.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/159420142 shutdown_connections 2023-10-30T13:23:40.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/159420142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe2a50 0x55bc0dfecaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/159420142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dfe2510 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/159420142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe1810 0x55bc0dfe1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/159420142 >> 172.21.15.46:0/159420142 conn(0x55bc0df97fc0 msgr2=0x55bc0dfdf5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:40.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/159420142 shutdown_connections 2023-10-30T13:23:40.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/159420142 wait complete. 2023-10-30T13:23:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 -- start start 2023-10-30T13:23:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 Processor -- start 2023-10-30T13:23:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0dfdbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dfdc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2a50 0x55bc0dfe7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ddf1740 con 0x55bc0dfe2130 2023-10-30T13:23:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ddb0d10 con 0x55bc0dfe2a50 2023-10-30T13:23:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ddb09b0 con 0x55bc0dfe1810 2023-10-30T13:23:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4ff4ef3000 10 client.0 _ll_register_callbacks cb 0x55bc0df8c390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:23:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4feaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dfdc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4feaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dfdc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51300/0 (socket says 172.21.15.46:51300) 2023-10-30T13:23:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4feaffd640 1 -- 172.21.15.46:0/2539301511 learned_addr learned my addr 172.21.15.46:0/2539301511 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0dfdbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4febfff640 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2a50 0x55bc0dfe7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4feaffd640 1 -- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 msgr2=0x55bc0dfdbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4feaffd640 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0dfdbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4feaffd640 1 -- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2a50 msgr2=0x55bc0dfe7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4feaffd640 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2a50 0x55bc0dfe7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4feaffd640 1 -- 172.21.15.46:0/2539301511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec0c6f30 con 0x55bc0dfe2130 2023-10-30T13:23:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0dfdbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.972+0000 7f4febfff640 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2a50 0x55bc0dfe7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4feaffd640 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dfdc490 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4fe001ed30 tx=0x7f4fe00046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e3e55b9241dce5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4fca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2023-10-30T13:23:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4fca7fc640 1 -- 172.21.15.46:0/2539301511 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0003820 con 0x55bc0dfe2130 2023-10-30T13:23:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4fca7fc640 1 -- 172.21.15.46:0/2539301511 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fe0003980 con 0x55bc0dfe2130 2023-10-30T13:23:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2539301511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc0dda5370 con 0x55bc0dfe2130 2023-10-30T13:23:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4fca7fc640 1 -- 172.21.15.46:0/2539301511 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fe0003b40 con 0x55bc0dfe2130 2023-10-30T13:23:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 10 client.4671 fetch_fsmap learned FSMap version 12 2023-10-30T13:23:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2539301511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55bc0ddb3460 con 0x55bc0dfe2130 2023-10-30T13:23:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4fca7fc640 1 -- 172.21.15.46:0/2539301511 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bc0dda5370 con 0x55bc0dfe2130 2023-10-30T13:23:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4fca7fc640 1 -- 172.21.15.46:0/2539301511 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f4fe0003ca0 con 0x55bc0dfe2130 2023-10-30T13:23:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 10 client.4671 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T13:23:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 10 client.4671 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-30T13:23:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 20 client.4671 populate_metadata read hostname 'smithi046' 2023-10-30T13:23:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0dfe89a0 0x55bc0dfe8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 4 client.4671 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T13:23:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2539301511 --> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc0dd8b2d0 con 0x55bc0dfe89a0 2023-10-30T13:23:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0dfe89a0 0x55bc0dfe8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0dfe89a0 0x55bc0dfe8d80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7d44f52a7f656686 server_cookie=d2a947980494e283 in_seq=0 out_seq=0 2023-10-30T13:23:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4fca7fc640 10 client.4671 ms_handle_connect on v2:172.21.15.154:6834/3109262047 2023-10-30T13:23:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4fca7fc640 1 -- 172.21.15.46:0/2539301511 <== mds.1 v2:172.21.15.154:6834/3109262047 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bc0dd8b2d0 con 0x55bc0dfe89a0 2023-10-30T13:23:40.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4fca7fc640 10 client.4671 handle_command_reply: tid=0 2023-10-30T13:23:40.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 client.4671 shutdown 2023-10-30T13:23:40.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2539301511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 msgr2=0x55bc0dfdc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:40.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dfdc490 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4fe001ed30 tx=0x7f4fe00046d0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2539301511 shutdown_connections 2023-10-30T13:23:40.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0dfe89a0 0x55bc0dfe8d80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2a50 0x55bc0dfe7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dfdc490 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/2539301511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0dfdbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.976+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2539301511 >> 172.21.15.46:0/2539301511 conn(0x55bc0df97fc0 msgr2=0x55bc0dfe1090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.980+0000 7f4fca7fc640 0 client.4671 ms_handle_reset on v2:172.21.15.154:6834/3109262047 2023-10-30T13:23:40.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.980+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2539301511 shutdown_connections 2023-10-30T13:23:40.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.980+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2539301511 wait complete. 2023-10-30T13:23:40.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.980+0000 7f4ff4ef3000 20 client.4671 trim_cache size 0 max 16384 2023-10-30T13:23:41.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.992+0000 7f4ff4ef3000 1 Processor -- start 2023-10-30T13:23:41.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.992+0000 7f4ff4ef3000 1 -- start start 2023-10-30T13:23:41.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0dff83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dff8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2a50 0x55bc0dff8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0daab940 con 0x55bc0dfe2130 2023-10-30T13:23:41.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0dfe3060 con 0x55bc0dfe2a50 2023-10-30T13:23:41.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ddf68b0 con 0x55bc0dfe1810 2023-10-30T13:23:41.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4feaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dff8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4feaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dff8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51316/0 (socket says 172.21.15.46:51316) 2023-10-30T13:23:41.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4feaffd640 1 -- 172.21.15.46:0/2271675138 learned_addr learned my addr 172.21.15.46:0/2271675138 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:41.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4febfff640 1 --2- 172.21.15.46:0/2271675138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2a50 0x55bc0dff8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/2271675138 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0dff83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4feaffd640 1 -- 172.21.15.46:0/2271675138 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 msgr2=0x55bc0dff83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4feaffd640 1 --2- 172.21.15.46:0/2271675138 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0dff83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4feaffd640 1 -- 172.21.15.46:0/2271675138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2a50 msgr2=0x55bc0dff8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4feaffd640 1 --2- 172.21.15.46:0/2271675138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2a50 0x55bc0dff8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4feaffd640 1 -- 172.21.15.46:0/2271675138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc0dff3f50 con 0x55bc0dfe2130 2023-10-30T13:23:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4feaffd640 1 --2- 172.21.15.46:0/2271675138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dff8930 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0009010 tx=0x7f4fe0009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d525f51f896b64bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:41.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4fd1ffb640 1 -- 172.21.15.46:0/2271675138 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0003620 con 0x55bc0dfe2130 2023-10-30T13:23:41.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4fd1ffb640 1 -- 172.21.15.46:0/2271675138 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe0000e90 con 0x55bc0dfe2130 2023-10-30T13:23:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4fd1ffb640 1 -- 172.21.15.46:0/2271675138 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe00039a0 con 0x55bc0dfe2130 2023-10-30T13:23:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2271675138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 msgr2=0x55bc0dff8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/2271675138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dff8930 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0009010 tx=0x7f4fe0009240 comp rx=0 tx=0).stop 2023-10-30T13:23:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2271675138 shutdown_connections 2023-10-30T13:23:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/2271675138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2a50 0x55bc0dff8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/2271675138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2130 0x55bc0dff8930 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/2271675138 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0dff83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2271675138 >> 172.21.15.46:0/2271675138 conn(0x55bc0df97fc0 msgr2=0x55bc0dfa64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2271675138 shutdown_connections 2023-10-30T13:23:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/2271675138 wait complete. 2023-10-30T13:23:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 -- start start 2023-10-30T13:23:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:40.996+0000 7f4ff4ef3000 1 Processor -- start 2023-10-30T13:23:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0e042660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2130 0x55bc0e042ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2a50 0x55bc0e043300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0daeb1b0 con 0x55bc0dfe2a50 2023-10-30T13:23:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0dded0b0 con 0x55bc0dfe2130 2023-10-30T13:23:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ddb04d0 con 0x55bc0dfe1810 2023-10-30T13:23:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 10 client.0 _ll_register_callbacks cb 0x7f4fe00096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:23:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4febfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2a50 0x55bc0e043300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4febfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2a50 0x55bc0e043300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51326/0 (socket says 172.21.15.46:51326) 2023-10-30T13:23:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4febfff640 1 -- 172.21.15.46:0/604944671 learned_addr learned my addr 172.21.15.46:0/604944671 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4feaffd640 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2130 0x55bc0e042ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0e042660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4febfff640 1 -- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 msgr2=0x55bc0e042660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4febfff640 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0e042660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4febfff640 1 -- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2130 msgr2=0x55bc0e042ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4febfff640 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2130 0x55bc0e042ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4febfff640 1 -- 172.21.15.46:0/604944671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc0ddb3460 con 0x55bc0dfe2a50 2023-10-30T13:23:41.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0e042660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:41.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4feaffd640 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2130 0x55bc0e042ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:41.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4febfff640 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2a50 0x55bc0e043300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4001340 tx=0x7f4fe40014b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5d72bae4bc3d2c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:41.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4fca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2023-10-30T13:23:41.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4fca7fc640 1 -- 172.21.15.46:0/604944671 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe405cff0 con 0x55bc0dfe2a50 2023-10-30T13:23:41.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4fca7fc640 1 -- 172.21.15.46:0/604944671 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fe405cb60 con 0x55bc0dfe2a50 2023-10-30T13:23:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/604944671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc0dd896f0 con 0x55bc0dfe2a50 2023-10-30T13:23:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4fca7fc640 1 -- 172.21.15.46:0/604944671 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fe405cd20 con 0x55bc0dfe2a50 2023-10-30T13:23:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 10 client.4677 fetch_fsmap learned FSMap version 12 2023-10-30T13:23:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/604944671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55bc0dd99760 con 0x55bc0dfe2a50 2023-10-30T13:23:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4fca7fc640 1 -- 172.21.15.46:0/604944671 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe4076020 con 0x55bc0dfe2a50 2023-10-30T13:23:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4fca7fc640 1 -- 172.21.15.46:0/604944671 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f4fe4019020 con 0x55bc0dfe2a50 2023-10-30T13:23:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 10 client.4677 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T13:23:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 10 client.4677 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-30T13:23:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 20 client.4677 populate_metadata read hostname 'smithi046' 2023-10-30T13:23:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0e04c570 0x55bc0e044210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 4 client.4677 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T13:23:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/604944671 --> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bc0ddea720 con 0x55bc0e04c570 2023-10-30T13:23:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0e04c570 0x55bc0e044210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4feb7fe640 1 -- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0e04c570 msgr2=0x55bc0e044210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2023-10-30T13:23:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4feb7fe640 1 -- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0e04c570 msgr2=0x55bc0e044210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:23:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0e04c570 0x55bc0e044210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:23:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.000+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0e04c570 0x55bc0e044210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:23:41.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0e04c570 0x55bc0e044210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4feb7fe640 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0e04c570 0x55bc0e044210 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=860088c2822564b8 server_cookie=55705994b052824f in_seq=0 out_seq=0 2023-10-30T13:23:41.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4fca7fc640 10 client.4677 ms_handle_connect on v2:172.21.15.154:6834/3109262047 2023-10-30T13:23:41.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4fca7fc640 1 -- 172.21.15.46:0/604944671 <== mds.1 v2:172.21.15.154:6834/3109262047 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55bc0ddea720 con 0x55bc0e04c570 2023-10-30T13:23:41.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4fca7fc640 10 client.4677 handle_command_reply: tid=0 2023-10-30T13:23:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff4ef3000 1 client.4677 shutdown 2023-10-30T13:23:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/604944671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2a50 msgr2=0x55bc0e043300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2a50 0x55bc0e043300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4001340 tx=0x7f4fe40014b0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/604944671 shutdown_connections 2023-10-30T13:23:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:6834/3109262047,v1:172.21.15.154:6835/3109262047] conn(0x55bc0e04c570 0x55bc0e044210 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc0dfe2a50 0x55bc0e043300 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bc0dfe2130 0x55bc0e042ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff4ef3000 1 --2- 172.21.15.46:0/604944671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bc0dfe1810 0x55bc0e042660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4fca7fc640 0 client.4677 ms_handle_reset on v2:172.21.15.154:6834/3109262047 2023-10-30T13:23:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/604944671 >> 172.21.15.46:0/604944671 conn(0x55bc0df97fc0 msgr2=0x55bc0dffb640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/604944671 shutdown_connections 2023-10-30T13:23:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff4ef3000 1 -- 172.21.15.46:0/604944671 wait complete. 2023-10-30T13:23:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff4ef3000 20 client.4677 trim_cache size 0 max 16384 2023-10-30T13:23:41.215 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-10-30T13:23:41.215 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2", 2023-10-30T13:23:41.215 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 1, 2023-10-30T13:23:41.215 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4288, 2023-10-30T13:23:41.216 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2023-10-30T13:23:41.216 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2023-10-30T13:23:41.216 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2023-10-30T13:23:41.216 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 16.199435457, 2023-10-30T13:23:41.216 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 12, 2023-10-30T13:23:41.217 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 20, 2023-10-30T13:23:41.217 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 20, 2023-10-30T13:23:41.217 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 28.322395662000002 2023-10-30T13:23:41.217 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-10-30T13:23:41.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff1b65640 1 -- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4fd80664d0 msgr2=0x7f4fd8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff1b65640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4fd80664d0 0x7f4fd8068990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc007a60 tx=0x7f4fdc008040 comp rx=0 tx=0).stop 2023-10-30T13:23:41.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff1b65640 1 -- 172.21.15.46:0/2108406013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fec14c6b0 msgr2=0x7f4fec0c2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff1b65640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fec14c6b0 0x7f4fec0c2df0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400c7a0 tx=0x7f4fe400cc60 comp rx=0 tx=0).stop 2023-10-30T13:23:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff1b65640 1 -- 172.21.15.46:0/2108406013 shutdown_connections 2023-10-30T13:23:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff1b65640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4fd80664d0 0x7f4fd8068990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff1b65640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fec14c6b0 0x7f4fec0c2df0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff1b65640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fec151910 0x7f4fec06ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff1b65640 1 --2- 172.21.15.46:0/2108406013 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec06a970 0x7f4fec14d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff1b65640 1 -- 172.21.15.46:0/2108406013 >> 172.21.15.46:0/2108406013 conn(0x7f4fec0c7910 msgr2=0x7f4fec0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff1b65640 1 -- 172.21.15.46:0/2108406013 shutdown_connections 2023-10-30T13:23:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.204+0000 7f4ff1b65640 1 -- 172.21.15.46:0/2108406013 wait complete. 2023-10-30T13:23:41.231 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-30T13:23:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.368+0000 7f0cef8c9640 1 Processor -- start 2023-10-30T13:23:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.368+0000 7f0cef8c9640 1 -- start start 2023-10-30T13:23:41.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.368+0000 7f0cef8c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0ce80c4430 0x7f0ce80ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.368+0000 7f0cef8c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce8159830 0x7f0ce8159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.368+0000 7f0cef8c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce80cb780 con 0x7f0ce8153c60 2023-10-30T13:23:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.368+0000 7f0cef8c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce8154040 con 0x7f0ce80c4430 2023-10-30T13:23:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.368+0000 7f0cef8c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce815a2e0 con 0x7f0ce8159830 2023-10-30T13:23:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.368+0000 7f0ced665640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0ce8153c60 0x7f0ce80cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49814/0 (socket says 172.21.15.46:49814) 2023-10-30T13:23:41.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.368+0000 7f0ced665640 1 -- 172.21.15.46:0/2034111067 learned_addr learned my addr 172.21.15.46:0/2034111067 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:41.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.368+0000 7f0cede66640 1 --2- 172.21.15.46:0/2034111067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce8159830 0x7f0ce8159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.368+0000 7f0cece64640 1 --2- 172.21.15.46:0/2034111067 >> v2:172.21.15.154:3300/0 conn(0x7f0ce80c4430 0x7f0ce80ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636893351 0 0) 0x7f0ce80cb780 con 0x7f0ce8153c60 2023-10-30T13:23:41.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd4003530 con 0x7f0ce8153c60 2023-10-30T13:23:41.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 765058121 0 0) 0x7f0cd4003530 con 0x7f0ce8153c60 2023-10-30T13:23:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce8159830 msgr2=0x7f0ce8159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 --2- 172.21.15.46:0/2034111067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce8159830 0x7f0ce8159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 >> v2:172.21.15.154:3300/0 conn(0x7f0ce80c4430 msgr2=0x7f0ce80ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 --2- 172.21.15.46:0/2034111067 >> v2:172.21.15.154:3300/0 conn(0x7f0ce80c4430 0x7f0ce80ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce815a5c0 con 0x7f0ce8153c60 2023-10-30T13:23:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f0ce4001680 con 0x7f0ce8153c60 2023-10-30T13:23:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 >> v1:172.21.15.46:6789/0 conn(0x7f0ce8153c60 legacy=0x7f0ce80cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 --2- 172.21.15.46:0/2034111067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce8159830 0x7f0cd4003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 --2- 172.21.15.46:0/2034111067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cd4003f00 0x7f0cd40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 --2- 172.21.15.46:0/2034111067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4004d10 0x7f0cd4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce8154040 con 0x7f0cd4004d10 2023-10-30T13:23:41.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce815a2e0 con 0x7f0cd4003f00 2023-10-30T13:23:41.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd400b990 con 0x7f0ce8159830 2023-10-30T13:23:41.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cede66640 1 --2- 172.21.15.46:0/2034111067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4004d10 0x7f0cd4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cece64640 1 --2- 172.21.15.46:0/2034111067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cd4003f00 0x7f0cd40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0ced665640 1 --2- 172.21.15.46:0/2034111067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce8159830 0x7f0cd4003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cede66640 1 -- 172.21.15.46:0/2034111067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce8159830 msgr2=0x7f0cd4003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cede66640 1 --2- 172.21.15.46:0/2034111067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce8159830 0x7f0cd4003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cede66640 1 -- 172.21.15.46:0/2034111067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cd4003f00 msgr2=0x7f0cd40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cede66640 1 --2- 172.21.15.46:0/2034111067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cd4003f00 0x7f0cd40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.372+0000 7f0cede66640 1 -- 172.21.15.46:0/2034111067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd4003530 con 0x7f0cd4004d10 2023-10-30T13:23:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cede66640 1 --2- 172.21.15.46:0/2034111067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4004d10 0x7f0cd4005190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009fc0 tx=0x7f0ce001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d028c8b38a814465 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce00a5c70 con 0x7f0cd4004d10 2023-10-30T13:23:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce000f040 con 0x7f0cd4004d10 2023-10-30T13:23:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cdffff640 1 -- 172.21.15.46:0/2034111067 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce00ad440 con 0x7f0cd4004d10 2023-10-30T13:23:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 -- 172.21.15.46:0/2034111067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4004d10 msgr2=0x7f0cd4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 --2- 172.21.15.46:0/2034111067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4004d10 0x7f0cd4005190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009fc0 tx=0x7f0ce001cc00 comp rx=0 tx=0).stop 2023-10-30T13:23:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cede66640 1 -- 172.21.15.46:0/2034111067 reap_dead start 2023-10-30T13:23:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 -- 172.21.15.46:0/2034111067 shutdown_connections 2023-10-30T13:23:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 -- 172.21.15.46:0/2034111067 >> 172.21.15.46:0/2034111067 conn(0x7f0ce806bfe0 msgr2=0x7f0ce806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:41.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 -- 172.21.15.46:0/2034111067 shutdown_connections 2023-10-30T13:23:41.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 -- 172.21.15.46:0/2034111067 wait complete. 2023-10-30T13:23:41.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 Processor -- start 2023-10-30T13:23:41.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 -- start start 2023-10-30T13:23:41.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4004d10 0x7f0ce80ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce81541a0 0x7f0ce8154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce8146a50 0x7f0ce8146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80b41f0 con 0x7f0cd4004d10 2023-10-30T13:23:41.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce8147410 con 0x7f0ce8146a50 2023-10-30T13:23:41.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0cef8c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce81ee870 con 0x7f0ce81541a0 2023-10-30T13:23:41.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.376+0000 7f0ced665640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4004d10 0x7f0ce80ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0ced665640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4004d10 0x7f0ce80ce8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51342/0 (socket says 172.21.15.46:51342) 2023-10-30T13:23:41.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0ced665640 1 -- 172.21.15.46:0/1823985165 learned_addr learned my addr 172.21.15.46:0/1823985165 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:41.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0cede66640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce8146a50 0x7f0ce8146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0cece64640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce81541a0 0x7f0ce8154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0ced665640 1 -- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce81541a0 msgr2=0x7f0ce8154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0ced665640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce81541a0 0x7f0ce8154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0ced665640 1 -- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce8146a50 msgr2=0x7f0ce8146ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0ced665640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce8146a50 0x7f0ce8146ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0ced665640 1 -- 172.21.15.46:0/1823985165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce81eeb30 con 0x7f0cd4004d10 2023-10-30T13:23:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0cece64640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce81541a0 0x7f0ce8154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0cede66640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce8146a50 0x7f0ce8146ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0ced665640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4004d10 0x7f0ce80ce8a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400eaf0 tx=0x7f0ce4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0cddffb640 1 -- 172.21.15.46:0/1823985165 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4004a00 con 0x7f0cd4004d10 2023-10-30T13:23:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0cddffb640 1 -- 172.21.15.46:0/1823985165 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce4004b80 con 0x7f0cd4004d10 2023-10-30T13:23:41.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0cddffb640 1 -- 172.21.15.46:0/1823985165 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce400c7e0 con 0x7f0cd4004d10 2023-10-30T13:23:41.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0cef8c9640 1 -- 172.21.15.46:0/1823985165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce81eee10 con 0x7f0cd4004d10 2023-10-30T13:23:41.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0cef8c9640 1 -- 172.21.15.46:0/1823985165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce81ef3d0 con 0x7f0cd4004d10 2023-10-30T13:23:41.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0cddffb640 1 -- 172.21.15.46:0/1823985165 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ce400c940 con 0x7f0cd4004d10 2023-10-30T13:23:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.380+0000 7f0cddffb640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ccc0664d0 0x7f0ccc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cddffb640 1 -- 172.21.15.46:0/1823985165 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ce4083b30 con 0x7f0cd4004d10 2023-10-30T13:23:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cece64640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ccc0664d0 0x7f0ccc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 Processor -- start 2023-10-30T13:23:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 -- start start 2023-10-30T13:23:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cece64640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ccc0664d0 0x7f0ccc068990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80cf2a0 tx=0x7f0cd8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 0x5588b96c3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b96c4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c4a50 0x5588b96ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588b9477cc0 con 0x5588b96c3810 2023-10-30T13:23:41.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588b94da920 con 0x5588b96c4a50 2023-10-30T13:23:41.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588b94d7e50 con 0x5588b96c4130 2023-10-30T13:23:41.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0ced665640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 0x5588b96c3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0ced665640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 0x5588b96c3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51354/0 (socket says 172.21.15.46:51354) 2023-10-30T13:23:41.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0ced665640 1 -- 172.21.15.46:0/2348573076 learned_addr learned my addr 172.21.15.46:0/2348573076 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:41.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cede66640 1 --2- 172.21.15.46:0/2348573076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c4a50 0x5588b96ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cece64640 1 --2- 172.21.15.46:0/2348573076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b96c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0ced665640 1 -- 172.21.15.46:0/2348573076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 msgr2=0x5588b96c4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0ced665640 1 --2- 172.21.15.46:0/2348573076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b96c4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0ced665640 1 -- 172.21.15.46:0/2348573076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c4a50 msgr2=0x5588b96ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0ced665640 1 --2- 172.21.15.46:0/2348573076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c4a50 0x5588b96ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0ced665640 1 -- 172.21.15.46:0/2348573076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588b91cc890 con 0x5588b96c3810 2023-10-30T13:23:41.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0ced665640 1 --2- 172.21.15.46:0/2348573076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 0x5588b96c3bf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4082080 tx=0x7f0ce40554b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00943093ffe5112 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:41.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0ccaffd640 1 -- 172.21.15.46:0/2348573076 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4053010 con 0x5588b96c3810 2023-10-30T13:23:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0ccaffd640 1 -- 172.21.15.46:0/2348573076 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce4051380 con 0x5588b96c3810 2023-10-30T13:23:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0ccaffd640 1 -- 172.21.15.46:0/2348573076 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce405cd20 con 0x5588b96c3810 2023-10-30T13:23:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 -- 172.21.15.46:0/2348573076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 msgr2=0x5588b96c3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/2348573076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 0x5588b96c3bf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4082080 tx=0x7f0ce40554b0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 -- 172.21.15.46:0/2348573076 shutdown_connections 2023-10-30T13:23:41.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/2348573076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c4a50 0x5588b96ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/2348573076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b96c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/2348573076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 0x5588b96c3bf0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 -- 172.21.15.46:0/2348573076 >> 172.21.15.46:0/2348573076 conn(0x5588b9679fc0 msgr2=0x5588b96c15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:41.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 -- 172.21.15.46:0/2348573076 shutdown_connections 2023-10-30T13:23:41.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.384+0000 7f0cf2c46000 1 -- 172.21.15.46:0/2348573076 wait complete. 2023-10-30T13:23:41.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 1 -- start start 2023-10-30T13:23:41.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 1 Processor -- start 2023-10-30T13:23:41.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c3810 0x5588b96bdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4130 0x5588b96be490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4a50 0x5588b96cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588b94d3740 con 0x5588b96c4130 2023-10-30T13:23:41.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588b9492d10 con 0x5588b96c3810 2023-10-30T13:23:41.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588b94929b0 con 0x5588b96c4a50 2023-10-30T13:23:41.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 10 client.0 _ll_register_callbacks cb 0x5588b966e390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:23:41.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cece64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4130 0x5588b96be490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cece64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4130 0x5588b96be490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51358/0 (socket says 172.21.15.46:51358) 2023-10-30T13:23:41.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cece64640 1 -- 172.21.15.46:0/978136766 learned_addr learned my addr 172.21.15.46:0/978136766 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:41.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0ced665640 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c3810 0x5588b96bdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cede66640 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4a50 0x5588b96cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cece64640 1 -- 172.21.15.46:0/978136766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4a50 msgr2=0x5588b96cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cece64640 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4a50 0x5588b96cca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cece64640 1 -- 172.21.15.46:0/978136766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c3810 msgr2=0x5588b96bdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cece64640 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c3810 0x5588b96bdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cece64640 1 -- 172.21.15.46:0/978136766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce81ef3d0 con 0x5588b96c4130 2023-10-30T13:23:41.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cede66640 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4a50 0x5588b96cca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:41.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cece64640 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4130 0x5588b96be490 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8015a00 tx=0x7f0cd8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08ffa5e16bf2b0f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:41.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cc3fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2023-10-30T13:23:41.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cc3fff640 1 -- 172.21.15.46:0/978136766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8014d40 con 0x5588b96c4130 2023-10-30T13:23:41.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cc3fff640 1 -- 172.21.15.46:0/978136766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0cd8023420 con 0x5588b96c4130 2023-10-30T13:23:41.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 1 -- 172.21.15.46:0/978136766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5588b9487370 con 0x5588b96c4130 2023-10-30T13:23:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cc3fff640 1 -- 172.21.15.46:0/978136766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cd8014ea0 con 0x5588b96c4130 2023-10-30T13:23:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 10 client.4689 fetch_fsmap learned FSMap version 12 2023-10-30T13:23:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 1 -- 172.21.15.46:0/978136766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x5588b9495460 con 0x5588b96c4130 2023-10-30T13:23:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cc3fff640 1 -- 172.21.15.46:0/978136766 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5588b9487370 con 0x5588b96c4130 2023-10-30T13:23:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cc3fff640 1 -- 172.21.15.46:0/978136766 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f0cd8023620 con 0x5588b96c4130 2023-10-30T13:23:41.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 10 client.4689 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T13:23:41.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 10 client.4689 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-30T13:23:41.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 20 client.4689 populate_metadata read hostname 'smithi046' 2023-10-30T13:23:41.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.46:6834/275749119,v1:172.21.15.46:6836/275749119] conn(0x5588b96cd7f0 0x5588b96cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 4 client.4689 mds_command: new command op to 4306 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T13:23:41.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0cf2c46000 1 -- 172.21.15.46:0/978136766 --> [v2:172.21.15.46:6834/275749119,v1:172.21.15.46:6836/275749119] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588b946d2d0 con 0x5588b96cd7f0 2023-10-30T13:23:41.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.388+0000 7f0ced665640 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.46:6834/275749119,v1:172.21.15.46:6836/275749119] conn(0x5588b96cd7f0 0x5588b96cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0ced665640 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.46:6834/275749119,v1:172.21.15.46:6836/275749119] conn(0x5588b96cd7f0 0x5588b96cdbd0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eb69e42f3c7c22d server_cookie=24993897fc40ccc7 in_seq=0 out_seq=0 2023-10-30T13:23:41.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cc3fff640 10 client.4689 ms_handle_connect on v2:172.21.15.46:6834/275749119 2023-10-30T13:23:41.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cc3fff640 1 -- 172.21.15.46:0/978136766 <== mds.0 v2:172.21.15.46:6834/275749119 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5588b946d2d0 con 0x5588b96cd7f0 2023-10-30T13:23:41.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cc3fff640 10 client.4689 handle_command_reply: tid=0 2023-10-30T13:23:41.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cf2c46000 1 client.4689 shutdown 2023-10-30T13:23:41.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cf2c46000 1 -- 172.21.15.46:0/978136766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4130 msgr2=0x5588b96be490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4130 0x5588b96be490 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8015a00 tx=0x7f0cd8004740 comp rx=0 tx=0).stop 2023-10-30T13:23:41.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cf2c46000 1 -- 172.21.15.46:0/978136766 shutdown_connections 2023-10-30T13:23:41.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.46:6834/275749119,v1:172.21.15.46:6836/275749119] conn(0x5588b96cd7f0 0x5588b96cdbd0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4a50 0x5588b96cca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4130 0x5588b96be490 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/978136766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c3810 0x5588b96bdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cc3fff640 0 client.4689 ms_handle_reset on v2:172.21.15.46:6834/275749119 2023-10-30T13:23:41.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cf2c46000 1 -- 172.21.15.46:0/978136766 >> 172.21.15.46:0/978136766 conn(0x5588b9679fc0 msgr2=0x5588b96c3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:41.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cf2c46000 1 -- 172.21.15.46:0/978136766 shutdown_connections 2023-10-30T13:23:41.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cf2c46000 1 -- 172.21.15.46:0/978136766 wait complete. 2023-10-30T13:23:41.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.392+0000 7f0cf2c46000 20 client.4689 trim_cache size 0 max 16384 2023-10-30T13:23:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 Processor -- start 2023-10-30T13:23:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 -- start start 2023-10-30T13:23:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c3810 0x5588b9722650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b9722b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4a50 0x5588b97230d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588b918d940 con 0x5588b96c4a50 2023-10-30T13:23:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588b94d7740 con 0x5588b96c3810 2023-10-30T13:23:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588b94816c0 con 0x5588b96c4130 2023-10-30T13:23:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cede66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4a50 0x5588b97230d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cede66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4a50 0x5588b97230d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51364/0 (socket says 172.21.15.46:51364) 2023-10-30T13:23:41.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cede66640 1 -- 172.21.15.46:0/1006380640 learned_addr learned my addr 172.21.15.46:0/1006380640 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:41.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cece64640 1 --2- 172.21.15.46:0/1006380640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b9722b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0ced665640 1 --2- 172.21.15.46:0/1006380640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c3810 0x5588b9722650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cede66640 1 -- 172.21.15.46:0/1006380640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 msgr2=0x5588b9722b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cede66640 1 --2- 172.21.15.46:0/1006380640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b9722b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cede66640 1 -- 172.21.15.46:0/1006380640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c3810 msgr2=0x5588b9722650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cede66640 1 --2- 172.21.15.46:0/1006380640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c3810 0x5588b9722650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cede66640 1 -- 172.21.15.46:0/1006380640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588b947b760 con 0x5588b96c4a50 2023-10-30T13:23:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cede66640 1 --2- 172.21.15.46:0/1006380640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4a50 0x5588b97230d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0ce001c670 tx=0x7f0ce001b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce003a25a9d8355 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:41.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0ccb7fe640 1 -- 172.21.15.46:0/1006380640 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce00a5b30 con 0x5588b96c4a50 2023-10-30T13:23:41.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0ccb7fe640 1 -- 172.21.15.46:0/1006380640 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce00a5c90 con 0x5588b96c4a50 2023-10-30T13:23:41.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0ccb7fe640 1 -- 172.21.15.46:0/1006380640 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce00039a0 con 0x5588b96c4a50 2023-10-30T13:23:41.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 -- 172.21.15.46:0/1006380640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4a50 msgr2=0x5588b97230d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/1006380640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4a50 0x5588b97230d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0ce001c670 tx=0x7f0ce001b4a0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 -- 172.21.15.46:0/1006380640 shutdown_connections 2023-10-30T13:23:41.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/1006380640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c4a50 0x5588b97230d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/1006380640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b9722b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/1006380640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c3810 0x5588b9722650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 -- 172.21.15.46:0/1006380640 >> 172.21.15.46:0/1006380640 conn(0x5588b9679fc0 msgr2=0x5588b967f660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:41.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 -- 172.21.15.46:0/1006380640 shutdown_connections 2023-10-30T13:23:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.408+0000 7f0cf2c46000 1 -- 172.21.15.46:0/1006380640 wait complete. 2023-10-30T13:23:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 1 -- start start 2023-10-30T13:23:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 1 Processor -- start 2023-10-30T13:23:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 0x5588b971b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b971b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c4a50 0x5588b971c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588b94d0b40 con 0x5588b96c3810 2023-10-30T13:23:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588b94d88b0 con 0x5588b96c4a50 2023-10-30T13:23:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588b94924d0 con 0x5588b96c4130 2023-10-30T13:23:41.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 10 client.0 _ll_register_callbacks cb 0x7f0ce4051030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:23:41.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0ced665640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 0x5588b971b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0ced665640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 0x5588b971b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51380/0 (socket says 172.21.15.46:51380) 2023-10-30T13:23:41.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0ced665640 1 -- 172.21.15.46:0/390821676 learned_addr learned my addr 172.21.15.46:0/390821676 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:41.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cece64640 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b971b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cede66640 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c4a50 0x5588b971c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0ced665640 1 -- 172.21.15.46:0/390821676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 msgr2=0x5588b971b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0ced665640 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b971b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0ced665640 1 -- 172.21.15.46:0/390821676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c4a50 msgr2=0x5588b971c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0ced665640 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c4a50 0x5588b971c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0ced665640 1 -- 172.21.15.46:0/390821676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd4003530 con 0x5588b96c3810 2023-10-30T13:23:41.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cece64640 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b971b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:41.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0ced665640 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 0x5588b971b200 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4054e30 tx=0x7f0ce4001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2951b5b5c19b14a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:41.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cc3fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2023-10-30T13:23:41.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cc3fff640 1 -- 172.21.15.46:0/390821676 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4080040 con 0x5588b96c3810 2023-10-30T13:23:41.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cc3fff640 1 -- 172.21.15.46:0/390821676 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ce4072e40 con 0x5588b96c3810 2023-10-30T13:23:41.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 1 -- 172.21.15.46:0/390821676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5588b946b6f0 con 0x5588b96c3810 2023-10-30T13:23:41.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cc3fff640 1 -- 172.21.15.46:0/390821676 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ce4052e70 con 0x5588b96c3810 2023-10-30T13:23:41.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 10 client.4695 fetch_fsmap learned FSMap version 12 2023-10-30T13:23:41.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 1 -- 172.21.15.46:0/390821676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x5588b91cccc0 con 0x5588b96c3810 2023-10-30T13:23:41.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cc3fff640 1 -- 172.21.15.46:0/390821676 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5588b946b6f0 con 0x5588b96c3810 2023-10-30T13:23:41.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cc3fff640 1 -- 172.21.15.46:0/390821676 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f0ce4060080 con 0x5588b96c3810 2023-10-30T13:23:41.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 10 client.4695 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T13:23:41.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 10 client.4695 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-30T13:23:41.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 20 client.4695 populate_metadata read hostname 'smithi046' 2023-10-30T13:23:41.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.46:6834/275749119,v1:172.21.15.46:6836/275749119] conn(0x5588b971cb70 0x5588b971cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 4 client.4695 mds_command: new command op to 4306 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T13:23:41.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cf2c46000 1 -- 172.21.15.46:0/390821676 --> [v2:172.21.15.46:6834/275749119,v1:172.21.15.46:6836/275749119] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5588b94cc720 con 0x5588b971cb70 2023-10-30T13:23:41.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cece64640 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.46:6834/275749119,v1:172.21.15.46:6836/275749119] conn(0x5588b971cb70 0x5588b971cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cc3fff640 10 client.4695 ms_handle_connect on v2:172.21.15.46:6834/275749119 2023-10-30T13:23:41.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.412+0000 7f0cece64640 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.46:6834/275749119,v1:172.21.15.46:6836/275749119] conn(0x5588b971cb70 0x5588b971cf50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba5778589fe2ab0 server_cookie=68ee349bb9c87b53 in_seq=0 out_seq=0 2023-10-30T13:23:41.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cc3fff640 1 -- 172.21.15.46:0/390821676 <== mds.0 v2:172.21.15.46:6834/275749119 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5588b94cc720 con 0x5588b971cb70 2023-10-30T13:23:41.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cc3fff640 10 client.4695 handle_command_reply: tid=0 2023-10-30T13:23:41.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cf2c46000 1 client.4695 shutdown 2023-10-30T13:23:41.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cf2c46000 1 -- 172.21.15.46:0/390821676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 msgr2=0x5588b971b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 0x5588b971b200 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4054e30 tx=0x7f0ce4001770 comp rx=0 tx=0).stop 2023-10-30T13:23:41.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cf2c46000 1 -- 172.21.15.46:0/390821676 shutdown_connections 2023-10-30T13:23:41.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.46:6834/275749119,v1:172.21.15.46:6836/275749119] conn(0x5588b971cb70 0x5588b971cf50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588b96c4a50 0x5588b971c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588b96c4130 0x5588b971b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cf2c46000 1 --2- 172.21.15.46:0/390821676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588b96c3810 0x5588b971b200 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cf2c46000 1 -- 172.21.15.46:0/390821676 >> 172.21.15.46:0/390821676 conn(0x5588b9679fc0 msgr2=0x5588b967f660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:41.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cc3fff640 0 client.4695 ms_handle_reset on v2:172.21.15.46:6834/275749119 2023-10-30T13:23:41.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cf2c46000 1 -- 172.21.15.46:0/390821676 shutdown_connections 2023-10-30T13:23:41.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cf2c46000 1 -- 172.21.15.46:0/390821676 wait complete. 2023-10-30T13:23:41.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cf2c46000 20 client.4695 trim_cache size 0 max 16384 2023-10-30T13:23:41.441 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-10-30T13:23:41.442 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2", 2023-10-30T13:23:41.442 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2023-10-30T13:23:41.442 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4306, 2023-10-30T13:23:41.442 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2023-10-30T13:23:41.442 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2023-10-30T13:23:41.443 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2023-10-30T13:23:41.443 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 21.471007617000001, 2023-10-30T13:23:41.443 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 12, 2023-10-30T13:23:41.443 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 20, 2023-10-30T13:23:41.443 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 20, 2023-10-30T13:23:41.444 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 28.510171542999998 2023-10-30T13:23:41.444 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-10-30T13:23:41.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cef8c9640 1 -- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ccc0664d0 msgr2=0x7f0ccc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cef8c9640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ccc0664d0 0x7f0ccc068990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80cf2a0 tx=0x7f0cd8008040 comp rx=0 tx=0).stop 2023-10-30T13:23:41.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cef8c9640 1 -- 172.21.15.46:0/1823985165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4004d10 msgr2=0x7f0ce80ce8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cef8c9640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4004d10 0x7f0ce80ce8a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400eaf0 tx=0x7f0ce4005e80 comp rx=0 tx=0).stop 2023-10-30T13:23:41.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cef8c9640 1 -- 172.21.15.46:0/1823985165 shutdown_connections 2023-10-30T13:23:41.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cef8c9640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ccc0664d0 0x7f0ccc068990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cef8c9640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce8146a50 0x7f0ce8146ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cef8c9640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce81541a0 0x7f0ce8154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cef8c9640 1 --2- 172.21.15.46:0/1823985165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4004d10 0x7f0ce80ce8a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cef8c9640 1 -- 172.21.15.46:0/1823985165 >> 172.21.15.46:0/1823985165 conn(0x7f0ce806bfe0 msgr2=0x7f0ce80cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:41.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cef8c9640 1 -- 172.21.15.46:0/1823985165 shutdown_connections 2023-10-30T13:23:41.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.416+0000 7f0cef8c9640 1 -- 172.21.15.46:0/1823985165 wait complete. 2023-10-30T13:23:41.448 INFO:teuthology.run_tasks:Running task kclient... 2023-10-30T13:23:41.472 INFO:tasks.kclient:Mounting kernel clients... 2023-10-30T13:23:41.472 INFO:tasks.kclient:config is {'client.0': {}} 2023-10-30T13:23:41.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:41.473 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-10-30T13:23:41.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:41.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2023-10-30T13:23:41.549 INFO:teuthology.orchestra.run.smithi046.stderr:Cannot find device "ceph-brx" 2023-10-30T13:23:41.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:23:41.551 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-10-30T13:23:41.552 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-30T13:23:41.552 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-30T13:23:41.552 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-30T13:23:41.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5209d1d640 1 Processor -- start 2023-10-30T13:23:41.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5209d1d640 1 -- start start 2023-10-30T13:23:41.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5209d1d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f52041538a0 0x7f520415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5209d1d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f52040cf160 0x7f52040cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5209d1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52040cb800 con 0x7f52040ca0e0 2023-10-30T13:23:41.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5209d1d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520415dc30 con 0x7f52040cf160 2023-10-30T13:23:41.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5209d1d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52040cb330 con 0x7f52041538a0 2023-10-30T13:23:41.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5203fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f52040cf160 0x7f52040cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5202ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f52040ca0e0 0x7f520415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49830/0 (socket says 172.21.15.46:49830) 2023-10-30T13:23:41.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5202ffd640 1 -- 172.21.15.46:0/40009948 learned_addr learned my addr 172.21.15.46:0/40009948 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:41.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f52037fe640 1 --2- 172.21.15.46:0/40009948 >> v2:172.21.15.154:3300/0 conn(0x7f52041538a0 0x7f520415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5203fff640 1 -- 172.21.15.46:0/40009948 >> v2:172.21.15.154:3300/0 conn(0x7f52041538a0 msgr2=0x7f520415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5203fff640 1 --2- 172.21.15.46:0/40009948 >> v2:172.21.15.154:3300/0 conn(0x7f52041538a0 0x7f520415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5203fff640 1 -- 172.21.15.46:0/40009948 >> v1:172.21.15.46:6789/0 conn(0x7f52040ca0e0 legacy=0x7f520415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f5203fff640 1 -- 172.21.15.46:0/40009948 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52040cad70 con 0x7f52040cf160 2023-10-30T13:23:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.728+0000 7f52027fc640 1 -- 172.21.15.46:0/40009948 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259964699 0 0) 0x7f52040cb800 con 0x7f52040ca0e0 2023-10-30T13:23:41.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.732+0000 7f5203fff640 1 --2- 172.21.15.46:0/40009948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52040cf160 0x7f52040cf590 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f52040cfd10 tx=0x7f51fc01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d967bfd4ef2ae210 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:41.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.732+0000 7f52027fc640 1 -- 172.21.15.46:0/40009948 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc0b0440 con 0x7f52040cf160 2023-10-30T13:23:41.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.732+0000 7f52027fc640 1 -- 172.21.15.46:0/40009948 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc0b05a0 con 0x7f52040cf160 2023-10-30T13:23:41.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.732+0000 7f52027fc640 1 -- 172.21.15.46:0/40009948 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc0b0930 con 0x7f52040cf160 2023-10-30T13:23:41.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.732+0000 7f5209d1d640 1 -- 172.21.15.46:0/40009948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52040cf160 msgr2=0x7f52040cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.732+0000 7f5209d1d640 1 --2- 172.21.15.46:0/40009948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52040cf160 0x7f52040cf590 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f52040cfd10 tx=0x7f51fc01f0f0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.732+0000 7f5209d1d640 1 -- 172.21.15.46:0/40009948 shutdown_connections 2023-10-30T13:23:41.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.732+0000 7f5209d1d640 1 --2- 172.21.15.46:0/40009948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52040cf160 0x7f52040cf590 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.732+0000 7f5209d1d640 1 --2- 172.21.15.46:0/40009948 >> v2:172.21.15.154:3300/0 conn(0x7f52041538a0 0x7f520415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.732+0000 7f5209d1d640 1 -- 172.21.15.46:0/40009948 >> 172.21.15.46:0/40009948 conn(0x7f520406bfe0 msgr2=0x7f520406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.732+0000 7f5209d1d640 1 -- 172.21.15.46:0/40009948 shutdown_connections 2023-10-30T13:23:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.732+0000 7f5209d1d640 1 -- 172.21.15.46:0/40009948 wait complete. 2023-10-30T13:23:41.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5209d1d640 1 Processor -- start 2023-10-30T13:23:41.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5209d1d640 1 -- start start 2023-10-30T13:23:41.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5209d1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52040ca0e0 0x7f52040cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5209d1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52040cf160 0x7f52041e3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5209d1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52041538a0 0x7f52041e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5209d1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52041e31d0 con 0x7f52040cf160 2023-10-30T13:23:41.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5209d1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5204148450 con 0x7f52040ca0e0 2023-10-30T13:23:41.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5209d1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5204148730 con 0x7f52041538a0 2023-10-30T13:23:41.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5202ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52040cf160 0x7f52041e3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5202ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52040cf160 0x7f52041e3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51398/0 (socket says 172.21.15.46:51398) 2023-10-30T13:23:41.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5202ffd640 1 -- 172.21.15.46:0/3970909075 learned_addr learned my addr 172.21.15.46:0/3970909075 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:41.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5203fff640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52041538a0 0x7f52041e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f52037fe640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52040ca0e0 0x7f52040cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5202ffd640 1 -- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52041538a0 msgr2=0x7f52041e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5202ffd640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52041538a0 0x7f52041e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5202ffd640 1 -- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52040ca0e0 msgr2=0x7f52040cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5202ffd640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52040ca0e0 0x7f52040cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5202ffd640 1 -- 172.21.15.46:0/3970909075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc009a70 con 0x7f52040cf160 2023-10-30T13:23:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f52037fe640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52040ca0e0 0x7f52040cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5202ffd640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52040cf160 0x7f52041e3660 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f51f800c480 tx=0x7f51f800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:41.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f51ebfff640 1 -- 172.21.15.46:0/3970909075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f80074e0 con 0x7f52040cf160 2023-10-30T13:23:41.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f51ebfff640 1 -- 172.21.15.46:0/3970909075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f8007660 con 0x7f52040cf160 2023-10-30T13:23:41.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5209d1d640 1 -- 172.21.15.46:0/3970909075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52041e2b80 con 0x7f52040cf160 2023-10-30T13:23:41.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f51ebfff640 1 -- 172.21.15.46:0/3970909075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8004a30 con 0x7f52040cf160 2023-10-30T13:23:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.736+0000 7f5209d1d640 1 -- 172.21.15.46:0/3970909075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52041e57a0 con 0x7f52040cf160 2023-10-30T13:23:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.740+0000 7f5209d1d640 1 -- 172.21.15.46:0/3970909075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52040b2a50 con 0x7f52040cf160 2023-10-30T13:23:41.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.740+0000 7f51ebfff640 1 -- 172.21.15.46:0/3970909075 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51f8004b90 con 0x7f52040cf160 2023-10-30T13:23:41.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.740+0000 7f51ebfff640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51f00664a0 0x7f51f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:41.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.740+0000 7f51ebfff640 1 -- 172.21.15.46:0/3970909075 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51f8083420 con 0x7f52040cf160 2023-10-30T13:23:41.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.740+0000 7f52037fe640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51f00664a0 0x7f51f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:41.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.740+0000 7f52037fe640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51f00664a0 0x7f51f0068960 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009bb0 tx=0x7f51f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:41.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.744+0000 7f51ebfff640 1 -- 172.21.15.46:0/3970909075 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51f8054320 con 0x7f52040cf160 2023-10-30T13:23:41.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.940+0000 7f5209d1d640 1 -- 172.21.15.46:0/3970909075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5204159850 con 0x7f52040cf160 2023-10-30T13:23:41.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.940+0000 7f51ebfff640 1 -- 172.21.15.46:0/3970909075 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v12) v1 ==== 53+0+73 (secure 0 0 0) 0x7f51f8018060 con 0x7f52040cf160 2023-10-30T13:23:41.949 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-30T13:23:41.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.944+0000 7f5209d1d640 1 -- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51f00664a0 msgr2=0x7f51f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.944+0000 7f5209d1d640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51f00664a0 0x7f51f0068960 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009bb0 tx=0x7f51f4008040 comp rx=0 tx=0).stop 2023-10-30T13:23:41.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.944+0000 7f5209d1d640 1 -- 172.21.15.46:0/3970909075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52040cf160 msgr2=0x7f52041e3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:41.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.944+0000 7f5209d1d640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52040cf160 0x7f52041e3660 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f51f800c480 tx=0x7f51f800c940 comp rx=0 tx=0).stop 2023-10-30T13:23:41.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.944+0000 7f5209d1d640 1 -- 172.21.15.46:0/3970909075 shutdown_connections 2023-10-30T13:23:41.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.944+0000 7f5209d1d640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51f00664a0 0x7f51f0068960 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.944+0000 7f5209d1d640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52041538a0 0x7f52041e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.944+0000 7f5209d1d640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52040cf160 0x7f52041e3660 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.944+0000 7f5209d1d640 1 --2- 172.21.15.46:0/3970909075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52040ca0e0 0x7f52040cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:41.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.944+0000 7f5209d1d640 1 -- 172.21.15.46:0/3970909075 >> 172.21.15.46:0/3970909075 conn(0x7f520406bfe0 msgr2=0x7f5204066390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:41.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.944+0000 7f5209d1d640 1 -- 172.21.15.46:0/3970909075 shutdown_connections 2023-10-30T13:23:41.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:41.944+0000 7f5209d1d640 1 -- 172.21.15.46:0/3970909075 wait complete. 2023-10-30T13:23:41.969 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-30T13:23:41.970 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-30T13:23:41.970 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2023-10-30T13:23:41.970 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-30T13:23:41.971 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-30T13:23:41.971 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-30T13:23:41.971 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-30T13:23:41.971 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-30T13:23:41.972 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-30T13:23:41.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:41.972 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2023-10-30T13:23:41.982 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-30T13:23:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-30T13:23:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2023-10-30T13:23:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-10-30T13:23:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2023-10-30T13:23:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-10-30T13:23:41.983 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T13:23:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2023-10-30T13:23:41.983 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2023-10-30T13:23:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T13:23:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2023-10-30T13:23:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2023-10-30T13:23:41.984 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-30T13:23:41.984 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2023-10-30T13:23:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2023-10-30T13:23:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-10-30T13:23:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42397sec preferred_lft 42397sec 2023-10-30T13:23:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2023-10-30T13:23:41.985 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-10-30T13:23:41.986 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T13:23:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2023-10-30T13:23:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2023-10-30T13:23:41.986 INFO:teuthology.orchestra.run.smithi046.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-30T13:23:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 02:42:3a:5d:8b:55 brd ff:ff:ff:ff:ff:ff 2023-10-30T13:23:41.986 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-30T13:23:41.987 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-10-30T13:23:41.988 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-30T13:23:41.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:41.988 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-10-30T13:23:41.989 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-10-30T13:23:41.989 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2023-10-30T13:23:41.989 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2023-10-30T13:23:41.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2023-10-30T13:23:41.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-30T13:23:41.990 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-10-30T13:23:42.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:42.101 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-30T13:23:42.122 INFO:teuthology.orchestra.run.smithi046.stdout:1 2023-10-30T13:23:42.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:42.124 DEBUG:teuthology.orchestra.run.smithi046:> route 2023-10-30T13:23:42.189 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2023-10-30T13:23:42.190 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-30T13:23:42.190 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-10-30T13:23:42.190 INFO:teuthology.orchestra.run.smithi046.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-30T13:23:42.190 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-10-30T13:23:42.191 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-30T13:23:42.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:42.191 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-10-30T13:23:42.192 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-10-30T13:23:42.192 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-30T13:23:42.192 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-30T13:23:42.193 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-30T13:23:42.193 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-10-30T13:23:42.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:42.286 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-10-30T13:23:42.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:42.339 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2023-10-30T13:23:42.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:42.390 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-10-30T13:23:42.390 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-10-30T13:23:42.391 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-30T13:23:42.391 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-30T13:23:42.391 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-10-30T13:23:42.473 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-30T13:23:42.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:42.474 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-10-30T13:23:42.474 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-10-30T13:23:42.474 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-30T13:23:42.474 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-30T13:23:42.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-30T13:23:42.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-30T13:23:42.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-30T13:23:42.476 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-10-30T13:23:42.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:42.638 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-10-30T13:23:42.638 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-10-30T13:23:42.639 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2023-10-30T13:23:42.639 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2023-10-30T13:23:42.639 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-10-30T13:23:42.681 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-30T13:23:42.681 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-30T13:23:42.738 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-30T13:23:42.738 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-30T13:23:42.739 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-30T13:23:42.789 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-30T13:23:42.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:42.790 DEBUG:teuthology.orchestra.run.smithi046:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T13:23:42.897 INFO:teuthology.orchestra.run.smithi046.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T13:23:42.897 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T13:23:42.897 INFO:teuthology.orchestra.run.smithi046.stdout:invalid new device string format 2023-10-30T13:23:43.011 INFO:teuthology.orchestra.run.smithi046.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T13:23:43.012 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T13:23:43.012 INFO:teuthology.orchestra.run.smithi046.stdout:invalid new device string format 2023-10-30T13:23:43.012 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: resolved to: "172.21.15.46:3300,172.21.15.154:3300,172.21.15.154:3301" 2023-10-30T13:23:43.012 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-30T13:23:43.024 INFO:teuthology.orchestra.run.smithi046.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T13:23:43.025 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T13:23:43.025 INFO:teuthology.orchestra.run.smithi046.stdout:invalid new device string format 2023-10-30T13:23:43.025 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: resolved to: "172.21.15.46:3300,172.21.15.154:3300,172.21.15.154:3301" 2023-10-30T13:23:43.025 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-30T13:23:43.026 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: resolved to: "172.21.15.46:6789,172.21.15.154:6789,172.21.15.154:6790" 2023-10-30T13:23:43.026 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.46:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2023-10-30T13:23:43.026 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2" will pass to kernel 2023-10-30T13:23:43.028 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-30T13:23:43.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:43.029 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-30T13:23:43.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:43.052 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T13:23:43.122 INFO:teuthology.orchestra.run.smithi046.stdout:client4586 2023-10-30T13:23:43.123 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:23:43.123 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T13:23:43.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:43.175 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T13:23:43.238 INFO:teuthology.orchestra.run.smithi046.stdout:client4586 2023-10-30T13:23:43.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:43.239 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T13:23:43.298 INFO:teuthology.orchestra.run.smithi046.stdout:client4586 2023-10-30T13:23:43.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:23:43.298 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2.client4586/status') 2023-10-30T13:23:43.370 INFO:teuthology.orchestra.run.smithi046.stdout:instance: client.4586 (3)172.21.15.46:0/1676631590 2023-10-30T13:23:43.370 INFO:teuthology.orchestra.run.smithi046.stdout:blocklisted: false 2023-10-30T13:23:43.371 INFO:teuthology.orchestra.run.smithi046.stderr:0+1 records in 2023-10-30T13:23:43.371 INFO:teuthology.orchestra.run.smithi046.stderr:0+1 records out 2023-10-30T13:23:43.371 INFO:teuthology.orchestra.run.smithi046.stderr:70 bytes copied, 5.1995e-05 s, 1.3 MB/s 2023-10-30T13:23:43.373 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-10-30T13:23:43.385 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:43.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f381071d640 1 Processor -- start 2023-10-30T13:23:43.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f381071d640 1 -- start start 2023-10-30T13:23:43.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f381071d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38081538a0 0x7f380815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f381071d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f38080ca0e0 0x7f380815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f381071d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f38080cf160 0x7f38080cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f381071d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38080cb800 con 0x7f38080ca0e0 2023-10-30T13:23:43.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f381071d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380815dc30 con 0x7f38080cf160 2023-10-30T13:23:43.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f381071d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38080cb330 con 0x7f38081538a0 2023-10-30T13:23:43.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f380dcb8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f38080ca0e0 0x7f380815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f380dcb8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f38080ca0e0 0x7f380815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51406/0 (socket says 172.21.15.46:51406) 2023-10-30T13:23:43.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f380dcb8640 1 -- 172.21.15.46:0/3598088539 learned_addr learned my addr 172.21.15.46:0/3598088539 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:43.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f380e4b9640 1 --2- 172.21.15.46:0/3598088539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38081538a0 0x7f380815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f380ecba640 1 --2- 172.21.15.46:0/3598088539 >> v2:172.21.15.154:3300/0 conn(0x7f38080cf160 0x7f38080cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f380dcb8640 1 -- 172.21.15.46:0/3598088539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38081538a0 msgr2=0x7f380815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:43.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f380dcb8640 1 --2- 172.21.15.46:0/3598088539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38081538a0 0x7f380815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f380dcb8640 1 -- 172.21.15.46:0/3598088539 >> v2:172.21.15.154:3300/0 conn(0x7f38080cf160 msgr2=0x7f38080cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:43.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f380dcb8640 1 --2- 172.21.15.46:0/3598088539 >> v2:172.21.15.154:3300/0 conn(0x7f38080cf160 0x7f38080cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.552+0000 7f380dcb8640 1 -- 172.21.15.46:0/3598088539 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38080cad70 con 0x7f38080ca0e0 2023-10-30T13:23:43.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f380dcb8640 1 --2- 172.21.15.46:0/3598088539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38080ca0e0 0x7f380815a090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f380806cb00 tx=0x7f37f801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b756de4272db0ff9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:43.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f380d4b7640 1 -- 172.21.15.46:0/3598088539 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f80a89b0 con 0x7f38080ca0e0 2023-10-30T13:23:43.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f380d4b7640 1 -- 172.21.15.46:0/3598088539 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f80a8b10 con 0x7f38080ca0e0 2023-10-30T13:23:43.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f380d4b7640 1 -- 172.21.15.46:0/3598088539 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f80a8ea0 con 0x7f38080ca0e0 2023-10-30T13:23:43.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f381071d640 1 -- 172.21.15.46:0/3598088539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38080ca0e0 msgr2=0x7f380815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:43.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f381071d640 1 --2- 172.21.15.46:0/3598088539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38080ca0e0 0x7f380815a090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f380806cb00 tx=0x7f37f801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f381071d640 1 -- 172.21.15.46:0/3598088539 shutdown_connections 2023-10-30T13:23:43.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f381071d640 1 --2- 172.21.15.46:0/3598088539 >> v2:172.21.15.154:3300/0 conn(0x7f38080cf160 0x7f38080cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f381071d640 1 --2- 172.21.15.46:0/3598088539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38080ca0e0 0x7f380815a090 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f381071d640 1 --2- 172.21.15.46:0/3598088539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38081538a0 0x7f380815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f381071d640 1 -- 172.21.15.46:0/3598088539 >> 172.21.15.46:0/3598088539 conn(0x7f380806bfe0 msgr2=0x7f380806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:43.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f381071d640 1 -- 172.21.15.46:0/3598088539 shutdown_connections 2023-10-30T13:23:43.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.556+0000 7f381071d640 1 -- 172.21.15.46:0/3598088539 wait complete. 2023-10-30T13:23:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f381071d640 1 Processor -- start 2023-10-30T13:23:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f381071d640 1 -- start start 2023-10-30T13:23:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f381071d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38080ca0e0 0x7f380815d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f381071d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38080cf160 0x7f380814a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f381071d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38081538a0 0x7f38081f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f381071d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38081f2fd0 con 0x7f38080ca0e0 2023-10-30T13:23:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f381071d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380815ad70 con 0x7f38081538a0 2023-10-30T13:23:43.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f381071d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380814bab0 con 0x7f38080cf160 2023-10-30T13:23:43.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f380e4b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38080ca0e0 0x7f380815d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f380e4b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38080ca0e0 0x7f380815d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51408/0 (socket says 172.21.15.46:51408) 2023-10-30T13:23:43.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f380e4b9640 1 -- 172.21.15.46:0/1624247424 learned_addr learned my addr 172.21.15.46:0/1624247424 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:43.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f380ecba640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38081538a0 0x7f38081f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f380dcb8640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38080cf160 0x7f380814a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f380e4b9640 1 -- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38080cf160 msgr2=0x7f380814a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:43.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f380e4b9640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38080cf160 0x7f380814a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f380e4b9640 1 -- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38081538a0 msgr2=0x7f38081f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:43.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f380e4b9640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38081538a0 0x7f38081f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f380e4b9640 1 -- 172.21.15.46:0/1624247424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f80094d0 con 0x7f38080ca0e0 2023-10-30T13:23:43.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f380e4b9640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38080ca0e0 0x7f380815d410 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f380400e8a0 tx=0x7f380400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:43.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f37feffd640 1 -- 172.21.15.46:0/1624247424 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3804009510 con 0x7f38080ca0e0 2023-10-30T13:23:43.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f37feffd640 1 -- 172.21.15.46:0/1624247424 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380400c940 con 0x7f38080ca0e0 2023-10-30T13:23:43.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f37feffd640 1 -- 172.21.15.46:0/1624247424 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380400ccb0 con 0x7f38080ca0e0 2023-10-30T13:23:43.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f381071d640 1 -- 172.21.15.46:0/1624247424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38080ce130 con 0x7f38080ca0e0 2023-10-30T13:23:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.560+0000 7f381071d640 1 -- 172.21.15.46:0/1624247424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38080ce380 con 0x7f38080ca0e0 2023-10-30T13:23:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.564+0000 7f381071d640 1 -- 172.21.15.46:0/1624247424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38080b2a50 con 0x7f38080ca0e0 2023-10-30T13:23:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.564+0000 7f37feffd640 1 -- 172.21.15.46:0/1624247424 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3804013470 con 0x7f38080ca0e0 2023-10-30T13:23:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.564+0000 7f37feffd640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f37ec0664d0 0x7f37ec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.564+0000 7f37feffd640 1 -- 172.21.15.46:0/1624247424 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3804084060 con 0x7f38080ca0e0 2023-10-30T13:23:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.564+0000 7f380dcb8640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f37ec0664d0 0x7f37ec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.564+0000 7f380dcb8640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f37ec0664d0 0x7f37ec068990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f37f80b2000 tx=0x7f37f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.568+0000 7f37feffd640 1 -- 172.21.15.46:0/1624247424 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f38040b5090 con 0x7f38080ca0e0 2023-10-30T13:23:43.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:23:43.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:23:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.728+0000 7f381071d640 1 -- 172.21.15.46:0/1624247424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f380804e170 con 0x7f38080ca0e0 2023-10-30T13:23:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f37feffd640 1 -- 172.21.15.46:0/1624247424 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f3804021090 con 0x7f38080ca0e0 2023-10-30T13:23:43.738 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:43.738 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:20.665723+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:43.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f381071d640 1 -- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f37ec0664d0 msgr2=0x7f37ec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:43.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f381071d640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f37ec0664d0 0x7f37ec068990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f37f80b2000 tx=0x7f37f8008040 comp rx=0 tx=0).stop 2023-10-30T13:23:43.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f381071d640 1 -- 172.21.15.46:0/1624247424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38080ca0e0 msgr2=0x7f380815d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:43.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f381071d640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38080ca0e0 0x7f380815d410 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f380400e8a0 tx=0x7f380400eca0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f381071d640 1 -- 172.21.15.46:0/1624247424 shutdown_connections 2023-10-30T13:23:43.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f381071d640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f37ec0664d0 0x7f37ec068990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f381071d640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38081538a0 0x7f38081f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f381071d640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38080cf160 0x7f380814a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f381071d640 1 --2- 172.21.15.46:0/1624247424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38080ca0e0 0x7f380815d410 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f381071d640 1 -- 172.21.15.46:0/1624247424 >> 172.21.15.46:0/1624247424 conn(0x7f380806bfe0 msgr2=0x7f38081483a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:43.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f381071d640 1 -- 172.21.15.46:0/1624247424 shutdown_connections 2023-10-30T13:23:43.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.732+0000 7f381071d640 1 -- 172.21.15.46:0/1624247424 wait complete. 2023-10-30T13:23:43.759 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:23:06.788949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:23:17.648827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:23:18.326967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:23:43.760 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:23:43.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6cb3e640 1 Processor -- start 2023-10-30T13:23:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6cb3e640 1 -- start start 2023-10-30T13:23:43.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6cb3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7c680c5810 0x7f7c680c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6cb3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c680c6cb0 con 0x7f7c680c6130 2023-10-30T13:23:43.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6cb3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c680c6620 con 0x7f7c6806a980 2023-10-30T13:23:43.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6cb3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c68071d90 con 0x7f7c680c5810 2023-10-30T13:23:43.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c65d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7c680c6130 0x7f7c6806a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49842/0 (socket says 172.21.15.46:49842) 2023-10-30T13:23:43.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c65d9b640 1 -- 172.21.15.46:0/133351449 learned_addr learned my addr 172.21.15.46:0/133351449 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:43.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6659c640 1 --2- 172.21.15.46:0/133351449 >> v2:172.21.15.154:3300/0 conn(0x7f7c680c5810 0x7f7c680c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046823974 0 0) 0x7f7c680c6cb0 con 0x7f7c680c6130 2023-10-30T13:23:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c50003530 con 0x7f7c680c6130 2023-10-30T13:23:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 163037450 0 0) 0x7f7c50003530 con 0x7f7c680c6130 2023-10-30T13:23:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 >> v2:172.21.15.154:3300/0 conn(0x7f7c680c5810 msgr2=0x7f7c680c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 --2- 172.21.15.46:0/133351449 >> v2:172.21.15.154:3300/0 conn(0x7f7c680c5810 0x7f7c680c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 >> v1:172.21.15.154:6789/0 conn(0x7f7c6806a980 legacy=0x7f7c68161f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:43.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c681582f0 con 0x7f7c680c6130 2023-10-30T13:23:43.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f7c5c001680 con 0x7f7c680c6130 2023-10-30T13:23:43.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 >> v1:172.21.15.46:6789/0 conn(0x7f7c680c6130 legacy=0x7f7c6806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:43.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 --2- 172.21.15.46:0/133351449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c50003980 0x7f7c50003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 --2- 172.21.15.46:0/133351449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c500047d0 0x7f7c50004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 --2- 172.21.15.46:0/133351449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c500052b0 0x7f7c5000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5000c150 con 0x7f7c500047d0 2023-10-30T13:23:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5000c2c0 con 0x7f7c50003980 2023-10-30T13:23:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5000c480 con 0x7f7c500052b0 2023-10-30T13:23:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c65d9b640 1 --2- 172.21.15.46:0/133351449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c500047d0 0x7f7c50004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c6659c640 1 --2- 172.21.15.46:0/133351449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c50003980 0x7f7c50003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c66d9d640 1 --2- 172.21.15.46:0/133351449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c500052b0 0x7f7c5000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c65d9b640 1 -- 172.21.15.46:0/133351449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c500052b0 msgr2=0x7f7c5000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:43.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c65d9b640 1 --2- 172.21.15.46:0/133351449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c500052b0 0x7f7c5000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c65d9b640 1 -- 172.21.15.46:0/133351449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c50003980 msgr2=0x7f7c50003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:43.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c65d9b640 1 --2- 172.21.15.46:0/133351449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c50003980 0x7f7c50003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c65d9b640 1 -- 172.21.15.46:0/133351449 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c50003530 con 0x7f7c500047d0 2023-10-30T13:23:43.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c66d9d640 1 --2- 172.21.15.46:0/133351449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c500052b0 0x7f7c5000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:43.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.944+0000 7f7c66d9d640 1 -- 172.21.15.46:0/133351449 reap_dead start 2023-10-30T13:23:43.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.948+0000 7f7c65d9b640 1 --2- 172.21.15.46:0/133351449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c500047d0 0x7f7c50004c50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7c6806f5d0 tx=0x7f7c5c01cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d617a55bfeff11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:43.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.948+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c005ae0 con 0x7f7c500047d0 2023-10-30T13:23:43.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.948+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5c005c40 con 0x7f7c500047d0 2023-10-30T13:23:43.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.948+0000 7f7c6559a640 1 -- 172.21.15.46:0/133351449 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0a59f0 con 0x7f7c500047d0 2023-10-30T13:23:43.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.948+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/133351449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c500047d0 msgr2=0x7f7c50004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:43.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.948+0000 7f7c6cb3e640 1 --2- 172.21.15.46:0/133351449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c500047d0 0x7f7c50004c50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7c6806f5d0 tx=0x7f7c5c01cbc0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.948+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/133351449 shutdown_connections 2023-10-30T13:23:43.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.948+0000 7f7c6cb3e640 1 --2- 172.21.15.46:0/133351449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c500047d0 0x7f7c50004c50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.948+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/133351449 >> 172.21.15.46:0/133351449 conn(0x7f7c680c7910 msgr2=0x7f7c680c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.948+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/133351449 shutdown_connections 2023-10-30T13:23:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.948+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/133351449 wait complete. 2023-10-30T13:23:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c6cb3e640 1 Processor -- start 2023-10-30T13:23:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c6cb3e640 1 -- start start 2023-10-30T13:23:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c6cb3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c500047d0 0x7f7c681f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c6cb3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c680c2ff0 0x7f7c680c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c6cb3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c681484b0 0x7f7c68149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c6cb3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c681f2ba0 con 0x7f7c680c2ff0 2023-10-30T13:23:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c6cb3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680c2a30 con 0x7f7c500047d0 2023-10-30T13:23:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c6cb3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680c2d10 con 0x7f7c681484b0 2023-10-30T13:23:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c65d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c680c2ff0 0x7f7c680c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c65d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c680c2ff0 0x7f7c680c3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51428/0 (socket says 172.21.15.46:51428) 2023-10-30T13:23:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c65d9b640 1 -- 172.21.15.46:0/912258767 learned_addr learned my addr 172.21.15.46:0/912258767 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c6659c640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c500047d0 0x7f7c681f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c66d9d640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c681484b0 0x7f7c68149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c65d9b640 1 -- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c681484b0 msgr2=0x7f7c68149160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:43.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c65d9b640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c681484b0 0x7f7c68149160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c65d9b640 1 -- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c500047d0 msgr2=0x7f7c681f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:43.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c65d9b640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c500047d0 0x7f7c681f2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:43.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c65d9b640 1 -- 172.21.15.46:0/912258767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c681f24e0 con 0x7f7c680c2ff0 2023-10-30T13:23:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c66d9d640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c681484b0 0x7f7c68149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c65d9b640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c680c2ff0 0x7f7c680c3f50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c001940 tx=0x7f7c50005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c56ffd640 1 -- 172.21.15.46:0/912258767 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0b4070 con 0x7f7c680c2ff0 2023-10-30T13:23:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c56ffd640 1 -- 172.21.15.46:0/912258767 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5c001500 con 0x7f7c680c2ff0 2023-10-30T13:23:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/912258767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c681f27c0 con 0x7f7c680c2ff0 2023-10-30T13:23:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.952+0000 7f7c56ffd640 1 -- 172.21.15.46:0/912258767 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0a57b0 con 0x7f7c680c2ff0 2023-10-30T13:23:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.956+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/912258767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c680c25f0 con 0x7f7c680c2ff0 2023-10-30T13:23:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.956+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/912258767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c681f2ce0 con 0x7f7c680c2ff0 2023-10-30T13:23:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.956+0000 7f7c56ffd640 1 -- 172.21.15.46:0/912258767 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7c5c0a5910 con 0x7f7c680c2ff0 2023-10-30T13:23:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.956+0000 7f7c56ffd640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7c480664d0 0x7f7c48068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.956+0000 7f7c56ffd640 1 -- 172.21.15.46:0/912258767 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c5c11d8d0 con 0x7f7c680c2ff0 2023-10-30T13:23:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.956+0000 7f7c6659c640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7c480664d0 0x7f7c48068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.960+0000 7f7c6659c640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7c480664d0 0x7f7c48068990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7c58009c40 tx=0x7f7c58006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:43.960+0000 7f7c56ffd640 1 -- 172.21.15.46:0/912258767 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7c681f2ce0 con 0x7f7c680c2ff0 2023-10-30T13:23:44.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.116+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/912258767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7c681f2ce0 con 0x7f7c680c2ff0 2023-10-30T13:23:44.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.116+0000 7f7c56ffd640 1 -- 172.21.15.46:0/912258767 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7c681f2ce0 con 0x7f7c680c2ff0 2023-10-30T13:23:44.124 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-10-30T13:23:44.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.116+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7c480664d0 msgr2=0x7f7c48068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.116+0000 7f7c6cb3e640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7c480664d0 0x7f7c48068990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7c58009c40 tx=0x7f7c58006f90 comp rx=0 tx=0).stop 2023-10-30T13:23:44.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.116+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/912258767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c680c2ff0 msgr2=0x7f7c680c3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.116+0000 7f7c6cb3e640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c680c2ff0 0x7f7c680c3f50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c001940 tx=0x7f7c50005800 comp rx=0 tx=0).stop 2023-10-30T13:23:44.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.120+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/912258767 shutdown_connections 2023-10-30T13:23:44.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.120+0000 7f7c6cb3e640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7c480664d0 0x7f7c48068990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.120+0000 7f7c6cb3e640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c681484b0 0x7f7c68149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.120+0000 7f7c6cb3e640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c680c2ff0 0x7f7c680c3f50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.120+0000 7f7c6cb3e640 1 --2- 172.21.15.46:0/912258767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c500047d0 0x7f7c681f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.120+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/912258767 >> 172.21.15.46:0/912258767 conn(0x7f7c680c7910 msgr2=0x7f7c6814f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:44.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.120+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/912258767 shutdown_connections 2023-10-30T13:23:44.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.120+0000 7f7c6cb3e640 1 -- 172.21.15.46:0/912258767 wait complete. 2023-10-30T13:23:44.144 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:23:44.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f9628faa640 1 Processor -- start 2023-10-30T13:23:44.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f9628faa640 1 -- start start 2023-10-30T13:23:44.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f9628faa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9624153980 0x7f96240caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f9628faa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962415a920 con 0x7f9624159930 2023-10-30T13:23:44.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f9628faa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96240cb930 con 0x7f96240cb1e0 2023-10-30T13:23:44.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f9628faa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9624159dc0 con 0x7f9624153980 2023-10-30T13:23:44.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f961bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9624159930 0x7f96240cf1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49856/0 (socket says 172.21.15.46:49856) 2023-10-30T13:23:44.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f961bfff640 1 -- 172.21.15.46:0/66319590 learned_addr learned my addr 172.21.15.46:0/66319590 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:44.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f962259c640 1 --2- 172.21.15.46:0/66319590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9624153980 0x7f96240caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f9621d9b640 1 -- 172.21.15.46:0/66319590 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430751570 0 0) 0x7f962415a920 con 0x7f9624159930 2023-10-30T13:23:44.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f9621d9b640 1 -- 172.21.15.46:0/66319590 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9610003530 con 0x7f9624159930 2023-10-30T13:23:44.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f9621d9b640 1 -- 172.21.15.46:0/66319590 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717905956 0 0) 0x7f96240cb930 con 0x7f96240cb1e0 2023-10-30T13:23:44.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f9621d9b640 1 -- 172.21.15.46:0/66319590 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f962415a920 con 0x7f96240cb1e0 2023-10-30T13:23:44.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f9621d9b640 1 -- 172.21.15.46:0/66319590 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4179105268 0 0) 0x7f9610003530 con 0x7f9624159930 2023-10-30T13:23:44.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f962259c640 1 -- 172.21.15.46:0/66319590 >> v1:172.21.15.154:6789/0 conn(0x7f96240cb1e0 legacy=0x7f962415d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f962259c640 1 -- 172.21.15.46:0/66319590 >> v1:172.21.15.46:6789/0 conn(0x7f9624159930 legacy=0x7f96240cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.284+0000 7f962259c640 1 -- 172.21.15.46:0/66319590 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962415a080 con 0x7f9624153980 2023-10-30T13:23:44.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.288+0000 7f962259c640 1 --2- 172.21.15.46:0/66319590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9624153980 0x7f96240caca0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f96140097f0 tx=0x7f961401c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=290566239ed45e23 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:44.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.288+0000 7f9621d9b640 1 -- 172.21.15.46:0/66319590 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c001680 con 0x7f9624153980 2023-10-30T13:23:44.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.288+0000 7f9621d9b640 1 -- 172.21.15.46:0/66319590 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96140a7050 con 0x7f9624153980 2023-10-30T13:23:44.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.288+0000 7f9621d9b640 1 -- 172.21.15.46:0/66319590 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96140ab620 con 0x7f9624153980 2023-10-30T13:23:44.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.288+0000 7f9628faa640 1 -- 172.21.15.46:0/66319590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9624153980 msgr2=0x7f96240caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.288+0000 7f9628faa640 1 --2- 172.21.15.46:0/66319590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9624153980 0x7f96240caca0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f96140097f0 tx=0x7f961401c3b0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.288+0000 7f9628faa640 1 -- 172.21.15.46:0/66319590 shutdown_connections 2023-10-30T13:23:44.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.288+0000 7f9628faa640 1 --2- 172.21.15.46:0/66319590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9624153980 0x7f96240caca0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.288+0000 7f9628faa640 1 -- 172.21.15.46:0/66319590 >> 172.21.15.46:0/66319590 conn(0x7f962406c0f0 msgr2=0x7f962406c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:44.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.288+0000 7f9628faa640 1 -- 172.21.15.46:0/66319590 shutdown_connections 2023-10-30T13:23:44.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.288+0000 7f9628faa640 1 -- 172.21.15.46:0/66319590 wait complete. 2023-10-30T13:23:44.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f9628faa640 1 Processor -- start 2023-10-30T13:23:44.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f9628faa640 1 -- start start 2023-10-30T13:23:44.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f9628faa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96240cb1e0 0x7f9624152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f9628faa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9624153980 0x7f9624148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f9628faa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96241f1fd0 0x7f96241f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f9628faa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96241f29c0 con 0x7f9624153980 2023-10-30T13:23:44.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f9628faa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96240c86a0 con 0x7f96241f1fd0 2023-10-30T13:23:44.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f9628faa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96240c8940 con 0x7f96240cb1e0 2023-10-30T13:23:44.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f961bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9624153980 0x7f9624148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f961bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9624153980 0x7f9624148bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51440/0 (socket says 172.21.15.46:51440) 2023-10-30T13:23:44.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f961bfff640 1 -- 172.21.15.46:0/3942162944 learned_addr learned my addr 172.21.15.46:0/3942162944 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:44.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f9622d9d640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96241f1fd0 0x7f96241f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f962259c640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96240cb1e0 0x7f9624152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f961bfff640 1 -- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96240cb1e0 msgr2=0x7f9624152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f961bfff640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96240cb1e0 0x7f9624152830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f961bfff640 1 -- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96241f1fd0 msgr2=0x7f96241f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f961bfff640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96241f1fd0 0x7f96241f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f961bfff640 1 -- 172.21.15.46:0/3942162944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614009420 con 0x7f9624153980 2023-10-30T13:23:44.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f9622d9d640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96241f1fd0 0x7f96241f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:44.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f961bfff640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9624153980 0x7f9624148bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f960c00c780 tx=0x7f960c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:44.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f961affd640 1 -- 172.21.15.46:0/3942162944 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c005ae0 con 0x7f9624153980 2023-10-30T13:23:44.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f961affd640 1 -- 172.21.15.46:0/3942162944 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c005c40 con 0x7f9624153980 2023-10-30T13:23:44.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f961affd640 1 -- 172.21.15.46:0/3942162944 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c007070 con 0x7f9624153980 2023-10-30T13:23:44.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f9628faa640 1 -- 172.21.15.46:0/3942162944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96240c8c20 con 0x7f9624153980 2023-10-30T13:23:44.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.292+0000 7f9628faa640 1 -- 172.21.15.46:0/3942162944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96240c6f60 con 0x7f9624153980 2023-10-30T13:23:44.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.296+0000 7f9628faa640 1 -- 172.21.15.46:0/3942162944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962404e170 con 0x7f9624153980 2023-10-30T13:23:44.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.296+0000 7f961affd640 1 -- 172.21.15.46:0/3942162944 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f960c0071f0 con 0x7f9624153980 2023-10-30T13:23:44.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.296+0000 7f961affd640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96040664a0 0x7f9604068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.296+0000 7f961affd640 1 -- 172.21.15.46:0/3942162944 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f960c083690 con 0x7f9624153980 2023-10-30T13:23:44.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.296+0000 7f962259c640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96040664a0 0x7f9604068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.296+0000 7f962259c640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96040664a0 0x7f9604068960 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f96140ad000 tx=0x7f96140a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:44.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.300+0000 7f961affd640 1 -- 172.21.15.46:0/3942162944 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f960c054500 con 0x7f9624153980 2023-10-30T13:23:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 -- 172.21.15.46:0/3942162944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f96240caa80 con 0x7f9624153980 2023-10-30T13:23:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f961affd640 1 -- 172.21.15.46:0/3942162944 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f960c054090 con 0x7f9624153980 2023-10-30T13:23:44.467 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-10-30T13:23:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 -- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96040664a0 msgr2=0x7f9604068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96040664a0 0x7f9604068960 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f96140ad000 tx=0x7f96140a6000 comp rx=0 tx=0).stop 2023-10-30T13:23:44.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 -- 172.21.15.46:0/3942162944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9624153980 msgr2=0x7f9624148bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9624153980 0x7f9624148bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f960c00c780 tx=0x7f960c00cc40 comp rx=0 tx=0).stop 2023-10-30T13:23:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 -- 172.21.15.46:0/3942162944 shutdown_connections 2023-10-30T13:23:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96040664a0 0x7f9604068960 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96241f1fd0 0x7f96241f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9624153980 0x7f9624148bd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 --2- 172.21.15.46:0/3942162944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96240cb1e0 0x7f9624152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 -- 172.21.15.46:0/3942162944 >> 172.21.15.46:0/3942162944 conn(0x7f962406c0f0 msgr2=0x7f96240cd890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 -- 172.21.15.46:0/3942162944 shutdown_connections 2023-10-30T13:23:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.460+0000 7f9628faa640 1 -- 172.21.15.46:0/3942162944 wait complete. 2023-10-30T13:23:44.488 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:23:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.628+0000 7fc66efa7640 1 Processor -- start 2023-10-30T13:23:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.628+0000 7fc66efa7640 1 -- start start 2023-10-30T13:23:44.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.628+0000 7fc66efa7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 0x7fc66815a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.628+0000 7fc66efa7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc6680cb4c0 0x7fc6680cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.628+0000 7fc66efa7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66815a6b0 con 0x7fc668153c20 2023-10-30T13:23:44.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.628+0000 7fc66efa7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66815dbc0 con 0x7fc6680cb4c0 2023-10-30T13:23:44.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.628+0000 7fc66efa7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66815d730 con 0x7fc668153840 2023-10-30T13:23:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.628+0000 7fc66e7a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc6680cb4c0 0x7fc6680cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.628+0000 7fc66dfa5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 0x7fc66815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.628+0000 7fc66dfa5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 0x7fc66815a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:36328/0 (socket says 172.21.15.46:36328) 2023-10-30T13:23:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.628+0000 7fc66dfa5640 1 -- 172.21.15.46:0/1009597085 learned_addr learned my addr 172.21.15.46:0/1009597085 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.632+0000 7fc66dfa5640 1 -- 172.21.15.46:0/1009597085 >> v2:172.21.15.154:3300/0 conn(0x7fc6680cb4c0 msgr2=0x7fc6680cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.632+0000 7fc66dfa5640 1 --2- 172.21.15.46:0/1009597085 >> v2:172.21.15.154:3300/0 conn(0x7fc6680cb4c0 0x7fc6680cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.632+0000 7fc66dfa5640 1 -- 172.21.15.46:0/1009597085 >> v1:172.21.15.154:6789/0 conn(0x7fc668153c20 legacy=0x7fc6680c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.632+0000 7fc66dfa5640 1 -- 172.21.15.46:0/1009597085 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6680ca8f0 con 0x7fc668153840 2023-10-30T13:23:44.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.632+0000 7fc66dfa5640 1 --2- 172.21.15.46:0/1009597085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 0x7fc66815a170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc664009870 tx=0x7fc66401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=485391b0abb6bf4e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:44.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.632+0000 7fc66cfa3640 1 -- 172.21.15.46:0/1009597085 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6640a8a30 con 0x7fc668153840 2023-10-30T13:23:44.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.632+0000 7fc66cfa3640 1 -- 172.21.15.46:0/1009597085 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6640a8b90 con 0x7fc668153840 2023-10-30T13:23:44.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.632+0000 7fc66cfa3640 1 -- 172.21.15.46:0/1009597085 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6640b0440 con 0x7fc668153840 2023-10-30T13:23:44.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 -- 172.21.15.46:0/1009597085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 msgr2=0x7fc66815a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 --2- 172.21.15.46:0/1009597085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 0x7fc66815a170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc664009870 tx=0x7fc66401ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:44.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 -- 172.21.15.46:0/1009597085 shutdown_connections 2023-10-30T13:23:44.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 --2- 172.21.15.46:0/1009597085 >> v2:172.21.15.154:3300/0 conn(0x7fc6680cb4c0 0x7fc6680cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 --2- 172.21.15.46:0/1009597085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 0x7fc66815a170 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 -- 172.21.15.46:0/1009597085 >> 172.21.15.46:0/1009597085 conn(0x7fc66806be40 msgr2=0x7fc66806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:44.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 -- 172.21.15.46:0/1009597085 shutdown_connections 2023-10-30T13:23:44.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 -- 172.21.15.46:0/1009597085 wait complete. 2023-10-30T13:23:44.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 Processor -- start 2023-10-30T13:23:44.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 -- start start 2023-10-30T13:23:44.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6680cb4c0 0x7fc668158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 0x7fc6681553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc668153c20 0x7fc6681558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66815b260 con 0x7fc6680cb4c0 2023-10-30T13:23:44.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66815bde0 con 0x7fc668153c20 2023-10-30T13:23:44.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66efa7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66815c930 con 0x7fc668153840 2023-10-30T13:23:44.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66dfa5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6680cb4c0 0x7fc668158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66dfa5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6680cb4c0 0x7fc668158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51442/0 (socket says 172.21.15.46:51442) 2023-10-30T13:23:44.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66dfa5640 1 -- 172.21.15.46:0/2978415530 learned_addr learned my addr 172.21.15.46:0/2978415530 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66e7a6640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc668153c20 0x7fc6681558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.636+0000 7fc66d7a4640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 0x7fc6681553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc66dfa5640 1 -- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 msgr2=0x7fc6681553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc66dfa5640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 0x7fc6681553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc66dfa5640 1 -- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc668153c20 msgr2=0x7fc6681558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc66dfa5640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc668153c20 0x7fc6681558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc66dfa5640 1 -- 172.21.15.46:0/2978415530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6640094d0 con 0x7fc6680cb4c0 2023-10-30T13:23:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc66d7a4640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 0x7fc6681553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc66e7a6640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc668153c20 0x7fc6681558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:44.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc66dfa5640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6680cb4c0 0x7fc668158810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc66401d3e0 tx=0x7fc66401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc65e7fc640 1 -- 172.21.15.46:0/2978415530 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6640b0dd0 con 0x7fc6680cb4c0 2023-10-30T13:23:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc65e7fc640 1 -- 172.21.15.46:0/2978415530 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6640bb460 con 0x7fc6680cb4c0 2023-10-30T13:23:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc65e7fc640 1 -- 172.21.15.46:0/2978415530 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6640bb810 con 0x7fc6680cb4c0 2023-10-30T13:23:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc66efa7640 1 -- 172.21.15.46:0/2978415530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66815b4b0 con 0x7fc6680cb4c0 2023-10-30T13:23:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc66efa7640 1 -- 172.21.15.46:0/2978415530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66815ba70 con 0x7fc6680cb4c0 2023-10-30T13:23:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc66efa7640 1 -- 172.21.15.46:0/2978415530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66804e9c0 con 0x7fc6680cb4c0 2023-10-30T13:23:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc65e7fc640 1 -- 172.21.15.46:0/2978415530 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6640b83f0 con 0x7fc6680cb4c0 2023-10-30T13:23:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc65e7fc640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc64c0664d0 0x7fc64c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.640+0000 7fc65e7fc640 1 -- 172.21.15.46:0/2978415530 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc664121810 con 0x7fc6680cb4c0 2023-10-30T13:23:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.644+0000 7fc66d7a4640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc64c0664d0 0x7fc64c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.644+0000 7fc66d7a4640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc64c0664d0 0x7fc64c068990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc668157270 tx=0x7fc658009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.644+0000 7fc65e7fc640 1 -- 172.21.15.46:0/2978415530 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6640eeb60 con 0x7fc6680cb4c0 2023-10-30T13:23:44.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.804+0000 7fc66efa7640 1 -- 172.21.15.46:0/2978415530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc66806c6f0 con 0x7fc6680cb4c0 2023-10-30T13:23:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.804+0000 7fc65e7fc640 1 -- 172.21.15.46:0/2978415530 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc6640c0020 con 0x7fc6680cb4c0 2023-10-30T13:23:44.813 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-10-30T13:23:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.808+0000 7fc66efa7640 1 -- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc64c0664d0 msgr2=0x7fc64c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.808+0000 7fc66efa7640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc64c0664d0 0x7fc64c068990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc668157270 tx=0x7fc658009040 comp rx=0 tx=0).stop 2023-10-30T13:23:44.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.808+0000 7fc66efa7640 1 -- 172.21.15.46:0/2978415530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6680cb4c0 msgr2=0x7fc668158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.808+0000 7fc66efa7640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6680cb4c0 0x7fc668158810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc66401d3e0 tx=0x7fc66401bad0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.808+0000 7fc66efa7640 1 -- 172.21.15.46:0/2978415530 shutdown_connections 2023-10-30T13:23:44.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.808+0000 7fc66efa7640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc64c0664d0 0x7fc64c068990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.808+0000 7fc66efa7640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc668153c20 0x7fc6681558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.808+0000 7fc66efa7640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc668153840 0x7fc6681553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.808+0000 7fc66efa7640 1 --2- 172.21.15.46:0/2978415530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6680cb4c0 0x7fc668158810 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.808+0000 7fc66efa7640 1 -- 172.21.15.46:0/2978415530 >> 172.21.15.46:0/2978415530 conn(0x7fc66806be40 msgr2=0x7fc668151070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:44.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.808+0000 7fc66efa7640 1 -- 172.21.15.46:0/2978415530 shutdown_connections 2023-10-30T13:23:44.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.808+0000 7fc66efa7640 1 -- 172.21.15.46:0/2978415530 wait complete. 2023-10-30T13:23:44.833 INFO:tasks.mds_thrash:mds thrasher using random seed: 1698672224 2023-10-30T13:23:44.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:44.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.968+0000 7fdde713e640 1 Processor -- start 2023-10-30T13:23:44.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdde713e640 1 -- start start 2023-10-30T13:23:44.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdde713e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde01538a0 0x7fdde015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdde713e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdde00ca0e0 0x7fdde015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdde713e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde00cb800 con 0x7fdde00cf160 2023-10-30T13:23:44.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdde713e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde015dc30 con 0x7fdde00ca0e0 2023-10-30T13:23:44.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdde713e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00cb330 con 0x7fdde01538a0 2023-10-30T13:23:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdddffff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdde00ca0e0 0x7fdde015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdde4eda640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde01538a0 0x7fdde015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdddffff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdde00ca0e0 0x7fdde015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:33030/0 (socket says 172.21.15.46:33030) 2023-10-30T13:23:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdddffff640 1 -- 172.21.15.46:0/2312705942 learned_addr learned my addr 172.21.15.46:0/2312705942 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdde4eda640 1 -- 172.21.15.46:0/2312705942 >> v2:172.21.15.154:3300/0 conn(0x7fdde00ca0e0 msgr2=0x7fdde015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdde4eda640 1 --2- 172.21.15.46:0/2312705942 >> v2:172.21.15.154:3300/0 conn(0x7fdde00ca0e0 0x7fdde015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdde4eda640 1 -- 172.21.15.46:0/2312705942 >> v1:172.21.15.154:6789/0 conn(0x7fdde00cf160 legacy=0x7fdde00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdde4eda640 1 -- 172.21.15.46:0/2312705942 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde00cad70 con 0x7fdde01538a0 2023-10-30T13:23:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.972+0000 7fdddffff640 1 --2- 172.21.15.46:0/2312705942 >> v2:172.21.15.154:3300/0 conn(0x7fdde00ca0e0 0x7fdde015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.976+0000 7fdde4eda640 1 --2- 172.21.15.46:0/2312705942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde01538a0 0x7fdde015d6f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fddd40097c0 tx=0x7fddd401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c11297da5317c98c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:44.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.976+0000 7fdddf7fe640 1 -- 172.21.15.46:0/2312705942 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd40a8970 con 0x7fdde01538a0 2023-10-30T13:23:44.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.976+0000 7fdddf7fe640 1 -- 172.21.15.46:0/2312705942 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd40a8ad0 con 0x7fdde01538a0 2023-10-30T13:23:44.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.976+0000 7fdddf7fe640 1 -- 172.21.15.46:0/2312705942 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd40a8e60 con 0x7fdde01538a0 2023-10-30T13:23:44.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.976+0000 7fdde713e640 1 -- 172.21.15.46:0/2312705942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde01538a0 msgr2=0x7fdde015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.976+0000 7fdde713e640 1 --2- 172.21.15.46:0/2312705942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde01538a0 0x7fdde015d6f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fddd40097c0 tx=0x7fddd401ec70 comp rx=0 tx=0).stop 2023-10-30T13:23:44.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.976+0000 7fdde713e640 1 -- 172.21.15.46:0/2312705942 shutdown_connections 2023-10-30T13:23:44.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.976+0000 7fdde713e640 1 --2- 172.21.15.46:0/2312705942 >> v2:172.21.15.154:3300/0 conn(0x7fdde00ca0e0 0x7fdde015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.976+0000 7fdde713e640 1 --2- 172.21.15.46:0/2312705942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde01538a0 0x7fdde015d6f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.976+0000 7fdde713e640 1 -- 172.21.15.46:0/2312705942 >> 172.21.15.46:0/2312705942 conn(0x7fdde006bfe0 msgr2=0x7fdde006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:44.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.976+0000 7fdde713e640 1 -- 172.21.15.46:0/2312705942 shutdown_connections 2023-10-30T13:23:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.976+0000 7fdde713e640 1 -- 172.21.15.46:0/2312705942 wait complete. 2023-10-30T13:23:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde713e640 1 Processor -- start 2023-10-30T13:23:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde713e640 1 -- start start 2023-10-30T13:23:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde713e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde00ca0e0 0x7fdde00ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde713e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde00cf160 0x7fdde00b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde713e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde01538a0 0x7fdde00b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde713e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde01efcc0 con 0x7fdde01538a0 2023-10-30T13:23:44.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde713e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde01f01f0 con 0x7fdde00ca0e0 2023-10-30T13:23:44.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde713e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde01eff10 con 0x7fdde00cf160 2023-10-30T13:23:44.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde56db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde01538a0 0x7fdde00b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde56db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde01538a0 0x7fdde00b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51452/0 (socket says 172.21.15.46:51452) 2023-10-30T13:23:44.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde56db640 1 -- 172.21.15.46:0/2535022556 learned_addr learned my addr 172.21.15.46:0/2535022556 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:44.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde4eda640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde00ca0e0 0x7fdde00ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdddffff640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde00cf160 0x7fdde00b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde56db640 1 -- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde00cf160 msgr2=0x7fdde00b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde56db640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde00cf160 0x7fdde00b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde56db640 1 -- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde00ca0e0 msgr2=0x7fdde00ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde56db640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde00ca0e0 0x7fdde00ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde56db640 1 -- 172.21.15.46:0/2535022556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc0094d0 con 0x7fdde01538a0 2023-10-30T13:23:44.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde56db640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde01538a0 0x7fdde00b42f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c6c0 tx=0x7fddd800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:44.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdddd7fa640 1 -- 172.21.15.46:0/2535022556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd8003b80 con 0x7fdde01538a0 2023-10-30T13:23:44.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdddd7fa640 1 -- 172.21.15.46:0/2535022556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd8003ce0 con 0x7fdde01538a0 2023-10-30T13:23:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdddd7fa640 1 -- 172.21.15.46:0/2535022556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd80052f0 con 0x7fdde01538a0 2023-10-30T13:23:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde713e640 1 -- 172.21.15.46:0/2535022556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd4009420 con 0x7fdde01538a0 2023-10-30T13:23:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.980+0000 7fdde713e640 1 -- 172.21.15.46:0/2535022556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde00b35b0 con 0x7fdde01538a0 2023-10-30T13:23:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.984+0000 7fdde713e640 1 -- 172.21.15.46:0/2535022556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde00b2a50 con 0x7fdde01538a0 2023-10-30T13:23:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.984+0000 7fdddd7fa640 1 -- 172.21.15.46:0/2535022556 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fddd8005450 con 0x7fdde01538a0 2023-10-30T13:23:44.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.984+0000 7fdddd7fa640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddc40664d0 0x7fddc4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.984+0000 7fdddd7fa640 1 -- 172.21.15.46:0/2535022556 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddd8083a70 con 0x7fdde01538a0 2023-10-30T13:23:44.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.984+0000 7fdde4eda640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddc40664d0 0x7fddc4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.984+0000 7fdde4eda640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddc40664d0 0x7fddc4068990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fddd40b2000 tx=0x7fddd40aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:44.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:44.988+0000 7fdddd7fa640 1 -- 172.21.15.46:0/2535022556 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fddd8050f90 con 0x7fdde01538a0 2023-10-30T13:23:45.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.144+0000 7fdde713e640 1 -- 172.21.15.46:0/2535022556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdde004e170 con 0x7fdde01538a0 2023-10-30T13:23:45.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdddd7fa640 1 -- 172.21.15.46:0/2535022556 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fddd80545b0 con 0x7fdde01538a0 2023-10-30T13:23:45.154 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:45.155 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:20.665723+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:45.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdde713e640 1 -- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddc40664d0 msgr2=0x7fddc4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdde713e640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddc40664d0 0x7fddc4068990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fddd40b2000 tx=0x7fddd40aa070 comp rx=0 tx=0).stop 2023-10-30T13:23:45.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdde713e640 1 -- 172.21.15.46:0/2535022556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde01538a0 msgr2=0x7fdde00b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdde713e640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde01538a0 0x7fdde00b42f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c6c0 tx=0x7fddd800cb80 comp rx=0 tx=0).stop 2023-10-30T13:23:45.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdde713e640 1 -- 172.21.15.46:0/2535022556 shutdown_connections 2023-10-30T13:23:45.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdde713e640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddc40664d0 0x7fddc4068990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdde713e640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde01538a0 0x7fdde00b42f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdde713e640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde00cf160 0x7fdde00b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdde713e640 1 --2- 172.21.15.46:0/2535022556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde00ca0e0 0x7fdde00ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdde713e640 1 -- 172.21.15.46:0/2535022556 >> 172.21.15.46:0/2535022556 conn(0x7fdde006bfe0 msgr2=0x7fdde0152870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdde713e640 1 -- 172.21.15.46:0/2535022556 shutdown_connections 2023-10-30T13:23:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.148+0000 7fdde713e640 1 -- 172.21.15.46:0/2535022556 wait complete. 2023-10-30T13:23:45.176 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:23:06.788949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:23:17.648827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:23:18.326967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:23:45.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:23:45.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f4704cea640 1 Processor -- start 2023-10-30T13:23:45.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f4704cea640 1 -- start start 2023-10-30T13:23:45.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f4704cea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4700153980 0x7f47000caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:45.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f4704cea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4700159930 0x7f47000cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:45.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f4704cea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470015a920 con 0x7f47000cf1a0 2023-10-30T13:23:45.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f4704cea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47000cb930 con 0x7f4700159930 2023-10-30T13:23:45.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f4704cea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4700159dc0 con 0x7f4700153980 2023-10-30T13:23:45.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f46f7fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4700159930 0x7f47000cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:45.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f46fe59c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4700153980 0x7f47000caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:45.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f46f7fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4700159930 0x7f47000cb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:33052/0 (socket says 172.21.15.46:33052) 2023-10-30T13:23:45.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f46f7fff640 1 -- 172.21.15.46:0/915883178 learned_addr learned my addr 172.21.15.46:0/915883178 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f46fed9d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f47000cf1a0 0x7f470015d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:37894/0 (socket says 172.21.15.46:37894) 2023-10-30T13:23:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f46f7fff640 1 -- 172.21.15.46:0/915883178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4700153980 msgr2=0x7f47000caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f46f7fff640 1 --2- 172.21.15.46:0/915883178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4700153980 0x7f47000caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f46f7fff640 1 -- 172.21.15.46:0/915883178 >> v1:172.21.15.154:6789/0 conn(0x7f47000cf1a0 legacy=0x7f470015d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.320+0000 7f46f7fff640 1 -- 172.21.15.46:0/915883178 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470015a080 con 0x7f4700159930 2023-10-30T13:23:45.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.316+0000 7f46fdd9b640 1 -- 172.21.15.46:0/915883178 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456351084 0 0) 0x7f470015a920 con 0x7f47000cf1a0 2023-10-30T13:23:45.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.320+0000 7f46f7fff640 1 --2- 172.21.15.46:0/915883178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4700159930 0x7f47000cb1e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f46e8009870 tx=0x7f46e801c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f04d3ee07d84d705 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:45.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.320+0000 7f46fdd9b640 1 -- 172.21.15.46:0/915883178 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e80aa670 con 0x7f4700159930 2023-10-30T13:23:45.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.320+0000 7f46fdd9b640 1 -- 172.21.15.46:0/915883178 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e80aa7d0 con 0x7f4700159930 2023-10-30T13:23:45.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.320+0000 7f46fdd9b640 1 -- 172.21.15.46:0/915883178 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e80aab60 con 0x7f4700159930 2023-10-30T13:23:45.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.320+0000 7f4704cea640 1 -- 172.21.15.46:0/915883178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4700159930 msgr2=0x7f47000cb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.320+0000 7f4704cea640 1 --2- 172.21.15.46:0/915883178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4700159930 0x7f47000cb1e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f46e8009870 tx=0x7f46e801c510 comp rx=0 tx=0).stop 2023-10-30T13:23:45.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.320+0000 7f4704cea640 1 -- 172.21.15.46:0/915883178 shutdown_connections 2023-10-30T13:23:45.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.320+0000 7f4704cea640 1 --2- 172.21.15.46:0/915883178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4700159930 0x7f47000cb1e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.320+0000 7f4704cea640 1 --2- 172.21.15.46:0/915883178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4700153980 0x7f47000caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.320+0000 7f4704cea640 1 -- 172.21.15.46:0/915883178 >> 172.21.15.46:0/915883178 conn(0x7f470006c0f0 msgr2=0x7f470006c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:45.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f4704cea640 1 -- 172.21.15.46:0/915883178 shutdown_connections 2023-10-30T13:23:45.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f4704cea640 1 -- 172.21.15.46:0/915883178 wait complete. 2023-10-30T13:23:45.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f4704cea640 1 Processor -- start 2023-10-30T13:23:45.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f4704cea640 1 -- start start 2023-10-30T13:23:45.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f4704cea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47000cf1a0 0x7f4700142180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:45.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f4704cea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4700153980 0x7f47000cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:45.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f4704cea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4700152f90 0x7f47000cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:45.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f4704cea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4700151190 con 0x7f47000cf1a0 2023-10-30T13:23:45.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f4704cea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47001437a0 con 0x7f4700152f90 2023-10-30T13:23:45.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f4704cea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4700151430 con 0x7f4700153980 2023-10-30T13:23:45.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f46fe59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47000cf1a0 0x7f4700142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:45.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f46fe59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47000cf1a0 0x7f4700142180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51456/0 (socket says 172.21.15.46:51456) 2023-10-30T13:23:45.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f46fe59c640 1 -- 172.21.15.46:0/584263808 learned_addr learned my addr 172.21.15.46:0/584263808 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:45.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f46fed9d640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4700152f90 0x7f47000cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:45.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f46f7fff640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4700153980 0x7f47000cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:45.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f46fe59c640 1 -- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4700153980 msgr2=0x7f47000cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f46fe59c640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4700153980 0x7f47000cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f46fe59c640 1 -- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4700152f90 msgr2=0x7f47000cdb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f46fe59c640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4700152f90 0x7f47000cdb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f46fe59c640 1 -- 172.21.15.46:0/584263808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e80094d0 con 0x7f47000cf1a0 2023-10-30T13:23:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.324+0000 7f46fe59c640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47000cf1a0 0x7f4700142180 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f46f000e960 tx=0x7f46f000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.328+0000 7f46f6ffd640 1 -- 172.21.15.46:0/584263808 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0009690 con 0x7f47000cf1a0 2023-10-30T13:23:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.328+0000 7f46f6ffd640 1 -- 172.21.15.46:0/584263808 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f000ca00 con 0x7f47000cf1a0 2023-10-30T13:23:45.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.328+0000 7f46f6ffd640 1 -- 172.21.15.46:0/584263808 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f000cd70 con 0x7f47000cf1a0 2023-10-30T13:23:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.328+0000 7f4704cea640 1 -- 172.21.15.46:0/584263808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47001439d0 con 0x7f47000cf1a0 2023-10-30T13:23:45.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.328+0000 7f4704cea640 1 -- 172.21.15.46:0/584263808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47001426c0 con 0x7f47000cf1a0 2023-10-30T13:23:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.328+0000 7f4704cea640 1 -- 172.21.15.46:0/584263808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4700151aa0 con 0x7f47000cf1a0 2023-10-30T13:23:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.328+0000 7f46f6ffd640 1 -- 172.21.15.46:0/584263808 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f46f0013470 con 0x7f47000cf1a0 2023-10-30T13:23:45.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.328+0000 7f46f6ffd640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f46e00664d0 0x7f46e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.328+0000 7f46f6ffd640 1 -- 172.21.15.46:0/584263808 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46f0084370 con 0x7f47000cf1a0 2023-10-30T13:23:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.328+0000 7f46f7fff640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f46e00664d0 0x7f46e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.332+0000 7f46f7fff640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f46e00664d0 0x7f46e0068990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f46e80ae000 tx=0x7f46e80b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.332+0000 7f46f6ffd640 1 -- 172.21.15.46:0/584263808 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f46f00518a0 con 0x7f47000cf1a0 2023-10-30T13:23:45.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.488+0000 7f4704cea640 1 -- 172.21.15.46:0/584263808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4700143e80 con 0x7f47000cf1a0 2023-10-30T13:23:45.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.488+0000 7f46f6ffd640 1 -- 172.21.15.46:0/584263808 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f46f0054ec0 con 0x7f47000cf1a0 2023-10-30T13:23:45.497 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-10-30T13:23:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.492+0000 7f4704cea640 1 -- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f46e00664d0 msgr2=0x7f46e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.492+0000 7f4704cea640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f46e00664d0 0x7f46e0068990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f46e80ae000 tx=0x7f46e80b5040 comp rx=0 tx=0).stop 2023-10-30T13:23:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.492+0000 7f4704cea640 1 -- 172.21.15.46:0/584263808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47000cf1a0 msgr2=0x7f4700142180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.492+0000 7f4704cea640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47000cf1a0 0x7f4700142180 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f46f000e960 tx=0x7f46f000ee20 comp rx=0 tx=0).stop 2023-10-30T13:23:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.492+0000 7f4704cea640 1 -- 172.21.15.46:0/584263808 shutdown_connections 2023-10-30T13:23:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.492+0000 7f4704cea640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f46e00664d0 0x7f46e0068990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.492+0000 7f4704cea640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4700152f90 0x7f47000cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.492+0000 7f4704cea640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4700153980 0x7f47000cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.492+0000 7f4704cea640 1 --2- 172.21.15.46:0/584263808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47000cf1a0 0x7f4700142180 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.492+0000 7f4704cea640 1 -- 172.21.15.46:0/584263808 >> 172.21.15.46:0/584263808 conn(0x7f470006c0f0 msgr2=0x7f4700157910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.492+0000 7f4704cea640 1 -- 172.21.15.46:0/584263808 shutdown_connections 2023-10-30T13:23:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.492+0000 7f4704cea640 1 -- 172.21.15.46:0/584263808 wait complete. 2023-10-30T13:23:45.518 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:23:45.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.656+0000 7f2198f72640 1 Processor -- start 2023-10-30T13:23:45.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.656+0000 7f2198f72640 1 -- start start 2023-10-30T13:23:45.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.660+0000 7f2198f72640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f21940cd6b0 0x7f219406afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.660+0000 7f2198f72640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219406b4e0 con 0x7f219406c9e0 2023-10-30T13:23:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.660+0000 7f2198f72640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21941599b0 con 0x7f21941570d0 2023-10-30T13:23:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.660+0000 7f2198f72640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f219415a630 con 0x7f21940cd6b0 2023-10-30T13:23:45.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.660+0000 7f2192d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f21940cd6b0 0x7f219406afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.660+0000 7f2192d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f21940cd6b0 0x7f219406afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51460/0 (socket says 172.21.15.46:51460) 2023-10-30T13:23:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.660+0000 7f2192d9d640 1 -- 172.21.15.46:0/1419107232 learned_addr learned my addr 172.21.15.46:0/1419107232 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:45.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.660+0000 7f218bfff640 1 --1- 172.21.15.46:0/1419107232 >> v1:172.21.15.46:6789/0 conn(0x7f219406c9e0 0x7f21940ccfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49864/0 (socket says 172.21.15.46:49864) 2023-10-30T13:23:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.660+0000 7f2192d9d640 1 -- 172.21.15.46:0/1419107232 >> v1:172.21.15.154:6789/0 conn(0x7f21941570d0 legacy=0x7f2194071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.660+0000 7f2192d9d640 1 -- 172.21.15.46:0/1419107232 >> v1:172.21.15.46:6789/0 conn(0x7f219406c9e0 legacy=0x7f21940ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.660+0000 7f2192d9d640 1 -- 172.21.15.46:0/1419107232 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2194159c90 con 0x7f21940cd6b0 2023-10-30T13:23:45.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.660+0000 7f2192d9d640 1 --2- 172.21.15.46:0/1419107232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21940cd6b0 0x7f219406afa0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f218c009e50 tx=0x7f218c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8100ff3ecf87ec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:45.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2191d9b640 1 -- 172.21.15.46:0/1419107232 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c0af400 con 0x7f21940cd6b0 2023-10-30T13:23:45.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2191d9b640 1 -- 172.21.15.46:0/1419107232 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218c0ab050 con 0x7f21940cd6b0 2023-10-30T13:23:45.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2191d9b640 1 -- 172.21.15.46:0/1419107232 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c0af780 con 0x7f21940cd6b0 2023-10-30T13:23:45.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 -- 172.21.15.46:0/1419107232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21940cd6b0 msgr2=0x7f219406afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 --2- 172.21.15.46:0/1419107232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21940cd6b0 0x7f219406afa0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f218c009e50 tx=0x7f218c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 -- 172.21.15.46:0/1419107232 shutdown_connections 2023-10-30T13:23:45.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 --2- 172.21.15.46:0/1419107232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21940cd6b0 0x7f219406afa0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 -- 172.21.15.46:0/1419107232 >> 172.21.15.46:0/1419107232 conn(0x7f2194069e90 msgr2=0x7f219406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:45.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 -- 172.21.15.46:0/1419107232 shutdown_connections 2023-10-30T13:23:45.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 -- 172.21.15.46:0/1419107232 wait complete. 2023-10-30T13:23:45.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 Processor -- start 2023-10-30T13:23:45.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 -- start start 2023-10-30T13:23:45.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f219406c9e0 0x7f21941ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:45.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21940cd6b0 0x7f2194065a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:45.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21941570d0 0x7f2194065fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:45.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2194157710 con 0x7f21940cd6b0 2023-10-30T13:23:45.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21941ef2a0 con 0x7f219406c9e0 2023-10-30T13:23:45.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.664+0000 7f2198f72640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2194159230 con 0x7f21941570d0 2023-10-30T13:23:45.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21940cd6b0 0x7f2194065a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:45.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21940cd6b0 0x7f2194065a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51462/0 (socket says 172.21.15.46:51462) 2023-10-30T13:23:45.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218bfff640 1 -- 172.21.15.46:0/4242191846 learned_addr learned my addr 172.21.15.46:0/4242191846 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:45.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f2192d9d640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21941570d0 0x7f2194065fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:45.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f219259c640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f219406c9e0 0x7f21941ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:45.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218bfff640 1 -- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21941570d0 msgr2=0x7f2194065fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218bfff640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21941570d0 0x7f2194065fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218bfff640 1 -- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f219406c9e0 msgr2=0x7f21941ef9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218bfff640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f219406c9e0 0x7f21941ef9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218bfff640 1 -- 172.21.15.46:0/4242191846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c009b30 con 0x7f21940cd6b0 2023-10-30T13:23:45.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f219259c640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f219406c9e0 0x7f21941ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:45.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218bfff640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21940cd6b0 0x7f2194065a80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f217c00c430 tx=0x7f217c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:45.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218affd640 1 -- 172.21.15.46:0/4242191846 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c0074e0 con 0x7f21940cd6b0 2023-10-30T13:23:45.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218affd640 1 -- 172.21.15.46:0/4242191846 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217c007640 con 0x7f21940cd6b0 2023-10-30T13:23:45.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f2198f72640 1 -- 172.21.15.46:0/4242191846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21941589c0 con 0x7f21940cd6b0 2023-10-30T13:23:45.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218affd640 1 -- 172.21.15.46:0/4242191846 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c004a10 con 0x7f21940cd6b0 2023-10-30T13:23:45.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f2198f72640 1 -- 172.21.15.46:0/4242191846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2194158c30 con 0x7f21940cd6b0 2023-10-30T13:23:45.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f2198f72640 1 -- 172.21.15.46:0/4242191846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2194158e20 con 0x7f21940cd6b0 2023-10-30T13:23:45.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218affd640 1 -- 172.21.15.46:0/4242191846 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f217c004b70 con 0x7f21940cd6b0 2023-10-30T13:23:45.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.668+0000 7f218affd640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21740664a0 0x7f2174068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:45.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.672+0000 7f218affd640 1 -- 172.21.15.46:0/4242191846 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f217c083740 con 0x7f21940cd6b0 2023-10-30T13:23:45.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.672+0000 7f219259c640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21740664a0 0x7f2174068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:45.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.672+0000 7f219259c640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21740664a0 0x7f2174068960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2184000f70 tx=0x7f2184009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:45.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.672+0000 7f218affd640 1 -- 172.21.15.46:0/4242191846 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f217c050a90 con 0x7f21940cd6b0 2023-10-30T13:23:45.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.832+0000 7f2198f72640 1 -- 172.21.15.46:0/4242191846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f219406b4e0 con 0x7f21940cd6b0 2023-10-30T13:23:45.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.832+0000 7f218affd640 1 -- 172.21.15.46:0/4242191846 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f219406b4e0 con 0x7f21940cd6b0 2023-10-30T13:23:45.840 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-10-30T13:23:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.836+0000 7f2198f72640 1 -- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21740664a0 msgr2=0x7f2174068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.836+0000 7f2198f72640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21740664a0 0x7f2174068960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2184000f70 tx=0x7f2184009040 comp rx=0 tx=0).stop 2023-10-30T13:23:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.836+0000 7f2198f72640 1 -- 172.21.15.46:0/4242191846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21940cd6b0 msgr2=0x7f2194065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.836+0000 7f2198f72640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21940cd6b0 0x7f2194065a80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f217c00c430 tx=0x7f217c00c8f0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.836+0000 7f2198f72640 1 -- 172.21.15.46:0/4242191846 shutdown_connections 2023-10-30T13:23:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.836+0000 7f2198f72640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21740664a0 0x7f2174068960 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.836+0000 7f2198f72640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21941570d0 0x7f2194065fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.836+0000 7f2198f72640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21940cd6b0 0x7f2194065a80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.836+0000 7f2198f72640 1 --2- 172.21.15.46:0/4242191846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f219406c9e0 0x7f21941ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.836+0000 7f2198f72640 1 -- 172.21.15.46:0/4242191846 >> 172.21.15.46:0/4242191846 conn(0x7f2194069e90 msgr2=0x7f21940cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.836+0000 7f2198f72640 1 -- 172.21.15.46:0/4242191846 shutdown_connections 2023-10-30T13:23:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:45.836+0000 7f2198f72640 1 -- 172.21.15.46:0/4242191846 wait complete. 2023-10-30T13:23:45.861 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:23:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c58cd6640 1 Processor -- start 2023-10-30T13:23:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c58cd6640 1 -- start start 2023-10-30T13:23:46.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c58cd6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c54153980 0x7f5c540caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c58cd6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5c54159930 0x7f5c540cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c58cd6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5415a920 con 0x7f5c540cf1a0 2023-10-30T13:23:46.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c58cd6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c540cb930 con 0x7f5c54153980 2023-10-30T13:23:46.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c58cd6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c54159dc0 con 0x7f5c54159930 2023-10-30T13:23:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c5259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c54153980 0x7f5c540caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c52d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5c540cf1a0 0x7f5c5415d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49876/0 (socket says 172.21.15.46:49876) 2023-10-30T13:23:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c52d9d640 1 -- 172.21.15.46:0/2607198304 learned_addr learned my addr 172.21.15.46:0/2607198304 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c4bfff640 1 --2- 172.21.15.46:0/2607198304 >> v2:172.21.15.154:3300/0 conn(0x7f5c54159930 0x7f5c540cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c5259c640 1 -- 172.21.15.46:0/2607198304 >> v2:172.21.15.154:3300/0 conn(0x7f5c54159930 msgr2=0x7f5c540cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c5259c640 1 --2- 172.21.15.46:0/2607198304 >> v2:172.21.15.154:3300/0 conn(0x7f5c54159930 0x7f5c540cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c5259c640 1 -- 172.21.15.46:0/2607198304 >> v1:172.21.15.46:6789/0 conn(0x7f5c540cf1a0 legacy=0x7f5c5415d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c5259c640 1 -- 172.21.15.46:0/2607198304 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5415a080 con 0x7f5c54153980 2023-10-30T13:23:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c51d9b640 1 -- 172.21.15.46:0/2607198304 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102085290 0 0) 0x7f5c5415a920 con 0x7f5c540cf1a0 2023-10-30T13:23:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.000+0000 7f5c4bfff640 1 --2- 172.21.15.46:0/2607198304 >> v2:172.21.15.154:3300/0 conn(0x7f5c54159930 0x7f5c540cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.004+0000 7f5c5259c640 1 --2- 172.21.15.46:0/2607198304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c54153980 0x7f5c540caca0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5c540742c0 tx=0x7f5c4401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd13ada439b4223 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.004+0000 7f5c51d9b640 1 -- 172.21.15.46:0/2607198304 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c440a7070 con 0x7f5c54153980 2023-10-30T13:23:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.004+0000 7f5c51d9b640 1 -- 172.21.15.46:0/2607198304 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c440ab430 con 0x7f5c54153980 2023-10-30T13:23:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.004+0000 7f5c51d9b640 1 -- 172.21.15.46:0/2607198304 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c440ab7c0 con 0x7f5c54153980 2023-10-30T13:23:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.004+0000 7f5c58cd6640 1 -- 172.21.15.46:0/2607198304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c54153980 msgr2=0x7f5c540caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.004+0000 7f5c58cd6640 1 --2- 172.21.15.46:0/2607198304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c54153980 0x7f5c540caca0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5c540742c0 tx=0x7f5c4401c430 comp rx=0 tx=0).stop 2023-10-30T13:23:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.004+0000 7f5c58cd6640 1 -- 172.21.15.46:0/2607198304 shutdown_connections 2023-10-30T13:23:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.004+0000 7f5c58cd6640 1 --2- 172.21.15.46:0/2607198304 >> v2:172.21.15.154:3300/0 conn(0x7f5c54159930 0x7f5c540cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.004+0000 7f5c58cd6640 1 --2- 172.21.15.46:0/2607198304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c54153980 0x7f5c540caca0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.004+0000 7f5c58cd6640 1 -- 172.21.15.46:0/2607198304 >> 172.21.15.46:0/2607198304 conn(0x7f5c5406c0f0 msgr2=0x7f5c5406c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.004+0000 7f5c58cd6640 1 -- 172.21.15.46:0/2607198304 shutdown_connections 2023-10-30T13:23:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.004+0000 7f5c58cd6640 1 -- 172.21.15.46:0/2607198304 wait complete. 2023-10-30T13:23:46.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c58cd6640 1 Processor -- start 2023-10-30T13:23:46.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c58cd6640 1 -- start start 2023-10-30T13:23:46.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c58cd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540cf1a0 0x7f5c540ce860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c58cd6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c54153980 0x7f5c54149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c58cd6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c5413d770 0x7f5c5413dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c58cd6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c541531b0 con 0x7f5c540cf1a0 2023-10-30T13:23:46.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c58cd6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c54152010 con 0x7f5c5413d770 2023-10-30T13:23:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c58cd6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5413e160 con 0x7f5c54153980 2023-10-30T13:23:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c5259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540cf1a0 0x7f5c540ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c5259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540cf1a0 0x7f5c540ce860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51484/0 (socket says 172.21.15.46:51484) 2023-10-30T13:23:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c5259c640 1 -- 172.21.15.46:0/3900398057 learned_addr learned my addr 172.21.15.46:0/3900398057 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c52d9d640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c5413d770 0x7f5c5413dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c4bfff640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c54153980 0x7f5c54149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c5259c640 1 -- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c54153980 msgr2=0x7f5c54149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c5259c640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c54153980 0x7f5c54149cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c5259c640 1 -- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c5413d770 msgr2=0x7f5c5413dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c5259c640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c5413d770 0x7f5c5413dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c5259c640 1 -- 172.21.15.46:0/3900398057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c44009420 con 0x7f5c540cf1a0 2023-10-30T13:23:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c5259c640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540cf1a0 0x7f5c540ce860 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5c440ad040 tx=0x7f5c4401ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c4affd640 1 -- 172.21.15.46:0/3900398057 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c440b5810 con 0x7f5c540cf1a0 2023-10-30T13:23:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c4affd640 1 -- 172.21.15.46:0/3900398057 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c440b5990 con 0x7f5c540cf1a0 2023-10-30T13:23:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c58cd6640 1 -- 172.21.15.46:0/3900398057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c541511b0 con 0x7f5c540cf1a0 2023-10-30T13:23:46.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c4affd640 1 -- 172.21.15.46:0/3900398057 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c440b5d40 con 0x7f5c540cf1a0 2023-10-30T13:23:46.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.008+0000 7f5c58cd6640 1 -- 172.21.15.46:0/3900398057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c541516e0 con 0x7f5c540cf1a0 2023-10-30T13:23:46.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.012+0000 7f5c58cd6640 1 -- 172.21.15.46:0/3900398057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5404e170 con 0x7f5c540cf1a0 2023-10-30T13:23:46.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.012+0000 7f5c4affd640 1 -- 172.21.15.46:0/3900398057 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5c440b4390 con 0x7f5c540cf1a0 2023-10-30T13:23:46.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.012+0000 7f5c4affd640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c340664a0 0x7f5c34068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.012+0000 7f5c4affd640 1 -- 172.21.15.46:0/3900398057 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c4411e710 con 0x7f5c540cf1a0 2023-10-30T13:23:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.012+0000 7f5c4bfff640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c340664a0 0x7f5c34068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.012+0000 7f5c4bfff640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c340664a0 0x7f5c34068960 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c006fd0 tx=0x7f5c3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.016+0000 7f5c4affd640 1 -- 172.21.15.46:0/3900398057 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5c440ebb90 con 0x7f5c540cf1a0 2023-10-30T13:23:46.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.172+0000 7f5c58cd6640 1 -- 172.21.15.46:0/3900398057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5c541f5280 con 0x7f5c540cf1a0 2023-10-30T13:23:46.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.172+0000 7f5c4affd640 1 -- 172.21.15.46:0/3900398057 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5c440ef1b0 con 0x7f5c540cf1a0 2023-10-30T13:23:46.181 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-10-30T13:23:46.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.176+0000 7f5c58cd6640 1 -- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c340664a0 msgr2=0x7f5c34068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.176+0000 7f5c58cd6640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c340664a0 0x7f5c34068960 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c006fd0 tx=0x7f5c3c008040 comp rx=0 tx=0).stop 2023-10-30T13:23:46.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.176+0000 7f5c58cd6640 1 -- 172.21.15.46:0/3900398057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540cf1a0 msgr2=0x7f5c540ce860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.176+0000 7f5c58cd6640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540cf1a0 0x7f5c540ce860 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5c440ad040 tx=0x7f5c4401ac70 comp rx=0 tx=0).stop 2023-10-30T13:23:46.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.176+0000 7f5c58cd6640 1 -- 172.21.15.46:0/3900398057 shutdown_connections 2023-10-30T13:23:46.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.176+0000 7f5c58cd6640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c340664a0 0x7f5c34068960 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.176+0000 7f5c58cd6640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c5413d770 0x7f5c5413dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.176+0000 7f5c58cd6640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c54153980 0x7f5c54149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.176+0000 7f5c58cd6640 1 --2- 172.21.15.46:0/3900398057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c540cf1a0 0x7f5c540ce860 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.176+0000 7f5c58cd6640 1 -- 172.21.15.46:0/3900398057 >> 172.21.15.46:0/3900398057 conn(0x7f5c5406c0f0 msgr2=0x7f5c54157f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:46.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.176+0000 7f5c58cd6640 1 -- 172.21.15.46:0/3900398057 shutdown_connections 2023-10-30T13:23:46.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.176+0000 7f5c58cd6640 1 -- 172.21.15.46:0/3900398057 wait complete. 2023-10-30T13:23:46.202 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-10-30T13:23:46.202 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:46.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.336+0000 7f6df909c640 1 Processor -- start 2023-10-30T13:23:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.336+0000 7f6df909c640 1 -- start start 2023-10-30T13:23:46.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df909c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6df40c6b80 0x7f6df40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df909c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6df413f4a0 0x7f6df40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df909c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df40c0a50 con 0x7f6df40c6f60 2023-10-30T13:23:46.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df909c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df406f6e0 con 0x7f6df40c6b80 2023-10-30T13:23:46.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df909c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df406f9c0 con 0x7f6df413f4a0 2023-10-30T13:23:46.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6df40c6b80 0x7f6df40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6df40c6b80 0x7f6df40bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51488/0 (socket says 172.21.15.46:51488) 2023-10-30T13:23:46.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6df40c6f60 0x7f6df413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49890/0 (socket says 172.21.15.46:49890) 2023-10-30T13:23:46.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df2d9d640 1 -- 172.21.15.46:0/3544232681 learned_addr learned my addr 172.21.15.46:0/3544232681 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:46.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df359e640 1 --2- 172.21.15.46:0/3544232681 >> v2:172.21.15.154:3300/0 conn(0x7f6df413f4a0 0x7f6df40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df2d9d640 1 -- 172.21.15.46:0/3544232681 >> v2:172.21.15.154:3300/0 conn(0x7f6df413f4a0 msgr2=0x7f6df40c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:46.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df2d9d640 1 --2- 172.21.15.46:0/3544232681 >> v2:172.21.15.154:3300/0 conn(0x7f6df413f4a0 0x7f6df40c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df2d9d640 1 -- 172.21.15.46:0/3544232681 >> v1:172.21.15.46:6789/0 conn(0x7f6df40c6f60 legacy=0x7f6df413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df2d9d640 1 -- 172.21.15.46:0/3544232681 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df406fca0 con 0x7f6df40c6b80 2023-10-30T13:23:46.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.340+0000 7f6df1d9b640 1 -- 172.21.15.46:0/3544232681 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332589573 0 0) 0x7f6df40c0a50 con 0x7f6df40c6f60 2023-10-30T13:23:46.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df2d9d640 1 --2- 172.21.15.46:0/3544232681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40c6b80 0x7f6df40bfec0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6df40c8980 tx=0x7f6de401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a434a13065b4dd7f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:46.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df1d9b640 1 -- 172.21.15.46:0/3544232681 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de40ab610 con 0x7f6df40c6b80 2023-10-30T13:23:46.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df1d9b640 1 -- 172.21.15.46:0/3544232681 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de40ab770 con 0x7f6df40c6b80 2023-10-30T13:23:46.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df1d9b640 1 -- 172.21.15.46:0/3544232681 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de40abb00 con 0x7f6df40c6b80 2023-10-30T13:23:46.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df909c640 1 -- 172.21.15.46:0/3544232681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40c6b80 msgr2=0x7f6df40bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df909c640 1 --2- 172.21.15.46:0/3544232681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40c6b80 0x7f6df40bfec0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6df40c8980 tx=0x7f6de401c500 comp rx=0 tx=0).stop 2023-10-30T13:23:46.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df909c640 1 -- 172.21.15.46:0/3544232681 shutdown_connections 2023-10-30T13:23:46.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df909c640 1 --2- 172.21.15.46:0/3544232681 >> v2:172.21.15.154:3300/0 conn(0x7f6df413f4a0 0x7f6df40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df909c640 1 --2- 172.21.15.46:0/3544232681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40c6b80 0x7f6df40bfec0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df909c640 1 -- 172.21.15.46:0/3544232681 >> 172.21.15.46:0/3544232681 conn(0x7f6df40c7910 msgr2=0x7f6df40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:46.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df909c640 1 -- 172.21.15.46:0/3544232681 shutdown_connections 2023-10-30T13:23:46.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df909c640 1 -- 172.21.15.46:0/3544232681 wait complete. 2023-10-30T13:23:46.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df909c640 1 Processor -- start 2023-10-30T13:23:46.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.344+0000 7f6df909c640 1 -- start start 2023-10-30T13:23:46.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df909c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6df40c6b80 0x7f6df4156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df909c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40c6f60 0x7f6df40c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df909c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6df413f4a0 0x7f6df4138620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df909c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df40cabf0 con 0x7f6df40c6f60 2023-10-30T13:23:46.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df909c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df40c8d40 con 0x7f6df413f4a0 2023-10-30T13:23:46.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df909c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df40ca910 con 0x7f6df40c6b80 2023-10-30T13:23:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40c6f60 0x7f6df40c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40c6f60 0x7f6df40c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51496/0 (socket says 172.21.15.46:51496) 2023-10-30T13:23:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df259c640 1 -- 172.21.15.46:0/3900536502 learned_addr learned my addr 172.21.15.46:0/3900536502 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df359e640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6df413f4a0 0x7f6df4138620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df2d9d640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6df40c6b80 0x7f6df4156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df259c640 1 -- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6df40c6b80 msgr2=0x7f6df4156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df259c640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6df40c6b80 0x7f6df4156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df259c640 1 -- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6df413f4a0 msgr2=0x7f6df4138620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df259c640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6df413f4a0 0x7f6df4138620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df259c640 1 -- 172.21.15.46:0/3900536502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de40094d0 con 0x7f6df40c6f60 2023-10-30T13:23:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df259c640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40c6f60 0x7f6df40c9b80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6de80011f0 tx=0x7f6de800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6ddb7fe640 1 -- 172.21.15.46:0/3900536502 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de80074e0 con 0x7f6df40c6f60 2023-10-30T13:23:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6ddb7fe640 1 -- 172.21.15.46:0/3900536502 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de8007640 con 0x7f6df40c6f60 2023-10-30T13:23:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df909c640 1 -- 172.21.15.46:0/3900536502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df40c8fe0 con 0x7f6df40c6f60 2023-10-30T13:23:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6ddb7fe640 1 -- 172.21.15.46:0/3900536502 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de8004a10 con 0x7f6df40c6f60 2023-10-30T13:23:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df909c640 1 -- 172.21.15.46:0/3900536502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df40ca0c0 con 0x7f6df40c6f60 2023-10-30T13:23:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6df909c640 1 -- 172.21.15.46:0/3900536502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df404e170 con 0x7f6df40c6f60 2023-10-30T13:23:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.348+0000 7f6ddb7fe640 1 -- 172.21.15.46:0/3900536502 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6de8004b70 con 0x7f6df40c6f60 2023-10-30T13:23:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.352+0000 7f6ddb7fe640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6de00664a0 0x7f6de0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.352+0000 7f6ddb7fe640 1 -- 172.21.15.46:0/3900536502 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6de8083400 con 0x7f6df40c6f60 2023-10-30T13:23:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.352+0000 7f6df2d9d640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6de00664a0 0x7f6de0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.352+0000 7f6df2d9d640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6de00664a0 0x7f6de0068960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6de40af000 tx=0x7f6de40a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.356+0000 7f6ddb7fe640 1 -- 172.21.15.46:0/3900536502 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6de8050890 con 0x7f6df40c6f60 2023-10-30T13:23:46.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.544+0000 7f6df909c640 1 -- 172.21.15.46:0/3900536502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6df40c8070 con 0x7f6df40c6f60 2023-10-30T13:23:46.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.544+0000 7f6ddb7fe640 1 -- 172.21.15.46:0/3900536502 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f6de8053eb0 con 0x7f6df40c6f60 2023-10-30T13:23:46.553 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:46.553 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:35.740725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-10-30T13:23:46.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.548+0000 7f6df909c640 1 -- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6de00664a0 msgr2=0x7f6de0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.548+0000 7f6df909c640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6de00664a0 0x7f6de0068960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6de40af000 tx=0x7f6de40a7000 comp rx=0 tx=0).stop 2023-10-30T13:23:46.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.548+0000 7f6df909c640 1 -- 172.21.15.46:0/3900536502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40c6f60 msgr2=0x7f6df40c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.548+0000 7f6df909c640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40c6f60 0x7f6df40c9b80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6de80011f0 tx=0x7f6de800c9b0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.548+0000 7f6df909c640 1 -- 172.21.15.46:0/3900536502 shutdown_connections 2023-10-30T13:23:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.548+0000 7f6df909c640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6de00664a0 0x7f6de0068960 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.548+0000 7f6df909c640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6df413f4a0 0x7f6df4138620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.548+0000 7f6df909c640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6df40c6f60 0x7f6df40c9b80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.548+0000 7f6df909c640 1 --2- 172.21.15.46:0/3900536502 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6df40c6b80 0x7f6df4156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.548+0000 7f6df909c640 1 -- 172.21.15.46:0/3900536502 >> 172.21.15.46:0/3900536502 conn(0x7f6df40c7910 msgr2=0x7f6df4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:46.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.548+0000 7f6df909c640 1 -- 172.21.15.46:0/3900536502 shutdown_connections 2023-10-30T13:23:46.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.548+0000 7f6df909c640 1 -- 172.21.15.46:0/3900536502 wait complete. 2023-10-30T13:23:46.573 INFO:tasks.mds_thrash:Ready to start thrashing 2023-10-30T13:23:46.574 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-10-30T13:23:46.574 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:23:46.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.708+0000 7f2eab59e640 1 Processor -- start 2023-10-30T13:23:46.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2eab59e640 1 -- start start 2023-10-30T13:23:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2eab59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2eac15cd50 0x7f2eac0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2eab59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2eac0cefa0 0x7f2eac0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2eab59e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eac15a6c0 con 0x7f2eac1538f0 2023-10-30T13:23:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2eab59e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eac15d300 con 0x7f2eac15cd50 2023-10-30T13:23:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2eab59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac15d5e0 con 0x7f2eac0cefa0 2023-10-30T13:23:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2ea9d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2eac15cd50 0x7f2eac0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2ea9d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2eac15cd50 0x7f2eac0c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47322/0 (socket says 172.21.15.46:47322) 2023-10-30T13:23:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2ea9d9b640 1 -- 172.21.15.46:0/2962956834 learned_addr learned my addr 172.21.15.46:0/2962956834 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2eaad9d640 1 --2- 172.21.15.46:0/2962956834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2eac0cefa0 0x7f2eac0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2ea9d9b640 1 -- 172.21.15.46:0/2962956834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2eac0cefa0 msgr2=0x7f2eac0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2ea9d9b640 1 --2- 172.21.15.46:0/2962956834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2eac0cefa0 0x7f2eac0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2ea9d9b640 1 -- 172.21.15.46:0/2962956834 >> v1:172.21.15.154:6789/0 conn(0x7f2eac1538f0 legacy=0x7f2eac159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.712+0000 7f2ea9d9b640 1 -- 172.21.15.46:0/2962956834 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eac15d8c0 con 0x7f2eac15cd50 2023-10-30T13:23:46.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.716+0000 7f2ea9d9b640 1 --2- 172.21.15.46:0/2962956834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac15cd50 0x7f2eac0c9f90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009870 tx=0x7f2ea001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f05b0eb412fc32d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.716+0000 7f2ea959a640 1 -- 172.21.15.46:0/2962956834 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00a8a30 con 0x7f2eac15cd50 2023-10-30T13:23:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.716+0000 7f2ea959a640 1 -- 172.21.15.46:0/2962956834 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea00a8b90 con 0x7f2eac15cd50 2023-10-30T13:23:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.716+0000 7f2ea959a640 1 -- 172.21.15.46:0/2962956834 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00b0440 con 0x7f2eac15cd50 2023-10-30T13:23:46.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.716+0000 7f2eab59e640 1 -- 172.21.15.46:0/2962956834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac15cd50 msgr2=0x7f2eac0c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.716+0000 7f2eab59e640 1 --2- 172.21.15.46:0/2962956834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac15cd50 0x7f2eac0c9f90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009870 tx=0x7f2ea001ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:46.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.716+0000 7f2eab59e640 1 -- 172.21.15.46:0/2962956834 shutdown_connections 2023-10-30T13:23:46.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.716+0000 7f2eab59e640 1 --2- 172.21.15.46:0/2962956834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2eac0cefa0 0x7f2eac0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.716+0000 7f2eab59e640 1 --2- 172.21.15.46:0/2962956834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac15cd50 0x7f2eac0c9f90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.716+0000 7f2eab59e640 1 -- 172.21.15.46:0/2962956834 >> 172.21.15.46:0/2962956834 conn(0x7f2eac06a640 msgr2=0x7f2eac065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:46.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.716+0000 7f2eab59e640 1 -- 172.21.15.46:0/2962956834 shutdown_connections 2023-10-30T13:23:46.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.716+0000 7f2eab59e640 1 -- 172.21.15.46:0/2962956834 wait complete. 2023-10-30T13:23:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eab59e640 1 Processor -- start 2023-10-30T13:23:46.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eab59e640 1 -- start start 2023-10-30T13:23:46.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eab59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2eac0cefa0 0x7f2eac15a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eab59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac1538f0 0x7f2eac15b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eab59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2eac15cd50 0x7f2eac0b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eab59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac15c500 con 0x7f2eac1538f0 2023-10-30T13:23:46.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eab59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac15ae60 con 0x7f2eac0cefa0 2023-10-30T13:23:46.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eab59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac14ddf0 con 0x7f2eac15cd50 2023-10-30T13:23:46.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2ea9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac1538f0 0x7f2eac15b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2ea9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac1538f0 0x7f2eac15b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47328/0 (socket says 172.21.15.46:47328) 2023-10-30T13:23:46.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2ea9d9b640 1 -- 172.21.15.46:0/2212291926 learned_addr learned my addr 172.21.15.46:0/2212291926 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:46.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eaad9d640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2eac15cd50 0x7f2eac0b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eaa59c640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2eac0cefa0 0x7f2eac15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2ea9d9b640 1 -- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2eac15cd50 msgr2=0x7f2eac0b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2ea9d9b640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2eac15cd50 0x7f2eac0b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2ea9d9b640 1 -- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2eac0cefa0 msgr2=0x7f2eac15a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2ea9d9b640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2eac0cefa0 0x7f2eac15a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2ea9d9b640 1 -- 172.21.15.46:0/2212291926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea00094d0 con 0x7f2eac1538f0 2023-10-30T13:23:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2ea9d9b640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac1538f0 0x7f2eac15b410 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00b3040 tx=0x7f2ea001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2e9affd640 1 -- 172.21.15.46:0/2212291926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00be070 con 0x7f2eac1538f0 2023-10-30T13:23:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2e9affd640 1 -- 172.21.15.46:0/2212291926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea00ba400 con 0x7f2eac1538f0 2023-10-30T13:23:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2e9affd640 1 -- 172.21.15.46:0/2212291926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00ba7d0 con 0x7f2eac1538f0 2023-10-30T13:23:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eab59e640 1 -- 172.21.15.46:0/2212291926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eac14db10 con 0x7f2eac1538f0 2023-10-30T13:23:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eab59e640 1 -- 172.21.15.46:0/2212291926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eac1ef2e0 con 0x7f2eac1538f0 2023-10-30T13:23:46.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.720+0000 7f2eab59e640 1 -- 172.21.15.46:0/2212291926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eac04e170 con 0x7f2eac1538f0 2023-10-30T13:23:46.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.724+0000 7f2e9affd640 1 -- 172.21.15.46:0/2212291926 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ea00c3410 con 0x7f2eac1538f0 2023-10-30T13:23:46.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.724+0000 7f2e9affd640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2e8c0664a0 0x7f2e8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.724+0000 7f2e9affd640 1 -- 172.21.15.46:0/2212291926 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ea0121a20 con 0x7f2eac1538f0 2023-10-30T13:23:46.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.724+0000 7f2eaa59c640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2e8c0664a0 0x7f2e8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.724+0000 7f2eaa59c640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2e8c0664a0 0x7f2e8c068960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c007fd0 tx=0x7f2e9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:46.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.728+0000 7f2e9affd640 1 -- 172.21.15.46:0/2212291926 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ea00eee00 con 0x7f2eac1538f0 2023-10-30T13:23:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.884+0000 7f2eab59e640 1 -- 172.21.15.46:0/2212291926 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2eac1eee80 con 0x7f2e8c0664a0 2023-10-30T13:23:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2e9affd640 1 -- 172.21.15.46:0/2212291926 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264626 (secure 0 0 0) 0x7f2e9c0014f0 con 0x7f2e8c0664a0 2023-10-30T13:23:46.912 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:46.914 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T13:23:46.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2eab59e640 1 -- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2e8c0664a0 msgr2=0x7f2e8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2eab59e640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2e8c0664a0 0x7f2e8c068960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c007fd0 tx=0x7f2e9c009040 comp rx=0 tx=0).stop 2023-10-30T13:23:46.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2eab59e640 1 -- 172.21.15.46:0/2212291926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac1538f0 msgr2=0x7f2eac15b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2eab59e640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac1538f0 0x7f2eac15b410 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00b3040 tx=0x7f2ea001bad0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2eab59e640 1 -- 172.21.15.46:0/2212291926 shutdown_connections 2023-10-30T13:23:46.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2eab59e640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2e8c0664a0 0x7f2e8c068960 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2eab59e640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2eac15cd50 0x7f2eac0b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2eab59e640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac1538f0 0x7f2eac15b410 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2eab59e640 1 --2- 172.21.15.46:0/2212291926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2eac0cefa0 0x7f2eac15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2eab59e640 1 -- 172.21.15.46:0/2212291926 >> 172.21.15.46:0/2212291926 conn(0x7f2eac06a640 msgr2=0x7f2eac0c47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:46.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2eab59e640 1 -- 172.21.15.46:0/2212291926 shutdown_connections 2023-10-30T13:23:46.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:46.904+0000 7f2eab59e640 1 -- 172.21.15.46:0/2212291926 wait complete. 2023-10-30T13:23:46.932 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-30T13:23:46.718944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596835,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":287,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235676,"kb_used_data":2908,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739044,"statfs":{"total":773094113280,"available":772852781056,"internally_reserved":0,"allocated":2977792,"data_stored":1757797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2651,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004545"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612886+0000","last_change":"2023-10-30T13:23:19.719271+0000","last_active":"2023-10-30T13:23:21.612886+0000","last_peered":"2023-10-30T13:23:21.612886+0000","last_clean":"2023-10-30T13:23:21.612886+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T13:23:21.612886+0000","last_undegraded":"2023-10-30T13:23:21.612886+0000","last_fullsized":"2023-10-30T13:23:21.612886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:41:37.614073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670716+0000","last_change":"2023-10-30T13:23:19.687152+0000","last_active":"2023-10-30T13:23:20.670716+0000","last_peered":"2023-10-30T13:23:20.670716+0000","last_clean":"2023-10-30T13:23:20.670716+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T13:23:20.670716+0000","last_undegraded":"2023-10-30T13:23:20.670716+0000","last_fullsized":"2023-10-30T13:23:20.670716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107815+0000","last_change":"2023-10-30T13:23:19.689554+0000","last_active":"2023-10-30T13:23:21.107815+0000","last_peered":"2023-10-30T13:23:21.107815+0000","last_clean":"2023-10-30T13:23:21.107815+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T13:23:21.107815+0000","last_undegraded":"2023-10-30T13:23:21.107815+0000","last_fullsized":"2023-10-30T13:23:21.107815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:56:13.795044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611622+0000","last_change":"2023-10-30T13:23:19.714018+0000","last_active":"2023-10-30T13:23:21.611622+0000","last_peered":"2023-10-30T13:23:21.611622+0000","last_clean":"2023-10-30T13:23:21.611622+0000","last_became_active":"2023-10-30T13:23:19.711673+0000","last_became_peered":"2023-10-30T13:23:19.711673+0000","last_unstale":"2023-10-30T13:23:21.611622+0000","last_undegraded":"2023-10-30T13:23:21.611622+0000","last_fullsized":"2023-10-30T13:23:21.611622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:28:16.658840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611546+0000","last_change":"2023-10-30T13:23:19.719999+0000","last_active":"2023-10-30T13:23:21.611546+0000","last_peered":"2023-10-30T13:23:21.611546+0000","last_clean":"2023-10-30T13:23:21.611546+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T13:23:21.611546+0000","last_undegraded":"2023-10-30T13:23:21.611546+0000","last_fullsized":"2023-10-30T13:23:21.611546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:46:31.287013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612300+0000","last_change":"2023-10-30T13:23:19.703474+0000","last_active":"2023-10-30T13:23:21.612300+0000","last_peered":"2023-10-30T13:23:21.612300+0000","last_clean":"2023-10-30T13:23:21.612300+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T13:23:21.612300+0000","last_undegraded":"2023-10-30T13:23:21.612300+0000","last_fullsized":"2023-10-30T13:23:21.612300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:53:58.118821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613234+0000","last_change":"2023-10-30T13:23:19.726985+0000","last_active":"2023-10-30T13:23:21.613234+0000","last_peered":"2023-10-30T13:23:21.613234+0000","last_clean":"2023-10-30T13:23:21.613234+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T13:23:21.613234+0000","last_undegraded":"2023-10-30T13:23:21.613234+0000","last_fullsized":"2023-10-30T13:23:21.613234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:51:13.152339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.004933+0000","last_change":"2023-10-30T13:23:19.714153+0000","last_active":"2023-10-30T13:23:21.004933+0000","last_peered":"2023-10-30T13:23:21.004933+0000","last_clean":"2023-10-30T13:23:21.004933+0000","last_became_active":"2023-10-30T13:23:19.713876+0000","last_became_peered":"2023-10-30T13:23:19.713876+0000","last_unstale":"2023-10-30T13:23:21.004933+0000","last_undegraded":"2023-10-30T13:23:21.004933+0000","last_fullsized":"2023-10-30T13:23:21.004933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:05:13.588780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613220+0000","last_change":"2023-10-30T13:23:19.724583+0000","last_active":"2023-10-30T13:23:21.613220+0000","last_peered":"2023-10-30T13:23:21.613220+0000","last_clean":"2023-10-30T13:23:21.613220+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T13:23:21.613220+0000","last_undegraded":"2023-10-30T13:23:21.613220+0000","last_fullsized":"2023-10-30T13:23:21.613220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:47.936982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677145+0000","last_change":"2023-10-30T13:23:19.704399+0000","last_active":"2023-10-30T13:23:20.677145+0000","last_peered":"2023-10-30T13:23:20.677145+0000","last_clean":"2023-10-30T13:23:20.677145+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T13:23:20.677145+0000","last_undegraded":"2023-10-30T13:23:20.677145+0000","last_fullsized":"2023-10-30T13:23:20.677145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676522+0000","last_change":"2023-10-30T13:23:19.726077+0000","last_active":"2023-10-30T13:23:20.676522+0000","last_peered":"2023-10-30T13:23:20.676522+0000","last_clean":"2023-10-30T13:23:20.676522+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T13:23:20.676522+0000","last_undegraded":"2023-10-30T13:23:20.676522+0000","last_fullsized":"2023-10-30T13:23:20.676522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675959+0000","last_change":"2023-10-30T13:23:19.720995+0000","last_active":"2023-10-30T13:23:20.675959+0000","last_peered":"2023-10-30T13:23:20.675959+0000","last_clean":"2023-10-30T13:23:20.675959+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T13:23:20.675959+0000","last_undegraded":"2023-10-30T13:23:20.675959+0000","last_fullsized":"2023-10-30T13:23:20.675959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:44.490568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005253+0000","last_change":"2023-10-30T13:23:19.692509+0000","last_active":"2023-10-30T13:23:21.005253+0000","last_peered":"2023-10-30T13:23:21.005253+0000","last_clean":"2023-10-30T13:23:21.005253+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T13:23:21.005253+0000","last_undegraded":"2023-10-30T13:23:21.005253+0000","last_fullsized":"2023-10-30T13:23:21.005253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:15:17.956816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107580+0000","last_change":"2023-10-30T13:23:19.685435+0000","last_active":"2023-10-30T13:23:21.107580+0000","last_peered":"2023-10-30T13:23:21.107580+0000","last_clean":"2023-10-30T13:23:21.107580+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T13:23:21.107580+0000","last_undegraded":"2023-10-30T13:23:21.107580+0000","last_fullsized":"2023-10-30T13:23:21.107580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:39.035298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670272+0000","last_change":"2023-10-30T13:23:19.706721+0000","last_active":"2023-10-30T13:23:20.670272+0000","last_peered":"2023-10-30T13:23:20.670272+0000","last_clean":"2023-10-30T13:23:20.670272+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T13:23:20.670272+0000","last_undegraded":"2023-10-30T13:23:20.670272+0000","last_fullsized":"2023-10-30T13:23:20.670272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.610717+0000","last_change":"2023-10-30T13:23:19.710432+0000","last_active":"2023-10-30T13:23:21.610717+0000","last_peered":"2023-10-30T13:23:21.610717+0000","last_clean":"2023-10-30T13:23:21.610717+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T13:23:21.610717+0000","last_undegraded":"2023-10-30T13:23:21.610717+0000","last_fullsized":"2023-10-30T13:23:21.610717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:48:51.992826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612857+0000","last_change":"2023-10-30T13:23:19.741225+0000","last_active":"2023-10-30T13:23:21.612857+0000","last_peered":"2023-10-30T13:23:21.612857+0000","last_clean":"2023-10-30T13:23:21.612857+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T13:23:21.612857+0000","last_undegraded":"2023-10-30T13:23:21.612857+0000","last_fullsized":"2023-10-30T13:23:21.612857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:27.310070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.608570+0000","last_change":"2023-10-30T13:23:19.701024+0000","last_active":"2023-10-30T13:23:21.608570+0000","last_peered":"2023-10-30T13:23:21.608570+0000","last_clean":"2023-10-30T13:23:21.608570+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T13:23:21.608570+0000","last_undegraded":"2023-10-30T13:23:21.608570+0000","last_fullsized":"2023-10-30T13:23:21.608570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:14.460383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612314+0000","last_change":"2023-10-30T13:23:19.723929+0000","last_active":"2023-10-30T13:23:21.612314+0000","last_peered":"2023-10-30T13:23:21.612314+0000","last_clean":"2023-10-30T13:23:21.612314+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T13:23:21.612314+0000","last_undegraded":"2023-10-30T13:23:21.612314+0000","last_fullsized":"2023-10-30T13:23:21.612314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:41.023731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674806+0000","last_change":"2023-10-30T13:23:19.725101+0000","last_active":"2023-10-30T13:23:20.674806+0000","last_peered":"2023-10-30T13:23:20.674806+0000","last_clean":"2023-10-30T13:23:20.674806+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T13:23:20.674806+0000","last_undegraded":"2023-10-30T13:23:20.674806+0000","last_fullsized":"2023-10-30T13:23:20.674806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107754+0000","last_change":"2023-10-30T13:23:19.721813+0000","last_active":"2023-10-30T13:23:21.107754+0000","last_peered":"2023-10-30T13:23:21.107754+0000","last_clean":"2023-10-30T13:23:21.107754+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T13:23:21.107754+0000","last_undegraded":"2023-10-30T13:23:21.107754+0000","last_fullsized":"2023-10-30T13:23:21.107754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:34:17.212695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612670+0000","last_change":"2023-10-30T13:23:19.724763+0000","last_active":"2023-10-30T13:23:21.612670+0000","last_peered":"2023-10-30T13:23:21.612670+0000","last_clean":"2023-10-30T13:23:21.612670+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T13:23:21.612670+0000","last_undegraded":"2023-10-30T13:23:21.612670+0000","last_fullsized":"2023-10-30T13:23:21.612670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:28:53.428578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005009+0000","last_change":"2023-10-30T13:23:19.716706+0000","last_active":"2023-10-30T13:23:21.005009+0000","last_peered":"2023-10-30T13:23:21.005009+0000","last_clean":"2023-10-30T13:23:21.005009+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T13:23:21.005009+0000","last_undegraded":"2023-10-30T13:23:21.005009+0000","last_fullsized":"2023-10-30T13:23:21.005009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:07.718271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.610920+0000","last_change":"2023-10-30T13:23:19.704791+0000","last_active":"2023-10-30T13:23:21.610920+0000","last_peered":"2023-10-30T13:23:21.610920+0000","last_clean":"2023-10-30T13:23:21.610920+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T13:23:21.610920+0000","last_undegraded":"2023-10-30T13:23:21.610920+0000","last_fullsized":"2023-10-30T13:23:21.610920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:54.750514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005422+0000","last_change":"2023-10-30T13:23:19.717027+0000","last_active":"2023-10-30T13:23:21.005422+0000","last_peered":"2023-10-30T13:23:21.005422+0000","last_clean":"2023-10-30T13:23:21.005422+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T13:23:21.005422+0000","last_undegraded":"2023-10-30T13:23:21.005422+0000","last_fullsized":"2023-10-30T13:23:21.005422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:38.788372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677247+0000","last_change":"2023-10-30T13:23:19.706002+0000","last_active":"2023-10-30T13:23:20.677247+0000","last_peered":"2023-10-30T13:23:20.677247+0000","last_clean":"2023-10-30T13:23:20.677247+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T13:23:20.677247+0000","last_undegraded":"2023-10-30T13:23:20.677247+0000","last_fullsized":"2023-10-30T13:23:20.677247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:46.882474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670564+0000","last_change":"2023-10-30T13:23:19.678472+0000","last_active":"2023-10-30T13:23:20.670564+0000","last_peered":"2023-10-30T13:23:20.670564+0000","last_clean":"2023-10-30T13:23:20.670564+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T13:23:20.670564+0000","last_undegraded":"2023-10-30T13:23:20.670564+0000","last_fullsized":"2023-10-30T13:23:20.670564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005470+0000","last_change":"2023-10-30T13:23:19.702155+0000","last_active":"2023-10-30T13:23:21.005470+0000","last_peered":"2023-10-30T13:23:21.005470+0000","last_clean":"2023-10-30T13:23:21.005470+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T13:23:21.005470+0000","last_undegraded":"2023-10-30T13:23:21.005470+0000","last_fullsized":"2023-10-30T13:23:21.005470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:30.131186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107903+0000","last_change":"2023-10-30T13:23:19.687820+0000","last_active":"2023-10-30T13:23:21.107903+0000","last_peered":"2023-10-30T13:23:21.107903+0000","last_clean":"2023-10-30T13:23:21.107903+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T13:23:21.107903+0000","last_undegraded":"2023-10-30T13:23:21.107903+0000","last_fullsized":"2023-10-30T13:23:21.107903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:46.088882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612897+0000","last_change":"2023-10-30T13:23:19.702442+0000","last_active":"2023-10-30T13:23:21.612897+0000","last_peered":"2023-10-30T13:23:21.612897+0000","last_clean":"2023-10-30T13:23:21.612897+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T13:23:21.612897+0000","last_undegraded":"2023-10-30T13:23:21.612897+0000","last_fullsized":"2023-10-30T13:23:21.612897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:58:50.083326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.608295+0000","last_change":"2023-10-30T13:23:19.711364+0000","last_active":"2023-10-30T13:23:21.608295+0000","last_peered":"2023-10-30T13:23:21.608295+0000","last_clean":"2023-10-30T13:23:21.608295+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T13:23:21.608295+0000","last_undegraded":"2023-10-30T13:23:21.608295+0000","last_fullsized":"2023-10-30T13:23:21.608295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:15.250397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674450+0000","last_change":"2023-10-30T13:23:19.698485+0000","last_active":"2023-10-30T13:23:20.674450+0000","last_peered":"2023-10-30T13:23:20.674450+0000","last_clean":"2023-10-30T13:23:20.674450+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T13:23:20.674450+0000","last_undegraded":"2023-10-30T13:23:20.674450+0000","last_fullsized":"2023-10-30T13:23:20.674450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675615+0000","last_change":"2023-10-30T13:23:19.703223+0000","last_active":"2023-10-30T13:23:20.675615+0000","last_peered":"2023-10-30T13:23:20.675615+0000","last_clean":"2023-10-30T13:23:20.675615+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T13:23:20.675615+0000","last_undegraded":"2023-10-30T13:23:20.675615+0000","last_fullsized":"2023-10-30T13:23:20.675615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:28.065471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.036846+0000","last_change":"2023-10-30T13:23:18.963933+0000","last_active":"2023-10-30T13:23:30.036846+0000","last_peered":"2023-10-30T13:23:30.036846+0000","last_clean":"2023-10-30T13:23:30.036846+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T13:23:30.036846+0000","last_undegraded":"2023-10-30T13:23:30.036846+0000","last_fullsized":"2023-10-30T13:23:30.036846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:48:58.196567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107913+0000","last_change":"2023-10-30T13:23:19.717819+0000","last_active":"2023-10-30T13:23:21.107913+0000","last_peered":"2023-10-30T13:23:21.107913+0000","last_clean":"2023-10-30T13:23:21.107913+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T13:23:21.107913+0000","last_undegraded":"2023-10-30T13:23:21.107913+0000","last_fullsized":"2023-10-30T13:23:21.107913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:15.403400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.737858+0000","last_change":"2023-10-30T13:23:18.976615+0000","last_active":"2023-10-30T13:23:35.737858+0000","last_peered":"2023-10-30T13:23:35.737858+0000","last_clean":"2023-10-30T13:23:35.737858+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T13:23:35.737858+0000","last_undegraded":"2023-10-30T13:23:35.737858+0000","last_fullsized":"2023-10-30T13:23:35.737858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:59.048396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.733720+0000","last_change":"2023-10-30T13:23:18.977357+0000","last_active":"2023-10-30T13:23:35.733720+0000","last_peered":"2023-10-30T13:23:35.733720+0000","last_clean":"2023-10-30T13:23:35.733720+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T13:23:35.733720+0000","last_undegraded":"2023-10-30T13:23:35.733720+0000","last_fullsized":"2023-10-30T13:23:35.733720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:53.502681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005000+0000","last_change":"2023-10-30T13:23:19.689197+0000","last_active":"2023-10-30T13:23:21.005000+0000","last_peered":"2023-10-30T13:23:21.005000+0000","last_clean":"2023-10-30T13:23:21.005000+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T13:23:21.005000+0000","last_undegraded":"2023-10-30T13:23:21.005000+0000","last_fullsized":"2023-10-30T13:23:21.005000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:59:47.946429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611057+0000","last_change":"2023-10-30T13:23:18.989299+0000","last_active":"2023-10-30T13:23:21.611057+0000","last_peered":"2023-10-30T13:23:21.611057+0000","last_clean":"2023-10-30T13:23:21.611057+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T13:23:21.611057+0000","last_undegraded":"2023-10-30T13:23:21.611057+0000","last_fullsized":"2023-10-30T13:23:21.611057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:44.510184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670217+0000","last_change":"2023-10-30T13:23:19.679342+0000","last_active":"2023-10-30T13:23:20.670217+0000","last_peered":"2023-10-30T13:23:20.670217+0000","last_clean":"2023-10-30T13:23:20.670217+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T13:23:20.670217+0000","last_undegraded":"2023-10-30T13:23:20.670217+0000","last_fullsized":"2023-10-30T13:23:20.670217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.028003+0000","last_change":"2023-10-30T13:23:18.989515+0000","last_active":"2023-10-30T13:23:30.028003+0000","last_peered":"2023-10-30T13:23:30.028003+0000","last_clean":"2023-10-30T13:23:30.028003+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T13:23:30.028003+0000","last_undegraded":"2023-10-30T13:23:30.028003+0000","last_fullsized":"2023-10-30T13:23:30.028003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:35.133762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612308+0000","last_change":"2023-10-30T13:23:19.732859+0000","last_active":"2023-10-30T13:23:21.612308+0000","last_peered":"2023-10-30T13:23:21.612308+0000","last_clean":"2023-10-30T13:23:21.612308+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T13:23:21.612308+0000","last_undegraded":"2023-10-30T13:23:21.612308+0000","last_fullsized":"2023-10-30T13:23:21.612308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:27.629521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.027734+0000","last_change":"2023-10-30T13:23:18.991454+0000","last_active":"2023-10-30T13:23:30.027734+0000","last_peered":"2023-10-30T13:23:30.027734+0000","last_clean":"2023-10-30T13:23:30.027734+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T13:23:30.027734+0000","last_undegraded":"2023-10-30T13:23:30.027734+0000","last_fullsized":"2023-10-30T13:23:30.027734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:13.450999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107915+0000","last_change":"2023-10-30T13:23:19.722347+0000","last_active":"2023-10-30T13:23:21.107915+0000","last_peered":"2023-10-30T13:23:21.107915+0000","last_clean":"2023-10-30T13:23:21.107915+0000","last_became_active":"2023-10-30T13:23:19.722077+0000","last_became_peered":"2023-10-30T13:23:19.722077+0000","last_unstale":"2023-10-30T13:23:21.107915+0000","last_undegraded":"2023-10-30T13:23:21.107915+0000","last_fullsized":"2023-10-30T13:23:21.107915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:23:21.544691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.967593+0000","last_change":"2023-10-30T13:23:18.982337+0000","last_active":"2023-10-30T13:23:24.967593+0000","last_peered":"2023-10-30T13:23:24.967593+0000","last_clean":"2023-10-30T13:23:24.967593+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T13:23:24.967593+0000","last_undegraded":"2023-10-30T13:23:24.967593+0000","last_fullsized":"2023-10-30T13:23:24.967593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:09:39.996740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676299+0000","last_change":"2023-10-30T13:23:19.706643+0000","last_active":"2023-10-30T13:23:20.676299+0000","last_peered":"2023-10-30T13:23:20.676299+0000","last_clean":"2023-10-30T13:23:20.676299+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T13:23:20.676299+0000","last_undegraded":"2023-10-30T13:23:20.676299+0000","last_fullsized":"2023-10-30T13:23:20.676299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:29.238430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'11","reported_seq":23,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:43.022865+0000","last_change":"2023-10-30T13:23:18.969882+0000","last_active":"2023-10-30T13:23:43.022865+0000","last_peered":"2023-10-30T13:23:43.022865+0000","last_clean":"2023-10-30T13:23:43.022865+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T13:23:43.022865+0000","last_undegraded":"2023-10-30T13:23:43.022865+0000","last_fullsized":"2023-10-30T13:23:43.022865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:17.093448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611588+0000","last_change":"2023-10-30T13:23:19.728295+0000","last_active":"2023-10-30T13:23:21.611588+0000","last_peered":"2023-10-30T13:23:21.611588+0000","last_clean":"2023-10-30T13:23:21.611588+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T13:23:21.611588+0000","last_undegraded":"2023-10-30T13:23:21.611588+0000","last_fullsized":"2023-10-30T13:23:21.611588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:37:54.545985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.732729+0000","last_change":"2023-10-30T13:23:18.971632+0000","last_active":"2023-10-30T13:23:35.732729+0000","last_peered":"2023-10-30T13:23:35.732729+0000","last_clean":"2023-10-30T13:23:35.732729+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T13:23:35.732729+0000","last_undegraded":"2023-10-30T13:23:35.732729+0000","last_fullsized":"2023-10-30T13:23:35.732729+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:54.428499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107653+0000","last_change":"2023-10-30T13:23:19.718168+0000","last_active":"2023-10-30T13:23:21.107653+0000","last_peered":"2023-10-30T13:23:21.107653+0000","last_clean":"2023-10-30T13:23:21.107653+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T13:23:21.107653+0000","last_undegraded":"2023-10-30T13:23:21.107653+0000","last_fullsized":"2023-10-30T13:23:21.107653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:54.832358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.029264+0000","last_change":"2023-10-30T13:23:18.978420+0000","last_active":"2023-10-30T13:23:30.029264+0000","last_peered":"2023-10-30T13:23:30.029264+0000","last_clean":"2023-10-30T13:23:30.029264+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T13:23:30.029264+0000","last_undegraded":"2023-10-30T13:23:30.029264+0000","last_fullsized":"2023-10-30T13:23:30.029264+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:10:44.511282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611634+0000","last_change":"2023-10-30T13:23:19.721434+0000","last_active":"2023-10-30T13:23:21.611634+0000","last_peered":"2023-10-30T13:23:21.611634+0000","last_clean":"2023-10-30T13:23:21.611634+0000","last_became_active":"2023-10-30T13:23:19.720704+0000","last_became_peered":"2023-10-30T13:23:19.720704+0000","last_unstale":"2023-10-30T13:23:21.611634+0000","last_undegraded":"2023-10-30T13:23:21.611634+0000","last_fullsized":"2023-10-30T13:23:21.611634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:45.728513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.737551+0000","last_change":"2023-10-30T13:23:18.972707+0000","last_active":"2023-10-30T13:23:35.737551+0000","last_peered":"2023-10-30T13:23:35.737551+0000","last_clean":"2023-10-30T13:23:35.737551+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T13:23:35.737551+0000","last_undegraded":"2023-10-30T13:23:35.737551+0000","last_fullsized":"2023-10-30T13:23:35.737551+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674810+0000","last_change":"2023-10-30T13:23:19.698274+0000","last_active":"2023-10-30T13:23:20.674810+0000","last_peered":"2023-10-30T13:23:20.674810+0000","last_clean":"2023-10-30T13:23:20.674810+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T13:23:20.674810+0000","last_undegraded":"2023-10-30T13:23:20.674810+0000","last_fullsized":"2023-10-30T13:23:20.674810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'15","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.734036+0000","last_change":"2023-10-30T13:23:18.948296+0000","last_active":"2023-10-30T13:23:35.734036+0000","last_peered":"2023-10-30T13:23:35.734036+0000","last_clean":"2023-10-30T13:23:35.734036+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T13:23:35.734036+0000","last_undegraded":"2023-10-30T13:23:35.734036+0000","last_fullsized":"2023-10-30T13:23:35.734036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:55.308329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611416+0000","last_change":"2023-10-30T13:23:19.728647+0000","last_active":"2023-10-30T13:23:21.611416+0000","last_peered":"2023-10-30T13:23:21.611416+0000","last_clean":"2023-10-30T13:23:21.611416+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T13:23:21.611416+0000","last_undegraded":"2023-10-30T13:23:21.611416+0000","last_fullsized":"2023-10-30T13:23:21.611416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:09.123809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.963649+0000","last_change":"2023-10-30T13:23:18.962343+0000","last_active":"2023-10-30T13:23:24.963649+0000","last_peered":"2023-10-30T13:23:24.963649+0000","last_clean":"2023-10-30T13:23:24.963649+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T13:23:24.963649+0000","last_undegraded":"2023-10-30T13:23:24.963649+0000","last_fullsized":"2023-10-30T13:23:24.963649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:36.082164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611928+0000","last_change":"2023-10-30T13:23:19.720770+0000","last_active":"2023-10-30T13:23:21.611928+0000","last_peered":"2023-10-30T13:23:21.611928+0000","last_clean":"2023-10-30T13:23:21.611928+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T13:23:21.611928+0000","last_undegraded":"2023-10-30T13:23:21.611928+0000","last_fullsized":"2023-10-30T13:23:21.611928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:37.378858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.962456+0000","last_change":"2023-10-30T13:23:18.949081+0000","last_active":"2023-10-30T13:23:24.962456+0000","last_peered":"2023-10-30T13:23:24.962456+0000","last_clean":"2023-10-30T13:23:24.962456+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T13:23:24.962456+0000","last_undegraded":"2023-10-30T13:23:24.962456+0000","last_fullsized":"2023-10-30T13:23:24.962456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:00.356885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613552+0000","last_change":"2023-10-30T13:23:19.724180+0000","last_active":"2023-10-30T13:23:21.613552+0000","last_peered":"2023-10-30T13:23:21.613552+0000","last_clean":"2023-10-30T13:23:21.613552+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T13:23:21.613552+0000","last_undegraded":"2023-10-30T13:23:21.613552+0000","last_fullsized":"2023-10-30T13:23:21.613552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:14.049454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.981649+0000","last_change":"2023-10-30T13:23:18.970780+0000","last_active":"2023-10-30T13:23:20.981649+0000","last_peered":"2023-10-30T13:23:20.981649+0000","last_clean":"2023-10-30T13:23:20.981649+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T13:23:20.981649+0000","last_undegraded":"2023-10-30T13:23:20.981649+0000","last_fullsized":"2023-10-30T13:23:20.981649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:38:51.113629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611979+0000","last_change":"2023-10-30T13:23:19.704834+0000","last_active":"2023-10-30T13:23:21.611979+0000","last_peered":"2023-10-30T13:23:21.611979+0000","last_clean":"2023-10-30T13:23:21.611979+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T13:23:21.611979+0000","last_undegraded":"2023-10-30T13:23:21.611979+0000","last_fullsized":"2023-10-30T13:23:21.611979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:11.266083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.734210+0000","last_change":"2023-10-30T13:23:18.977920+0000","last_active":"2023-10-30T13:23:35.734210+0000","last_peered":"2023-10-30T13:23:35.734210+0000","last_clean":"2023-10-30T13:23:35.734210+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T13:23:35.734210+0000","last_undegraded":"2023-10-30T13:23:35.734210+0000","last_fullsized":"2023-10-30T13:23:35.734210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:07:59.139948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611883+0000","last_change":"2023-10-30T13:23:19.724497+0000","last_active":"2023-10-30T13:23:21.611883+0000","last_peered":"2023-10-30T13:23:21.611883+0000","last_clean":"2023-10-30T13:23:21.611883+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T13:23:21.611883+0000","last_undegraded":"2023-10-30T13:23:21.611883+0000","last_fullsized":"2023-10-30T13:23:21.611883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:23.836868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.686506+0000","last_change":"2023-10-30T13:23:08.208031+0000","last_active":"2023-10-30T13:23:19.686506+0000","last_peered":"2023-10-30T13:23:19.686506+0000","last_clean":"2023-10-30T13:23:19.686506+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T13:23:19.686506+0000","last_undegraded":"2023-10-30T13:23:19.686506+0000","last_fullsized":"2023-10-30T13:23:19.686506+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_clean_scrub_stamp":"2023-10-30T13:23:07.179866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:22.691367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.027055+0000","last_change":"2023-10-30T13:23:18.979328+0000","last_active":"2023-10-30T13:23:30.027055+0000","last_peered":"2023-10-30T13:23:30.027055+0000","last_clean":"2023-10-30T13:23:30.027055+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T13:23:30.027055+0000","last_undegraded":"2023-10-30T13:23:30.027055+0000","last_fullsized":"2023-10-30T13:23:30.027055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:04.942593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005198+0000","last_change":"2023-10-30T13:23:19.711222+0000","last_active":"2023-10-30T13:23:21.005198+0000","last_peered":"2023-10-30T13:23:21.005198+0000","last_clean":"2023-10-30T13:23:21.005198+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T13:23:21.005198+0000","last_undegraded":"2023-10-30T13:23:21.005198+0000","last_fullsized":"2023-10-30T13:23:21.005198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:07.213498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.731662+0000","last_change":"2023-10-30T13:23:18.985318+0000","last_active":"2023-10-30T13:23:35.731662+0000","last_peered":"2023-10-30T13:23:35.731662+0000","last_clean":"2023-10-30T13:23:35.731662+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T13:23:35.731662+0000","last_undegraded":"2023-10-30T13:23:35.731662+0000","last_fullsized":"2023-10-30T13:23:35.731662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:16.754084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107580+0000","last_change":"2023-10-30T13:23:19.717938+0000","last_active":"2023-10-30T13:23:21.107580+0000","last_peered":"2023-10-30T13:23:21.107580+0000","last_clean":"2023-10-30T13:23:21.107580+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T13:23:21.107580+0000","last_undegraded":"2023-10-30T13:23:21.107580+0000","last_fullsized":"2023-10-30T13:23:21.107580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:56:01.624954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.732971+0000","last_change":"2023-10-30T13:23:18.949565+0000","last_active":"2023-10-30T13:23:35.732971+0000","last_peered":"2023-10-30T13:23:35.732971+0000","last_clean":"2023-10-30T13:23:35.732971+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T13:23:35.732971+0000","last_undegraded":"2023-10-30T13:23:35.732971+0000","last_fullsized":"2023-10-30T13:23:35.732971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:55:46.029368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670291+0000","last_change":"2023-10-30T13:23:19.678037+0000","last_active":"2023-10-30T13:23:20.670291+0000","last_peered":"2023-10-30T13:23:20.670291+0000","last_clean":"2023-10-30T13:23:20.670291+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T13:23:20.670291+0000","last_undegraded":"2023-10-30T13:23:20.670291+0000","last_fullsized":"2023-10-30T13:23:20.670291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613192+0000","last_change":"2023-10-30T13:23:18.984535+0000","last_active":"2023-10-30T13:23:21.613192+0000","last_peered":"2023-10-30T13:23:21.613192+0000","last_clean":"2023-10-30T13:23:21.613192+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T13:23:21.613192+0000","last_undegraded":"2023-10-30T13:23:21.613192+0000","last_fullsized":"2023-10-30T13:23:21.613192+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:28.405405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675980+0000","last_change":"2023-10-30T13:23:19.691867+0000","last_active":"2023-10-30T13:23:20.675980+0000","last_peered":"2023-10-30T13:23:20.675980+0000","last_clean":"2023-10-30T13:23:20.675980+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T13:23:20.675980+0000","last_undegraded":"2023-10-30T13:23:20.675980+0000","last_fullsized":"2023-10-30T13:23:20.675980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:02.276042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670494+0000","last_change":"2023-10-30T13:23:18.958210+0000","last_active":"2023-10-30T13:23:20.670494+0000","last_peered":"2023-10-30T13:23:20.670494+0000","last_clean":"2023-10-30T13:23:20.670494+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T13:23:20.670494+0000","last_undegraded":"2023-10-30T13:23:20.670494+0000","last_fullsized":"2023-10-30T13:23:20.670494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:34.653005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613343+0000","last_change":"2023-10-30T13:23:19.713174+0000","last_active":"2023-10-30T13:23:21.613343+0000","last_peered":"2023-10-30T13:23:21.613343+0000","last_clean":"2023-10-30T13:23:21.613343+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T13:23:21.613343+0000","last_undegraded":"2023-10-30T13:23:21.613343+0000","last_fullsized":"2023-10-30T13:23:21.613343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:43.400223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612618+0000","last_change":"2023-10-30T13:23:18.985185+0000","last_active":"2023-10-30T13:23:21.612618+0000","last_peered":"2023-10-30T13:23:21.612618+0000","last_clean":"2023-10-30T13:23:21.612618+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T13:23:21.612618+0000","last_undegraded":"2023-10-30T13:23:21.612618+0000","last_fullsized":"2023-10-30T13:23:21.612618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:36.230072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670523+0000","last_change":"2023-10-30T13:23:19.680185+0000","last_active":"2023-10-30T13:23:20.670523+0000","last_peered":"2023-10-30T13:23:20.670523+0000","last_clean":"2023-10-30T13:23:20.670523+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T13:23:20.670523+0000","last_undegraded":"2023-10-30T13:23:20.670523+0000","last_fullsized":"2023-10-30T13:23:20.670523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.733893+0000","last_change":"2023-10-30T13:23:18.978085+0000","last_active":"2023-10-30T13:23:35.733893+0000","last_peered":"2023-10-30T13:23:35.733893+0000","last_clean":"2023-10-30T13:23:35.733893+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T13:23:35.733893+0000","last_undegraded":"2023-10-30T13:23:35.733893+0000","last_fullsized":"2023-10-30T13:23:35.733893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:47.588972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005467+0000","last_change":"2023-10-30T13:23:19.704291+0000","last_active":"2023-10-30T13:23:21.005467+0000","last_peered":"2023-10-30T13:23:21.005467+0000","last_clean":"2023-10-30T13:23:21.005467+0000","last_became_active":"2023-10-30T13:23:19.704066+0000","last_became_peered":"2023-10-30T13:23:19.704066+0000","last_unstale":"2023-10-30T13:23:21.005467+0000","last_undegraded":"2023-10-30T13:23:21.005467+0000","last_fullsized":"2023-10-30T13:23:21.005467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:13.754194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.027200+0000","last_change":"2023-10-30T13:23:18.970566+0000","last_active":"2023-10-30T13:23:30.027200+0000","last_peered":"2023-10-30T13:23:30.027200+0000","last_clean":"2023-10-30T13:23:30.027200+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T13:23:30.027200+0000","last_undegraded":"2023-10-30T13:23:30.027200+0000","last_fullsized":"2023-10-30T13:23:30.027200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:03.851889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107903+0000","last_change":"2023-10-30T13:23:19.685481+0000","last_active":"2023-10-30T13:23:21.107903+0000","last_peered":"2023-10-30T13:23:21.107903+0000","last_clean":"2023-10-30T13:23:21.107903+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T13:23:21.107903+0000","last_undegraded":"2023-10-30T13:23:21.107903+0000","last_fullsized":"2023-10-30T13:23:21.107903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:33:50.672215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.037430+0000","last_change":"2023-10-30T13:23:18.986394+0000","last_active":"2023-10-30T13:23:30.037430+0000","last_peered":"2023-10-30T13:23:30.037430+0000","last_clean":"2023-10-30T13:23:30.037430+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T13:23:30.037430+0000","last_undegraded":"2023-10-30T13:23:30.037430+0000","last_fullsized":"2023-10-30T13:23:30.037430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:59.986981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675944+0000","last_change":"2023-10-30T13:23:19.681239+0000","last_active":"2023-10-30T13:23:20.675944+0000","last_peered":"2023-10-30T13:23:20.675944+0000","last_clean":"2023-10-30T13:23:20.675944+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T13:23:20.675944+0000","last_undegraded":"2023-10-30T13:23:20.675944+0000","last_fullsized":"2023-10-30T13:23:20.675944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:00.121240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.036720+0000","last_change":"2023-10-30T13:23:18.963905+0000","last_active":"2023-10-30T13:23:30.036720+0000","last_peered":"2023-10-30T13:23:30.036720+0000","last_clean":"2023-10-30T13:23:30.036720+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T13:23:30.036720+0000","last_undegraded":"2023-10-30T13:23:30.036720+0000","last_fullsized":"2023-10-30T13:23:30.036720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005091+0000","last_change":"2023-10-30T13:23:19.678697+0000","last_active":"2023-10-30T13:23:21.005091+0000","last_peered":"2023-10-30T13:23:21.005091+0000","last_clean":"2023-10-30T13:23:21.005091+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T13:23:21.005091+0000","last_undegraded":"2023-10-30T13:23:21.005091+0000","last_fullsized":"2023-10-30T13:23:21.005091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:57.442011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.981565+0000","last_change":"2023-10-30T13:23:18.952302+0000","last_active":"2023-10-30T13:23:20.981565+0000","last_peered":"2023-10-30T13:23:20.981565+0000","last_clean":"2023-10-30T13:23:20.981565+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T13:23:20.981565+0000","last_undegraded":"2023-10-30T13:23:20.981565+0000","last_fullsized":"2023-10-30T13:23:20.981565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:12.827417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005323+0000","last_change":"2023-10-30T13:23:19.688716+0000","last_active":"2023-10-30T13:23:21.005323+0000","last_peered":"2023-10-30T13:23:21.005323+0000","last_clean":"2023-10-30T13:23:21.005323+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T13:23:21.005323+0000","last_undegraded":"2023-10-30T13:23:21.005323+0000","last_fullsized":"2023-10-30T13:23:21.005323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:26:04.209304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.022861+0000","last_change":"2023-10-30T13:23:18.984912+0000","last_active":"2023-10-30T13:23:30.022861+0000","last_peered":"2023-10-30T13:23:30.022861+0000","last_clean":"2023-10-30T13:23:30.022861+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T13:23:30.022861+0000","last_undegraded":"2023-10-30T13:23:30.022861+0000","last_fullsized":"2023-10-30T13:23:30.022861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:27:16.205042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674560+0000","last_change":"2023-10-30T13:23:19.718922+0000","last_active":"2023-10-30T13:23:20.674560+0000","last_peered":"2023-10-30T13:23:20.674560+0000","last_clean":"2023-10-30T13:23:20.674560+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T13:23:20.674560+0000","last_undegraded":"2023-10-30T13:23:20.674560+0000","last_fullsized":"2023-10-30T13:23:20.674560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612299+0000","last_change":"2023-10-30T13:23:19.732322+0000","last_active":"2023-10-30T13:23:21.612299+0000","last_peered":"2023-10-30T13:23:21.612299+0000","last_clean":"2023-10-30T13:23:21.612299+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T13:23:21.612299+0000","last_undegraded":"2023-10-30T13:23:21.612299+0000","last_fullsized":"2023-10-30T13:23:21.612299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:21.479138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.968310+0000","last_change":"2023-10-30T13:23:18.993372+0000","last_active":"2023-10-30T13:23:24.968310+0000","last_peered":"2023-10-30T13:23:24.968310+0000","last_clean":"2023-10-30T13:23:24.968310+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T13:23:24.968310+0000","last_undegraded":"2023-10-30T13:23:24.968310+0000","last_fullsized":"2023-10-30T13:23:24.968310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:41.370678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611741+0000","last_change":"2023-10-30T13:23:19.732769+0000","last_active":"2023-10-30T13:23:21.611741+0000","last_peered":"2023-10-30T13:23:21.611741+0000","last_clean":"2023-10-30T13:23:21.611741+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T13:23:21.611741+0000","last_undegraded":"2023-10-30T13:23:21.611741+0000","last_fullsized":"2023-10-30T13:23:21.611741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:56.378098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.732313+0000","last_change":"2023-10-30T13:23:18.987633+0000","last_active":"2023-10-30T13:23:35.732313+0000","last_peered":"2023-10-30T13:23:35.732313+0000","last_clean":"2023-10-30T13:23:35.732313+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T13:23:35.732313+0000","last_undegraded":"2023-10-30T13:23:35.732313+0000","last_fullsized":"2023-10-30T13:23:35.732313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:09:33.324898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107906+0000","last_change":"2023-10-30T13:23:19.722311+0000","last_active":"2023-10-30T13:23:21.107906+0000","last_peered":"2023-10-30T13:23:21.107906+0000","last_clean":"2023-10-30T13:23:21.107906+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T13:23:21.107906+0000","last_undegraded":"2023-10-30T13:23:21.107906+0000","last_fullsized":"2023-10-30T13:23:21.107906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:44.204074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.962089+0000","last_change":"2023-10-30T13:23:18.958470+0000","last_active":"2023-10-30T13:23:24.962089+0000","last_peered":"2023-10-30T13:23:24.962089+0000","last_clean":"2023-10-30T13:23:24.962089+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T13:23:24.962089+0000","last_undegraded":"2023-10-30T13:23:24.962089+0000","last_fullsized":"2023-10-30T13:23:24.962089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:21.507744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005410+0000","last_change":"2023-10-30T13:23:19.699742+0000","last_active":"2023-10-30T13:23:21.005410+0000","last_peered":"2023-10-30T13:23:21.005410+0000","last_clean":"2023-10-30T13:23:21.005410+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T13:23:21.005410+0000","last_undegraded":"2023-10-30T13:23:21.005410+0000","last_fullsized":"2023-10-30T13:23:21.005410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:13.504378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'9","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.735807+0000","last_change":"2023-10-30T13:23:18.980952+0000","last_active":"2023-10-30T13:23:35.735807+0000","last_peered":"2023-10-30T13:23:35.735807+0000","last_clean":"2023-10-30T13:23:35.735807+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T13:23:35.735807+0000","last_undegraded":"2023-10-30T13:23:35.735807+0000","last_fullsized":"2023-10-30T13:23:35.735807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:44.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6467,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":12934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":819200,"data_stored":662582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":74433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2569999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1379999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":71612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":74467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":221184,"data_stored":70697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":71842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":811008,"data_stored":660907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:23:46.938 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:23:47.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f752610f640 1 Processor -- start 2023-10-30T13:23:47.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f752610f640 1 -- start start 2023-10-30T13:23:47.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f752610f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f752006d290 0x7f752006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f752610f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75200c6c10 con 0x7f752015a410 2023-10-30T13:23:47.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f752610f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75200c6ef0 con 0x7f7520069e90 2023-10-30T13:23:47.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f752610f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75200c8570 con 0x7f752006d290 2023-10-30T13:23:47.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f752015a410 0x7f752006cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58946/0 (socket says 172.21.15.46:58946) 2023-10-30T13:23:47.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751effd640 1 -- 172.21.15.46:0/4149691035 learned_addr learned my addr 172.21.15.46:0/4149691035 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:47.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751ffff640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f752006d290 0x7f752006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281659204 0 0) 0x7f75200c6c10 con 0x7f752015a410 2023-10-30T13:23:47.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7508003530 con 0x7f752015a410 2023-10-30T13:23:47.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522324460 0 0) 0x7f75200c6ef0 con 0x7f7520069e90 2023-10-30T13:23:47.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75200c6c10 con 0x7f7520069e90 2023-10-30T13:23:47.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 993245068 0 0) 0x7f7508003530 con 0x7f752015a410 2023-10-30T13:23:47.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f752006d290 msgr2=0x7f752006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f752006d290 0x7f752006a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 >> v1:172.21.15.154:6789/0 conn(0x7f7520069e90 legacy=0x7f75200c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75200c71d0 con 0x7f752015a410 2023-10-30T13:23:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f7514001680 con 0x7f752015a410 2023-10-30T13:23:47.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 >> v1:172.21.15.46:6789/0 conn(0x7f752015a410 legacy=0x7f752006cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f752006d290 0x7f7508003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75080040a0 0x7f75080047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7508004d10 0x7f7508005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75200c8570 con 0x7f75080040a0 2023-10-30T13:23:47.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750800ba70 con 0x7f7508004d10 2023-10-30T13:23:47.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750800bc30 con 0x7f752006d290 2023-10-30T13:23:47.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.108+0000 7f751effd640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75080040a0 0x7f75080047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751f7fe640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f752006d290 0x7f7508003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751ffff640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7508004d10 0x7f7508005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751effd640 1 -- 172.21.15.46:0/4149691035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f752006d290 msgr2=0x7f7508003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:47.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751effd640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f752006d290 0x7f7508003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751effd640 1 -- 172.21.15.46:0/4149691035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7508004d10 msgr2=0x7f7508005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:47.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751effd640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7508004d10 0x7f7508005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751effd640 1 -- 172.21.15.46:0/4149691035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7508003530 con 0x7f75080040a0 2023-10-30T13:23:47.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751ffff640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7508004d10 0x7f7508005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:47.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751effd640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75080040a0 0x7f75080047d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f751400c160 tx=0x7f751401cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d289a8214cf8bea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:47.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75140058e0 con 0x7f75080040a0 2023-10-30T13:23:47.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7514005a60 con 0x7f75080040a0 2023-10-30T13:23:47.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751e7fc640 1 -- 172.21.15.46:0/4149691035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7514005e50 con 0x7f75080040a0 2023-10-30T13:23:47.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f752610f640 1 -- 172.21.15.46:0/4149691035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75080040a0 msgr2=0x7f75080047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f752610f640 1 --2- 172.21.15.46:0/4149691035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75080040a0 0x7f75080047d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f751400c160 tx=0x7f751401cba0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f751ffff640 1 -- 172.21.15.46:0/4149691035 reap_dead start 2023-10-30T13:23:47.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f752610f640 1 -- 172.21.15.46:0/4149691035 shutdown_connections 2023-10-30T13:23:47.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f752610f640 1 -- 172.21.15.46:0/4149691035 >> 172.21.15.46:0/4149691035 conn(0x7f752006dfd0 msgr2=0x7f752006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:47.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f752610f640 1 -- 172.21.15.46:0/4149691035 shutdown_connections 2023-10-30T13:23:47.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.112+0000 7f752610f640 1 -- 172.21.15.46:0/4149691035 wait complete. 2023-10-30T13:23:47.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f752610f640 1 Processor -- start 2023-10-30T13:23:47.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f752610f640 1 -- start start 2023-10-30T13:23:47.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f752610f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75080040a0 0x7f7520156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f752610f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7520159060 0x7f7520156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f752610f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f752014ef10 0x7f752014f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f752610f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752015dd80 con 0x7f75080040a0 2023-10-30T13:23:47.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f752610f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752014f8d0 con 0x7f752014ef10 2023-10-30T13:23:47.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f752610f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752015d7c0 con 0x7f7520159060 2023-10-30T13:23:47.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f751f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75080040a0 0x7f7520156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f751f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75080040a0 0x7f7520156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47338/0 (socket says 172.21.15.46:47338) 2023-10-30T13:23:47.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f751f7fe640 1 -- 172.21.15.46:0/1049248948 learned_addr learned my addr 172.21.15.46:0/1049248948 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:47.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f751ffff640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f752014ef10 0x7f752014f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f751effd640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7520159060 0x7f7520156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f751f7fe640 1 -- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7520159060 msgr2=0x7f7520156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f751f7fe640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7520159060 0x7f7520156e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f751f7fe640 1 -- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f752014ef10 msgr2=0x7f752014f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f751f7fe640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f752014ef10 0x7f752014f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f751f7fe640 1 -- 172.21.15.46:0/1049248948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752015daa0 con 0x7f75080040a0 2023-10-30T13:23:47.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f751f7fe640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75080040a0 0x7f7520156950 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7510011ee0 tx=0x7f751000f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:47.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f7507fff640 1 -- 172.21.15.46:0/1049248948 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510015610 con 0x7f75080040a0 2023-10-30T13:23:47.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f7507fff640 1 -- 172.21.15.46:0/1049248948 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7510015770 con 0x7f75080040a0 2023-10-30T13:23:47.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f7507fff640 1 -- 172.21.15.46:0/1049248948 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510015b20 con 0x7f75080040a0 2023-10-30T13:23:47.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.116+0000 7f752610f640 1 -- 172.21.15.46:0/1049248948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f752014e450 con 0x7f75080040a0 2023-10-30T13:23:47.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.120+0000 7f752610f640 1 -- 172.21.15.46:0/1049248948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f752014ea10 con 0x7f75080040a0 2023-10-30T13:23:47.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.120+0000 7f752610f640 1 -- 172.21.15.46:0/1049248948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752014e2b0 con 0x7f75080040a0 2023-10-30T13:23:47.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.120+0000 7f7507fff640 1 -- 172.21.15.46:0/1049248948 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7510015c80 con 0x7f75080040a0 2023-10-30T13:23:47.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.120+0000 7f7507fff640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f750c0664d0 0x7f750c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.120+0000 7f7507fff640 1 -- 172.21.15.46:0/1049248948 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7510087980 con 0x7f75080040a0 2023-10-30T13:23:47.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.120+0000 7f751effd640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f750c0664d0 0x7f750c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.120+0000 7f751effd640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f750c0664d0 0x7f750c068990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f75140aa000 tx=0x7f7514001a00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:47.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.124+0000 7f7507fff640 1 -- 172.21.15.46:0/1049248948 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f752014e2b0 con 0x7f75080040a0 2023-10-30T13:23:47.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.280+0000 7f752610f640 1 -- 172.21.15.46:0/1049248948 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f75200c7790 con 0x7f750c0664d0 2023-10-30T13:23:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.300+0000 7f7507fff640 1 -- 172.21.15.46:0/1049248948 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264626 (secure 0 0 0) 0x7f7514001500 con 0x7f750c0664d0 2023-10-30T13:23:47.309 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:47.310 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T13:23:47.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.304+0000 7f752610f640 1 -- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f750c0664d0 msgr2=0x7f750c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.304+0000 7f752610f640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f750c0664d0 0x7f750c068990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f75140aa000 tx=0x7f7514001a00 comp rx=0 tx=0).stop 2023-10-30T13:23:47.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.304+0000 7f752610f640 1 -- 172.21.15.46:0/1049248948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75080040a0 msgr2=0x7f7520156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.304+0000 7f752610f640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75080040a0 0x7f7520156950 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7510011ee0 tx=0x7f751000f540 comp rx=0 tx=0).stop 2023-10-30T13:23:47.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.304+0000 7f752610f640 1 -- 172.21.15.46:0/1049248948 shutdown_connections 2023-10-30T13:23:47.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.304+0000 7f752610f640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f750c0664d0 0x7f750c068990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.304+0000 7f752610f640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f752014ef10 0x7f752014f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.304+0000 7f752610f640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7520159060 0x7f7520156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.304+0000 7f752610f640 1 --2- 172.21.15.46:0/1049248948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75080040a0 0x7f7520156950 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.304+0000 7f752610f640 1 -- 172.21.15.46:0/1049248948 >> 172.21.15.46:0/1049248948 conn(0x7f752006dfd0 msgr2=0x7f7520071840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:47.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.304+0000 7f752610f640 1 -- 172.21.15.46:0/1049248948 shutdown_connections 2023-10-30T13:23:47.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.304+0000 7f752610f640 1 -- 172.21.15.46:0/1049248948 wait complete. 2023-10-30T13:23:47.329 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-30T13:23:46.718944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596835,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":287,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235676,"kb_used_data":2908,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739044,"statfs":{"total":773094113280,"available":772852781056,"internally_reserved":0,"allocated":2977792,"data_stored":1757797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2651,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004545"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612886+0000","last_change":"2023-10-30T13:23:19.719271+0000","last_active":"2023-10-30T13:23:21.612886+0000","last_peered":"2023-10-30T13:23:21.612886+0000","last_clean":"2023-10-30T13:23:21.612886+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T13:23:21.612886+0000","last_undegraded":"2023-10-30T13:23:21.612886+0000","last_fullsized":"2023-10-30T13:23:21.612886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:41:37.614073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670716+0000","last_change":"2023-10-30T13:23:19.687152+0000","last_active":"2023-10-30T13:23:20.670716+0000","last_peered":"2023-10-30T13:23:20.670716+0000","last_clean":"2023-10-30T13:23:20.670716+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T13:23:20.670716+0000","last_undegraded":"2023-10-30T13:23:20.670716+0000","last_fullsized":"2023-10-30T13:23:20.670716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107815+0000","last_change":"2023-10-30T13:23:19.689554+0000","last_active":"2023-10-30T13:23:21.107815+0000","last_peered":"2023-10-30T13:23:21.107815+0000","last_clean":"2023-10-30T13:23:21.107815+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T13:23:21.107815+0000","last_undegraded":"2023-10-30T13:23:21.107815+0000","last_fullsized":"2023-10-30T13:23:21.107815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:56:13.795044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611622+0000","last_change":"2023-10-30T13:23:19.714018+0000","last_active":"2023-10-30T13:23:21.611622+0000","last_peered":"2023-10-30T13:23:21.611622+0000","last_clean":"2023-10-30T13:23:21.611622+0000","last_became_active":"2023-10-30T13:23:19.711673+0000","last_became_peered":"2023-10-30T13:23:19.711673+0000","last_unstale":"2023-10-30T13:23:21.611622+0000","last_undegraded":"2023-10-30T13:23:21.611622+0000","last_fullsized":"2023-10-30T13:23:21.611622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:28:16.658840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611546+0000","last_change":"2023-10-30T13:23:19.719999+0000","last_active":"2023-10-30T13:23:21.611546+0000","last_peered":"2023-10-30T13:23:21.611546+0000","last_clean":"2023-10-30T13:23:21.611546+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T13:23:21.611546+0000","last_undegraded":"2023-10-30T13:23:21.611546+0000","last_fullsized":"2023-10-30T13:23:21.611546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:46:31.287013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612300+0000","last_change":"2023-10-30T13:23:19.703474+0000","last_active":"2023-10-30T13:23:21.612300+0000","last_peered":"2023-10-30T13:23:21.612300+0000","last_clean":"2023-10-30T13:23:21.612300+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T13:23:21.612300+0000","last_undegraded":"2023-10-30T13:23:21.612300+0000","last_fullsized":"2023-10-30T13:23:21.612300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:53:58.118821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613234+0000","last_change":"2023-10-30T13:23:19.726985+0000","last_active":"2023-10-30T13:23:21.613234+0000","last_peered":"2023-10-30T13:23:21.613234+0000","last_clean":"2023-10-30T13:23:21.613234+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T13:23:21.613234+0000","last_undegraded":"2023-10-30T13:23:21.613234+0000","last_fullsized":"2023-10-30T13:23:21.613234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:51:13.152339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.004933+0000","last_change":"2023-10-30T13:23:19.714153+0000","last_active":"2023-10-30T13:23:21.004933+0000","last_peered":"2023-10-30T13:23:21.004933+0000","last_clean":"2023-10-30T13:23:21.004933+0000","last_became_active":"2023-10-30T13:23:19.713876+0000","last_became_peered":"2023-10-30T13:23:19.713876+0000","last_unstale":"2023-10-30T13:23:21.004933+0000","last_undegraded":"2023-10-30T13:23:21.004933+0000","last_fullsized":"2023-10-30T13:23:21.004933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:05:13.588780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613220+0000","last_change":"2023-10-30T13:23:19.724583+0000","last_active":"2023-10-30T13:23:21.613220+0000","last_peered":"2023-10-30T13:23:21.613220+0000","last_clean":"2023-10-30T13:23:21.613220+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T13:23:21.613220+0000","last_undegraded":"2023-10-30T13:23:21.613220+0000","last_fullsized":"2023-10-30T13:23:21.613220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:47.936982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677145+0000","last_change":"2023-10-30T13:23:19.704399+0000","last_active":"2023-10-30T13:23:20.677145+0000","last_peered":"2023-10-30T13:23:20.677145+0000","last_clean":"2023-10-30T13:23:20.677145+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T13:23:20.677145+0000","last_undegraded":"2023-10-30T13:23:20.677145+0000","last_fullsized":"2023-10-30T13:23:20.677145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676522+0000","last_change":"2023-10-30T13:23:19.726077+0000","last_active":"2023-10-30T13:23:20.676522+0000","last_peered":"2023-10-30T13:23:20.676522+0000","last_clean":"2023-10-30T13:23:20.676522+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T13:23:20.676522+0000","last_undegraded":"2023-10-30T13:23:20.676522+0000","last_fullsized":"2023-10-30T13:23:20.676522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675959+0000","last_change":"2023-10-30T13:23:19.720995+0000","last_active":"2023-10-30T13:23:20.675959+0000","last_peered":"2023-10-30T13:23:20.675959+0000","last_clean":"2023-10-30T13:23:20.675959+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T13:23:20.675959+0000","last_undegraded":"2023-10-30T13:23:20.675959+0000","last_fullsized":"2023-10-30T13:23:20.675959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:44.490568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005253+0000","last_change":"2023-10-30T13:23:19.692509+0000","last_active":"2023-10-30T13:23:21.005253+0000","last_peered":"2023-10-30T13:23:21.005253+0000","last_clean":"2023-10-30T13:23:21.005253+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T13:23:21.005253+0000","last_undegraded":"2023-10-30T13:23:21.005253+0000","last_fullsized":"2023-10-30T13:23:21.005253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:15:17.956816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107580+0000","last_change":"2023-10-30T13:23:19.685435+0000","last_active":"2023-10-30T13:23:21.107580+0000","last_peered":"2023-10-30T13:23:21.107580+0000","last_clean":"2023-10-30T13:23:21.107580+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T13:23:21.107580+0000","last_undegraded":"2023-10-30T13:23:21.107580+0000","last_fullsized":"2023-10-30T13:23:21.107580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:39.035298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670272+0000","last_change":"2023-10-30T13:23:19.706721+0000","last_active":"2023-10-30T13:23:20.670272+0000","last_peered":"2023-10-30T13:23:20.670272+0000","last_clean":"2023-10-30T13:23:20.670272+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T13:23:20.670272+0000","last_undegraded":"2023-10-30T13:23:20.670272+0000","last_fullsized":"2023-10-30T13:23:20.670272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.610717+0000","last_change":"2023-10-30T13:23:19.710432+0000","last_active":"2023-10-30T13:23:21.610717+0000","last_peered":"2023-10-30T13:23:21.610717+0000","last_clean":"2023-10-30T13:23:21.610717+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T13:23:21.610717+0000","last_undegraded":"2023-10-30T13:23:21.610717+0000","last_fullsized":"2023-10-30T13:23:21.610717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:48:51.992826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612857+0000","last_change":"2023-10-30T13:23:19.741225+0000","last_active":"2023-10-30T13:23:21.612857+0000","last_peered":"2023-10-30T13:23:21.612857+0000","last_clean":"2023-10-30T13:23:21.612857+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T13:23:21.612857+0000","last_undegraded":"2023-10-30T13:23:21.612857+0000","last_fullsized":"2023-10-30T13:23:21.612857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:27.310070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.608570+0000","last_change":"2023-10-30T13:23:19.701024+0000","last_active":"2023-10-30T13:23:21.608570+0000","last_peered":"2023-10-30T13:23:21.608570+0000","last_clean":"2023-10-30T13:23:21.608570+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T13:23:21.608570+0000","last_undegraded":"2023-10-30T13:23:21.608570+0000","last_fullsized":"2023-10-30T13:23:21.608570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:14.460383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612314+0000","last_change":"2023-10-30T13:23:19.723929+0000","last_active":"2023-10-30T13:23:21.612314+0000","last_peered":"2023-10-30T13:23:21.612314+0000","last_clean":"2023-10-30T13:23:21.612314+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T13:23:21.612314+0000","last_undegraded":"2023-10-30T13:23:21.612314+0000","last_fullsized":"2023-10-30T13:23:21.612314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:41.023731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674806+0000","last_change":"2023-10-30T13:23:19.725101+0000","last_active":"2023-10-30T13:23:20.674806+0000","last_peered":"2023-10-30T13:23:20.674806+0000","last_clean":"2023-10-30T13:23:20.674806+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T13:23:20.674806+0000","last_undegraded":"2023-10-30T13:23:20.674806+0000","last_fullsized":"2023-10-30T13:23:20.674806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107754+0000","last_change":"2023-10-30T13:23:19.721813+0000","last_active":"2023-10-30T13:23:21.107754+0000","last_peered":"2023-10-30T13:23:21.107754+0000","last_clean":"2023-10-30T13:23:21.107754+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T13:23:21.107754+0000","last_undegraded":"2023-10-30T13:23:21.107754+0000","last_fullsized":"2023-10-30T13:23:21.107754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:34:17.212695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612670+0000","last_change":"2023-10-30T13:23:19.724763+0000","last_active":"2023-10-30T13:23:21.612670+0000","last_peered":"2023-10-30T13:23:21.612670+0000","last_clean":"2023-10-30T13:23:21.612670+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T13:23:21.612670+0000","last_undegraded":"2023-10-30T13:23:21.612670+0000","last_fullsized":"2023-10-30T13:23:21.612670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:28:53.428578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005009+0000","last_change":"2023-10-30T13:23:19.716706+0000","last_active":"2023-10-30T13:23:21.005009+0000","last_peered":"2023-10-30T13:23:21.005009+0000","last_clean":"2023-10-30T13:23:21.005009+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T13:23:21.005009+0000","last_undegraded":"2023-10-30T13:23:21.005009+0000","last_fullsized":"2023-10-30T13:23:21.005009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:07.718271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.610920+0000","last_change":"2023-10-30T13:23:19.704791+0000","last_active":"2023-10-30T13:23:21.610920+0000","last_peered":"2023-10-30T13:23:21.610920+0000","last_clean":"2023-10-30T13:23:21.610920+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T13:23:21.610920+0000","last_undegraded":"2023-10-30T13:23:21.610920+0000","last_fullsized":"2023-10-30T13:23:21.610920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:54.750514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005422+0000","last_change":"2023-10-30T13:23:19.717027+0000","last_active":"2023-10-30T13:23:21.005422+0000","last_peered":"2023-10-30T13:23:21.005422+0000","last_clean":"2023-10-30T13:23:21.005422+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T13:23:21.005422+0000","last_undegraded":"2023-10-30T13:23:21.005422+0000","last_fullsized":"2023-10-30T13:23:21.005422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:38.788372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677247+0000","last_change":"2023-10-30T13:23:19.706002+0000","last_active":"2023-10-30T13:23:20.677247+0000","last_peered":"2023-10-30T13:23:20.677247+0000","last_clean":"2023-10-30T13:23:20.677247+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T13:23:20.677247+0000","last_undegraded":"2023-10-30T13:23:20.677247+0000","last_fullsized":"2023-10-30T13:23:20.677247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:46.882474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670564+0000","last_change":"2023-10-30T13:23:19.678472+0000","last_active":"2023-10-30T13:23:20.670564+0000","last_peered":"2023-10-30T13:23:20.670564+0000","last_clean":"2023-10-30T13:23:20.670564+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T13:23:20.670564+0000","last_undegraded":"2023-10-30T13:23:20.670564+0000","last_fullsized":"2023-10-30T13:23:20.670564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005470+0000","last_change":"2023-10-30T13:23:19.702155+0000","last_active":"2023-10-30T13:23:21.005470+0000","last_peered":"2023-10-30T13:23:21.005470+0000","last_clean":"2023-10-30T13:23:21.005470+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T13:23:21.005470+0000","last_undegraded":"2023-10-30T13:23:21.005470+0000","last_fullsized":"2023-10-30T13:23:21.005470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:30.131186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107903+0000","last_change":"2023-10-30T13:23:19.687820+0000","last_active":"2023-10-30T13:23:21.107903+0000","last_peered":"2023-10-30T13:23:21.107903+0000","last_clean":"2023-10-30T13:23:21.107903+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T13:23:21.107903+0000","last_undegraded":"2023-10-30T13:23:21.107903+0000","last_fullsized":"2023-10-30T13:23:21.107903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:46.088882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612897+0000","last_change":"2023-10-30T13:23:19.702442+0000","last_active":"2023-10-30T13:23:21.612897+0000","last_peered":"2023-10-30T13:23:21.612897+0000","last_clean":"2023-10-30T13:23:21.612897+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T13:23:21.612897+0000","last_undegraded":"2023-10-30T13:23:21.612897+0000","last_fullsized":"2023-10-30T13:23:21.612897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:58:50.083326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.608295+0000","last_change":"2023-10-30T13:23:19.711364+0000","last_active":"2023-10-30T13:23:21.608295+0000","last_peered":"2023-10-30T13:23:21.608295+0000","last_clean":"2023-10-30T13:23:21.608295+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T13:23:21.608295+0000","last_undegraded":"2023-10-30T13:23:21.608295+0000","last_fullsized":"2023-10-30T13:23:21.608295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:15.250397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674450+0000","last_change":"2023-10-30T13:23:19.698485+0000","last_active":"2023-10-30T13:23:20.674450+0000","last_peered":"2023-10-30T13:23:20.674450+0000","last_clean":"2023-10-30T13:23:20.674450+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T13:23:20.674450+0000","last_undegraded":"2023-10-30T13:23:20.674450+0000","last_fullsized":"2023-10-30T13:23:20.674450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675615+0000","last_change":"2023-10-30T13:23:19.703223+0000","last_active":"2023-10-30T13:23:20.675615+0000","last_peered":"2023-10-30T13:23:20.675615+0000","last_clean":"2023-10-30T13:23:20.675615+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T13:23:20.675615+0000","last_undegraded":"2023-10-30T13:23:20.675615+0000","last_fullsized":"2023-10-30T13:23:20.675615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:28.065471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.036846+0000","last_change":"2023-10-30T13:23:18.963933+0000","last_active":"2023-10-30T13:23:30.036846+0000","last_peered":"2023-10-30T13:23:30.036846+0000","last_clean":"2023-10-30T13:23:30.036846+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T13:23:30.036846+0000","last_undegraded":"2023-10-30T13:23:30.036846+0000","last_fullsized":"2023-10-30T13:23:30.036846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:48:58.196567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107913+0000","last_change":"2023-10-30T13:23:19.717819+0000","last_active":"2023-10-30T13:23:21.107913+0000","last_peered":"2023-10-30T13:23:21.107913+0000","last_clean":"2023-10-30T13:23:21.107913+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T13:23:21.107913+0000","last_undegraded":"2023-10-30T13:23:21.107913+0000","last_fullsized":"2023-10-30T13:23:21.107913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:15.403400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.737858+0000","last_change":"2023-10-30T13:23:18.976615+0000","last_active":"2023-10-30T13:23:35.737858+0000","last_peered":"2023-10-30T13:23:35.737858+0000","last_clean":"2023-10-30T13:23:35.737858+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T13:23:35.737858+0000","last_undegraded":"2023-10-30T13:23:35.737858+0000","last_fullsized":"2023-10-30T13:23:35.737858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:59.048396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.733720+0000","last_change":"2023-10-30T13:23:18.977357+0000","last_active":"2023-10-30T13:23:35.733720+0000","last_peered":"2023-10-30T13:23:35.733720+0000","last_clean":"2023-10-30T13:23:35.733720+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T13:23:35.733720+0000","last_undegraded":"2023-10-30T13:23:35.733720+0000","last_fullsized":"2023-10-30T13:23:35.733720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:53.502681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005000+0000","last_change":"2023-10-30T13:23:19.689197+0000","last_active":"2023-10-30T13:23:21.005000+0000","last_peered":"2023-10-30T13:23:21.005000+0000","last_clean":"2023-10-30T13:23:21.005000+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T13:23:21.005000+0000","last_undegraded":"2023-10-30T13:23:21.005000+0000","last_fullsized":"2023-10-30T13:23:21.005000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:59:47.946429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611057+0000","last_change":"2023-10-30T13:23:18.989299+0000","last_active":"2023-10-30T13:23:21.611057+0000","last_peered":"2023-10-30T13:23:21.611057+0000","last_clean":"2023-10-30T13:23:21.611057+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T13:23:21.611057+0000","last_undegraded":"2023-10-30T13:23:21.611057+0000","last_fullsized":"2023-10-30T13:23:21.611057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:44.510184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670217+0000","last_change":"2023-10-30T13:23:19.679342+0000","last_active":"2023-10-30T13:23:20.670217+0000","last_peered":"2023-10-30T13:23:20.670217+0000","last_clean":"2023-10-30T13:23:20.670217+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T13:23:20.670217+0000","last_undegraded":"2023-10-30T13:23:20.670217+0000","last_fullsized":"2023-10-30T13:23:20.670217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.028003+0000","last_change":"2023-10-30T13:23:18.989515+0000","last_active":"2023-10-30T13:23:30.028003+0000","last_peered":"2023-10-30T13:23:30.028003+0000","last_clean":"2023-10-30T13:23:30.028003+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T13:23:30.028003+0000","last_undegraded":"2023-10-30T13:23:30.028003+0000","last_fullsized":"2023-10-30T13:23:30.028003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:35.133762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612308+0000","last_change":"2023-10-30T13:23:19.732859+0000","last_active":"2023-10-30T13:23:21.612308+0000","last_peered":"2023-10-30T13:23:21.612308+0000","last_clean":"2023-10-30T13:23:21.612308+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T13:23:21.612308+0000","last_undegraded":"2023-10-30T13:23:21.612308+0000","last_fullsized":"2023-10-30T13:23:21.612308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:27.629521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.027734+0000","last_change":"2023-10-30T13:23:18.991454+0000","last_active":"2023-10-30T13:23:30.027734+0000","last_peered":"2023-10-30T13:23:30.027734+0000","last_clean":"2023-10-30T13:23:30.027734+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T13:23:30.027734+0000","last_undegraded":"2023-10-30T13:23:30.027734+0000","last_fullsized":"2023-10-30T13:23:30.027734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:13.450999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107915+0000","last_change":"2023-10-30T13:23:19.722347+0000","last_active":"2023-10-30T13:23:21.107915+0000","last_peered":"2023-10-30T13:23:21.107915+0000","last_clean":"2023-10-30T13:23:21.107915+0000","last_became_active":"2023-10-30T13:23:19.722077+0000","last_became_peered":"2023-10-30T13:23:19.722077+0000","last_unstale":"2023-10-30T13:23:21.107915+0000","last_undegraded":"2023-10-30T13:23:21.107915+0000","last_fullsized":"2023-10-30T13:23:21.107915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:23:21.544691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.967593+0000","last_change":"2023-10-30T13:23:18.982337+0000","last_active":"2023-10-30T13:23:24.967593+0000","last_peered":"2023-10-30T13:23:24.967593+0000","last_clean":"2023-10-30T13:23:24.967593+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T13:23:24.967593+0000","last_undegraded":"2023-10-30T13:23:24.967593+0000","last_fullsized":"2023-10-30T13:23:24.967593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:09:39.996740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676299+0000","last_change":"2023-10-30T13:23:19.706643+0000","last_active":"2023-10-30T13:23:20.676299+0000","last_peered":"2023-10-30T13:23:20.676299+0000","last_clean":"2023-10-30T13:23:20.676299+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T13:23:20.676299+0000","last_undegraded":"2023-10-30T13:23:20.676299+0000","last_fullsized":"2023-10-30T13:23:20.676299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:29.238430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'11","reported_seq":23,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:43.022865+0000","last_change":"2023-10-30T13:23:18.969882+0000","last_active":"2023-10-30T13:23:43.022865+0000","last_peered":"2023-10-30T13:23:43.022865+0000","last_clean":"2023-10-30T13:23:43.022865+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T13:23:43.022865+0000","last_undegraded":"2023-10-30T13:23:43.022865+0000","last_fullsized":"2023-10-30T13:23:43.022865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:17.093448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611588+0000","last_change":"2023-10-30T13:23:19.728295+0000","last_active":"2023-10-30T13:23:21.611588+0000","last_peered":"2023-10-30T13:23:21.611588+0000","last_clean":"2023-10-30T13:23:21.611588+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T13:23:21.611588+0000","last_undegraded":"2023-10-30T13:23:21.611588+0000","last_fullsized":"2023-10-30T13:23:21.611588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:37:54.545985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.732729+0000","last_change":"2023-10-30T13:23:18.971632+0000","last_active":"2023-10-30T13:23:35.732729+0000","last_peered":"2023-10-30T13:23:35.732729+0000","last_clean":"2023-10-30T13:23:35.732729+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T13:23:35.732729+0000","last_undegraded":"2023-10-30T13:23:35.732729+0000","last_fullsized":"2023-10-30T13:23:35.732729+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:54.428499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107653+0000","last_change":"2023-10-30T13:23:19.718168+0000","last_active":"2023-10-30T13:23:21.107653+0000","last_peered":"2023-10-30T13:23:21.107653+0000","last_clean":"2023-10-30T13:23:21.107653+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T13:23:21.107653+0000","last_undegraded":"2023-10-30T13:23:21.107653+0000","last_fullsized":"2023-10-30T13:23:21.107653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:54.832358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.029264+0000","last_change":"2023-10-30T13:23:18.978420+0000","last_active":"2023-10-30T13:23:30.029264+0000","last_peered":"2023-10-30T13:23:30.029264+0000","last_clean":"2023-10-30T13:23:30.029264+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T13:23:30.029264+0000","last_undegraded":"2023-10-30T13:23:30.029264+0000","last_fullsized":"2023-10-30T13:23:30.029264+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:10:44.511282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611634+0000","last_change":"2023-10-30T13:23:19.721434+0000","last_active":"2023-10-30T13:23:21.611634+0000","last_peered":"2023-10-30T13:23:21.611634+0000","last_clean":"2023-10-30T13:23:21.611634+0000","last_became_active":"2023-10-30T13:23:19.720704+0000","last_became_peered":"2023-10-30T13:23:19.720704+0000","last_unstale":"2023-10-30T13:23:21.611634+0000","last_undegraded":"2023-10-30T13:23:21.611634+0000","last_fullsized":"2023-10-30T13:23:21.611634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:45.728513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.737551+0000","last_change":"2023-10-30T13:23:18.972707+0000","last_active":"2023-10-30T13:23:35.737551+0000","last_peered":"2023-10-30T13:23:35.737551+0000","last_clean":"2023-10-30T13:23:35.737551+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T13:23:35.737551+0000","last_undegraded":"2023-10-30T13:23:35.737551+0000","last_fullsized":"2023-10-30T13:23:35.737551+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674810+0000","last_change":"2023-10-30T13:23:19.698274+0000","last_active":"2023-10-30T13:23:20.674810+0000","last_peered":"2023-10-30T13:23:20.674810+0000","last_clean":"2023-10-30T13:23:20.674810+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T13:23:20.674810+0000","last_undegraded":"2023-10-30T13:23:20.674810+0000","last_fullsized":"2023-10-30T13:23:20.674810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'15","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.734036+0000","last_change":"2023-10-30T13:23:18.948296+0000","last_active":"2023-10-30T13:23:35.734036+0000","last_peered":"2023-10-30T13:23:35.734036+0000","last_clean":"2023-10-30T13:23:35.734036+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T13:23:35.734036+0000","last_undegraded":"2023-10-30T13:23:35.734036+0000","last_fullsized":"2023-10-30T13:23:35.734036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:55.308329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611416+0000","last_change":"2023-10-30T13:23:19.728647+0000","last_active":"2023-10-30T13:23:21.611416+0000","last_peered":"2023-10-30T13:23:21.611416+0000","last_clean":"2023-10-30T13:23:21.611416+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T13:23:21.611416+0000","last_undegraded":"2023-10-30T13:23:21.611416+0000","last_fullsized":"2023-10-30T13:23:21.611416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:09.123809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.963649+0000","last_change":"2023-10-30T13:23:18.962343+0000","last_active":"2023-10-30T13:23:24.963649+0000","last_peered":"2023-10-30T13:23:24.963649+0000","last_clean":"2023-10-30T13:23:24.963649+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T13:23:24.963649+0000","last_undegraded":"2023-10-30T13:23:24.963649+0000","last_fullsized":"2023-10-30T13:23:24.963649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:36.082164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611928+0000","last_change":"2023-10-30T13:23:19.720770+0000","last_active":"2023-10-30T13:23:21.611928+0000","last_peered":"2023-10-30T13:23:21.611928+0000","last_clean":"2023-10-30T13:23:21.611928+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T13:23:21.611928+0000","last_undegraded":"2023-10-30T13:23:21.611928+0000","last_fullsized":"2023-10-30T13:23:21.611928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:37.378858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.962456+0000","last_change":"2023-10-30T13:23:18.949081+0000","last_active":"2023-10-30T13:23:24.962456+0000","last_peered":"2023-10-30T13:23:24.962456+0000","last_clean":"2023-10-30T13:23:24.962456+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T13:23:24.962456+0000","last_undegraded":"2023-10-30T13:23:24.962456+0000","last_fullsized":"2023-10-30T13:23:24.962456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:00.356885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613552+0000","last_change":"2023-10-30T13:23:19.724180+0000","last_active":"2023-10-30T13:23:21.613552+0000","last_peered":"2023-10-30T13:23:21.613552+0000","last_clean":"2023-10-30T13:23:21.613552+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T13:23:21.613552+0000","last_undegraded":"2023-10-30T13:23:21.613552+0000","last_fullsized":"2023-10-30T13:23:21.613552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:14.049454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.981649+0000","last_change":"2023-10-30T13:23:18.970780+0000","last_active":"2023-10-30T13:23:20.981649+0000","last_peered":"2023-10-30T13:23:20.981649+0000","last_clean":"2023-10-30T13:23:20.981649+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T13:23:20.981649+0000","last_undegraded":"2023-10-30T13:23:20.981649+0000","last_fullsized":"2023-10-30T13:23:20.981649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:38:51.113629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611979+0000","last_change":"2023-10-30T13:23:19.704834+0000","last_active":"2023-10-30T13:23:21.611979+0000","last_peered":"2023-10-30T13:23:21.611979+0000","last_clean":"2023-10-30T13:23:21.611979+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T13:23:21.611979+0000","last_undegraded":"2023-10-30T13:23:21.611979+0000","last_fullsized":"2023-10-30T13:23:21.611979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:11.266083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.734210+0000","last_change":"2023-10-30T13:23:18.977920+0000","last_active":"2023-10-30T13:23:35.734210+0000","last_peered":"2023-10-30T13:23:35.734210+0000","last_clean":"2023-10-30T13:23:35.734210+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T13:23:35.734210+0000","last_undegraded":"2023-10-30T13:23:35.734210+0000","last_fullsized":"2023-10-30T13:23:35.734210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:07:59.139948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611883+0000","last_change":"2023-10-30T13:23:19.724497+0000","last_active":"2023-10-30T13:23:21.611883+0000","last_peered":"2023-10-30T13:23:21.611883+0000","last_clean":"2023-10-30T13:23:21.611883+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T13:23:21.611883+0000","last_undegraded":"2023-10-30T13:23:21.611883+0000","last_fullsized":"2023-10-30T13:23:21.611883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:23.836868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.686506+0000","last_change":"2023-10-30T13:23:08.208031+0000","last_active":"2023-10-30T13:23:19.686506+0000","last_peered":"2023-10-30T13:23:19.686506+0000","last_clean":"2023-10-30T13:23:19.686506+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T13:23:19.686506+0000","last_undegraded":"2023-10-30T13:23:19.686506+0000","last_fullsized":"2023-10-30T13:23:19.686506+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_clean_scrub_stamp":"2023-10-30T13:23:07.179866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:22.691367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.027055+0000","last_change":"2023-10-30T13:23:18.979328+0000","last_active":"2023-10-30T13:23:30.027055+0000","last_peered":"2023-10-30T13:23:30.027055+0000","last_clean":"2023-10-30T13:23:30.027055+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T13:23:30.027055+0000","last_undegraded":"2023-10-30T13:23:30.027055+0000","last_fullsized":"2023-10-30T13:23:30.027055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:04.942593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005198+0000","last_change":"2023-10-30T13:23:19.711222+0000","last_active":"2023-10-30T13:23:21.005198+0000","last_peered":"2023-10-30T13:23:21.005198+0000","last_clean":"2023-10-30T13:23:21.005198+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T13:23:21.005198+0000","last_undegraded":"2023-10-30T13:23:21.005198+0000","last_fullsized":"2023-10-30T13:23:21.005198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:07.213498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.731662+0000","last_change":"2023-10-30T13:23:18.985318+0000","last_active":"2023-10-30T13:23:35.731662+0000","last_peered":"2023-10-30T13:23:35.731662+0000","last_clean":"2023-10-30T13:23:35.731662+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T13:23:35.731662+0000","last_undegraded":"2023-10-30T13:23:35.731662+0000","last_fullsized":"2023-10-30T13:23:35.731662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:16.754084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107580+0000","last_change":"2023-10-30T13:23:19.717938+0000","last_active":"2023-10-30T13:23:21.107580+0000","last_peered":"2023-10-30T13:23:21.107580+0000","last_clean":"2023-10-30T13:23:21.107580+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T13:23:21.107580+0000","last_undegraded":"2023-10-30T13:23:21.107580+0000","last_fullsized":"2023-10-30T13:23:21.107580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:56:01.624954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.732971+0000","last_change":"2023-10-30T13:23:18.949565+0000","last_active":"2023-10-30T13:23:35.732971+0000","last_peered":"2023-10-30T13:23:35.732971+0000","last_clean":"2023-10-30T13:23:35.732971+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T13:23:35.732971+0000","last_undegraded":"2023-10-30T13:23:35.732971+0000","last_fullsized":"2023-10-30T13:23:35.732971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:55:46.029368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670291+0000","last_change":"2023-10-30T13:23:19.678037+0000","last_active":"2023-10-30T13:23:20.670291+0000","last_peered":"2023-10-30T13:23:20.670291+0000","last_clean":"2023-10-30T13:23:20.670291+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T13:23:20.670291+0000","last_undegraded":"2023-10-30T13:23:20.670291+0000","last_fullsized":"2023-10-30T13:23:20.670291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613192+0000","last_change":"2023-10-30T13:23:18.984535+0000","last_active":"2023-10-30T13:23:21.613192+0000","last_peered":"2023-10-30T13:23:21.613192+0000","last_clean":"2023-10-30T13:23:21.613192+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T13:23:21.613192+0000","last_undegraded":"2023-10-30T13:23:21.613192+0000","last_fullsized":"2023-10-30T13:23:21.613192+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:28.405405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675980+0000","last_change":"2023-10-30T13:23:19.691867+0000","last_active":"2023-10-30T13:23:20.675980+0000","last_peered":"2023-10-30T13:23:20.675980+0000","last_clean":"2023-10-30T13:23:20.675980+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T13:23:20.675980+0000","last_undegraded":"2023-10-30T13:23:20.675980+0000","last_fullsized":"2023-10-30T13:23:20.675980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:02.276042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670494+0000","last_change":"2023-10-30T13:23:18.958210+0000","last_active":"2023-10-30T13:23:20.670494+0000","last_peered":"2023-10-30T13:23:20.670494+0000","last_clean":"2023-10-30T13:23:20.670494+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T13:23:20.670494+0000","last_undegraded":"2023-10-30T13:23:20.670494+0000","last_fullsized":"2023-10-30T13:23:20.670494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:34.653005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613343+0000","last_change":"2023-10-30T13:23:19.713174+0000","last_active":"2023-10-30T13:23:21.613343+0000","last_peered":"2023-10-30T13:23:21.613343+0000","last_clean":"2023-10-30T13:23:21.613343+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T13:23:21.613343+0000","last_undegraded":"2023-10-30T13:23:21.613343+0000","last_fullsized":"2023-10-30T13:23:21.613343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:43.400223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612618+0000","last_change":"2023-10-30T13:23:18.985185+0000","last_active":"2023-10-30T13:23:21.612618+0000","last_peered":"2023-10-30T13:23:21.612618+0000","last_clean":"2023-10-30T13:23:21.612618+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T13:23:21.612618+0000","last_undegraded":"2023-10-30T13:23:21.612618+0000","last_fullsized":"2023-10-30T13:23:21.612618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:36.230072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670523+0000","last_change":"2023-10-30T13:23:19.680185+0000","last_active":"2023-10-30T13:23:20.670523+0000","last_peered":"2023-10-30T13:23:20.670523+0000","last_clean":"2023-10-30T13:23:20.670523+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T13:23:20.670523+0000","last_undegraded":"2023-10-30T13:23:20.670523+0000","last_fullsized":"2023-10-30T13:23:20.670523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.733893+0000","last_change":"2023-10-30T13:23:18.978085+0000","last_active":"2023-10-30T13:23:35.733893+0000","last_peered":"2023-10-30T13:23:35.733893+0000","last_clean":"2023-10-30T13:23:35.733893+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T13:23:35.733893+0000","last_undegraded":"2023-10-30T13:23:35.733893+0000","last_fullsized":"2023-10-30T13:23:35.733893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:47.588972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005467+0000","last_change":"2023-10-30T13:23:19.704291+0000","last_active":"2023-10-30T13:23:21.005467+0000","last_peered":"2023-10-30T13:23:21.005467+0000","last_clean":"2023-10-30T13:23:21.005467+0000","last_became_active":"2023-10-30T13:23:19.704066+0000","last_became_peered":"2023-10-30T13:23:19.704066+0000","last_unstale":"2023-10-30T13:23:21.005467+0000","last_undegraded":"2023-10-30T13:23:21.005467+0000","last_fullsized":"2023-10-30T13:23:21.005467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:13.754194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.027200+0000","last_change":"2023-10-30T13:23:18.970566+0000","last_active":"2023-10-30T13:23:30.027200+0000","last_peered":"2023-10-30T13:23:30.027200+0000","last_clean":"2023-10-30T13:23:30.027200+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T13:23:30.027200+0000","last_undegraded":"2023-10-30T13:23:30.027200+0000","last_fullsized":"2023-10-30T13:23:30.027200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:03.851889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107903+0000","last_change":"2023-10-30T13:23:19.685481+0000","last_active":"2023-10-30T13:23:21.107903+0000","last_peered":"2023-10-30T13:23:21.107903+0000","last_clean":"2023-10-30T13:23:21.107903+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T13:23:21.107903+0000","last_undegraded":"2023-10-30T13:23:21.107903+0000","last_fullsized":"2023-10-30T13:23:21.107903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:33:50.672215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.037430+0000","last_change":"2023-10-30T13:23:18.986394+0000","last_active":"2023-10-30T13:23:30.037430+0000","last_peered":"2023-10-30T13:23:30.037430+0000","last_clean":"2023-10-30T13:23:30.037430+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T13:23:30.037430+0000","last_undegraded":"2023-10-30T13:23:30.037430+0000","last_fullsized":"2023-10-30T13:23:30.037430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:59.986981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675944+0000","last_change":"2023-10-30T13:23:19.681239+0000","last_active":"2023-10-30T13:23:20.675944+0000","last_peered":"2023-10-30T13:23:20.675944+0000","last_clean":"2023-10-30T13:23:20.675944+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T13:23:20.675944+0000","last_undegraded":"2023-10-30T13:23:20.675944+0000","last_fullsized":"2023-10-30T13:23:20.675944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:00.121240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.036720+0000","last_change":"2023-10-30T13:23:18.963905+0000","last_active":"2023-10-30T13:23:30.036720+0000","last_peered":"2023-10-30T13:23:30.036720+0000","last_clean":"2023-10-30T13:23:30.036720+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T13:23:30.036720+0000","last_undegraded":"2023-10-30T13:23:30.036720+0000","last_fullsized":"2023-10-30T13:23:30.036720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005091+0000","last_change":"2023-10-30T13:23:19.678697+0000","last_active":"2023-10-30T13:23:21.005091+0000","last_peered":"2023-10-30T13:23:21.005091+0000","last_clean":"2023-10-30T13:23:21.005091+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T13:23:21.005091+0000","last_undegraded":"2023-10-30T13:23:21.005091+0000","last_fullsized":"2023-10-30T13:23:21.005091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:57.442011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.981565+0000","last_change":"2023-10-30T13:23:18.952302+0000","last_active":"2023-10-30T13:23:20.981565+0000","last_peered":"2023-10-30T13:23:20.981565+0000","last_clean":"2023-10-30T13:23:20.981565+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T13:23:20.981565+0000","last_undegraded":"2023-10-30T13:23:20.981565+0000","last_fullsized":"2023-10-30T13:23:20.981565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:12.827417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005323+0000","last_change":"2023-10-30T13:23:19.688716+0000","last_active":"2023-10-30T13:23:21.005323+0000","last_peered":"2023-10-30T13:23:21.005323+0000","last_clean":"2023-10-30T13:23:21.005323+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T13:23:21.005323+0000","last_undegraded":"2023-10-30T13:23:21.005323+0000","last_fullsized":"2023-10-30T13:23:21.005323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:26:04.209304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.022861+0000","last_change":"2023-10-30T13:23:18.984912+0000","last_active":"2023-10-30T13:23:30.022861+0000","last_peered":"2023-10-30T13:23:30.022861+0000","last_clean":"2023-10-30T13:23:30.022861+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T13:23:30.022861+0000","last_undegraded":"2023-10-30T13:23:30.022861+0000","last_fullsized":"2023-10-30T13:23:30.022861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:27:16.205042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674560+0000","last_change":"2023-10-30T13:23:19.718922+0000","last_active":"2023-10-30T13:23:20.674560+0000","last_peered":"2023-10-30T13:23:20.674560+0000","last_clean":"2023-10-30T13:23:20.674560+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T13:23:20.674560+0000","last_undegraded":"2023-10-30T13:23:20.674560+0000","last_fullsized":"2023-10-30T13:23:20.674560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612299+0000","last_change":"2023-10-30T13:23:19.732322+0000","last_active":"2023-10-30T13:23:21.612299+0000","last_peered":"2023-10-30T13:23:21.612299+0000","last_clean":"2023-10-30T13:23:21.612299+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T13:23:21.612299+0000","last_undegraded":"2023-10-30T13:23:21.612299+0000","last_fullsized":"2023-10-30T13:23:21.612299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:21.479138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.968310+0000","last_change":"2023-10-30T13:23:18.993372+0000","last_active":"2023-10-30T13:23:24.968310+0000","last_peered":"2023-10-30T13:23:24.968310+0000","last_clean":"2023-10-30T13:23:24.968310+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T13:23:24.968310+0000","last_undegraded":"2023-10-30T13:23:24.968310+0000","last_fullsized":"2023-10-30T13:23:24.968310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:41.370678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611741+0000","last_change":"2023-10-30T13:23:19.732769+0000","last_active":"2023-10-30T13:23:21.611741+0000","last_peered":"2023-10-30T13:23:21.611741+0000","last_clean":"2023-10-30T13:23:21.611741+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T13:23:21.611741+0000","last_undegraded":"2023-10-30T13:23:21.611741+0000","last_fullsized":"2023-10-30T13:23:21.611741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:56.378098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.732313+0000","last_change":"2023-10-30T13:23:18.987633+0000","last_active":"2023-10-30T13:23:35.732313+0000","last_peered":"2023-10-30T13:23:35.732313+0000","last_clean":"2023-10-30T13:23:35.732313+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T13:23:35.732313+0000","last_undegraded":"2023-10-30T13:23:35.732313+0000","last_fullsized":"2023-10-30T13:23:35.732313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:09:33.324898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107906+0000","last_change":"2023-10-30T13:23:19.722311+0000","last_active":"2023-10-30T13:23:21.107906+0000","last_peered":"2023-10-30T13:23:21.107906+0000","last_clean":"2023-10-30T13:23:21.107906+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T13:23:21.107906+0000","last_undegraded":"2023-10-30T13:23:21.107906+0000","last_fullsized":"2023-10-30T13:23:21.107906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:44.204074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.962089+0000","last_change":"2023-10-30T13:23:18.958470+0000","last_active":"2023-10-30T13:23:24.962089+0000","last_peered":"2023-10-30T13:23:24.962089+0000","last_clean":"2023-10-30T13:23:24.962089+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T13:23:24.962089+0000","last_undegraded":"2023-10-30T13:23:24.962089+0000","last_fullsized":"2023-10-30T13:23:24.962089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:21.507744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005410+0000","last_change":"2023-10-30T13:23:19.699742+0000","last_active":"2023-10-30T13:23:21.005410+0000","last_peered":"2023-10-30T13:23:21.005410+0000","last_clean":"2023-10-30T13:23:21.005410+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T13:23:21.005410+0000","last_undegraded":"2023-10-30T13:23:21.005410+0000","last_fullsized":"2023-10-30T13:23:21.005410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:13.504378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'9","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.735807+0000","last_change":"2023-10-30T13:23:18.980952+0000","last_active":"2023-10-30T13:23:35.735807+0000","last_peered":"2023-10-30T13:23:35.735807+0000","last_clean":"2023-10-30T13:23:35.735807+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T13:23:35.735807+0000","last_undegraded":"2023-10-30T13:23:35.735807+0000","last_fullsized":"2023-10-30T13:23:35.735807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:44.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6467,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":12934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":819200,"data_stored":662582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":74433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2569999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1379999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":71612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":74467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":221184,"data_stored":70697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":71842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":811008,"data_stored":660907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:23:47.334 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-10-30T13:23:47.335 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:23:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.508+0000 7f44d71b8640 1 Processor -- start 2023-10-30T13:23:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.508+0000 7f44d71b8640 1 -- start start 2023-10-30T13:23:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44d71b8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44d00c6b80 0x7f44d00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44d71b8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f44d00c6f60 0x7f44d00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44d71b8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d00c0a50 con 0x7f44d00c8430 2023-10-30T13:23:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44d71b8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d006f6e0 con 0x7f44d00c6f60 2023-10-30T13:23:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44d71b8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d006f9c0 con 0x7f44d00c6b80 2023-10-30T13:23:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44d4f54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44d00c6b80 0x7f44d00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44cffff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f44d00c6f60 0x7f44d00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44cffff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f44d00c6f60 0x7f44d00c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:32874/0 (socket says 172.21.15.46:32874) 2023-10-30T13:23:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44cffff640 1 -- 172.21.15.46:0/42187557 learned_addr learned my addr 172.21.15.46:0/42187557 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44cffff640 1 -- 172.21.15.46:0/42187557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44d00c6b80 msgr2=0x7f44d00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44cffff640 1 --2- 172.21.15.46:0/42187557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44d00c6b80 0x7f44d00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44cffff640 1 -- 172.21.15.46:0/42187557 >> v1:172.21.15.154:6789/0 conn(0x7f44d00c8430 legacy=0x7f44d013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.512+0000 7f44cffff640 1 -- 172.21.15.46:0/42187557 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d006fca0 con 0x7f44d00c6f60 2023-10-30T13:23:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44cffff640 1 --2- 172.21.15.46:0/42187557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d00c6f60 0x7f44d00c0400 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009870 tx=0x7f44bc01c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dc41e81564e6c4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44cf7fe640 1 -- 172.21.15.46:0/42187557 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc0ab650 con 0x7f44d00c6f60 2023-10-30T13:23:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44cf7fe640 1 -- 172.21.15.46:0/42187557 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44bc0ab7b0 con 0x7f44d00c6f60 2023-10-30T13:23:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44cf7fe640 1 -- 172.21.15.46:0/42187557 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc0abb40 con 0x7f44d00c6f60 2023-10-30T13:23:47.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44d71b8640 1 -- 172.21.15.46:0/42187557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d00c6f60 msgr2=0x7f44d00c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44d71b8640 1 --2- 172.21.15.46:0/42187557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d00c6f60 0x7f44d00c0400 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009870 tx=0x7f44bc01c540 comp rx=0 tx=0).stop 2023-10-30T13:23:47.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44d71b8640 1 -- 172.21.15.46:0/42187557 shutdown_connections 2023-10-30T13:23:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44d71b8640 1 --2- 172.21.15.46:0/42187557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d00c6f60 0x7f44d00c0400 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44d71b8640 1 --2- 172.21.15.46:0/42187557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44d00c6b80 0x7f44d00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44d71b8640 1 -- 172.21.15.46:0/42187557 >> 172.21.15.46:0/42187557 conn(0x7f44d00c7910 msgr2=0x7f44d00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44d71b8640 1 -- 172.21.15.46:0/42187557 shutdown_connections 2023-10-30T13:23:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44d71b8640 1 -- 172.21.15.46:0/42187557 wait complete. 2023-10-30T13:23:47.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44d71b8640 1 Processor -- start 2023-10-30T13:23:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.516+0000 7f44d71b8640 1 -- start start 2023-10-30T13:23:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44d71b8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44d00c6b80 0x7f44d01582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44d71b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d00c6f60 0x7f44d0158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44d71b8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d00c8430 0x7f44d00c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44d71b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d0157cc0 con 0x7f44d00c6f60 2023-10-30T13:23:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44d71b8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d0139cd0 con 0x7f44d00c8430 2023-10-30T13:23:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44d71b8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d013b180 con 0x7f44d00c6b80 2023-10-30T13:23:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d00c6f60 0x7f44d0158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44d4f54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44d00c6b80 0x7f44d01582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d00c6f60 0x7f44d0158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47352/0 (socket says 172.21.15.46:47352) 2023-10-30T13:23:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cffff640 1 -- 172.21.15.46:0/1668030323 learned_addr learned my addr 172.21.15.46:0/1668030323 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44d5755640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d00c8430 0x7f44d00c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cffff640 1 -- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44d00c6b80 msgr2=0x7f44d01582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cffff640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44d00c6b80 0x7f44d01582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cffff640 1 -- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d00c8430 msgr2=0x7f44d00c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cffff640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d00c8430 0x7f44d00c8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cffff640 1 -- 172.21.15.46:0/1668030323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc0094d0 con 0x7f44d00c6f60 2023-10-30T13:23:47.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cffff640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d00c6f60 0x7f44d0158810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f44bc01a170 tx=0x7f44bc01ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:47.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cd7fa640 1 -- 172.21.15.46:0/1668030323 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc0b4800 con 0x7f44d00c6f60 2023-10-30T13:23:47.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cd7fa640 1 -- 172.21.15.46:0/1668030323 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44bc0b4960 con 0x7f44d00c6f60 2023-10-30T13:23:47.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cd7fa640 1 -- 172.21.15.46:0/1668030323 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc0b4cf0 con 0x7f44d00c6f60 2023-10-30T13:23:47.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44d71b8640 1 -- 172.21.15.46:0/1668030323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d0139f30 con 0x7f44d00c6f60 2023-10-30T13:23:47.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44d71b8640 1 -- 172.21.15.46:0/1668030323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d0157670 con 0x7f44d00c6f60 2023-10-30T13:23:47.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44d71b8640 1 -- 172.21.15.46:0/1668030323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d00c8c10 con 0x7f44d00c6f60 2023-10-30T13:23:47.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.520+0000 7f44cd7fa640 1 -- 172.21.15.46:0/1668030323 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f44bc0aa430 con 0x7f44d00c6f60 2023-10-30T13:23:47.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.524+0000 7f44cd7fa640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f44b40664a0 0x7f44b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.524+0000 7f44cd7fa640 1 -- 172.21.15.46:0/1668030323 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44bc11eb70 con 0x7f44d00c6f60 2023-10-30T13:23:47.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.524+0000 7f44d4f54640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f44b40664a0 0x7f44b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.524+0000 7f44d4f54640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f44b40664a0 0x7f44b4068960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009b60 tx=0x7f44c4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:47.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.528+0000 7f44cd7fa640 1 -- 172.21.15.46:0/1668030323 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f44d00c8c10 con 0x7f44d00c6f60 2023-10-30T13:23:47.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.680+0000 7f44d71b8640 1 -- 172.21.15.46:0/1668030323 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f44d00c8c10 con 0x7f44b40664a0 2023-10-30T13:23:47.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.700+0000 7f44cd7fa640 1 -- 172.21.15.46:0/1668030323 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264626 (secure 0 0 0) 0x7f44d00c8c10 con 0x7f44b40664a0 2023-10-30T13:23:47.710 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T13:23:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.704+0000 7f44d71b8640 1 -- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f44b40664a0 msgr2=0x7f44b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.704+0000 7f44d71b8640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f44b40664a0 0x7f44b4068960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009b60 tx=0x7f44c4009190 comp rx=0 tx=0).stop 2023-10-30T13:23:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.704+0000 7f44d71b8640 1 -- 172.21.15.46:0/1668030323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d00c6f60 msgr2=0x7f44d0158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.704+0000 7f44d71b8640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d00c6f60 0x7f44d0158810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f44bc01a170 tx=0x7f44bc01ad80 comp rx=0 tx=0).stop 2023-10-30T13:23:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.704+0000 7f44d71b8640 1 -- 172.21.15.46:0/1668030323 shutdown_connections 2023-10-30T13:23:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.704+0000 7f44d71b8640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f44b40664a0 0x7f44b4068960 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.704+0000 7f44d71b8640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d00c8430 0x7f44d00c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.704+0000 7f44d71b8640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d00c6f60 0x7f44d0158810 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.704+0000 7f44d71b8640 1 --2- 172.21.15.46:0/1668030323 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44d00c6b80 0x7f44d01582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.704+0000 7f44d71b8640 1 -- 172.21.15.46:0/1668030323 >> 172.21.15.46:0/1668030323 conn(0x7f44d00c7910 msgr2=0x7f44d0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.704+0000 7f44d71b8640 1 -- 172.21.15.46:0/1668030323 shutdown_connections 2023-10-30T13:23:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.704+0000 7f44d71b8640 1 -- 172.21.15.46:0/1668030323 wait complete. 2023-10-30T13:23:47.731 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-30T13:23:46.718944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596835,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":287,"num_write_kb":2457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235676,"kb_used_data":2908,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739044,"statfs":{"total":773094113280,"available":772852781056,"internally_reserved":0,"allocated":2977792,"data_stored":1757797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2651,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004545"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612886+0000","last_change":"2023-10-30T13:23:19.719271+0000","last_active":"2023-10-30T13:23:21.612886+0000","last_peered":"2023-10-30T13:23:21.612886+0000","last_clean":"2023-10-30T13:23:21.612886+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T13:23:21.612886+0000","last_undegraded":"2023-10-30T13:23:21.612886+0000","last_fullsized":"2023-10-30T13:23:21.612886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:41:37.614073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670716+0000","last_change":"2023-10-30T13:23:19.687152+0000","last_active":"2023-10-30T13:23:20.670716+0000","last_peered":"2023-10-30T13:23:20.670716+0000","last_clean":"2023-10-30T13:23:20.670716+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T13:23:20.670716+0000","last_undegraded":"2023-10-30T13:23:20.670716+0000","last_fullsized":"2023-10-30T13:23:20.670716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107815+0000","last_change":"2023-10-30T13:23:19.689554+0000","last_active":"2023-10-30T13:23:21.107815+0000","last_peered":"2023-10-30T13:23:21.107815+0000","last_clean":"2023-10-30T13:23:21.107815+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T13:23:21.107815+0000","last_undegraded":"2023-10-30T13:23:21.107815+0000","last_fullsized":"2023-10-30T13:23:21.107815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:56:13.795044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611622+0000","last_change":"2023-10-30T13:23:19.714018+0000","last_active":"2023-10-30T13:23:21.611622+0000","last_peered":"2023-10-30T13:23:21.611622+0000","last_clean":"2023-10-30T13:23:21.611622+0000","last_became_active":"2023-10-30T13:23:19.711673+0000","last_became_peered":"2023-10-30T13:23:19.711673+0000","last_unstale":"2023-10-30T13:23:21.611622+0000","last_undegraded":"2023-10-30T13:23:21.611622+0000","last_fullsized":"2023-10-30T13:23:21.611622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:28:16.658840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611546+0000","last_change":"2023-10-30T13:23:19.719999+0000","last_active":"2023-10-30T13:23:21.611546+0000","last_peered":"2023-10-30T13:23:21.611546+0000","last_clean":"2023-10-30T13:23:21.611546+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T13:23:21.611546+0000","last_undegraded":"2023-10-30T13:23:21.611546+0000","last_fullsized":"2023-10-30T13:23:21.611546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:46:31.287013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612300+0000","last_change":"2023-10-30T13:23:19.703474+0000","last_active":"2023-10-30T13:23:21.612300+0000","last_peered":"2023-10-30T13:23:21.612300+0000","last_clean":"2023-10-30T13:23:21.612300+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T13:23:21.612300+0000","last_undegraded":"2023-10-30T13:23:21.612300+0000","last_fullsized":"2023-10-30T13:23:21.612300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:53:58.118821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613234+0000","last_change":"2023-10-30T13:23:19.726985+0000","last_active":"2023-10-30T13:23:21.613234+0000","last_peered":"2023-10-30T13:23:21.613234+0000","last_clean":"2023-10-30T13:23:21.613234+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T13:23:21.613234+0000","last_undegraded":"2023-10-30T13:23:21.613234+0000","last_fullsized":"2023-10-30T13:23:21.613234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:51:13.152339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.004933+0000","last_change":"2023-10-30T13:23:19.714153+0000","last_active":"2023-10-30T13:23:21.004933+0000","last_peered":"2023-10-30T13:23:21.004933+0000","last_clean":"2023-10-30T13:23:21.004933+0000","last_became_active":"2023-10-30T13:23:19.713876+0000","last_became_peered":"2023-10-30T13:23:19.713876+0000","last_unstale":"2023-10-30T13:23:21.004933+0000","last_undegraded":"2023-10-30T13:23:21.004933+0000","last_fullsized":"2023-10-30T13:23:21.004933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:05:13.588780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613220+0000","last_change":"2023-10-30T13:23:19.724583+0000","last_active":"2023-10-30T13:23:21.613220+0000","last_peered":"2023-10-30T13:23:21.613220+0000","last_clean":"2023-10-30T13:23:21.613220+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T13:23:21.613220+0000","last_undegraded":"2023-10-30T13:23:21.613220+0000","last_fullsized":"2023-10-30T13:23:21.613220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:47.936982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677145+0000","last_change":"2023-10-30T13:23:19.704399+0000","last_active":"2023-10-30T13:23:20.677145+0000","last_peered":"2023-10-30T13:23:20.677145+0000","last_clean":"2023-10-30T13:23:20.677145+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T13:23:20.677145+0000","last_undegraded":"2023-10-30T13:23:20.677145+0000","last_fullsized":"2023-10-30T13:23:20.677145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:19.748739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676522+0000","last_change":"2023-10-30T13:23:19.726077+0000","last_active":"2023-10-30T13:23:20.676522+0000","last_peered":"2023-10-30T13:23:20.676522+0000","last_clean":"2023-10-30T13:23:20.676522+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T13:23:20.676522+0000","last_undegraded":"2023-10-30T13:23:20.676522+0000","last_fullsized":"2023-10-30T13:23:20.676522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675959+0000","last_change":"2023-10-30T13:23:19.720995+0000","last_active":"2023-10-30T13:23:20.675959+0000","last_peered":"2023-10-30T13:23:20.675959+0000","last_clean":"2023-10-30T13:23:20.675959+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T13:23:20.675959+0000","last_undegraded":"2023-10-30T13:23:20.675959+0000","last_fullsized":"2023-10-30T13:23:20.675959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:44.490568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005253+0000","last_change":"2023-10-30T13:23:19.692509+0000","last_active":"2023-10-30T13:23:21.005253+0000","last_peered":"2023-10-30T13:23:21.005253+0000","last_clean":"2023-10-30T13:23:21.005253+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T13:23:21.005253+0000","last_undegraded":"2023-10-30T13:23:21.005253+0000","last_fullsized":"2023-10-30T13:23:21.005253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:15:17.956816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107580+0000","last_change":"2023-10-30T13:23:19.685435+0000","last_active":"2023-10-30T13:23:21.107580+0000","last_peered":"2023-10-30T13:23:21.107580+0000","last_clean":"2023-10-30T13:23:21.107580+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T13:23:21.107580+0000","last_undegraded":"2023-10-30T13:23:21.107580+0000","last_fullsized":"2023-10-30T13:23:21.107580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:05:39.035298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670272+0000","last_change":"2023-10-30T13:23:19.706721+0000","last_active":"2023-10-30T13:23:20.670272+0000","last_peered":"2023-10-30T13:23:20.670272+0000","last_clean":"2023-10-30T13:23:20.670272+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T13:23:20.670272+0000","last_undegraded":"2023-10-30T13:23:20.670272+0000","last_fullsized":"2023-10-30T13:23:20.670272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.610717+0000","last_change":"2023-10-30T13:23:19.710432+0000","last_active":"2023-10-30T13:23:21.610717+0000","last_peered":"2023-10-30T13:23:21.610717+0000","last_clean":"2023-10-30T13:23:21.610717+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T13:23:21.610717+0000","last_undegraded":"2023-10-30T13:23:21.610717+0000","last_fullsized":"2023-10-30T13:23:21.610717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:48:51.992826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612857+0000","last_change":"2023-10-30T13:23:19.741225+0000","last_active":"2023-10-30T13:23:21.612857+0000","last_peered":"2023-10-30T13:23:21.612857+0000","last_clean":"2023-10-30T13:23:21.612857+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T13:23:21.612857+0000","last_undegraded":"2023-10-30T13:23:21.612857+0000","last_fullsized":"2023-10-30T13:23:21.612857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:27.310070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.608570+0000","last_change":"2023-10-30T13:23:19.701024+0000","last_active":"2023-10-30T13:23:21.608570+0000","last_peered":"2023-10-30T13:23:21.608570+0000","last_clean":"2023-10-30T13:23:21.608570+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T13:23:21.608570+0000","last_undegraded":"2023-10-30T13:23:21.608570+0000","last_fullsized":"2023-10-30T13:23:21.608570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:14.460383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612314+0000","last_change":"2023-10-30T13:23:19.723929+0000","last_active":"2023-10-30T13:23:21.612314+0000","last_peered":"2023-10-30T13:23:21.612314+0000","last_clean":"2023-10-30T13:23:21.612314+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T13:23:21.612314+0000","last_undegraded":"2023-10-30T13:23:21.612314+0000","last_fullsized":"2023-10-30T13:23:21.612314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:41.023731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674806+0000","last_change":"2023-10-30T13:23:19.725101+0000","last_active":"2023-10-30T13:23:20.674806+0000","last_peered":"2023-10-30T13:23:20.674806+0000","last_clean":"2023-10-30T13:23:20.674806+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T13:23:20.674806+0000","last_undegraded":"2023-10-30T13:23:20.674806+0000","last_fullsized":"2023-10-30T13:23:20.674806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107754+0000","last_change":"2023-10-30T13:23:19.721813+0000","last_active":"2023-10-30T13:23:21.107754+0000","last_peered":"2023-10-30T13:23:21.107754+0000","last_clean":"2023-10-30T13:23:21.107754+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T13:23:21.107754+0000","last_undegraded":"2023-10-30T13:23:21.107754+0000","last_fullsized":"2023-10-30T13:23:21.107754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:34:17.212695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612670+0000","last_change":"2023-10-30T13:23:19.724763+0000","last_active":"2023-10-30T13:23:21.612670+0000","last_peered":"2023-10-30T13:23:21.612670+0000","last_clean":"2023-10-30T13:23:21.612670+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T13:23:21.612670+0000","last_undegraded":"2023-10-30T13:23:21.612670+0000","last_fullsized":"2023-10-30T13:23:21.612670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:28:53.428578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005009+0000","last_change":"2023-10-30T13:23:19.716706+0000","last_active":"2023-10-30T13:23:21.005009+0000","last_peered":"2023-10-30T13:23:21.005009+0000","last_clean":"2023-10-30T13:23:21.005009+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T13:23:21.005009+0000","last_undegraded":"2023-10-30T13:23:21.005009+0000","last_fullsized":"2023-10-30T13:23:21.005009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:07.718271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.610920+0000","last_change":"2023-10-30T13:23:19.704791+0000","last_active":"2023-10-30T13:23:21.610920+0000","last_peered":"2023-10-30T13:23:21.610920+0000","last_clean":"2023-10-30T13:23:21.610920+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T13:23:21.610920+0000","last_undegraded":"2023-10-30T13:23:21.610920+0000","last_fullsized":"2023-10-30T13:23:21.610920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:54.750514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005422+0000","last_change":"2023-10-30T13:23:19.717027+0000","last_active":"2023-10-30T13:23:21.005422+0000","last_peered":"2023-10-30T13:23:21.005422+0000","last_clean":"2023-10-30T13:23:21.005422+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T13:23:21.005422+0000","last_undegraded":"2023-10-30T13:23:21.005422+0000","last_fullsized":"2023-10-30T13:23:21.005422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:38.788372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.677247+0000","last_change":"2023-10-30T13:23:19.706002+0000","last_active":"2023-10-30T13:23:20.677247+0000","last_peered":"2023-10-30T13:23:20.677247+0000","last_clean":"2023-10-30T13:23:20.677247+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T13:23:20.677247+0000","last_undegraded":"2023-10-30T13:23:20.677247+0000","last_fullsized":"2023-10-30T13:23:20.677247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:46.882474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670564+0000","last_change":"2023-10-30T13:23:19.678472+0000","last_active":"2023-10-30T13:23:20.670564+0000","last_peered":"2023-10-30T13:23:20.670564+0000","last_clean":"2023-10-30T13:23:20.670564+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T13:23:20.670564+0000","last_undegraded":"2023-10-30T13:23:20.670564+0000","last_fullsized":"2023-10-30T13:23:20.670564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:58.567977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005470+0000","last_change":"2023-10-30T13:23:19.702155+0000","last_active":"2023-10-30T13:23:21.005470+0000","last_peered":"2023-10-30T13:23:21.005470+0000","last_clean":"2023-10-30T13:23:21.005470+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T13:23:21.005470+0000","last_undegraded":"2023-10-30T13:23:21.005470+0000","last_fullsized":"2023-10-30T13:23:21.005470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:30.131186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107903+0000","last_change":"2023-10-30T13:23:19.687820+0000","last_active":"2023-10-30T13:23:21.107903+0000","last_peered":"2023-10-30T13:23:21.107903+0000","last_clean":"2023-10-30T13:23:21.107903+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T13:23:21.107903+0000","last_undegraded":"2023-10-30T13:23:21.107903+0000","last_fullsized":"2023-10-30T13:23:21.107903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:46.088882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612897+0000","last_change":"2023-10-30T13:23:19.702442+0000","last_active":"2023-10-30T13:23:21.612897+0000","last_peered":"2023-10-30T13:23:21.612897+0000","last_clean":"2023-10-30T13:23:21.612897+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T13:23:21.612897+0000","last_undegraded":"2023-10-30T13:23:21.612897+0000","last_fullsized":"2023-10-30T13:23:21.612897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:58:50.083326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.608295+0000","last_change":"2023-10-30T13:23:19.711364+0000","last_active":"2023-10-30T13:23:21.608295+0000","last_peered":"2023-10-30T13:23:21.608295+0000","last_clean":"2023-10-30T13:23:21.608295+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T13:23:21.608295+0000","last_undegraded":"2023-10-30T13:23:21.608295+0000","last_fullsized":"2023-10-30T13:23:21.608295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:15.250397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674450+0000","last_change":"2023-10-30T13:23:19.698485+0000","last_active":"2023-10-30T13:23:20.674450+0000","last_peered":"2023-10-30T13:23:20.674450+0000","last_clean":"2023-10-30T13:23:20.674450+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T13:23:20.674450+0000","last_undegraded":"2023-10-30T13:23:20.674450+0000","last_fullsized":"2023-10-30T13:23:20.674450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675615+0000","last_change":"2023-10-30T13:23:19.703223+0000","last_active":"2023-10-30T13:23:20.675615+0000","last_peered":"2023-10-30T13:23:20.675615+0000","last_clean":"2023-10-30T13:23:20.675615+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T13:23:20.675615+0000","last_undegraded":"2023-10-30T13:23:20.675615+0000","last_fullsized":"2023-10-30T13:23:20.675615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:28.065471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.036846+0000","last_change":"2023-10-30T13:23:18.963933+0000","last_active":"2023-10-30T13:23:30.036846+0000","last_peered":"2023-10-30T13:23:30.036846+0000","last_clean":"2023-10-30T13:23:30.036846+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T13:23:30.036846+0000","last_undegraded":"2023-10-30T13:23:30.036846+0000","last_fullsized":"2023-10-30T13:23:30.036846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:48:58.196567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107913+0000","last_change":"2023-10-30T13:23:19.717819+0000","last_active":"2023-10-30T13:23:21.107913+0000","last_peered":"2023-10-30T13:23:21.107913+0000","last_clean":"2023-10-30T13:23:21.107913+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T13:23:21.107913+0000","last_undegraded":"2023-10-30T13:23:21.107913+0000","last_fullsized":"2023-10-30T13:23:21.107913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:15.403400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.737858+0000","last_change":"2023-10-30T13:23:18.976615+0000","last_active":"2023-10-30T13:23:35.737858+0000","last_peered":"2023-10-30T13:23:35.737858+0000","last_clean":"2023-10-30T13:23:35.737858+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T13:23:35.737858+0000","last_undegraded":"2023-10-30T13:23:35.737858+0000","last_fullsized":"2023-10-30T13:23:35.737858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:59.048396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.733720+0000","last_change":"2023-10-30T13:23:18.977357+0000","last_active":"2023-10-30T13:23:35.733720+0000","last_peered":"2023-10-30T13:23:35.733720+0000","last_clean":"2023-10-30T13:23:35.733720+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T13:23:35.733720+0000","last_undegraded":"2023-10-30T13:23:35.733720+0000","last_fullsized":"2023-10-30T13:23:35.733720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:53.502681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005000+0000","last_change":"2023-10-30T13:23:19.689197+0000","last_active":"2023-10-30T13:23:21.005000+0000","last_peered":"2023-10-30T13:23:21.005000+0000","last_clean":"2023-10-30T13:23:21.005000+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T13:23:21.005000+0000","last_undegraded":"2023-10-30T13:23:21.005000+0000","last_fullsized":"2023-10-30T13:23:21.005000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:59:47.946429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611057+0000","last_change":"2023-10-30T13:23:18.989299+0000","last_active":"2023-10-30T13:23:21.611057+0000","last_peered":"2023-10-30T13:23:21.611057+0000","last_clean":"2023-10-30T13:23:21.611057+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T13:23:21.611057+0000","last_undegraded":"2023-10-30T13:23:21.611057+0000","last_fullsized":"2023-10-30T13:23:21.611057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:44.510184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670217+0000","last_change":"2023-10-30T13:23:19.679342+0000","last_active":"2023-10-30T13:23:20.670217+0000","last_peered":"2023-10-30T13:23:20.670217+0000","last_clean":"2023-10-30T13:23:20.670217+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T13:23:20.670217+0000","last_undegraded":"2023-10-30T13:23:20.670217+0000","last_fullsized":"2023-10-30T13:23:20.670217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:48.838401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.028003+0000","last_change":"2023-10-30T13:23:18.989515+0000","last_active":"2023-10-30T13:23:30.028003+0000","last_peered":"2023-10-30T13:23:30.028003+0000","last_clean":"2023-10-30T13:23:30.028003+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T13:23:30.028003+0000","last_undegraded":"2023-10-30T13:23:30.028003+0000","last_fullsized":"2023-10-30T13:23:30.028003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:35.133762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612308+0000","last_change":"2023-10-30T13:23:19.732859+0000","last_active":"2023-10-30T13:23:21.612308+0000","last_peered":"2023-10-30T13:23:21.612308+0000","last_clean":"2023-10-30T13:23:21.612308+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T13:23:21.612308+0000","last_undegraded":"2023-10-30T13:23:21.612308+0000","last_fullsized":"2023-10-30T13:23:21.612308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:27.629521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.027734+0000","last_change":"2023-10-30T13:23:18.991454+0000","last_active":"2023-10-30T13:23:30.027734+0000","last_peered":"2023-10-30T13:23:30.027734+0000","last_clean":"2023-10-30T13:23:30.027734+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T13:23:30.027734+0000","last_undegraded":"2023-10-30T13:23:30.027734+0000","last_fullsized":"2023-10-30T13:23:30.027734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:13.450999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107915+0000","last_change":"2023-10-30T13:23:19.722347+0000","last_active":"2023-10-30T13:23:21.107915+0000","last_peered":"2023-10-30T13:23:21.107915+0000","last_clean":"2023-10-30T13:23:21.107915+0000","last_became_active":"2023-10-30T13:23:19.722077+0000","last_became_peered":"2023-10-30T13:23:19.722077+0000","last_unstale":"2023-10-30T13:23:21.107915+0000","last_undegraded":"2023-10-30T13:23:21.107915+0000","last_fullsized":"2023-10-30T13:23:21.107915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:23:21.544691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.967593+0000","last_change":"2023-10-30T13:23:18.982337+0000","last_active":"2023-10-30T13:23:24.967593+0000","last_peered":"2023-10-30T13:23:24.967593+0000","last_clean":"2023-10-30T13:23:24.967593+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T13:23:24.967593+0000","last_undegraded":"2023-10-30T13:23:24.967593+0000","last_fullsized":"2023-10-30T13:23:24.967593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:09:39.996740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.676299+0000","last_change":"2023-10-30T13:23:19.706643+0000","last_active":"2023-10-30T13:23:20.676299+0000","last_peered":"2023-10-30T13:23:20.676299+0000","last_clean":"2023-10-30T13:23:20.676299+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T13:23:20.676299+0000","last_undegraded":"2023-10-30T13:23:20.676299+0000","last_fullsized":"2023-10-30T13:23:20.676299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:29.238430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'11","reported_seq":23,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:43.022865+0000","last_change":"2023-10-30T13:23:18.969882+0000","last_active":"2023-10-30T13:23:43.022865+0000","last_peered":"2023-10-30T13:23:43.022865+0000","last_clean":"2023-10-30T13:23:43.022865+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T13:23:43.022865+0000","last_undegraded":"2023-10-30T13:23:43.022865+0000","last_fullsized":"2023-10-30T13:23:43.022865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:17.093448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611588+0000","last_change":"2023-10-30T13:23:19.728295+0000","last_active":"2023-10-30T13:23:21.611588+0000","last_peered":"2023-10-30T13:23:21.611588+0000","last_clean":"2023-10-30T13:23:21.611588+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T13:23:21.611588+0000","last_undegraded":"2023-10-30T13:23:21.611588+0000","last_fullsized":"2023-10-30T13:23:21.611588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:37:54.545985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.732729+0000","last_change":"2023-10-30T13:23:18.971632+0000","last_active":"2023-10-30T13:23:35.732729+0000","last_peered":"2023-10-30T13:23:35.732729+0000","last_clean":"2023-10-30T13:23:35.732729+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T13:23:35.732729+0000","last_undegraded":"2023-10-30T13:23:35.732729+0000","last_fullsized":"2023-10-30T13:23:35.732729+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:54.428499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107653+0000","last_change":"2023-10-30T13:23:19.718168+0000","last_active":"2023-10-30T13:23:21.107653+0000","last_peered":"2023-10-30T13:23:21.107653+0000","last_clean":"2023-10-30T13:23:21.107653+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T13:23:21.107653+0000","last_undegraded":"2023-10-30T13:23:21.107653+0000","last_fullsized":"2023-10-30T13:23:21.107653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:54.832358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.029264+0000","last_change":"2023-10-30T13:23:18.978420+0000","last_active":"2023-10-30T13:23:30.029264+0000","last_peered":"2023-10-30T13:23:30.029264+0000","last_clean":"2023-10-30T13:23:30.029264+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T13:23:30.029264+0000","last_undegraded":"2023-10-30T13:23:30.029264+0000","last_fullsized":"2023-10-30T13:23:30.029264+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:10:44.511282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611634+0000","last_change":"2023-10-30T13:23:19.721434+0000","last_active":"2023-10-30T13:23:21.611634+0000","last_peered":"2023-10-30T13:23:21.611634+0000","last_clean":"2023-10-30T13:23:21.611634+0000","last_became_active":"2023-10-30T13:23:19.720704+0000","last_became_peered":"2023-10-30T13:23:19.720704+0000","last_unstale":"2023-10-30T13:23:21.611634+0000","last_undegraded":"2023-10-30T13:23:21.611634+0000","last_fullsized":"2023-10-30T13:23:21.611634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:45.728513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.737551+0000","last_change":"2023-10-30T13:23:18.972707+0000","last_active":"2023-10-30T13:23:35.737551+0000","last_peered":"2023-10-30T13:23:35.737551+0000","last_clean":"2023-10-30T13:23:35.737551+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T13:23:35.737551+0000","last_undegraded":"2023-10-30T13:23:35.737551+0000","last_fullsized":"2023-10-30T13:23:35.737551+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674810+0000","last_change":"2023-10-30T13:23:19.698274+0000","last_active":"2023-10-30T13:23:20.674810+0000","last_peered":"2023-10-30T13:23:20.674810+0000","last_clean":"2023-10-30T13:23:20.674810+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T13:23:20.674810+0000","last_undegraded":"2023-10-30T13:23:20.674810+0000","last_fullsized":"2023-10-30T13:23:20.674810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'15","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.734036+0000","last_change":"2023-10-30T13:23:18.948296+0000","last_active":"2023-10-30T13:23:35.734036+0000","last_peered":"2023-10-30T13:23:35.734036+0000","last_clean":"2023-10-30T13:23:35.734036+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T13:23:35.734036+0000","last_undegraded":"2023-10-30T13:23:35.734036+0000","last_fullsized":"2023-10-30T13:23:35.734036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:55.308329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611416+0000","last_change":"2023-10-30T13:23:19.728647+0000","last_active":"2023-10-30T13:23:21.611416+0000","last_peered":"2023-10-30T13:23:21.611416+0000","last_clean":"2023-10-30T13:23:21.611416+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T13:23:21.611416+0000","last_undegraded":"2023-10-30T13:23:21.611416+0000","last_fullsized":"2023-10-30T13:23:21.611416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:09.123809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.963649+0000","last_change":"2023-10-30T13:23:18.962343+0000","last_active":"2023-10-30T13:23:24.963649+0000","last_peered":"2023-10-30T13:23:24.963649+0000","last_clean":"2023-10-30T13:23:24.963649+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T13:23:24.963649+0000","last_undegraded":"2023-10-30T13:23:24.963649+0000","last_fullsized":"2023-10-30T13:23:24.963649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:36.082164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611928+0000","last_change":"2023-10-30T13:23:19.720770+0000","last_active":"2023-10-30T13:23:21.611928+0000","last_peered":"2023-10-30T13:23:21.611928+0000","last_clean":"2023-10-30T13:23:21.611928+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T13:23:21.611928+0000","last_undegraded":"2023-10-30T13:23:21.611928+0000","last_fullsized":"2023-10-30T13:23:21.611928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:37.378858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.962456+0000","last_change":"2023-10-30T13:23:18.949081+0000","last_active":"2023-10-30T13:23:24.962456+0000","last_peered":"2023-10-30T13:23:24.962456+0000","last_clean":"2023-10-30T13:23:24.962456+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T13:23:24.962456+0000","last_undegraded":"2023-10-30T13:23:24.962456+0000","last_fullsized":"2023-10-30T13:23:24.962456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:00.356885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613552+0000","last_change":"2023-10-30T13:23:19.724180+0000","last_active":"2023-10-30T13:23:21.613552+0000","last_peered":"2023-10-30T13:23:21.613552+0000","last_clean":"2023-10-30T13:23:21.613552+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T13:23:21.613552+0000","last_undegraded":"2023-10-30T13:23:21.613552+0000","last_fullsized":"2023-10-30T13:23:21.613552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:14.049454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.981649+0000","last_change":"2023-10-30T13:23:18.970780+0000","last_active":"2023-10-30T13:23:20.981649+0000","last_peered":"2023-10-30T13:23:20.981649+0000","last_clean":"2023-10-30T13:23:20.981649+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T13:23:20.981649+0000","last_undegraded":"2023-10-30T13:23:20.981649+0000","last_fullsized":"2023-10-30T13:23:20.981649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:38:51.113629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611979+0000","last_change":"2023-10-30T13:23:19.704834+0000","last_active":"2023-10-30T13:23:21.611979+0000","last_peered":"2023-10-30T13:23:21.611979+0000","last_clean":"2023-10-30T13:23:21.611979+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T13:23:21.611979+0000","last_undegraded":"2023-10-30T13:23:21.611979+0000","last_fullsized":"2023-10-30T13:23:21.611979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:11.266083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.734210+0000","last_change":"2023-10-30T13:23:18.977920+0000","last_active":"2023-10-30T13:23:35.734210+0000","last_peered":"2023-10-30T13:23:35.734210+0000","last_clean":"2023-10-30T13:23:35.734210+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T13:23:35.734210+0000","last_undegraded":"2023-10-30T13:23:35.734210+0000","last_fullsized":"2023-10-30T13:23:35.734210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:07:59.139948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611883+0000","last_change":"2023-10-30T13:23:19.724497+0000","last_active":"2023-10-30T13:23:21.611883+0000","last_peered":"2023-10-30T13:23:21.611883+0000","last_clean":"2023-10-30T13:23:21.611883+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T13:23:21.611883+0000","last_undegraded":"2023-10-30T13:23:21.611883+0000","last_fullsized":"2023-10-30T13:23:21.611883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:23.836868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:23:19.686506+0000","last_change":"2023-10-30T13:23:08.208031+0000","last_active":"2023-10-30T13:23:19.686506+0000","last_peered":"2023-10-30T13:23:19.686506+0000","last_clean":"2023-10-30T13:23:19.686506+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T13:23:19.686506+0000","last_undegraded":"2023-10-30T13:23:19.686506+0000","last_fullsized":"2023-10-30T13:23:19.686506+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_clean_scrub_stamp":"2023-10-30T13:23:07.179866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:22.691367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.027055+0000","last_change":"2023-10-30T13:23:18.979328+0000","last_active":"2023-10-30T13:23:30.027055+0000","last_peered":"2023-10-30T13:23:30.027055+0000","last_clean":"2023-10-30T13:23:30.027055+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T13:23:30.027055+0000","last_undegraded":"2023-10-30T13:23:30.027055+0000","last_fullsized":"2023-10-30T13:23:30.027055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:04.942593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005198+0000","last_change":"2023-10-30T13:23:19.711222+0000","last_active":"2023-10-30T13:23:21.005198+0000","last_peered":"2023-10-30T13:23:21.005198+0000","last_clean":"2023-10-30T13:23:21.005198+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T13:23:21.005198+0000","last_undegraded":"2023-10-30T13:23:21.005198+0000","last_fullsized":"2023-10-30T13:23:21.005198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:07.213498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.731662+0000","last_change":"2023-10-30T13:23:18.985318+0000","last_active":"2023-10-30T13:23:35.731662+0000","last_peered":"2023-10-30T13:23:35.731662+0000","last_clean":"2023-10-30T13:23:35.731662+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T13:23:35.731662+0000","last_undegraded":"2023-10-30T13:23:35.731662+0000","last_fullsized":"2023-10-30T13:23:35.731662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:16.754084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107580+0000","last_change":"2023-10-30T13:23:19.717938+0000","last_active":"2023-10-30T13:23:21.107580+0000","last_peered":"2023-10-30T13:23:21.107580+0000","last_clean":"2023-10-30T13:23:21.107580+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T13:23:21.107580+0000","last_undegraded":"2023-10-30T13:23:21.107580+0000","last_fullsized":"2023-10-30T13:23:21.107580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:56:01.624954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.732971+0000","last_change":"2023-10-30T13:23:18.949565+0000","last_active":"2023-10-30T13:23:35.732971+0000","last_peered":"2023-10-30T13:23:35.732971+0000","last_clean":"2023-10-30T13:23:35.732971+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T13:23:35.732971+0000","last_undegraded":"2023-10-30T13:23:35.732971+0000","last_fullsized":"2023-10-30T13:23:35.732971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:55:46.029368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670291+0000","last_change":"2023-10-30T13:23:19.678037+0000","last_active":"2023-10-30T13:23:20.670291+0000","last_peered":"2023-10-30T13:23:20.670291+0000","last_clean":"2023-10-30T13:23:20.670291+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T13:23:20.670291+0000","last_undegraded":"2023-10-30T13:23:20.670291+0000","last_fullsized":"2023-10-30T13:23:20.670291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:51.424848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613192+0000","last_change":"2023-10-30T13:23:18.984535+0000","last_active":"2023-10-30T13:23:21.613192+0000","last_peered":"2023-10-30T13:23:21.613192+0000","last_clean":"2023-10-30T13:23:21.613192+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T13:23:21.613192+0000","last_undegraded":"2023-10-30T13:23:21.613192+0000","last_fullsized":"2023-10-30T13:23:21.613192+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:28.405405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675980+0000","last_change":"2023-10-30T13:23:19.691867+0000","last_active":"2023-10-30T13:23:20.675980+0000","last_peered":"2023-10-30T13:23:20.675980+0000","last_clean":"2023-10-30T13:23:20.675980+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T13:23:20.675980+0000","last_undegraded":"2023-10-30T13:23:20.675980+0000","last_fullsized":"2023-10-30T13:23:20.675980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:02.276042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670494+0000","last_change":"2023-10-30T13:23:18.958210+0000","last_active":"2023-10-30T13:23:20.670494+0000","last_peered":"2023-10-30T13:23:20.670494+0000","last_clean":"2023-10-30T13:23:20.670494+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T13:23:20.670494+0000","last_undegraded":"2023-10-30T13:23:20.670494+0000","last_fullsized":"2023-10-30T13:23:20.670494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:34.653005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.613343+0000","last_change":"2023-10-30T13:23:19.713174+0000","last_active":"2023-10-30T13:23:21.613343+0000","last_peered":"2023-10-30T13:23:21.613343+0000","last_clean":"2023-10-30T13:23:21.613343+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T13:23:21.613343+0000","last_undegraded":"2023-10-30T13:23:21.613343+0000","last_fullsized":"2023-10-30T13:23:21.613343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:43.400223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612618+0000","last_change":"2023-10-30T13:23:18.985185+0000","last_active":"2023-10-30T13:23:21.612618+0000","last_peered":"2023-10-30T13:23:21.612618+0000","last_clean":"2023-10-30T13:23:21.612618+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T13:23:21.612618+0000","last_undegraded":"2023-10-30T13:23:21.612618+0000","last_fullsized":"2023-10-30T13:23:21.612618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:36.230072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.670523+0000","last_change":"2023-10-30T13:23:19.680185+0000","last_active":"2023-10-30T13:23:20.670523+0000","last_peered":"2023-10-30T13:23:20.670523+0000","last_clean":"2023-10-30T13:23:20.670523+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T13:23:20.670523+0000","last_undegraded":"2023-10-30T13:23:20.670523+0000","last_fullsized":"2023-10-30T13:23:20.670523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.394599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.733893+0000","last_change":"2023-10-30T13:23:18.978085+0000","last_active":"2023-10-30T13:23:35.733893+0000","last_peered":"2023-10-30T13:23:35.733893+0000","last_clean":"2023-10-30T13:23:35.733893+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T13:23:35.733893+0000","last_undegraded":"2023-10-30T13:23:35.733893+0000","last_fullsized":"2023-10-30T13:23:35.733893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:47.588972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005467+0000","last_change":"2023-10-30T13:23:19.704291+0000","last_active":"2023-10-30T13:23:21.005467+0000","last_peered":"2023-10-30T13:23:21.005467+0000","last_clean":"2023-10-30T13:23:21.005467+0000","last_became_active":"2023-10-30T13:23:19.704066+0000","last_became_peered":"2023-10-30T13:23:19.704066+0000","last_unstale":"2023-10-30T13:23:21.005467+0000","last_undegraded":"2023-10-30T13:23:21.005467+0000","last_fullsized":"2023-10-30T13:23:21.005467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:13.754194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.027200+0000","last_change":"2023-10-30T13:23:18.970566+0000","last_active":"2023-10-30T13:23:30.027200+0000","last_peered":"2023-10-30T13:23:30.027200+0000","last_clean":"2023-10-30T13:23:30.027200+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T13:23:30.027200+0000","last_undegraded":"2023-10-30T13:23:30.027200+0000","last_fullsized":"2023-10-30T13:23:30.027200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:03.851889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107903+0000","last_change":"2023-10-30T13:23:19.685481+0000","last_active":"2023-10-30T13:23:21.107903+0000","last_peered":"2023-10-30T13:23:21.107903+0000","last_clean":"2023-10-30T13:23:21.107903+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T13:23:21.107903+0000","last_undegraded":"2023-10-30T13:23:21.107903+0000","last_fullsized":"2023-10-30T13:23:21.107903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:33:50.672215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.037430+0000","last_change":"2023-10-30T13:23:18.986394+0000","last_active":"2023-10-30T13:23:30.037430+0000","last_peered":"2023-10-30T13:23:30.037430+0000","last_clean":"2023-10-30T13:23:30.037430+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T13:23:30.037430+0000","last_undegraded":"2023-10-30T13:23:30.037430+0000","last_fullsized":"2023-10-30T13:23:30.037430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:59.986981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.675944+0000","last_change":"2023-10-30T13:23:19.681239+0000","last_active":"2023-10-30T13:23:20.675944+0000","last_peered":"2023-10-30T13:23:20.675944+0000","last_clean":"2023-10-30T13:23:20.675944+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T13:23:20.675944+0000","last_undegraded":"2023-10-30T13:23:20.675944+0000","last_fullsized":"2023-10-30T13:23:20.675944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:00.121240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.036720+0000","last_change":"2023-10-30T13:23:18.963905+0000","last_active":"2023-10-30T13:23:30.036720+0000","last_peered":"2023-10-30T13:23:30.036720+0000","last_clean":"2023-10-30T13:23:30.036720+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T13:23:30.036720+0000","last_undegraded":"2023-10-30T13:23:30.036720+0000","last_fullsized":"2023-10-30T13:23:30.036720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005091+0000","last_change":"2023-10-30T13:23:19.678697+0000","last_active":"2023-10-30T13:23:21.005091+0000","last_peered":"2023-10-30T13:23:21.005091+0000","last_clean":"2023-10-30T13:23:21.005091+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T13:23:21.005091+0000","last_undegraded":"2023-10-30T13:23:21.005091+0000","last_fullsized":"2023-10-30T13:23:21.005091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:57.442011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.981565+0000","last_change":"2023-10-30T13:23:18.952302+0000","last_active":"2023-10-30T13:23:20.981565+0000","last_peered":"2023-10-30T13:23:20.981565+0000","last_clean":"2023-10-30T13:23:20.981565+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T13:23:20.981565+0000","last_undegraded":"2023-10-30T13:23:20.981565+0000","last_fullsized":"2023-10-30T13:23:20.981565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:12.827417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005323+0000","last_change":"2023-10-30T13:23:19.688716+0000","last_active":"2023-10-30T13:23:21.005323+0000","last_peered":"2023-10-30T13:23:21.005323+0000","last_clean":"2023-10-30T13:23:21.005323+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T13:23:21.005323+0000","last_undegraded":"2023-10-30T13:23:21.005323+0000","last_fullsized":"2023-10-30T13:23:21.005323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:26:04.209304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:30.022861+0000","last_change":"2023-10-30T13:23:18.984912+0000","last_active":"2023-10-30T13:23:30.022861+0000","last_peered":"2023-10-30T13:23:30.022861+0000","last_clean":"2023-10-30T13:23:30.022861+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T13:23:30.022861+0000","last_undegraded":"2023-10-30T13:23:30.022861+0000","last_fullsized":"2023-10-30T13:23:30.022861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:27:16.205042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:20.674560+0000","last_change":"2023-10-30T13:23:19.718922+0000","last_active":"2023-10-30T13:23:20.674560+0000","last_peered":"2023-10-30T13:23:20.674560+0000","last_clean":"2023-10-30T13:23:20.674560+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T13:23:20.674560+0000","last_undegraded":"2023-10-30T13:23:20.674560+0000","last_fullsized":"2023-10-30T13:23:20.674560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:35.959266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.612299+0000","last_change":"2023-10-30T13:23:19.732322+0000","last_active":"2023-10-30T13:23:21.612299+0000","last_peered":"2023-10-30T13:23:21.612299+0000","last_clean":"2023-10-30T13:23:21.612299+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T13:23:21.612299+0000","last_undegraded":"2023-10-30T13:23:21.612299+0000","last_fullsized":"2023-10-30T13:23:21.612299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:21.479138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.968310+0000","last_change":"2023-10-30T13:23:18.993372+0000","last_active":"2023-10-30T13:23:24.968310+0000","last_peered":"2023-10-30T13:23:24.968310+0000","last_clean":"2023-10-30T13:23:24.968310+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T13:23:24.968310+0000","last_undegraded":"2023-10-30T13:23:24.968310+0000","last_fullsized":"2023-10-30T13:23:24.968310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:41.370678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.611741+0000","last_change":"2023-10-30T13:23:19.732769+0000","last_active":"2023-10-30T13:23:21.611741+0000","last_peered":"2023-10-30T13:23:21.611741+0000","last_clean":"2023-10-30T13:23:21.611741+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T13:23:21.611741+0000","last_undegraded":"2023-10-30T13:23:21.611741+0000","last_fullsized":"2023-10-30T13:23:21.611741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:56.378098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.732313+0000","last_change":"2023-10-30T13:23:18.987633+0000","last_active":"2023-10-30T13:23:35.732313+0000","last_peered":"2023-10-30T13:23:35.732313+0000","last_clean":"2023-10-30T13:23:35.732313+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T13:23:35.732313+0000","last_undegraded":"2023-10-30T13:23:35.732313+0000","last_fullsized":"2023-10-30T13:23:35.732313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:09:33.324898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.107906+0000","last_change":"2023-10-30T13:23:19.722311+0000","last_active":"2023-10-30T13:23:21.107906+0000","last_peered":"2023-10-30T13:23:21.107906+0000","last_clean":"2023-10-30T13:23:21.107906+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T13:23:21.107906+0000","last_undegraded":"2023-10-30T13:23:21.107906+0000","last_fullsized":"2023-10-30T13:23:21.107906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:44.204074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:24.962089+0000","last_change":"2023-10-30T13:23:18.958470+0000","last_active":"2023-10-30T13:23:24.962089+0000","last_peered":"2023-10-30T13:23:24.962089+0000","last_clean":"2023-10-30T13:23:24.962089+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T13:23:24.962089+0000","last_undegraded":"2023-10-30T13:23:24.962089+0000","last_fullsized":"2023-10-30T13:23:24.962089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:21.507744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:21.005410+0000","last_change":"2023-10-30T13:23:19.699742+0000","last_active":"2023-10-30T13:23:21.005410+0000","last_peered":"2023-10-30T13:23:21.005410+0000","last_clean":"2023-10-30T13:23:21.005410+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T13:23:21.005410+0000","last_undegraded":"2023-10-30T13:23:21.005410+0000","last_fullsized":"2023-10-30T13:23:21.005410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:13.504378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'9","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:23:35.735807+0000","last_change":"2023-10-30T13:23:18.980952+0000","last_active":"2023-10-30T13:23:35.735807+0000","last_peered":"2023-10-30T13:23:35.735807+0000","last_clean":"2023-10-30T13:23:35.735807+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T13:23:35.735807+0000","last_undegraded":"2023-10-30T13:23:35.735807+0000","last_fullsized":"2023-10-30T13:23:35.735807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:44.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6467,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":12934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":819200,"data_stored":662582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":74433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2569999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1379999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":71612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":74467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":221184,"data_stored":70697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":71842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":811008,"data_stored":660907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:23:47.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:47.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.912+0000 7f29c8c6b640 1 Processor -- start 2023-10-30T13:23:47.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.912+0000 7f29c8c6b640 1 -- start start 2023-10-30T13:23:47.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c8c6b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29c41538a0 0x7f29c415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c8c6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29c40ca0e0 0x7f29c415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c8c6b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c40cb800 con 0x7f29c40cf160 2023-10-30T13:23:47.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c8c6b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c415dc30 con 0x7f29c41538a0 2023-10-30T13:23:47.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c8c6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40cb330 con 0x7f29c40ca0e0 2023-10-30T13:23:47.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29c41538a0 0x7f29c415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29c41538a0 0x7f29c415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47358/0 (socket says 172.21.15.46:47358) 2023-10-30T13:23:47.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c259c640 1 -- 172.21.15.46:0/1872625926 learned_addr learned my addr 172.21.15.46:0/1872625926 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:47.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29bbfff640 1 --2- 172.21.15.46:0/1872625926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29c40ca0e0 0x7f29c415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c259c640 1 -- 172.21.15.46:0/1872625926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29c40ca0e0 msgr2=0x7f29c415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:47.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c259c640 1 --2- 172.21.15.46:0/1872625926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29c40ca0e0 0x7f29c415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c259c640 1 -- 172.21.15.46:0/1872625926 >> v1:172.21.15.154:6789/0 conn(0x7f29c40cf160 legacy=0x7f29c40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c259c640 1 -- 172.21.15.46:0/1872625926 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c40cad70 con 0x7f29c41538a0 2023-10-30T13:23:47.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.916+0000 7f29c259c640 1 --2- 172.21.15.46:0/1872625926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c41538a0 0x7f29c415d6f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009870 tx=0x7f29b401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f8a6a70f27c0043 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:47.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c1d9b640 1 -- 172.21.15.46:0/1872625926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b40a8a30 con 0x7f29c41538a0 2023-10-30T13:23:47.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c1d9b640 1 -- 172.21.15.46:0/1872625926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b40a8b90 con 0x7f29c41538a0 2023-10-30T13:23:47.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c1d9b640 1 -- 172.21.15.46:0/1872625926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b40b0440 con 0x7f29c41538a0 2023-10-30T13:23:47.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1872625926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c41538a0 msgr2=0x7f29c415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 --2- 172.21.15.46:0/1872625926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c41538a0 0x7f29c415d6f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009870 tx=0x7f29b401ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:47.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1872625926 shutdown_connections 2023-10-30T13:23:47.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 --2- 172.21.15.46:0/1872625926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29c40ca0e0 0x7f29c415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 --2- 172.21.15.46:0/1872625926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c41538a0 0x7f29c415d6f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1872625926 >> 172.21.15.46:0/1872625926 conn(0x7f29c406bfe0 msgr2=0x7f29c406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:47.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1872625926 shutdown_connections 2023-10-30T13:23:47.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1872625926 wait complete. 2023-10-30T13:23:47.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 Processor -- start 2023-10-30T13:23:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 -- start start 2023-10-30T13:23:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29c40ca0e0 0x7f29c40ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29c40cf160 0x7f29c41e42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c41538a0 0x7f29c4147a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c41590a0 con 0x7f29c41538a0 2023-10-30T13:23:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c41e3560 con 0x7f29c40cf160 2023-10-30T13:23:47.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c8c6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c41f2e60 con 0x7f29c40ca0e0 2023-10-30T13:23:47.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c41538a0 0x7f29c4147a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c259c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29c40ca0e0 0x7f29c40ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c41538a0 0x7f29c4147a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47362/0 (socket says 172.21.15.46:47362) 2023-10-30T13:23:47.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.920+0000 7f29c2d9d640 1 -- 172.21.15.46:0/1882838932 learned_addr learned my addr 172.21.15.46:0/1882838932 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:47.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29bbfff640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29c40cf160 0x7f29c41e42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29c2d9d640 1 -- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29c40ca0e0 msgr2=0x7f29c40ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29c2d9d640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29c40ca0e0 0x7f29c40ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29c2d9d640 1 -- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29c40cf160 msgr2=0x7f29c41e42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29c2d9d640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29c40cf160 0x7f29c41e42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29c2d9d640 1 -- 172.21.15.46:0/1882838932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b40094d0 con 0x7f29c41538a0 2023-10-30T13:23:47.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29c2d9d640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c41538a0 0x7f29c4147a80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00c6c0 tx=0x7f29bc00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:47.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29baffd640 1 -- 172.21.15.46:0/1882838932 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc003b80 con 0x7f29c41538a0 2023-10-30T13:23:47.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29baffd640 1 -- 172.21.15.46:0/1882838932 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29bc003ce0 con 0x7f29c41538a0 2023-10-30T13:23:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29baffd640 1 -- 172.21.15.46:0/1882838932 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc0052f0 con 0x7f29c41538a0 2023-10-30T13:23:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1882838932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c41592b0 con 0x7f29c41538a0 2023-10-30T13:23:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1882838932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c41f3120 con 0x7f29c41538a0 2023-10-30T13:23:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1882838932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c40b2a50 con 0x7f29c41538a0 2023-10-30T13:23:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29baffd640 1 -- 172.21.15.46:0/1882838932 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29bc005450 con 0x7f29c41538a0 2023-10-30T13:23:47.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29baffd640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29a40664f0 0x7f29a40689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:47.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.924+0000 7f29baffd640 1 -- 172.21.15.46:0/1882838932 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29bc083b60 con 0x7f29c41538a0 2023-10-30T13:23:47.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.928+0000 7f29c259c640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29a40664f0 0x7f29a40689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:47.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.928+0000 7f29c259c640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29a40664f0 0x7f29a40689b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f29b40b2000 tx=0x7f29b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:47.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:47.928+0000 7f29baffd640 1 -- 172.21.15.46:0/1882838932 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29bc050fe0 con 0x7f29c41538a0 2023-10-30T13:23:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1882838932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29c4159870 con 0x7f29c41538a0 2023-10-30T13:23:48.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29baffd640 1 -- 172.21.15.46:0/1882838932 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f29c4159870 con 0x7f29c41538a0 2023-10-30T13:23:48.099 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:48.099 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:20.665723+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:48.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29a40664f0 msgr2=0x7f29a40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29a40664f0 0x7f29a40689b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f29b40b2000 tx=0x7f29b4009040 comp rx=0 tx=0).stop 2023-10-30T13:23:48.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1882838932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c41538a0 msgr2=0x7f29c4147a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c41538a0 0x7f29c4147a80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00c6c0 tx=0x7f29bc00cb80 comp rx=0 tx=0).stop 2023-10-30T13:23:48.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1882838932 shutdown_connections 2023-10-30T13:23:48.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29a40664f0 0x7f29a40689b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c41538a0 0x7f29c4147a80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29c40cf160 0x7f29c41e42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 --2- 172.21.15.46:0/1882838932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29c40ca0e0 0x7f29c40ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1882838932 >> 172.21.15.46:0/1882838932 conn(0x7f29c406bfe0 msgr2=0x7f29c4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:48.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1882838932 shutdown_connections 2023-10-30T13:23:48.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.092+0000 7f29c8c6b640 1 -- 172.21.15.46:0/1882838932 wait complete. 2023-10-30T13:23:48.121 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:23:06.788949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:23:17.648827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:23:18.326967+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:23:48.122 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:23:48.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.260+0000 7f53154dc640 1 Processor -- start 2023-10-30T13:23:48.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.260+0000 7f53154dc640 1 -- start start 2023-10-30T13:23:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f53154dc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5310153840 0x7f531015a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f53154dc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f53100cb4c0 0x7f53100cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f53154dc640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531015a6b0 con 0x7f5310153c20 2023-10-30T13:23:48.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f53154dc640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531015dbc0 con 0x7f53100cb4c0 2023-10-30T13:23:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f53154dc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531015d730 con 0x7f5310153840 2023-10-30T13:23:48.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f530ffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5310153840 0x7f531015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f5314cdb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f53100cb4c0 0x7f53100cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f530ffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5310153840 0x7f531015a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:37648/0 (socket says 172.21.15.46:37648) 2023-10-30T13:23:48.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f530ffff640 1 -- 172.21.15.46:0/363518775 learned_addr learned my addr 172.21.15.46:0/363518775 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:48.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f530ffff640 1 -- 172.21.15.46:0/363518775 >> v2:172.21.15.154:3300/0 conn(0x7f53100cb4c0 msgr2=0x7f53100cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:48.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f530ffff640 1 --2- 172.21.15.46:0/363518775 >> v2:172.21.15.154:3300/0 conn(0x7f53100cb4c0 0x7f53100cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f530ffff640 1 -- 172.21.15.46:0/363518775 >> v1:172.21.15.154:6789/0 conn(0x7f5310153c20 legacy=0x7f53100c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.264+0000 7f530ffff640 1 -- 172.21.15.46:0/363518775 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53100ca8f0 con 0x7f5310153840 2023-10-30T13:23:48.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f530ffff640 1 --2- 172.21.15.46:0/363518775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5310153840 0x7f531015a170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5300009870 tx=0x7f530001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e716c7d374dabcd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:48.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f530effd640 1 -- 172.21.15.46:0/363518775 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000a8a30 con 0x7f5310153840 2023-10-30T13:23:48.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f530effd640 1 -- 172.21.15.46:0/363518775 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53000a8b90 con 0x7f5310153840 2023-10-30T13:23:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f530effd640 1 -- 172.21.15.46:0/363518775 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000b0440 con 0x7f5310153840 2023-10-30T13:23:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f53154dc640 1 -- 172.21.15.46:0/363518775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5310153840 msgr2=0x7f531015a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f53154dc640 1 --2- 172.21.15.46:0/363518775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5310153840 0x7f531015a170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5300009870 tx=0x7f530001ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:48.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f53154dc640 1 -- 172.21.15.46:0/363518775 shutdown_connections 2023-10-30T13:23:48.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f53154dc640 1 --2- 172.21.15.46:0/363518775 >> v2:172.21.15.154:3300/0 conn(0x7f53100cb4c0 0x7f53100cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f53154dc640 1 --2- 172.21.15.46:0/363518775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5310153840 0x7f531015a170 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f53154dc640 1 -- 172.21.15.46:0/363518775 >> 172.21.15.46:0/363518775 conn(0x7f531006be40 msgr2=0x7f531006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:48.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f53154dc640 1 -- 172.21.15.46:0/363518775 shutdown_connections 2023-10-30T13:23:48.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f53154dc640 1 -- 172.21.15.46:0/363518775 wait complete. 2023-10-30T13:23:48.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f53154dc640 1 Processor -- start 2023-10-30T13:23:48.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.268+0000 7f53154dc640 1 -- start start 2023-10-30T13:23:48.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f53154dc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53100cb4c0 0x7f5310158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f53154dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5310153840 0x7f5310155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f53154dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5310153c20 0x7f5310155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f53154dc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5310155d90 con 0x7f5310153c20 2023-10-30T13:23:48.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f53154dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5310154cc0 con 0x7f5310153840 2023-10-30T13:23:48.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f53154dc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531015be60 con 0x7f53100cb4c0 2023-10-30T13:23:48.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f5314cdb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5310153c20 0x7f5310155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f5314cdb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5310153c20 0x7f5310155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47376/0 (socket says 172.21.15.46:47376) 2023-10-30T13:23:48.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f5314cdb640 1 -- 172.21.15.46:0/3428689894 learned_addr learned my addr 172.21.15.46:0/3428689894 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:48.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f530ffff640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53100cb4c0 0x7f5310158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f530f7fe640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5310153840 0x7f5310155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f5314cdb640 1 -- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53100cb4c0 msgr2=0x7f5310158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f5314cdb640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53100cb4c0 0x7f5310158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f5314cdb640 1 -- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5310153840 msgr2=0x7f5310155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f5314cdb640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5310153840 0x7f5310155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f5314cdb640 1 -- 172.21.15.46:0/3428689894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53000094d0 con 0x7f5310153c20 2023-10-30T13:23:48.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f5314cdb640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5310153c20 0x7f5310155850 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f530800efc0 tx=0x7f530800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:48.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f530cff9640 1 -- 172.21.15.46:0/3428689894 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308014480 con 0x7f5310153c20 2023-10-30T13:23:48.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f530cff9640 1 -- 172.21.15.46:0/3428689894 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530800f040 con 0x7f5310153c20 2023-10-30T13:23:48.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f530cff9640 1 -- 172.21.15.46:0/3428689894 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308014820 con 0x7f5310153c20 2023-10-30T13:23:48.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f53154dc640 1 -- 172.21.15.46:0/3428689894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531015c140 con 0x7f5310153c20 2023-10-30T13:23:48.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f53154dc640 1 -- 172.21.15.46:0/3428689894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5310158010 con 0x7f5310153c20 2023-10-30T13:23:48.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.272+0000 7f53154dc640 1 -- 172.21.15.46:0/3428689894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531004e9c0 con 0x7f5310153c20 2023-10-30T13:23:48.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.276+0000 7f530cff9640 1 -- 172.21.15.46:0/3428689894 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5308014980 con 0x7f5310153c20 2023-10-30T13:23:48.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.276+0000 7f530cff9640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f52fc0664a0 0x7f52fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.276+0000 7f530cff9640 1 -- 172.21.15.46:0/3428689894 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5308084c60 con 0x7f5310153c20 2023-10-30T13:23:48.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.276+0000 7f530ffff640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f52fc0664a0 0x7f52fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.276+0000 7f530ffff640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f52fc0664a0 0x7f52fc068960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f53000b2000 tx=0x7f53000aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:48.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.280+0000 7f530cff9640 1 -- 172.21.15.46:0/3428689894 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f53080520f0 con 0x7f5310153c20 2023-10-30T13:23:48.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 -- 172.21.15.46:0/3428689894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53100cb000 con 0x7f5310153c20 2023-10-30T13:23:48.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f530cff9640 1 -- 172.21.15.46:0/3428689894 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5308018020 con 0x7f5310153c20 2023-10-30T13:23:48.443 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-10-30T13:23:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 -- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f52fc0664a0 msgr2=0x7f52fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f52fc0664a0 0x7f52fc068960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f53000b2000 tx=0x7f53000aa070 comp rx=0 tx=0).stop 2023-10-30T13:23:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 -- 172.21.15.46:0/3428689894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5310153c20 msgr2=0x7f5310155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5310153c20 0x7f5310155850 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f530800efc0 tx=0x7f530800c490 comp rx=0 tx=0).stop 2023-10-30T13:23:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 -- 172.21.15.46:0/3428689894 shutdown_connections 2023-10-30T13:23:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f52fc0664a0 0x7f52fc068960 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5310153c20 0x7f5310155850 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5310153840 0x7f5310155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 --2- 172.21.15.46:0/3428689894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53100cb4c0 0x7f5310158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 -- 172.21.15.46:0/3428689894 >> 172.21.15.46:0/3428689894 conn(0x7f531006be40 msgr2=0x7f5310152420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:48.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 -- 172.21.15.46:0/3428689894 shutdown_connections 2023-10-30T13:23:48.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.436+0000 7f53154dc640 1 -- 172.21.15.46:0/3428689894 wait complete. 2023-10-30T13:23:48.464 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:23:48.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb584b3e640 1 Processor -- start 2023-10-30T13:23:48.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb584b3e640 1 -- start start 2023-10-30T13:23:48.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb584b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb58006f880 0x7fb58006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb584b3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb58006a7b0 0x7fb580161f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb584b3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5800c6cb0 con 0x7fb5800c5810 2023-10-30T13:23:48.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb584b3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb580071d90 con 0x7fb58006a7b0 2023-10-30T13:23:48.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb584b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5801585d0 con 0x7fb58006f880 2023-10-30T13:23:48.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb57ed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb58006a7b0 0x7fb580161f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb57e59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb5800c5810 0x7fb5800c5bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58948/0 (socket says 172.21.15.46:58948) 2023-10-30T13:23:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb57e59c640 1 -- 172.21.15.46:0/268200919 learned_addr learned my addr 172.21.15.46:0/268200919 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:23:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb57dd9b640 1 --2- 172.21.15.46:0/268200919 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb58006f880 0x7fb58006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb57ed9d640 1 -- 172.21.15.46:0/268200919 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb58006f880 msgr2=0x7fb58006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb57ed9d640 1 --2- 172.21.15.46:0/268200919 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb58006f880 0x7fb58006a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb57d59a640 1 -- 172.21.15.46:0/268200919 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641944657 0 0) 0x7fb5800c6cb0 con 0x7fb5800c5810 2023-10-30T13:23:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb57ed9d640 1 -- 172.21.15.46:0/268200919 >> v1:172.21.15.46:6789/0 conn(0x7fb5800c5810 legacy=0x7fb5800c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.604+0000 7fb57ed9d640 1 -- 172.21.15.46:0/268200919 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5800c6620 con 0x7fb58006a7b0 2023-10-30T13:23:48.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.608+0000 7fb57ed9d640 1 --2- 172.21.15.46:0/268200919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb58006a7b0 0x7fb580161f10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb578009dc0 tx=0x7fb57801c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41dab4227366c1f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:48.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.608+0000 7fb57d59a640 1 -- 172.21.15.46:0/268200919 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5780a66e0 con 0x7fb58006a7b0 2023-10-30T13:23:48.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.608+0000 7fb57d59a640 1 -- 172.21.15.46:0/268200919 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5780a6840 con 0x7fb58006a7b0 2023-10-30T13:23:48.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.608+0000 7fb57d59a640 1 -- 172.21.15.46:0/268200919 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5780a6bd0 con 0x7fb58006a7b0 2023-10-30T13:23:48.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.608+0000 7fb584b3e640 1 -- 172.21.15.46:0/268200919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb58006a7b0 msgr2=0x7fb580161f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.608+0000 7fb584b3e640 1 --2- 172.21.15.46:0/268200919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb58006a7b0 0x7fb580161f10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb578009dc0 tx=0x7fb57801c810 comp rx=0 tx=0).stop 2023-10-30T13:23:48.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.608+0000 7fb584b3e640 1 -- 172.21.15.46:0/268200919 shutdown_connections 2023-10-30T13:23:48.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.608+0000 7fb584b3e640 1 --2- 172.21.15.46:0/268200919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb58006a7b0 0x7fb580161f10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.608+0000 7fb584b3e640 1 --2- 172.21.15.46:0/268200919 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb58006f880 0x7fb58006a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.608+0000 7fb584b3e640 1 -- 172.21.15.46:0/268200919 >> 172.21.15.46:0/268200919 conn(0x7fb5800c7910 msgr2=0x7fb5800c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:48.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.608+0000 7fb584b3e640 1 -- 172.21.15.46:0/268200919 shutdown_connections 2023-10-30T13:23:48.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.608+0000 7fb584b3e640 1 -- 172.21.15.46:0/268200919 wait complete. 2023-10-30T13:23:48.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb584b3e640 1 Processor -- start 2023-10-30T13:23:48.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb584b3e640 1 -- start start 2023-10-30T13:23:48.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb584b3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb58006a7b0 0x7fb5801520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb584b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb58006f880 0x7fb5801f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb584b3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5800c5810 0x7fb5801f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb584b3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb580159820 con 0x7fb58006a7b0 2023-10-30T13:23:48.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb584b3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58015aad0 con 0x7fb5800c5810 2023-10-30T13:23:48.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb584b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5801587f0 con 0x7fb58006f880 2023-10-30T13:23:48.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb57e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb58006a7b0 0x7fb5801520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb57e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb58006a7b0 0x7fb5801520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47396/0 (socket says 172.21.15.46:47396) 2023-10-30T13:23:48.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb57e59c640 1 -- 172.21.15.46:0/1053193726 learned_addr learned my addr 172.21.15.46:0/1053193726 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:48.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb57ed9d640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5800c5810 0x7fb5801f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb57dd9b640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb58006f880 0x7fb5801f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb57e59c640 1 -- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb58006f880 msgr2=0x7fb5801f3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb57e59c640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb58006f880 0x7fb5801f3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb57e59c640 1 -- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5800c5810 msgr2=0x7fb5801f3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb57e59c640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5800c5810 0x7fb5801f3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb57e59c640 1 -- 172.21.15.46:0/1053193726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb578009a70 con 0x7fb58006a7b0 2023-10-30T13:23:48.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb57dd9b640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb58006f880 0x7fb5801f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:48.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb57e59c640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb58006a7b0 0x7fb5801520c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb5700010f0 tx=0x7fb57000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:48.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb56effd640 1 -- 172.21.15.46:0/1053193726 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570007270 con 0x7fb58006a7b0 2023-10-30T13:23:48.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb56effd640 1 -- 172.21.15.46:0/1053193726 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5700073d0 con 0x7fb58006a7b0 2023-10-30T13:23:48.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb584b3e640 1 -- 172.21.15.46:0/1053193726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58015a1a0 con 0x7fb58006a7b0 2023-10-30T13:23:48.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb56effd640 1 -- 172.21.15.46:0/1053193726 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700057b0 con 0x7fb58006a7b0 2023-10-30T13:23:48.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb584b3e640 1 -- 172.21.15.46:0/1053193726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58015a760 con 0x7fb58006a7b0 2023-10-30T13:23:48.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.612+0000 7fb584b3e640 1 -- 172.21.15.46:0/1053193726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5800c75d0 con 0x7fb58006a7b0 2023-10-30T13:23:48.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.616+0000 7fb56effd640 1 -- 172.21.15.46:0/1053193726 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb570005910 con 0x7fb58006a7b0 2023-10-30T13:23:48.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.616+0000 7fb56effd640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5600664d0 0x7fb560068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.616+0000 7fb56effd640 1 -- 172.21.15.46:0/1053193726 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb570083450 con 0x7fb58006a7b0 2023-10-30T13:23:48.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.616+0000 7fb57dd9b640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5600664d0 0x7fb560068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.616+0000 7fb57dd9b640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5600664d0 0x7fb560068990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb5800c47e0 tx=0x7fb574008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:48.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.620+0000 7fb56effd640 1 -- 172.21.15.46:0/1053193726 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb5700508e0 con 0x7fb58006a7b0 2023-10-30T13:23:48.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.776+0000 7fb584b3e640 1 -- 172.21.15.46:0/1053193726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb58004e170 con 0x7fb58006a7b0 2023-10-30T13:23:48.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb56effd640 1 -- 172.21.15.46:0/1053193726 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb570017020 con 0x7fb58006a7b0 2023-10-30T13:23:48.786 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-10-30T13:23:48.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb584b3e640 1 -- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5600664d0 msgr2=0x7fb560068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb584b3e640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5600664d0 0x7fb560068990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb5800c47e0 tx=0x7fb574008040 comp rx=0 tx=0).stop 2023-10-30T13:23:48.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb584b3e640 1 -- 172.21.15.46:0/1053193726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb58006a7b0 msgr2=0x7fb5801520c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb584b3e640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb58006a7b0 0x7fb5801520c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb5700010f0 tx=0x7fb57000cbe0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb584b3e640 1 -- 172.21.15.46:0/1053193726 shutdown_connections 2023-10-30T13:23:48.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb584b3e640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5600664d0 0x7fb560068990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb584b3e640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5800c5810 0x7fb5801f3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb584b3e640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb58006f880 0x7fb5801f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb584b3e640 1 --2- 172.21.15.46:0/1053193726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb58006a7b0 0x7fb5801520c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb584b3e640 1 -- 172.21.15.46:0/1053193726 >> 172.21.15.46:0/1053193726 conn(0x7fb5800c7910 msgr2=0x7fb58006d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:48.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb584b3e640 1 -- 172.21.15.46:0/1053193726 shutdown_connections 2023-10-30T13:23:48.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.780+0000 7fb584b3e640 1 -- 172.21.15.46:0/1053193726 wait complete. 2023-10-30T13:23:48.807 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:23:48.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.944+0000 7f003179d640 1 Processor -- start 2023-10-30T13:23:48.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f003179d640 1 -- start start 2023-10-30T13:23:48.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f003179d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f002c0c5810 0x7f002c0c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f003179d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f002c0c6130 0x7f002c06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f003179d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c0c6cb0 con 0x7f002c06f880 2023-10-30T13:23:48.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f003179d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f002c071d90 con 0x7f002c0c5810 2023-10-30T13:23:48.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f003179d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f002c1585d0 con 0x7f002c0c6130 2023-10-30T13:23:48.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f002affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f002c0c5810 0x7f002c0c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f002affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f002c0c5810 0x7f002c0c5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47406/0 (socket says 172.21.15.46:47406) 2023-10-30T13:23:48.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f002affd640 1 -- 172.21.15.46:0/2848007425 learned_addr learned my addr 172.21.15.46:0/2848007425 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:48.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f002a7fc640 1 --2- 172.21.15.46:0/2848007425 >> v2:172.21.15.154:3300/0 conn(0x7f002c0c6130 0x7f002c06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f002affd640 1 -- 172.21.15.46:0/2848007425 >> v2:172.21.15.154:3300/0 conn(0x7f002c0c6130 msgr2=0x7f002c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:48.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f002affd640 1 --2- 172.21.15.46:0/2848007425 >> v2:172.21.15.154:3300/0 conn(0x7f002c0c6130 0x7f002c06a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f002affd640 1 -- 172.21.15.46:0/2848007425 >> v1:172.21.15.154:6789/0 conn(0x7f002c06f880 legacy=0x7f002c15ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:48.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.948+0000 7f002affd640 1 -- 172.21.15.46:0/2848007425 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c0c6620 con 0x7f002c0c5810 2023-10-30T13:23:48.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f002affd640 1 --2- 172.21.15.46:0/2848007425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f002c0c5810 0x7f002c0c5bf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f002c071fe0 tx=0x7f001c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3bf82deb5575b27 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:48.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f0029ffb640 1 -- 172.21.15.46:0/2848007425 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c0ab600 con 0x7f002c0c5810 2023-10-30T13:23:48.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f0029ffb640 1 -- 172.21.15.46:0/2848007425 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001c0a7050 con 0x7f002c0c5810 2023-10-30T13:23:48.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f0029ffb640 1 -- 172.21.15.46:0/2848007425 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c0ab980 con 0x7f002c0c5810 2023-10-30T13:23:48.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f003179d640 1 -- 172.21.15.46:0/2848007425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f002c0c5810 msgr2=0x7f002c0c5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f003179d640 1 --2- 172.21.15.46:0/2848007425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f002c0c5810 0x7f002c0c5bf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f002c071fe0 tx=0x7f001c01c380 comp rx=0 tx=0).stop 2023-10-30T13:23:48.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f003179d640 1 -- 172.21.15.46:0/2848007425 shutdown_connections 2023-10-30T13:23:48.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f003179d640 1 --2- 172.21.15.46:0/2848007425 >> v2:172.21.15.154:3300/0 conn(0x7f002c0c6130 0x7f002c06a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f003179d640 1 --2- 172.21.15.46:0/2848007425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f002c0c5810 0x7f002c0c5bf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f003179d640 1 -- 172.21.15.46:0/2848007425 >> 172.21.15.46:0/2848007425 conn(0x7f002c0c7910 msgr2=0x7f002c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:48.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f003179d640 1 -- 172.21.15.46:0/2848007425 shutdown_connections 2023-10-30T13:23:48.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f003179d640 1 -- 172.21.15.46:0/2848007425 wait complete. 2023-10-30T13:23:48.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f003179d640 1 Processor -- start 2023-10-30T13:23:48.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.952+0000 7f003179d640 1 -- start start 2023-10-30T13:23:48.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f003179d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f002c06f880 0x7f002c14b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f003179d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f002c0c5810 0x7f002c0c0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f003179d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f002c0c6130 0x7f002c0c0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f003179d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0cac80 con 0x7f002c0c5810 2023-10-30T13:23:48.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f003179d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0c9a30 con 0x7f002c06f880 2023-10-30T13:23:48.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f003179d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c15a650 con 0x7f002c0c6130 2023-10-30T13:23:48.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f002a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f002c0c5810 0x7f002c0c0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f002a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f002c0c5810 0x7f002c0c0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47414/0 (socket says 172.21.15.46:47414) 2023-10-30T13:23:48.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f002a7fc640 1 -- 172.21.15.46:0/1389190212 learned_addr learned my addr 172.21.15.46:0/1389190212 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:48.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f002affd640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f002c06f880 0x7f002c14b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f002b7fe640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f002c0c6130 0x7f002c0c0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f002a7fc640 1 -- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f002c0c6130 msgr2=0x7f002c0c0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f002a7fc640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f002c0c6130 0x7f002c0c0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f002a7fc640 1 -- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f002c06f880 msgr2=0x7f002c14b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f002a7fc640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f002c06f880 0x7f002c14b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f002a7fc640 1 -- 172.21.15.46:0/1389190212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c009480 con 0x7f002c0c5810 2023-10-30T13:23:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f002a7fc640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f002c0c5810 0x7f002c0c0a60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f002000d7d0 tx=0x7f002000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f00137fe640 1 -- 172.21.15.46:0/1389190212 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002000b6c0 con 0x7f002c0c5810 2023-10-30T13:23:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f00137fe640 1 -- 172.21.15.46:0/1389190212 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002000b820 con 0x7f002c0c5810 2023-10-30T13:23:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f003179d640 1 -- 172.21.15.46:0/1389190212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002c0c8e20 con 0x7f002c0c5810 2023-10-30T13:23:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f00137fe640 1 -- 172.21.15.46:0/1389190212 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002000bbb0 con 0x7f002c0c5810 2023-10-30T13:23:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f003179d640 1 -- 172.21.15.46:0/1389190212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002c0c93e0 con 0x7f002c0c5810 2023-10-30T13:23:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.956+0000 7f003179d640 1 -- 172.21.15.46:0/1389190212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002c0c9bc0 con 0x7f002c0c5810 2023-10-30T13:23:48.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.960+0000 7f00137fe640 1 -- 172.21.15.46:0/1389190212 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f002000bd10 con 0x7f002c0c5810 2023-10-30T13:23:48.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.960+0000 7f00137fe640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00180664a0 0x7f0018068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:48.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.960+0000 7f00137fe640 1 -- 172.21.15.46:0/1389190212 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00200842d0 con 0x7f002c0c5810 2023-10-30T13:23:48.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.960+0000 7f002affd640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00180664a0 0x7f0018068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:48.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.960+0000 7f002affd640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00180664a0 0x7f0018068960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f001c0ad000 tx=0x7f001c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:48.964+0000 7f00137fe640 1 -- 172.21.15.46:0/1389190212 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0020051740 con 0x7f002c0c5810 2023-10-30T13:23:49.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 -- 172.21.15.46:0/1389190212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f002c0b0960 con 0x7f002c0c5810 2023-10-30T13:23:49.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f00137fe640 1 -- 172.21.15.46:0/1389190212 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0020054d60 con 0x7f002c0c5810 2023-10-30T13:23:49.127 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-10-30T13:23:49.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 -- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00180664a0 msgr2=0x7f0018068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00180664a0 0x7f0018068960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f001c0ad000 tx=0x7f001c009040 comp rx=0 tx=0).stop 2023-10-30T13:23:49.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 -- 172.21.15.46:0/1389190212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f002c0c5810 msgr2=0x7f002c0c0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f002c0c5810 0x7f002c0c0a60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f002000d7d0 tx=0x7f002000dc90 comp rx=0 tx=0).stop 2023-10-30T13:23:49.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 -- 172.21.15.46:0/1389190212 shutdown_connections 2023-10-30T13:23:49.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00180664a0 0x7f0018068960 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f002c0c6130 0x7f002c0c0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f002c0c5810 0x7f002c0c0a60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 --2- 172.21.15.46:0/1389190212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f002c06f880 0x7f002c14b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 -- 172.21.15.46:0/1389190212 >> 172.21.15.46:0/1389190212 conn(0x7f002c0c7910 msgr2=0x7f002c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 -- 172.21.15.46:0/1389190212 shutdown_connections 2023-10-30T13:23:49.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.120+0000 7f003179d640 1 -- 172.21.15.46:0/1389190212 wait complete. 2023-10-30T13:23:49.148 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde88ea640 1 Processor -- start 2023-10-30T13:23:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde88ea640 1 -- start start 2023-10-30T13:23:49.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde88ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdde406f7a0 0x7fdde406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde88ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdde4071110 0x7fdde4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde88ea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde4066cf0 con 0x7fdde40c6bd0 2023-10-30T13:23:49.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde88ea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde40700c0 con 0x7fdde4071110 2023-10-30T13:23:49.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde88ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde40c70d0 con 0x7fdde406f7a0 2023-10-30T13:23:49.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdde4071110 0x7fdde4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdde4071110 0x7fdde4158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47430/0 (socket says 172.21.15.46:47430) 2023-10-30T13:23:49.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde2d9d640 1 -- 172.21.15.46:0/2832056666 learned_addr learned my addr 172.21.15.46:0/2832056666 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:49.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde259c640 1 --2- 172.21.15.46:0/2832056666 >> v2:172.21.15.154:3300/0 conn(0x7fdde406f7a0 0x7fdde406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde2d9d640 1 -- 172.21.15.46:0/2832056666 >> v2:172.21.15.154:3300/0 conn(0x7fdde406f7a0 msgr2=0x7fdde406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:49.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde2d9d640 1 --2- 172.21.15.46:0/2832056666 >> v2:172.21.15.154:3300/0 conn(0x7fdde406f7a0 0x7fdde406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.284+0000 7fdde2d9d640 1 -- 172.21.15.46:0/2832056666 >> v1:172.21.15.154:6789/0 conn(0x7fdde40c6bd0 legacy=0x7fdde4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde2d9d640 1 -- 172.21.15.46:0/2832056666 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde40c7390 con 0x7fdde4071110 2023-10-30T13:23:49.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde2d9d640 1 --2- 172.21.15.46:0/2832056666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde4071110 0x7fdde4158110 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdde406c830 tx=0x7fdddc01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4b7a82b783e993 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:49.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde159a640 1 -- 172.21.15.46:0/2832056666 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc0af400 con 0x7fdde4071110 2023-10-30T13:23:49.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde159a640 1 -- 172.21.15.46:0/2832056666 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdddc0ab050 con 0x7fdde4071110 2023-10-30T13:23:49.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde159a640 1 -- 172.21.15.46:0/2832056666 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc0af780 con 0x7fdde4071110 2023-10-30T13:23:49.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde88ea640 1 -- 172.21.15.46:0/2832056666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde4071110 msgr2=0x7fdde4158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde88ea640 1 --2- 172.21.15.46:0/2832056666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde4071110 0x7fdde4158110 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdde406c830 tx=0x7fdddc01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde88ea640 1 -- 172.21.15.46:0/2832056666 shutdown_connections 2023-10-30T13:23:49.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde88ea640 1 --2- 172.21.15.46:0/2832056666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde4071110 0x7fdde4158110 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde88ea640 1 --2- 172.21.15.46:0/2832056666 >> v2:172.21.15.154:3300/0 conn(0x7fdde406f7a0 0x7fdde406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde88ea640 1 -- 172.21.15.46:0/2832056666 >> 172.21.15.46:0/2832056666 conn(0x7fdde40c7910 msgr2=0x7fdde40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:49.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde88ea640 1 -- 172.21.15.46:0/2832056666 shutdown_connections 2023-10-30T13:23:49.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.288+0000 7fdde88ea640 1 -- 172.21.15.46:0/2832056666 wait complete. 2023-10-30T13:23:49.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde88ea640 1 Processor -- start 2023-10-30T13:23:49.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde88ea640 1 -- start start 2023-10-30T13:23:49.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde88ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde406f7a0 0x7fdde41575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde88ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde4071110 0x7fdde4152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde88ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde40c6bd0 0x7fdde414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde88ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde406c560 con 0x7fdde406f7a0 2023-10-30T13:23:49.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde88ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde40c20b0 con 0x7fdde40c6bd0 2023-10-30T13:23:49.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde88ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde40c9340 con 0x7fdde4071110 2023-10-30T13:23:49.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde406f7a0 0x7fdde41575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde406f7a0 0x7fdde41575a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47442/0 (socket says 172.21.15.46:47442) 2023-10-30T13:23:49.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde259c640 1 -- 172.21.15.46:0/544262068 learned_addr learned my addr 172.21.15.46:0/544262068 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:49.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde2d9d640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde40c6bd0 0x7fdde414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde1d9b640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde4071110 0x7fdde4152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde259c640 1 -- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde4071110 msgr2=0x7fdde4152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde259c640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde4071110 0x7fdde4152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde259c640 1 -- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde40c6bd0 msgr2=0x7fdde414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde259c640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde40c6bd0 0x7fdde414efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde259c640 1 -- 172.21.15.46:0/544262068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc009b30 con 0x7fdde406f7a0 2023-10-30T13:23:49.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde259c640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde406f7a0 0x7fdde41575a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fddd400d5f0 tx=0x7fddd400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:49.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fddd2ffd640 1 -- 172.21.15.46:0/544262068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd40125f0 con 0x7fdde406f7a0 2023-10-30T13:23:49.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fddd2ffd640 1 -- 172.21.15.46:0/544262068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd4012750 con 0x7fdde406f7a0 2023-10-30T13:23:49.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fddd2ffd640 1 -- 172.21.15.46:0/544262068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd4012ae0 con 0x7fdde406f7a0 2023-10-30T13:23:49.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.292+0000 7fdde88ea640 1 -- 172.21.15.46:0/544262068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde4154b40 con 0x7fdde406f7a0 2023-10-30T13:23:49.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.296+0000 7fdde88ea640 1 -- 172.21.15.46:0/544262068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde40c9060 con 0x7fdde406f7a0 2023-10-30T13:23:49.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.296+0000 7fdde88ea640 1 -- 172.21.15.46:0/544262068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde4070330 con 0x7fdde406f7a0 2023-10-30T13:23:49.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.296+0000 7fddd2ffd640 1 -- 172.21.15.46:0/544262068 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fddd4012c40 con 0x7fdde406f7a0 2023-10-30T13:23:49.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.296+0000 7fddd2ffd640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddc40664a0 0x7fddc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.296+0000 7fddd2ffd640 1 -- 172.21.15.46:0/544262068 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddd4084270 con 0x7fdde406f7a0 2023-10-30T13:23:49.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.296+0000 7fdde1d9b640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddc40664a0 0x7fddc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.300+0000 7fdde1d9b640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddc40664a0 0x7fddc4068960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdde41e58f0 tx=0x7fddd8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.300+0000 7fddd2ffd640 1 -- 172.21.15.46:0/544262068 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdde4070330 con 0x7fdde406f7a0 2023-10-30T13:23:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 -- 172.21.15.46:0/544262068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdde4070330 con 0x7fdde406f7a0 2023-10-30T13:23:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fddd2ffd640 1 -- 172.21.15.46:0/544262068 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7fdde4070330 con 0x7fdde406f7a0 2023-10-30T13:23:49.511 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:49.511 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:35.740725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-10-30T13:23:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 -- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddc40664a0 msgr2=0x7fddc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddc40664a0 0x7fddc4068960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdde41e58f0 tx=0x7fddd8009040 comp rx=0 tx=0).stop 2023-10-30T13:23:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 -- 172.21.15.46:0/544262068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde406f7a0 msgr2=0x7fdde41575a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde406f7a0 0x7fdde41575a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fddd400d5f0 tx=0x7fddd400dab0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 -- 172.21.15.46:0/544262068 shutdown_connections 2023-10-30T13:23:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddc40664a0 0x7fddc4068960 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde40c6bd0 0x7fdde414efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde4071110 0x7fdde4152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 --2- 172.21.15.46:0/544262068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde406f7a0 0x7fdde41575a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 -- 172.21.15.46:0/544262068 >> 172.21.15.46:0/544262068 conn(0x7fdde40c7910 msgr2=0x7fdde40c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 -- 172.21.15.46:0/544262068 shutdown_connections 2023-10-30T13:23:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.504+0000 7fdde88ea640 1 -- 172.21.15.46:0/544262068 wait complete. 2023-10-30T13:23:49.532 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:23:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.668+0000 7f3a11aea640 1 Processor -- start 2023-10-30T13:23:49.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.668+0000 7f3a11aea640 1 -- start start 2023-10-30T13:23:49.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.668+0000 7f3a11aea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3a0c070a00 0x7f3a0c070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.668+0000 7f3a11aea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3a0c069e90 0x7f3a0c06a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.668+0000 7f3a11aea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c0713b0 con 0x7f3a0c1539a0 2023-10-30T13:23:49.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.668+0000 7f3a11aea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a0c06fbb0 con 0x7f3a0c070a00 2023-10-30T13:23:49.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.668+0000 7f3a11aea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a0c153d80 con 0x7f3a0c069e90 2023-10-30T13:23:49.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.672+0000 7f3a0affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3a0c070a00 0x7f3a0c070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.672+0000 7f3a0affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3a0c070a00 0x7f3a0c070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47444/0 (socket says 172.21.15.46:47444) 2023-10-30T13:23:49.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.672+0000 7f3a0b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3a0c1539a0 0x7f3a0c15a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58954/0 (socket says 172.21.15.46:58954) 2023-10-30T13:23:49.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.672+0000 7f3a0affd640 1 -- 172.21.15.46:0/2911692606 learned_addr learned my addr 172.21.15.46:0/2911692606 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:49.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.672+0000 7f3a0bfff640 1 --2- 172.21.15.46:0/2911692606 >> v2:172.21.15.154:3300/0 conn(0x7f3a0c069e90 0x7f3a0c06a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.672+0000 7f3a0affd640 1 -- 172.21.15.46:0/2911692606 >> v2:172.21.15.154:3300/0 conn(0x7f3a0c069e90 msgr2=0x7f3a0c06a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:49.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.672+0000 7f3a0affd640 1 --2- 172.21.15.46:0/2911692606 >> v2:172.21.15.154:3300/0 conn(0x7f3a0c069e90 0x7f3a0c06a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.672+0000 7f3a0affd640 1 -- 172.21.15.46:0/2911692606 >> v1:172.21.15.46:6789/0 conn(0x7f3a0c1539a0 legacy=0x7f3a0c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.672+0000 7f3a0affd640 1 -- 172.21.15.46:0/2911692606 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c0c5c50 con 0x7f3a0c070a00 2023-10-30T13:23:49.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.672+0000 7f3a0a7fc640 1 -- 172.21.15.46:0/2911692606 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914611140 0 0) 0x7f3a0c0713b0 con 0x7f3a0c1539a0 2023-10-30T13:23:49.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.672+0000 7f3a0bfff640 1 --2- 172.21.15.46:0/2911692606 >> v2:172.21.15.154:3300/0 conn(0x7f3a0c069e90 0x7f3a0c06a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:49.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.672+0000 7f3a0affd640 1 --2- 172.21.15.46:0/2911692606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a0c070a00 0x7f3a0c070e70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0b1080 tx=0x7f3a0001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d070cee1fbf8e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:49.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.676+0000 7f3a0a7fc640 1 -- 172.21.15.46:0/2911692606 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00004230 con 0x7f3a0c070a00 2023-10-30T13:23:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.676+0000 7f3a0a7fc640 1 -- 172.21.15.46:0/2911692606 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a000a7050 con 0x7f3a0c070a00 2023-10-30T13:23:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.676+0000 7f3a0a7fc640 1 -- 172.21.15.46:0/2911692606 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00003620 con 0x7f3a0c070a00 2023-10-30T13:23:49.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.676+0000 7f3a11aea640 1 -- 172.21.15.46:0/2911692606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a0c070a00 msgr2=0x7f3a0c070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.676+0000 7f3a11aea640 1 --2- 172.21.15.46:0/2911692606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a0c070a00 0x7f3a0c070e70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0b1080 tx=0x7f3a0001c440 comp rx=0 tx=0).stop 2023-10-30T13:23:49.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.676+0000 7f3a11aea640 1 -- 172.21.15.46:0/2911692606 shutdown_connections 2023-10-30T13:23:49.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.676+0000 7f3a11aea640 1 --2- 172.21.15.46:0/2911692606 >> v2:172.21.15.154:3300/0 conn(0x7f3a0c069e90 0x7f3a0c06a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.676+0000 7f3a11aea640 1 --2- 172.21.15.46:0/2911692606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a0c070a00 0x7f3a0c070e70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.676+0000 7f3a11aea640 1 -- 172.21.15.46:0/2911692606 >> 172.21.15.46:0/2911692606 conn(0x7f3a0c0c7910 msgr2=0x7f3a0c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:49.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.676+0000 7f3a11aea640 1 -- 172.21.15.46:0/2911692606 shutdown_connections 2023-10-30T13:23:49.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.676+0000 7f3a11aea640 1 -- 172.21.15.46:0/2911692606 wait complete. 2023-10-30T13:23:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.676+0000 7f3a11aea640 1 Processor -- start 2023-10-30T13:23:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a11aea640 1 -- start start 2023-10-30T13:23:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a11aea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a0c069e90 0x7f3a0c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a11aea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a0c070a00 0x7f3a0c144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a11aea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a0c1539a0 0x7f3a0c141870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a11aea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c157ed0 con 0x7f3a0c070a00 2023-10-30T13:23:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a11aea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c0c07f0 con 0x7f3a0c1539a0 2023-10-30T13:23:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a11aea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c143f90 con 0x7f3a0c069e90 2023-10-30T13:23:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a0affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a0c070a00 0x7f3a0c144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a0affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a0c070a00 0x7f3a0c144960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47454/0 (socket says 172.21.15.46:47454) 2023-10-30T13:23:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a0affd640 1 -- 172.21.15.46:0/3483781272 learned_addr learned my addr 172.21.15.46:0/3483781272 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:23:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a0b7fe640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a0c069e90 0x7f3a0c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a0bfff640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a0c1539a0 0x7f3a0c141870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a0affd640 1 -- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a0c069e90 msgr2=0x7f3a0c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a0affd640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a0c069e90 0x7f3a0c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a0affd640 1 -- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a0c1539a0 msgr2=0x7f3a0c141870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a0affd640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a0c1539a0 0x7f3a0c141870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a0affd640 1 -- 172.21.15.46:0/3483781272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a000094d0 con 0x7f3a0c070a00 2023-10-30T13:23:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a0affd640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a0c070a00 0x7f3a0c144960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3a00019050 tx=0x7f3a000191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:49.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f39f3fff640 1 -- 172.21.15.46:0/3483781272 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a000bb070 con 0x7f3a0c070a00 2023-10-30T13:23:49.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f39f3fff640 1 -- 172.21.15.46:0/3483781272 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a00003a70 con 0x7f3a0c070a00 2023-10-30T13:23:49.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f39f3fff640 1 -- 172.21.15.46:0/3483781272 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00003e20 con 0x7f3a0c070a00 2023-10-30T13:23:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a11aea640 1 -- 172.21.15.46:0/3483781272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0c157810 con 0x7f3a0c070a00 2023-10-30T13:23:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a11aea640 1 -- 172.21.15.46:0/3483781272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0c14d770 con 0x7f3a0c070a00 2023-10-30T13:23:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.680+0000 7f3a11aea640 1 -- 172.21.15.46:0/3483781272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0c04e170 con 0x7f3a0c070a00 2023-10-30T13:23:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.684+0000 7f39f3fff640 1 -- 172.21.15.46:0/3483781272 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3a000b2040 con 0x7f3a0c070a00 2023-10-30T13:23:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.684+0000 7f39f3fff640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f39f80664a0 0x7f39f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.684+0000 7f39f3fff640 1 -- 172.21.15.46:0/3483781272 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a0011d930 con 0x7f3a0c070a00 2023-10-30T13:23:49.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.684+0000 7f3a0b7fe640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f39f80664a0 0x7f39f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.684+0000 7f3a0b7fe640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f39f80664a0 0x7f39f8068960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f39fc000f70 tx=0x7f39fc007df0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:49.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.688+0000 7f39f3fff640 1 -- 172.21.15.46:0/3483781272 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3a000ead30 con 0x7f3a0c070a00 2023-10-30T13:23:49.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.848+0000 7f3a11aea640 1 -- 172.21.15.46:0/3483781272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a0c0c72f0 con 0x7f3a0c070a00 2023-10-30T13:23:49.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.848+0000 7f39f3fff640 1 -- 172.21.15.46:0/3483781272 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f3a000b0020 con 0x7f3a0c070a00 2023-10-30T13:23:49.855 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:23:49.856 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T13:23:20.665723+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:23:49.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.852+0000 7f3a11aea640 1 -- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f39f80664a0 msgr2=0x7f39f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.852+0000 7f3a11aea640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f39f80664a0 0x7f39f8068960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f39fc000f70 tx=0x7f39fc007df0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.852+0000 7f3a11aea640 1 -- 172.21.15.46:0/3483781272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a0c070a00 msgr2=0x7f3a0c144960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.852+0000 7f3a11aea640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a0c070a00 0x7f3a0c144960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3a00019050 tx=0x7f3a000191e0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.852+0000 7f3a11aea640 1 -- 172.21.15.46:0/3483781272 shutdown_connections 2023-10-30T13:23:49.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.852+0000 7f3a11aea640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f39f80664a0 0x7f39f8068960 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.852+0000 7f3a11aea640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a0c1539a0 0x7f3a0c141870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.852+0000 7f3a11aea640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a0c070a00 0x7f3a0c144960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.852+0000 7f3a11aea640 1 --2- 172.21.15.46:0/3483781272 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a0c069e90 0x7f3a0c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.852+0000 7f3a11aea640 1 -- 172.21.15.46:0/3483781272 >> 172.21.15.46:0/3483781272 conn(0x7f3a0c0c7910 msgr2=0x7f3a0c1529c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:49.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.852+0000 7f3a11aea640 1 -- 172.21.15.46:0/3483781272 shutdown_connections 2023-10-30T13:23:49.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:23:49.852+0000 7f3a11aea640 1 -- 172.21.15.46:0/3483781272 wait complete. 2023-10-30T13:23:49.877 DEBUG:tasks.mds_thrash:Yielding 2023-10-30T13:23:49.878 INFO:teuthology.run_tasks:Running task workunit... 2023-10-30T13:23:49.890 INFO:tasks.workunit:Pulling workunits from ref 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:23:49.891 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-30T13:23:49.891 INFO:tasks.workunit:timeout=6h 2023-10-30T13:23:49.891 INFO:tasks.workunit:cleanup=True 2023-10-30T13:23:49.892 DEBUG:teuthology.orchestra.run.smithi046:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-30T13:23:49.892 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-10-30T13:23:49.892 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 66 secs before thrashing 2023-10-30T13:23:49.902 INFO:teuthology.orchestra.run.smithi046.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-10-30T13:23:49.902 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-30T13:23:49.903 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 2bh/43d Inode: 1 Links: 2 2023-10-30T13:23:49.903 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:23:49.903 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-10-30T13:23:49.903 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-10-30 13:23:24.950609852 +0000 2023-10-30T13:23:49.904 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-10-30 13:23:43.040360186 +0000 2023-10-30T13:23:49.904 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2023-10-30 13:23:24.950609852 +0000 2023-10-30T13:23:49.904 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-10-30T13:23:49.904 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-10-30T13:23:49.973 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:23:50.027 INFO:tasks.workunit.client.0.smithi046.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-30T13:24:13.733 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:24:13.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:24:43.799 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:24:43.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:24:52.769 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 65% (7911/12071) Updating files: 66% (7967/12071) Updating files: 67% (8088/12071) Updating files: 68% (8209/12071) Updating files: 69% (8329/12071) Updating files: 70% (8450/12071) Updating files: 71% (8571/12071) Updating files: 72% (8692/12071) Updating files: 73% (8812/12071) Updating files: 74% (8933/12071) Updating files: 75% (9054/12071) Updating files: 76% (9174/12071) Updating files: 77% (9295/12071) Updating files: 78% (9416/12071) Updating files: 79% (9537/12071) Updating files: 80% (9657/12071) Updating files: 81% (9778/12071) Updating files: 82% (9899/12071) Updating files: 83% (10019/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 86% (10382/12071) Updating files: 87% (10502/12071) Updating files: 88% (10623/12071) Updating files: 89% (10744/12071) Updating files: 90% (10864/12071) Updating files: 91% (10985/12071) Updating files: 92% (11106/12071) Updating files: 93% (11227/12071) Updating files: 94% (11347/12071) Updating files: 95% (11468/12071) Updating files: 96% (11589/12071) Updating files: 97% (11709/12071) Updating files: 98% (11830/12071) Updating files: 99% (11951/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T13:24:53.348 INFO:tasks.workunit.client.0.smithi046.stderr:Note: switching to '00c1c13cea21a686b046584596fbeac5315a7d71'. 2023-10-30T13:24:53.348 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-10-30T13:24:53.349 INFO:tasks.workunit.client.0.smithi046.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-30T13:24:53.349 INFO:tasks.workunit.client.0.smithi046.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-30T13:24:53.349 INFO:tasks.workunit.client.0.smithi046.stderr:state without impacting any branches by switching back to a branch. 2023-10-30T13:24:53.349 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-10-30T13:24:53.350 INFO:tasks.workunit.client.0.smithi046.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-30T13:24:53.350 INFO:tasks.workunit.client.0.smithi046.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-30T13:24:53.350 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-10-30T13:24:53.350 INFO:tasks.workunit.client.0.smithi046.stderr: git switch -c 2023-10-30T13:24:53.351 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-10-30T13:24:53.351 INFO:tasks.workunit.client.0.smithi046.stderr:Or undo this operation with: 2023-10-30T13:24:53.351 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-10-30T13:24:53.351 INFO:tasks.workunit.client.0.smithi046.stderr: git switch - 2023-10-30T13:24:53.351 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-10-30T13:24:53.352 INFO:tasks.workunit.client.0.smithi046.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-30T13:24:53.352 INFO:tasks.workunit.client.0.smithi046.stderr: 2023-10-30T13:24:53.352 INFO:tasks.workunit.client.0.smithi046.stderr:HEAD is now at 00c1c13cea2 mds/MDBalancer: ignore queued callbacks if MDS is not active 2023-10-30T13:24:53.353 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-30T13:24:53.363 INFO:tasks.workunit.client.0.smithi046.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-30T13:24:53.366 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T13:24:53.367 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-30T13:24:54.109 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-30T13:24:54.230 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-30T13:24:54.296 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T13:24:54.298 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T13:24:54.299 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-30T13:24:54.354 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T13:24:54.359 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T13:24:54.360 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-30T13:24:54.409 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-10-30T13:24:54.411 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-10-30T13:24:54.411 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-10-30T13:24:54.470 INFO:tasks.workunit.client.0.smithi046.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-10-30T13:24:55.893 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:56.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.035+0000 7f60803b8640 1 Processor -- start 2023-10-30T13:24:56.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.035+0000 7f60803b8640 1 -- start start 2023-10-30T13:24:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f60803b8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6078151ad0 0x7f60781580c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f60803b8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6078151eb0 con 0x7f6078065e10 2023-10-30T13:24:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f60803b8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60780c5a40 con 0x7f60780c29e0 2023-10-30T13:24:56.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f60803b8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6078157900 con 0x7f6078151ad0 2023-10-30T13:24:56.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607e955640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6078065e10 0x7f60781620e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43216/0 (socket says 172.21.15.46:43216) 2023-10-30T13:24:56.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607e955640 1 -- 172.21.15.46:0/224130240 learned_addr learned my addr 172.21.15.46:0/224130240 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:24:56.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607e154640 1 --2- 172.21.15.46:0/224130240 >> v2:172.21.15.154:3300/0 conn(0x7f6078151ad0 0x7f60781580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359710092 0 0) 0x7f60700014a0 con 0x7f6078065e10 2023-10-30T13:24:56.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6064003530 con 0x7f6078065e10 2023-10-30T13:24:56.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2142459632 0 0) 0x7f6064003530 con 0x7f6078065e10 2023-10-30T13:24:56.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 >> v2:172.21.15.154:3300/0 conn(0x7f6078151ad0 msgr2=0x7f60781580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:56.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 --2- 172.21.15.46:0/224130240 >> v2:172.21.15.154:3300/0 conn(0x7f6078151ad0 0x7f60781580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 >> v1:172.21.15.154:6789/0 conn(0x7f60780c29e0 legacy=0x7f60780c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60780c5f60 con 0x7f6078065e10 2023-10-30T13:24:56.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f6070001b90 con 0x7f6078065e10 2023-10-30T13:24:56.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 >> v1:172.21.15.46:6789/0 conn(0x7f6078065e10 legacy=0x7f60781620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 --2- 172.21.15.46:0/224130240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6064003980 0x7f6064003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 --2- 172.21.15.46:0/224130240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6064004380 0x7f6064004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 --2- 172.21.15.46:0/224130240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6064005050 0x7f606400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6078157900 con 0x7f6064005050 2023-10-30T13:24:56.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606400bf00 con 0x7f6064004380 2023-10-30T13:24:56.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606400c0c0 con 0x7f6064003980 2023-10-30T13:24:56.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607e955640 1 --2- 172.21.15.46:0/224130240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6064005050 0x7f606400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607d953640 1 --2- 172.21.15.46:0/224130240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6064004380 0x7f6064004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607e154640 1 --2- 172.21.15.46:0/224130240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6064003980 0x7f6064003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607e955640 1 -- 172.21.15.46:0/224130240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6064003980 msgr2=0x7f6064003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:56.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607e955640 1 --2- 172.21.15.46:0/224130240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6064003980 0x7f6064003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607e955640 1 -- 172.21.15.46:0/224130240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6064004380 msgr2=0x7f6064004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:56.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607e955640 1 --2- 172.21.15.46:0/224130240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6064004380 0x7f6064004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607e955640 1 -- 172.21.15.46:0/224130240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064003530 con 0x7f6064005050 2023-10-30T13:24:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.039+0000 7f607e154640 1 --2- 172.21.15.46:0/224130240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6064003980 0x7f6064003d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f607e955640 1 -- 172.21.15.46:0/224130240 reap_dead start 2023-10-30T13:24:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f607e955640 1 --2- 172.21.15.46:0/224130240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6064005050 0x7f606400b990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f607000a780 tx=0x7f607001d260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f2a17ee8ca7e3b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60700ad630 con 0x7f6064005050 2023-10-30T13:24:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60700a8070 con 0x7f6064005050 2023-10-30T13:24:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f607d152640 1 -- 172.21.15.46:0/224130240 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60700ad9d0 con 0x7f6064005050 2023-10-30T13:24:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f60803b8640 1 -- 172.21.15.46:0/224130240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6064005050 msgr2=0x7f606400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f60803b8640 1 --2- 172.21.15.46:0/224130240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6064005050 0x7f606400b990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f607000a780 tx=0x7f607001d260 comp rx=0 tx=0).stop 2023-10-30T13:24:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f60803b8640 1 -- 172.21.15.46:0/224130240 shutdown_connections 2023-10-30T13:24:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f60803b8640 1 --2- 172.21.15.46:0/224130240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6064005050 0x7f606400b990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f60803b8640 1 -- 172.21.15.46:0/224130240 >> 172.21.15.46:0/224130240 conn(0x7f607806c6e0 msgr2=0x7f6078069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f60803b8640 1 -- 172.21.15.46:0/224130240 shutdown_connections 2023-10-30T13:24:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f60803b8640 1 -- 172.21.15.46:0/224130240 wait complete. 2023-10-30T13:24:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.043+0000 7f60803b8640 1 Processor -- start 2023-10-30T13:24:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f60803b8640 1 -- start start 2023-10-30T13:24:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f60803b8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6064005050 0x7f607814e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f60803b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6078070bb0 0x7f60781eec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f60803b8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078156630 0x7f6078156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f60803b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6078156ff0 con 0x7f6078070bb0 2023-10-30T13:24:56.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f60803b8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6078154760 con 0x7f6064005050 2023-10-30T13:24:56.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f60803b8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6078070530 con 0x7f6078156630 2023-10-30T13:24:56.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f607d953640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6078070bb0 0x7f60781eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f607d953640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6078070bb0 0x7f60781eec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35614/0 (socket says 172.21.15.46:35614) 2023-10-30T13:24:56.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f607d953640 1 -- 172.21.15.46:0/3645653033 learned_addr learned my addr 172.21.15.46:0/3645653033 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:24:56.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f607e154640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6064005050 0x7f607814e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f607e955640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078156630 0x7f6078156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f607d953640 1 -- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078156630 msgr2=0x7f6078156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f607d953640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078156630 0x7f6078156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f607d953640 1 -- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6064005050 msgr2=0x7f607814e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f607d953640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6064005050 0x7f607814e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f607d953640 1 -- 172.21.15.46:0/3645653033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60780707d0 con 0x7f6078070bb0 2023-10-30T13:24:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f607e955640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078156630 0x7f6078156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f607d953640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6078070bb0 0x7f60781eec70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f606800f870 tx=0x7f606800fd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f606effd640 1 -- 172.21.15.46:0/3645653033 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068001d10 con 0x7f6078070bb0 2023-10-30T13:24:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f606effd640 1 -- 172.21.15.46:0/3645653033 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6068005320 con 0x7f6078070bb0 2023-10-30T13:24:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f60803b8640 1 -- 172.21.15.46:0/3645653033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60781f21e0 con 0x7f6078070bb0 2023-10-30T13:24:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f606effd640 1 -- 172.21.15.46:0/3645653033 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068006a20 con 0x7f6078070bb0 2023-10-30T13:24:56.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f60803b8640 1 -- 172.21.15.46:0/3645653033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60781f27a0 con 0x7f6078070bb0 2023-10-30T13:24:56.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.047+0000 7f60803b8640 1 -- 172.21.15.46:0/3645653033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607804e170 con 0x7f6078070bb0 2023-10-30T13:24:56.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.051+0000 7f606effd640 1 -- 172.21.15.46:0/3645653033 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6068006b80 con 0x7f6078070bb0 2023-10-30T13:24:56.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.051+0000 7f606effd640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f605c0664a0 0x7f605c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.051+0000 7f606effd640 1 -- 172.21.15.46:0/3645653033 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6068083850 con 0x7f6078070bb0 2023-10-30T13:24:56.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.051+0000 7f607e154640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f605c0664a0 0x7f605c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.051+0000 7f607e154640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f605c0664a0 0x7f605c068960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f60740099b0 tx=0x7f6074008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:56.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.055+0000 7f606effd640 1 -- 172.21.15.46:0/3645653033 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6068050c40 con 0x7f6078070bb0 2023-10-30T13:24:56.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 -- 172.21.15.46:0/3645653033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60780c65b0 con 0x7f6078070bb0 2023-10-30T13:24:56.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f606effd640 1 -- 172.21.15.46:0/3645653033 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f606801a090 con 0x7f6078070bb0 2023-10-30T13:24:56.251 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:24:56.251 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:35.740725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:56.252 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-10-30T13:24:56.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 -- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f605c0664a0 msgr2=0x7f605c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f605c0664a0 0x7f605c068960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f60740099b0 tx=0x7f6074008040 comp rx=0 tx=0).stop 2023-10-30T13:24:56.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 -- 172.21.15.46:0/3645653033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6078070bb0 msgr2=0x7f60781eec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6078070bb0 0x7f60781eec70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f606800f870 tx=0x7f606800fd30 comp rx=0 tx=0).stop 2023-10-30T13:24:56.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 -- 172.21.15.46:0/3645653033 shutdown_connections 2023-10-30T13:24:56.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f605c0664a0 0x7f605c068960 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078156630 0x7f6078156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6078070bb0 0x7f60781eec70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 --2- 172.21.15.46:0/3645653033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6064005050 0x7f607814e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 -- 172.21.15.46:0/3645653033 >> 172.21.15.46:0/3645653033 conn(0x7f607806c6e0 msgr2=0x7f6078150010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:56.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 -- 172.21.15.46:0/3645653033 shutdown_connections 2023-10-30T13:24:56.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.247+0000 7f60803b8640 1 -- 172.21.15.46:0/3645653033 wait complete. 2023-10-30T13:24:56.272 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=2) 2023-10-30T13:24:56.273 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:24:56.273 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:24:56.287 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:24:56.287 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:56.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1eac9640 1 Processor -- start 2023-10-30T13:24:56.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1eac9640 1 -- start start 2023-10-30T13:24:56.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1eac9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde18153d70 0x7fde1806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1eac9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde1806e470 0x7fde180cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1eac9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1806afa0 con 0x7fde180c8490 2023-10-30T13:24:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1eac9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1806d780 con 0x7fde18153d70 2023-10-30T13:24:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1eac9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde18154150 con 0x7fde1806e470 2023-10-30T13:24:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1c865640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde18153d70 0x7fde1806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1d066640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fde180c8490 0x7fde180c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34512/0 (socket says 172.21.15.46:34512) 2023-10-30T13:24:56.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1d066640 1 -- 172.21.15.46:0/561807343 learned_addr learned my addr 172.21.15.46:0/561807343 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:24:56.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1c865640 1 --2- 172.21.15.46:0/561807343 >> v2:172.21.15.46:3300/0 conn(0x7fde18153d70 0x7fde1806ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41012/0 (socket says 172.21.15.46:41012) 2023-10-30T13:24:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde17fff640 1 --2- 172.21.15.46:0/561807343 >> v2:172.21.15.154:3300/0 conn(0x7fde1806e470 0x7fde180cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1c865640 1 -- 172.21.15.46:0/561807343 >> v2:172.21.15.154:3300/0 conn(0x7fde1806e470 msgr2=0x7fde180cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1c865640 1 --2- 172.21.15.46:0/561807343 >> v2:172.21.15.154:3300/0 conn(0x7fde1806e470 0x7fde180cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1c865640 1 -- 172.21.15.46:0/561807343 >> v1:172.21.15.46:6789/0 conn(0x7fde180c8490 legacy=0x7fde180c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde1c865640 1 -- 172.21.15.46:0/561807343 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1806e030 con 0x7fde18153d70 2023-10-30T13:24:56.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.471+0000 7fde177fe640 1 -- 172.21.15.46:0/561807343 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638454249 0 0) 0x7fde1806afa0 con 0x7fde180c8490 2023-10-30T13:24:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.475+0000 7fde1c865640 1 --2- 172.21.15.46:0/561807343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde18153d70 0x7fde1806ca80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fde18070230 tx=0x7fde0c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879e100515e58719 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.475+0000 7fde177fe640 1 -- 172.21.15.46:0/561807343 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c0a7070 con 0x7fde18153d70 2023-10-30T13:24:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.475+0000 7fde177fe640 1 -- 172.21.15.46:0/561807343 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c0ab430 con 0x7fde18153d70 2023-10-30T13:24:56.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.475+0000 7fde177fe640 1 -- 172.21.15.46:0/561807343 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c0ab7c0 con 0x7fde18153d70 2023-10-30T13:24:56.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.475+0000 7fde1eac9640 1 -- 172.21.15.46:0/561807343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde18153d70 msgr2=0x7fde1806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.475+0000 7fde1eac9640 1 --2- 172.21.15.46:0/561807343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde18153d70 0x7fde1806ca80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fde18070230 tx=0x7fde0c01c430 comp rx=0 tx=0).stop 2023-10-30T13:24:56.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.475+0000 7fde1eac9640 1 -- 172.21.15.46:0/561807343 shutdown_connections 2023-10-30T13:24:56.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.475+0000 7fde1eac9640 1 --2- 172.21.15.46:0/561807343 >> v2:172.21.15.154:3300/0 conn(0x7fde1806e470 0x7fde180cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.475+0000 7fde1eac9640 1 --2- 172.21.15.46:0/561807343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde18153d70 0x7fde1806ca80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.475+0000 7fde1eac9640 1 -- 172.21.15.46:0/561807343 >> 172.21.15.46:0/561807343 conn(0x7fde18071ea0 msgr2=0x7fde18065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:56.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.475+0000 7fde1eac9640 1 -- 172.21.15.46:0/561807343 shutdown_connections 2023-10-30T13:24:56.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.475+0000 7fde1eac9640 1 -- 172.21.15.46:0/561807343 wait complete. 2023-10-30T13:24:56.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1eac9640 1 Processor -- start 2023-10-30T13:24:56.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1eac9640 1 -- start start 2023-10-30T13:24:56.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1eac9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1806e470 0x7fde18144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1eac9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde180c8490 0x7fde180cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1eac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde18153d70 0x7fde18144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1eac9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde18155320 con 0x7fde1806e470 2023-10-30T13:24:56.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1eac9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde181f6640 con 0x7fde180c8490 2023-10-30T13:24:56.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1eac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde181f5da0 con 0x7fde18153d70 2023-10-30T13:24:56.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1c865640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1806e470 0x7fde18144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1c865640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1806e470 0x7fde18144bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41016/0 (socket says 172.21.15.46:41016) 2023-10-30T13:24:56.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1c865640 1 -- 172.21.15.46:0/3339796108 learned_addr learned my addr 172.21.15.46:0/3339796108 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:24:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde17fff640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde180c8490 0x7fde180cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1d066640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde18153d70 0x7fde18144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1c865640 1 -- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde18153d70 msgr2=0x7fde18144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1c865640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde18153d70 0x7fde18144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1c865640 1 -- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde180c8490 msgr2=0x7fde180cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1c865640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde180c8490 0x7fde180cb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1c865640 1 -- 172.21.15.46:0/3339796108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c009420 con 0x7fde1806e470 2023-10-30T13:24:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1c865640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1806e470 0x7fde18144bd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fde0c01a060 tx=0x7fde0c01ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde157fa640 1 -- 172.21.15.46:0/3339796108 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c0b5810 con 0x7fde1806e470 2023-10-30T13:24:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde157fa640 1 -- 172.21.15.46:0/3339796108 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c0b5990 con 0x7fde1806e470 2023-10-30T13:24:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde157fa640 1 -- 172.21.15.46:0/3339796108 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c0b5d40 con 0x7fde1806e470 2023-10-30T13:24:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1eac9640 1 -- 172.21.15.46:0/3339796108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde18145210 con 0x7fde1806e470 2023-10-30T13:24:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1eac9640 1 -- 172.21.15.46:0/3339796108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde181f5ff0 con 0x7fde1806e470 2023-10-30T13:24:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.479+0000 7fde1eac9640 1 -- 172.21.15.46:0/3339796108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1804e170 con 0x7fde1806e470 2023-10-30T13:24:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.483+0000 7fde157fa640 1 -- 172.21.15.46:0/3339796108 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fde0c0b4390 con 0x7fde1806e470 2023-10-30T13:24:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.483+0000 7fde157fa640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddfc0664d0 0x7fddfc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.483+0000 7fde157fa640 1 -- 172.21.15.46:0/3339796108 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fde0c11e3a0 con 0x7fde1806e470 2023-10-30T13:24:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.483+0000 7fde17fff640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddfc0664d0 0x7fddfc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.483+0000 7fde17fff640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddfc0664d0 0x7fddfc068990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fde18145d20 tx=0x7fde04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.487+0000 7fde157fa640 1 -- 172.21.15.46:0/3339796108 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fde0c0eb7a0 con 0x7fde1806e470 2023-10-30T13:24:56.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.679+0000 7fde1eac9640 1 -- 172.21.15.46:0/3339796108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde1806c2a0 con 0x7fde1806e470 2023-10-30T13:24:56.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde157fa640 1 -- 172.21.15.46:0/3339796108 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7fde0c006e20 con 0x7fde1806e470 2023-10-30T13:24:56.686 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:24:56.687 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:35.740725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:56.688 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-10-30T13:24:56.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde1eac9640 1 -- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddfc0664d0 msgr2=0x7fddfc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde1eac9640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddfc0664d0 0x7fddfc068990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fde18145d20 tx=0x7fde04008040 comp rx=0 tx=0).stop 2023-10-30T13:24:56.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde1eac9640 1 -- 172.21.15.46:0/3339796108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1806e470 msgr2=0x7fde18144bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:56.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde1eac9640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1806e470 0x7fde18144bd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fde0c01a060 tx=0x7fde0c01ac70 comp rx=0 tx=0).stop 2023-10-30T13:24:56.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde1eac9640 1 -- 172.21.15.46:0/3339796108 shutdown_connections 2023-10-30T13:24:56.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde1eac9640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddfc0664d0 0x7fddfc068990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde1eac9640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde18153d70 0x7fde18144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde1eac9640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde180c8490 0x7fde180cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde1eac9640 1 --2- 172.21.15.46:0/3339796108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde1806e470 0x7fde18144bd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:56.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde1eac9640 1 -- 172.21.15.46:0/3339796108 >> 172.21.15.46:0/3339796108 conn(0x7fde18071ea0 msgr2=0x7fde180c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:56.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde1eac9640 1 -- 172.21.15.46:0/3339796108 shutdown_connections 2023-10-30T13:24:56.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:56.683+0000 7fde1eac9640 1 -- 172.21.15.46:0/3339796108 wait complete. 2023-10-30T13:24:56.707 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:24:57.392 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir foo 2023-10-30T13:24:57.399 INFO:tasks.workunit.client.0.smithi046.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-10-30T13:24:57.495 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir foo/.snap/barsnap 2023-10-30T13:24:58.708 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.859+0000 7f6012fd3640 1 Processor -- start 2023-10-30T13:24:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.859+0000 7f6012fd3640 1 -- start start 2023-10-30T13:24:58.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.859+0000 7f6012fd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f600c06f7a0 0x7f600c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:58.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.859+0000 7f6012fd3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f600c0c6bd0 0x7f600c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:58.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.859+0000 7f6012fd3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c06a3d0 con 0x7f600c071110 2023-10-30T13:24:58.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.859+0000 7f6012fd3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f600c0700c0 con 0x7f600c0c6bd0 2023-10-30T13:24:58.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.859+0000 7f6012fd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c066cf0 con 0x7f600c06f7a0 2023-10-30T13:24:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.863+0000 7f6010d6f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f600c06f7a0 0x7f600c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.863+0000 7f600bfff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f600c0c6bd0 0x7f600c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:58.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.863+0000 7f6010d6f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f600c06f7a0 0x7f600c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:58554/0 (socket says 172.21.15.46:58554) 2023-10-30T13:24:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.863+0000 7f6010d6f640 1 -- 172.21.15.46:0/2612528385 learned_addr learned my addr 172.21.15.46:0/2612528385 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:24:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.863+0000 7f6010d6f640 1 -- 172.21.15.46:0/2612528385 >> v2:172.21.15.154:3300/0 conn(0x7f600c0c6bd0 msgr2=0x7f600c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.863+0000 7f6010d6f640 1 --2- 172.21.15.46:0/2612528385 >> v2:172.21.15.154:3300/0 conn(0x7f600c0c6bd0 0x7f600c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.863+0000 7f6010d6f640 1 -- 172.21.15.46:0/2612528385 >> v1:172.21.15.154:6789/0 conn(0x7f600c071110 legacy=0x7f600c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:58.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.863+0000 7f6010d6f640 1 -- 172.21.15.46:0/2612528385 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c0c70d0 con 0x7f600c06f7a0 2023-10-30T13:24:58.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.863+0000 7f6010d6f640 1 --2- 172.21.15.46:0/2612528385 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f600c06f7a0 0x7f600c06fb80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6000009870 tx=0x7f600001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3460cfeaae8c225 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:58.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f600b7fe640 1 -- 172.21.15.46:0/2612528385 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60000a8a30 con 0x7f600c06f7a0 2023-10-30T13:24:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f600b7fe640 1 -- 172.21.15.46:0/2612528385 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60000a8b90 con 0x7f600c06f7a0 2023-10-30T13:24:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f600b7fe640 1 -- 172.21.15.46:0/2612528385 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60000b0440 con 0x7f600c06f7a0 2023-10-30T13:24:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f6012fd3640 1 -- 172.21.15.46:0/2612528385 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f600c06f7a0 msgr2=0x7f600c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f6012fd3640 1 --2- 172.21.15.46:0/2612528385 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f600c06f7a0 0x7f600c06fb80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6000009870 tx=0x7f600001ed30 comp rx=0 tx=0).stop 2023-10-30T13:24:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f6012fd3640 1 -- 172.21.15.46:0/2612528385 shutdown_connections 2023-10-30T13:24:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f6012fd3640 1 --2- 172.21.15.46:0/2612528385 >> v2:172.21.15.154:3300/0 conn(0x7f600c0c6bd0 0x7f600c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f6012fd3640 1 --2- 172.21.15.46:0/2612528385 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f600c06f7a0 0x7f600c06fb80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f6012fd3640 1 -- 172.21.15.46:0/2612528385 >> 172.21.15.46:0/2612528385 conn(0x7f600c0c7910 msgr2=0x7f600c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f6012fd3640 1 -- 172.21.15.46:0/2612528385 shutdown_connections 2023-10-30T13:24:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f6012fd3640 1 -- 172.21.15.46:0/2612528385 wait complete. 2023-10-30T13:24:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f6012fd3640 1 Processor -- start 2023-10-30T13:24:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.867+0000 7f6012fd3640 1 -- start start 2023-10-30T13:24:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6012fd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f600c06f7a0 0x7f600c0c0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6012fd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f600c071110 0x7f600c1e2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6012fd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f600c0c6bd0 0x7f600c1e2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6012fd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0ae800 con 0x7f600c06f7a0 2023-10-30T13:24:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6012fd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0adc80 con 0x7f600c0c6bd0 2023-10-30T13:24:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6012fd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0adf60 con 0x7f600c071110 2023-10-30T13:24:58.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6010d6f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f600c06f7a0 0x7f600c0c0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:58.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6010d6f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f600c06f7a0 0x7f600c0c0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41026/0 (socket says 172.21.15.46:41026) 2023-10-30T13:24:58.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6010d6f640 1 -- 172.21.15.46:0/3297550458 learned_addr learned my addr 172.21.15.46:0/3297550458 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:24:58.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f600bfff640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f600c071110 0x7f600c1e2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:58.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6011570640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f600c0c6bd0 0x7f600c1e2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:58.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6010d6f640 1 -- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f600c071110 msgr2=0x7f600c1e2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:58.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6010d6f640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f600c071110 0x7f600c1e2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:58.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6010d6f640 1 -- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f600c0c6bd0 msgr2=0x7f600c1e2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:58.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6010d6f640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f600c0c6bd0 0x7f600c1e2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:58.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6010d6f640 1 -- 172.21.15.46:0/3297550458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60000094d0 con 0x7f600c06f7a0 2023-10-30T13:24:58.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6010d6f640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f600c06f7a0 0x7f600c0c0540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6000000c10 tx=0x7f600001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:58.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f60097fa640 1 -- 172.21.15.46:0/3297550458 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60000bf070 con 0x7f600c06f7a0 2023-10-30T13:24:58.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f60097fa640 1 -- 172.21.15.46:0/3297550458 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60000b1be0 con 0x7f600c06f7a0 2023-10-30T13:24:58.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f60097fa640 1 -- 172.21.15.46:0/3297550458 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60000b9410 con 0x7f600c06f7a0 2023-10-30T13:24:58.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6012fd3640 1 -- 172.21.15.46:0/3297550458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600c0ae240 con 0x7f600c06f7a0 2023-10-30T13:24:58.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6012fd3640 1 -- 172.21.15.46:0/3297550458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600c1e1d20 con 0x7f600c06f7a0 2023-10-30T13:24:58.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f6012fd3640 1 -- 172.21.15.46:0/3297550458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f600c04e170 con 0x7f600c06f7a0 2023-10-30T13:24:58.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.871+0000 7f60097fa640 1 -- 172.21.15.46:0/3297550458 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60000b7040 con 0x7f600c06f7a0 2023-10-30T13:24:58.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.875+0000 7f60097fa640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ff00664a0 0x7f5ff0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:58.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.875+0000 7f60097fa640 1 -- 172.21.15.46:0/3297550458 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6000121250 con 0x7f600c06f7a0 2023-10-30T13:24:58.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.875+0000 7f600bfff640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ff00664a0 0x7f5ff0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:58.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.875+0000 7f600bfff640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ff00664a0 0x7f5ff0068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f600c0c8280 tx=0x7f5ff8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:58.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:58.879+0000 7f60097fa640 1 -- 172.21.15.46:0/3297550458 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60000ee510 con 0x7f600c06f7a0 2023-10-30T13:24:59.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.067+0000 7f6012fd3640 1 -- 172.21.15.46:0/3297550458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f600c0c4990 con 0x7f600c06f7a0 2023-10-30T13:24:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.067+0000 7f60097fa640 1 -- 172.21.15.46:0/3297550458 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f60000b5090 con 0x7f600c06f7a0 2023-10-30T13:24:59.073 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:24:59.073 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:35.740725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-10-30T13:24:59.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.071+0000 7f6012fd3640 1 -- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ff00664a0 msgr2=0x7f5ff0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:59.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.071+0000 7f6012fd3640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ff00664a0 0x7f5ff0068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f600c0c8280 tx=0x7f5ff8008040 comp rx=0 tx=0).stop 2023-10-30T13:24:59.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.071+0000 7f6012fd3640 1 -- 172.21.15.46:0/3297550458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f600c06f7a0 msgr2=0x7f600c0c0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:59.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.071+0000 7f6012fd3640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f600c06f7a0 0x7f600c0c0540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6000000c10 tx=0x7f600001bad0 comp rx=0 tx=0).stop 2023-10-30T13:24:59.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.071+0000 7f6012fd3640 1 -- 172.21.15.46:0/3297550458 shutdown_connections 2023-10-30T13:24:59.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.071+0000 7f6012fd3640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ff00664a0 0x7f5ff0068960 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:59.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.071+0000 7f6012fd3640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f600c0c6bd0 0x7f600c1e2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:59.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.071+0000 7f6012fd3640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f600c071110 0x7f600c1e2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:59.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.071+0000 7f6012fd3640 1 --2- 172.21.15.46:0/3297550458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f600c06f7a0 0x7f600c0c0540 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:59.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.071+0000 7f6012fd3640 1 -- 172.21.15.46:0/3297550458 >> 172.21.15.46:0/3297550458 conn(0x7f600c0c7910 msgr2=0x7f600c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:59.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.071+0000 7f6012fd3640 1 -- 172.21.15.46:0/3297550458 shutdown_connections 2023-10-30T13:24:59.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:24:59.071+0000 7f6012fd3640 1 -- 172.21.15.46:0/3297550458 wait complete. 2023-10-30T13:24:59.094 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:25:01.094 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:01.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.243+0000 7f7fd5ba3640 1 Processor -- start 2023-10-30T13:25:01.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.243+0000 7f7fd5ba3640 1 -- start start 2023-10-30T13:25:01.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fd5ba3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7fd01538f0 0x7f7fd0159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fd5ba3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd00ca6a0 con 0x7f7fd015cd50 2023-10-30T13:25:01.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fd5ba3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd015d300 con 0x7f7fd00cefa0 2023-10-30T13:25:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fd5ba3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fd015d5e0 con 0x7f7fd01538f0 2023-10-30T13:25:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7fd015cd50 0x7f7fd00c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34522/0 (socket says 172.21.15.46:34522) 2023-10-30T13:25:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcffff640 1 -- 172.21.15.46:0/3304345319 learned_addr learned my addr 172.21.15.46:0/3304345319 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:25:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fd4ba1640 1 --2- 172.21.15.46:0/3304345319 >> v2:172.21.15.154:3300/0 conn(0x7f7fd01538f0 0x7f7fd0159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186672930 0 0) 0x7f7fd00ca6a0 con 0x7f7fd015cd50 2023-10-30T13:25:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc0003530 con 0x7f7fd015cd50 2023-10-30T13:25:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4241040860 0 0) 0x7f7fc0003530 con 0x7f7fd015cd50 2023-10-30T13:25:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 >> v2:172.21.15.154:3300/0 conn(0x7f7fd01538f0 msgr2=0x7f7fd0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 --2- 172.21.15.46:0/3304345319 >> v2:172.21.15.154:3300/0 conn(0x7f7fd01538f0 0x7f7fd0159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 >> v1:172.21.15.154:6789/0 conn(0x7f7fd00cefa0 legacy=0x7f7fd01640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd015d8c0 con 0x7f7fd015cd50 2023-10-30T13:25:01.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f7fbc001680 con 0x7f7fd015cd50 2023-10-30T13:25:01.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 >> v1:172.21.15.46:6789/0 conn(0x7f7fd015cd50 legacy=0x7f7fd00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 --2- 172.21.15.46:0/3304345319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc0003980 0x7f7fc0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 --2- 172.21.15.46:0/3304345319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fc00047d0 0x7f7fc0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 --2- 172.21.15.46:0/3304345319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc0005190 0x7f7fc000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd015d5e0 con 0x7f7fc00047d0 2023-10-30T13:25:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc000c040 con 0x7f7fc0005190 2023-10-30T13:25:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc000c200 con 0x7f7fc0003980 2023-10-30T13:25:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcffff640 1 --2- 172.21.15.46:0/3304345319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fc00047d0 0x7f7fc0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fd4ba1640 1 --2- 172.21.15.46:0/3304345319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc0003980 0x7f7fc0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fd53a2640 1 --2- 172.21.15.46:0/3304345319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc0005190 0x7f7fc000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcffff640 1 -- 172.21.15.46:0/3304345319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc0003980 msgr2=0x7f7fc0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcffff640 1 --2- 172.21.15.46:0/3304345319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc0003980 0x7f7fc0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcffff640 1 -- 172.21.15.46:0/3304345319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc0005190 msgr2=0x7f7fc000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:01.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcffff640 1 --2- 172.21.15.46:0/3304345319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc0005190 0x7f7fc000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fcffff640 1 -- 172.21.15.46:0/3304345319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc0003530 con 0x7f7fc00047d0 2023-10-30T13:25:01.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fd53a2640 1 --2- 172.21.15.46:0/3304345319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc0005190 0x7f7fc000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:25:01.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.247+0000 7f7fd53a2640 1 -- 172.21.15.46:0/3304345319 reap_dead start 2023-10-30T13:25:01.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.251+0000 7f7fcffff640 1 --2- 172.21.15.46:0/3304345319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fc00047d0 0x7f7fc0004c50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00c240 tx=0x7f7fbc01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df80a98df1f822cb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:01.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.251+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc0a5b40 con 0x7f7fc00047d0 2023-10-30T13:25:01.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.251+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fbc0a5ca0 con 0x7f7fc00047d0 2023-10-30T13:25:01.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.251+0000 7f7fcf7fe640 1 -- 172.21.15.46:0/3304345319 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc005b20 con 0x7f7fc00047d0 2023-10-30T13:25:01.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.251+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/3304345319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fc00047d0 msgr2=0x7f7fc0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.251+0000 7f7fd5ba3640 1 --2- 172.21.15.46:0/3304345319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fc00047d0 0x7f7fc0004c50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00c240 tx=0x7f7fbc01cc40 comp rx=0 tx=0).stop 2023-10-30T13:25:01.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.251+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/3304345319 shutdown_connections 2023-10-30T13:25:01.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.251+0000 7f7fd5ba3640 1 --2- 172.21.15.46:0/3304345319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fc00047d0 0x7f7fc0004c50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.251+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/3304345319 >> 172.21.15.46:0/3304345319 conn(0x7f7fd006a640 msgr2=0x7f7fd0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:01.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.251+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/3304345319 shutdown_connections 2023-10-30T13:25:01.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.251+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/3304345319 wait complete. 2023-10-30T13:25:01.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd5ba3640 1 Processor -- start 2023-10-30T13:25:01.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd5ba3640 1 -- start start 2023-10-30T13:25:01.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd5ba3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc00047d0 0x7f7fd01ef850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd5ba3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fd01591f0 0x7f7fd015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd5ba3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fd015c7c0 0x7f7fd013ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd5ba3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd00b40c0 con 0x7f7fd01591f0 2023-10-30T13:25:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd5ba3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd01ef3c0 con 0x7f7fc00047d0 2023-10-30T13:25:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd5ba3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd00b3950 con 0x7f7fd015c7c0 2023-10-30T13:25:01.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fcffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fd01591f0 0x7f7fd015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fcffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fd01591f0 0x7f7fd015c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41042/0 (socket says 172.21.15.46:41042) 2023-10-30T13:25:01.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fcffff640 1 -- 172.21.15.46:0/4022264612 learned_addr learned my addr 172.21.15.46:0/4022264612 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:01.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd4ba1640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc00047d0 0x7f7fd01ef850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd53a2640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fd015c7c0 0x7f7fd013ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fcffff640 1 -- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fd015c7c0 msgr2=0x7f7fd013ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fcffff640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fd015c7c0 0x7f7fd013ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fcffff640 1 -- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc00047d0 msgr2=0x7f7fd01ef850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fcffff640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc00047d0 0x7f7fd01ef850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fcffff640 1 -- 172.21.15.46:0/4022264612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd013c550 con 0x7f7fd01591f0 2023-10-30T13:25:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd53a2640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fd015c7c0 0x7f7fd013ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fcffff640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fd01591f0 0x7f7fd015c280 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00c240 tx=0x7f7fc0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fcd7fa640 1 -- 172.21.15.46:0/4022264612 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc0b3070 con 0x7f7fd01591f0 2023-10-30T13:25:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fcd7fa640 1 -- 172.21.15.46:0/4022264612 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fbc000ee0 con 0x7f7fd01591f0 2023-10-30T13:25:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fcd7fa640 1 -- 172.21.15.46:0/4022264612 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc001680 con 0x7f7fd01591f0 2023-10-30T13:25:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/4022264612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd00b3bc0 con 0x7f7fd01591f0 2023-10-30T13:25:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/4022264612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd01ef050 con 0x7f7fd01591f0 2023-10-30T13:25:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.255+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/4022264612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd004e170 con 0x7f7fd01591f0 2023-10-30T13:25:01.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.259+0000 7f7fcd7fa640 1 -- 172.21.15.46:0/4022264612 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7fbc007140 con 0x7f7fd01591f0 2023-10-30T13:25:01.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.259+0000 7f7fcd7fa640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7fb40664a0 0x7f7fb4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.259+0000 7f7fcd7fa640 1 -- 172.21.15.46:0/4022264612 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7fbc11d7b0 con 0x7f7fd01591f0 2023-10-30T13:25:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.259+0000 7f7fd4ba1640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7fb40664a0 0x7f7fb4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.259+0000 7f7fd4ba1640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7fb40664a0 0x7f7fb4068960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4009b60 tx=0x7f7fc4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:01.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.263+0000 7f7fcd7fa640 1 -- 172.21.15.46:0/4022264612 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7fbc0eab20 con 0x7f7fd01591f0 2023-10-30T13:25:01.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.451+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/4022264612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fd01ef500 con 0x7f7fd01591f0 2023-10-30T13:25:01.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.451+0000 7f7fcd7fa640 1 -- 172.21.15.46:0/4022264612 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f7fbc0ee140 con 0x7f7fd01591f0 2023-10-30T13:25:01.457 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:01.457 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:35.740725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:01.457 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-10-30T13:25:01.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.455+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7fb40664a0 msgr2=0x7f7fb4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.455+0000 7f7fd5ba3640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7fb40664a0 0x7f7fb4068960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4009b60 tx=0x7f7fc4008040 comp rx=0 tx=0).stop 2023-10-30T13:25:01.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.455+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/4022264612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fd01591f0 msgr2=0x7f7fd015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.455+0000 7f7fd5ba3640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fd01591f0 0x7f7fd015c280 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00c240 tx=0x7f7fc0005800 comp rx=0 tx=0).stop 2023-10-30T13:25:01.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.455+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/4022264612 shutdown_connections 2023-10-30T13:25:01.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.455+0000 7f7fd5ba3640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7fb40664a0 0x7f7fb4068960 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.455+0000 7f7fd5ba3640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fd015c7c0 0x7f7fd013ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.455+0000 7f7fd5ba3640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fd01591f0 0x7f7fd015c280 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.455+0000 7f7fd5ba3640 1 --2- 172.21.15.46:0/4022264612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc00047d0 0x7f7fd01ef850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.455+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/4022264612 >> 172.21.15.46:0/4022264612 conn(0x7f7fd006a640 msgr2=0x7f7fd00c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:01.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.455+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/4022264612 shutdown_connections 2023-10-30T13:25:01.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:01.455+0000 7f7fd5ba3640 1 -- 172.21.15.46:0/4022264612 wait complete. 2023-10-30T13:25:01.478 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:25:03.479 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:03.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.631+0000 7f5a62eea640 1 Processor -- start 2023-10-30T13:25:03.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.631+0000 7f5a62eea640 1 -- start start 2023-10-30T13:25:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.631+0000 7f5a62eea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a5c15d7f0 0x7f5a5c15a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.631+0000 7f5a62eea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c15a640 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.631+0000 7f5a62eea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c0c86a0 con 0x7f5a5c06fa00 2023-10-30T13:25:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.631+0000 7f5a62eea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a5c15cc50 con 0x7f5a5c15d7f0 2023-10-30T13:25:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.635+0000 7f5a61487640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a5c15d7f0 0x7f5a5c15a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.635+0000 7f5a60c86640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5a5c0c3c20 0x7f5a5c0ccd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34526/0 (socket says 172.21.15.46:34526) 2023-10-30T13:25:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.635+0000 7f5a60c86640 1 -- 172.21.15.46:0/2043428442 learned_addr learned my addr 172.21.15.46:0/2043428442 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:25:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.635+0000 7f5a61487640 1 -- 172.21.15.46:0/2043428442 >> v1:172.21.15.154:6789/0 conn(0x7f5a5c06fa00 legacy=0x7f5a5c15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.635+0000 7f5a61487640 1 -- 172.21.15.46:0/2043428442 >> v1:172.21.15.46:6789/0 conn(0x7f5a5c0c3c20 legacy=0x7f5a5c0ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.635+0000 7f5a61487640 1 -- 172.21.15.46:0/2043428442 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c0c88f0 con 0x7f5a5c15d7f0 2023-10-30T13:25:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.635+0000 7f5a5b7fe640 1 -- 172.21.15.46:0/2043428442 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944026376 0 0) 0x7f5a5c15a640 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.635+0000 7f5a61487640 1 --2- 172.21.15.46:0/2043428442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a5c15d7f0 0x7f5a5c15a100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5a54009d90 tx=0x7f5a5401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbec2264a7e34d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.635+0000 7f5a5b7fe640 1 -- 172.21.15.46:0/2043428442 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a540af600 con 0x7f5a5c15d7f0 2023-10-30T13:25:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.635+0000 7f5a5b7fe640 1 -- 172.21.15.46:0/2043428442 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a540ab050 con 0x7f5a5c15d7f0 2023-10-30T13:25:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.635+0000 7f5a5b7fe640 1 -- 172.21.15.46:0/2043428442 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a540af980 con 0x7f5a5c15d7f0 2023-10-30T13:25:03.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 -- 172.21.15.46:0/2043428442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a5c15d7f0 msgr2=0x7f5a5c15a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:03.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 --2- 172.21.15.46:0/2043428442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a5c15d7f0 0x7f5a5c15a100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5a54009d90 tx=0x7f5a5401f130 comp rx=0 tx=0).stop 2023-10-30T13:25:03.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 -- 172.21.15.46:0/2043428442 shutdown_connections 2023-10-30T13:25:03.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 --2- 172.21.15.46:0/2043428442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a5c15d7f0 0x7f5a5c15a100 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:03.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 -- 172.21.15.46:0/2043428442 >> 172.21.15.46:0/2043428442 conn(0x7f5a5c071790 msgr2=0x7f5a5c071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:03.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 -- 172.21.15.46:0/2043428442 shutdown_connections 2023-10-30T13:25:03.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 -- 172.21.15.46:0/2043428442 wait complete. 2023-10-30T13:25:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 Processor -- start 2023-10-30T13:25:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 -- start start 2023-10-30T13:25:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a5c06fa00 0x7f5a5c149200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a5c0c3c20 0x7f5a5c0cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a5c15d7f0 0x7f5a5c1521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c0cb5e0 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c149740 con 0x7f5a5c15d7f0 2023-10-30T13:25:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a62eea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c1f0570 con 0x7f5a5c06fa00 2023-10-30T13:25:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a5c0c3c20 0x7f5a5c0cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a5c0c3c20 0x7f5a5c0cc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41070/0 (socket says 172.21.15.46:41070) 2023-10-30T13:25:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a5bfff640 1 -- 172.21.15.46:0/3744892688 learned_addr learned my addr 172.21.15.46:0/3744892688 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:03.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.639+0000 7f5a60c86640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a5c06fa00 0x7f5a5c149200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:03.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a61487640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a5c15d7f0 0x7f5a5c1521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:03.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a5bfff640 1 -- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a5c06fa00 msgr2=0x7f5a5c149200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:03.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a5bfff640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a5c06fa00 0x7f5a5c149200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:03.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a5bfff640 1 -- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a5c15d7f0 msgr2=0x7f5a5c1521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:03.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a5bfff640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a5c15d7f0 0x7f5a5c1521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:03.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a5bfff640 1 -- 172.21.15.46:0/3744892688 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a54009a70 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a5bfff640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a5c0c3c20 0x7f5a5c0cc340 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800c3c0 tx=0x7f5a4800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:03.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a597fa640 1 -- 172.21.15.46:0/3744892688 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a480052d0 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a597fa640 1 -- 172.21.15.46:0/3744892688 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a48005430 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a597fa640 1 -- 172.21.15.46:0/3744892688 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a48003840 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a62eea640 1 -- 172.21.15.46:0/3744892688 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5c14dac0 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a62eea640 1 -- 172.21.15.46:0/3744892688 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5c0cb7f0 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a62eea640 1 -- 172.21.15.46:0/3744892688 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a5c1f06e0 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a597fa640 1 -- 172.21.15.46:0/3744892688 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5a480039a0 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a597fa640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5a400664a0 0x7f5a40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.643+0000 7f5a597fa640 1 -- 172.21.15.46:0/3744892688 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f5a48083450 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.647+0000 7f5a60c86640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5a400664a0 0x7f5a40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:03.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.647+0000 7f5a60c86640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5a400664a0 0x7f5a40068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5a50000f70 tx=0x7f5a50007240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:03.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.647+0000 7f5a597fa640 1 -- 172.21.15.46:0/3744892688 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5a48050850 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.843+0000 7f5a62eea640 1 -- 172.21.15.46:0/3744892688 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a5c04e170 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.843+0000 7f5a597fa640 1 -- 172.21.15.46:0/3744892688 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f5a48053e70 con 0x7f5a5c0c3c20 2023-10-30T13:25:03.848 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:03.848 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:35.740725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-10-30T13:25:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.843+0000 7f5a62eea640 1 -- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5a400664a0 msgr2=0x7f5a40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.843+0000 7f5a62eea640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5a400664a0 0x7f5a40068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5a50000f70 tx=0x7f5a50007240 comp rx=0 tx=0).stop 2023-10-30T13:25:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.843+0000 7f5a62eea640 1 -- 172.21.15.46:0/3744892688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a5c0c3c20 msgr2=0x7f5a5c0cc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.843+0000 7f5a62eea640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a5c0c3c20 0x7f5a5c0cc340 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800c3c0 tx=0x7f5a4800c880 comp rx=0 tx=0).stop 2023-10-30T13:25:03.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.847+0000 7f5a62eea640 1 -- 172.21.15.46:0/3744892688 shutdown_connections 2023-10-30T13:25:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.847+0000 7f5a62eea640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5a400664a0 0x7f5a40068960 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.847+0000 7f5a62eea640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a5c15d7f0 0x7f5a5c1521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.847+0000 7f5a62eea640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a5c0c3c20 0x7f5a5c0cc340 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.847+0000 7f5a62eea640 1 --2- 172.21.15.46:0/3744892688 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a5c06fa00 0x7f5a5c149200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.847+0000 7f5a62eea640 1 -- 172.21.15.46:0/3744892688 >> 172.21.15.46:0/3744892688 conn(0x7f5a5c071790 msgr2=0x7f5a5c073720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.847+0000 7f5a62eea640 1 -- 172.21.15.46:0/3744892688 shutdown_connections 2023-10-30T13:25:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:03.847+0000 7f5a62eea640 1 -- 172.21.15.46:0/3744892688 wait complete. 2023-10-30T13:25:03.869 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:25:05.870 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:06.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb112ba7640 1 Processor -- start 2023-10-30T13:25:06.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb112ba7640 1 -- start start 2023-10-30T13:25:06.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb112ba7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb10c15cd50 0x7fb10c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:06.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb112ba7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb10c0cefa0 0x7fb10c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:06.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb112ba7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10c15a6c0 con 0x7fb10c1538f0 2023-10-30T13:25:06.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb112ba7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10c15d300 con 0x7fb10c15cd50 2023-10-30T13:25:06.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb112ba7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c15d5e0 con 0x7fb10c0cefa0 2023-10-30T13:25:06.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb1113a4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb10c15cd50 0x7fb10c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:06.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb1123a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb10c0cefa0 0x7fb10c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:06.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb1113a4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb10c15cd50 0x7fb10c0c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:48638/0 (socket says 172.21.15.46:48638) 2023-10-30T13:25:06.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb1113a4640 1 -- 172.21.15.46:0/1457355305 learned_addr learned my addr 172.21.15.46:0/1457355305 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:06.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb1123a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb10c0cefa0 0x7fb10c0cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:58596/0 (socket says 172.21.15.46:58596) 2023-10-30T13:25:06.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb111ba5640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fb10c1538f0 0x7fb10c159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:35918/0 (socket says 172.21.15.46:35918) 2023-10-30T13:25:06.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb1113a4640 1 -- 172.21.15.46:0/1457355305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb10c0cefa0 msgr2=0x7fb10c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:06.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb1113a4640 1 --2- 172.21.15.46:0/1457355305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb10c0cefa0 0x7fb10c0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:06.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb1113a4640 1 -- 172.21.15.46:0/1457355305 >> v1:172.21.15.154:6789/0 conn(0x7fb10c1538f0 legacy=0x7fb10c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:06.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.015+0000 7fb1113a4640 1 -- 172.21.15.46:0/1457355305 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10c15d8c0 con 0x7fb10c15cd50 2023-10-30T13:25:06.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.019+0000 7fb1113a4640 1 --2- 172.21.15.46:0/1457355305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c15cd50 0x7fb10c0c9f90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009870 tx=0x7fb0fc01ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb0b158ddd3f45cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:06.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.019+0000 7fb110ba3640 1 -- 172.21.15.46:0/1457355305 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0a8a30 con 0x7fb10c15cd50 2023-10-30T13:25:06.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.019+0000 7fb110ba3640 1 -- 172.21.15.46:0/1457355305 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc0a8b90 con 0x7fb10c15cd50 2023-10-30T13:25:06.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.019+0000 7fb110ba3640 1 -- 172.21.15.46:0/1457355305 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0b0440 con 0x7fb10c15cd50 2023-10-30T13:25:06.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.019+0000 7fb112ba7640 1 -- 172.21.15.46:0/1457355305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c15cd50 msgr2=0x7fb10c0c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:06.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.019+0000 7fb112ba7640 1 --2- 172.21.15.46:0/1457355305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c15cd50 0x7fb10c0c9f90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009870 tx=0x7fb0fc01ed30 comp rx=0 tx=0).stop 2023-10-30T13:25:06.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.019+0000 7fb112ba7640 1 -- 172.21.15.46:0/1457355305 shutdown_connections 2023-10-30T13:25:06.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.019+0000 7fb112ba7640 1 --2- 172.21.15.46:0/1457355305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb10c0cefa0 0x7fb10c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:06.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.019+0000 7fb112ba7640 1 --2- 172.21.15.46:0/1457355305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c15cd50 0x7fb10c0c9f90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:06.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.019+0000 7fb112ba7640 1 -- 172.21.15.46:0/1457355305 >> 172.21.15.46:0/1457355305 conn(0x7fb10c06a640 msgr2=0x7fb10c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:06.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.019+0000 7fb112ba7640 1 -- 172.21.15.46:0/1457355305 shutdown_connections 2023-10-30T13:25:06.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.019+0000 7fb112ba7640 1 -- 172.21.15.46:0/1457355305 wait complete. 2023-10-30T13:25:06.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb112ba7640 1 Processor -- start 2023-10-30T13:25:06.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb112ba7640 1 -- start start 2023-10-30T13:25:06.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb112ba7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c0cefa0 0x7fb10c158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:06.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb112ba7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb10c1538f0 0x7fb10c0b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:06.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb112ba7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb10c15cd50 0x7fb10c144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:06.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb112ba7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0b4300 con 0x7fb10c15cd50 2023-10-30T13:25:06.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb112ba7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0b2ff0 con 0x7fb10c0cefa0 2023-10-30T13:25:06.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb112ba7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c150f90 con 0x7fb10c1538f0 2023-10-30T13:25:06.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1123a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb10c15cd50 0x7fb10c144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:06.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1123a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb10c15cd50 0x7fb10c144700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41072/0 (socket says 172.21.15.46:41072) 2023-10-30T13:25:06.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1123a6640 1 -- 172.21.15.46:0/3666366223 learned_addr learned my addr 172.21.15.46:0/3666366223 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:06.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb111ba5640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c0cefa0 0x7fb10c158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:06.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1113a4640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb10c1538f0 0x7fb10c0b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1123a6640 1 -- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb10c1538f0 msgr2=0x7fb10c0b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1123a6640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb10c1538f0 0x7fb10c0b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1123a6640 1 -- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c0cefa0 msgr2=0x7fb10c158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:06.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1123a6640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c0cefa0 0x7fb10c158f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1123a6640 1 -- 172.21.15.46:0/3666366223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc0094d0 con 0x7fb10c15cd50 2023-10-30T13:25:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1113a4640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb10c1538f0 0x7fb10c0b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb111ba5640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c0cefa0 0x7fb10c158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1123a6640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb10c15cd50 0x7fb10c144700 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb10400efc0 tx=0x7fb10400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1027fc640 1 -- 172.21.15.46:0/3666366223 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104014480 con 0x7fb10c15cd50 2023-10-30T13:25:06.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1027fc640 1 -- 172.21.15.46:0/3666366223 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10400f040 con 0x7fb10c15cd50 2023-10-30T13:25:06.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb112ba7640 1 -- 172.21.15.46:0/3666366223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb10c151270 con 0x7fb10c15cd50 2023-10-30T13:25:06.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb1027fc640 1 -- 172.21.15.46:0/3666366223 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104014820 con 0x7fb10c15cd50 2023-10-30T13:25:06.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.023+0000 7fb112ba7640 1 -- 172.21.15.46:0/3666366223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10c151550 con 0x7fb10c15cd50 2023-10-30T13:25:06.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.027+0000 7fb112ba7640 1 -- 172.21.15.46:0/3666366223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10c0b34d0 con 0x7fb10c15cd50 2023-10-30T13:25:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.027+0000 7fb1027fc640 1 -- 172.21.15.46:0/3666366223 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb104014980 con 0x7fb10c15cd50 2023-10-30T13:25:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.027+0000 7fb1027fc640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0f00664d0 0x7fb0f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.027+0000 7fb1027fc640 1 -- 172.21.15.46:0/3666366223 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb104084b80 con 0x7fb10c15cd50 2023-10-30T13:25:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.027+0000 7fb111ba5640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0f00664d0 0x7fb0f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.027+0000 7fb111ba5640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0f00664d0 0x7fb0f0068990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb108000f70 tx=0x7fb108007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.031+0000 7fb1027fc640 1 -- 172.21.15.46:0/3666366223 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb10c0b34d0 con 0x7fb10c15cd50 2023-10-30T13:25:06.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.223+0000 7fb112ba7640 1 -- 172.21.15.46:0/3666366223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb10c0b34d0 con 0x7fb10c15cd50 2023-10-30T13:25:06.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.223+0000 7fb1027fc640 1 -- 172.21.15.46:0/3666366223 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7fb10c0b34d0 con 0x7fb10c15cd50 2023-10-30T13:25:06.228 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:06.229 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:35.740725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:06.229 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-10-30T13:25:06.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.227+0000 7fb112ba7640 1 -- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0f00664d0 msgr2=0x7fb0f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:06.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.227+0000 7fb112ba7640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0f00664d0 0x7fb0f0068990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb108000f70 tx=0x7fb108007d50 comp rx=0 tx=0).stop 2023-10-30T13:25:06.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.227+0000 7fb112ba7640 1 -- 172.21.15.46:0/3666366223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb10c15cd50 msgr2=0x7fb10c144700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:06.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.227+0000 7fb112ba7640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb10c15cd50 0x7fb10c144700 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb10400efc0 tx=0x7fb10400c490 comp rx=0 tx=0).stop 2023-10-30T13:25:06.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.227+0000 7fb112ba7640 1 -- 172.21.15.46:0/3666366223 shutdown_connections 2023-10-30T13:25:06.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.227+0000 7fb112ba7640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0f00664d0 0x7fb0f0068990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:06.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.227+0000 7fb112ba7640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb10c15cd50 0x7fb10c144700 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:06.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.227+0000 7fb112ba7640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb10c1538f0 0x7fb10c0b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:06.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.227+0000 7fb112ba7640 1 --2- 172.21.15.46:0/3666366223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c0cefa0 0x7fb10c158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:06.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.227+0000 7fb112ba7640 1 -- 172.21.15.46:0/3666366223 >> 172.21.15.46:0/3666366223 conn(0x7fb10c06a640 msgr2=0x7fb10c0cf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:06.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.227+0000 7fb112ba7640 1 -- 172.21.15.46:0/3666366223 shutdown_connections 2023-10-30T13:25:06.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:06.227+0000 7fb112ba7640 1 -- 172.21.15.46:0/3666366223 wait complete. 2023-10-30T13:25:06.250 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:25:08.251 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:08.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.399+0000 7f2e1cb1d640 1 Processor -- start 2023-10-30T13:25:08.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.399+0000 7f2e1cb1d640 1 -- start start 2023-10-30T13:25:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.403+0000 7f2e1cb1d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2e180cf160 0x7f2e180cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.403+0000 7f2e1cb1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e180cb800 con 0x7f2e1815d6f0 2023-10-30T13:25:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.403+0000 7f2e1cb1d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e180cb330 con 0x7f2e181538a0 2023-10-30T13:25:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.403+0000 7f2e1cb1d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e180cad70 con 0x7f2e180cf160 2023-10-30T13:25:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.403+0000 7f2e16d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2e180cf160 0x7f2e180cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.403+0000 7f2e16d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2e180cf160 0x7f2e180cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58840/0 (socket says 172.21.15.46:58840) 2023-10-30T13:25:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.403+0000 7f2e16d9d640 1 -- 172.21.15.46:0/1855382185 learned_addr learned my addr 172.21.15.46:0/1855382185 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.403+0000 7f2e16d9d640 1 -- 172.21.15.46:0/1855382185 >> v1:172.21.15.154:6789/0 conn(0x7f2e181538a0 legacy=0x7f2e180ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:08.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.403+0000 7f2e16d9d640 1 -- 172.21.15.46:0/1855382185 >> v1:172.21.15.46:6789/0 conn(0x7f2e1815d6f0 legacy=0x7f2e1815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:08.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.403+0000 7f2e16d9d640 1 -- 172.21.15.46:0/1855382185 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e180cb050 con 0x7f2e180cf160 2023-10-30T13:25:08.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.407+0000 7f2e16d9d640 1 --2- 172.21.15.46:0/1855382185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e180cf160 0x7f2e180cf590 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009e50 tx=0x7f2e1001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467f20ee6e9adf5a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:08.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.407+0000 7f2e1559a640 1 -- 172.21.15.46:0/1855382185 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e100af400 con 0x7f2e180cf160 2023-10-30T13:25:08.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.407+0000 7f2e1559a640 1 -- 172.21.15.46:0/1855382185 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e100ab050 con 0x7f2e180cf160 2023-10-30T13:25:08.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.407+0000 7f2e1559a640 1 -- 172.21.15.46:0/1855382185 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e100af780 con 0x7f2e180cf160 2023-10-30T13:25:08.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.407+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/1855382185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e180cf160 msgr2=0x7f2e180cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:08.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.407+0000 7f2e1cb1d640 1 --2- 172.21.15.46:0/1855382185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e180cf160 0x7f2e180cf590 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009e50 tx=0x7f2e1001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:25:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.407+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/1855382185 shutdown_connections 2023-10-30T13:25:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.407+0000 7f2e1cb1d640 1 --2- 172.21.15.46:0/1855382185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e180cf160 0x7f2e180cf590 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.407+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/1855382185 >> 172.21.15.46:0/1855382185 conn(0x7f2e1806bfe0 msgr2=0x7f2e1806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.407+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/1855382185 shutdown_connections 2023-10-30T13:25:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.407+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/1855382185 wait complete. 2023-10-30T13:25:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.407+0000 7f2e1cb1d640 1 Processor -- start 2023-10-30T13:25:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1cb1d640 1 -- start start 2023-10-30T13:25:08.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1cb1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e180cf160 0x7f2e18154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:08.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1cb1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e181538a0 0x7f2e180b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:08.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1cb1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e1815d6f0 0x7f2e181e2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:08.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1cb1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e181559b0 con 0x7f2e180cf160 2023-10-30T13:25:08.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1cb1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18159410 con 0x7f2e1815d6f0 2023-10-30T13:25:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1cb1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e181e3de0 con 0x7f2e181538a0 2023-10-30T13:25:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e180cf160 0x7f2e18154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e180cf160 0x7f2e18154dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58854/0 (socket says 172.21.15.46:58854) 2023-10-30T13:25:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1659c640 1 -- 172.21.15.46:0/291620822 learned_addr learned my addr 172.21.15.46:0/291620822 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e15d9b640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e181538a0 0x7f2e180b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:08.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e16d9d640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e1815d6f0 0x7f2e181e2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:08.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1659c640 1 -- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e181538a0 msgr2=0x7f2e180b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:08.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1659c640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e181538a0 0x7f2e180b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:08.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1659c640 1 -- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e1815d6f0 msgr2=0x7f2e181e2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:08.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1659c640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e1815d6f0 0x7f2e181e2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:08.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1659c640 1 -- 172.21.15.46:0/291620822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e10009b30 con 0x7f2e180cf160 2023-10-30T13:25:08.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e15d9b640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e181538a0 0x7f2e180b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:08.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1659c640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e180cf160 0x7f2e18154dc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800c530 tx=0x7f2e0800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:08.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e06ffd640 1 -- 172.21.15.46:0/291620822 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08005220 con 0x7f2e180cf160 2023-10-30T13:25:08.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e06ffd640 1 -- 172.21.15.46:0/291620822 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e08005380 con 0x7f2e180cf160 2023-10-30T13:25:08.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/291620822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e18156ce0 con 0x7f2e180cf160 2023-10-30T13:25:08.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e06ffd640 1 -- 172.21.15.46:0/291620822 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08007270 con 0x7f2e180cf160 2023-10-30T13:25:08.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/291620822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e18155300 con 0x7f2e180cf160 2023-10-30T13:25:08.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.411+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/291620822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1804e170 con 0x7f2e180cf160 2023-10-30T13:25:08.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.415+0000 7f2e06ffd640 1 -- 172.21.15.46:0/291620822 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2e080073d0 con 0x7f2e180cf160 2023-10-30T13:25:08.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.415+0000 7f2e06ffd640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2df80664a0 0x7f2df8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:08.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.415+0000 7f2e06ffd640 1 -- 172.21.15.46:0/291620822 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f2e080837a0 con 0x7f2e180cf160 2023-10-30T13:25:08.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.415+0000 7f2e15d9b640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2df80664a0 0x7f2df8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:08.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.415+0000 7f2e15d9b640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2df80664a0 0x7f2df8068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2e180c47a0 tx=0x7f2e0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:08.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.419+0000 7f2e06ffd640 1 -- 172.21.15.46:0/291620822 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2e08050b80 con 0x7f2e180cf160 2023-10-30T13:25:08.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.607+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/291620822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e180b1dd0 con 0x7f2e180cf160 2023-10-30T13:25:08.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.607+0000 7f2e06ffd640 1 -- 172.21.15.46:0/291620822 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f2e080541a0 con 0x7f2e180cf160 2023-10-30T13:25:08.612 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:35.740725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:08.613 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-10-30T13:25:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.611+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2df80664a0 msgr2=0x7f2df8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.611+0000 7f2e1cb1d640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2df80664a0 0x7f2df8068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2e180c47a0 tx=0x7f2e0c009040 comp rx=0 tx=0).stop 2023-10-30T13:25:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.611+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/291620822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e180cf160 msgr2=0x7f2e18154dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.611+0000 7f2e1cb1d640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e180cf160 0x7f2e18154dc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800c530 tx=0x7f2e0800c9f0 comp rx=0 tx=0).stop 2023-10-30T13:25:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.611+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/291620822 shutdown_connections 2023-10-30T13:25:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.611+0000 7f2e1cb1d640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2df80664a0 0x7f2df8068960 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.611+0000 7f2e1cb1d640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e1815d6f0 0x7f2e181e2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.611+0000 7f2e1cb1d640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e181538a0 0x7f2e180b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.611+0000 7f2e1cb1d640 1 --2- 172.21.15.46:0/291620822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e180cf160 0x7f2e18154dc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.611+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/291620822 >> 172.21.15.46:0/291620822 conn(0x7f2e1806bfe0 msgr2=0x7f2e180735c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.611+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/291620822 shutdown_connections 2023-10-30T13:25:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:08.611+0000 7f2e1cb1d640 1 -- 172.21.15.46:0/291620822 wait complete. 2023-10-30T13:25:08.633 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:25:10.634 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:10.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.783+0000 7f0f18580640 1 Processor -- start 2023-10-30T13:25:10.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.783+0000 7f0f18580640 1 -- start start 2023-10-30T13:25:10.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f18580640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f10153d70 0x7f0f1006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:10.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f18580640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f1006e470 0x7f0f100cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:10.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f18580640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1006afa0 con 0x7f0f100c8490 2023-10-30T13:25:10.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f18580640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1006d780 con 0x7f0f1006e470 2023-10-30T13:25:10.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f18580640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f10154150 con 0x7f0f10153d70 2023-10-30T13:25:10.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f15b1b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f1006e470 0x7f0f100cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:10.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f16b1d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0f100c8490 0x7f0f100c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50056/0 (socket says 172.21.15.46:50056) 2023-10-30T13:25:10.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f16b1d640 1 -- 172.21.15.46:0/3739335757 learned_addr learned my addr 172.21.15.46:0/3739335757 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:25:10.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f1631c640 1 --2- 172.21.15.46:0/3739335757 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f10153d70 0x7f0f1006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:10.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f15b1b640 1 -- 172.21.15.46:0/3739335757 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f10153d70 msgr2=0x7f0f1006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:10.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f15b1b640 1 --2- 172.21.15.46:0/3739335757 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f10153d70 0x7f0f1006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:10.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f15b1b640 1 -- 172.21.15.46:0/3739335757 >> v1:172.21.15.46:6789/0 conn(0x7f0f100c8490 legacy=0x7f0f100c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:10.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f15b1b640 1 -- 172.21.15.46:0/3739335757 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1006e030 con 0x7f0f1006e470 2023-10-30T13:25:10.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.787+0000 7f0f1531a640 1 -- 172.21.15.46:0/3739335757 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326329882 0 0) 0x7f0f1006afa0 con 0x7f0f100c8490 2023-10-30T13:25:10.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f15b1b640 1 --2- 172.21.15.46:0/3739335757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1006e470 0x7f0f100cce00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0f000097c0 tx=0x7f0f0001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b9ac3daad196fd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:10.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f1531a640 1 -- 172.21.15.46:0/3739335757 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000ab650 con 0x7f0f1006e470 2023-10-30T13:25:10.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f1531a640 1 -- 172.21.15.46:0/3739335757 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f000ab7b0 con 0x7f0f1006e470 2023-10-30T13:25:10.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f1531a640 1 -- 172.21.15.46:0/3739335757 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000abb40 con 0x7f0f1006e470 2023-10-30T13:25:10.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 -- 172.21.15.46:0/3739335757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1006e470 msgr2=0x7f0f100cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:10.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 --2- 172.21.15.46:0/3739335757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1006e470 0x7f0f100cce00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0f000097c0 tx=0x7f0f0001c450 comp rx=0 tx=0).stop 2023-10-30T13:25:10.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 -- 172.21.15.46:0/3739335757 shutdown_connections 2023-10-30T13:25:10.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 --2- 172.21.15.46:0/3739335757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1006e470 0x7f0f100cce00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:10.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 --2- 172.21.15.46:0/3739335757 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f10153d70 0x7f0f1006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:10.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 -- 172.21.15.46:0/3739335757 >> 172.21.15.46:0/3739335757 conn(0x7f0f10071ea0 msgr2=0x7f0f10065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:10.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 -- 172.21.15.46:0/3739335757 shutdown_connections 2023-10-30T13:25:10.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 -- 172.21.15.46:0/3739335757 wait complete. 2023-10-30T13:25:10.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 Processor -- start 2023-10-30T13:25:10.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 -- start start 2023-10-30T13:25:10.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1006e470 0x7f0f10156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:10.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f100c8490 0x7f0f101f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:10.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f10153d70 0x7f0f101f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:10.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f100c3060 con 0x7f0f1006e470 2023-10-30T13:25:10.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f10147bd0 con 0x7f0f10153d70 2023-10-30T13:25:10.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.791+0000 7f0f18580640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f100b15c0 con 0x7f0f100c8490 2023-10-30T13:25:10.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f1631c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1006e470 0x7f0f10156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:10.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f1631c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1006e470 0x7f0f10156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58880/0 (socket says 172.21.15.46:58880) 2023-10-30T13:25:10.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f1631c640 1 -- 172.21.15.46:0/2569630645 learned_addr learned my addr 172.21.15.46:0/2569630645 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:10.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f15b1b640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f100c8490 0x7f0f101f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:10.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f16b1d640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f10153d70 0x7f0f101f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f1631c640 1 -- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f100c8490 msgr2=0x7f0f101f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f1631c640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f100c8490 0x7f0f101f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f1631c640 1 -- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f10153d70 msgr2=0x7f0f101f5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:10.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f1631c640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f10153d70 0x7f0f101f5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f1631c640 1 -- 172.21.15.46:0/2569630645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f00009420 con 0x7f0f1006e470 2023-10-30T13:25:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f15b1b640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f100c8490 0x7f0f101f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:25:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f16b1d640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f10153d70 0x7f0f101f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:25:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f1631c640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1006e470 0x7f0f10156500 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00e960 tx=0x7f0f0c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f06ffd640 1 -- 172.21.15.46:0/2569630645 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c009690 con 0x7f0f1006e470 2023-10-30T13:25:10.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f06ffd640 1 -- 172.21.15.46:0/2569630645 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c00ca00 con 0x7f0f1006e470 2023-10-30T13:25:10.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f06ffd640 1 -- 172.21.15.46:0/2569630645 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c00cd70 con 0x7f0f1006e470 2023-10-30T13:25:10.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f18580640 1 -- 172.21.15.46:0/2569630645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f100c7320 con 0x7f0f1006e470 2023-10-30T13:25:10.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f18580640 1 -- 172.21.15.46:0/2569630645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f100c6d60 con 0x7f0f1006e470 2023-10-30T13:25:10.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f18580640 1 -- 172.21.15.46:0/2569630645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1004e170 con 0x7f0f1006e470 2023-10-30T13:25:10.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f06ffd640 1 -- 172.21.15.46:0/2569630645 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0f0c013470 con 0x7f0f1006e470 2023-10-30T13:25:10.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.795+0000 7f0f06ffd640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ef40664a0 0x7f0ef4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:10.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.799+0000 7f0f06ffd640 1 -- 172.21.15.46:0/2569630645 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0f0c084530 con 0x7f0f1006e470 2023-10-30T13:25:10.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.799+0000 7f0f15b1b640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ef40664a0 0x7f0ef4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:10.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.799+0000 7f0f15b1b640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ef40664a0 0x7f0ef4068960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0f000af000 tx=0x7f0f00008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:10.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.799+0000 7f0f06ffd640 1 -- 172.21.15.46:0/2569630645 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0f0c051910 con 0x7f0f1006e470 2023-10-30T13:25:10.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.995+0000 7f0f18580640 1 -- 172.21.15.46:0/2569630645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f1006dd20 con 0x7f0f1006e470 2023-10-30T13:25:10.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.995+0000 7f0f06ffd640 1 -- 172.21.15.46:0/2569630645 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7f0f0c054f30 con 0x7f0f1006e470 2023-10-30T13:25:10.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:11.000 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:35.740725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:11.000 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-10-30T13:25:11.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.995+0000 7f0f18580640 1 -- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ef40664a0 msgr2=0x7f0ef4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:11.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.995+0000 7f0f18580640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ef40664a0 0x7f0ef4068960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0f000af000 tx=0x7f0f00008040 comp rx=0 tx=0).stop 2023-10-30T13:25:11.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.995+0000 7f0f18580640 1 -- 172.21.15.46:0/2569630645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1006e470 msgr2=0x7f0f10156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:11.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.995+0000 7f0f18580640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1006e470 0x7f0f10156500 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00e960 tx=0x7f0f0c00ee20 comp rx=0 tx=0).stop 2023-10-30T13:25:11.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.995+0000 7f0f18580640 1 -- 172.21.15.46:0/2569630645 shutdown_connections 2023-10-30T13:25:11.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.995+0000 7f0f18580640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ef40664a0 0x7f0ef4068960 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:11.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.995+0000 7f0f18580640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f10153d70 0x7f0f101f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:11.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.995+0000 7f0f18580640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f100c8490 0x7f0f101f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:11.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.995+0000 7f0f18580640 1 --2- 172.21.15.46:0/2569630645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f1006e470 0x7f0f10156500 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:11.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.995+0000 7f0f18580640 1 -- 172.21.15.46:0/2569630645 >> 172.21.15.46:0/2569630645 conn(0x7f0f10071ea0 msgr2=0x7f0f10150ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:11.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.999+0000 7f0f18580640 1 -- 172.21.15.46:0/2569630645 shutdown_connections 2023-10-30T13:25:11.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:10.999+0000 7f0f18580640 1 -- 172.21.15.46:0/2569630645 wait complete. 2023-10-30T13:25:11.021 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:25:13.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:13.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.171+0000 7fd2c1809640 1 Processor -- start 2023-10-30T13:25:13.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2c1809640 1 -- start start 2023-10-30T13:25:13.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2c1809640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd2bc06e470 0x7fd2bc06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:13.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2c1809640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2bc0c8490 0x7fd2bc0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:13.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2c1809640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc06afa0 con 0x7fd2bc153d70 2023-10-30T13:25:13.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2c1809640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2bc06d780 con 0x7fd2bc06e470 2023-10-30T13:25:13.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2c1809640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc154150 con 0x7fd2bc0c8490 2023-10-30T13:25:13.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2baffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd2bc153d70 0x7fd2bc0cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50068/0 (socket says 172.21.15.46:50068) 2023-10-30T13:25:13.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2baffd640 1 -- 172.21.15.46:0/1479258316 learned_addr learned my addr 172.21.15.46:0/1479258316 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:25:13.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2b3fff640 1 --2- 172.21.15.46:0/1479258316 >> v2:172.21.15.154:3300/0 conn(0x7fd2bc06e470 0x7fd2bc06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:13.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2bb7fe640 1 --2- 172.21.15.46:0/1479258316 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2bc0c8490 0x7fd2bc0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:13.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231195097 0 0) 0x7fd2bc06afa0 con 0x7fd2bc153d70 2023-10-30T13:25:13.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2a8003530 con 0x7fd2bc153d70 2023-10-30T13:25:13.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1130790358 0 0) 0x7fd2a8003530 con 0x7fd2bc153d70 2023-10-30T13:25:13.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2bc0c8490 msgr2=0x7fd2bc0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:13.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 --2- 172.21.15.46:0/1479258316 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2bc0c8490 0x7fd2bc0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:13.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 >> v2:172.21.15.154:3300/0 conn(0x7fd2bc06e470 msgr2=0x7fd2bc06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:13.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 --2- 172.21.15.46:0/1479258316 >> v2:172.21.15.154:3300/0 conn(0x7fd2bc06e470 0x7fd2bc06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:13.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc06e030 con 0x7fd2bc153d70 2023-10-30T13:25:13.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fd2ac001680 con 0x7fd2bc153d70 2023-10-30T13:25:13.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 >> v1:172.21.15.46:6789/0 conn(0x7fd2bc153d70 legacy=0x7fd2bc0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:13.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 --2- 172.21.15.46:0/1479258316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a8003980 0x7fd2a8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:13.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 --2- 172.21.15.46:0/1479258316 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2bc0c8490 0x7fd2a80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:13.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 --2- 172.21.15.46:0/1479258316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a8004d10 0x7fd2a8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:13.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc06d780 con 0x7fd2a8003980 2023-10-30T13:25:13.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc154150 con 0x7fd2a8004d10 2023-10-30T13:25:13.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a800b990 con 0x7fd2bc0c8490 2023-10-30T13:25:13.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2baffd640 1 --2- 172.21.15.46:0/1479258316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a8003980 0x7fd2a8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:13.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2b3fff640 1 --2- 172.21.15.46:0/1479258316 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2bc0c8490 0x7fd2a80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:13.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2bb7fe640 1 --2- 172.21.15.46:0/1479258316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a8004d10 0x7fd2a8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:13.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2baffd640 1 -- 172.21.15.46:0/1479258316 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2bc0c8490 msgr2=0x7fd2a80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:13.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2baffd640 1 --2- 172.21.15.46:0/1479258316 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2bc0c8490 0x7fd2a80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:13.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2baffd640 1 -- 172.21.15.46:0/1479258316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a8004d10 msgr2=0x7fd2a8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:13.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2baffd640 1 --2- 172.21.15.46:0/1479258316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a8004d10 0x7fd2a8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:13.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.175+0000 7fd2baffd640 1 -- 172.21.15.46:0/1479258316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a8003530 con 0x7fd2a8003980 2023-10-30T13:25:13.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.179+0000 7fd2baffd640 1 --2- 172.21.15.46:0/1479258316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a8003980 0x7fd2a8003da0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00c240 tx=0x7fd2ac01cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6605ce30765cae1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:13.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.179+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0a5910 con 0x7fd2a8003980 2023-10-30T13:25:13.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.179+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac0a5a70 con 0x7fd2a8003980 2023-10-30T13:25:13.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.179+0000 7fd2ba7fc640 1 -- 172.21.15.46:0/1479258316 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0a5e40 con 0x7fd2a8003980 2023-10-30T13:25:13.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.179+0000 7fd2c1809640 1 -- 172.21.15.46:0/1479258316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a8003980 msgr2=0x7fd2a8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:13.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.179+0000 7fd2c1809640 1 --2- 172.21.15.46:0/1479258316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a8003980 0x7fd2a8003da0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00c240 tx=0x7fd2ac01cc30 comp rx=0 tx=0).stop 2023-10-30T13:25:13.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.179+0000 7fd2bb7fe640 1 -- 172.21.15.46:0/1479258316 reap_dead start 2023-10-30T13:25:13.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.179+0000 7fd2c1809640 1 -- 172.21.15.46:0/1479258316 shutdown_connections 2023-10-30T13:25:13.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.179+0000 7fd2c1809640 1 -- 172.21.15.46:0/1479258316 >> 172.21.15.46:0/1479258316 conn(0x7fd2bc071ea0 msgr2=0x7fd2bc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:13.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.179+0000 7fd2c1809640 1 -- 172.21.15.46:0/1479258316 shutdown_connections 2023-10-30T13:25:13.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.179+0000 7fd2c1809640 1 -- 172.21.15.46:0/1479258316 wait complete. 2023-10-30T13:25:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2c1809640 1 Processor -- start 2023-10-30T13:25:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2c1809640 1 -- start start 2023-10-30T13:25:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2c1809640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2a8003980 0x7fd2bc1f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:13.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2c1809640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2bc1f5d50 0x7fd2bc14f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:13.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2c1809640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc14f8a0 0x7fd2bc0af030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:13.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2c1809640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0c22b0 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2c1809640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0b0210 con 0x7fd2bc1f5d50 2023-10-30T13:25:13.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2c1809640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc1f2dd0 con 0x7fd2a8003980 2023-10-30T13:25:13.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc14f8a0 0x7fd2bc0af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:13.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc14f8a0 0x7fd2bc0af030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58894/0 (socket says 172.21.15.46:58894) 2023-10-30T13:25:13.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2bb7fe640 1 -- 172.21.15.46:0/1964537485 learned_addr learned my addr 172.21.15.46:0/1964537485 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:13.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2b3fff640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2bc1f5d50 0x7fd2bc14f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:13.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2baffd640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2a8003980 0x7fd2bc1f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:13.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2bb7fe640 1 -- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2a8003980 msgr2=0x7fd2bc1f27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:13.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2bb7fe640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2a8003980 0x7fd2bc1f27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:13.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2bb7fe640 1 -- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2bc1f5d50 msgr2=0x7fd2bc14f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:13.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2bb7fe640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2bc1f5d50 0x7fd2bc14f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:13.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2bb7fe640 1 -- 172.21.15.46:0/1964537485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc1f3090 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2bb7fe640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc14f8a0 0x7fd2bc0af030 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0700c0 tx=0x7fd2b40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:13.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2b8ff9640 1 -- 172.21.15.46:0/1964537485 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b400e070 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2b8ff9640 1 -- 172.21.15.46:0/1964537485 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b4011e10 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2b8ff9640 1 -- 172.21.15.46:0/1964537485 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b40039a0 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2c1809640 1 -- 172.21.15.46:0/1964537485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2bc1f3370 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2c1809640 1 -- 172.21.15.46:0/1964537485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2bc0afb50 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.183+0000 7fd2c1809640 1 -- 172.21.15.46:0/1964537485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2bc0b0350 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.187+0000 7fd2b8ff9640 1 -- 172.21.15.46:0/1964537485 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd2b4003b00 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.187+0000 7fd2b8ff9640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd29c0664a0 0x7fd29c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.187+0000 7fd2b8ff9640 1 -- 172.21.15.46:0/1964537485 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd2b4084680 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.187+0000 7fd2baffd640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd29c0664a0 0x7fd29c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.187+0000 7fd2baffd640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd29c0664a0 0x7fd29c068960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0aa000 tx=0x7fd2ac001a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.191+0000 7fd2b8ff9640 1 -- 172.21.15.46:0/1964537485 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd2bc0b0350 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.379+0000 7fd2c1809640 1 -- 172.21.15.46:0/1964537485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2bc0b0350 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.379+0000 7fd2b8ff9640 1 -- 172.21.15.46:0/1964537485 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6516 (secure 0 0 0) 0x7fd2bc0b0350 con 0x7fd2bc14f8a0 2023-10-30T13:25:13.385 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:13.385 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:23:35.740725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4271":{"gid":4271,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.46:6837/994489214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":994489214},{"type":"v1","addr":"172.21.15.46:6837","nonce":994489214}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:13.385 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-10-30T13:25:13.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.383+0000 7fd2c1809640 1 -- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd29c0664a0 msgr2=0x7fd29c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:13.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.383+0000 7fd2c1809640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd29c0664a0 0x7fd29c068960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0aa000 tx=0x7fd2ac001a80 comp rx=0 tx=0).stop 2023-10-30T13:25:13.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.383+0000 7fd2c1809640 1 -- 172.21.15.46:0/1964537485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc14f8a0 msgr2=0x7fd2bc0af030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:13.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.383+0000 7fd2c1809640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc14f8a0 0x7fd2bc0af030 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0700c0 tx=0x7fd2b40115a0 comp rx=0 tx=0).stop 2023-10-30T13:25:13.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.383+0000 7fd2c1809640 1 -- 172.21.15.46:0/1964537485 shutdown_connections 2023-10-30T13:25:13.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.383+0000 7fd2c1809640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd29c0664a0 0x7fd29c068960 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:13.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.383+0000 7fd2c1809640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc14f8a0 0x7fd2bc0af030 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:13.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.383+0000 7fd2c1809640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2bc1f5d50 0x7fd2bc14f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:13.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.383+0000 7fd2c1809640 1 --2- 172.21.15.46:0/1964537485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2a8003980 0x7fd2bc1f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:13.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.383+0000 7fd2c1809640 1 -- 172.21.15.46:0/1964537485 >> 172.21.15.46:0/1964537485 conn(0x7fd2bc071ea0 msgr2=0x7fd2bc14e470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:13.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.383+0000 7fd2c1809640 1 -- 172.21.15.46:0/1964537485 shutdown_connections 2023-10-30T13:25:13.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:13.383+0000 7fd2c1809640 1 -- 172.21.15.46:0/1964537485 wait complete. 2023-10-30T13:25:13.406 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:25:13.707 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-10-30T13:25:13.741 INFO:tasks.workunit.client.0.smithi046.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-10-30T13:25:13.872 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:25:13.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:25:13.901 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo passed: files are identical 2023-10-30T13:25:13.902 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir foo/.snap/barsnap 2023-10-30T13:25:13.902 INFO:tasks.workunit.client.0.smithi046.stdout:passed: files are identical 2023-10-30T13:25:13.956 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T13:25:13.957 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo OK 2023-10-30T13:25:13.958 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:25:13.958 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:25:14.677 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2023-10-30T13:25:14.677 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2023-10-30T13:25:15.030 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 1 2023-10-30T13:25:15.031 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 2 2023-10-30T13:25:15.032 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 3 2023-10-30T13:25:15.032 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' -e file4 ']' 2023-10-30T13:25:15.033 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/snap1 2023-10-30T13:25:15.050 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 4 2023-10-30T13:25:15.052 INFO:tasks.workunit.client.0.smithi046.stderr:++ ls 2023-10-30T13:25:15.058 INFO:tasks.workunit.client.0.smithi046.stderr:+ now='file1 2023-10-30T13:25:15.059 INFO:tasks.workunit.client.0.smithi046.stderr:file2 2023-10-30T13:25:15.059 INFO:tasks.workunit.client.0.smithi046.stderr:file3 2023-10-30T13:25:15.059 INFO:tasks.workunit.client.0.smithi046.stderr:file4' 2023-10-30T13:25:15.060 INFO:tasks.workunit.client.0.smithi046.stderr:++ ls .snap/snap1 2023-10-30T13:25:15.065 INFO:tasks.workunit.client.0.smithi046.stderr:+ then='file1 2023-10-30T13:25:15.065 INFO:tasks.workunit.client.0.smithi046.stderr:file2 2023-10-30T13:25:15.065 INFO:tasks.workunit.client.0.smithi046.stderr:file3' 2023-10-30T13:25:15.066 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/snap1 2023-10-30T13:25:15.077 INFO:tasks.workunit.client.0.smithi046.stderr:+ '[' 'file1 2023-10-30T13:25:15.077 INFO:tasks.workunit.client.0.smithi046.stderr:file2 2023-10-30T13:25:15.077 INFO:tasks.workunit.client.0.smithi046.stderr:file3 2023-10-30T13:25:15.077 INFO:tasks.workunit.client.0.smithi046.stderr:file4' = 'file1 2023-10-30T13:25:15.078 INFO:tasks.workunit.client.0.smithi046.stderr:file2 2023-10-30T13:25:15.078 INFO:tasks.workunit.client.0.smithi046.stderr:file3' ']' 2023-10-30T13:25:15.078 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 1 2023-10-30T13:25:15.088 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 2 2023-10-30T13:25:15.098 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 3 2023-10-30T13:25:15.108 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/snap1 2023-10-30T13:25:15.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo 4 2023-10-30T13:25:15.128 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/snap1 2023-10-30T13:25:15.141 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm file1 file2 file3 file4 2023-10-30T13:25:15.149 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo OK 2023-10-30T13:25:15.150 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T13:25:15.151 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:25:15.151 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:25:15.178 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2023-10-30T13:25:15.179 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2023-10-30T13:25:15.239 INFO:tasks.workunit.client.0.smithi046.stdout:Create dir 100 to 199 ... 2023-10-30T13:25:15.241 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 100 2023-10-30T13:25:15.247 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 101 2023-10-30T13:25:15.256 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 102 2023-10-30T13:25:15.266 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 103 2023-10-30T13:25:15.275 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 104 2023-10-30T13:25:15.283 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 105 2023-10-30T13:25:15.292 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 106 2023-10-30T13:25:15.302 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 107 2023-10-30T13:25:15.310 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 108 2023-10-30T13:25:15.407 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483c972640 1 Processor -- start 2023-10-30T13:25:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483c972640 1 -- start start 2023-10-30T13:25:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483c972640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48381538a0 0x7f483815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483c972640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f48380ca0e0 0x7f483815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483c972640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48380cf160 0x7f48380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483c972640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48380cb800 con 0x7f48381538a0 2023-10-30T13:25:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483c972640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483815dc30 con 0x7f48380ca0e0 2023-10-30T13:25:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483c972640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48380cb330 con 0x7f48380cf160 2023-10-30T13:25:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48381538a0 0x7f483815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48381538a0 0x7f483815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58900/0 (socket says 172.21.15.46:58900) 2023-10-30T13:25:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483659c640 1 -- 172.21.15.46:0/839061488 learned_addr learned my addr 172.21.15.46:0/839061488 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f4836d9d640 1 --2- 172.21.15.46:0/839061488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48380cf160 0x7f48380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f482ffff640 1 --2- 172.21.15.46:0/839061488 >> v2:172.21.15.154:3300/0 conn(0x7f48380ca0e0 0x7f483815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483659c640 1 -- 172.21.15.46:0/839061488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48380cf160 msgr2=0x7f48380cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483659c640 1 --2- 172.21.15.46:0/839061488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48380cf160 0x7f48380cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483659c640 1 -- 172.21.15.46:0/839061488 >> v2:172.21.15.154:3300/0 conn(0x7f48380ca0e0 msgr2=0x7f483815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483659c640 1 --2- 172.21.15.46:0/839061488 >> v2:172.21.15.154:3300/0 conn(0x7f48380ca0e0 0x7f483815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.547+0000 7f483659c640 1 -- 172.21.15.46:0/839061488 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48380cad70 con 0x7f48381538a0 2023-10-30T13:25:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f483659c640 1 --2- 172.21.15.46:0/839061488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48381538a0 0x7f483815d6f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f48380b30a0 tx=0x7f482801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b29ba5154717b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f4835d9b640 1 -- 172.21.15.46:0/839061488 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48280a89b0 con 0x7f48381538a0 2023-10-30T13:25:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f4835d9b640 1 -- 172.21.15.46:0/839061488 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48280a8b10 con 0x7f48381538a0 2023-10-30T13:25:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f4835d9b640 1 -- 172.21.15.46:0/839061488 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48280a8ea0 con 0x7f48381538a0 2023-10-30T13:25:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f483c972640 1 -- 172.21.15.46:0/839061488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48381538a0 msgr2=0x7f483815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f483c972640 1 --2- 172.21.15.46:0/839061488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48381538a0 0x7f483815d6f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f48380b30a0 tx=0x7f482801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f483c972640 1 -- 172.21.15.46:0/839061488 shutdown_connections 2023-10-30T13:25:15.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f483c972640 1 --2- 172.21.15.46:0/839061488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48380cf160 0x7f48380cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f483c972640 1 --2- 172.21.15.46:0/839061488 >> v2:172.21.15.154:3300/0 conn(0x7f48380ca0e0 0x7f483815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f483c972640 1 --2- 172.21.15.46:0/839061488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48381538a0 0x7f483815d6f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f483c972640 1 -- 172.21.15.46:0/839061488 >> 172.21.15.46:0/839061488 conn(0x7f483806bfe0 msgr2=0x7f483806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:15.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f483c972640 1 -- 172.21.15.46:0/839061488 shutdown_connections 2023-10-30T13:25:15.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.551+0000 7f483c972640 1 -- 172.21.15.46:0/839061488 wait complete. 2023-10-30T13:25:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f483c972640 1 Processor -- start 2023-10-30T13:25:15.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f483c972640 1 -- start start 2023-10-30T13:25:15.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f483c972640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48380ca0e0 0x7f483815b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f483c972640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48380cf160 0x7f48381f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f483c972640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48381538a0 0x7f48380ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f483c972640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483815aec0 con 0x7f48380cf160 2023-10-30T13:25:15.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f483c972640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4838150dc0 con 0x7f48380ca0e0 2023-10-30T13:25:15.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f483c972640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48380b46b0 con 0x7f48381538a0 2023-10-30T13:25:15.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f482ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48380cf160 0x7f48381f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f482ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48380cf160 0x7f48381f2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58904/0 (socket says 172.21.15.46:58904) 2023-10-30T13:25:15.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f482ffff640 1 -- 172.21.15.46:0/1535167425 learned_addr learned my addr 172.21.15.46:0/1535167425 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f483659c640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48380ca0e0 0x7f483815b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f4836d9d640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48381538a0 0x7f48380ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f482ffff640 1 -- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48381538a0 msgr2=0x7f48380ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f482ffff640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48381538a0 0x7f48380ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f482ffff640 1 -- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48380ca0e0 msgr2=0x7f483815b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f482ffff640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48380ca0e0 0x7f483815b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f482ffff640 1 -- 172.21.15.46:0/1535167425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48280094d0 con 0x7f48380cf160 2023-10-30T13:25:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f4836d9d640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48381538a0 0x7f48380ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f482ffff640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48380cf160 0x7f48381f2580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f482000d830 tx=0x7f482000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:15.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f482effd640 1 -- 172.21.15.46:0/1535167425 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48200096a0 con 0x7f48380cf160 2023-10-30T13:25:15.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f482effd640 1 -- 172.21.15.46:0/1535167425 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4820009800 con 0x7f48380cf160 2023-10-30T13:25:15.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f482effd640 1 -- 172.21.15.46:0/1535167425 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4820009b70 con 0x7f48380cf160 2023-10-30T13:25:15.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f483c972640 1 -- 172.21.15.46:0/1535167425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483815a8b0 con 0x7f48380cf160 2023-10-30T13:25:15.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.555+0000 7f483c972640 1 -- 172.21.15.46:0/1535167425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48380b4c30 con 0x7f48380cf160 2023-10-30T13:25:15.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.559+0000 7f483c972640 1 -- 172.21.15.46:0/1535167425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48380b2a50 con 0x7f48380cf160 2023-10-30T13:25:15.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.559+0000 7f482effd640 1 -- 172.21.15.46:0/1535167425 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4820014410 con 0x7f48380cf160 2023-10-30T13:25:15.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.559+0000 7f482effd640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48180664a0 0x7f4818068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.559+0000 7f482effd640 1 -- 172.21.15.46:0/1535167425 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f4820084a10 con 0x7f48380cf160 2023-10-30T13:25:15.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.559+0000 7f483659c640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48180664a0 0x7f4818068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.559+0000 7f483659c640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48180664a0 0x7f4818068960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f48280b2000 tx=0x7f4828009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:15.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.563+0000 7f482effd640 1 -- 172.21.15.46:0/1535167425 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4820051c70 con 0x7f48380cf160 2023-10-30T13:25:15.572 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 109 2023-10-30T13:25:15.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.755+0000 7f483c972640 1 -- 172.21.15.46:0/1535167425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f483815b070 con 0x7f48380cf160 2023-10-30T13:25:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.755+0000 7f482effd640 1 -- 172.21.15.46:0/1535167425 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5730 (secure 0 0 0) 0x7f4820055290 con 0x7f48380cf160 2023-10-30T13:25:15.760 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:15.760 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:13.695245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:15.761 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 13 2023-10-30T13:25:15.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.759+0000 7f483c972640 1 -- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48180664a0 msgr2=0x7f4818068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:15.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.759+0000 7f483c972640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48180664a0 0x7f4818068960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f48280b2000 tx=0x7f4828009040 comp rx=0 tx=0).stop 2023-10-30T13:25:15.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.759+0000 7f483c972640 1 -- 172.21.15.46:0/1535167425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48380cf160 msgr2=0x7f48381f2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:15.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.759+0000 7f483c972640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48380cf160 0x7f48381f2580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f482000d830 tx=0x7f482000dcf0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.759+0000 7f483c972640 1 -- 172.21.15.46:0/1535167425 shutdown_connections 2023-10-30T13:25:15.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.759+0000 7f483c972640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48180664a0 0x7f4818068960 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.759+0000 7f483c972640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48381538a0 0x7f48380ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.759+0000 7f483c972640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48380cf160 0x7f48381f2580 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.759+0000 7f483c972640 1 --2- 172.21.15.46:0/1535167425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48380ca0e0 0x7f483815b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.759+0000 7f483c972640 1 -- 172.21.15.46:0/1535167425 >> 172.21.15.46:0/1535167425 conn(0x7f483806bfe0 msgr2=0x7f48381524c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:15.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.759+0000 7f483c972640 1 -- 172.21.15.46:0/1535167425 shutdown_connections 2023-10-30T13:25:15.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.759+0000 7f483c972640 1 -- 172.21.15.46:0/1535167425 wait complete. 2023-10-30T13:25:15.782 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T13:25:15.782 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:25:15.782 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:15.834 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 110 2023-10-30T13:25:15.844 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 111 2023-10-30T13:25:15.854 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 112 2023-10-30T13:25:15.867 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 113 2023-10-30T13:25:15.875 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 114 2023-10-30T13:25:15.885 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 115 2023-10-30T13:25:15.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.943+0000 7f5c7d972640 1 Processor -- start 2023-10-30T13:25:15.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c7d972640 1 -- start start 2023-10-30T13:25:15.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c7d972640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c781538a0 0x7f5c7815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c7d972640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c780ca0e0 0x7f5c7815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c7d972640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c780cb800 con 0x7f5c780cf160 2023-10-30T13:25:15.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c7d972640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7815dc30 con 0x7f5c781538a0 2023-10-30T13:25:15.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c7d972640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c780cb330 con 0x7f5c780ca0e0 2023-10-30T13:25:15.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c76ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c781538a0 0x7f5c7815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c777fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5c780cf160 0x7f5c780cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50080/0 (socket says 172.21.15.46:50080) 2023-10-30T13:25:15.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c777fe640 1 -- 172.21.15.46:0/3717119155 learned_addr learned my addr 172.21.15.46:0/3717119155 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:25:15.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c767fc640 1 --2- 172.21.15.46:0/3717119155 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c780ca0e0 0x7f5c7815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c76ffd640 1 -- 172.21.15.46:0/3717119155 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c780ca0e0 msgr2=0x7f5c7815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:15.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c76ffd640 1 --2- 172.21.15.46:0/3717119155 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c780ca0e0 0x7f5c7815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c76ffd640 1 -- 172.21.15.46:0/3717119155 >> v1:172.21.15.46:6789/0 conn(0x7f5c780cf160 legacy=0x7f5c780cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:15.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c76ffd640 1 -- 172.21.15.46:0/3717119155 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c780cad70 con 0x7f5c781538a0 2023-10-30T13:25:15.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.947+0000 7f5c75ffb640 1 -- 172.21.15.46:0/3717119155 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393127121 0 0) 0x7f5c780cb800 con 0x7f5c780cf160 2023-10-30T13:25:15.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c76ffd640 1 --2- 172.21.15.46:0/3717119155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c781538a0 0x7f5c7815d6f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5c680097c0 tx=0x7f5c6801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915cc335fcb77042 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:15.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c75ffb640 1 -- 172.21.15.46:0/3717119155 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c680a8970 con 0x7f5c781538a0 2023-10-30T13:25:15.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c75ffb640 1 -- 172.21.15.46:0/3717119155 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c680a8ad0 con 0x7f5c781538a0 2023-10-30T13:25:15.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c75ffb640 1 -- 172.21.15.46:0/3717119155 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c680a8e60 con 0x7f5c781538a0 2023-10-30T13:25:15.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c7d972640 1 -- 172.21.15.46:0/3717119155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c781538a0 msgr2=0x7f5c7815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:15.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c7d972640 1 --2- 172.21.15.46:0/3717119155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c781538a0 0x7f5c7815d6f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5c680097c0 tx=0x7f5c6801ec70 comp rx=0 tx=0).stop 2023-10-30T13:25:15.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c7d972640 1 -- 172.21.15.46:0/3717119155 shutdown_connections 2023-10-30T13:25:15.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c7d972640 1 --2- 172.21.15.46:0/3717119155 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c780ca0e0 0x7f5c7815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c7d972640 1 --2- 172.21.15.46:0/3717119155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c781538a0 0x7f5c7815d6f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c7d972640 1 -- 172.21.15.46:0/3717119155 >> 172.21.15.46:0/3717119155 conn(0x7f5c7806bfe0 msgr2=0x7f5c7806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:15.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c7d972640 1 -- 172.21.15.46:0/3717119155 shutdown_connections 2023-10-30T13:25:15.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c7d972640 1 -- 172.21.15.46:0/3717119155 wait complete. 2023-10-30T13:25:15.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c7d972640 1 Processor -- start 2023-10-30T13:25:15.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.951+0000 7f5c7d972640 1 -- start start 2023-10-30T13:25:15.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c7d972640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c780ca0e0 0x7f5c78140530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c7d972640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c780cf160 0x7f5c78146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c7d972640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c781538a0 0x7f5c78147110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c7d972640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c780ce8c0 con 0x7f5c780cf160 2023-10-30T13:25:15.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c7d972640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7814eed0 con 0x7f5c780ca0e0 2023-10-30T13:25:15.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c7d972640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c781f3290 con 0x7f5c781538a0 2023-10-30T13:25:15.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c767fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c780cf160 0x7f5c78146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c767fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c780cf160 0x7f5c78146bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58922/0 (socket says 172.21.15.46:58922) 2023-10-30T13:25:15.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c767fc640 1 -- 172.21.15.46:0/1516686264 learned_addr learned my addr 172.21.15.46:0/1516686264 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:15.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c777fe640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c781538a0 0x7f5c78147110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c76ffd640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c780ca0e0 0x7f5c78140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c767fc640 1 -- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c781538a0 msgr2=0x7f5c78147110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:15.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c767fc640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c781538a0 0x7f5c78147110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c767fc640 1 -- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c780ca0e0 msgr2=0x7f5c78140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:15.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c767fc640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c780ca0e0 0x7f5c78140530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:15.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c767fc640 1 -- 172.21.15.46:0/1516686264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c68009420 con 0x7f5c780cf160 2023-10-30T13:25:15.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c76ffd640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c780ca0e0 0x7f5c78140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:15.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c767fc640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c780cf160 0x7f5c78146bd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00d800 tx=0x7f5c6c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:15.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c5ffff640 1 -- 172.21.15.46:0/1516686264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c00b660 con 0x7f5c780cf160 2023-10-30T13:25:15.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c5ffff640 1 -- 172.21.15.46:0/1516686264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c00b7c0 con 0x7f5c780cf160 2023-10-30T13:25:15.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c5ffff640 1 -- 172.21.15.46:0/1516686264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c00bb30 con 0x7f5c780cf160 2023-10-30T13:25:15.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c7d972640 1 -- 172.21.15.46:0/1516686264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c781f2430 con 0x7f5c780cf160 2023-10-30T13:25:15.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c7d972640 1 -- 172.21.15.46:0/1516686264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c781f29f0 con 0x7f5c780cf160 2023-10-30T13:25:15.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c7d972640 1 -- 172.21.15.46:0/1516686264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c780b2a50 con 0x7f5c780cf160 2023-10-30T13:25:15.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.955+0000 7f5c5ffff640 1 -- 172.21.15.46:0/1516686264 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5c6c00bc90 con 0x7f5c780cf160 2023-10-30T13:25:15.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.959+0000 7f5c5ffff640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c640664a0 0x7f5c64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:15.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.959+0000 7f5c5ffff640 1 -- 172.21.15.46:0/1516686264 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5996+0+0 (secure 0 0 0) 0x7f5c6c084630 con 0x7f5c780cf160 2023-10-30T13:25:15.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.959+0000 7f5c76ffd640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c640664a0 0x7f5c64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:15.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.959+0000 7f5c76ffd640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c640664a0 0x7f5c64068960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5c680b2000 tx=0x7f5c68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:15.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:15.963+0000 7f5c5ffff640 1 -- 172.21.15.46:0/1516686264 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5c6c051860 con 0x7f5c780cf160 2023-10-30T13:25:16.013 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 116 2023-10-30T13:25:16.024 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 117 2023-10-30T13:25:16.034 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 118 2023-10-30T13:25:16.043 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 119 2023-10-30T13:25:16.052 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 120 2023-10-30T13:25:16.062 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 121 2023-10-30T13:25:16.094 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 122 2023-10-30T13:25:16.104 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 123 2023-10-30T13:25:16.114 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 124 2023-10-30T13:25:16.123 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 125 2023-10-30T13:25:16.132 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 126 2023-10-30T13:25:16.141 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 127 2023-10-30T13:25:16.150 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 128 2023-10-30T13:25:16.159 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 129 2023-10-30T13:25:16.168 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 130 2023-10-30T13:25:16.177 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 131 2023-10-30T13:25:16.186 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 132 2023-10-30T13:25:16.195 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 133 2023-10-30T13:25:16.204 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 134 2023-10-30T13:25:16.213 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 135 2023-10-30T13:25:16.222 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 136 2023-10-30T13:25:16.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.227+0000 7f5c7d972640 1 -- 172.21.15.46:0/1516686264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c7804e170 con 0x7f5c780cf160 2023-10-30T13:25:16.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.227+0000 7f5c5ffff640 1 -- 172.21.15.46:0/1516686264 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5730 (secure 0 0 0) 0x7f5c6c011340 con 0x7f5c780cf160 2023-10-30T13:25:16.232 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:16.232 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:13.695245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:16.233 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 13 2023-10-30T13:25:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.231+0000 7f5c7d972640 1 -- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c640664a0 msgr2=0x7f5c64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:16.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.231+0000 7f5c7d972640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c640664a0 0x7f5c64068960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5c680b2000 tx=0x7f5c68008040 comp rx=0 tx=0).stop 2023-10-30T13:25:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.231+0000 7f5c7d972640 1 -- 172.21.15.46:0/1516686264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c780cf160 msgr2=0x7f5c78146bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.231+0000 7f5c7d972640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c780cf160 0x7f5c78146bd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00d800 tx=0x7f5c6c00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:25:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.231+0000 7f5c7d972640 1 -- 172.21.15.46:0/1516686264 shutdown_connections 2023-10-30T13:25:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.231+0000 7f5c7d972640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c640664a0 0x7f5c64068960 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:16.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.231+0000 7f5c7d972640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c781538a0 0x7f5c78147110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:16.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.231+0000 7f5c7d972640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c780cf160 0x7f5c78146bd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:16.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.231+0000 7f5c7d972640 1 --2- 172.21.15.46:0/1516686264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c780ca0e0 0x7f5c78140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:16.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.231+0000 7f5c7d972640 1 -- 172.21.15.46:0/1516686264 >> 172.21.15.46:0/1516686264 conn(0x7f5c7806bfe0 msgr2=0x7f5c781527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:16.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.231+0000 7f5c7d972640 1 -- 172.21.15.46:0/1516686264 shutdown_connections 2023-10-30T13:25:16.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:16.231+0000 7f5c7d972640 1 -- 172.21.15.46:0/1516686264 wait complete. 2023-10-30T13:25:16.255 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 13, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 13, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4259": { "addr": "172.21.15.46:6833/875791418", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 875791418, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 875791418, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4259, "incarnation": 13, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_4288": { "addr": "172.21.15.154:6835/3109262047", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3109262047, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3109262047, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4306": { "addr": "172.21.15.46:6836/275749119", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 275749119, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 275749119, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 23, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:25:13.695245+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4288, "mds_2": 4259 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:25:16.411 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 137 2023-10-30T13:25:16.728 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 138 2023-10-30T13:25:16.738 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 139 2023-10-30T13:25:16.748 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 140 2023-10-30T13:25:16.758 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 141 2023-10-30T13:25:16.768 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 142 2023-10-30T13:25:16.778 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 143 2023-10-30T13:25:16.788 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 144 2023-10-30T13:25:16.798 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 145 2023-10-30T13:25:17.435 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 146 2023-10-30T13:25:17.619 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 147 2023-10-30T13:25:17.628 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 148 2023-10-30T13:25:17.637 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 149 2023-10-30T13:25:17.877 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 150 2023-10-30T13:25:17.887 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 151 2023-10-30T13:25:18.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:18.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.395+0000 7f1964f72640 1 Processor -- start 2023-10-30T13:25:18.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.395+0000 7f1964f72640 1 -- start start 2023-10-30T13:25:18.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.395+0000 7f1964f72640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f19601539a0 0x7f196015a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:18.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.395+0000 7f1964f72640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1960069e90 0x7f196006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:18.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.395+0000 7f1964f72640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196015a8c0 con 0x7f1960070a00 2023-10-30T13:25:18.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.395+0000 7f1964f72640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f196006fbb0 con 0x7f19601539a0 2023-10-30T13:25:18.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.395+0000 7f1964f72640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1960153d80 con 0x7f1960069e90 2023-10-30T13:25:18.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.395+0000 7f195e59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f19601539a0 0x7f196015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:18.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.395+0000 7f195e59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f19601539a0 0x7f196015a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49470/0 (socket says 172.21.15.46:49470) 2023-10-30T13:25:18.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.395+0000 7f195e59c640 1 -- 172.21.15.46:0/2027304144 learned_addr learned my addr 172.21.15.46:0/2027304144 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:18.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.399+0000 7f195ed9d640 1 --2- 172.21.15.46:0/2027304144 >> v2:172.21.15.154:3300/0 conn(0x7f1960069e90 0x7f196006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:18.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.399+0000 7f195e59c640 1 -- 172.21.15.46:0/2027304144 >> v2:172.21.15.154:3300/0 conn(0x7f1960069e90 msgr2=0x7f196006a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:18.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.399+0000 7f195e59c640 1 --2- 172.21.15.46:0/2027304144 >> v2:172.21.15.154:3300/0 conn(0x7f1960069e90 0x7f196006a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:18.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.399+0000 7f195e59c640 1 -- 172.21.15.46:0/2027304144 >> v1:172.21.15.154:6789/0 conn(0x7f1960070a00 legacy=0x7f1960070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:18.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.399+0000 7f195e59c640 1 -- 172.21.15.46:0/2027304144 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19600c5c50 con 0x7f19601539a0 2023-10-30T13:25:18.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.399+0000 7f195e59c640 1 --2- 172.21.15.46:0/2027304144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19601539a0 0x7f196015a380 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f19600c8900 tx=0x7f195001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3303a0df9a5f96bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:18.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.399+0000 7f195dd9b640 1 -- 172.21.15.46:0/2027304144 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500ac640 con 0x7f19601539a0 2023-10-30T13:25:18.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.399+0000 7f195dd9b640 1 -- 172.21.15.46:0/2027304144 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19500ac7a0 con 0x7f19601539a0 2023-10-30T13:25:18.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.399+0000 7f195dd9b640 1 -- 172.21.15.46:0/2027304144 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500acb30 con 0x7f19601539a0 2023-10-30T13:25:18.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 -- 172.21.15.46:0/2027304144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19601539a0 msgr2=0x7f196015a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:18.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 --2- 172.21.15.46:0/2027304144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19601539a0 0x7f196015a380 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f19600c8900 tx=0x7f195001c400 comp rx=0 tx=0).stop 2023-10-30T13:25:18.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 -- 172.21.15.46:0/2027304144 shutdown_connections 2023-10-30T13:25:18.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 --2- 172.21.15.46:0/2027304144 >> v2:172.21.15.154:3300/0 conn(0x7f1960069e90 0x7f196006a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:18.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 --2- 172.21.15.46:0/2027304144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19601539a0 0x7f196015a380 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:18.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 -- 172.21.15.46:0/2027304144 >> 172.21.15.46:0/2027304144 conn(0x7f19600c7910 msgr2=0x7f19600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:18.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 -- 172.21.15.46:0/2027304144 shutdown_connections 2023-10-30T13:25:18.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 -- 172.21.15.46:0/2027304144 wait complete. 2023-10-30T13:25:18.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 Processor -- start 2023-10-30T13:25:18.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 -- start start 2023-10-30T13:25:18.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1960069e90 0x7f19600c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:18.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1960070a00 0x7f1960159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:18.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19601539a0 0x7f1960159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:18.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1960157770 con 0x7f1960069e90 2023-10-30T13:25:18.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196015a550 con 0x7f19601539a0 2023-10-30T13:25:18.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1964f72640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19600c9620 con 0x7f1960070a00 2023-10-30T13:25:18.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f195e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1960069e90 0x7f19600c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f195e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1960069e90 0x7f19600c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49472/0 (socket says 172.21.15.46:49472) 2023-10-30T13:25:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f195e59c640 1 -- 172.21.15.46:0/3538258640 learned_addr learned my addr 172.21.15.46:0/3538258640 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f1957fff640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1960070a00 0x7f1960159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f195ed9d640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19601539a0 0x7f1960159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f195e59c640 1 -- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1960070a00 msgr2=0x7f1960159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:18.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f195e59c640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1960070a00 0x7f1960159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:18.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f195e59c640 1 -- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19601539a0 msgr2=0x7f1960159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:18.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f195e59c640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19601539a0 0x7f1960159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:18.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.403+0000 7f195e59c640 1 -- 172.21.15.46:0/3538258640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19500094d0 con 0x7f1960069e90 2023-10-30T13:25:18.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.407+0000 7f195e59c640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1960069e90 0x7f19600c12a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f19500af040 tx=0x7f19500191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:18.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.407+0000 7f1956ffd640 1 -- 172.21.15.46:0/3538258640 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500b5640 con 0x7f1960069e90 2023-10-30T13:25:18.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.407+0000 7f1956ffd640 1 -- 172.21.15.46:0/3538258640 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19500b57c0 con 0x7f1960069e90 2023-10-30T13:25:18.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.407+0000 7f1964f72640 1 -- 172.21.15.46:0/3538258640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19600c9900 con 0x7f1960069e90 2023-10-30T13:25:18.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.407+0000 7f1956ffd640 1 -- 172.21.15.46:0/3538258640 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500b5b70 con 0x7f1960069e90 2023-10-30T13:25:18.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.407+0000 7f1964f72640 1 -- 172.21.15.46:0/3538258640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196015a250 con 0x7f1960069e90 2023-10-30T13:25:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.407+0000 7f1964f72640 1 -- 172.21.15.46:0/3538258640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19600c9e20 con 0x7f1960069e90 2023-10-30T13:25:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.407+0000 7f1956ffd640 1 -- 172.21.15.46:0/3538258640 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f19500b3070 con 0x7f1960069e90 2023-10-30T13:25:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.407+0000 7f1956ffd640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19400664a0 0x7f1940068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.407+0000 7f1956ffd640 1 -- 172.21.15.46:0/3538258640 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5996+0+0 (secure 0 0 0) 0x7f195011e170 con 0x7f1960069e90 2023-10-30T13:25:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.407+0000 7f1957fff640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19400664a0 0x7f1940068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.411+0000 7f1957fff640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19400664a0 0x7f1940068960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f19600cadb0 tx=0x7f1948009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:18.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.411+0000 7f1956ffd640 1 -- 172.21.15.46:0/3538258640 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f19500eb4c0 con 0x7f1960069e90 2023-10-30T13:25:18.427 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 152 2023-10-30T13:25:18.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.603+0000 7f1964f72640 1 -- 172.21.15.46:0/3538258640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f196004e170 con 0x7f1960069e90 2023-10-30T13:25:18.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.603+0000 7f1956ffd640 1 -- 172.21.15.46:0/3538258640 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5730 (secure 0 0 0) 0x7f19500eeae0 con 0x7f1960069e90 2023-10-30T13:25:18.609 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:18.609 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:13.695245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:18.610 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 13 2023-10-30T13:25:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.607+0000 7f1964f72640 1 -- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19400664a0 msgr2=0x7f1940068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.607+0000 7f1964f72640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19400664a0 0x7f1940068960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f19600cadb0 tx=0x7f1948009040 comp rx=0 tx=0).stop 2023-10-30T13:25:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.607+0000 7f1964f72640 1 -- 172.21.15.46:0/3538258640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1960069e90 msgr2=0x7f19600c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.607+0000 7f1964f72640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1960069e90 0x7f19600c12a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f19500af040 tx=0x7f19500191a0 comp rx=0 tx=0).stop 2023-10-30T13:25:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.607+0000 7f1964f72640 1 -- 172.21.15.46:0/3538258640 shutdown_connections 2023-10-30T13:25:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.607+0000 7f1964f72640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19400664a0 0x7f1940068960 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.607+0000 7f1964f72640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19601539a0 0x7f1960159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.607+0000 7f1964f72640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1960070a00 0x7f1960159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.607+0000 7f1964f72640 1 --2- 172.21.15.46:0/3538258640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1960069e90 0x7f19600c12a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.607+0000 7f1964f72640 1 -- 172.21.15.46:0/3538258640 >> 172.21.15.46:0/3538258640 conn(0x7f19600c7910 msgr2=0x7f196006b340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:18.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.607+0000 7f1964f72640 1 -- 172.21.15.46:0/3538258640 shutdown_connections 2023-10-30T13:25:18.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:18.607+0000 7f1964f72640 1 -- 172.21.15.46:0/3538258640 wait complete. 2023-10-30T13:25:18.630 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:25:18.645 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 153 2023-10-30T13:25:18.654 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 154 2023-10-30T13:25:18.663 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 155 2023-10-30T13:25:18.673 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 156 2023-10-30T13:25:18.683 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 157 2023-10-30T13:25:18.692 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 158 2023-10-30T13:25:18.701 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 159 2023-10-30T13:25:18.712 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 160 2023-10-30T13:25:18.725 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 161 2023-10-30T13:25:18.737 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 162 2023-10-30T13:25:18.751 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 163 2023-10-30T13:25:18.765 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 164 2023-10-30T13:25:18.778 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 165 2023-10-30T13:25:18.791 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 166 2023-10-30T13:25:19.419 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 167 2023-10-30T13:25:19.559 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 168 2023-10-30T13:25:19.570 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 169 2023-10-30T13:25:19.908 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 170 2023-10-30T13:25:19.921 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 171 2023-10-30T13:25:19.932 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 172 2023-10-30T13:25:19.944 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 173 2023-10-30T13:25:20.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:20.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.771+0000 7f81c3bd3640 1 Processor -- start 2023-10-30T13:25:20.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.771+0000 7f81c3bd3640 1 -- start start 2023-10-30T13:25:20.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.771+0000 7f81c3bd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81bc1580e0 0x7f81bc070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:20.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.771+0000 7f81c3bd3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f81bc0c6540 0x7f81bc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:20.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.771+0000 7f81c3bd3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f81bc069e90 0x7f81bc06a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:20.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.771+0000 7f81c3bd3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81bc0c5810 con 0x7f81bc069e90 2023-10-30T13:25:20.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.771+0000 7f81c3bd3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81bc0c6960 con 0x7f81bc0c6540 2023-10-30T13:25:20.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.771+0000 7f81c3bd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc0c6bb0 con 0x7f81bc1580e0 2023-10-30T13:25:20.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c2170640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f81bc069e90 0x7f81bc06a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:20.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c2170640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f81bc069e90 0x7f81bc06a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49474/0 (socket says 172.21.15.46:49474) 2023-10-30T13:25:20.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c2170640 1 -- 172.21.15.46:0/49973528 learned_addr learned my addr 172.21.15.46:0/49973528 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:20.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c196f640 1 --2- 172.21.15.46:0/49973528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81bc1580e0 0x7f81bc070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:20.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c116e640 1 --2- 172.21.15.46:0/49973528 >> v2:172.21.15.154:3300/0 conn(0x7f81bc0c6540 0x7f81bc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:20.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c2170640 1 -- 172.21.15.46:0/49973528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81bc1580e0 msgr2=0x7f81bc070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:20.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c2170640 1 --2- 172.21.15.46:0/49973528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81bc1580e0 0x7f81bc070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c2170640 1 -- 172.21.15.46:0/49973528 >> v2:172.21.15.154:3300/0 conn(0x7f81bc0c6540 msgr2=0x7f81bc15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:20.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c2170640 1 --2- 172.21.15.46:0/49973528 >> v2:172.21.15.154:3300/0 conn(0x7f81bc0c6540 0x7f81bc15e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c2170640 1 -- 172.21.15.46:0/49973528 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc0c6e70 con 0x7f81bc069e90 2023-10-30T13:25:20.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c116e640 1 --2- 172.21.15.46:0/49973528 >> v2:172.21.15.154:3300/0 conn(0x7f81bc0c6540 0x7f81bc15e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:20.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c2170640 1 --2- 172.21.15.46:0/49973528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81bc069e90 0x7f81bc06a320 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f81bc06f230 tx=0x7f81b401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b8c1192639a0ec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:20.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c096d640 1 -- 172.21.15.46:0/49973528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b40a5760 con 0x7f81bc069e90 2023-10-30T13:25:20.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c096d640 1 -- 172.21.15.46:0/49973528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b40a58c0 con 0x7f81bc069e90 2023-10-30T13:25:20.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.775+0000 7f81c096d640 1 -- 172.21.15.46:0/49973528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b40a5c50 con 0x7f81bc069e90 2023-10-30T13:25:20.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 -- 172.21.15.46:0/49973528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81bc069e90 msgr2=0x7f81bc06a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:20.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 --2- 172.21.15.46:0/49973528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81bc069e90 0x7f81bc06a320 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f81bc06f230 tx=0x7f81b401c9d0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 -- 172.21.15.46:0/49973528 shutdown_connections 2023-10-30T13:25:20.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 --2- 172.21.15.46:0/49973528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81bc069e90 0x7f81bc06a320 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 --2- 172.21.15.46:0/49973528 >> v2:172.21.15.154:3300/0 conn(0x7f81bc0c6540 0x7f81bc15e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 --2- 172.21.15.46:0/49973528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81bc1580e0 0x7f81bc070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 -- 172.21.15.46:0/49973528 >> 172.21.15.46:0/49973528 conn(0x7f81bc0c7910 msgr2=0x7f81bc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 -- 172.21.15.46:0/49973528 shutdown_connections 2023-10-30T13:25:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 -- 172.21.15.46:0/49973528 wait complete. 2023-10-30T13:25:20.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 Processor -- start 2023-10-30T13:25:20.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 -- start start 2023-10-30T13:25:20.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81bc069e90 0x7f81bc160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:20.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81bc0c6540 0x7f81bc0ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:20.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81bc1580e0 0x7f81bc1563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:20.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc160020 con 0x7f81bc069e90 2023-10-30T13:25:20.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc15ef20 con 0x7f81bc1580e0 2023-10-30T13:25:20.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c3bd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc160cd0 con 0x7f81bc0c6540 2023-10-30T13:25:20.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c196f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81bc069e90 0x7f81bc160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:20.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c196f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81bc069e90 0x7f81bc160620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49478/0 (socket says 172.21.15.46:49478) 2023-10-30T13:25:20.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c196f640 1 -- 172.21.15.46:0/1986883262 learned_addr learned my addr 172.21.15.46:0/1986883262 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:20.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c116e640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81bc0c6540 0x7f81bc0ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:20.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c2170640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81bc1580e0 0x7f81bc1563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:20.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c196f640 1 -- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81bc0c6540 msgr2=0x7f81bc0ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:20.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c196f640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81bc0c6540 0x7f81bc0ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c196f640 1 -- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81bc1580e0 msgr2=0x7f81bc1563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:20.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c196f640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81bc1580e0 0x7f81bc1563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.779+0000 7f81c196f640 1 -- 172.21.15.46:0/1986883262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b4009b30 con 0x7f81bc069e90 2023-10-30T13:25:20.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.783+0000 7f81c196f640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81bc069e90 0x7f81bc160620 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f81b800e8a0 tx=0x7f81b800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:20.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.783+0000 7f81b27fc640 1 -- 172.21.15.46:0/1986883262 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8009510 con 0x7f81bc069e90 2023-10-30T13:25:20.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.783+0000 7f81b27fc640 1 -- 172.21.15.46:0/1986883262 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b800c940 con 0x7f81bc069e90 2023-10-30T13:25:20.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.783+0000 7f81b27fc640 1 -- 172.21.15.46:0/1986883262 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b800ccb0 con 0x7f81bc069e90 2023-10-30T13:25:20.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.783+0000 7f81c3bd3640 1 -- 172.21.15.46:0/1986883262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81bc15f680 con 0x7f81bc069e90 2023-10-30T13:25:20.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.783+0000 7f81c3bd3640 1 -- 172.21.15.46:0/1986883262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81bc15fc40 con 0x7f81bc069e90 2023-10-30T13:25:20.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.783+0000 7f81c3bd3640 1 -- 172.21.15.46:0/1986883262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81bc04e170 con 0x7f81bc069e90 2023-10-30T13:25:20.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.783+0000 7f81b27fc640 1 -- 172.21.15.46:0/1986883262 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f81b8013470 con 0x7f81bc069e90 2023-10-30T13:25:20.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.783+0000 7f81b27fc640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81a00664d0 0x7f81a0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:20.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.783+0000 7f81b27fc640 1 -- 172.21.15.46:0/1986883262 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5996+0+0 (secure 0 0 0) 0x7f81b8083ff0 con 0x7f81bc069e90 2023-10-30T13:25:20.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.783+0000 7f81c116e640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81a00664d0 0x7f81a0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:20.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.787+0000 7f81c116e640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81a00664d0 0x7f81a0068990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f81bc157b00 tx=0x7f81ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:20.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.787+0000 7f81b27fc640 1 -- 172.21.15.46:0/1986883262 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f81b80513f0 con 0x7f81bc069e90 2023-10-30T13:25:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 -- 172.21.15.46:0/1986883262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81bc0c7cf0 con 0x7f81bc069e90 2023-10-30T13:25:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81b27fc640 1 -- 172.21.15.46:0/1986883262 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5732 (secure 0 0 0) 0x7f81b8054a10 con 0x7f81bc069e90 2023-10-30T13:25:20.984 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:20.984 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:18.713882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:resolve","state_seq":33,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 14 2023-10-30T13:25:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 -- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81a00664d0 msgr2=0x7f81a0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81a00664d0 0x7f81a0068990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f81bc157b00 tx=0x7f81ac008040 comp rx=0 tx=0).stop 2023-10-30T13:25:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 -- 172.21.15.46:0/1986883262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81bc069e90 msgr2=0x7f81bc160620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81bc069e90 0x7f81bc160620 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f81b800e8a0 tx=0x7f81b800eca0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 -- 172.21.15.46:0/1986883262 shutdown_connections 2023-10-30T13:25:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81a00664d0 0x7f81a0068990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81bc1580e0 0x7f81bc1563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81bc0c6540 0x7f81bc0ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 --2- 172.21.15.46:0/1986883262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81bc069e90 0x7f81bc160620 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 -- 172.21.15.46:0/1986883262 >> 172.21.15.46:0/1986883262 conn(0x7f81bc0c7910 msgr2=0x7f81bc154540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 -- 172.21.15.46:0/1986883262 shutdown_connections 2023-10-30T13:25:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:20.979+0000 7f81c3bd3640 1 -- 172.21.15.46:0/1986883262 wait complete. 2023-10-30T13:25:21.004 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:25:21.452 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 174 2023-10-30T13:25:21.711 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 175 2023-10-30T13:25:21.726 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 176 2023-10-30T13:25:21.741 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 177 2023-10-30T13:25:21.755 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 178 2023-10-30T13:25:21.770 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 179 2023-10-30T13:25:21.781 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 180 2023-10-30T13:25:21.791 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 181 2023-10-30T13:25:22.346 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 182 2023-10-30T13:25:22.356 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 183 2023-10-30T13:25:22.367 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 184 2023-10-30T13:25:22.377 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 185 2023-10-30T13:25:22.387 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 186 2023-10-30T13:25:22.398 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 187 2023-10-30T13:25:22.408 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 188 2023-10-30T13:25:22.419 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 189 2023-10-30T13:25:22.429 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 190 2023-10-30T13:25:22.438 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 191 2023-10-30T13:25:22.447 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 192 2023-10-30T13:25:22.459 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 193 2023-10-30T13:25:22.557 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 194 2023-10-30T13:25:22.574 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 195 2023-10-30T13:25:22.777 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 196 2023-10-30T13:25:23.006 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:23.038 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 197 2023-10-30T13:25:23.054 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 198 2023-10-30T13:25:23.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.155+0000 7f91a7017640 1 Processor -- start 2023-10-30T13:25:23.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.155+0000 7f91a7017640 1 -- start start 2023-10-30T13:25:23.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.159+0000 7f91a7017640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f91a00c6b80 0x7f91a00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.159+0000 7f91a7017640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a00c0a50 con 0x7f91a00c6f60 2023-10-30T13:25:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.159+0000 7f91a7017640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a006f6e0 con 0x7f91a013f4a0 2023-10-30T13:25:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.159+0000 7f91a7017640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a006f9c0 con 0x7f91a00c6b80 2023-10-30T13:25:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.159+0000 7f919ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f91a00c6f60 0x7f91a013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49662/0 (socket says 172.21.15.46:49662) 2023-10-30T13:25:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.159+0000 7f919ffff640 1 -- 172.21.15.46:0/2382302331 learned_addr learned my addr 172.21.15.46:0/2382302331 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:25:23.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.159+0000 7f91a4db3640 1 --2- 172.21.15.46:0/2382302331 >> v2:172.21.15.154:3300/0 conn(0x7f91a00c6b80 0x7f91a00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:23.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.159+0000 7f919f7fe640 1 -- 172.21.15.46:0/2382302331 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114485643 0 0) 0x7f91a00c0a50 con 0x7f91a00c6f60 2023-10-30T13:25:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.159+0000 7f919f7fe640 1 -- 172.21.15.46:0/2382302331 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9190005740 con 0x7f91a00c6f60 2023-10-30T13:25:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.159+0000 7f91a4db3640 1 -- 172.21.15.46:0/2382302331 >> v1:172.21.15.154:6789/0 conn(0x7f91a013f4a0 legacy=0x7f91a01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.159+0000 7f91a4db3640 1 -- 172.21.15.46:0/2382302331 >> v1:172.21.15.46:6789/0 conn(0x7f91a00c6f60 legacy=0x7f91a013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.159+0000 7f91a4db3640 1 -- 172.21.15.46:0/2382302331 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a006fca0 con 0x7f91a00c6b80 2023-10-30T13:25:23.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.163+0000 7f91a4db3640 1 --2- 172.21.15.46:0/2382302331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a00c6b80 0x7f91a00bfec0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f91940097c0 tx=0x7f919401c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=261eda764eb63282 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:23.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.163+0000 7f919f7fe640 1 -- 172.21.15.46:0/2382302331 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91940ab600 con 0x7f91a00c6b80 2023-10-30T13:25:23.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.163+0000 7f919f7fe640 1 -- 172.21.15.46:0/2382302331 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91940a7050 con 0x7f91a00c6b80 2023-10-30T13:25:23.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.163+0000 7f919f7fe640 1 -- 172.21.15.46:0/2382302331 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91940ab980 con 0x7f91a00c6b80 2023-10-30T13:25:23.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.163+0000 7f91a7017640 1 -- 172.21.15.46:0/2382302331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a00c6b80 msgr2=0x7f91a00bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:23.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.163+0000 7f91a7017640 1 --2- 172.21.15.46:0/2382302331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a00c6b80 0x7f91a00bfec0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f91940097c0 tx=0x7f919401c480 comp rx=0 tx=0).stop 2023-10-30T13:25:23.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.163+0000 7f91a7017640 1 -- 172.21.15.46:0/2382302331 shutdown_connections 2023-10-30T13:25:23.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.163+0000 7f91a7017640 1 --2- 172.21.15.46:0/2382302331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a00c6b80 0x7f91a00bfec0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:23.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.163+0000 7f91a7017640 1 -- 172.21.15.46:0/2382302331 >> 172.21.15.46:0/2382302331 conn(0x7f91a00c7910 msgr2=0x7f91a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:23.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.163+0000 7f91a7017640 1 -- 172.21.15.46:0/2382302331 shutdown_connections 2023-10-30T13:25:23.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.163+0000 7f91a7017640 1 -- 172.21.15.46:0/2382302331 wait complete. 2023-10-30T13:25:23.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a7017640 1 Processor -- start 2023-10-30T13:25:23.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a7017640 1 -- start start 2023-10-30T13:25:23.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a7017640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a00c6b80 0x7f91a013d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:23.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a7017640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a00c6f60 0x7f91a00af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:23.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a7017640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a013f4a0 0x7f91a0154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:23.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a7017640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a0154350 con 0x7f91a013f4a0 2023-10-30T13:25:23.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a7017640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a0154630 con 0x7f91a00c6f60 2023-10-30T13:25:23.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a7017640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a013dbd0 con 0x7f91a00c6b80 2023-10-30T13:25:23.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a55b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a013f4a0 0x7f91a0154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:23.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a55b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a013f4a0 0x7f91a0154ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49480/0 (socket says 172.21.15.46:49480) 2023-10-30T13:25:23.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a55b4640 1 -- 172.21.15.46:0/2296895149 learned_addr learned my addr 172.21.15.46:0/2296895149 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:23.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f919ffff640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a00c6f60 0x7f91a00af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:23.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a4db3640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a00c6b80 0x7f91a013d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:23.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a55b4640 1 -- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a00c6b80 msgr2=0x7f91a013d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:23.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a55b4640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a00c6b80 0x7f91a013d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:23.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a55b4640 1 -- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a00c6f60 msgr2=0x7f91a00af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:23.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a55b4640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a00c6f60 0x7f91a00af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:23.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a55b4640 1 -- 172.21.15.46:0/2296895149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9194009420 con 0x7f91a013f4a0 2023-10-30T13:25:23.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a55b4640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a013f4a0 0x7f91a0154ac0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f919800c730 tx=0x7f919800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:23.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f919d7fa640 1 -- 172.21.15.46:0/2296895149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198003b80 con 0x7f91a013f4a0 2023-10-30T13:25:23.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f919d7fa640 1 -- 172.21.15.46:0/2296895149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9198003ce0 con 0x7f91a013f4a0 2023-10-30T13:25:23.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a7017640 1 -- 172.21.15.46:0/2296895149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a013deb0 con 0x7f91a013f4a0 2023-10-30T13:25:23.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f919d7fa640 1 -- 172.21.15.46:0/2296895149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919800aa30 con 0x7f91a013f4a0 2023-10-30T13:25:23.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a7017640 1 -- 172.21.15.46:0/2296895149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a013e470 con 0x7f91a013f4a0 2023-10-30T13:25:23.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.167+0000 7f91a7017640 1 -- 172.21.15.46:0/2296895149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a013e630 con 0x7f91a013f4a0 2023-10-30T13:25:23.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.171+0000 7f919d7fa640 1 -- 172.21.15.46:0/2296895149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f919800ab90 con 0x7f91a013f4a0 2023-10-30T13:25:23.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.171+0000 7f919d7fa640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f91840664d0 0x7f9184068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:23.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.171+0000 7f919d7fa640 1 -- 172.21.15.46:0/2296895149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5996+0+0 (secure 0 0 0) 0x7f9198083bf0 con 0x7f91a013f4a0 2023-10-30T13:25:23.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.171+0000 7f91a4db3640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f91840664d0 0x7f9184068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:23.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.171+0000 7f91a4db3640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f91840664d0 0x7f9184068990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f91940ad000 tx=0x7f9194008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:23.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.175+0000 7f919d7fa640 1 -- 172.21.15.46:0/2296895149 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9198050fe0 con 0x7f91a013f4a0 2023-10-30T13:25:23.312 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 199 2023-10-30T13:25:23.327 INFO:tasks.workunit.client.0.smithi046.stdout:Take first snapshot .snap/test1 2023-10-30T13:25:23.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.383+0000 7f91a7017640 1 -- 172.21.15.46:0/2296895149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91a004e170 con 0x7f91a013f4a0 2023-10-30T13:25:23.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.383+0000 7f919d7fa640 1 -- 172.21.15.46:0/2296895149 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5732 (secure 0 0 0) 0x7f9198054600 con 0x7f91a013f4a0 2023-10-30T13:25:23.387 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:23.388 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:18.713882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:resolve","state_seq":33,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:23.388 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 14 2023-10-30T13:25:23.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.383+0000 7f91a7017640 1 -- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f91840664d0 msgr2=0x7f9184068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:23.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.383+0000 7f91a7017640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f91840664d0 0x7f9184068990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f91940ad000 tx=0x7f9194008040 comp rx=0 tx=0).stop 2023-10-30T13:25:23.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.383+0000 7f91a7017640 1 -- 172.21.15.46:0/2296895149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a013f4a0 msgr2=0x7f91a0154ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:23.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.383+0000 7f91a7017640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a013f4a0 0x7f91a0154ac0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f919800c730 tx=0x7f919800cbf0 comp rx=0 tx=0).stop 2023-10-30T13:25:23.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.387+0000 7f91a7017640 1 -- 172.21.15.46:0/2296895149 shutdown_connections 2023-10-30T13:25:23.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.387+0000 7f91a7017640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f91840664d0 0x7f9184068990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:23.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.387+0000 7f91a7017640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a013f4a0 0x7f91a0154ac0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:23.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.387+0000 7f91a7017640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a00c6f60 0x7f91a00af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:23.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.387+0000 7f91a7017640 1 --2- 172.21.15.46:0/2296895149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a00c6b80 0x7f91a013d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:23.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.387+0000 7f91a7017640 1 -- 172.21.15.46:0/2296895149 >> 172.21.15.46:0/2296895149 conn(0x7f91a00c7910 msgr2=0x7f91a006aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:23.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.387+0000 7f91a7017640 1 -- 172.21.15.46:0/2296895149 shutdown_connections 2023-10-30T13:25:23.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:23.387+0000 7f91a7017640 1 -- 172.21.15.46:0/2296895149 wait complete. 2023-10-30T13:25:23.412 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:25:24.731 INFO:tasks.workunit.client.0.smithi046.stdout:Create dir 200 to 299 ... 2023-10-30T13:25:24.733 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 200 2023-10-30T13:25:24.741 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 201 2023-10-30T13:25:24.755 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 202 2023-10-30T13:25:24.768 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 203 2023-10-30T13:25:24.782 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 204 2023-10-30T13:25:24.795 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 205 2023-10-30T13:25:24.808 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 206 2023-10-30T13:25:24.822 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 207 2023-10-30T13:25:24.835 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 208 2023-10-30T13:25:24.848 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 209 2023-10-30T13:25:24.860 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 210 2023-10-30T13:25:24.873 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 211 2023-10-30T13:25:25.227 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 212 2023-10-30T13:25:25.413 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:25.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.551+0000 7f5388046640 1 Processor -- start 2023-10-30T13:25:25.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.551+0000 7f5388046640 1 -- start start 2023-10-30T13:25:25.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5388046640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f53800c05d0 0x7f538013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:25.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5388046640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53800c0a50 con 0x7f53800c6f60 2023-10-30T13:25:25.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5388046640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538006f6e0 con 0x7f53800c6b80 2023-10-30T13:25:25.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5388046640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538006f9c0 con 0x7f53800c05d0 2023-10-30T13:25:25.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f53865e3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f53800c6f60 0x7f53801566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49668/0 (socket says 172.21.15.46:49668) 2023-10-30T13:25:25.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f53865e3640 1 -- 172.21.15.46:0/236115958 learned_addr learned my addr 172.21.15.46:0/236115958 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:25:25.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f53855e1640 1 --2- 172.21.15.46:0/236115958 >> v2:172.21.15.154:3300/0 conn(0x7f53800c05d0 0x7f538013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:25.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443560079 0 0) 0x7f53800c0a50 con 0x7f53800c6f60 2023-10-30T13:25:25.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536c005740 con 0x7f53800c6f60 2023-10-30T13:25:25.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3704028566 0 0) 0x7f536c005740 con 0x7f53800c6f60 2023-10-30T13:25:25.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 >> v2:172.21.15.154:3300/0 conn(0x7f53800c05d0 msgr2=0x7f538013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:25.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 --2- 172.21.15.46:0/236115958 >> v2:172.21.15.154:3300/0 conn(0x7f53800c05d0 0x7f538013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:25.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 >> v1:172.21.15.154:6789/0 conn(0x7f53800c6b80 legacy=0x7f53800bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:25.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538006fca0 con 0x7f53800c6f60 2023-10-30T13:25:25.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f5378001a00 con 0x7f53800c6f60 2023-10-30T13:25:25.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f53855e1640 1 --2- 172.21.15.46:0/236115958 >> v2:172.21.15.154:3300/0 conn(0x7f53800c05d0 0x7f538013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:25:25.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 >> v1:172.21.15.46:6789/0 conn(0x7f53800c6f60 legacy=0x7f53801566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:25.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 --2- 172.21.15.46:0/236115958 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f536c005b90 0x7f536c005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:25.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 --2- 172.21.15.46:0/236115958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f536c0069e0 0x7f536c006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:25.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 --2- 172.21.15.46:0/236115958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 0x7f536c00dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:25.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538006f9c0 con 0x7f536c0073a0 2023-10-30T13:25:25.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c00e240 con 0x7f536c0069e0 2023-10-30T13:25:25.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536c00e400 con 0x7f536c005b90 2023-10-30T13:25:25.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f53865e3640 1 --2- 172.21.15.46:0/236115958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 0x7f536c00dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:25.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f5385de2640 1 --2- 172.21.15.46:0/236115958 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f536c005b90 0x7f536c005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:25.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f53855e1640 1 --2- 172.21.15.46:0/236115958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f536c0069e0 0x7f536c006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:25.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f53865e3640 1 -- 172.21.15.46:0/236115958 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f536c005b90 msgr2=0x7f536c005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:25.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f53865e3640 1 --2- 172.21.15.46:0/236115958 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f536c005b90 0x7f536c005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:25.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f53865e3640 1 -- 172.21.15.46:0/236115958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f536c0069e0 msgr2=0x7f536c006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:25.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f53865e3640 1 --2- 172.21.15.46:0/236115958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f536c0069e0 0x7f536c006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:25.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f53865e3640 1 -- 172.21.15.46:0/236115958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c005740 con 0x7f536c0073a0 2023-10-30T13:25:25.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.555+0000 7f53855e1640 1 --2- 172.21.15.46:0/236115958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f536c0069e0 0x7f536c006e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:25.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.559+0000 7f53865e3640 1 -- 172.21.15.46:0/236115958 reap_dead start 2023-10-30T13:25:25.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.559+0000 7f53865e3640 1 --2- 172.21.15.46:0/236115958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 0x7f536c00dba0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f537800a830 tx=0x7f537801d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef179d303269e92 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:25.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.559+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53780072d0 con 0x7f536c0073a0 2023-10-30T13:25:25.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.559+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53780a8070 con 0x7f536c0073a0 2023-10-30T13:25:25.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.559+0000 7f5384de0640 1 -- 172.21.15.46:0/236115958 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53780076b0 con 0x7f536c0073a0 2023-10-30T13:25:25.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.559+0000 7f5388046640 1 -- 172.21.15.46:0/236115958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 msgr2=0x7f536c00dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:25.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.559+0000 7f5388046640 1 --2- 172.21.15.46:0/236115958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 0x7f536c00dba0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f537800a830 tx=0x7f537801d340 comp rx=0 tx=0).stop 2023-10-30T13:25:25.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.559+0000 7f5388046640 1 -- 172.21.15.46:0/236115958 shutdown_connections 2023-10-30T13:25:25.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.559+0000 7f5388046640 1 --2- 172.21.15.46:0/236115958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 0x7f536c00dba0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:25.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.559+0000 7f5388046640 1 -- 172.21.15.46:0/236115958 >> 172.21.15.46:0/236115958 conn(0x7f53800c7910 msgr2=0x7f53800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:25.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.559+0000 7f5388046640 1 -- 172.21.15.46:0/236115958 shutdown_connections 2023-10-30T13:25:25.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.559+0000 7f5388046640 1 -- 172.21.15.46:0/236115958 wait complete. 2023-10-30T13:25:25.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5388046640 1 Processor -- start 2023-10-30T13:25:25.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5388046640 1 -- start start 2023-10-30T13:25:25.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5388046640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 0x7f53801ee000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:25.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5388046640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538013e460 0x7f53800c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:25.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5388046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53800c9b00 0x7f53800c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:25.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5388046640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53801ee540 con 0x7f536c0073a0 2023-10-30T13:25:25.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5388046640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53801ea3e0 con 0x7f538013e460 2023-10-30T13:25:25.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5388046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53801ea6a0 con 0x7f53800c9b00 2023-10-30T13:25:25.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5385de2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 0x7f53801ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:25.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5385de2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 0x7f53801ee000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49488/0 (socket says 172.21.15.46:49488) 2023-10-30T13:25:25.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5385de2640 1 -- 172.21.15.46:0/2123327879 learned_addr learned my addr 172.21.15.46:0/2123327879 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:25.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f53855e1640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538013e460 0x7f53800c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:25.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f53865e3640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53800c9b00 0x7f53800c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:25.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5385de2640 1 -- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53800c9b00 msgr2=0x7f53800c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:25.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5385de2640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53800c9b00 0x7f53800c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:25.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5385de2640 1 -- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538013e460 msgr2=0x7f53800c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:25.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5385de2640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538013e460 0x7f53800c0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:25.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5385de2640 1 -- 172.21.15.46:0/2123327879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53801ea980 con 0x7f536c0073a0 2023-10-30T13:25:25.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5385de2640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 0x7f53801ee000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f537c00f5f0 tx=0x7f537c00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:25.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f53767fc640 1 -- 172.21.15.46:0/2123327879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537c0052e0 con 0x7f536c0073a0 2023-10-30T13:25:25.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f53767fc640 1 -- 172.21.15.46:0/2123327879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537c005440 con 0x7f536c0073a0 2023-10-30T13:25:25.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f53767fc640 1 -- 172.21.15.46:0/2123327879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537c003820 con 0x7f536c0073a0 2023-10-30T13:25:25.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5388046640 1 -- 172.21.15.46:0/2123327879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53801eac60 con 0x7f536c0073a0 2023-10-30T13:25:25.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5388046640 1 -- 172.21.15.46:0/2123327879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53800c9300 con 0x7f536c0073a0 2023-10-30T13:25:25.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.563+0000 7f5388046640 1 -- 172.21.15.46:0/2123327879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538004e170 con 0x7f536c0073a0 2023-10-30T13:25:25.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.567+0000 7f53767fc640 1 -- 172.21.15.46:0/2123327879 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f537c003980 con 0x7f536c0073a0 2023-10-30T13:25:25.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.567+0000 7f53767fc640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f53640664d0 0x7f5364068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:25.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.567+0000 7f53767fc640 1 -- 172.21.15.46:0/2123327879 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f537c083fd0 con 0x7f536c0073a0 2023-10-30T13:25:25.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.567+0000 7f53855e1640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f53640664d0 0x7f5364068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:25.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.567+0000 7f53855e1640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f53640664d0 0x7f5364068990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5380072470 tx=0x7f5370008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:25.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.571+0000 7f53767fc640 1 -- 172.21.15.46:0/2123327879 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f537c051240 con 0x7f536c0073a0 2023-10-30T13:25:25.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.759+0000 7f5388046640 1 -- 172.21.15.46:0/2123327879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53800b0a50 con 0x7f536c0073a0 2023-10-30T13:25:25.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.759+0000 7f53767fc640 1 -- 172.21.15.46:0/2123327879 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f537c054860 con 0x7f536c0073a0 2023-10-30T13:25:25.765 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:25.765 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:24.713414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:25.765 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 15 2023-10-30T13:25:25.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.763+0000 7f5388046640 1 -- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f53640664d0 msgr2=0x7f5364068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.763+0000 7f5388046640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f53640664d0 0x7f5364068990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5380072470 tx=0x7f5370008040 comp rx=0 tx=0).stop 2023-10-30T13:25:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.763+0000 7f5388046640 1 -- 172.21.15.46:0/2123327879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 msgr2=0x7f53801ee000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.763+0000 7f5388046640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 0x7f53801ee000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f537c00f5f0 tx=0x7f537c00fab0 comp rx=0 tx=0).stop 2023-10-30T13:25:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.763+0000 7f5388046640 1 -- 172.21.15.46:0/2123327879 shutdown_connections 2023-10-30T13:25:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.763+0000 7f5388046640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f53640664d0 0x7f5364068990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.763+0000 7f5388046640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53800c9b00 0x7f53800c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.763+0000 7f5388046640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538013e460 0x7f53800c0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.763+0000 7f5388046640 1 --2- 172.21.15.46:0/2123327879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f536c0073a0 0x7f53801ee000 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.763+0000 7f5388046640 1 -- 172.21.15.46:0/2123327879 >> 172.21.15.46:0/2123327879 conn(0x7f53800c7910 msgr2=0x7f53800bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.763+0000 7f5388046640 1 -- 172.21.15.46:0/2123327879 shutdown_connections 2023-10-30T13:25:25.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:25.763+0000 7f5388046640 1 -- 172.21.15.46:0/2123327879 wait complete. 2023-10-30T13:25:25.786 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:25:25.789 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 213 2023-10-30T13:25:25.798 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 214 2023-10-30T13:25:25.806 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 215 2023-10-30T13:25:25.814 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 216 2023-10-30T13:25:26.420 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 217 2023-10-30T13:25:26.430 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 218 2023-10-30T13:25:26.691 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 219 2023-10-30T13:25:26.701 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 220 2023-10-30T13:25:26.710 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 221 2023-10-30T13:25:27.419 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 222 2023-10-30T13:25:27.701 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 223 2023-10-30T13:25:27.713 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 224 2023-10-30T13:25:27.724 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 225 2023-10-30T13:25:27.735 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 226 2023-10-30T13:25:27.746 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 227 2023-10-30T13:25:27.756 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 228 2023-10-30T13:25:27.765 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 229 2023-10-30T13:25:27.774 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 230 2023-10-30T13:25:27.783 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 231 2023-10-30T13:25:27.787 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:27.793 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 232 2023-10-30T13:25:27.803 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 233 2023-10-30T13:25:27.813 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 234 2023-10-30T13:25:27.823 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 235 2023-10-30T13:25:27.833 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 236 2023-10-30T13:25:27.842 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 237 2023-10-30T13:25:27.852 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 238 2023-10-30T13:25:27.861 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 239 2023-10-30T13:25:27.869 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 240 2023-10-30T13:25:27.878 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 241 2023-10-30T13:25:27.887 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 242 2023-10-30T13:25:27.897 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 243 2023-10-30T13:25:27.907 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 244 2023-10-30T13:25:27.920 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 245 2023-10-30T13:25:27.934 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 246 2023-10-30T13:25:27.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.943+0000 7fc0dcbc5640 1 Processor -- start 2023-10-30T13:25:27.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.943+0000 7fc0dcbc5640 1 -- start start 2023-10-30T13:25:27.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0dcbc5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc0d81538a0 0x7fc0d815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:27.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0dcbc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0d80cf160 0x7fc0d80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:27.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0dcbc5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d80cb800 con 0x7fc0d80ca0e0 2023-10-30T13:25:27.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0dcbc5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d815dc30 con 0x7fc0d81538a0 2023-10-30T13:25:27.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0dcbc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80cb330 con 0x7fc0d80cf160 2023-10-30T13:25:27.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0d659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc0d81538a0 0x7fc0d815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:27.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0d659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc0d81538a0 0x7fc0d815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46964/0 (socket says 172.21.15.46:46964) 2023-10-30T13:25:27.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0d659c640 1 -- 172.21.15.46:0/3686954061 learned_addr learned my addr 172.21.15.46:0/3686954061 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:27.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0d5d9b640 1 --1- 172.21.15.46:0/3686954061 >> v1:172.21.15.46:6789/0 conn(0x7fc0d80ca0e0 0x7fc0d815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49456/0 (socket says 172.21.15.46:49456) 2023-10-30T13:25:27.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0d6d9d640 1 --2- 172.21.15.46:0/3686954061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0d80cf160 0x7fc0d80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:27.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0d659c640 1 -- 172.21.15.46:0/3686954061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0d80cf160 msgr2=0x7fc0d80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:27.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0d659c640 1 --2- 172.21.15.46:0/3686954061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0d80cf160 0x7fc0d80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:27.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0d659c640 1 -- 172.21.15.46:0/3686954061 >> v1:172.21.15.46:6789/0 conn(0x7fc0d80ca0e0 legacy=0x7fc0d815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:27.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.947+0000 7fc0d659c640 1 -- 172.21.15.46:0/3686954061 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d80cad70 con 0x7fc0d81538a0 2023-10-30T13:25:27.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0d659c640 1 --2- 172.21.15.46:0/3686954061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d81538a0 0x7fc0d815d6f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8009870 tx=0x7fc0c801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa2d574b8506b2d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:27.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0d559a640 1 -- 172.21.15.46:0/3686954061 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80a8a30 con 0x7fc0d81538a0 2023-10-30T13:25:27.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0d559a640 1 -- 172.21.15.46:0/3686954061 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c80a8b90 con 0x7fc0d81538a0 2023-10-30T13:25:27.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0d559a640 1 -- 172.21.15.46:0/3686954061 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80b0440 con 0x7fc0d81538a0 2023-10-30T13:25:27.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/3686954061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d81538a0 msgr2=0x7fc0d815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0dcbc5640 1 --2- 172.21.15.46:0/3686954061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d81538a0 0x7fc0d815d6f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8009870 tx=0x7fc0c801ed30 comp rx=0 tx=0).stop 2023-10-30T13:25:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/3686954061 shutdown_connections 2023-10-30T13:25:27.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0dcbc5640 1 --2- 172.21.15.46:0/3686954061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0d80cf160 0x7fc0d80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0dcbc5640 1 --2- 172.21.15.46:0/3686954061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d81538a0 0x7fc0d815d6f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/3686954061 >> 172.21.15.46:0/3686954061 conn(0x7fc0d806bfe0 msgr2=0x7fc0d806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/3686954061 shutdown_connections 2023-10-30T13:25:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/3686954061 wait complete. 2023-10-30T13:25:27.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0dcbc5640 1 Processor -- start 2023-10-30T13:25:27.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.951+0000 7fc0dcbc5640 1 -- start start 2023-10-30T13:25:27.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0dcbc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80ca0e0 0x7fc0d81e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:27.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0dcbc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0d80cf160 0x7fc0d815b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:27.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0dcbc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0d81538a0 0x7fc0d815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:27.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0dcbc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d814d4f0 con 0x7fc0d80ca0e0 2023-10-30T13:25:27.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0dcbc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d81e4d40 con 0x7fc0d80cf160 2023-10-30T13:25:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0dcbc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d815b2a0 con 0x7fc0d81538a0 2023-10-30T13:25:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0d659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80ca0e0 0x7fc0d81e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0d659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80ca0e0 0x7fc0d81e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46966/0 (socket says 172.21.15.46:46966) 2023-10-30T13:25:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0d659c640 1 -- 172.21.15.46:0/4060792657 learned_addr learned my addr 172.21.15.46:0/4060792657 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:27.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0d6d9d640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0d81538a0 0x7fc0d815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0d5d9b640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0d80cf160 0x7fc0d815b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0d659c640 1 -- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0d81538a0 msgr2=0x7fc0d815bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0d659c640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0d81538a0 0x7fc0d815bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:27.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0d659c640 1 -- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0d80cf160 msgr2=0x7fc0d815b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0d659c640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0d80cf160 0x7fc0d815b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0d659c640 1 -- 172.21.15.46:0/4060792657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c80094d0 con 0x7fc0d80ca0e0 2023-10-30T13:25:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0d659c640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80ca0e0 0x7fc0d81e4090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80b3040 tx=0x7fc0c801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0c6ffd640 1 -- 172.21.15.46:0/4060792657 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80ba600 con 0x7fc0d80ca0e0 2023-10-30T13:25:27.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0c6ffd640 1 -- 172.21.15.46:0/4060792657 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c80ba780 con 0x7fc0d80ca0e0 2023-10-30T13:25:27.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0c6ffd640 1 -- 172.21.15.46:0/4060792657 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80bab30 con 0x7fc0d80ca0e0 2023-10-30T13:25:27.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/4060792657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d80cef10 con 0x7fc0d80ca0e0 2023-10-30T13:25:27.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/4060792657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d80c50b0 con 0x7fc0d80ca0e0 2023-10-30T13:25:27.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.955+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/4060792657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d80b2a50 con 0x7fc0d80ca0e0 2023-10-30T13:25:27.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.959+0000 7fc0c6ffd640 1 -- 172.21.15.46:0/4060792657 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc0c80bac90 con 0x7fc0d80ca0e0 2023-10-30T13:25:27.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.959+0000 7fc0c6ffd640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0b80664a0 0x7fc0b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:27.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.959+0000 7fc0c6ffd640 1 -- 172.21.15.46:0/4060792657 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fc0c81224e0 con 0x7fc0d80ca0e0 2023-10-30T13:25:27.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.959+0000 7fc0d5d9b640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0b80664a0 0x7fc0b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:27.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.959+0000 7fc0d5d9b640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0b80664a0 0x7fc0b8068960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc0d80c4660 tx=0x7fc0cc007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:27.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:27.963+0000 7fc0c6ffd640 1 -- 172.21.15.46:0/4060792657 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc0c80ef7e0 con 0x7fc0d80ca0e0 2023-10-30T13:25:28.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.151+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/4060792657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0d8151940 con 0x7fc0d80ca0e0 2023-10-30T13:25:28.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.151+0000 7fc0c6ffd640 1 -- 172.21.15.46:0/4060792657 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5734 (secure 0 0 0) 0x7fc0c80f2e00 con 0x7fc0d80ca0e0 2023-10-30T13:25:28.156 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:28.156 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:24.713414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:28.158 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 15 2023-10-30T13:25:28.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.155+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0b80664a0 msgr2=0x7fc0b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:28.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.155+0000 7fc0dcbc5640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0b80664a0 0x7fc0b8068960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc0d80c4660 tx=0x7fc0cc007c90 comp rx=0 tx=0).stop 2023-10-30T13:25:28.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.155+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/4060792657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80ca0e0 msgr2=0x7fc0d81e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:28.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.155+0000 7fc0dcbc5640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80ca0e0 0x7fc0d81e4090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80b3040 tx=0x7fc0c801bad0 comp rx=0 tx=0).stop 2023-10-30T13:25:28.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.155+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/4060792657 shutdown_connections 2023-10-30T13:25:28.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.155+0000 7fc0dcbc5640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0b80664a0 0x7fc0b8068960 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:28.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.155+0000 7fc0dcbc5640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0d81538a0 0x7fc0d815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:28.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.155+0000 7fc0dcbc5640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0d80cf160 0x7fc0d815b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:28.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.155+0000 7fc0dcbc5640 1 --2- 172.21.15.46:0/4060792657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80ca0e0 0x7fc0d81e4090 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:28.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.155+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/4060792657 >> 172.21.15.46:0/4060792657 conn(0x7fc0d806bfe0 msgr2=0x7fc0d8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:28.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.155+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/4060792657 shutdown_connections 2023-10-30T13:25:28.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:28.155+0000 7fc0dcbc5640 1 -- 172.21.15.46:0/4060792657 wait complete. 2023-10-30T13:25:28.177 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4259": { "addr": "172.21.15.46:6833/875791418", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 875791418, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 875791418, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4259, "incarnation": 13, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:reconnect", "state_seq": 35 }, "gid_4288": { "addr": "172.21.15.154:6835/3109262047", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3109262047, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3109262047, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4306": { "addr": "172.21.15.46:6836/275749119", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 275749119, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 275749119, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 23, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:25:24.713414+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4288, "mds_2": 4259 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:25:28.411 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 247 2023-10-30T13:25:28.887 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 248 2023-10-30T13:25:28.902 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 249 2023-10-30T13:25:28.916 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 250 2023-10-30T13:25:28.931 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 251 2023-10-30T13:25:28.940 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 252 2023-10-30T13:25:28.950 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 253 2023-10-30T13:25:28.959 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 254 2023-10-30T13:25:28.969 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 255 2023-10-30T13:25:28.978 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 256 2023-10-30T13:25:28.988 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 257 2023-10-30T13:25:28.997 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 258 2023-10-30T13:25:29.007 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 259 2023-10-30T13:25:29.017 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 260 2023-10-30T13:25:29.028 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 261 2023-10-30T13:25:29.038 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 262 2023-10-30T13:25:29.047 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 263 2023-10-30T13:25:29.057 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 264 2023-10-30T13:25:29.067 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 265 2023-10-30T13:25:29.077 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 266 2023-10-30T13:25:29.086 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 267 2023-10-30T13:25:29.096 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 268 2023-10-30T13:25:29.105 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 269 2023-10-30T13:25:29.114 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 270 2023-10-30T13:25:29.124 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 271 2023-10-30T13:25:29.133 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 272 2023-10-30T13:25:29.143 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 273 2023-10-30T13:25:29.152 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 274 2023-10-30T13:25:29.161 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 275 2023-10-30T13:25:29.170 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 276 2023-10-30T13:25:29.435 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 277 2023-10-30T13:25:29.784 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 278 2023-10-30T13:25:30.043 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 279 2023-10-30T13:25:30.060 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 280 2023-10-30T13:25:30.078 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 281 2023-10-30T13:25:30.178 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.319+0000 7f6f2d13e640 1 Processor -- start 2023-10-30T13:25:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.319+0000 7f6f2d13e640 1 -- start start 2023-10-30T13:25:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.319+0000 7f6f2d13e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6f280c05d0 0x7f6f2813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.319+0000 7f6f2d13e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f280c0a50 con 0x7f6f280c6b80 2023-10-30T13:25:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.319+0000 7f6f2d13e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2806f6e0 con 0x7f6f280c6f60 2023-10-30T13:25:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.319+0000 7f6f2d13e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f2806f9c0 con 0x7f6f280c05d0 2023-10-30T13:25:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.319+0000 7f6f26d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6f280c6b80 0x7f6f280bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49466/0 (socket says 172.21.15.46:49466) 2023-10-30T13:25:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.319+0000 7f6f26d9d640 1 -- 172.21.15.46:0/2508981152 learned_addr learned my addr 172.21.15.46:0/2508981152 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:25:30.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f2659c640 1 --2- 172.21.15.46:0/2508981152 >> v2:172.21.15.154:3300/0 conn(0x7f6f280c05d0 0x7f6f2813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:30.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784069886 0 0) 0x7f6f280c0a50 con 0x7f6f280c6b80 2023-10-30T13:25:30.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f10003530 con 0x7f6f280c6b80 2023-10-30T13:25:30.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 530140461 0 0) 0x7f6f10003530 con 0x7f6f280c6b80 2023-10-30T13:25:30.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 >> v2:172.21.15.154:3300/0 conn(0x7f6f280c05d0 msgr2=0x7f6f2813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:30.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 --2- 172.21.15.46:0/2508981152 >> v2:172.21.15.154:3300/0 conn(0x7f6f280c05d0 0x7f6f2813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:30.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 >> v1:172.21.15.154:6789/0 conn(0x7f6f280c6f60 legacy=0x7f6f281566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:30.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2806fca0 con 0x7f6f280c6b80 2023-10-30T13:25:30.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f6f18001680 con 0x7f6f280c6b80 2023-10-30T13:25:30.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 >> v1:172.21.15.46:6789/0 conn(0x7f6f280c6b80 legacy=0x7f6f280bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:30.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 --2- 172.21.15.46:0/2508981152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 0x7f6f10003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:30.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 --2- 172.21.15.46:0/2508981152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f100047d0 0x7f6f10004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:30.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 --2- 172.21.15.46:0/2508981152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f10005190 0x7f6f1000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:30.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2806f9c0 con 0x7f6f10003980 2023-10-30T13:25:30.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1000c030 con 0x7f6f100047d0 2023-10-30T13:25:30.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1000c1f0 con 0x7f6f10005190 2023-10-30T13:25:30.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f26d9d640 1 --2- 172.21.15.46:0/2508981152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 0x7f6f10003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:30.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f2659c640 1 --2- 172.21.15.46:0/2508981152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f100047d0 0x7f6f10004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:30.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f2759e640 1 --2- 172.21.15.46:0/2508981152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f10005190 0x7f6f1000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:30.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f26d9d640 1 -- 172.21.15.46:0/2508981152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f10005190 msgr2=0x7f6f1000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:30.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f26d9d640 1 --2- 172.21.15.46:0/2508981152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f10005190 0x7f6f1000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:30.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f26d9d640 1 -- 172.21.15.46:0/2508981152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f100047d0 msgr2=0x7f6f10004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:30.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f26d9d640 1 --2- 172.21.15.46:0/2508981152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f100047d0 0x7f6f10004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:30.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f26d9d640 1 -- 172.21.15.46:0/2508981152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f10003530 con 0x7f6f10003980 2023-10-30T13:25:30.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.323+0000 7f6f2759e640 1 -- 172.21.15.46:0/2508981152 reap_dead start 2023-10-30T13:25:30.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f26d9d640 1 --2- 172.21.15.46:0/2508981152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 0x7f6f10003da0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c240 tx=0x7f6f1801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ed4cb4e2db9633 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:30.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f18005ae0 con 0x7f6f10003980 2023-10-30T13:25:30.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f18005c40 con 0x7f6f10003980 2023-10-30T13:25:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f25d9b640 1 -- 172.21.15.46:0/2508981152 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f180a5a70 con 0x7f6f10003980 2023-10-30T13:25:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f2d13e640 1 -- 172.21.15.46:0/2508981152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 msgr2=0x7f6f10003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f2d13e640 1 --2- 172.21.15.46:0/2508981152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 0x7f6f10003da0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c240 tx=0x7f6f1801cc40 comp rx=0 tx=0).stop 2023-10-30T13:25:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f2d13e640 1 -- 172.21.15.46:0/2508981152 shutdown_connections 2023-10-30T13:25:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f2d13e640 1 --2- 172.21.15.46:0/2508981152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 0x7f6f10003da0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f2d13e640 1 -- 172.21.15.46:0/2508981152 >> 172.21.15.46:0/2508981152 conn(0x7f6f280c7910 msgr2=0x7f6f280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:30.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f2d13e640 1 -- 172.21.15.46:0/2508981152 shutdown_connections 2023-10-30T13:25:30.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f2d13e640 1 -- 172.21.15.46:0/2508981152 wait complete. 2023-10-30T13:25:30.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f2d13e640 1 Processor -- start 2023-10-30T13:25:30.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.327+0000 7f6f2d13e640 1 -- start start 2023-10-30T13:25:30.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f2d13e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 0x7f6f2813b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:30.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f2d13e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f280ca7e0 0x7f6f2813c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:30.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f2d13e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f281382e0 0x7f6f28138760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:30.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f2d13e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2813a800 con 0x7f6f10003980 2023-10-30T13:25:30.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f2d13e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f281f23b0 con 0x7f6f281382e0 2023-10-30T13:25:30.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f2d13e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f28139310 con 0x7f6f280ca7e0 2023-10-30T13:25:30.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f26d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 0x7f6f2813b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:30.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f26d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 0x7f6f2813b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46988/0 (socket says 172.21.15.46:46988) 2023-10-30T13:25:30.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f26d9d640 1 -- 172.21.15.46:0/3063529513 learned_addr learned my addr 172.21.15.46:0/3063529513 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:30.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f2659c640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f280ca7e0 0x7f6f2813c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:30.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f2759e640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f281382e0 0x7f6f28138760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:30.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f26d9d640 1 -- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f280ca7e0 msgr2=0x7f6f2813c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:30.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f26d9d640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f280ca7e0 0x7f6f2813c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:30.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f26d9d640 1 -- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f281382e0 msgr2=0x7f6f28138760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:30.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f26d9d640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f281382e0 0x7f6f28138760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:30.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f26d9d640 1 -- 172.21.15.46:0/3063529513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2813b2a0 con 0x7f6f10003980 2023-10-30T13:25:30.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f26d9d640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 0x7f6f2813b680 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c240 tx=0x7f6f10005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:30.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f0f7fe640 1 -- 172.21.15.46:0/3063529513 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f180b4070 con 0x7f6f10003980 2023-10-30T13:25:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f0f7fe640 1 -- 172.21.15.46:0/3063529513 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f18001650 con 0x7f6f10003980 2023-10-30T13:25:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f0f7fe640 1 -- 172.21.15.46:0/3063529513 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f180a5a50 con 0x7f6f10003980 2023-10-30T13:25:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f2d13e640 1 -- 172.21.15.46:0/3063529513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f2813bbc0 con 0x7f6f10003980 2023-10-30T13:25:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f2d13e640 1 -- 172.21.15.46:0/3063529513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f28139840 con 0x7f6f10003980 2023-10-30T13:25:30.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f2d13e640 1 -- 172.21.15.46:0/3063529513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f2804e170 con 0x7f6f10003980 2023-10-30T13:25:30.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.331+0000 7f6f0f7fe640 1 -- 172.21.15.46:0/3063529513 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6f180a5bb0 con 0x7f6f10003980 2023-10-30T13:25:30.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.335+0000 7f6f0f7fe640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6f140664d0 0x7f6f14068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:30.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.335+0000 7f6f0f7fe640 1 -- 172.21.15.46:0/3063529513 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f6f1811d920 con 0x7f6f10003980 2023-10-30T13:25:30.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.335+0000 7f6f2659c640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6f140664d0 0x7f6f14068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:30.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.335+0000 7f6f2659c640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6f140664d0 0x7f6f14068990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6f28159590 tx=0x7f6f1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:30.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.339+0000 7f6f0f7fe640 1 -- 172.21.15.46:0/3063529513 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6f180eaca0 con 0x7f6f10003980 2023-10-30T13:25:30.430 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 282 2023-10-30T13:25:30.451 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 283 2023-10-30T13:25:30.466 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 284 2023-10-30T13:25:30.480 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 285 2023-10-30T13:25:30.494 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 286 2023-10-30T13:25:30.509 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 287 2023-10-30T13:25:30.523 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 288 2023-10-30T13:25:30.537 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 289 2023-10-30T13:25:30.551 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 290 2023-10-30T13:25:30.565 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 291 2023-10-30T13:25:30.579 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 292 2023-10-30T13:25:30.594 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 293 2023-10-30T13:25:30.608 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 294 2023-10-30T13:25:30.622 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 295 2023-10-30T13:25:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.623+0000 7f6f2d13e640 1 -- 172.21.15.46:0/3063529513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f2815a220 con 0x7f6f10003980 2023-10-30T13:25:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.623+0000 7f6f0f7fe640 1 -- 172.21.15.46:0/3063529513 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f6f1811d030 con 0x7f6f10003980 2023-10-30T13:25:30.629 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:30.629 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:24.713414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:30.630 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 15 2023-10-30T13:25:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.627+0000 7f6f2d13e640 1 -- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6f140664d0 msgr2=0x7f6f14068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.627+0000 7f6f2d13e640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6f140664d0 0x7f6f14068990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6f28159590 tx=0x7f6f1c008040 comp rx=0 tx=0).stop 2023-10-30T13:25:30.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.627+0000 7f6f2d13e640 1 -- 172.21.15.46:0/3063529513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 msgr2=0x7f6f2813b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:30.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.627+0000 7f6f2d13e640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 0x7f6f2813b680 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c240 tx=0x7f6f10005800 comp rx=0 tx=0).stop 2023-10-30T13:25:30.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.627+0000 7f6f2d13e640 1 -- 172.21.15.46:0/3063529513 shutdown_connections 2023-10-30T13:25:30.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.627+0000 7f6f2d13e640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6f140664d0 0x7f6f14068990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:30.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.627+0000 7f6f2d13e640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f281382e0 0x7f6f28138760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:30.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.627+0000 7f6f2d13e640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f280ca7e0 0x7f6f2813c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:30.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.627+0000 7f6f2d13e640 1 --2- 172.21.15.46:0/3063529513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f10003980 0x7f6f2813b680 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:30.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.627+0000 7f6f2d13e640 1 -- 172.21.15.46:0/3063529513 >> 172.21.15.46:0/3063529513 conn(0x7f6f280c7910 msgr2=0x7f6f280c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:30.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.627+0000 7f6f2d13e640 1 -- 172.21.15.46:0/3063529513 shutdown_connections 2023-10-30T13:25:30.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:30.627+0000 7f6f2d13e640 1 -- 172.21.15.46:0/3063529513 wait complete. 2023-10-30T13:25:30.637 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 296 2023-10-30T13:25:30.652 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 297 2023-10-30T13:25:30.656 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:25:30.664 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 298 2023-10-30T13:25:30.678 INFO:tasks.workunit.client.0.smithi046.stdout: create dir 299 2023-10-30T13:25:30.690 INFO:tasks.workunit.client.0.smithi046.stdout:Create a snapshot in every first level dir ... 2023-10-30T13:25:30.749 INFO:tasks.workunit.client.0.smithi046.stdout: create 100/.snap/snap-subdir-test 2023-10-30T13:25:32.658 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:32.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.807+0000 7fc1636d6640 1 Processor -- start 2023-10-30T13:25:32.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc1636d6640 1 -- start start 2023-10-30T13:25:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc1636d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc15c06e470 0x7fc15c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc1636d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc15c0c8490 0x7fc15c0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc1636d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc15c06afa0 con 0x7fc15c153d70 2023-10-30T13:25:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc1636d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc15c06d780 con 0x7fc15c06e470 2023-10-30T13:25:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc1636d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c154150 con 0x7fc15c0c8490 2023-10-30T13:25:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc161472640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc15c153d70 0x7fc15c0cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49468/0 (socket says 172.21.15.46:49468) 2023-10-30T13:25:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc161472640 1 -- 172.21.15.46:0/2933221011 learned_addr learned my addr 172.21.15.46:0/2933221011 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:25:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc161c73640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc15c0c8490 0x7fc15c0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc160c71640 1 --2- 172.21.15.46:0/2933221011 >> v2:172.21.15.154:3300/0 conn(0x7fc15c06e470 0x7fc15c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674965106 0 0) 0x7fc15c06afa0 con 0x7fc15c153d70 2023-10-30T13:25:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc148003530 con 0x7fc15c153d70 2023-10-30T13:25:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2608761383 0 0) 0x7fc148003530 con 0x7fc15c153d70 2023-10-30T13:25:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc15c0c8490 msgr2=0x7fc15c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc15c0c8490 0x7fc15c0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 >> v2:172.21.15.154:3300/0 conn(0x7fc15c06e470 msgr2=0x7fc15c06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 --2- 172.21.15.46:0/2933221011 >> v2:172.21.15.154:3300/0 conn(0x7fc15c06e470 0x7fc15c06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c06e030 con 0x7fc15c153d70 2023-10-30T13:25:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fc158001680 con 0x7fc15c153d70 2023-10-30T13:25:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 >> v1:172.21.15.46:6789/0 conn(0x7fc15c153d70 legacy=0x7fc15c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc15c0c8490 0x7fc148003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc148003f00 0x7fc1480047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc148004d10 0x7fc148005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c06d780 con 0x7fc148004d10 2023-10-30T13:25:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c154150 con 0x7fc148003f00 2023-10-30T13:25:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14800b990 con 0x7fc15c0c8490 2023-10-30T13:25:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc161c73640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc148004d10 0x7fc148005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc160c71640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc148003f00 0x7fc1480047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc161472640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc15c0c8490 0x7fc148003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc161c73640 1 -- 172.21.15.46:0/2933221011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc15c0c8490 msgr2=0x7fc148003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc161c73640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc15c0c8490 0x7fc148003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc161c73640 1 -- 172.21.15.46:0/2933221011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc148003f00 msgr2=0x7fc1480047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc161c73640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc148003f00 0x7fc1480047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc161c73640 1 -- 172.21.15.46:0/2933221011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148003530 con 0x7fc148004d10 2023-10-30T13:25:32.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.811+0000 7fc160c71640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc148003f00 0x7fc1480047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:32.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.815+0000 7fc161c73640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc148004d10 0x7fc148005190 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc154009fc0 tx=0x7fc15401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1397b5544bd7c42 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.815+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1540a5c70 con 0x7fc148004d10 2023-10-30T13:25:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.815+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15400f040 con 0x7fc148004d10 2023-10-30T13:25:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.815+0000 7fc153fff640 1 -- 172.21.15.46:0/2933221011 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1540ad440 con 0x7fc148004d10 2023-10-30T13:25:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.815+0000 7fc1636d6640 1 -- 172.21.15.46:0/2933221011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc148004d10 msgr2=0x7fc148005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:32.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.815+0000 7fc1636d6640 1 --2- 172.21.15.46:0/2933221011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc148004d10 0x7fc148005190 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc154009fc0 tx=0x7fc15401cc00 comp rx=0 tx=0).stop 2023-10-30T13:25:32.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.815+0000 7fc161c73640 1 -- 172.21.15.46:0/2933221011 reap_dead start 2023-10-30T13:25:32.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.815+0000 7fc1636d6640 1 -- 172.21.15.46:0/2933221011 shutdown_connections 2023-10-30T13:25:32.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.815+0000 7fc1636d6640 1 -- 172.21.15.46:0/2933221011 >> 172.21.15.46:0/2933221011 conn(0x7fc15c071ea0 msgr2=0x7fc15c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:32.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.815+0000 7fc1636d6640 1 -- 172.21.15.46:0/2933221011 shutdown_connections 2023-10-30T13:25:32.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.815+0000 7fc1636d6640 1 -- 172.21.15.46:0/2933221011 wait complete. 2023-10-30T13:25:32.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc1636d6640 1 Processor -- start 2023-10-30T13:25:32.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc1636d6640 1 -- start start 2023-10-30T13:25:32.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc1636d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc148004d10 0x7fc15c14bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:32.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc1636d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc15c1472f0 0x7fc15c142890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:32.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc1636d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc15c15d910 0x7fc15c142dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:32.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc1636d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c148920 con 0x7fc15c1472f0 2023-10-30T13:25:32.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc1636d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c147830 con 0x7fc15c15d910 2023-10-30T13:25:32.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc1636d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c147af0 con 0x7fc148004d10 2023-10-30T13:25:32.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc160c71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc15c1472f0 0x7fc15c142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:32.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc160c71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc15c1472f0 0x7fc15c142890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47012/0 (socket says 172.21.15.46:47012) 2023-10-30T13:25:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc160c71640 1 -- 172.21.15.46:0/2187496071 learned_addr learned my addr 172.21.15.46:0/2187496071 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc161472640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc148004d10 0x7fc15c14bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc161c73640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc15c15d910 0x7fc15c142dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc160c71640 1 -- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc148004d10 msgr2=0x7fc15c14bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc160c71640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc148004d10 0x7fc15c14bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:32.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc160c71640 1 -- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc15c15d910 msgr2=0x7fc15c142dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:32.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc160c71640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc15c15d910 0x7fc15c142dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:32.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc160c71640 1 -- 172.21.15.46:0/2187496071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c147dd0 con 0x7fc15c1472f0 2023-10-30T13:25:32.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc161472640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc148004d10 0x7fc15c14bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:32.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc160c71640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc15c1472f0 0x7fc15c142890 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0115f0 tx=0x7fc14c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:32.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc151ffb640 1 -- 172.21.15.46:0/2187496071 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c004220 con 0x7fc15c1472f0 2023-10-30T13:25:32.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc151ffb640 1 -- 172.21.15.46:0/2187496071 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14c004380 con 0x7fc15c1472f0 2023-10-30T13:25:32.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc151ffb640 1 -- 172.21.15.46:0/2187496071 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c00f910 con 0x7fc15c1472f0 2023-10-30T13:25:32.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc1636d6640 1 -- 172.21.15.46:0/2187496071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15c1497c0 con 0x7fc15c1472f0 2023-10-30T13:25:32.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.819+0000 7fc1636d6640 1 -- 172.21.15.46:0/2187496071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15c149d80 con 0x7fc15c1472f0 2023-10-30T13:25:32.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.823+0000 7fc1636d6640 1 -- 172.21.15.46:0/2187496071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15c04e170 con 0x7fc15c1472f0 2023-10-30T13:25:32.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.823+0000 7fc151ffb640 1 -- 172.21.15.46:0/2187496071 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc14c00fa70 con 0x7fc15c1472f0 2023-10-30T13:25:32.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.823+0000 7fc151ffb640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc1400664d0 0x7fc140068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:32.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.823+0000 7fc151ffb640 1 -- 172.21.15.46:0/2187496071 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fc14c0845c0 con 0x7fc15c1472f0 2023-10-30T13:25:32.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.823+0000 7fc161472640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc1400664d0 0x7fc140068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:32.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.823+0000 7fc161472640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc1400664d0 0x7fc140068990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc1580016c0 tx=0x7fc158009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:32.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:32.827+0000 7fc151ffb640 1 -- 172.21.15.46:0/2187496071 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc14c051830 con 0x7fc15c1472f0 2023-10-30T13:25:33.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 -- 172.21.15.46:0/2187496071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc15c14de00 con 0x7fc15c1472f0 2023-10-30T13:25:33.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc151ffb640 1 -- 172.21.15.46:0/2187496071 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5731 (secure 0 0 0) 0x7fc14c022090 con 0x7fc15c1472f0 2023-10-30T13:25:33.023 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:33.023 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:29.892206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:33.024 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 16 2023-10-30T13:25:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 -- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc1400664d0 msgr2=0x7fc140068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc1400664d0 0x7fc140068990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc1580016c0 tx=0x7fc158009040 comp rx=0 tx=0).stop 2023-10-30T13:25:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 -- 172.21.15.46:0/2187496071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc15c1472f0 msgr2=0x7fc15c142890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc15c1472f0 0x7fc15c142890 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0115f0 tx=0x7fc14c011ab0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 -- 172.21.15.46:0/2187496071 shutdown_connections 2023-10-30T13:25:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc1400664d0 0x7fc140068990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc15c15d910 0x7fc15c142dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc15c1472f0 0x7fc15c142890 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 --2- 172.21.15.46:0/2187496071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc148004d10 0x7fc15c14bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 -- 172.21.15.46:0/2187496071 >> 172.21.15.46:0/2187496071 conn(0x7fc15c071ea0 msgr2=0x7fc15c0c7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:33.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 -- 172.21.15.46:0/2187496071 shutdown_connections 2023-10-30T13:25:33.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:33.019+0000 7fc1636d6640 1 -- 172.21.15.46:0/2187496071 wait complete. 2023-10-30T13:25:33.044 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:25:35.045 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:35.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.191+0000 7f6b1053e640 1 Processor -- start 2023-10-30T13:25:35.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.191+0000 7f6b1053e640 1 -- start start 2023-10-30T13:25:35.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b1053e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b080c05d0 0x7f6b0813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b1053e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b080c0a50 con 0x7f6b080c6b80 2023-10-30T13:25:35.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b1053e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0806f6e0 con 0x7f6b080c6f60 2023-10-30T13:25:35.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b1053e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0806f9c0 con 0x7f6b080c05d0 2023-10-30T13:25:35.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0e2da640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6b080c6b80 0x7f6b080bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49472/0 (socket says 172.21.15.46:49472) 2023-10-30T13:25:35.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0e2da640 1 -- 172.21.15.46:0/2775502365 learned_addr learned my addr 172.21.15.46:0/2775502365 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:25:35.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0dad9640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b080c05d0 0x7f6b0813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615390448 0 0) 0x7f6b080c0a50 con 0x7f6b080c6b80 2023-10-30T13:25:35.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6af4005740 con 0x7f6b080c6b80 2023-10-30T13:25:35.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1841582759 0 0) 0x7f6af4005740 con 0x7f6b080c6b80 2023-10-30T13:25:35.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b080c05d0 msgr2=0x7f6b0813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:35.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b080c05d0 0x7f6b0813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 >> v1:172.21.15.154:6789/0 conn(0x7f6b080c6f60 legacy=0x7f6b081566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0806fca0 con 0x7f6b080c6b80 2023-10-30T13:25:35.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f6b04001680 con 0x7f6b080c6b80 2023-10-30T13:25:35.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 >> v1:172.21.15.46:6789/0 conn(0x7f6b080c6b80 legacy=0x7f6b080bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4005bc0 0x7f6af4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af4006570 0x7f6af4006a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b080c05d0 0x7f6af40072a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0806f9c0 con 0x7f6af4005bc0 2023-10-30T13:25:35.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af400dbd0 con 0x7f6af4006570 2023-10-30T13:25:35.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af400dd90 con 0x7f6b080c05d0 2023-10-30T13:25:35.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0e2da640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4005bc0 0x7f6af4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0e2da640 1 -- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b080c05d0 msgr2=0x7f6af40072a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:35.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0eadb640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b080c05d0 0x7f6af40072a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0dad9640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af4006570 0x7f6af4006a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0e2da640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b080c05d0 0x7f6af40072a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0e2da640 1 -- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af4006570 msgr2=0x7f6af4006a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:35.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0e2da640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af4006570 0x7f6af4006a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0e2da640 1 -- 172.21.15.46:0/2775502365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af4005740 con 0x7f6af4005bc0 2023-10-30T13:25:35.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0eadb640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b080c05d0 0x7f6af40072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:25:35.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.195+0000 7f6b0dad9640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af4006570 0x7f6af4006a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:25:35.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b0e2da640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4005bc0 0x7f6af4005fa0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6b0400c190 tx=0x7f6b0401cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628101b87d2d961e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:35.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b04005ae0 con 0x7f6af4005bc0 2023-10-30T13:25:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b04005c60 con 0x7f6af4005bc0 2023-10-30T13:25:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b0d2d8640 1 -- 172.21.15.46:0/2775502365 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b040a57d0 con 0x7f6af4005bc0 2023-10-30T13:25:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b1053e640 1 -- 172.21.15.46:0/2775502365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4005bc0 msgr2=0x7f6af4005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b1053e640 1 --2- 172.21.15.46:0/2775502365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4005bc0 0x7f6af4005fa0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6b0400c190 tx=0x7f6b0401cbe0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b0eadb640 1 -- 172.21.15.46:0/2775502365 reap_dead start 2023-10-30T13:25:35.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b1053e640 1 -- 172.21.15.46:0/2775502365 shutdown_connections 2023-10-30T13:25:35.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b1053e640 1 -- 172.21.15.46:0/2775502365 >> 172.21.15.46:0/2775502365 conn(0x7f6b080c7910 msgr2=0x7f6b080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:35.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b1053e640 1 -- 172.21.15.46:0/2775502365 shutdown_connections 2023-10-30T13:25:35.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b1053e640 1 -- 172.21.15.46:0/2775502365 wait complete. 2023-10-30T13:25:35.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b1053e640 1 Processor -- start 2023-10-30T13:25:35.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.199+0000 7f6b1053e640 1 -- start start 2023-10-30T13:25:35.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b1053e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6af4005bc0 0x7f6b0813c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b1053e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b080c8c10 0x7f6b08158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b1053e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b08159270 0x7f6b08136db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b1053e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080cac10 con 0x7f6b080c8c10 2023-10-30T13:25:35.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b1053e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08158490 con 0x7f6b08159270 2023-10-30T13:25:35.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b1053e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08158e60 con 0x7f6af4005bc0 2023-10-30T13:25:35.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b0dad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b080c8c10 0x7f6b08158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b0dad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b080c8c10 0x7f6b08158920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47042/0 (socket says 172.21.15.46:47042) 2023-10-30T13:25:35.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b0dad9640 1 -- 172.21.15.46:0/3156080400 learned_addr learned my addr 172.21.15.46:0/3156080400 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:35.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b0eadb640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b08159270 0x7f6b08136db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b0e2da640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6af4005bc0 0x7f6b0813c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b0dad9640 1 -- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6af4005bc0 msgr2=0x7f6b0813c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b0dad9640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6af4005bc0 0x7f6b0813c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b0dad9640 1 -- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b08159270 msgr2=0x7f6b08136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b0dad9640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b08159270 0x7f6b08136db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b0dad9640 1 -- 172.21.15.46:0/3156080400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b08157880 con 0x7f6b080c8c10 2023-10-30T13:25:35.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b0dad9640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b080c8c10 0x7f6b08158920 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6af80108d0 tx=0x7f6af8010d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:35.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6afeffd640 1 -- 172.21.15.46:0/3156080400 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8004220 con 0x7f6b080c8c10 2023-10-30T13:25:35.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6afeffd640 1 -- 172.21.15.46:0/3156080400 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af8004380 con 0x7f6b080c8c10 2023-10-30T13:25:35.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6afeffd640 1 -- 172.21.15.46:0/3156080400 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af800eba0 con 0x7f6b080c8c10 2023-10-30T13:25:35.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b1053e640 1 -- 172.21.15.46:0/3156080400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b08157b60 con 0x7f6b080c8c10 2023-10-30T13:25:35.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b1053e640 1 -- 172.21.15.46:0/3156080400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b08158120 con 0x7f6b080c8c10 2023-10-30T13:25:35.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6b1053e640 1 -- 172.21.15.46:0/3156080400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b081582e0 con 0x7f6b080c8c10 2023-10-30T13:25:35.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.203+0000 7f6afeffd640 1 -- 172.21.15.46:0/3156080400 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6af800ed00 con 0x7f6b080c8c10 2023-10-30T13:25:35.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.207+0000 7f6afeffd640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6aec0664a0 0x7f6aec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.207+0000 7f6afeffd640 1 -- 172.21.15.46:0/3156080400 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f6af8083e40 con 0x7f6b080c8c10 2023-10-30T13:25:35.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.207+0000 7f6b0e2da640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6aec0664a0 0x7f6aec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.207+0000 7f6b0e2da640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6aec0664a0 0x7f6aec068960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6b040ab000 tx=0x7f6b040a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:35.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.211+0000 7f6afeffd640 1 -- 172.21.15.46:0/3156080400 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6af80511d0 con 0x7f6b080c8c10 2023-10-30T13:25:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 -- 172.21.15.46:0/3156080400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b08158600 con 0x7f6b080c8c10 2023-10-30T13:25:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6afeffd640 1 -- 172.21.15.46:0/3156080400 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5731 (secure 0 0 0) 0x7f6af80547f0 con 0x7f6b080c8c10 2023-10-30T13:25:35.403 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:35.403 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:29.892206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 16 2023-10-30T13:25:35.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 -- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6aec0664a0 msgr2=0x7f6aec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6aec0664a0 0x7f6aec068960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6b040ab000 tx=0x7f6b040a7070 comp rx=0 tx=0).stop 2023-10-30T13:25:35.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 -- 172.21.15.46:0/3156080400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b080c8c10 msgr2=0x7f6b08158920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b080c8c10 0x7f6b08158920 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6af80108d0 tx=0x7f6af8010d90 comp rx=0 tx=0).stop 2023-10-30T13:25:35.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 -- 172.21.15.46:0/3156080400 shutdown_connections 2023-10-30T13:25:35.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6aec0664a0 0x7f6aec068960 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b08159270 0x7f6b08136db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b080c8c10 0x7f6b08158920 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 --2- 172.21.15.46:0/3156080400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6af4005bc0 0x7f6b0813c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 -- 172.21.15.46:0/3156080400 >> 172.21.15.46:0/3156080400 conn(0x7f6b080c7910 msgr2=0x7f6b080c16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 -- 172.21.15.46:0/3156080400 shutdown_connections 2023-10-30T13:25:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:35.399+0000 7f6b1053e640 1 -- 172.21.15.46:0/3156080400 wait complete. 2023-10-30T13:25:35.426 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:25:36.445 INFO:tasks.workunit.client.0.smithi046.stdout: create 100/30 file after the snapshot 2023-10-30T13:25:36.638 INFO:tasks.workunit.client.0.smithi046.stdout: create 100/31 file after the snapshot 2023-10-30T13:25:36.639 INFO:tasks.workunit.client.0.smithi046.stdout: create 100/32 file after the snapshot 2023-10-30T13:25:36.639 INFO:tasks.workunit.client.0.smithi046.stdout: create 100/33 file after the snapshot 2023-10-30T13:25:36.639 INFO:tasks.workunit.client.0.smithi046.stdout: create 100/34 file after the snapshot 2023-10-30T13:25:36.639 INFO:tasks.workunit.client.0.smithi046.stdout: create 100/35 file after the snapshot 2023-10-30T13:25:36.640 INFO:tasks.workunit.client.0.smithi046.stdout: create 100/36 file after the snapshot 2023-10-30T13:25:36.640 INFO:tasks.workunit.client.0.smithi046.stdout: create 100/37 file after the snapshot 2023-10-30T13:25:36.640 INFO:tasks.workunit.client.0.smithi046.stdout: create 100/38 file after the snapshot 2023-10-30T13:25:36.640 INFO:tasks.workunit.client.0.smithi046.stdout: create 100/39 file after the snapshot 2023-10-30T13:25:36.640 INFO:tasks.workunit.client.0.smithi046.stdout: create 101/.snap/snap-subdir-test 2023-10-30T13:25:37.427 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:37.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.567+0000 7f641f91d640 1 Processor -- start 2023-10-30T13:25:37.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.567+0000 7f641f91d640 1 -- start start 2023-10-30T13:25:37.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.567+0000 7f641f91d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6418070a00 0x7f6418070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:37.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.567+0000 7f641f91d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6418069e90 0x7f641806a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:37.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.567+0000 7f641f91d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64180713b0 con 0x7f64181539a0 2023-10-30T13:25:37.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.567+0000 7f641f91d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f641806fbb0 con 0x7f6418070a00 2023-10-30T13:25:37.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.567+0000 7f641f91d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6418153d80 con 0x7f6418069e90 2023-10-30T13:25:37.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.567+0000 7f641ceb8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6418070a00 0x7f6418070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:37.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.571+0000 7f641ceb8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6418070a00 0x7f6418070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48568/0 (socket says 172.21.15.46:48568) 2023-10-30T13:25:37.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.571+0000 7f641ceb8640 1 -- 172.21.15.46:0/3482764291 learned_addr learned my addr 172.21.15.46:0/3482764291 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:37.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.571+0000 7f641deba640 1 --2- 172.21.15.46:0/3482764291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6418069e90 0x7f641806a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:37.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.571+0000 7f641ceb8640 1 -- 172.21.15.46:0/3482764291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6418069e90 msgr2=0x7f641806a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:37.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.571+0000 7f641ceb8640 1 --2- 172.21.15.46:0/3482764291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6418069e90 0x7f641806a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.571+0000 7f641ceb8640 1 -- 172.21.15.46:0/3482764291 >> v1:172.21.15.154:6789/0 conn(0x7f64181539a0 legacy=0x7f641815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:37.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.571+0000 7f641ceb8640 1 -- 172.21.15.46:0/3482764291 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64180c5c50 con 0x7f6418070a00 2023-10-30T13:25:37.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.571+0000 7f641ceb8640 1 --2- 172.21.15.46:0/3482764291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6418070a00 0x7f6418070e70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6408009870 tx=0x7f640801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faaed1386b9c371b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:37.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.571+0000 7f640ffff640 1 -- 172.21.15.46:0/3482764291 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64080ad640 con 0x7f6418070a00 2023-10-30T13:25:37.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.571+0000 7f640ffff640 1 -- 172.21.15.46:0/3482764291 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64080ad7a0 con 0x7f6418070a00 2023-10-30T13:25:37.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.571+0000 7f640ffff640 1 -- 172.21.15.46:0/3482764291 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64080adb30 con 0x7f6418070a00 2023-10-30T13:25:37.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 -- 172.21.15.46:0/3482764291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6418070a00 msgr2=0x7f6418070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:37.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 --2- 172.21.15.46:0/3482764291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6418070a00 0x7f6418070e70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6408009870 tx=0x7f640801c480 comp rx=0 tx=0).stop 2023-10-30T13:25:37.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 -- 172.21.15.46:0/3482764291 shutdown_connections 2023-10-30T13:25:37.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 --2- 172.21.15.46:0/3482764291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6418069e90 0x7f641806a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 --2- 172.21.15.46:0/3482764291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6418070a00 0x7f6418070e70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 -- 172.21.15.46:0/3482764291 >> 172.21.15.46:0/3482764291 conn(0x7f64180c7910 msgr2=0x7f64180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:37.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 -- 172.21.15.46:0/3482764291 shutdown_connections 2023-10-30T13:25:37.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 -- 172.21.15.46:0/3482764291 wait complete. 2023-10-30T13:25:37.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 Processor -- start 2023-10-30T13:25:37.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 -- start start 2023-10-30T13:25:37.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6418069e90 0x7f6418147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:37.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6418070a00 0x7f6418158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:37.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64181539a0 0x7f64181f5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:37.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6418147f80 con 0x7f6418070a00 2023-10-30T13:25:37.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64181496d0 con 0x7f6418069e90 2023-10-30T13:25:37.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641f91d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64180c17a0 con 0x7f64181539a0 2023-10-30T13:25:37.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641ceb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6418070a00 0x7f6418158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:37.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641ceb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6418070a00 0x7f6418158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48576/0 (socket says 172.21.15.46:48576) 2023-10-30T13:25:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641ceb8640 1 -- 172.21.15.46:0/350356793 learned_addr learned my addr 172.21.15.46:0/350356793 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641d6b9640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6418069e90 0x7f6418147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641deba640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64181539a0 0x7f64181f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641ceb8640 1 -- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64181539a0 msgr2=0x7f64181f5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641ceb8640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64181539a0 0x7f64181f5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641ceb8640 1 -- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6418069e90 msgr2=0x7f6418147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641ceb8640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6418069e90 0x7f6418147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641ceb8640 1 -- 172.21.15.46:0/350356793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64080094d0 con 0x7f6418070a00 2023-10-30T13:25:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641d6b9640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6418069e90 0x7f6418147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.575+0000 7f641ceb8640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6418070a00 0x7f6418158a20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f64080b0040 tx=0x7f6408019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:37.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.579+0000 7f640dffb640 1 -- 172.21.15.46:0/350356793 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64080bc070 con 0x7f6418070a00 2023-10-30T13:25:37.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.579+0000 7f640dffb640 1 -- 172.21.15.46:0/350356793 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64080a6040 con 0x7f6418070a00 2023-10-30T13:25:37.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.579+0000 7f640dffb640 1 -- 172.21.15.46:0/350356793 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64080b7850 con 0x7f6418070a00 2023-10-30T13:25:37.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.579+0000 7f641f91d640 1 -- 172.21.15.46:0/350356793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64180c1a80 con 0x7f6418070a00 2023-10-30T13:25:37.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.579+0000 7f641f91d640 1 -- 172.21.15.46:0/350356793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6418149c50 con 0x7f6418070a00 2023-10-30T13:25:37.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.579+0000 7f641f91d640 1 -- 172.21.15.46:0/350356793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6418070ec0 con 0x7f6418070a00 2023-10-30T13:25:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.579+0000 7f640dffb640 1 -- 172.21.15.46:0/350356793 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64080b79b0 con 0x7f6418070a00 2023-10-30T13:25:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.579+0000 7f640dffb640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63fc0664a0 0x7f63fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.579+0000 7f640dffb640 1 -- 172.21.15.46:0/350356793 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f640811e400 con 0x7f6418070a00 2023-10-30T13:25:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.583+0000 7f641d6b9640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63fc0664a0 0x7f63fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.583+0000 7f641d6b9640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63fc0664a0 0x7f63fc068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6414000f70 tx=0x7f6414009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:37.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.583+0000 7f640dffb640 1 -- 172.21.15.46:0/350356793 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64080eb790 con 0x7f6418070a00 2023-10-30T13:25:37.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.775+0000 7f641f91d640 1 -- 172.21.15.46:0/350356793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6418149e40 con 0x7f6418070a00 2023-10-30T13:25:37.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.775+0000 7f640dffb640 1 -- 172.21.15.46:0/350356793 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5731 (secure 0 0 0) 0x7f64080ad600 con 0x7f6418070a00 2023-10-30T13:25:37.779 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:37.780 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:35.725213+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":39,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 17 2023-10-30T13:25:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.775+0000 7f641f91d640 1 -- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63fc0664a0 msgr2=0x7f63fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.775+0000 7f641f91d640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63fc0664a0 0x7f63fc068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6414000f70 tx=0x7f6414009040 comp rx=0 tx=0).stop 2023-10-30T13:25:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.775+0000 7f641f91d640 1 -- 172.21.15.46:0/350356793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6418070a00 msgr2=0x7f6418158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.775+0000 7f641f91d640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6418070a00 0x7f6418158a20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f64080b0040 tx=0x7f6408019220 comp rx=0 tx=0).stop 2023-10-30T13:25:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.779+0000 7f641f91d640 1 -- 172.21.15.46:0/350356793 shutdown_connections 2023-10-30T13:25:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.779+0000 7f641f91d640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63fc0664a0 0x7f63fc068960 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.779+0000 7f641f91d640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64181539a0 0x7f64181f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.779+0000 7f641f91d640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6418070a00 0x7f6418158a20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.779+0000 7f641f91d640 1 --2- 172.21.15.46:0/350356793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6418069e90 0x7f6418147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.779+0000 7f641f91d640 1 -- 172.21.15.46:0/350356793 >> 172.21.15.46:0/350356793 conn(0x7f64180c7910 msgr2=0x7f641806b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.779+0000 7f641f91d640 1 -- 172.21.15.46:0/350356793 shutdown_connections 2023-10-30T13:25:37.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:37.779+0000 7f641f91d640 1 -- 172.21.15.46:0/350356793 wait complete. 2023-10-30T13:25:37.800 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=2, replacing gid=4271 2023-10-30T13:25:37.800 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.e 2023-10-30T13:25:38.923 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T13:25:38.919+0000 7fce56864640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:25:40.955 INFO:tasks.workunit.client.0.smithi046.stdout: create 101/30 file after the snapshot 2023-10-30T13:25:40.956 INFO:tasks.workunit.client.0.smithi046.stdout: create 101/31 file after the snapshot 2023-10-30T13:25:40.957 INFO:tasks.workunit.client.0.smithi046.stdout: create 101/32 file after the snapshot 2023-10-30T13:25:40.957 INFO:tasks.workunit.client.0.smithi046.stdout: create 101/33 file after the snapshot 2023-10-30T13:25:40.957 INFO:tasks.workunit.client.0.smithi046.stdout: create 101/34 file after the snapshot 2023-10-30T13:25:40.958 INFO:tasks.workunit.client.0.smithi046.stdout: create 101/35 file after the snapshot 2023-10-30T13:25:40.958 INFO:tasks.workunit.client.0.smithi046.stdout: create 101/36 file after the snapshot 2023-10-30T13:25:40.958 INFO:tasks.workunit.client.0.smithi046.stdout: create 101/37 file after the snapshot 2023-10-30T13:25:40.959 INFO:tasks.workunit.client.0.smithi046.stdout: create 101/38 file after the snapshot 2023-10-30T13:25:40.959 INFO:tasks.workunit.client.0.smithi046.stdout: create 101/39 file after the snapshot 2023-10-30T13:25:40.959 INFO:tasks.workunit.client.0.smithi046.stdout: create 102/.snap/snap-subdir-test 2023-10-30T13:25:41.687 INFO:tasks.workunit.client.0.smithi046.stdout: create 102/30 file after the snapshot 2023-10-30T13:25:41.688 INFO:tasks.workunit.client.0.smithi046.stdout: create 102/31 file after the snapshot 2023-10-30T13:25:41.688 INFO:tasks.workunit.client.0.smithi046.stdout: create 102/32 file after the snapshot 2023-10-30T13:25:41.689 INFO:tasks.workunit.client.0.smithi046.stdout: create 102/33 file after the snapshot 2023-10-30T13:25:41.690 INFO:tasks.workunit.client.0.smithi046.stdout: create 102/34 file after the snapshot 2023-10-30T13:25:41.690 INFO:tasks.workunit.client.0.smithi046.stdout: create 102/35 file after the snapshot 2023-10-30T13:25:41.690 INFO:tasks.workunit.client.0.smithi046.stdout: create 102/36 file after the snapshot 2023-10-30T13:25:41.690 INFO:tasks.workunit.client.0.smithi046.stdout: create 102/37 file after the snapshot 2023-10-30T13:25:41.691 INFO:tasks.workunit.client.0.smithi046.stdout: create 102/38 file after the snapshot 2023-10-30T13:25:41.691 INFO:tasks.workunit.client.0.smithi046.stdout: create 102/39 file after the snapshot 2023-10-30T13:25:41.691 INFO:tasks.workunit.client.0.smithi046.stdout: create 103/.snap/snap-subdir-test 2023-10-30T13:25:41.733 INFO:tasks.workunit.client.0.smithi046.stdout: create 103/30 file after the snapshot 2023-10-30T13:25:41.735 INFO:tasks.workunit.client.0.smithi046.stdout: create 103/31 file after the snapshot 2023-10-30T13:25:41.735 INFO:tasks.workunit.client.0.smithi046.stdout: create 103/32 file after the snapshot 2023-10-30T13:25:41.735 INFO:tasks.workunit.client.0.smithi046.stdout: create 103/33 file after the snapshot 2023-10-30T13:25:41.735 INFO:tasks.workunit.client.0.smithi046.stdout: create 103/34 file after the snapshot 2023-10-30T13:25:41.736 INFO:tasks.workunit.client.0.smithi046.stdout: create 103/35 file after the snapshot 2023-10-30T13:25:41.736 INFO:tasks.workunit.client.0.smithi046.stdout: create 103/36 file after the snapshot 2023-10-30T13:25:41.736 INFO:tasks.workunit.client.0.smithi046.stdout: create 103/37 file after the snapshot 2023-10-30T13:25:41.736 INFO:tasks.workunit.client.0.smithi046.stdout: create 103/38 file after the snapshot 2023-10-30T13:25:41.736 INFO:tasks.workunit.client.0.smithi046.stdout: create 103/39 file after the snapshot 2023-10-30T13:25:41.736 INFO:tasks.workunit.client.0.smithi046.stdout: create 104/.snap/snap-subdir-test 2023-10-30T13:25:41.777 INFO:tasks.workunit.client.0.smithi046.stdout: create 104/30 file after the snapshot 2023-10-30T13:25:41.778 INFO:tasks.workunit.client.0.smithi046.stdout: create 104/31 file after the snapshot 2023-10-30T13:25:41.779 INFO:tasks.workunit.client.0.smithi046.stdout: create 104/32 file after the snapshot 2023-10-30T13:25:41.779 INFO:tasks.workunit.client.0.smithi046.stdout: create 104/33 file after the snapshot 2023-10-30T13:25:41.779 INFO:tasks.workunit.client.0.smithi046.stdout: create 104/34 file after the snapshot 2023-10-30T13:25:41.779 INFO:tasks.workunit.client.0.smithi046.stdout: create 104/35 file after the snapshot 2023-10-30T13:25:41.779 INFO:tasks.workunit.client.0.smithi046.stdout: create 104/36 file after the snapshot 2023-10-30T13:25:41.780 INFO:tasks.workunit.client.0.smithi046.stdout: create 104/37 file after the snapshot 2023-10-30T13:25:41.780 INFO:tasks.workunit.client.0.smithi046.stdout: create 104/38 file after the snapshot 2023-10-30T13:25:41.780 INFO:tasks.workunit.client.0.smithi046.stdout: create 104/39 file after the snapshot 2023-10-30T13:25:41.780 INFO:tasks.workunit.client.0.smithi046.stdout: create 105/.snap/snap-subdir-test 2023-10-30T13:25:41.808 INFO:tasks.workunit.client.0.smithi046.stdout: create 105/30 file after the snapshot 2023-10-30T13:25:41.810 INFO:tasks.workunit.client.0.smithi046.stdout: create 105/31 file after the snapshot 2023-10-30T13:25:41.810 INFO:tasks.workunit.client.0.smithi046.stdout: create 105/32 file after the snapshot 2023-10-30T13:25:41.810 INFO:tasks.workunit.client.0.smithi046.stdout: create 105/33 file after the snapshot 2023-10-30T13:25:41.810 INFO:tasks.workunit.client.0.smithi046.stdout: create 105/34 file after the snapshot 2023-10-30T13:25:41.811 INFO:tasks.workunit.client.0.smithi046.stdout: create 105/35 file after the snapshot 2023-10-30T13:25:41.811 INFO:tasks.workunit.client.0.smithi046.stdout: create 105/36 file after the snapshot 2023-10-30T13:25:41.811 INFO:tasks.workunit.client.0.smithi046.stdout: create 105/37 file after the snapshot 2023-10-30T13:25:41.811 INFO:tasks.workunit.client.0.smithi046.stdout: create 105/38 file after the snapshot 2023-10-30T13:25:41.811 INFO:tasks.workunit.client.0.smithi046.stdout: create 105/39 file after the snapshot 2023-10-30T13:25:41.811 INFO:tasks.workunit.client.0.smithi046.stdout: create 106/.snap/snap-subdir-test 2023-10-30T13:25:41.838 INFO:tasks.workunit.client.0.smithi046.stdout: create 106/30 file after the snapshot 2023-10-30T13:25:41.840 INFO:tasks.workunit.client.0.smithi046.stdout: create 106/31 file after the snapshot 2023-10-30T13:25:41.840 INFO:tasks.workunit.client.0.smithi046.stdout: create 106/32 file after the snapshot 2023-10-30T13:25:41.840 INFO:tasks.workunit.client.0.smithi046.stdout: create 106/33 file after the snapshot 2023-10-30T13:25:41.840 INFO:tasks.workunit.client.0.smithi046.stdout: create 106/34 file after the snapshot 2023-10-30T13:25:41.841 INFO:tasks.workunit.client.0.smithi046.stdout: create 106/35 file after the snapshot 2023-10-30T13:25:41.841 INFO:tasks.workunit.client.0.smithi046.stdout: create 106/36 file after the snapshot 2023-10-30T13:25:41.841 INFO:tasks.workunit.client.0.smithi046.stdout: create 106/37 file after the snapshot 2023-10-30T13:25:41.841 INFO:tasks.workunit.client.0.smithi046.stdout: create 106/38 file after the snapshot 2023-10-30T13:25:41.841 INFO:tasks.workunit.client.0.smithi046.stdout: create 106/39 file after the snapshot 2023-10-30T13:25:41.842 INFO:tasks.workunit.client.0.smithi046.stdout: create 107/.snap/snap-subdir-test 2023-10-30T13:25:41.868 INFO:tasks.workunit.client.0.smithi046.stdout: create 107/30 file after the snapshot 2023-10-30T13:25:41.870 INFO:tasks.workunit.client.0.smithi046.stdout: create 107/31 file after the snapshot 2023-10-30T13:25:41.870 INFO:tasks.workunit.client.0.smithi046.stdout: create 107/32 file after the snapshot 2023-10-30T13:25:41.871 INFO:tasks.workunit.client.0.smithi046.stdout: create 107/33 file after the snapshot 2023-10-30T13:25:41.871 INFO:tasks.workunit.client.0.smithi046.stdout: create 107/34 file after the snapshot 2023-10-30T13:25:41.871 INFO:tasks.workunit.client.0.smithi046.stdout: create 107/35 file after the snapshot 2023-10-30T13:25:41.871 INFO:tasks.workunit.client.0.smithi046.stdout: create 107/36 file after the snapshot 2023-10-30T13:25:41.872 INFO:tasks.workunit.client.0.smithi046.stdout: create 107/37 file after the snapshot 2023-10-30T13:25:41.872 INFO:tasks.workunit.client.0.smithi046.stdout: create 107/38 file after the snapshot 2023-10-30T13:25:41.872 INFO:tasks.workunit.client.0.smithi046.stdout: create 107/39 file after the snapshot 2023-10-30T13:25:41.872 INFO:tasks.workunit.client.0.smithi046.stdout: create 108/.snap/snap-subdir-test 2023-10-30T13:25:41.900 INFO:tasks.workunit.client.0.smithi046.stdout: create 108/30 file after the snapshot 2023-10-30T13:25:41.902 INFO:tasks.workunit.client.0.smithi046.stdout: create 108/31 file after the snapshot 2023-10-30T13:25:41.902 INFO:tasks.workunit.client.0.smithi046.stdout: create 108/32 file after the snapshot 2023-10-30T13:25:41.902 INFO:tasks.workunit.client.0.smithi046.stdout: create 108/33 file after the snapshot 2023-10-30T13:25:41.902 INFO:tasks.workunit.client.0.smithi046.stdout: create 108/34 file after the snapshot 2023-10-30T13:25:41.902 INFO:tasks.workunit.client.0.smithi046.stdout: create 108/35 file after the snapshot 2023-10-30T13:25:41.903 INFO:tasks.workunit.client.0.smithi046.stdout: create 108/36 file after the snapshot 2023-10-30T13:25:41.903 INFO:tasks.workunit.client.0.smithi046.stdout: create 108/37 file after the snapshot 2023-10-30T13:25:41.903 INFO:tasks.workunit.client.0.smithi046.stdout: create 108/38 file after the snapshot 2023-10-30T13:25:41.903 INFO:tasks.workunit.client.0.smithi046.stdout: create 108/39 file after the snapshot 2023-10-30T13:25:41.903 INFO:tasks.workunit.client.0.smithi046.stdout: create 109/.snap/snap-subdir-test 2023-10-30T13:25:42.182 INFO:tasks.workunit.client.0.smithi046.stdout: create 109/30 file after the snapshot 2023-10-30T13:25:42.183 INFO:tasks.workunit.client.0.smithi046.stdout: create 109/31 file after the snapshot 2023-10-30T13:25:42.183 INFO:tasks.workunit.client.0.smithi046.stdout: create 109/32 file after the snapshot 2023-10-30T13:25:42.183 INFO:tasks.workunit.client.0.smithi046.stdout: create 109/33 file after the snapshot 2023-10-30T13:25:42.183 INFO:tasks.workunit.client.0.smithi046.stdout: create 109/34 file after the snapshot 2023-10-30T13:25:42.184 INFO:tasks.workunit.client.0.smithi046.stdout: create 109/35 file after the snapshot 2023-10-30T13:25:42.184 INFO:tasks.workunit.client.0.smithi046.stdout: create 109/36 file after the snapshot 2023-10-30T13:25:42.185 INFO:tasks.workunit.client.0.smithi046.stdout: create 109/37 file after the snapshot 2023-10-30T13:25:42.185 INFO:tasks.workunit.client.0.smithi046.stdout: create 109/38 file after the snapshot 2023-10-30T13:25:42.185 INFO:tasks.workunit.client.0.smithi046.stdout: create 109/39 file after the snapshot 2023-10-30T13:25:42.185 INFO:tasks.workunit.client.0.smithi046.stdout: create 110/.snap/snap-subdir-test 2023-10-30T13:25:43.177 INFO:tasks.workunit.client.0.smithi046.stdout: create 110/30 file after the snapshot 2023-10-30T13:25:43.178 INFO:tasks.workunit.client.0.smithi046.stdout: create 110/31 file after the snapshot 2023-10-30T13:25:43.178 INFO:tasks.workunit.client.0.smithi046.stdout: create 110/32 file after the snapshot 2023-10-30T13:25:43.179 INFO:tasks.workunit.client.0.smithi046.stdout: create 110/33 file after the snapshot 2023-10-30T13:25:43.179 INFO:tasks.workunit.client.0.smithi046.stdout: create 110/34 file after the snapshot 2023-10-30T13:25:43.179 INFO:tasks.workunit.client.0.smithi046.stdout: create 110/35 file after the snapshot 2023-10-30T13:25:43.180 INFO:tasks.workunit.client.0.smithi046.stdout: create 110/36 file after the snapshot 2023-10-30T13:25:43.180 INFO:tasks.workunit.client.0.smithi046.stdout: create 110/37 file after the snapshot 2023-10-30T13:25:43.180 INFO:tasks.workunit.client.0.smithi046.stdout: create 110/38 file after the snapshot 2023-10-30T13:25:43.180 INFO:tasks.workunit.client.0.smithi046.stdout: create 110/39 file after the snapshot 2023-10-30T13:25:43.181 INFO:tasks.workunit.client.0.smithi046.stdout: create 111/.snap/snap-subdir-test 2023-10-30T13:25:43.211 INFO:tasks.workunit.client.0.smithi046.stdout: create 111/30 file after the snapshot 2023-10-30T13:25:43.213 INFO:tasks.workunit.client.0.smithi046.stdout: create 111/31 file after the snapshot 2023-10-30T13:25:43.213 INFO:tasks.workunit.client.0.smithi046.stdout: create 111/32 file after the snapshot 2023-10-30T13:25:43.214 INFO:tasks.workunit.client.0.smithi046.stdout: create 111/33 file after the snapshot 2023-10-30T13:25:43.214 INFO:tasks.workunit.client.0.smithi046.stdout: create 111/34 file after the snapshot 2023-10-30T13:25:43.214 INFO:tasks.workunit.client.0.smithi046.stdout: create 111/35 file after the snapshot 2023-10-30T13:25:43.214 INFO:tasks.workunit.client.0.smithi046.stdout: create 111/36 file after the snapshot 2023-10-30T13:25:43.214 INFO:tasks.workunit.client.0.smithi046.stdout: create 111/37 file after the snapshot 2023-10-30T13:25:43.215 INFO:tasks.workunit.client.0.smithi046.stdout: create 111/38 file after the snapshot 2023-10-30T13:25:43.215 INFO:tasks.workunit.client.0.smithi046.stdout: create 111/39 file after the snapshot 2023-10-30T13:25:43.215 INFO:tasks.workunit.client.0.smithi046.stdout: create 112/.snap/snap-subdir-test 2023-10-30T13:25:43.249 INFO:tasks.workunit.client.0.smithi046.stdout: create 112/30 file after the snapshot 2023-10-30T13:25:43.251 INFO:tasks.workunit.client.0.smithi046.stdout: create 112/31 file after the snapshot 2023-10-30T13:25:43.251 INFO:tasks.workunit.client.0.smithi046.stdout: create 112/32 file after the snapshot 2023-10-30T13:25:43.251 INFO:tasks.workunit.client.0.smithi046.stdout: create 112/33 file after the snapshot 2023-10-30T13:25:43.251 INFO:tasks.workunit.client.0.smithi046.stdout: create 112/34 file after the snapshot 2023-10-30T13:25:43.252 INFO:tasks.workunit.client.0.smithi046.stdout: create 112/35 file after the snapshot 2023-10-30T13:25:43.252 INFO:tasks.workunit.client.0.smithi046.stdout: create 112/36 file after the snapshot 2023-10-30T13:25:43.252 INFO:tasks.workunit.client.0.smithi046.stdout: create 112/37 file after the snapshot 2023-10-30T13:25:43.252 INFO:tasks.workunit.client.0.smithi046.stdout: create 112/38 file after the snapshot 2023-10-30T13:25:43.253 INFO:tasks.workunit.client.0.smithi046.stdout: create 112/39 file after the snapshot 2023-10-30T13:25:43.253 INFO:tasks.workunit.client.0.smithi046.stdout: create 113/.snap/snap-subdir-test 2023-10-30T13:25:43.280 INFO:tasks.workunit.client.0.smithi046.stdout: create 113/30 file after the snapshot 2023-10-30T13:25:43.283 INFO:tasks.workunit.client.0.smithi046.stdout: create 113/31 file after the snapshot 2023-10-30T13:25:43.283 INFO:tasks.workunit.client.0.smithi046.stdout: create 113/32 file after the snapshot 2023-10-30T13:25:43.283 INFO:tasks.workunit.client.0.smithi046.stdout: create 113/33 file after the snapshot 2023-10-30T13:25:43.284 INFO:tasks.workunit.client.0.smithi046.stdout: create 113/34 file after the snapshot 2023-10-30T13:25:43.284 INFO:tasks.workunit.client.0.smithi046.stdout: create 113/35 file after the snapshot 2023-10-30T13:25:43.284 INFO:tasks.workunit.client.0.smithi046.stdout: create 113/36 file after the snapshot 2023-10-30T13:25:43.285 INFO:tasks.workunit.client.0.smithi046.stdout: create 113/37 file after the snapshot 2023-10-30T13:25:43.285 INFO:tasks.workunit.client.0.smithi046.stdout: create 113/38 file after the snapshot 2023-10-30T13:25:43.285 INFO:tasks.workunit.client.0.smithi046.stdout: create 113/39 file after the snapshot 2023-10-30T13:25:43.285 INFO:tasks.workunit.client.0.smithi046.stdout: create 114/.snap/snap-subdir-test 2023-10-30T13:25:43.318 INFO:tasks.workunit.client.0.smithi046.stdout: create 114/30 file after the snapshot 2023-10-30T13:25:43.319 INFO:tasks.workunit.client.0.smithi046.stdout: create 114/31 file after the snapshot 2023-10-30T13:25:43.320 INFO:tasks.workunit.client.0.smithi046.stdout: create 114/32 file after the snapshot 2023-10-30T13:25:43.320 INFO:tasks.workunit.client.0.smithi046.stdout: create 114/33 file after the snapshot 2023-10-30T13:25:43.320 INFO:tasks.workunit.client.0.smithi046.stdout: create 114/34 file after the snapshot 2023-10-30T13:25:43.320 INFO:tasks.workunit.client.0.smithi046.stdout: create 114/35 file after the snapshot 2023-10-30T13:25:43.321 INFO:tasks.workunit.client.0.smithi046.stdout: create 114/36 file after the snapshot 2023-10-30T13:25:43.321 INFO:tasks.workunit.client.0.smithi046.stdout: create 114/37 file after the snapshot 2023-10-30T13:25:43.321 INFO:tasks.workunit.client.0.smithi046.stdout: create 114/38 file after the snapshot 2023-10-30T13:25:43.321 INFO:tasks.workunit.client.0.smithi046.stdout: create 114/39 file after the snapshot 2023-10-30T13:25:43.321 INFO:tasks.workunit.client.0.smithi046.stdout: create 115/.snap/snap-subdir-test 2023-10-30T13:25:43.938 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:25:43.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:25:44.334 INFO:tasks.workunit.client.0.smithi046.stdout: create 115/30 file after the snapshot 2023-10-30T13:25:44.336 INFO:tasks.workunit.client.0.smithi046.stdout: create 115/31 file after the snapshot 2023-10-30T13:25:44.336 INFO:tasks.workunit.client.0.smithi046.stdout: create 115/32 file after the snapshot 2023-10-30T13:25:44.336 INFO:tasks.workunit.client.0.smithi046.stdout: create 115/33 file after the snapshot 2023-10-30T13:25:44.337 INFO:tasks.workunit.client.0.smithi046.stdout: create 115/34 file after the snapshot 2023-10-30T13:25:44.337 INFO:tasks.workunit.client.0.smithi046.stdout: create 115/35 file after the snapshot 2023-10-30T13:25:44.337 INFO:tasks.workunit.client.0.smithi046.stdout: create 115/36 file after the snapshot 2023-10-30T13:25:44.337 INFO:tasks.workunit.client.0.smithi046.stdout: create 115/37 file after the snapshot 2023-10-30T13:25:44.337 INFO:tasks.workunit.client.0.smithi046.stdout: create 115/38 file after the snapshot 2023-10-30T13:25:44.338 INFO:tasks.workunit.client.0.smithi046.stdout: create 115/39 file after the snapshot 2023-10-30T13:25:44.338 INFO:tasks.workunit.client.0.smithi046.stdout: create 116/.snap/snap-subdir-test 2023-10-30T13:25:44.368 INFO:tasks.workunit.client.0.smithi046.stdout: create 116/30 file after the snapshot 2023-10-30T13:25:44.370 INFO:tasks.workunit.client.0.smithi046.stdout: create 116/31 file after the snapshot 2023-10-30T13:25:44.370 INFO:tasks.workunit.client.0.smithi046.stdout: create 116/32 file after the snapshot 2023-10-30T13:25:44.370 INFO:tasks.workunit.client.0.smithi046.stdout: create 116/33 file after the snapshot 2023-10-30T13:25:44.370 INFO:tasks.workunit.client.0.smithi046.stdout: create 116/34 file after the snapshot 2023-10-30T13:25:44.371 INFO:tasks.workunit.client.0.smithi046.stdout: create 116/35 file after the snapshot 2023-10-30T13:25:44.371 INFO:tasks.workunit.client.0.smithi046.stdout: create 116/36 file after the snapshot 2023-10-30T13:25:44.371 INFO:tasks.workunit.client.0.smithi046.stdout: create 116/37 file after the snapshot 2023-10-30T13:25:44.371 INFO:tasks.workunit.client.0.smithi046.stdout: create 116/38 file after the snapshot 2023-10-30T13:25:44.371 INFO:tasks.workunit.client.0.smithi046.stdout: create 116/39 file after the snapshot 2023-10-30T13:25:44.372 INFO:tasks.workunit.client.0.smithi046.stdout: create 117/.snap/snap-subdir-test 2023-10-30T13:25:44.434 INFO:tasks.workunit.client.0.smithi046.stdout: create 117/30 file after the snapshot 2023-10-30T13:25:44.436 INFO:tasks.workunit.client.0.smithi046.stdout: create 117/31 file after the snapshot 2023-10-30T13:25:44.436 INFO:tasks.workunit.client.0.smithi046.stdout: create 117/32 file after the snapshot 2023-10-30T13:25:44.436 INFO:tasks.workunit.client.0.smithi046.stdout: create 117/33 file after the snapshot 2023-10-30T13:25:44.436 INFO:tasks.workunit.client.0.smithi046.stdout: create 117/34 file after the snapshot 2023-10-30T13:25:44.436 INFO:tasks.workunit.client.0.smithi046.stdout: create 117/35 file after the snapshot 2023-10-30T13:25:44.437 INFO:tasks.workunit.client.0.smithi046.stdout: create 117/36 file after the snapshot 2023-10-30T13:25:44.437 INFO:tasks.workunit.client.0.smithi046.stdout: create 117/37 file after the snapshot 2023-10-30T13:25:44.437 INFO:tasks.workunit.client.0.smithi046.stdout: create 117/38 file after the snapshot 2023-10-30T13:25:44.437 INFO:tasks.workunit.client.0.smithi046.stdout: create 117/39 file after the snapshot 2023-10-30T13:25:44.437 INFO:tasks.workunit.client.0.smithi046.stdout: create 118/.snap/snap-subdir-test 2023-10-30T13:25:45.032 INFO:tasks.workunit.client.0.smithi046.stdout: create 118/30 file after the snapshot 2023-10-30T13:25:45.034 INFO:tasks.workunit.client.0.smithi046.stdout: create 118/31 file after the snapshot 2023-10-30T13:25:45.034 INFO:tasks.workunit.client.0.smithi046.stdout: create 118/32 file after the snapshot 2023-10-30T13:25:45.034 INFO:tasks.workunit.client.0.smithi046.stdout: create 118/33 file after the snapshot 2023-10-30T13:25:45.034 INFO:tasks.workunit.client.0.smithi046.stdout: create 118/34 file after the snapshot 2023-10-30T13:25:45.034 INFO:tasks.workunit.client.0.smithi046.stdout: create 118/35 file after the snapshot 2023-10-30T13:25:45.035 INFO:tasks.workunit.client.0.smithi046.stdout: create 118/36 file after the snapshot 2023-10-30T13:25:45.035 INFO:tasks.workunit.client.0.smithi046.stdout: create 118/37 file after the snapshot 2023-10-30T13:25:45.035 INFO:tasks.workunit.client.0.smithi046.stdout: create 118/38 file after the snapshot 2023-10-30T13:25:45.035 INFO:tasks.workunit.client.0.smithi046.stdout: create 118/39 file after the snapshot 2023-10-30T13:25:45.036 INFO:tasks.workunit.client.0.smithi046.stdout: create 119/.snap/snap-subdir-test 2023-10-30T13:25:45.343 INFO:tasks.workunit.client.0.smithi046.stdout: create 119/30 file after the snapshot 2023-10-30T13:25:45.345 INFO:tasks.workunit.client.0.smithi046.stdout: create 119/31 file after the snapshot 2023-10-30T13:25:45.345 INFO:tasks.workunit.client.0.smithi046.stdout: create 119/32 file after the snapshot 2023-10-30T13:25:45.345 INFO:tasks.workunit.client.0.smithi046.stdout: create 119/33 file after the snapshot 2023-10-30T13:25:45.345 INFO:tasks.workunit.client.0.smithi046.stdout: create 119/34 file after the snapshot 2023-10-30T13:25:45.346 INFO:tasks.workunit.client.0.smithi046.stdout: create 119/35 file after the snapshot 2023-10-30T13:25:45.346 INFO:tasks.workunit.client.0.smithi046.stdout: create 119/36 file after the snapshot 2023-10-30T13:25:45.346 INFO:tasks.workunit.client.0.smithi046.stdout: create 119/37 file after the snapshot 2023-10-30T13:25:45.346 INFO:tasks.workunit.client.0.smithi046.stdout: create 119/38 file after the snapshot 2023-10-30T13:25:45.346 INFO:tasks.workunit.client.0.smithi046.stdout: create 119/39 file after the snapshot 2023-10-30T13:25:45.347 INFO:tasks.workunit.client.0.smithi046.stdout: create 120/.snap/snap-subdir-test 2023-10-30T13:25:45.376 INFO:tasks.workunit.client.0.smithi046.stdout: create 120/30 file after the snapshot 2023-10-30T13:25:45.378 INFO:tasks.workunit.client.0.smithi046.stdout: create 120/31 file after the snapshot 2023-10-30T13:25:45.378 INFO:tasks.workunit.client.0.smithi046.stdout: create 120/32 file after the snapshot 2023-10-30T13:25:45.379 INFO:tasks.workunit.client.0.smithi046.stdout: create 120/33 file after the snapshot 2023-10-30T13:25:45.379 INFO:tasks.workunit.client.0.smithi046.stdout: create 120/34 file after the snapshot 2023-10-30T13:25:45.379 INFO:tasks.workunit.client.0.smithi046.stdout: create 120/35 file after the snapshot 2023-10-30T13:25:45.379 INFO:tasks.workunit.client.0.smithi046.stdout: create 120/36 file after the snapshot 2023-10-30T13:25:45.379 INFO:tasks.workunit.client.0.smithi046.stdout: create 120/37 file after the snapshot 2023-10-30T13:25:45.380 INFO:tasks.workunit.client.0.smithi046.stdout: create 120/38 file after the snapshot 2023-10-30T13:25:45.380 INFO:tasks.workunit.client.0.smithi046.stdout: create 120/39 file after the snapshot 2023-10-30T13:25:45.380 INFO:tasks.workunit.client.0.smithi046.stdout: create 121/.snap/snap-subdir-test 2023-10-30T13:25:45.408 INFO:tasks.workunit.client.0.smithi046.stdout: create 121/30 file after the snapshot 2023-10-30T13:25:45.410 INFO:tasks.workunit.client.0.smithi046.stdout: create 121/31 file after the snapshot 2023-10-30T13:25:45.410 INFO:tasks.workunit.client.0.smithi046.stdout: create 121/32 file after the snapshot 2023-10-30T13:25:45.410 INFO:tasks.workunit.client.0.smithi046.stdout: create 121/33 file after the snapshot 2023-10-30T13:25:45.410 INFO:tasks.workunit.client.0.smithi046.stdout: create 121/34 file after the snapshot 2023-10-30T13:25:45.411 INFO:tasks.workunit.client.0.smithi046.stdout: create 121/35 file after the snapshot 2023-10-30T13:25:45.411 INFO:tasks.workunit.client.0.smithi046.stdout: create 121/36 file after the snapshot 2023-10-30T13:25:45.411 INFO:tasks.workunit.client.0.smithi046.stdout: create 121/37 file after the snapshot 2023-10-30T13:25:45.411 INFO:tasks.workunit.client.0.smithi046.stdout: create 121/38 file after the snapshot 2023-10-30T13:25:45.411 INFO:tasks.workunit.client.0.smithi046.stdout: create 121/39 file after the snapshot 2023-10-30T13:25:45.412 INFO:tasks.workunit.client.0.smithi046.stdout: create 122/.snap/snap-subdir-test 2023-10-30T13:25:45.689 INFO:tasks.workunit.client.0.smithi046.stdout: create 122/30 file after the snapshot 2023-10-30T13:25:45.690 INFO:tasks.workunit.client.0.smithi046.stdout: create 122/31 file after the snapshot 2023-10-30T13:25:45.690 INFO:tasks.workunit.client.0.smithi046.stdout: create 122/32 file after the snapshot 2023-10-30T13:25:45.691 INFO:tasks.workunit.client.0.smithi046.stdout: create 122/33 file after the snapshot 2023-10-30T13:25:45.691 INFO:tasks.workunit.client.0.smithi046.stdout: create 122/34 file after the snapshot 2023-10-30T13:25:45.691 INFO:tasks.workunit.client.0.smithi046.stdout: create 122/35 file after the snapshot 2023-10-30T13:25:45.691 INFO:tasks.workunit.client.0.smithi046.stdout: create 122/36 file after the snapshot 2023-10-30T13:25:45.691 INFO:tasks.workunit.client.0.smithi046.stdout: create 122/37 file after the snapshot 2023-10-30T13:25:45.692 INFO:tasks.workunit.client.0.smithi046.stdout: create 122/38 file after the snapshot 2023-10-30T13:25:45.692 INFO:tasks.workunit.client.0.smithi046.stdout: create 122/39 file after the snapshot 2023-10-30T13:25:45.692 INFO:tasks.workunit.client.0.smithi046.stdout: create 123/.snap/snap-subdir-test 2023-10-30T13:25:45.720 INFO:tasks.workunit.client.0.smithi046.stdout: create 123/30 file after the snapshot 2023-10-30T13:25:45.722 INFO:tasks.workunit.client.0.smithi046.stdout: create 123/31 file after the snapshot 2023-10-30T13:25:45.722 INFO:tasks.workunit.client.0.smithi046.stdout: create 123/32 file after the snapshot 2023-10-30T13:25:45.722 INFO:tasks.workunit.client.0.smithi046.stdout: create 123/33 file after the snapshot 2023-10-30T13:25:45.723 INFO:tasks.workunit.client.0.smithi046.stdout: create 123/34 file after the snapshot 2023-10-30T13:25:45.723 INFO:tasks.workunit.client.0.smithi046.stdout: create 123/35 file after the snapshot 2023-10-30T13:25:45.723 INFO:tasks.workunit.client.0.smithi046.stdout: create 123/36 file after the snapshot 2023-10-30T13:25:45.723 INFO:tasks.workunit.client.0.smithi046.stdout: create 123/37 file after the snapshot 2023-10-30T13:25:45.723 INFO:tasks.workunit.client.0.smithi046.stdout: create 123/38 file after the snapshot 2023-10-30T13:25:45.724 INFO:tasks.workunit.client.0.smithi046.stdout: create 123/39 file after the snapshot 2023-10-30T13:25:45.724 INFO:tasks.workunit.client.0.smithi046.stdout: create 124/.snap/snap-subdir-test 2023-10-30T13:25:45.750 INFO:tasks.workunit.client.0.smithi046.stdout: create 124/30 file after the snapshot 2023-10-30T13:25:45.752 INFO:tasks.workunit.client.0.smithi046.stdout: create 124/31 file after the snapshot 2023-10-30T13:25:45.752 INFO:tasks.workunit.client.0.smithi046.stdout: create 124/32 file after the snapshot 2023-10-30T13:25:45.752 INFO:tasks.workunit.client.0.smithi046.stdout: create 124/33 file after the snapshot 2023-10-30T13:25:45.753 INFO:tasks.workunit.client.0.smithi046.stdout: create 124/34 file after the snapshot 2023-10-30T13:25:45.753 INFO:tasks.workunit.client.0.smithi046.stdout: create 124/35 file after the snapshot 2023-10-30T13:25:45.753 INFO:tasks.workunit.client.0.smithi046.stdout: create 124/36 file after the snapshot 2023-10-30T13:25:45.753 INFO:tasks.workunit.client.0.smithi046.stdout: create 124/37 file after the snapshot 2023-10-30T13:25:45.753 INFO:tasks.workunit.client.0.smithi046.stdout: create 124/38 file after the snapshot 2023-10-30T13:25:45.754 INFO:tasks.workunit.client.0.smithi046.stdout: create 124/39 file after the snapshot 2023-10-30T13:25:45.754 INFO:tasks.workunit.client.0.smithi046.stdout: create 125/.snap/snap-subdir-test 2023-10-30T13:25:45.781 INFO:tasks.workunit.client.0.smithi046.stdout: create 125/30 file after the snapshot 2023-10-30T13:25:45.782 INFO:tasks.workunit.client.0.smithi046.stdout: create 125/31 file after the snapshot 2023-10-30T13:25:45.782 INFO:tasks.workunit.client.0.smithi046.stdout: create 125/32 file after the snapshot 2023-10-30T13:25:45.783 INFO:tasks.workunit.client.0.smithi046.stdout: create 125/33 file after the snapshot 2023-10-30T13:25:45.783 INFO:tasks.workunit.client.0.smithi046.stdout: create 125/34 file after the snapshot 2023-10-30T13:25:45.783 INFO:tasks.workunit.client.0.smithi046.stdout: create 125/35 file after the snapshot 2023-10-30T13:25:45.783 INFO:tasks.workunit.client.0.smithi046.stdout: create 125/36 file after the snapshot 2023-10-30T13:25:45.783 INFO:tasks.workunit.client.0.smithi046.stdout: create 125/37 file after the snapshot 2023-10-30T13:25:45.783 INFO:tasks.workunit.client.0.smithi046.stdout: create 125/38 file after the snapshot 2023-10-30T13:25:45.784 INFO:tasks.workunit.client.0.smithi046.stdout: create 125/39 file after the snapshot 2023-10-30T13:25:45.784 INFO:tasks.workunit.client.0.smithi046.stdout: create 126/.snap/snap-subdir-test 2023-10-30T13:25:45.809 INFO:tasks.workunit.client.0.smithi046.stdout: create 126/30 file after the snapshot 2023-10-30T13:25:45.811 INFO:tasks.workunit.client.0.smithi046.stdout: create 126/31 file after the snapshot 2023-10-30T13:25:45.811 INFO:tasks.workunit.client.0.smithi046.stdout: create 126/32 file after the snapshot 2023-10-30T13:25:45.811 INFO:tasks.workunit.client.0.smithi046.stdout: create 126/33 file after the snapshot 2023-10-30T13:25:45.812 INFO:tasks.workunit.client.0.smithi046.stdout: create 126/34 file after the snapshot 2023-10-30T13:25:45.812 INFO:tasks.workunit.client.0.smithi046.stdout: create 126/35 file after the snapshot 2023-10-30T13:25:45.812 INFO:tasks.workunit.client.0.smithi046.stdout: create 126/36 file after the snapshot 2023-10-30T13:25:45.812 INFO:tasks.workunit.client.0.smithi046.stdout: create 126/37 file after the snapshot 2023-10-30T13:25:45.812 INFO:tasks.workunit.client.0.smithi046.stdout: create 126/38 file after the snapshot 2023-10-30T13:25:45.812 INFO:tasks.workunit.client.0.smithi046.stdout: create 126/39 file after the snapshot 2023-10-30T13:25:45.813 INFO:tasks.workunit.client.0.smithi046.stdout: create 127/.snap/snap-subdir-test 2023-10-30T13:25:46.084 INFO:tasks.workunit.client.0.smithi046.stdout: create 127/30 file after the snapshot 2023-10-30T13:25:46.085 INFO:tasks.workunit.client.0.smithi046.stdout: create 127/31 file after the snapshot 2023-10-30T13:25:46.085 INFO:tasks.workunit.client.0.smithi046.stdout: create 127/32 file after the snapshot 2023-10-30T13:25:46.086 INFO:tasks.workunit.client.0.smithi046.stdout: create 127/33 file after the snapshot 2023-10-30T13:25:46.087 INFO:tasks.workunit.client.0.smithi046.stdout: create 127/34 file after the snapshot 2023-10-30T13:25:46.087 INFO:tasks.workunit.client.0.smithi046.stdout: create 127/35 file after the snapshot 2023-10-30T13:25:46.087 INFO:tasks.workunit.client.0.smithi046.stdout: create 127/36 file after the snapshot 2023-10-30T13:25:46.087 INFO:tasks.workunit.client.0.smithi046.stdout: create 127/37 file after the snapshot 2023-10-30T13:25:46.087 INFO:tasks.workunit.client.0.smithi046.stdout: create 127/38 file after the snapshot 2023-10-30T13:25:46.088 INFO:tasks.workunit.client.0.smithi046.stdout: create 127/39 file after the snapshot 2023-10-30T13:25:46.088 INFO:tasks.workunit.client.0.smithi046.stdout: create 128/.snap/snap-subdir-test 2023-10-30T13:25:46.114 INFO:tasks.workunit.client.0.smithi046.stdout: create 128/30 file after the snapshot 2023-10-30T13:25:46.115 INFO:tasks.workunit.client.0.smithi046.stdout: create 128/31 file after the snapshot 2023-10-30T13:25:46.115 INFO:tasks.workunit.client.0.smithi046.stdout: create 128/32 file after the snapshot 2023-10-30T13:25:46.116 INFO:tasks.workunit.client.0.smithi046.stdout: create 128/33 file after the snapshot 2023-10-30T13:25:46.116 INFO:tasks.workunit.client.0.smithi046.stdout: create 128/34 file after the snapshot 2023-10-30T13:25:46.117 INFO:tasks.workunit.client.0.smithi046.stdout: create 128/35 file after the snapshot 2023-10-30T13:25:46.117 INFO:tasks.workunit.client.0.smithi046.stdout: create 128/36 file after the snapshot 2023-10-30T13:25:46.117 INFO:tasks.workunit.client.0.smithi046.stdout: create 128/37 file after the snapshot 2023-10-30T13:25:46.117 INFO:tasks.workunit.client.0.smithi046.stdout: create 128/38 file after the snapshot 2023-10-30T13:25:46.118 INFO:tasks.workunit.client.0.smithi046.stdout: create 128/39 file after the snapshot 2023-10-30T13:25:46.118 INFO:tasks.workunit.client.0.smithi046.stdout: create 129/.snap/snap-subdir-test 2023-10-30T13:25:46.147 INFO:tasks.workunit.client.0.smithi046.stdout: create 129/30 file after the snapshot 2023-10-30T13:25:46.148 INFO:tasks.workunit.client.0.smithi046.stdout: create 129/31 file after the snapshot 2023-10-30T13:25:46.149 INFO:tasks.workunit.client.0.smithi046.stdout: create 129/32 file after the snapshot 2023-10-30T13:25:46.149 INFO:tasks.workunit.client.0.smithi046.stdout: create 129/33 file after the snapshot 2023-10-30T13:25:46.149 INFO:tasks.workunit.client.0.smithi046.stdout: create 129/34 file after the snapshot 2023-10-30T13:25:46.149 INFO:tasks.workunit.client.0.smithi046.stdout: create 129/35 file after the snapshot 2023-10-30T13:25:46.149 INFO:tasks.workunit.client.0.smithi046.stdout: create 129/36 file after the snapshot 2023-10-30T13:25:46.150 INFO:tasks.workunit.client.0.smithi046.stdout: create 129/37 file after the snapshot 2023-10-30T13:25:46.150 INFO:tasks.workunit.client.0.smithi046.stdout: create 129/38 file after the snapshot 2023-10-30T13:25:46.150 INFO:tasks.workunit.client.0.smithi046.stdout: create 129/39 file after the snapshot 2023-10-30T13:25:46.150 INFO:tasks.workunit.client.0.smithi046.stdout: create 130/.snap/snap-subdir-test 2023-10-30T13:25:46.187 INFO:tasks.workunit.client.0.smithi046.stdout: create 130/30 file after the snapshot 2023-10-30T13:25:46.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 130/31 file after the snapshot 2023-10-30T13:25:46.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 130/32 file after the snapshot 2023-10-30T13:25:46.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 130/33 file after the snapshot 2023-10-30T13:25:46.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 130/34 file after the snapshot 2023-10-30T13:25:46.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 130/35 file after the snapshot 2023-10-30T13:25:46.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 130/36 file after the snapshot 2023-10-30T13:25:46.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 130/37 file after the snapshot 2023-10-30T13:25:46.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 130/38 file after the snapshot 2023-10-30T13:25:46.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 130/39 file after the snapshot 2023-10-30T13:25:46.191 INFO:tasks.workunit.client.0.smithi046.stdout: create 131/.snap/snap-subdir-test 2023-10-30T13:25:46.219 INFO:tasks.workunit.client.0.smithi046.stdout: create 131/30 file after the snapshot 2023-10-30T13:25:46.220 INFO:tasks.workunit.client.0.smithi046.stdout: create 131/31 file after the snapshot 2023-10-30T13:25:46.220 INFO:tasks.workunit.client.0.smithi046.stdout: create 131/32 file after the snapshot 2023-10-30T13:25:46.221 INFO:tasks.workunit.client.0.smithi046.stdout: create 131/33 file after the snapshot 2023-10-30T13:25:46.222 INFO:tasks.workunit.client.0.smithi046.stdout: create 131/34 file after the snapshot 2023-10-30T13:25:46.222 INFO:tasks.workunit.client.0.smithi046.stdout: create 131/35 file after the snapshot 2023-10-30T13:25:46.222 INFO:tasks.workunit.client.0.smithi046.stdout: create 131/36 file after the snapshot 2023-10-30T13:25:46.222 INFO:tasks.workunit.client.0.smithi046.stdout: create 131/37 file after the snapshot 2023-10-30T13:25:46.222 INFO:tasks.workunit.client.0.smithi046.stdout: create 131/38 file after the snapshot 2023-10-30T13:25:46.223 INFO:tasks.workunit.client.0.smithi046.stdout: create 131/39 file after the snapshot 2023-10-30T13:25:46.223 INFO:tasks.workunit.client.0.smithi046.stdout: create 132/.snap/snap-subdir-test 2023-10-30T13:25:46.253 INFO:tasks.workunit.client.0.smithi046.stdout: create 132/30 file after the snapshot 2023-10-30T13:25:46.255 INFO:tasks.workunit.client.0.smithi046.stdout: create 132/31 file after the snapshot 2023-10-30T13:25:46.255 INFO:tasks.workunit.client.0.smithi046.stdout: create 132/32 file after the snapshot 2023-10-30T13:25:46.255 INFO:tasks.workunit.client.0.smithi046.stdout: create 132/33 file after the snapshot 2023-10-30T13:25:46.256 INFO:tasks.workunit.client.0.smithi046.stdout: create 132/34 file after the snapshot 2023-10-30T13:25:46.256 INFO:tasks.workunit.client.0.smithi046.stdout: create 132/35 file after the snapshot 2023-10-30T13:25:46.256 INFO:tasks.workunit.client.0.smithi046.stdout: create 132/36 file after the snapshot 2023-10-30T13:25:46.256 INFO:tasks.workunit.client.0.smithi046.stdout: create 132/37 file after the snapshot 2023-10-30T13:25:46.256 INFO:tasks.workunit.client.0.smithi046.stdout: create 132/38 file after the snapshot 2023-10-30T13:25:46.257 INFO:tasks.workunit.client.0.smithi046.stdout: create 132/39 file after the snapshot 2023-10-30T13:25:46.257 INFO:tasks.workunit.client.0.smithi046.stdout: create 133/.snap/snap-subdir-test 2023-10-30T13:25:46.288 INFO:tasks.workunit.client.0.smithi046.stdout: create 133/30 file after the snapshot 2023-10-30T13:25:46.289 INFO:tasks.workunit.client.0.smithi046.stdout: create 133/31 file after the snapshot 2023-10-30T13:25:46.290 INFO:tasks.workunit.client.0.smithi046.stdout: create 133/32 file after the snapshot 2023-10-30T13:25:46.290 INFO:tasks.workunit.client.0.smithi046.stdout: create 133/33 file after the snapshot 2023-10-30T13:25:46.290 INFO:tasks.workunit.client.0.smithi046.stdout: create 133/34 file after the snapshot 2023-10-30T13:25:46.290 INFO:tasks.workunit.client.0.smithi046.stdout: create 133/35 file after the snapshot 2023-10-30T13:25:46.290 INFO:tasks.workunit.client.0.smithi046.stdout: create 133/36 file after the snapshot 2023-10-30T13:25:46.291 INFO:tasks.workunit.client.0.smithi046.stdout: create 133/37 file after the snapshot 2023-10-30T13:25:46.291 INFO:tasks.workunit.client.0.smithi046.stdout: create 133/38 file after the snapshot 2023-10-30T13:25:46.291 INFO:tasks.workunit.client.0.smithi046.stdout: create 133/39 file after the snapshot 2023-10-30T13:25:46.291 INFO:tasks.workunit.client.0.smithi046.stdout: create 134/.snap/snap-subdir-test 2023-10-30T13:25:46.320 INFO:tasks.workunit.client.0.smithi046.stdout: create 134/30 file after the snapshot 2023-10-30T13:25:46.322 INFO:tasks.workunit.client.0.smithi046.stdout: create 134/31 file after the snapshot 2023-10-30T13:25:46.322 INFO:tasks.workunit.client.0.smithi046.stdout: create 134/32 file after the snapshot 2023-10-30T13:25:46.322 INFO:tasks.workunit.client.0.smithi046.stdout: create 134/33 file after the snapshot 2023-10-30T13:25:46.322 INFO:tasks.workunit.client.0.smithi046.stdout: create 134/34 file after the snapshot 2023-10-30T13:25:46.323 INFO:tasks.workunit.client.0.smithi046.stdout: create 134/35 file after the snapshot 2023-10-30T13:25:46.323 INFO:tasks.workunit.client.0.smithi046.stdout: create 134/36 file after the snapshot 2023-10-30T13:25:46.323 INFO:tasks.workunit.client.0.smithi046.stdout: create 134/37 file after the snapshot 2023-10-30T13:25:46.323 INFO:tasks.workunit.client.0.smithi046.stdout: create 134/38 file after the snapshot 2023-10-30T13:25:46.323 INFO:tasks.workunit.client.0.smithi046.stdout: create 134/39 file after the snapshot 2023-10-30T13:25:46.323 INFO:tasks.workunit.client.0.smithi046.stdout: create 135/.snap/snap-subdir-test 2023-10-30T13:25:46.801 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T13:25:46.801 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:25:46.802 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:25:46.806 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:25:46.806 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:46.900 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T13:25:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.955+0000 7f0b35394640 1 Processor -- start 2023-10-30T13:25:46.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.955+0000 7f0b35394640 1 -- start start 2023-10-30T13:25:46.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b35394640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b300c6bd0 0x7f0b3006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:46.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b35394640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0b30071110 0x7f0b30158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:46.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b35394640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b30066cf0 con 0x7f0b3006f7a0 2023-10-30T13:25:46.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b35394640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b300700c0 con 0x7f0b30071110 2023-10-30T13:25:46.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b35394640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300c70d0 con 0x7f0b300c6bd0 2023-10-30T13:25:46.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b27fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b300c6bd0 0x7f0b3006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:46.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b27fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b300c6bd0 0x7f0b3006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:47248/0 (socket says 172.21.15.46:47248) 2023-10-30T13:25:46.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b27fff640 1 -- 172.21.15.46:0/2801878158 learned_addr learned my addr 172.21.15.46:0/2801878158 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:46.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b2f7fe640 1 --2- 172.21.15.46:0/2801878158 >> v2:172.21.15.154:3300/0 conn(0x7f0b30071110 0x7f0b30158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:46.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b27fff640 1 -- 172.21.15.46:0/2801878158 >> v2:172.21.15.154:3300/0 conn(0x7f0b30071110 msgr2=0x7f0b30158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:46.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b27fff640 1 --2- 172.21.15.46:0/2801878158 >> v2:172.21.15.154:3300/0 conn(0x7f0b30071110 0x7f0b30158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:46.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b27fff640 1 -- 172.21.15.46:0/2801878158 >> v1:172.21.15.154:6789/0 conn(0x7f0b3006f7a0 legacy=0x7f0b30069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:46.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.959+0000 7f0b27fff640 1 -- 172.21.15.46:0/2801878158 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b300c7390 con 0x7f0b300c6bd0 2023-10-30T13:25:46.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b27fff640 1 --2- 172.21.15.46:0/2801878158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b300c6bd0 0x7f0b3006fb80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009870 tx=0x7f0b1801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ace3adbe7bc46cd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:46.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b2e7fc640 1 -- 172.21.15.46:0/2801878158 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b180a8a30 con 0x7f0b300c6bd0 2023-10-30T13:25:46.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b2e7fc640 1 -- 172.21.15.46:0/2801878158 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b180a8b90 con 0x7f0b300c6bd0 2023-10-30T13:25:46.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b2e7fc640 1 -- 172.21.15.46:0/2801878158 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b180b0440 con 0x7f0b300c6bd0 2023-10-30T13:25:46.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b35394640 1 -- 172.21.15.46:0/2801878158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b300c6bd0 msgr2=0x7f0b3006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:46.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b35394640 1 --2- 172.21.15.46:0/2801878158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b300c6bd0 0x7f0b3006fb80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009870 tx=0x7f0b1801ed30 comp rx=0 tx=0).stop 2023-10-30T13:25:46.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b35394640 1 -- 172.21.15.46:0/2801878158 shutdown_connections 2023-10-30T13:25:46.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b35394640 1 --2- 172.21.15.46:0/2801878158 >> v2:172.21.15.154:3300/0 conn(0x7f0b30071110 0x7f0b30158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:46.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b35394640 1 --2- 172.21.15.46:0/2801878158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b300c6bd0 0x7f0b3006fb80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:46.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b35394640 1 -- 172.21.15.46:0/2801878158 >> 172.21.15.46:0/2801878158 conn(0x7f0b300c7910 msgr2=0x7f0b300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:46.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b35394640 1 -- 172.21.15.46:0/2801878158 shutdown_connections 2023-10-30T13:25:46.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b35394640 1 -- 172.21.15.46:0/2801878158 wait complete. 2023-10-30T13:25:46.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b35394640 1 Processor -- start 2023-10-30T13:25:46.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.963+0000 7f0b35394640 1 -- start start 2023-10-30T13:25:46.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b35394640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3006f7a0 0x7f0b3015aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:46.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b35394640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b30071110 0x7f0b3014eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b35394640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b300c6bd0 0x7f0b301f0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b35394640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3014b0d0 con 0x7f0b3006f7a0 2023-10-30T13:25:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b35394640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b30159be0 con 0x7f0b30071110 2023-10-30T13:25:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b35394640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3015b4a0 con 0x7f0b300c6bd0 2023-10-30T13:25:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b2effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3006f7a0 0x7f0b3015aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:46.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b2effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3006f7a0 0x7f0b3015aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43162/0 (socket says 172.21.15.46:43162) 2023-10-30T13:25:46.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b2effd640 1 -- 172.21.15.46:0/2180034505 learned_addr learned my addr 172.21.15.46:0/2180034505 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:46.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b2f7fe640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b300c6bd0 0x7f0b301f0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:46.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b27fff640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b30071110 0x7f0b3014eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:46.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b2effd640 1 -- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b300c6bd0 msgr2=0x7f0b301f0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b2effd640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b300c6bd0 0x7f0b301f0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b2effd640 1 -- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b30071110 msgr2=0x7f0b3014eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b2effd640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b30071110 0x7f0b3014eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b2effd640 1 -- 172.21.15.46:0/2180034505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b180094d0 con 0x7f0b3006f7a0 2023-10-30T13:25:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b2effd640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3006f7a0 0x7f0b3015aa60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000c3c0 tx=0x7f0b2000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b277fe640 1 -- 172.21.15.46:0/2180034505 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b200052d0 con 0x7f0b3006f7a0 2023-10-30T13:25:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b277fe640 1 -- 172.21.15.46:0/2180034505 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b20005430 con 0x7f0b3006f7a0 2023-10-30T13:25:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b35394640 1 -- 172.21.15.46:0/2180034505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3014a770 con 0x7f0b3006f7a0 2023-10-30T13:25:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b277fe640 1 -- 172.21.15.46:0/2180034505 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20003840 con 0x7f0b3006f7a0 2023-10-30T13:25:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b35394640 1 -- 172.21.15.46:0/2180034505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3014aa10 con 0x7f0b3006f7a0 2023-10-30T13:25:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b35394640 1 -- 172.21.15.46:0/2180034505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3004e170 con 0x7f0b3006f7a0 2023-10-30T13:25:46.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.967+0000 7f0b277fe640 1 -- 172.21.15.46:0/2180034505 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0b200039a0 con 0x7f0b3006f7a0 2023-10-30T13:25:46.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.971+0000 7f0b277fe640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0b100664a0 0x7f0b10068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:46.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.971+0000 7f0b277fe640 1 -- 172.21.15.46:0/2180034505 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f0b20083470 con 0x7f0b3006f7a0 2023-10-30T13:25:46.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.971+0000 7f0b27fff640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0b100664a0 0x7f0b10068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:46.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.971+0000 7f0b27fff640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0b100664a0 0x7f0b10068960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0b180b2000 tx=0x7f0b18009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:46.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:46.975+0000 7f0b277fe640 1 -- 172.21.15.46:0/2180034505 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0b20050820 con 0x7f0b3006f7a0 2023-10-30T13:25:47.066 INFO:tasks.workunit.client.0.smithi046.stdout: create 135/30 file after the snapshot 2023-10-30T13:25:47.067 INFO:tasks.workunit.client.0.smithi046.stdout: create 135/31 file after the snapshot 2023-10-30T13:25:47.067 INFO:tasks.workunit.client.0.smithi046.stdout: create 135/32 file after the snapshot 2023-10-30T13:25:47.067 INFO:tasks.workunit.client.0.smithi046.stdout: create 135/33 file after the snapshot 2023-10-30T13:25:47.068 INFO:tasks.workunit.client.0.smithi046.stdout: create 135/34 file after the snapshot 2023-10-30T13:25:47.068 INFO:tasks.workunit.client.0.smithi046.stdout: create 135/35 file after the snapshot 2023-10-30T13:25:47.068 INFO:tasks.workunit.client.0.smithi046.stdout: create 135/36 file after the snapshot 2023-10-30T13:25:47.068 INFO:tasks.workunit.client.0.smithi046.stdout: create 135/37 file after the snapshot 2023-10-30T13:25:47.068 INFO:tasks.workunit.client.0.smithi046.stdout: create 135/38 file after the snapshot 2023-10-30T13:25:47.069 INFO:tasks.workunit.client.0.smithi046.stdout: create 135/39 file after the snapshot 2023-10-30T13:25:47.069 INFO:tasks.workunit.client.0.smithi046.stdout: create 136/.snap/snap-subdir-test 2023-10-30T13:25:47.098 INFO:tasks.workunit.client.0.smithi046.stdout: create 136/30 file after the snapshot 2023-10-30T13:25:47.099 INFO:tasks.workunit.client.0.smithi046.stdout: create 136/31 file after the snapshot 2023-10-30T13:25:47.099 INFO:tasks.workunit.client.0.smithi046.stdout: create 136/32 file after the snapshot 2023-10-30T13:25:47.100 INFO:tasks.workunit.client.0.smithi046.stdout: create 136/33 file after the snapshot 2023-10-30T13:25:47.100 INFO:tasks.workunit.client.0.smithi046.stdout: create 136/34 file after the snapshot 2023-10-30T13:25:47.100 INFO:tasks.workunit.client.0.smithi046.stdout: create 136/35 file after the snapshot 2023-10-30T13:25:47.100 INFO:tasks.workunit.client.0.smithi046.stdout: create 136/36 file after the snapshot 2023-10-30T13:25:47.100 INFO:tasks.workunit.client.0.smithi046.stdout: create 136/37 file after the snapshot 2023-10-30T13:25:47.101 INFO:tasks.workunit.client.0.smithi046.stdout: create 136/38 file after the snapshot 2023-10-30T13:25:47.101 INFO:tasks.workunit.client.0.smithi046.stdout: create 136/39 file after the snapshot 2023-10-30T13:25:47.101 INFO:tasks.workunit.client.0.smithi046.stdout: create 137/.snap/snap-subdir-test 2023-10-30T13:25:47.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.179+0000 7f0b35394640 1 -- 172.21.15.46:0/2180034505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b3014b210 con 0x7f0b3006f7a0 2023-10-30T13:25:47.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.179+0000 7f0b277fe640 1 -- 172.21.15.46:0/2180034505 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5731 (secure 0 0 0) 0x7f0b20053e40 con 0x7f0b3006f7a0 2023-10-30T13:25:47.184 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:47.185 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:35.725213+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":39,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:47.185 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 17 2023-10-30T13:25:47.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.183+0000 7f0b35394640 1 -- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0b100664a0 msgr2=0x7f0b10068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:47.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.183+0000 7f0b35394640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0b100664a0 0x7f0b10068960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0b180b2000 tx=0x7f0b18009040 comp rx=0 tx=0).stop 2023-10-30T13:25:47.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.183+0000 7f0b35394640 1 -- 172.21.15.46:0/2180034505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3006f7a0 msgr2=0x7f0b3015aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:47.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.183+0000 7f0b35394640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3006f7a0 0x7f0b3015aa60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000c3c0 tx=0x7f0b2000c880 comp rx=0 tx=0).stop 2023-10-30T13:25:47.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.183+0000 7f0b35394640 1 -- 172.21.15.46:0/2180034505 shutdown_connections 2023-10-30T13:25:47.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.183+0000 7f0b35394640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0b100664a0 0x7f0b10068960 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:47.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.183+0000 7f0b35394640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b300c6bd0 0x7f0b301f0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:47.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.183+0000 7f0b35394640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b30071110 0x7f0b3014eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:47.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.183+0000 7f0b35394640 1 --2- 172.21.15.46:0/2180034505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3006f7a0 0x7f0b3015aa60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:47.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.183+0000 7f0b35394640 1 -- 172.21.15.46:0/2180034505 >> 172.21.15.46:0/2180034505 conn(0x7f0b300c7910 msgr2=0x7f0b300c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:47.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.183+0000 7f0b35394640 1 -- 172.21.15.46:0/2180034505 shutdown_connections 2023-10-30T13:25:47.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:47.183+0000 7f0b35394640 1 -- 172.21.15.46:0/2180034505 wait complete. 2023-10-30T13:25:47.205 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T13:25:47.537 INFO:tasks.workunit.client.0.smithi046.stdout: create 137/30 file after the snapshot 2023-10-30T13:25:47.539 INFO:tasks.workunit.client.0.smithi046.stdout: create 137/31 file after the snapshot 2023-10-30T13:25:47.539 INFO:tasks.workunit.client.0.smithi046.stdout: create 137/32 file after the snapshot 2023-10-30T13:25:47.539 INFO:tasks.workunit.client.0.smithi046.stdout: create 137/33 file after the snapshot 2023-10-30T13:25:47.539 INFO:tasks.workunit.client.0.smithi046.stdout: create 137/34 file after the snapshot 2023-10-30T13:25:47.540 INFO:tasks.workunit.client.0.smithi046.stdout: create 137/35 file after the snapshot 2023-10-30T13:25:47.540 INFO:tasks.workunit.client.0.smithi046.stdout: create 137/36 file after the snapshot 2023-10-30T13:25:47.540 INFO:tasks.workunit.client.0.smithi046.stdout: create 137/37 file after the snapshot 2023-10-30T13:25:47.540 INFO:tasks.workunit.client.0.smithi046.stdout: create 137/38 file after the snapshot 2023-10-30T13:25:47.540 INFO:tasks.workunit.client.0.smithi046.stdout: create 137/39 file after the snapshot 2023-10-30T13:25:47.541 INFO:tasks.workunit.client.0.smithi046.stdout: create 138/.snap/snap-subdir-test 2023-10-30T13:25:47.569 INFO:tasks.workunit.client.0.smithi046.stdout: create 138/30 file after the snapshot 2023-10-30T13:25:47.570 INFO:tasks.workunit.client.0.smithi046.stdout: create 138/31 file after the snapshot 2023-10-30T13:25:47.571 INFO:tasks.workunit.client.0.smithi046.stdout: create 138/32 file after the snapshot 2023-10-30T13:25:47.571 INFO:tasks.workunit.client.0.smithi046.stdout: create 138/33 file after the snapshot 2023-10-30T13:25:47.571 INFO:tasks.workunit.client.0.smithi046.stdout: create 138/34 file after the snapshot 2023-10-30T13:25:47.571 INFO:tasks.workunit.client.0.smithi046.stdout: create 138/35 file after the snapshot 2023-10-30T13:25:47.571 INFO:tasks.workunit.client.0.smithi046.stdout: create 138/36 file after the snapshot 2023-10-30T13:25:47.572 INFO:tasks.workunit.client.0.smithi046.stdout: create 138/37 file after the snapshot 2023-10-30T13:25:47.572 INFO:tasks.workunit.client.0.smithi046.stdout: create 138/38 file after the snapshot 2023-10-30T13:25:47.572 INFO:tasks.workunit.client.0.smithi046.stdout: create 138/39 file after the snapshot 2023-10-30T13:25:47.572 INFO:tasks.workunit.client.0.smithi046.stdout: create 139/.snap/snap-subdir-test 2023-10-30T13:25:47.599 INFO:tasks.workunit.client.0.smithi046.stdout: create 139/30 file after the snapshot 2023-10-30T13:25:47.601 INFO:tasks.workunit.client.0.smithi046.stdout: create 139/31 file after the snapshot 2023-10-30T13:25:47.601 INFO:tasks.workunit.client.0.smithi046.stdout: create 139/32 file after the snapshot 2023-10-30T13:25:47.601 INFO:tasks.workunit.client.0.smithi046.stdout: create 139/33 file after the snapshot 2023-10-30T13:25:47.601 INFO:tasks.workunit.client.0.smithi046.stdout: create 139/34 file after the snapshot 2023-10-30T13:25:47.602 INFO:tasks.workunit.client.0.smithi046.stdout: create 139/35 file after the snapshot 2023-10-30T13:25:47.602 INFO:tasks.workunit.client.0.smithi046.stdout: create 139/36 file after the snapshot 2023-10-30T13:25:47.602 INFO:tasks.workunit.client.0.smithi046.stdout: create 139/37 file after the snapshot 2023-10-30T13:25:47.602 INFO:tasks.workunit.client.0.smithi046.stdout: create 139/38 file after the snapshot 2023-10-30T13:25:47.602 INFO:tasks.workunit.client.0.smithi046.stdout: create 139/39 file after the snapshot 2023-10-30T13:25:47.603 INFO:tasks.workunit.client.0.smithi046.stdout: create 140/.snap/snap-subdir-test 2023-10-30T13:25:47.881 INFO:tasks.workunit.client.0.smithi046.stdout: create 140/30 file after the snapshot 2023-10-30T13:25:47.883 INFO:tasks.workunit.client.0.smithi046.stdout: create 140/31 file after the snapshot 2023-10-30T13:25:47.883 INFO:tasks.workunit.client.0.smithi046.stdout: create 140/32 file after the snapshot 2023-10-30T13:25:47.883 INFO:tasks.workunit.client.0.smithi046.stdout: create 140/33 file after the snapshot 2023-10-30T13:25:47.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 140/34 file after the snapshot 2023-10-30T13:25:47.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 140/35 file after the snapshot 2023-10-30T13:25:47.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 140/36 file after the snapshot 2023-10-30T13:25:47.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 140/37 file after the snapshot 2023-10-30T13:25:47.885 INFO:tasks.workunit.client.0.smithi046.stdout: create 140/38 file after the snapshot 2023-10-30T13:25:47.885 INFO:tasks.workunit.client.0.smithi046.stdout: create 140/39 file after the snapshot 2023-10-30T13:25:47.885 INFO:tasks.workunit.client.0.smithi046.stdout: create 141/.snap/snap-subdir-test 2023-10-30T13:25:48.165 INFO:tasks.workunit.client.0.smithi046.stdout: create 141/30 file after the snapshot 2023-10-30T13:25:48.167 INFO:tasks.workunit.client.0.smithi046.stdout: create 141/31 file after the snapshot 2023-10-30T13:25:48.167 INFO:tasks.workunit.client.0.smithi046.stdout: create 141/32 file after the snapshot 2023-10-30T13:25:48.167 INFO:tasks.workunit.client.0.smithi046.stdout: create 141/33 file after the snapshot 2023-10-30T13:25:48.168 INFO:tasks.workunit.client.0.smithi046.stdout: create 141/34 file after the snapshot 2023-10-30T13:25:48.168 INFO:tasks.workunit.client.0.smithi046.stdout: create 141/35 file after the snapshot 2023-10-30T13:25:48.168 INFO:tasks.workunit.client.0.smithi046.stdout: create 141/36 file after the snapshot 2023-10-30T13:25:48.168 INFO:tasks.workunit.client.0.smithi046.stdout: create 141/37 file after the snapshot 2023-10-30T13:25:48.168 INFO:tasks.workunit.client.0.smithi046.stdout: create 141/38 file after the snapshot 2023-10-30T13:25:48.169 INFO:tasks.workunit.client.0.smithi046.stdout: create 141/39 file after the snapshot 2023-10-30T13:25:48.169 INFO:tasks.workunit.client.0.smithi046.stdout: create 142/.snap/snap-subdir-test 2023-10-30T13:25:48.198 INFO:tasks.workunit.client.0.smithi046.stdout: create 142/30 file after the snapshot 2023-10-30T13:25:48.200 INFO:tasks.workunit.client.0.smithi046.stdout: create 142/31 file after the snapshot 2023-10-30T13:25:48.200 INFO:tasks.workunit.client.0.smithi046.stdout: create 142/32 file after the snapshot 2023-10-30T13:25:48.201 INFO:tasks.workunit.client.0.smithi046.stdout: create 142/33 file after the snapshot 2023-10-30T13:25:48.201 INFO:tasks.workunit.client.0.smithi046.stdout: create 142/34 file after the snapshot 2023-10-30T13:25:48.201 INFO:tasks.workunit.client.0.smithi046.stdout: create 142/35 file after the snapshot 2023-10-30T13:25:48.201 INFO:tasks.workunit.client.0.smithi046.stdout: create 142/36 file after the snapshot 2023-10-30T13:25:48.201 INFO:tasks.workunit.client.0.smithi046.stdout: create 142/37 file after the snapshot 2023-10-30T13:25:48.202 INFO:tasks.workunit.client.0.smithi046.stdout: create 142/38 file after the snapshot 2023-10-30T13:25:48.202 INFO:tasks.workunit.client.0.smithi046.stdout: create 142/39 file after the snapshot 2023-10-30T13:25:48.203 INFO:tasks.workunit.client.0.smithi046.stdout: create 143/.snap/snap-subdir-test 2023-10-30T13:25:48.229 INFO:tasks.workunit.client.0.smithi046.stdout: create 143/30 file after the snapshot 2023-10-30T13:25:48.231 INFO:tasks.workunit.client.0.smithi046.stdout: create 143/31 file after the snapshot 2023-10-30T13:25:48.231 INFO:tasks.workunit.client.0.smithi046.stdout: create 143/32 file after the snapshot 2023-10-30T13:25:48.231 INFO:tasks.workunit.client.0.smithi046.stdout: create 143/33 file after the snapshot 2023-10-30T13:25:48.231 INFO:tasks.workunit.client.0.smithi046.stdout: create 143/34 file after the snapshot 2023-10-30T13:25:48.231 INFO:tasks.workunit.client.0.smithi046.stdout: create 143/35 file after the snapshot 2023-10-30T13:25:48.232 INFO:tasks.workunit.client.0.smithi046.stdout: create 143/36 file after the snapshot 2023-10-30T13:25:48.232 INFO:tasks.workunit.client.0.smithi046.stdout: create 143/37 file after the snapshot 2023-10-30T13:25:48.232 INFO:tasks.workunit.client.0.smithi046.stdout: create 143/38 file after the snapshot 2023-10-30T13:25:48.232 INFO:tasks.workunit.client.0.smithi046.stdout: create 143/39 file after the snapshot 2023-10-30T13:25:48.232 INFO:tasks.workunit.client.0.smithi046.stdout: create 144/.snap/snap-subdir-test 2023-10-30T13:25:48.262 INFO:tasks.workunit.client.0.smithi046.stdout: create 144/30 file after the snapshot 2023-10-30T13:25:48.264 INFO:tasks.workunit.client.0.smithi046.stdout: create 144/31 file after the snapshot 2023-10-30T13:25:48.264 INFO:tasks.workunit.client.0.smithi046.stdout: create 144/32 file after the snapshot 2023-10-30T13:25:48.264 INFO:tasks.workunit.client.0.smithi046.stdout: create 144/33 file after the snapshot 2023-10-30T13:25:48.264 INFO:tasks.workunit.client.0.smithi046.stdout: create 144/34 file after the snapshot 2023-10-30T13:25:48.265 INFO:tasks.workunit.client.0.smithi046.stdout: create 144/35 file after the snapshot 2023-10-30T13:25:48.265 INFO:tasks.workunit.client.0.smithi046.stdout: create 144/36 file after the snapshot 2023-10-30T13:25:48.265 INFO:tasks.workunit.client.0.smithi046.stdout: create 144/37 file after the snapshot 2023-10-30T13:25:48.265 INFO:tasks.workunit.client.0.smithi046.stdout: create 144/38 file after the snapshot 2023-10-30T13:25:48.265 INFO:tasks.workunit.client.0.smithi046.stdout: create 144/39 file after the snapshot 2023-10-30T13:25:48.266 INFO:tasks.workunit.client.0.smithi046.stdout: create 145/.snap/snap-subdir-test 2023-10-30T13:25:48.825 INFO:tasks.workunit.client.0.smithi046.stdout: create 145/30 file after the snapshot 2023-10-30T13:25:48.827 INFO:tasks.workunit.client.0.smithi046.stdout: create 145/31 file after the snapshot 2023-10-30T13:25:48.827 INFO:tasks.workunit.client.0.smithi046.stdout: create 145/32 file after the snapshot 2023-10-30T13:25:48.827 INFO:tasks.workunit.client.0.smithi046.stdout: create 145/33 file after the snapshot 2023-10-30T13:25:48.827 INFO:tasks.workunit.client.0.smithi046.stdout: create 145/34 file after the snapshot 2023-10-30T13:25:48.827 INFO:tasks.workunit.client.0.smithi046.stdout: create 145/35 file after the snapshot 2023-10-30T13:25:48.828 INFO:tasks.workunit.client.0.smithi046.stdout: create 145/36 file after the snapshot 2023-10-30T13:25:48.828 INFO:tasks.workunit.client.0.smithi046.stdout: create 145/37 file after the snapshot 2023-10-30T13:25:48.828 INFO:tasks.workunit.client.0.smithi046.stdout: create 145/38 file after the snapshot 2023-10-30T13:25:48.828 INFO:tasks.workunit.client.0.smithi046.stdout: create 145/39 file after the snapshot 2023-10-30T13:25:48.828 INFO:tasks.workunit.client.0.smithi046.stdout: create 146/.snap/snap-subdir-test 2023-10-30T13:25:49.110 INFO:tasks.workunit.client.0.smithi046.stdout: create 146/30 file after the snapshot 2023-10-30T13:25:49.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:49.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f210533e640 1 Processor -- start 2023-10-30T13:25:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f210533e640 1 -- start start 2023-10-30T13:25:49.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f210533e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f21000c6bd0 0x7f210006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:49.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f210533e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2100071110 0x7f2100158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:49.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f210533e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2100066cf0 con 0x7f210006f7a0 2023-10-30T13:25:49.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f210533e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21000700c0 con 0x7f21000c6bd0 2023-10-30T13:25:49.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f210533e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21000c70d0 con 0x7f2100071110 2023-10-30T13:25:49.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f20feffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f210006f7a0 0x7f2100069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58460/0 (socket says 172.21.15.46:58460) 2023-10-30T13:25:49.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f20feffd640 1 -- 172.21.15.46:0/2421995324 learned_addr learned my addr 172.21.15.46:0/2421995324 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:25:49.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f20f7fff640 1 --2- 172.21.15.46:0/2421995324 >> v2:172.21.15.46:3300/0 conn(0x7f21000c6bd0 0x7f210006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:49.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f20ff7fe640 1 --2- 172.21.15.46:0/2421995324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2100071110 0x7f2100158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f20fe7fc640 1 -- 172.21.15.46:0/2421995324 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646264850 0 0) 0x7f2100066cf0 con 0x7f210006f7a0 2023-10-30T13:25:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f20f7fff640 1 -- 172.21.15.46:0/2421995324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2100071110 msgr2=0x7f2100158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f20f7fff640 1 --2- 172.21.15.46:0/2421995324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2100071110 0x7f2100158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f20f7fff640 1 -- 172.21.15.46:0/2421995324 >> v1:172.21.15.46:6789/0 conn(0x7f210006f7a0 legacy=0x7f2100069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.351+0000 7f20f7fff640 1 -- 172.21.15.46:0/2421995324 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21000c7390 con 0x7f21000c6bd0 2023-10-30T13:25:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.355+0000 7f20f7fff640 1 --2- 172.21.15.46:0/2421995324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21000c6bd0 0x7f210006fb80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f20e80097c0 tx=0x7f20e801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4647a9cefcd84cb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.355+0000 7f20fe7fc640 1 -- 172.21.15.46:0/2421995324 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e80a8970 con 0x7f21000c6bd0 2023-10-30T13:25:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.355+0000 7f20fe7fc640 1 -- 172.21.15.46:0/2421995324 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e80a8ad0 con 0x7f21000c6bd0 2023-10-30T13:25:49.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.355+0000 7f20fe7fc640 1 -- 172.21.15.46:0/2421995324 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e80a8e60 con 0x7f21000c6bd0 2023-10-30T13:25:49.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.355+0000 7f210533e640 1 -- 172.21.15.46:0/2421995324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21000c6bd0 msgr2=0x7f210006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:49.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.355+0000 7f210533e640 1 --2- 172.21.15.46:0/2421995324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21000c6bd0 0x7f210006fb80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f20e80097c0 tx=0x7f20e801ec70 comp rx=0 tx=0).stop 2023-10-30T13:25:49.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.355+0000 7f210533e640 1 -- 172.21.15.46:0/2421995324 shutdown_connections 2023-10-30T13:25:49.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.355+0000 7f210533e640 1 --2- 172.21.15.46:0/2421995324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2100071110 0x7f2100158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:49.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.355+0000 7f210533e640 1 --2- 172.21.15.46:0/2421995324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21000c6bd0 0x7f210006fb80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:49.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.355+0000 7f210533e640 1 -- 172.21.15.46:0/2421995324 >> 172.21.15.46:0/2421995324 conn(0x7f21000c7910 msgr2=0x7f21000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:49.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.355+0000 7f210533e640 1 -- 172.21.15.46:0/2421995324 shutdown_connections 2023-10-30T13:25:49.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.355+0000 7f210533e640 1 -- 172.21.15.46:0/2421995324 wait complete. 2023-10-30T13:25:49.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f210533e640 1 Processor -- start 2023-10-30T13:25:49.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f210533e640 1 -- start start 2023-10-30T13:25:49.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f210533e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f210006f7a0 0x7f2100157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:49.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f210533e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2100071110 0x7f21000c3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:49.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f210533e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21000c6bd0 0x7f2100151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:49.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f210533e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2100154ae0 con 0x7f21000c6bd0 2023-10-30T13:25:49.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f210533e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2100155c30 con 0x7f210006f7a0 2023-10-30T13:25:49.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f210533e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21000c9780 con 0x7f2100071110 2023-10-30T13:25:49.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21000c6bd0 0x7f2100151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:49.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21000c6bd0 0x7f2100151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43180/0 (socket says 172.21.15.46:43180) 2023-10-30T13:25:49.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20ff7fe640 1 -- 172.21.15.46:0/3365986027 learned_addr learned my addr 172.21.15.46:0/3365986027 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:25:49.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20feffd640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f210006f7a0 0x7f2100157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:49.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20f7fff640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2100071110 0x7f21000c3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20ff7fe640 1 -- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2100071110 msgr2=0x7f21000c3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20ff7fe640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2100071110 0x7f21000c3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20ff7fe640 1 -- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f210006f7a0 msgr2=0x7f2100157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20ff7fe640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f210006f7a0 0x7f2100157930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20ff7fe640 1 -- 172.21.15.46:0/3365986027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e8009420 con 0x7f21000c6bd0 2023-10-30T13:25:49.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20ff7fe640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21000c6bd0 0x7f2100151940 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f20f8007e40 tx=0x7f20f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:49.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20f77fe640 1 -- 172.21.15.46:0/3365986027 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8014620 con 0x7f21000c6bd0 2023-10-30T13:25:49.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20f77fe640 1 -- 172.21.15.46:0/3365986027 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f800ee90 con 0x7f21000c6bd0 2023-10-30T13:25:49.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f20f77fe640 1 -- 172.21.15.46:0/3365986027 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f80149e0 con 0x7f21000c6bd0 2023-10-30T13:25:49.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f210533e640 1 -- 172.21.15.46:0/3365986027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2100157f50 con 0x7f21000c6bd0 2023-10-30T13:25:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f210533e640 1 -- 172.21.15.46:0/3365986027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2100155300 con 0x7f21000c6bd0 2023-10-30T13:25:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.359+0000 7f210533e640 1 -- 172.21.15.46:0/3365986027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2100070330 con 0x7f21000c6bd0 2023-10-30T13:25:49.369 INFO:tasks.workunit.client.0.smithi046.stdout: create 146/31 file after the snapshot 2023-10-30T13:25:49.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.363+0000 7f20f77fe640 1 -- 172.21.15.46:0/3365986027 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f20f8014b40 con 0x7f21000c6bd0 2023-10-30T13:25:49.371 INFO:tasks.workunit.client.0.smithi046.stdout: create 146/32 file after the snapshot 2023-10-30T13:25:49.371 INFO:tasks.workunit.client.0.smithi046.stdout: create 146/33 file after the snapshot 2023-10-30T13:25:49.371 INFO:tasks.workunit.client.0.smithi046.stdout: create 146/34 file after the snapshot 2023-10-30T13:25:49.371 INFO:tasks.workunit.client.0.smithi046.stdout: create 146/35 file after the snapshot 2023-10-30T13:25:49.372 INFO:tasks.workunit.client.0.smithi046.stdout: create 146/36 file after the snapshot 2023-10-30T13:25:49.372 INFO:tasks.workunit.client.0.smithi046.stdout: create 146/37 file after the snapshot 2023-10-30T13:25:49.372 INFO:tasks.workunit.client.0.smithi046.stdout: create 146/38 file after the snapshot 2023-10-30T13:25:49.372 INFO:tasks.workunit.client.0.smithi046.stdout: create 146/39 file after the snapshot 2023-10-30T13:25:49.372 INFO:tasks.workunit.client.0.smithi046.stdout: create 147/.snap/snap-subdir-test 2023-10-30T13:25:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.363+0000 7f20f77fe640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20e00664a0 0x7f20e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.363+0000 7f20f77fe640 1 -- 172.21.15.46:0/3365986027 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f20f80847e0 con 0x7f21000c6bd0 2023-10-30T13:25:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.363+0000 7f20feffd640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20e00664a0 0x7f20e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.363+0000 7f20feffd640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20e00664a0 0x7f20e0068960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f20f0000f70 tx=0x7f20f0007df0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.367+0000 7f20f77fe640 1 -- 172.21.15.46:0/3365986027 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2100070330 con 0x7f21000c6bd0 2023-10-30T13:25:49.398 INFO:tasks.workunit.client.0.smithi046.stdout: create 147/30 file after the snapshot 2023-10-30T13:25:49.401 INFO:tasks.workunit.client.0.smithi046.stdout: create 147/31 file after the snapshot 2023-10-30T13:25:49.401 INFO:tasks.workunit.client.0.smithi046.stdout: create 147/32 file after the snapshot 2023-10-30T13:25:49.401 INFO:tasks.workunit.client.0.smithi046.stdout: create 147/33 file after the snapshot 2023-10-30T13:25:49.401 INFO:tasks.workunit.client.0.smithi046.stdout: create 147/34 file after the snapshot 2023-10-30T13:25:49.401 INFO:tasks.workunit.client.0.smithi046.stdout: create 147/35 file after the snapshot 2023-10-30T13:25:49.402 INFO:tasks.workunit.client.0.smithi046.stdout: create 147/36 file after the snapshot 2023-10-30T13:25:49.402 INFO:tasks.workunit.client.0.smithi046.stdout: create 147/37 file after the snapshot 2023-10-30T13:25:49.402 INFO:tasks.workunit.client.0.smithi046.stdout: create 147/38 file after the snapshot 2023-10-30T13:25:49.402 INFO:tasks.workunit.client.0.smithi046.stdout: create 147/39 file after the snapshot 2023-10-30T13:25:49.402 INFO:tasks.workunit.client.0.smithi046.stdout: create 148/.snap/snap-subdir-test 2023-10-30T13:25:49.439 INFO:tasks.workunit.client.0.smithi046.stdout: create 148/30 file after the snapshot 2023-10-30T13:25:49.441 INFO:tasks.workunit.client.0.smithi046.stdout: create 148/31 file after the snapshot 2023-10-30T13:25:49.442 INFO:tasks.workunit.client.0.smithi046.stdout: create 148/32 file after the snapshot 2023-10-30T13:25:49.443 INFO:tasks.workunit.client.0.smithi046.stdout: create 148/33 file after the snapshot 2023-10-30T13:25:49.443 INFO:tasks.workunit.client.0.smithi046.stdout: create 148/34 file after the snapshot 2023-10-30T13:25:49.443 INFO:tasks.workunit.client.0.smithi046.stdout: create 148/35 file after the snapshot 2023-10-30T13:25:49.443 INFO:tasks.workunit.client.0.smithi046.stdout: create 148/36 file after the snapshot 2023-10-30T13:25:49.444 INFO:tasks.workunit.client.0.smithi046.stdout: create 148/37 file after the snapshot 2023-10-30T13:25:49.444 INFO:tasks.workunit.client.0.smithi046.stdout: create 148/38 file after the snapshot 2023-10-30T13:25:49.444 INFO:tasks.workunit.client.0.smithi046.stdout: create 148/39 file after the snapshot 2023-10-30T13:25:49.444 INFO:tasks.workunit.client.0.smithi046.stdout: create 149/.snap/snap-subdir-test 2023-10-30T13:25:49.481 INFO:tasks.workunit.client.0.smithi046.stdout: create 149/30 file after the snapshot 2023-10-30T13:25:49.483 INFO:tasks.workunit.client.0.smithi046.stdout: create 149/31 file after the snapshot 2023-10-30T13:25:49.483 INFO:tasks.workunit.client.0.smithi046.stdout: create 149/32 file after the snapshot 2023-10-30T13:25:49.483 INFO:tasks.workunit.client.0.smithi046.stdout: create 149/33 file after the snapshot 2023-10-30T13:25:49.483 INFO:tasks.workunit.client.0.smithi046.stdout: create 149/34 file after the snapshot 2023-10-30T13:25:49.483 INFO:tasks.workunit.client.0.smithi046.stdout: create 149/35 file after the snapshot 2023-10-30T13:25:49.484 INFO:tasks.workunit.client.0.smithi046.stdout: create 149/36 file after the snapshot 2023-10-30T13:25:49.484 INFO:tasks.workunit.client.0.smithi046.stdout: create 149/37 file after the snapshot 2023-10-30T13:25:49.484 INFO:tasks.workunit.client.0.smithi046.stdout: create 149/38 file after the snapshot 2023-10-30T13:25:49.484 INFO:tasks.workunit.client.0.smithi046.stdout: create 149/39 file after the snapshot 2023-10-30T13:25:49.484 INFO:tasks.workunit.client.0.smithi046.stdout: create 150/.snap/snap-subdir-test 2023-10-30T13:25:49.523 INFO:tasks.workunit.client.0.smithi046.stdout: create 150/30 file after the snapshot 2023-10-30T13:25:49.524 INFO:tasks.workunit.client.0.smithi046.stdout: create 150/31 file after the snapshot 2023-10-30T13:25:49.524 INFO:tasks.workunit.client.0.smithi046.stdout: create 150/32 file after the snapshot 2023-10-30T13:25:49.525 INFO:tasks.workunit.client.0.smithi046.stdout: create 150/33 file after the snapshot 2023-10-30T13:25:49.525 INFO:tasks.workunit.client.0.smithi046.stdout: create 150/34 file after the snapshot 2023-10-30T13:25:49.525 INFO:tasks.workunit.client.0.smithi046.stdout: create 150/35 file after the snapshot 2023-10-30T13:25:49.525 INFO:tasks.workunit.client.0.smithi046.stdout: create 150/36 file after the snapshot 2023-10-30T13:25:49.525 INFO:tasks.workunit.client.0.smithi046.stdout: create 150/37 file after the snapshot 2023-10-30T13:25:49.526 INFO:tasks.workunit.client.0.smithi046.stdout: create 150/38 file after the snapshot 2023-10-30T13:25:49.526 INFO:tasks.workunit.client.0.smithi046.stdout: create 150/39 file after the snapshot 2023-10-30T13:25:49.526 INFO:tasks.workunit.client.0.smithi046.stdout: create 151/.snap/snap-subdir-test 2023-10-30T13:25:49.562 INFO:tasks.workunit.client.0.smithi046.stdout: create 151/30 file after the snapshot 2023-10-30T13:25:49.564 INFO:tasks.workunit.client.0.smithi046.stdout: create 151/31 file after the snapshot 2023-10-30T13:25:49.564 INFO:tasks.workunit.client.0.smithi046.stdout: create 151/32 file after the snapshot 2023-10-30T13:25:49.564 INFO:tasks.workunit.client.0.smithi046.stdout: create 151/33 file after the snapshot 2023-10-30T13:25:49.564 INFO:tasks.workunit.client.0.smithi046.stdout: create 151/34 file after the snapshot 2023-10-30T13:25:49.565 INFO:tasks.workunit.client.0.smithi046.stdout: create 151/35 file after the snapshot 2023-10-30T13:25:49.565 INFO:tasks.workunit.client.0.smithi046.stdout: create 151/36 file after the snapshot 2023-10-30T13:25:49.565 INFO:tasks.workunit.client.0.smithi046.stdout: create 151/37 file after the snapshot 2023-10-30T13:25:49.565 INFO:tasks.workunit.client.0.smithi046.stdout: create 151/38 file after the snapshot 2023-10-30T13:25:49.565 INFO:tasks.workunit.client.0.smithi046.stdout: create 151/39 file after the snapshot 2023-10-30T13:25:49.565 INFO:tasks.workunit.client.0.smithi046.stdout: create 152/.snap/snap-subdir-test 2023-10-30T13:25:49.601 INFO:tasks.workunit.client.0.smithi046.stdout: create 152/30 file after the snapshot 2023-10-30T13:25:49.603 INFO:tasks.workunit.client.0.smithi046.stdout: create 152/31 file after the snapshot 2023-10-30T13:25:49.604 INFO:tasks.workunit.client.0.smithi046.stdout: create 152/32 file after the snapshot 2023-10-30T13:25:49.604 INFO:tasks.workunit.client.0.smithi046.stdout: create 152/33 file after the snapshot 2023-10-30T13:25:49.604 INFO:tasks.workunit.client.0.smithi046.stdout: create 152/34 file after the snapshot 2023-10-30T13:25:49.605 INFO:tasks.workunit.client.0.smithi046.stdout: create 152/35 file after the snapshot 2023-10-30T13:25:49.605 INFO:tasks.workunit.client.0.smithi046.stdout: create 152/36 file after the snapshot 2023-10-30T13:25:49.605 INFO:tasks.workunit.client.0.smithi046.stdout: create 152/37 file after the snapshot 2023-10-30T13:25:49.606 INFO:tasks.workunit.client.0.smithi046.stdout: create 152/38 file after the snapshot 2023-10-30T13:25:49.606 INFO:tasks.workunit.client.0.smithi046.stdout: create 152/39 file after the snapshot 2023-10-30T13:25:49.606 INFO:tasks.workunit.client.0.smithi046.stdout: create 153/.snap/snap-subdir-test 2023-10-30T13:25:49.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 -- 172.21.15.46:0/3365986027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2100070330 con 0x7f21000c6bd0 2023-10-30T13:25:49.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f20f77fe640 1 -- 172.21.15.46:0/3365986027 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6522 (secure 0 0 0) 0x7f2100070330 con 0x7f21000c6bd0 2023-10-30T13:25:49.611 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:25:49.611 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4950,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:25:35.725213+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":39,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:49.611 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 18 2023-10-30T13:25:49.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 -- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20e00664a0 msgr2=0x7f20e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:49.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20e00664a0 0x7f20e0068960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f20f0000f70 tx=0x7f20f0007df0 comp rx=0 tx=0).stop 2023-10-30T13:25:49.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 -- 172.21.15.46:0/3365986027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21000c6bd0 msgr2=0x7f2100151940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:49.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21000c6bd0 0x7f2100151940 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f20f8007e40 tx=0x7f20f800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:25:49.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 -- 172.21.15.46:0/3365986027 shutdown_connections 2023-10-30T13:25:49.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20e00664a0 0x7f20e0068960 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:49.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21000c6bd0 0x7f2100151940 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:49.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2100071110 0x7f21000c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:49.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 --2- 172.21.15.46:0/3365986027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f210006f7a0 0x7f2100157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:49.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 -- 172.21.15.46:0/3365986027 >> 172.21.15.46:0/3365986027 conn(0x7f21000c7910 msgr2=0x7f21000c8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:49.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 -- 172.21.15.46:0/3365986027 shutdown_connections 2023-10-30T13:25:49.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:25:49.607+0000 7f210533e640 1 -- 172.21.15.46:0/3365986027 wait complete. 2023-10-30T13:25:49.637 INFO:tasks.workunit.client.0.smithi046.stdout: create 153/30 file after the snapshot 2023-10-30T13:25:49.638 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T13:25:49.639 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 23 secs before thrashing 2023-10-30T13:25:49.640 INFO:tasks.workunit.client.0.smithi046.stdout: create 153/31 file after the snapshot 2023-10-30T13:25:49.640 INFO:tasks.workunit.client.0.smithi046.stdout: create 153/32 file after the snapshot 2023-10-30T13:25:49.640 INFO:tasks.workunit.client.0.smithi046.stdout: create 153/33 file after the snapshot 2023-10-30T13:25:49.641 INFO:tasks.workunit.client.0.smithi046.stdout: create 153/34 file after the snapshot 2023-10-30T13:25:49.641 INFO:tasks.workunit.client.0.smithi046.stdout: create 153/35 file after the snapshot 2023-10-30T13:25:49.641 INFO:tasks.workunit.client.0.smithi046.stdout: create 153/36 file after the snapshot 2023-10-30T13:25:49.641 INFO:tasks.workunit.client.0.smithi046.stdout: create 153/37 file after the snapshot 2023-10-30T13:25:49.641 INFO:tasks.workunit.client.0.smithi046.stdout: create 153/38 file after the snapshot 2023-10-30T13:25:49.642 INFO:tasks.workunit.client.0.smithi046.stdout: create 153/39 file after the snapshot 2023-10-30T13:25:49.642 INFO:tasks.workunit.client.0.smithi046.stdout: create 154/.snap/snap-subdir-test 2023-10-30T13:25:49.676 INFO:tasks.workunit.client.0.smithi046.stdout: create 154/30 file after the snapshot 2023-10-30T13:25:49.678 INFO:tasks.workunit.client.0.smithi046.stdout: create 154/31 file after the snapshot 2023-10-30T13:25:49.678 INFO:tasks.workunit.client.0.smithi046.stdout: create 154/32 file after the snapshot 2023-10-30T13:25:49.678 INFO:tasks.workunit.client.0.smithi046.stdout: create 154/33 file after the snapshot 2023-10-30T13:25:49.678 INFO:tasks.workunit.client.0.smithi046.stdout: create 154/34 file after the snapshot 2023-10-30T13:25:49.678 INFO:tasks.workunit.client.0.smithi046.stdout: create 154/35 file after the snapshot 2023-10-30T13:25:49.679 INFO:tasks.workunit.client.0.smithi046.stdout: create 154/36 file after the snapshot 2023-10-30T13:25:49.679 INFO:tasks.workunit.client.0.smithi046.stdout: create 154/37 file after the snapshot 2023-10-30T13:25:49.679 INFO:tasks.workunit.client.0.smithi046.stdout: create 154/38 file after the snapshot 2023-10-30T13:25:49.680 INFO:tasks.workunit.client.0.smithi046.stdout: create 154/39 file after the snapshot 2023-10-30T13:25:49.680 INFO:tasks.workunit.client.0.smithi046.stdout: create 155/.snap/snap-subdir-test 2023-10-30T13:25:49.716 INFO:tasks.workunit.client.0.smithi046.stdout: create 155/30 file after the snapshot 2023-10-30T13:25:49.718 INFO:tasks.workunit.client.0.smithi046.stdout: create 155/31 file after the snapshot 2023-10-30T13:25:49.718 INFO:tasks.workunit.client.0.smithi046.stdout: create 155/32 file after the snapshot 2023-10-30T13:25:49.718 INFO:tasks.workunit.client.0.smithi046.stdout: create 155/33 file after the snapshot 2023-10-30T13:25:49.719 INFO:tasks.workunit.client.0.smithi046.stdout: create 155/34 file after the snapshot 2023-10-30T13:25:49.719 INFO:tasks.workunit.client.0.smithi046.stdout: create 155/35 file after the snapshot 2023-10-30T13:25:49.719 INFO:tasks.workunit.client.0.smithi046.stdout: create 155/36 file after the snapshot 2023-10-30T13:25:49.719 INFO:tasks.workunit.client.0.smithi046.stdout: create 155/37 file after the snapshot 2023-10-30T13:25:49.719 INFO:tasks.workunit.client.0.smithi046.stdout: create 155/38 file after the snapshot 2023-10-30T13:25:49.720 INFO:tasks.workunit.client.0.smithi046.stdout: create 155/39 file after the snapshot 2023-10-30T13:25:49.720 INFO:tasks.workunit.client.0.smithi046.stdout: create 156/.snap/snap-subdir-test 2023-10-30T13:25:49.757 INFO:tasks.workunit.client.0.smithi046.stdout: create 156/30 file after the snapshot 2023-10-30T13:25:49.760 INFO:tasks.workunit.client.0.smithi046.stdout: create 156/31 file after the snapshot 2023-10-30T13:25:49.760 INFO:tasks.workunit.client.0.smithi046.stdout: create 156/32 file after the snapshot 2023-10-30T13:25:49.760 INFO:tasks.workunit.client.0.smithi046.stdout: create 156/33 file after the snapshot 2023-10-30T13:25:49.761 INFO:tasks.workunit.client.0.smithi046.stdout: create 156/34 file after the snapshot 2023-10-30T13:25:49.761 INFO:tasks.workunit.client.0.smithi046.stdout: create 156/35 file after the snapshot 2023-10-30T13:25:49.761 INFO:tasks.workunit.client.0.smithi046.stdout: create 156/36 file after the snapshot 2023-10-30T13:25:49.762 INFO:tasks.workunit.client.0.smithi046.stdout: create 156/37 file after the snapshot 2023-10-30T13:25:49.762 INFO:tasks.workunit.client.0.smithi046.stdout: create 156/38 file after the snapshot 2023-10-30T13:25:49.763 INFO:tasks.workunit.client.0.smithi046.stdout: create 156/39 file after the snapshot 2023-10-30T13:25:49.763 INFO:tasks.workunit.client.0.smithi046.stdout: create 157/.snap/snap-subdir-test 2023-10-30T13:25:49.798 INFO:tasks.workunit.client.0.smithi046.stdout: create 157/30 file after the snapshot 2023-10-30T13:25:49.799 INFO:tasks.workunit.client.0.smithi046.stdout: create 157/31 file after the snapshot 2023-10-30T13:25:49.799 INFO:tasks.workunit.client.0.smithi046.stdout: create 157/32 file after the snapshot 2023-10-30T13:25:49.800 INFO:tasks.workunit.client.0.smithi046.stdout: create 157/33 file after the snapshot 2023-10-30T13:25:49.800 INFO:tasks.workunit.client.0.smithi046.stdout: create 157/34 file after the snapshot 2023-10-30T13:25:49.801 INFO:tasks.workunit.client.0.smithi046.stdout: create 157/35 file after the snapshot 2023-10-30T13:25:49.801 INFO:tasks.workunit.client.0.smithi046.stdout: create 157/36 file after the snapshot 2023-10-30T13:25:49.801 INFO:tasks.workunit.client.0.smithi046.stdout: create 157/37 file after the snapshot 2023-10-30T13:25:49.802 INFO:tasks.workunit.client.0.smithi046.stdout: create 157/38 file after the snapshot 2023-10-30T13:25:49.802 INFO:tasks.workunit.client.0.smithi046.stdout: create 157/39 file after the snapshot 2023-10-30T13:25:49.802 INFO:tasks.workunit.client.0.smithi046.stdout: create 158/.snap/snap-subdir-test 2023-10-30T13:25:49.837 INFO:tasks.workunit.client.0.smithi046.stdout: create 158/30 file after the snapshot 2023-10-30T13:25:49.839 INFO:tasks.workunit.client.0.smithi046.stdout: create 158/31 file after the snapshot 2023-10-30T13:25:49.839 INFO:tasks.workunit.client.0.smithi046.stdout: create 158/32 file after the snapshot 2023-10-30T13:25:49.840 INFO:tasks.workunit.client.0.smithi046.stdout: create 158/33 file after the snapshot 2023-10-30T13:25:49.840 INFO:tasks.workunit.client.0.smithi046.stdout: create 158/34 file after the snapshot 2023-10-30T13:25:49.840 INFO:tasks.workunit.client.0.smithi046.stdout: create 158/35 file after the snapshot 2023-10-30T13:25:49.840 INFO:tasks.workunit.client.0.smithi046.stdout: create 158/36 file after the snapshot 2023-10-30T13:25:49.840 INFO:tasks.workunit.client.0.smithi046.stdout: create 158/37 file after the snapshot 2023-10-30T13:25:49.841 INFO:tasks.workunit.client.0.smithi046.stdout: create 158/38 file after the snapshot 2023-10-30T13:25:49.841 INFO:tasks.workunit.client.0.smithi046.stdout: create 158/39 file after the snapshot 2023-10-30T13:25:49.841 INFO:tasks.workunit.client.0.smithi046.stdout: create 159/.snap/snap-subdir-test 2023-10-30T13:25:49.881 INFO:tasks.workunit.client.0.smithi046.stdout: create 159/30 file after the snapshot 2023-10-30T13:25:49.883 INFO:tasks.workunit.client.0.smithi046.stdout: create 159/31 file after the snapshot 2023-10-30T13:25:49.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 159/32 file after the snapshot 2023-10-30T13:25:49.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 159/33 file after the snapshot 2023-10-30T13:25:49.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 159/34 file after the snapshot 2023-10-30T13:25:49.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 159/35 file after the snapshot 2023-10-30T13:25:49.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 159/36 file after the snapshot 2023-10-30T13:25:49.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 159/37 file after the snapshot 2023-10-30T13:25:49.885 INFO:tasks.workunit.client.0.smithi046.stdout: create 159/38 file after the snapshot 2023-10-30T13:25:49.885 INFO:tasks.workunit.client.0.smithi046.stdout: create 159/39 file after the snapshot 2023-10-30T13:25:49.885 INFO:tasks.workunit.client.0.smithi046.stdout: create 160/.snap/snap-subdir-test 2023-10-30T13:25:50.222 INFO:tasks.workunit.client.0.smithi046.stdout: create 160/30 file after the snapshot 2023-10-30T13:25:50.224 INFO:tasks.workunit.client.0.smithi046.stdout: create 160/31 file after the snapshot 2023-10-30T13:25:50.224 INFO:tasks.workunit.client.0.smithi046.stdout: create 160/32 file after the snapshot 2023-10-30T13:25:50.224 INFO:tasks.workunit.client.0.smithi046.stdout: create 160/33 file after the snapshot 2023-10-30T13:25:50.224 INFO:tasks.workunit.client.0.smithi046.stdout: create 160/34 file after the snapshot 2023-10-30T13:25:50.225 INFO:tasks.workunit.client.0.smithi046.stdout: create 160/35 file after the snapshot 2023-10-30T13:25:50.225 INFO:tasks.workunit.client.0.smithi046.stdout: create 160/36 file after the snapshot 2023-10-30T13:25:50.225 INFO:tasks.workunit.client.0.smithi046.stdout: create 160/37 file after the snapshot 2023-10-30T13:25:50.225 INFO:tasks.workunit.client.0.smithi046.stdout: create 160/38 file after the snapshot 2023-10-30T13:25:50.226 INFO:tasks.workunit.client.0.smithi046.stdout: create 160/39 file after the snapshot 2023-10-30T13:25:50.226 INFO:tasks.workunit.client.0.smithi046.stdout: create 161/.snap/snap-subdir-test 2023-10-30T13:25:50.256 INFO:tasks.workunit.client.0.smithi046.stdout: create 161/30 file after the snapshot 2023-10-30T13:25:50.259 INFO:tasks.workunit.client.0.smithi046.stdout: create 161/31 file after the snapshot 2023-10-30T13:25:50.259 INFO:tasks.workunit.client.0.smithi046.stdout: create 161/32 file after the snapshot 2023-10-30T13:25:50.259 INFO:tasks.workunit.client.0.smithi046.stdout: create 161/33 file after the snapshot 2023-10-30T13:25:50.260 INFO:tasks.workunit.client.0.smithi046.stdout: create 161/34 file after the snapshot 2023-10-30T13:25:50.260 INFO:tasks.workunit.client.0.smithi046.stdout: create 161/35 file after the snapshot 2023-10-30T13:25:50.261 INFO:tasks.workunit.client.0.smithi046.stdout: create 161/36 file after the snapshot 2023-10-30T13:25:50.261 INFO:tasks.workunit.client.0.smithi046.stdout: create 161/37 file after the snapshot 2023-10-30T13:25:50.262 INFO:tasks.workunit.client.0.smithi046.stdout: create 161/38 file after the snapshot 2023-10-30T13:25:50.262 INFO:tasks.workunit.client.0.smithi046.stdout: create 161/39 file after the snapshot 2023-10-30T13:25:50.262 INFO:tasks.workunit.client.0.smithi046.stdout: create 162/.snap/snap-subdir-test 2023-10-30T13:25:51.106 INFO:tasks.workunit.client.0.smithi046.stdout: create 162/30 file after the snapshot 2023-10-30T13:25:51.108 INFO:tasks.workunit.client.0.smithi046.stdout: create 162/31 file after the snapshot 2023-10-30T13:25:51.109 INFO:tasks.workunit.client.0.smithi046.stdout: create 162/32 file after the snapshot 2023-10-30T13:25:51.109 INFO:tasks.workunit.client.0.smithi046.stdout: create 162/33 file after the snapshot 2023-10-30T13:25:51.109 INFO:tasks.workunit.client.0.smithi046.stdout: create 162/34 file after the snapshot 2023-10-30T13:25:51.109 INFO:tasks.workunit.client.0.smithi046.stdout: create 162/35 file after the snapshot 2023-10-30T13:25:51.109 INFO:tasks.workunit.client.0.smithi046.stdout: create 162/36 file after the snapshot 2023-10-30T13:25:51.109 INFO:tasks.workunit.client.0.smithi046.stdout: create 162/37 file after the snapshot 2023-10-30T13:25:51.110 INFO:tasks.workunit.client.0.smithi046.stdout: create 162/38 file after the snapshot 2023-10-30T13:25:51.110 INFO:tasks.workunit.client.0.smithi046.stdout: create 162/39 file after the snapshot 2023-10-30T13:25:51.110 INFO:tasks.workunit.client.0.smithi046.stdout: create 163/.snap/snap-subdir-test 2023-10-30T13:25:51.147 INFO:tasks.workunit.client.0.smithi046.stdout: create 163/30 file after the snapshot 2023-10-30T13:25:51.149 INFO:tasks.workunit.client.0.smithi046.stdout: create 163/31 file after the snapshot 2023-10-30T13:25:51.149 INFO:tasks.workunit.client.0.smithi046.stdout: create 163/32 file after the snapshot 2023-10-30T13:25:51.149 INFO:tasks.workunit.client.0.smithi046.stdout: create 163/33 file after the snapshot 2023-10-30T13:25:51.150 INFO:tasks.workunit.client.0.smithi046.stdout: create 163/34 file after the snapshot 2023-10-30T13:25:51.150 INFO:tasks.workunit.client.0.smithi046.stdout: create 163/35 file after the snapshot 2023-10-30T13:25:51.150 INFO:tasks.workunit.client.0.smithi046.stdout: create 163/36 file after the snapshot 2023-10-30T13:25:51.150 INFO:tasks.workunit.client.0.smithi046.stdout: create 163/37 file after the snapshot 2023-10-30T13:25:51.150 INFO:tasks.workunit.client.0.smithi046.stdout: create 163/38 file after the snapshot 2023-10-30T13:25:51.151 INFO:tasks.workunit.client.0.smithi046.stdout: create 163/39 file after the snapshot 2023-10-30T13:25:51.151 INFO:tasks.workunit.client.0.smithi046.stdout: create 164/.snap/snap-subdir-test 2023-10-30T13:25:51.405 INFO:tasks.workunit.client.0.smithi046.stdout: create 164/30 file after the snapshot 2023-10-30T13:25:51.407 INFO:tasks.workunit.client.0.smithi046.stdout: create 164/31 file after the snapshot 2023-10-30T13:25:51.408 INFO:tasks.workunit.client.0.smithi046.stdout: create 164/32 file after the snapshot 2023-10-30T13:25:51.408 INFO:tasks.workunit.client.0.smithi046.stdout: create 164/33 file after the snapshot 2023-10-30T13:25:51.408 INFO:tasks.workunit.client.0.smithi046.stdout: create 164/34 file after the snapshot 2023-10-30T13:25:51.409 INFO:tasks.workunit.client.0.smithi046.stdout: create 164/35 file after the snapshot 2023-10-30T13:25:51.409 INFO:tasks.workunit.client.0.smithi046.stdout: create 164/36 file after the snapshot 2023-10-30T13:25:51.409 INFO:tasks.workunit.client.0.smithi046.stdout: create 164/37 file after the snapshot 2023-10-30T13:25:51.409 INFO:tasks.workunit.client.0.smithi046.stdout: create 164/38 file after the snapshot 2023-10-30T13:25:51.410 INFO:tasks.workunit.client.0.smithi046.stdout: create 164/39 file after the snapshot 2023-10-30T13:25:51.410 INFO:tasks.workunit.client.0.smithi046.stdout: create 165/.snap/snap-subdir-test 2023-10-30T13:25:51.442 INFO:tasks.workunit.client.0.smithi046.stdout: create 165/30 file after the snapshot 2023-10-30T13:25:51.444 INFO:tasks.workunit.client.0.smithi046.stdout: create 165/31 file after the snapshot 2023-10-30T13:25:51.445 INFO:tasks.workunit.client.0.smithi046.stdout: create 165/32 file after the snapshot 2023-10-30T13:25:51.445 INFO:tasks.workunit.client.0.smithi046.stdout: create 165/33 file after the snapshot 2023-10-30T13:25:51.445 INFO:tasks.workunit.client.0.smithi046.stdout: create 165/34 file after the snapshot 2023-10-30T13:25:51.446 INFO:tasks.workunit.client.0.smithi046.stdout: create 165/35 file after the snapshot 2023-10-30T13:25:51.446 INFO:tasks.workunit.client.0.smithi046.stdout: create 165/36 file after the snapshot 2023-10-30T13:25:51.446 INFO:tasks.workunit.client.0.smithi046.stdout: create 165/37 file after the snapshot 2023-10-30T13:25:51.447 INFO:tasks.workunit.client.0.smithi046.stdout: create 165/38 file after the snapshot 2023-10-30T13:25:51.447 INFO:tasks.workunit.client.0.smithi046.stdout: create 165/39 file after the snapshot 2023-10-30T13:25:51.447 INFO:tasks.workunit.client.0.smithi046.stdout: create 166/.snap/snap-subdir-test 2023-10-30T13:25:51.740 INFO:tasks.workunit.client.0.smithi046.stdout: create 166/30 file after the snapshot 2023-10-30T13:25:51.742 INFO:tasks.workunit.client.0.smithi046.stdout: create 166/31 file after the snapshot 2023-10-30T13:25:51.742 INFO:tasks.workunit.client.0.smithi046.stdout: create 166/32 file after the snapshot 2023-10-30T13:25:51.743 INFO:tasks.workunit.client.0.smithi046.stdout: create 166/33 file after the snapshot 2023-10-30T13:25:51.743 INFO:tasks.workunit.client.0.smithi046.stdout: create 166/34 file after the snapshot 2023-10-30T13:25:51.743 INFO:tasks.workunit.client.0.smithi046.stdout: create 166/35 file after the snapshot 2023-10-30T13:25:51.743 INFO:tasks.workunit.client.0.smithi046.stdout: create 166/36 file after the snapshot 2023-10-30T13:25:51.743 INFO:tasks.workunit.client.0.smithi046.stdout: create 166/37 file after the snapshot 2023-10-30T13:25:51.744 INFO:tasks.workunit.client.0.smithi046.stdout: create 166/38 file after the snapshot 2023-10-30T13:25:51.744 INFO:tasks.workunit.client.0.smithi046.stdout: create 166/39 file after the snapshot 2023-10-30T13:25:51.744 INFO:tasks.workunit.client.0.smithi046.stdout: create 167/.snap/snap-subdir-test 2023-10-30T13:25:51.774 INFO:tasks.workunit.client.0.smithi046.stdout: create 167/30 file after the snapshot 2023-10-30T13:25:51.776 INFO:tasks.workunit.client.0.smithi046.stdout: create 167/31 file after the snapshot 2023-10-30T13:25:51.776 INFO:tasks.workunit.client.0.smithi046.stdout: create 167/32 file after the snapshot 2023-10-30T13:25:51.776 INFO:tasks.workunit.client.0.smithi046.stdout: create 167/33 file after the snapshot 2023-10-30T13:25:51.776 INFO:tasks.workunit.client.0.smithi046.stdout: create 167/34 file after the snapshot 2023-10-30T13:25:51.776 INFO:tasks.workunit.client.0.smithi046.stdout: create 167/35 file after the snapshot 2023-10-30T13:25:51.777 INFO:tasks.workunit.client.0.smithi046.stdout: create 167/36 file after the snapshot 2023-10-30T13:25:51.777 INFO:tasks.workunit.client.0.smithi046.stdout: create 167/37 file after the snapshot 2023-10-30T13:25:51.777 INFO:tasks.workunit.client.0.smithi046.stdout: create 167/38 file after the snapshot 2023-10-30T13:25:51.777 INFO:tasks.workunit.client.0.smithi046.stdout: create 167/39 file after the snapshot 2023-10-30T13:25:51.777 INFO:tasks.workunit.client.0.smithi046.stdout: create 168/.snap/snap-subdir-test 2023-10-30T13:25:52.431 INFO:tasks.workunit.client.0.smithi046.stdout: create 168/30 file after the snapshot 2023-10-30T13:25:52.433 INFO:tasks.workunit.client.0.smithi046.stdout: create 168/31 file after the snapshot 2023-10-30T13:25:52.433 INFO:tasks.workunit.client.0.smithi046.stdout: create 168/32 file after the snapshot 2023-10-30T13:25:52.433 INFO:tasks.workunit.client.0.smithi046.stdout: create 168/33 file after the snapshot 2023-10-30T13:25:52.433 INFO:tasks.workunit.client.0.smithi046.stdout: create 168/34 file after the snapshot 2023-10-30T13:25:52.433 INFO:tasks.workunit.client.0.smithi046.stdout: create 168/35 file after the snapshot 2023-10-30T13:25:52.434 INFO:tasks.workunit.client.0.smithi046.stdout: create 168/36 file after the snapshot 2023-10-30T13:25:52.434 INFO:tasks.workunit.client.0.smithi046.stdout: create 168/37 file after the snapshot 2023-10-30T13:25:52.434 INFO:tasks.workunit.client.0.smithi046.stdout: create 168/38 file after the snapshot 2023-10-30T13:25:52.434 INFO:tasks.workunit.client.0.smithi046.stdout: create 168/39 file after the snapshot 2023-10-30T13:25:52.434 INFO:tasks.workunit.client.0.smithi046.stdout: create 169/.snap/snap-subdir-test 2023-10-30T13:25:52.466 INFO:tasks.workunit.client.0.smithi046.stdout: create 169/30 file after the snapshot 2023-10-30T13:25:52.468 INFO:tasks.workunit.client.0.smithi046.stdout: create 169/31 file after the snapshot 2023-10-30T13:25:52.468 INFO:tasks.workunit.client.0.smithi046.stdout: create 169/32 file after the snapshot 2023-10-30T13:25:52.468 INFO:tasks.workunit.client.0.smithi046.stdout: create 169/33 file after the snapshot 2023-10-30T13:25:52.468 INFO:tasks.workunit.client.0.smithi046.stdout: create 169/34 file after the snapshot 2023-10-30T13:25:52.468 INFO:tasks.workunit.client.0.smithi046.stdout: create 169/35 file after the snapshot 2023-10-30T13:25:52.469 INFO:tasks.workunit.client.0.smithi046.stdout: create 169/36 file after the snapshot 2023-10-30T13:25:52.469 INFO:tasks.workunit.client.0.smithi046.stdout: create 169/37 file after the snapshot 2023-10-30T13:25:52.469 INFO:tasks.workunit.client.0.smithi046.stdout: create 169/38 file after the snapshot 2023-10-30T13:25:52.469 INFO:tasks.workunit.client.0.smithi046.stdout: create 169/39 file after the snapshot 2023-10-30T13:25:52.469 INFO:tasks.workunit.client.0.smithi046.stdout: create 170/.snap/snap-subdir-test 2023-10-30T13:25:52.500 INFO:tasks.workunit.client.0.smithi046.stdout: create 170/30 file after the snapshot 2023-10-30T13:25:52.502 INFO:tasks.workunit.client.0.smithi046.stdout: create 170/31 file after the snapshot 2023-10-30T13:25:52.502 INFO:tasks.workunit.client.0.smithi046.stdout: create 170/32 file after the snapshot 2023-10-30T13:25:52.502 INFO:tasks.workunit.client.0.smithi046.stdout: create 170/33 file after the snapshot 2023-10-30T13:25:52.502 INFO:tasks.workunit.client.0.smithi046.stdout: create 170/34 file after the snapshot 2023-10-30T13:25:52.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 170/35 file after the snapshot 2023-10-30T13:25:52.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 170/36 file after the snapshot 2023-10-30T13:25:52.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 170/37 file after the snapshot 2023-10-30T13:25:52.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 170/38 file after the snapshot 2023-10-30T13:25:52.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 170/39 file after the snapshot 2023-10-30T13:25:52.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 171/.snap/snap-subdir-test 2023-10-30T13:25:52.534 INFO:tasks.workunit.client.0.smithi046.stdout: create 171/30 file after the snapshot 2023-10-30T13:25:52.535 INFO:tasks.workunit.client.0.smithi046.stdout: create 171/31 file after the snapshot 2023-10-30T13:25:52.536 INFO:tasks.workunit.client.0.smithi046.stdout: create 171/32 file after the snapshot 2023-10-30T13:25:52.536 INFO:tasks.workunit.client.0.smithi046.stdout: create 171/33 file after the snapshot 2023-10-30T13:25:52.536 INFO:tasks.workunit.client.0.smithi046.stdout: create 171/34 file after the snapshot 2023-10-30T13:25:52.536 INFO:tasks.workunit.client.0.smithi046.stdout: create 171/35 file after the snapshot 2023-10-30T13:25:52.536 INFO:tasks.workunit.client.0.smithi046.stdout: create 171/36 file after the snapshot 2023-10-30T13:25:52.537 INFO:tasks.workunit.client.0.smithi046.stdout: create 171/37 file after the snapshot 2023-10-30T13:25:52.537 INFO:tasks.workunit.client.0.smithi046.stdout: create 171/38 file after the snapshot 2023-10-30T13:25:52.537 INFO:tasks.workunit.client.0.smithi046.stdout: create 171/39 file after the snapshot 2023-10-30T13:25:52.537 INFO:tasks.workunit.client.0.smithi046.stdout: create 172/.snap/snap-subdir-test 2023-10-30T13:25:52.567 INFO:tasks.workunit.client.0.smithi046.stdout: create 172/30 file after the snapshot 2023-10-30T13:25:52.569 INFO:tasks.workunit.client.0.smithi046.stdout: create 172/31 file after the snapshot 2023-10-30T13:25:52.569 INFO:tasks.workunit.client.0.smithi046.stdout: create 172/32 file after the snapshot 2023-10-30T13:25:52.569 INFO:tasks.workunit.client.0.smithi046.stdout: create 172/33 file after the snapshot 2023-10-30T13:25:52.570 INFO:tasks.workunit.client.0.smithi046.stdout: create 172/34 file after the snapshot 2023-10-30T13:25:52.570 INFO:tasks.workunit.client.0.smithi046.stdout: create 172/35 file after the snapshot 2023-10-30T13:25:52.570 INFO:tasks.workunit.client.0.smithi046.stdout: create 172/36 file after the snapshot 2023-10-30T13:25:52.570 INFO:tasks.workunit.client.0.smithi046.stdout: create 172/37 file after the snapshot 2023-10-30T13:25:52.570 INFO:tasks.workunit.client.0.smithi046.stdout: create 172/38 file after the snapshot 2023-10-30T13:25:52.570 INFO:tasks.workunit.client.0.smithi046.stdout: create 172/39 file after the snapshot 2023-10-30T13:25:52.571 INFO:tasks.workunit.client.0.smithi046.stdout: create 173/.snap/snap-subdir-test 2023-10-30T13:25:52.602 INFO:tasks.workunit.client.0.smithi046.stdout: create 173/30 file after the snapshot 2023-10-30T13:25:52.603 INFO:tasks.workunit.client.0.smithi046.stdout: create 173/31 file after the snapshot 2023-10-30T13:25:52.604 INFO:tasks.workunit.client.0.smithi046.stdout: create 173/32 file after the snapshot 2023-10-30T13:25:52.604 INFO:tasks.workunit.client.0.smithi046.stdout: create 173/33 file after the snapshot 2023-10-30T13:25:52.604 INFO:tasks.workunit.client.0.smithi046.stdout: create 173/34 file after the snapshot 2023-10-30T13:25:52.604 INFO:tasks.workunit.client.0.smithi046.stdout: create 173/35 file after the snapshot 2023-10-30T13:25:52.604 INFO:tasks.workunit.client.0.smithi046.stdout: create 173/36 file after the snapshot 2023-10-30T13:25:52.605 INFO:tasks.workunit.client.0.smithi046.stdout: create 173/37 file after the snapshot 2023-10-30T13:25:52.605 INFO:tasks.workunit.client.0.smithi046.stdout: create 173/38 file after the snapshot 2023-10-30T13:25:52.605 INFO:tasks.workunit.client.0.smithi046.stdout: create 173/39 file after the snapshot 2023-10-30T13:25:52.605 INFO:tasks.workunit.client.0.smithi046.stdout: create 174/.snap/snap-subdir-test 2023-10-30T13:25:52.634 INFO:tasks.workunit.client.0.smithi046.stdout: create 174/30 file after the snapshot 2023-10-30T13:25:52.635 INFO:tasks.workunit.client.0.smithi046.stdout: create 174/31 file after the snapshot 2023-10-30T13:25:52.635 INFO:tasks.workunit.client.0.smithi046.stdout: create 174/32 file after the snapshot 2023-10-30T13:25:52.635 INFO:tasks.workunit.client.0.smithi046.stdout: create 174/33 file after the snapshot 2023-10-30T13:25:52.636 INFO:tasks.workunit.client.0.smithi046.stdout: create 174/34 file after the snapshot 2023-10-30T13:25:52.636 INFO:tasks.workunit.client.0.smithi046.stdout: create 174/35 file after the snapshot 2023-10-30T13:25:52.636 INFO:tasks.workunit.client.0.smithi046.stdout: create 174/36 file after the snapshot 2023-10-30T13:25:52.636 INFO:tasks.workunit.client.0.smithi046.stdout: create 174/37 file after the snapshot 2023-10-30T13:25:52.636 INFO:tasks.workunit.client.0.smithi046.stdout: create 174/38 file after the snapshot 2023-10-30T13:25:52.637 INFO:tasks.workunit.client.0.smithi046.stdout: create 174/39 file after the snapshot 2023-10-30T13:25:52.637 INFO:tasks.workunit.client.0.smithi046.stdout: create 175/.snap/snap-subdir-test 2023-10-30T13:25:52.665 INFO:tasks.workunit.client.0.smithi046.stdout: create 175/30 file after the snapshot 2023-10-30T13:25:52.667 INFO:tasks.workunit.client.0.smithi046.stdout: create 175/31 file after the snapshot 2023-10-30T13:25:52.667 INFO:tasks.workunit.client.0.smithi046.stdout: create 175/32 file after the snapshot 2023-10-30T13:25:52.667 INFO:tasks.workunit.client.0.smithi046.stdout: create 175/33 file after the snapshot 2023-10-30T13:25:52.667 INFO:tasks.workunit.client.0.smithi046.stdout: create 175/34 file after the snapshot 2023-10-30T13:25:52.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 175/35 file after the snapshot 2023-10-30T13:25:52.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 175/36 file after the snapshot 2023-10-30T13:25:52.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 175/37 file after the snapshot 2023-10-30T13:25:52.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 175/38 file after the snapshot 2023-10-30T13:25:52.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 175/39 file after the snapshot 2023-10-30T13:25:52.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 176/.snap/snap-subdir-test 2023-10-30T13:25:52.698 INFO:tasks.workunit.client.0.smithi046.stdout: create 176/30 file after the snapshot 2023-10-30T13:25:52.699 INFO:tasks.workunit.client.0.smithi046.stdout: create 176/31 file after the snapshot 2023-10-30T13:25:52.699 INFO:tasks.workunit.client.0.smithi046.stdout: create 176/32 file after the snapshot 2023-10-30T13:25:52.700 INFO:tasks.workunit.client.0.smithi046.stdout: create 176/33 file after the snapshot 2023-10-30T13:25:52.700 INFO:tasks.workunit.client.0.smithi046.stdout: create 176/34 file after the snapshot 2023-10-30T13:25:52.700 INFO:tasks.workunit.client.0.smithi046.stdout: create 176/35 file after the snapshot 2023-10-30T13:25:52.700 INFO:tasks.workunit.client.0.smithi046.stdout: create 176/36 file after the snapshot 2023-10-30T13:25:52.700 INFO:tasks.workunit.client.0.smithi046.stdout: create 176/37 file after the snapshot 2023-10-30T13:25:52.701 INFO:tasks.workunit.client.0.smithi046.stdout: create 176/38 file after the snapshot 2023-10-30T13:25:52.701 INFO:tasks.workunit.client.0.smithi046.stdout: create 176/39 file after the snapshot 2023-10-30T13:25:52.701 INFO:tasks.workunit.client.0.smithi046.stdout: create 177/.snap/snap-subdir-test 2023-10-30T13:25:52.724 INFO:tasks.workunit.client.0.smithi046.stdout: create 177/30 file after the snapshot 2023-10-30T13:25:52.726 INFO:tasks.workunit.client.0.smithi046.stdout: create 177/31 file after the snapshot 2023-10-30T13:25:52.726 INFO:tasks.workunit.client.0.smithi046.stdout: create 177/32 file after the snapshot 2023-10-30T13:25:52.726 INFO:tasks.workunit.client.0.smithi046.stdout: create 177/33 file after the snapshot 2023-10-30T13:25:52.726 INFO:tasks.workunit.client.0.smithi046.stdout: create 177/34 file after the snapshot 2023-10-30T13:25:52.727 INFO:tasks.workunit.client.0.smithi046.stdout: create 177/35 file after the snapshot 2023-10-30T13:25:52.727 INFO:tasks.workunit.client.0.smithi046.stdout: create 177/36 file after the snapshot 2023-10-30T13:25:52.727 INFO:tasks.workunit.client.0.smithi046.stdout: create 177/37 file after the snapshot 2023-10-30T13:25:52.727 INFO:tasks.workunit.client.0.smithi046.stdout: create 177/38 file after the snapshot 2023-10-30T13:25:52.727 INFO:tasks.workunit.client.0.smithi046.stdout: create 177/39 file after the snapshot 2023-10-30T13:25:52.727 INFO:tasks.workunit.client.0.smithi046.stdout: create 178/.snap/snap-subdir-test 2023-10-30T13:25:52.750 INFO:tasks.workunit.client.0.smithi046.stdout: create 178/30 file after the snapshot 2023-10-30T13:25:52.751 INFO:tasks.workunit.client.0.smithi046.stdout: create 178/31 file after the snapshot 2023-10-30T13:25:52.752 INFO:tasks.workunit.client.0.smithi046.stdout: create 178/32 file after the snapshot 2023-10-30T13:25:52.752 INFO:tasks.workunit.client.0.smithi046.stdout: create 178/33 file after the snapshot 2023-10-30T13:25:52.752 INFO:tasks.workunit.client.0.smithi046.stdout: create 178/34 file after the snapshot 2023-10-30T13:25:52.752 INFO:tasks.workunit.client.0.smithi046.stdout: create 178/35 file after the snapshot 2023-10-30T13:25:52.753 INFO:tasks.workunit.client.0.smithi046.stdout: create 178/36 file after the snapshot 2023-10-30T13:25:52.753 INFO:tasks.workunit.client.0.smithi046.stdout: create 178/37 file after the snapshot 2023-10-30T13:25:52.753 INFO:tasks.workunit.client.0.smithi046.stdout: create 178/38 file after the snapshot 2023-10-30T13:25:52.753 INFO:tasks.workunit.client.0.smithi046.stdout: create 178/39 file after the snapshot 2023-10-30T13:25:52.753 INFO:tasks.workunit.client.0.smithi046.stdout: create 179/.snap/snap-subdir-test 2023-10-30T13:25:53.188 INFO:tasks.workunit.client.0.smithi046.stdout: create 179/30 file after the snapshot 2023-10-30T13:25:53.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 179/31 file after the snapshot 2023-10-30T13:25:53.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 179/32 file after the snapshot 2023-10-30T13:25:53.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 179/33 file after the snapshot 2023-10-30T13:25:53.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 179/34 file after the snapshot 2023-10-30T13:25:53.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 179/35 file after the snapshot 2023-10-30T13:25:53.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 179/36 file after the snapshot 2023-10-30T13:25:53.191 INFO:tasks.workunit.client.0.smithi046.stdout: create 179/37 file after the snapshot 2023-10-30T13:25:53.191 INFO:tasks.workunit.client.0.smithi046.stdout: create 179/38 file after the snapshot 2023-10-30T13:25:53.191 INFO:tasks.workunit.client.0.smithi046.stdout: create 179/39 file after the snapshot 2023-10-30T13:25:53.191 INFO:tasks.workunit.client.0.smithi046.stdout: create 180/.snap/snap-subdir-test 2023-10-30T13:25:53.214 INFO:tasks.workunit.client.0.smithi046.stdout: create 180/30 file after the snapshot 2023-10-30T13:25:53.216 INFO:tasks.workunit.client.0.smithi046.stdout: create 180/31 file after the snapshot 2023-10-30T13:25:53.216 INFO:tasks.workunit.client.0.smithi046.stdout: create 180/32 file after the snapshot 2023-10-30T13:25:53.216 INFO:tasks.workunit.client.0.smithi046.stdout: create 180/33 file after the snapshot 2023-10-30T13:25:53.216 INFO:tasks.workunit.client.0.smithi046.stdout: create 180/34 file after the snapshot 2023-10-30T13:25:53.217 INFO:tasks.workunit.client.0.smithi046.stdout: create 180/35 file after the snapshot 2023-10-30T13:25:53.217 INFO:tasks.workunit.client.0.smithi046.stdout: create 180/36 file after the snapshot 2023-10-30T13:25:53.217 INFO:tasks.workunit.client.0.smithi046.stdout: create 180/37 file after the snapshot 2023-10-30T13:25:53.217 INFO:tasks.workunit.client.0.smithi046.stdout: create 180/38 file after the snapshot 2023-10-30T13:25:53.217 INFO:tasks.workunit.client.0.smithi046.stdout: create 180/39 file after the snapshot 2023-10-30T13:25:53.218 INFO:tasks.workunit.client.0.smithi046.stdout: create 181/.snap/snap-subdir-test 2023-10-30T13:25:53.240 INFO:tasks.workunit.client.0.smithi046.stdout: create 181/30 file after the snapshot 2023-10-30T13:25:53.242 INFO:tasks.workunit.client.0.smithi046.stdout: create 181/31 file after the snapshot 2023-10-30T13:25:53.242 INFO:tasks.workunit.client.0.smithi046.stdout: create 181/32 file after the snapshot 2023-10-30T13:25:53.242 INFO:tasks.workunit.client.0.smithi046.stdout: create 181/33 file after the snapshot 2023-10-30T13:25:53.242 INFO:tasks.workunit.client.0.smithi046.stdout: create 181/34 file after the snapshot 2023-10-30T13:25:53.243 INFO:tasks.workunit.client.0.smithi046.stdout: create 181/35 file after the snapshot 2023-10-30T13:25:53.243 INFO:tasks.workunit.client.0.smithi046.stdout: create 181/36 file after the snapshot 2023-10-30T13:25:53.243 INFO:tasks.workunit.client.0.smithi046.stdout: create 181/37 file after the snapshot 2023-10-30T13:25:53.243 INFO:tasks.workunit.client.0.smithi046.stdout: create 181/38 file after the snapshot 2023-10-30T13:25:53.243 INFO:tasks.workunit.client.0.smithi046.stdout: create 181/39 file after the snapshot 2023-10-30T13:25:53.244 INFO:tasks.workunit.client.0.smithi046.stdout: create 182/.snap/snap-subdir-test 2023-10-30T13:25:53.266 INFO:tasks.workunit.client.0.smithi046.stdout: create 182/30 file after the snapshot 2023-10-30T13:25:53.268 INFO:tasks.workunit.client.0.smithi046.stdout: create 182/31 file after the snapshot 2023-10-30T13:25:53.268 INFO:tasks.workunit.client.0.smithi046.stdout: create 182/32 file after the snapshot 2023-10-30T13:25:53.268 INFO:tasks.workunit.client.0.smithi046.stdout: create 182/33 file after the snapshot 2023-10-30T13:25:53.269 INFO:tasks.workunit.client.0.smithi046.stdout: create 182/34 file after the snapshot 2023-10-30T13:25:53.269 INFO:tasks.workunit.client.0.smithi046.stdout: create 182/35 file after the snapshot 2023-10-30T13:25:53.269 INFO:tasks.workunit.client.0.smithi046.stdout: create 182/36 file after the snapshot 2023-10-30T13:25:53.269 INFO:tasks.workunit.client.0.smithi046.stdout: create 182/37 file after the snapshot 2023-10-30T13:25:53.269 INFO:tasks.workunit.client.0.smithi046.stdout: create 182/38 file after the snapshot 2023-10-30T13:25:53.270 INFO:tasks.workunit.client.0.smithi046.stdout: create 182/39 file after the snapshot 2023-10-30T13:25:53.270 INFO:tasks.workunit.client.0.smithi046.stdout: create 183/.snap/snap-subdir-test 2023-10-30T13:25:53.292 INFO:tasks.workunit.client.0.smithi046.stdout: create 183/30 file after the snapshot 2023-10-30T13:25:53.293 INFO:tasks.workunit.client.0.smithi046.stdout: create 183/31 file after the snapshot 2023-10-30T13:25:53.293 INFO:tasks.workunit.client.0.smithi046.stdout: create 183/32 file after the snapshot 2023-10-30T13:25:53.294 INFO:tasks.workunit.client.0.smithi046.stdout: create 183/33 file after the snapshot 2023-10-30T13:25:53.294 INFO:tasks.workunit.client.0.smithi046.stdout: create 183/34 file after the snapshot 2023-10-30T13:25:53.294 INFO:tasks.workunit.client.0.smithi046.stdout: create 183/35 file after the snapshot 2023-10-30T13:25:53.294 INFO:tasks.workunit.client.0.smithi046.stdout: create 183/36 file after the snapshot 2023-10-30T13:25:53.294 INFO:tasks.workunit.client.0.smithi046.stdout: create 183/37 file after the snapshot 2023-10-30T13:25:53.295 INFO:tasks.workunit.client.0.smithi046.stdout: create 183/38 file after the snapshot 2023-10-30T13:25:53.295 INFO:tasks.workunit.client.0.smithi046.stdout: create 183/39 file after the snapshot 2023-10-30T13:25:53.295 INFO:tasks.workunit.client.0.smithi046.stdout: create 184/.snap/snap-subdir-test 2023-10-30T13:25:53.318 INFO:tasks.workunit.client.0.smithi046.stdout: create 184/30 file after the snapshot 2023-10-30T13:25:53.319 INFO:tasks.workunit.client.0.smithi046.stdout: create 184/31 file after the snapshot 2023-10-30T13:25:53.320 INFO:tasks.workunit.client.0.smithi046.stdout: create 184/32 file after the snapshot 2023-10-30T13:25:53.320 INFO:tasks.workunit.client.0.smithi046.stdout: create 184/33 file after the snapshot 2023-10-30T13:25:53.320 INFO:tasks.workunit.client.0.smithi046.stdout: create 184/34 file after the snapshot 2023-10-30T13:25:53.320 INFO:tasks.workunit.client.0.smithi046.stdout: create 184/35 file after the snapshot 2023-10-30T13:25:53.320 INFO:tasks.workunit.client.0.smithi046.stdout: create 184/36 file after the snapshot 2023-10-30T13:25:53.320 INFO:tasks.workunit.client.0.smithi046.stdout: create 184/37 file after the snapshot 2023-10-30T13:25:53.321 INFO:tasks.workunit.client.0.smithi046.stdout: create 184/38 file after the snapshot 2023-10-30T13:25:53.321 INFO:tasks.workunit.client.0.smithi046.stdout: create 184/39 file after the snapshot 2023-10-30T13:25:53.321 INFO:tasks.workunit.client.0.smithi046.stdout: create 185/.snap/snap-subdir-test 2023-10-30T13:25:53.346 INFO:tasks.workunit.client.0.smithi046.stdout: create 185/30 file after the snapshot 2023-10-30T13:25:53.348 INFO:tasks.workunit.client.0.smithi046.stdout: create 185/31 file after the snapshot 2023-10-30T13:25:53.348 INFO:tasks.workunit.client.0.smithi046.stdout: create 185/32 file after the snapshot 2023-10-30T13:25:53.348 INFO:tasks.workunit.client.0.smithi046.stdout: create 185/33 file after the snapshot 2023-10-30T13:25:53.348 INFO:tasks.workunit.client.0.smithi046.stdout: create 185/34 file after the snapshot 2023-10-30T13:25:53.349 INFO:tasks.workunit.client.0.smithi046.stdout: create 185/35 file after the snapshot 2023-10-30T13:25:53.349 INFO:tasks.workunit.client.0.smithi046.stdout: create 185/36 file after the snapshot 2023-10-30T13:25:53.349 INFO:tasks.workunit.client.0.smithi046.stdout: create 185/37 file after the snapshot 2023-10-30T13:25:53.349 INFO:tasks.workunit.client.0.smithi046.stdout: create 185/38 file after the snapshot 2023-10-30T13:25:53.349 INFO:tasks.workunit.client.0.smithi046.stdout: create 185/39 file after the snapshot 2023-10-30T13:25:53.349 INFO:tasks.workunit.client.0.smithi046.stdout: create 186/.snap/snap-subdir-test 2023-10-30T13:25:53.947 INFO:tasks.workunit.client.0.smithi046.stdout: create 186/30 file after the snapshot 2023-10-30T13:25:53.948 INFO:tasks.workunit.client.0.smithi046.stdout: create 186/31 file after the snapshot 2023-10-30T13:25:53.949 INFO:tasks.workunit.client.0.smithi046.stdout: create 186/32 file after the snapshot 2023-10-30T13:25:53.949 INFO:tasks.workunit.client.0.smithi046.stdout: create 186/33 file after the snapshot 2023-10-30T13:25:53.949 INFO:tasks.workunit.client.0.smithi046.stdout: create 186/34 file after the snapshot 2023-10-30T13:25:53.949 INFO:tasks.workunit.client.0.smithi046.stdout: create 186/35 file after the snapshot 2023-10-30T13:25:53.949 INFO:tasks.workunit.client.0.smithi046.stdout: create 186/36 file after the snapshot 2023-10-30T13:25:53.949 INFO:tasks.workunit.client.0.smithi046.stdout: create 186/37 file after the snapshot 2023-10-30T13:25:53.950 INFO:tasks.workunit.client.0.smithi046.stdout: create 186/38 file after the snapshot 2023-10-30T13:25:53.950 INFO:tasks.workunit.client.0.smithi046.stdout: create 186/39 file after the snapshot 2023-10-30T13:25:53.950 INFO:tasks.workunit.client.0.smithi046.stdout: create 187/.snap/snap-subdir-test 2023-10-30T13:25:54.291 INFO:tasks.workunit.client.0.smithi046.stdout: create 187/30 file after the snapshot 2023-10-30T13:25:54.294 INFO:tasks.workunit.client.0.smithi046.stdout: create 187/31 file after the snapshot 2023-10-30T13:25:54.294 INFO:tasks.workunit.client.0.smithi046.stdout: create 187/32 file after the snapshot 2023-10-30T13:25:54.294 INFO:tasks.workunit.client.0.smithi046.stdout: create 187/33 file after the snapshot 2023-10-30T13:25:54.294 INFO:tasks.workunit.client.0.smithi046.stdout: create 187/34 file after the snapshot 2023-10-30T13:25:54.294 INFO:tasks.workunit.client.0.smithi046.stdout: create 187/35 file after the snapshot 2023-10-30T13:25:54.295 INFO:tasks.workunit.client.0.smithi046.stdout: create 187/36 file after the snapshot 2023-10-30T13:25:54.295 INFO:tasks.workunit.client.0.smithi046.stdout: create 187/37 file after the snapshot 2023-10-30T13:25:54.295 INFO:tasks.workunit.client.0.smithi046.stdout: create 187/38 file after the snapshot 2023-10-30T13:25:54.295 INFO:tasks.workunit.client.0.smithi046.stdout: create 187/39 file after the snapshot 2023-10-30T13:25:54.295 INFO:tasks.workunit.client.0.smithi046.stdout: create 188/.snap/snap-subdir-test 2023-10-30T13:25:54.333 INFO:tasks.workunit.client.0.smithi046.stdout: create 188/30 file after the snapshot 2023-10-30T13:25:54.335 INFO:tasks.workunit.client.0.smithi046.stdout: create 188/31 file after the snapshot 2023-10-30T13:25:54.335 INFO:tasks.workunit.client.0.smithi046.stdout: create 188/32 file after the snapshot 2023-10-30T13:25:54.335 INFO:tasks.workunit.client.0.smithi046.stdout: create 188/33 file after the snapshot 2023-10-30T13:25:54.335 INFO:tasks.workunit.client.0.smithi046.stdout: create 188/34 file after the snapshot 2023-10-30T13:25:54.336 INFO:tasks.workunit.client.0.smithi046.stdout: create 188/35 file after the snapshot 2023-10-30T13:25:54.336 INFO:tasks.workunit.client.0.smithi046.stdout: create 188/36 file after the snapshot 2023-10-30T13:25:54.336 INFO:tasks.workunit.client.0.smithi046.stdout: create 188/37 file after the snapshot 2023-10-30T13:25:54.336 INFO:tasks.workunit.client.0.smithi046.stdout: create 188/38 file after the snapshot 2023-10-30T13:25:54.336 INFO:tasks.workunit.client.0.smithi046.stdout: create 188/39 file after the snapshot 2023-10-30T13:25:54.337 INFO:tasks.workunit.client.0.smithi046.stdout: create 189/.snap/snap-subdir-test 2023-10-30T13:25:54.367 INFO:tasks.workunit.client.0.smithi046.stdout: create 189/30 file after the snapshot 2023-10-30T13:25:54.368 INFO:tasks.workunit.client.0.smithi046.stdout: create 189/31 file after the snapshot 2023-10-30T13:25:54.368 INFO:tasks.workunit.client.0.smithi046.stdout: create 189/32 file after the snapshot 2023-10-30T13:25:54.368 INFO:tasks.workunit.client.0.smithi046.stdout: create 189/33 file after the snapshot 2023-10-30T13:25:54.369 INFO:tasks.workunit.client.0.smithi046.stdout: create 189/34 file after the snapshot 2023-10-30T13:25:54.369 INFO:tasks.workunit.client.0.smithi046.stdout: create 189/35 file after the snapshot 2023-10-30T13:25:54.369 INFO:tasks.workunit.client.0.smithi046.stdout: create 189/36 file after the snapshot 2023-10-30T13:25:54.369 INFO:tasks.workunit.client.0.smithi046.stdout: create 189/37 file after the snapshot 2023-10-30T13:25:54.369 INFO:tasks.workunit.client.0.smithi046.stdout: create 189/38 file after the snapshot 2023-10-30T13:25:54.370 INFO:tasks.workunit.client.0.smithi046.stdout: create 189/39 file after the snapshot 2023-10-30T13:25:54.370 INFO:tasks.workunit.client.0.smithi046.stdout: create 190/.snap/snap-subdir-test 2023-10-30T13:25:54.400 INFO:tasks.workunit.client.0.smithi046.stdout: create 190/30 file after the snapshot 2023-10-30T13:25:54.402 INFO:tasks.workunit.client.0.smithi046.stdout: create 190/31 file after the snapshot 2023-10-30T13:25:54.402 INFO:tasks.workunit.client.0.smithi046.stdout: create 190/32 file after the snapshot 2023-10-30T13:25:54.402 INFO:tasks.workunit.client.0.smithi046.stdout: create 190/33 file after the snapshot 2023-10-30T13:25:54.403 INFO:tasks.workunit.client.0.smithi046.stdout: create 190/34 file after the snapshot 2023-10-30T13:25:54.403 INFO:tasks.workunit.client.0.smithi046.stdout: create 190/35 file after the snapshot 2023-10-30T13:25:54.403 INFO:tasks.workunit.client.0.smithi046.stdout: create 190/36 file after the snapshot 2023-10-30T13:25:54.403 INFO:tasks.workunit.client.0.smithi046.stdout: create 190/37 file after the snapshot 2023-10-30T13:25:54.403 INFO:tasks.workunit.client.0.smithi046.stdout: create 190/38 file after the snapshot 2023-10-30T13:25:54.403 INFO:tasks.workunit.client.0.smithi046.stdout: create 190/39 file after the snapshot 2023-10-30T13:25:54.404 INFO:tasks.workunit.client.0.smithi046.stdout: create 191/.snap/snap-subdir-test 2023-10-30T13:25:54.831 INFO:tasks.workunit.client.0.smithi046.stdout: create 191/30 file after the snapshot 2023-10-30T13:25:54.832 INFO:tasks.workunit.client.0.smithi046.stdout: create 191/31 file after the snapshot 2023-10-30T13:25:54.832 INFO:tasks.workunit.client.0.smithi046.stdout: create 191/32 file after the snapshot 2023-10-30T13:25:54.833 INFO:tasks.workunit.client.0.smithi046.stdout: create 191/33 file after the snapshot 2023-10-30T13:25:54.833 INFO:tasks.workunit.client.0.smithi046.stdout: create 191/34 file after the snapshot 2023-10-30T13:25:54.833 INFO:tasks.workunit.client.0.smithi046.stdout: create 191/35 file after the snapshot 2023-10-30T13:25:54.833 INFO:tasks.workunit.client.0.smithi046.stdout: create 191/36 file after the snapshot 2023-10-30T13:25:54.833 INFO:tasks.workunit.client.0.smithi046.stdout: create 191/37 file after the snapshot 2023-10-30T13:25:54.834 INFO:tasks.workunit.client.0.smithi046.stdout: create 191/38 file after the snapshot 2023-10-30T13:25:54.834 INFO:tasks.workunit.client.0.smithi046.stdout: create 191/39 file after the snapshot 2023-10-30T13:25:54.834 INFO:tasks.workunit.client.0.smithi046.stdout: create 192/.snap/snap-subdir-test 2023-10-30T13:25:54.863 INFO:tasks.workunit.client.0.smithi046.stdout: create 192/30 file after the snapshot 2023-10-30T13:25:54.865 INFO:tasks.workunit.client.0.smithi046.stdout: create 192/31 file after the snapshot 2023-10-30T13:25:54.865 INFO:tasks.workunit.client.0.smithi046.stdout: create 192/32 file after the snapshot 2023-10-30T13:25:54.865 INFO:tasks.workunit.client.0.smithi046.stdout: create 192/33 file after the snapshot 2023-10-30T13:25:54.865 INFO:tasks.workunit.client.0.smithi046.stdout: create 192/34 file after the snapshot 2023-10-30T13:25:54.866 INFO:tasks.workunit.client.0.smithi046.stdout: create 192/35 file after the snapshot 2023-10-30T13:25:54.866 INFO:tasks.workunit.client.0.smithi046.stdout: create 192/36 file after the snapshot 2023-10-30T13:25:54.866 INFO:tasks.workunit.client.0.smithi046.stdout: create 192/37 file after the snapshot 2023-10-30T13:25:54.866 INFO:tasks.workunit.client.0.smithi046.stdout: create 192/38 file after the snapshot 2023-10-30T13:25:54.867 INFO:tasks.workunit.client.0.smithi046.stdout: create 192/39 file after the snapshot 2023-10-30T13:25:54.867 INFO:tasks.workunit.client.0.smithi046.stdout: create 193/.snap/snap-subdir-test 2023-10-30T13:25:54.898 INFO:tasks.workunit.client.0.smithi046.stdout: create 193/30 file after the snapshot 2023-10-30T13:25:54.900 INFO:tasks.workunit.client.0.smithi046.stdout: create 193/31 file after the snapshot 2023-10-30T13:25:54.900 INFO:tasks.workunit.client.0.smithi046.stdout: create 193/32 file after the snapshot 2023-10-30T13:25:54.900 INFO:tasks.workunit.client.0.smithi046.stdout: create 193/33 file after the snapshot 2023-10-30T13:25:54.900 INFO:tasks.workunit.client.0.smithi046.stdout: create 193/34 file after the snapshot 2023-10-30T13:25:54.901 INFO:tasks.workunit.client.0.smithi046.stdout: create 193/35 file after the snapshot 2023-10-30T13:25:54.901 INFO:tasks.workunit.client.0.smithi046.stdout: create 193/36 file after the snapshot 2023-10-30T13:25:54.901 INFO:tasks.workunit.client.0.smithi046.stdout: create 193/37 file after the snapshot 2023-10-30T13:25:54.901 INFO:tasks.workunit.client.0.smithi046.stdout: create 193/38 file after the snapshot 2023-10-30T13:25:54.901 INFO:tasks.workunit.client.0.smithi046.stdout: create 193/39 file after the snapshot 2023-10-30T13:25:54.902 INFO:tasks.workunit.client.0.smithi046.stdout: create 194/.snap/snap-subdir-test 2023-10-30T13:25:54.931 INFO:tasks.workunit.client.0.smithi046.stdout: create 194/30 file after the snapshot 2023-10-30T13:25:54.932 INFO:tasks.workunit.client.0.smithi046.stdout: create 194/31 file after the snapshot 2023-10-30T13:25:54.933 INFO:tasks.workunit.client.0.smithi046.stdout: create 194/32 file after the snapshot 2023-10-30T13:25:54.933 INFO:tasks.workunit.client.0.smithi046.stdout: create 194/33 file after the snapshot 2023-10-30T13:25:54.933 INFO:tasks.workunit.client.0.smithi046.stdout: create 194/34 file after the snapshot 2023-10-30T13:25:54.933 INFO:tasks.workunit.client.0.smithi046.stdout: create 194/35 file after the snapshot 2023-10-30T13:25:54.933 INFO:tasks.workunit.client.0.smithi046.stdout: create 194/36 file after the snapshot 2023-10-30T13:25:54.934 INFO:tasks.workunit.client.0.smithi046.stdout: create 194/37 file after the snapshot 2023-10-30T13:25:54.934 INFO:tasks.workunit.client.0.smithi046.stdout: create 194/38 file after the snapshot 2023-10-30T13:25:54.934 INFO:tasks.workunit.client.0.smithi046.stdout: create 194/39 file after the snapshot 2023-10-30T13:25:54.934 INFO:tasks.workunit.client.0.smithi046.stdout: create 195/.snap/snap-subdir-test 2023-10-30T13:25:54.965 INFO:tasks.workunit.client.0.smithi046.stdout: create 195/30 file after the snapshot 2023-10-30T13:25:54.967 INFO:tasks.workunit.client.0.smithi046.stdout: create 195/31 file after the snapshot 2023-10-30T13:25:54.967 INFO:tasks.workunit.client.0.smithi046.stdout: create 195/32 file after the snapshot 2023-10-30T13:25:54.967 INFO:tasks.workunit.client.0.smithi046.stdout: create 195/33 file after the snapshot 2023-10-30T13:25:54.968 INFO:tasks.workunit.client.0.smithi046.stdout: create 195/34 file after the snapshot 2023-10-30T13:25:54.968 INFO:tasks.workunit.client.0.smithi046.stdout: create 195/35 file after the snapshot 2023-10-30T13:25:54.968 INFO:tasks.workunit.client.0.smithi046.stdout: create 195/36 file after the snapshot 2023-10-30T13:25:54.968 INFO:tasks.workunit.client.0.smithi046.stdout: create 195/37 file after the snapshot 2023-10-30T13:25:54.968 INFO:tasks.workunit.client.0.smithi046.stdout: create 195/38 file after the snapshot 2023-10-30T13:25:54.969 INFO:tasks.workunit.client.0.smithi046.stdout: create 195/39 file after the snapshot 2023-10-30T13:25:54.969 INFO:tasks.workunit.client.0.smithi046.stdout: create 196/.snap/snap-subdir-test 2023-10-30T13:25:55.154 INFO:tasks.workunit.client.0.smithi046.stdout: create 196/30 file after the snapshot 2023-10-30T13:25:55.156 INFO:tasks.workunit.client.0.smithi046.stdout: create 196/31 file after the snapshot 2023-10-30T13:25:55.156 INFO:tasks.workunit.client.0.smithi046.stdout: create 196/32 file after the snapshot 2023-10-30T13:25:55.156 INFO:tasks.workunit.client.0.smithi046.stdout: create 196/33 file after the snapshot 2023-10-30T13:25:55.156 INFO:tasks.workunit.client.0.smithi046.stdout: create 196/34 file after the snapshot 2023-10-30T13:25:55.157 INFO:tasks.workunit.client.0.smithi046.stdout: create 196/35 file after the snapshot 2023-10-30T13:25:55.157 INFO:tasks.workunit.client.0.smithi046.stdout: create 196/36 file after the snapshot 2023-10-30T13:25:55.157 INFO:tasks.workunit.client.0.smithi046.stdout: create 196/37 file after the snapshot 2023-10-30T13:25:55.157 INFO:tasks.workunit.client.0.smithi046.stdout: create 196/38 file after the snapshot 2023-10-30T13:25:55.157 INFO:tasks.workunit.client.0.smithi046.stdout: create 196/39 file after the snapshot 2023-10-30T13:25:55.158 INFO:tasks.workunit.client.0.smithi046.stdout: create 197/.snap/snap-subdir-test 2023-10-30T13:25:55.185 INFO:tasks.workunit.client.0.smithi046.stdout: create 197/30 file after the snapshot 2023-10-30T13:25:55.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 197/31 file after the snapshot 2023-10-30T13:25:55.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 197/32 file after the snapshot 2023-10-30T13:25:55.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 197/33 file after the snapshot 2023-10-30T13:25:55.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 197/34 file after the snapshot 2023-10-30T13:25:55.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 197/35 file after the snapshot 2023-10-30T13:25:55.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 197/36 file after the snapshot 2023-10-30T13:25:55.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 197/37 file after the snapshot 2023-10-30T13:25:55.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 197/38 file after the snapshot 2023-10-30T13:25:55.191 INFO:tasks.workunit.client.0.smithi046.stdout: create 197/39 file after the snapshot 2023-10-30T13:25:55.191 INFO:tasks.workunit.client.0.smithi046.stdout: create 198/.snap/snap-subdir-test 2023-10-30T13:25:55.210 INFO:tasks.workunit.client.0.smithi046.stdout: create 198/30 file after the snapshot 2023-10-30T13:25:55.211 INFO:tasks.workunit.client.0.smithi046.stdout: create 198/31 file after the snapshot 2023-10-30T13:25:55.212 INFO:tasks.workunit.client.0.smithi046.stdout: create 198/32 file after the snapshot 2023-10-30T13:25:55.212 INFO:tasks.workunit.client.0.smithi046.stdout: create 198/33 file after the snapshot 2023-10-30T13:25:55.212 INFO:tasks.workunit.client.0.smithi046.stdout: create 198/34 file after the snapshot 2023-10-30T13:25:55.212 INFO:tasks.workunit.client.0.smithi046.stdout: create 198/35 file after the snapshot 2023-10-30T13:25:55.212 INFO:tasks.workunit.client.0.smithi046.stdout: create 198/36 file after the snapshot 2023-10-30T13:25:55.213 INFO:tasks.workunit.client.0.smithi046.stdout: create 198/37 file after the snapshot 2023-10-30T13:25:55.213 INFO:tasks.workunit.client.0.smithi046.stdout: create 198/38 file after the snapshot 2023-10-30T13:25:55.213 INFO:tasks.workunit.client.0.smithi046.stdout: create 198/39 file after the snapshot 2023-10-30T13:25:55.213 INFO:tasks.workunit.client.0.smithi046.stdout: create 199/.snap/snap-subdir-test 2023-10-30T13:25:55.235 INFO:tasks.workunit.client.0.smithi046.stdout: create 199/30 file after the snapshot 2023-10-30T13:25:55.237 INFO:tasks.workunit.client.0.smithi046.stdout: create 199/31 file after the snapshot 2023-10-30T13:25:55.237 INFO:tasks.workunit.client.0.smithi046.stdout: create 199/32 file after the snapshot 2023-10-30T13:25:55.237 INFO:tasks.workunit.client.0.smithi046.stdout: create 199/33 file after the snapshot 2023-10-30T13:25:55.237 INFO:tasks.workunit.client.0.smithi046.stdout: create 199/34 file after the snapshot 2023-10-30T13:25:55.238 INFO:tasks.workunit.client.0.smithi046.stdout: create 199/35 file after the snapshot 2023-10-30T13:25:55.238 INFO:tasks.workunit.client.0.smithi046.stdout: create 199/36 file after the snapshot 2023-10-30T13:25:55.238 INFO:tasks.workunit.client.0.smithi046.stdout: create 199/37 file after the snapshot 2023-10-30T13:25:55.238 INFO:tasks.workunit.client.0.smithi046.stdout: create 199/38 file after the snapshot 2023-10-30T13:25:55.238 INFO:tasks.workunit.client.0.smithi046.stdout: create 199/39 file after the snapshot 2023-10-30T13:25:55.239 INFO:tasks.workunit.client.0.smithi046.stdout: create 200/.snap/snap-subdir-test 2023-10-30T13:25:55.421 INFO:tasks.workunit.client.0.smithi046.stdout: create 200/30 file after the snapshot 2023-10-30T13:25:55.423 INFO:tasks.workunit.client.0.smithi046.stdout: create 200/31 file after the snapshot 2023-10-30T13:25:55.423 INFO:tasks.workunit.client.0.smithi046.stdout: create 200/32 file after the snapshot 2023-10-30T13:25:55.423 INFO:tasks.workunit.client.0.smithi046.stdout: create 200/33 file after the snapshot 2023-10-30T13:25:55.423 INFO:tasks.workunit.client.0.smithi046.stdout: create 200/34 file after the snapshot 2023-10-30T13:25:55.423 INFO:tasks.workunit.client.0.smithi046.stdout: create 200/35 file after the snapshot 2023-10-30T13:25:55.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 200/36 file after the snapshot 2023-10-30T13:25:55.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 200/37 file after the snapshot 2023-10-30T13:25:55.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 200/38 file after the snapshot 2023-10-30T13:25:55.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 200/39 file after the snapshot 2023-10-30T13:25:55.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 201/.snap/snap-subdir-test 2023-10-30T13:25:55.447 INFO:tasks.workunit.client.0.smithi046.stdout: create 201/30 file after the snapshot 2023-10-30T13:25:55.449 INFO:tasks.workunit.client.0.smithi046.stdout: create 201/31 file after the snapshot 2023-10-30T13:25:55.449 INFO:tasks.workunit.client.0.smithi046.stdout: create 201/32 file after the snapshot 2023-10-30T13:25:55.449 INFO:tasks.workunit.client.0.smithi046.stdout: create 201/33 file after the snapshot 2023-10-30T13:25:55.449 INFO:tasks.workunit.client.0.smithi046.stdout: create 201/34 file after the snapshot 2023-10-30T13:25:55.449 INFO:tasks.workunit.client.0.smithi046.stdout: create 201/35 file after the snapshot 2023-10-30T13:25:55.450 INFO:tasks.workunit.client.0.smithi046.stdout: create 201/36 file after the snapshot 2023-10-30T13:25:55.450 INFO:tasks.workunit.client.0.smithi046.stdout: create 201/37 file after the snapshot 2023-10-30T13:25:55.450 INFO:tasks.workunit.client.0.smithi046.stdout: create 201/38 file after the snapshot 2023-10-30T13:25:55.450 INFO:tasks.workunit.client.0.smithi046.stdout: create 201/39 file after the snapshot 2023-10-30T13:25:55.450 INFO:tasks.workunit.client.0.smithi046.stdout: create 202/.snap/snap-subdir-test 2023-10-30T13:25:55.475 INFO:tasks.workunit.client.0.smithi046.stdout: create 202/30 file after the snapshot 2023-10-30T13:25:55.477 INFO:tasks.workunit.client.0.smithi046.stdout: create 202/31 file after the snapshot 2023-10-30T13:25:55.477 INFO:tasks.workunit.client.0.smithi046.stdout: create 202/32 file after the snapshot 2023-10-30T13:25:55.477 INFO:tasks.workunit.client.0.smithi046.stdout: create 202/33 file after the snapshot 2023-10-30T13:25:55.477 INFO:tasks.workunit.client.0.smithi046.stdout: create 202/34 file after the snapshot 2023-10-30T13:25:55.478 INFO:tasks.workunit.client.0.smithi046.stdout: create 202/35 file after the snapshot 2023-10-30T13:25:55.478 INFO:tasks.workunit.client.0.smithi046.stdout: create 202/36 file after the snapshot 2023-10-30T13:25:55.478 INFO:tasks.workunit.client.0.smithi046.stdout: create 202/37 file after the snapshot 2023-10-30T13:25:55.478 INFO:tasks.workunit.client.0.smithi046.stdout: create 202/38 file after the snapshot 2023-10-30T13:25:55.478 INFO:tasks.workunit.client.0.smithi046.stdout: create 202/39 file after the snapshot 2023-10-30T13:25:55.479 INFO:tasks.workunit.client.0.smithi046.stdout: create 203/.snap/snap-subdir-test 2023-10-30T13:25:55.501 INFO:tasks.workunit.client.0.smithi046.stdout: create 203/30 file after the snapshot 2023-10-30T13:25:55.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 203/31 file after the snapshot 2023-10-30T13:25:55.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 203/32 file after the snapshot 2023-10-30T13:25:55.504 INFO:tasks.workunit.client.0.smithi046.stdout: create 203/33 file after the snapshot 2023-10-30T13:25:55.504 INFO:tasks.workunit.client.0.smithi046.stdout: create 203/34 file after the snapshot 2023-10-30T13:25:55.504 INFO:tasks.workunit.client.0.smithi046.stdout: create 203/35 file after the snapshot 2023-10-30T13:25:55.504 INFO:tasks.workunit.client.0.smithi046.stdout: create 203/36 file after the snapshot 2023-10-30T13:25:55.504 INFO:tasks.workunit.client.0.smithi046.stdout: create 203/37 file after the snapshot 2023-10-30T13:25:55.505 INFO:tasks.workunit.client.0.smithi046.stdout: create 203/38 file after the snapshot 2023-10-30T13:25:55.505 INFO:tasks.workunit.client.0.smithi046.stdout: create 203/39 file after the snapshot 2023-10-30T13:25:55.505 INFO:tasks.workunit.client.0.smithi046.stdout: create 204/.snap/snap-subdir-test 2023-10-30T13:25:55.528 INFO:tasks.workunit.client.0.smithi046.stdout: create 204/30 file after the snapshot 2023-10-30T13:25:55.530 INFO:tasks.workunit.client.0.smithi046.stdout: create 204/31 file after the snapshot 2023-10-30T13:25:55.530 INFO:tasks.workunit.client.0.smithi046.stdout: create 204/32 file after the snapshot 2023-10-30T13:25:55.530 INFO:tasks.workunit.client.0.smithi046.stdout: create 204/33 file after the snapshot 2023-10-30T13:25:55.530 INFO:tasks.workunit.client.0.smithi046.stdout: create 204/34 file after the snapshot 2023-10-30T13:25:55.531 INFO:tasks.workunit.client.0.smithi046.stdout: create 204/35 file after the snapshot 2023-10-30T13:25:55.531 INFO:tasks.workunit.client.0.smithi046.stdout: create 204/36 file after the snapshot 2023-10-30T13:25:55.531 INFO:tasks.workunit.client.0.smithi046.stdout: create 204/37 file after the snapshot 2023-10-30T13:25:55.531 INFO:tasks.workunit.client.0.smithi046.stdout: create 204/38 file after the snapshot 2023-10-30T13:25:55.531 INFO:tasks.workunit.client.0.smithi046.stdout: create 204/39 file after the snapshot 2023-10-30T13:25:55.532 INFO:tasks.workunit.client.0.smithi046.stdout: create 205/.snap/snap-subdir-test 2023-10-30T13:25:55.555 INFO:tasks.workunit.client.0.smithi046.stdout: create 205/30 file after the snapshot 2023-10-30T13:25:55.557 INFO:tasks.workunit.client.0.smithi046.stdout: create 205/31 file after the snapshot 2023-10-30T13:25:55.557 INFO:tasks.workunit.client.0.smithi046.stdout: create 205/32 file after the snapshot 2023-10-30T13:25:55.558 INFO:tasks.workunit.client.0.smithi046.stdout: create 205/33 file after the snapshot 2023-10-30T13:25:55.558 INFO:tasks.workunit.client.0.smithi046.stdout: create 205/34 file after the snapshot 2023-10-30T13:25:55.558 INFO:tasks.workunit.client.0.smithi046.stdout: create 205/35 file after the snapshot 2023-10-30T13:25:55.558 INFO:tasks.workunit.client.0.smithi046.stdout: create 205/36 file after the snapshot 2023-10-30T13:25:55.558 INFO:tasks.workunit.client.0.smithi046.stdout: create 205/37 file after the snapshot 2023-10-30T13:25:55.559 INFO:tasks.workunit.client.0.smithi046.stdout: create 205/38 file after the snapshot 2023-10-30T13:25:55.559 INFO:tasks.workunit.client.0.smithi046.stdout: create 205/39 file after the snapshot 2023-10-30T13:25:55.559 INFO:tasks.workunit.client.0.smithi046.stdout: create 206/.snap/snap-subdir-test 2023-10-30T13:25:55.581 INFO:tasks.workunit.client.0.smithi046.stdout: create 206/30 file after the snapshot 2023-10-30T13:25:55.583 INFO:tasks.workunit.client.0.smithi046.stdout: create 206/31 file after the snapshot 2023-10-30T13:25:55.583 INFO:tasks.workunit.client.0.smithi046.stdout: create 206/32 file after the snapshot 2023-10-30T13:25:55.583 INFO:tasks.workunit.client.0.smithi046.stdout: create 206/33 file after the snapshot 2023-10-30T13:25:55.584 INFO:tasks.workunit.client.0.smithi046.stdout: create 206/34 file after the snapshot 2023-10-30T13:25:55.584 INFO:tasks.workunit.client.0.smithi046.stdout: create 206/35 file after the snapshot 2023-10-30T13:25:55.584 INFO:tasks.workunit.client.0.smithi046.stdout: create 206/36 file after the snapshot 2023-10-30T13:25:55.584 INFO:tasks.workunit.client.0.smithi046.stdout: create 206/37 file after the snapshot 2023-10-30T13:25:55.584 INFO:tasks.workunit.client.0.smithi046.stdout: create 206/38 file after the snapshot 2023-10-30T13:25:55.585 INFO:tasks.workunit.client.0.smithi046.stdout: create 206/39 file after the snapshot 2023-10-30T13:25:55.585 INFO:tasks.workunit.client.0.smithi046.stdout: create 207/.snap/snap-subdir-test 2023-10-30T13:25:55.608 INFO:tasks.workunit.client.0.smithi046.stdout: create 207/30 file after the snapshot 2023-10-30T13:25:55.609 INFO:tasks.workunit.client.0.smithi046.stdout: create 207/31 file after the snapshot 2023-10-30T13:25:55.609 INFO:tasks.workunit.client.0.smithi046.stdout: create 207/32 file after the snapshot 2023-10-30T13:25:55.610 INFO:tasks.workunit.client.0.smithi046.stdout: create 207/33 file after the snapshot 2023-10-30T13:25:55.610 INFO:tasks.workunit.client.0.smithi046.stdout: create 207/34 file after the snapshot 2023-10-30T13:25:55.610 INFO:tasks.workunit.client.0.smithi046.stdout: create 207/35 file after the snapshot 2023-10-30T13:25:55.610 INFO:tasks.workunit.client.0.smithi046.stdout: create 207/36 file after the snapshot 2023-10-30T13:25:55.610 INFO:tasks.workunit.client.0.smithi046.stdout: create 207/37 file after the snapshot 2023-10-30T13:25:55.610 INFO:tasks.workunit.client.0.smithi046.stdout: create 207/38 file after the snapshot 2023-10-30T13:25:55.611 INFO:tasks.workunit.client.0.smithi046.stdout: create 207/39 file after the snapshot 2023-10-30T13:25:55.611 INFO:tasks.workunit.client.0.smithi046.stdout: create 208/.snap/snap-subdir-test 2023-10-30T13:25:56.116 INFO:tasks.workunit.client.0.smithi046.stdout: create 208/30 file after the snapshot 2023-10-30T13:25:56.118 INFO:tasks.workunit.client.0.smithi046.stdout: create 208/31 file after the snapshot 2023-10-30T13:25:56.118 INFO:tasks.workunit.client.0.smithi046.stdout: create 208/32 file after the snapshot 2023-10-30T13:25:56.118 INFO:tasks.workunit.client.0.smithi046.stdout: create 208/33 file after the snapshot 2023-10-30T13:25:56.118 INFO:tasks.workunit.client.0.smithi046.stdout: create 208/34 file after the snapshot 2023-10-30T13:25:56.118 INFO:tasks.workunit.client.0.smithi046.stdout: create 208/35 file after the snapshot 2023-10-30T13:25:56.119 INFO:tasks.workunit.client.0.smithi046.stdout: create 208/36 file after the snapshot 2023-10-30T13:25:56.119 INFO:tasks.workunit.client.0.smithi046.stdout: create 208/37 file after the snapshot 2023-10-30T13:25:56.119 INFO:tasks.workunit.client.0.smithi046.stdout: create 208/38 file after the snapshot 2023-10-30T13:25:56.119 INFO:tasks.workunit.client.0.smithi046.stdout: create 208/39 file after the snapshot 2023-10-30T13:25:56.120 INFO:tasks.workunit.client.0.smithi046.stdout: create 209/.snap/snap-subdir-test 2023-10-30T13:25:56.152 INFO:tasks.workunit.client.0.smithi046.stdout: create 209/30 file after the snapshot 2023-10-30T13:25:56.156 INFO:tasks.workunit.client.0.smithi046.stdout: create 209/31 file after the snapshot 2023-10-30T13:25:56.156 INFO:tasks.workunit.client.0.smithi046.stdout: create 209/32 file after the snapshot 2023-10-30T13:25:56.156 INFO:tasks.workunit.client.0.smithi046.stdout: create 209/33 file after the snapshot 2023-10-30T13:25:56.156 INFO:tasks.workunit.client.0.smithi046.stdout: create 209/34 file after the snapshot 2023-10-30T13:25:56.157 INFO:tasks.workunit.client.0.smithi046.stdout: create 209/35 file after the snapshot 2023-10-30T13:25:56.157 INFO:tasks.workunit.client.0.smithi046.stdout: create 209/36 file after the snapshot 2023-10-30T13:25:56.157 INFO:tasks.workunit.client.0.smithi046.stdout: create 209/37 file after the snapshot 2023-10-30T13:25:56.157 INFO:tasks.workunit.client.0.smithi046.stdout: create 209/38 file after the snapshot 2023-10-30T13:25:56.157 INFO:tasks.workunit.client.0.smithi046.stdout: create 209/39 file after the snapshot 2023-10-30T13:25:56.157 INFO:tasks.workunit.client.0.smithi046.stdout: create 210/.snap/snap-subdir-test 2023-10-30T13:25:56.191 INFO:tasks.workunit.client.0.smithi046.stdout: create 210/30 file after the snapshot 2023-10-30T13:25:56.194 INFO:tasks.workunit.client.0.smithi046.stdout: create 210/31 file after the snapshot 2023-10-30T13:25:56.194 INFO:tasks.workunit.client.0.smithi046.stdout: create 210/32 file after the snapshot 2023-10-30T13:25:56.195 INFO:tasks.workunit.client.0.smithi046.stdout: create 210/33 file after the snapshot 2023-10-30T13:25:56.195 INFO:tasks.workunit.client.0.smithi046.stdout: create 210/34 file after the snapshot 2023-10-30T13:25:56.195 INFO:tasks.workunit.client.0.smithi046.stdout: create 210/35 file after the snapshot 2023-10-30T13:25:56.195 INFO:tasks.workunit.client.0.smithi046.stdout: create 210/36 file after the snapshot 2023-10-30T13:25:56.195 INFO:tasks.workunit.client.0.smithi046.stdout: create 210/37 file after the snapshot 2023-10-30T13:25:56.196 INFO:tasks.workunit.client.0.smithi046.stdout: create 210/38 file after the snapshot 2023-10-30T13:25:56.196 INFO:tasks.workunit.client.0.smithi046.stdout: create 210/39 file after the snapshot 2023-10-30T13:25:56.196 INFO:tasks.workunit.client.0.smithi046.stdout: create 211/.snap/snap-subdir-test 2023-10-30T13:25:56.483 INFO:tasks.workunit.client.0.smithi046.stdout: create 211/30 file after the snapshot 2023-10-30T13:25:56.486 INFO:tasks.workunit.client.0.smithi046.stdout: create 211/31 file after the snapshot 2023-10-30T13:25:56.486 INFO:tasks.workunit.client.0.smithi046.stdout: create 211/32 file after the snapshot 2023-10-30T13:25:56.487 INFO:tasks.workunit.client.0.smithi046.stdout: create 211/33 file after the snapshot 2023-10-30T13:25:56.487 INFO:tasks.workunit.client.0.smithi046.stdout: create 211/34 file after the snapshot 2023-10-30T13:25:56.487 INFO:tasks.workunit.client.0.smithi046.stdout: create 211/35 file after the snapshot 2023-10-30T13:25:56.487 INFO:tasks.workunit.client.0.smithi046.stdout: create 211/36 file after the snapshot 2023-10-30T13:25:56.487 INFO:tasks.workunit.client.0.smithi046.stdout: create 211/37 file after the snapshot 2023-10-30T13:25:56.487 INFO:tasks.workunit.client.0.smithi046.stdout: create 211/38 file after the snapshot 2023-10-30T13:25:56.488 INFO:tasks.workunit.client.0.smithi046.stdout: create 211/39 file after the snapshot 2023-10-30T13:25:56.488 INFO:tasks.workunit.client.0.smithi046.stdout: create 212/.snap/snap-subdir-test 2023-10-30T13:25:56.535 INFO:tasks.workunit.client.0.smithi046.stdout: create 212/30 file after the snapshot 2023-10-30T13:25:56.537 INFO:tasks.workunit.client.0.smithi046.stdout: create 212/31 file after the snapshot 2023-10-30T13:25:56.537 INFO:tasks.workunit.client.0.smithi046.stdout: create 212/32 file after the snapshot 2023-10-30T13:25:56.537 INFO:tasks.workunit.client.0.smithi046.stdout: create 212/33 file after the snapshot 2023-10-30T13:25:56.538 INFO:tasks.workunit.client.0.smithi046.stdout: create 212/34 file after the snapshot 2023-10-30T13:25:56.538 INFO:tasks.workunit.client.0.smithi046.stdout: create 212/35 file after the snapshot 2023-10-30T13:25:56.538 INFO:tasks.workunit.client.0.smithi046.stdout: create 212/36 file after the snapshot 2023-10-30T13:25:56.538 INFO:tasks.workunit.client.0.smithi046.stdout: create 212/37 file after the snapshot 2023-10-30T13:25:56.538 INFO:tasks.workunit.client.0.smithi046.stdout: create 212/38 file after the snapshot 2023-10-30T13:25:56.539 INFO:tasks.workunit.client.0.smithi046.stdout: create 212/39 file after the snapshot 2023-10-30T13:25:56.539 INFO:tasks.workunit.client.0.smithi046.stdout: create 213/.snap/snap-subdir-test 2023-10-30T13:25:56.568 INFO:tasks.workunit.client.0.smithi046.stdout: create 213/30 file after the snapshot 2023-10-30T13:25:56.570 INFO:tasks.workunit.client.0.smithi046.stdout: create 213/31 file after the snapshot 2023-10-30T13:25:56.570 INFO:tasks.workunit.client.0.smithi046.stdout: create 213/32 file after the snapshot 2023-10-30T13:25:56.571 INFO:tasks.workunit.client.0.smithi046.stdout: create 213/33 file after the snapshot 2023-10-30T13:25:56.571 INFO:tasks.workunit.client.0.smithi046.stdout: create 213/34 file after the snapshot 2023-10-30T13:25:56.571 INFO:tasks.workunit.client.0.smithi046.stdout: create 213/35 file after the snapshot 2023-10-30T13:25:56.571 INFO:tasks.workunit.client.0.smithi046.stdout: create 213/36 file after the snapshot 2023-10-30T13:25:56.571 INFO:tasks.workunit.client.0.smithi046.stdout: create 213/37 file after the snapshot 2023-10-30T13:25:56.572 INFO:tasks.workunit.client.0.smithi046.stdout: create 213/38 file after the snapshot 2023-10-30T13:25:56.572 INFO:tasks.workunit.client.0.smithi046.stdout: create 213/39 file after the snapshot 2023-10-30T13:25:56.572 INFO:tasks.workunit.client.0.smithi046.stdout: create 214/.snap/snap-subdir-test 2023-10-30T13:25:56.611 INFO:tasks.workunit.client.0.smithi046.stdout: create 214/30 file after the snapshot 2023-10-30T13:25:56.613 INFO:tasks.workunit.client.0.smithi046.stdout: create 214/31 file after the snapshot 2023-10-30T13:25:56.613 INFO:tasks.workunit.client.0.smithi046.stdout: create 214/32 file after the snapshot 2023-10-30T13:25:56.614 INFO:tasks.workunit.client.0.smithi046.stdout: create 214/33 file after the snapshot 2023-10-30T13:25:56.614 INFO:tasks.workunit.client.0.smithi046.stdout: create 214/34 file after the snapshot 2023-10-30T13:25:56.614 INFO:tasks.workunit.client.0.smithi046.stdout: create 214/35 file after the snapshot 2023-10-30T13:25:56.617 INFO:tasks.workunit.client.0.smithi046.stdout: create 214/36 file after the snapshot 2023-10-30T13:25:56.617 INFO:tasks.workunit.client.0.smithi046.stdout: create 214/37 file after the snapshot 2023-10-30T13:25:56.617 INFO:tasks.workunit.client.0.smithi046.stdout: create 214/38 file after the snapshot 2023-10-30T13:25:56.617 INFO:tasks.workunit.client.0.smithi046.stdout: create 214/39 file after the snapshot 2023-10-30T13:25:56.618 INFO:tasks.workunit.client.0.smithi046.stdout: create 215/.snap/snap-subdir-test 2023-10-30T13:25:56.655 INFO:tasks.workunit.client.0.smithi046.stdout: create 215/30 file after the snapshot 2023-10-30T13:25:56.657 INFO:tasks.workunit.client.0.smithi046.stdout: create 215/31 file after the snapshot 2023-10-30T13:25:56.657 INFO:tasks.workunit.client.0.smithi046.stdout: create 215/32 file after the snapshot 2023-10-30T13:25:56.657 INFO:tasks.workunit.client.0.smithi046.stdout: create 215/33 file after the snapshot 2023-10-30T13:25:56.658 INFO:tasks.workunit.client.0.smithi046.stdout: create 215/34 file after the snapshot 2023-10-30T13:25:56.658 INFO:tasks.workunit.client.0.smithi046.stdout: create 215/35 file after the snapshot 2023-10-30T13:25:56.658 INFO:tasks.workunit.client.0.smithi046.stdout: create 215/36 file after the snapshot 2023-10-30T13:25:56.658 INFO:tasks.workunit.client.0.smithi046.stdout: create 215/37 file after the snapshot 2023-10-30T13:25:56.658 INFO:tasks.workunit.client.0.smithi046.stdout: create 215/38 file after the snapshot 2023-10-30T13:25:56.659 INFO:tasks.workunit.client.0.smithi046.stdout: create 215/39 file after the snapshot 2023-10-30T13:25:56.659 INFO:tasks.workunit.client.0.smithi046.stdout: create 216/.snap/snap-subdir-test 2023-10-30T13:25:57.609 INFO:tasks.workunit.client.0.smithi046.stdout: create 216/30 file after the snapshot 2023-10-30T13:25:58.090 INFO:tasks.workunit.client.0.smithi046.stdout: create 216/31 file after the snapshot 2023-10-30T13:25:58.091 INFO:tasks.workunit.client.0.smithi046.stdout: create 216/32 file after the snapshot 2023-10-30T13:25:58.091 INFO:tasks.workunit.client.0.smithi046.stdout: create 216/33 file after the snapshot 2023-10-30T13:25:58.091 INFO:tasks.workunit.client.0.smithi046.stdout: create 216/34 file after the snapshot 2023-10-30T13:25:58.091 INFO:tasks.workunit.client.0.smithi046.stdout: create 216/35 file after the snapshot 2023-10-30T13:25:58.092 INFO:tasks.workunit.client.0.smithi046.stdout: create 216/36 file after the snapshot 2023-10-30T13:25:58.092 INFO:tasks.workunit.client.0.smithi046.stdout: create 216/37 file after the snapshot 2023-10-30T13:25:58.093 INFO:tasks.workunit.client.0.smithi046.stdout: create 216/38 file after the snapshot 2023-10-30T13:25:58.093 INFO:tasks.workunit.client.0.smithi046.stdout: create 216/39 file after the snapshot 2023-10-30T13:25:58.093 INFO:tasks.workunit.client.0.smithi046.stdout: create 217/.snap/snap-subdir-test 2023-10-30T13:25:58.124 INFO:tasks.workunit.client.0.smithi046.stdout: create 217/30 file after the snapshot 2023-10-30T13:25:58.125 INFO:tasks.workunit.client.0.smithi046.stdout: create 217/31 file after the snapshot 2023-10-30T13:25:58.125 INFO:tasks.workunit.client.0.smithi046.stdout: create 217/32 file after the snapshot 2023-10-30T13:25:58.125 INFO:tasks.workunit.client.0.smithi046.stdout: create 217/33 file after the snapshot 2023-10-30T13:25:58.126 INFO:tasks.workunit.client.0.smithi046.stdout: create 217/34 file after the snapshot 2023-10-30T13:25:58.126 INFO:tasks.workunit.client.0.smithi046.stdout: create 217/35 file after the snapshot 2023-10-30T13:25:58.127 INFO:tasks.workunit.client.0.smithi046.stdout: create 217/36 file after the snapshot 2023-10-30T13:25:58.127 INFO:tasks.workunit.client.0.smithi046.stdout: create 217/37 file after the snapshot 2023-10-30T13:25:58.127 INFO:tasks.workunit.client.0.smithi046.stdout: create 217/38 file after the snapshot 2023-10-30T13:25:58.127 INFO:tasks.workunit.client.0.smithi046.stdout: create 217/39 file after the snapshot 2023-10-30T13:25:58.127 INFO:tasks.workunit.client.0.smithi046.stdout: create 218/.snap/snap-subdir-test 2023-10-30T13:25:58.157 INFO:tasks.workunit.client.0.smithi046.stdout: create 218/30 file after the snapshot 2023-10-30T13:25:58.159 INFO:tasks.workunit.client.0.smithi046.stdout: create 218/31 file after the snapshot 2023-10-30T13:25:58.159 INFO:tasks.workunit.client.0.smithi046.stdout: create 218/32 file after the snapshot 2023-10-30T13:25:58.159 INFO:tasks.workunit.client.0.smithi046.stdout: create 218/33 file after the snapshot 2023-10-30T13:25:58.159 INFO:tasks.workunit.client.0.smithi046.stdout: create 218/34 file after the snapshot 2023-10-30T13:25:58.159 INFO:tasks.workunit.client.0.smithi046.stdout: create 218/35 file after the snapshot 2023-10-30T13:25:58.160 INFO:tasks.workunit.client.0.smithi046.stdout: create 218/36 file after the snapshot 2023-10-30T13:25:58.160 INFO:tasks.workunit.client.0.smithi046.stdout: create 218/37 file after the snapshot 2023-10-30T13:25:58.160 INFO:tasks.workunit.client.0.smithi046.stdout: create 218/38 file after the snapshot 2023-10-30T13:25:58.160 INFO:tasks.workunit.client.0.smithi046.stdout: create 218/39 file after the snapshot 2023-10-30T13:25:58.161 INFO:tasks.workunit.client.0.smithi046.stdout: create 219/.snap/snap-subdir-test 2023-10-30T13:25:58.951 INFO:tasks.workunit.client.0.smithi046.stdout: create 219/30 file after the snapshot 2023-10-30T13:25:58.952 INFO:tasks.workunit.client.0.smithi046.stdout: create 219/31 file after the snapshot 2023-10-30T13:25:58.953 INFO:tasks.workunit.client.0.smithi046.stdout: create 219/32 file after the snapshot 2023-10-30T13:25:58.953 INFO:tasks.workunit.client.0.smithi046.stdout: create 219/33 file after the snapshot 2023-10-30T13:25:58.953 INFO:tasks.workunit.client.0.smithi046.stdout: create 219/34 file after the snapshot 2023-10-30T13:25:58.954 INFO:tasks.workunit.client.0.smithi046.stdout: create 219/35 file after the snapshot 2023-10-30T13:25:58.954 INFO:tasks.workunit.client.0.smithi046.stdout: create 219/36 file after the snapshot 2023-10-30T13:25:58.954 INFO:tasks.workunit.client.0.smithi046.stdout: create 219/37 file after the snapshot 2023-10-30T13:25:58.955 INFO:tasks.workunit.client.0.smithi046.stdout: create 219/38 file after the snapshot 2023-10-30T13:25:58.955 INFO:tasks.workunit.client.0.smithi046.stdout: create 219/39 file after the snapshot 2023-10-30T13:25:58.955 INFO:tasks.workunit.client.0.smithi046.stdout: create 220/.snap/snap-subdir-test 2023-10-30T13:25:59.146 INFO:tasks.workunit.client.0.smithi046.stdout: create 220/30 file after the snapshot 2023-10-30T13:25:59.147 INFO:tasks.workunit.client.0.smithi046.stdout: create 220/31 file after the snapshot 2023-10-30T13:25:59.148 INFO:tasks.workunit.client.0.smithi046.stdout: create 220/32 file after the snapshot 2023-10-30T13:25:59.148 INFO:tasks.workunit.client.0.smithi046.stdout: create 220/33 file after the snapshot 2023-10-30T13:25:59.148 INFO:tasks.workunit.client.0.smithi046.stdout: create 220/34 file after the snapshot 2023-10-30T13:25:59.149 INFO:tasks.workunit.client.0.smithi046.stdout: create 220/35 file after the snapshot 2023-10-30T13:25:59.149 INFO:tasks.workunit.client.0.smithi046.stdout: create 220/36 file after the snapshot 2023-10-30T13:25:59.149 INFO:tasks.workunit.client.0.smithi046.stdout: create 220/37 file after the snapshot 2023-10-30T13:25:59.150 INFO:tasks.workunit.client.0.smithi046.stdout: create 220/38 file after the snapshot 2023-10-30T13:25:59.150 INFO:tasks.workunit.client.0.smithi046.stdout: create 220/39 file after the snapshot 2023-10-30T13:25:59.150 INFO:tasks.workunit.client.0.smithi046.stdout: create 221/.snap/snap-subdir-test 2023-10-30T13:25:59.187 INFO:tasks.workunit.client.0.smithi046.stdout: create 221/30 file after the snapshot 2023-10-30T13:25:59.188 INFO:tasks.workunit.client.0.smithi046.stdout: create 221/31 file after the snapshot 2023-10-30T13:25:59.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 221/32 file after the snapshot 2023-10-30T13:25:59.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 221/33 file after the snapshot 2023-10-30T13:25:59.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 221/34 file after the snapshot 2023-10-30T13:25:59.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 221/35 file after the snapshot 2023-10-30T13:25:59.189 INFO:tasks.workunit.client.0.smithi046.stdout: create 221/36 file after the snapshot 2023-10-30T13:25:59.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 221/37 file after the snapshot 2023-10-30T13:25:59.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 221/38 file after the snapshot 2023-10-30T13:25:59.190 INFO:tasks.workunit.client.0.smithi046.stdout: create 221/39 file after the snapshot 2023-10-30T13:25:59.191 INFO:tasks.workunit.client.0.smithi046.stdout: create 222/.snap/snap-subdir-test 2023-10-30T13:25:59.224 INFO:tasks.workunit.client.0.smithi046.stdout: create 222/30 file after the snapshot 2023-10-30T13:25:59.226 INFO:tasks.workunit.client.0.smithi046.stdout: create 222/31 file after the snapshot 2023-10-30T13:25:59.226 INFO:tasks.workunit.client.0.smithi046.stdout: create 222/32 file after the snapshot 2023-10-30T13:25:59.226 INFO:tasks.workunit.client.0.smithi046.stdout: create 222/33 file after the snapshot 2023-10-30T13:25:59.226 INFO:tasks.workunit.client.0.smithi046.stdout: create 222/34 file after the snapshot 2023-10-30T13:25:59.227 INFO:tasks.workunit.client.0.smithi046.stdout: create 222/35 file after the snapshot 2023-10-30T13:25:59.227 INFO:tasks.workunit.client.0.smithi046.stdout: create 222/36 file after the snapshot 2023-10-30T13:25:59.227 INFO:tasks.workunit.client.0.smithi046.stdout: create 222/37 file after the snapshot 2023-10-30T13:25:59.227 INFO:tasks.workunit.client.0.smithi046.stdout: create 222/38 file after the snapshot 2023-10-30T13:25:59.227 INFO:tasks.workunit.client.0.smithi046.stdout: create 222/39 file after the snapshot 2023-10-30T13:25:59.228 INFO:tasks.workunit.client.0.smithi046.stdout: create 223/.snap/snap-subdir-test 2023-10-30T13:25:59.255 INFO:tasks.workunit.client.0.smithi046.stdout: create 223/30 file after the snapshot 2023-10-30T13:25:59.257 INFO:tasks.workunit.client.0.smithi046.stdout: create 223/31 file after the snapshot 2023-10-30T13:25:59.257 INFO:tasks.workunit.client.0.smithi046.stdout: create 223/32 file after the snapshot 2023-10-30T13:25:59.257 INFO:tasks.workunit.client.0.smithi046.stdout: create 223/33 file after the snapshot 2023-10-30T13:25:59.258 INFO:tasks.workunit.client.0.smithi046.stdout: create 223/34 file after the snapshot 2023-10-30T13:25:59.258 INFO:tasks.workunit.client.0.smithi046.stdout: create 223/35 file after the snapshot 2023-10-30T13:25:59.258 INFO:tasks.workunit.client.0.smithi046.stdout: create 223/36 file after the snapshot 2023-10-30T13:25:59.258 INFO:tasks.workunit.client.0.smithi046.stdout: create 223/37 file after the snapshot 2023-10-30T13:25:59.258 INFO:tasks.workunit.client.0.smithi046.stdout: create 223/38 file after the snapshot 2023-10-30T13:25:59.259 INFO:tasks.workunit.client.0.smithi046.stdout: create 223/39 file after the snapshot 2023-10-30T13:25:59.259 INFO:tasks.workunit.client.0.smithi046.stdout: create 224/.snap/snap-subdir-test 2023-10-30T13:25:59.286 INFO:tasks.workunit.client.0.smithi046.stdout: create 224/30 file after the snapshot 2023-10-30T13:25:59.290 INFO:tasks.workunit.client.0.smithi046.stdout: create 224/31 file after the snapshot 2023-10-30T13:25:59.290 INFO:tasks.workunit.client.0.smithi046.stdout: create 224/32 file after the snapshot 2023-10-30T13:25:59.290 INFO:tasks.workunit.client.0.smithi046.stdout: create 224/33 file after the snapshot 2023-10-30T13:25:59.290 INFO:tasks.workunit.client.0.smithi046.stdout: create 224/34 file after the snapshot 2023-10-30T13:25:59.290 INFO:tasks.workunit.client.0.smithi046.stdout: create 224/35 file after the snapshot 2023-10-30T13:25:59.291 INFO:tasks.workunit.client.0.smithi046.stdout: create 224/36 file after the snapshot 2023-10-30T13:25:59.291 INFO:tasks.workunit.client.0.smithi046.stdout: create 224/37 file after the snapshot 2023-10-30T13:25:59.291 INFO:tasks.workunit.client.0.smithi046.stdout: create 224/38 file after the snapshot 2023-10-30T13:25:59.291 INFO:tasks.workunit.client.0.smithi046.stdout: create 224/39 file after the snapshot 2023-10-30T13:25:59.291 INFO:tasks.workunit.client.0.smithi046.stdout: create 225/.snap/snap-subdir-test 2023-10-30T13:25:59.421 INFO:tasks.workunit.client.0.smithi046.stdout: create 225/30 file after the snapshot 2023-10-30T13:25:59.422 INFO:tasks.workunit.client.0.smithi046.stdout: create 225/31 file after the snapshot 2023-10-30T13:25:59.423 INFO:tasks.workunit.client.0.smithi046.stdout: create 225/32 file after the snapshot 2023-10-30T13:25:59.423 INFO:tasks.workunit.client.0.smithi046.stdout: create 225/33 file after the snapshot 2023-10-30T13:25:59.423 INFO:tasks.workunit.client.0.smithi046.stdout: create 225/34 file after the snapshot 2023-10-30T13:25:59.423 INFO:tasks.workunit.client.0.smithi046.stdout: create 225/35 file after the snapshot 2023-10-30T13:25:59.423 INFO:tasks.workunit.client.0.smithi046.stdout: create 225/36 file after the snapshot 2023-10-30T13:25:59.423 INFO:tasks.workunit.client.0.smithi046.stdout: create 225/37 file after the snapshot 2023-10-30T13:25:59.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 225/38 file after the snapshot 2023-10-30T13:25:59.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 225/39 file after the snapshot 2023-10-30T13:25:59.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 226/.snap/snap-subdir-test 2023-10-30T13:25:59.454 INFO:tasks.workunit.client.0.smithi046.stdout: create 226/30 file after the snapshot 2023-10-30T13:25:59.456 INFO:tasks.workunit.client.0.smithi046.stdout: create 226/31 file after the snapshot 2023-10-30T13:25:59.456 INFO:tasks.workunit.client.0.smithi046.stdout: create 226/32 file after the snapshot 2023-10-30T13:25:59.456 INFO:tasks.workunit.client.0.smithi046.stdout: create 226/33 file after the snapshot 2023-10-30T13:25:59.457 INFO:tasks.workunit.client.0.smithi046.stdout: create 226/34 file after the snapshot 2023-10-30T13:25:59.457 INFO:tasks.workunit.client.0.smithi046.stdout: create 226/35 file after the snapshot 2023-10-30T13:25:59.457 INFO:tasks.workunit.client.0.smithi046.stdout: create 226/36 file after the snapshot 2023-10-30T13:25:59.457 INFO:tasks.workunit.client.0.smithi046.stdout: create 226/37 file after the snapshot 2023-10-30T13:25:59.457 INFO:tasks.workunit.client.0.smithi046.stdout: create 226/38 file after the snapshot 2023-10-30T13:25:59.458 INFO:tasks.workunit.client.0.smithi046.stdout: create 226/39 file after the snapshot 2023-10-30T13:25:59.458 INFO:tasks.workunit.client.0.smithi046.stdout: create 227/.snap/snap-subdir-test 2023-10-30T13:25:59.486 INFO:tasks.workunit.client.0.smithi046.stdout: create 227/30 file after the snapshot 2023-10-30T13:25:59.487 INFO:tasks.workunit.client.0.smithi046.stdout: create 227/31 file after the snapshot 2023-10-30T13:25:59.487 INFO:tasks.workunit.client.0.smithi046.stdout: create 227/32 file after the snapshot 2023-10-30T13:25:59.487 INFO:tasks.workunit.client.0.smithi046.stdout: create 227/33 file after the snapshot 2023-10-30T13:25:59.488 INFO:tasks.workunit.client.0.smithi046.stdout: create 227/34 file after the snapshot 2023-10-30T13:25:59.488 INFO:tasks.workunit.client.0.smithi046.stdout: create 227/35 file after the snapshot 2023-10-30T13:25:59.488 INFO:tasks.workunit.client.0.smithi046.stdout: create 227/36 file after the snapshot 2023-10-30T13:25:59.488 INFO:tasks.workunit.client.0.smithi046.stdout: create 227/37 file after the snapshot 2023-10-30T13:25:59.489 INFO:tasks.workunit.client.0.smithi046.stdout: create 227/38 file after the snapshot 2023-10-30T13:25:59.489 INFO:tasks.workunit.client.0.smithi046.stdout: create 227/39 file after the snapshot 2023-10-30T13:25:59.489 INFO:tasks.workunit.client.0.smithi046.stdout: create 228/.snap/snap-subdir-test 2023-10-30T13:26:00.031 INFO:tasks.workunit.client.0.smithi046.stdout: create 228/30 file after the snapshot 2023-10-30T13:26:00.032 INFO:tasks.workunit.client.0.smithi046.stdout: create 228/31 file after the snapshot 2023-10-30T13:26:00.033 INFO:tasks.workunit.client.0.smithi046.stdout: create 228/32 file after the snapshot 2023-10-30T13:26:00.033 INFO:tasks.workunit.client.0.smithi046.stdout: create 228/33 file after the snapshot 2023-10-30T13:26:00.033 INFO:tasks.workunit.client.0.smithi046.stdout: create 228/34 file after the snapshot 2023-10-30T13:26:00.033 INFO:tasks.workunit.client.0.smithi046.stdout: create 228/35 file after the snapshot 2023-10-30T13:26:00.034 INFO:tasks.workunit.client.0.smithi046.stdout: create 228/36 file after the snapshot 2023-10-30T13:26:00.034 INFO:tasks.workunit.client.0.smithi046.stdout: create 228/37 file after the snapshot 2023-10-30T13:26:00.034 INFO:tasks.workunit.client.0.smithi046.stdout: create 228/38 file after the snapshot 2023-10-30T13:26:00.035 INFO:tasks.workunit.client.0.smithi046.stdout: create 228/39 file after the snapshot 2023-10-30T13:26:00.035 INFO:tasks.workunit.client.0.smithi046.stdout: create 229/.snap/snap-subdir-test 2023-10-30T13:26:00.431 INFO:tasks.workunit.client.0.smithi046.stdout: create 229/30 file after the snapshot 2023-10-30T13:26:00.432 INFO:tasks.workunit.client.0.smithi046.stdout: create 229/31 file after the snapshot 2023-10-30T13:26:00.432 INFO:tasks.workunit.client.0.smithi046.stdout: create 229/32 file after the snapshot 2023-10-30T13:26:00.432 INFO:tasks.workunit.client.0.smithi046.stdout: create 229/33 file after the snapshot 2023-10-30T13:26:00.433 INFO:tasks.workunit.client.0.smithi046.stdout: create 229/34 file after the snapshot 2023-10-30T13:26:00.433 INFO:tasks.workunit.client.0.smithi046.stdout: create 229/35 file after the snapshot 2023-10-30T13:26:00.434 INFO:tasks.workunit.client.0.smithi046.stdout: create 229/36 file after the snapshot 2023-10-30T13:26:00.434 INFO:tasks.workunit.client.0.smithi046.stdout: create 229/37 file after the snapshot 2023-10-30T13:26:00.434 INFO:tasks.workunit.client.0.smithi046.stdout: create 229/38 file after the snapshot 2023-10-30T13:26:00.434 INFO:tasks.workunit.client.0.smithi046.stdout: create 229/39 file after the snapshot 2023-10-30T13:26:00.435 INFO:tasks.workunit.client.0.smithi046.stdout: create 230/.snap/snap-subdir-test 2023-10-30T13:26:00.469 INFO:tasks.workunit.client.0.smithi046.stdout: create 230/30 file after the snapshot 2023-10-30T13:26:00.470 INFO:tasks.workunit.client.0.smithi046.stdout: create 230/31 file after the snapshot 2023-10-30T13:26:00.471 INFO:tasks.workunit.client.0.smithi046.stdout: create 230/32 file after the snapshot 2023-10-30T13:26:00.471 INFO:tasks.workunit.client.0.smithi046.stdout: create 230/33 file after the snapshot 2023-10-30T13:26:00.471 INFO:tasks.workunit.client.0.smithi046.stdout: create 230/34 file after the snapshot 2023-10-30T13:26:00.471 INFO:tasks.workunit.client.0.smithi046.stdout: create 230/35 file after the snapshot 2023-10-30T13:26:00.472 INFO:tasks.workunit.client.0.smithi046.stdout: create 230/36 file after the snapshot 2023-10-30T13:26:00.472 INFO:tasks.workunit.client.0.smithi046.stdout: create 230/37 file after the snapshot 2023-10-30T13:26:00.472 INFO:tasks.workunit.client.0.smithi046.stdout: create 230/38 file after the snapshot 2023-10-30T13:26:00.473 INFO:tasks.workunit.client.0.smithi046.stdout: create 230/39 file after the snapshot 2023-10-30T13:26:00.473 INFO:tasks.workunit.client.0.smithi046.stdout: create 231/.snap/snap-subdir-test 2023-10-30T13:26:00.760 INFO:tasks.workunit.client.0.smithi046.stdout: create 231/30 file after the snapshot 2023-10-30T13:26:00.764 INFO:tasks.workunit.client.0.smithi046.stdout: create 231/31 file after the snapshot 2023-10-30T13:26:00.764 INFO:tasks.workunit.client.0.smithi046.stdout: create 231/32 file after the snapshot 2023-10-30T13:26:00.764 INFO:tasks.workunit.client.0.smithi046.stdout: create 231/33 file after the snapshot 2023-10-30T13:26:00.764 INFO:tasks.workunit.client.0.smithi046.stdout: create 231/34 file after the snapshot 2023-10-30T13:26:00.765 INFO:tasks.workunit.client.0.smithi046.stdout: create 231/35 file after the snapshot 2023-10-30T13:26:00.766 INFO:tasks.workunit.client.0.smithi046.stdout: create 231/36 file after the snapshot 2023-10-30T13:26:00.766 INFO:tasks.workunit.client.0.smithi046.stdout: create 231/37 file after the snapshot 2023-10-30T13:26:00.766 INFO:tasks.workunit.client.0.smithi046.stdout: create 231/38 file after the snapshot 2023-10-30T13:26:00.766 INFO:tasks.workunit.client.0.smithi046.stdout: create 231/39 file after the snapshot 2023-10-30T13:26:00.767 INFO:tasks.workunit.client.0.smithi046.stdout: create 232/.snap/snap-subdir-test 2023-10-30T13:26:01.731 INFO:tasks.workunit.client.0.smithi046.stdout: create 232/30 file after the snapshot 2023-10-30T13:26:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: create 232/31 file after the snapshot 2023-10-30T13:26:01.732 INFO:tasks.workunit.client.0.smithi046.stdout: create 232/32 file after the snapshot 2023-10-30T13:26:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: create 232/33 file after the snapshot 2023-10-30T13:26:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: create 232/34 file after the snapshot 2023-10-30T13:26:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: create 232/35 file after the snapshot 2023-10-30T13:26:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: create 232/36 file after the snapshot 2023-10-30T13:26:01.733 INFO:tasks.workunit.client.0.smithi046.stdout: create 232/37 file after the snapshot 2023-10-30T13:26:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: create 232/38 file after the snapshot 2023-10-30T13:26:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: create 232/39 file after the snapshot 2023-10-30T13:26:01.734 INFO:tasks.workunit.client.0.smithi046.stdout: create 233/.snap/snap-subdir-test 2023-10-30T13:26:02.487 INFO:tasks.workunit.client.0.smithi046.stdout: create 233/30 file after the snapshot 2023-10-30T13:26:02.488 INFO:tasks.workunit.client.0.smithi046.stdout: create 233/31 file after the snapshot 2023-10-30T13:26:02.489 INFO:tasks.workunit.client.0.smithi046.stdout: create 233/32 file after the snapshot 2023-10-30T13:26:02.489 INFO:tasks.workunit.client.0.smithi046.stdout: create 233/33 file after the snapshot 2023-10-30T13:26:02.489 INFO:tasks.workunit.client.0.smithi046.stdout: create 233/34 file after the snapshot 2023-10-30T13:26:02.489 INFO:tasks.workunit.client.0.smithi046.stdout: create 233/35 file after the snapshot 2023-10-30T13:26:02.489 INFO:tasks.workunit.client.0.smithi046.stdout: create 233/36 file after the snapshot 2023-10-30T13:26:02.490 INFO:tasks.workunit.client.0.smithi046.stdout: create 233/37 file after the snapshot 2023-10-30T13:26:02.490 INFO:tasks.workunit.client.0.smithi046.stdout: create 233/38 file after the snapshot 2023-10-30T13:26:02.490 INFO:tasks.workunit.client.0.smithi046.stdout: create 233/39 file after the snapshot 2023-10-30T13:26:02.490 INFO:tasks.workunit.client.0.smithi046.stdout: create 234/.snap/snap-subdir-test 2023-10-30T13:26:02.777 INFO:tasks.workunit.client.0.smithi046.stdout: create 234/30 file after the snapshot 2023-10-30T13:26:02.779 INFO:tasks.workunit.client.0.smithi046.stdout: create 234/31 file after the snapshot 2023-10-30T13:26:02.779 INFO:tasks.workunit.client.0.smithi046.stdout: create 234/32 file after the snapshot 2023-10-30T13:26:02.779 INFO:tasks.workunit.client.0.smithi046.stdout: create 234/33 file after the snapshot 2023-10-30T13:26:02.779 INFO:tasks.workunit.client.0.smithi046.stdout: create 234/34 file after the snapshot 2023-10-30T13:26:02.780 INFO:tasks.workunit.client.0.smithi046.stdout: create 234/35 file after the snapshot 2023-10-30T13:26:02.780 INFO:tasks.workunit.client.0.smithi046.stdout: create 234/36 file after the snapshot 2023-10-30T13:26:02.780 INFO:tasks.workunit.client.0.smithi046.stdout: create 234/37 file after the snapshot 2023-10-30T13:26:02.780 INFO:tasks.workunit.client.0.smithi046.stdout: create 234/38 file after the snapshot 2023-10-30T13:26:02.780 INFO:tasks.workunit.client.0.smithi046.stdout: create 234/39 file after the snapshot 2023-10-30T13:26:02.781 INFO:tasks.workunit.client.0.smithi046.stdout: create 235/.snap/snap-subdir-test 2023-10-30T13:26:02.820 INFO:tasks.workunit.client.0.smithi046.stdout: create 235/30 file after the snapshot 2023-10-30T13:26:02.821 INFO:tasks.workunit.client.0.smithi046.stdout: create 235/31 file after the snapshot 2023-10-30T13:26:02.821 INFO:tasks.workunit.client.0.smithi046.stdout: create 235/32 file after the snapshot 2023-10-30T13:26:02.821 INFO:tasks.workunit.client.0.smithi046.stdout: create 235/33 file after the snapshot 2023-10-30T13:26:02.822 INFO:tasks.workunit.client.0.smithi046.stdout: create 235/34 file after the snapshot 2023-10-30T13:26:02.822 INFO:tasks.workunit.client.0.smithi046.stdout: create 235/35 file after the snapshot 2023-10-30T13:26:02.823 INFO:tasks.workunit.client.0.smithi046.stdout: create 235/36 file after the snapshot 2023-10-30T13:26:02.823 INFO:tasks.workunit.client.0.smithi046.stdout: create 235/37 file after the snapshot 2023-10-30T13:26:02.823 INFO:tasks.workunit.client.0.smithi046.stdout: create 235/38 file after the snapshot 2023-10-30T13:26:02.823 INFO:tasks.workunit.client.0.smithi046.stdout: create 235/39 file after the snapshot 2023-10-30T13:26:02.823 INFO:tasks.workunit.client.0.smithi046.stdout: create 236/.snap/snap-subdir-test 2023-10-30T13:26:02.862 INFO:tasks.workunit.client.0.smithi046.stdout: create 236/30 file after the snapshot 2023-10-30T13:26:02.863 INFO:tasks.workunit.client.0.smithi046.stdout: create 236/31 file after the snapshot 2023-10-30T13:26:02.863 INFO:tasks.workunit.client.0.smithi046.stdout: create 236/32 file after the snapshot 2023-10-30T13:26:02.863 INFO:tasks.workunit.client.0.smithi046.stdout: create 236/33 file after the snapshot 2023-10-30T13:26:02.863 INFO:tasks.workunit.client.0.smithi046.stdout: create 236/34 file after the snapshot 2023-10-30T13:26:02.864 INFO:tasks.workunit.client.0.smithi046.stdout: create 236/35 file after the snapshot 2023-10-30T13:26:02.864 INFO:tasks.workunit.client.0.smithi046.stdout: create 236/36 file after the snapshot 2023-10-30T13:26:02.865 INFO:tasks.workunit.client.0.smithi046.stdout: create 236/37 file after the snapshot 2023-10-30T13:26:02.865 INFO:tasks.workunit.client.0.smithi046.stdout: create 236/38 file after the snapshot 2023-10-30T13:26:02.865 INFO:tasks.workunit.client.0.smithi046.stdout: create 236/39 file after the snapshot 2023-10-30T13:26:02.865 INFO:tasks.workunit.client.0.smithi046.stdout: create 237/.snap/snap-subdir-test 2023-10-30T13:26:02.906 INFO:tasks.workunit.client.0.smithi046.stdout: create 237/30 file after the snapshot 2023-10-30T13:26:02.908 INFO:tasks.workunit.client.0.smithi046.stdout: create 237/31 file after the snapshot 2023-10-30T13:26:02.908 INFO:tasks.workunit.client.0.smithi046.stdout: create 237/32 file after the snapshot 2023-10-30T13:26:02.908 INFO:tasks.workunit.client.0.smithi046.stdout: create 237/33 file after the snapshot 2023-10-30T13:26:02.908 INFO:tasks.workunit.client.0.smithi046.stdout: create 237/34 file after the snapshot 2023-10-30T13:26:02.908 INFO:tasks.workunit.client.0.smithi046.stdout: create 237/35 file after the snapshot 2023-10-30T13:26:02.909 INFO:tasks.workunit.client.0.smithi046.stdout: create 237/36 file after the snapshot 2023-10-30T13:26:02.909 INFO:tasks.workunit.client.0.smithi046.stdout: create 237/37 file after the snapshot 2023-10-30T13:26:02.909 INFO:tasks.workunit.client.0.smithi046.stdout: create 237/38 file after the snapshot 2023-10-30T13:26:02.909 INFO:tasks.workunit.client.0.smithi046.stdout: create 237/39 file after the snapshot 2023-10-30T13:26:02.910 INFO:tasks.workunit.client.0.smithi046.stdout: create 238/.snap/snap-subdir-test 2023-10-30T13:26:02.953 INFO:tasks.workunit.client.0.smithi046.stdout: create 238/30 file after the snapshot 2023-10-30T13:26:02.954 INFO:tasks.workunit.client.0.smithi046.stdout: create 238/31 file after the snapshot 2023-10-30T13:26:02.954 INFO:tasks.workunit.client.0.smithi046.stdout: create 238/32 file after the snapshot 2023-10-30T13:26:02.955 INFO:tasks.workunit.client.0.smithi046.stdout: create 238/33 file after the snapshot 2023-10-30T13:26:02.955 INFO:tasks.workunit.client.0.smithi046.stdout: create 238/34 file after the snapshot 2023-10-30T13:26:02.955 INFO:tasks.workunit.client.0.smithi046.stdout: create 238/35 file after the snapshot 2023-10-30T13:26:02.955 INFO:tasks.workunit.client.0.smithi046.stdout: create 238/36 file after the snapshot 2023-10-30T13:26:02.956 INFO:tasks.workunit.client.0.smithi046.stdout: create 238/37 file after the snapshot 2023-10-30T13:26:02.956 INFO:tasks.workunit.client.0.smithi046.stdout: create 238/38 file after the snapshot 2023-10-30T13:26:02.956 INFO:tasks.workunit.client.0.smithi046.stdout: create 238/39 file after the snapshot 2023-10-30T13:26:02.956 INFO:tasks.workunit.client.0.smithi046.stdout: create 239/.snap/snap-subdir-test 2023-10-30T13:26:03.446 INFO:tasks.workunit.client.0.smithi046.stdout: create 239/30 file after the snapshot 2023-10-30T13:26:03.448 INFO:tasks.workunit.client.0.smithi046.stdout: create 239/31 file after the snapshot 2023-10-30T13:26:03.448 INFO:tasks.workunit.client.0.smithi046.stdout: create 239/32 file after the snapshot 2023-10-30T13:26:03.448 INFO:tasks.workunit.client.0.smithi046.stdout: create 239/33 file after the snapshot 2023-10-30T13:26:03.448 INFO:tasks.workunit.client.0.smithi046.stdout: create 239/34 file after the snapshot 2023-10-30T13:26:03.449 INFO:tasks.workunit.client.0.smithi046.stdout: create 239/35 file after the snapshot 2023-10-30T13:26:03.449 INFO:tasks.workunit.client.0.smithi046.stdout: create 239/36 file after the snapshot 2023-10-30T13:26:03.449 INFO:tasks.workunit.client.0.smithi046.stdout: create 239/37 file after the snapshot 2023-10-30T13:26:03.449 INFO:tasks.workunit.client.0.smithi046.stdout: create 239/38 file after the snapshot 2023-10-30T13:26:03.449 INFO:tasks.workunit.client.0.smithi046.stdout: create 239/39 file after the snapshot 2023-10-30T13:26:03.449 INFO:tasks.workunit.client.0.smithi046.stdout: create 240/.snap/snap-subdir-test 2023-10-30T13:26:03.498 INFO:tasks.workunit.client.0.smithi046.stdout: create 240/30 file after the snapshot 2023-10-30T13:26:03.499 INFO:tasks.workunit.client.0.smithi046.stdout: create 240/31 file after the snapshot 2023-10-30T13:26:03.499 INFO:tasks.workunit.client.0.smithi046.stdout: create 240/32 file after the snapshot 2023-10-30T13:26:03.500 INFO:tasks.workunit.client.0.smithi046.stdout: create 240/33 file after the snapshot 2023-10-30T13:26:03.500 INFO:tasks.workunit.client.0.smithi046.stdout: create 240/34 file after the snapshot 2023-10-30T13:26:03.500 INFO:tasks.workunit.client.0.smithi046.stdout: create 240/35 file after the snapshot 2023-10-30T13:26:03.500 INFO:tasks.workunit.client.0.smithi046.stdout: create 240/36 file after the snapshot 2023-10-30T13:26:03.500 INFO:tasks.workunit.client.0.smithi046.stdout: create 240/37 file after the snapshot 2023-10-30T13:26:03.501 INFO:tasks.workunit.client.0.smithi046.stdout: create 240/38 file after the snapshot 2023-10-30T13:26:03.501 INFO:tasks.workunit.client.0.smithi046.stdout: create 240/39 file after the snapshot 2023-10-30T13:26:03.501 INFO:tasks.workunit.client.0.smithi046.stdout: create 241/.snap/snap-subdir-test 2023-10-30T13:26:03.532 INFO:tasks.workunit.client.0.smithi046.stdout: create 241/30 file after the snapshot 2023-10-30T13:26:03.533 INFO:tasks.workunit.client.0.smithi046.stdout: create 241/31 file after the snapshot 2023-10-30T13:26:03.534 INFO:tasks.workunit.client.0.smithi046.stdout: create 241/32 file after the snapshot 2023-10-30T13:26:03.534 INFO:tasks.workunit.client.0.smithi046.stdout: create 241/33 file after the snapshot 2023-10-30T13:26:03.534 INFO:tasks.workunit.client.0.smithi046.stdout: create 241/34 file after the snapshot 2023-10-30T13:26:03.534 INFO:tasks.workunit.client.0.smithi046.stdout: create 241/35 file after the snapshot 2023-10-30T13:26:03.534 INFO:tasks.workunit.client.0.smithi046.stdout: create 241/36 file after the snapshot 2023-10-30T13:26:03.535 INFO:tasks.workunit.client.0.smithi046.stdout: create 241/37 file after the snapshot 2023-10-30T13:26:03.535 INFO:tasks.workunit.client.0.smithi046.stdout: create 241/38 file after the snapshot 2023-10-30T13:26:03.535 INFO:tasks.workunit.client.0.smithi046.stdout: create 241/39 file after the snapshot 2023-10-30T13:26:03.535 INFO:tasks.workunit.client.0.smithi046.stdout: create 242/.snap/snap-subdir-test 2023-10-30T13:26:03.564 INFO:tasks.workunit.client.0.smithi046.stdout: create 242/30 file after the snapshot 2023-10-30T13:26:03.565 INFO:tasks.workunit.client.0.smithi046.stdout: create 242/31 file after the snapshot 2023-10-30T13:26:03.566 INFO:tasks.workunit.client.0.smithi046.stdout: create 242/32 file after the snapshot 2023-10-30T13:26:03.566 INFO:tasks.workunit.client.0.smithi046.stdout: create 242/33 file after the snapshot 2023-10-30T13:26:03.566 INFO:tasks.workunit.client.0.smithi046.stdout: create 242/34 file after the snapshot 2023-10-30T13:26:03.566 INFO:tasks.workunit.client.0.smithi046.stdout: create 242/35 file after the snapshot 2023-10-30T13:26:03.567 INFO:tasks.workunit.client.0.smithi046.stdout: create 242/36 file after the snapshot 2023-10-30T13:26:03.567 INFO:tasks.workunit.client.0.smithi046.stdout: create 242/37 file after the snapshot 2023-10-30T13:26:03.567 INFO:tasks.workunit.client.0.smithi046.stdout: create 242/38 file after the snapshot 2023-10-30T13:26:03.567 INFO:tasks.workunit.client.0.smithi046.stdout: create 242/39 file after the snapshot 2023-10-30T13:26:03.567 INFO:tasks.workunit.client.0.smithi046.stdout: create 243/.snap/snap-subdir-test 2023-10-30T13:26:03.597 INFO:tasks.workunit.client.0.smithi046.stdout: create 243/30 file after the snapshot 2023-10-30T13:26:03.598 INFO:tasks.workunit.client.0.smithi046.stdout: create 243/31 file after the snapshot 2023-10-30T13:26:03.598 INFO:tasks.workunit.client.0.smithi046.stdout: create 243/32 file after the snapshot 2023-10-30T13:26:03.599 INFO:tasks.workunit.client.0.smithi046.stdout: create 243/33 file after the snapshot 2023-10-30T13:26:03.599 INFO:tasks.workunit.client.0.smithi046.stdout: create 243/34 file after the snapshot 2023-10-30T13:26:03.599 INFO:tasks.workunit.client.0.smithi046.stdout: create 243/35 file after the snapshot 2023-10-30T13:26:03.599 INFO:tasks.workunit.client.0.smithi046.stdout: create 243/36 file after the snapshot 2023-10-30T13:26:03.599 INFO:tasks.workunit.client.0.smithi046.stdout: create 243/37 file after the snapshot 2023-10-30T13:26:03.600 INFO:tasks.workunit.client.0.smithi046.stdout: create 243/38 file after the snapshot 2023-10-30T13:26:03.600 INFO:tasks.workunit.client.0.smithi046.stdout: create 243/39 file after the snapshot 2023-10-30T13:26:03.600 INFO:tasks.workunit.client.0.smithi046.stdout: create 244/.snap/snap-subdir-test 2023-10-30T13:26:03.635 INFO:tasks.workunit.client.0.smithi046.stdout: create 244/30 file after the snapshot 2023-10-30T13:26:03.636 INFO:tasks.workunit.client.0.smithi046.stdout: create 244/31 file after the snapshot 2023-10-30T13:26:03.636 INFO:tasks.workunit.client.0.smithi046.stdout: create 244/32 file after the snapshot 2023-10-30T13:26:03.636 INFO:tasks.workunit.client.0.smithi046.stdout: create 244/33 file after the snapshot 2023-10-30T13:26:03.637 INFO:tasks.workunit.client.0.smithi046.stdout: create 244/34 file after the snapshot 2023-10-30T13:26:03.637 INFO:tasks.workunit.client.0.smithi046.stdout: create 244/35 file after the snapshot 2023-10-30T13:26:03.638 INFO:tasks.workunit.client.0.smithi046.stdout: create 244/36 file after the snapshot 2023-10-30T13:26:03.638 INFO:tasks.workunit.client.0.smithi046.stdout: create 244/37 file after the snapshot 2023-10-30T13:26:03.638 INFO:tasks.workunit.client.0.smithi046.stdout: create 244/38 file after the snapshot 2023-10-30T13:26:03.638 INFO:tasks.workunit.client.0.smithi046.stdout: create 244/39 file after the snapshot 2023-10-30T13:26:03.639 INFO:tasks.workunit.client.0.smithi046.stdout: create 245/.snap/snap-subdir-test 2023-10-30T13:26:03.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 245/30 file after the snapshot 2023-10-30T13:26:03.669 INFO:tasks.workunit.client.0.smithi046.stdout: create 245/31 file after the snapshot 2023-10-30T13:26:03.670 INFO:tasks.workunit.client.0.smithi046.stdout: create 245/32 file after the snapshot 2023-10-30T13:26:03.670 INFO:tasks.workunit.client.0.smithi046.stdout: create 245/33 file after the snapshot 2023-10-30T13:26:03.670 INFO:tasks.workunit.client.0.smithi046.stdout: create 245/34 file after the snapshot 2023-10-30T13:26:03.671 INFO:tasks.workunit.client.0.smithi046.stdout: create 245/35 file after the snapshot 2023-10-30T13:26:03.672 INFO:tasks.workunit.client.0.smithi046.stdout: create 245/36 file after the snapshot 2023-10-30T13:26:03.672 INFO:tasks.workunit.client.0.smithi046.stdout: create 245/37 file after the snapshot 2023-10-30T13:26:03.672 INFO:tasks.workunit.client.0.smithi046.stdout: create 245/38 file after the snapshot 2023-10-30T13:26:03.672 INFO:tasks.workunit.client.0.smithi046.stdout: create 245/39 file after the snapshot 2023-10-30T13:26:03.673 INFO:tasks.workunit.client.0.smithi046.stdout: create 246/.snap/snap-subdir-test 2023-10-30T13:26:03.698 INFO:tasks.workunit.client.0.smithi046.stdout: create 246/30 file after the snapshot 2023-10-30T13:26:03.701 INFO:tasks.workunit.client.0.smithi046.stdout: create 246/31 file after the snapshot 2023-10-30T13:26:03.701 INFO:tasks.workunit.client.0.smithi046.stdout: create 246/32 file after the snapshot 2023-10-30T13:26:03.701 INFO:tasks.workunit.client.0.smithi046.stdout: create 246/33 file after the snapshot 2023-10-30T13:26:03.701 INFO:tasks.workunit.client.0.smithi046.stdout: create 246/34 file after the snapshot 2023-10-30T13:26:03.702 INFO:tasks.workunit.client.0.smithi046.stdout: create 246/35 file after the snapshot 2023-10-30T13:26:03.702 INFO:tasks.workunit.client.0.smithi046.stdout: create 246/36 file after the snapshot 2023-10-30T13:26:03.702 INFO:tasks.workunit.client.0.smithi046.stdout: create 246/37 file after the snapshot 2023-10-30T13:26:03.702 INFO:tasks.workunit.client.0.smithi046.stdout: create 246/38 file after the snapshot 2023-10-30T13:26:03.702 INFO:tasks.workunit.client.0.smithi046.stdout: create 246/39 file after the snapshot 2023-10-30T13:26:03.703 INFO:tasks.workunit.client.0.smithi046.stdout: create 247/.snap/snap-subdir-test 2023-10-30T13:26:03.739 INFO:tasks.workunit.client.0.smithi046.stdout: create 247/30 file after the snapshot 2023-10-30T13:26:03.740 INFO:tasks.workunit.client.0.smithi046.stdout: create 247/31 file after the snapshot 2023-10-30T13:26:03.741 INFO:tasks.workunit.client.0.smithi046.stdout: create 247/32 file after the snapshot 2023-10-30T13:26:03.741 INFO:tasks.workunit.client.0.smithi046.stdout: create 247/33 file after the snapshot 2023-10-30T13:26:03.741 INFO:tasks.workunit.client.0.smithi046.stdout: create 247/34 file after the snapshot 2023-10-30T13:26:03.741 INFO:tasks.workunit.client.0.smithi046.stdout: create 247/35 file after the snapshot 2023-10-30T13:26:03.741 INFO:tasks.workunit.client.0.smithi046.stdout: create 247/36 file after the snapshot 2023-10-30T13:26:03.741 INFO:tasks.workunit.client.0.smithi046.stdout: create 247/37 file after the snapshot 2023-10-30T13:26:03.742 INFO:tasks.workunit.client.0.smithi046.stdout: create 247/38 file after the snapshot 2023-10-30T13:26:03.742 INFO:tasks.workunit.client.0.smithi046.stdout: create 247/39 file after the snapshot 2023-10-30T13:26:03.742 INFO:tasks.workunit.client.0.smithi046.stdout: create 248/.snap/snap-subdir-test 2023-10-30T13:26:04.039 INFO:tasks.workunit.client.0.smithi046.stdout: create 248/30 file after the snapshot 2023-10-30T13:26:04.040 INFO:tasks.workunit.client.0.smithi046.stdout: create 248/31 file after the snapshot 2023-10-30T13:26:04.040 INFO:tasks.workunit.client.0.smithi046.stdout: create 248/32 file after the snapshot 2023-10-30T13:26:04.041 INFO:tasks.workunit.client.0.smithi046.stdout: create 248/33 file after the snapshot 2023-10-30T13:26:04.041 INFO:tasks.workunit.client.0.smithi046.stdout: create 248/34 file after the snapshot 2023-10-30T13:26:04.041 INFO:tasks.workunit.client.0.smithi046.stdout: create 248/35 file after the snapshot 2023-10-30T13:26:04.041 INFO:tasks.workunit.client.0.smithi046.stdout: create 248/36 file after the snapshot 2023-10-30T13:26:04.041 INFO:tasks.workunit.client.0.smithi046.stdout: create 248/37 file after the snapshot 2023-10-30T13:26:04.042 INFO:tasks.workunit.client.0.smithi046.stdout: create 248/38 file after the snapshot 2023-10-30T13:26:04.042 INFO:tasks.workunit.client.0.smithi046.stdout: create 248/39 file after the snapshot 2023-10-30T13:26:04.042 INFO:tasks.workunit.client.0.smithi046.stdout: create 249/.snap/snap-subdir-test 2023-10-30T13:26:04.077 INFO:tasks.workunit.client.0.smithi046.stdout: create 249/30 file after the snapshot 2023-10-30T13:26:04.079 INFO:tasks.workunit.client.0.smithi046.stdout: create 249/31 file after the snapshot 2023-10-30T13:26:04.079 INFO:tasks.workunit.client.0.smithi046.stdout: create 249/32 file after the snapshot 2023-10-30T13:26:04.079 INFO:tasks.workunit.client.0.smithi046.stdout: create 249/33 file after the snapshot 2023-10-30T13:26:04.079 INFO:tasks.workunit.client.0.smithi046.stdout: create 249/34 file after the snapshot 2023-10-30T13:26:04.080 INFO:tasks.workunit.client.0.smithi046.stdout: create 249/35 file after the snapshot 2023-10-30T13:26:04.080 INFO:tasks.workunit.client.0.smithi046.stdout: create 249/36 file after the snapshot 2023-10-30T13:26:04.080 INFO:tasks.workunit.client.0.smithi046.stdout: create 249/37 file after the snapshot 2023-10-30T13:26:04.080 INFO:tasks.workunit.client.0.smithi046.stdout: create 249/38 file after the snapshot 2023-10-30T13:26:04.080 INFO:tasks.workunit.client.0.smithi046.stdout: create 249/39 file after the snapshot 2023-10-30T13:26:04.080 INFO:tasks.workunit.client.0.smithi046.stdout: create 250/.snap/snap-subdir-test 2023-10-30T13:26:04.374 INFO:tasks.workunit.client.0.smithi046.stdout: create 250/30 file after the snapshot 2023-10-30T13:26:04.375 INFO:tasks.workunit.client.0.smithi046.stdout: create 250/31 file after the snapshot 2023-10-30T13:26:04.375 INFO:tasks.workunit.client.0.smithi046.stdout: create 250/32 file after the snapshot 2023-10-30T13:26:04.375 INFO:tasks.workunit.client.0.smithi046.stdout: create 250/33 file after the snapshot 2023-10-30T13:26:04.376 INFO:tasks.workunit.client.0.smithi046.stdout: create 250/34 file after the snapshot 2023-10-30T13:26:04.376 INFO:tasks.workunit.client.0.smithi046.stdout: create 250/35 file after the snapshot 2023-10-30T13:26:04.377 INFO:tasks.workunit.client.0.smithi046.stdout: create 250/36 file after the snapshot 2023-10-30T13:26:04.377 INFO:tasks.workunit.client.0.smithi046.stdout: create 250/37 file after the snapshot 2023-10-30T13:26:04.377 INFO:tasks.workunit.client.0.smithi046.stdout: create 250/38 file after the snapshot 2023-10-30T13:26:04.377 INFO:tasks.workunit.client.0.smithi046.stdout: create 250/39 file after the snapshot 2023-10-30T13:26:04.377 INFO:tasks.workunit.client.0.smithi046.stdout: create 251/.snap/snap-subdir-test 2023-10-30T13:26:04.408 INFO:tasks.workunit.client.0.smithi046.stdout: create 251/30 file after the snapshot 2023-10-30T13:26:04.409 INFO:tasks.workunit.client.0.smithi046.stdout: create 251/31 file after the snapshot 2023-10-30T13:26:04.409 INFO:tasks.workunit.client.0.smithi046.stdout: create 251/32 file after the snapshot 2023-10-30T13:26:04.410 INFO:tasks.workunit.client.0.smithi046.stdout: create 251/33 file after the snapshot 2023-10-30T13:26:04.410 INFO:tasks.workunit.client.0.smithi046.stdout: create 251/34 file after the snapshot 2023-10-30T13:26:04.410 INFO:tasks.workunit.client.0.smithi046.stdout: create 251/35 file after the snapshot 2023-10-30T13:26:04.410 INFO:tasks.workunit.client.0.smithi046.stdout: create 251/36 file after the snapshot 2023-10-30T13:26:04.410 INFO:tasks.workunit.client.0.smithi046.stdout: create 251/37 file after the snapshot 2023-10-30T13:26:04.411 INFO:tasks.workunit.client.0.smithi046.stdout: create 251/38 file after the snapshot 2023-10-30T13:26:04.411 INFO:tasks.workunit.client.0.smithi046.stdout: create 251/39 file after the snapshot 2023-10-30T13:26:04.411 INFO:tasks.workunit.client.0.smithi046.stdout: create 252/.snap/snap-subdir-test 2023-10-30T13:26:04.702 INFO:tasks.workunit.client.0.smithi046.stdout: create 252/30 file after the snapshot 2023-10-30T13:26:04.704 INFO:tasks.workunit.client.0.smithi046.stdout: create 252/31 file after the snapshot 2023-10-30T13:26:04.704 INFO:tasks.workunit.client.0.smithi046.stdout: create 252/32 file after the snapshot 2023-10-30T13:26:04.705 INFO:tasks.workunit.client.0.smithi046.stdout: create 252/33 file after the snapshot 2023-10-30T13:26:04.705 INFO:tasks.workunit.client.0.smithi046.stdout: create 252/34 file after the snapshot 2023-10-30T13:26:04.705 INFO:tasks.workunit.client.0.smithi046.stdout: create 252/35 file after the snapshot 2023-10-30T13:26:04.705 INFO:tasks.workunit.client.0.smithi046.stdout: create 252/36 file after the snapshot 2023-10-30T13:26:04.705 INFO:tasks.workunit.client.0.smithi046.stdout: create 252/37 file after the snapshot 2023-10-30T13:26:04.706 INFO:tasks.workunit.client.0.smithi046.stdout: create 252/38 file after the snapshot 2023-10-30T13:26:04.706 INFO:tasks.workunit.client.0.smithi046.stdout: create 252/39 file after the snapshot 2023-10-30T13:26:04.706 INFO:tasks.workunit.client.0.smithi046.stdout: create 253/.snap/snap-subdir-test 2023-10-30T13:26:04.736 INFO:tasks.workunit.client.0.smithi046.stdout: create 253/30 file after the snapshot 2023-10-30T13:26:04.738 INFO:tasks.workunit.client.0.smithi046.stdout: create 253/31 file after the snapshot 2023-10-30T13:26:04.738 INFO:tasks.workunit.client.0.smithi046.stdout: create 253/32 file after the snapshot 2023-10-30T13:26:04.739 INFO:tasks.workunit.client.0.smithi046.stdout: create 253/33 file after the snapshot 2023-10-30T13:26:04.739 INFO:tasks.workunit.client.0.smithi046.stdout: create 253/34 file after the snapshot 2023-10-30T13:26:04.739 INFO:tasks.workunit.client.0.smithi046.stdout: create 253/35 file after the snapshot 2023-10-30T13:26:04.739 INFO:tasks.workunit.client.0.smithi046.stdout: create 253/36 file after the snapshot 2023-10-30T13:26:04.739 INFO:tasks.workunit.client.0.smithi046.stdout: create 253/37 file after the snapshot 2023-10-30T13:26:04.740 INFO:tasks.workunit.client.0.smithi046.stdout: create 253/38 file after the snapshot 2023-10-30T13:26:04.740 INFO:tasks.workunit.client.0.smithi046.stdout: create 253/39 file after the snapshot 2023-10-30T13:26:04.740 INFO:tasks.workunit.client.0.smithi046.stdout: create 254/.snap/snap-subdir-test 2023-10-30T13:26:04.770 INFO:tasks.workunit.client.0.smithi046.stdout: create 254/30 file after the snapshot 2023-10-30T13:26:04.771 INFO:tasks.workunit.client.0.smithi046.stdout: create 254/31 file after the snapshot 2023-10-30T13:26:04.772 INFO:tasks.workunit.client.0.smithi046.stdout: create 254/32 file after the snapshot 2023-10-30T13:26:04.772 INFO:tasks.workunit.client.0.smithi046.stdout: create 254/33 file after the snapshot 2023-10-30T13:26:04.772 INFO:tasks.workunit.client.0.smithi046.stdout: create 254/34 file after the snapshot 2023-10-30T13:26:04.772 INFO:tasks.workunit.client.0.smithi046.stdout: create 254/35 file after the snapshot 2023-10-30T13:26:04.773 INFO:tasks.workunit.client.0.smithi046.stdout: create 254/36 file after the snapshot 2023-10-30T13:26:04.773 INFO:tasks.workunit.client.0.smithi046.stdout: create 254/37 file after the snapshot 2023-10-30T13:26:04.773 INFO:tasks.workunit.client.0.smithi046.stdout: create 254/38 file after the snapshot 2023-10-30T13:26:04.773 INFO:tasks.workunit.client.0.smithi046.stdout: create 254/39 file after the snapshot 2023-10-30T13:26:04.773 INFO:tasks.workunit.client.0.smithi046.stdout: create 255/.snap/snap-subdir-test 2023-10-30T13:26:04.804 INFO:tasks.workunit.client.0.smithi046.stdout: create 255/30 file after the snapshot 2023-10-30T13:26:04.806 INFO:tasks.workunit.client.0.smithi046.stdout: create 255/31 file after the snapshot 2023-10-30T13:26:04.807 INFO:tasks.workunit.client.0.smithi046.stdout: create 255/32 file after the snapshot 2023-10-30T13:26:04.807 INFO:tasks.workunit.client.0.smithi046.stdout: create 255/33 file after the snapshot 2023-10-30T13:26:04.807 INFO:tasks.workunit.client.0.smithi046.stdout: create 255/34 file after the snapshot 2023-10-30T13:26:04.807 INFO:tasks.workunit.client.0.smithi046.stdout: create 255/35 file after the snapshot 2023-10-30T13:26:04.807 INFO:tasks.workunit.client.0.smithi046.stdout: create 255/36 file after the snapshot 2023-10-30T13:26:04.808 INFO:tasks.workunit.client.0.smithi046.stdout: create 255/37 file after the snapshot 2023-10-30T13:26:04.808 INFO:tasks.workunit.client.0.smithi046.stdout: create 255/38 file after the snapshot 2023-10-30T13:26:04.808 INFO:tasks.workunit.client.0.smithi046.stdout: create 255/39 file after the snapshot 2023-10-30T13:26:04.808 INFO:tasks.workunit.client.0.smithi046.stdout: create 256/.snap/snap-subdir-test 2023-10-30T13:26:04.843 INFO:tasks.workunit.client.0.smithi046.stdout: create 256/30 file after the snapshot 2023-10-30T13:26:04.845 INFO:tasks.workunit.client.0.smithi046.stdout: create 256/31 file after the snapshot 2023-10-30T13:26:04.845 INFO:tasks.workunit.client.0.smithi046.stdout: create 256/32 file after the snapshot 2023-10-30T13:26:04.845 INFO:tasks.workunit.client.0.smithi046.stdout: create 256/33 file after the snapshot 2023-10-30T13:26:04.845 INFO:tasks.workunit.client.0.smithi046.stdout: create 256/34 file after the snapshot 2023-10-30T13:26:04.846 INFO:tasks.workunit.client.0.smithi046.stdout: create 256/35 file after the snapshot 2023-10-30T13:26:04.846 INFO:tasks.workunit.client.0.smithi046.stdout: create 256/36 file after the snapshot 2023-10-30T13:26:04.846 INFO:tasks.workunit.client.0.smithi046.stdout: create 256/37 file after the snapshot 2023-10-30T13:26:04.846 INFO:tasks.workunit.client.0.smithi046.stdout: create 256/38 file after the snapshot 2023-10-30T13:26:04.846 INFO:tasks.workunit.client.0.smithi046.stdout: create 256/39 file after the snapshot 2023-10-30T13:26:04.846 INFO:tasks.workunit.client.0.smithi046.stdout: create 257/.snap/snap-subdir-test 2023-10-30T13:26:04.881 INFO:tasks.workunit.client.0.smithi046.stdout: create 257/30 file after the snapshot 2023-10-30T13:26:04.883 INFO:tasks.workunit.client.0.smithi046.stdout: create 257/31 file after the snapshot 2023-10-30T13:26:04.883 INFO:tasks.workunit.client.0.smithi046.stdout: create 257/32 file after the snapshot 2023-10-30T13:26:04.883 INFO:tasks.workunit.client.0.smithi046.stdout: create 257/33 file after the snapshot 2023-10-30T13:26:04.883 INFO:tasks.workunit.client.0.smithi046.stdout: create 257/34 file after the snapshot 2023-10-30T13:26:04.883 INFO:tasks.workunit.client.0.smithi046.stdout: create 257/35 file after the snapshot 2023-10-30T13:26:04.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 257/36 file after the snapshot 2023-10-30T13:26:04.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 257/37 file after the snapshot 2023-10-30T13:26:04.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 257/38 file after the snapshot 2023-10-30T13:26:04.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 257/39 file after the snapshot 2023-10-30T13:26:04.884 INFO:tasks.workunit.client.0.smithi046.stdout: create 258/.snap/snap-subdir-test 2023-10-30T13:26:04.915 INFO:tasks.workunit.client.0.smithi046.stdout: create 258/30 file after the snapshot 2023-10-30T13:26:04.917 INFO:tasks.workunit.client.0.smithi046.stdout: create 258/31 file after the snapshot 2023-10-30T13:26:04.917 INFO:tasks.workunit.client.0.smithi046.stdout: create 258/32 file after the snapshot 2023-10-30T13:26:04.917 INFO:tasks.workunit.client.0.smithi046.stdout: create 258/33 file after the snapshot 2023-10-30T13:26:04.917 INFO:tasks.workunit.client.0.smithi046.stdout: create 258/34 file after the snapshot 2023-10-30T13:26:04.917 INFO:tasks.workunit.client.0.smithi046.stdout: create 258/35 file after the snapshot 2023-10-30T13:26:04.918 INFO:tasks.workunit.client.0.smithi046.stdout: create 258/36 file after the snapshot 2023-10-30T13:26:04.918 INFO:tasks.workunit.client.0.smithi046.stdout: create 258/37 file after the snapshot 2023-10-30T13:26:04.918 INFO:tasks.workunit.client.0.smithi046.stdout: create 258/38 file after the snapshot 2023-10-30T13:26:04.918 INFO:tasks.workunit.client.0.smithi046.stdout: create 258/39 file after the snapshot 2023-10-30T13:26:04.918 INFO:tasks.workunit.client.0.smithi046.stdout: create 259/.snap/snap-subdir-test 2023-10-30T13:26:05.791 INFO:tasks.workunit.client.0.smithi046.stdout: create 259/30 file after the snapshot 2023-10-30T13:26:05.793 INFO:tasks.workunit.client.0.smithi046.stdout: create 259/31 file after the snapshot 2023-10-30T13:26:05.794 INFO:tasks.workunit.client.0.smithi046.stdout: create 259/32 file after the snapshot 2023-10-30T13:26:05.794 INFO:tasks.workunit.client.0.smithi046.stdout: create 259/33 file after the snapshot 2023-10-30T13:26:05.794 INFO:tasks.workunit.client.0.smithi046.stdout: create 259/34 file after the snapshot 2023-10-30T13:26:05.794 INFO:tasks.workunit.client.0.smithi046.stdout: create 259/35 file after the snapshot 2023-10-30T13:26:05.794 INFO:tasks.workunit.client.0.smithi046.stdout: create 259/36 file after the snapshot 2023-10-30T13:26:05.795 INFO:tasks.workunit.client.0.smithi046.stdout: create 259/37 file after the snapshot 2023-10-30T13:26:05.795 INFO:tasks.workunit.client.0.smithi046.stdout: create 259/38 file after the snapshot 2023-10-30T13:26:05.795 INFO:tasks.workunit.client.0.smithi046.stdout: create 259/39 file after the snapshot 2023-10-30T13:26:05.795 INFO:tasks.workunit.client.0.smithi046.stdout: create 260/.snap/snap-subdir-test 2023-10-30T13:26:05.828 INFO:tasks.workunit.client.0.smithi046.stdout: create 260/30 file after the snapshot 2023-10-30T13:26:05.830 INFO:tasks.workunit.client.0.smithi046.stdout: create 260/31 file after the snapshot 2023-10-30T13:26:05.831 INFO:tasks.workunit.client.0.smithi046.stdout: create 260/32 file after the snapshot 2023-10-30T13:26:05.831 INFO:tasks.workunit.client.0.smithi046.stdout: create 260/33 file after the snapshot 2023-10-30T13:26:05.831 INFO:tasks.workunit.client.0.smithi046.stdout: create 260/34 file after the snapshot 2023-10-30T13:26:05.831 INFO:tasks.workunit.client.0.smithi046.stdout: create 260/35 file after the snapshot 2023-10-30T13:26:05.831 INFO:tasks.workunit.client.0.smithi046.stdout: create 260/36 file after the snapshot 2023-10-30T13:26:05.832 INFO:tasks.workunit.client.0.smithi046.stdout: create 260/37 file after the snapshot 2023-10-30T13:26:05.832 INFO:tasks.workunit.client.0.smithi046.stdout: create 260/38 file after the snapshot 2023-10-30T13:26:05.832 INFO:tasks.workunit.client.0.smithi046.stdout: create 260/39 file after the snapshot 2023-10-30T13:26:05.832 INFO:tasks.workunit.client.0.smithi046.stdout: create 261/.snap/snap-subdir-test 2023-10-30T13:26:05.857 INFO:tasks.workunit.client.0.smithi046.stdout: create 261/30 file after the snapshot 2023-10-30T13:26:05.859 INFO:tasks.workunit.client.0.smithi046.stdout: create 261/31 file after the snapshot 2023-10-30T13:26:05.859 INFO:tasks.workunit.client.0.smithi046.stdout: create 261/32 file after the snapshot 2023-10-30T13:26:05.859 INFO:tasks.workunit.client.0.smithi046.stdout: create 261/33 file after the snapshot 2023-10-30T13:26:05.859 INFO:tasks.workunit.client.0.smithi046.stdout: create 261/34 file after the snapshot 2023-10-30T13:26:05.860 INFO:tasks.workunit.client.0.smithi046.stdout: create 261/35 file after the snapshot 2023-10-30T13:26:05.860 INFO:tasks.workunit.client.0.smithi046.stdout: create 261/36 file after the snapshot 2023-10-30T13:26:05.860 INFO:tasks.workunit.client.0.smithi046.stdout: create 261/37 file after the snapshot 2023-10-30T13:26:05.860 INFO:tasks.workunit.client.0.smithi046.stdout: create 261/38 file after the snapshot 2023-10-30T13:26:05.861 INFO:tasks.workunit.client.0.smithi046.stdout: create 261/39 file after the snapshot 2023-10-30T13:26:05.861 INFO:tasks.workunit.client.0.smithi046.stdout: create 262/.snap/snap-subdir-test 2023-10-30T13:26:05.886 INFO:tasks.workunit.client.0.smithi046.stdout: create 262/30 file after the snapshot 2023-10-30T13:26:05.888 INFO:tasks.workunit.client.0.smithi046.stdout: create 262/31 file after the snapshot 2023-10-30T13:26:05.888 INFO:tasks.workunit.client.0.smithi046.stdout: create 262/32 file after the snapshot 2023-10-30T13:26:05.888 INFO:tasks.workunit.client.0.smithi046.stdout: create 262/33 file after the snapshot 2023-10-30T13:26:05.889 INFO:tasks.workunit.client.0.smithi046.stdout: create 262/34 file after the snapshot 2023-10-30T13:26:05.889 INFO:tasks.workunit.client.0.smithi046.stdout: create 262/35 file after the snapshot 2023-10-30T13:26:05.889 INFO:tasks.workunit.client.0.smithi046.stdout: create 262/36 file after the snapshot 2023-10-30T13:26:05.889 INFO:tasks.workunit.client.0.smithi046.stdout: create 262/37 file after the snapshot 2023-10-30T13:26:05.890 INFO:tasks.workunit.client.0.smithi046.stdout: create 262/38 file after the snapshot 2023-10-30T13:26:05.890 INFO:tasks.workunit.client.0.smithi046.stdout: create 262/39 file after the snapshot 2023-10-30T13:26:05.890 INFO:tasks.workunit.client.0.smithi046.stdout: create 263/.snap/snap-subdir-test 2023-10-30T13:26:06.728 INFO:tasks.workunit.client.0.smithi046.stdout: create 263/30 file after the snapshot 2023-10-30T13:26:06.784 INFO:tasks.workunit.client.0.smithi046.stdout: create 263/31 file after the snapshot 2023-10-30T13:26:06.785 INFO:tasks.workunit.client.0.smithi046.stdout: create 263/32 file after the snapshot 2023-10-30T13:26:06.785 INFO:tasks.workunit.client.0.smithi046.stdout: create 263/33 file after the snapshot 2023-10-30T13:26:06.785 INFO:tasks.workunit.client.0.smithi046.stdout: create 263/34 file after the snapshot 2023-10-30T13:26:06.785 INFO:tasks.workunit.client.0.smithi046.stdout: create 263/35 file after the snapshot 2023-10-30T13:26:06.786 INFO:tasks.workunit.client.0.smithi046.stdout: create 263/36 file after the snapshot 2023-10-30T13:26:06.786 INFO:tasks.workunit.client.0.smithi046.stdout: create 263/37 file after the snapshot 2023-10-30T13:26:06.786 INFO:tasks.workunit.client.0.smithi046.stdout: create 263/38 file after the snapshot 2023-10-30T13:26:06.786 INFO:tasks.workunit.client.0.smithi046.stdout: create 263/39 file after the snapshot 2023-10-30T13:26:06.787 INFO:tasks.workunit.client.0.smithi046.stdout: create 264/.snap/snap-subdir-test 2023-10-30T13:26:06.787 INFO:tasks.workunit.client.0.smithi046.stdout: create 264/30 file after the snapshot 2023-10-30T13:26:06.787 INFO:tasks.workunit.client.0.smithi046.stdout: create 264/31 file after the snapshot 2023-10-30T13:26:06.787 INFO:tasks.workunit.client.0.smithi046.stdout: create 264/32 file after the snapshot 2023-10-30T13:26:06.787 INFO:tasks.workunit.client.0.smithi046.stdout: create 264/33 file after the snapshot 2023-10-30T13:26:06.788 INFO:tasks.workunit.client.0.smithi046.stdout: create 264/34 file after the snapshot 2023-10-30T13:26:06.788 INFO:tasks.workunit.client.0.smithi046.stdout: create 264/35 file after the snapshot 2023-10-30T13:26:06.788 INFO:tasks.workunit.client.0.smithi046.stdout: create 264/36 file after the snapshot 2023-10-30T13:26:06.788 INFO:tasks.workunit.client.0.smithi046.stdout: create 264/37 file after the snapshot 2023-10-30T13:26:06.789 INFO:tasks.workunit.client.0.smithi046.stdout: create 264/38 file after the snapshot 2023-10-30T13:26:06.789 INFO:tasks.workunit.client.0.smithi046.stdout: create 264/39 file after the snapshot 2023-10-30T13:26:06.789 INFO:tasks.workunit.client.0.smithi046.stdout: create 265/.snap/snap-subdir-test 2023-10-30T13:26:06.797 INFO:tasks.workunit.client.0.smithi046.stdout: create 265/30 file after the snapshot 2023-10-30T13:26:06.799 INFO:tasks.workunit.client.0.smithi046.stdout: create 265/31 file after the snapshot 2023-10-30T13:26:06.799 INFO:tasks.workunit.client.0.smithi046.stdout: create 265/32 file after the snapshot 2023-10-30T13:26:06.799 INFO:tasks.workunit.client.0.smithi046.stdout: create 265/33 file after the snapshot 2023-10-30T13:26:06.799 INFO:tasks.workunit.client.0.smithi046.stdout: create 265/34 file after the snapshot 2023-10-30T13:26:06.799 INFO:tasks.workunit.client.0.smithi046.stdout: create 265/35 file after the snapshot 2023-10-30T13:26:06.800 INFO:tasks.workunit.client.0.smithi046.stdout: create 265/36 file after the snapshot 2023-10-30T13:26:06.800 INFO:tasks.workunit.client.0.smithi046.stdout: create 265/37 file after the snapshot 2023-10-30T13:26:06.800 INFO:tasks.workunit.client.0.smithi046.stdout: create 265/38 file after the snapshot 2023-10-30T13:26:06.800 INFO:tasks.workunit.client.0.smithi046.stdout: create 265/39 file after the snapshot 2023-10-30T13:26:06.801 INFO:tasks.workunit.client.0.smithi046.stdout: create 266/.snap/snap-subdir-test 2023-10-30T13:26:06.827 INFO:tasks.workunit.client.0.smithi046.stdout: create 266/30 file after the snapshot 2023-10-30T13:26:06.829 INFO:tasks.workunit.client.0.smithi046.stdout: create 266/31 file after the snapshot 2023-10-30T13:26:06.829 INFO:tasks.workunit.client.0.smithi046.stdout: create 266/32 file after the snapshot 2023-10-30T13:26:06.830 INFO:tasks.workunit.client.0.smithi046.stdout: create 266/33 file after the snapshot 2023-10-30T13:26:06.830 INFO:tasks.workunit.client.0.smithi046.stdout: create 266/34 file after the snapshot 2023-10-30T13:26:06.830 INFO:tasks.workunit.client.0.smithi046.stdout: create 266/35 file after the snapshot 2023-10-30T13:26:06.830 INFO:tasks.workunit.client.0.smithi046.stdout: create 266/36 file after the snapshot 2023-10-30T13:26:06.830 INFO:tasks.workunit.client.0.smithi046.stdout: create 266/37 file after the snapshot 2023-10-30T13:26:06.831 INFO:tasks.workunit.client.0.smithi046.stdout: create 266/38 file after the snapshot 2023-10-30T13:26:06.831 INFO:tasks.workunit.client.0.smithi046.stdout: create 266/39 file after the snapshot 2023-10-30T13:26:06.831 INFO:tasks.workunit.client.0.smithi046.stdout: create 267/.snap/snap-subdir-test 2023-10-30T13:26:06.859 INFO:tasks.workunit.client.0.smithi046.stdout: create 267/30 file after the snapshot 2023-10-30T13:26:06.861 INFO:tasks.workunit.client.0.smithi046.stdout: create 267/31 file after the snapshot 2023-10-30T13:26:06.861 INFO:tasks.workunit.client.0.smithi046.stdout: create 267/32 file after the snapshot 2023-10-30T13:26:06.861 INFO:tasks.workunit.client.0.smithi046.stdout: create 267/33 file after the snapshot 2023-10-30T13:26:06.861 INFO:tasks.workunit.client.0.smithi046.stdout: create 267/34 file after the snapshot 2023-10-30T13:26:06.861 INFO:tasks.workunit.client.0.smithi046.stdout: create 267/35 file after the snapshot 2023-10-30T13:26:06.862 INFO:tasks.workunit.client.0.smithi046.stdout: create 267/36 file after the snapshot 2023-10-30T13:26:06.862 INFO:tasks.workunit.client.0.smithi046.stdout: create 267/37 file after the snapshot 2023-10-30T13:26:06.862 INFO:tasks.workunit.client.0.smithi046.stdout: create 267/38 file after the snapshot 2023-10-30T13:26:06.862 INFO:tasks.workunit.client.0.smithi046.stdout: create 267/39 file after the snapshot 2023-10-30T13:26:06.862 INFO:tasks.workunit.client.0.smithi046.stdout: create 268/.snap/snap-subdir-test 2023-10-30T13:26:06.892 INFO:tasks.workunit.client.0.smithi046.stdout: create 268/30 file after the snapshot 2023-10-30T13:26:06.893 INFO:tasks.workunit.client.0.smithi046.stdout: create 268/31 file after the snapshot 2023-10-30T13:26:06.894 INFO:tasks.workunit.client.0.smithi046.stdout: create 268/32 file after the snapshot 2023-10-30T13:26:06.894 INFO:tasks.workunit.client.0.smithi046.stdout: create 268/33 file after the snapshot 2023-10-30T13:26:06.894 INFO:tasks.workunit.client.0.smithi046.stdout: create 268/34 file after the snapshot 2023-10-30T13:26:06.894 INFO:tasks.workunit.client.0.smithi046.stdout: create 268/35 file after the snapshot 2023-10-30T13:26:06.894 INFO:tasks.workunit.client.0.smithi046.stdout: create 268/36 file after the snapshot 2023-10-30T13:26:06.895 INFO:tasks.workunit.client.0.smithi046.stdout: create 268/37 file after the snapshot 2023-10-30T13:26:06.895 INFO:tasks.workunit.client.0.smithi046.stdout: create 268/38 file after the snapshot 2023-10-30T13:26:06.895 INFO:tasks.workunit.client.0.smithi046.stdout: create 268/39 file after the snapshot 2023-10-30T13:26:06.895 INFO:tasks.workunit.client.0.smithi046.stdout: create 269/.snap/snap-subdir-test 2023-10-30T13:26:06.924 INFO:tasks.workunit.client.0.smithi046.stdout: create 269/30 file after the snapshot 2023-10-30T13:26:06.926 INFO:tasks.workunit.client.0.smithi046.stdout: create 269/31 file after the snapshot 2023-10-30T13:26:06.926 INFO:tasks.workunit.client.0.smithi046.stdout: create 269/32 file after the snapshot 2023-10-30T13:26:06.926 INFO:tasks.workunit.client.0.smithi046.stdout: create 269/33 file after the snapshot 2023-10-30T13:26:06.927 INFO:tasks.workunit.client.0.smithi046.stdout: create 269/34 file after the snapshot 2023-10-30T13:26:06.927 INFO:tasks.workunit.client.0.smithi046.stdout: create 269/35 file after the snapshot 2023-10-30T13:26:06.927 INFO:tasks.workunit.client.0.smithi046.stdout: create 269/36 file after the snapshot 2023-10-30T13:26:06.927 INFO:tasks.workunit.client.0.smithi046.stdout: create 269/37 file after the snapshot 2023-10-30T13:26:06.928 INFO:tasks.workunit.client.0.smithi046.stdout: create 269/38 file after the snapshot 2023-10-30T13:26:06.928 INFO:tasks.workunit.client.0.smithi046.stdout: create 269/39 file after the snapshot 2023-10-30T13:26:06.928 INFO:tasks.workunit.client.0.smithi046.stdout: create 270/.snap/snap-subdir-test 2023-10-30T13:26:07.422 INFO:tasks.workunit.client.0.smithi046.stdout: create 270/30 file after the snapshot 2023-10-30T13:26:07.423 INFO:tasks.workunit.client.0.smithi046.stdout: create 270/31 file after the snapshot 2023-10-30T13:26:07.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 270/32 file after the snapshot 2023-10-30T13:26:07.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 270/33 file after the snapshot 2023-10-30T13:26:07.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 270/34 file after the snapshot 2023-10-30T13:26:07.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 270/35 file after the snapshot 2023-10-30T13:26:07.424 INFO:tasks.workunit.client.0.smithi046.stdout: create 270/36 file after the snapshot 2023-10-30T13:26:07.425 INFO:tasks.workunit.client.0.smithi046.stdout: create 270/37 file after the snapshot 2023-10-30T13:26:07.425 INFO:tasks.workunit.client.0.smithi046.stdout: create 270/38 file after the snapshot 2023-10-30T13:26:07.425 INFO:tasks.workunit.client.0.smithi046.stdout: create 270/39 file after the snapshot 2023-10-30T13:26:07.425 INFO:tasks.workunit.client.0.smithi046.stdout: create 271/.snap/snap-subdir-test 2023-10-30T13:26:07.455 INFO:tasks.workunit.client.0.smithi046.stdout: create 271/30 file after the snapshot 2023-10-30T13:26:07.457 INFO:tasks.workunit.client.0.smithi046.stdout: create 271/31 file after the snapshot 2023-10-30T13:26:07.457 INFO:tasks.workunit.client.0.smithi046.stdout: create 271/32 file after the snapshot 2023-10-30T13:26:07.458 INFO:tasks.workunit.client.0.smithi046.stdout: create 271/33 file after the snapshot 2023-10-30T13:26:07.458 INFO:tasks.workunit.client.0.smithi046.stdout: create 271/34 file after the snapshot 2023-10-30T13:26:07.458 INFO:tasks.workunit.client.0.smithi046.stdout: create 271/35 file after the snapshot 2023-10-30T13:26:07.458 INFO:tasks.workunit.client.0.smithi046.stdout: create 271/36 file after the snapshot 2023-10-30T13:26:07.458 INFO:tasks.workunit.client.0.smithi046.stdout: create 271/37 file after the snapshot 2023-10-30T13:26:07.459 INFO:tasks.workunit.client.0.smithi046.stdout: create 271/38 file after the snapshot 2023-10-30T13:26:07.459 INFO:tasks.workunit.client.0.smithi046.stdout: create 271/39 file after the snapshot 2023-10-30T13:26:07.459 INFO:tasks.workunit.client.0.smithi046.stdout: create 272/.snap/snap-subdir-test 2023-10-30T13:26:08.464 INFO:tasks.workunit.client.0.smithi046.stdout: create 272/30 file after the snapshot 2023-10-30T13:26:08.466 INFO:tasks.workunit.client.0.smithi046.stdout: create 272/31 file after the snapshot 2023-10-30T13:26:08.466 INFO:tasks.workunit.client.0.smithi046.stdout: create 272/32 file after the snapshot 2023-10-30T13:26:08.466 INFO:tasks.workunit.client.0.smithi046.stdout: create 272/33 file after the snapshot 2023-10-30T13:26:08.466 INFO:tasks.workunit.client.0.smithi046.stdout: create 272/34 file after the snapshot 2023-10-30T13:26:08.466 INFO:tasks.workunit.client.0.smithi046.stdout: create 272/35 file after the snapshot 2023-10-30T13:26:08.467 INFO:tasks.workunit.client.0.smithi046.stdout: create 272/36 file after the snapshot 2023-10-30T13:26:08.467 INFO:tasks.workunit.client.0.smithi046.stdout: create 272/37 file after the snapshot 2023-10-30T13:26:08.467 INFO:tasks.workunit.client.0.smithi046.stdout: create 272/38 file after the snapshot 2023-10-30T13:26:08.468 INFO:tasks.workunit.client.0.smithi046.stdout: create 272/39 file after the snapshot 2023-10-30T13:26:08.468 INFO:tasks.workunit.client.0.smithi046.stdout: create 273/.snap/snap-subdir-test 2023-10-30T13:26:08.500 INFO:tasks.workunit.client.0.smithi046.stdout: create 273/30 file after the snapshot 2023-10-30T13:26:08.502 INFO:tasks.workunit.client.0.smithi046.stdout: create 273/31 file after the snapshot 2023-10-30T13:26:08.502 INFO:tasks.workunit.client.0.smithi046.stdout: create 273/32 file after the snapshot 2023-10-30T13:26:08.502 INFO:tasks.workunit.client.0.smithi046.stdout: create 273/33 file after the snapshot 2023-10-30T13:26:08.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 273/34 file after the snapshot 2023-10-30T13:26:08.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 273/35 file after the snapshot 2023-10-30T13:26:08.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 273/36 file after the snapshot 2023-10-30T13:26:08.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 273/37 file after the snapshot 2023-10-30T13:26:08.503 INFO:tasks.workunit.client.0.smithi046.stdout: create 273/38 file after the snapshot 2023-10-30T13:26:08.504 INFO:tasks.workunit.client.0.smithi046.stdout: create 273/39 file after the snapshot 2023-10-30T13:26:08.504 INFO:tasks.workunit.client.0.smithi046.stdout: create 274/.snap/snap-subdir-test 2023-10-30T13:26:08.529 INFO:tasks.workunit.client.0.smithi046.stdout: create 274/30 file after the snapshot 2023-10-30T13:26:08.531 INFO:tasks.workunit.client.0.smithi046.stdout: create 274/31 file after the snapshot 2023-10-30T13:26:08.531 INFO:tasks.workunit.client.0.smithi046.stdout: create 274/32 file after the snapshot 2023-10-30T13:26:08.531 INFO:tasks.workunit.client.0.smithi046.stdout: create 274/33 file after the snapshot 2023-10-30T13:26:08.532 INFO:tasks.workunit.client.0.smithi046.stdout: create 274/34 file after the snapshot 2023-10-30T13:26:08.532 INFO:tasks.workunit.client.0.smithi046.stdout: create 274/35 file after the snapshot 2023-10-30T13:26:08.532 INFO:tasks.workunit.client.0.smithi046.stdout: create 274/36 file after the snapshot 2023-10-30T13:26:08.532 INFO:tasks.workunit.client.0.smithi046.stdout: create 274/37 file after the snapshot 2023-10-30T13:26:08.532 INFO:tasks.workunit.client.0.smithi046.stdout: create 274/38 file after the snapshot 2023-10-30T13:26:08.532 INFO:tasks.workunit.client.0.smithi046.stdout: create 274/39 file after the snapshot 2023-10-30T13:26:08.533 INFO:tasks.workunit.client.0.smithi046.stdout: create 275/.snap/snap-subdir-test 2023-10-30T13:26:08.559 INFO:tasks.workunit.client.0.smithi046.stdout: create 275/30 file after the snapshot 2023-10-30T13:26:08.561 INFO:tasks.workunit.client.0.smithi046.stdout: create 275/31 file after the snapshot 2023-10-30T13:26:08.561 INFO:tasks.workunit.client.0.smithi046.stdout: create 275/32 file after the snapshot 2023-10-30T13:26:08.561 INFO:tasks.workunit.client.0.smithi046.stdout: create 275/33 file after the snapshot 2023-10-30T13:26:08.561 INFO:tasks.workunit.client.0.smithi046.stdout: create 275/34 file after the snapshot 2023-10-30T13:26:08.561 INFO:tasks.workunit.client.0.smithi046.stdout: create 275/35 file after the snapshot 2023-10-30T13:26:08.562 INFO:tasks.workunit.client.0.smithi046.stdout: create 275/36 file after the snapshot 2023-10-30T13:26:08.562 INFO:tasks.workunit.client.0.smithi046.stdout: create 275/37 file after the snapshot 2023-10-30T13:26:08.562 INFO:tasks.workunit.client.0.smithi046.stdout: create 275/38 file after the snapshot 2023-10-30T13:26:08.562 INFO:tasks.workunit.client.0.smithi046.stdout: create 275/39 file after the snapshot 2023-10-30T13:26:08.562 INFO:tasks.workunit.client.0.smithi046.stdout: create 276/.snap/snap-subdir-test 2023-10-30T13:26:08.929 INFO:tasks.workunit.client.0.smithi046.stdout: create 276/30 file after the snapshot 2023-10-30T13:26:08.931 INFO:tasks.workunit.client.0.smithi046.stdout: create 276/31 file after the snapshot 2023-10-30T13:26:08.931 INFO:tasks.workunit.client.0.smithi046.stdout: create 276/32 file after the snapshot 2023-10-30T13:26:08.931 INFO:tasks.workunit.client.0.smithi046.stdout: create 276/33 file after the snapshot 2023-10-30T13:26:08.931 INFO:tasks.workunit.client.0.smithi046.stdout: create 276/34 file after the snapshot 2023-10-30T13:26:08.932 INFO:tasks.workunit.client.0.smithi046.stdout: create 276/35 file after the snapshot 2023-10-30T13:26:08.932 INFO:tasks.workunit.client.0.smithi046.stdout: create 276/36 file after the snapshot 2023-10-30T13:26:08.932 INFO:tasks.workunit.client.0.smithi046.stdout: create 276/37 file after the snapshot 2023-10-30T13:26:08.932 INFO:tasks.workunit.client.0.smithi046.stdout: create 276/38 file after the snapshot 2023-10-30T13:26:08.932 INFO:tasks.workunit.client.0.smithi046.stdout: create 276/39 file after the snapshot 2023-10-30T13:26:08.933 INFO:tasks.workunit.client.0.smithi046.stdout: create 277/.snap/snap-subdir-test 2023-10-30T13:26:08.963 INFO:tasks.workunit.client.0.smithi046.stdout: create 277/30 file after the snapshot 2023-10-30T13:26:08.965 INFO:tasks.workunit.client.0.smithi046.stdout: create 277/31 file after the snapshot 2023-10-30T13:26:08.965 INFO:tasks.workunit.client.0.smithi046.stdout: create 277/32 file after the snapshot 2023-10-30T13:26:08.965 INFO:tasks.workunit.client.0.smithi046.stdout: create 277/33 file after the snapshot 2023-10-30T13:26:08.966 INFO:tasks.workunit.client.0.smithi046.stdout: create 277/34 file after the snapshot 2023-10-30T13:26:08.966 INFO:tasks.workunit.client.0.smithi046.stdout: create 277/35 file after the snapshot 2023-10-30T13:26:08.966 INFO:tasks.workunit.client.0.smithi046.stdout: create 277/36 file after the snapshot 2023-10-30T13:26:08.966 INFO:tasks.workunit.client.0.smithi046.stdout: create 277/37 file after the snapshot 2023-10-30T13:26:08.967 INFO:tasks.workunit.client.0.smithi046.stdout: create 277/38 file after the snapshot 2023-10-30T13:26:08.967 INFO:tasks.workunit.client.0.smithi046.stdout: create 277/39 file after the snapshot 2023-10-30T13:26:08.967 INFO:tasks.workunit.client.0.smithi046.stdout: create 278/.snap/snap-subdir-test 2023-10-30T13:26:08.999 INFO:tasks.workunit.client.0.smithi046.stdout: create 278/30 file after the snapshot 2023-10-30T13:26:09.001 INFO:tasks.workunit.client.0.smithi046.stdout: create 278/31 file after the snapshot 2023-10-30T13:26:09.001 INFO:tasks.workunit.client.0.smithi046.stdout: create 278/32 file after the snapshot 2023-10-30T13:26:09.001 INFO:tasks.workunit.client.0.smithi046.stdout: create 278/33 file after the snapshot 2023-10-30T13:26:09.002 INFO:tasks.workunit.client.0.smithi046.stdout: create 278/34 file after the snapshot 2023-10-30T13:26:09.002 INFO:tasks.workunit.client.0.smithi046.stdout: create 278/35 file after the snapshot 2023-10-30T13:26:09.002 INFO:tasks.workunit.client.0.smithi046.stdout: create 278/36 file after the snapshot 2023-10-30T13:26:09.002 INFO:tasks.workunit.client.0.smithi046.stdout: create 278/37 file after the snapshot 2023-10-30T13:26:09.002 INFO:tasks.workunit.client.0.smithi046.stdout: create 278/38 file after the snapshot 2023-10-30T13:26:09.003 INFO:tasks.workunit.client.0.smithi046.stdout: create 278/39 file after the snapshot 2023-10-30T13:26:09.003 INFO:tasks.workunit.client.0.smithi046.stdout: create 279/.snap/snap-subdir-test 2023-10-30T13:26:09.026 INFO:tasks.workunit.client.0.smithi046.stdout: create 279/30 file after the snapshot 2023-10-30T13:26:09.027 INFO:tasks.workunit.client.0.smithi046.stdout: create 279/31 file after the snapshot 2023-10-30T13:26:09.027 INFO:tasks.workunit.client.0.smithi046.stdout: create 279/32 file after the snapshot 2023-10-30T13:26:09.028 INFO:tasks.workunit.client.0.smithi046.stdout: create 279/33 file after the snapshot 2023-10-30T13:26:09.028 INFO:tasks.workunit.client.0.smithi046.stdout: create 279/34 file after the snapshot 2023-10-30T13:26:09.028 INFO:tasks.workunit.client.0.smithi046.stdout: create 279/35 file after the snapshot 2023-10-30T13:26:09.029 INFO:tasks.workunit.client.0.smithi046.stdout: create 279/36 file after the snapshot 2023-10-30T13:26:09.029 INFO:tasks.workunit.client.0.smithi046.stdout: create 279/37 file after the snapshot 2023-10-30T13:26:09.029 INFO:tasks.workunit.client.0.smithi046.stdout: create 279/38 file after the snapshot 2023-10-30T13:26:09.029 INFO:tasks.workunit.client.0.smithi046.stdout: create 279/39 file after the snapshot 2023-10-30T13:26:09.029 INFO:tasks.workunit.client.0.smithi046.stdout: create 280/.snap/snap-subdir-test 2023-10-30T13:26:09.056 INFO:tasks.workunit.client.0.smithi046.stdout: create 280/30 file after the snapshot 2023-10-30T13:26:09.057 INFO:tasks.workunit.client.0.smithi046.stdout: create 280/31 file after the snapshot 2023-10-30T13:26:09.057 INFO:tasks.workunit.client.0.smithi046.stdout: create 280/32 file after the snapshot 2023-10-30T13:26:09.058 INFO:tasks.workunit.client.0.smithi046.stdout: create 280/33 file after the snapshot 2023-10-30T13:26:09.058 INFO:tasks.workunit.client.0.smithi046.stdout: create 280/34 file after the snapshot 2023-10-30T13:26:09.058 INFO:tasks.workunit.client.0.smithi046.stdout: create 280/35 file after the snapshot 2023-10-30T13:26:09.058 INFO:tasks.workunit.client.0.smithi046.stdout: create 280/36 file after the snapshot 2023-10-30T13:26:09.059 INFO:tasks.workunit.client.0.smithi046.stdout: create 280/37 file after the snapshot 2023-10-30T13:26:09.059 INFO:tasks.workunit.client.0.smithi046.stdout: create 280/38 file after the snapshot 2023-10-30T13:26:09.059 INFO:tasks.workunit.client.0.smithi046.stdout: create 280/39 file after the snapshot 2023-10-30T13:26:09.059 INFO:tasks.workunit.client.0.smithi046.stdout: create 281/.snap/snap-subdir-test 2023-10-30T13:26:09.087 INFO:tasks.workunit.client.0.smithi046.stdout: create 281/30 file after the snapshot 2023-10-30T13:26:09.088 INFO:tasks.workunit.client.0.smithi046.stdout: create 281/31 file after the snapshot 2023-10-30T13:26:09.088 INFO:tasks.workunit.client.0.smithi046.stdout: create 281/32 file after the snapshot 2023-10-30T13:26:09.088 INFO:tasks.workunit.client.0.smithi046.stdout: create 281/33 file after the snapshot 2023-10-30T13:26:09.089 INFO:tasks.workunit.client.0.smithi046.stdout: create 281/34 file after the snapshot 2023-10-30T13:26:09.089 INFO:tasks.workunit.client.0.smithi046.stdout: create 281/35 file after the snapshot 2023-10-30T13:26:09.089 INFO:tasks.workunit.client.0.smithi046.stdout: create 281/36 file after the snapshot 2023-10-30T13:26:09.089 INFO:tasks.workunit.client.0.smithi046.stdout: create 281/37 file after the snapshot 2023-10-30T13:26:09.089 INFO:tasks.workunit.client.0.smithi046.stdout: create 281/38 file after the snapshot 2023-10-30T13:26:09.090 INFO:tasks.workunit.client.0.smithi046.stdout: create 281/39 file after the snapshot 2023-10-30T13:26:09.090 INFO:tasks.workunit.client.0.smithi046.stdout: create 282/.snap/snap-subdir-test 2023-10-30T13:26:09.118 INFO:tasks.workunit.client.0.smithi046.stdout: create 282/30 file after the snapshot 2023-10-30T13:26:09.120 INFO:tasks.workunit.client.0.smithi046.stdout: create 282/31 file after the snapshot 2023-10-30T13:26:09.120 INFO:tasks.workunit.client.0.smithi046.stdout: create 282/32 file after the snapshot 2023-10-30T13:26:09.120 INFO:tasks.workunit.client.0.smithi046.stdout: create 282/33 file after the snapshot 2023-10-30T13:26:09.120 INFO:tasks.workunit.client.0.smithi046.stdout: create 282/34 file after the snapshot 2023-10-30T13:26:09.121 INFO:tasks.workunit.client.0.smithi046.stdout: create 282/35 file after the snapshot 2023-10-30T13:26:09.121 INFO:tasks.workunit.client.0.smithi046.stdout: create 282/36 file after the snapshot 2023-10-30T13:26:09.121 INFO:tasks.workunit.client.0.smithi046.stdout: create 282/37 file after the snapshot 2023-10-30T13:26:09.121 INFO:tasks.workunit.client.0.smithi046.stdout: create 282/38 file after the snapshot 2023-10-30T13:26:09.121 INFO:tasks.workunit.client.0.smithi046.stdout: create 282/39 file after the snapshot 2023-10-30T13:26:09.122 INFO:tasks.workunit.client.0.smithi046.stdout: create 283/.snap/snap-subdir-test 2023-10-30T13:26:09.404 INFO:tasks.workunit.client.0.smithi046.stdout: create 283/30 file after the snapshot 2023-10-30T13:26:09.405 INFO:tasks.workunit.client.0.smithi046.stdout: create 283/31 file after the snapshot 2023-10-30T13:26:09.406 INFO:tasks.workunit.client.0.smithi046.stdout: create 283/32 file after the snapshot 2023-10-30T13:26:09.406 INFO:tasks.workunit.client.0.smithi046.stdout: create 283/33 file after the snapshot 2023-10-30T13:26:09.406 INFO:tasks.workunit.client.0.smithi046.stdout: create 283/34 file after the snapshot 2023-10-30T13:26:09.406 INFO:tasks.workunit.client.0.smithi046.stdout: create 283/35 file after the snapshot 2023-10-30T13:26:09.406 INFO:tasks.workunit.client.0.smithi046.stdout: create 283/36 file after the snapshot 2023-10-30T13:26:09.407 INFO:tasks.workunit.client.0.smithi046.stdout: create 283/37 file after the snapshot 2023-10-30T13:26:09.407 INFO:tasks.workunit.client.0.smithi046.stdout: create 283/38 file after the snapshot 2023-10-30T13:26:09.407 INFO:tasks.workunit.client.0.smithi046.stdout: create 283/39 file after the snapshot 2023-10-30T13:26:09.407 INFO:tasks.workunit.client.0.smithi046.stdout: create 284/.snap/snap-subdir-test 2023-10-30T13:26:09.433 INFO:tasks.workunit.client.0.smithi046.stdout: create 284/30 file after the snapshot 2023-10-30T13:26:09.435 INFO:tasks.workunit.client.0.smithi046.stdout: create 284/31 file after the snapshot 2023-10-30T13:26:09.435 INFO:tasks.workunit.client.0.smithi046.stdout: create 284/32 file after the snapshot 2023-10-30T13:26:09.436 INFO:tasks.workunit.client.0.smithi046.stdout: create 284/33 file after the snapshot 2023-10-30T13:26:09.436 INFO:tasks.workunit.client.0.smithi046.stdout: create 284/34 file after the snapshot 2023-10-30T13:26:09.436 INFO:tasks.workunit.client.0.smithi046.stdout: create 284/35 file after the snapshot 2023-10-30T13:26:09.437 INFO:tasks.workunit.client.0.smithi046.stdout: create 284/36 file after the snapshot 2023-10-30T13:26:09.437 INFO:tasks.workunit.client.0.smithi046.stdout: create 284/37 file after the snapshot 2023-10-30T13:26:09.437 INFO:tasks.workunit.client.0.smithi046.stdout: create 284/38 file after the snapshot 2023-10-30T13:26:09.438 INFO:tasks.workunit.client.0.smithi046.stdout: create 284/39 file after the snapshot 2023-10-30T13:26:09.438 INFO:tasks.workunit.client.0.smithi046.stdout: create 285/.snap/snap-subdir-test 2023-10-30T13:26:09.461 INFO:tasks.workunit.client.0.smithi046.stdout: create 285/30 file after the snapshot 2023-10-30T13:26:09.462 INFO:tasks.workunit.client.0.smithi046.stdout: create 285/31 file after the snapshot 2023-10-30T13:26:09.462 INFO:tasks.workunit.client.0.smithi046.stdout: create 285/32 file after the snapshot 2023-10-30T13:26:09.463 INFO:tasks.workunit.client.0.smithi046.stdout: create 285/33 file after the snapshot 2023-10-30T13:26:09.463 INFO:tasks.workunit.client.0.smithi046.stdout: create 285/34 file after the snapshot 2023-10-30T13:26:09.463 INFO:tasks.workunit.client.0.smithi046.stdout: create 285/35 file after the snapshot 2023-10-30T13:26:09.463 INFO:tasks.workunit.client.0.smithi046.stdout: create 285/36 file after the snapshot 2023-10-30T13:26:09.463 INFO:tasks.workunit.client.0.smithi046.stdout: create 285/37 file after the snapshot 2023-10-30T13:26:09.464 INFO:tasks.workunit.client.0.smithi046.stdout: create 285/38 file after the snapshot 2023-10-30T13:26:09.464 INFO:tasks.workunit.client.0.smithi046.stdout: create 285/39 file after the snapshot 2023-10-30T13:26:09.464 INFO:tasks.workunit.client.0.smithi046.stdout: create 286/.snap/snap-subdir-test 2023-10-30T13:26:09.489 INFO:tasks.workunit.client.0.smithi046.stdout: create 286/30 file after the snapshot 2023-10-30T13:26:09.491 INFO:tasks.workunit.client.0.smithi046.stdout: create 286/31 file after the snapshot 2023-10-30T13:26:09.491 INFO:tasks.workunit.client.0.smithi046.stdout: create 286/32 file after the snapshot 2023-10-30T13:26:09.491 INFO:tasks.workunit.client.0.smithi046.stdout: create 286/33 file after the snapshot 2023-10-30T13:26:09.492 INFO:tasks.workunit.client.0.smithi046.stdout: create 286/34 file after the snapshot 2023-10-30T13:26:09.492 INFO:tasks.workunit.client.0.smithi046.stdout: create 286/35 file after the snapshot 2023-10-30T13:26:09.492 INFO:tasks.workunit.client.0.smithi046.stdout: create 286/36 file after the snapshot 2023-10-30T13:26:09.492 INFO:tasks.workunit.client.0.smithi046.stdout: create 286/37 file after the snapshot 2023-10-30T13:26:09.492 INFO:tasks.workunit.client.0.smithi046.stdout: create 286/38 file after the snapshot 2023-10-30T13:26:09.492 INFO:tasks.workunit.client.0.smithi046.stdout: create 286/39 file after the snapshot 2023-10-30T13:26:09.493 INFO:tasks.workunit.client.0.smithi046.stdout: create 287/.snap/snap-subdir-test 2023-10-30T13:26:09.517 INFO:tasks.workunit.client.0.smithi046.stdout: create 287/30 file after the snapshot 2023-10-30T13:26:09.519 INFO:tasks.workunit.client.0.smithi046.stdout: create 287/31 file after the snapshot 2023-10-30T13:26:09.519 INFO:tasks.workunit.client.0.smithi046.stdout: create 287/32 file after the snapshot 2023-10-30T13:26:09.519 INFO:tasks.workunit.client.0.smithi046.stdout: create 287/33 file after the snapshot 2023-10-30T13:26:09.520 INFO:tasks.workunit.client.0.smithi046.stdout: create 287/34 file after the snapshot 2023-10-30T13:26:09.520 INFO:tasks.workunit.client.0.smithi046.stdout: create 287/35 file after the snapshot 2023-10-30T13:26:09.520 INFO:tasks.workunit.client.0.smithi046.stdout: create 287/36 file after the snapshot 2023-10-30T13:26:09.520 INFO:tasks.workunit.client.0.smithi046.stdout: create 287/37 file after the snapshot 2023-10-30T13:26:09.520 INFO:tasks.workunit.client.0.smithi046.stdout: create 287/38 file after the snapshot 2023-10-30T13:26:09.521 INFO:tasks.workunit.client.0.smithi046.stdout: create 287/39 file after the snapshot 2023-10-30T13:26:09.521 INFO:tasks.workunit.client.0.smithi046.stdout: create 288/.snap/snap-subdir-test 2023-10-30T13:26:09.548 INFO:tasks.workunit.client.0.smithi046.stdout: create 288/30 file after the snapshot 2023-10-30T13:26:09.550 INFO:tasks.workunit.client.0.smithi046.stdout: create 288/31 file after the snapshot 2023-10-30T13:26:09.551 INFO:tasks.workunit.client.0.smithi046.stdout: create 288/32 file after the snapshot 2023-10-30T13:26:09.551 INFO:tasks.workunit.client.0.smithi046.stdout: create 288/33 file after the snapshot 2023-10-30T13:26:09.551 INFO:tasks.workunit.client.0.smithi046.stdout: create 288/34 file after the snapshot 2023-10-30T13:26:09.551 INFO:tasks.workunit.client.0.smithi046.stdout: create 288/35 file after the snapshot 2023-10-30T13:26:09.551 INFO:tasks.workunit.client.0.smithi046.stdout: create 288/36 file after the snapshot 2023-10-30T13:26:09.552 INFO:tasks.workunit.client.0.smithi046.stdout: create 288/37 file after the snapshot 2023-10-30T13:26:09.552 INFO:tasks.workunit.client.0.smithi046.stdout: create 288/38 file after the snapshot 2023-10-30T13:26:09.552 INFO:tasks.workunit.client.0.smithi046.stdout: create 288/39 file after the snapshot 2023-10-30T13:26:09.552 INFO:tasks.workunit.client.0.smithi046.stdout: create 289/.snap/snap-subdir-test 2023-10-30T13:26:09.578 INFO:tasks.workunit.client.0.smithi046.stdout: create 289/30 file after the snapshot 2023-10-30T13:26:09.579 INFO:tasks.workunit.client.0.smithi046.stdout: create 289/31 file after the snapshot 2023-10-30T13:26:09.580 INFO:tasks.workunit.client.0.smithi046.stdout: create 289/32 file after the snapshot 2023-10-30T13:26:09.580 INFO:tasks.workunit.client.0.smithi046.stdout: create 289/33 file after the snapshot 2023-10-30T13:26:09.580 INFO:tasks.workunit.client.0.smithi046.stdout: create 289/34 file after the snapshot 2023-10-30T13:26:09.580 INFO:tasks.workunit.client.0.smithi046.stdout: create 289/35 file after the snapshot 2023-10-30T13:26:09.580 INFO:tasks.workunit.client.0.smithi046.stdout: create 289/36 file after the snapshot 2023-10-30T13:26:09.581 INFO:tasks.workunit.client.0.smithi046.stdout: create 289/37 file after the snapshot 2023-10-30T13:26:09.581 INFO:tasks.workunit.client.0.smithi046.stdout: create 289/38 file after the snapshot 2023-10-30T13:26:09.581 INFO:tasks.workunit.client.0.smithi046.stdout: create 289/39 file after the snapshot 2023-10-30T13:26:09.581 INFO:tasks.workunit.client.0.smithi046.stdout: create 290/.snap/snap-subdir-test 2023-10-30T13:26:09.607 INFO:tasks.workunit.client.0.smithi046.stdout: create 290/30 file after the snapshot 2023-10-30T13:26:09.608 INFO:tasks.workunit.client.0.smithi046.stdout: create 290/31 file after the snapshot 2023-10-30T13:26:09.609 INFO:tasks.workunit.client.0.smithi046.stdout: create 290/32 file after the snapshot 2023-10-30T13:26:09.609 INFO:tasks.workunit.client.0.smithi046.stdout: create 290/33 file after the snapshot 2023-10-30T13:26:09.609 INFO:tasks.workunit.client.0.smithi046.stdout: create 290/34 file after the snapshot 2023-10-30T13:26:09.609 INFO:tasks.workunit.client.0.smithi046.stdout: create 290/35 file after the snapshot 2023-10-30T13:26:09.609 INFO:tasks.workunit.client.0.smithi046.stdout: create 290/36 file after the snapshot 2023-10-30T13:26:09.609 INFO:tasks.workunit.client.0.smithi046.stdout: create 290/37 file after the snapshot 2023-10-30T13:26:09.610 INFO:tasks.workunit.client.0.smithi046.stdout: create 290/38 file after the snapshot 2023-10-30T13:26:09.610 INFO:tasks.workunit.client.0.smithi046.stdout: create 290/39 file after the snapshot 2023-10-30T13:26:09.610 INFO:tasks.workunit.client.0.smithi046.stdout: create 291/.snap/snap-subdir-test 2023-10-30T13:26:09.636 INFO:tasks.workunit.client.0.smithi046.stdout: create 291/30 file after the snapshot 2023-10-30T13:26:09.637 INFO:tasks.workunit.client.0.smithi046.stdout: create 291/31 file after the snapshot 2023-10-30T13:26:09.638 INFO:tasks.workunit.client.0.smithi046.stdout: create 291/32 file after the snapshot 2023-10-30T13:26:09.638 INFO:tasks.workunit.client.0.smithi046.stdout: create 291/33 file after the snapshot 2023-10-30T13:26:09.638 INFO:tasks.workunit.client.0.smithi046.stdout: create 291/34 file after the snapshot 2023-10-30T13:26:09.638 INFO:tasks.workunit.client.0.smithi046.stdout: create 291/35 file after the snapshot 2023-10-30T13:26:09.638 INFO:tasks.workunit.client.0.smithi046.stdout: create 291/36 file after the snapshot 2023-10-30T13:26:09.639 INFO:tasks.workunit.client.0.smithi046.stdout: create 291/37 file after the snapshot 2023-10-30T13:26:09.639 INFO:tasks.workunit.client.0.smithi046.stdout: create 291/38 file after the snapshot 2023-10-30T13:26:09.639 INFO:tasks.workunit.client.0.smithi046.stdout: create 291/39 file after the snapshot 2023-10-30T13:26:09.639 INFO:tasks.workunit.client.0.smithi046.stdout: create 292/.snap/snap-subdir-test 2023-10-30T13:26:09.665 INFO:tasks.workunit.client.0.smithi046.stdout: create 292/30 file after the snapshot 2023-10-30T13:26:09.667 INFO:tasks.workunit.client.0.smithi046.stdout: create 292/31 file after the snapshot 2023-10-30T13:26:09.667 INFO:tasks.workunit.client.0.smithi046.stdout: create 292/32 file after the snapshot 2023-10-30T13:26:09.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 292/33 file after the snapshot 2023-10-30T13:26:09.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 292/34 file after the snapshot 2023-10-30T13:26:09.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 292/35 file after the snapshot 2023-10-30T13:26:09.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 292/36 file after the snapshot 2023-10-30T13:26:09.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 292/37 file after the snapshot 2023-10-30T13:26:09.668 INFO:tasks.workunit.client.0.smithi046.stdout: create 292/38 file after the snapshot 2023-10-30T13:26:09.669 INFO:tasks.workunit.client.0.smithi046.stdout: create 292/39 file after the snapshot 2023-10-30T13:26:09.669 INFO:tasks.workunit.client.0.smithi046.stdout: create 293/.snap/snap-subdir-test 2023-10-30T13:26:09.693 INFO:tasks.workunit.client.0.smithi046.stdout: create 293/30 file after the snapshot 2023-10-30T13:26:09.695 INFO:tasks.workunit.client.0.smithi046.stdout: create 293/31 file after the snapshot 2023-10-30T13:26:09.696 INFO:tasks.workunit.client.0.smithi046.stdout: create 293/32 file after the snapshot 2023-10-30T13:26:09.696 INFO:tasks.workunit.client.0.smithi046.stdout: create 293/33 file after the snapshot 2023-10-30T13:26:09.696 INFO:tasks.workunit.client.0.smithi046.stdout: create 293/34 file after the snapshot 2023-10-30T13:26:09.696 INFO:tasks.workunit.client.0.smithi046.stdout: create 293/35 file after the snapshot 2023-10-30T13:26:09.696 INFO:tasks.workunit.client.0.smithi046.stdout: create 293/36 file after the snapshot 2023-10-30T13:26:09.697 INFO:tasks.workunit.client.0.smithi046.stdout: create 293/37 file after the snapshot 2023-10-30T13:26:09.697 INFO:tasks.workunit.client.0.smithi046.stdout: create 293/38 file after the snapshot 2023-10-30T13:26:09.697 INFO:tasks.workunit.client.0.smithi046.stdout: create 293/39 file after the snapshot 2023-10-30T13:26:09.697 INFO:tasks.workunit.client.0.smithi046.stdout: create 294/.snap/snap-subdir-test 2023-10-30T13:26:09.727 INFO:tasks.workunit.client.0.smithi046.stdout: create 294/30 file after the snapshot 2023-10-30T13:26:09.729 INFO:tasks.workunit.client.0.smithi046.stdout: create 294/31 file after the snapshot 2023-10-30T13:26:09.729 INFO:tasks.workunit.client.0.smithi046.stdout: create 294/32 file after the snapshot 2023-10-30T13:26:09.729 INFO:tasks.workunit.client.0.smithi046.stdout: create 294/33 file after the snapshot 2023-10-30T13:26:09.730 INFO:tasks.workunit.client.0.smithi046.stdout: create 294/34 file after the snapshot 2023-10-30T13:26:09.730 INFO:tasks.workunit.client.0.smithi046.stdout: create 294/35 file after the snapshot 2023-10-30T13:26:09.730 INFO:tasks.workunit.client.0.smithi046.stdout: create 294/36 file after the snapshot 2023-10-30T13:26:09.730 INFO:tasks.workunit.client.0.smithi046.stdout: create 294/37 file after the snapshot 2023-10-30T13:26:09.730 INFO:tasks.workunit.client.0.smithi046.stdout: create 294/38 file after the snapshot 2023-10-30T13:26:09.731 INFO:tasks.workunit.client.0.smithi046.stdout: create 294/39 file after the snapshot 2023-10-30T13:26:09.731 INFO:tasks.workunit.client.0.smithi046.stdout: create 295/.snap/snap-subdir-test 2023-10-30T13:26:10.007 INFO:tasks.workunit.client.0.smithi046.stdout: create 295/30 file after the snapshot 2023-10-30T13:26:10.008 INFO:tasks.workunit.client.0.smithi046.stdout: create 295/31 file after the snapshot 2023-10-30T13:26:10.009 INFO:tasks.workunit.client.0.smithi046.stdout: create 295/32 file after the snapshot 2023-10-30T13:26:10.009 INFO:tasks.workunit.client.0.smithi046.stdout: create 295/33 file after the snapshot 2023-10-30T13:26:10.009 INFO:tasks.workunit.client.0.smithi046.stdout: create 295/34 file after the snapshot 2023-10-30T13:26:10.009 INFO:tasks.workunit.client.0.smithi046.stdout: create 295/35 file after the snapshot 2023-10-30T13:26:10.009 INFO:tasks.workunit.client.0.smithi046.stdout: create 295/36 file after the snapshot 2023-10-30T13:26:10.010 INFO:tasks.workunit.client.0.smithi046.stdout: create 295/37 file after the snapshot 2023-10-30T13:26:10.010 INFO:tasks.workunit.client.0.smithi046.stdout: create 295/38 file after the snapshot 2023-10-30T13:26:10.010 INFO:tasks.workunit.client.0.smithi046.stdout: create 295/39 file after the snapshot 2023-10-30T13:26:10.010 INFO:tasks.workunit.client.0.smithi046.stdout: create 296/.snap/snap-subdir-test 2023-10-30T13:26:10.038 INFO:tasks.workunit.client.0.smithi046.stdout: create 296/30 file after the snapshot 2023-10-30T13:26:10.040 INFO:tasks.workunit.client.0.smithi046.stdout: create 296/31 file after the snapshot 2023-10-30T13:26:10.040 INFO:tasks.workunit.client.0.smithi046.stdout: create 296/32 file after the snapshot 2023-10-30T13:26:10.040 INFO:tasks.workunit.client.0.smithi046.stdout: create 296/33 file after the snapshot 2023-10-30T13:26:10.040 INFO:tasks.workunit.client.0.smithi046.stdout: create 296/34 file after the snapshot 2023-10-30T13:26:10.041 INFO:tasks.workunit.client.0.smithi046.stdout: create 296/35 file after the snapshot 2023-10-30T13:26:10.041 INFO:tasks.workunit.client.0.smithi046.stdout: create 296/36 file after the snapshot 2023-10-30T13:26:10.041 INFO:tasks.workunit.client.0.smithi046.stdout: create 296/37 file after the snapshot 2023-10-30T13:26:10.041 INFO:tasks.workunit.client.0.smithi046.stdout: create 296/38 file after the snapshot 2023-10-30T13:26:10.041 INFO:tasks.workunit.client.0.smithi046.stdout: create 296/39 file after the snapshot 2023-10-30T13:26:10.041 INFO:tasks.workunit.client.0.smithi046.stdout: create 297/.snap/snap-subdir-test 2023-10-30T13:26:10.066 INFO:tasks.workunit.client.0.smithi046.stdout: create 297/30 file after the snapshot 2023-10-30T13:26:10.068 INFO:tasks.workunit.client.0.smithi046.stdout: create 297/31 file after the snapshot 2023-10-30T13:26:10.068 INFO:tasks.workunit.client.0.smithi046.stdout: create 297/32 file after the snapshot 2023-10-30T13:26:10.069 INFO:tasks.workunit.client.0.smithi046.stdout: create 297/33 file after the snapshot 2023-10-30T13:26:10.069 INFO:tasks.workunit.client.0.smithi046.stdout: create 297/34 file after the snapshot 2023-10-30T13:26:10.069 INFO:tasks.workunit.client.0.smithi046.stdout: create 297/35 file after the snapshot 2023-10-30T13:26:10.069 INFO:tasks.workunit.client.0.smithi046.stdout: create 297/36 file after the snapshot 2023-10-30T13:26:10.069 INFO:tasks.workunit.client.0.smithi046.stdout: create 297/37 file after the snapshot 2023-10-30T13:26:10.070 INFO:tasks.workunit.client.0.smithi046.stdout: create 297/38 file after the snapshot 2023-10-30T13:26:10.070 INFO:tasks.workunit.client.0.smithi046.stdout: create 297/39 file after the snapshot 2023-10-30T13:26:10.070 INFO:tasks.workunit.client.0.smithi046.stdout: create 298/.snap/snap-subdir-test 2023-10-30T13:26:10.096 INFO:tasks.workunit.client.0.smithi046.stdout: create 298/30 file after the snapshot 2023-10-30T13:26:10.098 INFO:tasks.workunit.client.0.smithi046.stdout: create 298/31 file after the snapshot 2023-10-30T13:26:10.098 INFO:tasks.workunit.client.0.smithi046.stdout: create 298/32 file after the snapshot 2023-10-30T13:26:10.098 INFO:tasks.workunit.client.0.smithi046.stdout: create 298/33 file after the snapshot 2023-10-30T13:26:10.099 INFO:tasks.workunit.client.0.smithi046.stdout: create 298/34 file after the snapshot 2023-10-30T13:26:10.099 INFO:tasks.workunit.client.0.smithi046.stdout: create 298/35 file after the snapshot 2023-10-30T13:26:10.099 INFO:tasks.workunit.client.0.smithi046.stdout: create 298/36 file after the snapshot 2023-10-30T13:26:10.099 INFO:tasks.workunit.client.0.smithi046.stdout: create 298/37 file after the snapshot 2023-10-30T13:26:10.099 INFO:tasks.workunit.client.0.smithi046.stdout: create 298/38 file after the snapshot 2023-10-30T13:26:10.099 INFO:tasks.workunit.client.0.smithi046.stdout: create 298/39 file after the snapshot 2023-10-30T13:26:10.100 INFO:tasks.workunit.client.0.smithi046.stdout: create 299/.snap/snap-subdir-test 2023-10-30T13:26:10.123 INFO:tasks.workunit.client.0.smithi046.stdout: create 299/30 file after the snapshot 2023-10-30T13:26:10.125 INFO:tasks.workunit.client.0.smithi046.stdout: create 299/31 file after the snapshot 2023-10-30T13:26:10.125 INFO:tasks.workunit.client.0.smithi046.stdout: create 299/32 file after the snapshot 2023-10-30T13:26:10.126 INFO:tasks.workunit.client.0.smithi046.stdout: create 299/33 file after the snapshot 2023-10-30T13:26:10.126 INFO:tasks.workunit.client.0.smithi046.stdout: create 299/34 file after the snapshot 2023-10-30T13:26:10.126 INFO:tasks.workunit.client.0.smithi046.stdout: create 299/35 file after the snapshot 2023-10-30T13:26:10.126 INFO:tasks.workunit.client.0.smithi046.stdout: create 299/36 file after the snapshot 2023-10-30T13:26:10.126 INFO:tasks.workunit.client.0.smithi046.stdout: create 299/37 file after the snapshot 2023-10-30T13:26:10.127 INFO:tasks.workunit.client.0.smithi046.stdout: create 299/38 file after the snapshot 2023-10-30T13:26:10.127 INFO:tasks.workunit.client.0.smithi046.stdout: create 299/39 file after the snapshot 2023-10-30T13:26:10.127 INFO:tasks.workunit.client.0.smithi046.stdout:Take second snapshot .snap/test2 2023-10-30T13:26:10.443 INFO:tasks.workunit.client.0.smithi046.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2023-10-30T13:26:12.111 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2023-10-30T13:26:12.112 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2023-10-30T13:26:12.112 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2023-10-30T13:26:12.112 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2023-10-30T13:26:12.112 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2023-10-30T13:26:12.112 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2023-10-30T13:26:12.112 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2023-10-30T13:26:12.113 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2023-10-30T13:26:12.113 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2023-10-30T13:26:12.113 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2023-10-30T13:26:12.113 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2023-10-30T13:26:12.113 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2023-10-30T13:26:12.114 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2023-10-30T13:26:12.114 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2023-10-30T13:26:12.114 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2023-10-30T13:26:12.114 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2023-10-30T13:26:12.114 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2023-10-30T13:26:12.114 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2023-10-30T13:26:12.115 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2023-10-30T13:26:12.115 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2023-10-30T13:26:12.115 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2023-10-30T13:26:12.115 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2023-10-30T13:26:12.115 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2023-10-30T13:26:12.116 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2023-10-30T13:26:12.116 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2023-10-30T13:26:12.116 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2023-10-30T13:26:12.116 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2023-10-30T13:26:12.116 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2023-10-30T13:26:12.116 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2023-10-30T13:26:12.117 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2023-10-30T13:26:12.117 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2023-10-30T13:26:12.117 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2023-10-30T13:26:12.117 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2023-10-30T13:26:12.117 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2023-10-30T13:26:12.117 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2023-10-30T13:26:12.118 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2023-10-30T13:26:12.118 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2023-10-30T13:26:12.118 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2023-10-30T13:26:12.118 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2023-10-30T13:26:12.118 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2023-10-30T13:26:12.119 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2023-10-30T13:26:12.119 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2023-10-30T13:26:12.119 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2023-10-30T13:26:12.119 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2023-10-30T13:26:12.119 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2023-10-30T13:26:12.120 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2023-10-30T13:26:12.120 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2023-10-30T13:26:12.120 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2023-10-30T13:26:12.120 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2023-10-30T13:26:12.120 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2023-10-30T13:26:12.121 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2023-10-30T13:26:12.121 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2023-10-30T13:26:12.121 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2023-10-30T13:26:12.121 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2023-10-30T13:26:12.121 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2023-10-30T13:26:12.121 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2023-10-30T13:26:12.122 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2023-10-30T13:26:12.122 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2023-10-30T13:26:12.122 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2023-10-30T13:26:12.122 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2023-10-30T13:26:12.122 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2023-10-30T13:26:12.123 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2023-10-30T13:26:12.123 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2023-10-30T13:26:12.123 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2023-10-30T13:26:12.123 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2023-10-30T13:26:12.124 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2023-10-30T13:26:12.124 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2023-10-30T13:26:12.124 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2023-10-30T13:26:12.124 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2023-10-30T13:26:12.124 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2023-10-30T13:26:12.124 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2023-10-30T13:26:12.125 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2023-10-30T13:26:12.125 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2023-10-30T13:26:12.125 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2023-10-30T13:26:12.125 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2023-10-30T13:26:12.125 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2023-10-30T13:26:12.125 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2023-10-30T13:26:12.126 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2023-10-30T13:26:12.126 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2023-10-30T13:26:12.126 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2023-10-30T13:26:12.126 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2023-10-30T13:26:12.640 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:12.772 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2023-10-30T13:26:12.772 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2023-10-30T13:26:12.773 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2023-10-30T13:26:12.773 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2023-10-30T13:26:12.773 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2023-10-30T13:26:12.773 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2023-10-30T13:26:12.773 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2023-10-30T13:26:12.774 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2023-10-30T13:26:12.774 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2023-10-30T13:26:12.774 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2023-10-30T13:26:12.774 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2023-10-30T13:26:12.774 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2023-10-30T13:26:12.775 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2023-10-30T13:26:12.775 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2023-10-30T13:26:12.775 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2023-10-30T13:26:12.775 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2023-10-30T13:26:12.775 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2023-10-30T13:26:12.776 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2023-10-30T13:26:12.776 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2023-10-30T13:26:12.776 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2023-10-30T13:26:12.776 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2023-10-30T13:26:12.776 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2023-10-30T13:26:12.776 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2023-10-30T13:26:12.777 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2023-10-30T13:26:12.777 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2023-10-30T13:26:12.777 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2023-10-30T13:26:12.777 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2023-10-30T13:26:12.777 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2023-10-30T13:26:12.778 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2023-10-30T13:26:12.778 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2023-10-30T13:26:12.778 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2023-10-30T13:26:12.778 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2023-10-30T13:26:12.778 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2023-10-30T13:26:12.779 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2023-10-30T13:26:12.779 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2023-10-30T13:26:12.779 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2023-10-30T13:26:12.779 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2023-10-30T13:26:12.779 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2023-10-30T13:26:12.780 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2023-10-30T13:26:12.780 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2023-10-30T13:26:12.780 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2023-10-30T13:26:12.780 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2023-10-30T13:26:12.780 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2023-10-30T13:26:12.781 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2023-10-30T13:26:12.781 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2023-10-30T13:26:12.781 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2023-10-30T13:26:12.781 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2023-10-30T13:26:12.781 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2023-10-30T13:26:12.782 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2023-10-30T13:26:12.782 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2023-10-30T13:26:12.782 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2023-10-30T13:26:12.782 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2023-10-30T13:26:12.782 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2023-10-30T13:26:12.783 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2023-10-30T13:26:12.783 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2023-10-30T13:26:12.783 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2023-10-30T13:26:12.783 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2023-10-30T13:26:12.784 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2023-10-30T13:26:12.784 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2023-10-30T13:26:12.784 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2023-10-30T13:26:12.784 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2023-10-30T13:26:12.784 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2023-10-30T13:26:12.785 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2023-10-30T13:26:12.785 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2023-10-30T13:26:12.785 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2023-10-30T13:26:12.785 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2023-10-30T13:26:12.785 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2023-10-30T13:26:12.786 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2023-10-30T13:26:12.786 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2023-10-30T13:26:12.786 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2023-10-30T13:26:12.786 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2023-10-30T13:26:12.786 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2023-10-30T13:26:12.787 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2023-10-30T13:26:12.787 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2023-10-30T13:26:12.787 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2023-10-30T13:26:12.787 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2023-10-30T13:26:12.787 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2023-10-30T13:26:12.787 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2023-10-30T13:26:12.788 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2023-10-30T13:26:12.788 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2023-10-30T13:26:12.788 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2023-10-30T13:26:12.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.794+0000 7fa6a4db8640 1 Processor -- start 2023-10-30T13:26:12.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.794+0000 7fa6a4db8640 1 -- start start 2023-10-30T13:26:12.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.794+0000 7fa6a4db8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa6a006f7a0 0x7fa6a006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:12.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.794+0000 7fa6a4db8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6a00c6bd0 0x7fa6a0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:12.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.794+0000 7fa6a4db8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa6a0071110 0x7fa6a0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:12.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.794+0000 7fa6a4db8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a00700c0 con 0x7fa6a00c6bd0 2023-10-30T13:26:12.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.794+0000 7fa6a4db8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a006a3d0 con 0x7fa6a0071110 2023-10-30T13:26:12.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.794+0000 7fa6a4db8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a0066cf0 con 0x7fa6a006f7a0 2023-10-30T13:26:12.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.794+0000 7fa697fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6a00c6bd0 0x7fa6a0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:12.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.794+0000 7fa697fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6a00c6bd0 0x7fa6a0069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39354/0 (socket says 172.21.15.46:39354) 2023-10-30T13:26:12.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.794+0000 7fa697fff640 1 -- 172.21.15.46:0/958219018 learned_addr learned my addr 172.21.15.46:0/958219018 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:12.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.798+0000 7fa69ed9d640 1 --2- 172.21.15.46:0/958219018 >> v2:172.21.15.154:3300/0 conn(0x7fa6a0071110 0x7fa6a0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:12.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.798+0000 7fa69e59c640 1 --2- 172.21.15.46:0/958219018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa6a006f7a0 0x7fa6a006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:12.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.798+0000 7fa697fff640 1 -- 172.21.15.46:0/958219018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa6a006f7a0 msgr2=0x7fa6a006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:12.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.798+0000 7fa697fff640 1 --2- 172.21.15.46:0/958219018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa6a006f7a0 0x7fa6a006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:12.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.798+0000 7fa697fff640 1 -- 172.21.15.46:0/958219018 >> v2:172.21.15.154:3300/0 conn(0x7fa6a0071110 msgr2=0x7fa6a0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:12.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.798+0000 7fa697fff640 1 --2- 172.21.15.46:0/958219018 >> v2:172.21.15.154:3300/0 conn(0x7fa6a0071110 0x7fa6a0158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:12.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.798+0000 7fa697fff640 1 -- 172.21.15.46:0/958219018 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a00c70d0 con 0x7fa6a00c6bd0 2023-10-30T13:26:12.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.798+0000 7fa697fff640 1 --2- 172.21.15.46:0/958219018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a00c6bd0 0x7fa6a0069e90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa6a006c7d0 tx=0x7fa68801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac99da089f16115 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:12.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.798+0000 7fa69dd9b640 1 -- 172.21.15.46:0/958219018 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6880a89b0 con 0x7fa6a00c6bd0 2023-10-30T13:26:12.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.798+0000 7fa69dd9b640 1 -- 172.21.15.46:0/958219018 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6880a8b10 con 0x7fa6a00c6bd0 2023-10-30T13:26:12.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.798+0000 7fa69dd9b640 1 -- 172.21.15.46:0/958219018 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6880a8ea0 con 0x7fa6a00c6bd0 2023-10-30T13:26:12.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 -- 172.21.15.46:0/958219018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a00c6bd0 msgr2=0x7fa6a0069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:12.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 --2- 172.21.15.46:0/958219018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a00c6bd0 0x7fa6a0069e90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa6a006c7d0 tx=0x7fa68801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:26:12.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 -- 172.21.15.46:0/958219018 shutdown_connections 2023-10-30T13:26:12.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 --2- 172.21.15.46:0/958219018 >> v2:172.21.15.154:3300/0 conn(0x7fa6a0071110 0x7fa6a0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:12.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 --2- 172.21.15.46:0/958219018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a00c6bd0 0x7fa6a0069e90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:12.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 --2- 172.21.15.46:0/958219018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa6a006f7a0 0x7fa6a006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:12.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 -- 172.21.15.46:0/958219018 >> 172.21.15.46:0/958219018 conn(0x7fa6a00c7910 msgr2=0x7fa6a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:12.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 -- 172.21.15.46:0/958219018 shutdown_connections 2023-10-30T13:26:12.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 -- 172.21.15.46:0/958219018 wait complete. 2023-10-30T13:26:12.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 Processor -- start 2023-10-30T13:26:12.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 -- start start 2023-10-30T13:26:12.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa6a006f7a0 0x7fa6a014ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:12.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a0071110 0x7fa6a014b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:12.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6a00c6bd0 0x7fa6a00c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:12.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a014b4c0 con 0x7fa6a0071110 2023-10-30T13:26:12.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a014d3a0 con 0x7fa6a00c6bd0 2023-10-30T13:26:12.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa6a4db8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a00c8e40 con 0x7fa6a006f7a0 2023-10-30T13:26:12.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa697fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a0071110 0x7fa6a014b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:12.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa697fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a0071110 0x7fa6a014b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39362/0 (socket says 172.21.15.46:39362) 2023-10-30T13:26:12.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa69e59c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa6a006f7a0 0x7fa6a014ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa697fff640 1 -- 172.21.15.46:0/2566719798 learned_addr learned my addr 172.21.15.46:0/2566719798 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa69ed9d640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6a00c6bd0 0x7fa6a00c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa697fff640 1 -- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa6a006f7a0 msgr2=0x7fa6a014ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa697fff640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa6a006f7a0 0x7fa6a014ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:12.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa697fff640 1 -- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6a00c6bd0 msgr2=0x7fa6a00c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:12.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa697fff640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6a00c6bd0 0x7fa6a00c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:12.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa697fff640 1 -- 172.21.15.46:0/2566719798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6880094d0 con 0x7fa6a0071110 2023-10-30T13:26:12.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa69ed9d640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6a00c6bd0 0x7fa6a00c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:12.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.802+0000 7fa697fff640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a0071110 0x7fa6a014b8e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa68801d3a0 tx=0x7fa68801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:12.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.806+0000 7fa696ffd640 1 -- 172.21.15.46:0/2566719798 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6880ba600 con 0x7fa6a0071110 2023-10-30T13:26:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.806+0000 7fa696ffd640 1 -- 172.21.15.46:0/2566719798 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6880ba760 con 0x7fa6a0071110 2023-10-30T13:26:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.806+0000 7fa6a4db8640 1 -- 172.21.15.46:0/2566719798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a014af40 con 0x7fa6a0071110 2023-10-30T13:26:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.806+0000 7fa696ffd640 1 -- 172.21.15.46:0/2566719798 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6880bab10 con 0x7fa6a0071110 2023-10-30T13:26:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.806+0000 7fa6a4db8640 1 -- 172.21.15.46:0/2566719798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a014b1e0 con 0x7fa6a0071110 2023-10-30T13:26:12.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.806+0000 7fa6a4db8640 1 -- 172.21.15.46:0/2566719798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a00c8c10 con 0x7fa6a0071110 2023-10-30T13:26:12.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.806+0000 7fa696ffd640 1 -- 172.21.15.46:0/2566719798 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa6880bac70 con 0x7fa6a0071110 2023-10-30T13:26:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.806+0000 7fa696ffd640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa6800664a0 0x7fa680068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.806+0000 7fa696ffd640 1 -- 172.21.15.46:0/2566719798 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fa688122480 con 0x7fa6a0071110 2023-10-30T13:26:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.806+0000 7fa69e59c640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa6800664a0 0x7fa680068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.810+0000 7fa69e59c640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa6800664a0 0x7fa680068960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa6900045f0 tx=0x7fa690008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:12.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:12.810+0000 7fa696ffd640 1 -- 172.21.15.46:0/2566719798 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6a00c8c10 con 0x7fa6a0071110 2023-10-30T13:26:13.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.018+0000 7fa6a4db8640 1 -- 172.21.15.46:0/2566719798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6a00c8c10 con 0x7fa6a0071110 2023-10-30T13:26:13.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.018+0000 7fa696ffd640 1 -- 172.21.15.46:0/2566719798 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fa6a00c8c10 con 0x7fa6a0071110 2023-10-30T13:26:13.025 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:13.025 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4950,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:00.013027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":39,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:13.026 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-10-30T13:26:13.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.022+0000 7fa6a4db8640 1 -- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa6800664a0 msgr2=0x7fa680068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:13.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.022+0000 7fa6a4db8640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa6800664a0 0x7fa680068960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa6900045f0 tx=0x7fa690008040 comp rx=0 tx=0).stop 2023-10-30T13:26:13.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.022+0000 7fa6a4db8640 1 -- 172.21.15.46:0/2566719798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a0071110 msgr2=0x7fa6a014b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:13.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.022+0000 7fa6a4db8640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a0071110 0x7fa6a014b8e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa68801d3a0 tx=0x7fa68801ba90 comp rx=0 tx=0).stop 2023-10-30T13:26:13.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.022+0000 7fa6a4db8640 1 -- 172.21.15.46:0/2566719798 shutdown_connections 2023-10-30T13:26:13.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.022+0000 7fa6a4db8640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa6800664a0 0x7fa680068960 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.022+0000 7fa6a4db8640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6a00c6bd0 0x7fa6a00c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.022+0000 7fa6a4db8640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a0071110 0x7fa6a014b8e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.022+0000 7fa6a4db8640 1 --2- 172.21.15.46:0/2566719798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa6a006f7a0 0x7fa6a014ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.022+0000 7fa6a4db8640 1 -- 172.21.15.46:0/2566719798 >> 172.21.15.46:0/2566719798 conn(0x7fa6a00c7910 msgr2=0x7fa6a006be40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:13.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.022+0000 7fa6a4db8640 1 -- 172.21.15.46:0/2566719798 shutdown_connections 2023-10-30T13:26:13.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.022+0000 7fa6a4db8640 1 -- 172.21.15.46:0/2566719798 wait complete. 2023-10-30T13:26:13.047 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=2) 2023-10-30T13:26:13.047 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:26:13.047 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:26:13.062 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:26:13.063 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:13.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f636869a640 1 Processor -- start 2023-10-30T13:26:13.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f636869a640 1 -- start start 2023-10-30T13:26:13.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f636869a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f636015d6f0 0x7f636015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:13.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f636869a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63600cf160 0x7f63600cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:13.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f636869a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63600cb800 con 0x7f63601538a0 2023-10-30T13:26:13.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f636869a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63600cb330 con 0x7f636015d6f0 2023-10-30T13:26:13.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f636869a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63600cad70 con 0x7f63600cf160 2023-10-30T13:26:13.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f6366436640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f63601538a0 0x7f63600ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52468/0 (socket says 172.21.15.46:52468) 2023-10-30T13:26:13.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f6366436640 1 -- 172.21.15.46:0/3549014717 learned_addr learned my addr 172.21.15.46:0/3549014717 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:26:13.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f6366c37640 1 --2- 172.21.15.46:0/3549014717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63600cf160 0x7f63600cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:13.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f6365c35640 1 --2- 172.21.15.46:0/3549014717 >> v2:172.21.15.154:3300/0 conn(0x7f636015d6f0 0x7f636015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:13.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900569308 0 0) 0x7f63600cb800 con 0x7f63601538a0 2023-10-30T13:26:13.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.242+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f634c003530 con 0x7f63601538a0 2023-10-30T13:26:13.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1767547141 0 0) 0x7f634c003530 con 0x7f63601538a0 2023-10-30T13:26:13.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63600cf160 msgr2=0x7f63600cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:13.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 --2- 172.21.15.46:0/3549014717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63600cf160 0x7f63600cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 >> v2:172.21.15.154:3300/0 conn(0x7f636015d6f0 msgr2=0x7f636015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:13.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 --2- 172.21.15.46:0/3549014717 >> v2:172.21.15.154:3300/0 conn(0x7f636015d6f0 0x7f636015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63600cb050 con 0x7f63601538a0 2023-10-30T13:26:13.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f635c001680 con 0x7f63601538a0 2023-10-30T13:26:13.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 >> v1:172.21.15.46:6789/0 conn(0x7f63601538a0 legacy=0x7f63600ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:13.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 --2- 172.21.15.46:0/3549014717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63600cf160 0x7f634c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:13.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 --2- 172.21.15.46:0/3549014717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f634c003f00 0x7f634c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:13.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 --2- 172.21.15.46:0/3549014717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f634c004d10 0x7f634c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:13.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63600cb330 con 0x7f634c004d10 2023-10-30T13:26:13.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63600cad70 con 0x7f634c003f00 2023-10-30T13:26:13.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634c00b9c0 con 0x7f63600cf160 2023-10-30T13:26:13.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6366c37640 1 --2- 172.21.15.46:0/3549014717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f634c004d10 0x7f634c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:13.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6366436640 1 --2- 172.21.15.46:0/3549014717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63600cf160 0x7f634c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:13.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6365c35640 1 --2- 172.21.15.46:0/3549014717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f634c003f00 0x7f634c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:13.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6366c37640 1 -- 172.21.15.46:0/3549014717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63600cf160 msgr2=0x7f634c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:13.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6366c37640 1 --2- 172.21.15.46:0/3549014717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63600cf160 0x7f634c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6366c37640 1 -- 172.21.15.46:0/3549014717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f634c003f00 msgr2=0x7f634c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:13.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6366c37640 1 --2- 172.21.15.46:0/3549014717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f634c003f00 0x7f634c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6366c37640 1 -- 172.21.15.46:0/3549014717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c003530 con 0x7f634c004d10 2023-10-30T13:26:13.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.246+0000 7f6366c37640 1 --2- 172.21.15.46:0/3549014717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f634c004d10 0x7f634c005190 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6358009fc0 tx=0x7f635801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a207a6ecc4243b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:13.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63580a5c70 con 0x7f634c004d10 2023-10-30T13:26:13.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635800f040 con 0x7f634c004d10 2023-10-30T13:26:13.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f6365434640 1 -- 172.21.15.46:0/3549014717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63580ad440 con 0x7f634c004d10 2023-10-30T13:26:13.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 -- 172.21.15.46:0/3549014717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f634c004d10 msgr2=0x7f634c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:13.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 --2- 172.21.15.46:0/3549014717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f634c004d10 0x7f634c005190 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6358009fc0 tx=0x7f635801cc60 comp rx=0 tx=0).stop 2023-10-30T13:26:13.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f6366c37640 1 -- 172.21.15.46:0/3549014717 reap_dead start 2023-10-30T13:26:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 -- 172.21.15.46:0/3549014717 shutdown_connections 2023-10-30T13:26:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 -- 172.21.15.46:0/3549014717 >> 172.21.15.46:0/3549014717 conn(0x7f636006bfe0 msgr2=0x7f636006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 -- 172.21.15.46:0/3549014717 shutdown_connections 2023-10-30T13:26:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 -- 172.21.15.46:0/3549014717 wait complete. 2023-10-30T13:26:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 Processor -- start 2023-10-30T13:26:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 -- start start 2023-10-30T13:26:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f634c004d10 0x7f6360154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6360155180 0x7f636014f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f636014fbe0 0x7f6360155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6360150090 con 0x7f6360155180 2023-10-30T13:26:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6360153e20 con 0x7f634c004d10 2023-10-30T13:26:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f636869a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6360154100 con 0x7f636014fbe0 2023-10-30T13:26:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f6365c35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6360155180 0x7f636014f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f6365c35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6360155180 0x7f636014f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39374/0 (socket says 172.21.15.46:39374) 2023-10-30T13:26:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.250+0000 7f6365c35640 1 -- 172.21.15.46:0/911526100 learned_addr learned my addr 172.21.15.46:0/911526100 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6366436640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f634c004d10 0x7f6360154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6366c37640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f636014fbe0 0x7f6360155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6365c35640 1 -- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f636014fbe0 msgr2=0x7f6360155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6365c35640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f636014fbe0 0x7f6360155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6365c35640 1 -- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f634c004d10 msgr2=0x7f6360154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6365c35640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f634c004d10 0x7f6360154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6365c35640 1 -- 172.21.15.46:0/911526100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63600c64d0 con 0x7f6360155180 2023-10-30T13:26:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6365c35640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6360155180 0x7f636014f6a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f63500119c0 tx=0x7f6350011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6356ffd640 1 -- 172.21.15.46:0/911526100 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635000fb50 con 0x7f6360155180 2023-10-30T13:26:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6356ffd640 1 -- 172.21.15.46:0/911526100 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635000fcb0 con 0x7f6360155180 2023-10-30T13:26:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6356ffd640 1 -- 172.21.15.46:0/911526100 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350014440 con 0x7f6360155180 2023-10-30T13:26:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f636869a640 1 -- 172.21.15.46:0/911526100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63600c67b0 con 0x7f6360155180 2023-10-30T13:26:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f636869a640 1 -- 172.21.15.46:0/911526100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63600c6d70 con 0x7f6360155180 2023-10-30T13:26:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f636869a640 1 -- 172.21.15.46:0/911526100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636004e170 con 0x7f6360155180 2023-10-30T13:26:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6356ffd640 1 -- 172.21.15.46:0/911526100 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f63500145a0 con 0x7f6360155180 2023-10-30T13:26:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6356ffd640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63440664d0 0x7f6344068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.254+0000 7f6356ffd640 1 -- 172.21.15.46:0/911526100 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f6350084920 con 0x7f6360155180 2023-10-30T13:26:13.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.258+0000 7f6366436640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63440664d0 0x7f6344068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:13.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.258+0000 7f6366436640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63440664d0 0x7f6344068990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f635c001ec0 tx=0x7f635c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:13.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.258+0000 7f6356ffd640 1 -- 172.21.15.46:0/911526100 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6350051ba0 con 0x7f6360155180 2023-10-30T13:26:13.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.450+0000 7f636869a640 1 -- 172.21.15.46:0/911526100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63600ca9b0 con 0x7f6360155180 2023-10-30T13:26:13.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f6356ffd640 1 -- 172.21.15.46:0/911526100 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f6350022090 con 0x7f6360155180 2023-10-30T13:26:13.458 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:13.458 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4950,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:00.013027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":39,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-10-30T13:26:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f636869a640 1 -- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63440664d0 msgr2=0x7f6344068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f636869a640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63440664d0 0x7f6344068990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f635c001ec0 tx=0x7f635c009040 comp rx=0 tx=0).stop 2023-10-30T13:26:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f636869a640 1 -- 172.21.15.46:0/911526100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6360155180 msgr2=0x7f636014f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f636869a640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6360155180 0x7f636014f6a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f63500119c0 tx=0x7f6350011e80 comp rx=0 tx=0).stop 2023-10-30T13:26:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f636869a640 1 -- 172.21.15.46:0/911526100 shutdown_connections 2023-10-30T13:26:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f636869a640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63440664d0 0x7f6344068990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f636869a640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f636014fbe0 0x7f6360155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f636869a640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6360155180 0x7f636014f6a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f636869a640 1 --2- 172.21.15.46:0/911526100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f634c004d10 0x7f6360154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:13.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f636869a640 1 -- 172.21.15.46:0/911526100 >> 172.21.15.46:0/911526100 conn(0x7f636006bfe0 msgr2=0x7f63600c5920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:13.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f636869a640 1 -- 172.21.15.46:0/911526100 shutdown_connections 2023-10-30T13:26:13.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:13.454+0000 7f636869a640 1 -- 172.21.15.46:0/911526100 wait complete. 2023-10-30T13:26:13.479 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:26:13.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:13.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:15.480 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:15.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.630+0000 7f47dd7d3640 1 Processor -- start 2023-10-30T13:26:15.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.630+0000 7f47dd7d3640 1 -- start start 2023-10-30T13:26:15.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.630+0000 7f47dd7d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f47d815d2c0 0x7f47d815d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:15.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.630+0000 7f47dd7d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d815ce30 con 0x7f47d80c90d0 2023-10-30T13:26:15.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.630+0000 7f47dd7d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d8159880 con 0x7f47d80c8cf0 2023-10-30T13:26:15.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.630+0000 7f47dd7d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d815dc70 con 0x7f47d815d2c0 2023-10-30T13:26:15.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.630+0000 7f47cffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f47d80c90d0 0x7f47d80cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52482/0 (socket says 172.21.15.46:52482) 2023-10-30T13:26:15.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.630+0000 7f47cffff640 1 -- 172.21.15.46:0/2902936451 learned_addr learned my addr 172.21.15.46:0/2902936451 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:26:15.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d77fe640 1 --2- 172.21.15.46:0/2902936451 >> v2:172.21.15.154:3300/0 conn(0x7f47d815d2c0 0x7f47d815d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:15.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336417610 0 0) 0x7f47d815ce30 con 0x7f47d80c90d0 2023-10-30T13:26:15.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c4003530 con 0x7f47d80c90d0 2023-10-30T13:26:15.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3573335879 0 0) 0x7f47c4003530 con 0x7f47d80c90d0 2023-10-30T13:26:15.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 >> v2:172.21.15.154:3300/0 conn(0x7f47d815d2c0 msgr2=0x7f47d815d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:15.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 --2- 172.21.15.46:0/2902936451 >> v2:172.21.15.154:3300/0 conn(0x7f47d815d2c0 0x7f47d815d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:15.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 >> v1:172.21.15.154:6789/0 conn(0x7f47d80c8cf0 legacy=0x7f47d80c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:15.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d8159ae0 con 0x7f47d80c90d0 2023-10-30T13:26:15.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f47c0001680 con 0x7f47d80c90d0 2023-10-30T13:26:15.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 >> v1:172.21.15.46:6789/0 conn(0x7f47d80c90d0 legacy=0x7f47d80cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:15.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 --2- 172.21.15.46:0/2902936451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47c4003980 0x7f47c4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:15.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 --2- 172.21.15.46:0/2902936451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 0x7f47c4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:15.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 --2- 172.21.15.46:0/2902936451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47c4005190 0x7f47c400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:15.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d815dc70 con 0x7f47c40047d0 2023-10-30T13:26:15.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c400c030 con 0x7f47c4005190 2023-10-30T13:26:15.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c400c1f0 con 0x7f47c4003980 2023-10-30T13:26:15.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47cffff640 1 --2- 172.21.15.46:0/2902936451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 0x7f47c4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:15.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d6ffd640 1 --2- 172.21.15.46:0/2902936451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47c4003980 0x7f47c4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:15.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d77fe640 1 --2- 172.21.15.46:0/2902936451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47c4005190 0x7f47c400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:15.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47cffff640 1 -- 172.21.15.46:0/2902936451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47c4003980 msgr2=0x7f47c4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:15.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47cffff640 1 --2- 172.21.15.46:0/2902936451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47c4003980 0x7f47c4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:15.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47cffff640 1 -- 172.21.15.46:0/2902936451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47c4005190 msgr2=0x7f47c400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:15.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47cffff640 1 --2- 172.21.15.46:0/2902936451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47c4005190 0x7f47c400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:15.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d77fe640 1 -- 172.21.15.46:0/2902936451 reap_dead start 2023-10-30T13:26:15.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47cffff640 1 -- 172.21.15.46:0/2902936451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c4003530 con 0x7f47c40047d0 2023-10-30T13:26:15.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.634+0000 7f47d6ffd640 1 --2- 172.21.15.46:0/2902936451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47c4003980 0x7f47c4003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:15.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47cffff640 1 --2- 172.21.15.46:0/2902936451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 0x7f47c4004c50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f47c000c240 tx=0x7f47c001f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b875e18fa48be96 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:15.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c0005ae0 con 0x7f47c40047d0 2023-10-30T13:26:15.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c00ab070 con 0x7f47c40047d0 2023-10-30T13:26:15.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47d67fc640 1 -- 172.21.15.46:0/2902936451 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c0007070 con 0x7f47c40047d0 2023-10-30T13:26:15.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2902936451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 msgr2=0x7f47c4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:15.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47dd7d3640 1 --2- 172.21.15.46:0/2902936451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 0x7f47c4004c50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f47c000c240 tx=0x7f47c001f370 comp rx=0 tx=0).stop 2023-10-30T13:26:15.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2902936451 shutdown_connections 2023-10-30T13:26:15.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47dd7d3640 1 --2- 172.21.15.46:0/2902936451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 0x7f47c4004c50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:15.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2902936451 >> 172.21.15.46:0/2902936451 conn(0x7f47d806be40 msgr2=0x7f47d806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:15.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2902936451 shutdown_connections 2023-10-30T13:26:15.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2902936451 wait complete. 2023-10-30T13:26:15.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47dd7d3640 1 Processor -- start 2023-10-30T13:26:15.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.638+0000 7f47dd7d3640 1 -- start start 2023-10-30T13:26:15.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47dd7d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 0x7f47d81ef3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:15.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47dd7d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d80b3210 0x7f47d815c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:15.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47dd7d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47d80c4e90 0x7f47d80c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:15.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47dd7d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80ce8d0 con 0x7f47c40047d0 2023-10-30T13:26:15.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47dd7d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80c5850 con 0x7f47d80b3210 2023-10-30T13:26:15.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47dd7d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d80ceb20 con 0x7f47d80c4e90 2023-10-30T13:26:15.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47d6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 0x7f47d81ef3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:15.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47d6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 0x7f47d81ef3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39400/0 (socket says 172.21.15.46:39400) 2023-10-30T13:26:15.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47d6ffd640 1 -- 172.21.15.46:0/2748513076 learned_addr learned my addr 172.21.15.46:0/2748513076 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:15.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47cffff640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d80b3210 0x7f47d815c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:15.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47d77fe640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47d80c4e90 0x7f47d80c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:15.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47d6ffd640 1 -- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47d80c4e90 msgr2=0x7f47d80c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:15.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47d6ffd640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47d80c4e90 0x7f47d80c5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:15.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47d6ffd640 1 -- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d80b3210 msgr2=0x7f47d815c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:15.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47d6ffd640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d80b3210 0x7f47d815c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:15.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47d6ffd640 1 -- 172.21.15.46:0/2748513076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d80cee00 con 0x7f47c40047d0 2023-10-30T13:26:15.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47cffff640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d80b3210 0x7f47d815c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:26:15.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47d77fe640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47d80c4e90 0x7f47d80c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:26:15.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47d6ffd640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 0x7f47d81ef3e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f47c800e4b0 tx=0x7f47c800e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:15.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47cf7fe640 1 -- 172.21.15.46:0/2748513076 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8017070 con 0x7f47c40047d0 2023-10-30T13:26:15.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47cf7fe640 1 -- 172.21.15.46:0/2748513076 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c80052e0 con 0x7f47c40047d0 2023-10-30T13:26:15.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47cf7fe640 1 -- 172.21.15.46:0/2748513076 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8007ea0 con 0x7f47c40047d0 2023-10-30T13:26:15.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2748513076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d80cf0e0 con 0x7f47c40047d0 2023-10-30T13:26:15.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2748513076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d815c9a0 con 0x7f47c40047d0 2023-10-30T13:26:15.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2748513076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d80cf320 con 0x7f47c40047d0 2023-10-30T13:26:15.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.642+0000 7f47cf7fe640 1 -- 172.21.15.46:0/2748513076 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f47c8007020 con 0x7f47c40047d0 2023-10-30T13:26:15.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.646+0000 7f47cf7fe640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f47b80664d0 0x7f47b8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:15.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.646+0000 7f47cf7fe640 1 -- 172.21.15.46:0/2748513076 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f47c8083660 con 0x7f47c40047d0 2023-10-30T13:26:15.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.646+0000 7f47cffff640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f47b80664d0 0x7f47b8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:15.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.646+0000 7f47cffff640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f47b80664d0 0x7f47b8068990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f47c00af000 tx=0x7f47c00aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:15.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.650+0000 7f47cf7fe640 1 -- 172.21.15.46:0/2748513076 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f47c8050960 con 0x7f47c40047d0 2023-10-30T13:26:15.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.842+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2748513076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47d804e170 con 0x7f47c40047d0 2023-10-30T13:26:15.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.842+0000 7f47cf7fe640 1 -- 172.21.15.46:0/2748513076 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f47c8053f80 con 0x7f47c40047d0 2023-10-30T13:26:15.849 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:15.850 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4950,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:00.013027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":39,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:15.850 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-10-30T13:26:15.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.846+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f47b80664d0 msgr2=0x7f47b8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:15.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.846+0000 7f47dd7d3640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f47b80664d0 0x7f47b8068990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f47c00af000 tx=0x7f47c00aa000 comp rx=0 tx=0).stop 2023-10-30T13:26:15.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.846+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2748513076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 msgr2=0x7f47d81ef3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:15.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.846+0000 7f47dd7d3640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 0x7f47d81ef3e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f47c800e4b0 tx=0x7f47c800e970 comp rx=0 tx=0).stop 2023-10-30T13:26:15.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.846+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2748513076 shutdown_connections 2023-10-30T13:26:15.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.846+0000 7f47dd7d3640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f47b80664d0 0x7f47b8068990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:15.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.846+0000 7f47dd7d3640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47d80c4e90 0x7f47d80c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:15.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.846+0000 7f47dd7d3640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d80b3210 0x7f47d815c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:15.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.846+0000 7f47dd7d3640 1 --2- 172.21.15.46:0/2748513076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c40047d0 0x7f47d81ef3e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:15.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.846+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2748513076 >> 172.21.15.46:0/2748513076 conn(0x7f47d806be40 msgr2=0x7f47d8158290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:15.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.846+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2748513076 shutdown_connections 2023-10-30T13:26:15.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:15.846+0000 7f47dd7d3640 1 -- 172.21.15.46:0/2748513076 wait complete. 2023-10-30T13:26:15.870 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:26:17.871 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:18.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.018+0000 7f966173e640 1 Processor -- start 2023-10-30T13:26:18.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.018+0000 7f966173e640 1 -- start start 2023-10-30T13:26:18.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f966173e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f965c06e470 0x7f965c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:18.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f966173e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f965c0c8490 0x7f965c0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:18.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f966173e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965c06afa0 con 0x7f965c153d70 2023-10-30T13:26:18.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f966173e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f965c06d780 con 0x7f965c06e470 2023-10-30T13:26:18.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f966173e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965c154150 con 0x7f965c0c8490 2023-10-30T13:26:18.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f9653fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f965c06e470 0x7f965c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:18.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f965b7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f965c0c8490 0x7f965c0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:18.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f9653fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f965c06e470 0x7f965c06ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:52914/0 (socket says 172.21.15.46:52914) 2023-10-30T13:26:18.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f9653fff640 1 -- 172.21.15.46:0/2467746897 learned_addr learned my addr 172.21.15.46:0/2467746897 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:18.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f9653fff640 1 -- 172.21.15.46:0/2467746897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f965c0c8490 msgr2=0x7f965c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:18.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f9653fff640 1 --2- 172.21.15.46:0/2467746897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f965c0c8490 0x7f965c0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:18.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f9653fff640 1 -- 172.21.15.46:0/2467746897 >> v1:172.21.15.154:6789/0 conn(0x7f965c153d70 legacy=0x7f965c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:18.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.022+0000 7f9653fff640 1 -- 172.21.15.46:0/2467746897 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c06e030 con 0x7f965c06e470 2023-10-30T13:26:18.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.026+0000 7f9653fff640 1 --2- 172.21.15.46:0/2467746897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f965c06e470 0x7f965c06ca80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9644009870 tx=0x7f964401c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b54490d17befd82f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:18.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.026+0000 7f965a7fc640 1 -- 172.21.15.46:0/2467746897 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96440aa670 con 0x7f965c06e470 2023-10-30T13:26:18.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.026+0000 7f965a7fc640 1 -- 172.21.15.46:0/2467746897 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96440aa7d0 con 0x7f965c06e470 2023-10-30T13:26:18.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.026+0000 7f965a7fc640 1 -- 172.21.15.46:0/2467746897 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96440aab60 con 0x7f965c06e470 2023-10-30T13:26:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.026+0000 7f966173e640 1 -- 172.21.15.46:0/2467746897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f965c06e470 msgr2=0x7f965c06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.026+0000 7f966173e640 1 --2- 172.21.15.46:0/2467746897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f965c06e470 0x7f965c06ca80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9644009870 tx=0x7f964401c510 comp rx=0 tx=0).stop 2023-10-30T13:26:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.026+0000 7f966173e640 1 -- 172.21.15.46:0/2467746897 shutdown_connections 2023-10-30T13:26:18.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.026+0000 7f966173e640 1 --2- 172.21.15.46:0/2467746897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f965c0c8490 0x7f965c0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:18.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.026+0000 7f966173e640 1 --2- 172.21.15.46:0/2467746897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f965c06e470 0x7f965c06ca80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:18.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.026+0000 7f966173e640 1 -- 172.21.15.46:0/2467746897 >> 172.21.15.46:0/2467746897 conn(0x7f965c071ea0 msgr2=0x7f965c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:18.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.026+0000 7f966173e640 1 -- 172.21.15.46:0/2467746897 shutdown_connections 2023-10-30T13:26:18.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.026+0000 7f966173e640 1 -- 172.21.15.46:0/2467746897 wait complete. 2023-10-30T13:26:18.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f966173e640 1 Processor -- start 2023-10-30T13:26:18.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f966173e640 1 -- start start 2023-10-30T13:26:18.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f966173e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f965c06e470 0x7f965c14aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:18.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f966173e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c0c8490 0x7f965c14b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f966173e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f965c153d70 0x7f965c150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f966173e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c14c600 con 0x7f965c0c8490 2023-10-30T13:26:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f966173e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c14cba0 con 0x7f965c153d70 2023-10-30T13:26:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f966173e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965c14afe0 con 0x7f965c06e470 2023-10-30T13:26:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f9653fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c0c8490 0x7f965c14b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:18.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f9653fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c0c8490 0x7f965c14b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56014/0 (socket says 172.21.15.46:56014) 2023-10-30T13:26:18.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f9653fff640 1 -- 172.21.15.46:0/4158951082 learned_addr learned my addr 172.21.15.46:0/4158951082 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:18.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f965affd640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f965c06e470 0x7f965c14aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:18.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f965b7fe640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f965c153d70 0x7f965c150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f9653fff640 1 -- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f965c06e470 msgr2=0x7f965c14aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f9653fff640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f965c06e470 0x7f965c14aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f9653fff640 1 -- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f965c153d70 msgr2=0x7f965c150900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f9653fff640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f965c153d70 0x7f965c150900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f9653fff640 1 -- 172.21.15.46:0/4158951082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96440094d0 con 0x7f965c0c8490 2023-10-30T13:26:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f9653fff640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c0c8490 0x7f965c14b8d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f964401a140 tx=0x7f964401ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f96537fe640 1 -- 172.21.15.46:0/4158951082 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96440b3800 con 0x7f965c0c8490 2023-10-30T13:26:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f96537fe640 1 -- 172.21.15.46:0/4158951082 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96440b3980 con 0x7f965c0c8490 2023-10-30T13:26:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f966173e640 1 -- 172.21.15.46:0/4158951082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f965c14f4e0 con 0x7f965c0c8490 2023-10-30T13:26:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f96537fe640 1 -- 172.21.15.46:0/4158951082 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96440b3d30 con 0x7f965c0c8490 2023-10-30T13:26:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f966173e640 1 -- 172.21.15.46:0/4158951082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965c14faa0 con 0x7f965c0c8490 2023-10-30T13:26:18.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.030+0000 7f966173e640 1 -- 172.21.15.46:0/4158951082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f965c0cb720 con 0x7f965c0c8490 2023-10-30T13:26:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.034+0000 7f96537fe640 1 -- 172.21.15.46:0/4158951082 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f96440b2380 con 0x7f965c0c8490 2023-10-30T13:26:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.034+0000 7f96537fe640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f963c0664d0 0x7f963c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.034+0000 7f96537fe640 1 -- 172.21.15.46:0/4158951082 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f964411e5f0 con 0x7f965c0c8490 2023-10-30T13:26:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.034+0000 7f965affd640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f963c0664d0 0x7f963c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.034+0000 7f965affd640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f963c0664d0 0x7f963c068990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f964c007fd0 tx=0x7f964c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:18.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.038+0000 7f96537fe640 1 -- 172.21.15.46:0/4158951082 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f96440eb9a0 con 0x7f965c0c8490 2023-10-30T13:26:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 -- 172.21.15.46:0/4158951082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f965c04e170 con 0x7f965c0c8490 2023-10-30T13:26:18.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f96537fe640 1 -- 172.21.15.46:0/4158951082 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f96440aa630 con 0x7f965c0c8490 2023-10-30T13:26:18.231 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:18.232 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4950,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:00.013027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":39,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:18.232 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-10-30T13:26:18.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 -- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f963c0664d0 msgr2=0x7f963c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:18.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f963c0664d0 0x7f963c068990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f964c007fd0 tx=0x7f964c009040 comp rx=0 tx=0).stop 2023-10-30T13:26:18.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 -- 172.21.15.46:0/4158951082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c0c8490 msgr2=0x7f965c14b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c0c8490 0x7f965c14b8d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f964401a140 tx=0x7f964401ad50 comp rx=0 tx=0).stop 2023-10-30T13:26:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 -- 172.21.15.46:0/4158951082 shutdown_connections 2023-10-30T13:26:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f963c0664d0 0x7f963c068990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f965c153d70 0x7f965c150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c0c8490 0x7f965c14b8d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 --2- 172.21.15.46:0/4158951082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f965c06e470 0x7f965c14aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 -- 172.21.15.46:0/4158951082 >> 172.21.15.46:0/4158951082 conn(0x7f965c071ea0 msgr2=0x7f965c0cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 -- 172.21.15.46:0/4158951082 shutdown_connections 2023-10-30T13:26:18.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:18.226+0000 7f966173e640 1 -- 172.21.15.46:0/4158951082 wait complete. 2023-10-30T13:26:18.252 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:26:20.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:20.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.402+0000 7f76c9b65640 1 Processor -- start 2023-10-30T13:26:20.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.402+0000 7f76c9b65640 1 -- start start 2023-10-30T13:26:20.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.402+0000 7f76c9b65640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f76c406f7a0 0x7f76c406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:20.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.402+0000 7f76c9b65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76c40c6bd0 0x7f76c4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:20.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.402+0000 7f76c9b65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76c4071110 0x7f76c4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:20.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.402+0000 7f76c9b65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c40700c0 con 0x7f76c40c6bd0 2023-10-30T13:26:20.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.402+0000 7f76c9b65640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c406a3d0 con 0x7f76c406f7a0 2023-10-30T13:26:20.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.402+0000 7f76c9b65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c4066cf0 con 0x7f76c4071110 2023-10-30T13:26:20.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.402+0000 7f76c2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76c40c6bd0 0x7f76c4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:20.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76c40c6bd0 0x7f76c4069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56022/0 (socket says 172.21.15.46:56022) 2023-10-30T13:26:20.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c2ffd640 1 -- 172.21.15.46:0/3987512811 learned_addr learned my addr 172.21.15.46:0/3987512811 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:20.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c3fff640 1 --2- 172.21.15.46:0/3987512811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76c4071110 0x7f76c4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:20.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c37fe640 1 --2- 172.21.15.46:0/3987512811 >> v2:172.21.15.154:3300/0 conn(0x7f76c406f7a0 0x7f76c406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:20.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c2ffd640 1 -- 172.21.15.46:0/3987512811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76c4071110 msgr2=0x7f76c4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:20.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c2ffd640 1 --2- 172.21.15.46:0/3987512811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76c4071110 0x7f76c4158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:20.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c2ffd640 1 -- 172.21.15.46:0/3987512811 >> v2:172.21.15.154:3300/0 conn(0x7f76c406f7a0 msgr2=0x7f76c406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:20.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c2ffd640 1 --2- 172.21.15.46:0/3987512811 >> v2:172.21.15.154:3300/0 conn(0x7f76c406f7a0 0x7f76c406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:20.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c2ffd640 1 -- 172.21.15.46:0/3987512811 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c40c70d0 con 0x7f76c40c6bd0 2023-10-30T13:26:20.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c2ffd640 1 --2- 172.21.15.46:0/3987512811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c40c6bd0 0x7f76c4069e90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f76c406c830 tx=0x7f76b801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa97ca94fc9d708 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c27fc640 1 -- 172.21.15.46:0/3987512811 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b80a89b0 con 0x7f76c40c6bd0 2023-10-30T13:26:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c27fc640 1 -- 172.21.15.46:0/3987512811 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b80a8b10 con 0x7f76c40c6bd0 2023-10-30T13:26:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.406+0000 7f76c27fc640 1 -- 172.21.15.46:0/3987512811 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b80a8ea0 con 0x7f76c40c6bd0 2023-10-30T13:26:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 -- 172.21.15.46:0/3987512811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c40c6bd0 msgr2=0x7f76c4069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 --2- 172.21.15.46:0/3987512811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c40c6bd0 0x7f76c4069e90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f76c406c830 tx=0x7f76b801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:26:20.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 -- 172.21.15.46:0/3987512811 shutdown_connections 2023-10-30T13:26:20.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 --2- 172.21.15.46:0/3987512811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76c4071110 0x7f76c4158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:20.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 --2- 172.21.15.46:0/3987512811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c40c6bd0 0x7f76c4069e90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:20.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 --2- 172.21.15.46:0/3987512811 >> v2:172.21.15.154:3300/0 conn(0x7f76c406f7a0 0x7f76c406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:20.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 -- 172.21.15.46:0/3987512811 >> 172.21.15.46:0/3987512811 conn(0x7f76c40c7910 msgr2=0x7f76c40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:20.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 -- 172.21.15.46:0/3987512811 shutdown_connections 2023-10-30T13:26:20.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 -- 172.21.15.46:0/3987512811 wait complete. 2023-10-30T13:26:20.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 Processor -- start 2023-10-30T13:26:20.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 -- start start 2023-10-30T13:26:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76c406f7a0 0x7f76c414aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c4071110 0x7f76c414bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76c40c6bd0 0x7f76c4148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c41efee0 con 0x7f76c4071110 2023-10-30T13:26:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c406c1d0 con 0x7f76c406f7a0 2023-10-30T13:26:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c9b65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c406c4b0 con 0x7f76c40c6bd0 2023-10-30T13:26:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c4071110 0x7f76c414bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c4071110 0x7f76c414bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56030/0 (socket says 172.21.15.46:56030) 2023-10-30T13:26:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c2ffd640 1 -- 172.21.15.46:0/2625926867 learned_addr learned my addr 172.21.15.46:0/2625926867 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c37fe640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76c406f7a0 0x7f76c414aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c3fff640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76c40c6bd0 0x7f76c4148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:20.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c2ffd640 1 -- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76c40c6bd0 msgr2=0x7f76c4148930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:20.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c2ffd640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76c40c6bd0 0x7f76c4148930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:20.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c2ffd640 1 -- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76c406f7a0 msgr2=0x7f76c414aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:20.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c2ffd640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76c406f7a0 0x7f76c414aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:20.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.410+0000 7f76c2ffd640 1 -- 172.21.15.46:0/2625926867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b80094d0 con 0x7f76c4071110 2023-10-30T13:26:20.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.414+0000 7f76c2ffd640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c4071110 0x7f76c414bfe0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f76b80b3040 tx=0x7f76b801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:20.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.414+0000 7f76abfff640 1 -- 172.21.15.46:0/2625926867 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b80b9620 con 0x7f76c4071110 2023-10-30T13:26:20.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.414+0000 7f76abfff640 1 -- 172.21.15.46:0/2625926867 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b80b9780 con 0x7f76c4071110 2023-10-30T13:26:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.414+0000 7f76abfff640 1 -- 172.21.15.46:0/2625926867 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b80b9b30 con 0x7f76c4071110 2023-10-30T13:26:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.414+0000 7f76c9b65640 1 -- 172.21.15.46:0/2625926867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c41f0180 con 0x7f76c4071110 2023-10-30T13:26:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.414+0000 7f76c9b65640 1 -- 172.21.15.46:0/2625926867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c41f0740 con 0x7f76c4071110 2023-10-30T13:26:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.414+0000 7f76c9b65640 1 -- 172.21.15.46:0/2625926867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c404e170 con 0x7f76c4071110 2023-10-30T13:26:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.414+0000 7f76abfff640 1 -- 172.21.15.46:0/2625926867 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f76b80b7070 con 0x7f76c4071110 2023-10-30T13:26:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.414+0000 7f76abfff640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76b00664a0 0x7f76b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.414+0000 7f76abfff640 1 -- 172.21.15.46:0/2625926867 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f76b8121f90 con 0x7f76c4071110 2023-10-30T13:26:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.414+0000 7f76c37fe640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76b00664a0 0x7f76b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.418+0000 7f76c37fe640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76b00664a0 0x7f76b0068960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f76b400b990 tx=0x7f76b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:20.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.418+0000 7f76abfff640 1 -- 172.21.15.46:0/2625926867 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f76b80ef200 con 0x7f76c4071110 2023-10-30T13:26:20.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 -- 172.21.15.46:0/2625926867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76c41f0930 con 0x7f76c4071110 2023-10-30T13:26:20.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76abfff640 1 -- 172.21.15.46:0/2625926867 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f76b80b5070 con 0x7f76c4071110 2023-10-30T13:26:20.615 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:20.615 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4950,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:00.013027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":39,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:20.615 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-10-30T13:26:20.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 -- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76b00664a0 msgr2=0x7f76b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:20.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76b00664a0 0x7f76b0068960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f76b400b990 tx=0x7f76b4008040 comp rx=0 tx=0).stop 2023-10-30T13:26:20.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 -- 172.21.15.46:0/2625926867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c4071110 msgr2=0x7f76c414bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:20.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c4071110 0x7f76c414bfe0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f76b80b3040 tx=0x7f76b801ba90 comp rx=0 tx=0).stop 2023-10-30T13:26:20.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 -- 172.21.15.46:0/2625926867 shutdown_connections 2023-10-30T13:26:20.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76b00664a0 0x7f76b0068960 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:20.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76c40c6bd0 0x7f76c4148930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:20.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c4071110 0x7f76c414bfe0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:20.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 --2- 172.21.15.46:0/2625926867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76c406f7a0 0x7f76c414aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:20.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 -- 172.21.15.46:0/2625926867 >> 172.21.15.46:0/2625926867 conn(0x7f76c40c7910 msgr2=0x7f76c40c3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:20.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 -- 172.21.15.46:0/2625926867 shutdown_connections 2023-10-30T13:26:20.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:20.610+0000 7f76c9b65640 1 -- 172.21.15.46:0/2625926867 wait complete. 2023-10-30T13:26:20.635 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:26:22.637 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:22.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.786+0000 7f6547a91640 1 Processor -- start 2023-10-30T13:26:22.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.786+0000 7f6547a91640 1 -- start start 2023-10-30T13:26:22.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.786+0000 7f6547a91640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65401580e0 0x7f6540070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:22.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.786+0000 7f6547a91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f654015e700 0x7f654006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:22.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.786+0000 7f6547a91640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65400c5810 con 0x7f65400c6540 2023-10-30T13:26:22.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.786+0000 7f6547a91640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65400c6990 con 0x7f65401580e0 2023-10-30T13:26:22.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.786+0000 7f6547a91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654015eb90 con 0x7f654015e700 2023-10-30T13:26:22.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654582d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65401580e0 0x7f6540070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:22.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654582d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65401580e0 0x7f6540070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56032/0 (socket says 172.21.15.46:56032) 2023-10-30T13:26:22.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654582d640 1 -- 172.21.15.46:0/4085188900 learned_addr learned my addr 172.21.15.46:0/4085188900 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:22.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654502c640 1 --1- 172.21.15.46:0/4085188900 >> v1:172.21.15.46:6789/0 conn(0x7f65400c6540 0x7f6540069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53558/0 (socket says 172.21.15.46:53558) 2023-10-30T13:26:22.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654602e640 1 --2- 172.21.15.46:0/4085188900 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f654015e700 0x7f654006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:22.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654582d640 1 -- 172.21.15.46:0/4085188900 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f654015e700 msgr2=0x7f654006a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:22.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654582d640 1 --2- 172.21.15.46:0/4085188900 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f654015e700 0x7f654006a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:22.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654582d640 1 -- 172.21.15.46:0/4085188900 >> v1:172.21.15.46:6789/0 conn(0x7f65400c6540 legacy=0x7f6540069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:22.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654582d640 1 -- 172.21.15.46:0/4085188900 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65400c6bb0 con 0x7f65401580e0 2023-10-30T13:26:22.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654482b640 1 -- 172.21.15.46:0/4085188900 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610336956 0 0) 0x7f65400c5810 con 0x7f65400c6540 2023-10-30T13:26:22.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654582d640 1 --2- 172.21.15.46:0/4085188900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65401580e0 0x7f6540070740 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f653c009870 tx=0x7f653c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36a9da9cb82044f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:22.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654482b640 1 -- 172.21.15.46:0/4085188900 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c0aa670 con 0x7f65401580e0 2023-10-30T13:26:22.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654482b640 1 -- 172.21.15.46:0/4085188900 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653c0aa7d0 con 0x7f65401580e0 2023-10-30T13:26:22.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.790+0000 7f654482b640 1 -- 172.21.15.46:0/4085188900 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c0aab60 con 0x7f65401580e0 2023-10-30T13:26:22.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 -- 172.21.15.46:0/4085188900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65401580e0 msgr2=0x7f6540070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:22.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 --2- 172.21.15.46:0/4085188900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65401580e0 0x7f6540070740 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f653c009870 tx=0x7f653c01c510 comp rx=0 tx=0).stop 2023-10-30T13:26:22.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 -- 172.21.15.46:0/4085188900 shutdown_connections 2023-10-30T13:26:22.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 --2- 172.21.15.46:0/4085188900 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f654015e700 0x7f654006a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:22.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 --2- 172.21.15.46:0/4085188900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65401580e0 0x7f6540070740 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:22.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 -- 172.21.15.46:0/4085188900 >> 172.21.15.46:0/4085188900 conn(0x7f65400c7910 msgr2=0x7f65400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:22.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 -- 172.21.15.46:0/4085188900 shutdown_connections 2023-10-30T13:26:22.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 -- 172.21.15.46:0/4085188900 wait complete. 2023-10-30T13:26:22.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 Processor -- start 2023-10-30T13:26:22.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 -- start start 2023-10-30T13:26:22.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f65400c6540 0x7f65400c0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:22.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65401580e0 0x7f65400c0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:22.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654015e700 0x7f65400c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:22.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6540160e70 con 0x7f654015e700 2023-10-30T13:26:22.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401f6e80 con 0x7f65401580e0 2023-10-30T13:26:22.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f6547a91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65400c02d0 con 0x7f65400c6540 2023-10-30T13:26:22.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f654602e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654015e700 0x7f65400c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:22.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f654602e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654015e700 0x7f65400c93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56038/0 (socket says 172.21.15.46:56038) 2023-10-30T13:26:22.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f654602e640 1 -- 172.21.15.46:0/2077112321 learned_addr learned my addr 172.21.15.46:0/2077112321 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:22.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f654582d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f65400c6540 0x7f65400c0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:22.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f654502c640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65401580e0 0x7f65400c0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:22.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f654602e640 1 -- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f65400c6540 msgr2=0x7f65400c0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:22.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f654602e640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f65400c6540 0x7f65400c0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:22.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f654602e640 1 -- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65401580e0 msgr2=0x7f65400c0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:22.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f654602e640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65401580e0 0x7f65400c0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:22.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.794+0000 7f654602e640 1 -- 172.21.15.46:0/2077112321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c0094d0 con 0x7f654015e700 2023-10-30T13:26:22.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.798+0000 7f654602e640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654015e700 0x7f65400c93e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6538007e90 tx=0x7f653800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:22.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.798+0000 7f65367fc640 1 -- 172.21.15.46:0/2077112321 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6538014420 con 0x7f654015e700 2023-10-30T13:26:22.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.798+0000 7f65367fc640 1 -- 172.21.15.46:0/2077112321 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6538014580 con 0x7f654015e700 2023-10-30T13:26:22.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.798+0000 7f65367fc640 1 -- 172.21.15.46:0/2077112321 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6538014930 con 0x7f654015e700 2023-10-30T13:26:22.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.798+0000 7f6547a91640 1 -- 172.21.15.46:0/2077112321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65401f6920 con 0x7f654015e700 2023-10-30T13:26:22.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.798+0000 7f6547a91640 1 -- 172.21.15.46:0/2077112321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65401f6bc0 con 0x7f654015e700 2023-10-30T13:26:22.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.798+0000 7f6547a91640 1 -- 172.21.15.46:0/2077112321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f654004e170 con 0x7f654015e700 2023-10-30T13:26:22.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.798+0000 7f65367fc640 1 -- 172.21.15.46:0/2077112321 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6538014a90 con 0x7f654015e700 2023-10-30T13:26:22.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.798+0000 7f65367fc640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f65240664f0 0x7f65240689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:22.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.798+0000 7f65367fc640 1 -- 172.21.15.46:0/2077112321 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f6538084640 con 0x7f654015e700 2023-10-30T13:26:22.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.798+0000 7f654582d640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f65240664f0 0x7f65240689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:22.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.802+0000 7f654582d640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f65240664f0 0x7f65240689b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f653c0ae000 tx=0x7f653c0b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:22.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.802+0000 7f65367fc640 1 -- 172.21.15.46:0/2077112321 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6538051b50 con 0x7f654015e700 2023-10-30T13:26:23.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 -- 172.21.15.46:0/2077112321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65400c0410 con 0x7f654015e700 2023-10-30T13:26:23.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f65367fc640 1 -- 172.21.15.46:0/2077112321 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f6538055170 con 0x7f654015e700 2023-10-30T13:26:23.003 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:23.003 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4950,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:00.013027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":39,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:23.004 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-10-30T13:26:23.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 -- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f65240664f0 msgr2=0x7f65240689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:23.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f65240664f0 0x7f65240689b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f653c0ae000 tx=0x7f653c0b5040 comp rx=0 tx=0).stop 2023-10-30T13:26:23.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 -- 172.21.15.46:0/2077112321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654015e700 msgr2=0x7f65400c93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:23.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654015e700 0x7f65400c93e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6538007e90 tx=0x7f653800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 -- 172.21.15.46:0/2077112321 shutdown_connections 2023-10-30T13:26:23.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f65240664f0 0x7f65240689b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654015e700 0x7f65400c93e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65401580e0 0x7f65400c0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 --2- 172.21.15.46:0/2077112321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f65400c6540 0x7f65400c0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 -- 172.21.15.46:0/2077112321 >> 172.21.15.46:0/2077112321 conn(0x7f65400c7910 msgr2=0x7f654006b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:23.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 -- 172.21.15.46:0/2077112321 shutdown_connections 2023-10-30T13:26:23.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:22.998+0000 7f6547a91640 1 -- 172.21.15.46:0/2077112321 wait complete. 2023-10-30T13:26:23.024 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:26:25.025 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:25.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.174+0000 7fb737dc4640 1 Processor -- start 2023-10-30T13:26:25.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.174+0000 7fb737dc4640 1 -- start start 2023-10-30T13:26:25.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.174+0000 7fb737dc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb73014fd50 0x7fb7300c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:25.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.174+0000 7fb737dc4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb730065f10 0x7fb730071250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:25.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.174+0000 7fb737dc4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7301560a0 con 0x7fb730065a80 2023-10-30T13:26:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.174+0000 7fb737dc4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb730156380 con 0x7fb730065f10 2023-10-30T13:26:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.174+0000 7fb737dc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb730156660 con 0x7fb73014fd50 2023-10-30T13:26:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.174+0000 7fb736361640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb730065f10 0x7fb730071250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.174+0000 7fb73535f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb730065a80 0x7fb7300cb550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53562/0 (socket says 172.21.15.46:53562) 2023-10-30T13:26:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.174+0000 7fb73535f640 1 -- 172.21.15.46:0/2731106692 learned_addr learned my addr 172.21.15.46:0/2731106692 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:26:25.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.174+0000 7fb736361640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb730065f10 0x7fb730071250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56040/0 (socket says 172.21.15.46:56040) 2023-10-30T13:26:25.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.178+0000 7fb735b60640 1 --2- 172.21.15.46:0/2731106692 >> v2:172.21.15.154:3300/0 conn(0x7fb73014fd50 0x7fb7300c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.178+0000 7fb736361640 1 -- 172.21.15.46:0/2731106692 >> v2:172.21.15.154:3300/0 conn(0x7fb73014fd50 msgr2=0x7fb7300c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.178+0000 7fb736361640 1 --2- 172.21.15.46:0/2731106692 >> v2:172.21.15.154:3300/0 conn(0x7fb73014fd50 0x7fb7300c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.178+0000 7fb736361640 1 -- 172.21.15.46:0/2731106692 >> v1:172.21.15.46:6789/0 conn(0x7fb730065a80 legacy=0x7fb7300cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.178+0000 7fb736361640 1 -- 172.21.15.46:0/2731106692 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb730156940 con 0x7fb730065f10 2023-10-30T13:26:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.178+0000 7fb734b5e640 1 -- 172.21.15.46:0/2731106692 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987349062 0 0) 0x7fb7301560a0 con 0x7fb730065a80 2023-10-30T13:26:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.178+0000 7fb736361640 1 --2- 172.21.15.46:0/2731106692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb730065f10 0x7fb730071250 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb7300c90f0 tx=0x7fb72801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efbd92104e8e7252 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.178+0000 7fb734b5e640 1 -- 172.21.15.46:0/2731106692 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280a5980 con 0x7fb730065f10 2023-10-30T13:26:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.178+0000 7fb734b5e640 1 -- 172.21.15.46:0/2731106692 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7280a5ae0 con 0x7fb730065f10 2023-10-30T13:26:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.178+0000 7fb734b5e640 1 -- 172.21.15.46:0/2731106692 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280a5e70 con 0x7fb730065f10 2023-10-30T13:26:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 -- 172.21.15.46:0/2731106692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb730065f10 msgr2=0x7fb730071250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 --2- 172.21.15.46:0/2731106692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb730065f10 0x7fb730071250 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb7300c90f0 tx=0x7fb72801c8e0 comp rx=0 tx=0).stop 2023-10-30T13:26:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 -- 172.21.15.46:0/2731106692 shutdown_connections 2023-10-30T13:26:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 --2- 172.21.15.46:0/2731106692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb730065f10 0x7fb730071250 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 --2- 172.21.15.46:0/2731106692 >> v2:172.21.15.154:3300/0 conn(0x7fb73014fd50 0x7fb7300c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 -- 172.21.15.46:0/2731106692 >> 172.21.15.46:0/2731106692 conn(0x7fb73006c6e0 msgr2=0x7fb730069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:25.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 -- 172.21.15.46:0/2731106692 shutdown_connections 2023-10-30T13:26:25.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 -- 172.21.15.46:0/2731106692 wait complete. 2023-10-30T13:26:25.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 Processor -- start 2023-10-30T13:26:25.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 -- start start 2023-10-30T13:26:25.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb730065a80 0x7fb730143580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:25.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb730065f10 0x7fb730140250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb73014fd50 0x7fb730143f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb730143ac0 con 0x7fb73014fd50 2023-10-30T13:26:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7301f1810 con 0x7fb730065a80 2023-10-30T13:26:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb737dc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb730143d30 con 0x7fb730065f10 2023-10-30T13:26:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb736361640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb73014fd50 0x7fb730143f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb736361640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb73014fd50 0x7fb730143f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56042/0 (socket says 172.21.15.46:56042) 2023-10-30T13:26:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb736361640 1 -- 172.21.15.46:0/1667363025 learned_addr learned my addr 172.21.15.46:0/1667363025 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb73535f640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb730065f10 0x7fb730140250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:25.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb735b60640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb730065a80 0x7fb730143580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:25.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb736361640 1 -- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb730065f10 msgr2=0x7fb730140250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:25.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb736361640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb730065f10 0x7fb730140250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:25.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb736361640 1 -- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb730065a80 msgr2=0x7fb730143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:25.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb736361640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb730065a80 0x7fb730143580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:25.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb736361640 1 -- 172.21.15.46:0/1667363025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb728009a70 con 0x7fb73014fd50 2023-10-30T13:26:25.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.182+0000 7fb73535f640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb730065f10 0x7fb730140250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:25.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.186+0000 7fb736361640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb73014fd50 0x7fb730143f90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb7280194f0 tx=0x7fb728019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:25.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.186+0000 7fb7267fc640 1 -- 172.21.15.46:0/1667363025 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280b9860 con 0x7fb73014fd50 2023-10-30T13:26:25.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.186+0000 7fb7267fc640 1 -- 172.21.15.46:0/1667363025 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7280b99c0 con 0x7fb73014fd50 2023-10-30T13:26:25.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.186+0000 7fb7267fc640 1 -- 172.21.15.46:0/1667363025 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280b9d70 con 0x7fb73014fd50 2023-10-30T13:26:25.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.186+0000 7fb737dc4640 1 -- 172.21.15.46:0/1667363025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb730140790 con 0x7fb73014fd50 2023-10-30T13:26:25.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.186+0000 7fb737dc4640 1 -- 172.21.15.46:0/1667363025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73015ecb0 con 0x7fb73014fd50 2023-10-30T13:26:25.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.186+0000 7fb737dc4640 1 -- 172.21.15.46:0/1667363025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73004e170 con 0x7fb73014fd50 2023-10-30T13:26:25.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.186+0000 7fb7267fc640 1 -- 172.21.15.46:0/1667363025 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb7280ad440 con 0x7fb73014fd50 2023-10-30T13:26:25.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.186+0000 7fb7267fc640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7140664a0 0x7fb714068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:25.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.186+0000 7fb7267fc640 1 -- 172.21.15.46:0/1667363025 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fb72811f270 con 0x7fb73014fd50 2023-10-30T13:26:25.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.186+0000 7fb735b60640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7140664a0 0x7fb714068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:25.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.190+0000 7fb735b60640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7140664a0 0x7fb714068960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009ac0 tx=0x7fb72c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:25.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.190+0000 7fb7267fc640 1 -- 172.21.15.46:0/1667363025 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb7280ec720 con 0x7fb73014fd50 2023-10-30T13:26:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.378+0000 7fb737dc4640 1 -- 172.21.15.46:0/1667363025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb73014d0c0 con 0x7fb73014fd50 2023-10-30T13:26:25.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.378+0000 7fb7267fc640 1 -- 172.21.15.46:0/1667363025 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fb7280a5940 con 0x7fb73014fd50 2023-10-30T13:26:25.385 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:25.385 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4950,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:00.013027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":39,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:25.386 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-10-30T13:26:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.382+0000 7fb737dc4640 1 -- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7140664a0 msgr2=0x7fb714068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.382+0000 7fb737dc4640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7140664a0 0x7fb714068960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009ac0 tx=0x7fb72c009190 comp rx=0 tx=0).stop 2023-10-30T13:26:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.382+0000 7fb737dc4640 1 -- 172.21.15.46:0/1667363025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb73014fd50 msgr2=0x7fb730143f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:25.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.382+0000 7fb737dc4640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb73014fd50 0x7fb730143f90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb7280194f0 tx=0x7fb728019680 comp rx=0 tx=0).stop 2023-10-30T13:26:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.382+0000 7fb737dc4640 1 -- 172.21.15.46:0/1667363025 shutdown_connections 2023-10-30T13:26:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.382+0000 7fb737dc4640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7140664a0 0x7fb714068960 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.382+0000 7fb737dc4640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb73014fd50 0x7fb730143f90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.382+0000 7fb737dc4640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb730065f10 0x7fb730140250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:25.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.382+0000 7fb737dc4640 1 --2- 172.21.15.46:0/1667363025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb730065a80 0x7fb730143580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.382+0000 7fb737dc4640 1 -- 172.21.15.46:0/1667363025 >> 172.21.15.46:0/1667363025 conn(0x7fb73006c6e0 msgr2=0x7fb730069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.382+0000 7fb737dc4640 1 -- 172.21.15.46:0/1667363025 shutdown_connections 2023-10-30T13:26:25.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:25.382+0000 7fb737dc4640 1 -- 172.21.15.46:0/1667363025 wait complete. 2023-10-30T13:26:25.406 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:26:27.407 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:27.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc956b1d640 1 Processor -- start 2023-10-30T13:26:27.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc956b1d640 1 -- start start 2023-10-30T13:26:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc956b1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9501538a0 0x7fc95015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:27.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc956b1d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9500cf160 0x7fc9500cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:27.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc956b1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9500cb800 con 0x7fc9500ca0e0 2023-10-30T13:26:27.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc956b1d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc95015dc30 con 0x7fc9500cf160 2023-10-30T13:26:27.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc956b1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500cb330 con 0x7fc9501538a0 2023-10-30T13:26:27.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc94ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc9500ca0e0 0x7fc95015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41068/0 (socket says 172.21.15.46:41068) 2023-10-30T13:26:27.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc94ffff640 1 -- 172.21.15.46:0/4213342176 learned_addr learned my addr 172.21.15.46:0/4213342176 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:26:27.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc9550ba640 1 --2- 172.21.15.46:0/4213342176 >> v2:172.21.15.154:3300/0 conn(0x7fc9500cf160 0x7fc9500cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:27.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc9548b9640 1 --2- 172.21.15.46:0/4213342176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9501538a0 0x7fc95015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc94f7fe640 1 -- 172.21.15.46:0/4213342176 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435142871 0 0) 0x7fc9500cb800 con 0x7fc9500ca0e0 2023-10-30T13:26:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc94f7fe640 1 -- 172.21.15.46:0/4213342176 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc940003530 con 0x7fc9500ca0e0 2023-10-30T13:26:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc9550ba640 1 -- 172.21.15.46:0/4213342176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9501538a0 msgr2=0x7fc95015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc9550ba640 1 --2- 172.21.15.46:0/4213342176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9501538a0 0x7fc95015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc9550ba640 1 -- 172.21.15.46:0/4213342176 >> v1:172.21.15.46:6789/0 conn(0x7fc9500ca0e0 legacy=0x7fc95015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc94f7fe640 1 -- 172.21.15.46:0/4213342176 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1263703811 0 0) 0x7fc940003530 con 0x7fc9500ca0e0 2023-10-30T13:26:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.558+0000 7fc9550ba640 1 -- 172.21.15.46:0/4213342176 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9500cad70 con 0x7fc9500cf160 2023-10-30T13:26:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.562+0000 7fc9550ba640 1 --2- 172.21.15.46:0/4213342176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9500cf160 0x7fc9500cf590 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc948009d90 tx=0x7fc94801f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81944bd757af419b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.562+0000 7fc94f7fe640 1 -- 172.21.15.46:0/4213342176 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c001680 con 0x7fc9500cf160 2023-10-30T13:26:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.562+0000 7fc94f7fe640 1 -- 172.21.15.46:0/4213342176 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9480a8bb0 con 0x7fc9500cf160 2023-10-30T13:26:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.562+0000 7fc94f7fe640 1 -- 172.21.15.46:0/4213342176 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9480b0440 con 0x7fc9500cf160 2023-10-30T13:26:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.562+0000 7fc956b1d640 1 -- 172.21.15.46:0/4213342176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9500cf160 msgr2=0x7fc9500cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.562+0000 7fc956b1d640 1 --2- 172.21.15.46:0/4213342176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9500cf160 0x7fc9500cf590 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc948009d90 tx=0x7fc94801f060 comp rx=0 tx=0).stop 2023-10-30T13:26:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.562+0000 7fc956b1d640 1 -- 172.21.15.46:0/4213342176 shutdown_connections 2023-10-30T13:26:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.562+0000 7fc956b1d640 1 --2- 172.21.15.46:0/4213342176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9500cf160 0x7fc9500cf590 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.562+0000 7fc956b1d640 1 --2- 172.21.15.46:0/4213342176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9501538a0 0x7fc95015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.562+0000 7fc956b1d640 1 -- 172.21.15.46:0/4213342176 >> 172.21.15.46:0/4213342176 conn(0x7fc95006bfe0 msgr2=0x7fc95006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.562+0000 7fc956b1d640 1 -- 172.21.15.46:0/4213342176 shutdown_connections 2023-10-30T13:26:27.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.562+0000 7fc956b1d640 1 -- 172.21.15.46:0/4213342176 wait complete. 2023-10-30T13:26:27.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc956b1d640 1 Processor -- start 2023-10-30T13:26:27.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc956b1d640 1 -- start start 2023-10-30T13:26:27.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc956b1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9500ca0e0 0x7fc95015b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:27.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc956b1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9500cf160 0x7fc95015bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:27.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc956b1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9501538a0 0x7fc95014db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:27.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc956b1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95015a8b0 con 0x7fc9500ca0e0 2023-10-30T13:26:27.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc956b1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9501505d0 con 0x7fc9501538a0 2023-10-30T13:26:27.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc956b1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95015c1a0 con 0x7fc9500cf160 2023-10-30T13:26:27.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc9548b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9500ca0e0 0x7fc95015b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:27.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc9548b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9500ca0e0 0x7fc95015b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35234/0 (socket says 172.21.15.46:35234) 2023-10-30T13:26:27.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc9548b9640 1 -- 172.21.15.46:0/268790090 learned_addr learned my addr 172.21.15.46:0/268790090 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:27.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc9550ba640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9501538a0 0x7fc95014db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:27.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc94ffff640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9500cf160 0x7fc95015bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:27.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc9548b9640 1 -- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9500cf160 msgr2=0x7fc95015bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:27.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc9548b9640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9500cf160 0x7fc95015bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:27.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc9548b9640 1 -- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9501538a0 msgr2=0x7fc95014db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:27.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc9548b9640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9501538a0 0x7fc95014db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:27.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc9548b9640 1 -- 172.21.15.46:0/268790090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc948009a70 con 0x7fc9500ca0e0 2023-10-30T13:26:27.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc9548b9640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9500ca0e0 0x7fc95015b5e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc94400e870 tx=0x7fc94400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:27.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc94d7fa640 1 -- 172.21.15.46:0/268790090 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc944009510 con 0x7fc9500ca0e0 2023-10-30T13:26:27.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc94d7fa640 1 -- 172.21.15.46:0/268790090 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94400eec0 con 0x7fc9500ca0e0 2023-10-30T13:26:27.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc956b1d640 1 -- 172.21.15.46:0/268790090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95015c480 con 0x7fc9500ca0e0 2023-10-30T13:26:27.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc94d7fa640 1 -- 172.21.15.46:0/268790090 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94400cb40 con 0x7fc9500ca0e0 2023-10-30T13:26:27.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.566+0000 7fc956b1d640 1 -- 172.21.15.46:0/268790090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95015ade0 con 0x7fc9500ca0e0 2023-10-30T13:26:27.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.570+0000 7fc956b1d640 1 -- 172.21.15.46:0/268790090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9500b2a50 con 0x7fc9500ca0e0 2023-10-30T13:26:27.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.570+0000 7fc94d7fa640 1 -- 172.21.15.46:0/268790090 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc94400cca0 con 0x7fc9500ca0e0 2023-10-30T13:26:27.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.570+0000 7fc94d7fa640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9340664a0 0x7fc934068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.570+0000 7fc94d7fa640 1 -- 172.21.15.46:0/268790090 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fc9440841c0 con 0x7fc9500ca0e0 2023-10-30T13:26:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.570+0000 7fc94ffff640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9340664a0 0x7fc934068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.574+0000 7fc94ffff640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9340664a0 0x7fc934068960 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc950151d80 tx=0x7fc93c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.574+0000 7fc94d7fa640 1 -- 172.21.15.46:0/268790090 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc944051430 con 0x7fc9500ca0e0 2023-10-30T13:26:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.762+0000 7fc956b1d640 1 -- 172.21.15.46:0/268790090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc95004e170 con 0x7fc9500ca0e0 2023-10-30T13:26:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.762+0000 7fc94d7fa640 1 -- 172.21.15.46:0/268790090 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fc944054a50 con 0x7fc9500ca0e0 2023-10-30T13:26:27.769 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:27.769 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4950,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:00.013027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":13,"state":"up:active","state_seq":39,"addr":"172.21.15.46:6833/875791418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":875791418},{"type":"v1","addr":"172.21.15.46:6833","nonce":875791418}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:27.770 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-10-30T13:26:27.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.766+0000 7fc956b1d640 1 -- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9340664a0 msgr2=0x7fc934068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:27.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.766+0000 7fc956b1d640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9340664a0 0x7fc934068960 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc950151d80 tx=0x7fc93c009040 comp rx=0 tx=0).stop 2023-10-30T13:26:27.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.766+0000 7fc956b1d640 1 -- 172.21.15.46:0/268790090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9500ca0e0 msgr2=0x7fc95015b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:27.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.766+0000 7fc956b1d640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9500ca0e0 0x7fc95015b5e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc94400e870 tx=0x7fc94400ec70 comp rx=0 tx=0).stop 2023-10-30T13:26:27.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.766+0000 7fc956b1d640 1 -- 172.21.15.46:0/268790090 shutdown_connections 2023-10-30T13:26:27.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.766+0000 7fc956b1d640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9340664a0 0x7fc934068960 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:27.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.766+0000 7fc956b1d640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9501538a0 0x7fc95014db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:27.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.766+0000 7fc956b1d640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9500cf160 0x7fc95015bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:27.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.766+0000 7fc956b1d640 1 --2- 172.21.15.46:0/268790090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9500ca0e0 0x7fc95015b5e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:27.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.766+0000 7fc956b1d640 1 -- 172.21.15.46:0/268790090 >> 172.21.15.46:0/268790090 conn(0x7fc95006bfe0 msgr2=0x7fc950065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.766+0000 7fc956b1d640 1 -- 172.21.15.46:0/268790090 shutdown_connections 2023-10-30T13:26:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:27.766+0000 7fc956b1d640 1 -- 172.21.15.46:0/268790090 wait complete. 2023-10-30T13:26:27.790 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:26:29.791 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.938+0000 7f259e638640 1 Processor -- start 2023-10-30T13:26:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.938+0000 7f259e638640 1 -- start start 2023-10-30T13:26:29.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.942+0000 7f259e638640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25980717b0 0x7f259815ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:29.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.942+0000 7f259e638640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259815b060 con 0x7f25980c6290 2023-10-30T13:26:29.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.942+0000 7f259e638640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25980c4ab0 con 0x7f2598071b90 2023-10-30T13:26:29.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.942+0000 7f259e638640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2598157530 con 0x7f25980717b0 2023-10-30T13:26:29.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.942+0000 7f259d636640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25980717b0 0x7f259815ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:29.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.942+0000 7f259de37640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f25980c6290 0x7f2598161eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41082/0 (socket says 172.21.15.46:41082) 2023-10-30T13:26:29.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.942+0000 7f259de37640 1 -- 172.21.15.46:0/2730910577 learned_addr learned my addr 172.21.15.46:0/2730910577 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:26:29.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.942+0000 7f259d636640 1 -- 172.21.15.46:0/2730910577 >> v1:172.21.15.154:6789/0 conn(0x7f2598071b90 legacy=0x7f25981513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:29.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.942+0000 7f259d636640 1 -- 172.21.15.46:0/2730910577 >> v1:172.21.15.46:6789/0 conn(0x7f25980c6290 legacy=0x7f2598161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:29.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.942+0000 7f259d636640 1 -- 172.21.15.46:0/2730910577 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25980c4d90 con 0x7f25980717b0 2023-10-30T13:26:29.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f259d636640 1 --2- 172.21.15.46:0/2730910577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25980717b0 0x7f259815ab20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f25940097a0 tx=0x7f259401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923a927bbde62d62 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f258ffff640 1 -- 172.21.15.46:0/2730910577 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25940a8950 con 0x7f25980717b0 2023-10-30T13:26:29.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f258ffff640 1 -- 172.21.15.46:0/2730910577 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25940a8ab0 con 0x7f25980717b0 2023-10-30T13:26:29.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f258ffff640 1 -- 172.21.15.46:0/2730910577 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25940a8e40 con 0x7f25980717b0 2023-10-30T13:26:29.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f259e638640 1 -- 172.21.15.46:0/2730910577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25980717b0 msgr2=0x7f259815ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:29.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f259e638640 1 --2- 172.21.15.46:0/2730910577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25980717b0 0x7f259815ab20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f25940097a0 tx=0x7f259401ec50 comp rx=0 tx=0).stop 2023-10-30T13:26:29.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f259e638640 1 -- 172.21.15.46:0/2730910577 shutdown_connections 2023-10-30T13:26:29.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f259e638640 1 --2- 172.21.15.46:0/2730910577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25980717b0 0x7f259815ab20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:29.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f259e638640 1 -- 172.21.15.46:0/2730910577 >> 172.21.15.46:0/2730910577 conn(0x7f25980c75d0 msgr2=0x7f25980c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:29.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f259e638640 1 -- 172.21.15.46:0/2730910577 shutdown_connections 2023-10-30T13:26:29.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f259e638640 1 -- 172.21.15.46:0/2730910577 wait complete. 2023-10-30T13:26:29.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f259e638640 1 Processor -- start 2023-10-30T13:26:29.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.946+0000 7f259e638640 1 -- start start 2023-10-30T13:26:29.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259e638640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25980717b0 0x7f2598154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:29.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259e638640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2598071b90 0x7f2598145530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:29.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259e638640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25980c6290 0x7f2598145a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:29.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259e638640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259814a470 con 0x7f25980c6290 2023-10-30T13:26:29.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259e638640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2598152600 con 0x7f2598071b90 2023-10-30T13:26:29.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259e638640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25981528e0 con 0x7f25980717b0 2023-10-30T13:26:29.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259de37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25980c6290 0x7f2598145a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:29.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259de37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25980c6290 0x7f2598145a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35246/0 (socket says 172.21.15.46:35246) 2023-10-30T13:26:29.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259de37640 1 -- 172.21.15.46:0/3288569729 learned_addr learned my addr 172.21.15.46:0/3288569729 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:29.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259d636640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25980717b0 0x7f2598154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:29.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259ce35640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2598071b90 0x7f2598145530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:29.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259de37640 1 -- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25980717b0 msgr2=0x7f2598154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259de37640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25980717b0 0x7f2598154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259de37640 1 -- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2598071b90 msgr2=0x7f2598145530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259de37640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2598071b90 0x7f2598145530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259de37640 1 -- 172.21.15.46:0/3288569729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25940093b0 con 0x7f25980c6290 2023-10-30T13:26:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259de37640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25980c6290 0x7f2598145a70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f259000ca80 tx=0x7f259000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f258dffb640 1 -- 172.21.15.46:0/3288569729 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259000ac30 con 0x7f25980c6290 2023-10-30T13:26:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f258dffb640 1 -- 172.21.15.46:0/3288569729 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259000ad90 con 0x7f25980c6290 2023-10-30T13:26:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f258dffb640 1 -- 172.21.15.46:0/3288569729 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2590012600 con 0x7f25980c6290 2023-10-30T13:26:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259e638640 1 -- 172.21.15.46:0/3288569729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2598152bc0 con 0x7f25980c6290 2023-10-30T13:26:29.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259e638640 1 -- 172.21.15.46:0/3288569729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2598153180 con 0x7f25980c6290 2023-10-30T13:26:29.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f259e638640 1 -- 172.21.15.46:0/3288569729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2598153370 con 0x7f25980c6290 2023-10-30T13:26:29.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.950+0000 7f258dffb640 1 -- 172.21.15.46:0/3288569729 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2590007070 con 0x7f25980c6290 2023-10-30T13:26:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.954+0000 7f258dffb640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f257c0664d0 0x7f257c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.954+0000 7f258dffb640 1 -- 172.21.15.46:0/3288569729 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f2590083e00 con 0x7f25980c6290 2023-10-30T13:26:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.954+0000 7f259d636640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f257c0664d0 0x7f257c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.954+0000 7f259d636640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f257c0664d0 0x7f257c068990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f25940b2000 tx=0x7f2594008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:29.958+0000 7f258dffb640 1 -- 172.21.15.46:0/3288569729 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2590051160 con 0x7f25980c6290 2023-10-30T13:26:30.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.150+0000 7f259e638640 1 -- 172.21.15.46:0/3288569729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f259804e170 con 0x7f25980c6290 2023-10-30T13:26:30.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.150+0000 7f258dffb640 1 -- 172.21.15.46:0/3288569729 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f2590017070 con 0x7f25980c6290 2023-10-30T13:26:30.157 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:30.158 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:28.713862+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:30.158 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 21 2023-10-30T13:26:30.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.154+0000 7f259e638640 1 -- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f257c0664d0 msgr2=0x7f257c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:30.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.154+0000 7f259e638640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f257c0664d0 0x7f257c068990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f25940b2000 tx=0x7f2594008040 comp rx=0 tx=0).stop 2023-10-30T13:26:30.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.154+0000 7f259e638640 1 -- 172.21.15.46:0/3288569729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25980c6290 msgr2=0x7f2598145a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:30.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.154+0000 7f259e638640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25980c6290 0x7f2598145a70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f259000ca80 tx=0x7f259000cf40 comp rx=0 tx=0).stop 2023-10-30T13:26:30.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.154+0000 7f259e638640 1 -- 172.21.15.46:0/3288569729 shutdown_connections 2023-10-30T13:26:30.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.154+0000 7f259e638640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f257c0664d0 0x7f257c068990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.154+0000 7f259e638640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25980c6290 0x7f2598145a70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.154+0000 7f259e638640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2598071b90 0x7f2598145530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.154+0000 7f259e638640 1 --2- 172.21.15.46:0/3288569729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25980717b0 0x7f2598154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.154+0000 7f259e638640 1 -- 172.21.15.46:0/3288569729 >> 172.21.15.46:0/3288569729 conn(0x7f25980c75d0 msgr2=0x7f25980c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:30.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.154+0000 7f259e638640 1 -- 172.21.15.46:0/3288569729 shutdown_connections 2023-10-30T13:26:30.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.154+0000 7f259e638640 1 -- 172.21.15.46:0/3288569729 wait complete. 2023-10-30T13:26:30.178 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:26:30.179 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:26:30.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.314+0000 7feaaa9c4640 1 Processor -- start 2023-10-30T13:26:30.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaaa9c4640 1 -- start start 2023-10-30T13:26:30.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaaa9c4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feaa41538a0 0x7feaa415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:30.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaaa9c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa40cf160 0x7feaa40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:30.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaaa9c4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa40cb800 con 0x7feaa40ca0e0 2023-10-30T13:26:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaaa9c4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaa415dc30 con 0x7feaa41538a0 2023-10-30T13:26:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaaa9c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa40cb330 con 0x7feaa40cf160 2023-10-30T13:26:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaa3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feaa41538a0 0x7feaa415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaa3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feaa41538a0 0x7feaa415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35258/0 (socket says 172.21.15.46:35258) 2023-10-30T13:26:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaa3fff640 1 -- 172.21.15.46:0/4103616668 learned_addr learned my addr 172.21.15.46:0/4103616668 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:30.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaa37fe640 1 --1- 172.21.15.46:0/4103616668 >> v1:172.21.15.46:6789/0 conn(0x7feaa40ca0e0 0x7feaa415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41096/0 (socket says 172.21.15.46:41096) 2023-10-30T13:26:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaa8f61640 1 --2- 172.21.15.46:0/4103616668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa40cf160 0x7feaa40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaa3fff640 1 -- 172.21.15.46:0/4103616668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa40cf160 msgr2=0x7feaa40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaa3fff640 1 --2- 172.21.15.46:0/4103616668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa40cf160 0x7feaa40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaa3fff640 1 -- 172.21.15.46:0/4103616668 >> v1:172.21.15.46:6789/0 conn(0x7feaa40ca0e0 legacy=0x7feaa415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:30.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaa3fff640 1 -- 172.21.15.46:0/4103616668 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa40cad70 con 0x7feaa41538a0 2023-10-30T13:26:30.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.318+0000 7feaa2ffd640 1 -- 172.21.15.46:0/4103616668 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112854079 0 0) 0x7feaa40cb800 con 0x7feaa40ca0e0 2023-10-30T13:26:30.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaa3fff640 1 --2- 172.21.15.46:0/4103616668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa41538a0 0x7feaa415d6f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fea94009870 tx=0x7fea9401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9daa4bddb8d00f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:30.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaa2ffd640 1 -- 172.21.15.46:0/4103616668 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea940a8a30 con 0x7feaa41538a0 2023-10-30T13:26:30.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaa2ffd640 1 -- 172.21.15.46:0/4103616668 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea940a8b90 con 0x7feaa41538a0 2023-10-30T13:26:30.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaa2ffd640 1 -- 172.21.15.46:0/4103616668 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea940b0440 con 0x7feaa41538a0 2023-10-30T13:26:30.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaaa9c4640 1 -- 172.21.15.46:0/4103616668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa41538a0 msgr2=0x7feaa415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:30.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaaa9c4640 1 --2- 172.21.15.46:0/4103616668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa41538a0 0x7feaa415d6f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fea94009870 tx=0x7fea9401ed30 comp rx=0 tx=0).stop 2023-10-30T13:26:30.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaaa9c4640 1 -- 172.21.15.46:0/4103616668 shutdown_connections 2023-10-30T13:26:30.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaaa9c4640 1 --2- 172.21.15.46:0/4103616668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa40cf160 0x7feaa40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaaa9c4640 1 --2- 172.21.15.46:0/4103616668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa41538a0 0x7feaa415d6f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaaa9c4640 1 -- 172.21.15.46:0/4103616668 >> 172.21.15.46:0/4103616668 conn(0x7feaa406bfe0 msgr2=0x7feaa406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:30.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaaa9c4640 1 -- 172.21.15.46:0/4103616668 shutdown_connections 2023-10-30T13:26:30.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaaa9c4640 1 -- 172.21.15.46:0/4103616668 wait complete. 2023-10-30T13:26:30.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaaa9c4640 1 Processor -- start 2023-10-30T13:26:30.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.322+0000 7feaaa9c4640 1 -- start start 2023-10-30T13:26:30.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaaa9c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaa40ca0e0 0x7feaa415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:30.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaaa9c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa40cf160 0x7feaa415d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:30.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaaa9c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa41538a0 0x7feaa41ef350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:30.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaaa9c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa415bbc0 con 0x7feaa40cf160 2023-10-30T13:26:30.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaaa9c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa415bea0 con 0x7feaa40ca0e0 2023-10-30T13:26:30.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaaa9c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa415c180 con 0x7feaa41538a0 2023-10-30T13:26:30.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa40cf160 0x7feaa415d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:30.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa40cf160 0x7feaa415d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35268/0 (socket says 172.21.15.46:35268) 2023-10-30T13:26:30.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa37fe640 1 -- 172.21.15.46:0/595690826 learned_addr learned my addr 172.21.15.46:0/595690826 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:30.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa3fff640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaa40ca0e0 0x7feaa415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:30.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa8f61640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa41538a0 0x7feaa41ef350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:30.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa37fe640 1 -- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa41538a0 msgr2=0x7feaa41ef350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:30.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa37fe640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa41538a0 0x7feaa41ef350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa37fe640 1 -- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaa40ca0e0 msgr2=0x7feaa415b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:30.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa37fe640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaa40ca0e0 0x7feaa415b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa37fe640 1 -- 172.21.15.46:0/595690826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea940094d0 con 0x7feaa40cf160 2023-10-30T13:26:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa37fe640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa40cf160 0x7feaa415d340 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fea980011f0 tx=0x7fea9800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa0ff9640 1 -- 172.21.15.46:0/595690826 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea980074e0 con 0x7feaa40cf160 2023-10-30T13:26:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa0ff9640 1 -- 172.21.15.46:0/595690826 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea98007640 con 0x7feaa40cf160 2023-10-30T13:26:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaa0ff9640 1 -- 172.21.15.46:0/595690826 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea98004a10 con 0x7feaa40cf160 2023-10-30T13:26:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaaa9c4640 1 -- 172.21.15.46:0/595690826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa415c460 con 0x7feaa40cf160 2023-10-30T13:26:30.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaaa9c4640 1 -- 172.21.15.46:0/595690826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa40d0060 con 0x7feaa40cf160 2023-10-30T13:26:30.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.326+0000 7feaaa9c4640 1 -- 172.21.15.46:0/595690826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa40b2a50 con 0x7feaa40cf160 2023-10-30T13:26:30.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.330+0000 7feaa0ff9640 1 -- 172.21.15.46:0/595690826 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fea98004b70 con 0x7feaa40cf160 2023-10-30T13:26:30.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.330+0000 7feaa0ff9640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea900664d0 0x7fea90068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:30.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.330+0000 7feaa0ff9640 1 -- 172.21.15.46:0/595690826 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fea98083090 con 0x7feaa40cf160 2023-10-30T13:26:30.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.330+0000 7feaa3fff640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea900664d0 0x7fea90068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:30.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.330+0000 7feaa3fff640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea900664d0 0x7fea90068990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fea940b2000 tx=0x7fea94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:30.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.334+0000 7feaa0ff9640 1 -- 172.21.15.46:0/595690826 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fea98050470 con 0x7feaa40cf160 2023-10-30T13:26:30.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.522+0000 7feaaa9c4640 1 -- 172.21.15.46:0/595690826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feaa4159850 con 0x7feaa40cf160 2023-10-30T13:26:30.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.522+0000 7feaa0ff9640 1 -- 172.21.15.46:0/595690826 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5736 (secure 0 0 0) 0x7fea98053a90 con 0x7feaa40cf160 2023-10-30T13:26:30.529 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:30.529 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:28.713862+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:30.530 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 21 2023-10-30T13:26:30.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.526+0000 7feaaa9c4640 1 -- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea900664d0 msgr2=0x7fea90068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:30.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.526+0000 7feaaa9c4640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea900664d0 0x7fea90068990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fea940b2000 tx=0x7fea94009040 comp rx=0 tx=0).stop 2023-10-30T13:26:30.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.526+0000 7feaaa9c4640 1 -- 172.21.15.46:0/595690826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa40cf160 msgr2=0x7feaa415d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:30.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.526+0000 7feaaa9c4640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa40cf160 0x7feaa415d340 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fea980011f0 tx=0x7fea9800c9b0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.526+0000 7feaaa9c4640 1 -- 172.21.15.46:0/595690826 shutdown_connections 2023-10-30T13:26:30.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.526+0000 7feaaa9c4640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea900664d0 0x7fea90068990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.526+0000 7feaaa9c4640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa41538a0 0x7feaa41ef350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.526+0000 7feaaa9c4640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaa40cf160 0x7feaa415d340 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.526+0000 7feaaa9c4640 1 --2- 172.21.15.46:0/595690826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaa40ca0e0 0x7feaa415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:30.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.526+0000 7feaaa9c4640 1 -- 172.21.15.46:0/595690826 >> 172.21.15.46:0/595690826 conn(0x7feaa406bfe0 msgr2=0x7feaa41527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:30.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.526+0000 7feaaa9c4640 1 -- 172.21.15.46:0/595690826 shutdown_connections 2023-10-30T13:26:30.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:30.526+0000 7feaaa9c4640 1 -- 172.21.15.46:0/595690826 wait complete. 2023-10-30T13:26:30.550 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4288": { "addr": "172.21.15.154:6835/3109262047", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3109262047, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3109262047, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4306": { "addr": "172.21.15.46:6836/275749119", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 275749119, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 275749119, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4950": { "addr": "172.21.15.46:6837/1315246492", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 1315246492, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 1315246492, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4950, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:26:28.713862+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4288, "mds_2": 4950 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:26:32.551 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:32.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.698+0000 7f2c3b209640 1 Processor -- start 2023-10-30T13:26:32.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.698+0000 7f2c3b209640 1 -- start start 2023-10-30T13:26:32.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c3b209640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2c3414fc00 0x7f2c34156330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:32.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c3b209640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c34157230 0x7f2c3406a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:32.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c3b209640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c34156870 con 0x7f2c34156da0 2023-10-30T13:26:32.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c3b209640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c34072180 con 0x7f2c34157230 2023-10-30T13:26:32.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c3b209640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c34071cf0 con 0x7f2c3414fc00 2023-10-30T13:26:32.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c397a6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c34157230 0x7f2c3406a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:32.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c397a6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c34157230 0x7f2c3406a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35282/0 (socket says 172.21.15.46:35282) 2023-10-30T13:26:32.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c397a6640 1 -- 172.21.15.46:0/863258189 learned_addr learned my addr 172.21.15.46:0/863258189 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:32.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c38fa5640 1 --2- 172.21.15.46:0/863258189 >> v2:172.21.15.154:3300/0 conn(0x7f2c3414fc00 0x7f2c34156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:32.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c397a6640 1 -- 172.21.15.46:0/863258189 >> v2:172.21.15.154:3300/0 conn(0x7f2c3414fc00 msgr2=0x7f2c34156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:32.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c397a6640 1 --2- 172.21.15.46:0/863258189 >> v2:172.21.15.154:3300/0 conn(0x7f2c3414fc00 0x7f2c34156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:32.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c397a6640 1 -- 172.21.15.46:0/863258189 >> v1:172.21.15.154:6789/0 conn(0x7f2c34156da0 legacy=0x7f2c34069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:32.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.702+0000 7f2c397a6640 1 -- 172.21.15.46:0/863258189 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c34157660 con 0x7f2c34157230 2023-10-30T13:26:32.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.706+0000 7f2c397a6640 1 --2- 172.21.15.46:0/863258189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c34157230 0x7f2c3406a5a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2c3406eea0 tx=0x7f2c2c01fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7d2509a621e64d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.706+0000 7f2c337fe640 1 -- 172.21.15.46:0/863258189 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c0a9950 con 0x7f2c34157230 2023-10-30T13:26:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.706+0000 7f2c337fe640 1 -- 172.21.15.46:0/863258189 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2c0a9ab0 con 0x7f2c34157230 2023-10-30T13:26:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.706+0000 7f2c337fe640 1 -- 172.21.15.46:0/863258189 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c0a9e40 con 0x7f2c34157230 2023-10-30T13:26:32.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.706+0000 7f2c3b209640 1 -- 172.21.15.46:0/863258189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c34157230 msgr2=0x7f2c3406a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:32.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.706+0000 7f2c3b209640 1 --2- 172.21.15.46:0/863258189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c34157230 0x7f2c3406a5a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2c3406eea0 tx=0x7f2c2c01fd40 comp rx=0 tx=0).stop 2023-10-30T13:26:32.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.706+0000 7f2c3b209640 1 -- 172.21.15.46:0/863258189 shutdown_connections 2023-10-30T13:26:32.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.706+0000 7f2c3b209640 1 --2- 172.21.15.46:0/863258189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c34157230 0x7f2c3406a5a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:32.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.706+0000 7f2c3b209640 1 --2- 172.21.15.46:0/863258189 >> v2:172.21.15.154:3300/0 conn(0x7f2c3414fc00 0x7f2c34156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:32.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.706+0000 7f2c3b209640 1 -- 172.21.15.46:0/863258189 >> 172.21.15.46:0/863258189 conn(0x7f2c340c7910 msgr2=0x7f2c340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:32.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.706+0000 7f2c3b209640 1 -- 172.21.15.46:0/863258189 shutdown_connections 2023-10-30T13:26:32.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.706+0000 7f2c3b209640 1 -- 172.21.15.46:0/863258189 wait complete. 2023-10-30T13:26:32.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c3b209640 1 Processor -- start 2023-10-30T13:26:32.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c3b209640 1 -- start start 2023-10-30T13:26:32.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c3b209640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3414fc00 0x7f2c340c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:32.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c3b209640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c34156da0 0x7f2c341f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:32.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c3b209640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c34157230 0x7f2c34154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:32.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c3b209640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c34153820 con 0x7f2c3414fc00 2023-10-30T13:26:32.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c3b209640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c34153de0 con 0x7f2c34157230 2023-10-30T13:26:32.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c3b209640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c340c7310 con 0x7f2c34156da0 2023-10-30T13:26:32.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c38fa5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3414fc00 0x7f2c340c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:32.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c38fa5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3414fc00 0x7f2c340c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35298/0 (socket says 172.21.15.46:35298) 2023-10-30T13:26:32.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c38fa5640 1 -- 172.21.15.46:0/2751443175 learned_addr learned my addr 172.21.15.46:0/2751443175 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:32.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c33fff640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c34156da0 0x7f2c341f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:32.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c397a6640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c34157230 0x7f2c34154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:32.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c38fa5640 1 -- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c34156da0 msgr2=0x7f2c341f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:32.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c38fa5640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c34156da0 0x7f2c341f2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:32.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c38fa5640 1 -- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c34157230 msgr2=0x7f2c34154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:32.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c38fa5640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c34157230 0x7f2c34154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:32.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c38fa5640 1 -- 172.21.15.46:0/2751443175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c00c3d0 con 0x7f2c3414fc00 2023-10-30T13:26:32.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c38fa5640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3414fc00 0x7f2c340c6dd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800e930 tx=0x7f2c2800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:32.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c317fa640 1 -- 172.21.15.46:0/2751443175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c28009690 con 0x7f2c3414fc00 2023-10-30T13:26:32.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c317fa640 1 -- 172.21.15.46:0/2751443175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2800ca00 con 0x7f2c3414fc00 2023-10-30T13:26:32.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c317fa640 1 -- 172.21.15.46:0/2751443175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2800cd70 con 0x7f2c3414fc00 2023-10-30T13:26:32.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c3b209640 1 -- 172.21.15.46:0/2751443175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c341f2d80 con 0x7f2c3414fc00 2023-10-30T13:26:32.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c3b209640 1 -- 172.21.15.46:0/2751443175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c340ca5f0 con 0x7f2c3414fc00 2023-10-30T13:26:32.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.710+0000 7f2c3b209640 1 -- 172.21.15.46:0/2751443175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3404e170 con 0x7f2c3414fc00 2023-10-30T13:26:32.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.714+0000 7f2c317fa640 1 -- 172.21.15.46:0/2751443175 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c28013470 con 0x7f2c3414fc00 2023-10-30T13:26:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.714+0000 7f2c317fa640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c180664d0 0x7f2c18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.714+0000 7f2c317fa640 1 -- 172.21.15.46:0/2751443175 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f2c28084290 con 0x7f2c3414fc00 2023-10-30T13:26:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.714+0000 7f2c33fff640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c180664d0 0x7f2c18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.714+0000 7f2c33fff640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c180664d0 0x7f2c18068990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2c340b06e0 tx=0x7f2c20009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:32.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.718+0000 7f2c317fa640 1 -- 172.21.15.46:0/2751443175 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c280515f0 con 0x7f2c3414fc00 2023-10-30T13:26:32.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.910+0000 7f2c3b209640 1 -- 172.21.15.46:0/2751443175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c340b0af0 con 0x7f2c3414fc00 2023-10-30T13:26:32.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.910+0000 7f2c317fa640 1 -- 172.21.15.46:0/2751443175 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f2c28054c10 con 0x7f2c3414fc00 2023-10-30T13:26:32.916 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:32.917 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:28.713862+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 21 2023-10-30T13:26:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.914+0000 7f2c3b209640 1 -- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c180664d0 msgr2=0x7f2c18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:32.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.914+0000 7f2c3b209640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c180664d0 0x7f2c18068990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2c340b06e0 tx=0x7f2c20009040 comp rx=0 tx=0).stop 2023-10-30T13:26:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.914+0000 7f2c3b209640 1 -- 172.21.15.46:0/2751443175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3414fc00 msgr2=0x7f2c340c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.914+0000 7f2c3b209640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3414fc00 0x7f2c340c6dd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800e930 tx=0x7f2c2800edf0 comp rx=0 tx=0).stop 2023-10-30T13:26:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.914+0000 7f2c3b209640 1 -- 172.21.15.46:0/2751443175 shutdown_connections 2023-10-30T13:26:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.914+0000 7f2c3b209640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c180664d0 0x7f2c18068990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.914+0000 7f2c3b209640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c34157230 0x7f2c34154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:32.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.914+0000 7f2c3b209640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c34156da0 0x7f2c341f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:32.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.914+0000 7f2c3b209640 1 --2- 172.21.15.46:0/2751443175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3414fc00 0x7f2c340c6dd0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:32.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.914+0000 7f2c3b209640 1 -- 172.21.15.46:0/2751443175 >> 172.21.15.46:0/2751443175 conn(0x7f2c340c7910 msgr2=0x7f2c3414e950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:32.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.914+0000 7f2c3b209640 1 -- 172.21.15.46:0/2751443175 shutdown_connections 2023-10-30T13:26:32.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:32.914+0000 7f2c3b209640 1 -- 172.21.15.46:0/2751443175 wait complete. 2023-10-30T13:26:32.938 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:26:34.939 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:35.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.086+0000 7f6c04dc4640 1 Processor -- start 2023-10-30T13:26:35.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.086+0000 7f6c04dc4640 1 -- start start 2023-10-30T13:26:35.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.090+0000 7f6c04dc4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6c00071110 0x7f6c0006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:35.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.090+0000 7f6c04dc4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c00066cf0 con 0x7f6c0006f7a0 2023-10-30T13:26:35.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.090+0000 7f6c04dc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c000c70d0 con 0x7f6c000c6bd0 2023-10-30T13:26:35.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.090+0000 7f6c04dc4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c000700c0 con 0x7f6c00071110 2023-10-30T13:26:35.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.090+0000 7f6bfe59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6c0006f7a0 0x7f6c00069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41112/0 (socket says 172.21.15.46:41112) 2023-10-30T13:26:35.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.090+0000 7f6bfe59c640 1 -- 172.21.15.46:0/3416868878 learned_addr learned my addr 172.21.15.46:0/3416868878 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:26:35.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.090+0000 7f6bfed9d640 1 --2- 172.21.15.46:0/3416868878 >> v2:172.21.15.46:3300/0 conn(0x7f6c00071110 0x7f6c0006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:35.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.090+0000 7f6bfd59a640 1 -- 172.21.15.46:0/3416868878 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024248302 0 0) 0x7f6c00066cf0 con 0x7f6c0006f7a0 2023-10-30T13:26:35.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.090+0000 7f6bfd59a640 1 -- 172.21.15.46:0/3416868878 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6be8003530 con 0x7f6c0006f7a0 2023-10-30T13:26:35.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.090+0000 7f6bfed9d640 1 -- 172.21.15.46:0/3416868878 >> v1:172.21.15.154:6789/0 conn(0x7f6c000c6bd0 legacy=0x7f6c00158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.090+0000 7f6bfed9d640 1 -- 172.21.15.46:0/3416868878 >> v1:172.21.15.46:6789/0 conn(0x7f6c0006f7a0 legacy=0x7f6c00069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.090+0000 7f6bfed9d640 1 -- 172.21.15.46:0/3416868878 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c000c7390 con 0x7f6c00071110 2023-10-30T13:26:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6bfed9d640 1 --2- 172.21.15.46:0/3416868878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c00071110 0x7f6c0006fb80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8009d90 tx=0x7f6bf801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e82ac90b02fe64 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6bfd59a640 1 -- 172.21.15.46:0/3416868878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf80af600 con 0x7f6c00071110 2023-10-30T13:26:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6bfd59a640 1 -- 172.21.15.46:0/3416868878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf80ab050 con 0x7f6c00071110 2023-10-30T13:26:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6bfd59a640 1 -- 172.21.15.46:0/3416868878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf80af980 con 0x7f6c00071110 2023-10-30T13:26:35.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3416868878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c00071110 msgr2=0x7f6c0006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:35.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6c04dc4640 1 --2- 172.21.15.46:0/3416868878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c00071110 0x7f6c0006fb80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8009d90 tx=0x7f6bf801f130 comp rx=0 tx=0).stop 2023-10-30T13:26:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3416868878 shutdown_connections 2023-10-30T13:26:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6c04dc4640 1 --2- 172.21.15.46:0/3416868878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c00071110 0x7f6c0006fb80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3416868878 >> 172.21.15.46:0/3416868878 conn(0x7f6c000c7910 msgr2=0x7f6c000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3416868878 shutdown_connections 2023-10-30T13:26:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3416868878 wait complete. 2023-10-30T13:26:35.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6c04dc4640 1 Processor -- start 2023-10-30T13:26:35.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.094+0000 7f6c04dc4640 1 -- start start 2023-10-30T13:26:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6c04dc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0006f7a0 0x7f6c00154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6c04dc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c00071110 0x7f6c00152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6c04dc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c000c6bd0 0x7f6c000c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:35.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6c04dc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c001f1f00 con 0x7f6c0006f7a0 2023-10-30T13:26:35.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6c04dc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c00155090 con 0x7f6c00071110 2023-10-30T13:26:35.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6c04dc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c001f2760 con 0x7f6c000c6bd0 2023-10-30T13:26:35.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6bfe59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0006f7a0 0x7f6c00154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:35.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6bfe59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0006f7a0 0x7f6c00154b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35304/0 (socket says 172.21.15.46:35304) 2023-10-30T13:26:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6bfe59c640 1 -- 172.21.15.46:0/3085827482 learned_addr learned my addr 172.21.15.46:0/3085827482 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6bfdd9b640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c00071110 0x7f6c00152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6bfed9d640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c000c6bd0 0x7f6c000c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6bfe59c640 1 -- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c000c6bd0 msgr2=0x7f6c000c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6bfe59c640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c000c6bd0 0x7f6c000c5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6bfe59c640 1 -- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c00071110 msgr2=0x7f6c00152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:35.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6bfe59c640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c00071110 0x7f6c00152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:35.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6bfe59c640 1 -- 172.21.15.46:0/3085827482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf8009a70 con 0x7f6c0006f7a0 2023-10-30T13:26:35.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6bfdd9b640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c00071110 0x7f6c00152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:35.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6bfe59c640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0006f7a0 0x7f6c00154b50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000c420 tx=0x7f6bf000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:35.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6beeffd640 1 -- 172.21.15.46:0/3085827482 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0005cf0 con 0x7f6c0006f7a0 2023-10-30T13:26:35.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6beeffd640 1 -- 172.21.15.46:0/3085827482 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf0005e50 con 0x7f6c0006f7a0 2023-10-30T13:26:35.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6beeffd640 1 -- 172.21.15.46:0/3085827482 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf00072d0 con 0x7f6c0006f7a0 2023-10-30T13:26:35.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3085827482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c00151f20 con 0x7f6c0006f7a0 2023-10-30T13:26:35.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3085827482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c001f23f0 con 0x7f6c0006f7a0 2023-10-30T13:26:35.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3085827482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0004e170 con 0x7f6c0006f7a0 2023-10-30T13:26:35.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.098+0000 7f6beeffd640 1 -- 172.21.15.46:0/3085827482 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6bf0007430 con 0x7f6c0006f7a0 2023-10-30T13:26:35.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.102+0000 7f6beeffd640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6be00664a0 0x7f6be0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:35.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.102+0000 7f6beeffd640 1 -- 172.21.15.46:0/3085827482 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f6bf00834e0 con 0x7f6c0006f7a0 2023-10-30T13:26:35.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.102+0000 7f6bfdd9b640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6be00664a0 0x7f6be0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:35.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.102+0000 7f6bfdd9b640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6be00664a0 0x7f6be0068960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6c000c8610 tx=0x7f6bf4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:35.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.106+0000 7f6beeffd640 1 -- 172.21.15.46:0/3085827482 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6bf00508c0 con 0x7f6c0006f7a0 2023-10-30T13:26:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.294+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3085827482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c000c3650 con 0x7f6c0006f7a0 2023-10-30T13:26:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6beeffd640 1 -- 172.21.15.46:0/3085827482 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f6bf0053ee0 con 0x7f6c0006f7a0 2023-10-30T13:26:35.302 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:35.302 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:33.765225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:resolve","state_seq":13,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 22 2023-10-30T13:26:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6be00664a0 msgr2=0x7f6be0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6c04dc4640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6be00664a0 0x7f6be0068960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6c000c8610 tx=0x7f6bf4009040 comp rx=0 tx=0).stop 2023-10-30T13:26:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3085827482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0006f7a0 msgr2=0x7f6c00154b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6c04dc4640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0006f7a0 0x7f6c00154b50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000c420 tx=0x7f6bf000c8e0 comp rx=0 tx=0).stop 2023-10-30T13:26:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3085827482 shutdown_connections 2023-10-30T13:26:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6c04dc4640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6be00664a0 0x7f6be0068960 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6c04dc4640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c000c6bd0 0x7f6c000c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6c04dc4640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c00071110 0x7f6c00152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6c04dc4640 1 --2- 172.21.15.46:0/3085827482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0006f7a0 0x7f6c00154b50 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3085827482 >> 172.21.15.46:0/3085827482 conn(0x7f6c000c7910 msgr2=0x7f6c000c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3085827482 shutdown_connections 2023-10-30T13:26:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:35.298+0000 7f6c04dc4640 1 -- 172.21.15.46:0/3085827482 wait complete. 2023-10-30T13:26:35.323 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:26:37.323 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb56a8ea640 1 Processor -- start 2023-10-30T13:26:37.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb56a8ea640 1 -- start start 2023-10-30T13:26:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb56a8ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5641538a0 0x7fb56415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb56a8ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5640ca0e0 0x7fb56415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb56a8ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb5640cf160 0x7fb5640cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb56a8ea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5640cb800 con 0x7fb5640ca0e0 2023-10-30T13:26:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb56a8ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56415dc30 con 0x7fb5640cf160 2023-10-30T13:26:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb56a8ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5640cb330 con 0x7fb5641538a0 2023-10-30T13:26:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb5637fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5640ca0e0 0x7fb56415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb5637fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5640ca0e0 0x7fb56415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47196/0 (socket says 172.21.15.46:47196) 2023-10-30T13:26:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb5637fe640 1 -- 172.21.15.46:0/2853704832 learned_addr learned my addr 172.21.15.46:0/2853704832 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb563fff640 1 --2- 172.21.15.46:0/2853704832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5641538a0 0x7fb56415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb568e87640 1 --2- 172.21.15.46:0/2853704832 >> v2:172.21.15.154:3300/0 conn(0x7fb5640cf160 0x7fb5640cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb5637fe640 1 -- 172.21.15.46:0/2853704832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5641538a0 msgr2=0x7fb56415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb5637fe640 1 --2- 172.21.15.46:0/2853704832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5641538a0 0x7fb56415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb5637fe640 1 -- 172.21.15.46:0/2853704832 >> v2:172.21.15.154:3300/0 conn(0x7fb5640cf160 msgr2=0x7fb5640cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb5637fe640 1 --2- 172.21.15.46:0/2853704832 >> v2:172.21.15.154:3300/0 conn(0x7fb5640cf160 0x7fb5640cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb5637fe640 1 -- 172.21.15.46:0/2853704832 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5640cad70 con 0x7fb5640ca0e0 2023-10-30T13:26:37.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb563fff640 1 --2- 172.21.15.46:0/2853704832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5641538a0 0x7fb56415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:37.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.474+0000 7fb568e87640 1 --2- 172.21.15.46:0/2853704832 >> v2:172.21.15.154:3300/0 conn(0x7fb5640cf160 0x7fb5640cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb5637fe640 1 --2- 172.21.15.46:0/2853704832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640ca0e0 0x7fb56415a090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb56406cb00 tx=0x7fb55801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18b8c613b2f3bfe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb562ffd640 1 -- 172.21.15.46:0/2853704832 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5580a89b0 con 0x7fb5640ca0e0 2023-10-30T13:26:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb562ffd640 1 -- 172.21.15.46:0/2853704832 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5580a8b10 con 0x7fb5640ca0e0 2023-10-30T13:26:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb562ffd640 1 -- 172.21.15.46:0/2853704832 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5580a8ea0 con 0x7fb5640ca0e0 2023-10-30T13:26:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb56a8ea640 1 -- 172.21.15.46:0/2853704832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640ca0e0 msgr2=0x7fb56415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb56a8ea640 1 --2- 172.21.15.46:0/2853704832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640ca0e0 0x7fb56415a090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb56406cb00 tx=0x7fb55801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:26:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb56a8ea640 1 -- 172.21.15.46:0/2853704832 shutdown_connections 2023-10-30T13:26:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb56a8ea640 1 --2- 172.21.15.46:0/2853704832 >> v2:172.21.15.154:3300/0 conn(0x7fb5640cf160 0x7fb5640cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb56a8ea640 1 --2- 172.21.15.46:0/2853704832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640ca0e0 0x7fb56415a090 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb56a8ea640 1 --2- 172.21.15.46:0/2853704832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5641538a0 0x7fb56415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb56a8ea640 1 -- 172.21.15.46:0/2853704832 >> 172.21.15.46:0/2853704832 conn(0x7fb56406bfe0 msgr2=0x7fb56406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:37.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb56a8ea640 1 -- 172.21.15.46:0/2853704832 shutdown_connections 2023-10-30T13:26:37.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.478+0000 7fb56a8ea640 1 -- 172.21.15.46:0/2853704832 wait complete. 2023-10-30T13:26:37.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb56a8ea640 1 Processor -- start 2023-10-30T13:26:37.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb56a8ea640 1 -- start start 2023-10-30T13:26:37.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb56a8ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5640ca0e0 0x7fb5640b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:37.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb56a8ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640cf160 0x7fb5641483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:37.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb56a8ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5641538a0 0x7fb5641f7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:37.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb56a8ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56414f1d0 con 0x7fb5640cf160 2023-10-30T13:26:37.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb56a8ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5641f7060 con 0x7fb5641538a0 2023-10-30T13:26:37.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb56a8ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56414e0b0 con 0x7fb5640ca0e0 2023-10-30T13:26:37.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb5637fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640cf160 0x7fb5641483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:37.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb5637fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640cf160 0x7fb5641483f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47212/0 (socket says 172.21.15.46:47212) 2023-10-30T13:26:37.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb5637fe640 1 -- 172.21.15.46:0/975835164 learned_addr learned my addr 172.21.15.46:0/975835164 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:37.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb563fff640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5640ca0e0 0x7fb5640b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:37.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb568e87640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5641538a0 0x7fb5641f7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:37.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb5637fe640 1 -- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5640ca0e0 msgr2=0x7fb5640b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:37.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb5637fe640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5640ca0e0 0x7fb5640b44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:37.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb5637fe640 1 -- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5641538a0 msgr2=0x7fb5641f7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:37.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb5637fe640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5641538a0 0x7fb5641f7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:37.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb5637fe640 1 -- 172.21.15.46:0/975835164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5580094d0 con 0x7fb5640cf160 2023-10-30T13:26:37.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb563fff640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5640ca0e0 0x7fb5640b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:37.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb5637fe640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640cf160 0x7fb5641483f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb55801d3a0 tx=0x7fb55801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:37.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb560ff9640 1 -- 172.21.15.46:0/975835164 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5580ba600 con 0x7fb5640cf160 2023-10-30T13:26:37.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb560ff9640 1 -- 172.21.15.46:0/975835164 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5580ba760 con 0x7fb5640cf160 2023-10-30T13:26:37.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb56a8ea640 1 -- 172.21.15.46:0/975835164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56414e390 con 0x7fb5640cf160 2023-10-30T13:26:37.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb560ff9640 1 -- 172.21.15.46:0/975835164 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5580bab10 con 0x7fb5640cf160 2023-10-30T13:26:37.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.482+0000 7fb56a8ea640 1 -- 172.21.15.46:0/975835164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56414e950 con 0x7fb5640cf160 2023-10-30T13:26:37.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.486+0000 7fb56a8ea640 1 -- 172.21.15.46:0/975835164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5640b2a50 con 0x7fb5640cf160 2023-10-30T13:26:37.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.486+0000 7fb560ff9640 1 -- 172.21.15.46:0/975835164 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb5580bac70 con 0x7fb5640cf160 2023-10-30T13:26:37.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.486+0000 7fb560ff9640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5500664d0 0x7fb550068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:37.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.486+0000 7fb560ff9640 1 -- 172.21.15.46:0/975835164 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fb5581226b0 con 0x7fb5640cf160 2023-10-30T13:26:37.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.486+0000 7fb563fff640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5500664d0 0x7fb550068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:37.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.486+0000 7fb563fff640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5500664d0 0x7fb550068990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb55400b990 tx=0x7fb554008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:37.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.490+0000 7fb560ff9640 1 -- 172.21.15.46:0/975835164 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb5580ef940 con 0x7fb5640cf160 2023-10-30T13:26:37.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.678+0000 7fb56a8ea640 1 -- 172.21.15.46:0/975835164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb56414eb40 con 0x7fb5640cf160 2023-10-30T13:26:37.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.678+0000 7fb560ff9640 1 -- 172.21.15.46:0/975835164 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fb5580f2f60 con 0x7fb5640cf160 2023-10-30T13:26:37.686 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:37.686 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:33.765225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:resolve","state_seq":13,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:37.687 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 22 2023-10-30T13:26:37.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.682+0000 7fb56a8ea640 1 -- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5500664d0 msgr2=0x7fb550068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:37.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.682+0000 7fb56a8ea640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5500664d0 0x7fb550068990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb55400b990 tx=0x7fb554008040 comp rx=0 tx=0).stop 2023-10-30T13:26:37.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.682+0000 7fb56a8ea640 1 -- 172.21.15.46:0/975835164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640cf160 msgr2=0x7fb5641483f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:37.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.682+0000 7fb56a8ea640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640cf160 0x7fb5641483f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb55801d3a0 tx=0x7fb55801ba90 comp rx=0 tx=0).stop 2023-10-30T13:26:37.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.682+0000 7fb56a8ea640 1 -- 172.21.15.46:0/975835164 shutdown_connections 2023-10-30T13:26:37.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.682+0000 7fb56a8ea640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5500664d0 0x7fb550068990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:37.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.682+0000 7fb56a8ea640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5641538a0 0x7fb5641f7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:37.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.682+0000 7fb56a8ea640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5640cf160 0x7fb5641483f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:37.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.682+0000 7fb56a8ea640 1 --2- 172.21.15.46:0/975835164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5640ca0e0 0x7fb5640b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:37.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.682+0000 7fb56a8ea640 1 -- 172.21.15.46:0/975835164 >> 172.21.15.46:0/975835164 conn(0x7fb56406bfe0 msgr2=0x7fb564065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:37.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.682+0000 7fb56a8ea640 1 -- 172.21.15.46:0/975835164 shutdown_connections 2023-10-30T13:26:37.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:37.682+0000 7fb56a8ea640 1 -- 172.21.15.46:0/975835164 wait complete. 2023-10-30T13:26:37.706 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:26:39.707 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:39.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.858+0000 7fa60d71d640 1 Processor -- start 2023-10-30T13:26:39.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.858+0000 7fa60d71d640 1 -- start start 2023-10-30T13:26:39.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.862+0000 7fa60d71d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa60806f7a0 0x7fa60806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:39.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.862+0000 7fa60d71d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa608066cf0 con 0x7fa608071110 2023-10-30T13:26:39.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.862+0000 7fa60d71d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6080c70d0 con 0x7fa6080c6bd0 2023-10-30T13:26:39.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.862+0000 7fa60d71d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6080700c0 con 0x7fa60806f7a0 2023-10-30T13:26:39.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.862+0000 7fa606ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa60806f7a0 0x7fa60806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:39.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.862+0000 7fa6077fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa608071110 0x7fa608158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56956/0 (socket says 172.21.15.46:56956) 2023-10-30T13:26:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.862+0000 7fa6077fe640 1 -- 172.21.15.46:0/1143364564 learned_addr learned my addr 172.21.15.46:0/1143364564 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:26:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.862+0000 7fa606ffd640 1 -- 172.21.15.46:0/1143364564 >> v1:172.21.15.154:6789/0 conn(0x7fa6080c6bd0 legacy=0x7fa608069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.862+0000 7fa606ffd640 1 -- 172.21.15.46:0/1143364564 >> v1:172.21.15.46:6789/0 conn(0x7fa608071110 legacy=0x7fa608158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.862+0000 7fa606ffd640 1 -- 172.21.15.46:0/1143364564 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6080c7390 con 0x7fa60806f7a0 2023-10-30T13:26:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.862+0000 7fa605ffb640 1 -- 172.21.15.46:0/1143364564 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441461219 0 0) 0x7fa608066cf0 con 0x7fa608071110 2023-10-30T13:26:39.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.862+0000 7fa606ffd640 1 --2- 172.21.15.46:0/1143364564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa60806f7a0 0x7fa60806fb80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80097c0 tx=0x7fa5f801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37b99f6a1897de server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:39.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.866+0000 7fa605ffb640 1 -- 172.21.15.46:0/1143364564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f80a8970 con 0x7fa60806f7a0 2023-10-30T13:26:39.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.866+0000 7fa605ffb640 1 -- 172.21.15.46:0/1143364564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f80a8ad0 con 0x7fa60806f7a0 2023-10-30T13:26:39.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.866+0000 7fa605ffb640 1 -- 172.21.15.46:0/1143364564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f80a8e60 con 0x7fa60806f7a0 2023-10-30T13:26:39.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.866+0000 7fa60d71d640 1 -- 172.21.15.46:0/1143364564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa60806f7a0 msgr2=0x7fa60806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:39.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.866+0000 7fa60d71d640 1 --2- 172.21.15.46:0/1143364564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa60806f7a0 0x7fa60806fb80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80097c0 tx=0x7fa5f801ec70 comp rx=0 tx=0).stop 2023-10-30T13:26:39.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.866+0000 7fa60d71d640 1 -- 172.21.15.46:0/1143364564 shutdown_connections 2023-10-30T13:26:39.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.866+0000 7fa60d71d640 1 --2- 172.21.15.46:0/1143364564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa60806f7a0 0x7fa60806fb80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:39.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.866+0000 7fa60d71d640 1 -- 172.21.15.46:0/1143364564 >> 172.21.15.46:0/1143364564 conn(0x7fa6080c7910 msgr2=0x7fa6080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:39.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.866+0000 7fa60d71d640 1 -- 172.21.15.46:0/1143364564 shutdown_connections 2023-10-30T13:26:39.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.866+0000 7fa60d71d640 1 -- 172.21.15.46:0/1143364564 wait complete. 2023-10-30T13:26:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.866+0000 7fa60d71d640 1 Processor -- start 2023-10-30T13:26:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.866+0000 7fa60d71d640 1 -- start start 2023-10-30T13:26:39.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa60d71d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa60806f7a0 0x7fa6080c1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:39.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa60d71d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa608071110 0x7fa6080c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:39.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa60d71d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080c6bd0 0x7fa6081e3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:39.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa60d71d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6081e2bc0 con 0x7fa6080c6bd0 2023-10-30T13:26:39.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa60d71d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6081e2ea0 con 0x7fa608071110 2023-10-30T13:26:39.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa60d71d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6081ef7c0 con 0x7fa60806f7a0 2023-10-30T13:26:39.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa6077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080c6bd0 0x7fa6081e3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:39.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa6077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080c6bd0 0x7fa6081e3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47234/0 (socket says 172.21.15.46:47234) 2023-10-30T13:26:39.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa6077fe640 1 -- 172.21.15.46:0/2601784830 learned_addr learned my addr 172.21.15.46:0/2601784830 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:39.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa6067fc640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa608071110 0x7fa6080c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:39.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa606ffd640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa60806f7a0 0x7fa6080c1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:39.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa6077fe640 1 -- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa60806f7a0 msgr2=0x7fa6080c1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:39.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa6077fe640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa60806f7a0 0x7fa6080c1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:39.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa6077fe640 1 -- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa608071110 msgr2=0x7fa6080c0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:39.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa6077fe640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa608071110 0x7fa6080c0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:39.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa6077fe640 1 -- 172.21.15.46:0/2601784830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f8009420 con 0x7fa6080c6bd0 2023-10-30T13:26:39.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa6077fe640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080c6bd0 0x7fa6081e3330 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa60000c730 tx=0x7fa60000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:39.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa5ef7fe640 1 -- 172.21.15.46:0/2601784830 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60000ce40 con 0x7fa6080c6bd0 2023-10-30T13:26:39.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa5ef7fe640 1 -- 172.21.15.46:0/2601784830 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60000a5e0 con 0x7fa6080c6bd0 2023-10-30T13:26:39.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa5ef7fe640 1 -- 172.21.15.46:0/2601784830 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60000a9d0 con 0x7fa6080c6bd0 2023-10-30T13:26:39.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa60d71d640 1 -- 172.21.15.46:0/2601784830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6081efa80 con 0x7fa6080c6bd0 2023-10-30T13:26:39.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa60d71d640 1 -- 172.21.15.46:0/2601784830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6081e1f50 con 0x7fa6080c6bd0 2023-10-30T13:26:39.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa60d71d640 1 -- 172.21.15.46:0/2601784830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60804e170 con 0x7fa6080c6bd0 2023-10-30T13:26:39.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.870+0000 7fa5ef7fe640 1 -- 172.21.15.46:0/2601784830 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa60000ab30 con 0x7fa6080c6bd0 2023-10-30T13:26:39.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.874+0000 7fa5ef7fe640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5f40664a0 0x7fa5f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:39.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.874+0000 7fa5ef7fe640 1 -- 172.21.15.46:0/2601784830 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fa600083390 con 0x7fa6080c6bd0 2023-10-30T13:26:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.874+0000 7fa606ffd640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5f40664a0 0x7fa5f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.874+0000 7fa606ffd640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5f40664a0 0x7fa5f4068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80b2000 tx=0x7fa5f80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:39.878+0000 7fa5ef7fe640 1 -- 172.21.15.46:0/2601784830 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa600050780 con 0x7fa6080c6bd0 2023-10-30T13:26:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.070+0000 7fa60d71d640 1 -- 172.21.15.46:0/2601784830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6081e2fe0 con 0x7fa6080c6bd0 2023-10-30T13:26:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.070+0000 7fa5ef7fe640 1 -- 172.21.15.46:0/2601784830 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fa600018090 con 0x7fa6080c6bd0 2023-10-30T13:26:40.077 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:40.077 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:33.765225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:resolve","state_seq":13,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 22 2023-10-30T13:26:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.074+0000 7fa60d71d640 1 -- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5f40664a0 msgr2=0x7fa5f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.074+0000 7fa60d71d640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5f40664a0 0x7fa5f4068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80b2000 tx=0x7fa5f80aa070 comp rx=0 tx=0).stop 2023-10-30T13:26:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.074+0000 7fa60d71d640 1 -- 172.21.15.46:0/2601784830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080c6bd0 msgr2=0x7fa6081e3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.074+0000 7fa60d71d640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080c6bd0 0x7fa6081e3330 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa60000c730 tx=0x7fa60000cbf0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.074+0000 7fa60d71d640 1 -- 172.21.15.46:0/2601784830 shutdown_connections 2023-10-30T13:26:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.074+0000 7fa60d71d640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5f40664a0 0x7fa5f4068960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.074+0000 7fa60d71d640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080c6bd0 0x7fa6081e3330 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.074+0000 7fa60d71d640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa608071110 0x7fa6080c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.074+0000 7fa60d71d640 1 --2- 172.21.15.46:0/2601784830 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa60806f7a0 0x7fa6080c1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.074+0000 7fa60d71d640 1 -- 172.21.15.46:0/2601784830 >> 172.21.15.46:0/2601784830 conn(0x7fa6080c7910 msgr2=0x7fa6080c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:40.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.074+0000 7fa60d71d640 1 -- 172.21.15.46:0/2601784830 shutdown_connections 2023-10-30T13:26:40.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:40.074+0000 7fa60d71d640 1 -- 172.21.15.46:0/2601784830 wait complete. 2023-10-30T13:26:40.098 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:26:42.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:42.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdfff205640 1 Processor -- start 2023-10-30T13:26:42.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdfff205640 1 -- start start 2023-10-30T13:26:42.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdfff205640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdff81538a0 0x7fdff815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:42.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdfff205640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdff80ca0e0 0x7fdff815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:42.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdfff205640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff80cb800 con 0x7fdff80cf160 2023-10-30T13:26:42.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdfff205640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff815dc30 con 0x7fdff80ca0e0 2023-10-30T13:26:42.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdfff205640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff80cb330 con 0x7fdff81538a0 2023-10-30T13:26:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdffcfa1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdff81538a0 0x7fdff815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdff7fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdff80ca0e0 0x7fdff815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:42.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdff7fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdff80ca0e0 0x7fdff815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:53874/0 (socket says 172.21.15.46:53874) 2023-10-30T13:26:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdff7fff640 1 -- 172.21.15.46:0/598464327 learned_addr learned my addr 172.21.15.46:0/598464327 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdff7fff640 1 -- 172.21.15.46:0/598464327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdff81538a0 msgr2=0x7fdff815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdff7fff640 1 --2- 172.21.15.46:0/598464327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdff81538a0 0x7fdff815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdff7fff640 1 -- 172.21.15.46:0/598464327 >> v1:172.21.15.154:6789/0 conn(0x7fdff80cf160 legacy=0x7fdff80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:42.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.250+0000 7fdff7fff640 1 -- 172.21.15.46:0/598464327 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff80cad70 con 0x7fdff80ca0e0 2023-10-30T13:26:42.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.254+0000 7fdff7fff640 1 --2- 172.21.15.46:0/598464327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80ca0e0 0x7fdff815a090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009870 tx=0x7fdfe401ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e108533dcf04413d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.254+0000 7fdff77fe640 1 -- 172.21.15.46:0/598464327 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe40a8a30 con 0x7fdff80ca0e0 2023-10-30T13:26:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.254+0000 7fdff77fe640 1 -- 172.21.15.46:0/598464327 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe40a8b90 con 0x7fdff80ca0e0 2023-10-30T13:26:42.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.254+0000 7fdff77fe640 1 -- 172.21.15.46:0/598464327 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe40b0440 con 0x7fdff80ca0e0 2023-10-30T13:26:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.254+0000 7fdfff205640 1 -- 172.21.15.46:0/598464327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80ca0e0 msgr2=0x7fdff815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.254+0000 7fdfff205640 1 --2- 172.21.15.46:0/598464327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80ca0e0 0x7fdff815a090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009870 tx=0x7fdfe401ed30 comp rx=0 tx=0).stop 2023-10-30T13:26:42.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.254+0000 7fdfff205640 1 -- 172.21.15.46:0/598464327 shutdown_connections 2023-10-30T13:26:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.254+0000 7fdfff205640 1 --2- 172.21.15.46:0/598464327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80ca0e0 0x7fdff815a090 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.254+0000 7fdfff205640 1 --2- 172.21.15.46:0/598464327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdff81538a0 0x7fdff815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.254+0000 7fdfff205640 1 -- 172.21.15.46:0/598464327 >> 172.21.15.46:0/598464327 conn(0x7fdff806bfe0 msgr2=0x7fdff806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.254+0000 7fdfff205640 1 -- 172.21.15.46:0/598464327 shutdown_connections 2023-10-30T13:26:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.254+0000 7fdfff205640 1 -- 172.21.15.46:0/598464327 wait complete. 2023-10-30T13:26:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdfff205640 1 Processor -- start 2023-10-30T13:26:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdfff205640 1 -- start start 2023-10-30T13:26:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdfff205640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdff80ca0e0 0x7fdff81f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdfff205640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80cf160 0x7fdff80cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdfff205640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdff81538a0 0x7fdff815cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdfff205640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff80b3e30 con 0x7fdff80ca0e0 2023-10-30T13:26:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdfff205640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff80b39c0 con 0x7fdff80cf160 2023-10-30T13:26:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdfff205640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff80b3720 con 0x7fdff81538a0 2023-10-30T13:26:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdffcfa1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdff80ca0e0 0x7fdff81f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdffcfa1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdff80ca0e0 0x7fdff81f25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47238/0 (socket says 172.21.15.46:47238) 2023-10-30T13:26:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdffcfa1640 1 -- 172.21.15.46:0/1819009091 learned_addr learned my addr 172.21.15.46:0/1819009091 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdff7fff640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80cf160 0x7fdff80cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdffd7a2640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdff81538a0 0x7fdff815cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdffcfa1640 1 -- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdff81538a0 msgr2=0x7fdff815cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdffcfa1640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdff81538a0 0x7fdff815cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdffcfa1640 1 -- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80cf160 msgr2=0x7fdff80cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdffcfa1640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80cf160 0x7fdff80cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdffcfa1640 1 -- 172.21.15.46:0/1819009091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe40094d0 con 0x7fdff80ca0e0 2023-10-30T13:26:42.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdffcfa1640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdff80ca0e0 0x7fdff81f25d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00e870 tx=0x7fdfec00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdff57fa640 1 -- 172.21.15.46:0/1819009091 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec009510 con 0x7fdff80ca0e0 2023-10-30T13:26:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdff57fa640 1 -- 172.21.15.46:0/1819009091 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfec00eec0 con 0x7fdff80ca0e0 2023-10-30T13:26:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdff57fa640 1 -- 172.21.15.46:0/1819009091 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec00cb40 con 0x7fdff80ca0e0 2023-10-30T13:26:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdfff205640 1 -- 172.21.15.46:0/1819009091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff81f2f10 con 0x7fdff80ca0e0 2023-10-30T13:26:42.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.258+0000 7fdfff205640 1 -- 172.21.15.46:0/1819009091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff80b42c0 con 0x7fdff80ca0e0 2023-10-30T13:26:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.262+0000 7fdfff205640 1 -- 172.21.15.46:0/1819009091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff80b2a50 con 0x7fdff80ca0e0 2023-10-30T13:26:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.262+0000 7fdff57fa640 1 -- 172.21.15.46:0/1819009091 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdfec00cca0 con 0x7fdff80ca0e0 2023-10-30T13:26:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.262+0000 7fdff57fa640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdfdc0664a0 0x7fdfdc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.262+0000 7fdff57fa640 1 -- 172.21.15.46:0/1819009091 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fdfec084170 con 0x7fdff80ca0e0 2023-10-30T13:26:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.262+0000 7fdff7fff640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdfdc0664a0 0x7fdfdc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.262+0000 7fdff7fff640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdfdc0664a0 0x7fdfdc068960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40b2000 tx=0x7fdfe4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.266+0000 7fdff57fa640 1 -- 172.21.15.46:0/1819009091 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdfec051430 con 0x7fdff80ca0e0 2023-10-30T13:26:42.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.458+0000 7fdfff205640 1 -- 172.21.15.46:0/1819009091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdff804e170 con 0x7fdff80ca0e0 2023-10-30T13:26:42.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.458+0000 7fdff57fa640 1 -- 172.21.15.46:0/1819009091 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fdfec054a50 con 0x7fdff80ca0e0 2023-10-30T13:26:42.464 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:42.464 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:39.732409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2023-10-30T13:26:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.458+0000 7fdfff205640 1 -- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdfdc0664a0 msgr2=0x7fdfdc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.458+0000 7fdfff205640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdfdc0664a0 0x7fdfdc068960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40b2000 tx=0x7fdfe4009040 comp rx=0 tx=0).stop 2023-10-30T13:26:42.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.458+0000 7fdfff205640 1 -- 172.21.15.46:0/1819009091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdff80ca0e0 msgr2=0x7fdff81f25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:42.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.458+0000 7fdfff205640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdff80ca0e0 0x7fdff81f25d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00e870 tx=0x7fdfec00ec70 comp rx=0 tx=0).stop 2023-10-30T13:26:42.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.462+0000 7fdfff205640 1 -- 172.21.15.46:0/1819009091 shutdown_connections 2023-10-30T13:26:42.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.462+0000 7fdfff205640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdfdc0664a0 0x7fdfdc068960 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.462+0000 7fdfff205640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdff81538a0 0x7fdff815cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.462+0000 7fdfff205640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80cf160 0x7fdff80cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.462+0000 7fdfff205640 1 --2- 172.21.15.46:0/1819009091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdff80ca0e0 0x7fdff81f25d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.462+0000 7fdfff205640 1 -- 172.21.15.46:0/1819009091 >> 172.21.15.46:0/1819009091 conn(0x7fdff806bfe0 msgr2=0x7fdff806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.462+0000 7fdfff205640 1 -- 172.21.15.46:0/1819009091 shutdown_connections 2023-10-30T13:26:42.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:42.462+0000 7fdfff205640 1 -- 172.21.15.46:0/1819009091 wait complete. 2023-10-30T13:26:42.485 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4288": { "addr": "172.21.15.154:6835/3109262047", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3109262047, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3109262047, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4306": { "addr": "172.21.15.46:6836/275749119", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 275749119, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 275749119, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4950": { "addr": "172.21.15.46:6837/1315246492", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 1315246492, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 1315246492, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4950, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:reconnect", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:26:39.732409+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4288, "mds_2": 4950 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:26:44.094 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:44.099 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:44.487 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:44.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.634+0000 7f02ce4d6640 1 Processor -- start 2023-10-30T13:26:44.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.634+0000 7f02ce4d6640 1 -- start start 2023-10-30T13:26:44.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.638+0000 7f02ce4d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02c80c6f60 0x7f02c81566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:44.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.638+0000 7f02ce4d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c80c0a50 con 0x7f02c80c6b80 2023-10-30T13:26:44.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.638+0000 7f02ce4d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c806f6e0 con 0x7f02c80c05d0 2023-10-30T13:26:44.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.638+0000 7f02ce4d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c806f9c0 con 0x7f02c80c6f60 2023-10-30T13:26:44.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.638+0000 7f02c7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f02c80c6b80 0x7f02c80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56962/0 (socket says 172.21.15.46:56962) 2023-10-30T13:26:44.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.638+0000 7f02c7fff640 1 -- 172.21.15.46:0/1755539440 learned_addr learned my addr 172.21.15.46:0/1755539440 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:26:44.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.638+0000 7f02cca73640 1 --2- 172.21.15.46:0/1755539440 >> v2:172.21.15.46:3300/0 conn(0x7f02c80c6f60 0x7f02c81566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:44.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.638+0000 7f02cca73640 1 -- 172.21.15.46:0/1755539440 >> v1:172.21.15.154:6789/0 conn(0x7f02c80c05d0 legacy=0x7f02c813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:44.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.638+0000 7f02cca73640 1 -- 172.21.15.46:0/1755539440 >> v1:172.21.15.46:6789/0 conn(0x7f02c80c6b80 legacy=0x7f02c80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:44.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.638+0000 7f02cca73640 1 -- 172.21.15.46:0/1755539440 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c806fca0 con 0x7f02c80c6f60 2023-10-30T13:26:44.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.638+0000 7f02c6ffd640 1 -- 172.21.15.46:0/1755539440 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755607006 0 0) 0x7f02c80c0a50 con 0x7f02c80c6b80 2023-10-30T13:26:44.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02cca73640 1 --2- 172.21.15.46:0/1755539440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02c80c6f60 0x7f02c81566a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009d90 tx=0x7f02c001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6016b8f9673a8dc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:44.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02c6ffd640 1 -- 172.21.15.46:0/1755539440 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c00a5920 con 0x7f02c80c6f60 2023-10-30T13:26:44.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02c6ffd640 1 -- 172.21.15.46:0/1755539440 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c00a5a80 con 0x7f02c80c6f60 2023-10-30T13:26:44.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02c6ffd640 1 -- 172.21.15.46:0/1755539440 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c00a5e10 con 0x7f02c80c6f60 2023-10-30T13:26:44.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02ce4d6640 1 -- 172.21.15.46:0/1755539440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02c80c6f60 msgr2=0x7f02c81566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:44.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02ce4d6640 1 --2- 172.21.15.46:0/1755539440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02c80c6f60 0x7f02c81566a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009d90 tx=0x7f02c001c950 comp rx=0 tx=0).stop 2023-10-30T13:26:44.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02ce4d6640 1 -- 172.21.15.46:0/1755539440 shutdown_connections 2023-10-30T13:26:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02ce4d6640 1 --2- 172.21.15.46:0/1755539440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02c80c6f60 0x7f02c81566a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02ce4d6640 1 -- 172.21.15.46:0/1755539440 >> 172.21.15.46:0/1755539440 conn(0x7f02c80c7910 msgr2=0x7f02c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02ce4d6640 1 -- 172.21.15.46:0/1755539440 shutdown_connections 2023-10-30T13:26:44.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02ce4d6640 1 -- 172.21.15.46:0/1755539440 wait complete. 2023-10-30T13:26:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02ce4d6640 1 Processor -- start 2023-10-30T13:26:44.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.642+0000 7f02ce4d6640 1 -- start start 2023-10-30T13:26:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02ce4d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02c80c05d0 0x7f02c81536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02ce4d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02c80c6b80 0x7f02c80c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02ce4d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c80c6f60 0x7f02c80c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02ce4d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c8137670 con 0x7f02c80c6b80 2023-10-30T13:26:44.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02ce4d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c8137a80 con 0x7f02c80c6f60 2023-10-30T13:26:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02ce4d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c8153c00 con 0x7f02c80c05d0 2023-10-30T13:26:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02c80c6b80 0x7f02c80c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02c80c6b80 0x7f02c80c92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47254/0 (socket says 172.21.15.46:47254) 2023-10-30T13:26:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c77fe640 1 -- 172.21.15.46:0/782583879 learned_addr learned my addr 172.21.15.46:0/782583879 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:44.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02cca73640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c80c6f60 0x7f02c80c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c7fff640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02c80c05d0 0x7f02c81536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c77fe640 1 -- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02c80c05d0 msgr2=0x7f02c81536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c77fe640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02c80c05d0 0x7f02c81536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c77fe640 1 -- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c80c6f60 msgr2=0x7f02c80c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c77fe640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c80c6f60 0x7f02c80c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c77fe640 1 -- 172.21.15.46:0/782583879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c0009a70 con 0x7f02c80c6b80 2023-10-30T13:26:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02cca73640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c80c6f60 0x7f02c80c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c7fff640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02c80c05d0 0x7f02c81536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c77fe640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02c80c6b80 0x7f02c80c92d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00c4d0 tx=0x7f02bc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c4ff9640 1 -- 172.21.15.46:0/782583879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc0052d0 con 0x7f02c80c6b80 2023-10-30T13:26:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c4ff9640 1 -- 172.21.15.46:0/782583879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02bc005430 con 0x7f02c80c6b80 2023-10-30T13:26:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02ce4d6640 1 -- 172.21.15.46:0/782583879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c80c63c0 con 0x7f02c80c6b80 2023-10-30T13:26:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c4ff9640 1 -- 172.21.15.46:0/782583879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc012830 con 0x7f02c80c6b80 2023-10-30T13:26:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02ce4d6640 1 -- 172.21.15.46:0/782583879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c8137390 con 0x7f02c80c6b80 2023-10-30T13:26:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02ce4d6640 1 -- 172.21.15.46:0/782583879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c8072180 con 0x7f02c80c6b80 2023-10-30T13:26:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.646+0000 7f02c4ff9640 1 -- 172.21.15.46:0/782583879 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02bc012990 con 0x7f02c80c6b80 2023-10-30T13:26:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.650+0000 7f02c4ff9640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02b40664a0 0x7f02b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.650+0000 7f02c4ff9640 1 -- 172.21.15.46:0/782583879 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f02bc083b80 con 0x7f02c80c6b80 2023-10-30T13:26:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.650+0000 7f02c7fff640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02b40664a0 0x7f02b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.650+0000 7f02c7fff640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02b40664a0 0x7f02b4068960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f02b8000f70 tx=0x7f02b8007c60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.654+0000 7f02c4ff9640 1 -- 172.21.15.46:0/782583879 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f02bc050e20 con 0x7f02c80c6b80 2023-10-30T13:26:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 -- 172.21.15.46:0/782583879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02c804e170 con 0x7f02c80c6b80 2023-10-30T13:26:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02c4ff9640 1 -- 172.21.15.46:0/782583879 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f02bc054440 con 0x7f02c80c6b80 2023-10-30T13:26:44.850 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:44.851 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:39.732409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2023-10-30T13:26:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 -- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02b40664a0 msgr2=0x7f02b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02b40664a0 0x7f02b4068960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f02b8000f70 tx=0x7f02b8007c60 comp rx=0 tx=0).stop 2023-10-30T13:26:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 -- 172.21.15.46:0/782583879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02c80c6b80 msgr2=0x7f02c80c92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02c80c6b80 0x7f02c80c92d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00c4d0 tx=0x7f02bc00c990 comp rx=0 tx=0).stop 2023-10-30T13:26:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 -- 172.21.15.46:0/782583879 shutdown_connections 2023-10-30T13:26:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02b40664a0 0x7f02b4068960 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c80c6f60 0x7f02c80c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02c80c6b80 0x7f02c80c92d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 --2- 172.21.15.46:0/782583879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02c80c05d0 0x7f02c81536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 -- 172.21.15.46:0/782583879 >> 172.21.15.46:0/782583879 conn(0x7f02c80c7910 msgr2=0x7f02c80bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 -- 172.21.15.46:0/782583879 shutdown_connections 2023-10-30T13:26:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:44.846+0000 7f02ce4d6640 1 -- 172.21.15.46:0/782583879 wait complete. 2023-10-30T13:26:44.871 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:26:46.872 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:47.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43b5b72640 1 Processor -- start 2023-10-30T13:26:47.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43b5b72640 1 -- start start 2023-10-30T13:26:47.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43b5b72640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43b00c6bd0 0x7f43b006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:47.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43b5b72640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b0066cf0 con 0x7f43b006f7a0 2023-10-30T13:26:47.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43b5b72640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b00c70d0 con 0x7f43b0071110 2023-10-30T13:26:47.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43b5b72640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b00700c0 con 0x7f43b00c6bd0 2023-10-30T13:26:47.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43af7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f43b006f7a0 0x7f43b0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35992/0 (socket says 172.21.15.46:35992) 2023-10-30T13:26:47.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43af7fe640 1 -- 172.21.15.46:0/2415581707 learned_addr learned my addr 172.21.15.46:0/2415581707 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:26:47.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43aeffd640 1 --2- 172.21.15.46:0/2415581707 >> v2:172.21.15.46:3300/0 conn(0x7f43b00c6bd0 0x7f43b006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:47.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43ae7fc640 1 -- 172.21.15.46:0/2415581707 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110948316 0 0) 0x7f43b0066cf0 con 0x7f43b006f7a0 2023-10-30T13:26:47.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43ae7fc640 1 -- 172.21.15.46:0/2415581707 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4398003530 con 0x7f43b006f7a0 2023-10-30T13:26:47.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43aeffd640 1 -- 172.21.15.46:0/2415581707 >> v1:172.21.15.154:6789/0 conn(0x7f43b0071110 legacy=0x7f43b0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:47.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43aeffd640 1 -- 172.21.15.46:0/2415581707 >> v1:172.21.15.46:6789/0 conn(0x7f43b006f7a0 legacy=0x7f43b0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:47.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.022+0000 7f43aeffd640 1 -- 172.21.15.46:0/2415581707 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b00c7390 con 0x7f43b00c6bd0 2023-10-30T13:26:47.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.026+0000 7f43aeffd640 1 --2- 172.21.15.46:0/2415581707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b00c6bd0 0x7f43b006fb80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f43a40097c0 tx=0x7f43a401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98872c4342f29ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:47.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.026+0000 7f43ae7fc640 1 -- 172.21.15.46:0/2415581707 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a40a8970 con 0x7f43b00c6bd0 2023-10-30T13:26:47.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.026+0000 7f43ae7fc640 1 -- 172.21.15.46:0/2415581707 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a40a8ad0 con 0x7f43b00c6bd0 2023-10-30T13:26:47.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.026+0000 7f43ae7fc640 1 -- 172.21.15.46:0/2415581707 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a40a8e60 con 0x7f43b00c6bd0 2023-10-30T13:26:47.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.026+0000 7f43b5b72640 1 -- 172.21.15.46:0/2415581707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b00c6bd0 msgr2=0x7f43b006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:47.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.026+0000 7f43b5b72640 1 --2- 172.21.15.46:0/2415581707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b00c6bd0 0x7f43b006fb80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f43a40097c0 tx=0x7f43a401ec70 comp rx=0 tx=0).stop 2023-10-30T13:26:47.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.026+0000 7f43b5b72640 1 -- 172.21.15.46:0/2415581707 shutdown_connections 2023-10-30T13:26:47.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.026+0000 7f43b5b72640 1 --2- 172.21.15.46:0/2415581707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b00c6bd0 0x7f43b006fb80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:47.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.026+0000 7f43b5b72640 1 -- 172.21.15.46:0/2415581707 >> 172.21.15.46:0/2415581707 conn(0x7f43b00c7910 msgr2=0x7f43b00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:47.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.026+0000 7f43b5b72640 1 -- 172.21.15.46:0/2415581707 shutdown_connections 2023-10-30T13:26:47.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.026+0000 7f43b5b72640 1 -- 172.21.15.46:0/2415581707 wait complete. 2023-10-30T13:26:47.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43b5b72640 1 Processor -- start 2023-10-30T13:26:47.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43b5b72640 1 -- start start 2023-10-30T13:26:47.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43b5b72640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b006f7a0 0x7f43b014edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:47.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43b5b72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b0071110 0x7f43b0154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:47.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43b5b72640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43b00c6bd0 0x7f43b0154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:47.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43b5b72640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b01e6890 con 0x7f43b0071110 2023-10-30T13:26:47.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43b5b72640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b014f300 con 0x7f43b006f7a0 2023-10-30T13:26:47.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43b5b72640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b01e61a0 con 0x7f43b00c6bd0 2023-10-30T13:26:47.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43aeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b0071110 0x7f43b0154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:47.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43aeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b0071110 0x7f43b0154590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44146/0 (socket says 172.21.15.46:44146) 2023-10-30T13:26:47.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43aeffd640 1 -- 172.21.15.46:0/3638921363 learned_addr learned my addr 172.21.15.46:0/3638921363 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:47.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43af7fe640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b006f7a0 0x7f43b014edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:47.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43affff640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43b00c6bd0 0x7f43b0154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:47.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43aeffd640 1 -- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43b00c6bd0 msgr2=0x7f43b0154ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:47.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43aeffd640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43b00c6bd0 0x7f43b0154ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:47.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43aeffd640 1 -- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b006f7a0 msgr2=0x7f43b014edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:47.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43aeffd640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b006f7a0 0x7f43b014edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:47.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43aeffd640 1 -- 172.21.15.46:0/3638921363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a4009420 con 0x7f43b0071110 2023-10-30T13:26:47.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43aeffd640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b0071110 0x7f43b0154590 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f43a401b880 tx=0x7f43a401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f4397fff640 1 -- 172.21.15.46:0/3638921363 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a40bb640 con 0x7f43b0071110 2023-10-30T13:26:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f4397fff640 1 -- 172.21.15.46:0/3638921363 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a40bb7c0 con 0x7f43b0071110 2023-10-30T13:26:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f4397fff640 1 -- 172.21.15.46:0/3638921363 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a40bbb70 con 0x7f43b0071110 2023-10-30T13:26:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43b5b72640 1 -- 172.21.15.46:0/3638921363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b00c7cf0 con 0x7f43b0071110 2023-10-30T13:26:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.030+0000 7f43b5b72640 1 -- 172.21.15.46:0/3638921363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b01ef380 con 0x7f43b0071110 2023-10-30T13:26:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.034+0000 7f43b5b72640 1 -- 172.21.15.46:0/3638921363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b01ef510 con 0x7f43b0071110 2023-10-30T13:26:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.034+0000 7f4397fff640 1 -- 172.21.15.46:0/3638921363 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f43a40b7040 con 0x7f43b0071110 2023-10-30T13:26:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.034+0000 7f4397fff640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f439c0664d0 0x7f439c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.034+0000 7f4397fff640 1 -- 172.21.15.46:0/3638921363 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f43a4122370 con 0x7f43b0071110 2023-10-30T13:26:47.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.034+0000 7f43af7fe640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f439c0664d0 0x7f439c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:47.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.034+0000 7f43af7fe640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f439c0664d0 0x7f439c068990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f43a00011f0 tx=0x7f43a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:47.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.038+0000 7f4397fff640 1 -- 172.21.15.46:0/3638921363 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f43a40ef610 con 0x7f43b0071110 2023-10-30T13:26:47.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.226+0000 7f43b5b72640 1 -- 172.21.15.46:0/3638921363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43b004e170 con 0x7f43b0071110 2023-10-30T13:26:47.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f4397fff640 1 -- 172.21.15.46:0/3638921363 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f43a40f2c30 con 0x7f43b0071110 2023-10-30T13:26:47.234 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:47.235 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:45.740902+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2023-10-30T13:26:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f43b5b72640 1 -- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f439c0664d0 msgr2=0x7f439c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f43b5b72640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f439c0664d0 0x7f439c068990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f43a00011f0 tx=0x7f43a0009040 comp rx=0 tx=0).stop 2023-10-30T13:26:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f43b5b72640 1 -- 172.21.15.46:0/3638921363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b0071110 msgr2=0x7f43b0154590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f43b5b72640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b0071110 0x7f43b0154590 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f43a401b880 tx=0x7f43a401ba10 comp rx=0 tx=0).stop 2023-10-30T13:26:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f43b5b72640 1 -- 172.21.15.46:0/3638921363 shutdown_connections 2023-10-30T13:26:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f43b5b72640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f439c0664d0 0x7f439c068990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f43b5b72640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43b00c6bd0 0x7f43b0154ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f43b5b72640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b0071110 0x7f43b0154590 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f43b5b72640 1 --2- 172.21.15.46:0/3638921363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b006f7a0 0x7f43b014edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f43b5b72640 1 -- 172.21.15.46:0/3638921363 >> 172.21.15.46:0/3638921363 conn(0x7f43b00c7910 msgr2=0x7f43b00c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f43b5b72640 1 -- 172.21.15.46:0/3638921363 shutdown_connections 2023-10-30T13:26:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:47.230+0000 7f43b5b72640 1 -- 172.21.15.46:0/3638921363 wait complete. 2023-10-30T13:26:47.255 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:26:49.256 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.406+0000 7fa817d01640 1 Processor -- start 2023-10-30T13:26:49.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.406+0000 7fa817d01640 1 -- start start 2023-10-30T13:26:49.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.406+0000 7fa817d01640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa81006e470 0x7fa81006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.406+0000 7fa817d01640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8100c8490 0x7fa8100c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.406+0000 7fa817d01640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81006afa0 con 0x7fa810153d70 2023-10-30T13:26:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.406+0000 7fa817d01640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa81006d780 con 0x7fa81006e470 2023-10-30T13:26:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.406+0000 7fa817d01640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa810154150 con 0x7fa8100c8490 2023-10-30T13:26:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.406+0000 7fa81529c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa81006e470 0x7fa81006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa81529c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa81006e470 0x7fa81006ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44160/0 (socket says 172.21.15.46:44160) 2023-10-30T13:26:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa81529c640 1 -- 172.21.15.46:0/1410545097 learned_addr learned my addr 172.21.15.46:0/1410545097 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa815a9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa810153d70 0x7fa8100cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36004/0 (socket says 172.21.15.46:36004) 2023-10-30T13:26:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa81629e640 1 --2- 172.21.15.46:0/1410545097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8100c8490 0x7fa8100c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa81529c640 1 -- 172.21.15.46:0/1410545097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8100c8490 msgr2=0x7fa8100c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa81529c640 1 --2- 172.21.15.46:0/1410545097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8100c8490 0x7fa8100c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa81529c640 1 -- 172.21.15.46:0/1410545097 >> v1:172.21.15.46:6789/0 conn(0x7fa810153d70 legacy=0x7fa8100cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa81529c640 1 -- 172.21.15.46:0/1410545097 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81006e030 con 0x7fa81006e470 2023-10-30T13:26:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa814a9b640 1 -- 172.21.15.46:0/1410545097 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159638979 0 0) 0x7fa81006afa0 con 0x7fa810153d70 2023-10-30T13:26:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa81529c640 1 --2- 172.21.15.46:0/1410545097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa81006e470 0x7fa81006ca80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa800009870 tx=0x7fa80001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea189ed34467a440 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:49.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa814a9b640 1 -- 172.21.15.46:0/1410545097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8000aa670 con 0x7fa81006e470 2023-10-30T13:26:49.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa814a9b640 1 -- 172.21.15.46:0/1410545097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8000aa7d0 con 0x7fa81006e470 2023-10-30T13:26:49.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.410+0000 7fa814a9b640 1 -- 172.21.15.46:0/1410545097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8000aab60 con 0x7fa81006e470 2023-10-30T13:26:49.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 -- 172.21.15.46:0/1410545097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa81006e470 msgr2=0x7fa81006ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 --2- 172.21.15.46:0/1410545097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa81006e470 0x7fa81006ca80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa800009870 tx=0x7fa80001c510 comp rx=0 tx=0).stop 2023-10-30T13:26:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 -- 172.21.15.46:0/1410545097 shutdown_connections 2023-10-30T13:26:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 --2- 172.21.15.46:0/1410545097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8100c8490 0x7fa8100c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 --2- 172.21.15.46:0/1410545097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa81006e470 0x7fa81006ca80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 -- 172.21.15.46:0/1410545097 >> 172.21.15.46:0/1410545097 conn(0x7fa810071ea0 msgr2=0x7fa810065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 -- 172.21.15.46:0/1410545097 shutdown_connections 2023-10-30T13:26:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 -- 172.21.15.46:0/1410545097 wait complete. 2023-10-30T13:26:49.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 Processor -- start 2023-10-30T13:26:49.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 -- start start 2023-10-30T13:26:49.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81006e470 0x7fa8101f5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:49.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8100c8490 0x7fa8101f5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:49.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa810153d70 0x7fa810156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:49.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa810147800 con 0x7fa810153d70 2023-10-30T13:26:49.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa810152bd0 con 0x7fa81006e470 2023-10-30T13:26:49.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa817d01640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8101f6600 con 0x7fa8100c8490 2023-10-30T13:26:49.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa81629e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa810153d70 0x7fa810156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:49.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa81629e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa810153d70 0x7fa810156750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44162/0 (socket says 172.21.15.46:44162) 2023-10-30T13:26:49.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa81629e640 1 -- 172.21.15.46:0/4151636035 learned_addr learned my addr 172.21.15.46:0/4151636035 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:49.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa81529c640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8100c8490 0x7fa8101f5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:49.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa815a9d640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81006e470 0x7fa8101f5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:49.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa81629e640 1 -- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8100c8490 msgr2=0x7fa8101f5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:49.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa81629e640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8100c8490 0x7fa8101f5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:49.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa81629e640 1 -- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81006e470 msgr2=0x7fa8101f5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:49.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa81629e640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81006e470 0x7fa8101f5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:49.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.414+0000 7fa81629e640 1 -- 172.21.15.46:0/4151636035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8000094d0 con 0x7fa810153d70 2023-10-30T13:26:49.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.418+0000 7fa81629e640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa810153d70 0x7fa810156750 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa808007e40 tx=0x7fa80800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:49.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.418+0000 7fa8067fc640 1 -- 172.21.15.46:0/4151636035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808014620 con 0x7fa810153d70 2023-10-30T13:26:49.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.418+0000 7fa8067fc640 1 -- 172.21.15.46:0/4151636035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80800ee90 con 0x7fa810153d70 2023-10-30T13:26:49.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.418+0000 7fa817d01640 1 -- 172.21.15.46:0/4151636035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa810152e70 con 0x7fa810153d70 2023-10-30T13:26:49.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.418+0000 7fa8067fc640 1 -- 172.21.15.46:0/4151636035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8080149e0 con 0x7fa810153d70 2023-10-30T13:26:49.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.418+0000 7fa817d01640 1 -- 172.21.15.46:0/4151636035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8101552d0 con 0x7fa810153d70 2023-10-30T13:26:49.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.418+0000 7fa817d01640 1 -- 172.21.15.46:0/4151636035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8101530b0 con 0x7fa810153d70 2023-10-30T13:26:49.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.418+0000 7fa8067fc640 1 -- 172.21.15.46:0/4151636035 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa808014b40 con 0x7fa810153d70 2023-10-30T13:26:49.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.418+0000 7fa8067fc640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7f40664a0 0x7fa7f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:49.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.418+0000 7fa8067fc640 1 -- 172.21.15.46:0/4151636035 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fa808084a00 con 0x7fa810153d70 2023-10-30T13:26:49.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.418+0000 7fa815a9d640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7f40664a0 0x7fa7f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:49.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.422+0000 7fa815a9d640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7f40664a0 0x7fa7f4068960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa80c000f70 tx=0x7fa80c007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:49.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.422+0000 7fa8067fc640 1 -- 172.21.15.46:0/4151636035 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa8101530b0 con 0x7fa810153d70 2023-10-30T13:26:49.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.610+0000 7fa817d01640 1 -- 172.21.15.46:0/4151636035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8101472f0 con 0x7fa810153d70 2023-10-30T13:26:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.610+0000 7fa8067fc640 1 -- 172.21.15.46:0/4151636035 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fa8101472f0 con 0x7fa810153d70 2023-10-30T13:26:49.617 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:49.618 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:45.740902+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2023-10-30T13:26:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.614+0000 7fa817d01640 1 -- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7f40664a0 msgr2=0x7fa7f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.614+0000 7fa817d01640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7f40664a0 0x7fa7f4068960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa80c000f70 tx=0x7fa80c007c10 comp rx=0 tx=0).stop 2023-10-30T13:26:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.614+0000 7fa817d01640 1 -- 172.21.15.46:0/4151636035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa810153d70 msgr2=0x7fa810156750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:49.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.614+0000 7fa817d01640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa810153d70 0x7fa810156750 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa808007e40 tx=0x7fa80800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:26:49.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.614+0000 7fa817d01640 1 -- 172.21.15.46:0/4151636035 shutdown_connections 2023-10-30T13:26:49.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.614+0000 7fa817d01640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7f40664a0 0x7fa7f4068960 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:49.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.614+0000 7fa817d01640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa810153d70 0x7fa810156750 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:49.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.614+0000 7fa817d01640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8100c8490 0x7fa8101f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:49.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.614+0000 7fa817d01640 1 --2- 172.21.15.46:0/4151636035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81006e470 0x7fa8101f5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:49.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.614+0000 7fa817d01640 1 -- 172.21.15.46:0/4151636035 >> 172.21.15.46:0/4151636035 conn(0x7fa810071ea0 msgr2=0x7fa81006f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:49.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.614+0000 7fa817d01640 1 -- 172.21.15.46:0/4151636035 shutdown_connections 2023-10-30T13:26:49.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:49.614+0000 7fa817d01640 1 -- 172.21.15.46:0/4151636035 wait complete. 2023-10-30T13:26:49.638 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:26:51.639 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:51.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d30246640 1 Processor -- start 2023-10-30T13:26:51.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d30246640 1 -- start start 2023-10-30T13:26:51.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d30246640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6d280c05d0 0x7f6d2813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:51.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d30246640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d280c6f60 0x7f6d2813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:51.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d30246640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d280c0a50 con 0x7f6d280c6b80 2023-10-30T13:26:51.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d30246640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2806f6e0 con 0x7f6d280c05d0 2023-10-30T13:26:51.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d30246640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2806f9c0 con 0x7f6d280c6f60 2023-10-30T13:26:51.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d2dfe2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6d280c6b80 0x7f6d280bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36016/0 (socket says 172.21.15.46:36016) 2023-10-30T13:26:51.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d2dfe2640 1 -- 172.21.15.46:0/1481532015 learned_addr learned my addr 172.21.15.46:0/1481532015 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:26:51.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d2d7e1640 1 --2- 172.21.15.46:0/1481532015 >> v2:172.21.15.46:3300/0 conn(0x7f6d280c05d0 0x7f6d2813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:51.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d2e7e3640 1 --2- 172.21.15.46:0/1481532015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d280c6f60 0x7f6d2813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:51.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d2cfe0640 1 -- 172.21.15.46:0/1481532015 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597965990 0 0) 0x7f6d280c0a50 con 0x7f6d280c6b80 2023-10-30T13:26:51.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d2cfe0640 1 -- 172.21.15.46:0/1481532015 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d14003530 con 0x7f6d280c6b80 2023-10-30T13:26:51.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d2d7e1640 1 -- 172.21.15.46:0/1481532015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d280c6f60 msgr2=0x7f6d2813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:51.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d2d7e1640 1 --2- 172.21.15.46:0/1481532015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d280c6f60 0x7f6d2813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:51.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d2d7e1640 1 -- 172.21.15.46:0/1481532015 >> v1:172.21.15.46:6789/0 conn(0x7f6d280c6b80 legacy=0x7f6d280bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:51.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.790+0000 7f6d2d7e1640 1 -- 172.21.15.46:0/1481532015 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2806fca0 con 0x7f6d280c05d0 2023-10-30T13:26:51.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.794+0000 7f6d2d7e1640 1 --2- 172.21.15.46:0/1481532015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d280c05d0 0x7f6d2813ed90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6d180097c0 tx=0x7f6d1801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65672ddb9c4c4435 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:51.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.794+0000 7f6d2cfe0640 1 -- 172.21.15.46:0/1481532015 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d180ad640 con 0x7f6d280c05d0 2023-10-30T13:26:51.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.794+0000 7f6d2cfe0640 1 -- 172.21.15.46:0/1481532015 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d180ad7a0 con 0x7f6d280c05d0 2023-10-30T13:26:51.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.794+0000 7f6d2cfe0640 1 -- 172.21.15.46:0/1481532015 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d180adb30 con 0x7f6d280c05d0 2023-10-30T13:26:51.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.794+0000 7f6d30246640 1 -- 172.21.15.46:0/1481532015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d280c05d0 msgr2=0x7f6d2813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:51.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.794+0000 7f6d30246640 1 --2- 172.21.15.46:0/1481532015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d280c05d0 0x7f6d2813ed90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6d180097c0 tx=0x7f6d1801c480 comp rx=0 tx=0).stop 2023-10-30T13:26:51.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.794+0000 7f6d30246640 1 -- 172.21.15.46:0/1481532015 shutdown_connections 2023-10-30T13:26:51.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.794+0000 7f6d30246640 1 --2- 172.21.15.46:0/1481532015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d280c6f60 0x7f6d2813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:51.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.794+0000 7f6d30246640 1 --2- 172.21.15.46:0/1481532015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d280c05d0 0x7f6d2813ed90 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:51.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.794+0000 7f6d30246640 1 -- 172.21.15.46:0/1481532015 >> 172.21.15.46:0/1481532015 conn(0x7f6d280c7910 msgr2=0x7f6d280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:51.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.794+0000 7f6d30246640 1 -- 172.21.15.46:0/1481532015 shutdown_connections 2023-10-30T13:26:51.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.794+0000 7f6d30246640 1 -- 172.21.15.46:0/1481532015 wait complete. 2023-10-30T13:26:51.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d30246640 1 Processor -- start 2023-10-30T13:26:51.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d30246640 1 -- start start 2023-10-30T13:26:51.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d30246640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d280c05d0 0x7f6d28150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:51.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d30246640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d280c6b80 0x7f6d28157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:51.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d30246640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d280c6f60 0x7f6d28158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:51.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d30246640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d281f63c0 con 0x7f6d280c6f60 2023-10-30T13:26:51.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d30246640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d281f5560 con 0x7f6d280c05d0 2023-10-30T13:26:51.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d30246640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d281f5840 con 0x7f6d280c6b80 2023-10-30T13:26:51.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d2e7e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d280c6f60 0x7f6d28158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:51.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d2e7e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d280c6f60 0x7f6d28158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44186/0 (socket says 172.21.15.46:44186) 2023-10-30T13:26:51.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d2e7e3640 1 -- 172.21.15.46:0/3658667854 learned_addr learned my addr 172.21.15.46:0/3658667854 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:51.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d2dfe2640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d280c05d0 0x7f6d28150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:51.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d2d7e1640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d280c6b80 0x7f6d28157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:51.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d2e7e3640 1 -- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d280c6b80 msgr2=0x7f6d28157ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:51.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d2e7e3640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d280c6b80 0x7f6d28157ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:51.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d2e7e3640 1 -- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d280c05d0 msgr2=0x7f6d28150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:51.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d2e7e3640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d280c05d0 0x7f6d28150690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:51.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d2e7e3640 1 -- 172.21.15.46:0/3658667854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d18009420 con 0x7f6d280c6f60 2023-10-30T13:26:51.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d2dfe2640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d280c05d0 0x7f6d28150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:51.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d2e7e3640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d280c6f60 0x7f6d28158010 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6d20007e60 tx=0x7f6d2000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:51.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d1e7fc640 1 -- 172.21.15.46:0/3658667854 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d20014420 con 0x7f6d280c6f60 2023-10-30T13:26:51.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d1e7fc640 1 -- 172.21.15.46:0/3658667854 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d200145a0 con 0x7f6d280c6f60 2023-10-30T13:26:51.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d30246640 1 -- 172.21.15.46:0/3658667854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d281f5b20 con 0x7f6d280c6f60 2023-10-30T13:26:51.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d1e7fc640 1 -- 172.21.15.46:0/3658667854 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d20014950 con 0x7f6d280c6f60 2023-10-30T13:26:51.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.798+0000 7f6d30246640 1 -- 172.21.15.46:0/3658667854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d281f60e0 con 0x7f6d280c6f60 2023-10-30T13:26:51.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.802+0000 7f6d30246640 1 -- 172.21.15.46:0/3658667854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d281577f0 con 0x7f6d280c6f60 2023-10-30T13:26:51.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.802+0000 7f6d1e7fc640 1 -- 172.21.15.46:0/3658667854 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6d20014ab0 con 0x7f6d280c6f60 2023-10-30T13:26:51.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.802+0000 7f6d1e7fc640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d0c0664a0 0x7f6d0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:51.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.802+0000 7f6d1e7fc640 1 -- 172.21.15.46:0/3658667854 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f6d200848f0 con 0x7f6d280c6f60 2023-10-30T13:26:51.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.802+0000 7f6d2dfe2640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d0c0664a0 0x7f6d0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:51.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.802+0000 7f6d2dfe2640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d0c0664a0 0x7f6d0c068960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6d240011f0 tx=0x7f6d24009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:51.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.806+0000 7f6d1e7fc640 1 -- 172.21.15.46:0/3658667854 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6d281577f0 con 0x7f6d280c6f60 2023-10-30T13:26:52.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.994+0000 7f6d30246640 1 -- 172.21.15.46:0/3658667854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d281577f0 con 0x7f6d280c6f60 2023-10-30T13:26:52.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d1e7fc640 1 -- 172.21.15.46:0/3658667854 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f6d281577f0 con 0x7f6d280c6f60 2023-10-30T13:26:52.002 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:52.002 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:50.864877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:52.003 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 25 2023-10-30T13:26:52.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d30246640 1 -- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d0c0664a0 msgr2=0x7f6d0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:52.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d30246640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d0c0664a0 0x7f6d0c068960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6d240011f0 tx=0x7f6d24009040 comp rx=0 tx=0).stop 2023-10-30T13:26:52.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d30246640 1 -- 172.21.15.46:0/3658667854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d280c6f60 msgr2=0x7f6d28158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:52.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d30246640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d280c6f60 0x7f6d28158010 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6d20007e60 tx=0x7f6d2000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d30246640 1 -- 172.21.15.46:0/3658667854 shutdown_connections 2023-10-30T13:26:52.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d30246640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d0c0664a0 0x7f6d0c068960 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d30246640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d280c6f60 0x7f6d28158010 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d30246640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d280c6b80 0x7f6d28157ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d30246640 1 --2- 172.21.15.46:0/3658667854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d280c05d0 0x7f6d28150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d30246640 1 -- 172.21.15.46:0/3658667854 >> 172.21.15.46:0/3658667854 conn(0x7f6d280c7910 msgr2=0x7f6d2806bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:52.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d30246640 1 -- 172.21.15.46:0/3658667854 shutdown_connections 2023-10-30T13:26:52.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:51.998+0000 7f6d30246640 1 -- 172.21.15.46:0/3658667854 wait complete. 2023-10-30T13:26:52.023 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=2, replacing gid=4259 2023-10-30T13:26:52.023 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.a 2023-10-30T13:26:52.024 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:26:52.024 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:26:52.024 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:26:52.029 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:26:52.030 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:52.118 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-10-30T13:26:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232eac9640 1 Processor -- start 2023-10-30T13:26:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232eac9640 1 -- start start 2023-10-30T13:26:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232eac9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f232815d6f0 0x7f232815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232eac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23280cf160 0x7f23280cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232eac9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23280cb800 con 0x7f23281538a0 2023-10-30T13:26:52.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232eac9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23280cb330 con 0x7f232815d6f0 2023-10-30T13:26:52.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232eac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23280cad70 con 0x7f23280cf160 2023-10-30T13:26:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232d066640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23280cf160 0x7f23280cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f2327fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f232815d6f0 0x7f232815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232d066640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23280cf160 0x7f23280cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:60688/0 (socket says 172.21.15.46:60688) 2023-10-30T13:26:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232d066640 1 -- 172.21.15.46:0/3915372202 learned_addr learned my addr 172.21.15.46:0/3915372202 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232d066640 1 -- 172.21.15.46:0/3915372202 >> v2:172.21.15.154:3300/0 conn(0x7f232815d6f0 msgr2=0x7f232815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232d066640 1 --2- 172.21.15.46:0/3915372202 >> v2:172.21.15.154:3300/0 conn(0x7f232815d6f0 0x7f232815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232d066640 1 -- 172.21.15.46:0/3915372202 >> v1:172.21.15.154:6789/0 conn(0x7f23281538a0 legacy=0x7f23280ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:52.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.174+0000 7f232d066640 1 -- 172.21.15.46:0/3915372202 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23280cb050 con 0x7f23280cf160 2023-10-30T13:26:52.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.178+0000 7f232d066640 1 --2- 172.21.15.46:0/3915372202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23280cf160 0x7f23280cf590 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2320009e50 tx=0x7f232001f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=829b5dc841e8c553 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:52.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.178+0000 7f23277fe640 1 -- 172.21.15.46:0/3915372202 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23200af400 con 0x7f23280cf160 2023-10-30T13:26:52.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.178+0000 7f23277fe640 1 -- 172.21.15.46:0/3915372202 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23200ab050 con 0x7f23280cf160 2023-10-30T13:26:52.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.178+0000 7f23277fe640 1 -- 172.21.15.46:0/3915372202 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23200af780 con 0x7f23280cf160 2023-10-30T13:26:52.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.178+0000 7f232eac9640 1 -- 172.21.15.46:0/3915372202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23280cf160 msgr2=0x7f23280cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:52.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.178+0000 7f232eac9640 1 --2- 172.21.15.46:0/3915372202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23280cf160 0x7f23280cf590 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2320009e50 tx=0x7f232001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.178+0000 7f232eac9640 1 -- 172.21.15.46:0/3915372202 shutdown_connections 2023-10-30T13:26:52.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.178+0000 7f232eac9640 1 --2- 172.21.15.46:0/3915372202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23280cf160 0x7f23280cf590 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.178+0000 7f232eac9640 1 --2- 172.21.15.46:0/3915372202 >> v2:172.21.15.154:3300/0 conn(0x7f232815d6f0 0x7f232815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.178+0000 7f232eac9640 1 -- 172.21.15.46:0/3915372202 >> 172.21.15.46:0/3915372202 conn(0x7f232806bfe0 msgr2=0x7f232806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.178+0000 7f232eac9640 1 -- 172.21.15.46:0/3915372202 shutdown_connections 2023-10-30T13:26:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.178+0000 7f232eac9640 1 -- 172.21.15.46:0/3915372202 wait complete. 2023-10-30T13:26:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232eac9640 1 Processor -- start 2023-10-30T13:26:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232eac9640 1 -- start start 2023-10-30T13:26:52.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232eac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23280cf160 0x7f232814e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:52.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232eac9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23281538a0 0x7f23280b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:52.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232eac9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f232815d6f0 0x7f23280cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:52.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232eac9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23281efd70 con 0x7f232815d6f0 2023-10-30T13:26:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232eac9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23281f0260 con 0x7f23281538a0 2023-10-30T13:26:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232eac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23280ce7b0 con 0x7f23280cf160 2023-10-30T13:26:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232d066640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f232815d6f0 0x7f23280cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232d066640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f232815d6f0 0x7f23280cd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44216/0 (socket says 172.21.15.46:44216) 2023-10-30T13:26:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232d066640 1 -- 172.21.15.46:0/2658163601 learned_addr learned my addr 172.21.15.46:0/2658163601 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:52.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f2327fff640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23281538a0 0x7f23280b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:52.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232c865640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23280cf160 0x7f232814e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:52.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232d066640 1 -- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23280cf160 msgr2=0x7f232814e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:52.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232d066640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23280cf160 0x7f232814e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232d066640 1 -- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23281538a0 msgr2=0x7f23280b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:52.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232d066640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23281538a0 0x7f23280b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232d066640 1 -- 172.21.15.46:0/2658163601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2320009b30 con 0x7f232815d6f0 2023-10-30T13:26:52.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232d066640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f232815d6f0 0x7f23280cd120 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f232001ce20 tx=0x7f232001bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:52.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f23257fa640 1 -- 172.21.15.46:0/2658163601 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23200afc20 con 0x7f232815d6f0 2023-10-30T13:26:52.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f23257fa640 1 -- 172.21.15.46:0/2658163601 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23200afd80 con 0x7f232815d6f0 2023-10-30T13:26:52.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f23257fa640 1 -- 172.21.15.46:0/2658163601 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23200ba660 con 0x7f232815d6f0 2023-10-30T13:26:52.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232eac9640 1 -- 172.21.15.46:0/2658163601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232814e760 con 0x7f232815d6f0 2023-10-30T13:26:52.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.182+0000 7f232eac9640 1 -- 172.21.15.46:0/2658163601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23280cea50 con 0x7f232815d6f0 2023-10-30T13:26:52.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.186+0000 7f232eac9640 1 -- 172.21.15.46:0/2658163601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23280b2a50 con 0x7f232815d6f0 2023-10-30T13:26:52.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.186+0000 7f23257fa640 1 -- 172.21.15.46:0/2658163601 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f23200b6040 con 0x7f232815d6f0 2023-10-30T13:26:52.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.186+0000 7f23257fa640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f230c0664d0 0x7f230c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:52.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.186+0000 7f23257fa640 1 -- 172.21.15.46:0/2658163601 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f2320121ab0 con 0x7f232815d6f0 2023-10-30T13:26:52.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.186+0000 7f232c865640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f230c0664d0 0x7f230c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:52.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.190+0000 7f232c865640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f230c0664d0 0x7f230c068990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f231c007fd0 tx=0x7f231c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:52.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.190+0000 7f23257fa640 1 -- 172.21.15.46:0/2658163601 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23200eef50 con 0x7f232815d6f0 2023-10-30T13:26:52.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.378+0000 7f232eac9640 1 -- 172.21.15.46:0/2658163601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f232804e170 con 0x7f232815d6f0 2023-10-30T13:26:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.378+0000 7f23257fa640 1 -- 172.21.15.46:0/2658163601 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f2320009510 con 0x7f232815d6f0 2023-10-30T13:26:52.385 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:52.386 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:50.864877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:52.387 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 25 2023-10-30T13:26:52.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.382+0000 7f232eac9640 1 -- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f230c0664d0 msgr2=0x7f230c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:52.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.382+0000 7f232eac9640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f230c0664d0 0x7f230c068990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f231c007fd0 tx=0x7f231c009040 comp rx=0 tx=0).stop 2023-10-30T13:26:52.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.382+0000 7f232eac9640 1 -- 172.21.15.46:0/2658163601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f232815d6f0 msgr2=0x7f23280cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:52.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.382+0000 7f232eac9640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f232815d6f0 0x7f23280cd120 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f232001ce20 tx=0x7f232001bf90 comp rx=0 tx=0).stop 2023-10-30T13:26:52.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.382+0000 7f232eac9640 1 -- 172.21.15.46:0/2658163601 shutdown_connections 2023-10-30T13:26:52.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.382+0000 7f232eac9640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f230c0664d0 0x7f230c068990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.382+0000 7f232eac9640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f232815d6f0 0x7f23280cd120 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.382+0000 7f232eac9640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23281538a0 0x7f23280b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.382+0000 7f232eac9640 1 --2- 172.21.15.46:0/2658163601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23280cf160 0x7f232814e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.382+0000 7f232eac9640 1 -- 172.21.15.46:0/2658163601 >> 172.21.15.46:0/2658163601 conn(0x7f232806bfe0 msgr2=0x7f23280c48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.382+0000 7f232eac9640 1 -- 172.21.15.46:0/2658163601 shutdown_connections 2023-10-30T13:26:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:52.382+0000 7f232eac9640 1 -- 172.21.15.46:0/2658163601 wait complete. 2023-10-30T13:26:52.406 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T13:26:53.928 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T13:26:53.922+0000 7fce56864640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:26:54.407 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71f380640 1 Processor -- start 2023-10-30T13:26:54.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71f380640 1 -- start start 2023-10-30T13:26:54.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71f380640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe718069e90 0x7fe71815a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:54.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71f380640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe71806a270 0x7fe7180c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:54.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71f380640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe71806cb80 0x7fe71806d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:54.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71f380640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7180c8570 con 0x7fe718069e90 2023-10-30T13:26:54.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71f380640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe71806d550 con 0x7fe71806cb80 2023-10-30T13:26:54.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71f380640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180c6c10 con 0x7fe71806a270 2023-10-30T13:26:54.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71d11c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe718069e90 0x7fe71815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:54.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71d11c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe718069e90 0x7fe71815a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44224/0 (socket says 172.21.15.46:44224) 2023-10-30T13:26:54.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71d11c640 1 -- 172.21.15.46:0/105261697 learned_addr learned my addr 172.21.15.46:0/105261697 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:54.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71d91d640 1 --2- 172.21.15.46:0/105261697 >> v2:172.21.15.154:3300/0 conn(0x7fe71806cb80 0x7fe71806d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:54.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71d11c640 1 -- 172.21.15.46:0/105261697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe71806a270 msgr2=0x7fe7180c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:54.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71c91b640 1 --2- 172.21.15.46:0/105261697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe71806a270 0x7fe7180c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:54.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71d11c640 1 --2- 172.21.15.46:0/105261697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe71806a270 0x7fe7180c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:54.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71d11c640 1 -- 172.21.15.46:0/105261697 >> v2:172.21.15.154:3300/0 conn(0x7fe71806cb80 msgr2=0x7fe71806d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:54.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71d11c640 1 --2- 172.21.15.46:0/105261697 >> v2:172.21.15.154:3300/0 conn(0x7fe71806cb80 0x7fe71806d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:54.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71d11c640 1 -- 172.21.15.46:0/105261697 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7180c6ef0 con 0x7fe718069e90 2023-10-30T13:26:54.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71d91d640 1 --2- 172.21.15.46:0/105261697 >> v2:172.21.15.154:3300/0 conn(0x7fe71806cb80 0x7fe71806d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:26:54.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.558+0000 7fe71c91b640 1 --2- 172.21.15.46:0/105261697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe71806a270 0x7fe7180c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:26:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe71d11c640 1 --2- 172.21.15.46:0/105261697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe718069e90 0x7fe71815a410 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe71806ff00 tx=0x7fe71401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e652a718df6994f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe70ffff640 1 -- 172.21.15.46:0/105261697 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7140ac410 con 0x7fe718069e90 2023-10-30T13:26:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe70ffff640 1 -- 172.21.15.46:0/105261697 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7140ac570 con 0x7fe718069e90 2023-10-30T13:26:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe70ffff640 1 -- 172.21.15.46:0/105261697 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7140ac900 con 0x7fe718069e90 2023-10-30T13:26:54.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe71f380640 1 -- 172.21.15.46:0/105261697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe718069e90 msgr2=0x7fe71815a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:54.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe71f380640 1 --2- 172.21.15.46:0/105261697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe718069e90 0x7fe71815a410 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe71806ff00 tx=0x7fe71401c500 comp rx=0 tx=0).stop 2023-10-30T13:26:54.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe71f380640 1 -- 172.21.15.46:0/105261697 shutdown_connections 2023-10-30T13:26:54.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe71f380640 1 --2- 172.21.15.46:0/105261697 >> v2:172.21.15.154:3300/0 conn(0x7fe71806cb80 0x7fe71806d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:54.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe71f380640 1 --2- 172.21.15.46:0/105261697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe71806a270 0x7fe7180c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:54.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe71f380640 1 --2- 172.21.15.46:0/105261697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe718069e90 0x7fe71815a410 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:54.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe71f380640 1 -- 172.21.15.46:0/105261697 >> 172.21.15.46:0/105261697 conn(0x7fe71806dfd0 msgr2=0x7fe71806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:54.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe71f380640 1 -- 172.21.15.46:0/105261697 shutdown_connections 2023-10-30T13:26:54.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.562+0000 7fe71f380640 1 -- 172.21.15.46:0/105261697 wait complete. 2023-10-30T13:26:54.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71f380640 1 Processor -- start 2023-10-30T13:26:54.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71f380640 1 -- start start 2023-10-30T13:26:54.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71f380640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe718069e90 0x7fe718150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:54.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71f380640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe71806a270 0x7fe7181f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:54.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71f380640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe71806cb80 0x7fe71814f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:54.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71f380640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180c59c0 con 0x7fe718069e90 2023-10-30T13:26:54.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71f380640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180c4980 con 0x7fe71806a270 2023-10-30T13:26:54.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71f380640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180c4c60 con 0x7fe71806cb80 2023-10-30T13:26:54.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71d11c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe718069e90 0x7fe718150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:54.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71d11c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe718069e90 0x7fe718150c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44232/0 (socket says 172.21.15.46:44232) 2023-10-30T13:26:54.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71d11c640 1 -- 172.21.15.46:0/4124087175 learned_addr learned my addr 172.21.15.46:0/4124087175 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:26:54.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71c91b640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe71806a270 0x7fe7181f4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:54.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71d91d640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe71806cb80 0x7fe71814f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71d11c640 1 -- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe71806cb80 msgr2=0x7fe71814f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71d11c640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe71806cb80 0x7fe71814f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71d11c640 1 -- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe71806a270 msgr2=0x7fe7181f4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71d11c640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe71806a270 0x7fe7181f4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71d11c640 1 -- 172.21.15.46:0/4124087175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7140094d0 con 0x7fe718069e90 2023-10-30T13:26:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71d91d640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe71806cb80 0x7fe71814f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:26:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71d11c640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe718069e90 0x7fe718150c90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe7140af040 tx=0x7fe7140192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe70dffb640 1 -- 172.21.15.46:0/4124087175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7140ac410 con 0x7fe718069e90 2023-10-30T13:26:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe70dffb640 1 -- 172.21.15.46:0/4124087175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7140b65f0 con 0x7fe718069e90 2023-10-30T13:26:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe70dffb640 1 -- 172.21.15.46:0/4124087175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7140b69a0 con 0x7fe718069e90 2023-10-30T13:26:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71f380640 1 -- 172.21.15.46:0/4124087175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7180c4f40 con 0x7fe718069e90 2023-10-30T13:26:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.566+0000 7fe71f380640 1 -- 172.21.15.46:0/4124087175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7180c5650 con 0x7fe718069e90 2023-10-30T13:26:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.570+0000 7fe71f380640 1 -- 172.21.15.46:0/4124087175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe71806eac0 con 0x7fe718069e90 2023-10-30T13:26:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.570+0000 7fe70dffb640 1 -- 172.21.15.46:0/4124087175 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe7140b6b00 con 0x7fe718069e90 2023-10-30T13:26:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.570+0000 7fe70dffb640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe6fc0664a0 0x7fe6fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:54.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.570+0000 7fe70dffb640 1 -- 172.21.15.46:0/4124087175 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fe71411eb40 con 0x7fe718069e90 2023-10-30T13:26:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.570+0000 7fe71c91b640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe6fc0664a0 0x7fe6fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.570+0000 7fe71c91b640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe6fc0664a0 0x7fe6fc068960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe7181f2710 tx=0x7fe708008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:54.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.574+0000 7fe70dffb640 1 -- 172.21.15.46:0/4124087175 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe7140ebe70 con 0x7fe718069e90 2023-10-30T13:26:54.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 -- 172.21.15.46:0/4124087175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe71806c630 con 0x7fe718069e90 2023-10-30T13:26:54.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe70dffb640 1 -- 172.21.15.46:0/4124087175 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fe7140b1020 con 0x7fe718069e90 2023-10-30T13:26:54.771 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:26:54.771 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:26:50.864877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:54.772 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 26 2023-10-30T13:26:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 -- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe6fc0664a0 msgr2=0x7fe6fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe6fc0664a0 0x7fe6fc068960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe7181f2710 tx=0x7fe708008040 comp rx=0 tx=0).stop 2023-10-30T13:26:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 -- 172.21.15.46:0/4124087175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe718069e90 msgr2=0x7fe718150c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe718069e90 0x7fe718150c90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe7140af040 tx=0x7fe7140192a0 comp rx=0 tx=0).stop 2023-10-30T13:26:54.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 -- 172.21.15.46:0/4124087175 shutdown_connections 2023-10-30T13:26:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe6fc0664a0 0x7fe6fc068960 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe71806cb80 0x7fe71814f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe71806a270 0x7fe7181f4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 --2- 172.21.15.46:0/4124087175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe718069e90 0x7fe718150c90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:54.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 -- 172.21.15.46:0/4124087175 >> 172.21.15.46:0/4124087175 conn(0x7fe71806dfd0 msgr2=0x7fe71806e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 -- 172.21.15.46:0/4124087175 shutdown_connections 2023-10-30T13:26:54.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:26:54.766+0000 7fe71f380640 1 -- 172.21.15.46:0/4124087175 wait complete. 2023-10-30T13:26:54.792 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:26:54.792 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 116 secs before thrashing 2023-10-30T13:26:57.468 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2023-10-30T13:26:57.468 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2023-10-30T13:26:57.469 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2023-10-30T13:26:57.469 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2023-10-30T13:26:57.469 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2023-10-30T13:26:57.469 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2023-10-30T13:26:57.469 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2023-10-30T13:26:57.470 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2023-10-30T13:26:57.470 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2023-10-30T13:26:57.470 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2023-10-30T13:26:57.471 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2023-10-30T13:26:57.471 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2023-10-30T13:26:57.471 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2023-10-30T13:26:57.472 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2023-10-30T13:26:57.472 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2023-10-30T13:26:57.472 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2023-10-30T13:26:57.473 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2023-10-30T13:26:57.473 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2023-10-30T13:26:57.473 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2023-10-30T13:26:57.474 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2023-10-30T13:26:57.474 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2023-10-30T13:26:57.475 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2023-10-30T13:26:57.475 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2023-10-30T13:26:57.475 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2023-10-30T13:26:57.475 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2023-10-30T13:26:57.476 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2023-10-30T13:26:57.476 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2023-10-30T13:26:57.476 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2023-10-30T13:26:57.477 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2023-10-30T13:26:57.477 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2023-10-30T13:26:57.477 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2023-10-30T13:26:57.477 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2023-10-30T13:26:57.478 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2023-10-30T13:26:57.478 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2023-10-30T13:26:57.478 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2023-10-30T13:26:57.478 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2023-10-30T13:26:57.478 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2023-10-30T13:26:57.479 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2023-10-30T13:26:57.479 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2023-10-30T13:26:57.479 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2023-10-30T13:26:57.479 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2023-10-30T13:26:57.479 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2023-10-30T13:26:57.480 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2023-10-30T13:26:57.480 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2023-10-30T13:26:57.480 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2023-10-30T13:26:57.480 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2023-10-30T13:26:57.480 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2023-10-30T13:26:57.481 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2023-10-30T13:26:57.481 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2023-10-30T13:26:57.481 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2023-10-30T13:26:57.481 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2023-10-30T13:26:57.481 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2023-10-30T13:26:57.482 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2023-10-30T13:26:57.482 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2023-10-30T13:26:57.482 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2023-10-30T13:26:57.482 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2023-10-30T13:26:57.482 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2023-10-30T13:26:57.483 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2023-10-30T13:26:57.483 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2023-10-30T13:26:57.483 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2023-10-30T13:26:57.483 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2023-10-30T13:26:57.483 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2023-10-30T13:26:57.483 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2023-10-30T13:26:57.484 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2023-10-30T13:26:57.484 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2023-10-30T13:26:57.484 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2023-10-30T13:26:57.484 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2023-10-30T13:26:57.484 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2023-10-30T13:26:57.485 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2023-10-30T13:26:57.485 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2023-10-30T13:26:57.485 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2023-10-30T13:26:57.485 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2023-10-30T13:26:57.485 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2023-10-30T13:26:57.486 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2023-10-30T13:26:57.486 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2023-10-30T13:26:57.486 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2023-10-30T13:26:57.486 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2023-10-30T13:26:57.487 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2023-10-30T13:26:57.487 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2023-10-30T13:26:57.487 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2023-10-30T13:26:57.487 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2023-10-30T13:26:57.617 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2023-10-30T13:26:57.617 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2023-10-30T13:26:57.617 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2023-10-30T13:26:57.617 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2023-10-30T13:26:57.618 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2023-10-30T13:26:57.618 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2023-10-30T13:26:57.618 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2023-10-30T13:26:57.618 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2023-10-30T13:26:57.618 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2023-10-30T13:26:57.618 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2023-10-30T13:26:57.619 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2023-10-30T13:26:57.619 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2023-10-30T13:26:57.619 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2023-10-30T13:26:57.619 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2023-10-30T13:26:57.619 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2023-10-30T13:26:57.619 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2023-10-30T13:26:57.620 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2023-10-30T13:26:57.620 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2023-10-30T13:26:57.620 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2023-10-30T13:26:57.620 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2023-10-30T13:26:57.620 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2023-10-30T13:26:57.620 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2023-10-30T13:26:57.621 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2023-10-30T13:26:57.621 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2023-10-30T13:26:57.621 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2023-10-30T13:26:57.621 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2023-10-30T13:26:57.621 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2023-10-30T13:26:57.622 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2023-10-30T13:26:57.622 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2023-10-30T13:26:57.622 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2023-10-30T13:26:57.622 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2023-10-30T13:26:57.622 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2023-10-30T13:26:57.622 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2023-10-30T13:26:57.623 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2023-10-30T13:26:57.623 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2023-10-30T13:26:57.623 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2023-10-30T13:26:57.623 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2023-10-30T13:26:57.623 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2023-10-30T13:26:57.623 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2023-10-30T13:26:57.624 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2023-10-30T13:26:57.624 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2023-10-30T13:26:57.624 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2023-10-30T13:26:57.624 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2023-10-30T13:26:57.624 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2023-10-30T13:26:57.624 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2023-10-30T13:26:57.625 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2023-10-30T13:26:57.625 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2023-10-30T13:26:57.625 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2023-10-30T13:26:57.625 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2023-10-30T13:26:57.625 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2023-10-30T13:26:57.625 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2023-10-30T13:26:57.626 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2023-10-30T13:26:57.626 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2023-10-30T13:26:57.626 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2023-10-30T13:26:57.626 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2023-10-30T13:26:57.626 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2023-10-30T13:26:57.627 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2023-10-30T13:26:57.627 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2023-10-30T13:26:57.627 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2023-10-30T13:26:57.627 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2023-10-30T13:26:57.627 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2023-10-30T13:26:57.627 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2023-10-30T13:26:57.628 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2023-10-30T13:26:57.628 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2023-10-30T13:26:57.628 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2023-10-30T13:26:57.628 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2023-10-30T13:26:57.628 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2023-10-30T13:26:57.628 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2023-10-30T13:26:57.629 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2023-10-30T13:26:57.629 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2023-10-30T13:26:57.629 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2023-10-30T13:26:57.629 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2023-10-30T13:26:57.629 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2023-10-30T13:26:57.629 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2023-10-30T13:26:57.630 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2023-10-30T13:26:57.630 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2023-10-30T13:26:57.630 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2023-10-30T13:26:57.630 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2023-10-30T13:26:57.630 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2023-10-30T13:26:57.630 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2023-10-30T13:26:57.631 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2023-10-30T13:26:58.000 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2023-10-30T13:26:58.000 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2023-10-30T13:26:58.001 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2023-10-30T13:26:58.001 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2023-10-30T13:26:58.001 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2023-10-30T13:26:58.001 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2023-10-30T13:26:58.001 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2023-10-30T13:26:58.001 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2023-10-30T13:26:58.002 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2023-10-30T13:26:58.002 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2023-10-30T13:26:58.002 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2023-10-30T13:26:58.002 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2023-10-30T13:26:58.002 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2023-10-30T13:26:58.002 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2023-10-30T13:26:58.003 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2023-10-30T13:26:58.003 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2023-10-30T13:26:58.003 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2023-10-30T13:26:58.003 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2023-10-30T13:26:58.003 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2023-10-30T13:26:58.003 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2023-10-30T13:26:58.004 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2023-10-30T13:26:58.004 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2023-10-30T13:26:58.004 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2023-10-30T13:26:58.004 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2023-10-30T13:26:58.004 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2023-10-30T13:26:58.004 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2023-10-30T13:26:58.005 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2023-10-30T13:26:58.005 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2023-10-30T13:26:58.005 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2023-10-30T13:26:58.005 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2023-10-30T13:26:58.005 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2023-10-30T13:26:58.005 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2023-10-30T13:26:58.006 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2023-10-30T13:26:58.006 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2023-10-30T13:26:58.006 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2023-10-30T13:26:58.006 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2023-10-30T13:26:58.006 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2023-10-30T13:26:58.006 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2023-10-30T13:26:58.007 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2023-10-30T13:26:58.007 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2023-10-30T13:26:58.007 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2023-10-30T13:26:58.007 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2023-10-30T13:26:58.007 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2023-10-30T13:26:58.007 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2023-10-30T13:26:58.008 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2023-10-30T13:26:58.008 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2023-10-30T13:26:58.008 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2023-10-30T13:26:58.008 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2023-10-30T13:26:58.008 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2023-10-30T13:26:58.008 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2023-10-30T13:26:58.009 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2023-10-30T13:26:58.009 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2023-10-30T13:26:58.009 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2023-10-30T13:26:58.009 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2023-10-30T13:26:58.009 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2023-10-30T13:26:58.009 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2023-10-30T13:26:58.010 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2023-10-30T13:26:58.010 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2023-10-30T13:26:58.010 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2023-10-30T13:26:58.010 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2023-10-30T13:26:58.010 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2023-10-30T13:26:58.011 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2023-10-30T13:26:58.011 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2023-10-30T13:26:58.011 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2023-10-30T13:26:58.011 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2023-10-30T13:26:58.011 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2023-10-30T13:26:58.011 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2023-10-30T13:26:58.011 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2023-10-30T13:26:58.012 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2023-10-30T13:26:58.012 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2023-10-30T13:26:58.012 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2023-10-30T13:26:58.012 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2023-10-30T13:26:58.012 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2023-10-30T13:26:58.012 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2023-10-30T13:26:58.013 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2023-10-30T13:26:58.013 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2023-10-30T13:26:58.013 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2023-10-30T13:26:58.013 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2023-10-30T13:26:58.013 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2023-10-30T13:26:58.014 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2023-10-30T13:26:58.014 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2023-10-30T13:26:58.554 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2023-10-30T13:26:58.554 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2023-10-30T13:26:58.555 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2023-10-30T13:26:58.555 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2023-10-30T13:26:58.555 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2023-10-30T13:26:58.555 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2023-10-30T13:26:58.555 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2023-10-30T13:26:58.555 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2023-10-30T13:26:58.556 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2023-10-30T13:26:58.556 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2023-10-30T13:26:58.556 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2023-10-30T13:26:58.556 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2023-10-30T13:26:58.556 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2023-10-30T13:26:58.557 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2023-10-30T13:26:58.557 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2023-10-30T13:26:58.557 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2023-10-30T13:26:58.557 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2023-10-30T13:26:58.557 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2023-10-30T13:26:58.558 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2023-10-30T13:26:58.558 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2023-10-30T13:26:58.558 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2023-10-30T13:26:58.558 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2023-10-30T13:26:58.558 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2023-10-30T13:26:58.559 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2023-10-30T13:26:58.559 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2023-10-30T13:26:58.559 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2023-10-30T13:26:58.559 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2023-10-30T13:26:58.559 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2023-10-30T13:26:58.560 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2023-10-30T13:26:58.560 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2023-10-30T13:26:58.560 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2023-10-30T13:26:58.560 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2023-10-30T13:26:58.560 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2023-10-30T13:26:58.560 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2023-10-30T13:26:58.561 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2023-10-30T13:26:58.561 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2023-10-30T13:26:58.561 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2023-10-30T13:26:58.561 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2023-10-30T13:26:58.561 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2023-10-30T13:26:58.562 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2023-10-30T13:26:58.562 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2023-10-30T13:26:58.562 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2023-10-30T13:26:58.562 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2023-10-30T13:26:58.562 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2023-10-30T13:26:58.563 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2023-10-30T13:26:58.563 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2023-10-30T13:26:58.563 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2023-10-30T13:26:58.563 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2023-10-30T13:26:58.563 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2023-10-30T13:26:58.563 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2023-10-30T13:26:58.564 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2023-10-30T13:26:58.564 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2023-10-30T13:26:58.564 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2023-10-30T13:26:58.564 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2023-10-30T13:26:58.564 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2023-10-30T13:26:58.565 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2023-10-30T13:26:58.565 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2023-10-30T13:26:58.565 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2023-10-30T13:26:58.565 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2023-10-30T13:26:58.565 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2023-10-30T13:26:58.566 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2023-10-30T13:26:58.566 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2023-10-30T13:26:58.566 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2023-10-30T13:26:58.566 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2023-10-30T13:26:58.566 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2023-10-30T13:26:58.567 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2023-10-30T13:26:58.567 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2023-10-30T13:26:58.567 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2023-10-30T13:26:58.567 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2023-10-30T13:26:58.567 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2023-10-30T13:26:58.567 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2023-10-30T13:26:58.568 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2023-10-30T13:26:58.568 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2023-10-30T13:26:58.568 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2023-10-30T13:26:58.568 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2023-10-30T13:26:58.568 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2023-10-30T13:26:58.569 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2023-10-30T13:26:58.569 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2023-10-30T13:26:58.569 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2023-10-30T13:26:58.569 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2023-10-30T13:26:58.569 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2023-10-30T13:26:58.718 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2023-10-30T13:26:58.718 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2023-10-30T13:26:58.718 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2023-10-30T13:26:58.718 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2023-10-30T13:26:58.719 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2023-10-30T13:26:58.719 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2023-10-30T13:26:58.719 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2023-10-30T13:26:58.719 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2023-10-30T13:26:58.719 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2023-10-30T13:26:58.719 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2023-10-30T13:26:58.720 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2023-10-30T13:26:58.720 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2023-10-30T13:26:58.720 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2023-10-30T13:26:58.720 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2023-10-30T13:26:58.720 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2023-10-30T13:26:58.720 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2023-10-30T13:26:58.721 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2023-10-30T13:26:58.721 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2023-10-30T13:26:58.721 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2023-10-30T13:26:58.721 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2023-10-30T13:26:58.721 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2023-10-30T13:26:58.721 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2023-10-30T13:26:58.722 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2023-10-30T13:26:58.722 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2023-10-30T13:26:58.722 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2023-10-30T13:26:58.722 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2023-10-30T13:26:58.722 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2023-10-30T13:26:58.723 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2023-10-30T13:26:58.723 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2023-10-30T13:26:58.723 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2023-10-30T13:26:58.723 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2023-10-30T13:26:58.723 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2023-10-30T13:26:58.723 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2023-10-30T13:26:58.724 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2023-10-30T13:26:58.724 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2023-10-30T13:26:58.724 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2023-10-30T13:26:58.724 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2023-10-30T13:26:58.725 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2023-10-30T13:26:58.725 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2023-10-30T13:26:58.725 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2023-10-30T13:26:58.725 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2023-10-30T13:26:58.725 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2023-10-30T13:26:58.726 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2023-10-30T13:26:58.726 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2023-10-30T13:26:58.726 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2023-10-30T13:26:58.726 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2023-10-30T13:26:58.726 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2023-10-30T13:26:58.726 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2023-10-30T13:26:58.727 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2023-10-30T13:26:58.727 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2023-10-30T13:26:58.727 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2023-10-30T13:26:58.727 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2023-10-30T13:26:58.727 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2023-10-30T13:26:58.728 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2023-10-30T13:26:58.728 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2023-10-30T13:26:58.728 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2023-10-30T13:26:58.728 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2023-10-30T13:26:58.728 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2023-10-30T13:26:58.728 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2023-10-30T13:26:58.729 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2023-10-30T13:26:58.729 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2023-10-30T13:26:58.729 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2023-10-30T13:26:58.729 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2023-10-30T13:26:58.729 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2023-10-30T13:26:58.729 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2023-10-30T13:26:58.730 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2023-10-30T13:26:58.730 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2023-10-30T13:26:58.730 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2023-10-30T13:26:58.730 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2023-10-30T13:26:58.730 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2023-10-30T13:26:58.731 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2023-10-30T13:26:58.731 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2023-10-30T13:26:58.731 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2023-10-30T13:26:58.731 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2023-10-30T13:26:58.731 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2023-10-30T13:26:58.731 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2023-10-30T13:26:58.732 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2023-10-30T13:26:58.732 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2023-10-30T13:26:58.732 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2023-10-30T13:26:58.732 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2023-10-30T13:26:58.732 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2023-10-30T13:26:58.733 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2023-10-30T13:27:00.028 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2023-10-30T13:27:00.029 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2023-10-30T13:27:00.029 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2023-10-30T13:27:00.029 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2023-10-30T13:27:00.029 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2023-10-30T13:27:00.030 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2023-10-30T13:27:00.030 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2023-10-30T13:27:00.030 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2023-10-30T13:27:00.030 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2023-10-30T13:27:00.030 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2023-10-30T13:27:00.031 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2023-10-30T13:27:00.031 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2023-10-30T13:27:00.031 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2023-10-30T13:27:00.031 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2023-10-30T13:27:00.031 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2023-10-30T13:27:00.032 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2023-10-30T13:27:00.032 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2023-10-30T13:27:00.032 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2023-10-30T13:27:00.032 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2023-10-30T13:27:00.032 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2023-10-30T13:27:00.033 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2023-10-30T13:27:00.033 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2023-10-30T13:27:00.033 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2023-10-30T13:27:00.033 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2023-10-30T13:27:00.033 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2023-10-30T13:27:00.033 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2023-10-30T13:27:00.034 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2023-10-30T13:27:00.034 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2023-10-30T13:27:00.034 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2023-10-30T13:27:00.034 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2023-10-30T13:27:00.034 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2023-10-30T13:27:00.035 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2023-10-30T13:27:00.035 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2023-10-30T13:27:00.035 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2023-10-30T13:27:00.035 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2023-10-30T13:27:00.035 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2023-10-30T13:27:00.036 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2023-10-30T13:27:00.036 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2023-10-30T13:27:00.036 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2023-10-30T13:27:00.036 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2023-10-30T13:27:00.036 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2023-10-30T13:27:00.037 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2023-10-30T13:27:00.037 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2023-10-30T13:27:00.037 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2023-10-30T13:27:00.037 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2023-10-30T13:27:00.037 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2023-10-30T13:27:00.038 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2023-10-30T13:27:00.038 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2023-10-30T13:27:00.038 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2023-10-30T13:27:00.038 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2023-10-30T13:27:00.038 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2023-10-30T13:27:00.039 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2023-10-30T13:27:00.039 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2023-10-30T13:27:00.039 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2023-10-30T13:27:00.039 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2023-10-30T13:27:00.039 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2023-10-30T13:27:00.040 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2023-10-30T13:27:00.040 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2023-10-30T13:27:00.040 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2023-10-30T13:27:00.040 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2023-10-30T13:27:00.040 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2023-10-30T13:27:00.041 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2023-10-30T13:27:00.041 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2023-10-30T13:27:00.041 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2023-10-30T13:27:00.041 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2023-10-30T13:27:00.041 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2023-10-30T13:27:00.041 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2023-10-30T13:27:00.042 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2023-10-30T13:27:00.042 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2023-10-30T13:27:00.042 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2023-10-30T13:27:00.042 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2023-10-30T13:27:00.042 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2023-10-30T13:27:00.043 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2023-10-30T13:27:00.043 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2023-10-30T13:27:00.043 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2023-10-30T13:27:00.043 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2023-10-30T13:27:00.043 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2023-10-30T13:27:00.044 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2023-10-30T13:27:00.044 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2023-10-30T13:27:00.044 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2023-10-30T13:27:00.044 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2023-10-30T13:27:00.507 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2023-10-30T13:27:00.507 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2023-10-30T13:27:00.508 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2023-10-30T13:27:00.508 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2023-10-30T13:27:00.508 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2023-10-30T13:27:00.508 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2023-10-30T13:27:00.509 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2023-10-30T13:27:00.509 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2023-10-30T13:27:00.509 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2023-10-30T13:27:00.509 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2023-10-30T13:27:00.509 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2023-10-30T13:27:00.509 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2023-10-30T13:27:00.510 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2023-10-30T13:27:00.510 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2023-10-30T13:27:00.510 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2023-10-30T13:27:00.510 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2023-10-30T13:27:00.510 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2023-10-30T13:27:00.510 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2023-10-30T13:27:00.511 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2023-10-30T13:27:00.511 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2023-10-30T13:27:00.511 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2023-10-30T13:27:00.511 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2023-10-30T13:27:00.511 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2023-10-30T13:27:00.512 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2023-10-30T13:27:00.512 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2023-10-30T13:27:00.512 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2023-10-30T13:27:00.512 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2023-10-30T13:27:00.512 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2023-10-30T13:27:00.512 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2023-10-30T13:27:00.513 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2023-10-30T13:27:00.513 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2023-10-30T13:27:00.513 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2023-10-30T13:27:00.513 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2023-10-30T13:27:00.513 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2023-10-30T13:27:00.513 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2023-10-30T13:27:00.514 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2023-10-30T13:27:00.514 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2023-10-30T13:27:00.514 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2023-10-30T13:27:00.514 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2023-10-30T13:27:00.514 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2023-10-30T13:27:00.514 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2023-10-30T13:27:00.515 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2023-10-30T13:27:00.515 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2023-10-30T13:27:00.515 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2023-10-30T13:27:00.515 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2023-10-30T13:27:00.515 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2023-10-30T13:27:00.515 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2023-10-30T13:27:00.516 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2023-10-30T13:27:00.516 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2023-10-30T13:27:00.516 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2023-10-30T13:27:00.516 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2023-10-30T13:27:00.516 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2023-10-30T13:27:00.517 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2023-10-30T13:27:00.517 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2023-10-30T13:27:00.517 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2023-10-30T13:27:00.517 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2023-10-30T13:27:00.517 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2023-10-30T13:27:00.518 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2023-10-30T13:27:00.518 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2023-10-30T13:27:00.518 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2023-10-30T13:27:00.518 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2023-10-30T13:27:00.518 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2023-10-30T13:27:00.519 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2023-10-30T13:27:00.519 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2023-10-30T13:27:00.519 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2023-10-30T13:27:00.519 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2023-10-30T13:27:00.519 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2023-10-30T13:27:00.519 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2023-10-30T13:27:00.520 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2023-10-30T13:27:00.520 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2023-10-30T13:27:00.520 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2023-10-30T13:27:00.520 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2023-10-30T13:27:00.520 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2023-10-30T13:27:00.520 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2023-10-30T13:27:00.521 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2023-10-30T13:27:00.521 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2023-10-30T13:27:00.521 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2023-10-30T13:27:00.521 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2023-10-30T13:27:00.521 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2023-10-30T13:27:00.522 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2023-10-30T13:27:00.522 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2023-10-30T13:27:02.123 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2023-10-30T13:27:02.124 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2023-10-30T13:27:02.124 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2023-10-30T13:27:02.124 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2023-10-30T13:27:02.124 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2023-10-30T13:27:02.125 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2023-10-30T13:27:02.125 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2023-10-30T13:27:02.125 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2023-10-30T13:27:02.125 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2023-10-30T13:27:02.125 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2023-10-30T13:27:02.126 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2023-10-30T13:27:02.126 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2023-10-30T13:27:02.126 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2023-10-30T13:27:02.126 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2023-10-30T13:27:02.126 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2023-10-30T13:27:02.127 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2023-10-30T13:27:02.127 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2023-10-30T13:27:02.127 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2023-10-30T13:27:02.127 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2023-10-30T13:27:02.127 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2023-10-30T13:27:02.128 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2023-10-30T13:27:02.128 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2023-10-30T13:27:02.128 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2023-10-30T13:27:02.128 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2023-10-30T13:27:02.128 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2023-10-30T13:27:02.128 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2023-10-30T13:27:02.129 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2023-10-30T13:27:02.129 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2023-10-30T13:27:02.129 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2023-10-30T13:27:02.129 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2023-10-30T13:27:02.129 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2023-10-30T13:27:02.130 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2023-10-30T13:27:02.130 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2023-10-30T13:27:02.130 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2023-10-30T13:27:02.130 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2023-10-30T13:27:02.130 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2023-10-30T13:27:02.131 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2023-10-30T13:27:02.131 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2023-10-30T13:27:02.131 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2023-10-30T13:27:02.131 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2023-10-30T13:27:02.131 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2023-10-30T13:27:02.132 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2023-10-30T13:27:02.132 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2023-10-30T13:27:02.132 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2023-10-30T13:27:02.132 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2023-10-30T13:27:02.132 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2023-10-30T13:27:02.133 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2023-10-30T13:27:02.133 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2023-10-30T13:27:02.133 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2023-10-30T13:27:02.133 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2023-10-30T13:27:02.133 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2023-10-30T13:27:02.133 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2023-10-30T13:27:02.134 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2023-10-30T13:27:02.134 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2023-10-30T13:27:02.134 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2023-10-30T13:27:02.134 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2023-10-30T13:27:02.134 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2023-10-30T13:27:02.135 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2023-10-30T13:27:02.135 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2023-10-30T13:27:02.135 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2023-10-30T13:27:02.135 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2023-10-30T13:27:02.135 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2023-10-30T13:27:02.136 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2023-10-30T13:27:02.136 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2023-10-30T13:27:02.136 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2023-10-30T13:27:02.136 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2023-10-30T13:27:02.136 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2023-10-30T13:27:02.137 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2023-10-30T13:27:02.137 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2023-10-30T13:27:02.137 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2023-10-30T13:27:02.137 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2023-10-30T13:27:02.137 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2023-10-30T13:27:02.138 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2023-10-30T13:27:02.138 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2023-10-30T13:27:02.138 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2023-10-30T13:27:02.138 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2023-10-30T13:27:02.138 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2023-10-30T13:27:02.139 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2023-10-30T13:27:02.139 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2023-10-30T13:27:02.139 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2023-10-30T13:27:02.139 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2023-10-30T13:27:03.342 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2023-10-30T13:27:03.342 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2023-10-30T13:27:03.343 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2023-10-30T13:27:03.343 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2023-10-30T13:27:03.343 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2023-10-30T13:27:03.343 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2023-10-30T13:27:03.343 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2023-10-30T13:27:03.344 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2023-10-30T13:27:03.344 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2023-10-30T13:27:03.344 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2023-10-30T13:27:03.344 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2023-10-30T13:27:03.344 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2023-10-30T13:27:03.344 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2023-10-30T13:27:03.345 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2023-10-30T13:27:03.345 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2023-10-30T13:27:03.345 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2023-10-30T13:27:03.345 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2023-10-30T13:27:03.345 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2023-10-30T13:27:03.346 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2023-10-30T13:27:03.346 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2023-10-30T13:27:03.346 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2023-10-30T13:27:03.346 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2023-10-30T13:27:03.346 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2023-10-30T13:27:03.347 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2023-10-30T13:27:03.347 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2023-10-30T13:27:03.347 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2023-10-30T13:27:03.347 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2023-10-30T13:27:03.347 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2023-10-30T13:27:03.347 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2023-10-30T13:27:03.348 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2023-10-30T13:27:03.348 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2023-10-30T13:27:03.348 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2023-10-30T13:27:03.348 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2023-10-30T13:27:03.348 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2023-10-30T13:27:03.349 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2023-10-30T13:27:03.349 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2023-10-30T13:27:03.349 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2023-10-30T13:27:03.349 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2023-10-30T13:27:03.349 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2023-10-30T13:27:03.349 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2023-10-30T13:27:03.350 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2023-10-30T13:27:03.350 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2023-10-30T13:27:03.350 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2023-10-30T13:27:03.350 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2023-10-30T13:27:03.350 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2023-10-30T13:27:03.351 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2023-10-30T13:27:03.351 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2023-10-30T13:27:03.351 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2023-10-30T13:27:03.351 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2023-10-30T13:27:03.351 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2023-10-30T13:27:03.351 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2023-10-30T13:27:03.352 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2023-10-30T13:27:03.352 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2023-10-30T13:27:03.352 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2023-10-30T13:27:03.352 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2023-10-30T13:27:03.352 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2023-10-30T13:27:03.352 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2023-10-30T13:27:03.353 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2023-10-30T13:27:03.353 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2023-10-30T13:27:03.353 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2023-10-30T13:27:03.353 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2023-10-30T13:27:03.353 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2023-10-30T13:27:03.353 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2023-10-30T13:27:03.354 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2023-10-30T13:27:03.354 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2023-10-30T13:27:03.354 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2023-10-30T13:27:03.354 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2023-10-30T13:27:03.354 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2023-10-30T13:27:03.355 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2023-10-30T13:27:03.355 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2023-10-30T13:27:03.355 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2023-10-30T13:27:03.355 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2023-10-30T13:27:03.355 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2023-10-30T13:27:03.356 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2023-10-30T13:27:03.356 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2023-10-30T13:27:03.356 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2023-10-30T13:27:03.356 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2023-10-30T13:27:03.356 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2023-10-30T13:27:03.356 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2023-10-30T13:27:03.357 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2023-10-30T13:27:03.357 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2023-10-30T13:27:03.509 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2023-10-30T13:27:03.509 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2023-10-30T13:27:03.509 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2023-10-30T13:27:03.510 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2023-10-30T13:27:03.510 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2023-10-30T13:27:03.510 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2023-10-30T13:27:03.510 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2023-10-30T13:27:03.510 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2023-10-30T13:27:03.510 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2023-10-30T13:27:03.511 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2023-10-30T13:27:03.511 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2023-10-30T13:27:03.511 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2023-10-30T13:27:03.511 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2023-10-30T13:27:03.511 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2023-10-30T13:27:03.511 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2023-10-30T13:27:03.512 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2023-10-30T13:27:03.512 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2023-10-30T13:27:03.512 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2023-10-30T13:27:03.512 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2023-10-30T13:27:03.512 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2023-10-30T13:27:03.512 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2023-10-30T13:27:03.513 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2023-10-30T13:27:03.513 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2023-10-30T13:27:03.513 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2023-10-30T13:27:03.513 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2023-10-30T13:27:03.513 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2023-10-30T13:27:03.514 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2023-10-30T13:27:03.514 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2023-10-30T13:27:03.514 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2023-10-30T13:27:03.514 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2023-10-30T13:27:03.514 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2023-10-30T13:27:03.514 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2023-10-30T13:27:03.515 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2023-10-30T13:27:03.515 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2023-10-30T13:27:03.515 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2023-10-30T13:27:03.515 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2023-10-30T13:27:03.515 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2023-10-30T13:27:03.515 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2023-10-30T13:27:03.516 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2023-10-30T13:27:03.516 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2023-10-30T13:27:03.516 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2023-10-30T13:27:03.516 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2023-10-30T13:27:03.516 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2023-10-30T13:27:03.517 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2023-10-30T13:27:03.517 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2023-10-30T13:27:03.517 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2023-10-30T13:27:03.517 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2023-10-30T13:27:03.517 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2023-10-30T13:27:03.517 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2023-10-30T13:27:03.518 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2023-10-30T13:27:03.518 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2023-10-30T13:27:03.518 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2023-10-30T13:27:03.518 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2023-10-30T13:27:03.518 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2023-10-30T13:27:03.518 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2023-10-30T13:27:03.519 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2023-10-30T13:27:03.519 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2023-10-30T13:27:03.519 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2023-10-30T13:27:03.519 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2023-10-30T13:27:03.519 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2023-10-30T13:27:03.519 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2023-10-30T13:27:03.520 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2023-10-30T13:27:03.520 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2023-10-30T13:27:03.520 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2023-10-30T13:27:03.520 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2023-10-30T13:27:03.520 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2023-10-30T13:27:03.520 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2023-10-30T13:27:03.521 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2023-10-30T13:27:03.521 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2023-10-30T13:27:03.521 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2023-10-30T13:27:03.521 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2023-10-30T13:27:03.521 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2023-10-30T13:27:03.521 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2023-10-30T13:27:03.522 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2023-10-30T13:27:03.522 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2023-10-30T13:27:03.522 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2023-10-30T13:27:03.522 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2023-10-30T13:27:03.522 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2023-10-30T13:27:03.522 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2023-10-30T13:27:03.523 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2023-10-30T13:27:03.523 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2023-10-30T13:27:04.490 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2023-10-30T13:27:04.490 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2023-10-30T13:27:04.490 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2023-10-30T13:27:04.490 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2023-10-30T13:27:04.491 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2023-10-30T13:27:04.491 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2023-10-30T13:27:04.491 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2023-10-30T13:27:04.491 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2023-10-30T13:27:04.491 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2023-10-30T13:27:04.492 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2023-10-30T13:27:04.492 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2023-10-30T13:27:04.492 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2023-10-30T13:27:04.492 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2023-10-30T13:27:04.492 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2023-10-30T13:27:04.493 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2023-10-30T13:27:04.493 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2023-10-30T13:27:04.493 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2023-10-30T13:27:04.493 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2023-10-30T13:27:04.494 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2023-10-30T13:27:04.494 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2023-10-30T13:27:04.494 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2023-10-30T13:27:04.494 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2023-10-30T13:27:04.494 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2023-10-30T13:27:04.495 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2023-10-30T13:27:04.495 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2023-10-30T13:27:04.495 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2023-10-30T13:27:04.495 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2023-10-30T13:27:04.495 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2023-10-30T13:27:04.496 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2023-10-30T13:27:04.496 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2023-10-30T13:27:04.496 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2023-10-30T13:27:04.496 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2023-10-30T13:27:04.496 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2023-10-30T13:27:04.497 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2023-10-30T13:27:04.497 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2023-10-30T13:27:04.497 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2023-10-30T13:27:04.497 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2023-10-30T13:27:04.497 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2023-10-30T13:27:04.498 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2023-10-30T13:27:04.498 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2023-10-30T13:27:04.498 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2023-10-30T13:27:04.498 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2023-10-30T13:27:04.498 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2023-10-30T13:27:04.499 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2023-10-30T13:27:04.499 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2023-10-30T13:27:04.499 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2023-10-30T13:27:04.499 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2023-10-30T13:27:04.499 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2023-10-30T13:27:04.500 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2023-10-30T13:27:04.500 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2023-10-30T13:27:04.500 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2023-10-30T13:27:04.500 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2023-10-30T13:27:04.500 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2023-10-30T13:27:04.501 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2023-10-30T13:27:04.501 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2023-10-30T13:27:04.501 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2023-10-30T13:27:04.501 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2023-10-30T13:27:04.501 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2023-10-30T13:27:04.502 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2023-10-30T13:27:04.502 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2023-10-30T13:27:04.502 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2023-10-30T13:27:04.502 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2023-10-30T13:27:04.502 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2023-10-30T13:27:04.503 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2023-10-30T13:27:04.503 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2023-10-30T13:27:04.503 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2023-10-30T13:27:04.503 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2023-10-30T13:27:04.503 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2023-10-30T13:27:04.504 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2023-10-30T13:27:04.504 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2023-10-30T13:27:04.504 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2023-10-30T13:27:04.504 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2023-10-30T13:27:04.504 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2023-10-30T13:27:04.505 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2023-10-30T13:27:04.505 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2023-10-30T13:27:04.505 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2023-10-30T13:27:04.505 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2023-10-30T13:27:04.505 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2023-10-30T13:27:04.506 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2023-10-30T13:27:04.506 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2023-10-30T13:27:04.506 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2023-10-30T13:27:05.459 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2023-10-30T13:27:05.460 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2023-10-30T13:27:05.460 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2023-10-30T13:27:05.460 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2023-10-30T13:27:05.460 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2023-10-30T13:27:05.460 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2023-10-30T13:27:05.460 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2023-10-30T13:27:05.461 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2023-10-30T13:27:05.461 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2023-10-30T13:27:05.461 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2023-10-30T13:27:05.461 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2023-10-30T13:27:05.461 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2023-10-30T13:27:05.461 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2023-10-30T13:27:05.462 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2023-10-30T13:27:05.462 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2023-10-30T13:27:05.462 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2023-10-30T13:27:05.462 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2023-10-30T13:27:05.462 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2023-10-30T13:27:05.463 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2023-10-30T13:27:05.463 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2023-10-30T13:27:05.463 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2023-10-30T13:27:05.463 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2023-10-30T13:27:05.463 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2023-10-30T13:27:05.463 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2023-10-30T13:27:05.464 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2023-10-30T13:27:05.464 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2023-10-30T13:27:05.464 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2023-10-30T13:27:05.464 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2023-10-30T13:27:05.464 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2023-10-30T13:27:05.464 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2023-10-30T13:27:05.465 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2023-10-30T13:27:05.465 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2023-10-30T13:27:05.465 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2023-10-30T13:27:05.465 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2023-10-30T13:27:05.465 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2023-10-30T13:27:05.466 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2023-10-30T13:27:05.466 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2023-10-30T13:27:05.466 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2023-10-30T13:27:05.466 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2023-10-30T13:27:05.466 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2023-10-30T13:27:05.466 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2023-10-30T13:27:05.467 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2023-10-30T13:27:05.467 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2023-10-30T13:27:05.467 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2023-10-30T13:27:05.467 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2023-10-30T13:27:05.467 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2023-10-30T13:27:05.468 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2023-10-30T13:27:05.468 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2023-10-30T13:27:05.468 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2023-10-30T13:27:05.468 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2023-10-30T13:27:05.468 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2023-10-30T13:27:05.468 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2023-10-30T13:27:05.469 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2023-10-30T13:27:05.469 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2023-10-30T13:27:05.469 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2023-10-30T13:27:05.469 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2023-10-30T13:27:05.469 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2023-10-30T13:27:05.469 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2023-10-30T13:27:05.470 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2023-10-30T13:27:05.470 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2023-10-30T13:27:05.470 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2023-10-30T13:27:05.470 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2023-10-30T13:27:05.470 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2023-10-30T13:27:05.470 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2023-10-30T13:27:05.471 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2023-10-30T13:27:05.471 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2023-10-30T13:27:05.471 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2023-10-30T13:27:05.471 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2023-10-30T13:27:05.471 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2023-10-30T13:27:05.471 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2023-10-30T13:27:05.472 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2023-10-30T13:27:05.472 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2023-10-30T13:27:05.472 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2023-10-30T13:27:05.472 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2023-10-30T13:27:05.472 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2023-10-30T13:27:05.472 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2023-10-30T13:27:05.473 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2023-10-30T13:27:05.473 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2023-10-30T13:27:05.473 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2023-10-30T13:27:05.473 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2023-10-30T13:27:05.473 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2023-10-30T13:27:06.854 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2023-10-30T13:27:06.855 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2023-10-30T13:27:06.855 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2023-10-30T13:27:06.855 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2023-10-30T13:27:06.855 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2023-10-30T13:27:06.856 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2023-10-30T13:27:06.856 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2023-10-30T13:27:06.856 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2023-10-30T13:27:06.856 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2023-10-30T13:27:06.856 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2023-10-30T13:27:06.857 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2023-10-30T13:27:06.857 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2023-10-30T13:27:06.857 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2023-10-30T13:27:06.857 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2023-10-30T13:27:06.857 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2023-10-30T13:27:06.857 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2023-10-30T13:27:06.858 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2023-10-30T13:27:06.858 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2023-10-30T13:27:06.858 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2023-10-30T13:27:06.858 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2023-10-30T13:27:06.858 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2023-10-30T13:27:06.859 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2023-10-30T13:27:06.859 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2023-10-30T13:27:06.859 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2023-10-30T13:27:06.859 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2023-10-30T13:27:06.859 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2023-10-30T13:27:06.859 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2023-10-30T13:27:06.860 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2023-10-30T13:27:06.860 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2023-10-30T13:27:06.860 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2023-10-30T13:27:06.860 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2023-10-30T13:27:06.860 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2023-10-30T13:27:06.861 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2023-10-30T13:27:06.861 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2023-10-30T13:27:06.861 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2023-10-30T13:27:06.861 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2023-10-30T13:27:06.861 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2023-10-30T13:27:06.861 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2023-10-30T13:27:06.862 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2023-10-30T13:27:06.862 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2023-10-30T13:27:06.862 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2023-10-30T13:27:06.862 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2023-10-30T13:27:06.862 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2023-10-30T13:27:06.862 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2023-10-30T13:27:06.863 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2023-10-30T13:27:06.863 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2023-10-30T13:27:06.863 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2023-10-30T13:27:06.863 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2023-10-30T13:27:06.863 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2023-10-30T13:27:06.864 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2023-10-30T13:27:06.864 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2023-10-30T13:27:06.864 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2023-10-30T13:27:06.864 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2023-10-30T13:27:06.864 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2023-10-30T13:27:06.864 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2023-10-30T13:27:06.865 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2023-10-30T13:27:06.865 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2023-10-30T13:27:06.865 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2023-10-30T13:27:06.865 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2023-10-30T13:27:06.865 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2023-10-30T13:27:06.866 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2023-10-30T13:27:06.866 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2023-10-30T13:27:06.866 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2023-10-30T13:27:06.866 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2023-10-30T13:27:06.866 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2023-10-30T13:27:06.867 INFO:tasks.workunit.client.0.smithi046.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2023-10-30T13:27:06.867 INFO:tasks.workunit.client.0.smithi046.stdout:Take third snapshot .snap/test3 2023-10-30T13:27:06.912 INFO:tasks.workunit.client.0.smithi046.stdout:Delete the snapshots... 2023-10-30T13:27:13.248 INFO:tasks.workunit.client.0.smithi046.stdout:.//.snap/test1 2023-10-30T13:27:13.248 INFO:tasks.workunit.client.0.smithi046.stdout:.//.snap/test2 2023-10-30T13:27:13.249 INFO:tasks.workunit.client.0.smithi046.stdout:.//.snap/test3 2023-10-30T13:27:13.255 INFO:tasks.workunit.client.0.smithi046.stdout:./274/.snap/snap-subdir-test 2023-10-30T13:27:13.260 INFO:tasks.workunit.client.0.smithi046.stdout:./166/.snap/snap-subdir-test 2023-10-30T13:27:13.266 INFO:tasks.workunit.client.0.smithi046.stdout:./256/.snap/snap-subdir-test 2023-10-30T13:27:13.273 INFO:tasks.workunit.client.0.smithi046.stdout:./247/.snap/snap-subdir-test 2023-10-30T13:27:13.281 INFO:tasks.workunit.client.0.smithi046.stdout:./248/.snap/snap-subdir-test 2023-10-30T13:27:13.285 INFO:tasks.workunit.client.0.smithi046.stdout:./147/.snap/snap-subdir-test 2023-10-30T13:27:13.290 INFO:tasks.workunit.client.0.smithi046.stdout:./151/.snap/snap-subdir-test 2023-10-30T13:27:13.299 INFO:tasks.workunit.client.0.smithi046.stdout:./170/.snap/snap-subdir-test 2023-10-30T13:27:13.303 INFO:tasks.workunit.client.0.smithi046.stdout:./167/.snap/snap-subdir-test 2023-10-30T13:27:13.307 INFO:tasks.workunit.client.0.smithi046.stdout:./140/.snap/snap-subdir-test 2023-10-30T13:27:13.312 INFO:tasks.workunit.client.0.smithi046.stdout:./142/.snap/snap-subdir-test 2023-10-30T13:27:13.316 INFO:tasks.workunit.client.0.smithi046.stdout:./150/.snap/snap-subdir-test 2023-10-30T13:27:13.325 INFO:tasks.workunit.client.0.smithi046.stdout:./273/.snap/snap-subdir-test 2023-10-30T13:27:13.331 INFO:tasks.workunit.client.0.smithi046.stdout:./269/.snap/snap-subdir-test 2023-10-30T13:27:13.340 INFO:tasks.workunit.client.0.smithi046.stdout:./105/.snap/snap-subdir-test 2023-10-30T13:27:13.344 INFO:tasks.workunit.client.0.smithi046.stdout:./210/.snap/snap-subdir-test 2023-10-30T13:27:13.352 INFO:tasks.workunit.client.0.smithi046.stdout:./265/.snap/snap-subdir-test 2023-10-30T13:27:13.357 INFO:tasks.workunit.client.0.smithi046.stdout:./218/.snap/snap-subdir-test 2023-10-30T13:27:13.365 INFO:tasks.workunit.client.0.smithi046.stdout:./290/.snap/snap-subdir-test 2023-10-30T13:27:13.370 INFO:tasks.workunit.client.0.smithi046.stdout:./149/.snap/snap-subdir-test 2023-10-30T13:27:13.380 INFO:tasks.workunit.client.0.smithi046.stdout:./109/.snap/snap-subdir-test 2023-10-30T13:27:13.389 INFO:tasks.workunit.client.0.smithi046.stdout:./292/.snap/snap-subdir-test 2023-10-30T13:27:13.396 INFO:tasks.workunit.client.0.smithi046.stdout:./242/.snap/snap-subdir-test 2023-10-30T13:27:13.401 INFO:tasks.workunit.client.0.smithi046.stdout:./158/.snap/snap-subdir-test 2023-10-30T13:27:13.408 INFO:tasks.workunit.client.0.smithi046.stdout:./243/.snap/snap-subdir-test 2023-10-30T13:27:13.438 INFO:tasks.workunit.client.0.smithi046.stdout:./106/.snap/snap-subdir-test 2023-10-30T13:27:13.629 INFO:tasks.workunit.client.0.smithi046.stdout:./250/.snap/snap-subdir-test 2023-10-30T13:27:13.634 INFO:tasks.workunit.client.0.smithi046.stdout:./221/.snap/snap-subdir-test 2023-10-30T13:27:13.643 INFO:tasks.workunit.client.0.smithi046.stdout:./244/.snap/snap-subdir-test 2023-10-30T13:27:13.648 INFO:tasks.workunit.client.0.smithi046.stdout:./258/.snap/snap-subdir-test 2023-10-30T13:27:13.653 INFO:tasks.workunit.client.0.smithi046.stdout:./268/.snap/snap-subdir-test 2023-10-30T13:27:13.660 INFO:tasks.workunit.client.0.smithi046.stdout:./189/.snap/snap-subdir-test 2023-10-30T13:27:13.666 INFO:tasks.workunit.client.0.smithi046.stdout:./285/.snap/snap-subdir-test 2023-10-30T13:27:13.670 INFO:tasks.workunit.client.0.smithi046.stdout:./208/.snap/snap-subdir-test 2023-10-30T13:27:13.679 INFO:tasks.workunit.client.0.smithi046.stdout:./288/.snap/snap-subdir-test 2023-10-30T13:27:13.683 INFO:tasks.workunit.client.0.smithi046.stdout:./209/.snap/snap-subdir-test 2023-10-30T13:27:13.690 INFO:tasks.workunit.client.0.smithi046.stdout:./253/.snap/snap-subdir-test 2023-10-30T13:27:13.698 INFO:tasks.workunit.client.0.smithi046.stdout:./249/.snap/snap-subdir-test 2023-10-30T13:27:13.707 INFO:tasks.workunit.client.0.smithi046.stdout:./133/.snap/snap-subdir-test 2023-10-30T13:27:13.713 INFO:tasks.workunit.client.0.smithi046.stdout:./291/.snap/snap-subdir-test 2023-10-30T13:27:13.717 INFO:tasks.workunit.client.0.smithi046.stdout:./226/.snap/snap-subdir-test 2023-10-30T13:27:13.726 INFO:tasks.workunit.client.0.smithi046.stdout:./173/.snap/snap-subdir-test 2023-10-30T13:27:13.735 INFO:tasks.workunit.client.0.smithi046.stdout:./174/.snap/snap-subdir-test 2023-10-30T13:27:13.743 INFO:tasks.workunit.client.0.smithi046.stdout:./299/.snap/snap-subdir-test 2023-10-30T13:27:13.751 INFO:tasks.workunit.client.0.smithi046.stdout:./262/.snap/snap-subdir-test 2023-10-30T13:27:13.757 INFO:tasks.workunit.client.0.smithi046.stdout:./231/.snap/snap-subdir-test 2023-10-30T13:27:13.761 INFO:tasks.workunit.client.0.smithi046.stdout:./214/.snap/snap-subdir-test 2023-10-30T13:27:13.770 INFO:tasks.workunit.client.0.smithi046.stdout:./281/.snap/snap-subdir-test 2023-10-30T13:27:13.777 INFO:tasks.workunit.client.0.smithi046.stdout:./289/.snap/snap-subdir-test 2023-10-30T13:27:13.784 INFO:tasks.workunit.client.0.smithi046.stdout:./283/.snap/snap-subdir-test 2023-10-30T13:27:13.792 INFO:tasks.workunit.client.0.smithi046.stdout:./232/.snap/snap-subdir-test 2023-10-30T13:27:13.801 INFO:tasks.workunit.client.0.smithi046.stdout:./298/.snap/snap-subdir-test 2023-10-30T13:27:13.806 INFO:tasks.workunit.client.0.smithi046.stdout:./233/.snap/snap-subdir-test 2023-10-30T13:27:13.811 INFO:tasks.workunit.client.0.smithi046.stdout:./155/.snap/snap-subdir-test 2023-10-30T13:27:14.167 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:14.171 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:14.779 INFO:tasks.workunit.client.0.smithi046.stdout:./114/.snap/snap-subdir-test 2023-10-30T13:27:15.023 INFO:tasks.workunit.client.0.smithi046.stdout:./159/.snap/snap-subdir-test 2023-10-30T13:27:15.474 INFO:tasks.workunit.client.0.smithi046.stdout:./187/.snap/snap-subdir-test 2023-10-30T13:27:20.583 INFO:tasks.workunit.client.0.smithi046.stdout:./185/.snap/snap-subdir-test 2023-10-30T13:27:21.297 INFO:tasks.workunit.client.0.smithi046.stdout:./293/.snap/snap-subdir-test 2023-10-30T13:27:21.307 INFO:tasks.workunit.client.0.smithi046.stdout:./228/.snap/snap-subdir-test 2023-10-30T13:27:21.316 INFO:tasks.workunit.client.0.smithi046.stdout:./120/.snap/snap-subdir-test 2023-10-30T13:27:21.327 INFO:tasks.workunit.client.0.smithi046.stdout:./110/.snap/snap-subdir-test 2023-10-30T13:27:21.334 INFO:tasks.workunit.client.0.smithi046.stdout:./267/.snap/snap-subdir-test 2023-10-30T13:27:21.339 INFO:tasks.workunit.client.0.smithi046.stdout:./202/.snap/snap-subdir-test 2023-10-30T13:27:21.343 INFO:tasks.workunit.client.0.smithi046.stdout:./137/.snap/snap-subdir-test 2023-10-30T13:27:21.350 INFO:tasks.workunit.client.0.smithi046.stdout:./246/.snap/snap-subdir-test 2023-10-30T13:27:21.355 INFO:tasks.workunit.client.0.smithi046.stdout:./219/.snap/snap-subdir-test 2023-10-30T13:27:21.360 INFO:tasks.workunit.client.0.smithi046.stdout:./198/.snap/snap-subdir-test 2023-10-30T13:27:21.364 INFO:tasks.workunit.client.0.smithi046.stdout:./136/.snap/snap-subdir-test 2023-10-30T13:27:21.373 INFO:tasks.workunit.client.0.smithi046.stdout:./270/.snap/snap-subdir-test 2023-10-30T13:27:21.381 INFO:tasks.workunit.client.0.smithi046.stdout:./125/.snap/snap-subdir-test 2023-10-30T13:27:21.389 INFO:tasks.workunit.client.0.smithi046.stdout:./194/.snap/snap-subdir-test 2023-10-30T13:27:21.397 INFO:tasks.workunit.client.0.smithi046.stdout:./119/.snap/snap-subdir-test 2023-10-30T13:27:21.402 INFO:tasks.workunit.client.0.smithi046.stdout:./297/.snap/snap-subdir-test 2023-10-30T13:27:21.411 INFO:tasks.workunit.client.0.smithi046.stdout:./230/.snap/snap-subdir-test 2023-10-30T13:27:21.420 INFO:tasks.workunit.client.0.smithi046.stdout:./260/.snap/snap-subdir-test 2023-10-30T13:27:21.426 INFO:tasks.workunit.client.0.smithi046.stdout:./157/.snap/snap-subdir-test 2023-10-30T13:27:21.432 INFO:tasks.workunit.client.0.smithi046.stdout:./172/.snap/snap-subdir-test 2023-10-30T13:27:22.367 INFO:tasks.workunit.client.0.smithi046.stdout:./277/.snap/snap-subdir-test 2023-10-30T13:27:22.376 INFO:tasks.workunit.client.0.smithi046.stdout:./186/.snap/snap-subdir-test 2023-10-30T13:27:22.386 INFO:tasks.workunit.client.0.smithi046.stdout:./237/.snap/snap-subdir-test 2023-10-30T13:27:22.393 INFO:tasks.workunit.client.0.smithi046.stdout:./190/.snap/snap-subdir-test 2023-10-30T13:27:22.398 INFO:tasks.workunit.client.0.smithi046.stdout:./213/.snap/snap-subdir-test 2023-10-30T13:27:22.662 INFO:tasks.workunit.client.0.smithi046.stdout:./104/.snap/snap-subdir-test 2023-10-30T13:27:22.666 INFO:tasks.workunit.client.0.smithi046.stdout:./164/.snap/snap-subdir-test 2023-10-30T13:27:22.678 INFO:tasks.workunit.client.0.smithi046.stdout:./112/.snap/snap-subdir-test 2023-10-30T13:27:22.687 INFO:tasks.workunit.client.0.smithi046.stdout:./295/.snap/snap-subdir-test 2023-10-30T13:27:22.697 INFO:tasks.workunit.client.0.smithi046.stdout:./272/.snap/snap-subdir-test 2023-10-30T13:27:22.706 INFO:tasks.workunit.client.0.smithi046.stdout:./100/.snap/snap-subdir-test 2023-10-30T13:27:22.714 INFO:tasks.workunit.client.0.smithi046.stdout:./134/.snap/snap-subdir-test 2023-10-30T13:27:22.718 INFO:tasks.workunit.client.0.smithi046.stdout:./200/.snap/snap-subdir-test 2023-10-30T13:27:22.729 INFO:tasks.workunit.client.0.smithi046.stdout:./115/.snap/snap-subdir-test 2023-10-30T13:27:22.738 INFO:tasks.workunit.client.0.smithi046.stdout:./284/.snap/snap-subdir-test 2023-10-30T13:27:22.744 INFO:tasks.workunit.client.0.smithi046.stdout:./280/.snap/snap-subdir-test 2023-10-30T13:27:22.753 INFO:tasks.workunit.client.0.smithi046.stdout:./116/.snap/snap-subdir-test 2023-10-30T13:27:22.762 INFO:tasks.workunit.client.0.smithi046.stdout:./245/.snap/snap-subdir-test 2023-10-30T13:27:22.766 INFO:tasks.workunit.client.0.smithi046.stdout:./195/.snap/snap-subdir-test 2023-10-30T13:27:22.770 INFO:tasks.workunit.client.0.smithi046.stdout:./138/.snap/snap-subdir-test 2023-10-30T13:27:22.778 INFO:tasks.workunit.client.0.smithi046.stdout:./135/.snap/snap-subdir-test 2023-10-30T13:27:22.792 INFO:tasks.workunit.client.0.smithi046.stdout:./117/.snap/snap-subdir-test 2023-10-30T13:27:22.804 INFO:tasks.workunit.client.0.smithi046.stdout:./241/.snap/snap-subdir-test 2023-10-30T13:27:22.817 INFO:tasks.workunit.client.0.smithi046.stdout:./143/.snap/snap-subdir-test 2023-10-30T13:27:22.827 INFO:tasks.workunit.client.0.smithi046.stdout:./180/.snap/snap-subdir-test 2023-10-30T13:27:22.841 INFO:tasks.workunit.client.0.smithi046.stdout:./126/.snap/snap-subdir-test 2023-10-30T13:27:22.849 INFO:tasks.workunit.client.0.smithi046.stdout:./161/.snap/snap-subdir-test 2023-10-30T13:27:22.861 INFO:tasks.workunit.client.0.smithi046.stdout:./204/.snap/snap-subdir-test 2023-10-30T13:27:22.874 INFO:tasks.workunit.client.0.smithi046.stdout:./176/.snap/snap-subdir-test 2023-10-30T13:27:22.886 INFO:tasks.workunit.client.0.smithi046.stdout:./255/.snap/snap-subdir-test 2023-10-30T13:27:23.253 INFO:tasks.workunit.client.0.smithi046.stdout:./215/.snap/snap-subdir-test 2023-10-30T13:27:23.267 INFO:tasks.workunit.client.0.smithi046.stdout:./111/.snap/snap-subdir-test 2023-10-30T13:27:23.774 INFO:tasks.workunit.client.0.smithi046.stdout:./238/.snap/snap-subdir-test 2023-10-30T13:27:23.788 INFO:tasks.workunit.client.0.smithi046.stdout:./191/.snap/snap-subdir-test 2023-10-30T13:27:23.803 INFO:tasks.workunit.client.0.smithi046.stdout:./184/.snap/snap-subdir-test 2023-10-30T13:27:23.818 INFO:tasks.workunit.client.0.smithi046.stdout:./192/.snap/snap-subdir-test 2023-10-30T13:27:23.830 INFO:tasks.workunit.client.0.smithi046.stdout:./103/.snap/snap-subdir-test 2023-10-30T13:27:23.844 INFO:tasks.workunit.client.0.smithi046.stdout:./128/.snap/snap-subdir-test 2023-10-30T13:27:23.858 INFO:tasks.workunit.client.0.smithi046.stdout:./229/.snap/snap-subdir-test 2023-10-30T13:27:23.874 INFO:tasks.workunit.client.0.smithi046.stdout:./144/.snap/snap-subdir-test 2023-10-30T13:27:23.887 INFO:tasks.workunit.client.0.smithi046.stdout:./146/.snap/snap-subdir-test 2023-10-30T13:27:23.901 INFO:tasks.workunit.client.0.smithi046.stdout:./227/.snap/snap-subdir-test 2023-10-30T13:27:23.914 INFO:tasks.workunit.client.0.smithi046.stdout:./278/.snap/snap-subdir-test 2023-10-30T13:27:23.919 INFO:tasks.workunit.client.0.smithi046.stdout:./141/.snap/snap-subdir-test 2023-10-30T13:27:23.933 INFO:tasks.workunit.client.0.smithi046.stdout:./234/.snap/snap-subdir-test 2023-10-30T13:27:23.944 INFO:tasks.workunit.client.0.smithi046.stdout:./225/.snap/snap-subdir-test 2023-10-30T13:27:23.958 INFO:tasks.workunit.client.0.smithi046.stdout:./182/.snap/snap-subdir-test 2023-10-30T13:27:24.752 INFO:tasks.workunit.client.0.smithi046.stdout:./107/.snap/snap-subdir-test 2023-10-30T13:27:24.763 INFO:tasks.workunit.client.0.smithi046.stdout:./178/.snap/snap-subdir-test 2023-10-30T13:27:24.771 INFO:tasks.workunit.client.0.smithi046.stdout:./266/.snap/snap-subdir-test 2023-10-30T13:27:24.781 INFO:tasks.workunit.client.0.smithi046.stdout:./240/.snap/snap-subdir-test 2023-10-30T13:27:24.786 INFO:tasks.workunit.client.0.smithi046.stdout:./156/.snap/snap-subdir-test 2023-10-30T13:27:24.794 INFO:tasks.workunit.client.0.smithi046.stdout:./131/.snap/snap-subdir-test 2023-10-30T13:27:24.804 INFO:tasks.workunit.client.0.smithi046.stdout:./123/.snap/snap-subdir-test 2023-10-30T13:27:24.809 INFO:tasks.workunit.client.0.smithi046.stdout:./160/.snap/snap-subdir-test 2023-10-30T13:27:25.065 INFO:tasks.workunit.client.0.smithi046.stdout:./165/.snap/snap-subdir-test 2023-10-30T13:27:25.076 INFO:tasks.workunit.client.0.smithi046.stdout:./101/.snap/snap-subdir-test 2023-10-30T13:27:25.081 INFO:tasks.workunit.client.0.smithi046.stdout:./148/.snap/snap-subdir-test 2023-10-30T13:27:25.090 INFO:tasks.workunit.client.0.smithi046.stdout:./168/.snap/snap-subdir-test 2023-10-30T13:27:25.100 INFO:tasks.workunit.client.0.smithi046.stdout:./179/.snap/snap-subdir-test 2023-10-30T13:27:25.107 INFO:tasks.workunit.client.0.smithi046.stdout:./257/.snap/snap-subdir-test 2023-10-30T13:27:25.114 INFO:tasks.workunit.client.0.smithi046.stdout:./252/.snap/snap-subdir-test 2023-10-30T13:27:25.123 INFO:tasks.workunit.client.0.smithi046.stdout:./132/.snap/snap-subdir-test 2023-10-30T13:27:25.131 INFO:tasks.workunit.client.0.smithi046.stdout:./236/.snap/snap-subdir-test 2023-10-30T13:27:25.135 INFO:tasks.workunit.client.0.smithi046.stdout:./211/.snap/snap-subdir-test 2023-10-30T13:27:25.139 INFO:tasks.workunit.client.0.smithi046.stdout:./199/.snap/snap-subdir-test 2023-10-30T13:27:25.144 INFO:tasks.workunit.client.0.smithi046.stdout:./205/.snap/snap-subdir-test 2023-10-30T13:27:25.149 INFO:tasks.workunit.client.0.smithi046.stdout:./163/.snap/snap-subdir-test 2023-10-30T13:27:25.157 INFO:tasks.workunit.client.0.smithi046.stdout:./261/.snap/snap-subdir-test 2023-10-30T13:27:25.161 INFO:tasks.workunit.client.0.smithi046.stdout:./201/.snap/snap-subdir-test 2023-10-30T13:27:25.171 INFO:tasks.workunit.client.0.smithi046.stdout:./282/.snap/snap-subdir-test 2023-10-30T13:27:25.179 INFO:tasks.workunit.client.0.smithi046.stdout:./296/.snap/snap-subdir-test 2023-10-30T13:27:25.184 INFO:tasks.workunit.client.0.smithi046.stdout:./217/.snap/snap-subdir-test 2023-10-30T13:27:25.189 INFO:tasks.workunit.client.0.smithi046.stdout:./153/.snap/snap-subdir-test 2023-10-30T13:27:25.193 INFO:tasks.workunit.client.0.smithi046.stdout:./206/.snap/snap-subdir-test 2023-10-30T13:27:25.201 INFO:tasks.workunit.client.0.smithi046.stdout:./122/.snap/snap-subdir-test 2023-10-30T13:27:25.205 INFO:tasks.workunit.client.0.smithi046.stdout:./162/.snap/snap-subdir-test 2023-10-30T13:27:25.209 INFO:tasks.workunit.client.0.smithi046.stdout:./197/.snap/snap-subdir-test 2023-10-30T13:27:25.215 INFO:tasks.workunit.client.0.smithi046.stdout:./216/.snap/snap-subdir-test 2023-10-30T13:27:31.006 INFO:tasks.workunit.client.0.smithi046.stdout:./254/.snap/snap-subdir-test 2023-10-30T13:27:31.014 INFO:tasks.workunit.client.0.smithi046.stdout:./113/.snap/snap-subdir-test 2023-10-30T13:27:31.019 INFO:tasks.workunit.client.0.smithi046.stdout:./152/.snap/snap-subdir-test 2023-10-30T13:27:31.028 INFO:tasks.workunit.client.0.smithi046.stdout:./121/.snap/snap-subdir-test 2023-10-30T13:27:31.288 INFO:tasks.workunit.client.0.smithi046.stdout:./118/.snap/snap-subdir-test 2023-10-30T13:27:31.292 INFO:tasks.workunit.client.0.smithi046.stdout:./196/.snap/snap-subdir-test 2023-10-30T13:27:31.300 INFO:tasks.workunit.client.0.smithi046.stdout:./124/.snap/snap-subdir-test 2023-10-30T13:27:31.309 INFO:tasks.workunit.client.0.smithi046.stdout:./239/.snap/snap-subdir-test 2023-10-30T13:27:31.318 INFO:tasks.workunit.client.0.smithi046.stdout:./188/.snap/snap-subdir-test 2023-10-30T13:27:31.327 INFO:tasks.workunit.client.0.smithi046.stdout:./287/.snap/snap-subdir-test 2023-10-30T13:27:31.334 INFO:tasks.workunit.client.0.smithi046.stdout:./275/.snap/snap-subdir-test 2023-10-30T13:27:31.342 INFO:tasks.workunit.client.0.smithi046.stdout:./259/.snap/snap-subdir-test 2023-10-30T13:27:31.347 INFO:tasks.workunit.client.0.smithi046.stdout:./224/.snap/snap-subdir-test 2023-10-30T13:27:31.487 INFO:tasks.workunit.client.0.smithi046.stdout:./264/.snap/snap-subdir-test 2023-10-30T13:27:31.492 INFO:tasks.workunit.client.0.smithi046.stdout:./145/.snap/snap-subdir-test 2023-10-30T13:27:31.498 INFO:tasks.workunit.client.0.smithi046.stdout:./279/.snap/snap-subdir-test 2023-10-30T13:27:31.508 INFO:tasks.workunit.client.0.smithi046.stdout:./263/.snap/snap-subdir-test 2023-10-30T13:27:31.521 INFO:tasks.workunit.client.0.smithi046.stdout:./130/.snap/snap-subdir-test 2023-10-30T13:27:31.534 INFO:tasks.workunit.client.0.smithi046.stdout:./193/.snap/snap-subdir-test 2023-10-30T13:27:31.548 INFO:tasks.workunit.client.0.smithi046.stdout:./223/.snap/snap-subdir-test 2023-10-30T13:27:31.555 INFO:tasks.workunit.client.0.smithi046.stdout:./207/.snap/snap-subdir-test 2023-10-30T13:27:31.568 INFO:tasks.workunit.client.0.smithi046.stdout:./276/.snap/snap-subdir-test 2023-10-30T13:27:31.579 INFO:tasks.workunit.client.0.smithi046.stdout:./271/.snap/snap-subdir-test 2023-10-30T13:27:32.535 INFO:tasks.workunit.client.0.smithi046.stdout:./251/.snap/snap-subdir-test 2023-10-30T13:27:32.548 INFO:tasks.workunit.client.0.smithi046.stdout:./175/.snap/snap-subdir-test 2023-10-30T13:27:32.562 INFO:tasks.workunit.client.0.smithi046.stdout:./222/.snap/snap-subdir-test 2023-10-30T13:27:32.567 INFO:tasks.workunit.client.0.smithi046.stdout:./220/.snap/snap-subdir-test 2023-10-30T13:27:32.575 INFO:tasks.workunit.client.0.smithi046.stdout:./203/.snap/snap-subdir-test 2023-10-30T13:27:33.251 INFO:tasks.workunit.client.0.smithi046.stdout:./181/.snap/snap-subdir-test 2023-10-30T13:27:33.263 INFO:tasks.workunit.client.0.smithi046.stdout:./286/.snap/snap-subdir-test 2023-10-30T13:27:33.277 INFO:tasks.workunit.client.0.smithi046.stdout:./127/.snap/snap-subdir-test 2023-10-30T13:27:33.291 INFO:tasks.workunit.client.0.smithi046.stdout:./108/.snap/snap-subdir-test 2023-10-30T13:27:33.431 INFO:tasks.workunit.client.0.smithi046.stdout:./183/.snap/snap-subdir-test 2023-10-30T13:27:33.582 INFO:tasks.workunit.client.0.smithi046.stdout:./294/.snap/snap-subdir-test 2023-10-30T13:27:34.029 INFO:tasks.workunit.client.0.smithi046.stdout:./235/.snap/snap-subdir-test 2023-10-30T13:27:34.040 INFO:tasks.workunit.client.0.smithi046.stdout:./177/.snap/snap-subdir-test 2023-10-30T13:27:34.053 INFO:tasks.workunit.client.0.smithi046.stdout:./169/.snap/snap-subdir-test 2023-10-30T13:27:34.059 INFO:tasks.workunit.client.0.smithi046.stdout:./212/.snap/snap-subdir-test 2023-10-30T13:27:34.065 INFO:tasks.workunit.client.0.smithi046.stdout:./154/.snap/snap-subdir-test 2023-10-30T13:27:34.078 INFO:tasks.workunit.client.0.smithi046.stdout:./171/.snap/snap-subdir-test 2023-10-30T13:27:34.086 INFO:tasks.workunit.client.0.smithi046.stdout:./139/.snap/snap-subdir-test 2023-10-30T13:27:34.100 INFO:tasks.workunit.client.0.smithi046.stdout:./102/.snap/snap-subdir-test 2023-10-30T13:27:34.114 INFO:tasks.workunit.client.0.smithi046.stdout:./129/.snap/snap-subdir-test 2023-10-30T13:27:44.321 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:44.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:46.146 INFO:tasks.workunit.client.0.smithi046.stdout:Delete all the files and directories ... 2023-10-30T13:27:48.867 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/19' 2023-10-30T13:27:48.867 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/15' 2023-10-30T13:27:48.867 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/31' 2023-10-30T13:27:48.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/36' 2023-10-30T13:27:48.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/20' 2023-10-30T13:27:48.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/12' 2023-10-30T13:27:48.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/39' 2023-10-30T13:27:48.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/10' 2023-10-30T13:27:48.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/34' 2023-10-30T13:27:48.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/38' 2023-10-30T13:27:48.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/11' 2023-10-30T13:27:48.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/16' 2023-10-30T13:27:48.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/32' 2023-10-30T13:27:48.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/30' 2023-10-30T13:27:48.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/14' 2023-10-30T13:27:48.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/18' 2023-10-30T13:27:48.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/13' 2023-10-30T13:27:48.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/33' 2023-10-30T13:27:48.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/17' 2023-10-30T13:27:48.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/35' 2023-10-30T13:27:48.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './100/37' 2023-10-30T13:27:48.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './100' 2023-10-30T13:27:48.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/19' 2023-10-30T13:27:48.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/15' 2023-10-30T13:27:48.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/31' 2023-10-30T13:27:48.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/36' 2023-10-30T13:27:48.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/20' 2023-10-30T13:27:48.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/12' 2023-10-30T13:27:48.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/39' 2023-10-30T13:27:48.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/10' 2023-10-30T13:27:48.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/34' 2023-10-30T13:27:48.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/38' 2023-10-30T13:27:48.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/11' 2023-10-30T13:27:48.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/16' 2023-10-30T13:27:48.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/32' 2023-10-30T13:27:48.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/30' 2023-10-30T13:27:48.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/14' 2023-10-30T13:27:48.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/18' 2023-10-30T13:27:48.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/13' 2023-10-30T13:27:48.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/33' 2023-10-30T13:27:48.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/17' 2023-10-30T13:27:48.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/35' 2023-10-30T13:27:48.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './101/37' 2023-10-30T13:27:48.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './101' 2023-10-30T13:27:48.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/19' 2023-10-30T13:27:48.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/15' 2023-10-30T13:27:48.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/31' 2023-10-30T13:27:48.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/36' 2023-10-30T13:27:48.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/20' 2023-10-30T13:27:48.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/12' 2023-10-30T13:27:48.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/39' 2023-10-30T13:27:48.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/10' 2023-10-30T13:27:48.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/34' 2023-10-30T13:27:48.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/38' 2023-10-30T13:27:48.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/11' 2023-10-30T13:27:48.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/16' 2023-10-30T13:27:48.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/32' 2023-10-30T13:27:48.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/30' 2023-10-30T13:27:48.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/14' 2023-10-30T13:27:48.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/18' 2023-10-30T13:27:48.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/13' 2023-10-30T13:27:48.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/33' 2023-10-30T13:27:48.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/17' 2023-10-30T13:27:48.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/35' 2023-10-30T13:27:48.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './102/37' 2023-10-30T13:27:48.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './102' 2023-10-30T13:27:48.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/19' 2023-10-30T13:27:48.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/15' 2023-10-30T13:27:48.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/31' 2023-10-30T13:27:48.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/36' 2023-10-30T13:27:48.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/20' 2023-10-30T13:27:48.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/12' 2023-10-30T13:27:48.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/39' 2023-10-30T13:27:48.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/10' 2023-10-30T13:27:48.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/34' 2023-10-30T13:27:48.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/38' 2023-10-30T13:27:48.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/11' 2023-10-30T13:27:48.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/16' 2023-10-30T13:27:48.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/32' 2023-10-30T13:27:48.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/30' 2023-10-30T13:27:48.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/14' 2023-10-30T13:27:48.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/18' 2023-10-30T13:27:48.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/13' 2023-10-30T13:27:48.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/33' 2023-10-30T13:27:48.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/17' 2023-10-30T13:27:48.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/35' 2023-10-30T13:27:48.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './103/37' 2023-10-30T13:27:48.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './103' 2023-10-30T13:27:48.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/19' 2023-10-30T13:27:48.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/15' 2023-10-30T13:27:48.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/31' 2023-10-30T13:27:48.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/36' 2023-10-30T13:27:48.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/20' 2023-10-30T13:27:48.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/12' 2023-10-30T13:27:48.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/39' 2023-10-30T13:27:48.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/10' 2023-10-30T13:27:48.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/34' 2023-10-30T13:27:48.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/38' 2023-10-30T13:27:48.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/11' 2023-10-30T13:27:48.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/16' 2023-10-30T13:27:48.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/32' 2023-10-30T13:27:48.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/30' 2023-10-30T13:27:48.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/14' 2023-10-30T13:27:48.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/18' 2023-10-30T13:27:48.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/13' 2023-10-30T13:27:48.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/33' 2023-10-30T13:27:48.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/17' 2023-10-30T13:27:48.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/35' 2023-10-30T13:27:48.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './104/37' 2023-10-30T13:27:48.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './104' 2023-10-30T13:27:48.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/19' 2023-10-30T13:27:48.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/15' 2023-10-30T13:27:48.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/31' 2023-10-30T13:27:48.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/36' 2023-10-30T13:27:48.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/20' 2023-10-30T13:27:48.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/12' 2023-10-30T13:27:48.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/39' 2023-10-30T13:27:48.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/10' 2023-10-30T13:27:48.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/34' 2023-10-30T13:27:48.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/38' 2023-10-30T13:27:48.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/11' 2023-10-30T13:27:48.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/16' 2023-10-30T13:27:48.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/32' 2023-10-30T13:27:48.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/30' 2023-10-30T13:27:48.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/14' 2023-10-30T13:27:48.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/18' 2023-10-30T13:27:48.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/13' 2023-10-30T13:27:48.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/33' 2023-10-30T13:27:48.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/17' 2023-10-30T13:27:48.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/35' 2023-10-30T13:27:48.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './105/37' 2023-10-30T13:27:48.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './105' 2023-10-30T13:27:48.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/19' 2023-10-30T13:27:48.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/15' 2023-10-30T13:27:48.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/31' 2023-10-30T13:27:48.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/36' 2023-10-30T13:27:48.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/20' 2023-10-30T13:27:48.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/12' 2023-10-30T13:27:48.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/39' 2023-10-30T13:27:48.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/10' 2023-10-30T13:27:48.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/34' 2023-10-30T13:27:48.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/38' 2023-10-30T13:27:48.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/11' 2023-10-30T13:27:48.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/16' 2023-10-30T13:27:48.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/32' 2023-10-30T13:27:48.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/30' 2023-10-30T13:27:48.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/14' 2023-10-30T13:27:48.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/18' 2023-10-30T13:27:48.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/13' 2023-10-30T13:27:48.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/33' 2023-10-30T13:27:48.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/17' 2023-10-30T13:27:48.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/35' 2023-10-30T13:27:48.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './106/37' 2023-10-30T13:27:48.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './106' 2023-10-30T13:27:48.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/19' 2023-10-30T13:27:48.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/15' 2023-10-30T13:27:48.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/31' 2023-10-30T13:27:48.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/36' 2023-10-30T13:27:48.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/20' 2023-10-30T13:27:48.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/12' 2023-10-30T13:27:48.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/39' 2023-10-30T13:27:48.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/10' 2023-10-30T13:27:48.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/34' 2023-10-30T13:27:48.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/38' 2023-10-30T13:27:48.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/11' 2023-10-30T13:27:48.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/16' 2023-10-30T13:27:48.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/32' 2023-10-30T13:27:48.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/30' 2023-10-30T13:27:48.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/14' 2023-10-30T13:27:48.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/18' 2023-10-30T13:27:48.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/13' 2023-10-30T13:27:48.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/33' 2023-10-30T13:27:48.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/17' 2023-10-30T13:27:48.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/35' 2023-10-30T13:27:48.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './107/37' 2023-10-30T13:27:48.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './107' 2023-10-30T13:27:48.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/19' 2023-10-30T13:27:48.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/15' 2023-10-30T13:27:48.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/31' 2023-10-30T13:27:48.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/36' 2023-10-30T13:27:48.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/20' 2023-10-30T13:27:48.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/12' 2023-10-30T13:27:48.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/39' 2023-10-30T13:27:48.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/10' 2023-10-30T13:27:48.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/34' 2023-10-30T13:27:48.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/38' 2023-10-30T13:27:48.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/11' 2023-10-30T13:27:48.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/16' 2023-10-30T13:27:48.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/32' 2023-10-30T13:27:48.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/30' 2023-10-30T13:27:48.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/14' 2023-10-30T13:27:48.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/18' 2023-10-30T13:27:48.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/13' 2023-10-30T13:27:48.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/33' 2023-10-30T13:27:48.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/17' 2023-10-30T13:27:48.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/35' 2023-10-30T13:27:48.905 INFO:tasks.workunit.client.0.smithi046.stdout:removed './108/37' 2023-10-30T13:27:48.905 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './108' 2023-10-30T13:27:48.905 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/19' 2023-10-30T13:27:48.905 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/15' 2023-10-30T13:27:48.905 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/31' 2023-10-30T13:27:48.906 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/36' 2023-10-30T13:27:48.906 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/20' 2023-10-30T13:27:48.906 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/12' 2023-10-30T13:27:48.906 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/39' 2023-10-30T13:27:48.906 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/10' 2023-10-30T13:27:48.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/34' 2023-10-30T13:27:48.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/38' 2023-10-30T13:27:48.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/11' 2023-10-30T13:27:48.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/16' 2023-10-30T13:27:48.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/32' 2023-10-30T13:27:48.908 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/30' 2023-10-30T13:27:50.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/14' 2023-10-30T13:27:50.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/18' 2023-10-30T13:27:50.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/13' 2023-10-30T13:27:50.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/33' 2023-10-30T13:27:50.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/17' 2023-10-30T13:27:50.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/35' 2023-10-30T13:27:50.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './109/37' 2023-10-30T13:27:50.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './109' 2023-10-30T13:27:50.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/19' 2023-10-30T13:27:50.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/15' 2023-10-30T13:27:50.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/31' 2023-10-30T13:27:50.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/36' 2023-10-30T13:27:50.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/20' 2023-10-30T13:27:50.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/12' 2023-10-30T13:27:50.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/39' 2023-10-30T13:27:50.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/10' 2023-10-30T13:27:50.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/34' 2023-10-30T13:27:50.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/38' 2023-10-30T13:27:50.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/11' 2023-10-30T13:27:50.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/16' 2023-10-30T13:27:50.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/32' 2023-10-30T13:27:50.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/30' 2023-10-30T13:27:50.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/14' 2023-10-30T13:27:50.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/18' 2023-10-30T13:27:50.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/13' 2023-10-30T13:27:50.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/33' 2023-10-30T13:27:50.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/17' 2023-10-30T13:27:50.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/35' 2023-10-30T13:27:50.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './110/37' 2023-10-30T13:27:50.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './110' 2023-10-30T13:27:50.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/19' 2023-10-30T13:27:50.670 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/15' 2023-10-30T13:27:50.670 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/31' 2023-10-30T13:27:50.670 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/36' 2023-10-30T13:27:50.670 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/20' 2023-10-30T13:27:50.670 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/12' 2023-10-30T13:27:50.671 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/39' 2023-10-30T13:27:50.671 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/10' 2023-10-30T13:27:50.671 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/34' 2023-10-30T13:27:50.671 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/38' 2023-10-30T13:27:50.671 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/11' 2023-10-30T13:27:50.671 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/16' 2023-10-30T13:27:50.672 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/32' 2023-10-30T13:27:50.672 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/30' 2023-10-30T13:27:50.672 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/14' 2023-10-30T13:27:50.672 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/18' 2023-10-30T13:27:50.672 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/13' 2023-10-30T13:27:50.673 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/33' 2023-10-30T13:27:50.673 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/17' 2023-10-30T13:27:50.673 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/35' 2023-10-30T13:27:50.673 INFO:tasks.workunit.client.0.smithi046.stdout:removed './111/37' 2023-10-30T13:27:50.673 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './111' 2023-10-30T13:27:50.673 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/19' 2023-10-30T13:27:50.674 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/15' 2023-10-30T13:27:50.674 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/31' 2023-10-30T13:27:50.674 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/36' 2023-10-30T13:27:50.674 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/20' 2023-10-30T13:27:50.674 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/12' 2023-10-30T13:27:50.675 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/39' 2023-10-30T13:27:50.675 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/10' 2023-10-30T13:27:50.675 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/34' 2023-10-30T13:27:50.675 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/38' 2023-10-30T13:27:50.675 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/11' 2023-10-30T13:27:50.675 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/16' 2023-10-30T13:27:50.676 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/32' 2023-10-30T13:27:50.676 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/30' 2023-10-30T13:27:50.676 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/14' 2023-10-30T13:27:50.676 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/18' 2023-10-30T13:27:50.676 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/13' 2023-10-30T13:27:50.676 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/33' 2023-10-30T13:27:50.677 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/17' 2023-10-30T13:27:50.677 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/35' 2023-10-30T13:27:50.677 INFO:tasks.workunit.client.0.smithi046.stdout:removed './112/37' 2023-10-30T13:27:50.677 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './112' 2023-10-30T13:27:50.677 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/19' 2023-10-30T13:27:50.677 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/15' 2023-10-30T13:27:50.678 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/31' 2023-10-30T13:27:50.678 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/36' 2023-10-30T13:27:50.678 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/20' 2023-10-30T13:27:50.678 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/12' 2023-10-30T13:27:50.678 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/39' 2023-10-30T13:27:50.679 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/10' 2023-10-30T13:27:50.679 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/34' 2023-10-30T13:27:50.679 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/38' 2023-10-30T13:27:50.679 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/11' 2023-10-30T13:27:50.679 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/16' 2023-10-30T13:27:50.679 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/32' 2023-10-30T13:27:50.680 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/30' 2023-10-30T13:27:50.680 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/14' 2023-10-30T13:27:50.680 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/18' 2023-10-30T13:27:50.680 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/13' 2023-10-30T13:27:50.680 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/33' 2023-10-30T13:27:50.680 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/17' 2023-10-30T13:27:50.681 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/35' 2023-10-30T13:27:50.681 INFO:tasks.workunit.client.0.smithi046.stdout:removed './113/37' 2023-10-30T13:27:50.681 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './113' 2023-10-30T13:27:50.681 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/19' 2023-10-30T13:27:50.681 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/15' 2023-10-30T13:27:50.681 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/31' 2023-10-30T13:27:50.682 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/36' 2023-10-30T13:27:50.682 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/20' 2023-10-30T13:27:50.682 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/12' 2023-10-30T13:27:50.682 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/39' 2023-10-30T13:27:50.682 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/10' 2023-10-30T13:27:50.683 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/34' 2023-10-30T13:27:50.683 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/38' 2023-10-30T13:27:50.683 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/11' 2023-10-30T13:27:50.683 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/16' 2023-10-30T13:27:50.683 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/32' 2023-10-30T13:27:50.683 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/30' 2023-10-30T13:27:50.684 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/14' 2023-10-30T13:27:50.684 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/18' 2023-10-30T13:27:50.684 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/13' 2023-10-30T13:27:50.684 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/33' 2023-10-30T13:27:50.684 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/17' 2023-10-30T13:27:50.684 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/35' 2023-10-30T13:27:50.685 INFO:tasks.workunit.client.0.smithi046.stdout:removed './114/37' 2023-10-30T13:27:50.685 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './114' 2023-10-30T13:27:50.685 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/19' 2023-10-30T13:27:50.685 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/15' 2023-10-30T13:27:50.685 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/31' 2023-10-30T13:27:50.686 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/36' 2023-10-30T13:27:50.686 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/20' 2023-10-30T13:27:50.686 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/12' 2023-10-30T13:27:50.686 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/39' 2023-10-30T13:27:50.686 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/10' 2023-10-30T13:27:50.686 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/34' 2023-10-30T13:27:50.687 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/38' 2023-10-30T13:27:50.687 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/11' 2023-10-30T13:27:50.687 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/16' 2023-10-30T13:27:50.687 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/32' 2023-10-30T13:27:50.687 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/30' 2023-10-30T13:27:50.688 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/14' 2023-10-30T13:27:50.688 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/18' 2023-10-30T13:27:50.688 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/13' 2023-10-30T13:27:50.688 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/33' 2023-10-30T13:27:50.688 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/17' 2023-10-30T13:27:50.688 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/35' 2023-10-30T13:27:50.689 INFO:tasks.workunit.client.0.smithi046.stdout:removed './115/37' 2023-10-30T13:27:50.689 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './115' 2023-10-30T13:27:50.689 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/19' 2023-10-30T13:27:50.689 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/15' 2023-10-30T13:27:50.689 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/31' 2023-10-30T13:27:50.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/36' 2023-10-30T13:27:50.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/20' 2023-10-30T13:27:50.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/12' 2023-10-30T13:27:50.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/39' 2023-10-30T13:27:50.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/10' 2023-10-30T13:27:50.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/34' 2023-10-30T13:27:50.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/38' 2023-10-30T13:27:50.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/11' 2023-10-30T13:27:50.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/16' 2023-10-30T13:27:50.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/32' 2023-10-30T13:27:50.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/30' 2023-10-30T13:27:50.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/14' 2023-10-30T13:27:50.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/18' 2023-10-30T13:27:50.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/13' 2023-10-30T13:27:50.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/33' 2023-10-30T13:27:50.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/17' 2023-10-30T13:27:50.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/35' 2023-10-30T13:27:50.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './116/37' 2023-10-30T13:27:50.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './116' 2023-10-30T13:27:50.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/19' 2023-10-30T13:27:50.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/15' 2023-10-30T13:27:50.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/31' 2023-10-30T13:27:50.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/36' 2023-10-30T13:27:50.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/20' 2023-10-30T13:27:50.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/12' 2023-10-30T13:27:50.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/39' 2023-10-30T13:27:50.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/10' 2023-10-30T13:27:50.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/34' 2023-10-30T13:27:50.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/38' 2023-10-30T13:27:50.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/11' 2023-10-30T13:27:50.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/16' 2023-10-30T13:27:50.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/32' 2023-10-30T13:27:50.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/30' 2023-10-30T13:27:50.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/14' 2023-10-30T13:27:50.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/18' 2023-10-30T13:27:50.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/13' 2023-10-30T13:27:50.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/33' 2023-10-30T13:27:50.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/17' 2023-10-30T13:27:50.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/35' 2023-10-30T13:27:50.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './117/37' 2023-10-30T13:27:50.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './117' 2023-10-30T13:27:50.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/19' 2023-10-30T13:27:50.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/15' 2023-10-30T13:27:50.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/31' 2023-10-30T13:27:50.698 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/36' 2023-10-30T13:27:50.698 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/20' 2023-10-30T13:27:50.698 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/12' 2023-10-30T13:27:50.698 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/39' 2023-10-30T13:27:50.698 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/10' 2023-10-30T13:27:50.699 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/34' 2023-10-30T13:27:50.699 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/38' 2023-10-30T13:27:50.699 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/11' 2023-10-30T13:27:50.699 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/16' 2023-10-30T13:27:50.699 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/32' 2023-10-30T13:27:50.699 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/30' 2023-10-30T13:27:50.699 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/14' 2023-10-30T13:27:50.700 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/18' 2023-10-30T13:27:50.700 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/13' 2023-10-30T13:27:50.700 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/33' 2023-10-30T13:27:50.700 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/17' 2023-10-30T13:27:50.700 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/35' 2023-10-30T13:27:50.700 INFO:tasks.workunit.client.0.smithi046.stdout:removed './118/37' 2023-10-30T13:27:50.701 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './118' 2023-10-30T13:27:50.701 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/19' 2023-10-30T13:27:50.701 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/15' 2023-10-30T13:27:50.701 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/31' 2023-10-30T13:27:50.701 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/36' 2023-10-30T13:27:50.701 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/20' 2023-10-30T13:27:50.702 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/12' 2023-10-30T13:27:52.212 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/39' 2023-10-30T13:27:52.213 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/10' 2023-10-30T13:27:52.213 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/34' 2023-10-30T13:27:52.213 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/38' 2023-10-30T13:27:52.213 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/11' 2023-10-30T13:27:52.214 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/16' 2023-10-30T13:27:52.214 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/32' 2023-10-30T13:27:52.214 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/30' 2023-10-30T13:27:52.214 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/14' 2023-10-30T13:27:52.214 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/18' 2023-10-30T13:27:52.215 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/13' 2023-10-30T13:27:52.215 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/33' 2023-10-30T13:27:52.215 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/17' 2023-10-30T13:27:52.215 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/35' 2023-10-30T13:27:52.215 INFO:tasks.workunit.client.0.smithi046.stdout:removed './119/37' 2023-10-30T13:27:52.215 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './119' 2023-10-30T13:27:52.216 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/19' 2023-10-30T13:27:52.216 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/15' 2023-10-30T13:27:52.216 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/31' 2023-10-30T13:27:52.216 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/36' 2023-10-30T13:27:52.216 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/20' 2023-10-30T13:27:52.217 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/12' 2023-10-30T13:27:52.217 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/39' 2023-10-30T13:27:52.217 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/10' 2023-10-30T13:27:52.217 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/34' 2023-10-30T13:27:52.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/38' 2023-10-30T13:27:52.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/11' 2023-10-30T13:27:52.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/16' 2023-10-30T13:27:52.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/32' 2023-10-30T13:27:52.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/30' 2023-10-30T13:27:52.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/14' 2023-10-30T13:27:52.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/18' 2023-10-30T13:27:52.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/13' 2023-10-30T13:27:52.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/33' 2023-10-30T13:27:52.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/17' 2023-10-30T13:27:52.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/35' 2023-10-30T13:27:52.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './120/37' 2023-10-30T13:27:52.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './120' 2023-10-30T13:27:52.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/19' 2023-10-30T13:27:52.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/15' 2023-10-30T13:27:52.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/31' 2023-10-30T13:27:52.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/36' 2023-10-30T13:27:52.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/20' 2023-10-30T13:27:52.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/12' 2023-10-30T13:27:52.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/39' 2023-10-30T13:27:52.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/10' 2023-10-30T13:27:52.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/34' 2023-10-30T13:27:52.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/38' 2023-10-30T13:27:52.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/11' 2023-10-30T13:27:52.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/16' 2023-10-30T13:27:52.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/32' 2023-10-30T13:27:52.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/30' 2023-10-30T13:27:52.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/14' 2023-10-30T13:27:52.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/18' 2023-10-30T13:27:52.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/13' 2023-10-30T13:27:52.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/33' 2023-10-30T13:27:52.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/17' 2023-10-30T13:27:52.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/35' 2023-10-30T13:27:52.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './121/37' 2023-10-30T13:27:52.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './121' 2023-10-30T13:27:52.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/19' 2023-10-30T13:27:52.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/15' 2023-10-30T13:27:52.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/31' 2023-10-30T13:27:52.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/36' 2023-10-30T13:27:52.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/20' 2023-10-30T13:27:52.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/12' 2023-10-30T13:27:52.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/39' 2023-10-30T13:27:52.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/10' 2023-10-30T13:27:52.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/34' 2023-10-30T13:27:52.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/38' 2023-10-30T13:27:52.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/11' 2023-10-30T13:27:52.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/16' 2023-10-30T13:27:52.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/32' 2023-10-30T13:27:52.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/30' 2023-10-30T13:27:52.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/14' 2023-10-30T13:27:52.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/18' 2023-10-30T13:27:52.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/13' 2023-10-30T13:27:52.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/33' 2023-10-30T13:27:52.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/17' 2023-10-30T13:27:52.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/35' 2023-10-30T13:27:52.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './122/37' 2023-10-30T13:27:52.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './122' 2023-10-30T13:27:52.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/19' 2023-10-30T13:27:52.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/15' 2023-10-30T13:27:52.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/31' 2023-10-30T13:27:52.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/36' 2023-10-30T13:27:52.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/20' 2023-10-30T13:27:52.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/12' 2023-10-30T13:27:52.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/39' 2023-10-30T13:27:52.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/10' 2023-10-30T13:27:52.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/34' 2023-10-30T13:27:52.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/38' 2023-10-30T13:27:52.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/11' 2023-10-30T13:27:52.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/16' 2023-10-30T13:27:52.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/32' 2023-10-30T13:27:52.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/30' 2023-10-30T13:27:52.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/14' 2023-10-30T13:27:52.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/18' 2023-10-30T13:27:52.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/13' 2023-10-30T13:27:52.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/33' 2023-10-30T13:27:52.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/17' 2023-10-30T13:27:52.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/35' 2023-10-30T13:27:52.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './123/37' 2023-10-30T13:27:52.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './123' 2023-10-30T13:27:52.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/19' 2023-10-30T13:27:52.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/15' 2023-10-30T13:27:52.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/31' 2023-10-30T13:27:52.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/36' 2023-10-30T13:27:52.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/20' 2023-10-30T13:27:52.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/12' 2023-10-30T13:27:52.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/39' 2023-10-30T13:27:52.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/10' 2023-10-30T13:27:52.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/34' 2023-10-30T13:27:52.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/38' 2023-10-30T13:27:52.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/11' 2023-10-30T13:27:52.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/16' 2023-10-30T13:27:52.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/32' 2023-10-30T13:27:52.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/30' 2023-10-30T13:27:52.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/14' 2023-10-30T13:27:52.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/18' 2023-10-30T13:27:52.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/13' 2023-10-30T13:27:52.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/33' 2023-10-30T13:27:52.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/17' 2023-10-30T13:27:52.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/35' 2023-10-30T13:27:52.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './124/37' 2023-10-30T13:27:52.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './124' 2023-10-30T13:27:52.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/19' 2023-10-30T13:27:52.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/15' 2023-10-30T13:27:52.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/31' 2023-10-30T13:27:52.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/36' 2023-10-30T13:27:52.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/20' 2023-10-30T13:27:52.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/12' 2023-10-30T13:27:52.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/39' 2023-10-30T13:27:52.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/10' 2023-10-30T13:27:52.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/34' 2023-10-30T13:27:52.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/38' 2023-10-30T13:27:52.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/11' 2023-10-30T13:27:52.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/16' 2023-10-30T13:27:52.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/32' 2023-10-30T13:27:52.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/30' 2023-10-30T13:27:52.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/14' 2023-10-30T13:27:52.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/18' 2023-10-30T13:27:52.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/13' 2023-10-30T13:27:52.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/33' 2023-10-30T13:27:52.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/17' 2023-10-30T13:27:52.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/35' 2023-10-30T13:27:52.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './125/37' 2023-10-30T13:27:52.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './125' 2023-10-30T13:27:52.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/19' 2023-10-30T13:27:52.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/15' 2023-10-30T13:27:52.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/31' 2023-10-30T13:27:52.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/36' 2023-10-30T13:27:52.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/20' 2023-10-30T13:27:52.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/12' 2023-10-30T13:27:52.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/39' 2023-10-30T13:27:52.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/10' 2023-10-30T13:27:52.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/34' 2023-10-30T13:27:52.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/38' 2023-10-30T13:27:52.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/11' 2023-10-30T13:27:52.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/16' 2023-10-30T13:27:52.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/32' 2023-10-30T13:27:52.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/30' 2023-10-30T13:27:52.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/14' 2023-10-30T13:27:52.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/18' 2023-10-30T13:27:52.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/13' 2023-10-30T13:27:52.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/33' 2023-10-30T13:27:52.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/17' 2023-10-30T13:27:52.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/35' 2023-10-30T13:27:52.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './126/37' 2023-10-30T13:27:52.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './126' 2023-10-30T13:27:52.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/19' 2023-10-30T13:27:52.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/15' 2023-10-30T13:27:52.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/31' 2023-10-30T13:27:52.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/36' 2023-10-30T13:27:52.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/20' 2023-10-30T13:27:52.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/12' 2023-10-30T13:27:52.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/39' 2023-10-30T13:27:52.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/10' 2023-10-30T13:27:52.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/34' 2023-10-30T13:27:52.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/38' 2023-10-30T13:27:52.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/11' 2023-10-30T13:27:52.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/16' 2023-10-30T13:27:52.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/32' 2023-10-30T13:27:52.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/30' 2023-10-30T13:27:52.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/14' 2023-10-30T13:27:52.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/18' 2023-10-30T13:27:52.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/13' 2023-10-30T13:27:52.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/33' 2023-10-30T13:27:52.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/17' 2023-10-30T13:27:52.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/35' 2023-10-30T13:27:52.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './127/37' 2023-10-30T13:27:52.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './127' 2023-10-30T13:27:52.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/19' 2023-10-30T13:27:52.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/15' 2023-10-30T13:27:52.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/31' 2023-10-30T13:27:52.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/36' 2023-10-30T13:27:52.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/20' 2023-10-30T13:27:52.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/12' 2023-10-30T13:27:52.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/39' 2023-10-30T13:27:52.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/10' 2023-10-30T13:27:52.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/34' 2023-10-30T13:27:52.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/38' 2023-10-30T13:27:52.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/11' 2023-10-30T13:27:52.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/16' 2023-10-30T13:27:52.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/32' 2023-10-30T13:27:52.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/30' 2023-10-30T13:27:52.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/14' 2023-10-30T13:27:52.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/18' 2023-10-30T13:27:52.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/13' 2023-10-30T13:27:52.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/33' 2023-10-30T13:27:52.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/17' 2023-10-30T13:27:52.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/35' 2023-10-30T13:27:53.574 INFO:tasks.workunit.client.0.smithi046.stdout:removed './128/37' 2023-10-30T13:27:53.574 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './128' 2023-10-30T13:27:53.574 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/19' 2023-10-30T13:27:53.575 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/15' 2023-10-30T13:27:53.575 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/31' 2023-10-30T13:27:53.575 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/36' 2023-10-30T13:27:53.575 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/20' 2023-10-30T13:27:53.575 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/12' 2023-10-30T13:27:53.575 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/39' 2023-10-30T13:27:53.576 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/10' 2023-10-30T13:27:53.576 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/34' 2023-10-30T13:27:53.576 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/38' 2023-10-30T13:27:53.576 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/11' 2023-10-30T13:27:53.576 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/16' 2023-10-30T13:27:53.577 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/32' 2023-10-30T13:27:53.577 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/30' 2023-10-30T13:27:53.577 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/14' 2023-10-30T13:27:53.577 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/18' 2023-10-30T13:27:53.577 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/13' 2023-10-30T13:27:53.577 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/33' 2023-10-30T13:27:53.578 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/17' 2023-10-30T13:27:53.578 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/35' 2023-10-30T13:27:53.578 INFO:tasks.workunit.client.0.smithi046.stdout:removed './129/37' 2023-10-30T13:27:53.578 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './129' 2023-10-30T13:27:53.578 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/19' 2023-10-30T13:27:53.578 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/15' 2023-10-30T13:27:53.579 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/31' 2023-10-30T13:27:53.579 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/36' 2023-10-30T13:27:53.579 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/20' 2023-10-30T13:27:53.579 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/12' 2023-10-30T13:27:53.579 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/39' 2023-10-30T13:27:53.580 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/10' 2023-10-30T13:27:53.580 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/34' 2023-10-30T13:27:53.580 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/38' 2023-10-30T13:27:53.580 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/11' 2023-10-30T13:27:53.580 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/16' 2023-10-30T13:27:53.580 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/32' 2023-10-30T13:27:53.581 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/30' 2023-10-30T13:27:53.581 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/14' 2023-10-30T13:27:53.581 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/18' 2023-10-30T13:27:53.581 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/13' 2023-10-30T13:27:53.581 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/33' 2023-10-30T13:27:53.582 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/17' 2023-10-30T13:27:53.582 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/35' 2023-10-30T13:27:53.582 INFO:tasks.workunit.client.0.smithi046.stdout:removed './130/37' 2023-10-30T13:27:53.582 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './130' 2023-10-30T13:27:53.582 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/19' 2023-10-30T13:27:53.582 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/15' 2023-10-30T13:27:53.583 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/31' 2023-10-30T13:27:53.583 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/36' 2023-10-30T13:27:53.583 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/20' 2023-10-30T13:27:53.583 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/12' 2023-10-30T13:27:53.583 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/39' 2023-10-30T13:27:53.584 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/10' 2023-10-30T13:27:53.584 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/34' 2023-10-30T13:27:53.584 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/38' 2023-10-30T13:27:53.584 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/11' 2023-10-30T13:27:53.584 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/16' 2023-10-30T13:27:53.584 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/32' 2023-10-30T13:27:53.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/30' 2023-10-30T13:27:53.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/14' 2023-10-30T13:27:53.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/18' 2023-10-30T13:27:53.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/13' 2023-10-30T13:27:53.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/33' 2023-10-30T13:27:53.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/17' 2023-10-30T13:27:53.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/35' 2023-10-30T13:27:53.586 INFO:tasks.workunit.client.0.smithi046.stdout:removed './131/37' 2023-10-30T13:27:53.586 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './131' 2023-10-30T13:27:53.586 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/19' 2023-10-30T13:27:53.586 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/15' 2023-10-30T13:27:53.586 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/31' 2023-10-30T13:27:53.587 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/36' 2023-10-30T13:27:53.587 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/20' 2023-10-30T13:27:53.587 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/12' 2023-10-30T13:27:53.587 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/39' 2023-10-30T13:27:53.587 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/10' 2023-10-30T13:27:53.587 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/34' 2023-10-30T13:27:53.588 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/38' 2023-10-30T13:27:53.588 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/11' 2023-10-30T13:27:53.588 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/16' 2023-10-30T13:27:53.588 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/32' 2023-10-30T13:27:53.588 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/30' 2023-10-30T13:27:53.588 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/14' 2023-10-30T13:27:53.589 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/18' 2023-10-30T13:27:53.589 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/13' 2023-10-30T13:27:53.589 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/33' 2023-10-30T13:27:53.589 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/17' 2023-10-30T13:27:53.589 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/35' 2023-10-30T13:27:53.589 INFO:tasks.workunit.client.0.smithi046.stdout:removed './132/37' 2023-10-30T13:27:53.590 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './132' 2023-10-30T13:27:53.590 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/19' 2023-10-30T13:27:53.590 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/15' 2023-10-30T13:27:53.590 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/31' 2023-10-30T13:27:53.590 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/36' 2023-10-30T13:27:53.590 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/20' 2023-10-30T13:27:53.591 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/12' 2023-10-30T13:27:53.591 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/39' 2023-10-30T13:27:53.591 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/10' 2023-10-30T13:27:53.591 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/34' 2023-10-30T13:27:53.591 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/38' 2023-10-30T13:27:53.591 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/11' 2023-10-30T13:27:53.592 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/16' 2023-10-30T13:27:53.592 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/32' 2023-10-30T13:27:53.592 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/30' 2023-10-30T13:27:53.592 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/14' 2023-10-30T13:27:53.592 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/18' 2023-10-30T13:27:53.592 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/13' 2023-10-30T13:27:53.593 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/33' 2023-10-30T13:27:53.593 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/17' 2023-10-30T13:27:53.593 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/35' 2023-10-30T13:27:53.593 INFO:tasks.workunit.client.0.smithi046.stdout:removed './133/37' 2023-10-30T13:27:53.593 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './133' 2023-10-30T13:27:53.594 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/19' 2023-10-30T13:27:53.594 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/15' 2023-10-30T13:27:53.594 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/31' 2023-10-30T13:27:53.594 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/36' 2023-10-30T13:27:53.594 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/20' 2023-10-30T13:27:53.595 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/12' 2023-10-30T13:27:53.595 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/39' 2023-10-30T13:27:53.595 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/10' 2023-10-30T13:27:53.595 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/34' 2023-10-30T13:27:53.595 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/38' 2023-10-30T13:27:53.595 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/11' 2023-10-30T13:27:53.596 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/16' 2023-10-30T13:27:53.596 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/32' 2023-10-30T13:27:53.596 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/30' 2023-10-30T13:27:53.596 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/14' 2023-10-30T13:27:53.596 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/18' 2023-10-30T13:27:53.596 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/13' 2023-10-30T13:27:53.597 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/33' 2023-10-30T13:27:53.597 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/17' 2023-10-30T13:27:53.597 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/35' 2023-10-30T13:27:53.597 INFO:tasks.workunit.client.0.smithi046.stdout:removed './134/37' 2023-10-30T13:27:53.597 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './134' 2023-10-30T13:27:53.598 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/19' 2023-10-30T13:27:53.598 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/15' 2023-10-30T13:27:53.598 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/31' 2023-10-30T13:27:53.598 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/36' 2023-10-30T13:27:53.598 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/20' 2023-10-30T13:27:53.599 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/12' 2023-10-30T13:27:53.599 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/39' 2023-10-30T13:27:53.599 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/10' 2023-10-30T13:27:53.599 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/34' 2023-10-30T13:27:53.599 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/38' 2023-10-30T13:27:53.599 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/11' 2023-10-30T13:27:53.600 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/16' 2023-10-30T13:27:53.600 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/32' 2023-10-30T13:27:53.600 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/30' 2023-10-30T13:27:53.600 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/14' 2023-10-30T13:27:53.600 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/18' 2023-10-30T13:27:53.601 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/13' 2023-10-30T13:27:53.601 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/33' 2023-10-30T13:27:53.601 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/17' 2023-10-30T13:27:53.601 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/35' 2023-10-30T13:27:53.601 INFO:tasks.workunit.client.0.smithi046.stdout:removed './135/37' 2023-10-30T13:27:53.601 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './135' 2023-10-30T13:27:53.602 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/19' 2023-10-30T13:27:53.602 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/15' 2023-10-30T13:27:53.602 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/31' 2023-10-30T13:27:53.602 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/36' 2023-10-30T13:27:53.602 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/20' 2023-10-30T13:27:53.603 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/12' 2023-10-30T13:27:53.603 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/39' 2023-10-30T13:27:53.603 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/10' 2023-10-30T13:27:53.603 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/34' 2023-10-30T13:27:53.603 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/38' 2023-10-30T13:27:53.603 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/11' 2023-10-30T13:27:53.604 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/16' 2023-10-30T13:27:53.604 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/32' 2023-10-30T13:27:53.604 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/30' 2023-10-30T13:27:53.604 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/14' 2023-10-30T13:27:53.604 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/18' 2023-10-30T13:27:53.605 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/13' 2023-10-30T13:27:53.605 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/33' 2023-10-30T13:27:53.605 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/17' 2023-10-30T13:27:53.605 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/35' 2023-10-30T13:27:53.605 INFO:tasks.workunit.client.0.smithi046.stdout:removed './136/37' 2023-10-30T13:27:53.605 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './136' 2023-10-30T13:27:53.606 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/19' 2023-10-30T13:27:53.606 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/15' 2023-10-30T13:27:53.606 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/31' 2023-10-30T13:27:53.606 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/36' 2023-10-30T13:27:53.606 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/20' 2023-10-30T13:27:53.607 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/12' 2023-10-30T13:27:53.607 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/39' 2023-10-30T13:27:53.607 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/10' 2023-10-30T13:27:53.607 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/34' 2023-10-30T13:27:53.607 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/38' 2023-10-30T13:27:53.608 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/11' 2023-10-30T13:27:53.608 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/16' 2023-10-30T13:27:53.608 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/32' 2023-10-30T13:27:53.608 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/30' 2023-10-30T13:27:53.608 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/14' 2023-10-30T13:27:53.608 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/18' 2023-10-30T13:27:53.609 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/13' 2023-10-30T13:27:53.609 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/33' 2023-10-30T13:27:53.609 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/17' 2023-10-30T13:27:53.609 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/35' 2023-10-30T13:27:53.609 INFO:tasks.workunit.client.0.smithi046.stdout:removed './137/37' 2023-10-30T13:27:53.610 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './137' 2023-10-30T13:27:53.610 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/19' 2023-10-30T13:27:53.610 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/15' 2023-10-30T13:27:53.610 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/31' 2023-10-30T13:27:53.610 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/36' 2023-10-30T13:27:53.610 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/20' 2023-10-30T13:27:53.611 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/12' 2023-10-30T13:27:53.611 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/39' 2023-10-30T13:27:53.611 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/10' 2023-10-30T13:27:53.611 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/34' 2023-10-30T13:27:53.611 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/38' 2023-10-30T13:27:53.612 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/11' 2023-10-30T13:27:53.612 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/16' 2023-10-30T13:27:57.300 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/32' 2023-10-30T13:27:57.300 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/30' 2023-10-30T13:27:57.300 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/14' 2023-10-30T13:27:57.301 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/18' 2023-10-30T13:27:57.301 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/13' 2023-10-30T13:27:57.301 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/33' 2023-10-30T13:27:57.301 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/17' 2023-10-30T13:27:57.301 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/35' 2023-10-30T13:27:57.302 INFO:tasks.workunit.client.0.smithi046.stdout:removed './138/37' 2023-10-30T13:27:57.302 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './138' 2023-10-30T13:27:57.302 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/19' 2023-10-30T13:27:57.302 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/15' 2023-10-30T13:27:57.302 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/31' 2023-10-30T13:27:57.303 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/36' 2023-10-30T13:27:57.303 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/20' 2023-10-30T13:27:57.303 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/12' 2023-10-30T13:27:57.303 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/39' 2023-10-30T13:27:57.303 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/10' 2023-10-30T13:27:57.304 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/34' 2023-10-30T13:27:57.304 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/38' 2023-10-30T13:27:57.304 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/11' 2023-10-30T13:27:57.304 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/16' 2023-10-30T13:27:57.304 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/32' 2023-10-30T13:27:57.304 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/30' 2023-10-30T13:27:57.305 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/14' 2023-10-30T13:27:57.305 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/18' 2023-10-30T13:27:57.305 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/13' 2023-10-30T13:27:57.305 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/33' 2023-10-30T13:27:57.305 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/17' 2023-10-30T13:27:57.306 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/35' 2023-10-30T13:27:57.306 INFO:tasks.workunit.client.0.smithi046.stdout:removed './139/37' 2023-10-30T13:27:57.306 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './139' 2023-10-30T13:27:57.306 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/19' 2023-10-30T13:27:57.306 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/15' 2023-10-30T13:27:57.307 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/31' 2023-10-30T13:27:57.307 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/36' 2023-10-30T13:27:57.307 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/20' 2023-10-30T13:27:57.307 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/12' 2023-10-30T13:27:57.307 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/39' 2023-10-30T13:27:57.308 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/10' 2023-10-30T13:27:57.308 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/34' 2023-10-30T13:27:57.308 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/38' 2023-10-30T13:27:57.308 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/11' 2023-10-30T13:27:57.308 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/16' 2023-10-30T13:27:57.309 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/32' 2023-10-30T13:27:57.309 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/30' 2023-10-30T13:27:57.309 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/14' 2023-10-30T13:27:57.309 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/18' 2023-10-30T13:27:57.309 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/13' 2023-10-30T13:27:57.309 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/33' 2023-10-30T13:27:57.310 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/17' 2023-10-30T13:27:57.310 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/35' 2023-10-30T13:27:57.310 INFO:tasks.workunit.client.0.smithi046.stdout:removed './140/37' 2023-10-30T13:27:57.310 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './140' 2023-10-30T13:27:57.310 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/19' 2023-10-30T13:27:57.311 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/15' 2023-10-30T13:27:57.311 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/31' 2023-10-30T13:27:57.311 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/36' 2023-10-30T13:27:57.311 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/20' 2023-10-30T13:27:57.311 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/12' 2023-10-30T13:27:57.311 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/39' 2023-10-30T13:27:57.312 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/10' 2023-10-30T13:27:57.312 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/34' 2023-10-30T13:27:57.312 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/38' 2023-10-30T13:27:57.312 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/11' 2023-10-30T13:27:57.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/16' 2023-10-30T13:27:57.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/32' 2023-10-30T13:27:57.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/30' 2023-10-30T13:27:57.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/14' 2023-10-30T13:27:57.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/18' 2023-10-30T13:27:57.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/13' 2023-10-30T13:27:57.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/33' 2023-10-30T13:27:57.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/17' 2023-10-30T13:27:57.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/35' 2023-10-30T13:27:57.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './141/37' 2023-10-30T13:27:57.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './141' 2023-10-30T13:27:57.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/19' 2023-10-30T13:27:57.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/15' 2023-10-30T13:27:57.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/31' 2023-10-30T13:27:57.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/36' 2023-10-30T13:27:57.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/20' 2023-10-30T13:27:57.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/12' 2023-10-30T13:27:57.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/39' 2023-10-30T13:27:57.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/10' 2023-10-30T13:27:57.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/34' 2023-10-30T13:27:57.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/38' 2023-10-30T13:27:57.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/11' 2023-10-30T13:27:57.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/16' 2023-10-30T13:27:57.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/32' 2023-10-30T13:27:57.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/30' 2023-10-30T13:27:57.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/14' 2023-10-30T13:27:57.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/18' 2023-10-30T13:27:57.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/13' 2023-10-30T13:27:57.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/33' 2023-10-30T13:27:57.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/17' 2023-10-30T13:27:57.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/35' 2023-10-30T13:27:57.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './142/37' 2023-10-30T13:27:57.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './142' 2023-10-30T13:27:57.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/19' 2023-10-30T13:27:57.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/15' 2023-10-30T13:27:57.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/31' 2023-10-30T13:27:57.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/36' 2023-10-30T13:27:57.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/20' 2023-10-30T13:27:57.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/12' 2023-10-30T13:27:57.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/39' 2023-10-30T13:27:57.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/10' 2023-10-30T13:27:57.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/34' 2023-10-30T13:27:57.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/38' 2023-10-30T13:27:57.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/11' 2023-10-30T13:27:57.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/16' 2023-10-30T13:27:57.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/32' 2023-10-30T13:27:57.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/30' 2023-10-30T13:27:57.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/14' 2023-10-30T13:27:57.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/18' 2023-10-30T13:27:57.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/13' 2023-10-30T13:27:57.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/33' 2023-10-30T13:27:57.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/17' 2023-10-30T13:27:57.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/35' 2023-10-30T13:27:57.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './143/37' 2023-10-30T13:27:57.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './143' 2023-10-30T13:27:57.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/19' 2023-10-30T13:27:57.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/15' 2023-10-30T13:27:57.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/31' 2023-10-30T13:27:57.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/36' 2023-10-30T13:27:57.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/20' 2023-10-30T13:27:57.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/12' 2023-10-30T13:27:57.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/39' 2023-10-30T13:27:57.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/10' 2023-10-30T13:27:57.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/34' 2023-10-30T13:27:57.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/38' 2023-10-30T13:27:57.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/11' 2023-10-30T13:27:57.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/16' 2023-10-30T13:27:57.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/32' 2023-10-30T13:27:57.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/30' 2023-10-30T13:27:57.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/14' 2023-10-30T13:27:57.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/18' 2023-10-30T13:27:57.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/13' 2023-10-30T13:27:57.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/33' 2023-10-30T13:27:57.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/17' 2023-10-30T13:27:57.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/35' 2023-10-30T13:27:57.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './144/37' 2023-10-30T13:27:57.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './144' 2023-10-30T13:27:57.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/19' 2023-10-30T13:27:57.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/15' 2023-10-30T13:27:57.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/31' 2023-10-30T13:27:57.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/36' 2023-10-30T13:27:57.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/20' 2023-10-30T13:27:57.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/12' 2023-10-30T13:27:57.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/39' 2023-10-30T13:27:57.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/10' 2023-10-30T13:27:57.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/34' 2023-10-30T13:27:57.329 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/38' 2023-10-30T13:27:57.329 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/11' 2023-10-30T13:27:57.329 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/16' 2023-10-30T13:27:57.329 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/32' 2023-10-30T13:27:57.329 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/30' 2023-10-30T13:27:57.329 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/14' 2023-10-30T13:27:57.330 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/18' 2023-10-30T13:27:57.330 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/13' 2023-10-30T13:27:57.330 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/33' 2023-10-30T13:27:57.330 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/17' 2023-10-30T13:27:57.330 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/35' 2023-10-30T13:27:57.331 INFO:tasks.workunit.client.0.smithi046.stdout:removed './145/37' 2023-10-30T13:27:57.331 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './145' 2023-10-30T13:27:57.331 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/19' 2023-10-30T13:27:57.331 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/15' 2023-10-30T13:27:57.331 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/31' 2023-10-30T13:27:57.331 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/36' 2023-10-30T13:27:57.332 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/20' 2023-10-30T13:27:57.332 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/12' 2023-10-30T13:27:57.332 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/39' 2023-10-30T13:27:57.332 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/10' 2023-10-30T13:27:57.332 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/34' 2023-10-30T13:27:57.333 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/38' 2023-10-30T13:27:57.333 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/11' 2023-10-30T13:27:57.333 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/16' 2023-10-30T13:27:57.333 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/32' 2023-10-30T13:27:57.333 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/30' 2023-10-30T13:27:57.333 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/14' 2023-10-30T13:27:57.334 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/18' 2023-10-30T13:27:57.334 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/13' 2023-10-30T13:27:57.334 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/33' 2023-10-30T13:27:57.334 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/17' 2023-10-30T13:27:57.334 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/35' 2023-10-30T13:27:57.335 INFO:tasks.workunit.client.0.smithi046.stdout:removed './146/37' 2023-10-30T13:27:57.335 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './146' 2023-10-30T13:27:57.335 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/19' 2023-10-30T13:27:57.335 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/15' 2023-10-30T13:27:57.335 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/31' 2023-10-30T13:27:57.335 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/36' 2023-10-30T13:27:57.336 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/20' 2023-10-30T13:27:57.336 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/12' 2023-10-30T13:27:57.336 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/39' 2023-10-30T13:27:57.336 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/10' 2023-10-30T13:27:57.336 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/34' 2023-10-30T13:27:57.337 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/38' 2023-10-30T13:27:57.337 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/11' 2023-10-30T13:27:57.337 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/16' 2023-10-30T13:27:57.337 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/32' 2023-10-30T13:27:57.337 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/30' 2023-10-30T13:27:57.337 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/14' 2023-10-30T13:27:57.338 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/18' 2023-10-30T13:27:57.338 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/13' 2023-10-30T13:27:57.338 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/33' 2023-10-30T13:27:57.338 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/17' 2023-10-30T13:27:57.338 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/35' 2023-10-30T13:27:57.339 INFO:tasks.workunit.client.0.smithi046.stdout:removed './147/37' 2023-10-30T13:27:57.339 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './147' 2023-10-30T13:27:57.339 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/19' 2023-10-30T13:27:57.339 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/15' 2023-10-30T13:27:57.339 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/31' 2023-10-30T13:27:57.339 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/36' 2023-10-30T13:28:00.217 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/20' 2023-10-30T13:28:00.217 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/12' 2023-10-30T13:28:00.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/39' 2023-10-30T13:28:00.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/10' 2023-10-30T13:28:00.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/34' 2023-10-30T13:28:00.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/38' 2023-10-30T13:28:00.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/11' 2023-10-30T13:28:00.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/16' 2023-10-30T13:28:00.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/32' 2023-10-30T13:28:00.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/30' 2023-10-30T13:28:00.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/14' 2023-10-30T13:28:00.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/18' 2023-10-30T13:28:00.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/13' 2023-10-30T13:28:00.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/33' 2023-10-30T13:28:00.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/17' 2023-10-30T13:28:00.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/35' 2023-10-30T13:28:00.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './148/37' 2023-10-30T13:28:00.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './148' 2023-10-30T13:28:00.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/19' 2023-10-30T13:28:00.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/15' 2023-10-30T13:28:00.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/31' 2023-10-30T13:28:00.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/36' 2023-10-30T13:28:00.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/20' 2023-10-30T13:28:00.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/12' 2023-10-30T13:28:00.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/39' 2023-10-30T13:28:00.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/10' 2023-10-30T13:28:00.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/34' 2023-10-30T13:28:00.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/38' 2023-10-30T13:28:00.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/11' 2023-10-30T13:28:00.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/16' 2023-10-30T13:28:00.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/32' 2023-10-30T13:28:00.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/30' 2023-10-30T13:28:00.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/14' 2023-10-30T13:28:00.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/18' 2023-10-30T13:28:00.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/13' 2023-10-30T13:28:00.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/33' 2023-10-30T13:28:00.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/17' 2023-10-30T13:28:00.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/35' 2023-10-30T13:28:00.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './149/37' 2023-10-30T13:28:00.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './149' 2023-10-30T13:28:00.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/19' 2023-10-30T13:28:00.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/15' 2023-10-30T13:28:00.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/31' 2023-10-30T13:28:00.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/36' 2023-10-30T13:28:00.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/20' 2023-10-30T13:28:00.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/12' 2023-10-30T13:28:00.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/39' 2023-10-30T13:28:00.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/10' 2023-10-30T13:28:00.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/34' 2023-10-30T13:28:00.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/38' 2023-10-30T13:28:00.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/11' 2023-10-30T13:28:00.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/16' 2023-10-30T13:28:00.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/32' 2023-10-30T13:28:00.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/30' 2023-10-30T13:28:00.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/14' 2023-10-30T13:28:00.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/18' 2023-10-30T13:28:00.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/13' 2023-10-30T13:28:00.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/33' 2023-10-30T13:28:00.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/17' 2023-10-30T13:28:00.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/35' 2023-10-30T13:28:00.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './150/37' 2023-10-30T13:28:00.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './150' 2023-10-30T13:28:00.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/19' 2023-10-30T13:28:00.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/15' 2023-10-30T13:28:00.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/31' 2023-10-30T13:28:00.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/36' 2023-10-30T13:28:00.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/20' 2023-10-30T13:28:00.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/12' 2023-10-30T13:28:00.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/39' 2023-10-30T13:28:00.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/10' 2023-10-30T13:28:00.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/34' 2023-10-30T13:28:00.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/38' 2023-10-30T13:28:00.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/11' 2023-10-30T13:28:00.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/16' 2023-10-30T13:28:00.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/32' 2023-10-30T13:28:00.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/30' 2023-10-30T13:28:00.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/14' 2023-10-30T13:28:00.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/18' 2023-10-30T13:28:00.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/13' 2023-10-30T13:28:00.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/33' 2023-10-30T13:28:00.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/17' 2023-10-30T13:28:00.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/35' 2023-10-30T13:28:00.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './151/37' 2023-10-30T13:28:00.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './151' 2023-10-30T13:28:00.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/19' 2023-10-30T13:28:00.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/15' 2023-10-30T13:28:00.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/31' 2023-10-30T13:28:00.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/36' 2023-10-30T13:28:00.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/20' 2023-10-30T13:28:00.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/12' 2023-10-30T13:28:00.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/39' 2023-10-30T13:28:00.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/10' 2023-10-30T13:28:00.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/34' 2023-10-30T13:28:00.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/38' 2023-10-30T13:28:00.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/11' 2023-10-30T13:28:00.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/16' 2023-10-30T13:28:00.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/32' 2023-10-30T13:28:00.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/30' 2023-10-30T13:28:00.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/14' 2023-10-30T13:28:00.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/18' 2023-10-30T13:28:00.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/13' 2023-10-30T13:28:00.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/33' 2023-10-30T13:28:00.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/17' 2023-10-30T13:28:00.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/35' 2023-10-30T13:28:00.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './152/37' 2023-10-30T13:28:00.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './152' 2023-10-30T13:28:00.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/19' 2023-10-30T13:28:00.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/15' 2023-10-30T13:28:00.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/31' 2023-10-30T13:28:00.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/36' 2023-10-30T13:28:00.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/20' 2023-10-30T13:28:00.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/12' 2023-10-30T13:28:00.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/39' 2023-10-30T13:28:00.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/10' 2023-10-30T13:28:00.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/34' 2023-10-30T13:28:00.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/38' 2023-10-30T13:28:00.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/11' 2023-10-30T13:28:00.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/16' 2023-10-30T13:28:00.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/32' 2023-10-30T13:28:00.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/30' 2023-10-30T13:28:00.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/14' 2023-10-30T13:28:00.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/18' 2023-10-30T13:28:00.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/13' 2023-10-30T13:28:00.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/33' 2023-10-30T13:28:00.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/17' 2023-10-30T13:28:00.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/35' 2023-10-30T13:28:00.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './153/37' 2023-10-30T13:28:00.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './153' 2023-10-30T13:28:00.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/19' 2023-10-30T13:28:00.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/15' 2023-10-30T13:28:00.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/31' 2023-10-30T13:28:00.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/36' 2023-10-30T13:28:00.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/20' 2023-10-30T13:28:00.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/12' 2023-10-30T13:28:00.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/39' 2023-10-30T13:28:00.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/10' 2023-10-30T13:28:00.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/34' 2023-10-30T13:28:00.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/38' 2023-10-30T13:28:00.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/11' 2023-10-30T13:28:00.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/16' 2023-10-30T13:28:00.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/32' 2023-10-30T13:28:00.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/30' 2023-10-30T13:28:00.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/14' 2023-10-30T13:28:00.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/18' 2023-10-30T13:28:00.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/13' 2023-10-30T13:28:00.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/33' 2023-10-30T13:28:00.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/17' 2023-10-30T13:28:00.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/35' 2023-10-30T13:28:00.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './154/37' 2023-10-30T13:28:00.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './154' 2023-10-30T13:28:00.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/19' 2023-10-30T13:28:00.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/15' 2023-10-30T13:28:00.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/31' 2023-10-30T13:28:00.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/36' 2023-10-30T13:28:00.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/20' 2023-10-30T13:28:00.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/12' 2023-10-30T13:28:00.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/39' 2023-10-30T13:28:00.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/10' 2023-10-30T13:28:00.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/34' 2023-10-30T13:28:00.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/38' 2023-10-30T13:28:00.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/11' 2023-10-30T13:28:00.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/16' 2023-10-30T13:28:00.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/32' 2023-10-30T13:28:00.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/30' 2023-10-30T13:28:00.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/14' 2023-10-30T13:28:00.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/18' 2023-10-30T13:28:00.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/13' 2023-10-30T13:28:00.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/33' 2023-10-30T13:28:00.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/17' 2023-10-30T13:28:00.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/35' 2023-10-30T13:28:00.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './155/37' 2023-10-30T13:28:00.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './155' 2023-10-30T13:28:00.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/19' 2023-10-30T13:28:00.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/15' 2023-10-30T13:28:00.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/31' 2023-10-30T13:28:00.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/36' 2023-10-30T13:28:00.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/20' 2023-10-30T13:28:00.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/12' 2023-10-30T13:28:00.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/39' 2023-10-30T13:28:00.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/10' 2023-10-30T13:28:00.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/34' 2023-10-30T13:28:00.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/38' 2023-10-30T13:28:00.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/11' 2023-10-30T13:28:00.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/16' 2023-10-30T13:28:00.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/32' 2023-10-30T13:28:00.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/30' 2023-10-30T13:28:00.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/14' 2023-10-30T13:28:00.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/18' 2023-10-30T13:28:00.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/13' 2023-10-30T13:28:00.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/33' 2023-10-30T13:28:00.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/17' 2023-10-30T13:28:00.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/35' 2023-10-30T13:28:00.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './156/37' 2023-10-30T13:28:00.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './156' 2023-10-30T13:28:00.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/19' 2023-10-30T13:28:00.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/15' 2023-10-30T13:28:00.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/31' 2023-10-30T13:28:00.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/36' 2023-10-30T13:28:00.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/20' 2023-10-30T13:28:00.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/12' 2023-10-30T13:28:00.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/39' 2023-10-30T13:28:00.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/10' 2023-10-30T13:28:00.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/34' 2023-10-30T13:28:00.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/38' 2023-10-30T13:28:00.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/11' 2023-10-30T13:28:00.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/16' 2023-10-30T13:28:00.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/32' 2023-10-30T13:28:00.254 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/30' 2023-10-30T13:28:00.254 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/14' 2023-10-30T13:28:00.254 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/18' 2023-10-30T13:28:00.254 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/13' 2023-10-30T13:28:00.254 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/33' 2023-10-30T13:28:09.214 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/17' 2023-10-30T13:28:09.214 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/35' 2023-10-30T13:28:09.215 INFO:tasks.workunit.client.0.smithi046.stdout:removed './157/37' 2023-10-30T13:28:09.215 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './157' 2023-10-30T13:28:09.215 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/19' 2023-10-30T13:28:09.215 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/15' 2023-10-30T13:28:09.215 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/31' 2023-10-30T13:28:09.216 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/36' 2023-10-30T13:28:09.216 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/20' 2023-10-30T13:28:09.216 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/12' 2023-10-30T13:28:09.216 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/39' 2023-10-30T13:28:09.216 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/10' 2023-10-30T13:28:09.216 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/34' 2023-10-30T13:28:09.217 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/38' 2023-10-30T13:28:09.217 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/11' 2023-10-30T13:28:09.217 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/16' 2023-10-30T13:28:09.217 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/32' 2023-10-30T13:28:09.217 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/30' 2023-10-30T13:28:09.217 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/14' 2023-10-30T13:28:09.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/18' 2023-10-30T13:28:09.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/13' 2023-10-30T13:28:09.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/33' 2023-10-30T13:28:09.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/17' 2023-10-30T13:28:09.218 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/35' 2023-10-30T13:28:09.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './158/37' 2023-10-30T13:28:09.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './158' 2023-10-30T13:28:09.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/19' 2023-10-30T13:28:09.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/15' 2023-10-30T13:28:09.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/31' 2023-10-30T13:28:09.219 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/36' 2023-10-30T13:28:09.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/20' 2023-10-30T13:28:09.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/12' 2023-10-30T13:28:09.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/39' 2023-10-30T13:28:09.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/10' 2023-10-30T13:28:09.220 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/34' 2023-10-30T13:28:09.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/38' 2023-10-30T13:28:09.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/11' 2023-10-30T13:28:09.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/16' 2023-10-30T13:28:09.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/32' 2023-10-30T13:28:09.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/30' 2023-10-30T13:28:09.221 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/14' 2023-10-30T13:28:09.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/18' 2023-10-30T13:28:09.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/13' 2023-10-30T13:28:09.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/33' 2023-10-30T13:28:09.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/17' 2023-10-30T13:28:09.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/35' 2023-10-30T13:28:09.222 INFO:tasks.workunit.client.0.smithi046.stdout:removed './159/37' 2023-10-30T13:28:09.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './159' 2023-10-30T13:28:09.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/19' 2023-10-30T13:28:09.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/15' 2023-10-30T13:28:09.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/31' 2023-10-30T13:28:09.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/36' 2023-10-30T13:28:09.223 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/20' 2023-10-30T13:28:09.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/12' 2023-10-30T13:28:09.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/39' 2023-10-30T13:28:09.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/10' 2023-10-30T13:28:09.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/34' 2023-10-30T13:28:09.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/38' 2023-10-30T13:28:09.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/11' 2023-10-30T13:28:09.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/16' 2023-10-30T13:28:09.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/32' 2023-10-30T13:28:09.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/30' 2023-10-30T13:28:09.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/14' 2023-10-30T13:28:09.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/18' 2023-10-30T13:28:09.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/13' 2023-10-30T13:28:09.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/33' 2023-10-30T13:28:09.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/17' 2023-10-30T13:28:09.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/35' 2023-10-30T13:28:09.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './160/37' 2023-10-30T13:28:09.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './160' 2023-10-30T13:28:09.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/19' 2023-10-30T13:28:09.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/15' 2023-10-30T13:28:09.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/31' 2023-10-30T13:28:09.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/36' 2023-10-30T13:28:09.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/20' 2023-10-30T13:28:09.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/12' 2023-10-30T13:28:09.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/39' 2023-10-30T13:28:09.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/10' 2023-10-30T13:28:09.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/34' 2023-10-30T13:28:09.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/38' 2023-10-30T13:28:09.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/11' 2023-10-30T13:28:09.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/16' 2023-10-30T13:28:09.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/32' 2023-10-30T13:28:09.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/30' 2023-10-30T13:28:09.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/14' 2023-10-30T13:28:09.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/18' 2023-10-30T13:28:09.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/13' 2023-10-30T13:28:09.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/33' 2023-10-30T13:28:09.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/17' 2023-10-30T13:28:09.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/35' 2023-10-30T13:28:09.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './161/37' 2023-10-30T13:28:09.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './161' 2023-10-30T13:28:09.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/19' 2023-10-30T13:28:09.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/15' 2023-10-30T13:28:09.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/31' 2023-10-30T13:28:09.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/36' 2023-10-30T13:28:09.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/20' 2023-10-30T13:28:09.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/12' 2023-10-30T13:28:09.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/39' 2023-10-30T13:28:09.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/10' 2023-10-30T13:28:09.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/34' 2023-10-30T13:28:09.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/38' 2023-10-30T13:28:09.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/11' 2023-10-30T13:28:09.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/16' 2023-10-30T13:28:09.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/32' 2023-10-30T13:28:09.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/30' 2023-10-30T13:28:09.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/14' 2023-10-30T13:28:09.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/18' 2023-10-30T13:28:09.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/13' 2023-10-30T13:28:09.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/33' 2023-10-30T13:28:09.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/17' 2023-10-30T13:28:09.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/35' 2023-10-30T13:28:09.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './162/37' 2023-10-30T13:28:09.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './162' 2023-10-30T13:28:09.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/19' 2023-10-30T13:28:09.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/15' 2023-10-30T13:28:09.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/31' 2023-10-30T13:28:09.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/36' 2023-10-30T13:28:09.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/20' 2023-10-30T13:28:09.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/12' 2023-10-30T13:28:09.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/39' 2023-10-30T13:28:09.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/10' 2023-10-30T13:28:09.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/34' 2023-10-30T13:28:09.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/38' 2023-10-30T13:28:09.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/11' 2023-10-30T13:28:09.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/16' 2023-10-30T13:28:09.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/32' 2023-10-30T13:28:09.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/30' 2023-10-30T13:28:09.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/14' 2023-10-30T13:28:09.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/18' 2023-10-30T13:28:09.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/13' 2023-10-30T13:28:09.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/33' 2023-10-30T13:28:09.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/17' 2023-10-30T13:28:09.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/35' 2023-10-30T13:28:09.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './163/37' 2023-10-30T13:28:09.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './163' 2023-10-30T13:28:09.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/19' 2023-10-30T13:28:09.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/15' 2023-10-30T13:28:09.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/31' 2023-10-30T13:28:09.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/36' 2023-10-30T13:28:09.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/20' 2023-10-30T13:28:09.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/12' 2023-10-30T13:28:09.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/39' 2023-10-30T13:28:09.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/10' 2023-10-30T13:28:09.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/34' 2023-10-30T13:28:09.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/38' 2023-10-30T13:28:09.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/11' 2023-10-30T13:28:09.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/16' 2023-10-30T13:28:09.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/32' 2023-10-30T13:28:09.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/30' 2023-10-30T13:28:09.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/14' 2023-10-30T13:28:09.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/18' 2023-10-30T13:28:09.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/13' 2023-10-30T13:28:09.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/33' 2023-10-30T13:28:09.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/17' 2023-10-30T13:28:09.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/35' 2023-10-30T13:28:09.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './164/37' 2023-10-30T13:28:09.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './164' 2023-10-30T13:28:09.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/19' 2023-10-30T13:28:09.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/15' 2023-10-30T13:28:09.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/31' 2023-10-30T13:28:09.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/36' 2023-10-30T13:28:09.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/20' 2023-10-30T13:28:09.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/12' 2023-10-30T13:28:09.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/39' 2023-10-30T13:28:09.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/10' 2023-10-30T13:28:09.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/34' 2023-10-30T13:28:09.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/38' 2023-10-30T13:28:09.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/11' 2023-10-30T13:28:09.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/16' 2023-10-30T13:28:09.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/32' 2023-10-30T13:28:09.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/30' 2023-10-30T13:28:09.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/14' 2023-10-30T13:28:09.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/18' 2023-10-30T13:28:09.247 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/13' 2023-10-30T13:28:09.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/33' 2023-10-30T13:28:09.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/17' 2023-10-30T13:28:09.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/35' 2023-10-30T13:28:09.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './165/37' 2023-10-30T13:28:09.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './165' 2023-10-30T13:28:09.248 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/19' 2023-10-30T13:28:09.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/15' 2023-10-30T13:28:09.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/31' 2023-10-30T13:28:09.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/36' 2023-10-30T13:28:09.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/20' 2023-10-30T13:28:09.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/12' 2023-10-30T13:28:09.249 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/39' 2023-10-30T13:28:09.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/10' 2023-10-30T13:28:09.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/34' 2023-10-30T13:28:09.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/38' 2023-10-30T13:28:09.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/11' 2023-10-30T13:28:09.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/16' 2023-10-30T13:28:09.250 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/32' 2023-10-30T13:28:09.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/30' 2023-10-30T13:28:09.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/14' 2023-10-30T13:28:09.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/18' 2023-10-30T13:28:09.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/13' 2023-10-30T13:28:09.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/33' 2023-10-30T13:28:09.251 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/17' 2023-10-30T13:28:09.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/35' 2023-10-30T13:28:09.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './166/37' 2023-10-30T13:28:09.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './166' 2023-10-30T13:28:09.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/19' 2023-10-30T13:28:09.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/15' 2023-10-30T13:28:09.252 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/31' 2023-10-30T13:28:09.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/36' 2023-10-30T13:28:09.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/20' 2023-10-30T13:28:09.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/12' 2023-10-30T13:28:09.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/39' 2023-10-30T13:28:09.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/10' 2023-10-30T13:28:09.253 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/34' 2023-10-30T13:28:09.254 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/38' 2023-10-30T13:28:10.552 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/11' 2023-10-30T13:28:10.552 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/16' 2023-10-30T13:28:10.553 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/32' 2023-10-30T13:28:10.553 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/30' 2023-10-30T13:28:10.553 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/14' 2023-10-30T13:28:10.553 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/18' 2023-10-30T13:28:10.553 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/13' 2023-10-30T13:28:10.554 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/33' 2023-10-30T13:28:10.554 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/17' 2023-10-30T13:28:10.554 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/35' 2023-10-30T13:28:10.554 INFO:tasks.workunit.client.0.smithi046.stdout:removed './167/37' 2023-10-30T13:28:10.554 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './167' 2023-10-30T13:28:10.555 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/19' 2023-10-30T13:28:10.555 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/15' 2023-10-30T13:28:10.555 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/31' 2023-10-30T13:28:10.555 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/36' 2023-10-30T13:28:10.555 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/20' 2023-10-30T13:28:10.555 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/12' 2023-10-30T13:28:10.556 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/39' 2023-10-30T13:28:10.556 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/10' 2023-10-30T13:28:10.556 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/34' 2023-10-30T13:28:10.556 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/38' 2023-10-30T13:28:10.556 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/11' 2023-10-30T13:28:10.557 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/16' 2023-10-30T13:28:10.557 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/32' 2023-10-30T13:28:10.557 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/30' 2023-10-30T13:28:10.557 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/14' 2023-10-30T13:28:10.557 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/18' 2023-10-30T13:28:10.557 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/13' 2023-10-30T13:28:10.558 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/33' 2023-10-30T13:28:10.558 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/17' 2023-10-30T13:28:10.558 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/35' 2023-10-30T13:28:10.558 INFO:tasks.workunit.client.0.smithi046.stdout:removed './168/37' 2023-10-30T13:28:10.558 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './168' 2023-10-30T13:28:10.559 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/19' 2023-10-30T13:28:10.559 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/15' 2023-10-30T13:28:10.559 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/31' 2023-10-30T13:28:10.559 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/36' 2023-10-30T13:28:10.559 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/20' 2023-10-30T13:28:10.559 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/12' 2023-10-30T13:28:10.560 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/39' 2023-10-30T13:28:10.560 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/10' 2023-10-30T13:28:10.560 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/34' 2023-10-30T13:28:10.560 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/38' 2023-10-30T13:28:10.560 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/11' 2023-10-30T13:28:10.560 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/16' 2023-10-30T13:28:10.561 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/32' 2023-10-30T13:28:10.561 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/30' 2023-10-30T13:28:10.561 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/14' 2023-10-30T13:28:10.561 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/18' 2023-10-30T13:28:10.561 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/13' 2023-10-30T13:28:10.561 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/33' 2023-10-30T13:28:10.562 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/17' 2023-10-30T13:28:10.562 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/35' 2023-10-30T13:28:10.562 INFO:tasks.workunit.client.0.smithi046.stdout:removed './169/37' 2023-10-30T13:28:10.562 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './169' 2023-10-30T13:28:10.562 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/19' 2023-10-30T13:28:10.563 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/15' 2023-10-30T13:28:10.563 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/31' 2023-10-30T13:28:10.563 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/36' 2023-10-30T13:28:10.563 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/20' 2023-10-30T13:28:10.563 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/12' 2023-10-30T13:28:10.563 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/39' 2023-10-30T13:28:10.564 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/10' 2023-10-30T13:28:10.564 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/34' 2023-10-30T13:28:10.564 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/38' 2023-10-30T13:28:10.564 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/11' 2023-10-30T13:28:10.564 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/16' 2023-10-30T13:28:10.564 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/32' 2023-10-30T13:28:10.565 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/30' 2023-10-30T13:28:10.565 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/14' 2023-10-30T13:28:10.565 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/18' 2023-10-30T13:28:10.565 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/13' 2023-10-30T13:28:10.565 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/33' 2023-10-30T13:28:10.565 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/17' 2023-10-30T13:28:10.566 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/35' 2023-10-30T13:28:10.566 INFO:tasks.workunit.client.0.smithi046.stdout:removed './170/37' 2023-10-30T13:28:10.566 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './170' 2023-10-30T13:28:10.566 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/19' 2023-10-30T13:28:10.566 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/15' 2023-10-30T13:28:10.566 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/31' 2023-10-30T13:28:10.567 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/36' 2023-10-30T13:28:10.567 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/20' 2023-10-30T13:28:10.567 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/12' 2023-10-30T13:28:10.567 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/39' 2023-10-30T13:28:10.567 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/10' 2023-10-30T13:28:10.567 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/34' 2023-10-30T13:28:10.568 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/38' 2023-10-30T13:28:10.568 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/11' 2023-10-30T13:28:10.568 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/16' 2023-10-30T13:28:10.568 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/32' 2023-10-30T13:28:10.568 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/30' 2023-10-30T13:28:10.568 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/14' 2023-10-30T13:28:10.569 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/18' 2023-10-30T13:28:10.569 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/13' 2023-10-30T13:28:10.569 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/33' 2023-10-30T13:28:10.569 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/17' 2023-10-30T13:28:10.569 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/35' 2023-10-30T13:28:10.569 INFO:tasks.workunit.client.0.smithi046.stdout:removed './171/37' 2023-10-30T13:28:10.570 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './171' 2023-10-30T13:28:10.570 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/19' 2023-10-30T13:28:10.570 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/15' 2023-10-30T13:28:10.570 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/31' 2023-10-30T13:28:10.570 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/36' 2023-10-30T13:28:10.570 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/20' 2023-10-30T13:28:10.571 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/12' 2023-10-30T13:28:10.571 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/39' 2023-10-30T13:28:10.571 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/10' 2023-10-30T13:28:10.571 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/34' 2023-10-30T13:28:10.571 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/38' 2023-10-30T13:28:10.571 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/11' 2023-10-30T13:28:10.572 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/16' 2023-10-30T13:28:10.572 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/32' 2023-10-30T13:28:10.572 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/30' 2023-10-30T13:28:10.572 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/14' 2023-10-30T13:28:10.572 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/18' 2023-10-30T13:28:10.572 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/13' 2023-10-30T13:28:10.573 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/33' 2023-10-30T13:28:10.573 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/17' 2023-10-30T13:28:10.573 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/35' 2023-10-30T13:28:10.573 INFO:tasks.workunit.client.0.smithi046.stdout:removed './172/37' 2023-10-30T13:28:10.573 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './172' 2023-10-30T13:28:10.573 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/19' 2023-10-30T13:28:10.574 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/15' 2023-10-30T13:28:10.574 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/31' 2023-10-30T13:28:10.574 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/36' 2023-10-30T13:28:10.574 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/20' 2023-10-30T13:28:10.574 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/12' 2023-10-30T13:28:10.574 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/39' 2023-10-30T13:28:10.575 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/10' 2023-10-30T13:28:10.575 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/34' 2023-10-30T13:28:10.575 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/38' 2023-10-30T13:28:10.575 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/11' 2023-10-30T13:28:10.575 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/16' 2023-10-30T13:28:10.575 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/32' 2023-10-30T13:28:10.576 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/30' 2023-10-30T13:28:10.576 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/14' 2023-10-30T13:28:10.576 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/18' 2023-10-30T13:28:10.576 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/13' 2023-10-30T13:28:10.576 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/33' 2023-10-30T13:28:10.576 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/17' 2023-10-30T13:28:10.577 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/35' 2023-10-30T13:28:10.577 INFO:tasks.workunit.client.0.smithi046.stdout:removed './173/37' 2023-10-30T13:28:10.577 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './173' 2023-10-30T13:28:10.577 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/19' 2023-10-30T13:28:10.577 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/15' 2023-10-30T13:28:10.577 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/31' 2023-10-30T13:28:10.578 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/36' 2023-10-30T13:28:10.578 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/20' 2023-10-30T13:28:10.578 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/12' 2023-10-30T13:28:10.578 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/39' 2023-10-30T13:28:10.578 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/10' 2023-10-30T13:28:10.578 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/34' 2023-10-30T13:28:10.579 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/38' 2023-10-30T13:28:10.579 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/11' 2023-10-30T13:28:10.579 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/16' 2023-10-30T13:28:10.579 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/32' 2023-10-30T13:28:10.579 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/30' 2023-10-30T13:28:10.579 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/14' 2023-10-30T13:28:10.580 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/18' 2023-10-30T13:28:10.580 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/13' 2023-10-30T13:28:10.580 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/33' 2023-10-30T13:28:10.580 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/17' 2023-10-30T13:28:10.580 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/35' 2023-10-30T13:28:10.580 INFO:tasks.workunit.client.0.smithi046.stdout:removed './174/37' 2023-10-30T13:28:10.581 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './174' 2023-10-30T13:28:10.581 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/19' 2023-10-30T13:28:10.581 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/15' 2023-10-30T13:28:10.581 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/31' 2023-10-30T13:28:10.581 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/36' 2023-10-30T13:28:10.581 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/20' 2023-10-30T13:28:10.582 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/12' 2023-10-30T13:28:10.582 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/39' 2023-10-30T13:28:10.582 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/10' 2023-10-30T13:28:10.582 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/34' 2023-10-30T13:28:10.582 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/38' 2023-10-30T13:28:10.582 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/11' 2023-10-30T13:28:10.583 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/16' 2023-10-30T13:28:10.583 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/32' 2023-10-30T13:28:10.583 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/30' 2023-10-30T13:28:10.583 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/14' 2023-10-30T13:28:10.583 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/18' 2023-10-30T13:28:10.584 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/13' 2023-10-30T13:28:10.584 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/33' 2023-10-30T13:28:10.584 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/17' 2023-10-30T13:28:10.584 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/35' 2023-10-30T13:28:10.584 INFO:tasks.workunit.client.0.smithi046.stdout:removed './175/37' 2023-10-30T13:28:10.584 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './175' 2023-10-30T13:28:10.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/19' 2023-10-30T13:28:10.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/15' 2023-10-30T13:28:10.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/31' 2023-10-30T13:28:10.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/36' 2023-10-30T13:28:10.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/20' 2023-10-30T13:28:10.585 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/12' 2023-10-30T13:28:10.586 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/39' 2023-10-30T13:28:10.586 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/10' 2023-10-30T13:28:10.586 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/34' 2023-10-30T13:28:10.586 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/38' 2023-10-30T13:28:10.586 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/11' 2023-10-30T13:28:10.586 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/16' 2023-10-30T13:28:10.587 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/32' 2023-10-30T13:28:10.587 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/30' 2023-10-30T13:28:10.587 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/14' 2023-10-30T13:28:10.587 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/18' 2023-10-30T13:28:10.587 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/13' 2023-10-30T13:28:10.587 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/33' 2023-10-30T13:28:10.588 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/17' 2023-10-30T13:28:10.588 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/35' 2023-10-30T13:28:10.588 INFO:tasks.workunit.client.0.smithi046.stdout:removed './176/37' 2023-10-30T13:28:10.588 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './176' 2023-10-30T13:28:10.588 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/19' 2023-10-30T13:28:10.588 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/15' 2023-10-30T13:28:12.044 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/31' 2023-10-30T13:28:12.045 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/36' 2023-10-30T13:28:12.045 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/20' 2023-10-30T13:28:12.045 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/12' 2023-10-30T13:28:12.045 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/39' 2023-10-30T13:28:12.046 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/10' 2023-10-30T13:28:12.046 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/34' 2023-10-30T13:28:12.046 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/38' 2023-10-30T13:28:12.046 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/11' 2023-10-30T13:28:12.046 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/16' 2023-10-30T13:28:12.047 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/32' 2023-10-30T13:28:12.047 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/30' 2023-10-30T13:28:12.047 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/14' 2023-10-30T13:28:12.047 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/18' 2023-10-30T13:28:12.047 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/13' 2023-10-30T13:28:12.048 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/33' 2023-10-30T13:28:12.048 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/17' 2023-10-30T13:28:12.048 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/35' 2023-10-30T13:28:12.048 INFO:tasks.workunit.client.0.smithi046.stdout:removed './177/37' 2023-10-30T13:28:12.048 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './177' 2023-10-30T13:28:12.048 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/19' 2023-10-30T13:28:12.049 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/15' 2023-10-30T13:28:12.049 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/31' 2023-10-30T13:28:12.049 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/36' 2023-10-30T13:28:12.049 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/20' 2023-10-30T13:28:12.049 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/12' 2023-10-30T13:28:12.050 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/39' 2023-10-30T13:28:12.050 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/10' 2023-10-30T13:28:12.050 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/34' 2023-10-30T13:28:12.050 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/38' 2023-10-30T13:28:12.050 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/11' 2023-10-30T13:28:12.051 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/16' 2023-10-30T13:28:12.051 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/32' 2023-10-30T13:28:12.051 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/30' 2023-10-30T13:28:12.051 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/14' 2023-10-30T13:28:12.051 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/18' 2023-10-30T13:28:12.051 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/13' 2023-10-30T13:28:12.052 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/33' 2023-10-30T13:28:12.052 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/17' 2023-10-30T13:28:12.052 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/35' 2023-10-30T13:28:12.052 INFO:tasks.workunit.client.0.smithi046.stdout:removed './178/37' 2023-10-30T13:28:12.052 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './178' 2023-10-30T13:28:12.053 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/19' 2023-10-30T13:28:12.053 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/15' 2023-10-30T13:28:12.053 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/31' 2023-10-30T13:28:12.053 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/36' 2023-10-30T13:28:12.053 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/20' 2023-10-30T13:28:12.053 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/12' 2023-10-30T13:28:12.054 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/39' 2023-10-30T13:28:12.054 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/10' 2023-10-30T13:28:12.054 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/34' 2023-10-30T13:28:12.054 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/38' 2023-10-30T13:28:12.054 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/11' 2023-10-30T13:28:12.055 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/16' 2023-10-30T13:28:12.055 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/32' 2023-10-30T13:28:12.055 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/30' 2023-10-30T13:28:12.055 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/14' 2023-10-30T13:28:12.055 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/18' 2023-10-30T13:28:12.056 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/13' 2023-10-30T13:28:12.056 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/33' 2023-10-30T13:28:12.056 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/17' 2023-10-30T13:28:12.056 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/35' 2023-10-30T13:28:12.056 INFO:tasks.workunit.client.0.smithi046.stdout:removed './179/37' 2023-10-30T13:28:12.056 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './179' 2023-10-30T13:28:12.057 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/19' 2023-10-30T13:28:12.057 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/15' 2023-10-30T13:28:12.057 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/31' 2023-10-30T13:28:12.057 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/36' 2023-10-30T13:28:12.057 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/20' 2023-10-30T13:28:12.058 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/12' 2023-10-30T13:28:12.058 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/39' 2023-10-30T13:28:12.058 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/10' 2023-10-30T13:28:12.058 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/34' 2023-10-30T13:28:12.058 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/38' 2023-10-30T13:28:12.058 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/11' 2023-10-30T13:28:12.059 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/16' 2023-10-30T13:28:12.059 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/32' 2023-10-30T13:28:12.059 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/30' 2023-10-30T13:28:12.059 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/14' 2023-10-30T13:28:12.059 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/18' 2023-10-30T13:28:12.059 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/13' 2023-10-30T13:28:12.060 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/33' 2023-10-30T13:28:12.060 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/17' 2023-10-30T13:28:12.060 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/35' 2023-10-30T13:28:12.060 INFO:tasks.workunit.client.0.smithi046.stdout:removed './180/37' 2023-10-30T13:28:12.060 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './180' 2023-10-30T13:28:12.060 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/19' 2023-10-30T13:28:12.061 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/15' 2023-10-30T13:28:12.061 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/31' 2023-10-30T13:28:12.061 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/36' 2023-10-30T13:28:12.061 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/20' 2023-10-30T13:28:12.061 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/12' 2023-10-30T13:28:12.062 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/39' 2023-10-30T13:28:12.062 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/10' 2023-10-30T13:28:12.062 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/34' 2023-10-30T13:28:12.062 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/38' 2023-10-30T13:28:12.062 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/11' 2023-10-30T13:28:12.062 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/16' 2023-10-30T13:28:12.063 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/32' 2023-10-30T13:28:12.063 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/30' 2023-10-30T13:28:12.063 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/14' 2023-10-30T13:28:12.063 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/18' 2023-10-30T13:28:12.063 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/13' 2023-10-30T13:28:12.064 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/33' 2023-10-30T13:28:12.064 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/17' 2023-10-30T13:28:12.064 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/35' 2023-10-30T13:28:12.064 INFO:tasks.workunit.client.0.smithi046.stdout:removed './181/37' 2023-10-30T13:28:12.064 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './181' 2023-10-30T13:28:12.064 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/19' 2023-10-30T13:28:12.065 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/15' 2023-10-30T13:28:12.065 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/31' 2023-10-30T13:28:12.065 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/36' 2023-10-30T13:28:12.065 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/20' 2023-10-30T13:28:12.065 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/12' 2023-10-30T13:28:12.065 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/39' 2023-10-30T13:28:12.066 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/10' 2023-10-30T13:28:12.066 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/34' 2023-10-30T13:28:12.066 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/38' 2023-10-30T13:28:12.066 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/11' 2023-10-30T13:28:12.066 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/16' 2023-10-30T13:28:12.067 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/32' 2023-10-30T13:28:12.067 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/30' 2023-10-30T13:28:12.067 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/14' 2023-10-30T13:28:12.067 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/18' 2023-10-30T13:28:12.067 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/13' 2023-10-30T13:28:12.067 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/33' 2023-10-30T13:28:12.068 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/17' 2023-10-30T13:28:12.068 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/35' 2023-10-30T13:28:12.068 INFO:tasks.workunit.client.0.smithi046.stdout:removed './182/37' 2023-10-30T13:28:12.068 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './182' 2023-10-30T13:28:12.068 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/19' 2023-10-30T13:28:12.069 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/15' 2023-10-30T13:28:12.069 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/31' 2023-10-30T13:28:12.069 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/36' 2023-10-30T13:28:12.069 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/20' 2023-10-30T13:28:12.069 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/12' 2023-10-30T13:28:12.069 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/39' 2023-10-30T13:28:12.070 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/10' 2023-10-30T13:28:12.070 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/34' 2023-10-30T13:28:12.070 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/38' 2023-10-30T13:28:12.070 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/11' 2023-10-30T13:28:12.070 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/16' 2023-10-30T13:28:12.071 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/32' 2023-10-30T13:28:12.071 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/30' 2023-10-30T13:28:12.071 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/14' 2023-10-30T13:28:12.071 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/18' 2023-10-30T13:28:12.071 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/13' 2023-10-30T13:28:12.072 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/33' 2023-10-30T13:28:12.072 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/17' 2023-10-30T13:28:12.072 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/35' 2023-10-30T13:28:12.072 INFO:tasks.workunit.client.0.smithi046.stdout:removed './183/37' 2023-10-30T13:28:12.072 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './183' 2023-10-30T13:28:12.072 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/19' 2023-10-30T13:28:12.073 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/15' 2023-10-30T13:28:12.073 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/31' 2023-10-30T13:28:12.073 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/36' 2023-10-30T13:28:12.073 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/20' 2023-10-30T13:28:12.073 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/12' 2023-10-30T13:28:12.073 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/39' 2023-10-30T13:28:12.074 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/10' 2023-10-30T13:28:12.074 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/34' 2023-10-30T13:28:12.074 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/38' 2023-10-30T13:28:12.074 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/11' 2023-10-30T13:28:12.074 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/16' 2023-10-30T13:28:12.075 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/32' 2023-10-30T13:28:12.075 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/30' 2023-10-30T13:28:12.075 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/14' 2023-10-30T13:28:12.075 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/18' 2023-10-30T13:28:12.075 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/13' 2023-10-30T13:28:12.075 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/33' 2023-10-30T13:28:12.076 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/17' 2023-10-30T13:28:12.076 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/35' 2023-10-30T13:28:12.076 INFO:tasks.workunit.client.0.smithi046.stdout:removed './184/37' 2023-10-30T13:28:12.076 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './184' 2023-10-30T13:28:12.076 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/19' 2023-10-30T13:28:12.076 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/15' 2023-10-30T13:28:12.077 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/31' 2023-10-30T13:28:12.077 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/36' 2023-10-30T13:28:12.077 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/20' 2023-10-30T13:28:12.077 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/12' 2023-10-30T13:28:12.077 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/39' 2023-10-30T13:28:12.078 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/10' 2023-10-30T13:28:12.078 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/34' 2023-10-30T13:28:12.078 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/38' 2023-10-30T13:28:12.078 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/11' 2023-10-30T13:28:12.078 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/16' 2023-10-30T13:28:12.079 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/32' 2023-10-30T13:28:12.079 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/30' 2023-10-30T13:28:12.079 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/14' 2023-10-30T13:28:12.079 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/18' 2023-10-30T13:28:12.079 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/13' 2023-10-30T13:28:12.079 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/33' 2023-10-30T13:28:12.080 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/17' 2023-10-30T13:28:12.080 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/35' 2023-10-30T13:28:12.080 INFO:tasks.workunit.client.0.smithi046.stdout:removed './185/37' 2023-10-30T13:28:12.080 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './185' 2023-10-30T13:28:12.080 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/19' 2023-10-30T13:28:12.080 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/15' 2023-10-30T13:28:12.081 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/31' 2023-10-30T13:28:12.081 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/36' 2023-10-30T13:28:12.081 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/20' 2023-10-30T13:28:12.081 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/12' 2023-10-30T13:28:12.081 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/39' 2023-10-30T13:28:12.081 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/10' 2023-10-30T13:28:12.082 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/34' 2023-10-30T13:28:12.082 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/38' 2023-10-30T13:28:12.082 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/11' 2023-10-30T13:28:12.082 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/16' 2023-10-30T13:28:12.083 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/32' 2023-10-30T13:28:12.083 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/30' 2023-10-30T13:28:12.083 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/14' 2023-10-30T13:28:12.083 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/18' 2023-10-30T13:28:14.560 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:28:14.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:28:14.689 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/13' 2023-10-30T13:28:14.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/33' 2023-10-30T13:28:14.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/17' 2023-10-30T13:28:14.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/35' 2023-10-30T13:28:14.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './186/37' 2023-10-30T13:28:14.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './186' 2023-10-30T13:28:14.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/19' 2023-10-30T13:28:14.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/15' 2023-10-30T13:28:14.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/31' 2023-10-30T13:28:14.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/36' 2023-10-30T13:28:14.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/20' 2023-10-30T13:28:14.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/12' 2023-10-30T13:28:14.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/39' 2023-10-30T13:28:14.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/10' 2023-10-30T13:28:14.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/34' 2023-10-30T13:28:14.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/38' 2023-10-30T13:28:14.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/11' 2023-10-30T13:28:14.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/16' 2023-10-30T13:28:14.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/32' 2023-10-30T13:28:14.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/30' 2023-10-30T13:28:14.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/14' 2023-10-30T13:28:14.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/18' 2023-10-30T13:28:14.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/13' 2023-10-30T13:28:14.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/33' 2023-10-30T13:28:14.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/17' 2023-10-30T13:28:14.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/35' 2023-10-30T13:28:14.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './187/37' 2023-10-30T13:28:14.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './187' 2023-10-30T13:28:14.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/19' 2023-10-30T13:28:14.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/15' 2023-10-30T13:28:14.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/31' 2023-10-30T13:28:14.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/36' 2023-10-30T13:28:14.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/20' 2023-10-30T13:28:14.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/12' 2023-10-30T13:28:14.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/39' 2023-10-30T13:28:14.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/10' 2023-10-30T13:28:14.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/34' 2023-10-30T13:28:14.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/38' 2023-10-30T13:28:14.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/11' 2023-10-30T13:28:14.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/16' 2023-10-30T13:28:14.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/32' 2023-10-30T13:28:14.698 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/30' 2023-10-30T13:28:14.698 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/14' 2023-10-30T13:28:14.698 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/18' 2023-10-30T13:28:14.698 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/13' 2023-10-30T13:28:14.698 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/33' 2023-10-30T13:28:14.699 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/17' 2023-10-30T13:28:14.699 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/35' 2023-10-30T13:28:14.699 INFO:tasks.workunit.client.0.smithi046.stdout:removed './188/37' 2023-10-30T13:28:14.699 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './188' 2023-10-30T13:28:14.699 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/19' 2023-10-30T13:28:14.700 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/15' 2023-10-30T13:28:14.700 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/31' 2023-10-30T13:28:14.700 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/36' 2023-10-30T13:28:14.700 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/20' 2023-10-30T13:28:14.700 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/12' 2023-10-30T13:28:14.701 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/39' 2023-10-30T13:28:14.701 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/10' 2023-10-30T13:28:14.701 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/34' 2023-10-30T13:28:14.701 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/38' 2023-10-30T13:28:14.701 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/11' 2023-10-30T13:28:14.701 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/16' 2023-10-30T13:28:14.702 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/32' 2023-10-30T13:28:14.702 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/30' 2023-10-30T13:28:14.702 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/14' 2023-10-30T13:28:14.702 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/18' 2023-10-30T13:28:14.702 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/13' 2023-10-30T13:28:14.703 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/33' 2023-10-30T13:28:14.703 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/17' 2023-10-30T13:28:14.703 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/35' 2023-10-30T13:28:14.703 INFO:tasks.workunit.client.0.smithi046.stdout:removed './189/37' 2023-10-30T13:28:14.703 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './189' 2023-10-30T13:28:14.703 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/19' 2023-10-30T13:28:14.704 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/15' 2023-10-30T13:28:14.704 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/31' 2023-10-30T13:28:14.704 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/36' 2023-10-30T13:28:14.704 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/20' 2023-10-30T13:28:14.704 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/12' 2023-10-30T13:28:14.704 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/39' 2023-10-30T13:28:14.705 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/10' 2023-10-30T13:28:14.705 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/34' 2023-10-30T13:28:14.705 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/38' 2023-10-30T13:28:14.705 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/11' 2023-10-30T13:28:14.705 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/16' 2023-10-30T13:28:14.706 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/32' 2023-10-30T13:28:14.706 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/30' 2023-10-30T13:28:14.706 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/14' 2023-10-30T13:28:14.706 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/18' 2023-10-30T13:28:14.706 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/13' 2023-10-30T13:28:14.707 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/33' 2023-10-30T13:28:14.707 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/17' 2023-10-30T13:28:14.707 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/35' 2023-10-30T13:28:14.707 INFO:tasks.workunit.client.0.smithi046.stdout:removed './190/37' 2023-10-30T13:28:14.707 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './190' 2023-10-30T13:28:14.707 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/19' 2023-10-30T13:28:14.708 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/15' 2023-10-30T13:28:14.708 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/31' 2023-10-30T13:28:14.708 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/36' 2023-10-30T13:28:14.708 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/20' 2023-10-30T13:28:14.708 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/12' 2023-10-30T13:28:14.709 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/39' 2023-10-30T13:28:14.709 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/10' 2023-10-30T13:28:14.709 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/34' 2023-10-30T13:28:14.709 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/38' 2023-10-30T13:28:14.709 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/11' 2023-10-30T13:28:14.710 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/16' 2023-10-30T13:28:14.710 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/32' 2023-10-30T13:28:14.710 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/30' 2023-10-30T13:28:14.710 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/14' 2023-10-30T13:28:14.710 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/18' 2023-10-30T13:28:14.710 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/13' 2023-10-30T13:28:14.711 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/33' 2023-10-30T13:28:14.711 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/17' 2023-10-30T13:28:14.711 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/35' 2023-10-30T13:28:14.711 INFO:tasks.workunit.client.0.smithi046.stdout:removed './191/37' 2023-10-30T13:28:14.711 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './191' 2023-10-30T13:28:14.712 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/19' 2023-10-30T13:28:14.712 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/15' 2023-10-30T13:28:14.712 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/31' 2023-10-30T13:28:14.712 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/36' 2023-10-30T13:28:14.712 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/20' 2023-10-30T13:28:14.713 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/12' 2023-10-30T13:28:14.713 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/39' 2023-10-30T13:28:14.713 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/10' 2023-10-30T13:28:14.713 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/34' 2023-10-30T13:28:14.713 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/38' 2023-10-30T13:28:14.714 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/11' 2023-10-30T13:28:14.714 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/16' 2023-10-30T13:28:14.714 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/32' 2023-10-30T13:28:14.714 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/30' 2023-10-30T13:28:14.714 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/14' 2023-10-30T13:28:14.736 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/18' 2023-10-30T13:28:14.736 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/13' 2023-10-30T13:28:14.737 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/33' 2023-10-30T13:28:14.737 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/17' 2023-10-30T13:28:14.737 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/35' 2023-10-30T13:28:14.737 INFO:tasks.workunit.client.0.smithi046.stdout:removed './192/37' 2023-10-30T13:28:14.737 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './192' 2023-10-30T13:28:14.738 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/19' 2023-10-30T13:28:14.738 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/15' 2023-10-30T13:28:14.738 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/31' 2023-10-30T13:28:14.738 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/36' 2023-10-30T13:28:14.739 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/20' 2023-10-30T13:28:14.739 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/12' 2023-10-30T13:28:14.739 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/39' 2023-10-30T13:28:14.739 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/10' 2023-10-30T13:28:14.739 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/34' 2023-10-30T13:28:14.739 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/38' 2023-10-30T13:28:14.740 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/11' 2023-10-30T13:28:14.740 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/16' 2023-10-30T13:28:14.740 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/32' 2023-10-30T13:28:14.740 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/30' 2023-10-30T13:28:14.740 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/14' 2023-10-30T13:28:14.741 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/18' 2023-10-30T13:28:14.741 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/13' 2023-10-30T13:28:14.741 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/33' 2023-10-30T13:28:14.741 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/17' 2023-10-30T13:28:14.741 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/35' 2023-10-30T13:28:14.742 INFO:tasks.workunit.client.0.smithi046.stdout:removed './193/37' 2023-10-30T13:28:14.742 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './193' 2023-10-30T13:28:14.742 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/19' 2023-10-30T13:28:14.742 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/15' 2023-10-30T13:28:14.742 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/31' 2023-10-30T13:28:14.743 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/36' 2023-10-30T13:28:14.743 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/20' 2023-10-30T13:28:14.743 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/12' 2023-10-30T13:28:14.743 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/39' 2023-10-30T13:28:14.743 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/10' 2023-10-30T13:28:14.743 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/34' 2023-10-30T13:28:14.744 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/38' 2023-10-30T13:28:14.744 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/11' 2023-10-30T13:28:14.744 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/16' 2023-10-30T13:28:14.744 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/32' 2023-10-30T13:28:14.744 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/30' 2023-10-30T13:28:14.745 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/14' 2023-10-30T13:28:14.745 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/18' 2023-10-30T13:28:14.745 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/13' 2023-10-30T13:28:14.745 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/33' 2023-10-30T13:28:14.745 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/17' 2023-10-30T13:28:14.745 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/35' 2023-10-30T13:28:14.746 INFO:tasks.workunit.client.0.smithi046.stdout:removed './194/37' 2023-10-30T13:28:14.746 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './194' 2023-10-30T13:28:14.746 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/19' 2023-10-30T13:28:14.746 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/15' 2023-10-30T13:28:14.746 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/31' 2023-10-30T13:28:14.747 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/36' 2023-10-30T13:28:14.747 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/20' 2023-10-30T13:28:14.747 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/12' 2023-10-30T13:28:14.747 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/39' 2023-10-30T13:28:14.747 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/10' 2023-10-30T13:28:14.748 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/34' 2023-10-30T13:28:14.748 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/38' 2023-10-30T13:28:14.748 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/11' 2023-10-30T13:28:14.748 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/16' 2023-10-30T13:28:14.748 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/32' 2023-10-30T13:28:14.749 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/30' 2023-10-30T13:28:14.749 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/14' 2023-10-30T13:28:14.749 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/18' 2023-10-30T13:28:14.749 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/13' 2023-10-30T13:28:14.749 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/33' 2023-10-30T13:28:14.750 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/17' 2023-10-30T13:28:14.750 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/35' 2023-10-30T13:28:14.750 INFO:tasks.workunit.client.0.smithi046.stdout:removed './195/37' 2023-10-30T13:28:14.750 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './195' 2023-10-30T13:28:14.750 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/19' 2023-10-30T13:28:14.750 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/15' 2023-10-30T13:28:14.751 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/31' 2023-10-30T13:28:14.751 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/36' 2023-10-30T13:28:14.751 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/20' 2023-10-30T13:28:14.751 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/12' 2023-10-30T13:28:14.751 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/39' 2023-10-30T13:28:14.752 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/10' 2023-10-30T13:28:18.067 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/34' 2023-10-30T13:28:18.067 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/38' 2023-10-30T13:28:18.067 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/11' 2023-10-30T13:28:18.067 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/16' 2023-10-30T13:28:18.068 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/32' 2023-10-30T13:28:18.068 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/30' 2023-10-30T13:28:18.068 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/14' 2023-10-30T13:28:18.068 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/18' 2023-10-30T13:28:18.068 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/13' 2023-10-30T13:28:18.068 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/33' 2023-10-30T13:28:18.069 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/17' 2023-10-30T13:28:18.069 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/35' 2023-10-30T13:28:18.069 INFO:tasks.workunit.client.0.smithi046.stdout:removed './196/37' 2023-10-30T13:28:18.069 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './196' 2023-10-30T13:28:18.069 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/19' 2023-10-30T13:28:18.069 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/15' 2023-10-30T13:28:18.070 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/31' 2023-10-30T13:28:18.070 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/36' 2023-10-30T13:28:18.070 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/20' 2023-10-30T13:28:18.070 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/12' 2023-10-30T13:28:18.070 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/39' 2023-10-30T13:28:18.071 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/10' 2023-10-30T13:28:18.071 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/34' 2023-10-30T13:28:18.071 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/38' 2023-10-30T13:28:18.071 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/11' 2023-10-30T13:28:18.071 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/16' 2023-10-30T13:28:18.072 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/32' 2023-10-30T13:28:18.072 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/30' 2023-10-30T13:28:18.072 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/14' 2023-10-30T13:28:18.072 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/18' 2023-10-30T13:28:18.072 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/13' 2023-10-30T13:28:18.073 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/33' 2023-10-30T13:28:18.073 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/17' 2023-10-30T13:28:18.073 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/35' 2023-10-30T13:28:18.073 INFO:tasks.workunit.client.0.smithi046.stdout:removed './197/37' 2023-10-30T13:28:18.073 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './197' 2023-10-30T13:28:18.073 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/19' 2023-10-30T13:28:18.074 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/15' 2023-10-30T13:28:18.074 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/31' 2023-10-30T13:28:18.074 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/36' 2023-10-30T13:28:18.074 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/20' 2023-10-30T13:28:18.074 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/12' 2023-10-30T13:28:18.075 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/39' 2023-10-30T13:28:18.075 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/10' 2023-10-30T13:28:18.075 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/34' 2023-10-30T13:28:18.075 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/38' 2023-10-30T13:28:18.075 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/11' 2023-10-30T13:28:18.076 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/16' 2023-10-30T13:28:18.076 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/32' 2023-10-30T13:28:18.076 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/30' 2023-10-30T13:28:18.076 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/14' 2023-10-30T13:28:18.076 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/18' 2023-10-30T13:28:18.076 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/13' 2023-10-30T13:28:18.077 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/33' 2023-10-30T13:28:18.077 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/17' 2023-10-30T13:28:18.077 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/35' 2023-10-30T13:28:18.077 INFO:tasks.workunit.client.0.smithi046.stdout:removed './198/37' 2023-10-30T13:28:18.077 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './198' 2023-10-30T13:28:18.078 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/19' 2023-10-30T13:28:18.078 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/15' 2023-10-30T13:28:18.078 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/31' 2023-10-30T13:28:18.078 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/36' 2023-10-30T13:28:18.078 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/20' 2023-10-30T13:28:18.079 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/12' 2023-10-30T13:28:18.079 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/39' 2023-10-30T13:28:18.079 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/10' 2023-10-30T13:28:18.079 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/34' 2023-10-30T13:28:18.079 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/38' 2023-10-30T13:28:18.079 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/11' 2023-10-30T13:28:18.080 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/16' 2023-10-30T13:28:18.080 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/32' 2023-10-30T13:28:18.080 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/30' 2023-10-30T13:28:18.080 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/14' 2023-10-30T13:28:18.080 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/18' 2023-10-30T13:28:18.081 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/13' 2023-10-30T13:28:18.081 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/33' 2023-10-30T13:28:18.081 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/17' 2023-10-30T13:28:18.081 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/35' 2023-10-30T13:28:18.081 INFO:tasks.workunit.client.0.smithi046.stdout:removed './199/37' 2023-10-30T13:28:18.081 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './199' 2023-10-30T13:28:18.082 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/21' 2023-10-30T13:28:18.082 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/27' 2023-10-30T13:28:18.082 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/31' 2023-10-30T13:28:18.082 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/36' 2023-10-30T13:28:18.082 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/20' 2023-10-30T13:28:18.083 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/39' 2023-10-30T13:28:18.083 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/34' 2023-10-30T13:28:18.083 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/38' 2023-10-30T13:28:18.083 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/28' 2023-10-30T13:28:18.083 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/24' 2023-10-30T13:28:18.084 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/29' 2023-10-30T13:28:18.084 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/32' 2023-10-30T13:28:18.084 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/22' 2023-10-30T13:28:18.084 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/25' 2023-10-30T13:28:18.084 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/30' 2023-10-30T13:28:18.084 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/33' 2023-10-30T13:28:18.085 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/23' 2023-10-30T13:28:18.085 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/35' 2023-10-30T13:28:18.085 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/26' 2023-10-30T13:28:18.085 INFO:tasks.workunit.client.0.smithi046.stdout:removed './200/37' 2023-10-30T13:28:18.085 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './200' 2023-10-30T13:28:18.086 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/21' 2023-10-30T13:28:18.086 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/27' 2023-10-30T13:28:18.086 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/31' 2023-10-30T13:28:18.086 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/36' 2023-10-30T13:28:18.086 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/20' 2023-10-30T13:28:18.087 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/39' 2023-10-30T13:28:18.087 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/34' 2023-10-30T13:28:18.087 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/38' 2023-10-30T13:28:18.087 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/28' 2023-10-30T13:28:18.087 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/24' 2023-10-30T13:28:18.088 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/29' 2023-10-30T13:28:18.088 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/32' 2023-10-30T13:28:18.088 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/22' 2023-10-30T13:28:18.088 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/25' 2023-10-30T13:28:18.088 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/30' 2023-10-30T13:28:18.088 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/33' 2023-10-30T13:28:18.089 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/23' 2023-10-30T13:28:18.089 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/35' 2023-10-30T13:28:18.089 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/26' 2023-10-30T13:28:18.089 INFO:tasks.workunit.client.0.smithi046.stdout:removed './201/37' 2023-10-30T13:28:18.089 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './201' 2023-10-30T13:28:18.090 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/21' 2023-10-30T13:28:18.090 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/27' 2023-10-30T13:28:18.090 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/31' 2023-10-30T13:28:18.090 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/36' 2023-10-30T13:28:18.090 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/20' 2023-10-30T13:28:18.091 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/39' 2023-10-30T13:28:18.091 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/34' 2023-10-30T13:28:18.091 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/38' 2023-10-30T13:28:18.091 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/28' 2023-10-30T13:28:18.091 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/24' 2023-10-30T13:28:18.091 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/29' 2023-10-30T13:28:18.092 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/32' 2023-10-30T13:28:18.092 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/22' 2023-10-30T13:28:18.092 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/25' 2023-10-30T13:28:18.092 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/30' 2023-10-30T13:28:18.092 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/33' 2023-10-30T13:28:18.093 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/23' 2023-10-30T13:28:18.093 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/35' 2023-10-30T13:28:18.093 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/26' 2023-10-30T13:28:18.093 INFO:tasks.workunit.client.0.smithi046.stdout:removed './202/37' 2023-10-30T13:28:18.093 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './202' 2023-10-30T13:28:18.094 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/21' 2023-10-30T13:28:18.094 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/27' 2023-10-30T13:28:18.094 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/31' 2023-10-30T13:28:18.094 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/36' 2023-10-30T13:28:18.094 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/20' 2023-10-30T13:28:18.094 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/39' 2023-10-30T13:28:18.095 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/34' 2023-10-30T13:28:18.095 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/38' 2023-10-30T13:28:18.095 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/28' 2023-10-30T13:28:18.095 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/24' 2023-10-30T13:28:18.095 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/29' 2023-10-30T13:28:18.096 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/32' 2023-10-30T13:28:18.096 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/22' 2023-10-30T13:28:18.096 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/25' 2023-10-30T13:28:18.096 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/30' 2023-10-30T13:28:18.096 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/33' 2023-10-30T13:28:18.097 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/23' 2023-10-30T13:28:18.097 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/35' 2023-10-30T13:28:18.097 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/26' 2023-10-30T13:28:18.097 INFO:tasks.workunit.client.0.smithi046.stdout:removed './203/37' 2023-10-30T13:28:18.097 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './203' 2023-10-30T13:28:18.097 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/21' 2023-10-30T13:28:18.098 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/27' 2023-10-30T13:28:18.098 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/31' 2023-10-30T13:28:18.098 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/36' 2023-10-30T13:28:18.098 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/20' 2023-10-30T13:28:18.098 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/39' 2023-10-30T13:28:18.099 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/34' 2023-10-30T13:28:18.099 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/38' 2023-10-30T13:28:18.099 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/28' 2023-10-30T13:28:18.099 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/24' 2023-10-30T13:28:18.099 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/29' 2023-10-30T13:28:18.100 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/32' 2023-10-30T13:28:18.100 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/22' 2023-10-30T13:28:18.100 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/25' 2023-10-30T13:28:18.100 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/30' 2023-10-30T13:28:18.100 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/33' 2023-10-30T13:28:18.100 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/23' 2023-10-30T13:28:18.101 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/35' 2023-10-30T13:28:18.101 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/26' 2023-10-30T13:28:18.101 INFO:tasks.workunit.client.0.smithi046.stdout:removed './204/37' 2023-10-30T13:28:18.101 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './204' 2023-10-30T13:28:18.101 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/21' 2023-10-30T13:28:18.102 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/27' 2023-10-30T13:28:18.102 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/31' 2023-10-30T13:28:18.102 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/36' 2023-10-30T13:28:18.102 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/20' 2023-10-30T13:28:18.102 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/39' 2023-10-30T13:28:18.103 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/34' 2023-10-30T13:28:18.103 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/38' 2023-10-30T13:28:18.103 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/28' 2023-10-30T13:28:18.103 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/24' 2023-10-30T13:28:18.103 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/29' 2023-10-30T13:28:18.103 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/32' 2023-10-30T13:28:18.104 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/22' 2023-10-30T13:28:18.104 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/25' 2023-10-30T13:28:18.104 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/30' 2023-10-30T13:28:18.104 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/33' 2023-10-30T13:28:18.104 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/23' 2023-10-30T13:28:18.105 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/35' 2023-10-30T13:28:18.105 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/26' 2023-10-30T13:28:18.105 INFO:tasks.workunit.client.0.smithi046.stdout:removed './205/37' 2023-10-30T13:28:18.105 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './205' 2023-10-30T13:28:18.105 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/21' 2023-10-30T13:28:18.105 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/27' 2023-10-30T13:28:18.106 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/31' 2023-10-30T13:28:18.106 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/36' 2023-10-30T13:28:18.106 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/20' 2023-10-30T13:28:18.106 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/39' 2023-10-30T13:28:29.533 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/34' 2023-10-30T13:28:29.534 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/38' 2023-10-30T13:28:29.534 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/28' 2023-10-30T13:28:29.534 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/24' 2023-10-30T13:28:29.534 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/29' 2023-10-30T13:28:29.535 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/32' 2023-10-30T13:28:29.535 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/22' 2023-10-30T13:28:29.535 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/25' 2023-10-30T13:28:29.535 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/30' 2023-10-30T13:28:29.535 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/33' 2023-10-30T13:28:29.535 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/23' 2023-10-30T13:28:29.536 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/35' 2023-10-30T13:28:29.536 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/26' 2023-10-30T13:28:29.536 INFO:tasks.workunit.client.0.smithi046.stdout:removed './206/37' 2023-10-30T13:28:29.536 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './206' 2023-10-30T13:28:29.536 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/21' 2023-10-30T13:28:29.537 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/27' 2023-10-30T13:28:29.537 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/31' 2023-10-30T13:28:29.537 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/36' 2023-10-30T13:28:29.537 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/20' 2023-10-30T13:28:29.537 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/39' 2023-10-30T13:28:29.538 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/34' 2023-10-30T13:28:29.538 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/38' 2023-10-30T13:28:29.538 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/28' 2023-10-30T13:28:29.538 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/24' 2023-10-30T13:28:29.538 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/29' 2023-10-30T13:28:29.539 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/32' 2023-10-30T13:28:29.539 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/22' 2023-10-30T13:28:29.539 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/25' 2023-10-30T13:28:29.539 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/30' 2023-10-30T13:28:29.539 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/33' 2023-10-30T13:28:29.539 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/23' 2023-10-30T13:28:29.540 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/35' 2023-10-30T13:28:29.540 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/26' 2023-10-30T13:28:29.540 INFO:tasks.workunit.client.0.smithi046.stdout:removed './207/37' 2023-10-30T13:28:29.540 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './207' 2023-10-30T13:28:29.540 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/21' 2023-10-30T13:28:29.541 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/27' 2023-10-30T13:28:29.541 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/31' 2023-10-30T13:28:29.541 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/36' 2023-10-30T13:28:29.541 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/20' 2023-10-30T13:28:29.541 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/39' 2023-10-30T13:28:29.542 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/34' 2023-10-30T13:28:29.542 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/38' 2023-10-30T13:28:29.542 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/28' 2023-10-30T13:28:29.542 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/24' 2023-10-30T13:28:29.542 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/29' 2023-10-30T13:28:29.543 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/32' 2023-10-30T13:28:29.543 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/22' 2023-10-30T13:28:29.543 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/25' 2023-10-30T13:28:29.543 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/30' 2023-10-30T13:28:29.543 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/33' 2023-10-30T13:28:29.543 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/23' 2023-10-30T13:28:29.544 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/35' 2023-10-30T13:28:29.544 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/26' 2023-10-30T13:28:29.544 INFO:tasks.workunit.client.0.smithi046.stdout:removed './208/37' 2023-10-30T13:28:29.544 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './208' 2023-10-30T13:28:29.544 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/21' 2023-10-30T13:28:29.544 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/27' 2023-10-30T13:28:29.545 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/31' 2023-10-30T13:28:29.545 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/36' 2023-10-30T13:28:29.545 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/20' 2023-10-30T13:28:29.545 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/39' 2023-10-30T13:28:29.545 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/34' 2023-10-30T13:28:29.546 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/38' 2023-10-30T13:28:29.546 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/28' 2023-10-30T13:28:29.546 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/24' 2023-10-30T13:28:29.546 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/29' 2023-10-30T13:28:29.546 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/32' 2023-10-30T13:28:29.547 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/22' 2023-10-30T13:28:29.547 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/25' 2023-10-30T13:28:29.547 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/30' 2023-10-30T13:28:29.547 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/33' 2023-10-30T13:28:29.547 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/23' 2023-10-30T13:28:29.547 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/35' 2023-10-30T13:28:29.548 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/26' 2023-10-30T13:28:29.548 INFO:tasks.workunit.client.0.smithi046.stdout:removed './209/37' 2023-10-30T13:28:29.548 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './209' 2023-10-30T13:28:29.548 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/21' 2023-10-30T13:28:29.548 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/27' 2023-10-30T13:28:29.549 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/31' 2023-10-30T13:28:29.549 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/36' 2023-10-30T13:28:29.549 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/20' 2023-10-30T13:28:29.549 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/39' 2023-10-30T13:28:29.549 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/34' 2023-10-30T13:28:29.549 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/38' 2023-10-30T13:28:29.550 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/28' 2023-10-30T13:28:29.550 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/24' 2023-10-30T13:28:29.550 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/29' 2023-10-30T13:28:29.550 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/32' 2023-10-30T13:28:29.550 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/22' 2023-10-30T13:28:29.551 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/25' 2023-10-30T13:28:29.551 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/30' 2023-10-30T13:28:29.551 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/33' 2023-10-30T13:28:29.551 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/23' 2023-10-30T13:28:29.551 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/35' 2023-10-30T13:28:29.552 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/26' 2023-10-30T13:28:29.552 INFO:tasks.workunit.client.0.smithi046.stdout:removed './210/37' 2023-10-30T13:28:29.552 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './210' 2023-10-30T13:28:29.552 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/21' 2023-10-30T13:28:29.552 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/27' 2023-10-30T13:28:29.552 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/31' 2023-10-30T13:28:29.553 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/36' 2023-10-30T13:28:29.553 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/20' 2023-10-30T13:28:29.553 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/39' 2023-10-30T13:28:29.553 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/34' 2023-10-30T13:28:29.553 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/38' 2023-10-30T13:28:29.554 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/28' 2023-10-30T13:28:29.554 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/24' 2023-10-30T13:28:29.554 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/29' 2023-10-30T13:28:29.554 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/32' 2023-10-30T13:28:29.554 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/22' 2023-10-30T13:28:29.554 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/25' 2023-10-30T13:28:29.555 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/30' 2023-10-30T13:28:29.555 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/33' 2023-10-30T13:28:29.555 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/23' 2023-10-30T13:28:29.555 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/35' 2023-10-30T13:28:29.555 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/26' 2023-10-30T13:28:29.556 INFO:tasks.workunit.client.0.smithi046.stdout:removed './211/37' 2023-10-30T13:28:29.556 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './211' 2023-10-30T13:28:29.556 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/21' 2023-10-30T13:28:29.556 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/27' 2023-10-30T13:28:29.556 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/31' 2023-10-30T13:28:29.556 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/36' 2023-10-30T13:28:29.557 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/20' 2023-10-30T13:28:29.557 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/39' 2023-10-30T13:28:29.557 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/34' 2023-10-30T13:28:29.557 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/38' 2023-10-30T13:28:29.557 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/28' 2023-10-30T13:28:29.558 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/24' 2023-10-30T13:28:29.558 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/29' 2023-10-30T13:28:29.558 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/32' 2023-10-30T13:28:29.558 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/22' 2023-10-30T13:28:29.558 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/25' 2023-10-30T13:28:29.559 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/30' 2023-10-30T13:28:29.559 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/33' 2023-10-30T13:28:29.559 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/23' 2023-10-30T13:28:29.559 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/35' 2023-10-30T13:28:29.559 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/26' 2023-10-30T13:28:29.559 INFO:tasks.workunit.client.0.smithi046.stdout:removed './212/37' 2023-10-30T13:28:29.560 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './212' 2023-10-30T13:28:29.560 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/21' 2023-10-30T13:28:29.560 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/27' 2023-10-30T13:28:29.560 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/31' 2023-10-30T13:28:29.560 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/36' 2023-10-30T13:28:29.561 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/20' 2023-10-30T13:28:29.561 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/39' 2023-10-30T13:28:29.561 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/34' 2023-10-30T13:28:29.561 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/38' 2023-10-30T13:28:29.561 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/28' 2023-10-30T13:28:29.561 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/24' 2023-10-30T13:28:29.562 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/29' 2023-10-30T13:28:29.562 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/32' 2023-10-30T13:28:29.562 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/22' 2023-10-30T13:28:29.562 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/25' 2023-10-30T13:28:29.562 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/30' 2023-10-30T13:28:29.563 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/33' 2023-10-30T13:28:29.563 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/23' 2023-10-30T13:28:29.563 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/35' 2023-10-30T13:28:29.563 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/26' 2023-10-30T13:28:29.563 INFO:tasks.workunit.client.0.smithi046.stdout:removed './213/37' 2023-10-30T13:28:29.563 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './213' 2023-10-30T13:28:29.564 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/21' 2023-10-30T13:28:29.564 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/27' 2023-10-30T13:28:29.564 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/31' 2023-10-30T13:28:29.564 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/36' 2023-10-30T13:28:29.564 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/20' 2023-10-30T13:28:29.565 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/39' 2023-10-30T13:28:29.565 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/34' 2023-10-30T13:28:29.565 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/38' 2023-10-30T13:28:29.565 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/28' 2023-10-30T13:28:29.565 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/24' 2023-10-30T13:28:29.565 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/29' 2023-10-30T13:28:29.566 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/32' 2023-10-30T13:28:29.566 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/22' 2023-10-30T13:28:29.566 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/25' 2023-10-30T13:28:29.566 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/30' 2023-10-30T13:28:29.566 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/33' 2023-10-30T13:28:29.567 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/23' 2023-10-30T13:28:29.567 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/35' 2023-10-30T13:28:29.567 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/26' 2023-10-30T13:28:29.567 INFO:tasks.workunit.client.0.smithi046.stdout:removed './214/37' 2023-10-30T13:28:29.567 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './214' 2023-10-30T13:28:29.568 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/21' 2023-10-30T13:28:29.568 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/27' 2023-10-30T13:28:29.568 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/31' 2023-10-30T13:28:29.568 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/36' 2023-10-30T13:28:29.568 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/20' 2023-10-30T13:28:29.568 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/39' 2023-10-30T13:28:29.569 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/34' 2023-10-30T13:28:29.569 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/38' 2023-10-30T13:28:29.569 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/28' 2023-10-30T13:28:29.569 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/24' 2023-10-30T13:28:29.569 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/29' 2023-10-30T13:28:29.570 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/32' 2023-10-30T13:28:29.570 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/22' 2023-10-30T13:28:29.570 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/25' 2023-10-30T13:28:29.570 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/30' 2023-10-30T13:28:29.570 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/33' 2023-10-30T13:28:29.570 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/23' 2023-10-30T13:28:29.571 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/35' 2023-10-30T13:28:29.571 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/26' 2023-10-30T13:28:29.571 INFO:tasks.workunit.client.0.smithi046.stdout:removed './215/37' 2023-10-30T13:28:29.571 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './215' 2023-10-30T13:28:29.571 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/21' 2023-10-30T13:28:29.572 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/27' 2023-10-30T13:28:29.572 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/31' 2023-10-30T13:28:29.572 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/36' 2023-10-30T13:28:29.572 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/20' 2023-10-30T13:28:29.572 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/39' 2023-10-30T13:28:29.573 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/34' 2023-10-30T13:28:29.573 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/38' 2023-10-30T13:28:32.288 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/28' 2023-10-30T13:28:32.288 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/24' 2023-10-30T13:28:32.289 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/29' 2023-10-30T13:28:32.289 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/32' 2023-10-30T13:28:32.289 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/22' 2023-10-30T13:28:32.289 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/25' 2023-10-30T13:28:32.289 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/30' 2023-10-30T13:28:32.290 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/33' 2023-10-30T13:28:32.290 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/23' 2023-10-30T13:28:32.290 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/35' 2023-10-30T13:28:32.290 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/26' 2023-10-30T13:28:32.290 INFO:tasks.workunit.client.0.smithi046.stdout:removed './216/37' 2023-10-30T13:28:32.291 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './216' 2023-10-30T13:28:32.291 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/21' 2023-10-30T13:28:32.291 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/27' 2023-10-30T13:28:32.291 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/31' 2023-10-30T13:28:32.291 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/36' 2023-10-30T13:28:32.291 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/20' 2023-10-30T13:28:32.292 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/39' 2023-10-30T13:28:32.292 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/34' 2023-10-30T13:28:32.292 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/38' 2023-10-30T13:28:32.292 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/28' 2023-10-30T13:28:32.292 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/24' 2023-10-30T13:28:32.293 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/29' 2023-10-30T13:28:32.293 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/32' 2023-10-30T13:28:32.293 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/22' 2023-10-30T13:28:32.293 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/25' 2023-10-30T13:28:32.293 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/30' 2023-10-30T13:28:32.294 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/33' 2023-10-30T13:28:32.294 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/23' 2023-10-30T13:28:32.294 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/35' 2023-10-30T13:28:32.294 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/26' 2023-10-30T13:28:32.294 INFO:tasks.workunit.client.0.smithi046.stdout:removed './217/37' 2023-10-30T13:28:32.295 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './217' 2023-10-30T13:28:32.295 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/21' 2023-10-30T13:28:32.295 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/27' 2023-10-30T13:28:32.295 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/31' 2023-10-30T13:28:32.295 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/36' 2023-10-30T13:28:32.296 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/20' 2023-10-30T13:28:32.296 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/39' 2023-10-30T13:28:32.296 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/34' 2023-10-30T13:28:32.296 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/38' 2023-10-30T13:28:32.296 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/28' 2023-10-30T13:28:32.296 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/24' 2023-10-30T13:28:32.297 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/29' 2023-10-30T13:28:32.297 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/32' 2023-10-30T13:28:32.297 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/22' 2023-10-30T13:28:32.297 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/25' 2023-10-30T13:28:32.298 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/30' 2023-10-30T13:28:32.298 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/33' 2023-10-30T13:28:32.298 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/23' 2023-10-30T13:28:32.298 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/35' 2023-10-30T13:28:32.298 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/26' 2023-10-30T13:28:32.298 INFO:tasks.workunit.client.0.smithi046.stdout:removed './218/37' 2023-10-30T13:28:32.299 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './218' 2023-10-30T13:28:32.299 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/21' 2023-10-30T13:28:32.299 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/27' 2023-10-30T13:28:32.299 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/31' 2023-10-30T13:28:32.299 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/36' 2023-10-30T13:28:32.300 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/20' 2023-10-30T13:28:32.300 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/39' 2023-10-30T13:28:32.300 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/34' 2023-10-30T13:28:32.300 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/38' 2023-10-30T13:28:32.300 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/28' 2023-10-30T13:28:32.301 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/24' 2023-10-30T13:28:32.301 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/29' 2023-10-30T13:28:32.301 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/32' 2023-10-30T13:28:32.301 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/22' 2023-10-30T13:28:32.301 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/25' 2023-10-30T13:28:32.301 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/30' 2023-10-30T13:28:32.302 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/33' 2023-10-30T13:28:32.302 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/23' 2023-10-30T13:28:32.302 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/35' 2023-10-30T13:28:32.302 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/26' 2023-10-30T13:28:32.302 INFO:tasks.workunit.client.0.smithi046.stdout:removed './219/37' 2023-10-30T13:28:32.303 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './219' 2023-10-30T13:28:32.303 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/21' 2023-10-30T13:28:32.303 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/27' 2023-10-30T13:28:32.303 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/31' 2023-10-30T13:28:32.303 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/36' 2023-10-30T13:28:32.304 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/20' 2023-10-30T13:28:32.304 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/39' 2023-10-30T13:28:32.304 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/34' 2023-10-30T13:28:32.304 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/38' 2023-10-30T13:28:32.304 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/28' 2023-10-30T13:28:32.305 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/24' 2023-10-30T13:28:32.305 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/29' 2023-10-30T13:28:32.305 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/32' 2023-10-30T13:28:32.305 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/22' 2023-10-30T13:28:32.305 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/25' 2023-10-30T13:28:32.306 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/30' 2023-10-30T13:28:32.306 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/33' 2023-10-30T13:28:32.306 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/23' 2023-10-30T13:28:32.306 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/35' 2023-10-30T13:28:32.306 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/26' 2023-10-30T13:28:32.307 INFO:tasks.workunit.client.0.smithi046.stdout:removed './220/37' 2023-10-30T13:28:32.307 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './220' 2023-10-30T13:28:32.307 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/21' 2023-10-30T13:28:32.307 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/27' 2023-10-30T13:28:32.307 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/31' 2023-10-30T13:28:32.307 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/36' 2023-10-30T13:28:32.308 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/20' 2023-10-30T13:28:32.308 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/39' 2023-10-30T13:28:32.308 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/34' 2023-10-30T13:28:32.308 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/38' 2023-10-30T13:28:32.308 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/28' 2023-10-30T13:28:32.309 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/24' 2023-10-30T13:28:32.309 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/29' 2023-10-30T13:28:32.309 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/32' 2023-10-30T13:28:32.309 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/22' 2023-10-30T13:28:32.309 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/25' 2023-10-30T13:28:32.310 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/30' 2023-10-30T13:28:32.310 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/33' 2023-10-30T13:28:32.310 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/23' 2023-10-30T13:28:32.310 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/35' 2023-10-30T13:28:32.310 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/26' 2023-10-30T13:28:32.310 INFO:tasks.workunit.client.0.smithi046.stdout:removed './221/37' 2023-10-30T13:28:32.311 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './221' 2023-10-30T13:28:32.311 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/21' 2023-10-30T13:28:32.311 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/27' 2023-10-30T13:28:32.311 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/31' 2023-10-30T13:28:32.311 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/36' 2023-10-30T13:28:32.312 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/20' 2023-10-30T13:28:32.312 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/39' 2023-10-30T13:28:32.312 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/34' 2023-10-30T13:28:32.312 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/38' 2023-10-30T13:28:32.312 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/28' 2023-10-30T13:28:32.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/24' 2023-10-30T13:28:32.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/29' 2023-10-30T13:28:32.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/32' 2023-10-30T13:28:32.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/22' 2023-10-30T13:28:32.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/25' 2023-10-30T13:28:32.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/30' 2023-10-30T13:28:32.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/33' 2023-10-30T13:28:32.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/23' 2023-10-30T13:28:32.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/35' 2023-10-30T13:28:32.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/26' 2023-10-30T13:28:32.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './222/37' 2023-10-30T13:28:32.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './222' 2023-10-30T13:28:32.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/21' 2023-10-30T13:28:32.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/27' 2023-10-30T13:28:32.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/31' 2023-10-30T13:28:32.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/36' 2023-10-30T13:28:32.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/20' 2023-10-30T13:28:32.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/39' 2023-10-30T13:28:32.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/34' 2023-10-30T13:28:32.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/38' 2023-10-30T13:28:32.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/28' 2023-10-30T13:28:32.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/24' 2023-10-30T13:28:32.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/29' 2023-10-30T13:28:32.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/32' 2023-10-30T13:28:32.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/22' 2023-10-30T13:28:32.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/25' 2023-10-30T13:28:32.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/30' 2023-10-30T13:28:32.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/33' 2023-10-30T13:28:32.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/23' 2023-10-30T13:28:32.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/35' 2023-10-30T13:28:32.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/26' 2023-10-30T13:28:32.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './223/37' 2023-10-30T13:28:32.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './223' 2023-10-30T13:28:32.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/21' 2023-10-30T13:28:32.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/27' 2023-10-30T13:28:32.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/31' 2023-10-30T13:28:32.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/36' 2023-10-30T13:28:32.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/20' 2023-10-30T13:28:32.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/39' 2023-10-30T13:28:32.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/34' 2023-10-30T13:28:32.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/38' 2023-10-30T13:28:32.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/28' 2023-10-30T13:28:32.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/24' 2023-10-30T13:28:32.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/29' 2023-10-30T13:28:32.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/32' 2023-10-30T13:28:32.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/22' 2023-10-30T13:28:32.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/25' 2023-10-30T13:28:32.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/30' 2023-10-30T13:28:32.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/33' 2023-10-30T13:28:32.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/23' 2023-10-30T13:28:32.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/35' 2023-10-30T13:28:32.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/26' 2023-10-30T13:28:32.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './224/37' 2023-10-30T13:28:32.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './224' 2023-10-30T13:28:32.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/21' 2023-10-30T13:28:32.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/27' 2023-10-30T13:28:32.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/31' 2023-10-30T13:28:32.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/36' 2023-10-30T13:28:32.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/20' 2023-10-30T13:28:32.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/39' 2023-10-30T13:28:32.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/34' 2023-10-30T13:28:32.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/38' 2023-10-30T13:28:32.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/28' 2023-10-30T13:28:32.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/24' 2023-10-30T13:28:32.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/29' 2023-10-30T13:28:32.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/32' 2023-10-30T13:28:32.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/22' 2023-10-30T13:28:32.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/25' 2023-10-30T13:28:32.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/30' 2023-10-30T13:28:32.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/33' 2023-10-30T13:28:32.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/23' 2023-10-30T13:28:32.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/35' 2023-10-30T13:28:32.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/26' 2023-10-30T13:28:32.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './225/37' 2023-10-30T13:28:32.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './225' 2023-10-30T13:28:32.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/21' 2023-10-30T13:28:32.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/27' 2023-10-30T13:28:32.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/31' 2023-10-30T13:28:32.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/36' 2023-10-30T13:28:32.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/20' 2023-10-30T13:28:32.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/39' 2023-10-30T13:28:32.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/34' 2023-10-30T13:28:32.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/38' 2023-10-30T13:28:32.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/28' 2023-10-30T13:28:32.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/24' 2023-10-30T13:28:34.396 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/29' 2023-10-30T13:28:34.396 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/32' 2023-10-30T13:28:34.397 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/22' 2023-10-30T13:28:34.397 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/25' 2023-10-30T13:28:34.397 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/30' 2023-10-30T13:28:34.397 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/33' 2023-10-30T13:28:34.397 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/23' 2023-10-30T13:28:34.398 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/35' 2023-10-30T13:28:34.398 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/26' 2023-10-30T13:28:34.398 INFO:tasks.workunit.client.0.smithi046.stdout:removed './226/37' 2023-10-30T13:28:34.398 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './226' 2023-10-30T13:28:34.398 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/21' 2023-10-30T13:28:34.399 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/27' 2023-10-30T13:28:34.399 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/31' 2023-10-30T13:28:34.399 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/36' 2023-10-30T13:28:34.399 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/20' 2023-10-30T13:28:34.399 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/39' 2023-10-30T13:28:34.400 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/34' 2023-10-30T13:28:34.400 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/38' 2023-10-30T13:28:34.400 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/28' 2023-10-30T13:28:34.400 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/24' 2023-10-30T13:28:34.400 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/29' 2023-10-30T13:28:34.400 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/32' 2023-10-30T13:28:34.401 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/22' 2023-10-30T13:28:34.401 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/25' 2023-10-30T13:28:34.401 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/30' 2023-10-30T13:28:34.401 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/33' 2023-10-30T13:28:34.401 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/23' 2023-10-30T13:28:34.402 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/35' 2023-10-30T13:28:34.402 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/26' 2023-10-30T13:28:34.402 INFO:tasks.workunit.client.0.smithi046.stdout:removed './227/37' 2023-10-30T13:28:34.402 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './227' 2023-10-30T13:28:34.402 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/21' 2023-10-30T13:28:34.403 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/27' 2023-10-30T13:28:34.403 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/31' 2023-10-30T13:28:34.403 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/36' 2023-10-30T13:28:34.403 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/20' 2023-10-30T13:28:34.403 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/39' 2023-10-30T13:28:34.403 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/34' 2023-10-30T13:28:34.404 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/38' 2023-10-30T13:28:34.404 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/28' 2023-10-30T13:28:34.404 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/24' 2023-10-30T13:28:34.404 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/29' 2023-10-30T13:28:34.404 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/32' 2023-10-30T13:28:34.404 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/22' 2023-10-30T13:28:34.405 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/25' 2023-10-30T13:28:34.405 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/30' 2023-10-30T13:28:34.405 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/33' 2023-10-30T13:28:34.405 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/23' 2023-10-30T13:28:34.405 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/35' 2023-10-30T13:28:34.406 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/26' 2023-10-30T13:28:34.406 INFO:tasks.workunit.client.0.smithi046.stdout:removed './228/37' 2023-10-30T13:28:34.406 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './228' 2023-10-30T13:28:34.406 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/21' 2023-10-30T13:28:34.406 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/27' 2023-10-30T13:28:34.407 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/31' 2023-10-30T13:28:34.407 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/36' 2023-10-30T13:28:34.407 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/20' 2023-10-30T13:28:34.407 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/39' 2023-10-30T13:28:34.407 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/34' 2023-10-30T13:28:34.408 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/38' 2023-10-30T13:28:34.408 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/28' 2023-10-30T13:28:34.408 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/24' 2023-10-30T13:28:34.408 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/29' 2023-10-30T13:28:34.408 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/32' 2023-10-30T13:28:34.408 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/22' 2023-10-30T13:28:34.409 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/25' 2023-10-30T13:28:34.409 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/30' 2023-10-30T13:28:34.409 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/33' 2023-10-30T13:28:34.409 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/23' 2023-10-30T13:28:34.409 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/35' 2023-10-30T13:28:34.410 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/26' 2023-10-30T13:28:34.410 INFO:tasks.workunit.client.0.smithi046.stdout:removed './229/37' 2023-10-30T13:28:34.410 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './229' 2023-10-30T13:28:34.410 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/21' 2023-10-30T13:28:34.410 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/27' 2023-10-30T13:28:34.411 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/31' 2023-10-30T13:28:34.411 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/36' 2023-10-30T13:28:34.411 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/20' 2023-10-30T13:28:34.411 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/39' 2023-10-30T13:28:34.411 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/34' 2023-10-30T13:28:34.411 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/38' 2023-10-30T13:28:34.412 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/28' 2023-10-30T13:28:34.412 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/24' 2023-10-30T13:28:34.412 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/29' 2023-10-30T13:28:34.412 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/32' 2023-10-30T13:28:34.412 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/22' 2023-10-30T13:28:34.412 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/25' 2023-10-30T13:28:34.413 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/30' 2023-10-30T13:28:34.413 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/33' 2023-10-30T13:28:34.413 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/23' 2023-10-30T13:28:34.413 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/35' 2023-10-30T13:28:34.413 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/26' 2023-10-30T13:28:34.414 INFO:tasks.workunit.client.0.smithi046.stdout:removed './230/37' 2023-10-30T13:28:34.414 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './230' 2023-10-30T13:28:34.414 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/21' 2023-10-30T13:28:34.414 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/27' 2023-10-30T13:28:34.414 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/31' 2023-10-30T13:28:34.414 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/36' 2023-10-30T13:28:34.415 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/20' 2023-10-30T13:28:34.415 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/39' 2023-10-30T13:28:34.415 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/34' 2023-10-30T13:28:34.415 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/38' 2023-10-30T13:28:34.415 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/28' 2023-10-30T13:28:34.416 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/24' 2023-10-30T13:28:34.416 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/29' 2023-10-30T13:28:34.416 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/32' 2023-10-30T13:28:34.416 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/22' 2023-10-30T13:28:34.416 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/25' 2023-10-30T13:28:34.417 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/30' 2023-10-30T13:28:34.417 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/33' 2023-10-30T13:28:34.417 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/23' 2023-10-30T13:28:34.417 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/35' 2023-10-30T13:28:34.417 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/26' 2023-10-30T13:28:34.417 INFO:tasks.workunit.client.0.smithi046.stdout:removed './231/37' 2023-10-30T13:28:34.418 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './231' 2023-10-30T13:28:34.418 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/21' 2023-10-30T13:28:34.418 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/27' 2023-10-30T13:28:34.418 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/31' 2023-10-30T13:28:34.418 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/36' 2023-10-30T13:28:34.419 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/20' 2023-10-30T13:28:34.419 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/39' 2023-10-30T13:28:34.419 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/34' 2023-10-30T13:28:34.419 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/38' 2023-10-30T13:28:34.419 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/28' 2023-10-30T13:28:34.420 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/24' 2023-10-30T13:28:34.420 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/29' 2023-10-30T13:28:34.420 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/32' 2023-10-30T13:28:34.420 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/22' 2023-10-30T13:28:34.420 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/25' 2023-10-30T13:28:34.420 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/30' 2023-10-30T13:28:34.421 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/33' 2023-10-30T13:28:34.421 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/23' 2023-10-30T13:28:34.421 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/35' 2023-10-30T13:28:34.421 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/26' 2023-10-30T13:28:34.421 INFO:tasks.workunit.client.0.smithi046.stdout:removed './232/37' 2023-10-30T13:28:34.421 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './232' 2023-10-30T13:28:34.422 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/21' 2023-10-30T13:28:34.422 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/27' 2023-10-30T13:28:34.422 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/31' 2023-10-30T13:28:34.422 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/36' 2023-10-30T13:28:34.422 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/20' 2023-10-30T13:28:34.423 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/39' 2023-10-30T13:28:34.423 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/34' 2023-10-30T13:28:34.423 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/38' 2023-10-30T13:28:34.423 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/28' 2023-10-30T13:28:34.423 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/24' 2023-10-30T13:28:34.424 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/29' 2023-10-30T13:28:34.424 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/32' 2023-10-30T13:28:34.424 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/22' 2023-10-30T13:28:34.424 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/25' 2023-10-30T13:28:34.424 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/30' 2023-10-30T13:28:34.424 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/33' 2023-10-30T13:28:34.425 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/23' 2023-10-30T13:28:34.425 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/35' 2023-10-30T13:28:34.425 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/26' 2023-10-30T13:28:34.425 INFO:tasks.workunit.client.0.smithi046.stdout:removed './233/37' 2023-10-30T13:28:34.425 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './233' 2023-10-30T13:28:34.426 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/21' 2023-10-30T13:28:34.426 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/27' 2023-10-30T13:28:34.426 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/31' 2023-10-30T13:28:34.426 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/36' 2023-10-30T13:28:34.427 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/20' 2023-10-30T13:28:34.427 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/39' 2023-10-30T13:28:34.427 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/34' 2023-10-30T13:28:34.427 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/38' 2023-10-30T13:28:34.427 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/28' 2023-10-30T13:28:34.427 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/24' 2023-10-30T13:28:34.428 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/29' 2023-10-30T13:28:34.428 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/32' 2023-10-30T13:28:34.428 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/22' 2023-10-30T13:28:34.428 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/25' 2023-10-30T13:28:34.428 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/30' 2023-10-30T13:28:34.428 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/33' 2023-10-30T13:28:34.429 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/23' 2023-10-30T13:28:34.429 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/35' 2023-10-30T13:28:34.429 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/26' 2023-10-30T13:28:34.429 INFO:tasks.workunit.client.0.smithi046.stdout:removed './234/37' 2023-10-30T13:28:34.429 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './234' 2023-10-30T13:28:34.430 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/21' 2023-10-30T13:28:34.430 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/27' 2023-10-30T13:28:34.430 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/31' 2023-10-30T13:28:34.430 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/36' 2023-10-30T13:28:34.430 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/20' 2023-10-30T13:28:34.431 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/39' 2023-10-30T13:28:34.431 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/34' 2023-10-30T13:28:34.431 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/38' 2023-10-30T13:28:34.431 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/28' 2023-10-30T13:28:34.431 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/24' 2023-10-30T13:28:34.431 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/29' 2023-10-30T13:28:34.432 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/32' 2023-10-30T13:28:34.432 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/22' 2023-10-30T13:28:34.432 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/25' 2023-10-30T13:28:34.432 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/30' 2023-10-30T13:28:34.432 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/33' 2023-10-30T13:28:34.433 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/23' 2023-10-30T13:28:34.433 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/35' 2023-10-30T13:28:34.433 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/26' 2023-10-30T13:28:34.433 INFO:tasks.workunit.client.0.smithi046.stdout:removed './235/37' 2023-10-30T13:28:34.433 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './235' 2023-10-30T13:28:34.433 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/21' 2023-10-30T13:28:34.434 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/27' 2023-10-30T13:28:34.434 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/31' 2023-10-30T13:28:34.434 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/36' 2023-10-30T13:28:34.434 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/20' 2023-10-30T13:28:34.434 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/39' 2023-10-30T13:28:34.435 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/34' 2023-10-30T13:28:34.435 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/38' 2023-10-30T13:28:34.435 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/28' 2023-10-30T13:28:34.435 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/24' 2023-10-30T13:28:34.435 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/29' 2023-10-30T13:28:34.435 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/32' 2023-10-30T13:28:35.475 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/22' 2023-10-30T13:28:35.475 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/25' 2023-10-30T13:28:35.476 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/30' 2023-10-30T13:28:35.476 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/33' 2023-10-30T13:28:35.476 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/23' 2023-10-30T13:28:35.476 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/35' 2023-10-30T13:28:35.476 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/26' 2023-10-30T13:28:35.477 INFO:tasks.workunit.client.0.smithi046.stdout:removed './236/37' 2023-10-30T13:28:35.477 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './236' 2023-10-30T13:28:35.477 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/21' 2023-10-30T13:28:35.477 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/27' 2023-10-30T13:28:35.477 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/31' 2023-10-30T13:28:35.478 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/36' 2023-10-30T13:28:35.478 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/20' 2023-10-30T13:28:35.478 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/39' 2023-10-30T13:28:35.478 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/34' 2023-10-30T13:28:35.478 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/38' 2023-10-30T13:28:35.479 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/28' 2023-10-30T13:28:35.479 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/24' 2023-10-30T13:28:35.479 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/29' 2023-10-30T13:28:35.479 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/32' 2023-10-30T13:28:35.479 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/22' 2023-10-30T13:28:35.479 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/25' 2023-10-30T13:28:35.480 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/30' 2023-10-30T13:28:35.480 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/33' 2023-10-30T13:28:35.480 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/23' 2023-10-30T13:28:35.480 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/35' 2023-10-30T13:28:35.480 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/26' 2023-10-30T13:28:35.481 INFO:tasks.workunit.client.0.smithi046.stdout:removed './237/37' 2023-10-30T13:28:35.481 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './237' 2023-10-30T13:28:35.481 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/21' 2023-10-30T13:28:35.481 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/27' 2023-10-30T13:28:35.481 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/31' 2023-10-30T13:28:35.482 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/36' 2023-10-30T13:28:35.482 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/20' 2023-10-30T13:28:35.482 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/39' 2023-10-30T13:28:35.482 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/34' 2023-10-30T13:28:35.482 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/38' 2023-10-30T13:28:35.483 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/28' 2023-10-30T13:28:35.483 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/24' 2023-10-30T13:28:35.483 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/29' 2023-10-30T13:28:35.483 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/32' 2023-10-30T13:28:35.483 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/22' 2023-10-30T13:28:35.483 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/25' 2023-10-30T13:28:35.484 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/30' 2023-10-30T13:28:35.484 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/33' 2023-10-30T13:28:35.484 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/23' 2023-10-30T13:28:35.484 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/35' 2023-10-30T13:28:35.484 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/26' 2023-10-30T13:28:35.485 INFO:tasks.workunit.client.0.smithi046.stdout:removed './238/37' 2023-10-30T13:28:35.485 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './238' 2023-10-30T13:28:35.485 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/21' 2023-10-30T13:28:35.485 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/27' 2023-10-30T13:28:35.486 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/31' 2023-10-30T13:28:35.486 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/36' 2023-10-30T13:28:35.486 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/20' 2023-10-30T13:28:35.486 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/39' 2023-10-30T13:28:35.486 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/34' 2023-10-30T13:28:35.487 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/38' 2023-10-30T13:28:35.487 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/28' 2023-10-30T13:28:35.487 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/24' 2023-10-30T13:28:35.487 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/29' 2023-10-30T13:28:35.487 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/32' 2023-10-30T13:28:35.488 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/22' 2023-10-30T13:28:35.488 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/25' 2023-10-30T13:28:35.488 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/30' 2023-10-30T13:28:35.488 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/33' 2023-10-30T13:28:35.488 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/23' 2023-10-30T13:28:35.488 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/35' 2023-10-30T13:28:35.489 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/26' 2023-10-30T13:28:35.489 INFO:tasks.workunit.client.0.smithi046.stdout:removed './239/37' 2023-10-30T13:28:35.489 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './239' 2023-10-30T13:28:35.489 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/21' 2023-10-30T13:28:35.489 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/27' 2023-10-30T13:28:35.490 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/31' 2023-10-30T13:28:35.490 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/36' 2023-10-30T13:28:35.490 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/20' 2023-10-30T13:28:35.490 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/39' 2023-10-30T13:28:35.490 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/34' 2023-10-30T13:28:35.491 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/38' 2023-10-30T13:28:35.491 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/28' 2023-10-30T13:28:35.491 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/24' 2023-10-30T13:28:35.491 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/29' 2023-10-30T13:28:35.491 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/32' 2023-10-30T13:28:35.491 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/22' 2023-10-30T13:28:35.492 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/25' 2023-10-30T13:28:35.492 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/30' 2023-10-30T13:28:35.492 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/33' 2023-10-30T13:28:35.492 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/23' 2023-10-30T13:28:35.492 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/35' 2023-10-30T13:28:35.493 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/26' 2023-10-30T13:28:35.493 INFO:tasks.workunit.client.0.smithi046.stdout:removed './240/37' 2023-10-30T13:28:35.493 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './240' 2023-10-30T13:28:35.493 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/21' 2023-10-30T13:28:35.493 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/27' 2023-10-30T13:28:35.493 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/31' 2023-10-30T13:28:35.494 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/36' 2023-10-30T13:28:35.494 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/20' 2023-10-30T13:28:35.494 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/39' 2023-10-30T13:28:35.494 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/34' 2023-10-30T13:28:35.494 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/38' 2023-10-30T13:28:35.495 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/28' 2023-10-30T13:28:35.495 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/24' 2023-10-30T13:28:35.495 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/29' 2023-10-30T13:28:35.495 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/32' 2023-10-30T13:28:35.495 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/22' 2023-10-30T13:28:35.496 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/25' 2023-10-30T13:28:35.496 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/30' 2023-10-30T13:28:35.496 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/33' 2023-10-30T13:28:35.496 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/23' 2023-10-30T13:28:35.496 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/35' 2023-10-30T13:28:35.496 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/26' 2023-10-30T13:28:35.497 INFO:tasks.workunit.client.0.smithi046.stdout:removed './241/37' 2023-10-30T13:28:35.497 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './241' 2023-10-30T13:28:35.497 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/21' 2023-10-30T13:28:35.497 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/27' 2023-10-30T13:28:35.497 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/31' 2023-10-30T13:28:35.498 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/36' 2023-10-30T13:28:35.498 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/20' 2023-10-30T13:28:35.498 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/39' 2023-10-30T13:28:35.498 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/34' 2023-10-30T13:28:35.498 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/38' 2023-10-30T13:28:35.499 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/28' 2023-10-30T13:28:35.499 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/24' 2023-10-30T13:28:35.499 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/29' 2023-10-30T13:28:35.499 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/32' 2023-10-30T13:28:35.499 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/22' 2023-10-30T13:28:35.500 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/25' 2023-10-30T13:28:35.500 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/30' 2023-10-30T13:28:35.500 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/33' 2023-10-30T13:28:35.500 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/23' 2023-10-30T13:28:35.500 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/35' 2023-10-30T13:28:35.500 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/26' 2023-10-30T13:28:35.501 INFO:tasks.workunit.client.0.smithi046.stdout:removed './242/37' 2023-10-30T13:28:35.501 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './242' 2023-10-30T13:28:35.501 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/21' 2023-10-30T13:28:35.501 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/27' 2023-10-30T13:28:35.501 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/31' 2023-10-30T13:28:35.502 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/36' 2023-10-30T13:28:35.502 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/20' 2023-10-30T13:28:35.502 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/39' 2023-10-30T13:28:35.502 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/34' 2023-10-30T13:28:35.502 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/38' 2023-10-30T13:28:35.503 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/28' 2023-10-30T13:28:35.503 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/24' 2023-10-30T13:28:35.503 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/29' 2023-10-30T13:28:35.503 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/32' 2023-10-30T13:28:35.503 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/22' 2023-10-30T13:28:35.503 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/25' 2023-10-30T13:28:35.504 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/30' 2023-10-30T13:28:35.504 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/33' 2023-10-30T13:28:35.504 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/23' 2023-10-30T13:28:35.504 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/35' 2023-10-30T13:28:35.504 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/26' 2023-10-30T13:28:35.505 INFO:tasks.workunit.client.0.smithi046.stdout:removed './243/37' 2023-10-30T13:28:35.505 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './243' 2023-10-30T13:28:35.505 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/21' 2023-10-30T13:28:35.505 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/27' 2023-10-30T13:28:35.505 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/31' 2023-10-30T13:28:35.506 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/36' 2023-10-30T13:28:35.506 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/20' 2023-10-30T13:28:35.506 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/39' 2023-10-30T13:28:35.506 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/34' 2023-10-30T13:28:35.506 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/38' 2023-10-30T13:28:35.507 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/28' 2023-10-30T13:28:35.507 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/24' 2023-10-30T13:28:35.507 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/29' 2023-10-30T13:28:35.507 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/32' 2023-10-30T13:28:35.507 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/22' 2023-10-30T13:28:35.507 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/25' 2023-10-30T13:28:35.508 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/30' 2023-10-30T13:28:35.508 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/33' 2023-10-30T13:28:35.508 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/23' 2023-10-30T13:28:35.508 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/35' 2023-10-30T13:28:35.508 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/26' 2023-10-30T13:28:35.509 INFO:tasks.workunit.client.0.smithi046.stdout:removed './244/37' 2023-10-30T13:28:35.509 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './244' 2023-10-30T13:28:35.509 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/21' 2023-10-30T13:28:35.509 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/27' 2023-10-30T13:28:35.509 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/31' 2023-10-30T13:28:35.510 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/36' 2023-10-30T13:28:35.510 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/20' 2023-10-30T13:28:35.510 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/39' 2023-10-30T13:28:35.510 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/34' 2023-10-30T13:28:35.510 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/38' 2023-10-30T13:28:35.511 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/28' 2023-10-30T13:28:35.511 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/24' 2023-10-30T13:28:35.511 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/29' 2023-10-30T13:28:35.511 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/32' 2023-10-30T13:28:35.511 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/22' 2023-10-30T13:28:35.512 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/25' 2023-10-30T13:28:35.512 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/30' 2023-10-30T13:28:35.512 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/33' 2023-10-30T13:28:35.512 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/23' 2023-10-30T13:28:35.512 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/35' 2023-10-30T13:28:35.512 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/26' 2023-10-30T13:28:35.513 INFO:tasks.workunit.client.0.smithi046.stdout:removed './245/37' 2023-10-30T13:28:35.513 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './245' 2023-10-30T13:28:35.513 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/21' 2023-10-30T13:28:35.513 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/27' 2023-10-30T13:28:35.513 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/31' 2023-10-30T13:28:35.513 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/36' 2023-10-30T13:28:35.514 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/20' 2023-10-30T13:28:35.514 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/39' 2023-10-30T13:28:35.514 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/34' 2023-10-30T13:28:35.514 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/38' 2023-10-30T13:28:35.515 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/28' 2023-10-30T13:28:35.515 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/24' 2023-10-30T13:28:35.515 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/29' 2023-10-30T13:28:35.515 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/32' 2023-10-30T13:28:35.515 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/22' 2023-10-30T13:28:37.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/25' 2023-10-30T13:28:37.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/30' 2023-10-30T13:28:37.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/33' 2023-10-30T13:28:37.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/23' 2023-10-30T13:28:37.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/35' 2023-10-30T13:28:37.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/26' 2023-10-30T13:28:37.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './246/37' 2023-10-30T13:28:37.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './246' 2023-10-30T13:28:37.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/21' 2023-10-30T13:28:37.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/27' 2023-10-30T13:28:37.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/31' 2023-10-30T13:28:37.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/36' 2023-10-30T13:28:37.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/20' 2023-10-30T13:28:37.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/39' 2023-10-30T13:28:37.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/34' 2023-10-30T13:28:37.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/38' 2023-10-30T13:28:37.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/28' 2023-10-30T13:28:37.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/24' 2023-10-30T13:28:37.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/29' 2023-10-30T13:28:37.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/32' 2023-10-30T13:28:37.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/22' 2023-10-30T13:28:37.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/25' 2023-10-30T13:28:37.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/30' 2023-10-30T13:28:37.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/33' 2023-10-30T13:28:37.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/23' 2023-10-30T13:28:37.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/35' 2023-10-30T13:28:37.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/26' 2023-10-30T13:28:37.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './247/37' 2023-10-30T13:28:37.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './247' 2023-10-30T13:28:37.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/21' 2023-10-30T13:28:37.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/27' 2023-10-30T13:28:37.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/31' 2023-10-30T13:28:37.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/36' 2023-10-30T13:28:37.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/20' 2023-10-30T13:28:37.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/39' 2023-10-30T13:28:37.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/34' 2023-10-30T13:28:37.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/38' 2023-10-30T13:28:37.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/28' 2023-10-30T13:28:37.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/24' 2023-10-30T13:28:37.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/29' 2023-10-30T13:28:37.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/32' 2023-10-30T13:28:37.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/22' 2023-10-30T13:28:37.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/25' 2023-10-30T13:28:37.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/30' 2023-10-30T13:28:37.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/33' 2023-10-30T13:28:37.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/23' 2023-10-30T13:28:37.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/35' 2023-10-30T13:28:37.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/26' 2023-10-30T13:28:37.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './248/37' 2023-10-30T13:28:37.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './248' 2023-10-30T13:28:37.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/21' 2023-10-30T13:28:37.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/27' 2023-10-30T13:28:37.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/31' 2023-10-30T13:28:37.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/36' 2023-10-30T13:28:37.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/20' 2023-10-30T13:28:37.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/39' 2023-10-30T13:28:37.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/34' 2023-10-30T13:28:37.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/38' 2023-10-30T13:28:37.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/28' 2023-10-30T13:28:37.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/24' 2023-10-30T13:28:37.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/29' 2023-10-30T13:28:37.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/32' 2023-10-30T13:28:37.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/22' 2023-10-30T13:28:37.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/25' 2023-10-30T13:28:37.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/30' 2023-10-30T13:28:37.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/33' 2023-10-30T13:28:37.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/23' 2023-10-30T13:28:37.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/35' 2023-10-30T13:28:37.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/26' 2023-10-30T13:28:37.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './249/37' 2023-10-30T13:28:37.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './249' 2023-10-30T13:28:37.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/21' 2023-10-30T13:28:37.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/27' 2023-10-30T13:28:37.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/31' 2023-10-30T13:28:37.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/36' 2023-10-30T13:28:37.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/20' 2023-10-30T13:28:37.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/39' 2023-10-30T13:28:37.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/34' 2023-10-30T13:28:37.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/38' 2023-10-30T13:28:37.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/28' 2023-10-30T13:28:37.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/24' 2023-10-30T13:28:37.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/29' 2023-10-30T13:28:37.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/32' 2023-10-30T13:28:37.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/22' 2023-10-30T13:28:37.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/25' 2023-10-30T13:28:37.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/30' 2023-10-30T13:28:37.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/33' 2023-10-30T13:28:37.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/23' 2023-10-30T13:28:37.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/35' 2023-10-30T13:28:37.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/26' 2023-10-30T13:28:37.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './250/37' 2023-10-30T13:28:37.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './250' 2023-10-30T13:28:37.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/21' 2023-10-30T13:28:37.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/27' 2023-10-30T13:28:37.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/31' 2023-10-30T13:28:37.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/36' 2023-10-30T13:28:37.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/20' 2023-10-30T13:28:37.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/39' 2023-10-30T13:28:37.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/34' 2023-10-30T13:28:37.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/38' 2023-10-30T13:28:37.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/28' 2023-10-30T13:28:37.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/24' 2023-10-30T13:28:37.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/29' 2023-10-30T13:28:37.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/32' 2023-10-30T13:28:37.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/22' 2023-10-30T13:28:37.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/25' 2023-10-30T13:28:37.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/30' 2023-10-30T13:28:37.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/33' 2023-10-30T13:28:37.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/23' 2023-10-30T13:28:37.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/35' 2023-10-30T13:28:37.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/26' 2023-10-30T13:28:37.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './251/37' 2023-10-30T13:28:37.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './251' 2023-10-30T13:28:37.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/21' 2023-10-30T13:28:37.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/27' 2023-10-30T13:28:37.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/31' 2023-10-30T13:28:37.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/36' 2023-10-30T13:28:37.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/20' 2023-10-30T13:28:37.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/39' 2023-10-30T13:28:37.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/34' 2023-10-30T13:28:37.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/38' 2023-10-30T13:28:37.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/28' 2023-10-30T13:28:37.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/24' 2023-10-30T13:28:37.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/29' 2023-10-30T13:28:37.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/32' 2023-10-30T13:28:37.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/22' 2023-10-30T13:28:37.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/25' 2023-10-30T13:28:37.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/30' 2023-10-30T13:28:37.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/33' 2023-10-30T13:28:37.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/23' 2023-10-30T13:28:37.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/35' 2023-10-30T13:28:37.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/26' 2023-10-30T13:28:37.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './252/37' 2023-10-30T13:28:37.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './252' 2023-10-30T13:28:37.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/21' 2023-10-30T13:28:37.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/27' 2023-10-30T13:28:37.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/31' 2023-10-30T13:28:37.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/36' 2023-10-30T13:28:37.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/20' 2023-10-30T13:28:37.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/39' 2023-10-30T13:28:37.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/34' 2023-10-30T13:28:37.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/38' 2023-10-30T13:28:37.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/28' 2023-10-30T13:28:37.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/24' 2023-10-30T13:28:37.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/29' 2023-10-30T13:28:37.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/32' 2023-10-30T13:28:37.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/22' 2023-10-30T13:28:37.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/25' 2023-10-30T13:28:37.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/30' 2023-10-30T13:28:37.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/33' 2023-10-30T13:28:37.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/23' 2023-10-30T13:28:37.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/35' 2023-10-30T13:28:37.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/26' 2023-10-30T13:28:37.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './253/37' 2023-10-30T13:28:37.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './253' 2023-10-30T13:28:37.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/21' 2023-10-30T13:28:37.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/27' 2023-10-30T13:28:37.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/31' 2023-10-30T13:28:37.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/36' 2023-10-30T13:28:37.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/20' 2023-10-30T13:28:37.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/39' 2023-10-30T13:28:37.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/34' 2023-10-30T13:28:37.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/38' 2023-10-30T13:28:37.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/28' 2023-10-30T13:28:37.905 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/24' 2023-10-30T13:28:37.905 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/29' 2023-10-30T13:28:37.905 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/32' 2023-10-30T13:28:37.905 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/22' 2023-10-30T13:28:37.905 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/25' 2023-10-30T13:28:37.905 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/30' 2023-10-30T13:28:37.906 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/33' 2023-10-30T13:28:37.906 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/23' 2023-10-30T13:28:37.906 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/35' 2023-10-30T13:28:37.906 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/26' 2023-10-30T13:28:37.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './254/37' 2023-10-30T13:28:37.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './254' 2023-10-30T13:28:37.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/21' 2023-10-30T13:28:37.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/27' 2023-10-30T13:28:37.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/31' 2023-10-30T13:28:37.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/36' 2023-10-30T13:28:37.908 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/20' 2023-10-30T13:28:37.908 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/39' 2023-10-30T13:28:37.908 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/34' 2023-10-30T13:28:37.908 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/38' 2023-10-30T13:28:37.908 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/28' 2023-10-30T13:28:37.909 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/24' 2023-10-30T13:28:37.909 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/29' 2023-10-30T13:28:37.909 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/32' 2023-10-30T13:28:37.909 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/22' 2023-10-30T13:28:37.909 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/25' 2023-10-30T13:28:37.910 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/30' 2023-10-30T13:28:37.910 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/33' 2023-10-30T13:28:37.910 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/23' 2023-10-30T13:28:37.910 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/35' 2023-10-30T13:28:37.910 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/26' 2023-10-30T13:28:37.911 INFO:tasks.workunit.client.0.smithi046.stdout:removed './255/37' 2023-10-30T13:28:37.911 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './255' 2023-10-30T13:28:37.911 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/21' 2023-10-30T13:28:37.911 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/27' 2023-10-30T13:28:37.911 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/31' 2023-10-30T13:28:37.912 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/36' 2023-10-30T13:28:37.912 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/20' 2023-10-30T13:28:37.912 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/39' 2023-10-30T13:28:37.912 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/34' 2023-10-30T13:28:37.912 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/38' 2023-10-30T13:28:37.912 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/28' 2023-10-30T13:28:37.913 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/24' 2023-10-30T13:28:37.913 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/29' 2023-10-30T13:28:37.913 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/32' 2023-10-30T13:28:37.913 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/22' 2023-10-30T13:28:37.913 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/25' 2023-10-30T13:28:37.914 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/30' 2023-10-30T13:28:39.435 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/33' 2023-10-30T13:28:39.436 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/23' 2023-10-30T13:28:39.436 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/35' 2023-10-30T13:28:39.436 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/26' 2023-10-30T13:28:39.436 INFO:tasks.workunit.client.0.smithi046.stdout:removed './256/37' 2023-10-30T13:28:39.437 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './256' 2023-10-30T13:28:39.437 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/21' 2023-10-30T13:28:39.437 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/27' 2023-10-30T13:28:39.437 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/31' 2023-10-30T13:28:39.437 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/36' 2023-10-30T13:28:39.438 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/20' 2023-10-30T13:28:39.438 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/39' 2023-10-30T13:28:39.438 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/34' 2023-10-30T13:28:39.438 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/38' 2023-10-30T13:28:39.438 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/28' 2023-10-30T13:28:39.439 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/24' 2023-10-30T13:28:39.439 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/29' 2023-10-30T13:28:39.439 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/32' 2023-10-30T13:28:39.439 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/22' 2023-10-30T13:28:39.439 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/25' 2023-10-30T13:28:39.440 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/30' 2023-10-30T13:28:39.440 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/33' 2023-10-30T13:28:39.440 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/23' 2023-10-30T13:28:39.440 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/35' 2023-10-30T13:28:39.440 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/26' 2023-10-30T13:28:39.441 INFO:tasks.workunit.client.0.smithi046.stdout:removed './257/37' 2023-10-30T13:28:39.441 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './257' 2023-10-30T13:28:39.441 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/21' 2023-10-30T13:28:39.441 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/27' 2023-10-30T13:28:39.441 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/31' 2023-10-30T13:28:39.442 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/36' 2023-10-30T13:28:39.442 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/20' 2023-10-30T13:28:39.442 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/39' 2023-10-30T13:28:39.442 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/34' 2023-10-30T13:28:39.442 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/38' 2023-10-30T13:28:39.443 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/28' 2023-10-30T13:28:39.443 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/24' 2023-10-30T13:28:39.443 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/29' 2023-10-30T13:28:39.443 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/32' 2023-10-30T13:28:39.443 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/22' 2023-10-30T13:28:39.444 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/25' 2023-10-30T13:28:39.444 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/30' 2023-10-30T13:28:39.444 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/33' 2023-10-30T13:28:39.444 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/23' 2023-10-30T13:28:39.445 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/35' 2023-10-30T13:28:39.445 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/26' 2023-10-30T13:28:39.445 INFO:tasks.workunit.client.0.smithi046.stdout:removed './258/37' 2023-10-30T13:28:39.445 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './258' 2023-10-30T13:28:39.445 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/21' 2023-10-30T13:28:39.446 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/27' 2023-10-30T13:28:39.446 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/31' 2023-10-30T13:28:39.446 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/36' 2023-10-30T13:28:39.446 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/20' 2023-10-30T13:28:39.447 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/39' 2023-10-30T13:28:39.447 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/34' 2023-10-30T13:28:39.447 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/38' 2023-10-30T13:28:39.447 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/28' 2023-10-30T13:28:39.447 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/24' 2023-10-30T13:28:39.447 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/29' 2023-10-30T13:28:39.448 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/32' 2023-10-30T13:28:39.448 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/22' 2023-10-30T13:28:39.448 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/25' 2023-10-30T13:28:39.448 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/30' 2023-10-30T13:28:39.448 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/33' 2023-10-30T13:28:39.449 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/23' 2023-10-30T13:28:39.449 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/35' 2023-10-30T13:28:39.449 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/26' 2023-10-30T13:28:39.449 INFO:tasks.workunit.client.0.smithi046.stdout:removed './259/37' 2023-10-30T13:28:39.449 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './259' 2023-10-30T13:28:39.450 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/21' 2023-10-30T13:28:39.450 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/27' 2023-10-30T13:28:39.450 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/31' 2023-10-30T13:28:39.450 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/36' 2023-10-30T13:28:39.450 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/20' 2023-10-30T13:28:39.450 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/39' 2023-10-30T13:28:39.451 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/34' 2023-10-30T13:28:39.451 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/38' 2023-10-30T13:28:39.451 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/28' 2023-10-30T13:28:39.451 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/24' 2023-10-30T13:28:39.451 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/29' 2023-10-30T13:28:39.451 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/32' 2023-10-30T13:28:39.452 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/22' 2023-10-30T13:28:39.452 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/25' 2023-10-30T13:28:39.452 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/30' 2023-10-30T13:28:39.452 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/33' 2023-10-30T13:28:39.452 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/23' 2023-10-30T13:28:39.453 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/35' 2023-10-30T13:28:39.453 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/26' 2023-10-30T13:28:39.453 INFO:tasks.workunit.client.0.smithi046.stdout:removed './260/37' 2023-10-30T13:28:39.453 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './260' 2023-10-30T13:28:39.453 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/21' 2023-10-30T13:28:39.454 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/27' 2023-10-30T13:28:39.454 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/31' 2023-10-30T13:28:39.454 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/36' 2023-10-30T13:28:39.454 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/20' 2023-10-30T13:28:39.454 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/39' 2023-10-30T13:28:39.454 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/34' 2023-10-30T13:28:39.455 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/38' 2023-10-30T13:28:39.455 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/28' 2023-10-30T13:28:39.455 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/24' 2023-10-30T13:28:39.455 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/29' 2023-10-30T13:28:39.455 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/32' 2023-10-30T13:28:39.455 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/22' 2023-10-30T13:28:39.456 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/25' 2023-10-30T13:28:39.456 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/30' 2023-10-30T13:28:39.456 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/33' 2023-10-30T13:28:39.456 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/23' 2023-10-30T13:28:39.456 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/35' 2023-10-30T13:28:39.457 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/26' 2023-10-30T13:28:39.457 INFO:tasks.workunit.client.0.smithi046.stdout:removed './261/37' 2023-10-30T13:28:39.457 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './261' 2023-10-30T13:28:39.457 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/21' 2023-10-30T13:28:39.457 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/27' 2023-10-30T13:28:39.458 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/31' 2023-10-30T13:28:39.458 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/36' 2023-10-30T13:28:39.458 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/20' 2023-10-30T13:28:39.458 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/39' 2023-10-30T13:28:39.458 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/34' 2023-10-30T13:28:39.459 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/38' 2023-10-30T13:28:39.459 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/28' 2023-10-30T13:28:39.459 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/24' 2023-10-30T13:28:39.459 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/29' 2023-10-30T13:28:39.459 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/32' 2023-10-30T13:28:39.460 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/22' 2023-10-30T13:28:39.460 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/25' 2023-10-30T13:28:39.460 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/30' 2023-10-30T13:28:39.460 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/33' 2023-10-30T13:28:39.460 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/23' 2023-10-30T13:28:39.460 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/35' 2023-10-30T13:28:39.461 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/26' 2023-10-30T13:28:39.461 INFO:tasks.workunit.client.0.smithi046.stdout:removed './262/37' 2023-10-30T13:28:39.461 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './262' 2023-10-30T13:28:39.461 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/21' 2023-10-30T13:28:39.462 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/27' 2023-10-30T13:28:39.462 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/31' 2023-10-30T13:28:39.462 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/36' 2023-10-30T13:28:39.462 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/20' 2023-10-30T13:28:39.462 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/39' 2023-10-30T13:28:39.463 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/34' 2023-10-30T13:28:39.463 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/38' 2023-10-30T13:28:39.463 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/28' 2023-10-30T13:28:39.463 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/24' 2023-10-30T13:28:39.463 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/29' 2023-10-30T13:28:39.463 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/32' 2023-10-30T13:28:39.464 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/22' 2023-10-30T13:28:39.464 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/25' 2023-10-30T13:28:39.464 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/30' 2023-10-30T13:28:39.464 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/33' 2023-10-30T13:28:39.464 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/23' 2023-10-30T13:28:39.465 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/35' 2023-10-30T13:28:39.465 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/26' 2023-10-30T13:28:39.465 INFO:tasks.workunit.client.0.smithi046.stdout:removed './263/37' 2023-10-30T13:28:39.465 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './263' 2023-10-30T13:28:39.465 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/21' 2023-10-30T13:28:39.466 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/27' 2023-10-30T13:28:39.466 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/31' 2023-10-30T13:28:39.466 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/36' 2023-10-30T13:28:39.466 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/20' 2023-10-30T13:28:39.466 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/39' 2023-10-30T13:28:39.467 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/34' 2023-10-30T13:28:39.467 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/38' 2023-10-30T13:28:39.467 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/28' 2023-10-30T13:28:39.467 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/24' 2023-10-30T13:28:39.467 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/29' 2023-10-30T13:28:39.467 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/32' 2023-10-30T13:28:39.468 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/22' 2023-10-30T13:28:39.468 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/25' 2023-10-30T13:28:39.468 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/30' 2023-10-30T13:28:39.468 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/33' 2023-10-30T13:28:39.468 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/23' 2023-10-30T13:28:39.469 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/35' 2023-10-30T13:28:39.469 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/26' 2023-10-30T13:28:39.469 INFO:tasks.workunit.client.0.smithi046.stdout:removed './264/37' 2023-10-30T13:28:39.469 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './264' 2023-10-30T13:28:39.469 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/21' 2023-10-30T13:28:39.470 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/27' 2023-10-30T13:28:39.470 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/31' 2023-10-30T13:28:39.470 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/36' 2023-10-30T13:28:39.470 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/20' 2023-10-30T13:28:39.470 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/39' 2023-10-30T13:28:39.471 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/34' 2023-10-30T13:28:39.471 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/38' 2023-10-30T13:28:39.471 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/28' 2023-10-30T13:28:39.471 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/24' 2023-10-30T13:28:39.471 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/29' 2023-10-30T13:28:39.471 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/32' 2023-10-30T13:28:39.472 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/22' 2023-10-30T13:28:39.472 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/25' 2023-10-30T13:28:39.472 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/30' 2023-10-30T13:28:39.472 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/33' 2023-10-30T13:28:39.473 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/23' 2023-10-30T13:28:39.473 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/35' 2023-10-30T13:28:39.473 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/26' 2023-10-30T13:28:39.473 INFO:tasks.workunit.client.0.smithi046.stdout:removed './265/37' 2023-10-30T13:28:39.473 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './265' 2023-10-30T13:28:39.473 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/21' 2023-10-30T13:28:39.474 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/27' 2023-10-30T13:28:39.474 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/31' 2023-10-30T13:28:39.474 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/36' 2023-10-30T13:28:39.474 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/20' 2023-10-30T13:28:39.474 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/39' 2023-10-30T13:28:39.475 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/34' 2023-10-30T13:28:39.475 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/38' 2023-10-30T13:28:39.475 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/28' 2023-10-30T13:28:39.475 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/24' 2023-10-30T13:28:39.475 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/29' 2023-10-30T13:28:39.476 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/32' 2023-10-30T13:28:39.476 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/22' 2023-10-30T13:28:39.476 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/25' 2023-10-30T13:28:39.476 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/30' 2023-10-30T13:28:39.476 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/33' 2023-10-30T13:28:39.476 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/23' 2023-10-30T13:28:40.782 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/35' 2023-10-30T13:28:40.782 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/26' 2023-10-30T13:28:40.783 INFO:tasks.workunit.client.0.smithi046.stdout:removed './266/37' 2023-10-30T13:28:40.783 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './266' 2023-10-30T13:28:40.783 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/21' 2023-10-30T13:28:40.783 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/27' 2023-10-30T13:28:40.783 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/31' 2023-10-30T13:28:40.784 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/36' 2023-10-30T13:28:40.784 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/20' 2023-10-30T13:28:40.784 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/39' 2023-10-30T13:28:40.784 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/34' 2023-10-30T13:28:40.784 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/38' 2023-10-30T13:28:40.785 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/28' 2023-10-30T13:28:40.785 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/24' 2023-10-30T13:28:40.785 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/29' 2023-10-30T13:28:40.785 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/32' 2023-10-30T13:28:40.785 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/22' 2023-10-30T13:28:40.786 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/25' 2023-10-30T13:28:40.786 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/30' 2023-10-30T13:28:40.786 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/33' 2023-10-30T13:28:40.786 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/23' 2023-10-30T13:28:40.786 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/35' 2023-10-30T13:28:40.786 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/26' 2023-10-30T13:28:40.787 INFO:tasks.workunit.client.0.smithi046.stdout:removed './267/37' 2023-10-30T13:28:40.787 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './267' 2023-10-30T13:28:40.787 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/21' 2023-10-30T13:28:40.787 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/27' 2023-10-30T13:28:40.787 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/31' 2023-10-30T13:28:40.788 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/36' 2023-10-30T13:28:40.788 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/20' 2023-10-30T13:28:40.788 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/39' 2023-10-30T13:28:40.788 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/34' 2023-10-30T13:28:40.788 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/38' 2023-10-30T13:28:40.788 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/28' 2023-10-30T13:28:40.789 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/24' 2023-10-30T13:28:40.789 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/29' 2023-10-30T13:28:40.789 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/32' 2023-10-30T13:28:40.789 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/22' 2023-10-30T13:28:40.789 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/25' 2023-10-30T13:28:40.790 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/30' 2023-10-30T13:28:40.790 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/33' 2023-10-30T13:28:40.790 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/23' 2023-10-30T13:28:40.790 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/35' 2023-10-30T13:28:40.790 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/26' 2023-10-30T13:28:40.790 INFO:tasks.workunit.client.0.smithi046.stdout:removed './268/37' 2023-10-30T13:28:40.791 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './268' 2023-10-30T13:28:40.791 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/21' 2023-10-30T13:28:40.791 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/27' 2023-10-30T13:28:40.791 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/31' 2023-10-30T13:28:40.791 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/36' 2023-10-30T13:28:40.792 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/20' 2023-10-30T13:28:40.792 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/39' 2023-10-30T13:28:40.792 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/34' 2023-10-30T13:28:40.792 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/38' 2023-10-30T13:28:40.792 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/28' 2023-10-30T13:28:40.792 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/24' 2023-10-30T13:28:40.793 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/29' 2023-10-30T13:28:40.793 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/32' 2023-10-30T13:28:40.793 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/22' 2023-10-30T13:28:40.793 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/25' 2023-10-30T13:28:40.793 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/30' 2023-10-30T13:28:40.794 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/33' 2023-10-30T13:28:40.794 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/23' 2023-10-30T13:28:40.794 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/35' 2023-10-30T13:28:40.794 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/26' 2023-10-30T13:28:40.794 INFO:tasks.workunit.client.0.smithi046.stdout:removed './269/37' 2023-10-30T13:28:40.794 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './269' 2023-10-30T13:28:40.795 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/21' 2023-10-30T13:28:40.795 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/27' 2023-10-30T13:28:40.795 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/31' 2023-10-30T13:28:40.795 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/36' 2023-10-30T13:28:40.795 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/20' 2023-10-30T13:28:40.796 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/39' 2023-10-30T13:28:40.796 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/34' 2023-10-30T13:28:40.796 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/38' 2023-10-30T13:28:40.796 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/28' 2023-10-30T13:28:40.796 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/24' 2023-10-30T13:28:40.796 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/29' 2023-10-30T13:28:40.797 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/32' 2023-10-30T13:28:40.797 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/22' 2023-10-30T13:28:40.797 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/25' 2023-10-30T13:28:40.797 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/30' 2023-10-30T13:28:40.797 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/33' 2023-10-30T13:28:40.798 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/23' 2023-10-30T13:28:40.798 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/35' 2023-10-30T13:28:40.798 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/26' 2023-10-30T13:28:40.798 INFO:tasks.workunit.client.0.smithi046.stdout:removed './270/37' 2023-10-30T13:28:40.798 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './270' 2023-10-30T13:28:40.798 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/21' 2023-10-30T13:28:40.799 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/27' 2023-10-30T13:28:40.799 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/31' 2023-10-30T13:28:40.799 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/36' 2023-10-30T13:28:40.799 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/20' 2023-10-30T13:28:40.799 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/39' 2023-10-30T13:28:40.800 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/34' 2023-10-30T13:28:40.800 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/38' 2023-10-30T13:28:40.800 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/28' 2023-10-30T13:28:40.800 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/24' 2023-10-30T13:28:40.800 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/29' 2023-10-30T13:28:40.800 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/32' 2023-10-30T13:28:40.801 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/22' 2023-10-30T13:28:40.801 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/25' 2023-10-30T13:28:40.801 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/30' 2023-10-30T13:28:40.801 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/33' 2023-10-30T13:28:40.801 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/23' 2023-10-30T13:28:40.802 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/35' 2023-10-30T13:28:40.802 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/26' 2023-10-30T13:28:40.802 INFO:tasks.workunit.client.0.smithi046.stdout:removed './271/37' 2023-10-30T13:28:40.802 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './271' 2023-10-30T13:28:40.802 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/21' 2023-10-30T13:28:40.802 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/27' 2023-10-30T13:28:40.803 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/31' 2023-10-30T13:28:40.803 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/36' 2023-10-30T13:28:40.803 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/20' 2023-10-30T13:28:40.803 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/39' 2023-10-30T13:28:40.803 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/34' 2023-10-30T13:28:40.804 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/38' 2023-10-30T13:28:40.804 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/28' 2023-10-30T13:28:40.804 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/24' 2023-10-30T13:28:40.804 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/29' 2023-10-30T13:28:40.804 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/32' 2023-10-30T13:28:40.804 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/22' 2023-10-30T13:28:40.805 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/25' 2023-10-30T13:28:40.805 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/30' 2023-10-30T13:28:40.805 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/33' 2023-10-30T13:28:40.805 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/23' 2023-10-30T13:28:40.805 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/35' 2023-10-30T13:28:40.805 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/26' 2023-10-30T13:28:40.806 INFO:tasks.workunit.client.0.smithi046.stdout:removed './272/37' 2023-10-30T13:28:40.806 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './272' 2023-10-30T13:28:40.806 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/21' 2023-10-30T13:28:40.806 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/27' 2023-10-30T13:28:40.806 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/31' 2023-10-30T13:28:40.807 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/36' 2023-10-30T13:28:40.807 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/20' 2023-10-30T13:28:40.807 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/39' 2023-10-30T13:28:40.807 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/34' 2023-10-30T13:28:40.807 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/38' 2023-10-30T13:28:40.807 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/28' 2023-10-30T13:28:40.808 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/24' 2023-10-30T13:28:40.808 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/29' 2023-10-30T13:28:40.808 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/32' 2023-10-30T13:28:40.808 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/22' 2023-10-30T13:28:40.808 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/25' 2023-10-30T13:28:40.809 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/30' 2023-10-30T13:28:40.809 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/33' 2023-10-30T13:28:40.809 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/23' 2023-10-30T13:28:40.809 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/35' 2023-10-30T13:28:40.809 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/26' 2023-10-30T13:28:40.809 INFO:tasks.workunit.client.0.smithi046.stdout:removed './273/37' 2023-10-30T13:28:40.810 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './273' 2023-10-30T13:28:40.810 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/21' 2023-10-30T13:28:40.810 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/27' 2023-10-30T13:28:40.810 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/31' 2023-10-30T13:28:40.810 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/36' 2023-10-30T13:28:40.811 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/20' 2023-10-30T13:28:40.811 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/39' 2023-10-30T13:28:40.811 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/34' 2023-10-30T13:28:40.811 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/38' 2023-10-30T13:28:40.811 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/28' 2023-10-30T13:28:40.811 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/24' 2023-10-30T13:28:40.812 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/29' 2023-10-30T13:28:40.812 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/32' 2023-10-30T13:28:40.812 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/22' 2023-10-30T13:28:40.812 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/25' 2023-10-30T13:28:40.812 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/30' 2023-10-30T13:28:40.813 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/33' 2023-10-30T13:28:40.813 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/23' 2023-10-30T13:28:40.813 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/35' 2023-10-30T13:28:40.813 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/26' 2023-10-30T13:28:40.813 INFO:tasks.workunit.client.0.smithi046.stdout:removed './274/37' 2023-10-30T13:28:40.813 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './274' 2023-10-30T13:28:40.814 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/21' 2023-10-30T13:28:40.814 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/27' 2023-10-30T13:28:40.814 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/31' 2023-10-30T13:28:40.814 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/36' 2023-10-30T13:28:40.814 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/20' 2023-10-30T13:28:40.815 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/39' 2023-10-30T13:28:40.815 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/34' 2023-10-30T13:28:40.815 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/38' 2023-10-30T13:28:40.815 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/28' 2023-10-30T13:28:40.815 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/24' 2023-10-30T13:28:40.815 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/29' 2023-10-30T13:28:40.816 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/32' 2023-10-30T13:28:40.816 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/22' 2023-10-30T13:28:40.816 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/25' 2023-10-30T13:28:40.816 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/30' 2023-10-30T13:28:40.816 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/33' 2023-10-30T13:28:40.817 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/23' 2023-10-30T13:28:40.817 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/35' 2023-10-30T13:28:40.817 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/26' 2023-10-30T13:28:40.817 INFO:tasks.workunit.client.0.smithi046.stdout:removed './275/37' 2023-10-30T13:28:40.817 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './275' 2023-10-30T13:28:40.817 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/21' 2023-10-30T13:28:40.818 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/27' 2023-10-30T13:28:40.818 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/31' 2023-10-30T13:28:40.818 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/36' 2023-10-30T13:28:40.818 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/20' 2023-10-30T13:28:40.818 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/39' 2023-10-30T13:28:40.819 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/34' 2023-10-30T13:28:40.819 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/38' 2023-10-30T13:28:40.819 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/28' 2023-10-30T13:28:40.819 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/24' 2023-10-30T13:28:40.819 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/29' 2023-10-30T13:28:40.819 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/32' 2023-10-30T13:28:40.820 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/22' 2023-10-30T13:28:40.820 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/25' 2023-10-30T13:28:40.820 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/30' 2023-10-30T13:28:40.820 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/33' 2023-10-30T13:28:40.820 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/23' 2023-10-30T13:28:40.821 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/35' 2023-10-30T13:28:40.821 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/26' 2023-10-30T13:28:42.659 INFO:tasks.workunit.client.0.smithi046.stdout:removed './276/37' 2023-10-30T13:28:42.659 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './276' 2023-10-30T13:28:42.660 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/21' 2023-10-30T13:28:42.660 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/27' 2023-10-30T13:28:42.660 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/31' 2023-10-30T13:28:42.660 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/36' 2023-10-30T13:28:42.660 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/20' 2023-10-30T13:28:42.661 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/39' 2023-10-30T13:28:42.661 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/34' 2023-10-30T13:28:42.661 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/38' 2023-10-30T13:28:42.661 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/28' 2023-10-30T13:28:42.661 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/24' 2023-10-30T13:28:42.662 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/29' 2023-10-30T13:28:42.662 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/32' 2023-10-30T13:28:42.662 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/22' 2023-10-30T13:28:42.662 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/25' 2023-10-30T13:28:42.662 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/30' 2023-10-30T13:28:42.663 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/33' 2023-10-30T13:28:42.663 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/23' 2023-10-30T13:28:42.663 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/35' 2023-10-30T13:28:42.663 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/26' 2023-10-30T13:28:42.663 INFO:tasks.workunit.client.0.smithi046.stdout:removed './277/37' 2023-10-30T13:28:42.663 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './277' 2023-10-30T13:28:42.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/21' 2023-10-30T13:28:42.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/27' 2023-10-30T13:28:42.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/31' 2023-10-30T13:28:42.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/36' 2023-10-30T13:28:42.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/20' 2023-10-30T13:28:42.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/39' 2023-10-30T13:28:42.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/34' 2023-10-30T13:28:42.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/38' 2023-10-30T13:28:42.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/28' 2023-10-30T13:28:42.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/24' 2023-10-30T13:28:42.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/29' 2023-10-30T13:28:42.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/32' 2023-10-30T13:28:42.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/22' 2023-10-30T13:28:42.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/25' 2023-10-30T13:28:42.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/30' 2023-10-30T13:28:42.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/33' 2023-10-30T13:28:42.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/23' 2023-10-30T13:28:42.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/35' 2023-10-30T13:28:42.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/26' 2023-10-30T13:28:42.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './278/37' 2023-10-30T13:28:42.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './278' 2023-10-30T13:28:42.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/21' 2023-10-30T13:28:42.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/27' 2023-10-30T13:28:42.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/31' 2023-10-30T13:28:42.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/36' 2023-10-30T13:28:42.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/20' 2023-10-30T13:28:42.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/39' 2023-10-30T13:28:42.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/34' 2023-10-30T13:28:42.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/38' 2023-10-30T13:28:42.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/28' 2023-10-30T13:28:42.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/24' 2023-10-30T13:28:42.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/29' 2023-10-30T13:28:42.670 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/32' 2023-10-30T13:28:42.670 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/22' 2023-10-30T13:28:42.670 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/25' 2023-10-30T13:28:42.670 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/30' 2023-10-30T13:28:42.670 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/33' 2023-10-30T13:28:42.670 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/23' 2023-10-30T13:28:42.671 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/35' 2023-10-30T13:28:42.671 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/26' 2023-10-30T13:28:42.671 INFO:tasks.workunit.client.0.smithi046.stdout:removed './279/37' 2023-10-30T13:28:42.671 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './279' 2023-10-30T13:28:42.671 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/21' 2023-10-30T13:28:42.672 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/27' 2023-10-30T13:28:42.672 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/31' 2023-10-30T13:28:42.672 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/36' 2023-10-30T13:28:42.672 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/20' 2023-10-30T13:28:42.672 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/39' 2023-10-30T13:28:42.672 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/34' 2023-10-30T13:28:42.673 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/38' 2023-10-30T13:28:42.673 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/28' 2023-10-30T13:28:42.673 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/24' 2023-10-30T13:28:42.673 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/29' 2023-10-30T13:28:42.673 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/32' 2023-10-30T13:28:42.674 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/22' 2023-10-30T13:28:42.674 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/25' 2023-10-30T13:28:42.674 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/30' 2023-10-30T13:28:42.674 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/33' 2023-10-30T13:28:42.674 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/23' 2023-10-30T13:28:42.674 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/35' 2023-10-30T13:28:42.675 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/26' 2023-10-30T13:28:42.675 INFO:tasks.workunit.client.0.smithi046.stdout:removed './280/37' 2023-10-30T13:28:42.675 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './280' 2023-10-30T13:28:42.675 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/21' 2023-10-30T13:28:42.675 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/27' 2023-10-30T13:28:42.676 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/31' 2023-10-30T13:28:42.676 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/36' 2023-10-30T13:28:42.676 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/20' 2023-10-30T13:28:42.676 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/39' 2023-10-30T13:28:42.676 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/34' 2023-10-30T13:28:42.676 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/38' 2023-10-30T13:28:42.677 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/28' 2023-10-30T13:28:42.677 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/24' 2023-10-30T13:28:42.677 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/29' 2023-10-30T13:28:42.677 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/32' 2023-10-30T13:28:42.677 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/22' 2023-10-30T13:28:42.678 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/25' 2023-10-30T13:28:42.678 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/30' 2023-10-30T13:28:42.678 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/33' 2023-10-30T13:28:42.678 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/23' 2023-10-30T13:28:42.678 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/35' 2023-10-30T13:28:42.678 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/26' 2023-10-30T13:28:42.679 INFO:tasks.workunit.client.0.smithi046.stdout:removed './281/37' 2023-10-30T13:28:42.679 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './281' 2023-10-30T13:28:42.679 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/21' 2023-10-30T13:28:42.679 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/27' 2023-10-30T13:28:42.679 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/31' 2023-10-30T13:28:42.680 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/36' 2023-10-30T13:28:42.680 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/20' 2023-10-30T13:28:42.680 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/39' 2023-10-30T13:28:42.680 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/34' 2023-10-30T13:28:42.680 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/38' 2023-10-30T13:28:42.680 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/28' 2023-10-30T13:28:42.681 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/24' 2023-10-30T13:28:42.681 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/29' 2023-10-30T13:28:42.681 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/32' 2023-10-30T13:28:42.681 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/22' 2023-10-30T13:28:42.681 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/25' 2023-10-30T13:28:42.682 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/30' 2023-10-30T13:28:42.682 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/33' 2023-10-30T13:28:42.682 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/23' 2023-10-30T13:28:42.682 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/35' 2023-10-30T13:28:42.682 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/26' 2023-10-30T13:28:42.682 INFO:tasks.workunit.client.0.smithi046.stdout:removed './282/37' 2023-10-30T13:28:42.683 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './282' 2023-10-30T13:28:42.683 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/21' 2023-10-30T13:28:42.683 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/27' 2023-10-30T13:28:42.683 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/31' 2023-10-30T13:28:42.683 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/36' 2023-10-30T13:28:42.684 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/20' 2023-10-30T13:28:42.684 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/39' 2023-10-30T13:28:42.684 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/34' 2023-10-30T13:28:42.684 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/38' 2023-10-30T13:28:42.684 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/28' 2023-10-30T13:28:42.684 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/24' 2023-10-30T13:28:42.685 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/29' 2023-10-30T13:28:42.685 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/32' 2023-10-30T13:28:42.685 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/22' 2023-10-30T13:28:42.685 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/25' 2023-10-30T13:28:42.685 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/30' 2023-10-30T13:28:42.686 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/33' 2023-10-30T13:28:42.686 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/23' 2023-10-30T13:28:42.686 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/35' 2023-10-30T13:28:42.686 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/26' 2023-10-30T13:28:42.686 INFO:tasks.workunit.client.0.smithi046.stdout:removed './283/37' 2023-10-30T13:28:42.686 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './283' 2023-10-30T13:28:42.687 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/21' 2023-10-30T13:28:42.687 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/27' 2023-10-30T13:28:42.687 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/31' 2023-10-30T13:28:42.687 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/36' 2023-10-30T13:28:42.687 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/20' 2023-10-30T13:28:42.688 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/39' 2023-10-30T13:28:42.688 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/34' 2023-10-30T13:28:42.688 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/38' 2023-10-30T13:28:42.688 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/28' 2023-10-30T13:28:42.688 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/24' 2023-10-30T13:28:42.688 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/29' 2023-10-30T13:28:42.689 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/32' 2023-10-30T13:28:42.689 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/22' 2023-10-30T13:28:42.689 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/25' 2023-10-30T13:28:42.689 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/30' 2023-10-30T13:28:42.689 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/33' 2023-10-30T13:28:42.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/23' 2023-10-30T13:28:42.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/35' 2023-10-30T13:28:42.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/26' 2023-10-30T13:28:42.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './284/37' 2023-10-30T13:28:42.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './284' 2023-10-30T13:28:42.690 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/21' 2023-10-30T13:28:42.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/27' 2023-10-30T13:28:42.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/31' 2023-10-30T13:28:42.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/36' 2023-10-30T13:28:42.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/20' 2023-10-30T13:28:42.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/39' 2023-10-30T13:28:42.691 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/34' 2023-10-30T13:28:42.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/38' 2023-10-30T13:28:42.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/28' 2023-10-30T13:28:42.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/24' 2023-10-30T13:28:42.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/29' 2023-10-30T13:28:42.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/32' 2023-10-30T13:28:42.692 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/22' 2023-10-30T13:28:42.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/25' 2023-10-30T13:28:42.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/30' 2023-10-30T13:28:42.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/33' 2023-10-30T13:28:42.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/23' 2023-10-30T13:28:42.693 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/35' 2023-10-30T13:28:42.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/26' 2023-10-30T13:28:42.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './285/37' 2023-10-30T13:28:42.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './285' 2023-10-30T13:28:42.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/21' 2023-10-30T13:28:42.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/27' 2023-10-30T13:28:42.694 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/31' 2023-10-30T13:28:42.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/36' 2023-10-30T13:28:42.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/20' 2023-10-30T13:28:42.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/39' 2023-10-30T13:28:42.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/34' 2023-10-30T13:28:42.695 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/38' 2023-10-30T13:28:42.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/28' 2023-10-30T13:28:42.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/24' 2023-10-30T13:28:42.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/29' 2023-10-30T13:28:42.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/32' 2023-10-30T13:28:42.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/22' 2023-10-30T13:28:42.696 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/25' 2023-10-30T13:28:42.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/30' 2023-10-30T13:28:42.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/33' 2023-10-30T13:28:42.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/23' 2023-10-30T13:28:42.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/35' 2023-10-30T13:28:42.697 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/26' 2023-10-30T13:28:42.698 INFO:tasks.workunit.client.0.smithi046.stdout:removed './286/37' 2023-10-30T13:28:42.698 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './286' 2023-10-30T13:28:44.690 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:28:44.695 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:28:50.793 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:50.864 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/21' 2023-10-30T13:28:50.864 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/27' 2023-10-30T13:28:50.865 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/31' 2023-10-30T13:28:50.865 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/36' 2023-10-30T13:28:50.865 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/20' 2023-10-30T13:28:50.865 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/39' 2023-10-30T13:28:50.865 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/34' 2023-10-30T13:28:50.866 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/38' 2023-10-30T13:28:50.866 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/28' 2023-10-30T13:28:50.866 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/24' 2023-10-30T13:28:50.866 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/29' 2023-10-30T13:28:50.867 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/32' 2023-10-30T13:28:50.867 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/22' 2023-10-30T13:28:50.867 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/25' 2023-10-30T13:28:50.867 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/30' 2023-10-30T13:28:50.867 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/33' 2023-10-30T13:28:50.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/23' 2023-10-30T13:28:50.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/35' 2023-10-30T13:28:50.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/26' 2023-10-30T13:28:50.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './287/37' 2023-10-30T13:28:50.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './287' 2023-10-30T13:28:50.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/21' 2023-10-30T13:28:50.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/27' 2023-10-30T13:28:50.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/31' 2023-10-30T13:28:50.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/36' 2023-10-30T13:28:50.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/20' 2023-10-30T13:28:50.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/39' 2023-10-30T13:28:50.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/34' 2023-10-30T13:28:50.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/38' 2023-10-30T13:28:50.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/28' 2023-10-30T13:28:50.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/24' 2023-10-30T13:28:50.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/29' 2023-10-30T13:28:50.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/32' 2023-10-30T13:28:50.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/22' 2023-10-30T13:28:50.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/25' 2023-10-30T13:28:50.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/30' 2023-10-30T13:28:50.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/33' 2023-10-30T13:28:50.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/23' 2023-10-30T13:28:50.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/35' 2023-10-30T13:28:50.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/26' 2023-10-30T13:28:50.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './288/37' 2023-10-30T13:28:50.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './288' 2023-10-30T13:28:50.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/21' 2023-10-30T13:28:50.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/27' 2023-10-30T13:28:50.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/31' 2023-10-30T13:28:50.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/36' 2023-10-30T13:28:50.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/20' 2023-10-30T13:28:50.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/39' 2023-10-30T13:28:50.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/34' 2023-10-30T13:28:50.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/38' 2023-10-30T13:28:50.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/28' 2023-10-30T13:28:50.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/24' 2023-10-30T13:28:50.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/29' 2023-10-30T13:28:50.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/32' 2023-10-30T13:28:50.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/22' 2023-10-30T13:28:50.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/25' 2023-10-30T13:28:50.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/30' 2023-10-30T13:28:50.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/33' 2023-10-30T13:28:50.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/23' 2023-10-30T13:28:50.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/35' 2023-10-30T13:28:50.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/26' 2023-10-30T13:28:50.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './289/37' 2023-10-30T13:28:50.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './289' 2023-10-30T13:28:50.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/21' 2023-10-30T13:28:50.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/27' 2023-10-30T13:28:50.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/31' 2023-10-30T13:28:50.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/36' 2023-10-30T13:28:50.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/20' 2023-10-30T13:28:50.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/39' 2023-10-30T13:28:50.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/34' 2023-10-30T13:28:50.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/38' 2023-10-30T13:28:50.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/28' 2023-10-30T13:28:50.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/24' 2023-10-30T13:28:50.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/29' 2023-10-30T13:28:50.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/32' 2023-10-30T13:28:50.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/22' 2023-10-30T13:28:50.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/25' 2023-10-30T13:28:50.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/30' 2023-10-30T13:28:50.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/33' 2023-10-30T13:28:50.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/23' 2023-10-30T13:28:50.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/35' 2023-10-30T13:28:50.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/26' 2023-10-30T13:28:50.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './290/37' 2023-10-30T13:28:50.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './290' 2023-10-30T13:28:50.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/21' 2023-10-30T13:28:50.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/27' 2023-10-30T13:28:50.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/31' 2023-10-30T13:28:50.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/36' 2023-10-30T13:28:50.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/20' 2023-10-30T13:28:50.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/39' 2023-10-30T13:28:50.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/34' 2023-10-30T13:28:50.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/38' 2023-10-30T13:28:50.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/28' 2023-10-30T13:28:50.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/24' 2023-10-30T13:28:50.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/29' 2023-10-30T13:28:50.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/32' 2023-10-30T13:28:50.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/22' 2023-10-30T13:28:50.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/25' 2023-10-30T13:28:50.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/30' 2023-10-30T13:28:50.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/33' 2023-10-30T13:28:50.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/23' 2023-10-30T13:28:50.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/35' 2023-10-30T13:28:50.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/26' 2023-10-30T13:28:50.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './291/37' 2023-10-30T13:28:50.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './291' 2023-10-30T13:28:50.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/21' 2023-10-30T13:28:50.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/27' 2023-10-30T13:28:50.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/31' 2023-10-30T13:28:50.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/36' 2023-10-30T13:28:50.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/20' 2023-10-30T13:28:50.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/39' 2023-10-30T13:28:50.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/34' 2023-10-30T13:28:50.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/38' 2023-10-30T13:28:50.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/28' 2023-10-30T13:28:50.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/24' 2023-10-30T13:28:50.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/29' 2023-10-30T13:28:50.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/32' 2023-10-30T13:28:50.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/22' 2023-10-30T13:28:50.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/25' 2023-10-30T13:28:50.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/30' 2023-10-30T13:28:50.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/33' 2023-10-30T13:28:50.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/23' 2023-10-30T13:28:50.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/35' 2023-10-30T13:28:50.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/26' 2023-10-30T13:28:50.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './292/37' 2023-10-30T13:28:50.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './292' 2023-10-30T13:28:50.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/21' 2023-10-30T13:28:50.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/27' 2023-10-30T13:28:50.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/31' 2023-10-30T13:28:50.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/36' 2023-10-30T13:28:50.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/20' 2023-10-30T13:28:50.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/39' 2023-10-30T13:28:50.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/34' 2023-10-30T13:28:50.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/38' 2023-10-30T13:28:50.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/28' 2023-10-30T13:28:50.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/24' 2023-10-30T13:28:50.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/29' 2023-10-30T13:28:50.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/32' 2023-10-30T13:28:50.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/22' 2023-10-30T13:28:50.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/25' 2023-10-30T13:28:50.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/30' 2023-10-30T13:28:50.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/33' 2023-10-30T13:28:50.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/23' 2023-10-30T13:28:50.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/35' 2023-10-30T13:28:50.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/26' 2023-10-30T13:28:50.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './293/37' 2023-10-30T13:28:50.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './293' 2023-10-30T13:28:50.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/21' 2023-10-30T13:28:50.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/27' 2023-10-30T13:28:50.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/31' 2023-10-30T13:28:50.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/36' 2023-10-30T13:28:50.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/20' 2023-10-30T13:28:50.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/39' 2023-10-30T13:28:50.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/34' 2023-10-30T13:28:50.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/38' 2023-10-30T13:28:50.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/28' 2023-10-30T13:28:50.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/24' 2023-10-30T13:28:50.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/29' 2023-10-30T13:28:50.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/32' 2023-10-30T13:28:50.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/22' 2023-10-30T13:28:50.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/25' 2023-10-30T13:28:50.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/30' 2023-10-30T13:28:50.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/33' 2023-10-30T13:28:50.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/23' 2023-10-30T13:28:50.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/35' 2023-10-30T13:28:50.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/26' 2023-10-30T13:28:50.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './294/37' 2023-10-30T13:28:50.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './294' 2023-10-30T13:28:50.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/21' 2023-10-30T13:28:50.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/27' 2023-10-30T13:28:50.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/31' 2023-10-30T13:28:50.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/36' 2023-10-30T13:28:50.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/20' 2023-10-30T13:28:50.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/39' 2023-10-30T13:28:50.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/34' 2023-10-30T13:28:50.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/38' 2023-10-30T13:28:50.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/28' 2023-10-30T13:28:50.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/24' 2023-10-30T13:28:50.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/29' 2023-10-30T13:28:50.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/32' 2023-10-30T13:28:50.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/22' 2023-10-30T13:28:50.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/25' 2023-10-30T13:28:50.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/30' 2023-10-30T13:28:50.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/33' 2023-10-30T13:28:50.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/23' 2023-10-30T13:28:50.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/35' 2023-10-30T13:28:50.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/26' 2023-10-30T13:28:50.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './295/37' 2023-10-30T13:28:50.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './295' 2023-10-30T13:28:50.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/21' 2023-10-30T13:28:50.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/27' 2023-10-30T13:28:50.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/31' 2023-10-30T13:28:50.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/36' 2023-10-30T13:28:50.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/20' 2023-10-30T13:28:50.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/39' 2023-10-30T13:28:50.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/34' 2023-10-30T13:28:50.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/38' 2023-10-30T13:28:50.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/28' 2023-10-30T13:28:50.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/24' 2023-10-30T13:28:50.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/29' 2023-10-30T13:28:50.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/32' 2023-10-30T13:28:50.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/22' 2023-10-30T13:28:50.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/25' 2023-10-30T13:28:50.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/30' 2023-10-30T13:28:50.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/33' 2023-10-30T13:28:50.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/23' 2023-10-30T13:28:50.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/35' 2023-10-30T13:28:50.903 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/26' 2023-10-30T13:28:50.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './296/37' 2023-10-30T13:28:50.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './296' 2023-10-30T13:28:50.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/21' 2023-10-30T13:28:50.904 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/27' 2023-10-30T13:28:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.969+0000 7f0f5aa6b640 1 Processor -- start 2023-10-30T13:28:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.969+0000 7f0f5aa6b640 1 -- start start 2023-10-30T13:28:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f5aa6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f5406f7a0 0x7f0f5406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f5aa6b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f540c6bd0 0x7f0f54069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f5aa6b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5406a3d0 con 0x7f0f54071110 2023-10-30T13:28:50.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f5aa6b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f540700c0 con 0x7f0f540c6bd0 2023-10-30T13:28:50.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f5aa6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f54066cf0 con 0x7f0f5406f7a0 2023-10-30T13:28:50.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f53fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f540c6bd0 0x7f0f54069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:50.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f53fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f540c6bd0 0x7f0f54069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49034/0 (socket says 172.21.15.46:49034) 2023-10-30T13:28:50.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f53fff640 1 -- 172.21.15.46:0/1536760904 learned_addr learned my addr 172.21.15.46:0/1536760904 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:28:50.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f58807640 1 --2- 172.21.15.46:0/1536760904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f5406f7a0 0x7f0f5406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:50.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f53fff640 1 -- 172.21.15.46:0/1536760904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f5406f7a0 msgr2=0x7f0f5406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:50.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f53fff640 1 --2- 172.21.15.46:0/1536760904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f5406f7a0 0x7f0f5406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:50.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f53fff640 1 -- 172.21.15.46:0/1536760904 >> v1:172.21.15.154:6789/0 conn(0x7f0f54071110 legacy=0x7f0f54158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:50.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f53fff640 1 -- 172.21.15.46:0/1536760904 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f540c70d0 con 0x7f0f540c6bd0 2023-10-30T13:28:50.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f58807640 1 --2- 172.21.15.46:0/1536760904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f5406f7a0 0x7f0f5406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:50.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.973+0000 7f0f53fff640 1 --2- 172.21.15.46:0/1536760904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f540c6bd0 0x7f0f54069e90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0f40009870 tx=0x7f0f4001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa3838a1196b74d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:50.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f537fe640 1 -- 172.21.15.46:0/1536760904 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f400a8a30 con 0x7f0f540c6bd0 2023-10-30T13:28:50.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f537fe640 1 -- 172.21.15.46:0/1536760904 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f400a8b90 con 0x7f0f540c6bd0 2023-10-30T13:28:50.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f537fe640 1 -- 172.21.15.46:0/1536760904 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f400b0440 con 0x7f0f540c6bd0 2023-10-30T13:28:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/1536760904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f540c6bd0 msgr2=0x7f0f54069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 --2- 172.21.15.46:0/1536760904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f540c6bd0 0x7f0f54069e90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0f40009870 tx=0x7f0f4001ed30 comp rx=0 tx=0).stop 2023-10-30T13:28:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/1536760904 shutdown_connections 2023-10-30T13:28:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 --2- 172.21.15.46:0/1536760904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f540c6bd0 0x7f0f54069e90 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:50.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 --2- 172.21.15.46:0/1536760904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f5406f7a0 0x7f0f5406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:50.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/1536760904 >> 172.21.15.46:0/1536760904 conn(0x7f0f540c7910 msgr2=0x7f0f540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:50.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/1536760904 shutdown_connections 2023-10-30T13:28:50.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/1536760904 wait complete. 2023-10-30T13:28:50.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 Processor -- start 2023-10-30T13:28:50.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 -- start start 2023-10-30T13:28:50.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f5406f7a0 0x7f0f541ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:50.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f54071110 0x7f0f541ef670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:50.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f540c6bd0 0x7f0f540af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:50.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f540ae150 con 0x7f0f540c6bd0 2023-10-30T13:28:50.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f540ca700 con 0x7f0f5406f7a0 2023-10-30T13:28:50.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f5aa6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f540ca3d0 con 0x7f0f54071110 2023-10-30T13:28:50.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f59008640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f540c6bd0 0x7f0f540af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:50.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f59008640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f540c6bd0 0x7f0f540af8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49042/0 (socket says 172.21.15.46:49042) 2023-10-30T13:28:50.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.977+0000 7f0f59008640 1 -- 172.21.15.46:0/3681674479 learned_addr learned my addr 172.21.15.46:0/3681674479 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:28:50.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f58807640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f5406f7a0 0x7f0f541ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:50.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f53fff640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f54071110 0x7f0f541ef670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:50.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f59008640 1 -- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f54071110 msgr2=0x7f0f541ef670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:50.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f59008640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f54071110 0x7f0f541ef670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:50.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f59008640 1 -- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f5406f7a0 msgr2=0x7f0f541ef130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:50.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f59008640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f5406f7a0 0x7f0f541ef130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:50.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f59008640 1 -- 172.21.15.46:0/3681674479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f400094d0 con 0x7f0f540c6bd0 2023-10-30T13:28:50.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f59008640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f540c6bd0 0x7f0f540af8b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c00c730 tx=0x7f0f4c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:50.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f517fa640 1 -- 172.21.15.46:0/3681674479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c003b80 con 0x7f0f540c6bd0 2023-10-30T13:28:50.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f517fa640 1 -- 172.21.15.46:0/3681674479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4c003ce0 con 0x7f0f540c6bd0 2023-10-30T13:28:50.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f517fa640 1 -- 172.21.15.46:0/3681674479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c00aa30 con 0x7f0f540c6bd0 2023-10-30T13:28:50.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/3681674479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f540c0190 con 0x7f0f540c6bd0 2023-10-30T13:28:50.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/3681674479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f54150740 con 0x7f0f540c6bd0 2023-10-30T13:28:50.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/3681674479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f540ca540 con 0x7f0f540c6bd0 2023-10-30T13:28:50.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.981+0000 7f0f517fa640 1 -- 172.21.15.46:0/3681674479 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0f4c00ab90 con 0x7f0f540c6bd0 2023-10-30T13:28:50.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.985+0000 7f0f517fa640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f380664a0 0x7f0f38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:50.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.985+0000 7f0f517fa640 1 -- 172.21.15.46:0/3681674479 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f0f4c0840b0 con 0x7f0f540c6bd0 2023-10-30T13:28:50.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.985+0000 7f0f58807640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f380664a0 0x7f0f38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:50.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.985+0000 7f0f58807640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f380664a0 0x7f0f38068960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800ba10 tx=0x7f0f48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:50.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:50.989+0000 7f0f517fa640 1 -- 172.21.15.46:0/3681674479 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0f540ca540 con 0x7f0f540c6bd0 2023-10-30T13:28:51.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/3681674479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f540ca540 con 0x7f0f540c6bd0 2023-10-30T13:28:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f517fa640 1 -- 172.21.15.46:0/3681674479 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f0f540ca540 con 0x7f0f540c6bd0 2023-10-30T13:28:51.199 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:28:51.199 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:28:00.722346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 28 2023-10-30T13:28:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f380664a0 msgr2=0x7f0f38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f380664a0 0x7f0f38068960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800ba10 tx=0x7f0f48008040 comp rx=0 tx=0).stop 2023-10-30T13:28:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/3681674479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f540c6bd0 msgr2=0x7f0f540af8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f540c6bd0 0x7f0f540af8b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c00c730 tx=0x7f0f4c00cbf0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/3681674479 shutdown_connections 2023-10-30T13:28:51.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f380664a0 0x7f0f38068960 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f540c6bd0 0x7f0f540af8b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f54071110 0x7f0f541ef670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 --2- 172.21.15.46:0/3681674479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f5406f7a0 0x7f0f541ef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/3681674479 >> 172.21.15.46:0/3681674479 conn(0x7f0f540c7910 msgr2=0x7f0f540c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/3681674479 shutdown_connections 2023-10-30T13:28:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.193+0000 7f0f5aa6b640 1 -- 172.21.15.46:0/3681674479 wait complete. 2023-10-30T13:28:51.219 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=1) 2023-10-30T13:28:51.220 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T13:28:51.220 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:28:51.242 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T13:28:51.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:51.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6274165640 1 Processor -- start 2023-10-30T13:28:51.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6274165640 1 -- start start 2023-10-30T13:28:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6274165640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c0cf160 0x7f626c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:51.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6274165640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c0cb800 con 0x7f626c1538a0 2023-10-30T13:28:51.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6274165640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c0cb330 con 0x7f626c15d6f0 2023-10-30T13:28:51.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6274165640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0cad70 con 0x7f626c0cf160 2023-10-30T13:28:51.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6271f01640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f626c1538a0 0x7f626c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58744/0 (socket says 172.21.15.46:58744) 2023-10-30T13:28:51.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6271f01640 1 -- 172.21.15.46:0/2518291061 learned_addr learned my addr 172.21.15.46:0/2518291061 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:28:51.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6272702640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c0cf160 0x7f626c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:51.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734088462 0 0) 0x7f626c0cb800 con 0x7f626c1538a0 2023-10-30T13:28:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6258003530 con 0x7f626c1538a0 2023-10-30T13:28:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871307134 0 0) 0x7f626c0cb330 con 0x7f626c15d6f0 2023-10-30T13:28:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f626c0cb800 con 0x7f626c15d6f0 2023-10-30T13:28:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2932886239 0 0) 0x7f6258003530 con 0x7f626c1538a0 2023-10-30T13:28:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c0cf160 msgr2=0x7f626c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c0cf160 0x7f626c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 >> v1:172.21.15.154:6789/0 conn(0x7f626c15d6f0 legacy=0x7f626c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c0cb050 con 0x7f626c1538a0 2023-10-30T13:28:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f6268001680 con 0x7f626c1538a0 2023-10-30T13:28:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 >> v1:172.21.15.46:6789/0 conn(0x7f626c1538a0 legacy=0x7f626c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6258003b40 0x7f6258003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c0cf160 0x7f62580047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6258004d10 0x7f6258005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0cad70 con 0x7f6258003b40 2023-10-30T13:28:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625800ba40 con 0x7f6258004d10 2023-10-30T13:28:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625800bc00 con 0x7f626c0cf160 2023-10-30T13:28:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6271f01640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6258003b40 0x7f6258003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6271700640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c0cf160 0x7f62580047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.385+0000 7f6272702640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6258004d10 0x7f6258005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6271f01640 1 -- 172.21.15.46:0/2518291061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c0cf160 msgr2=0x7f62580047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6271f01640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c0cf160 0x7f62580047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6271f01640 1 -- 172.21.15.46:0/2518291061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6258004d10 msgr2=0x7f6258005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6271f01640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6258004d10 0x7f6258005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6271f01640 1 -- 172.21.15.46:0/2518291061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6258003530 con 0x7f6258003b40 2023-10-30T13:28:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6272702640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6258004d10 0x7f6258005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:51.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6271f01640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6258003b40 0x7f6258003f40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6268007fb0 tx=0x7f626801cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7411ad4e18ccc1d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:51.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268005ae0 con 0x7f6258003b40 2023-10-30T13:28:51.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6268005c40 con 0x7f6258003b40 2023-10-30T13:28:51.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6270eff640 1 -- 172.21.15.46:0/2518291061 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62680a5820 con 0x7f6258003b40 2023-10-30T13:28:51.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6274165640 1 -- 172.21.15.46:0/2518291061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6258003b40 msgr2=0x7f6258003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:51.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6274165640 1 --2- 172.21.15.46:0/2518291061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6258003b40 0x7f6258003f40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6268007fb0 tx=0x7f626801cb20 comp rx=0 tx=0).stop 2023-10-30T13:28:51.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6272702640 1 -- 172.21.15.46:0/2518291061 reap_dead start 2023-10-30T13:28:51.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6274165640 1 -- 172.21.15.46:0/2518291061 shutdown_connections 2023-10-30T13:28:51.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6274165640 1 -- 172.21.15.46:0/2518291061 >> 172.21.15.46:0/2518291061 conn(0x7f626c06bfe0 msgr2=0x7f626c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:51.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6274165640 1 -- 172.21.15.46:0/2518291061 shutdown_connections 2023-10-30T13:28:51.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.389+0000 7f6274165640 1 -- 172.21.15.46:0/2518291061 wait complete. 2023-10-30T13:28:51.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6274165640 1 Processor -- start 2023-10-30T13:28:51.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6274165640 1 -- start start 2023-10-30T13:28:51.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6274165640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6258003b40 0x7f626c1e47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:51.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6274165640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c1e2ae0 0x7f626c1e4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:51.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6274165640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c1e6170 0x7f626c14fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:51.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6274165640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c1efd70 con 0x7f6258003b40 2023-10-30T13:28:51.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6274165640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c1e4340 con 0x7f626c1e6170 2023-10-30T13:28:51.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6274165640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c1f0260 con 0x7f626c1e2ae0 2023-10-30T13:28:51.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6271f01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6258003b40 0x7f626c1e47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:51.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6271f01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6258003b40 0x7f626c1e47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49056/0 (socket says 172.21.15.46:49056) 2023-10-30T13:28:51.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6271f01640 1 -- 172.21.15.46:0/2611594671 learned_addr learned my addr 172.21.15.46:0/2611594671 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:28:51.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6271700640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c1e2ae0 0x7f626c1e4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:51.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6272702640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c1e6170 0x7f626c14fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:51.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6271f01640 1 -- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c1e2ae0 msgr2=0x7f626c1e4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:51.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6271f01640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c1e2ae0 0x7f626c1e4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6271f01640 1 -- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c1e6170 msgr2=0x7f626c14fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:51.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6271f01640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c1e6170 0x7f626c14fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6271f01640 1 -- 172.21.15.46:0/2611594671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c1e3a50 con 0x7f6258003b40 2023-10-30T13:28:51.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6271700640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c1e2ae0 0x7f626c1e4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:51.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6271f01640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6258003b40 0x7f626c1e47d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f626801c950 tx=0x7f6258005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:51.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f62627fc640 1 -- 172.21.15.46:0/2611594671 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62680b4070 con 0x7f6258003b40 2023-10-30T13:28:51.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f62627fc640 1 -- 172.21.15.46:0/2611594671 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62680a5a30 con 0x7f6258003b40 2023-10-30T13:28:51.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f62627fc640 1 -- 172.21.15.46:0/2611594671 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62680a5dc0 con 0x7f6258003b40 2023-10-30T13:28:51.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6274165640 1 -- 172.21.15.46:0/2611594671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626c1e6870 con 0x7f6258003b40 2023-10-30T13:28:51.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.393+0000 7f6274165640 1 -- 172.21.15.46:0/2611594671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626c1e3cf0 con 0x7f6258003b40 2023-10-30T13:28:51.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.397+0000 7f6274165640 1 -- 172.21.15.46:0/2611594671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626c04e170 con 0x7f6258003b40 2023-10-30T13:28:51.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.397+0000 7f62627fc640 1 -- 172.21.15.46:0/2611594671 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6268007090 con 0x7f6258003b40 2023-10-30T13:28:51.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.397+0000 7f62627fc640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62500664a0 0x7f6250068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:51.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.397+0000 7f62627fc640 1 -- 172.21.15.46:0/2611594671 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f625c001d30 con 0x7f6258003b40 2023-10-30T13:28:51.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.397+0000 7f6271700640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62500664a0 0x7f6250068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:51.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.401+0000 7f6271700640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62500664a0 0x7f6250068960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f626c1ecf20 tx=0x7f625c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:51.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.401+0000 7f62627fc640 1 -- 172.21.15.46:0/2611594671 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f62680eaed0 con 0x7f6258003b40 2023-10-30T13:28:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.601+0000 7f6274165640 1 -- 172.21.15.46:0/2611594671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f626c0b1dd0 con 0x7f6258003b40 2023-10-30T13:28:51.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.601+0000 7f62627fc640 1 -- 172.21.15.46:0/2611594671 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f62680bf090 con 0x7f6258003b40 2023-10-30T13:28:51.610 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:28:51.610 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:28:00.722346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:51.611 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 28 2023-10-30T13:28:51.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.605+0000 7f6274165640 1 -- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62500664a0 msgr2=0x7f6250068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:51.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.605+0000 7f6274165640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62500664a0 0x7f6250068960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f626c1ecf20 tx=0x7f625c009040 comp rx=0 tx=0).stop 2023-10-30T13:28:51.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.605+0000 7f6274165640 1 -- 172.21.15.46:0/2611594671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6258003b40 msgr2=0x7f626c1e47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:51.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.605+0000 7f6274165640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6258003b40 0x7f626c1e47d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f626801c950 tx=0x7f6258005800 comp rx=0 tx=0).stop 2023-10-30T13:28:51.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.605+0000 7f6274165640 1 -- 172.21.15.46:0/2611594671 shutdown_connections 2023-10-30T13:28:51.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.605+0000 7f6274165640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62500664a0 0x7f6250068960 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.605+0000 7f6274165640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c1e6170 0x7f626c14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.605+0000 7f6274165640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c1e2ae0 0x7f626c1e4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.605+0000 7f6274165640 1 --2- 172.21.15.46:0/2611594671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6258003b40 0x7f626c1e47d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.605+0000 7f6274165640 1 -- 172.21.15.46:0/2611594671 >> 172.21.15.46:0/2611594671 conn(0x7f626c06bfe0 msgr2=0x7f626c0c5300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.605+0000 7f6274165640 1 -- 172.21.15.46:0/2611594671 shutdown_connections 2023-10-30T13:28:51.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:51.605+0000 7f6274165640 1 -- 172.21.15.46:0/2611594671 wait complete. 2023-10-30T13:28:51.630 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:28:53.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:53.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f8830479640 1 Processor -- start 2023-10-30T13:28:53.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f8830479640 1 -- start start 2023-10-30T13:28:53.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f8830479640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f88281538a0 0x7f882815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:53.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f8830479640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88280cb800 con 0x7f88280ca0e0 2023-10-30T13:28:53.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f8830479640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882815dc30 con 0x7f88280cf160 2023-10-30T13:28:53.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f8830479640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88280cb330 con 0x7f88281538a0 2023-10-30T13:28:53.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882da14640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f88280ca0e0 0x7f882815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58750/0 (socket says 172.21.15.46:58750) 2023-10-30T13:28:53.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882da14640 1 -- 172.21.15.46:0/199044082 learned_addr learned my addr 172.21.15.46:0/199044082 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:28:53.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882e215640 1 --2- 172.21.15.46:0/199044082 >> v2:172.21.15.154:3300/0 conn(0x7f88281538a0 0x7f882815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224937004 0 0) 0x7f88280cb800 con 0x7f88280ca0e0 2023-10-30T13:28:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8814003530 con 0x7f88280ca0e0 2023-10-30T13:28:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1730149084 0 0) 0x7f8814003530 con 0x7f88280ca0e0 2023-10-30T13:28:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 >> v2:172.21.15.154:3300/0 conn(0x7f88281538a0 msgr2=0x7f882815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 --2- 172.21.15.46:0/199044082 >> v2:172.21.15.154:3300/0 conn(0x7f88281538a0 0x7f882815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:53.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 >> v1:172.21.15.154:6789/0 conn(0x7f88280cf160 legacy=0x7f8828164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:53.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88280cad70 con 0x7f88280ca0e0 2023-10-30T13:28:53.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f8818001680 con 0x7f88280ca0e0 2023-10-30T13:28:53.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 >> v1:172.21.15.46:6789/0 conn(0x7f88280ca0e0 legacy=0x7f882815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:53.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 --2- 172.21.15.46:0/199044082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814003980 0x7f8814003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:53.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 --2- 172.21.15.46:0/199044082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88140047d0 0x7f8814004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:53.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 --2- 172.21.15.46:0/199044082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8814005190 0x7f881400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:53.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88280cb330 con 0x7f88140047d0 2023-10-30T13:28:53.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881400c040 con 0x7f8814003980 2023-10-30T13:28:53.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881400c200 con 0x7f8814005190 2023-10-30T13:28:53.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.781+0000 7f882da14640 1 --2- 172.21.15.46:0/199044082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88140047d0 0x7f8814004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:53.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f882e215640 1 --2- 172.21.15.46:0/199044082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814003980 0x7f8814003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:53.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f882ea16640 1 --2- 172.21.15.46:0/199044082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8814005190 0x7f881400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:53.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f882da14640 1 -- 172.21.15.46:0/199044082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8814005190 msgr2=0x7f881400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:53.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f882da14640 1 --2- 172.21.15.46:0/199044082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8814005190 0x7f881400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:53.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f882da14640 1 -- 172.21.15.46:0/199044082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814003980 msgr2=0x7f8814003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:53.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f882da14640 1 --2- 172.21.15.46:0/199044082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814003980 0x7f8814003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:53.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f882da14640 1 -- 172.21.15.46:0/199044082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8814003530 con 0x7f88140047d0 2023-10-30T13:28:53.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f882ea16640 1 -- 172.21.15.46:0/199044082 reap_dead start 2023-10-30T13:28:53.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f882da14640 1 --2- 172.21.15.46:0/199044082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88140047d0 0x7f8814004c50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f881800c1c0 tx=0x7f881801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8847aa3a1313e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:53.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88180a5b20 con 0x7f88140047d0 2023-10-30T13:28:53.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88180a5c80 con 0x7f88140047d0 2023-10-30T13:28:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f882d213640 1 -- 172.21.15.46:0/199044082 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8818005b20 con 0x7f88140047d0 2023-10-30T13:28:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f8830479640 1 -- 172.21.15.46:0/199044082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88140047d0 msgr2=0x7f8814004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.785+0000 7f8830479640 1 --2- 172.21.15.46:0/199044082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88140047d0 0x7f8814004c50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f881800c1c0 tx=0x7f881801cc20 comp rx=0 tx=0).stop 2023-10-30T13:28:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 -- 172.21.15.46:0/199044082 shutdown_connections 2023-10-30T13:28:53.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 --2- 172.21.15.46:0/199044082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88140047d0 0x7f8814004c50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:53.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 -- 172.21.15.46:0/199044082 >> 172.21.15.46:0/199044082 conn(0x7f882806bfe0 msgr2=0x7f882806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:53.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 -- 172.21.15.46:0/199044082 shutdown_connections 2023-10-30T13:28:53.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 -- 172.21.15.46:0/199044082 wait complete. 2023-10-30T13:28:53.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 Processor -- start 2023-10-30T13:28:53.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 -- start start 2023-10-30T13:28:53.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88140047d0 0x7f8828158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:53.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88281590d0 0x7f88280b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:53.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88280b4360 0x7f8828145580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:53.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88280b3a00 con 0x7f88280b4360 2023-10-30T13:28:53.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88281596e0 con 0x7f88140047d0 2023-10-30T13:28:53.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f8830479640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88280b47e0 con 0x7f88281590d0 2023-10-30T13:28:53.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f882ea16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88280b4360 0x7f8828145580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:53.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f882ea16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88280b4360 0x7f8828145580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49082/0 (socket says 172.21.15.46:49082) 2023-10-30T13:28:53.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f882ea16640 1 -- 172.21.15.46:0/4102073267 learned_addr learned my addr 172.21.15.46:0/4102073267 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:28:53.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f882e215640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88140047d0 0x7f8828158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:53.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f882da14640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88281590d0 0x7f88280b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:53.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f882ea16640 1 -- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88281590d0 msgr2=0x7f88280b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:53.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f882ea16640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88281590d0 0x7f88280b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:53.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f882ea16640 1 -- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88140047d0 msgr2=0x7f8828158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:53.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f882ea16640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88140047d0 0x7f8828158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:53.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f882ea16640 1 -- 172.21.15.46:0/4102073267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88281e5980 con 0x7f88280b4360 2023-10-30T13:28:53.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f882ea16640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88280b4360 0x7f8828145580 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8828072ce0 tx=0x7f882000e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:53.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f881effd640 1 -- 172.21.15.46:0/4102073267 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820003b90 con 0x7f88280b4360 2023-10-30T13:28:53.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.789+0000 7f881effd640 1 -- 172.21.15.46:0/4102073267 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8820003d30 con 0x7f88280b4360 2023-10-30T13:28:53.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.793+0000 7f881effd640 1 -- 172.21.15.46:0/4102073267 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820003ea0 con 0x7f88280b4360 2023-10-30T13:28:53.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.793+0000 7f8830479640 1 -- 172.21.15.46:0/4102073267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8828158210 con 0x7f88280b4360 2023-10-30T13:28:53.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.793+0000 7f8830479640 1 -- 172.21.15.46:0/4102073267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88280b3620 con 0x7f88280b4360 2023-10-30T13:28:53.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.793+0000 7f8830479640 1 -- 172.21.15.46:0/4102073267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88281594b0 con 0x7f88280b4360 2023-10-30T13:28:53.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.793+0000 7f881effd640 1 -- 172.21.15.46:0/4102073267 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8820005310 con 0x7f88280b4360 2023-10-30T13:28:53.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.793+0000 7f881effd640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f880c0664d0 0x7f880c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:53.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.793+0000 7f881effd640 1 -- 172.21.15.46:0/4102073267 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f8820083840 con 0x7f88280b4360 2023-10-30T13:28:53.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.793+0000 7f882e215640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f880c0664d0 0x7f880c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:53.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.797+0000 7f882e215640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f880c0664d0 0x7f880c068990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8824009b60 tx=0x7f8824008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:53.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.797+0000 7f881effd640 1 -- 172.21.15.46:0/4102073267 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8820050900 con 0x7f88280b4360 2023-10-30T13:28:53.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.989+0000 7f8830479640 1 -- 172.21.15.46:0/4102073267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88280b31a0 con 0x7f88280b4360 2023-10-30T13:28:53.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.989+0000 7f881effd640 1 -- 172.21.15.46:0/4102073267 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f88280b31a0 con 0x7f88280b4360 2023-10-30T13:28:53.997 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:28:53.997 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:28:00.722346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:53.997 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 28 2023-10-30T13:28:53.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.993+0000 7f8830479640 1 -- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f880c0664d0 msgr2=0x7f880c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:53.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.993+0000 7f8830479640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f880c0664d0 0x7f880c068990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8824009b60 tx=0x7f8824008040 comp rx=0 tx=0).stop 2023-10-30T13:28:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.993+0000 7f8830479640 1 -- 172.21.15.46:0/4102073267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88280b4360 msgr2=0x7f8828145580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.993+0000 7f8830479640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88280b4360 0x7f8828145580 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8828072ce0 tx=0x7f882000e960 comp rx=0 tx=0).stop 2023-10-30T13:28:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.993+0000 7f8830479640 1 -- 172.21.15.46:0/4102073267 shutdown_connections 2023-10-30T13:28:54.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.993+0000 7f8830479640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f880c0664d0 0x7f880c068990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:54.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.993+0000 7f8830479640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88280b4360 0x7f8828145580 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:54.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.993+0000 7f8830479640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88281590d0 0x7f88280b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:54.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.993+0000 7f8830479640 1 --2- 172.21.15.46:0/4102073267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88140047d0 0x7f8828158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:54.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.993+0000 7f8830479640 1 -- 172.21.15.46:0/4102073267 >> 172.21.15.46:0/4102073267 conn(0x7f882806bfe0 msgr2=0x7f88280c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:54.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.993+0000 7f8830479640 1 -- 172.21.15.46:0/4102073267 shutdown_connections 2023-10-30T13:28:54.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:53.993+0000 7f8830479640 1 -- 172.21.15.46:0/4102073267 wait complete. 2023-10-30T13:28:54.018 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:28:56.019 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.165+0000 7fc0f89d3640 1 Processor -- start 2023-10-30T13:28:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f89d3640 1 -- start start 2023-10-30T13:28:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f89d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0f4151800 0x7fc0f406e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f89d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc0f40c8260 0x7fc0f40cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f89d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f406e090 con 0x7fc0f40c1ee0 2023-10-30T13:28:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f89d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f406dc70 con 0x7fc0f40c8260 2023-10-30T13:28:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f89d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f40c8a50 con 0x7fc0f4151800 2023-10-30T13:28:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc0f40c8260 0x7fc0f40cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc0f40c8260 0x7fc0f40cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49094/0 (socket says 172.21.15.46:49094) 2023-10-30T13:28:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f2d9d640 1 -- 172.21.15.46:0/3778722371 learned_addr learned my addr 172.21.15.46:0/3778722371 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:28:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0ebfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc0f40c1ee0 0x7fc0f40c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58766/0 (socket says 172.21.15.46:58766) 2023-10-30T13:28:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f259c640 1 --2- 172.21.15.46:0/3778722371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0f4151800 0x7fc0f406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f2d9d640 1 -- 172.21.15.46:0/3778722371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0f4151800 msgr2=0x7fc0f406e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f2d9d640 1 --2- 172.21.15.46:0/3778722371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0f4151800 0x7fc0f406e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f2d9d640 1 -- 172.21.15.46:0/3778722371 >> v1:172.21.15.46:6789/0 conn(0x7fc0f40c1ee0 legacy=0x7fc0f40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f2d9d640 1 -- 172.21.15.46:0/3778722371 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f40c3b10 con 0x7fc0f40c8260 2023-10-30T13:28:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f1d9b640 1 -- 172.21.15.46:0/3778722371 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031135448 0 0) 0x7fc0f406e090 con 0x7fc0f40c1ee0 2023-10-30T13:28:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.169+0000 7fc0f259c640 1 --2- 172.21.15.46:0/3778722371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0f4151800 0x7fc0f406e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f2d9d640 1 --2- 172.21.15.46:0/3778722371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0f40c8260 0x7fc0f40cce00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009e50 tx=0x7fc0ec01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6010bab8549e023 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f1d9b640 1 -- 172.21.15.46:0/3778722371 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0a5720 con 0x7fc0f40c8260 2023-10-30T13:28:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f1d9b640 1 -- 172.21.15.46:0/3778722371 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec0a5880 con 0x7fc0f40c8260 2023-10-30T13:28:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f1d9b640 1 -- 172.21.15.46:0/3778722371 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0a5c10 con 0x7fc0f40c8260 2023-10-30T13:28:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f89d3640 1 -- 172.21.15.46:0/3778722371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0f40c8260 msgr2=0x7fc0f40cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f89d3640 1 --2- 172.21.15.46:0/3778722371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0f40c8260 0x7fc0f40cce00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009e50 tx=0x7fc0ec01c950 comp rx=0 tx=0).stop 2023-10-30T13:28:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f89d3640 1 -- 172.21.15.46:0/3778722371 shutdown_connections 2023-10-30T13:28:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f89d3640 1 --2- 172.21.15.46:0/3778722371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0f40c8260 0x7fc0f40cce00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f89d3640 1 --2- 172.21.15.46:0/3778722371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0f4151800 0x7fc0f406e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f89d3640 1 -- 172.21.15.46:0/3778722371 >> 172.21.15.46:0/3778722371 conn(0x7fc0f4071ea0 msgr2=0x7fc0f4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f89d3640 1 -- 172.21.15.46:0/3778722371 shutdown_connections 2023-10-30T13:28:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f89d3640 1 -- 172.21.15.46:0/3778722371 wait complete. 2023-10-30T13:28:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f89d3640 1 Processor -- start 2023-10-30T13:28:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.173+0000 7fc0f89d3640 1 -- start start 2023-10-30T13:28:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f89d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0f40c1ee0 0x7fc0f4143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f89d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0f40c8260 0x7fc0f414a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f89d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0f4151800 0x7fc0f414a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f89d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f4149730 con 0x7fc0f4151800 2023-10-30T13:28:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f89d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f40c59e0 con 0x7fc0f40c1ee0 2023-10-30T13:28:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f89d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f41f65b0 con 0x7fc0f40c8260 2023-10-30T13:28:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0f4151800 0x7fc0f414a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0f4151800 0x7fc0f414a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49108/0 (socket says 172.21.15.46:49108) 2023-10-30T13:28:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f2d9d640 1 -- 172.21.15.46:0/624821209 learned_addr learned my addr 172.21.15.46:0/624821209 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:28:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0ebfff640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0f40c8260 0x7fc0f414a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f259c640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0f40c1ee0 0x7fc0f4143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f2d9d640 1 -- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0f40c8260 msgr2=0x7fc0f414a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f2d9d640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0f40c8260 0x7fc0f414a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f2d9d640 1 -- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0f40c1ee0 msgr2=0x7fc0f4143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f2d9d640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0f40c1ee0 0x7fc0f4143ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f2d9d640 1 -- 172.21.15.46:0/624821209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ec009b30 con 0x7fc0f4151800 2023-10-30T13:28:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f259c640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0f40c1ee0 0x7fc0f4143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0ebfff640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0f40c8260 0x7fc0f414a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f2d9d640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0f4151800 0x7fc0f414a7f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0af040 tx=0x7fc0ec0196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0eaffd640 1 -- 172.21.15.46:0/624821209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0b8670 con 0x7fc0f4151800 2023-10-30T13:28:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0eaffd640 1 -- 172.21.15.46:0/624821209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec0b87d0 con 0x7fc0f4151800 2023-10-30T13:28:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0eaffd640 1 -- 172.21.15.46:0/624821209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0b8b80 con 0x7fc0f4151800 2023-10-30T13:28:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f89d3640 1 -- 172.21.15.46:0/624821209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f40c5ca0 con 0x7fc0f4151800 2023-10-30T13:28:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f89d3640 1 -- 172.21.15.46:0/624821209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f4149ab0 con 0x7fc0f4151800 2023-10-30T13:28:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.177+0000 7fc0f89d3640 1 -- 172.21.15.46:0/624821209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f404e170 con 0x7fc0f4151800 2023-10-30T13:28:56.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.181+0000 7fc0eaffd640 1 -- 172.21.15.46:0/624821209 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc0ec0b8ce0 con 0x7fc0f4151800 2023-10-30T13:28:56.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.181+0000 7fc0eaffd640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0d40664a0 0x7fc0d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:56.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.181+0000 7fc0eaffd640 1 -- 172.21.15.46:0/624821209 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6678+0+0 (secure 0 0 0) 0x7fc0ec11f7d0 con 0x7fc0f4151800 2023-10-30T13:28:56.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.181+0000 7fc0f259c640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0d40664a0 0x7fc0d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:56.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.181+0000 7fc0f259c640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0d40664a0 0x7fc0d4068960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400b990 tx=0x7fc0e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:56.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.185+0000 7fc0eaffd640 1 -- 172.21.15.46:0/624821209 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc0ec0ec9b0 con 0x7fc0f4151800 2023-10-30T13:28:56.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.377+0000 7fc0f89d3640 1 -- 172.21.15.46:0/624821209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0f40c6de0 con 0x7fc0f4151800 2023-10-30T13:28:56.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.377+0000 7fc0eaffd640 1 -- 172.21.15.46:0/624821209 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fc0ec0effd0 con 0x7fc0f4151800 2023-10-30T13:28:56.384 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:28:56.384 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:28:00.722346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:56.385 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 28 2023-10-30T13:28:56.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.381+0000 7fc0f89d3640 1 -- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0d40664a0 msgr2=0x7fc0d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:56.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.381+0000 7fc0f89d3640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0d40664a0 0x7fc0d4068960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400b990 tx=0x7fc0e4008040 comp rx=0 tx=0).stop 2023-10-30T13:28:56.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.381+0000 7fc0f89d3640 1 -- 172.21.15.46:0/624821209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0f4151800 msgr2=0x7fc0f414a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:56.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.381+0000 7fc0f89d3640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0f4151800 0x7fc0f414a7f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0af040 tx=0x7fc0ec0196f0 comp rx=0 tx=0).stop 2023-10-30T13:28:56.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.381+0000 7fc0f89d3640 1 -- 172.21.15.46:0/624821209 shutdown_connections 2023-10-30T13:28:56.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.381+0000 7fc0f89d3640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0d40664a0 0x7fc0d4068960 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:56.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.381+0000 7fc0f89d3640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0f4151800 0x7fc0f414a7f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:56.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.381+0000 7fc0f89d3640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0f40c8260 0x7fc0f414a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:56.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.381+0000 7fc0f89d3640 1 --2- 172.21.15.46:0/624821209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0f40c1ee0 0x7fc0f4143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:56.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.381+0000 7fc0f89d3640 1 -- 172.21.15.46:0/624821209 >> 172.21.15.46:0/624821209 conn(0x7fc0f4071ea0 msgr2=0x7fc0f40c8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:56.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.381+0000 7fc0f89d3640 1 -- 172.21.15.46:0/624821209 shutdown_connections 2023-10-30T13:28:56.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:56.381+0000 7fc0f89d3640 1 -- 172.21.15.46:0/624821209 wait complete. 2023-10-30T13:28:56.406 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:28:58.407 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d45e9c640 1 Processor -- start 2023-10-30T13:28:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d45e9c640 1 -- start start 2023-10-30T13:28:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d45e9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8d401538a0 0x7f8d4015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d45e9c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d400ca0e0 0x7f8d4015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d45e9c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d400cb800 con 0x7f8d400cf160 2023-10-30T13:28:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d45e9c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d4015dc30 con 0x7f8d401538a0 2023-10-30T13:28:58.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d45e9c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400cb330 con 0x7f8d400ca0e0 2023-10-30T13:28:58.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d3f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8d401538a0 0x7f8d4015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:58.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d3ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8d400cf160 0x7f8d400cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59072/0 (socket says 172.21.15.46:59072) 2023-10-30T13:28:58.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d3ffff640 1 -- 172.21.15.46:0/1745424130 learned_addr learned my addr 172.21.15.46:0/1745424130 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:28:58.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d3f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8d401538a0 0x7f8d4015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33968/0 (socket says 172.21.15.46:33968) 2023-10-30T13:28:58.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d3effd640 1 --2- 172.21.15.46:0/1745424130 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d400ca0e0 0x7f8d4015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:58.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d3f7fe640 1 -- 172.21.15.46:0/1745424130 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d400ca0e0 msgr2=0x7f8d4015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:58.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d3f7fe640 1 --2- 172.21.15.46:0/1745424130 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d400ca0e0 0x7f8d4015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:58.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d3f7fe640 1 -- 172.21.15.46:0/1745424130 >> v1:172.21.15.46:6789/0 conn(0x7f8d400cf160 legacy=0x7f8d400cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:58.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d3f7fe640 1 -- 172.21.15.46:0/1745424130 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400cad70 con 0x7f8d401538a0 2023-10-30T13:28:58.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.557+0000 7f8d3e7fc640 1 -- 172.21.15.46:0/1745424130 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516227789 0 0) 0x7f8d400cb800 con 0x7f8d400cf160 2023-10-30T13:28:58.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.561+0000 7f8d3f7fe640 1 --2- 172.21.15.46:0/1745424130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d401538a0 0x7f8d4015d6f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8d300097c0 tx=0x7f8d3001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4320df1339d6256 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:58.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.561+0000 7f8d3e7fc640 1 -- 172.21.15.46:0/1745424130 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d300a8970 con 0x7f8d401538a0 2023-10-30T13:28:58.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.561+0000 7f8d3e7fc640 1 -- 172.21.15.46:0/1745424130 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d300a8ad0 con 0x7f8d401538a0 2023-10-30T13:28:58.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.561+0000 7f8d3e7fc640 1 -- 172.21.15.46:0/1745424130 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d300a8e60 con 0x7f8d401538a0 2023-10-30T13:28:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.561+0000 7f8d45e9c640 1 -- 172.21.15.46:0/1745424130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d401538a0 msgr2=0x7f8d4015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.561+0000 7f8d45e9c640 1 --2- 172.21.15.46:0/1745424130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d401538a0 0x7f8d4015d6f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8d300097c0 tx=0x7f8d3001ec70 comp rx=0 tx=0).stop 2023-10-30T13:28:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.561+0000 7f8d45e9c640 1 -- 172.21.15.46:0/1745424130 shutdown_connections 2023-10-30T13:28:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.561+0000 7f8d45e9c640 1 --2- 172.21.15.46:0/1745424130 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d400ca0e0 0x7f8d4015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.561+0000 7f8d45e9c640 1 --2- 172.21.15.46:0/1745424130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d401538a0 0x7f8d4015d6f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.561+0000 7f8d45e9c640 1 -- 172.21.15.46:0/1745424130 >> 172.21.15.46:0/1745424130 conn(0x7f8d4006bfe0 msgr2=0x7f8d4006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.561+0000 7f8d45e9c640 1 -- 172.21.15.46:0/1745424130 shutdown_connections 2023-10-30T13:28:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.561+0000 7f8d45e9c640 1 -- 172.21.15.46:0/1745424130 wait complete. 2023-10-30T13:28:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d45e9c640 1 Processor -- start 2023-10-30T13:28:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d45e9c640 1 -- start start 2023-10-30T13:28:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d45e9c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d400ca0e0 0x7f8d40146e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:58.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d45e9c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d400cf160 0x7f8d4014e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:58.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d45e9c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d401538a0 0x7f8d4014eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:58.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d45e9c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d40158a50 con 0x7f8d400ca0e0 2023-10-30T13:28:58.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d45e9c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400cebb0 con 0x7f8d400cf160 2023-10-30T13:28:58.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d45e9c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400cee70 con 0x7f8d401538a0 2023-10-30T13:28:58.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d3f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d400ca0e0 0x7f8d40146e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:58.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d3f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d400ca0e0 0x7f8d40146e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33980/0 (socket says 172.21.15.46:33980) 2023-10-30T13:28:58.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d3f7fe640 1 -- 172.21.15.46:0/773159052 learned_addr learned my addr 172.21.15.46:0/773159052 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:28:58.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d3ffff640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d401538a0 0x7f8d4014eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:58.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d3effd640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d400cf160 0x7f8d4014e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:58.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d3f7fe640 1 -- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d401538a0 msgr2=0x7f8d4014eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:58.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d3f7fe640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d401538a0 0x7f8d4014eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:58.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d3f7fe640 1 -- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d400cf160 msgr2=0x7f8d4014e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:58.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d3f7fe640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d400cf160 0x7f8d4014e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:58.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d3f7fe640 1 -- 172.21.15.46:0/773159052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d30009420 con 0x7f8d400ca0e0 2023-10-30T13:28:58.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d3f7fe640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d400ca0e0 0x7f8d40146e60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8d300b3040 tx=0x7f8d3001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:58.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d27fff640 1 -- 172.21.15.46:0/773159052 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d300b9630 con 0x7f8d400ca0e0 2023-10-30T13:28:58.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d27fff640 1 -- 172.21.15.46:0/773159052 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d300b97b0 con 0x7f8d400ca0e0 2023-10-30T13:28:58.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.565+0000 7f8d27fff640 1 -- 172.21.15.46:0/773159052 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d300b9b60 con 0x7f8d400ca0e0 2023-10-30T13:28:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.569+0000 7f8d45e9c640 1 -- 172.21.15.46:0/773159052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d40157380 con 0x7f8d400ca0e0 2023-10-30T13:28:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.569+0000 7f8d45e9c640 1 -- 172.21.15.46:0/773159052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d40157940 con 0x7f8d400ca0e0 2023-10-30T13:28:58.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.569+0000 7f8d45e9c640 1 -- 172.21.15.46:0/773159052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d400b2a50 con 0x7f8d400ca0e0 2023-10-30T13:28:58.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.569+0000 7f8d27fff640 1 -- 172.21.15.46:0/773159052 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8d300b7040 con 0x7f8d400ca0e0 2023-10-30T13:28:58.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.569+0000 7f8d27fff640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d2c0664a0 0x7f8d2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.569+0000 7f8d27fff640 1 -- 172.21.15.46:0/773159052 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f8d30122170 con 0x7f8d400ca0e0 2023-10-30T13:28:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.569+0000 7f8d3effd640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d2c0664a0 0x7f8d2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.573+0000 7f8d3effd640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d2c0664a0 0x7f8d2c068960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8d40146bd0 tx=0x7f8d34008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.573+0000 7f8d27fff640 1 -- 172.21.15.46:0/773159052 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8d300ef2a0 con 0x7f8d400ca0e0 2023-10-30T13:28:58.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.761+0000 7f8d45e9c640 1 -- 172.21.15.46:0/773159052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d401f66e0 con 0x7f8d400ca0e0 2023-10-30T13:28:58.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d27fff640 1 -- 172.21.15.46:0/773159052 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f8d300f28c0 con 0x7f8d400ca0e0 2023-10-30T13:28:58.770 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:28:58.770 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:28:00.722346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:58.772 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 28 2023-10-30T13:28:58.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d45e9c640 1 -- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d2c0664a0 msgr2=0x7f8d2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:58.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d45e9c640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d2c0664a0 0x7f8d2c068960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8d40146bd0 tx=0x7f8d34008040 comp rx=0 tx=0).stop 2023-10-30T13:28:58.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d45e9c640 1 -- 172.21.15.46:0/773159052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d400ca0e0 msgr2=0x7f8d40146e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:58.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d45e9c640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d400ca0e0 0x7f8d40146e60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8d300b3040 tx=0x7f8d3001ba10 comp rx=0 tx=0).stop 2023-10-30T13:28:58.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d45e9c640 1 -- 172.21.15.46:0/773159052 shutdown_connections 2023-10-30T13:28:58.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d45e9c640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d2c0664a0 0x7f8d2c068960 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:58.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d45e9c640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d401538a0 0x7f8d4014eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:58.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d45e9c640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d400cf160 0x7f8d4014e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:58.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d45e9c640 1 --2- 172.21.15.46:0/773159052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d400ca0e0 0x7f8d40146e60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:58.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d45e9c640 1 -- 172.21.15.46:0/773159052 >> 172.21.15.46:0/773159052 conn(0x7f8d4006bfe0 msgr2=0x7f8d401527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:58.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d45e9c640 1 -- 172.21.15.46:0/773159052 shutdown_connections 2023-10-30T13:28:58.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:28:58.765+0000 7f8d45e9c640 1 -- 172.21.15.46:0/773159052 wait complete. 2023-10-30T13:28:58.792 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:29:00.793 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:00.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.941+0000 7f7c28594640 1 Processor -- start 2023-10-30T13:29:00.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.941+0000 7f7c28594640 1 -- start start 2023-10-30T13:29:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.945+0000 7f7c28594640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c2006f7a0 0x7f7c2006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:00.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.945+0000 7f7c28594640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c20066cf0 con 0x7f7c20071110 2023-10-30T13:29:00.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.945+0000 7f7c28594640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c200c70d0 con 0x7f7c200c6bd0 2023-10-30T13:29:00.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.945+0000 7f7c28594640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c200700c0 con 0x7f7c2006f7a0 2023-10-30T13:29:00.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.945+0000 7f7c26330640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c2006f7a0 0x7f7c2006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:00.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.945+0000 7f7c26b31640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7c20071110 0x7f7c20158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59088/0 (socket says 172.21.15.46:59088) 2023-10-30T13:29:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.945+0000 7f7c26b31640 1 -- 172.21.15.46:0/3609885564 learned_addr learned my addr 172.21.15.46:0/3609885564 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.945+0000 7f7c2532e640 1 -- 172.21.15.46:0/3609885564 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730105802 0 0) 0x7f7c20066cf0 con 0x7f7c20071110 2023-10-30T13:29:00.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.945+0000 7f7c2532e640 1 -- 172.21.15.46:0/3609885564 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c0c003530 con 0x7f7c20071110 2023-10-30T13:29:00.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.945+0000 7f7c26330640 1 -- 172.21.15.46:0/3609885564 >> v1:172.21.15.154:6789/0 conn(0x7f7c200c6bd0 legacy=0x7f7c20069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:00.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.945+0000 7f7c26330640 1 -- 172.21.15.46:0/3609885564 >> v1:172.21.15.46:6789/0 conn(0x7f7c20071110 legacy=0x7f7c20158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:00.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.945+0000 7f7c26330640 1 -- 172.21.15.46:0/3609885564 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c200c7390 con 0x7f7c2006f7a0 2023-10-30T13:29:00.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c26330640 1 --2- 172.21.15.46:0/3609885564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2006f7a0 0x7f7c2006fb80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0097c0 tx=0x7f7c1c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d24137a42676b95 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:00.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c2532e640 1 -- 172.21.15.46:0/3609885564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c0a8970 con 0x7f7c2006f7a0 2023-10-30T13:29:00.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c2532e640 1 -- 172.21.15.46:0/3609885564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c0a8ad0 con 0x7f7c2006f7a0 2023-10-30T13:29:00.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c2532e640 1 -- 172.21.15.46:0/3609885564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c0a8e60 con 0x7f7c2006f7a0 2023-10-30T13:29:00.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c28594640 1 -- 172.21.15.46:0/3609885564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2006f7a0 msgr2=0x7f7c2006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:00.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c28594640 1 --2- 172.21.15.46:0/3609885564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2006f7a0 0x7f7c2006fb80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0097c0 tx=0x7f7c1c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:29:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c28594640 1 -- 172.21.15.46:0/3609885564 shutdown_connections 2023-10-30T13:29:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c28594640 1 --2- 172.21.15.46:0/3609885564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2006f7a0 0x7f7c2006fb80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c28594640 1 -- 172.21.15.46:0/3609885564 >> 172.21.15.46:0/3609885564 conn(0x7f7c200c7910 msgr2=0x7f7c200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c28594640 1 -- 172.21.15.46:0/3609885564 shutdown_connections 2023-10-30T13:29:00.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c28594640 1 -- 172.21.15.46:0/3609885564 wait complete. 2023-10-30T13:29:00.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c28594640 1 Processor -- start 2023-10-30T13:29:00.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.949+0000 7f7c28594640 1 -- start start 2023-10-30T13:29:00.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c28594640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c2006f7a0 0x7f7c20155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c28594640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c20071110 0x7f7c200c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c28594640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200c6bd0 0x7f7c201571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c28594640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20156200 con 0x7f7c20071110 2023-10-30T13:29:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c28594640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c200c15b0 con 0x7f7c200c6bd0 2023-10-30T13:29:00.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c28594640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20156550 con 0x7f7c2006f7a0 2023-10-30T13:29:00.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c25b2f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c20071110 0x7f7c200c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:00.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c25b2f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c20071110 0x7f7c200c37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33998/0 (socket says 172.21.15.46:33998) 2023-10-30T13:29:00.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c25b2f640 1 -- 172.21.15.46:0/3545491111 learned_addr learned my addr 172.21.15.46:0/3545491111 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:00.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c26330640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c2006f7a0 0x7f7c20155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:00.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c26b31640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200c6bd0 0x7f7c201571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:00.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c25b2f640 1 -- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c2006f7a0 msgr2=0x7f7c20155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:00.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c25b2f640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c2006f7a0 0x7f7c20155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:00.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c25b2f640 1 -- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200c6bd0 msgr2=0x7f7c201571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:00.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c25b2f640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200c6bd0 0x7f7c201571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:00.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c25b2f640 1 -- 172.21.15.46:0/3545491111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c009420 con 0x7f7c20071110 2023-10-30T13:29:00.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c25b2f640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c20071110 0x7f7c200c37d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000c4d0 tx=0x7f7c1000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:00.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c16ffd640 1 -- 172.21.15.46:0/3545491111 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c100052d0 con 0x7f7c20071110 2023-10-30T13:29:00.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c16ffd640 1 -- 172.21.15.46:0/3545491111 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c10005430 con 0x7f7c20071110 2023-10-30T13:29:00.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c16ffd640 1 -- 172.21.15.46:0/3545491111 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10012830 con 0x7f7c20071110 2023-10-30T13:29:00.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c28594640 1 -- 172.21.15.46:0/3545491111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c200c0d10 con 0x7f7c20071110 2023-10-30T13:29:00.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c28594640 1 -- 172.21.15.46:0/3545491111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c200c12d0 con 0x7f7c20071110 2023-10-30T13:29:00.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.953+0000 7f7c28594640 1 -- 172.21.15.46:0/3545491111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2004e170 con 0x7f7c20071110 2023-10-30T13:29:00.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.957+0000 7f7c16ffd640 1 -- 172.21.15.46:0/3545491111 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7c10012990 con 0x7f7c20071110 2023-10-30T13:29:00.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.957+0000 7f7c16ffd640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7c040664d0 0x7f7c04068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:00.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.957+0000 7f7c16ffd640 1 -- 172.21.15.46:0/3545491111 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f7c10083d20 con 0x7f7c20071110 2023-10-30T13:29:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.957+0000 7f7c26330640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7c040664d0 0x7f7c04068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.957+0000 7f7c26330640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7c040664d0 0x7f7c04068990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0b2000 tx=0x7f7c1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:00.961+0000 7f7c16ffd640 1 -- 172.21.15.46:0/3545491111 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7c10050dc0 con 0x7f7c20071110 2023-10-30T13:29:01.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 -- 172.21.15.46:0/3545491111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c200c8100 con 0x7f7c20071110 2023-10-30T13:29:01.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c16ffd640 1 -- 172.21.15.46:0/3545491111 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f7c100543e0 con 0x7f7c20071110 2023-10-30T13:29:01.159 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:01.159 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:28:00.722346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:01.159 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 28 2023-10-30T13:29:01.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 -- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7c040664d0 msgr2=0x7f7c04068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7c040664d0 0x7f7c04068990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0b2000 tx=0x7f7c1c008040 comp rx=0 tx=0).stop 2023-10-30T13:29:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 -- 172.21.15.46:0/3545491111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c20071110 msgr2=0x7f7c200c37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c20071110 0x7f7c200c37d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000c4d0 tx=0x7f7c1000c990 comp rx=0 tx=0).stop 2023-10-30T13:29:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 -- 172.21.15.46:0/3545491111 shutdown_connections 2023-10-30T13:29:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7c040664d0 0x7f7c04068990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:01.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200c6bd0 0x7f7c201571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:01.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c20071110 0x7f7c200c37d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:01.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 --2- 172.21.15.46:0/3545491111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c2006f7a0 0x7f7c20155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:01.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 -- 172.21.15.46:0/3545491111 >> 172.21.15.46:0/3545491111 conn(0x7f7c200c7910 msgr2=0x7f7c200c92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:01.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 -- 172.21.15.46:0/3545491111 shutdown_connections 2023-10-30T13:29:01.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:01.153+0000 7f7c28594640 1 -- 172.21.15.46:0/3545491111 wait complete. 2023-10-30T13:29:01.180 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:29:03.181 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.329+0000 7f5dcf009640 1 Processor -- start 2023-10-30T13:29:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.329+0000 7f5dcf009640 1 -- start start 2023-10-30T13:29:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dcf009640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5dc81538a0 0x7f5dc815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dcf009640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dc80cf160 0x7f5dc80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dcf009640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc80cb800 con 0x7f5dc80ca0e0 2023-10-30T13:29:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dcf009640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc815dc30 con 0x7f5dc81538a0 2023-10-30T13:29:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dcf009640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc80cb330 con 0x7f5dc80cf160 2023-10-30T13:29:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5dc80ca0e0 0x7f5dc815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59098/0 (socket says 172.21.15.46:59098) 2023-10-30T13:29:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc7fff640 1 -- 172.21.15.46:0/2861561550 learned_addr learned my addr 172.21.15.46:0/2861561550 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dccda5640 1 --2- 172.21.15.46:0/2861561550 >> v2:172.21.15.154:3300/0 conn(0x7f5dc81538a0 0x7f5dc815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dcd5a6640 1 --2- 172.21.15.46:0/2861561550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dc80cf160 0x7f5dc80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622463293 0 0) 0x7f5dc80cb800 con 0x7f5dc80ca0e0 2023-10-30T13:29:03.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db8003530 con 0x7f5dc80ca0e0 2023-10-30T13:29:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3601946235 0 0) 0x7f5db8003530 con 0x7f5dc80ca0e0 2023-10-30T13:29:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dc80cf160 msgr2=0x7f5dc80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 --2- 172.21.15.46:0/2861561550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dc80cf160 0x7f5dc80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 >> v2:172.21.15.154:3300/0 conn(0x7f5dc81538a0 msgr2=0x7f5dc815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 --2- 172.21.15.46:0/2861561550 >> v2:172.21.15.154:3300/0 conn(0x7f5dc81538a0 0x7f5dc815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc80cad70 con 0x7f5dc80ca0e0 2023-10-30T13:29:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f5db4001680 con 0x7f5dc80ca0e0 2023-10-30T13:29:03.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 >> v1:172.21.15.46:6789/0 conn(0x7f5dc80ca0e0 legacy=0x7f5dc815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:03.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 --2- 172.21.15.46:0/2861561550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db8003980 0x7f5db8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:03.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 --2- 172.21.15.46:0/2861561550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dc80cf160 0x7f5db80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:03.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 --2- 172.21.15.46:0/2861561550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8004d10 0x7f5db8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:03.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc815dc30 con 0x7f5db8004d10 2023-10-30T13:29:03.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc80cb330 con 0x7f5db8003980 2023-10-30T13:29:03.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db800b990 con 0x7f5dc80cf160 2023-10-30T13:29:03.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dcd5a6640 1 --2- 172.21.15.46:0/2861561550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8004d10 0x7f5db8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:03.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dccda5640 1 --2- 172.21.15.46:0/2861561550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db8003980 0x7f5db8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:03.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dc7fff640 1 --2- 172.21.15.46:0/2861561550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dc80cf160 0x7f5db80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:03.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dcd5a6640 1 -- 172.21.15.46:0/2861561550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dc80cf160 msgr2=0x7f5db80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:03.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dcd5a6640 1 --2- 172.21.15.46:0/2861561550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dc80cf160 0x7f5db80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:03.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dcd5a6640 1 -- 172.21.15.46:0/2861561550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db8003980 msgr2=0x7f5db8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:03.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dcd5a6640 1 --2- 172.21.15.46:0/2861561550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db8003980 0x7f5db8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:03.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.333+0000 7f5dcd5a6640 1 -- 172.21.15.46:0/2861561550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db8003530 con 0x7f5db8004d10 2023-10-30T13:29:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dcd5a6640 1 --2- 172.21.15.46:0/2861561550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8004d10 0x7f5db8005190 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009fc0 tx=0x7f5dc001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca9e09237084e17 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc00a5c70 con 0x7f5db8004d10 2023-10-30T13:29:03.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc000f040 con 0x7f5db8004d10 2023-10-30T13:29:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dc77fe640 1 -- 172.21.15.46:0/2861561550 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc00ad440 con 0x7f5db8004d10 2023-10-30T13:29:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dcf009640 1 -- 172.21.15.46:0/2861561550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8004d10 msgr2=0x7f5db8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dcf009640 1 --2- 172.21.15.46:0/2861561550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8004d10 0x7f5db8005190 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009fc0 tx=0x7f5dc001cc60 comp rx=0 tx=0).stop 2023-10-30T13:29:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dcd5a6640 1 -- 172.21.15.46:0/2861561550 reap_dead start 2023-10-30T13:29:03.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dcf009640 1 -- 172.21.15.46:0/2861561550 shutdown_connections 2023-10-30T13:29:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dcf009640 1 -- 172.21.15.46:0/2861561550 >> 172.21.15.46:0/2861561550 conn(0x7f5dc806bfe0 msgr2=0x7f5dc806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dcf009640 1 -- 172.21.15.46:0/2861561550 shutdown_connections 2023-10-30T13:29:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dcf009640 1 -- 172.21.15.46:0/2861561550 wait complete. 2023-10-30T13:29:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dcf009640 1 Processor -- start 2023-10-30T13:29:03.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.337+0000 7f5dcf009640 1 -- start start 2023-10-30T13:29:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcf009640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5db8004d10 0x7f5dc815b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcf009640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5dc8145960 0x7f5dc8145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcf009640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dc81f29e0 0x7f5dc81f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcf009640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc815d280 con 0x7f5dc81f29e0 2023-10-30T13:29:03.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcf009640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc8159620 con 0x7f5dc8145960 2023-10-30T13:29:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcf009640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc815d520 con 0x7f5db8004d10 2023-10-30T13:29:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcd5a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dc81f29e0 0x7f5dc81f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcd5a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dc81f29e0 0x7f5dc81f2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34018/0 (socket says 172.21.15.46:34018) 2023-10-30T13:29:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcd5a6640 1 -- 172.21.15.46:0/886084967 learned_addr learned my addr 172.21.15.46:0/886084967 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dccda5640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5db8004d10 0x7f5dc815b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dc7fff640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5dc8145960 0x7f5dc8145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcd5a6640 1 -- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5db8004d10 msgr2=0x7f5dc815b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcd5a6640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5db8004d10 0x7f5dc815b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcd5a6640 1 -- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5dc8145960 msgr2=0x7f5dc8145dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:03.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcd5a6640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5dc8145960 0x7f5dc8145dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcd5a6640 1 -- 172.21.15.46:0/886084967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc815a8b0 con 0x7f5dc81f29e0 2023-10-30T13:29:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dccda5640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5db8004d10 0x7f5dc815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dc7fff640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5dc8145960 0x7f5dc8145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcd5a6640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dc81f29e0 0x7f5dc81f2e60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00b0040 tx=0x7f5dc0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:03.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dc57fa640 1 -- 172.21.15.46:0/886084967 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc00039a0 con 0x7f5dc81f29e0 2023-10-30T13:29:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dc57fa640 1 -- 172.21.15.46:0/886084967 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc0003b00 con 0x7f5dc81f29e0 2023-10-30T13:29:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dc57fa640 1 -- 172.21.15.46:0/886084967 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0003e90 con 0x7f5dc81f29e0 2023-10-30T13:29:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcf009640 1 -- 172.21.15.46:0/886084967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc815ab90 con 0x7f5dc81f29e0 2023-10-30T13:29:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcf009640 1 -- 172.21.15.46:0/886084967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc815b150 con 0x7f5dc81f29e0 2023-10-30T13:29:03.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.341+0000 7f5dcf009640 1 -- 172.21.15.46:0/886084967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc804e170 con 0x7f5dc81f29e0 2023-10-30T13:29:03.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.345+0000 7f5dc57fa640 1 -- 172.21.15.46:0/886084967 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5dc00b4040 con 0x7f5dc81f29e0 2023-10-30T13:29:03.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.345+0000 7f5dc57fa640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5dac0664d0 0x7f5dac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:03.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.345+0000 7f5dc57fa640 1 -- 172.21.15.46:0/886084967 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f5dc011e160 con 0x7f5dc81f29e0 2023-10-30T13:29:03.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.345+0000 7f5dccda5640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5dac0664d0 0x7f5dac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:03.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.345+0000 7f5dccda5640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5dac0664d0 0x7f5dac068990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009c60 tx=0x7f5dbc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:03.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.349+0000 7f5dc57fa640 1 -- 172.21.15.46:0/886084967 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5dc00eb190 con 0x7f5dc81f29e0 2023-10-30T13:29:03.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.537+0000 7f5dcf009640 1 -- 172.21.15.46:0/886084967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dc815d280 con 0x7f5dc81f29e0 2023-10-30T13:29:03.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.537+0000 7f5dc57fa640 1 -- 172.21.15.46:0/886084967 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f5dc00be090 con 0x7f5dc81f29e0 2023-10-30T13:29:03.545 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:03.545 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:28:00.722346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:03.545 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 28 2023-10-30T13:29:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.541+0000 7f5dcf009640 1 -- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5dac0664d0 msgr2=0x7f5dac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.541+0000 7f5dcf009640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5dac0664d0 0x7f5dac068990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009c60 tx=0x7f5dbc008040 comp rx=0 tx=0).stop 2023-10-30T13:29:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.541+0000 7f5dcf009640 1 -- 172.21.15.46:0/886084967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dc81f29e0 msgr2=0x7f5dc81f2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.541+0000 7f5dcf009640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dc81f29e0 0x7f5dc81f2e60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00b0040 tx=0x7f5dc0009750 comp rx=0 tx=0).stop 2023-10-30T13:29:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.541+0000 7f5dcf009640 1 -- 172.21.15.46:0/886084967 shutdown_connections 2023-10-30T13:29:03.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.541+0000 7f5dcf009640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5dac0664d0 0x7f5dac068990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:03.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.541+0000 7f5dcf009640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dc81f29e0 0x7f5dc81f2e60 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:03.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.541+0000 7f5dcf009640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5dc8145960 0x7f5dc8145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:03.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.541+0000 7f5dcf009640 1 --2- 172.21.15.46:0/886084967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5db8004d10 0x7f5dc815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:03.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.541+0000 7f5dcf009640 1 -- 172.21.15.46:0/886084967 >> 172.21.15.46:0/886084967 conn(0x7f5dc806bfe0 msgr2=0x7f5dc8149410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:03.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.541+0000 7f5dcf009640 1 -- 172.21.15.46:0/886084967 shutdown_connections 2023-10-30T13:29:03.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:03.541+0000 7f5dcf009640 1 -- 172.21.15.46:0/886084967 wait complete. 2023-10-30T13:29:03.566 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:29:05.567 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:05.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff27f4bb640 1 Processor -- start 2023-10-30T13:29:05.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff27f4bb640 1 -- start start 2023-10-30T13:29:05.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff27f4bb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff27815d6f0 0x7ff27815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:05.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff27f4bb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff2780cf160 0x7ff2780cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:05.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff27f4bb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2780cb800 con 0x7ff2781538a0 2023-10-30T13:29:05.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff27f4bb640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2780cb330 con 0x7ff2780cf160 2023-10-30T13:29:05.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff27f4bb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780cad70 con 0x7ff27815d6f0 2023-10-30T13:29:05.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff27d257640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff2781538a0 0x7ff2780ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59112/0 (socket says 172.21.15.46:59112) 2023-10-30T13:29:05.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff27d257640 1 -- 172.21.15.46:0/3438401215 learned_addr learned my addr 172.21.15.46:0/3438401215 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:05.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff27da58640 1 --2- 172.21.15.46:0/3438401215 >> v2:172.21.15.154:3300/0 conn(0x7ff2780cf160 0x7ff2780cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:05.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff27ca56640 1 --2- 172.21.15.46:0/3438401215 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff27815d6f0 0x7ff27815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:05.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184980070 0 0) 0x7ff2780cb800 con 0x7ff2781538a0 2023-10-30T13:29:05.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff264003530 con 0x7ff2781538a0 2023-10-30T13:29:05.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2357219768 0 0) 0x7ff264003530 con 0x7ff2781538a0 2023-10-30T13:29:05.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff27815d6f0 msgr2=0x7ff27815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:05.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 --2- 172.21.15.46:0/3438401215 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff27815d6f0 0x7ff27815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:05.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 >> v2:172.21.15.154:3300/0 conn(0x7ff2780cf160 msgr2=0x7ff2780cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:05.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 --2- 172.21.15.46:0/3438401215 >> v2:172.21.15.154:3300/0 conn(0x7ff2780cf160 0x7ff2780cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:05.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2780cb050 con 0x7ff2781538a0 2023-10-30T13:29:05.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7ff274001680 con 0x7ff2781538a0 2023-10-30T13:29:05.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 >> v1:172.21.15.46:6789/0 conn(0x7ff2781538a0 legacy=0x7ff2780ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:05.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 --2- 172.21.15.46:0/3438401215 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff27815d6f0 0x7ff264003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:05.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 --2- 172.21.15.46:0/3438401215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff264003f00 0x7ff2640047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:05.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 --2- 172.21.15.46:0/3438401215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff264004d10 0x7ff264005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:05.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780cb330 con 0x7ff264004d10 2023-10-30T13:29:05.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780cad70 con 0x7ff264003f00 2023-10-30T13:29:05.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.717+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26400b9c0 con 0x7ff27815d6f0 2023-10-30T13:29:05.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff27da58640 1 --2- 172.21.15.46:0/3438401215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff264004d10 0x7ff264005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:05.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff27d257640 1 --2- 172.21.15.46:0/3438401215 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff27815d6f0 0x7ff264003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:05.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff27ca56640 1 --2- 172.21.15.46:0/3438401215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff264003f00 0x7ff2640047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:05.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff27da58640 1 -- 172.21.15.46:0/3438401215 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff27815d6f0 msgr2=0x7ff264003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:05.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff27da58640 1 --2- 172.21.15.46:0/3438401215 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff27815d6f0 0x7ff264003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:05.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff27da58640 1 -- 172.21.15.46:0/3438401215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff264003f00 msgr2=0x7ff2640047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:05.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff27da58640 1 --2- 172.21.15.46:0/3438401215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff264003f00 0x7ff2640047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:05.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff27da58640 1 -- 172.21.15.46:0/3438401215 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff264003530 con 0x7ff264004d10 2023-10-30T13:29:05.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff27da58640 1 --2- 172.21.15.46:0/3438401215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff264004d10 0x7ff264005190 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff270009fc0 tx=0x7ff27001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451e98eca31a875c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:05.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2700a5cb0 con 0x7ff264004d10 2023-10-30T13:29:05.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27000f040 con 0x7ff264004d10 2023-10-30T13:29:05.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff26ffff640 1 -- 172.21.15.46:0/3438401215 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2700ad680 con 0x7ff264004d10 2023-10-30T13:29:05.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff27f4bb640 1 -- 172.21.15.46:0/3438401215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff264004d10 msgr2=0x7ff264005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:05.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff27f4bb640 1 --2- 172.21.15.46:0/3438401215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff264004d10 0x7ff264005190 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff270009fc0 tx=0x7ff27001cca0 comp rx=0 tx=0).stop 2023-10-30T13:29:05.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.721+0000 7ff27da58640 1 -- 172.21.15.46:0/3438401215 reap_dead start 2023-10-30T13:29:05.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27f4bb640 1 -- 172.21.15.46:0/3438401215 shutdown_connections 2023-10-30T13:29:05.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27f4bb640 1 -- 172.21.15.46:0/3438401215 >> 172.21.15.46:0/3438401215 conn(0x7ff27806bfe0 msgr2=0x7ff27806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:05.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27f4bb640 1 -- 172.21.15.46:0/3438401215 shutdown_connections 2023-10-30T13:29:05.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27f4bb640 1 -- 172.21.15.46:0/3438401215 wait complete. 2023-10-30T13:29:05.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27f4bb640 1 Processor -- start 2023-10-30T13:29:05.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27f4bb640 1 -- start start 2023-10-30T13:29:05.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27f4bb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff264004d10 0x7ff278145b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:05.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27f4bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2780b3110 0x7ff2780b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:05.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27f4bb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff278158020 0x7ff2781590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:05.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27f4bb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff278144ef0 con 0x7ff2780b3110 2023-10-30T13:29:05.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27f4bb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2781462c0 con 0x7ff278158020 2023-10-30T13:29:05.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27f4bb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff278158560 con 0x7ff264004d10 2023-10-30T13:29:05.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27ca56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2780b3110 0x7ff2780b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:05.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27ca56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2780b3110 0x7ff2780b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34048/0 (socket says 172.21.15.46:34048) 2023-10-30T13:29:05.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27ca56640 1 -- 172.21.15.46:0/880390883 learned_addr learned my addr 172.21.15.46:0/880390883 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:05.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27d257640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff264004d10 0x7ff278145b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:05.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27da58640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff278158020 0x7ff2781590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:05.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27ca56640 1 -- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff264004d10 msgr2=0x7ff278145b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:05.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27ca56640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff264004d10 0x7ff278145b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:05.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27ca56640 1 -- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff278158020 msgr2=0x7ff2781590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:05.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27ca56640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff278158020 0x7ff2781590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:05.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27ca56640 1 -- 172.21.15.46:0/880390883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff278159630 con 0x7ff2780b3110 2023-10-30T13:29:05.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.725+0000 7ff27ca56640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2780b3110 0x7ff2780b4af0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff268011560 tx=0x7ff268011960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:05.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.729+0000 7ff26dffb640 1 -- 172.21.15.46:0/880390883 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268015930 con 0x7ff2780b3110 2023-10-30T13:29:05.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.729+0000 7ff26dffb640 1 -- 172.21.15.46:0/880390883 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff268015a90 con 0x7ff2780b3110 2023-10-30T13:29:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.729+0000 7ff26dffb640 1 -- 172.21.15.46:0/880390883 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268015e20 con 0x7ff2780b3110 2023-10-30T13:29:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.729+0000 7ff27f4bb640 1 -- 172.21.15.46:0/880390883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2780b4530 con 0x7ff2780b3110 2023-10-30T13:29:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.729+0000 7ff27f4bb640 1 -- 172.21.15.46:0/880390883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2781448a0 con 0x7ff2780b3110 2023-10-30T13:29:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.729+0000 7ff27f4bb640 1 -- 172.21.15.46:0/880390883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27804e170 con 0x7ff2780b3110 2023-10-30T13:29:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.729+0000 7ff26dffb640 1 -- 172.21.15.46:0/880390883 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff26801d400 con 0x7ff2780b3110 2023-10-30T13:29:05.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.729+0000 7ff26dffb640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff25c0664a0 0x7ff25c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:05.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.729+0000 7ff26dffb640 1 -- 172.21.15.46:0/880390883 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6678+0+0 (secure 0 0 0) 0x7ff268084980 con 0x7ff2780b3110 2023-10-30T13:29:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.729+0000 7ff27d257640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff25c0664a0 0x7ff25c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.733+0000 7ff27d257640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff25c0664a0 0x7ff25c068960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff274001ec0 tx=0x7ff274007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.733+0000 7ff26dffb640 1 -- 172.21.15.46:0/880390883 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff268051a20 con 0x7ff2780b3110 2023-10-30T13:29:05.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.921+0000 7ff27f4bb640 1 -- 172.21.15.46:0/880390883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2780c5c30 con 0x7ff2780b3110 2023-10-30T13:29:05.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff26dffb640 1 -- 172.21.15.46:0/880390883 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6531 (secure 0 0 0) 0x7ff268082e60 con 0x7ff2780b3110 2023-10-30T13:29:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:05.930 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:28:00.722346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:05.930 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 28 2023-10-30T13:29:05.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff27f4bb640 1 -- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff25c0664a0 msgr2=0x7ff25c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:05.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff27f4bb640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff25c0664a0 0x7ff25c068960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff274001ec0 tx=0x7ff274007040 comp rx=0 tx=0).stop 2023-10-30T13:29:05.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff27f4bb640 1 -- 172.21.15.46:0/880390883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2780b3110 msgr2=0x7ff2780b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:05.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff27f4bb640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2780b3110 0x7ff2780b4af0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff268011560 tx=0x7ff268011960 comp rx=0 tx=0).stop 2023-10-30T13:29:05.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff27f4bb640 1 -- 172.21.15.46:0/880390883 shutdown_connections 2023-10-30T13:29:05.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff27f4bb640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff25c0664a0 0x7ff25c068960 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:05.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff27f4bb640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff278158020 0x7ff2781590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:05.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff27f4bb640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2780b3110 0x7ff2780b4af0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:05.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff27f4bb640 1 --2- 172.21.15.46:0/880390883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff264004d10 0x7ff278145b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:05.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff27f4bb640 1 -- 172.21.15.46:0/880390883 >> 172.21.15.46:0/880390883 conn(0x7ff27806bfe0 msgr2=0x7ff2780cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:05.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff27f4bb640 1 -- 172.21.15.46:0/880390883 shutdown_connections 2023-10-30T13:29:05.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:05.925+0000 7ff27f4bb640 1 -- 172.21.15.46:0/880390883 wait complete. 2023-10-30T13:29:05.951 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:29:07.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.101+0000 7f72b86ea640 1 Processor -- start 2023-10-30T13:29:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.101+0000 7f72b86ea640 1 -- start start 2023-10-30T13:29:08.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b86ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72b00c6bd0 0x7f72b006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b86ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72b0071110 0x7f72b0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b86ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b0066cf0 con 0x7f72b006f7a0 2023-10-30T13:29:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b86ea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b00700c0 con 0x7f72b00c6bd0 2023-10-30T13:29:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b86ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b00c70d0 con 0x7f72b0071110 2023-10-30T13:29:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b6486640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f72b006f7a0 0x7f72b0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50306/0 (socket says 172.21.15.46:50306) 2023-10-30T13:29:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b5c85640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72b00c6bd0 0x7f72b006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b6486640 1 -- 172.21.15.46:0/2321366220 learned_addr learned my addr 172.21.15.46:0/2321366220 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b6c87640 1 --2- 172.21.15.46:0/2321366220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72b0071110 0x7f72b0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b5c85640 1 -- 172.21.15.46:0/2321366220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72b0071110 msgr2=0x7f72b0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b5c85640 1 --2- 172.21.15.46:0/2321366220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72b0071110 0x7f72b0158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b5c85640 1 -- 172.21.15.46:0/2321366220 >> v1:172.21.15.46:6789/0 conn(0x7f72b006f7a0 legacy=0x7f72b0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b5c85640 1 -- 172.21.15.46:0/2321366220 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b00c7390 con 0x7f72b00c6bd0 2023-10-30T13:29:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b5484640 1 -- 172.21.15.46:0/2321366220 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699930720 0 0) 0x7f72b0066cf0 con 0x7f72b006f7a0 2023-10-30T13:29:08.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b5c85640 1 --2- 172.21.15.46:0/2321366220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b00c6bd0 0x7f72b006fb80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f72a00097c0 tx=0x7f72a001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f1f43729ec155a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.105+0000 7f72b5484640 1 -- 172.21.15.46:0/2321366220 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a00a8970 con 0x7f72b00c6bd0 2023-10-30T13:29:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b5484640 1 -- 172.21.15.46:0/2321366220 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a00a8ad0 con 0x7f72b00c6bd0 2023-10-30T13:29:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b5484640 1 -- 172.21.15.46:0/2321366220 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a00a8e60 con 0x7f72b00c6bd0 2023-10-30T13:29:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 -- 172.21.15.46:0/2321366220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b00c6bd0 msgr2=0x7f72b006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 --2- 172.21.15.46:0/2321366220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b00c6bd0 0x7f72b006fb80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f72a00097c0 tx=0x7f72a001ec70 comp rx=0 tx=0).stop 2023-10-30T13:29:08.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 -- 172.21.15.46:0/2321366220 shutdown_connections 2023-10-30T13:29:08.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 --2- 172.21.15.46:0/2321366220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72b0071110 0x7f72b0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:08.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 --2- 172.21.15.46:0/2321366220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b00c6bd0 0x7f72b006fb80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:08.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 -- 172.21.15.46:0/2321366220 >> 172.21.15.46:0/2321366220 conn(0x7f72b00c7910 msgr2=0x7f72b00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:08.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 -- 172.21.15.46:0/2321366220 shutdown_connections 2023-10-30T13:29:08.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 -- 172.21.15.46:0/2321366220 wait complete. 2023-10-30T13:29:08.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 Processor -- start 2023-10-30T13:29:08.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 -- start start 2023-10-30T13:29:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72b006f7a0 0x7f72b00c0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b0071110 0x7f72b014f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72b00c6bd0 0x7f72b014fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b00c9460 con 0x7f72b0071110 2023-10-30T13:29:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b01ef330 con 0x7f72b006f7a0 2023-10-30T13:29:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b86ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b01e4170 con 0x7f72b00c6bd0 2023-10-30T13:29:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b5c85640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b0071110 0x7f72b014f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b5c85640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b0071110 0x7f72b014f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41848/0 (socket says 172.21.15.46:41848) 2023-10-30T13:29:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.109+0000 7f72b5c85640 1 -- 172.21.15.46:0/706415646 learned_addr learned my addr 172.21.15.46:0/706415646 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72b6c87640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72b00c6bd0 0x7f72b014fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72b6486640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72b006f7a0 0x7f72b00c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72b5c85640 1 -- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72b00c6bd0 msgr2=0x7f72b014fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72b5c85640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72b00c6bd0 0x7f72b014fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:08.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72b5c85640 1 -- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72b006f7a0 msgr2=0x7f72b00c0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:08.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72b5c85640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72b006f7a0 0x7f72b00c0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:08.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72b5c85640 1 -- 172.21.15.46:0/706415646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a0009420 con 0x7f72b0071110 2023-10-30T13:29:08.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72b5c85640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b0071110 0x7f72b014f790 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f72a001d320 tx=0x7f72a001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:08.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72a6ffd640 1 -- 172.21.15.46:0/706415646 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a00bb640 con 0x7f72b0071110 2023-10-30T13:29:08.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72a6ffd640 1 -- 172.21.15.46:0/706415646 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a00bb7c0 con 0x7f72b0071110 2023-10-30T13:29:08.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72a6ffd640 1 -- 172.21.15.46:0/706415646 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a00bbb70 con 0x7f72b0071110 2023-10-30T13:29:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72b86ea640 1 -- 172.21.15.46:0/706415646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b01e4450 con 0x7f72b0071110 2023-10-30T13:29:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72b86ea640 1 -- 172.21.15.46:0/706415646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b00c08b0 con 0x7f72b0071110 2023-10-30T13:29:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72b86ea640 1 -- 172.21.15.46:0/706415646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b00c8f50 con 0x7f72b0071110 2023-10-30T13:29:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72a6ffd640 1 -- 172.21.15.46:0/706415646 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f72a00b7040 con 0x7f72b0071110 2023-10-30T13:29:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72a6ffd640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72940664d0 0x7f7294068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:08.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.113+0000 7f72a6ffd640 1 -- 172.21.15.46:0/706415646 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6678+0+0 (secure 0 0 0) 0x7f72a01226d0 con 0x7f72b0071110 2023-10-30T13:29:08.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.117+0000 7f72b6486640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72940664d0 0x7f7294068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:08.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.117+0000 7f72b6486640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72940664d0 0x7f7294068990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f72ac000f70 tx=0x7f72ac007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:08.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.117+0000 7f72a6ffd640 1 -- 172.21.15.46:0/706415646 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f72b00c8f50 con 0x7f72b0071110 2023-10-30T13:29:08.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.309+0000 7f72b86ea640 1 -- 172.21.15.46:0/706415646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72b00c8f50 con 0x7f72b0071110 2023-10-30T13:29:08.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.309+0000 7f72a6ffd640 1 -- 172.21.15.46:0/706415646 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f72b00c8f50 con 0x7f72b0071110 2023-10-30T13:29:08.316 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:08.316 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:28:00.722346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4288,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.154:6835/3109262047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3109262047},{"type":"v1","addr":"172.21.15.154:6835","nonce":3109262047}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:08.317 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 28 2023-10-30T13:29:08.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.313+0000 7f72b86ea640 1 -- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72940664d0 msgr2=0x7f7294068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:08.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.313+0000 7f72b86ea640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72940664d0 0x7f7294068990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f72ac000f70 tx=0x7f72ac007c10 comp rx=0 tx=0).stop 2023-10-30T13:29:08.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.313+0000 7f72b86ea640 1 -- 172.21.15.46:0/706415646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b0071110 msgr2=0x7f72b014f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:08.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.313+0000 7f72b86ea640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b0071110 0x7f72b014f790 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f72a001d320 tx=0x7f72a001ba10 comp rx=0 tx=0).stop 2023-10-30T13:29:08.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.313+0000 7f72b86ea640 1 -- 172.21.15.46:0/706415646 shutdown_connections 2023-10-30T13:29:08.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.313+0000 7f72b86ea640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72940664d0 0x7f7294068990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:08.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.313+0000 7f72b86ea640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72b00c6bd0 0x7f72b014fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:08.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.313+0000 7f72b86ea640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b0071110 0x7f72b014f790 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:08.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.313+0000 7f72b86ea640 1 --2- 172.21.15.46:0/706415646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72b006f7a0 0x7f72b00c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:08.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.313+0000 7f72b86ea640 1 -- 172.21.15.46:0/706415646 >> 172.21.15.46:0/706415646 conn(0x7f72b00c7910 msgr2=0x7f72b006a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:08.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.313+0000 7f72b86ea640 1 -- 172.21.15.46:0/706415646 shutdown_connections 2023-10-30T13:29:08.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:08.313+0000 7f72b86ea640 1 -- 172.21.15.46:0/706415646 wait complete. 2023-10-30T13:29:08.337 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:29:10.338 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:10.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a8f1b8640 1 Processor -- start 2023-10-30T13:29:10.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a8f1b8640 1 -- start start 2023-10-30T13:29:10.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a8f1b8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8a8815d6f0 0x7f8a8815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a8f1b8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a880cb800 con 0x7f8a880cf160 2023-10-30T13:29:10.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a8f1b8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a880cb330 con 0x7f8a881538a0 2023-10-30T13:29:10.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a8f1b8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a880cad70 con 0x7f8a8815d6f0 2023-10-30T13:29:10.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a8d755640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8a880cf160 0x7f8a88164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50312/0 (socket says 172.21.15.46:50312) 2023-10-30T13:29:10.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a8d755640 1 -- 172.21.15.46:0/1716645229 learned_addr learned my addr 172.21.15.46:0/1716645229 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:10.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a87fff640 1 --2- 172.21.15.46:0/1716645229 >> v2:172.21.15.154:3300/0 conn(0x7f8a8815d6f0 0x7f8a8815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141781585 0 0) 0x7f8a880cb800 con 0x7f8a880cf160 2023-10-30T13:29:10.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a78003530 con 0x7f8a880cf160 2023-10-30T13:29:10.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4242074024 0 0) 0x7f8a78003530 con 0x7f8a880cf160 2023-10-30T13:29:10.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 >> v2:172.21.15.154:3300/0 conn(0x7f8a8815d6f0 msgr2=0x7f8a8815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:10.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 --2- 172.21.15.46:0/1716645229 >> v2:172.21.15.154:3300/0 conn(0x7f8a8815d6f0 0x7f8a8815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 >> v1:172.21.15.154:6789/0 conn(0x7f8a881538a0 legacy=0x7f8a880ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:10.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a880cb050 con 0x7f8a880cf160 2023-10-30T13:29:10.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f8a80001a00 con 0x7f8a880cf160 2023-10-30T13:29:10.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 >> v1:172.21.15.46:6789/0 conn(0x7f8a880cf160 legacy=0x7f8a88164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:10.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 --2- 172.21.15.46:0/1716645229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a78003980 0x7f8a78003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 --2- 172.21.15.46:0/1716645229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a780047d0 0x7f8a78004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 --2- 172.21.15.46:0/1716645229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a78005190 0x7f8a7800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a880cad70 con 0x7f8a780047d0 2023-10-30T13:29:10.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7800c040 con 0x7f8a78005190 2023-10-30T13:29:10.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7800c200 con 0x7f8a78003980 2023-10-30T13:29:10.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.489+0000 7f8a87fff640 1 --2- 172.21.15.46:0/1716645229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a780047d0 0x7f8a78004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a8d755640 1 --2- 172.21.15.46:0/1716645229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a78005190 0x7f8a7800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a8cf54640 1 --2- 172.21.15.46:0/1716645229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a78003980 0x7f8a78003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a87fff640 1 -- 172.21.15.46:0/1716645229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a78003980 msgr2=0x7f8a78003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:10.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a87fff640 1 --2- 172.21.15.46:0/1716645229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a78003980 0x7f8a78003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a87fff640 1 -- 172.21.15.46:0/1716645229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a78005190 msgr2=0x7f8a7800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a87fff640 1 --2- 172.21.15.46:0/1716645229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a78005190 0x7f8a7800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a87fff640 1 -- 172.21.15.46:0/1716645229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a78003530 con 0x7f8a780047d0 2023-10-30T13:29:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a8d755640 1 -- 172.21.15.46:0/1716645229 reap_dead start 2023-10-30T13:29:10.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a87fff640 1 --2- 172.21.15.46:0/1716645229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a780047d0 0x7f8a78004c50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8a74009ae0 tx=0x7f8a7401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285e405be62fe54 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:10.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a740a5640 con 0x7f8a780047d0 2023-10-30T13:29:10.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a740a57a0 con 0x7f8a780047d0 2023-10-30T13:29:10.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a877fe640 1 -- 172.21.15.46:0/1716645229 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a740a5b50 con 0x7f8a780047d0 2023-10-30T13:29:10.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1716645229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a780047d0 msgr2=0x7f8a78004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:10.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.493+0000 7f8a8f1b8640 1 --2- 172.21.15.46:0/1716645229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a780047d0 0x7f8a78004c50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8a74009ae0 tx=0x7f8a7401ca50 comp rx=0 tx=0).stop 2023-10-30T13:29:10.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1716645229 shutdown_connections 2023-10-30T13:29:10.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 --2- 172.21.15.46:0/1716645229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a780047d0 0x7f8a78004c50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1716645229 >> 172.21.15.46:0/1716645229 conn(0x7f8a8806bfe0 msgr2=0x7f8a8806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:10.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1716645229 shutdown_connections 2023-10-30T13:29:10.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1716645229 wait complete. 2023-10-30T13:29:10.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 Processor -- start 2023-10-30T13:29:10.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 -- start start 2023-10-30T13:29:10.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a780047d0 0x7f8a8814ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a881efcc0 0x7f8a881e45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a88159420 0x7f8a881586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a88158520 con 0x7f8a88159420 2023-10-30T13:29:10.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a880b3330 con 0x7f8a881efcc0 2023-10-30T13:29:10.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8f1b8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a881e4160 con 0x7f8a780047d0 2023-10-30T13:29:10.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8d755640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a88159420 0x7f8a881586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8d755640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a88159420 0x7f8a881586e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41866/0 (socket says 172.21.15.46:41866) 2023-10-30T13:29:10.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8d755640 1 -- 172.21.15.46:0/1663520478 learned_addr learned my addr 172.21.15.46:0/1663520478 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:10.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a87fff640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a881efcc0 0x7f8a881e45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8cf54640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a780047d0 0x7f8a8814ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8d755640 1 -- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a780047d0 msgr2=0x7f8a8814ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:10.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8d755640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a780047d0 0x7f8a8814ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8d755640 1 -- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a881efcc0 msgr2=0x7f8a881e45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8d755640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a881efcc0 0x7f8a881e45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8d755640 1 -- 172.21.15.46:0/1663520478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a881e3e30 con 0x7f8a88159420 2023-10-30T13:29:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8cf54640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a780047d0 0x7f8a8814ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.497+0000 7f8a8d755640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a88159420 0x7f8a881586e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8a8806cb00 tx=0x7f8a80007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:10.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.501+0000 7f8a857fa640 1 -- 172.21.15.46:0/1663520478 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8000aad0 con 0x7f8a88159420 2023-10-30T13:29:10.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.501+0000 7f8a857fa640 1 -- 172.21.15.46:0/1663520478 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8000ac30 con 0x7f8a88159420 2023-10-30T13:29:10.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.501+0000 7f8a857fa640 1 -- 172.21.15.46:0/1663520478 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a800133f0 con 0x7f8a88159420 2023-10-30T13:29:10.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.501+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1663520478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a881f0180 con 0x7f8a88159420 2023-10-30T13:29:10.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.501+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1663520478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a881e31c0 con 0x7f8a88159420 2023-10-30T13:29:10.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.501+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1663520478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a88159870 con 0x7f8a88159420 2023-10-30T13:29:10.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.501+0000 7f8a857fa640 1 -- 172.21.15.46:0/1663520478 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8a80013550 con 0x7f8a88159420 2023-10-30T13:29:10.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.501+0000 7f8a857fa640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8a6c0664a0 0x7f8a6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.501+0000 7f8a857fa640 1 -- 172.21.15.46:0/1663520478 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f8a80084670 con 0x7f8a88159420 2023-10-30T13:29:10.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.501+0000 7f8a8cf54640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8a6c0664a0 0x7f8a6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.505+0000 7f8a8cf54640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8a6c0664a0 0x7f8a6c068960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c000c10 tx=0x7f8a7c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:10.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.505+0000 7f8a857fa640 1 -- 172.21.15.46:0/1663520478 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8a88159870 con 0x7f8a88159420 2023-10-30T13:29:10.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.697+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1663520478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a88159870 con 0x7f8a88159420 2023-10-30T13:29:10.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.697+0000 7f8a857fa640 1 -- 172.21.15.46:0/1663520478 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f8a88159870 con 0x7f8a88159420 2023-10-30T13:29:10.704 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:10.704 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:08.773431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:10.704 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 29 2023-10-30T13:29:10.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.697+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8a6c0664a0 msgr2=0x7f8a6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:10.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.697+0000 7f8a8f1b8640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8a6c0664a0 0x7f8a6c068960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c000c10 tx=0x7f8a7c009040 comp rx=0 tx=0).stop 2023-10-30T13:29:10.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.697+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1663520478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a88159420 msgr2=0x7f8a881586e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:10.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.697+0000 7f8a8f1b8640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a88159420 0x7f8a881586e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8a8806cb00 tx=0x7f8a80007bd0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.701+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1663520478 shutdown_connections 2023-10-30T13:29:10.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.701+0000 7f8a8f1b8640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8a6c0664a0 0x7f8a6c068960 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.701+0000 7f8a8f1b8640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a88159420 0x7f8a881586e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.701+0000 7f8a8f1b8640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a881efcc0 0x7f8a881e45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.701+0000 7f8a8f1b8640 1 --2- 172.21.15.46:0/1663520478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a780047d0 0x7f8a8814ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.701+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1663520478 >> 172.21.15.46:0/1663520478 conn(0x7f8a8806bfe0 msgr2=0x7f8a88065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.701+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1663520478 shutdown_connections 2023-10-30T13:29:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.701+0000 7f8a8f1b8640 1 -- 172.21.15.46:0/1663520478 wait complete. 2023-10-30T13:29:10.725 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-30T13:29:10.725 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:29:10.725 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:10.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7f5002640 1 Processor -- start 2023-10-30T13:29:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7f5002640 1 -- start start 2023-10-30T13:29:10.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7f5002640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc7f00c6a50 0x7fc7f015d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7f5002640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f00c8b20 0x7fc7f0159780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7f5002640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7f0159cc0 0x7fc7f006afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7f5002640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f006f4c0 con 0x7fc7f0159cc0 2023-10-30T13:29:10.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7f5002640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f006f6f0 con 0x7fc7f00c6a50 2023-10-30T13:29:10.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7f5002640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f006f9d0 con 0x7fc7f00c8b20 2023-10-30T13:29:10.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7ef59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7f0159cc0 0x7fc7f006afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7ef59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7f0159cc0 0x7fc7f006afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41870/0 (socket says 172.21.15.46:41870) 2023-10-30T13:29:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7ef59e640 1 -- 172.21.15.46:0/289922812 learned_addr learned my addr 172.21.15.46:0/289922812 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7e7fff640 1 --2- 172.21.15.46:0/289922812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f00c8b20 0x7fc7f0159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7eed9d640 1 --2- 172.21.15.46:0/289922812 >> v2:172.21.15.154:3300/0 conn(0x7fc7f00c6a50 0x7fc7f015d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7ef59e640 1 -- 172.21.15.46:0/289922812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f00c8b20 msgr2=0x7fc7f0159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7ef59e640 1 --2- 172.21.15.46:0/289922812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f00c8b20 0x7fc7f0159780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7ef59e640 1 -- 172.21.15.46:0/289922812 >> v2:172.21.15.154:3300/0 conn(0x7fc7f00c6a50 msgr2=0x7fc7f015d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7ef59e640 1 --2- 172.21.15.46:0/289922812 >> v2:172.21.15.154:3300/0 conn(0x7fc7f00c6a50 0x7fc7f015d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.865+0000 7fc7ef59e640 1 -- 172.21.15.46:0/289922812 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f0071580 con 0x7fc7f0159cc0 2023-10-30T13:29:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7ef59e640 1 --2- 172.21.15.46:0/289922812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f0159cc0 0x7fc7f006afa0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0072d60 tx=0x7fc7e801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642170120e054a28 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7ee59c640 1 -- 172.21.15.46:0/289922812 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e80af400 con 0x7fc7f0159cc0 2023-10-30T13:29:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7ee59c640 1 -- 172.21.15.46:0/289922812 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e80ab050 con 0x7fc7f0159cc0 2023-10-30T13:29:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7ee59c640 1 -- 172.21.15.46:0/289922812 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e80af780 con 0x7fc7f0159cc0 2023-10-30T13:29:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7f5002640 1 -- 172.21.15.46:0/289922812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f0159cc0 msgr2=0x7fc7f006afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7f5002640 1 --2- 172.21.15.46:0/289922812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f0159cc0 0x7fc7f006afa0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0072d60 tx=0x7fc7e801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7f5002640 1 -- 172.21.15.46:0/289922812 shutdown_connections 2023-10-30T13:29:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7f5002640 1 --2- 172.21.15.46:0/289922812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f0159cc0 0x7fc7f006afa0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7f5002640 1 --2- 172.21.15.46:0/289922812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f00c8b20 0x7fc7f0159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7f5002640 1 --2- 172.21.15.46:0/289922812 >> v2:172.21.15.154:3300/0 conn(0x7fc7f00c6a50 0x7fc7f015d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7f5002640 1 -- 172.21.15.46:0/289922812 >> 172.21.15.46:0/289922812 conn(0x7fc7f0065a80 msgr2=0x7fc7f0065e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:10.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7f5002640 1 -- 172.21.15.46:0/289922812 shutdown_connections 2023-10-30T13:29:10.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7f5002640 1 -- 172.21.15.46:0/289922812 wait complete. 2023-10-30T13:29:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.869+0000 7fc7f5002640 1 Processor -- start 2023-10-30T13:29:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7f5002640 1 -- start start 2023-10-30T13:29:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7f5002640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f00c6a50 0x7fc7f0155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7f5002640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f00c8b20 0x7fc7f014e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7f5002640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f0159cc0 0x7fc7f014e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7f5002640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f0156c80 con 0x7fc7f0159cc0 2023-10-30T13:29:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7f5002640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f0156f60 con 0x7fc7f00c6a50 2023-10-30T13:29:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7f5002640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f0154270 con 0x7fc7f00c8b20 2023-10-30T13:29:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7ef59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f0159cc0 0x7fc7f014e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7ef59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f0159cc0 0x7fc7f014e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41880/0 (socket says 172.21.15.46:41880) 2023-10-30T13:29:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7ef59e640 1 -- 172.21.15.46:0/2036182832 learned_addr learned my addr 172.21.15.46:0/2036182832 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7e7fff640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f00c8b20 0x7fc7f014e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7eed9d640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f00c6a50 0x7fc7f0155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7ef59e640 1 -- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f00c8b20 msgr2=0x7fc7f014e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7ef59e640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f00c8b20 0x7fc7f014e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7ef59e640 1 -- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f00c6a50 msgr2=0x7fc7f0155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7ef59e640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f00c6a50 0x7fc7f0155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7ef59e640 1 -- 172.21.15.46:0/2036182832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e8009b30 con 0x7fc7f0159cc0 2023-10-30T13:29:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7ef59e640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f0159cc0 0x7fc7f014e650 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc7e80b2040 tx=0x7fc7e801bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7e77fe640 1 -- 172.21.15.46:0/2036182832 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e80afc80 con 0x7fc7f0159cc0 2023-10-30T13:29:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7e77fe640 1 -- 172.21.15.46:0/2036182832 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e80afe00 con 0x7fc7f0159cc0 2023-10-30T13:29:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7e77fe640 1 -- 172.21.15.46:0/2036182832 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e80ba660 con 0x7fc7f0159cc0 2023-10-30T13:29:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7f5002640 1 -- 172.21.15.46:0/2036182832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f0154550 con 0x7fc7f0159cc0 2023-10-30T13:29:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7f5002640 1 -- 172.21.15.46:0/2036182832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f0156440 con 0x7fc7f0159cc0 2023-10-30T13:29:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.873+0000 7fc7f5002640 1 -- 172.21.15.46:0/2036182832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f004e170 con 0x7fc7f0159cc0 2023-10-30T13:29:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.877+0000 7fc7e77fe640 1 -- 172.21.15.46:0/2036182832 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7e80b6040 con 0x7fc7f0159cc0 2023-10-30T13:29:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.877+0000 7fc7e77fe640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d00664a0 0x7fc7d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.877+0000 7fc7e77fe640 1 -- 172.21.15.46:0/2036182832 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fc7e81221a0 con 0x7fc7f0159cc0 2023-10-30T13:29:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.877+0000 7fc7eed9d640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d00664a0 0x7fc7d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.877+0000 7fc7eed9d640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d00664a0 0x7fc7d0068960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009a40 tx=0x7fc7e0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:10.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:10.881+0000 7fc7e77fe640 1 -- 172.21.15.46:0/2036182832 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7e80ef690 con 0x7fc7f0159cc0 2023-10-30T13:29:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 -- 172.21.15.46:0/2036182832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7f00c6ed0 con 0x7fc7f0159cc0 2023-10-30T13:29:11.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7e77fe640 1 -- 172.21.15.46:0/2036182832 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5735 (secure 0 0 0) 0x7fc7e8009510 con 0x7fc7f0159cc0 2023-10-30T13:29:11.083 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:11.083 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:08.773431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:11.084 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 29 2023-10-30T13:29:11.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 -- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d00664a0 msgr2=0x7fc7d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:11.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d00664a0 0x7fc7d0068960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009a40 tx=0x7fc7e0009190 comp rx=0 tx=0).stop 2023-10-30T13:29:11.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 -- 172.21.15.46:0/2036182832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f0159cc0 msgr2=0x7fc7f014e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:11.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f0159cc0 0x7fc7f014e650 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc7e80b2040 tx=0x7fc7e801bf60 comp rx=0 tx=0).stop 2023-10-30T13:29:11.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 -- 172.21.15.46:0/2036182832 shutdown_connections 2023-10-30T13:29:11.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d00664a0 0x7fc7d0068960 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:11.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f0159cc0 0x7fc7f014e650 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:11.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f00c8b20 0x7fc7f014e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:11.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 --2- 172.21.15.46:0/2036182832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f00c6a50 0x7fc7f0155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:11.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 -- 172.21.15.46:0/2036182832 >> 172.21.15.46:0/2036182832 conn(0x7fc7f0065a80 msgr2=0x7fc7f0153400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:11.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 -- 172.21.15.46:0/2036182832 shutdown_connections 2023-10-30T13:29:11.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:11.077+0000 7fc7f5002640 1 -- 172.21.15.46:0/2036182832 wait complete. 2023-10-30T13:29:11.105 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 29, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 29, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4306": { "addr": "172.21.15.46:6836/275749119", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 275749119, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 275749119, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4950": { "addr": "172.21.15.46:6837/1315246492", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 1315246492, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 1315246492, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4950, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_5106": { "addr": "172.21.15.46:6833/603884510", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 603884510, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 603884510, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5106, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:29:08.773431+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 5106, "mds_2": 4950 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:29:13.106 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:13.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.253+0000 7fc3d7809640 1 Processor -- start 2023-10-30T13:29:13.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d7809640 1 -- start start 2023-10-30T13:29:13.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d7809640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc3d01538a0 0x7fc3d015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:13.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d7809640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3d00cf160 0x7fc3d00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d7809640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d00cb800 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d7809640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d015dc30 con 0x7fc3d00cf160 2023-10-30T13:29:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d7809640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d00cb330 con 0x7fc3d01538a0 2023-10-30T13:29:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d5da6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3d00cf160 0x7fc3d00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d5da6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3d00cf160 0x7fc3d00cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41894/0 (socket says 172.21.15.46:41894) 2023-10-30T13:29:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d5da6640 1 -- 172.21.15.46:0/2282387170 learned_addr learned my addr 172.21.15.46:0/2282387170 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d55a5640 1 --2- 172.21.15.46:0/2282387170 >> v2:172.21.15.154:3300/0 conn(0x7fc3d01538a0 0x7fc3d015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d5da6640 1 -- 172.21.15.46:0/2282387170 >> v2:172.21.15.154:3300/0 conn(0x7fc3d01538a0 msgr2=0x7fc3d015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d5da6640 1 --2- 172.21.15.46:0/2282387170 >> v2:172.21.15.154:3300/0 conn(0x7fc3d01538a0 0x7fc3d015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d5da6640 1 -- 172.21.15.46:0/2282387170 >> v1:172.21.15.154:6789/0 conn(0x7fc3d00ca0e0 legacy=0x7fc3d015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.257+0000 7fc3d5da6640 1 -- 172.21.15.46:0/2282387170 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d00cad70 con 0x7fc3d00cf160 2023-10-30T13:29:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3d5da6640 1 --2- 172.21.15.46:0/2282387170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d00cf160 0x7fc3d00cf590 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00cfd10 tx=0x7fc3c801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463d175110987f0f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3c7fff640 1 -- 172.21.15.46:0/2282387170 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c80af400 con 0x7fc3d00cf160 2023-10-30T13:29:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3c7fff640 1 -- 172.21.15.46:0/2282387170 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c80ab050 con 0x7fc3d00cf160 2023-10-30T13:29:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3c7fff640 1 -- 172.21.15.46:0/2282387170 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c80af780 con 0x7fc3d00cf160 2023-10-30T13:29:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3d7809640 1 -- 172.21.15.46:0/2282387170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d00cf160 msgr2=0x7fc3d00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3d7809640 1 --2- 172.21.15.46:0/2282387170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d00cf160 0x7fc3d00cf590 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00cfd10 tx=0x7fc3c801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:29:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3d7809640 1 -- 172.21.15.46:0/2282387170 shutdown_connections 2023-10-30T13:29:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3d7809640 1 --2- 172.21.15.46:0/2282387170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d00cf160 0x7fc3d00cf590 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3d7809640 1 --2- 172.21.15.46:0/2282387170 >> v2:172.21.15.154:3300/0 conn(0x7fc3d01538a0 0x7fc3d015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3d7809640 1 -- 172.21.15.46:0/2282387170 >> 172.21.15.46:0/2282387170 conn(0x7fc3d006bfe0 msgr2=0x7fc3d006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3d7809640 1 -- 172.21.15.46:0/2282387170 shutdown_connections 2023-10-30T13:29:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3d7809640 1 -- 172.21.15.46:0/2282387170 wait complete. 2023-10-30T13:29:13.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3d7809640 1 Processor -- start 2023-10-30T13:29:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.261+0000 7fc3d7809640 1 -- start start 2023-10-30T13:29:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d7809640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d00ca0e0 0x7fc3d01480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d7809640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d00cf160 0x7fc3d0158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d7809640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d01538a0 0x7fc3d01f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d7809640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d01e5330 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d7809640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d00b4f30 con 0x7fc3d00cf160 2023-10-30T13:29:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d7809640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d01e4d70 con 0x7fc3d01538a0 2023-10-30T13:29:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d55a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d00ca0e0 0x7fc3d01480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d55a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d00ca0e0 0x7fc3d01480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41902/0 (socket says 172.21.15.46:41902) 2023-10-30T13:29:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d55a5640 1 -- 172.21.15.46:0/919183443 learned_addr learned my addr 172.21.15.46:0/919183443 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d4da4640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d00cf160 0x7fc3d0158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d5da6640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d01538a0 0x7fc3d01f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d55a5640 1 -- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d01538a0 msgr2=0x7fc3d01f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:13.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d55a5640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d01538a0 0x7fc3d01f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:13.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d55a5640 1 -- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d00cf160 msgr2=0x7fc3d0158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:13.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d55a5640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d00cf160 0x7fc3d0158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:13.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d55a5640 1 -- 172.21.15.46:0/919183443 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c8009b30 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d5da6640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d01538a0 0x7fc3d01f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:29:13.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d55a5640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d00ca0e0 0x7fc3d01480f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00e930 tx=0x7fc3cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:13.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3c5ffb640 1 -- 172.21.15.46:0/919183443 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc009690 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3c5ffb640 1 -- 172.21.15.46:0/919183443 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3cc00ca00 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d7809640 1 -- 172.21.15.46:0/919183443 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d01e5050 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3c5ffb640 1 -- 172.21.15.46:0/919183443 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc00cd70 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d7809640 1 -- 172.21.15.46:0/919183443 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d00b4bc0 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.265+0000 7fc3d7809640 1 -- 172.21.15.46:0/919183443 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d00b4d80 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.269+0000 7fc3c5ffb640 1 -- 172.21.15.46:0/919183443 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3cc013470 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.269+0000 7fc3c5ffb640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3b40664a0 0x7fc3b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:13.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.269+0000 7fc3c5ffb640 1 -- 172.21.15.46:0/919183443 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fc3cc084ad0 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.269+0000 7fc3d4da4640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3b40664a0 0x7fc3b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:13.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.269+0000 7fc3d4da4640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3b40664a0 0x7fc3b4068960 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc3d01e45c0 tx=0x7fc3c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:13.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.273+0000 7fc3c5ffb640 1 -- 172.21.15.46:0/919183443 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc3cc051df0 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.461+0000 7fc3d7809640 1 -- 172.21.15.46:0/919183443 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3d00b2a50 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.461+0000 7fc3c5ffb640 1 -- 172.21.15.46:0/919183443 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5735 (secure 0 0 0) 0x7fc3cc021090 con 0x7fc3d00ca0e0 2023-10-30T13:29:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:13.470 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:08.773431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 29 2023-10-30T13:29:13.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.465+0000 7fc3d7809640 1 -- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3b40664a0 msgr2=0x7fc3b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:13.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.465+0000 7fc3d7809640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3b40664a0 0x7fc3b4068960 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc3d01e45c0 tx=0x7fc3c0009040 comp rx=0 tx=0).stop 2023-10-30T13:29:13.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.465+0000 7fc3d7809640 1 -- 172.21.15.46:0/919183443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d00ca0e0 msgr2=0x7fc3d01480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:13.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.465+0000 7fc3d7809640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d00ca0e0 0x7fc3d01480f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00e930 tx=0x7fc3cc00edf0 comp rx=0 tx=0).stop 2023-10-30T13:29:13.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.465+0000 7fc3d7809640 1 -- 172.21.15.46:0/919183443 shutdown_connections 2023-10-30T13:29:13.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.465+0000 7fc3d7809640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3b40664a0 0x7fc3b4068960 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:13.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.465+0000 7fc3d7809640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d01538a0 0x7fc3d01f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:13.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.465+0000 7fc3d7809640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d00cf160 0x7fc3d0158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:13.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.465+0000 7fc3d7809640 1 --2- 172.21.15.46:0/919183443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d00ca0e0 0x7fc3d01480f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:13.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.465+0000 7fc3d7809640 1 -- 172.21.15.46:0/919183443 >> 172.21.15.46:0/919183443 conn(0x7fc3d006bfe0 msgr2=0x7fc3d006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:13.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.465+0000 7fc3d7809640 1 -- 172.21.15.46:0/919183443 shutdown_connections 2023-10-30T13:29:13.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:13.465+0000 7fc3d7809640 1 -- 172.21.15.46:0/919183443 wait complete. 2023-10-30T13:29:13.490 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:29:15.491 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:15.552 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:29:15.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:29:15.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f3e4c9640 1 Processor -- start 2023-10-30T13:29:15.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f3e4c9640 1 -- start start 2023-10-30T13:29:15.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f3e4c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f380cf160 0x7f1f380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:15.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f3e4c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380cb800 con 0x7f1f3815d6f0 2023-10-30T13:29:15.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f3e4c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380cb330 con 0x7f1f381538a0 2023-10-30T13:29:15.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f3e4c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f380cad70 con 0x7f1f380cf160 2023-10-30T13:29:15.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f3ca66640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f380cf160 0x7f1f380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:15.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f377fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1f3815d6f0 0x7f1f3815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50314/0 (socket says 172.21.15.46:50314) 2023-10-30T13:29:15.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f377fe640 1 -- 172.21.15.46:0/2167180532 learned_addr learned my addr 172.21.15.46:0/2167180532 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:15.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f3ca66640 1 -- 172.21.15.46:0/2167180532 >> v1:172.21.15.154:6789/0 conn(0x7f1f381538a0 legacy=0x7f1f380ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:15.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f3ca66640 1 -- 172.21.15.46:0/2167180532 >> v1:172.21.15.46:6789/0 conn(0x7f1f3815d6f0 legacy=0x7f1f3815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:15.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f3ca66640 1 -- 172.21.15.46:0/2167180532 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f380cb050 con 0x7f1f380cf160 2023-10-30T13:29:15.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.640+0000 7f1f36ffd640 1 -- 172.21.15.46:0/2167180532 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163740935 0 0) 0x7f1f380cb800 con 0x7f1f3815d6f0 2023-10-30T13:29:15.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.644+0000 7f1f3ca66640 1 --2- 172.21.15.46:0/2167180532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f380cf160 0x7f1f380cf590 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009d90 tx=0x7f1f3001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53392aa2cf591815 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:15.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.644+0000 7f1f36ffd640 1 -- 172.21.15.46:0/2167180532 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f300af600 con 0x7f1f380cf160 2023-10-30T13:29:15.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.644+0000 7f1f36ffd640 1 -- 172.21.15.46:0/2167180532 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f300ab050 con 0x7f1f380cf160 2023-10-30T13:29:15.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.644+0000 7f1f36ffd640 1 -- 172.21.15.46:0/2167180532 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f300af980 con 0x7f1f380cf160 2023-10-30T13:29:15.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.644+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/2167180532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f380cf160 msgr2=0x7f1f380cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:15.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.644+0000 7f1f3e4c9640 1 --2- 172.21.15.46:0/2167180532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f380cf160 0x7f1f380cf590 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009d90 tx=0x7f1f3001f130 comp rx=0 tx=0).stop 2023-10-30T13:29:15.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.644+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/2167180532 shutdown_connections 2023-10-30T13:29:15.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.644+0000 7f1f3e4c9640 1 --2- 172.21.15.46:0/2167180532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f380cf160 0x7f1f380cf590 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:15.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.644+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/2167180532 >> 172.21.15.46:0/2167180532 conn(0x7f1f3806bfe0 msgr2=0x7f1f3806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:15.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.644+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/2167180532 shutdown_connections 2023-10-30T13:29:15.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.644+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/2167180532 wait complete. 2023-10-30T13:29:15.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3e4c9640 1 Processor -- start 2023-10-30T13:29:15.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3e4c9640 1 -- start start 2023-10-30T13:29:15.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3e4c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f380cf160 0x7f1f3814fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:15.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3e4c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f381538a0 0x7f1f381500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:15.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3e4c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3815d6f0 0x7f1f380b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:15.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3e4c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3814f630 con 0x7f1f3815d6f0 2023-10-30T13:29:15.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3e4c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f381519c0 con 0x7f1f380cf160 2023-10-30T13:29:15.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3e4c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3814dc80 con 0x7f1f381538a0 2023-10-30T13:29:15.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3ca66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3815d6f0 0x7f1f380b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:15.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3ca66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3815d6f0 0x7f1f380b45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41918/0 (socket says 172.21.15.46:41918) 2023-10-30T13:29:15.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3ca66640 1 -- 172.21.15.46:0/877834372 learned_addr learned my addr 172.21.15.46:0/877834372 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:15.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f377fe640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f381538a0 0x7f1f381500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:15.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f37fff640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f380cf160 0x7f1f3814fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:15.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3ca66640 1 -- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f381538a0 msgr2=0x7f1f381500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:15.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3ca66640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f381538a0 0x7f1f381500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:15.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3ca66640 1 -- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f380cf160 msgr2=0x7f1f3814fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:15.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3ca66640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f380cf160 0x7f1f3814fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:15.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3ca66640 1 -- 172.21.15.46:0/877834372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f30009a70 con 0x7f1f3815d6f0 2023-10-30T13:29:15.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f377fe640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f381538a0 0x7f1f381500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:15.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3ca66640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3815d6f0 0x7f1f380b45c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1f300b2040 tx=0x7f1f3001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:15.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f34ff9640 1 -- 172.21.15.46:0/877834372 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f300b9410 con 0x7f1f3815d6f0 2023-10-30T13:29:15.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f34ff9640 1 -- 172.21.15.46:0/877834372 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f300b9570 con 0x7f1f3815d6f0 2023-10-30T13:29:15.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f34ff9640 1 -- 172.21.15.46:0/877834372 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f300b9920 con 0x7f1f3815d6f0 2023-10-30T13:29:15.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/877834372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f38151700 con 0x7f1f3815d6f0 2023-10-30T13:29:15.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.648+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/877834372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3814e200 con 0x7f1f3815d6f0 2023-10-30T13:29:15.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.652+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/877834372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3804e170 con 0x7f1f3815d6f0 2023-10-30T13:29:15.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.652+0000 7f1f34ff9640 1 -- 172.21.15.46:0/877834372 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1f300b6050 con 0x7f1f3815d6f0 2023-10-30T13:29:15.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.652+0000 7f1f34ff9640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1f240664d0 0x7f1f24068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:15.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.652+0000 7f1f34ff9640 1 -- 172.21.15.46:0/877834372 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f1f30122550 con 0x7f1f3815d6f0 2023-10-30T13:29:15.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.652+0000 7f1f37fff640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1f240664d0 0x7f1f24068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:15.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.652+0000 7f1f37fff640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1f240664d0 0x7f1f24068990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1f28007fd0 tx=0x7f1f28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:15.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.656+0000 7f1f34ff9640 1 -- 172.21.15.46:0/877834372 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1f300ef880 con 0x7f1f3815d6f0 2023-10-30T13:29:15.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.848+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/877834372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f380b1750 con 0x7f1f3815d6f0 2023-10-30T13:29:15.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.848+0000 7f1f34ff9640 1 -- 172.21.15.46:0/877834372 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f1f300f2ea0 con 0x7f1f3815d6f0 2023-10-30T13:29:15.857 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:15.857 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:14.210718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:resolve","state_seq":37,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:15.858 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 30 2023-10-30T13:29:15.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.852+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1f240664d0 msgr2=0x7f1f24068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:15.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.852+0000 7f1f3e4c9640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1f240664d0 0x7f1f24068990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1f28007fd0 tx=0x7f1f28009040 comp rx=0 tx=0).stop 2023-10-30T13:29:15.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.852+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/877834372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3815d6f0 msgr2=0x7f1f380b45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:15.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.852+0000 7f1f3e4c9640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3815d6f0 0x7f1f380b45c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1f300b2040 tx=0x7f1f3001bed0 comp rx=0 tx=0).stop 2023-10-30T13:29:15.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.852+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/877834372 shutdown_connections 2023-10-30T13:29:15.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.852+0000 7f1f3e4c9640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1f240664d0 0x7f1f24068990 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:15.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.852+0000 7f1f3e4c9640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3815d6f0 0x7f1f380b45c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:15.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.852+0000 7f1f3e4c9640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f381538a0 0x7f1f381500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:15.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.852+0000 7f1f3e4c9640 1 --2- 172.21.15.46:0/877834372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f380cf160 0x7f1f3814fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:15.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.852+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/877834372 >> 172.21.15.46:0/877834372 conn(0x7f1f3806bfe0 msgr2=0x7f1f380cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:15.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.852+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/877834372 shutdown_connections 2023-10-30T13:29:15.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:15.852+0000 7f1f3e4c9640 1 -- 172.21.15.46:0/877834372 wait complete. 2023-10-30T13:29:15.877 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:29:17.878 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:18.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe56286b640 1 Processor -- start 2023-10-30T13:29:18.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe56286b640 1 -- start start 2023-10-30T13:29:18.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe56286b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c0cf160 0x7fe55c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:18.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe56286b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c0cb800 con 0x7fe55c1538a0 2023-10-30T13:29:18.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe56286b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c0cb330 con 0x7fe55c15d6f0 2023-10-30T13:29:18.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe56286b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c0cad70 con 0x7fe55c0cf160 2023-10-30T13:29:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe55c1538a0 0x7fe55c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35986/0 (socket says 172.21.15.46:35986) 2023-10-30T13:29:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55bfff640 1 -- 172.21.15.46:0/747735204 learned_addr learned my addr 172.21.15.46:0/747735204 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe560e08640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c0cf160 0x7fe55c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451275140 0 0) 0x7fe55c0cb800 con 0x7fe55c1538a0 2023-10-30T13:29:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe544003530 con 0x7fe55c1538a0 2023-10-30T13:29:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383641166 0 0) 0x7fe55c0cb330 con 0x7fe55c15d6f0 2023-10-30T13:29:18.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe55c0cb800 con 0x7fe55c15d6f0 2023-10-30T13:29:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3129190572 0 0) 0x7fe544003530 con 0x7fe55c1538a0 2023-10-30T13:29:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c0cf160 msgr2=0x7fe55c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c0cf160 0x7fe55c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 >> v1:172.21.15.154:6789/0 conn(0x7fe55c15d6f0 legacy=0x7fe55c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:18.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c0cb050 con 0x7fe55c1538a0 2023-10-30T13:29:18.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fe54c001680 con 0x7fe55c1538a0 2023-10-30T13:29:18.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 >> v1:172.21.15.46:6789/0 conn(0x7fe55c1538a0 legacy=0x7fe55c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:18.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe544003b40 0x7fe544003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:18.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c0cf160 0x7fe5440047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:18.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe544004d10 0x7fe544005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c0cad70 con 0x7fe544004d10 2023-10-30T13:29:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54400ba40 con 0x7fe544003b40 2023-10-30T13:29:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.028+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54400bbb0 con 0x7fe55c0cf160 2023-10-30T13:29:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe560e08640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe544004d10 0x7fe544005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe55b7fe640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c0cf160 0x7fe5440047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:18.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe55bfff640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe544003b40 0x7fe544003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:18.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe560e08640 1 -- 172.21.15.46:0/747735204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c0cf160 msgr2=0x7fe5440047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:18.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe560e08640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c0cf160 0x7fe5440047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:18.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe560e08640 1 -- 172.21.15.46:0/747735204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe544003b40 msgr2=0x7fe544003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:18.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe560e08640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe544003b40 0x7fe544003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:18.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe560e08640 1 -- 172.21.15.46:0/747735204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe544003530 con 0x7fe544004d10 2023-10-30T13:29:18.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe55bfff640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe544003b40 0x7fe544003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:18.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe560e08640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe544004d10 0x7fe544005190 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe554009fc0 tx=0x7fe55401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9558995e0219d3e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:18.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5540a5c70 con 0x7fe544004d10 2023-10-30T13:29:18.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55400f040 con 0x7fe544004d10 2023-10-30T13:29:18.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.032+0000 7fe55affd640 1 -- 172.21.15.46:0/747735204 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5540ad680 con 0x7fe544004d10 2023-10-30T13:29:18.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 -- 172.21.15.46:0/747735204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe544004d10 msgr2=0x7fe544005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:18.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 --2- 172.21.15.46:0/747735204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe544004d10 0x7fe544005190 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe554009fc0 tx=0x7fe55401cc60 comp rx=0 tx=0).stop 2023-10-30T13:29:18.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe560e08640 1 -- 172.21.15.46:0/747735204 reap_dead start 2023-10-30T13:29:18.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 -- 172.21.15.46:0/747735204 shutdown_connections 2023-10-30T13:29:18.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 -- 172.21.15.46:0/747735204 >> 172.21.15.46:0/747735204 conn(0x7fe55c06bfe0 msgr2=0x7fe55c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:18.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 -- 172.21.15.46:0/747735204 shutdown_connections 2023-10-30T13:29:18.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 -- 172.21.15.46:0/747735204 wait complete. 2023-10-30T13:29:18.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 Processor -- start 2023-10-30T13:29:18.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 -- start start 2023-10-30T13:29:18.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c158940 0x7fe55c0c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:18.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe55c0ced10 0x7fe55c0c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:18.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe55c158d20 0x7fe55c14fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:18.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c1ef420 con 0x7fe55c158d20 2023-10-30T13:29:18.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c1ef6c0 con 0x7fe55c0ced10 2023-10-30T13:29:18.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe56286b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c1ef9a0 con 0x7fe55c158940 2023-10-30T13:29:18.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe560e08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe55c158d20 0x7fe55c14fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:18.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe560e08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe55c158d20 0x7fe55c14fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49912/0 (socket says 172.21.15.46:49912) 2023-10-30T13:29:18.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe560e08640 1 -- 172.21.15.46:0/3150395902 learned_addr learned my addr 172.21.15.46:0/3150395902 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe55b7fe640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe55c0ced10 0x7fe55c0c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe55bfff640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c158940 0x7fe55c0c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe560e08640 1 -- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c158940 msgr2=0x7fe55c0c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe560e08640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c158940 0x7fe55c0c4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe560e08640 1 -- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe55c0ced10 msgr2=0x7fe55c0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe560e08640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe55c0ced10 0x7fe55c0c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.036+0000 7fe560e08640 1 -- 172.21.15.46:0/3150395902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c1efc80 con 0x7fe55c158d20 2023-10-30T13:29:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.040+0000 7fe560e08640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe55c158d20 0x7fe55c14fba0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe554002240 tx=0x7fe554009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.040+0000 7fe558ff9640 1 -- 172.21.15.46:0/3150395902 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5540aead0 con 0x7fe55c158d20 2023-10-30T13:29:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.040+0000 7fe558ff9640 1 -- 172.21.15.46:0/3150395902 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5540aec30 con 0x7fe55c158d20 2023-10-30T13:29:18.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.040+0000 7fe558ff9640 1 -- 172.21.15.46:0/3150395902 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5540039a0 con 0x7fe55c158d20 2023-10-30T13:29:18.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.040+0000 7fe56286b640 1 -- 172.21.15.46:0/3150395902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55c0cdf90 con 0x7fe55c158d20 2023-10-30T13:29:18.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.040+0000 7fe56286b640 1 -- 172.21.15.46:0/3150395902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55c0cda90 con 0x7fe55c158d20 2023-10-30T13:29:18.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.040+0000 7fe56286b640 1 -- 172.21.15.46:0/3150395902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55c1efe10 con 0x7fe55c158d20 2023-10-30T13:29:18.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.040+0000 7fe558ff9640 1 -- 172.21.15.46:0/3150395902 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5540b4040 con 0x7fe55c158d20 2023-10-30T13:29:18.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.044+0000 7fe558ff9640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe5480664a0 0x7fe548068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:18.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.044+0000 7fe558ff9640 1 -- 172.21.15.46:0/3150395902 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fe55411dd20 con 0x7fe55c158d20 2023-10-30T13:29:18.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.044+0000 7fe55bfff640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe5480664a0 0x7fe548068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:18.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.044+0000 7fe55bfff640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe5480664a0 0x7fe548068960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe54c004850 tx=0x7fe54c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:18.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.048+0000 7fe558ff9640 1 -- 172.21.15.46:0/3150395902 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5540eb190 con 0x7fe55c158d20 2023-10-30T13:29:18.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.236+0000 7fe56286b640 1 -- 172.21.15.46:0/3150395902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe55c159b40 con 0x7fe55c158d20 2023-10-30T13:29:18.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.236+0000 7fe558ff9640 1 -- 172.21.15.46:0/3150395902 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fe5540be090 con 0x7fe55c158d20 2023-10-30T13:29:18.243 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:18.244 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:14.210718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:resolve","state_seq":37,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:18.244 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 30 2023-10-30T13:29:18.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.236+0000 7fe56286b640 1 -- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe5480664a0 msgr2=0x7fe548068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:18.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.236+0000 7fe56286b640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe5480664a0 0x7fe548068960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe54c004850 tx=0x7fe54c009040 comp rx=0 tx=0).stop 2023-10-30T13:29:18.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.236+0000 7fe56286b640 1 -- 172.21.15.46:0/3150395902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe55c158d20 msgr2=0x7fe55c14fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.236+0000 7fe56286b640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe55c158d20 0x7fe55c14fba0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe554002240 tx=0x7fe554009750 comp rx=0 tx=0).stop 2023-10-30T13:29:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.236+0000 7fe56286b640 1 -- 172.21.15.46:0/3150395902 shutdown_connections 2023-10-30T13:29:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.236+0000 7fe56286b640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe5480664a0 0x7fe548068960 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.236+0000 7fe56286b640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe55c158d20 0x7fe55c14fba0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.236+0000 7fe56286b640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe55c0ced10 0x7fe55c0c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.236+0000 7fe56286b640 1 --2- 172.21.15.46:0/3150395902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe55c158940 0x7fe55c0c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.236+0000 7fe56286b640 1 -- 172.21.15.46:0/3150395902 >> 172.21.15.46:0/3150395902 conn(0x7fe55c06bfe0 msgr2=0x7fe55c0b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.240+0000 7fe56286b640 1 -- 172.21.15.46:0/3150395902 shutdown_connections 2023-10-30T13:29:18.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:18.240+0000 7fe56286b640 1 -- 172.21.15.46:0/3150395902 wait complete. 2023-10-30T13:29:18.264 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:29:20.265 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:20.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb11f1d640 1 Processor -- start 2023-10-30T13:29:20.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb11f1d640 1 -- start start 2023-10-30T13:29:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb11f1d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdb0c0c05d0 0x7fdb0c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb11f1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb0c0c6f60 0x7fdb0c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb11f1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c0c0a50 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb11f1d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb0c06f6e0 con 0x7fdb0c0c05d0 2023-10-30T13:29:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb11f1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c06f9c0 con 0x7fdb0c0c6f60 2023-10-30T13:29:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb0affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdb0c0c05d0 0x7fdb0c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb0b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdb0c0c6b80 0x7fdb0c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35998/0 (socket says 172.21.15.46:35998) 2023-10-30T13:29:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb0affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdb0c0c05d0 0x7fdb0c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49922/0 (socket says 172.21.15.46:49922) 2023-10-30T13:29:20.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb0b7fe640 1 -- 172.21.15.46:0/3293995538 learned_addr learned my addr 172.21.15.46:0/3293995538 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:20.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb0bfff640 1 --2- 172.21.15.46:0/3293995538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb0c0c6f60 0x7fdb0c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:20.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb0affd640 1 -- 172.21.15.46:0/3293995538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb0c0c6f60 msgr2=0x7fdb0c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:20.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb0affd640 1 --2- 172.21.15.46:0/3293995538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb0c0c6f60 0x7fdb0c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:20.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb0a7fc640 1 -- 172.21.15.46:0/3293995538 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151813147 0 0) 0x7fdb0c0c0a50 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb0affd640 1 -- 172.21.15.46:0/3293995538 >> v1:172.21.15.46:6789/0 conn(0x7fdb0c0c6b80 legacy=0x7fdb0c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:20.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.416+0000 7fdb0affd640 1 -- 172.21.15.46:0/3293995538 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c06fca0 con 0x7fdb0c0c05d0 2023-10-30T13:29:20.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.420+0000 7fdb0affd640 1 --2- 172.21.15.46:0/3293995538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb0c0c05d0 0x7fdb0c13ed90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdb000097c0 tx=0x7fdb0001e690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cdcc52d70b7f8f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.420+0000 7fdb0a7fc640 1 -- 172.21.15.46:0/3293995538 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb000ab670 con 0x7fdb0c0c05d0 2023-10-30T13:29:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.420+0000 7fdb0a7fc640 1 -- 172.21.15.46:0/3293995538 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb000ab7d0 con 0x7fdb0c0c05d0 2023-10-30T13:29:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.420+0000 7fdb0a7fc640 1 -- 172.21.15.46:0/3293995538 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb000abb60 con 0x7fdb0c0c05d0 2023-10-30T13:29:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.420+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3293995538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb0c0c05d0 msgr2=0x7fdb0c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:20.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.420+0000 7fdb11f1d640 1 --2- 172.21.15.46:0/3293995538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb0c0c05d0 0x7fdb0c13ed90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdb000097c0 tx=0x7fdb0001e690 comp rx=0 tx=0).stop 2023-10-30T13:29:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.420+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3293995538 shutdown_connections 2023-10-30T13:29:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.420+0000 7fdb11f1d640 1 --2- 172.21.15.46:0/3293995538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb0c0c6f60 0x7fdb0c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.420+0000 7fdb11f1d640 1 --2- 172.21.15.46:0/3293995538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb0c0c05d0 0x7fdb0c13ed90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.420+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3293995538 >> 172.21.15.46:0/3293995538 conn(0x7fdb0c0c7910 msgr2=0x7fdb0c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:20.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.420+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3293995538 shutdown_connections 2023-10-30T13:29:20.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.420+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3293995538 wait complete. 2023-10-30T13:29:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb11f1d640 1 Processor -- start 2023-10-30T13:29:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb11f1d640 1 -- start start 2023-10-30T13:29:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb11f1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb0c0c05d0 0x7fdb0c154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb11f1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb0c0c6b80 0x7fdb0c1522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:20.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb11f1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb0c0c6f60 0x7fdb0c154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb11f1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c0caaf0 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb11f1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c0c9e60 con 0x7fdb0c0c6f60 2023-10-30T13:29:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb11f1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c0ca850 con 0x7fdb0c0c05d0 2023-10-30T13:29:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb0c0c6b80 0x7fdb0c1522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:20.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb0c0c6b80 0x7fdb0c1522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49924/0 (socket says 172.21.15.46:49924) 2023-10-30T13:29:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0affd640 1 -- 172.21.15.46:0/3816229904 learned_addr learned my addr 172.21.15.46:0/3816229904 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0b7fe640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb0c0c05d0 0x7fdb0c154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0bfff640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb0c0c6f60 0x7fdb0c154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0affd640 1 -- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb0c0c05d0 msgr2=0x7fdb0c154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:20.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0affd640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb0c0c05d0 0x7fdb0c154480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0affd640 1 -- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb0c0c6f60 msgr2=0x7fdb0c154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0affd640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb0c0c6f60 0x7fdb0c154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0affd640 1 -- 172.21.15.46:0/3816229904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb00009420 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0b7fe640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb0c0c05d0 0x7fdb0c154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:20.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0bfff640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb0c0c6f60 0x7fdb0c154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb0affd640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb0c0c6b80 0x7fdb0c1522e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdb000ad040 tx=0x7fdb0001ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdaf3fff640 1 -- 172.21.15.46:0/3816229904 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb00004230 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdaf3fff640 1 -- 172.21.15.46:0/3816229904 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb00004390 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdaf3fff640 1 -- 172.21.15.46:0/3816229904 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb00003860 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3816229904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0c0c95c0 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.424+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3816229904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0c0c9b80 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.428+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3816229904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0c04e170 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.428+0000 7fdaf3fff640 1 -- 172.21.15.46:0/3816229904 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdb000039c0 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.428+0000 7fdaf3fff640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdaf80664a0 0x7fdaf8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.428+0000 7fdaf3fff640 1 -- 172.21.15.46:0/3816229904 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fdb0011f0f0 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.428+0000 7fdb0b7fe640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdaf80664a0 0x7fdaf8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.428+0000 7fdb0b7fe640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdaf80664a0 0x7fdaf8068960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdafc007fd0 tx=0x7fdafc007c20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:20.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.432+0000 7fdaf3fff640 1 -- 172.21.15.46:0/3816229904 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdb000ec2d0 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.620+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3816229904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb0c070b80 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.620+0000 7fdaf3fff640 1 -- 172.21.15.46:0/3816229904 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fdb000c3090 con 0x7fdb0c0c6b80 2023-10-30T13:29:20.629 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:20.630 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:14.210718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:resolve","state_seq":37,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:20.630 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 30 2023-10-30T13:29:20.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.624+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdaf80664a0 msgr2=0x7fdaf8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:20.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.624+0000 7fdb11f1d640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdaf80664a0 0x7fdaf8068960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdafc007fd0 tx=0x7fdafc007c20 comp rx=0 tx=0).stop 2023-10-30T13:29:20.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.624+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3816229904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb0c0c6b80 msgr2=0x7fdb0c1522e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:20.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.624+0000 7fdb11f1d640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb0c0c6b80 0x7fdb0c1522e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdb000ad040 tx=0x7fdb0001ced0 comp rx=0 tx=0).stop 2023-10-30T13:29:20.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.624+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3816229904 shutdown_connections 2023-10-30T13:29:20.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.624+0000 7fdb11f1d640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdaf80664a0 0x7fdaf8068960 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:20.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.624+0000 7fdb11f1d640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb0c0c6f60 0x7fdb0c154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:20.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.624+0000 7fdb11f1d640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb0c0c6b80 0x7fdb0c1522e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:20.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.624+0000 7fdb11f1d640 1 --2- 172.21.15.46:0/3816229904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb0c0c05d0 0x7fdb0c154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:20.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.624+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3816229904 >> 172.21.15.46:0/3816229904 conn(0x7fdb0c0c7910 msgr2=0x7fdb0c0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.624+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3816229904 shutdown_connections 2023-10-30T13:29:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:20.624+0000 7fdb11f1d640 1 -- 172.21.15.46:0/3816229904 wait complete. 2023-10-30T13:29:20.650 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:29:22.652 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:22.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.800+0000 7f0fc7b65640 1 Processor -- start 2023-10-30T13:29:22.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.800+0000 7f0fc7b65640 1 -- start start 2023-10-30T13:29:22.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.800+0000 7f0fc7b65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fc0153bf0 0x7f0fc0071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:22.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.800+0000 7f0fc7b65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc00c6fe0 0x7f0fc00c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:22.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.800+0000 7f0fc7b65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc00c60e0 con 0x7f0fc00cb550 2023-10-30T13:29:22.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.800+0000 7f0fc7b65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc015d870 con 0x7f0fc0153bf0 2023-10-30T13:29:22.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.800+0000 7f0fc7b65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00c63a0 con 0x7f0fc00c6fe0 2023-10-30T13:29:22.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.800+0000 7f0fc5901640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fc0153bf0 0x7f0fc0071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:22.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.800+0000 7f0fc5901640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fc0153bf0 0x7f0fc0071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49930/0 (socket says 172.21.15.46:49930) 2023-10-30T13:29:22.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.800+0000 7f0fc5901640 1 -- 172.21.15.46:0/526345741 learned_addr learned my addr 172.21.15.46:0/526345741 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:22.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.804+0000 7f0fc5100640 1 --2- 172.21.15.46:0/526345741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc00c6fe0 0x7f0fc00c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:22.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.804+0000 7f0fc5901640 1 -- 172.21.15.46:0/526345741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc00c6fe0 msgr2=0x7f0fc00c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:22.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.804+0000 7f0fc5901640 1 --2- 172.21.15.46:0/526345741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc00c6fe0 0x7f0fc00c7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:22.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.804+0000 7f0fc5901640 1 -- 172.21.15.46:0/526345741 >> v1:172.21.15.154:6789/0 conn(0x7f0fc00cb550 legacy=0x7f0fc0159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:22.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.804+0000 7f0fc5901640 1 -- 172.21.15.46:0/526345741 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc00c6680 con 0x7f0fc0153bf0 2023-10-30T13:29:22.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.804+0000 7f0fc5901640 1 --2- 172.21.15.46:0/526345741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc0153bf0 0x7f0fc0071260 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009870 tx=0x7f0fbc01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d90839a7f381f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:22.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.804+0000 7f0fc48ff640 1 -- 172.21.15.46:0/526345741 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc015a820 con 0x7f0fc0153bf0 2023-10-30T13:29:22.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.804+0000 7f0fc48ff640 1 -- 172.21.15.46:0/526345741 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc0aa670 con 0x7f0fc0153bf0 2023-10-30T13:29:22.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.804+0000 7f0fc48ff640 1 -- 172.21.15.46:0/526345741 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc0aaa00 con 0x7f0fc0153bf0 2023-10-30T13:29:22.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 -- 172.21.15.46:0/526345741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc0153bf0 msgr2=0x7f0fc0071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:22.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 --2- 172.21.15.46:0/526345741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc0153bf0 0x7f0fc0071260 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009870 tx=0x7f0fbc01c510 comp rx=0 tx=0).stop 2023-10-30T13:29:22.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 -- 172.21.15.46:0/526345741 shutdown_connections 2023-10-30T13:29:22.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 --2- 172.21.15.46:0/526345741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc00c6fe0 0x7f0fc00c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:22.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 --2- 172.21.15.46:0/526345741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc0153bf0 0x7f0fc0071260 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:22.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 -- 172.21.15.46:0/526345741 >> 172.21.15.46:0/526345741 conn(0x7f0fc006c6e0 msgr2=0x7f0fc0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:22.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 -- 172.21.15.46:0/526345741 shutdown_connections 2023-10-30T13:29:22.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 -- 172.21.15.46:0/526345741 wait complete. 2023-10-30T13:29:22.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 Processor -- start 2023-10-30T13:29:22.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 -- start start 2023-10-30T13:29:22.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc00c6fe0 0x7f0fc014cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:22.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc00cb550 0x7f0fc0159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:22.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0153bf0 0x7f0fc01564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:22.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc0151bf0 con 0x7f0fc00c6fe0 2023-10-30T13:29:22.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc0156030 con 0x7f0fc00cb550 2023-10-30T13:29:22.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc7b65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc014bcd0 con 0x7f0fc0153bf0 2023-10-30T13:29:22.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc5901640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc00c6fe0 0x7f0fc014cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:22.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc5901640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc00c6fe0 0x7f0fc014cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49940/0 (socket says 172.21.15.46:49940) 2023-10-30T13:29:22.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc5901640 1 -- 172.21.15.46:0/985014648 learned_addr learned my addr 172.21.15.46:0/985014648 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:22.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc5100640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc00cb550 0x7f0fc0159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:22.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc6102640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0153bf0 0x7f0fc01564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:22.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc5901640 1 -- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0153bf0 msgr2=0x7f0fc01564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:22.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc5901640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0153bf0 0x7f0fc01564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:22.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc5901640 1 -- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc00cb550 msgr2=0x7f0fc0159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:22.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc5901640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc00cb550 0x7f0fc0159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:22.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc5901640 1 -- 172.21.15.46:0/985014648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc0094d0 con 0x7f0fc00c6fe0 2023-10-30T13:29:22.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.808+0000 7f0fc5901640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc00c6fe0 0x7f0fc014cbc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc01a140 tx=0x7f0fbc01ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:22.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.812+0000 7f0fb67fc640 1 -- 172.21.15.46:0/985014648 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc0b3800 con 0x7f0fc00c6fe0 2023-10-30T13:29:22.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.812+0000 7f0fb67fc640 1 -- 172.21.15.46:0/985014648 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc0b3960 con 0x7f0fc00c6fe0 2023-10-30T13:29:22.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.812+0000 7f0fb67fc640 1 -- 172.21.15.46:0/985014648 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc0b3d10 con 0x7f0fc00c6fe0 2023-10-30T13:29:22.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.812+0000 7f0fc7b65640 1 -- 172.21.15.46:0/985014648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc014bf90 con 0x7f0fc00c6fe0 2023-10-30T13:29:22.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.812+0000 7f0fc7b65640 1 -- 172.21.15.46:0/985014648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc0155cc0 con 0x7f0fc00c6fe0 2023-10-30T13:29:22.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.812+0000 7f0fc7b65640 1 -- 172.21.15.46:0/985014648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc004e170 con 0x7f0fc00c6fe0 2023-10-30T13:29:22.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.812+0000 7f0fb67fc640 1 -- 172.21.15.46:0/985014648 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0fbc0b2380 con 0x7f0fc00c6fe0 2023-10-30T13:29:22.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.812+0000 7f0fb67fc640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0fa40664d0 0x7f0fa4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:22.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.812+0000 7f0fb67fc640 1 -- 172.21.15.46:0/985014648 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f0fbc11e850 con 0x7f0fc00c6fe0 2023-10-30T13:29:22.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.812+0000 7f0fc5100640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0fa40664d0 0x7f0fa4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:22.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.816+0000 7f0fc5100640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0fa40664d0 0x7f0fa4068990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0159ae0 tx=0x7f0fb0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:22.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:22.816+0000 7f0fb67fc640 1 -- 172.21.15.46:0/985014648 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0fbc0ebb70 con 0x7f0fc00c6fe0 2023-10-30T13:29:23.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fc7b65640 1 -- 172.21.15.46:0/985014648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fc01572a0 con 0x7f0fc00c6fe0 2023-10-30T13:29:23.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fb67fc640 1 -- 172.21.15.46:0/985014648 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f0fbc0ef190 con 0x7f0fc00c6fe0 2023-10-30T13:29:23.015 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:23.016 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:19.980413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:reconnect","state_seq":39,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:23.016 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2023-10-30T13:29:23.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fc7b65640 1 -- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0fa40664d0 msgr2=0x7f0fa4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:23.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fc7b65640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0fa40664d0 0x7f0fa4068990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0159ae0 tx=0x7f0fb0008040 comp rx=0 tx=0).stop 2023-10-30T13:29:23.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fc7b65640 1 -- 172.21.15.46:0/985014648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc00c6fe0 msgr2=0x7f0fc014cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:23.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fc7b65640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc00c6fe0 0x7f0fc014cbc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc01a140 tx=0x7f0fbc01ad50 comp rx=0 tx=0).stop 2023-10-30T13:29:23.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fc7b65640 1 -- 172.21.15.46:0/985014648 shutdown_connections 2023-10-30T13:29:23.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fc7b65640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0fa40664d0 0x7f0fa4068990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:23.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fc7b65640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0153bf0 0x7f0fc01564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:23.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fc7b65640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc00cb550 0x7f0fc0159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:23.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fc7b65640 1 --2- 172.21.15.46:0/985014648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc00c6fe0 0x7f0fc014cbc0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:23.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fc7b65640 1 -- 172.21.15.46:0/985014648 >> 172.21.15.46:0/985014648 conn(0x7f0fc006c6e0 msgr2=0x7f0fc0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:23.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.008+0000 7f0fc7b65640 1 -- 172.21.15.46:0/985014648 shutdown_connections 2023-10-30T13:29:23.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:23.012+0000 7f0fc7b65640 1 -- 172.21.15.46:0/985014648 wait complete. 2023-10-30T13:29:23.036 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 31, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 31, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4306": { "addr": "172.21.15.46:6836/275749119", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 275749119, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 275749119, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4950": { "addr": "172.21.15.46:6837/1315246492", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 1315246492, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 1315246492, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4950, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_5106": { "addr": "172.21.15.46:6833/603884510", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 603884510, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 603884510, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5106, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:reconnect", "state_seq": 39 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:29:19.980413+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 5106, "mds_2": 4950 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:29:25.037 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.184+0000 7f814233e640 1 Processor -- start 2023-10-30T13:29:25.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.184+0000 7f814233e640 1 -- start start 2023-10-30T13:29:25.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f814233e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1538a0 0x7f813c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:25.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f814233e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c0cb800 con 0x7f813c0cf160 2023-10-30T13:29:25.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f814233e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c15dc30 con 0x7f813c0ca0e0 2023-10-30T13:29:25.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f814233e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0cb330 con 0x7f813c1538a0 2023-10-30T13:29:25.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f81408db640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f813c0cf160 0x7f813c164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36006/0 (socket says 172.21.15.46:36006) 2023-10-30T13:29:25.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f81408db640 1 -- 172.21.15.46:0/2605326842 learned_addr learned my addr 172.21.15.46:0/2605326842 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:25.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813bfff640 1 --2- 172.21.15.46:0/2605326842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1538a0 0x7f813c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:25.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542945022 0 0) 0x7f813c0cb800 con 0x7f813c0cf160 2023-10-30T13:29:25.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8124003530 con 0x7f813c0cf160 2023-10-30T13:29:25.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1939745297 0 0) 0x7f8124003530 con 0x7f813c0cf160 2023-10-30T13:29:25.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1538a0 msgr2=0x7f813c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:25.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 --2- 172.21.15.46:0/2605326842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1538a0 0x7f813c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:25.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 >> v1:172.21.15.154:6789/0 conn(0x7f813c0ca0e0 legacy=0x7f813c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:25.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c0cad70 con 0x7f813c0cf160 2023-10-30T13:29:25.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f8134001a00 con 0x7f813c0cf160 2023-10-30T13:29:25.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 >> v1:172.21.15.46:6789/0 conn(0x7f813c0cf160 legacy=0x7f813c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:25.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 --2- 172.21.15.46:0/2605326842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81240039b0 0x7f8124003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:25.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 --2- 172.21.15.46:0/2605326842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81240047d0 0x7f8124004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:25.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 --2- 172.21.15.46:0/2605326842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1538a0 0x7f8124005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:25.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0cb330 con 0x7f81240047d0 2023-10-30T13:29:25.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812400bad0 con 0x7f81240039b0 2023-10-30T13:29:25.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812400bc90 con 0x7f813c1538a0 2023-10-30T13:29:25.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813b7fe640 1 --2- 172.21.15.46:0/2605326842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81240047d0 0x7f8124004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:25.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813bfff640 1 --2- 172.21.15.46:0/2605326842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81240039b0 0x7f8124003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:25.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f81408db640 1 --2- 172.21.15.46:0/2605326842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1538a0 0x7f8124005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:25.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813b7fe640 1 -- 172.21.15.46:0/2605326842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1538a0 msgr2=0x7f8124005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:25.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813b7fe640 1 --2- 172.21.15.46:0/2605326842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1538a0 0x7f8124005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:25.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813b7fe640 1 -- 172.21.15.46:0/2605326842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81240039b0 msgr2=0x7f8124003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:25.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813b7fe640 1 --2- 172.21.15.46:0/2605326842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81240039b0 0x7f8124003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:25.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.188+0000 7f813b7fe640 1 -- 172.21.15.46:0/2605326842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8124003530 con 0x7f81240047d0 2023-10-30T13:29:25.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.192+0000 7f813b7fe640 1 --2- 172.21.15.46:0/2605326842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81240047d0 0x7f8124004c50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f81300015b0 tx=0x7f813001c200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9153dc506bfd4162 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:25.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.192+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81300052d0 con 0x7f81240047d0 2023-10-30T13:29:25.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.192+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81300a7050 con 0x7f81240047d0 2023-10-30T13:29:25.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.192+0000 7f813affd640 1 -- 172.21.15.46:0/2605326842 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130003600 con 0x7f81240047d0 2023-10-30T13:29:25.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.192+0000 7f814233e640 1 -- 172.21.15.46:0/2605326842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81240047d0 msgr2=0x7f8124004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:25.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.192+0000 7f814233e640 1 --2- 172.21.15.46:0/2605326842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81240047d0 0x7f8124004c50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f81300015b0 tx=0x7f813001c200 comp rx=0 tx=0).stop 2023-10-30T13:29:25.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.192+0000 7f81408db640 1 -- 172.21.15.46:0/2605326842 reap_dead start 2023-10-30T13:29:25.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.192+0000 7f814233e640 1 -- 172.21.15.46:0/2605326842 shutdown_connections 2023-10-30T13:29:25.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.192+0000 7f814233e640 1 -- 172.21.15.46:0/2605326842 >> 172.21.15.46:0/2605326842 conn(0x7f813c06bfe0 msgr2=0x7f813c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:25.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.192+0000 7f814233e640 1 -- 172.21.15.46:0/2605326842 shutdown_connections 2023-10-30T13:29:25.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.192+0000 7f814233e640 1 -- 172.21.15.46:0/2605326842 wait complete. 2023-10-30T13:29:25.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.192+0000 7f814233e640 1 Processor -- start 2023-10-30T13:29:25.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f814233e640 1 -- start start 2023-10-30T13:29:25.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f814233e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81240047d0 0x7f813c1efdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:25.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f814233e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1e6770 0x7f813c0b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:25.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f814233e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f813c1e31b0 0x7f813c145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:25.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f814233e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c1ef350 con 0x7f81240047d0 2023-10-30T13:29:25.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f814233e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c1f3360 con 0x7f813c1e31b0 2023-10-30T13:29:25.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f814233e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c1ef5f0 con 0x7f813c1e6770 2023-10-30T13:29:25.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f813bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81240047d0 0x7f813c1efdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:25.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f813bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81240047d0 0x7f813c1efdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49968/0 (socket says 172.21.15.46:49968) 2023-10-30T13:29:25.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f813bfff640 1 -- 172.21.15.46:0/1637159832 learned_addr learned my addr 172.21.15.46:0/1637159832 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:25.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f813b7fe640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1e6770 0x7f813c0b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:25.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f81408db640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f813c1e31b0 0x7f813c145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:25.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f813bfff640 1 -- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1e6770 msgr2=0x7f813c0b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:25.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f813bfff640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1e6770 0x7f813c0b4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:25.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f813bfff640 1 -- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f813c1e31b0 msgr2=0x7f813c145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:25.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f813bfff640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f813c1e31b0 0x7f813c145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:25.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f813bfff640 1 -- 172.21.15.46:0/1637159832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c1ef8d0 con 0x7f81240047d0 2023-10-30T13:29:25.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f813b7fe640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1e6770 0x7f813c0b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:25.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f813bfff640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81240047d0 0x7f813c1efdf0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f812c0119d0 tx=0x7f812c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:25.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f8138ff9640 1 -- 172.21.15.46:0/1637159832 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c00fb50 con 0x7f81240047d0 2023-10-30T13:29:25.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f8138ff9640 1 -- 172.21.15.46:0/1637159832 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c00fcb0 con 0x7f81240047d0 2023-10-30T13:29:25.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f8138ff9640 1 -- 172.21.15.46:0/1637159832 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c014440 con 0x7f81240047d0 2023-10-30T13:29:25.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f814233e640 1 -- 172.21.15.46:0/1637159832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c1f3640 con 0x7f81240047d0 2023-10-30T13:29:25.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f814233e640 1 -- 172.21.15.46:0/1637159832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c0b3910 con 0x7f81240047d0 2023-10-30T13:29:25.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.196+0000 7f814233e640 1 -- 172.21.15.46:0/1637159832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c04e170 con 0x7f81240047d0 2023-10-30T13:29:25.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.200+0000 7f8138ff9640 1 -- 172.21.15.46:0/1637159832 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f812c0145a0 con 0x7f81240047d0 2023-10-30T13:29:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.200+0000 7f8138ff9640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81280664d0 0x7f8128068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.200+0000 7f8138ff9640 1 -- 172.21.15.46:0/1637159832 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f812c084a60 con 0x7f81240047d0 2023-10-30T13:29:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.200+0000 7f813b7fe640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81280664d0 0x7f8128068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:25.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.200+0000 7f813b7fe640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81280664d0 0x7f8128068990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f81240057b0 tx=0x7f81300011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:25.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.204+0000 7f8138ff9640 1 -- 172.21.15.46:0/1637159832 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f812c051d00 con 0x7f81240047d0 2023-10-30T13:29:25.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.392+0000 7f814233e640 1 -- 172.21.15.46:0/1637159832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f813c1efa60 con 0x7f81240047d0 2023-10-30T13:29:25.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.392+0000 7f8138ff9640 1 -- 172.21.15.46:0/1637159832 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f812c022070 con 0x7f81240047d0 2023-10-30T13:29:25.401 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:25.401 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:19.980413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:reconnect","state_seq":39,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:25.402 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2023-10-30T13:29:25.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.396+0000 7f814233e640 1 -- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81280664d0 msgr2=0x7f8128068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:25.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.396+0000 7f814233e640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81280664d0 0x7f8128068990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f81240057b0 tx=0x7f81300011a0 comp rx=0 tx=0).stop 2023-10-30T13:29:25.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.396+0000 7f814233e640 1 -- 172.21.15.46:0/1637159832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81240047d0 msgr2=0x7f813c1efdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:25.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.396+0000 7f814233e640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81240047d0 0x7f813c1efdf0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f812c0119d0 tx=0x7f812c011e90 comp rx=0 tx=0).stop 2023-10-30T13:29:25.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.396+0000 7f814233e640 1 -- 172.21.15.46:0/1637159832 shutdown_connections 2023-10-30T13:29:25.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.396+0000 7f814233e640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81280664d0 0x7f8128068990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:25.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.396+0000 7f814233e640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f813c1e31b0 0x7f813c145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:25.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.396+0000 7f814233e640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f813c1e6770 0x7f813c0b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:25.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.396+0000 7f814233e640 1 --2- 172.21.15.46:0/1637159832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81240047d0 0x7f813c1efdf0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:25.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.396+0000 7f814233e640 1 -- 172.21.15.46:0/1637159832 >> 172.21.15.46:0/1637159832 conn(0x7f813c06bfe0 msgr2=0x7f813c1526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:25.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.396+0000 7f814233e640 1 -- 172.21.15.46:0/1637159832 shutdown_connections 2023-10-30T13:29:25.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:25.396+0000 7f814233e640 1 -- 172.21.15.46:0/1637159832 wait complete. 2023-10-30T13:29:25.422 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:29:27.423 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:27.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.568+0000 7ff1f9e79640 1 Processor -- start 2023-10-30T13:29:27.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.568+0000 7ff1f9e79640 1 -- start start 2023-10-30T13:29:27.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f9e79640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff1f41539a0 0x7ff1f415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f9e79640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1f4070a00 0x7ff1f4070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f9e79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f4069e90 0x7ff1f406a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f9e79640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1f406fbb0 con 0x7ff1f41539a0 2023-10-30T13:29:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f9e79640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1f415a8c0 con 0x7ff1f4070a00 2023-10-30T13:29:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f9e79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f40713b0 con 0x7ff1f4069e90 2023-10-30T13:29:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff1f41539a0 0x7ff1f415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff1f41539a0 0x7ff1f415a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42086/0 (socket says 172.21.15.46:42086) 2023-10-30T13:29:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f37fe640 1 -- 172.21.15.46:0/617020890 learned_addr learned my addr 172.21.15.46:0/617020890 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f3fff640 1 --2- 172.21.15.46:0/617020890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f4069e90 0x7ff1f406a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f2ffd640 1 --2- 172.21.15.46:0/617020890 >> v2:172.21.15.154:3300/0 conn(0x7ff1f4070a00 0x7ff1f4070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:27.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f37fe640 1 -- 172.21.15.46:0/617020890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f4069e90 msgr2=0x7ff1f406a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:27.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f37fe640 1 --2- 172.21.15.46:0/617020890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f4069e90 0x7ff1f406a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:27.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f37fe640 1 -- 172.21.15.46:0/617020890 >> v2:172.21.15.154:3300/0 conn(0x7ff1f4070a00 msgr2=0x7ff1f4070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:27.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f37fe640 1 --2- 172.21.15.46:0/617020890 >> v2:172.21.15.154:3300/0 conn(0x7ff1f4070a00 0x7ff1f4070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:27.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.572+0000 7ff1f37fe640 1 -- 172.21.15.46:0/617020890 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4153d80 con 0x7ff1f41539a0 2023-10-30T13:29:27.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f37fe640 1 --2- 172.21.15.46:0/617020890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1f41539a0 0x7ff1f415a380 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4072640 tx=0x7ff1e401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617e796b66b2862d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:27.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f27fc640 1 -- 172.21.15.46:0/617020890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e40ac640 con 0x7ff1f41539a0 2023-10-30T13:29:27.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f27fc640 1 -- 172.21.15.46:0/617020890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e40ac7a0 con 0x7ff1f41539a0 2023-10-30T13:29:27.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f27fc640 1 -- 172.21.15.46:0/617020890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e40acb30 con 0x7ff1f41539a0 2023-10-30T13:29:27.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f9e79640 1 -- 172.21.15.46:0/617020890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1f41539a0 msgr2=0x7ff1f415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:27.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f9e79640 1 --2- 172.21.15.46:0/617020890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1f41539a0 0x7ff1f415a380 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4072640 tx=0x7ff1e401c440 comp rx=0 tx=0).stop 2023-10-30T13:29:27.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f9e79640 1 -- 172.21.15.46:0/617020890 shutdown_connections 2023-10-30T13:29:27.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f9e79640 1 --2- 172.21.15.46:0/617020890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f4069e90 0x7ff1f406a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:27.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f9e79640 1 --2- 172.21.15.46:0/617020890 >> v2:172.21.15.154:3300/0 conn(0x7ff1f4070a00 0x7ff1f4070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:27.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f9e79640 1 --2- 172.21.15.46:0/617020890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1f41539a0 0x7ff1f415a380 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f9e79640 1 -- 172.21.15.46:0/617020890 >> 172.21.15.46:0/617020890 conn(0x7ff1f40c7910 msgr2=0x7ff1f40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f9e79640 1 -- 172.21.15.46:0/617020890 shutdown_connections 2023-10-30T13:29:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f9e79640 1 -- 172.21.15.46:0/617020890 wait complete. 2023-10-30T13:29:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f9e79640 1 Processor -- start 2023-10-30T13:29:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.576+0000 7ff1f9e79640 1 -- start start 2023-10-30T13:29:27.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f9e79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1f4069e90 0x7ff1f41509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:27.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f9e79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f4070a00 0x7ff1f4151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:27.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f9e79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f41539a0 0x7ff1f40ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:27.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f9e79640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f415c150 con 0x7ff1f4069e90 2023-10-30T13:29:27.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f9e79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f4150fd0 con 0x7ff1f41539a0 2023-10-30T13:29:27.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f9e79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f415b4d0 con 0x7ff1f4070a00 2023-10-30T13:29:27.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1f4069e90 0x7ff1f41509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:27.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1f4069e90 0x7ff1f41509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42088/0 (socket says 172.21.15.46:42088) 2023-10-30T13:29:27.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f37fe640 1 -- 172.21.15.46:0/1361313157 learned_addr learned my addr 172.21.15.46:0/1361313157 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:27.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f2ffd640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f4070a00 0x7ff1f4151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:27.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f3fff640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f41539a0 0x7ff1f40ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:27.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f37fe640 1 -- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f4070a00 msgr2=0x7ff1f4151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:27.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f37fe640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f4070a00 0x7ff1f4151620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:27.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f37fe640 1 -- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f41539a0 msgr2=0x7ff1f40ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:27.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f37fe640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f41539a0 0x7ff1f40ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:27.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f37fe640 1 -- 172.21.15.46:0/1361313157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e40094d0 con 0x7ff1f4069e90 2023-10-30T13:29:27.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f3fff640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f41539a0 0x7ff1f40ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:27.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f37fe640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1f4069e90 0x7ff1f41509b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff1e40af040 tx=0x7ff1e401ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:27.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1dbfff640 1 -- 172.21.15.46:0/1361313157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e40b7830 con 0x7ff1f4069e90 2023-10-30T13:29:27.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1dbfff640 1 -- 172.21.15.46:0/1361313157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e40b7990 con 0x7ff1f4069e90 2023-10-30T13:29:27.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1dbfff640 1 -- 172.21.15.46:0/1361313157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e40b7d20 con 0x7ff1f4069e90 2023-10-30T13:29:27.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f9e79640 1 -- 172.21.15.46:0/1361313157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f415b7b0 con 0x7ff1f4069e90 2023-10-30T13:29:27.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f9e79640 1 -- 172.21.15.46:0/1361313157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f415bd70 con 0x7ff1f4069e90 2023-10-30T13:29:27.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1f9e79640 1 -- 172.21.15.46:0/1361313157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f40ae0b0 con 0x7ff1f4069e90 2023-10-30T13:29:27.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.580+0000 7ff1dbfff640 1 -- 172.21.15.46:0/1361313157 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1e40b3040 con 0x7ff1f4069e90 2023-10-30T13:29:27.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.584+0000 7ff1dbfff640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1e00664a0 0x7ff1e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:27.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.584+0000 7ff1dbfff640 1 -- 172.21.15.46:0/1361313157 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7ff1e411eac0 con 0x7ff1f4069e90 2023-10-30T13:29:27.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.584+0000 7ff1f2ffd640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1e00664a0 0x7ff1e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:27.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.584+0000 7ff1f2ffd640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1e00664a0 0x7ff1e0068960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4152dd0 tx=0x7ff1e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:27.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.588+0000 7ff1dbfff640 1 -- 172.21.15.46:0/1361313157 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1e40ebde0 con 0x7ff1f4069e90 2023-10-30T13:29:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.780+0000 7ff1f9e79640 1 -- 172.21.15.46:0/1361313157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1f404e170 con 0x7ff1f4069e90 2023-10-30T13:29:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.780+0000 7ff1dbfff640 1 -- 172.21.15.46:0/1361313157 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5736 (secure 0 0 0) 0x7ff1e40ef400 con 0x7ff1f4069e90 2023-10-30T13:29:27.788 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:27.788 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:25.989306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:27.789 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 32 2023-10-30T13:29:27.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.780+0000 7ff1f9e79640 1 -- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1e00664a0 msgr2=0x7ff1e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.780+0000 7ff1f9e79640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1e00664a0 0x7ff1e0068960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4152dd0 tx=0x7ff1e8008040 comp rx=0 tx=0).stop 2023-10-30T13:29:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.780+0000 7ff1f9e79640 1 -- 172.21.15.46:0/1361313157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1f4069e90 msgr2=0x7ff1f41509b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.780+0000 7ff1f9e79640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1f4069e90 0x7ff1f41509b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff1e40af040 tx=0x7ff1e401ac80 comp rx=0 tx=0).stop 2023-10-30T13:29:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.780+0000 7ff1f9e79640 1 -- 172.21.15.46:0/1361313157 shutdown_connections 2023-10-30T13:29:27.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.780+0000 7ff1f9e79640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1e00664a0 0x7ff1e0068960 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:27.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.780+0000 7ff1f9e79640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f41539a0 0x7ff1f40ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:27.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.780+0000 7ff1f9e79640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f4070a00 0x7ff1f4151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:27.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.780+0000 7ff1f9e79640 1 --2- 172.21.15.46:0/1361313157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1f4069e90 0x7ff1f41509b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:27.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.780+0000 7ff1f9e79640 1 -- 172.21.15.46:0/1361313157 >> 172.21.15.46:0/1361313157 conn(0x7ff1f40c7910 msgr2=0x7ff1f41589c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:27.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.784+0000 7ff1f9e79640 1 -- 172.21.15.46:0/1361313157 shutdown_connections 2023-10-30T13:29:27.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:27.784+0000 7ff1f9e79640 1 -- 172.21.15.46:0/1361313157 wait complete. 2023-10-30T13:29:27.808 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:29:29.809 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb73dbb8640 1 Processor -- start 2023-10-30T13:29:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb73dbb8640 1 -- start start 2023-10-30T13:29:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb73dbb8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb7381538a0 0x7fb73815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb73dbb8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7380cb800 con 0x7fb7380ca0e0 2023-10-30T13:29:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb73dbb8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73815dc30 con 0x7fb7380cf160 2023-10-30T13:29:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb73dbb8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7380cb330 con 0x7fb7381538a0 2023-10-30T13:29:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb7377fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb7381538a0 0x7fb73815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb7377fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb7381538a0 0x7fb73815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42092/0 (socket says 172.21.15.46:42092) 2023-10-30T13:29:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb7377fe640 1 -- 172.21.15.46:0/62159147 learned_addr learned my addr 172.21.15.46:0/62159147 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb7377fe640 1 -- 172.21.15.46:0/62159147 >> v1:172.21.15.154:6789/0 conn(0x7fb7380cf160 legacy=0x7fb738164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:29.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb7377fe640 1 -- 172.21.15.46:0/62159147 >> v1:172.21.15.46:6789/0 conn(0x7fb7380ca0e0 legacy=0x7fb73815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:29.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb7377fe640 1 -- 172.21.15.46:0/62159147 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7380cad70 con 0x7fb7381538a0 2023-10-30T13:29:29.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.956+0000 7fb7367fc640 1 -- 172.21.15.46:0/62159147 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060928965 0 0) 0x7fb7380cb800 con 0x7fb7380ca0e0 2023-10-30T13:29:29.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.960+0000 7fb7377fe640 1 --2- 172.21.15.46:0/62159147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7381538a0 0x7fb73815d6f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb728009870 tx=0x7fb72801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d2bb8944e1fab2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:29.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.960+0000 7fb7367fc640 1 -- 172.21.15.46:0/62159147 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280a8a30 con 0x7fb7381538a0 2023-10-30T13:29:29.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.960+0000 7fb7367fc640 1 -- 172.21.15.46:0/62159147 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7280a8b90 con 0x7fb7381538a0 2023-10-30T13:29:29.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.960+0000 7fb7367fc640 1 -- 172.21.15.46:0/62159147 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7280b0440 con 0x7fb7381538a0 2023-10-30T13:29:29.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.960+0000 7fb73dbb8640 1 -- 172.21.15.46:0/62159147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7381538a0 msgr2=0x7fb73815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.960+0000 7fb73dbb8640 1 --2- 172.21.15.46:0/62159147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7381538a0 0x7fb73815d6f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb728009870 tx=0x7fb72801ed30 comp rx=0 tx=0).stop 2023-10-30T13:29:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.960+0000 7fb73dbb8640 1 -- 172.21.15.46:0/62159147 shutdown_connections 2023-10-30T13:29:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.960+0000 7fb73dbb8640 1 --2- 172.21.15.46:0/62159147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7381538a0 0x7fb73815d6f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.960+0000 7fb73dbb8640 1 -- 172.21.15.46:0/62159147 >> 172.21.15.46:0/62159147 conn(0x7fb73806bfe0 msgr2=0x7fb73806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:29.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.960+0000 7fb73dbb8640 1 -- 172.21.15.46:0/62159147 shutdown_connections 2023-10-30T13:29:29.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.960+0000 7fb73dbb8640 1 -- 172.21.15.46:0/62159147 wait complete. 2023-10-30T13:29:29.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb73dbb8640 1 Processor -- start 2023-10-30T13:29:29.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb73dbb8640 1 -- start start 2023-10-30T13:29:29.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb73dbb8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7380ca0e0 0x7fb738146180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:29.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb73dbb8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7380cf160 0x7fb7380ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:29.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb73dbb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7381538a0 0x7fb7380ce670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:29.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb73dbb8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7381466c0 con 0x7fb7381538a0 2023-10-30T13:29:29.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb73dbb8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7381e6600 con 0x7fb7380ca0e0 2023-10-30T13:29:29.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb73dbb8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb738157b60 con 0x7fb7380cf160 2023-10-30T13:29:29.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb737fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7381538a0 0x7fb7380ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:29.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb7377fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7380ca0e0 0x7fb738146180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:29.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb7377fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7380ca0e0 0x7fb738146180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:56862/0 (socket says 172.21.15.46:56862) 2023-10-30T13:29:29.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb7377fe640 1 -- 172.21.15.46:0/3209027106 learned_addr learned my addr 172.21.15.46:0/3209027106 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:29.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb737fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7381538a0 0x7fb7380ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42102/0 (socket says 172.21.15.46:42102) 2023-10-30T13:29:29.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb736ffd640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7380cf160 0x7fb7380ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:29.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb737fff640 1 -- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7380cf160 msgr2=0x7fb7380ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:29.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb737fff640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7380cf160 0x7fb7380ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:29.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb737fff640 1 -- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7380ca0e0 msgr2=0x7fb738146180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:29.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb737fff640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7380ca0e0 0x7fb738146180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:29.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb737fff640 1 -- 172.21.15.46:0/3209027106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7280094d0 con 0x7fb7381538a0 2023-10-30T13:29:29.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb737fff640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7381538a0 0x7fb7380ce670 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb73000c700 tx=0x7fb73000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:29.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb71ffff640 1 -- 172.21.15.46:0/3209027106 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730003b80 con 0x7fb7381538a0 2023-10-30T13:29:29.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb71ffff640 1 -- 172.21.15.46:0/3209027106 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb730003ce0 con 0x7fb7381538a0 2023-10-30T13:29:29.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb73dbb8640 1 -- 172.21.15.46:0/3209027106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb738157e40 con 0x7fb7381538a0 2023-10-30T13:29:29.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb71ffff640 1 -- 172.21.15.46:0/3209027106 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730005310 con 0x7fb7381538a0 2023-10-30T13:29:29.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.964+0000 7fb73dbb8640 1 -- 172.21.15.46:0/3209027106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7381e2ae0 con 0x7fb7381538a0 2023-10-30T13:29:29.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.968+0000 7fb73dbb8640 1 -- 172.21.15.46:0/3209027106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73804e170 con 0x7fb7381538a0 2023-10-30T13:29:29.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.968+0000 7fb71ffff640 1 -- 172.21.15.46:0/3209027106 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb730005470 con 0x7fb7381538a0 2023-10-30T13:29:29.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.968+0000 7fb71ffff640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7240664d0 0x7fb724068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:29.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.968+0000 7fb71ffff640 1 -- 172.21.15.46:0/3209027106 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fb7300839e0 con 0x7fb7381538a0 2023-10-30T13:29:29.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.968+0000 7fb7377fe640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7240664d0 0x7fb724068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:29.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.968+0000 7fb7377fe640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7240664d0 0x7fb724068990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb728009870 tx=0x7fb728009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:29.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:29.972+0000 7fb71ffff640 1 -- 172.21.15.46:0/3209027106 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb730050d80 con 0x7fb7381538a0 2023-10-30T13:29:30.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 -- 172.21.15.46:0/3209027106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb73815dc30 con 0x7fb7381538a0 2023-10-30T13:29:30.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb71ffff640 1 -- 172.21.15.46:0/3209027106 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5736 (secure 0 0 0) 0x7fb73815dc30 con 0x7fb7381538a0 2023-10-30T13:29:30.170 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:30.171 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:25.989306+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 32 2023-10-30T13:29:30.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 -- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7240664d0 msgr2=0x7fb724068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:30.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7240664d0 0x7fb724068990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb728009870 tx=0x7fb728009040 comp rx=0 tx=0).stop 2023-10-30T13:29:30.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 -- 172.21.15.46:0/3209027106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7381538a0 msgr2=0x7fb7380ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:30.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7381538a0 0x7fb7380ce670 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb73000c700 tx=0x7fb73000cbc0 comp rx=0 tx=0).stop 2023-10-30T13:29:30.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 -- 172.21.15.46:0/3209027106 shutdown_connections 2023-10-30T13:29:30.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7240664d0 0x7fb724068990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:30.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7381538a0 0x7fb7380ce670 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:30.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7380cf160 0x7fb7380ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:30.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 --2- 172.21.15.46:0/3209027106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7380ca0e0 0x7fb738146180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:30.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 -- 172.21.15.46:0/3209027106 >> 172.21.15.46:0/3209027106 conn(0x7fb73806bfe0 msgr2=0x7fb7381526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:30.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 -- 172.21.15.46:0/3209027106 shutdown_connections 2023-10-30T13:29:30.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:30.164+0000 7fb73dbb8640 1 -- 172.21.15.46:0/3209027106 wait complete. 2023-10-30T13:29:30.192 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:29:32.193 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:32.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.340+0000 7f8e8ab0f640 1 Processor -- start 2023-10-30T13:29:32.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.340+0000 7f8e8ab0f640 1 -- start start 2023-10-30T13:29:32.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e8ab0f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8e840c6b80 0x7f8e840bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:32.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e8ab0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e840c6f60 0x7f8e840c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:32.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e8ab0f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e840c0a50 con 0x7f8e840c8430 2023-10-30T13:29:32.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e8ab0f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8406f6e0 con 0x7f8e840c6b80 2023-10-30T13:29:32.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e8ab0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8406f9c0 con 0x7f8e840c6f60 2023-10-30T13:29:32.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e83fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e840c6f60 0x7f8e840c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:32.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e888ab640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8e840c6b80 0x7f8e840bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:32.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e890ac640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f8e840c8430 0x7f8e8413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:48714/0 (socket says 172.21.15.46:48714) 2023-10-30T13:29:32.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e890ac640 1 -- 172.21.15.46:0/3568755100 learned_addr learned my addr 172.21.15.46:0/3568755100 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:32.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e83fff640 1 -- 172.21.15.46:0/3568755100 >> v2:172.21.15.154:3300/0 conn(0x7f8e840c6b80 msgr2=0x7f8e840bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:32.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e83fff640 1 --2- 172.21.15.46:0/3568755100 >> v2:172.21.15.154:3300/0 conn(0x7f8e840c6b80 0x7f8e840bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:32.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e83fff640 1 -- 172.21.15.46:0/3568755100 >> v1:172.21.15.154:6789/0 conn(0x7f8e840c8430 legacy=0x7f8e8413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:32.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e83fff640 1 -- 172.21.15.46:0/3568755100 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8406fca0 con 0x7f8e840c6f60 2023-10-30T13:29:32.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.344+0000 7f8e888ab640 1 --2- 172.21.15.46:0/3568755100 >> v2:172.21.15.154:3300/0 conn(0x7f8e840c6b80 0x7f8e840bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:29:32.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.348+0000 7f8e83fff640 1 --2- 172.21.15.46:0/3568755100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e840c6f60 0x7f8e840c0400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000ba60 tx=0x7f8e7001eac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dac319fe12b5ee4e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:32.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.348+0000 7f8e837fe640 1 -- 172.21.15.46:0/3568755100 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e700a7aa0 con 0x7f8e840c6f60 2023-10-30T13:29:32.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.348+0000 7f8e837fe640 1 -- 172.21.15.46:0/3568755100 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e700a7c00 con 0x7f8e840c6f60 2023-10-30T13:29:32.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.348+0000 7f8e837fe640 1 -- 172.21.15.46:0/3568755100 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e700af440 con 0x7f8e840c6f60 2023-10-30T13:29:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.348+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/3568755100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e840c6f60 msgr2=0x7f8e840c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.348+0000 7f8e8ab0f640 1 --2- 172.21.15.46:0/3568755100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e840c6f60 0x7f8e840c0400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000ba60 tx=0x7f8e7001eac0 comp rx=0 tx=0).stop 2023-10-30T13:29:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.348+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/3568755100 shutdown_connections 2023-10-30T13:29:32.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.348+0000 7f8e8ab0f640 1 --2- 172.21.15.46:0/3568755100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e840c6f60 0x7f8e840c0400 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:32.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.348+0000 7f8e8ab0f640 1 --2- 172.21.15.46:0/3568755100 >> v2:172.21.15.154:3300/0 conn(0x7f8e840c6b80 0x7f8e840bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:32.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.348+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/3568755100 >> 172.21.15.46:0/3568755100 conn(0x7f8e840c7910 msgr2=0x7f8e840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:32.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.348+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/3568755100 shutdown_connections 2023-10-30T13:29:32.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.348+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/3568755100 wait complete. 2023-10-30T13:29:32.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e8ab0f640 1 Processor -- start 2023-10-30T13:29:32.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e8ab0f640 1 -- start start 2023-10-30T13:29:32.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e8ab0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e840c6b80 0x7f8e84130750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:32.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e8ab0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e840c6f60 0x7f8e840c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:32.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e8ab0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e840c8430 0x7f8e841f13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:32.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e8ab0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e84134260 con 0x7f8e840c6f60 2023-10-30T13:29:32.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e8ab0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e841336c0 con 0x7f8e840c6b80 2023-10-30T13:29:32.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e8ab0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e841eec20 con 0x7f8e840c8430 2023-10-30T13:29:32.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e83fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e840c6f60 0x7f8e840c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:32.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e83fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e840c6f60 0x7f8e840c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42106/0 (socket says 172.21.15.46:42106) 2023-10-30T13:29:32.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e83fff640 1 -- 172.21.15.46:0/173909396 learned_addr learned my addr 172.21.15.46:0/173909396 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:32.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e888ab640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e840c6b80 0x7f8e84130750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:32.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e890ac640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e840c8430 0x7f8e841f13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:32.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e83fff640 1 -- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e840c8430 msgr2=0x7f8e841f13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e83fff640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e840c8430 0x7f8e841f13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e83fff640 1 -- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e840c6b80 msgr2=0x7f8e84130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e83fff640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e840c6b80 0x7f8e84130750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e83fff640 1 -- 172.21.15.46:0/173909396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e78009420 con 0x7f8e840c6f60 2023-10-30T13:29:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e888ab640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e840c6b80 0x7f8e84130750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e83fff640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e840c6f60 0x7f8e840c8c10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8e7001c6f0 tx=0x7f8e7001b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e817fa640 1 -- 172.21.15.46:0/173909396 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e700b9600 con 0x7f8e840c6f60 2023-10-30T13:29:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e817fa640 1 -- 172.21.15.46:0/173909396 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e700b9780 con 0x7f8e840c6f60 2023-10-30T13:29:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e817fa640 1 -- 172.21.15.46:0/173909396 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e700b9b30 con 0x7f8e840c6f60 2023-10-30T13:29:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/173909396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7000b6c0 con 0x7f8e840c6f60 2023-10-30T13:29:32.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/173909396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e84133c10 con 0x7f8e840c6f60 2023-10-30T13:29:32.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.352+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/173909396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e840b0b90 con 0x7f8e840c6f60 2023-10-30T13:29:32.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.356+0000 7f8e817fa640 1 -- 172.21.15.46:0/173909396 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e700b9c90 con 0x7f8e840c6f60 2023-10-30T13:29:32.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.356+0000 7f8e817fa640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e680664d0 0x7f8e68068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:32.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.356+0000 7f8e817fa640 1 -- 172.21.15.46:0/173909396 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f8e70121810 con 0x7f8e840c6f60 2023-10-30T13:29:32.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.356+0000 7f8e888ab640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e680664d0 0x7f8e68068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:32.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.356+0000 7f8e888ab640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e680664d0 0x7f8e68068990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009d40 tx=0x7f8e78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:32.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.360+0000 7f8e817fa640 1 -- 172.21.15.46:0/173909396 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e700eeb10 con 0x7f8e840c6f60 2023-10-30T13:29:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/173909396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e8404e170 con 0x7f8e840c6f60 2023-10-30T13:29:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e817fa640 1 -- 172.21.15.46:0/173909396 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f8e700f2130 con 0x7f8e840c6f60 2023-10-30T13:29:32.555 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:32.555 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:31.028439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:clientreplay","state_seq":43,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:32.556 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 33 2023-10-30T13:29:32.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e680664d0 msgr2=0x7f8e68068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:32.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e680664d0 0x7f8e68068990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009d40 tx=0x7f8e78008040 comp rx=0 tx=0).stop 2023-10-30T13:29:32.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/173909396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e840c6f60 msgr2=0x7f8e840c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:32.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e840c6f60 0x7f8e840c8c10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8e7001c6f0 tx=0x7f8e7001b860 comp rx=0 tx=0).stop 2023-10-30T13:29:32.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/173909396 shutdown_connections 2023-10-30T13:29:32.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e680664d0 0x7f8e68068990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:32.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e840c8430 0x7f8e841f13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:32.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e840c6f60 0x7f8e840c8c10 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:32.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 --2- 172.21.15.46:0/173909396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e840c6b80 0x7f8e84130750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:32.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/173909396 >> 172.21.15.46:0/173909396 conn(0x7f8e840c7910 msgr2=0x7f8e8406a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:32.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/173909396 shutdown_connections 2023-10-30T13:29:32.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:32.548+0000 7f8e8ab0f640 1 -- 172.21.15.46:0/173909396 wait complete. 2023-10-30T13:29:32.576 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:29:34.576 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.724+0000 7f071b605640 1 Processor -- start 2023-10-30T13:29:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.724+0000 7f071b605640 1 -- start start 2023-10-30T13:29:34.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.724+0000 7f071b605640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0714153d70 0x7f071406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:34.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.724+0000 7f071b605640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f07140c8490 0x7f07140c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:34.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.724+0000 7f071b605640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f071406afa0 con 0x7f071406e470 2023-10-30T13:29:34.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.724+0000 7f071b605640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f071406d780 con 0x7f0714153d70 2023-10-30T13:29:34.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.724+0000 7f071b605640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0714154150 con 0x7f07140c8490 2023-10-30T13:29:34.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.724+0000 7f0718ba0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f071406e470 0x7f07140cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53228/0 (socket says 172.21.15.46:53228) 2023-10-30T13:29:34.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.724+0000 7f0718ba0640 1 -- 172.21.15.46:0/3637740974 learned_addr learned my addr 172.21.15.46:0/3637740974 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:34.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.724+0000 7f07193a1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0714153d70 0x7f071406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:34.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.724+0000 7f0719ba2640 1 --2- 172.21.15.46:0/3637740974 >> v2:172.21.15.154:3300/0 conn(0x7f07140c8490 0x7f07140c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:34.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.728+0000 7f07193a1640 1 -- 172.21.15.46:0/3637740974 >> v2:172.21.15.154:3300/0 conn(0x7f07140c8490 msgr2=0x7f07140c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:34.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.728+0000 7f07193a1640 1 --2- 172.21.15.46:0/3637740974 >> v2:172.21.15.154:3300/0 conn(0x7f07140c8490 0x7f07140c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:34.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.728+0000 7f07193a1640 1 -- 172.21.15.46:0/3637740974 >> v1:172.21.15.46:6789/0 conn(0x7f071406e470 legacy=0x7f07140cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:34.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.728+0000 7f07193a1640 1 -- 172.21.15.46:0/3637740974 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071406e030 con 0x7f0714153d70 2023-10-30T13:29:34.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.728+0000 7f0719ba2640 1 --2- 172.21.15.46:0/3637740974 >> v2:172.21.15.154:3300/0 conn(0x7f07140c8490 0x7f07140c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:34.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.728+0000 7f070bfff640 1 -- 172.21.15.46:0/3637740974 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130568497 0 0) 0x7f071406afa0 con 0x7f071406e470 2023-10-30T13:29:34.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.728+0000 7f07193a1640 1 --2- 172.21.15.46:0/3637740974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0714153d70 0x7f071406ca80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f07140cddf0 tx=0x7f071001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f527b26e2dd421d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:34.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.728+0000 7f070bfff640 1 -- 172.21.15.46:0/3637740974 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07100a7070 con 0x7f0714153d70 2023-10-30T13:29:34.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.728+0000 7f070bfff640 1 -- 172.21.15.46:0/3637740974 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07100ab430 con 0x7f0714153d70 2023-10-30T13:29:34.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.728+0000 7f070bfff640 1 -- 172.21.15.46:0/3637740974 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07100ab7c0 con 0x7f0714153d70 2023-10-30T13:29:34.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.728+0000 7f071b605640 1 -- 172.21.15.46:0/3637740974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0714153d70 msgr2=0x7f071406ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:34.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.728+0000 7f071b605640 1 --2- 172.21.15.46:0/3637740974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0714153d70 0x7f071406ca80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f07140cddf0 tx=0x7f071001c430 comp rx=0 tx=0).stop 2023-10-30T13:29:34.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 -- 172.21.15.46:0/3637740974 shutdown_connections 2023-10-30T13:29:34.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 --2- 172.21.15.46:0/3637740974 >> v2:172.21.15.154:3300/0 conn(0x7f07140c8490 0x7f07140c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:34.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 --2- 172.21.15.46:0/3637740974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0714153d70 0x7f071406ca80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:34.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 -- 172.21.15.46:0/3637740974 >> 172.21.15.46:0/3637740974 conn(0x7f0714071ea0 msgr2=0x7f0714065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:34.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 -- 172.21.15.46:0/3637740974 shutdown_connections 2023-10-30T13:29:34.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 -- 172.21.15.46:0/3637740974 wait complete. 2023-10-30T13:29:34.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 Processor -- start 2023-10-30T13:29:34.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 -- start start 2023-10-30T13:29:34.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f071406e470 0x7f07140cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:34.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07140c8490 0x7f07140cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:34.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0714153d70 0x7f0714150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:34.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0714145800 con 0x7f07140c8490 2023-10-30T13:29:34.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0714146310 con 0x7f0714153d70 2023-10-30T13:29:34.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f071b605640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07141451c0 con 0x7f071406e470 2023-10-30T13:29:34.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f0718ba0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07140c8490 0x7f07140cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:34.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f0718ba0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07140c8490 0x7f07140cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42116/0 (socket says 172.21.15.46:42116) 2023-10-30T13:29:34.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f0718ba0640 1 -- 172.21.15.46:0/3600259374 learned_addr learned my addr 172.21.15.46:0/3600259374 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:34.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f07193a1640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f071406e470 0x7f07140cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:34.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f0719ba2640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0714153d70 0x7f0714150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:34.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f0718ba0640 1 -- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f071406e470 msgr2=0x7f07140cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:34.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f0718ba0640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f071406e470 0x7f07140cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:34.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f0718ba0640 1 -- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0714153d70 msgr2=0x7f0714150380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:34.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f0718ba0640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0714153d70 0x7f0714150380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:34.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f0718ba0640 1 -- 172.21.15.46:0/3600259374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0710009420 con 0x7f07140c8490 2023-10-30T13:29:34.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f0719ba2640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0714153d70 0x7f0714150380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:34.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.732+0000 7f0718ba0640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07140c8490 0x7f07140cc710 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f070400c520 tx=0x7f070400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:34.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.736+0000 7f0709ffb640 1 -- 172.21.15.46:0/3600259374 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07040074e0 con 0x7f07140c8490 2023-10-30T13:29:34.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.736+0000 7f0709ffb640 1 -- 172.21.15.46:0/3600259374 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0704007640 con 0x7f07140c8490 2023-10-30T13:29:34.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.736+0000 7f0709ffb640 1 -- 172.21.15.46:0/3600259374 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07040049f0 con 0x7f07140c8490 2023-10-30T13:29:34.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.736+0000 7f071b605640 1 -- 172.21.15.46:0/3600259374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0714146540 con 0x7f07140c8490 2023-10-30T13:29:34.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.736+0000 7f071b605640 1 -- 172.21.15.46:0/3600259374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0714163e40 con 0x7f07140c8490 2023-10-30T13:29:34.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.736+0000 7f071b605640 1 -- 172.21.15.46:0/3600259374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0714164060 con 0x7f07140c8490 2023-10-30T13:29:34.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.736+0000 7f0709ffb640 1 -- 172.21.15.46:0/3600259374 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0704004b50 con 0x7f07140c8490 2023-10-30T13:29:34.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.736+0000 7f0709ffb640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06f8066520 0x7f06f80689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:34.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.736+0000 7f0709ffb640 1 -- 172.21.15.46:0/3600259374 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f0704083600 con 0x7f07140c8490 2023-10-30T13:29:34.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.736+0000 7f07193a1640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06f8066520 0x7f06f80689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:34.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.740+0000 7f07193a1640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06f8066520 0x7f06f80689e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f07100af000 tx=0x7f07100a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:34.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.740+0000 7f0709ffb640 1 -- 172.21.15.46:0/3600259374 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0704050910 con 0x7f07140c8490 2023-10-30T13:29:34.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.928+0000 7f071b605640 1 -- 172.21.15.46:0/3600259374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f071404e170 con 0x7f07140c8490 2023-10-30T13:29:34.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f0709ffb640 1 -- 172.21.15.46:0/3600259374 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f0704018020 con 0x7f07140c8490 2023-10-30T13:29:34.938 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:34.938 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:31.028439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:clientreplay","state_seq":43,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:34.939 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 33 2023-10-30T13:29:34.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f071b605640 1 -- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06f8066520 msgr2=0x7f06f80689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:34.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f071b605640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06f8066520 0x7f06f80689e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f07100af000 tx=0x7f07100a7000 comp rx=0 tx=0).stop 2023-10-30T13:29:34.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f071b605640 1 -- 172.21.15.46:0/3600259374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07140c8490 msgr2=0x7f07140cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:34.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f071b605640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07140c8490 0x7f07140cc710 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f070400c520 tx=0x7f070400c9e0 comp rx=0 tx=0).stop 2023-10-30T13:29:34.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f071b605640 1 -- 172.21.15.46:0/3600259374 shutdown_connections 2023-10-30T13:29:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f071b605640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06f8066520 0x7f06f80689e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f071b605640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0714153d70 0x7f0714150380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f071b605640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07140c8490 0x7f07140cc710 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f071b605640 1 --2- 172.21.15.46:0/3600259374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f071406e470 0x7f07140cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f071b605640 1 -- 172.21.15.46:0/3600259374 >> 172.21.15.46:0/3600259374 conn(0x7f0714071ea0 msgr2=0x7f07140c78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f071b605640 1 -- 172.21.15.46:0/3600259374 shutdown_connections 2023-10-30T13:29:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:34.932+0000 7f071b605640 1 -- 172.21.15.46:0/3600259374 wait complete. 2023-10-30T13:29:34.959 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 33, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 33, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4306": { "addr": "172.21.15.46:6836/275749119", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 275749119, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 275749119, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4950": { "addr": "172.21.15.46:6837/1315246492", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 1315246492, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 1315246492, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4950, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_5106": { "addr": "172.21.15.46:6833/603884510", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 603884510, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 603884510, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5106, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:clientreplay", "state_seq": 43 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:29:31.028439+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 5106, "mds_2": 4950 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:29:36.960 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:37.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.112+0000 7fa3457c4640 1 Processor -- start 2023-10-30T13:29:37.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.112+0000 7fa3457c4640 1 -- start start 2023-10-30T13:29:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.112+0000 7fa3457c4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3401538a0 0x7fa34015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.112+0000 7fa3457c4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3400cb800 con 0x7fa3400cf160 2023-10-30T13:29:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.112+0000 7fa3457c4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34015dc30 con 0x7fa3400ca0e0 2023-10-30T13:29:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.112+0000 7fa3457c4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3400cb330 con 0x7fa3401538a0 2023-10-30T13:29:37.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.112+0000 7fa33effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3401538a0 0x7fa34015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:37.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.112+0000 7fa33effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3401538a0 0x7fa34015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56374/0 (socket says 172.21.15.46:56374) 2023-10-30T13:29:37.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.112+0000 7fa33effd640 1 -- 172.21.15.46:0/3902365717 learned_addr learned my addr 172.21.15.46:0/3902365717 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:37.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.112+0000 7fa33effd640 1 -- 172.21.15.46:0/3902365717 >> v1:172.21.15.154:6789/0 conn(0x7fa3400ca0e0 legacy=0x7fa34015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:37.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.112+0000 7fa33effd640 1 -- 172.21.15.46:0/3902365717 >> v1:172.21.15.46:6789/0 conn(0x7fa3400cf160 legacy=0x7fa340164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:37.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.112+0000 7fa33effd640 1 -- 172.21.15.46:0/3902365717 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3400cad70 con 0x7fa3401538a0 2023-10-30T13:29:37.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.116+0000 7fa33effd640 1 --2- 172.21.15.46:0/3902365717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3401538a0 0x7fa34015d6f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa330009870 tx=0x7fa33001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476ef3bceccd7b81 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:37.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.116+0000 7fa33e7fc640 1 -- 172.21.15.46:0/3902365717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3300a8a30 con 0x7fa3401538a0 2023-10-30T13:29:37.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.116+0000 7fa33e7fc640 1 -- 172.21.15.46:0/3902365717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3300a8b90 con 0x7fa3401538a0 2023-10-30T13:29:37.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.116+0000 7fa33e7fc640 1 -- 172.21.15.46:0/3902365717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3300b0440 con 0x7fa3401538a0 2023-10-30T13:29:37.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.116+0000 7fa3457c4640 1 -- 172.21.15.46:0/3902365717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3401538a0 msgr2=0x7fa34015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:37.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.116+0000 7fa3457c4640 1 --2- 172.21.15.46:0/3902365717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3401538a0 0x7fa34015d6f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa330009870 tx=0x7fa33001ed30 comp rx=0 tx=0).stop 2023-10-30T13:29:37.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.116+0000 7fa3457c4640 1 -- 172.21.15.46:0/3902365717 shutdown_connections 2023-10-30T13:29:37.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.116+0000 7fa3457c4640 1 --2- 172.21.15.46:0/3902365717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3401538a0 0x7fa34015d6f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:37.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.116+0000 7fa3457c4640 1 -- 172.21.15.46:0/3902365717 >> 172.21.15.46:0/3902365717 conn(0x7fa34006bfe0 msgr2=0x7fa34006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:37.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.116+0000 7fa3457c4640 1 -- 172.21.15.46:0/3902365717 shutdown_connections 2023-10-30T13:29:37.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.116+0000 7fa3457c4640 1 -- 172.21.15.46:0/3902365717 wait complete. 2023-10-30T13:29:37.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3457c4640 1 Processor -- start 2023-10-30T13:29:37.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3457c4640 1 -- start start 2023-10-30T13:29:37.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3457c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3400ca0e0 0x7fa340143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:37.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3457c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3400cf160 0x7fa3400ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:37.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3457c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3401538a0 0x7fa340144d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:37.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3457c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa340157600 con 0x7fa3400cf160 2023-10-30T13:29:37.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3457c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa340157850 con 0x7fa3400ca0e0 2023-10-30T13:29:37.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3457c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa340157b30 con 0x7fa3401538a0 2023-10-30T13:29:37.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa337fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3400cf160 0x7fa3400ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:37.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa337fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3400cf160 0x7fa3400ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56388/0 (socket says 172.21.15.46:56388) 2023-10-30T13:29:37.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa337fff640 1 -- 172.21.15.46:0/1824014615 learned_addr learned my addr 172.21.15.46:0/1824014615 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:37.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa33effd640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3400ca0e0 0x7fa340143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa33f7fe640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3401538a0 0x7fa340144d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa337fff640 1 -- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3401538a0 msgr2=0x7fa340144d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa337fff640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3401538a0 0x7fa340144d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa337fff640 1 -- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3400ca0e0 msgr2=0x7fa340143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:37.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa337fff640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3400ca0e0 0x7fa340143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:37.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa337fff640 1 -- 172.21.15.46:0/1824014615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3300094d0 con 0x7fa3400cf160 2023-10-30T13:29:37.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa33f7fe640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3401538a0 0x7fa340144d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:37.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa337fff640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3400cf160 0x7fa3400ce260 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa32800c3c0 tx=0x7fa32800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3377fe640 1 -- 172.21.15.46:0/1824014615 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3280052d0 con 0x7fa3400cf160 2023-10-30T13:29:37.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3377fe640 1 -- 172.21.15.46:0/1824014615 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa328005430 con 0x7fa3400cf160 2023-10-30T13:29:37.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3457c4640 1 -- 172.21.15.46:0/1824014615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa340157e10 con 0x7fa3400cf160 2023-10-30T13:29:37.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3377fe640 1 -- 172.21.15.46:0/1824014615 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328003840 con 0x7fa3400cf160 2023-10-30T13:29:37.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3457c4640 1 -- 172.21.15.46:0/1824014615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3401583d0 con 0x7fa3400cf160 2023-10-30T13:29:37.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.120+0000 7fa3457c4640 1 -- 172.21.15.46:0/1824014615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34004e170 con 0x7fa3400cf160 2023-10-30T13:29:37.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.124+0000 7fa3377fe640 1 -- 172.21.15.46:0/1824014615 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3280039a0 con 0x7fa3400cf160 2023-10-30T13:29:37.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.124+0000 7fa3377fe640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3200664a0 0x7fa320068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:37.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.124+0000 7fa3377fe640 1 -- 172.21.15.46:0/1824014615 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fa328083420 con 0x7fa3400cf160 2023-10-30T13:29:37.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.124+0000 7fa33effd640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3200664a0 0x7fa320068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:37.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.124+0000 7fa33effd640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3200664a0 0x7fa320068960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa3300b2000 tx=0x7fa330009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:37.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.128+0000 7fa3377fe640 1 -- 172.21.15.46:0/1824014615 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3280506b0 con 0x7fa3400cf160 2023-10-30T13:29:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.316+0000 7fa3457c4640 1 -- 172.21.15.46:0/1824014615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa34015dc30 con 0x7fa3400cf160 2023-10-30T13:29:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.316+0000 7fa3377fe640 1 -- 172.21.15.46:0/1824014615 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fa34015dc30 con 0x7fa3400cf160 2023-10-30T13:29:37.324 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:37.324 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:31.028439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:clientreplay","state_seq":43,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 33 2023-10-30T13:29:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.320+0000 7fa3457c4640 1 -- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3200664a0 msgr2=0x7fa320068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.320+0000 7fa3457c4640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3200664a0 0x7fa320068960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa3300b2000 tx=0x7fa330009040 comp rx=0 tx=0).stop 2023-10-30T13:29:37.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.320+0000 7fa3457c4640 1 -- 172.21.15.46:0/1824014615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3400cf160 msgr2=0x7fa3400ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:37.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.320+0000 7fa3457c4640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3400cf160 0x7fa3400ce260 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa32800c3c0 tx=0x7fa32800c880 comp rx=0 tx=0).stop 2023-10-30T13:29:37.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.320+0000 7fa3457c4640 1 -- 172.21.15.46:0/1824014615 shutdown_connections 2023-10-30T13:29:37.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.320+0000 7fa3457c4640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3200664a0 0x7fa320068960 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:37.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.320+0000 7fa3457c4640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3401538a0 0x7fa340144d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:37.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.320+0000 7fa3457c4640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3400cf160 0x7fa3400ce260 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:37.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.320+0000 7fa3457c4640 1 --2- 172.21.15.46:0/1824014615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3400ca0e0 0x7fa340143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:37.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.320+0000 7fa3457c4640 1 -- 172.21.15.46:0/1824014615 >> 172.21.15.46:0/1824014615 conn(0x7fa34006bfe0 msgr2=0x7fa3401511d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:37.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.320+0000 7fa3457c4640 1 -- 172.21.15.46:0/1824014615 shutdown_connections 2023-10-30T13:29:37.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:37.320+0000 7fa3457c4640 1 -- 172.21.15.46:0/1824014615 wait complete. 2023-10-30T13:29:37.346 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:29:38.939 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T13:29:38.932+0000 7fce56864640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:29:39.347 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:39.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.496+0000 7fbf4849e640 1 Processor -- start 2023-10-30T13:29:39.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.496+0000 7fbf4849e640 1 -- start start 2023-10-30T13:29:39.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.496+0000 7fbf4849e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf400c6fe0 0x7fbf40071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:39.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.496+0000 7fbf4849e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbf400c5f60 0x7fbf400cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:39.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.496+0000 7fbf4849e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf400700b0 con 0x7fbf40153bf0 2023-10-30T13:29:39.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.496+0000 7fbf4849e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf400c6810 con 0x7fbf400c5f60 2023-10-30T13:29:39.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.496+0000 7fbf4849e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf40159900 con 0x7fbf400c6fe0 2023-10-30T13:29:39.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.496+0000 7fbf4623a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbf40153bf0 0x7fbf400c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58812/0 (socket says 172.21.15.46:58812) 2023-10-30T13:29:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.496+0000 7fbf4623a640 1 -- 172.21.15.46:0/704813244 learned_addr learned my addr 172.21.15.46:0/704813244 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.496+0000 7fbf45a39640 1 --2- 172.21.15.46:0/704813244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf400c6fe0 0x7fbf40071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.496+0000 7fbf46a3b640 1 --2- 172.21.15.46:0/704813244 >> v2:172.21.15.154:3300/0 conn(0x7fbf400c5f60 0x7fbf400cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.500+0000 7fbf45238640 1 -- 172.21.15.46:0/704813244 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366345604 0 0) 0x7fbf400700b0 con 0x7fbf40153bf0 2023-10-30T13:29:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.500+0000 7fbf45238640 1 -- 172.21.15.46:0/704813244 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf2c003530 con 0x7fbf40153bf0 2023-10-30T13:29:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.500+0000 7fbf45a39640 1 -- 172.21.15.46:0/704813244 >> v2:172.21.15.154:3300/0 conn(0x7fbf400c5f60 msgr2=0x7fbf400cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.500+0000 7fbf45a39640 1 --2- 172.21.15.46:0/704813244 >> v2:172.21.15.154:3300/0 conn(0x7fbf400c5f60 0x7fbf400cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.500+0000 7fbf45a39640 1 -- 172.21.15.46:0/704813244 >> v1:172.21.15.46:6789/0 conn(0x7fbf40153bf0 legacy=0x7fbf400c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.500+0000 7fbf45a39640 1 -- 172.21.15.46:0/704813244 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf400c6530 con 0x7fbf400c6fe0 2023-10-30T13:29:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.500+0000 7fbf46a3b640 1 --2- 172.21.15.46:0/704813244 >> v2:172.21.15.154:3300/0 conn(0x7fbf400c5f60 0x7fbf400cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:29:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.500+0000 7fbf45a39640 1 --2- 172.21.15.46:0/704813244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf400c6fe0 0x7fbf40071260 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbf300097c0 tx=0x7fbf3001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=689deb1d8cf7c295 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.500+0000 7fbf45238640 1 -- 172.21.15.46:0/704813244 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf300a8970 con 0x7fbf400c6fe0 2023-10-30T13:29:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.500+0000 7fbf45238640 1 -- 172.21.15.46:0/704813244 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf300a8ad0 con 0x7fbf400c6fe0 2023-10-30T13:29:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.500+0000 7fbf45238640 1 -- 172.21.15.46:0/704813244 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf300a8e40 con 0x7fbf400c6fe0 2023-10-30T13:29:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 -- 172.21.15.46:0/704813244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf400c6fe0 msgr2=0x7fbf40071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:39.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 --2- 172.21.15.46:0/704813244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf400c6fe0 0x7fbf40071260 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbf300097c0 tx=0x7fbf3001ec70 comp rx=0 tx=0).stop 2023-10-30T13:29:39.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 -- 172.21.15.46:0/704813244 shutdown_connections 2023-10-30T13:29:39.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 --2- 172.21.15.46:0/704813244 >> v2:172.21.15.154:3300/0 conn(0x7fbf400c5f60 0x7fbf400cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:39.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 --2- 172.21.15.46:0/704813244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf400c6fe0 0x7fbf40071260 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:39.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 -- 172.21.15.46:0/704813244 >> 172.21.15.46:0/704813244 conn(0x7fbf4006c6e0 msgr2=0x7fbf40069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:39.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 -- 172.21.15.46:0/704813244 shutdown_connections 2023-10-30T13:29:39.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 -- 172.21.15.46:0/704813244 wait complete. 2023-10-30T13:29:39.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 Processor -- start 2023-10-30T13:29:39.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 -- start start 2023-10-30T13:29:39.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf400c5f60 0x7fbf4014e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:39.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf400c6fe0 0x7fbf400c5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf40153bf0 0x7fbf4014f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4014dfa0 con 0x7fbf400c5f60 2023-10-30T13:29:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4014f230 con 0x7fbf400c6fe0 2023-10-30T13:29:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4849e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf400c40f0 con 0x7fbf40153bf0 2023-10-30T13:29:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4623a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf400c5f60 0x7fbf4014e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4623a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf400c5f60 0x7fbf4014e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56396/0 (socket says 172.21.15.46:56396) 2023-10-30T13:29:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf4623a640 1 -- 172.21.15.46:0/2213431400 learned_addr learned my addr 172.21.15.46:0/2213431400 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf45a39640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf400c6fe0 0x7fbf400c5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.504+0000 7fbf46a3b640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf40153bf0 0x7fbf4014f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf4623a640 1 -- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf40153bf0 msgr2=0x7fbf4014f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf4623a640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf40153bf0 0x7fbf4014f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf4623a640 1 -- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf400c6fe0 msgr2=0x7fbf400c5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf4623a640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf400c6fe0 0x7fbf400c5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf4623a640 1 -- 172.21.15.46:0/2213431400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf38009a70 con 0x7fbf400c5f60 2023-10-30T13:29:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf4623a640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf400c5f60 0x7fbf4014e8f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00c3c0 tx=0x7fbf3c00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf36ffd640 1 -- 172.21.15.46:0/2213431400 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c005db0 con 0x7fbf400c5f60 2023-10-30T13:29:39.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf36ffd640 1 -- 172.21.15.46:0/2213431400 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c0070d0 con 0x7fbf400c5f60 2023-10-30T13:29:39.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf36ffd640 1 -- 172.21.15.46:0/2213431400 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c0074a0 con 0x7fbf400c5f60 2023-10-30T13:29:39.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf4849e640 1 -- 172.21.15.46:0/2213431400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf30009420 con 0x7fbf400c5f60 2023-10-30T13:29:39.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf4849e640 1 -- 172.21.15.46:0/2213431400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf400c4690 con 0x7fbf400c5f60 2023-10-30T13:29:39.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf4849e640 1 -- 172.21.15.46:0/2213431400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf40158700 con 0x7fbf400c5f60 2023-10-30T13:29:39.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf36ffd640 1 -- 172.21.15.46:0/2213431400 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbf3c007600 con 0x7fbf400c5f60 2023-10-30T13:29:39.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf36ffd640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf240664f0 0x7fbf240689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:39.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.508+0000 7fbf36ffd640 1 -- 172.21.15.46:0/2213431400 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fbf3c083470 con 0x7fbf400c5f60 2023-10-30T13:29:39.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.512+0000 7fbf45a39640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf240664f0 0x7fbf240689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:39.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.512+0000 7fbf45a39640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf240664f0 0x7fbf240689b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbf300b2000 tx=0x7fbf300aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:39.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.516+0000 7fbf36ffd640 1 -- 172.21.15.46:0/2213431400 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbf3c0509a0 con 0x7fbf400c5f60 2023-10-30T13:29:39.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 -- 172.21.15.46:0/2213431400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf40159dc0 con 0x7fbf400c5f60 2023-10-30T13:29:39.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf36ffd640 1 -- 172.21.15.46:0/2213431400 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5736 (secure 0 0 0) 0x7fbf3c053fc0 con 0x7fbf400c5f60 2023-10-30T13:29:39.711 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:39.711 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:36.923344+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:39.712 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 34 2023-10-30T13:29:39.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 -- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf240664f0 msgr2=0x7fbf240689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:39.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf240664f0 0x7fbf240689b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbf300b2000 tx=0x7fbf300aa070 comp rx=0 tx=0).stop 2023-10-30T13:29:39.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 -- 172.21.15.46:0/2213431400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf400c5f60 msgr2=0x7fbf4014e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:39.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf400c5f60 0x7fbf4014e8f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00c3c0 tx=0x7fbf3c00c7c0 comp rx=0 tx=0).stop 2023-10-30T13:29:39.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 -- 172.21.15.46:0/2213431400 shutdown_connections 2023-10-30T13:29:39.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf240664f0 0x7fbf240689b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:39.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf40153bf0 0x7fbf4014f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:39.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf400c6fe0 0x7fbf400c5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:39.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 --2- 172.21.15.46:0/2213431400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf400c5f60 0x7fbf4014e8f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:39.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 -- 172.21.15.46:0/2213431400 >> 172.21.15.46:0/2213431400 conn(0x7fbf4006c6e0 msgr2=0x7fbf40157ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:39.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 -- 172.21.15.46:0/2213431400 shutdown_connections 2023-10-30T13:29:39.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:39.704+0000 7fbf4849e640 1 -- 172.21.15.46:0/2213431400 wait complete. 2023-10-30T13:29:39.732 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=4288 2023-10-30T13:29:39.732 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.b 2023-10-30T13:29:43.406 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/31' 2023-10-30T13:29:43.407 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/36' 2023-10-30T13:29:43.407 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/20' 2023-10-30T13:29:43.407 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/39' 2023-10-30T13:29:43.407 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/34' 2023-10-30T13:29:43.407 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/38' 2023-10-30T13:29:43.408 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/28' 2023-10-30T13:29:43.408 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/24' 2023-10-30T13:29:43.408 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/29' 2023-10-30T13:29:43.408 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/32' 2023-10-30T13:29:43.408 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/22' 2023-10-30T13:29:43.409 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/25' 2023-10-30T13:29:43.409 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/30' 2023-10-30T13:29:43.409 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/33' 2023-10-30T13:29:43.409 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/23' 2023-10-30T13:29:43.409 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/35' 2023-10-30T13:29:43.410 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/26' 2023-10-30T13:29:43.410 INFO:tasks.workunit.client.0.smithi046.stdout:removed './297/37' 2023-10-30T13:29:43.410 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './297' 2023-10-30T13:29:43.410 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/21' 2023-10-30T13:29:43.410 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/27' 2023-10-30T13:29:43.411 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/31' 2023-10-30T13:29:43.411 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/36' 2023-10-30T13:29:43.411 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/20' 2023-10-30T13:29:43.411 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/39' 2023-10-30T13:29:43.411 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/34' 2023-10-30T13:29:43.411 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/38' 2023-10-30T13:29:43.412 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/28' 2023-10-30T13:29:43.412 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/24' 2023-10-30T13:29:43.412 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/29' 2023-10-30T13:29:43.412 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/32' 2023-10-30T13:29:43.412 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/22' 2023-10-30T13:29:43.413 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/25' 2023-10-30T13:29:43.413 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/30' 2023-10-30T13:29:43.413 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/33' 2023-10-30T13:29:43.413 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/23' 2023-10-30T13:29:43.413 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/35' 2023-10-30T13:29:43.414 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/26' 2023-10-30T13:29:43.414 INFO:tasks.workunit.client.0.smithi046.stdout:removed './298/37' 2023-10-30T13:29:43.414 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './298' 2023-10-30T13:29:43.414 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/21' 2023-10-30T13:29:43.414 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/27' 2023-10-30T13:29:43.415 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/31' 2023-10-30T13:29:43.415 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/36' 2023-10-30T13:29:43.415 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/20' 2023-10-30T13:29:43.415 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/39' 2023-10-30T13:29:43.415 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/34' 2023-10-30T13:29:43.416 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/38' 2023-10-30T13:29:43.416 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/28' 2023-10-30T13:29:43.416 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/24' 2023-10-30T13:29:43.416 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/29' 2023-10-30T13:29:43.416 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/32' 2023-10-30T13:29:43.416 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/22' 2023-10-30T13:29:43.417 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/25' 2023-10-30T13:29:43.417 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/30' 2023-10-30T13:29:43.417 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/33' 2023-10-30T13:29:43.417 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/23' 2023-10-30T13:29:43.417 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/35' 2023-10-30T13:29:43.418 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/26' 2023-10-30T13:29:43.418 INFO:tasks.workunit.client.0.smithi046.stdout:removed './299/37' 2023-10-30T13:29:43.418 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './299' 2023-10-30T13:29:43.418 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/166/19' 2023-10-30T13:29:43.418 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/166/15' 2023-10-30T13:29:43.419 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/166/20' 2023-10-30T13:29:43.419 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/166/12' 2023-10-30T13:29:43.419 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/166/10' 2023-10-30T13:29:43.419 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/166/11' 2023-10-30T13:29:43.419 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/166/16' 2023-10-30T13:29:43.420 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/166/14' 2023-10-30T13:29:43.420 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/166/18' 2023-10-30T13:29:43.420 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/166/13' 2023-10-30T13:29:43.420 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/166/17' 2023-10-30T13:29:43.420 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/166' 2023-10-30T13:29:43.421 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/147/19' 2023-10-30T13:29:43.421 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/147/15' 2023-10-30T13:29:43.421 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/147/20' 2023-10-30T13:29:43.421 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/147/12' 2023-10-30T13:29:43.421 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/147/10' 2023-10-30T13:29:43.421 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/147/11' 2023-10-30T13:29:43.422 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/147/16' 2023-10-30T13:29:43.422 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/147/14' 2023-10-30T13:29:43.422 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/147/18' 2023-10-30T13:29:43.422 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/147/13' 2023-10-30T13:29:43.422 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/147/17' 2023-10-30T13:29:43.423 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/147' 2023-10-30T13:29:43.423 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/151/19' 2023-10-30T13:29:43.423 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/151/15' 2023-10-30T13:29:43.423 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/151/20' 2023-10-30T13:29:43.423 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/151/12' 2023-10-30T13:29:43.424 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/151/10' 2023-10-30T13:29:43.424 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/151/11' 2023-10-30T13:29:43.424 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/151/16' 2023-10-30T13:29:43.424 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/151/14' 2023-10-30T13:29:43.424 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/151/18' 2023-10-30T13:29:43.425 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/151/13' 2023-10-30T13:29:43.425 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/151/17' 2023-10-30T13:29:43.425 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/151' 2023-10-30T13:29:43.425 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/170/19' 2023-10-30T13:29:43.425 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/170/15' 2023-10-30T13:29:43.426 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/170/20' 2023-10-30T13:29:43.426 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/170/12' 2023-10-30T13:29:43.426 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/170/10' 2023-10-30T13:29:43.426 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/170/11' 2023-10-30T13:29:43.426 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/170/16' 2023-10-30T13:29:43.427 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/170/14' 2023-10-30T13:29:43.427 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/170/18' 2023-10-30T13:29:43.427 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/170/13' 2023-10-30T13:29:43.427 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/170/17' 2023-10-30T13:29:43.427 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/170' 2023-10-30T13:29:43.427 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/167/19' 2023-10-30T13:29:43.428 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/167/15' 2023-10-30T13:29:43.428 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/167/20' 2023-10-30T13:29:43.428 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/167/12' 2023-10-30T13:29:43.428 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/167/10' 2023-10-30T13:29:43.428 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/167/11' 2023-10-30T13:29:43.429 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/167/16' 2023-10-30T13:29:43.429 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/167/14' 2023-10-30T13:29:43.429 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/167/18' 2023-10-30T13:29:43.429 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/167/13' 2023-10-30T13:29:43.429 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/167/17' 2023-10-30T13:29:43.429 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/167' 2023-10-30T13:29:43.430 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/140/19' 2023-10-30T13:29:43.430 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/140/15' 2023-10-30T13:29:43.430 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/140/20' 2023-10-30T13:29:43.430 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/140/12' 2023-10-30T13:29:43.430 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/140/10' 2023-10-30T13:29:43.431 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/140/11' 2023-10-30T13:29:43.431 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/140/16' 2023-10-30T13:29:43.431 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/140/14' 2023-10-30T13:29:43.431 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/140/18' 2023-10-30T13:29:43.431 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/140/13' 2023-10-30T13:29:43.432 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/140/17' 2023-10-30T13:29:43.432 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/140' 2023-10-30T13:29:43.432 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/142/19' 2023-10-30T13:29:43.432 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/142/15' 2023-10-30T13:29:43.432 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/142/20' 2023-10-30T13:29:43.432 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/142/12' 2023-10-30T13:29:43.433 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/142/10' 2023-10-30T13:29:43.733 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-30T13:29:43.734 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T13:29:43.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T13:29:43.737 INFO:tasks.ceph.mds.b:Started 2023-10-30T13:29:43.737 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:43.813 INFO:tasks.ceph.mds.b.smithi154.stdout:starting mds.b at 2023-10-30T13:29:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8aa2ad6640 1 Processor -- start 2023-10-30T13:29:43.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8aa2ad6640 1 -- start start 2023-10-30T13:29:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8aa2ad6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a9c0cf160 0x7f8a9c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:43.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8aa2ad6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c0cb800 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8aa2ad6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c0cb330 con 0x7f8a9c1538a0 2023-10-30T13:29:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8aa2ad6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0cad70 con 0x7f8a9c0cf160 2023-10-30T13:29:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8a9bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8a9c15d6f0 0x7f8a9c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58826/0 (socket says 172.21.15.46:58826) 2023-10-30T13:29:43.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8a9bfff640 1 -- 172.21.15.46:0/3906737585 learned_addr learned my addr 172.21.15.46:0/3906737585 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:43.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8aa1073640 1 --2- 172.21.15.46:0/3906737585 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a9c0cf160 0x7f8a9c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:43.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8a9b7fe640 1 -- 172.21.15.46:0/3906737585 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425745790 0 0) 0x7f8a9c0cb800 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8a9b7fe640 1 -- 172.21.15.46:0/3906737585 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a8c003530 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8a9b7fe640 1 -- 172.21.15.46:0/3906737585 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489639878 0 0) 0x7f8a9c0cb330 con 0x7f8a9c1538a0 2023-10-30T13:29:43.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8a9b7fe640 1 -- 172.21.15.46:0/3906737585 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a9c0cb800 con 0x7f8a9c1538a0 2023-10-30T13:29:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8aa1073640 1 -- 172.21.15.46:0/3906737585 >> v1:172.21.15.154:6789/0 conn(0x7f8a9c1538a0 legacy=0x7f8a9c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8aa1073640 1 -- 172.21.15.46:0/3906737585 >> v1:172.21.15.46:6789/0 conn(0x7f8a9c15d6f0 legacy=0x7f8a9c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8a9b7fe640 1 -- 172.21.15.46:0/3906737585 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1486891133 0 0) 0x7f8a8c003530 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.888+0000 7f8aa1073640 1 -- 172.21.15.46:0/3906737585 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c0cb050 con 0x7f8a9c0cf160 2023-10-30T13:29:43.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.892+0000 7f8aa1073640 1 --2- 172.21.15.46:0/3906737585 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a9c0cf160 0x7f8a9c0cf590 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009dc0 tx=0x7f8a9401f080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94631e039d04db43 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:43.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.892+0000 7f8a9b7fe640 1 -- 172.21.15.46:0/3906737585 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88001680 con 0x7f8a9c0cf160 2023-10-30T13:29:43.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.896+0000 7f8a9b7fe640 1 -- 172.21.15.46:0/3906737585 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a940a8c10 con 0x7f8a9c0cf160 2023-10-30T13:29:43.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.896+0000 7f8a9b7fe640 1 -- 172.21.15.46:0/3906737585 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a940b0440 con 0x7f8a9c0cf160 2023-10-30T13:29:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.896+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/3906737585 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a9c0cf160 msgr2=0x7f8a9c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.896+0000 7f8aa2ad6640 1 --2- 172.21.15.46:0/3906737585 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a9c0cf160 0x7f8a9c0cf590 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009dc0 tx=0x7f8a9401f080 comp rx=0 tx=0).stop 2023-10-30T13:29:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.896+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/3906737585 shutdown_connections 2023-10-30T13:29:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.896+0000 7f8aa2ad6640 1 --2- 172.21.15.46:0/3906737585 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a9c0cf160 0x7f8a9c0cf590 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.896+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/3906737585 >> 172.21.15.46:0/3906737585 conn(0x7f8a9c06bfe0 msgr2=0x7f8a9c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:43.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.896+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/3906737585 shutdown_connections 2023-10-30T13:29:43.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.896+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/3906737585 wait complete. 2023-10-30T13:29:43.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.896+0000 7f8aa2ad6640 1 Processor -- start 2023-10-30T13:29:43.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa2ad6640 1 -- start start 2023-10-30T13:29:43.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa2ad6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a9c0cf160 0x7f8a9c0cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa2ad6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a9c1538a0 0x7f8a9c0ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa2ad6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a9c15d6f0 0x7f8a9c14cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa2ad6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0cd8d0 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa2ad6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c159270 con 0x7f8a9c1538a0 2023-10-30T13:29:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa2ad6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c158530 con 0x7f8a9c0cf160 2023-10-30T13:29:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa1073640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a9c15d6f0 0x7f8a9c14cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa1073640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a9c15d6f0 0x7f8a9c14cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56408/0 (socket says 172.21.15.46:56408) 2023-10-30T13:29:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa1073640 1 -- 172.21.15.46:0/2815976368 learned_addr learned my addr 172.21.15.46:0/2815976368 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:43.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa0872640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a9c0cf160 0x7f8a9c0cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:43.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8a9bfff640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a9c1538a0 0x7f8a9c0ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:43.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa1073640 1 -- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a9c0cf160 msgr2=0x7f8a9c0cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:43.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa1073640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a9c0cf160 0x7f8a9c0cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa1073640 1 -- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a9c1538a0 msgr2=0x7f8a9c0ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:43.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa1073640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a9c1538a0 0x7f8a9c0ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa1073640 1 -- 172.21.15.46:0/2815976368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a94009a70 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa1073640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a9c15d6f0 0x7f8a9c14cc70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8a9401ccb0 tx=0x7f8a9401be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:43.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8a997fa640 1 -- 172.21.15.46:0/2815976368 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a940c2070 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8a997fa640 1 -- 172.21.15.46:0/2815976368 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a94009450 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/2815976368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c0cd050 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8a997fa640 1 -- 172.21.15.46:0/2815976368 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a940ba410 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.900+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/2815976368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c158250 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.904+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/2815976368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9c15db00 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.904+0000 7f8a997fa640 1 -- 172.21.15.46:0/2815976368 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8a940b7050 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.904+0000 7f8a997fa640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8a800664d0 0x7f8a80068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:43.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.904+0000 7f8a997fa640 1 -- 172.21.15.46:0/2815976368 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f8a94121990 con 0x7f8a9c15d6f0 2023-10-30T13:29:43.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.904+0000 7f8aa0872640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8a800664d0 0x7f8a80068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:43.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.904+0000 7f8aa0872640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8a800664d0 0x7f8a80068990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8a900011f0 tx=0x7f8a90009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:43.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:43.908+0000 7f8a997fa640 1 -- 172.21.15.46:0/2815976368 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8a9c15db00 con 0x7f8a9c15d6f0 2023-10-30T13:29:44.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/2815976368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a9c0cda10 con 0x7f8a9c15d6f0 2023-10-30T13:29:44.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8a997fa640 1 -- 172.21.15.46:0/2815976368 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f8a9c0cda10 con 0x7f8a9c15d6f0 2023-10-30T13:29:44.135 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:44.135 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:36.923344+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:44.135 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 34 2023-10-30T13:29:44.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8a800664d0 msgr2=0x7f8a80068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:44.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8a800664d0 0x7f8a80068990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8a900011f0 tx=0x7f8a90009040 comp rx=0 tx=0).stop 2023-10-30T13:29:44.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/2815976368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a9c15d6f0 msgr2=0x7f8a9c14cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:44.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a9c15d6f0 0x7f8a9c14cc70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8a9401ccb0 tx=0x7f8a9401be20 comp rx=0 tx=0).stop 2023-10-30T13:29:44.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/2815976368 shutdown_connections 2023-10-30T13:29:44.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8a800664d0 0x7f8a80068990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:44.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a9c15d6f0 0x7f8a9c14cc70 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:44.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a9c1538a0 0x7f8a9c0ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:44.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 --2- 172.21.15.46:0/2815976368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a9c0cf160 0x7f8a9c0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:44.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/2815976368 >> 172.21.15.46:0/2815976368 conn(0x7f8a9c06bfe0 msgr2=0x7f8a9c15aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:44.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/2815976368 shutdown_connections 2023-10-30T13:29:44.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:44.128+0000 7f8aa2ad6640 1 -- 172.21.15.46:0/2815976368 wait complete. 2023-10-30T13:29:44.155 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-30T13:29:44.867 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/142/11' 2023-10-30T13:29:44.867 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/142/16' 2023-10-30T13:29:44.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/142/14' 2023-10-30T13:29:44.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/142/18' 2023-10-30T13:29:44.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/142/13' 2023-10-30T13:29:44.868 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/142/17' 2023-10-30T13:29:44.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/142' 2023-10-30T13:29:44.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/150/19' 2023-10-30T13:29:44.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/150/15' 2023-10-30T13:29:44.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/150/20' 2023-10-30T13:29:44.869 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/150/12' 2023-10-30T13:29:44.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/150/10' 2023-10-30T13:29:44.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/150/11' 2023-10-30T13:29:44.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/150/16' 2023-10-30T13:29:44.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/150/14' 2023-10-30T13:29:44.870 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/150/18' 2023-10-30T13:29:44.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/150/13' 2023-10-30T13:29:44.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/150/17' 2023-10-30T13:29:44.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/150' 2023-10-30T13:29:44.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/105/19' 2023-10-30T13:29:44.871 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/105/15' 2023-10-30T13:29:44.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/105/20' 2023-10-30T13:29:44.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/105/12' 2023-10-30T13:29:44.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/105/10' 2023-10-30T13:29:44.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/105/11' 2023-10-30T13:29:44.872 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/105/16' 2023-10-30T13:29:44.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/105/14' 2023-10-30T13:29:44.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/105/18' 2023-10-30T13:29:44.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/105/13' 2023-10-30T13:29:44.873 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/105/17' 2023-10-30T13:29:44.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/105' 2023-10-30T13:29:44.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/149/19' 2023-10-30T13:29:44.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/149/15' 2023-10-30T13:29:44.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/149/20' 2023-10-30T13:29:44.874 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/149/12' 2023-10-30T13:29:44.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/149/10' 2023-10-30T13:29:44.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/149/11' 2023-10-30T13:29:44.875 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/149/16' 2023-10-30T13:29:44.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/149/14' 2023-10-30T13:29:44.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/149/18' 2023-10-30T13:29:44.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/149/13' 2023-10-30T13:29:44.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/149/17' 2023-10-30T13:29:44.876 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/149' 2023-10-30T13:29:44.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/109/19' 2023-10-30T13:29:44.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/109/15' 2023-10-30T13:29:44.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/109/20' 2023-10-30T13:29:44.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/109/12' 2023-10-30T13:29:44.877 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/109/10' 2023-10-30T13:29:44.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/109/11' 2023-10-30T13:29:44.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/109/16' 2023-10-30T13:29:44.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/109/14' 2023-10-30T13:29:44.878 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/109/18' 2023-10-30T13:29:44.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/109/13' 2023-10-30T13:29:44.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/109/17' 2023-10-30T13:29:44.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/109' 2023-10-30T13:29:44.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/158/19' 2023-10-30T13:29:44.879 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/158/15' 2023-10-30T13:29:44.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/158/20' 2023-10-30T13:29:44.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/158/12' 2023-10-30T13:29:44.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/158/10' 2023-10-30T13:29:44.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/158/11' 2023-10-30T13:29:44.880 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/158/16' 2023-10-30T13:29:44.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/158/14' 2023-10-30T13:29:44.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/158/18' 2023-10-30T13:29:44.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/158/13' 2023-10-30T13:29:44.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/158/17' 2023-10-30T13:29:44.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/158' 2023-10-30T13:29:44.881 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/106/19' 2023-10-30T13:29:44.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/106/15' 2023-10-30T13:29:44.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/106/20' 2023-10-30T13:29:44.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/106/12' 2023-10-30T13:29:44.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/106/10' 2023-10-30T13:29:44.882 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/106/11' 2023-10-30T13:29:44.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/106/16' 2023-10-30T13:29:44.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/106/14' 2023-10-30T13:29:44.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/106/18' 2023-10-30T13:29:44.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/106/13' 2023-10-30T13:29:44.883 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/106/17' 2023-10-30T13:29:44.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/106' 2023-10-30T13:29:44.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/189/19' 2023-10-30T13:29:44.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/189/15' 2023-10-30T13:29:44.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/189/20' 2023-10-30T13:29:44.884 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/189/12' 2023-10-30T13:29:44.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/189/10' 2023-10-30T13:29:44.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/189/11' 2023-10-30T13:29:44.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/189/16' 2023-10-30T13:29:44.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/189/14' 2023-10-30T13:29:44.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/189/18' 2023-10-30T13:29:44.885 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/189/13' 2023-10-30T13:29:44.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/189/17' 2023-10-30T13:29:44.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/189' 2023-10-30T13:29:44.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/133/19' 2023-10-30T13:29:44.886 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/133/15' 2023-10-30T13:29:44.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/133/20' 2023-10-30T13:29:44.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/133/12' 2023-10-30T13:29:44.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/133/10' 2023-10-30T13:29:44.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/133/11' 2023-10-30T13:29:44.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/133/16' 2023-10-30T13:29:44.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/133/14' 2023-10-30T13:29:44.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/133/18' 2023-10-30T13:29:44.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/133/13' 2023-10-30T13:29:44.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/133/17' 2023-10-30T13:29:44.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/133' 2023-10-30T13:29:44.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/173/19' 2023-10-30T13:29:44.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/173/15' 2023-10-30T13:29:44.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/173/20' 2023-10-30T13:29:44.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/173/12' 2023-10-30T13:29:44.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/173/10' 2023-10-30T13:29:44.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/173/11' 2023-10-30T13:29:45.600 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:29:45.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:29:46.156 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.304+0000 7fd76e394640 1 Processor -- start 2023-10-30T13:29:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.304+0000 7fd76e394640 1 -- start start 2023-10-30T13:29:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.304+0000 7fd76e394640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd7680c6bd0 0x7fd76806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.304+0000 7fd76e394640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd768066cf0 con 0x7fd76806f7a0 2023-10-30T13:29:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.304+0000 7fd76e394640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7680c70d0 con 0x7fd768071110 2023-10-30T13:29:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.304+0000 7fd76e394640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7680700c0 con 0x7fd7680c6bd0 2023-10-30T13:29:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.304+0000 7fd767fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd76806f7a0 0x7fd768069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58836/0 (socket says 172.21.15.46:58836) 2023-10-30T13:29:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.304+0000 7fd767fff640 1 -- 172.21.15.46:0/1329343854 learned_addr learned my addr 172.21.15.46:0/1329343854 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:29:46.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.304+0000 7fd7677fe640 1 --2- 172.21.15.46:0/1329343854 >> v2:172.21.15.154:3300/0 conn(0x7fd7680c6bd0 0x7fd76806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:46.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.304+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529555243 0 0) 0x7fd768066cf0 con 0x7fd76806f7a0 2023-10-30T13:29:46.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.304+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd750003530 con 0x7fd76806f7a0 2023-10-30T13:29:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3093418178 0 0) 0x7fd750003530 con 0x7fd76806f7a0 2023-10-30T13:29:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 >> v2:172.21.15.154:3300/0 conn(0x7fd7680c6bd0 msgr2=0x7fd76806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 --2- 172.21.15.46:0/1329343854 >> v2:172.21.15.154:3300/0 conn(0x7fd7680c6bd0 0x7fd76806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 >> v1:172.21.15.154:6789/0 conn(0x7fd768071110 legacy=0x7fd768158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7680c7390 con 0x7fd76806f7a0 2023-10-30T13:29:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fd758001680 con 0x7fd76806f7a0 2023-10-30T13:29:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 >> v1:172.21.15.46:6789/0 conn(0x7fd76806f7a0 legacy=0x7fd768069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 --2- 172.21.15.46:0/1329343854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd750003980 0x7fd750003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 --2- 172.21.15.46:0/1329343854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7500047d0 0x7fd750004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 --2- 172.21.15.46:0/1329343854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd750005190 0x7fd75000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7680700c0 con 0x7fd750005190 2023-10-30T13:29:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75000c040 con 0x7fd7500047d0 2023-10-30T13:29:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75000c200 con 0x7fd750003980 2023-10-30T13:29:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd76c931640 1 --2- 172.21.15.46:0/1329343854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd750005190 0x7fd75000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd7677fe640 1 --2- 172.21.15.46:0/1329343854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7500047d0 0x7fd750004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd767fff640 1 --2- 172.21.15.46:0/1329343854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd750003980 0x7fd750003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd76c931640 1 -- 172.21.15.46:0/1329343854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd750003980 msgr2=0x7fd750003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd76c931640 1 --2- 172.21.15.46:0/1329343854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd750003980 0x7fd750003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd76c931640 1 -- 172.21.15.46:0/1329343854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7500047d0 msgr2=0x7fd750004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd76c931640 1 --2- 172.21.15.46:0/1329343854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7500047d0 0x7fd750004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:46.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd76c931640 1 -- 172.21.15.46:0/1329343854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd750003530 con 0x7fd750005190 2023-10-30T13:29:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.308+0000 7fd76c931640 1 -- 172.21.15.46:0/1329343854 reap_dead start 2023-10-30T13:29:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76c931640 1 --2- 172.21.15.46:0/1329343854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd750005190 0x7fd75000b990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd760001e70 tx=0x7fd76001c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8987b628b373b9d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760003b80 con 0x7fd750005190 2023-10-30T13:29:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd760003d20 con 0x7fd750005190 2023-10-30T13:29:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd766ffd640 1 -- 172.21.15.46:0/1329343854 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760005260 con 0x7fd750005190 2023-10-30T13:29:46.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 -- 172.21.15.46:0/1329343854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd750005190 msgr2=0x7fd75000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:46.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 --2- 172.21.15.46:0/1329343854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd750005190 0x7fd75000b990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd760001e70 tx=0x7fd76001c760 comp rx=0 tx=0).stop 2023-10-30T13:29:46.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 -- 172.21.15.46:0/1329343854 shutdown_connections 2023-10-30T13:29:46.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 --2- 172.21.15.46:0/1329343854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd750005190 0x7fd75000b990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:46.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 -- 172.21.15.46:0/1329343854 >> 172.21.15.46:0/1329343854 conn(0x7fd7680c7910 msgr2=0x7fd7680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 -- 172.21.15.46:0/1329343854 shutdown_connections 2023-10-30T13:29:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 -- 172.21.15.46:0/1329343854 wait complete. 2023-10-30T13:29:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 Processor -- start 2023-10-30T13:29:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 -- start start 2023-10-30T13:29:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd750005190 0x7fd76814f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:46.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd76815b980 0x7fd76814fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:46.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7681f1000 0x7fd7681f1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:46.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd768157a40 con 0x7fd7681f1000 2023-10-30T13:29:46.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd768157000 con 0x7fd76815b980 2023-10-30T13:29:46.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.312+0000 7fd76e394640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7681f0b70 con 0x7fd750005190 2023-10-30T13:29:46.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd76c931640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7681f1000 0x7fd7681f1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:46.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd76c931640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7681f1000 0x7fd7681f1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56436/0 (socket says 172.21.15.46:56436) 2023-10-30T13:29:46.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd76c931640 1 -- 172.21.15.46:0/664021946 learned_addr learned my addr 172.21.15.46:0/664021946 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:29:46.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd7677fe640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd76815b980 0x7fd76814fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:46.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd767fff640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd750005190 0x7fd76814f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd76c931640 1 -- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd750005190 msgr2=0x7fd76814f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd76c931640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd750005190 0x7fd76814f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd76c931640 1 -- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd76815b980 msgr2=0x7fd76814fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd76c931640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd76815b980 0x7fd76814fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd76c931640 1 -- 172.21.15.46:0/664021946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7681f19d0 con 0x7fd7681f1000 2023-10-30T13:29:46.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd76c931640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7681f1000 0x7fd7681f1490 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd760098170 tx=0x7fd750005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:46.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd764ff9640 1 -- 172.21.15.46:0/664021946 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7600ac070 con 0x7fd7681f1000 2023-10-30T13:29:46.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd764ff9640 1 -- 172.21.15.46:0/664021946 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd760003e90 con 0x7fd7681f1000 2023-10-30T13:29:46.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd764ff9640 1 -- 172.21.15.46:0/664021946 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760006d60 con 0x7fd7681f1000 2023-10-30T13:29:46.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd76e394640 1 -- 172.21.15.46:0/664021946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7681582e0 con 0x7fd7681f1000 2023-10-30T13:29:46.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd76e394640 1 -- 172.21.15.46:0/664021946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd768156d20 con 0x7fd7681f1000 2023-10-30T13:29:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd76e394640 1 -- 172.21.15.46:0/664021946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76804e170 con 0x7fd7681f1000 2023-10-30T13:29:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd764ff9640 1 -- 172.21.15.46:0/664021946 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd760007030 con 0x7fd7681f1000 2023-10-30T13:29:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd764ff9640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd7540664d0 0x7fd754068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.316+0000 7fd764ff9640 1 -- 172.21.15.46:0/664021946 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fd76011d310 con 0x7fd7681f1000 2023-10-30T13:29:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.320+0000 7fd767fff640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd7540664d0 0x7fd754068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.320+0000 7fd767fff640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd7540664d0 0x7fd754068990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd758000c10 tx=0x7fd758009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.320+0000 7fd764ff9640 1 -- 172.21.15.46:0/664021946 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd7600ea4f0 con 0x7fd7681f1000 2023-10-30T13:29:46.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 -- 172.21.15.46:0/664021946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7681f0cb0 con 0x7fd7681f1000 2023-10-30T13:29:46.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd764ff9640 1 -- 172.21.15.46:0/664021946 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6527 (secure 0 0 0) 0x7fd7600b8020 con 0x7fd7681f1000 2023-10-30T13:29:46.519 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:29:46.519 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:36.923344+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:46.520 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 35 2023-10-30T13:29:46.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 -- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd7540664d0 msgr2=0x7fd754068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:46.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd7540664d0 0x7fd754068990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd758000c10 tx=0x7fd758009040 comp rx=0 tx=0).stop 2023-10-30T13:29:46.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 -- 172.21.15.46:0/664021946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7681f1000 msgr2=0x7fd7681f1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:46.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7681f1000 0x7fd7681f1490 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd760098170 tx=0x7fd750005800 comp rx=0 tx=0).stop 2023-10-30T13:29:46.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 -- 172.21.15.46:0/664021946 shutdown_connections 2023-10-30T13:29:46.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd7540664d0 0x7fd754068990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:46.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7681f1000 0x7fd7681f1490 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:46.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd76815b980 0x7fd76814fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:46.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 --2- 172.21.15.46:0/664021946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd750005190 0x7fd76814f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:46.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 -- 172.21.15.46:0/664021946 >> 172.21.15.46:0/664021946 conn(0x7fd7680c7910 msgr2=0x7fd7680c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:46.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 -- 172.21.15.46:0/664021946 shutdown_connections 2023-10-30T13:29:46.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:29:46.512+0000 7fd76e394640 1 -- 172.21.15.46:0/664021946 wait complete. 2023-10-30T13:29:46.540 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-30T13:29:46.540 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 29 secs before thrashing 2023-10-30T13:29:52.711 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/173/16' 2023-10-30T13:29:52.711 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/173/14' 2023-10-30T13:29:52.711 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/173/18' 2023-10-30T13:29:52.712 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/173/13' 2023-10-30T13:29:52.712 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/173/17' 2023-10-30T13:29:52.712 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/173' 2023-10-30T13:29:52.712 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/174/19' 2023-10-30T13:29:52.712 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/174/15' 2023-10-30T13:29:52.712 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/174/20' 2023-10-30T13:29:52.713 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/174/12' 2023-10-30T13:29:52.713 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/174/10' 2023-10-30T13:29:52.713 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/174/11' 2023-10-30T13:29:52.713 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/174/16' 2023-10-30T13:29:52.713 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/174/14' 2023-10-30T13:29:52.713 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/174/18' 2023-10-30T13:29:52.714 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/174/13' 2023-10-30T13:29:52.714 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/174/17' 2023-10-30T13:29:52.714 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/174' 2023-10-30T13:29:52.714 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/155/19' 2023-10-30T13:29:52.714 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/155/15' 2023-10-30T13:29:52.715 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/155/20' 2023-10-30T13:29:52.715 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/155/12' 2023-10-30T13:29:52.715 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/155/10' 2023-10-30T13:29:52.715 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/155/11' 2023-10-30T13:29:52.715 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/155/16' 2023-10-30T13:29:52.715 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/155/14' 2023-10-30T13:29:52.716 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/155/18' 2023-10-30T13:29:52.716 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/155/13' 2023-10-30T13:29:52.716 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/155/17' 2023-10-30T13:29:52.716 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/155' 2023-10-30T13:29:52.716 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/114/19' 2023-10-30T13:29:52.716 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/114/15' 2023-10-30T13:29:52.717 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/114/20' 2023-10-30T13:29:52.717 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/114/12' 2023-10-30T13:29:52.717 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/114/10' 2023-10-30T13:29:52.717 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/114/11' 2023-10-30T13:29:52.717 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/114/16' 2023-10-30T13:29:52.718 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/114/14' 2023-10-30T13:29:52.718 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/114/18' 2023-10-30T13:29:52.718 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/114/13' 2023-10-30T13:29:52.718 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/114/17' 2023-10-30T13:29:52.718 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/114' 2023-10-30T13:29:52.719 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/159/19' 2023-10-30T13:29:52.719 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/159/15' 2023-10-30T13:29:52.719 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/159/20' 2023-10-30T13:29:52.719 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/159/12' 2023-10-30T13:29:52.719 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/159/10' 2023-10-30T13:29:52.719 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/159/11' 2023-10-30T13:29:52.720 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/159/16' 2023-10-30T13:29:52.720 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/159/14' 2023-10-30T13:29:52.720 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/159/18' 2023-10-30T13:29:52.720 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/159/13' 2023-10-30T13:29:52.720 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/159/17' 2023-10-30T13:29:52.721 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/159' 2023-10-30T13:29:52.721 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/187/19' 2023-10-30T13:29:52.721 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/187/15' 2023-10-30T13:29:52.721 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/187/20' 2023-10-30T13:29:52.721 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/187/12' 2023-10-30T13:29:52.721 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/187/10' 2023-10-30T13:29:52.722 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/187/11' 2023-10-30T13:29:52.722 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/187/16' 2023-10-30T13:29:52.722 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/187/14' 2023-10-30T13:29:52.722 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/187/18' 2023-10-30T13:29:52.722 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/187/13' 2023-10-30T13:29:52.722 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/187/17' 2023-10-30T13:29:52.723 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/187' 2023-10-30T13:29:52.723 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/185/19' 2023-10-30T13:29:52.723 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/185/15' 2023-10-30T13:29:52.723 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/185/20' 2023-10-30T13:29:52.723 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/185/12' 2023-10-30T13:29:52.723 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/185/10' 2023-10-30T13:29:52.724 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/185/11' 2023-10-30T13:29:52.724 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/185/16' 2023-10-30T13:29:52.724 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/185/14' 2023-10-30T13:29:52.724 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/185/18' 2023-10-30T13:29:52.724 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/185/13' 2023-10-30T13:29:52.724 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/185/17' 2023-10-30T13:29:52.725 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/185' 2023-10-30T13:29:52.725 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/120/19' 2023-10-30T13:29:52.725 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/120/15' 2023-10-30T13:29:52.725 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/120/20' 2023-10-30T13:29:52.725 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/120/12' 2023-10-30T13:29:52.725 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/120/10' 2023-10-30T13:29:52.726 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/120/11' 2023-10-30T13:29:52.726 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/120/16' 2023-10-30T13:29:52.726 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/120/14' 2023-10-30T13:29:52.726 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/120/18' 2023-10-30T13:29:52.726 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/120/13' 2023-10-30T13:29:52.727 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/120/17' 2023-10-30T13:29:52.727 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/120' 2023-10-30T13:29:52.727 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/110/19' 2023-10-30T13:29:52.727 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/110/15' 2023-10-30T13:29:52.727 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/110/20' 2023-10-30T13:29:52.727 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/110/12' 2023-10-30T13:29:52.728 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/110/10' 2023-10-30T13:29:52.728 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/110/11' 2023-10-30T13:29:52.728 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/110/16' 2023-10-30T13:29:52.728 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/110/14' 2023-10-30T13:29:52.728 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/110/18' 2023-10-30T13:29:52.728 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/110/13' 2023-10-30T13:29:52.729 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/110/17' 2023-10-30T13:29:52.729 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/110' 2023-10-30T13:29:52.729 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/137/19' 2023-10-30T13:29:52.729 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/137/15' 2023-10-30T13:29:52.729 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/137/20' 2023-10-30T13:29:52.730 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/137/12' 2023-10-30T13:29:52.730 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/137/10' 2023-10-30T13:29:52.730 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/137/11' 2023-10-30T13:29:52.730 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/137/16' 2023-10-30T13:29:58.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/137/14' 2023-10-30T13:29:58.224 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/137/18' 2023-10-30T13:29:58.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/137/13' 2023-10-30T13:29:58.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/137/17' 2023-10-30T13:29:58.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/137' 2023-10-30T13:29:58.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/198/19' 2023-10-30T13:29:58.225 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/198/15' 2023-10-30T13:29:58.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/198/20' 2023-10-30T13:29:58.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/198/12' 2023-10-30T13:29:58.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/198/10' 2023-10-30T13:29:58.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/198/11' 2023-10-30T13:29:58.226 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/198/16' 2023-10-30T13:29:58.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/198/14' 2023-10-30T13:29:58.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/198/18' 2023-10-30T13:29:58.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/198/13' 2023-10-30T13:29:58.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/198/17' 2023-10-30T13:29:58.227 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/198' 2023-10-30T13:29:58.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/136/19' 2023-10-30T13:29:58.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/136/15' 2023-10-30T13:29:58.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/136/20' 2023-10-30T13:29:58.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/136/12' 2023-10-30T13:29:58.228 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/136/10' 2023-10-30T13:29:58.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/136/11' 2023-10-30T13:29:58.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/136/16' 2023-10-30T13:29:58.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/136/14' 2023-10-30T13:29:58.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/136/18' 2023-10-30T13:29:58.229 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/136/13' 2023-10-30T13:29:58.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/136/17' 2023-10-30T13:29:58.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/136' 2023-10-30T13:29:58.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/125/19' 2023-10-30T13:29:58.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/125/15' 2023-10-30T13:29:58.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/125/20' 2023-10-30T13:29:58.230 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/125/12' 2023-10-30T13:29:58.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/125/10' 2023-10-30T13:29:58.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/125/11' 2023-10-30T13:29:58.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/125/16' 2023-10-30T13:29:58.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/125/14' 2023-10-30T13:29:58.231 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/125/18' 2023-10-30T13:29:58.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/125/13' 2023-10-30T13:29:58.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/125/17' 2023-10-30T13:29:58.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/125' 2023-10-30T13:29:58.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/194/19' 2023-10-30T13:29:58.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/194/15' 2023-10-30T13:29:58.232 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/194/20' 2023-10-30T13:29:58.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/194/12' 2023-10-30T13:29:58.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/194/10' 2023-10-30T13:29:58.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/194/11' 2023-10-30T13:29:58.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/194/16' 2023-10-30T13:29:58.233 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/194/14' 2023-10-30T13:29:58.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/194/18' 2023-10-30T13:29:58.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/194/13' 2023-10-30T13:29:58.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/194/17' 2023-10-30T13:29:58.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/194' 2023-10-30T13:29:58.234 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/119/19' 2023-10-30T13:29:58.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/119/15' 2023-10-30T13:29:58.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/119/20' 2023-10-30T13:29:58.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/119/12' 2023-10-30T13:29:58.235 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/119/10' 2023-10-30T13:29:58.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/119/11' 2023-10-30T13:29:58.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/119/16' 2023-10-30T13:29:58.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/119/14' 2023-10-30T13:29:58.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/119/18' 2023-10-30T13:29:58.236 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/119/13' 2023-10-30T13:29:58.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/119/17' 2023-10-30T13:29:58.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/119' 2023-10-30T13:29:58.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/157/19' 2023-10-30T13:29:58.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/157/15' 2023-10-30T13:29:58.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/157/20' 2023-10-30T13:29:58.237 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/157/12' 2023-10-30T13:29:58.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/157/10' 2023-10-30T13:29:58.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/157/11' 2023-10-30T13:29:58.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/157/16' 2023-10-30T13:29:58.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/157/14' 2023-10-30T13:29:58.238 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/157/18' 2023-10-30T13:29:58.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/157/13' 2023-10-30T13:29:58.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/157/17' 2023-10-30T13:29:58.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/157' 2023-10-30T13:29:58.239 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/172/19' 2023-10-30T13:29:58.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/172/15' 2023-10-30T13:29:58.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/172/20' 2023-10-30T13:29:58.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/172/12' 2023-10-30T13:29:58.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/172/10' 2023-10-30T13:29:58.240 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/172/11' 2023-10-30T13:29:58.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/172/16' 2023-10-30T13:29:58.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/172/14' 2023-10-30T13:29:58.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/172/18' 2023-10-30T13:29:58.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/172/13' 2023-10-30T13:29:58.241 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/172/17' 2023-10-30T13:29:58.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/172' 2023-10-30T13:29:58.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/186/19' 2023-10-30T13:29:58.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/186/15' 2023-10-30T13:29:58.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/186/20' 2023-10-30T13:29:58.242 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/186/12' 2023-10-30T13:29:58.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/186/10' 2023-10-30T13:29:58.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/186/11' 2023-10-30T13:29:58.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/186/16' 2023-10-30T13:29:58.243 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/186/14' 2023-10-30T13:29:58.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/186/18' 2023-10-30T13:29:58.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/186/13' 2023-10-30T13:29:58.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/186/17' 2023-10-30T13:29:58.244 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/186' 2023-10-30T13:29:58.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/190/19' 2023-10-30T13:29:58.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/190/15' 2023-10-30T13:29:58.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/190/20' 2023-10-30T13:29:58.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/190/12' 2023-10-30T13:29:58.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/190/10' 2023-10-30T13:29:58.245 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/190/11' 2023-10-30T13:29:58.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/190/16' 2023-10-30T13:29:58.246 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/190/14' 2023-10-30T13:29:59.932 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/190/18' 2023-10-30T13:29:59.932 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/190/13' 2023-10-30T13:29:59.932 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/190/17' 2023-10-30T13:29:59.932 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/190' 2023-10-30T13:29:59.933 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/104/19' 2023-10-30T13:29:59.933 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/104/15' 2023-10-30T13:29:59.933 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/104/20' 2023-10-30T13:29:59.933 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/104/12' 2023-10-30T13:29:59.933 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/104/10' 2023-10-30T13:29:59.934 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/104/11' 2023-10-30T13:29:59.934 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/104/16' 2023-10-30T13:29:59.934 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/104/14' 2023-10-30T13:29:59.934 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/104/18' 2023-10-30T13:29:59.934 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/104/13' 2023-10-30T13:29:59.935 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/104/17' 2023-10-30T13:29:59.935 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/104' 2023-10-30T13:29:59.935 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/164/19' 2023-10-30T13:29:59.935 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/164/15' 2023-10-30T13:29:59.935 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/164/20' 2023-10-30T13:29:59.935 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/164/12' 2023-10-30T13:29:59.936 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/164/10' 2023-10-30T13:29:59.936 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/164/11' 2023-10-30T13:29:59.936 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/164/16' 2023-10-30T13:29:59.936 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/164/14' 2023-10-30T13:29:59.936 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/164/18' 2023-10-30T13:29:59.937 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/164/13' 2023-10-30T13:29:59.937 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/164/17' 2023-10-30T13:29:59.937 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/164' 2023-10-30T13:29:59.937 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/112/19' 2023-10-30T13:29:59.937 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/112/15' 2023-10-30T13:29:59.937 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/112/20' 2023-10-30T13:29:59.938 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/112/12' 2023-10-30T13:29:59.938 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/112/10' 2023-10-30T13:29:59.938 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/112/11' 2023-10-30T13:29:59.938 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/112/16' 2023-10-30T13:29:59.938 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/112/14' 2023-10-30T13:29:59.939 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/112/18' 2023-10-30T13:29:59.939 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/112/13' 2023-10-30T13:29:59.939 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/112/17' 2023-10-30T13:29:59.939 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/112' 2023-10-30T13:29:59.939 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/100/19' 2023-10-30T13:29:59.939 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/100/15' 2023-10-30T13:29:59.940 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/100/20' 2023-10-30T13:29:59.940 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/100/12' 2023-10-30T13:29:59.940 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/100/10' 2023-10-30T13:29:59.940 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/100/11' 2023-10-30T13:29:59.940 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/100/16' 2023-10-30T13:29:59.941 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/100/14' 2023-10-30T13:29:59.941 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/100/18' 2023-10-30T13:29:59.941 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/100/13' 2023-10-30T13:29:59.941 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/100/17' 2023-10-30T13:29:59.941 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/100' 2023-10-30T13:29:59.941 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/134/19' 2023-10-30T13:29:59.942 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/134/15' 2023-10-30T13:29:59.942 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/134/20' 2023-10-30T13:29:59.942 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/134/12' 2023-10-30T13:29:59.942 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/134/10' 2023-10-30T13:29:59.942 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/134/11' 2023-10-30T13:29:59.943 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/134/16' 2023-10-30T13:29:59.943 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/134/14' 2023-10-30T13:29:59.943 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/134/18' 2023-10-30T13:29:59.943 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/134/13' 2023-10-30T13:29:59.943 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/134/17' 2023-10-30T13:29:59.944 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/134' 2023-10-30T13:29:59.944 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/115/19' 2023-10-30T13:29:59.944 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/115/15' 2023-10-30T13:29:59.944 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/115/20' 2023-10-30T13:29:59.944 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/115/12' 2023-10-30T13:29:59.945 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/115/10' 2023-10-30T13:29:59.945 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/115/11' 2023-10-30T13:29:59.945 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/115/16' 2023-10-30T13:29:59.945 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/115/14' 2023-10-30T13:29:59.945 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/115/18' 2023-10-30T13:29:59.945 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/115/13' 2023-10-30T13:29:59.946 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/115/17' 2023-10-30T13:29:59.946 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/115' 2023-10-30T13:29:59.946 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/116/19' 2023-10-30T13:29:59.946 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/116/15' 2023-10-30T13:29:59.946 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/116/20' 2023-10-30T13:29:59.947 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/116/12' 2023-10-30T13:29:59.947 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/116/10' 2023-10-30T13:29:59.947 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/116/11' 2023-10-30T13:29:59.947 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/116/16' 2023-10-30T13:29:59.947 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/116/14' 2023-10-30T13:29:59.948 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/116/18' 2023-10-30T13:29:59.948 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/116/13' 2023-10-30T13:29:59.948 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/116/17' 2023-10-30T13:29:59.948 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/116' 2023-10-30T13:29:59.948 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/195/19' 2023-10-30T13:29:59.949 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/195/15' 2023-10-30T13:29:59.949 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/195/20' 2023-10-30T13:29:59.949 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/195/12' 2023-10-30T13:29:59.949 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/195/10' 2023-10-30T13:29:59.949 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/195/11' 2023-10-30T13:29:59.950 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/195/16' 2023-10-30T13:29:59.950 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/195/14' 2023-10-30T13:29:59.950 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/195/18' 2023-10-30T13:29:59.950 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/195/13' 2023-10-30T13:29:59.950 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/195/17' 2023-10-30T13:29:59.951 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/195' 2023-10-30T13:29:59.951 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/138/19' 2023-10-30T13:29:59.951 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/138/15' 2023-10-30T13:29:59.951 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/138/20' 2023-10-30T13:29:59.952 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/138/12' 2023-10-30T13:29:59.952 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/138/10' 2023-10-30T13:29:59.952 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/138/11' 2023-10-30T13:29:59.952 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/138/16' 2023-10-30T13:29:59.952 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/138/14' 2023-10-30T13:29:59.952 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/138/18' 2023-10-30T13:30:01.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/138/13' 2023-10-30T13:30:01.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/138/17' 2023-10-30T13:30:01.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/138' 2023-10-30T13:30:01.887 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/135/19' 2023-10-30T13:30:01.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/135/15' 2023-10-30T13:30:01.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/135/20' 2023-10-30T13:30:01.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/135/12' 2023-10-30T13:30:01.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/135/10' 2023-10-30T13:30:01.888 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/135/11' 2023-10-30T13:30:01.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/135/16' 2023-10-30T13:30:01.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/135/14' 2023-10-30T13:30:01.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/135/18' 2023-10-30T13:30:01.889 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/135/13' 2023-10-30T13:30:01.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/135/17' 2023-10-30T13:30:01.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/135' 2023-10-30T13:30:01.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/117/19' 2023-10-30T13:30:01.890 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/117/15' 2023-10-30T13:30:01.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/117/20' 2023-10-30T13:30:01.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/117/12' 2023-10-30T13:30:01.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/117/10' 2023-10-30T13:30:01.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/117/11' 2023-10-30T13:30:01.891 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/117/16' 2023-10-30T13:30:01.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/117/14' 2023-10-30T13:30:01.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/117/18' 2023-10-30T13:30:01.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/117/13' 2023-10-30T13:30:01.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/117/17' 2023-10-30T13:30:01.892 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/117' 2023-10-30T13:30:01.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/143/19' 2023-10-30T13:30:01.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/143/15' 2023-10-30T13:30:01.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/143/20' 2023-10-30T13:30:01.893 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/143/12' 2023-10-30T13:30:01.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/143/10' 2023-10-30T13:30:01.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/143/11' 2023-10-30T13:30:01.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/143/16' 2023-10-30T13:30:01.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/143/14' 2023-10-30T13:30:01.894 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/143/18' 2023-10-30T13:30:01.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/143/13' 2023-10-30T13:30:01.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/143/17' 2023-10-30T13:30:01.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/143' 2023-10-30T13:30:01.895 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/180/19' 2023-10-30T13:30:01.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/180/15' 2023-10-30T13:30:01.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/180/20' 2023-10-30T13:30:01.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/180/12' 2023-10-30T13:30:01.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/180/10' 2023-10-30T13:30:01.896 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/180/11' 2023-10-30T13:30:01.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/180/16' 2023-10-30T13:30:01.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/180/14' 2023-10-30T13:30:01.897 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/180/18' 2023-10-30T13:30:01.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/180/13' 2023-10-30T13:30:01.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/180/17' 2023-10-30T13:30:01.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/180' 2023-10-30T13:30:01.898 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/126/19' 2023-10-30T13:30:01.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/126/15' 2023-10-30T13:30:01.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/126/20' 2023-10-30T13:30:01.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/126/12' 2023-10-30T13:30:01.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/126/10' 2023-10-30T13:30:01.899 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/126/11' 2023-10-30T13:30:01.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/126/16' 2023-10-30T13:30:01.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/126/14' 2023-10-30T13:30:01.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/126/18' 2023-10-30T13:30:01.900 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/126/13' 2023-10-30T13:30:01.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/126/17' 2023-10-30T13:30:01.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/126' 2023-10-30T13:30:01.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/161/19' 2023-10-30T13:30:01.901 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/161/15' 2023-10-30T13:30:01.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/161/20' 2023-10-30T13:30:01.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/161/12' 2023-10-30T13:30:01.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/161/10' 2023-10-30T13:30:01.902 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/161/11' 2023-10-30T13:30:01.905 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/161/16' 2023-10-30T13:30:01.906 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/161/14' 2023-10-30T13:30:01.906 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/161/18' 2023-10-30T13:30:01.906 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/161/13' 2023-10-30T13:30:01.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/161/17' 2023-10-30T13:30:01.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/161' 2023-10-30T13:30:01.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/176/19' 2023-10-30T13:30:01.907 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/176/15' 2023-10-30T13:30:01.908 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/176/20' 2023-10-30T13:30:01.908 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/176/12' 2023-10-30T13:30:01.908 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/176/10' 2023-10-30T13:30:01.908 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/176/11' 2023-10-30T13:30:01.908 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/176/16' 2023-10-30T13:30:01.909 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/176/14' 2023-10-30T13:30:01.909 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/176/18' 2023-10-30T13:30:01.909 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/176/13' 2023-10-30T13:30:01.909 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/176/17' 2023-10-30T13:30:01.910 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/176' 2023-10-30T13:30:01.910 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/111/19' 2023-10-30T13:30:01.910 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/111/15' 2023-10-30T13:30:01.910 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/111/20' 2023-10-30T13:30:01.910 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/111/12' 2023-10-30T13:30:01.911 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/111/10' 2023-10-30T13:30:01.911 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/111/11' 2023-10-30T13:30:01.911 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/111/16' 2023-10-30T13:30:01.911 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/111/14' 2023-10-30T13:30:01.912 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/111/18' 2023-10-30T13:30:01.912 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/111/13' 2023-10-30T13:30:01.912 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/111/17' 2023-10-30T13:30:01.912 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/111' 2023-10-30T13:30:01.912 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/191/19' 2023-10-30T13:30:01.913 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/191/15' 2023-10-30T13:30:01.913 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/191/20' 2023-10-30T13:30:01.913 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/191/12' 2023-10-30T13:30:01.913 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/191/10' 2023-10-30T13:30:01.913 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/191/11' 2023-10-30T13:30:01.914 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/191/16' 2023-10-30T13:30:01.914 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/191/14' 2023-10-30T13:30:01.914 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/191/18' 2023-10-30T13:30:01.914 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/191/13' 2023-10-30T13:30:03.312 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/191/17' 2023-10-30T13:30:03.312 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/191' 2023-10-30T13:30:03.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/184/19' 2023-10-30T13:30:03.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/184/15' 2023-10-30T13:30:03.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/184/20' 2023-10-30T13:30:03.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/184/12' 2023-10-30T13:30:03.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/184/10' 2023-10-30T13:30:03.313 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/184/11' 2023-10-30T13:30:03.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/184/16' 2023-10-30T13:30:03.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/184/14' 2023-10-30T13:30:03.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/184/18' 2023-10-30T13:30:03.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/184/13' 2023-10-30T13:30:03.314 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/184/17' 2023-10-30T13:30:03.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/184' 2023-10-30T13:30:03.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/192/19' 2023-10-30T13:30:03.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/192/15' 2023-10-30T13:30:03.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/192/20' 2023-10-30T13:30:03.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/192/12' 2023-10-30T13:30:03.315 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/192/10' 2023-10-30T13:30:03.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/192/11' 2023-10-30T13:30:03.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/192/16' 2023-10-30T13:30:03.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/192/14' 2023-10-30T13:30:03.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/192/18' 2023-10-30T13:30:03.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/192/13' 2023-10-30T13:30:03.316 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/192/17' 2023-10-30T13:30:03.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/192' 2023-10-30T13:30:03.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/103/19' 2023-10-30T13:30:03.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/103/15' 2023-10-30T13:30:03.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/103/20' 2023-10-30T13:30:03.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/103/12' 2023-10-30T13:30:03.317 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/103/10' 2023-10-30T13:30:03.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/103/11' 2023-10-30T13:30:03.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/103/16' 2023-10-30T13:30:03.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/103/14' 2023-10-30T13:30:03.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/103/18' 2023-10-30T13:30:03.318 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/103/13' 2023-10-30T13:30:03.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/103/17' 2023-10-30T13:30:03.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/103' 2023-10-30T13:30:03.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/128/19' 2023-10-30T13:30:03.319 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/128/15' 2023-10-30T13:30:03.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/128/20' 2023-10-30T13:30:03.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/128/12' 2023-10-30T13:30:03.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/128/10' 2023-10-30T13:30:03.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/128/11' 2023-10-30T13:30:03.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/128/16' 2023-10-30T13:30:03.320 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/128/14' 2023-10-30T13:30:03.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/128/18' 2023-10-30T13:30:03.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/128/13' 2023-10-30T13:30:03.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/128/17' 2023-10-30T13:30:03.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/128' 2023-10-30T13:30:03.321 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/144/19' 2023-10-30T13:30:03.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/144/15' 2023-10-30T13:30:03.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/144/20' 2023-10-30T13:30:03.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/144/12' 2023-10-30T13:30:03.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/144/10' 2023-10-30T13:30:03.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/144/11' 2023-10-30T13:30:03.322 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/144/16' 2023-10-30T13:30:03.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/144/14' 2023-10-30T13:30:03.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/144/18' 2023-10-30T13:30:03.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/144/13' 2023-10-30T13:30:03.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/144/17' 2023-10-30T13:30:03.323 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/144' 2023-10-30T13:30:03.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/146/19' 2023-10-30T13:30:03.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/146/15' 2023-10-30T13:30:03.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/146/20' 2023-10-30T13:30:03.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/146/12' 2023-10-30T13:30:03.324 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/146/10' 2023-10-30T13:30:03.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/146/11' 2023-10-30T13:30:03.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/146/16' 2023-10-30T13:30:03.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/146/14' 2023-10-30T13:30:03.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/146/18' 2023-10-30T13:30:03.325 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/146/13' 2023-10-30T13:30:03.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/146/17' 2023-10-30T13:30:03.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/146' 2023-10-30T13:30:03.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/141/19' 2023-10-30T13:30:03.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/141/15' 2023-10-30T13:30:03.326 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/141/20' 2023-10-30T13:30:03.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/141/12' 2023-10-30T13:30:03.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/141/10' 2023-10-30T13:30:03.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/141/11' 2023-10-30T13:30:03.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/141/16' 2023-10-30T13:30:03.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/141/14' 2023-10-30T13:30:03.327 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/141/18' 2023-10-30T13:30:03.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/141/13' 2023-10-30T13:30:03.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/141/17' 2023-10-30T13:30:03.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/141' 2023-10-30T13:30:03.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/182/19' 2023-10-30T13:30:03.328 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/182/15' 2023-10-30T13:30:03.329 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/182/20' 2023-10-30T13:30:03.329 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/182/12' 2023-10-30T13:30:03.329 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/182/10' 2023-10-30T13:30:03.329 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/182/11' 2023-10-30T13:30:03.329 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/182/16' 2023-10-30T13:30:03.329 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/182/14' 2023-10-30T13:30:03.330 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/182/18' 2023-10-30T13:30:03.330 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/182/13' 2023-10-30T13:30:03.330 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/182/17' 2023-10-30T13:30:03.330 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/182' 2023-10-30T13:30:03.330 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/107/19' 2023-10-30T13:30:03.331 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/107/15' 2023-10-30T13:30:03.331 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/107/20' 2023-10-30T13:30:03.331 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/107/12' 2023-10-30T13:30:03.331 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/107/10' 2023-10-30T13:30:03.331 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/107/11' 2023-10-30T13:30:03.331 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/107/16' 2023-10-30T13:30:03.332 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/107/14' 2023-10-30T13:30:03.332 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/107/18' 2023-10-30T13:30:03.332 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/107/13' 2023-10-30T13:30:03.332 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/107/17' 2023-10-30T13:30:11.102 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/107' 2023-10-30T13:30:11.102 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/178/19' 2023-10-30T13:30:11.103 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/178/15' 2023-10-30T13:30:11.103 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/178/20' 2023-10-30T13:30:11.103 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/178/12' 2023-10-30T13:30:11.103 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/178/10' 2023-10-30T13:30:11.103 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/178/11' 2023-10-30T13:30:11.104 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/178/16' 2023-10-30T13:30:11.104 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/178/14' 2023-10-30T13:30:11.104 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/178/18' 2023-10-30T13:30:11.104 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/178/13' 2023-10-30T13:30:11.104 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/178/17' 2023-10-30T13:30:11.105 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/178' 2023-10-30T13:30:11.105 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/156/19' 2023-10-30T13:30:11.105 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/156/15' 2023-10-30T13:30:11.105 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/156/20' 2023-10-30T13:30:11.105 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/156/12' 2023-10-30T13:30:11.106 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/156/10' 2023-10-30T13:30:11.106 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/156/11' 2023-10-30T13:30:11.106 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/156/16' 2023-10-30T13:30:11.106 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/156/14' 2023-10-30T13:30:11.106 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/156/18' 2023-10-30T13:30:11.107 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/156/13' 2023-10-30T13:30:11.107 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/156/17' 2023-10-30T13:30:11.107 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/156' 2023-10-30T13:30:11.107 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/131/19' 2023-10-30T13:30:11.107 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/131/15' 2023-10-30T13:30:11.107 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/131/20' 2023-10-30T13:30:11.108 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/131/12' 2023-10-30T13:30:11.108 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/131/10' 2023-10-30T13:30:11.108 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/131/11' 2023-10-30T13:30:11.108 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/131/16' 2023-10-30T13:30:11.108 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/131/14' 2023-10-30T13:30:11.109 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/131/18' 2023-10-30T13:30:11.109 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/131/13' 2023-10-30T13:30:11.109 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/131/17' 2023-10-30T13:30:11.109 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/131' 2023-10-30T13:30:11.109 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/123/19' 2023-10-30T13:30:11.109 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/123/15' 2023-10-30T13:30:11.110 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/123/20' 2023-10-30T13:30:11.110 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/123/12' 2023-10-30T13:30:11.110 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/123/10' 2023-10-30T13:30:11.110 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/123/11' 2023-10-30T13:30:11.110 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/123/16' 2023-10-30T13:30:11.111 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/123/14' 2023-10-30T13:30:11.111 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/123/18' 2023-10-30T13:30:11.111 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/123/13' 2023-10-30T13:30:11.111 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/123/17' 2023-10-30T13:30:11.111 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/123' 2023-10-30T13:30:11.111 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/160/19' 2023-10-30T13:30:11.112 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/160/15' 2023-10-30T13:30:11.112 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/160/20' 2023-10-30T13:30:11.112 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/160/12' 2023-10-30T13:30:11.112 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/160/10' 2023-10-30T13:30:11.113 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/160/11' 2023-10-30T13:30:11.113 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/160/16' 2023-10-30T13:30:11.113 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/160/14' 2023-10-30T13:30:11.113 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/160/18' 2023-10-30T13:30:11.114 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/160/13' 2023-10-30T13:30:11.114 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/160/17' 2023-10-30T13:30:11.114 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/160' 2023-10-30T13:30:11.114 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/165/19' 2023-10-30T13:30:11.114 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/165/15' 2023-10-30T13:30:11.115 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/165/20' 2023-10-30T13:30:11.115 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/165/12' 2023-10-30T13:30:11.115 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/165/10' 2023-10-30T13:30:11.115 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/165/11' 2023-10-30T13:30:11.115 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/165/16' 2023-10-30T13:30:11.116 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/165/14' 2023-10-30T13:30:11.116 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/165/18' 2023-10-30T13:30:11.116 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/165/13' 2023-10-30T13:30:11.116 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/165/17' 2023-10-30T13:30:11.117 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/165' 2023-10-30T13:30:11.117 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/101/19' 2023-10-30T13:30:11.117 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/101/15' 2023-10-30T13:30:11.117 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/101/20' 2023-10-30T13:30:11.117 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/101/12' 2023-10-30T13:30:11.117 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/101/10' 2023-10-30T13:30:11.118 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/101/11' 2023-10-30T13:30:11.118 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/101/16' 2023-10-30T13:30:11.118 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/101/14' 2023-10-30T13:30:11.118 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/101/18' 2023-10-30T13:30:11.118 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/101/13' 2023-10-30T13:30:11.119 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/101/17' 2023-10-30T13:30:11.119 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/101' 2023-10-30T13:30:11.119 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/148/19' 2023-10-30T13:30:11.119 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/148/15' 2023-10-30T13:30:11.119 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/148/20' 2023-10-30T13:30:11.120 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/148/12' 2023-10-30T13:30:11.120 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/148/10' 2023-10-30T13:30:11.120 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/148/11' 2023-10-30T13:30:11.120 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/148/16' 2023-10-30T13:30:11.120 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/148/14' 2023-10-30T13:30:11.121 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/148/18' 2023-10-30T13:30:11.121 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/148/13' 2023-10-30T13:30:11.121 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/148/17' 2023-10-30T13:30:11.121 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/148' 2023-10-30T13:30:11.121 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/168/19' 2023-10-30T13:30:11.121 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/168/15' 2023-10-30T13:30:11.122 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/168/20' 2023-10-30T13:30:11.122 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/168/12' 2023-10-30T13:30:11.122 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/168/10' 2023-10-30T13:30:11.122 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/168/11' 2023-10-30T13:30:11.122 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/168/16' 2023-10-30T13:30:11.123 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/168/14' 2023-10-30T13:30:11.123 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/168/18' 2023-10-30T13:30:11.123 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/168/13' 2023-10-30T13:30:11.123 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/168/17' 2023-10-30T13:30:11.124 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/168' 2023-10-30T13:30:12.649 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/179/19' 2023-10-30T13:30:12.649 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/179/15' 2023-10-30T13:30:12.649 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/179/20' 2023-10-30T13:30:12.650 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/179/12' 2023-10-30T13:30:12.650 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/179/10' 2023-10-30T13:30:12.650 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/179/11' 2023-10-30T13:30:12.650 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/179/16' 2023-10-30T13:30:12.651 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/179/14' 2023-10-30T13:30:12.651 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/179/18' 2023-10-30T13:30:12.651 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/179/13' 2023-10-30T13:30:12.651 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/179/17' 2023-10-30T13:30:12.651 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/179' 2023-10-30T13:30:12.651 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/132/19' 2023-10-30T13:30:12.652 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/132/15' 2023-10-30T13:30:12.652 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/132/20' 2023-10-30T13:30:12.652 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/132/12' 2023-10-30T13:30:12.652 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/132/10' 2023-10-30T13:30:12.652 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/132/11' 2023-10-30T13:30:12.652 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/132/16' 2023-10-30T13:30:12.653 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/132/14' 2023-10-30T13:30:12.653 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/132/18' 2023-10-30T13:30:12.653 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/132/13' 2023-10-30T13:30:12.653 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/132/17' 2023-10-30T13:30:12.653 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/132' 2023-10-30T13:30:12.654 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/199/19' 2023-10-30T13:30:12.654 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/199/15' 2023-10-30T13:30:12.654 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/199/20' 2023-10-30T13:30:12.654 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/199/12' 2023-10-30T13:30:12.654 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/199/10' 2023-10-30T13:30:12.655 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/199/11' 2023-10-30T13:30:12.655 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/199/16' 2023-10-30T13:30:12.655 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/199/14' 2023-10-30T13:30:12.655 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/199/18' 2023-10-30T13:30:12.655 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/199/13' 2023-10-30T13:30:12.656 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/199/17' 2023-10-30T13:30:12.656 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/199' 2023-10-30T13:30:12.656 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/163/19' 2023-10-30T13:30:12.656 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/163/15' 2023-10-30T13:30:12.656 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/163/20' 2023-10-30T13:30:12.657 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/163/12' 2023-10-30T13:30:12.657 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/163/10' 2023-10-30T13:30:12.657 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/163/11' 2023-10-30T13:30:12.657 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/163/16' 2023-10-30T13:30:12.657 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/163/14' 2023-10-30T13:30:12.657 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/163/18' 2023-10-30T13:30:12.658 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/163/13' 2023-10-30T13:30:12.658 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/163/17' 2023-10-30T13:30:12.658 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/163' 2023-10-30T13:30:12.658 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/153/19' 2023-10-30T13:30:12.658 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/153/15' 2023-10-30T13:30:12.659 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/153/20' 2023-10-30T13:30:12.659 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/153/12' 2023-10-30T13:30:12.659 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/153/10' 2023-10-30T13:30:12.659 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/153/11' 2023-10-30T13:30:12.659 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/153/16' 2023-10-30T13:30:12.660 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/153/14' 2023-10-30T13:30:12.660 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/153/18' 2023-10-30T13:30:12.660 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/153/13' 2023-10-30T13:30:12.660 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/153/17' 2023-10-30T13:30:12.660 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/153' 2023-10-30T13:30:12.661 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/122/19' 2023-10-30T13:30:12.661 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/122/15' 2023-10-30T13:30:12.661 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/122/20' 2023-10-30T13:30:12.661 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/122/12' 2023-10-30T13:30:12.661 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/122/10' 2023-10-30T13:30:12.662 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/122/11' 2023-10-30T13:30:12.662 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/122/16' 2023-10-30T13:30:12.662 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/122/14' 2023-10-30T13:30:12.662 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/122/18' 2023-10-30T13:30:12.662 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/122/13' 2023-10-30T13:30:12.662 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/122/17' 2023-10-30T13:30:12.663 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/122' 2023-10-30T13:30:12.663 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/162/19' 2023-10-30T13:30:12.663 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/162/15' 2023-10-30T13:30:12.663 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/162/20' 2023-10-30T13:30:12.663 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/162/12' 2023-10-30T13:30:12.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/162/10' 2023-10-30T13:30:12.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/162/11' 2023-10-30T13:30:12.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/162/16' 2023-10-30T13:30:12.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/162/14' 2023-10-30T13:30:12.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/162/18' 2023-10-30T13:30:12.664 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/162/13' 2023-10-30T13:30:12.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/162/17' 2023-10-30T13:30:12.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/162' 2023-10-30T13:30:12.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/197/19' 2023-10-30T13:30:12.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/197/15' 2023-10-30T13:30:12.665 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/197/20' 2023-10-30T13:30:12.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/197/12' 2023-10-30T13:30:12.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/197/10' 2023-10-30T13:30:12.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/197/11' 2023-10-30T13:30:12.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/197/16' 2023-10-30T13:30:12.666 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/197/14' 2023-10-30T13:30:12.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/197/18' 2023-10-30T13:30:12.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/197/13' 2023-10-30T13:30:12.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/197/17' 2023-10-30T13:30:12.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/197' 2023-10-30T13:30:12.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/113/19' 2023-10-30T13:30:12.667 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/113/15' 2023-10-30T13:30:12.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/113/20' 2023-10-30T13:30:12.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/113/12' 2023-10-30T13:30:12.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/113/10' 2023-10-30T13:30:12.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/113/11' 2023-10-30T13:30:12.668 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/113/16' 2023-10-30T13:30:12.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/113/14' 2023-10-30T13:30:12.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/113/18' 2023-10-30T13:30:12.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/113/13' 2023-10-30T13:30:12.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/113/17' 2023-10-30T13:30:12.669 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/113' 2023-10-30T13:30:12.670 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/152/19' 2023-10-30T13:30:15.191 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/152/15' 2023-10-30T13:30:15.191 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/152/20' 2023-10-30T13:30:15.191 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/152/12' 2023-10-30T13:30:15.192 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/152/10' 2023-10-30T13:30:15.192 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/152/11' 2023-10-30T13:30:15.192 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/152/16' 2023-10-30T13:30:15.193 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/152/14' 2023-10-30T13:30:15.193 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/152/18' 2023-10-30T13:30:15.193 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/152/13' 2023-10-30T13:30:15.193 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/152/17' 2023-10-30T13:30:15.193 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/152' 2023-10-30T13:30:15.194 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/121/19' 2023-10-30T13:30:15.194 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/121/15' 2023-10-30T13:30:15.194 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/121/20' 2023-10-30T13:30:15.194 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/121/12' 2023-10-30T13:30:15.194 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/121/10' 2023-10-30T13:30:15.195 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/121/11' 2023-10-30T13:30:15.195 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/121/16' 2023-10-30T13:30:15.195 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/121/14' 2023-10-30T13:30:15.195 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/121/18' 2023-10-30T13:30:15.195 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/121/13' 2023-10-30T13:30:15.195 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/121/17' 2023-10-30T13:30:15.196 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/121' 2023-10-30T13:30:15.196 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/118/19' 2023-10-30T13:30:15.196 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/118/15' 2023-10-30T13:30:15.196 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/118/20' 2023-10-30T13:30:15.197 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/118/12' 2023-10-30T13:30:15.197 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/118/10' 2023-10-30T13:30:15.197 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/118/11' 2023-10-30T13:30:15.197 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/118/16' 2023-10-30T13:30:15.197 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/118/14' 2023-10-30T13:30:15.198 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/118/18' 2023-10-30T13:30:15.198 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/118/13' 2023-10-30T13:30:15.198 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/118/17' 2023-10-30T13:30:15.198 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/118' 2023-10-30T13:30:15.198 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/196/19' 2023-10-30T13:30:15.199 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/196/15' 2023-10-30T13:30:15.199 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/196/20' 2023-10-30T13:30:15.199 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/196/12' 2023-10-30T13:30:15.199 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/196/10' 2023-10-30T13:30:15.199 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/196/11' 2023-10-30T13:30:15.200 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/196/16' 2023-10-30T13:30:15.200 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/196/14' 2023-10-30T13:30:15.200 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/196/18' 2023-10-30T13:30:15.200 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/196/13' 2023-10-30T13:30:15.200 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/196/17' 2023-10-30T13:30:15.200 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/196' 2023-10-30T13:30:15.201 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/124/19' 2023-10-30T13:30:15.201 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/124/15' 2023-10-30T13:30:15.201 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/124/20' 2023-10-30T13:30:15.201 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/124/12' 2023-10-30T13:30:15.201 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/124/10' 2023-10-30T13:30:15.201 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/124/11' 2023-10-30T13:30:15.202 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/124/16' 2023-10-30T13:30:15.202 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/124/14' 2023-10-30T13:30:15.202 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/124/18' 2023-10-30T13:30:15.202 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/124/13' 2023-10-30T13:30:15.203 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/124/17' 2023-10-30T13:30:15.203 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/124' 2023-10-30T13:30:15.203 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/188/19' 2023-10-30T13:30:15.203 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/188/15' 2023-10-30T13:30:15.203 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/188/20' 2023-10-30T13:30:15.203 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/188/12' 2023-10-30T13:30:15.204 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/188/10' 2023-10-30T13:30:15.204 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/188/11' 2023-10-30T13:30:15.204 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/188/16' 2023-10-30T13:30:15.204 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/188/14' 2023-10-30T13:30:15.204 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/188/18' 2023-10-30T13:30:15.205 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/188/13' 2023-10-30T13:30:15.205 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/188/17' 2023-10-30T13:30:15.205 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/188' 2023-10-30T13:30:15.205 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/145/19' 2023-10-30T13:30:15.205 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/145/15' 2023-10-30T13:30:15.205 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/145/20' 2023-10-30T13:30:15.206 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/145/12' 2023-10-30T13:30:15.206 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/145/10' 2023-10-30T13:30:15.206 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/145/11' 2023-10-30T13:30:15.206 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/145/16' 2023-10-30T13:30:15.206 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/145/14' 2023-10-30T13:30:15.207 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/145/18' 2023-10-30T13:30:15.207 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/145/13' 2023-10-30T13:30:15.207 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/145/17' 2023-10-30T13:30:15.207 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/145' 2023-10-30T13:30:15.208 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/130/19' 2023-10-30T13:30:15.208 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/130/15' 2023-10-30T13:30:15.208 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/130/20' 2023-10-30T13:30:15.208 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/130/12' 2023-10-30T13:30:15.208 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/130/10' 2023-10-30T13:30:15.208 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/130/11' 2023-10-30T13:30:15.209 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/130/16' 2023-10-30T13:30:15.209 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/130/14' 2023-10-30T13:30:15.209 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/130/18' 2023-10-30T13:30:15.209 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/130/13' 2023-10-30T13:30:15.209 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/130/17' 2023-10-30T13:30:15.210 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/130' 2023-10-30T13:30:15.210 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/193/19' 2023-10-30T13:30:15.210 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/193/15' 2023-10-30T13:30:15.210 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/193/20' 2023-10-30T13:30:15.210 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/193/12' 2023-10-30T13:30:15.211 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/193/10' 2023-10-30T13:30:15.211 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/193/11' 2023-10-30T13:30:15.211 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/193/16' 2023-10-30T13:30:15.211 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/193/14' 2023-10-30T13:30:15.212 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/193/18' 2023-10-30T13:30:15.212 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/193/13' 2023-10-30T13:30:15.212 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/193/17' 2023-10-30T13:30:15.212 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/193' 2023-10-30T13:30:15.213 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/175/19' 2023-10-30T13:30:15.213 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/175/15' 2023-10-30T13:30:15.541 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:15.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.680+0000 7ff6a3609640 1 Processor -- start 2023-10-30T13:30:15.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.680+0000 7ff6a3609640 1 -- start start 2023-10-30T13:30:15.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.680+0000 7ff6a3609640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69c0c6fe0 0x7ff69c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:15.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.680+0000 7ff6a3609640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff69c0c5f60 0x7ff69c0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:15.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.680+0000 7ff6a3609640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff69c0700b0 con 0x7ff69c153bf0 2023-10-30T13:30:15.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.680+0000 7ff6a3609640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69c0c6810 con 0x7ff69c0c5f60 2023-10-30T13:30:15.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.680+0000 7ff6a3609640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c159900 con 0x7ff69c0c6fe0 2023-10-30T13:30:15.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.680+0000 7ff6a1ba6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff69c0c5f60 0x7ff69c0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:15.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.680+0000 7ff6a1ba6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff69c0c5f60 0x7ff69c0cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44606/0 (socket says 172.21.15.46:44606) 2023-10-30T13:30:15.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.684+0000 7ff6a1ba6640 1 -- 172.21.15.46:0/1266165821 learned_addr learned my addr 172.21.15.46:0/1266165821 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:15.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.684+0000 7ff6a0ba4640 1 --2- 172.21.15.46:0/1266165821 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69c0c6fe0 0x7ff69c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:15.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.684+0000 7ff6a1ba6640 1 -- 172.21.15.46:0/1266165821 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69c0c6fe0 msgr2=0x7ff69c071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:15.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.684+0000 7ff6a1ba6640 1 --2- 172.21.15.46:0/1266165821 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69c0c6fe0 0x7ff69c071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:15.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.684+0000 7ff6a1ba6640 1 -- 172.21.15.46:0/1266165821 >> v1:172.21.15.154:6789/0 conn(0x7ff69c153bf0 legacy=0x7ff69c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:15.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.684+0000 7ff6a1ba6640 1 -- 172.21.15.46:0/1266165821 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c0c6530 con 0x7ff69c0c5f60 2023-10-30T13:30:15.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.684+0000 7ff6a1ba6640 1 --2- 172.21.15.46:0/1266165821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69c0c5f60 0x7ff69c0cb570 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff694009e50 tx=0x7ff69401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9cbbf5077cf5122 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:15.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.684+0000 7ff693fff640 1 -- 172.21.15.46:0/1266165821 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6940af400 con 0x7ff69c0c5f60 2023-10-30T13:30:15.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.684+0000 7ff693fff640 1 -- 172.21.15.46:0/1266165821 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6940ab050 con 0x7ff69c0c5f60 2023-10-30T13:30:15.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.684+0000 7ff693fff640 1 -- 172.21.15.46:0/1266165821 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6940af780 con 0x7ff69c0c5f60 2023-10-30T13:30:15.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 -- 172.21.15.46:0/1266165821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69c0c5f60 msgr2=0x7ff69c0cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:15.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 --2- 172.21.15.46:0/1266165821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69c0c5f60 0x7ff69c0cb570 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff694009e50 tx=0x7ff69401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:30:15.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 -- 172.21.15.46:0/1266165821 shutdown_connections 2023-10-30T13:30:15.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 --2- 172.21.15.46:0/1266165821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69c0c5f60 0x7ff69c0cb570 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:15.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 --2- 172.21.15.46:0/1266165821 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69c0c6fe0 0x7ff69c071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:15.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 -- 172.21.15.46:0/1266165821 >> 172.21.15.46:0/1266165821 conn(0x7ff69c06c6e0 msgr2=0x7ff69c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:15.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 -- 172.21.15.46:0/1266165821 shutdown_connections 2023-10-30T13:30:15.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 -- 172.21.15.46:0/1266165821 wait complete. 2023-10-30T13:30:15.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 Processor -- start 2023-10-30T13:30:15.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 -- start start 2023-10-30T13:30:15.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69c0c5f60 0x7ff69c152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:15.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69c0c6fe0 0x7ff69c143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:15.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69c153bf0 0x7ff69c143a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:15.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c143f80 con 0x7ff69c0c5f60 2023-10-30T13:30:15.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c1485a0 con 0x7ff69c153bf0 2023-10-30T13:30:15.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a3609640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c14dea0 con 0x7ff69c0c6fe0 2023-10-30T13:30:15.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a13a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69c0c5f60 0x7ff69c152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:15.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a13a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69c0c5f60 0x7ff69c152a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44616/0 (socket says 172.21.15.46:44616) 2023-10-30T13:30:15.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a13a5640 1 -- 172.21.15.46:0/3902805057 learned_addr learned my addr 172.21.15.46:0/3902805057 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:15.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a0ba4640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69c0c6fe0 0x7ff69c143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:15.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.688+0000 7ff6a1ba6640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69c153bf0 0x7ff69c143a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:15.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff6a13a5640 1 -- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69c0c6fe0 msgr2=0x7ff69c143500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:15.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff6a13a5640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69c0c6fe0 0x7ff69c143500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:15.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff6a13a5640 1 -- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69c153bf0 msgr2=0x7ff69c143a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:15.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff6a13a5640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69c153bf0 0x7ff69c143a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:15.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff6a13a5640 1 -- 172.21.15.46:0/3902805057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff694009b30 con 0x7ff69c0c5f60 2023-10-30T13:30:15.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff6a13a5640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69c0c5f60 0x7ff69c152a60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff69800c3c0 tx=0x7ff69800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:15.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff691ffb640 1 -- 172.21.15.46:0/3902805057 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6980052d0 con 0x7ff69c0c5f60 2023-10-30T13:30:15.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff691ffb640 1 -- 172.21.15.46:0/3902805057 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff698005430 con 0x7ff69c0c5f60 2023-10-30T13:30:15.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff6a3609640 1 -- 172.21.15.46:0/3902805057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69c14f990 con 0x7ff69c0c5f60 2023-10-30T13:30:15.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff691ffb640 1 -- 172.21.15.46:0/3902805057 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698003840 con 0x7ff69c0c5f60 2023-10-30T13:30:15.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff6a3609640 1 -- 172.21.15.46:0/3902805057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69c14e0f0 con 0x7ff69c0c5f60 2023-10-30T13:30:15.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff6a3609640 1 -- 172.21.15.46:0/3902805057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69c04e170 con 0x7ff69c0c5f60 2023-10-30T13:30:15.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff691ffb640 1 -- 172.21.15.46:0/3902805057 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff6980039a0 con 0x7ff69c0c5f60 2023-10-30T13:30:15.705 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:30:15.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.692+0000 7ff691ffb640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6800664a0 0x7ff680068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:15.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.696+0000 7ff691ffb640 1 -- 172.21.15.46:0/3902805057 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7ff6980834d0 con 0x7ff69c0c5f60 2023-10-30T13:30:15.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.696+0000 7ff6a0ba4640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6800664a0 0x7ff680068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:15.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.696+0000 7ff6a0ba4640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6800664a0 0x7ff680068960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff69c153530 tx=0x7ff68c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:15.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.700+0000 7ff691ffb640 1 -- 172.21.15.46:0/3902805057 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6980507f0 con 0x7ff69c0c5f60 2023-10-30T13:30:15.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:30:15.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.892+0000 7ff6a3609640 1 -- 172.21.15.46:0/3902805057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff69c06fe90 con 0x7ff69c0c5f60 2023-10-30T13:30:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.892+0000 7ff691ffb640 1 -- 172.21.15.46:0/3902805057 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6530 (secure 0 0 0) 0x7ff698053e10 con 0x7ff69c0c5f60 2023-10-30T13:30:15.900 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:15.901 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:52.914924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:15.902 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 37 2023-10-30T13:30:15.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.896+0000 7ff6a3609640 1 -- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6800664a0 msgr2=0x7ff680068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:15.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.896+0000 7ff6a3609640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6800664a0 0x7ff680068960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff69c153530 tx=0x7ff68c008040 comp rx=0 tx=0).stop 2023-10-30T13:30:15.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.896+0000 7ff6a3609640 1 -- 172.21.15.46:0/3902805057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69c0c5f60 msgr2=0x7ff69c152a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:15.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.896+0000 7ff6a3609640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69c0c5f60 0x7ff69c152a60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff69800c3c0 tx=0x7ff69800c880 comp rx=0 tx=0).stop 2023-10-30T13:30:15.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.896+0000 7ff6a3609640 1 -- 172.21.15.46:0/3902805057 shutdown_connections 2023-10-30T13:30:15.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.896+0000 7ff6a3609640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6800664a0 0x7ff680068960 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:15.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.896+0000 7ff6a3609640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69c153bf0 0x7ff69c143a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:15.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.896+0000 7ff6a3609640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69c0c6fe0 0x7ff69c143500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:15.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.896+0000 7ff6a3609640 1 --2- 172.21.15.46:0/3902805057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff69c0c5f60 0x7ff69c152a60 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:15.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.896+0000 7ff6a3609640 1 -- 172.21.15.46:0/3902805057 >> 172.21.15.46:0/3902805057 conn(0x7ff69c06c6e0 msgr2=0x7ff69c070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:15.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.896+0000 7ff6a3609640 1 -- 172.21.15.46:0/3902805057 shutdown_connections 2023-10-30T13:30:15.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:15.896+0000 7ff6a3609640 1 -- 172.21.15.46:0/3902805057 wait complete. 2023-10-30T13:30:15.921 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T13:30:15.921 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:30:15.921 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:30:15.939 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:30:15.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:16.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.116+0000 7fe4797d3640 1 Processor -- start 2023-10-30T13:30:16.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.116+0000 7fe4797d3640 1 -- start start 2023-10-30T13:30:16.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe4797d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4740c6bd0 0x7fe47406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:16.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe4797d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe474071110 0x7fe474158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:16.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe4797d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe474066cf0 con 0x7fe47406f7a0 2023-10-30T13:30:16.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe4797d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4740700c0 con 0x7fe4740c6bd0 2023-10-30T13:30:16.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe4797d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4740c70d0 con 0x7fe474071110 2023-10-30T13:30:16.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe46bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4740c6bd0 0x7fe47406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:16.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe46bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4740c6bd0 0x7fe47406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44622/0 (socket says 172.21.15.46:44622) 2023-10-30T13:30:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe46bfff640 1 -- 172.21.15.46:0/1670710530 learned_addr learned my addr 172.21.15.46:0/1670710530 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe472ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe47406f7a0 0x7fe474069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54690/0 (socket says 172.21.15.46:54690) 2023-10-30T13:30:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe4737fe640 1 --2- 172.21.15.46:0/1670710530 >> v2:172.21.15.154:3300/0 conn(0x7fe474071110 0x7fe474158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe46bfff640 1 -- 172.21.15.46:0/1670710530 >> v2:172.21.15.154:3300/0 conn(0x7fe474071110 msgr2=0x7fe474158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe46bfff640 1 --2- 172.21.15.46:0/1670710530 >> v2:172.21.15.154:3300/0 conn(0x7fe474071110 0x7fe474158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:16.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe46bfff640 1 -- 172.21.15.46:0/1670710530 >> v1:172.21.15.46:6789/0 conn(0x7fe47406f7a0 legacy=0x7fe474069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:16.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.120+0000 7fe46bfff640 1 -- 172.21.15.46:0/1670710530 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4740c7390 con 0x7fe4740c6bd0 2023-10-30T13:30:16.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe46bfff640 1 --2- 172.21.15.46:0/1670710530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4740c6bd0 0x7fe47406fb80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe47406cdd0 tx=0x7fe45c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5386de4e28e477 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:16.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4727fc640 1 -- 172.21.15.46:0/1670710530 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c0a89f0 con 0x7fe4740c6bd0 2023-10-30T13:30:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4727fc640 1 -- 172.21.15.46:0/1670710530 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45c0a8b50 con 0x7fe4740c6bd0 2023-10-30T13:30:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4727fc640 1 -- 172.21.15.46:0/1670710530 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c0b0440 con 0x7fe4740c6bd0 2023-10-30T13:30:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4797d3640 1 -- 172.21.15.46:0/1670710530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4740c6bd0 msgr2=0x7fe47406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4797d3640 1 --2- 172.21.15.46:0/1670710530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4740c6bd0 0x7fe47406fb80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe47406cdd0 tx=0x7fe45c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:30:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4797d3640 1 -- 172.21.15.46:0/1670710530 shutdown_connections 2023-10-30T13:30:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4797d3640 1 --2- 172.21.15.46:0/1670710530 >> v2:172.21.15.154:3300/0 conn(0x7fe474071110 0x7fe474158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4797d3640 1 --2- 172.21.15.46:0/1670710530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4740c6bd0 0x7fe47406fb80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4797d3640 1 -- 172.21.15.46:0/1670710530 >> 172.21.15.46:0/1670710530 conn(0x7fe4740c7910 msgr2=0x7fe4740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4797d3640 1 -- 172.21.15.46:0/1670710530 shutdown_connections 2023-10-30T13:30:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4797d3640 1 -- 172.21.15.46:0/1670710530 wait complete. 2023-10-30T13:30:16.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4797d3640 1 Processor -- start 2023-10-30T13:30:16.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4797d3640 1 -- start start 2023-10-30T13:30:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4797d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47406f7a0 0x7fe4741567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.124+0000 7fe4797d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe474071110 0x7fe4741eef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe4797d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4740c6bd0 0x7fe4740c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe4797d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4740c0b50 con 0x7fe47406f7a0 2023-10-30T13:30:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe4797d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4740c0590 con 0x7fe4740c6bd0 2023-10-30T13:30:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe4797d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4740c0870 con 0x7fe474071110 2023-10-30T13:30:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe472ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47406f7a0 0x7fe4741567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe472ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47406f7a0 0x7fe4741567e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44632/0 (socket says 172.21.15.46:44632) 2023-10-30T13:30:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe472ffd640 1 -- 172.21.15.46:0/1630437255 learned_addr learned my addr 172.21.15.46:0/1630437255 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe46bfff640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe474071110 0x7fe4741eef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:16.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe4737fe640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4740c6bd0 0x7fe4740c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:16.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe472ffd640 1 -- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe474071110 msgr2=0x7fe4741eef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:16.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe472ffd640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe474071110 0x7fe4741eef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:16.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe472ffd640 1 -- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4740c6bd0 msgr2=0x7fe4740c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:16.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe472ffd640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4740c6bd0 0x7fe4740c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe472ffd640 1 -- 172.21.15.46:0/1630437255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c0094d0 con 0x7fe47406f7a0 2023-10-30T13:30:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe472ffd640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47406f7a0 0x7fe4741567e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe46400c430 tx=0x7fe46400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe46b7fe640 1 -- 172.21.15.46:0/1630437255 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4640074e0 con 0x7fe47406f7a0 2023-10-30T13:30:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe46b7fe640 1 -- 172.21.15.46:0/1630437255 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe464007640 con 0x7fe47406f7a0 2023-10-30T13:30:16.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe46b7fe640 1 -- 172.21.15.46:0/1630437255 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464004a10 con 0x7fe47406f7a0 2023-10-30T13:30:16.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe4797d3640 1 -- 172.21.15.46:0/1630437255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4741f25e0 con 0x7fe47406f7a0 2023-10-30T13:30:16.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe4797d3640 1 -- 172.21.15.46:0/1630437255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4741f2ba0 con 0x7fe47406f7a0 2023-10-30T13:30:16.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe4797d3640 1 -- 172.21.15.46:0/1630437255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe47404e170 con 0x7fe47406f7a0 2023-10-30T13:30:16.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe46b7fe640 1 -- 172.21.15.46:0/1630437255 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe464004b70 con 0x7fe47406f7a0 2023-10-30T13:30:16.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.128+0000 7fe46b7fe640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4540664d0 0x7fe454068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:16.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.132+0000 7fe46b7fe640 1 -- 172.21.15.46:0/1630437255 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fe4640839c0 con 0x7fe47406f7a0 2023-10-30T13:30:16.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.132+0000 7fe46bfff640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4540664d0 0x7fe454068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:16.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.132+0000 7fe46bfff640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4540664d0 0x7fe454068990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0b2000 tx=0x7fe45c0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:16.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.132+0000 7fe46b7fe640 1 -- 172.21.15.46:0/1630437255 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe464050c30 con 0x7fe47406f7a0 2023-10-30T13:30:16.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.324+0000 7fe4797d3640 1 -- 172.21.15.46:0/1630437255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe47406a7f0 con 0x7fe47406f7a0 2023-10-30T13:30:16.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.324+0000 7fe46b7fe640 1 -- 172.21.15.46:0/1630437255 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fe464054250 con 0x7fe47406f7a0 2023-10-30T13:30:16.332 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:16.332 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:52.914924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 37 2023-10-30T13:30:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.324+0000 7fe4797d3640 1 -- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4540664d0 msgr2=0x7fe454068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.324+0000 7fe4797d3640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4540664d0 0x7fe454068990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0b2000 tx=0x7fe45c0aa070 comp rx=0 tx=0).stop 2023-10-30T13:30:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.324+0000 7fe4797d3640 1 -- 172.21.15.46:0/1630437255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47406f7a0 msgr2=0x7fe4741567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.324+0000 7fe4797d3640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47406f7a0 0x7fe4741567e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe46400c430 tx=0x7fe46400c8f0 comp rx=0 tx=0).stop 2023-10-30T13:30:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.328+0000 7fe4797d3640 1 -- 172.21.15.46:0/1630437255 shutdown_connections 2023-10-30T13:30:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.328+0000 7fe4797d3640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4540664d0 0x7fe454068990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.328+0000 7fe4797d3640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4740c6bd0 0x7fe4740c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.328+0000 7fe4797d3640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe474071110 0x7fe4741eef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.328+0000 7fe4797d3640 1 --2- 172.21.15.46:0/1630437255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47406f7a0 0x7fe4741567e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.328+0000 7fe4797d3640 1 -- 172.21.15.46:0/1630437255 >> 172.21.15.46:0/1630437255 conn(0x7fe4740c7910 msgr2=0x7fe4740c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.328+0000 7fe4797d3640 1 -- 172.21.15.46:0/1630437255 shutdown_connections 2023-10-30T13:30:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:16.328+0000 7fe4797d3640 1 -- 172.21.15.46:0/1630437255 wait complete. 2023-10-30T13:30:16.353 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:30:18.354 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:18.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.500+0000 7fd427879640 1 Processor -- start 2023-10-30T13:30:18.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd427879640 1 -- start start 2023-10-30T13:30:18.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd427879640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4200bfec0 0x7fd42006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:18.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd427879640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd420069e90 0x7fd42006a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:18.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd427879640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42006fa80 con 0x7fd420159e60 2023-10-30T13:30:18.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd427879640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd420071290 con 0x7fd420069e90 2023-10-30T13:30:18.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd427879640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd420071570 con 0x7fd4200bfec0 2023-10-30T13:30:18.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd425e16640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd420069e90 0x7fd42006a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:18.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd425e16640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd420069e90 0x7fd42006a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55096/0 (socket says 172.21.15.46:55096) 2023-10-30T13:30:18.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd425e16640 1 -- 172.21.15.46:0/3598801570 learned_addr learned my addr 172.21.15.46:0/3598801570 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:18.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd425615640 1 --2- 172.21.15.46:0/3598801570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4200bfec0 0x7fd42006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:18.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd425e16640 1 -- 172.21.15.46:0/3598801570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4200bfec0 msgr2=0x7fd42006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:18.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd425e16640 1 --2- 172.21.15.46:0/3598801570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4200bfec0 0x7fd42006f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:18.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd425e16640 1 -- 172.21.15.46:0/3598801570 >> v1:172.21.15.154:6789/0 conn(0x7fd420159e60 legacy=0x7fd4200c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:18.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.504+0000 7fd425e16640 1 -- 172.21.15.46:0/3598801570 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd420071850 con 0x7fd420069e90 2023-10-30T13:30:18.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd425e16640 1 --2- 172.21.15.46:0/3598801570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd420069e90 0x7fd42006a320 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd41800d150 tx=0x7fd418021400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866c001e9f49f9be server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:18.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd417fff640 1 -- 172.21.15.46:0/3598801570 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41800b2d0 con 0x7fd420069e90 2023-10-30T13:30:18.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd417fff640 1 -- 172.21.15.46:0/3598801570 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4180ad050 con 0x7fd420069e90 2023-10-30T13:30:18.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd417fff640 1 -- 172.21.15.46:0/3598801570 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd418009820 con 0x7fd420069e90 2023-10-30T13:30:18.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd427879640 1 -- 172.21.15.46:0/3598801570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd420069e90 msgr2=0x7fd42006a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd427879640 1 --2- 172.21.15.46:0/3598801570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd420069e90 0x7fd42006a320 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd41800d150 tx=0x7fd418021400 comp rx=0 tx=0).stop 2023-10-30T13:30:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd427879640 1 -- 172.21.15.46:0/3598801570 shutdown_connections 2023-10-30T13:30:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd427879640 1 --2- 172.21.15.46:0/3598801570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd420069e90 0x7fd42006a320 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd427879640 1 --2- 172.21.15.46:0/3598801570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4200bfec0 0x7fd42006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd427879640 1 -- 172.21.15.46:0/3598801570 >> 172.21.15.46:0/3598801570 conn(0x7fd4200c7910 msgr2=0x7fd4200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd427879640 1 -- 172.21.15.46:0/3598801570 shutdown_connections 2023-10-30T13:30:18.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd427879640 1 -- 172.21.15.46:0/3598801570 wait complete. 2023-10-30T13:30:18.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd427879640 1 Processor -- start 2023-10-30T13:30:18.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.508+0000 7fd427879640 1 -- start start 2023-10-30T13:30:18.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd427879640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd420069e90 0x7fd4201e03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:18.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd427879640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4200bfec0 0x7fd4201e0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:18.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd427879640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd420159e60 0x7fd42014d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd427879640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4201ef850 con 0x7fd420069e90 2023-10-30T13:30:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd427879640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd420070ed0 con 0x7fd420159e60 2023-10-30T13:30:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd427879640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4201eec60 con 0x7fd4200bfec0 2023-10-30T13:30:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd425615640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd420069e90 0x7fd4201e03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd425615640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd420069e90 0x7fd4201e03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55110/0 (socket says 172.21.15.46:55110) 2023-10-30T13:30:18.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd425615640 1 -- 172.21.15.46:0/248689254 learned_addr learned my addr 172.21.15.46:0/248689254 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:18.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd424e14640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4200bfec0 0x7fd4201e0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:18.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd425e16640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd420159e60 0x7fd42014d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:18.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd425615640 1 -- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4200bfec0 msgr2=0x7fd4201e0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:18.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd425615640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4200bfec0 0x7fd4201e0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:18.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd425615640 1 -- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd420159e60 msgr2=0x7fd42014d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:18.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd425615640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd420159e60 0x7fd42014d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:18.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd425615640 1 -- 172.21.15.46:0/248689254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41800bd40 con 0x7fd420069e90 2023-10-30T13:30:18.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd425615640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd420069e90 0x7fd4201e03d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00d800 tx=0x7fd41c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:18.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd415ffb640 1 -- 172.21.15.46:0/248689254 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41c00b6c0 con 0x7fd420069e90 2023-10-30T13:30:18.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd415ffb640 1 -- 172.21.15.46:0/248689254 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41c00b860 con 0x7fd420069e90 2023-10-30T13:30:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd427879640 1 -- 172.21.15.46:0/248689254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4200c66d0 con 0x7fd420069e90 2023-10-30T13:30:18.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd415ffb640 1 -- 172.21.15.46:0/248689254 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41c00bc10 con 0x7fd420069e90 2023-10-30T13:30:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd427879640 1 -- 172.21.15.46:0/248689254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4201ef200 con 0x7fd420069e90 2023-10-30T13:30:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.512+0000 7fd427879640 1 -- 172.21.15.46:0/248689254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd42004e170 con 0x7fd420069e90 2023-10-30T13:30:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.516+0000 7fd415ffb640 1 -- 172.21.15.46:0/248689254 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd41c01e460 con 0x7fd420069e90 2023-10-30T13:30:18.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.516+0000 7fd415ffb640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd4040664d0 0x7fd404068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:18.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.516+0000 7fd415ffb640 1 -- 172.21.15.46:0/248689254 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fd41c084590 con 0x7fd420069e90 2023-10-30T13:30:18.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.516+0000 7fd424e14640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd4040664d0 0x7fd404068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:18.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.516+0000 7fd424e14640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd4040664d0 0x7fd404068990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd4200c94e0 tx=0x7fd410009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:18.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.520+0000 7fd415ffb640 1 -- 172.21.15.46:0/248689254 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd41c051880 con 0x7fd420069e90 2023-10-30T13:30:18.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.708+0000 7fd427879640 1 -- 172.21.15.46:0/248689254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd420070250 con 0x7fd420069e90 2023-10-30T13:30:18.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.708+0000 7fd415ffb640 1 -- 172.21.15.46:0/248689254 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fd41c054ea0 con 0x7fd420069e90 2023-10-30T13:30:18.717 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:18.717 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:52.914924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:18.718 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 37 2023-10-30T13:30:18.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.712+0000 7fd427879640 1 -- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd4040664d0 msgr2=0x7fd404068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:18.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.712+0000 7fd427879640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd4040664d0 0x7fd404068990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd4200c94e0 tx=0x7fd410009040 comp rx=0 tx=0).stop 2023-10-30T13:30:18.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.712+0000 7fd427879640 1 -- 172.21.15.46:0/248689254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd420069e90 msgr2=0x7fd4201e03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:18.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.712+0000 7fd427879640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd420069e90 0x7fd4201e03d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00d800 tx=0x7fd41c00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:30:18.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.712+0000 7fd427879640 1 -- 172.21.15.46:0/248689254 shutdown_connections 2023-10-30T13:30:18.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.712+0000 7fd427879640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd4040664d0 0x7fd404068990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:18.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.712+0000 7fd427879640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd420159e60 0x7fd42014d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:18.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.712+0000 7fd427879640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4200bfec0 0x7fd4201e0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:18.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.712+0000 7fd427879640 1 --2- 172.21.15.46:0/248689254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd420069e90 0x7fd4201e03d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:18.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.712+0000 7fd427879640 1 -- 172.21.15.46:0/248689254 >> 172.21.15.46:0/248689254 conn(0x7fd4200c7910 msgr2=0x7fd4200c9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:18.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.712+0000 7fd427879640 1 -- 172.21.15.46:0/248689254 shutdown_connections 2023-10-30T13:30:18.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:18.712+0000 7fd427879640 1 -- 172.21.15.46:0/248689254 wait complete. 2023-10-30T13:30:18.738 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:30:20.739 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:20.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.884+0000 7f011a274640 1 Processor -- start 2023-10-30T13:30:20.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.884+0000 7f011a274640 1 -- start start 2023-10-30T13:30:20.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f011a274640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0114070450 0x7f01140c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f011a274640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f011406ded0 0x7f011406e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f011a274640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0114071fe0 con 0x7f01141539a0 2023-10-30T13:30:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f011a274640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01140722c0 con 0x7f011406ded0 2023-10-30T13:30:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f011a274640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011415cb00 con 0x7f0114070450 2023-10-30T13:30:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f0119a73640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f011406ded0 0x7f011406e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f0119272640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f01141539a0 0x7f01140717c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45764/0 (socket says 172.21.15.46:45764) 2023-10-30T13:30:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f0119272640 1 -- 172.21.15.46:0/2928527298 learned_addr learned my addr 172.21.15.46:0/2928527298 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:30:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f0118a71640 1 --2- 172.21.15.46:0/2928527298 >> v2:172.21.15.154:3300/0 conn(0x7f0114070450 0x7f01140c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f0119a73640 1 -- 172.21.15.46:0/2928527298 >> v2:172.21.15.154:3300/0 conn(0x7f0114070450 msgr2=0x7f01140c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f0119a73640 1 --2- 172.21.15.46:0/2928527298 >> v2:172.21.15.154:3300/0 conn(0x7f0114070450 0x7f01140c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:20.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f0119a73640 1 -- 172.21.15.46:0/2928527298 >> v1:172.21.15.46:6789/0 conn(0x7f01141539a0 legacy=0x7f01140717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:20.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.888+0000 7f0119a73640 1 -- 172.21.15.46:0/2928527298 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011415cde0 con 0x7f011406ded0 2023-10-30T13:30:20.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f0119a73640 1 --2- 172.21.15.46:0/2928527298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f011406ded0 0x7f011406e300 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f011406c2b0 tx=0x7f010c01c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af268dfb5bf092c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:20.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f010bfff640 1 -- 172.21.15.46:0/2928527298 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c0a5850 con 0x7f011406ded0 2023-10-30T13:30:20.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f010bfff640 1 -- 172.21.15.46:0/2928527298 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c0a59b0 con 0x7f011406ded0 2023-10-30T13:30:20.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f010bfff640 1 -- 172.21.15.46:0/2928527298 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c0a5d40 con 0x7f011406ded0 2023-10-30T13:30:20.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f011a274640 1 -- 172.21.15.46:0/2928527298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f011406ded0 msgr2=0x7f011406e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:20.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f011a274640 1 --2- 172.21.15.46:0/2928527298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f011406ded0 0x7f011406e300 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f011406c2b0 tx=0x7f010c01c880 comp rx=0 tx=0).stop 2023-10-30T13:30:20.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f011a274640 1 -- 172.21.15.46:0/2928527298 shutdown_connections 2023-10-30T13:30:20.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f011a274640 1 --2- 172.21.15.46:0/2928527298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f011406ded0 0x7f011406e300 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:20.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f011a274640 1 --2- 172.21.15.46:0/2928527298 >> v2:172.21.15.154:3300/0 conn(0x7f0114070450 0x7f01140c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:20.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f011a274640 1 -- 172.21.15.46:0/2928527298 >> 172.21.15.46:0/2928527298 conn(0x7f01140cbe40 msgr2=0x7f01140cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:20.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f011a274640 1 -- 172.21.15.46:0/2928527298 shutdown_connections 2023-10-30T13:30:20.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f011a274640 1 -- 172.21.15.46:0/2928527298 wait complete. 2023-10-30T13:30:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f011a274640 1 Processor -- start 2023-10-30T13:30:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.892+0000 7f011a274640 1 -- start start 2023-10-30T13:30:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f011a274640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f011406ded0 0x7f01140c3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f011a274640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0114070450 0x7f0114144040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f011a274640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01141539a0 0x7f011414e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f011a274640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011414da80 con 0x7f0114070450 2023-10-30T13:30:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f011a274640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c3910 con 0x7f01141539a0 2023-10-30T13:30:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f011a274640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01141f31d0 con 0x7f011406ded0 2023-10-30T13:30:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0118a71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0114070450 0x7f0114144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0118a71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0114070450 0x7f0114144040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55128/0 (socket says 172.21.15.46:55128) 2023-10-30T13:30:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0118a71640 1 -- 172.21.15.46:0/2685603424 learned_addr learned my addr 172.21.15.46:0/2685603424 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0119272640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f011406ded0 0x7f01140c3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0119a73640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01141539a0 0x7f011414e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:20.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0118a71640 1 -- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f011406ded0 msgr2=0x7f01140c3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:20.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0118a71640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f011406ded0 0x7f01140c3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:20.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0118a71640 1 -- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01141539a0 msgr2=0x7f011414e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:20.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0118a71640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01141539a0 0x7f011414e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:20.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0118a71640 1 -- 172.21.15.46:0/2685603424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010c009a70 con 0x7f0114070450 2023-10-30T13:30:20.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0119a73640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01141539a0 0x7f011414e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:20.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0119272640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f011406ded0 0x7f01140c3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:20.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0118a71640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0114070450 0x7f0114144040 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f010400d800 tx=0x7f010400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:20.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0109ffb640 1 -- 172.21.15.46:0/2685603424 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010400b6c0 con 0x7f0114070450 2023-10-30T13:30:20.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0109ffb640 1 -- 172.21.15.46:0/2685603424 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010400b820 con 0x7f0114070450 2023-10-30T13:30:20.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f011a274640 1 -- 172.21.15.46:0/2685603424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011414e1b0 con 0x7f0114070450 2023-10-30T13:30:20.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0109ffb640 1 -- 172.21.15.46:0/2685603424 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010400bb90 con 0x7f0114070450 2023-10-30T13:30:20.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f011a274640 1 -- 172.21.15.46:0/2685603424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011414d4c0 con 0x7f0114070450 2023-10-30T13:30:20.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f011a274640 1 -- 172.21.15.46:0/2685603424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011404e170 con 0x7f0114070450 2023-10-30T13:30:20.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.896+0000 7f0109ffb640 1 -- 172.21.15.46:0/2685603424 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f010400bcf0 con 0x7f0114070450 2023-10-30T13:30:20.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.900+0000 7f0109ffb640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00f80664a0 0x7f00f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:20.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.900+0000 7f0109ffb640 1 -- 172.21.15.46:0/2685603424 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f01040845c0 con 0x7f0114070450 2023-10-30T13:30:20.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.900+0000 7f0119272640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00f80664a0 0x7f00f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:20.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.900+0000 7f0119272640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00f80664a0 0x7f00f8068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0110005890 tx=0x7f0110009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:20.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:20.904+0000 7f0109ffb640 1 -- 172.21.15.46:0/2685603424 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0104051830 con 0x7f0114070450 2023-10-30T13:30:21.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.096+0000 7f011a274640 1 -- 172.21.15.46:0/2685603424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0114163c70 con 0x7f0114070450 2023-10-30T13:30:21.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.096+0000 7f0109ffb640 1 -- 172.21.15.46:0/2685603424 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f0104054e50 con 0x7f0114070450 2023-10-30T13:30:21.103 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:21.104 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:52.914924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:21.104 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 37 2023-10-30T13:30:21.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.096+0000 7f011a274640 1 -- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00f80664a0 msgr2=0x7f00f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:21.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.096+0000 7f011a274640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00f80664a0 0x7f00f8068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0110005890 tx=0x7f0110009040 comp rx=0 tx=0).stop 2023-10-30T13:30:21.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.096+0000 7f011a274640 1 -- 172.21.15.46:0/2685603424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0114070450 msgr2=0x7f0114144040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:21.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.096+0000 7f011a274640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0114070450 0x7f0114144040 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f010400d800 tx=0x7f010400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:30:21.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.100+0000 7f011a274640 1 -- 172.21.15.46:0/2685603424 shutdown_connections 2023-10-30T13:30:21.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.100+0000 7f011a274640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00f80664a0 0x7f00f8068960 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:21.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.100+0000 7f011a274640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01141539a0 0x7f011414e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:21.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.100+0000 7f011a274640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0114070450 0x7f0114144040 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:21.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.100+0000 7f011a274640 1 --2- 172.21.15.46:0/2685603424 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f011406ded0 0x7f01140c3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:21.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.100+0000 7f011a274640 1 -- 172.21.15.46:0/2685603424 >> 172.21.15.46:0/2685603424 conn(0x7f01140cbe40 msgr2=0x7f011414c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:21.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.100+0000 7f011a274640 1 -- 172.21.15.46:0/2685603424 shutdown_connections 2023-10-30T13:30:21.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:21.100+0000 7f011a274640 1 -- 172.21.15.46:0/2685603424 wait complete. 2023-10-30T13:30:21.125 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:30:23.126 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.268+0000 7f2c9a474640 1 Processor -- start 2023-10-30T13:30:23.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c9a474640 1 -- start start 2023-10-30T13:30:23.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c9a474640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c941538f0 0x7f2c94159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:23.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c9a474640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c9415cd50 0x7f2c940c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c9a474640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9415d300 con 0x7f2c940cefa0 2023-10-30T13:30:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c9a474640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9415d5e0 con 0x7f2c941538f0 2023-10-30T13:30:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c9a474640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9415d8c0 con 0x7f2c9415cd50 2023-10-30T13:30:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c99472640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c941538f0 0x7f2c94159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:23.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c99472640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c941538f0 0x7f2c94159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55138/0 (socket says 172.21.15.46:55138) 2023-10-30T13:30:23.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c99472640 1 -- 172.21.15.46:0/944646510 learned_addr learned my addr 172.21.15.46:0/944646510 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:23.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c98c71640 1 --2- 172.21.15.46:0/944646510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c9415cd50 0x7f2c940c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:23.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c99472640 1 -- 172.21.15.46:0/944646510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c9415cd50 msgr2=0x7f2c940c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:23.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c99472640 1 --2- 172.21.15.46:0/944646510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c9415cd50 0x7f2c940c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:23.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c99472640 1 -- 172.21.15.46:0/944646510 >> v1:172.21.15.154:6789/0 conn(0x7f2c940cefa0 legacy=0x7f2c940cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:23.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.272+0000 7f2c99472640 1 -- 172.21.15.46:0/944646510 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9415dba0 con 0x7f2c941538f0 2023-10-30T13:30:23.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c99472640 1 --2- 172.21.15.46:0/944646510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c941538f0 0x7f2c94159fb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009870 tx=0x7f2c9001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b17eda273a98fa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:23.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c8bfff640 1 -- 172.21.15.46:0/944646510 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c900a8a30 con 0x7f2c941538f0 2023-10-30T13:30:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c8bfff640 1 -- 172.21.15.46:0/944646510 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c900a8b90 con 0x7f2c941538f0 2023-10-30T13:30:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c8bfff640 1 -- 172.21.15.46:0/944646510 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c900b0440 con 0x7f2c941538f0 2023-10-30T13:30:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c9a474640 1 -- 172.21.15.46:0/944646510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c941538f0 msgr2=0x7f2c94159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c9a474640 1 --2- 172.21.15.46:0/944646510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c941538f0 0x7f2c94159fb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009870 tx=0x7f2c9001ed30 comp rx=0 tx=0).stop 2023-10-30T13:30:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c9a474640 1 -- 172.21.15.46:0/944646510 shutdown_connections 2023-10-30T13:30:23.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c9a474640 1 --2- 172.21.15.46:0/944646510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c9415cd50 0x7f2c940c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:23.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c9a474640 1 --2- 172.21.15.46:0/944646510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c941538f0 0x7f2c94159fb0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:23.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c9a474640 1 -- 172.21.15.46:0/944646510 >> 172.21.15.46:0/944646510 conn(0x7f2c9406a640 msgr2=0x7f2c94065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:23.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c9a474640 1 -- 172.21.15.46:0/944646510 shutdown_connections 2023-10-30T13:30:23.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c9a474640 1 -- 172.21.15.46:0/944646510 wait complete. 2023-10-30T13:30:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.276+0000 7f2c9a474640 1 Processor -- start 2023-10-30T13:30:23.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c9a474640 1 -- start start 2023-10-30T13:30:23.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c9a474640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c940cefa0 0x7f2c9415a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:23.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c9a474640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c941538f0 0x7f2c9415b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:23.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c9a474640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c9415cd50 0x7f2c9415b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:23.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c9a474640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c940b42a0 con 0x7f2c940cefa0 2023-10-30T13:30:23.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c9a474640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94150050 con 0x7f2c941538f0 2023-10-30T13:30:23.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c9a474640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c940b4eb0 con 0x7f2c9415cd50 2023-10-30T13:30:23.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c99472640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c940cefa0 0x7f2c9415a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:23.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c99472640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c940cefa0 0x7f2c9415a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55146/0 (socket says 172.21.15.46:55146) 2023-10-30T13:30:23.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c99472640 1 -- 172.21.15.46:0/32924680 learned_addr learned my addr 172.21.15.46:0/32924680 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:23.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c99c73640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c9415cd50 0x7f2c9415b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:23.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c98c71640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c941538f0 0x7f2c9415b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:23.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c99472640 1 -- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c9415cd50 msgr2=0x7f2c9415b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:23.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c99472640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c9415cd50 0x7f2c9415b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:23.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c99472640 1 -- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c941538f0 msgr2=0x7f2c9415b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c99472640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c941538f0 0x7f2c9415b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c99472640 1 -- 172.21.15.46:0/32924680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c900094d0 con 0x7f2c940cefa0 2023-10-30T13:30:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c99472640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c940cefa0 0x7f2c9415a900 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2c900b3040 tx=0x7f2c9001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c89ffb640 1 -- 172.21.15.46:0/32924680 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c900be070 con 0x7f2c940cefa0 2023-10-30T13:30:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c89ffb640 1 -- 172.21.15.46:0/32924680 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c900ba400 con 0x7f2c940cefa0 2023-10-30T13:30:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c89ffb640 1 -- 172.21.15.46:0/32924680 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c900ba7d0 con 0x7f2c940cefa0 2023-10-30T13:30:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c9a474640 1 -- 172.21.15.46:0/32924680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c941502f0 con 0x7f2c940cefa0 2023-10-30T13:30:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c9a474640 1 -- 172.21.15.46:0/32924680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c940b5190 con 0x7f2c940cefa0 2023-10-30T13:30:23.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.280+0000 7f2c9a474640 1 -- 172.21.15.46:0/32924680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c940b46d0 con 0x7f2c940cefa0 2023-10-30T13:30:23.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.284+0000 7f2c89ffb640 1 -- 172.21.15.46:0/32924680 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c900c3410 con 0x7f2c940cefa0 2023-10-30T13:30:23.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.284+0000 7f2c89ffb640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c780664d0 0x7f2c78068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:23.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.284+0000 7f2c89ffb640 1 -- 172.21.15.46:0/32924680 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f2c90121c70 con 0x7f2c940cefa0 2023-10-30T13:30:23.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.284+0000 7f2c98c71640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c780664d0 0x7f2c78068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:23.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.284+0000 7f2c98c71640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c780664d0 0x7f2c78068990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2c940c9df0 tx=0x7f2c84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:23.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.288+0000 7f2c89ffb640 1 -- 172.21.15.46:0/32924680 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c940b46d0 con 0x7f2c940cefa0 2023-10-30T13:30:23.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.476+0000 7f2c9a474640 1 -- 172.21.15.46:0/32924680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c940b46d0 con 0x7f2c940cefa0 2023-10-30T13:30:23.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.476+0000 7f2c89ffb640 1 -- 172.21.15.46:0/32924680 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f2c940b46d0 con 0x7f2c940cefa0 2023-10-30T13:30:23.484 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:23.484 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:52.914924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:23.484 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 37 2023-10-30T13:30:23.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.480+0000 7f2c9a474640 1 -- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c780664d0 msgr2=0x7f2c78068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:23.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.480+0000 7f2c9a474640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c780664d0 0x7f2c78068990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2c940c9df0 tx=0x7f2c84008040 comp rx=0 tx=0).stop 2023-10-30T13:30:23.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.480+0000 7f2c9a474640 1 -- 172.21.15.46:0/32924680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c940cefa0 msgr2=0x7f2c9415a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:23.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.480+0000 7f2c9a474640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c940cefa0 0x7f2c9415a900 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2c900b3040 tx=0x7f2c9001bad0 comp rx=0 tx=0).stop 2023-10-30T13:30:23.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.480+0000 7f2c9a474640 1 -- 172.21.15.46:0/32924680 shutdown_connections 2023-10-30T13:30:23.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.480+0000 7f2c9a474640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c780664d0 0x7f2c78068990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:23.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.480+0000 7f2c9a474640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c9415cd50 0x7f2c9415b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:23.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.480+0000 7f2c9a474640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c941538f0 0x7f2c9415b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:23.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.480+0000 7f2c9a474640 1 --2- 172.21.15.46:0/32924680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c940cefa0 0x7f2c9415a900 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:23.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.480+0000 7f2c9a474640 1 -- 172.21.15.46:0/32924680 >> 172.21.15.46:0/32924680 conn(0x7f2c9406a640 msgr2=0x7f2c94158040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:23.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.480+0000 7f2c9a474640 1 -- 172.21.15.46:0/32924680 shutdown_connections 2023-10-30T13:30:23.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:23.480+0000 7f2c9a474640 1 -- 172.21.15.46:0/32924680 wait complete. 2023-10-30T13:30:23.505 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:30:25.507 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:25.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefff6ea640 1 Processor -- start 2023-10-30T13:30:25.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefff6ea640 1 -- start start 2023-10-30T13:30:25.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefff6ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff806a720 0x7feff806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:25.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefff6ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff8071290 con 0x7feff80bfec0 2023-10-30T13:30:25.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefff6ea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff8071570 con 0x7feff8159e60 2023-10-30T13:30:25.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefff6ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff8071850 con 0x7feff806a720 2023-10-30T13:30:25.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffcc85640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7feff80bfec0 0x7feff80c1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45776/0 (socket says 172.21.15.46:45776) 2023-10-30T13:30:25.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffcc85640 1 -- 172.21.15.46:0/1321903890 learned_addr learned my addr 172.21.15.46:0/1321903890 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:30:25.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffdc87640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff806a720 0x7feff806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:25.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891847998 0 0) 0x7feff8071290 con 0x7feff80bfec0 2023-10-30T13:30:25.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe4003530 con 0x7feff80bfec0 2023-10-30T13:30:25.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1539752378 0 0) 0x7fefe4003530 con 0x7feff80bfec0 2023-10-30T13:30:25.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff806a720 msgr2=0x7feff806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff806a720 0x7feff806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 >> v1:172.21.15.154:6789/0 conn(0x7feff8159e60 legacy=0x7feff806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff8071b30 con 0x7feff80bfec0 2023-10-30T13:30:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fefe8001680 con 0x7feff80bfec0 2023-10-30T13:30:25.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 >> v1:172.21.15.46:6789/0 conn(0x7feff80bfec0 legacy=0x7feff80c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefe40039b0 0x7fefe4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe4004360 0x7fefe4004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe4004fe0 0x7fefe400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff8071850 con 0x7fefe40039b0 2023-10-30T13:30:25.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe400e0e0 con 0x7fefe4004fe0 2023-10-30T13:30:25.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe400e220 con 0x7fefe4004360 2023-10-30T13:30:25.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffdc87640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff806a720 0x7feff806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:30:25.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffd486640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefe40039b0 0x7fefe4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:25.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffcc85640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe4004360 0x7fefe4004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:25.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffdc87640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe4004fe0 0x7fefe400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:25.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffd486640 1 -- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe4004360 msgr2=0x7fefe4004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:25.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffd486640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe4004360 0x7fefe4004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:25.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffd486640 1 -- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe4004fe0 msgr2=0x7fefe400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:25.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffd486640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe4004fe0 0x7fefe400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:25.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffd486640 1 -- 172.21.15.46:0/1321903890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe4003530 con 0x7fefe40039b0 2023-10-30T13:30:25.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.656+0000 7feffcc85640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe4004360 0x7fefe4004830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:25.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.660+0000 7feffd486640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefe40039b0 0x7fefe4003d90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7feff4001860 tx=0x7feff401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c994f302fd6c2721 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:25.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.660+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff40a8890 con 0x7fefe40039b0 2023-10-30T13:30:25.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.660+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff40a89f0 con 0x7fefe40039b0 2023-10-30T13:30:25.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.660+0000 7fefeffff640 1 -- 172.21.15.46:0/1321903890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff40a8da0 con 0x7fefe40039b0 2023-10-30T13:30:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.660+0000 7fefff6ea640 1 -- 172.21.15.46:0/1321903890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefe40039b0 msgr2=0x7fefe4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.660+0000 7fefff6ea640 1 --2- 172.21.15.46:0/1321903890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefe40039b0 0x7fefe4003d90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7feff4001860 tx=0x7feff401ec70 comp rx=0 tx=0).stop 2023-10-30T13:30:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.660+0000 7feffdc87640 1 -- 172.21.15.46:0/1321903890 reap_dead start 2023-10-30T13:30:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.660+0000 7fefff6ea640 1 -- 172.21.15.46:0/1321903890 shutdown_connections 2023-10-30T13:30:25.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.660+0000 7fefff6ea640 1 -- 172.21.15.46:0/1321903890 >> 172.21.15.46:0/1321903890 conn(0x7feff80c7910 msgr2=0x7feff80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.660+0000 7fefff6ea640 1 -- 172.21.15.46:0/1321903890 shutdown_connections 2023-10-30T13:30:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.660+0000 7fefff6ea640 1 -- 172.21.15.46:0/1321903890 wait complete. 2023-10-30T13:30:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7fefff6ea640 1 Processor -- start 2023-10-30T13:30:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7fefff6ea640 1 -- start start 2023-10-30T13:30:25.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7fefff6ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe40039b0 0x7feff81eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7fefff6ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff80c3590 0x7feff814d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7fefff6ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feff814dbb0 0x7feff80c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7fefff6ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff80c1cf0 con 0x7feff814dbb0 2023-10-30T13:30:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7fefff6ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff80c2d40 con 0x7fefe40039b0 2023-10-30T13:30:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7fefff6ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff80c2220 con 0x7feff80c3590 2023-10-30T13:30:25.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffdc87640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feff814dbb0 0x7feff80c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:25.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffdc87640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feff814dbb0 0x7feff80c3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55170/0 (socket says 172.21.15.46:55170) 2023-10-30T13:30:25.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffdc87640 1 -- 172.21.15.46:0/3453564876 learned_addr learned my addr 172.21.15.46:0/3453564876 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:25.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffcc85640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff80c3590 0x7feff814d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:25.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffd486640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe40039b0 0x7feff81eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:25.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffdc87640 1 -- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff80c3590 msgr2=0x7feff814d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:25.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffdc87640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff80c3590 0x7feff814d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:25.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffdc87640 1 -- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe40039b0 msgr2=0x7feff81eedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:25.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffdc87640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe40039b0 0x7feff81eedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:25.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffdc87640 1 -- 172.21.15.46:0/3453564876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff814e030 con 0x7feff814dbb0 2023-10-30T13:30:25.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffcc85640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff80c3590 0x7feff814d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:25.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffd486640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe40039b0 0x7feff81eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:25.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7feffdc87640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feff814dbb0 0x7feff80c3d40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7feff806ab00 tx=0x7feff00075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:25.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7fefedffb640 1 -- 172.21.15.46:0/3453564876 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff00039a0 con 0x7feff814dbb0 2023-10-30T13:30:25.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7fefedffb640 1 -- 172.21.15.46:0/3453564876 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff0010050 con 0x7feff814dbb0 2023-10-30T13:30:25.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7fefedffb640 1 -- 172.21.15.46:0/3453564876 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0003d40 con 0x7feff814dbb0 2023-10-30T13:30:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.664+0000 7fefff6ea640 1 -- 172.21.15.46:0/3453564876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff81f2dd0 con 0x7feff814dbb0 2023-10-30T13:30:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.668+0000 7fefff6ea640 1 -- 172.21.15.46:0/3453564876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff80c1f40 con 0x7feff814dbb0 2023-10-30T13:30:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.668+0000 7fefff6ea640 1 -- 172.21.15.46:0/3453564876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff804e170 con 0x7feff814dbb0 2023-10-30T13:30:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.668+0000 7fefedffb640 1 -- 172.21.15.46:0/3453564876 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feff0004030 con 0x7feff814dbb0 2023-10-30T13:30:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.668+0000 7fefedffb640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fefdc0664a0 0x7fefdc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.668+0000 7fefedffb640 1 -- 172.21.15.46:0/3453564876 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7feff0083e80 con 0x7feff814dbb0 2023-10-30T13:30:25.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.668+0000 7feffd486640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fefdc0664a0 0x7fefdc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:25.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.672+0000 7feffd486640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fefdc0664a0 0x7fefdc068960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7feff40b2000 tx=0x7feff40aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:25.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.672+0000 7fefedffb640 1 -- 172.21.15.46:0/3453564876 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feff0051180 con 0x7feff814dbb0 2023-10-30T13:30:25.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.860+0000 7fefff6ea640 1 -- 172.21.15.46:0/3453564876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feff80c2e80 con 0x7feff814dbb0 2023-10-30T13:30:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.860+0000 7fefedffb640 1 -- 172.21.15.46:0/3453564876 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6530 (secure 0 0 0) 0x7feff001a090 con 0x7feff814dbb0 2023-10-30T13:30:25.871 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:25.871 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:52.914924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:25.872 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 37 2023-10-30T13:30:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.864+0000 7fefff6ea640 1 -- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fefdc0664a0 msgr2=0x7fefdc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.864+0000 7fefff6ea640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fefdc0664a0 0x7fefdc068960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7feff40b2000 tx=0x7feff40aa070 comp rx=0 tx=0).stop 2023-10-30T13:30:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.864+0000 7fefff6ea640 1 -- 172.21.15.46:0/3453564876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feff814dbb0 msgr2=0x7feff80c3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.864+0000 7fefff6ea640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feff814dbb0 0x7feff80c3d40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7feff806ab00 tx=0x7feff00075c0 comp rx=0 tx=0).stop 2023-10-30T13:30:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.864+0000 7fefff6ea640 1 -- 172.21.15.46:0/3453564876 shutdown_connections 2023-10-30T13:30:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.864+0000 7fefff6ea640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fefdc0664a0 0x7fefdc068960 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.864+0000 7fefff6ea640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feff814dbb0 0x7feff80c3d40 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.864+0000 7fefff6ea640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff80c3590 0x7feff814d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.864+0000 7fefff6ea640 1 --2- 172.21.15.46:0/3453564876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe40039b0 0x7feff81eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.864+0000 7fefff6ea640 1 -- 172.21.15.46:0/3453564876 >> 172.21.15.46:0/3453564876 conn(0x7feff80c7910 msgr2=0x7feff81578c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.864+0000 7fefff6ea640 1 -- 172.21.15.46:0/3453564876 shutdown_connections 2023-10-30T13:30:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:25.864+0000 7fefff6ea640 1 -- 172.21.15.46:0/3453564876 wait complete. 2023-10-30T13:30:25.890 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:30:27.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:28.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f26261c5640 1 Processor -- start 2023-10-30T13:30:28.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f26261c5640 1 -- start start 2023-10-30T13:30:28.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f26261c5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26201538a0 0x7f262015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:28.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f26261c5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f26200ca0e0 0x7f262015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:28.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f26261c5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26200cb800 con 0x7f26200cf160 2023-10-30T13:30:28.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f26261c5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f262015dc30 con 0x7f26201538a0 2023-10-30T13:30:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f26261c5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26200cb330 con 0x7f26200ca0e0 2023-10-30T13:30:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f261f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26201538a0 0x7f262015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f261f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f26201538a0 0x7f262015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55404/0 (socket says 172.21.15.46:55404) 2023-10-30T13:30:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f261f7fe640 1 -- 172.21.15.46:0/77251197 learned_addr learned my addr 172.21.15.46:0/77251197 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f261effd640 1 --2- 172.21.15.46:0/77251197 >> v2:172.21.15.154:3300/0 conn(0x7f26200ca0e0 0x7f262015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:28.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f261f7fe640 1 -- 172.21.15.46:0/77251197 >> v2:172.21.15.154:3300/0 conn(0x7f26200ca0e0 msgr2=0x7f262015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:28.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f261f7fe640 1 --2- 172.21.15.46:0/77251197 >> v2:172.21.15.154:3300/0 conn(0x7f26200ca0e0 0x7f262015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:28.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f261f7fe640 1 -- 172.21.15.46:0/77251197 >> v1:172.21.15.154:6789/0 conn(0x7f26200cf160 legacy=0x7f26200cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:28.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.040+0000 7f261f7fe640 1 -- 172.21.15.46:0/77251197 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26200cad70 con 0x7f26201538a0 2023-10-30T13:30:28.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.044+0000 7f261f7fe640 1 --2- 172.21.15.46:0/77251197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26201538a0 0x7f262015d6f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f26200b30a0 tx=0x7f261001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4717467b75cce1b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:28.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.044+0000 7f261e7fc640 1 -- 172.21.15.46:0/77251197 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26100a89b0 con 0x7f26201538a0 2023-10-30T13:30:28.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.044+0000 7f261e7fc640 1 -- 172.21.15.46:0/77251197 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26100a8b10 con 0x7f26201538a0 2023-10-30T13:30:28.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.044+0000 7f261e7fc640 1 -- 172.21.15.46:0/77251197 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26100a8ea0 con 0x7f26201538a0 2023-10-30T13:30:28.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.044+0000 7f26261c5640 1 -- 172.21.15.46:0/77251197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26201538a0 msgr2=0x7f262015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:28.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.044+0000 7f26261c5640 1 --2- 172.21.15.46:0/77251197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26201538a0 0x7f262015d6f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f26200b30a0 tx=0x7f261001ecb0 comp rx=0 tx=0).stop 2023-10-30T13:30:28.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.044+0000 7f26261c5640 1 -- 172.21.15.46:0/77251197 shutdown_connections 2023-10-30T13:30:28.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.044+0000 7f26261c5640 1 --2- 172.21.15.46:0/77251197 >> v2:172.21.15.154:3300/0 conn(0x7f26200ca0e0 0x7f262015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:28.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.044+0000 7f26261c5640 1 --2- 172.21.15.46:0/77251197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26201538a0 0x7f262015d6f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:28.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.044+0000 7f26261c5640 1 -- 172.21.15.46:0/77251197 >> 172.21.15.46:0/77251197 conn(0x7f262006bfe0 msgr2=0x7f262006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:28.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.044+0000 7f26261c5640 1 -- 172.21.15.46:0/77251197 shutdown_connections 2023-10-30T13:30:28.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.044+0000 7f26261c5640 1 -- 172.21.15.46:0/77251197 wait complete. 2023-10-30T13:30:28.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f26261c5640 1 Processor -- start 2023-10-30T13:30:28.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f26261c5640 1 -- start start 2023-10-30T13:30:28.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f26261c5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26200ca0e0 0x7f26200b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:28.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f26261c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26200cf160 0x7f26201466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:28.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f26261c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26201538a0 0x7f26201498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:28.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f26261c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26200cda00 con 0x7f26200cf160 2023-10-30T13:30:28.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f26261c5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26201e2ae0 con 0x7f26200ca0e0 2023-10-30T13:30:28.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f26261c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26200b3570 con 0x7f26201538a0 2023-10-30T13:30:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f261effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26200cf160 0x7f26201466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f261effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26200cf160 0x7f26201466e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55408/0 (socket says 172.21.15.46:55408) 2023-10-30T13:30:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f261effd640 1 -- 172.21.15.46:0/2191855985 learned_addr learned my addr 172.21.15.46:0/2191855985 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f261f7fe640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26200ca0e0 0x7f26200b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f261ffff640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26201538a0 0x7f26201498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f261effd640 1 -- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26201538a0 msgr2=0x7f26201498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f261effd640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26201538a0 0x7f26201498e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f261effd640 1 -- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26200ca0e0 msgr2=0x7f26200b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f261effd640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26200ca0e0 0x7f26200b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f261effd640 1 -- 172.21.15.46:0/2191855985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26100094d0 con 0x7f26200cf160 2023-10-30T13:30:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f261f7fe640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26200ca0e0 0x7f26200b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f261effd640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26200cf160 0x7f26201466e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f261400d800 tx=0x7f261400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f2607fff640 1 -- 172.21.15.46:0/2191855985 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261400b6c0 con 0x7f26200cf160 2023-10-30T13:30:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f2607fff640 1 -- 172.21.15.46:0/2191855985 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261400b820 con 0x7f26200cf160 2023-10-30T13:30:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f26261c5640 1 -- 172.21.15.46:0/2191855985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2620145ff0 con 0x7f26200cf160 2023-10-30T13:30:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f2607fff640 1 -- 172.21.15.46:0/2191855985 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261400bb90 con 0x7f26200cf160 2023-10-30T13:30:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.048+0000 7f26261c5640 1 -- 172.21.15.46:0/2191855985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26200ccfb0 con 0x7f26200cf160 2023-10-30T13:30:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.052+0000 7f26261c5640 1 -- 172.21.15.46:0/2191855985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26200b2a50 con 0x7f26200cf160 2023-10-30T13:30:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.052+0000 7f2607fff640 1 -- 172.21.15.46:0/2191855985 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f261400bcf0 con 0x7f26200cf160 2023-10-30T13:30:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.052+0000 7f2607fff640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f260c0664a0 0x7f260c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.052+0000 7f2607fff640 1 -- 172.21.15.46:0/2191855985 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f2614084510 con 0x7f26200cf160 2023-10-30T13:30:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.052+0000 7f261f7fe640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f260c0664a0 0x7f260c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:28.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.052+0000 7f261f7fe640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f260c0664a0 0x7f260c068960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f26100b2000 tx=0x7f2610008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:28.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.056+0000 7f2607fff640 1 -- 172.21.15.46:0/2191855985 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2614051890 con 0x7f26200cf160 2023-10-30T13:30:28.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 -- 172.21.15.46:0/2191855985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f262004e170 con 0x7f26200cf160 2023-10-30T13:30:28.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f2607fff640 1 -- 172.21.15.46:0/2191855985 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f2614054eb0 con 0x7f26200cf160 2023-10-30T13:30:28.255 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:28.255 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:29:52.914924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6836/275749119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":275749119},{"type":"v1","addr":"172.21.15.46:6836","nonce":275749119}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:28.255 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 37 2023-10-30T13:30:28.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 -- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f260c0664a0 msgr2=0x7f260c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f260c0664a0 0x7f260c068960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f26100b2000 tx=0x7f2610008040 comp rx=0 tx=0).stop 2023-10-30T13:30:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 -- 172.21.15.46:0/2191855985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26200cf160 msgr2=0x7f26201466e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26200cf160 0x7f26201466e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f261400d800 tx=0x7f261400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:30:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 -- 172.21.15.46:0/2191855985 shutdown_connections 2023-10-30T13:30:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f260c0664a0 0x7f260c068960 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26201538a0 0x7f26201498e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26200cf160 0x7f26201466e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 --2- 172.21.15.46:0/2191855985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26200ca0e0 0x7f26200b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 -- 172.21.15.46:0/2191855985 >> 172.21.15.46:0/2191855985 conn(0x7f262006bfe0 msgr2=0x7f26200c5460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 -- 172.21.15.46:0/2191855985 shutdown_connections 2023-10-30T13:30:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:28.248+0000 7f26261c5640 1 -- 172.21.15.46:0/2191855985 wait complete. 2023-10-30T13:30:28.276 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:30:30.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:30.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.420+0000 7f3eded3b640 1 Processor -- start 2023-10-30T13:30:30.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3eded3b640 1 -- start start 2023-10-30T13:30:30.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3eded3b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed815cd50 0x7f3ed80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3eded3b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3ed80cefa0 0x7f3ed80cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3eded3b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed815a6c0 con 0x7f3ed81538f0 2023-10-30T13:30:30.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3eded3b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed815d300 con 0x7f3ed80cefa0 2023-10-30T13:30:30.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3eded3b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed815d5e0 con 0x7f3ed815cd50 2023-10-30T13:30:30.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3eddd39640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3ed81538f0 0x7f3ed8159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46756/0 (socket says 172.21.15.46:46756) 2023-10-30T13:30:30.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3eddd39640 1 -- 172.21.15.46:0/1149252710 learned_addr learned my addr 172.21.15.46:0/1149252710 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:30:30.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edd538640 1 --2- 172.21.15.46:0/1149252710 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed815cd50 0x7f3ed80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3ede53a640 1 --2- 172.21.15.46:0/1149252710 >> v2:172.21.15.154:3300/0 conn(0x7f3ed80cefa0 0x7f3ed80cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710271617 0 0) 0x7f3ed815a6c0 con 0x7f3ed81538f0 2023-10-30T13:30:30.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec4003530 con 0x7f3ed81538f0 2023-10-30T13:30:30.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2362184850 0 0) 0x7f3ec4003530 con 0x7f3ed81538f0 2023-10-30T13:30:30.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed815cd50 msgr2=0x7f3ed80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:30.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 --2- 172.21.15.46:0/1149252710 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed815cd50 0x7f3ed80c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 >> v2:172.21.15.154:3300/0 conn(0x7f3ed80cefa0 msgr2=0x7f3ed80cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:30.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 --2- 172.21.15.46:0/1149252710 >> v2:172.21.15.154:3300/0 conn(0x7f3ed80cefa0 0x7f3ed80cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed815d8c0 con 0x7f3ed81538f0 2023-10-30T13:30:30.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f3ed4001680 con 0x7f3ed81538f0 2023-10-30T13:30:30.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 >> v1:172.21.15.46:6789/0 conn(0x7f3ed81538f0 legacy=0x7f3ed8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:30.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 --2- 172.21.15.46:0/1149252710 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed815cd50 0x7f3ec4003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 --2- 172.21.15.46:0/1149252710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ec4003f00 0x7f3ec40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 --2- 172.21.15.46:0/1149252710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec4004d10 0x7f3ec4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed815d300 con 0x7f3ec4004d10 2023-10-30T13:30:30.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed815d5e0 con 0x7f3ec4003f00 2023-10-30T13:30:30.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec400b9c0 con 0x7f3ed815cd50 2023-10-30T13:30:30.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3ede53a640 1 --2- 172.21.15.46:0/1149252710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec4004d10 0x7f3ec4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3eddd39640 1 --2- 172.21.15.46:0/1149252710 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed815cd50 0x7f3ec4003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3edd538640 1 --2- 172.21.15.46:0/1149252710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ec4003f00 0x7f3ec40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3ede53a640 1 -- 172.21.15.46:0/1149252710 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed815cd50 msgr2=0x7f3ec4003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:30.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3ede53a640 1 --2- 172.21.15.46:0/1149252710 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed815cd50 0x7f3ec4003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3ede53a640 1 -- 172.21.15.46:0/1149252710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ec4003f00 msgr2=0x7f3ec40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:30.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3ede53a640 1 --2- 172.21.15.46:0/1149252710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ec4003f00 0x7f3ec40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.424+0000 7f3ede53a640 1 -- 172.21.15.46:0/1149252710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec4003530 con 0x7f3ec4004d10 2023-10-30T13:30:30.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.428+0000 7f3ede53a640 1 --2- 172.21.15.46:0/1149252710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec4004d10 0x7f3ec4005190 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009fc0 tx=0x7f3ed001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3799885ba35c016 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:30.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.428+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed00a5cb0 con 0x7f3ec4004d10 2023-10-30T13:30:30.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.428+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed000f040 con 0x7f3ec4004d10 2023-10-30T13:30:30.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.428+0000 7f3edcd37640 1 -- 172.21.15.46:0/1149252710 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed00ad680 con 0x7f3ec4004d10 2023-10-30T13:30:30.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.428+0000 7f3eded3b640 1 -- 172.21.15.46:0/1149252710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec4004d10 msgr2=0x7f3ec4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:30.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.428+0000 7f3eded3b640 1 --2- 172.21.15.46:0/1149252710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec4004d10 0x7f3ec4005190 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009fc0 tx=0x7f3ed001cca0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.428+0000 7f3ede53a640 1 -- 172.21.15.46:0/1149252710 reap_dead start 2023-10-30T13:30:30.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.428+0000 7f3eded3b640 1 -- 172.21.15.46:0/1149252710 shutdown_connections 2023-10-30T13:30:30.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.428+0000 7f3eded3b640 1 -- 172.21.15.46:0/1149252710 >> 172.21.15.46:0/1149252710 conn(0x7f3ed806a640 msgr2=0x7f3ed8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:30.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.428+0000 7f3eded3b640 1 -- 172.21.15.46:0/1149252710 shutdown_connections 2023-10-30T13:30:30.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.428+0000 7f3eded3b640 1 -- 172.21.15.46:0/1149252710 wait complete. 2023-10-30T13:30:30.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3eded3b640 1 Processor -- start 2023-10-30T13:30:30.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3eded3b640 1 -- start start 2023-10-30T13:30:30.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3eded3b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed80ce460 0x7f3ed81ef420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3eded3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ed80ce840 0x7f3ed81ef960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3eded3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed80b35e0 0x7f3ed813e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3eded3b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed81eef70 con 0x7f3ed80ce840 2023-10-30T13:30:30.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3eded3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed81f2690 con 0x7f3ed80b35e0 2023-10-30T13:30:30.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3eded3b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed81f2970 con 0x7f3ed80ce460 2023-10-30T13:30:30.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3edd538640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ed80ce840 0x7f3ed81ef960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3edd538640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ed80ce840 0x7f3ed81ef960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55426/0 (socket says 172.21.15.46:55426) 2023-10-30T13:30:30.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3edd538640 1 -- 172.21.15.46:0/1432397077 learned_addr learned my addr 172.21.15.46:0/1432397077 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3ede53a640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed80b35e0 0x7f3ed813e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3eddd39640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed80ce460 0x7f3ed81ef420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3edd538640 1 -- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed80ce460 msgr2=0x7f3ed81ef420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3edd538640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed80ce460 0x7f3ed81ef420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3edd538640 1 -- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed80b35e0 msgr2=0x7f3ed813e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:30.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3edd538640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed80b35e0 0x7f3ed813e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3edd538640 1 -- 172.21.15.46:0/1432397077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed81f2c50 con 0x7f3ed80ce840 2023-10-30T13:30:30.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3ede53a640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed80b35e0 0x7f3ed813e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:30.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3edd538640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ed80ce840 0x7f3ed81ef960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8011560 tx=0x7f3ec8011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:30.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3ece7fc640 1 -- 172.21.15.46:0/1432397077 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8015930 con 0x7f3ed80ce840 2023-10-30T13:30:30.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3ece7fc640 1 -- 172.21.15.46:0/1432397077 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec8015a90 con 0x7f3ed80ce840 2023-10-30T13:30:30.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3ece7fc640 1 -- 172.21.15.46:0/1432397077 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8015e60 con 0x7f3ed80ce840 2023-10-30T13:30:30.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3eded3b640 1 -- 172.21.15.46:0/1432397077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed81f2f30 con 0x7f3ed80ce840 2023-10-30T13:30:30.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.432+0000 7f3eded3b640 1 -- 172.21.15.46:0/1432397077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed81f3210 con 0x7f3ed80ce840 2023-10-30T13:30:30.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.436+0000 7f3eded3b640 1 -- 172.21.15.46:0/1432397077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed804e170 con 0x7f3ed80ce840 2023-10-30T13:30:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.436+0000 7f3ece7fc640 1 -- 172.21.15.46:0/1432397077 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3ec800f460 con 0x7f3ed80ce840 2023-10-30T13:30:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.436+0000 7f3ece7fc640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3ebc0664d0 0x7f3ebc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.436+0000 7f3ece7fc640 1 -- 172.21.15.46:0/1432397077 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f3ec80848e0 con 0x7f3ed80ce840 2023-10-30T13:30:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.436+0000 7f3eddd39640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3ebc0664d0 0x7f3ebc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.436+0000 7f3eddd39640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3ebc0664d0 0x7f3ebc068990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4004850 tx=0x7f3ed4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.440+0000 7f3ece7fc640 1 -- 172.21.15.46:0/1432397077 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3ec8051ad0 con 0x7f3ed80ce840 2023-10-30T13:30:30.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.628+0000 7f3eded3b640 1 -- 172.21.15.46:0/1432397077 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ed81ef0b0 con 0x7f3ed80ce840 2023-10-30T13:30:30.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.628+0000 7f3ece7fc640 1 -- 172.21.15.46:0/1432397077 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f3ed81ef0b0 con 0x7f3ed80ce840 2023-10-30T13:30:30.638 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:30.638 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:28.793737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:30.640 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 38 2023-10-30T13:30:30.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.632+0000 7f3eded3b640 1 -- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3ebc0664d0 msgr2=0x7f3ebc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:30.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.632+0000 7f3eded3b640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3ebc0664d0 0x7f3ebc068990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4004850 tx=0x7f3ed4009040 comp rx=0 tx=0).stop 2023-10-30T13:30:30.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.632+0000 7f3eded3b640 1 -- 172.21.15.46:0/1432397077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ed80ce840 msgr2=0x7f3ed81ef960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:30.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.632+0000 7f3eded3b640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ed80ce840 0x7f3ed81ef960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8011560 tx=0x7f3ec8011a20 comp rx=0 tx=0).stop 2023-10-30T13:30:30.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.632+0000 7f3eded3b640 1 -- 172.21.15.46:0/1432397077 shutdown_connections 2023-10-30T13:30:30.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.632+0000 7f3eded3b640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3ebc0664d0 0x7f3ebc068990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.632+0000 7f3eded3b640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed80b35e0 0x7f3ed813e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.632+0000 7f3eded3b640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ed80ce840 0x7f3ed81ef960 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.632+0000 7f3eded3b640 1 --2- 172.21.15.46:0/1432397077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed80ce460 0x7f3ed81ef420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.632+0000 7f3eded3b640 1 -- 172.21.15.46:0/1432397077 >> 172.21.15.46:0/1432397077 conn(0x7f3ed806a640 msgr2=0x7f3ed8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:30.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.632+0000 7f3eded3b640 1 -- 172.21.15.46:0/1432397077 shutdown_connections 2023-10-30T13:30:30.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.632+0000 7f3eded3b640 1 -- 172.21.15.46:0/1432397077 wait complete. 2023-10-30T13:30:30.658 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:30:30.658 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:30:30.659 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:30.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f8882bf5640 1 Processor -- start 2023-10-30T13:30:30.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f8882bf5640 1 -- start start 2023-10-30T13:30:30.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f8882bf5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f887c159e60 0x7f887c06f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f8882bf5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f887c0bfec0 0x7f887c0c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f8882bf5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f887c06a720 0x7f887c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f8882bf5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f887c06fa80 con 0x7f887c0bfec0 2023-10-30T13:30:30.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f8882bf5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f887c071290 con 0x7f887c159e60 2023-10-30T13:30:30.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f8882bf5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887c071570 con 0x7f887c06a720 2023-10-30T13:30:30.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f887bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f887c0bfec0 0x7f887c0c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f887bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f887c0bfec0 0x7f887c0c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55442/0 (socket says 172.21.15.46:55442) 2023-10-30T13:30:30.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f887bfff640 1 -- 172.21.15.46:0/3304599762 learned_addr learned my addr 172.21.15.46:0/3304599762 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:30.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f8880991640 1 --2- 172.21.15.46:0/3304599762 >> v2:172.21.15.154:3300/0 conn(0x7f887c159e60 0x7f887c06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f8881192640 1 --2- 172.21.15.46:0/3304599762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f887c06a720 0x7f887c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f887bfff640 1 -- 172.21.15.46:0/3304599762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f887c06a720 msgr2=0x7f887c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:30.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f887bfff640 1 --2- 172.21.15.46:0/3304599762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f887c06a720 0x7f887c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f887bfff640 1 -- 172.21.15.46:0/3304599762 >> v2:172.21.15.154:3300/0 conn(0x7f887c159e60 msgr2=0x7f887c06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f887bfff640 1 --2- 172.21.15.46:0/3304599762 >> v2:172.21.15.154:3300/0 conn(0x7f887c159e60 0x7f887c06f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.796+0000 7f887bfff640 1 -- 172.21.15.46:0/3304599762 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c071850 con 0x7f887c0bfec0 2023-10-30T13:30:30.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f887bfff640 1 --2- 172.21.15.46:0/3304599762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f887c0bfec0 0x7f887c0c1a40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f887c0c8950 tx=0x7f886801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879bf7b302d88963 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:30.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f887b7fe640 1 -- 172.21.15.46:0/3304599762 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88680a89b0 con 0x7f887c0bfec0 2023-10-30T13:30:30.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f887b7fe640 1 -- 172.21.15.46:0/3304599762 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88680a8b10 con 0x7f887c0bfec0 2023-10-30T13:30:30.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f887b7fe640 1 -- 172.21.15.46:0/3304599762 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88680a8ea0 con 0x7f887c0bfec0 2023-10-30T13:30:30.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f8882bf5640 1 -- 172.21.15.46:0/3304599762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f887c0bfec0 msgr2=0x7f887c0c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:30.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f8882bf5640 1 --2- 172.21.15.46:0/3304599762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f887c0bfec0 0x7f887c0c1a40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f887c0c8950 tx=0x7f886801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f8882bf5640 1 -- 172.21.15.46:0/3304599762 shutdown_connections 2023-10-30T13:30:30.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f8882bf5640 1 --2- 172.21.15.46:0/3304599762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f887c06a720 0x7f887c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f8882bf5640 1 --2- 172.21.15.46:0/3304599762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f887c0bfec0 0x7f887c0c1a40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f8882bf5640 1 --2- 172.21.15.46:0/3304599762 >> v2:172.21.15.154:3300/0 conn(0x7f887c159e60 0x7f887c06f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f8882bf5640 1 -- 172.21.15.46:0/3304599762 >> 172.21.15.46:0/3304599762 conn(0x7f887c0c7910 msgr2=0x7f887c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:30.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f8882bf5640 1 -- 172.21.15.46:0/3304599762 shutdown_connections 2023-10-30T13:30:30.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.800+0000 7f8882bf5640 1 -- 172.21.15.46:0/3304599762 wait complete. 2023-10-30T13:30:30.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f8882bf5640 1 Processor -- start 2023-10-30T13:30:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f8882bf5640 1 -- start start 2023-10-30T13:30:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f8882bf5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f887c06a720 0x7f887c158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f8882bf5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f887c0bfec0 0x7f887c15ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f8882bf5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f887c159e60 0x7f887c15efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f8882bf5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c14ddf0 con 0x7f887c0bfec0 2023-10-30T13:30:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f8882bf5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c154f80 con 0x7f887c159e60 2023-10-30T13:30:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f8882bf5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887c1f2a00 con 0x7f887c06a720 2023-10-30T13:30:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f887bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f887c0bfec0 0x7f887c15ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f887bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f887c0bfec0 0x7f887c15ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55452/0 (socket says 172.21.15.46:55452) 2023-10-30T13:30:30.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f887bfff640 1 -- 172.21.15.46:0/1341799654 learned_addr learned my addr 172.21.15.46:0/1341799654 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:30.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f8880991640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f887c06a720 0x7f887c158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f8881192640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f887c159e60 0x7f887c15efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f887bfff640 1 -- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f887c06a720 msgr2=0x7f887c158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:30.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f887bfff640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f887c06a720 0x7f887c158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f887bfff640 1 -- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f887c159e60 msgr2=0x7f887c15efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f887bfff640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f887c159e60 0x7f887c15efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f887bfff640 1 -- 172.21.15.46:0/1341799654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88680094d0 con 0x7f887c0bfec0 2023-10-30T13:30:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f887bfff640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f887c0bfec0 0x7f887c15ea80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8868000c10 tx=0x7f886801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f88797fa640 1 -- 172.21.15.46:0/1341799654 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88680c1070 con 0x7f887c0bfec0 2023-10-30T13:30:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f88797fa640 1 -- 172.21.15.46:0/1341799654 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88680b0bb0 con 0x7f887c0bfec0 2023-10-30T13:30:30.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f88797fa640 1 -- 172.21.15.46:0/1341799654 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88680bb420 con 0x7f887c0bfec0 2023-10-30T13:30:30.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.804+0000 7f8882bf5640 1 -- 172.21.15.46:0/1341799654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f887c14e020 con 0x7f887c0bfec0 2023-10-30T13:30:30.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.808+0000 7f8882bf5640 1 -- 172.21.15.46:0/1341799654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887c14d880 con 0x7f887c0bfec0 2023-10-30T13:30:30.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.808+0000 7f8882bf5640 1 -- 172.21.15.46:0/1341799654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887c14d6e0 con 0x7f887c0bfec0 2023-10-30T13:30:30.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.808+0000 7f88797fa640 1 -- 172.21.15.46:0/1341799654 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f88680b7040 con 0x7f887c0bfec0 2023-10-30T13:30:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.808+0000 7f88797fa640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88600664d0 0x7f8860068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:30.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.808+0000 7f88797fa640 1 -- 172.21.15.46:0/1341799654 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f8868121880 con 0x7f887c0bfec0 2023-10-30T13:30:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.808+0000 7f8880991640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88600664d0 0x7f8860068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.812+0000 7f8880991640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88600664d0 0x7f8860068990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8870009ac0 tx=0x7f8870009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:30.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:30.812+0000 7f88797fa640 1 -- 172.21.15.46:0/1341799654 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f887c14d6e0 con 0x7f887c0bfec0 2023-10-30T13:30:31.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.008+0000 7f8882bf5640 1 -- 172.21.15.46:0/1341799654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f887c15e7a0 con 0x7f887c0bfec0 2023-10-30T13:30:31.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.008+0000 7f88797fa640 1 -- 172.21.15.46:0/1341799654 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f887c15e7a0 con 0x7f887c0bfec0 2023-10-30T13:30:31.016 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:31.017 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:28.793737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:31.017 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 38 2023-10-30T13:30:31.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.012+0000 7f8882bf5640 1 -- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88600664d0 msgr2=0x7f8860068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:31.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.012+0000 7f8882bf5640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88600664d0 0x7f8860068990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8870009ac0 tx=0x7f8870009190 comp rx=0 tx=0).stop 2023-10-30T13:30:31.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.012+0000 7f8882bf5640 1 -- 172.21.15.46:0/1341799654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f887c0bfec0 msgr2=0x7f887c15ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:31.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.012+0000 7f8882bf5640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f887c0bfec0 0x7f887c15ea80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8868000c10 tx=0x7f886801ba90 comp rx=0 tx=0).stop 2023-10-30T13:30:31.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.012+0000 7f8882bf5640 1 -- 172.21.15.46:0/1341799654 shutdown_connections 2023-10-30T13:30:31.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.012+0000 7f8882bf5640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88600664d0 0x7f8860068990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.012+0000 7f8882bf5640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f887c159e60 0x7f887c15efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.012+0000 7f8882bf5640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f887c0bfec0 0x7f887c15ea80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.012+0000 7f8882bf5640 1 --2- 172.21.15.46:0/1341799654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f887c06a720 0x7f887c158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.012+0000 7f8882bf5640 1 -- 172.21.15.46:0/1341799654 >> 172.21.15.46:0/1341799654 conn(0x7f887c0c7910 msgr2=0x7f887c0c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.012+0000 7f8882bf5640 1 -- 172.21.15.46:0/1341799654 shutdown_connections 2023-10-30T13:30:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:31.012+0000 7f8882bf5640 1 -- 172.21.15.46:0/1341799654 wait complete. 2023-10-30T13:30:31.039 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 38, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 38, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4950": { "addr": "172.21.15.46:6837/1315246492", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 1315246492, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 1315246492, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4950, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_5106": { "addr": "172.21.15.46:6833/603884510", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 603884510, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 603884510, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5106, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 45 }, "gid_5266": { "addr": "172.21.15.154:6835/651441434", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 651441434, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 651441434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5266, "incarnation": 38, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:30:28.793737+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5266, "mds_1": 5106, "mds_2": 4950 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:30:33.039 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.184+0000 7fa797ac9640 1 Processor -- start 2023-10-30T13:30:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.184+0000 7fa797ac9640 1 -- start start 2023-10-30T13:30:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa797ac9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa7901539a0 0x7fa79015a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa797ac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa790070a00 0x7fa790070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:33.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa797ac9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa790069e90 0x7fa79006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa797ac9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa79006fbb0 con 0x7fa790069e90 2023-10-30T13:30:33.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa797ac9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa79015a8c0 con 0x7fa7901539a0 2023-10-30T13:30:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa797ac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa790071380 con 0x7fa790070a00 2023-10-30T13:30:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa796066640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa790069e90 0x7fa79006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:33.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa796066640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa790069e90 0x7fa79006a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55458/0 (socket says 172.21.15.46:55458) 2023-10-30T13:30:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa796066640 1 -- 172.21.15.46:0/1364495247 learned_addr learned my addr 172.21.15.46:0/1364495247 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa795064640 1 --2- 172.21.15.46:0/1364495247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa790070a00 0x7fa790070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:33.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa795865640 1 --2- 172.21.15.46:0/1364495247 >> v2:172.21.15.154:3300/0 conn(0x7fa7901539a0 0x7fa79015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa796066640 1 -- 172.21.15.46:0/1364495247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa790070a00 msgr2=0x7fa790070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa796066640 1 --2- 172.21.15.46:0/1364495247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa790070a00 0x7fa790070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:33.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa796066640 1 -- 172.21.15.46:0/1364495247 >> v2:172.21.15.154:3300/0 conn(0x7fa7901539a0 msgr2=0x7fa79015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa796066640 1 --2- 172.21.15.46:0/1364495247 >> v2:172.21.15.154:3300/0 conn(0x7fa7901539a0 0x7fa79015a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:33.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa796066640 1 -- 172.21.15.46:0/1364495247 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa790153d80 con 0x7fa790069e90 2023-10-30T13:30:33.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.188+0000 7fa795064640 1 --2- 172.21.15.46:0/1364495247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa790070a00 0x7fa790070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa796066640 1 --2- 172.21.15.46:0/1364495247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa790069e90 0x7fa79006a300 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa7900c8950 tx=0x7fa78801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898ddf60dc538dae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa794863640 1 -- 172.21.15.46:0/1364495247 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7880a56a0 con 0x7fa790069e90 2023-10-30T13:30:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa794863640 1 -- 172.21.15.46:0/1364495247 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7880a5800 con 0x7fa790069e90 2023-10-30T13:30:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa794863640 1 -- 172.21.15.46:0/1364495247 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7880a5b90 con 0x7fa790069e90 2023-10-30T13:30:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa797ac9640 1 -- 172.21.15.46:0/1364495247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa790069e90 msgr2=0x7fa79006a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa797ac9640 1 --2- 172.21.15.46:0/1364495247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa790069e90 0x7fa79006a300 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa7900c8950 tx=0x7fa78801c910 comp rx=0 tx=0).stop 2023-10-30T13:30:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa797ac9640 1 -- 172.21.15.46:0/1364495247 shutdown_connections 2023-10-30T13:30:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa797ac9640 1 --2- 172.21.15.46:0/1364495247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa790069e90 0x7fa79006a300 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa797ac9640 1 --2- 172.21.15.46:0/1364495247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa790070a00 0x7fa790070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa797ac9640 1 --2- 172.21.15.46:0/1364495247 >> v2:172.21.15.154:3300/0 conn(0x7fa7901539a0 0x7fa79015a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa797ac9640 1 -- 172.21.15.46:0/1364495247 >> 172.21.15.46:0/1364495247 conn(0x7fa7900c7910 msgr2=0x7fa7900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa797ac9640 1 -- 172.21.15.46:0/1364495247 shutdown_connections 2023-10-30T13:30:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa797ac9640 1 -- 172.21.15.46:0/1364495247 wait complete. 2023-10-30T13:30:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa797ac9640 1 Processor -- start 2023-10-30T13:30:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.192+0000 7fa797ac9640 1 -- start start 2023-10-30T13:30:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa797ac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa790069e90 0x7fa7901f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa797ac9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa790070a00 0x7fa7901f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa797ac9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7901539a0 0x7fa7901497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa797ac9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901f3100 con 0x7fa7901539a0 2023-10-30T13:30:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa797ac9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79015bdb0 con 0x7fa790070a00 2023-10-30T13:30:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa797ac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901f0630 con 0x7fa790069e90 2023-10-30T13:30:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa796066640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7901539a0 0x7fa7901497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa796066640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7901539a0 0x7fa7901497e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55460/0 (socket says 172.21.15.46:55460) 2023-10-30T13:30:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa796066640 1 -- 172.21.15.46:0/353372595 learned_addr learned my addr 172.21.15.46:0/353372595 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa795865640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa790069e90 0x7fa7901f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa795064640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa790070a00 0x7fa7901f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa796066640 1 -- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa790069e90 msgr2=0x7fa7901f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa796066640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa790069e90 0x7fa7901f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa796066640 1 -- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa790070a00 msgr2=0x7fa7901f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa796066640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa790070a00 0x7fa7901f00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa796066640 1 -- 172.21.15.46:0/353372595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa788009b30 con 0x7fa7901539a0 2023-10-30T13:30:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa795064640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa790070a00 0x7fa7901f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa796066640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7901539a0 0x7fa7901497e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa7880af040 tx=0x7fa7880196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa7867fc640 1 -- 172.21.15.46:0/353372595 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7880b8670 con 0x7fa7901539a0 2023-10-30T13:30:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa7867fc640 1 -- 172.21.15.46:0/353372595 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7880b87f0 con 0x7fa7901539a0 2023-10-30T13:30:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa797ac9640 1 -- 172.21.15.46:0/353372595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa79015aee0 con 0x7fa7901539a0 2023-10-30T13:30:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa7867fc640 1 -- 172.21.15.46:0/353372595 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7880b8ba0 con 0x7fa7901539a0 2023-10-30T13:30:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa797ac9640 1 -- 172.21.15.46:0/353372595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa79015b480 con 0x7fa7901539a0 2023-10-30T13:30:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa797ac9640 1 -- 172.21.15.46:0/353372595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79015bf60 con 0x7fa7901539a0 2023-10-30T13:30:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.196+0000 7fa7867fc640 1 -- 172.21.15.46:0/353372595 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7880b8d00 con 0x7fa7901539a0 2023-10-30T13:30:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.200+0000 7fa7867fc640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7740664a0 0x7fa774068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.200+0000 7fa7867fc640 1 -- 172.21.15.46:0/353372595 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fa78811f500 con 0x7fa7901539a0 2023-10-30T13:30:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.200+0000 7fa795865640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7740664a0 0x7fa774068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:33.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.200+0000 7fa795865640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7740664a0 0x7fa774068960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa78c007a60 tx=0x7fa78c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:33.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.204+0000 7fa7867fc640 1 -- 172.21.15.46:0/353372595 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa79015bf60 con 0x7fa7901539a0 2023-10-30T13:30:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 -- 172.21.15.46:0/353372595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa79015bf60 con 0x7fa7901539a0 2023-10-30T13:30:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa7867fc640 1 -- 172.21.15.46:0/353372595 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fa79015bf60 con 0x7fa7901539a0 2023-10-30T13:30:33.398 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:28.793737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 38 2023-10-30T13:30:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 -- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7740664a0 msgr2=0x7fa774068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7740664a0 0x7fa774068960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa78c007a60 tx=0x7fa78c008040 comp rx=0 tx=0).stop 2023-10-30T13:30:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 -- 172.21.15.46:0/353372595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7901539a0 msgr2=0x7fa7901497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7901539a0 0x7fa7901497e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa7880af040 tx=0x7fa7880196b0 comp rx=0 tx=0).stop 2023-10-30T13:30:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 -- 172.21.15.46:0/353372595 shutdown_connections 2023-10-30T13:30:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7740664a0 0x7fa774068960 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7901539a0 0x7fa7901497e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa790070a00 0x7fa7901f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 --2- 172.21.15.46:0/353372595 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa790069e90 0x7fa7901f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 -- 172.21.15.46:0/353372595 >> 172.21.15.46:0/353372595 conn(0x7fa7900c7910 msgr2=0x7fa79014dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 -- 172.21.15.46:0/353372595 shutdown_connections 2023-10-30T13:30:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:33.392+0000 7fa797ac9640 1 -- 172.21.15.46:0/353372595 wait complete. 2023-10-30T13:30:33.419 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:35.420 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:35.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.568+0000 7f331c73e640 1 Processor -- start 2023-10-30T13:30:35.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.568+0000 7f331c73e640 1 -- start start 2023-10-30T13:30:35.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.568+0000 7f331c73e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3314153d70 0x7f331406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:35.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.568+0000 7f331c73e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331406afa0 con 0x7f331406e470 2023-10-30T13:30:35.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.568+0000 7f331c73e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3314154150 con 0x7f33140c8490 2023-10-30T13:30:35.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.568+0000 7f331c73e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f331406d780 con 0x7f3314153d70 2023-10-30T13:30:35.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.568+0000 7f3319cd9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f331406e470 0x7f33140cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46766/0 (socket says 172.21.15.46:46766) 2023-10-30T13:30:35.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.568+0000 7f3319cd9640 1 -- 172.21.15.46:0/2948416934 learned_addr learned my addr 172.21.15.46:0/2948416934 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:30:35.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.568+0000 7f331a4da640 1 --2- 172.21.15.46:0/2948416934 >> v2:172.21.15.154:3300/0 conn(0x7f3314153d70 0x7f331406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:35.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.572+0000 7f33194d8640 1 -- 172.21.15.46:0/2948416934 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982694126 0 0) 0x7f331406afa0 con 0x7f331406e470 2023-10-30T13:30:35.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.572+0000 7f33194d8640 1 -- 172.21.15.46:0/2948416934 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3300003530 con 0x7f331406e470 2023-10-30T13:30:35.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.572+0000 7f331a4da640 1 -- 172.21.15.46:0/2948416934 >> v1:172.21.15.154:6789/0 conn(0x7f33140c8490 legacy=0x7f33140c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:35.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.572+0000 7f33194d8640 1 -- 172.21.15.46:0/2948416934 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963891386 0 0) 0x7f3314154150 con 0x7f33140c8490 2023-10-30T13:30:35.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.572+0000 7f331a4da640 1 -- 172.21.15.46:0/2948416934 >> v1:172.21.15.46:6789/0 conn(0x7f331406e470 legacy=0x7f33140cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:35.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.572+0000 7f331a4da640 1 -- 172.21.15.46:0/2948416934 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331406e030 con 0x7f3314153d70 2023-10-30T13:30:35.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.572+0000 7f331a4da640 1 --2- 172.21.15.46:0/2948416934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3314153d70 0x7f331406ca80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f33100097c0 tx=0x7f331001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c252f0051ccf4843 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.572+0000 7f33194d8640 1 -- 172.21.15.46:0/2948416934 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33100ab650 con 0x7f3314153d70 2023-10-30T13:30:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.572+0000 7f33194d8640 1 -- 172.21.15.46:0/2948416934 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33100ab7b0 con 0x7f3314153d70 2023-10-30T13:30:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.572+0000 7f33194d8640 1 -- 172.21.15.46:0/2948416934 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33100abb20 con 0x7f3314153d70 2023-10-30T13:30:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 -- 172.21.15.46:0/2948416934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3314153d70 msgr2=0x7f331406ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 --2- 172.21.15.46:0/2948416934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3314153d70 0x7f331406ca80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f33100097c0 tx=0x7f331001c450 comp rx=0 tx=0).stop 2023-10-30T13:30:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 -- 172.21.15.46:0/2948416934 shutdown_connections 2023-10-30T13:30:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 --2- 172.21.15.46:0/2948416934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3314153d70 0x7f331406ca80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 -- 172.21.15.46:0/2948416934 >> 172.21.15.46:0/2948416934 conn(0x7f3314071ea0 msgr2=0x7f3314065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 -- 172.21.15.46:0/2948416934 shutdown_connections 2023-10-30T13:30:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 -- 172.21.15.46:0/2948416934 wait complete. 2023-10-30T13:30:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 Processor -- start 2023-10-30T13:30:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 -- start start 2023-10-30T13:30:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331406e470 0x7f3314142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:35.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33140c8490 0x7f3314149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3314153d70 0x7f3314147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33141f6760 con 0x7f331406e470 2023-10-30T13:30:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3314146c60 con 0x7f33140c8490 2023-10-30T13:30:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331c73e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3314146f20 con 0x7f3314153d70 2023-10-30T13:30:35.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331a4da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331406e470 0x7f3314142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331a4da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331406e470 0x7f3314142810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55466/0 (socket says 172.21.15.46:55466) 2023-10-30T13:30:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331a4da640 1 -- 172.21.15.46:0/243838293 learned_addr learned my addr 172.21.15.46:0/243838293 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331acdb640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3314153d70 0x7f3314147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f3319cd9640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33140c8490 0x7f3314149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331a4da640 1 -- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3314153d70 msgr2=0x7f3314147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331a4da640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3314153d70 0x7f3314147fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331a4da640 1 -- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33140c8490 msgr2=0x7f3314149720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331a4da640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33140c8490 0x7f3314149720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f331a4da640 1 -- 172.21.15.46:0/243838293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3310009420 con 0x7f331406e470 2023-10-30T13:30:35.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.576+0000 7f3319cd9640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33140c8490 0x7f3314149720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:30:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.580+0000 7f331a4da640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331406e470 0x7f3314142810 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f331001ab00 tx=0x7f33100191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.580+0000 7f330affd640 1 -- 172.21.15.46:0/243838293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33100044d0 con 0x7f331406e470 2023-10-30T13:30:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.580+0000 7f330affd640 1 -- 172.21.15.46:0/243838293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33100ab920 con 0x7f331406e470 2023-10-30T13:30:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.580+0000 7f331c73e640 1 -- 172.21.15.46:0/243838293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33141f5b50 con 0x7f331406e470 2023-10-30T13:30:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.580+0000 7f330affd640 1 -- 172.21.15.46:0/243838293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33100abc90 con 0x7f331406e470 2023-10-30T13:30:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.580+0000 7f331c73e640 1 -- 172.21.15.46:0/243838293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33141f6110 con 0x7f331406e470 2023-10-30T13:30:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.580+0000 7f331c73e640 1 -- 172.21.15.46:0/243838293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331404e170 con 0x7f331406e470 2023-10-30T13:30:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.580+0000 7f330affd640 1 -- 172.21.15.46:0/243838293 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3310003620 con 0x7f331406e470 2023-10-30T13:30:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.580+0000 7f330affd640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32f80664d0 0x7f32f8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.580+0000 7f330affd640 1 -- 172.21.15.46:0/243838293 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f331011e510 con 0x7f331406e470 2023-10-30T13:30:35.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.580+0000 7f3319cd9640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32f80664d0 0x7f32f8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:35.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.584+0000 7f3319cd9640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32f80664d0 0x7f32f8068990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f331414a560 tx=0x7f3304009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:35.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.584+0000 7f330affd640 1 -- 172.21.15.46:0/243838293 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33100eb710 con 0x7f331406e470 2023-10-30T13:30:35.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.776+0000 7f331c73e640 1 -- 172.21.15.46:0/243838293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3314153280 con 0x7f331406e470 2023-10-30T13:30:35.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.776+0000 7f330affd640 1 -- 172.21.15.46:0/243838293 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f33100eed30 con 0x7f331406e470 2023-10-30T13:30:35.785 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:35.785 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:28.793737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:35.785 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 38 2023-10-30T13:30:35.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.780+0000 7f331c73e640 1 -- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32f80664d0 msgr2=0x7f32f8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:35.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.780+0000 7f331c73e640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32f80664d0 0x7f32f8068990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f331414a560 tx=0x7f3304009040 comp rx=0 tx=0).stop 2023-10-30T13:30:35.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.780+0000 7f331c73e640 1 -- 172.21.15.46:0/243838293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331406e470 msgr2=0x7f3314142810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:35.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.780+0000 7f331c73e640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331406e470 0x7f3314142810 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f331001ab00 tx=0x7f33100191f0 comp rx=0 tx=0).stop 2023-10-30T13:30:35.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.780+0000 7f331c73e640 1 -- 172.21.15.46:0/243838293 shutdown_connections 2023-10-30T13:30:35.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.780+0000 7f331c73e640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32f80664d0 0x7f32f8068990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:35.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.780+0000 7f331c73e640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3314153d70 0x7f3314147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:35.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.780+0000 7f331c73e640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33140c8490 0x7f3314149720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:35.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.780+0000 7f331c73e640 1 --2- 172.21.15.46:0/243838293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331406e470 0x7f3314142810 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:35.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.780+0000 7f331c73e640 1 -- 172.21.15.46:0/243838293 >> 172.21.15.46:0/243838293 conn(0x7f3314071ea0 msgr2=0x7f331406b730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.780+0000 7f331c73e640 1 -- 172.21.15.46:0/243838293 shutdown_connections 2023-10-30T13:30:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:35.780+0000 7f331c73e640 1 -- 172.21.15.46:0/243838293 wait complete. 2023-10-30T13:30:35.806 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:37.807 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:37.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f7479640 1 Processor -- start 2023-10-30T13:30:37.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f7479640 1 -- start start 2023-10-30T13:30:37.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f7479640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8f01538a0 0x7fd8f015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:37.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f7479640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f00ca0e0 0x7fd8f015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:37.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f7479640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00cb800 con 0x7fd8f00cf160 2023-10-30T13:30:37.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f7479640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f015dc30 con 0x7fd8f01538a0 2023-10-30T13:30:37.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f7479640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f00cb330 con 0x7fd8f00ca0e0 2023-10-30T13:30:37.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f5215640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8f01538a0 0x7fd8f015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f5215640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8f01538a0 0x7fd8f015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51750/0 (socket says 172.21.15.46:51750) 2023-10-30T13:30:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f5215640 1 -- 172.21.15.46:0/302430025 learned_addr learned my addr 172.21.15.46:0/302430025 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f4a14640 1 --2- 172.21.15.46:0/302430025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f00ca0e0 0x7fd8f015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:37.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f5215640 1 -- 172.21.15.46:0/302430025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f00ca0e0 msgr2=0x7fd8f015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:37.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f5215640 1 --2- 172.21.15.46:0/302430025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f00ca0e0 0x7fd8f015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:37.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f5215640 1 -- 172.21.15.46:0/302430025 >> v1:172.21.15.154:6789/0 conn(0x7fd8f00cf160 legacy=0x7fd8f00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:37.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.956+0000 7fd8f5215640 1 -- 172.21.15.46:0/302430025 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f00cad70 con 0x7fd8f01538a0 2023-10-30T13:30:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.960+0000 7fd8f5215640 1 --2- 172.21.15.46:0/302430025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f01538a0 0x7fd8f015d6f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009870 tx=0x7fd8ec01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c721501b9bb8e6a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.960+0000 7fd8e7fff640 1 -- 172.21.15.46:0/302430025 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec0a8a30 con 0x7fd8f01538a0 2023-10-30T13:30:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.960+0000 7fd8e7fff640 1 -- 172.21.15.46:0/302430025 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec0a8b90 con 0x7fd8f01538a0 2023-10-30T13:30:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.960+0000 7fd8e7fff640 1 -- 172.21.15.46:0/302430025 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec0b0440 con 0x7fd8f01538a0 2023-10-30T13:30:37.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.960+0000 7fd8f7479640 1 -- 172.21.15.46:0/302430025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f01538a0 msgr2=0x7fd8f015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:37.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.960+0000 7fd8f7479640 1 --2- 172.21.15.46:0/302430025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f01538a0 0x7fd8f015d6f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009870 tx=0x7fd8ec01ed30 comp rx=0 tx=0).stop 2023-10-30T13:30:37.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.960+0000 7fd8f7479640 1 -- 172.21.15.46:0/302430025 shutdown_connections 2023-10-30T13:30:37.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.960+0000 7fd8f7479640 1 --2- 172.21.15.46:0/302430025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f00ca0e0 0x7fd8f015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:37.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.960+0000 7fd8f7479640 1 --2- 172.21.15.46:0/302430025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f01538a0 0x7fd8f015d6f0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:37.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.960+0000 7fd8f7479640 1 -- 172.21.15.46:0/302430025 >> 172.21.15.46:0/302430025 conn(0x7fd8f006bfe0 msgr2=0x7fd8f006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:37.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.960+0000 7fd8f7479640 1 -- 172.21.15.46:0/302430025 shutdown_connections 2023-10-30T13:30:37.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.960+0000 7fd8f7479640 1 -- 172.21.15.46:0/302430025 wait complete. 2023-10-30T13:30:37.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f7479640 1 Processor -- start 2023-10-30T13:30:37.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f7479640 1 -- start start 2023-10-30T13:30:37.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f7479640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00ca0e0 0x7fd8f0159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:37.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f7479640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f00cf160 0x7fd8f01e4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:37.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f7479640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f01538a0 0x7fd8f015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:37.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f7479640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f01e4950 con 0x7fd8f01538a0 2023-10-30T13:30:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f7479640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f01e2ae0 con 0x7fd8f00ca0e0 2023-10-30T13:30:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f7479640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f01e4060 con 0x7fd8f00cf160 2023-10-30T13:30:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f5a16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f01538a0 0x7fd8f015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f4a14640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f00cf160 0x7fd8f01e4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f5215640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00ca0e0 0x7fd8f0159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f5a16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f01538a0 0x7fd8f015b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51766/0 (socket says 172.21.15.46:51766) 2023-10-30T13:30:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f4a14640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f00cf160 0x7fd8f01e4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:34388/0 (socket says 172.21.15.46:34388) 2023-10-30T13:30:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f5a16640 1 -- 172.21.15.46:0/4135030737 learned_addr learned my addr 172.21.15.46:0/4135030737 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f5a16640 1 -- 172.21.15.46:0/4135030737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f00cf160 msgr2=0x7fd8f01e4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f5a16640 1 --2- 172.21.15.46:0/4135030737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f00cf160 0x7fd8f01e4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f5a16640 1 -- 172.21.15.46:0/4135030737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00ca0e0 msgr2=0x7fd8f0159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f5a16640 1 --2- 172.21.15.46:0/4135030737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00ca0e0 0x7fd8f0159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f5a16640 1 -- 172.21.15.46:0/4135030737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec0094d0 con 0x7fd8f01538a0 2023-10-30T13:30:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f5a16640 1 --2- 172.21.15.46:0/4135030737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f01538a0 0x7fd8f015b7d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c6c0 tx=0x7fd8e800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8e5ffb640 1 -- 172.21.15.46:0/4135030737 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8003b80 con 0x7fd8f01538a0 2023-10-30T13:30:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8e5ffb640 1 -- 172.21.15.46:0/4135030737 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e8003ce0 con 0x7fd8f01538a0 2023-10-30T13:30:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f7479640 1 -- 172.21.15.46:0/4135030737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f01e2d40 con 0x7fd8f01538a0 2023-10-30T13:30:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8e5ffb640 1 -- 172.21.15.46:0/4135030737 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80052f0 con 0x7fd8f01538a0 2023-10-30T13:30:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.964+0000 7fd8f7479640 1 -- 172.21.15.46:0/4135030737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f01e4300 con 0x7fd8f01538a0 2023-10-30T13:30:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.968+0000 7fd8f7479640 1 -- 172.21.15.46:0/4135030737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f00b2a50 con 0x7fd8f01538a0 2023-10-30T13:30:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.968+0000 7fd8e5ffb640 1 -- 172.21.15.46:0/4135030737 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd8e8005450 con 0x7fd8f01538a0 2023-10-30T13:30:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.968+0000 7fd8e5ffb640 1 --2- 172.21.15.46:0/4135030737 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8d40664d0 0x7fd8d4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.968+0000 7fd8e5ffb640 1 -- 172.21.15.46:0/4135030737 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fd8e8083d90 con 0x7fd8f01538a0 2023-10-30T13:30:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.968+0000 7fd8f5215640 1 --2- 172.21.15.46:0/4135030737 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8d40664d0 0x7fd8d4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:37.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.968+0000 7fd8f5215640 1 --2- 172.21.15.46:0/4135030737 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8d40664d0 0x7fd8d4068990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0b2000 tx=0x7fd8ec0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:37.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:37.972+0000 7fd8e5ffb640 1 -- 172.21.15.46:0/4135030737 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd8e8051030 con 0x7fd8f01538a0 2023-10-30T13:30:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 -- 172.21.15.46:0/4135030737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8f004e170 con 0x7fd8f01538a0 2023-10-30T13:30:38.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8e5ffb640 1 -- 172.21.15.46:0/4135030737 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fd8e8054650 con 0x7fd8f01538a0 2023-10-30T13:30:38.170 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:38.173 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:35.317055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:resolve","state_seq":14,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 39 2023-10-30T13:30:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 -- 172.21.15.46:0/4135030737 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8d40664d0 msgr2=0x7fd8d4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 --2- 172.21.15.46:0/4135030737 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8d40664d0 0x7fd8d4068990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0b2000 tx=0x7fd8ec0aa070 comp rx=0 tx=0).stop 2023-10-30T13:30:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 -- 172.21.15.46:0/4135030737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f01538a0 msgr2=0x7fd8f015b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 --2- 172.21.15.46:0/4135030737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f01538a0 0x7fd8f015b7d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c6c0 tx=0x7fd8e800cb80 comp rx=0 tx=0).stop 2023-10-30T13:30:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 -- 172.21.15.46:0/4135030737 shutdown_connections 2023-10-30T13:30:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 --2- 172.21.15.46:0/4135030737 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8d40664d0 0x7fd8d4068990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 --2- 172.21.15.46:0/4135030737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f01538a0 0x7fd8f015b7d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 --2- 172.21.15.46:0/4135030737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f00cf160 0x7fd8f01e4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 --2- 172.21.15.46:0/4135030737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00ca0e0 0x7fd8f0159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 -- 172.21.15.46:0/4135030737 >> 172.21.15.46:0/4135030737 conn(0x7fd8f006bfe0 msgr2=0x7fd8f0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 -- 172.21.15.46:0/4135030737 shutdown_connections 2023-10-30T13:30:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:38.164+0000 7fd8f7479640 1 -- 172.21.15.46:0/4135030737 wait complete. 2023-10-30T13:30:38.191 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:40.192 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e8d65640 1 Processor -- start 2023-10-30T13:30:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e8d65640 1 -- start start 2023-10-30T13:30:40.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e8d65640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd3e41538a0 0x7fd3e415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:40.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e8d65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3e40cf160 0x7fd3e40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:40.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e8d65640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e40cb800 con 0x7fd3e40ca0e0 2023-10-30T13:30:40.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e8d65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e415dc30 con 0x7fd3e40cf160 2023-10-30T13:30:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e8d65640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e40cb330 con 0x7fd3e41538a0 2023-10-30T13:30:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e1d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd3e40ca0e0 0x7fd3e415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49936/0 (socket says 172.21.15.46:49936) 2023-10-30T13:30:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e1d9b640 1 -- 172.21.15.46:0/2860556903 learned_addr learned my addr 172.21.15.46:0/2860556903 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:30:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e2d9d640 1 --2- 172.21.15.46:0/2860556903 >> v2:172.21.15.46:3300/0 conn(0x7fd3e40cf160 0x7fd3e40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:40.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e259c640 1 --2- 172.21.15.46:0/2860556903 >> v2:172.21.15.154:3300/0 conn(0x7fd3e41538a0 0x7fd3e415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:40.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e2d9d640 1 -- 172.21.15.46:0/2860556903 >> v2:172.21.15.154:3300/0 conn(0x7fd3e41538a0 msgr2=0x7fd3e415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:40.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e2d9d640 1 --2- 172.21.15.46:0/2860556903 >> v2:172.21.15.154:3300/0 conn(0x7fd3e41538a0 0x7fd3e415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:40.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e2d9d640 1 -- 172.21.15.46:0/2860556903 >> v1:172.21.15.46:6789/0 conn(0x7fd3e40ca0e0 legacy=0x7fd3e415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:40.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e2d9d640 1 -- 172.21.15.46:0/2860556903 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e40cad70 con 0x7fd3e40cf160 2023-10-30T13:30:40.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.340+0000 7fd3e159a640 1 -- 172.21.15.46:0/2860556903 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511685559 0 0) 0x7fd3e40cb800 con 0x7fd3e40ca0e0 2023-10-30T13:30:40.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.344+0000 7fd3e2d9d640 1 --2- 172.21.15.46:0/2860556903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3e40cf160 0x7fd3e40cf590 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd3e40cfd10 tx=0x7fd3dc01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92dd75edc789930 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:40.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.344+0000 7fd3e159a640 1 -- 172.21.15.46:0/2860556903 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0b0440 con 0x7fd3e40cf160 2023-10-30T13:30:40.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.344+0000 7fd3e159a640 1 -- 172.21.15.46:0/2860556903 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3dc0b05a0 con 0x7fd3e40cf160 2023-10-30T13:30:40.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.344+0000 7fd3e159a640 1 -- 172.21.15.46:0/2860556903 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0b0930 con 0x7fd3e40cf160 2023-10-30T13:30:40.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.344+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2860556903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3e40cf160 msgr2=0x7fd3e40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:40.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.344+0000 7fd3e8d65640 1 --2- 172.21.15.46:0/2860556903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3e40cf160 0x7fd3e40cf590 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd3e40cfd10 tx=0x7fd3dc01f0f0 comp rx=0 tx=0).stop 2023-10-30T13:30:40.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.344+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2860556903 shutdown_connections 2023-10-30T13:30:40.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.344+0000 7fd3e8d65640 1 --2- 172.21.15.46:0/2860556903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3e40cf160 0x7fd3e40cf590 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:40.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.344+0000 7fd3e8d65640 1 --2- 172.21.15.46:0/2860556903 >> v2:172.21.15.154:3300/0 conn(0x7fd3e41538a0 0x7fd3e415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:40.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.344+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2860556903 >> 172.21.15.46:0/2860556903 conn(0x7fd3e406bfe0 msgr2=0x7fd3e406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:40.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.344+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2860556903 shutdown_connections 2023-10-30T13:30:40.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.344+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2860556903 wait complete. 2023-10-30T13:30:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e8d65640 1 Processor -- start 2023-10-30T13:30:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e8d65640 1 -- start start 2023-10-30T13:30:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e8d65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40ca0e0 0x7fd3e40cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e8d65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3e40cf160 0x7fd3e41e3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:40.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e8d65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e41538a0 0x7fd3e41e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:40.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e8d65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e41e31d0 con 0x7fd3e40cf160 2023-10-30T13:30:40.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e8d65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e4148450 con 0x7fd3e41538a0 2023-10-30T13:30:40.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e8d65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e4148730 con 0x7fd3e40ca0e0 2023-10-30T13:30:40.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3e40cf160 0x7fd3e41e3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3e40cf160 0x7fd3e41e3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51772/0 (socket says 172.21.15.46:51772) 2023-10-30T13:30:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e1d9b640 1 -- 172.21.15.46:0/2322987708 learned_addr learned my addr 172.21.15.46:0/2322987708 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e2d9d640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e41538a0 0x7fd3e41e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e259c640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40ca0e0 0x7fd3e40cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e1d9b640 1 -- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40ca0e0 msgr2=0x7fd3e40cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:40.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e1d9b640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40ca0e0 0x7fd3e40cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:40.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e1d9b640 1 -- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e41538a0 msgr2=0x7fd3e41e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:40.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e1d9b640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e41538a0 0x7fd3e41e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:40.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e1d9b640 1 -- 172.21.15.46:0/2322987708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc009a70 con 0x7fd3e40cf160 2023-10-30T13:30:40.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e1d9b640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3e40cf160 0x7fd3e41e3660 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800c480 tx=0x7fd3d800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:40.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3d2ffd640 1 -- 172.21.15.46:0/2322987708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d80074e0 con 0x7fd3e40cf160 2023-10-30T13:30:40.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3d2ffd640 1 -- 172.21.15.46:0/2322987708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d8007660 con 0x7fd3e40cf160 2023-10-30T13:30:40.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2322987708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e41e2b80 con 0x7fd3e40cf160 2023-10-30T13:30:40.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3d2ffd640 1 -- 172.21.15.46:0/2322987708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8004a30 con 0x7fd3e40cf160 2023-10-30T13:30:40.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.348+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2322987708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e41e57a0 con 0x7fd3e40cf160 2023-10-30T13:30:40.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.352+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2322987708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e40b2a50 con 0x7fd3e40cf160 2023-10-30T13:30:40.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.352+0000 7fd3d2ffd640 1 -- 172.21.15.46:0/2322987708 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd3d8004b90 con 0x7fd3e40cf160 2023-10-30T13:30:40.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.352+0000 7fd3d2ffd640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3c40664a0 0x7fd3c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:40.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.352+0000 7fd3d2ffd640 1 -- 172.21.15.46:0/2322987708 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fd3d8083560 con 0x7fd3e40cf160 2023-10-30T13:30:40.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.352+0000 7fd3e259c640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3c40664a0 0x7fd3c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:40.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.356+0000 7fd3e259c640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3c40664a0 0x7fd3c4068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd3d400b9d0 tx=0x7fd3d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:40.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.356+0000 7fd3d2ffd640 1 -- 172.21.15.46:0/2322987708 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd3d8054320 con 0x7fd3e40cf160 2023-10-30T13:30:40.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.544+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2322987708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3e4159850 con 0x7fd3e40cf160 2023-10-30T13:30:40.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3d2ffd640 1 -- 172.21.15.46:0/2322987708 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fd3d8053eb0 con 0x7fd3e40cf160 2023-10-30T13:30:40.553 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:40.554 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:35.317055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:resolve","state_seq":14,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:40.554 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 39 2023-10-30T13:30:40.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3c40664a0 msgr2=0x7fd3c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:40.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3e8d65640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3c40664a0 0x7fd3c4068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd3d400b9d0 tx=0x7fd3d4008040 comp rx=0 tx=0).stop 2023-10-30T13:30:40.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2322987708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3e40cf160 msgr2=0x7fd3e41e3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:40.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3e8d65640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3e40cf160 0x7fd3e41e3660 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800c480 tx=0x7fd3d800c940 comp rx=0 tx=0).stop 2023-10-30T13:30:40.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2322987708 shutdown_connections 2023-10-30T13:30:40.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3e8d65640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3c40664a0 0x7fd3c4068960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:40.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3e8d65640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e41538a0 0x7fd3e41e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:40.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3e8d65640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3e40cf160 0x7fd3e41e3660 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:40.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3e8d65640 1 --2- 172.21.15.46:0/2322987708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40ca0e0 0x7fd3e40cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:40.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2322987708 >> 172.21.15.46:0/2322987708 conn(0x7fd3e406bfe0 msgr2=0x7fd3e4066390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:40.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2322987708 shutdown_connections 2023-10-30T13:30:40.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:40.548+0000 7fd3e8d65640 1 -- 172.21.15.46:0/2322987708 wait complete. 2023-10-30T13:30:40.575 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:42.576 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe46d0f640 1 Processor -- start 2023-10-30T13:30:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe46d0f640 1 -- start start 2023-10-30T13:30:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe46d0f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbe401538a0 0x7fbe4015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe46d0f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe400cb800 con 0x7fbe400cf160 2023-10-30T13:30:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe46d0f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4015dc30 con 0x7fbe400ca0e0 2023-10-30T13:30:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe46d0f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe400cb330 con 0x7fbe401538a0 2023-10-30T13:30:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe452ac640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbe400cf160 0x7fbe40164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49944/0 (socket says 172.21.15.46:49944) 2023-10-30T13:30:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe452ac640 1 -- 172.21.15.46:0/4219340482 learned_addr learned my addr 172.21.15.46:0/4219340482 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:30:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe44aab640 1 --2- 172.21.15.46:0/4219340482 >> v2:172.21.15.154:3300/0 conn(0x7fbe401538a0 0x7fbe4015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe3f7fe640 1 -- 172.21.15.46:0/4219340482 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765442752 0 0) 0x7fbe400cb800 con 0x7fbe400cf160 2023-10-30T13:30:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe3f7fe640 1 -- 172.21.15.46:0/4219340482 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe30003530 con 0x7fbe400cf160 2023-10-30T13:30:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe3f7fe640 1 -- 172.21.15.46:0/4219340482 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2498428997 0 0) 0x7fbe30003530 con 0x7fbe400cf160 2023-10-30T13:30:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe44aab640 1 -- 172.21.15.46:0/4219340482 >> v1:172.21.15.154:6789/0 conn(0x7fbe400ca0e0 legacy=0x7fbe4015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.724+0000 7fbe44aab640 1 -- 172.21.15.46:0/4219340482 >> v1:172.21.15.46:6789/0 conn(0x7fbe400cf160 legacy=0x7fbe40164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.728+0000 7fbe44aab640 1 -- 172.21.15.46:0/4219340482 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe400cad70 con 0x7fbe401538a0 2023-10-30T13:30:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.728+0000 7fbe44aab640 1 --2- 172.21.15.46:0/4219340482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe401538a0 0x7fbe4015d6f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbe340097c0 tx=0x7fbe3401eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=343391e57a921ab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.728+0000 7fbe3f7fe640 1 -- 172.21.15.46:0/4219340482 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe38001a00 con 0x7fbe401538a0 2023-10-30T13:30:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.728+0000 7fbe3f7fe640 1 -- 172.21.15.46:0/4219340482 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe340a8610 con 0x7fbe401538a0 2023-10-30T13:30:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.728+0000 7fbe3f7fe640 1 -- 172.21.15.46:0/4219340482 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe340a8980 con 0x7fbe401538a0 2023-10-30T13:30:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.728+0000 7fbe46d0f640 1 -- 172.21.15.46:0/4219340482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe401538a0 msgr2=0x7fbe4015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.728+0000 7fbe46d0f640 1 --2- 172.21.15.46:0/4219340482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe401538a0 0x7fbe4015d6f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbe340097c0 tx=0x7fbe3401eb90 comp rx=0 tx=0).stop 2023-10-30T13:30:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 -- 172.21.15.46:0/4219340482 shutdown_connections 2023-10-30T13:30:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 --2- 172.21.15.46:0/4219340482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe401538a0 0x7fbe4015d6f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 -- 172.21.15.46:0/4219340482 >> 172.21.15.46:0/4219340482 conn(0x7fbe4006bfe0 msgr2=0x7fbe4006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 -- 172.21.15.46:0/4219340482 shutdown_connections 2023-10-30T13:30:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 -- 172.21.15.46:0/4219340482 wait complete. 2023-10-30T13:30:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 Processor -- start 2023-10-30T13:30:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 -- start start 2023-10-30T13:30:42.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe400ca0e0 0x7fbe400b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe400cf160 0x7fbe40148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe401538a0 0x7fbe400cd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe400ce960 con 0x7fbe400cf160 2023-10-30T13:30:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe40158420 con 0x7fbe400ca0e0 2023-10-30T13:30:42.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe46d0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe400b41e0 con 0x7fbe401538a0 2023-10-30T13:30:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe3ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe400cf160 0x7fbe40148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe3ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe400cf160 0x7fbe40148ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51782/0 (socket says 172.21.15.46:51782) 2023-10-30T13:30:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe3ffff640 1 -- 172.21.15.46:0/1805918108 learned_addr learned my addr 172.21.15.46:0/1805918108 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe44aab640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe400ca0e0 0x7fbe400b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:42.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe452ac640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe401538a0 0x7fbe400cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe3ffff640 1 -- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe401538a0 msgr2=0x7fbe400cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe3ffff640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe401538a0 0x7fbe400cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe3ffff640 1 -- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe400ca0e0 msgr2=0x7fbe400b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe3ffff640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe400ca0e0 0x7fbe400b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe3ffff640 1 -- 172.21.15.46:0/1805918108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe34009420 con 0x7fbe400cf160 2023-10-30T13:30:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe452ac640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe401538a0 0x7fbe400cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe44aab640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe400ca0e0 0x7fbe400b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.732+0000 7fbe3ffff640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe400cf160 0x7fbe40148ed0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00c3c0 tx=0x7fbe2c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.736+0000 7fbe3d7fa640 1 -- 172.21.15.46:0/1805918108 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c0052d0 con 0x7fbe400cf160 2023-10-30T13:30:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.736+0000 7fbe3d7fa640 1 -- 172.21.15.46:0/1805918108 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2c005430 con 0x7fbe400cf160 2023-10-30T13:30:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.736+0000 7fbe3d7fa640 1 -- 172.21.15.46:0/1805918108 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c003840 con 0x7fbe400cf160 2023-10-30T13:30:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.736+0000 7fbe46d0f640 1 -- 172.21.15.46:0/1805918108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe400b37e0 con 0x7fbe400cf160 2023-10-30T13:30:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.736+0000 7fbe46d0f640 1 -- 172.21.15.46:0/1805918108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4015d890 con 0x7fbe400cf160 2023-10-30T13:30:42.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.736+0000 7fbe46d0f640 1 -- 172.21.15.46:0/1805918108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4004e170 con 0x7fbe400cf160 2023-10-30T13:30:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.736+0000 7fbe3d7fa640 1 -- 172.21.15.46:0/1805918108 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbe2c0039a0 con 0x7fbe400cf160 2023-10-30T13:30:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.736+0000 7fbe3d7fa640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe240664a0 0x7fbe24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:42.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.736+0000 7fbe3d7fa640 1 -- 172.21.15.46:0/1805918108 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fbe2c0834c0 con 0x7fbe400cf160 2023-10-30T13:30:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.736+0000 7fbe44aab640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe240664a0 0x7fbe24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.740+0000 7fbe44aab640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe240664a0 0x7fbe24068960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbe340b2000 tx=0x7fbe340aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:42.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.740+0000 7fbe3d7fa640 1 -- 172.21.15.46:0/1805918108 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbe2c0506b0 con 0x7fbe400cf160 2023-10-30T13:30:42.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 -- 172.21.15.46:0/1805918108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe4015dc30 con 0x7fbe400cf160 2023-10-30T13:30:42.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe3d7fa640 1 -- 172.21.15.46:0/1805918108 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5744 (secure 0 0 0) 0x7fbe4015dc30 con 0x7fbe400cf160 2023-10-30T13:30:42.939 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:42.939 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:41.026756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:42.939 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 40 2023-10-30T13:30:42.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 -- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe240664a0 msgr2=0x7fbe24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:42.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe240664a0 0x7fbe24068960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbe340b2000 tx=0x7fbe340aa070 comp rx=0 tx=0).stop 2023-10-30T13:30:42.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 -- 172.21.15.46:0/1805918108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe400cf160 msgr2=0x7fbe40148ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:42.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe400cf160 0x7fbe40148ed0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00c3c0 tx=0x7fbe2c00c880 comp rx=0 tx=0).stop 2023-10-30T13:30:42.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 -- 172.21.15.46:0/1805918108 shutdown_connections 2023-10-30T13:30:42.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe240664a0 0x7fbe24068960 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe401538a0 0x7fbe400cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe400cf160 0x7fbe40148ed0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 --2- 172.21.15.46:0/1805918108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe400ca0e0 0x7fbe400b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 -- 172.21.15.46:0/1805918108 >> 172.21.15.46:0/1805918108 conn(0x7fbe4006bfe0 msgr2=0x7fbe401516c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 -- 172.21.15.46:0/1805918108 shutdown_connections 2023-10-30T13:30:42.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:42.932+0000 7fbe46d0f640 1 -- 172.21.15.46:0/1805918108 wait complete. 2023-10-30T13:30:42.960 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 40, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 40, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4950": { "addr": "172.21.15.46:6837/1315246492", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 1315246492, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 1315246492, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4950, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_5106": { "addr": "172.21.15.46:6833/603884510", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 603884510, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 603884510, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5106, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 45 }, "gid_5266": { "addr": "172.21.15.154:6835/651441434", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 651441434, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 651441434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5266, "incarnation": 38, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:30:41.026756+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5266, "mds_1": 5106, "mds_2": 4950 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:30:44.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:45.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f048d846640 1 Processor -- start 2023-10-30T13:30:45.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f048d846640 1 -- start start 2023-10-30T13:30:45.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f048d846640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f048815d6f0 0x7f048815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:45.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f048d846640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f04880cf160 0x7f04880cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:45.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f048d846640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04880cb800 con 0x7f04881538a0 2023-10-30T13:30:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f048d846640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04880cb330 con 0x7f048815d6f0 2023-10-30T13:30:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f048d846640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04880cad70 con 0x7f04880cf160 2023-10-30T13:30:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f047ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f048815d6f0 0x7f048815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f047ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f048815d6f0 0x7f048815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51790/0 (socket says 172.21.15.46:51790) 2023-10-30T13:30:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f047ffff640 1 -- 172.21.15.46:0/2296841263 learned_addr learned my addr 172.21.15.46:0/2296841263 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f04877fe640 1 --2- 172.21.15.46:0/2296841263 >> v2:172.21.15.154:3300/0 conn(0x7f04880cf160 0x7f04880cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f047ffff640 1 -- 172.21.15.46:0/2296841263 >> v2:172.21.15.154:3300/0 conn(0x7f04880cf160 msgr2=0x7f04880cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f047ffff640 1 --2- 172.21.15.46:0/2296841263 >> v2:172.21.15.154:3300/0 conn(0x7f04880cf160 0x7f04880cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f047ffff640 1 -- 172.21.15.46:0/2296841263 >> v1:172.21.15.154:6789/0 conn(0x7f04881538a0 legacy=0x7f04880ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.112+0000 7f047ffff640 1 -- 172.21.15.46:0/2296841263 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04880cb050 con 0x7f048815d6f0 2023-10-30T13:30:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.116+0000 7f047ffff640 1 --2- 172.21.15.46:0/2296841263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048815d6f0 0x7f048815a090 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f04880b5330 tx=0x7f047001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c512ba5b8cefbdf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.116+0000 7f04867fc640 1 -- 172.21.15.46:0/2296841263 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04700a89f0 con 0x7f048815d6f0 2023-10-30T13:30:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.116+0000 7f04867fc640 1 -- 172.21.15.46:0/2296841263 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04700a8b50 con 0x7f048815d6f0 2023-10-30T13:30:45.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.116+0000 7f04867fc640 1 -- 172.21.15.46:0/2296841263 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04700b0440 con 0x7f048815d6f0 2023-10-30T13:30:45.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.116+0000 7f048d846640 1 -- 172.21.15.46:0/2296841263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048815d6f0 msgr2=0x7f048815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:45.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.116+0000 7f048d846640 1 --2- 172.21.15.46:0/2296841263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048815d6f0 0x7f048815a090 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f04880b5330 tx=0x7f047001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:30:45.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.116+0000 7f048d846640 1 -- 172.21.15.46:0/2296841263 shutdown_connections 2023-10-30T13:30:45.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.116+0000 7f048d846640 1 --2- 172.21.15.46:0/2296841263 >> v2:172.21.15.154:3300/0 conn(0x7f04880cf160 0x7f04880cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:45.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.116+0000 7f048d846640 1 --2- 172.21.15.46:0/2296841263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048815d6f0 0x7f048815a090 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:45.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.116+0000 7f048d846640 1 -- 172.21.15.46:0/2296841263 >> 172.21.15.46:0/2296841263 conn(0x7f048806bfe0 msgr2=0x7f048806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:45.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.116+0000 7f048d846640 1 -- 172.21.15.46:0/2296841263 shutdown_connections 2023-10-30T13:30:45.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.116+0000 7f048d846640 1 -- 172.21.15.46:0/2296841263 wait complete. 2023-10-30T13:30:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f048d846640 1 Processor -- start 2023-10-30T13:30:45.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f048d846640 1 -- start start 2023-10-30T13:30:45.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f048d846640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04880cf160 0x7f04881e3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:45.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f048d846640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04881538a0 0x7f04880b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:45.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f048d846640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048815d6f0 0x7f04880b4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f048d846640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048814e2d0 con 0x7f048815d6f0 2023-10-30T13:30:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f048d846640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04880b33b0 con 0x7f04880cf160 2023-10-30T13:30:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f048d846640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04880cd9e0 con 0x7f04881538a0 2023-10-30T13:30:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f04877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048815d6f0 0x7f04880b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f04877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048815d6f0 0x7f04880b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51804/0 (socket says 172.21.15.46:51804) 2023-10-30T13:30:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f04877fe640 1 -- 172.21.15.46:0/2728860727 learned_addr learned my addr 172.21.15.46:0/2728860727 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f047ffff640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04881538a0 0x7f04880b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f0486ffd640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04880cf160 0x7f04881e3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f04877fe640 1 -- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04881538a0 msgr2=0x7f04880b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:45.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f04877fe640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04881538a0 0x7f04880b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:45.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f04877fe640 1 -- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04880cf160 msgr2=0x7f04881e3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:45.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f04877fe640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04880cf160 0x7f04881e3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:45.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f04877fe640 1 -- 172.21.15.46:0/2728860727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04700094d0 con 0x7f048815d6f0 2023-10-30T13:30:45.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f047ffff640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04881538a0 0x7f04880b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f0486ffd640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04880cf160 0x7f04881e3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f04877fe640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048815d6f0 0x7f04880b4720 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0480007dc0 tx=0x7f048000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:45.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f0484ff9640 1 -- 172.21.15.46:0/2728860727 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0480014420 con 0x7f048815d6f0 2023-10-30T13:30:45.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f0484ff9640 1 -- 172.21.15.46:0/2728860727 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04800145a0 con 0x7f048815d6f0 2023-10-30T13:30:45.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f0484ff9640 1 -- 172.21.15.46:0/2728860727 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0480014950 con 0x7f048815d6f0 2023-10-30T13:30:45.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f048d846640 1 -- 172.21.15.46:0/2728860727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04880ceaa0 con 0x7f048815d6f0 2023-10-30T13:30:45.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.120+0000 7f048d846640 1 -- 172.21.15.46:0/2728860727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04880cdd80 con 0x7f048815d6f0 2023-10-30T13:30:45.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.124+0000 7f048d846640 1 -- 172.21.15.46:0/2728860727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04880b2a50 con 0x7f048815d6f0 2023-10-30T13:30:45.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.124+0000 7f0484ff9640 1 -- 172.21.15.46:0/2728860727 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0480014ab0 con 0x7f048815d6f0 2023-10-30T13:30:45.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.124+0000 7f0484ff9640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04680664a0 0x7f0468068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:45.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.124+0000 7f0484ff9640 1 -- 172.21.15.46:0/2728860727 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f0480084a20 con 0x7f048815d6f0 2023-10-30T13:30:45.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.124+0000 7f0486ffd640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04680664a0 0x7f0468068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:45.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.124+0000 7f0486ffd640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04680664a0 0x7f0468068960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0478000f70 tx=0x7f0478009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:45.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.128+0000 7f0484ff9640 1 -- 172.21.15.46:0/2728860727 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0480051c20 con 0x7f048815d6f0 2023-10-30T13:30:45.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.316+0000 7f048d846640 1 -- 172.21.15.46:0/2728860727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04880cec30 con 0x7f048815d6f0 2023-10-30T13:30:45.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.316+0000 7f0484ff9640 1 -- 172.21.15.46:0/2728860727 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f0480055240 con 0x7f048815d6f0 2023-10-30T13:30:45.325 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:45.326 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:41.026756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 40 2023-10-30T13:30:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.320+0000 7f048d846640 1 -- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04680664a0 msgr2=0x7f0468068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:45.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.320+0000 7f048d846640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04680664a0 0x7f0468068960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0478000f70 tx=0x7f0478009040 comp rx=0 tx=0).stop 2023-10-30T13:30:45.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.320+0000 7f048d846640 1 -- 172.21.15.46:0/2728860727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048815d6f0 msgr2=0x7f04880b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:45.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.320+0000 7f048d846640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048815d6f0 0x7f04880b4720 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0480007dc0 tx=0x7f048000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:30:45.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.320+0000 7f048d846640 1 -- 172.21.15.46:0/2728860727 shutdown_connections 2023-10-30T13:30:45.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.320+0000 7f048d846640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04680664a0 0x7f0468068960 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:45.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.320+0000 7f048d846640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048815d6f0 0x7f04880b4720 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:45.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.320+0000 7f048d846640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04881538a0 0x7f04880b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:45.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.320+0000 7f048d846640 1 --2- 172.21.15.46:0/2728860727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04880cf160 0x7f04881e3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:45.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.320+0000 7f048d846640 1 -- 172.21.15.46:0/2728860727 >> 172.21.15.46:0/2728860727 conn(0x7f048806bfe0 msgr2=0x7f04881e5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:45.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.320+0000 7f048d846640 1 -- 172.21.15.46:0/2728860727 shutdown_connections 2023-10-30T13:30:45.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:45.320+0000 7f048d846640 1 -- 172.21.15.46:0/2728860727 wait complete. 2023-10-30T13:30:45.347 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:45.906 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:30:45.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:30:47.348 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.496+0000 7f193f46b640 1 Processor -- start 2023-10-30T13:30:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.496+0000 7f193f46b640 1 -- start start 2023-10-30T13:30:47.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.496+0000 7f193f46b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f19380c6f60 0x7f19381566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:47.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.496+0000 7f193f46b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19380c0a50 con 0x7f19380c6b80 2023-10-30T13:30:47.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.496+0000 7f193f46b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193806f6e0 con 0x7f19380c05d0 2023-10-30T13:30:47.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.496+0000 7f193f46b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193806f9c0 con 0x7f19380c6f60 2023-10-30T13:30:47.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.496+0000 7f193d207640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f19380c6b80 0x7f19380bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49092/0 (socket says 172.21.15.46:49092) 2023-10-30T13:30:47.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.496+0000 7f193d207640 1 -- 172.21.15.46:0/14713165 learned_addr learned my addr 172.21.15.46:0/14713165 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:30:47.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.496+0000 7f193da08640 1 --2- 172.21.15.46:0/14713165 >> v2:172.21.15.154:3300/0 conn(0x7f19380c6f60 0x7f19381566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:47.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.496+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727414654 0 0) 0x7f19380c0a50 con 0x7f19380c6b80 2023-10-30T13:30:47.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.496+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1924005740 con 0x7f19380c6b80 2023-10-30T13:30:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3935064638 0 0) 0x7f1924005740 con 0x7f19380c6b80 2023-10-30T13:30:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 >> v2:172.21.15.154:3300/0 conn(0x7f19380c6f60 msgr2=0x7f19381566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 --2- 172.21.15.46:0/14713165 >> v2:172.21.15.154:3300/0 conn(0x7f19380c6f60 0x7f19381566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 >> v1:172.21.15.154:6789/0 conn(0x7f19380c05d0 legacy=0x7f193813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193806fca0 con 0x7f19380c6b80 2023-10-30T13:30:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f1934001680 con 0x7f19380c6b80 2023-10-30T13:30:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 >> v1:172.21.15.46:6789/0 conn(0x7f19380c6b80 legacy=0x7f19380bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 --2- 172.21.15.46:0/14713165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1924005b90 0x7f1924005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 --2- 172.21.15.46:0/14713165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 0x7f1924006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 --2- 172.21.15.46:0/14713165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19240073a0 0x7f192400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193806f9c0 con 0x7f19240069e0 2023-10-30T13:30:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192400e240 con 0x7f19240073a0 2023-10-30T13:30:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192400e400 con 0x7f1924005b90 2023-10-30T13:30:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f193ca06640 1 --2- 172.21.15.46:0/14713165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 0x7f1924006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f193da08640 1 --2- 172.21.15.46:0/14713165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19240073a0 0x7f192400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f193d207640 1 --2- 172.21.15.46:0/14713165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1924005b90 0x7f1924005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f193ca06640 1 -- 172.21.15.46:0/14713165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1924005b90 msgr2=0x7f1924005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f193ca06640 1 --2- 172.21.15.46:0/14713165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1924005b90 0x7f1924005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f193ca06640 1 -- 172.21.15.46:0/14713165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19240073a0 msgr2=0x7f192400dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f193ca06640 1 --2- 172.21.15.46:0/14713165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19240073a0 0x7f192400dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f193ca06640 1 -- 172.21.15.46:0/14713165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1924005740 con 0x7f19240069e0 2023-10-30T13:30:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.500+0000 7f193da08640 1 -- 172.21.15.46:0/14713165 reap_dead start 2023-10-30T13:30:47.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193ca06640 1 --2- 172.21.15.46:0/14713165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 0x7f1924006e60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1928001750 tx=0x7f192801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602603fdbf098aa5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19280052d0 con 0x7f19240069e0 2023-10-30T13:30:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19280a7050 con 0x7f19240069e0 2023-10-30T13:30:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f192ffff640 1 -- 172.21.15.46:0/14713165 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1928003840 con 0x7f19240069e0 2023-10-30T13:30:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 -- 172.21.15.46:0/14713165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 msgr2=0x7f1924006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 --2- 172.21.15.46:0/14713165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 0x7f1924006e60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1928001750 tx=0x7f192801c3a0 comp rx=0 tx=0).stop 2023-10-30T13:30:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 -- 172.21.15.46:0/14713165 shutdown_connections 2023-10-30T13:30:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 --2- 172.21.15.46:0/14713165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 0x7f1924006e60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 -- 172.21.15.46:0/14713165 >> 172.21.15.46:0/14713165 conn(0x7f19380c7910 msgr2=0x7f19380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 -- 172.21.15.46:0/14713165 shutdown_connections 2023-10-30T13:30:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 -- 172.21.15.46:0/14713165 wait complete. 2023-10-30T13:30:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 Processor -- start 2023-10-30T13:30:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 -- start start 2023-10-30T13:30:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 0x7f19381520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19380aea80 0x7f19380ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19380b0b50 0x7f193813dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19381ef4c0 con 0x7f19240069e0 2023-10-30T13:30:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1938152610 con 0x7f19380aea80 2023-10-30T13:30:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.504+0000 7f193f46b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19380c0420 con 0x7f19380b0b50 2023-10-30T13:30:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193d207640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 0x7f19381520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193d207640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 0x7f19381520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44322/0 (socket says 172.21.15.46:44322) 2023-10-30T13:30:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193d207640 1 -- 172.21.15.46:0/2200587415 learned_addr learned my addr 172.21.15.46:0/2200587415 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193da08640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19380b0b50 0x7f193813dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193ca06640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19380aea80 0x7f19380ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193d207640 1 -- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19380b0b50 msgr2=0x7f193813dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193d207640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19380b0b50 0x7f193813dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193d207640 1 -- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19380aea80 msgr2=0x7f19380ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193d207640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19380aea80 0x7f19380ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193d207640 1 -- 172.21.15.46:0/2200587415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19381ef9b0 con 0x7f19240069e0 2023-10-30T13:30:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193da08640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19380b0b50 0x7f193813dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193ca06640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19380aea80 0x7f19380ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193d207640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 0x7f19381520d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f19340124d0 tx=0x7f1934012990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f192dffb640 1 -- 172.21.15.46:0/2200587415 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19340148d0 con 0x7f19240069e0 2023-10-30T13:30:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f192dffb640 1 -- 172.21.15.46:0/2200587415 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1934014a30 con 0x7f19240069e0 2023-10-30T13:30:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f192dffb640 1 -- 172.21.15.46:0/2200587415 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934014dc0 con 0x7f19240069e0 2023-10-30T13:30:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193f46b640 1 -- 172.21.15.46:0/2200587415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1938151630 con 0x7f19240069e0 2023-10-30T13:30:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193f46b640 1 -- 172.21.15.46:0/2200587415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19381518d0 con 0x7f19240069e0 2023-10-30T13:30:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f193f46b640 1 -- 172.21.15.46:0/2200587415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19380b0f30 con 0x7f19240069e0 2023-10-30T13:30:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f192dffb640 1 -- 172.21.15.46:0/2200587415 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f19340153e0 con 0x7f19240069e0 2023-10-30T13:30:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.508+0000 7f192dffb640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f191c0664a0 0x7f191c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.512+0000 7f192dffb640 1 -- 172.21.15.46:0/2200587415 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f1934086940 con 0x7f19240069e0 2023-10-30T13:30:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.512+0000 7f193ca06640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f191c0664a0 0x7f191c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.512+0000 7f193ca06640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f191c0664a0 0x7f191c068960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f193806c9a0 tx=0x7f19280011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.512+0000 7f192dffb640 1 -- 172.21.15.46:0/2200587415 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f19380b0f30 con 0x7f19240069e0 2023-10-30T13:30:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.700+0000 7f193f46b640 1 -- 172.21.15.46:0/2200587415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19380b0f30 con 0x7f19240069e0 2023-10-30T13:30:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f192dffb640 1 -- 172.21.15.46:0/2200587415 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f19380b0f30 con 0x7f19240069e0 2023-10-30T13:30:47.710 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:47.710 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:47.037951+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 41 2023-10-30T13:30:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f193f46b640 1 -- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f191c0664a0 msgr2=0x7f191c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f193f46b640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f191c0664a0 0x7f191c068960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f193806c9a0 tx=0x7f19280011a0 comp rx=0 tx=0).stop 2023-10-30T13:30:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f193f46b640 1 -- 172.21.15.46:0/2200587415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 msgr2=0x7f19381520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f193f46b640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 0x7f19381520d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f19340124d0 tx=0x7f1934012990 comp rx=0 tx=0).stop 2023-10-30T13:30:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f193f46b640 1 -- 172.21.15.46:0/2200587415 shutdown_connections 2023-10-30T13:30:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f193f46b640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f191c0664a0 0x7f191c068960 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f193f46b640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19380b0b50 0x7f193813dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f193f46b640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19380aea80 0x7f19380ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f193f46b640 1 --2- 172.21.15.46:0/2200587415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19240069e0 0x7f19381520d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f193f46b640 1 -- 172.21.15.46:0/2200587415 >> 172.21.15.46:0/2200587415 conn(0x7f19380c7910 msgr2=0x7f193813bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f193f46b640 1 -- 172.21.15.46:0/2200587415 shutdown_connections 2023-10-30T13:30:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:47.704+0000 7f193f46b640 1 -- 172.21.15.46:0/2200587415 wait complete. 2023-10-30T13:30:47.732 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:49.733 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:49.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9509ad6640 1 Processor -- start 2023-10-30T13:30:49.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9509ad6640 1 -- start start 2023-10-30T13:30:49.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9509ad6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f95041538a0 0x7f950415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:49.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9509ad6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95040cf160 0x7f95040cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:49.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9509ad6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95040cb800 con 0x7f95040ca0e0 2023-10-30T13:30:49.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9509ad6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950415dc30 con 0x7f95041538a0 2023-10-30T13:30:49.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9509ad6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95040cb330 con 0x7f95040cf160 2023-10-30T13:30:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9502ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f95040ca0e0 0x7f950415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49108/0 (socket says 172.21.15.46:49108) 2023-10-30T13:30:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9502ffd640 1 -- 172.21.15.46:0/931701377 learned_addr learned my addr 172.21.15.46:0/931701377 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:30:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9503fff640 1 --2- 172.21.15.46:0/931701377 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95040cf160 0x7f95040cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f95037fe640 1 --2- 172.21.15.46:0/931701377 >> v2:172.21.15.154:3300/0 conn(0x7f95041538a0 0x7f950415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f95027fc640 1 -- 172.21.15.46:0/931701377 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151705606 0 0) 0x7f95040cb800 con 0x7f95040ca0e0 2023-10-30T13:30:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f95027fc640 1 -- 172.21.15.46:0/931701377 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ec003530 con 0x7f95040ca0e0 2023-10-30T13:30:49.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9503fff640 1 -- 172.21.15.46:0/931701377 >> v2:172.21.15.154:3300/0 conn(0x7f95041538a0 msgr2=0x7f950415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:49.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9503fff640 1 --2- 172.21.15.46:0/931701377 >> v2:172.21.15.154:3300/0 conn(0x7f95041538a0 0x7f950415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:49.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9503fff640 1 -- 172.21.15.46:0/931701377 >> v1:172.21.15.46:6789/0 conn(0x7f95040ca0e0 legacy=0x7f950415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:49.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f95027fc640 1 -- 172.21.15.46:0/931701377 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4015009335 0 0) 0x7f94ec003530 con 0x7f95040ca0e0 2023-10-30T13:30:49.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.884+0000 7f9503fff640 1 -- 172.21.15.46:0/931701377 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95040cad70 con 0x7f95040cf160 2023-10-30T13:30:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.888+0000 7f9503fff640 1 --2- 172.21.15.46:0/931701377 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95040cf160 0x7f95040cf590 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f94fc009d90 tx=0x7f94fc01f060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cf40a065fadf162 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.888+0000 7f95027fc640 1 -- 172.21.15.46:0/931701377 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8001680 con 0x7f95040cf160 2023-10-30T13:30:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.888+0000 7f95027fc640 1 -- 172.21.15.46:0/931701377 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94fc0a8bb0 con 0x7f95040cf160 2023-10-30T13:30:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.888+0000 7f95027fc640 1 -- 172.21.15.46:0/931701377 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc0b0440 con 0x7f95040cf160 2023-10-30T13:30:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.888+0000 7f9509ad6640 1 -- 172.21.15.46:0/931701377 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95040cf160 msgr2=0x7f95040cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.888+0000 7f9509ad6640 1 --2- 172.21.15.46:0/931701377 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95040cf160 0x7f95040cf590 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f94fc009d90 tx=0x7f94fc01f060 comp rx=0 tx=0).stop 2023-10-30T13:30:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.888+0000 7f9509ad6640 1 -- 172.21.15.46:0/931701377 shutdown_connections 2023-10-30T13:30:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.888+0000 7f9509ad6640 1 --2- 172.21.15.46:0/931701377 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95040cf160 0x7f95040cf590 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.888+0000 7f9509ad6640 1 --2- 172.21.15.46:0/931701377 >> v2:172.21.15.154:3300/0 conn(0x7f95041538a0 0x7f950415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.888+0000 7f9509ad6640 1 -- 172.21.15.46:0/931701377 >> 172.21.15.46:0/931701377 conn(0x7f950406bfe0 msgr2=0x7f950406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:49.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.888+0000 7f9509ad6640 1 -- 172.21.15.46:0/931701377 shutdown_connections 2023-10-30T13:30:49.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.888+0000 7f9509ad6640 1 -- 172.21.15.46:0/931701377 wait complete. 2023-10-30T13:30:49.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9509ad6640 1 Processor -- start 2023-10-30T13:30:49.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9509ad6640 1 -- start start 2023-10-30T13:30:49.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9509ad6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95040ca0e0 0x7f950415ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:49.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9509ad6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95040cf160 0x7f95041e4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:49.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9509ad6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95041538a0 0x7f95041e5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:49.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9509ad6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95041e4730 con 0x7f95040cf160 2023-10-30T13:30:49.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9509ad6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95040c91f0 con 0x7f95040ca0e0 2023-10-30T13:30:49.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9509ad6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95041e3e00 con 0x7f95041538a0 2023-10-30T13:30:49.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9502ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95040cf160 0x7f95041e4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:49.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9502ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95040cf160 0x7f95041e4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44336/0 (socket says 172.21.15.46:44336) 2023-10-30T13:30:49.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9502ffd640 1 -- 172.21.15.46:0/3964217405 learned_addr learned my addr 172.21.15.46:0/3964217405 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f95037fe640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95040ca0e0 0x7f950415ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9503fff640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95041538a0 0x7f95041e5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9502ffd640 1 -- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95041538a0 msgr2=0x7f95041e5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:49.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9502ffd640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95041538a0 0x7f95041e5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:49.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9502ffd640 1 -- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95040ca0e0 msgr2=0x7f950415ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:49.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9502ffd640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95040ca0e0 0x7f950415ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:49.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9502ffd640 1 -- 172.21.15.46:0/3964217405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc009a70 con 0x7f95040cf160 2023-10-30T13:30:49.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9502ffd640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95040cf160 0x7f95041e4bc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f94f800c760 tx=0x7f94f800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f94ebfff640 1 -- 172.21.15.46:0/3964217405 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8005ae0 con 0x7f95040cf160 2023-10-30T13:30:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f94ebfff640 1 -- 172.21.15.46:0/3964217405 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f8005c40 con 0x7f95040cf160 2023-10-30T13:30:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f94ebfff640 1 -- 172.21.15.46:0/3964217405 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8007070 con 0x7f95040cf160 2023-10-30T13:30:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9509ad6640 1 -- 172.21.15.46:0/3964217405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95041e40e0 con 0x7f95040cf160 2023-10-30T13:30:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.892+0000 7f9509ad6640 1 -- 172.21.15.46:0/3964217405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950415d200 con 0x7f95040cf160 2023-10-30T13:30:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.896+0000 7f9509ad6640 1 -- 172.21.15.46:0/3964217405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95040b2a50 con 0x7f95040cf160 2023-10-30T13:30:49.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.896+0000 7f94ebfff640 1 -- 172.21.15.46:0/3964217405 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94f80071d0 con 0x7f95040cf160 2023-10-30T13:30:49.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.896+0000 7f94ebfff640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94f00664a0 0x7f94f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:49.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.896+0000 7f94ebfff640 1 -- 172.21.15.46:0/3964217405 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f94f8083f60 con 0x7f95040cf160 2023-10-30T13:30:49.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.896+0000 7f95037fe640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94f00664a0 0x7f94f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:49.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.896+0000 7f95037fe640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94f00664a0 0x7f94f0068960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009a50 tx=0x7f94f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:49.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:49.900+0000 7f94ebfff640 1 -- 172.21.15.46:0/3964217405 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94f8054bf0 con 0x7f95040cf160 2023-10-30T13:30:50.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.088+0000 7f9509ad6640 1 -- 172.21.15.46:0/3964217405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f950404e170 con 0x7f95040cf160 2023-10-30T13:30:50.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.088+0000 7f94ebfff640 1 -- 172.21.15.46:0/3964217405 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f94f8054780 con 0x7f95040cf160 2023-10-30T13:30:50.097 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:50.097 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:47.037951+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:50.099 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 41 2023-10-30T13:30:50.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.092+0000 7f9509ad6640 1 -- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94f00664a0 msgr2=0x7f94f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:50.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.092+0000 7f9509ad6640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94f00664a0 0x7f94f0068960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009a50 tx=0x7f94f4008040 comp rx=0 tx=0).stop 2023-10-30T13:30:50.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.092+0000 7f9509ad6640 1 -- 172.21.15.46:0/3964217405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95040cf160 msgr2=0x7f95041e4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:50.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.092+0000 7f9509ad6640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95040cf160 0x7f95041e4bc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f94f800c760 tx=0x7f94f800cc20 comp rx=0 tx=0).stop 2023-10-30T13:30:50.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.092+0000 7f9509ad6640 1 -- 172.21.15.46:0/3964217405 shutdown_connections 2023-10-30T13:30:50.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.092+0000 7f9509ad6640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94f00664a0 0x7f94f0068960 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:50.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.092+0000 7f9509ad6640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95041538a0 0x7f95041e5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:50.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.092+0000 7f9509ad6640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95040cf160 0x7f95041e4bc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.092+0000 7f9509ad6640 1 --2- 172.21.15.46:0/3964217405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95040ca0e0 0x7f950415ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.092+0000 7f9509ad6640 1 -- 172.21.15.46:0/3964217405 >> 172.21.15.46:0/3964217405 conn(0x7f950406bfe0 msgr2=0x7f95041524c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.092+0000 7f9509ad6640 1 -- 172.21.15.46:0/3964217405 shutdown_connections 2023-10-30T13:30:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:50.092+0000 7f9509ad6640 1 -- 172.21.15.46:0/3964217405 wait complete. 2023-10-30T13:30:50.118 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:52.119 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:52.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff09beea640 1 Processor -- start 2023-10-30T13:30:52.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff09beea640 1 -- start start 2023-10-30T13:30:52.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff09beea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0940c6b80 0x7ff0940bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:52.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff09beea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff0940c6f60 0x7ff0940c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:52.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff09beea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0940c0a50 con 0x7ff0940c8430 2023-10-30T13:30:52.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff09beea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff09406f6e0 con 0x7ff0940c6b80 2023-10-30T13:30:52.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff09beea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff09406f9c0 con 0x7ff0940c6f60 2023-10-30T13:30:52.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff099c86640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0940c6b80 0x7ff0940bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:52.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff099c86640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0940c6b80 0x7ff0940bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44344/0 (socket says 172.21.15.46:44344) 2023-10-30T13:30:52.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff099c86640 1 -- 172.21.15.46:0/2736022858 learned_addr learned my addr 172.21.15.46:0/2736022858 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:52.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff099485640 1 --2- 172.21.15.46:0/2736022858 >> v2:172.21.15.154:3300/0 conn(0x7ff0940c6f60 0x7ff0940c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:52.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff099c86640 1 -- 172.21.15.46:0/2736022858 >> v2:172.21.15.154:3300/0 conn(0x7ff0940c6f60 msgr2=0x7ff0940c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:52.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff099c86640 1 --2- 172.21.15.46:0/2736022858 >> v2:172.21.15.154:3300/0 conn(0x7ff0940c6f60 0x7ff0940c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:52.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff099c86640 1 -- 172.21.15.46:0/2736022858 >> v1:172.21.15.154:6789/0 conn(0x7ff0940c8430 legacy=0x7ff09413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:52.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.268+0000 7ff099c86640 1 -- 172.21.15.46:0/2736022858 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09406fca0 con 0x7ff0940c6b80 2023-10-30T13:30:52.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.272+0000 7ff099c86640 1 --2- 172.21.15.46:0/2736022858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940c6b80 0x7ff0940bfec0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff094159b90 tx=0x7ff09001ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e92a286dfcab47 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:52.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.272+0000 7ff098c84640 1 -- 172.21.15.46:0/2736022858 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0900a7bb0 con 0x7ff0940c6b80 2023-10-30T13:30:52.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.272+0000 7ff098c84640 1 -- 172.21.15.46:0/2736022858 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0900a7d10 con 0x7ff0940c6b80 2023-10-30T13:30:52.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.272+0000 7ff098c84640 1 -- 172.21.15.46:0/2736022858 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0900af660 con 0x7ff0940c6b80 2023-10-30T13:30:52.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.272+0000 7ff09beea640 1 -- 172.21.15.46:0/2736022858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940c6b80 msgr2=0x7ff0940bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:52.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.272+0000 7ff09beea640 1 --2- 172.21.15.46:0/2736022858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940c6b80 0x7ff0940bfec0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff094159b90 tx=0x7ff09001ebd0 comp rx=0 tx=0).stop 2023-10-30T13:30:52.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.272+0000 7ff09beea640 1 -- 172.21.15.46:0/2736022858 shutdown_connections 2023-10-30T13:30:52.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.272+0000 7ff09beea640 1 --2- 172.21.15.46:0/2736022858 >> v2:172.21.15.154:3300/0 conn(0x7ff0940c6f60 0x7ff0940c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:52.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.272+0000 7ff09beea640 1 --2- 172.21.15.46:0/2736022858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940c6b80 0x7ff0940bfec0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:52.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.272+0000 7ff09beea640 1 -- 172.21.15.46:0/2736022858 >> 172.21.15.46:0/2736022858 conn(0x7ff0940c7910 msgr2=0x7ff0940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:52.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.272+0000 7ff09beea640 1 -- 172.21.15.46:0/2736022858 shutdown_connections 2023-10-30T13:30:52.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.272+0000 7ff09beea640 1 -- 172.21.15.46:0/2736022858 wait complete. 2023-10-30T13:30:52.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09beea640 1 Processor -- start 2023-10-30T13:30:52.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09beea640 1 -- start start 2023-10-30T13:30:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09beea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0940c6b80 0x7ff09415dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09beea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0940c6f60 0x7ff094158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09beea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940c8430 0x7ff094159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:52.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09beea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09406a430 con 0x7ff0940c8430 2023-10-30T13:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09beea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09406a710 con 0x7ff0940c6b80 2023-10-30T13:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09beea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09415fcd0 con 0x7ff0940c6f60 2023-10-30T13:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09a487640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940c8430 0x7ff094159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09a487640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940c8430 0x7ff094159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44350/0 (socket says 172.21.15.46:44350) 2023-10-30T13:30:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09a487640 1 -- 172.21.15.46:0/2050645114 learned_addr learned my addr 172.21.15.46:0/2050645114 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff099c86640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0940c6b80 0x7ff09415dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff099485640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0940c6f60 0x7ff094158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09a487640 1 -- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0940c6f60 msgr2=0x7ff094158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09a487640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0940c6f60 0x7ff094158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09a487640 1 -- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0940c6b80 msgr2=0x7ff09415dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09a487640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0940c6b80 0x7ff09415dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09a487640 1 -- 172.21.15.46:0/2050645114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09000b770 con 0x7ff0940c8430 2023-10-30T13:30:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09a487640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940c8430 0x7ff094159010 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00c730 tx=0x7ff08c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff08a7fc640 1 -- 172.21.15.46:0/2050645114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c003b80 con 0x7ff0940c8430 2023-10-30T13:30:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff08a7fc640 1 -- 172.21.15.46:0/2050645114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08c003ce0 con 0x7ff0940c8430 2023-10-30T13:30:52.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff08a7fc640 1 -- 172.21.15.46:0/2050645114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c00aa30 con 0x7ff0940c8430 2023-10-30T13:30:52.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09beea640 1 -- 172.21.15.46:0/2050645114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff094160220 con 0x7ff0940c8430 2023-10-30T13:30:52.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.276+0000 7ff09beea640 1 -- 172.21.15.46:0/2050645114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09415ff40 con 0x7ff0940c8430 2023-10-30T13:30:52.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.280+0000 7ff09beea640 1 -- 172.21.15.46:0/2050645114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff09406a880 con 0x7ff0940c8430 2023-10-30T13:30:52.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.280+0000 7ff08a7fc640 1 -- 172.21.15.46:0/2050645114 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff08c00ab90 con 0x7ff0940c8430 2023-10-30T13:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.280+0000 7ff08a7fc640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0780664a0 0x7ff078068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.280+0000 7ff08a7fc640 1 -- 172.21.15.46:0/2050645114 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7ff08c083ed0 con 0x7ff0940c8430 2023-10-30T13:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.280+0000 7ff099c86640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0780664a0 0x7ff078068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.284+0000 7ff099c86640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0780664a0 0x7ff078068960 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff0900b1000 tx=0x7ff090008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:52.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.284+0000 7ff08a7fc640 1 -- 172.21.15.46:0/2050645114 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff08c051030 con 0x7ff0940c8430 2023-10-30T13:30:52.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.472+0000 7ff09beea640 1 -- 172.21.15.46:0/2050645114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff09404e170 con 0x7ff0940c8430 2023-10-30T13:30:52.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff08a7fc640 1 -- 172.21.15.46:0/2050645114 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff08c054650 con 0x7ff0940c8430 2023-10-30T13:30:52.482 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:52.482 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:47.037951+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:52.483 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 41 2023-10-30T13:30:52.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff09beea640 1 -- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0780664a0 msgr2=0x7ff078068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:52.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff09beea640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0780664a0 0x7ff078068960 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff0900b1000 tx=0x7ff090008040 comp rx=0 tx=0).stop 2023-10-30T13:30:52.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff09beea640 1 -- 172.21.15.46:0/2050645114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940c8430 msgr2=0x7ff094159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:52.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff09beea640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940c8430 0x7ff094159010 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00c730 tx=0x7ff08c00cbf0 comp rx=0 tx=0).stop 2023-10-30T13:30:52.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff09beea640 1 -- 172.21.15.46:0/2050645114 shutdown_connections 2023-10-30T13:30:52.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff09beea640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0780664a0 0x7ff078068960 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:52.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff09beea640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940c8430 0x7ff094159010 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:52.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff09beea640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0940c6f60 0x7ff094158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:52.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff09beea640 1 --2- 172.21.15.46:0/2050645114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0940c6b80 0x7ff09415dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:52.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff09beea640 1 -- 172.21.15.46:0/2050645114 >> 172.21.15.46:0/2050645114 conn(0x7ff0940c7910 msgr2=0x7ff094138570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:52.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff09beea640 1 -- 172.21.15.46:0/2050645114 shutdown_connections 2023-10-30T13:30:52.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:52.476+0000 7ff09beea640 1 -- 172.21.15.46:0/2050645114 wait complete. 2023-10-30T13:30:52.503 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:54.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:54.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9439e46640 1 Processor -- start 2023-10-30T13:30:54.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9439e46640 1 -- start start 2023-10-30T13:30:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9439e46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94340c6b80 0x7f94340bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9439e46640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f94340c6f60 0x7f94340c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:54.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9439e46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94340c0a50 con 0x7f94340c8430 2023-10-30T13:30:54.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9439e46640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f943406f6e0 con 0x7f94340c6f60 2023-10-30T13:30:54.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9439e46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f943406f9c0 con 0x7f94340c6b80 2023-10-30T13:30:54.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9433fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f94340c8430 0x7f943413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49122/0 (socket says 172.21.15.46:49122) 2023-10-30T13:30:54.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9433fff640 1 -- 172.21.15.46:0/2631058162 learned_addr learned my addr 172.21.15.46:0/2631058162 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:30:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f94337fe640 1 --2- 172.21.15.46:0/2631058162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94340c6b80 0x7f94340bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f942bfff640 1 --2- 172.21.15.46:0/2631058162 >> v2:172.21.15.154:3300/0 conn(0x7f94340c6f60 0x7f94340c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431225915 0 0) 0x7f94340c0a50 con 0x7f94340c8430 2023-10-30T13:30:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9420003530 con 0x7f94340c8430 2023-10-30T13:30:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3352147841 0 0) 0x7f9420003530 con 0x7f94340c8430 2023-10-30T13:30:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94340c6b80 msgr2=0x7f94340bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:54.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 --2- 172.21.15.46:0/2631058162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94340c6b80 0x7f94340bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:54.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 >> v2:172.21.15.154:3300/0 conn(0x7f94340c6f60 msgr2=0x7f94340c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:54.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 --2- 172.21.15.46:0/2631058162 >> v2:172.21.15.154:3300/0 conn(0x7f94340c6f60 0x7f94340c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:54.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943406fca0 con 0x7f94340c8430 2023-10-30T13:30:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f942c001a00 con 0x7f94340c8430 2023-10-30T13:30:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 >> v1:172.21.15.46:6789/0 conn(0x7f94340c8430 legacy=0x7f943413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 --2- 172.21.15.46:0/2631058162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9420003980 0x7f9420003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 --2- 172.21.15.46:0/2631058162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94340c6b80 0x7f94200047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 --2- 172.21.15.46:0/2631058162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9420004d10 0x7f9420005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943406f6e0 con 0x7f9420003980 2023-10-30T13:30:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943406f9c0 con 0x7f9420004d10 2023-10-30T13:30:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942000b990 con 0x7f94340c6b80 2023-10-30T13:30:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f94337fe640 1 --2- 172.21.15.46:0/2631058162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9420003980 0x7f9420003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f942bfff640 1 --2- 172.21.15.46:0/2631058162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94340c6b80 0x7f94200047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f9433fff640 1 --2- 172.21.15.46:0/2631058162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9420004d10 0x7f9420005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f94337fe640 1 -- 172.21.15.46:0/2631058162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94340c6b80 msgr2=0x7f94200047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.652+0000 7f94337fe640 1 --2- 172.21.15.46:0/2631058162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94340c6b80 0x7f94200047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f94337fe640 1 -- 172.21.15.46:0/2631058162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9420004d10 msgr2=0x7f9420005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f94337fe640 1 --2- 172.21.15.46:0/2631058162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9420004d10 0x7f9420005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f94337fe640 1 -- 172.21.15.46:0/2631058162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9420003530 con 0x7f9420003980 2023-10-30T13:30:54.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f94337fe640 1 --2- 172.21.15.46:0/2631058162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9420003980 0x7f9420003da0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9424009ae0 tx=0x7f942401f100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5587229ddfc83a00 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:54.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94240ae600 con 0x7f9420003980 2023-10-30T13:30:54.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94240aa050 con 0x7f9420003980 2023-10-30T13:30:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f9432ffd640 1 -- 172.21.15.46:0/2631058162 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94240ae9a0 con 0x7f9420003980 2023-10-30T13:30:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f9439e46640 1 -- 172.21.15.46:0/2631058162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9420003980 msgr2=0x7f9420003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f9439e46640 1 --2- 172.21.15.46:0/2631058162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9420003980 0x7f9420003da0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9424009ae0 tx=0x7f942401f100 comp rx=0 tx=0).stop 2023-10-30T13:30:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f9433fff640 1 -- 172.21.15.46:0/2631058162 reap_dead start 2023-10-30T13:30:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f9439e46640 1 -- 172.21.15.46:0/2631058162 shutdown_connections 2023-10-30T13:30:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f9439e46640 1 -- 172.21.15.46:0/2631058162 >> 172.21.15.46:0/2631058162 conn(0x7f94340c7910 msgr2=0x7f94340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f9439e46640 1 -- 172.21.15.46:0/2631058162 shutdown_connections 2023-10-30T13:30:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.656+0000 7f9439e46640 1 -- 172.21.15.46:0/2631058162 wait complete. 2023-10-30T13:30:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9439e46640 1 Processor -- start 2023-10-30T13:30:54.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9439e46640 1 -- start start 2023-10-30T13:30:54.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9439e46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9420003980 0x7f943413d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:54.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9439e46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f943413de80 0x7f94340c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:54.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9439e46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943413e4c0 0x7f94340ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:54.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9439e46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94341f1920 con 0x7f943413e4c0 2023-10-30T13:30:54.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9439e46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94341551d0 con 0x7f943413de80 2023-10-30T13:30:54.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9439e46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94341f0d30 con 0x7f9420003980 2023-10-30T13:30:54.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9433fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943413e4c0 0x7f94340ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:54.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9433fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943413e4c0 0x7f94340ca2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44372/0 (socket says 172.21.15.46:44372) 2023-10-30T13:30:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9433fff640 1 -- 172.21.15.46:0/1408381428 learned_addr learned my addr 172.21.15.46:0/1408381428 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f94337fe640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9420003980 0x7f943413d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f942bfff640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f943413de80 0x7f94340c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9433fff640 1 -- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9420003980 msgr2=0x7f943413d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:54.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9433fff640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9420003980 0x7f943413d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9433fff640 1 -- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f943413de80 msgr2=0x7f94340c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9433fff640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f943413de80 0x7f94340c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9433fff640 1 -- 172.21.15.46:0/1408381428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9434155490 con 0x7f943413e4c0 2023-10-30T13:30:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f942bfff640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f943413de80 0x7f94340c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:54.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.660+0000 7f9433fff640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943413e4c0 0x7f94340ca2d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f942c0013e0 tx=0x7f942c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:54.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.664+0000 7f9430ff9640 1 -- 172.21.15.46:0/1408381428 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c007a80 con 0x7f943413e4c0 2023-10-30T13:30:54.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.664+0000 7f9430ff9640 1 -- 172.21.15.46:0/1408381428 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942c00ee20 con 0x7f943413e4c0 2023-10-30T13:30:54.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.664+0000 7f9430ff9640 1 -- 172.21.15.46:0/1408381428 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c013660 con 0x7f943413e4c0 2023-10-30T13:30:54.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.664+0000 7f9439e46640 1 -- 172.21.15.46:0/1408381428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94341f0ff0 con 0x7f943413e4c0 2023-10-30T13:30:54.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.664+0000 7f9439e46640 1 -- 172.21.15.46:0/1408381428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94341f15b0 con 0x7f943413e4c0 2023-10-30T13:30:54.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.664+0000 7f9439e46640 1 -- 172.21.15.46:0/1408381428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94341f1770 con 0x7f943413e4c0 2023-10-30T13:30:54.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.664+0000 7f9430ff9640 1 -- 172.21.15.46:0/1408381428 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f942c0137c0 con 0x7f943413e4c0 2023-10-30T13:30:54.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.664+0000 7f9430ff9640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94140664a0 0x7f9414068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:54.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.664+0000 7f9430ff9640 1 -- 172.21.15.46:0/1408381428 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f942c084510 con 0x7f943413e4c0 2023-10-30T13:30:54.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.664+0000 7f94337fe640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94140664a0 0x7f9414068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:54.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.668+0000 7f94337fe640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94140664a0 0x7f9414068960 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f94240b0000 tx=0x7f94240046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:54.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.672+0000 7f9430ff9640 1 -- 172.21.15.46:0/1408381428 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f942c051700 con 0x7f943413e4c0 2023-10-30T13:30:54.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 -- 172.21.15.46:0/1408381428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f943404e170 con 0x7f943413e4c0 2023-10-30T13:30:54.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9430ff9640 1 -- 172.21.15.46:0/1408381428 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f942c054d20 con 0x7f943413e4c0 2023-10-30T13:30:54.866 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:54.867 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:52.105215+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:clientreplay","state_seq":20,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:54.868 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 42 2023-10-30T13:30:54.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 -- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94140664a0 msgr2=0x7f9414068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:54.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94140664a0 0x7f9414068960 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f94240b0000 tx=0x7f94240046d0 comp rx=0 tx=0).stop 2023-10-30T13:30:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 -- 172.21.15.46:0/1408381428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943413e4c0 msgr2=0x7f94340ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943413e4c0 0x7f94340ca2d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f942c0013e0 tx=0x7f942c00e530 comp rx=0 tx=0).stop 2023-10-30T13:30:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 -- 172.21.15.46:0/1408381428 shutdown_connections 2023-10-30T13:30:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94140664a0 0x7f9414068960 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943413e4c0 0x7f94340ca2d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:54.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f943413de80 0x7f94340c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:54.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 --2- 172.21.15.46:0/1408381428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9420003980 0x7f943413d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:54.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 -- 172.21.15.46:0/1408381428 >> 172.21.15.46:0/1408381428 conn(0x7f94340c7910 msgr2=0x7f9434159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:54.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 -- 172.21.15.46:0/1408381428 shutdown_connections 2023-10-30T13:30:54.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:54.860+0000 7f9439e46640 1 -- 172.21.15.46:0/1408381428 wait complete. 2023-10-30T13:30:54.888 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 42, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 42, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4950": { "addr": "172.21.15.46:6837/1315246492", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 1315246492, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 1315246492, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4950, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_5106": { "addr": "172.21.15.46:6833/603884510", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 603884510, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 603884510, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5106, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 45 }, "gid_5266": { "addr": "172.21.15.154:6835/651441434", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 651441434, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 651441434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5266, "incarnation": 38, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:clientreplay", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:30:52.105215+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5266, "mds_1": 5106, "mds_2": 4950 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:30:56.890 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:57.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29dc5a1640 1 Processor -- start 2023-10-30T13:30:57.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29dc5a1640 1 -- start start 2023-10-30T13:30:57.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29dc5a1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f29d41538a0 0x7f29d415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:57.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29dc5a1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29d40cf160 0x7f29d40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:57.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29dc5a1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d40cb800 con 0x7f29d40ca0e0 2023-10-30T13:30:57.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29dc5a1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d415dc30 con 0x7f29d40cf160 2023-10-30T13:30:57.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29dc5a1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d40cb330 con 0x7f29d41538a0 2023-10-30T13:30:57.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29dab3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29d40cf160 0x7f29d40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:57.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29d9b3c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f29d40ca0e0 0x7f29d415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55544/0 (socket says 172.21.15.46:55544) 2023-10-30T13:30:57.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29d9b3c640 1 -- 172.21.15.46:0/1692651920 learned_addr learned my addr 172.21.15.46:0/1692651920 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:30:57.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29da33d640 1 --2- 172.21.15.46:0/1692651920 >> v2:172.21.15.154:3300/0 conn(0x7f29d41538a0 0x7f29d415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:57.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29dab3e640 1 -- 172.21.15.46:0/1692651920 >> v2:172.21.15.154:3300/0 conn(0x7f29d41538a0 msgr2=0x7f29d415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:57.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29dab3e640 1 --2- 172.21.15.46:0/1692651920 >> v2:172.21.15.154:3300/0 conn(0x7f29d41538a0 0x7f29d415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:57.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29dab3e640 1 -- 172.21.15.46:0/1692651920 >> v1:172.21.15.46:6789/0 conn(0x7f29d40ca0e0 legacy=0x7f29d415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:57.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29dab3e640 1 -- 172.21.15.46:0/1692651920 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d40cad70 con 0x7f29d40cf160 2023-10-30T13:30:57.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.040+0000 7f29d933b640 1 -- 172.21.15.46:0/1692651920 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709699036 0 0) 0x7f29d40cb800 con 0x7f29d40ca0e0 2023-10-30T13:30:57.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.044+0000 7f29dab3e640 1 --2- 172.21.15.46:0/1692651920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d40cf160 0x7f29d40cf590 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f29d40cfd10 tx=0x7f29cc01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa625b14b50de60a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:57.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.044+0000 7f29d933b640 1 -- 172.21.15.46:0/1692651920 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc0b0440 con 0x7f29d40cf160 2023-10-30T13:30:57.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.044+0000 7f29d933b640 1 -- 172.21.15.46:0/1692651920 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29cc0b05a0 con 0x7f29d40cf160 2023-10-30T13:30:57.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.044+0000 7f29d933b640 1 -- 172.21.15.46:0/1692651920 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc0b0930 con 0x7f29d40cf160 2023-10-30T13:30:57.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.044+0000 7f29dc5a1640 1 -- 172.21.15.46:0/1692651920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d40cf160 msgr2=0x7f29d40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:57.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.044+0000 7f29dc5a1640 1 --2- 172.21.15.46:0/1692651920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d40cf160 0x7f29d40cf590 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f29d40cfd10 tx=0x7f29cc01f0f0 comp rx=0 tx=0).stop 2023-10-30T13:30:57.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.044+0000 7f29dc5a1640 1 -- 172.21.15.46:0/1692651920 shutdown_connections 2023-10-30T13:30:57.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.044+0000 7f29dc5a1640 1 --2- 172.21.15.46:0/1692651920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d40cf160 0x7f29d40cf590 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:57.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.044+0000 7f29dc5a1640 1 --2- 172.21.15.46:0/1692651920 >> v2:172.21.15.154:3300/0 conn(0x7f29d41538a0 0x7f29d415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:57.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.044+0000 7f29dc5a1640 1 -- 172.21.15.46:0/1692651920 >> 172.21.15.46:0/1692651920 conn(0x7f29d406bfe0 msgr2=0x7f29d406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:57.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.044+0000 7f29dc5a1640 1 -- 172.21.15.46:0/1692651920 shutdown_connections 2023-10-30T13:30:57.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.044+0000 7f29dc5a1640 1 -- 172.21.15.46:0/1692651920 wait complete. 2023-10-30T13:30:57.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dc5a1640 1 Processor -- start 2023-10-30T13:30:57.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dc5a1640 1 -- start start 2023-10-30T13:30:57.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dc5a1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29d40ca0e0 0x7f29d40b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:57.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dc5a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29d40cf160 0x7f29d41490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:57.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dc5a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d41538a0 0x7f29d41efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:57.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dc5a1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d40b4940 con 0x7f29d41538a0 2023-10-30T13:30:57.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dc5a1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d4148b30 con 0x7f29d40ca0e0 2023-10-30T13:30:57.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dc5a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d40b3a50 con 0x7f29d40cf160 2023-10-30T13:30:57.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dab3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d41538a0 0x7f29d41efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:57.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dab3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d41538a0 0x7f29d41efe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48598/0 (socket says 172.21.15.46:48598) 2023-10-30T13:30:57.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dab3e640 1 -- 172.21.15.46:0/2381184944 learned_addr learned my addr 172.21.15.46:0/2381184944 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:57.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29d9b3c640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29d40cf160 0x7f29d41490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:57.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29da33d640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29d40ca0e0 0x7f29d40b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:57.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dab3e640 1 -- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29d40cf160 msgr2=0x7f29d41490b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:57.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dab3e640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29d40cf160 0x7f29d41490b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:57.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dab3e640 1 -- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29d40ca0e0 msgr2=0x7f29d40b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:57.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dab3e640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29d40ca0e0 0x7f29d40b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:57.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dab3e640 1 -- 172.21.15.46:0/2381184944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29cc009a70 con 0x7f29d41538a0 2023-10-30T13:30:57.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29d9b3c640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29d40cf160 0x7f29d41490b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:57.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dab3e640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d41538a0 0x7f29d41efe00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009e50 tx=0x7f29cc01be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:57.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29caffd640 1 -- 172.21.15.46:0/2381184944 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc0c2070 con 0x7f29d41538a0 2023-10-30T13:30:57.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29caffd640 1 -- 172.21.15.46:0/2381184944 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29cc009450 con 0x7f29d41538a0 2023-10-30T13:30:57.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29caffd640 1 -- 172.21.15.46:0/2381184944 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc0b0440 con 0x7f29d41538a0 2023-10-30T13:30:57.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dc5a1640 1 -- 172.21.15.46:0/2381184944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d40b3d30 con 0x7f29d41538a0 2023-10-30T13:30:57.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.048+0000 7f29dc5a1640 1 -- 172.21.15.46:0/2381184944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d40b42f0 con 0x7f29d41538a0 2023-10-30T13:30:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.052+0000 7f29dc5a1640 1 -- 172.21.15.46:0/2381184944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d40b2a50 con 0x7f29d41538a0 2023-10-30T13:30:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.052+0000 7f29caffd640 1 -- 172.21.15.46:0/2381184944 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29cc0b7050 con 0x7f29d41538a0 2023-10-30T13:30:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.052+0000 7f29caffd640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29b80664a0 0x7f29b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.052+0000 7f29caffd640 1 -- 172.21.15.46:0/2381184944 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f29cc1221e0 con 0x7f29d41538a0 2023-10-30T13:30:57.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.052+0000 7f29da33d640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29b80664a0 0x7f29b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:57.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.052+0000 7f29da33d640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29b80664a0 0x7f29b8068960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f29d000b9d0 tx=0x7f29d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:57.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.056+0000 7f29caffd640 1 -- 172.21.15.46:0/2381184944 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29cc0ef460 con 0x7f29d41538a0 2023-10-30T13:30:57.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 -- 172.21.15.46:0/2381184944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29d40cb020 con 0x7f29d41538a0 2023-10-30T13:30:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29caffd640 1 -- 172.21.15.46:0/2381184944 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f29cc0f2a80 con 0x7f29d41538a0 2023-10-30T13:30:57.254 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:57.255 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:52.105215+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:clientreplay","state_seq":20,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 42 2023-10-30T13:30:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 -- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29b80664a0 msgr2=0x7f29b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29b80664a0 0x7f29b8068960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f29d000b9d0 tx=0x7f29d0008040 comp rx=0 tx=0).stop 2023-10-30T13:30:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 -- 172.21.15.46:0/2381184944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d41538a0 msgr2=0x7f29d41efe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:57.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d41538a0 0x7f29d41efe00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009e50 tx=0x7f29cc01be90 comp rx=0 tx=0).stop 2023-10-30T13:30:57.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 -- 172.21.15.46:0/2381184944 shutdown_connections 2023-10-30T13:30:57.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29b80664a0 0x7f29b8068960 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:57.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d41538a0 0x7f29d41efe00 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:57.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29d40cf160 0x7f29d41490b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:57.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 --2- 172.21.15.46:0/2381184944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29d40ca0e0 0x7f29d40b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:57.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 -- 172.21.15.46:0/2381184944 >> 172.21.15.46:0/2381184944 conn(0x7f29d406bfe0 msgr2=0x7f29d4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:57.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 -- 172.21.15.46:0/2381184944 shutdown_connections 2023-10-30T13:30:57.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:57.248+0000 7f29dc5a1640 1 -- 172.21.15.46:0/2381184944 wait complete. 2023-10-30T13:30:57.276 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:59.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:59.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.424+0000 7f230fc6b640 1 Processor -- start 2023-10-30T13:30:59.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.424+0000 7f230fc6b640 1 -- start start 2023-10-30T13:30:59.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.424+0000 7f230fc6b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2308151800 0x7f230806e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:59.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.424+0000 7f230fc6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23080c8260 0x7f23080cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:59.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.424+0000 7f230fc6b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230806e090 con 0x7f23080c1ee0 2023-10-30T13:30:59.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.424+0000 7f230fc6b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f230806dc70 con 0x7f2308151800 2023-10-30T13:30:59.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.424+0000 7f230fc6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23080c8a50 con 0x7f23080c8260 2023-10-30T13:30:59.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.424+0000 7f230d206640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f23080c1ee0 0x7f23080c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55554/0 (socket says 172.21.15.46:55554) 2023-10-30T13:30:59.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.424+0000 7f230d206640 1 -- 172.21.15.46:0/2605555577 learned_addr learned my addr 172.21.15.46:0/2605555577 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:30:59.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230e208640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23080c8260 0x7f23080cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:59.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230da07640 1 --2- 172.21.15.46:0/2605555577 >> v2:172.21.15.154:3300/0 conn(0x7f2308151800 0x7f230806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:59.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904380082 0 0) 0x7f230806e090 con 0x7f23080c1ee0 2023-10-30T13:30:59.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f4003530 con 0x7f23080c1ee0 2023-10-30T13:30:59.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3155933642 0 0) 0x7f22f4003530 con 0x7f23080c1ee0 2023-10-30T13:30:59.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23080c8260 msgr2=0x7f23080cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:59.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23080c8260 0x7f23080cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:59.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 >> v2:172.21.15.154:3300/0 conn(0x7f2308151800 msgr2=0x7f230806e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:59.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 --2- 172.21.15.46:0/2605555577 >> v2:172.21.15.154:3300/0 conn(0x7f2308151800 0x7f230806e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:59.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080c3b10 con 0x7f23080c1ee0 2023-10-30T13:30:59.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f22f8001680 con 0x7f23080c1ee0 2023-10-30T13:30:59.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 >> v1:172.21.15.46:6789/0 conn(0x7f23080c1ee0 legacy=0x7f23080c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:59.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4003980 0x7f22f4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:59.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22f40047d0 0x7f22f4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:59.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23080c8260 0x7f22f4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:59.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230806dc70 con 0x7f22f40047d0 2023-10-30T13:30:59.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23080c8a50 con 0x7f22f4003980 2023-10-30T13:30:59.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f400b990 con 0x7f23080c8260 2023-10-30T13:30:59.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230d206640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22f40047d0 0x7f22f4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:59.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230e208640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23080c8260 0x7f22f4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:59.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230da07640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4003980 0x7f22f4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:59.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230d206640 1 -- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23080c8260 msgr2=0x7f22f4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:59.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230d206640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23080c8260 0x7f22f4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:59.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230d206640 1 -- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4003980 msgr2=0x7f22f4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:59.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230d206640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4003980 0x7f22f4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:59.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230d206640 1 -- 172.21.15.46:0/2605555577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f4003530 con 0x7f22f40047d0 2023-10-30T13:30:59.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230e208640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23080c8260 0x7f22f4005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:59.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.428+0000 7f230da07640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4003980 0x7f22f4003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:59.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230d206640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22f40047d0 0x7f22f4004c50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f22f800c210 tx=0x7f22f801f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6544b98cf0d279e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:59.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f80a8cb0 con 0x7f22f40047d0 2023-10-30T13:30:59.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f80a8e10 con 0x7f22f40047d0 2023-10-30T13:30:59.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230ca05640 1 -- 172.21.15.46:0/2605555577 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8005b00 con 0x7f22f40047d0 2023-10-30T13:30:59.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230fc6b640 1 -- 172.21.15.46:0/2605555577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22f40047d0 msgr2=0x7f22f4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:59.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230fc6b640 1 --2- 172.21.15.46:0/2605555577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22f40047d0 0x7f22f4004c50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f22f800c210 tx=0x7f22f801f390 comp rx=0 tx=0).stop 2023-10-30T13:30:59.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230e208640 1 -- 172.21.15.46:0/2605555577 reap_dead start 2023-10-30T13:30:59.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230fc6b640 1 -- 172.21.15.46:0/2605555577 shutdown_connections 2023-10-30T13:30:59.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230fc6b640 1 -- 172.21.15.46:0/2605555577 >> 172.21.15.46:0/2605555577 conn(0x7f2308071ea0 msgr2=0x7f2308065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:59.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230fc6b640 1 -- 172.21.15.46:0/2605555577 shutdown_connections 2023-10-30T13:30:59.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230fc6b640 1 -- 172.21.15.46:0/2605555577 wait complete. 2023-10-30T13:30:59.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230fc6b640 1 Processor -- start 2023-10-30T13:30:59.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230fc6b640 1 -- start start 2023-10-30T13:30:59.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.432+0000 7f230fc6b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f40047d0 0x7f2308156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:59.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230fc6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23081ee970 0x7f2308142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:59.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230fc6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2308153e90 0x7f2308150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:59.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230fc6b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2308154fb0 con 0x7f2308153e90 2023-10-30T13:30:59.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230fc6b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23081ee4e0 con 0x7f22f40047d0 2023-10-30T13:30:59.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230fc6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23081549f0 con 0x7f23081ee970 2023-10-30T13:30:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230e208640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2308153e90 0x7f2308150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230e208640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2308153e90 0x7f2308150700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48608/0 (socket says 172.21.15.46:48608) 2023-10-30T13:30:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230e208640 1 -- 172.21.15.46:0/1551597257 learned_addr learned my addr 172.21.15.46:0/1551597257 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:30:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230da07640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f40047d0 0x7f2308156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230d206640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23081ee970 0x7f2308142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230e208640 1 -- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23081ee970 msgr2=0x7f2308142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230e208640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23081ee970 0x7f2308142310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230e208640 1 -- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f40047d0 msgr2=0x7f2308156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230e208640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f40047d0 0x7f2308156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230e208640 1 -- 172.21.15.46:0/1551597257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2308154cd0 con 0x7f2308153e90 2023-10-30T13:30:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230d206640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23081ee970 0x7f2308142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230e208640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2308153e90 0x7f2308150700 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f23080c4200 tx=0x7f2300010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f22fe7fc640 1 -- 172.21.15.46:0/1551597257 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230001b070 con 0x7f2308153e90 2023-10-30T13:30:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f22fe7fc640 1 -- 172.21.15.46:0/1551597257 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230000f040 con 0x7f2308153e90 2023-10-30T13:30:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f22fe7fc640 1 -- 172.21.15.46:0/1551597257 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230000e050 con 0x7f2308153e90 2023-10-30T13:30:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230fc6b640 1 -- 172.21.15.46:0/1551597257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23081ede90 con 0x7f2308153e90 2023-10-30T13:30:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230fc6b640 1 -- 172.21.15.46:0/1551597257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2308142850 con 0x7f2308153e90 2023-10-30T13:30:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f230fc6b640 1 -- 172.21.15.46:0/1551597257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23081edcf0 con 0x7f2308153e90 2023-10-30T13:30:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.436+0000 7f22fe7fc640 1 -- 172.21.15.46:0/1551597257 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2300003be0 con 0x7f2308153e90 2023-10-30T13:30:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.440+0000 7f22fe7fc640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f22ec0664d0 0x7f22ec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.440+0000 7f22fe7fc640 1 -- 172.21.15.46:0/1551597257 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f2300084640 con 0x7f2308153e90 2023-10-30T13:30:59.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.440+0000 7f230da07640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f22ec0664d0 0x7f22ec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:59.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.440+0000 7f230da07640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f22ec0664d0 0x7f22ec068990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2304009a30 tx=0x7f2304008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:59.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.444+0000 7f22fe7fc640 1 -- 172.21.15.46:0/1551597257 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23081edcf0 con 0x7f2308153e90 2023-10-30T13:30:59.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.632+0000 7f230fc6b640 1 -- 172.21.15.46:0/1551597257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23081edcf0 con 0x7f2308153e90 2023-10-30T13:30:59.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.632+0000 7f22fe7fc640 1 -- 172.21.15.46:0/1551597257 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f23081edcf0 con 0x7f2308153e90 2023-10-30T13:30:59.641 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:30:59.641 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:30:57.974169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:59.641 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 43 2023-10-30T13:30:59.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.636+0000 7f230fc6b640 1 -- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f22ec0664d0 msgr2=0x7f22ec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:59.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.636+0000 7f230fc6b640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f22ec0664d0 0x7f22ec068990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2304009a30 tx=0x7f2304008040 comp rx=0 tx=0).stop 2023-10-30T13:30:59.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.636+0000 7f230fc6b640 1 -- 172.21.15.46:0/1551597257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2308153e90 msgr2=0x7f2308150700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:59.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.636+0000 7f230fc6b640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2308153e90 0x7f2308150700 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f23080c4200 tx=0x7f2300010590 comp rx=0 tx=0).stop 2023-10-30T13:30:59.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.636+0000 7f230fc6b640 1 -- 172.21.15.46:0/1551597257 shutdown_connections 2023-10-30T13:30:59.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.636+0000 7f230fc6b640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f22ec0664d0 0x7f22ec068990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:59.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.636+0000 7f230fc6b640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2308153e90 0x7f2308150700 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:59.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.636+0000 7f230fc6b640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23081ee970 0x7f2308142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:59.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.636+0000 7f230fc6b640 1 --2- 172.21.15.46:0/1551597257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f40047d0 0x7f2308156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:59.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.636+0000 7f230fc6b640 1 -- 172.21.15.46:0/1551597257 >> 172.21.15.46:0/1551597257 conn(0x7f2308071ea0 msgr2=0x7f230806c290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:59.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.636+0000 7f230fc6b640 1 -- 172.21.15.46:0/1551597257 shutdown_connections 2023-10-30T13:30:59.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:30:59.636+0000 7f230fc6b640 1 -- 172.21.15.46:0/1551597257 wait complete. 2023-10-30T13:30:59.662 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4306 2023-10-30T13:30:59.662 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.c 2023-10-30T13:31:04.805 INFO:tasks.ceph.mds.b.smithi154.stderr:2023-10-30T13:31:04.802+0000 7fce6e191640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:31:04.806 INFO:tasks.ceph.mds.b.smithi154.stderr:2023-10-30T13:31:04.802+0000 7fce6e191640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:31:04.806 INFO:tasks.ceph.mds.b.smithi154.stderr:2023-10-30T13:31:04.802+0000 7fce6e191640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:31:04.976 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/175/20' 2023-10-30T13:31:04.977 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/175/12' 2023-10-30T13:31:04.977 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/175/10' 2023-10-30T13:31:04.977 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/175/11' 2023-10-30T13:31:04.977 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/175/16' 2023-10-30T13:31:04.978 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/175/14' 2023-10-30T13:31:04.978 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/175/18' 2023-10-30T13:31:04.978 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/175/13' 2023-10-30T13:31:04.978 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/175/17' 2023-10-30T13:31:04.978 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/175' 2023-10-30T13:31:04.979 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/181/19' 2023-10-30T13:31:04.979 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/181/15' 2023-10-30T13:31:04.979 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/181/20' 2023-10-30T13:31:04.979 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/181/12' 2023-10-30T13:31:04.979 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/181/10' 2023-10-30T13:31:04.979 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/181/11' 2023-10-30T13:31:04.980 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/181/16' 2023-10-30T13:31:04.980 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/181/14' 2023-10-30T13:31:04.980 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/181/18' 2023-10-30T13:31:04.980 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/181/13' 2023-10-30T13:31:04.980 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/181/17' 2023-10-30T13:31:04.981 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/181' 2023-10-30T13:31:04.981 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/127/19' 2023-10-30T13:31:04.981 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/127/15' 2023-10-30T13:31:04.981 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/127/20' 2023-10-30T13:31:04.981 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/127/12' 2023-10-30T13:31:04.982 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/127/10' 2023-10-30T13:31:04.982 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/127/11' 2023-10-30T13:31:04.982 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/127/16' 2023-10-30T13:31:04.982 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/127/14' 2023-10-30T13:31:04.982 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/127/18' 2023-10-30T13:31:04.983 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/127/13' 2023-10-30T13:31:04.983 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/127/17' 2023-10-30T13:31:04.983 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/127' 2023-10-30T13:31:04.983 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/108/19' 2023-10-30T13:31:04.983 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/108/15' 2023-10-30T13:31:04.984 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/108/20' 2023-10-30T13:31:04.984 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/108/12' 2023-10-30T13:31:04.984 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/108/10' 2023-10-30T13:31:04.984 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/108/11' 2023-10-30T13:31:04.984 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/108/16' 2023-10-30T13:31:04.985 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/108/14' 2023-10-30T13:31:04.985 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/108/18' 2023-10-30T13:31:04.985 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/108/13' 2023-10-30T13:31:04.985 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/108/17' 2023-10-30T13:31:04.985 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/108' 2023-10-30T13:31:04.985 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/183/19' 2023-10-30T13:31:04.986 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/183/15' 2023-10-30T13:31:04.986 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/183/20' 2023-10-30T13:31:04.986 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/183/12' 2023-10-30T13:31:04.986 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/183/10' 2023-10-30T13:31:04.986 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/183/11' 2023-10-30T13:31:04.987 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/183/16' 2023-10-30T13:31:04.987 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/183/14' 2023-10-30T13:31:04.987 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/183/18' 2023-10-30T13:31:04.987 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/183/13' 2023-10-30T13:31:04.987 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/183/17' 2023-10-30T13:31:04.988 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/183' 2023-10-30T13:31:04.988 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/177/19' 2023-10-30T13:31:04.988 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/177/15' 2023-10-30T13:31:04.988 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/177/20' 2023-10-30T13:31:04.988 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/177/12' 2023-10-30T13:31:04.988 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/177/10' 2023-10-30T13:31:04.989 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/177/11' 2023-10-30T13:31:04.989 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/177/16' 2023-10-30T13:31:04.989 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/177/14' 2023-10-30T13:31:04.989 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/177/18' 2023-10-30T13:31:04.989 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/177/13' 2023-10-30T13:31:04.989 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/177/17' 2023-10-30T13:31:04.990 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/177' 2023-10-30T13:31:04.990 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/169/19' 2023-10-30T13:31:04.990 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/169/15' 2023-10-30T13:31:04.990 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/169/20' 2023-10-30T13:31:04.990 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/169/12' 2023-10-30T13:31:04.991 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/169/10' 2023-10-30T13:31:04.991 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/169/11' 2023-10-30T13:31:04.991 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/169/16' 2023-10-30T13:31:04.991 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/169/14' 2023-10-30T13:31:04.991 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/169/18' 2023-10-30T13:31:04.992 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/169/13' 2023-10-30T13:31:04.992 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/169/17' 2023-10-30T13:31:04.992 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/169' 2023-10-30T13:31:04.992 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/154/19' 2023-10-30T13:31:04.992 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/154/15' 2023-10-30T13:31:04.992 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/154/20' 2023-10-30T13:31:04.993 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/154/12' 2023-10-30T13:31:04.993 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/154/10' 2023-10-30T13:31:04.993 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/154/11' 2023-10-30T13:31:04.993 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/154/16' 2023-10-30T13:31:04.993 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/154/14' 2023-10-30T13:31:04.993 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/154/18' 2023-10-30T13:31:04.994 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/154/13' 2023-10-30T13:31:04.994 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/154/17' 2023-10-30T13:31:04.994 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/154' 2023-10-30T13:31:04.994 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/171/19' 2023-10-30T13:31:04.994 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/171/15' 2023-10-30T13:31:04.995 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/171/20' 2023-10-30T13:31:04.995 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/171/12' 2023-10-30T13:31:04.995 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/171/10' 2023-10-30T13:31:04.995 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/171/11' 2023-10-30T13:31:04.995 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/171/16' 2023-10-30T13:31:04.995 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/171/14' 2023-10-30T13:31:04.996 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/171/18' 2023-10-30T13:31:04.996 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/171/13' 2023-10-30T13:31:04.996 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/171/17' 2023-10-30T13:31:04.996 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/171' 2023-10-30T13:31:04.996 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/139/19' 2023-10-30T13:31:04.997 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/139/15' 2023-10-30T13:31:04.997 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/139/20' 2023-10-30T13:31:07.042 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/139/12' 2023-10-30T13:31:07.042 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/139/10' 2023-10-30T13:31:07.043 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/139/11' 2023-10-30T13:31:07.043 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/139/16' 2023-10-30T13:31:07.043 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/139/14' 2023-10-30T13:31:07.043 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/139/18' 2023-10-30T13:31:07.043 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/139/13' 2023-10-30T13:31:07.044 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/139/17' 2023-10-30T13:31:07.044 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/139' 2023-10-30T13:31:07.044 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/102/19' 2023-10-30T13:31:07.044 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/102/15' 2023-10-30T13:31:07.044 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/102/20' 2023-10-30T13:31:07.045 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/102/12' 2023-10-30T13:31:07.045 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/102/10' 2023-10-30T13:31:07.045 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/102/11' 2023-10-30T13:31:07.045 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/102/16' 2023-10-30T13:31:07.045 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/102/14' 2023-10-30T13:31:07.045 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/102/18' 2023-10-30T13:31:07.046 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/102/13' 2023-10-30T13:31:07.046 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/102/17' 2023-10-30T13:31:07.046 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/102' 2023-10-30T13:31:07.046 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/129/19' 2023-10-30T13:31:07.046 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/129/15' 2023-10-30T13:31:07.047 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/129/20' 2023-10-30T13:31:07.047 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/129/12' 2023-10-30T13:31:07.047 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/129/10' 2023-10-30T13:31:07.047 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/129/11' 2023-10-30T13:31:07.047 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/129/16' 2023-10-30T13:31:07.047 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/129/14' 2023-10-30T13:31:07.048 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/129/18' 2023-10-30T13:31:07.048 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/129/13' 2023-10-30T13:31:07.048 INFO:tasks.workunit.client.0.smithi046.stdout:removed './copyofsnap1/test1/129/17' 2023-10-30T13:31:07.048 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1/129' 2023-10-30T13:31:07.048 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1/test1' 2023-10-30T13:31:07.048 INFO:tasks.workunit.client.0.smithi046.stdout:removed directory './copyofsnap1' 2023-10-30T13:31:07.049 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T13:31:07.050 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:31:07.050 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:31:07.105 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2023-10-30T13:31:07.105 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2023-10-30T13:31:07.663 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:31:07.663 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:31:07.664 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:31:07.668 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:31:07.668 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:07.763 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-10-30T13:31:07.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25ae9c4640 1 Processor -- start 2023-10-30T13:31:07.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25ae9c4640 1 -- start start 2023-10-30T13:31:07.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25ae9c4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f25a81538a0 0x7f25a815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:07.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25ae9c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25a80cf160 0x7f25a80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:07.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25ae9c4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a80cb800 con 0x7f25a80ca0e0 2023-10-30T13:31:07.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25ae9c4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a815dc30 con 0x7f25a81538a0 2023-10-30T13:31:07.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25ae9c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a80cb330 con 0x7f25a80cf160 2023-10-30T13:31:07.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25a7fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f25a81538a0 0x7f25a815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:07.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25acf61640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25a80cf160 0x7f25a80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:07.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25acf61640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25a80cf160 0x7f25a80cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:40136/0 (socket says 172.21.15.46:40136) 2023-10-30T13:31:07.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25a7fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f25a81538a0 0x7f25a815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:35500/0 (socket says 172.21.15.46:35500) 2023-10-30T13:31:07.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25acf61640 1 -- 172.21.15.46:0/885755826 learned_addr learned my addr 172.21.15.46:0/885755826 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:07.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.819+0000 7f25a77fe640 1 --1- 172.21.15.46:0/885755826 >> v1:172.21.15.154:6789/0 conn(0x7f25a80ca0e0 0x7f25a815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:41760/0 (socket says 172.21.15.46:41760) 2023-10-30T13:31:07.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.823+0000 7f25acf61640 1 -- 172.21.15.46:0/885755826 >> v2:172.21.15.154:3300/0 conn(0x7f25a81538a0 msgr2=0x7f25a815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:07.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.823+0000 7f25acf61640 1 --2- 172.21.15.46:0/885755826 >> v2:172.21.15.154:3300/0 conn(0x7f25a81538a0 0x7f25a815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:07.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.823+0000 7f25acf61640 1 -- 172.21.15.46:0/885755826 >> v1:172.21.15.154:6789/0 conn(0x7f25a80ca0e0 legacy=0x7f25a815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:07.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.823+0000 7f25acf61640 1 -- 172.21.15.46:0/885755826 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a80cad70 con 0x7f25a80cf160 2023-10-30T13:31:07.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.823+0000 7f25acf61640 1 --2- 172.21.15.46:0/885755826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25a80cf160 0x7f25a80cf590 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f25a0009e50 tx=0x7f25a001f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef0840ea519e8a9c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:07.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.823+0000 7f25a6ffd640 1 -- 172.21.15.46:0/885755826 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a00af400 con 0x7f25a80cf160 2023-10-30T13:31:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.823+0000 7f25a6ffd640 1 -- 172.21.15.46:0/885755826 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a00ab050 con 0x7f25a80cf160 2023-10-30T13:31:07.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.823+0000 7f25a6ffd640 1 -- 172.21.15.46:0/885755826 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a00af780 con 0x7f25a80cf160 2023-10-30T13:31:07.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 -- 172.21.15.46:0/885755826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25a80cf160 msgr2=0x7f25a80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:07.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 --2- 172.21.15.46:0/885755826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25a80cf160 0x7f25a80cf590 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f25a0009e50 tx=0x7f25a001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:31:07.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 -- 172.21.15.46:0/885755826 shutdown_connections 2023-10-30T13:31:07.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 --2- 172.21.15.46:0/885755826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25a80cf160 0x7f25a80cf590 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:07.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 --2- 172.21.15.46:0/885755826 >> v2:172.21.15.154:3300/0 conn(0x7f25a81538a0 0x7f25a815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:07.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 -- 172.21.15.46:0/885755826 >> 172.21.15.46:0/885755826 conn(0x7f25a806bfe0 msgr2=0x7f25a806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:07.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 -- 172.21.15.46:0/885755826 shutdown_connections 2023-10-30T13:31:07.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 -- 172.21.15.46:0/885755826 wait complete. 2023-10-30T13:31:07.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 Processor -- start 2023-10-30T13:31:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 -- start start 2023-10-30T13:31:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25a80ca0e0 0x7f25a81e6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25a80cf160 0x7f25a80ce310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a81538a0 0x7f25a81ef110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:07.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a81e6c70 con 0x7f25a81538a0 2023-10-30T13:31:07.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a81f35c0 con 0x7f25a80cf160 2023-10-30T13:31:07.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25ae9c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a8159460 con 0x7f25a80ca0e0 2023-10-30T13:31:07.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25acf61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a81538a0 0x7f25a81ef110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:07.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25acf61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a81538a0 0x7f25a81ef110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59258/0 (socket says 172.21.15.46:59258) 2023-10-30T13:31:07.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25acf61640 1 -- 172.21.15.46:0/3773613793 learned_addr learned my addr 172.21.15.46:0/3773613793 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:07.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25a7fff640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25a80ca0e0 0x7f25a81e6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:07.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25a77fe640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25a80cf160 0x7f25a80ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:07.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25acf61640 1 -- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25a80ca0e0 msgr2=0x7f25a81e6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:07.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25acf61640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25a80ca0e0 0x7f25a81e6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:07.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25acf61640 1 -- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25a80cf160 msgr2=0x7f25a80ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:07.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25acf61640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25a80cf160 0x7f25a80ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:07.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.827+0000 7f25acf61640 1 -- 172.21.15.46:0/3773613793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a0009b30 con 0x7f25a81538a0 2023-10-30T13:31:07.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.831+0000 7f25acf61640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a81538a0 0x7f25a81ef110 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f25a001ce20 tx=0x7f25a001bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:07.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.831+0000 7f25a4ff9640 1 -- 172.21.15.46:0/3773613793 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0007000 con 0x7f25a81538a0 2023-10-30T13:31:07.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.831+0000 7f25a4ff9640 1 -- 172.21.15.46:0/3773613793 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a00afd00 con 0x7f25a81538a0 2023-10-30T13:31:07.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.831+0000 7f25ae9c4640 1 -- 172.21.15.46:0/3773613793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a81f2800 con 0x7f25a81538a0 2023-10-30T13:31:07.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.831+0000 7f25a4ff9640 1 -- 172.21.15.46:0/3773613793 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a00b9630 con 0x7f25a81538a0 2023-10-30T13:31:07.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.831+0000 7f25ae9c4640 1 -- 172.21.15.46:0/3773613793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a81f2dc0 con 0x7f25a81538a0 2023-10-30T13:31:07.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.831+0000 7f25ae9c4640 1 -- 172.21.15.46:0/3773613793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a80b2a50 con 0x7f25a81538a0 2023-10-30T13:31:07.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.831+0000 7f25a4ff9640 1 -- 172.21.15.46:0/3773613793 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25a00b6070 con 0x7f25a81538a0 2023-10-30T13:31:07.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.831+0000 7f25a4ff9640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25940664a0 0x7f2594068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:07.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.831+0000 7f25a4ff9640 1 -- 172.21.15.46:0/3773613793 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f25a0121ff0 con 0x7f25a81538a0 2023-10-30T13:31:07.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.831+0000 7f25a7fff640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25940664a0 0x7f2594068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:07.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.835+0000 7f25a7fff640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25940664a0 0x7f2594068960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2598009ac0 tx=0x7f2598009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:07.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:07.835+0000 7f25a4ff9640 1 -- 172.21.15.46:0/3773613793 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25a00ef220 con 0x7f25a81538a0 2023-10-30T13:31:08.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 -- 172.21.15.46:0/3773613793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25a804e170 con 0x7f25a81538a0 2023-10-30T13:31:08.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25a4ff9640 1 -- 172.21.15.46:0/3773613793 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f25a00b4070 con 0x7f25a81538a0 2023-10-30T13:31:08.034 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:08.035 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:01.861512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:08.035 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2023-10-30T13:31:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 -- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25940664a0 msgr2=0x7f2594068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25940664a0 0x7f2594068960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2598009ac0 tx=0x7f2598009190 comp rx=0 tx=0).stop 2023-10-30T13:31:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 -- 172.21.15.46:0/3773613793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a81538a0 msgr2=0x7f25a81ef110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a81538a0 0x7f25a81ef110 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f25a001ce20 tx=0x7f25a001bf90 comp rx=0 tx=0).stop 2023-10-30T13:31:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 -- 172.21.15.46:0/3773613793 shutdown_connections 2023-10-30T13:31:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25940664a0 0x7f2594068960 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a81538a0 0x7f25a81ef110 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25a80cf160 0x7f25a80ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 --2- 172.21.15.46:0/3773613793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25a80ca0e0 0x7f25a81e6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 -- 172.21.15.46:0/3773613793 >> 172.21.15.46:0/3773613793 conn(0x7f25a806bfe0 msgr2=0x7f25a8066420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:08.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 -- 172.21.15.46:0/3773613793 shutdown_connections 2023-10-30T13:31:08.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.031+0000 7f25ae9c4640 1 -- 172.21.15.46:0/3773613793 wait complete. 2023-10-30T13:31:08.056 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:31:08.056 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.195+0000 7f7254c46640 1 Processor -- start 2023-10-30T13:31:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.195+0000 7f7254c46640 1 -- start start 2023-10-30T13:31:08.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.195+0000 7f7254c46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72500c6b80 0x7f72500bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:08.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.195+0000 7f7254c46640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72500c6f60 0x7f72500c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:08.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.195+0000 7f7254c46640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72500c0a50 con 0x7f72500c8430 2023-10-30T13:31:08.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.195+0000 7f7254c46640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f725006f6e0 con 0x7f72500c6f60 2023-10-30T13:31:08.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.195+0000 7f7254c46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725006f9c0 con 0x7f72500c6b80 2023-10-30T13:31:08.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.195+0000 7f7247fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72500c6f60 0x7f72500c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:08.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.195+0000 7f7247fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72500c6f60 0x7f72500c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59262/0 (socket says 172.21.15.46:59262) 2023-10-30T13:31:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.195+0000 7f7247fff640 1 -- 172.21.15.46:0/2821748771 learned_addr learned my addr 172.21.15.46:0/2821748771 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.195+0000 7f724e59c640 1 --2- 172.21.15.46:0/2821748771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72500c6b80 0x7f72500bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.199+0000 7f7247fff640 1 -- 172.21.15.46:0/2821748771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72500c6b80 msgr2=0x7f72500bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.199+0000 7f7247fff640 1 --2- 172.21.15.46:0/2821748771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72500c6b80 0x7f72500bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.199+0000 7f7247fff640 1 -- 172.21.15.46:0/2821748771 >> v1:172.21.15.154:6789/0 conn(0x7f72500c8430 legacy=0x7f725013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.199+0000 7f7247fff640 1 -- 172.21.15.46:0/2821748771 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725006fca0 con 0x7f72500c6f60 2023-10-30T13:31:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.199+0000 7f7247fff640 1 --2- 172.21.15.46:0/2821748771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72500c6f60 0x7f72500c0400 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7238009870 tx=0x7f723801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bce5177fd564fa3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.199+0000 7f724dd9b640 1 -- 172.21.15.46:0/2821748771 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72380ab650 con 0x7f72500c6f60 2023-10-30T13:31:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.199+0000 7f724dd9b640 1 -- 172.21.15.46:0/2821748771 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72380ab7b0 con 0x7f72500c6f60 2023-10-30T13:31:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.199+0000 7f724dd9b640 1 -- 172.21.15.46:0/2821748771 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72380abb40 con 0x7f72500c6f60 2023-10-30T13:31:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.199+0000 7f7254c46640 1 -- 172.21.15.46:0/2821748771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72500c6f60 msgr2=0x7f72500c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.199+0000 7f7254c46640 1 --2- 172.21.15.46:0/2821748771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72500c6f60 0x7f72500c0400 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7238009870 tx=0x7f723801c540 comp rx=0 tx=0).stop 2023-10-30T13:31:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 -- 172.21.15.46:0/2821748771 shutdown_connections 2023-10-30T13:31:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 --2- 172.21.15.46:0/2821748771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72500c6f60 0x7f72500c0400 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 --2- 172.21.15.46:0/2821748771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72500c6b80 0x7f72500bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 -- 172.21.15.46:0/2821748771 >> 172.21.15.46:0/2821748771 conn(0x7f72500c7910 msgr2=0x7f72500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 -- 172.21.15.46:0/2821748771 shutdown_connections 2023-10-30T13:31:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 -- 172.21.15.46:0/2821748771 wait complete. 2023-10-30T13:31:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 Processor -- start 2023-10-30T13:31:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 -- start start 2023-10-30T13:31:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72500c6b80 0x7f72501580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72500c6f60 0x7f7250155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72500c8430 0x7f7250131300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72500ca800 con 0x7f72500c6b80 2023-10-30T13:31:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7250158630 con 0x7f72500c6f60 2023-10-30T13:31:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.203+0000 7f7254c46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7250138dc0 con 0x7f72500c8430 2023-10-30T13:31:08.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f724e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72500c6b80 0x7f72501580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:08.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f724e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72500c6b80 0x7f72501580f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59278/0 (socket says 172.21.15.46:59278) 2023-10-30T13:31:08.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f724e59c640 1 -- 172.21.15.46:0/870472107 learned_addr learned my addr 172.21.15.46:0/870472107 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:08.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f724ed9d640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72500c8430 0x7f7250131300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:08.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f7247fff640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72500c6f60 0x7f7250155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:08.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f724e59c640 1 -- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72500c8430 msgr2=0x7f7250131300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f724e59c640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72500c8430 0x7f7250131300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f724e59c640 1 -- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72500c6f60 msgr2=0x7f7250155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f724e59c640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72500c6f60 0x7f7250155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f724e59c640 1 -- 172.21.15.46:0/870472107 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72380094d0 con 0x7f72500c6b80 2023-10-30T13:31:08.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f724ed9d640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72500c8430 0x7f7250131300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:08.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f7247fff640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72500c6f60 0x7f7250155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f724e59c640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72500c6b80 0x7f72501580f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f724000d800 tx=0x7f724000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f7246ffd640 1 -- 172.21.15.46:0/870472107 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724000b6c0 con 0x7f72500c6b80 2023-10-30T13:31:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f7246ffd640 1 -- 172.21.15.46:0/870472107 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724000b820 con 0x7f72500c6b80 2023-10-30T13:31:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f7246ffd640 1 -- 172.21.15.46:0/870472107 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724000bb90 con 0x7f72500c6b80 2023-10-30T13:31:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f7254c46640 1 -- 172.21.15.46:0/870472107 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7250139090 con 0x7f72500c6b80 2023-10-30T13:31:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f7254c46640 1 -- 172.21.15.46:0/870472107 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7250133330 con 0x7f72500c6b80 2023-10-30T13:31:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f7254c46640 1 -- 172.21.15.46:0/870472107 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725004e170 con 0x7f72500c6b80 2023-10-30T13:31:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.207+0000 7f7246ffd640 1 -- 172.21.15.46:0/870472107 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f724000bcf0 con 0x7f72500c6b80 2023-10-30T13:31:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.211+0000 7f7246ffd640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72300664a0 0x7f7230068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.211+0000 7f7246ffd640 1 -- 172.21.15.46:0/870472107 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f7240084730 con 0x7f72500c6b80 2023-10-30T13:31:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.211+0000 7f7247fff640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72300664a0 0x7f7230068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.211+0000 7f7247fff640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72300664a0 0x7f7230068960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f72380af000 tx=0x7f72380a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.215+0000 7f7246ffd640 1 -- 172.21.15.46:0/870472107 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7240051890 con 0x7f72500c6b80 2023-10-30T13:31:08.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 -- 172.21.15.46:0/870472107 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72500c54f0 con 0x7f72500c6b80 2023-10-30T13:31:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7246ffd640 1 -- 172.21.15.46:0/870472107 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f7240011340 con 0x7f72500c6b80 2023-10-30T13:31:08.406 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:08.406 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:01.861512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2023-10-30T13:31:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 -- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72300664a0 msgr2=0x7f7230068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72300664a0 0x7f7230068960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f72380af000 tx=0x7f72380a7000 comp rx=0 tx=0).stop 2023-10-30T13:31:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 -- 172.21.15.46:0/870472107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72500c6b80 msgr2=0x7f72501580f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72500c6b80 0x7f72501580f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f724000d800 tx=0x7f724000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 -- 172.21.15.46:0/870472107 shutdown_connections 2023-10-30T13:31:08.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72300664a0 0x7f7230068960 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72500c8430 0x7f7250131300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72500c6f60 0x7f7250155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 --2- 172.21.15.46:0/870472107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72500c6b80 0x7f72501580f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 -- 172.21.15.46:0/870472107 >> 172.21.15.46:0/870472107 conn(0x7f72500c7910 msgr2=0x7f72500c10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:08.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 -- 172.21.15.46:0/870472107 shutdown_connections 2023-10-30T13:31:08.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.403+0000 7f7254c46640 1 -- 172.21.15.46:0/870472107 wait complete. 2023-10-30T13:31:08.427 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=2) 2023-10-30T13:31:08.427 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:31:08.428 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:31:08.444 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:31:08.444 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:08.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f071366b640 1 Processor -- start 2023-10-30T13:31:08.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f071366b640 1 -- start start 2023-10-30T13:31:08.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f071366b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f070c1538a0 0x7f070c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:08.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f071366b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f070c0ca0e0 0x7f070c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:08.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f071366b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070c0cb800 con 0x7f070c0cf160 2023-10-30T13:31:08.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f071366b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f070c15dc30 con 0x7f070c0ca0e0 2023-10-30T13:31:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f071366b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070c0cb330 con 0x7f070c1538a0 2023-10-30T13:31:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f0710c06640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f070c0ca0e0 0x7f070c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f0710c06640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f070c0ca0e0 0x7f070c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59284/0 (socket says 172.21.15.46:59284) 2023-10-30T13:31:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f0710c06640 1 -- 172.21.15.46:0/1883108905 learned_addr learned my addr 172.21.15.46:0/1883108905 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:08.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f0711407640 1 --2- 172.21.15.46:0/1883108905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f070c1538a0 0x7f070c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f0710c06640 1 -- 172.21.15.46:0/1883108905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f070c1538a0 msgr2=0x7f070c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f0710c06640 1 --2- 172.21.15.46:0/1883108905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f070c1538a0 0x7f070c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f0710c06640 1 -- 172.21.15.46:0/1883108905 >> v1:172.21.15.154:6789/0 conn(0x7f070c0cf160 legacy=0x7f070c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f0710c06640 1 -- 172.21.15.46:0/1883108905 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c0cad70 con 0x7f070c0ca0e0 2023-10-30T13:31:08.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.623+0000 7f0711407640 1 --2- 172.21.15.46:0/1883108905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f070c1538a0 0x7f070c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:08.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.627+0000 7f0710c06640 1 --2- 172.21.15.46:0/1883108905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f070c0ca0e0 0x7f070c15a090 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f06fc009870 tx=0x7f06fc01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1dff4bd04fc3947 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:08.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.627+0000 7f0703fff640 1 -- 172.21.15.46:0/1883108905 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0a8a30 con 0x7f070c0ca0e0 2023-10-30T13:31:08.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.627+0000 7f0703fff640 1 -- 172.21.15.46:0/1883108905 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc0a8b90 con 0x7f070c0ca0e0 2023-10-30T13:31:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.627+0000 7f0703fff640 1 -- 172.21.15.46:0/1883108905 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0b0440 con 0x7f070c0ca0e0 2023-10-30T13:31:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.627+0000 7f071366b640 1 -- 172.21.15.46:0/1883108905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f070c0ca0e0 msgr2=0x7f070c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.627+0000 7f071366b640 1 --2- 172.21.15.46:0/1883108905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f070c0ca0e0 0x7f070c15a090 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f06fc009870 tx=0x7f06fc01ed30 comp rx=0 tx=0).stop 2023-10-30T13:31:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.627+0000 7f071366b640 1 -- 172.21.15.46:0/1883108905 shutdown_connections 2023-10-30T13:31:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.627+0000 7f071366b640 1 --2- 172.21.15.46:0/1883108905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f070c0ca0e0 0x7f070c15a090 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.627+0000 7f071366b640 1 --2- 172.21.15.46:0/1883108905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f070c1538a0 0x7f070c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.627+0000 7f071366b640 1 -- 172.21.15.46:0/1883108905 >> 172.21.15.46:0/1883108905 conn(0x7f070c06bfe0 msgr2=0x7f070c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.627+0000 7f071366b640 1 -- 172.21.15.46:0/1883108905 shutdown_connections 2023-10-30T13:31:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.627+0000 7f071366b640 1 -- 172.21.15.46:0/1883108905 wait complete. 2023-10-30T13:31:08.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f071366b640 1 Processor -- start 2023-10-30T13:31:08.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f071366b640 1 -- start start 2023-10-30T13:31:08.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f071366b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f070c0ca0e0 0x7f070c147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:08.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f071366b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f070c0cf160 0x7f070c15c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:08.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f071366b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f070c1538a0 0x7f070c15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f071366b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070c1f0c50 con 0x7f070c0cf160 2023-10-30T13:31:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f071366b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070c1436f0 con 0x7f070c1538a0 2023-10-30T13:31:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f071366b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070c15acf0 con 0x7f070c0ca0e0 2023-10-30T13:31:08.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0710c06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f070c0cf160 0x7f070c15c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:08.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0710c06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f070c0cf160 0x7f070c15c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59298/0 (socket says 172.21.15.46:59298) 2023-10-30T13:31:08.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0710c06640 1 -- 172.21.15.46:0/2962777050 learned_addr learned my addr 172.21.15.46:0/2962777050 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:08.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0711407640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f070c0ca0e0 0x7f070c147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:08.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0711c08640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f070c1538a0 0x7f070c15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:08.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0710c06640 1 -- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f070c0ca0e0 msgr2=0x7f070c147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0710c06640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f070c0ca0e0 0x7f070c147e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0710c06640 1 -- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f070c1538a0 msgr2=0x7f070c15ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0710c06640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f070c1538a0 0x7f070c15ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0710c06640 1 -- 172.21.15.46:0/2962777050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc0094d0 con 0x7f070c0cf160 2023-10-30T13:31:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0711c08640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f070c1538a0 0x7f070c15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0710c06640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f070c0cf160 0x7f070c15c500 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0b3040 tx=0x7f06fc01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0701ffb640 1 -- 172.21.15.46:0/2962777050 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0be070 con 0x7f070c0cf160 2023-10-30T13:31:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0701ffb640 1 -- 172.21.15.46:0/2962777050 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc0ba400 con 0x7f070c0cf160 2023-10-30T13:31:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f0701ffb640 1 -- 172.21.15.46:0/2962777050 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0ba7d0 con 0x7f070c0cf160 2023-10-30T13:31:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f071366b640 1 -- 172.21.15.46:0/2962777050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070c15afd0 con 0x7f070c0cf160 2023-10-30T13:31:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.631+0000 7f071366b640 1 -- 172.21.15.46:0/2962777050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f070c15b590 con 0x7f070c0cf160 2023-10-30T13:31:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.635+0000 7f071366b640 1 -- 172.21.15.46:0/2962777050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f070c0b2a50 con 0x7f070c0cf160 2023-10-30T13:31:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.635+0000 7f0701ffb640 1 -- 172.21.15.46:0/2962777050 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06fc0c3410 con 0x7f070c0cf160 2023-10-30T13:31:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.635+0000 7f0701ffb640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06f00664a0 0x7f06f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.635+0000 7f0701ffb640 1 -- 172.21.15.46:0/2962777050 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f06fc121cc0 con 0x7f070c0cf160 2023-10-30T13:31:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.635+0000 7f0711407640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06f00664a0 0x7f06f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.639+0000 7f0711407640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06f00664a0 0x7f06f0068960 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07080045f0 tx=0x7f0708008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:08.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.639+0000 7f0701ffb640 1 -- 172.21.15.46:0/2962777050 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06fc128080 con 0x7f070c0cf160 2023-10-30T13:31:08.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.827+0000 7f071366b640 1 -- 172.21.15.46:0/2962777050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f070c1f0d90 con 0x7f070c0cf160 2023-10-30T13:31:08.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.827+0000 7f0701ffb640 1 -- 172.21.15.46:0/2962777050 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f070c1f0d90 con 0x7f070c0cf160 2023-10-30T13:31:08.833 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:08.833 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:01.861512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2023-10-30T13:31:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.831+0000 7f071366b640 1 -- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06f00664a0 msgr2=0x7f06f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.831+0000 7f071366b640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06f00664a0 0x7f06f0068960 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07080045f0 tx=0x7f0708008040 comp rx=0 tx=0).stop 2023-10-30T13:31:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.831+0000 7f071366b640 1 -- 172.21.15.46:0/2962777050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f070c0cf160 msgr2=0x7f070c15c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.831+0000 7f071366b640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f070c0cf160 0x7f070c15c500 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0b3040 tx=0x7f06fc01bad0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.831+0000 7f071366b640 1 -- 172.21.15.46:0/2962777050 shutdown_connections 2023-10-30T13:31:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.831+0000 7f071366b640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06f00664a0 0x7f06f0068960 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.831+0000 7f071366b640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f070c1538a0 0x7f070c15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.831+0000 7f071366b640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f070c0cf160 0x7f070c15c500 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.831+0000 7f071366b640 1 --2- 172.21.15.46:0/2962777050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f070c0ca0e0 0x7f070c147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:08.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.831+0000 7f071366b640 1 -- 172.21.15.46:0/2962777050 >> 172.21.15.46:0/2962777050 conn(0x7f070c06bfe0 msgr2=0x7f070c0cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:08.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.831+0000 7f071366b640 1 -- 172.21.15.46:0/2962777050 shutdown_connections 2023-10-30T13:31:08.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:08.831+0000 7f071366b640 1 -- 172.21.15.46:0/2962777050 wait complete. 2023-10-30T13:31:08.854 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:31:08.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~0s 2023-10-30T13:31:10.855 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:11.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b1b11d640 1 Processor -- start 2023-10-30T13:31:11.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b1b11d640 1 -- start start 2023-10-30T13:31:11.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b1b11d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8b14069e90 0x7f8b1415a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:11.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b1b11d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8b1406a270 0x7f8b140c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:11.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b1b11d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b1406cb80 0x7f8b1406d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:11.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b1b11d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b140c8570 con 0x7f8b1406a270 2023-10-30T13:31:11.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b1b11d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b1406d550 con 0x7f8b14069e90 2023-10-30T13:31:11.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b1b11d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140c6c10 con 0x7f8b1406cb80 2023-10-30T13:31:11.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b13fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8b1406a270 0x7f8b140c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:11.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b13fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8b1406a270 0x7f8b140c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59304/0 (socket says 172.21.15.46:59304) 2023-10-30T13:31:11.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b196ba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b1406cb80 0x7f8b1406d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:11.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b18eb9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8b14069e90 0x7f8b1415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:11.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b13fff640 1 -- 172.21.15.46:0/4089297378 learned_addr learned my addr 172.21.15.46:0/4089297378 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:11.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b13fff640 1 -- 172.21.15.46:0/4089297378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b1406cb80 msgr2=0x7f8b1406d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:11.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b13fff640 1 --2- 172.21.15.46:0/4089297378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b1406cb80 0x7f8b1406d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:11.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b13fff640 1 -- 172.21.15.46:0/4089297378 >> v2:172.21.15.154:3300/0 conn(0x7f8b14069e90 msgr2=0x7f8b1415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:11.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b13fff640 1 --2- 172.21.15.46:0/4089297378 >> v2:172.21.15.154:3300/0 conn(0x7f8b14069e90 0x7f8b1415a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:11.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.007+0000 7f8b13fff640 1 -- 172.21.15.46:0/4089297378 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b140c6ef0 con 0x7f8b1406a270 2023-10-30T13:31:11.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b13fff640 1 --2- 172.21.15.46:0/4089297378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1406a270 0x7f8b140c9cb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8b1406ed20 tx=0x7f8b0001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e329fa6c254b625e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:11.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b137fe640 1 -- 172.21.15.46:0/4089297378 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b000ac410 con 0x7f8b1406a270 2023-10-30T13:31:11.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b137fe640 1 -- 172.21.15.46:0/4089297378 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b000ac570 con 0x7f8b1406a270 2023-10-30T13:31:11.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b137fe640 1 -- 172.21.15.46:0/4089297378 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b000ac900 con 0x7f8b1406a270 2023-10-30T13:31:11.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b1b11d640 1 -- 172.21.15.46:0/4089297378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1406a270 msgr2=0x7f8b140c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:11.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b1b11d640 1 --2- 172.21.15.46:0/4089297378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1406a270 0x7f8b140c9cb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8b1406ed20 tx=0x7f8b0001c500 comp rx=0 tx=0).stop 2023-10-30T13:31:11.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b1b11d640 1 -- 172.21.15.46:0/4089297378 shutdown_connections 2023-10-30T13:31:11.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b1b11d640 1 --2- 172.21.15.46:0/4089297378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b1406cb80 0x7f8b1406d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:11.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b1b11d640 1 --2- 172.21.15.46:0/4089297378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1406a270 0x7f8b140c9cb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:11.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b1b11d640 1 --2- 172.21.15.46:0/4089297378 >> v2:172.21.15.154:3300/0 conn(0x7f8b14069e90 0x7f8b1415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:11.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b1b11d640 1 -- 172.21.15.46:0/4089297378 >> 172.21.15.46:0/4089297378 conn(0x7f8b1406dfd0 msgr2=0x7f8b1406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:11.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b1b11d640 1 -- 172.21.15.46:0/4089297378 shutdown_connections 2023-10-30T13:31:11.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.011+0000 7f8b1b11d640 1 -- 172.21.15.46:0/4089297378 wait complete. 2023-10-30T13:31:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b1b11d640 1 Processor -- start 2023-10-30T13:31:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b1b11d640 1 -- start start 2023-10-30T13:31:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b1b11d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b14069e90 0x7f8b14155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b1b11d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1406a270 0x7f8b14156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b1b11d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b1406cb80 0x7f8b140ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b1b11d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b141598d0 con 0x7f8b1406a270 2023-10-30T13:31:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b1b11d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1414c0b0 con 0x7f8b14069e90 2023-10-30T13:31:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b1b11d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b141588c0 con 0x7f8b1406cb80 2023-10-30T13:31:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b13fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1406a270 0x7f8b14156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b13fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1406a270 0x7f8b14156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59314/0 (socket says 172.21.15.46:59314) 2023-10-30T13:31:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b13fff640 1 -- 172.21.15.46:0/1166805263 learned_addr learned my addr 172.21.15.46:0/1166805263 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b18eb9640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b14069e90 0x7f8b14155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b196ba640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b1406cb80 0x7f8b140ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b13fff640 1 -- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b1406cb80 msgr2=0x7f8b140ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b13fff640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b1406cb80 0x7f8b140ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b13fff640 1 -- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b14069e90 msgr2=0x7f8b14155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b13fff640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b14069e90 0x7f8b14155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b13fff640 1 -- 172.21.15.46:0/1166805263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b000094d0 con 0x7f8b1406a270 2023-10-30T13:31:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b18eb9640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b14069e90 0x7f8b14155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.015+0000 7f8b13fff640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1406a270 0x7f8b14156790 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8b0001abb0 tx=0x7f8b000192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.019+0000 7f8b117fa640 1 -- 172.21.15.46:0/1166805263 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b000ac410 con 0x7f8b1406a270 2023-10-30T13:31:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.019+0000 7f8b117fa640 1 -- 172.21.15.46:0/1166805263 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b000b7480 con 0x7f8b1406a270 2023-10-30T13:31:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.019+0000 7f8b117fa640 1 -- 172.21.15.46:0/1166805263 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b000b7830 con 0x7f8b1406a270 2023-10-30T13:31:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.019+0000 7f8b1b11d640 1 -- 172.21.15.46:0/1166805263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b14159320 con 0x7f8b1406a270 2023-10-30T13:31:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.019+0000 7f8b1b11d640 1 -- 172.21.15.46:0/1166805263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b14157ff0 con 0x7f8b1406a270 2023-10-30T13:31:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.019+0000 7f8b1b11d640 1 -- 172.21.15.46:0/1166805263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1404e170 con 0x7f8b1406a270 2023-10-30T13:31:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.019+0000 7f8b117fa640 1 -- 172.21.15.46:0/1166805263 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8b000b7ba0 con 0x7f8b1406a270 2023-10-30T13:31:11.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.019+0000 7f8b117fa640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8af80664a0 0x7f8af8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:11.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.019+0000 7f8b117fa640 1 -- 172.21.15.46:0/1166805263 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f8b0011ebf0 con 0x7f8b1406a270 2023-10-30T13:31:11.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.019+0000 7f8b18eb9640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8af80664a0 0x7f8af8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:11.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.023+0000 7f8b18eb9640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8af80664a0 0x7f8af8068960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009b30 tx=0x7f8b08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:11.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.023+0000 7f8b117fa640 1 -- 172.21.15.46:0/1166805263 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8b000ebdf0 con 0x7f8b1406a270 2023-10-30T13:31:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 -- 172.21.15.46:0/1166805263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b14158ed0 con 0x7f8b1406a270 2023-10-30T13:31:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b117fa640 1 -- 172.21.15.46:0/1166805263 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f8b000ef410 con 0x7f8b1406a270 2023-10-30T13:31:11.219 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:11.219 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:01.861512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2023-10-30T13:31:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 -- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8af80664a0 msgr2=0x7f8af8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8af80664a0 0x7f8af8068960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009b30 tx=0x7f8b08008040 comp rx=0 tx=0).stop 2023-10-30T13:31:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 -- 172.21.15.46:0/1166805263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1406a270 msgr2=0x7f8b14156790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1406a270 0x7f8b14156790 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8b0001abb0 tx=0x7f8b000192a0 comp rx=0 tx=0).stop 2023-10-30T13:31:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 -- 172.21.15.46:0/1166805263 shutdown_connections 2023-10-30T13:31:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8af80664a0 0x7f8af8068960 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b1406cb80 0x7f8b140ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1406a270 0x7f8b14156790 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 --2- 172.21.15.46:0/1166805263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b14069e90 0x7f8b14155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 -- 172.21.15.46:0/1166805263 >> 172.21.15.46:0/1166805263 conn(0x7f8b1406dfd0 msgr2=0x7f8b1415abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 -- 172.21.15.46:0/1166805263 shutdown_connections 2023-10-30T13:31:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:11.215+0000 7f8b1b11d640 1 -- 172.21.15.46:0/1166805263 wait complete. 2023-10-30T13:31:11.241 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:31:13.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:13.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.391+0000 7fc7f9780640 1 Processor -- start 2023-10-30T13:31:13.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.391+0000 7fc7f9780640 1 -- start start 2023-10-30T13:31:13.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f9780640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc7f40c6b80 0x7fc7f40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:13.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f9780640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f40c0a50 con 0x7fc7f40c6f60 2023-10-30T13:31:13.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f9780640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f406f6e0 con 0x7fc7f413f4a0 2023-10-30T13:31:13.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f9780640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f406f9c0 con 0x7fc7f40c6b80 2023-10-30T13:31:13.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7ebfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc7f40c6f60 0x7fc7f413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44006/0 (socket says 172.21.15.46:44006) 2023-10-30T13:31:13.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7ebfff640 1 -- 172.21.15.46:0/567880433 learned_addr learned my addr 172.21.15.46:0/567880433 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:31:13.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f2ffd640 1 --2- 172.21.15.46:0/567880433 >> v2:172.21.15.154:3300/0 conn(0x7fc7f40c6b80 0x7fc7f40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:13.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864496706 0 0) 0x7fc7f40c0a50 con 0x7fc7f40c6f60 2023-10-30T13:31:13.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e0003530 con 0x7fc7f40c6f60 2023-10-30T13:31:13.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3834288721 0 0) 0x7fc7e0003530 con 0x7fc7f40c6f60 2023-10-30T13:31:13.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 >> v2:172.21.15.154:3300/0 conn(0x7fc7f40c6b80 msgr2=0x7fc7f40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:13.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 --2- 172.21.15.46:0/567880433 >> v2:172.21.15.154:3300/0 conn(0x7fc7f40c6b80 0x7fc7f40bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:13.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 >> v1:172.21.15.154:6789/0 conn(0x7fc7f413f4a0 legacy=0x7fc7f41566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:13.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f406fca0 con 0x7fc7f40c6f60 2023-10-30T13:31:13.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fc7dc001680 con 0x7fc7f40c6f60 2023-10-30T13:31:13.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f2ffd640 1 --2- 172.21.15.46:0/567880433 >> v2:172.21.15.154:3300/0 conn(0x7fc7f40c6b80 0x7fc7f40bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:31:13.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 >> v1:172.21.15.46:6789/0 conn(0x7fc7f40c6f60 legacy=0x7fc7f413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:13.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 --2- 172.21.15.46:0/567880433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e0003980 0x7fc7e0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:13.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 --2- 172.21.15.46:0/567880433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e00047d0 0x7fc7e0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:13.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 --2- 172.21.15.46:0/567880433 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e0005190 0x7fc7e000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:13.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f406f9c0 con 0x7fc7e0003980 2023-10-30T13:31:13.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e000c030 con 0x7fc7e00047d0 2023-10-30T13:31:13.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e000e400 con 0x7fc7e0005190 2023-10-30T13:31:13.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f2ffd640 1 --2- 172.21.15.46:0/567880433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e0003980 0x7fc7e0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:13.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7ebfff640 1 --2- 172.21.15.46:0/567880433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e00047d0 0x7fc7e0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:13.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f37fe640 1 --2- 172.21.15.46:0/567880433 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e0005190 0x7fc7e000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:13.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f2ffd640 1 -- 172.21.15.46:0/567880433 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e0005190 msgr2=0x7fc7e000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:13.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f2ffd640 1 --2- 172.21.15.46:0/567880433 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e0005190 0x7fc7e000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:13.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f2ffd640 1 -- 172.21.15.46:0/567880433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e00047d0 msgr2=0x7fc7e0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:13.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f2ffd640 1 --2- 172.21.15.46:0/567880433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e00047d0 0x7fc7e0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:13.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f2ffd640 1 -- 172.21.15.46:0/567880433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e0003530 con 0x7fc7e0003980 2023-10-30T13:31:13.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.395+0000 7fc7f37fe640 1 -- 172.21.15.46:0/567880433 reap_dead start 2023-10-30T13:31:13.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.399+0000 7fc7f2ffd640 1 --2- 172.21.15.46:0/567880433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e0003980 0x7fc7e0003da0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4009cb0 tx=0x7fc7e401cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f609112dc5a855 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.399+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e40a57a0 con 0x7fc7e0003980 2023-10-30T13:31:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.399+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e40a5900 con 0x7fc7e0003980 2023-10-30T13:31:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.399+0000 7fc7f27fc640 1 -- 172.21.15.46:0/567880433 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e40a5cf0 con 0x7fc7e0003980 2023-10-30T13:31:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.399+0000 7fc7f9780640 1 -- 172.21.15.46:0/567880433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e0003980 msgr2=0x7fc7e0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.399+0000 7fc7f9780640 1 --2- 172.21.15.46:0/567880433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e0003980 0x7fc7e0003da0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4009cb0 tx=0x7fc7e401cbf0 comp rx=0 tx=0).stop 2023-10-30T13:31:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.399+0000 7fc7f9780640 1 -- 172.21.15.46:0/567880433 shutdown_connections 2023-10-30T13:31:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.399+0000 7fc7f9780640 1 --2- 172.21.15.46:0/567880433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e0003980 0x7fc7e0003da0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.399+0000 7fc7f9780640 1 -- 172.21.15.46:0/567880433 >> 172.21.15.46:0/567880433 conn(0x7fc7f40c7910 msgr2=0x7fc7f40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.399+0000 7fc7f9780640 1 -- 172.21.15.46:0/567880433 shutdown_connections 2023-10-30T13:31:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.399+0000 7fc7f9780640 1 -- 172.21.15.46:0/567880433 wait complete. 2023-10-30T13:31:13.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f9780640 1 Processor -- start 2023-10-30T13:31:13.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f9780640 1 -- start start 2023-10-30T13:31:13.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f9780640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e0003980 0x7fc7f4159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:13.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f9780640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f4138360 0x7fc7f413d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:13.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f9780640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f4137840 0x7fc7f4137cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:13.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f9780640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f4138200 con 0x7fc7f4137840 2023-10-30T13:31:13.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f9780640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f4158ee0 con 0x7fc7e0003980 2023-10-30T13:31:13.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f9780640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f415a320 con 0x7fc7f4138360 2023-10-30T13:31:13.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f4137840 0x7fc7f4137cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:13.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f4137840 0x7fc7f4137cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59328/0 (socket says 172.21.15.46:59328) 2023-10-30T13:31:13.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f37fe640 1 -- 172.21.15.46:0/1167935552 learned_addr learned my addr 172.21.15.46:0/1167935552 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:13.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f2ffd640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e0003980 0x7fc7f4159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:13.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7ebfff640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f4138360 0x7fc7f413d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:13.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f37fe640 1 -- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f4138360 msgr2=0x7fc7f413d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:13.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f37fe640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f4138360 0x7fc7f413d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:13.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f37fe640 1 -- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e0003980 msgr2=0x7fc7f4159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f37fe640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e0003980 0x7fc7f4159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f37fe640 1 -- 172.21.15.46:0/1167935552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f41eecf0 con 0x7fc7f4137840 2023-10-30T13:31:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7ebfff640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f4138360 0x7fc7f413d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f37fe640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f4137840 0x7fc7f4137cc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc7f406f4d0 tx=0x7fc7ec00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7eb7fe640 1 -- 172.21.15.46:0/1167935552 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec00eed0 con 0x7fc7f4137840 2023-10-30T13:31:13.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7eb7fe640 1 -- 172.21.15.46:0/1167935552 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ec003980 con 0x7fc7f4137840 2023-10-30T13:31:13.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7eb7fe640 1 -- 172.21.15.46:0/1167935552 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec003d30 con 0x7fc7f4137840 2023-10-30T13:31:13.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f9780640 1 -- 172.21.15.46:0/1167935552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f41eefd0 con 0x7fc7f4137840 2023-10-30T13:31:13.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.403+0000 7fc7f9780640 1 -- 172.21.15.46:0/1167935552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f415a850 con 0x7fc7f4137840 2023-10-30T13:31:13.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.407+0000 7fc7f9780640 1 -- 172.21.15.46:0/1167935552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f415aa10 con 0x7fc7f4137840 2023-10-30T13:31:13.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.407+0000 7fc7eb7fe640 1 -- 172.21.15.46:0/1167935552 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7ec0050d0 con 0x7fc7f4137840 2023-10-30T13:31:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.407+0000 7fc7eb7fe640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d40664a0 0x7fc7d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.407+0000 7fc7eb7fe640 1 -- 172.21.15.46:0/1167935552 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fc7ec083c70 con 0x7fc7f4137840 2023-10-30T13:31:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.407+0000 7fc7f2ffd640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d40664a0 0x7fc7d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.407+0000 7fc7f2ffd640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d40664a0 0x7fc7d4068960 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc7e40af000 tx=0x7fc7e40a7040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.411+0000 7fc7eb7fe640 1 -- 172.21.15.46:0/1167935552 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7ec050ef0 con 0x7fc7f4137840 2023-10-30T13:31:13.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.603+0000 7fc7f9780640 1 -- 172.21.15.46:0/1167935552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7f415aa10 con 0x7fc7f4137840 2023-10-30T13:31:13.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.603+0000 7fc7eb7fe640 1 -- 172.21.15.46:0/1167935552 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fc7ec054510 con 0x7fc7f4137840 2023-10-30T13:31:13.608 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:13.609 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:01.861512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:13.609 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2023-10-30T13:31:13.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.607+0000 7fc7f9780640 1 -- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d40664a0 msgr2=0x7fc7d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:13.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.607+0000 7fc7f9780640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d40664a0 0x7fc7d4068960 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc7e40af000 tx=0x7fc7e40a7040 comp rx=0 tx=0).stop 2023-10-30T13:31:13.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.607+0000 7fc7f9780640 1 -- 172.21.15.46:0/1167935552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f4137840 msgr2=0x7fc7f4137cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:13.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.607+0000 7fc7f9780640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f4137840 0x7fc7f4137cc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc7f406f4d0 tx=0x7fc7ec00eb90 comp rx=0 tx=0).stop 2023-10-30T13:31:13.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.607+0000 7fc7f9780640 1 -- 172.21.15.46:0/1167935552 shutdown_connections 2023-10-30T13:31:13.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.607+0000 7fc7f9780640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7d40664a0 0x7fc7d4068960 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:13.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.607+0000 7fc7f9780640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f4137840 0x7fc7f4137cc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:13.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.607+0000 7fc7f9780640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f4138360 0x7fc7f413d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:13.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.607+0000 7fc7f9780640 1 --2- 172.21.15.46:0/1167935552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e0003980 0x7fc7f4159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:13.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.607+0000 7fc7f9780640 1 -- 172.21.15.46:0/1167935552 >> 172.21.15.46:0/1167935552 conn(0x7fc7f40c7910 msgr2=0x7fc7f40c19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:13.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.607+0000 7fc7f9780640 1 -- 172.21.15.46:0/1167935552 shutdown_connections 2023-10-30T13:31:13.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:13.607+0000 7fc7f9780640 1 -- 172.21.15.46:0/1167935552 wait complete. 2023-10-30T13:31:13.630 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:31:15.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:15.759 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e has been restored 2023-10-30T13:31:15.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c28a74640 1 Processor -- start 2023-10-30T13:31:15.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c28a74640 1 -- start start 2023-10-30T13:31:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c28a74640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c241538f0 0x7f2c24159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c28a74640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c240ca6a0 con 0x7f2c240cefa0 2023-10-30T13:31:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c28a74640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2415d300 con 0x7f2c2415cd50 2023-10-30T13:31:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c28a74640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2415d5e0 con 0x7f2c241538f0 2023-10-30T13:31:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c23fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2c240cefa0 0x7f2c241640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44022/0 (socket says 172.21.15.46:44022) 2023-10-30T13:31:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c23fff640 1 -- 172.21.15.46:0/2452506291 learned_addr learned my addr 172.21.15.46:0/2452506291 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:31:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c237fe640 1 --2- 172.21.15.46:0/2452506291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c241538f0 0x7f2c24159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c227fc640 1 -- 172.21.15.46:0/2452506291 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907846959 0 0) 0x7f2c240ca6a0 con 0x7f2c240cefa0 2023-10-30T13:31:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c227fc640 1 -- 172.21.15.46:0/2452506291 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c0c003530 con 0x7f2c240cefa0 2023-10-30T13:31:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c227fc640 1 -- 172.21.15.46:0/2452506291 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410417631 0 0) 0x7f2c2415d300 con 0x7f2c2415cd50 2023-10-30T13:31:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c227fc640 1 -- 172.21.15.46:0/2452506291 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c240ca6a0 con 0x7f2c2415cd50 2023-10-30T13:31:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c237fe640 1 -- 172.21.15.46:0/2452506291 >> v1:172.21.15.154:6789/0 conn(0x7f2c2415cd50 legacy=0x7f2c240c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c237fe640 1 -- 172.21.15.46:0/2452506291 >> v1:172.21.15.46:6789/0 conn(0x7f2c240cefa0 legacy=0x7f2c241640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.783+0000 7f2c237fe640 1 -- 172.21.15.46:0/2452506291 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2415d8c0 con 0x7f2c241538f0 2023-10-30T13:31:15.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.787+0000 7f2c237fe640 1 --2- 172.21.15.46:0/2452506291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c241538f0 0x7f2c24159fb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c140097f0 tx=0x7f2c1401ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6378ac9801355a37 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:15.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.787+0000 7f2c227fc640 1 -- 172.21.15.46:0/2452506291 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c140a89b0 con 0x7f2c241538f0 2023-10-30T13:31:15.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.787+0000 7f2c227fc640 1 -- 172.21.15.46:0/2452506291 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c140a8b10 con 0x7f2c241538f0 2023-10-30T13:31:15.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.787+0000 7f2c227fc640 1 -- 172.21.15.46:0/2452506291 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c140a8ea0 con 0x7f2c241538f0 2023-10-30T13:31:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 -- 172.21.15.46:0/2452506291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c241538f0 msgr2=0x7f2c24159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 --2- 172.21.15.46:0/2452506291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c241538f0 0x7f2c24159fb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c140097f0 tx=0x7f2c1401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:31:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 -- 172.21.15.46:0/2452506291 shutdown_connections 2023-10-30T13:31:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 --2- 172.21.15.46:0/2452506291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c241538f0 0x7f2c24159fb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 -- 172.21.15.46:0/2452506291 >> 172.21.15.46:0/2452506291 conn(0x7f2c2406a640 msgr2=0x7f2c24065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 -- 172.21.15.46:0/2452506291 shutdown_connections 2023-10-30T13:31:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 -- 172.21.15.46:0/2452506291 wait complete. 2023-10-30T13:31:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 Processor -- start 2023-10-30T13:31:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 -- start start 2023-10-30T13:31:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c240cefa0 0x7f2c2414f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c241538f0 0x7f2c24147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415cd50 0x7f2c24147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c241f3d50 con 0x7f2c2415cd50 2023-10-30T13:31:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c241f3160 con 0x7f2c240cefa0 2023-10-30T13:31:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c28a74640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2414df50 con 0x7f2c241538f0 2023-10-30T13:31:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415cd50 0x7f2c24147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415cd50 0x7f2c24147fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59334/0 (socket says 172.21.15.46:59334) 2023-10-30T13:31:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c23fff640 1 -- 172.21.15.46:0/1326986865 learned_addr learned my addr 172.21.15.46:0/1326986865 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c237fe640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c240cefa0 0x7f2c2414f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c22ffd640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c241538f0 0x7f2c24147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c23fff640 1 -- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c241538f0 msgr2=0x7f2c24147aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c23fff640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c241538f0 0x7f2c24147aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c23fff640 1 -- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c240cefa0 msgr2=0x7f2c2414f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c23fff640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c240cefa0 0x7f2c2414f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.791+0000 7f2c23fff640 1 -- 172.21.15.46:0/1326986865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c14009420 con 0x7f2c2415cd50 2023-10-30T13:31:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.795+0000 7f2c23fff640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415cd50 0x7f2c24147fe0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00cba0 tx=0x7f2c1c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:15.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.795+0000 7f2c0bfff640 1 -- 172.21.15.46:0/1326986865 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c00ace0 con 0x7f2c2415cd50 2023-10-30T13:31:15.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.795+0000 7f2c0bfff640 1 -- 172.21.15.46:0/1326986865 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1c00ae40 con 0x7f2c2415cd50 2023-10-30T13:31:15.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.795+0000 7f2c28a74640 1 -- 172.21.15.46:0/1326986865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c241f3420 con 0x7f2c2415cd50 2023-10-30T13:31:15.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.795+0000 7f2c0bfff640 1 -- 172.21.15.46:0/1326986865 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c012620 con 0x7f2c2415cd50 2023-10-30T13:31:15.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.795+0000 7f2c28a74640 1 -- 172.21.15.46:0/1326986865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c241f39e0 con 0x7f2c2415cd50 2023-10-30T13:31:15.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.795+0000 7f2c28a74640 1 -- 172.21.15.46:0/1326986865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c24159a70 con 0x7f2c2415cd50 2023-10-30T13:31:15.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.795+0000 7f2c0bfff640 1 -- 172.21.15.46:0/1326986865 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c1c007070 con 0x7f2c2415cd50 2023-10-30T13:31:15.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.795+0000 7f2c0bfff640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c100664a0 0x7f2c10068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:15.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.795+0000 7f2c0bfff640 1 -- 172.21.15.46:0/1326986865 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f2c1c084170 con 0x7f2c2415cd50 2023-10-30T13:31:15.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.795+0000 7f2c237fe640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c100664a0 0x7f2c10068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:15.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.799+0000 7f2c237fe640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c100664a0 0x7f2c10068960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2c140b2000 tx=0x7f2c14008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:15.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.799+0000 7f2c0bfff640 1 -- 172.21.15.46:0/1326986865 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c24159a70 con 0x7f2c2415cd50 2023-10-30T13:31:15.963 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:31:15.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:31:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 -- 172.21.15.46:0/1326986865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c241f3f00 con 0x7f2c2415cd50 2023-10-30T13:31:15.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c0bfff640 1 -- 172.21.15.46:0/1326986865 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f2c241f3f00 con 0x7f2c2415cd50 2023-10-30T13:31:15.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:15.999 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:01.861512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:15.999 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2023-10-30T13:31:16.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 -- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c100664a0 msgr2=0x7f2c10068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c100664a0 0x7f2c10068960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2c140b2000 tx=0x7f2c14008040 comp rx=0 tx=0).stop 2023-10-30T13:31:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 -- 172.21.15.46:0/1326986865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415cd50 msgr2=0x7f2c24147fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415cd50 0x7f2c24147fe0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00cba0 tx=0x7f2c1c00a430 comp rx=0 tx=0).stop 2023-10-30T13:31:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 -- 172.21.15.46:0/1326986865 shutdown_connections 2023-10-30T13:31:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c100664a0 0x7f2c10068960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415cd50 0x7f2c24147fe0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c241538f0 0x7f2c24147aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 --2- 172.21.15.46:0/1326986865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c240cefa0 0x7f2c2414f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 -- 172.21.15.46:0/1326986865 >> 172.21.15.46:0/1326986865 conn(0x7f2c2406a640 msgr2=0x7f2c240c4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 -- 172.21.15.46:0/1326986865 shutdown_connections 2023-10-30T13:31:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:15.995+0000 7f2c28a74640 1 -- 172.21.15.46:0/1326986865 wait complete. 2023-10-30T13:31:16.021 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:31:18.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:18.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.171+0000 7f29a8fab640 1 Processor -- start 2023-10-30T13:31:18.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.171+0000 7f29a8fab640 1 -- start start 2023-10-30T13:31:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a8fab640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29a40c6f60 0x7f29a41566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a8fab640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a40c0a50 con 0x7f29a40c05d0 2023-10-30T13:31:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a8fab640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a406f6e0 con 0x7f29a40c6b80 2023-10-30T13:31:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a8fab640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a406f9c0 con 0x7f29a40c6f60 2023-10-30T13:31:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f299bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f29a40c05d0 0x7f29a413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55146/0 (socket says 172.21.15.46:55146) 2023-10-30T13:31:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f299bfff640 1 -- 172.21.15.46:0/4204533026 learned_addr learned my addr 172.21.15.46:0/4204533026 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:31:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a2d9d640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29a40c6f60 0x7f29a41566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:18.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498375098 0 0) 0x7f29a40c0a50 con 0x7f29a40c05d0 2023-10-30T13:31:18.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2990003530 con 0x7f29a40c05d0 2023-10-30T13:31:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2332012717 0 0) 0x7f2990003530 con 0x7f29a40c05d0 2023-10-30T13:31:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29a40c6f60 msgr2=0x7f29a41566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29a40c6f60 0x7f29a41566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 >> v1:172.21.15.154:6789/0 conn(0x7f29a40c6b80 legacy=0x7f29a40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a406fca0 con 0x7f29a40c05d0 2023-10-30T13:31:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f298c001680 con 0x7f29a40c05d0 2023-10-30T13:31:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 >> v1:172.21.15.46:6789/0 conn(0x7f29a40c05d0 legacy=0x7f29a413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29900039b0 0x7f2990003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2990004360 0x7f2990004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2990004fe0 0x7f299000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a406f9c0 con 0x7f2990004fe0 2023-10-30T13:31:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299000bf00 con 0x7f29900039b0 2023-10-30T13:31:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299000c0c0 con 0x7f2990004360 2023-10-30T13:31:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a2d9d640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29a40c6f60 0x7f29a41566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:31:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a2d9d640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2990004fe0 0x7f299000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a259c640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29900039b0 0x7f2990003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f299bfff640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2990004360 0x7f2990004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a2d9d640 1 -- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2990004360 msgr2=0x7f2990004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a2d9d640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2990004360 0x7f2990004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a2d9d640 1 -- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29900039b0 msgr2=0x7f2990003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a2d9d640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29900039b0 0x7f2990003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f29a2d9d640 1 -- 172.21.15.46:0/4204533026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2990003530 con 0x7f2990004fe0 2023-10-30T13:31:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.175+0000 7f299bfff640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2990004360 0x7f2990004830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:18.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.179+0000 7f29a2d9d640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2990004fe0 0x7f299000b990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f29a40c7340 tx=0x7f299c01cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719aa7cb7a936019 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:18.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.179+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c0a5d50 con 0x7f2990004fe0 2023-10-30T13:31:18.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.179+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299c00f040 con 0x7f2990004fe0 2023-10-30T13:31:18.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.179+0000 7f29a1d9b640 1 -- 172.21.15.46:0/4204533026 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c003bc0 con 0x7f2990004fe0 2023-10-30T13:31:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.179+0000 7f29a8fab640 1 -- 172.21.15.46:0/4204533026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2990004fe0 msgr2=0x7f299000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.179+0000 7f29a8fab640 1 --2- 172.21.15.46:0/4204533026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2990004fe0 0x7f299000b990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f29a40c7340 tx=0x7f299c01cd00 comp rx=0 tx=0).stop 2023-10-30T13:31:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.179+0000 7f29a2d9d640 1 -- 172.21.15.46:0/4204533026 reap_dead start 2023-10-30T13:31:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.179+0000 7f29a8fab640 1 -- 172.21.15.46:0/4204533026 shutdown_connections 2023-10-30T13:31:18.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.179+0000 7f29a8fab640 1 -- 172.21.15.46:0/4204533026 >> 172.21.15.46:0/4204533026 conn(0x7f29a40c7910 msgr2=0x7f29a40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:18.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.179+0000 7f29a8fab640 1 -- 172.21.15.46:0/4204533026 shutdown_connections 2023-10-30T13:31:18.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.179+0000 7f29a8fab640 1 -- 172.21.15.46:0/4204533026 wait complete. 2023-10-30T13:31:18.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a8fab640 1 Processor -- start 2023-10-30T13:31:18.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a8fab640 1 -- start start 2023-10-30T13:31:18.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a8fab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2990004fe0 0x7f29a41ef000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:18.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a8fab640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29a40c8c10 0x7f29a413a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:18.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a8fab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29a413c7e0 0x7f29a413abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:18.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a8fab640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a413cf20 con 0x7f29a413c7e0 2023-10-30T13:31:18.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a8fab640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a41ee630 con 0x7f2990004fe0 2023-10-30T13:31:18.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a8fab640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a413e800 con 0x7f29a40c8c10 2023-10-30T13:31:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29a413c7e0 0x7f29a413abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29a413c7e0 0x7f29a413abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33146/0 (socket says 172.21.15.46:33146) 2023-10-30T13:31:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a2d9d640 1 -- 172.21.15.46:0/3175547363 learned_addr learned my addr 172.21.15.46:0/3175547363 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:18.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a259c640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2990004fe0 0x7f29a41ef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:18.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f299bfff640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29a40c8c10 0x7f29a413a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:18.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a2d9d640 1 -- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29a40c8c10 msgr2=0x7f29a413a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:18.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a2d9d640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29a40c8c10 0x7f29a413a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:18.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a2d9d640 1 -- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2990004fe0 msgr2=0x7f29a41ef000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:18.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a2d9d640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2990004fe0 0x7f29a41ef000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:18.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a2d9d640 1 -- 172.21.15.46:0/3175547363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a41ee840 con 0x7f29a413c7e0 2023-10-30T13:31:18.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a2d9d640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29a413c7e0 0x7f29a413abf0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f299c01cd40 tx=0x7f299c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:18.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f299affd640 1 -- 172.21.15.46:0/3175547363 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c0a5d50 con 0x7f29a413c7e0 2023-10-30T13:31:18.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f299affd640 1 -- 172.21.15.46:0/3175547363 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299c003e60 con 0x7f29a413c7e0 2023-10-30T13:31:18.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f299affd640 1 -- 172.21.15.46:0/3175547363 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c004230 con 0x7f29a413c7e0 2023-10-30T13:31:18.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a8fab640 1 -- 172.21.15.46:0/3175547363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a413e240 con 0x7f29a413c7e0 2023-10-30T13:31:18.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.183+0000 7f29a8fab640 1 -- 172.21.15.46:0/3175547363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a41eeae0 con 0x7f29a413c7e0 2023-10-30T13:31:18.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.187+0000 7f29a8fab640 1 -- 172.21.15.46:0/3175547363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a41eed00 con 0x7f29a413c7e0 2023-10-30T13:31:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.187+0000 7f299affd640 1 -- 172.21.15.46:0/3175547363 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f299c004390 con 0x7f29a413c7e0 2023-10-30T13:31:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.187+0000 7f299affd640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29840664a0 0x7f2984068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:18.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.187+0000 7f299affd640 1 -- 172.21.15.46:0/3175547363 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f299c11f6e0 con 0x7f29a413c7e0 2023-10-30T13:31:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.187+0000 7f29a259c640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29840664a0 0x7f2984068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.187+0000 7f29a259c640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29840664a0 0x7f2984068960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2994007fd0 tx=0x7f2994009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:18.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.191+0000 7f299affd640 1 -- 172.21.15.46:0/3175547363 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f299c0ec960 con 0x7f29a413c7e0 2023-10-30T13:31:18.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.379+0000 7f29a8fab640 1 -- 172.21.15.46:0/3175547363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29a41eed00 con 0x7f29a413c7e0 2023-10-30T13:31:18.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f299affd640 1 -- 172.21.15.46:0/3175547363 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f299c0eff80 con 0x7f29a413c7e0 2023-10-30T13:31:18.385 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:18.386 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:01.861512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:18.386 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2023-10-30T13:31:18.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f29a8fab640 1 -- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29840664a0 msgr2=0x7f2984068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:18.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f29a8fab640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29840664a0 0x7f2984068960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2994007fd0 tx=0x7f2994009040 comp rx=0 tx=0).stop 2023-10-30T13:31:18.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f29a8fab640 1 -- 172.21.15.46:0/3175547363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29a413c7e0 msgr2=0x7f29a413abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:18.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f29a8fab640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29a413c7e0 0x7f29a413abf0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f299c01cd40 tx=0x7f299c009750 comp rx=0 tx=0).stop 2023-10-30T13:31:18.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f29a8fab640 1 -- 172.21.15.46:0/3175547363 shutdown_connections 2023-10-30T13:31:18.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f29a8fab640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29840664a0 0x7f2984068960 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:18.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f29a8fab640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29a413c7e0 0x7f29a413abf0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:18.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f29a8fab640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29a40c8c10 0x7f29a413a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:18.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f29a8fab640 1 --2- 172.21.15.46:0/3175547363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2990004fe0 0x7f29a41ef000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:18.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f29a8fab640 1 -- 172.21.15.46:0/3175547363 >> 172.21.15.46:0/3175547363 conn(0x7f29a40c7910 msgr2=0x7f29a406bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:18.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f29a8fab640 1 -- 172.21.15.46:0/3175547363 shutdown_connections 2023-10-30T13:31:18.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:18.383+0000 7f29a8fab640 1 -- 172.21.15.46:0/3175547363 wait complete. 2023-10-30T13:31:18.407 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:31:20.408 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:20.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.559+0000 7f014c29e640 1 Processor -- start 2023-10-30T13:31:20.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.559+0000 7f014c29e640 1 -- start start 2023-10-30T13:31:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.559+0000 7f014c29e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f014415a410 0x7f014406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.559+0000 7f014c29e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f014406cb80 0x7f014406d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.559+0000 7f014c29e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014406d550 con 0x7f0144069e90 2023-10-30T13:31:20.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.559+0000 7f014c29e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01440c8570 con 0x7f014415a410 2023-10-30T13:31:20.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.559+0000 7f014c29e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014415a850 con 0x7f014406cb80 2023-10-30T13:31:20.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.559+0000 7f0149839640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f014415a410 0x7f014406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:20.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.559+0000 7f0149839640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f014415a410 0x7f014406a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33160/0 (socket says 172.21.15.46:33160) 2023-10-30T13:31:20.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.559+0000 7f0149839640 1 -- 172.21.15.46:0/2824368075 learned_addr learned my addr 172.21.15.46:0/2824368075 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.559+0000 7f014a03a640 1 --1- 172.21.15.46:0/2824368075 >> v1:172.21.15.46:6789/0 conn(0x7f0144069e90 0x7f01440c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55162/0 (socket says 172.21.15.46:55162) 2023-10-30T13:31:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f014a83b640 1 --2- 172.21.15.46:0/2824368075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f014406cb80 0x7f014406d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f0149839640 1 -- 172.21.15.46:0/2824368075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f014406cb80 msgr2=0x7f014406d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f0149839640 1 --2- 172.21.15.46:0/2824368075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f014406cb80 0x7f014406d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f0149839640 1 -- 172.21.15.46:0/2824368075 >> v1:172.21.15.46:6789/0 conn(0x7f0144069e90 legacy=0x7f01440c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:20.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f0149839640 1 -- 172.21.15.46:0/2824368075 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01440c6c10 con 0x7f014415a410 2023-10-30T13:31:20.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f014a83b640 1 --2- 172.21.15.46:0/2824368075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f014406cb80 0x7f014406d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:20.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f0149038640 1 -- 172.21.15.46:0/2824368075 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387047206 0 0) 0x7f014406d550 con 0x7f0144069e90 2023-10-30T13:31:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f0149839640 1 --2- 172.21.15.46:0/2824368075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014415a410 0x7f014406a270 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0134009870 tx=0x7f013401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44cb51ef6db48ae5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f0149038640 1 -- 172.21.15.46:0/2824368075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01340ab600 con 0x7f014415a410 2023-10-30T13:31:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f0149038640 1 -- 172.21.15.46:0/2824368075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01340a7050 con 0x7f014415a410 2023-10-30T13:31:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f0149038640 1 -- 172.21.15.46:0/2824368075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01340ab980 con 0x7f014415a410 2023-10-30T13:31:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f014c29e640 1 -- 172.21.15.46:0/2824368075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014415a410 msgr2=0x7f014406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.563+0000 7f014c29e640 1 --2- 172.21.15.46:0/2824368075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014415a410 0x7f014406a270 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0134009870 tx=0x7f013401c540 comp rx=0 tx=0).stop 2023-10-30T13:31:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 -- 172.21.15.46:0/2824368075 shutdown_connections 2023-10-30T13:31:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 --2- 172.21.15.46:0/2824368075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f014406cb80 0x7f014406d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:20.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 --2- 172.21.15.46:0/2824368075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014415a410 0x7f014406a270 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:20.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 -- 172.21.15.46:0/2824368075 >> 172.21.15.46:0/2824368075 conn(0x7f014406dfd0 msgr2=0x7f014406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:20.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 -- 172.21.15.46:0/2824368075 shutdown_connections 2023-10-30T13:31:20.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 -- 172.21.15.46:0/2824368075 wait complete. 2023-10-30T13:31:20.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 Processor -- start 2023-10-30T13:31:20.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 -- start start 2023-10-30T13:31:20.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0144069e90 0x7f0144155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:20.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014406cb80 0x7f0144158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:20.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f014415a410 0x7f0144154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:20.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0144153c00 con 0x7f0144069e90 2023-10-30T13:31:20.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0144152a50 con 0x7f014406cb80 2023-10-30T13:31:20.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014c29e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01441556b0 con 0x7f014415a410 2023-10-30T13:31:20.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014a03a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0144069e90 0x7f0144155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:20.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014a03a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0144069e90 0x7f0144155170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33164/0 (socket says 172.21.15.46:33164) 2023-10-30T13:31:20.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014a03a640 1 -- 172.21.15.46:0/2235369001 learned_addr learned my addr 172.21.15.46:0/2235369001 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:20.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014a83b640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f014415a410 0x7f0144154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:20.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f0149839640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014406cb80 0x7f0144158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014a03a640 1 -- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f014415a410 msgr2=0x7f0144154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014a03a640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f014415a410 0x7f0144154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014a03a640 1 -- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014406cb80 msgr2=0x7f0144158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014a03a640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014406cb80 0x7f0144158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014a03a640 1 -- 172.21.15.46:0/2235369001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01340094d0 con 0x7f0144069e90 2023-10-30T13:31:20.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014a83b640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f014415a410 0x7f0144154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:20.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.567+0000 7f014a03a640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0144069e90 0x7f0144155170 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f01400011f0 tx=0x7f014000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:20.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.571+0000 7f013affd640 1 -- 172.21.15.46:0/2235369001 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01400074e0 con 0x7f0144069e90 2023-10-30T13:31:20.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.571+0000 7f013affd640 1 -- 172.21.15.46:0/2235369001 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0140007640 con 0x7f0144069e90 2023-10-30T13:31:20.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.571+0000 7f013affd640 1 -- 172.21.15.46:0/2235369001 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140004a10 con 0x7f0144069e90 2023-10-30T13:31:20.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.571+0000 7f014c29e640 1 -- 172.21.15.46:0/2235369001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0144151cf0 con 0x7f0144069e90 2023-10-30T13:31:20.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.571+0000 7f014c29e640 1 -- 172.21.15.46:0/2235369001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01441513c0 con 0x7f0144069e90 2023-10-30T13:31:20.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.571+0000 7f014c29e640 1 -- 172.21.15.46:0/2235369001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014404e170 con 0x7f0144069e90 2023-10-30T13:31:20.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.571+0000 7f013affd640 1 -- 172.21.15.46:0/2235369001 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0140004b70 con 0x7f0144069e90 2023-10-30T13:31:20.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.571+0000 7f013affd640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01280664a0 0x7f0128068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:20.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.571+0000 7f013affd640 1 -- 172.21.15.46:0/2235369001 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f01400833b0 con 0x7f0144069e90 2023-10-30T13:31:20.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.571+0000 7f0149839640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01280664a0 0x7f0128068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:20.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.575+0000 7f0149839640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01280664a0 0x7f0128068960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f014406ed20 tx=0x7f0134008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:20.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.575+0000 7f013affd640 1 -- 172.21.15.46:0/2235369001 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0140050630 con 0x7f0144069e90 2023-10-30T13:31:20.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.767+0000 7f014c29e640 1 -- 172.21.15.46:0/2235369001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f014406ef60 con 0x7f0144069e90 2023-10-30T13:31:20.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.767+0000 7f013affd640 1 -- 172.21.15.46:0/2235369001 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f0140018090 con 0x7f0144069e90 2023-10-30T13:31:20.773 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:20.773 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:01.861512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:20.773 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2023-10-30T13:31:20.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.771+0000 7f014c29e640 1 -- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01280664a0 msgr2=0x7f0128068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:20.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.771+0000 7f014c29e640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01280664a0 0x7f0128068960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f014406ed20 tx=0x7f0134008040 comp rx=0 tx=0).stop 2023-10-30T13:31:20.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.771+0000 7f014c29e640 1 -- 172.21.15.46:0/2235369001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0144069e90 msgr2=0x7f0144155170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:20.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.771+0000 7f014c29e640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0144069e90 0x7f0144155170 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f01400011f0 tx=0x7f014000c9b0 comp rx=0 tx=0).stop 2023-10-30T13:31:20.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.771+0000 7f014c29e640 1 -- 172.21.15.46:0/2235369001 shutdown_connections 2023-10-30T13:31:20.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.771+0000 7f014c29e640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01280664a0 0x7f0128068960 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:20.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.771+0000 7f014c29e640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f014415a410 0x7f0144154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:20.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.771+0000 7f014c29e640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014406cb80 0x7f0144158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:20.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.771+0000 7f014c29e640 1 --2- 172.21.15.46:0/2235369001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0144069e90 0x7f0144155170 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:20.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.771+0000 7f014c29e640 1 -- 172.21.15.46:0/2235369001 >> 172.21.15.46:0/2235369001 conn(0x7f014406dfd0 msgr2=0x7f01440c9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:20.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.771+0000 7f014c29e640 1 -- 172.21.15.46:0/2235369001 shutdown_connections 2023-10-30T13:31:20.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:20.771+0000 7f014c29e640 1 -- 172.21.15.46:0/2235369001 wait complete. 2023-10-30T13:31:20.794 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:31:22.795 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:22.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f844e4d6640 1 Processor -- start 2023-10-30T13:31:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f844e4d6640 1 -- start start 2023-10-30T13:31:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f844e4d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f84481538a0 0x7f844815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f844e4d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84480ca0e0 0x7f844815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f844e4d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84480cb800 con 0x7f84480cf160 2023-10-30T13:31:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f844e4d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844815dc30 con 0x7f84481538a0 2023-10-30T13:31:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f844e4d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84480cb330 con 0x7f84480ca0e0 2023-10-30T13:31:22.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f84477fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84480ca0e0 0x7f844815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:22.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f8447fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f84481538a0 0x7f844815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:22.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f84477fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84480ca0e0 0x7f844815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:55184/0 (socket says 172.21.15.46:55184) 2023-10-30T13:31:22.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f844ca73640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f84480cf160 0x7f84480cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:56504/0 (socket says 172.21.15.46:56504) 2023-10-30T13:31:22.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f84477fe640 1 -- 172.21.15.46:0/3311155988 learned_addr learned my addr 172.21.15.46:0/3311155988 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:22.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.947+0000 7f8446ffd640 1 -- 172.21.15.46:0/3311155988 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763410043 0 0) 0x7f84480cb800 con 0x7f84480cf160 2023-10-30T13:31:22.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.951+0000 7f8446ffd640 1 -- 172.21.15.46:0/3311155988 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8430003530 con 0x7f84480cf160 2023-10-30T13:31:22.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.951+0000 7f84477fe640 1 -- 172.21.15.46:0/3311155988 >> v2:172.21.15.154:3300/0 conn(0x7f84481538a0 msgr2=0x7f844815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:22.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.951+0000 7f84477fe640 1 --2- 172.21.15.46:0/3311155988 >> v2:172.21.15.154:3300/0 conn(0x7f84481538a0 0x7f844815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:22.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.951+0000 7f84477fe640 1 -- 172.21.15.46:0/3311155988 >> v1:172.21.15.154:6789/0 conn(0x7f84480cf160 legacy=0x7f84480cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:22.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.951+0000 7f84477fe640 1 -- 172.21.15.46:0/3311155988 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84480cad70 con 0x7f84480ca0e0 2023-10-30T13:31:22.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.951+0000 7f84477fe640 1 --2- 172.21.15.46:0/3311155988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84480ca0e0 0x7f844815a090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f843c009870 tx=0x7f843c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=958abe8db9a023fc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:22.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.951+0000 7f8446ffd640 1 -- 172.21.15.46:0/3311155988 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c0a8a30 con 0x7f84480ca0e0 2023-10-30T13:31:22.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.951+0000 7f8446ffd640 1 -- 172.21.15.46:0/3311155988 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c0a8b90 con 0x7f84480ca0e0 2023-10-30T13:31:22.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.951+0000 7f8446ffd640 1 -- 172.21.15.46:0/3311155988 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c0b0440 con 0x7f84480ca0e0 2023-10-30T13:31:22.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 -- 172.21.15.46:0/3311155988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84480ca0e0 msgr2=0x7f844815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:22.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 --2- 172.21.15.46:0/3311155988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84480ca0e0 0x7f844815a090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f843c009870 tx=0x7f843c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:31:22.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 -- 172.21.15.46:0/3311155988 shutdown_connections 2023-10-30T13:31:22.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 --2- 172.21.15.46:0/3311155988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84480ca0e0 0x7f844815a090 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:22.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 --2- 172.21.15.46:0/3311155988 >> v2:172.21.15.154:3300/0 conn(0x7f84481538a0 0x7f844815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:22.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 -- 172.21.15.46:0/3311155988 >> 172.21.15.46:0/3311155988 conn(0x7f844806bfe0 msgr2=0x7f844806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:22.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 -- 172.21.15.46:0/3311155988 shutdown_connections 2023-10-30T13:31:22.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 -- 172.21.15.46:0/3311155988 wait complete. 2023-10-30T13:31:22.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 Processor -- start 2023-10-30T13:31:22.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 -- start start 2023-10-30T13:31:22.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84480ca0e0 0x7f84480b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:22.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84480cf160 0x7f844815c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:22.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84481538a0 0x7f844815cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:22.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844814f4c0 con 0x7f84481538a0 2023-10-30T13:31:22.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84481eefc0 con 0x7f84480ca0e0 2023-10-30T13:31:22.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844e4d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844815a8b0 con 0x7f84480cf160 2023-10-30T13:31:22.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844ca73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84481538a0 0x7f844815cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:22.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844ca73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84481538a0 0x7f844815cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33168/0 (socket says 172.21.15.46:33168) 2023-10-30T13:31:22.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844ca73640 1 -- 172.21.15.46:0/1753793517 learned_addr learned my addr 172.21.15.46:0/1753793517 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:22.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f84477fe640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84480cf160 0x7f844815c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:22.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f8447fff640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84480ca0e0 0x7f84480b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:22.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844ca73640 1 -- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84480cf160 msgr2=0x7f844815c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:22.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844ca73640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84480cf160 0x7f844815c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:22.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844ca73640 1 -- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84480ca0e0 msgr2=0x7f84480b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:22.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844ca73640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84480ca0e0 0x7f84480b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:22.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.955+0000 7f844ca73640 1 -- 172.21.15.46:0/1753793517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c0094d0 con 0x7f84481538a0 2023-10-30T13:31:22.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.959+0000 7f844ca73640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84481538a0 0x7f844815cd40 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f844000c770 tx=0x7f844000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:22.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.959+0000 7f8444ff9640 1 -- 172.21.15.46:0/1753793517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844000ce80 con 0x7f84481538a0 2023-10-30T13:31:22.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.959+0000 7f8444ff9640 1 -- 172.21.15.46:0/1753793517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844000a5e0 con 0x7f84481538a0 2023-10-30T13:31:22.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.959+0000 7f8444ff9640 1 -- 172.21.15.46:0/1753793517 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844000a950 con 0x7f84481538a0 2023-10-30T13:31:22.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.959+0000 7f844e4d6640 1 -- 172.21.15.46:0/1753793517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84481ef1d0 con 0x7f84481538a0 2023-10-30T13:31:22.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.959+0000 7f844e4d6640 1 -- 172.21.15.46:0/1753793517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844815abb0 con 0x7f84481538a0 2023-10-30T13:31:22.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.959+0000 7f844e4d6640 1 -- 172.21.15.46:0/1753793517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84480b2a30 con 0x7f84481538a0 2023-10-30T13:31:22.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.959+0000 7f8444ff9640 1 -- 172.21.15.46:0/1753793517 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f844000aab0 con 0x7f84481538a0 2023-10-30T13:31:22.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.959+0000 7f8444ff9640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84340664a0 0x7f8434068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:22.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.959+0000 7f8444ff9640 1 -- 172.21.15.46:0/1753793517 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f84400837c0 con 0x7f84481538a0 2023-10-30T13:31:22.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.959+0000 7f8447fff640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84340664a0 0x7f8434068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:22.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.963+0000 7f8447fff640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84340664a0 0x7f8434068960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8438009ac0 tx=0x7f8438009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:22.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:22.963+0000 7f8444ff9640 1 -- 172.21.15.46:0/1753793517 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8440050af0 con 0x7f84481538a0 2023-10-30T13:31:23.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.155+0000 7f844e4d6640 1 -- 172.21.15.46:0/1753793517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84480cfdd0 con 0x7f84481538a0 2023-10-30T13:31:23.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.155+0000 7f8444ff9640 1 -- 172.21.15.46:0/1753793517 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f8440054110 con 0x7f84481538a0 2023-10-30T13:31:23.159 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:23.159 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:01.861512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":4950},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4950":{"gid":4950,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6837/1315246492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1315246492},{"type":"v1","addr":"172.21.15.46:6837","nonce":1315246492}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:23.160 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 45 2023-10-30T13:31:23.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.155+0000 7f844e4d6640 1 -- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84340664a0 msgr2=0x7f8434068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:23.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.155+0000 7f844e4d6640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84340664a0 0x7f8434068960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8438009ac0 tx=0x7f8438009190 comp rx=0 tx=0).stop 2023-10-30T13:31:23.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.155+0000 7f844e4d6640 1 -- 172.21.15.46:0/1753793517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84481538a0 msgr2=0x7f844815cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:23.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.155+0000 7f844e4d6640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84481538a0 0x7f844815cd40 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f844000c770 tx=0x7f844000cc30 comp rx=0 tx=0).stop 2023-10-30T13:31:23.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.155+0000 7f844e4d6640 1 -- 172.21.15.46:0/1753793517 shutdown_connections 2023-10-30T13:31:23.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.155+0000 7f844e4d6640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84340664a0 0x7f8434068960 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:23.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.155+0000 7f844e4d6640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84481538a0 0x7f844815cd40 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:23.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.155+0000 7f844e4d6640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84480cf160 0x7f844815c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.155+0000 7f844e4d6640 1 --2- 172.21.15.46:0/1753793517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84480ca0e0 0x7f84480b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.155+0000 7f844e4d6640 1 -- 172.21.15.46:0/1753793517 >> 172.21.15.46:0/1753793517 conn(0x7f844806bfe0 msgr2=0x7f844806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.159+0000 7f844e4d6640 1 -- 172.21.15.46:0/1753793517 shutdown_connections 2023-10-30T13:31:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:23.159+0000 7f844e4d6640 1 -- 172.21.15.46:0/1753793517 wait complete. 2023-10-30T13:31:23.181 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:31:25.182 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:25.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.331+0000 7f0e1c446640 1 Processor -- start 2023-10-30T13:31:25.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.331+0000 7f0e1c446640 1 -- start start 2023-10-30T13:31:25.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1c446640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e1406f7a0 0x7f0e1406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:25.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1c446640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0e140c6bd0 0x7f0e14069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:25.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1c446640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e1406a3d0 con 0x7f0e14071110 2023-10-30T13:31:25.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1c446640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e140700c0 con 0x7f0e1406f7a0 2023-10-30T13:31:25.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1c446640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e14066cf0 con 0x7f0e140c6bd0 2023-10-30T13:31:25.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1a1e2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e1406f7a0 0x7f0e1406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:25.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1a1e2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e1406f7a0 0x7f0e1406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33176/0 (socket says 172.21.15.46:33176) 2023-10-30T13:31:25.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1a1e2640 1 -- 172.21.15.46:0/4216815223 learned_addr learned my addr 172.21.15.46:0/4216815223 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:25.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e199e1640 1 --2- 172.21.15.46:0/4216815223 >> v2:172.21.15.154:3300/0 conn(0x7f0e140c6bd0 0x7f0e14069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:25.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1a1e2640 1 -- 172.21.15.46:0/4216815223 >> v2:172.21.15.154:3300/0 conn(0x7f0e140c6bd0 msgr2=0x7f0e14069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:25.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1a1e2640 1 --2- 172.21.15.46:0/4216815223 >> v2:172.21.15.154:3300/0 conn(0x7f0e140c6bd0 0x7f0e14069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1a1e2640 1 -- 172.21.15.46:0/4216815223 >> v1:172.21.15.154:6789/0 conn(0x7f0e14071110 legacy=0x7f0e14158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:25.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1a1e2640 1 -- 172.21.15.46:0/4216815223 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e140c70d0 con 0x7f0e1406f7a0 2023-10-30T13:31:25.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.335+0000 7f0e1a1e2640 1 --2- 172.21.15.46:0/4216815223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e1406f7a0 0x7f0e1406fb80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0e1406f1c0 tx=0x7f0e1001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e58d63f9b88215 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:25.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e191e0640 1 -- 172.21.15.46:0/4216815223 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100a89b0 con 0x7f0e1406f7a0 2023-10-30T13:31:25.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e191e0640 1 -- 172.21.15.46:0/4216815223 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e100a8b10 con 0x7f0e1406f7a0 2023-10-30T13:31:25.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e191e0640 1 -- 172.21.15.46:0/4216815223 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100a8ea0 con 0x7f0e1406f7a0 2023-10-30T13:31:25.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 -- 172.21.15.46:0/4216815223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e1406f7a0 msgr2=0x7f0e1406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:25.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 --2- 172.21.15.46:0/4216815223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e1406f7a0 0x7f0e1406fb80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0e1406f1c0 tx=0x7f0e1001ecb0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 -- 172.21.15.46:0/4216815223 shutdown_connections 2023-10-30T13:31:25.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 --2- 172.21.15.46:0/4216815223 >> v2:172.21.15.154:3300/0 conn(0x7f0e140c6bd0 0x7f0e14069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 --2- 172.21.15.46:0/4216815223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e1406f7a0 0x7f0e1406fb80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 -- 172.21.15.46:0/4216815223 >> 172.21.15.46:0/4216815223 conn(0x7f0e140c7910 msgr2=0x7f0e140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:25.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 -- 172.21.15.46:0/4216815223 shutdown_connections 2023-10-30T13:31:25.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 -- 172.21.15.46:0/4216815223 wait complete. 2023-10-30T13:31:25.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 Processor -- start 2023-10-30T13:31:25.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 -- start start 2023-10-30T13:31:25.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e1406f7a0 0x7f0e14154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:25.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e14071110 0x7f0e141f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:25.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e140c6bd0 0x7f0e141f0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e141f3090 con 0x7f0e1406f7a0 2023-10-30T13:31:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e141f2db0 con 0x7f0e14071110 2023-10-30T13:31:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1c446640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e14153b90 con 0x7f0e140c6bd0 2023-10-30T13:31:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1a1e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e1406f7a0 0x7f0e14154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1a1e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e1406f7a0 0x7f0e14154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33192/0 (socket says 172.21.15.46:33192) 2023-10-30T13:31:25.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.339+0000 7f0e1a1e2640 1 -- 172.21.15.46:0/2409800355 learned_addr learned my addr 172.21.15.46:0/2409800355 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:25.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e199e1640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e14071110 0x7f0e141f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:25.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e1a9e3640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e140c6bd0 0x7f0e141f0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:25.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e1a1e2640 1 -- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e140c6bd0 msgr2=0x7f0e141f0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:25.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e1a1e2640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e140c6bd0 0x7f0e141f0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e1a1e2640 1 -- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e14071110 msgr2=0x7f0e141f23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:25.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e1a1e2640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e14071110 0x7f0e141f23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e1a1e2640 1 -- 172.21.15.46:0/2409800355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e100094d0 con 0x7f0e1406f7a0 2023-10-30T13:31:25.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e199e1640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e14071110 0x7f0e141f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:25.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e1a1e2640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e1406f7a0 0x7f0e14154c60 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0e100b3040 tx=0x7f0e1001ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:25.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e0affd640 1 -- 172.21.15.46:0/2409800355 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100b0e80 con 0x7f0e1406f7a0 2023-10-30T13:31:25.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e0affd640 1 -- 172.21.15.46:0/2409800355 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e100bb480 con 0x7f0e1406f7a0 2023-10-30T13:31:25.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e1c446640 1 -- 172.21.15.46:0/2409800355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e14153e70 con 0x7f0e1406f7a0 2023-10-30T13:31:25.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e0affd640 1 -- 172.21.15.46:0/2409800355 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100bb810 con 0x7f0e1406f7a0 2023-10-30T13:31:25.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e1c446640 1 -- 172.21.15.46:0/2409800355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e140c4960 con 0x7f0e1406f7a0 2023-10-30T13:31:25.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e1c446640 1 -- 172.21.15.46:0/2409800355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e14154320 con 0x7f0e1406f7a0 2023-10-30T13:31:25.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e0affd640 1 -- 172.21.15.46:0/2409800355 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e100bb970 con 0x7f0e1406f7a0 2023-10-30T13:31:25.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e0affd640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0df80664a0 0x7f0df8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:25.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.343+0000 7f0e0affd640 1 -- 172.21.15.46:0/2409800355 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f0e10122790 con 0x7f0e1406f7a0 2023-10-30T13:31:25.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.347+0000 7f0e199e1640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0df80664a0 0x7f0df8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:25.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.347+0000 7f0e199e1640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0df80664a0 0x7f0df8068960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0e1414f2d0 tx=0x7f0e04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:25.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.347+0000 7f0e0affd640 1 -- 172.21.15.46:0/2409800355 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e100ef8a0 con 0x7f0e1406f7a0 2023-10-30T13:31:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.535+0000 7f0e1c446640 1 -- 172.21.15.46:0/2409800355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e141f0b00 con 0x7f0e1406f7a0 2023-10-30T13:31:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e0affd640 1 -- 172.21.15.46:0/2409800355 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f0e100f2ec0 con 0x7f0e1406f7a0 2023-10-30T13:31:25.542 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:25.542 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:23.809922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:25.542 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 46 2023-10-30T13:31:25.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e1c446640 1 -- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0df80664a0 msgr2=0x7f0df8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:25.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e1c446640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0df80664a0 0x7f0df8068960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0e1414f2d0 tx=0x7f0e04008040 comp rx=0 tx=0).stop 2023-10-30T13:31:25.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e1c446640 1 -- 172.21.15.46:0/2409800355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e1406f7a0 msgr2=0x7f0e14154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:25.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e1c446640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e1406f7a0 0x7f0e14154c60 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0e100b3040 tx=0x7f0e1001ba50 comp rx=0 tx=0).stop 2023-10-30T13:31:25.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e1c446640 1 -- 172.21.15.46:0/2409800355 shutdown_connections 2023-10-30T13:31:25.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e1c446640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0df80664a0 0x7f0df8068960 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e1c446640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e140c6bd0 0x7f0e141f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e1c446640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e14071110 0x7f0e141f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e1c446640 1 --2- 172.21.15.46:0/2409800355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e1406f7a0 0x7f0e14154c60 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e1c446640 1 -- 172.21.15.46:0/2409800355 >> 172.21.15.46:0/2409800355 conn(0x7f0e140c7910 msgr2=0x7f0e140b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:25.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e1c446640 1 -- 172.21.15.46:0/2409800355 shutdown_connections 2023-10-30T13:31:25.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.539+0000 7f0e1c446640 1 -- 172.21.15.46:0/2409800355 wait complete. 2023-10-30T13:31:25.563 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T13:31:25.563 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:31:25.563 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:25.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad0866b640 1 Processor -- start 2023-10-30T13:31:25.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad0866b640 1 -- start start 2023-10-30T13:31:25.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad0866b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad001538a0 0x7fad0015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:25.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad0866b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad000cf160 0x7fad000cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:25.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad0866b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad000cb800 con 0x7fad000ca0e0 2023-10-30T13:31:25.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad0866b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad0015dc30 con 0x7fad000cf160 2023-10-30T13:31:25.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad0866b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad000cb330 con 0x7fad001538a0 2023-10-30T13:31:25.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad06c08640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad000cf160 0x7fad000cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:25.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad06c08640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad000cf160 0x7fad000cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33204/0 (socket says 172.21.15.46:33204) 2023-10-30T13:31:25.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad06c08640 1 -- 172.21.15.46:0/2175784279 learned_addr learned my addr 172.21.15.46:0/2175784279 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:25.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad06407640 1 --2- 172.21.15.46:0/2175784279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad001538a0 0x7fad0015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:25.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad06c08640 1 -- 172.21.15.46:0/2175784279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad001538a0 msgr2=0x7fad0015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:25.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad06c08640 1 --2- 172.21.15.46:0/2175784279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad001538a0 0x7fad0015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad06c08640 1 -- 172.21.15.46:0/2175784279 >> v1:172.21.15.154:6789/0 conn(0x7fad000ca0e0 legacy=0x7fad0015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:25.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad06c08640 1 -- 172.21.15.46:0/2175784279 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad000cad70 con 0x7fad000cf160 2023-10-30T13:31:25.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.703+0000 7fad06407640 1 --2- 172.21.15.46:0/2175784279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad001538a0 0x7fad0015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:25.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.707+0000 7fad06c08640 1 --2- 172.21.15.46:0/2175784279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad000cf160 0x7fad000cf590 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7facf8009e50 tx=0x7facf801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f1c6332c4f9905f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:25.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.707+0000 7fad05405640 1 -- 172.21.15.46:0/2175784279 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf80af400 con 0x7fad000cf160 2023-10-30T13:31:25.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.707+0000 7fad05405640 1 -- 172.21.15.46:0/2175784279 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf80ab050 con 0x7fad000cf160 2023-10-30T13:31:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.707+0000 7fad05405640 1 -- 172.21.15.46:0/2175784279 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf80af780 con 0x7fad000cf160 2023-10-30T13:31:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.707+0000 7fad0866b640 1 -- 172.21.15.46:0/2175784279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad000cf160 msgr2=0x7fad000cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.707+0000 7fad0866b640 1 --2- 172.21.15.46:0/2175784279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad000cf160 0x7fad000cf590 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7facf8009e50 tx=0x7facf801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.707+0000 7fad0866b640 1 -- 172.21.15.46:0/2175784279 shutdown_connections 2023-10-30T13:31:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.707+0000 7fad0866b640 1 --2- 172.21.15.46:0/2175784279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad000cf160 0x7fad000cf590 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.707+0000 7fad0866b640 1 --2- 172.21.15.46:0/2175784279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad001538a0 0x7fad0015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.707+0000 7fad0866b640 1 -- 172.21.15.46:0/2175784279 >> 172.21.15.46:0/2175784279 conn(0x7fad0006bfe0 msgr2=0x7fad0006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:25.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.707+0000 7fad0866b640 1 -- 172.21.15.46:0/2175784279 shutdown_connections 2023-10-30T13:31:25.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.707+0000 7fad0866b640 1 -- 172.21.15.46:0/2175784279 wait complete. 2023-10-30T13:31:25.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad0866b640 1 Processor -- start 2023-10-30T13:31:25.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad0866b640 1 -- start start 2023-10-30T13:31:25.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad0866b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad000ca0e0 0x7fad000ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad0866b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad000cf160 0x7fad0015c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad0866b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad001538a0 0x7fad00144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad0866b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad001442e0 con 0x7fad000cf160 2023-10-30T13:31:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad0866b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad00143fe0 con 0x7fad000ca0e0 2023-10-30T13:31:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad0866b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0014c570 con 0x7fad001538a0 2023-10-30T13:31:25.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad05c06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad000cf160 0x7fad0015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:25.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad06407640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad000ca0e0 0x7fad000ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:25.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad05c06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad000cf160 0x7fad0015c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33214/0 (socket says 172.21.15.46:33214) 2023-10-30T13:31:25.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad05c06640 1 -- 172.21.15.46:0/710935228 learned_addr learned my addr 172.21.15.46:0/710935228 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:25.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad06c08640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad001538a0 0x7fad00144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:25.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad05c06640 1 -- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad001538a0 msgr2=0x7fad00144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:25.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad05c06640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad001538a0 0x7fad00144770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad05c06640 1 -- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad000ca0e0 msgr2=0x7fad000ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:25.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad05c06640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad000ca0e0 0x7fad000ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad05c06640 1 -- 172.21.15.46:0/710935228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf8009b30 con 0x7fad000cf160 2023-10-30T13:31:25.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad05c06640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad000cf160 0x7fad0015c920 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7facf000c530 tx=0x7facf000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:25.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7facf6ffd640 1 -- 172.21.15.46:0/710935228 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf0005220 con 0x7fad000cf160 2023-10-30T13:31:25.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7facf6ffd640 1 -- 172.21.15.46:0/710935228 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf0005380 con 0x7fad000cf160 2023-10-30T13:31:25.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad0866b640 1 -- 172.21.15.46:0/710935228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad0014c850 con 0x7fad000cf160 2023-10-30T13:31:25.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7facf6ffd640 1 -- 172.21.15.46:0/710935228 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf0007270 con 0x7fad000cf160 2023-10-30T13:31:25.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.711+0000 7fad0866b640 1 -- 172.21.15.46:0/710935228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad0014ce10 con 0x7fad000cf160 2023-10-30T13:31:25.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.715+0000 7fad0866b640 1 -- 172.21.15.46:0/710935228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad000b2a50 con 0x7fad000cf160 2023-10-30T13:31:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.715+0000 7facf6ffd640 1 -- 172.21.15.46:0/710935228 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7facf00073d0 con 0x7fad000cf160 2023-10-30T13:31:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.715+0000 7facf6ffd640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7face40664a0 0x7face4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.715+0000 7facf6ffd640 1 -- 172.21.15.46:0/710935228 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6284+0+0 (secure 0 0 0) 0x7facf00837e0 con 0x7fad000cf160 2023-10-30T13:31:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.715+0000 7fad06407640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7face40664a0 0x7face4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.715+0000 7fad06407640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7face40664a0 0x7face4068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7facfc0045f0 tx=0x7facfc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:25.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.719+0000 7facf6ffd640 1 -- 172.21.15.46:0/710935228 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7facf0050b60 con 0x7fad000cf160 2023-10-30T13:31:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 -- 172.21.15.46:0/710935228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad0004e170 con 0x7fad000cf160 2023-10-30T13:31:25.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7facf6ffd640 1 -- 172.21.15.46:0/710935228 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5737 (secure 0 0 0) 0x7facf0054180 con 0x7fad000cf160 2023-10-30T13:31:25.914 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:25.915 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:23.809922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:25.915 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 46 2023-10-30T13:31:25.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 -- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7face40664a0 msgr2=0x7face4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:25.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7face40664a0 0x7face4068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7facfc0045f0 tx=0x7facfc008040 comp rx=0 tx=0).stop 2023-10-30T13:31:25.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 -- 172.21.15.46:0/710935228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad000cf160 msgr2=0x7fad0015c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad000cf160 0x7fad0015c920 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7facf000c530 tx=0x7facf000c9f0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 -- 172.21.15.46:0/710935228 shutdown_connections 2023-10-30T13:31:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7face40664a0 0x7face4068960 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad001538a0 0x7fad00144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad000cf160 0x7fad0015c920 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 --2- 172.21.15.46:0/710935228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad000ca0e0 0x7fad000ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 -- 172.21.15.46:0/710935228 >> 172.21.15.46:0/710935228 conn(0x7fad0006bfe0 msgr2=0x7fad00151320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 -- 172.21.15.46:0/710935228 shutdown_connections 2023-10-30T13:31:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:25.911+0000 7fad0866b640 1 -- 172.21.15.46:0/710935228 wait complete. 2023-10-30T13:31:25.936 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 46, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 46, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5106": { "addr": "172.21.15.46:6833/603884510", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 603884510, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 603884510, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5106, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 45 }, "gid_5266": { "addr": "172.21.15.154:6835/651441434", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 651441434, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 651441434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5266, "incarnation": 38, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 22 }, "gid_5442": { "addr": "172.21.15.46:6836/3955931693", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3955931693, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3955931693, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5442, "incarnation": 46, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:31:23.809922+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5266, "mds_1": 5106, "mds_2": 5442 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:31:27.937 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3dbaa95640 1 Processor -- start 2023-10-30T13:31:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3dbaa95640 1 -- start start 2023-10-30T13:31:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3dbaa95640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3db415cd50 0x7f3db40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3dbaa95640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db415a6c0 con 0x7f3db41538f0 2023-10-30T13:31:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3dbaa95640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db415d300 con 0x7f3db40cefa0 2023-10-30T13:31:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3dbaa95640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db415d5e0 con 0x7f3db415cd50 2023-10-30T13:31:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3db9292640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3db415cd50 0x7f3db40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3db9a93640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3db41538f0 0x7f3db4159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37146/0 (socket says 172.21.15.46:37146) 2023-10-30T13:31:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3db9a93640 1 -- 172.21.15.46:0/3126503140 learned_addr learned my addr 172.21.15.46:0/3126503140 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:31:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3db9292640 1 -- 172.21.15.46:0/3126503140 >> v1:172.21.15.154:6789/0 conn(0x7f3db40cefa0 legacy=0x7f3db41640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3db9292640 1 -- 172.21.15.46:0/3126503140 >> v1:172.21.15.46:6789/0 conn(0x7f3db41538f0 legacy=0x7f3db4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3db8a91640 1 -- 172.21.15.46:0/3126503140 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498322803 0 0) 0x7f3db415a6c0 con 0x7f3db41538f0 2023-10-30T13:31:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.091+0000 7f3db9292640 1 -- 172.21.15.46:0/3126503140 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db415d8c0 con 0x7f3db415cd50 2023-10-30T13:31:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.095+0000 7f3db9292640 1 --2- 172.21.15.46:0/3126503140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db415cd50 0x7f3db40c9f90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3da40097c0 tx=0x7f3da401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dcbef93588d1189 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.095+0000 7f3db8a91640 1 -- 172.21.15.46:0/3126503140 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da40a8970 con 0x7f3db415cd50 2023-10-30T13:31:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.095+0000 7f3db8a91640 1 -- 172.21.15.46:0/3126503140 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da40a8ad0 con 0x7f3db415cd50 2023-10-30T13:31:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.095+0000 7f3db8a91640 1 -- 172.21.15.46:0/3126503140 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da40a8e60 con 0x7f3db415cd50 2023-10-30T13:31:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.095+0000 7f3dbaa95640 1 -- 172.21.15.46:0/3126503140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db415cd50 msgr2=0x7f3db40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.095+0000 7f3dbaa95640 1 --2- 172.21.15.46:0/3126503140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db415cd50 0x7f3db40c9f90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3da40097c0 tx=0x7f3da401ec70 comp rx=0 tx=0).stop 2023-10-30T13:31:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.095+0000 7f3dbaa95640 1 -- 172.21.15.46:0/3126503140 shutdown_connections 2023-10-30T13:31:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.095+0000 7f3dbaa95640 1 --2- 172.21.15.46:0/3126503140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db415cd50 0x7f3db40c9f90 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.095+0000 7f3dbaa95640 1 -- 172.21.15.46:0/3126503140 >> 172.21.15.46:0/3126503140 conn(0x7f3db406a640 msgr2=0x7f3db4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.095+0000 7f3dbaa95640 1 -- 172.21.15.46:0/3126503140 shutdown_connections 2023-10-30T13:31:28.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.095+0000 7f3dbaa95640 1 -- 172.21.15.46:0/3126503140 wait complete. 2023-10-30T13:31:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.095+0000 7f3dbaa95640 1 Processor -- start 2023-10-30T13:31:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3dbaa95640 1 -- start start 2023-10-30T13:31:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3dbaa95640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3db40cefa0 0x7f3db4151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3dbaa95640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db41538f0 0x7f3db4158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3dbaa95640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3db415cd50 0x7f3db413c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3dbaa95640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db4152010 con 0x7f3db41538f0 2023-10-30T13:31:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3dbaa95640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db41582a0 con 0x7f3db415cd50 2023-10-30T13:31:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3dbaa95640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db415c550 con 0x7f3db40cefa0 2023-10-30T13:31:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3db9292640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db41538f0 0x7f3db4158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3db9292640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db41538f0 0x7f3db4158cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54786/0 (socket says 172.21.15.46:54786) 2023-10-30T13:31:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3db9292640 1 -- 172.21.15.46:0/4053224760 learned_addr learned my addr 172.21.15.46:0/4053224760 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3db9a93640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3db40cefa0 0x7f3db4151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3dba294640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3db415cd50 0x7f3db413c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3db9292640 1 -- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3db40cefa0 msgr2=0x7f3db4151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3db9292640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3db40cefa0 0x7f3db4151960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3db9292640 1 -- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3db415cd50 msgr2=0x7f3db413c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:28.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3db9292640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3db415cd50 0x7f3db413c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:28.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3db9292640 1 -- 172.21.15.46:0/4053224760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da4009420 con 0x7f3db41538f0 2023-10-30T13:31:28.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3db9a93640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3db40cefa0 0x7f3db4151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:28.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3db9292640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db41538f0 0x7f3db4158cf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3da401b880 tx=0x7f3da401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3daa7fc640 1 -- 172.21.15.46:0/4053224760 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da40b0ea0 con 0x7f3db41538f0 2023-10-30T13:31:28.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3daa7fc640 1 -- 172.21.15.46:0/4053224760 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da40b9410 con 0x7f3db41538f0 2023-10-30T13:31:28.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3daa7fc640 1 -- 172.21.15.46:0/4053224760 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da40b97a0 con 0x7f3db41538f0 2023-10-30T13:31:28.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3dbaa95640 1 -- 172.21.15.46:0/4053224760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db415c830 con 0x7f3db41538f0 2023-10-30T13:31:28.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3dbaa95640 1 -- 172.21.15.46:0/4053224760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db41587d0 con 0x7f3db41538f0 2023-10-30T13:31:28.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.099+0000 7f3dbaa95640 1 -- 172.21.15.46:0/4053224760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db404e170 con 0x7f3db41538f0 2023-10-30T13:31:28.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.103+0000 7f3daa7fc640 1 -- 172.21.15.46:0/4053224760 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3da40b7040 con 0x7f3db41538f0 2023-10-30T13:31:28.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.103+0000 7f3daa7fc640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d980664a0 0x7f3d98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:28.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.103+0000 7f3daa7fc640 1 -- 172.21.15.46:0/4053224760 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f3da4121e20 con 0x7f3db41538f0 2023-10-30T13:31:28.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.103+0000 7f3db9a93640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d980664a0 0x7f3d98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:28.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.103+0000 7f3db9a93640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d980664a0 0x7f3d98068960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3db0000f70 tx=0x7f3db0007c60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:28.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.107+0000 7f3daa7fc640 1 -- 172.21.15.46:0/4053224760 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3da40eefc0 con 0x7f3db41538f0 2023-10-30T13:31:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.299+0000 7f3dbaa95640 1 -- 172.21.15.46:0/4053224760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3db4159a60 con 0x7f3db41538f0 2023-10-30T13:31:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.299+0000 7f3daa7fc640 1 -- 172.21.15.46:0/4053224760 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f3da40b5020 con 0x7f3db41538f0 2023-10-30T13:31:28.304 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:28.304 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:23.809922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:28.304 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 46 2023-10-30T13:31:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.303+0000 7f3dbaa95640 1 -- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d980664a0 msgr2=0x7f3d98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.303+0000 7f3dbaa95640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d980664a0 0x7f3d98068960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3db0000f70 tx=0x7f3db0007c60 comp rx=0 tx=0).stop 2023-10-30T13:31:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.303+0000 7f3dbaa95640 1 -- 172.21.15.46:0/4053224760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db41538f0 msgr2=0x7f3db4158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.303+0000 7f3dbaa95640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db41538f0 0x7f3db4158cf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3da401b880 tx=0x7f3da401ba10 comp rx=0 tx=0).stop 2023-10-30T13:31:28.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.303+0000 7f3dbaa95640 1 -- 172.21.15.46:0/4053224760 shutdown_connections 2023-10-30T13:31:28.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.303+0000 7f3dbaa95640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d980664a0 0x7f3d98068960 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:28.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.303+0000 7f3dbaa95640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3db415cd50 0x7f3db413c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:28.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.303+0000 7f3dbaa95640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db41538f0 0x7f3db4158cf0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:28.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.303+0000 7f3dbaa95640 1 --2- 172.21.15.46:0/4053224760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3db40cefa0 0x7f3db4151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:28.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.303+0000 7f3dbaa95640 1 -- 172.21.15.46:0/4053224760 >> 172.21.15.46:0/4053224760 conn(0x7f3db406a640 msgr2=0x7f3db40cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:28.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.303+0000 7f3dbaa95640 1 -- 172.21.15.46:0/4053224760 shutdown_connections 2023-10-30T13:31:28.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:28.303+0000 7f3dbaa95640 1 -- 172.21.15.46:0/4053224760 wait complete. 2023-10-30T13:31:28.325 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:31:30.326 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:30.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe0a611d640 1 Processor -- start 2023-10-30T13:31:30.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe0a611d640 1 -- start start 2023-10-30T13:31:30.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe0a611d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe0a00c6b80 0x7fe0a00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:30.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe0a611d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe0a00c6f60 0x7fe0a00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:30.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe0a611d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a00c0a50 con 0x7fe0a00c8430 2023-10-30T13:31:30.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe0a611d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a006f6e0 con 0x7fe0a00c6b80 2023-10-30T13:31:30.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe0a611d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a006f9c0 con 0x7fe0a00c6f60 2023-10-30T13:31:30.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe09f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe0a00c6b80 0x7fe0a00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:30.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe09ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe0a00c8430 0x7fe0a013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37154/0 (socket says 172.21.15.46:37154) 2023-10-30T13:31:30.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe09ffff640 1 -- 172.21.15.46:0/3224030171 learned_addr learned my addr 172.21.15.46:0/3224030171 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:31:30.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe097fff640 1 --2- 172.21.15.46:0/3224030171 >> v2:172.21.15.154:3300/0 conn(0x7fe0a00c6f60 0x7fe0a00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:30.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe09f7fe640 1 -- 172.21.15.46:0/3224030171 >> v2:172.21.15.154:3300/0 conn(0x7fe0a00c6f60 msgr2=0x7fe0a00c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:30.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe09f7fe640 1 --2- 172.21.15.46:0/3224030171 >> v2:172.21.15.154:3300/0 conn(0x7fe0a00c6f60 0x7fe0a00c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:30.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe09f7fe640 1 -- 172.21.15.46:0/3224030171 >> v1:172.21.15.46:6789/0 conn(0x7fe0a00c8430 legacy=0x7fe0a013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:30.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe09f7fe640 1 -- 172.21.15.46:0/3224030171 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a006fca0 con 0x7fe0a00c6b80 2023-10-30T13:31:30.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.479+0000 7fe09effd640 1 -- 172.21.15.46:0/3224030171 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265630166 0 0) 0x7fe0a00c0a50 con 0x7fe0a00c8430 2023-10-30T13:31:30.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe09f7fe640 1 --2- 172.21.15.46:0/3224030171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a00c6b80 0x7fe0a00bfec0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0159b90 tx=0x7fe09001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e5f42e786c66fdf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:30.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe09effd640 1 -- 172.21.15.46:0/3224030171 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0900ab600 con 0x7fe0a00c6b80 2023-10-30T13:31:30.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe09effd640 1 -- 172.21.15.46:0/3224030171 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0900a7050 con 0x7fe0a00c6b80 2023-10-30T13:31:30.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe09effd640 1 -- 172.21.15.46:0/3224030171 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0900ab980 con 0x7fe0a00c6b80 2023-10-30T13:31:30.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe0a611d640 1 -- 172.21.15.46:0/3224030171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a00c6b80 msgr2=0x7fe0a00bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:30.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe0a611d640 1 --2- 172.21.15.46:0/3224030171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a00c6b80 0x7fe0a00bfec0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0159b90 tx=0x7fe09001c460 comp rx=0 tx=0).stop 2023-10-30T13:31:30.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe0a611d640 1 -- 172.21.15.46:0/3224030171 shutdown_connections 2023-10-30T13:31:30.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe0a611d640 1 --2- 172.21.15.46:0/3224030171 >> v2:172.21.15.154:3300/0 conn(0x7fe0a00c6f60 0x7fe0a00c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:30.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe0a611d640 1 --2- 172.21.15.46:0/3224030171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a00c6b80 0x7fe0a00bfec0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:30.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe0a611d640 1 -- 172.21.15.46:0/3224030171 >> 172.21.15.46:0/3224030171 conn(0x7fe0a00c7910 msgr2=0x7fe0a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:30.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe0a611d640 1 -- 172.21.15.46:0/3224030171 shutdown_connections 2023-10-30T13:31:30.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe0a611d640 1 -- 172.21.15.46:0/3224030171 wait complete. 2023-10-30T13:31:30.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.483+0000 7fe0a611d640 1 Processor -- start 2023-10-30T13:31:30.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe0a611d640 1 -- start start 2023-10-30T13:31:30.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe0a611d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0a00c6b80 0x7fe0a013e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:30.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe0a611d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a00c6f60 0x7fe0a013d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:30.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe0a611d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a00c8430 0x7fe0a01347c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:30.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe0a611d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a013e880 con 0x7fe0a00c8430 2023-10-30T13:31:30.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe0a611d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a01560f0 con 0x7fe0a00c6f60 2023-10-30T13:31:30.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe0a611d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a013c620 con 0x7fe0a00c6b80 2023-10-30T13:31:30.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a00c8430 0x7fe0a01347c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:30.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a00c8430 0x7fe0a01347c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54804/0 (socket says 172.21.15.46:54804) 2023-10-30T13:31:30.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09ffff640 1 -- 172.21.15.46:0/220206809 learned_addr learned my addr 172.21.15.46:0/220206809 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:30.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe097fff640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a00c6f60 0x7fe0a013d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:30.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09f7fe640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0a00c6b80 0x7fe0a013e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:30.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09ffff640 1 -- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0a00c6b80 msgr2=0x7fe0a013e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:30.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09ffff640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0a00c6b80 0x7fe0a013e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:30.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09ffff640 1 -- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a00c6f60 msgr2=0x7fe0a013d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:30.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09ffff640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a00c6f60 0x7fe0a013d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:30.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09ffff640 1 -- 172.21.15.46:0/220206809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe090009420 con 0x7fe0a00c8430 2023-10-30T13:31:30.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe097fff640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a00c6f60 0x7fe0a013d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:30.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09f7fe640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0a00c6b80 0x7fe0a013e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:30.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09ffff640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a00c8430 0x7fe0a01347c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe09800c780 tx=0x7fe09800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:30.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09cff9640 1 -- 172.21.15.46:0/220206809 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09800ce90 con 0x7fe0a00c8430 2023-10-30T13:31:30.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09cff9640 1 -- 172.21.15.46:0/220206809 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09800a640 con 0x7fe0a00c8430 2023-10-30T13:31:30.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe09cff9640 1 -- 172.21.15.46:0/220206809 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09800a9d0 con 0x7fe0a00c8430 2023-10-30T13:31:30.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe0a611d640 1 -- 172.21.15.46:0/220206809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a013c900 con 0x7fe0a00c8430 2023-10-30T13:31:30.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe0a611d640 1 -- 172.21.15.46:0/220206809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a013cbe0 con 0x7fe0a00c8430 2023-10-30T13:31:30.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.487+0000 7fe0a611d640 1 -- 172.21.15.46:0/220206809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a013d270 con 0x7fe0a00c8430 2023-10-30T13:31:30.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.491+0000 7fe09cff9640 1 -- 172.21.15.46:0/220206809 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe09800ab30 con 0x7fe0a00c8430 2023-10-30T13:31:30.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.491+0000 7fe09cff9640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0800664a0 0x7fe080068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:30.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.491+0000 7fe09cff9640 1 -- 172.21.15.46:0/220206809 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fe0980836a0 con 0x7fe0a00c8430 2023-10-30T13:31:30.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.491+0000 7fe09f7fe640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0800664a0 0x7fe080068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:30.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.491+0000 7fe09f7fe640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0800664a0 0x7fe080068960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe0900ad000 tx=0x7fe090008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:30.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.495+0000 7fe09cff9640 1 -- 172.21.15.46:0/220206809 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe0a013d270 con 0x7fe0a00c8430 2023-10-30T13:31:30.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.683+0000 7fe0a611d640 1 -- 172.21.15.46:0/220206809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0a013d270 con 0x7fe0a00c8430 2023-10-30T13:31:30.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.683+0000 7fe09cff9640 1 -- 172.21.15.46:0/220206809 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fe0a013d270 con 0x7fe0a00c8430 2023-10-30T13:31:30.689 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:30.690 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:29.230479+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:resolve","state_seq":7,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:30.690 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 47 2023-10-30T13:31:30.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.687+0000 7fe0a611d640 1 -- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0800664a0 msgr2=0x7fe080068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:30.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.687+0000 7fe0a611d640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0800664a0 0x7fe080068960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe0900ad000 tx=0x7fe090008040 comp rx=0 tx=0).stop 2023-10-30T13:31:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.687+0000 7fe0a611d640 1 -- 172.21.15.46:0/220206809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a00c8430 msgr2=0x7fe0a01347c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.687+0000 7fe0a611d640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a00c8430 0x7fe0a01347c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe09800c780 tx=0x7fe09800cc40 comp rx=0 tx=0).stop 2023-10-30T13:31:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.687+0000 7fe0a611d640 1 -- 172.21.15.46:0/220206809 shutdown_connections 2023-10-30T13:31:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.687+0000 7fe0a611d640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0800664a0 0x7fe080068960 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.687+0000 7fe0a611d640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a00c8430 0x7fe0a01347c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:30.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.687+0000 7fe0a611d640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a00c6f60 0x7fe0a013d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:30.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.687+0000 7fe0a611d640 1 --2- 172.21.15.46:0/220206809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0a00c6b80 0x7fe0a013e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:30.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.687+0000 7fe0a611d640 1 -- 172.21.15.46:0/220206809 >> 172.21.15.46:0/220206809 conn(0x7fe0a00c7910 msgr2=0x7fe0a013ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:30.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.687+0000 7fe0a611d640 1 -- 172.21.15.46:0/220206809 shutdown_connections 2023-10-30T13:31:30.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:30.687+0000 7fe0a611d640 1 -- 172.21.15.46:0/220206809 wait complete. 2023-10-30T13:31:30.710 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:31:32.711 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff69d3a0640 1 Processor -- start 2023-10-30T13:31:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff69d3a0640 1 -- start start 2023-10-30T13:31:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff69d3a0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff6981538f0 0x7ff698159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff69d3a0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69815cd50 0x7ff6980c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff69d3a0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff69815d300 con 0x7ff6980cefa0 2023-10-30T13:31:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff69d3a0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69815d5e0 con 0x7ff6981538f0 2023-10-30T13:31:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff69d3a0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69815d8c0 con 0x7ff69815cd50 2023-10-30T13:31:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff6977fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69815cd50 0x7ff6980c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff697fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff6981538f0 0x7ff698159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff6977fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69815cd50 0x7ff6980c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:36432/0 (socket says 172.21.15.46:36432) 2023-10-30T13:31:32.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff6977fe640 1 -- 172.21.15.46:0/2683325936 learned_addr learned my addr 172.21.15.46:0/2683325936 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff697fff640 1 -- 172.21.15.46:0/2683325936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69815cd50 msgr2=0x7ff6980c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff697fff640 1 --2- 172.21.15.46:0/2683325936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69815cd50 0x7ff6980c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff697fff640 1 -- 172.21.15.46:0/2683325936 >> v1:172.21.15.154:6789/0 conn(0x7ff6980cefa0 legacy=0x7ff6980cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff697fff640 1 -- 172.21.15.46:0/2683325936 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69815dba0 con 0x7ff6981538f0 2023-10-30T13:31:32.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.863+0000 7ff6977fe640 1 --2- 172.21.15.46:0/2683325936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69815cd50 0x7ff6980c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:31:32.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.867+0000 7ff697fff640 1 --2- 172.21.15.46:0/2683325936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6981538f0 0x7ff698159fb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff6880097c0 tx=0x7ff68801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af5f5ee27f22160d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.867+0000 7ff696ffd640 1 -- 172.21.15.46:0/2683325936 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6880a8970 con 0x7ff6981538f0 2023-10-30T13:31:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.867+0000 7ff696ffd640 1 -- 172.21.15.46:0/2683325936 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6880a8ad0 con 0x7ff6981538f0 2023-10-30T13:31:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.867+0000 7ff696ffd640 1 -- 172.21.15.46:0/2683325936 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6880a8e60 con 0x7ff6981538f0 2023-10-30T13:31:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.867+0000 7ff69d3a0640 1 -- 172.21.15.46:0/2683325936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6981538f0 msgr2=0x7ff698159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.867+0000 7ff69d3a0640 1 --2- 172.21.15.46:0/2683325936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6981538f0 0x7ff698159fb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff6880097c0 tx=0x7ff68801ec70 comp rx=0 tx=0).stop 2023-10-30T13:31:32.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.867+0000 7ff69d3a0640 1 -- 172.21.15.46:0/2683325936 shutdown_connections 2023-10-30T13:31:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.867+0000 7ff69d3a0640 1 --2- 172.21.15.46:0/2683325936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff69815cd50 0x7ff6980c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.867+0000 7ff69d3a0640 1 --2- 172.21.15.46:0/2683325936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6981538f0 0x7ff698159fb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.867+0000 7ff69d3a0640 1 -- 172.21.15.46:0/2683325936 >> 172.21.15.46:0/2683325936 conn(0x7ff69806a640 msgr2=0x7ff698065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.867+0000 7ff69d3a0640 1 -- 172.21.15.46:0/2683325936 shutdown_connections 2023-10-30T13:31:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.867+0000 7ff69d3a0640 1 -- 172.21.15.46:0/2683325936 wait complete. 2023-10-30T13:31:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff69d3a0640 1 Processor -- start 2023-10-30T13:31:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff69d3a0640 1 -- start start 2023-10-30T13:31:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff69d3a0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6980cefa0 0x7ff6980c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff69d3a0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981538f0 0x7ff69815ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff69d3a0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69815cd50 0x7ff69815b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff69d3a0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff698146120 con 0x7ff6981538f0 2023-10-30T13:31:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff69d3a0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff698146540 con 0x7ff69815cd50 2023-10-30T13:31:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff69d3a0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff698145a60 con 0x7ff6980cefa0 2023-10-30T13:31:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff6977fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981538f0 0x7ff69815ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff6977fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981538f0 0x7ff69815ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54806/0 (socket says 172.21.15.46:54806) 2023-10-30T13:31:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff6977fe640 1 -- 172.21.15.46:0/4111053505 learned_addr learned my addr 172.21.15.46:0/4111053505 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff697fff640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6980cefa0 0x7ff6980c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff69cb9f640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69815cd50 0x7ff69815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff6977fe640 1 -- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6980cefa0 msgr2=0x7ff6980c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff6977fe640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6980cefa0 0x7ff6980c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff6977fe640 1 -- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69815cd50 msgr2=0x7ff69815b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff6977fe640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69815cd50 0x7ff69815b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff6977fe640 1 -- 172.21.15.46:0/4111053505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c0094d0 con 0x7ff6981538f0 2023-10-30T13:31:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff6977fe640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981538f0 0x7ff69815ae00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00e930 tx=0x7ff68c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff694ff9640 1 -- 172.21.15.46:0/4111053505 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c00c8e0 con 0x7ff6981538f0 2023-10-30T13:31:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff694ff9640 1 -- 172.21.15.46:0/4111053505 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c00ca40 con 0x7ff6981538f0 2023-10-30T13:31:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff69d3a0640 1 -- 172.21.15.46:0/4111053505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff688009420 con 0x7ff6981538f0 2023-10-30T13:31:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.871+0000 7ff694ff9640 1 -- 172.21.15.46:0/4111053505 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c00cdb0 con 0x7ff6981538f0 2023-10-30T13:31:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.875+0000 7ff69d3a0640 1 -- 172.21.15.46:0/4111053505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69815c390 con 0x7ff6981538f0 2023-10-30T13:31:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.875+0000 7ff69d3a0640 1 -- 172.21.15.46:0/4111053505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69804e170 con 0x7ff6981538f0 2023-10-30T13:31:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.875+0000 7ff694ff9640 1 -- 172.21.15.46:0/4111053505 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff68c013470 con 0x7ff6981538f0 2023-10-30T13:31:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.875+0000 7ff694ff9640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6840664a0 0x7ff684068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:32.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.875+0000 7ff694ff9640 1 -- 172.21.15.46:0/4111053505 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6284+0+0 (secure 0 0 0) 0x7ff68c0845f0 con 0x7ff6981538f0 2023-10-30T13:31:32.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.875+0000 7ff697fff640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6840664a0 0x7ff684068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:32.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.879+0000 7ff697fff640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6840664a0 0x7ff684068960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff6880b2000 tx=0x7ff688008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:32.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:32.879+0000 7ff694ff9640 1 -- 172.21.15.46:0/4111053505 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff68c051850 con 0x7ff6981538f0 2023-10-30T13:31:33.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 -- 172.21.15.46:0/4111053505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6980b2290 con 0x7ff6981538f0 2023-10-30T13:31:33.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff694ff9640 1 -- 172.21.15.46:0/4111053505 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5738 (secure 0 0 0) 0x7ff68c054e70 con 0x7ff6981538f0 2023-10-30T13:31:33.075 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:33.075 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:29.230479+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:resolve","state_seq":7,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:33.076 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 47 2023-10-30T13:31:33.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 -- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6840664a0 msgr2=0x7ff684068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:33.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6840664a0 0x7ff684068960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff6880b2000 tx=0x7ff688008040 comp rx=0 tx=0).stop 2023-10-30T13:31:33.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 -- 172.21.15.46:0/4111053505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981538f0 msgr2=0x7ff69815ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:33.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981538f0 0x7ff69815ae00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00e930 tx=0x7ff68c00edf0 comp rx=0 tx=0).stop 2023-10-30T13:31:33.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 -- 172.21.15.46:0/4111053505 shutdown_connections 2023-10-30T13:31:33.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6840664a0 0x7ff684068960 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:33.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69815cd50 0x7ff69815b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:33.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981538f0 0x7ff69815ae00 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:33.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 --2- 172.21.15.46:0/4111053505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6980cefa0 0x7ff6980c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:33.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 -- 172.21.15.46:0/4111053505 >> 172.21.15.46:0/4111053505 conn(0x7ff69806a640 msgr2=0x7ff698065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:33.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 -- 172.21.15.46:0/4111053505 shutdown_connections 2023-10-30T13:31:33.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:33.071+0000 7ff69d3a0640 1 -- 172.21.15.46:0/4111053505 wait complete. 2023-10-30T13:31:33.096 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:31:35.097 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:35.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.247+0000 7ff1f4446640 1 Processor -- start 2023-10-30T13:31:35.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.247+0000 7ff1f4446640 1 -- start start 2023-10-30T13:31:35.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.247+0000 7ff1f4446640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec06f7a0 0x7ff1ec06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:35.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.247+0000 7ff1f4446640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ec066cf0 con 0x7ff1ec071110 2023-10-30T13:31:35.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.247+0000 7ff1f4446640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ec0c70d0 con 0x7ff1ec0c6bd0 2023-10-30T13:31:35.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.247+0000 7ff1f4446640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec0700c0 con 0x7ff1ec06f7a0 2023-10-30T13:31:35.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.247+0000 7ff1f29e3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff1ec071110 0x7ff1ec158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37156/0 (socket says 172.21.15.46:37156) 2023-10-30T13:31:35.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.247+0000 7ff1f29e3640 1 -- 172.21.15.46:0/1967894083 learned_addr learned my addr 172.21.15.46:0/1967894083 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:31:35.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.247+0000 7ff1f21e2640 1 --2- 172.21.15.46:0/1967894083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec06f7a0 0x7ff1ec06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:35.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.247+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969085789 0 0) 0x7ff1ec066cf0 con 0x7ff1ec071110 2023-10-30T13:31:35.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.247+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d8003530 con 0x7ff1ec071110 2023-10-30T13:31:35.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479127907 0 0) 0x7ff1ec0c70d0 con 0x7ff1ec0c6bd0 2023-10-30T13:31:35.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ec066cf0 con 0x7ff1ec0c6bd0 2023-10-30T13:31:35.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1070732179 0 0) 0x7ff1d8003530 con 0x7ff1ec071110 2023-10-30T13:31:35.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec06f7a0 msgr2=0x7ff1ec06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:35.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 --2- 172.21.15.46:0/1967894083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec06f7a0 0x7ff1ec06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:35.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 >> v1:172.21.15.154:6789/0 conn(0x7ff1ec0c6bd0 legacy=0x7ff1ec069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:35.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec0c7390 con 0x7ff1ec071110 2023-10-30T13:31:35.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7ff1e4001a00 con 0x7ff1ec071110 2023-10-30T13:31:35.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 >> v1:172.21.15.46:6789/0 conn(0x7ff1ec071110 legacy=0x7ff1ec158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:35.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 --2- 172.21.15.46:0/1967894083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d8003b40 0x7ff1d8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:35.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 --2- 172.21.15.46:0/1967894083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec06f7a0 0x7ff1d80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:35.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 --2- 172.21.15.46:0/1967894083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d8004d10 0x7ff1d8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:35.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec0700c0 con 0x7ff1d8004d10 2023-10-30T13:31:35.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d800ba40 con 0x7ff1d8003b40 2023-10-30T13:31:35.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d800bbb0 con 0x7ff1ec06f7a0 2023-10-30T13:31:35.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f29e3640 1 --2- 172.21.15.46:0/1967894083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d8004d10 0x7ff1d8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:35.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f21e2640 1 --2- 172.21.15.46:0/1967894083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d8003b40 0x7ff1d8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:35.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f19e1640 1 --2- 172.21.15.46:0/1967894083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec06f7a0 0x7ff1d80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:35.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f29e3640 1 -- 172.21.15.46:0/1967894083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec06f7a0 msgr2=0x7ff1d80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:35.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f29e3640 1 --2- 172.21.15.46:0/1967894083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec06f7a0 0x7ff1d80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:35.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f29e3640 1 -- 172.21.15.46:0/1967894083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d8003b40 msgr2=0x7ff1d8003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:35.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f29e3640 1 --2- 172.21.15.46:0/1967894083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d8003b40 0x7ff1d8003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:35.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f29e3640 1 -- 172.21.15.46:0/1967894083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d8003530 con 0x7ff1d8004d10 2023-10-30T13:31:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.251+0000 7ff1f29e3640 1 --2- 172.21.15.46:0/1967894083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d8004d10 0x7ff1d8005190 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400a7c0 tx=0x7ff1e401d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c274cf60fbe5ecfe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e40ad630 con 0x7ff1d8004d10 2023-10-30T13:31:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e40a8070 con 0x7ff1d8004d10 2023-10-30T13:31:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f11e0640 1 -- 172.21.15.46:0/1967894083 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e40ad9f0 con 0x7ff1d8004d10 2023-10-30T13:31:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 -- 172.21.15.46:0/1967894083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d8004d10 msgr2=0x7ff1d8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 --2- 172.21.15.46:0/1967894083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d8004d10 0x7ff1d8005190 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400a7c0 tx=0x7ff1e401d2a0 comp rx=0 tx=0).stop 2023-10-30T13:31:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f29e3640 1 -- 172.21.15.46:0/1967894083 reap_dead start 2023-10-30T13:31:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 -- 172.21.15.46:0/1967894083 shutdown_connections 2023-10-30T13:31:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 -- 172.21.15.46:0/1967894083 >> 172.21.15.46:0/1967894083 conn(0x7ff1ec0c7910 msgr2=0x7ff1ec0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:35.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 -- 172.21.15.46:0/1967894083 shutdown_connections 2023-10-30T13:31:35.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 -- 172.21.15.46:0/1967894083 wait complete. 2023-10-30T13:31:35.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 Processor -- start 2023-10-30T13:31:35.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 -- start start 2023-10-30T13:31:35.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d8004d10 0x7ff1ec14d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:35.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1ec155f60 0x7ff1ec157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:35.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec0ca8f0 0x7ff1ec157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:35.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec143440 con 0x7ff1d8004d10 2023-10-30T13:31:35.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec1efa10 con 0x7ff1ec155f60 2023-10-30T13:31:35.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f4446640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec1f0550 con 0x7ff1ec0ca8f0 2023-10-30T13:31:35.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.255+0000 7ff1f21e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d8004d10 0x7ff1ec14d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:35.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f21e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d8004d10 0x7ff1ec14d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54818/0 (socket says 172.21.15.46:54818) 2023-10-30T13:31:35.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f21e2640 1 -- 172.21.15.46:0/3848272809 learned_addr learned my addr 172.21.15.46:0/3848272809 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:35.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f19e1640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1ec155f60 0x7ff1ec157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:35.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f29e3640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec0ca8f0 0x7ff1ec157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:35.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f21e2640 1 -- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec0ca8f0 msgr2=0x7ff1ec157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:35.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f21e2640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec0ca8f0 0x7ff1ec157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:35.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f21e2640 1 -- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1ec155f60 msgr2=0x7ff1ec157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:35.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f21e2640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1ec155f60 0x7ff1ec157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:35.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f21e2640 1 -- 172.21.15.46:0/3848272809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec146cf0 con 0x7ff1d8004d10 2023-10-30T13:31:35.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f19e1640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1ec155f60 0x7ff1ec157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:35.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f21e2640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d8004d10 0x7ff1ec14d410 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80119d0 tx=0x7ff1e8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:35.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1e2ffd640 1 -- 172.21.15.46:0/3848272809 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e800fb50 con 0x7ff1d8004d10 2023-10-30T13:31:35.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1e2ffd640 1 -- 172.21.15.46:0/3848272809 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e800fcb0 con 0x7ff1d8004d10 2023-10-30T13:31:35.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1e2ffd640 1 -- 172.21.15.46:0/3848272809 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8014440 con 0x7ff1d8004d10 2023-10-30T13:31:35.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f4446640 1 -- 172.21.15.46:0/3848272809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ec1efc20 con 0x7ff1d8004d10 2023-10-30T13:31:35.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f4446640 1 -- 172.21.15.46:0/3848272809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ec1f01e0 con 0x7ff1d8004d10 2023-10-30T13:31:35.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1f4446640 1 -- 172.21.15.46:0/3848272809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ec1f0370 con 0x7ff1d8004d10 2023-10-30T13:31:35.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1e2ffd640 1 -- 172.21.15.46:0/3848272809 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1e80145a0 con 0x7ff1d8004d10 2023-10-30T13:31:35.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.259+0000 7ff1e2ffd640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1d0066470 0x7ff1d0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:35.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.263+0000 7ff1e2ffd640 1 -- 172.21.15.46:0/3848272809 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6284+0+0 (secure 0 0 0) 0x7ff1e8084a70 con 0x7ff1d8004d10 2023-10-30T13:31:35.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.263+0000 7ff1f19e1640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1d0066470 0x7ff1d0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:35.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.263+0000 7ff1f19e1640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1d0066470 0x7ff1d0068930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec14bf60 tx=0x7ff1dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:35.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.267+0000 7ff1e2ffd640 1 -- 172.21.15.46:0/3848272809 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1e8051cd0 con 0x7ff1d8004d10 2023-10-30T13:31:35.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.455+0000 7ff1f4446640 1 -- 172.21.15.46:0/3848272809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1ec04e170 con 0x7ff1d8004d10 2023-10-30T13:31:35.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.455+0000 7ff1e2ffd640 1 -- 172.21.15.46:0/3848272809 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5740 (secure 0 0 0) 0x7ff1e8022020 con 0x7ff1d8004d10 2023-10-30T13:31:35.461 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:35.461 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:35.051709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:35.461 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 48 2023-10-30T13:31:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.459+0000 7ff1f4446640 1 -- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1d0066470 msgr2=0x7ff1d0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.459+0000 7ff1f4446640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1d0066470 0x7ff1d0068930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec14bf60 tx=0x7ff1dc009040 comp rx=0 tx=0).stop 2023-10-30T13:31:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.459+0000 7ff1f4446640 1 -- 172.21.15.46:0/3848272809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d8004d10 msgr2=0x7ff1ec14d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.459+0000 7ff1f4446640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d8004d10 0x7ff1ec14d410 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80119d0 tx=0x7ff1e8011e90 comp rx=0 tx=0).stop 2023-10-30T13:31:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.459+0000 7ff1f4446640 1 -- 172.21.15.46:0/3848272809 shutdown_connections 2023-10-30T13:31:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.459+0000 7ff1f4446640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1d0066470 0x7ff1d0068930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.459+0000 7ff1f4446640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1ec0ca8f0 0x7ff1ec157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.459+0000 7ff1f4446640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1ec155f60 0x7ff1ec157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.459+0000 7ff1f4446640 1 --2- 172.21.15.46:0/3848272809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d8004d10 0x7ff1ec14d410 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.459+0000 7ff1f4446640 1 -- 172.21.15.46:0/3848272809 >> 172.21.15.46:0/3848272809 conn(0x7ff1ec0c7910 msgr2=0x7ff1ec150110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.459+0000 7ff1f4446640 1 -- 172.21.15.46:0/3848272809 shutdown_connections 2023-10-30T13:31:35.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:35.459+0000 7ff1f4446640 1 -- 172.21.15.46:0/3848272809 wait complete. 2023-10-30T13:31:35.481 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:31:37.482 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:37.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.631+0000 7fad604c9640 1 Processor -- start 2023-10-30T13:31:37.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.631+0000 7fad604c9640 1 -- start start 2023-10-30T13:31:37.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad604c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad580c6b80 0x7fad580bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:37.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad604c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad580c6f60 0x7fad580c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad604c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad580c0a50 con 0x7fad580c8430 2023-10-30T13:31:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad604c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5806f6e0 con 0x7fad580c6b80 2023-10-30T13:31:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad604c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5806f9c0 con 0x7fad580c6f60 2023-10-30T13:31:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad5e265640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad580c6b80 0x7fad580bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad5ea66640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fad580c8430 0x7fad5813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51976/0 (socket says 172.21.15.46:51976) 2023-10-30T13:31:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad5e265640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad580c6b80 0x7fad580bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51890/0 (socket says 172.21.15.46:51890) 2023-10-30T13:31:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad5ea66640 1 -- 172.21.15.46:0/3719312461 learned_addr learned my addr 172.21.15.46:0/3719312461 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:31:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad5da64640 1 --2- 172.21.15.46:0/3719312461 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad580c6f60 0x7fad580c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad5e265640 1 -- 172.21.15.46:0/3719312461 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad580c6f60 msgr2=0x7fad580c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad5e265640 1 --2- 172.21.15.46:0/3719312461 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad580c6f60 0x7fad580c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad5e265640 1 -- 172.21.15.46:0/3719312461 >> v1:172.21.15.46:6789/0 conn(0x7fad580c8430 legacy=0x7fad5813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:37.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.635+0000 7fad5e265640 1 -- 172.21.15.46:0/3719312461 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5806fca0 con 0x7fad580c6b80 2023-10-30T13:31:37.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad5e265640 1 --2- 172.21.15.46:0/3719312461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad580c6b80 0x7fad580bfec0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fad540097c0 tx=0x7fad5401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615705752af41ee9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:37.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad5d263640 1 -- 172.21.15.46:0/3719312461 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad540ab600 con 0x7fad580c6b80 2023-10-30T13:31:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad5d263640 1 -- 172.21.15.46:0/3719312461 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad540a7050 con 0x7fad580c6b80 2023-10-30T13:31:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad5d263640 1 -- 172.21.15.46:0/3719312461 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad540ab980 con 0x7fad580c6b80 2023-10-30T13:31:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad604c9640 1 -- 172.21.15.46:0/3719312461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad580c6b80 msgr2=0x7fad580bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad604c9640 1 --2- 172.21.15.46:0/3719312461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad580c6b80 0x7fad580bfec0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fad540097c0 tx=0x7fad5401c3f0 comp rx=0 tx=0).stop 2023-10-30T13:31:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad604c9640 1 -- 172.21.15.46:0/3719312461 shutdown_connections 2023-10-30T13:31:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad604c9640 1 --2- 172.21.15.46:0/3719312461 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad580c6f60 0x7fad580c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad604c9640 1 --2- 172.21.15.46:0/3719312461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad580c6b80 0x7fad580bfec0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad604c9640 1 -- 172.21.15.46:0/3719312461 >> 172.21.15.46:0/3719312461 conn(0x7fad580c7910 msgr2=0x7fad580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad604c9640 1 -- 172.21.15.46:0/3719312461 shutdown_connections 2023-10-30T13:31:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad604c9640 1 -- 172.21.15.46:0/3719312461 wait complete. 2023-10-30T13:31:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad604c9640 1 Processor -- start 2023-10-30T13:31:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.639+0000 7fad604c9640 1 -- start start 2023-10-30T13:31:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad604c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad580c6b80 0x7fad58155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad604c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad580c6f60 0x7fad581346c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad604c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad580c8430 0x7fad58134c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad604c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad58155010 con 0x7fad580c6f60 2023-10-30T13:31:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad604c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad581598f0 con 0x7fad580c6b80 2023-10-30T13:31:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad604c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5813c280 con 0x7fad580c8430 2023-10-30T13:31:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad5da64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad580c6f60 0x7fad581346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad5da64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad580c6f60 0x7fad581346c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51902/0 (socket says 172.21.15.46:51902) 2023-10-30T13:31:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad5da64640 1 -- 172.21.15.46:0/1693414183 learned_addr learned my addr 172.21.15.46:0/1693414183 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad5ea66640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad580c8430 0x7fad58134c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad5e265640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad580c6b80 0x7fad58155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad5da64640 1 -- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad580c8430 msgr2=0x7fad58134c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad5da64640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad580c8430 0x7fad58134c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad5da64640 1 -- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad580c6b80 msgr2=0x7fad58155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad5da64640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad580c6b80 0x7fad58155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad5da64640 1 -- 172.21.15.46:0/1693414183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad54009420 con 0x7fad580c6f60 2023-10-30T13:31:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad5e265640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad580c6b80 0x7fad58155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad5da64640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad580c6f60 0x7fad581346c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fad4800d800 tx=0x7fad4800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad4effd640 1 -- 172.21.15.46:0/1693414183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4800b6c0 con 0x7fad580c6f60 2023-10-30T13:31:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad4effd640 1 -- 172.21.15.46:0/1693414183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4800b820 con 0x7fad580c6f60 2023-10-30T13:31:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad4effd640 1 -- 172.21.15.46:0/1693414183 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4800bb90 con 0x7fad580c6f60 2023-10-30T13:31:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad604c9640 1 -- 172.21.15.46:0/1693414183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad58159330 con 0x7fad580c6f60 2023-10-30T13:31:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad604c9640 1 -- 172.21.15.46:0/1693414183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5813bcc0 con 0x7fad580c6f60 2023-10-30T13:31:37.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad604c9640 1 -- 172.21.15.46:0/1693414183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5804e170 con 0x7fad580c6f60 2023-10-30T13:31:37.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.643+0000 7fad4effd640 1 -- 172.21.15.46:0/1693414183 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fad4800bcf0 con 0x7fad580c6f60 2023-10-30T13:31:37.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.647+0000 7fad4effd640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad3c0664d0 0x7fad3c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:37.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.647+0000 7fad4effd640 1 -- 172.21.15.46:0/1693414183 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fad48084640 con 0x7fad580c6f60 2023-10-30T13:31:37.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.647+0000 7fad5e265640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad3c0664d0 0x7fad3c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:37.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.647+0000 7fad5e265640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad3c0664d0 0x7fad3c068990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fad540ad000 tx=0x7fad540a4f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:37.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.651+0000 7fad4effd640 1 -- 172.21.15.46:0/1693414183 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fad480517f0 con 0x7fad580c6f60 2023-10-30T13:31:37.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.839+0000 7fad604c9640 1 -- 172.21.15.46:0/1693414183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad5813f030 con 0x7fad580c6f60 2023-10-30T13:31:37.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.839+0000 7fad4effd640 1 -- 172.21.15.46:0/1693414183 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fad48054e10 con 0x7fad580c6f60 2023-10-30T13:31:37.844 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:37.845 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:35.051709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 48 2023-10-30T13:31:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.843+0000 7fad604c9640 1 -- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad3c0664d0 msgr2=0x7fad3c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.843+0000 7fad604c9640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad3c0664d0 0x7fad3c068990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fad540ad000 tx=0x7fad540a4f90 comp rx=0 tx=0).stop 2023-10-30T13:31:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.843+0000 7fad604c9640 1 -- 172.21.15.46:0/1693414183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad580c6f60 msgr2=0x7fad581346c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.843+0000 7fad604c9640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad580c6f60 0x7fad581346c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fad4800d800 tx=0x7fad4800dcc0 comp rx=0 tx=0).stop 2023-10-30T13:31:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.843+0000 7fad604c9640 1 -- 172.21.15.46:0/1693414183 shutdown_connections 2023-10-30T13:31:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.843+0000 7fad604c9640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad3c0664d0 0x7fad3c068990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.843+0000 7fad604c9640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad580c8430 0x7fad58134c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.843+0000 7fad604c9640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad580c6f60 0x7fad581346c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.843+0000 7fad604c9640 1 --2- 172.21.15.46:0/1693414183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad580c6b80 0x7fad58155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.843+0000 7fad604c9640 1 -- 172.21.15.46:0/1693414183 >> 172.21.15.46:0/1693414183 conn(0x7fad580c7910 msgr2=0x7fad580c0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.843+0000 7fad604c9640 1 -- 172.21.15.46:0/1693414183 shutdown_connections 2023-10-30T13:31:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:37.843+0000 7fad604c9640 1 -- 172.21.15.46:0/1693414183 wait complete. 2023-10-30T13:31:37.866 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5106": { "addr": "172.21.15.46:6833/603884510", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 603884510, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 603884510, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5106, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 45 }, "gid_5266": { "addr": "172.21.15.154:6835/651441434", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 651441434, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 651441434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5266, "incarnation": 38, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 22 }, "gid_5442": { "addr": "172.21.15.46:6836/3955931693", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3955931693, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3955931693, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5442, "incarnation": 46, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:reconnect", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:31:35.051709+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5266, "mds_1": 5106, "mds_2": 5442 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:31:39.867 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.015+0000 7fca6d4ea640 1 Processor -- start 2023-10-30T13:31:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.015+0000 7fca6d4ea640 1 -- start start 2023-10-30T13:31:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.019+0000 7fca6d4ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca680c6b80 0x7fca680bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.019+0000 7fca6d4ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca680c0a50 con 0x7fca680c6f60 2023-10-30T13:31:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.019+0000 7fca6d4ea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6806f6e0 con 0x7fca6813f4a0 2023-10-30T13:31:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.019+0000 7fca6d4ea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca6806f9c0 con 0x7fca680c6b80 2023-10-30T13:31:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.019+0000 7fca66ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca680c6b80 0x7fca680bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.019+0000 7fca667fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fca680c6f60 0x7fca6813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51992/0 (socket says 172.21.15.46:51992) 2023-10-30T13:31:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.019+0000 7fca667fc640 1 -- 172.21.15.46:0/2917970785 learned_addr learned my addr 172.21.15.46:0/2917970785 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:31:40.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.019+0000 7fca66ffd640 1 -- 172.21.15.46:0/2917970785 >> v1:172.21.15.154:6789/0 conn(0x7fca6813f4a0 legacy=0x7fca681566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:40.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.019+0000 7fca66ffd640 1 -- 172.21.15.46:0/2917970785 >> v1:172.21.15.46:6789/0 conn(0x7fca680c6f60 legacy=0x7fca6813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:40.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.019+0000 7fca66ffd640 1 -- 172.21.15.46:0/2917970785 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6806fca0 con 0x7fca680c6b80 2023-10-30T13:31:40.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.019+0000 7fca65ffb640 1 -- 172.21.15.46:0/2917970785 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465229297 0 0) 0x7fca680c0a50 con 0x7fca680c6f60 2023-10-30T13:31:40.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca66ffd640 1 --2- 172.21.15.46:0/2917970785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680c6b80 0x7fca680bfec0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fca580097c0 tx=0x7fca5801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de78ec1aafc9857f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:40.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca65ffb640 1 -- 172.21.15.46:0/2917970785 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca58004230 con 0x7fca680c6b80 2023-10-30T13:31:40.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca65ffb640 1 -- 172.21.15.46:0/2917970785 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca580a7050 con 0x7fca680c6b80 2023-10-30T13:31:40.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca65ffb640 1 -- 172.21.15.46:0/2917970785 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca58003620 con 0x7fca680c6b80 2023-10-30T13:31:40.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 -- 172.21.15.46:0/2917970785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680c6b80 msgr2=0x7fca680bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:40.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 --2- 172.21.15.46:0/2917970785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680c6b80 0x7fca680bfec0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fca580097c0 tx=0x7fca5801c440 comp rx=0 tx=0).stop 2023-10-30T13:31:40.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 -- 172.21.15.46:0/2917970785 shutdown_connections 2023-10-30T13:31:40.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 --2- 172.21.15.46:0/2917970785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680c6b80 0x7fca680bfec0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:40.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 -- 172.21.15.46:0/2917970785 >> 172.21.15.46:0/2917970785 conn(0x7fca680c7910 msgr2=0x7fca680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:40.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 -- 172.21.15.46:0/2917970785 shutdown_connections 2023-10-30T13:31:40.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 -- 172.21.15.46:0/2917970785 wait complete. 2023-10-30T13:31:40.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 Processor -- start 2023-10-30T13:31:40.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 -- start start 2023-10-30T13:31:40.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680c6b80 0x7fca6813de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:40.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680c6f60 0x7fca68150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:40.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca6813f4a0 0x7fca681514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:40.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca681353a0 con 0x7fca680c6b80 2023-10-30T13:31:40.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6813d9a0 con 0x7fca680c6f60 2023-10-30T13:31:40.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.023+0000 7fca6d4ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca681346f0 con 0x7fca6813f4a0 2023-10-30T13:31:40.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca66ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680c6b80 0x7fca6813de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:40.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca66ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680c6b80 0x7fca6813de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51920/0 (socket says 172.21.15.46:51920) 2023-10-30T13:31:40.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca66ffd640 1 -- 172.21.15.46:0/141591929 learned_addr learned my addr 172.21.15.46:0/141591929 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:40.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca667fc640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680c6f60 0x7fca68150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:40.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca677fe640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca6813f4a0 0x7fca681514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:40.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca66ffd640 1 -- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca6813f4a0 msgr2=0x7fca681514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:40.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca66ffd640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca6813f4a0 0x7fca681514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:40.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca66ffd640 1 -- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680c6f60 msgr2=0x7fca68150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:40.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca66ffd640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680c6f60 0x7fca68150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:40.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca66ffd640 1 -- 172.21.15.46:0/141591929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca58009420 con 0x7fca680c6b80 2023-10-30T13:31:40.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca677fe640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca6813f4a0 0x7fca681514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:31:40.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca66ffd640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680c6b80 0x7fca6813de50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fca580ae040 tx=0x7fca580191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:40.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca4f7fe640 1 -- 172.21.15.46:0/141591929 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca58003cf0 con 0x7fca680c6b80 2023-10-30T13:31:40.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca4f7fe640 1 -- 172.21.15.46:0/141591929 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca58003e50 con 0x7fca680c6b80 2023-10-30T13:31:40.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca4f7fe640 1 -- 172.21.15.46:0/141591929 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca580ab640 con 0x7fca680c6b80 2023-10-30T13:31:40.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca6d4ea640 1 -- 172.21.15.46:0/141591929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca681349d0 con 0x7fca680c6b80 2023-10-30T13:31:40.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca6d4ea640 1 -- 172.21.15.46:0/141591929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca68134f90 con 0x7fca680c6b80 2023-10-30T13:31:40.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca6d4ea640 1 -- 172.21.15.46:0/141591929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6804e170 con 0x7fca680c6b80 2023-10-30T13:31:40.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.027+0000 7fca4f7fe640 1 -- 172.21.15.46:0/141591929 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fca580b2040 con 0x7fca680c6b80 2023-10-30T13:31:40.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.031+0000 7fca4f7fe640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca540664d0 0x7fca54068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:40.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.031+0000 7fca4f7fe640 1 -- 172.21.15.46:0/141591929 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fca5811e220 con 0x7fca680c6b80 2023-10-30T13:31:40.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.031+0000 7fca667fc640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca540664d0 0x7fca54068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:40.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.031+0000 7fca667fc640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca540664d0 0x7fca54068990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fca6813aa20 tx=0x7fca5c007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:40.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.035+0000 7fca4f7fe640 1 -- 172.21.15.46:0/141591929 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fca580eb4f0 con 0x7fca680c6b80 2023-10-30T13:31:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 -- 172.21.15.46:0/141591929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca68156a00 con 0x7fca680c6b80 2023-10-30T13:31:40.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca4f7fe640 1 -- 172.21.15.46:0/141591929 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fca580eeb10 con 0x7fca680c6b80 2023-10-30T13:31:40.230 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:40.231 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:35.051709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:40.231 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 48 2023-10-30T13:31:40.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 -- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca540664d0 msgr2=0x7fca54068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:40.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca540664d0 0x7fca54068990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fca6813aa20 tx=0x7fca5c007b60 comp rx=0 tx=0).stop 2023-10-30T13:31:40.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 -- 172.21.15.46:0/141591929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680c6b80 msgr2=0x7fca6813de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:40.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680c6b80 0x7fca6813de50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fca580ae040 tx=0x7fca580191e0 comp rx=0 tx=0).stop 2023-10-30T13:31:40.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 -- 172.21.15.46:0/141591929 shutdown_connections 2023-10-30T13:31:40.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca540664d0 0x7fca54068990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:40.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca6813f4a0 0x7fca681514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:40.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680c6f60 0x7fca68150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:40.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 --2- 172.21.15.46:0/141591929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680c6b80 0x7fca6813de50 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 -- 172.21.15.46:0/141591929 >> 172.21.15.46:0/141591929 conn(0x7fca680c7910 msgr2=0x7fca680c60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 -- 172.21.15.46:0/141591929 shutdown_connections 2023-10-30T13:31:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:40.227+0000 7fca6d4ea640 1 -- 172.21.15.46:0/141591929 wait complete. 2023-10-30T13:31:40.252 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:31:42.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:42.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.403+0000 7f629d8c9640 1 Processor -- start 2023-10-30T13:31:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.403+0000 7f629d8c9640 1 -- start start 2023-10-30T13:31:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.403+0000 7f629d8c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62981570f0 0x7f629806ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.403+0000 7f629d8c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62980b2300 0x7f6298071b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.403+0000 7f629d8c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f629806b3b0 con 0x7f629806f370 2023-10-30T13:31:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.403+0000 7f629d8c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62980c38a0 con 0x7f62981570f0 2023-10-30T13:31:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.403+0000 7f629d8c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62980c3b20 con 0x7f62980b2300 2023-10-30T13:31:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.403+0000 7f6296ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62981570f0 0x7f629806ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.407+0000 7f6296ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62981570f0 0x7f629806ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51924/0 (socket says 172.21.15.46:51924) 2023-10-30T13:31:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.407+0000 7f6296ffd640 1 -- 172.21.15.46:0/3090362738 learned_addr learned my addr 172.21.15.46:0/3090362738 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:42.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.407+0000 7f628ffff640 1 --2- 172.21.15.46:0/3090362738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62980b2300 0x7f6298071b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:42.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.407+0000 7f6296ffd640 1 -- 172.21.15.46:0/3090362738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62980b2300 msgr2=0x7f6298071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:42.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.407+0000 7f6296ffd640 1 --2- 172.21.15.46:0/3090362738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62980b2300 0x7f6298071b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:42.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.407+0000 7f6296ffd640 1 -- 172.21.15.46:0/3090362738 >> v1:172.21.15.154:6789/0 conn(0x7f629806f370 legacy=0x7f62980ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:42.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.407+0000 7f6296ffd640 1 -- 172.21.15.46:0/3090362738 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62980ca9f0 con 0x7f62981570f0 2023-10-30T13:31:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.407+0000 7f6296ffd640 1 --2- 172.21.15.46:0/3090362738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62981570f0 0x7f629806ca30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6288009870 tx=0x7f628801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1661cec88696de server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.407+0000 7f62967fc640 1 -- 172.21.15.46:0/3090362738 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62880a7070 con 0x7f62981570f0 2023-10-30T13:31:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.407+0000 7f62967fc640 1 -- 172.21.15.46:0/3090362738 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62880ab430 con 0x7f62981570f0 2023-10-30T13:31:42.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.407+0000 7f62967fc640 1 -- 172.21.15.46:0/3090362738 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62880ab7c0 con 0x7f62981570f0 2023-10-30T13:31:42.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 -- 172.21.15.46:0/3090362738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62981570f0 msgr2=0x7f629806ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:42.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 --2- 172.21.15.46:0/3090362738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62981570f0 0x7f629806ca30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6288009870 tx=0x7f628801c540 comp rx=0 tx=0).stop 2023-10-30T13:31:42.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 -- 172.21.15.46:0/3090362738 shutdown_connections 2023-10-30T13:31:42.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 --2- 172.21.15.46:0/3090362738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62980b2300 0x7f6298071b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:42.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 --2- 172.21.15.46:0/3090362738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62981570f0 0x7f629806ca30 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:42.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 -- 172.21.15.46:0/3090362738 >> 172.21.15.46:0/3090362738 conn(0x7f6298069e90 msgr2=0x7f629806a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:42.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 -- 172.21.15.46:0/3090362738 shutdown_connections 2023-10-30T13:31:42.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 -- 172.21.15.46:0/3090362738 wait complete. 2023-10-30T13:31:42.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 Processor -- start 2023-10-30T13:31:42.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 -- start start 2023-10-30T13:31:42.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f629806f370 0x7f629814e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:42.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62980b2300 0x7f62980c20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:42.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62981570f0 0x7f62980c2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:42.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62981f2800 con 0x7f62980b2300 2023-10-30T13:31:42.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629814b9d0 con 0x7f629806f370 2023-10-30T13:31:42.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f629d8c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62981f2ac0 con 0x7f62981570f0 2023-10-30T13:31:42.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f628ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62980b2300 0x7f62980c20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:42.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f628ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62980b2300 0x7f62980c20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51940/0 (socket says 172.21.15.46:51940) 2023-10-30T13:31:42.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f628ffff640 1 -- 172.21.15.46:0/1886927970 learned_addr learned my addr 172.21.15.46:0/1886927970 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:42.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f6296ffd640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f629806f370 0x7f629814e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:42.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f62977fe640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62981570f0 0x7f62980c2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:42.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f628ffff640 1 -- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62981570f0 msgr2=0x7f62980c2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:42.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f628ffff640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62981570f0 0x7f62980c2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:42.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f628ffff640 1 -- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f629806f370 msgr2=0x7f629814e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:42.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f628ffff640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f629806f370 0x7f629814e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:42.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f628ffff640 1 -- 172.21.15.46:0/1886927970 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62880094d0 con 0x7f62980b2300 2023-10-30T13:31:42.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.411+0000 7f62977fe640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62981570f0 0x7f62980c2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:42.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.415+0000 7f628ffff640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62980b2300 0x7f62980c20c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f628000d5f0 tx=0x7f628000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:42.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.415+0000 7f6294ff9640 1 -- 172.21.15.46:0/1886927970 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62800125f0 con 0x7f62980b2300 2023-10-30T13:31:42.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.415+0000 7f6294ff9640 1 -- 172.21.15.46:0/1886927970 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6280012750 con 0x7f62980b2300 2023-10-30T13:31:42.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.415+0000 7f629d8c9640 1 -- 172.21.15.46:0/1886927970 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62981f2da0 con 0x7f62980b2300 2023-10-30T13:31:42.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.415+0000 7f6294ff9640 1 -- 172.21.15.46:0/1886927970 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6280012ae0 con 0x7f62980b2300 2023-10-30T13:31:42.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.415+0000 7f629d8c9640 1 -- 172.21.15.46:0/1886927970 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62981f3360 con 0x7f62980b2300 2023-10-30T13:31:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.415+0000 7f629d8c9640 1 -- 172.21.15.46:0/1886927970 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62981f3520 con 0x7f62980b2300 2023-10-30T13:31:42.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.415+0000 7f6294ff9640 1 -- 172.21.15.46:0/1886927970 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6280012c40 con 0x7f62980b2300 2023-10-30T13:31:42.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.415+0000 7f6294ff9640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62780664a0 0x7f6278068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:42.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.415+0000 7f6294ff9640 1 -- 172.21.15.46:0/1886927970 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f6280084630 con 0x7f62980b2300 2023-10-30T13:31:42.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.415+0000 7f6296ffd640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62780664a0 0x7f6278068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:42.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.419+0000 7f6296ffd640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62780664a0 0x7f6278068960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f62880af000 tx=0x7f6288008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:42.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.419+0000 7f6294ff9640 1 -- 172.21.15.46:0/1886927970 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6280051740 con 0x7f62980b2300 2023-10-30T13:31:42.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 -- 172.21.15.46:0/1886927970 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f629804e170 con 0x7f62980b2300 2023-10-30T13:31:42.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f6294ff9640 1 -- 172.21.15.46:0/1886927970 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f6280054d60 con 0x7f62980b2300 2023-10-30T13:31:42.618 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:42.619 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:40.078168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:42.619 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 49 2023-10-30T13:31:42.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 -- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62780664a0 msgr2=0x7f6278068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:42.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62780664a0 0x7f6278068960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f62880af000 tx=0x7f6288008040 comp rx=0 tx=0).stop 2023-10-30T13:31:42.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 -- 172.21.15.46:0/1886927970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62980b2300 msgr2=0x7f62980c20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:42.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62980b2300 0x7f62980c20c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f628000d5f0 tx=0x7f628000dab0 comp rx=0 tx=0).stop 2023-10-30T13:31:42.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 -- 172.21.15.46:0/1886927970 shutdown_connections 2023-10-30T13:31:42.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62780664a0 0x7f6278068960 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:42.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62981570f0 0x7f62980c2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:42.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62980b2300 0x7f62980c20c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:42.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 --2- 172.21.15.46:0/1886927970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f629806f370 0x7f629814e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:42.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 -- 172.21.15.46:0/1886927970 >> 172.21.15.46:0/1886927970 conn(0x7f6298069e90 msgr2=0x7f629806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:42.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 -- 172.21.15.46:0/1886927970 shutdown_connections 2023-10-30T13:31:42.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:42.615+0000 7f629d8c9640 1 -- 172.21.15.46:0/1886927970 wait complete. 2023-10-30T13:31:42.639 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:31:44.641 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:44.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.791+0000 7ffaa5cea640 1 Processor -- start 2023-10-30T13:31:44.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.791+0000 7ffaa5cea640 1 -- start start 2023-10-30T13:31:44.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffaa5cea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffaa0159e60 0x7ffaa006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffaa5cea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffaa006a720 0x7ffaa006ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffaa5cea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa006fa80 con 0x7ffaa00bfec0 2023-10-30T13:31:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffaa5cea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa0071290 con 0x7ffaa006a720 2023-10-30T13:31:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffaa5cea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa0071570 con 0x7ffaa0159e60 2023-10-30T13:31:44.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffa9ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffaa006a720 0x7ffaa006ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffa9ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffaa006a720 0x7ffaa006ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51944/0 (socket says 172.21.15.46:51944) 2023-10-30T13:31:44.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffa9ffff640 1 -- 172.21.15.46:0/2296264300 learned_addr learned my addr 172.21.15.46:0/2296264300 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:44.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffa9f7fe640 1 --2- 172.21.15.46:0/2296264300 >> v2:172.21.15.154:3300/0 conn(0x7ffaa0159e60 0x7ffaa006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffa9ffff640 1 -- 172.21.15.46:0/2296264300 >> v2:172.21.15.154:3300/0 conn(0x7ffaa0159e60 msgr2=0x7ffaa006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffa9ffff640 1 --2- 172.21.15.46:0/2296264300 >> v2:172.21.15.154:3300/0 conn(0x7ffaa0159e60 0x7ffaa006f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffa9ffff640 1 -- 172.21.15.46:0/2296264300 >> v1:172.21.15.154:6789/0 conn(0x7ffaa00bfec0 legacy=0x7ffaa00c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.795+0000 7ffa9ffff640 1 -- 172.21.15.46:0/2296264300 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa0071850 con 0x7ffaa006a720 2023-10-30T13:31:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffa9ffff640 1 --2- 172.21.15.46:0/2296264300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaa006a720 0x7ffaa006ab90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ffaa00c8950 tx=0x7ffa9801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f6227d3acd385d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffa9e7fc640 1 -- 172.21.15.46:0/2296264300 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa980af400 con 0x7ffaa006a720 2023-10-30T13:31:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffa9e7fc640 1 -- 172.21.15.46:0/2296264300 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa980ab050 con 0x7ffaa006a720 2023-10-30T13:31:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffa9e7fc640 1 -- 172.21.15.46:0/2296264300 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa980af780 con 0x7ffaa006a720 2023-10-30T13:31:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 -- 172.21.15.46:0/2296264300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaa006a720 msgr2=0x7ffaa006ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 --2- 172.21.15.46:0/2296264300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaa006a720 0x7ffaa006ab90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ffaa00c8950 tx=0x7ffa9801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 -- 172.21.15.46:0/2296264300 shutdown_connections 2023-10-30T13:31:44.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 --2- 172.21.15.46:0/2296264300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaa006a720 0x7ffaa006ab90 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 --2- 172.21.15.46:0/2296264300 >> v2:172.21.15.154:3300/0 conn(0x7ffaa0159e60 0x7ffaa006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 -- 172.21.15.46:0/2296264300 >> 172.21.15.46:0/2296264300 conn(0x7ffaa00c7910 msgr2=0x7ffaa00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:44.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 -- 172.21.15.46:0/2296264300 shutdown_connections 2023-10-30T13:31:44.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 -- 172.21.15.46:0/2296264300 wait complete. 2023-10-30T13:31:44.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 Processor -- start 2023-10-30T13:31:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 -- start start 2023-10-30T13:31:44.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaa006a720 0x7ffaa00b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaa00bfec0 0x7ffaa01f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffaa0159e60 0x7ffaa00c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00afcf0 con 0x7ffaa006a720 2023-10-30T13:31:44.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa0158960 con 0x7ffaa00bfec0 2023-10-30T13:31:44.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.799+0000 7ffaa5cea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00ca340 con 0x7ffaa0159e60 2023-10-30T13:31:44.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaa006a720 0x7ffaa00b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaa006a720 0x7ffaa00b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51954/0 (socket says 172.21.15.46:51954) 2023-10-30T13:31:44.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9f7fe640 1 -- 172.21.15.46:0/1482621030 learned_addr learned my addr 172.21.15.46:0/1482621030 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:44.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9ffff640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffaa0159e60 0x7ffaa00c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9effd640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaa00bfec0 0x7ffaa01f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9f7fe640 1 -- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffaa0159e60 msgr2=0x7ffaa00c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9f7fe640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffaa0159e60 0x7ffaa00c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9f7fe640 1 -- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaa00bfec0 msgr2=0x7ffaa01f2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9f7fe640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaa00bfec0 0x7ffaa01f2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9f7fe640 1 -- 172.21.15.46:0/1482621030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa98009b30 con 0x7ffaa006a720 2023-10-30T13:31:44.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9ffff640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffaa0159e60 0x7ffaa00c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:44.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9effd640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaa00bfec0 0x7ffaa01f2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:44.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa9f7fe640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaa006a720 0x7ffaa00b0550 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000d5f0 tx=0x7ffa9000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:44.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa87fff640 1 -- 172.21.15.46:0/1482621030 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa900125f0 con 0x7ffaa006a720 2023-10-30T13:31:44.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa87fff640 1 -- 172.21.15.46:0/1482621030 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa90012750 con 0x7ffaa006a720 2023-10-30T13:31:44.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa87fff640 1 -- 172.21.15.46:0/1482621030 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90012ae0 con 0x7ffaa006a720 2023-10-30T13:31:44.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffaa5cea640 1 -- 172.21.15.46:0/1482621030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa01efbd0 con 0x7ffaa006a720 2023-10-30T13:31:44.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffaa5cea640 1 -- 172.21.15.46:0/1482621030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa01eee10 con 0x7ffaa006a720 2023-10-30T13:31:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffaa5cea640 1 -- 172.21.15.46:0/1482621030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa004e170 con 0x7ffaa006a720 2023-10-30T13:31:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa87fff640 1 -- 172.21.15.46:0/1482621030 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffa90012c40 con 0x7ffaa006a720 2023-10-30T13:31:44.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.803+0000 7ffa87fff640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffa8c0664d0 0x7ffa8c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.807+0000 7ffa87fff640 1 -- 172.21.15.46:0/1482621030 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6284+0+0 (secure 0 0 0) 0x7ffa90084610 con 0x7ffaa006a720 2023-10-30T13:31:44.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.807+0000 7ffa9effd640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffa8c0664d0 0x7ffa8c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.807+0000 7ffa9effd640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffa8c0664d0 0x7ffa8c068990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0158c70 tx=0x7ffa94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.811+0000 7ffa87fff640 1 -- 172.21.15.46:0/1482621030 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffa900517b0 con 0x7ffaa006a720 2023-10-30T13:31:45.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.999+0000 7ffaa5cea640 1 -- 172.21.15.46:0/1482621030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffaa0070250 con 0x7ffaa006a720 2023-10-30T13:31:45.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:44.999+0000 7ffa87fff640 1 -- 172.21.15.46:0/1482621030 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5738 (secure 0 0 0) 0x7ffa90007380 con 0x7ffaa006a720 2023-10-30T13:31:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:45.004 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:40.078168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:45.004 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 49 2023-10-30T13:31:45.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:45.003+0000 7ffaa5cea640 1 -- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffa8c0664d0 msgr2=0x7ffa8c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:45.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:45.003+0000 7ffaa5cea640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffa8c0664d0 0x7ffa8c068990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0158c70 tx=0x7ffa94009040 comp rx=0 tx=0).stop 2023-10-30T13:31:45.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:45.003+0000 7ffaa5cea640 1 -- 172.21.15.46:0/1482621030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaa006a720 msgr2=0x7ffaa00b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:45.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:45.003+0000 7ffaa5cea640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaa006a720 0x7ffaa00b0550 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000d5f0 tx=0x7ffa9000dab0 comp rx=0 tx=0).stop 2023-10-30T13:31:45.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:45.003+0000 7ffaa5cea640 1 -- 172.21.15.46:0/1482621030 shutdown_connections 2023-10-30T13:31:45.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:45.003+0000 7ffaa5cea640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffa8c0664d0 0x7ffa8c068990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:45.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:45.003+0000 7ffaa5cea640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffaa0159e60 0x7ffaa00c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:45.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:45.003+0000 7ffaa5cea640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaa00bfec0 0x7ffaa01f2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:45.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:45.003+0000 7ffaa5cea640 1 --2- 172.21.15.46:0/1482621030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffaa006a720 0x7ffaa00b0550 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:45.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:45.003+0000 7ffaa5cea640 1 -- 172.21.15.46:0/1482621030 >> 172.21.15.46:0/1482621030 conn(0x7ffaa00c7910 msgr2=0x7ffaa00c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:45.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:45.003+0000 7ffaa5cea640 1 -- 172.21.15.46:0/1482621030 shutdown_connections 2023-10-30T13:31:45.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:45.003+0000 7ffaa5cea640 1 -- 172.21.15.46:0/1482621030 wait complete. 2023-10-30T13:31:45.025 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:31:45.869 INFO:tasks.workunit.client.0.smithi046.stderr:+ set -e 2023-10-30T13:31:45.870 INFO:tasks.workunit.client.0.smithi046.stderr:+ touch foo 2023-10-30T13:31:45.874 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod +x foo 2023-10-30T13:31:45.875 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/s 2023-10-30T13:31:45.894 INFO:tasks.workunit.client.0.smithi046.stderr:+ find .snap/s/foo -executable 2023-10-30T13:31:45.894 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep foo 2023-10-30T13:31:46.085 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:31:46.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:31:47.026 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.175+0000 7ff3e3bc4640 1 Processor -- start 2023-10-30T13:31:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e3bc4640 1 -- start start 2023-10-30T13:31:47.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e3bc4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3dc06f7a0 0x7ff3dc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e3bc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3dc0c6bd0 0x7ff3dc069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e3bc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff3dc071110 0x7ff3dc158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e3bc4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3dc0700c0 con 0x7ff3dc06f7a0 2023-10-30T13:31:47.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e3bc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3dc06a3d0 con 0x7ff3dc071110 2023-10-30T13:31:47.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e3bc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc066cf0 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e1960640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3dc06f7a0 0x7ff3dc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e1960640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3dc06f7a0 0x7ff3dc06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37572/0 (socket says 172.21.15.46:37572) 2023-10-30T13:31:47.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e1960640 1 -- 172.21.15.46:0/767410022 learned_addr learned my addr 172.21.15.46:0/767410022 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:47.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e115f640 1 --2- 172.21.15.46:0/767410022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3dc0c6bd0 0x7ff3dc069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e2161640 1 --2- 172.21.15.46:0/767410022 >> v2:172.21.15.154:3300/0 conn(0x7ff3dc071110 0x7ff3dc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e1960640 1 -- 172.21.15.46:0/767410022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3dc0c6bd0 msgr2=0x7ff3dc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:47.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e1960640 1 --2- 172.21.15.46:0/767410022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3dc0c6bd0 0x7ff3dc069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e1960640 1 -- 172.21.15.46:0/767410022 >> v2:172.21.15.154:3300/0 conn(0x7ff3dc071110 msgr2=0x7ff3dc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:47.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e1960640 1 --2- 172.21.15.46:0/767410022 >> v2:172.21.15.154:3300/0 conn(0x7ff3dc071110 0x7ff3dc158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.179+0000 7ff3e1960640 1 -- 172.21.15.46:0/767410022 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc0c70d0 con 0x7ff3dc06f7a0 2023-10-30T13:31:47.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e1960640 1 --2- 172.21.15.46:0/767410022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3dc06f7a0 0x7ff3dc06fb80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc06c7d0 tx=0x7ff3d801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883d802c5a5fe802 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:47.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e095e640 1 -- 172.21.15.46:0/767410022 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d80a89b0 con 0x7ff3dc06f7a0 2023-10-30T13:31:47.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e095e640 1 -- 172.21.15.46:0/767410022 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d80a8b10 con 0x7ff3dc06f7a0 2023-10-30T13:31:47.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e095e640 1 -- 172.21.15.46:0/767410022 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d80a8ea0 con 0x7ff3dc06f7a0 2023-10-30T13:31:47.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/767410022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3dc06f7a0 msgr2=0x7ff3dc06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:47.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e3bc4640 1 --2- 172.21.15.46:0/767410022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3dc06f7a0 0x7ff3dc06fb80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc06c7d0 tx=0x7ff3d801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/767410022 shutdown_connections 2023-10-30T13:31:47.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e3bc4640 1 --2- 172.21.15.46:0/767410022 >> v2:172.21.15.154:3300/0 conn(0x7ff3dc071110 0x7ff3dc158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e3bc4640 1 --2- 172.21.15.46:0/767410022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3dc0c6bd0 0x7ff3dc069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e3bc4640 1 --2- 172.21.15.46:0/767410022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3dc06f7a0 0x7ff3dc06fb80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/767410022 >> 172.21.15.46:0/767410022 conn(0x7ff3dc0c7910 msgr2=0x7ff3dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:47.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/767410022 shutdown_connections 2023-10-30T13:31:47.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/767410022 wait complete. 2023-10-30T13:31:47.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.183+0000 7ff3e3bc4640 1 Processor -- start 2023-10-30T13:31:47.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e3bc4640 1 -- start start 2023-10-30T13:31:47.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e3bc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3dc06f7a0 0x7ff3dc149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e3bc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3dc071110 0x7ff3dc0c1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e3bc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3dc0c6bd0 0x7ff3dc0c18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e3bc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc0c0f50 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e3bc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc14a120 con 0x7ff3dc06f7a0 2023-10-30T13:31:47.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e3bc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc0af3c0 con 0x7ff3dc071110 2023-10-30T13:31:47.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e2161640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3dc0c6bd0 0x7ff3dc0c18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e2161640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3dc0c6bd0 0x7ff3dc0c18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37578/0 (socket says 172.21.15.46:37578) 2023-10-30T13:31:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e2161640 1 -- 172.21.15.46:0/752763943 learned_addr learned my addr 172.21.15.46:0/752763943 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e115f640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3dc071110 0x7ff3dc0c1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e1960640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3dc06f7a0 0x7ff3dc149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e2161640 1 -- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3dc071110 msgr2=0x7ff3dc0c1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e2161640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3dc071110 0x7ff3dc0c1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e2161640 1 -- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3dc06f7a0 msgr2=0x7ff3dc149be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e2161640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3dc06f7a0 0x7ff3dc149be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e2161640 1 -- 172.21.15.46:0/752763943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d80094d0 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e2161640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3dc0c6bd0 0x7ff3dc0c18b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4007d50 tx=0x7ff3d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3d27fc640 1 -- 172.21.15.46:0/752763943 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4014420 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3d27fc640 1 -- 172.21.15.46:0/752763943 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d40145a0 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3d27fc640 1 -- 172.21.15.46:0/752763943 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4014950 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/752763943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3dc0aee00 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/752763943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3dc0c0890 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.187+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/752763943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3dc0c06f0 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.191+0000 7ff3d27fc640 1 -- 172.21.15.46:0/752763943 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff3d4014ab0 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.191+0000 7ff3d27fc640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff3c00664a0 0x7ff3c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.191+0000 7ff3d27fc640 1 -- 172.21.15.46:0/752763943 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6284+0+0 (secure 0 0 0) 0x7ff3d4084810 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.191+0000 7ff3e1960640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff3c00664a0 0x7ff3c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.191+0000 7ff3e1960640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff3c00664a0 0x7ff3c0068960 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff3d80b2000 tx=0x7ff3d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:47.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.195+0000 7ff3d27fc640 1 -- 172.21.15.46:0/752763943 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff3dc0c06f0 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.383+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/752763943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3dc0c1090 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.383+0000 7ff3d27fc640 1 -- 172.21.15.46:0/752763943 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5738 (secure 0 0 0) 0x7ff3dc0c1090 con 0x7ff3dc0c6bd0 2023-10-30T13:31:47.388 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:47.389 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:45.867209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:47.389 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 50 2023-10-30T13:31:47.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.387+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff3c00664a0 msgr2=0x7ff3c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:47.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.387+0000 7ff3e3bc4640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff3c00664a0 0x7ff3c0068960 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff3d80b2000 tx=0x7ff3d8008040 comp rx=0 tx=0).stop 2023-10-30T13:31:47.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.387+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/752763943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3dc0c6bd0 msgr2=0x7ff3dc0c18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:47.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.387+0000 7ff3e3bc4640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3dc0c6bd0 0x7ff3dc0c18b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4007d50 tx=0x7ff3d400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.387+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/752763943 shutdown_connections 2023-10-30T13:31:47.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.387+0000 7ff3e3bc4640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff3c00664a0 0x7ff3c0068960 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.387+0000 7ff3e3bc4640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3dc0c6bd0 0x7ff3dc0c18b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.387+0000 7ff3e3bc4640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3dc071110 0x7ff3dc0c1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.387+0000 7ff3e3bc4640 1 --2- 172.21.15.46:0/752763943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3dc06f7a0 0x7ff3dc149be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.387+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/752763943 >> 172.21.15.46:0/752763943 conn(0x7ff3dc0c7910 msgr2=0x7ff3dc150180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:47.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.387+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/752763943 shutdown_connections 2023-10-30T13:31:47.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:47.387+0000 7ff3e3bc4640 1 -- 172.21.15.46:0/752763943 wait complete. 2023-10-30T13:31:47.409 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=2, replacing gid=4950 2023-10-30T13:31:47.410 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.e 2023-10-30T13:31:50.969 INFO:tasks.workunit.client.0.smithi046.stdout:.snap/s/foo 2023-10-30T13:31:50.969 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/s 2023-10-30T13:31:50.982 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm foo 2023-10-30T13:31:50.996 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo OK 2023-10-30T13:31:50.997 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T13:31:50.997 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:31:50.997 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:31:51.414 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2023-10-30T13:31:51.414 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2023-10-30T13:31:53.974 INFO:tasks.workunit.client.0.smithi046.stderr:+ set -e 2023-10-30T13:31:53.975 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir foo 2023-10-30T13:31:53.980 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo x 2023-10-30T13:31:53.980 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:31:54.246 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir foo/.snap/ss 2023-10-30T13:31:54.261 INFO:tasks.workunit.client.0.smithi046.stderr:+ ln foo/x foo/xx 2023-10-30T13:31:54.264 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat foo/.snap/ss/x 2023-10-30T13:31:54.274 INFO:tasks.workunit.client.0.smithi046.stdout:x 2023-10-30T13:31:54.275 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir foo/.snap/ss 2023-10-30T13:31:54.287 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo a 2023-10-30T13:31:54.287 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo b 2023-10-30T13:31:54.287 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir foo/.snap/s 2023-10-30T13:31:54.304 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat foo/.snap/s/a 2023-10-30T13:31:54.316 INFO:tasks.workunit.client.0.smithi046.stderr:+ r=a 2023-10-30T13:31:54.316 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ -z a ] 2023-10-30T13:31:54.317 INFO:tasks.workunit.client.0.smithi046.stderr:+ ln foo/b foo/b2 2023-10-30T13:31:54.320 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat foo/.snap/s/b 2023-10-30T13:31:54.333 INFO:tasks.workunit.client.0.smithi046.stdout:b 2023-10-30T13:31:54.333 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo this used to hang: 2023-10-30T13:31:54.334 INFO:tasks.workunit.client.0.smithi046.stdout:this used to hang: 2023-10-30T13:31:54.334 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo more 2023-10-30T13:31:54.335 INFO:tasks.workunit.client.0.smithi046.stdout:oh, it didn't hang! good job. 2023-10-30T13:31:54.335 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo oh, it didn't hang! good job. 2023-10-30T13:31:54.335 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat foo/b 2023-10-30T13:31:54.335 INFO:tasks.workunit.client.0.smithi046.stdout:b 2023-10-30T13:31:54.336 INFO:tasks.workunit.client.0.smithi046.stdout:more 2023-10-30T13:31:54.336 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir foo/.snap/s 2023-10-30T13:31:54.344 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm -r foo 2023-10-30T13:31:54.377 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T13:31:54.378 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo OK 2023-10-30T13:31:54.379 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:31:54.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:31:54.411 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T13:31:54.413 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:31:54.413 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:31:54.417 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2023-10-30T13:31:54.417 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2023-10-30T13:31:54.463 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:31:54.464 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:54.551 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T13:31:54.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a36079640 1 Processor -- start 2023-10-30T13:31:54.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a36079640 1 -- start start 2023-10-30T13:31:54.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a36079640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a301538a0 0x7f1a3015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:54.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a36079640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a300ca0e0 0x7f1a3015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:54.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a36079640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a300cf160 0x7f1a300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:54.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a36079640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a300cb800 con 0x7f1a300ca0e0 2023-10-30T13:31:54.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a36079640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a3015dc30 con 0x7f1a300cf160 2023-10-30T13:31:54.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a36079640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300cb330 con 0x7f1a301538a0 2023-10-30T13:31:54.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a2effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a300ca0e0 0x7f1a3015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:54.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a2effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a300ca0e0 0x7f1a3015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37596/0 (socket says 172.21.15.46:37596) 2023-10-30T13:31:54.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a2effd640 1 -- 172.21.15.46:0/1940692769 learned_addr learned my addr 172.21.15.46:0/1940692769 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:54.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a2ffff640 1 --2- 172.21.15.46:0/1940692769 >> v2:172.21.15.154:3300/0 conn(0x7f1a300cf160 0x7f1a300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:54.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a2f7fe640 1 --2- 172.21.15.46:0/1940692769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a301538a0 0x7f1a3015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:54.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a2effd640 1 -- 172.21.15.46:0/1940692769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a301538a0 msgr2=0x7f1a3015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:54.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a2effd640 1 --2- 172.21.15.46:0/1940692769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a301538a0 0x7f1a3015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a2effd640 1 -- 172.21.15.46:0/1940692769 >> v2:172.21.15.154:3300/0 conn(0x7f1a300cf160 msgr2=0x7f1a300cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:54.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a2effd640 1 --2- 172.21.15.46:0/1940692769 >> v2:172.21.15.154:3300/0 conn(0x7f1a300cf160 0x7f1a300cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.607+0000 7f1a2effd640 1 -- 172.21.15.46:0/1940692769 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a300cad70 con 0x7f1a300ca0e0 2023-10-30T13:31:54.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a2effd640 1 --2- 172.21.15.46:0/1940692769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a300ca0e0 0x7f1a3015a090 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1a3006cb00 tx=0x7f1a2401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd880cfe081746e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:54.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a2e7fc640 1 -- 172.21.15.46:0/1940692769 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a240a89b0 con 0x7f1a300ca0e0 2023-10-30T13:31:54.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a2e7fc640 1 -- 172.21.15.46:0/1940692769 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a240a8b10 con 0x7f1a300ca0e0 2023-10-30T13:31:54.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a2e7fc640 1 -- 172.21.15.46:0/1940692769 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a240a8ea0 con 0x7f1a300ca0e0 2023-10-30T13:31:54.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a36079640 1 -- 172.21.15.46:0/1940692769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a300ca0e0 msgr2=0x7f1a3015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:54.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a36079640 1 --2- 172.21.15.46:0/1940692769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a300ca0e0 0x7f1a3015a090 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1a3006cb00 tx=0x7f1a2401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a36079640 1 -- 172.21.15.46:0/1940692769 shutdown_connections 2023-10-30T13:31:54.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a36079640 1 --2- 172.21.15.46:0/1940692769 >> v2:172.21.15.154:3300/0 conn(0x7f1a300cf160 0x7f1a300cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a36079640 1 --2- 172.21.15.46:0/1940692769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a300ca0e0 0x7f1a3015a090 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a36079640 1 --2- 172.21.15.46:0/1940692769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a301538a0 0x7f1a3015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a36079640 1 -- 172.21.15.46:0/1940692769 >> 172.21.15.46:0/1940692769 conn(0x7f1a3006bfe0 msgr2=0x7f1a3006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:54.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a36079640 1 -- 172.21.15.46:0/1940692769 shutdown_connections 2023-10-30T13:31:54.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.611+0000 7f1a36079640 1 -- 172.21.15.46:0/1940692769 wait complete. 2023-10-30T13:31:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a36079640 1 Processor -- start 2023-10-30T13:31:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a36079640 1 -- start start 2023-10-30T13:31:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a36079640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a300ca0e0 0x7f1a301e63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a36079640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a300cf160 0x7f1a300ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:54.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a36079640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a301538a0 0x7f1a300ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:54.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a36079640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a301f35b0 con 0x7f1a301538a0 2023-10-30T13:31:54.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a36079640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a301e6a10 con 0x7f1a300cf160 2023-10-30T13:31:54.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a36079640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3015d0c0 con 0x7f1a300ca0e0 2023-10-30T13:31:54.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a2ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a301538a0 0x7f1a300ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a2ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a301538a0 0x7f1a300ce540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37606/0 (socket says 172.21.15.46:37606) 2023-10-30T13:31:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a2ffff640 1 -- 172.21.15.46:0/909119351 learned_addr learned my addr 172.21.15.46:0/909119351 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a2f7fe640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a300ca0e0 0x7f1a301e63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a2effd640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a300cf160 0x7f1a300ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a2ffff640 1 -- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a300ca0e0 msgr2=0x7f1a301e63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a2ffff640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a300ca0e0 0x7f1a301e63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a2ffff640 1 -- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a300cf160 msgr2=0x7f1a300ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a2ffff640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a300cf160 0x7f1a300ce000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a2ffff640 1 -- 172.21.15.46:0/909119351 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a240094d0 con 0x7f1a301538a0 2023-10-30T13:31:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a2ffff640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a301538a0 0x7f1a300ce540 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800eed0 tx=0x7f1a2800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a17fff640 1 -- 172.21.15.46:0/909119351 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a28010040 con 0x7f1a301538a0 2023-10-30T13:31:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a17fff640 1 -- 172.21.15.46:0/909119351 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2800bd10 con 0x7f1a301538a0 2023-10-30T13:31:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a36079640 1 -- 172.21.15.46:0/909119351 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3015d3a0 con 0x7f1a301538a0 2023-10-30T13:31:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a17fff640 1 -- 172.21.15.46:0/909119351 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a28014640 con 0x7f1a301538a0 2023-10-30T13:31:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.615+0000 7f1a36079640 1 -- 172.21.15.46:0/909119351 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3015cdc0 con 0x7f1a301538a0 2023-10-30T13:31:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.619+0000 7f1a36079640 1 -- 172.21.15.46:0/909119351 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a301e6b50 con 0x7f1a301538a0 2023-10-30T13:31:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.619+0000 7f1a17fff640 1 -- 172.21.15.46:0/909119351 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1a280147a0 con 0x7f1a301538a0 2023-10-30T13:31:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.619+0000 7f1a17fff640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a1c0664a0 0x7f1a1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.619+0000 7f1a17fff640 1 -- 172.21.15.46:0/909119351 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f1a28084650 con 0x7f1a301538a0 2023-10-30T13:31:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.619+0000 7f1a2f7fe640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a1c0664a0 0x7f1a1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.619+0000 7f1a2f7fe640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a1c0664a0 0x7f1a1c068960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1a200079c0 tx=0x7f1a20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.623+0000 7f1a17fff640 1 -- 172.21.15.46:0/909119351 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a280518a0 con 0x7f1a301538a0 2023-10-30T13:31:54.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.811+0000 7f1a36079640 1 -- 172.21.15.46:0/909119351 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a300b2a50 con 0x7f1a301538a0 2023-10-30T13:31:54.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.811+0000 7f1a17fff640 1 -- 172.21.15.46:0/909119351 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f1a28054ec0 con 0x7f1a301538a0 2023-10-30T13:31:54.815 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:54.816 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:45.867209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:54.816 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 50 2023-10-30T13:31:54.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.811+0000 7f1a36079640 1 -- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a1c0664a0 msgr2=0x7f1a1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:54.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.811+0000 7f1a36079640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a1c0664a0 0x7f1a1c068960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1a200079c0 tx=0x7f1a20008040 comp rx=0 tx=0).stop 2023-10-30T13:31:54.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.815+0000 7f1a36079640 1 -- 172.21.15.46:0/909119351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a301538a0 msgr2=0x7f1a300ce540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:54.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.815+0000 7f1a36079640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a301538a0 0x7f1a300ce540 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800eed0 tx=0x7f1a2800b480 comp rx=0 tx=0).stop 2023-10-30T13:31:54.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.815+0000 7f1a36079640 1 -- 172.21.15.46:0/909119351 shutdown_connections 2023-10-30T13:31:54.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.815+0000 7f1a36079640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a1c0664a0 0x7f1a1c068960 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.815+0000 7f1a36079640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a301538a0 0x7f1a300ce540 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.815+0000 7f1a36079640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a300cf160 0x7f1a300ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.815+0000 7f1a36079640 1 --2- 172.21.15.46:0/909119351 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a300ca0e0 0x7f1a301e63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.815+0000 7f1a36079640 1 -- 172.21.15.46:0/909119351 >> 172.21.15.46:0/909119351 conn(0x7f1a3006bfe0 msgr2=0x7f1a301525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:54.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.815+0000 7f1a36079640 1 -- 172.21.15.46:0/909119351 shutdown_connections 2023-10-30T13:31:54.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:54.815+0000 7f1a36079640 1 -- 172.21.15.46:0/909119351 wait complete. 2023-10-30T13:31:54.837 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T13:31:56.838 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:56.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.983+0000 7f7627c6b640 1 Processor -- start 2023-10-30T13:31:56.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.983+0000 7f7627c6b640 1 -- start start 2023-10-30T13:31:56.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7627c6b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76200c6b80 0x7f76200bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:56.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7627c6b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f76200c6f60 0x7f76200c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:56.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7627c6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76200c8430 0x7f762013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:56.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7627c6b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76200c0a50 con 0x7f76200c6b80 2023-10-30T13:31:56.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7627c6b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f762006f6e0 con 0x7f76200c6f60 2023-10-30T13:31:56.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7627c6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762006f9c0 con 0x7f76200c8430 2023-10-30T13:31:56.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7625a07640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76200c6b80 0x7f76200bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:56.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7625a07640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76200c6b80 0x7f76200bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55494/0 (socket says 172.21.15.46:55494) 2023-10-30T13:31:56.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7625a07640 1 -- 172.21.15.46:0/60149792 learned_addr learned my addr 172.21.15.46:0/60149792 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:56.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7626208640 1 --2- 172.21.15.46:0/60149792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76200c8430 0x7f762013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:56.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7625206640 1 --2- 172.21.15.46:0/60149792 >> v2:172.21.15.154:3300/0 conn(0x7f76200c6f60 0x7f76200c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:56.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7625a07640 1 -- 172.21.15.46:0/60149792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76200c8430 msgr2=0x7f762013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:56.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7625a07640 1 --2- 172.21.15.46:0/60149792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76200c8430 0x7f762013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7625a07640 1 -- 172.21.15.46:0/60149792 >> v2:172.21.15.154:3300/0 conn(0x7f76200c6f60 msgr2=0x7f76200c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:56.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7625a07640 1 --2- 172.21.15.46:0/60149792 >> v2:172.21.15.154:3300/0 conn(0x7f76200c6f60 0x7f76200c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7625a07640 1 -- 172.21.15.46:0/60149792 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762006fca0 con 0x7f76200c6b80 2023-10-30T13:31:56.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.987+0000 7f7625a07640 1 --2- 172.21.15.46:0/60149792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76200c6b80 0x7f76200bfec0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7620159b90 tx=0x7f761c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e649734827455f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:56.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7624a05640 1 -- 172.21.15.46:0/60149792 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c0ac630 con 0x7f76200c6b80 2023-10-30T13:31:56.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7624a05640 1 -- 172.21.15.46:0/60149792 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c0ac790 con 0x7f76200c6b80 2023-10-30T13:31:56.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7624a05640 1 -- 172.21.15.46:0/60149792 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c0acb20 con 0x7f76200c6b80 2023-10-30T13:31:56.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 -- 172.21.15.46:0/60149792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76200c6b80 msgr2=0x7f76200bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:56.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 --2- 172.21.15.46:0/60149792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76200c6b80 0x7f76200bfec0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7620159b90 tx=0x7f761c01c500 comp rx=0 tx=0).stop 2023-10-30T13:31:56.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 -- 172.21.15.46:0/60149792 shutdown_connections 2023-10-30T13:31:56.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 --2- 172.21.15.46:0/60149792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76200c8430 0x7f762013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 --2- 172.21.15.46:0/60149792 >> v2:172.21.15.154:3300/0 conn(0x7f76200c6f60 0x7f76200c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 --2- 172.21.15.46:0/60149792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76200c6b80 0x7f76200bfec0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 -- 172.21.15.46:0/60149792 >> 172.21.15.46:0/60149792 conn(0x7f76200c7910 msgr2=0x7f76200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:56.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 -- 172.21.15.46:0/60149792 shutdown_connections 2023-10-30T13:31:56.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 -- 172.21.15.46:0/60149792 wait complete. 2023-10-30T13:31:56.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 Processor -- start 2023-10-30T13:31:56.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 -- start start 2023-10-30T13:31:56.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76200c6b80 0x7f7620155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:56.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76200c6f60 0x7f7620132ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:57.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76200c8430 0x7f7620133410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:57.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7620157810 con 0x7f76200c8430 2023-10-30T13:31:57.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7620132a40 con 0x7f76200c6b80 2023-10-30T13:31:57.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7627c6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7620159970 con 0x7f76200c6f60 2023-10-30T13:31:57.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.991+0000 7f7626208640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76200c8430 0x7f7620133410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:57.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7626208640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76200c8430 0x7f7620133410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55502/0 (socket says 172.21.15.46:55502) 2023-10-30T13:31:57.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7626208640 1 -- 172.21.15.46:0/3120108185 learned_addr learned my addr 172.21.15.46:0/3120108185 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:31:57.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7625206640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76200c6f60 0x7f7620132ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:57.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7625a07640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76200c6b80 0x7f7620155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:57.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7626208640 1 -- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76200c6f60 msgr2=0x7f7620132ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:57.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7626208640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76200c6f60 0x7f7620132ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:57.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7626208640 1 -- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76200c6b80 msgr2=0x7f7620155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:57.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7626208640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76200c6b80 0x7f7620155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:57.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7626208640 1 -- 172.21.15.46:0/3120108185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c0094d0 con 0x7f76200c8430 2023-10-30T13:31:57.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7626208640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76200c8430 0x7f7620133410 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7618007e90 tx=0x7f761800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:57.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f76167fc640 1 -- 172.21.15.46:0/3120108185 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618014420 con 0x7f76200c8430 2023-10-30T13:31:57.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f76167fc640 1 -- 172.21.15.46:0/3120108185 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7618014580 con 0x7f76200c8430 2023-10-30T13:31:57.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7627c6b640 1 -- 172.21.15.46:0/3120108185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7620157530 con 0x7f76200c8430 2023-10-30T13:31:57.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f76167fc640 1 -- 172.21.15.46:0/3120108185 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618014930 con 0x7f76200c8430 2023-10-30T13:31:57.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7627c6b640 1 -- 172.21.15.46:0/3120108185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7620159690 con 0x7f76200c8430 2023-10-30T13:31:57.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f7627c6b640 1 -- 172.21.15.46:0/3120108185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7620132bf0 con 0x7f76200c8430 2023-10-30T13:31:57.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f76167fc640 1 -- 172.21.15.46:0/3120108185 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7618014a90 con 0x7f76200c8430 2023-10-30T13:31:57.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.995+0000 7f76167fc640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76040664a0 0x7f7604068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:57.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.999+0000 7f76167fc640 1 -- 172.21.15.46:0/3120108185 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6340+0+0 (secure 0 0 0) 0x7f76180849e0 con 0x7f76200c8430 2023-10-30T13:31:57.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.999+0000 7f7625a07640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76040664a0 0x7f7604068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:57.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.999+0000 7f7625a07640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76040664a0 0x7f7604068960 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f761c0ae000 tx=0x7f761c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:57.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:56.999+0000 7f76167fc640 1 -- 172.21.15.46:0/3120108185 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7620132bf0 con 0x7f76200c8430 2023-10-30T13:31:57.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.191+0000 7f7627c6b640 1 -- 172.21.15.46:0/3120108185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7620132bf0 con 0x7f76200c8430 2023-10-30T13:31:57.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.191+0000 7f76167fc640 1 -- 172.21.15.46:0/3120108185 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6526 (secure 0 0 0) 0x7f7620132bf0 con 0x7f76200c8430 2023-10-30T13:31:57.196 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:31:57.196 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5574,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:31:45.867209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 51 2023-10-30T13:31:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.195+0000 7f7627c6b640 1 -- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76040664a0 msgr2=0x7f7604068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.195+0000 7f7627c6b640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76040664a0 0x7f7604068960 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f761c0ae000 tx=0x7f761c008040 comp rx=0 tx=0).stop 2023-10-30T13:31:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.195+0000 7f7627c6b640 1 -- 172.21.15.46:0/3120108185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76200c8430 msgr2=0x7f7620133410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:57.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.195+0000 7f7627c6b640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76200c8430 0x7f7620133410 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7618007e90 tx=0x7f761800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:31:57.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.195+0000 7f7627c6b640 1 -- 172.21.15.46:0/3120108185 shutdown_connections 2023-10-30T13:31:57.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.195+0000 7f7627c6b640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76040664a0 0x7f7604068960 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:57.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.195+0000 7f7627c6b640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76200c8430 0x7f7620133410 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:57.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.195+0000 7f7627c6b640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76200c6f60 0x7f7620132ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:57.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.195+0000 7f7627c6b640 1 --2- 172.21.15.46:0/3120108185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76200c6b80 0x7f7620155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:57.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.195+0000 7f7627c6b640 1 -- 172.21.15.46:0/3120108185 >> 172.21.15.46:0/3120108185 conn(0x7f76200c7910 msgr2=0x7f76200c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:57.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.195+0000 7f7627c6b640 1 -- 172.21.15.46:0/3120108185 shutdown_connections 2023-10-30T13:31:57.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:31:57.195+0000 7f7627c6b640 1 -- 172.21.15.46:0/3120108185 wait complete. 2023-10-30T13:31:57.217 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T13:31:57.217 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 90 secs before thrashing 2023-10-30T13:31:58.977 INFO:tasks.workunit.client.0.smithi046.stderr:+ set -e 2023-10-30T13:31:58.978 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:31:58.981 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/bar 2023-10-30T13:31:59.000 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/foo 2023-10-30T13:31:59.019 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/foo 2023-10-30T13:31:59.029 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:31:59.035 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/bar/a 2023-10-30T13:32:03.966 INFO:tasks.workunit.client.0.smithi046.stdout: File: .snap/bar/a 2023-10-30T13:32:03.967 INFO:tasks.workunit.client.0.smithi046.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-30T13:32:03.967 INFO:tasks.workunit.client.0.smithi046.stdout:Device: fah/250d Inode: 1099511633796 Links: 2 2023-10-30T13:32:03.967 INFO:tasks.workunit.client.0.smithi046.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2023-10-30T13:32:03.967 INFO:tasks.workunit.client.0.smithi046.stdout:Access: 2023-10-30 13:31:58.975468457 +0000 2023-10-30T13:32:03.967 INFO:tasks.workunit.client.0.smithi046.stdout:Modify: 2023-10-30 13:31:58.975468457 +0000 2023-10-30T13:32:03.968 INFO:tasks.workunit.client.0.smithi046.stdout:Change: 2023-10-30 13:31:58.975468457 +0000 2023-10-30T13:32:03.968 INFO:tasks.workunit.client.0.smithi046.stdout: Birth: 2023-10-30 13:31:58.975468457 +0000 2023-10-30T13:32:03.968 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/bar 2023-10-30T13:32:03.978 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo OK 2023-10-30T13:32:03.979 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T13:32:03.979 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:32:03.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:32:04.014 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2023-10-30T13:32:04.014 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2023-10-30T13:32:08.969 INFO:tasks.workunit.client.0.smithi046.stderr:+ set -e 2023-10-30T13:32:08.970 INFO:tasks.workunit.client.0.smithi046.stderr:+ seq 1 20 2023-10-30T13:32:08.971 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:08.975 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:08.977 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:08.998 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:09.011 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:09.012 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:09.262 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:09.278 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:09.279 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:09.279 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:09.279 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:09.279 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:09.280 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:09.280 INFO:tasks.workunit.client.0.smithi046.stdout: 49850 ? 00:00:00 ps 2023-10-30T13:32:09.280 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:09.301 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:09.307 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:10.021 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:10.025 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:10.027 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:10.048 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:10.623 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:10.624 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:10.711 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:10.727 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:10.727 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:10.727 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:10.728 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:10.728 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:10.728 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:10.728 INFO:tasks.workunit.client.0.smithi046.stdout: 49861 ? 00:00:00 ps 2023-10-30T13:32:10.729 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:10.748 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:10.754 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:10.770 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:10.775 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:10.777 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:10.796 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:10.807 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:10.808 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:10.863 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:10.881 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:10.881 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:10.881 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:10.881 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:10.881 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:10.881 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:10.882 INFO:tasks.workunit.client.0.smithi046.stdout: 49871 ? 00:00:00 ps 2023-10-30T13:32:10.882 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:10.922 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:10.926 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:10.938 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:10.942 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:10.943 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:10.958 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:10.968 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:10.969 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:11.103 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:11.118 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:11.118 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:11.118 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:11.118 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:11.119 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:11.119 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:11.119 INFO:tasks.workunit.client.0.smithi046.stdout: 49881 ? 00:00:00 ps 2023-10-30T13:32:11.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:11.144 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:11.152 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:11.165 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:11.169 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:11.171 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:11.188 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:11.200 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:11.201 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:11.261 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:11.276 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:11.276 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:11.277 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:11.277 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:11.277 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:11.277 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:11.277 INFO:tasks.workunit.client.0.smithi046.stdout: 49891 ? 00:00:00 ps 2023-10-30T13:32:11.278 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:11.314 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:11.320 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:11.332 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:11.336 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:11.338 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:11.355 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:11.377 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:11.379 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:11.428 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:11.443 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:11.443 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:11.444 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:11.444 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:11.444 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:11.444 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:11.444 INFO:tasks.workunit.client.0.smithi046.stdout: 49901 ? 00:00:00 ps 2023-10-30T13:32:11.445 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:11.463 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:11.468 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:11.481 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:11.485 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:11.487 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:11.505 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:11.515 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:11.516 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:11.562 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:11.576 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:11.577 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:11.577 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:11.577 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:11.577 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:11.577 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:11.578 INFO:tasks.workunit.client.0.smithi046.stdout: 49911 ? 00:00:00 ps 2023-10-30T13:32:11.578 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:11.598 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:11.603 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:11.616 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:11.620 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:11.622 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:11.641 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:11.651 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:11.652 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:11.704 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:11.719 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:11.719 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:11.719 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:11.719 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:11.719 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:11.720 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:11.720 INFO:tasks.workunit.client.0.smithi046.stdout: 49921 ? 00:00:00 ps 2023-10-30T13:32:11.720 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:11.739 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:11.744 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:11.757 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:11.761 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:11.763 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:11.781 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:11.793 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:11.794 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:11.846 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:11.861 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:11.861 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:11.861 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:11.861 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:11.861 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:11.862 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:11.862 INFO:tasks.workunit.client.0.smithi046.stdout: 49931 ? 00:00:00 ps 2023-10-30T13:32:11.862 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:11.920 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:11.927 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:11.942 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:11.946 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:11.948 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:11.965 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:11.976 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:11.977 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:12.055 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:12.069 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:12.069 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:12.070 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:12.070 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:12.070 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:12.070 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:12.070 INFO:tasks.workunit.client.0.smithi046.stdout: 49941 ? 00:00:00 ps 2023-10-30T13:32:12.071 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:12.091 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:12.097 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:12.111 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:12.115 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:12.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:12.140 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:12.151 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:12.152 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:12.204 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:12.219 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:12.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:12.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:12.220 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:12.220 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:12.220 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:12.220 INFO:tasks.workunit.client.0.smithi046.stdout: 49951 ? 00:00:00 ps 2023-10-30T13:32:12.221 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:12.241 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:12.246 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:12.260 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:12.265 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:12.267 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:12.489 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:12.501 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:12.502 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:12.589 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:12.603 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:12.604 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:12.604 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:12.604 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:12.604 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:12.604 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:12.605 INFO:tasks.workunit.client.0.smithi046.stdout: 49961 ? 00:00:00 ps 2023-10-30T13:32:12.605 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:12.627 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:12.633 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:12.647 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:12.651 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:12.653 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:12.674 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:12.686 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:12.687 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:12.739 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:12.753 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:12.754 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:12.754 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:12.754 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:12.754 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:12.754 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:12.755 INFO:tasks.workunit.client.0.smithi046.stdout: 49971 ? 00:00:00 ps 2023-10-30T13:32:12.755 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:12.777 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:12.784 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:12.798 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:12.802 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:12.804 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:13.563 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:13.601 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:13.602 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:13.848 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:13.863 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:13.863 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:13.863 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:13.863 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:13.864 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:13.864 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:13.864 INFO:tasks.workunit.client.0.smithi046.stdout: 49981 ? 00:00:00 ps 2023-10-30T13:32:13.865 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:13.888 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:13.894 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:13.917 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:13.924 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:13.930 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:14.027 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:14.038 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:14.039 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:14.198 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:14.212 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:14.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:14.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:14.213 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:14.213 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:14.213 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:14.214 INFO:tasks.workunit.client.0.smithi046.stdout: 49991 ? 00:00:00 ps 2023-10-30T13:32:14.214 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:14.234 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:14.239 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:14.252 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:14.256 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:14.258 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:14.276 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:14.284 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:14.285 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:14.348 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:14.363 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:14.363 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:14.363 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:14.363 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:14.364 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:14.364 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:14.364 INFO:tasks.workunit.client.0.smithi046.stdout: 50001 ? 00:00:00 ps 2023-10-30T13:32:14.365 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:14.380 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:14.384 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:14.396 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:14.400 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:14.403 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:14.417 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:14.427 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:14.428 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:14.473 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:14.488 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:14.488 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:14.488 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:14.489 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:14.489 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:14.489 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:14.489 INFO:tasks.workunit.client.0.smithi046.stdout: 50011 ? 00:00:00 ps 2023-10-30T13:32:14.490 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:14.505 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:14.510 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:14.520 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:14.524 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:14.526 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:14.541 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:14.552 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:14.553 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:14.606 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:14.621 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:14.621 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:14.622 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:14.622 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:14.622 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:14.622 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:14.622 INFO:tasks.workunit.client.0.smithi046.stdout: 50021 ? 00:00:00 ps 2023-10-30T13:32:14.623 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:14.642 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:14.646 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:14.656 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:14.660 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:14.663 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:14.678 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:14.687 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:14.688 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:14.761 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:14.776 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:14.776 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:14.777 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:14.777 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:14.777 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:14.777 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:14.777 INFO:tasks.workunit.client.0.smithi046.stdout: 50031 ? 00:00:00 ps 2023-10-30T13:32:14.778 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:14.794 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:14.799 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:14.809 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a 2023-10-30T13:32:14.814 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/one 2023-10-30T13:32:14.816 INFO:tasks.workunit.client.0.smithi046.stderr:+ cat 2023-10-30T13:32:14.831 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir a/.snap/two 2023-10-30T13:32:14.840 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a/foo 2023-10-30T13:32:14.841 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T13:32:14.907 INFO:tasks.workunit.client.0.smithi046.stderr:+ ps 2023-10-30T13:32:14.929 INFO:tasks.workunit.client.0.smithi046.stdout: PID TTY TIME CMD 2023-10-30T13:32:14.929 INFO:tasks.workunit.client.0.smithi046.stdout: 1195 ? 00:00:00 systemd 2023-10-30T13:32:14.929 INFO:tasks.workunit.client.0.smithi046.stdout: 1196 ? 00:00:00 (sd-pam) 2023-10-30T13:32:14.929 INFO:tasks.workunit.client.0.smithi046.stdout: 27902 ? 00:00:01 sshd 2023-10-30T13:32:14.930 INFO:tasks.workunit.client.0.smithi046.stdout: 49838 ? 00:00:00 timeout 2023-10-30T13:32:14.930 INFO:tasks.workunit.client.0.smithi046.stdout: 49842 ? 00:00:00 snaptest-double 2023-10-30T13:32:14.930 INFO:tasks.workunit.client.0.smithi046.stdout: 50041 ? 00:00:00 ps 2023-10-30T13:32:14.931 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:32:14.950 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm a/foo 2023-10-30T13:32:14.955 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir a 2023-10-30T13:32:14.967 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo OK 2023-10-30T13:32:14.968 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T13:32:14.968 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:32:14.968 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:32:15.013 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2023-10-30T13:32:15.013 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2023-10-30T13:32:16.162 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:32:16.167 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:32:18.991 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/foo 2023-10-30T13:32:19.232 INFO:tasks.workunit.client.0.smithi046.stdout:We want ENOENT, not ESTALE, here. 2023-10-30T13:32:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo We want ENOENT, not ESTALE, here. 2023-10-30T13:32:19.233 INFO:tasks.workunit.client.0.smithi046.stderr:+ seq 1 100 2023-10-30T13:32:19.235 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/1 2023-10-30T13:32:19.235 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.239 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2023-10-30T13:32:19.240 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/2 2023-10-30T13:32:19.240 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.242 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2023-10-30T13:32:19.243 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/3 2023-10-30T13:32:19.243 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.245 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2023-10-30T13:32:19.245 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/4 2023-10-30T13:32:19.246 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.248 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2023-10-30T13:32:19.248 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/5 2023-10-30T13:32:19.249 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.250 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2023-10-30T13:32:19.251 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/6 2023-10-30T13:32:19.252 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.253 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2023-10-30T13:32:19.254 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/7 2023-10-30T13:32:19.254 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.256 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2023-10-30T13:32:19.257 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/8 2023-10-30T13:32:19.257 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.259 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2023-10-30T13:32:19.259 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/9 2023-10-30T13:32:19.260 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.261 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2023-10-30T13:32:19.262 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/10 2023-10-30T13:32:19.263 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.264 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2023-10-30T13:32:19.265 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/11 2023-10-30T13:32:19.265 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.267 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2023-10-30T13:32:19.268 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/12 2023-10-30T13:32:19.268 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.270 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2023-10-30T13:32:19.271 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/13 2023-10-30T13:32:19.271 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.273 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2023-10-30T13:32:19.274 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/14 2023-10-30T13:32:19.274 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.275 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2023-10-30T13:32:19.276 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/15 2023-10-30T13:32:19.277 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.278 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2023-10-30T13:32:19.279 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/16 2023-10-30T13:32:19.280 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.281 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2023-10-30T13:32:19.282 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/17 2023-10-30T13:32:19.282 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.284 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2023-10-30T13:32:19.285 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/18 2023-10-30T13:32:19.285 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.287 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2023-10-30T13:32:19.288 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/19 2023-10-30T13:32:19.288 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.290 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2023-10-30T13:32:19.290 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/20 2023-10-30T13:32:19.291 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.292 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2023-10-30T13:32:19.293 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/21 2023-10-30T13:32:19.293 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.295 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2023-10-30T13:32:19.296 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/22 2023-10-30T13:32:19.296 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.297 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2023-10-30T13:32:19.298 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/23 2023-10-30T13:32:19.299 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.300 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2023-10-30T13:32:19.301 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/24 2023-10-30T13:32:19.301 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.303 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2023-10-30T13:32:19.304 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/25 2023-10-30T13:32:19.304 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.306 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2023-10-30T13:32:19.306 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/26 2023-10-30T13:32:19.307 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.308 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2023-10-30T13:32:19.309 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/27 2023-10-30T13:32:19.309 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.311 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2023-10-30T13:32:19.312 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/28 2023-10-30T13:32:19.312 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.314 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2023-10-30T13:32:19.315 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/29 2023-10-30T13:32:19.315 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.317 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2023-10-30T13:32:19.318 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/30 2023-10-30T13:32:19.318 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.319 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2023-10-30T13:32:19.320 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/31 2023-10-30T13:32:19.320 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.322 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2023-10-30T13:32:19.323 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/32 2023-10-30T13:32:19.323 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.325 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2023-10-30T13:32:19.326 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/33 2023-10-30T13:32:19.326 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.327 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2023-10-30T13:32:19.328 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/34 2023-10-30T13:32:19.328 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.330 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2023-10-30T13:32:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/35 2023-10-30T13:32:19.331 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.333 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2023-10-30T13:32:19.334 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/36 2023-10-30T13:32:19.334 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.335 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2023-10-30T13:32:19.336 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/37 2023-10-30T13:32:19.337 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.338 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2023-10-30T13:32:19.339 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/38 2023-10-30T13:32:19.339 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.341 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2023-10-30T13:32:19.341 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/39 2023-10-30T13:32:19.342 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.343 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2023-10-30T13:32:19.344 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/40 2023-10-30T13:32:19.344 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.346 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2023-10-30T13:32:19.347 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/41 2023-10-30T13:32:19.347 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.348 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2023-10-30T13:32:19.349 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/42 2023-10-30T13:32:19.350 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.351 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2023-10-30T13:32:19.352 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/43 2023-10-30T13:32:19.352 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.354 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2023-10-30T13:32:19.355 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/44 2023-10-30T13:32:19.355 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.357 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2023-10-30T13:32:19.357 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/45 2023-10-30T13:32:19.358 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.361 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2023-10-30T13:32:19.362 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/46 2023-10-30T13:32:19.362 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.362 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2023-10-30T13:32:19.363 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/47 2023-10-30T13:32:19.363 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.364 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2023-10-30T13:32:19.365 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/48 2023-10-30T13:32:19.366 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.367 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2023-10-30T13:32:19.368 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/49 2023-10-30T13:32:19.368 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.370 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2023-10-30T13:32:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/50 2023-10-30T13:32:19.371 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.372 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2023-10-30T13:32:19.373 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/51 2023-10-30T13:32:19.373 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.375 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2023-10-30T13:32:19.376 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/52 2023-10-30T13:32:19.376 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.378 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2023-10-30T13:32:19.379 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/53 2023-10-30T13:32:19.379 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.380 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2023-10-30T13:32:19.381 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/54 2023-10-30T13:32:19.381 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.383 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2023-10-30T13:32:19.384 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/55 2023-10-30T13:32:19.384 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.386 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2023-10-30T13:32:19.387 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/56 2023-10-30T13:32:19.387 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.388 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2023-10-30T13:32:19.389 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/57 2023-10-30T13:32:19.390 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.391 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2023-10-30T13:32:19.392 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/58 2023-10-30T13:32:19.392 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.394 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2023-10-30T13:32:19.395 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/59 2023-10-30T13:32:19.395 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.396 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2023-10-30T13:32:19.397 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/60 2023-10-30T13:32:19.398 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.399 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2023-10-30T13:32:19.400 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/61 2023-10-30T13:32:19.400 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.402 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2023-10-30T13:32:19.402 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/62 2023-10-30T13:32:19.403 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.404 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2023-10-30T13:32:19.405 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/63 2023-10-30T13:32:19.405 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.407 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2023-10-30T13:32:19.408 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/64 2023-10-30T13:32:19.408 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.410 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2023-10-30T13:32:19.411 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/65 2023-10-30T13:32:19.411 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.412 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2023-10-30T13:32:19.413 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/66 2023-10-30T13:32:19.413 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.415 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2023-10-30T13:32:19.416 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/67 2023-10-30T13:32:19.416 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.417 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2023-10-30T13:32:19.418 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/68 2023-10-30T13:32:19.419 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.420 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2023-10-30T13:32:19.421 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/69 2023-10-30T13:32:19.421 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.423 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2023-10-30T13:32:19.424 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/70 2023-10-30T13:32:19.424 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.425 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2023-10-30T13:32:19.426 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/71 2023-10-30T13:32:19.427 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.428 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2023-10-30T13:32:19.429 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/72 2023-10-30T13:32:19.429 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.431 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2023-10-30T13:32:19.431 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/73 2023-10-30T13:32:19.432 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.433 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2023-10-30T13:32:19.434 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/74 2023-10-30T13:32:19.434 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.436 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2023-10-30T13:32:19.437 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/75 2023-10-30T13:32:19.437 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.439 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2023-10-30T13:32:19.439 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/76 2023-10-30T13:32:19.440 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.441 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2023-10-30T13:32:19.442 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/77 2023-10-30T13:32:19.442 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.444 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2023-10-30T13:32:19.445 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/78 2023-10-30T13:32:19.445 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.447 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2023-10-30T13:32:19.448 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/79 2023-10-30T13:32:19.448 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.449 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2023-10-30T13:32:19.450 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/80 2023-10-30T13:32:19.451 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.452 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2023-10-30T13:32:19.453 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/81 2023-10-30T13:32:19.453 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.455 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2023-10-30T13:32:19.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/82 2023-10-30T13:32:19.456 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.457 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2023-10-30T13:32:19.458 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/83 2023-10-30T13:32:19.458 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.460 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2023-10-30T13:32:19.461 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/84 2023-10-30T13:32:19.461 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.462 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2023-10-30T13:32:19.463 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/85 2023-10-30T13:32:19.464 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.465 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2023-10-30T13:32:19.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/86 2023-10-30T13:32:19.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.467 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2023-10-30T13:32:19.468 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/87 2023-10-30T13:32:19.469 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.470 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2023-10-30T13:32:19.471 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/88 2023-10-30T13:32:19.471 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.473 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2023-10-30T13:32:19.473 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/89 2023-10-30T13:32:19.474 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.475 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2023-10-30T13:32:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/90 2023-10-30T13:32:19.476 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.478 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2023-10-30T13:32:19.479 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/91 2023-10-30T13:32:19.479 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.481 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2023-10-30T13:32:19.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/92 2023-10-30T13:32:19.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.483 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2023-10-30T13:32:19.484 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/93 2023-10-30T13:32:19.485 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.486 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2023-10-30T13:32:19.487 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/94 2023-10-30T13:32:19.487 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.489 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2023-10-30T13:32:19.490 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/95 2023-10-30T13:32:19.490 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.491 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2023-10-30T13:32:19.492 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/96 2023-10-30T13:32:19.492 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.494 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2023-10-30T13:32:19.494 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/97 2023-10-30T13:32:19.495 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.496 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2023-10-30T13:32:19.497 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/98 2023-10-30T13:32:19.497 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.499 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2023-10-30T13:32:19.500 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/99 2023-10-30T13:32:19.500 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.501 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2023-10-30T13:32:19.502 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/foo/100 2023-10-30T13:32:19.503 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep No such file 2023-10-30T13:32:19.504 INFO:tasks.workunit.client.0.smithi046.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2023-10-30T13:32:19.504 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/foo 2023-10-30T13:32:19.515 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo OK 2023-10-30T13:32:19.516 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T13:32:19.516 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:32:19.517 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:32:19.548 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-10-30T13:32:19.549 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-10-30T13:32:23.969 INFO:tasks.workunit.client.0.smithi046.stderr:+ set -e 2023-10-30T13:32:23.970 INFO:tasks.workunit.client.0.smithi046.stderr:+ sudo git config --global http.sslVerify false 2023-10-30T13:32:23.990 INFO:tasks.workunit.client.0.smithi046.stderr:+ sudo git config --global http.postBuffer 1048576000 2023-10-30T13:32:24.005 INFO:tasks.workunit.client.0.smithi046.stderr:+ retried=false 2023-10-30T13:32:24.005 INFO:tasks.workunit.client.0.smithi046.stderr:+ trap -- retry EXIT 2023-10-30T13:32:24.005 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm -rf ceph 2023-10-30T13:32:24.006 INFO:tasks.workunit.client.0.smithi046.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-10-30T13:32:24.011 INFO:tasks.workunit.client.0.smithi046.stderr:Cloning into 'ceph'... 2023-10-30T13:32:46.245 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:32:46.250 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:33:16.425 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:33:16.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:33:27.218 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:27.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.366+0000 7fd05ccc9640 1 Processor -- start 2023-10-30T13:33:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.366+0000 7fd05ccc9640 1 -- start start 2023-10-30T13:33:27.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd05ccc9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd0580c05d0 0x7fd05813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd05ccc9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0580c6f60 0x7fd05813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd05ccc9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0580c0a50 con 0x7fd0580c6b80 2023-10-30T13:33:27.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd05ccc9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05806f6e0 con 0x7fd0580c6f60 2023-10-30T13:33:27.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd05ccc9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05806f9c0 con 0x7fd0580c05d0 2023-10-30T13:33:27.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd05659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd0580c6b80 0x7fd0580bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53440/0 (socket says 172.21.15.46:53440) 2023-10-30T13:33:27.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd05659c640 1 -- 172.21.15.46:0/1600378937 learned_addr learned my addr 172.21.15.46:0/1600378937 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:33:27.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd056d9d640 1 --2- 172.21.15.46:0/1600378937 >> v2:172.21.15.46:3300/0 conn(0x7fd0580c6f60 0x7fd05813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd04ffff640 1 --2- 172.21.15.46:0/1600378937 >> v2:172.21.15.154:3300/0 conn(0x7fd0580c05d0 0x7fd05813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd055d9b640 1 -- 172.21.15.46:0/1600378937 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716197251 0 0) 0x7fd0580c0a50 con 0x7fd0580c6b80 2023-10-30T13:33:27.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd055d9b640 1 -- 172.21.15.46:0/1600378937 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd044003530 con 0x7fd0580c6b80 2023-10-30T13:33:27.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd056d9d640 1 -- 172.21.15.46:0/1600378937 >> v2:172.21.15.154:3300/0 conn(0x7fd0580c05d0 msgr2=0x7fd05813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd056d9d640 1 --2- 172.21.15.46:0/1600378937 >> v2:172.21.15.154:3300/0 conn(0x7fd0580c05d0 0x7fd05813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd056d9d640 1 -- 172.21.15.46:0/1600378937 >> v1:172.21.15.46:6789/0 conn(0x7fd0580c6b80 legacy=0x7fd0580bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd056d9d640 1 -- 172.21.15.46:0/1600378937 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05806fca0 con 0x7fd0580c6f60 2023-10-30T13:33:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd04ffff640 1 --2- 172.21.15.46:0/1600378937 >> v2:172.21.15.154:3300/0 conn(0x7fd0580c05d0 0x7fd05813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:33:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.370+0000 7fd056d9d640 1 --2- 172.21.15.46:0/1600378937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0580c6f60 0x7fd05813f2d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd05806c7e0 tx=0x7fd05001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ae506f378e0fb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:27.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd055d9b640 1 -- 172.21.15.46:0/1600378937 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0500a58e0 con 0x7fd0580c6f60 2023-10-30T13:33:27.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd055d9b640 1 -- 172.21.15.46:0/1600378937 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0500a5a40 con 0x7fd0580c6f60 2023-10-30T13:33:27.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd055d9b640 1 -- 172.21.15.46:0/1600378937 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0500a5dd0 con 0x7fd0580c6f60 2023-10-30T13:33:27.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 -- 172.21.15.46:0/1600378937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0580c6f60 msgr2=0x7fd05813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:27.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 --2- 172.21.15.46:0/1600378937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0580c6f60 0x7fd05813f2d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd05806c7e0 tx=0x7fd05001c910 comp rx=0 tx=0).stop 2023-10-30T13:33:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 -- 172.21.15.46:0/1600378937 shutdown_connections 2023-10-30T13:33:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 --2- 172.21.15.46:0/1600378937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0580c6f60 0x7fd05813f2d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 --2- 172.21.15.46:0/1600378937 >> v2:172.21.15.154:3300/0 conn(0x7fd0580c05d0 0x7fd05813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 -- 172.21.15.46:0/1600378937 >> 172.21.15.46:0/1600378937 conn(0x7fd0580c7910 msgr2=0x7fd0580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:27.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 -- 172.21.15.46:0/1600378937 shutdown_connections 2023-10-30T13:33:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 -- 172.21.15.46:0/1600378937 wait complete. 2023-10-30T13:33:27.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 Processor -- start 2023-10-30T13:33:27.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 -- start start 2023-10-30T13:33:27.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0580c05d0 0x7fd0581550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0580c6b80 0x7fd0580c1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0580c6f60 0x7fd0581ebbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0581eae90 con 0x7fd0580c6b80 2023-10-30T13:33:27.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0581eb730 con 0x7fd0580c6f60 2023-10-30T13:33:27.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.374+0000 7fd05ccc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0581f58b0 con 0x7fd0580c05d0 2023-10-30T13:33:27.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0580c6b80 0x7fd0580c1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0580c6b80 0x7fd0580c1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51680/0 (socket says 172.21.15.46:51680) 2023-10-30T13:33:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04ffff640 1 -- 172.21.15.46:0/2130196306 learned_addr learned my addr 172.21.15.46:0/2130196306 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd056d9d640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0580c6f60 0x7fd0581ebbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd05659c640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0580c05d0 0x7fd0581550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04ffff640 1 -- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0580c05d0 msgr2=0x7fd0581550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:27.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04ffff640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0580c05d0 0x7fd0581550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04ffff640 1 -- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0580c6f60 msgr2=0x7fd0581ebbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04ffff640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0580c6f60 0x7fd0581ebbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04ffff640 1 -- 172.21.15.46:0/2130196306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd050009a70 con 0x7fd0580c6b80 2023-10-30T13:33:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd056d9d640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0580c6f60 0x7fd0581ebbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:33:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd05659c640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0580c05d0 0x7fd0581550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:33:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04ffff640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0580c6b80 0x7fd0580c1140 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd04000e810 tx=0x7fd04000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04effd640 1 -- 172.21.15.46:0/2130196306 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04000c6d0 con 0x7fd0580c6b80 2023-10-30T13:33:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04effd640 1 -- 172.21.15.46:0/2130196306 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04000c830 con 0x7fd0580c6b80 2023-10-30T13:33:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd05ccc9640 1 -- 172.21.15.46:0/2130196306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0581f5b90 con 0x7fd0580c6b80 2023-10-30T13:33:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04effd640 1 -- 172.21.15.46:0/2130196306 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04000cbe0 con 0x7fd0580c6b80 2023-10-30T13:33:27.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd05ccc9640 1 -- 172.21.15.46:0/2130196306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0581eb3c0 con 0x7fd0580c6b80 2023-10-30T13:33:27.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd05ccc9640 1 -- 172.21.15.46:0/2130196306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0581eb8a0 con 0x7fd0580c6b80 2023-10-30T13:33:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.378+0000 7fd04effd640 1 -- 172.21.15.46:0/2130196306 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd04000cd40 con 0x7fd0580c6b80 2023-10-30T13:33:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.382+0000 7fd04effd640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0380664d0 0x7fd038068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.382+0000 7fd04effd640 1 -- 172.21.15.46:0/2130196306 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fd040083ef0 con 0x7fd0580c6b80 2023-10-30T13:33:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.382+0000 7fd05659c640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0380664d0 0x7fd038068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.382+0000 7fd05659c640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0380664d0 0x7fd038068990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd048005890 tx=0x7fd048009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:27.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.386+0000 7fd04effd640 1 -- 172.21.15.46:0/2130196306 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd040051070 con 0x7fd0580c6b80 2023-10-30T13:33:27.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.578+0000 7fd05ccc9640 1 -- 172.21.15.46:0/2130196306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd05804e170 con 0x7fd0580c6b80 2023-10-30T13:33:27.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.578+0000 7fd04effd640 1 -- 172.21.15.46:0/2130196306 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fd040054690 con 0x7fd0580c6b80 2023-10-30T13:33:27.584 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:27.584 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5574,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:32:58.952603+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 53 2023-10-30T13:33:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.578+0000 7fd05ccc9640 1 -- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0380664d0 msgr2=0x7fd038068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:27.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.578+0000 7fd05ccc9640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0380664d0 0x7fd038068990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd048005890 tx=0x7fd048009040 comp rx=0 tx=0).stop 2023-10-30T13:33:27.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.578+0000 7fd05ccc9640 1 -- 172.21.15.46:0/2130196306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0580c6b80 msgr2=0x7fd0580c1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.578+0000 7fd05ccc9640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0580c6b80 0x7fd0580c1140 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd04000e810 tx=0x7fd04000ecd0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.582+0000 7fd05ccc9640 1 -- 172.21.15.46:0/2130196306 shutdown_connections 2023-10-30T13:33:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.582+0000 7fd05ccc9640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0380664d0 0x7fd038068990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.582+0000 7fd05ccc9640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0580c6f60 0x7fd0581ebbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.582+0000 7fd05ccc9640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0580c6b80 0x7fd0580c1140 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.582+0000 7fd05ccc9640 1 --2- 172.21.15.46:0/2130196306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0580c05d0 0x7fd0581550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.582+0000 7fd05ccc9640 1 -- 172.21.15.46:0/2130196306 >> 172.21.15.46:0/2130196306 conn(0x7fd0580c7910 msgr2=0x7fd0580bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:27.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.582+0000 7fd05ccc9640 1 -- 172.21.15.46:0/2130196306 shutdown_connections 2023-10-30T13:33:27.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.582+0000 7fd05ccc9640 1 -- 172.21.15.46:0/2130196306 wait complete. 2023-10-30T13:33:27.604 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=1) 2023-10-30T13:33:27.605 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:33:27.605 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:33:27.620 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:33:27.621 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:27.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05aebc5640 1 Processor -- start 2023-10-30T13:33:27.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05aebc5640 1 -- start start 2023-10-30T13:33:27.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05aebc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a815d6f0 0x7f05a815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05aebc5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f05a80cf160 0x7f05a80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05aebc5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a80cb800 con 0x7f05a81538a0 2023-10-30T13:33:27.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05aebc5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a80cb330 con 0x7f05a80cf160 2023-10-30T13:33:27.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05aebc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80cad70 con 0x7f05a815d6f0 2023-10-30T13:33:27.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05ac961640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f05a81538a0 0x7f05a80ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53456/0 (socket says 172.21.15.46:53456) 2023-10-30T13:33:27.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05ac961640 1 -- 172.21.15.46:0/1095874658 learned_addr learned my addr 172.21.15.46:0/1095874658 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:33:27.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a7fff640 1 --2- 172.21.15.46:0/1095874658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a815d6f0 0x7f05a815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05ad162640 1 --2- 172.21.15.46:0/1095874658 >> v2:172.21.15.154:3300/0 conn(0x7f05a80cf160 0x7f05a80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38005511 0 0) 0x7f05a80cb800 con 0x7f05a81538a0 2023-10-30T13:33:27.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0598003530 con 0x7f05a81538a0 2023-10-30T13:33:27.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2910287352 0 0) 0x7f0598003530 con 0x7f05a81538a0 2023-10-30T13:33:27.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a815d6f0 msgr2=0x7f05a815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:27.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 --2- 172.21.15.46:0/1095874658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a815d6f0 0x7f05a815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 >> v2:172.21.15.154:3300/0 conn(0x7f05a80cf160 msgr2=0x7f05a80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:27.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 --2- 172.21.15.46:0/1095874658 >> v2:172.21.15.154:3300/0 conn(0x7f05a80cf160 0x7f05a80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a80cb050 con 0x7f05a81538a0 2023-10-30T13:33:27.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f059c001680 con 0x7f05a81538a0 2023-10-30T13:33:27.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 >> v1:172.21.15.46:6789/0 conn(0x7f05a81538a0 legacy=0x7f05a80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:27.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 --2- 172.21.15.46:0/1095874658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0598003980 0x7f0598003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 --2- 172.21.15.46:0/1095874658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a815d6f0 0x7f05980047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 --2- 172.21.15.46:0/1095874658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0598004d10 0x7f0598005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80cb330 con 0x7f0598004d10 2023-10-30T13:33:27.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80cad70 con 0x7f0598003980 2023-10-30T13:33:27.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059800b990 con 0x7f05a815d6f0 2023-10-30T13:33:27.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05ad162640 1 --2- 172.21.15.46:0/1095874658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0598004d10 0x7f0598005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05ac961640 1 --2- 172.21.15.46:0/1095874658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0598003980 0x7f0598003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05a7fff640 1 --2- 172.21.15.46:0/1095874658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a815d6f0 0x7f05980047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05ad162640 1 -- 172.21.15.46:0/1095874658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a815d6f0 msgr2=0x7f05980047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:27.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05ad162640 1 --2- 172.21.15.46:0/1095874658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a815d6f0 0x7f05980047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05ad162640 1 -- 172.21.15.46:0/1095874658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0598003980 msgr2=0x7f0598003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:27.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05ad162640 1 --2- 172.21.15.46:0/1095874658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0598003980 0x7f0598003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.814+0000 7f05ad162640 1 -- 172.21.15.46:0/1095874658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0598003530 con 0x7f0598004d10 2023-10-30T13:33:27.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.818+0000 7f05ad162640 1 --2- 172.21.15.46:0/1095874658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0598004d10 0x7f0598005190 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009fc0 tx=0x7f05a001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dea9b73735fd9f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:27.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.818+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a00a5cb0 con 0x7f0598004d10 2023-10-30T13:33:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.818+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a000f040 con 0x7f0598004d10 2023-10-30T13:33:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.818+0000 7f05a77fe640 1 -- 172.21.15.46:0/1095874658 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a00ad680 con 0x7f0598004d10 2023-10-30T13:33:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.818+0000 7f05aebc5640 1 -- 172.21.15.46:0/1095874658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0598004d10 msgr2=0x7f0598005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.818+0000 7f05aebc5640 1 --2- 172.21.15.46:0/1095874658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0598004d10 0x7f0598005190 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009fc0 tx=0x7f05a001cca0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.818+0000 7f05ad162640 1 -- 172.21.15.46:0/1095874658 reap_dead start 2023-10-30T13:33:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05aebc5640 1 -- 172.21.15.46:0/1095874658 shutdown_connections 2023-10-30T13:33:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05aebc5640 1 -- 172.21.15.46:0/1095874658 >> 172.21.15.46:0/1095874658 conn(0x7f05a806bfe0 msgr2=0x7f05a806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05aebc5640 1 -- 172.21.15.46:0/1095874658 shutdown_connections 2023-10-30T13:33:27.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05aebc5640 1 -- 172.21.15.46:0/1095874658 wait complete. 2023-10-30T13:33:27.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05aebc5640 1 Processor -- start 2023-10-30T13:33:27.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05aebc5640 1 -- start start 2023-10-30T13:33:27.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05aebc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0598004d10 0x7f05a80c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05aebc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a8156210 0x7f05a81569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05aebc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a81ef2f0 0x7f05a81ef770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05aebc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80c6ed0 con 0x7f05a81ef2f0 2023-10-30T13:33:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05aebc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a80c8030 con 0x7f0598004d10 2023-10-30T13:33:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05aebc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8155b50 con 0x7f05a8156210 2023-10-30T13:33:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05ad162640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a81ef2f0 0x7f05a81ef770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05ad162640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a81ef2f0 0x7f05a81ef770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51698/0 (socket says 172.21.15.46:51698) 2023-10-30T13:33:27.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05ad162640 1 -- 172.21.15.46:0/200169595 learned_addr learned my addr 172.21.15.46:0/200169595 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:27.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05a7fff640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a8156210 0x7f05a81569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05ac961640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0598004d10 0x7f05a80c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05ad162640 1 -- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a8156210 msgr2=0x7f05a81569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:27.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05ad162640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a8156210 0x7f05a81569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05ad162640 1 -- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0598004d10 msgr2=0x7f05a80c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:27.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05ad162640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0598004d10 0x7f05a80c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:27.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05ad162640 1 -- 172.21.15.46:0/200169595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a8155e30 con 0x7f05a81ef2f0 2023-10-30T13:33:27.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05a7fff640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a8156210 0x7f05a81569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:33:27.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05ac961640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0598004d10 0x7f05a80c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:33:27.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.822+0000 7f05ad162640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a81ef2f0 0x7f05a81ef770 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f05a00b0040 tx=0x7f05a0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:27.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.826+0000 7f05a57fa640 1 -- 172.21.15.46:0/200169595 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a00039a0 con 0x7f05a81ef2f0 2023-10-30T13:33:27.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.826+0000 7f05a57fa640 1 -- 172.21.15.46:0/200169595 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a0003b00 con 0x7f05a81ef2f0 2023-10-30T13:33:27.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.826+0000 7f05a57fa640 1 -- 172.21.15.46:0/200169595 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0003e90 con 0x7f05a81ef2f0 2023-10-30T13:33:27.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.826+0000 7f05aebc5640 1 -- 172.21.15.46:0/200169595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a80c82a0 con 0x7f05a81ef2f0 2023-10-30T13:33:27.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.826+0000 7f05aebc5640 1 -- 172.21.15.46:0/200169595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a80c8860 con 0x7f05a81ef2f0 2023-10-30T13:33:27.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.826+0000 7f05aebc5640 1 -- 172.21.15.46:0/200169595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a804e170 con 0x7f05a81ef2f0 2023-10-30T13:33:27.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.826+0000 7f05a57fa640 1 -- 172.21.15.46:0/200169595 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f05a00b4040 con 0x7f05a81ef2f0 2023-10-30T13:33:27.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.826+0000 7f05a57fa640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f058c0664a0 0x7f058c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:27.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.826+0000 7f05a57fa640 1 -- 172.21.15.46:0/200169595 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f05a011df60 con 0x7f05a81ef2f0 2023-10-30T13:33:27.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.830+0000 7f05ac961640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f058c0664a0 0x7f058c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:27.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.830+0000 7f05ac961640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f058c0664a0 0x7f058c068960 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f059c000c10 tx=0x7f059c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:27.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:27.830+0000 7f05a57fa640 1 -- 172.21.15.46:0/200169595 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f05a00eb190 con 0x7f05a81ef2f0 2023-10-30T13:33:28.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 -- 172.21.15.46:0/200169595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05a80b1dd0 con 0x7f05a81ef2f0 2023-10-30T13:33:28.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05a57fa640 1 -- 172.21.15.46:0/200169595 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f05a00b2090 con 0x7f05a81ef2f0 2023-10-30T13:33:28.030 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:28.031 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5574,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:32:58.952603+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:28.031 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 53 2023-10-30T13:33:28.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 -- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f058c0664a0 msgr2=0x7f058c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f058c0664a0 0x7f058c068960 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f059c000c10 tx=0x7f059c009040 comp rx=0 tx=0).stop 2023-10-30T13:33:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 -- 172.21.15.46:0/200169595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a81ef2f0 msgr2=0x7f05a81ef770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a81ef2f0 0x7f05a81ef770 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f05a00b0040 tx=0x7f05a0009750 comp rx=0 tx=0).stop 2023-10-30T13:33:28.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 -- 172.21.15.46:0/200169595 shutdown_connections 2023-10-30T13:33:28.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f058c0664a0 0x7f058c068960 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:28.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a81ef2f0 0x7f05a81ef770 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:28.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05a8156210 0x7f05a81569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:28.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 --2- 172.21.15.46:0/200169595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0598004d10 0x7f05a80c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:28.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 -- 172.21.15.46:0/200169595 >> 172.21.15.46:0/200169595 conn(0x7f05a806bfe0 msgr2=0x7f05a80c5390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:28.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 -- 172.21.15.46:0/200169595 shutdown_connections 2023-10-30T13:33:28.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:28.026+0000 7f05aebc5640 1 -- 172.21.15.46:0/200169595 wait complete. 2023-10-30T13:33:28.052 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:33:30.053 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.194+0000 7f6c64dd3640 1 Processor -- start 2023-10-30T13:33:30.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.194+0000 7f6c64dd3640 1 -- start start 2023-10-30T13:33:30.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c64dd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c600c6b80 0x7f6c600bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:30.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c64dd3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6c6013f4a0 0x7f6c600c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:30.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c64dd3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c600c0a50 con 0x7f6c600c6f60 2023-10-30T13:33:30.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c64dd3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c6006f6e0 con 0x7f6c6013f4a0 2023-10-30T13:33:30.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c64dd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6006f9c0 con 0x7f6c600c6b80 2023-10-30T13:33:30.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c57fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6c600c6f60 0x7f6c6013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53464/0 (socket says 172.21.15.46:53464) 2023-10-30T13:33:30.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c57fff640 1 -- 172.21.15.46:0/4116073506 learned_addr learned my addr 172.21.15.46:0/4116073506 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:33:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5e59c640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c600c6b80 0x7f6c600bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5ed9d640 1 --2- 172.21.15.46:0/4116073506 >> v2:172.21.15.154:3300/0 conn(0x7f6c6013f4a0 0x7f6c600c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775571928 0 0) 0x7f6c600c0a50 con 0x7f6c600c6f60 2023-10-30T13:33:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c4c003530 con 0x7f6c600c6f60 2023-10-30T13:33:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2632343479 0 0) 0x7f6c4c003530 con 0x7f6c600c6f60 2023-10-30T13:33:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c600c6b80 msgr2=0x7f6c600bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c600c6b80 0x7f6c600bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 >> v2:172.21.15.154:3300/0 conn(0x7f6c6013f4a0 msgr2=0x7f6c600c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 --2- 172.21.15.46:0/4116073506 >> v2:172.21.15.154:3300/0 conn(0x7f6c6013f4a0 0x7f6c600c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6006fca0 con 0x7f6c600c6f60 2023-10-30T13:33:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f6c48001680 con 0x7f6c600c6f60 2023-10-30T13:33:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 >> v1:172.21.15.46:6789/0 conn(0x7f6c600c6f60 legacy=0x7f6c6013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c4c003980 0x7f6c4c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c4c0047d0 0x7f6c4c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c600c6b80 0x7f6c4c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6006f6e0 con 0x7f6c4c003980 2023-10-30T13:33:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6006f9c0 con 0x7f6c4c0047d0 2023-10-30T13:33:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c00dba0 con 0x7f6c600c6b80 2023-10-30T13:33:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5e59c640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c4c003980 0x7f6c4c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c57fff640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c4c0047d0 0x7f6c4c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5ed9d640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c600c6b80 0x7f6c4c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5e59c640 1 -- 172.21.15.46:0/4116073506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c600c6b80 msgr2=0x7f6c4c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5e59c640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c600c6b80 0x7f6c4c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5e59c640 1 -- 172.21.15.46:0/4116073506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c4c0047d0 msgr2=0x7f6c4c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5e59c640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c4c0047d0 0x7f6c4c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c5e59c640 1 -- 172.21.15.46:0/4116073506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c003530 con 0x7f6c4c003980 2023-10-30T13:33:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.198+0000 7f6c57fff640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c4c0047d0 0x7f6c4c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:33:30.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.202+0000 7f6c5e59c640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c4c003980 0x7f6c4c003da0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6c50009ae0 tx=0x7f6c5001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751b54e1d657efb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:30.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.202+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c500a6460 con 0x7f6c4c003980 2023-10-30T13:33:30.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.202+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c500a65c0 con 0x7f6c4c003980 2023-10-30T13:33:30.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.202+0000 7f6c5dd9b640 1 -- 172.21.15.46:0/4116073506 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c500a6970 con 0x7f6c4c003980 2023-10-30T13:33:30.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.202+0000 7f6c64dd3640 1 -- 172.21.15.46:0/4116073506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c4c003980 msgr2=0x7f6c4c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:30.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.202+0000 7f6c64dd3640 1 --2- 172.21.15.46:0/4116073506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c4c003980 0x7f6c4c003da0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6c50009ae0 tx=0x7f6c5001ca50 comp rx=0 tx=0).stop 2023-10-30T13:33:30.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.202+0000 7f6c5ed9d640 1 -- 172.21.15.46:0/4116073506 reap_dead start 2023-10-30T13:33:30.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.202+0000 7f6c64dd3640 1 -- 172.21.15.46:0/4116073506 shutdown_connections 2023-10-30T13:33:30.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.202+0000 7f6c64dd3640 1 -- 172.21.15.46:0/4116073506 >> 172.21.15.46:0/4116073506 conn(0x7f6c600c7910 msgr2=0x7f6c600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:30.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.202+0000 7f6c64dd3640 1 -- 172.21.15.46:0/4116073506 shutdown_connections 2023-10-30T13:33:30.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.202+0000 7f6c64dd3640 1 -- 172.21.15.46:0/4116073506 wait complete. 2023-10-30T13:33:30.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c64dd3640 1 Processor -- start 2023-10-30T13:33:30.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c64dd3640 1 -- start start 2023-10-30T13:33:30.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c64dd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c4c003980 0x7f6c601f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:30.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c64dd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c6013cc30 0x7f6c60159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:30.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c64dd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c601f5cb0 0x7f6c601f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:30.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c64dd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6013de00 con 0x7f6c601f5cb0 2023-10-30T13:33:30.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c64dd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6015ad20 con 0x7f6c4c003980 2023-10-30T13:33:30.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c64dd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c601f30b0 con 0x7f6c6013cc30 2023-10-30T13:33:30.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c5ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c601f5cb0 0x7f6c601f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:30.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c5ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c601f5cb0 0x7f6c601f6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51704/0 (socket says 172.21.15.46:51704) 2023-10-30T13:33:30.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c5ed9d640 1 -- 172.21.15.46:0/790083907 learned_addr learned my addr 172.21.15.46:0/790083907 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:30.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c57fff640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c6013cc30 0x7f6c60159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:30.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c5e59c640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c4c003980 0x7f6c601f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c5ed9d640 1 -- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c6013cc30 msgr2=0x7f6c60159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c5ed9d640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c6013cc30 0x7f6c60159210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c5ed9d640 1 -- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c4c003980 msgr2=0x7f6c601f3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c5ed9d640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c4c003980 0x7f6c601f3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c5ed9d640 1 -- 172.21.15.46:0/790083907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c601f27a0 con 0x7f6c601f5cb0 2023-10-30T13:33:30.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c5ed9d640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c601f5cb0 0x7f6c601f6130 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6c6006f4f0 tx=0x7f6c580115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:30.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c56ffd640 1 -- 172.21.15.46:0/790083907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5800e070 con 0x7f6c601f5cb0 2023-10-30T13:33:30.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c56ffd640 1 -- 172.21.15.46:0/790083907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c58011e40 con 0x7f6c601f5cb0 2023-10-30T13:33:30.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c64dd3640 1 -- 172.21.15.46:0/790083907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c601f6750 con 0x7f6c601f5cb0 2023-10-30T13:33:30.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c56ffd640 1 -- 172.21.15.46:0/790083907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c580039a0 con 0x7f6c601f5cb0 2023-10-30T13:33:30.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.206+0000 7f6c64dd3640 1 -- 172.21.15.46:0/790083907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c601f2a60 con 0x7f6c601f5cb0 2023-10-30T13:33:30.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.210+0000 7f6c64dd3640 1 -- 172.21.15.46:0/790083907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6004e170 con 0x7f6c601f5cb0 2023-10-30T13:33:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.210+0000 7f6c56ffd640 1 -- 172.21.15.46:0/790083907 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6c58003b00 con 0x7f6c601f5cb0 2023-10-30T13:33:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.210+0000 7f6c56ffd640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6c400664d0 0x7f6c40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.210+0000 7f6c56ffd640 1 -- 172.21.15.46:0/790083907 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f6c58084750 con 0x7f6c601f5cb0 2023-10-30T13:33:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.210+0000 7f6c5e59c640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6c400664d0 0x7f6c40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.210+0000 7f6c5e59c640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6c400664d0 0x7f6c40068990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6c500ad000 tx=0x7f6c500b3040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:30.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.214+0000 7f6c56ffd640 1 -- 172.21.15.46:0/790083907 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6c58051a20 con 0x7f6c601f5cb0 2023-10-30T13:33:30.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 -- 172.21.15.46:0/790083907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c601f31f0 con 0x7f6c601f5cb0 2023-10-30T13:33:30.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c56ffd640 1 -- 172.21.15.46:0/790083907 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f6c58055040 con 0x7f6c601f5cb0 2023-10-30T13:33:30.411 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:30.411 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5574,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:28.965063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:30.412 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2023-10-30T13:33:30.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 -- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6c400664d0 msgr2=0x7f6c40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:30.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6c400664d0 0x7f6c40068990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6c500ad000 tx=0x7f6c500b3040 comp rx=0 tx=0).stop 2023-10-30T13:33:30.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 -- 172.21.15.46:0/790083907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c601f5cb0 msgr2=0x7f6c601f6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:30.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c601f5cb0 0x7f6c601f6130 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6c6006f4f0 tx=0x7f6c580115a0 comp rx=0 tx=0).stop 2023-10-30T13:33:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 -- 172.21.15.46:0/790083907 shutdown_connections 2023-10-30T13:33:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6c400664d0 0x7f6c40068990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c601f5cb0 0x7f6c601f6130 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c6013cc30 0x7f6c60159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 --2- 172.21.15.46:0/790083907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c4c003980 0x7f6c601f3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 -- 172.21.15.46:0/790083907 >> 172.21.15.46:0/790083907 conn(0x7f6c600c7910 msgr2=0x7f6c60159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 -- 172.21.15.46:0/790083907 shutdown_connections 2023-10-30T13:33:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:30.406+0000 7f6c64dd3640 1 -- 172.21.15.46:0/790083907 wait complete. 2023-10-30T13:33:30.432 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:33:32.433 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:32.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f53879640 1 Processor -- start 2023-10-30T13:33:32.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f53879640 1 -- start start 2023-10-30T13:33:32.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f53879640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6f4c1538a0 0x7f6f4c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:32.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f53879640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6f4c0cf160 0x7f6f4c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:32.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f53879640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c0cb800 con 0x7f6f4c0ca0e0 2023-10-30T13:33:32.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f53879640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4c15dc30 con 0x7f6f4c1538a0 2023-10-30T13:33:32.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f53879640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4c0cb330 con 0x7f6f4c0cf160 2023-10-30T13:33:32.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f51615640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6f4c1538a0 0x7f6f4c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:32.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f51615640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6f4c1538a0 0x7f6f4c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51720/0 (socket says 172.21.15.46:51720) 2023-10-30T13:33:32.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f51615640 1 -- 172.21.15.46:0/3444451122 learned_addr learned my addr 172.21.15.46:0/3444451122 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:32.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f51e16640 1 --2- 172.21.15.46:0/3444451122 >> v2:172.21.15.154:3300/0 conn(0x7f6f4c0cf160 0x7f6f4c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:32.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f51615640 1 -- 172.21.15.46:0/3444451122 >> v2:172.21.15.154:3300/0 conn(0x7f6f4c0cf160 msgr2=0x7f6f4c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:32.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f51615640 1 --2- 172.21.15.46:0/3444451122 >> v2:172.21.15.154:3300/0 conn(0x7f6f4c0cf160 0x7f6f4c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:32.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f51615640 1 -- 172.21.15.46:0/3444451122 >> v1:172.21.15.154:6789/0 conn(0x7f6f4c0ca0e0 legacy=0x7f6f4c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:32.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.598+0000 7f6f51615640 1 -- 172.21.15.46:0/3444451122 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c0cad70 con 0x7f6f4c1538a0 2023-10-30T13:33:32.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.602+0000 7f6f51615640 1 --2- 172.21.15.46:0/3444451122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f4c1538a0 0x7f6f4c15d6f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c0b5330 tx=0x7f6f4801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9b0239f4f5c89e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:32.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.602+0000 7f6f43fff640 1 -- 172.21.15.46:0/3444451122 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f480a89b0 con 0x7f6f4c1538a0 2023-10-30T13:33:32.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.602+0000 7f6f43fff640 1 -- 172.21.15.46:0/3444451122 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f480a8b10 con 0x7f6f4c1538a0 2023-10-30T13:33:32.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.602+0000 7f6f43fff640 1 -- 172.21.15.46:0/3444451122 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f480a8ea0 con 0x7f6f4c1538a0 2023-10-30T13:33:32.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.602+0000 7f6f53879640 1 -- 172.21.15.46:0/3444451122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f4c1538a0 msgr2=0x7f6f4c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:32.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.602+0000 7f6f53879640 1 --2- 172.21.15.46:0/3444451122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f4c1538a0 0x7f6f4c15d6f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c0b5330 tx=0x7f6f4801ecb0 comp rx=0 tx=0).stop 2023-10-30T13:33:32.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.602+0000 7f6f53879640 1 -- 172.21.15.46:0/3444451122 shutdown_connections 2023-10-30T13:33:32.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.602+0000 7f6f53879640 1 --2- 172.21.15.46:0/3444451122 >> v2:172.21.15.154:3300/0 conn(0x7f6f4c0cf160 0x7f6f4c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:32.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.602+0000 7f6f53879640 1 --2- 172.21.15.46:0/3444451122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f4c1538a0 0x7f6f4c15d6f0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:32.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.602+0000 7f6f53879640 1 -- 172.21.15.46:0/3444451122 >> 172.21.15.46:0/3444451122 conn(0x7f6f4c06bfe0 msgr2=0x7f6f4c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:32.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.602+0000 7f6f53879640 1 -- 172.21.15.46:0/3444451122 shutdown_connections 2023-10-30T13:33:32.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.602+0000 7f6f53879640 1 -- 172.21.15.46:0/3444451122 wait complete. 2023-10-30T13:33:32.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f53879640 1 Processor -- start 2023-10-30T13:33:32.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f53879640 1 -- start start 2023-10-30T13:33:32.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f53879640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f4c0ca0e0 0x7f6f4c0c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:32.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f53879640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f4c0cf160 0x7f6f4c157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:32.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f53879640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f4c1538a0 0x7f6f4c1e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:32.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f53879640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c1ef4c0 con 0x7f6f4c1538a0 2023-10-30T13:33:32.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f53879640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c0c9180 con 0x7f6f4c0cf160 2023-10-30T13:33:32.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f53879640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c1ef730 con 0x7f6f4c0ca0e0 2023-10-30T13:33:32.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f51e16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f4c1538a0 0x7f6f4c1e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:32.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f51e16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f4c1538a0 0x7f6f4c1e5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51724/0 (socket says 172.21.15.46:51724) 2023-10-30T13:33:32.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f51e16640 1 -- 172.21.15.46:0/1721063005 learned_addr learned my addr 172.21.15.46:0/1721063005 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:32.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f51615640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f4c0ca0e0 0x7f6f4c0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:32.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f50e14640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f4c0cf160 0x7f6f4c157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:32.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f51e16640 1 -- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f4c0ca0e0 msgr2=0x7f6f4c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:32.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f51e16640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f4c0ca0e0 0x7f6f4c0c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:32.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f51e16640 1 -- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f4c0cf160 msgr2=0x7f6f4c157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:32.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f51e16640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f4c0cf160 0x7f6f4c157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:32.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f51e16640 1 -- 172.21.15.46:0/1721063005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f480094d0 con 0x7f6f4c1538a0 2023-10-30T13:33:32.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f51e16640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f4c1538a0 0x7f6f4c1e5040 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6f44007d90 tx=0x7f6f4400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:32.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f41ffb640 1 -- 172.21.15.46:0/1721063005 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f44014420 con 0x7f6f4c1538a0 2023-10-30T13:33:32.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f41ffb640 1 -- 172.21.15.46:0/1721063005 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f440145a0 con 0x7f6f4c1538a0 2023-10-30T13:33:32.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f41ffb640 1 -- 172.21.15.46:0/1721063005 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f44014950 con 0x7f6f4c1538a0 2023-10-30T13:33:32.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f53879640 1 -- 172.21.15.46:0/1721063005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4c1efa10 con 0x7f6f4c1538a0 2023-10-30T13:33:32.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.606+0000 7f6f53879640 1 -- 172.21.15.46:0/1721063005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4c0c9460 con 0x7f6f4c1538a0 2023-10-30T13:33:32.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.610+0000 7f6f53879640 1 -- 172.21.15.46:0/1721063005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4c0b2a50 con 0x7f6f4c1538a0 2023-10-30T13:33:32.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.610+0000 7f6f41ffb640 1 -- 172.21.15.46:0/1721063005 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6f44014ab0 con 0x7f6f4c1538a0 2023-10-30T13:33:32.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.610+0000 7f6f41ffb640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6f300664a0 0x7f6f30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:32.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.610+0000 7f6f41ffb640 1 -- 172.21.15.46:0/1721063005 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f6f44084960 con 0x7f6f4c1538a0 2023-10-30T13:33:32.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.610+0000 7f6f51615640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6f300664a0 0x7f6f30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:32.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.610+0000 7f6f51615640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6f300664a0 0x7f6f30068960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6f480b2000 tx=0x7f6f48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:32.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.614+0000 7f6f41ffb640 1 -- 172.21.15.46:0/1721063005 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6f44051ba0 con 0x7f6f4c1538a0 2023-10-30T13:33:32.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 -- 172.21.15.46:0/1721063005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f4c04e170 con 0x7f6f4c1538a0 2023-10-30T13:33:32.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f41ffb640 1 -- 172.21.15.46:0/1721063005 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f6f440551c0 con 0x7f6f4c1538a0 2023-10-30T13:33:32.814 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:32.814 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5574,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:28.965063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:32.815 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2023-10-30T13:33:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 -- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6f300664a0 msgr2=0x7f6f30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6f300664a0 0x7f6f30068960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6f480b2000 tx=0x7f6f48009040 comp rx=0 tx=0).stop 2023-10-30T13:33:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 -- 172.21.15.46:0/1721063005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f4c1538a0 msgr2=0x7f6f4c1e5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f4c1538a0 0x7f6f4c1e5040 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6f44007d90 tx=0x7f6f4400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:33:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 -- 172.21.15.46:0/1721063005 shutdown_connections 2023-10-30T13:33:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6f300664a0 0x7f6f30068960 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f4c1538a0 0x7f6f4c1e5040 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f4c0cf160 0x7f6f4c157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 --2- 172.21.15.46:0/1721063005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f4c0ca0e0 0x7f6f4c0c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 -- 172.21.15.46:0/1721063005 >> 172.21.15.46:0/1721063005 conn(0x7f6f4c06bfe0 msgr2=0x7f6f4c1519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 -- 172.21.15.46:0/1721063005 shutdown_connections 2023-10-30T13:33:32.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:32.810+0000 7f6f53879640 1 -- 172.21.15.46:0/1721063005 wait complete. 2023-10-30T13:33:32.835 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:33:34.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:34.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.986+0000 7f13567e4640 1 Processor -- start 2023-10-30T13:33:34.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.986+0000 7f13567e4640 1 -- start start 2023-10-30T13:33:34.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.990+0000 7f13567e4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13500cefa0 0x7f13500cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:34.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.990+0000 7f13567e4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f135015a6c0 con 0x7f135015cd50 2023-10-30T13:33:34.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.990+0000 7f13567e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13500ca6a0 con 0x7f13501538f0 2023-10-30T13:33:34.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.990+0000 7f13567e4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13500cf8f0 con 0x7f13500cefa0 2023-10-30T13:33:34.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.990+0000 7f1355fe3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13500cefa0 0x7f13500cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:34.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.990+0000 7f13557e2640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f13501538f0 0x7f1350159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:34816/0 (socket says 172.21.15.46:34816) 2023-10-30T13:33:34.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.990+0000 7f13557e2640 1 -- 172.21.15.46:0/3199066930 learned_addr learned my addr 172.21.15.46:0/3199066930 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:33:34.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.990+0000 7f1355fe3640 1 -- 172.21.15.46:0/3199066930 >> v1:172.21.15.154:6789/0 conn(0x7f13501538f0 legacy=0x7f1350159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:34.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.990+0000 7f1355fe3640 1 -- 172.21.15.46:0/3199066930 >> v1:172.21.15.46:6789/0 conn(0x7f135015cd50 legacy=0x7f13500c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:34.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.990+0000 7f1355fe3640 1 -- 172.21.15.46:0/3199066930 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135015d300 con 0x7f13500cefa0 2023-10-30T13:33:34.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.990+0000 7f1347fff640 1 -- 172.21.15.46:0/3199066930 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638940397 0 0) 0x7f135015a6c0 con 0x7f135015cd50 2023-10-30T13:33:34.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.994+0000 7f1355fe3640 1 --2- 172.21.15.46:0/3199066930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13500cefa0 0x7f13500cf3b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1348009d90 tx=0x7f134801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd9bf48c1d56114 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:34.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.994+0000 7f1347fff640 1 -- 172.21.15.46:0/3199066930 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13480af600 con 0x7f13500cefa0 2023-10-30T13:33:34.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.994+0000 7f1347fff640 1 -- 172.21.15.46:0/3199066930 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13480ab050 con 0x7f13500cefa0 2023-10-30T13:33:34.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.994+0000 7f1347fff640 1 -- 172.21.15.46:0/3199066930 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13480af980 con 0x7f13500cefa0 2023-10-30T13:33:35.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.994+0000 7f13567e4640 1 -- 172.21.15.46:0/3199066930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13500cefa0 msgr2=0x7f13500cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:35.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.994+0000 7f13567e4640 1 --2- 172.21.15.46:0/3199066930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13500cefa0 0x7f13500cf3b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1348009d90 tx=0x7f134801f130 comp rx=0 tx=0).stop 2023-10-30T13:33:35.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.994+0000 7f13567e4640 1 -- 172.21.15.46:0/3199066930 shutdown_connections 2023-10-30T13:33:35.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.994+0000 7f13567e4640 1 --2- 172.21.15.46:0/3199066930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13500cefa0 0x7f13500cf3b0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:35.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.994+0000 7f13567e4640 1 -- 172.21.15.46:0/3199066930 >> 172.21.15.46:0/3199066930 conn(0x7f135006a640 msgr2=0x7f1350065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:35.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.994+0000 7f13567e4640 1 -- 172.21.15.46:0/3199066930 shutdown_connections 2023-10-30T13:33:35.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.994+0000 7f13567e4640 1 -- 172.21.15.46:0/3199066930 wait complete. 2023-10-30T13:33:35.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13567e4640 1 Processor -- start 2023-10-30T13:33:35.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13567e4640 1 -- start start 2023-10-30T13:33:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13567e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13500cefa0 0x7f1350157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13567e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13501538f0 0x7f13500b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13567e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f135015cd50 0x7f1350158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13567e4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1350153150 con 0x7f13500cefa0 2023-10-30T13:33:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13567e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13501521d0 con 0x7f135015cd50 2023-10-30T13:33:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13567e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1350152ef0 con 0x7f13501538f0 2023-10-30T13:33:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13557e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13500cefa0 0x7f1350157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13557e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13500cefa0 0x7f1350157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51742/0 (socket says 172.21.15.46:51742) 2023-10-30T13:33:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13557e2640 1 -- 172.21.15.46:0/3935250644 learned_addr learned my addr 172.21.15.46:0/3935250644 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f1354fe1640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13501538f0 0x7f13500b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f1355fe3640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f135015cd50 0x7f1350158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13557e2640 1 -- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13501538f0 msgr2=0x7f13500b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13557e2640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13501538f0 0x7f13500b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13557e2640 1 -- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f135015cd50 msgr2=0x7f1350158f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13557e2640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f135015cd50 0x7f1350158f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13557e2640 1 -- 172.21.15.46:0/3935250644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1348009a70 con 0x7f13500cefa0 2023-10-30T13:33:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f1355fe3640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f135015cd50 0x7f1350158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:33:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13557e2640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13500cefa0 0x7f1350157e50 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f134c00c500 tx=0x7f134c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f1345ffb640 1 -- 172.21.15.46:0/3935250644 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c005cf0 con 0x7f13500cefa0 2023-10-30T13:33:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f1345ffb640 1 -- 172.21.15.46:0/3935250644 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c005e50 con 0x7f13500cefa0 2023-10-30T13:33:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f1345ffb640 1 -- 172.21.15.46:0/3935250644 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c0072f0 con 0x7f13500cefa0 2023-10-30T13:33:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13567e4640 1 -- 172.21.15.46:0/3935250644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13500b5200 con 0x7f13500cefa0 2023-10-30T13:33:35.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:34.998+0000 7f13567e4640 1 -- 172.21.15.46:0/3935250644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13500b4f60 con 0x7f13500cefa0 2023-10-30T13:33:35.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.002+0000 7f13567e4640 1 -- 172.21.15.46:0/3935250644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135004e170 con 0x7f13500cefa0 2023-10-30T13:33:35.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.002+0000 7f1345ffb640 1 -- 172.21.15.46:0/3935250644 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f134c007450 con 0x7f13500cefa0 2023-10-30T13:33:35.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.002+0000 7f1345ffb640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13340664a0 0x7f1334068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:35.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.002+0000 7f1345ffb640 1 -- 172.21.15.46:0/3935250644 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f134c0835d0 con 0x7f13500cefa0 2023-10-30T13:33:35.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.002+0000 7f1354fe1640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13340664a0 0x7f1334068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:35.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.002+0000 7f1354fe1640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13340664a0 0x7f1334068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f13500ce930 tx=0x7f1340009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.006+0000 7f1345ffb640 1 -- 172.21.15.46:0/3935250644 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f134c050890 con 0x7f13500cefa0 2023-10-30T13:33:35.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 -- 172.21.15.46:0/3935250644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f135014d530 con 0x7f13500cefa0 2023-10-30T13:33:35.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f1345ffb640 1 -- 172.21.15.46:0/3935250644 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f134c053eb0 con 0x7f13500cefa0 2023-10-30T13:33:35.202 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:35.202 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5574,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:28.965063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:35.203 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2023-10-30T13:33:35.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 -- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13340664a0 msgr2=0x7f1334068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:35.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13340664a0 0x7f1334068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f13500ce930 tx=0x7f1340009040 comp rx=0 tx=0).stop 2023-10-30T13:33:35.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 -- 172.21.15.46:0/3935250644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13500cefa0 msgr2=0x7f1350157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:35.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13500cefa0 0x7f1350157e50 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f134c00c500 tx=0x7f134c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:33:35.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 -- 172.21.15.46:0/3935250644 shutdown_connections 2023-10-30T13:33:35.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13340664a0 0x7f1334068960 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:35.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f135015cd50 0x7f1350158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:35.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13501538f0 0x7f13500b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:35.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 --2- 172.21.15.46:0/3935250644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13500cefa0 0x7f1350157e50 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:35.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 -- 172.21.15.46:0/3935250644 >> 172.21.15.46:0/3935250644 conn(0x7f135006a640 msgr2=0x7f13500c9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:35.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 -- 172.21.15.46:0/3935250644 shutdown_connections 2023-10-30T13:33:35.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:35.198+0000 7f13567e4640 1 -- 172.21.15.46:0/3935250644 wait complete. 2023-10-30T13:33:35.223 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:33:37.224 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:37.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa34bb94640 1 Processor -- start 2023-10-30T13:33:37.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa34bb94640 1 -- start start 2023-10-30T13:33:37.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa34bb94640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa344153bf0 0x7fa344071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:37.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa34bb94640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3440cb550 0x7fa3440cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:37.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa34bb94640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3440c7b90 con 0x7fa3440c6fe0 2023-10-30T13:33:37.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa34bb94640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34415d870 con 0x7fa344153bf0 2023-10-30T13:33:37.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa34bb94640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440c60e0 con 0x7fa3440cb550 2023-10-30T13:33:37.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa34a131640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3440cb550 0x7fa3440cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa349930640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa344153bf0 0x7fa344071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa34a131640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3440cb550 0x7fa3440cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:56054/0 (socket says 172.21.15.46:56054) 2023-10-30T13:33:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa349930640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa344153bf0 0x7fa344071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:42500/0 (socket says 172.21.15.46:42500) 2023-10-30T13:33:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa34a131640 1 -- 172.21.15.46:0/1125587676 learned_addr learned my addr 172.21.15.46:0/1125587676 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:37.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.374+0000 7fa34912f640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa3440c6fe0 0x7fa3440c7480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:60144/0 (socket says 172.21.15.46:60144) 2023-10-30T13:33:37.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.378+0000 7fa34a131640 1 -- 172.21.15.46:0/1125587676 >> v2:172.21.15.154:3300/0 conn(0x7fa344153bf0 msgr2=0x7fa344071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:37.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.378+0000 7fa34a131640 1 --2- 172.21.15.46:0/1125587676 >> v2:172.21.15.154:3300/0 conn(0x7fa344153bf0 0x7fa344071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:37.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.378+0000 7fa34a131640 1 -- 172.21.15.46:0/1125587676 >> v1:172.21.15.154:6789/0 conn(0x7fa3440c6fe0 legacy=0x7fa3440c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:37.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.378+0000 7fa34a131640 1 -- 172.21.15.46:0/1125587676 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3440c63a0 con 0x7fa3440cb550 2023-10-30T13:33:37.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.378+0000 7fa34a131640 1 --2- 172.21.15.46:0/1125587676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3440cb550 0x7fa3440cb9c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa33c009e50 tx=0x7fa33c01c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=617bad9af797ea65 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:37.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.378+0000 7fa34892e640 1 -- 172.21.15.46:0/1125587676 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0a58a0 con 0x7fa3440cb550 2023-10-30T13:33:37.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.378+0000 7fa34892e640 1 -- 172.21.15.46:0/1125587676 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33c0a5a00 con 0x7fa3440cb550 2023-10-30T13:33:37.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.378+0000 7fa34892e640 1 -- 172.21.15.46:0/1125587676 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0a5d90 con 0x7fa3440cb550 2023-10-30T13:33:37.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 -- 172.21.15.46:0/1125587676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3440cb550 msgr2=0x7fa3440cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:37.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 --2- 172.21.15.46:0/1125587676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3440cb550 0x7fa3440cb9c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa33c009e50 tx=0x7fa33c01c9e0 comp rx=0 tx=0).stop 2023-10-30T13:33:37.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 -- 172.21.15.46:0/1125587676 shutdown_connections 2023-10-30T13:33:37.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 --2- 172.21.15.46:0/1125587676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3440cb550 0x7fa3440cb9c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:37.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 --2- 172.21.15.46:0/1125587676 >> v2:172.21.15.154:3300/0 conn(0x7fa344153bf0 0x7fa344071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:37.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 -- 172.21.15.46:0/1125587676 >> 172.21.15.46:0/1125587676 conn(0x7fa34406c6e0 msgr2=0x7fa344069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:37.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 -- 172.21.15.46:0/1125587676 shutdown_connections 2023-10-30T13:33:37.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 -- 172.21.15.46:0/1125587676 wait complete. 2023-10-30T13:33:37.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 Processor -- start 2023-10-30T13:33:37.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 -- start start 2023-10-30T13:33:37.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3440c6fe0 0x7fa3440bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:37.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3440cb550 0x7fa344149ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa344153bf0 0x7fa3440c06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34415aaa0 con 0x7fa3440c6fe0 2023-10-30T13:33:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3441f1ff0 con 0x7fa3440cb550 2023-10-30T13:33:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34bb94640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3441f22d0 con 0x7fa344153bf0 2023-10-30T13:33:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa349930640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3440c6fe0 0x7fa3440bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:37.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa349930640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3440c6fe0 0x7fa3440bfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42444/0 (socket says 172.21.15.46:42444) 2023-10-30T13:33:37.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa349930640 1 -- 172.21.15.46:0/1776255338 learned_addr learned my addr 172.21.15.46:0/1776255338 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:37.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34912f640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3440cb550 0x7fa344149ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:37.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.382+0000 7fa34a131640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa344153bf0 0x7fa3440c06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:37.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa349930640 1 -- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa344153bf0 msgr2=0x7fa3440c06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:37.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa349930640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa344153bf0 0x7fa3440c06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa349930640 1 -- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3440cb550 msgr2=0x7fa344149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa349930640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3440cb550 0x7fa344149ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa349930640 1 -- 172.21.15.46:0/1776255338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c009b30 con 0x7fa3440c6fe0 2023-10-30T13:33:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa349930640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3440c6fe0 0x7fa3440bfa10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa34000e870 tx=0x7fa34000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa33a7fc640 1 -- 172.21.15.46:0/1776255338 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340009510 con 0x7fa3440c6fe0 2023-10-30T13:33:37.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa33a7fc640 1 -- 172.21.15.46:0/1776255338 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34000eec0 con 0x7fa3440c6fe0 2023-10-30T13:33:37.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa33a7fc640 1 -- 172.21.15.46:0/1776255338 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34000cb40 con 0x7fa3440c6fe0 2023-10-30T13:33:37.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa34bb94640 1 -- 172.21.15.46:0/1776255338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3441f25b0 con 0x7fa3440c6fe0 2023-10-30T13:33:37.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa34bb94640 1 -- 172.21.15.46:0/1776255338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3441f2b70 con 0x7fa3440c6fe0 2023-10-30T13:33:37.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa34bb94640 1 -- 172.21.15.46:0/1776255338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34404e170 con 0x7fa3440c6fe0 2023-10-30T13:33:37.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa33a7fc640 1 -- 172.21.15.46:0/1776255338 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa34000cca0 con 0x7fa3440c6fe0 2023-10-30T13:33:37.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa33a7fc640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3280664d0 0x7fa328068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:37.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.386+0000 7fa33a7fc640 1 -- 172.21.15.46:0/1776255338 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fa340084230 con 0x7fa3440c6fe0 2023-10-30T13:33:37.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.390+0000 7fa34912f640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3280664d0 0x7fa328068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:37.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.390+0000 7fa34912f640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3280664d0 0x7fa328068990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa344072130 tx=0x7fa334007e50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.390+0000 7fa33a7fc640 1 -- 172.21.15.46:0/1776255338 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3400513d0 con 0x7fa3440c6fe0 2023-10-30T13:33:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 -- 172.21.15.46:0/1776255338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa34406e8d0 con 0x7fa3440c6fe0 2023-10-30T13:33:37.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa33a7fc640 1 -- 172.21.15.46:0/1776255338 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6527 (secure 0 0 0) 0x7fa3400549f0 con 0x7fa3440c6fe0 2023-10-30T13:33:37.586 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:37.586 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5574,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:28.965063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:37.587 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2023-10-30T13:33:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 -- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3280664d0 msgr2=0x7fa328068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3280664d0 0x7fa328068990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa344072130 tx=0x7fa334007e50 comp rx=0 tx=0).stop 2023-10-30T13:33:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 -- 172.21.15.46:0/1776255338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3440c6fe0 msgr2=0x7fa3440bfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3440c6fe0 0x7fa3440bfa10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa34000e870 tx=0x7fa34000ec70 comp rx=0 tx=0).stop 2023-10-30T13:33:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 -- 172.21.15.46:0/1776255338 shutdown_connections 2023-10-30T13:33:37.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3280664d0 0x7fa328068990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:37.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa344153bf0 0x7fa3440c06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:37.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3440cb550 0x7fa344149ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:37.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 --2- 172.21.15.46:0/1776255338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3440c6fe0 0x7fa3440bfa10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:37.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 -- 172.21.15.46:0/1776255338 >> 172.21.15.46:0/1776255338 conn(0x7fa34406c6e0 msgr2=0x7fa3440c7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:37.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 -- 172.21.15.46:0/1776255338 shutdown_connections 2023-10-30T13:33:37.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:37.582+0000 7fa34bb94640 1 -- 172.21.15.46:0/1776255338 wait complete. 2023-10-30T13:33:37.607 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:33:39.607 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:39.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.758+0000 7f0f3fa46640 1 Processor -- start 2023-10-30T13:33:39.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.758+0000 7f0f3fa46640 1 -- start start 2023-10-30T13:33:39.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.758+0000 7f0f3fa46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f381580e0 0x7f0f38070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:39.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.758+0000 7f0f3fa46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f380c5810 con 0x7f0f3815e700 2023-10-30T13:33:39.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.758+0000 7f0f3fa46640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3815eb90 con 0x7f0f380c6540 2023-10-30T13:33:39.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.758+0000 7f0f3fa46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3806acb0 con 0x7f0f381580e0 2023-10-30T13:33:39.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.758+0000 7f0f3dfe3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0f3815e700 0x7f0f3806a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56546/0 (socket says 172.21.15.46:56546) 2023-10-30T13:33:39.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.758+0000 7f0f3dfe3640 1 -- 172.21.15.46:0/2814882001 learned_addr learned my addr 172.21.15.46:0/2814882001 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:33:39.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.758+0000 7f0f3d7e2640 1 --2- 172.21.15.46:0/2814882001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f381580e0 0x7f0f38070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:39.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.762+0000 7f0f2ffff640 1 -- 172.21.15.46:0/2814882001 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939259223 0 0) 0x7f0f380c5810 con 0x7f0f3815e700 2023-10-30T13:33:39.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.762+0000 7f0f2ffff640 1 -- 172.21.15.46:0/2814882001 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f24003530 con 0x7f0f3815e700 2023-10-30T13:33:39.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.762+0000 7f0f2ffff640 1 -- 172.21.15.46:0/2814882001 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364099747 0 0) 0x7f0f3815eb90 con 0x7f0f380c6540 2023-10-30T13:33:39.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.762+0000 7f0f2ffff640 1 -- 172.21.15.46:0/2814882001 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f380c5810 con 0x7f0f380c6540 2023-10-30T13:33:39.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.762+0000 7f0f3d7e2640 1 -- 172.21.15.46:0/2814882001 >> v1:172.21.15.154:6789/0 conn(0x7f0f380c6540 legacy=0x7f0f38069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:39.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.762+0000 7f0f3d7e2640 1 -- 172.21.15.46:0/2814882001 >> v1:172.21.15.46:6789/0 conn(0x7f0f3815e700 legacy=0x7f0f3806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:39.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.762+0000 7f0f3d7e2640 1 -- 172.21.15.46:0/2814882001 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f380c6960 con 0x7f0f381580e0 2023-10-30T13:33:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.762+0000 7f0f3d7e2640 1 --2- 172.21.15.46:0/2814882001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f381580e0 0x7f0f38070740 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0f340097f0 tx=0x7f0f3401c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8be2a11de7f747b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.762+0000 7f0f2ffff640 1 -- 172.21.15.46:0/2814882001 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f340aa620 con 0x7f0f381580e0 2023-10-30T13:33:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f2ffff640 1 -- 172.21.15.46:0/2814882001 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f340aa780 con 0x7f0f381580e0 2023-10-30T13:33:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f2ffff640 1 -- 172.21.15.46:0/2814882001 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f340aaaf0 con 0x7f0f381580e0 2023-10-30T13:33:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 -- 172.21.15.46:0/2814882001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f381580e0 msgr2=0x7f0f38070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 --2- 172.21.15.46:0/2814882001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f381580e0 0x7f0f38070740 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0f340097f0 tx=0x7f0f3401c4c0 comp rx=0 tx=0).stop 2023-10-30T13:33:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 -- 172.21.15.46:0/2814882001 shutdown_connections 2023-10-30T13:33:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 --2- 172.21.15.46:0/2814882001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f381580e0 0x7f0f38070740 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 -- 172.21.15.46:0/2814882001 >> 172.21.15.46:0/2814882001 conn(0x7f0f380c7910 msgr2=0x7f0f380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 -- 172.21.15.46:0/2814882001 shutdown_connections 2023-10-30T13:33:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 -- 172.21.15.46:0/2814882001 wait complete. 2023-10-30T13:33:39.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 Processor -- start 2023-10-30T13:33:39.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 -- start start 2023-10-30T13:33:39.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f380c6540 0x7f0f380c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:39.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f381580e0 0x7f0f38147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:39.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f3815e700 0x7f0f381483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:39.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f380c9fb0 con 0x7f0f380c6540 2023-10-30T13:33:39.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f380c4fe0 con 0x7f0f381580e0 2023-10-30T13:33:39.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3fa46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f380ca200 con 0x7f0f3815e700 2023-10-30T13:33:39.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3d7e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f380c6540 0x7f0f380c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:39.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3d7e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f380c6540 0x7f0f380c1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42450/0 (socket says 172.21.15.46:42450) 2023-10-30T13:33:39.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.766+0000 7f0f3d7e2640 1 -- 172.21.15.46:0/37296611 learned_addr learned my addr 172.21.15.46:0/37296611 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:39.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f3dfe3640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f3815e700 0x7f0f381483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:39.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f3cfe1640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f381580e0 0x7f0f38147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:39.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f3d7e2640 1 -- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f3815e700 msgr2=0x7f0f381483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:39.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f3d7e2640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f3815e700 0x7f0f381483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:39.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f3d7e2640 1 -- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f381580e0 msgr2=0x7f0f38147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:39.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f3d7e2640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f381580e0 0x7f0f38147ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:39.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f3d7e2640 1 -- 172.21.15.46:0/37296611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f34009420 con 0x7f0f380c6540 2023-10-30T13:33:39.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f3d7e2640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f380c6540 0x7f0f380c1170 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0f340ac040 tx=0x7f0f34019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:39.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f2dffb640 1 -- 172.21.15.46:0/37296611 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f340aa8f0 con 0x7f0f380c6540 2023-10-30T13:33:39.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f2dffb640 1 -- 172.21.15.46:0/37296611 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f340aaa50 con 0x7f0f380c6540 2023-10-30T13:33:39.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f2dffb640 1 -- 172.21.15.46:0/37296611 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f340aadc0 con 0x7f0f380c6540 2023-10-30T13:33:39.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f3fa46640 1 -- 172.21.15.46:0/37296611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f380ca4e0 con 0x7f0f380c6540 2023-10-30T13:33:39.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f3fa46640 1 -- 172.21.15.46:0/37296611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f380caaa0 con 0x7f0f380c6540 2023-10-30T13:33:39.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f3fa46640 1 -- 172.21.15.46:0/37296611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3804e170 con 0x7f0f380c6540 2023-10-30T13:33:39.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f2dffb640 1 -- 172.21.15.46:0/37296611 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0f34003620 con 0x7f0f380c6540 2023-10-30T13:33:39.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.770+0000 7f0f2dffb640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f1c0664a0 0x7f0f1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:39.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.774+0000 7f0f2dffb640 1 -- 172.21.15.46:0/37296611 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f0f3411e930 con 0x7f0f380c6540 2023-10-30T13:33:39.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.774+0000 7f0f3cfe1640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f1c0664a0 0x7f0f1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:39.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.774+0000 7f0f3cfe1640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f1c0664a0 0x7f0f1c068960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0f380c9f40 tx=0x7f0f28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:39.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.778+0000 7f0f2dffb640 1 -- 172.21.15.46:0/37296611 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0f340ebab0 con 0x7f0f380c6540 2023-10-30T13:33:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.970+0000 7f0f3fa46640 1 -- 172.21.15.46:0/37296611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f3806aed0 con 0x7f0f380c6540 2023-10-30T13:33:39.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.970+0000 7f0f2dffb640 1 -- 172.21.15.46:0/37296611 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f0f340ef0d0 con 0x7f0f380c6540 2023-10-30T13:33:39.976 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:39.977 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5574,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:28.965063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:39.978 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2023-10-30T13:33:39.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.974+0000 7f0f3fa46640 1 -- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f1c0664a0 msgr2=0x7f0f1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:39.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.974+0000 7f0f3fa46640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f1c0664a0 0x7f0f1c068960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0f380c9f40 tx=0x7f0f28009040 comp rx=0 tx=0).stop 2023-10-30T13:33:39.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.974+0000 7f0f3fa46640 1 -- 172.21.15.46:0/37296611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f380c6540 msgr2=0x7f0f380c1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:39.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.974+0000 7f0f3fa46640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f380c6540 0x7f0f380c1170 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0f340ac040 tx=0x7f0f34019260 comp rx=0 tx=0).stop 2023-10-30T13:33:39.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.974+0000 7f0f3fa46640 1 -- 172.21.15.46:0/37296611 shutdown_connections 2023-10-30T13:33:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.974+0000 7f0f3fa46640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f1c0664a0 0x7f0f1c068960 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.974+0000 7f0f3fa46640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f3815e700 0x7f0f381483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:39.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.974+0000 7f0f3fa46640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f381580e0 0x7f0f38147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.974+0000 7f0f3fa46640 1 --2- 172.21.15.46:0/37296611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f380c6540 0x7f0f380c1170 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.974+0000 7f0f3fa46640 1 -- 172.21.15.46:0/37296611 >> 172.21.15.46:0/37296611 conn(0x7f0f380c7910 msgr2=0x7f0f38069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:39.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.974+0000 7f0f3fa46640 1 -- 172.21.15.46:0/37296611 shutdown_connections 2023-10-30T13:33:39.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:39.974+0000 7f0f3fa46640 1 -- 172.21.15.46:0/37296611 wait complete. 2023-10-30T13:33:39.997 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:33:41.998 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:42.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.146+0000 7fb020809640 1 Processor -- start 2023-10-30T13:33:42.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.146+0000 7fb020809640 1 -- start start 2023-10-30T13:33:42.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb020809640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb01c06f7a0 0x7fb01c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:42.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb020809640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c066cf0 con 0x7fb01c071110 2023-10-30T13:33:42.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb020809640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c0c70d0 con 0x7fb01c0c6bd0 2023-10-30T13:33:42.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb020809640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb01c0700c0 con 0x7fb01c06f7a0 2023-10-30T13:33:42.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb01ad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb01c071110 0x7fb01c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56548/0 (socket says 172.21.15.46:56548) 2023-10-30T13:33:42.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb01ad9d640 1 -- 172.21.15.46:0/4075156460 learned_addr learned my addr 172.21.15.46:0/4075156460 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:33:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb01a59c640 1 --2- 172.21.15.46:0/4075156460 >> v2:172.21.15.154:3300/0 conn(0x7fb01c06f7a0 0x7fb01c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469037077 0 0) 0x7fb01c066cf0 con 0x7fb01c071110 2023-10-30T13:33:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb008003530 con 0x7fb01c071110 2023-10-30T13:33:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3437001150 0 0) 0x7fb008003530 con 0x7fb01c071110 2023-10-30T13:33:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 >> v2:172.21.15.154:3300/0 conn(0x7fb01c06f7a0 msgr2=0x7fb01c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 --2- 172.21.15.46:0/4075156460 >> v2:172.21.15.154:3300/0 conn(0x7fb01c06f7a0 0x7fb01c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 >> v1:172.21.15.154:6789/0 conn(0x7fb01c0c6bd0 legacy=0x7fb01c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c0c7390 con 0x7fb01c071110 2023-10-30T13:33:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fb014001a00 con 0x7fb01c071110 2023-10-30T13:33:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 >> v1:172.21.15.46:6789/0 conn(0x7fb01c071110 legacy=0x7fb01c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 --2- 172.21.15.46:0/4075156460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb008003980 0x7fb008003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 --2- 172.21.15.46:0/4075156460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0080047d0 0x7fb008004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 --2- 172.21.15.46:0/4075156460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb008005190 0x7fb00800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c0700c0 con 0x7fb008005190 2023-10-30T13:33:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00800c040 con 0x7fb008003980 2023-10-30T13:33:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00800c200 con 0x7fb0080047d0 2023-10-30T13:33:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb01a59c640 1 --2- 172.21.15.46:0/4075156460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb008003980 0x7fb008003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb01ad9d640 1 --2- 172.21.15.46:0/4075156460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb008005190 0x7fb00800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb013fff640 1 --2- 172.21.15.46:0/4075156460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0080047d0 0x7fb008004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb01ad9d640 1 -- 172.21.15.46:0/4075156460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0080047d0 msgr2=0x7fb008004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb01ad9d640 1 --2- 172.21.15.46:0/4075156460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0080047d0 0x7fb008004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb01ad9d640 1 -- 172.21.15.46:0/4075156460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb008003980 msgr2=0x7fb008003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb01ad9d640 1 --2- 172.21.15.46:0/4075156460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb008003980 0x7fb008003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.150+0000 7fb01ad9d640 1 -- 172.21.15.46:0/4075156460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb008003530 con 0x7fb008005190 2023-10-30T13:33:42.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.154+0000 7fb01ad9d640 1 -- 172.21.15.46:0/4075156460 reap_dead start 2023-10-30T13:33:42.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.154+0000 7fb01ad9d640 1 --2- 172.21.15.46:0/4075156460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb008005190 0x7fb00800b990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb01400a830 tx=0x7fb01401d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c91964e31ffab162 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.154+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0140072d0 con 0x7fb008005190 2023-10-30T13:33:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.154+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0140a8070 con 0x7fb008005190 2023-10-30T13:33:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.154+0000 7fb019d9b640 1 -- 172.21.15.46:0/4075156460 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0140076b0 con 0x7fb008005190 2023-10-30T13:33:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.154+0000 7fb020809640 1 -- 172.21.15.46:0/4075156460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb008005190 msgr2=0x7fb00800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.154+0000 7fb020809640 1 --2- 172.21.15.46:0/4075156460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb008005190 0x7fb00800b990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb01400a830 tx=0x7fb01401d340 comp rx=0 tx=0).stop 2023-10-30T13:33:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.154+0000 7fb020809640 1 -- 172.21.15.46:0/4075156460 shutdown_connections 2023-10-30T13:33:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.154+0000 7fb020809640 1 --2- 172.21.15.46:0/4075156460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb008005190 0x7fb00800b990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.154+0000 7fb020809640 1 -- 172.21.15.46:0/4075156460 >> 172.21.15.46:0/4075156460 conn(0x7fb01c0c7910 msgr2=0x7fb01c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.154+0000 7fb020809640 1 -- 172.21.15.46:0/4075156460 shutdown_connections 2023-10-30T13:33:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.154+0000 7fb020809640 1 -- 172.21.15.46:0/4075156460 wait complete. 2023-10-30T13:33:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb020809640 1 Processor -- start 2023-10-30T13:33:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb020809640 1 -- start start 2023-10-30T13:33:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb020809640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb008005190 0x7fb01c1ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb020809640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb01c0c9540 0x7fb01c15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb020809640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c156940 0x7fb01c156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb020809640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c156520 con 0x7fb01c156940 2023-10-30T13:33:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb020809640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c1efde0 con 0x7fb008005190 2023-10-30T13:33:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb020809640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c1580e0 con 0x7fb01c0c9540 2023-10-30T13:33:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb01ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c156940 0x7fb01c156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb01ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c156940 0x7fb01c156dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42472/0 (socket says 172.21.15.46:42472) 2023-10-30T13:33:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb01ad9d640 1 -- 172.21.15.46:0/3915610520 learned_addr learned my addr 172.21.15.46:0/3915610520 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb01a59c640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb008005190 0x7fb01c1ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:42.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb013fff640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb01c0c9540 0x7fb01c15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:42.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb01ad9d640 1 -- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb01c0c9540 msgr2=0x7fb01c15b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:42.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb01ad9d640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb01c0c9540 0x7fb01c15b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:42.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb01ad9d640 1 -- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb008005190 msgr2=0x7fb01c1ef8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:42.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb01ad9d640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb008005190 0x7fb01c1ef8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:42.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb01ad9d640 1 -- 172.21.15.46:0/3915610520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c155600 con 0x7fb01c156940 2023-10-30T13:33:42.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb01ad9d640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c156940 0x7fb01c156dc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb014007040 tx=0x7fb008005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:42.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb012ffd640 1 -- 172.21.15.46:0/3915610520 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0140019c0 con 0x7fb01c156940 2023-10-30T13:33:42.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb012ffd640 1 -- 172.21.15.46:0/3915610520 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb014007450 con 0x7fb01c156940 2023-10-30T13:33:42.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb012ffd640 1 -- 172.21.15.46:0/3915610520 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0140077e0 con 0x7fb01c156940 2023-10-30T13:33:42.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb020809640 1 -- 172.21.15.46:0/3915610520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c1558a0 con 0x7fb01c156940 2023-10-30T13:33:42.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.158+0000 7fb020809640 1 -- 172.21.15.46:0/3915610520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01c155e60 con 0x7fb01c156940 2023-10-30T13:33:42.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.162+0000 7fb020809640 1 -- 172.21.15.46:0/3915610520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01c04e170 con 0x7fb01c156940 2023-10-30T13:33:42.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.162+0000 7fb012ffd640 1 -- 172.21.15.46:0/3915610520 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb0140aa070 con 0x7fb01c156940 2023-10-30T13:33:42.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.162+0000 7fb012ffd640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faffc0664d0 0x7faffc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:42.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.162+0000 7fb012ffd640 1 -- 172.21.15.46:0/3915610520 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fb01411d9f0 con 0x7fb01c156940 2023-10-30T13:33:42.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.162+0000 7fb01a59c640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faffc0664d0 0x7faffc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:42.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.162+0000 7fb01a59c640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faffc0664d0 0x7faffc068990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009ac0 tx=0x7fb00c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:42.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.166+0000 7fb012ffd640 1 -- 172.21.15.46:0/3915610520 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb0140eab10 con 0x7fb01c156940 2023-10-30T13:33:42.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.358+0000 7fb020809640 1 -- 172.21.15.46:0/3915610520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb01c070be0 con 0x7fb01c156940 2023-10-30T13:33:42.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.358+0000 7fb012ffd640 1 -- 172.21.15.46:0/3915610520 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6527 (secure 0 0 0) 0x7fb0140eca40 con 0x7fb01c156940 2023-10-30T13:33:42.365 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:42.366 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5574,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:28.965063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5106,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6833/603884510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":603884510},{"type":"v1","addr":"172.21.15.46:6833","nonce":603884510}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2023-10-30T13:33:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.362+0000 7fb020809640 1 -- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faffc0664d0 msgr2=0x7faffc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.362+0000 7fb020809640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faffc0664d0 0x7faffc068990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009ac0 tx=0x7fb00c009190 comp rx=0 tx=0).stop 2023-10-30T13:33:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.362+0000 7fb020809640 1 -- 172.21.15.46:0/3915610520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c156940 msgr2=0x7fb01c156dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.362+0000 7fb020809640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c156940 0x7fb01c156dc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb014007040 tx=0x7fb008005800 comp rx=0 tx=0).stop 2023-10-30T13:33:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.362+0000 7fb020809640 1 -- 172.21.15.46:0/3915610520 shutdown_connections 2023-10-30T13:33:42.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.362+0000 7fb020809640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faffc0664d0 0x7faffc068990 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:42.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.362+0000 7fb020809640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb01c156940 0x7fb01c156dc0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:42.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.362+0000 7fb020809640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb01c0c9540 0x7fb01c15b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:42.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.362+0000 7fb020809640 1 --2- 172.21.15.46:0/3915610520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb008005190 0x7fb01c1ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:42.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.362+0000 7fb020809640 1 -- 172.21.15.46:0/3915610520 >> 172.21.15.46:0/3915610520 conn(0x7fb01c0c7910 msgr2=0x7fb01c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.362+0000 7fb020809640 1 -- 172.21.15.46:0/3915610520 shutdown_connections 2023-10-30T13:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:42.362+0000 7fb020809640 1 -- 172.21.15.46:0/3915610520 wait complete. 2023-10-30T13:33:42.386 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:33:44.387 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:44.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f58486c9640 1 Processor -- start 2023-10-30T13:33:44.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f58486c9640 1 -- start start 2023-10-30T13:33:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f58486c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f584015d6f0 0x7f584015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f58486c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58400cf160 0x7f58400cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f58486c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58400cb800 con 0x7f58401538a0 2023-10-30T13:33:44.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f58486c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58400cb330 con 0x7f584015d6f0 2023-10-30T13:33:44.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f58486c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58400cad70 con 0x7f58400cf160 2023-10-30T13:33:44.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f5846c66640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58400cf160 0x7f58400cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f5845c64640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f584015d6f0 0x7f584015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f5846c66640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58400cf160 0x7f58400cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:56116/0 (socket says 172.21.15.46:56116) 2023-10-30T13:33:44.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f5846c66640 1 -- 172.21.15.46:0/1604599182 learned_addr learned my addr 172.21.15.46:0/1604599182 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:44.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f5846c66640 1 -- 172.21.15.46:0/1604599182 >> v2:172.21.15.154:3300/0 conn(0x7f584015d6f0 msgr2=0x7f584015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:44.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f5846c66640 1 --2- 172.21.15.46:0/1604599182 >> v2:172.21.15.154:3300/0 conn(0x7f584015d6f0 0x7f584015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f5846c66640 1 -- 172.21.15.46:0/1604599182 >> v1:172.21.15.154:6789/0 conn(0x7f58401538a0 legacy=0x7f58400ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:44.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.538+0000 7f5846c66640 1 -- 172.21.15.46:0/1604599182 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58400cb050 con 0x7f58400cf160 2023-10-30T13:33:44.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.542+0000 7f5846c66640 1 --2- 172.21.15.46:0/1604599182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58400cf160 0x7f58400cf590 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5838009e50 tx=0x7f583801f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90439677c98b4636 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:44.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.542+0000 7f5845463640 1 -- 172.21.15.46:0/1604599182 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58380af400 con 0x7f58400cf160 2023-10-30T13:33:44.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.542+0000 7f5845463640 1 -- 172.21.15.46:0/1604599182 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58380ab050 con 0x7f58400cf160 2023-10-30T13:33:44.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.542+0000 7f5845463640 1 -- 172.21.15.46:0/1604599182 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58380af780 con 0x7f58400cf160 2023-10-30T13:33:44.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.542+0000 7f58486c9640 1 -- 172.21.15.46:0/1604599182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58400cf160 msgr2=0x7f58400cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:44.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.542+0000 7f58486c9640 1 --2- 172.21.15.46:0/1604599182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58400cf160 0x7f58400cf590 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5838009e50 tx=0x7f583801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 -- 172.21.15.46:0/1604599182 shutdown_connections 2023-10-30T13:33:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 --2- 172.21.15.46:0/1604599182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58400cf160 0x7f58400cf590 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 --2- 172.21.15.46:0/1604599182 >> v2:172.21.15.154:3300/0 conn(0x7f584015d6f0 0x7f584015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 -- 172.21.15.46:0/1604599182 >> 172.21.15.46:0/1604599182 conn(0x7f584006bfe0 msgr2=0x7f584006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 -- 172.21.15.46:0/1604599182 shutdown_connections 2023-10-30T13:33:44.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 -- 172.21.15.46:0/1604599182 wait complete. 2023-10-30T13:33:44.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 Processor -- start 2023-10-30T13:33:44.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 -- start start 2023-10-30T13:33:44.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58400cf160 0x7f58400ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58401538a0 0x7f584015b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f584015d6f0 0x7f58400b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58400b3870 con 0x7f58401538a0 2023-10-30T13:33:44.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58400b32b0 con 0x7f584015d6f0 2023-10-30T13:33:44.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58400b3590 con 0x7f58400cf160 2023-10-30T13:33:44.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5845c64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58401538a0 0x7f584015b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5845c64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58401538a0 0x7f584015b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42486/0 (socket says 172.21.15.46:42486) 2023-10-30T13:33:44.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5845c64640 1 -- 172.21.15.46:0/3526556413 learned_addr learned my addr 172.21.15.46:0/3526556413 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:44.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5846c66640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f584015d6f0 0x7f58400b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5846465640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58400cf160 0x7f58400ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5845c64640 1 -- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58400cf160 msgr2=0x7f58400ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:44.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5845c64640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58400cf160 0x7f58400ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5845c64640 1 -- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f584015d6f0 msgr2=0x7f58400b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:44.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5845c64640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f584015d6f0 0x7f58400b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5845c64640 1 -- 172.21.15.46:0/3526556413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5838009b30 con 0x7f58401538a0 2023-10-30T13:33:44.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5845c64640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58401538a0 0x7f584015b6f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f583000e870 tx=0x7f583000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:44.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5836ffd640 1 -- 172.21.15.46:0/3526556413 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830009510 con 0x7f58401538a0 2023-10-30T13:33:44.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5836ffd640 1 -- 172.21.15.46:0/3526556413 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583000eec0 con 0x7f58401538a0 2023-10-30T13:33:44.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f5836ffd640 1 -- 172.21.15.46:0/3526556413 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583000cb40 con 0x7f58401538a0 2023-10-30T13:33:44.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.546+0000 7f58486c9640 1 -- 172.21.15.46:0/3526556413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5840149a90 con 0x7f58401538a0 2023-10-30T13:33:44.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.550+0000 7f58486c9640 1 -- 172.21.15.46:0/3526556413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58400cede0 con 0x7f58401538a0 2023-10-30T13:33:44.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.550+0000 7f58486c9640 1 -- 172.21.15.46:0/3526556413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f584015b540 con 0x7f58401538a0 2023-10-30T13:33:44.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.550+0000 7f5836ffd640 1 -- 172.21.15.46:0/3526556413 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f583000cca0 con 0x7f58401538a0 2023-10-30T13:33:44.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.550+0000 7f5836ffd640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58240664d0 0x7f5824068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.550+0000 7f5836ffd640 1 -- 172.21.15.46:0/3526556413 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f58300844f0 con 0x7f58401538a0 2023-10-30T13:33:44.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.550+0000 7f5846465640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58240664d0 0x7f5824068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.554+0000 7f5846465640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58240664d0 0x7f5824068990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f583c000f70 tx=0x7f583c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:44.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.554+0000 7f5836ffd640 1 -- 172.21.15.46:0/3526556413 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5830051770 con 0x7f58401538a0 2023-10-30T13:33:44.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.746+0000 7f58486c9640 1 -- 172.21.15.46:0/3526556413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58400b2a50 con 0x7f58401538a0 2023-10-30T13:33:44.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.746+0000 7f5836ffd640 1 -- 172.21.15.46:0/3526556413 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f5830054d90 con 0x7f58401538a0 2023-10-30T13:33:44.752 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:44.752 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:43.868255+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:44.753 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 55 2023-10-30T13:33:44.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.750+0000 7f58486c9640 1 -- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58240664d0 msgr2=0x7f5824068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:44.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.750+0000 7f58486c9640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58240664d0 0x7f5824068990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f583c000f70 tx=0x7f583c009040 comp rx=0 tx=0).stop 2023-10-30T13:33:44.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.750+0000 7f58486c9640 1 -- 172.21.15.46:0/3526556413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58401538a0 msgr2=0x7f584015b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:44.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.750+0000 7f58486c9640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58401538a0 0x7f584015b6f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f583000e870 tx=0x7f583000ec70 comp rx=0 tx=0).stop 2023-10-30T13:33:44.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.750+0000 7f58486c9640 1 -- 172.21.15.46:0/3526556413 shutdown_connections 2023-10-30T13:33:44.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.750+0000 7f58486c9640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58240664d0 0x7f5824068990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.750+0000 7f58486c9640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f584015d6f0 0x7f58400b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.750+0000 7f58486c9640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58401538a0 0x7f584015b6f0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.750+0000 7f58486c9640 1 --2- 172.21.15.46:0/3526556413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58400cf160 0x7f58400ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.750+0000 7f58486c9640 1 -- 172.21.15.46:0/3526556413 >> 172.21.15.46:0/3526556413 conn(0x7f584006bfe0 msgr2=0x7f584015aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:44.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.750+0000 7f58486c9640 1 -- 172.21.15.46:0/3526556413 shutdown_connections 2023-10-30T13:33:44.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.750+0000 7f58486c9640 1 -- 172.21.15.46:0/3526556413 wait complete. 2023-10-30T13:33:44.773 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:33:44.773 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:33:44.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.910+0000 7f0fecf94640 1 Processor -- start 2023-10-30T13:33:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.910+0000 7f0fecf94640 1 -- start start 2023-10-30T13:33:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fecf94640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fe80c6b80 0x7f0fe80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fecf94640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0fe813f4a0 0x7f0fe80c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fecf94640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe80c0a50 con 0x7f0fe80c6f60 2023-10-30T13:33:44.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fecf94640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe806f6e0 con 0x7f0fe813f4a0 2023-10-30T13:33:44.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fecf94640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe806f9c0 con 0x7f0fe80c6b80 2023-10-30T13:33:44.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe5d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0fe80c6f60 0x7f0fe813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56552/0 (socket says 172.21.15.46:56552) 2023-10-30T13:33:44.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe5d9b640 1 -- 172.21.15.46:0/549845678 learned_addr learned my addr 172.21.15.46:0/549845678 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:33:44.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe6d9d640 1 --2- 172.21.15.46:0/549845678 >> v2:172.21.15.154:3300/0 conn(0x7f0fe813f4a0 0x7f0fe80c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe659c640 1 --2- 172.21.15.46:0/549845678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fe80c6b80 0x7f0fe80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119621308 0 0) 0x7f0fe80c0a50 con 0x7f0fe80c6f60 2023-10-30T13:33:44.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd0005740 con 0x7f0fe80c6f60 2023-10-30T13:33:44.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1593548910 0 0) 0x7f0fd0005740 con 0x7f0fe80c6f60 2023-10-30T13:33:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fe80c6b80 msgr2=0x7f0fe80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 --2- 172.21.15.46:0/549845678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fe80c6b80 0x7f0fe80bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 >> v2:172.21.15.154:3300/0 conn(0x7f0fe813f4a0 msgr2=0x7f0fe80c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 --2- 172.21.15.46:0/549845678 >> v2:172.21.15.154:3300/0 conn(0x7f0fe813f4a0 0x7f0fe80c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe806fca0 con 0x7f0fe80c6f60 2023-10-30T13:33:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f0fdc001680 con 0x7f0fe80c6f60 2023-10-30T13:33:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe6d9d640 1 --2- 172.21.15.46:0/549845678 >> v2:172.21.15.154:3300/0 conn(0x7f0fe813f4a0 0x7f0fe80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:33:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 >> v1:172.21.15.46:6789/0 conn(0x7f0fe80c6f60 legacy=0x7f0fe813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 --2- 172.21.15.46:0/549845678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd0005b90 0x7f0fd0005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 --2- 172.21.15.46:0/549845678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fe80c6b80 0x7f0fd00069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 --2- 172.21.15.46:0/549845678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fd0006f20 0x7f0fd00073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe806f6e0 con 0x7f0fd0005b90 2023-10-30T13:33:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe806f9c0 con 0x7f0fd0006f20 2023-10-30T13:33:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd000dba0 con 0x7f0fe80c6b80 2023-10-30T13:33:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe659c640 1 --2- 172.21.15.46:0/549845678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd0005b90 0x7f0fd0005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe5d9b640 1 --2- 172.21.15.46:0/549845678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fe80c6b80 0x7f0fd00069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe6d9d640 1 --2- 172.21.15.46:0/549845678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fd0006f20 0x7f0fd00073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe659c640 1 -- 172.21.15.46:0/549845678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fe80c6b80 msgr2=0x7f0fd00069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe659c640 1 --2- 172.21.15.46:0/549845678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fe80c6b80 0x7f0fd00069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe659c640 1 -- 172.21.15.46:0/549845678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fd0006f20 msgr2=0x7f0fd00073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe659c640 1 --2- 172.21.15.46:0/549845678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fd0006f20 0x7f0fd00073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.914+0000 7f0fe659c640 1 -- 172.21.15.46:0/549845678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd0005740 con 0x7f0fd0005b90 2023-10-30T13:33:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.918+0000 7f0fe659c640 1 --2- 172.21.15.46:0/549845678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd0005b90 0x7f0fd0005fb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8009ae0 tx=0x7f0fd801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39aacb8f1189c2c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.918+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80a5530 con 0x7f0fd0005b90 2023-10-30T13:33:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.918+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd80a5690 con 0x7f0fd0005b90 2023-10-30T13:33:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.918+0000 7f0fe559a640 1 -- 172.21.15.46:0/549845678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80a5a40 con 0x7f0fd0005b90 2023-10-30T13:33:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.918+0000 7f0fecf94640 1 -- 172.21.15.46:0/549845678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd0005b90 msgr2=0x7f0fd0005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.918+0000 7f0fecf94640 1 --2- 172.21.15.46:0/549845678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd0005b90 0x7f0fd0005fb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8009ae0 tx=0x7f0fd801ca50 comp rx=0 tx=0).stop 2023-10-30T13:33:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.918+0000 7f0fe6d9d640 1 -- 172.21.15.46:0/549845678 reap_dead start 2023-10-30T13:33:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.918+0000 7f0fecf94640 1 -- 172.21.15.46:0/549845678 shutdown_connections 2023-10-30T13:33:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.918+0000 7f0fecf94640 1 -- 172.21.15.46:0/549845678 >> 172.21.15.46:0/549845678 conn(0x7f0fe80c7910 msgr2=0x7f0fe80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.918+0000 7f0fecf94640 1 -- 172.21.15.46:0/549845678 shutdown_connections 2023-10-30T13:33:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.918+0000 7f0fecf94640 1 -- 172.21.15.46:0/549845678 wait complete. 2023-10-30T13:33:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fecf94640 1 Processor -- start 2023-10-30T13:33:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fecf94640 1 -- start start 2023-10-30T13:33:44.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fecf94640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fd0005b90 0x7f0fe8155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fecf94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe80c4fa0 0x7f0fe8158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fecf94640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fe81f12a0 0x7f0fe8136540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fecf94640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8155430 con 0x7f0fe80c4fa0 2023-10-30T13:33:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fecf94640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8154510 con 0x7f0fe81f12a0 2023-10-30T13:33:44.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fecf94640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe81547b0 con 0x7f0fd0005b90 2023-10-30T13:33:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fe5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe80c4fa0 0x7f0fe8158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fe5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe80c4fa0 0x7f0fe8158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42510/0 (socket says 172.21.15.46:42510) 2023-10-30T13:33:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fe5d9b640 1 -- 172.21.15.46:0/223900911 learned_addr learned my addr 172.21.15.46:0/223900911 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fe659c640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fd0005b90 0x7f0fe8155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fe6d9d640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fe81f12a0 0x7f0fe8136540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fe5d9b640 1 -- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fd0005b90 msgr2=0x7f0fe8155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fe5d9b640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fd0005b90 0x7f0fe8155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fe5d9b640 1 -- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fe81f12a0 msgr2=0x7f0fe8136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fe5d9b640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fe81f12a0 0x7f0fe8136540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fe5d9b640 1 -- 172.21.15.46:0/223900911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe8154a90 con 0x7f0fe80c4fa0 2023-10-30T13:33:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fe5d9b640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe80c4fa0 0x7f0fe8158d80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00e7f0 tx=0x7f0fdc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fd6ffd640 1 -- 172.21.15.46:0/223900911 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc00c8d0 con 0x7f0fe80c4fa0 2023-10-30T13:33:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fd6ffd640 1 -- 172.21.15.46:0/223900911 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fdc00ca30 con 0x7f0fe80c4fa0 2023-10-30T13:33:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fd6ffd640 1 -- 172.21.15.46:0/223900911 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc00cde0 con 0x7f0fe80c4fa0 2023-10-30T13:33:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fecf94640 1 -- 172.21.15.46:0/223900911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe8154d70 con 0x7f0fe80c4fa0 2023-10-30T13:33:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.922+0000 7f0fecf94640 1 -- 172.21.15.46:0/223900911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe8155d90 con 0x7f0fe80c4fa0 2023-10-30T13:33:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.926+0000 7f0fecf94640 1 -- 172.21.15.46:0/223900911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe804e170 con 0x7f0fe80c4fa0 2023-10-30T13:33:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.926+0000 7f0fd6ffd640 1 -- 172.21.15.46:0/223900911 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0fdc01d460 con 0x7f0fe80c4fa0 2023-10-30T13:33:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.926+0000 7f0fd6ffd640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0fc8066490 0x7f0fc8068950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.926+0000 7f0fd6ffd640 1 -- 172.21.15.46:0/223900911 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f0fdc084060 con 0x7f0fe80c4fa0 2023-10-30T13:33:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.926+0000 7f0fe659c640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0fc8066490 0x7f0fc8068950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.926+0000 7f0fe659c640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0fc8066490 0x7f0fc8068950 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80af000 tx=0x7f0fd80046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:44.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:44.930+0000 7f0fd6ffd640 1 -- 172.21.15.46:0/223900911 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0fdc0b5090 con 0x7f0fe80c4fa0 2023-10-30T13:33:45.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 -- 172.21.15.46:0/223900911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fe813e790 con 0x7f0fe80c4fa0 2023-10-30T13:33:45.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fd6ffd640 1 -- 172.21.15.46:0/223900911 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f0fdc051230 con 0x7f0fe80c4fa0 2023-10-30T13:33:45.126 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:45.127 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:43.868255+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 55 2023-10-30T13:33:45.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 -- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0fc8066490 msgr2=0x7f0fc8068950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0fc8066490 0x7f0fc8068950 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80af000 tx=0x7f0fd80046d0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 -- 172.21.15.46:0/223900911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe80c4fa0 msgr2=0x7f0fe8158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe80c4fa0 0x7f0fe8158d80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00e7f0 tx=0x7f0fdc00ecb0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 -- 172.21.15.46:0/223900911 shutdown_connections 2023-10-30T13:33:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0fc8066490 0x7f0fc8068950 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fe81f12a0 0x7f0fe8136540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe80c4fa0 0x7f0fe8158d80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 --2- 172.21.15.46:0/223900911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fd0005b90 0x7f0fe8155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 -- 172.21.15.46:0/223900911 >> 172.21.15.46:0/223900911 conn(0x7f0fe80c7910 msgr2=0x7f0fe80c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 -- 172.21.15.46:0/223900911 shutdown_connections 2023-10-30T13:33:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:45.122+0000 7f0fecf94640 1 -- 172.21.15.46:0/223900911 wait complete. 2023-10-30T13:33:45.149 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 55, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 55, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5266": { "addr": "172.21.15.154:6835/651441434", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 651441434, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 651441434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5266, "incarnation": 38, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 22 }, "gid_5442": { "addr": "172.21.15.46:6836/3955931693", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3955931693, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3955931693, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5442, "incarnation": 46, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 13 }, "gid_5574": { "addr": "172.21.15.46:6837/126953235", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 126953235, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 126953235, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5574, "incarnation": 55, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:33:43.868255+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5266, "mds_1": 5574, "mds_2": 5442 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:33:46.538 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:33:46.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:33:47.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.298+0000 7fdc38046640 1 Processor -- start 2023-10-30T13:33:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.298+0000 7fdc38046640 1 -- start start 2023-10-30T13:33:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc38046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc301580e0 0x7fdc30070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:47.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc38046640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdc300c6540 0x7fdc3015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:47.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc38046640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc300c5810 con 0x7fdc30069e90 2023-10-30T13:33:47.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc38046640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc300c6960 con 0x7fdc300c6540 2023-10-30T13:33:47.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc38046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc300c6bb0 con 0x7fdc301580e0 2023-10-30T13:33:47.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc355e1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdc300c6540 0x7fdc3015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:47.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc355e1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdc300c6540 0x7fdc3015e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55324/0 (socket says 172.21.15.46:55324) 2023-10-30T13:33:47.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc355e1640 1 -- 172.21.15.46:0/1430028792 learned_addr learned my addr 172.21.15.46:0/1430028792 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:47.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc35de2640 1 --2- 172.21.15.46:0/1430028792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc301580e0 0x7fdc30070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:47.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc355e1640 1 -- 172.21.15.46:0/1430028792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc301580e0 msgr2=0x7fdc30070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:47.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc355e1640 1 --2- 172.21.15.46:0/1430028792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc301580e0 0x7fdc30070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:47.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc355e1640 1 -- 172.21.15.46:0/1430028792 >> v1:172.21.15.154:6789/0 conn(0x7fdc30069e90 legacy=0x7fdc3006a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:47.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.302+0000 7fdc355e1640 1 -- 172.21.15.46:0/1430028792 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc300c6e70 con 0x7fdc300c6540 2023-10-30T13:33:47.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc355e1640 1 --2- 172.21.15.46:0/1430028792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc300c6540 0x7fdc3015e700 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009870 tx=0x7fdc2001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf2e916a17ee671 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:47.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc34de0640 1 -- 172.21.15.46:0/1430028792 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc200a9400 con 0x7fdc300c6540 2023-10-30T13:33:47.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc34de0640 1 -- 172.21.15.46:0/1430028792 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc200a9560 con 0x7fdc300c6540 2023-10-30T13:33:47.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc34de0640 1 -- 172.21.15.46:0/1430028792 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc200a98f0 con 0x7fdc300c6540 2023-10-30T13:33:47.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc38046640 1 -- 172.21.15.46:0/1430028792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc300c6540 msgr2=0x7fdc3015e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:47.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc38046640 1 --2- 172.21.15.46:0/1430028792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc300c6540 0x7fdc3015e700 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009870 tx=0x7fdc2001c540 comp rx=0 tx=0).stop 2023-10-30T13:33:47.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc38046640 1 -- 172.21.15.46:0/1430028792 shutdown_connections 2023-10-30T13:33:47.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc38046640 1 --2- 172.21.15.46:0/1430028792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc300c6540 0x7fdc3015e700 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:47.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc38046640 1 --2- 172.21.15.46:0/1430028792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc301580e0 0x7fdc30070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:47.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc38046640 1 -- 172.21.15.46:0/1430028792 >> 172.21.15.46:0/1430028792 conn(0x7fdc300c7910 msgr2=0x7fdc300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:47.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc38046640 1 -- 172.21.15.46:0/1430028792 shutdown_connections 2023-10-30T13:33:47.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc38046640 1 -- 172.21.15.46:0/1430028792 wait complete. 2023-10-30T13:33:47.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc38046640 1 Processor -- start 2023-10-30T13:33:47.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.306+0000 7fdc38046640 1 -- start start 2023-10-30T13:33:47.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc38046640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc30069e90 0x7fdc300c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:47.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc38046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc300c6540 0x7fdc3014acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:47.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc38046640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc301580e0 0x7fdc3014b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:47.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc38046640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3015f2f0 con 0x7fdc301580e0 2023-10-30T13:33:47.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc38046640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc301f6f60 con 0x7fdc30069e90 2023-10-30T13:33:47.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc38046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc300ca880 con 0x7fdc300c6540 2023-10-30T13:33:47.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc365e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc301580e0 0x7fdc3014b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:47.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc365e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc301580e0 0x7fdc3014b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55336/0 (socket says 172.21.15.46:55336) 2023-10-30T13:33:47.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc365e3640 1 -- 172.21.15.46:0/1190073940 learned_addr learned my addr 172.21.15.46:0/1190073940 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:47.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc355e1640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc300c6540 0x7fdc3014acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:47.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc35de2640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc30069e90 0x7fdc300c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:47.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc365e3640 1 -- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc300c6540 msgr2=0x7fdc3014acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:47.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc365e3640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc300c6540 0x7fdc3014acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:47.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc365e3640 1 -- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc30069e90 msgr2=0x7fdc300c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:47.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc365e3640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc30069e90 0x7fdc300c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:47.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc365e3640 1 -- 172.21.15.46:0/1190073940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc200094d0 con 0x7fdc301580e0 2023-10-30T13:33:47.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc365e3640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc301580e0 0x7fdc3014b210 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800c7a0 tx=0x7fdc2800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:47.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc267fc640 1 -- 172.21.15.46:0/1190073940 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc28003b80 con 0x7fdc301580e0 2023-10-30T13:33:47.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc267fc640 1 -- 172.21.15.46:0/1190073940 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc28003ce0 con 0x7fdc301580e0 2023-10-30T13:33:47.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc267fc640 1 -- 172.21.15.46:0/1190073940 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2800aa90 con 0x7fdc301580e0 2023-10-30T13:33:47.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc38046640 1 -- 172.21.15.46:0/1190073940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc300cab20 con 0x7fdc301580e0 2023-10-30T13:33:47.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc38046640 1 -- 172.21.15.46:0/1190073940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc301f6910 con 0x7fdc301580e0 2023-10-30T13:33:47.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc38046640 1 -- 172.21.15.46:0/1190073940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc300c6000 con 0x7fdc301580e0 2023-10-30T13:33:47.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.310+0000 7fdc267fc640 1 -- 172.21.15.46:0/1190073940 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdc2800abf0 con 0x7fdc301580e0 2023-10-30T13:33:47.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.314+0000 7fdc267fc640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdc140664a0 0x7fdc14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:47.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.314+0000 7fdc267fc640 1 -- 172.21.15.46:0/1190073940 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fdc28083f50 con 0x7fdc301580e0 2023-10-30T13:33:47.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.314+0000 7fdc35de2640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdc140664a0 0x7fdc14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:47.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.314+0000 7fdc35de2640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdc140664a0 0x7fdc14068960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009ac0 tx=0x7fdc2c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:47.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.318+0000 7fdc267fc640 1 -- 172.21.15.46:0/1190073940 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdc300c6000 con 0x7fdc301580e0 2023-10-30T13:33:47.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.506+0000 7fdc38046640 1 -- 172.21.15.46:0/1190073940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc300c6000 con 0x7fdc301580e0 2023-10-30T13:33:47.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc267fc640 1 -- 172.21.15.46:0/1190073940 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fdc300c6000 con 0x7fdc301580e0 2023-10-30T13:33:47.513 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:47.514 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:43.868255+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 55 2023-10-30T13:33:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc38046640 1 -- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdc140664a0 msgr2=0x7fdc14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc38046640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdc140664a0 0x7fdc14068960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009ac0 tx=0x7fdc2c009190 comp rx=0 tx=0).stop 2023-10-30T13:33:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc38046640 1 -- 172.21.15.46:0/1190073940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc301580e0 msgr2=0x7fdc3014b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc38046640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc301580e0 0x7fdc3014b210 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800c7a0 tx=0x7fdc2800cc60 comp rx=0 tx=0).stop 2023-10-30T13:33:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc38046640 1 -- 172.21.15.46:0/1190073940 shutdown_connections 2023-10-30T13:33:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc38046640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdc140664a0 0x7fdc14068960 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc38046640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc301580e0 0x7fdc3014b210 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc38046640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc300c6540 0x7fdc3014acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc38046640 1 --2- 172.21.15.46:0/1190073940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc30069e90 0x7fdc300c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc38046640 1 -- 172.21.15.46:0/1190073940 >> 172.21.15.46:0/1190073940 conn(0x7fdc300c7910 msgr2=0x7fdc30155910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc38046640 1 -- 172.21.15.46:0/1190073940 shutdown_connections 2023-10-30T13:33:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:47.510+0000 7fdc38046640 1 -- 172.21.15.46:0/1190073940 wait complete. 2023-10-30T13:33:47.535 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:33:49.537 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:49.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f4386d3e640 1 Processor -- start 2023-10-30T13:33:49.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f4386d3e640 1 -- start start 2023-10-30T13:33:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f4386d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43801538a0 0x7f438015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f4386d3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f43800cf160 0x7f43800cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f4386d3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43800cb800 con 0x7f43800ca0e0 2023-10-30T13:33:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f4386d3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438015dc30 con 0x7f43800cf160 2023-10-30T13:33:49.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f4386d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43800cb330 con 0x7f43801538a0 2023-10-30T13:33:49.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f437ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f43800ca0e0 0x7f438015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48292/0 (socket says 172.21.15.46:48292) 2023-10-30T13:33:49.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f437ffff640 1 -- 172.21.15.46:0/917212376 learned_addr learned my addr 172.21.15.46:0/917212376 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:33:49.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f43852db640 1 --2- 172.21.15.46:0/917212376 >> v2:172.21.15.154:3300/0 conn(0x7f43800cf160 0x7f43800cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:49.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f4384ada640 1 --2- 172.21.15.46:0/917212376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43801538a0 0x7f438015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:49.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f437f7fe640 1 -- 172.21.15.46:0/917212376 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601740058 0 0) 0x7f43800cb800 con 0x7f43800ca0e0 2023-10-30T13:33:49.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f437f7fe640 1 -- 172.21.15.46:0/917212376 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4370003530 con 0x7f43800ca0e0 2023-10-30T13:33:49.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f43852db640 1 -- 172.21.15.46:0/917212376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43801538a0 msgr2=0x7f438015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:49.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f43852db640 1 --2- 172.21.15.46:0/917212376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43801538a0 0x7f438015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:49.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f437f7fe640 1 -- 172.21.15.46:0/917212376 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 603829463 0 0) 0x7f4370003530 con 0x7f43800ca0e0 2023-10-30T13:33:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f43852db640 1 -- 172.21.15.46:0/917212376 >> v1:172.21.15.46:6789/0 conn(0x7f43800ca0e0 legacy=0x7f438015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.678+0000 7f43852db640 1 -- 172.21.15.46:0/917212376 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43800cad70 con 0x7f43800cf160 2023-10-30T13:33:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.682+0000 7f43852db640 1 --2- 172.21.15.46:0/917212376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43800cf160 0x7f43800cf590 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4378009d90 tx=0x7f437801f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8c733d99920b838 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.682+0000 7f437f7fe640 1 -- 172.21.15.46:0/917212376 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436c001680 con 0x7f43800cf160 2023-10-30T13:33:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.682+0000 7f437f7fe640 1 -- 172.21.15.46:0/917212376 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43780a8bb0 con 0x7f43800cf160 2023-10-30T13:33:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.682+0000 7f437f7fe640 1 -- 172.21.15.46:0/917212376 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43780b0440 con 0x7f43800cf160 2023-10-30T13:33:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.682+0000 7f4386d3e640 1 -- 172.21.15.46:0/917212376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43800cf160 msgr2=0x7f43800cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.682+0000 7f4386d3e640 1 --2- 172.21.15.46:0/917212376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43800cf160 0x7f43800cf590 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4378009d90 tx=0x7f437801f060 comp rx=0 tx=0).stop 2023-10-30T13:33:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.682+0000 7f4386d3e640 1 -- 172.21.15.46:0/917212376 shutdown_connections 2023-10-30T13:33:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.682+0000 7f4386d3e640 1 --2- 172.21.15.46:0/917212376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43800cf160 0x7f43800cf590 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:49.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.682+0000 7f4386d3e640 1 --2- 172.21.15.46:0/917212376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43801538a0 0x7f438015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:49.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.682+0000 7f4386d3e640 1 -- 172.21.15.46:0/917212376 >> 172.21.15.46:0/917212376 conn(0x7f438006bfe0 msgr2=0x7f438006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:49.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.682+0000 7f4386d3e640 1 -- 172.21.15.46:0/917212376 shutdown_connections 2023-10-30T13:33:49.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.682+0000 7f4386d3e640 1 -- 172.21.15.46:0/917212376 wait complete. 2023-10-30T13:33:49.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4386d3e640 1 Processor -- start 2023-10-30T13:33:49.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4386d3e640 1 -- start start 2023-10-30T13:33:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4386d3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43800ca0e0 0x7f438015b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4386d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43800cf160 0x7f438015bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4386d3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43801538a0 0x7f438014db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4386d3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438015a8b0 con 0x7f43800ca0e0 2023-10-30T13:33:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4386d3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43801505d0 con 0x7f43801538a0 2023-10-30T13:33:49.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4386d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438015c1a0 con 0x7f43800cf160 2023-10-30T13:33:49.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4384ada640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43800ca0e0 0x7f438015b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:49.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4384ada640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43800ca0e0 0x7f438015b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55346/0 (socket says 172.21.15.46:55346) 2023-10-30T13:33:49.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4384ada640 1 -- 172.21.15.46:0/1904531718 learned_addr learned my addr 172.21.15.46:0/1904531718 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:49.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f43852db640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43801538a0 0x7f438014db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:49.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f437ffff640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43800cf160 0x7f438015bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:49.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4384ada640 1 -- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43800cf160 msgr2=0x7f438015bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:49.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4384ada640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43800cf160 0x7f438015bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:49.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4384ada640 1 -- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43801538a0 msgr2=0x7f438014db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:49.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4384ada640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43801538a0 0x7f438014db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4384ada640 1 -- 172.21.15.46:0/1904531718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4378009a70 con 0x7f43800ca0e0 2023-10-30T13:33:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4384ada640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43800ca0e0 0x7f438015b5e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f437400e870 tx=0x7f437400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f437d7fa640 1 -- 172.21.15.46:0/1904531718 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374009510 con 0x7f43800ca0e0 2023-10-30T13:33:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f437d7fa640 1 -- 172.21.15.46:0/1904531718 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437400eec0 con 0x7f43800ca0e0 2023-10-30T13:33:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f437d7fa640 1 -- 172.21.15.46:0/1904531718 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437400cb40 con 0x7f43800ca0e0 2023-10-30T13:33:49.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4386d3e640 1 -- 172.21.15.46:0/1904531718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438015c480 con 0x7f43800ca0e0 2023-10-30T13:33:49.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.686+0000 7f4386d3e640 1 -- 172.21.15.46:0/1904531718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438015ade0 con 0x7f43800ca0e0 2023-10-30T13:33:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.690+0000 7f4386d3e640 1 -- 172.21.15.46:0/1904531718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43800b2a50 con 0x7f43800ca0e0 2023-10-30T13:33:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.690+0000 7f437d7fa640 1 -- 172.21.15.46:0/1904531718 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f437400cca0 con 0x7f43800ca0e0 2023-10-30T13:33:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.690+0000 7f437d7fa640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43640664a0 0x7f4364068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.690+0000 7f437d7fa640 1 -- 172.21.15.46:0/1904531718 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f4374084380 con 0x7f43800ca0e0 2023-10-30T13:33:49.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.690+0000 7f437ffff640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43640664a0 0x7f4364068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:49.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.690+0000 7f437ffff640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43640664a0 0x7f4364068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4380151d80 tx=0x7f436c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:49.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.694+0000 7f437d7fa640 1 -- 172.21.15.46:0/1904531718 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4374051430 con 0x7f43800ca0e0 2023-10-30T13:33:49.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.882+0000 7f4386d3e640 1 -- 172.21.15.46:0/1904531718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f438004e170 con 0x7f43800ca0e0 2023-10-30T13:33:49.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.886+0000 7f437d7fa640 1 -- 172.21.15.46:0/1904531718 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f4374054a50 con 0x7f43800ca0e0 2023-10-30T13:33:49.891 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:49.891 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:49.288172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:resolve","state_seq":30,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:49.891 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 56 2023-10-30T13:33:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.886+0000 7f4386d3e640 1 -- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43640664a0 msgr2=0x7f4364068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.886+0000 7f4386d3e640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43640664a0 0x7f4364068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4380151d80 tx=0x7f436c009040 comp rx=0 tx=0).stop 2023-10-30T13:33:49.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.886+0000 7f4386d3e640 1 -- 172.21.15.46:0/1904531718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43800ca0e0 msgr2=0x7f438015b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:49.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.886+0000 7f4386d3e640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43800ca0e0 0x7f438015b5e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f437400e870 tx=0x7f437400ec70 comp rx=0 tx=0).stop 2023-10-30T13:33:49.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.890+0000 7f4386d3e640 1 -- 172.21.15.46:0/1904531718 shutdown_connections 2023-10-30T13:33:49.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.890+0000 7f4386d3e640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43640664a0 0x7f4364068960 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.890+0000 7f4386d3e640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43801538a0 0x7f438014db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.890+0000 7f4386d3e640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43800cf160 0x7f438015bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.890+0000 7f4386d3e640 1 --2- 172.21.15.46:0/1904531718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43800ca0e0 0x7f438015b5e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.890+0000 7f4386d3e640 1 -- 172.21.15.46:0/1904531718 >> 172.21.15.46:0/1904531718 conn(0x7f438006bfe0 msgr2=0x7f4380065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.890+0000 7f4386d3e640 1 -- 172.21.15.46:0/1904531718 shutdown_connections 2023-10-30T13:33:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:49.890+0000 7f4386d3e640 1 -- 172.21.15.46:0/1904531718 wait complete. 2023-10-30T13:33:49.912 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:33:51.913 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:52.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25d4d80640 1 Processor -- start 2023-10-30T13:33:52.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25d4d80640 1 -- start start 2023-10-30T13:33:52.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25d4d80640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25d015d6f0 0x7f25d015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:52.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25d4d80640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25d00cf160 0x7f25d00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:52.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25d4d80640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d00cb800 con 0x7f25d01538a0 2023-10-30T13:33:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25d4d80640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d00cb330 con 0x7f25d00cf160 2023-10-30T13:33:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25d4d80640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00cad70 con 0x7f25d015d6f0 2023-10-30T13:33:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25ce59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f25d01538a0 0x7f25d00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48298/0 (socket says 172.21.15.46:48298) 2023-10-30T13:33:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25ce59c640 1 -- 172.21.15.46:0/3839072678 learned_addr learned my addr 172.21.15.46:0/3839072678 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:33:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25ced9d640 1 --2- 172.21.15.46:0/3839072678 >> v2:172.21.15.46:3300/0 conn(0x7f25d00cf160 0x7f25d00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:52.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25c7fff640 1 --2- 172.21.15.46:0/3839072678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25d015d6f0 0x7f25d015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:52.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25cdd9b640 1 -- 172.21.15.46:0/3839072678 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265826715 0 0) 0x7f25d00cb800 con 0x7f25d01538a0 2023-10-30T13:33:52.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25cdd9b640 1 -- 172.21.15.46:0/3839072678 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25bc003530 con 0x7f25d01538a0 2023-10-30T13:33:52.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25ced9d640 1 -- 172.21.15.46:0/3839072678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25d015d6f0 msgr2=0x7f25d015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:52.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25ced9d640 1 --2- 172.21.15.46:0/3839072678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25d015d6f0 0x7f25d015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25ced9d640 1 -- 172.21.15.46:0/3839072678 >> v1:172.21.15.46:6789/0 conn(0x7f25d01538a0 legacy=0x7f25d00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:52.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.054+0000 7f25ced9d640 1 -- 172.21.15.46:0/3839072678 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d00cb050 con 0x7f25d00cf160 2023-10-30T13:33:52.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.058+0000 7f25ced9d640 1 --2- 172.21.15.46:0/3839072678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d00cf160 0x7f25d00cf590 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f25c8009d90 tx=0x7f25c801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b44812d8fe955ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:52.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.058+0000 7f25cdd9b640 1 -- 172.21.15.46:0/3839072678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c80af600 con 0x7f25d00cf160 2023-10-30T13:33:52.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.058+0000 7f25cdd9b640 1 -- 172.21.15.46:0/3839072678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c80ab050 con 0x7f25d00cf160 2023-10-30T13:33:52.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.058+0000 7f25cdd9b640 1 -- 172.21.15.46:0/3839072678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c80af980 con 0x7f25d00cf160 2023-10-30T13:33:52.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.058+0000 7f25d4d80640 1 -- 172.21.15.46:0/3839072678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d00cf160 msgr2=0x7f25d00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:52.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.058+0000 7f25d4d80640 1 --2- 172.21.15.46:0/3839072678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d00cf160 0x7f25d00cf590 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f25c8009d90 tx=0x7f25c801f130 comp rx=0 tx=0).stop 2023-10-30T13:33:52.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.058+0000 7f25d4d80640 1 -- 172.21.15.46:0/3839072678 shutdown_connections 2023-10-30T13:33:52.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.058+0000 7f25d4d80640 1 --2- 172.21.15.46:0/3839072678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d00cf160 0x7f25d00cf590 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.058+0000 7f25d4d80640 1 --2- 172.21.15.46:0/3839072678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25d015d6f0 0x7f25d015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.058+0000 7f25d4d80640 1 -- 172.21.15.46:0/3839072678 >> 172.21.15.46:0/3839072678 conn(0x7f25d006bfe0 msgr2=0x7f25d006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:52.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.058+0000 7f25d4d80640 1 -- 172.21.15.46:0/3839072678 shutdown_connections 2023-10-30T13:33:52.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.058+0000 7f25d4d80640 1 -- 172.21.15.46:0/3839072678 wait complete. 2023-10-30T13:33:52.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25d4d80640 1 Processor -- start 2023-10-30T13:33:52.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25d4d80640 1 -- start start 2023-10-30T13:33:52.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25d4d80640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25d00cf160 0x7f25d015c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:52.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25d4d80640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d01538a0 0x7f25d00cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:52.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25d4d80640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25d015d6f0 0x7f25d00cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:52.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25d4d80640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00ccfb0 con 0x7f25d01538a0 2023-10-30T13:33:52.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25d4d80640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00b36b0 con 0x7f25d015d6f0 2023-10-30T13:33:52.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25d4d80640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00b4760 con 0x7f25d00cf160 2023-10-30T13:33:52.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25ce59c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25d00cf160 0x7f25d015c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:52.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d01538a0 0x7f25d00cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:52.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25ce59c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25d00cf160 0x7f25d015c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:37352/0 (socket says 172.21.15.46:37352) 2023-10-30T13:33:52.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25ce59c640 1 -- 172.21.15.46:0/1124257270 learned_addr learned my addr 172.21.15.46:0/1124257270 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:52.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25ced9d640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25d015d6f0 0x7f25d00cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:52.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25ce59c640 1 -- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25d015d6f0 msgr2=0x7f25d00cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:52.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25ce59c640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25d015d6f0 0x7f25d00cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25ce59c640 1 -- 172.21.15.46:0/1124257270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d01538a0 msgr2=0x7f25d00cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:52.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25ce59c640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d01538a0 0x7f25d00cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25ce59c640 1 -- 172.21.15.46:0/1124257270 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c8009a70 con 0x7f25d00cf160 2023-10-30T13:33:52.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25c7fff640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d01538a0 0x7f25d00cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:33:52.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25ce59c640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25d00cf160 0x7f25d015c150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f25c000c500 tx=0x7f25c000c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:52.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25c6ffd640 1 -- 172.21.15.46:0/1124257270 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0005cf0 con 0x7f25d00cf160 2023-10-30T13:33:52.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25c6ffd640 1 -- 172.21.15.46:0/1124257270 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c0005e50 con 0x7f25d00cf160 2023-10-30T13:33:52.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25c6ffd640 1 -- 172.21.15.46:0/1124257270 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c00072f0 con 0x7f25d00cf160 2023-10-30T13:33:52.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25d4d80640 1 -- 172.21.15.46:0/1124257270 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d00cea40 con 0x7f25d00cf160 2023-10-30T13:33:52.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.062+0000 7f25d4d80640 1 -- 172.21.15.46:0/1124257270 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d00b3340 con 0x7f25d00cf160 2023-10-30T13:33:52.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.066+0000 7f25d4d80640 1 -- 172.21.15.46:0/1124257270 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d00b2a50 con 0x7f25d00cf160 2023-10-30T13:33:52.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.066+0000 7f25c6ffd640 1 -- 172.21.15.46:0/1124257270 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25c0007450 con 0x7f25d00cf160 2023-10-30T13:33:52.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.066+0000 7f25c6ffd640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25b00664d0 0x7f25b0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:52.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.066+0000 7f25c6ffd640 1 -- 172.21.15.46:0/1124257270 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f25c0083660 con 0x7f25d00cf160 2023-10-30T13:33:52.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.066+0000 7f25c7fff640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25b00664d0 0x7f25b0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:52.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.066+0000 7f25c7fff640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25b00664d0 0x7f25b0068990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f25d0152f00 tx=0x7f25b8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:52.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.070+0000 7f25c6ffd640 1 -- 172.21.15.46:0/1124257270 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25c00508e0 con 0x7f25d00cf160 2023-10-30T13:33:52.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.262+0000 7f25d4d80640 1 -- 172.21.15.46:0/1124257270 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25d004e170 con 0x7f25d00cf160 2023-10-30T13:33:52.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.262+0000 7f25c6ffd640 1 -- 172.21.15.46:0/1124257270 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f25c0053f00 con 0x7f25d00cf160 2023-10-30T13:33:52.268 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:52.268 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:49.288172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:resolve","state_seq":30,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:52.268 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 56 2023-10-30T13:33:52.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.266+0000 7f25d4d80640 1 -- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25b00664d0 msgr2=0x7f25b0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:52.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.266+0000 7f25d4d80640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25b00664d0 0x7f25b0068990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f25d0152f00 tx=0x7f25b8009190 comp rx=0 tx=0).stop 2023-10-30T13:33:52.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.266+0000 7f25d4d80640 1 -- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25d00cf160 msgr2=0x7f25d015c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:52.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.266+0000 7f25d4d80640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25d00cf160 0x7f25d015c150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f25c000c500 tx=0x7f25c000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.266+0000 7f25d4d80640 1 -- 172.21.15.46:0/1124257270 shutdown_connections 2023-10-30T13:33:52.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.266+0000 7f25d4d80640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25b00664d0 0x7f25b0068990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.266+0000 7f25d4d80640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25d015d6f0 0x7f25d00cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.266+0000 7f25d4d80640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d01538a0 0x7f25d00cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.266+0000 7f25d4d80640 1 --2- 172.21.15.46:0/1124257270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25d00cf160 0x7f25d015c150 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.266+0000 7f25d4d80640 1 -- 172.21.15.46:0/1124257270 >> 172.21.15.46:0/1124257270 conn(0x7f25d006bfe0 msgr2=0x7f25d0151d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:52.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.266+0000 7f25d4d80640 1 -- 172.21.15.46:0/1124257270 shutdown_connections 2023-10-30T13:33:52.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:52.266+0000 7f25d4d80640 1 -- 172.21.15.46:0/1124257270 wait complete. 2023-10-30T13:33:52.288 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:33:54.289 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30cd29c640 1 Processor -- start 2023-10-30T13:33:54.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30cd29c640 1 -- start start 2023-10-30T13:33:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30cd29c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30c81538a0 0x7f30c815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30cd29c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f30c80cf160 0x7f30c80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30cd29c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c80cb800 con 0x7f30c80ca0e0 2023-10-30T13:33:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30cd29c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c815dc30 con 0x7f30c81538a0 2023-10-30T13:33:54.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30cd29c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c80cb330 con 0x7f30c80cf160 2023-10-30T13:33:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30c6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30c81538a0 0x7f30c815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30bffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f30c80ca0e0 0x7f30c815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48310/0 (socket says 172.21.15.46:48310) 2023-10-30T13:33:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30c6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30c81538a0 0x7f30c815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55364/0 (socket says 172.21.15.46:55364) 2023-10-30T13:33:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30bffff640 1 -- 172.21.15.46:0/2062695629 learned_addr learned my addr 172.21.15.46:0/2062695629 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:33:54.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30c77fe640 1 --2- 172.21.15.46:0/2062695629 >> v2:172.21.15.154:3300/0 conn(0x7f30c80cf160 0x7f30c80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:54.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30c6ffd640 1 -- 172.21.15.46:0/2062695629 >> v2:172.21.15.154:3300/0 conn(0x7f30c80cf160 msgr2=0x7f30c80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:54.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30c6ffd640 1 --2- 172.21.15.46:0/2062695629 >> v2:172.21.15.154:3300/0 conn(0x7f30c80cf160 0x7f30c80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:54.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30c6ffd640 1 -- 172.21.15.46:0/2062695629 >> v1:172.21.15.46:6789/0 conn(0x7f30c80ca0e0 legacy=0x7f30c815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:54.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.442+0000 7f30c6ffd640 1 -- 172.21.15.46:0/2062695629 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c80cad70 con 0x7f30c81538a0 2023-10-30T13:33:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30c6ffd640 1 --2- 172.21.15.46:0/2062695629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30c81538a0 0x7f30c815d6f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f30c80b5330 tx=0x7f30b801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f5a5b02ccdac4c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30c67fc640 1 -- 172.21.15.46:0/2062695629 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b80a8950 con 0x7f30c81538a0 2023-10-30T13:33:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30c67fc640 1 -- 172.21.15.46:0/2062695629 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b80a8ab0 con 0x7f30c81538a0 2023-10-30T13:33:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30c67fc640 1 -- 172.21.15.46:0/2062695629 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b80a8e40 con 0x7f30c81538a0 2023-10-30T13:33:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30cd29c640 1 -- 172.21.15.46:0/2062695629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30c81538a0 msgr2=0x7f30c815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30cd29c640 1 --2- 172.21.15.46:0/2062695629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30c81538a0 0x7f30c815d6f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f30c80b5330 tx=0x7f30b801ec50 comp rx=0 tx=0).stop 2023-10-30T13:33:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30cd29c640 1 -- 172.21.15.46:0/2062695629 shutdown_connections 2023-10-30T13:33:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30cd29c640 1 --2- 172.21.15.46:0/2062695629 >> v2:172.21.15.154:3300/0 conn(0x7f30c80cf160 0x7f30c80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30cd29c640 1 --2- 172.21.15.46:0/2062695629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30c81538a0 0x7f30c815d6f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30cd29c640 1 -- 172.21.15.46:0/2062695629 >> 172.21.15.46:0/2062695629 conn(0x7f30c806bfe0 msgr2=0x7f30c806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30cd29c640 1 -- 172.21.15.46:0/2062695629 shutdown_connections 2023-10-30T13:33:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30cd29c640 1 -- 172.21.15.46:0/2062695629 wait complete. 2023-10-30T13:33:54.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30cd29c640 1 Processor -- start 2023-10-30T13:33:54.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.446+0000 7f30cd29c640 1 -- start start 2023-10-30T13:33:54.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30cd29c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30c80ca0e0 0x7f30c8158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:54.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30cd29c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30c80cf160 0x7f30c81e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30cd29c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30c81538a0 0x7f30c81e4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30cd29c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c815d9f0 con 0x7f30c81538a0 2023-10-30T13:33:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30cd29c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c815d710 con 0x7f30c80cf160 2023-10-30T13:33:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30cd29c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c81e3600 con 0x7f30c80ca0e0 2023-10-30T13:33:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30c81538a0 0x7f30c81e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:54.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30c81538a0 0x7f30c81e4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55372/0 (socket says 172.21.15.46:55372) 2023-10-30T13:33:54.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30c77fe640 1 -- 172.21.15.46:0/3784633306 learned_addr learned my addr 172.21.15.46:0/3784633306 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:54.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30bffff640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30c80cf160 0x7f30c81e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:54.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30c6ffd640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30c80ca0e0 0x7f30c8158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:54.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30c77fe640 1 -- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30c80ca0e0 msgr2=0x7f30c8158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:54.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30c77fe640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30c80ca0e0 0x7f30c8158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:54.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30c77fe640 1 -- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30c80cf160 msgr2=0x7f30c81e40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:54.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30c77fe640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30c80cf160 0x7f30c81e40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:54.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30c77fe640 1 -- 172.21.15.46:0/3784633306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b8009420 con 0x7f30c81538a0 2023-10-30T13:33:54.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30c77fe640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30c81538a0 0x7f30c81e4620 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f30c0007d90 tx=0x7f30c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:54.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30bf7fe640 1 -- 172.21.15.46:0/3784633306 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c0014420 con 0x7f30c81538a0 2023-10-30T13:33:54.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30bf7fe640 1 -- 172.21.15.46:0/3784633306 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c00145a0 con 0x7f30c81538a0 2023-10-30T13:33:54.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30cd29c640 1 -- 172.21.15.46:0/3784633306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c81e38e0 con 0x7f30c81538a0 2023-10-30T13:33:54.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30bf7fe640 1 -- 172.21.15.46:0/3784633306 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c0014950 con 0x7f30c81538a0 2023-10-30T13:33:54.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30cd29c640 1 -- 172.21.15.46:0/3784633306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c81595d0 con 0x7f30c81538a0 2023-10-30T13:33:54.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.450+0000 7f30cd29c640 1 -- 172.21.15.46:0/3784633306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c80b2a50 con 0x7f30c81538a0 2023-10-30T13:33:54.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.454+0000 7f30bf7fe640 1 -- 172.21.15.46:0/3784633306 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f30c0014ab0 con 0x7f30c81538a0 2023-10-30T13:33:54.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.454+0000 7f30bf7fe640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30a80664a0 0x7f30a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:54.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.454+0000 7f30bf7fe640 1 -- 172.21.15.46:0/3784633306 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f30c0084920 con 0x7f30c81538a0 2023-10-30T13:33:54.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.454+0000 7f30c6ffd640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30a80664a0 0x7f30a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:54.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.454+0000 7f30c6ffd640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30a80664a0 0x7f30a8068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f30b80b2000 tx=0x7f30b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:54.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.458+0000 7f30bf7fe640 1 -- 172.21.15.46:0/3784633306 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f30c0051bb0 con 0x7f30c81538a0 2023-10-30T13:33:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.646+0000 7f30cd29c640 1 -- 172.21.15.46:0/3784633306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30c81e3e00 con 0x7f30c81538a0 2023-10-30T13:33:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30bf7fe640 1 -- 172.21.15.46:0/3784633306 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f30c81e3e00 con 0x7f30c81538a0 2023-10-30T13:33:54.654 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:54.654 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:49.288172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:resolve","state_seq":30,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:54.654 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 56 2023-10-30T13:33:54.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30cd29c640 1 -- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30a80664a0 msgr2=0x7f30a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30cd29c640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30a80664a0 0x7f30a8068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f30b80b2000 tx=0x7f30b8008040 comp rx=0 tx=0).stop 2023-10-30T13:33:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30cd29c640 1 -- 172.21.15.46:0/3784633306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30c81538a0 msgr2=0x7f30c81e4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30cd29c640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30c81538a0 0x7f30c81e4620 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f30c0007d90 tx=0x7f30c000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:33:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30cd29c640 1 -- 172.21.15.46:0/3784633306 shutdown_connections 2023-10-30T13:33:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30cd29c640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30a80664a0 0x7f30a8068960 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30cd29c640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30c81538a0 0x7f30c81e4620 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30cd29c640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30c80cf160 0x7f30c81e40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30cd29c640 1 --2- 172.21.15.46:0/3784633306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30c80ca0e0 0x7f30c8158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30cd29c640 1 -- 172.21.15.46:0/3784633306 >> 172.21.15.46:0/3784633306 conn(0x7f30c806bfe0 msgr2=0x7f30c8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30cd29c640 1 -- 172.21.15.46:0/3784633306 shutdown_connections 2023-10-30T13:33:54.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:54.650+0000 7f30cd29c640 1 -- 172.21.15.46:0/3784633306 wait complete. 2023-10-30T13:33:54.675 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:33:56.676 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.826+0000 7f6623c46640 1 Processor -- start 2023-10-30T13:33:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.826+0000 7f6623c46640 1 -- start start 2023-10-30T13:33:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f6623c46640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f661c0c05d0 0x7f661c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:56.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f6623c46640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f661c0c6f60 0x7f661c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f6623c46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c0c0a50 con 0x7f661c0c6b80 2023-10-30T13:33:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f6623c46640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661c06f6e0 con 0x7f661c0c05d0 2023-10-30T13:33:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f6623c46640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661c06f9c0 con 0x7f661c0c6f60 2023-10-30T13:33:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f66219e2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f661c0c6b80 0x7f661c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48804/0 (socket says 172.21.15.46:48804) 2023-10-30T13:33:56.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f66219e2640 1 -- 172.21.15.46:0/4190685049 learned_addr learned my addr 172.21.15.46:0/4190685049 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:33:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f66211e1640 1 --2- 172.21.15.46:0/4190685049 >> v2:172.21.15.46:3300/0 conn(0x7f661c0c05d0 0x7f661c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f66221e3640 1 --2- 172.21.15.46:0/4190685049 >> v2:172.21.15.154:3300/0 conn(0x7f661c0c6f60 0x7f661c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f66209e0640 1 -- 172.21.15.46:0/4190685049 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404794221 0 0) 0x7f661c0c0a50 con 0x7f661c0c6b80 2023-10-30T13:33:56.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f66209e0640 1 -- 172.21.15.46:0/4190685049 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6608003530 con 0x7f661c0c6b80 2023-10-30T13:33:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f66211e1640 1 -- 172.21.15.46:0/4190685049 >> v2:172.21.15.154:3300/0 conn(0x7f661c0c6f60 msgr2=0x7f661c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f66211e1640 1 --2- 172.21.15.46:0/4190685049 >> v2:172.21.15.154:3300/0 conn(0x7f661c0c6f60 0x7f661c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f66209e0640 1 -- 172.21.15.46:0/4190685049 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1239596312 0 0) 0x7f6608003530 con 0x7f661c0c6b80 2023-10-30T13:33:56.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f66211e1640 1 -- 172.21.15.46:0/4190685049 >> v1:172.21.15.46:6789/0 conn(0x7f661c0c6b80 legacy=0x7f661c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:56.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f66211e1640 1 -- 172.21.15.46:0/4190685049 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c06fca0 con 0x7f661c0c05d0 2023-10-30T13:33:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.830+0000 7f66211e1640 1 --2- 172.21.15.46:0/4190685049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0c05d0 0x7f661c13ed90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f661c0c89b0 tx=0x7f660c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c079a18a60d5a5cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f66209e0640 1 -- 172.21.15.46:0/4190685049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c0a4ea0 con 0x7f661c0c05d0 2023-10-30T13:33:56.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f66209e0640 1 -- 172.21.15.46:0/4190685049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c0a7050 con 0x7f661c0c05d0 2023-10-30T13:33:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f66209e0640 1 -- 172.21.15.46:0/4190685049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6618001680 con 0x7f661c0c05d0 2023-10-30T13:33:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 -- 172.21.15.46:0/4190685049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0c05d0 msgr2=0x7f661c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 --2- 172.21.15.46:0/4190685049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0c05d0 0x7f661c13ed90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f661c0c89b0 tx=0x7f660c01c380 comp rx=0 tx=0).stop 2023-10-30T13:33:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 -- 172.21.15.46:0/4190685049 shutdown_connections 2023-10-30T13:33:56.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 --2- 172.21.15.46:0/4190685049 >> v2:172.21.15.154:3300/0 conn(0x7f661c0c6f60 0x7f661c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 --2- 172.21.15.46:0/4190685049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0c05d0 0x7f661c13ed90 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 -- 172.21.15.46:0/4190685049 >> 172.21.15.46:0/4190685049 conn(0x7f661c0c7910 msgr2=0x7f661c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 -- 172.21.15.46:0/4190685049 shutdown_connections 2023-10-30T13:33:56.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 -- 172.21.15.46:0/4190685049 wait complete. 2023-10-30T13:33:56.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 Processor -- start 2023-10-30T13:33:56.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 -- start start 2023-10-30T13:33:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661c0c05d0 0x7f661c0c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f661c0c6b80 0x7f661c12e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0c6f60 0x7f661c0c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c150470 con 0x7f661c0c6f60 2023-10-30T13:33:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c152020 con 0x7f661c0c6b80 2023-10-30T13:33:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.834+0000 7f6623c46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c14fde0 con 0x7f661c0c05d0 2023-10-30T13:33:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66221e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0c6f60 0x7f661c0c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66221e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0c6f60 0x7f661c0c94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52272/0 (socket says 172.21.15.46:52272) 2023-10-30T13:33:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66221e3640 1 -- 172.21.15.46:0/3532894755 learned_addr learned my addr 172.21.15.46:0/3532894755 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66211e1640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f661c0c6b80 0x7f661c12e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66219e2640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661c0c05d0 0x7f661c0c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:56.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66221e3640 1 -- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661c0c05d0 msgr2=0x7f661c0c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:56.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66221e3640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661c0c05d0 0x7f661c0c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:56.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66221e3640 1 -- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f661c0c6b80 msgr2=0x7f661c12e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:56.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66221e3640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f661c0c6b80 0x7f661c12e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:56.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66221e3640 1 -- 172.21.15.46:0/3532894755 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c009420 con 0x7f661c0c6f60 2023-10-30T13:33:56.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66221e3640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0c6f60 0x7f661c0c94b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f661400efc0 tx=0x7f661400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:56.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66127fc640 1 -- 172.21.15.46:0/3532894755 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614009ba0 con 0x7f661c0c6f60 2023-10-30T13:33:56.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66127fc640 1 -- 172.21.15.46:0/3532894755 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661400f040 con 0x7f661c0c6f60 2023-10-30T13:33:56.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66127fc640 1 -- 172.21.15.46:0/3532894755 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66140146c0 con 0x7f661c0c6f60 2023-10-30T13:33:56.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f6623c46640 1 -- 172.21.15.46:0/3532894755 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c0c0340 con 0x7f661c0c6f60 2023-10-30T13:33:56.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f6623c46640 1 -- 172.21.15.46:0/3532894755 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c1522c0 con 0x7f661c0c6f60 2023-10-30T13:33:56.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f6623c46640 1 -- 172.21.15.46:0/3532894755 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c151e80 con 0x7f661c0c6f60 2023-10-30T13:33:56.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66127fc640 1 -- 172.21.15.46:0/3532894755 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6614014820 con 0x7f661c0c6f60 2023-10-30T13:33:56.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.838+0000 7f66127fc640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66000664a0 0x7f6600068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:56.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.842+0000 7f66127fc640 1 -- 172.21.15.46:0/3532894755 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f66140849c0 con 0x7f661c0c6f60 2023-10-30T13:33:56.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.842+0000 7f66219e2640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66000664a0 0x7f6600068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:56.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.842+0000 7f66219e2640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66000664a0 0x7f6600068960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6618001190 tx=0x7f6618001120 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:56.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:56.846+0000 7f66127fc640 1 -- 172.21.15.46:0/3532894755 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f661c151e80 con 0x7f661c0c6f60 2023-10-30T13:33:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.034+0000 7f6623c46640 1 -- 172.21.15.46:0/3532894755 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f661c151e80 con 0x7f661c0c6f60 2023-10-30T13:33:57.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f66127fc640 1 -- 172.21.15.46:0/3532894755 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f661c151e80 con 0x7f661c0c6f60 2023-10-30T13:33:57.041 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:57.042 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:54.895187+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:57.042 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 57 2023-10-30T13:33:57.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f6623c46640 1 -- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66000664a0 msgr2=0x7f6600068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:57.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f6623c46640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66000664a0 0x7f6600068960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6618001190 tx=0x7f6618001120 comp rx=0 tx=0).stop 2023-10-30T13:33:57.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f6623c46640 1 -- 172.21.15.46:0/3532894755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0c6f60 msgr2=0x7f661c0c94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:57.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f6623c46640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0c6f60 0x7f661c0c94b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f661400efc0 tx=0x7f661400c490 comp rx=0 tx=0).stop 2023-10-30T13:33:57.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f6623c46640 1 -- 172.21.15.46:0/3532894755 shutdown_connections 2023-10-30T13:33:57.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f6623c46640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66000664a0 0x7f6600068960 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:57.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f6623c46640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0c6f60 0x7f661c0c94b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:57.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f6623c46640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f661c0c6b80 0x7f661c12e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:57.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f6623c46640 1 --2- 172.21.15.46:0/3532894755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661c0c05d0 0x7f661c0c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:57.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f6623c46640 1 -- 172.21.15.46:0/3532894755 >> 172.21.15.46:0/3532894755 conn(0x7f661c0c7910 msgr2=0x7f661c138de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:57.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f6623c46640 1 -- 172.21.15.46:0/3532894755 shutdown_connections 2023-10-30T13:33:57.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:57.038+0000 7f6623c46640 1 -- 172.21.15.46:0/3532894755 wait complete. 2023-10-30T13:33:57.063 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 57, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 57, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5266": { "addr": "172.21.15.154:6835/651441434", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 651441434, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 651441434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5266, "incarnation": 38, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 22 }, "gid_5442": { "addr": "172.21.15.46:6836/3955931693", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3955931693, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3955931693, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5442, "incarnation": 46, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 13 }, "gid_5574": { "addr": "172.21.15.46:6837/126953235", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 126953235, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 126953235, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5574, "incarnation": 55, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:reconnect", "state_seq": 32 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:33:54.895187+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5266, "mds_1": 5574, "mds_2": 5442 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:33:59.064 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.214+0000 7efe50646640 1 Processor -- start 2023-10-30T13:33:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.214+0000 7efe50646640 1 -- start start 2023-10-30T13:33:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.214+0000 7efe50646640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480c05d0 0x7efe4813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.214+0000 7efe50646640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe480c0a50 con 0x7efe480c6f60 2023-10-30T13:33:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.214+0000 7efe50646640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4806f6e0 con 0x7efe480c6b80 2023-10-30T13:33:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.214+0000 7efe50646640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4806f9c0 con 0x7efe480c05d0 2023-10-30T13:33:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.214+0000 7efe4ebe3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efe480c6f60 0x7efe481566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48818/0 (socket says 172.21.15.46:48818) 2023-10-30T13:33:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.214+0000 7efe4ebe3640 1 -- 172.21.15.46:0/2783129076 learned_addr learned my addr 172.21.15.46:0/2783129076 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:33:59.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.214+0000 7efe4dbe1640 1 --2- 172.21.15.46:0/2783129076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480c05d0 0x7efe4813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:59.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.214+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109582733 0 0) 0x7efe480c0a50 con 0x7efe480c6f60 2023-10-30T13:33:59.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe34003530 con 0x7efe480c6f60 2023-10-30T13:33:59.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091111033 0 0) 0x7efe4806f6e0 con 0x7efe480c6b80 2023-10-30T13:33:59.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe480c0a50 con 0x7efe480c6b80 2023-10-30T13:33:59.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3059107998 0 0) 0x7efe34003530 con 0x7efe480c6f60 2023-10-30T13:33:59.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480c05d0 msgr2=0x7efe4813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:59.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 --2- 172.21.15.46:0/2783129076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480c05d0 0x7efe4813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:59.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 >> v1:172.21.15.154:6789/0 conn(0x7efe480c6b80 legacy=0x7efe480bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:59.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4806fca0 con 0x7efe480c6f60 2023-10-30T13:33:59.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7efe40001a00 con 0x7efe480c6f60 2023-10-30T13:33:59.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 >> v1:172.21.15.46:6789/0 conn(0x7efe480c6f60 legacy=0x7efe481566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:59.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 --2- 172.21.15.46:0/2783129076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe34003b40 0x7efe34003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:59.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 --2- 172.21.15.46:0/2783129076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe340047d0 0x7efe34004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:59.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 --2- 172.21.15.46:0/2783129076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480c05d0 0x7efe34005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:59.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4806f9c0 con 0x7efe340047d0 2023-10-30T13:33:59.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3400dc50 con 0x7efe34003b40 2023-10-30T13:33:59.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3400de10 con 0x7efe480c05d0 2023-10-30T13:33:59.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4dbe1640 1 --2- 172.21.15.46:0/2783129076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe340047d0 0x7efe34004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:59.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4e3e2640 1 --2- 172.21.15.46:0/2783129076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe34003b40 0x7efe34003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4ebe3640 1 --2- 172.21.15.46:0/2783129076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480c05d0 0x7efe34005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4dbe1640 1 -- 172.21.15.46:0/2783129076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480c05d0 msgr2=0x7efe34005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4dbe1640 1 --2- 172.21.15.46:0/2783129076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480c05d0 0x7efe34005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4dbe1640 1 -- 172.21.15.46:0/2783129076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe34003b40 msgr2=0x7efe34003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4dbe1640 1 --2- 172.21.15.46:0/2783129076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe34003b40 0x7efe34003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.218+0000 7efe4dbe1640 1 -- 172.21.15.46:0/2783129076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe34003530 con 0x7efe340047d0 2023-10-30T13:33:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe4dbe1640 1 --2- 172.21.15.46:0/2783129076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe340047d0 0x7efe34004c50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7efe38009ab0 tx=0x7efe3801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8dde1c0fabec9a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe380a5600 con 0x7efe340047d0 2023-10-30T13:33:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe380a5760 con 0x7efe340047d0 2023-10-30T13:33:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe4d3e0640 1 -- 172.21.15.46:0/2783129076 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe380a5b10 con 0x7efe340047d0 2023-10-30T13:33:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe50646640 1 -- 172.21.15.46:0/2783129076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe340047d0 msgr2=0x7efe34004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe50646640 1 --2- 172.21.15.46:0/2783129076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe340047d0 0x7efe34004c50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7efe38009ab0 tx=0x7efe3801ca10 comp rx=0 tx=0).stop 2023-10-30T13:33:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe4ebe3640 1 -- 172.21.15.46:0/2783129076 reap_dead start 2023-10-30T13:33:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe50646640 1 -- 172.21.15.46:0/2783129076 shutdown_connections 2023-10-30T13:33:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe50646640 1 -- 172.21.15.46:0/2783129076 >> 172.21.15.46:0/2783129076 conn(0x7efe480c7910 msgr2=0x7efe480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe50646640 1 -- 172.21.15.46:0/2783129076 shutdown_connections 2023-10-30T13:33:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe50646640 1 -- 172.21.15.46:0/2783129076 wait complete. 2023-10-30T13:33:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe50646640 1 Processor -- start 2023-10-30T13:33:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.222+0000 7efe50646640 1 -- start start 2023-10-30T13:33:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe50646640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe340047d0 0x7efe480c5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe50646640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe481568e0 0x7efe481eaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe50646640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480ca500 0x7efe48153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:59.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe50646640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4813e500 con 0x7efe340047d0 2023-10-30T13:33:59.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe50646640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe481eaa60 con 0x7efe481568e0 2023-10-30T13:33:59.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe50646640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4813dc60 con 0x7efe480ca500 2023-10-30T13:33:59.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe4e3e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe340047d0 0x7efe480c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:59.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe4e3e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe340047d0 0x7efe480c5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52290/0 (socket says 172.21.15.46:52290) 2023-10-30T13:33:59.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe4e3e2640 1 -- 172.21.15.46:0/3538437133 learned_addr learned my addr 172.21.15.46:0/3538437133 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:33:59.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe4dbe1640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe481568e0 0x7efe481eaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:59.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe4ebe3640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480ca500 0x7efe48153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:59.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe4e3e2640 1 -- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480ca500 msgr2=0x7efe48153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:59.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe4e3e2640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480ca500 0x7efe48153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:59.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe4e3e2640 1 -- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe481568e0 msgr2=0x7efe481eaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:59.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe4e3e2640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe481568e0 0x7efe481eaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:59.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe4e3e2640 1 -- 172.21.15.46:0/3538437133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4813df40 con 0x7efe340047d0 2023-10-30T13:33:59.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe4e3e2640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe340047d0 0x7efe480c5190 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efe44005630 tx=0x7efe44012d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:59.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe3effd640 1 -- 172.21.15.46:0/3538437133 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe44010ba0 con 0x7efe340047d0 2023-10-30T13:33:59.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe3effd640 1 -- 172.21.15.46:0/3538437133 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe44010d40 con 0x7efe340047d0 2023-10-30T13:33:59.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe3effd640 1 -- 172.21.15.46:0/3538437133 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe440166c0 con 0x7efe340047d0 2023-10-30T13:33:59.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe50646640 1 -- 172.21.15.46:0/3538437133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe4813e220 con 0x7efe340047d0 2023-10-30T13:33:59.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe50646640 1 -- 172.21.15.46:0/3538437133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe481eb430 con 0x7efe340047d0 2023-10-30T13:33:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe50646640 1 -- 172.21.15.46:0/3538437133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe481ea550 con 0x7efe340047d0 2023-10-30T13:33:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.226+0000 7efe3effd640 1 -- 172.21.15.46:0/3538437133 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efe44016820 con 0x7efe340047d0 2023-10-30T13:33:59.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.230+0000 7efe3effd640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efe2c0664d0 0x7efe2c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:59.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.230+0000 7efe3effd640 1 -- 172.21.15.46:0/3538437133 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7efe44087990 con 0x7efe340047d0 2023-10-30T13:33:59.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.230+0000 7efe4dbe1640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efe2c0664d0 0x7efe2c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:59.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.230+0000 7efe4dbe1640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efe2c0664d0 0x7efe2c068990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efe38009ab0 tx=0x7efe380046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:59.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.234+0000 7efe3effd640 1 -- 172.21.15.46:0/3538437133 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efe481ea550 con 0x7efe340047d0 2023-10-30T13:33:59.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.422+0000 7efe50646640 1 -- 172.21.15.46:0/3538437133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe481ea550 con 0x7efe340047d0 2023-10-30T13:33:59.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.422+0000 7efe3effd640 1 -- 172.21.15.46:0/3538437133 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efe481ea550 con 0x7efe340047d0 2023-10-30T13:33:59.428 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:33:59.429 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:54.895187+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:59.430 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 57 2023-10-30T13:33:59.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.426+0000 7efe50646640 1 -- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efe2c0664d0 msgr2=0x7efe2c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:59.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.426+0000 7efe50646640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efe2c0664d0 0x7efe2c068990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efe38009ab0 tx=0x7efe380046d0 comp rx=0 tx=0).stop 2023-10-30T13:33:59.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.426+0000 7efe50646640 1 -- 172.21.15.46:0/3538437133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe340047d0 msgr2=0x7efe480c5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:59.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.426+0000 7efe50646640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe340047d0 0x7efe480c5190 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efe44005630 tx=0x7efe44012d00 comp rx=0 tx=0).stop 2023-10-30T13:33:59.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.426+0000 7efe50646640 1 -- 172.21.15.46:0/3538437133 shutdown_connections 2023-10-30T13:33:59.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.426+0000 7efe50646640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efe2c0664d0 0x7efe2c068990 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:59.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.426+0000 7efe50646640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe480ca500 0x7efe48153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:59.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.426+0000 7efe50646640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe481568e0 0x7efe481eaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:59.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.426+0000 7efe50646640 1 --2- 172.21.15.46:0/3538437133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe340047d0 0x7efe480c5190 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:59.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.426+0000 7efe50646640 1 -- 172.21.15.46:0/3538437133 >> 172.21.15.46:0/3538437133 conn(0x7efe480c7910 msgr2=0x7efe480c19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:59.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.426+0000 7efe50646640 1 -- 172.21.15.46:0/3538437133 shutdown_connections 2023-10-30T13:33:59.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:33:59.426+0000 7efe50646640 1 -- 172.21.15.46:0/3538437133 wait complete. 2023-10-30T13:33:59.450 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:34:01.451 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:01.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f328d46b640 1 Processor -- start 2023-10-30T13:34:01.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f328d46b640 1 -- start start 2023-10-30T13:34:01.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f328d46b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f328806f7a0 0x7f328806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:01.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f328d46b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3288066cf0 con 0x7f32880c6bd0 2023-10-30T13:34:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f328d46b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32880c70d0 con 0x7f3288071110 2023-10-30T13:34:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f328d46b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32880700c0 con 0x7f328806f7a0 2023-10-30T13:34:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f3286ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f328806f7a0 0x7f328806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f3286ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f328806f7a0 0x7f328806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52306/0 (socket says 172.21.15.46:52306) 2023-10-30T13:34:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f327ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f32880c6bd0 0x7f3288069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48830/0 (socket says 172.21.15.46:48830) 2023-10-30T13:34:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f3286ffd640 1 -- 172.21.15.46:0/1459457030 learned_addr learned my addr 172.21.15.46:0/1459457030 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f3286ffd640 1 -- 172.21.15.46:0/1459457030 >> v1:172.21.15.154:6789/0 conn(0x7f3288071110 legacy=0x7f3288158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f3286ffd640 1 -- 172.21.15.46:0/1459457030 >> v1:172.21.15.46:6789/0 conn(0x7f32880c6bd0 legacy=0x7f3288069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:01.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f32867fc640 1 -- 172.21.15.46:0/1459457030 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209223019 0 0) 0x7f3288066cf0 con 0x7f32880c6bd0 2023-10-30T13:34:01.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.602+0000 7f3286ffd640 1 -- 172.21.15.46:0/1459457030 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32880c7390 con 0x7f328806f7a0 2023-10-30T13:34:01.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.606+0000 7f3286ffd640 1 --2- 172.21.15.46:0/1459457030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f328806f7a0 0x7f328806fb80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3278009870 tx=0x7f327801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5111017322a99dfe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:01.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.606+0000 7f32867fc640 1 -- 172.21.15.46:0/1459457030 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32780a89f0 con 0x7f328806f7a0 2023-10-30T13:34:01.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.606+0000 7f32867fc640 1 -- 172.21.15.46:0/1459457030 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32780a8b50 con 0x7f328806f7a0 2023-10-30T13:34:01.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.606+0000 7f32867fc640 1 -- 172.21.15.46:0/1459457030 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32780b0440 con 0x7f328806f7a0 2023-10-30T13:34:01.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.606+0000 7f328d46b640 1 -- 172.21.15.46:0/1459457030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f328806f7a0 msgr2=0x7f328806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:01.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.606+0000 7f328d46b640 1 --2- 172.21.15.46:0/1459457030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f328806f7a0 0x7f328806fb80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3278009870 tx=0x7f327801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:34:01.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.606+0000 7f328d46b640 1 -- 172.21.15.46:0/1459457030 shutdown_connections 2023-10-30T13:34:01.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.606+0000 7f328d46b640 1 --2- 172.21.15.46:0/1459457030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f328806f7a0 0x7f328806fb80 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:01.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.606+0000 7f328d46b640 1 -- 172.21.15.46:0/1459457030 >> 172.21.15.46:0/1459457030 conn(0x7f32880c7910 msgr2=0x7f32880c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:01.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.606+0000 7f328d46b640 1 -- 172.21.15.46:0/1459457030 shutdown_connections 2023-10-30T13:34:01.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.606+0000 7f328d46b640 1 -- 172.21.15.46:0/1459457030 wait complete. 2023-10-30T13:34:01.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f328d46b640 1 Processor -- start 2023-10-30T13:34:01.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f328d46b640 1 -- start start 2023-10-30T13:34:01.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f328d46b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f328806f7a0 0x7f3288153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:01.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f328d46b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3288071110 0x7f32881f0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:01.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f328d46b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32880c6bd0 0x7f32880c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:01.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f328d46b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32881f0180 con 0x7f328806f7a0 2023-10-30T13:34:01.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f328d46b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3288155fc0 con 0x7f3288071110 2023-10-30T13:34:01.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f328d46b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3288156260 con 0x7f32880c6bd0 2023-10-30T13:34:01.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f3286ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f328806f7a0 0x7f3288153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:01.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f3286ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f328806f7a0 0x7f3288153970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52308/0 (socket says 172.21.15.46:52308) 2023-10-30T13:34:01.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f3286ffd640 1 -- 172.21.15.46:0/201260071 learned_addr learned my addr 172.21.15.46:0/201260071 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:01.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f327ffff640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3288071110 0x7f32881f0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:01.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f32877fe640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32880c6bd0 0x7f32880c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:01.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f3286ffd640 1 -- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32880c6bd0 msgr2=0x7f32880c3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:01.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f3286ffd640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32880c6bd0 0x7f32880c3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:01.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f3286ffd640 1 -- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3288071110 msgr2=0x7f32881f0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:01.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f3286ffd640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3288071110 0x7f32881f0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:01.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f3286ffd640 1 -- 172.21.15.46:0/201260071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780094d0 con 0x7f328806f7a0 2023-10-30T13:34:01.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f3286ffd640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f328806f7a0 0x7f3288153970 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f32780b3040 tx=0x7f327801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:01.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f327f7fe640 1 -- 172.21.15.46:0/201260071 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32780b0e40 con 0x7f328806f7a0 2023-10-30T13:34:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f327f7fe640 1 -- 172.21.15.46:0/201260071 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32780bb470 con 0x7f328806f7a0 2023-10-30T13:34:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f327f7fe640 1 -- 172.21.15.46:0/201260071 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32780bb820 con 0x7f328806f7a0 2023-10-30T13:34:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f328d46b640 1 -- 172.21.15.46:0/201260071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3288156540 con 0x7f328806f7a0 2023-10-30T13:34:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.610+0000 7f328d46b640 1 -- 172.21.15.46:0/201260071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32881efe10 con 0x7f328806f7a0 2023-10-30T13:34:01.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.614+0000 7f328d46b640 1 -- 172.21.15.46:0/201260071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328804e170 con 0x7f328806f7a0 2023-10-30T13:34:01.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.614+0000 7f327f7fe640 1 -- 172.21.15.46:0/201260071 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f32780b7070 con 0x7f328806f7a0 2023-10-30T13:34:01.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.614+0000 7f327f7fe640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32680664a0 0x7f3268068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:01.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.614+0000 7f327f7fe640 1 -- 172.21.15.46:0/201260071 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f32781224b0 con 0x7f328806f7a0 2023-10-30T13:34:01.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.614+0000 7f327ffff640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32680664a0 0x7f3268068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:01.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.614+0000 7f327ffff640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32680664a0 0x7f3268068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3288153860 tx=0x7f3270007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:01.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.618+0000 7f327f7fe640 1 -- 172.21.15.46:0/201260071 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f32780ef680 con 0x7f328806f7a0 2023-10-30T13:34:01.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.806+0000 7f328d46b640 1 -- 172.21.15.46:0/201260071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f328804e170 con 0x7f328806f7a0 2023-10-30T13:34:01.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f327f7fe640 1 -- 172.21.15.46:0/201260071 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f32780b5090 con 0x7f328806f7a0 2023-10-30T13:34:01.813 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:34:01.814 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:59.913534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:01.814 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 58 2023-10-30T13:34:01.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f328d46b640 1 -- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32680664a0 msgr2=0x7f3268068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:01.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f328d46b640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32680664a0 0x7f3268068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3288153860 tx=0x7f3270007b60 comp rx=0 tx=0).stop 2023-10-30T13:34:01.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f328d46b640 1 -- 172.21.15.46:0/201260071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f328806f7a0 msgr2=0x7f3288153970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:01.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f328d46b640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f328806f7a0 0x7f3288153970 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f32780b3040 tx=0x7f327801ba90 comp rx=0 tx=0).stop 2023-10-30T13:34:01.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f328d46b640 1 -- 172.21.15.46:0/201260071 shutdown_connections 2023-10-30T13:34:01.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f328d46b640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32680664a0 0x7f3268068960 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:01.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f328d46b640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32880c6bd0 0x7f32880c3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:01.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f328d46b640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3288071110 0x7f32881f0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:01.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f328d46b640 1 --2- 172.21.15.46:0/201260071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f328806f7a0 0x7f3288153970 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:01.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f328d46b640 1 -- 172.21.15.46:0/201260071 >> 172.21.15.46:0/201260071 conn(0x7f32880c7910 msgr2=0x7f32880c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:01.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f328d46b640 1 -- 172.21.15.46:0/201260071 shutdown_connections 2023-10-30T13:34:01.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:01.810+0000 7f328d46b640 1 -- 172.21.15.46:0/201260071 wait complete. 2023-10-30T13:34:01.834 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:34:03.836 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:03.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.986+0000 7fbfcaba0640 1 Processor -- start 2023-10-30T13:34:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.986+0000 7fbfcaba0640 1 -- start start 2023-10-30T13:34:03.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.986+0000 7fbfcaba0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc40c6840 0x7fbfc415a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:03.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.986+0000 7fbfcaba0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbfc40c6c20 0x7fbfc4156590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:03.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.986+0000 7fbfcaba0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc40c3b30 con 0x7fbfc40c3640 2023-10-30T13:34:03.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.986+0000 7fbfcaba0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc40c2aa0 con 0x7fbfc40c6c20 2023-10-30T13:34:03.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.986+0000 7fbfcaba0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40c3df0 con 0x7fbfc40c6840 2023-10-30T13:34:03.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.986+0000 7fbfca39f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbfc40c6c20 0x7fbfc4156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:03.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.986+0000 7fbfca39f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbfc40c6c20 0x7fbfc4156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52312/0 (socket says 172.21.15.46:52312) 2023-10-30T13:34:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.986+0000 7fbfca39f640 1 -- 172.21.15.46:0/921069165 learned_addr learned my addr 172.21.15.46:0/921069165 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.986+0000 7fbfc9b9e640 1 --2- 172.21.15.46:0/921069165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc40c6840 0x7fbfc415a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.990+0000 7fbfca39f640 1 -- 172.21.15.46:0/921069165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc40c6840 msgr2=0x7fbfc415a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:03.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.990+0000 7fbfca39f640 1 --2- 172.21.15.46:0/921069165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc40c6840 0x7fbfc415a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:03.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.990+0000 7fbfca39f640 1 -- 172.21.15.46:0/921069165 >> v1:172.21.15.46:6789/0 conn(0x7fbfc40c3640 legacy=0x7fbfc4153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:03.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.990+0000 7fbfca39f640 1 -- 172.21.15.46:0/921069165 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc4159ba0 con 0x7fbfc40c6c20 2023-10-30T13:34:03.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.990+0000 7fbfc8b9c640 1 -- 172.21.15.46:0/921069165 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165381743 0 0) 0x7fbfc40c3b30 con 0x7fbfc40c3640 2023-10-30T13:34:03.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.990+0000 7fbfca39f640 1 --2- 172.21.15.46:0/921069165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40c6c20 0x7fbfc4156590 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc00d150 tx=0x7fbfbc01ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ab96463182fff4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:03.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfc8b9c640 1 -- 172.21.15.46:0/921069165 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc00b2d0 con 0x7fbfc40c6c20 2023-10-30T13:34:03.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfc8b9c640 1 -- 172.21.15.46:0/921069165 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc00b430 con 0x7fbfc40c6c20 2023-10-30T13:34:03.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfc8b9c640 1 -- 172.21.15.46:0/921069165 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc0a7b10 con 0x7fbfc40c6c20 2023-10-30T13:34:03.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfcaba0640 1 -- 172.21.15.46:0/921069165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40c6c20 msgr2=0x7fbfc4156590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:03.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfcaba0640 1 --2- 172.21.15.46:0/921069165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40c6c20 0x7fbfc4156590 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc00d150 tx=0x7fbfbc01ec10 comp rx=0 tx=0).stop 2023-10-30T13:34:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfcaba0640 1 -- 172.21.15.46:0/921069165 shutdown_connections 2023-10-30T13:34:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfcaba0640 1 --2- 172.21.15.46:0/921069165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40c6c20 0x7fbfc4156590 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:03.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfcaba0640 1 --2- 172.21.15.46:0/921069165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc40c6840 0x7fbfc415a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:04.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfcaba0640 1 -- 172.21.15.46:0/921069165 >> 172.21.15.46:0/921069165 conn(0x7fbfc40c75d0 msgr2=0x7fbfc40c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:04.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfcaba0640 1 -- 172.21.15.46:0/921069165 shutdown_connections 2023-10-30T13:34:04.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfcaba0640 1 -- 172.21.15.46:0/921069165 wait complete. 2023-10-30T13:34:04.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfcaba0640 1 Processor -- start 2023-10-30T13:34:04.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.994+0000 7fbfcaba0640 1 -- start start 2023-10-30T13:34:04.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfcaba0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfc40c3640 0x7fbfc41eef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:04.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfcaba0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc40c6840 0x7fbfc4159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:04.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfcaba0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40c6c20 0x7fbfc414b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:04.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfcaba0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc41548f0 con 0x7fbfc40c6c20 2023-10-30T13:34:04.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfcaba0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc4159060 con 0x7fbfc40c3640 2023-10-30T13:34:04.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfcaba0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc41599c0 con 0x7fbfc40c6840 2023-10-30T13:34:04.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfca39f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40c6c20 0x7fbfc414b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:04.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfca39f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40c6c20 0x7fbfc414b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52326/0 (socket says 172.21.15.46:52326) 2023-10-30T13:34:04.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfca39f640 1 -- 172.21.15.46:0/4020254067 learned_addr learned my addr 172.21.15.46:0/4020254067 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:04.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfc9b9e640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfc40c3640 0x7fbfc41eef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:04.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfc939d640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc40c6840 0x7fbfc4159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:04.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfca39f640 1 -- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc40c6840 msgr2=0x7fbfc4159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:04.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfca39f640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc40c6840 0x7fbfc4159480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:04.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfca39f640 1 -- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfc40c3640 msgr2=0x7fbfc41eef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:04.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfca39f640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfc40c3640 0x7fbfc41eef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:04.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfca39f640 1 -- 172.21.15.46:0/4020254067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfbc00bd40 con 0x7fbfc40c6c20 2023-10-30T13:34:04.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfca39f640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40c6c20 0x7fbfc414b830 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc01c840 tx=0x7fbfbc01b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:04.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfba7fc640 1 -- 172.21.15.46:0/4020254067 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc009800 con 0x7fbfc40c6c20 2023-10-30T13:34:04.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfba7fc640 1 -- 172.21.15.46:0/4020254067 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc009960 con 0x7fbfc40c6c20 2023-10-30T13:34:04.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfba7fc640 1 -- 172.21.15.46:0/4020254067 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc009d10 con 0x7fbfc40c6c20 2023-10-30T13:34:04.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfcaba0640 1 -- 172.21.15.46:0/4020254067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc41554a0 con 0x7fbfc40c6c20 2023-10-30T13:34:04.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfcaba0640 1 -- 172.21.15.46:0/4020254067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc4158750 con 0x7fbfc40c6c20 2023-10-30T13:34:04.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfcaba0640 1 -- 172.21.15.46:0/4020254067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc41591a0 con 0x7fbfc40c6c20 2023-10-30T13:34:04.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:03.998+0000 7fbfba7fc640 1 -- 172.21.15.46:0/4020254067 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbfbc0ba470 con 0x7fbfc40c6c20 2023-10-30T13:34:04.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.002+0000 7fbfba7fc640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbfa80664a0 0x7fbfa8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:04.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.002+0000 7fbfba7fc640 1 -- 172.21.15.46:0/4020254067 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fbfbc121360 con 0x7fbfc40c6c20 2023-10-30T13:34:04.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.002+0000 7fbfc9b9e640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbfa80664a0 0x7fbfa8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:04.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.002+0000 7fbfc9b9e640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbfa80664a0 0x7fbfa8068960 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009b40 tx=0x7fbfc0006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:04.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.006+0000 7fbfba7fc640 1 -- 172.21.15.46:0/4020254067 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbfc41591a0 con 0x7fbfc40c6c20 2023-10-30T13:34:04.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.194+0000 7fbfcaba0640 1 -- 172.21.15.46:0/4020254067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfc41554a0 con 0x7fbfc40c6c20 2023-10-30T13:34:04.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.194+0000 7fbfba7fc640 1 -- 172.21.15.46:0/4020254067 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fbfc41554a0 con 0x7fbfc40c6c20 2023-10-30T13:34:04.200 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:34:04.200 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:33:59.913534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:04.201 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 58 2023-10-30T13:34:04.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.198+0000 7fbfcaba0640 1 -- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbfa80664a0 msgr2=0x7fbfa8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:04.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.198+0000 7fbfcaba0640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbfa80664a0 0x7fbfa8068960 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009b40 tx=0x7fbfc0006f90 comp rx=0 tx=0).stop 2023-10-30T13:34:04.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.198+0000 7fbfcaba0640 1 -- 172.21.15.46:0/4020254067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40c6c20 msgr2=0x7fbfc414b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:04.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.198+0000 7fbfcaba0640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40c6c20 0x7fbfc414b830 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc01c840 tx=0x7fbfbc01b9b0 comp rx=0 tx=0).stop 2023-10-30T13:34:04.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.198+0000 7fbfcaba0640 1 -- 172.21.15.46:0/4020254067 shutdown_connections 2023-10-30T13:34:04.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.198+0000 7fbfcaba0640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbfa80664a0 0x7fbfa8068960 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:04.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.198+0000 7fbfcaba0640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40c6c20 0x7fbfc414b830 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:04.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.198+0000 7fbfcaba0640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc40c6840 0x7fbfc4159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:04.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.198+0000 7fbfcaba0640 1 --2- 172.21.15.46:0/4020254067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfc40c3640 0x7fbfc41eef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:04.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.198+0000 7fbfcaba0640 1 -- 172.21.15.46:0/4020254067 >> 172.21.15.46:0/4020254067 conn(0x7fbfc40c75d0 msgr2=0x7fbfc4156bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:04.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.198+0000 7fbfcaba0640 1 -- 172.21.15.46:0/4020254067 shutdown_connections 2023-10-30T13:34:04.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:04.198+0000 7fbfcaba0640 1 -- 172.21.15.46:0/4020254067 wait complete. 2023-10-30T13:34:04.221 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:34:06.222 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:06.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f076dc06640 1 Processor -- start 2023-10-30T13:34:06.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f076dc06640 1 -- start start 2023-10-30T13:34:06.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f076dc06640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f076815cd50 0x7f07680c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:06.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f076dc06640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076815a6c0 con 0x7f07681538f0 2023-10-30T13:34:06.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f076dc06640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076815d300 con 0x7f07680cefa0 2023-10-30T13:34:06.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f076dc06640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076815d5e0 con 0x7f076815cd50 2023-10-30T13:34:06.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f0767fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f076815cd50 0x7f07680c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:06.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f076cc04640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f07681538f0 0x7f0768159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48852/0 (socket says 172.21.15.46:48852) 2023-10-30T13:34:06.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f076cc04640 1 -- 172.21.15.46:0/4088078914 learned_addr learned my addr 172.21.15.46:0/4088078914 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:34:06.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f07677fe640 1 -- 172.21.15.46:0/4088078914 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270483108 0 0) 0x7f076815a6c0 con 0x7f07681538f0 2023-10-30T13:34:06.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f07677fe640 1 -- 172.21.15.46:0/4088078914 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0758003530 con 0x7f07681538f0 2023-10-30T13:34:06.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f0767fff640 1 -- 172.21.15.46:0/4088078914 >> v1:172.21.15.154:6789/0 conn(0x7f07680cefa0 legacy=0x7f07681640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:06.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f0767fff640 1 -- 172.21.15.46:0/4088078914 >> v1:172.21.15.46:6789/0 conn(0x7f07681538f0 legacy=0x7f0768159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:06.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.374+0000 7f0767fff640 1 -- 172.21.15.46:0/4088078914 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076815d8c0 con 0x7f076815cd50 2023-10-30T13:34:06.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.378+0000 7f0767fff640 1 --2- 172.21.15.46:0/4088078914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f076815cd50 0x7f07680c9f90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f07540097c0 tx=0x7f075401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f24a433e09e3f496 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:06.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.378+0000 7f07677fe640 1 -- 172.21.15.46:0/4088078914 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07540a8970 con 0x7f076815cd50 2023-10-30T13:34:06.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.378+0000 7f07677fe640 1 -- 172.21.15.46:0/4088078914 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07540a8ad0 con 0x7f076815cd50 2023-10-30T13:34:06.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.378+0000 7f07677fe640 1 -- 172.21.15.46:0/4088078914 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07540a8e60 con 0x7f076815cd50 2023-10-30T13:34:06.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.378+0000 7f076dc06640 1 -- 172.21.15.46:0/4088078914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f076815cd50 msgr2=0x7f07680c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:06.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.378+0000 7f076dc06640 1 --2- 172.21.15.46:0/4088078914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f076815cd50 0x7f07680c9f90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f07540097c0 tx=0x7f075401ec70 comp rx=0 tx=0).stop 2023-10-30T13:34:06.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.378+0000 7f076dc06640 1 -- 172.21.15.46:0/4088078914 shutdown_connections 2023-10-30T13:34:06.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.378+0000 7f076dc06640 1 --2- 172.21.15.46:0/4088078914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f076815cd50 0x7f07680c9f90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:06.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.378+0000 7f076dc06640 1 -- 172.21.15.46:0/4088078914 >> 172.21.15.46:0/4088078914 conn(0x7f076806a640 msgr2=0x7f0768065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:06.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.378+0000 7f076dc06640 1 -- 172.21.15.46:0/4088078914 shutdown_connections 2023-10-30T13:34:06.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.378+0000 7f076dc06640 1 -- 172.21.15.46:0/4088078914 wait complete. 2023-10-30T13:34:06.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076dc06640 1 Processor -- start 2023-10-30T13:34:06.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076dc06640 1 -- start start 2023-10-30T13:34:06.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076dc06640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07680cefa0 0x7f07681f02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:06.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076dc06640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07681538f0 0x7f0768152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:06.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076dc06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076815cd50 0x7f0768151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:06.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076dc06640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07680b34a0 con 0x7f076815cd50 2023-10-30T13:34:06.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076dc06640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07681515e0 con 0x7f07681538f0 2023-10-30T13:34:06.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076dc06640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0768152ad0 con 0x7f07680cefa0 2023-10-30T13:34:06.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076d405640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076815cd50 0x7f0768151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:06.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076d405640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076815cd50 0x7f0768151a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52332/0 (socket says 172.21.15.46:52332) 2023-10-30T13:34:06.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076d405640 1 -- 172.21.15.46:0/155870855 learned_addr learned my addr 172.21.15.46:0/155870855 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:06.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f0767fff640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07681538f0 0x7f0768152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:06.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076cc04640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07680cefa0 0x7f07681f02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:06.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076d405640 1 -- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07680cefa0 msgr2=0x7f07681f02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:06.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076d405640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07680cefa0 0x7f07681f02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:06.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076d405640 1 -- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07681538f0 msgr2=0x7f0768152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:06.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076d405640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07681538f0 0x7f0768152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:06.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076d405640 1 -- 172.21.15.46:0/155870855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0754009420 con 0x7f076815cd50 2023-10-30T13:34:06.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076cc04640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07680cefa0 0x7f07681f02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:06.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076d405640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076815cd50 0x7f0768151a70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f076000c770 tx=0x7f076000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:06.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f07657fa640 1 -- 172.21.15.46:0/155870855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0760003b80 con 0x7f076815cd50 2023-10-30T13:34:06.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f07657fa640 1 -- 172.21.15.46:0/155870855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0760003ce0 con 0x7f076815cd50 2023-10-30T13:34:06.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f07657fa640 1 -- 172.21.15.46:0/155870855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076000aa90 con 0x7f076815cd50 2023-10-30T13:34:06.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076dc06640 1 -- 172.21.15.46:0/155870855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0768152d90 con 0x7f076815cd50 2023-10-30T13:34:06.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.382+0000 7f076dc06640 1 -- 172.21.15.46:0/155870855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0768150f90 con 0x7f076815cd50 2023-10-30T13:34:06.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.386+0000 7f076dc06640 1 -- 172.21.15.46:0/155870855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076804e170 con 0x7f076815cd50 2023-10-30T13:34:06.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.386+0000 7f07657fa640 1 -- 172.21.15.46:0/155870855 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f076000abf0 con 0x7f076815cd50 2023-10-30T13:34:06.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.386+0000 7f07657fa640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f074c0664a0 0x7f074c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:06.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.386+0000 7f07657fa640 1 -- 172.21.15.46:0/155870855 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f0760083f50 con 0x7f076815cd50 2023-10-30T13:34:06.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.386+0000 7f076cc04640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f074c0664a0 0x7f074c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:06.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.386+0000 7f076cc04640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f074c0664a0 0x7f074c068960 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f075c0011f0 tx=0x7f075c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:06.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.390+0000 7f07657fa640 1 -- 172.21.15.46:0/155870855 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07600510a0 con 0x7f076815cd50 2023-10-30T13:34:06.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.578+0000 7f076dc06640 1 -- 172.21.15.46:0/155870855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0768151720 con 0x7f076815cd50 2023-10-30T13:34:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.578+0000 7f07657fa640 1 -- 172.21.15.46:0/155870855 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f07600546c0 con 0x7f076815cd50 2023-10-30T13:34:06.584 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:34:06.585 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:34:05.920156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 59 2023-10-30T13:34:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.582+0000 7f076dc06640 1 -- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f074c0664a0 msgr2=0x7f074c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.582+0000 7f076dc06640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f074c0664a0 0x7f074c068960 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f075c0011f0 tx=0x7f075c009040 comp rx=0 tx=0).stop 2023-10-30T13:34:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.582+0000 7f076dc06640 1 -- 172.21.15.46:0/155870855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076815cd50 msgr2=0x7f0768151a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.582+0000 7f076dc06640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076815cd50 0x7f0768151a70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f076000c770 tx=0x7f076000cc30 comp rx=0 tx=0).stop 2023-10-30T13:34:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.582+0000 7f076dc06640 1 -- 172.21.15.46:0/155870855 shutdown_connections 2023-10-30T13:34:06.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.582+0000 7f076dc06640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f074c0664a0 0x7f074c068960 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:06.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.582+0000 7f076dc06640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076815cd50 0x7f0768151a70 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:06.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.582+0000 7f076dc06640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07681538f0 0x7f0768152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:06.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.582+0000 7f076dc06640 1 --2- 172.21.15.46:0/155870855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07680cefa0 0x7f07681f02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:06.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.582+0000 7f076dc06640 1 -- 172.21.15.46:0/155870855 >> 172.21.15.46:0/155870855 conn(0x7f076806a640 msgr2=0x7f07680c4490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:06.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.582+0000 7f076dc06640 1 -- 172.21.15.46:0/155870855 shutdown_connections 2023-10-30T13:34:06.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:06.582+0000 7f076dc06640 1 -- 172.21.15.46:0/155870855 wait complete. 2023-10-30T13:34:06.606 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=1, replacing gid=5106 2023-10-30T13:34:06.606 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.a 2023-10-30T13:34:11.608 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:34:11.608 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:34:11.608 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:34:11.613 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:34:11.613 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:11.710 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-10-30T13:34:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.762+0000 7fc5ba372640 1 Processor -- start 2023-10-30T13:34:11.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.762+0000 7fc5ba372640 1 -- start start 2023-10-30T13:34:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.762+0000 7fc5ba372640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc5b4153bf0 0x7fc5b4071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:11.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.762+0000 7fc5ba372640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5b40c5f60 0x7fc5b40cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.762+0000 7fc5ba372640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b40700b0 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.762+0000 7fc5ba372640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b40c6810 con 0x7fc5b4153bf0 2023-10-30T13:34:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.762+0000 7fc5ba372640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b4159900 con 0x7fc5b40c5f60 2023-10-30T13:34:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.766+0000 7fc5b3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc5b4153bf0 0x7fc5b4071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.766+0000 7fc5b37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc5b40c6fe0 0x7fc5b40c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35402/0 (socket says 172.21.15.46:35402) 2023-10-30T13:34:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.766+0000 7fc5b37fe640 1 -- 172.21.15.46:0/1890803905 learned_addr learned my addr 172.21.15.46:0/1890803905 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:34:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.766+0000 7fc5b890f640 1 --2- 172.21.15.46:0/1890803905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5b40c5f60 0x7fc5b40cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.766+0000 7fc5b3fff640 1 -- 172.21.15.46:0/1890803905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5b40c5f60 msgr2=0x7fc5b40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.766+0000 7fc5b3fff640 1 --2- 172.21.15.46:0/1890803905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5b40c5f60 0x7fc5b40cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.766+0000 7fc5b3fff640 1 -- 172.21.15.46:0/1890803905 >> v1:172.21.15.46:6789/0 conn(0x7fc5b40c6fe0 legacy=0x7fc5b40c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.766+0000 7fc5b2ffd640 1 -- 172.21.15.46:0/1890803905 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658195179 0 0) 0x7fc5b40700b0 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.766+0000 7fc5b3fff640 1 -- 172.21.15.46:0/1890803905 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b40c6530 con 0x7fc5b4153bf0 2023-10-30T13:34:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.766+0000 7fc5b3fff640 1 --2- 172.21.15.46:0/1890803905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5b4153bf0 0x7fc5b4071260 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc5a40097c0 tx=0x7fc5a401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5820b997386f8d4a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5b2ffd640 1 -- 172.21.15.46:0/1890803905 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a40a8970 con 0x7fc5b4153bf0 2023-10-30T13:34:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5b2ffd640 1 -- 172.21.15.46:0/1890803905 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a40a8ad0 con 0x7fc5b4153bf0 2023-10-30T13:34:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5b2ffd640 1 -- 172.21.15.46:0/1890803905 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a40a8e60 con 0x7fc5b4153bf0 2023-10-30T13:34:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 -- 172.21.15.46:0/1890803905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5b4153bf0 msgr2=0x7fc5b4071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 --2- 172.21.15.46:0/1890803905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5b4153bf0 0x7fc5b4071260 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc5a40097c0 tx=0x7fc5a401ec70 comp rx=0 tx=0).stop 2023-10-30T13:34:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 -- 172.21.15.46:0/1890803905 shutdown_connections 2023-10-30T13:34:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 --2- 172.21.15.46:0/1890803905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5b40c5f60 0x7fc5b40cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 --2- 172.21.15.46:0/1890803905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5b4153bf0 0x7fc5b4071260 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 -- 172.21.15.46:0/1890803905 >> 172.21.15.46:0/1890803905 conn(0x7fc5b406c6e0 msgr2=0x7fc5b4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 -- 172.21.15.46:0/1890803905 shutdown_connections 2023-10-30T13:34:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 -- 172.21.15.46:0/1890803905 wait complete. 2023-10-30T13:34:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 Processor -- start 2023-10-30T13:34:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 -- start start 2023-10-30T13:34:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5b40c5f60 0x7fc5b40ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5b40c6fe0 0x7fc5b414e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5b4153bf0 0x7fc5b414e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b414d1f0 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b41eefd0 con 0x7fc5b4153bf0 2023-10-30T13:34:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5ba372640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b41efdf0 con 0x7fc5b40c5f60 2023-10-30T13:34:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5b37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5b40c6fe0 0x7fc5b414e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5b37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5b40c6fe0 0x7fc5b414e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37090/0 (socket says 172.21.15.46:37090) 2023-10-30T13:34:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.770+0000 7fc5b37fe640 1 -- 172.21.15.46:0/717133604 learned_addr learned my addr 172.21.15.46:0/717133604 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5b3fff640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5b40c5f60 0x7fc5b40ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5b890f640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5b4153bf0 0x7fc5b414e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5b37fe640 1 -- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5b40c5f60 msgr2=0x7fc5b40ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5b37fe640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5b40c5f60 0x7fc5b40ca790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5b37fe640 1 -- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5b4153bf0 msgr2=0x7fc5b414e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5b37fe640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5b4153bf0 0x7fc5b414e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5b37fe640 1 -- 172.21.15.46:0/717133604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a4009420 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5b37fe640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5b40c6fe0 0x7fc5b414e050 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc5a80011f0 tx=0x7fc5a800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5b0ff9640 1 -- 172.21.15.46:0/717133604 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a80074e0 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5b0ff9640 1 -- 172.21.15.46:0/717133604 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a8007640 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5b0ff9640 1 -- 172.21.15.46:0/717133604 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a8004a10 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5ba372640 1 -- 172.21.15.46:0/717133604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b414f120 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5ba372640 1 -- 172.21.15.46:0/717133604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b41ef4c0 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5ba372640 1 -- 172.21.15.46:0/717133604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b404e170 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.774+0000 7fc5b0ff9640 1 -- 172.21.15.46:0/717133604 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc5a8004b70 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.778+0000 7fc5b0ff9640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5a00664a0 0x7fc5a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.778+0000 7fc5b0ff9640 1 -- 172.21.15.46:0/717133604 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fc5a8083340 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.778+0000 7fc5b3fff640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5a00664a0 0x7fc5a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.778+0000 7fc5b3fff640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5a00664a0 0x7fc5a0068960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc5a40b2000 tx=0x7fc5a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.782+0000 7fc5b0ff9640 1 -- 172.21.15.46:0/717133604 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc5a80504a0 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.974+0000 7fc5ba372640 1 -- 172.21.15.46:0/717133604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5b41590f0 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.974+0000 7fc5b0ff9640 1 -- 172.21.15.46:0/717133604 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fc5a8018090 con 0x7fc5b40c6fe0 2023-10-30T13:34:11.979 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:34:11.979 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:34:05.920156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:11.980 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 59 2023-10-30T13:34:11.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.974+0000 7fc5ba372640 1 -- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5a00664a0 msgr2=0x7fc5a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:11.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.974+0000 7fc5ba372640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5a00664a0 0x7fc5a0068960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc5a40b2000 tx=0x7fc5a4008040 comp rx=0 tx=0).stop 2023-10-30T13:34:11.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.974+0000 7fc5ba372640 1 -- 172.21.15.46:0/717133604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5b40c6fe0 msgr2=0x7fc5b414e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:11.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.974+0000 7fc5ba372640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5b40c6fe0 0x7fc5b414e050 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc5a80011f0 tx=0x7fc5a800c9b0 comp rx=0 tx=0).stop 2023-10-30T13:34:11.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.978+0000 7fc5ba372640 1 -- 172.21.15.46:0/717133604 shutdown_connections 2023-10-30T13:34:11.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.978+0000 7fc5ba372640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5a00664a0 0x7fc5a0068960 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.978+0000 7fc5ba372640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5b4153bf0 0x7fc5b414e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.978+0000 7fc5ba372640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5b40c6fe0 0x7fc5b414e050 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.978+0000 7fc5ba372640 1 --2- 172.21.15.46:0/717133604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5b40c5f60 0x7fc5b40ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.978+0000 7fc5ba372640 1 -- 172.21.15.46:0/717133604 >> 172.21.15.46:0/717133604 conn(0x7fc5b406c6e0 msgr2=0x7fc5b4151ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.978+0000 7fc5ba372640 1 -- 172.21.15.46:0/717133604 shutdown_connections 2023-10-30T13:34:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:11.978+0000 7fc5ba372640 1 -- 172.21.15.46:0/717133604 wait complete. 2023-10-30T13:34:12.001 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T13:34:14.002 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:14.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.150+0000 7f8dbe73f640 1 Processor -- start 2023-10-30T13:34:14.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.150+0000 7f8dbe73f640 1 -- start start 2023-10-30T13:34:14.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbe73f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8db80ccdc0 0x7f8db806f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:14.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbe73f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8db80c2490 0x7f8db80c2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:14.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbe73f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db80c2e80 con 0x7f8db80cd1a0 2023-10-30T13:34:14.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbe73f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db8070050 con 0x7f8db80c2490 2023-10-30T13:34:14.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbe73f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db8159860 con 0x7f8db80ccdc0 2023-10-30T13:34:14.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbcf3c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8db80cd1a0 0x7f8db815d510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35418/0 (socket says 172.21.15.46:35418) 2023-10-30T13:34:14.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbcf3c640 1 -- 172.21.15.46:0/1146058099 learned_addr learned my addr 172.21.15.46:0/1146058099 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:34:14.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbd73d640 1 --2- 172.21.15.46:0/1146058099 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8db80ccdc0 0x7f8db806f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:14.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbdf3e640 1 --2- 172.21.15.46:0/1146058099 >> v2:172.21.15.154:3300/0 conn(0x7f8db80c2490 0x7f8db80c2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:14.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8daffff640 1 -- 172.21.15.46:0/1146058099 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099035536 0 0) 0x7f8db80c2e80 con 0x7f8db80cd1a0 2023-10-30T13:34:14.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8daffff640 1 -- 172.21.15.46:0/1146058099 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8da4003530 con 0x7f8db80cd1a0 2023-10-30T13:34:14.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbd73d640 1 -- 172.21.15.46:0/1146058099 >> v2:172.21.15.154:3300/0 conn(0x7f8db80c2490 msgr2=0x7f8db80c2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:14.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbd73d640 1 --2- 172.21.15.46:0/1146058099 >> v2:172.21.15.154:3300/0 conn(0x7f8db80c2490 0x7f8db80c2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbd73d640 1 -- 172.21.15.46:0/1146058099 >> v1:172.21.15.46:6789/0 conn(0x7f8db80cd1a0 legacy=0x7f8db815d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:14.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.154+0000 7f8dbd73d640 1 -- 172.21.15.46:0/1146058099 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db8159b40 con 0x7f8db80ccdc0 2023-10-30T13:34:14.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8dbd73d640 1 --2- 172.21.15.46:0/1146058099 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8db80ccdc0 0x7f8db806f9f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8db40097c0 tx=0x7f8db401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7a14e0bff947333 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:14.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8daffff640 1 -- 172.21.15.46:0/1146058099 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db40a8970 con 0x7f8db80ccdc0 2023-10-30T13:34:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8daffff640 1 -- 172.21.15.46:0/1146058099 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db40a8ad0 con 0x7f8db80ccdc0 2023-10-30T13:34:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8daffff640 1 -- 172.21.15.46:0/1146058099 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db40a8e60 con 0x7f8db80ccdc0 2023-10-30T13:34:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1146058099 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8db80ccdc0 msgr2=0x7f8db806f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8dbe73f640 1 --2- 172.21.15.46:0/1146058099 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8db80ccdc0 0x7f8db806f9f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8db40097c0 tx=0x7f8db401ec70 comp rx=0 tx=0).stop 2023-10-30T13:34:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1146058099 shutdown_connections 2023-10-30T13:34:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8dbe73f640 1 --2- 172.21.15.46:0/1146058099 >> v2:172.21.15.154:3300/0 conn(0x7f8db80c2490 0x7f8db80c2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8dbe73f640 1 --2- 172.21.15.46:0/1146058099 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8db80ccdc0 0x7f8db806f9f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1146058099 >> 172.21.15.46:0/1146058099 conn(0x7f8db806dfc0 msgr2=0x7f8db8071710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1146058099 shutdown_connections 2023-10-30T13:34:14.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1146058099 wait complete. 2023-10-30T13:34:14.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8dbe73f640 1 Processor -- start 2023-10-30T13:34:14.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.158+0000 7f8dbe73f640 1 -- start start 2023-10-30T13:34:14.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbe73f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8db80c2490 0x7f8db81ef880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:14.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbe73f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8db80ccdc0 0x7f8db8158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:14.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbe73f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8db80cd1a0 0x7f8db80c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:14.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbe73f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db8158d00 con 0x7f8db80cd1a0 2023-10-30T13:34:14.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbe73f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db8158fe0 con 0x7f8db80c2490 2023-10-30T13:34:14.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbe73f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db81592c0 con 0x7f8db80ccdc0 2023-10-30T13:34:14.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbdf3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8db80cd1a0 0x7f8db80c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:14.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbcf3c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8db80ccdc0 0x7f8db8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:14.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbd73d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8db80c2490 0x7f8db81ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:14.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbdf3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8db80cd1a0 0x7f8db80c5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37100/0 (socket says 172.21.15.46:37100) 2023-10-30T13:34:14.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbdf3e640 1 -- 172.21.15.46:0/1321755481 learned_addr learned my addr 172.21.15.46:0/1321755481 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:14.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbdf3e640 1 -- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8db80ccdc0 msgr2=0x7f8db8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:14.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbdf3e640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8db80ccdc0 0x7f8db8158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbdf3e640 1 -- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8db80c2490 msgr2=0x7f8db81ef880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:14.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbdf3e640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8db80c2490 0x7f8db81ef880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbdf3e640 1 -- 172.21.15.46:0/1321755481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db4009420 con 0x7f8db80cd1a0 2023-10-30T13:34:14.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbd73d640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8db80c2490 0x7f8db81ef880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:34:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbdf3e640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8db80cd1a0 0x7f8db80c5030 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8db000efc0 tx=0x7f8db000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dadffb640 1 -- 172.21.15.46:0/1321755481 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db0009ba0 con 0x7f8db80cd1a0 2023-10-30T13:34:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dadffb640 1 -- 172.21.15.46:0/1321755481 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db000f040 con 0x7f8db80cd1a0 2023-10-30T13:34:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dadffb640 1 -- 172.21.15.46:0/1321755481 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db00146c0 con 0x7f8db80cd1a0 2023-10-30T13:34:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1321755481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db81ef080 con 0x7f8db80cd1a0 2023-10-30T13:34:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1321755481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db81494f0 con 0x7f8db80cd1a0 2023-10-30T13:34:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.162+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1321755481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db81ef5a0 con 0x7f8db80cd1a0 2023-10-30T13:34:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.166+0000 7f8dadffb640 1 -- 172.21.15.46:0/1321755481 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8db0014820 con 0x7f8db80cd1a0 2023-10-30T13:34:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.166+0000 7f8dadffb640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d9c0664a0 0x7f8d9c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.166+0000 7f8dadffb640 1 -- 172.21.15.46:0/1321755481 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f8db0084af0 con 0x7f8db80cd1a0 2023-10-30T13:34:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.166+0000 7f8dbd73d640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d9c0664a0 0x7f8d9c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.166+0000 7f8dbd73d640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d9c0664a0 0x7f8d9c068960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8db40b2000 tx=0x7f8db4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:14.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.174+0000 7f8dadffb640 1 -- 172.21.15.46:0/1321755481 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8db81ef5a0 con 0x7f8db80cd1a0 2023-10-30T13:34:14.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.370+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1321755481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8db81ef5a0 con 0x7f8db80cd1a0 2023-10-30T13:34:14.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.370+0000 7f8dadffb640 1 -- 172.21.15.46:0/1321755481 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f8db81ef5a0 con 0x7f8db80cd1a0 2023-10-30T13:34:14.375 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:34:14.376 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5721,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:34:05.920156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:14.376 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2023-10-30T13:34:14.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.374+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d9c0664a0 msgr2=0x7f8d9c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:14.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.374+0000 7f8dbe73f640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d9c0664a0 0x7f8d9c068960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8db40b2000 tx=0x7f8db4008040 comp rx=0 tx=0).stop 2023-10-30T13:34:14.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.374+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1321755481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8db80cd1a0 msgr2=0x7f8db80c5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:14.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.374+0000 7f8dbe73f640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8db80cd1a0 0x7f8db80c5030 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8db000efc0 tx=0x7f8db000c490 comp rx=0 tx=0).stop 2023-10-30T13:34:14.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.374+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1321755481 shutdown_connections 2023-10-30T13:34:14.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.374+0000 7f8dbe73f640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d9c0664a0 0x7f8d9c068960 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.374+0000 7f8dbe73f640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8db80cd1a0 0x7f8db80c5030 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.374+0000 7f8dbe73f640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8db80ccdc0 0x7f8db8158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.374+0000 7f8dbe73f640 1 --2- 172.21.15.46:0/1321755481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8db80c2490 0x7f8db81ef880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.374+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1321755481 >> 172.21.15.46:0/1321755481 conn(0x7f8db806dfc0 msgr2=0x7f8db806ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:14.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.374+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1321755481 shutdown_connections 2023-10-30T13:34:14.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:14.374+0000 7f8dbe73f640 1 -- 172.21.15.46:0/1321755481 wait complete. 2023-10-30T13:34:14.397 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:34:14.397 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 35 secs before thrashing 2023-10-30T13:34:16.684 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:34:16.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:34:46.739 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:34:46.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:34:49.398 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.545+0000 7f9c10d01640 1 Processor -- start 2023-10-30T13:34:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.545+0000 7f9c10d01640 1 -- start start 2023-10-30T13:34:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c10d01640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9c0c1538a0 0x7f9c0c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c10d01640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c0cb800 con 0x7f9c0c0ca0e0 2023-10-30T13:34:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c10d01640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c15dc30 con 0x7f9c0c0cf160 2023-10-30T13:34:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c10d01640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0c0cb330 con 0x7f9c0c1538a0 2023-10-30T13:34:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c09d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9c0c0ca0e0 0x7f9c0c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32904/0 (socket says 172.21.15.46:32904) 2023-10-30T13:34:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c09d9b640 1 -- 172.21.15.46:0/531932611 learned_addr learned my addr 172.21.15.46:0/531932611 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:34:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c0a59c640 1 --2- 172.21.15.46:0/531932611 >> v2:172.21.15.154:3300/0 conn(0x7f9c0c1538a0 0x7f9c0c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c0959a640 1 -- 172.21.15.46:0/531932611 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317531723 0 0) 0x7f9c0c0cb800 con 0x7f9c0c0ca0e0 2023-10-30T13:34:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c0959a640 1 -- 172.21.15.46:0/531932611 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf4003530 con 0x7f9c0c0ca0e0 2023-10-30T13:34:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c0a59c640 1 -- 172.21.15.46:0/531932611 >> v1:172.21.15.154:6789/0 conn(0x7f9c0c0cf160 legacy=0x7f9c0c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c0a59c640 1 -- 172.21.15.46:0/531932611 >> v1:172.21.15.46:6789/0 conn(0x7f9c0c0ca0e0 legacy=0x7f9c0c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c0959a640 1 -- 172.21.15.46:0/531932611 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1560147142 0 0) 0x7f9bf4003530 con 0x7f9c0c0ca0e0 2023-10-30T13:34:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.549+0000 7f9c0a59c640 1 -- 172.21.15.46:0/531932611 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c0cad70 con 0x7f9c0c1538a0 2023-10-30T13:34:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c0a59c640 1 --2- 172.21.15.46:0/531932611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c0c1538a0 0x7f9c0c15d6f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0097c0 tx=0x7f9bfc01eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7131e7b6c2015c41 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:49.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c0959a640 1 -- 172.21.15.46:0/531932611 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c00001680 con 0x7f9c0c1538a0 2023-10-30T13:34:49.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c0959a640 1 -- 172.21.15.46:0/531932611 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc0a8610 con 0x7f9c0c1538a0 2023-10-30T13:34:49.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c0959a640 1 -- 172.21.15.46:0/531932611 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc0a8980 con 0x7f9c0c1538a0 2023-10-30T13:34:49.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c10d01640 1 -- 172.21.15.46:0/531932611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c0c1538a0 msgr2=0x7f9c0c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:49.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c10d01640 1 --2- 172.21.15.46:0/531932611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c0c1538a0 0x7f9c0c15d6f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0097c0 tx=0x7f9bfc01eb90 comp rx=0 tx=0).stop 2023-10-30T13:34:49.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c10d01640 1 -- 172.21.15.46:0/531932611 shutdown_connections 2023-10-30T13:34:49.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c10d01640 1 --2- 172.21.15.46:0/531932611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c0c1538a0 0x7f9c0c15d6f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c10d01640 1 -- 172.21.15.46:0/531932611 >> 172.21.15.46:0/531932611 conn(0x7f9c0c06bfe0 msgr2=0x7f9c0c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:49.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c10d01640 1 -- 172.21.15.46:0/531932611 shutdown_connections 2023-10-30T13:34:49.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c10d01640 1 -- 172.21.15.46:0/531932611 wait complete. 2023-10-30T13:34:49.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c10d01640 1 Processor -- start 2023-10-30T13:34:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.553+0000 7f9c10d01640 1 -- start start 2023-10-30T13:34:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c10d01640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0c0ca0e0 0x7f9c0c1590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c10d01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c0c0cf160 0x7f9c0c0b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:49.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c10d01640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c0c1538a0 0x7f9c0c1e5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c10d01640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0ce910 con 0x7f9c0c0cf160 2023-10-30T13:34:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c10d01640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0b36e0 con 0x7f9c0c1538a0 2023-10-30T13:34:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c10d01640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0ceb20 con 0x7f9c0c0ca0e0 2023-10-30T13:34:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c09d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c0c0cf160 0x7f9c0c0b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:49.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c09d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c0c0cf160 0x7f9c0c0b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47404/0 (socket says 172.21.15.46:47404) 2023-10-30T13:34:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c09d9b640 1 -- 172.21.15.46:0/210090674 learned_addr learned my addr 172.21.15.46:0/210090674 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c0a59c640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0c0ca0e0 0x7f9c0c1590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c0ad9d640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c0c1538a0 0x7f9c0c1e5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:49.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c09d9b640 1 -- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0c0ca0e0 msgr2=0x7f9c0c1590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c09d9b640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0c0ca0e0 0x7f9c0c1590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c09d9b640 1 -- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c0c1538a0 msgr2=0x7f9c0c1e5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c09d9b640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c0c1538a0 0x7f9c0c1e5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c09d9b640 1 -- 172.21.15.46:0/210090674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc009420 con 0x7f9c0c0cf160 2023-10-30T13:34:49.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c09d9b640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c0c0cf160 0x7f9c0c0b31a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9c0000c780 tx=0x7f9c0000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9bfaffd640 1 -- 172.21.15.46:0/210090674 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c00005ae0 con 0x7f9c0c0cf160 2023-10-30T13:34:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9bfaffd640 1 -- 172.21.15.46:0/210090674 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c00005c40 con 0x7f9c0c0cf160 2023-10-30T13:34:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c10d01640 1 -- 172.21.15.46:0/210090674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c0c0cee00 con 0x7f9c0c0cf160 2023-10-30T13:34:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9bfaffd640 1 -- 172.21.15.46:0/210090674 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c00007070 con 0x7f9c0c0cf160 2023-10-30T13:34:49.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c10d01640 1 -- 172.21.15.46:0/210090674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0c0ce430 con 0x7f9c0c0cf160 2023-10-30T13:34:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9c10d01640 1 -- 172.21.15.46:0/210090674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0c04e170 con 0x7f9c0c0cf160 2023-10-30T13:34:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.557+0000 7f9bfaffd640 1 -- 172.21.15.46:0/210090674 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9c000071d0 con 0x7f9c0c0cf160 2023-10-30T13:34:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.561+0000 7f9bfaffd640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9bec0664d0 0x7f9bec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.561+0000 7f9bfaffd640 1 -- 172.21.15.46:0/210090674 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f9c00083d20 con 0x7f9c0c0cf160 2023-10-30T13:34:49.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.561+0000 7f9c0a59c640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9bec0664d0 0x7f9bec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:49.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.561+0000 7f9c0a59c640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9bec0664d0 0x7f9bec068990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0b2000 tx=0x7f9bfc0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:49.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.565+0000 7f9bfaffd640 1 -- 172.21.15.46:0/210090674 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9c00051010 con 0x7f9c0c0cf160 2023-10-30T13:34:49.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.757+0000 7f9c10d01640 1 -- 172.21.15.46:0/210090674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c0c0b1dd0 con 0x7f9c0c0cf160 2023-10-30T13:34:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.757+0000 7f9bfaffd640 1 -- 172.21.15.46:0/210090674 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f9c00054630 con 0x7f9c0c0cf160 2023-10-30T13:34:49.763 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:34:49.763 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5721,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:34:05.920156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2023-10-30T13:34:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.757+0000 7f9c10d01640 1 -- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9bec0664d0 msgr2=0x7f9bec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.757+0000 7f9c10d01640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9bec0664d0 0x7f9bec068990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0b2000 tx=0x7f9bfc0aa070 comp rx=0 tx=0).stop 2023-10-30T13:34:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.757+0000 7f9c10d01640 1 -- 172.21.15.46:0/210090674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c0c0cf160 msgr2=0x7f9c0c0b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.757+0000 7f9c10d01640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c0c0cf160 0x7f9c0c0b31a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9c0000c780 tx=0x7f9c0000cc40 comp rx=0 tx=0).stop 2023-10-30T13:34:49.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.761+0000 7f9c10d01640 1 -- 172.21.15.46:0/210090674 shutdown_connections 2023-10-30T13:34:49.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.761+0000 7f9c10d01640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9bec0664d0 0x7f9bec068990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.761+0000 7f9c10d01640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c0c1538a0 0x7f9c0c1e5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.761+0000 7f9c10d01640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c0c0cf160 0x7f9c0c0b31a0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.761+0000 7f9c10d01640 1 --2- 172.21.15.46:0/210090674 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0c0ca0e0 0x7f9c0c1590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.761+0000 7f9c10d01640 1 -- 172.21.15.46:0/210090674 >> 172.21.15.46:0/210090674 conn(0x7f9c0c06bfe0 msgr2=0x7f9c0c1517e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:49.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.761+0000 7f9c10d01640 1 -- 172.21.15.46:0/210090674 shutdown_connections 2023-10-30T13:34:49.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.761+0000 7f9c10d01640 1 -- 172.21.15.46:0/210090674 wait complete. 2023-10-30T13:34:49.784 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2023-10-30T13:34:49.784 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T13:34:49.784 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:34:49.801 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T13:34:49.801 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171b348640 1 Processor -- start 2023-10-30T13:34:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171b348640 1 -- start start 2023-10-30T13:34:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171b348640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17141538f0 0x7f1714159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171b348640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f171415cd50 0x7f17140c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171b348640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f17140cefa0 0x7f17140cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171b348640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171415d300 con 0x7f17140cefa0 2023-10-30T13:34:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171b348640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171415d5e0 con 0x7f171415cd50 2023-10-30T13:34:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171b348640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171415d8c0 con 0x7f17141538f0 2023-10-30T13:34:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171ab47640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f17140cefa0 0x7f17140cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171ab47640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f17140cefa0 0x7f17140cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47412/0 (socket says 172.21.15.46:47412) 2023-10-30T13:34:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171ab47640 1 -- 172.21.15.46:0/1612496285 learned_addr learned my addr 172.21.15.46:0/1612496285 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f1719b45640 1 --2- 172.21.15.46:0/1612496285 >> v2:172.21.15.154:3300/0 conn(0x7f171415cd50 0x7f17140c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171a346640 1 --2- 172.21.15.46:0/1612496285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17141538f0 0x7f1714159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171ab47640 1 -- 172.21.15.46:0/1612496285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17141538f0 msgr2=0x7f1714159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171ab47640 1 --2- 172.21.15.46:0/1612496285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17141538f0 0x7f1714159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171ab47640 1 -- 172.21.15.46:0/1612496285 >> v2:172.21.15.154:3300/0 conn(0x7f171415cd50 msgr2=0x7f17140c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:49.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171ab47640 1 --2- 172.21.15.46:0/1612496285 >> v2:172.21.15.154:3300/0 conn(0x7f171415cd50 0x7f17140c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.941+0000 7f171ab47640 1 -- 172.21.15.46:0/1612496285 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171415dba0 con 0x7f17140cefa0 2023-10-30T13:34:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f171ab47640 1 --2- 172.21.15.46:0/1612496285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17140cefa0 0x7f17140cf3f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1714066410 tx=0x7f170c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343d68cb86612635 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f1719344640 1 -- 172.21.15.46:0/1612496285 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c0af400 con 0x7f17140cefa0 2023-10-30T13:34:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f1719344640 1 -- 172.21.15.46:0/1612496285 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170c0ab050 con 0x7f17140cefa0 2023-10-30T13:34:49.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f1719344640 1 -- 172.21.15.46:0/1612496285 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c0af780 con 0x7f17140cefa0 2023-10-30T13:34:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f171b348640 1 -- 172.21.15.46:0/1612496285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17140cefa0 msgr2=0x7f17140cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f171b348640 1 --2- 172.21.15.46:0/1612496285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17140cefa0 0x7f17140cf3f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1714066410 tx=0x7f170c01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f171b348640 1 -- 172.21.15.46:0/1612496285 shutdown_connections 2023-10-30T13:34:49.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f171b348640 1 --2- 172.21.15.46:0/1612496285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17140cefa0 0x7f17140cf3f0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f171b348640 1 --2- 172.21.15.46:0/1612496285 >> v2:172.21.15.154:3300/0 conn(0x7f171415cd50 0x7f17140c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f171b348640 1 --2- 172.21.15.46:0/1612496285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17141538f0 0x7f1714159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f171b348640 1 -- 172.21.15.46:0/1612496285 >> 172.21.15.46:0/1612496285 conn(0x7f171406a640 msgr2=0x7f1714065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f171b348640 1 -- 172.21.15.46:0/1612496285 shutdown_connections 2023-10-30T13:34:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f171b348640 1 -- 172.21.15.46:0/1612496285 wait complete. 2023-10-30T13:34:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f171b348640 1 Processor -- start 2023-10-30T13:34:49.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.945+0000 7f171b348640 1 -- start start 2023-10-30T13:34:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171b348640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17140cefa0 0x7f17141590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171b348640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17141538f0 0x7f1714140b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171b348640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171415cd50 0x7f1714147a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171b348640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1714142790 con 0x7f171415cd50 2023-10-30T13:34:49.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171b348640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17141f4080 con 0x7f17141538f0 2023-10-30T13:34:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171b348640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1714142a50 con 0x7f17140cefa0 2023-10-30T13:34:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171ab47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171415cd50 0x7f1714147a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171ab47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171415cd50 0x7f1714147a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47428/0 (socket says 172.21.15.46:47428) 2023-10-30T13:34:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171ab47640 1 -- 172.21.15.46:0/3027234484 learned_addr learned my addr 172.21.15.46:0/3027234484 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f1719b45640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17141538f0 0x7f1714140b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171a346640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17140cefa0 0x7f17141590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171ab47640 1 -- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17140cefa0 msgr2=0x7f17141590d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171ab47640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17140cefa0 0x7f17141590d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171ab47640 1 -- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17141538f0 msgr2=0x7f1714140b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171ab47640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17141538f0 0x7f1714140b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171ab47640 1 -- 172.21.15.46:0/3027234484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c009b30 con 0x7f171415cd50 2023-10-30T13:34:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171ab47640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171415cd50 0x7f1714147a00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f170c0b2040 tx=0x7f170c01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f170affd640 1 -- 172.21.15.46:0/3027234484 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c0afde0 con 0x7f171415cd50 2023-10-30T13:34:49.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f170affd640 1 -- 172.21.15.46:0/3027234484 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170c009510 con 0x7f171415cd50 2023-10-30T13:34:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171b348640 1 -- 172.21.15.46:0/3027234484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17141f32c0 con 0x7f171415cd50 2023-10-30T13:34:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f170affd640 1 -- 172.21.15.46:0/3027234484 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c0ba660 con 0x7f171415cd50 2023-10-30T13:34:49.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171b348640 1 -- 172.21.15.46:0/3027234484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17141f3880 con 0x7f171415cd50 2023-10-30T13:34:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.949+0000 7f171b348640 1 -- 172.21.15.46:0/3027234484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171404e170 con 0x7f171415cd50 2023-10-30T13:34:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.953+0000 7f170affd640 1 -- 172.21.15.46:0/3027234484 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f170c0b6040 con 0x7f171415cd50 2023-10-30T13:34:49.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.953+0000 7f170affd640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16f80664f0 0x7f16f80689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.953+0000 7f170affd640 1 -- 172.21.15.46:0/3027234484 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f170c122460 con 0x7f171415cd50 2023-10-30T13:34:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.953+0000 7f171a346640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16f80664f0 0x7f16f80689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.953+0000 7f171a346640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16f80664f0 0x7f16f80689b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f171000ba30 tx=0x7f1710008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:49.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:49.957+0000 7f170affd640 1 -- 172.21.15.46:0/3027234484 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f170c0ef770 con 0x7f171415cd50 2023-10-30T13:34:50.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 -- 172.21.15.46:0/3027234484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17140b4f70 con 0x7f171415cd50 2023-10-30T13:34:50.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f170affd640 1 -- 172.21.15.46:0/3027234484 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f170c0f2d90 con 0x7f171415cd50 2023-10-30T13:34:50.154 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:34:50.154 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5721,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:34:05.920156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:50.155 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2023-10-30T13:34:50.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 -- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16f80664f0 msgr2=0x7f16f80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:50.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16f80664f0 0x7f16f80689b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f171000ba30 tx=0x7f1710008040 comp rx=0 tx=0).stop 2023-10-30T13:34:50.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 -- 172.21.15.46:0/3027234484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171415cd50 msgr2=0x7f1714147a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:50.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171415cd50 0x7f1714147a00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f170c0b2040 tx=0x7f170c01bf60 comp rx=0 tx=0).stop 2023-10-30T13:34:50.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 -- 172.21.15.46:0/3027234484 shutdown_connections 2023-10-30T13:34:50.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16f80664f0 0x7f16f80689b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:50.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f171415cd50 0x7f1714147a00 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:50.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17141538f0 0x7f1714140b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 --2- 172.21.15.46:0/3027234484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17140cefa0 0x7f17141590d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 -- 172.21.15.46:0/3027234484 >> 172.21.15.46:0/3027234484 conn(0x7f171406a640 msgr2=0x7f17140caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 -- 172.21.15.46:0/3027234484 shutdown_connections 2023-10-30T13:34:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:50.149+0000 7f171b348640 1 -- 172.21.15.46:0/3027234484 wait complete. 2023-10-30T13:34:50.176 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:34:52.177 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:52.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff19166b640 1 Processor -- start 2023-10-30T13:34:52.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff19166b640 1 -- start start 2023-10-30T13:34:52.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff19166b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff18c069e90 0x7ff18c15a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:52.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff19166b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff18c06cb80 0x7ff18c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:52.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff19166b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c06d530 con 0x7ff18c06a270 2023-10-30T13:34:52.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff19166b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18c0c8570 con 0x7ff18c06cb80 2023-10-30T13:34:52.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff19166b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18c0c6c10 con 0x7ff18c069e90 2023-10-30T13:34:52.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff18b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff18c06cb80 0x7ff18c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:52.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff183fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff18c06a270 0x7ff18c0c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32918/0 (socket says 172.21.15.46:32918) 2023-10-30T13:34:52.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff183fff640 1 -- 172.21.15.46:0/407430742 learned_addr learned my addr 172.21.15.46:0/407430742 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:34:52.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff18affd640 1 --2- 172.21.15.46:0/407430742 >> v2:172.21.15.154:3300/0 conn(0x7ff18c069e90 0x7ff18c15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff18b7fe640 1 -- 172.21.15.46:0/407430742 >> v2:172.21.15.154:3300/0 conn(0x7ff18c069e90 msgr2=0x7ff18c15a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff18b7fe640 1 --2- 172.21.15.46:0/407430742 >> v2:172.21.15.154:3300/0 conn(0x7ff18c069e90 0x7ff18c15a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff18b7fe640 1 -- 172.21.15.46:0/407430742 >> v1:172.21.15.46:6789/0 conn(0x7ff18c06a270 legacy=0x7ff18c0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff18b7fe640 1 -- 172.21.15.46:0/407430742 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c0c6ef0 con 0x7ff18c06cb80 2023-10-30T13:34:52.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.329+0000 7ff18a7fc640 1 -- 172.21.15.46:0/407430742 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447765471 0 0) 0x7ff18c06d530 con 0x7ff18c06a270 2023-10-30T13:34:52.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff18b7fe640 1 --2- 172.21.15.46:0/407430742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18c06cb80 0x7ff18c06cff0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff18c06ed20 tx=0x7ff18401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eacefc57caf97ac8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:52.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff18a7fc640 1 -- 172.21.15.46:0/407430742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1840a58c0 con 0x7ff18c06cb80 2023-10-30T13:34:52.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff18a7fc640 1 -- 172.21.15.46:0/407430742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1840a5a20 con 0x7ff18c06cb80 2023-10-30T13:34:52.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff18a7fc640 1 -- 172.21.15.46:0/407430742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1840a5db0 con 0x7ff18c06cb80 2023-10-30T13:34:52.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff19166b640 1 -- 172.21.15.46:0/407430742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18c06cb80 msgr2=0x7ff18c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:52.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff19166b640 1 --2- 172.21.15.46:0/407430742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18c06cb80 0x7ff18c06cff0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff18c06ed20 tx=0x7ff18401c910 comp rx=0 tx=0).stop 2023-10-30T13:34:52.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff19166b640 1 -- 172.21.15.46:0/407430742 shutdown_connections 2023-10-30T13:34:52.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff19166b640 1 --2- 172.21.15.46:0/407430742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18c06cb80 0x7ff18c06cff0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff19166b640 1 --2- 172.21.15.46:0/407430742 >> v2:172.21.15.154:3300/0 conn(0x7ff18c069e90 0x7ff18c15a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff19166b640 1 -- 172.21.15.46:0/407430742 >> 172.21.15.46:0/407430742 conn(0x7ff18c06dfd0 msgr2=0x7ff18c06e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:52.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff19166b640 1 -- 172.21.15.46:0/407430742 shutdown_connections 2023-10-30T13:34:52.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff19166b640 1 -- 172.21.15.46:0/407430742 wait complete. 2023-10-30T13:34:52.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff19166b640 1 Processor -- start 2023-10-30T13:34:52.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.333+0000 7ff19166b640 1 -- start start 2023-10-30T13:34:52.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff19166b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18c069e90 0x7ff18c145a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:52.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff19166b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff18c06a270 0x7ff18c140b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:52.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff19166b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18c06cb80 0x7ff18c14df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:52.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff19166b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c157f30 con 0x7ff18c06cb80 2023-10-30T13:34:52.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff19166b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c146090 con 0x7ff18c06a270 2023-10-30T13:34:52.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff19166b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c14d520 con 0x7ff18c069e90 2023-10-30T13:34:52.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff18b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18c06cb80 0x7ff18c14df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:52.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff18b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18c06cb80 0x7ff18c14df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47452/0 (socket says 172.21.15.46:47452) 2023-10-30T13:34:52.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff18b7fe640 1 -- 172.21.15.46:0/2270373678 learned_addr learned my addr 172.21.15.46:0/2270373678 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:52.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff18affd640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18c069e90 0x7ff18c145a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:52.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff183fff640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff18c06a270 0x7ff18c140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:52.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff18b7fe640 1 -- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18c069e90 msgr2=0x7ff18c145a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:52.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff18b7fe640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18c069e90 0x7ff18c145a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff18b7fe640 1 -- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff18c06a270 msgr2=0x7ff18c140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:52.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff18b7fe640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff18c06a270 0x7ff18c140b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff18b7fe640 1 -- 172.21.15.46:0/2270373678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff184009a70 con 0x7ff18c06cb80 2023-10-30T13:34:52.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff183fff640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff18c06a270 0x7ff18c140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:52.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff18b7fe640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18c06cb80 0x7ff18c14df70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff184019520 tx=0x7ff1840196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:52.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff1837fe640 1 -- 172.21.15.46:0/2270373678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1840b9860 con 0x7ff18c06cb80 2023-10-30T13:34:52.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff1837fe640 1 -- 172.21.15.46:0/2270373678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1840b99c0 con 0x7ff18c06cb80 2023-10-30T13:34:52.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff1837fe640 1 -- 172.21.15.46:0/2270373678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1840b9d70 con 0x7ff18c06cb80 2023-10-30T13:34:52.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff19166b640 1 -- 172.21.15.46:0/2270373678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18c1465e0 con 0x7ff18c06cb80 2023-10-30T13:34:52.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff19166b640 1 -- 172.21.15.46:0/2270373678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18c14d770 con 0x7ff18c06cb80 2023-10-30T13:34:52.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff19166b640 1 -- 172.21.15.46:0/2270373678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18c146b00 con 0x7ff18c06cb80 2023-10-30T13:34:52.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.337+0000 7ff1837fe640 1 -- 172.21.15.46:0/2270373678 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1840c4460 con 0x7ff18c06cb80 2023-10-30T13:34:52.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.341+0000 7ff1837fe640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff16c0664f0 0x7ff16c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:52.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.341+0000 7ff1837fe640 1 -- 172.21.15.46:0/2270373678 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7ff18411f3a0 con 0x7ff18c06cb80 2023-10-30T13:34:52.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.341+0000 7ff18affd640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff16c0664f0 0x7ff16c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:52.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.341+0000 7ff18affd640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff16c0664f0 0x7ff16c0689b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff18c1573d0 tx=0x7ff17c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:52.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.345+0000 7ff1837fe640 1 -- 172.21.15.46:0/2270373678 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1840efff0 con 0x7ff18c06cb80 2023-10-30T13:34:52.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 -- 172.21.15.46:0/2270373678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff18c06a650 con 0x7ff18c06cb80 2023-10-30T13:34:52.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff1837fe640 1 -- 172.21.15.46:0/2270373678 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6529 (secure 0 0 0) 0x7ff18c06a650 con 0x7ff18c06cb80 2023-10-30T13:34:52.545 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:34:52.546 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5721,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:34:05.920156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:52.546 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2023-10-30T13:34:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 -- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff16c0664f0 msgr2=0x7ff16c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:52.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff16c0664f0 0x7ff16c0689b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff18c1573d0 tx=0x7ff17c009190 comp rx=0 tx=0).stop 2023-10-30T13:34:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 -- 172.21.15.46:0/2270373678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18c06cb80 msgr2=0x7ff18c14df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18c06cb80 0x7ff18c14df70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff184019520 tx=0x7ff1840196b0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 -- 172.21.15.46:0/2270373678 shutdown_connections 2023-10-30T13:34:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff16c0664f0 0x7ff16c0689b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18c06cb80 0x7ff18c14df70 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff18c06a270 0x7ff18c140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 --2- 172.21.15.46:0/2270373678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18c069e90 0x7ff18c145a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 -- 172.21.15.46:0/2270373678 >> 172.21.15.46:0/2270373678 conn(0x7ff18c06dfd0 msgr2=0x7ff18c070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 -- 172.21.15.46:0/2270373678 shutdown_connections 2023-10-30T13:34:52.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:52.541+0000 7ff19166b640 1 -- 172.21.15.46:0/2270373678 wait complete. 2023-10-30T13:34:52.566 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:34:54.567 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:54.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.705+0000 7fa33a51d640 1 Processor -- start 2023-10-30T13:34:54.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.705+0000 7fa33a51d640 1 -- start start 2023-10-30T13:34:54.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.705+0000 7fa33a51d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3340c6bd0 0x7fa33406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:54.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.705+0000 7fa33a51d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa334071110 0x7fa334158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:54.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.705+0000 7fa33a51d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa334066cf0 con 0x7fa33406f7a0 2023-10-30T13:34:54.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.705+0000 7fa33a51d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3340700c0 con 0x7fa3340c6bd0 2023-10-30T13:34:54.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.705+0000 7fa33a51d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3340c70d0 con 0x7fa334071110 2023-10-30T13:34:54.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.705+0000 7fa3337fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3340c6bd0 0x7fa33406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:54.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.705+0000 7fa3337fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3340c6bd0 0x7fa33406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47462/0 (socket says 172.21.15.46:47462) 2023-10-30T13:34:54.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.705+0000 7fa3337fe640 1 -- 172.21.15.46:0/172002688 learned_addr learned my addr 172.21.15.46:0/172002688 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:54.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.705+0000 7fa338aba640 1 --2- 172.21.15.46:0/172002688 >> v2:172.21.15.154:3300/0 conn(0x7fa334071110 0x7fa334158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:54.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa3337fe640 1 -- 172.21.15.46:0/172002688 >> v2:172.21.15.154:3300/0 conn(0x7fa334071110 msgr2=0x7fa334158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:54.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa3337fe640 1 --2- 172.21.15.46:0/172002688 >> v2:172.21.15.154:3300/0 conn(0x7fa334071110 0x7fa334158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:54.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa3337fe640 1 -- 172.21.15.46:0/172002688 >> v1:172.21.15.154:6789/0 conn(0x7fa33406f7a0 legacy=0x7fa334069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:54.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa3337fe640 1 -- 172.21.15.46:0/172002688 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3340c7390 con 0x7fa3340c6bd0 2023-10-30T13:34:54.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa3337fe640 1 --2- 172.21.15.46:0/172002688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3340c6bd0 0x7fa33406fb80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa33406cdd0 tx=0x7fa32801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c6ff1c98150d77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:54.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa332ffd640 1 -- 172.21.15.46:0/172002688 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3280a89b0 con 0x7fa3340c6bd0 2023-10-30T13:34:54.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa332ffd640 1 -- 172.21.15.46:0/172002688 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3280a8b10 con 0x7fa3340c6bd0 2023-10-30T13:34:54.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa332ffd640 1 -- 172.21.15.46:0/172002688 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3280a8ea0 con 0x7fa3340c6bd0 2023-10-30T13:34:54.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa33a51d640 1 -- 172.21.15.46:0/172002688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3340c6bd0 msgr2=0x7fa33406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:54.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa33a51d640 1 --2- 172.21.15.46:0/172002688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3340c6bd0 0x7fa33406fb80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa33406cdd0 tx=0x7fa32801ecb0 comp rx=0 tx=0).stop 2023-10-30T13:34:54.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa33a51d640 1 -- 172.21.15.46:0/172002688 shutdown_connections 2023-10-30T13:34:54.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa33a51d640 1 --2- 172.21.15.46:0/172002688 >> v2:172.21.15.154:3300/0 conn(0x7fa334071110 0x7fa334158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:54.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa33a51d640 1 --2- 172.21.15.46:0/172002688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3340c6bd0 0x7fa33406fb80 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:54.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa33a51d640 1 -- 172.21.15.46:0/172002688 >> 172.21.15.46:0/172002688 conn(0x7fa3340c7910 msgr2=0x7fa3340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:54.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa33a51d640 1 -- 172.21.15.46:0/172002688 shutdown_connections 2023-10-30T13:34:54.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.709+0000 7fa33a51d640 1 -- 172.21.15.46:0/172002688 wait complete. 2023-10-30T13:34:54.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa33a51d640 1 Processor -- start 2023-10-30T13:34:54.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa33a51d640 1 -- start start 2023-10-30T13:34:54.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa33a51d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406f7a0 0x7fa3341f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:54.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa33a51d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa334071110 0x7fa3341ef5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa33a51d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3340c6bd0 0x7fa33414f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa33a51d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3341eebe0 con 0x7fa33406f7a0 2023-10-30T13:34:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa33a51d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3340c1f40 con 0x7fa334071110 2023-10-30T13:34:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa33a51d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3341f30a0 con 0x7fa3340c6bd0 2023-10-30T13:34:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa333fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406f7a0 0x7fa3341f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa333fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406f7a0 0x7fa3341f2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47476/0 (socket says 172.21.15.46:47476) 2023-10-30T13:34:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa333fff640 1 -- 172.21.15.46:0/2615282490 learned_addr learned my addr 172.21.15.46:0/2615282490 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa3337fe640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa334071110 0x7fa3341ef5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa338aba640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3340c6bd0 0x7fa33414f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa333fff640 1 -- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3340c6bd0 msgr2=0x7fa33414f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa333fff640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3340c6bd0 0x7fa33414f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa333fff640 1 -- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa334071110 msgr2=0x7fa3341ef5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa333fff640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa334071110 0x7fa3341ef5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa333fff640 1 -- 172.21.15.46:0/2615282490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3280094d0 con 0x7fa33406f7a0 2023-10-30T13:34:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa3337fe640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa334071110 0x7fa3341ef5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa333fff640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406f7a0 0x7fa3341f2b60 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa32400b410 tx=0x7fa32400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa330ff9640 1 -- 172.21.15.46:0/2615282490 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa324004020 con 0x7fa33406f7a0 2023-10-30T13:34:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa330ff9640 1 -- 172.21.15.46:0/2615282490 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa324004180 con 0x7fa33406f7a0 2023-10-30T13:34:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa330ff9640 1 -- 172.21.15.46:0/2615282490 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa324011630 con 0x7fa33406f7a0 2023-10-30T13:34:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.713+0000 7fa33a51d640 1 -- 172.21.15.46:0/2615282490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3341efb30 con 0x7fa33406f7a0 2023-10-30T13:34:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.717+0000 7fa33a51d640 1 -- 172.21.15.46:0/2615282490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3340c1980 con 0x7fa33406f7a0 2023-10-30T13:34:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.717+0000 7fa33a51d640 1 -- 172.21.15.46:0/2615282490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33404e170 con 0x7fa33406f7a0 2023-10-30T13:34:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.717+0000 7fa330ff9640 1 -- 172.21.15.46:0/2615282490 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa324011790 con 0x7fa33406f7a0 2023-10-30T13:34:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.717+0000 7fa330ff9640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3200664a0 0x7fa320068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.717+0000 7fa330ff9640 1 -- 172.21.15.46:0/2615282490 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fa324083e60 con 0x7fa33406f7a0 2023-10-30T13:34:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.717+0000 7fa3337fe640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3200664a0 0x7fa320068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.721+0000 7fa3337fe640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3200664a0 0x7fa320068960 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa3280b2000 tx=0x7fa3280aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.721+0000 7fa330ff9640 1 -- 172.21.15.46:0/2615282490 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa324051030 con 0x7fa33406f7a0 2023-10-30T13:34:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.909+0000 7fa33a51d640 1 -- 172.21.15.46:0/2615282490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3340c9830 con 0x7fa33406f7a0 2023-10-30T13:34:54.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa330ff9640 1 -- 172.21.15.46:0/2615282490 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6529 (secure 0 0 0) 0x7fa324015090 con 0x7fa33406f7a0 2023-10-30T13:34:54.917 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:34:54.917 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5721,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:34:05.920156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:54.918 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2023-10-30T13:34:54.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa33a51d640 1 -- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3200664a0 msgr2=0x7fa320068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:54.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa33a51d640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3200664a0 0x7fa320068960 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa3280b2000 tx=0x7fa3280aa070 comp rx=0 tx=0).stop 2023-10-30T13:34:54.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa33a51d640 1 -- 172.21.15.46:0/2615282490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406f7a0 msgr2=0x7fa3341f2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:54.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa33a51d640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406f7a0 0x7fa3341f2b60 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa32400b410 tx=0x7fa32400b8d0 comp rx=0 tx=0).stop 2023-10-30T13:34:54.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa33a51d640 1 -- 172.21.15.46:0/2615282490 shutdown_connections 2023-10-30T13:34:54.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa33a51d640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3200664a0 0x7fa320068960 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:54.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa33a51d640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3340c6bd0 0x7fa33414f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:54.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa33a51d640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa334071110 0x7fa3341ef5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:54.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa33a51d640 1 --2- 172.21.15.46:0/2615282490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa33406f7a0 0x7fa3341f2b60 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:54.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa33a51d640 1 -- 172.21.15.46:0/2615282490 >> 172.21.15.46:0/2615282490 conn(0x7fa3340c7910 msgr2=0x7fa3340c8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:54.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa33a51d640 1 -- 172.21.15.46:0/2615282490 shutdown_connections 2023-10-30T13:34:54.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:54.913+0000 7fa33a51d640 1 -- 172.21.15.46:0/2615282490 wait complete. 2023-10-30T13:34:54.938 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:34:56.939 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:57.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.089+0000 7ff134ef7640 1 Processor -- start 2023-10-30T13:34:57.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.089+0000 7ff134ef7640 1 -- start start 2023-10-30T13:34:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff134ef7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff1300c6fe0 0x7ff130071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff134ef7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1300c5f60 0x7ff1300cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff134ef7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1300700b0 con 0x7ff130153bf0 2023-10-30T13:34:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff134ef7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1300c6810 con 0x7ff1300c6fe0 2023-10-30T13:34:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff134ef7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff130159900 con 0x7ff1300c5f60 2023-10-30T13:34:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff12e59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff130153bf0 0x7ff1300c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43084/0 (socket says 172.21.15.46:43084) 2023-10-30T13:34:57.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff12e59c640 1 -- 172.21.15.46:0/276278522 learned_addr learned my addr 172.21.15.46:0/276278522 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:34:57.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff127fff640 1 --2- 172.21.15.46:0/276278522 >> v2:172.21.15.46:3300/0 conn(0x7ff1300c6fe0 0x7ff130071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:57.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff12ed9d640 1 --2- 172.21.15.46:0/276278522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1300c5f60 0x7ff1300cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:57.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff12dd9b640 1 -- 172.21.15.46:0/276278522 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741865535 0 0) 0x7ff1300700b0 con 0x7ff130153bf0 2023-10-30T13:34:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff12dd9b640 1 -- 172.21.15.46:0/276278522 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff11c003530 con 0x7ff130153bf0 2023-10-30T13:34:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff127fff640 1 -- 172.21.15.46:0/276278522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1300c5f60 msgr2=0x7ff1300cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff127fff640 1 --2- 172.21.15.46:0/276278522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1300c5f60 0x7ff1300cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff127fff640 1 -- 172.21.15.46:0/276278522 >> v1:172.21.15.46:6789/0 conn(0x7ff130153bf0 legacy=0x7ff1300c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.093+0000 7ff127fff640 1 -- 172.21.15.46:0/276278522 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1300c6530 con 0x7ff1300c6fe0 2023-10-30T13:34:57.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff127fff640 1 --2- 172.21.15.46:0/276278522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1300c6fe0 0x7ff130071260 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff1180097c0 tx=0x7ff11801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463fd551f89dee5c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:57.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff12dd9b640 1 -- 172.21.15.46:0/276278522 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1180a8970 con 0x7ff1300c6fe0 2023-10-30T13:34:57.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff12dd9b640 1 -- 172.21.15.46:0/276278522 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1180a8ad0 con 0x7ff1300c6fe0 2023-10-30T13:34:57.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff12dd9b640 1 -- 172.21.15.46:0/276278522 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1180a8e60 con 0x7ff1300c6fe0 2023-10-30T13:34:57.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff134ef7640 1 -- 172.21.15.46:0/276278522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1300c6fe0 msgr2=0x7ff130071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff134ef7640 1 --2- 172.21.15.46:0/276278522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1300c6fe0 0x7ff130071260 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff1180097c0 tx=0x7ff11801ec70 comp rx=0 tx=0).stop 2023-10-30T13:34:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff134ef7640 1 -- 172.21.15.46:0/276278522 shutdown_connections 2023-10-30T13:34:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff134ef7640 1 --2- 172.21.15.46:0/276278522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1300c5f60 0x7ff1300cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff134ef7640 1 --2- 172.21.15.46:0/276278522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1300c6fe0 0x7ff130071260 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff134ef7640 1 -- 172.21.15.46:0/276278522 >> 172.21.15.46:0/276278522 conn(0x7ff13006c6e0 msgr2=0x7ff130069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:57.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff134ef7640 1 -- 172.21.15.46:0/276278522 shutdown_connections 2023-10-30T13:34:57.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff134ef7640 1 -- 172.21.15.46:0/276278522 wait complete. 2023-10-30T13:34:57.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff134ef7640 1 Processor -- start 2023-10-30T13:34:57.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.097+0000 7ff134ef7640 1 -- start start 2023-10-30T13:34:57.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff134ef7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1300c5f60 0x7ff130154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:57.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff134ef7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1300c6fe0 0x7ff130156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:57.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff134ef7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff130153bf0 0x7ff13014c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:57.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff134ef7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff130156940 con 0x7ff130153bf0 2023-10-30T13:34:57.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff134ef7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1301559e0 con 0x7ff1300c6fe0 2023-10-30T13:34:57.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff134ef7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff130155cc0 con 0x7ff1300c5f60 2023-10-30T13:34:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff12ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff130153bf0 0x7ff13014c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff12ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff130153bf0 0x7ff13014c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36028/0 (socket says 172.21.15.46:36028) 2023-10-30T13:34:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff12ed9d640 1 -- 172.21.15.46:0/3883742194 learned_addr learned my addr 172.21.15.46:0/3883742194 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff12e59c640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1300c5f60 0x7ff130154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff127fff640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1300c6fe0 0x7ff130156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:57.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff12ed9d640 1 -- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1300c5f60 msgr2=0x7ff130154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:57.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff12ed9d640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1300c5f60 0x7ff130154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:57.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff12ed9d640 1 -- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1300c6fe0 msgr2=0x7ff130156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:57.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff12ed9d640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1300c6fe0 0x7ff130156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:57.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff12ed9d640 1 -- 172.21.15.46:0/3883742194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff118009420 con 0x7ff130153bf0 2023-10-30T13:34:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff12ed9d640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff130153bf0 0x7ff13014c120 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff12800efc0 tx=0x7ff12800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff126ffd640 1 -- 172.21.15.46:0/3883742194 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff128009ba0 con 0x7ff130153bf0 2023-10-30T13:34:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff126ffd640 1 -- 172.21.15.46:0/3883742194 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12800f040 con 0x7ff130153bf0 2023-10-30T13:34:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff126ffd640 1 -- 172.21.15.46:0/3883742194 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1280146c0 con 0x7ff130153bf0 2023-10-30T13:34:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff134ef7640 1 -- 172.21.15.46:0/3883742194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff130155fa0 con 0x7ff130153bf0 2023-10-30T13:34:57.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff134ef7640 1 -- 172.21.15.46:0/3883742194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff130156560 con 0x7ff130153bf0 2023-10-30T13:34:57.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.101+0000 7ff134ef7640 1 -- 172.21.15.46:0/3883742194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13014ddc0 con 0x7ff130153bf0 2023-10-30T13:34:57.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.105+0000 7ff126ffd640 1 -- 172.21.15.46:0/3883742194 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff128014820 con 0x7ff130153bf0 2023-10-30T13:34:57.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.105+0000 7ff126ffd640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1100664a0 0x7ff110068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:57.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.105+0000 7ff126ffd640 1 -- 172.21.15.46:0/3883742194 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7ff128084a70 con 0x7ff130153bf0 2023-10-30T13:34:57.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.105+0000 7ff12e59c640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1100664a0 0x7ff110068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:57.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.105+0000 7ff12e59c640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1100664a0 0x7ff110068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff1200011f0 tx=0x7ff120007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:57.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.109+0000 7ff126ffd640 1 -- 172.21.15.46:0/3883742194 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff13014ddc0 con 0x7ff130153bf0 2023-10-30T13:34:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.297+0000 7ff134ef7640 1 -- 172.21.15.46:0/3883742194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff13014ddc0 con 0x7ff130153bf0 2023-10-30T13:34:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.297+0000 7ff126ffd640 1 -- 172.21.15.46:0/3883742194 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6529 (secure 0 0 0) 0x7ff13014ddc0 con 0x7ff130153bf0 2023-10-30T13:34:57.303 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:34:57.303 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5721,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:34:05.920156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:57.304 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2023-10-30T13:34:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.301+0000 7ff134ef7640 1 -- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1100664a0 msgr2=0x7ff110068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.301+0000 7ff134ef7640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1100664a0 0x7ff110068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff1200011f0 tx=0x7ff120007d40 comp rx=0 tx=0).stop 2023-10-30T13:34:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.301+0000 7ff134ef7640 1 -- 172.21.15.46:0/3883742194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff130153bf0 msgr2=0x7ff13014c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.301+0000 7ff134ef7640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff130153bf0 0x7ff13014c120 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff12800efc0 tx=0x7ff12800c490 comp rx=0 tx=0).stop 2023-10-30T13:34:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.301+0000 7ff134ef7640 1 -- 172.21.15.46:0/3883742194 shutdown_connections 2023-10-30T13:34:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.301+0000 7ff134ef7640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1100664a0 0x7ff110068960 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.301+0000 7ff134ef7640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff130153bf0 0x7ff13014c120 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.301+0000 7ff134ef7640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1300c6fe0 0x7ff130156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.301+0000 7ff134ef7640 1 --2- 172.21.15.46:0/3883742194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1300c5f60 0x7ff130154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.301+0000 7ff134ef7640 1 -- 172.21.15.46:0/3883742194 >> 172.21.15.46:0/3883742194 conn(0x7ff13006c6e0 msgr2=0x7ff130071000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.301+0000 7ff134ef7640 1 -- 172.21.15.46:0/3883742194 shutdown_connections 2023-10-30T13:34:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:57.301+0000 7ff134ef7640 1 -- 172.21.15.46:0/3883742194 wait complete. 2023-10-30T13:34:57.324 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:34:59.325 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:59.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.473+0000 7f4e0452a640 1 Processor -- start 2023-10-30T13:34:59.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.473+0000 7f4e0452a640 1 -- start start 2023-10-30T13:34:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.473+0000 7f4e0452a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4dfc1539a0 0x7f4dfc15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.473+0000 7f4e0452a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4dfc069e90 0x7f4dfc06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.473+0000 7f4e0452a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dfc15a8c0 con 0x7f4dfc070a00 2023-10-30T13:34:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.473+0000 7f4e0452a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dfc06fbb0 con 0x7f4dfc069e90 2023-10-30T13:34:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.473+0000 7f4e0452a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dfc153d80 con 0x7f4dfc1539a0 2023-10-30T13:34:59.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e02ac7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4dfc069e90 0x7f4dfc06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:59.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e01ac5640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4dfc070a00 0x7f4dfc070e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43092/0 (socket says 172.21.15.46:43092) 2023-10-30T13:34:59.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e01ac5640 1 -- 172.21.15.46:0/3849524662 learned_addr learned my addr 172.21.15.46:0/3849524662 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:34:59.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e022c6640 1 --2- 172.21.15.46:0/3849524662 >> v2:172.21.15.154:3300/0 conn(0x7f4dfc1539a0 0x7f4dfc15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e012c4640 1 -- 172.21.15.46:0/3849524662 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720530963 0 0) 0x7f4dfc15a8c0 con 0x7f4dfc070a00 2023-10-30T13:34:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e012c4640 1 -- 172.21.15.46:0/3849524662 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de8003530 con 0x7f4dfc070a00 2023-10-30T13:34:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e02ac7640 1 -- 172.21.15.46:0/3849524662 >> v2:172.21.15.154:3300/0 conn(0x7f4dfc1539a0 msgr2=0x7f4dfc15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e02ac7640 1 --2- 172.21.15.46:0/3849524662 >> v2:172.21.15.154:3300/0 conn(0x7f4dfc1539a0 0x7f4dfc15a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e02ac7640 1 -- 172.21.15.46:0/3849524662 >> v1:172.21.15.46:6789/0 conn(0x7f4dfc070a00 legacy=0x7f4dfc070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e02ac7640 1 -- 172.21.15.46:0/3849524662 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc0c5c50 con 0x7f4dfc069e90 2023-10-30T13:34:59.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e02ac7640 1 --2- 172.21.15.46:0/3849524662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dfc069e90 0x7f4dfc06a300 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0c8950 tx=0x7f4df401c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cc27bba35a1207 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:59.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e012c4640 1 -- 172.21.15.46:0/3849524662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df40a5820 con 0x7f4dfc069e90 2023-10-30T13:34:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e012c4640 1 -- 172.21.15.46:0/3849524662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df40a5980 con 0x7f4dfc069e90 2023-10-30T13:34:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.477+0000 7f4e012c4640 1 -- 172.21.15.46:0/3849524662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df40a5d10 con 0x7f4dfc069e90 2023-10-30T13:34:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 -- 172.21.15.46:0/3849524662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dfc069e90 msgr2=0x7f4dfc06a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 --2- 172.21.15.46:0/3849524662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dfc069e90 0x7f4dfc06a300 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0c8950 tx=0x7f4df401c850 comp rx=0 tx=0).stop 2023-10-30T13:34:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 -- 172.21.15.46:0/3849524662 shutdown_connections 2023-10-30T13:34:59.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 --2- 172.21.15.46:0/3849524662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dfc069e90 0x7f4dfc06a300 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:59.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 --2- 172.21.15.46:0/3849524662 >> v2:172.21.15.154:3300/0 conn(0x7f4dfc1539a0 0x7f4dfc15a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:59.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 -- 172.21.15.46:0/3849524662 >> 172.21.15.46:0/3849524662 conn(0x7f4dfc0c7910 msgr2=0x7f4dfc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:59.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 -- 172.21.15.46:0/3849524662 shutdown_connections 2023-10-30T13:34:59.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 -- 172.21.15.46:0/3849524662 wait complete. 2023-10-30T13:34:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 Processor -- start 2023-10-30T13:34:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 -- start start 2023-10-30T13:34:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dfc069e90 0x7f4dfc0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dfc070a00 0x7f4dfc1490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4dfc1539a0 0x7f4dfc149620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc1440c0 con 0x7f4dfc070a00 2023-10-30T13:34:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc0c1da0 con 0x7f4dfc069e90 2023-10-30T13:34:59.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e0452a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc1442f0 con 0x7f4dfc1539a0 2023-10-30T13:34:59.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e01ac5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dfc070a00 0x7f4dfc1490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:59.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e01ac5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dfc070a00 0x7f4dfc1490e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36056/0 (socket says 172.21.15.46:36056) 2023-10-30T13:34:59.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e01ac5640 1 -- 172.21.15.46:0/416948019 learned_addr learned my addr 172.21.15.46:0/416948019 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:34:59.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e02ac7640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4dfc1539a0 0x7f4dfc149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:59.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.481+0000 7f4e022c6640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dfc069e90 0x7f4dfc0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:59.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4e01ac5640 1 -- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4dfc1539a0 msgr2=0x7f4dfc149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:59.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4e01ac5640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4dfc1539a0 0x7f4dfc149620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:59.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4e01ac5640 1 -- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dfc069e90 msgr2=0x7f4dfc0c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:59.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4e01ac5640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dfc069e90 0x7f4dfc0c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:59.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4e01ac5640 1 -- 172.21.15.46:0/416948019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df4009a70 con 0x7f4dfc070a00 2023-10-30T13:34:59.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4e01ac5640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dfc070a00 0x7f4dfc1490e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00c4d0 tx=0x7f4dec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:59.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4df2ffd640 1 -- 172.21.15.46:0/416948019 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec005cf0 con 0x7f4dfc070a00 2023-10-30T13:34:59.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4df2ffd640 1 -- 172.21.15.46:0/416948019 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dec005e50 con 0x7f4dfc070a00 2023-10-30T13:34:59.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4df2ffd640 1 -- 172.21.15.46:0/416948019 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec007270 con 0x7f4dfc070a00 2023-10-30T13:34:59.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4e0452a640 1 -- 172.21.15.46:0/416948019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dfc163d60 con 0x7f4dfc070a00 2023-10-30T13:34:59.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4e0452a640 1 -- 172.21.15.46:0/416948019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dfc0c17e0 con 0x7f4dfc070a00 2023-10-30T13:34:59.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4e0452a640 1 -- 172.21.15.46:0/416948019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dfc04e170 con 0x7f4dfc070a00 2023-10-30T13:34:59.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4df2ffd640 1 -- 172.21.15.46:0/416948019 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4dec0073d0 con 0x7f4dfc070a00 2023-10-30T13:34:59.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4df2ffd640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4de00664a0 0x7f4de0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:59.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4df2ffd640 1 -- 172.21.15.46:0/416948019 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f4dec0835a0 con 0x7f4dfc070a00 2023-10-30T13:34:59.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.485+0000 7f4e022c6640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4de00664a0 0x7f4de0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:59.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.489+0000 7f4e022c6640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4de00664a0 0x7f4de0068960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009ac0 tx=0x7f4df8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:59.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.489+0000 7f4df2ffd640 1 -- 172.21.15.46:0/416948019 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4dec050820 con 0x7f4dfc070a00 2023-10-30T13:34:59.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.681+0000 7f4e0452a640 1 -- 172.21.15.46:0/416948019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dfc152530 con 0x7f4dfc070a00 2023-10-30T13:34:59.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.681+0000 7f4df2ffd640 1 -- 172.21.15.46:0/416948019 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f4dec053e40 con 0x7f4dfc070a00 2023-10-30T13:34:59.688 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:34:59.689 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5721,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:34:05.920156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:59.689 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2023-10-30T13:34:59.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.685+0000 7f4e0452a640 1 -- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4de00664a0 msgr2=0x7f4de0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:59.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.685+0000 7f4e0452a640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4de00664a0 0x7f4de0068960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009ac0 tx=0x7f4df8009190 comp rx=0 tx=0).stop 2023-10-30T13:34:59.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.685+0000 7f4e0452a640 1 -- 172.21.15.46:0/416948019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dfc070a00 msgr2=0x7f4dfc1490e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:59.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.685+0000 7f4e0452a640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dfc070a00 0x7f4dfc1490e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00c4d0 tx=0x7f4dec00c990 comp rx=0 tx=0).stop 2023-10-30T13:34:59.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.685+0000 7f4e0452a640 1 -- 172.21.15.46:0/416948019 shutdown_connections 2023-10-30T13:34:59.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.685+0000 7f4e0452a640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4de00664a0 0x7f4de0068960 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:59.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.685+0000 7f4e0452a640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4dfc1539a0 0x7f4dfc149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:59.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.685+0000 7f4e0452a640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dfc070a00 0x7f4dfc1490e0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:59.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.685+0000 7f4e0452a640 1 --2- 172.21.15.46:0/416948019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dfc069e90 0x7f4dfc0c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:59.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.685+0000 7f4e0452a640 1 -- 172.21.15.46:0/416948019 >> 172.21.15.46:0/416948019 conn(0x7f4dfc0c7910 msgr2=0x7f4dfc14e550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:59.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.685+0000 7f4e0452a640 1 -- 172.21.15.46:0/416948019 shutdown_connections 2023-10-30T13:34:59.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:34:59.685+0000 7f4e0452a640 1 -- 172.21.15.46:0/416948019 wait complete. 2023-10-30T13:34:59.709 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:35:01.710 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd69032a640 1 Processor -- start 2023-10-30T13:35:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd69032a640 1 -- start start 2023-10-30T13:35:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd69032a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68815d6f0 0x7fd68815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd69032a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6880cb800 con 0x7fd6881538a0 2023-10-30T13:35:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd69032a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6880cb330 con 0x7fd6880cf160 2023-10-30T13:35:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd69032a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6880cad70 con 0x7fd68815d6f0 2023-10-30T13:35:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68e0c6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd6881538a0 0x7fd6880ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43098/0 (socket says 172.21.15.46:43098) 2023-10-30T13:35:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68e0c6640 1 -- 172.21.15.46:0/251567969 learned_addr learned my addr 172.21.15.46:0/251567969 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:35:01.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d8c5640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68815d6f0 0x7fd68815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718654542 0 0) 0x7fd6880cb800 con 0x7fd6881538a0 2023-10-30T13:35:01.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd674003530 con 0x7fd6881538a0 2023-10-30T13:35:01.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1323609019 0 0) 0x7fd674003530 con 0x7fd6881538a0 2023-10-30T13:35:01.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68815d6f0 msgr2=0x7fd68815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:01.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68815d6f0 0x7fd68815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 >> v1:172.21.15.154:6789/0 conn(0x7fd6880cf160 legacy=0x7fd688164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6880cb050 con 0x7fd6881538a0 2023-10-30T13:35:01.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fd684001680 con 0x7fd6881538a0 2023-10-30T13:35:01.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 >> v1:172.21.15.46:6789/0 conn(0x7fd6881538a0 legacy=0x7fd6880ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68815d6f0 0x7fd6740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd674003ef0 0x7fd6740047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd674004d10 0x7fd674005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.861+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6880cad70 con 0x7fd674003ef0 2023-10-30T13:35:01.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67400b9f0 con 0x7fd674004d10 2023-10-30T13:35:01.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67400bbb0 con 0x7fd68815d6f0 2023-10-30T13:35:01.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68d8c5640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd674003ef0 0x7fd6740047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68e0c6640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68815d6f0 0x7fd6740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68e8c7640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd674004d10 0x7fd674005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68d8c5640 1 -- 172.21.15.46:0/251567969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68815d6f0 msgr2=0x7fd6740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:01.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68d8c5640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd68815d6f0 0x7fd6740039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68d8c5640 1 -- 172.21.15.46:0/251567969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd674004d10 msgr2=0x7fd674005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:01.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68d8c5640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd674004d10 0x7fd674005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68d8c5640 1 -- 172.21.15.46:0/251567969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd674003530 con 0x7fd674003ef0 2023-10-30T13:35:01.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68e8c7640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd674004d10 0x7fd674005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:01.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68d8c5640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd674003ef0 0x7fd6740047d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd678009ae0 tx=0x7fd67801c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add46b4df255c1e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:01.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684007ea0 con 0x7fd674003ef0 2023-10-30T13:35:01.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6780a5580 con 0x7fd674003ef0 2023-10-30T13:35:01.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.865+0000 7fd68d0c4640 1 -- 172.21.15.46:0/251567969 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6780a5930 con 0x7fd674003ef0 2023-10-30T13:35:01.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 -- 172.21.15.46:0/251567969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd674003ef0 msgr2=0x7fd6740047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 --2- 172.21.15.46:0/251567969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd674003ef0 0x7fd6740047d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd678009ae0 tx=0x7fd67801c990 comp rx=0 tx=0).stop 2023-10-30T13:35:01.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd68e8c7640 1 -- 172.21.15.46:0/251567969 reap_dead start 2023-10-30T13:35:01.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 -- 172.21.15.46:0/251567969 shutdown_connections 2023-10-30T13:35:01.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 -- 172.21.15.46:0/251567969 >> 172.21.15.46:0/251567969 conn(0x7fd68806bfe0 msgr2=0x7fd68806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:01.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 -- 172.21.15.46:0/251567969 shutdown_connections 2023-10-30T13:35:01.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 -- 172.21.15.46:0/251567969 wait complete. 2023-10-30T13:35:01.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 Processor -- start 2023-10-30T13:35:01.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 -- start start 2023-10-30T13:35:01.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd674003ef0 0x7fd6881e6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6881509a0 0x7fd6880ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6880b3110 0x7fd6881f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68814e870 con 0x7fd674003ef0 2023-10-30T13:35:01.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68814dbf0 con 0x7fd6881509a0 2023-10-30T13:35:01.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd69032a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68814ded0 con 0x7fd6880b3110 2023-10-30T13:35:01.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd68e0c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd674003ef0 0x7fd6881e6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd68e0c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd674003ef0 0x7fd6881e6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36078/0 (socket says 172.21.15.46:36078) 2023-10-30T13:35:01.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd68e0c6640 1 -- 172.21.15.46:0/1629906774 learned_addr learned my addr 172.21.15.46:0/1629906774 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:01.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd68d8c5640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6881509a0 0x7fd6880ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.869+0000 7fd68e8c7640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6880b3110 0x7fd6881f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd68e0c6640 1 -- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6880b3110 msgr2=0x7fd6881f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd68e0c6640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6880b3110 0x7fd6881f2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd68e0c6640 1 -- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6881509a0 msgr2=0x7fd6880ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:01.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd68e0c6640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6881509a0 0x7fd6880ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:01.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd68e0c6640 1 -- 172.21.15.46:0/1629906774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68814e1b0 con 0x7fd674003ef0 2023-10-30T13:35:01.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd68e8c7640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6880b3110 0x7fd6881f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:01.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd68e0c6640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd674003ef0 0x7fd6881e6810 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd684001060 tx=0x7fd684012920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:01.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd67effd640 1 -- 172.21.15.46:0/1629906774 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684014810 con 0x7fd674003ef0 2023-10-30T13:35:01.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd67effd640 1 -- 172.21.15.46:0/1629906774 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd684014970 con 0x7fd674003ef0 2023-10-30T13:35:01.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd69032a640 1 -- 172.21.15.46:0/1629906774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68814e490 con 0x7fd674003ef0 2023-10-30T13:35:01.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd67effd640 1 -- 172.21.15.46:0/1629906774 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684014d20 con 0x7fd674003ef0 2023-10-30T13:35:01.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd69032a640 1 -- 172.21.15.46:0/1629906774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6880b3650 con 0x7fd674003ef0 2023-10-30T13:35:01.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd69032a640 1 -- 172.21.15.46:0/1629906774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68814e9b0 con 0x7fd674003ef0 2023-10-30T13:35:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd67effd640 1 -- 172.21.15.46:0/1629906774 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6840153e0 con 0x7fd674003ef0 2023-10-30T13:35:01.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd67effd640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd66c0664d0 0x7fd66c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd67effd640 1 -- 172.21.15.46:0/1629906774 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fd684086940 con 0x7fd674003ef0 2023-10-30T13:35:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.873+0000 7fd68d8c5640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd66c0664d0 0x7fd66c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.877+0000 7fd68d8c5640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd66c0664d0 0x7fd66c068990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd6780af000 tx=0x7fd6780046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:01.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:01.877+0000 7fd67effd640 1 -- 172.21.15.46:0/1629906774 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd68814e9b0 con 0x7fd674003ef0 2023-10-30T13:35:02.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.069+0000 7fd69032a640 1 -- 172.21.15.46:0/1629906774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd68814e9b0 con 0x7fd674003ef0 2023-10-30T13:35:02.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd67effd640 1 -- 172.21.15.46:0/1629906774 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6529 (secure 0 0 0) 0x7fd68814e9b0 con 0x7fd674003ef0 2023-10-30T13:35:02.077 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:02.078 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5721,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:34:05.920156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5266,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5266":{"gid":5266,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":22,"addr":"172.21.15.154:6835/651441434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":651441434},{"type":"v1","addr":"172.21.15.154:6835","nonce":651441434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:02.078 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2023-10-30T13:35:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd69032a640 1 -- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd66c0664d0 msgr2=0x7fd66c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd69032a640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd66c0664d0 0x7fd66c068990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd6780af000 tx=0x7fd6780046d0 comp rx=0 tx=0).stop 2023-10-30T13:35:02.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd69032a640 1 -- 172.21.15.46:0/1629906774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd674003ef0 msgr2=0x7fd6881e6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:02.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd69032a640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd674003ef0 0x7fd6881e6810 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd684001060 tx=0x7fd684012920 comp rx=0 tx=0).stop 2023-10-30T13:35:02.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd69032a640 1 -- 172.21.15.46:0/1629906774 shutdown_connections 2023-10-30T13:35:02.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd69032a640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd66c0664d0 0x7fd66c068990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:02.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd69032a640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6880b3110 0x7fd6881f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:02.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd69032a640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6881509a0 0x7fd6880ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd69032a640 1 --2- 172.21.15.46:0/1629906774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd674003ef0 0x7fd6881e6810 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd69032a640 1 -- 172.21.15.46:0/1629906774 >> 172.21.15.46:0/1629906774 conn(0x7fd68806bfe0 msgr2=0x7fd6881512d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd69032a640 1 -- 172.21.15.46:0/1629906774 shutdown_connections 2023-10-30T13:35:02.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:02.073+0000 7fd69032a640 1 -- 172.21.15.46:0/1629906774 wait complete. 2023-10-30T13:35:02.098 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T13:35:04.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:04.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd19a780640 1 Processor -- start 2023-10-30T13:35:04.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd19a780640 1 -- start start 2023-10-30T13:35:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd19a780640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1940bfec0 0x7fd1940c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd19a780640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd19406a720 0x7fd19406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd19a780640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd194071290 con 0x7fd194159e60 2023-10-30T13:35:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd19a780640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd194071570 con 0x7fd19406a720 2023-10-30T13:35:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd19a780640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd194071850 con 0x7fd1940bfec0 2023-10-30T13:35:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd193fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd194159e60 0x7fd19406f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43104/0 (socket says 172.21.15.46:43104) 2023-10-30T13:35:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd193fff640 1 -- 172.21.15.46:0/3606848076 learned_addr learned my addr 172.21.15.46:0/3606848076 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:35:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd198d1d640 1 --2- 172.21.15.46:0/3606848076 >> v2:172.21.15.154:3300/0 conn(0x7fd19406a720 0x7fd19406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd1937fe640 1 --2- 172.21.15.46:0/3606848076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1940bfec0 0x7fd1940c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63138971 0 0) 0x7fd194071290 con 0x7fd194159e60 2023-10-30T13:35:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd17c003530 con 0x7fd194159e60 2023-10-30T13:35:04.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3021169155 0 0) 0x7fd17c003530 con 0x7fd194159e60 2023-10-30T13:35:04.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1940bfec0 msgr2=0x7fd1940c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:04.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 --2- 172.21.15.46:0/3606848076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1940bfec0 0x7fd1940c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 >> v2:172.21.15.154:3300/0 conn(0x7fd19406a720 msgr2=0x7fd19406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 --2- 172.21.15.46:0/3606848076 >> v2:172.21.15.154:3300/0 conn(0x7fd19406a720 0x7fd19406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd194071b30 con 0x7fd194159e60 2023-10-30T13:35:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fd184001680 con 0x7fd194159e60 2023-10-30T13:35:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 >> v1:172.21.15.46:6789/0 conn(0x7fd194159e60 legacy=0x7fd19406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 --2- 172.21.15.46:0/3606848076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17c003980 0x7fd17c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 --2- 172.21.15.46:0/3606848076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1940bfec0 0x7fd17c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 --2- 172.21.15.46:0/3606848076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd17c004d10 0x7fd17c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd194071570 con 0x7fd17c004d10 2023-10-30T13:35:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd194071850 con 0x7fd17c003980 2023-10-30T13:35:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.249+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c00dba0 con 0x7fd1940bfec0 2023-10-30T13:35:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd198d1d640 1 --2- 172.21.15.46:0/3606848076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd17c004d10 0x7fd17c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd193fff640 1 --2- 172.21.15.46:0/3606848076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17c003980 0x7fd17c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd1937fe640 1 --2- 172.21.15.46:0/3606848076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1940bfec0 0x7fd17c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd198d1d640 1 -- 172.21.15.46:0/3606848076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1940bfec0 msgr2=0x7fd17c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd198d1d640 1 --2- 172.21.15.46:0/3606848076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1940bfec0 0x7fd17c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd198d1d640 1 -- 172.21.15.46:0/3606848076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17c003980 msgr2=0x7fd17c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd198d1d640 1 --2- 172.21.15.46:0/3606848076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17c003980 0x7fd17c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd198d1d640 1 -- 172.21.15.46:0/3606848076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c003530 con 0x7fd17c004d10 2023-10-30T13:35:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd198d1d640 1 --2- 172.21.15.46:0/3606848076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd17c004d10 0x7fd17c005190 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009fc0 tx=0x7fd18c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8b9cfa87b353a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c0a5cb0 con 0x7fd17c004d10 2023-10-30T13:35:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18c00f040 con 0x7fd17c004d10 2023-10-30T13:35:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd192ffd640 1 -- 172.21.15.46:0/3606848076 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c0ad680 con 0x7fd17c004d10 2023-10-30T13:35:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd19a780640 1 -- 172.21.15.46:0/3606848076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd17c004d10 msgr2=0x7fd17c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.253+0000 7fd19a780640 1 --2- 172.21.15.46:0/3606848076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd17c004d10 0x7fd17c005190 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009fc0 tx=0x7fd18c01cca0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd198d1d640 1 -- 172.21.15.46:0/3606848076 reap_dead start 2023-10-30T13:35:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd19a780640 1 -- 172.21.15.46:0/3606848076 shutdown_connections 2023-10-30T13:35:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd19a780640 1 -- 172.21.15.46:0/3606848076 >> 172.21.15.46:0/3606848076 conn(0x7fd1940c7910 msgr2=0x7fd1940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd19a780640 1 -- 172.21.15.46:0/3606848076 shutdown_connections 2023-10-30T13:35:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd19a780640 1 -- 172.21.15.46:0/3606848076 wait complete. 2023-10-30T13:35:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd19a780640 1 Processor -- start 2023-10-30T13:35:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd19a780640 1 -- start start 2023-10-30T13:35:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd19a780640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd17c004d10 0x7fd1941499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd19a780640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd19414a630 0x7fd1940c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd19a780640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1940c5950 0x7fd194146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd19a780640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941ee4a0 con 0x7fd19414a630 2023-10-30T13:35:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd19a780640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941485f0 con 0x7fd1940c5950 2023-10-30T13:35:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd19a780640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941ef470 con 0x7fd17c004d10 2023-10-30T13:35:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd1937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd19414a630 0x7fd1940c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd1937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd19414a630 0x7fd1940c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36094/0 (socket says 172.21.15.46:36094) 2023-10-30T13:35:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd1937fe640 1 -- 172.21.15.46:0/564481437 learned_addr learned my addr 172.21.15.46:0/564481437 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd193fff640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd17c004d10 0x7fd1941499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd198d1d640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1940c5950 0x7fd194146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd1937fe640 1 -- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd17c004d10 msgr2=0x7fd1941499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd1937fe640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd17c004d10 0x7fd1941499c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd1937fe640 1 -- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1940c5950 msgr2=0x7fd194146c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd1937fe640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1940c5950 0x7fd194146c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd1937fe640 1 -- 172.21.15.46:0/564481437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1940c45f0 con 0x7fd19414a630 2023-10-30T13:35:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd193fff640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd17c004d10 0x7fd1941499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.257+0000 7fd1937fe640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd19414a630 0x7fd1940c5410 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd188011640 tx=0x7fd188011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.261+0000 7fd190ff9640 1 -- 172.21.15.46:0/564481437 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18800f920 con 0x7fd19414a630 2023-10-30T13:35:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.261+0000 7fd190ff9640 1 -- 172.21.15.46:0/564481437 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18800fa80 con 0x7fd19414a630 2023-10-30T13:35:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.261+0000 7fd190ff9640 1 -- 172.21.15.46:0/564481437 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18800fdf0 con 0x7fd19414a630 2023-10-30T13:35:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.261+0000 7fd19a780640 1 -- 172.21.15.46:0/564481437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1941ee6b0 con 0x7fd19414a630 2023-10-30T13:35:04.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.261+0000 7fd19a780640 1 -- 172.21.15.46:0/564481437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1941eec70 con 0x7fd19414a630 2023-10-30T13:35:04.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.261+0000 7fd19a780640 1 -- 172.21.15.46:0/564481437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19404e170 con 0x7fd19414a630 2023-10-30T13:35:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.261+0000 7fd190ff9640 1 -- 172.21.15.46:0/564481437 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd188015460 con 0x7fd19414a630 2023-10-30T13:35:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.261+0000 7fd190ff9640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1800664d0 0x7fd180068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.261+0000 7fd190ff9640 1 -- 172.21.15.46:0/564481437 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fd188084690 con 0x7fd19414a630 2023-10-30T13:35:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.261+0000 7fd193fff640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1800664d0 0x7fd180068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.265+0000 7fd193fff640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1800664d0 0x7fd180068990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd184000c10 tx=0x7fd184009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.265+0000 7fd190ff9640 1 -- 172.21.15.46:0/564481437 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd188051780 con 0x7fd19414a630 2023-10-30T13:35:04.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.457+0000 7fd19a780640 1 -- 172.21.15.46:0/564481437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd194158db0 con 0x7fd19414a630 2023-10-30T13:35:04.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.457+0000 7fd190ff9640 1 -- 172.21.15.46:0/564481437 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5734 (secure 0 0 0) 0x7fd188054da0 con 0x7fd19414a630 2023-10-30T13:35:04.463 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:04.464 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:03.892152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:04.464 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 61 2023-10-30T13:35:04.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.461+0000 7fd19a780640 1 -- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1800664d0 msgr2=0x7fd180068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.461+0000 7fd19a780640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1800664d0 0x7fd180068990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd184000c10 tx=0x7fd184009040 comp rx=0 tx=0).stop 2023-10-30T13:35:04.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.461+0000 7fd19a780640 1 -- 172.21.15.46:0/564481437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd19414a630 msgr2=0x7fd1940c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.461+0000 7fd19a780640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd19414a630 0x7fd1940c5410 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd188011640 tx=0x7fd188011b00 comp rx=0 tx=0).stop 2023-10-30T13:35:04.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.461+0000 7fd19a780640 1 -- 172.21.15.46:0/564481437 shutdown_connections 2023-10-30T13:35:04.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.461+0000 7fd19a780640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1800664d0 0x7fd180068990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.461+0000 7fd19a780640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1940c5950 0x7fd194146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.461+0000 7fd19a780640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd19414a630 0x7fd1940c5410 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.461+0000 7fd19a780640 1 --2- 172.21.15.46:0/564481437 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd17c004d10 0x7fd1941499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.461+0000 7fd19a780640 1 -- 172.21.15.46:0/564481437 >> 172.21.15.46:0/564481437 conn(0x7fd1940c7910 msgr2=0x7fd1940c9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:04.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.461+0000 7fd19a780640 1 -- 172.21.15.46:0/564481437 shutdown_connections 2023-10-30T13:35:04.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.461+0000 7fd19a780640 1 -- 172.21.15.46:0/564481437 wait complete. 2023-10-30T13:35:04.484 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-30T13:35:04.484 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:35:04.485 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:04.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.621+0000 7f1c3de91640 1 Processor -- start 2023-10-30T13:35:04.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.621+0000 7f1c3de91640 1 -- start start 2023-10-30T13:35:04.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.621+0000 7f1c3de91640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c380c93d0 0x7f1c3806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.621+0000 7f1c3de91640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3806afa0 con 0x7f1c3806e470 2023-10-30T13:35:04.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.621+0000 7f1c3de91640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c38154150 con 0x7f1c38153d70 2023-10-30T13:35:04.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.621+0000 7f1c3de91640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c3806d780 con 0x7f1c380c93d0 2023-10-30T13:35:04.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.625+0000 7f1c37fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c380c93d0 0x7f1c3806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.625+0000 7f1c36ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1c3806e470 0x7f1c380c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43106/0 (socket says 172.21.15.46:43106) 2023-10-30T13:35:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.625+0000 7f1c36ffd640 1 -- 172.21.15.46:0/2528994125 learned_addr learned my addr 172.21.15.46:0/2528994125 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:35:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.625+0000 7f1c37fff640 1 -- 172.21.15.46:0/2528994125 >> v1:172.21.15.154:6789/0 conn(0x7f1c38153d70 legacy=0x7f1c380cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.625+0000 7f1c37fff640 1 -- 172.21.15.46:0/2528994125 >> v1:172.21.15.46:6789/0 conn(0x7f1c3806e470 legacy=0x7f1c380c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.625+0000 7f1c37fff640 1 -- 172.21.15.46:0/2528994125 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3806e030 con 0x7f1c380c93d0 2023-10-30T13:35:04.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.625+0000 7f1c367fc640 1 -- 172.21.15.46:0/2528994125 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799933581 0 0) 0x7f1c3806afa0 con 0x7f1c3806e470 2023-10-30T13:35:04.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.625+0000 7f1c37fff640 1 --2- 172.21.15.46:0/2528994125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c380c93d0 0x7f1c3806ca80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1c30009d90 tx=0x7f1c3001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18b2311252d2cee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.625+0000 7f1c367fc640 1 -- 172.21.15.46:0/2528994125 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c300a59e0 con 0x7f1c380c93d0 2023-10-30T13:35:04.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.625+0000 7f1c367fc640 1 -- 172.21.15.46:0/2528994125 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c300a5b40 con 0x7f1c380c93d0 2023-10-30T13:35:04.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.625+0000 7f1c367fc640 1 -- 172.21.15.46:0/2528994125 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c300ad440 con 0x7f1c380c93d0 2023-10-30T13:35:04.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 -- 172.21.15.46:0/2528994125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c380c93d0 msgr2=0x7f1c3806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 --2- 172.21.15.46:0/2528994125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c380c93d0 0x7f1c3806ca80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1c30009d90 tx=0x7f1c3001c920 comp rx=0 tx=0).stop 2023-10-30T13:35:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 -- 172.21.15.46:0/2528994125 shutdown_connections 2023-10-30T13:35:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 --2- 172.21.15.46:0/2528994125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c380c93d0 0x7f1c3806ca80 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 -- 172.21.15.46:0/2528994125 >> 172.21.15.46:0/2528994125 conn(0x7f1c38071ea0 msgr2=0x7f1c38065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 -- 172.21.15.46:0/2528994125 shutdown_connections 2023-10-30T13:35:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 -- 172.21.15.46:0/2528994125 wait complete. 2023-10-30T13:35:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 Processor -- start 2023-10-30T13:35:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 -- start start 2023-10-30T13:35:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3806e470 0x7f1c380cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c380c93d0 0x7f1c38146870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c38153d70 0x7f1c381f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c381f6720 con 0x7f1c3806e470 2023-10-30T13:35:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c38145110 con 0x7f1c380c93d0 2023-10-30T13:35:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c3de91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c381453f0 con 0x7f1c38153d70 2023-10-30T13:35:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3806e470 0x7f1c380cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3806e470 0x7f1c380cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36104/0 (socket says 172.21.15.46:36104) 2023-10-30T13:35:04.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c377fe640 1 -- 172.21.15.46:0/822819474 learned_addr learned my addr 172.21.15.46:0/822819474 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:04.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c36ffd640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c380c93d0 0x7f1c38146870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c37fff640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c38153d70 0x7f1c381f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c377fe640 1 -- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c38153d70 msgr2=0x7f1c381f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c377fe640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c38153d70 0x7f1c381f5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c377fe640 1 -- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c380c93d0 msgr2=0x7f1c38146870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c377fe640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c380c93d0 0x7f1c38146870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.629+0000 7f1c377fe640 1 -- 172.21.15.46:0/822819474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c30009a70 con 0x7f1c3806e470 2023-10-30T13:35:04.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.633+0000 7f1c377fe640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3806e470 0x7f1c380cc5c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1c2800c500 tx=0x7f1c2800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.633+0000 7f1c1ffff640 1 -- 172.21.15.46:0/822819474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c28005220 con 0x7f1c3806e470 2023-10-30T13:35:04.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.633+0000 7f1c1ffff640 1 -- 172.21.15.46:0/822819474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c28005380 con 0x7f1c3806e470 2023-10-30T13:35:04.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.633+0000 7f1c1ffff640 1 -- 172.21.15.46:0/822819474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c28007270 con 0x7f1c3806e470 2023-10-30T13:35:04.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.633+0000 7f1c3de91640 1 -- 172.21.15.46:0/822819474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3813e590 con 0x7f1c3806e470 2023-10-30T13:35:04.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.633+0000 7f1c3de91640 1 -- 172.21.15.46:0/822819474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3815a6c0 con 0x7f1c3806e470 2023-10-30T13:35:04.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.633+0000 7f1c3de91640 1 -- 172.21.15.46:0/822819474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c381455b0 con 0x7f1c3806e470 2023-10-30T13:35:04.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.633+0000 7f1c1ffff640 1 -- 172.21.15.46:0/822819474 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1c280073d0 con 0x7f1c3806e470 2023-10-30T13:35:04.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.633+0000 7f1c1ffff640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c240664a0 0x7f1c24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:04.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.633+0000 7f1c1ffff640 1 -- 172.21.15.46:0/822819474 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f1c28083a00 con 0x7f1c3806e470 2023-10-30T13:35:04.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.633+0000 7f1c36ffd640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c240664a0 0x7f1c24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:04.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.637+0000 7f1c36ffd640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c240664a0 0x7f1c24068960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c007bc0 tx=0x7f1c2c007b10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.637+0000 7f1c1ffff640 1 -- 172.21.15.46:0/822819474 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1c28050bb0 con 0x7f1c3806e470 2023-10-30T13:35:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 -- 172.21.15.46:0/822819474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c38144f70 con 0x7f1c3806e470 2023-10-30T13:35:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c1ffff640 1 -- 172.21.15.46:0/822819474 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f1c38144f70 con 0x7f1c3806e470 2023-10-30T13:35:04.834 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:04.834 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:03.892152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 61 2023-10-30T13:35:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 -- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c240664a0 msgr2=0x7f1c24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c240664a0 0x7f1c24068960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c007bc0 tx=0x7f1c2c007b10 comp rx=0 tx=0).stop 2023-10-30T13:35:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 -- 172.21.15.46:0/822819474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3806e470 msgr2=0x7f1c380cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3806e470 0x7f1c380cc5c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1c2800c500 tx=0x7f1c2800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 -- 172.21.15.46:0/822819474 shutdown_connections 2023-10-30T13:35:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c240664a0 0x7f1c24068960 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c38153d70 0x7f1c381f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c380c93d0 0x7f1c38146870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 --2- 172.21.15.46:0/822819474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3806e470 0x7f1c380cc5c0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 -- 172.21.15.46:0/822819474 >> 172.21.15.46:0/822819474 conn(0x7f1c38071ea0 msgr2=0x7f1c380c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 -- 172.21.15.46:0/822819474 shutdown_connections 2023-10-30T13:35:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:04.829+0000 7f1c3de91640 1 -- 172.21.15.46:0/822819474 wait complete. 2023-10-30T13:35:04.855 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 61, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 61, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5442": { "addr": "172.21.15.46:6836/3955931693", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3955931693, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3955931693, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5442, "incarnation": 46, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 13 }, "gid_5574": { "addr": "172.21.15.46:6837/126953235", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 126953235, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 126953235, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5574, "incarnation": 55, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_5721": { "addr": "172.21.15.46:6833/4044310929", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4044310929, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4044310929, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5721, "incarnation": 61, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:35:03.892152+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5721, "mds_1": 5574, "mds_2": 5442 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:35:06.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:07.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.005+0000 7f0adb86b640 1 Processor -- start 2023-10-30T13:35:07.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.005+0000 7f0adb86b640 1 -- start start 2023-10-30T13:35:07.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.005+0000 7f0adb86b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0ad415a410 0x7f0ad406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:07.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.005+0000 7f0adb86b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0ad406cb80 0x7f0ad406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:07.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.005+0000 7f0adb86b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad406d530 con 0x7f0ad4069e90 2023-10-30T13:35:07.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.005+0000 7f0adb86b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad40c8570 con 0x7f0ad406cb80 2023-10-30T13:35:07.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.005+0000 7f0adb86b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad415a850 con 0x7f0ad415a410 2023-10-30T13:35:07.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.009+0000 7f0ad9e08640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0ad406cb80 0x7f0ad406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:07.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.009+0000 7f0ad9e08640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0ad406cb80 0x7f0ad406cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39054/0 (socket says 172.21.15.46:39054) 2023-10-30T13:35:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.009+0000 7f0ad9e08640 1 -- 172.21.15.46:0/2703854477 learned_addr learned my addr 172.21.15.46:0/2703854477 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.009+0000 7f0ad8e06640 1 --2- 172.21.15.46:0/2703854477 >> v2:172.21.15.154:3300/0 conn(0x7f0ad415a410 0x7f0ad406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.009+0000 7f0ad9e08640 1 -- 172.21.15.46:0/2703854477 >> v2:172.21.15.154:3300/0 conn(0x7f0ad415a410 msgr2=0x7f0ad406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.009+0000 7f0ad9e08640 1 --2- 172.21.15.46:0/2703854477 >> v2:172.21.15.154:3300/0 conn(0x7f0ad415a410 0x7f0ad406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.009+0000 7f0ad9e08640 1 -- 172.21.15.46:0/2703854477 >> v1:172.21.15.154:6789/0 conn(0x7f0ad4069e90 legacy=0x7f0ad40c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.009+0000 7f0ad9e08640 1 -- 172.21.15.46:0/2703854477 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad40c6c10 con 0x7f0ad406cb80 2023-10-30T13:35:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.009+0000 7f0ad9e08640 1 --2- 172.21.15.46:0/2703854477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad406cb80 0x7f0ad406cff0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0ad406feb0 tx=0x7f0acc01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb26736f463239e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.009+0000 7f0acbfff640 1 -- 172.21.15.46:0/2703854477 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc0a5690 con 0x7f0ad406cb80 2023-10-30T13:35:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.009+0000 7f0acbfff640 1 -- 172.21.15.46:0/2703854477 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0acc0a57f0 con 0x7f0ad406cb80 2023-10-30T13:35:07.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.009+0000 7f0acbfff640 1 -- 172.21.15.46:0/2703854477 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc0a5b80 con 0x7f0ad406cb80 2023-10-30T13:35:07.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 -- 172.21.15.46:0/2703854477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad406cb80 msgr2=0x7f0ad406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:07.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 --2- 172.21.15.46:0/2703854477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad406cb80 0x7f0ad406cff0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0ad406feb0 tx=0x7f0acc01c9d0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 -- 172.21.15.46:0/2703854477 shutdown_connections 2023-10-30T13:35:07.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 --2- 172.21.15.46:0/2703854477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad406cb80 0x7f0ad406cff0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 --2- 172.21.15.46:0/2703854477 >> v2:172.21.15.154:3300/0 conn(0x7f0ad415a410 0x7f0ad406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 -- 172.21.15.46:0/2703854477 >> 172.21.15.46:0/2703854477 conn(0x7f0ad406dfd0 msgr2=0x7f0ad406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:07.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 -- 172.21.15.46:0/2703854477 shutdown_connections 2023-10-30T13:35:07.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 -- 172.21.15.46:0/2703854477 wait complete. 2023-10-30T13:35:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 Processor -- start 2023-10-30T13:35:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 -- start start 2023-10-30T13:35:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad4069e90 0x7f0ad4147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad406cb80 0x7f0ad406b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad415a410 0x7f0ad406bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad415d840 con 0x7f0ad415a410 2023-10-30T13:35:07.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad414e270 con 0x7f0ad4069e90 2023-10-30T13:35:07.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0adb86b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad414e550 con 0x7f0ad406cb80 2023-10-30T13:35:07.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0ad9e08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad415a410 0x7f0ad406bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:07.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0ad9e08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad415a410 0x7f0ad406bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39056/0 (socket says 172.21.15.46:39056) 2023-10-30T13:35:07.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0ad9e08640 1 -- 172.21.15.46:0/2565822475 learned_addr learned my addr 172.21.15.46:0/2565822475 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0ad8e06640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad406cb80 0x7f0ad406b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0ad9607640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad4069e90 0x7f0ad4147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0ad9e08640 1 -- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad406cb80 msgr2=0x7f0ad406b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0ad9e08640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad406cb80 0x7f0ad406b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0ad9e08640 1 -- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad4069e90 msgr2=0x7f0ad4147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0ad9e08640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad4069e90 0x7f0ad4147ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0ad9e08640 1 -- 172.21.15.46:0/2565822475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc009b30 con 0x7f0ad415a410 2023-10-30T13:35:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.013+0000 7f0ad9607640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad4069e90 0x7f0ad4147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.017+0000 7f0ad9e08640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad415a410 0x7f0ad406bac0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0af040 tx=0x7f0acc019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.017+0000 7f0ac9ffb640 1 -- 172.21.15.46:0/2565822475 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc0b8670 con 0x7f0ad415a410 2023-10-30T13:35:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.017+0000 7f0ac9ffb640 1 -- 172.21.15.46:0/2565822475 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0acc0b87f0 con 0x7f0ad415a410 2023-10-30T13:35:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.017+0000 7f0ac9ffb640 1 -- 172.21.15.46:0/2565822475 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc0b8ba0 con 0x7f0ad415a410 2023-10-30T13:35:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.017+0000 7f0adb86b640 1 -- 172.21.15.46:0/2565822475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad414da90 con 0x7f0ad415a410 2023-10-30T13:35:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.017+0000 7f0adb86b640 1 -- 172.21.15.46:0/2565822475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad414dd70 con 0x7f0ad415a410 2023-10-30T13:35:07.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.017+0000 7f0adb86b640 1 -- 172.21.15.46:0/2565822475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad404e170 con 0x7f0ad415a410 2023-10-30T13:35:07.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.017+0000 7f0ac9ffb640 1 -- 172.21.15.46:0/2565822475 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0acc0b8d00 con 0x7f0ad415a410 2023-10-30T13:35:07.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.017+0000 7f0ac9ffb640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ab80664a0 0x7f0ab8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:07.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.017+0000 7f0ac9ffb640 1 -- 172.21.15.46:0/2565822475 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f0acc11f6c0 con 0x7f0ad415a410 2023-10-30T13:35:07.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.017+0000 7f0ad9607640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ab80664a0 0x7f0ab8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:07.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.021+0000 7f0ad9607640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ab80664a0 0x7f0ab8068960 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0000f70 tx=0x7f0ad0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:07.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.021+0000 7f0ac9ffb640 1 -- 172.21.15.46:0/2565822475 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0acc0ec9b0 con 0x7f0ad415a410 2023-10-30T13:35:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 -- 172.21.15.46:0/2565822475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ad415d980 con 0x7f0ad415a410 2023-10-30T13:35:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0ac9ffb640 1 -- 172.21.15.46:0/2565822475 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f0acc009510 con 0x7f0ad415a410 2023-10-30T13:35:07.218 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:07.218 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:03.892152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:07.218 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 61 2023-10-30T13:35:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 -- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ab80664a0 msgr2=0x7f0ab8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ab80664a0 0x7f0ab8068960 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0000f70 tx=0x7f0ad0009040 comp rx=0 tx=0).stop 2023-10-30T13:35:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 -- 172.21.15.46:0/2565822475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad415a410 msgr2=0x7f0ad406bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad415a410 0x7f0ad406bac0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0af040 tx=0x7f0acc019770 comp rx=0 tx=0).stop 2023-10-30T13:35:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 -- 172.21.15.46:0/2565822475 shutdown_connections 2023-10-30T13:35:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ab80664a0 0x7f0ab8068960 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad415a410 0x7f0ad406bac0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad406cb80 0x7f0ad406b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 --2- 172.21.15.46:0/2565822475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad4069e90 0x7f0ad4147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:07.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 -- 172.21.15.46:0/2565822475 >> 172.21.15.46:0/2565822475 conn(0x7f0ad406dfd0 msgr2=0x7f0ad40c9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:07.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 -- 172.21.15.46:0/2565822475 shutdown_connections 2023-10-30T13:35:07.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:07.213+0000 7f0adb86b640 1 -- 172.21.15.46:0/2565822475 wait complete. 2023-10-30T13:35:07.239 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:09.240 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:09.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.381+0000 7f907efab640 1 Processor -- start 2023-10-30T13:35:09.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.381+0000 7f907efab640 1 -- start start 2023-10-30T13:35:09.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f907efab640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90781538a0 0x7f907815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:09.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f907efab640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f90780cf160 0x7f90780cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:09.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f907efab640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90780cb800 con 0x7f90780ca0e0 2023-10-30T13:35:09.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f907efab640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907815dc30 con 0x7f90781538a0 2023-10-30T13:35:09.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f907efab640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90780cb330 con 0x7f90780cf160 2023-10-30T13:35:09.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f9077fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f90780ca0e0 0x7f907815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33496/0 (socket says 172.21.15.46:33496) 2023-10-30T13:35:09.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f9077fff640 1 -- 172.21.15.46:0/3632027744 learned_addr learned my addr 172.21.15.46:0/3632027744 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:35:09.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f907cd47640 1 --2- 172.21.15.46:0/3632027744 >> v2:172.21.15.46:3300/0 conn(0x7f90781538a0 0x7f907815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:09.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f907d548640 1 --2- 172.21.15.46:0/3632027744 >> v2:172.21.15.154:3300/0 conn(0x7f90780cf160 0x7f90780cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:09.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f907cd47640 1 -- 172.21.15.46:0/3632027744 >> v2:172.21.15.154:3300/0 conn(0x7f90780cf160 msgr2=0x7f90780cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:09.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f90777fe640 1 -- 172.21.15.46:0/3632027744 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672715654 0 0) 0x7f90780cb800 con 0x7f90780ca0e0 2023-10-30T13:35:09.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f907cd47640 1 --2- 172.21.15.46:0/3632027744 >> v2:172.21.15.154:3300/0 conn(0x7f90780cf160 0x7f90780cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f907cd47640 1 -- 172.21.15.46:0/3632027744 >> v1:172.21.15.46:6789/0 conn(0x7f90780ca0e0 legacy=0x7f907815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:09.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f907cd47640 1 -- 172.21.15.46:0/3632027744 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90780cad70 con 0x7f90781538a0 2023-10-30T13:35:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.385+0000 7f907d548640 1 --2- 172.21.15.46:0/3632027744 >> v2:172.21.15.154:3300/0 conn(0x7f90780cf160 0x7f90780cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:09.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.389+0000 7f907cd47640 1 --2- 172.21.15.46:0/3632027744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90781538a0 0x7f907815d6f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f90780b5330 tx=0x7f906c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad00f2b17e7b02c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:09.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.389+0000 7f90777fe640 1 -- 172.21.15.46:0/3632027744 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c0a8950 con 0x7f90781538a0 2023-10-30T13:35:09.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.389+0000 7f90777fe640 1 -- 172.21.15.46:0/3632027744 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906c0a8ab0 con 0x7f90781538a0 2023-10-30T13:35:09.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.389+0000 7f90777fe640 1 -- 172.21.15.46:0/3632027744 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c0a8e40 con 0x7f90781538a0 2023-10-30T13:35:09.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.389+0000 7f907efab640 1 -- 172.21.15.46:0/3632027744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90781538a0 msgr2=0x7f907815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.389+0000 7f907efab640 1 --2- 172.21.15.46:0/3632027744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90781538a0 0x7f907815d6f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f90780b5330 tx=0x7f906c01ec50 comp rx=0 tx=0).stop 2023-10-30T13:35:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.389+0000 7f907efab640 1 -- 172.21.15.46:0/3632027744 shutdown_connections 2023-10-30T13:35:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.389+0000 7f907efab640 1 --2- 172.21.15.46:0/3632027744 >> v2:172.21.15.154:3300/0 conn(0x7f90780cf160 0x7f90780cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.389+0000 7f907efab640 1 --2- 172.21.15.46:0/3632027744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90781538a0 0x7f907815d6f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.389+0000 7f907efab640 1 -- 172.21.15.46:0/3632027744 >> 172.21.15.46:0/3632027744 conn(0x7f907806bfe0 msgr2=0x7f907806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:09.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.389+0000 7f907efab640 1 -- 172.21.15.46:0/3632027744 shutdown_connections 2023-10-30T13:35:09.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.389+0000 7f907efab640 1 -- 172.21.15.46:0/3632027744 wait complete. 2023-10-30T13:35:09.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907efab640 1 Processor -- start 2023-10-30T13:35:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907efab640 1 -- start start 2023-10-30T13:35:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907efab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90780ca0e0 0x7f9078146ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907efab640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90780cf160 0x7f90781efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907efab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90781538a0 0x7f90780b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:09.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907efab640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9078158ab0 con 0x7f90781538a0 2023-10-30T13:35:09.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907efab640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9078147a90 con 0x7f90780ca0e0 2023-10-30T13:35:09.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907efab640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90781ef970 con 0x7f90780cf160 2023-10-30T13:35:09.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907d548640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90781538a0 0x7f90780b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:09.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907d548640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90781538a0 0x7f90780b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39082/0 (socket says 172.21.15.46:39082) 2023-10-30T13:35:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907d548640 1 -- 172.21.15.46:0/1511824997 learned_addr learned my addr 172.21.15.46:0/1511824997 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907cd47640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90780ca0e0 0x7f9078146ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f9077fff640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90780cf160 0x7f90781efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907d548640 1 -- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90780cf160 msgr2=0x7f90781efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907d548640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90780cf160 0x7f90781efe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907d548640 1 -- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90780ca0e0 msgr2=0x7f9078146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907d548640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90780ca0e0 0x7f9078146ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907d548640 1 -- 172.21.15.46:0/1511824997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906c009420 con 0x7f90781538a0 2023-10-30T13:35:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907cd47640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90780ca0e0 0x7f9078146ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.393+0000 7f907d548640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90781538a0 0x7f90780b4580 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9070007de0 tx=0x7f907000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.397+0000 7f90757fa640 1 -- 172.21.15.46:0/1511824997 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90700039a0 con 0x7f90781538a0 2023-10-30T13:35:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.397+0000 7f90757fa640 1 -- 172.21.15.46:0/1511824997 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907000f040 con 0x7f90781538a0 2023-10-30T13:35:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.397+0000 7f907efab640 1 -- 172.21.15.46:0/1511824997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90780b5080 con 0x7f90781538a0 2023-10-30T13:35:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.397+0000 7f90757fa640 1 -- 172.21.15.46:0/1511824997 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070003d40 con 0x7f90781538a0 2023-10-30T13:35:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.397+0000 7f907efab640 1 -- 172.21.15.46:0/1511824997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90781474d0 con 0x7f90781538a0 2023-10-30T13:35:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.397+0000 7f907efab640 1 -- 172.21.15.46:0/1511824997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90780b2a50 con 0x7f90781538a0 2023-10-30T13:35:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.397+0000 7f90757fa640 1 -- 172.21.15.46:0/1511824997 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9070004030 con 0x7f90781538a0 2023-10-30T13:35:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.401+0000 7f90757fa640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f905c0664d0 0x7f905c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.401+0000 7f90757fa640 1 -- 172.21.15.46:0/1511824997 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f9070084650 con 0x7f90781538a0 2023-10-30T13:35:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.401+0000 7f907cd47640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f905c0664d0 0x7f905c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.401+0000 7f907cd47640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f905c0664d0 0x7f905c068990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f906c0b2000 tx=0x7f906c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.405+0000 7f90757fa640 1 -- 172.21.15.46:0/1511824997 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9070051630 con 0x7f90781538a0 2023-10-30T13:35:09.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.601+0000 7f907efab640 1 -- 172.21.15.46:0/1511824997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90781efae0 con 0x7f90781538a0 2023-10-30T13:35:09.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.601+0000 7f90757fa640 1 -- 172.21.15.46:0/1511824997 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f9070054c50 con 0x7f90781538a0 2023-10-30T13:35:09.609 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:09.609 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:03.892152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:09.609 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 61 2023-10-30T13:35:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.605+0000 7f907efab640 1 -- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f905c0664d0 msgr2=0x7f905c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.605+0000 7f907efab640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f905c0664d0 0x7f905c068990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f906c0b2000 tx=0x7f906c008040 comp rx=0 tx=0).stop 2023-10-30T13:35:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.605+0000 7f907efab640 1 -- 172.21.15.46:0/1511824997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90781538a0 msgr2=0x7f90780b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.605+0000 7f907efab640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90781538a0 0x7f90780b4580 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9070007de0 tx=0x7f907000d4a0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.605+0000 7f907efab640 1 -- 172.21.15.46:0/1511824997 shutdown_connections 2023-10-30T13:35:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.605+0000 7f907efab640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f905c0664d0 0x7f905c068990 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.605+0000 7f907efab640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90781538a0 0x7f90780b4580 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.605+0000 7f907efab640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90780cf160 0x7f90781efe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.605+0000 7f907efab640 1 --2- 172.21.15.46:0/1511824997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90780ca0e0 0x7f9078146ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:09.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.605+0000 7f907efab640 1 -- 172.21.15.46:0/1511824997 >> 172.21.15.46:0/1511824997 conn(0x7f907806bfe0 msgr2=0x7f90781526b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:09.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.605+0000 7f907efab640 1 -- 172.21.15.46:0/1511824997 shutdown_connections 2023-10-30T13:35:09.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:09.605+0000 7f907efab640 1 -- 172.21.15.46:0/1511824997 wait complete. 2023-10-30T13:35:09.632 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:11.633 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.781+0000 7fd834079640 1 Processor -- start 2023-10-30T13:35:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.781+0000 7fd834079640 1 -- start start 2023-10-30T13:35:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd834079640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd82c06f7a0 0x7fd82c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd834079640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd82c071110 0x7fd82c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd834079640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd82c066cf0 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd834079640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd82c0700c0 con 0x7fd82c071110 2023-10-30T13:35:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd834079640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd82c0c70d0 con 0x7fd82c06f7a0 2023-10-30T13:35:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd832616640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd82c071110 0x7fd82c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd831614640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd82c0c6bd0 0x7fd82c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33500/0 (socket says 172.21.15.46:33500) 2023-10-30T13:35:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd831614640 1 -- 172.21.15.46:0/1549700746 learned_addr learned my addr 172.21.15.46:0/1549700746 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:35:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd831e15640 1 --2- 172.21.15.46:0/1549700746 >> v2:172.21.15.154:3300/0 conn(0x7fd82c06f7a0 0x7fd82c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd832616640 1 -- 172.21.15.46:0/1549700746 >> v2:172.21.15.154:3300/0 conn(0x7fd82c06f7a0 msgr2=0x7fd82c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd832616640 1 --2- 172.21.15.46:0/1549700746 >> v2:172.21.15.154:3300/0 conn(0x7fd82c06f7a0 0x7fd82c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd832616640 1 -- 172.21.15.46:0/1549700746 >> v1:172.21.15.46:6789/0 conn(0x7fd82c0c6bd0 legacy=0x7fd82c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd832616640 1 -- 172.21.15.46:0/1549700746 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c0c7390 con 0x7fd82c071110 2023-10-30T13:35:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.785+0000 7fd830e13640 1 -- 172.21.15.46:0/1549700746 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803997224 0 0) 0x7fd82c066cf0 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd832616640 1 --2- 172.21.15.46:0/1549700746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd82c071110 0x7fd82c158110 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd82c06c830 tx=0x7fd82401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d433442392b346ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd830e13640 1 -- 172.21.15.46:0/1549700746 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240b0440 con 0x7fd82c071110 2023-10-30T13:35:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd830e13640 1 -- 172.21.15.46:0/1549700746 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8240b05a0 con 0x7fd82c071110 2023-10-30T13:35:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd830e13640 1 -- 172.21.15.46:0/1549700746 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240b0930 con 0x7fd82c071110 2023-10-30T13:35:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd834079640 1 -- 172.21.15.46:0/1549700746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd82c071110 msgr2=0x7fd82c158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd834079640 1 --2- 172.21.15.46:0/1549700746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd82c071110 0x7fd82c158110 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd82c06c830 tx=0x7fd82401f0f0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd834079640 1 -- 172.21.15.46:0/1549700746 shutdown_connections 2023-10-30T13:35:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd834079640 1 --2- 172.21.15.46:0/1549700746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd82c071110 0x7fd82c158110 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd834079640 1 --2- 172.21.15.46:0/1549700746 >> v2:172.21.15.154:3300/0 conn(0x7fd82c06f7a0 0x7fd82c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd834079640 1 -- 172.21.15.46:0/1549700746 >> 172.21.15.46:0/1549700746 conn(0x7fd82c0c7910 msgr2=0x7fd82c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd834079640 1 -- 172.21.15.46:0/1549700746 shutdown_connections 2023-10-30T13:35:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd834079640 1 -- 172.21.15.46:0/1549700746 wait complete. 2023-10-30T13:35:11.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd834079640 1 Processor -- start 2023-10-30T13:35:11.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.789+0000 7fd834079640 1 -- start start 2023-10-30T13:35:11.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd834079640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd82c06f7a0 0x7fd82c154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd834079640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd82c071110 0x7fd82c0ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd834079640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd82c0c6bd0 0x7fd82c152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd834079640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c153d30 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd834079640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c154580 con 0x7fd82c06f7a0 2023-10-30T13:35:11.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd834079640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c1529a0 con 0x7fd82c071110 2023-10-30T13:35:11.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd832616640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd82c0c6bd0 0x7fd82c152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd832616640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd82c0c6bd0 0x7fd82c152e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39094/0 (socket says 172.21.15.46:39094) 2023-10-30T13:35:11.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd832616640 1 -- 172.21.15.46:0/1425276391 learned_addr learned my addr 172.21.15.46:0/1425276391 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd831e15640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd82c06f7a0 0x7fd82c154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd831614640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd82c071110 0x7fd82c0ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd832616640 1 -- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd82c071110 msgr2=0x7fd82c0ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd832616640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd82c071110 0x7fd82c0ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd832616640 1 -- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd82c06f7a0 msgr2=0x7fd82c154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd832616640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd82c06f7a0 0x7fd82c154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd832616640 1 -- 172.21.15.46:0/1425276391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd824009a70 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd832616640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd82c0c6bd0 0x7fd82c152e30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd82401bd00 tx=0x7fd82401be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd8227fc640 1 -- 172.21.15.46:0/1425276391 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240c2070 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd8227fc640 1 -- 172.21.15.46:0/1425276391 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8240ba410 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd8227fc640 1 -- 172.21.15.46:0/1425276391 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240b0440 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd834079640 1 -- 172.21.15.46:0/1425276391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd82c1f2ce0 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd834079640 1 -- 172.21.15.46:0/1425276391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82c1e62b0 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd834079640 1 -- 172.21.15.46:0/1425276391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82c04e170 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.793+0000 7fd8227fc640 1 -- 172.21.15.46:0/1425276391 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd8240b7050 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.797+0000 7fd8227fc640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8100664d0 0x7fd810068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:11.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.797+0000 7fd8227fc640 1 -- 172.21.15.46:0/1425276391 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fd824121bf0 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.797+0000 7fd831e15640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8100664d0 0x7fd810068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:11.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.797+0000 7fd831e15640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8100664d0 0x7fd810068990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd828009ac0 tx=0x7fd828009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:11.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.801+0000 7fd8227fc640 1 -- 172.21.15.46:0/1425276391 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd8240eee10 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.989+0000 7fd834079640 1 -- 172.21.15.46:0/1425276391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd82c152ae0 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.989+0000 7fd8227fc640 1 -- 172.21.15.46:0/1425276391 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5734 (secure 0 0 0) 0x7fd8240f2430 con 0x7fd82c0c6bd0 2023-10-30T13:35:11.996 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:11.996 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:03.892152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:11.996 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 61 2023-10-30T13:35:11.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.993+0000 7fd834079640 1 -- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8100664d0 msgr2=0x7fd810068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.993+0000 7fd834079640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8100664d0 0x7fd810068990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd828009ac0 tx=0x7fd828009190 comp rx=0 tx=0).stop 2023-10-30T13:35:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.993+0000 7fd834079640 1 -- 172.21.15.46:0/1425276391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd82c0c6bd0 msgr2=0x7fd82c152e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.993+0000 7fd834079640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd82c0c6bd0 0x7fd82c152e30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd82401bd00 tx=0x7fd82401be90 comp rx=0 tx=0).stop 2023-10-30T13:35:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.993+0000 7fd834079640 1 -- 172.21.15.46:0/1425276391 shutdown_connections 2023-10-30T13:35:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.993+0000 7fd834079640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8100664d0 0x7fd810068990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.993+0000 7fd834079640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd82c0c6bd0 0x7fd82c152e30 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.993+0000 7fd834079640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd82c071110 0x7fd82c0ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.993+0000 7fd834079640 1 --2- 172.21.15.46:0/1425276391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd82c06f7a0 0x7fd82c154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:11.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.993+0000 7fd834079640 1 -- 172.21.15.46:0/1425276391 >> 172.21.15.46:0/1425276391 conn(0x7fd82c0c7910 msgr2=0x7fd82c0c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:11.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.993+0000 7fd834079640 1 -- 172.21.15.46:0/1425276391 shutdown_connections 2023-10-30T13:35:12.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:11.993+0000 7fd834079640 1 -- 172.21.15.46:0/1425276391 wait complete. 2023-10-30T13:35:12.016 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:14.017 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.165+0000 7f9631303640 1 Processor -- start 2023-10-30T13:35:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.165+0000 7f9631303640 1 -- start start 2023-10-30T13:35:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f9631303640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f962c071b90 0x7f962c15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f9631303640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c15b060 con 0x7f962c0c6290 2023-10-30T13:35:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f9631303640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c0c4ab0 con 0x7f962c0717b0 2023-10-30T13:35:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f9631303640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f962c157530 con 0x7f962c071b90 2023-10-30T13:35:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f9630b02640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f962c0c6290 0x7f962c161eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33504/0 (socket says 172.21.15.46:33504) 2023-10-30T13:35:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f9630b02640 1 -- 172.21.15.46:0/1289542461 learned_addr learned my addr 172.21.15.46:0/1289542461 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:35:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962b7fe640 1 --2- 172.21.15.46:0/1289542461 >> v2:172.21.15.154:3300/0 conn(0x7f962c071b90 0x7f962c15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506532693 0 0) 0x7f962c15b060 con 0x7f962c0c6290 2023-10-30T13:35:14.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9614003530 con 0x7f962c0c6290 2023-10-30T13:35:14.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3099248272 0 0) 0x7f9614003530 con 0x7f962c0c6290 2023-10-30T13:35:14.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 >> v2:172.21.15.154:3300/0 conn(0x7f962c071b90 msgr2=0x7f962c15ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:14.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 --2- 172.21.15.46:0/1289542461 >> v2:172.21.15.154:3300/0 conn(0x7f962c071b90 0x7f962c15ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 >> v1:172.21.15.154:6789/0 conn(0x7f962c0717b0 legacy=0x7f962c1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c0c4d90 con 0x7f962c0c6290 2023-10-30T13:35:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f9624001a00 con 0x7f962c0c6290 2023-10-30T13:35:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 >> v1:172.21.15.46:6789/0 conn(0x7f962c0c6290 legacy=0x7f962c161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 --2- 172.21.15.46:0/1289542461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9614003980 0x7f9614003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 --2- 172.21.15.46:0/1289542461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614004380 0x7f9614004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 --2- 172.21.15.46:0/1289542461 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9614005050 0x7f961400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961400bed0 con 0x7f9614004380 2023-10-30T13:35:14.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961400c040 con 0x7f9614003980 2023-10-30T13:35:14.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961400c200 con 0x7f9614005050 2023-10-30T13:35:14.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962b7fe640 1 --2- 172.21.15.46:0/1289542461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614004380 0x7f9614004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962bfff640 1 --2- 172.21.15.46:0/1289542461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9614003980 0x7f9614003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f9630b02640 1 --2- 172.21.15.46:0/1289542461 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9614005050 0x7f961400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962b7fe640 1 -- 172.21.15.46:0/1289542461 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9614005050 msgr2=0x7f961400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:14.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962b7fe640 1 --2- 172.21.15.46:0/1289542461 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9614005050 0x7f961400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962b7fe640 1 -- 172.21.15.46:0/1289542461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9614003980 msgr2=0x7f9614003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:14.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962b7fe640 1 --2- 172.21.15.46:0/1289542461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9614003980 0x7f9614003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962b7fe640 1 -- 172.21.15.46:0/1289542461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614003530 con 0x7f9614004380 2023-10-30T13:35:14.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f962bfff640 1 --2- 172.21.15.46:0/1289542461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9614003980 0x7f9614003d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:14.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.169+0000 7f9630b02640 1 -- 172.21.15.46:0/1289542461 reap_dead start 2023-10-30T13:35:14.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f962b7fe640 1 --2- 172.21.15.46:0/1289542461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614004380 0x7f9614004850 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9620009ae0 tx=0x7f962001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19fa1fe900a1d0a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:14.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96200a5530 con 0x7f9614004380 2023-10-30T13:35:14.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96200a5690 con 0x7f9614004380 2023-10-30T13:35:14.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f962affd640 1 -- 172.21.15.46:0/1289542461 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96200a5a40 con 0x7f9614004380 2023-10-30T13:35:14.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f9631303640 1 -- 172.21.15.46:0/1289542461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614004380 msgr2=0x7f9614004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f9631303640 1 --2- 172.21.15.46:0/1289542461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614004380 0x7f9614004850 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9620009ae0 tx=0x7f962001ca50 comp rx=0 tx=0).stop 2023-10-30T13:35:14.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f9631303640 1 -- 172.21.15.46:0/1289542461 shutdown_connections 2023-10-30T13:35:14.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f9631303640 1 --2- 172.21.15.46:0/1289542461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9614004380 0x7f9614004850 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f9631303640 1 -- 172.21.15.46:0/1289542461 >> 172.21.15.46:0/1289542461 conn(0x7f962c0c75d0 msgr2=0x7f962c0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:14.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f9631303640 1 -- 172.21.15.46:0/1289542461 shutdown_connections 2023-10-30T13:35:14.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f9631303640 1 -- 172.21.15.46:0/1289542461 wait complete. 2023-10-30T13:35:14.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f9631303640 1 Processor -- start 2023-10-30T13:35:14.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.173+0000 7f9631303640 1 -- start start 2023-10-30T13:35:14.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9631303640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9614004380 0x7f962c15b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9631303640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f962c14a5f0 0x7f962c14b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9631303640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f962c14bb70 0x7f962c1f1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9631303640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c0c93d0 con 0x7f962c14a5f0 2023-10-30T13:35:14.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9631303640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c14bff0 con 0x7f9614004380 2023-10-30T13:35:14.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9631303640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962c1587f0 con 0x7f962c14bb70 2023-10-30T13:35:14.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f962b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f962c14a5f0 0x7f962c14b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f962b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f962c14a5f0 0x7f962c14b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39136/0 (socket says 172.21.15.46:39136) 2023-10-30T13:35:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f962b7fe640 1 -- 172.21.15.46:0/465725698 learned_addr learned my addr 172.21.15.46:0/465725698 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f962bfff640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9614004380 0x7f962c15b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9630b02640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f962c14bb70 0x7f962c1f1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f962b7fe640 1 -- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f962c14bb70 msgr2=0x7f962c1f1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f962b7fe640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f962c14bb70 0x7f962c1f1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f962b7fe640 1 -- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9614004380 msgr2=0x7f962c15b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f962b7fe640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9614004380 0x7f962c15b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f962b7fe640 1 -- 172.21.15.46:0/465725698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c0c95c0 con 0x7f962c14a5f0 2023-10-30T13:35:14.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f962b7fe640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f962c14a5f0 0x7f962c14b630 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f962001c880 tx=0x7f96200197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:14.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9628ff9640 1 -- 172.21.15.46:0/465725698 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96200bd070 con 0x7f962c14a5f0 2023-10-30T13:35:14.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9628ff9640 1 -- 172.21.15.46:0/465725698 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9620004020 con 0x7f962c14a5f0 2023-10-30T13:35:14.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9628ff9640 1 -- 172.21.15.46:0/465725698 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96200043d0 con 0x7f962c14a5f0 2023-10-30T13:35:14.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9631303640 1 -- 172.21.15.46:0/465725698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962c1f2be0 con 0x7f962c14a5f0 2023-10-30T13:35:14.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9631303640 1 -- 172.21.15.46:0/465725698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962c14a1d0 con 0x7f962c14a5f0 2023-10-30T13:35:14.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.177+0000 7f9631303640 1 -- 172.21.15.46:0/465725698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962c14a3c0 con 0x7f962c14a5f0 2023-10-30T13:35:14.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.181+0000 7f9628ff9640 1 -- 172.21.15.46:0/465725698 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f96200b4070 con 0x7f962c14a5f0 2023-10-30T13:35:14.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.181+0000 7f9628ff9640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96180664a0 0x7f9618068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:14.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.181+0000 7f9628ff9640 1 -- 172.21.15.46:0/465725698 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f962011e410 con 0x7f962c14a5f0 2023-10-30T13:35:14.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.181+0000 7f962bfff640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96180664a0 0x7f9618068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:14.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.181+0000 7f962bfff640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96180664a0 0x7f9618068960 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f961c000c10 tx=0x7f961c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:14.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.185+0000 7f9628ff9640 1 -- 172.21.15.46:0/465725698 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f96200eb620 con 0x7f962c14a5f0 2023-10-30T13:35:14.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.373+0000 7f9631303640 1 -- 172.21.15.46:0/465725698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f962c1579d0 con 0x7f962c14a5f0 2023-10-30T13:35:14.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.373+0000 7f9628ff9640 1 -- 172.21.15.46:0/465725698 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f96200f5030 con 0x7f962c14a5f0 2023-10-30T13:35:14.380 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:14.380 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:11.030402+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:resolve","state_seq":16,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:14.380 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 62 2023-10-30T13:35:14.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.377+0000 7f9631303640 1 -- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96180664a0 msgr2=0x7f9618068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.377+0000 7f9631303640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96180664a0 0x7f9618068960 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f961c000c10 tx=0x7f961c009040 comp rx=0 tx=0).stop 2023-10-30T13:35:14.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.377+0000 7f9631303640 1 -- 172.21.15.46:0/465725698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f962c14a5f0 msgr2=0x7f962c14b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:14.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.377+0000 7f9631303640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f962c14a5f0 0x7f962c14b630 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f962001c880 tx=0x7f96200197f0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.377+0000 7f9631303640 1 -- 172.21.15.46:0/465725698 shutdown_connections 2023-10-30T13:35:14.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.377+0000 7f9631303640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96180664a0 0x7f9618068960 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.377+0000 7f9631303640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f962c14bb70 0x7f962c1f1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.377+0000 7f9631303640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f962c14a5f0 0x7f962c14b630 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.377+0000 7f9631303640 1 --2- 172.21.15.46:0/465725698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9614004380 0x7f962c15b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:14.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.377+0000 7f9631303640 1 -- 172.21.15.46:0/465725698 >> 172.21.15.46:0/465725698 conn(0x7f962c0c75d0 msgr2=0x7f962c0c9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:14.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.377+0000 7f9631303640 1 -- 172.21.15.46:0/465725698 shutdown_connections 2023-10-30T13:35:14.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:14.377+0000 7f9631303640 1 -- 172.21.15.46:0/465725698 wait complete. 2023-10-30T13:35:14.400 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:16.402 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:16.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.549+0000 7fa106772640 1 Processor -- start 2023-10-30T13:35:16.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.549+0000 7fa106772640 1 -- start start 2023-10-30T13:35:16.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa106772640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa100156f60 0x7fa100071b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa106772640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa10006fd50 0x7fa100072050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa106772640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa10006afa0 0x7fa10015ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa106772640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa10006bea0 con 0x7fa10006fd50 2023-10-30T13:35:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa106772640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1000caa80 con 0x7fa100156f60 2023-10-30T13:35:16.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa106772640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa100157340 con 0x7fa10006afa0 2023-10-30T13:35:16.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa0ff7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa10006fd50 0x7fa100072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa0ff7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa10006fd50 0x7fa100072050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44232/0 (socket says 172.21.15.46:44232) 2023-10-30T13:35:16.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa0ff7fe640 1 -- 172.21.15.46:0/3844775101 learned_addr learned my addr 172.21.15.46:0/3844775101 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:16.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa0fffff640 1 --2- 172.21.15.46:0/3844775101 >> v2:172.21.15.154:3300/0 conn(0x7fa100156f60 0x7fa100071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa104d0f640 1 --2- 172.21.15.46:0/3844775101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa10006afa0 0x7fa10015ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa0ff7fe640 1 -- 172.21.15.46:0/3844775101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa10006afa0 msgr2=0x7fa10015ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa0ff7fe640 1 --2- 172.21.15.46:0/3844775101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa10006afa0 0x7fa10015ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa0ff7fe640 1 -- 172.21.15.46:0/3844775101 >> v2:172.21.15.154:3300/0 conn(0x7fa100156f60 msgr2=0x7fa100071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa0ff7fe640 1 --2- 172.21.15.46:0/3844775101 >> v2:172.21.15.154:3300/0 conn(0x7fa100156f60 0x7fa100071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa0ff7fe640 1 -- 172.21.15.46:0/3844775101 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10006ca30 con 0x7fa10006fd50 2023-10-30T13:35:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.553+0000 7fa104d0f640 1 --2- 172.21.15.46:0/3844775101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa10006afa0 0x7fa10015ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:16.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa0ff7fe640 1 --2- 172.21.15.46:0/3844775101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006fd50 0x7fa100072050 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa1000ce070 tx=0x7fa0f401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ff0c7b131e3f79 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa0feffd640 1 -- 172.21.15.46:0/3844775101 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f40ac410 con 0x7fa10006fd50 2023-10-30T13:35:16.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa0feffd640 1 -- 172.21.15.46:0/3844775101 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f40ac570 con 0x7fa10006fd50 2023-10-30T13:35:16.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa0feffd640 1 -- 172.21.15.46:0/3844775101 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f40ac900 con 0x7fa10006fd50 2023-10-30T13:35:16.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa106772640 1 -- 172.21.15.46:0/3844775101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006fd50 msgr2=0x7fa100072050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa106772640 1 --2- 172.21.15.46:0/3844775101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006fd50 0x7fa100072050 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa1000ce070 tx=0x7fa0f401c4e0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa106772640 1 -- 172.21.15.46:0/3844775101 shutdown_connections 2023-10-30T13:35:16.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa106772640 1 --2- 172.21.15.46:0/3844775101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa10006afa0 0x7fa10015ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa106772640 1 --2- 172.21.15.46:0/3844775101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006fd50 0x7fa100072050 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa106772640 1 --2- 172.21.15.46:0/3844775101 >> v2:172.21.15.154:3300/0 conn(0x7fa100156f60 0x7fa100071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa106772640 1 -- 172.21.15.46:0/3844775101 >> 172.21.15.46:0/3844775101 conn(0x7fa100069e90 msgr2=0x7fa10006a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa106772640 1 -- 172.21.15.46:0/3844775101 shutdown_connections 2023-10-30T13:35:16.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa106772640 1 -- 172.21.15.46:0/3844775101 wait complete. 2023-10-30T13:35:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa106772640 1 Processor -- start 2023-10-30T13:35:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.557+0000 7fa106772640 1 -- start start 2023-10-30T13:35:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa106772640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006afa0 0x7fa1001f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa106772640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa10006fd50 0x7fa1000c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa106772640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa100156f60 0x7fa1000c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa106772640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1001567b0 con 0x7fa10006afa0 2023-10-30T13:35:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa106772640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10015d1c0 con 0x7fa10006fd50 2023-10-30T13:35:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa106772640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1001f5f10 con 0x7fa100156f60 2023-10-30T13:35:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006afa0 0x7fa1001f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006afa0 0x7fa1001f5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44242/0 (socket says 172.21.15.46:44242) 2023-10-30T13:35:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fffff640 1 -- 172.21.15.46:0/3632963487 learned_addr learned my addr 172.21.15.46:0/3632963487 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0ff7fe640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa10006fd50 0x7fa1000c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa104d0f640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa100156f60 0x7fa1000c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fffff640 1 -- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa100156f60 msgr2=0x7fa1000c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fffff640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa100156f60 0x7fa1000c8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fffff640 1 -- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa10006fd50 msgr2=0x7fa1000c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fffff640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa10006fd50 0x7fa1000c88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fffff640 1 -- 172.21.15.46:0/3632963487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f40094d0 con 0x7fa10006afa0 2023-10-30T13:35:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0ff7fe640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa10006fd50 0x7fa1000c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fffff640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006afa0 0x7fa1001f5260 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000d5f0 tx=0x7fa0f000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fcff9640 1 -- 172.21.15.46:0/3632963487 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f00125f0 con 0x7fa10006afa0 2023-10-30T13:35:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fcff9640 1 -- 172.21.15.46:0/3632963487 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f0012750 con 0x7fa10006afa0 2023-10-30T13:35:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa106772640 1 -- 172.21.15.46:0/3632963487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1000c8460 con 0x7fa10006afa0 2023-10-30T13:35:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fcff9640 1 -- 172.21.15.46:0/3632963487 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0012ae0 con 0x7fa10006afa0 2023-10-30T13:35:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa106772640 1 -- 172.21.15.46:0/3632963487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1001f5c30 con 0x7fa10006afa0 2023-10-30T13:35:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa106772640 1 -- 172.21.15.46:0/3632963487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa10004e170 con 0x7fa10006afa0 2023-10-30T13:35:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.561+0000 7fa0fcff9640 1 -- 172.21.15.46:0/3632963487 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa0f0012c40 con 0x7fa10006afa0 2023-10-30T13:35:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.565+0000 7fa0fcff9640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0ec0664a0 0x7fa0ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.565+0000 7fa0fcff9640 1 -- 172.21.15.46:0/3632963487 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fa0f00846b0 con 0x7fa10006afa0 2023-10-30T13:35:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.565+0000 7fa0ff7fe640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0ec0664a0 0x7fa0ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:16.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.565+0000 7fa0ff7fe640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0ec0664a0 0x7fa0ec068960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40ae000 tx=0x7fa0f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:16.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.569+0000 7fa0fcff9640 1 -- 172.21.15.46:0/3632963487 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa0f00517a0 con 0x7fa10006afa0 2023-10-30T13:35:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.753+0000 7fa106772640 1 -- 172.21.15.46:0/3632963487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa10006fb30 con 0x7fa10006afa0 2023-10-30T13:35:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa0fcff9640 1 -- 172.21.15.46:0/3632963487 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5736 (secure 0 0 0) 0x7fa0f0054dc0 con 0x7fa10006afa0 2023-10-30T13:35:16.761 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:16.761 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:11.030402+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:resolve","state_seq":16,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:16.762 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 62 2023-10-30T13:35:16.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa106772640 1 -- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0ec0664a0 msgr2=0x7fa0ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa106772640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0ec0664a0 0x7fa0ec068960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40ae000 tx=0x7fa0f4008040 comp rx=0 tx=0).stop 2023-10-30T13:35:16.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa106772640 1 -- 172.21.15.46:0/3632963487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006afa0 msgr2=0x7fa1001f5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:16.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa106772640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006afa0 0x7fa1001f5260 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000d5f0 tx=0x7fa0f000dab0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa106772640 1 -- 172.21.15.46:0/3632963487 shutdown_connections 2023-10-30T13:35:16.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa106772640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0ec0664a0 0x7fa0ec068960 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa106772640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa100156f60 0x7fa1000c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa106772640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa10006fd50 0x7fa1000c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa106772640 1 --2- 172.21.15.46:0/3632963487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006afa0 0x7fa1001f5260 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:16.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa106772640 1 -- 172.21.15.46:0/3632963487 >> 172.21.15.46:0/3632963487 conn(0x7fa100069e90 msgr2=0x7fa100154170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:16.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa106772640 1 -- 172.21.15.46:0/3632963487 shutdown_connections 2023-10-30T13:35:16.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:16.757+0000 7fa106772640 1 -- 172.21.15.46:0/3632963487 wait complete. 2023-10-30T13:35:16.782 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 62, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 62, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5442": { "addr": "172.21.15.46:6836/3955931693", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3955931693, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3955931693, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5442, "incarnation": 46, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 13 }, "gid_5574": { "addr": "172.21.15.46:6837/126953235", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 126953235, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 126953235, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5574, "incarnation": 55, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_5721": { "addr": "172.21.15.46:6833/4044310929", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4044310929, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4044310929, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5721, "incarnation": 61, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:resolve", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:35:11.030402+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5721, "mds_1": 5574, "mds_2": 5442 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:35:16.885 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:35:16.889 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:35:18.784 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.933+0000 7fa08550f640 1 Processor -- start 2023-10-30T13:35:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa08550f640 1 -- start start 2023-10-30T13:35:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa08550f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa080070a00 0x7fa080070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa08550f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa080069e90 0x7fa08006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa08550f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0800713b0 con 0x7fa0801539a0 2023-10-30T13:35:18.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa08550f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa08006fbb0 con 0x7fa080070a00 2023-10-30T13:35:18.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa08550f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa080153d80 con 0x7fa080069e90 2023-10-30T13:35:18.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa07f7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa080069e90 0x7fa08006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa077fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa080070a00 0x7fa080070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa07effd640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa0801539a0 0x7fa08015a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:58176/0 (socket says 172.21.15.46:58176) 2023-10-30T13:35:18.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa07effd640 1 -- 172.21.15.46:0/405800117 learned_addr learned my addr 172.21.15.46:0/405800117 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:35:18.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa07f7fe640 1 -- 172.21.15.46:0/405800117 >> v2:172.21.15.154:3300/0 conn(0x7fa080070a00 msgr2=0x7fa080070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:18.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa07f7fe640 1 --2- 172.21.15.46:0/405800117 >> v2:172.21.15.154:3300/0 conn(0x7fa080070a00 0x7fa080070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa07f7fe640 1 -- 172.21.15.46:0/405800117 >> v1:172.21.15.154:6789/0 conn(0x7fa0801539a0 legacy=0x7fa08015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa07f7fe640 1 -- 172.21.15.46:0/405800117 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0800c5c50 con 0x7fa080069e90 2023-10-30T13:35:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.937+0000 7fa077fff640 1 --2- 172.21.15.46:0/405800117 >> v2:172.21.15.154:3300/0 conn(0x7fa080070a00 0x7fa080070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:35:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.941+0000 7fa07f7fe640 1 --2- 172.21.15.46:0/405800117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa080069e90 0x7fa08006a300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa078009d90 tx=0x7fa07801c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39d70a76c284aa07 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.941+0000 7fa07e7fc640 1 -- 172.21.15.46:0/405800117 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0780a5860 con 0x7fa080069e90 2023-10-30T13:35:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.941+0000 7fa07e7fc640 1 -- 172.21.15.46:0/405800117 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0780a59c0 con 0x7fa080069e90 2023-10-30T13:35:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.941+0000 7fa07e7fc640 1 -- 172.21.15.46:0/405800117 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0780a5d50 con 0x7fa080069e90 2023-10-30T13:35:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.941+0000 7fa08550f640 1 -- 172.21.15.46:0/405800117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa080069e90 msgr2=0x7fa08006a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.941+0000 7fa08550f640 1 --2- 172.21.15.46:0/405800117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa080069e90 0x7fa08006a300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa078009d90 tx=0x7fa07801c890 comp rx=0 tx=0).stop 2023-10-30T13:35:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.941+0000 7fa08550f640 1 -- 172.21.15.46:0/405800117 shutdown_connections 2023-10-30T13:35:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.941+0000 7fa08550f640 1 --2- 172.21.15.46:0/405800117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa080069e90 0x7fa08006a300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.941+0000 7fa08550f640 1 --2- 172.21.15.46:0/405800117 >> v2:172.21.15.154:3300/0 conn(0x7fa080070a00 0x7fa080070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.941+0000 7fa08550f640 1 -- 172.21.15.46:0/405800117 >> 172.21.15.46:0/405800117 conn(0x7fa0800c7910 msgr2=0x7fa0800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.941+0000 7fa08550f640 1 -- 172.21.15.46:0/405800117 shutdown_connections 2023-10-30T13:35:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.941+0000 7fa08550f640 1 -- 172.21.15.46:0/405800117 wait complete. 2023-10-30T13:35:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa08550f640 1 Processor -- start 2023-10-30T13:35:18.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa08550f640 1 -- start start 2023-10-30T13:35:18.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa08550f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa080069e90 0x7fa080145a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa08550f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa080070a00 0x7fa0801f55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa08550f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0801539a0 0x7fa0801436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa08550f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa080159760 con 0x7fa080070a00 2023-10-30T13:35:18.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa08550f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa080158610 con 0x7fa080069e90 2023-10-30T13:35:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa08550f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa080157f10 con 0x7fa0801539a0 2023-10-30T13:35:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa077fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa080070a00 0x7fa0801f55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa077fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa080070a00 0x7fa0801f55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44256/0 (socket says 172.21.15.46:44256) 2023-10-30T13:35:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa077fff640 1 -- 172.21.15.46:0/3428487624 learned_addr learned my addr 172.21.15.46:0/3428487624 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa07effd640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa080069e90 0x7fa080145a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa07f7fe640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0801539a0 0x7fa0801436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa077fff640 1 -- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0801539a0 msgr2=0x7fa0801436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa077fff640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0801539a0 0x7fa0801436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa077fff640 1 -- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa080069e90 msgr2=0x7fa080145a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa077fff640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa080069e90 0x7fa080145a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa077fff640 1 -- 172.21.15.46:0/3428487624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa068009420 con 0x7fa080070a00 2023-10-30T13:35:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa07f7fe640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0801539a0 0x7fa0801436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa077fff640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa080070a00 0x7fa0801f55d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa06800ebe0 tx=0x7fa06800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa0777fe640 1 -- 172.21.15.46:0/3428487624 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06800eee0 con 0x7fa080070a00 2023-10-30T13:35:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa0777fe640 1 -- 172.21.15.46:0/3428487624 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06800cc30 con 0x7fa080070a00 2023-10-30T13:35:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa08550f640 1 -- 172.21.15.46:0/3428487624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa078009a70 con 0x7fa080070a00 2023-10-30T13:35:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa0777fe640 1 -- 172.21.15.46:0/3428487624 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068013470 con 0x7fa080070a00 2023-10-30T13:35:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa08550f640 1 -- 172.21.15.46:0/3428487624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa080158b90 con 0x7fa080070a00 2023-10-30T13:35:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.945+0000 7fa08550f640 1 -- 172.21.15.46:0/3428487624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08004e170 con 0x7fa080070a00 2023-10-30T13:35:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.949+0000 7fa0777fe640 1 -- 172.21.15.46:0/3428487624 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa0680135d0 con 0x7fa080070a00 2023-10-30T13:35:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.949+0000 7fa0777fe640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0600664a0 0x7fa060068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.949+0000 7fa0777fe640 1 -- 172.21.15.46:0/3428487624 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fa068084390 con 0x7fa080070a00 2023-10-30T13:35:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.949+0000 7fa07effd640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0600664a0 0x7fa060068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.949+0000 7fa07effd640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0600664a0 0x7fa060068960 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa070005890 tx=0x7fa070009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:18.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:18.953+0000 7fa0777fe640 1 -- 172.21.15.46:0/3428487624 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa0680515c0 con 0x7fa080070a00 2023-10-30T13:35:19.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.145+0000 7fa08550f640 1 -- 172.21.15.46:0/3428487624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0800c72f0 con 0x7fa080070a00 2023-10-30T13:35:19.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.145+0000 7fa0777fe640 1 -- 172.21.15.46:0/3428487624 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fa068054be0 con 0x7fa080070a00 2023-10-30T13:35:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:19.152 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:17.257386+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:19.152 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 63 2023-10-30T13:35:19.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.149+0000 7fa08550f640 1 -- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0600664a0 msgr2=0x7fa060068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.149+0000 7fa08550f640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0600664a0 0x7fa060068960 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa070005890 tx=0x7fa070009040 comp rx=0 tx=0).stop 2023-10-30T13:35:19.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.149+0000 7fa08550f640 1 -- 172.21.15.46:0/3428487624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa080070a00 msgr2=0x7fa0801f55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:19.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.149+0000 7fa08550f640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa080070a00 0x7fa0801f55d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa06800ebe0 tx=0x7fa06800c520 comp rx=0 tx=0).stop 2023-10-30T13:35:19.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.149+0000 7fa08550f640 1 -- 172.21.15.46:0/3428487624 shutdown_connections 2023-10-30T13:35:19.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.149+0000 7fa08550f640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0600664a0 0x7fa060068960 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.149+0000 7fa08550f640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0801539a0 0x7fa0801436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.149+0000 7fa08550f640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa080070a00 0x7fa0801f55d0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.149+0000 7fa08550f640 1 --2- 172.21.15.46:0/3428487624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa080069e90 0x7fa080145a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.149+0000 7fa08550f640 1 -- 172.21.15.46:0/3428487624 >> 172.21.15.46:0/3428487624 conn(0x7fa0800c7910 msgr2=0x7fa080150fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.149+0000 7fa08550f640 1 -- 172.21.15.46:0/3428487624 shutdown_connections 2023-10-30T13:35:19.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:19.149+0000 7fa08550f640 1 -- 172.21.15.46:0/3428487624 wait complete. 2023-10-30T13:35:19.172 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:21.173 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:21.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.321+0000 7f0c9371d640 1 Processor -- start 2023-10-30T13:35:21.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.321+0000 7f0c9371d640 1 -- start start 2023-10-30T13:35:21.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c9371d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c8c151800 0x7f0c8c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c9371d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0c8c0c1ee0 0x7f0c8c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c9371d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c8c06e090 con 0x7f0c8c0c9230 2023-10-30T13:35:21.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c9371d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c8c06dc70 con 0x7f0c8c151800 2023-10-30T13:35:21.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c9371d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c8c0c8a50 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c914b9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c8c151800 0x7f0c8c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c914b9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c8c151800 0x7f0c8c06e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44270/0 (socket says 172.21.15.46:44270) 2023-10-30T13:35:21.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c914b9640 1 -- 172.21.15.46:0/29377989 learned_addr learned my addr 172.21.15.46:0/29377989 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:21.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c90cb8640 1 --2- 172.21.15.46:0/29377989 >> v2:172.21.15.154:3300/0 conn(0x7f0c8c0c1ee0 0x7f0c8c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c914b9640 1 -- 172.21.15.46:0/29377989 >> v2:172.21.15.154:3300/0 conn(0x7f0c8c0c1ee0 msgr2=0x7f0c8c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:21.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c914b9640 1 --2- 172.21.15.46:0/29377989 >> v2:172.21.15.154:3300/0 conn(0x7f0c8c0c1ee0 0x7f0c8c0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c914b9640 1 -- 172.21.15.46:0/29377989 >> v1:172.21.15.154:6789/0 conn(0x7f0c8c0c9230 legacy=0x7f0c8c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:21.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c914b9640 1 -- 172.21.15.46:0/29377989 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8c0c3b10 con 0x7f0c8c151800 2023-10-30T13:35:21.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c914b9640 1 --2- 172.21.15.46:0/29377989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c8c151800 0x7f0c8c06e470 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c070110 tx=0x7f0c8801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a142325e7ceb9d6b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:21.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c83fff640 1 -- 172.21.15.46:0/29377989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880ab670 con 0x7f0c8c151800 2023-10-30T13:35:21.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.325+0000 7f0c83fff640 1 -- 172.21.15.46:0/29377989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c880ab7d0 con 0x7f0c8c151800 2023-10-30T13:35:21.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c83fff640 1 -- 172.21.15.46:0/29377989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880abb60 con 0x7f0c8c151800 2023-10-30T13:35:21.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 -- 172.21.15.46:0/29377989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c8c151800 msgr2=0x7f0c8c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 --2- 172.21.15.46:0/29377989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c8c151800 0x7f0c8c06e470 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c070110 tx=0x7f0c8801c470 comp rx=0 tx=0).stop 2023-10-30T13:35:21.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 -- 172.21.15.46:0/29377989 shutdown_connections 2023-10-30T13:35:21.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 --2- 172.21.15.46:0/29377989 >> v2:172.21.15.154:3300/0 conn(0x7f0c8c0c1ee0 0x7f0c8c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 --2- 172.21.15.46:0/29377989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c8c151800 0x7f0c8c06e470 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 -- 172.21.15.46:0/29377989 >> 172.21.15.46:0/29377989 conn(0x7f0c8c071ea0 msgr2=0x7f0c8c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:21.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 -- 172.21.15.46:0/29377989 shutdown_connections 2023-10-30T13:35:21.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 -- 172.21.15.46:0/29377989 wait complete. 2023-10-30T13:35:21.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 Processor -- start 2023-10-30T13:35:21.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 -- start start 2023-10-30T13:35:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c8c0c1ee0 0x7f0c8c153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c8c0c9230 0x7f0c8c1f0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c8c151800 0x7f0c8c1f1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c1f0970 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c0c6990 con 0x7f0c8c151800 2023-10-30T13:35:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c9371d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c1f3de0 con 0x7f0c8c0c9230 2023-10-30T13:35:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c914b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c8c0c1ee0 0x7f0c8c153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c914b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c8c0c1ee0 0x7f0c8c153940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44272/0 (socket says 172.21.15.46:44272) 2023-10-30T13:35:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c914b9640 1 -- 172.21.15.46:0/141219676 learned_addr learned my addr 172.21.15.46:0/141219676 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c90cb8640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c8c0c9230 0x7f0c8c1f0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c91cba640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c8c151800 0x7f0c8c1f1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c914b9640 1 -- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c8c0c9230 msgr2=0x7f0c8c1f0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.329+0000 7f0c914b9640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c8c0c9230 0x7f0c8c1f0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c914b9640 1 -- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c8c151800 msgr2=0x7f0c8c1f1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c914b9640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c8c151800 0x7f0c8c1f1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c914b9640 1 -- 172.21.15.46:0/141219676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c880094d0 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c91cba640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c8c151800 0x7f0c8c1f1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:35:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c914b9640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c8c0c1ee0 0x7f0c8c153940 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f0c8801a0a0 tx=0x7f0c8801acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c81ffb640 1 -- 172.21.15.46:0/141219676 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880b4800 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c81ffb640 1 -- 172.21.15.46:0/141219676 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c880b4960 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c9371d640 1 -- 172.21.15.46:0/141219676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8c1f4080 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c81ffb640 1 -- 172.21.15.46:0/141219676 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880b4cf0 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c9371d640 1 -- 172.21.15.46:0/141219676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8c1f4620 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c9371d640 1 -- 172.21.15.46:0/141219676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8c04e170 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c81ffb640 1 -- 172.21.15.46:0/141219676 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c880b3380 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c81ffb640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0c700664d0 0x7f0c70068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.333+0000 7f0c81ffb640 1 -- 172.21.15.46:0/141219676 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f0c8811e960 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.337+0000 7f0c90cb8640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0c700664d0 0x7f0c70068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.337+0000 7f0c90cb8640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0c700664d0 0x7f0c70068990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c147f00 tx=0x7f0c7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.337+0000 7f0c81ffb640 1 -- 172.21.15.46:0/141219676 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0c880eba70 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 -- 172.21.15.46:0/141219676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c8c0c6de0 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c81ffb640 1 -- 172.21.15.46:0/141219676 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f0c880c4020 con 0x7f0c8c0c1ee0 2023-10-30T13:35:21.534 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:21.534 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:17.257386+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:21.534 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 63 2023-10-30T13:35:21.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 -- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0c700664d0 msgr2=0x7f0c70068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0c700664d0 0x7f0c70068990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c147f00 tx=0x7f0c7c008040 comp rx=0 tx=0).stop 2023-10-30T13:35:21.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 -- 172.21.15.46:0/141219676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c8c0c1ee0 msgr2=0x7f0c8c153940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:21.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c8c0c1ee0 0x7f0c8c153940 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f0c8801a0a0 tx=0x7f0c8801acb0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 -- 172.21.15.46:0/141219676 shutdown_connections 2023-10-30T13:35:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0c700664d0 0x7f0c70068990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c8c151800 0x7f0c8c1f1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c8c0c9230 0x7f0c8c1f0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 --2- 172.21.15.46:0/141219676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c8c0c1ee0 0x7f0c8c153940 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 -- 172.21.15.46:0/141219676 >> 172.21.15.46:0/141219676 conn(0x7f0c8c071ea0 msgr2=0x7f0c8c155dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:21.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 -- 172.21.15.46:0/141219676 shutdown_connections 2023-10-30T13:35:21.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:21.529+0000 7f0c9371d640 1 -- 172.21.15.46:0/141219676 wait complete. 2023-10-30T13:35:21.555 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:23.555 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2566fc4640 1 Processor -- start 2023-10-30T13:35:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2566fc4640 1 -- start start 2023-10-30T13:35:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2566fc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f256006e470 0x7f256006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2566fc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f25600c8490 0x7f25600c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2566fc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f256006afa0 con 0x7f2560153d70 2023-10-30T13:35:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2566fc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f256006d780 con 0x7f25600c8490 2023-10-30T13:35:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2566fc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2560154150 con 0x7f256006e470 2023-10-30T13:35:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f255ffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f256006e470 0x7f256006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2565561640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f25600c8490 0x7f25600c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f255ffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f256006e470 0x7f256006ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:36548/0 (socket says 172.21.15.46:36548) 2023-10-30T13:35:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2565561640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f25600c8490 0x7f25600c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:51796/0 (socket says 172.21.15.46:51796) 2023-10-30T13:35:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f255ffff640 1 -- 172.21.15.46:0/308764895 learned_addr learned my addr 172.21.15.46:0/308764895 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2565561640 1 -- 172.21.15.46:0/308764895 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f256006e470 msgr2=0x7f256006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:23.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2565561640 1 --2- 172.21.15.46:0/308764895 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f256006e470 0x7f256006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2565561640 1 -- 172.21.15.46:0/308764895 >> v1:172.21.15.154:6789/0 conn(0x7f2560153d70 legacy=0x7f25600cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f2565561640 1 -- 172.21.15.46:0/308764895 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256006e030 con 0x7f25600c8490 2023-10-30T13:35:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.705+0000 7f255ffff640 1 --2- 172.21.15.46:0/308764895 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f256006e470 0x7f256006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:35:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.709+0000 7f2565561640 1 --2- 172.21.15.46:0/308764895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25600c8490 0x7f25600c8cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2558009d90 tx=0x7f255801c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bd2eda37a94c08c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.709+0000 7f255f7fe640 1 -- 172.21.15.46:0/308764895 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25580a59e0 con 0x7f25600c8490 2023-10-30T13:35:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.709+0000 7f255f7fe640 1 -- 172.21.15.46:0/308764895 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25580a5b40 con 0x7f25600c8490 2023-10-30T13:35:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.709+0000 7f255f7fe640 1 -- 172.21.15.46:0/308764895 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25580ad440 con 0x7f25600c8490 2023-10-30T13:35:23.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.709+0000 7f2566fc4640 1 -- 172.21.15.46:0/308764895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25600c8490 msgr2=0x7f25600c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.709+0000 7f2566fc4640 1 --2- 172.21.15.46:0/308764895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25600c8490 0x7f25600c8cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2558009d90 tx=0x7f255801c920 comp rx=0 tx=0).stop 2023-10-30T13:35:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 -- 172.21.15.46:0/308764895 shutdown_connections 2023-10-30T13:35:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 --2- 172.21.15.46:0/308764895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25600c8490 0x7f25600c8cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 --2- 172.21.15.46:0/308764895 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f256006e470 0x7f256006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 -- 172.21.15.46:0/308764895 >> 172.21.15.46:0/308764895 conn(0x7f2560071ea0 msgr2=0x7f2560065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:23.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 -- 172.21.15.46:0/308764895 shutdown_connections 2023-10-30T13:35:23.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 -- 172.21.15.46:0/308764895 wait complete. 2023-10-30T13:35:23.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 Processor -- start 2023-10-30T13:35:23.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 -- start start 2023-10-30T13:35:23.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f256006e470 0x7f25601524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25600c8490 0x7f256014bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2560153d70 0x7f256014c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256014aa00 con 0x7f25600c8490 2023-10-30T13:35:23.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256014e280 con 0x7f256006e470 2023-10-30T13:35:23.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2566fc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256014ac50 con 0x7f2560153d70 2023-10-30T13:35:23.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f255ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25600c8490 0x7f256014bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f255ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25600c8490 0x7f256014bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44282/0 (socket says 172.21.15.46:44282) 2023-10-30T13:35:23.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f255ffff640 1 -- 172.21.15.46:0/886776276 learned_addr learned my addr 172.21.15.46:0/886776276 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:23.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2564d60640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f256006e470 0x7f25601524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f2565561640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2560153d70 0x7f256014c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f255ffff640 1 -- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2560153d70 msgr2=0x7f256014c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f255ffff640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2560153d70 0x7f256014c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f255ffff640 1 -- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f256006e470 msgr2=0x7f25601524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f255ffff640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f256006e470 0x7f25601524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.713+0000 7f255ffff640 1 -- 172.21.15.46:0/886776276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c0094d0 con 0x7f25600c8490 2023-10-30T13:35:23.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.717+0000 7f255ffff640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25600c8490 0x7f256014bf60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f254c00e9d0 tx=0x7f254c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:23.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.717+0000 7f255d7fa640 1 -- 172.21.15.46:0/886776276 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c00ca60 con 0x7f25600c8490 2023-10-30T13:35:23.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.717+0000 7f255d7fa640 1 -- 172.21.15.46:0/886776276 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254c00cbc0 con 0x7f25600c8490 2023-10-30T13:35:23.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.717+0000 7f255d7fa640 1 -- 172.21.15.46:0/886776276 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c013470 con 0x7f25600c8490 2023-10-30T13:35:23.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.717+0000 7f2566fc4640 1 -- 172.21.15.46:0/886776276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2558009a70 con 0x7f25600c8490 2023-10-30T13:35:23.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.717+0000 7f2566fc4640 1 -- 172.21.15.46:0/886776276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256014b210 con 0x7f25600c8490 2023-10-30T13:35:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.717+0000 7f2566fc4640 1 -- 172.21.15.46:0/886776276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2560067280 con 0x7f25600c8490 2023-10-30T13:35:23.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.717+0000 7f255d7fa640 1 -- 172.21.15.46:0/886776276 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f254c0135d0 con 0x7f25600c8490 2023-10-30T13:35:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.717+0000 7f255d7fa640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25440664a0 0x7f2544068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.717+0000 7f255d7fa640 1 -- 172.21.15.46:0/886776276 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f254c084790 con 0x7f25600c8490 2023-10-30T13:35:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.721+0000 7f2564d60640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25440664a0 0x7f2544068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:23.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.721+0000 7f2564d60640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25440664a0 0x7f2544068960 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2554007fd0 tx=0x7f2554009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:23.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.721+0000 7f255d7fa640 1 -- 172.21.15.46:0/886776276 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f254c0519d0 con 0x7f25600c8490 2023-10-30T13:35:23.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.917+0000 7f2566fc4640 1 -- 172.21.15.46:0/886776276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f256004e170 con 0x7f25600c8490 2023-10-30T13:35:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.917+0000 7f255d7fa640 1 -- 172.21.15.46:0/886776276 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f254c054ff0 con 0x7f25600c8490 2023-10-30T13:35:23.924 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:23.925 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:23.070980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:23.925 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 64 2023-10-30T13:35:23.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.921+0000 7f2566fc4640 1 -- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25440664a0 msgr2=0x7f2544068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.921+0000 7f2566fc4640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25440664a0 0x7f2544068960 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2554007fd0 tx=0x7f2554009040 comp rx=0 tx=0).stop 2023-10-30T13:35:23.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.921+0000 7f2566fc4640 1 -- 172.21.15.46:0/886776276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25600c8490 msgr2=0x7f256014bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:23.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.921+0000 7f2566fc4640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25600c8490 0x7f256014bf60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f254c00e9d0 tx=0x7f254c00ee90 comp rx=0 tx=0).stop 2023-10-30T13:35:23.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.921+0000 7f2566fc4640 1 -- 172.21.15.46:0/886776276 shutdown_connections 2023-10-30T13:35:23.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.921+0000 7f2566fc4640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f25440664a0 0x7f2544068960 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.921+0000 7f2566fc4640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2560153d70 0x7f256014c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.921+0000 7f2566fc4640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25600c8490 0x7f256014bf60 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.921+0000 7f2566fc4640 1 --2- 172.21.15.46:0/886776276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f256006e470 0x7f25601524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:23.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.921+0000 7f2566fc4640 1 -- 172.21.15.46:0/886776276 >> 172.21.15.46:0/886776276 conn(0x7f2560071ea0 msgr2=0x7f256006f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:23.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.921+0000 7f2566fc4640 1 -- 172.21.15.46:0/886776276 shutdown_connections 2023-10-30T13:35:23.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:23.921+0000 7f2566fc4640 1 -- 172.21.15.46:0/886776276 wait complete. 2023-10-30T13:35:23.946 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:25.946 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:26.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.105+0000 7f8e19fa7640 1 Processor -- start 2023-10-30T13:35:26.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.105+0000 7f8e19fa7640 1 -- start start 2023-10-30T13:35:26.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.105+0000 7f8e19fa7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8e141538f0 0x7f8e14159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.105+0000 7f8e19fa7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e1415cd50 0x7f8e140c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.109+0000 7f8e19fa7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1415d300 con 0x7f8e140cefa0 2023-10-30T13:35:26.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.109+0000 7f8e19fa7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e1415d5e0 con 0x7f8e141538f0 2023-10-30T13:35:26.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.109+0000 7f8e19fa7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1415d8c0 con 0x7f8e1415cd50 2023-10-30T13:35:26.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.109+0000 7f8e13fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e1415cd50 0x7f8e140c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.109+0000 7f8e13fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e1415cd50 0x7f8e140c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:36554/0 (socket says 172.21.15.46:36554) 2023-10-30T13:35:26.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.109+0000 7f8e18fa5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8e141538f0 0x7f8e14159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.109+0000 7f8e13fff640 1 -- 172.21.15.46:0/1846655521 learned_addr learned my addr 172.21.15.46:0/1846655521 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:26.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.109+0000 7f8e13fff640 1 -- 172.21.15.46:0/1846655521 >> v2:172.21.15.154:3300/0 conn(0x7f8e141538f0 msgr2=0x7f8e14159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:26.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.109+0000 7f8e13fff640 1 --2- 172.21.15.46:0/1846655521 >> v2:172.21.15.154:3300/0 conn(0x7f8e141538f0 0x7f8e14159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.109+0000 7f8e13fff640 1 -- 172.21.15.46:0/1846655521 >> v1:172.21.15.154:6789/0 conn(0x7f8e140cefa0 legacy=0x7f8e140cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.109+0000 7f8e13fff640 1 -- 172.21.15.46:0/1846655521 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1415dba0 con 0x7f8e1415cd50 2023-10-30T13:35:26.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e13fff640 1 --2- 172.21.15.46:0/1846655521 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e1415cd50 0x7f8e140c9f90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009870 tx=0x7f8e0001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a66619e258cc076 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:26.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e137fe640 1 -- 172.21.15.46:0/1846655521 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000a8a30 con 0x7f8e1415cd50 2023-10-30T13:35:26.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e137fe640 1 -- 172.21.15.46:0/1846655521 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e000a8b90 con 0x7f8e1415cd50 2023-10-30T13:35:26.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e137fe640 1 -- 172.21.15.46:0/1846655521 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000b0440 con 0x7f8e1415cd50 2023-10-30T13:35:26.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e19fa7640 1 -- 172.21.15.46:0/1846655521 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e1415cd50 msgr2=0x7f8e140c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e19fa7640 1 --2- 172.21.15.46:0/1846655521 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e1415cd50 0x7f8e140c9f90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009870 tx=0x7f8e0001ed30 comp rx=0 tx=0).stop 2023-10-30T13:35:26.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e19fa7640 1 -- 172.21.15.46:0/1846655521 shutdown_connections 2023-10-30T13:35:26.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e19fa7640 1 --2- 172.21.15.46:0/1846655521 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e1415cd50 0x7f8e140c9f90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e19fa7640 1 --2- 172.21.15.46:0/1846655521 >> v2:172.21.15.154:3300/0 conn(0x7f8e141538f0 0x7f8e14159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e19fa7640 1 -- 172.21.15.46:0/1846655521 >> 172.21.15.46:0/1846655521 conn(0x7f8e1406a640 msgr2=0x7f8e14065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:26.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e19fa7640 1 -- 172.21.15.46:0/1846655521 shutdown_connections 2023-10-30T13:35:26.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e19fa7640 1 -- 172.21.15.46:0/1846655521 wait complete. 2023-10-30T13:35:26.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e19fa7640 1 Processor -- start 2023-10-30T13:35:26.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.113+0000 7f8e19fa7640 1 -- start start 2023-10-30T13:35:26.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e19fa7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e140cefa0 0x7f8e1415c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e19fa7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e141538f0 0x7f8e140b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e19fa7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e1415cd50 0x7f8e141f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e19fa7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e141455c0 con 0x7f8e141538f0 2023-10-30T13:35:26.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e19fa7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e141efec0 con 0x7f8e140cefa0 2023-10-30T13:35:26.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e19fa7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e141efa10 con 0x7f8e1415cd50 2023-10-30T13:35:26.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e13fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e141538f0 0x7f8e140b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e13fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e141538f0 0x7f8e140b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44290/0 (socket says 172.21.15.46:44290) 2023-10-30T13:35:26.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e13fff640 1 -- 172.21.15.46:0/3568314633 learned_addr learned my addr 172.21.15.46:0/3568314633 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:26.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e18fa5640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e140cefa0 0x7f8e1415c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e197a6640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e1415cd50 0x7f8e141f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e13fff640 1 -- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e1415cd50 msgr2=0x7f8e141f2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e13fff640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e1415cd50 0x7f8e141f2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e13fff640 1 -- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e140cefa0 msgr2=0x7f8e1415c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e13fff640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e140cefa0 0x7f8e1415c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e13fff640 1 -- 172.21.15.46:0/3568314633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e000094d0 con 0x7f8e141538f0 2023-10-30T13:35:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e18fa5640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e140cefa0 0x7f8e1415c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:26.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e13fff640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e141538f0 0x7f8e140b4170 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f8e0001c960 tx=0x7f8e0001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e117fa640 1 -- 172.21.15.46:0/3568314633 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000b1de0 con 0x7f8e141538f0 2023-10-30T13:35:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e117fa640 1 -- 172.21.15.46:0/3568314633 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e000bb440 con 0x7f8e141538f0 2023-10-30T13:35:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e117fa640 1 -- 172.21.15.46:0/3568314633 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000bb7b0 con 0x7f8e141538f0 2023-10-30T13:35:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e19fa7640 1 -- 172.21.15.46:0/3568314633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e141457f0 con 0x7f8e141538f0 2023-10-30T13:35:26.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e19fa7640 1 -- 172.21.15.46:0/3568314633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e140b50a0 con 0x7f8e141538f0 2023-10-30T13:35:26.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e19fa7640 1 -- 172.21.15.46:0/3568314633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1404e170 con 0x7f8e141538f0 2023-10-30T13:35:26.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.117+0000 7f8e117fa640 1 -- 172.21.15.46:0/3568314633 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e000b7040 con 0x7f8e141538f0 2023-10-30T13:35:26.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.121+0000 7f8e117fa640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8df80664a0 0x7f8df8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:26.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.121+0000 7f8e117fa640 1 -- 172.21.15.46:0/3568314633 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8e00121f80 con 0x7f8e141538f0 2023-10-30T13:35:26.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.121+0000 7f8e18fa5640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8df80664a0 0x7f8df8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:26.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.121+0000 7f8e18fa5640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8df80664a0 0x7f8df8068960 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009bb0 tx=0x7f8e08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:26.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.125+0000 7f8e117fa640 1 -- 172.21.15.46:0/3568314633 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e000ef1a0 con 0x7f8e141538f0 2023-10-30T13:35:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.313+0000 7f8e19fa7640 1 -- 172.21.15.46:0/3568314633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e140b2290 con 0x7f8e141538f0 2023-10-30T13:35:26.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.313+0000 7f8e117fa640 1 -- 172.21.15.46:0/3568314633 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f8e000f27c0 con 0x7f8e141538f0 2023-10-30T13:35:26.321 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:26.321 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:23.070980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 64 2023-10-30T13:35:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.317+0000 7f8e19fa7640 1 -- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8df80664a0 msgr2=0x7f8df8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.317+0000 7f8e19fa7640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8df80664a0 0x7f8df8068960 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009bb0 tx=0x7f8e08008040 comp rx=0 tx=0).stop 2023-10-30T13:35:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.317+0000 7f8e19fa7640 1 -- 172.21.15.46:0/3568314633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e141538f0 msgr2=0x7f8e140b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.317+0000 7f8e19fa7640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e141538f0 0x7f8e140b4170 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f8e0001c960 tx=0x7f8e0001bad0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.317+0000 7f8e19fa7640 1 -- 172.21.15.46:0/3568314633 shutdown_connections 2023-10-30T13:35:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.317+0000 7f8e19fa7640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8df80664a0 0x7f8df8068960 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.317+0000 7f8e19fa7640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e1415cd50 0x7f8e141f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.317+0000 7f8e19fa7640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e141538f0 0x7f8e140b4170 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.317+0000 7f8e19fa7640 1 --2- 172.21.15.46:0/3568314633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e140cefa0 0x7f8e1415c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.317+0000 7f8e19fa7640 1 -- 172.21.15.46:0/3568314633 >> 172.21.15.46:0/3568314633 conn(0x7f8e1406a640 msgr2=0x7f8e14158170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.317+0000 7f8e19fa7640 1 -- 172.21.15.46:0/3568314633 shutdown_connections 2023-10-30T13:35:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:26.317+0000 7f8e19fa7640 1 -- 172.21.15.46:0/3568314633 wait complete. 2023-10-30T13:35:26.342 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:28.343 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.485+0000 7fb028bd3640 1 Processor -- start 2023-10-30T13:35:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.485+0000 7fb028bd3640 1 -- start start 2023-10-30T13:35:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb028bd3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb0240c05d0 0x7fb02413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb028bd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0240c6f60 0x7fb02413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb028bd3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0240c0a50 con 0x7fb0240c6b80 2023-10-30T13:35:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb028bd3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02406f6e0 con 0x7fb0240c05d0 2023-10-30T13:35:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb028bd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02406f9c0 con 0x7fb0240c6f60 2023-10-30T13:35:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb01bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb0240c05d0 0x7fb02413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb01bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb0240c05d0 0x7fb02413ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49546/0 (socket says 172.21.15.46:49546) 2023-10-30T13:35:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb01bfff640 1 -- 172.21.15.46:0/2707603517 learned_addr learned my addr 172.21.15.46:0/2707603517 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb022d9d640 1 --2- 172.21.15.46:0/2707603517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0240c6f60 0x7fb02413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb01bfff640 1 -- 172.21.15.46:0/2707603517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0240c6f60 msgr2=0x7fb02413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb01bfff640 1 --2- 172.21.15.46:0/2707603517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0240c6f60 0x7fb02413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb01bfff640 1 -- 172.21.15.46:0/2707603517 >> v1:172.21.15.154:6789/0 conn(0x7fb0240c6b80 legacy=0x7fb0240bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:28.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.489+0000 7fb01bfff640 1 -- 172.21.15.46:0/2707603517 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02406fca0 con 0x7fb0240c05d0 2023-10-30T13:35:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb01bfff640 1 --2- 172.21.15.46:0/2707603517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0240c05d0 0x7fb02413ed90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009870 tx=0x7fb00c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c584b65801c7338 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb021d9b640 1 -- 172.21.15.46:0/2707603517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0ab650 con 0x7fb0240c05d0 2023-10-30T13:35:28.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb021d9b640 1 -- 172.21.15.46:0/2707603517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00c0ab7b0 con 0x7fb0240c05d0 2023-10-30T13:35:28.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb021d9b640 1 -- 172.21.15.46:0/2707603517 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0abb40 con 0x7fb0240c05d0 2023-10-30T13:35:28.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb028bd3640 1 -- 172.21.15.46:0/2707603517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0240c05d0 msgr2=0x7fb02413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:28.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb028bd3640 1 --2- 172.21.15.46:0/2707603517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0240c05d0 0x7fb02413ed90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009870 tx=0x7fb00c01c540 comp rx=0 tx=0).stop 2023-10-30T13:35:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb028bd3640 1 -- 172.21.15.46:0/2707603517 shutdown_connections 2023-10-30T13:35:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb028bd3640 1 --2- 172.21.15.46:0/2707603517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0240c6f60 0x7fb02413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb028bd3640 1 --2- 172.21.15.46:0/2707603517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0240c05d0 0x7fb02413ed90 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:28.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb028bd3640 1 -- 172.21.15.46:0/2707603517 >> 172.21.15.46:0/2707603517 conn(0x7fb0240c7910 msgr2=0x7fb0240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:28.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb028bd3640 1 -- 172.21.15.46:0/2707603517 shutdown_connections 2023-10-30T13:35:28.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb028bd3640 1 -- 172.21.15.46:0/2707603517 wait complete. 2023-10-30T13:35:28.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb028bd3640 1 Processor -- start 2023-10-30T13:35:28.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.493+0000 7fb028bd3640 1 -- start start 2023-10-30T13:35:28.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb028bd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0240c05d0 0x7fb02406b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:28.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb028bd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0240c6b80 0x7fb02412f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:28.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb028bd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0240c6f60 0x7fb02412e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:28.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb028bd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02414d010 con 0x7fb0240c6f60 2023-10-30T13:35:28.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb028bd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb024135140 con 0x7fb0240c6b80 2023-10-30T13:35:28.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb028bd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb024139750 con 0x7fb0240c05d0 2023-10-30T13:35:28.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb022d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0240c6f60 0x7fb02412e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:28.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb022d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0240c6f60 0x7fb02412e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49548/0 (socket says 172.21.15.46:49548) 2023-10-30T13:35:28.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb022d9d640 1 -- 172.21.15.46:0/2875962435 learned_addr learned my addr 172.21.15.46:0/2875962435 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:28.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb01bfff640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0240c6b80 0x7fb02412f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:28.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb02259c640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0240c05d0 0x7fb02406b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:28.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb022d9d640 1 -- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0240c05d0 msgr2=0x7fb02406b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:28.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb022d9d640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0240c05d0 0x7fb02406b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:28.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb022d9d640 1 -- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0240c6b80 msgr2=0x7fb02412f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:28.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb022d9d640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0240c6b80 0x7fb02412f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:28.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb022d9d640 1 -- 172.21.15.46:0/2875962435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c0094d0 con 0x7fb0240c6f60 2023-10-30T13:35:28.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb01bfff640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0240c6b80 0x7fb02412f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:28.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb022d9d640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0240c6f60 0x7fb02412e510 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb01c007e60 tx=0x7fb01c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:28.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb01affd640 1 -- 172.21.15.46:0/2875962435 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c014420 con 0x7fb0240c6f60 2023-10-30T13:35:28.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb01affd640 1 -- 172.21.15.46:0/2875962435 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c0145a0 con 0x7fb0240c6f60 2023-10-30T13:35:28.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb01affd640 1 -- 172.21.15.46:0/2875962435 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c014950 con 0x7fb0240c6f60 2023-10-30T13:35:28.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb028bd3640 1 -- 172.21.15.46:0/2875962435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02414d4a0 con 0x7fb0240c6f60 2023-10-30T13:35:28.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb028bd3640 1 -- 172.21.15.46:0/2875962435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb024139470 con 0x7fb0240c6f60 2023-10-30T13:35:28.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb028bd3640 1 -- 172.21.15.46:0/2875962435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02414d180 con 0x7fb0240c6f60 2023-10-30T13:35:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.497+0000 7fb01affd640 1 -- 172.21.15.46:0/2875962435 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb01c014ab0 con 0x7fb0240c6f60 2023-10-30T13:35:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.501+0000 7fb01affd640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0040664d0 0x7fb004068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:28.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.501+0000 7fb01affd640 1 -- 172.21.15.46:0/2875962435 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fb01c0849b0 con 0x7fb0240c6f60 2023-10-30T13:35:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.501+0000 7fb02259c640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0040664d0 0x7fb004068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.501+0000 7fb02259c640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0040664d0 0x7fb004068990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb014000f70 tx=0x7fb014009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:28.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.505+0000 7fb01affd640 1 -- 172.21.15.46:0/2875962435 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb01c051bd0 con 0x7fb0240c6f60 2023-10-30T13:35:28.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 -- 172.21.15.46:0/2875962435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb02404e170 con 0x7fb0240c6f60 2023-10-30T13:35:28.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb01affd640 1 -- 172.21.15.46:0/2875962435 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5735 (secure 0 0 0) 0x7fb01c0551f0 con 0x7fb0240c6f60 2023-10-30T13:35:28.698 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:28.698 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:23.070980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:28.699 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 64 2023-10-30T13:35:28.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 -- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0040664d0 msgr2=0x7fb004068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:28.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0040664d0 0x7fb004068990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb014000f70 tx=0x7fb014009040 comp rx=0 tx=0).stop 2023-10-30T13:35:28.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 -- 172.21.15.46:0/2875962435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0240c6f60 msgr2=0x7fb02412e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:28.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0240c6f60 0x7fb02412e510 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb01c007e60 tx=0x7fb01c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:35:28.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 -- 172.21.15.46:0/2875962435 shutdown_connections 2023-10-30T13:35:28.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0040664d0 0x7fb004068990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:28.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0240c6f60 0x7fb02412e510 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:28.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0240c6b80 0x7fb02412f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:28.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 --2- 172.21.15.46:0/2875962435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0240c05d0 0x7fb02406b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:28.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 -- 172.21.15.46:0/2875962435 >> 172.21.15.46:0/2875962435 conn(0x7fb0240c7910 msgr2=0x7fb0240c1970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:28.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 -- 172.21.15.46:0/2875962435 shutdown_connections 2023-10-30T13:35:28.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:28.693+0000 7fb028bd3640 1 -- 172.21.15.46:0/2875962435 wait complete. 2023-10-30T13:35:28.720 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 64, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 64, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5442": { "addr": "172.21.15.46:6836/3955931693", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3955931693, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3955931693, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5442, "incarnation": 46, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 13 }, "gid_5574": { "addr": "172.21.15.46:6837/126953235", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 126953235, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 126953235, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5574, "incarnation": 55, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_5721": { "addr": "172.21.15.46:6833/4044310929", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4044310929, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4044310929, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5721, "incarnation": 61, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:35:23.070980+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5721, "mds_1": 5574, "mds_2": 5442 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:35:30.720 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:30.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.869+0000 7f8c74bc4640 1 Processor -- start 2023-10-30T13:35:30.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.869+0000 7f8c74bc4640 1 -- start start 2023-10-30T13:35:30.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.869+0000 7f8c74bc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c700c6b80 0x7f8c700bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.869+0000 7f8c74bc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8c7013f4a0 0x7f8c700c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.869+0000 7f8c74bc4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c700c0a50 con 0x7f8c700c6f60 2023-10-30T13:35:30.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.869+0000 7f8c74bc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7006f6e0 con 0x7f8c7013f4a0 2023-10-30T13:35:30.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.869+0000 7f8c74bc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7006f9c0 con 0x7f8c700c6b80 2023-10-30T13:35:30.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.869+0000 7f8c67fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8c700c6f60 0x7f8c7013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46382/0 (socket says 172.21.15.46:46382) 2023-10-30T13:35:30.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.869+0000 7f8c67fff640 1 -- 172.21.15.46:0/1845878149 learned_addr learned my addr 172.21.15.46:0/1845878149 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:35:30.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.869+0000 7f8c6e59c640 1 --2- 172.21.15.46:0/1845878149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c700c6b80 0x7f8c700bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.873+0000 7f8c6ed9d640 1 --2- 172.21.15.46:0/1845878149 >> v2:172.21.15.154:3300/0 conn(0x7f8c7013f4a0 0x7f8c700c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.873+0000 7f8c6dd9b640 1 -- 172.21.15.46:0/1845878149 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680991073 0 0) 0x7f8c700c0a50 con 0x7f8c700c6f60 2023-10-30T13:35:30.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.873+0000 7f8c6dd9b640 1 -- 172.21.15.46:0/1845878149 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c5c003530 con 0x7f8c700c6f60 2023-10-30T13:35:30.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.873+0000 7f8c6e59c640 1 -- 172.21.15.46:0/1845878149 >> v2:172.21.15.154:3300/0 conn(0x7f8c7013f4a0 msgr2=0x7f8c700c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:30.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.873+0000 7f8c6e59c640 1 --2- 172.21.15.46:0/1845878149 >> v2:172.21.15.154:3300/0 conn(0x7f8c7013f4a0 0x7f8c700c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.873+0000 7f8c6e59c640 1 -- 172.21.15.46:0/1845878149 >> v1:172.21.15.46:6789/0 conn(0x7f8c700c6f60 legacy=0x7f8c7013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:30.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.873+0000 7f8c6e59c640 1 -- 172.21.15.46:0/1845878149 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7006fca0 con 0x7f8c700c6b80 2023-10-30T13:35:30.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.873+0000 7f8c6e59c640 1 --2- 172.21.15.46:0/1845878149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c700c6b80 0x7f8c700bfec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8c600097c0 tx=0x7f8c6001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52476f0ad67a152d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:30.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.873+0000 7f8c6dd9b640 1 -- 172.21.15.46:0/1845878149 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c600ad640 con 0x7f8c700c6b80 2023-10-30T13:35:30.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c6dd9b640 1 -- 172.21.15.46:0/1845878149 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c600ad7a0 con 0x7f8c700c6b80 2023-10-30T13:35:30.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c6dd9b640 1 -- 172.21.15.46:0/1845878149 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c600adb10 con 0x7f8c700c6b80 2023-10-30T13:35:30.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 -- 172.21.15.46:0/1845878149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c700c6b80 msgr2=0x7f8c700bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:30.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 --2- 172.21.15.46:0/1845878149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c700c6b80 0x7f8c700bfec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8c600097c0 tx=0x7f8c6001c480 comp rx=0 tx=0).stop 2023-10-30T13:35:30.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 -- 172.21.15.46:0/1845878149 shutdown_connections 2023-10-30T13:35:30.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 --2- 172.21.15.46:0/1845878149 >> v2:172.21.15.154:3300/0 conn(0x7f8c7013f4a0 0x7f8c700c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 --2- 172.21.15.46:0/1845878149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c700c6b80 0x7f8c700bfec0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 -- 172.21.15.46:0/1845878149 >> 172.21.15.46:0/1845878149 conn(0x7f8c700c7910 msgr2=0x7f8c700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:30.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 -- 172.21.15.46:0/1845878149 shutdown_connections 2023-10-30T13:35:30.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 -- 172.21.15.46:0/1845878149 wait complete. 2023-10-30T13:35:30.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 Processor -- start 2023-10-30T13:35:30.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 -- start start 2023-10-30T13:35:30.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700c6b80 0x7f8c7015f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c700c6f60 0x7f8c701380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c7013f4a0 0x7f8c70138610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7013e5a0 con 0x7f8c700c6b80 2023-10-30T13:35:30.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c701f6230 con 0x7f8c700c6f60 2023-10-30T13:35:30.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c74bc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c701f6770 con 0x7f8c7013f4a0 2023-10-30T13:35:30.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c6e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700c6b80 0x7f8c7015f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c6e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700c6b80 0x7f8c7015f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49564/0 (socket says 172.21.15.46:49564) 2023-10-30T13:35:30.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c6e59c640 1 -- 172.21.15.46:0/3963428146 learned_addr learned my addr 172.21.15.46:0/3963428146 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:30.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c67fff640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c700c6f60 0x7f8c701380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.877+0000 7f8c6ed9d640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c7013f4a0 0x7f8c70138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c6e59c640 1 -- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c7013f4a0 msgr2=0x7f8c70138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:30.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c6e59c640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c7013f4a0 0x7f8c70138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c6e59c640 1 -- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c700c6f60 msgr2=0x7f8c701380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:30.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c6e59c640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c700c6f60 0x7f8c701380d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c6e59c640 1 -- 172.21.15.46:0/3963428146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c60009420 con 0x7f8c700c6b80 2023-10-30T13:35:30.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c6e59c640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700c6b80 0x7f8c7015f3d0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8c60000c10 tx=0x7f8c60019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:30.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c66ffd640 1 -- 172.21.15.46:0/3963428146 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c600add20 con 0x7f8c700c6b80 2023-10-30T13:35:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c66ffd640 1 -- 172.21.15.46:0/3963428146 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c600a6070 con 0x7f8c700c6b80 2023-10-30T13:35:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c74bc4640 1 -- 172.21.15.46:0/3963428146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7013a5a0 con 0x7f8c700c6b80 2023-10-30T13:35:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c66ffd640 1 -- 172.21.15.46:0/3963428146 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c60003820 con 0x7f8c700c6b80 2023-10-30T13:35:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c74bc4640 1 -- 172.21.15.46:0/3963428146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c701f6490 con 0x7f8c700c6b80 2023-10-30T13:35:30.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c74bc4640 1 -- 172.21.15.46:0/3963428146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c70071e30 con 0x7f8c700c6b80 2023-10-30T13:35:30.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c66ffd640 1 -- 172.21.15.46:0/3963428146 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8c600b4040 con 0x7f8c700c6b80 2023-10-30T13:35:30.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c66ffd640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8c500664a0 0x7f8c50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:30.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c66ffd640 1 -- 172.21.15.46:0/3963428146 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8c6011e480 con 0x7f8c700c6b80 2023-10-30T13:35:30.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.881+0000 7f8c67fff640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8c500664a0 0x7f8c50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:30.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.885+0000 7f8c67fff640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8c500664a0 0x7f8c50068960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8c7013a340 tx=0x7f8c58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:30.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:30.885+0000 7f8c66ffd640 1 -- 172.21.15.46:0/3963428146 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8c600eb600 con 0x7f8c700c6b80 2023-10-30T13:35:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.077+0000 7f8c74bc4640 1 -- 172.21.15.46:0/3963428146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c7004e170 con 0x7f8c700c6b80 2023-10-30T13:35:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.077+0000 7f8c66ffd640 1 -- 172.21.15.46:0/3963428146 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8c600eec20 con 0x7f8c700c6b80 2023-10-30T13:35:31.084 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:31.084 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:28.173835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 65 2023-10-30T13:35:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.081+0000 7f8c74bc4640 1 -- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8c500664a0 msgr2=0x7f8c50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.081+0000 7f8c74bc4640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8c500664a0 0x7f8c50068960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8c7013a340 tx=0x7f8c58009040 comp rx=0 tx=0).stop 2023-10-30T13:35:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.081+0000 7f8c74bc4640 1 -- 172.21.15.46:0/3963428146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700c6b80 msgr2=0x7f8c7015f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.081+0000 7f8c74bc4640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700c6b80 0x7f8c7015f3d0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8c60000c10 tx=0x7f8c60019220 comp rx=0 tx=0).stop 2023-10-30T13:35:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.081+0000 7f8c74bc4640 1 -- 172.21.15.46:0/3963428146 shutdown_connections 2023-10-30T13:35:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.081+0000 7f8c74bc4640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8c500664a0 0x7f8c50068960 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.081+0000 7f8c74bc4640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c7013f4a0 0x7f8c70138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.081+0000 7f8c74bc4640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c700c6f60 0x7f8c701380d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.081+0000 7f8c74bc4640 1 --2- 172.21.15.46:0/3963428146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700c6b80 0x7f8c7015f3d0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.081+0000 7f8c74bc4640 1 -- 172.21.15.46:0/3963428146 >> 172.21.15.46:0/3963428146 conn(0x7f8c700c7910 msgr2=0x7f8c70159110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.081+0000 7f8c74bc4640 1 -- 172.21.15.46:0/3963428146 shutdown_connections 2023-10-30T13:35:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:31.081+0000 7f8c74bc4640 1 -- 172.21.15.46:0/3963428146 wait complete. 2023-10-30T13:35:31.105 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:33.106 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:33.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.253+0000 7f4253d2a640 1 Processor -- start 2023-10-30T13:35:33.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f4253d2a640 1 -- start start 2023-10-30T13:35:33.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f4253d2a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f424c0c6b80 0x7f424c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f4253d2a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f424c13f4a0 0x7f424c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f4253d2a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424c0c0a50 con 0x7f424c0c6f60 2023-10-30T13:35:33.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f4253d2a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424c06f6e0 con 0x7f424c13f4a0 2023-10-30T13:35:33.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f4253d2a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424c06f9c0 con 0x7f424c0c6b80 2023-10-30T13:35:33.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f42522c7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f424c13f4a0 0x7f424c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f42522c7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f424c13f4a0 0x7f424c0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49578/0 (socket says 172.21.15.46:49578) 2023-10-30T13:35:33.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f42522c7640 1 -- 172.21.15.46:0/943352199 learned_addr learned my addr 172.21.15.46:0/943352199 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:33.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f4251ac6640 1 --2- 172.21.15.46:0/943352199 >> v2:172.21.15.154:3300/0 conn(0x7f424c0c6b80 0x7f424c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f42522c7640 1 -- 172.21.15.46:0/943352199 >> v2:172.21.15.154:3300/0 conn(0x7f424c0c6b80 msgr2=0x7f424c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:33.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f42522c7640 1 --2- 172.21.15.46:0/943352199 >> v2:172.21.15.154:3300/0 conn(0x7f424c0c6b80 0x7f424c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f42522c7640 1 -- 172.21.15.46:0/943352199 >> v1:172.21.15.154:6789/0 conn(0x7f424c0c6f60 legacy=0x7f424c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.257+0000 7f42522c7640 1 -- 172.21.15.46:0/943352199 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c06fca0 con 0x7f424c13f4a0 2023-10-30T13:35:33.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f42522c7640 1 --2- 172.21.15.46:0/943352199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f424c13f4a0 0x7f424c0c0400 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f424c06f1b0 tx=0x7f424401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a2f093ba35fcff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:33.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4250ac4640 1 -- 172.21.15.46:0/943352199 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42440a57a0 con 0x7f424c13f4a0 2023-10-30T13:35:33.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4250ac4640 1 -- 172.21.15.46:0/943352199 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42440a5900 con 0x7f424c13f4a0 2023-10-30T13:35:33.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4250ac4640 1 -- 172.21.15.46:0/943352199 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42440a5c90 con 0x7f424c13f4a0 2023-10-30T13:35:33.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4253d2a640 1 -- 172.21.15.46:0/943352199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f424c13f4a0 msgr2=0x7f424c0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4253d2a640 1 --2- 172.21.15.46:0/943352199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f424c13f4a0 0x7f424c0c0400 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f424c06f1b0 tx=0x7f424401c9d0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4253d2a640 1 -- 172.21.15.46:0/943352199 shutdown_connections 2023-10-30T13:35:33.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4253d2a640 1 --2- 172.21.15.46:0/943352199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f424c13f4a0 0x7f424c0c0400 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4253d2a640 1 --2- 172.21.15.46:0/943352199 >> v2:172.21.15.154:3300/0 conn(0x7f424c0c6b80 0x7f424c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4253d2a640 1 -- 172.21.15.46:0/943352199 >> 172.21.15.46:0/943352199 conn(0x7f424c0c7910 msgr2=0x7f424c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:33.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4253d2a640 1 -- 172.21.15.46:0/943352199 shutdown_connections 2023-10-30T13:35:33.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4253d2a640 1 -- 172.21.15.46:0/943352199 wait complete. 2023-10-30T13:35:33.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4253d2a640 1 Processor -- start 2023-10-30T13:35:33.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.261+0000 7f4253d2a640 1 -- start start 2023-10-30T13:35:33.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f4253d2a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0c6b80 0x7f424c15f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f4253d2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f424c0c6f60 0x7f424c1587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f4253d2a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c13f4a0 0x7f424c1348e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f4253d2a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c13d030 con 0x7f424c0c6f60 2023-10-30T13:35:33.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f4253d2a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c13cd20 con 0x7f424c13f4a0 2023-10-30T13:35:33.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f4253d2a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424c1343e0 con 0x7f424c0c6b80 2023-10-30T13:35:33.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42512c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f424c0c6f60 0x7f424c1587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42512c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f424c0c6f60 0x7f424c1587e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49590/0 (socket says 172.21.15.46:49590) 2023-10-30T13:35:33.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42512c5640 1 -- 172.21.15.46:0/1626885216 learned_addr learned my addr 172.21.15.46:0/1626885216 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:33.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f4251ac6640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0c6b80 0x7f424c15f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42522c7640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c13f4a0 0x7f424c1348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42512c5640 1 -- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0c6b80 msgr2=0x7f424c15f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42512c5640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0c6b80 0x7f424c15f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42512c5640 1 -- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c13f4a0 msgr2=0x7f424c1348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42512c5640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c13f4a0 0x7f424c1348e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42512c5640 1 -- 172.21.15.46:0/1626885216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4244009b30 con 0x7f424c0c6f60 2023-10-30T13:35:33.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42512c5640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f424c0c6f60 0x7f424c1587e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c500 tx=0x7f423c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:33.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42427fc640 1 -- 172.21.15.46:0/1626885216 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c005220 con 0x7f424c0c6f60 2023-10-30T13:35:33.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42427fc640 1 -- 172.21.15.46:0/1626885216 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c005380 con 0x7f424c0c6f60 2023-10-30T13:35:33.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f4253d2a640 1 -- 172.21.15.46:0/1626885216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424c13d280 con 0x7f424c0c6f60 2023-10-30T13:35:33.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42427fc640 1 -- 172.21.15.46:0/1626885216 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c007270 con 0x7f424c0c6f60 2023-10-30T13:35:33.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f4253d2a640 1 -- 172.21.15.46:0/1626885216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424c13d840 con 0x7f424c0c6f60 2023-10-30T13:35:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f4253d2a640 1 -- 172.21.15.46:0/1626885216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424c04e170 con 0x7f424c0c6f60 2023-10-30T13:35:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.265+0000 7f42427fc640 1 -- 172.21.15.46:0/1626885216 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f423c0073d0 con 0x7f424c0c6f60 2023-10-30T13:35:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.269+0000 7f42427fc640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42300664a0 0x7f4230068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:33.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.269+0000 7f42427fc640 1 -- 172.21.15.46:0/1626885216 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f423c083b70 con 0x7f424c0c6f60 2023-10-30T13:35:33.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.269+0000 7f4251ac6640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42300664a0 0x7f4230068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:33.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.269+0000 7f4251ac6640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42300664a0 0x7f4230068960 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f424800b9d0 tx=0x7f4248008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:33.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.273+0000 7f42427fc640 1 -- 172.21.15.46:0/1626885216 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f423c050c70 con 0x7f424c0c6f60 2023-10-30T13:35:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.461+0000 7f4253d2a640 1 -- 172.21.15.46:0/1626885216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f424c13c330 con 0x7f424c0c6f60 2023-10-30T13:35:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.461+0000 7f42427fc640 1 -- 172.21.15.46:0/1626885216 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f423c054290 con 0x7f424c0c6f60 2023-10-30T13:35:33.469 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:33.469 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:28.173835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:33.469 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 65 2023-10-30T13:35:33.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.465+0000 7f4253d2a640 1 -- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42300664a0 msgr2=0x7f4230068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.465+0000 7f4253d2a640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42300664a0 0x7f4230068960 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f424800b9d0 tx=0x7f4248008040 comp rx=0 tx=0).stop 2023-10-30T13:35:33.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.465+0000 7f4253d2a640 1 -- 172.21.15.46:0/1626885216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f424c0c6f60 msgr2=0x7f424c1587e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:33.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.465+0000 7f4253d2a640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f424c0c6f60 0x7f424c1587e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c500 tx=0x7f423c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.465+0000 7f4253d2a640 1 -- 172.21.15.46:0/1626885216 shutdown_connections 2023-10-30T13:35:33.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.465+0000 7f4253d2a640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42300664a0 0x7f4230068960 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.465+0000 7f4253d2a640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c13f4a0 0x7f424c1348e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.465+0000 7f4253d2a640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f424c0c6f60 0x7f424c1587e0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.465+0000 7f4253d2a640 1 --2- 172.21.15.46:0/1626885216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0c6b80 0x7f424c15f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.465+0000 7f4253d2a640 1 -- 172.21.15.46:0/1626885216 >> 172.21.15.46:0/1626885216 conn(0x7f424c0c7910 msgr2=0x7f424c0c1680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.465+0000 7f4253d2a640 1 -- 172.21.15.46:0/1626885216 shutdown_connections 2023-10-30T13:35:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:33.465+0000 7f4253d2a640 1 -- 172.21.15.46:0/1626885216 wait complete. 2023-10-30T13:35:33.489 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:35.490 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:35.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.633+0000 7f8850f49640 1 Processor -- start 2023-10-30T13:35:35.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.633+0000 7f8850f49640 1 -- start start 2023-10-30T13:35:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f8850f49640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f884c15cd50 0x7f884c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f8850f49640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f884c0cefa0 0x7f884c0cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f8850f49640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f884c15a6c0 con 0x7f884c1538f0 2023-10-30T13:35:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f8850f49640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f884c0cf8f0 con 0x7f884c15cd50 2023-10-30T13:35:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f8850f49640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f884c15d300 con 0x7f884c0cefa0 2023-10-30T13:35:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f884affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f884c15cd50 0x7f884c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f884affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f884c15cd50 0x7f884c0c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49616/0 (socket says 172.21.15.46:49616) 2023-10-30T13:35:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f884affd640 1 -- 172.21.15.46:0/449805764 learned_addr learned my addr 172.21.15.46:0/449805764 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:35.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f884bfff640 1 --2- 172.21.15.46:0/449805764 >> v2:172.21.15.154:3300/0 conn(0x7f884c0cefa0 0x7f884c0cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f884affd640 1 -- 172.21.15.46:0/449805764 >> v2:172.21.15.154:3300/0 conn(0x7f884c0cefa0 msgr2=0x7f884c0cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:35.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f884affd640 1 --2- 172.21.15.46:0/449805764 >> v2:172.21.15.154:3300/0 conn(0x7f884c0cefa0 0x7f884c0cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f884affd640 1 -- 172.21.15.46:0/449805764 >> v1:172.21.15.154:6789/0 conn(0x7f884c1538f0 legacy=0x7f884c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.637+0000 7f884affd640 1 -- 172.21.15.46:0/449805764 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884c15d5e0 con 0x7f884c15cd50 2023-10-30T13:35:35.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f884affd640 1 --2- 172.21.15.46:0/449805764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884c15cd50 0x7f884c0c9f90 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f884c066a40 tx=0x7f884001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f90a50ea448d28e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f884a7fc640 1 -- 172.21.15.46:0/449805764 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88400a89f0 con 0x7f884c15cd50 2023-10-30T13:35:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f884a7fc640 1 -- 172.21.15.46:0/449805764 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88400a8b50 con 0x7f884c15cd50 2023-10-30T13:35:35.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f884a7fc640 1 -- 172.21.15.46:0/449805764 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88400b0440 con 0x7f884c15cd50 2023-10-30T13:35:35.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 -- 172.21.15.46:0/449805764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884c15cd50 msgr2=0x7f884c0c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 --2- 172.21.15.46:0/449805764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884c15cd50 0x7f884c0c9f90 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f884c066a40 tx=0x7f884001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 -- 172.21.15.46:0/449805764 shutdown_connections 2023-10-30T13:35:35.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 --2- 172.21.15.46:0/449805764 >> v2:172.21.15.154:3300/0 conn(0x7f884c0cefa0 0x7f884c0cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 --2- 172.21.15.46:0/449805764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884c15cd50 0x7f884c0c9f90 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 -- 172.21.15.46:0/449805764 >> 172.21.15.46:0/449805764 conn(0x7f884c06a640 msgr2=0x7f884c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:35.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 -- 172.21.15.46:0/449805764 shutdown_connections 2023-10-30T13:35:35.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 -- 172.21.15.46:0/449805764 wait complete. 2023-10-30T13:35:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 Processor -- start 2023-10-30T13:35:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 -- start start 2023-10-30T13:35:35.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f884c0cefa0 0x7f884c1453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f884c1538f0 0x7f884c15a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.641+0000 7f8850f49640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884c15cd50 0x7f884c158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f8850f49640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884c1f02f0 con 0x7f884c15cd50 2023-10-30T13:35:35.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f8850f49640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884c157a20 con 0x7f884c1538f0 2023-10-30T13:35:35.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f8850f49640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f884c1ef2e0 con 0x7f884c0cefa0 2023-10-30T13:35:35.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f884bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884c15cd50 0x7f884c158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f884bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884c15cd50 0x7f884c158ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49624/0 (socket says 172.21.15.46:49624) 2023-10-30T13:35:35.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f884bfff640 1 -- 172.21.15.46:0/975766045 learned_addr learned my addr 172.21.15.46:0/975766045 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:35.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f884affd640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f884c1538f0 0x7f884c15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f884b7fe640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f884c0cefa0 0x7f884c1453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f884bfff640 1 -- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f884c0cefa0 msgr2=0x7f884c1453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f884bfff640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f884c0cefa0 0x7f884c1453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f884bfff640 1 -- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f884c1538f0 msgr2=0x7f884c15a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f884bfff640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f884c1538f0 0x7f884c15a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f884bfff640 1 -- 172.21.15.46:0/975766045 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88400094d0 con 0x7f884c15cd50 2023-10-30T13:35:35.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f884bfff640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884c15cd50 0x7f884c158ee0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f884400efc0 tx=0x7f884400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:35.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f8833fff640 1 -- 172.21.15.46:0/975766045 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8844014480 con 0x7f884c15cd50 2023-10-30T13:35:35.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f8833fff640 1 -- 172.21.15.46:0/975766045 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884400f040 con 0x7f884c15cd50 2023-10-30T13:35:35.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f8850f49640 1 -- 172.21.15.46:0/975766045 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f884c1efe60 con 0x7f884c15cd50 2023-10-30T13:35:35.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f8833fff640 1 -- 172.21.15.46:0/975766045 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8844014820 con 0x7f884c15cd50 2023-10-30T13:35:35.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f8850f49640 1 -- 172.21.15.46:0/975766045 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f884c1ef530 con 0x7f884c15cd50 2023-10-30T13:35:35.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f8850f49640 1 -- 172.21.15.46:0/975766045 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f884c1efff0 con 0x7f884c15cd50 2023-10-30T13:35:35.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f8833fff640 1 -- 172.21.15.46:0/975766045 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8844014980 con 0x7f884c15cd50 2023-10-30T13:35:35.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.645+0000 7f8833fff640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88380664d0 0x7f8838068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:35.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.649+0000 7f8833fff640 1 -- 172.21.15.46:0/975766045 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8844084bc0 con 0x7f884c15cd50 2023-10-30T13:35:35.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.649+0000 7f884b7fe640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88380664d0 0x7f8838068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:35.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.649+0000 7f884b7fe640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88380664d0 0x7f8838068990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f883c000f70 tx=0x7f883c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:35.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.653+0000 7f8833fff640 1 -- 172.21.15.46:0/975766045 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8844051d50 con 0x7f884c15cd50 2023-10-30T13:35:35.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 -- 172.21.15.46:0/975766045 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f884c04e170 con 0x7f884c15cd50 2023-10-30T13:35:35.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8833fff640 1 -- 172.21.15.46:0/975766045 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f8844018020 con 0x7f884c15cd50 2023-10-30T13:35:35.850 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:35.851 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:34.731621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:35.851 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 66 2023-10-30T13:35:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 -- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88380664d0 msgr2=0x7f8838068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88380664d0 0x7f8838068990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f883c000f70 tx=0x7f883c009040 comp rx=0 tx=0).stop 2023-10-30T13:35:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 -- 172.21.15.46:0/975766045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884c15cd50 msgr2=0x7f884c158ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884c15cd50 0x7f884c158ee0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f884400efc0 tx=0x7f884400c490 comp rx=0 tx=0).stop 2023-10-30T13:35:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 -- 172.21.15.46:0/975766045 shutdown_connections 2023-10-30T13:35:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88380664d0 0x7f8838068990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f884c15cd50 0x7f884c158ee0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f884c1538f0 0x7f884c15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 --2- 172.21.15.46:0/975766045 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f884c0cefa0 0x7f884c1453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 -- 172.21.15.46:0/975766045 >> 172.21.15.46:0/975766045 conn(0x7f884c06a640 msgr2=0x7f884c0cd710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:35.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 -- 172.21.15.46:0/975766045 shutdown_connections 2023-10-30T13:35:35.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:35.845+0000 7f8850f49640 1 -- 172.21.15.46:0/975766045 wait complete. 2023-10-30T13:35:35.871 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5266 2023-10-30T13:35:35.871 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.b 2023-10-30T13:35:42.010 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-10-30T13:35:42.005+0000 7ff482536640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:35:42.010 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-10-30T13:35:42.005+0000 7ff482536640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:35:42.010 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-10-30T13:35:42.005+0000 7ff482536640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:35:44.872 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-30T13:35:44.872 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T13:35:44.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T13:35:44.876 INFO:tasks.ceph.mds.b:Started 2023-10-30T13:35:44.876 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:44.954 INFO:tasks.ceph.mds.b.smithi154.stdout:starting mds.b at 2023-10-30T13:35:45.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.025+0000 7f42d933e640 1 Processor -- start 2023-10-30T13:35:45.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.025+0000 7f42d933e640 1 -- start start 2023-10-30T13:35:45.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.025+0000 7f42d933e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f42d406b060 0x7f42d4140270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.025+0000 7f42d933e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d406f350 con 0x7f42d406ac80 2023-10-30T13:35:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.025+0000 7f42d933e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d40c5710 con 0x7f42d415b3d0 2023-10-30T13:35:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.025+0000 7f42d933e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d406f5f0 con 0x7f42d406b060 2023-10-30T13:35:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d2ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f42d406ac80 0x7f42d415acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35682/0 (socket says 172.21.15.46:35682) 2023-10-30T13:35:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d2ffd640 1 -- 172.21.15.46:0/1017924246 learned_addr learned my addr 172.21.15.46:0/1017924246 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:35:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d27fc640 1 --2- 172.21.15.46:0/1017924246 >> v2:172.21.15.154:3300/0 conn(0x7f42d406b060 0x7f42d4140270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:45.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437793278 0 0) 0x7f42d406f350 con 0x7f42d406ac80 2023-10-30T13:35:45.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc003530 con 0x7f42d406ac80 2023-10-30T13:35:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 523739209 0 0) 0x7f42bc003530 con 0x7f42d406ac80 2023-10-30T13:35:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 >> v2:172.21.15.154:3300/0 conn(0x7f42d406b060 msgr2=0x7f42d4140270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 --2- 172.21.15.46:0/1017924246 >> v2:172.21.15.154:3300/0 conn(0x7f42d406b060 0x7f42d4140270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 >> v1:172.21.15.154:6789/0 conn(0x7f42d415b3d0 legacy=0x7f42d4071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:45.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d406f8d0 con 0x7f42d406ac80 2023-10-30T13:35:45.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f42c4001680 con 0x7f42d406ac80 2023-10-30T13:35:45.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 >> v1:172.21.15.46:6789/0 conn(0x7f42d406ac80 legacy=0x7f42d415acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:45.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 --2- 172.21.15.46:0/1017924246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42bc003980 0x7f42bc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 --2- 172.21.15.46:0/1017924246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42bc0047d0 0x7f42bc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 --2- 172.21.15.46:0/1017924246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42bc005190 0x7f42bc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d406f5f0 con 0x7f42bc003980 2023-10-30T13:35:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc00c030 con 0x7f42bc005190 2023-10-30T13:35:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc00c1f0 con 0x7f42bc0047d0 2023-10-30T13:35:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d2ffd640 1 --2- 172.21.15.46:0/1017924246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42bc003980 0x7f42bc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d37fe640 1 --2- 172.21.15.46:0/1017924246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42bc005190 0x7f42bc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d27fc640 1 --2- 172.21.15.46:0/1017924246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42bc0047d0 0x7f42bc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d2ffd640 1 -- 172.21.15.46:0/1017924246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42bc0047d0 msgr2=0x7f42bc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d2ffd640 1 --2- 172.21.15.46:0/1017924246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42bc0047d0 0x7f42bc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:45.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d2ffd640 1 -- 172.21.15.46:0/1017924246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42bc005190 msgr2=0x7f42bc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:45.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d2ffd640 1 --2- 172.21.15.46:0/1017924246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42bc005190 0x7f42bc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:45.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d2ffd640 1 -- 172.21.15.46:0/1017924246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc003530 con 0x7f42bc003980 2023-10-30T13:35:45.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d37fe640 1 --2- 172.21.15.46:0/1017924246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42bc005190 0x7f42bc00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:45.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.029+0000 7f42d37fe640 1 -- 172.21.15.46:0/1017924246 reap_dead start 2023-10-30T13:35:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.033+0000 7f42d2ffd640 1 --2- 172.21.15.46:0/1017924246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42bc003980 0x7f42bc003da0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f42c400c1c0 tx=0x7f42c401cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b001555d928296 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.033+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c40a5b00 con 0x7f42bc003980 2023-10-30T13:35:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.033+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c40a5c60 con 0x7f42bc003980 2023-10-30T13:35:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.033+0000 7f42d1ffb640 1 -- 172.21.15.46:0/1017924246 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c4005b20 con 0x7f42bc003980 2023-10-30T13:35:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.033+0000 7f42d933e640 1 -- 172.21.15.46:0/1017924246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42bc003980 msgr2=0x7f42bc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:45.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.033+0000 7f42d933e640 1 --2- 172.21.15.46:0/1017924246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42bc003980 0x7f42bc003da0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f42c400c1c0 tx=0x7f42c401cc20 comp rx=0 tx=0).stop 2023-10-30T13:35:45.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.033+0000 7f42d933e640 1 -- 172.21.15.46:0/1017924246 shutdown_connections 2023-10-30T13:35:45.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.033+0000 7f42d933e640 1 --2- 172.21.15.46:0/1017924246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42bc003980 0x7f42bc003da0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:45.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.033+0000 7f42d933e640 1 -- 172.21.15.46:0/1017924246 >> 172.21.15.46:0/1017924246 conn(0x7f42d40cac50 msgr2=0x7f42d40cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:45.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.033+0000 7f42d933e640 1 -- 172.21.15.46:0/1017924246 shutdown_connections 2023-10-30T13:35:45.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.033+0000 7f42d933e640 1 -- 172.21.15.46:0/1017924246 wait complete. 2023-10-30T13:35:45.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d933e640 1 Processor -- start 2023-10-30T13:35:45.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d933e640 1 -- start start 2023-10-30T13:35:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d933e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42bc003980 0x7f42d4159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d933e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42d413c1a0 0x7f42d41f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d933e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42d41f15a0 0x7f42d4154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d933e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d41f1a50 con 0x7f42d41f15a0 2023-10-30T13:35:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d933e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d413ad50 con 0x7f42d413c1a0 2023-10-30T13:35:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d933e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d413b030 con 0x7f42bc003980 2023-10-30T13:35:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42d41f15a0 0x7f42d4154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42d41f15a0 0x7f42d4154050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50138/0 (socket says 172.21.15.46:50138) 2023-10-30T13:35:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d37fe640 1 -- 172.21.15.46:0/3947039803 learned_addr learned my addr 172.21.15.46:0/3947039803 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:35:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d27fc640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42d413c1a0 0x7f42d41f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d2ffd640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42bc003980 0x7f42d4159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d37fe640 1 -- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42bc003980 msgr2=0x7f42d4159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d37fe640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42bc003980 0x7f42d4159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d37fe640 1 -- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42d413c1a0 msgr2=0x7f42d41f1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d37fe640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42d413c1a0 0x7f42d41f1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d37fe640 1 -- 172.21.15.46:0/3947039803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d4139bd0 con 0x7f42d41f15a0 2023-10-30T13:35:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.037+0000 7f42d37fe640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42d41f15a0 0x7f42d4154050 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f42d415b7b0 tx=0x7f42cc00e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.041+0000 7f42bb7fe640 1 -- 172.21.15.46:0/3947039803 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc003b90 con 0x7f42d41f15a0 2023-10-30T13:35:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.041+0000 7f42bb7fe640 1 -- 172.21.15.46:0/3947039803 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc003d10 con 0x7f42d41f15a0 2023-10-30T13:35:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.041+0000 7f42bb7fe640 1 -- 172.21.15.46:0/3947039803 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc005310 con 0x7f42d41f15a0 2023-10-30T13:35:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.041+0000 7f42d933e640 1 -- 172.21.15.46:0/3947039803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d4139eb0 con 0x7f42d41f15a0 2023-10-30T13:35:45.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.041+0000 7f42d933e640 1 -- 172.21.15.46:0/3947039803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d413a470 con 0x7f42d41f15a0 2023-10-30T13:35:45.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.041+0000 7f42d933e640 1 -- 172.21.15.46:0/3947039803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d413a660 con 0x7f42d41f15a0 2023-10-30T13:35:45.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.041+0000 7f42bb7fe640 1 -- 172.21.15.46:0/3947039803 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f42cc005470 con 0x7f42d41f15a0 2023-10-30T13:35:45.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.041+0000 7f42bb7fe640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42c00664d0 0x7f42c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:45.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.041+0000 7f42bb7fe640 1 -- 172.21.15.46:0/3947039803 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f42cc083950 con 0x7f42d41f15a0 2023-10-30T13:35:45.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.045+0000 7f42d2ffd640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42c00664d0 0x7f42c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:45.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.045+0000 7f42d2ffd640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42c00664d0 0x7f42c0068990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f42c40ab000 tx=0x7f42c40b0040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:45.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.049+0000 7f42bb7fe640 1 -- 172.21.15.46:0/3947039803 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f42cc050af0 con 0x7f42d41f15a0 2023-10-30T13:35:45.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.241+0000 7f42d933e640 1 -- 172.21.15.46:0/3947039803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42d404e170 con 0x7f42d41f15a0 2023-10-30T13:35:45.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.241+0000 7f42bb7fe640 1 -- 172.21.15.46:0/3947039803 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f42cc019020 con 0x7f42d41f15a0 2023-10-30T13:35:45.248 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:35:45.249 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:35:34.731621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:45.249 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 67 2023-10-30T13:35:45.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.245+0000 7f42d933e640 1 -- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42c00664d0 msgr2=0x7f42c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:45.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.245+0000 7f42d933e640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42c00664d0 0x7f42c0068990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f42c40ab000 tx=0x7f42c40b0040 comp rx=0 tx=0).stop 2023-10-30T13:35:45.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.245+0000 7f42d933e640 1 -- 172.21.15.46:0/3947039803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42d41f15a0 msgr2=0x7f42d4154050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:45.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.245+0000 7f42d933e640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42d41f15a0 0x7f42d4154050 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f42d415b7b0 tx=0x7f42cc00e960 comp rx=0 tx=0).stop 2023-10-30T13:35:45.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.245+0000 7f42d933e640 1 -- 172.21.15.46:0/3947039803 shutdown_connections 2023-10-30T13:35:45.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.245+0000 7f42d933e640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42c00664d0 0x7f42c0068990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:45.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.245+0000 7f42d933e640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42d41f15a0 0x7f42d4154050 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:45.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.245+0000 7f42d933e640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42d413c1a0 0x7f42d41f1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:45.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.245+0000 7f42d933e640 1 --2- 172.21.15.46:0/3947039803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42bc003980 0x7f42d4159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:45.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.245+0000 7f42d933e640 1 -- 172.21.15.46:0/3947039803 >> 172.21.15.46:0/3947039803 conn(0x7f42d40cac50 msgr2=0x7f42d40c7500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:45.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.245+0000 7f42d933e640 1 -- 172.21.15.46:0/3947039803 shutdown_connections 2023-10-30T13:35:45.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:35:45.245+0000 7f42d933e640 1 -- 172.21.15.46:0/3947039803 wait complete. 2023-10-30T13:35:45.269 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-30T13:35:45.270 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 53 secs before thrashing 2023-10-30T13:35:46.951 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:35:46.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:17.080 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:17.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:38.271 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:38.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.424+0000 7f4e37ad6640 1 Processor -- start 2023-10-30T13:36:38.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e37ad6640 1 -- start start 2023-10-30T13:36:38.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e37ad6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4e3015d6f0 0x7f4e3015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:38.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e37ad6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300cf160 0x7f4e300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:38.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e37ad6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e300cb800 con 0x7f4e301538a0 2023-10-30T13:36:38.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e37ad6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e300cb330 con 0x7f4e3015d6f0 2023-10-30T13:36:38.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e37ad6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e300cad70 con 0x7f4e300cf160 2023-10-30T13:36:38.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e35872640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4e301538a0 0x7f4e300ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57080/0 (socket says 172.21.15.46:57080) 2023-10-30T13:36:38.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e35872640 1 -- 172.21.15.46:0/494835798 learned_addr learned my addr 172.21.15.46:0/494835798 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:36:38.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e36073640 1 --2- 172.21.15.46:0/494835798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300cf160 0x7f4e300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:38.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e35071640 1 --2- 172.21.15.46:0/494835798 >> v2:172.21.15.154:3300/0 conn(0x7f4e3015d6f0 0x7f4e3015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:38.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e34870640 1 -- 172.21.15.46:0/494835798 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686411134 0 0) 0x7f4e300cb800 con 0x7f4e301538a0 2023-10-30T13:36:38.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e34870640 1 -- 172.21.15.46:0/494835798 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e1c003530 con 0x7f4e301538a0 2023-10-30T13:36:38.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e34870640 1 -- 172.21.15.46:0/494835798 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 360686386 0 0) 0x7f4e1c003530 con 0x7f4e301538a0 2023-10-30T13:36:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e36073640 1 -- 172.21.15.46:0/494835798 >> v2:172.21.15.154:3300/0 conn(0x7f4e3015d6f0 msgr2=0x7f4e3015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e36073640 1 --2- 172.21.15.46:0/494835798 >> v2:172.21.15.154:3300/0 conn(0x7f4e3015d6f0 0x7f4e3015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e36073640 1 -- 172.21.15.46:0/494835798 >> v1:172.21.15.46:6789/0 conn(0x7f4e301538a0 legacy=0x7f4e300ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.428+0000 7f4e36073640 1 -- 172.21.15.46:0/494835798 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e300cb050 con 0x7f4e300cf160 2023-10-30T13:36:38.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.432+0000 7f4e36073640 1 --2- 172.21.15.46:0/494835798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300cf160 0x7f4e300cf590 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009d90 tx=0x7f4e2801f060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7798bef9bc0bab5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:38.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.432+0000 7f4e34870640 1 -- 172.21.15.46:0/494835798 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c001680 con 0x7f4e300cf160 2023-10-30T13:36:38.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.432+0000 7f4e34870640 1 -- 172.21.15.46:0/494835798 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e280a8bb0 con 0x7f4e300cf160 2023-10-30T13:36:38.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.432+0000 7f4e34870640 1 -- 172.21.15.46:0/494835798 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e280b0440 con 0x7f4e300cf160 2023-10-30T13:36:38.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.432+0000 7f4e37ad6640 1 -- 172.21.15.46:0/494835798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300cf160 msgr2=0x7f4e300cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:38.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.432+0000 7f4e37ad6640 1 --2- 172.21.15.46:0/494835798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300cf160 0x7f4e300cf590 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009d90 tx=0x7f4e2801f060 comp rx=0 tx=0).stop 2023-10-30T13:36:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.432+0000 7f4e37ad6640 1 -- 172.21.15.46:0/494835798 shutdown_connections 2023-10-30T13:36:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.432+0000 7f4e37ad6640 1 --2- 172.21.15.46:0/494835798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300cf160 0x7f4e300cf590 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.432+0000 7f4e37ad6640 1 --2- 172.21.15.46:0/494835798 >> v2:172.21.15.154:3300/0 conn(0x7f4e3015d6f0 0x7f4e3015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.432+0000 7f4e37ad6640 1 -- 172.21.15.46:0/494835798 >> 172.21.15.46:0/494835798 conn(0x7f4e3006bfe0 msgr2=0x7f4e3006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.432+0000 7f4e37ad6640 1 -- 172.21.15.46:0/494835798 shutdown_connections 2023-10-30T13:36:38.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.432+0000 7f4e37ad6640 1 -- 172.21.15.46:0/494835798 wait complete. 2023-10-30T13:36:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e37ad6640 1 Processor -- start 2023-10-30T13:36:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e37ad6640 1 -- start start 2023-10-30T13:36:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e37ad6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300cf160 0x7f4e301f0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e37ad6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e301538a0 0x7f4e300c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e37ad6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e3015d6f0 0x7f4e300c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e37ad6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e300c8f70 con 0x7f4e3015d6f0 2023-10-30T13:36:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e37ad6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e301e33a0 con 0x7f4e301538a0 2023-10-30T13:36:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e37ad6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e301e2ae0 con 0x7f4e300cf160 2023-10-30T13:36:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e36073640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e3015d6f0 0x7f4e300c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:38.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e36073640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e3015d6f0 0x7f4e300c8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44356/0 (socket says 172.21.15.46:44356) 2023-10-30T13:36:38.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e36073640 1 -- 172.21.15.46:0/2298270803 learned_addr learned my addr 172.21.15.46:0/2298270803 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:38.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e35872640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300cf160 0x7f4e301f0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:38.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e35071640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e301538a0 0x7f4e300c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:38.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e36073640 1 -- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300cf160 msgr2=0x7f4e301f0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:38.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e36073640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300cf160 0x7f4e301f0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e36073640 1 -- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e301538a0 msgr2=0x7f4e300c84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:38.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e36073640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e301538a0 0x7f4e300c84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e36073640 1 -- 172.21.15.46:0/2298270803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e28009a70 con 0x7f4e3015d6f0 2023-10-30T13:36:38.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e35071640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e301538a0 0x7f4e300c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:38.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e36073640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e3015d6f0 0x7f4e300c8a30 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4e280b3040 tx=0x7f4e2801be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:38.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e267fc640 1 -- 172.21.15.46:0/2298270803 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28009450 con 0x7f4e3015d6f0 2023-10-30T13:36:38.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e267fc640 1 -- 172.21.15.46:0/2298270803 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e280bc450 con 0x7f4e3015d6f0 2023-10-30T13:36:38.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e267fc640 1 -- 172.21.15.46:0/2298270803 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e280bc7e0 con 0x7f4e3015d6f0 2023-10-30T13:36:38.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e37ad6640 1 -- 172.21.15.46:0/2298270803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3014f080 con 0x7f4e3015d6f0 2023-10-30T13:36:38.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.436+0000 7f4e37ad6640 1 -- 172.21.15.46:0/2298270803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e301e3030 con 0x7f4e3015d6f0 2023-10-30T13:36:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.440+0000 7f4e37ad6640 1 -- 172.21.15.46:0/2298270803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e300b2a50 con 0x7f4e3015d6f0 2023-10-30T13:36:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.440+0000 7f4e267fc640 1 -- 172.21.15.46:0/2298270803 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e280bc940 con 0x7f4e3015d6f0 2023-10-30T13:36:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.440+0000 7f4e267fc640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e140664a0 0x7f4e14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.440+0000 7f4e267fc640 1 -- 172.21.15.46:0/2298270803 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f4e28122c40 con 0x7f4e3015d6f0 2023-10-30T13:36:38.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.440+0000 7f4e35872640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e140664a0 0x7f4e14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.444+0000 7f4e35872640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e140664a0 0x7f4e14068960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c001ba0 tx=0x7f4e2c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:38.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.448+0000 7f4e267fc640 1 -- 172.21.15.46:0/2298270803 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e280efd30 con 0x7f4e3015d6f0 2023-10-30T13:36:38.536 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 0% (97/12071) Updating files: 1% (121/12071) Updating files: 1% (140/12071) Updating files: 2% (242/12071) Updating files: 2% (301/12071) Updating files: 3% (363/12071) Updating files: 4% (483/12071) Updating files: 5% (604/12071) Updating files: 5% (684/12071) Updating files: 5% (686/12071) Updating files: 6% (725/12071) Updating files: 7% (845/12071) Updating files: 7% (898/12071) Updating files: 7% (937/12071) Updating files: 8% (966/12071) Updating files: 8% (996/12071) Updating files: 8% (1000/12071) Updating files: 9% (1087/12071) Updating files: 10% (1208/12071) Updating files: 10% (1265/12071) Updating files: 10% (1266/12071) Updating files: 11% (1328/12071) Updating files: 11% (1411/12071) Updating files: 12% (1449/12071) Updating files: 12% (1462/12071) Updating files: 12% (1520/12071) Updating files: 12% (1537/12071) Updating files: 13% (1570/12071) Updating files: 13% (1604/12071) Updating files: 14% (1690/12071) Updating files: 14% (1794/12071) Updating files: 15% (1811/12071) Updating files: 15% (1863/12071) Updating files: 16% (1932/12071) Updating files: 16% (2050/12071) Updating files: 17% (2053/12071) Updating files: 18% (2173/12071) Updating files: 18% (2176/12071) Updating files: 19% (2294/12071) Updating files: 19% (2299/12071) Updating files: 19% (2380/12071) Updating files: 19% (2393/12071) Updating files: 19% (2405/12071) Updating files: 20% (2415/12071) Updating files: 20% (2528/12071) Updating files: 21% (2535/12071) Updating files: 22% (2656/12071) Updating files: 22% (2685/12071) Updating files: 23% (2777/12071) Updating files: 23% (2859/12071) Updating files: 24% (2898/12071) Updating files: 24% (2920/12071) Updating files: 25% (3018/12071) Updating files: 26% (3139/12071) Updating files: 26% (3155/12071) Updating files: 26% (3226/12071) Updating files: 27% (3260/12071) Updating files: 27% (3267/12071) Updating files: 28% (3380/12071) Updating files: 28% (3391/12071) Updating files: 28% (3417/12071) Updating files: 28% (3460/12071) Updating files: 29% (3501/12071) Updating files: 29% (3525/12071) Updating files: 29% (3537/12071) Updating files: 29% (3611/12071) Updating files: 30% (3622/12071) Updating files: 31% (3743/12071) Updating files: 31% (3767/12071) Updating files: 32% (3863/12071) Updating files: 33% (3984/12071) Updating files: 33% (4004/12071) Updating files: 34% (4105/12071) Updating files: 34% (4177/12071) Updating files: 35% (4225/12071) Updating files: 36% (4346/12071) Updating files: 36% (4360/12071) Updating files: 36% (4362/12071) Updating files: 37% (4467/12071) Updating files: 38% (4587/12071) Updating files: 38% (4630/12071) Updating files: 39% (4708/12071) Updating files: 39% (4735/12071) Updating files: 39% (4761/12071) Updating files: 40% (4829/12071) Updating files: 41% (4950/12071) Updating files: 42% (5070/12071) Updating files: 42% (5083/12071) Updating files: 42% (5190/12071) Updating files: 43% (5191/12071) Updating files: 43% (5197/12071) Updating files: 44% (5312/12071) Updating files: 44% (5399/12071) Updating files: 45% (5432/12071) Updating files: 46% (5553/12071) Updating files: 47% (5674/12071) Updating files: 47% (5678/12071) Updating files: 47% (5743/12071) Updating files: 48% (5795/12071) Updating files: 49% (5915/12071) Updating files: 49% (5927/12071) Updating files: 49% (5965/12071) Updating files: 50% (6036/12071) Updating files: 50% (6097/12071) Updating files: 50% (6139/12071) Updating files: 51% (6157/12071) Updating files: 52% (6277/12071) Updating files: 52% (6330/12071) Updating files: 53% (6398/12071) Updating files: 53% (6495/12071) Updating files: 54% (6519/12071) Updating files: 54% (6607/12071) Updating files: 55% (6640/12071) Updating files: 56% (6760/12071) Updating files: 56% (6805/12071) Updating files: 57% (6881/12071) Updating files: 58% (7002/12071) Updating files: 59% (7122/12071) Updating files: 59% (7198/12071) Updating files: 60% (7243/12071) Updating files: 61% (7364/12071) Updating files: 61% (7403/12071) Updating files: 62% (7485/12071) Updating files: 63% (7605/12071) Updating files: 64% (7726/12071) Updating files: 64% (7787/12071) Updating files: 64% (7793/12071) Updating files: 65% (7847/12071) Updating files: 66% (7967/12071) Updating files: 67% (8088/12071) Updating files: 67% (8126/12071) Updating files: 68% (8209/12071) Updating files: 68% (8252/12071) Updating files: 68% (8281/12071) Updating files: 69% (8329/12071) Updating files: 69% (8344/12071) Updating files: 70% (8450/12071) Updating files: 70% (8485/12071) Updating files: 70% (8521/12071) Updating files: 70% (8563/12071) Updating files: 71% (8571/12071) Updating files: 71% (8605/12071) Updating files: 72% (8692/12071) Updating files: 73% (8812/12071) Updating files: 73% (8836/12071) Updating files: 73% (8844/12071) Updating files: 73% (8931/12071) Updating files: 74% (8933/12071) Updating files: 74% (8949/12071) Updating files: 74% (8957/12071) Updating files: 75% (9054/12071) Updating files: 75% (9136/12071) Updating files: 76% (9174/12071) Updating files: 77% (9295/12071) Updating files: 77% (9314/12071) Updating files: 78% (9416/12071) Updating files: 78% (9446/12071) Updating files: 79% (9537/12071) Updating files: 79% (9611/12071) Updating files: 79% (9614/12071) Updating files: 79% (9625/12071) Updating files: 80% (9657/12071) Updating files: 80% (9732/12071) Updating files: 81% (9778/12071) Updating files: 81% (9847/12071) Updating files: 82% (9899/12071) Updating files: 83% (10019/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 85% (10347/12071) Updating files: 86% (10382/12071) Updating files: 86% (10455/12071) Updating files: 87% (10502/12071) Updating files: 87% (10535/12071) Updating files: 88% (10623/12071) Updating files: 88% (10680/12071) Updating files: 89% (10744/12071) Updating files: 90% (10864/12071) Updating files: 90% (10870/12071) Updating files: 91% (10985/12071) Updating files: 92% (11106/12071) Updating files: 92% (11137/12071) Updating files: 93% (11227/12071) Updating files: 94% (11347/12071) Updating files: 94% (11416/12071) Updating files: 95% (11468/12071) Updating files: 96% (11589/12071) Updating files: 96% (11603/12071) Updating files: 96% (11670/12071) Updating files: 97% (11709/12071) Updating files: 98% (11830/12071) Updating files: 98% (11832/12071) Updating files: 98% (11845/12071) Updating files: 99% (11951/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T13:36:38.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.664+0000 7f4e37ad6640 1 -- 172.21.15.46:0/2298270803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e30159810 con 0x7f4e3015d6f0 2023-10-30T13:36:38.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.664+0000 7f4e267fc640 1 -- 172.21.15.46:0/2298270803 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f4e280b5090 con 0x7f4e3015d6f0 2023-10-30T13:36:38.672 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:36:38.672 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:16.036572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:38.672 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 71 2023-10-30T13:36:38.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.668+0000 7f4e37ad6640 1 -- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e140664a0 msgr2=0x7f4e14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:38.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.668+0000 7f4e37ad6640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e140664a0 0x7f4e14068960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c001ba0 tx=0x7f4e2c009040 comp rx=0 tx=0).stop 2023-10-30T13:36:38.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.668+0000 7f4e37ad6640 1 -- 172.21.15.46:0/2298270803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e3015d6f0 msgr2=0x7f4e300c8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:38.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.668+0000 7f4e37ad6640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e3015d6f0 0x7f4e300c8a30 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4e280b3040 tx=0x7f4e2801be00 comp rx=0 tx=0).stop 2023-10-30T13:36:38.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.668+0000 7f4e37ad6640 1 -- 172.21.15.46:0/2298270803 shutdown_connections 2023-10-30T13:36:38.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.668+0000 7f4e37ad6640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e140664a0 0x7f4e14068960 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.668+0000 7f4e37ad6640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e3015d6f0 0x7f4e300c8a30 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.668+0000 7f4e37ad6640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e301538a0 0x7f4e300c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.668+0000 7f4e37ad6640 1 --2- 172.21.15.46:0/2298270803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300cf160 0x7f4e301f0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.668+0000 7f4e37ad6640 1 -- 172.21.15.46:0/2298270803 >> 172.21.15.46:0/2298270803 conn(0x7f4e3006bfe0 msgr2=0x7f4e3015b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:38.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.668+0000 7f4e37ad6640 1 -- 172.21.15.46:0/2298270803 shutdown_connections 2023-10-30T13:36:38.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.668+0000 7f4e37ad6640 1 -- 172.21.15.46:0/2298270803 wait complete. 2023-10-30T13:36:38.694 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=2) 2023-10-30T13:36:38.694 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:36:38.694 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:36:38.712 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:36:38.712 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:38.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.892+0000 7f4616e09640 1 Processor -- start 2023-10-30T13:36:38.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f4616e09640 1 -- start start 2023-10-30T13:36:38.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f4616e09640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46101538a0 0x7f461015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:38.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f4616e09640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46100cf160 0x7f46100cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:38.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f4616e09640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46100cb800 con 0x7f46100ca0e0 2023-10-30T13:36:38.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f4616e09640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461015dc30 con 0x7f46100cf160 2023-10-30T13:36:38.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f4616e09640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46100cb330 con 0x7f46101538a0 2023-10-30T13:36:38.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f46153a6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46100cf160 0x7f46100cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:38.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f46153a6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46100cf160 0x7f46100cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44366/0 (socket says 172.21.15.46:44366) 2023-10-30T13:36:38.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f46153a6640 1 -- 172.21.15.46:0/865160076 learned_addr learned my addr 172.21.15.46:0/865160076 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:38.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f4614ba5640 1 --2- 172.21.15.46:0/865160076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46101538a0 0x7f461015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:38.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f46153a6640 1 -- 172.21.15.46:0/865160076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46101538a0 msgr2=0x7f461015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:38.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f46153a6640 1 --2- 172.21.15.46:0/865160076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46101538a0 0x7f461015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f46153a6640 1 -- 172.21.15.46:0/865160076 >> v1:172.21.15.154:6789/0 conn(0x7f46100ca0e0 legacy=0x7f461015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:38.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.896+0000 7f46153a6640 1 -- 172.21.15.46:0/865160076 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46100cad70 con 0x7f46100cf160 2023-10-30T13:36:38.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f46153a6640 1 --2- 172.21.15.46:0/865160076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46100cf160 0x7f46100cf590 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4608009e50 tx=0x7f460801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1668ee2b2c3573b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:38.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f460f7fe640 1 -- 172.21.15.46:0/865160076 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46080af400 con 0x7f46100cf160 2023-10-30T13:36:38.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f460f7fe640 1 -- 172.21.15.46:0/865160076 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46080ab050 con 0x7f46100cf160 2023-10-30T13:36:38.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f460f7fe640 1 -- 172.21.15.46:0/865160076 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46080af780 con 0x7f46100cf160 2023-10-30T13:36:38.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f4616e09640 1 -- 172.21.15.46:0/865160076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46100cf160 msgr2=0x7f46100cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:38.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f4616e09640 1 --2- 172.21.15.46:0/865160076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46100cf160 0x7f46100cf590 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4608009e50 tx=0x7f460801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f4616e09640 1 -- 172.21.15.46:0/865160076 shutdown_connections 2023-10-30T13:36:38.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f4616e09640 1 --2- 172.21.15.46:0/865160076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46100cf160 0x7f46100cf590 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f4616e09640 1 --2- 172.21.15.46:0/865160076 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46101538a0 0x7f461015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f4616e09640 1 -- 172.21.15.46:0/865160076 >> 172.21.15.46:0/865160076 conn(0x7f461006bfe0 msgr2=0x7f461006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:38.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f4616e09640 1 -- 172.21.15.46:0/865160076 shutdown_connections 2023-10-30T13:36:38.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f4616e09640 1 -- 172.21.15.46:0/865160076 wait complete. 2023-10-30T13:36:38.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f4616e09640 1 Processor -- start 2023-10-30T13:36:38.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.900+0000 7f4616e09640 1 -- start start 2023-10-30T13:36:38.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4616e09640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46100ca0e0 0x7f461015a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:38.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4616e09640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46100cf160 0x7f4610145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:38.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4616e09640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46101538a0 0x7f461015b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:38.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4616e09640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4610142b70 con 0x7f46100ca0e0 2023-10-30T13:36:38.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4616e09640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46101f3940 con 0x7f46101538a0 2023-10-30T13:36:38.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4616e09640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4610146310 con 0x7f46100cf160 2023-10-30T13:36:38.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4614ba5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46100ca0e0 0x7f461015a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:38.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4614ba5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46100ca0e0 0x7f461015a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44382/0 (socket says 172.21.15.46:44382) 2023-10-30T13:36:38.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4614ba5640 1 -- 172.21.15.46:0/3098018981 learned_addr learned my addr 172.21.15.46:0/3098018981 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:38.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f460ffff640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46100cf160 0x7f4610145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:38.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f46153a6640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46101538a0 0x7f461015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:38.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4614ba5640 1 -- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46100cf160 msgr2=0x7f4610145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:38.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4614ba5640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46100cf160 0x7f4610145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4614ba5640 1 -- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46101538a0 msgr2=0x7f461015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:38.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4614ba5640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46101538a0 0x7f461015b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:38.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4614ba5640 1 -- 172.21.15.46:0/3098018981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4608009b30 con 0x7f46100ca0e0 2023-10-30T13:36:38.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4614ba5640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46100ca0e0 0x7f461015a8b0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f460400d5f0 tx=0x7f460400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:38.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f460d7fa640 1 -- 172.21.15.46:0/3098018981 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46040125f0 con 0x7f46100ca0e0 2023-10-30T13:36:38.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f460d7fa640 1 -- 172.21.15.46:0/3098018981 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4604012750 con 0x7f46100ca0e0 2023-10-30T13:36:38.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f460d7fa640 1 -- 172.21.15.46:0/3098018981 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4604012b00 con 0x7f46100ca0e0 2023-10-30T13:36:38.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4616e09640 1 -- 172.21.15.46:0/3098018981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46101465f0 con 0x7f46100ca0e0 2023-10-30T13:36:38.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4616e09640 1 -- 172.21.15.46:0/3098018981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46101f2e60 con 0x7f46100ca0e0 2023-10-30T13:36:38.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.904+0000 7f4616e09640 1 -- 172.21.15.46:0/3098018981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46100b2a50 con 0x7f46100ca0e0 2023-10-30T13:36:38.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.908+0000 7f460d7fa640 1 -- 172.21.15.46:0/3098018981 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4604012c60 con 0x7f46100ca0e0 2023-10-30T13:36:38.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.908+0000 7f460d7fa640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45f40664a0 0x7f45f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:38.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.908+0000 7f460d7fa640 1 -- 172.21.15.46:0/3098018981 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f4604084730 con 0x7f46100ca0e0 2023-10-30T13:36:38.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.908+0000 7f460ffff640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45f40664a0 0x7f45f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:38.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.908+0000 7f460ffff640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45f40664a0 0x7f45f4068960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f46101432f0 tx=0x7f45fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:38.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:38.912+0000 7f460d7fa640 1 -- 172.21.15.46:0/3098018981 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4604051840 con 0x7f46100ca0e0 2023-10-30T13:36:39.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.104+0000 7f4616e09640 1 -- 172.21.15.46:0/3098018981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f461004e170 con 0x7f46100ca0e0 2023-10-30T13:36:39.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f460d7fa640 1 -- 172.21.15.46:0/3098018981 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f4604054e60 con 0x7f46100ca0e0 2023-10-30T13:36:39.114 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:36:39.114 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:16.036572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:39.115 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 71 2023-10-30T13:36:39.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f4616e09640 1 -- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45f40664a0 msgr2=0x7f45f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:39.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f4616e09640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45f40664a0 0x7f45f4068960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f46101432f0 tx=0x7f45fc009040 comp rx=0 tx=0).stop 2023-10-30T13:36:39.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f4616e09640 1 -- 172.21.15.46:0/3098018981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46100ca0e0 msgr2=0x7f461015a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:39.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f4616e09640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46100ca0e0 0x7f461015a8b0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f460400d5f0 tx=0x7f460400dab0 comp rx=0 tx=0).stop 2023-10-30T13:36:39.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f4616e09640 1 -- 172.21.15.46:0/3098018981 shutdown_connections 2023-10-30T13:36:39.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f4616e09640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45f40664a0 0x7f45f4068960 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:39.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f4616e09640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46101538a0 0x7f461015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f4616e09640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46100cf160 0x7f4610145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f4616e09640 1 --2- 172.21.15.46:0/3098018981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46100ca0e0 0x7f461015a8b0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f4616e09640 1 -- 172.21.15.46:0/3098018981 >> 172.21.15.46:0/3098018981 conn(0x7f461006bfe0 msgr2=0x7f4610159f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f4616e09640 1 -- 172.21.15.46:0/3098018981 shutdown_connections 2023-10-30T13:36:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:39.108+0000 7f4616e09640 1 -- 172.21.15.46:0/3098018981 wait complete. 2023-10-30T13:36:39.134 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:36:41.135 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.284+0000 7f117010f640 1 Processor -- start 2023-10-30T13:36:41.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.284+0000 7f117010f640 1 -- start start 2023-10-30T13:36:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.284+0000 7f117010f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f116806e470 0x7f116806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.284+0000 7f117010f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116806afa0 con 0x7f1168153d70 2023-10-30T13:36:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.284+0000 7f117010f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1168154150 con 0x7f11680c8490 2023-10-30T13:36:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.284+0000 7f117010f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116806d780 con 0x7f116806e470 2023-10-30T13:36:41.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.284+0000 7f116deab640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1168153d70 0x7f11680cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57088/0 (socket says 172.21.15.46:57088) 2023-10-30T13:36:41.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.284+0000 7f116deab640 1 -- 172.21.15.46:0/2885257708 learned_addr learned my addr 172.21.15.46:0/2885257708 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:36:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.284+0000 7f116d6aa640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f116806e470 0x7f116806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:41.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422543193 0 0) 0x7f116806afa0 con 0x7f1168153d70 2023-10-30T13:36:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1154003530 con 0x7f1168153d70 2023-10-30T13:36:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 431203173 0 0) 0x7f1154003530 con 0x7f1168153d70 2023-10-30T13:36:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f116806e470 msgr2=0x7f116806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f116806e470 0x7f116806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:41.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 >> v1:172.21.15.154:6789/0 conn(0x7f11680c8490 legacy=0x7f11680c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:41.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116806e030 con 0x7f1168153d70 2023-10-30T13:36:41.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f1164001680 con 0x7f1168153d70 2023-10-30T13:36:41.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116d6aa640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f116806e470 0x7f116806ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:36:41.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 >> v1:172.21.15.46:6789/0 conn(0x7f1168153d70 legacy=0x7f11680cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:41.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f116806e470 0x7f11540039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:41.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1154003ef0 0x7f11540047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:41.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1154004d10 0x7f1154005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:41.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116806d780 con 0x7f1154003ef0 2023-10-30T13:36:41.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115400b9f0 con 0x7f1154004d10 2023-10-30T13:36:41.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115400bbb0 con 0x7f116806e470 2023-10-30T13:36:41.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116d6aa640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1154003ef0 0x7f11540047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116deab640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f116806e470 0x7f11540039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116e6ac640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1154004d10 0x7f1154005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116d6aa640 1 -- 172.21.15.46:0/2885257708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f116806e470 msgr2=0x7f11540039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116d6aa640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f116806e470 0x7f11540039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116d6aa640 1 -- 172.21.15.46:0/2885257708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1154004d10 msgr2=0x7f1154005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116d6aa640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1154004d10 0x7f1154005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116d6aa640 1 -- 172.21.15.46:0/2885257708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1158009420 con 0x7f1154003ef0 2023-10-30T13:36:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.288+0000 7f116d6aa640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1154003ef0 0x7f11540047d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f1158009c70 tx=0x7f115801cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de23c8e206272bad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158004030 con 0x7f1154003ef0 2023-10-30T13:36:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1158004190 con 0x7f1154003ef0 2023-10-30T13:36:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f116cea9640 1 -- 172.21.15.46:0/2885257708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11580a5730 con 0x7f1154003ef0 2023-10-30T13:36:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 -- 172.21.15.46:0/2885257708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1154003ef0 msgr2=0x7f11540047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:41.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 --2- 172.21.15.46:0/2885257708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1154003ef0 0x7f11540047d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f1158009c70 tx=0x7f115801cb30 comp rx=0 tx=0).stop 2023-10-30T13:36:41.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f116e6ac640 1 -- 172.21.15.46:0/2885257708 reap_dead start 2023-10-30T13:36:41.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 -- 172.21.15.46:0/2885257708 shutdown_connections 2023-10-30T13:36:41.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 -- 172.21.15.46:0/2885257708 >> 172.21.15.46:0/2885257708 conn(0x7f1168071ea0 msgr2=0x7f1168065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 -- 172.21.15.46:0/2885257708 shutdown_connections 2023-10-30T13:36:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 -- 172.21.15.46:0/2885257708 wait complete. 2023-10-30T13:36:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 Processor -- start 2023-10-30T13:36:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 -- start start 2023-10-30T13:36:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1154003ef0 0x7f116815b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11681f2af0 0x7f11681efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11681f02a0 0x7f1168148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11681ef940 con 0x7f11681f02a0 2023-10-30T13:36:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11681f2fc0 con 0x7f11681f2af0 2023-10-30T13:36:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.292+0000 7f117010f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11681ef580 con 0x7f1154003ef0 2023-10-30T13:36:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f116e6ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11681f02a0 0x7f1168148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f116deab640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1154003ef0 0x7f116815b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f116d6aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11681f2af0 0x7f11681efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f116e6ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11681f02a0 0x7f1168148db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44402/0 (socket says 172.21.15.46:44402) 2023-10-30T13:36:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f116e6ac640 1 -- 172.21.15.46:0/2199453884 learned_addr learned my addr 172.21.15.46:0/2199453884 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f116e6ac640 1 -- 172.21.15.46:0/2199453884 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1154003ef0 msgr2=0x7f116815b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f116e6ac640 1 --2- 172.21.15.46:0/2199453884 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1154003ef0 0x7f116815b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:41.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f116e6ac640 1 -- 172.21.15.46:0/2199453884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11681f2af0 msgr2=0x7f11681efd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:41.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f116e6ac640 1 --2- 172.21.15.46:0/2199453884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11681f2af0 0x7f11681efd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:41.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f116e6ac640 1 -- 172.21.15.46:0/2199453884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116814bd40 con 0x7f11681f02a0 2023-10-30T13:36:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f116e6ac640 1 --2- 172.21.15.46:0/2199453884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11681f02a0 0x7f1168148db0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f11680700c0 tx=0x7f1160010eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f115e7fc640 1 -- 172.21.15.46:0/2199453884 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160003d90 con 0x7f11681f02a0 2023-10-30T13:36:41.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f115e7fc640 1 -- 172.21.15.46:0/2199453884 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116000d070 con 0x7f11681f02a0 2023-10-30T13:36:41.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f115e7fc640 1 -- 172.21.15.46:0/2199453884 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116000ea10 con 0x7f11681f02a0 2023-10-30T13:36:41.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f117010f640 1 -- 172.21.15.46:0/2199453884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116814c470 con 0x7f11681f02a0 2023-10-30T13:36:41.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f117010f640 1 -- 172.21.15.46:0/2199453884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11681f3560 con 0x7f11681f02a0 2023-10-30T13:36:41.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f117010f640 1 -- 172.21.15.46:0/2199453884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11681efa80 con 0x7f11681f02a0 2023-10-30T13:36:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.296+0000 7f115e7fc640 1 -- 172.21.15.46:0/2199453884 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f116000eb70 con 0x7f11681f02a0 2023-10-30T13:36:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.300+0000 7f115e7fc640 1 --2- 172.21.15.46:0/2199453884 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f114c0664a0 0x7f114c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.300+0000 7f115e7fc640 1 -- 172.21.15.46:0/2199453884 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f11600871f0 con 0x7f11681f02a0 2023-10-30T13:36:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.300+0000 7f116deab640 1 --2- 172.21.15.46:0/2199453884 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f114c0664a0 0x7f114c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.300+0000 7f116deab640 1 --2- 172.21.15.46:0/2199453884 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f114c0664a0 0x7f114c068960 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1164001060 tx=0x7f1164000f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.304+0000 7f115e7fc640 1 -- 172.21.15.46:0/2199453884 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f11681efa80 con 0x7f11681f02a0 2023-10-30T13:36:41.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 -- 172.21.15.46:0/2199453884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11681efa80 con 0x7f11681f02a0 2023-10-30T13:36:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f115e7fc640 1 -- 172.21.15.46:0/2199453884 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f11681efa80 con 0x7f11681f02a0 2023-10-30T13:36:41.502 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:36:41.502 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:16.036572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 71 2023-10-30T13:36:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 -- 172.21.15.46:0/2199453884 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f114c0664a0 msgr2=0x7f114c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 --2- 172.21.15.46:0/2199453884 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f114c0664a0 0x7f114c068960 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1164001060 tx=0x7f1164000f90 comp rx=0 tx=0).stop 2023-10-30T13:36:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 -- 172.21.15.46:0/2199453884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11681f02a0 msgr2=0x7f1168148db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 --2- 172.21.15.46:0/2199453884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11681f02a0 0x7f1168148db0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f11680700c0 tx=0x7f1160010eb0 comp rx=0 tx=0).stop 2023-10-30T13:36:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 -- 172.21.15.46:0/2199453884 shutdown_connections 2023-10-30T13:36:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 --2- 172.21.15.46:0/2199453884 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f114c0664a0 0x7f114c068960 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 --2- 172.21.15.46:0/2199453884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11681f02a0 0x7f1168148db0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 --2- 172.21.15.46:0/2199453884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11681f2af0 0x7f11681efd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 --2- 172.21.15.46:0/2199453884 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1154003ef0 0x7f116815b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:41.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 -- 172.21.15.46:0/2199453884 >> 172.21.15.46:0/2199453884 conn(0x7f1168071ea0 msgr2=0x7f1168152f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:41.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 -- 172.21.15.46:0/2199453884 shutdown_connections 2023-10-30T13:36:41.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:41.496+0000 7f117010f640 1 -- 172.21.15.46:0/2199453884 wait complete. 2023-10-30T13:36:41.523 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:36:43.524 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:43.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.672+0000 7f13dcac9640 1 Processor -- start 2023-10-30T13:36:43.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.672+0000 7f13dcac9640 1 -- start start 2023-10-30T13:36:43.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.672+0000 7f13dcac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d81580e0 0x7f13d8070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:43.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.672+0000 7f13dcac9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13d80c6540 0x7f13d815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:43.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.672+0000 7f13dcac9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f13d8069e90 0x7f13d806a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:43.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.672+0000 7f13dcac9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d80c5810 con 0x7f13d80c6540 2023-10-30T13:36:43.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.672+0000 7f13dcac9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d80c6960 con 0x7f13d8069e90 2023-10-30T13:36:43.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.672+0000 7f13dcac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80c6bb0 con 0x7f13d81580e0 2023-10-30T13:36:43.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13cffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13d80c6540 0x7f13d815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:43.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13cffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13d80c6540 0x7f13d815e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44406/0 (socket says 172.21.15.46:44406) 2023-10-30T13:36:43.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13cffff640 1 -- 172.21.15.46:0/3379606540 learned_addr learned my addr 172.21.15.46:0/3379606540 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:43.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13d659c640 1 --2- 172.21.15.46:0/3379606540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d81580e0 0x7f13d8070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:43.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13d6d9d640 1 --2- 172.21.15.46:0/3379606540 >> v2:172.21.15.154:3300/0 conn(0x7f13d8069e90 0x7f13d806a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:43.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13cffff640 1 -- 172.21.15.46:0/3379606540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d81580e0 msgr2=0x7f13d8070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:43.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13cffff640 1 --2- 172.21.15.46:0/3379606540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d81580e0 0x7f13d8070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13cffff640 1 -- 172.21.15.46:0/3379606540 >> v2:172.21.15.154:3300/0 conn(0x7f13d8069e90 msgr2=0x7f13d806a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:43.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13cffff640 1 --2- 172.21.15.46:0/3379606540 >> v2:172.21.15.154:3300/0 conn(0x7f13d8069e90 0x7f13d806a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13cffff640 1 -- 172.21.15.46:0/3379606540 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d80c6e70 con 0x7f13d80c6540 2023-10-30T13:36:43.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13cffff640 1 --2- 172.21.15.46:0/3379606540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d80c6540 0x7f13d815e700 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f13d80c54b0 tx=0x7f13c001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12133d83920a66b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:43.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13d5d9b640 1 -- 172.21.15.46:0/3379606540 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c00ac630 con 0x7f13d80c6540 2023-10-30T13:36:43.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13d5d9b640 1 -- 172.21.15.46:0/3379606540 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c00ac790 con 0x7f13d80c6540 2023-10-30T13:36:43.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.676+0000 7f13d5d9b640 1 -- 172.21.15.46:0/3379606540 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c00acb20 con 0x7f13d80c6540 2023-10-30T13:36:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 -- 172.21.15.46:0/3379606540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d80c6540 msgr2=0x7f13d815e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 --2- 172.21.15.46:0/3379606540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d80c6540 0x7f13d815e700 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f13d80c54b0 tx=0x7f13c001c500 comp rx=0 tx=0).stop 2023-10-30T13:36:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 -- 172.21.15.46:0/3379606540 shutdown_connections 2023-10-30T13:36:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 --2- 172.21.15.46:0/3379606540 >> v2:172.21.15.154:3300/0 conn(0x7f13d8069e90 0x7f13d806a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 --2- 172.21.15.46:0/3379606540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d80c6540 0x7f13d815e700 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 --2- 172.21.15.46:0/3379606540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d81580e0 0x7f13d8070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 -- 172.21.15.46:0/3379606540 >> 172.21.15.46:0/3379606540 conn(0x7f13d80c7910 msgr2=0x7f13d80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:43.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 -- 172.21.15.46:0/3379606540 shutdown_connections 2023-10-30T13:36:43.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 -- 172.21.15.46:0/3379606540 wait complete. 2023-10-30T13:36:43.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 Processor -- start 2023-10-30T13:36:43.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 -- start start 2023-10-30T13:36:43.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d8069e90 0x7f13d81f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:43.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d80c6540 0x7f13d80c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:43.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d81580e0 0x7f13d80c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:43.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d81f69f0 con 0x7f13d80c6540 2023-10-30T13:36:43.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80c9810 con 0x7f13d8069e90 2023-10-30T13:36:43.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13dcac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80c8f70 con 0x7f13d81580e0 2023-10-30T13:36:43.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d80c6540 0x7f13d80c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:43.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d80c6540 0x7f13d80c2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44412/0 (socket says 172.21.15.46:44412) 2023-10-30T13:36:43.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13cffff640 1 -- 172.21.15.46:0/3278606591 learned_addr learned my addr 172.21.15.46:0/3278606591 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:43.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13d659c640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d8069e90 0x7f13d81f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:43.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13d6d9d640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d81580e0 0x7f13d80c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13cffff640 1 -- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d81580e0 msgr2=0x7f13d80c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.680+0000 7f13cffff640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d81580e0 0x7f13d80c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13cffff640 1 -- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d8069e90 msgr2=0x7f13d81f6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13cffff640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d8069e90 0x7f13d81f6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13cffff640 1 -- 172.21.15.46:0/3278606591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c00094d0 con 0x7f13d80c6540 2023-10-30T13:36:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13d6d9d640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d81580e0 0x7f13d80c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13cffff640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d80c6540 0x7f13d80c2cb0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f13c001cda0 tx=0x7f13c00192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13ceffd640 1 -- 172.21.15.46:0/3278606591 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c00b6600 con 0x7f13d80c6540 2023-10-30T13:36:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13ceffd640 1 -- 172.21.15.46:0/3278606591 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c00b6780 con 0x7f13d80c6540 2023-10-30T13:36:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13ceffd640 1 -- 172.21.15.46:0/3278606591 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c00b6b30 con 0x7f13d80c6540 2023-10-30T13:36:43.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13dcac9640 1 -- 172.21.15.46:0/3278606591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d80c9250 con 0x7f13d80c6540 2023-10-30T13:36:43.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13dcac9640 1 -- 172.21.15.46:0/3278606591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d81f6680 con 0x7f13d80c6540 2023-10-30T13:36:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13dcac9640 1 -- 172.21.15.46:0/3278606591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d804e170 con 0x7f13d80c6540 2023-10-30T13:36:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13ceffd640 1 -- 172.21.15.46:0/3278606591 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13c00b6c90 con 0x7f13d80c6540 2023-10-30T13:36:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13ceffd640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13b80664a0 0x7f13b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13ceffd640 1 -- 172.21.15.46:0/3278606591 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f13c011e700 con 0x7f13d80c6540 2023-10-30T13:36:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.684+0000 7f13d659c640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13b80664a0 0x7f13b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.688+0000 7f13d659c640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13b80664a0 0x7f13b8068960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f13c80079c0 tx=0x7f13c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:43.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.688+0000 7f13ceffd640 1 -- 172.21.15.46:0/3278606591 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f13c00eb880 con 0x7f13d80c6540 2023-10-30T13:36:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.880+0000 7f13dcac9640 1 -- 172.21.15.46:0/3278606591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13d80c5d10 con 0x7f13d80c6540 2023-10-30T13:36:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.880+0000 7f13ceffd640 1 -- 172.21.15.46:0/3278606591 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f13c00eeea0 con 0x7f13d80c6540 2023-10-30T13:36:43.887 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:36:43.887 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:16.036572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:43.888 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 71 2023-10-30T13:36:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.884+0000 7f13dcac9640 1 -- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13b80664a0 msgr2=0x7f13b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.884+0000 7f13dcac9640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13b80664a0 0x7f13b8068960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f13c80079c0 tx=0x7f13c8008040 comp rx=0 tx=0).stop 2023-10-30T13:36:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.884+0000 7f13dcac9640 1 -- 172.21.15.46:0/3278606591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d80c6540 msgr2=0x7f13d80c2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.884+0000 7f13dcac9640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d80c6540 0x7f13d80c2cb0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f13c001cda0 tx=0x7f13c00192a0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.884+0000 7f13dcac9640 1 -- 172.21.15.46:0/3278606591 shutdown_connections 2023-10-30T13:36:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.884+0000 7f13dcac9640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13b80664a0 0x7f13b8068960 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.884+0000 7f13dcac9640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d81580e0 0x7f13d80c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.884+0000 7f13dcac9640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13d80c6540 0x7f13d80c2cb0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.884+0000 7f13dcac9640 1 --2- 172.21.15.46:0/3278606591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d8069e90 0x7f13d81f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.884+0000 7f13dcac9640 1 -- 172.21.15.46:0/3278606591 >> 172.21.15.46:0/3278606591 conn(0x7f13d80c7910 msgr2=0x7f13d8153050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.884+0000 7f13dcac9640 1 -- 172.21.15.46:0/3278606591 shutdown_connections 2023-10-30T13:36:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:43.884+0000 7f13dcac9640 1 -- 172.21.15.46:0/3278606591 wait complete. 2023-10-30T13:36:43.909 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:36:45.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:46.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44baf1d640 1 Processor -- start 2023-10-30T13:36:46.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44baf1d640 1 -- start start 2023-10-30T13:36:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44baf1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44b415d6f0 0x7f44b415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44baf1d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44b40cf160 0x7f44b40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44baf1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b40cb800 con 0x7f44b41538a0 2023-10-30T13:36:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44baf1d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b40cb330 con 0x7f44b40cf160 2023-10-30T13:36:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44baf1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b40cad70 con 0x7f44b415d6f0 2023-10-30T13:36:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44b8cb9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f44b41538a0 0x7f44b40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57098/0 (socket says 172.21.15.46:57098) 2023-10-30T13:36:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44b8cb9640 1 -- 172.21.15.46:0/3153944893 learned_addr learned my addr 172.21.15.46:0/3153944893 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:36:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44b94ba640 1 --2- 172.21.15.46:0/3153944893 >> v2:172.21.15.46:3300/0 conn(0x7f44b40cf160 0x7f44b40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44b3fff640 1 --2- 172.21.15.46:0/3153944893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44b415d6f0 0x7f44b415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44b94ba640 1 -- 172.21.15.46:0/3153944893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44b415d6f0 msgr2=0x7f44b415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44b94ba640 1 --2- 172.21.15.46:0/3153944893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44b415d6f0 0x7f44b415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44b94ba640 1 -- 172.21.15.46:0/3153944893 >> v1:172.21.15.46:6789/0 conn(0x7f44b41538a0 legacy=0x7f44b40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44b94ba640 1 -- 172.21.15.46:0/3153944893 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b40cb050 con 0x7f44b40cf160 2023-10-30T13:36:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.060+0000 7f44b37fe640 1 -- 172.21.15.46:0/3153944893 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175999240 0 0) 0x7f44b40cb800 con 0x7f44b41538a0 2023-10-30T13:36:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.064+0000 7f44b94ba640 1 --2- 172.21.15.46:0/3153944893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44b40cf160 0x7f44b40cf590 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f44ac009d90 tx=0x7f44ac01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797c12da9b85492a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.064+0000 7f44b37fe640 1 -- 172.21.15.46:0/3153944893 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac0af600 con 0x7f44b40cf160 2023-10-30T13:36:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.064+0000 7f44b37fe640 1 -- 172.21.15.46:0/3153944893 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ac0ab050 con 0x7f44b40cf160 2023-10-30T13:36:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.064+0000 7f44b37fe640 1 -- 172.21.15.46:0/3153944893 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac0af980 con 0x7f44b40cf160 2023-10-30T13:36:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.064+0000 7f44baf1d640 1 -- 172.21.15.46:0/3153944893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44b40cf160 msgr2=0x7f44b40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.064+0000 7f44baf1d640 1 --2- 172.21.15.46:0/3153944893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44b40cf160 0x7f44b40cf590 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f44ac009d90 tx=0x7f44ac01f130 comp rx=0 tx=0).stop 2023-10-30T13:36:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.064+0000 7f44baf1d640 1 -- 172.21.15.46:0/3153944893 shutdown_connections 2023-10-30T13:36:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.064+0000 7f44baf1d640 1 --2- 172.21.15.46:0/3153944893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44b40cf160 0x7f44b40cf590 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.064+0000 7f44baf1d640 1 --2- 172.21.15.46:0/3153944893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44b415d6f0 0x7f44b415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.064+0000 7f44baf1d640 1 -- 172.21.15.46:0/3153944893 >> 172.21.15.46:0/3153944893 conn(0x7f44b406bfe0 msgr2=0x7f44b406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.064+0000 7f44baf1d640 1 -- 172.21.15.46:0/3153944893 shutdown_connections 2023-10-30T13:36:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.064+0000 7f44baf1d640 1 -- 172.21.15.46:0/3153944893 wait complete. 2023-10-30T13:36:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44baf1d640 1 Processor -- start 2023-10-30T13:36:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44baf1d640 1 -- start start 2023-10-30T13:36:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44baf1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44b40cf160 0x7f44b41e2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44baf1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44b41538a0 0x7f44b40b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44baf1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44b415d6f0 0x7f44b41e4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44baf1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b41e5860 con 0x7f44b415d6f0 2023-10-30T13:36:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44baf1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b4157d60 con 0x7f44b41538a0 2023-10-30T13:36:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44baf1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b4158690 con 0x7f44b40cf160 2023-10-30T13:36:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b94ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44b415d6f0 0x7f44b41e4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b94ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44b415d6f0 0x7f44b41e4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44434/0 (socket says 172.21.15.46:44434) 2023-10-30T13:36:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b94ba640 1 -- 172.21.15.46:0/1856411541 learned_addr learned my addr 172.21.15.46:0/1856411541 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b8cb9640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44b40cf160 0x7f44b41e2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b3fff640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44b41538a0 0x7f44b40b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b94ba640 1 -- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44b40cf160 msgr2=0x7f44b41e2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:46.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b94ba640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44b40cf160 0x7f44b41e2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:46.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b94ba640 1 -- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44b41538a0 msgr2=0x7f44b40b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:46.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b94ba640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44b41538a0 0x7f44b40b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:46.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b94ba640 1 -- 172.21.15.46:0/1856411541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ac009a70 con 0x7f44b415d6f0 2023-10-30T13:36:46.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b8cb9640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44b40cf160 0x7f44b41e2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b94ba640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44b415d6f0 0x7f44b41e4d10 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0b2040 tx=0x7f44ac01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b17fa640 1 -- 172.21.15.46:0/1856411541 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac009450 con 0x7f44b415d6f0 2023-10-30T13:36:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b17fa640 1 -- 172.21.15.46:0/1856411541 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ac0b9410 con 0x7f44b415d6f0 2023-10-30T13:36:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44b17fa640 1 -- 172.21.15.46:0/1856411541 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac0b97c0 con 0x7f44b415d6f0 2023-10-30T13:36:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44baf1d640 1 -- 172.21.15.46:0/1856411541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b4158950 con 0x7f44b415d6f0 2023-10-30T13:36:46.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44baf1d640 1 -- 172.21.15.46:0/1856411541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b4157710 con 0x7f44b415d6f0 2023-10-30T13:36:46.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.068+0000 7f44baf1d640 1 -- 172.21.15.46:0/1856411541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b40b2a50 con 0x7f44b415d6f0 2023-10-30T13:36:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.072+0000 7f44b17fa640 1 -- 172.21.15.46:0/1856411541 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f44ac0b6050 con 0x7f44b415d6f0 2023-10-30T13:36:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.072+0000 7f44b17fa640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f44980664a0 0x7f4498068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.072+0000 7f44b17fa640 1 -- 172.21.15.46:0/1856411541 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f44ac122610 con 0x7f44b415d6f0 2023-10-30T13:36:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.072+0000 7f44b8cb9640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f44980664a0 0x7f4498068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.072+0000 7f44b8cb9640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f44980664a0 0x7f4498068960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f44a8007fd0 tx=0x7f44a8007c20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:46.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.076+0000 7f44b17fa640 1 -- 172.21.15.46:0/1856411541 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f44ac0ef820 con 0x7f44b415d6f0 2023-10-30T13:36:46.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 -- 172.21.15.46:0/1856411541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44b404e170 con 0x7f44b415d6f0 2023-10-30T13:36:46.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44b17fa640 1 -- 172.21.15.46:0/1856411541 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f44ac0f2e40 con 0x7f44b415d6f0 2023-10-30T13:36:46.274 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:36:46.274 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:16.036572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:46.274 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 71 2023-10-30T13:36:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 -- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f44980664a0 msgr2=0x7f4498068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:46.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f44980664a0 0x7f4498068960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f44a8007fd0 tx=0x7f44a8007c20 comp rx=0 tx=0).stop 2023-10-30T13:36:46.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 -- 172.21.15.46:0/1856411541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44b415d6f0 msgr2=0x7f44b41e4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:46.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44b415d6f0 0x7f44b41e4d10 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0b2040 tx=0x7f44ac01bed0 comp rx=0 tx=0).stop 2023-10-30T13:36:46.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 -- 172.21.15.46:0/1856411541 shutdown_connections 2023-10-30T13:36:46.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f44980664a0 0x7f4498068960 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:46.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44b415d6f0 0x7f44b41e4d10 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:46.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44b41538a0 0x7f44b40b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:46.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 --2- 172.21.15.46:0/1856411541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44b40cf160 0x7f44b41e2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:46.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 -- 172.21.15.46:0/1856411541 >> 172.21.15.46:0/1856411541 conn(0x7f44b406bfe0 msgr2=0x7f44b4151a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:46.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 -- 172.21.15.46:0/1856411541 shutdown_connections 2023-10-30T13:36:46.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:46.268+0000 7f44baf1d640 1 -- 172.21.15.46:0/1856411541 wait complete. 2023-10-30T13:36:46.295 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:36:47.134 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:47.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:48.296 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:48.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.580+0000 7f64994ea640 1 Processor -- start 2023-10-30T13:36:48.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.580+0000 7f64994ea640 1 -- start start 2023-10-30T13:36:48.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.588+0000 7f64994ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6494152ec0 0x7f6494159790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:48.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.588+0000 7f64994ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6494071dc0 0x7f64940715a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:48.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.588+0000 7f64994ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64940c8430 0x7f64941557e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:48.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.588+0000 7f64994ea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f649406fd10 con 0x7f6494152ec0 2023-10-30T13:36:48.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.588+0000 7f64994ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6494070b90 con 0x7f6494071dc0 2023-10-30T13:36:48.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.588+0000 7f64994ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64941532a0 con 0x7f64940c8430 2023-10-30T13:36:48.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.588+0000 7f64937fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64940c8430 0x7f64941557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.588+0000 7f64937fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64940c8430 0x7f64941557e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:52494/0 (socket says 172.21.15.46:52494) 2023-10-30T13:36:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.588+0000 7f64937fe640 1 -- 172.21.15.46:0/1358699890 learned_addr learned my addr 172.21.15.46:0/1358699890 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.592+0000 7f64927fc640 1 --2- 172.21.15.46:0/1358699890 >> v2:172.21.15.154:3300/0 conn(0x7f6494071dc0 0x7f64940715a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:48.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.592+0000 7f64937fe640 1 -- 172.21.15.46:0/1358699890 >> v2:172.21.15.154:3300/0 conn(0x7f6494071dc0 msgr2=0x7f64940715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:48.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.592+0000 7f64937fe640 1 --2- 172.21.15.46:0/1358699890 >> v2:172.21.15.154:3300/0 conn(0x7f6494071dc0 0x7f64940715a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.592+0000 7f64937fe640 1 -- 172.21.15.46:0/1358699890 >> v2:172.21.15.46:3300/0 conn(0x7f6494152ec0 msgr2=0x7f6494159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:48.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.592+0000 7f64937fe640 1 --2- 172.21.15.46:0/1358699890 >> v2:172.21.15.46:3300/0 conn(0x7f6494152ec0 0x7f6494159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.592+0000 7f64937fe640 1 -- 172.21.15.46:0/1358699890 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64940c7280 con 0x7f64940c8430 2023-10-30T13:36:48.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.592+0000 7f64927fc640 1 --2- 172.21.15.46:0/1358699890 >> v2:172.21.15.154:3300/0 conn(0x7f6494071dc0 0x7f64940715a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:36:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.604+0000 7f64937fe640 1 --2- 172.21.15.46:0/1358699890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64940c8430 0x7f64941557e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6494070540 tx=0x7f648c01c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c4dd8a6a45e845f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.604+0000 7f6491ffb640 1 -- 172.21.15.46:0/1358699890 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c0a5820 con 0x7f64940c8430 2023-10-30T13:36:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.604+0000 7f6491ffb640 1 -- 172.21.15.46:0/1358699890 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648c0a5980 con 0x7f64940c8430 2023-10-30T13:36:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.604+0000 7f6491ffb640 1 -- 172.21.15.46:0/1358699890 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c0a5d10 con 0x7f64940c8430 2023-10-30T13:36:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.608+0000 7f64994ea640 1 -- 172.21.15.46:0/1358699890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64940c8430 msgr2=0x7f64941557e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.608+0000 7f64994ea640 1 --2- 172.21.15.46:0/1358699890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64940c8430 0x7f64941557e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6494070540 tx=0x7f648c01c9a0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.608+0000 7f64994ea640 1 -- 172.21.15.46:0/1358699890 shutdown_connections 2023-10-30T13:36:48.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.608+0000 7f64994ea640 1 --2- 172.21.15.46:0/1358699890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64940c8430 0x7f64941557e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.608+0000 7f64994ea640 1 --2- 172.21.15.46:0/1358699890 >> v2:172.21.15.154:3300/0 conn(0x7f6494071dc0 0x7f64940715a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.608+0000 7f64994ea640 1 --2- 172.21.15.46:0/1358699890 >> v2:172.21.15.46:3300/0 conn(0x7f6494152ec0 0x7f6494159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.608+0000 7f64994ea640 1 -- 172.21.15.46:0/1358699890 >> 172.21.15.46:0/1358699890 conn(0x7f64940c7910 msgr2=0x7f64940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:48.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.608+0000 7f64994ea640 1 -- 172.21.15.46:0/1358699890 shutdown_connections 2023-10-30T13:36:48.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.608+0000 7f64994ea640 1 -- 172.21.15.46:0/1358699890 wait complete. 2023-10-30T13:36:48.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.616+0000 7f64994ea640 1 Processor -- start 2023-10-30T13:36:48.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.616+0000 7f64994ea640 1 -- start start 2023-10-30T13:36:48.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.616+0000 7f64994ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6494071dc0 0x7f6494158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:48.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.616+0000 7f64994ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64940c8430 0x7f6494158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:48.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.616+0000 7f64994ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6494152ec0 0x7f64941ff470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:48.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.616+0000 7f64994ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6494159470 con 0x7f6494071dc0 2023-10-30T13:36:48.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.616+0000 7f64994ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64940c5e50 con 0x7f64940c8430 2023-10-30T13:36:48.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.616+0000 7f64994ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64941596a0 con 0x7f6494152ec0 2023-10-30T13:36:48.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.616+0000 7f6492ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6494071dc0 0x7f6494158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:48.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.616+0000 7f6492ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6494071dc0 0x7f6494158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57944/0 (socket says 172.21.15.46:57944) 2023-10-30T13:36:48.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.616+0000 7f6492ffd640 1 -- 172.21.15.46:0/3518857177 learned_addr learned my addr 172.21.15.46:0/3518857177 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:48.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.620+0000 7f6492ffd640 1 -- 172.21.15.46:0/3518857177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6494152ec0 msgr2=0x7f64941ff470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T13:36:48.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.620+0000 7f6492ffd640 1 --2- 172.21.15.46:0/3518857177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6494152ec0 0x7f64941ff470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.620+0000 7f6492ffd640 1 -- 172.21.15.46:0/3518857177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64940c8430 msgr2=0x7f6494158bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T13:36:48.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.620+0000 7f6492ffd640 1 --2- 172.21.15.46:0/3518857177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64940c8430 0x7f6494158bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.620+0000 7f6492ffd640 1 -- 172.21.15.46:0/3518857177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c009b30 con 0x7f6494071dc0 2023-10-30T13:36:48.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.620+0000 7f6492ffd640 1 --2- 172.21.15.46:0/3518857177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6494071dc0 0x7f6494158690 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f648800b410 tx=0x7f648800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:48.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.624+0000 7f647b7fe640 1 -- 172.21.15.46:0/3518857177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6488011830 con 0x7f6494071dc0 2023-10-30T13:36:48.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.624+0000 7f647b7fe640 1 -- 172.21.15.46:0/3518857177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64880119b0 con 0x7f6494071dc0 2023-10-30T13:36:48.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.624+0000 7f647b7fe640 1 -- 172.21.15.46:0/3518857177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6488011d60 con 0x7f6494071dc0 2023-10-30T13:36:48.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.628+0000 7f64994ea640 1 -- 172.21.15.46:0/3518857177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6494159960 con 0x7f6494071dc0 2023-10-30T13:36:48.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.628+0000 7f64994ea640 1 -- 172.21.15.46:0/3518857177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64941537e0 con 0x7f6494071dc0 2023-10-30T13:36:48.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.632+0000 7f64994ea640 1 -- 172.21.15.46:0/3518857177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649404e170 con 0x7f6494071dc0 2023-10-30T13:36:48.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.636+0000 7f647b7fe640 1 -- 172.21.15.46:0/3518857177 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f648801a400 con 0x7f6494071dc0 2023-10-30T13:36:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.636+0000 7f647b7fe640 1 --2- 172.21.15.46:0/3518857177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6480066520 0x7f64800689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:48.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.636+0000 7f647b7fe640 1 -- 172.21.15.46:0/3518857177 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f6488084170 con 0x7f6494071dc0 2023-10-30T13:36:48.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.636+0000 7f64927fc640 1 --2- 172.21.15.46:0/3518857177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6480066520 0x7f64800689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:48.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.640+0000 7f64927fc640 1 --2- 172.21.15.46:0/3518857177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6480066520 0x7f64800689e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6494151960 tx=0x7f6484008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:48.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.648+0000 7f647b7fe640 1 -- 172.21.15.46:0/3518857177 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6488051350 con 0x7f6494071dc0 2023-10-30T13:36:48.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.848+0000 7f64994ea640 1 -- 172.21.15.46:0/3518857177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f649414f150 con 0x7f6494071dc0 2023-10-30T13:36:48.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.848+0000 7f647b7fe640 1 -- 172.21.15.46:0/3518857177 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f6488015090 con 0x7f6494071dc0 2023-10-30T13:36:48.856 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:36:48.856 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:16.036572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:48.856 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 71 2023-10-30T13:36:48.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.852+0000 7f64994ea640 1 -- 172.21.15.46:0/3518857177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6480066520 msgr2=0x7f64800689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:48.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.852+0000 7f64994ea640 1 --2- 172.21.15.46:0/3518857177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6480066520 0x7f64800689e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6494151960 tx=0x7f6484008040 comp rx=0 tx=0).stop 2023-10-30T13:36:48.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.852+0000 7f64994ea640 1 -- 172.21.15.46:0/3518857177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6494071dc0 msgr2=0x7f6494158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:48.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.852+0000 7f64994ea640 1 --2- 172.21.15.46:0/3518857177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6494071dc0 0x7f6494158690 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f648800b410 tx=0x7f648800b8d0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.852+0000 7f64994ea640 1 -- 172.21.15.46:0/3518857177 shutdown_connections 2023-10-30T13:36:48.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.852+0000 7f64994ea640 1 --2- 172.21.15.46:0/3518857177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6480066520 0x7f64800689e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.852+0000 7f64994ea640 1 --2- 172.21.15.46:0/3518857177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6494152ec0 0x7f64941ff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.852+0000 7f64994ea640 1 --2- 172.21.15.46:0/3518857177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64940c8430 0x7f6494158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.852+0000 7f64994ea640 1 --2- 172.21.15.46:0/3518857177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6494071dc0 0x7f6494158690 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:48.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.852+0000 7f64994ea640 1 -- 172.21.15.46:0/3518857177 >> 172.21.15.46:0/3518857177 conn(0x7f64940c7910 msgr2=0x7f64940c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:48.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.852+0000 7f64994ea640 1 -- 172.21.15.46:0/3518857177 shutdown_connections 2023-10-30T13:36:48.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:48.852+0000 7f64994ea640 1 -- 172.21.15.46:0/3518857177 wait complete. 2023-10-30T13:36:48.878 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:36:50.879 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:51.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.020+0000 7fbe600d6640 1 Processor -- start 2023-10-30T13:36:51.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.020+0000 7fbe600d6640 1 -- start start 2023-10-30T13:36:51.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe600d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbe580c05d0 0x7fbe5813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:51.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe600d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe580c6f60 0x7fbe5813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:51.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe600d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe580c0a50 con 0x7fbe580c6b80 2023-10-30T13:36:51.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe600d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe5806f6e0 con 0x7fbe580c05d0 2023-10-30T13:36:51.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe600d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5806f9c0 con 0x7fbe580c6f60 2023-10-30T13:36:51.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5d671640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbe580c05d0 0x7fbe5813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:51.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5de72640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbe580c6b80 0x7fbe580bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58040/0 (socket says 172.21.15.46:58040) 2023-10-30T13:36:51.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5de72640 1 -- 172.21.15.46:0/535709961 learned_addr learned my addr 172.21.15.46:0/535709961 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:36:51.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5e673640 1 --2- 172.21.15.46:0/535709961 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe580c6f60 0x7fbe5813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:51.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5d671640 1 -- 172.21.15.46:0/535709961 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe580c6f60 msgr2=0x7fbe5813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:51.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5d671640 1 --2- 172.21.15.46:0/535709961 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe580c6f60 0x7fbe5813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:51.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5d671640 1 -- 172.21.15.46:0/535709961 >> v1:172.21.15.46:6789/0 conn(0x7fbe580c6b80 legacy=0x7fbe580bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:51.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5d671640 1 -- 172.21.15.46:0/535709961 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5806fca0 con 0x7fbe580c05d0 2023-10-30T13:36:51.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5ce70640 1 -- 172.21.15.46:0/535709961 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930640505 0 0) 0x7fbe580c0a50 con 0x7fbe580c6b80 2023-10-30T13:36:51.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5d671640 1 --2- 172.21.15.46:0/535709961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe580c05d0 0x7fbe5813ed90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fbe480097c0 tx=0x7fbe4801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae18b1d9f628b6c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:51.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5ce70640 1 -- 172.21.15.46:0/535709961 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe480ad640 con 0x7fbe580c05d0 2023-10-30T13:36:51.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5ce70640 1 -- 172.21.15.46:0/535709961 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe480ad7a0 con 0x7fbe580c05d0 2023-10-30T13:36:51.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.024+0000 7fbe5ce70640 1 -- 172.21.15.46:0/535709961 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe480adb30 con 0x7fbe580c05d0 2023-10-30T13:36:51.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 -- 172.21.15.46:0/535709961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe580c05d0 msgr2=0x7fbe5813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:51.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 --2- 172.21.15.46:0/535709961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe580c05d0 0x7fbe5813ed90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fbe480097c0 tx=0x7fbe4801c480 comp rx=0 tx=0).stop 2023-10-30T13:36:51.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 -- 172.21.15.46:0/535709961 shutdown_connections 2023-10-30T13:36:51.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 --2- 172.21.15.46:0/535709961 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe580c6f60 0x7fbe5813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:51.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 --2- 172.21.15.46:0/535709961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe580c05d0 0x7fbe5813ed90 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:51.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 -- 172.21.15.46:0/535709961 >> 172.21.15.46:0/535709961 conn(0x7fbe580c7910 msgr2=0x7fbe580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:51.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 -- 172.21.15.46:0/535709961 shutdown_connections 2023-10-30T13:36:51.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 -- 172.21.15.46:0/535709961 wait complete. 2023-10-30T13:36:51.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 Processor -- start 2023-10-30T13:36:51.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 -- start start 2023-10-30T13:36:51.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe580c05d0 0x7fbe580c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:51.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe580c6b80 0x7fbe580c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:51.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe580c6f60 0x7fbe5814e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe580c8050 con 0x7fbe580c6f60 2023-10-30T13:36:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe581f0170 con 0x7fbe580c6b80 2023-10-30T13:36:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe600d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe581f0600 con 0x7fbe580c05d0 2023-10-30T13:36:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe5e673640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe580c6f60 0x7fbe5814e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe5e673640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe580c6f60 0x7fbe5814e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57960/0 (socket says 172.21.15.46:57960) 2023-10-30T13:36:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe5e673640 1 -- 172.21.15.46:0/1994772582 learned_addr learned my addr 172.21.15.46:0/1994772582 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe5de72640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe580c05d0 0x7fbe580c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.028+0000 7fbe5d671640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe580c6b80 0x7fbe580c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe5e673640 1 -- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe580c05d0 msgr2=0x7fbe580c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe5e673640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe580c05d0 0x7fbe580c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe5e673640 1 -- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe580c6b80 msgr2=0x7fbe580c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe5e673640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe580c6b80 0x7fbe580c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe5e673640 1 -- 172.21.15.46:0/1994772582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe48009420 con 0x7fbe580c6f60 2023-10-30T13:36:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe5e673640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe580c6f60 0x7fbe5814e1a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fbe50007e60 tx=0x7fbe5000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe4e7fc640 1 -- 172.21.15.46:0/1994772582 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe50014420 con 0x7fbe580c6f60 2023-10-30T13:36:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe4e7fc640 1 -- 172.21.15.46:0/1994772582 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe500145a0 con 0x7fbe580c6f60 2023-10-30T13:36:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe4e7fc640 1 -- 172.21.15.46:0/1994772582 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe50014950 con 0x7fbe580c6f60 2023-10-30T13:36:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe600d6640 1 -- 172.21.15.46:0/1994772582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe580c7d90 con 0x7fbe580c6f60 2023-10-30T13:36:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe600d6640 1 -- 172.21.15.46:0/1994772582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe581efb20 con 0x7fbe580c6f60 2023-10-30T13:36:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe600d6640 1 -- 172.21.15.46:0/1994772582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe5804e170 con 0x7fbe580c6f60 2023-10-30T13:36:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe4e7fc640 1 -- 172.21.15.46:0/1994772582 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbe50014ab0 con 0x7fbe580c6f60 2023-10-30T13:36:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe4e7fc640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe3c0664a0 0x7fbe3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe4e7fc640 1 -- 172.21.15.46:0/1994772582 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fbe50084b40 con 0x7fbe580c6f60 2023-10-30T13:36:51.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.032+0000 7fbe5de72640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe3c0664a0 0x7fbe3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:51.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.036+0000 7fbe5de72640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe3c0664a0 0x7fbe3c068960 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbe54007fd0 tx=0x7fbe54007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:51.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.036+0000 7fbe4e7fc640 1 -- 172.21.15.46:0/1994772582 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbe50051ba0 con 0x7fbe580c6f60 2023-10-30T13:36:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 -- 172.21.15.46:0/1994772582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe581f02b0 con 0x7fbe580c6f60 2023-10-30T13:36:51.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe4e7fc640 1 -- 172.21.15.46:0/1994772582 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6535 (secure 0 0 0) 0x7fbe500551c0 con 0x7fbe580c6f60 2023-10-30T13:36:51.234 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:36:51.234 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:16.036572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:51.235 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 71 2023-10-30T13:36:51.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 -- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe3c0664a0 msgr2=0x7fbe3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:51.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe3c0664a0 0x7fbe3c068960 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbe54007fd0 tx=0x7fbe54007d60 comp rx=0 tx=0).stop 2023-10-30T13:36:51.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 -- 172.21.15.46:0/1994772582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe580c6f60 msgr2=0x7fbe5814e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:51.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe580c6f60 0x7fbe5814e1a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fbe50007e60 tx=0x7fbe5000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:36:51.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 -- 172.21.15.46:0/1994772582 shutdown_connections 2023-10-30T13:36:51.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe3c0664a0 0x7fbe3c068960 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:51.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe580c6f60 0x7fbe5814e1a0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:51.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe580c6b80 0x7fbe580c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:51.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 --2- 172.21.15.46:0/1994772582 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe580c05d0 0x7fbe580c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:51.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 -- 172.21.15.46:0/1994772582 >> 172.21.15.46:0/1994772582 conn(0x7fbe580c7910 msgr2=0x7fbe5813ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:51.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 -- 172.21.15.46:0/1994772582 shutdown_connections 2023-10-30T13:36:51.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:51.228+0000 7fbe600d6640 1 -- 172.21.15.46:0/1994772582 wait complete. 2023-10-30T13:36:51.255 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:36:53.256 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:53.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.404+0000 7f853669a640 1 Processor -- start 2023-10-30T13:36:53.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.404+0000 7f853669a640 1 -- start start 2023-10-30T13:36:53.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.404+0000 7f853669a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f85300c6b80 0x7f85300bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:53.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.404+0000 7f853669a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85300c6f60 0x7f85300c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:53.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.404+0000 7f853669a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85300c0a50 con 0x7f85300c8430 2023-10-30T13:36:53.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.404+0000 7f853669a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f853006f6e0 con 0x7f85300c6b80 2023-10-30T13:36:53.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.404+0000 7f853669a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853006f9c0 con 0x7f85300c6f60 2023-10-30T13:36:53.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.404+0000 7f8534c37640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f85300c8430 0x7f853013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58056/0 (socket says 172.21.15.46:58056) 2023-10-30T13:36:53.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.404+0000 7f8534c37640 1 -- 172.21.15.46:0/1571023748 learned_addr learned my addr 172.21.15.46:0/1571023748 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:36:53.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.404+0000 7f852ffff640 1 --2- 172.21.15.46:0/1571023748 >> v2:172.21.15.154:3300/0 conn(0x7f85300c6b80 0x7f85300bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:53.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.404+0000 7f852f7fe640 1 --2- 172.21.15.46:0/1571023748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85300c6f60 0x7f85300c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:53.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.408+0000 7f852effd640 1 -- 172.21.15.46:0/1571023748 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771812187 0 0) 0x7f85300c0a50 con 0x7f85300c8430 2023-10-30T13:36:53.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.408+0000 7f852effd640 1 -- 172.21.15.46:0/1571023748 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8518003530 con 0x7f85300c8430 2023-10-30T13:36:53.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.408+0000 7f852ffff640 1 -- 172.21.15.46:0/1571023748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85300c6f60 msgr2=0x7f85300c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:53.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.408+0000 7f852ffff640 1 --2- 172.21.15.46:0/1571023748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85300c6f60 0x7f85300c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:53.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.408+0000 7f852ffff640 1 -- 172.21.15.46:0/1571023748 >> v1:172.21.15.46:6789/0 conn(0x7f85300c8430 legacy=0x7f853013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:53.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.408+0000 7f852effd640 1 -- 172.21.15.46:0/1571023748 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1357367411 0 0) 0x7f8518003530 con 0x7f85300c8430 2023-10-30T13:36:53.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.408+0000 7f852ffff640 1 -- 172.21.15.46:0/1571023748 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853006fca0 con 0x7f85300c6b80 2023-10-30T13:36:53.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.408+0000 7f852f7fe640 1 --2- 172.21.15.46:0/1571023748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85300c6f60 0x7f85300c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:36:53.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.408+0000 7f852ffff640 1 --2- 172.21.15.46:0/1571023748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85300c6b80 0x7f85300bfec0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f85200097c0 tx=0x7f852001c310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e865ea095441e81 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:53.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f852effd640 1 -- 172.21.15.46:0/1571023748 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8528001a00 con 0x7f85300c6b80 2023-10-30T13:36:53.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f852effd640 1 -- 172.21.15.46:0/1571023748 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85200a7050 con 0x7f85300c6b80 2023-10-30T13:36:53.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f852effd640 1 -- 172.21.15.46:0/1571023748 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85200ab600 con 0x7f85300c6b80 2023-10-30T13:36:53.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f853669a640 1 -- 172.21.15.46:0/1571023748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85300c6b80 msgr2=0x7f85300bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:53.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f853669a640 1 --2- 172.21.15.46:0/1571023748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85300c6b80 0x7f85300bfec0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f85200097c0 tx=0x7f852001c310 comp rx=0 tx=0).stop 2023-10-30T13:36:53.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f853669a640 1 -- 172.21.15.46:0/1571023748 shutdown_connections 2023-10-30T13:36:53.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f853669a640 1 --2- 172.21.15.46:0/1571023748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85300c6f60 0x7f85300c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:53.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f853669a640 1 --2- 172.21.15.46:0/1571023748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85300c6b80 0x7f85300bfec0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:53.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f853669a640 1 -- 172.21.15.46:0/1571023748 >> 172.21.15.46:0/1571023748 conn(0x7f85300c7910 msgr2=0x7f85300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:53.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f853669a640 1 -- 172.21.15.46:0/1571023748 shutdown_connections 2023-10-30T13:36:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f853669a640 1 -- 172.21.15.46:0/1571023748 wait complete. 2023-10-30T13:36:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f853669a640 1 Processor -- start 2023-10-30T13:36:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.412+0000 7f853669a640 1 -- start start 2023-10-30T13:36:53.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f853669a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85300c6b80 0x7f85301f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:53.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f853669a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85300c6f60 0x7f853013d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:53.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f853669a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85300c8430 0x7f85300c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:53.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f853669a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85301f3250 con 0x7f85300c6b80 2023-10-30T13:36:53.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f853669a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85301386f0 con 0x7f85300c6f60 2023-10-30T13:36:53.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f853669a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85301543c0 con 0x7f85300c8430 2023-10-30T13:36:53.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85300c6b80 0x7f85301f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:53.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85300c6b80 0x7f85301f2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57968/0 (socket says 172.21.15.46:57968) 2023-10-30T13:36:53.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852ffff640 1 -- 172.21.15.46:0/1085686299 learned_addr learned my addr 172.21.15.46:0/1085686299 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:53.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f8534c37640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85300c8430 0x7f85300c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:53.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852f7fe640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85300c6f60 0x7f853013d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:53.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852ffff640 1 -- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85300c8430 msgr2=0x7f85300c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:53.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852ffff640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85300c8430 0x7f85300c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:53.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852ffff640 1 -- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85300c6f60 msgr2=0x7f853013d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:53.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852ffff640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85300c6f60 0x7f853013d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:53.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852ffff640 1 -- 172.21.15.46:0/1085686299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8524009420 con 0x7f85300c6b80 2023-10-30T13:36:53.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852f7fe640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85300c6f60 0x7f853013d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:36:53.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852ffff640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85300c6b80 0x7f85301f2d10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8520000c10 tx=0x7f85200190b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852cff9640 1 -- 172.21.15.46:0/1085686299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8520003ab0 con 0x7f85300c6b80 2023-10-30T13:36:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852cff9640 1 -- 172.21.15.46:0/1085686299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8520003c10 con 0x7f85300c6b80 2023-10-30T13:36:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852cff9640 1 -- 172.21.15.46:0/1085686299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85200b5410 con 0x7f85300c6b80 2023-10-30T13:36:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f853669a640 1 -- 172.21.15.46:0/1085686299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8520009420 con 0x7f85300c6b80 2023-10-30T13:36:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f853669a640 1 -- 172.21.15.46:0/1085686299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853013d5c0 con 0x7f85300c6b80 2023-10-30T13:36:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f853669a640 1 -- 172.21.15.46:0/1085686299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853004e170 con 0x7f85300c6b80 2023-10-30T13:36:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.416+0000 7f852cff9640 1 -- 172.21.15.46:0/1085686299 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f85200b2050 con 0x7f85300c6b80 2023-10-30T13:36:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.420+0000 7f852cff9640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f851c0664a0 0x7f851c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.420+0000 7f852cff9640 1 -- 172.21.15.46:0/1085686299 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f852011e620 con 0x7f85300c6b80 2023-10-30T13:36:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.420+0000 7f852f7fe640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f851c0664a0 0x7f851c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.420+0000 7f852f7fe640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f851c0664a0 0x7f851c068960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f85300c1950 tx=0x7f8524008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:53.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.424+0000 7f852cff9640 1 -- 172.21.15.46:0/1085686299 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85200eb8c0 con 0x7f85300c6b80 2023-10-30T13:36:53.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 -- 172.21.15.46:0/1085686299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85300b0d20 con 0x7f85300c6b80 2023-10-30T13:36:53.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f852cff9640 1 -- 172.21.15.46:0/1085686299 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f85200eeee0 con 0x7f85300c6b80 2023-10-30T13:36:53.719 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:36:53.720 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:16.036572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5442":{"gid":5442,"name":"c","rank":2,"incarnation":46,"state":"up:active","state_seq":13,"addr":"172.21.15.46:6836/3955931693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3955931693},{"type":"v1","addr":"172.21.15.46:6836","nonce":3955931693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:53.721 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 71 2023-10-30T13:36:53.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 -- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f851c0664a0 msgr2=0x7f851c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:53.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f851c0664a0 0x7f851c068960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f85300c1950 tx=0x7f8524008040 comp rx=0 tx=0).stop 2023-10-30T13:36:53.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 -- 172.21.15.46:0/1085686299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85300c6b80 msgr2=0x7f85301f2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:53.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85300c6b80 0x7f85301f2d10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8520000c10 tx=0x7f85200190b0 comp rx=0 tx=0).stop 2023-10-30T13:36:53.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 -- 172.21.15.46:0/1085686299 shutdown_connections 2023-10-30T13:36:53.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f851c0664a0 0x7f851c068960 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:53.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85300c8430 0x7f85300c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:53.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85300c6f60 0x7f853013d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:53.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 --2- 172.21.15.46:0/1085686299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85300c6b80 0x7f85301f2d10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:53.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 -- 172.21.15.46:0/1085686299 >> 172.21.15.46:0/1085686299 conn(0x7f85300c7910 msgr2=0x7f85300c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:53.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 -- 172.21.15.46:0/1085686299 shutdown_connections 2023-10-30T13:36:53.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:53.712+0000 7f853669a640 1 -- 172.21.15.46:0/1085686299 wait complete. 2023-10-30T13:36:53.740 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:36:55.741 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:55.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.888+0000 7f3b3359e640 1 Processor -- start 2023-10-30T13:36:55.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3359e640 1 -- start start 2023-10-30T13:36:55.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3359e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3b341538f0 0x7f3b34159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:55.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3359e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b3415cd50 0x7f3b340c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:55.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3359e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3415d300 con 0x7f3b340cefa0 2023-10-30T13:36:55.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3359e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3415d5e0 con 0x7f3b341538f0 2023-10-30T13:36:55.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3359e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3415d8c0 con 0x7f3b3415cd50 2023-10-30T13:36:55.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3259c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3b341538f0 0x7f3b34159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:55.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3259c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3b341538f0 0x7f3b34159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:56560/0 (socket says 172.21.15.46:56560) 2023-10-30T13:36:55.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3259c640 1 -- 172.21.15.46:0/2910227082 learned_addr learned my addr 172.21.15.46:0/2910227082 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:55.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b31d9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b3415cd50 0x7f3b340c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:55.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b32d9d640 1 --1- 172.21.15.46:0/2910227082 >> v1:172.21.15.154:6789/0 conn(0x7f3b340cefa0 0x7f3b340cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:33416/0 (socket says 172.21.15.46:33416) 2023-10-30T13:36:55.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3259c640 1 -- 172.21.15.46:0/2910227082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b3415cd50 msgr2=0x7f3b340c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:55.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3259c640 1 --2- 172.21.15.46:0/2910227082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b3415cd50 0x7f3b340c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:55.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3259c640 1 -- 172.21.15.46:0/2910227082 >> v1:172.21.15.154:6789/0 conn(0x7f3b340cefa0 legacy=0x7f3b340cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:55.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b3259c640 1 -- 172.21.15.46:0/2910227082 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3415dba0 con 0x7f3b341538f0 2023-10-30T13:36:55.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.892+0000 7f3b31d9b640 1 --2- 172.21.15.46:0/2910227082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b3415cd50 0x7f3b340c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:36:55.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.896+0000 7f3b3259c640 1 --2- 172.21.15.46:0/2910227082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b341538f0 0x7f3b34159fb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009870 tx=0x7f3b2401ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4715cdab348d43f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:55.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.896+0000 7f3b3159a640 1 -- 172.21.15.46:0/2910227082 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b240a8a30 con 0x7f3b341538f0 2023-10-30T13:36:55.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.896+0000 7f3b3159a640 1 -- 172.21.15.46:0/2910227082 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b240a8b90 con 0x7f3b341538f0 2023-10-30T13:36:55.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.896+0000 7f3b3159a640 1 -- 172.21.15.46:0/2910227082 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b240b0440 con 0x7f3b341538f0 2023-10-30T13:36:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.896+0000 7f3b3359e640 1 -- 172.21.15.46:0/2910227082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b341538f0 msgr2=0x7f3b34159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.896+0000 7f3b3359e640 1 --2- 172.21.15.46:0/2910227082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b341538f0 0x7f3b34159fb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009870 tx=0x7f3b2401ed30 comp rx=0 tx=0).stop 2023-10-30T13:36:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.896+0000 7f3b3359e640 1 -- 172.21.15.46:0/2910227082 shutdown_connections 2023-10-30T13:36:55.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.896+0000 7f3b3359e640 1 --2- 172.21.15.46:0/2910227082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b3415cd50 0x7f3b340c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:55.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.896+0000 7f3b3359e640 1 --2- 172.21.15.46:0/2910227082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b341538f0 0x7f3b34159fb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:55.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.896+0000 7f3b3359e640 1 -- 172.21.15.46:0/2910227082 >> 172.21.15.46:0/2910227082 conn(0x7f3b3406a640 msgr2=0x7f3b34065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:55.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.896+0000 7f3b3359e640 1 -- 172.21.15.46:0/2910227082 shutdown_connections 2023-10-30T13:36:55.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.896+0000 7f3b3359e640 1 -- 172.21.15.46:0/2910227082 wait complete. 2023-10-30T13:36:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b3359e640 1 Processor -- start 2023-10-30T13:36:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b3359e640 1 -- start start 2023-10-30T13:36:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b3359e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b340cefa0 0x7f3b340c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b3359e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b341538f0 0x7f3b3415ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b3359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3415cd50 0x7f3b3415b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b3359e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b34146120 con 0x7f3b3415cd50 2023-10-30T13:36:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b3359e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b34146540 con 0x7f3b341538f0 2023-10-30T13:36:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b3359e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b34145a60 con 0x7f3b340cefa0 2023-10-30T13:36:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b32d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3415cd50 0x7f3b3415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b32d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3415cd50 0x7f3b3415b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57972/0 (socket says 172.21.15.46:57972) 2023-10-30T13:36:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b32d9d640 1 -- 172.21.15.46:0/4273165623 learned_addr learned my addr 172.21.15.46:0/4273165623 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b3259c640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b340cefa0 0x7f3b340c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:55.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b31d9b640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b341538f0 0x7f3b3415ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b32d9d640 1 -- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b340cefa0 msgr2=0x7f3b340c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b32d9d640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b340cefa0 0x7f3b340c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b32d9d640 1 -- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b341538f0 msgr2=0x7f3b3415ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b32d9d640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b341538f0 0x7f3b3415ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:55.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b32d9d640 1 -- 172.21.15.46:0/4273165623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b28009420 con 0x7f3b3415cd50 2023-10-30T13:36:55.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b32d9d640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3415cd50 0x7f3b3415b340 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c00c7a0 tx=0x7f3b2c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b22ffd640 1 -- 172.21.15.46:0/4273165623 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c00ce50 con 0x7f3b3415cd50 2023-10-30T13:36:55.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b22ffd640 1 -- 172.21.15.46:0/4273165623 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c00a5e0 con 0x7f3b3415cd50 2023-10-30T13:36:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b3359e640 1 -- 172.21.15.46:0/4273165623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b240094d0 con 0x7f3b3415cd50 2023-10-30T13:36:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b22ffd640 1 -- 172.21.15.46:0/4273165623 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c00a950 con 0x7f3b3415cd50 2023-10-30T13:36:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.900+0000 7f3b3359e640 1 -- 172.21.15.46:0/4273165623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3415c390 con 0x7f3b3415cd50 2023-10-30T13:36:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.904+0000 7f3b3359e640 1 -- 172.21.15.46:0/4273165623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b34146260 con 0x7f3b3415cd50 2023-10-30T13:36:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.904+0000 7f3b22ffd640 1 -- 172.21.15.46:0/4273165623 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3b2c00aab0 con 0x7f3b3415cd50 2023-10-30T13:36:55.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.904+0000 7f3b22ffd640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3b140664a0 0x7f3b14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:55.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.904+0000 7f3b22ffd640 1 -- 172.21.15.46:0/4273165623 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f3b2c0837e0 con 0x7f3b3415cd50 2023-10-30T13:36:55.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.904+0000 7f3b3259c640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3b140664a0 0x7f3b14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:55.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.904+0000 7f3b3259c640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3b140664a0 0x7f3b14068960 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3b240b2000 tx=0x7f3b24009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:55.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:55.908+0000 7f3b22ffd640 1 -- 172.21.15.46:0/4273165623 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3b34146260 con 0x7f3b3415cd50 2023-10-30T13:36:56.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 -- 172.21.15.46:0/4273165623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b34146260 con 0x7f3b3415cd50 2023-10-30T13:36:56.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b22ffd640 1 -- 172.21.15.46:0/4273165623 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f3b34146260 con 0x7f3b3415cd50 2023-10-30T13:36:56.106 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:36:56.106 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:53.920796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:56.106 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 72 2023-10-30T13:36:56.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 -- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3b140664a0 msgr2=0x7f3b14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3b140664a0 0x7f3b14068960 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3b240b2000 tx=0x7f3b24009040 comp rx=0 tx=0).stop 2023-10-30T13:36:56.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 -- 172.21.15.46:0/4273165623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3415cd50 msgr2=0x7f3b3415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3415cd50 0x7f3b3415b340 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c00c7a0 tx=0x7f3b2c00cc60 comp rx=0 tx=0).stop 2023-10-30T13:36:56.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 -- 172.21.15.46:0/4273165623 shutdown_connections 2023-10-30T13:36:56.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3b140664a0 0x7f3b14068960 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b3415cd50 0x7f3b3415b340 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b341538f0 0x7f3b3415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 --2- 172.21.15.46:0/4273165623 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b340cefa0 0x7f3b340c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 -- 172.21.15.46:0/4273165623 >> 172.21.15.46:0/4273165623 conn(0x7f3b3406a640 msgr2=0x7f3b34065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:56.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 -- 172.21.15.46:0/4273165623 shutdown_connections 2023-10-30T13:36:56.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.100+0000 7f3b3359e640 1 -- 172.21.15.46:0/4273165623 wait complete. 2023-10-30T13:36:56.127 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:36:56.127 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:36:56.128 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:56.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.264+0000 7faf7efd3640 1 Processor -- start 2023-10-30T13:36:56.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.264+0000 7faf7efd3640 1 -- start start 2023-10-30T13:36:56.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.264+0000 7faf7efd3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faf7806f7a0 0x7faf7806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.264+0000 7faf7efd3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf780c6bd0 0x7faf78069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.264+0000 7faf7efd3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7806a3d0 con 0x7faf78071110 2023-10-30T13:36:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf7efd3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf780700c0 con 0x7faf780c6bd0 2023-10-30T13:36:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf7efd3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf78066cf0 con 0x7faf7806f7a0 2023-10-30T13:36:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf77fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf780c6bd0 0x7faf78069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf77fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf780c6bd0 0x7faf78069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57976/0 (socket says 172.21.15.46:57976) 2023-10-30T13:36:56.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf77fff640 1 -- 172.21.15.46:0/1333207478 learned_addr learned my addr 172.21.15.46:0/1333207478 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:56.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf7cd6f640 1 --2- 172.21.15.46:0/1333207478 >> v2:172.21.15.154:3300/0 conn(0x7faf7806f7a0 0x7faf7806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf77fff640 1 -- 172.21.15.46:0/1333207478 >> v2:172.21.15.154:3300/0 conn(0x7faf7806f7a0 msgr2=0x7faf7806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:56.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf77fff640 1 --2- 172.21.15.46:0/1333207478 >> v2:172.21.15.154:3300/0 conn(0x7faf7806f7a0 0x7faf7806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf77fff640 1 -- 172.21.15.46:0/1333207478 >> v1:172.21.15.154:6789/0 conn(0x7faf78071110 legacy=0x7faf78158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf77fff640 1 -- 172.21.15.46:0/1333207478 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf780c70d0 con 0x7faf780c6bd0 2023-10-30T13:36:56.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf77fff640 1 --2- 172.21.15.46:0/1333207478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf780c6bd0 0x7faf78069e90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7faf7806c830 tx=0x7faf6401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d9cdf94d5e4deb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:56.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf777fe640 1 -- 172.21.15.46:0/1333207478 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf640a89f0 con 0x7faf780c6bd0 2023-10-30T13:36:56.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf777fe640 1 -- 172.21.15.46:0/1333207478 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf640a8b50 con 0x7faf780c6bd0 2023-10-30T13:36:56.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.268+0000 7faf777fe640 1 -- 172.21.15.46:0/1333207478 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf640b0440 con 0x7faf780c6bd0 2023-10-30T13:36:56.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 -- 172.21.15.46:0/1333207478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf780c6bd0 msgr2=0x7faf78069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 --2- 172.21.15.46:0/1333207478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf780c6bd0 0x7faf78069e90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7faf7806c830 tx=0x7faf6401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 -- 172.21.15.46:0/1333207478 shutdown_connections 2023-10-30T13:36:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 --2- 172.21.15.46:0/1333207478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf780c6bd0 0x7faf78069e90 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 --2- 172.21.15.46:0/1333207478 >> v2:172.21.15.154:3300/0 conn(0x7faf7806f7a0 0x7faf7806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 -- 172.21.15.46:0/1333207478 >> 172.21.15.46:0/1333207478 conn(0x7faf780c7910 msgr2=0x7faf780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:56.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 -- 172.21.15.46:0/1333207478 shutdown_connections 2023-10-30T13:36:56.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 -- 172.21.15.46:0/1333207478 wait complete. 2023-10-30T13:36:56.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 Processor -- start 2023-10-30T13:36:56.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 -- start start 2023-10-30T13:36:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7806f7a0 0x7faf780afa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf78071110 0x7faf780c0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf780c6bd0 0x7faf780c0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf780bfec0 con 0x7faf78071110 2023-10-30T13:36:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf780c01a0 con 0x7faf7806f7a0 2023-10-30T13:36:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7efd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf780ae460 con 0x7faf780c6bd0 2023-10-30T13:36:56.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf77fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf78071110 0x7faf780c0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf77fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf78071110 0x7faf780c0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57990/0 (socket says 172.21.15.46:57990) 2023-10-30T13:36:56.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf77fff640 1 -- 172.21.15.46:0/2981873745 learned_addr learned my addr 172.21.15.46:0/2981873745 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:56.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7d570640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf780c6bd0 0x7faf780c0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf7cd6f640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7806f7a0 0x7faf780afa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.272+0000 7faf77fff640 1 -- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf780c6bd0 msgr2=0x7faf780c0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf77fff640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf780c6bd0 0x7faf780c0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf77fff640 1 -- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7806f7a0 msgr2=0x7faf780afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf77fff640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7806f7a0 0x7faf780afa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf77fff640 1 -- 172.21.15.46:0/2981873745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf640094d0 con 0x7faf78071110 2023-10-30T13:36:56.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf7d570640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf780c6bd0 0x7faf780c0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:56.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf77fff640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf78071110 0x7faf780c0630 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7faf6401b900 tx=0x7faf6401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:56.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf757fa640 1 -- 172.21.15.46:0/2981873745 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf640b1e90 con 0x7faf78071110 2023-10-30T13:36:56.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf757fa640 1 -- 172.21.15.46:0/2981873745 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf640bb440 con 0x7faf78071110 2023-10-30T13:36:56.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf7efd3640 1 -- 172.21.15.46:0/2981873745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf780ae740 con 0x7faf78071110 2023-10-30T13:36:56.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf757fa640 1 -- 172.21.15.46:0/2981873745 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf640bb7b0 con 0x7faf78071110 2023-10-30T13:36:56.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf7efd3640 1 -- 172.21.15.46:0/2981873745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf780aed00 con 0x7faf78071110 2023-10-30T13:36:56.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf7efd3640 1 -- 172.21.15.46:0/2981873745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf780c0310 con 0x7faf78071110 2023-10-30T13:36:56.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf757fa640 1 -- 172.21.15.46:0/2981873745 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faf640b7040 con 0x7faf78071110 2023-10-30T13:36:56.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf757fa640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faf5c0664a0 0x7faf5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:56.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf757fa640 1 -- 172.21.15.46:0/2981873745 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7faf641220d0 con 0x7faf78071110 2023-10-30T13:36:56.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.276+0000 7faf7cd6f640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faf5c0664a0 0x7faf5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:56.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.280+0000 7faf7cd6f640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faf5c0664a0 0x7faf5c068960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00b930 tx=0x7faf6c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:56.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.280+0000 7faf757fa640 1 -- 172.21.15.46:0/2981873745 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faf640ef2a0 con 0x7faf78071110 2023-10-30T13:36:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.468+0000 7faf7efd3640 1 -- 172.21.15.46:0/2981873745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf780aeef0 con 0x7faf78071110 2023-10-30T13:36:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf757fa640 1 -- 172.21.15.46:0/2981873745 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5743 (secure 0 0 0) 0x7faf640f28c0 con 0x7faf78071110 2023-10-30T13:36:56.477 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:36:56.477 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:53.920796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:56.477 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 72 2023-10-30T13:36:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf7efd3640 1 -- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faf5c0664a0 msgr2=0x7faf5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf7efd3640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faf5c0664a0 0x7faf5c068960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00b930 tx=0x7faf6c008040 comp rx=0 tx=0).stop 2023-10-30T13:36:56.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf7efd3640 1 -- 172.21.15.46:0/2981873745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf78071110 msgr2=0x7faf780c0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:56.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf7efd3640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf78071110 0x7faf780c0630 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7faf6401b900 tx=0x7faf6401ba90 comp rx=0 tx=0).stop 2023-10-30T13:36:56.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf7efd3640 1 -- 172.21.15.46:0/2981873745 shutdown_connections 2023-10-30T13:36:56.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf7efd3640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faf5c0664a0 0x7faf5c068960 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf7efd3640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf780c6bd0 0x7faf780c0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf7efd3640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf78071110 0x7faf780c0630 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf7efd3640 1 --2- 172.21.15.46:0/2981873745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7806f7a0 0x7faf780afa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf7efd3640 1 -- 172.21.15.46:0/2981873745 >> 172.21.15.46:0/2981873745 conn(0x7faf780c7910 msgr2=0x7faf780c9040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf7efd3640 1 -- 172.21.15.46:0/2981873745 shutdown_connections 2023-10-30T13:36:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:56.472+0000 7faf7efd3640 1 -- 172.21.15.46:0/2981873745 wait complete. 2023-10-30T13:36:56.498 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 72, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 72, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5574": { "addr": "172.21.15.46:6837/126953235", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 126953235, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 126953235, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5574, "incarnation": 55, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_5721": { "addr": "172.21.15.46:6833/4044310929", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4044310929, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4044310929, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5721, "incarnation": 61, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 24 }, "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:36:53.920796+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5721, "mds_1": 5574, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:36:58.499 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:58.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd12b80640 1 Processor -- start 2023-10-30T13:36:58.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd12b80640 1 -- start start 2023-10-30T13:36:58.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd12b80640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd0c0c6b80 0x7fbd0c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd12b80640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd0c0c6f60 0x7fbd0c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd12b80640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0c0c0a50 con 0x7fbd0c0c8430 2023-10-30T13:36:58.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd12b80640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd0c06f6e0 con 0x7fbd0c0c6b80 2023-10-30T13:36:58.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd12b80640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c06f9c0 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd1091c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd0c0c6b80 0x7fbd0c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd1091c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd0c0c6b80 0x7fbd0c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55570/0 (socket says 172.21.15.46:55570) 2023-10-30T13:36:58.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd1091c640 1 -- 172.21.15.46:0/2104319686 learned_addr learned my addr 172.21.15.46:0/2104319686 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:58.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd1111d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbd0c0c8430 0x7fbd0c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56022/0 (socket says 172.21.15.46:56022) 2023-10-30T13:36:58.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd0bfff640 1 --2- 172.21.15.46:0/2104319686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd0c0c6f60 0x7fbd0c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd1091c640 1 -- 172.21.15.46:0/2104319686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd0c0c6f60 msgr2=0x7fbd0c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:58.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd1091c640 1 --2- 172.21.15.46:0/2104319686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd0c0c6f60 0x7fbd0c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd1091c640 1 -- 172.21.15.46:0/2104319686 >> v1:172.21.15.46:6789/0 conn(0x7fbd0c0c8430 legacy=0x7fbd0c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd1091c640 1 -- 172.21.15.46:0/2104319686 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c06fca0 con 0x7fbd0c0c6b80 2023-10-30T13:36:58.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd0bfff640 1 --2- 172.21.15.46:0/2104319686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd0c0c6f60 0x7fbd0c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:58.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.648+0000 7fbd0b7fe640 1 -- 172.21.15.46:0/2104319686 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199898396 0 0) 0x7fbd0c0c0a50 con 0x7fbd0c0c8430 2023-10-30T13:36:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.652+0000 7fbd1091c640 1 --2- 172.21.15.46:0/2104319686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd0c0c6b80 0x7fbd0c0bfec0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000bb10 tx=0x7fbd0001eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc8943a07d4c2099 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.652+0000 7fbd0b7fe640 1 -- 172.21.15.46:0/2104319686 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd000a7b60 con 0x7fbd0c0c6b80 2023-10-30T13:36:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.652+0000 7fbd0b7fe640 1 -- 172.21.15.46:0/2104319686 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd000a7cc0 con 0x7fbd0c0c6b80 2023-10-30T13:36:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.652+0000 7fbd0b7fe640 1 -- 172.21.15.46:0/2104319686 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd000af660 con 0x7fbd0c0c6b80 2023-10-30T13:36:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.652+0000 7fbd12b80640 1 -- 172.21.15.46:0/2104319686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd0c0c6b80 msgr2=0x7fbd0c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.652+0000 7fbd12b80640 1 --2- 172.21.15.46:0/2104319686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd0c0c6b80 0x7fbd0c0bfec0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000bb10 tx=0x7fbd0001eb80 comp rx=0 tx=0).stop 2023-10-30T13:36:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.652+0000 7fbd12b80640 1 -- 172.21.15.46:0/2104319686 shutdown_connections 2023-10-30T13:36:58.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.652+0000 7fbd12b80640 1 --2- 172.21.15.46:0/2104319686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd0c0c6f60 0x7fbd0c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.652+0000 7fbd12b80640 1 --2- 172.21.15.46:0/2104319686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd0c0c6b80 0x7fbd0c0bfec0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.652+0000 7fbd12b80640 1 -- 172.21.15.46:0/2104319686 >> 172.21.15.46:0/2104319686 conn(0x7fbd0c0c7910 msgr2=0x7fbd0c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:58.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.652+0000 7fbd12b80640 1 -- 172.21.15.46:0/2104319686 shutdown_connections 2023-10-30T13:36:58.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.652+0000 7fbd12b80640 1 -- 172.21.15.46:0/2104319686 wait complete. 2023-10-30T13:36:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd12b80640 1 Processor -- start 2023-10-30T13:36:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd12b80640 1 -- start start 2023-10-30T13:36:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd12b80640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd0c0c6b80 0x7fbd0c155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd12b80640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd0c0c6f60 0x7fbd0c1f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd12b80640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd0c0c8430 0x7fbd0c1f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd12b80640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c138240 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd12b80640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c0c8c10 con 0x7fbd0c0c8430 2023-10-30T13:36:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd12b80640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c0c12a0 con 0x7fbd0c0c6b80 2023-10-30T13:36:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd0bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd0c0c6f60 0x7fbd0c1f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd0bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd0c0c6f60 0x7fbd0c1f30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55572/0 (socket says 172.21.15.46:55572) 2023-10-30T13:36:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd0bfff640 1 -- 172.21.15.46:0/111978178 learned_addr learned my addr 172.21.15.46:0/111978178 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:36:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd1091c640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd0c0c6b80 0x7fbd0c155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd1111d640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd0c0c8430 0x7fbd0c1f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd0bfff640 1 -- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd0c0c6b80 msgr2=0x7fbd0c155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd0bfff640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd0c0c6b80 0x7fbd0c155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd0bfff640 1 -- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd0c0c8430 msgr2=0x7fbd0c1f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd0bfff640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd0c0c8430 0x7fbd0c1f3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd0bfff640 1 -- 172.21.15.46:0/111978178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0000b770 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd1111d640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd0c0c8430 0x7fbd0c1f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd0bfff640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd0c0c6f60 0x7fbd0c1f30c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800e810 tx=0x7fbcf800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd097fa640 1 -- 172.21.15.46:0/111978178 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf800c8d0 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd097fa640 1 -- 172.21.15.46:0/111978178 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf800ca30 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd097fa640 1 -- 172.21.15.46:0/111978178 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf800cdc0 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd12b80640 1 -- 172.21.15.46:0/111978178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0c0c8e60 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.656+0000 7fbd12b80640 1 -- 172.21.15.46:0/111978178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0c0c9420 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.660+0000 7fbd12b80640 1 -- 172.21.15.46:0/111978178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0c04e170 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.660+0000 7fbd097fa640 1 -- 172.21.15.46:0/111978178 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbcf8013410 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.660+0000 7fbd097fa640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbcf00664d0 0x7fbcf0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.660+0000 7fbd097fa640 1 -- 172.21.15.46:0/111978178 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fbcf8083fb0 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.660+0000 7fbd1091c640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbcf00664d0 0x7fbcf0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.660+0000 7fbd1091c640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbcf00664d0 0x7fbcf0068990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbd000b1000 tx=0x7fbd000a9070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.664+0000 7fbd097fa640 1 -- 172.21.15.46:0/111978178 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbcf8051100 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.852+0000 7fbd12b80640 1 -- 172.21.15.46:0/111978178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd0c0c16c0 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.852+0000 7fbd097fa640 1 -- 172.21.15.46:0/111978178 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5743 (secure 0 0 0) 0x7fbcf8054720 con 0x7fbd0c0c6f60 2023-10-30T13:36:58.859 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:36:58.859 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:53.920796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 72 2023-10-30T13:36:58.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.852+0000 7fbd12b80640 1 -- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbcf00664d0 msgr2=0x7fbcf0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.852+0000 7fbd12b80640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbcf00664d0 0x7fbcf0068990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbd000b1000 tx=0x7fbd000a9070 comp rx=0 tx=0).stop 2023-10-30T13:36:58.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.852+0000 7fbd12b80640 1 -- 172.21.15.46:0/111978178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd0c0c6f60 msgr2=0x7fbd0c1f30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:58.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.852+0000 7fbd12b80640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd0c0c6f60 0x7fbd0c1f30c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800e810 tx=0x7fbcf800ecd0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.852+0000 7fbd12b80640 1 -- 172.21.15.46:0/111978178 shutdown_connections 2023-10-30T13:36:58.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.852+0000 7fbd12b80640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbcf00664d0 0x7fbcf0068990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.852+0000 7fbd12b80640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd0c0c8430 0x7fbd0c1f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.852+0000 7fbd12b80640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd0c0c6f60 0x7fbd0c1f30c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.852+0000 7fbd12b80640 1 --2- 172.21.15.46:0/111978178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd0c0c6b80 0x7fbd0c155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.852+0000 7fbd12b80640 1 -- 172.21.15.46:0/111978178 >> 172.21.15.46:0/111978178 conn(0x7fbd0c0c7910 msgr2=0x7fbd0c0c5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.856+0000 7fbd12b80640 1 -- 172.21.15.46:0/111978178 shutdown_connections 2023-10-30T13:36:58.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:36:58.856+0000 7fbd12b80640 1 -- 172.21.15.46:0/111978178 wait complete. 2023-10-30T13:36:58.880 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:00.881 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:01.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8b6ed6640 1 Processor -- start 2023-10-30T13:37:01.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8b6ed6640 1 -- start start 2023-10-30T13:37:01.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8b6ed6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa8b00c6bd0 0x7fa8b006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:01.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8b6ed6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b0066cf0 con 0x7fa8b006f7a0 2023-10-30T13:37:01.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8b6ed6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00c70d0 con 0x7fa8b0071110 2023-10-30T13:37:01.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8b6ed6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b00700c0 con 0x7fa8b00c6bd0 2023-10-30T13:37:01.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8b4c72640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa8b006f7a0 0x7fa8b0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56036/0 (socket says 172.21.15.46:56036) 2023-10-30T13:37:01.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8b4c72640 1 -- 172.21.15.46:0/3715965613 learned_addr learned my addr 172.21.15.46:0/3715965613 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:37:01.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8affff640 1 --2- 172.21.15.46:0/3715965613 >> v2:172.21.15.154:3300/0 conn(0x7fa8b00c6bd0 0x7fa8b006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504222648 0 0) 0x7fa8b0066cf0 con 0x7fa8b006f7a0 2023-10-30T13:37:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a0003530 con 0x7fa8b006f7a0 2023-10-30T13:37:01.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830288080 0 0) 0x7fa8b00c70d0 con 0x7fa8b0071110 2023-10-30T13:37:01.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b0066cf0 con 0x7fa8b0071110 2023-10-30T13:37:01.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1467661422 0 0) 0x7fa8a0003530 con 0x7fa8b006f7a0 2023-10-30T13:37:01.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 >> v2:172.21.15.154:3300/0 conn(0x7fa8b00c6bd0 msgr2=0x7fa8b006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:01.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 --2- 172.21.15.46:0/3715965613 >> v2:172.21.15.154:3300/0 conn(0x7fa8b00c6bd0 0x7fa8b006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 >> v1:172.21.15.154:6789/0 conn(0x7fa8b0071110 legacy=0x7fa8b0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b00c7390 con 0x7fa8b006f7a0 2023-10-30T13:37:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fa8a4001680 con 0x7fa8b006f7a0 2023-10-30T13:37:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 >> v1:172.21.15.46:6789/0 conn(0x7fa8b006f7a0 legacy=0x7fa8b0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 --2- 172.21.15.46:0/3715965613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a0003b60 0x7fa8a0003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 --2- 172.21.15.46:0/3715965613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8a00047d0 0x7fa8a0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 --2- 172.21.15.46:0/3715965613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a0005190 0x7fa8a000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b00700c0 con 0x7fa8a00047d0 2023-10-30T13:37:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a000bfb0 con 0x7fa8a0005190 2023-10-30T13:37:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a000c170 con 0x7fa8a0003b60 2023-10-30T13:37:01.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8affff640 1 --2- 172.21.15.46:0/3715965613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8a00047d0 0x7fa8a0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:01.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8b4c72640 1 --2- 172.21.15.46:0/3715965613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a0003b60 0x7fa8a0003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:01.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8b5473640 1 --2- 172.21.15.46:0/3715965613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a0005190 0x7fa8a000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:01.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8affff640 1 -- 172.21.15.46:0/3715965613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a0003b60 msgr2=0x7fa8a0003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:01.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8affff640 1 --2- 172.21.15.46:0/3715965613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a0003b60 0x7fa8a0003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:01.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8affff640 1 -- 172.21.15.46:0/3715965613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a0005190 msgr2=0x7fa8a000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:01.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8affff640 1 --2- 172.21.15.46:0/3715965613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a0005190 0x7fa8a000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:01.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8affff640 1 -- 172.21.15.46:0/3715965613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a0003530 con 0x7fa8a00047d0 2023-10-30T13:37:01.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.032+0000 7fa8b5473640 1 -- 172.21.15.46:0/3715965613 reap_dead start 2023-10-30T13:37:01.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.036+0000 7fa8affff640 1 --2- 172.21.15.46:0/3715965613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8a00047d0 0x7fa8a0004c50 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa89c009ab0 tx=0x7fa89c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25eb86ba7c070b9b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:01.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.036+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c0a5620 con 0x7fa8a00047d0 2023-10-30T13:37:01.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.036+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89c0a5780 con 0x7fa8a00047d0 2023-10-30T13:37:01.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.036+0000 7fa8af7fe640 1 -- 172.21.15.46:0/3715965613 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c0a5b30 con 0x7fa8a00047d0 2023-10-30T13:37:01.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.036+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/3715965613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8a00047d0 msgr2=0x7fa8a0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:01.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.036+0000 7fa8b6ed6640 1 --2- 172.21.15.46:0/3715965613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8a00047d0 0x7fa8a0004c50 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa89c009ab0 tx=0x7fa89c01ca30 comp rx=0 tx=0).stop 2023-10-30T13:37:01.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.036+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/3715965613 shutdown_connections 2023-10-30T13:37:01.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.036+0000 7fa8b6ed6640 1 --2- 172.21.15.46:0/3715965613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8a00047d0 0x7fa8a0004c50 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:01.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.036+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/3715965613 >> 172.21.15.46:0/3715965613 conn(0x7fa8b00c7910 msgr2=0x7fa8b00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:01.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.036+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/3715965613 shutdown_connections 2023-10-30T13:37:01.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.036+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/3715965613 wait complete. 2023-10-30T13:37:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b6ed6640 1 Processor -- start 2023-10-30T13:37:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b6ed6640 1 -- start start 2023-10-30T13:37:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b6ed6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a00047d0 0x7fa8b00c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b6ed6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8b014aa90 0x7fa8b01ef4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b6ed6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8b00c1140 0x7fa8b0146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:01.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b6ed6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b0150020 con 0x7fa8b00c1140 2023-10-30T13:37:01.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b6ed6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b014af80 con 0x7fa8b014aa90 2023-10-30T13:37:01.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b6ed6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b014b260 con 0x7fa8a00047d0 2023-10-30T13:37:01.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b5473640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8b00c1140 0x7fa8b0146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:01.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b4c72640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a00047d0 0x7fa8b00c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b5473640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8b00c1140 0x7fa8b0146690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55596/0 (socket says 172.21.15.46:55596) 2023-10-30T13:37:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b5473640 1 -- 172.21.15.46:0/2072043853 learned_addr learned my addr 172.21.15.46:0/2072043853 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8affff640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8b014aa90 0x7fa8b01ef4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b5473640 1 -- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a00047d0 msgr2=0x7fa8b00c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b5473640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a00047d0 0x7fa8b00c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b5473640 1 -- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8b014aa90 msgr2=0x7fa8b01ef4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:01.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b5473640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8b014aa90 0x7fa8b01ef4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:01.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b5473640 1 -- 172.21.15.46:0/2072043853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b014b540 con 0x7fa8b00c1140 2023-10-30T13:37:01.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b5473640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8b00c1140 0x7fa8b0146690 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa8b006c7d0 tx=0x7fa8a800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:01.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8ad7fa640 1 -- 172.21.15.46:0/2072043853 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8006b60 con 0x7fa8b00c1140 2023-10-30T13:37:01.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8ad7fa640 1 -- 172.21.15.46:0/2072043853 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a8006ce0 con 0x7fa8b00c1140 2023-10-30T13:37:01.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8ad7fa640 1 -- 172.21.15.46:0/2072043853 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8003bc0 con 0x7fa8b00c1140 2023-10-30T13:37:01.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.040+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/2072043853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b014b820 con 0x7fa8b00c1140 2023-10-30T13:37:01.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.044+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/2072043853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b01eec80 con 0x7fa8b00c1140 2023-10-30T13:37:01.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.044+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/2072043853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b0070610 con 0x7fa8b00c1140 2023-10-30T13:37:01.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.044+0000 7fa8ad7fa640 1 -- 172.21.15.46:0/2072043853 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa8a8003d20 con 0x7fa8b00c1140 2023-10-30T13:37:01.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.044+0000 7fa8ad7fa640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8940664d0 0x7fa894068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:01.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.044+0000 7fa8ad7fa640 1 -- 172.21.15.46:0/2072043853 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fa8a80839f0 con 0x7fa8b00c1140 2023-10-30T13:37:01.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.044+0000 7fa8b4c72640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8940664d0 0x7fa894068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:01.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.048+0000 7fa8b4c72640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8940664d0 0x7fa894068990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa8a40017b0 tx=0x7fa8a4001480 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:01.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.048+0000 7fa8ad7fa640 1 -- 172.21.15.46:0/2072043853 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa8b0070610 con 0x7fa8b00c1140 2023-10-30T13:37:01.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.240+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/2072043853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8b0070610 con 0x7fa8b00c1140 2023-10-30T13:37:01.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.240+0000 7fa8ad7fa640 1 -- 172.21.15.46:0/2072043853 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5745 (secure 0 0 0) 0x7fa8b0070610 con 0x7fa8b00c1140 2023-10-30T13:37:01.248 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:01.249 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:59.650067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:resolve","state_seq":20,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:01.249 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 73 2023-10-30T13:37:01.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.244+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8940664d0 msgr2=0x7fa894068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.244+0000 7fa8b6ed6640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8940664d0 0x7fa894068990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa8a40017b0 tx=0x7fa8a4001480 comp rx=0 tx=0).stop 2023-10-30T13:37:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.244+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/2072043853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8b00c1140 msgr2=0x7fa8b0146690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.244+0000 7fa8b6ed6640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8b00c1140 0x7fa8b0146690 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa8b006c7d0 tx=0x7fa8a800eb90 comp rx=0 tx=0).stop 2023-10-30T13:37:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.244+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/2072043853 shutdown_connections 2023-10-30T13:37:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.244+0000 7fa8b6ed6640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8940664d0 0x7fa894068990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.244+0000 7fa8b6ed6640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8b00c1140 0x7fa8b0146690 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.244+0000 7fa8b6ed6640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8b014aa90 0x7fa8b01ef4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.244+0000 7fa8b6ed6640 1 --2- 172.21.15.46:0/2072043853 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a00047d0 0x7fa8b00c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.244+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/2072043853 >> 172.21.15.46:0/2072043853 conn(0x7fa8b00c7910 msgr2=0x7fa8b00c2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.244+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/2072043853 shutdown_connections 2023-10-30T13:37:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:01.244+0000 7fa8b6ed6640 1 -- 172.21.15.46:0/2072043853 wait complete. 2023-10-30T13:37:01.270 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:03.271 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:03.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136bd3e640 1 Processor -- start 2023-10-30T13:37:03.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136bd3e640 1 -- start start 2023-10-30T13:37:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136bd3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f136415d6f0 0x7f136415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:03.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136bd3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13640cf160 0x7f13640cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:03.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136bd3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13640cb800 con 0x7f13641538a0 2023-10-30T13:37:03.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136bd3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13640cb330 con 0x7f136415d6f0 2023-10-30T13:37:03.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136bd3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13640cad70 con 0x7f13640cf160 2023-10-30T13:37:03.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136a2db640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13640cf160 0x7f13640cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:03.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136a2db640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13640cf160 0x7f13640cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:60646/0 (socket says 172.21.15.46:60646) 2023-10-30T13:37:03.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f13692d9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f136415d6f0 0x7f136415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:03.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136a2db640 1 -- 172.21.15.46:0/2525812036 learned_addr learned my addr 172.21.15.46:0/2525812036 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:03.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136a2db640 1 -- 172.21.15.46:0/2525812036 >> v2:172.21.15.154:3300/0 conn(0x7f136415d6f0 msgr2=0x7f136415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:03.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136a2db640 1 --2- 172.21.15.46:0/2525812036 >> v2:172.21.15.154:3300/0 conn(0x7f136415d6f0 0x7f136415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:03.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136a2db640 1 -- 172.21.15.46:0/2525812036 >> v1:172.21.15.154:6789/0 conn(0x7f13641538a0 legacy=0x7f13640ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:03.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.420+0000 7f136a2db640 1 -- 172.21.15.46:0/2525812036 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13640cb050 con 0x7f13640cf160 2023-10-30T13:37:03.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.424+0000 7f136a2db640 1 --2- 172.21.15.46:0/2525812036 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13640cf160 0x7f13640cf590 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f135c009e50 tx=0x7f135c01f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a05cd298da9b9b43 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:03.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.424+0000 7f1368ad8640 1 -- 172.21.15.46:0/2525812036 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c0af400 con 0x7f13640cf160 2023-10-30T13:37:03.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.424+0000 7f1368ad8640 1 -- 172.21.15.46:0/2525812036 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135c0ab050 con 0x7f13640cf160 2023-10-30T13:37:03.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.424+0000 7f1368ad8640 1 -- 172.21.15.46:0/2525812036 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c0af780 con 0x7f13640cf160 2023-10-30T13:37:03.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.424+0000 7f136bd3e640 1 -- 172.21.15.46:0/2525812036 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13640cf160 msgr2=0x7f13640cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:03.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.424+0000 7f136bd3e640 1 --2- 172.21.15.46:0/2525812036 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13640cf160 0x7f13640cf590 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f135c009e50 tx=0x7f135c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:37:03.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.424+0000 7f136bd3e640 1 -- 172.21.15.46:0/2525812036 shutdown_connections 2023-10-30T13:37:03.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.424+0000 7f136bd3e640 1 --2- 172.21.15.46:0/2525812036 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13640cf160 0x7f13640cf590 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:03.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.424+0000 7f136bd3e640 1 --2- 172.21.15.46:0/2525812036 >> v2:172.21.15.154:3300/0 conn(0x7f136415d6f0 0x7f136415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:03.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.424+0000 7f136bd3e640 1 -- 172.21.15.46:0/2525812036 >> 172.21.15.46:0/2525812036 conn(0x7f136406bfe0 msgr2=0x7f136406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:03.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.424+0000 7f136bd3e640 1 -- 172.21.15.46:0/2525812036 shutdown_connections 2023-10-30T13:37:03.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f136bd3e640 1 -- 172.21.15.46:0/2525812036 wait complete. 2023-10-30T13:37:03.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f136bd3e640 1 Processor -- start 2023-10-30T13:37:03.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f136bd3e640 1 -- start start 2023-10-30T13:37:03.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f136bd3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13640cf160 0x7f136414e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:03.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f136bd3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13641538a0 0x7f13640b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:03.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f136bd3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f136415d6f0 0x7f13640cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:03.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f136bd3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13641efd70 con 0x7f13640cf160 2023-10-30T13:37:03.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f136bd3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13641f0260 con 0x7f13641538a0 2023-10-30T13:37:03.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f136bd3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13640ce7b0 con 0x7f136415d6f0 2023-10-30T13:37:03.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f1369ada640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13640cf160 0x7f136414e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:03.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f1369ada640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13640cf160 0x7f136414e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55608/0 (socket says 172.21.15.46:55608) 2023-10-30T13:37:03.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f1369ada640 1 -- 172.21.15.46:0/1524326089 learned_addr learned my addr 172.21.15.46:0/1524326089 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:03.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f13692d9640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13641538a0 0x7f13640b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:03.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f136a2db640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f136415d6f0 0x7f13640cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:03.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f1369ada640 1 -- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f136415d6f0 msgr2=0x7f13640cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:03.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f1369ada640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f136415d6f0 0x7f13640cd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:03.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f1369ada640 1 -- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13641538a0 msgr2=0x7f13640b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:03.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f1369ada640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13641538a0 0x7f13640b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:03.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f1369ada640 1 -- 172.21.15.46:0/1524326089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c009b30 con 0x7f13640cf160 2023-10-30T13:37:03.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f1369ada640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13640cf160 0x7f136414e220 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f136000c500 tx=0x7f136000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:03.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f135a7fc640 1 -- 172.21.15.46:0/1524326089 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360005220 con 0x7f13640cf160 2023-10-30T13:37:03.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f135a7fc640 1 -- 172.21.15.46:0/1524326089 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1360005380 con 0x7f13640cf160 2023-10-30T13:37:03.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.428+0000 7f135a7fc640 1 -- 172.21.15.46:0/1524326089 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360007270 con 0x7f13640cf160 2023-10-30T13:37:03.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.432+0000 7f136bd3e640 1 -- 172.21.15.46:0/1524326089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136414e760 con 0x7f13640cf160 2023-10-30T13:37:03.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.432+0000 7f136bd3e640 1 -- 172.21.15.46:0/1524326089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13640cea50 con 0x7f13640cf160 2023-10-30T13:37:03.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.432+0000 7f136bd3e640 1 -- 172.21.15.46:0/1524326089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13640b2a50 con 0x7f13640cf160 2023-10-30T13:37:03.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.432+0000 7f135a7fc640 1 -- 172.21.15.46:0/1524326089 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13600073d0 con 0x7f13640cf160 2023-10-30T13:37:03.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.432+0000 7f135a7fc640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13480664d0 0x7f1348068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.432+0000 7f135a7fc640 1 -- 172.21.15.46:0/1524326089 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f13600839c0 con 0x7f13640cf160 2023-10-30T13:37:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.432+0000 7f13692d9640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13480664d0 0x7f1348068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:03.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.436+0000 7f13692d9640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13480664d0 0x7f1348068990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f13640c98a0 tx=0x7f1354009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:03.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.436+0000 7f135a7fc640 1 -- 172.21.15.46:0/1524326089 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1360050b20 con 0x7f13640cf160 2023-10-30T13:37:03.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.628+0000 7f136bd3e640 1 -- 172.21.15.46:0/1524326089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f136404e170 con 0x7f13640cf160 2023-10-30T13:37:03.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.628+0000 7f135a7fc640 1 -- 172.21.15.46:0/1524326089 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f1360054140 con 0x7f13640cf160 2023-10-30T13:37:03.636 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:03.637 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:59.650067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:resolve","state_seq":20,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 73 2023-10-30T13:37:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.632+0000 7f136bd3e640 1 -- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13480664d0 msgr2=0x7f1348068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.632+0000 7f136bd3e640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13480664d0 0x7f1348068990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f13640c98a0 tx=0x7f1354009190 comp rx=0 tx=0).stop 2023-10-30T13:37:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.632+0000 7f136bd3e640 1 -- 172.21.15.46:0/1524326089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13640cf160 msgr2=0x7f136414e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.632+0000 7f136bd3e640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13640cf160 0x7f136414e220 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f136000c500 tx=0x7f136000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:37:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.632+0000 7f136bd3e640 1 -- 172.21.15.46:0/1524326089 shutdown_connections 2023-10-30T13:37:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.632+0000 7f136bd3e640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13480664d0 0x7f1348068990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.632+0000 7f136bd3e640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f136415d6f0 0x7f13640cd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.632+0000 7f136bd3e640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13641538a0 0x7f13640b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.632+0000 7f136bd3e640 1 --2- 172.21.15.46:0/1524326089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13640cf160 0x7f136414e220 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.632+0000 7f136bd3e640 1 -- 172.21.15.46:0/1524326089 >> 172.21.15.46:0/1524326089 conn(0x7f136406bfe0 msgr2=0x7f13640c48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.632+0000 7f136bd3e640 1 -- 172.21.15.46:0/1524326089 shutdown_connections 2023-10-30T13:37:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:03.632+0000 7f136bd3e640 1 -- 172.21.15.46:0/1524326089 wait complete. 2023-10-30T13:37:03.658 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:05.660 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:05.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.808+0000 7fbd38d1d640 1 Processor -- start 2023-10-30T13:37:05.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.808+0000 7fbd38d1d640 1 -- start start 2023-10-30T13:37:05.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.808+0000 7fbd38d1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd341518a0 0x7fbd34158a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:05.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.808+0000 7fbd38d1d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbd341581b0 0x7fbd3406fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:05.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.808+0000 7fbd38d1d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd34151c80 con 0x7fbd340c1f50 2023-10-30T13:37:05.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.808+0000 7fbd38d1d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd340c30c0 con 0x7fbd341581b0 2023-10-30T13:37:05.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.808+0000 7fbd38d1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340c4120 con 0x7fbd341518a0 2023-10-30T13:37:05.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.812+0000 7fbd3259c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd341518a0 0x7fbd34158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:05.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.812+0000 7fbd3259c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd341518a0 0x7fbd34158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:60676/0 (socket says 172.21.15.46:60676) 2023-10-30T13:37:05.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.812+0000 7fbd31d9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbd341581b0 0x7fbd3406fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:05.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.812+0000 7fbd3259c640 1 -- 172.21.15.46:0/275023753 learned_addr learned my addr 172.21.15.46:0/275023753 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:05.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.812+0000 7fbd3159a640 1 -- 172.21.15.46:0/275023753 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591420991 0 0) 0x7fbd34151c80 con 0x7fbd340c1f50 2023-10-30T13:37:05.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.812+0000 7fbd3159a640 1 -- 172.21.15.46:0/275023753 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd1c003530 con 0x7fbd340c1f50 2023-10-30T13:37:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.812+0000 7fbd3259c640 1 -- 172.21.15.46:0/275023753 >> v2:172.21.15.154:3300/0 conn(0x7fbd341581b0 msgr2=0x7fbd3406fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.812+0000 7fbd3259c640 1 --2- 172.21.15.46:0/275023753 >> v2:172.21.15.154:3300/0 conn(0x7fbd341581b0 0x7fbd3406fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.812+0000 7fbd3259c640 1 -- 172.21.15.46:0/275023753 >> v1:172.21.15.154:6789/0 conn(0x7fbd340c1f50 legacy=0x7fbd340ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.812+0000 7fbd3259c640 1 -- 172.21.15.46:0/275023753 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd340c3b60 con 0x7fbd341518a0 2023-10-30T13:37:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.812+0000 7fbd31d9b640 1 --2- 172.21.15.46:0/275023753 >> v2:172.21.15.154:3300/0 conn(0x7fbd341581b0 0x7fbd3406fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:37:05.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.812+0000 7fbd3259c640 1 --2- 172.21.15.46:0/275023753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd341518a0 0x7fbd34158a20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbd24009870 tx=0x7fbd2401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abcbd8f478f49196 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:05.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd3159a640 1 -- 172.21.15.46:0/275023753 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd240a8a30 con 0x7fbd341518a0 2023-10-30T13:37:05.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd3159a640 1 -- 172.21.15.46:0/275023753 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd240a8b90 con 0x7fbd341518a0 2023-10-30T13:37:05.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd3159a640 1 -- 172.21.15.46:0/275023753 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd240b0440 con 0x7fbd341518a0 2023-10-30T13:37:05.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 -- 172.21.15.46:0/275023753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd341518a0 msgr2=0x7fbd34158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:05.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 --2- 172.21.15.46:0/275023753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd341518a0 0x7fbd34158a20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbd24009870 tx=0x7fbd2401ed30 comp rx=0 tx=0).stop 2023-10-30T13:37:05.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 -- 172.21.15.46:0/275023753 shutdown_connections 2023-10-30T13:37:05.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 --2- 172.21.15.46:0/275023753 >> v2:172.21.15.154:3300/0 conn(0x7fbd341581b0 0x7fbd3406fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:05.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 --2- 172.21.15.46:0/275023753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd341518a0 0x7fbd34158a20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:05.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 -- 172.21.15.46:0/275023753 >> 172.21.15.46:0/275023753 conn(0x7fbd34069e90 msgr2=0x7fbd3406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:05.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 -- 172.21.15.46:0/275023753 shutdown_connections 2023-10-30T13:37:05.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 -- 172.21.15.46:0/275023753 wait complete. 2023-10-30T13:37:05.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 Processor -- start 2023-10-30T13:37:05.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 -- start start 2023-10-30T13:37:05.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd340c1f50 0x7fbd34155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:05.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd341518a0 0x7fbd3406c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:05.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd341581b0 0x7fbd3414cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:05.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3406b160 con 0x7fbd341518a0 2023-10-30T13:37:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3406b440 con 0x7fbd341581b0 2023-10-30T13:37:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.816+0000 7fbd38d1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3406b720 con 0x7fbd340c1f50 2023-10-30T13:37:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd31d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd341518a0 0x7fbd3406c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd31d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd341518a0 0x7fbd3406c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55620/0 (socket says 172.21.15.46:55620) 2023-10-30T13:37:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd31d9b640 1 -- 172.21.15.46:0/4215983504 learned_addr learned my addr 172.21.15.46:0/4215983504 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:05.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd32d9d640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd341581b0 0x7fbd3414cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:05.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd3259c640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd340c1f50 0x7fbd34155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:05.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd31d9b640 1 -- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd340c1f50 msgr2=0x7fbd34155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:05.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd31d9b640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd340c1f50 0x7fbd34155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:05.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd31d9b640 1 -- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd341581b0 msgr2=0x7fbd3414cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:05.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd31d9b640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd341581b0 0x7fbd3414cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:05.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd31d9b640 1 -- 172.21.15.46:0/4215983504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd28009420 con 0x7fbd341518a0 2023-10-30T13:37:05.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd3259c640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd340c1f50 0x7fbd34155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:37:05.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd31d9b640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd341518a0 0x7fbd3406c4e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fbd2800e980 tx=0x7fbd2800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:05.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd22ffd640 1 -- 172.21.15.46:0/4215983504 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2800ca00 con 0x7fbd341518a0 2023-10-30T13:37:05.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd22ffd640 1 -- 172.21.15.46:0/4215983504 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2800cb60 con 0x7fbd341518a0 2023-10-30T13:37:05.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd22ffd640 1 -- 172.21.15.46:0/4215983504 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd28013470 con 0x7fbd341518a0 2023-10-30T13:37:05.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd38d1d640 1 -- 172.21.15.46:0/4215983504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd240094d0 con 0x7fbd341518a0 2023-10-30T13:37:05.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd38d1d640 1 -- 172.21.15.46:0/4215983504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd3406bce0 con 0x7fbd341518a0 2023-10-30T13:37:05.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd38d1d640 1 -- 172.21.15.46:0/4215983504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd3404e170 con 0x7fbd341518a0 2023-10-30T13:37:05.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.820+0000 7fbd22ffd640 1 -- 172.21.15.46:0/4215983504 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbd280135d0 con 0x7fbd341518a0 2023-10-30T13:37:05.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.824+0000 7fbd22ffd640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbd140664d0 0x7fbd14068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:05.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.824+0000 7fbd22ffd640 1 -- 172.21.15.46:0/4215983504 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fbd280847d0 con 0x7fbd341518a0 2023-10-30T13:37:05.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.824+0000 7fbd3259c640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbd140664d0 0x7fbd14068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:05.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.824+0000 7fbd3259c640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbd140664d0 0x7fbd14068990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbd240b2000 tx=0x7fbd240aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:05.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:05.828+0000 7fbd22ffd640 1 -- 172.21.15.46:0/4215983504 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbd28051900 con 0x7fbd341518a0 2023-10-30T13:37:06.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.016+0000 7fbd38d1d640 1 -- 172.21.15.46:0/4215983504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd341481f0 con 0x7fbd341518a0 2023-10-30T13:37:06.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.016+0000 7fbd22ffd640 1 -- 172.21.15.46:0/4215983504 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5745 (secure 0 0 0) 0x7fbd28054f20 con 0x7fbd341518a0 2023-10-30T13:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:36:59.650067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:resolve","state_seq":20,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:06.025 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 73 2023-10-30T13:37:06.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.020+0000 7fbd38d1d640 1 -- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbd140664d0 msgr2=0x7fbd14068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:06.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.020+0000 7fbd38d1d640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbd140664d0 0x7fbd14068990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbd240b2000 tx=0x7fbd240aa070 comp rx=0 tx=0).stop 2023-10-30T13:37:06.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.020+0000 7fbd38d1d640 1 -- 172.21.15.46:0/4215983504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd341518a0 msgr2=0x7fbd3406c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:06.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.020+0000 7fbd38d1d640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd341518a0 0x7fbd3406c4e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fbd2800e980 tx=0x7fbd2800ee40 comp rx=0 tx=0).stop 2023-10-30T13:37:06.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.020+0000 7fbd38d1d640 1 -- 172.21.15.46:0/4215983504 shutdown_connections 2023-10-30T13:37:06.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.020+0000 7fbd38d1d640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbd140664d0 0x7fbd14068990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:06.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.020+0000 7fbd38d1d640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd341581b0 0x7fbd3414cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:06.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.020+0000 7fbd38d1d640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd341518a0 0x7fbd3406c4e0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:06.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.020+0000 7fbd38d1d640 1 --2- 172.21.15.46:0/4215983504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd340c1f50 0x7fbd34155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:06.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.020+0000 7fbd38d1d640 1 -- 172.21.15.46:0/4215983504 >> 172.21.15.46:0/4215983504 conn(0x7fbd34069e90 msgr2=0x7fbd3414f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:06.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.020+0000 7fbd38d1d640 1 -- 172.21.15.46:0/4215983504 shutdown_connections 2023-10-30T13:37:06.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:06.020+0000 7fbd38d1d640 1 -- 172.21.15.46:0/4215983504 wait complete. 2023-10-30T13:37:06.045 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:08.046 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f994ea17640 1 Processor -- start 2023-10-30T13:37:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f994ea17640 1 -- start start 2023-10-30T13:37:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f994ea17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cf160 0x7f99480cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f994ea17640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480cb800 con 0x7f994815d6f0 2023-10-30T13:37:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f994ea17640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480cb330 con 0x7f99481538a0 2023-10-30T13:37:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f994ea17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99480cad70 con 0x7f99480cf160 2023-10-30T13:37:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f99477fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f994815d6f0 0x7f994815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51704/0 (socket says 172.21.15.46:51704) 2023-10-30T13:37:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f99477fe640 1 -- 172.21.15.46:0/4067490819 learned_addr learned my addr 172.21.15.46:0/4067490819 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:37:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f994cfb4640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cf160 0x7f99480cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482391782 0 0) 0x7f99480cb800 con 0x7f994815d6f0 2023-10-30T13:37:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9930003530 con 0x7f994815d6f0 2023-10-30T13:37:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122927717 0 0) 0x7f99480cb330 con 0x7f99481538a0 2023-10-30T13:37:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99480cb800 con 0x7f99481538a0 2023-10-30T13:37:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3049037352 0 0) 0x7f9930003530 con 0x7f994815d6f0 2023-10-30T13:37:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cf160 msgr2=0x7f99480cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cf160 0x7f99480cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 >> v1:172.21.15.154:6789/0 conn(0x7f99481538a0 legacy=0x7f99480ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480cb050 con 0x7f994815d6f0 2023-10-30T13:37:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f993c001680 con 0x7f994815d6f0 2023-10-30T13:37:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 >> v1:172.21.15.46:6789/0 conn(0x7f994815d6f0 legacy=0x7f994815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9930003b40 0x7f9930003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99300047d0 0x7f9930004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9930005190 0x7f993000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480cad70 con 0x7f9930005190 2023-10-30T13:37:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993000bf00 con 0x7f9930003b40 2023-10-30T13:37:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f993000c0c0 con 0x7f99300047d0 2023-10-30T13:37:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.196+0000 7f994cfb4640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cf160 0x7f99480cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:37:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.200+0000 7f9947fff640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9930003b40 0x7f9930003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.200+0000 7f994cfb4640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9930005190 0x7f993000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.200+0000 7f99477fe640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99300047d0 0x7f9930004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.200+0000 7f994cfb4640 1 -- 172.21.15.46:0/4067490819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99300047d0 msgr2=0x7f9930004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.200+0000 7f994cfb4640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99300047d0 0x7f9930004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.200+0000 7f994cfb4640 1 -- 172.21.15.46:0/4067490819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9930003b40 msgr2=0x7f9930003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:08.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.200+0000 7f994cfb4640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9930003b40 0x7f9930003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:08.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.200+0000 7f994cfb4640 1 -- 172.21.15.46:0/4067490819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9930003530 con 0x7f9930005190 2023-10-30T13:37:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.200+0000 7f994cfb4640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9930005190 0x7f993000b990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f99480cf540 tx=0x7f994001cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d257966e9f4f6fb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.200+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99400a5d50 con 0x7f9930005190 2023-10-30T13:37:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.200+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994000f040 con 0x7f9930005190 2023-10-30T13:37:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.200+0000 7f9946ffd640 1 -- 172.21.15.46:0/4067490819 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940003bc0 con 0x7f9930005190 2023-10-30T13:37:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 -- 172.21.15.46:0/4067490819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9930005190 msgr2=0x7f993000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 --2- 172.21.15.46:0/4067490819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9930005190 0x7f993000b990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f99480cf540 tx=0x7f994001cd00 comp rx=0 tx=0).stop 2023-10-30T13:37:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994cfb4640 1 -- 172.21.15.46:0/4067490819 reap_dead start 2023-10-30T13:37:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 -- 172.21.15.46:0/4067490819 shutdown_connections 2023-10-30T13:37:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 -- 172.21.15.46:0/4067490819 >> 172.21.15.46:0/4067490819 conn(0x7f994806bfe0 msgr2=0x7f994806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 -- 172.21.15.46:0/4067490819 shutdown_connections 2023-10-30T13:37:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 -- 172.21.15.46:0/4067490819 wait complete. 2023-10-30T13:37:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 Processor -- start 2023-10-30T13:37:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 -- start start 2023-10-30T13:37:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9930005190 0x7f99480c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cfd40 0x7f99480c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9948153fe0 0x7f99481585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480c7590 con 0x7f9930005190 2023-10-30T13:37:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480c64d0 con 0x7f9948153fe0 2023-10-30T13:37:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994ea17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99480c7030 con 0x7f99480cfd40 2023-10-30T13:37:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f9947fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9930005190 0x7f99480c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f9947fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9930005190 0x7f99480c7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45870/0 (socket says 172.21.15.46:45870) 2023-10-30T13:37:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f9947fff640 1 -- 172.21.15.46:0/3851325721 learned_addr learned my addr 172.21.15.46:0/3851325721 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:08.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f99477fe640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cfd40 0x7f99480c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:08.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f994cfb4640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9948153fe0 0x7f99481585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:08.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f9947fff640 1 -- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cfd40 msgr2=0x7f99480c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:08.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f9947fff640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cfd40 0x7f99480c8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:08.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.204+0000 7f9947fff640 1 -- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9948153fe0 msgr2=0x7f99481585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:08.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f9947fff640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9948153fe0 0x7f99481585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:08.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f9947fff640 1 -- 172.21.15.46:0/3851325721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480c6790 con 0x7f9930005190 2023-10-30T13:37:08.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f9947fff640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9930005190 0x7f99480c7a20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f993800f7f0 tx=0x7f993800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:08.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f9944ff9640 1 -- 172.21.15.46:0/3851325721 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993800fea0 con 0x7f9930005190 2023-10-30T13:37:08.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f9944ff9640 1 -- 172.21.15.46:0/3851325721 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99380052e0 con 0x7f9930005190 2023-10-30T13:37:08.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f994ea17640 1 -- 172.21.15.46:0/3851325721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99480c6a70 con 0x7f9930005190 2023-10-30T13:37:08.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f9944ff9640 1 -- 172.21.15.46:0/3851325721 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993800d950 con 0x7f9930005190 2023-10-30T13:37:08.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f994ea17640 1 -- 172.21.15.46:0/3851325721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9948155320 con 0x7f9930005190 2023-10-30T13:37:08.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f994ea17640 1 -- 172.21.15.46:0/3851325721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994804e170 con 0x7f9930005190 2023-10-30T13:37:08.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f9944ff9640 1 -- 172.21.15.46:0/3851325721 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f993800dab0 con 0x7f9930005190 2023-10-30T13:37:08.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f9944ff9640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99340664a0 0x7f9934068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:08.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f9944ff9640 1 -- 172.21.15.46:0/3851325721 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f99380842e0 con 0x7f9930005190 2023-10-30T13:37:08.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.208+0000 7f99477fe640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99340664a0 0x7f9934068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:08.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.212+0000 7f99477fe640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99340664a0 0x7f9934068960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f99480d01e0 tx=0x7f993c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:08.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.212+0000 7f9944ff9640 1 -- 172.21.15.46:0/3851325721 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9938051380 con 0x7f9930005190 2023-10-30T13:37:08.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.404+0000 7f994ea17640 1 -- 172.21.15.46:0/3851325721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99480b1dd0 con 0x7f9930005190 2023-10-30T13:37:08.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.404+0000 7f9944ff9640 1 -- 172.21.15.46:0/3851325721 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f99380549a0 con 0x7f9930005190 2023-10-30T13:37:08.411 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:08.411 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:05.227707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 74 2023-10-30T13:37:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.408+0000 7f994ea17640 1 -- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99340664a0 msgr2=0x7f9934068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.408+0000 7f994ea17640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99340664a0 0x7f9934068960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f99480d01e0 tx=0x7f993c009040 comp rx=0 tx=0).stop 2023-10-30T13:37:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.408+0000 7f994ea17640 1 -- 172.21.15.46:0/3851325721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9930005190 msgr2=0x7f99480c7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.408+0000 7f994ea17640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9930005190 0x7f99480c7a20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f993800f7f0 tx=0x7f993800fcb0 comp rx=0 tx=0).stop 2023-10-30T13:37:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.408+0000 7f994ea17640 1 -- 172.21.15.46:0/3851325721 shutdown_connections 2023-10-30T13:37:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.408+0000 7f994ea17640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99340664a0 0x7f9934068960 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.408+0000 7f994ea17640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9948153fe0 0x7f99481585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.408+0000 7f994ea17640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cfd40 0x7f99480c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.408+0000 7f994ea17640 1 --2- 172.21.15.46:0/3851325721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9930005190 0x7f99480c7a20 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.408+0000 7f994ea17640 1 -- 172.21.15.46:0/3851325721 >> 172.21.15.46:0/3851325721 conn(0x7f994806bfe0 msgr2=0x7f99480c4980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.408+0000 7f994ea17640 1 -- 172.21.15.46:0/3851325721 shutdown_connections 2023-10-30T13:37:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:08.408+0000 7f994ea17640 1 -- 172.21.15.46:0/3851325721 wait complete. 2023-10-30T13:37:08.433 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 74, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 74, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5574": { "addr": "172.21.15.46:6837/126953235", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 126953235, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 126953235, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5574, "incarnation": 55, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_5721": { "addr": "172.21.15.46:6833/4044310929", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4044310929, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4044310929, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5721, "incarnation": 61, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 24 }, "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:reconnect", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:37:05.227707+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5721, "mds_1": 5574, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:37:10.434 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:10.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.580+0000 7f02b410f640 1 Processor -- start 2023-10-30T13:37:10.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b410f640 1 -- start start 2023-10-30T13:37:10.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b410f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f02ac0c6b80 0x7f02ac0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:10.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b410f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac0c0a50 con 0x7f02ac0c6f60 2023-10-30T13:37:10.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b410f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac06f6e0 con 0x7f02ac13f4a0 2023-10-30T13:37:10.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b410f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ac06f9c0 con 0x7f02ac0c6b80 2023-10-30T13:37:10.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b16aa640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f02ac0c6f60 0x7f02ac13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51718/0 (socket says 172.21.15.46:51718) 2023-10-30T13:37:10.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b16aa640 1 -- 172.21.15.46:0/3470986409 learned_addr learned my addr 172.21.15.46:0/3470986409 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:37:10.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b1eab640 1 --2- 172.21.15.46:0/3470986409 >> v2:172.21.15.154:3300/0 conn(0x7f02ac0c6b80 0x7f02ac0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:10.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b0ea9640 1 -- 172.21.15.46:0/3470986409 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623594021 0 0) 0x7f02ac0c0a50 con 0x7f02ac0c6f60 2023-10-30T13:37:10.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b0ea9640 1 -- 172.21.15.46:0/3470986409 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0298003530 con 0x7f02ac0c6f60 2023-10-30T13:37:10.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b1eab640 1 -- 172.21.15.46:0/3470986409 >> v1:172.21.15.154:6789/0 conn(0x7f02ac13f4a0 legacy=0x7f02ac1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:10.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b1eab640 1 -- 172.21.15.46:0/3470986409 >> v1:172.21.15.46:6789/0 conn(0x7f02ac0c6f60 legacy=0x7f02ac13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:10.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b0ea9640 1 -- 172.21.15.46:0/3470986409 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 825392557 0 0) 0x7f0298003530 con 0x7f02ac0c6f60 2023-10-30T13:37:10.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.584+0000 7f02b1eab640 1 -- 172.21.15.46:0/3470986409 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac06fca0 con 0x7f02ac0c6b80 2023-10-30T13:37:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.588+0000 7f02b1eab640 1 --2- 172.21.15.46:0/3470986409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac0c6b80 0x7f02ac0bfec0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f02a80097c0 tx=0x7f02a801e5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d77e4327e6f86a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.588+0000 7f02b0ea9640 1 -- 172.21.15.46:0/3470986409 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c001680 con 0x7f02ac0c6b80 2023-10-30T13:37:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.588+0000 7f02b0ea9640 1 -- 172.21.15.46:0/3470986409 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a80a9050 con 0x7f02ac0c6b80 2023-10-30T13:37:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.588+0000 7f02b0ea9640 1 -- 172.21.15.46:0/3470986409 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8004250 con 0x7f02ac0c6b80 2023-10-30T13:37:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.588+0000 7f02b410f640 1 -- 172.21.15.46:0/3470986409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac0c6b80 msgr2=0x7f02ac0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:10.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.588+0000 7f02b410f640 1 --2- 172.21.15.46:0/3470986409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac0c6b80 0x7f02ac0bfec0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f02a80097c0 tx=0x7f02a801e5b0 comp rx=0 tx=0).stop 2023-10-30T13:37:10.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.588+0000 7f02b410f640 1 -- 172.21.15.46:0/3470986409 shutdown_connections 2023-10-30T13:37:10.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.588+0000 7f02b410f640 1 --2- 172.21.15.46:0/3470986409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac0c6b80 0x7f02ac0bfec0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:10.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.588+0000 7f02b410f640 1 -- 172.21.15.46:0/3470986409 >> 172.21.15.46:0/3470986409 conn(0x7f02ac0c7910 msgr2=0x7f02ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:10.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.588+0000 7f02b410f640 1 -- 172.21.15.46:0/3470986409 shutdown_connections 2023-10-30T13:37:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.588+0000 7f02b410f640 1 -- 172.21.15.46:0/3470986409 wait complete. 2023-10-30T13:37:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b410f640 1 Processor -- start 2023-10-30T13:37:10.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b410f640 1 -- start start 2023-10-30T13:37:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b410f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac0c6b80 0x7f02ac0c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b410f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac0c6f60 0x7f02ac0c3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b410f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac13f4a0 0x7f02ac0c3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b410f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac1f2250 con 0x7f02ac0c6f60 2023-10-30T13:37:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b410f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac1f3070 con 0x7f02ac13f4a0 2023-10-30T13:37:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b410f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac1f2480 con 0x7f02ac0c6b80 2023-10-30T13:37:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b16aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac0c6f60 0x7f02ac0c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b16aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac0c6f60 0x7f02ac0c3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45872/0 (socket says 172.21.15.46:45872) 2023-10-30T13:37:10.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b16aa640 1 -- 172.21.15.46:0/3902710574 learned_addr learned my addr 172.21.15.46:0/3902710574 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:10.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b1eab640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac0c6b80 0x7f02ac0c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:10.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b26ac640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac13f4a0 0x7f02ac0c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:10.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b16aa640 1 -- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac0c6b80 msgr2=0x7f02ac0c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:10.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b16aa640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac0c6b80 0x7f02ac0c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:10.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b16aa640 1 -- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac13f4a0 msgr2=0x7f02ac0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:10.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b16aa640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac13f4a0 0x7f02ac0c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:10.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b16aa640 1 -- 172.21.15.46:0/3902710574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a8009420 con 0x7f02ac0c6f60 2023-10-30T13:37:10.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b16aa640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac0c6f60 0x7f02ac0c3300 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f029c00c760 tx=0x7f029c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:10.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02a27fc640 1 -- 172.21.15.46:0/3902710574 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c005ae0 con 0x7f02ac0c6f60 2023-10-30T13:37:10.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02a27fc640 1 -- 172.21.15.46:0/3902710574 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029c005c40 con 0x7f02ac0c6f60 2023-10-30T13:37:10.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02a27fc640 1 -- 172.21.15.46:0/3902710574 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c007070 con 0x7f02ac0c6f60 2023-10-30T13:37:10.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b410f640 1 -- 172.21.15.46:0/3902710574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac1f2740 con 0x7f02ac0c6f60 2023-10-30T13:37:10.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.592+0000 7f02b410f640 1 -- 172.21.15.46:0/3902710574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ac1f2d00 con 0x7f02ac0c6f60 2023-10-30T13:37:10.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.596+0000 7f02b410f640 1 -- 172.21.15.46:0/3902710574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ac1f31b0 con 0x7f02ac0c6f60 2023-10-30T13:37:10.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.596+0000 7f02a27fc640 1 -- 172.21.15.46:0/3902710574 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f029c0071d0 con 0x7f02ac0c6f60 2023-10-30T13:37:10.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.596+0000 7f02a27fc640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02900664d0 0x7f0290068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:10.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.596+0000 7f02a27fc640 1 -- 172.21.15.46:0/3902710574 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f029c083fe0 con 0x7f02ac0c6f60 2023-10-30T13:37:10.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.596+0000 7f02b1eab640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02900664d0 0x7f0290068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:10.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.596+0000 7f02b1eab640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02900664d0 0x7f0290068990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f02a80af000 tx=0x7f02a80a8000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:10.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.600+0000 7f02a27fc640 1 -- 172.21.15.46:0/3902710574 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f029c054b70 con 0x7f02ac0c6f60 2023-10-30T13:37:10.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.788+0000 7f02b410f640 1 -- 172.21.15.46:0/3902710574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02ac04e170 con 0x7f02ac0c6f60 2023-10-30T13:37:10.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.788+0000 7f02a27fc640 1 -- 172.21.15.46:0/3902710574 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f029c054700 con 0x7f02ac0c6f60 2023-10-30T13:37:10.796 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:10.796 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:05.227707+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:10.797 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 74 2023-10-30T13:37:10.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.792+0000 7f02b410f640 1 -- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02900664d0 msgr2=0x7f0290068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:10.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.792+0000 7f02b410f640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02900664d0 0x7f0290068990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f02a80af000 tx=0x7f02a80a8000 comp rx=0 tx=0).stop 2023-10-30T13:37:10.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.792+0000 7f02b410f640 1 -- 172.21.15.46:0/3902710574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac0c6f60 msgr2=0x7f02ac0c3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:10.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.792+0000 7f02b410f640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac0c6f60 0x7f02ac0c3300 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f029c00c760 tx=0x7f029c00cc20 comp rx=0 tx=0).stop 2023-10-30T13:37:10.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.792+0000 7f02b410f640 1 -- 172.21.15.46:0/3902710574 shutdown_connections 2023-10-30T13:37:10.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.792+0000 7f02b410f640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02900664d0 0x7f0290068990 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:10.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.792+0000 7f02b410f640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac13f4a0 0x7f02ac0c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:10.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.792+0000 7f02b410f640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02ac0c6f60 0x7f02ac0c3300 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:10.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.792+0000 7f02b410f640 1 --2- 172.21.15.46:0/3902710574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac0c6b80 0x7f02ac0c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:10.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.792+0000 7f02b410f640 1 -- 172.21.15.46:0/3902710574 >> 172.21.15.46:0/3902710574 conn(0x7f02ac0c7910 msgr2=0x7f02ac139be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:10.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.792+0000 7f02b410f640 1 -- 172.21.15.46:0/3902710574 shutdown_connections 2023-10-30T13:37:10.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:10.792+0000 7f02b410f640 1 -- 172.21.15.46:0/3902710574 wait complete. 2023-10-30T13:37:10.817 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:12.818 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:12.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11c91d640 1 Processor -- start 2023-10-30T13:37:12.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11c91d640 1 -- start start 2023-10-30T13:37:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11c91d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff118153d70 0x7ff11806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11c91d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff11806e470 0x7ff1180cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11c91d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11806afa0 con 0x7ff1180c8490 2023-10-30T13:37:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11c91d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff11806d780 con 0x7ff11806e470 2023-10-30T13:37:12.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11c91d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff118154150 con 0x7ff118153d70 2023-10-30T13:37:12.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11659c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff118153d70 0x7ff11806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:12.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff115d9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff11806e470 0x7ff1180cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:12.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff116d9d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff1180c8490 0x7ff1180c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:41358/0 (socket says 172.21.15.46:41358) 2023-10-30T13:37:12.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11659c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff118153d70 0x7ff11806ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:50770/0 (socket says 172.21.15.46:50770) 2023-10-30T13:37:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff116d9d640 1 -- 172.21.15.46:0/3594070054 learned_addr learned my addr 172.21.15.46:0/3594070054 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:37:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11559a640 1 -- 172.21.15.46:0/3594070054 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284227383 0 0) 0x7ff11806afa0 con 0x7ff1180c8490 2023-10-30T13:37:12.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11559a640 1 -- 172.21.15.46:0/3594070054 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff100003530 con 0x7ff1180c8490 2023-10-30T13:37:12.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11659c640 1 -- 172.21.15.46:0/3594070054 >> v2:172.21.15.154:3300/0 conn(0x7ff11806e470 msgr2=0x7ff1180cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:12.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11659c640 1 --2- 172.21.15.46:0/3594070054 >> v2:172.21.15.154:3300/0 conn(0x7ff11806e470 0x7ff1180cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:12.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11659c640 1 -- 172.21.15.46:0/3594070054 >> v1:172.21.15.154:6789/0 conn(0x7ff1180c8490 legacy=0x7ff1180c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:12.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff11659c640 1 -- 172.21.15.46:0/3594070054 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11806e030 con 0x7ff118153d70 2023-10-30T13:37:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.968+0000 7ff115d9b640 1 --2- 172.21.15.46:0/3594070054 >> v2:172.21.15.154:3300/0 conn(0x7ff11806e470 0x7ff1180cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:37:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.972+0000 7ff11659c640 1 --2- 172.21.15.46:0/3594070054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff118153d70 0x7ff11806ca80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff1080097c0 tx=0x7ff10801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1658b22058ca7778 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.972+0000 7ff11559a640 1 -- 172.21.15.46:0/3594070054 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1080ab650 con 0x7ff118153d70 2023-10-30T13:37:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.972+0000 7ff11559a640 1 -- 172.21.15.46:0/3594070054 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1080ab7b0 con 0x7ff118153d70 2023-10-30T13:37:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.972+0000 7ff11559a640 1 -- 172.21.15.46:0/3594070054 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1080abb40 con 0x7ff118153d70 2023-10-30T13:37:12.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.972+0000 7ff11c91d640 1 -- 172.21.15.46:0/3594070054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff118153d70 msgr2=0x7ff11806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.972+0000 7ff11c91d640 1 --2- 172.21.15.46:0/3594070054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff118153d70 0x7ff11806ca80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff1080097c0 tx=0x7ff10801c450 comp rx=0 tx=0).stop 2023-10-30T13:37:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.972+0000 7ff11c91d640 1 -- 172.21.15.46:0/3594070054 shutdown_connections 2023-10-30T13:37:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.972+0000 7ff11c91d640 1 --2- 172.21.15.46:0/3594070054 >> v2:172.21.15.154:3300/0 conn(0x7ff11806e470 0x7ff1180cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.972+0000 7ff11c91d640 1 --2- 172.21.15.46:0/3594070054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff118153d70 0x7ff11806ca80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:12.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.972+0000 7ff11c91d640 1 -- 172.21.15.46:0/3594070054 >> 172.21.15.46:0/3594070054 conn(0x7ff118071ea0 msgr2=0x7ff118065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:12.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.972+0000 7ff11c91d640 1 -- 172.21.15.46:0/3594070054 shutdown_connections 2023-10-30T13:37:12.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.972+0000 7ff11c91d640 1 -- 172.21.15.46:0/3594070054 wait complete. 2023-10-30T13:37:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11c91d640 1 Processor -- start 2023-10-30T13:37:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11c91d640 1 -- start start 2023-10-30T13:37:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11c91d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11806e470 0x7ff118154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11c91d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1180c8490 0x7ff118145ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11c91d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff118153d70 0x7ff118146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11c91d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff118141200 con 0x7ff11806e470 2023-10-30T13:37:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11c91d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff118145650 con 0x7ff118153d70 2023-10-30T13:37:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11c91d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180cafb0 con 0x7ff1180c8490 2023-10-30T13:37:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11806e470 0x7ff118154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:12.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11806e470 0x7ff118154590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45886/0 (socket says 172.21.15.46:45886) 2023-10-30T13:37:12.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11659c640 1 -- 172.21.15.46:0/3935332345 learned_addr learned my addr 172.21.15.46:0/3935332345 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:12.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff116d9d640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff118153d70 0x7ff118146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:12.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff115d9b640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1180c8490 0x7ff118145ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:12.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11659c640 1 -- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1180c8490 msgr2=0x7ff118145ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:12.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11659c640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1180c8490 0x7ff118145ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:12.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11659c640 1 -- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff118153d70 msgr2=0x7ff118146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:12.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11659c640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff118153d70 0x7ff118146020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:12.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11659c640 1 -- 172.21.15.46:0/3935332345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10c009420 con 0x7ff11806e470 2023-10-30T13:37:12.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff116d9d640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff118153d70 0x7ff118146020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff115d9b640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1180c8490 0x7ff118145ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11659c640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11806e470 0x7ff118154590 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff1080ad040 tx=0x7ff10801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff106ffd640 1 -- 172.21.15.46:0/3935332345 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1080bf610 con 0x7ff11806e470 2023-10-30T13:37:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff106ffd640 1 -- 172.21.15.46:0/3935332345 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1080bf770 con 0x7ff11806e470 2023-10-30T13:37:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff106ffd640 1 -- 172.21.15.46:0/3935332345 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1080bfae0 con 0x7ff11806e470 2023-10-30T13:37:12.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11c91d640 1 -- 172.21.15.46:0/3935332345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff108009420 con 0x7ff11806e470 2023-10-30T13:37:12.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.976+0000 7ff11c91d640 1 -- 172.21.15.46:0/3935332345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1180cc630 con 0x7ff11806e470 2023-10-30T13:37:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.980+0000 7ff11c91d640 1 -- 172.21.15.46:0/3935332345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11804e170 con 0x7ff11806e470 2023-10-30T13:37:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.980+0000 7ff106ffd640 1 -- 172.21.15.46:0/3935332345 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1080bfc40 con 0x7ff11806e470 2023-10-30T13:37:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.980+0000 7ff106ffd640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0f80664a0 0x7ff0f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.980+0000 7ff106ffd640 1 -- 172.21.15.46:0/3935332345 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7ff10811ec60 con 0x7ff11806e470 2023-10-30T13:37:12.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.980+0000 7ff115d9b640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0f80664a0 0x7ff0f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:12.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.984+0000 7ff115d9b640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0f80664a0 0x7ff0f8068960 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff1181f3a40 tx=0x7ff10c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:12.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:12.984+0000 7ff106ffd640 1 -- 172.21.15.46:0/3935332345 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1080ebd10 con 0x7ff11806e470 2023-10-30T13:37:13.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 -- 172.21.15.46:0/3935332345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff11806c2a0 con 0x7ff11806e470 2023-10-30T13:37:13.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff106ffd640 1 -- 172.21.15.46:0/3935332345 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5744 (secure 0 0 0) 0x7ff1080ef330 con 0x7ff11806e470 2023-10-30T13:37:13.182 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:13.183 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:11.197969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:13.183 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 75 2023-10-30T13:37:13.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 -- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0f80664a0 msgr2=0x7ff0f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:13.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0f80664a0 0x7ff0f8068960 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff1181f3a40 tx=0x7ff10c008040 comp rx=0 tx=0).stop 2023-10-30T13:37:13.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 -- 172.21.15.46:0/3935332345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11806e470 msgr2=0x7ff118154590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:13.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11806e470 0x7ff118154590 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff1080ad040 tx=0x7ff10801ac90 comp rx=0 tx=0).stop 2023-10-30T13:37:13.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 -- 172.21.15.46:0/3935332345 shutdown_connections 2023-10-30T13:37:13.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0f80664a0 0x7ff0f8068960 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:13.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff118153d70 0x7ff118146020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:13.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1180c8490 0x7ff118145ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:13.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 --2- 172.21.15.46:0/3935332345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11806e470 0x7ff118154590 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:13.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 -- 172.21.15.46:0/3935332345 >> 172.21.15.46:0/3935332345 conn(0x7ff118071ea0 msgr2=0x7ff118146d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:13.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 -- 172.21.15.46:0/3935332345 shutdown_connections 2023-10-30T13:37:13.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:13.176+0000 7ff11c91d640 1 -- 172.21.15.46:0/3935332345 wait complete. 2023-10-30T13:37:13.204 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:15.205 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:15.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.352+0000 7f5ec3aa9640 1 Processor -- start 2023-10-30T13:37:15.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.352+0000 7f5ec3aa9640 1 -- start start 2023-10-30T13:37:15.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.352+0000 7f5ec3aa9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ebc0c6540 0x7f5ebc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.352+0000 7f5ec3aa9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc0c5810 con 0x7f5ebc070740 2023-10-30T13:37:15.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec3aa9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc0c69b0 con 0x7f5ebc1580e0 2023-10-30T13:37:15.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec3aa9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ebc06acb0 con 0x7f5ebc0c6540 2023-10-30T13:37:15.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec2046640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ebc0c6540 0x7f5ebc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec1044640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5ebc070740 0x7f5ebc06a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51734/0 (socket says 172.21.15.46:51734) 2023-10-30T13:37:15.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec1044640 1 -- 172.21.15.46:0/3911396240 learned_addr learned my addr 172.21.15.46:0/3911396240 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:37:15.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec2046640 1 -- 172.21.15.46:0/3911396240 >> v1:172.21.15.154:6789/0 conn(0x7f5ebc1580e0 legacy=0x7f5ebc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec2046640 1 -- 172.21.15.46:0/3911396240 >> v1:172.21.15.46:6789/0 conn(0x7f5ebc070740 legacy=0x7f5ebc06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec2046640 1 -- 172.21.15.46:0/3911396240 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc0c6bb0 con 0x7f5ebc0c6540 2023-10-30T13:37:15.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec0843640 1 -- 172.21.15.46:0/3911396240 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078519174 0 0) 0x7f5ebc0c5810 con 0x7f5ebc070740 2023-10-30T13:37:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec2046640 1 --2- 172.21.15.46:0/3911396240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ebc0c6540 0x7f5ebc15e700 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4009d90 tx=0x7f5eb401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ae7f68f9a43a54 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec0843640 1 -- 172.21.15.46:0/3911396240 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb40a5920 con 0x7f5ebc0c6540 2023-10-30T13:37:15.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec0843640 1 -- 172.21.15.46:0/3911396240 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb40a5a80 con 0x7f5ebc0c6540 2023-10-30T13:37:15.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.356+0000 7f5ec0843640 1 -- 172.21.15.46:0/3911396240 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb40a5e10 con 0x7f5ebc0c6540 2023-10-30T13:37:15.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/3911396240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ebc0c6540 msgr2=0x7f5ebc15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 --2- 172.21.15.46:0/3911396240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ebc0c6540 0x7f5ebc15e700 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4009d90 tx=0x7f5eb401c950 comp rx=0 tx=0).stop 2023-10-30T13:37:15.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/3911396240 shutdown_connections 2023-10-30T13:37:15.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 --2- 172.21.15.46:0/3911396240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ebc0c6540 0x7f5ebc15e700 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/3911396240 >> 172.21.15.46:0/3911396240 conn(0x7f5ebc0c7910 msgr2=0x7f5ebc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/3911396240 shutdown_connections 2023-10-30T13:37:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/3911396240 wait complete. 2023-10-30T13:37:15.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 Processor -- start 2023-10-30T13:37:15.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 -- start start 2023-10-30T13:37:15.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ebc070740 0x7f5ebc146c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ebc0c6540 0x7f5ebc1f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ebc1580e0 0x7f5ebc14cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc1fa320 con 0x7f5ebc070740 2023-10-30T13:37:15.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc149910 con 0x7f5ebc1580e0 2023-10-30T13:37:15.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec3aa9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc1fa7b0 con 0x7f5ebc0c6540 2023-10-30T13:37:15.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec1845640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ebc070740 0x7f5ebc146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec1845640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ebc070740 0x7f5ebc146c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45910/0 (socket says 172.21.15.46:45910) 2023-10-30T13:37:15.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec1845640 1 -- 172.21.15.46:0/145935177 learned_addr learned my addr 172.21.15.46:0/145935177 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:15.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec1044640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ebc0c6540 0x7f5ebc1f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.360+0000 7f5ec2046640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ebc1580e0 0x7f5ebc14cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5ec1845640 1 -- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ebc0c6540 msgr2=0x7f5ebc1f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5ec1845640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ebc0c6540 0x7f5ebc1f6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5ec1845640 1 -- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ebc1580e0 msgr2=0x7f5ebc14cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5ec1845640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ebc1580e0 0x7f5ebc14cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5ec1845640 1 -- 172.21.15.46:0/145935177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb4009a70 con 0x7f5ebc070740 2023-10-30T13:37:15.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5ec1044640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ebc0c6540 0x7f5ebc1f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:15.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5ec2046640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ebc1580e0 0x7f5ebc14cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:15.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5ec1845640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ebc070740 0x7f5ebc146c10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f5eb800c3f0 tx=0x7f5eb800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:15.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5eb27fc640 1 -- 172.21.15.46:0/145935177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb80052d0 con 0x7f5ebc070740 2023-10-30T13:37:15.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5eb27fc640 1 -- 172.21.15.46:0/145935177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb8005430 con 0x7f5ebc070740 2023-10-30T13:37:15.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/145935177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ebc149630 con 0x7f5ebc070740 2023-10-30T13:37:15.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5eb27fc640 1 -- 172.21.15.46:0/145935177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8003840 con 0x7f5ebc070740 2023-10-30T13:37:15.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/145935177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ebc1f9cd0 con 0x7f5ebc070740 2023-10-30T13:37:15.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/145935177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ebc04e170 con 0x7f5ebc070740 2023-10-30T13:37:15.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5eb27fc640 1 -- 172.21.15.46:0/145935177 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5eb80039a0 con 0x7f5ebc070740 2023-10-30T13:37:15.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5eb27fc640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ea00664d0 0x7f5ea0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5eb27fc640 1 -- 172.21.15.46:0/145935177 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f5eb80837d0 con 0x7f5ebc070740 2023-10-30T13:37:15.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.364+0000 7f5ec1044640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ea00664d0 0x7f5ea0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.368+0000 7f5ec1044640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ea00664d0 0x7f5ea0068990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc06a9d0 tx=0x7f5eac007c60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:15.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.372+0000 7f5eb27fc640 1 -- 172.21.15.46:0/145935177 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5eb8050870 con 0x7f5ebc070740 2023-10-30T13:37:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.556+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/145935177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ebc06aed0 con 0x7f5ebc070740 2023-10-30T13:37:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.556+0000 7f5eb27fc640 1 -- 172.21.15.46:0/145935177 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f5eb8053e90 con 0x7f5ebc070740 2023-10-30T13:37:15.564 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:15.565 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:11.197969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:15.565 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 75 2023-10-30T13:37:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.560+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ea00664d0 msgr2=0x7f5ea0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.560+0000 7f5ec3aa9640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ea00664d0 0x7f5ea0068990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc06a9d0 tx=0x7f5eac007c60 comp rx=0 tx=0).stop 2023-10-30T13:37:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.560+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/145935177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ebc070740 msgr2=0x7f5ebc146c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.560+0000 7f5ec3aa9640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ebc070740 0x7f5ebc146c10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f5eb800c3f0 tx=0x7f5eb800c8b0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.560+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/145935177 shutdown_connections 2023-10-30T13:37:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.560+0000 7f5ec3aa9640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ea00664d0 0x7f5ea0068990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.560+0000 7f5ec3aa9640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ebc1580e0 0x7f5ebc14cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.560+0000 7f5ec3aa9640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ebc0c6540 0x7f5ebc1f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.560+0000 7f5ec3aa9640 1 --2- 172.21.15.46:0/145935177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ebc070740 0x7f5ebc146c10 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.560+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/145935177 >> 172.21.15.46:0/145935177 conn(0x7f5ebc0c7910 msgr2=0x7f5ebc153a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:15.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.560+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/145935177 shutdown_connections 2023-10-30T13:37:15.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:15.560+0000 7f5ec3aa9640 1 -- 172.21.15.46:0/145935177 wait complete. 2023-10-30T13:37:15.586 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:17.549 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:37:17.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:37:17.587 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:17.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.740+0000 7f868d3c5640 1 Processor -- start 2023-10-30T13:37:17.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.740+0000 7f868d3c5640 1 -- start start 2023-10-30T13:37:17.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.744+0000 7f868d3c5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f868806f7a0 0x7f868806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:17.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.744+0000 7f868d3c5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8688066cf0 con 0x7f8688071110 2023-10-30T13:37:17.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.744+0000 7f868d3c5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86880c70d0 con 0x7f86880c6bd0 2023-10-30T13:37:17.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.744+0000 7f868d3c5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86880700c0 con 0x7f868806f7a0 2023-10-30T13:37:17.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.744+0000 7f8686ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f868806f7a0 0x7f868806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:17.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.744+0000 7f86877fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8688071110 0x7f8688158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48584/0 (socket says 172.21.15.46:48584) 2023-10-30T13:37:17.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.744+0000 7f86877fe640 1 -- 172.21.15.46:0/2048067277 learned_addr learned my addr 172.21.15.46:0/2048067277 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:37:17.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.744+0000 7f8686ffd640 1 -- 172.21.15.46:0/2048067277 >> v1:172.21.15.154:6789/0 conn(0x7f86880c6bd0 legacy=0x7f8688069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:17.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.744+0000 7f8686ffd640 1 -- 172.21.15.46:0/2048067277 >> v1:172.21.15.46:6789/0 conn(0x7f8688071110 legacy=0x7f8688158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:17.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.744+0000 7f8686ffd640 1 -- 172.21.15.46:0/2048067277 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86880c7390 con 0x7f868806f7a0 2023-10-30T13:37:17.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.744+0000 7f86867fc640 1 -- 172.21.15.46:0/2048067277 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882598364 0 0) 0x7f8688066cf0 con 0x7f8688071110 2023-10-30T13:37:17.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f8686ffd640 1 --2- 172.21.15.46:0/2048067277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f868806f7a0 0x7f868806fb80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f86780097c0 tx=0x7f867801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647849ac987a40aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:17.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f86867fc640 1 -- 172.21.15.46:0/2048067277 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86780a8970 con 0x7f868806f7a0 2023-10-30T13:37:17.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f86867fc640 1 -- 172.21.15.46:0/2048067277 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86780a8ad0 con 0x7f868806f7a0 2023-10-30T13:37:17.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f86867fc640 1 -- 172.21.15.46:0/2048067277 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86780a8e60 con 0x7f868806f7a0 2023-10-30T13:37:17.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 -- 172.21.15.46:0/2048067277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f868806f7a0 msgr2=0x7f868806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:17.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 --2- 172.21.15.46:0/2048067277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f868806f7a0 0x7f868806fb80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f86780097c0 tx=0x7f867801ec70 comp rx=0 tx=0).stop 2023-10-30T13:37:17.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 -- 172.21.15.46:0/2048067277 shutdown_connections 2023-10-30T13:37:17.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 --2- 172.21.15.46:0/2048067277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f868806f7a0 0x7f868806fb80 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:17.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 -- 172.21.15.46:0/2048067277 >> 172.21.15.46:0/2048067277 conn(0x7f86880c7910 msgr2=0x7f86880c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:17.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 -- 172.21.15.46:0/2048067277 shutdown_connections 2023-10-30T13:37:17.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 -- 172.21.15.46:0/2048067277 wait complete. 2023-10-30T13:37:17.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 Processor -- start 2023-10-30T13:37:17.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 -- start start 2023-10-30T13:37:17.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f868806f7a0 0x7f86881554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:17.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8688071110 0x7f868814f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:17.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86880c6bd0 0x7f8688153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:17.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8688152a20 con 0x7f86880c6bd0 2023-10-30T13:37:17.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86880c9020 con 0x7f8688071110 2023-10-30T13:37:17.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f868d3c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86880c9300 con 0x7f868806f7a0 2023-10-30T13:37:17.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f86877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86880c6bd0 0x7f8688153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:17.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f8686ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f868806f7a0 0x7f86881554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:17.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f86877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86880c6bd0 0x7f8688153df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47300/0 (socket says 172.21.15.46:47300) 2023-10-30T13:37:17.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.748+0000 7f86877fe640 1 -- 172.21.15.46:0/913651452 learned_addr learned my addr 172.21.15.46:0/913651452 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:17.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f867ffff640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8688071110 0x7f868814f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:17.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f86877fe640 1 -- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f868806f7a0 msgr2=0x7f86881554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:17.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f86877fe640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f868806f7a0 0x7f86881554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:17.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f86877fe640 1 -- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8688071110 msgr2=0x7f868814f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:17.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f86877fe640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8688071110 0x7f868814f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:17.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f86877fe640 1 -- 172.21.15.46:0/913651452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8678009420 con 0x7f86880c6bd0 2023-10-30T13:37:17.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f86877fe640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86880c6bd0 0x7f8688153df0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f868000c780 tx=0x7f868000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:17.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f867f7fe640 1 -- 172.21.15.46:0/913651452 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868000ce90 con 0x7f86880c6bd0 2023-10-30T13:37:17.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f867f7fe640 1 -- 172.21.15.46:0/913651452 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868000a640 con 0x7f86880c6bd0 2023-10-30T13:37:17.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f867f7fe640 1 -- 172.21.15.46:0/913651452 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868000a9d0 con 0x7f86880c6bd0 2023-10-30T13:37:17.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f868d3c5640 1 -- 172.21.15.46:0/913651452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8688152c70 con 0x7f86880c6bd0 2023-10-30T13:37:17.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f868d3c5640 1 -- 172.21.15.46:0/913651452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8688155a30 con 0x7f86880c6bd0 2023-10-30T13:37:17.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f868d3c5640 1 -- 172.21.15.46:0/913651452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868804e170 con 0x7f86880c6bd0 2023-10-30T13:37:17.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.752+0000 7f867f7fe640 1 -- 172.21.15.46:0/913651452 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f868000ab30 con 0x7f86880c6bd0 2023-10-30T13:37:17.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.756+0000 7f867f7fe640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f86680664a0 0x7f8668068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:17.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.756+0000 7f867f7fe640 1 -- 172.21.15.46:0/913651452 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f8680083650 con 0x7f86880c6bd0 2023-10-30T13:37:17.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.756+0000 7f8686ffd640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f86680664a0 0x7f8668068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:17.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.756+0000 7f8686ffd640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f86680664a0 0x7f8668068960 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f86780b2000 tx=0x7f8678008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:17.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.760+0000 7f867f7fe640 1 -- 172.21.15.46:0/913651452 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8680050830 con 0x7f86880c6bd0 2023-10-30T13:37:17.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 -- 172.21.15.46:0/913651452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8688153120 con 0x7f86880c6bd0 2023-10-30T13:37:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f867f7fe640 1 -- 172.21.15.46:0/913651452 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f8680018090 con 0x7f86880c6bd0 2023-10-30T13:37:17.953 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:17.954 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:11.197969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 75 2023-10-30T13:37:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 -- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f86680664a0 msgr2=0x7f8668068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f86680664a0 0x7f8668068960 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f86780b2000 tx=0x7f8678008040 comp rx=0 tx=0).stop 2023-10-30T13:37:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 -- 172.21.15.46:0/913651452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86880c6bd0 msgr2=0x7f8688153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86880c6bd0 0x7f8688153df0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f868000c780 tx=0x7f868000cc40 comp rx=0 tx=0).stop 2023-10-30T13:37:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 -- 172.21.15.46:0/913651452 shutdown_connections 2023-10-30T13:37:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f86680664a0 0x7f8668068960 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86880c6bd0 0x7f8688153df0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:17.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8688071110 0x7f868814f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:17.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 --2- 172.21.15.46:0/913651452 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f868806f7a0 0x7f86881554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:17.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 -- 172.21.15.46:0/913651452 >> 172.21.15.46:0/913651452 conn(0x7f86880c7910 msgr2=0x7f86880c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:17.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 -- 172.21.15.46:0/913651452 shutdown_connections 2023-10-30T13:37:17.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:17.948+0000 7f868d3c5640 1 -- 172.21.15.46:0/913651452 wait complete. 2023-10-30T13:37:17.974 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:19.975 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:20.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.124+0000 7fe44053e640 1 Processor -- start 2023-10-30T13:37:20.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.124+0000 7fe44053e640 1 -- start start 2023-10-30T13:37:20.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe44053e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4380c6b80 0x7fe4380bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe44053e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4380c6f60 0x7fe4380c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe44053e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4380c0a50 con 0x7fe4380c8430 2023-10-30T13:37:20.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe44053e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe43806f6e0 con 0x7fe4380c6f60 2023-10-30T13:37:20.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe44053e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43806f9c0 con 0x7fe4380c6b80 2023-10-30T13:37:20.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe43dad9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4380c6f60 0x7fe4380c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe43eadb640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe4380c8430 0x7fe43813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48598/0 (socket says 172.21.15.46:48598) 2023-10-30T13:37:20.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe43eadb640 1 -- 172.21.15.46:0/1892517357 learned_addr learned my addr 172.21.15.46:0/1892517357 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:37:20.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe43e2da640 1 --2- 172.21.15.46:0/1892517357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4380c6b80 0x7fe4380bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe43dad9640 1 -- 172.21.15.46:0/1892517357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4380c6b80 msgr2=0x7fe4380bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:20.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe43dad9640 1 --2- 172.21.15.46:0/1892517357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4380c6b80 0x7fe4380bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe43dad9640 1 -- 172.21.15.46:0/1892517357 >> v1:172.21.15.46:6789/0 conn(0x7fe4380c8430 legacy=0x7fe43813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe43dad9640 1 -- 172.21.15.46:0/1892517357 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43806fca0 con 0x7fe4380c6f60 2023-10-30T13:37:20.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe43d2d8640 1 -- 172.21.15.46:0/1892517357 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632821058 0 0) 0x7fe4380c0a50 con 0x7fe4380c8430 2023-10-30T13:37:20.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.128+0000 7fe43e2da640 1 --2- 172.21.15.46:0/1892517357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4380c6b80 0x7fe4380bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:20.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe43dad9640 1 --2- 172.21.15.46:0/1892517357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380c6f60 0x7fe4380c0400 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe4280097c0 tx=0x7fe42801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c18eb8ebfdfe8c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:20.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe43d2d8640 1 -- 172.21.15.46:0/1892517357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4280ad640 con 0x7fe4380c6f60 2023-10-30T13:37:20.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe43d2d8640 1 -- 172.21.15.46:0/1892517357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4280ad7a0 con 0x7fe4380c6f60 2023-10-30T13:37:20.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe43d2d8640 1 -- 172.21.15.46:0/1892517357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4280adb30 con 0x7fe4380c6f60 2023-10-30T13:37:20.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe44053e640 1 -- 172.21.15.46:0/1892517357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380c6f60 msgr2=0x7fe4380c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe44053e640 1 --2- 172.21.15.46:0/1892517357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380c6f60 0x7fe4380c0400 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe4280097c0 tx=0x7fe42801c480 comp rx=0 tx=0).stop 2023-10-30T13:37:20.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe44053e640 1 -- 172.21.15.46:0/1892517357 shutdown_connections 2023-10-30T13:37:20.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe44053e640 1 --2- 172.21.15.46:0/1892517357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380c6f60 0x7fe4380c0400 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe44053e640 1 --2- 172.21.15.46:0/1892517357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4380c6b80 0x7fe4380bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe44053e640 1 -- 172.21.15.46:0/1892517357 >> 172.21.15.46:0/1892517357 conn(0x7fe4380c7910 msgr2=0x7fe4380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:20.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe44053e640 1 -- 172.21.15.46:0/1892517357 shutdown_connections 2023-10-30T13:37:20.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe44053e640 1 -- 172.21.15.46:0/1892517357 wait complete. 2023-10-30T13:37:20.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe44053e640 1 Processor -- start 2023-10-30T13:37:20.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.132+0000 7fe44053e640 1 -- start start 2023-10-30T13:37:20.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe44053e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380c6b80 0x7fe4381391d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe44053e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4380c6f60 0x7fe438159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe44053e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4380c8430 0x7fe438150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe44053e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43813c7e0 con 0x7fe4380c6b80 2023-10-30T13:37:20.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe44053e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43813cd60 con 0x7fe4380c8430 2023-10-30T13:37:20.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe44053e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43813e740 con 0x7fe4380c6f60 2023-10-30T13:37:20.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43e2da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380c6b80 0x7fe4381391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43e2da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380c6b80 0x7fe4381391d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47318/0 (socket says 172.21.15.46:47318) 2023-10-30T13:37:20.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43e2da640 1 -- 172.21.15.46:0/2065015748 learned_addr learned my addr 172.21.15.46:0/2065015748 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:20.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43dad9640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4380c6f60 0x7fe438159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43eadb640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4380c8430 0x7fe438150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43e2da640 1 -- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4380c6f60 msgr2=0x7fe438159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43e2da640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4380c6f60 0x7fe438159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43e2da640 1 -- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4380c8430 msgr2=0x7fe438150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43e2da640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4380c8430 0x7fe438150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43e2da640 1 -- 172.21.15.46:0/2065015748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe428009420 con 0x7fe4380c6b80 2023-10-30T13:37:20.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43dad9640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4380c6f60 0x7fe438159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:37:20.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43eadb640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4380c8430 0x7fe438150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:37:20.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe43e2da640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380c6b80 0x7fe4381391d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe43400e810 tx=0x7fe43400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:20.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe42effd640 1 -- 172.21.15.46:0/2065015748 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43400c8d0 con 0x7fe4380c6b80 2023-10-30T13:37:20.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe42effd640 1 -- 172.21.15.46:0/2065015748 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43400ca30 con 0x7fe4380c6b80 2023-10-30T13:37:20.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe42effd640 1 -- 172.21.15.46:0/2065015748 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43400cdc0 con 0x7fe4380c6b80 2023-10-30T13:37:20.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe44053e640 1 -- 172.21.15.46:0/2065015748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe438158b70 con 0x7fe4380c6b80 2023-10-30T13:37:20.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe44053e640 1 -- 172.21.15.46:0/2065015748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43813c9f0 con 0x7fe4380c6b80 2023-10-30T13:37:20.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe44053e640 1 -- 172.21.15.46:0/2065015748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe43804e170 con 0x7fe4380c6b80 2023-10-30T13:37:20.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.136+0000 7fe42effd640 1 -- 172.21.15.46:0/2065015748 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe434013410 con 0x7fe4380c6b80 2023-10-30T13:37:20.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.140+0000 7fe42effd640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe41c0664f0 0x7fe41c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.140+0000 7fe42effd640 1 -- 172.21.15.46:0/2065015748 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fe434083f70 con 0x7fe4380c6b80 2023-10-30T13:37:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.140+0000 7fe43dad9640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe41c0664f0 0x7fe41c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.140+0000 7fe43dad9640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe41c0664f0 0x7fe41c0689b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe4280af000 tx=0x7fe4280a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.144+0000 7fe42effd640 1 -- 172.21.15.46:0/2065015748 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe434051120 con 0x7fe4380c6b80 2023-10-30T13:37:20.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.332+0000 7fe44053e640 1 -- 172.21.15.46:0/2065015748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4381572e0 con 0x7fe4380c6b80 2023-10-30T13:37:20.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.332+0000 7fe42effd640 1 -- 172.21.15.46:0/2065015748 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5750 (secure 0 0 0) 0x7fe434054740 con 0x7fe4380c6b80 2023-10-30T13:37:20.340 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:20.340 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:17.076197+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:20.340 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 76 2023-10-30T13:37:20.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.336+0000 7fe44053e640 1 -- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe41c0664f0 msgr2=0x7fe41c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.336+0000 7fe44053e640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe41c0664f0 0x7fe41c0689b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe4280af000 tx=0x7fe4280a7070 comp rx=0 tx=0).stop 2023-10-30T13:37:20.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.336+0000 7fe44053e640 1 -- 172.21.15.46:0/2065015748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380c6b80 msgr2=0x7fe4381391d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.336+0000 7fe44053e640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380c6b80 0x7fe4381391d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe43400e810 tx=0x7fe43400ecd0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.336+0000 7fe44053e640 1 -- 172.21.15.46:0/2065015748 shutdown_connections 2023-10-30T13:37:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.336+0000 7fe44053e640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe41c0664f0 0x7fe41c0689b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.336+0000 7fe44053e640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4380c8430 0x7fe438150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.336+0000 7fe44053e640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4380c6f60 0x7fe438159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.336+0000 7fe44053e640 1 --2- 172.21.15.46:0/2065015748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380c6b80 0x7fe4381391d0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.336+0000 7fe44053e640 1 -- 172.21.15.46:0/2065015748 >> 172.21.15.46:0/2065015748 conn(0x7fe4380c7910 msgr2=0x7fe4380c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.336+0000 7fe44053e640 1 -- 172.21.15.46:0/2065015748 shutdown_connections 2023-10-30T13:37:20.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:20.336+0000 7fe44053e640 1 -- 172.21.15.46:0/2065015748 wait complete. 2023-10-30T13:37:20.361 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 76, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 76, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5574": { "addr": "172.21.15.46:6837/126953235", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 126953235, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 126953235, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5574, "incarnation": 55, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_5721": { "addr": "172.21.15.46:6833/4044310929", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4044310929, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4044310929, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5721, "incarnation": 61, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 24 }, "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:clientreplay", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:37:17.076197+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5721, "mds_1": 5574, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:37:22.362 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:22.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.508+0000 7fc492b3e640 1 Processor -- start 2023-10-30T13:37:22.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc492b3e640 1 -- start start 2023-10-30T13:37:22.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc492b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc48c1538a0 0x7fc48c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc492b3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc48c0ca0e0 0x7fc48c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc492b3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48c0cb800 con 0x7fc48c0cf160 2023-10-30T13:37:22.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc492b3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48c15dc30 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc492b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c0cb330 con 0x7fc48c1538a0 2023-10-30T13:37:22.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc48bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc48c0ca0e0 0x7fc48c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc48bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc48c0ca0e0 0x7fc48c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47328/0 (socket says 172.21.15.46:47328) 2023-10-30T13:37:22.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc48bfff640 1 -- 172.21.15.46:0/1726607754 learned_addr learned my addr 172.21.15.46:0/1726607754 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:22.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc4908da640 1 --2- 172.21.15.46:0/1726607754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc48c1538a0 0x7fc48c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc48bfff640 1 -- 172.21.15.46:0/1726607754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc48c1538a0 msgr2=0x7fc48c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:22.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc48bfff640 1 --2- 172.21.15.46:0/1726607754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc48c1538a0 0x7fc48c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc48bfff640 1 -- 172.21.15.46:0/1726607754 >> v1:172.21.15.154:6789/0 conn(0x7fc48c0cf160 legacy=0x7fc48c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:22.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.512+0000 7fc48bfff640 1 -- 172.21.15.46:0/1726607754 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c0cad70 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc48bfff640 1 --2- 172.21.15.46:0/1726607754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc48c0ca0e0 0x7fc48c15a090 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc478009870 tx=0x7fc47801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39960d8aee4ec98c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:22.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc48b7fe640 1 -- 172.21.15.46:0/1726607754 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4780a8a30 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc48b7fe640 1 -- 172.21.15.46:0/1726607754 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4780a8b90 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc48b7fe640 1 -- 172.21.15.46:0/1726607754 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4780b0440 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc492b3e640 1 -- 172.21.15.46:0/1726607754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc48c0ca0e0 msgr2=0x7fc48c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:22.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc492b3e640 1 --2- 172.21.15.46:0/1726607754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc48c0ca0e0 0x7fc48c15a090 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc478009870 tx=0x7fc47801ed30 comp rx=0 tx=0).stop 2023-10-30T13:37:22.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc492b3e640 1 -- 172.21.15.46:0/1726607754 shutdown_connections 2023-10-30T13:37:22.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc492b3e640 1 --2- 172.21.15.46:0/1726607754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc48c0ca0e0 0x7fc48c15a090 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc492b3e640 1 --2- 172.21.15.46:0/1726607754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc48c1538a0 0x7fc48c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc492b3e640 1 -- 172.21.15.46:0/1726607754 >> 172.21.15.46:0/1726607754 conn(0x7fc48c06bfe0 msgr2=0x7fc48c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:22.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc492b3e640 1 -- 172.21.15.46:0/1726607754 shutdown_connections 2023-10-30T13:37:22.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc492b3e640 1 -- 172.21.15.46:0/1726607754 wait complete. 2023-10-30T13:37:22.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc492b3e640 1 Processor -- start 2023-10-30T13:37:22.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.516+0000 7fc492b3e640 1 -- start start 2023-10-30T13:37:22.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc492b3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc48c0ca0e0 0x7fc48c147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc492b3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc48c0cf160 0x7fc48c15c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc492b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc48c1538a0 0x7fc48c15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc492b3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c1f0c50 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc492b3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c1436f0 con 0x7fc48c0cf160 2023-10-30T13:37:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc492b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c15acf0 con 0x7fc48c1538a0 2023-10-30T13:37:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4908da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc48c0ca0e0 0x7fc48c147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4908da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc48c0ca0e0 0x7fc48c147e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47338/0 (socket says 172.21.15.46:47338) 2023-10-30T13:37:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4908da640 1 -- 172.21.15.46:0/1523767322 learned_addr learned my addr 172.21.15.46:0/1523767322 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:22.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4910db640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc48c1538a0 0x7fc48c15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc48bfff640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc48c0cf160 0x7fc48c15c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4908da640 1 -- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc48c1538a0 msgr2=0x7fc48c15ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:22.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4908da640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc48c1538a0 0x7fc48c15ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4908da640 1 -- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc48c0cf160 msgr2=0x7fc48c15c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:22.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4908da640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc48c0cf160 0x7fc48c15c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4908da640 1 -- 172.21.15.46:0/1523767322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4780094d0 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4908da640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc48c0ca0e0 0x7fc48c147e90 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fc48000d5f0 tx=0x7fc48000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:22.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4897fa640 1 -- 172.21.15.46:0/1523767322 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4800125f0 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4897fa640 1 -- 172.21.15.46:0/1523767322 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc480012750 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc492b3e640 1 -- 172.21.15.46:0/1523767322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc48c15afd0 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc4897fa640 1 -- 172.21.15.46:0/1523767322 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480012ae0 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc492b3e640 1 -- 172.21.15.46:0/1523767322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48c15b590 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.520+0000 7fc492b3e640 1 -- 172.21.15.46:0/1523767322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48c0b2a50 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.524+0000 7fc4897fa640 1 -- 172.21.15.46:0/1523767322 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc480012c40 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.524+0000 7fc4897fa640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc4700664a0 0x7fc470068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.524+0000 7fc4897fa640 1 -- 172.21.15.46:0/1523767322 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fc480084570 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.524+0000 7fc48bfff640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc4700664a0 0x7fc470068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.524+0000 7fc48bfff640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc4700664a0 0x7fc470068960 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc4780b2000 tx=0x7fc478009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.528+0000 7fc4897fa640 1 -- 172.21.15.46:0/1523767322 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc4800516d0 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.716+0000 7fc492b3e640 1 -- 172.21.15.46:0/1523767322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc48c04e170 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.716+0000 7fc4897fa640 1 -- 172.21.15.46:0/1523767322 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5750 (secure 0 0 0) 0x7fc480007380 con 0x7fc48c0ca0e0 2023-10-30T13:37:22.724 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:22.724 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:17.076197+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 76 2023-10-30T13:37:22.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.720+0000 7fc492b3e640 1 -- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc4700664a0 msgr2=0x7fc470068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:22.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.720+0000 7fc492b3e640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc4700664a0 0x7fc470068960 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc4780b2000 tx=0x7fc478009040 comp rx=0 tx=0).stop 2023-10-30T13:37:22.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.720+0000 7fc492b3e640 1 -- 172.21.15.46:0/1523767322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc48c0ca0e0 msgr2=0x7fc48c147e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:22.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.720+0000 7fc492b3e640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc48c0ca0e0 0x7fc48c147e90 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fc48000d5f0 tx=0x7fc48000dab0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.720+0000 7fc492b3e640 1 -- 172.21.15.46:0/1523767322 shutdown_connections 2023-10-30T13:37:22.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.720+0000 7fc492b3e640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc4700664a0 0x7fc470068960 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.720+0000 7fc492b3e640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc48c1538a0 0x7fc48c15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.720+0000 7fc492b3e640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc48c0cf160 0x7fc48c15c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.720+0000 7fc492b3e640 1 --2- 172.21.15.46:0/1523767322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc48c0ca0e0 0x7fc48c147e90 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.720+0000 7fc492b3e640 1 -- 172.21.15.46:0/1523767322 >> 172.21.15.46:0/1523767322 conn(0x7fc48c06bfe0 msgr2=0x7fc48c0cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:22.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.720+0000 7fc492b3e640 1 -- 172.21.15.46:0/1523767322 shutdown_connections 2023-10-30T13:37:22.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:22.720+0000 7fc492b3e640 1 -- 172.21.15.46:0/1523767322 wait complete. 2023-10-30T13:37:22.745 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:24.746 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:24.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe1aaea640 1 Processor -- start 2023-10-30T13:37:24.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe1aaea640 1 -- start start 2023-10-30T13:37:24.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe1aaea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbe141538a0 0x7fbe1415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:24.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe1aaea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe140ca0e0 0x7fbe1415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:24.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe1aaea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe140cb800 con 0x7fbe140cf160 2023-10-30T13:37:24.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe1aaea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe1415dc30 con 0x7fbe141538a0 2023-10-30T13:37:24.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe1aaea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe140cb330 con 0x7fbe140ca0e0 2023-10-30T13:37:24.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe18886640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbe141538a0 0x7fbe1415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe19087640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbe140cf160 0x7fbe140cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48600/0 (socket says 172.21.15.46:48600) 2023-10-30T13:37:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe19087640 1 -- 172.21.15.46:0/887431217 learned_addr learned my addr 172.21.15.46:0/887431217 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:37:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe13fff640 1 --2- 172.21.15.46:0/887431217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe140ca0e0 0x7fbe1415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe18886640 1 -- 172.21.15.46:0/887431217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe140ca0e0 msgr2=0x7fbe1415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe18886640 1 --2- 172.21.15.46:0/887431217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe140ca0e0 0x7fbe1415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:24.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe18886640 1 -- 172.21.15.46:0/887431217 >> v1:172.21.15.46:6789/0 conn(0x7fbe140cf160 legacy=0x7fbe140cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:24.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe137fe640 1 -- 172.21.15.46:0/887431217 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501074000 0 0) 0x7fbe140cb800 con 0x7fbe140cf160 2023-10-30T13:37:24.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.896+0000 7fbe18886640 1 -- 172.21.15.46:0/887431217 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe140cad70 con 0x7fbe141538a0 2023-10-30T13:37:24.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.900+0000 7fbe18886640 1 --2- 172.21.15.46:0/887431217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe141538a0 0x7fbe1415d6f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fbe080097c0 tx=0x7fbe0801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f09f7fbb2c91b57 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:24.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.900+0000 7fbe137fe640 1 -- 172.21.15.46:0/887431217 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe080a8970 con 0x7fbe141538a0 2023-10-30T13:37:24.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.900+0000 7fbe137fe640 1 -- 172.21.15.46:0/887431217 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe080a8ad0 con 0x7fbe141538a0 2023-10-30T13:37:24.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.900+0000 7fbe137fe640 1 -- 172.21.15.46:0/887431217 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe080a8e60 con 0x7fbe141538a0 2023-10-30T13:37:24.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.900+0000 7fbe1aaea640 1 -- 172.21.15.46:0/887431217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe141538a0 msgr2=0x7fbe1415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:24.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.900+0000 7fbe1aaea640 1 --2- 172.21.15.46:0/887431217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe141538a0 0x7fbe1415d6f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fbe080097c0 tx=0x7fbe0801ec70 comp rx=0 tx=0).stop 2023-10-30T13:37:24.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.900+0000 7fbe1aaea640 1 -- 172.21.15.46:0/887431217 shutdown_connections 2023-10-30T13:37:24.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.900+0000 7fbe1aaea640 1 --2- 172.21.15.46:0/887431217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe140ca0e0 0x7fbe1415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:24.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.900+0000 7fbe1aaea640 1 --2- 172.21.15.46:0/887431217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe141538a0 0x7fbe1415d6f0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:24.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.900+0000 7fbe1aaea640 1 -- 172.21.15.46:0/887431217 >> 172.21.15.46:0/887431217 conn(0x7fbe1406bfe0 msgr2=0x7fbe1406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:24.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.900+0000 7fbe1aaea640 1 -- 172.21.15.46:0/887431217 shutdown_connections 2023-10-30T13:37:24.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.900+0000 7fbe1aaea640 1 -- 172.21.15.46:0/887431217 wait complete. 2023-10-30T13:37:24.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe1aaea640 1 Processor -- start 2023-10-30T13:37:24.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe1aaea640 1 -- start start 2023-10-30T13:37:24.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe1aaea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe140ca0e0 0x7fbe14140530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:24.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe1aaea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe140cf160 0x7fbe14146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:24.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe1aaea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe141538a0 0x7fbe14147110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:24.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe1aaea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe140ce8c0 con 0x7fbe141538a0 2023-10-30T13:37:24.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe1aaea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1414eed0 con 0x7fbe140ca0e0 2023-10-30T13:37:24.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe1aaea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe141f3290 con 0x7fbe140cf160 2023-10-30T13:37:24.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe19087640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe141538a0 0x7fbe14147110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:24.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe19087640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe141538a0 0x7fbe14147110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47360/0 (socket says 172.21.15.46:47360) 2023-10-30T13:37:24.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe19087640 1 -- 172.21.15.46:0/147620048 learned_addr learned my addr 172.21.15.46:0/147620048 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:24.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe18886640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe140ca0e0 0x7fbe14140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:24.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe13fff640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe140cf160 0x7fbe14146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:24.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe19087640 1 -- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe140cf160 msgr2=0x7fbe14146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:24.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe19087640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe140cf160 0x7fbe14146bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:24.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe19087640 1 -- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe140ca0e0 msgr2=0x7fbe14140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe19087640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe140ca0e0 0x7fbe14140530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe19087640 1 -- 172.21.15.46:0/147620048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe08009420 con 0x7fbe141538a0 2023-10-30T13:37:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe13fff640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe140cf160 0x7fbe14146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe18886640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe140ca0e0 0x7fbe14140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:24.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe19087640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe141538a0 0x7fbe14147110 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c001570 tx=0x7fbe0c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:24.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe117fa640 1 -- 172.21.15.46:0/147620048 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c00ce40 con 0x7fbe141538a0 2023-10-30T13:37:24.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe117fa640 1 -- 172.21.15.46:0/147620048 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0c00a5e0 con 0x7fbe141538a0 2023-10-30T13:37:24.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe117fa640 1 -- 172.21.15.46:0/147620048 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c00a990 con 0x7fbe141538a0 2023-10-30T13:37:24.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe1aaea640 1 -- 172.21.15.46:0/147620048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe141f2430 con 0x7fbe141538a0 2023-10-30T13:37:24.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.904+0000 7fbe1aaea640 1 -- 172.21.15.46:0/147620048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe141f29f0 con 0x7fbe141538a0 2023-10-30T13:37:24.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.908+0000 7fbe1aaea640 1 -- 172.21.15.46:0/147620048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe140b2a50 con 0x7fbe141538a0 2023-10-30T13:37:24.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.908+0000 7fbe117fa640 1 -- 172.21.15.46:0/147620048 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbe0c00aaf0 con 0x7fbe141538a0 2023-10-30T13:37:24.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.908+0000 7fbe117fa640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbdf80664a0 0x7fbdf8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:24.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.908+0000 7fbe117fa640 1 -- 172.21.15.46:0/147620048 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fbe0c083680 con 0x7fbe141538a0 2023-10-30T13:37:24.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.908+0000 7fbe18886640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbdf80664a0 0x7fbdf8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:24.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.908+0000 7fbe18886640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbdf80664a0 0x7fbdf8068960 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbe080b2000 tx=0x7fbe08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:24.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:24.912+0000 7fbe117fa640 1 -- 172.21.15.46:0/147620048 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbe0c050690 con 0x7fbe141538a0 2023-10-30T13:37:25.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.100+0000 7fbe1aaea640 1 -- 172.21.15.46:0/147620048 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe1404e170 con 0x7fbe141538a0 2023-10-30T13:37:25.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.100+0000 7fbe117fa640 1 -- 172.21.15.46:0/147620048 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5744 (secure 0 0 0) 0x7fbe0c018090 con 0x7fbe141538a0 2023-10-30T13:37:25.108 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:25.109 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:23.243175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:25.109 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 77 2023-10-30T13:37:25.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.104+0000 7fbe1aaea640 1 -- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbdf80664a0 msgr2=0x7fbdf8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:25.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.104+0000 7fbe1aaea640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbdf80664a0 0x7fbdf8068960 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbe080b2000 tx=0x7fbe08008040 comp rx=0 tx=0).stop 2023-10-30T13:37:25.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.104+0000 7fbe1aaea640 1 -- 172.21.15.46:0/147620048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe141538a0 msgr2=0x7fbe14147110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:25.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.104+0000 7fbe1aaea640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe141538a0 0x7fbe14147110 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c001570 tx=0x7fbe0c00cc50 comp rx=0 tx=0).stop 2023-10-30T13:37:25.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.104+0000 7fbe1aaea640 1 -- 172.21.15.46:0/147620048 shutdown_connections 2023-10-30T13:37:25.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.104+0000 7fbe1aaea640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbdf80664a0 0x7fbdf8068960 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:25.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.104+0000 7fbe1aaea640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe141538a0 0x7fbe14147110 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:25.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.104+0000 7fbe1aaea640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe140cf160 0x7fbe14146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:25.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.104+0000 7fbe1aaea640 1 --2- 172.21.15.46:0/147620048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe140ca0e0 0x7fbe14140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:25.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.104+0000 7fbe1aaea640 1 -- 172.21.15.46:0/147620048 >> 172.21.15.46:0/147620048 conn(0x7fbe1406bfe0 msgr2=0x7fbe141527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:25.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.104+0000 7fbe1aaea640 1 -- 172.21.15.46:0/147620048 shutdown_connections 2023-10-30T13:37:25.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:25.104+0000 7fbe1aaea640 1 -- 172.21.15.46:0/147620048 wait complete. 2023-10-30T13:37:25.129 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=2, replacing gid=5442 2023-10-30T13:37:25.130 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.c 2023-10-30T13:37:27.131 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:37:27.133 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:37:27.133 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:37:27.138 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:37:27.138 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:27.239 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-10-30T13:37:27.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.288+0000 7f1ddc432640 1 Processor -- start 2023-10-30T13:37:27.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.288+0000 7f1ddc432640 1 -- start start 2023-10-30T13:37:27.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1ddc432640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd4069e90 0x7f1dd415a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:27.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1ddc432640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1dd406cb80 0x7f1dd406d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:27.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1ddc432640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd406d550 con 0x7f1dd406a270 2023-10-30T13:37:27.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1ddc432640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd40c8570 con 0x7f1dd406cb80 2023-10-30T13:37:27.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1ddc432640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd40c6c10 con 0x7f1dd4069e90 2023-10-30T13:37:27.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1dda9cf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1dd406cb80 0x7f1dd406d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:27.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1dd99cd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1dd406a270 0x7f1dd40c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33644/0 (socket says 172.21.15.46:33644) 2023-10-30T13:37:27.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1dd99cd640 1 -- 172.21.15.46:0/98752370 learned_addr learned my addr 172.21.15.46:0/98752370 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:37:27.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1dda9cf640 1 --2- 172.21.15.46:0/98752370 >> v2:172.21.15.46:3300/0 conn(0x7f1dd406cb80 0x7f1dd406d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54634/0 (socket says 172.21.15.46:54634) 2023-10-30T13:37:27.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1dda1ce640 1 --2- 172.21.15.46:0/98752370 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd4069e90 0x7f1dd415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:27.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1dda9cf640 1 -- 172.21.15.46:0/98752370 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd4069e90 msgr2=0x7f1dd415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:27.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1dda9cf640 1 --2- 172.21.15.46:0/98752370 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd4069e90 0x7f1dd415a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1dda9cf640 1 -- 172.21.15.46:0/98752370 >> v1:172.21.15.46:6789/0 conn(0x7f1dd406a270 legacy=0x7f1dd40c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:27.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1dda9cf640 1 -- 172.21.15.46:0/98752370 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd40c6ef0 con 0x7f1dd406cb80 2023-10-30T13:37:27.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.292+0000 7f1dd91cc640 1 -- 172.21.15.46:0/98752370 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839005139 0 0) 0x7f1dd406d550 con 0x7f1dd406a270 2023-10-30T13:37:27.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1dda9cf640 1 --2- 172.21.15.46:0/98752370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd406cb80 0x7f1dd406d010 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc009d90 tx=0x7f1dcc01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87493fd9970dde2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:27.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1dd91cc640 1 -- 172.21.15.46:0/98752370 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc0a5900 con 0x7f1dd406cb80 2023-10-30T13:37:27.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1dd91cc640 1 -- 172.21.15.46:0/98752370 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dcc0a5a60 con 0x7f1dd406cb80 2023-10-30T13:37:27.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1dd91cc640 1 -- 172.21.15.46:0/98752370 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc0a5df0 con 0x7f1dd406cb80 2023-10-30T13:37:27.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 -- 172.21.15.46:0/98752370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd406cb80 msgr2=0x7f1dd406d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:27.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 --2- 172.21.15.46:0/98752370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd406cb80 0x7f1dd406d010 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc009d90 tx=0x7f1dcc01c950 comp rx=0 tx=0).stop 2023-10-30T13:37:27.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 -- 172.21.15.46:0/98752370 shutdown_connections 2023-10-30T13:37:27.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 --2- 172.21.15.46:0/98752370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd406cb80 0x7f1dd406d010 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 --2- 172.21.15.46:0/98752370 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd4069e90 0x7f1dd415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 -- 172.21.15.46:0/98752370 >> 172.21.15.46:0/98752370 conn(0x7f1dd406dfd0 msgr2=0x7f1dd406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:27.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 -- 172.21.15.46:0/98752370 shutdown_connections 2023-10-30T13:37:27.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 -- 172.21.15.46:0/98752370 wait complete. 2023-10-30T13:37:27.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 Processor -- start 2023-10-30T13:37:27.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 -- start start 2023-10-30T13:37:27.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd4069e90 0x7f1dd41f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:27.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd406a270 0x7f1dd41569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:27.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd406cb80 0x7f1dd4154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:27.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd4071500 con 0x7f1dd406a270 2023-10-30T13:37:27.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd4155b90 con 0x7f1dd4069e90 2023-10-30T13:37:27.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.296+0000 7f1ddc432640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd41540f0 con 0x7f1dd406cb80 2023-10-30T13:37:27.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dd99cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd406a270 0x7f1dd41569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:27.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dd99cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd406a270 0x7f1dd41569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54640/0 (socket says 172.21.15.46:54640) 2023-10-30T13:37:27.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dd99cd640 1 -- 172.21.15.46:0/1183102936 learned_addr learned my addr 172.21.15.46:0/1183102936 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:27.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dda1ce640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd4069e90 0x7f1dd41f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:27.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dda9cf640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd406cb80 0x7f1dd4154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:27.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dd99cd640 1 -- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd406cb80 msgr2=0x7f1dd4154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:27.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dd99cd640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd406cb80 0x7f1dd4154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dd99cd640 1 -- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd4069e90 msgr2=0x7f1dd41f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:27.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dd99cd640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd4069e90 0x7f1dd41f2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dd99cd640 1 -- 172.21.15.46:0/1183102936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc009a70 con 0x7f1dd406a270 2023-10-30T13:37:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dd99cd640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd406a270 0x7f1dd41569b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c520 tx=0x7f1dc400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dcaffd640 1 -- 172.21.15.46:0/1183102936 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc40074e0 con 0x7f1dd406a270 2023-10-30T13:37:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dcaffd640 1 -- 172.21.15.46:0/1183102936 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc4007640 con 0x7f1dd406a270 2023-10-30T13:37:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dcaffd640 1 -- 172.21.15.46:0/1183102936 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc40049f0 con 0x7f1dd406a270 2023-10-30T13:37:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1ddc432640 1 -- 172.21.15.46:0/1183102936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd41543d0 con 0x7f1dd406a270 2023-10-30T13:37:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1ddc432640 1 -- 172.21.15.46:0/1183102936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd4071220 con 0x7f1dd406a270 2023-10-30T13:37:27.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1ddc432640 1 -- 172.21.15.46:0/1183102936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd4154560 con 0x7f1dd406a270 2023-10-30T13:37:27.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.300+0000 7f1dcaffd640 1 -- 172.21.15.46:0/1183102936 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1dc4004b50 con 0x7f1dd406a270 2023-10-30T13:37:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.304+0000 7f1dcaffd640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1db80664a0 0x7f1db8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.304+0000 7f1dcaffd640 1 -- 172.21.15.46:0/1183102936 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f1dc4083340 con 0x7f1dd406a270 2023-10-30T13:37:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.304+0000 7f1dda1ce640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1db80664a0 0x7f1db8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.304+0000 7f1dda1ce640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1db80664a0 0x7f1db8068960 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009b60 tx=0x7f1dd0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.308+0000 7f1dcaffd640 1 -- 172.21.15.46:0/1183102936 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1dc40505c0 con 0x7f1dd406a270 2023-10-30T13:37:27.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 -- 172.21.15.46:0/1183102936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dd404e170 con 0x7f1dd406a270 2023-10-30T13:37:27.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1dcaffd640 1 -- 172.21.15.46:0/1183102936 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f1dc4018090 con 0x7f1dd406a270 2023-10-30T13:37:27.502 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:27.502 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:23.243175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:27.503 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 77 2023-10-30T13:37:27.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 -- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1db80664a0 msgr2=0x7f1db8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:27.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1db80664a0 0x7f1db8068960 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009b60 tx=0x7f1dd0009190 comp rx=0 tx=0).stop 2023-10-30T13:37:27.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 -- 172.21.15.46:0/1183102936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd406a270 msgr2=0x7f1dd41569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:27.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd406a270 0x7f1dd41569b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c520 tx=0x7f1dc400c9e0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 -- 172.21.15.46:0/1183102936 shutdown_connections 2023-10-30T13:37:27.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1db80664a0 0x7f1db8068960 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd406cb80 0x7f1dd4154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd406a270 0x7f1dd41569b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 --2- 172.21.15.46:0/1183102936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd4069e90 0x7f1dd41f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 -- 172.21.15.46:0/1183102936 >> 172.21.15.46:0/1183102936 conn(0x7f1dd406dfd0 msgr2=0x7f1dd414f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:27.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 -- 172.21.15.46:0/1183102936 shutdown_connections 2023-10-30T13:37:27.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:27.496+0000 7f1ddc432640 1 -- 172.21.15.46:0/1183102936 wait complete. 2023-10-30T13:37:27.523 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:37:29.149 INFO:tasks.workunit.client.0.smithi046.stderr:+ trap - EXIT 2023-10-30T13:37:29.150 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ceph 2023-10-30T13:37:29.150 INFO:tasks.workunit.client.0.smithi046.stderr:+ seq 1 90 2023-10-30T13:37:29.152 INFO:tasks.workunit.client.0.smithi046.stderr:+ versions=1 2023-10-30T13:37:29.152 INFO:tasks.workunit.client.0.smithi046.stderr:2 2023-10-30T13:37:29.152 INFO:tasks.workunit.client.0.smithi046.stderr:3 2023-10-30T13:37:29.152 INFO:tasks.workunit.client.0.smithi046.stderr:4 2023-10-30T13:37:29.152 INFO:tasks.workunit.client.0.smithi046.stderr:5 2023-10-30T13:37:29.153 INFO:tasks.workunit.client.0.smithi046.stderr:6 2023-10-30T13:37:29.153 INFO:tasks.workunit.client.0.smithi046.stderr:7 2023-10-30T13:37:29.153 INFO:tasks.workunit.client.0.smithi046.stderr:8 2023-10-30T13:37:29.153 INFO:tasks.workunit.client.0.smithi046.stderr:9 2023-10-30T13:37:29.153 INFO:tasks.workunit.client.0.smithi046.stderr:10 2023-10-30T13:37:29.153 INFO:tasks.workunit.client.0.smithi046.stderr:11 2023-10-30T13:37:29.154 INFO:tasks.workunit.client.0.smithi046.stderr:12 2023-10-30T13:37:29.154 INFO:tasks.workunit.client.0.smithi046.stderr:13 2023-10-30T13:37:29.154 INFO:tasks.workunit.client.0.smithi046.stderr:14 2023-10-30T13:37:29.154 INFO:tasks.workunit.client.0.smithi046.stderr:15 2023-10-30T13:37:29.154 INFO:tasks.workunit.client.0.smithi046.stderr:16 2023-10-30T13:37:29.155 INFO:tasks.workunit.client.0.smithi046.stderr:17 2023-10-30T13:37:29.155 INFO:tasks.workunit.client.0.smithi046.stderr:18 2023-10-30T13:37:29.155 INFO:tasks.workunit.client.0.smithi046.stderr:19 2023-10-30T13:37:29.155 INFO:tasks.workunit.client.0.smithi046.stderr:20 2023-10-30T13:37:29.155 INFO:tasks.workunit.client.0.smithi046.stderr:21 2023-10-30T13:37:29.155 INFO:tasks.workunit.client.0.smithi046.stderr:22 2023-10-30T13:37:29.155 INFO:tasks.workunit.client.0.smithi046.stderr:23 2023-10-30T13:37:29.156 INFO:tasks.workunit.client.0.smithi046.stderr:24 2023-10-30T13:37:29.156 INFO:tasks.workunit.client.0.smithi046.stderr:25 2023-10-30T13:37:29.156 INFO:tasks.workunit.client.0.smithi046.stderr:26 2023-10-30T13:37:29.156 INFO:tasks.workunit.client.0.smithi046.stderr:27 2023-10-30T13:37:29.156 INFO:tasks.workunit.client.0.smithi046.stderr:28 2023-10-30T13:37:29.156 INFO:tasks.workunit.client.0.smithi046.stderr:29 2023-10-30T13:37:29.157 INFO:tasks.workunit.client.0.smithi046.stderr:30 2023-10-30T13:37:29.157 INFO:tasks.workunit.client.0.smithi046.stderr:31 2023-10-30T13:37:29.157 INFO:tasks.workunit.client.0.smithi046.stderr:32 2023-10-30T13:37:29.157 INFO:tasks.workunit.client.0.smithi046.stderr:33 2023-10-30T13:37:29.157 INFO:tasks.workunit.client.0.smithi046.stderr:34 2023-10-30T13:37:29.157 INFO:tasks.workunit.client.0.smithi046.stderr:35 2023-10-30T13:37:29.158 INFO:tasks.workunit.client.0.smithi046.stderr:36 2023-10-30T13:37:29.158 INFO:tasks.workunit.client.0.smithi046.stderr:37 2023-10-30T13:37:29.158 INFO:tasks.workunit.client.0.smithi046.stderr:38 2023-10-30T13:37:29.158 INFO:tasks.workunit.client.0.smithi046.stderr:39 2023-10-30T13:37:29.158 INFO:tasks.workunit.client.0.smithi046.stderr:40 2023-10-30T13:37:29.159 INFO:tasks.workunit.client.0.smithi046.stderr:41 2023-10-30T13:37:29.159 INFO:tasks.workunit.client.0.smithi046.stderr:42 2023-10-30T13:37:29.159 INFO:tasks.workunit.client.0.smithi046.stderr:43 2023-10-30T13:37:29.159 INFO:tasks.workunit.client.0.smithi046.stderr:44 2023-10-30T13:37:29.159 INFO:tasks.workunit.client.0.smithi046.stderr:45 2023-10-30T13:37:29.160 INFO:tasks.workunit.client.0.smithi046.stderr:46 2023-10-30T13:37:29.160 INFO:tasks.workunit.client.0.smithi046.stderr:47 2023-10-30T13:37:29.160 INFO:tasks.workunit.client.0.smithi046.stderr:48 2023-10-30T13:37:29.160 INFO:tasks.workunit.client.0.smithi046.stderr:49 2023-10-30T13:37:29.160 INFO:tasks.workunit.client.0.smithi046.stderr:50 2023-10-30T13:37:29.160 INFO:tasks.workunit.client.0.smithi046.stderr:51 2023-10-30T13:37:29.161 INFO:tasks.workunit.client.0.smithi046.stderr:52 2023-10-30T13:37:29.161 INFO:tasks.workunit.client.0.smithi046.stderr:53 2023-10-30T13:37:29.161 INFO:tasks.workunit.client.0.smithi046.stderr:54 2023-10-30T13:37:29.161 INFO:tasks.workunit.client.0.smithi046.stderr:55 2023-10-30T13:37:29.161 INFO:tasks.workunit.client.0.smithi046.stderr:56 2023-10-30T13:37:29.161 INFO:tasks.workunit.client.0.smithi046.stderr:57 2023-10-30T13:37:29.162 INFO:tasks.workunit.client.0.smithi046.stderr:58 2023-10-30T13:37:29.162 INFO:tasks.workunit.client.0.smithi046.stderr:59 2023-10-30T13:37:29.162 INFO:tasks.workunit.client.0.smithi046.stderr:60 2023-10-30T13:37:29.162 INFO:tasks.workunit.client.0.smithi046.stderr:61 2023-10-30T13:37:29.162 INFO:tasks.workunit.client.0.smithi046.stderr:62 2023-10-30T13:37:29.163 INFO:tasks.workunit.client.0.smithi046.stderr:63 2023-10-30T13:37:29.163 INFO:tasks.workunit.client.0.smithi046.stderr:64 2023-10-30T13:37:29.163 INFO:tasks.workunit.client.0.smithi046.stderr:65 2023-10-30T13:37:29.163 INFO:tasks.workunit.client.0.smithi046.stderr:66 2023-10-30T13:37:29.163 INFO:tasks.workunit.client.0.smithi046.stderr:67 2023-10-30T13:37:29.163 INFO:tasks.workunit.client.0.smithi046.stderr:68 2023-10-30T13:37:29.164 INFO:tasks.workunit.client.0.smithi046.stderr:69 2023-10-30T13:37:29.164 INFO:tasks.workunit.client.0.smithi046.stderr:70 2023-10-30T13:37:29.164 INFO:tasks.workunit.client.0.smithi046.stderr:71 2023-10-30T13:37:29.164 INFO:tasks.workunit.client.0.smithi046.stderr:72 2023-10-30T13:37:29.164 INFO:tasks.workunit.client.0.smithi046.stderr:73 2023-10-30T13:37:29.164 INFO:tasks.workunit.client.0.smithi046.stderr:74 2023-10-30T13:37:29.165 INFO:tasks.workunit.client.0.smithi046.stderr:75 2023-10-30T13:37:29.165 INFO:tasks.workunit.client.0.smithi046.stderr:76 2023-10-30T13:37:29.165 INFO:tasks.workunit.client.0.smithi046.stderr:77 2023-10-30T13:37:29.165 INFO:tasks.workunit.client.0.smithi046.stderr:78 2023-10-30T13:37:29.165 INFO:tasks.workunit.client.0.smithi046.stderr:79 2023-10-30T13:37:29.165 INFO:tasks.workunit.client.0.smithi046.stderr:80 2023-10-30T13:37:29.166 INFO:tasks.workunit.client.0.smithi046.stderr:81 2023-10-30T13:37:29.166 INFO:tasks.workunit.client.0.smithi046.stderr:82 2023-10-30T13:37:29.166 INFO:tasks.workunit.client.0.smithi046.stderr:83 2023-10-30T13:37:29.166 INFO:tasks.workunit.client.0.smithi046.stderr:84 2023-10-30T13:37:29.166 INFO:tasks.workunit.client.0.smithi046.stderr:85 2023-10-30T13:37:29.166 INFO:tasks.workunit.client.0.smithi046.stderr:86 2023-10-30T13:37:29.167 INFO:tasks.workunit.client.0.smithi046.stderr:87 2023-10-30T13:37:29.167 INFO:tasks.workunit.client.0.smithi046.stderr:88 2023-10-30T13:37:29.167 INFO:tasks.workunit.client.0.smithi046.stderr:89 2023-10-30T13:37:29.167 INFO:tasks.workunit.client.0.smithi046.stderr:90 2023-10-30T13:37:29.167 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 1 -eq 48 ] 2023-10-30T13:37:29.168 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.1 2023-10-30T13:37:29.168 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.1 2023-10-30T13:37:29.168 INFO:tasks.workunit.client.0.smithi046.stdout:v0.1 2023-10-30T13:37:29.168 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.1 2023-10-30T13:37:29.524 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:29.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.668+0000 7fde3c217640 1 Processor -- start 2023-10-30T13:37:29.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.668+0000 7fde3c217640 1 -- start start 2023-10-30T13:37:29.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.668+0000 7fde3c217640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde341538a0 0x7fde3415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.668+0000 7fde3c217640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde340ca0e0 0x7fde3415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.668+0000 7fde3c217640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde340cb800 con 0x7fde340cf160 2023-10-30T13:37:29.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.668+0000 7fde3c217640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde3415dc30 con 0x7fde340ca0e0 2023-10-30T13:37:29.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.668+0000 7fde3c217640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde340cb330 con 0x7fde341538a0 2023-10-30T13:37:29.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.668+0000 7fde3a7b4640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fde340cf160 0x7fde340cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33650/0 (socket says 172.21.15.46:33650) 2023-10-30T13:37:29.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.668+0000 7fde3a7b4640 1 -- 172.21.15.46:0/2077385169 learned_addr learned my addr 172.21.15.46:0/2077385169 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:37:29.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.668+0000 7fde397b2640 1 --2- 172.21.15.46:0/2077385169 >> v2:172.21.15.154:3300/0 conn(0x7fde340ca0e0 0x7fde3415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.668+0000 7fde39fb3640 1 --2- 172.21.15.46:0/2077385169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde341538a0 0x7fde3415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.668+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684770143 0 0) 0x7fde340cb800 con 0x7fde340cf160 2023-10-30T13:37:29.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde20003530 con 0x7fde340cf160 2023-10-30T13:37:29.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2525943726 0 0) 0x7fde20003530 con 0x7fde340cf160 2023-10-30T13:37:29.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde341538a0 msgr2=0x7fde3415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:29.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 --2- 172.21.15.46:0/2077385169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde341538a0 0x7fde3415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 >> v2:172.21.15.154:3300/0 conn(0x7fde340ca0e0 msgr2=0x7fde3415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:29.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 --2- 172.21.15.46:0/2077385169 >> v2:172.21.15.154:3300/0 conn(0x7fde340ca0e0 0x7fde3415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde340cad70 con 0x7fde340cf160 2023-10-30T13:37:29.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fde2c001a00 con 0x7fde340cf160 2023-10-30T13:37:29.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 >> v1:172.21.15.46:6789/0 conn(0x7fde340cf160 legacy=0x7fde340cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:29.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 --2- 172.21.15.46:0/2077385169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde20003980 0x7fde20003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 --2- 172.21.15.46:0/2077385169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde341538a0 0x7fde200047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 --2- 172.21.15.46:0/2077385169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde20004d10 0x7fde20005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3415dc30 con 0x7fde20003980 2023-10-30T13:37:29.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde340cb330 con 0x7fde20004d10 2023-10-30T13:37:29.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2000b990 con 0x7fde341538a0 2023-10-30T13:37:29.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde39fb3640 1 --2- 172.21.15.46:0/2077385169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde20003980 0x7fde20003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde397b2640 1 --2- 172.21.15.46:0/2077385169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde341538a0 0x7fde200047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde3a7b4640 1 --2- 172.21.15.46:0/2077385169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde20004d10 0x7fde20005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde39fb3640 1 -- 172.21.15.46:0/2077385169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde341538a0 msgr2=0x7fde200047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:29.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde39fb3640 1 --2- 172.21.15.46:0/2077385169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde341538a0 0x7fde200047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde39fb3640 1 -- 172.21.15.46:0/2077385169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde20004d10 msgr2=0x7fde20005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:29.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde39fb3640 1 --2- 172.21.15.46:0/2077385169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde20004d10 0x7fde20005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde39fb3640 1 -- 172.21.15.46:0/2077385169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde20003530 con 0x7fde20003980 2023-10-30T13:37:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.672+0000 7fde39fb3640 1 --2- 172.21.15.46:0/2077385169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde20003980 0x7fde20003da0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fde30009ae0 tx=0x7fde3001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=190d78a522a1ebb1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde300a5530 con 0x7fde20003980 2023-10-30T13:37:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde300a5690 con 0x7fde20003980 2023-10-30T13:37:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde38fb1640 1 -- 172.21.15.46:0/2077385169 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde300a5a40 con 0x7fde20003980 2023-10-30T13:37:29.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 -- 172.21.15.46:0/2077385169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde20003980 msgr2=0x7fde20003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 --2- 172.21.15.46:0/2077385169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde20003980 0x7fde20003da0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fde30009ae0 tx=0x7fde3001ca50 comp rx=0 tx=0).stop 2023-10-30T13:37:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3a7b4640 1 -- 172.21.15.46:0/2077385169 reap_dead start 2023-10-30T13:37:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 -- 172.21.15.46:0/2077385169 shutdown_connections 2023-10-30T13:37:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 -- 172.21.15.46:0/2077385169 >> 172.21.15.46:0/2077385169 conn(0x7fde3406bfe0 msgr2=0x7fde3406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 -- 172.21.15.46:0/2077385169 shutdown_connections 2023-10-30T13:37:29.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 -- 172.21.15.46:0/2077385169 wait complete. 2023-10-30T13:37:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 Processor -- start 2023-10-30T13:37:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 -- start start 2023-10-30T13:37:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde20003980 0x7fde341efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde34158a60 0x7fde34144390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde34144ea0 0x7fde340b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde34159000 con 0x7fde34158a60 2023-10-30T13:37:29.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde341592c0 con 0x7fde34144ea0 2023-10-30T13:37:29.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde3c217640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde341595a0 con 0x7fde20003980 2023-10-30T13:37:29.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde397b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde34158a60 0x7fde34144390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde397b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde34158a60 0x7fde34144390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54658/0 (socket says 172.21.15.46:54658) 2023-10-30T13:37:29.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.676+0000 7fde397b2640 1 -- 172.21.15.46:0/3604793088 learned_addr learned my addr 172.21.15.46:0/3604793088 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:37:29.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde3a7b4640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde34144ea0 0x7fde340b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde39fb3640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde20003980 0x7fde341efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde397b2640 1 -- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde20003980 msgr2=0x7fde341efcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde397b2640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde20003980 0x7fde341efcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde397b2640 1 -- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde34144ea0 msgr2=0x7fde340b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde397b2640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde34144ea0 0x7fde340b4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde397b2640 1 -- 172.21.15.46:0/3604793088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde341f27f0 con 0x7fde34158a60 2023-10-30T13:37:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde397b2640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde34158a60 0x7fde34144390 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fde24011660 tx=0x7fde24011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde2a7fc640 1 -- 172.21.15.46:0/3604793088 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2400f9c0 con 0x7fde34158a60 2023-10-30T13:37:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde2a7fc640 1 -- 172.21.15.46:0/3604793088 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2400fb20 con 0x7fde34158a60 2023-10-30T13:37:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde3c217640 1 -- 172.21.15.46:0/3604793088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde341f2ad0 con 0x7fde34158a60 2023-10-30T13:37:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde2a7fc640 1 -- 172.21.15.46:0/3604793088 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2400fe90 con 0x7fde34158a60 2023-10-30T13:37:29.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde3c217640 1 -- 172.21.15.46:0/3604793088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde341f3090 con 0x7fde34158a60 2023-10-30T13:37:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde3c217640 1 -- 172.21.15.46:0/3604793088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde340c4070 con 0x7fde34158a60 2023-10-30T13:37:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde2a7fc640 1 -- 172.21.15.46:0/3604793088 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fde24015460 con 0x7fde34158a60 2023-10-30T13:37:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde2a7fc640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde18066470 0x7fde18068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.680+0000 7fde2a7fc640 1 -- 172.21.15.46:0/3604793088 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fde24084ae0 con 0x7fde34158a60 2023-10-30T13:37:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.684+0000 7fde39fb3640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde18066470 0x7fde18068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.684+0000 7fde39fb3640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde18066470 0x7fde18068930 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fde300af000 tx=0x7fde300046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:29.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.684+0000 7fde2a7fc640 1 -- 172.21.15.46:0/3604793088 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fde24051c30 con 0x7fde34158a60 2023-10-30T13:37:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.876+0000 7fde3c217640 1 -- 172.21.15.46:0/3604793088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde3404e170 con 0x7fde34158a60 2023-10-30T13:37:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.876+0000 7fde2a7fc640 1 -- 172.21.15.46:0/3604793088 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+6535 (secure 0 0 0) 0x7fde24055250 con 0x7fde34158a60 2023-10-30T13:37:29.883 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:37:29.883 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:23.243175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:29.883 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 78 2023-10-30T13:37:29.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.876+0000 7fde3c217640 1 -- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde18066470 msgr2=0x7fde18068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:29.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.876+0000 7fde3c217640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde18066470 0x7fde18068930 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fde300af000 tx=0x7fde300046d0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.876+0000 7fde3c217640 1 -- 172.21.15.46:0/3604793088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde34158a60 msgr2=0x7fde34144390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:29.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.876+0000 7fde3c217640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde34158a60 0x7fde34144390 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fde24011660 tx=0x7fde24011b20 comp rx=0 tx=0).stop 2023-10-30T13:37:29.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.876+0000 7fde3c217640 1 -- 172.21.15.46:0/3604793088 shutdown_connections 2023-10-30T13:37:29.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.876+0000 7fde3c217640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde18066470 0x7fde18068930 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.876+0000 7fde3c217640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde34144ea0 0x7fde340b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.876+0000 7fde3c217640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde34158a60 0x7fde34144390 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.876+0000 7fde3c217640 1 --2- 172.21.15.46:0/3604793088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde20003980 0x7fde341efcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.880+0000 7fde3c217640 1 -- 172.21.15.46:0/3604793088 >> 172.21.15.46:0/3604793088 conn(0x7fde3406bfe0 msgr2=0x7fde34151860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:29.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.880+0000 7fde3c217640 1 -- 172.21.15.46:0/3604793088 shutdown_connections 2023-10-30T13:37:29.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:37:29.880+0000 7fde3c217640 1 -- 172.21.15.46:0/3604793088 wait complete. 2023-10-30T13:37:29.904 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:37:29.905 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 83 secs before thrashing 2023-10-30T13:37:47.669 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:37:47.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:17.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:17.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:47.829 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:47.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:52.905 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:38:53.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.043+0000 7fc60b59e640 1 Processor -- start 2023-10-30T13:38:53.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.043+0000 7fc60b59e640 1 -- start start 2023-10-30T13:38:53.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc60b59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc60c0717b0 0x7fc60c15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc60b59e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc60c071b90 0x7fc60c1513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc60b59e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c15b060 con 0x7fc60c0c6290 2023-10-30T13:38:53.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc60b59e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc60c157530 con 0x7fc60c0717b0 2023-10-30T13:38:53.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc60b59e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc60c0c4ab0 con 0x7fc60c071b90 2023-10-30T13:38:53.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc60a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc60c0717b0 0x7fc60c15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc60a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc60c0717b0 0x7fc60c15ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49146/0 (socket says 172.21.15.46:49146) 2023-10-30T13:38:53.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc60a59c640 1 -- 172.21.15.46:0/2510601138 learned_addr learned my addr 172.21.15.46:0/2510601138 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:38:53.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc603fff640 1 --2- 172.21.15.46:0/2510601138 >> v2:172.21.15.154:3300/0 conn(0x7fc60c071b90 0x7fc60c1513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc60a59c640 1 -- 172.21.15.46:0/2510601138 >> v2:172.21.15.154:3300/0 conn(0x7fc60c071b90 msgr2=0x7fc60c1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:53.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc60a59c640 1 --2- 172.21.15.46:0/2510601138 >> v2:172.21.15.154:3300/0 conn(0x7fc60c071b90 0x7fc60c1513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc60a59c640 1 -- 172.21.15.46:0/2510601138 >> v1:172.21.15.154:6789/0 conn(0x7fc60c0c6290 legacy=0x7fc60c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:53.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.047+0000 7fc60a59c640 1 -- 172.21.15.46:0/2510601138 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c0c4d90 con 0x7fc60c0717b0 2023-10-30T13:38:53.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc60a59c640 1 --2- 172.21.15.46:0/2510601138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c0717b0 0x7fc60c15ab20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc60c06b310 tx=0x7fc5fc01ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2efab05a5cb5fd26 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:53.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc609d9b640 1 -- 172.21.15.46:0/2510601138 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc0a89b0 con 0x7fc60c0717b0 2023-10-30T13:38:53.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc609d9b640 1 -- 172.21.15.46:0/2510601138 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5fc0a8b10 con 0x7fc60c0717b0 2023-10-30T13:38:53.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc609d9b640 1 -- 172.21.15.46:0/2510601138 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc0a8ea0 con 0x7fc60c0717b0 2023-10-30T13:38:53.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc60b59e640 1 -- 172.21.15.46:0/2510601138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c0717b0 msgr2=0x7fc60c15ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc60b59e640 1 --2- 172.21.15.46:0/2510601138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c0717b0 0x7fc60c15ab20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc60c06b310 tx=0x7fc5fc01ecb0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc60b59e640 1 -- 172.21.15.46:0/2510601138 shutdown_connections 2023-10-30T13:38:53.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc60b59e640 1 --2- 172.21.15.46:0/2510601138 >> v2:172.21.15.154:3300/0 conn(0x7fc60c071b90 0x7fc60c1513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc60b59e640 1 --2- 172.21.15.46:0/2510601138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c0717b0 0x7fc60c15ab20 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc60b59e640 1 -- 172.21.15.46:0/2510601138 >> 172.21.15.46:0/2510601138 conn(0x7fc60c0c75d0 msgr2=0x7fc60c0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:53.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc60b59e640 1 -- 172.21.15.46:0/2510601138 shutdown_connections 2023-10-30T13:38:53.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc60b59e640 1 -- 172.21.15.46:0/2510601138 wait complete. 2023-10-30T13:38:53.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc60b59e640 1 Processor -- start 2023-10-30T13:38:53.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.051+0000 7fc60b59e640 1 -- start start 2023-10-30T13:38:53.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60b59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc60c0717b0 0x7fc60c1ef520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60b59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc60c071b90 0x7fc60c159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c0c6290 0x7fc60c156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60b59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c14a4b0 con 0x7fc60c0c6290 2023-10-30T13:38:53.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60b59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c149e70 con 0x7fc60c0717b0 2023-10-30T13:38:53.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60b59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c06fc30 con 0x7fc60c071b90 2023-10-30T13:38:53.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c0c6290 0x7fc60c156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c0c6290 0x7fc60c156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49150/0 (socket says 172.21.15.46:49150) 2023-10-30T13:38:53.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60ad9d640 1 -- 172.21.15.46:0/3156905202 learned_addr learned my addr 172.21.15.46:0/3156905202 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:38:53.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60a59c640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc60c0717b0 0x7fc60c1ef520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc603fff640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc60c071b90 0x7fc60c159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60ad9d640 1 -- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc60c071b90 msgr2=0x7fc60c159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60ad9d640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc60c071b90 0x7fc60c159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60ad9d640 1 -- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc60c0717b0 msgr2=0x7fc60c1ef520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60ad9d640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc60c0717b0 0x7fc60c1ef520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60ad9d640 1 -- 172.21.15.46:0/3156905202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc009470 con 0x7fc60c0c6290 2023-10-30T13:38:53.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc603fff640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc60c071b90 0x7fc60c159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:38:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60ad9d640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c0c6290 0x7fc60c156a30 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc60400b9a0 tx=0x7fc60400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc602ffd640 1 -- 172.21.15.46:0/3156905202 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60400cb40 con 0x7fc60c0c6290 2023-10-30T13:38:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc602ffd640 1 -- 172.21.15.46:0/3156905202 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60400cca0 con 0x7fc60c0c6290 2023-10-30T13:38:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc602ffd640 1 -- 172.21.15.46:0/3156905202 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc604014680 con 0x7fc60c0c6290 2023-10-30T13:38:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60b59e640 1 -- 172.21.15.46:0/3156905202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60c1efa60 con 0x7fc60c0c6290 2023-10-30T13:38:53.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60b59e640 1 -- 172.21.15.46:0/3156905202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60c070f50 con 0x7fc60c0c6290 2023-10-30T13:38:53.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc60b59e640 1 -- 172.21.15.46:0/3156905202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60c04e170 con 0x7fc60c0c6290 2023-10-30T13:38:53.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.055+0000 7fc602ffd640 1 -- 172.21.15.46:0/3156905202 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc60401d3b0 con 0x7fc60c0c6290 2023-10-30T13:38:53.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.059+0000 7fc602ffd640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5ec0664a0 0x7fc5ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.059+0000 7fc602ffd640 1 -- 172.21.15.46:0/3156905202 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fc604083ee0 con 0x7fc60c0c6290 2023-10-30T13:38:53.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.059+0000 7fc60a59c640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5ec0664a0 0x7fc5ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.059+0000 7fc60a59c640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5ec0664a0 0x7fc5ec068960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0b2000 tx=0x7fc5fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.063+0000 7fc602ffd640 1 -- 172.21.15.46:0/3156905202 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6040510d0 con 0x7fc60c0c6290 2023-10-30T13:38:53.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.251+0000 7fc60b59e640 1 -- 172.21.15.46:0/3156905202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc60c1576f0 con 0x7fc60c0c6290 2023-10-30T13:38:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.251+0000 7fc602ffd640 1 -- 172.21.15.46:0/3156905202 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fc6040546f0 con 0x7fc60c0c6290 2023-10-30T13:38:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:38:53.255 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:44.121877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:38:53.256 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 80 2023-10-30T13:38:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.255+0000 7fc60b59e640 1 -- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5ec0664a0 msgr2=0x7fc5ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.255+0000 7fc60b59e640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5ec0664a0 0x7fc5ec068960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0b2000 tx=0x7fc5fc009040 comp rx=0 tx=0).stop 2023-10-30T13:38:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.255+0000 7fc60b59e640 1 -- 172.21.15.46:0/3156905202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c0c6290 msgr2=0x7fc60c156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.255+0000 7fc60b59e640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c0c6290 0x7fc60c156a30 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc60400b9a0 tx=0x7fc60400be60 comp rx=0 tx=0).stop 2023-10-30T13:38:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.255+0000 7fc60b59e640 1 -- 172.21.15.46:0/3156905202 shutdown_connections 2023-10-30T13:38:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.255+0000 7fc60b59e640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5ec0664a0 0x7fc5ec068960 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.255+0000 7fc60b59e640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c0c6290 0x7fc60c156a30 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.255+0000 7fc60b59e640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc60c071b90 0x7fc60c159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.255+0000 7fc60b59e640 1 --2- 172.21.15.46:0/3156905202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc60c0717b0 0x7fc60c1ef520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.255+0000 7fc60b59e640 1 -- 172.21.15.46:0/3156905202 >> 172.21.15.46:0/3156905202 conn(0x7fc60c0c75d0 msgr2=0x7fc60c0c9780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.255+0000 7fc60b59e640 1 -- 172.21.15.46:0/3156905202 shutdown_connections 2023-10-30T13:38:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.255+0000 7fc60b59e640 1 -- 172.21.15.46:0/3156905202 wait complete. 2023-10-30T13:38:53.277 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=1) 2023-10-30T13:38:53.277 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:38:53.278 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:38:53.296 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:38:53.297 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:38:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f506adc4640 1 Processor -- start 2023-10-30T13:38:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f506adc4640 1 -- start start 2023-10-30T13:38:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f506adc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50641538a0 0x7f506415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f506adc4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50640cf160 0x7f50640cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f506adc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50640cb800 con 0x7f50640ca0e0 2023-10-30T13:38:53.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f506adc4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506415dc30 con 0x7f50640cf160 2023-10-30T13:38:53.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f506adc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50640cb330 con 0x7f50641538a0 2023-10-30T13:38:53.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f5069361640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50640cf160 0x7f50640cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f5069361640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50640cf160 0x7f50640cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49166/0 (socket says 172.21.15.46:49166) 2023-10-30T13:38:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f5069361640 1 -- 172.21.15.46:0/2386556975 learned_addr learned my addr 172.21.15.46:0/2386556975 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:38:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f5068b60640 1 --2- 172.21.15.46:0/2386556975 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50641538a0 0x7f506415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f5069361640 1 -- 172.21.15.46:0/2386556975 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50641538a0 msgr2=0x7f506415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f5069361640 1 --2- 172.21.15.46:0/2386556975 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50641538a0 0x7f506415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f5069361640 1 -- 172.21.15.46:0/2386556975 >> v1:172.21.15.154:6789/0 conn(0x7f50640ca0e0 legacy=0x7f506415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.479+0000 7f5069361640 1 -- 172.21.15.46:0/2386556975 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50640cad70 con 0x7f50640cf160 2023-10-30T13:38:53.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.483+0000 7f5069361640 1 --2- 172.21.15.46:0/2386556975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50640cf160 0x7f50640cf590 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f505c009e50 tx=0x7f505c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47fa7e9ffd71492 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:53.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.483+0000 7f50637fe640 1 -- 172.21.15.46:0/2386556975 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c0af400 con 0x7f50640cf160 2023-10-30T13:38:53.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.483+0000 7f50637fe640 1 -- 172.21.15.46:0/2386556975 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c0ab050 con 0x7f50640cf160 2023-10-30T13:38:53.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.483+0000 7f50637fe640 1 -- 172.21.15.46:0/2386556975 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c0af780 con 0x7f50640cf160 2023-10-30T13:38:53.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.483+0000 7f506adc4640 1 -- 172.21.15.46:0/2386556975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50640cf160 msgr2=0x7f50640cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.483+0000 7f506adc4640 1 --2- 172.21.15.46:0/2386556975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50640cf160 0x7f50640cf590 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f505c009e50 tx=0x7f505c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.483+0000 7f506adc4640 1 -- 172.21.15.46:0/2386556975 shutdown_connections 2023-10-30T13:38:53.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.483+0000 7f506adc4640 1 --2- 172.21.15.46:0/2386556975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50640cf160 0x7f50640cf590 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.483+0000 7f506adc4640 1 --2- 172.21.15.46:0/2386556975 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50641538a0 0x7f506415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.483+0000 7f506adc4640 1 -- 172.21.15.46:0/2386556975 >> 172.21.15.46:0/2386556975 conn(0x7f506406bfe0 msgr2=0x7f506406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:53.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.483+0000 7f506adc4640 1 -- 172.21.15.46:0/2386556975 shutdown_connections 2023-10-30T13:38:53.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.483+0000 7f506adc4640 1 -- 172.21.15.46:0/2386556975 wait complete. 2023-10-30T13:38:53.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f506adc4640 1 Processor -- start 2023-10-30T13:38:53.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f506adc4640 1 -- start start 2023-10-30T13:38:53.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f506adc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50640ca0e0 0x7f506415a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f506adc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50640cf160 0x7f506415b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f506adc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50641538a0 0x7f506415b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f506adc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50641e3960 con 0x7f50640cf160 2023-10-30T13:38:53.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f506adc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50641e4750 con 0x7f50640ca0e0 2023-10-30T13:38:53.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f506adc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50641e3ba0 con 0x7f50641538a0 2023-10-30T13:38:53.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f5063fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50640cf160 0x7f506415b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f5063fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50640cf160 0x7f506415b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49178/0 (socket says 172.21.15.46:49178) 2023-10-30T13:38:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f5063fff640 1 -- 172.21.15.46:0/3120732580 learned_addr learned my addr 172.21.15.46:0/3120732580 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:38:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f5068b60640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50640ca0e0 0x7f506415a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f5069361640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50641538a0 0x7f506415b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f5063fff640 1 -- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50641538a0 msgr2=0x7f506415b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f5063fff640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50641538a0 0x7f506415b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f5063fff640 1 -- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50640ca0e0 msgr2=0x7f506415a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f5063fff640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50640ca0e0 0x7f506415a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f5063fff640 1 -- 172.21.15.46:0/3120732580 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c009b30 con 0x7f50640cf160 2023-10-30T13:38:53.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f5063fff640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50640cf160 0x7f506415b300 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f505000c500 tx=0x7f505000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:53.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f50617fa640 1 -- 172.21.15.46:0/3120732580 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5050005220 con 0x7f50640cf160 2023-10-30T13:38:53.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f50617fa640 1 -- 172.21.15.46:0/3120732580 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5050005380 con 0x7f50640cf160 2023-10-30T13:38:53.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f50617fa640 1 -- 172.21.15.46:0/3120732580 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5050007270 con 0x7f50640cf160 2023-10-30T13:38:53.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f506adc4640 1 -- 172.21.15.46:0/3120732580 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50641e3430 con 0x7f50640cf160 2023-10-30T13:38:53.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.487+0000 7f506adc4640 1 -- 172.21.15.46:0/3120732580 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50641e3db0 con 0x7f50640cf160 2023-10-30T13:38:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.491+0000 7f506adc4640 1 -- 172.21.15.46:0/3120732580 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50640b2a50 con 0x7f50640cf160 2023-10-30T13:38:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.491+0000 7f50617fa640 1 -- 172.21.15.46:0/3120732580 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f50500073d0 con 0x7f50640cf160 2023-10-30T13:38:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.491+0000 7f50617fa640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50480664a0 0x7f5048068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.491+0000 7f50617fa640 1 -- 172.21.15.46:0/3120732580 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f5050083a60 con 0x7f50640cf160 2023-10-30T13:38:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.491+0000 7f5068b60640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50480664a0 0x7f5048068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:53.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.491+0000 7f5068b60640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50480664a0 0x7f5048068960 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f50580079c0 tx=0x7f5058008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:53.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.495+0000 7f50617fa640 1 -- 172.21.15.46:0/3120732580 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5050050bc0 con 0x7f50640cf160 2023-10-30T13:38:53.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.683+0000 7f506adc4640 1 -- 172.21.15.46:0/3120732580 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f506404e170 con 0x7f50640cf160 2023-10-30T13:38:53.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f50617fa640 1 -- 172.21.15.46:0/3120732580 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f50500541e0 con 0x7f50640cf160 2023-10-30T13:38:53.689 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:38:53.689 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:44.121877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:38:53.689 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 80 2023-10-30T13:38:53.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f506adc4640 1 -- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50480664a0 msgr2=0x7f5048068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f506adc4640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50480664a0 0x7f5048068960 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f50580079c0 tx=0x7f5058008040 comp rx=0 tx=0).stop 2023-10-30T13:38:53.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f506adc4640 1 -- 172.21.15.46:0/3120732580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50640cf160 msgr2=0x7f506415b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:53.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f506adc4640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50640cf160 0x7f506415b300 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f505000c500 tx=0x7f505000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f506adc4640 1 -- 172.21.15.46:0/3120732580 shutdown_connections 2023-10-30T13:38:53.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f506adc4640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50480664a0 0x7f5048068960 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f506adc4640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50641538a0 0x7f506415b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f506adc4640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50640cf160 0x7f506415b300 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f506adc4640 1 --2- 172.21.15.46:0/3120732580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50640ca0e0 0x7f506415a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:53.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f506adc4640 1 -- 172.21.15.46:0/3120732580 >> 172.21.15.46:0/3120732580 conn(0x7f506406bfe0 msgr2=0x7f50640c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:53.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f506adc4640 1 -- 172.21.15.46:0/3120732580 shutdown_connections 2023-10-30T13:38:53.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:53.687+0000 7f506adc4640 1 -- 172.21.15.46:0/3120732580 wait complete. 2023-10-30T13:38:53.711 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:38:55.712 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:38:55.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e62c9640 1 Processor -- start 2023-10-30T13:38:55.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e62c9640 1 -- start start 2023-10-30T13:38:55.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e62c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60e0159e60 0x7f60e006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:55.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e62c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e00bfec0 0x7f60e00c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:55.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e62c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60e006a720 0x7f60e006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:55.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e62c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e006fa80 con 0x7f60e006a720 2023-10-30T13:38:55.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e62c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e0071290 con 0x7f60e0159e60 2023-10-30T13:38:55.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e62c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e0071570 con 0x7f60e00bfec0 2023-10-30T13:38:55.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e4866640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60e006a720 0x7f60e006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:55.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e4866640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60e006a720 0x7f60e006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49186/0 (socket says 172.21.15.46:49186) 2023-10-30T13:38:55.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e4866640 1 -- 172.21.15.46:0/2824304459 learned_addr learned my addr 172.21.15.46:0/2824304459 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:38:55.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60df7fe640 1 --2- 172.21.15.46:0/2824304459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e00bfec0 0x7f60e00c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:55.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60dffff640 1 --2- 172.21.15.46:0/2824304459 >> v2:172.21.15.154:3300/0 conn(0x7f60e0159e60 0x7f60e006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:55.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e4866640 1 -- 172.21.15.46:0/2824304459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e00bfec0 msgr2=0x7f60e00c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:55.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e4866640 1 --2- 172.21.15.46:0/2824304459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e00bfec0 0x7f60e00c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:55.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e4866640 1 -- 172.21.15.46:0/2824304459 >> v2:172.21.15.154:3300/0 conn(0x7f60e0159e60 msgr2=0x7f60e006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:55.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e4866640 1 --2- 172.21.15.46:0/2824304459 >> v2:172.21.15.154:3300/0 conn(0x7f60e0159e60 0x7f60e006f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:55.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.867+0000 7f60e4866640 1 -- 172.21.15.46:0/2824304459 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e0071850 con 0x7f60e006a720 2023-10-30T13:38:55.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60e4866640 1 --2- 172.21.15.46:0/2824304459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60e006a720 0x7f60e006abb0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f60e00c8950 tx=0x7f60d801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f642451498854429 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:55.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60deffd640 1 -- 172.21.15.46:0/2824304459 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d80af400 con 0x7f60e006a720 2023-10-30T13:38:55.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60deffd640 1 -- 172.21.15.46:0/2824304459 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d80ab050 con 0x7f60e006a720 2023-10-30T13:38:55.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60deffd640 1 -- 172.21.15.46:0/2824304459 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d80af780 con 0x7f60e006a720 2023-10-30T13:38:55.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60e62c9640 1 -- 172.21.15.46:0/2824304459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60e006a720 msgr2=0x7f60e006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:55.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60e62c9640 1 --2- 172.21.15.46:0/2824304459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60e006a720 0x7f60e006abb0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f60e00c8950 tx=0x7f60d801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:38:55.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60e62c9640 1 -- 172.21.15.46:0/2824304459 shutdown_connections 2023-10-30T13:38:55.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60e62c9640 1 --2- 172.21.15.46:0/2824304459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60e006a720 0x7f60e006abb0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:55.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60e62c9640 1 --2- 172.21.15.46:0/2824304459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e00bfec0 0x7f60e00c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:55.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60e62c9640 1 --2- 172.21.15.46:0/2824304459 >> v2:172.21.15.154:3300/0 conn(0x7f60e0159e60 0x7f60e006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:55.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60e62c9640 1 -- 172.21.15.46:0/2824304459 >> 172.21.15.46:0/2824304459 conn(0x7f60e00c7910 msgr2=0x7f60e00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:55.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60e62c9640 1 -- 172.21.15.46:0/2824304459 shutdown_connections 2023-10-30T13:38:55.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.871+0000 7f60e62c9640 1 -- 172.21.15.46:0/2824304459 wait complete. 2023-10-30T13:38:55.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60e62c9640 1 Processor -- start 2023-10-30T13:38:55.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60e62c9640 1 -- start start 2023-10-30T13:38:55.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60e62c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60e006a720 0x7f60e014c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:55.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60e62c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e00bfec0 0x7f60e00c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:55.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60e62c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e0159e60 0x7f60e01467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:55.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60e62c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e00c4a10 con 0x7f60e006a720 2023-10-30T13:38:55.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60e62c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e014cf30 con 0x7f60e00bfec0 2023-10-30T13:38:55.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60e62c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e0154a60 con 0x7f60e0159e60 2023-10-30T13:38:55.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60dffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60e006a720 0x7f60e014c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:55.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60dffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60e006a720 0x7f60e014c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49200/0 (socket says 172.21.15.46:49200) 2023-10-30T13:38:55.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60dffff640 1 -- 172.21.15.46:0/972269036 learned_addr learned my addr 172.21.15.46:0/972269036 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:38:55.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60e4866640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e0159e60 0x7f60e01467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:55.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60df7fe640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e00bfec0 0x7f60e00c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:55.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60dffff640 1 -- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e0159e60 msgr2=0x7f60e01467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:55.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60dffff640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e0159e60 0x7f60e01467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60dffff640 1 -- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e00bfec0 msgr2=0x7f60e00c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60dffff640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e00bfec0 0x7f60e00c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60dffff640 1 -- 172.21.15.46:0/972269036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d8009b30 con 0x7f60e006a720 2023-10-30T13:38:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60dffff640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60e006a720 0x7f60e014c9f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f60d000e870 tx=0x7f60d000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.875+0000 7f60dcff9640 1 -- 172.21.15.46:0/972269036 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d0009510 con 0x7f60e006a720 2023-10-30T13:38:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.879+0000 7f60dcff9640 1 -- 172.21.15.46:0/972269036 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d000eec0 con 0x7f60e006a720 2023-10-30T13:38:55.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.879+0000 7f60e62c9640 1 -- 172.21.15.46:0/972269036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e0156710 con 0x7f60e006a720 2023-10-30T13:38:55.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.879+0000 7f60dcff9640 1 -- 172.21.15.46:0/972269036 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d000cb40 con 0x7f60e006a720 2023-10-30T13:38:55.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.879+0000 7f60e62c9640 1 -- 172.21.15.46:0/972269036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e00c4290 con 0x7f60e006a720 2023-10-30T13:38:55.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.879+0000 7f60e62c9640 1 -- 172.21.15.46:0/972269036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e004e170 con 0x7f60e006a720 2023-10-30T13:38:55.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.879+0000 7f60dcff9640 1 -- 172.21.15.46:0/972269036 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60d000cca0 con 0x7f60e006a720 2023-10-30T13:38:55.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.879+0000 7f60dcff9640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60cc0664d0 0x7f60cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:55.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.879+0000 7f60dcff9640 1 -- 172.21.15.46:0/972269036 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f60d0084390 con 0x7f60e006a720 2023-10-30T13:38:55.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.879+0000 7f60df7fe640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60cc0664d0 0x7f60cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:55.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.883+0000 7f60df7fe640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60cc0664d0 0x7f60cc068990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f60e01539e0 tx=0x7f60d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:55.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:55.883+0000 7f60dcff9640 1 -- 172.21.15.46:0/972269036 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60d0051430 con 0x7f60e006a720 2023-10-30T13:38:56.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.071+0000 7f60e62c9640 1 -- 172.21.15.46:0/972269036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60e0069f90 con 0x7f60e006a720 2023-10-30T13:38:56.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60dcff9640 1 -- 172.21.15.46:0/972269036 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f60d0054a50 con 0x7f60e006a720 2023-10-30T13:38:56.077 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:38:56.077 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:44.121877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:38:56.077 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 80 2023-10-30T13:38:56.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60e62c9640 1 -- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60cc0664d0 msgr2=0x7f60cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:56.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60e62c9640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60cc0664d0 0x7f60cc068990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f60e01539e0 tx=0x7f60d4008040 comp rx=0 tx=0).stop 2023-10-30T13:38:56.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60e62c9640 1 -- 172.21.15.46:0/972269036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60e006a720 msgr2=0x7f60e014c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:56.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60e62c9640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60e006a720 0x7f60e014c9f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f60d000e870 tx=0x7f60d000ec70 comp rx=0 tx=0).stop 2023-10-30T13:38:56.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60e62c9640 1 -- 172.21.15.46:0/972269036 shutdown_connections 2023-10-30T13:38:56.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60e62c9640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60cc0664d0 0x7f60cc068990 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60e62c9640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e0159e60 0x7f60e01467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60e62c9640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e00bfec0 0x7f60e00c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60e62c9640 1 --2- 172.21.15.46:0/972269036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60e006a720 0x7f60e014c9f0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:56.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60e62c9640 1 -- 172.21.15.46:0/972269036 >> 172.21.15.46:0/972269036 conn(0x7f60e00c7910 msgr2=0x7f60e014f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:56.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60e62c9640 1 -- 172.21.15.46:0/972269036 shutdown_connections 2023-10-30T13:38:56.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:56.075+0000 7f60e62c9640 1 -- 172.21.15.46:0/972269036 wait complete. 2023-10-30T13:38:56.098 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:38:58.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:38:58.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.251+0000 7f65207a1640 1 Processor -- start 2023-10-30T13:38:58.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.251+0000 7f65207a1640 1 -- start start 2023-10-30T13:38:58.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.251+0000 7f65207a1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6518153d70 0x7f651806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:58.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.251+0000 7f65207a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f651806e470 0x7f65180cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:58.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.251+0000 7f65207a1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651806afa0 con 0x7f65180c8490 2023-10-30T13:38:58.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.251+0000 7f65207a1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f651806d780 con 0x7f6518153d70 2023-10-30T13:38:58.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.251+0000 7f65207a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6518154150 con 0x7f651806e470 2023-10-30T13:38:58.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.251+0000 7f651dd3c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f651806e470 0x7f65180cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:58.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.251+0000 7f651e53d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6518153d70 0x7f651806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:58.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.251+0000 7f651dd3c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f651806e470 0x7f65180cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:57746/0 (socket says 172.21.15.46:57746) 2023-10-30T13:38:58.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.251+0000 7f651dd3c640 1 -- 172.21.15.46:0/3971050100 learned_addr learned my addr 172.21.15.46:0/3971050100 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:38:58.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.255+0000 7f651dd3c640 1 -- 172.21.15.46:0/3971050100 >> v2:172.21.15.154:3300/0 conn(0x7f6518153d70 msgr2=0x7f651806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:58.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.255+0000 7f651dd3c640 1 --2- 172.21.15.46:0/3971050100 >> v2:172.21.15.154:3300/0 conn(0x7f6518153d70 0x7f651806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.255+0000 7f651dd3c640 1 -- 172.21.15.46:0/3971050100 >> v1:172.21.15.154:6789/0 conn(0x7f65180c8490 legacy=0x7f65180c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.255+0000 7f651dd3c640 1 -- 172.21.15.46:0/3971050100 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651806e030 con 0x7f651806e470 2023-10-30T13:38:58.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.255+0000 7f651dd3c640 1 --2- 172.21.15.46:0/3971050100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f651806e470 0x7f65180cce00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f650801c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caa8ced576a66a95 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:58.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.255+0000 7f651d53b640 1 -- 172.21.15.46:0/3971050100 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65080aa670 con 0x7f651806e470 2023-10-30T13:38:58.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.255+0000 7f651d53b640 1 -- 172.21.15.46:0/3971050100 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65080aa7d0 con 0x7f651806e470 2023-10-30T13:38:58.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.255+0000 7f651d53b640 1 -- 172.21.15.46:0/3971050100 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65080aab60 con 0x7f651806e470 2023-10-30T13:38:58.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 -- 172.21.15.46:0/3971050100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f651806e470 msgr2=0x7f65180cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 --2- 172.21.15.46:0/3971050100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f651806e470 0x7f65180cce00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f650801c510 comp rx=0 tx=0).stop 2023-10-30T13:38:58.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 -- 172.21.15.46:0/3971050100 shutdown_connections 2023-10-30T13:38:58.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 --2- 172.21.15.46:0/3971050100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f651806e470 0x7f65180cce00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 --2- 172.21.15.46:0/3971050100 >> v2:172.21.15.154:3300/0 conn(0x7f6518153d70 0x7f651806ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 -- 172.21.15.46:0/3971050100 >> 172.21.15.46:0/3971050100 conn(0x7f6518071ea0 msgr2=0x7f6518065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:58.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 -- 172.21.15.46:0/3971050100 shutdown_connections 2023-10-30T13:38:58.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 -- 172.21.15.46:0/3971050100 wait complete. 2023-10-30T13:38:58.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 Processor -- start 2023-10-30T13:38:58.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 -- start start 2023-10-30T13:38:58.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f651806e470 0x7f65180c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:58.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65180c8490 0x7f651806f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:58.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6518153d70 0x7f6518147700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:58.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65180af870 con 0x7f651806e470 2023-10-30T13:38:58.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65180c2010 con 0x7f65180c8490 2023-10-30T13:38:58.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f65207a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65180c22f0 con 0x7f6518153d70 2023-10-30T13:38:58.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f651e53d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f651806e470 0x7f65180c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:58.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f651e53d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f651806e470 0x7f65180c30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39470/0 (socket says 172.21.15.46:39470) 2023-10-30T13:38:58.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f651e53d640 1 -- 172.21.15.46:0/1885581977 learned_addr learned my addr 172.21.15.46:0/1885581977 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:38:58.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f651dd3c640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65180c8490 0x7f651806f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:58.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.259+0000 7f651ed3e640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6518153d70 0x7f6518147700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:58.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f651e53d640 1 -- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6518153d70 msgr2=0x7f6518147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f651e53d640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6518153d70 0x7f6518147700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f651e53d640 1 -- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65180c8490 msgr2=0x7f651806f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f651e53d640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65180c8490 0x7f651806f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f651e53d640 1 -- 172.21.15.46:0/1885581977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65080094d0 con 0x7f651806e470 2023-10-30T13:38:58.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f651ed3e640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6518153d70 0x7f6518147700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:38:58.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f651e53d640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f651806e470 0x7f65180c30b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f651400e870 tx=0x7f651400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:58.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f650effd640 1 -- 172.21.15.46:0/1885581977 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6514009510 con 0x7f651806e470 2023-10-30T13:38:58.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f650effd640 1 -- 172.21.15.46:0/1885581977 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651400eec0 con 0x7f651806e470 2023-10-30T13:38:58.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f65207a1640 1 -- 172.21.15.46:0/1885581977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65180c25d0 con 0x7f651806e470 2023-10-30T13:38:58.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f650effd640 1 -- 172.21.15.46:0/1885581977 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651400cb40 con 0x7f651806e470 2023-10-30T13:38:58.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f65207a1640 1 -- 172.21.15.46:0/1885581977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65180c2b90 con 0x7f651806e470 2023-10-30T13:38:58.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f65207a1640 1 -- 172.21.15.46:0/1885581977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65180c35f0 con 0x7f651806e470 2023-10-30T13:38:58.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f650effd640 1 -- 172.21.15.46:0/1885581977 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f651400cca0 con 0x7f651806e470 2023-10-30T13:38:58.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f650effd640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64fc0664a0 0x7f64fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:58.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f650effd640 1 -- 172.21.15.46:0/1885581977 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f6514084350 con 0x7f651806e470 2023-10-30T13:38:58.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.263+0000 7f651dd3c640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64fc0664a0 0x7f64fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:58.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.267+0000 7f651dd3c640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64fc0664a0 0x7f64fc068960 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f65080ae000 tx=0x7f65080a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:58.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.267+0000 7f650effd640 1 -- 172.21.15.46:0/1885581977 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6514051480 con 0x7f651806e470 2023-10-30T13:38:58.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.459+0000 7f65207a1640 1 -- 172.21.15.46:0/1885581977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f651806dd20 con 0x7f651806e470 2023-10-30T13:38:58.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.459+0000 7f650effd640 1 -- 172.21.15.46:0/1885581977 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f6514054aa0 con 0x7f651806e470 2023-10-30T13:38:58.463 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:38:58.463 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:44.121877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:38:58.464 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 80 2023-10-30T13:38:58.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.463+0000 7f65207a1640 1 -- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64fc0664a0 msgr2=0x7f64fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.463+0000 7f65207a1640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64fc0664a0 0x7f64fc068960 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f65080ae000 tx=0x7f65080a7000 comp rx=0 tx=0).stop 2023-10-30T13:38:58.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.463+0000 7f65207a1640 1 -- 172.21.15.46:0/1885581977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f651806e470 msgr2=0x7f65180c30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:58.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.463+0000 7f65207a1640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f651806e470 0x7f65180c30b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f651400e870 tx=0x7f651400ec70 comp rx=0 tx=0).stop 2023-10-30T13:38:58.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.463+0000 7f65207a1640 1 -- 172.21.15.46:0/1885581977 shutdown_connections 2023-10-30T13:38:58.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.463+0000 7f65207a1640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64fc0664a0 0x7f64fc068960 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.463+0000 7f65207a1640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6518153d70 0x7f6518147700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.463+0000 7f65207a1640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65180c8490 0x7f651806f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.463+0000 7f65207a1640 1 --2- 172.21.15.46:0/1885581977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f651806e470 0x7f65180c30b0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:58.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.463+0000 7f65207a1640 1 -- 172.21.15.46:0/1885581977 >> 172.21.15.46:0/1885581977 conn(0x7f6518071ea0 msgr2=0x7f651814dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:58.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.463+0000 7f65207a1640 1 -- 172.21.15.46:0/1885581977 shutdown_connections 2023-10-30T13:38:58.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:38:58.463+0000 7f65207a1640 1 -- 172.21.15.46:0/1885581977 wait complete. 2023-10-30T13:38:58.484 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:39:00.485 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:00.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.635+0000 7fa803c46640 1 Processor -- start 2023-10-30T13:39:00.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.635+0000 7fa803c46640 1 -- start start 2023-10-30T13:39:00.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa803c46640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa7fc06d290 0x7fa7fc06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:00.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa803c46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc0c6c10 con 0x7fa7fc069e90 2023-10-30T13:39:00.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa803c46640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc0c6ef0 con 0x7fa7fc15a410 2023-10-30T13:39:00.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa803c46640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7fc0c8570 con 0x7fa7fc06d290 2023-10-30T13:39:00.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8019e2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa7fc069e90 0x7fa7fc0c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35826/0 (socket says 172.21.15.46:35826) 2023-10-30T13:39:00.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8019e2640 1 -- 172.21.15.46:0/114437498 learned_addr learned my addr 172.21.15.46:0/114437498 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:39:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8021e3640 1 --2- 172.21.15.46:0/114437498 >> v2:172.21.15.154:3300/0 conn(0x7fa7fc06d290 0x7fa7fc06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193072515 0 0) 0x7fa7fc0c6c10 con 0x7fa7fc069e90 2023-10-30T13:39:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e8003530 con 0x7fa7fc069e90 2023-10-30T13:39:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3542417402 0 0) 0x7fa7e8003530 con 0x7fa7fc069e90 2023-10-30T13:39:00.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 >> v2:172.21.15.154:3300/0 conn(0x7fa7fc06d290 msgr2=0x7fa7fc06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:00.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 --2- 172.21.15.46:0/114437498 >> v2:172.21.15.154:3300/0 conn(0x7fa7fc06d290 0x7fa7fc06a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:00.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 >> v1:172.21.15.154:6789/0 conn(0x7fa7fc15a410 legacy=0x7fa7fc06cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:00.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc0c71d0 con 0x7fa7fc069e90 2023-10-30T13:39:00.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fa7f8001680 con 0x7fa7fc069e90 2023-10-30T13:39:00.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 >> v1:172.21.15.46:6789/0 conn(0x7fa7fc069e90 legacy=0x7fa7fc0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:00.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 --2- 172.21.15.46:0/114437498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e8003980 0x7fa7e8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:00.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 --2- 172.21.15.46:0/114437498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7e80047d0 0x7fa7e8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:00.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 --2- 172.21.15.46:0/114437498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e8005190 0x7fa7e800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:00.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc0c8570 con 0x7fa7e8003980 2023-10-30T13:39:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e800c030 con 0x7fa7e8005190 2023-10-30T13:39:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e800c1f0 con 0x7fa7e80047d0 2023-10-30T13:39:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8019e2640 1 --2- 172.21.15.46:0/114437498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e8003980 0x7fa7e8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8021e3640 1 --2- 172.21.15.46:0/114437498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e8005190 0x7fa7e800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:00.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8011e1640 1 --2- 172.21.15.46:0/114437498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7e80047d0 0x7fa7e8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:00.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8019e2640 1 -- 172.21.15.46:0/114437498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7e80047d0 msgr2=0x7fa7e8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:00.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8019e2640 1 --2- 172.21.15.46:0/114437498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7e80047d0 0x7fa7e8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:00.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8019e2640 1 -- 172.21.15.46:0/114437498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e8005190 msgr2=0x7fa7e800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:00.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8019e2640 1 --2- 172.21.15.46:0/114437498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e8005190 0x7fa7e800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:00.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8019e2640 1 -- 172.21.15.46:0/114437498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e8003530 con 0x7fa7e8003980 2023-10-30T13:39:00.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8021e3640 1 --2- 172.21.15.46:0/114437498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e8005190 0x7fa7e800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:00.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.639+0000 7fa8021e3640 1 -- 172.21.15.46:0/114437498 reap_dead start 2023-10-30T13:39:00.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.643+0000 7fa8019e2640 1 --2- 172.21.15.46:0/114437498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e8003980 0x7fa7e8003da0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800c240 tx=0x7fa7f801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57da12097c4573f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.643+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8005ae0 con 0x7fa7e8003980 2023-10-30T13:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.643+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f8005c60 con 0x7fa7e8003980 2023-10-30T13:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.643+0000 7fa8009e0640 1 -- 172.21.15.46:0/114437498 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8007070 con 0x7fa7e8003980 2023-10-30T13:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.643+0000 7fa803c46640 1 -- 172.21.15.46:0/114437498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e8003980 msgr2=0x7fa7e8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:00.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.643+0000 7fa803c46640 1 --2- 172.21.15.46:0/114437498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e8003980 0x7fa7e8003da0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800c240 tx=0x7fa7f801cc00 comp rx=0 tx=0).stop 2023-10-30T13:39:00.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.643+0000 7fa803c46640 1 -- 172.21.15.46:0/114437498 shutdown_connections 2023-10-30T13:39:00.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.643+0000 7fa803c46640 1 --2- 172.21.15.46:0/114437498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7e8003980 0x7fa7e8003da0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:00.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.643+0000 7fa803c46640 1 -- 172.21.15.46:0/114437498 >> 172.21.15.46:0/114437498 conn(0x7fa7fc06dfd0 msgr2=0x7fa7fc06e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:00.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.643+0000 7fa803c46640 1 -- 172.21.15.46:0/114437498 shutdown_connections 2023-10-30T13:39:00.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.643+0000 7fa803c46640 1 -- 172.21.15.46:0/114437498 wait complete. 2023-10-30T13:39:00.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa803c46640 1 Processor -- start 2023-10-30T13:39:00.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa803c46640 1 -- start start 2023-10-30T13:39:00.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa803c46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7e8003980 0x7fa7fc1519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:00.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa803c46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7fc154830 0x7fa7fc1f5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:00.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa803c46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7fc1f5b60 0x7fa7fc1586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:00.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa803c46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc152420 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa803c46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc0c4640 con 0x7fa7fc154830 2023-10-30T13:39:00.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa803c46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc0c59f0 con 0x7fa7e8003980 2023-10-30T13:39:00.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa8021e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7fc1f5b60 0x7fa7fc1586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:00.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa8021e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7fc1f5b60 0x7fa7fc1586c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39484/0 (socket says 172.21.15.46:39484) 2023-10-30T13:39:00.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa8021e3640 1 -- 172.21.15.46:0/1556097332 learned_addr learned my addr 172.21.15.46:0/1556097332 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:00.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa8011e1640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7fc154830 0x7fa7fc1f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:00.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa8019e2640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7e8003980 0x7fa7fc1519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:00.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa8021e3640 1 -- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7e8003980 msgr2=0x7fa7fc1519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:00.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa8021e3640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7e8003980 0x7fa7fc1519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:00.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa8021e3640 1 -- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7fc154830 msgr2=0x7fa7fc1f5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:00.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa8021e3640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7fc154830 0x7fa7fc1f5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:00.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa8021e3640 1 -- 172.21.15.46:0/1556097332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc151f10 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa8011e1640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7fc154830 0x7fa7fc1f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:00.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa8021e3640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7fc1f5b60 0x7fa7fc1586c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc06ef00 tx=0x7fa7f4010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:00.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa7f27fc640 1 -- 172.21.15.46:0/1556097332 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f400e050 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa7f27fc640 1 -- 172.21.15.46:0/1556097332 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f400f040 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa803c46640 1 -- 172.21.15.46:0/1556097332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc1f5fe0 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa7f27fc640 1 -- 172.21.15.46:0/1556097332 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4007ee0 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa803c46640 1 -- 172.21.15.46:0/1556097332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc0c50c0 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.647+0000 7fa803c46640 1 -- 172.21.15.46:0/1556097332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc04e170 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.651+0000 7fa7f27fc640 1 -- 172.21.15.46:0/1556097332 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7f4015440 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.651+0000 7fa7f27fc640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7e00664a0 0x7fa7e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:00.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.651+0000 7fa7f27fc640 1 -- 172.21.15.46:0/1556097332 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fa7f40843f0 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.651+0000 7fa8019e2640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7e00664a0 0x7fa7e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:00.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.651+0000 7fa8019e2640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7e00664a0 0x7fa7e0068960 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80ab000 tx=0x7fa7f80b1040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:00.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.655+0000 7fa7f27fc640 1 -- 172.21.15.46:0/1556097332 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7f4051520 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.843+0000 7fa803c46640 1 -- 172.21.15.46:0/1556097332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7fc06ef60 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa7f27fc640 1 -- 172.21.15.46:0/1556097332 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fa7f4054b40 con 0x7fa7fc1f5b60 2023-10-30T13:39:00.849 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:00.849 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:44.121877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:00.849 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 80 2023-10-30T13:39:00.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa803c46640 1 -- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7e00664a0 msgr2=0x7fa7e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:00.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa803c46640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7e00664a0 0x7fa7e0068960 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80ab000 tx=0x7fa7f80b1040 comp rx=0 tx=0).stop 2023-10-30T13:39:00.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa803c46640 1 -- 172.21.15.46:0/1556097332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7fc1f5b60 msgr2=0x7fa7fc1586c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:00.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa803c46640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7fc1f5b60 0x7fa7fc1586c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc06ef00 tx=0x7fa7f4010590 comp rx=0 tx=0).stop 2023-10-30T13:39:00.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa803c46640 1 -- 172.21.15.46:0/1556097332 shutdown_connections 2023-10-30T13:39:00.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa803c46640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7e00664a0 0x7fa7e0068960 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:00.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa803c46640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7fc1f5b60 0x7fa7fc1586c0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:00.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa803c46640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7fc154830 0x7fa7fc1f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:00.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa803c46640 1 --2- 172.21.15.46:0/1556097332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7e8003980 0x7fa7fc1519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:00.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa803c46640 1 -- 172.21.15.46:0/1556097332 >> 172.21.15.46:0/1556097332 conn(0x7fa7fc06dfd0 msgr2=0x7fa7fc06e570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:00.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa803c46640 1 -- 172.21.15.46:0/1556097332 shutdown_connections 2023-10-30T13:39:00.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:00.847+0000 7fa803c46640 1 -- 172.21.15.46:0/1556097332 wait complete. 2023-10-30T13:39:00.870 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:39:02.871 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.023+0000 7fdb3b1c4640 1 Processor -- start 2023-10-30T13:39:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.023+0000 7fdb3b1c4640 1 -- start start 2023-10-30T13:39:03.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.023+0000 7fdb3b1c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb340c6bd0 0x7fdb3406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb3b1c4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb34066cf0 con 0x7fdb34071110 2023-10-30T13:39:03.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb3b1c4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb340c70d0 con 0x7fdb3406f7a0 2023-10-30T13:39:03.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb3b1c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb340700c0 con 0x7fdb340c6bd0 2023-10-30T13:39:03.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb39761640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdb34071110 0x7fdb34158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35838/0 (socket says 172.21.15.46:35838) 2023-10-30T13:39:03.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb39761640 1 -- 172.21.15.46:0/1654335536 learned_addr learned my addr 172.21.15.46:0/1654335536 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:39:03.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb33fff640 1 --2- 172.21.15.46:0/1654335536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb340c6bd0 0x7fdb3406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991209364 0 0) 0x7fdb34066cf0 con 0x7fdb34071110 2023-10-30T13:39:03.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb24003530 con 0x7fdb34071110 2023-10-30T13:39:03.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4051881726 0 0) 0x7fdb24003530 con 0x7fdb34071110 2023-10-30T13:39:03.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb340c6bd0 msgr2=0x7fdb3406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:03.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 --2- 172.21.15.46:0/1654335536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb340c6bd0 0x7fdb3406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 >> v1:172.21.15.154:6789/0 conn(0x7fdb3406f7a0 legacy=0x7fdb34069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb340c7390 con 0x7fdb34071110 2023-10-30T13:39:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fdb2c001a00 con 0x7fdb34071110 2023-10-30T13:39:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 >> v1:172.21.15.46:6789/0 conn(0x7fdb34071110 legacy=0x7fdb34158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 --2- 172.21.15.46:0/1654335536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb240039b0 0x7fdb24003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 --2- 172.21.15.46:0/1654335536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb24004360 0x7fdb24004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 --2- 172.21.15.46:0/1654335536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb340c6bd0 0x7fdb24005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb340700c0 con 0x7fdb240039b0 2023-10-30T13:39:03.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2400b990 con 0x7fdb24004360 2023-10-30T13:39:03.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2400bb50 con 0x7fdb340c6bd0 2023-10-30T13:39:03.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb38f60640 1 --2- 172.21.15.46:0/1654335536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb240039b0 0x7fdb24003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb39761640 1 --2- 172.21.15.46:0/1654335536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb340c6bd0 0x7fdb24005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb33fff640 1 --2- 172.21.15.46:0/1654335536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb24004360 0x7fdb24004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb38f60640 1 -- 172.21.15.46:0/1654335536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb340c6bd0 msgr2=0x7fdb24005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:03.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb38f60640 1 --2- 172.21.15.46:0/1654335536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb340c6bd0 0x7fdb24005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb38f60640 1 -- 172.21.15.46:0/1654335536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb24004360 msgr2=0x7fdb24004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:03.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb38f60640 1 --2- 172.21.15.46:0/1654335536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb24004360 0x7fdb24004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.027+0000 7fdb38f60640 1 -- 172.21.15.46:0/1654335536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb24003530 con 0x7fdb240039b0 2023-10-30T13:39:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb38f60640 1 --2- 172.21.15.46:0/1654335536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb240039b0 0x7fdb24003d90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fdb28001910 tx=0x7fdb2801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2945b4d699d7f36 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb280a88f0 con 0x7fdb240039b0 2023-10-30T13:39:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb280a8a70 con 0x7fdb240039b0 2023-10-30T13:39:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb337fe640 1 -- 172.21.15.46:0/1654335536 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb280a8e20 con 0x7fdb240039b0 2023-10-30T13:39:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/1654335536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb240039b0 msgr2=0x7fdb24003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb3b1c4640 1 --2- 172.21.15.46:0/1654335536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb240039b0 0x7fdb24003d90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fdb28001910 tx=0x7fdb2801ecd0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb39761640 1 -- 172.21.15.46:0/1654335536 reap_dead start 2023-10-30T13:39:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/1654335536 shutdown_connections 2023-10-30T13:39:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/1654335536 >> 172.21.15.46:0/1654335536 conn(0x7fdb340c7910 msgr2=0x7fdb340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/1654335536 shutdown_connections 2023-10-30T13:39:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/1654335536 wait complete. 2023-10-30T13:39:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb3b1c4640 1 Processor -- start 2023-10-30T13:39:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.031+0000 7fdb3b1c4640 1 -- start start 2023-10-30T13:39:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb3b1c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb240039b0 0x7fdb3406b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb3b1c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb3414d930 0x7fdb3414fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb3b1c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb3414a2f0 0x7fdb341503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb3b1c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3414fa00 con 0x7fdb3414d930 2023-10-30T13:39:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb3b1c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3414a930 con 0x7fdb3414a2f0 2023-10-30T13:39:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb3b1c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb341f0830 con 0x7fdb240039b0 2023-10-30T13:39:03.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb33fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb3414d930 0x7fdb3414fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb33fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb3414d930 0x7fdb3414fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39504/0 (socket says 172.21.15.46:39504) 2023-10-30T13:39:03.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb33fff640 1 -- 172.21.15.46:0/3584576511 learned_addr learned my addr 172.21.15.46:0/3584576511 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:03.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb38f60640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb240039b0 0x7fdb3406b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb39761640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb3414a2f0 0x7fdb341503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb33fff640 1 -- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb240039b0 msgr2=0x7fdb3406b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb33fff640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb240039b0 0x7fdb3406b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb33fff640 1 -- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb3414a2f0 msgr2=0x7fdb341503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb33fff640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb3414a2f0 0x7fdb341503d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb33fff640 1 -- 172.21.15.46:0/3584576511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb341f0b10 con 0x7fdb3414d930 2023-10-30T13:39:03.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb33fff640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb3414d930 0x7fdb3414fe90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fdb200115e0 tx=0x7fdb20011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:03.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb317fa640 1 -- 172.21.15.46:0/3584576511 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2000f8f0 con 0x7fdb3414d930 2023-10-30T13:39:03.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb317fa640 1 -- 172.21.15.46:0/3584576511 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2000fa50 con 0x7fdb3414d930 2023-10-30T13:39:03.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/3584576511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3414f0d0 con 0x7fdb3414d930 2023-10-30T13:39:03.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb317fa640 1 -- 172.21.15.46:0/3584576511 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2000fdc0 con 0x7fdb3414d930 2023-10-30T13:39:03.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/3584576511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3414f690 con 0x7fdb3414d930 2023-10-30T13:39:03.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/3584576511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3404e170 con 0x7fdb3414d930 2023-10-30T13:39:03.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.035+0000 7fdb317fa640 1 -- 172.21.15.46:0/3584576511 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdb20015460 con 0x7fdb3414d930 2023-10-30T13:39:03.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.039+0000 7fdb317fa640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdb180664d0 0x7fdb18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:03.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.039+0000 7fdb317fa640 1 -- 172.21.15.46:0/3584576511 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fdb20084a20 con 0x7fdb3414d930 2023-10-30T13:39:03.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.039+0000 7fdb38f60640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdb180664d0 0x7fdb18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:03.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.039+0000 7fdb38f60640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdb180664d0 0x7fdb18068990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdb28001910 tx=0x7fdb28001330 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.043+0000 7fdb317fa640 1 -- 172.21.15.46:0/3584576511 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdb20051ac0 con 0x7fdb3414d930 2023-10-30T13:39:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.231+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/3584576511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb3406a5e0 con 0x7fdb3414d930 2023-10-30T13:39:03.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.231+0000 7fdb317fa640 1 -- 172.21.15.46:0/3584576511 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fdb200550e0 con 0x7fdb3414d930 2023-10-30T13:39:03.236 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:03.236 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:44.121877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:03.237 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 80 2023-10-30T13:39:03.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.235+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdb180664d0 msgr2=0x7fdb18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.235+0000 7fdb3b1c4640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdb180664d0 0x7fdb18068990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdb28001910 tx=0x7fdb28001330 comp rx=0 tx=0).stop 2023-10-30T13:39:03.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.235+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/3584576511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb3414d930 msgr2=0x7fdb3414fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:03.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.235+0000 7fdb3b1c4640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb3414d930 0x7fdb3414fe90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fdb200115e0 tx=0x7fdb20011aa0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.235+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/3584576511 shutdown_connections 2023-10-30T13:39:03.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.235+0000 7fdb3b1c4640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdb180664d0 0x7fdb18068990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.235+0000 7fdb3b1c4640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb3414a2f0 0x7fdb341503d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.235+0000 7fdb3b1c4640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb3414d930 0x7fdb3414fe90 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.235+0000 7fdb3b1c4640 1 --2- 172.21.15.46:0/3584576511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb240039b0 0x7fdb3406b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:03.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.235+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/3584576511 >> 172.21.15.46:0/3584576511 conn(0x7fdb340c7910 msgr2=0x7fdb340ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:03.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.235+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/3584576511 shutdown_connections 2023-10-30T13:39:03.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:03.235+0000 7fdb3b1c4640 1 -- 172.21.15.46:0/3584576511 wait complete. 2023-10-30T13:39:03.257 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:39:05.259 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:05.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e2e54640 1 Processor -- start 2023-10-30T13:39:05.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e2e54640 1 -- start start 2023-10-30T13:39:05.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e2e54640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff6dc15d6f0 0x7ff6dc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e2e54640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6dc0cf160 0x7ff6dc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e2e54640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6dc0cb800 con 0x7ff6dc1538a0 2023-10-30T13:39:05.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e2e54640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6dc0cb330 con 0x7ff6dc0cf160 2023-10-30T13:39:05.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e2e54640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6dc0cad70 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e13f1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6dc0cf160 0x7ff6dc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e13f1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6dc0cf160 0x7ff6dc0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39508/0 (socket says 172.21.15.46:39508) 2023-10-30T13:39:05.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e13f1640 1 -- 172.21.15.46:0/3695944431 learned_addr learned my addr 172.21.15.46:0/3695944431 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:05.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6dbfff640 1 --2- 172.21.15.46:0/3695944431 >> v2:172.21.15.154:3300/0 conn(0x7ff6dc15d6f0 0x7ff6dc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e13f1640 1 -- 172.21.15.46:0/3695944431 >> v2:172.21.15.154:3300/0 conn(0x7ff6dc15d6f0 msgr2=0x7ff6dc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:05.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e13f1640 1 --2- 172.21.15.46:0/3695944431 >> v2:172.21.15.154:3300/0 conn(0x7ff6dc15d6f0 0x7ff6dc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e13f1640 1 -- 172.21.15.46:0/3695944431 >> v1:172.21.15.154:6789/0 conn(0x7ff6dc1538a0 legacy=0x7ff6dc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:05.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.411+0000 7ff6e13f1640 1 -- 172.21.15.46:0/3695944431 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc0cb050 con 0x7ff6dc0cf160 2023-10-30T13:39:05.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.415+0000 7ff6e13f1640 1 --2- 172.21.15.46:0/3695944431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc0cf590 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0b30a0 tx=0x7ff6d401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24642dc1c85fe70f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:05.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.415+0000 7ff6db7fe640 1 -- 172.21.15.46:0/3695944431 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40af400 con 0x7ff6dc0cf160 2023-10-30T13:39:05.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.415+0000 7ff6db7fe640 1 -- 172.21.15.46:0/3695944431 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d40ab050 con 0x7ff6dc0cf160 2023-10-30T13:39:05.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.415+0000 7ff6db7fe640 1 -- 172.21.15.46:0/3695944431 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40af780 con 0x7ff6dc0cf160 2023-10-30T13:39:05.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.415+0000 7ff6e2e54640 1 -- 172.21.15.46:0/3695944431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc0cf160 msgr2=0x7ff6dc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:05.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.415+0000 7ff6e2e54640 1 --2- 172.21.15.46:0/3695944431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc0cf590 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0b30a0 tx=0x7ff6d401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.415+0000 7ff6e2e54640 1 -- 172.21.15.46:0/3695944431 shutdown_connections 2023-10-30T13:39:05.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.415+0000 7ff6e2e54640 1 --2- 172.21.15.46:0/3695944431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc0cf590 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.415+0000 7ff6e2e54640 1 --2- 172.21.15.46:0/3695944431 >> v2:172.21.15.154:3300/0 conn(0x7ff6dc15d6f0 0x7ff6dc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.415+0000 7ff6e2e54640 1 -- 172.21.15.46:0/3695944431 >> 172.21.15.46:0/3695944431 conn(0x7ff6dc06bfe0 msgr2=0x7ff6dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:05.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.415+0000 7ff6e2e54640 1 -- 172.21.15.46:0/3695944431 shutdown_connections 2023-10-30T13:39:05.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.415+0000 7ff6e2e54640 1 -- 172.21.15.46:0/3695944431 wait complete. 2023-10-30T13:39:05.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e2e54640 1 Processor -- start 2023-10-30T13:39:05.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e2e54640 1 -- start start 2023-10-30T13:39:05.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e2e54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc1f0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e2e54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc1538a0 0x7ff6dc0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e2e54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc15d6f0 0x7ff6dc1f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e2e54640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc1f0070 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e2e54640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc1e37b0 con 0x7ff6dc0cf160 2023-10-30T13:39:05.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e2e54640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc1e2be0 con 0x7ff6dc1538a0 2023-10-30T13:39:05.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e13f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc15d6f0 0x7ff6dc1f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e13f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc15d6f0 0x7ff6dc1f2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39524/0 (socket says 172.21.15.46:39524) 2023-10-30T13:39:05.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e13f1640 1 -- 172.21.15.46:0/4017133987 learned_addr learned my addr 172.21.15.46:0/4017133987 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:05.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e0bf0640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc1f0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6dbfff640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc1538a0 0x7ff6dc0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e13f1640 1 -- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc1538a0 msgr2=0x7ff6dc0b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:05.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e13f1640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc1538a0 0x7ff6dc0b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e13f1640 1 -- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc0cf160 msgr2=0x7ff6dc1f0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:05.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e13f1640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc1f0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e13f1640 1 -- 172.21.15.46:0/4017133987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d4009b30 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e13f1640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc15d6f0 0x7ff6dc1f2600 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff6d401cdf0 tx=0x7ff6d401bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:05.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6d97fa640 1 -- 172.21.15.46:0/4017133987 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40afd10 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6d97fa640 1 -- 172.21.15.46:0/4017133987 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d40afe90 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6d97fa640 1 -- 172.21.15.46:0/4017133987 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40ba680 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e2e54640 1 -- 172.21.15.46:0/4017133987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6dc1e2e80 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e2e54640 1 -- 172.21.15.46:0/4017133987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6dc1e3440 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.419+0000 7ff6e2e54640 1 -- 172.21.15.46:0/4017133987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6dc0b2a50 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.423+0000 7ff6d97fa640 1 -- 172.21.15.46:0/4017133987 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff6d40b6040 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.423+0000 7ff6d97fa640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6c00664d0 0x7ff6c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:05.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.423+0000 7ff6d97fa640 1 -- 172.21.15.46:0/4017133987 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7ff6d41221d0 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.423+0000 7ff6e0bf0640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6c00664d0 0x7ff6c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:05.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.423+0000 7ff6e0bf0640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6c00664d0 0x7ff6c0068990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0000f70 tx=0x7ff6d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:05.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.427+0000 7ff6d97fa640 1 -- 172.21.15.46:0/4017133987 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6d40ef1e0 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 -- 172.21.15.46:0/4017133987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6dc1e38f0 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6d97fa640 1 -- 172.21.15.46:0/4017133987 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6538 (secure 0 0 0) 0x7ff6d4009510 con 0x7ff6dc15d6f0 2023-10-30T13:39:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:05.622 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:44.121877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:05.622 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 80 2023-10-30T13:39:05.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 -- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6c00664d0 msgr2=0x7ff6c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:05.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6c00664d0 0x7ff6c0068990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0000f70 tx=0x7ff6d0009040 comp rx=0 tx=0).stop 2023-10-30T13:39:05.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 -- 172.21.15.46:0/4017133987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc15d6f0 msgr2=0x7ff6dc1f2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:05.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc15d6f0 0x7ff6dc1f2600 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff6d401cdf0 tx=0x7ff6d401bf60 comp rx=0 tx=0).stop 2023-10-30T13:39:05.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 -- 172.21.15.46:0/4017133987 shutdown_connections 2023-10-30T13:39:05.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6c00664d0 0x7ff6c0068990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc15d6f0 0x7ff6dc1f2600 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc1538a0 0x7ff6dc0b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 --2- 172.21.15.46:0/4017133987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc1f0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:05.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 -- 172.21.15.46:0/4017133987 >> 172.21.15.46:0/4017133987 conn(0x7ff6dc06bfe0 msgr2=0x7ff6dc15b340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:05.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 -- 172.21.15.46:0/4017133987 shutdown_connections 2023-10-30T13:39:05.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:05.619+0000 7ff6e2e54640 1 -- 172.21.15.46:0/4017133987 wait complete. 2023-10-30T13:39:05.643 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:39:07.644 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:07.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f285d7c5640 1 Processor -- start 2023-10-30T13:39:07.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f285d7c5640 1 -- start start 2023-10-30T13:39:07.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f285d7c5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f28580cf160 0x7f28580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:07.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f285d7c5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28580cb800 con 0x7f28581538a0 2023-10-30T13:39:07.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f285d7c5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28580cb330 con 0x7f285815d6f0 2023-10-30T13:39:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f285d7c5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28580cad70 con 0x7f28580cf160 2023-10-30T13:39:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f2856ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f28581538a0 0x7f28580ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33580/0 (socket says 172.21.15.46:33580) 2023-10-30T13:39:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f2856ffd640 1 -- 172.21.15.46:0/2716392996 learned_addr learned my addr 172.21.15.46:0/2716392996 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:39:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f28577fe640 1 --2- 172.21.15.46:0/2716392996 >> v2:172.21.15.154:3300/0 conn(0x7f28580cf160 0x7f28580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588776458 0 0) 0x7f28580cb800 con 0x7f28581538a0 2023-10-30T13:39:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2844003530 con 0x7f28581538a0 2023-10-30T13:39:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3497969202 0 0) 0x7f2844003530 con 0x7f28581538a0 2023-10-30T13:39:07.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 >> v2:172.21.15.154:3300/0 conn(0x7f28580cf160 msgr2=0x7f28580cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:07.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f28567fc640 1 --2- 172.21.15.46:0/2716392996 >> v2:172.21.15.154:3300/0 conn(0x7f28580cf160 0x7f28580cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:07.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 >> v1:172.21.15.154:6789/0 conn(0x7f285815d6f0 legacy=0x7f285815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:07.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28580cb050 con 0x7f28581538a0 2023-10-30T13:39:07.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f2848001680 con 0x7f28581538a0 2023-10-30T13:39:07.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 >> v1:172.21.15.46:6789/0 conn(0x7f28581538a0 legacy=0x7f28580ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:07.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.795+0000 7f28567fc640 1 --2- 172.21.15.46:0/2716392996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2844003980 0x7f2844003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:07.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f28567fc640 1 --2- 172.21.15.46:0/2716392996 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28440047d0 0x7f2844004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:07.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f28567fc640 1 --2- 172.21.15.46:0/2716392996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2844005190 0x7f284400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:07.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28580cad70 con 0x7f2844003980 2023-10-30T13:39:07.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284400c040 con 0x7f2844005190 2023-10-30T13:39:07.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284400c200 con 0x7f28440047d0 2023-10-30T13:39:07.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f2856ffd640 1 --2- 172.21.15.46:0/2716392996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2844003980 0x7f2844003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:07.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f28577fe640 1 --2- 172.21.15.46:0/2716392996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2844005190 0x7f284400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f284ffff640 1 --2- 172.21.15.46:0/2716392996 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28440047d0 0x7f2844004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f2856ffd640 1 -- 172.21.15.46:0/2716392996 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28440047d0 msgr2=0x7f2844004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f2856ffd640 1 --2- 172.21.15.46:0/2716392996 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28440047d0 0x7f2844004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f2856ffd640 1 -- 172.21.15.46:0/2716392996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2844005190 msgr2=0x7f284400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f2856ffd640 1 --2- 172.21.15.46:0/2716392996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2844005190 0x7f284400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f2856ffd640 1 -- 172.21.15.46:0/2716392996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2844003530 con 0x7f2844003980 2023-10-30T13:39:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f28577fe640 1 --2- 172.21.15.46:0/2716392996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2844005190 0x7f284400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f28577fe640 1 -- 172.21.15.46:0/2716392996 reap_dead start 2023-10-30T13:39:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f2856ffd640 1 --2- 172.21.15.46:0/2716392996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2844003980 0x7f2844003da0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f284800c240 tx=0x7f284801f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0ae50e185e1824 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848005ae0 con 0x7f2844003980 2023-10-30T13:39:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28480ab070 con 0x7f2844003980 2023-10-30T13:39:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.799+0000 7f28567fc640 1 -- 172.21.15.46:0/2716392996 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848007070 con 0x7f2844003980 2023-10-30T13:39:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 -- 172.21.15.46:0/2716392996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2844003980 msgr2=0x7f2844003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:07.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 --2- 172.21.15.46:0/2716392996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2844003980 0x7f2844003da0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f284800c240 tx=0x7f284801f370 comp rx=0 tx=0).stop 2023-10-30T13:39:07.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 -- 172.21.15.46:0/2716392996 shutdown_connections 2023-10-30T13:39:07.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 --2- 172.21.15.46:0/2716392996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2844003980 0x7f2844003da0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:07.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 -- 172.21.15.46:0/2716392996 >> 172.21.15.46:0/2716392996 conn(0x7f285806bfe0 msgr2=0x7f285806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:07.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 -- 172.21.15.46:0/2716392996 shutdown_connections 2023-10-30T13:39:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 -- 172.21.15.46:0/2716392996 wait complete. 2023-10-30T13:39:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 Processor -- start 2023-10-30T13:39:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 -- start start 2023-10-30T13:39:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2844003980 0x7f2858157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28580ce910 0x7f285815b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:07.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28580cd2e0 0x7f28580cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:07.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28580cdca0 con 0x7f28580ce910 2023-10-30T13:39:07.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28581f0350 con 0x7f2844003980 2023-10-30T13:39:07.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f285d7c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28581ef4f0 con 0x7f28580cd2e0 2023-10-30T13:39:07.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f284ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28580ce910 0x7f285815b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:07.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f284ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28580ce910 0x7f285815b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38552/0 (socket says 172.21.15.46:38552) 2023-10-30T13:39:07.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f284ffff640 1 -- 172.21.15.46:0/3091734270 learned_addr learned my addr 172.21.15.46:0/3091734270 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:07.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f2856ffd640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2844003980 0x7f2858157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:07.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f28577fe640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28580cd2e0 0x7f28580cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:07.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f284ffff640 1 -- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28580cd2e0 msgr2=0x7f28580cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f284ffff640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28580cd2e0 0x7f28580cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f284ffff640 1 -- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2844003980 msgr2=0x7f2858157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f284ffff640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2844003980 0x7f2858157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f284ffff640 1 -- 172.21.15.46:0/3091734270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28581ef7d0 con 0x7f28580ce910 2023-10-30T13:39:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.803+0000 7f2856ffd640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2844003980 0x7f2858157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.807+0000 7f284ffff640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28580ce910 0x7f285815b790 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2840001ec0 tx=0x7f284000e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.807+0000 7f284f7fe640 1 -- 172.21.15.46:0/3091734270 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28400052e0 con 0x7f28580ce910 2023-10-30T13:39:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.807+0000 7f284f7fe640 1 -- 172.21.15.46:0/3091734270 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2840005440 con 0x7f28580ce910 2023-10-30T13:39:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.807+0000 7f284f7fe640 1 -- 172.21.15.46:0/3091734270 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840003820 con 0x7f28580ce910 2023-10-30T13:39:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.807+0000 7f285d7c5640 1 -- 172.21.15.46:0/3091734270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28581efab0 con 0x7f28580ce910 2023-10-30T13:39:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.807+0000 7f285d7c5640 1 -- 172.21.15.46:0/3091734270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28581f0070 con 0x7f28580ce910 2023-10-30T13:39:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.807+0000 7f285d7c5640 1 -- 172.21.15.46:0/3091734270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285804e170 con 0x7f28580ce910 2023-10-30T13:39:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.807+0000 7f284f7fe640 1 -- 172.21.15.46:0/3091734270 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2840003980 con 0x7f28580ce910 2023-10-30T13:39:07.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.807+0000 7f284f7fe640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28380664a0 0x7f2838068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:07.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.807+0000 7f284f7fe640 1 -- 172.21.15.46:0/3091734270 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f2840083980 con 0x7f28580ce910 2023-10-30T13:39:07.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.811+0000 7f2856ffd640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28380664a0 0x7f2838068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:07.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.811+0000 7f2856ffd640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28380664a0 0x7f2838068960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f28480af000 tx=0x7f28480aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:07.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:07.811+0000 7f284f7fe640 1 -- 172.21.15.46:0/3091734270 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2840050b70 con 0x7f28580ce910 2023-10-30T13:39:08.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.007+0000 7f285d7c5640 1 -- 172.21.15.46:0/3091734270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28580b1dd0 con 0x7f28580ce910 2023-10-30T13:39:08.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.007+0000 7f284f7fe640 1 -- 172.21.15.46:0/3091734270 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f2840019090 con 0x7f28580ce910 2023-10-30T13:39:08.012 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:08.012 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:37:44.121877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":5574,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5574":{"gid":5574,"name":"e","rank":1,"incarnation":55,"state":"up:active","state_seq":36,"addr":"172.21.15.46:6837/126953235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":126953235},{"type":"v1","addr":"172.21.15.46:6837","nonce":126953235}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:08.012 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 80 2023-10-30T13:39:08.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.011+0000 7f285d7c5640 1 -- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28380664a0 msgr2=0x7f2838068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.011+0000 7f285d7c5640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28380664a0 0x7f2838068960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f28480af000 tx=0x7f28480aa000 comp rx=0 tx=0).stop 2023-10-30T13:39:08.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.011+0000 7f285d7c5640 1 -- 172.21.15.46:0/3091734270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28580ce910 msgr2=0x7f285815b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:08.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.011+0000 7f285d7c5640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28580ce910 0x7f285815b790 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2840001ec0 tx=0x7f284000e8a0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.011+0000 7f285d7c5640 1 -- 172.21.15.46:0/3091734270 shutdown_connections 2023-10-30T13:39:08.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.011+0000 7f285d7c5640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28380664a0 0x7f2838068960 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.011+0000 7f285d7c5640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28580cd2e0 0x7f28580cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.011+0000 7f285d7c5640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28580ce910 0x7f285815b790 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.011+0000 7f285d7c5640 1 --2- 172.21.15.46:0/3091734270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2844003980 0x7f2858157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:08.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.011+0000 7f285d7c5640 1 -- 172.21.15.46:0/3091734270 >> 172.21.15.46:0/3091734270 conn(0x7f285806bfe0 msgr2=0x7f28580c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:08.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.011+0000 7f285d7c5640 1 -- 172.21.15.46:0/3091734270 shutdown_connections 2023-10-30T13:39:08.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:08.011+0000 7f285d7c5640 1 -- 172.21.15.46:0/3091734270 wait complete. 2023-10-30T13:39:08.033 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:39:10.035 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:10.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f818751d640 1 Processor -- start 2023-10-30T13:39:10.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f818751d640 1 -- start start 2023-10-30T13:39:10.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f818751d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f818015d6f0 0x7f818015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f818751d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f81800cf160 0x7f81800cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f818751d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81800cb800 con 0x7f81801538a0 2023-10-30T13:39:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f818751d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81800cb330 con 0x7f81800cf160 2023-10-30T13:39:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f818751d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81800cad70 con 0x7f818015d6f0 2023-10-30T13:39:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f81852b9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f81801538a0 0x7f81800ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33584/0 (socket says 172.21.15.46:33584) 2023-10-30T13:39:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f81852b9640 1 -- 172.21.15.46:0/3841444520 learned_addr learned my addr 172.21.15.46:0/3841444520 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:39:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f8185aba640 1 --2- 172.21.15.46:0/3841444520 >> v2:172.21.15.46:3300/0 conn(0x7f81800cf160 0x7f81800cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f8184ab8640 1 --2- 172.21.15.46:0/3841444520 >> v2:172.21.15.154:3300/0 conn(0x7f818015d6f0 0x7f818015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f8185aba640 1 -- 172.21.15.46:0/3841444520 >> v2:172.21.15.154:3300/0 conn(0x7f818015d6f0 msgr2=0x7f818015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f8185aba640 1 --2- 172.21.15.46:0/3841444520 >> v2:172.21.15.154:3300/0 conn(0x7f818015d6f0 0x7f818015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f8185aba640 1 -- 172.21.15.46:0/3841444520 >> v1:172.21.15.46:6789/0 conn(0x7f81801538a0 legacy=0x7f81800ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.187+0000 7f8185aba640 1 -- 172.21.15.46:0/3841444520 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81800cb050 con 0x7f81800cf160 2023-10-30T13:39:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.191+0000 7f8185aba640 1 --2- 172.21.15.46:0/3841444520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81800cf160 0x7f81800cf590 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f81800b30a0 tx=0x7f817801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff9346023d8a2f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.191+0000 7f8177fff640 1 -- 172.21.15.46:0/3841444520 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81780b0440 con 0x7f81800cf160 2023-10-30T13:39:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.191+0000 7f8177fff640 1 -- 172.21.15.46:0/3841444520 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81780b05a0 con 0x7f81800cf160 2023-10-30T13:39:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.191+0000 7f8177fff640 1 -- 172.21.15.46:0/3841444520 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81780b0930 con 0x7f81800cf160 2023-10-30T13:39:10.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.191+0000 7f818751d640 1 -- 172.21.15.46:0/3841444520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81800cf160 msgr2=0x7f81800cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.191+0000 7f818751d640 1 --2- 172.21.15.46:0/3841444520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81800cf160 0x7f81800cf590 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f81800b30a0 tx=0x7f817801f0f0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.191+0000 7f818751d640 1 -- 172.21.15.46:0/3841444520 shutdown_connections 2023-10-30T13:39:10.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.191+0000 7f818751d640 1 --2- 172.21.15.46:0/3841444520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81800cf160 0x7f81800cf590 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.191+0000 7f818751d640 1 --2- 172.21.15.46:0/3841444520 >> v2:172.21.15.154:3300/0 conn(0x7f818015d6f0 0x7f818015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.191+0000 7f818751d640 1 -- 172.21.15.46:0/3841444520 >> 172.21.15.46:0/3841444520 conn(0x7f818006bfe0 msgr2=0x7f818006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:10.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.191+0000 7f818751d640 1 -- 172.21.15.46:0/3841444520 shutdown_connections 2023-10-30T13:39:10.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.191+0000 7f818751d640 1 -- 172.21.15.46:0/3841444520 wait complete. 2023-10-30T13:39:10.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f818751d640 1 Processor -- start 2023-10-30T13:39:10.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f818751d640 1 -- start start 2023-10-30T13:39:10.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f818751d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81800cf160 0x7f818014eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:10.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f818751d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81801538a0 0x7f818014f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:10.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f818751d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f818015d6f0 0x7f81801f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:10.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f818751d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81800b43c0 con 0x7f81800cf160 2023-10-30T13:39:10.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f818751d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81801f33b0 con 0x7f81801538a0 2023-10-30T13:39:10.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f818751d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81800b4660 con 0x7f818015d6f0 2023-10-30T13:39:10.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f81852b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81800cf160 0x7f818014eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:10.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f81852b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81800cf160 0x7f818014eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38568/0 (socket says 172.21.15.46:38568) 2023-10-30T13:39:10.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f81852b9640 1 -- 172.21.15.46:0/2018006506 learned_addr learned my addr 172.21.15.46:0/2018006506 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:10.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f8184ab8640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81801538a0 0x7f818014f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:10.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f8185aba640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f818015d6f0 0x7f81801f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:10.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f81852b9640 1 -- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f818015d6f0 msgr2=0x7f81801f2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f81852b9640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f818015d6f0 0x7f81801f2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f81852b9640 1 -- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81801538a0 msgr2=0x7f818014f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f81852b9640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81801538a0 0x7f818014f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f81852b9640 1 -- 172.21.15.46:0/2018006506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8178009a70 con 0x7f81800cf160 2023-10-30T13:39:10.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f81852b9640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81800cf160 0x7f818014eb90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f817c00c500 tx=0x7f817c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:10.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f8175ffb640 1 -- 172.21.15.46:0/2018006506 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c005cf0 con 0x7f81800cf160 2023-10-30T13:39:10.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f8175ffb640 1 -- 172.21.15.46:0/2018006506 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c005e50 con 0x7f81800cf160 2023-10-30T13:39:10.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f8175ffb640 1 -- 172.21.15.46:0/2018006506 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c0072f0 con 0x7f81800cf160 2023-10-30T13:39:10.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f818751d640 1 -- 172.21.15.46:0/2018006506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81800b4940 con 0x7f81800cf160 2023-10-30T13:39:10.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.195+0000 7f818751d640 1 -- 172.21.15.46:0/2018006506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81800b4f00 con 0x7f81800cf160 2023-10-30T13:39:10.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.199+0000 7f818751d640 1 -- 172.21.15.46:0/2018006506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81800b2a50 con 0x7f81800cf160 2023-10-30T13:39:10.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.199+0000 7f8175ffb640 1 -- 172.21.15.46:0/2018006506 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f817c007450 con 0x7f81800cf160 2023-10-30T13:39:10.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.199+0000 7f8175ffb640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81640664a0 0x7f8164068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:10.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.199+0000 7f8175ffb640 1 -- 172.21.15.46:0/2018006506 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f817c0836d0 con 0x7f81800cf160 2023-10-30T13:39:10.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.199+0000 7f8184ab8640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81640664a0 0x7f8164068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:10.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.199+0000 7f8184ab8640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81640664a0 0x7f8164068960 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f818015c510 tx=0x7f8170008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:10.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.203+0000 7f8175ffb640 1 -- 172.21.15.46:0/2018006506 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f817c050850 con 0x7f81800cf160 2023-10-30T13:39:10.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.391+0000 7f818751d640 1 -- 172.21.15.46:0/2018006506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f818004e170 con 0x7f81800cf160 2023-10-30T13:39:10.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f8175ffb640 1 -- 172.21.15.46:0/2018006506 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f817c053e70 con 0x7f81800cf160 2023-10-30T13:39:10.396 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:08.955220+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:10.397 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 81 2023-10-30T13:39:10.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f818751d640 1 -- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81640664a0 msgr2=0x7f8164068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f818751d640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81640664a0 0x7f8164068960 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f818015c510 tx=0x7f8170008040 comp rx=0 tx=0).stop 2023-10-30T13:39:10.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f818751d640 1 -- 172.21.15.46:0/2018006506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81800cf160 msgr2=0x7f818014eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f818751d640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81800cf160 0x7f818014eb90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f817c00c500 tx=0x7f817c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f818751d640 1 -- 172.21.15.46:0/2018006506 shutdown_connections 2023-10-30T13:39:10.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f818751d640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81640664a0 0x7f8164068960 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f818751d640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f818015d6f0 0x7f81801f2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f818751d640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81801538a0 0x7f818014f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f818751d640 1 --2- 172.21.15.46:0/2018006506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81800cf160 0x7f818014eb90 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f818751d640 1 -- 172.21.15.46:0/2018006506 >> 172.21.15.46:0/2018006506 conn(0x7f818006bfe0 msgr2=0x7f818015b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:10.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f818751d640 1 -- 172.21.15.46:0/2018006506 shutdown_connections 2023-10-30T13:39:10.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.395+0000 7f818751d640 1 -- 172.21.15.46:0/2018006506 wait complete. 2023-10-30T13:39:10.418 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T13:39:10.418 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:39:10.418 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:10.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc40ae46640 1 Processor -- start 2023-10-30T13:39:10.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc40ae46640 1 -- start start 2023-10-30T13:39:10.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc40ae46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc4041538a0 0x7fc40415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:10.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc40ae46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4040cb800 con 0x7fc4040cf160 2023-10-30T13:39:10.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc40ae46640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40415dc30 con 0x7fc4040ca0e0 2023-10-30T13:39:10.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc40ae46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4040cb330 con 0x7fc4041538a0 2023-10-30T13:39:10.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc4093e3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc4040cf160 0x7fc404164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33598/0 (socket says 172.21.15.46:33598) 2023-10-30T13:39:10.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc4093e3640 1 -- 172.21.15.46:0/3222842485 learned_addr learned my addr 172.21.15.46:0/3222842485 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:39:10.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc408be2640 1 --2- 172.21.15.46:0/3222842485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc4041538a0 0x7fc40415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:10.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc4037fe640 1 -- 172.21.15.46:0/3222842485 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148005557 0 0) 0x7fc4040cb800 con 0x7fc4040cf160 2023-10-30T13:39:10.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc4037fe640 1 -- 172.21.15.46:0/3222842485 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f4003530 con 0x7fc4040cf160 2023-10-30T13:39:10.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc4037fe640 1 -- 172.21.15.46:0/3222842485 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344604707 0 0) 0x7fc3f0001120 con 0x7fc4040ca0e0 2023-10-30T13:39:10.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc408be2640 1 -- 172.21.15.46:0/3222842485 >> v1:172.21.15.154:6789/0 conn(0x7fc4040ca0e0 legacy=0x7fc40415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc408be2640 1 -- 172.21.15.46:0/3222842485 >> v1:172.21.15.46:6789/0 conn(0x7fc4040cf160 legacy=0x7fc404164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.559+0000 7fc408be2640 1 -- 172.21.15.46:0/3222842485 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4040cad70 con 0x7fc4041538a0 2023-10-30T13:39:10.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.563+0000 7fc408be2640 1 --2- 172.21.15.46:0/3222842485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc4041538a0 0x7fc40415d6f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80097f0 tx=0x7fc3f801ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f42cfff59c006a10 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:10.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.563+0000 7fc4037fe640 1 -- 172.21.15.46:0/3222842485 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f80a89b0 con 0x7fc4041538a0 2023-10-30T13:39:10.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.563+0000 7fc4037fe640 1 -- 172.21.15.46:0/3222842485 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f80a8b10 con 0x7fc4041538a0 2023-10-30T13:39:10.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.563+0000 7fc4037fe640 1 -- 172.21.15.46:0/3222842485 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f80a8e80 con 0x7fc4041538a0 2023-10-30T13:39:10.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.563+0000 7fc40ae46640 1 -- 172.21.15.46:0/3222842485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc4041538a0 msgr2=0x7fc40415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.563+0000 7fc40ae46640 1 --2- 172.21.15.46:0/3222842485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc4041538a0 0x7fc40415d6f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80097f0 tx=0x7fc3f801ecb0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.563+0000 7fc40ae46640 1 -- 172.21.15.46:0/3222842485 shutdown_connections 2023-10-30T13:39:10.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.563+0000 7fc40ae46640 1 --2- 172.21.15.46:0/3222842485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc4041538a0 0x7fc40415d6f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.563+0000 7fc40ae46640 1 -- 172.21.15.46:0/3222842485 >> 172.21.15.46:0/3222842485 conn(0x7fc40406bfe0 msgr2=0x7fc40406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:10.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.563+0000 7fc40ae46640 1 -- 172.21.15.46:0/3222842485 shutdown_connections 2023-10-30T13:39:10.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.563+0000 7fc40ae46640 1 -- 172.21.15.46:0/3222842485 wait complete. 2023-10-30T13:39:10.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc40ae46640 1 Processor -- start 2023-10-30T13:39:10.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc40ae46640 1 -- start start 2023-10-30T13:39:10.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc40ae46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc4040ca0e0 0x7fc404150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:10.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc40ae46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4040cf160 0x7fc404065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:10.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc40ae46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4041538a0 0x7fc4040b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:10.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc40ae46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40415cdd0 con 0x7fc4040cf160 2023-10-30T13:39:10.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc40ae46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4040b35b0 con 0x7fc4041538a0 2023-10-30T13:39:10.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc40ae46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40415d390 con 0x7fc4040ca0e0 2023-10-30T13:39:10.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc403fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4040cf160 0x7fc404065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:10.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc403fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4040cf160 0x7fc404065ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38578/0 (socket says 172.21.15.46:38578) 2023-10-30T13:39:10.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc403fff640 1 -- 172.21.15.46:0/4188898778 learned_addr learned my addr 172.21.15.46:0/4188898778 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:10.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc4093e3640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4041538a0 0x7fc4040b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:10.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc408be2640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc4040ca0e0 0x7fc404150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:10.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc403fff640 1 -- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc4040ca0e0 msgr2=0x7fc404150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc403fff640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc4040ca0e0 0x7fc404150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc403fff640 1 -- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4041538a0 msgr2=0x7fc4040b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc403fff640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4041538a0 0x7fc4040b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc403fff640 1 -- 172.21.15.46:0/4188898778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f8009420 con 0x7fc4040cf160 2023-10-30T13:39:10.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc403fff640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4040cf160 0x7fc404065ce0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000c750 tx=0x7fc3f000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:10.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc4017fa640 1 -- 172.21.15.46:0/4188898778 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0005270 con 0x7fc4040cf160 2023-10-30T13:39:10.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc4017fa640 1 -- 172.21.15.46:0/4188898778 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f00053f0 con 0x7fc4040cf160 2023-10-30T13:39:10.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc40ae46640 1 -- 172.21.15.46:0/4188898778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40415d670 con 0x7fc4040cf160 2023-10-30T13:39:10.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc4017fa640 1 -- 172.21.15.46:0/4188898778 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f000a880 con 0x7fc4040cf160 2023-10-30T13:39:10.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.567+0000 7fc40ae46640 1 -- 172.21.15.46:0/4188898778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4040b3240 con 0x7fc4040cf160 2023-10-30T13:39:10.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.571+0000 7fc40ae46640 1 -- 172.21.15.46:0/4188898778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc404151270 con 0x7fc4040cf160 2023-10-30T13:39:10.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.571+0000 7fc4017fa640 1 -- 172.21.15.46:0/4188898778 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3f000a9e0 con 0x7fc4040cf160 2023-10-30T13:39:10.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.571+0000 7fc4017fa640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3e80664a0 0x7fc3e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:10.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.571+0000 7fc4017fa640 1 -- 172.21.15.46:0/4188898778 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fc3f0083690 con 0x7fc4040cf160 2023-10-30T13:39:10.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.571+0000 7fc408be2640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3e80664a0 0x7fc3e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:10.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.571+0000 7fc408be2640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3e80664a0 0x7fc3e8068960 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80b2000 tx=0x7fc3f80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:10.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.575+0000 7fc4017fa640 1 -- 172.21.15.46:0/4188898778 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc3f0050760 con 0x7fc4040cf160 2023-10-30T13:39:10.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.767+0000 7fc40ae46640 1 -- 172.21.15.46:0/4188898778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc40415dc30 con 0x7fc4040cf160 2023-10-30T13:39:10.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.767+0000 7fc4017fa640 1 -- 172.21.15.46:0/4188898778 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5746 (secure 0 0 0) 0x7fc40415dc30 con 0x7fc4040cf160 2023-10-30T13:39:10.770 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:10.771 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:08.955220+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:10.771 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 81 2023-10-30T13:39:10.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.767+0000 7fc40ae46640 1 -- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3e80664a0 msgr2=0x7fc3e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.767+0000 7fc40ae46640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3e80664a0 0x7fc3e8068960 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80b2000 tx=0x7fc3f80aa070 comp rx=0 tx=0).stop 2023-10-30T13:39:10.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.767+0000 7fc40ae46640 1 -- 172.21.15.46:0/4188898778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4040cf160 msgr2=0x7fc404065ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:10.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.767+0000 7fc40ae46640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4040cf160 0x7fc404065ce0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000c750 tx=0x7fc3f000cc10 comp rx=0 tx=0).stop 2023-10-30T13:39:10.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.771+0000 7fc40ae46640 1 -- 172.21.15.46:0/4188898778 shutdown_connections 2023-10-30T13:39:10.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.771+0000 7fc40ae46640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3e80664a0 0x7fc3e8068960 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.771+0000 7fc40ae46640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4041538a0 0x7fc4040b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.771+0000 7fc40ae46640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4040cf160 0x7fc404065ce0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.771+0000 7fc40ae46640 1 --2- 172.21.15.46:0/4188898778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc4040ca0e0 0x7fc404150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:10.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.771+0000 7fc40ae46640 1 -- 172.21.15.46:0/4188898778 >> 172.21.15.46:0/4188898778 conn(0x7fc40406bfe0 msgr2=0x7fc404151930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:10.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.771+0000 7fc40ae46640 1 -- 172.21.15.46:0/4188898778 shutdown_connections 2023-10-30T13:39:10.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:10.771+0000 7fc40ae46640 1 -- 172.21.15.46:0/4188898778 wait complete. 2023-10-30T13:39:10.793 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 81, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 81, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5721": { "addr": "172.21.15.46:6833/4044310929", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4044310929, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4044310929, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5721, "incarnation": 61, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 24 }, "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 28 }, "gid_6036": { "addr": "172.21.15.46:6836/3037060424", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3037060424, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3037060424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6036, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:39:08.955220+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5721, "mds_1": 6036, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:39:12.794 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:12.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.943+0000 7ff1d1fc4640 1 Processor -- start 2023-10-30T13:39:12.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.943+0000 7ff1d1fc4640 1 -- start start 2023-10-30T13:39:12.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.943+0000 7ff1d1fc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1cc06f7a0 0x7ff1cc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:12.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.943+0000 7ff1d1fc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1cc071110 0x7ff1cc158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:12.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.947+0000 7ff1d1fc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1cc066cf0 con 0x7ff1cc0c6bd0 2023-10-30T13:39:12.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.947+0000 7ff1d1fc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1cc0700c0 con 0x7ff1cc06f7a0 2023-10-30T13:39:12.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.947+0000 7ff1d1fc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc0c70d0 con 0x7ff1cc071110 2023-10-30T13:39:12.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.947+0000 7ff1cb7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1cc06f7a0 0x7ff1cc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:12.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.947+0000 7ff1cbfff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1cc071110 0x7ff1cc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:12.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.947+0000 7ff1caffd640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff1cc0c6bd0 0x7ff1cc069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:47170/0 (socket says 172.21.15.46:47170) 2023-10-30T13:39:12.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.947+0000 7ff1cb7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1cc06f7a0 0x7ff1cc06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:58500/0 (socket says 172.21.15.46:58500) 2023-10-30T13:39:12.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.947+0000 7ff1caffd640 1 -- 172.21.15.46:0/136429300 learned_addr learned my addr 172.21.15.46:0/136429300 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:39:12.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.947+0000 7ff1cb7fe640 1 -- 172.21.15.46:0/136429300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1cc071110 msgr2=0x7ff1cc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:12.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.947+0000 7ff1cb7fe640 1 --2- 172.21.15.46:0/136429300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1cc071110 0x7ff1cc158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:12.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.947+0000 7ff1cb7fe640 1 -- 172.21.15.46:0/136429300 >> v1:172.21.15.154:6789/0 conn(0x7ff1cc0c6bd0 legacy=0x7ff1cc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:12.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.947+0000 7ff1cb7fe640 1 -- 172.21.15.46:0/136429300 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc0c7390 con 0x7ff1cc06f7a0 2023-10-30T13:39:12.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1cb7fe640 1 --2- 172.21.15.46:0/136429300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc06f7a0 0x7ff1cc06fb80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0097c0 tx=0x7ff1bc01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40be50709a2e1b07 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:12.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1ca7fc640 1 -- 172.21.15.46:0/136429300 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0a8970 con 0x7ff1cc06f7a0 2023-10-30T13:39:12.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1ca7fc640 1 -- 172.21.15.46:0/136429300 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc0a8ad0 con 0x7ff1cc06f7a0 2023-10-30T13:39:12.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1ca7fc640 1 -- 172.21.15.46:0/136429300 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0a8e60 con 0x7ff1cc06f7a0 2023-10-30T13:39:12.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/136429300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc06f7a0 msgr2=0x7ff1cc06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:12.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 --2- 172.21.15.46:0/136429300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc06f7a0 0x7ff1cc06fb80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0097c0 tx=0x7ff1bc01ec70 comp rx=0 tx=0).stop 2023-10-30T13:39:12.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/136429300 shutdown_connections 2023-10-30T13:39:12.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 --2- 172.21.15.46:0/136429300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1cc071110 0x7ff1cc158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:12.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 --2- 172.21.15.46:0/136429300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc06f7a0 0x7ff1cc06fb80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:12.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/136429300 >> 172.21.15.46:0/136429300 conn(0x7ff1cc0c7910 msgr2=0x7ff1cc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:12.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/136429300 shutdown_connections 2023-10-30T13:39:12.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/136429300 wait complete. 2023-10-30T13:39:12.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 Processor -- start 2023-10-30T13:39:12.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 -- start start 2023-10-30T13:39:12.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc06f7a0 0x7ff1cc0c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:12.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1cc071110 0x7ff1cc1f0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:12.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1cc0c6bd0 0x7ff1cc1f0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:12.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc0afc90 con 0x7ff1cc071110 2023-10-30T13:39:12.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc156ad0 con 0x7ff1cc06f7a0 2023-10-30T13:39:12.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.951+0000 7ff1d1fc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc157c90 con 0x7ff1cc0c6bd0 2023-10-30T13:39:12.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1caffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1cc071110 0x7ff1cc1f0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:12.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1caffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1cc071110 0x7ff1cc1f0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38588/0 (socket says 172.21.15.46:38588) 2023-10-30T13:39:12.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1caffd640 1 -- 172.21.15.46:0/918559346 learned_addr learned my addr 172.21.15.46:0/918559346 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:12.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1cb7fe640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc06f7a0 0x7ff1cc0c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:12.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1cbfff640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1cc0c6bd0 0x7ff1cc1f0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:12.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1caffd640 1 -- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1cc0c6bd0 msgr2=0x7ff1cc1f0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:12.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1caffd640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1cc0c6bd0 0x7ff1cc1f0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:12.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1caffd640 1 -- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc06f7a0 msgr2=0x7ff1cc0c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:12.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1caffd640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc06f7a0 0x7ff1cc0c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:12.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1caffd640 1 -- 172.21.15.46:0/918559346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc009420 con 0x7ff1cc071110 2023-10-30T13:39:12.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1caffd640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1cc071110 0x7ff1cc1f0130 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000c4d0 tx=0x7ff1c000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:12.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1b3fff640 1 -- 172.21.15.46:0/918559346 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0005cf0 con 0x7ff1cc071110 2023-10-30T13:39:12.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1b3fff640 1 -- 172.21.15.46:0/918559346 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c0005e50 con 0x7ff1cc071110 2023-10-30T13:39:12.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/918559346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1cc0afec0 con 0x7ff1cc071110 2023-10-30T13:39:12.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1b3fff640 1 -- 172.21.15.46:0/918559346 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0007270 con 0x7ff1cc071110 2023-10-30T13:39:12.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/918559346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc1561a0 con 0x7ff1cc071110 2023-10-30T13:39:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/918559346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1cc156c80 con 0x7ff1cc071110 2023-10-30T13:39:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.955+0000 7ff1b3fff640 1 -- 172.21.15.46:0/918559346 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1c00073d0 con 0x7ff1cc071110 2023-10-30T13:39:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.959+0000 7ff1b3fff640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1b80664a0 0x7ff1b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:12.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.959+0000 7ff1b3fff640 1 -- 172.21.15.46:0/918559346 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7ff1c00836e0 con 0x7ff1cc071110 2023-10-30T13:39:12.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.959+0000 7ff1cb7fe640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1b80664a0 0x7ff1b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:12.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.959+0000 7ff1cb7fe640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1b80664a0 0x7ff1b8068960 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0b2000 tx=0x7ff1bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:12.959+0000 7ff1b3fff640 1 -- 172.21.15.46:0/918559346 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1cc156c80 con 0x7ff1cc071110 2023-10-30T13:39:13.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.151+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/918559346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1cc156c80 con 0x7ff1cc071110 2023-10-30T13:39:13.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.151+0000 7ff1b3fff640 1 -- 172.21.15.46:0/918559346 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5746 (secure 0 0 0) 0x7ff1cc156c80 con 0x7ff1cc071110 2023-10-30T13:39:13.155 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:13.155 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:08.955220+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:13.155 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 81 2023-10-30T13:39:13.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.151+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1b80664a0 msgr2=0x7ff1b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:13.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.151+0000 7ff1d1fc4640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1b80664a0 0x7ff1b8068960 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0b2000 tx=0x7ff1bc008040 comp rx=0 tx=0).stop 2023-10-30T13:39:13.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.151+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/918559346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1cc071110 msgr2=0x7ff1cc1f0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:13.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.151+0000 7ff1d1fc4640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1cc071110 0x7ff1cc1f0130 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000c4d0 tx=0x7ff1c000c990 comp rx=0 tx=0).stop 2023-10-30T13:39:13.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.155+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/918559346 shutdown_connections 2023-10-30T13:39:13.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.155+0000 7ff1d1fc4640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1b80664a0 0x7ff1b8068960 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.155+0000 7ff1d1fc4640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1cc0c6bd0 0x7ff1cc1f0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.155+0000 7ff1d1fc4640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1cc071110 0x7ff1cc1f0130 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.155+0000 7ff1d1fc4640 1 --2- 172.21.15.46:0/918559346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc06f7a0 0x7ff1cc0c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:13.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.155+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/918559346 >> 172.21.15.46:0/918559346 conn(0x7ff1cc0c7910 msgr2=0x7ff1cc0c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:13.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.155+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/918559346 shutdown_connections 2023-10-30T13:39:13.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:13.155+0000 7ff1d1fc4640 1 -- 172.21.15.46:0/918559346 wait complete. 2023-10-30T13:39:13.177 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:15.177 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:15.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f834d02f640 1 Processor -- start 2023-10-30T13:39:15.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f834d02f640 1 -- start start 2023-10-30T13:39:15.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f834d02f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83480cefa0 0x7f83480cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:15.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f834d02f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834815a6c0 con 0x7f83481538f0 2023-10-30T13:39:15.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f834d02f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83480ca6a0 con 0x7f834815cd50 2023-10-30T13:39:15.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f834d02f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f834815d300 con 0x7f83480cefa0 2023-10-30T13:39:15.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f8347fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f83481538f0 0x7f8348159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33610/0 (socket says 172.21.15.46:33610) 2023-10-30T13:39:15.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f8347fff640 1 -- 172.21.15.46:0/1798995947 learned_addr learned my addr 172.21.15.46:0/1798995947 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:39:15.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f834c82e640 1 --2- 172.21.15.46:0/1798995947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83480cefa0 0x7f83480cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:15.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f8346ffd640 1 -- 172.21.15.46:0/1798995947 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299775929 0 0) 0x7f834815a6c0 con 0x7f83481538f0 2023-10-30T13:39:15.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f8346ffd640 1 -- 172.21.15.46:0/1798995947 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8330003530 con 0x7f83481538f0 2023-10-30T13:39:15.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f8346ffd640 1 -- 172.21.15.46:0/1798995947 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476341606 0 0) 0x7f83480ca6a0 con 0x7f834815cd50 2023-10-30T13:39:15.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f8346ffd640 1 -- 172.21.15.46:0/1798995947 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f834815a6c0 con 0x7f834815cd50 2023-10-30T13:39:15.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f8346ffd640 1 -- 172.21.15.46:0/1798995947 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1205945033 0 0) 0x7f8330003530 con 0x7f83481538f0 2023-10-30T13:39:15.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f834c82e640 1 -- 172.21.15.46:0/1798995947 >> v1:172.21.15.154:6789/0 conn(0x7f834815cd50 legacy=0x7f83480c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:15.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f834c82e640 1 -- 172.21.15.46:0/1798995947 >> v1:172.21.15.46:6789/0 conn(0x7f83481538f0 legacy=0x7f8348159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:15.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.323+0000 7f834c82e640 1 -- 172.21.15.46:0/1798995947 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834815d5e0 con 0x7f83480cefa0 2023-10-30T13:39:15.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.327+0000 7f834c82e640 1 --2- 172.21.15.46:0/1798995947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83480cefa0 0x7f83480cf3d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8340009dc0 tx=0x7f834001f080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9ff877b663417c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:15.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.327+0000 7f8346ffd640 1 -- 172.21.15.46:0/1798995947 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338001680 con 0x7f83480cefa0 2023-10-30T13:39:15.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.327+0000 7f8346ffd640 1 -- 172.21.15.46:0/1798995947 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83400a8c10 con 0x7f83480cefa0 2023-10-30T13:39:15.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.327+0000 7f8346ffd640 1 -- 172.21.15.46:0/1798995947 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83400b0440 con 0x7f83480cefa0 2023-10-30T13:39:15.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.327+0000 7f834d02f640 1 -- 172.21.15.46:0/1798995947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83480cefa0 msgr2=0x7f83480cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:15.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.327+0000 7f834d02f640 1 --2- 172.21.15.46:0/1798995947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83480cefa0 0x7f83480cf3d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8340009dc0 tx=0x7f834001f080 comp rx=0 tx=0).stop 2023-10-30T13:39:15.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.327+0000 7f834d02f640 1 -- 172.21.15.46:0/1798995947 shutdown_connections 2023-10-30T13:39:15.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.327+0000 7f834d02f640 1 --2- 172.21.15.46:0/1798995947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83480cefa0 0x7f83480cf3d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.327+0000 7f834d02f640 1 -- 172.21.15.46:0/1798995947 >> 172.21.15.46:0/1798995947 conn(0x7f834806a640 msgr2=0x7f8348065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:15.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.327+0000 7f834d02f640 1 -- 172.21.15.46:0/1798995947 shutdown_connections 2023-10-30T13:39:15.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.327+0000 7f834d02f640 1 -- 172.21.15.46:0/1798995947 wait complete. 2023-10-30T13:39:15.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f834d02f640 1 Processor -- start 2023-10-30T13:39:15.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f834d02f640 1 -- start start 2023-10-30T13:39:15.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f834d02f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83480cefa0 0x7f8348152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:15.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f834d02f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83481538f0 0x7f83480b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:15.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f834d02f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f834815cd50 0x7f834813e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:15.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f834d02f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83480b49b0 con 0x7f83480cefa0 2023-10-30T13:39:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f834d02f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83480b30f0 con 0x7f83481538f0 2023-10-30T13:39:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f834d02f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83480b4be0 con 0x7f834815cd50 2023-10-30T13:39:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f8347fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83480cefa0 0x7f8348152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f8347fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83480cefa0 0x7f8348152380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38590/0 (socket says 172.21.15.46:38590) 2023-10-30T13:39:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f8347fff640 1 -- 172.21.15.46:0/730440525 learned_addr learned my addr 172.21.15.46:0/730440525 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f834c82e640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f834815cd50 0x7f834813e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f83477fe640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83481538f0 0x7f83480b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f8347fff640 1 -- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f834815cd50 msgr2=0x7f834813e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f8347fff640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f834815cd50 0x7f834813e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f8347fff640 1 -- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83481538f0 msgr2=0x7f83480b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f8347fff640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83481538f0 0x7f83480b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f8347fff640 1 -- 172.21.15.46:0/730440525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8340009a70 con 0x7f83480cefa0 2023-10-30T13:39:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f83477fe640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83481538f0 0x7f83480b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f8347fff640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83480cefa0 0x7f8348152380 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f833800c750 tx=0x7f833800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:15.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f8344ff9640 1 -- 172.21.15.46:0/730440525 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338005af0 con 0x7f83480cefa0 2023-10-30T13:39:15.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f8344ff9640 1 -- 172.21.15.46:0/730440525 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8338005c50 con 0x7f83480cefa0 2023-10-30T13:39:15.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f834d02f640 1 -- 172.21.15.46:0/730440525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834815c510 con 0x7f83480cefa0 2023-10-30T13:39:15.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f8344ff9640 1 -- 172.21.15.46:0/730440525 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338007070 con 0x7f83480cefa0 2023-10-30T13:39:15.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.331+0000 7f834d02f640 1 -- 172.21.15.46:0/730440525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83480b3390 con 0x7f83480cefa0 2023-10-30T13:39:15.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.335+0000 7f834d02f640 1 -- 172.21.15.46:0/730440525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83480cfc10 con 0x7f83480cefa0 2023-10-30T13:39:15.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.335+0000 7f8344ff9640 1 -- 172.21.15.46:0/730440525 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83380071d0 con 0x7f83480cefa0 2023-10-30T13:39:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.335+0000 7f8344ff9640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83340664a0 0x7f8334068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.335+0000 7f8344ff9640 1 -- 172.21.15.46:0/730440525 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f83380838d0 con 0x7f83480cefa0 2023-10-30T13:39:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.335+0000 7f83477fe640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83340664a0 0x7f8334068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.339+0000 7f83477fe640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83340664a0 0x7f8334068960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f83480c54d0 tx=0x7f833c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:15.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.343+0000 7f8344ff9640 1 -- 172.21.15.46:0/730440525 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8338050af0 con 0x7f83480cefa0 2023-10-30T13:39:15.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.543+0000 7f834d02f640 1 -- 172.21.15.46:0/730440525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f834804e170 con 0x7f83480cefa0 2023-10-30T13:39:15.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.543+0000 7f8344ff9640 1 -- 172.21.15.46:0/730440525 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f8338017090 con 0x7f83480cefa0 2023-10-30T13:39:15.548 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:15.548 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:08.955220+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:15.549 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 81 2023-10-30T13:39:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.547+0000 7f834d02f640 1 -- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83340664a0 msgr2=0x7f8334068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.547+0000 7f834d02f640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83340664a0 0x7f8334068960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f83480c54d0 tx=0x7f833c009040 comp rx=0 tx=0).stop 2023-10-30T13:39:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.547+0000 7f834d02f640 1 -- 172.21.15.46:0/730440525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83480cefa0 msgr2=0x7f8348152380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.547+0000 7f834d02f640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83480cefa0 0x7f8348152380 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f833800c750 tx=0x7f833800cc10 comp rx=0 tx=0).stop 2023-10-30T13:39:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.547+0000 7f834d02f640 1 -- 172.21.15.46:0/730440525 shutdown_connections 2023-10-30T13:39:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.547+0000 7f834d02f640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83340664a0 0x7f8334068960 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.547+0000 7f834d02f640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f834815cd50 0x7f834813e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.547+0000 7f834d02f640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83481538f0 0x7f83480b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.547+0000 7f834d02f640 1 --2- 172.21.15.46:0/730440525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83480cefa0 0x7f8348152380 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.547+0000 7f834d02f640 1 -- 172.21.15.46:0/730440525 >> 172.21.15.46:0/730440525 conn(0x7f834806a640 msgr2=0x7f83480c3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.547+0000 7f834d02f640 1 -- 172.21.15.46:0/730440525 shutdown_connections 2023-10-30T13:39:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:15.547+0000 7f834d02f640 1 -- 172.21.15.46:0/730440525 wait complete. 2023-10-30T13:39:15.570 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:17.570 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf1e7d3640 1 Processor -- start 2023-10-30T13:39:17.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf1e7d3640 1 -- start start 2023-10-30T13:39:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf1e7d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf18153d70 0x7fbf1806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf1e7d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf1806e470 0x7fbf180cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:17.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf1e7d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf1806afa0 con 0x7fbf180c8490 2023-10-30T13:39:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf1e7d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf1806d780 con 0x7fbf18153d70 2023-10-30T13:39:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf1e7d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf18154150 con 0x7fbf1806e470 2023-10-30T13:39:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf17fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf18153d70 0x7fbf1806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf17fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf18153d70 0x7fbf1806ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57924/0 (socket says 172.21.15.46:57924) 2023-10-30T13:39:17.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf17fff640 1 -- 172.21.15.46:0/2968132794 learned_addr learned my addr 172.21.15.46:0/2968132794 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf177fe640 1 --2- 172.21.15.46:0/2968132794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf1806e470 0x7fbf180cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf17fff640 1 -- 172.21.15.46:0/2968132794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf1806e470 msgr2=0x7fbf180cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf17fff640 1 --2- 172.21.15.46:0/2968132794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf1806e470 0x7fbf180cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf17fff640 1 -- 172.21.15.46:0/2968132794 >> v1:172.21.15.46:6789/0 conn(0x7fbf180c8490 legacy=0x7fbf180c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.723+0000 7fbf17fff640 1 -- 172.21.15.46:0/2968132794 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1806e030 con 0x7fbf18153d70 2023-10-30T13:39:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.727+0000 7fbf17fff640 1 --2- 172.21.15.46:0/2968132794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf18153d70 0x7fbf1806ca80 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009870 tx=0x7fbf0801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc16a4c494f455b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.727+0000 7fbf16ffd640 1 -- 172.21.15.46:0/2968132794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf080aa670 con 0x7fbf18153d70 2023-10-30T13:39:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.727+0000 7fbf16ffd640 1 -- 172.21.15.46:0/2968132794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf080aa7d0 con 0x7fbf18153d70 2023-10-30T13:39:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.727+0000 7fbf16ffd640 1 -- 172.21.15.46:0/2968132794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf080aab60 con 0x7fbf18153d70 2023-10-30T13:39:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.727+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/2968132794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf18153d70 msgr2=0x7fbf1806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.727+0000 7fbf1e7d3640 1 --2- 172.21.15.46:0/2968132794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf18153d70 0x7fbf1806ca80 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009870 tx=0x7fbf0801c510 comp rx=0 tx=0).stop 2023-10-30T13:39:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.727+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/2968132794 shutdown_connections 2023-10-30T13:39:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.727+0000 7fbf1e7d3640 1 --2- 172.21.15.46:0/2968132794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf1806e470 0x7fbf180cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.727+0000 7fbf1e7d3640 1 --2- 172.21.15.46:0/2968132794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf18153d70 0x7fbf1806ca80 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.727+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/2968132794 >> 172.21.15.46:0/2968132794 conn(0x7fbf18071ea0 msgr2=0x7fbf18065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.727+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/2968132794 shutdown_connections 2023-10-30T13:39:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.727+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/2968132794 wait complete. 2023-10-30T13:39:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf1e7d3640 1 Processor -- start 2023-10-30T13:39:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf1e7d3640 1 -- start start 2023-10-30T13:39:17.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf1e7d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806e470 0x7fbf18149b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf1e7d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf180c8490 0x7fbf181f67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf1e7d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf18153d70 0x7fbf181f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf1e7d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf18155510 con 0x7fbf1806e470 2023-10-30T13:39:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf1e7d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf181f6330 con 0x7fbf18153d70 2023-10-30T13:39:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf1e7d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1815dce0 con 0x7fbf180c8490 2023-10-30T13:39:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806e470 0x7fbf18149b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806e470 0x7fbf18149b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57936/0 (socket says 172.21.15.46:57936) 2023-10-30T13:39:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf17fff640 1 -- 172.21.15.46:0/1719649524 learned_addr learned my addr 172.21.15.46:0/1719649524 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:17.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf177fe640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf180c8490 0x7fbf181f67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf1cd70640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf18153d70 0x7fbf181f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf17fff640 1 -- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf180c8490 msgr2=0x7fbf181f67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf17fff640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf180c8490 0x7fbf181f67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf17fff640 1 -- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf18153d70 msgr2=0x7fbf181f6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:17.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf17fff640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf18153d70 0x7fbf181f6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf17fff640 1 -- 172.21.15.46:0/1719649524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf080094d0 con 0x7fbf1806e470 2023-10-30T13:39:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf17fff640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806e470 0x7fbf18149b20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbf0801abc0 tx=0x7fbf0801ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf14ff9640 1 -- 172.21.15.46:0/1719649524 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf080b3800 con 0x7fbf1806e470 2023-10-30T13:39:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf14ff9640 1 -- 172.21.15.46:0/1719649524 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf080b3980 con 0x7fbf1806e470 2023-10-30T13:39:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf14ff9640 1 -- 172.21.15.46:0/1719649524 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf080b3d30 con 0x7fbf1806e470 2023-10-30T13:39:17.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/1719649524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf18154420 con 0x7fbf1806e470 2023-10-30T13:39:17.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.731+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/1719649524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf181549e0 con 0x7fbf1806e470 2023-10-30T13:39:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.735+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/1719649524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1804e170 con 0x7fbf1806e470 2023-10-30T13:39:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.735+0000 7fbf14ff9640 1 -- 172.21.15.46:0/1719649524 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbf080b2380 con 0x7fbf1806e470 2023-10-30T13:39:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.735+0000 7fbf14ff9640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf040664d0 0x7fbf04068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:17.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.735+0000 7fbf14ff9640 1 -- 172.21.15.46:0/1719649524 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fbf0811ea20 con 0x7fbf1806e470 2023-10-30T13:39:17.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.735+0000 7fbf177fe640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf040664d0 0x7fbf04068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:17.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.735+0000 7fbf177fe640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf040664d0 0x7fbf04068990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fbf180c7130 tx=0x7fbf0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:17.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.739+0000 7fbf14ff9640 1 -- 172.21.15.46:0/1719649524 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbf080ebb10 con 0x7fbf1806e470 2023-10-30T13:39:17.885 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:39:17.889 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:39:17.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.927+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/1719649524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf181509e0 con 0x7fbf1806e470 2023-10-30T13:39:17.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.927+0000 7fbf14ff9640 1 -- 172.21.15.46:0/1719649524 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5748 (secure 0 0 0) 0x7fbf080c3090 con 0x7fbf1806e470 2023-10-30T13:39:17.931 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:17.931 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:15.683301+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:resolve","state_seq":29,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:17.932 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 82 2023-10-30T13:39:17.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.931+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf040664d0 msgr2=0x7fbf04068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:17.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.931+0000 7fbf1e7d3640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf040664d0 0x7fbf04068990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fbf180c7130 tx=0x7fbf0c008040 comp rx=0 tx=0).stop 2023-10-30T13:39:17.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.931+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/1719649524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806e470 msgr2=0x7fbf18149b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:17.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.931+0000 7fbf1e7d3640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806e470 0x7fbf18149b20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbf0801abc0 tx=0x7fbf0801ad50 comp rx=0 tx=0).stop 2023-10-30T13:39:17.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.931+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/1719649524 shutdown_connections 2023-10-30T13:39:17.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.931+0000 7fbf1e7d3640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf040664d0 0x7fbf04068990 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:17.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.931+0000 7fbf1e7d3640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf18153d70 0x7fbf181f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:17.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.931+0000 7fbf1e7d3640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf180c8490 0x7fbf181f67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:17.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.931+0000 7fbf1e7d3640 1 --2- 172.21.15.46:0/1719649524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806e470 0x7fbf18149b20 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:17.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.931+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/1719649524 >> 172.21.15.46:0/1719649524 conn(0x7fbf18071ea0 msgr2=0x7fbf1814e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:17.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.931+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/1719649524 shutdown_connections 2023-10-30T13:39:17.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:17.931+0000 7fbf1e7d3640 1 -- 172.21.15.46:0/1719649524 wait complete. 2023-10-30T13:39:17.953 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:19.954 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.107+0000 7f4f5f7aa640 1 Processor -- start 2023-10-30T13:39:20.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.107+0000 7f4f5f7aa640 1 -- start start 2023-10-30T13:39:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.107+0000 7f4f5f7aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f580c6540 0x7f4f5815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.107+0000 7f4f5f7aa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f580c69d0 con 0x7f4f581580e0 2023-10-30T13:39:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.107+0000 7f4f5f7aa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f580c5810 con 0x7f4f58070740 2023-10-30T13:39:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.107+0000 7f4f5f7aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5806acb0 con 0x7f4f580c6540 2023-10-30T13:39:20.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.107+0000 7f4f5d546640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4f581580e0 0x7f4f58069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57054/0 (socket says 172.21.15.46:57054) 2023-10-30T13:39:20.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.107+0000 7f4f5d546640 1 -- 172.21.15.46:0/3217839811 learned_addr learned my addr 172.21.15.46:0/3217839811 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:39:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.111+0000 7f4f5dd47640 1 --2- 172.21.15.46:0/3217839811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f580c6540 0x7f4f5815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.111+0000 7f4f4ffff640 1 -- 172.21.15.46:0/3217839811 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597154071 0 0) 0x7f4f580c69d0 con 0x7f4f581580e0 2023-10-30T13:39:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.111+0000 7f4f4ffff640 1 -- 172.21.15.46:0/3217839811 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f44003530 con 0x7f4f581580e0 2023-10-30T13:39:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.111+0000 7f4f4ffff640 1 -- 172.21.15.46:0/3217839811 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595104004 0 0) 0x7f4f580c5810 con 0x7f4f58070740 2023-10-30T13:39:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.111+0000 7f4f4ffff640 1 -- 172.21.15.46:0/3217839811 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f580c69d0 con 0x7f4f58070740 2023-10-30T13:39:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.111+0000 7f4f4ffff640 1 -- 172.21.15.46:0/3217839811 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3114848691 0 0) 0x7f4f44003530 con 0x7f4f581580e0 2023-10-30T13:39:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.111+0000 7f4f5dd47640 1 -- 172.21.15.46:0/3217839811 >> v1:172.21.15.154:6789/0 conn(0x7f4f58070740 legacy=0x7f4f5806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.111+0000 7f4f5dd47640 1 -- 172.21.15.46:0/3217839811 >> v1:172.21.15.46:6789/0 conn(0x7f4f581580e0 legacy=0x7f4f58069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.111+0000 7f4f5dd47640 1 -- 172.21.15.46:0/3217839811 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f580c6bb0 con 0x7f4f580c6540 2023-10-30T13:39:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.111+0000 7f4f5dd47640 1 --2- 172.21.15.46:0/3217839811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f580c6540 0x7f4f5815e700 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4f50009dc0 tx=0x7f4f5001c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3ab0fa3f8773d7c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f4ffff640 1 -- 172.21.15.46:0/3217839811 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54001680 con 0x7f4f580c6540 2023-10-30T13:39:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f4ffff640 1 -- 172.21.15.46:0/3217839811 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f500a55e0 con 0x7f4f580c6540 2023-10-30T13:39:20.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f4ffff640 1 -- 172.21.15.46:0/3217839811 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500a5950 con 0x7f4f580c6540 2023-10-30T13:39:20.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/3217839811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f580c6540 msgr2=0x7f4f5815e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 --2- 172.21.15.46:0/3217839811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f580c6540 0x7f4f5815e700 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4f50009dc0 tx=0x7f4f5001c890 comp rx=0 tx=0).stop 2023-10-30T13:39:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/3217839811 shutdown_connections 2023-10-30T13:39:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 --2- 172.21.15.46:0/3217839811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f580c6540 0x7f4f5815e700 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/3217839811 >> 172.21.15.46:0/3217839811 conn(0x7f4f580c7910 msgr2=0x7f4f580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/3217839811 shutdown_connections 2023-10-30T13:39:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/3217839811 wait complete. 2023-10-30T13:39:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 Processor -- start 2023-10-30T13:39:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 -- start start 2023-10-30T13:39:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f58070740 0x7f4f58149130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f580c6540 0x7f4f58071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f581580e0 0x7f4f58071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f581487a0 con 0x7f4f581580e0 2023-10-30T13:39:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f581478b0 con 0x7f4f580c6540 2023-10-30T13:39:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5f7aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f58147b90 con 0x7f4f58070740 2023-10-30T13:39:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5dd47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f581580e0 0x7f4f58071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5dd47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f581580e0 0x7f4f58071940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57950/0 (socket says 172.21.15.46:57950) 2023-10-30T13:39:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5dd47640 1 -- 172.21.15.46:0/2304195572 learned_addr learned my addr 172.21.15.46:0/2304195572 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5d546640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f58070740 0x7f4f58149130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.115+0000 7f4f5cd45640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f580c6540 0x7f4f58071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f5dd47640 1 -- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f58070740 msgr2=0x7f4f58149130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f5dd47640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f58070740 0x7f4f58149130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f5dd47640 1 -- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f580c6540 msgr2=0x7f4f58071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f5dd47640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f580c6540 0x7f4f58071400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f5dd47640 1 -- 172.21.15.46:0/2304195572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f50009a70 con 0x7f4f581580e0 2023-10-30T13:39:20.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f5d546640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f58070740 0x7f4f58149130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:20.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f5dd47640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f581580e0 0x7f4f58071940 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4f5001af40 tx=0x7f4f50019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:20.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f4dffb640 1 -- 172.21.15.46:0/2304195572 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500bd070 con 0x7f4f581580e0 2023-10-30T13:39:20.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f4dffb640 1 -- 172.21.15.46:0/2304195572 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f500a5960 con 0x7f4f581580e0 2023-10-30T13:39:20.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/2304195572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f58147e70 con 0x7f4f581580e0 2023-10-30T13:39:20.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f4dffb640 1 -- 172.21.15.46:0/2304195572 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500a5cd0 con 0x7f4f581580e0 2023-10-30T13:39:20.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/2304195572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f58148430 con 0x7f4f581580e0 2023-10-30T13:39:20.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/2304195572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5804e170 con 0x7f4f581580e0 2023-10-30T13:39:20.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f4dffb640 1 -- 172.21.15.46:0/2304195572 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4f50004040 con 0x7f4f581580e0 2023-10-30T13:39:20.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f4dffb640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f3c0664a0 0x7f4f3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:20.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.119+0000 7f4f4dffb640 1 -- 172.21.15.46:0/2304195572 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f4f5011ea30 con 0x7f4f581580e0 2023-10-30T13:39:20.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.123+0000 7f4f5d546640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f3c0664a0 0x7f4f3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:20.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.123+0000 7f4f5d546640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f3c0664a0 0x7f4f3c068960 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f54000fc0 tx=0x7f4f54009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:20.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.127+0000 7f4f4dffb640 1 -- 172.21.15.46:0/2304195572 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4f500ebb10 con 0x7f4f581580e0 2023-10-30T13:39:20.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/2304195572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f580c5290 con 0x7f4f581580e0 2023-10-30T13:39:20.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f4dffb640 1 -- 172.21.15.46:0/2304195572 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5748 (secure 0 0 0) 0x7f4f500b2090 con 0x7f4f581580e0 2023-10-30T13:39:20.322 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:20.322 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:15.683301+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:resolve","state_seq":29,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:20.323 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 82 2023-10-30T13:39:20.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f3c0664a0 msgr2=0x7f4f3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f3c0664a0 0x7f4f3c068960 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f54000fc0 tx=0x7f4f54009040 comp rx=0 tx=0).stop 2023-10-30T13:39:20.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/2304195572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f581580e0 msgr2=0x7f4f58071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:20.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f581580e0 0x7f4f58071940 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4f5001af40 tx=0x7f4f50019630 comp rx=0 tx=0).stop 2023-10-30T13:39:20.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/2304195572 shutdown_connections 2023-10-30T13:39:20.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f3c0664a0 0x7f4f3c068960 secure :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f54000fc0 tx=0x7f4f54009040 comp rx=0 tx=0).stop 2023-10-30T13:39:20.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f581580e0 0x7f4f58071940 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f580c6540 0x7f4f58071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 --2- 172.21.15.46:0/2304195572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f58070740 0x7f4f58149130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/2304195572 >> 172.21.15.46:0/2304195572 conn(0x7f4f580c7910 msgr2=0x7f4f58149fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/2304195572 shutdown_connections 2023-10-30T13:39:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:20.319+0000 7f4f5f7aa640 1 -- 172.21.15.46:0/2304195572 wait complete. 2023-10-30T13:39:20.343 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:22.343 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:22.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab144c9640 1 Processor -- start 2023-10-30T13:39:22.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab144c9640 1 -- start start 2023-10-30T13:39:22.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab144c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fab0c1538a0 0x7fab0c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:22.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab144c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fab0c0ca0e0 0x7fab0c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:22.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab144c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0c0cb800 con 0x7fab0c0cf160 2023-10-30T13:39:22.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab144c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab0c15dc30 con 0x7fab0c1538a0 2023-10-30T13:39:22.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab144c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab0c0cb330 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab12265640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fab0c1538a0 0x7fab0c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:22.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab12265640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fab0c1538a0 0x7fab0c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57958/0 (socket says 172.21.15.46:57958) 2023-10-30T13:39:22.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab12265640 1 -- 172.21.15.46:0/1492931568 learned_addr learned my addr 172.21.15.46:0/1492931568 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:22.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab11a64640 1 --2- 172.21.15.46:0/1492931568 >> v2:172.21.15.154:3300/0 conn(0x7fab0c0ca0e0 0x7fab0c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:22.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab12265640 1 -- 172.21.15.46:0/1492931568 >> v2:172.21.15.154:3300/0 conn(0x7fab0c0ca0e0 msgr2=0x7fab0c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:22.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab12265640 1 --2- 172.21.15.46:0/1492931568 >> v2:172.21.15.154:3300/0 conn(0x7fab0c0ca0e0 0x7fab0c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab12265640 1 -- 172.21.15.46:0/1492931568 >> v1:172.21.15.154:6789/0 conn(0x7fab0c0cf160 legacy=0x7fab0c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:22.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.495+0000 7fab12265640 1 -- 172.21.15.46:0/1492931568 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c0cad70 con 0x7fab0c1538a0 2023-10-30T13:39:22.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.499+0000 7fab12265640 1 --2- 172.21.15.46:0/1492931568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab0c1538a0 0x7fab0c15d6f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0b30a0 tx=0x7fab0801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7869c01e6be2e813 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:22.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.499+0000 7fab11263640 1 -- 172.21.15.46:0/1492931568 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab080a89f0 con 0x7fab0c1538a0 2023-10-30T13:39:22.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.499+0000 7fab11263640 1 -- 172.21.15.46:0/1492931568 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab080a8b50 con 0x7fab0c1538a0 2023-10-30T13:39:22.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.499+0000 7fab11263640 1 -- 172.21.15.46:0/1492931568 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab080b0440 con 0x7fab0c1538a0 2023-10-30T13:39:22.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.499+0000 7fab144c9640 1 -- 172.21.15.46:0/1492931568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab0c1538a0 msgr2=0x7fab0c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:22.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.499+0000 7fab144c9640 1 --2- 172.21.15.46:0/1492931568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab0c1538a0 0x7fab0c15d6f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0b30a0 tx=0x7fab0801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.499+0000 7fab144c9640 1 -- 172.21.15.46:0/1492931568 shutdown_connections 2023-10-30T13:39:22.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.499+0000 7fab144c9640 1 --2- 172.21.15.46:0/1492931568 >> v2:172.21.15.154:3300/0 conn(0x7fab0c0ca0e0 0x7fab0c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.499+0000 7fab144c9640 1 --2- 172.21.15.46:0/1492931568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab0c1538a0 0x7fab0c15d6f0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.499+0000 7fab144c9640 1 -- 172.21.15.46:0/1492931568 >> 172.21.15.46:0/1492931568 conn(0x7fab0c06bfe0 msgr2=0x7fab0c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:22.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.499+0000 7fab144c9640 1 -- 172.21.15.46:0/1492931568 shutdown_connections 2023-10-30T13:39:22.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.499+0000 7fab144c9640 1 -- 172.21.15.46:0/1492931568 wait complete. 2023-10-30T13:39:22.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab144c9640 1 Processor -- start 2023-10-30T13:39:22.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab144c9640 1 -- start start 2023-10-30T13:39:22.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab144c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab0c0ca0e0 0x7fab0c158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:22.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab144c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab0c0cf160 0x7fab0c0c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:22.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab144c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab0c1538a0 0x7fab0c1ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:22.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab144c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c1e4db0 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab144c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c1ef520 con 0x7fab0c0cf160 2023-10-30T13:39:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab144c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c1f2960 con 0x7fab0c1538a0 2023-10-30T13:39:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab12265640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab0c0ca0e0 0x7fab0c158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab12265640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab0c0ca0e0 0x7fab0c158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57968/0 (socket says 172.21.15.46:57968) 2023-10-30T13:39:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab12265640 1 -- 172.21.15.46:0/257045907 learned_addr learned my addr 172.21.15.46:0/257045907 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab11a64640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab0c0cf160 0x7fab0c0c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab12a66640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab0c1538a0 0x7fab0c1ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab12265640 1 -- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab0c1538a0 msgr2=0x7fab0c1ef9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab12265640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab0c1538a0 0x7fab0c1ef9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab12265640 1 -- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab0c0cf160 msgr2=0x7fab0c0c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab12265640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab0c0cf160 0x7fab0c0c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab12265640 1 -- 172.21.15.46:0/257045907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab080094d0 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab12265640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab0c0ca0e0 0x7fab0c158440 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fab0801b900 tx=0x7fab0801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:22.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab02ffd640 1 -- 172.21.15.46:0/257045907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab080ba600 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab02ffd640 1 -- 172.21.15.46:0/257045907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab080ba760 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab144c9640 1 -- 172.21.15.46:0/257045907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0c1f2c20 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab02ffd640 1 -- 172.21.15.46:0/257045907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab080bab10 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab144c9640 1 -- 172.21.15.46:0/257045907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0c1f31e0 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.503+0000 7fab144c9640 1 -- 172.21.15.46:0/257045907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0c0b2a50 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.507+0000 7fab02ffd640 1 -- 172.21.15.46:0/257045907 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fab080bac70 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.507+0000 7fab02ffd640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faaf00664d0 0x7faaf0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:22.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.507+0000 7fab02ffd640 1 -- 172.21.15.46:0/257045907 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fab08122770 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.507+0000 7fab11a64640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faaf00664d0 0x7faaf0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:22.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.507+0000 7fab11a64640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faaf00664d0 0x7faaf0068990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0cf0d0 tx=0x7faafc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:22.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.511+0000 7fab02ffd640 1 -- 172.21.15.46:0/257045907 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fab080ef8d0 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.699+0000 7fab144c9640 1 -- 172.21.15.46:0/257045907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab0c159850 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.699+0000 7fab02ffd640 1 -- 172.21.15.46:0/257045907 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5750 (secure 0 0 0) 0x7fab080f2ef0 con 0x7fab0c0ca0e0 2023-10-30T13:39:22.704 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:22.704 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:21.135605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:22.704 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 83 2023-10-30T13:39:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.703+0000 7fab144c9640 1 -- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faaf00664d0 msgr2=0x7faaf0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.703+0000 7fab144c9640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faaf00664d0 0x7faaf0068990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0cf0d0 tx=0x7faafc008040 comp rx=0 tx=0).stop 2023-10-30T13:39:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.703+0000 7fab144c9640 1 -- 172.21.15.46:0/257045907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab0c0ca0e0 msgr2=0x7fab0c158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:22.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.703+0000 7fab144c9640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab0c0ca0e0 0x7fab0c158440 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fab0801b900 tx=0x7fab0801ba90 comp rx=0 tx=0).stop 2023-10-30T13:39:22.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.703+0000 7fab144c9640 1 -- 172.21.15.46:0/257045907 shutdown_connections 2023-10-30T13:39:22.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.703+0000 7fab144c9640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faaf00664d0 0x7faaf0068990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.703+0000 7fab144c9640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab0c1538a0 0x7fab0c1ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.703+0000 7fab144c9640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab0c0cf160 0x7fab0c0c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.703+0000 7fab144c9640 1 --2- 172.21.15.46:0/257045907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab0c0ca0e0 0x7fab0c158440 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:22.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.703+0000 7fab144c9640 1 -- 172.21.15.46:0/257045907 >> 172.21.15.46:0/257045907 conn(0x7fab0c06bfe0 msgr2=0x7fab0c151f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:22.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.703+0000 7fab144c9640 1 -- 172.21.15.46:0/257045907 shutdown_connections 2023-10-30T13:39:22.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:22.703+0000 7fab144c9640 1 -- 172.21.15.46:0/257045907 wait complete. 2023-10-30T13:39:22.725 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 83, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 83, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5721": { "addr": "172.21.15.46:6833/4044310929", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4044310929, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4044310929, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5721, "incarnation": 61, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 24 }, "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 28 }, "gid_6036": { "addr": "172.21.15.46:6836/3037060424", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3037060424, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3037060424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6036, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:reconnect", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:39:21.135605+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5721, "mds_1": 6036, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:39:24.726 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.875+0000 7f93cb046640 1 Processor -- start 2023-10-30T13:39:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.875+0000 7f93cb046640 1 -- start start 2023-10-30T13:39:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93cb046640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f93c40c6b80 0x7f93c40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93cb046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f93c413f4a0 0x7f93c40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93cb046640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c40c0a50 con 0x7f93c40c6f60 2023-10-30T13:39:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93cb046640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c406f6e0 con 0x7f93c40c6b80 2023-10-30T13:39:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93cb046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c406f9c0 con 0x7f93c413f4a0 2023-10-30T13:39:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93c8de2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f93c40c6b80 0x7f93c40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:24.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93c8de2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f93c40c6b80 0x7f93c40bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57976/0 (socket says 172.21.15.46:57976) 2023-10-30T13:39:24.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93c8de2640 1 -- 172.21.15.46:0/164689907 learned_addr learned my addr 172.21.15.46:0/164689907 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:24.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93c95e3640 1 --2- 172.21.15.46:0/164689907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f93c413f4a0 0x7f93c40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:24.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93c8de2640 1 -- 172.21.15.46:0/164689907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f93c413f4a0 msgr2=0x7f93c40c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:24.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93c8de2640 1 --2- 172.21.15.46:0/164689907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f93c413f4a0 0x7f93c40c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:24.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93c8de2640 1 -- 172.21.15.46:0/164689907 >> v1:172.21.15.46:6789/0 conn(0x7f93c40c6f60 legacy=0x7f93c413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:24.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93c8de2640 1 -- 172.21.15.46:0/164689907 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c406fca0 con 0x7f93c40c6b80 2023-10-30T13:39:24.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.879+0000 7f93c95e3640 1 --2- 172.21.15.46:0/164689907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f93c413f4a0 0x7f93c40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:39:24.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93c8de2640 1 --2- 172.21.15.46:0/164689907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93c40c6b80 0x7f93c40bfec0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f93b8009870 tx=0x7f93b801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7208b162694a6b30 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:24.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93c37fe640 1 -- 172.21.15.46:0/164689907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b80ab650 con 0x7f93c40c6b80 2023-10-30T13:39:24.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93c37fe640 1 -- 172.21.15.46:0/164689907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b80ab7b0 con 0x7f93c40c6b80 2023-10-30T13:39:24.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93c37fe640 1 -- 172.21.15.46:0/164689907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b80abb40 con 0x7f93c40c6b80 2023-10-30T13:39:24.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93cb046640 1 -- 172.21.15.46:0/164689907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93c40c6b80 msgr2=0x7f93c40bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:24.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93cb046640 1 --2- 172.21.15.46:0/164689907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93c40c6b80 0x7f93c40bfec0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f93b8009870 tx=0x7f93b801c540 comp rx=0 tx=0).stop 2023-10-30T13:39:24.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93cb046640 1 -- 172.21.15.46:0/164689907 shutdown_connections 2023-10-30T13:39:24.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93cb046640 1 --2- 172.21.15.46:0/164689907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f93c413f4a0 0x7f93c40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:24.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93cb046640 1 --2- 172.21.15.46:0/164689907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93c40c6b80 0x7f93c40bfec0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:24.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93cb046640 1 -- 172.21.15.46:0/164689907 >> 172.21.15.46:0/164689907 conn(0x7f93c40c7910 msgr2=0x7f93c40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:24.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93cb046640 1 -- 172.21.15.46:0/164689907 shutdown_connections 2023-10-30T13:39:24.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93cb046640 1 -- 172.21.15.46:0/164689907 wait complete. 2023-10-30T13:39:24.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93cb046640 1 Processor -- start 2023-10-30T13:39:24.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.883+0000 7f93cb046640 1 -- start start 2023-10-30T13:39:24.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93cb046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f93c40c6b80 0x7f93c4159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:24.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93cb046640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93c40c6f60 0x7f93c413c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:24.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93cb046640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93c413f4a0 0x7f93c4157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:24.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93cb046640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c413e880 con 0x7f93c413f4a0 2023-10-30T13:39:24.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93cb046640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c41599a0 con 0x7f93c40c6f60 2023-10-30T13:39:24.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93cb046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c413dfe0 con 0x7f93c40c6b80 2023-10-30T13:39:24.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c95e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93c413f4a0 0x7f93c4157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:24.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c95e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93c413f4a0 0x7f93c4157de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57982/0 (socket says 172.21.15.46:57982) 2023-10-30T13:39:24.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c95e3640 1 -- 172.21.15.46:0/2841798934 learned_addr learned my addr 172.21.15.46:0/2841798934 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:24.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c3fff640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93c40c6f60 0x7f93c413c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:24.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c8de2640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f93c40c6b80 0x7f93c4159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:24.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c95e3640 1 -- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f93c40c6b80 msgr2=0x7f93c4159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:24.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c95e3640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f93c40c6b80 0x7f93c4159380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:24.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c95e3640 1 -- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93c40c6f60 msgr2=0x7f93c413c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:24.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c95e3640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93c40c6f60 0x7f93c413c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:24.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c95e3640 1 -- 172.21.15.46:0/2841798934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b80094d0 con 0x7f93c413f4a0 2023-10-30T13:39:24.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c95e3640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93c413f4a0 0x7f93c4157de0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f93bc00cd50 tx=0x7f93bc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:24.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c17fa640 1 -- 172.21.15.46:0/2841798934 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc00f680 con 0x7f93c413f4a0 2023-10-30T13:39:24.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c17fa640 1 -- 172.21.15.46:0/2841798934 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93bc00f800 con 0x7f93c413f4a0 2023-10-30T13:39:24.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93c17fa640 1 -- 172.21.15.46:0/2841798934 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc00fbb0 con 0x7f93c413f4a0 2023-10-30T13:39:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93cb046640 1 -- 172.21.15.46:0/2841798934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c413e2c0 con 0x7f93c413f4a0 2023-10-30T13:39:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93cb046640 1 -- 172.21.15.46:0/2841798934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c41578c0 con 0x7f93c413f4a0 2023-10-30T13:39:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.887+0000 7f93cb046640 1 -- 172.21.15.46:0/2841798934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c404e170 con 0x7f93c413f4a0 2023-10-30T13:39:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.891+0000 7f93c17fa640 1 -- 172.21.15.46:0/2841798934 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f93bc00b040 con 0x7f93c413f4a0 2023-10-30T13:39:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.891+0000 7f93c17fa640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f93a80664a0 0x7f93a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:24.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.891+0000 7f93c17fa640 1 -- 172.21.15.46:0/2841798934 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f93bc084c50 con 0x7f93c413f4a0 2023-10-30T13:39:24.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.891+0000 7f93c8de2640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f93a80664a0 0x7f93a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:24.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.891+0000 7f93c8de2640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f93a80664a0 0x7f93a8068960 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f93b80af000 tx=0x7f93b80a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:24.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:24.895+0000 7f93c17fa640 1 -- 172.21.15.46:0/2841798934 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f93bc051d20 con 0x7f93c413f4a0 2023-10-30T13:39:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.083+0000 7f93cb046640 1 -- 172.21.15.46:0/2841798934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93c406c5f0 con 0x7f93c413f4a0 2023-10-30T13:39:25.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93c17fa640 1 -- 172.21.15.46:0/2841798934 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5750 (secure 0 0 0) 0x7f93bc01d020 con 0x7f93c413f4a0 2023-10-30T13:39:25.089 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:25.089 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:21.135605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:25.089 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 83 2023-10-30T13:39:25.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93cb046640 1 -- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f93a80664a0 msgr2=0x7f93a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:25.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93cb046640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f93a80664a0 0x7f93a8068960 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f93b80af000 tx=0x7f93b80a7000 comp rx=0 tx=0).stop 2023-10-30T13:39:25.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93cb046640 1 -- 172.21.15.46:0/2841798934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93c413f4a0 msgr2=0x7f93c4157de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:25.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93cb046640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93c413f4a0 0x7f93c4157de0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f93bc00cd50 tx=0x7f93bc00a650 comp rx=0 tx=0).stop 2023-10-30T13:39:25.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93cb046640 1 -- 172.21.15.46:0/2841798934 shutdown_connections 2023-10-30T13:39:25.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93cb046640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f93a80664a0 0x7f93a8068960 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93cb046640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93c413f4a0 0x7f93c4157de0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93cb046640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93c40c6f60 0x7f93c413c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93cb046640 1 --2- 172.21.15.46:0/2841798934 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f93c40c6b80 0x7f93c4159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:25.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93cb046640 1 -- 172.21.15.46:0/2841798934 >> 172.21.15.46:0/2841798934 conn(0x7f93c40c7910 msgr2=0x7f93c40c0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:25.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93cb046640 1 -- 172.21.15.46:0/2841798934 shutdown_connections 2023-10-30T13:39:25.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:25.087+0000 7f93cb046640 1 -- 172.21.15.46:0/2841798934 wait complete. 2023-10-30T13:39:25.110 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:27.111 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:27.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.263+0000 7f10f5165640 1 Processor -- start 2023-10-30T13:39:27.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10f5165640 1 -- start start 2023-10-30T13:39:27.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10f5165640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10f00c6b80 0x7f10f00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:27.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10f5165640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10f00c6f60 0x7f10f00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:27.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10f5165640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f00c0a50 con 0x7f10f00c8430 2023-10-30T13:39:27.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10f5165640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f006f6e0 con 0x7f10f00c6f60 2023-10-30T13:39:27.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10f5165640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f006f9c0 con 0x7f10f00c6b80 2023-10-30T13:39:27.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10e7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10f00c6f60 0x7f10f00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:27.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10e7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10f00c6f60 0x7f10f00c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47634/0 (socket says 172.21.15.46:47634) 2023-10-30T13:39:27.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10e7fff640 1 -- 172.21.15.46:0/2218135289 learned_addr learned my addr 172.21.15.46:0/2218135289 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:27.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10eed9d640 1 --2- 172.21.15.46:0/2218135289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10f00c6b80 0x7f10f00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:27.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10e7fff640 1 -- 172.21.15.46:0/2218135289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10f00c6b80 msgr2=0x7f10f00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:27.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10e7fff640 1 --2- 172.21.15.46:0/2218135289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10f00c6b80 0x7f10f00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10e7fff640 1 -- 172.21.15.46:0/2218135289 >> v1:172.21.15.46:6789/0 conn(0x7f10f00c8430 legacy=0x7f10f013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.267+0000 7f10e7fff640 1 -- 172.21.15.46:0/2218135289 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f006fca0 con 0x7f10f00c6f60 2023-10-30T13:39:27.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10e7fff640 1 --2- 172.21.15.46:0/2218135289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10f00c6f60 0x7f10f00c0400 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f10d800bb10 tx=0x7f10d801ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a9bf3b23eca8f1c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:27.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10ee59c640 1 -- 172.21.15.46:0/2218135289 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d80a7bf0 con 0x7f10f00c6f60 2023-10-30T13:39:27.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10ee59c640 1 -- 172.21.15.46:0/2218135289 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d80a7d50 con 0x7f10f00c6f60 2023-10-30T13:39:27.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10ee59c640 1 -- 172.21.15.46:0/2218135289 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d80af660 con 0x7f10f00c6f60 2023-10-30T13:39:27.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10f5165640 1 -- 172.21.15.46:0/2218135289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10f00c6f60 msgr2=0x7f10f00c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10f5165640 1 --2- 172.21.15.46:0/2218135289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10f00c6f60 0x7f10f00c0400 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f10d800bb10 tx=0x7f10d801ec10 comp rx=0 tx=0).stop 2023-10-30T13:39:27.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10f5165640 1 -- 172.21.15.46:0/2218135289 shutdown_connections 2023-10-30T13:39:27.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10f5165640 1 --2- 172.21.15.46:0/2218135289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10f00c6f60 0x7f10f00c0400 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10f5165640 1 --2- 172.21.15.46:0/2218135289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10f00c6b80 0x7f10f00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10f5165640 1 -- 172.21.15.46:0/2218135289 >> 172.21.15.46:0/2218135289 conn(0x7f10f00c7910 msgr2=0x7f10f00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:27.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10f5165640 1 -- 172.21.15.46:0/2218135289 shutdown_connections 2023-10-30T13:39:27.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10f5165640 1 -- 172.21.15.46:0/2218135289 wait complete. 2023-10-30T13:39:27.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10f5165640 1 Processor -- start 2023-10-30T13:39:27.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.271+0000 7f10f5165640 1 -- start start 2023-10-30T13:39:27.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10f5165640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10f00c6b80 0x7f10f01f2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:27.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10f5165640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10f00c6f60 0x7f10f013aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:27.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10f5165640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10f00c8430 0x7f10f01555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:27.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10f5165640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0158470 con 0x7f10f00c6f60 2023-10-30T13:39:27.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10f5165640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f01f2f70 con 0x7f10f00c8430 2023-10-30T13:39:27.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10f5165640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0155150 con 0x7f10f00c6b80 2023-10-30T13:39:27.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10f00c6f60 0x7f10f013aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:27.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10f00c6f60 0x7f10f013aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47646/0 (socket says 172.21.15.46:47646) 2023-10-30T13:39:27.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10e7fff640 1 -- 172.21.15.46:0/3114285472 learned_addr learned my addr 172.21.15.46:0/3114285472 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:27.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10eed9d640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10f00c6b80 0x7f10f01f2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:27.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10ef59e640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10f00c8430 0x7f10f01555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10e7fff640 1 -- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10f00c6b80 msgr2=0x7f10f01f2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10e7fff640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10f00c6b80 0x7f10f01f2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10e7fff640 1 -- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10f00c8430 msgr2=0x7f10f01555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10e7fff640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10f00c8430 0x7f10f01555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10e7fff640 1 -- 172.21.15.46:0/3114285472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d800b770 con 0x7f10f00c6f60 2023-10-30T13:39:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10ef59e640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10f00c8430 0x7f10f01555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10e7fff640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10f00c6f60 0x7f10f013aad0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f10d800b2c0 tx=0x7f10d801b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10e77fe640 1 -- 172.21.15.46:0/3114285472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d80ba460 con 0x7f10f00c6f60 2023-10-30T13:39:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10e77fe640 1 -- 172.21.15.46:0/3114285472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d80ba5c0 con 0x7f10f00c6f60 2023-10-30T13:39:27.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10e77fe640 1 -- 172.21.15.46:0/3114285472 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d80ba970 con 0x7f10f00c6f60 2023-10-30T13:39:27.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10f5165640 1 -- 172.21.15.46:0/3114285472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f01f26d0 con 0x7f10f00c6f60 2023-10-30T13:39:27.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10f5165640 1 -- 172.21.15.46:0/3114285472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f01f2c90 con 0x7f10f00c6f60 2023-10-30T13:39:27.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.275+0000 7f10f5165640 1 -- 172.21.15.46:0/3114285472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f004e170 con 0x7f10f00c6f60 2023-10-30T13:39:27.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.279+0000 7f10e77fe640 1 -- 172.21.15.46:0/3114285472 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f10d80c2360 con 0x7f10f00c6f60 2023-10-30T13:39:27.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.279+0000 7f10e77fe640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f10d00664a0 0x7f10d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:27.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.279+0000 7f10e77fe640 1 -- 172.21.15.46:0/3114285472 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f10d811ec20 con 0x7f10f00c6f60 2023-10-30T13:39:27.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.279+0000 7f10eed9d640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f10d00664a0 0x7f10d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:27.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.279+0000 7f10eed9d640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f10d00664a0 0x7f10d0068960 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009b60 tx=0x7f10e0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:27.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.283+0000 7f10e77fe640 1 -- 172.21.15.46:0/3114285472 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f10d80ebd10 con 0x7f10f00c6f60 2023-10-30T13:39:27.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.475+0000 7f10f5165640 1 -- 172.21.15.46:0/3114285472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10f0155290 con 0x7f10f00c6f60 2023-10-30T13:39:27.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.475+0000 7f10e77fe640 1 -- 172.21.15.46:0/3114285472 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f10d80bf020 con 0x7f10f00c6f60 2023-10-30T13:39:27.480 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:27.480 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:26.986768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:27.480 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 84 2023-10-30T13:39:27.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.479+0000 7f10f5165640 1 -- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f10d00664a0 msgr2=0x7f10d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.479+0000 7f10f5165640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f10d00664a0 0x7f10d0068960 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009b60 tx=0x7f10e0009190 comp rx=0 tx=0).stop 2023-10-30T13:39:27.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.479+0000 7f10f5165640 1 -- 172.21.15.46:0/3114285472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10f00c6f60 msgr2=0x7f10f013aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:27.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.479+0000 7f10f5165640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10f00c6f60 0x7f10f013aad0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f10d800b2c0 tx=0x7f10d801b9b0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.479+0000 7f10f5165640 1 -- 172.21.15.46:0/3114285472 shutdown_connections 2023-10-30T13:39:27.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.479+0000 7f10f5165640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f10d00664a0 0x7f10d0068960 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.479+0000 7f10f5165640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10f00c8430 0x7f10f01555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.479+0000 7f10f5165640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10f00c6f60 0x7f10f013aad0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.479+0000 7f10f5165640 1 --2- 172.21.15.46:0/3114285472 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10f00c6b80 0x7f10f01f2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:27.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.479+0000 7f10f5165640 1 -- 172.21.15.46:0/3114285472 >> 172.21.15.46:0/3114285472 conn(0x7f10f00c7910 msgr2=0x7f10f00c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:27.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.479+0000 7f10f5165640 1 -- 172.21.15.46:0/3114285472 shutdown_connections 2023-10-30T13:39:27.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:27.479+0000 7f10f5165640 1 -- 172.21.15.46:0/3114285472 wait complete. 2023-10-30T13:39:27.501 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:29.503 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:29.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f2862e05640 1 Processor -- start 2023-10-30T13:39:29.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f2862e05640 1 -- start start 2023-10-30T13:39:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f2862e05640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f285c15d6f0 0x7f285c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f2862e05640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f285c0cf160 0x7f285c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f2862e05640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285c0cb800 con 0x7f285c1538a0 2023-10-30T13:39:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f2862e05640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285c0cb330 con 0x7f285c15d6f0 2023-10-30T13:39:29.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f2862e05640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0cad70 con 0x7f285c0cf160 2023-10-30T13:39:29.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f2860ba1640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f285c1538a0 0x7f285c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42192/0 (socket says 172.21.15.46:42192) 2023-10-30T13:39:29.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f2860ba1640 1 -- 172.21.15.46:0/2606868000 learned_addr learned my addr 172.21.15.46:0/2606868000 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:39:29.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285bfff640 1 --2- 172.21.15.46:0/2606868000 >> v2:172.21.15.154:3300/0 conn(0x7f285c15d6f0 0x7f285c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:29.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f28613a2640 1 --2- 172.21.15.46:0/2606868000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f285c0cf160 0x7f285c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:29.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442307386 0 0) 0x7f285c0cb800 con 0x7f285c1538a0 2023-10-30T13:39:29.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f284c003530 con 0x7f285c1538a0 2023-10-30T13:39:29.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 395826180 0 0) 0x7f284c003530 con 0x7f285c1538a0 2023-10-30T13:39:29.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f285c0cf160 msgr2=0x7f285c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:29.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 --2- 172.21.15.46:0/2606868000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f285c0cf160 0x7f285c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:29.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 >> v2:172.21.15.154:3300/0 conn(0x7f285c15d6f0 msgr2=0x7f285c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:29.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 --2- 172.21.15.46:0/2606868000 >> v2:172.21.15.154:3300/0 conn(0x7f285c15d6f0 0x7f285c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:29.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c0cb050 con 0x7f285c1538a0 2023-10-30T13:39:29.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f2850001680 con 0x7f285c1538a0 2023-10-30T13:39:29.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 >> v1:172.21.15.46:6789/0 conn(0x7f285c1538a0 legacy=0x7f285c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:29.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 --2- 172.21.15.46:0/2606868000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f284c003980 0x7f284c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:29.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 --2- 172.21.15.46:0/2606868000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f285c0cf160 0x7f284c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:29.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 --2- 172.21.15.46:0/2606868000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c004d10 0x7f284c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:29.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0cb330 con 0x7f284c003980 2023-10-30T13:39:29.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0cad70 con 0x7f284c004d10 2023-10-30T13:39:29.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.655+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284c00b990 con 0x7f285c0cf160 2023-10-30T13:39:29.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.659+0000 7f2860ba1640 1 --2- 172.21.15.46:0/2606868000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f284c003980 0x7f284c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:29.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.659+0000 7f28613a2640 1 --2- 172.21.15.46:0/2606868000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c004d10 0x7f284c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:29.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.659+0000 7f285bfff640 1 --2- 172.21.15.46:0/2606868000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f285c0cf160 0x7f284c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:29.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.659+0000 7f2860ba1640 1 -- 172.21.15.46:0/2606868000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f285c0cf160 msgr2=0x7f284c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:29.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.659+0000 7f2860ba1640 1 --2- 172.21.15.46:0/2606868000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f285c0cf160 0x7f284c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:29.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.659+0000 7f2860ba1640 1 -- 172.21.15.46:0/2606868000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c004d10 msgr2=0x7f284c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:29.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.659+0000 7f2860ba1640 1 --2- 172.21.15.46:0/2606868000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c004d10 0x7f284c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:29.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.659+0000 7f2860ba1640 1 -- 172.21.15.46:0/2606868000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c003530 con 0x7f284c003980 2023-10-30T13:39:29.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.659+0000 7f2860ba1640 1 --2- 172.21.15.46:0/2606868000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f284c003980 0x7f284c003da0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f285000c210 tx=0x7f285001f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7438b52c9d52c4a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:29.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.659+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850005ae0 con 0x7f284c003980 2023-10-30T13:39:29.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.659+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28500ab070 con 0x7f284c003980 2023-10-30T13:39:29.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.659+0000 7f285b7fe640 1 -- 172.21.15.46:0/2606868000 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850005ea0 con 0x7f284c003980 2023-10-30T13:39:29.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 -- 172.21.15.46:0/2606868000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f284c003980 msgr2=0x7f284c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:29.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 --2- 172.21.15.46:0/2606868000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f284c003980 0x7f284c003da0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f285000c210 tx=0x7f285001f3b0 comp rx=0 tx=0).stop 2023-10-30T13:39:29.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f28613a2640 1 -- 172.21.15.46:0/2606868000 reap_dead start 2023-10-30T13:39:29.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 -- 172.21.15.46:0/2606868000 shutdown_connections 2023-10-30T13:39:29.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 -- 172.21.15.46:0/2606868000 >> 172.21.15.46:0/2606868000 conn(0x7f285c06bfe0 msgr2=0x7f285c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:29.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 -- 172.21.15.46:0/2606868000 shutdown_connections 2023-10-30T13:39:29.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 -- 172.21.15.46:0/2606868000 wait complete. 2023-10-30T13:39:29.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 Processor -- start 2023-10-30T13:39:29.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 -- start start 2023-10-30T13:39:29.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f284c003980 0x7f285c1e3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:29.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f285c1e59a0 0x7f285c15c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:29.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f285c1f28c0 0x7f285c158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:29.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0b48b0 con 0x7f285c1e59a0 2023-10-30T13:39:29.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c148420 con 0x7f285c1f28c0 2023-10-30T13:39:29.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2862e05640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285c1e4e20 con 0x7f284c003980 2023-10-30T13:39:29.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f285bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f285c1e59a0 0x7f285c15c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:29.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f285bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f285c1e59a0 0x7f285c15c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47662/0 (socket says 172.21.15.46:47662) 2023-10-30T13:39:29.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f285bfff640 1 -- 172.21.15.46:0/2131716860 learned_addr learned my addr 172.21.15.46:0/2131716860 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:29.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f2860ba1640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f284c003980 0x7f285c1e3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:29.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.663+0000 7f28613a2640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f285c1f28c0 0x7f285c158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:29.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f285bfff640 1 -- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f284c003980 msgr2=0x7f285c1e3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:29.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f285bfff640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f284c003980 0x7f285c1e3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:29.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f285bfff640 1 -- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f285c1f28c0 msgr2=0x7f285c158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:29.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f285bfff640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f285c1f28c0 0x7f285c158d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:29.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f285bfff640 1 -- 172.21.15.46:0/2131716860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c157820 con 0x7f285c1e59a0 2023-10-30T13:39:29.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f285bfff640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f285c1e59a0 0x7f285c15c150 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2848011610 tx=0x7f2848011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:29.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f28597fa640 1 -- 172.21.15.46:0/2131716860 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848015870 con 0x7f285c1e59a0 2023-10-30T13:39:29.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f28597fa640 1 -- 172.21.15.46:0/2131716860 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28480159d0 con 0x7f285c1e59a0 2023-10-30T13:39:29.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f2862e05640 1 -- 172.21.15.46:0/2131716860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285c0b32b0 con 0x7f285c1e59a0 2023-10-30T13:39:29.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f28597fa640 1 -- 172.21.15.46:0/2131716860 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848015d40 con 0x7f285c1e59a0 2023-10-30T13:39:29.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f2862e05640 1 -- 172.21.15.46:0/2131716860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285c1e4580 con 0x7f285c1e59a0 2023-10-30T13:39:29.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f2862e05640 1 -- 172.21.15.46:0/2131716860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285c0b3110 con 0x7f285c1e59a0 2023-10-30T13:39:29.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f28597fa640 1 -- 172.21.15.46:0/2131716860 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f28480263f0 con 0x7f285c1e59a0 2023-10-30T13:39:29.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f28597fa640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28400664a0 0x7f2840068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:29.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.667+0000 7f28597fa640 1 -- 172.21.15.46:0/2131716860 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f2848084700 con 0x7f285c1e59a0 2023-10-30T13:39:29.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.671+0000 7f2860ba1640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28400664a0 0x7f2840068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:29.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.671+0000 7f2860ba1640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28400664a0 0x7f2840068960 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f28500ae000 tx=0x7f28500b4040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:29.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.671+0000 7f28597fa640 1 -- 172.21.15.46:0/2131716860 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f285c0b3110 con 0x7f285c1e59a0 2023-10-30T13:39:29.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 -- 172.21.15.46:0/2131716860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f285c157820 con 0x7f285c1e59a0 2023-10-30T13:39:29.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f28597fa640 1 -- 172.21.15.46:0/2131716860 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f285c157820 con 0x7f285c1e59a0 2023-10-30T13:39:29.866 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:29.866 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:26.986768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:29.868 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 84 2023-10-30T13:39:29.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 -- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28400664a0 msgr2=0x7f2840068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:29.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28400664a0 0x7f2840068960 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f28500ae000 tx=0x7f28500b4040 comp rx=0 tx=0).stop 2023-10-30T13:39:29.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 -- 172.21.15.46:0/2131716860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f285c1e59a0 msgr2=0x7f285c15c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:29.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f285c1e59a0 0x7f285c15c150 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2848011610 tx=0x7f2848011ad0 comp rx=0 tx=0).stop 2023-10-30T13:39:29.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 -- 172.21.15.46:0/2131716860 shutdown_connections 2023-10-30T13:39:29.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28400664a0 0x7f2840068960 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:29.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f285c1f28c0 0x7f285c158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:29.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f285c1e59a0 0x7f285c15c150 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:29.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 --2- 172.21.15.46:0/2131716860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f284c003980 0x7f285c1e3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:29.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 -- 172.21.15.46:0/2131716860 >> 172.21.15.46:0/2131716860 conn(0x7f285c06bfe0 msgr2=0x7f285c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:29.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 -- 172.21.15.46:0/2131716860 shutdown_connections 2023-10-30T13:39:29.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:29.863+0000 7f2862e05640 1 -- 172.21.15.46:0/2131716860 wait complete. 2023-10-30T13:39:29.887 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:31.888 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.039+0000 7fda32b65640 1 Processor -- start 2023-10-30T13:39:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.039+0000 7fda32b65640 1 -- start start 2023-10-30T13:39:32.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.043+0000 7fda32b65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda2c0c6bd0 0x7fda2c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.043+0000 7fda32b65640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2c066cf0 con 0x7fda2c06f7a0 2023-10-30T13:39:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.043+0000 7fda32b65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2c0c70d0 con 0x7fda2c071110 2023-10-30T13:39:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.043+0000 7fda32b65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2c0700c0 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.043+0000 7fda23fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda2c0c6bd0 0x7fda2c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:32.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.043+0000 7fda23fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda2c0c6bd0 0x7fda2c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47678/0 (socket says 172.21.15.46:47678) 2023-10-30T13:39:32.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.043+0000 7fda23fff640 1 -- 172.21.15.46:0/2029782311 learned_addr learned my addr 172.21.15.46:0/2029782311 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:32.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.043+0000 7fda23fff640 1 -- 172.21.15.46:0/2029782311 >> v1:172.21.15.154:6789/0 conn(0x7fda2c071110 legacy=0x7fda2c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.043+0000 7fda23fff640 1 -- 172.21.15.46:0/2029782311 >> v1:172.21.15.46:6789/0 conn(0x7fda2c06f7a0 legacy=0x7fda2c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.043+0000 7fda23fff640 1 -- 172.21.15.46:0/2029782311 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c0c7390 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda23fff640 1 --2- 172.21.15.46:0/2029782311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2c0c6bd0 0x7fda2c06fb80 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fda1c009870 tx=0x7fda1c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929233a42eb23b4b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:32.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda237fe640 1 -- 172.21.15.46:0/2029782311 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c0a8a30 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda237fe640 1 -- 172.21.15.46:0/2029782311 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c0a8b90 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda237fe640 1 -- 172.21.15.46:0/2029782311 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c0b0440 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda32b65640 1 -- 172.21.15.46:0/2029782311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2c0c6bd0 msgr2=0x7fda2c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:32.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda32b65640 1 --2- 172.21.15.46:0/2029782311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2c0c6bd0 0x7fda2c06fb80 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fda1c009870 tx=0x7fda1c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:39:32.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda32b65640 1 -- 172.21.15.46:0/2029782311 shutdown_connections 2023-10-30T13:39:32.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda32b65640 1 --2- 172.21.15.46:0/2029782311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2c0c6bd0 0x7fda2c06fb80 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda32b65640 1 -- 172.21.15.46:0/2029782311 >> 172.21.15.46:0/2029782311 conn(0x7fda2c0c7910 msgr2=0x7fda2c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda32b65640 1 -- 172.21.15.46:0/2029782311 shutdown_connections 2023-10-30T13:39:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda32b65640 1 -- 172.21.15.46:0/2029782311 wait complete. 2023-10-30T13:39:32.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda32b65640 1 Processor -- start 2023-10-30T13:39:32.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.047+0000 7fda32b65640 1 -- start start 2023-10-30T13:39:32.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda32b65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda2c06f7a0 0x7fda2c1e4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:32.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda32b65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda2c071110 0x7fda2c1f01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:32.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda32b65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2c0c6bd0 0x7fda2c148b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:32.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda32b65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c1564a0 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda32b65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c155a80 con 0x7fda2c06f7a0 2023-10-30T13:39:32.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda32b65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c153020 con 0x7fda2c071110 2023-10-30T13:39:32.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda31102640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2c0c6bd0 0x7fda2c148b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:32.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda31102640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2c0c6bd0 0x7fda2c148b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47692/0 (socket says 172.21.15.46:47692) 2023-10-30T13:39:32.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda31102640 1 -- 172.21.15.46:0/111424616 learned_addr learned my addr 172.21.15.46:0/111424616 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:32.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda30901640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda2c06f7a0 0x7fda2c1e4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:32.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda23fff640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda2c071110 0x7fda2c1f01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:32.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda31102640 1 -- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda2c071110 msgr2=0x7fda2c1f01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:32.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda31102640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda2c071110 0x7fda2c1f01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:32.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda31102640 1 -- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda2c06f7a0 msgr2=0x7fda2c1e4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:32.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda31102640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda2c06f7a0 0x7fda2c1e4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:32.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda31102640 1 -- 172.21.15.46:0/111424616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c0094d0 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda30901640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda2c06f7a0 0x7fda2c1e4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:32.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda23fff640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda2c071110 0x7fda2c1f01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:32.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda31102640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2c0c6bd0 0x7fda2c148b00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fda2400c760 tx=0x7fda2400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:32.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda217fa640 1 -- 172.21.15.46:0/111424616 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24003b80 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda217fa640 1 -- 172.21.15.46:0/111424616 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda24003ce0 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda217fa640 1 -- 172.21.15.46:0/111424616 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2400aa30 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda32b65640 1 -- 172.21.15.46:0/111424616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2c152a60 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda32b65640 1 -- 172.21.15.46:0/111424616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2c155430 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda32b65640 1 -- 172.21.15.46:0/111424616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2c04e170 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.051+0000 7fda217fa640 1 -- 172.21.15.46:0/111424616 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fda2400ab90 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.055+0000 7fda217fa640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda180664a0 0x7fda18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:32.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.055+0000 7fda217fa640 1 -- 172.21.15.46:0/111424616 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fda24083f60 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.055+0000 7fda30901640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda180664a0 0x7fda18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:32.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.055+0000 7fda30901640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda180664a0 0x7fda18068960 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fda28000f70 tx=0x7fda28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:32.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.059+0000 7fda217fa640 1 -- 172.21.15.46:0/111424616 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fda240510d0 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 -- 172.21.15.46:0/111424616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda2c155bc0 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda217fa640 1 -- 172.21.15.46:0/111424616 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5747 (secure 0 0 0) 0x7fda240546f0 con 0x7fda2c0c6bd0 2023-10-30T13:39:32.254 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:32.254 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:26.986768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:32.255 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 84 2023-10-30T13:39:32.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 -- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda180664a0 msgr2=0x7fda18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:32.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda180664a0 0x7fda18068960 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fda28000f70 tx=0x7fda28009040 comp rx=0 tx=0).stop 2023-10-30T13:39:32.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 -- 172.21.15.46:0/111424616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2c0c6bd0 msgr2=0x7fda2c148b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:32.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2c0c6bd0 0x7fda2c148b00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fda2400c760 tx=0x7fda2400cc20 comp rx=0 tx=0).stop 2023-10-30T13:39:32.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 -- 172.21.15.46:0/111424616 shutdown_connections 2023-10-30T13:39:32.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda180664a0 0x7fda18068960 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:32.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2c0c6bd0 0x7fda2c148b00 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:32.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda2c071110 0x7fda2c1f01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:32.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 --2- 172.21.15.46:0/111424616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda2c06f7a0 0x7fda2c1e4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:32.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 -- 172.21.15.46:0/111424616 >> 172.21.15.46:0/111424616 conn(0x7fda2c0c7910 msgr2=0x7fda2c0c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:32.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 -- 172.21.15.46:0/111424616 shutdown_connections 2023-10-30T13:39:32.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:32.251+0000 7fda32b65640 1 -- 172.21.15.46:0/111424616 wait complete. 2023-10-30T13:39:32.275 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:34.276 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:34.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.427+0000 7f5584679640 1 Processor -- start 2023-10-30T13:39:34.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.427+0000 7f5584679640 1 -- start start 2023-10-30T13:39:34.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5584679640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f557c153bf0 0x7f557c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:34.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5584679640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f557c0c5f60 0x7f557c0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:34.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5584679640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557c0700b0 con 0x7f557c0c6fe0 2023-10-30T13:39:34.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5584679640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f557c0c6810 con 0x7f557c0c5f60 2023-10-30T13:39:34.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5584679640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557c159900 con 0x7f557c153bf0 2023-10-30T13:39:34.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5582c16640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f557c0c5f60 0x7f557c0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:34.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5582c16640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f557c0c5f60 0x7f557c0cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47698/0 (socket says 172.21.15.46:47698) 2023-10-30T13:39:34.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5582c16640 1 -- 172.21.15.46:0/2167147230 learned_addr learned my addr 172.21.15.46:0/2167147230 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:34.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5582415640 1 --2- 172.21.15.46:0/2167147230 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f557c153bf0 0x7f557c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:34.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5582c16640 1 -- 172.21.15.46:0/2167147230 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f557c153bf0 msgr2=0x7f557c071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:34.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5582c16640 1 --2- 172.21.15.46:0/2167147230 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f557c153bf0 0x7f557c071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:34.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5582c16640 1 -- 172.21.15.46:0/2167147230 >> v1:172.21.15.154:6789/0 conn(0x7f557c0c6fe0 legacy=0x7f557c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:34.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5582c16640 1 -- 172.21.15.46:0/2167147230 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c0c6530 con 0x7f557c0c5f60 2023-10-30T13:39:34.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5582c16640 1 --2- 172.21.15.46:0/2167147230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c0c5f60 0x7f557c0cb570 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5574009e50 tx=0x7f557401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf6384713fc4d49 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:34.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.431+0000 7f5581413640 1 -- 172.21.15.46:0/2167147230 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55740af400 con 0x7f557c0c5f60 2023-10-30T13:39:34.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5581413640 1 -- 172.21.15.46:0/2167147230 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55740ab050 con 0x7f557c0c5f60 2023-10-30T13:39:34.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5581413640 1 -- 172.21.15.46:0/2167147230 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55740af780 con 0x7f557c0c5f60 2023-10-30T13:39:34.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 -- 172.21.15.46:0/2167147230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c0c5f60 msgr2=0x7f557c0cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:34.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 --2- 172.21.15.46:0/2167147230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c0c5f60 0x7f557c0cb570 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5574009e50 tx=0x7f557401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:39:34.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 -- 172.21.15.46:0/2167147230 shutdown_connections 2023-10-30T13:39:34.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 --2- 172.21.15.46:0/2167147230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c0c5f60 0x7f557c0cb570 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:34.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 --2- 172.21.15.46:0/2167147230 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f557c153bf0 0x7f557c071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 -- 172.21.15.46:0/2167147230 >> 172.21.15.46:0/2167147230 conn(0x7f557c06c6e0 msgr2=0x7f557c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 -- 172.21.15.46:0/2167147230 shutdown_connections 2023-10-30T13:39:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 -- 172.21.15.46:0/2167147230 wait complete. 2023-10-30T13:39:34.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 Processor -- start 2023-10-30T13:39:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 -- start start 2023-10-30T13:39:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f557c0c5f60 0x7f557c148390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c0c6fe0 0x7f557c1f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f557c153bf0 0x7f557c0c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557c1f2850 con 0x7f557c0c6fe0 2023-10-30T13:39:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557c1570a0 con 0x7f557c0c5f60 2023-10-30T13:39:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5584679640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557c0ca040 con 0x7f557c153bf0 2023-10-30T13:39:34.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5581c14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c0c6fe0 0x7f557c1f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:34.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5581c14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c0c6fe0 0x7f557c1f2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47702/0 (socket says 172.21.15.46:47702) 2023-10-30T13:39:34.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5581c14640 1 -- 172.21.15.46:0/3867674610 learned_addr learned my addr 172.21.15.46:0/3867674610 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:34.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.435+0000 7f5582415640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f557c0c5f60 0x7f557c148390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:34.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5582c16640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f557c153bf0 0x7f557c0c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:34.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5581c14640 1 -- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f557c153bf0 msgr2=0x7f557c0c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:34.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5581c14640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f557c153bf0 0x7f557c0c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:34.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5581c14640 1 -- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f557c0c5f60 msgr2=0x7f557c148390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:34.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5581c14640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f557c0c5f60 0x7f557c148390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:34.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5581c14640 1 -- 172.21.15.46:0/3867674610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5574009b30 con 0x7f557c0c6fe0 2023-10-30T13:39:34.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5581c14640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c0c6fe0 0x7f557c1f2ce0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f556c00c500 tx=0x7f556c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5572ffd640 1 -- 172.21.15.46:0/3867674610 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c005220 con 0x7f557c0c6fe0 2023-10-30T13:39:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5572ffd640 1 -- 172.21.15.46:0/3867674610 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556c005380 con 0x7f557c0c6fe0 2023-10-30T13:39:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5572ffd640 1 -- 172.21.15.46:0/3867674610 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c007270 con 0x7f557c0c6fe0 2023-10-30T13:39:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5584679640 1 -- 172.21.15.46:0/3867674610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f557c1f2220 con 0x7f557c0c6fe0 2023-10-30T13:39:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5584679640 1 -- 172.21.15.46:0/3867674610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f557c0c4130 con 0x7f557c0c6fe0 2023-10-30T13:39:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5584679640 1 -- 172.21.15.46:0/3867674610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f557c04e170 con 0x7f557c0c6fe0 2023-10-30T13:39:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5572ffd640 1 -- 172.21.15.46:0/3867674610 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f556c0073d0 con 0x7f557c0c6fe0 2023-10-30T13:39:34.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5572ffd640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f55600664a0 0x7f5560068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:34.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.439+0000 7f5572ffd640 1 -- 172.21.15.46:0/3867674610 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f556c083a70 con 0x7f557c0c6fe0 2023-10-30T13:39:34.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.443+0000 7f5582415640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f55600664a0 0x7f5560068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:34.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.443+0000 7f5582415640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f55600664a0 0x7f5560068960 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f55780079c0 tx=0x7f5578008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:34.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.443+0000 7f5572ffd640 1 -- 172.21.15.46:0/3867674610 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f556c050b70 con 0x7f557c0c6fe0 2023-10-30T13:39:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 -- 172.21.15.46:0/3867674610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f557c06fe90 con 0x7f557c0c6fe0 2023-10-30T13:39:34.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5572ffd640 1 -- 172.21.15.46:0/3867674610 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5753 (secure 0 0 0) 0x7f556c054190 con 0x7f557c0c6fe0 2023-10-30T13:39:34.638 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:34.638 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:33.032103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 85 2023-10-30T13:39:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 -- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f55600664a0 msgr2=0x7f5560068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f55600664a0 0x7f5560068960 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f55780079c0 tx=0x7f5578008040 comp rx=0 tx=0).stop 2023-10-30T13:39:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 -- 172.21.15.46:0/3867674610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c0c6fe0 msgr2=0x7f557c1f2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c0c6fe0 0x7f557c1f2ce0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f556c00c500 tx=0x7f556c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:39:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 -- 172.21.15.46:0/3867674610 shutdown_connections 2023-10-30T13:39:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f55600664a0 0x7f5560068960 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f557c153bf0 0x7f557c0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c0c6fe0 0x7f557c1f2ce0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 --2- 172.21.15.46:0/3867674610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f557c0c5f60 0x7f557c148390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 -- 172.21.15.46:0/3867674610 >> 172.21.15.46:0/3867674610 conn(0x7f557c06c6e0 msgr2=0x7f557c157d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 -- 172.21.15.46:0/3867674610 shutdown_connections 2023-10-30T13:39:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:34.635+0000 7f5584679640 1 -- 172.21.15.46:0/3867674610 wait complete. 2023-10-30T13:39:34.659 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 85, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 85, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5721": { "addr": "172.21.15.46:6833/4044310929", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4044310929, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4044310929, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5721, "incarnation": 61, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 24 }, "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 28 }, "gid_6036": { "addr": "172.21.15.46:6836/3037060424", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3037060424, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3037060424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6036, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:clientreplay", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:39:33.032103+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5721, "mds_1": 6036, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:39:36.660 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:36.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.811+0000 7f73c0617640 1 Processor -- start 2023-10-30T13:39:36.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.811+0000 7f73c0617640 1 -- start start 2023-10-30T13:39:36.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73c0617640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73b80c05d0 0x7f73b813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:36.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73c0617640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b80c0a50 con 0x7f73b80c6f60 2023-10-30T13:39:36.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73c0617640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b806f6e0 con 0x7f73b80c6b80 2023-10-30T13:39:36.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73c0617640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b806f9c0 con 0x7f73b80c05d0 2023-10-30T13:39:36.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73bebb4640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f73b80c6f60 0x7f73b81566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54572/0 (socket says 172.21.15.46:54572) 2023-10-30T13:39:36.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73bebb4640 1 -- 172.21.15.46:0/634258114 learned_addr learned my addr 172.21.15.46:0/634258114 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:39:36.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73bdbb2640 1 --2- 172.21.15.46:0/634258114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73b80c05d0 0x7f73b813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:36.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73bd3b1640 1 -- 172.21.15.46:0/634258114 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193065559 0 0) 0x7f73b80c0a50 con 0x7f73b80c6f60 2023-10-30T13:39:36.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73bd3b1640 1 -- 172.21.15.46:0/634258114 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a4003530 con 0x7f73b80c6f60 2023-10-30T13:39:36.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73bd3b1640 1 -- 172.21.15.46:0/634258114 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370786279 0 0) 0x7f73b806f6e0 con 0x7f73b80c6b80 2023-10-30T13:39:36.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73bd3b1640 1 -- 172.21.15.46:0/634258114 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b80c0a50 con 0x7f73b80c6b80 2023-10-30T13:39:36.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73bd3b1640 1 -- 172.21.15.46:0/634258114 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1413870792 0 0) 0x7f73a4003530 con 0x7f73b80c6f60 2023-10-30T13:39:36.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73bdbb2640 1 -- 172.21.15.46:0/634258114 >> v1:172.21.15.154:6789/0 conn(0x7f73b80c6b80 legacy=0x7f73b80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:36.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73bdbb2640 1 -- 172.21.15.46:0/634258114 >> v1:172.21.15.46:6789/0 conn(0x7f73b80c6f60 legacy=0x7f73b81566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:36.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.815+0000 7f73bdbb2640 1 -- 172.21.15.46:0/634258114 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b806fca0 con 0x7f73b80c05d0 2023-10-30T13:39:36.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73bdbb2640 1 --2- 172.21.15.46:0/634258114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73b80c05d0 0x7f73b813ed90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f73a80097f0 tx=0x7f73a801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c95f3d3bf1e0c664 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:36.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73bd3b1640 1 -- 172.21.15.46:0/634258114 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0001a00 con 0x7f73b80c05d0 2023-10-30T13:39:36.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73bd3b1640 1 -- 172.21.15.46:0/634258114 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a80a7050 con 0x7f73b80c05d0 2023-10-30T13:39:36.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73bd3b1640 1 -- 172.21.15.46:0/634258114 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a80ab400 con 0x7f73b80c05d0 2023-10-30T13:39:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73c0617640 1 -- 172.21.15.46:0/634258114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73b80c05d0 msgr2=0x7f73b813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73c0617640 1 --2- 172.21.15.46:0/634258114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73b80c05d0 0x7f73b813ed90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f73a80097f0 tx=0x7f73a801c3e0 comp rx=0 tx=0).stop 2023-10-30T13:39:36.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73c0617640 1 -- 172.21.15.46:0/634258114 shutdown_connections 2023-10-30T13:39:36.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73c0617640 1 --2- 172.21.15.46:0/634258114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73b80c05d0 0x7f73b813ed90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:36.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73c0617640 1 -- 172.21.15.46:0/634258114 >> 172.21.15.46:0/634258114 conn(0x7f73b80c7910 msgr2=0x7f73b80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:36.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73c0617640 1 -- 172.21.15.46:0/634258114 shutdown_connections 2023-10-30T13:39:36.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73c0617640 1 -- 172.21.15.46:0/634258114 wait complete. 2023-10-30T13:39:36.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73c0617640 1 Processor -- start 2023-10-30T13:39:36.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.819+0000 7f73c0617640 1 -- start start 2023-10-30T13:39:36.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73c0617640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73b80c05d0 0x7f73b8158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:36.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73c0617640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73b80c6b80 0x7f73b813d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:36.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73c0617640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73b80c6f60 0x7f73b813d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:36.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73c0617640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b8136910 con 0x7f73b80c6b80 2023-10-30T13:39:36.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73c0617640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b813cd20 con 0x7f73b80c6f60 2023-10-30T13:39:36.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73c0617640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b81552e0 con 0x7f73b80c05d0 2023-10-30T13:39:36.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73bdbb2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73b80c6b80 0x7f73b813d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:36.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73bdbb2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73b80c6b80 0x7f73b813d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53908/0 (socket says 172.21.15.46:53908) 2023-10-30T13:39:36.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73bdbb2640 1 -- 172.21.15.46:0/3584720266 learned_addr learned my addr 172.21.15.46:0/3584720266 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:36.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73be3b3640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73b80c05d0 0x7f73b8158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:36.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73bebb4640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73b80c6f60 0x7f73b813d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:36.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73bdbb2640 1 -- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73b80c05d0 msgr2=0x7f73b8158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:36.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73bdbb2640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73b80c05d0 0x7f73b8158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:36.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73bdbb2640 1 -- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73b80c6f60 msgr2=0x7f73b813d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:36.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73bdbb2640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73b80c6f60 0x7f73b813d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:36.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73bdbb2640 1 -- 172.21.15.46:0/3584720266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a8009420 con 0x7f73b80c6b80 2023-10-30T13:39:36.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73bdbb2640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73b80c6b80 0x7f73b813d1b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f73a80097f0 tx=0x7f73a8019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:36.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73aeffd640 1 -- 172.21.15.46:0/3584720266 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a80abd00 con 0x7f73b80c6b80 2023-10-30T13:39:36.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73aeffd640 1 -- 172.21.15.46:0/3584720266 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a80abe60 con 0x7f73b80c6b80 2023-10-30T13:39:36.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73aeffd640 1 -- 172.21.15.46:0/3584720266 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a80b4620 con 0x7f73b80c6b80 2023-10-30T13:39:36.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73c0617640 1 -- 172.21.15.46:0/3584720266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b81396f0 con 0x7f73b80c6b80 2023-10-30T13:39:36.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73c0617640 1 -- 172.21.15.46:0/3584720266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b813c6d0 con 0x7f73b80c6b80 2023-10-30T13:39:36.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.823+0000 7f73c0617640 1 -- 172.21.15.46:0/3584720266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b804e170 con 0x7f73b80c6b80 2023-10-30T13:39:36.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.827+0000 7f73aeffd640 1 -- 172.21.15.46:0/3584720266 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f73a80b2040 con 0x7f73b80c6b80 2023-10-30T13:39:36.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.827+0000 7f73aeffd640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f739c0664a0 0x7f739c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:36.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.827+0000 7f73aeffd640 1 -- 172.21.15.46:0/3584720266 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f73a811ddf0 con 0x7f73b80c6b80 2023-10-30T13:39:36.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.827+0000 7f73be3b3640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f739c0664a0 0x7f739c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:36.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.827+0000 7f73be3b3640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f739c0664a0 0x7f739c068960 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f73b40011f0 tx=0x7f73b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:36.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:36.831+0000 7f73aeffd640 1 -- 172.21.15.46:0/3584720266 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f73a80eae50 con 0x7f73b80c6b80 2023-10-30T13:39:37.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 -- 172.21.15.46:0/3584720266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73b813ce60 con 0x7f73b80c6b80 2023-10-30T13:39:37.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73aeffd640 1 -- 172.21.15.46:0/3584720266 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5753 (secure 0 0 0) 0x7f73a80b0030 con 0x7f73b80c6b80 2023-10-30T13:39:37.030 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:37.030 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:33.032103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:37.030 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 85 2023-10-30T13:39:37.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 -- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f739c0664a0 msgr2=0x7f739c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f739c0664a0 0x7f739c068960 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f73b40011f0 tx=0x7f73b4009040 comp rx=0 tx=0).stop 2023-10-30T13:39:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 -- 172.21.15.46:0/3584720266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73b80c6b80 msgr2=0x7f73b813d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73b80c6b80 0x7f73b813d1b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f73a80097f0 tx=0x7f73a8019180 comp rx=0 tx=0).stop 2023-10-30T13:39:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 -- 172.21.15.46:0/3584720266 shutdown_connections 2023-10-30T13:39:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f739c0664a0 0x7f739c068960 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73b80c6f60 0x7f73b813d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73b80c6b80 0x7f73b813d1b0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 --2- 172.21.15.46:0/3584720266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73b80c05d0 0x7f73b8158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 -- 172.21.15.46:0/3584720266 >> 172.21.15.46:0/3584720266 conn(0x7f73b80c7910 msgr2=0x7f73b813a790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 -- 172.21.15.46:0/3584720266 shutdown_connections 2023-10-30T13:39:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:37.027+0000 7f73c0617640 1 -- 172.21.15.46:0/3584720266 wait complete. 2023-10-30T13:39:37.051 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:39:39.052 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.203+0000 7fe356491640 1 Processor -- start 2023-10-30T13:39:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe356491640 1 -- start start 2023-10-30T13:39:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe356491640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3500c05d0 0x7fe35013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe356491640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3500c6f60 0x7fe35013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe356491640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3500c0a50 con 0x7fe3500c6b80 2023-10-30T13:39:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe356491640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe35006f6e0 con 0x7fe3500c05d0 2023-10-30T13:39:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe356491640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35006f9c0 con 0x7fe3500c6f60 2023-10-30T13:39:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe34f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3500c05d0 0x7fe35013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe34f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3500c05d0 0x7fe35013ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53910/0 (socket says 172.21.15.46:53910) 2023-10-30T13:39:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe34f7fe640 1 -- 172.21.15.46:0/3648617787 learned_addr learned my addr 172.21.15.46:0/3648617787 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe354a2e640 1 --2- 172.21.15.46:0/3648617787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3500c6f60 0x7fe35013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe34f7fe640 1 -- 172.21.15.46:0/3648617787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3500c6f60 msgr2=0x7fe35013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe34f7fe640 1 --2- 172.21.15.46:0/3648617787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3500c6f60 0x7fe35013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe34f7fe640 1 -- 172.21.15.46:0/3648617787 >> v1:172.21.15.46:6789/0 conn(0x7fe3500c6b80 legacy=0x7fe3500bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.207+0000 7fe34f7fe640 1 -- 172.21.15.46:0/3648617787 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35006fca0 con 0x7fe3500c05d0 2023-10-30T13:39:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe34f7fe640 1 --2- 172.21.15.46:0/3648617787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3500c05d0 0x7fe35013ed90 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe344009870 tx=0x7fe34401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8ea5a29e01bfec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe34effd640 1 -- 172.21.15.46:0/3648617787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3440ab650 con 0x7fe3500c05d0 2023-10-30T13:39:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe34effd640 1 -- 172.21.15.46:0/3648617787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3440ab7b0 con 0x7fe3500c05d0 2023-10-30T13:39:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe34effd640 1 -- 172.21.15.46:0/3648617787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3440abb40 con 0x7fe3500c05d0 2023-10-30T13:39:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe356491640 1 -- 172.21.15.46:0/3648617787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3500c05d0 msgr2=0x7fe35013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe356491640 1 --2- 172.21.15.46:0/3648617787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3500c05d0 0x7fe35013ed90 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe344009870 tx=0x7fe34401c540 comp rx=0 tx=0).stop 2023-10-30T13:39:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe356491640 1 -- 172.21.15.46:0/3648617787 shutdown_connections 2023-10-30T13:39:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe356491640 1 --2- 172.21.15.46:0/3648617787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3500c6f60 0x7fe35013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe356491640 1 --2- 172.21.15.46:0/3648617787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3500c05d0 0x7fe35013ed90 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe356491640 1 -- 172.21.15.46:0/3648617787 >> 172.21.15.46:0/3648617787 conn(0x7fe3500c7910 msgr2=0x7fe3500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe356491640 1 -- 172.21.15.46:0/3648617787 shutdown_connections 2023-10-30T13:39:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe356491640 1 -- 172.21.15.46:0/3648617787 wait complete. 2023-10-30T13:39:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe356491640 1 Processor -- start 2023-10-30T13:39:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.211+0000 7fe356491640 1 -- start start 2023-10-30T13:39:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe356491640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3500c05d0 0x7fe35013a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe356491640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3500c6b80 0x7fe3500c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe356491640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3500c6f60 0x7fe3501526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe356491640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3500ca650 con 0x7fe3500c6f60 2023-10-30T13:39:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe356491640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe350135d80 con 0x7fe3500c6b80 2023-10-30T13:39:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe356491640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe350136270 con 0x7fe3500c05d0 2023-10-30T13:39:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe354a2e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3500c6f60 0x7fe3501526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe354a2e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3500c6f60 0x7fe3501526c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53916/0 (socket says 172.21.15.46:53916) 2023-10-30T13:39:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe354a2e640 1 -- 172.21.15.46:0/4153640551 learned_addr learned my addr 172.21.15.46:0/4153640551 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe34f7fe640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3500c6b80 0x7fe3500c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe34ffff640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3500c05d0 0x7fe35013a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe354a2e640 1 -- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3500c05d0 msgr2=0x7fe35013a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe354a2e640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3500c05d0 0x7fe35013a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe354a2e640 1 -- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3500c6b80 msgr2=0x7fe3500c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe354a2e640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3500c6b80 0x7fe3500c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe354a2e640 1 -- 172.21.15.46:0/4153640551 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3440094d0 con 0x7fe3500c6f60 2023-10-30T13:39:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe354a2e640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3500c6f60 0x7fe3501526c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe348007e60 tx=0x7fe34800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe34cff9640 1 -- 172.21.15.46:0/4153640551 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348014420 con 0x7fe3500c6f60 2023-10-30T13:39:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe34cff9640 1 -- 172.21.15.46:0/4153640551 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3480145a0 con 0x7fe3500c6f60 2023-10-30T13:39:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe356491640 1 -- 172.21.15.46:0/4153640551 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35013d340 con 0x7fe3500c6f60 2023-10-30T13:39:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe34cff9640 1 -- 172.21.15.46:0/4153640551 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348014950 con 0x7fe3500c6f60 2023-10-30T13:39:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe356491640 1 -- 172.21.15.46:0/4153640551 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3500c8d40 con 0x7fe3500c6f60 2023-10-30T13:39:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.215+0000 7fe356491640 1 -- 172.21.15.46:0/4153640551 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35013d4d0 con 0x7fe3500c6f60 2023-10-30T13:39:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.219+0000 7fe34cff9640 1 -- 172.21.15.46:0/4153640551 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe348014ab0 con 0x7fe3500c6f60 2023-10-30T13:39:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.219+0000 7fe34cff9640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe33c0664f0 0x7fe33c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.219+0000 7fe34cff9640 1 -- 172.21.15.46:0/4153640551 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fe348084a20 con 0x7fe3500c6f60 2023-10-30T13:39:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.219+0000 7fe34ffff640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe33c0664f0 0x7fe33c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.219+0000 7fe34ffff640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe33c0664f0 0x7fe33c0689b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe340000f70 tx=0x7fe340009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.223+0000 7fe34cff9640 1 -- 172.21.15.46:0/4153640551 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe348051b70 con 0x7fe3500c6f60 2023-10-30T13:39:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 -- 172.21.15.46:0/4153640551 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe35013c3d0 con 0x7fe3500c6f60 2023-10-30T13:39:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe34cff9640 1 -- 172.21.15.46:0/4153640551 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5747 (secure 0 0 0) 0x7fe348055190 con 0x7fe3500c6f60 2023-10-30T13:39:39.418 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:39.418 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:38.428153+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 86 2023-10-30T13:39:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 -- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe33c0664f0 msgr2=0x7fe33c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe33c0664f0 0x7fe33c0689b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe340000f70 tx=0x7fe340009040 comp rx=0 tx=0).stop 2023-10-30T13:39:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 -- 172.21.15.46:0/4153640551 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3500c6f60 msgr2=0x7fe3501526c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3500c6f60 0x7fe3501526c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe348007e60 tx=0x7fe34800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 -- 172.21.15.46:0/4153640551 shutdown_connections 2023-10-30T13:39:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe33c0664f0 0x7fe33c0689b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3500c6f60 0x7fe3501526c0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3500c6b80 0x7fe3500c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 --2- 172.21.15.46:0/4153640551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3500c05d0 0x7fe35013a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 -- 172.21.15.46:0/4153640551 >> 172.21.15.46:0/4153640551 conn(0x7fe3500c7910 msgr2=0x7fe3501532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 -- 172.21.15.46:0/4153640551 shutdown_connections 2023-10-30T13:39:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:39.415+0000 7fe356491640 1 -- 172.21.15.46:0/4153640551 wait complete. 2023-10-30T13:39:39.439 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=5574 2023-10-30T13:39:39.439 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.e 2023-10-30T13:39:41.440 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T13:39:41.441 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:39:41.441 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:39:41.445 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:39:41.446 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:41.543 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T13:39:41.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff1185aa640 1 Processor -- start 2023-10-30T13:39:41.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff1185aa640 1 -- start start 2023-10-30T13:39:41.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff1185aa640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff110071110 0x7ff11006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff1185aa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff110066cf0 con 0x7ff1100c6bd0 2023-10-30T13:39:41.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff1185aa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1100c70d0 con 0x7ff11006f7a0 2023-10-30T13:39:41.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff1185aa640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1100700c0 con 0x7ff110071110 2023-10-30T13:39:41.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115b45640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff1100c6bd0 0x7ff110158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54592/0 (socket says 172.21.15.46:54592) 2023-10-30T13:39:41.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115b45640 1 -- 172.21.15.46:0/2641116819 learned_addr learned my addr 172.21.15.46:0/2641116819 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:39:41.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff116b47640 1 --2- 172.21.15.46:0/2641116819 >> v2:172.21.15.154:3300/0 conn(0x7ff110071110 0x7ff11006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330291396 0 0) 0x7ff110066cf0 con 0x7ff1100c6bd0 2023-10-30T13:39:41.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0fc003530 con 0x7ff1100c6bd0 2023-10-30T13:39:41.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1449132898 0 0) 0x7ff0fc003530 con 0x7ff1100c6bd0 2023-10-30T13:39:41.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 >> v2:172.21.15.154:3300/0 conn(0x7ff110071110 msgr2=0x7ff11006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:41.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 --2- 172.21.15.46:0/2641116819 >> v2:172.21.15.154:3300/0 conn(0x7ff110071110 0x7ff11006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 >> v1:172.21.15.154:6789/0 conn(0x7ff11006f7a0 legacy=0x7ff110069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1100c7390 con 0x7ff1100c6bd0 2023-10-30T13:39:41.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7ff100001680 con 0x7ff1100c6bd0 2023-10-30T13:39:41.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 >> v1:172.21.15.46:6789/0 conn(0x7ff1100c6bd0 legacy=0x7ff110158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 --2- 172.21.15.46:0/2641116819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0fc003980 0x7ff0fc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 --2- 172.21.15.46:0/2641116819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 0x7ff0fc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 --2- 172.21.15.46:0/2641116819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0fc005190 0x7ff0fc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1100700c0 con 0x7ff0fc0047d0 2023-10-30T13:39:41.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc00c040 con 0x7ff0fc003980 2023-10-30T13:39:41.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc00c200 con 0x7ff0fc005190 2023-10-30T13:39:41.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff115b45640 1 --2- 172.21.15.46:0/2641116819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 0x7ff0fc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff116346640 1 --2- 172.21.15.46:0/2641116819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0fc003980 0x7ff0fc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.599+0000 7ff116b47640 1 --2- 172.21.15.46:0/2641116819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0fc005190 0x7ff0fc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.603+0000 7ff115b45640 1 -- 172.21.15.46:0/2641116819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0fc005190 msgr2=0x7ff0fc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:41.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.603+0000 7ff115b45640 1 --2- 172.21.15.46:0/2641116819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0fc005190 0x7ff0fc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.603+0000 7ff115b45640 1 -- 172.21.15.46:0/2641116819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0fc003980 msgr2=0x7ff0fc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:41.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.603+0000 7ff115b45640 1 --2- 172.21.15.46:0/2641116819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0fc003980 0x7ff0fc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.603+0000 7ff115b45640 1 -- 172.21.15.46:0/2641116819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc003530 con 0x7ff0fc0047d0 2023-10-30T13:39:41.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.603+0000 7ff116b47640 1 -- 172.21.15.46:0/2641116819 reap_dead start 2023-10-30T13:39:41.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.603+0000 7ff115b45640 1 --2- 172.21.15.46:0/2641116819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 0x7ff0fc004c50 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff10000c210 tx=0x7ff10001cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c51931f0dde14b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:41.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.603+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff100005ae0 con 0x7ff0fc0047d0 2023-10-30T13:39:41.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.603+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff100005c40 con 0x7ff0fc0047d0 2023-10-30T13:39:41.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.603+0000 7ff115344640 1 -- 172.21.15.46:0/2641116819 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1000a5ae0 con 0x7ff0fc0047d0 2023-10-30T13:39:41.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.603+0000 7ff1185aa640 1 -- 172.21.15.46:0/2641116819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 msgr2=0x7ff0fc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.603+0000 7ff1185aa640 1 --2- 172.21.15.46:0/2641116819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 0x7ff0fc004c50 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff10000c210 tx=0x7ff10001cba0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 -- 172.21.15.46:0/2641116819 shutdown_connections 2023-10-30T13:39:41.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 --2- 172.21.15.46:0/2641116819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 0x7ff0fc004c50 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 -- 172.21.15.46:0/2641116819 >> 172.21.15.46:0/2641116819 conn(0x7ff1100c7910 msgr2=0x7ff1100c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:41.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 -- 172.21.15.46:0/2641116819 shutdown_connections 2023-10-30T13:39:41.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 -- 172.21.15.46:0/2641116819 wait complete. 2023-10-30T13:39:41.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 Processor -- start 2023-10-30T13:39:41.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 -- start start 2023-10-30T13:39:41.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 0x7ff110150240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff110145fd0 0x7ff1100c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1100c2740 0x7ff1100c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1101eb5b0 con 0x7ff0fc0047d0 2023-10-30T13:39:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1101ea950 con 0x7ff1100c2740 2023-10-30T13:39:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff1185aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff110150aa0 con 0x7ff110145fd0 2023-10-30T13:39:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff116346640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 0x7ff110150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff116346640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 0x7ff110150240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53954/0 (socket says 172.21.15.46:53954) 2023-10-30T13:39:41.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff116346640 1 -- 172.21.15.46:0/1376829992 learned_addr learned my addr 172.21.15.46:0/1376829992 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:41.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff115b45640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff110145fd0 0x7ff1100c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff116b47640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1100c2740 0x7ff1100c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff116346640 1 -- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff110145fd0 msgr2=0x7ff1100c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff116346640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff110145fd0 0x7ff1100c3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff116346640 1 -- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1100c2740 msgr2=0x7ff1100c2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff116346640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1100c2740 0x7ff1100c2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff116346640 1 -- 172.21.15.46:0/1376829992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1101eac10 con 0x7ff0fc0047d0 2023-10-30T13:39:41.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff115b45640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff110145fd0 0x7ff1100c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:41.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff116b47640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1100c2740 0x7ff1100c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:41.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.607+0000 7ff116346640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 0x7ff110150240 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00f570 tx=0x7ff10c00fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:41.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.611+0000 7ff106ffd640 1 -- 172.21.15.46:0/1376829992 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c0052e0 con 0x7ff0fc0047d0 2023-10-30T13:39:41.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.611+0000 7ff106ffd640 1 -- 172.21.15.46:0/1376829992 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c005440 con 0x7ff0fc0047d0 2023-10-30T13:39:41.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.611+0000 7ff106ffd640 1 -- 172.21.15.46:0/1376829992 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c003840 con 0x7ff0fc0047d0 2023-10-30T13:39:41.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.611+0000 7ff1185aa640 1 -- 172.21.15.46:0/1376829992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1101eaef0 con 0x7ff0fc0047d0 2023-10-30T13:39:41.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.611+0000 7ff1185aa640 1 -- 172.21.15.46:0/1376829992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff110150780 con 0x7ff0fc0047d0 2023-10-30T13:39:41.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.611+0000 7ff1185aa640 1 -- 172.21.15.46:0/1376829992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1101eb820 con 0x7ff0fc0047d0 2023-10-30T13:39:41.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.611+0000 7ff106ffd640 1 -- 172.21.15.46:0/1376829992 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff10c0039a0 con 0x7ff0fc0047d0 2023-10-30T13:39:41.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.611+0000 7ff106ffd640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0f40664a0 0x7ff0f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.611+0000 7ff106ffd640 1 -- 172.21.15.46:0/1376829992 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7ff10c0841d0 con 0x7ff0fc0047d0 2023-10-30T13:39:41.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.611+0000 7ff115b45640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0f40664a0 0x7ff0f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.615+0000 7ff115b45640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0f40664a0 0x7ff0f4068960 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff1000ab000 tx=0x7ff1000b1040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:41.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.615+0000 7ff106ffd640 1 -- 172.21.15.46:0/1376829992 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff10c051180 con 0x7ff0fc0047d0 2023-10-30T13:39:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.803+0000 7ff1185aa640 1 -- 172.21.15.46:0/1376829992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff11004e170 con 0x7ff0fc0047d0 2023-10-30T13:39:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.803+0000 7ff106ffd640 1 -- 172.21.15.46:0/1376829992 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5747 (secure 0 0 0) 0x7ff10c0547a0 con 0x7ff0fc0047d0 2023-10-30T13:39:41.808 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:41.808 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:38.428153+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 86 2023-10-30T13:39:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.807+0000 7ff1185aa640 1 -- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0f40664a0 msgr2=0x7ff0f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.807+0000 7ff1185aa640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0f40664a0 0x7ff0f4068960 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff1000ab000 tx=0x7ff1000b1040 comp rx=0 tx=0).stop 2023-10-30T13:39:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.807+0000 7ff1185aa640 1 -- 172.21.15.46:0/1376829992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 msgr2=0x7ff110150240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.807+0000 7ff1185aa640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 0x7ff110150240 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00f570 tx=0x7ff10c00fa30 comp rx=0 tx=0).stop 2023-10-30T13:39:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.807+0000 7ff1185aa640 1 -- 172.21.15.46:0/1376829992 shutdown_connections 2023-10-30T13:39:41.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.807+0000 7ff1185aa640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0f40664a0 0x7ff0f4068960 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.807+0000 7ff1185aa640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1100c2740 0x7ff1100c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.807+0000 7ff1185aa640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff110145fd0 0x7ff1100c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.807+0000 7ff1185aa640 1 --2- 172.21.15.46:0/1376829992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0047d0 0x7ff110150240 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.807+0000 7ff1185aa640 1 -- 172.21.15.46:0/1376829992 >> 172.21.15.46:0/1376829992 conn(0x7ff1100c7910 msgr2=0x7ff11006b250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:41.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.807+0000 7ff1185aa640 1 -- 172.21.15.46:0/1376829992 shutdown_connections 2023-10-30T13:39:41.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:41.807+0000 7ff1185aa640 1 -- 172.21.15.46:0/1376829992 wait complete. 2023-10-30T13:39:41.828 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T13:39:43.829 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:43.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.983+0000 7fc8ee654640 1 Processor -- start 2023-10-30T13:39:43.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.983+0000 7fc8ee654640 1 -- start start 2023-10-30T13:39:43.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.983+0000 7fc8ee654640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8e80c6b80 0x7fc8e80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:43.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.983+0000 7fc8ee654640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8e813f4a0 0x7fc8e80c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:43.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.983+0000 7fc8ee654640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e80c0a50 con 0x7fc8e80c6f60 2023-10-30T13:39:43.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.983+0000 7fc8ee654640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8e806f6e0 con 0x7fc8e80c6b80 2023-10-30T13:39:43.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.983+0000 7fc8ee654640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e806f9c0 con 0x7fc8e813f4a0 2023-10-30T13:39:43.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.983+0000 7fc8e7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8e80c6b80 0x7fc8e80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:43.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.983+0000 7fc8e7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8e80c6b80 0x7fc8e80bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53968/0 (socket says 172.21.15.46:53968) 2023-10-30T13:39:43.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.983+0000 7fc8e7fff640 1 -- 172.21.15.46:0/2523578513 learned_addr learned my addr 172.21.15.46:0/2523578513 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:43.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.983+0000 7fc8ecbf1640 1 --2- 172.21.15.46:0/2523578513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8e813f4a0 0x7fc8e80c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:43.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8e7fff640 1 -- 172.21.15.46:0/2523578513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8e813f4a0 msgr2=0x7fc8e80c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:43.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8e7fff640 1 --2- 172.21.15.46:0/2523578513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8e813f4a0 0x7fc8e80c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8e7fff640 1 -- 172.21.15.46:0/2523578513 >> v1:172.21.15.46:6789/0 conn(0x7fc8e80c6f60 legacy=0x7fc8e813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:43.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8e7fff640 1 -- 172.21.15.46:0/2523578513 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e806fca0 con 0x7fc8e80c6b80 2023-10-30T13:39:43.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8e6ffd640 1 -- 172.21.15.46:0/2523578513 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632406677 0 0) 0x7fc8e80c0a50 con 0x7fc8e80c6f60 2023-10-30T13:39:43.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8e7fff640 1 --2- 172.21.15.46:0/2523578513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8e80c6b80 0x7fc8e80bfec0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009870 tx=0x7fc8d801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85627c2374006120 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:43.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8e6ffd640 1 -- 172.21.15.46:0/2523578513 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d80ab650 con 0x7fc8e80c6b80 2023-10-30T13:39:43.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8e6ffd640 1 -- 172.21.15.46:0/2523578513 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d80ab7b0 con 0x7fc8e80c6b80 2023-10-30T13:39:43.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8e6ffd640 1 -- 172.21.15.46:0/2523578513 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d80abb40 con 0x7fc8e80c6b80 2023-10-30T13:39:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8ee654640 1 -- 172.21.15.46:0/2523578513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8e80c6b80 msgr2=0x7fc8e80bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8ee654640 1 --2- 172.21.15.46:0/2523578513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8e80c6b80 0x7fc8e80bfec0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009870 tx=0x7fc8d801c540 comp rx=0 tx=0).stop 2023-10-30T13:39:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8ee654640 1 -- 172.21.15.46:0/2523578513 shutdown_connections 2023-10-30T13:39:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8ee654640 1 --2- 172.21.15.46:0/2523578513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8e813f4a0 0x7fc8e80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.987+0000 7fc8ee654640 1 --2- 172.21.15.46:0/2523578513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8e80c6b80 0x7fc8e80bfec0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ee654640 1 -- 172.21.15.46:0/2523578513 >> 172.21.15.46:0/2523578513 conn(0x7fc8e80c7910 msgr2=0x7fc8e80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ee654640 1 -- 172.21.15.46:0/2523578513 shutdown_connections 2023-10-30T13:39:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ee654640 1 -- 172.21.15.46:0/2523578513 wait complete. 2023-10-30T13:39:43.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ee654640 1 Processor -- start 2023-10-30T13:39:43.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ee654640 1 -- start start 2023-10-30T13:39:43.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ee654640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8e80c6b80 0x7fc8e81591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:43.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ee654640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8e80c6f60 0x7fc8e80ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ee654640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8e813f4a0 0x7fc8e80c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ee654640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e8155640 con 0x7fc8e813f4a0 2023-10-30T13:39:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ee654640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e81f5e70 con 0x7fc8e80c6b80 2023-10-30T13:39:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ee654640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e81f5b90 con 0x7fc8e80c6f60 2023-10-30T13:39:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ecbf1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8e813f4a0 0x7fc8e80c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:44.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ecbf1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8e813f4a0 0x7fc8e80c8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53976/0 (socket says 172.21.15.46:53976) 2023-10-30T13:39:44.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ecbf1640 1 -- 172.21.15.46:0/2813510021 learned_addr learned my addr 172.21.15.46:0/2813510021 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:39:44.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8e7fff640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8e80c6b80 0x7fc8e81591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:44.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8e77fe640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8e80c6f60 0x7fc8e80ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:44.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ecbf1640 1 -- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8e80c6f60 msgr2=0x7fc8e80ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:44.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ecbf1640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8e80c6f60 0x7fc8e80ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:44.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ecbf1640 1 -- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8e80c6b80 msgr2=0x7fc8e81591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:44.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ecbf1640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8e80c6b80 0x7fc8e81591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:44.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ecbf1640 1 -- 172.21.15.46:0/2813510021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d80094d0 con 0x7fc8e813f4a0 2023-10-30T13:39:44.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8e77fe640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8e80c6f60 0x7fc8e80ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:44.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.991+0000 7fc8ecbf1640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8e813f4a0 0x7fc8e80c8ef0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0007e90 tx=0x7fc8e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:44.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.995+0000 7fc8e4ff9640 1 -- 172.21.15.46:0/2813510021 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0014420 con 0x7fc8e813f4a0 2023-10-30T13:39:44.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.995+0000 7fc8e4ff9640 1 -- 172.21.15.46:0/2813510021 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e0014580 con 0x7fc8e813f4a0 2023-10-30T13:39:44.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.995+0000 7fc8ee654640 1 -- 172.21.15.46:0/2813510021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e81f4fc0 con 0x7fc8e813f4a0 2023-10-30T13:39:44.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.995+0000 7fc8e4ff9640 1 -- 172.21.15.46:0/2813510021 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0014930 con 0x7fc8e813f4a0 2023-10-30T13:39:44.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.995+0000 7fc8ee654640 1 -- 172.21.15.46:0/2813510021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e81f5540 con 0x7fc8e813f4a0 2023-10-30T13:39:44.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.995+0000 7fc8ee654640 1 -- 172.21.15.46:0/2813510021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e80c8c10 con 0x7fc8e813f4a0 2023-10-30T13:39:44.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.995+0000 7fc8e4ff9640 1 -- 172.21.15.46:0/2813510021 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc8e0014a90 con 0x7fc8e813f4a0 2023-10-30T13:39:44.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.995+0000 7fc8e4ff9640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8d40664a0 0x7fc8d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:44.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.995+0000 7fc8e4ff9640 1 -- 172.21.15.46:0/2813510021 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fc8e0084ab0 con 0x7fc8e813f4a0 2023-10-30T13:39:44.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.995+0000 7fc8e7fff640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8d40664a0 0x7fc8d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:44.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.999+0000 7fc8e7fff640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8d40664a0 0x7fc8d4068960 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc8d80af000 tx=0x7fc8d80b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:44.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:43.999+0000 7fc8e4ff9640 1 -- 172.21.15.46:0/2813510021 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc8e80c8c10 con 0x7fc8e813f4a0 2023-10-30T13:39:44.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 -- 172.21.15.46:0/2813510021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8e81f5cd0 con 0x7fc8e813f4a0 2023-10-30T13:39:44.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8e4ff9640 1 -- 172.21.15.46:0/2813510021 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fc8e81f5cd0 con 0x7fc8e813f4a0 2023-10-30T13:39:44.205 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:39:44.205 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6201,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:39:38.428153+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:44.206 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 87 2023-10-30T13:39:44.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 -- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8d40664a0 msgr2=0x7fc8d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8d40664a0 0x7fc8d4068960 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc8d80af000 tx=0x7fc8d80b6040 comp rx=0 tx=0).stop 2023-10-30T13:39:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 -- 172.21.15.46:0/2813510021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8e813f4a0 msgr2=0x7fc8e80c8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8e813f4a0 0x7fc8e80c8ef0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0007e90 tx=0x7fc8e000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:39:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 -- 172.21.15.46:0/2813510021 shutdown_connections 2023-10-30T13:39:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8d40664a0 0x7fc8d4068960 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8e813f4a0 0x7fc8e80c8ef0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8e80c6f60 0x7fc8e80ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 --2- 172.21.15.46:0/2813510021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8e80c6b80 0x7fc8e81591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 -- 172.21.15.46:0/2813510021 >> 172.21.15.46:0/2813510021 conn(0x7fc8e80c7910 msgr2=0x7fc8e813ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 -- 172.21.15.46:0/2813510021 shutdown_connections 2023-10-30T13:39:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:39:44.203+0000 7fc8ee654640 1 -- 172.21.15.46:0/2813510021 wait complete. 2023-10-30T13:39:44.227 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T13:39:44.227 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 73 secs before thrashing 2023-10-30T13:39:47.947 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:39:47.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:18.302 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:18.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:48.359 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:48.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:57.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:57.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7fedad068640 1 Processor -- start 2023-10-30T13:40:57.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7fedad068640 1 -- start start 2023-10-30T13:40:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7fedad068640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feda81538a0 0x7feda815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7fedad068640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feda80cf160 0x7feda80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7fedad068640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda80cb800 con 0x7feda80ca0e0 2023-10-30T13:40:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7fedad068640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda815dc30 con 0x7feda80cf160 2023-10-30T13:40:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7fedad068640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda80cb330 con 0x7feda81538a0 2023-10-30T13:40:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7feda6d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feda81538a0 0x7feda815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7feda759e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feda80cf160 0x7feda80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7feda759e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feda80cf160 0x7feda80cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:42070/0 (socket says 172.21.15.46:42070) 2023-10-30T13:40:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7feda6d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feda81538a0 0x7feda815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:49178/0 (socket says 172.21.15.46:49178) 2023-10-30T13:40:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7feda759e640 1 -- 172.21.15.46:0/2130024501 learned_addr learned my addr 172.21.15.46:0/2130024501 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:40:57.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7feda6d9d640 1 -- 172.21.15.46:0/2130024501 >> v2:172.21.15.154:3300/0 conn(0x7feda80cf160 msgr2=0x7feda80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:57.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7feda6d9d640 1 --2- 172.21.15.46:0/2130024501 >> v2:172.21.15.154:3300/0 conn(0x7feda80cf160 0x7feda80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7feda6d9d640 1 -- 172.21.15.46:0/2130024501 >> v1:172.21.15.154:6789/0 conn(0x7feda80ca0e0 legacy=0x7feda815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:57.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7feda6d9d640 1 -- 172.21.15.46:0/2130024501 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda80cad70 con 0x7feda81538a0 2023-10-30T13:40:57.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.374+0000 7feda759e640 1 --2- 172.21.15.46:0/2130024501 >> v2:172.21.15.154:3300/0 conn(0x7feda80cf160 0x7feda80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:40:57.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.378+0000 7feda6d9d640 1 --2- 172.21.15.46:0/2130024501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feda81538a0 0x7feda815d6f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fed980097c0 tx=0x7fed9801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60010449457e42d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:57.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.378+0000 7feda5d9b640 1 -- 172.21.15.46:0/2130024501 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed980a8970 con 0x7feda81538a0 2023-10-30T13:40:57.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.378+0000 7feda5d9b640 1 -- 172.21.15.46:0/2130024501 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed980a8ad0 con 0x7feda81538a0 2023-10-30T13:40:57.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.378+0000 7feda5d9b640 1 -- 172.21.15.46:0/2130024501 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed980a8e60 con 0x7feda81538a0 2023-10-30T13:40:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.382+0000 7fedad068640 1 -- 172.21.15.46:0/2130024501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feda81538a0 msgr2=0x7feda815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.382+0000 7fedad068640 1 --2- 172.21.15.46:0/2130024501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feda81538a0 0x7feda815d6f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fed980097c0 tx=0x7fed9801ec70 comp rx=0 tx=0).stop 2023-10-30T13:40:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.382+0000 7fedad068640 1 -- 172.21.15.46:0/2130024501 shutdown_connections 2023-10-30T13:40:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.382+0000 7fedad068640 1 --2- 172.21.15.46:0/2130024501 >> v2:172.21.15.154:3300/0 conn(0x7feda80cf160 0x7feda80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.382+0000 7fedad068640 1 --2- 172.21.15.46:0/2130024501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feda81538a0 0x7feda815d6f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.382+0000 7fedad068640 1 -- 172.21.15.46:0/2130024501 >> 172.21.15.46:0/2130024501 conn(0x7feda806bfe0 msgr2=0x7feda806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.382+0000 7fedad068640 1 -- 172.21.15.46:0/2130024501 shutdown_connections 2023-10-30T13:40:57.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.382+0000 7fedad068640 1 -- 172.21.15.46:0/2130024501 wait complete. 2023-10-30T13:40:57.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.382+0000 7fedad068640 1 Processor -- start 2023-10-30T13:40:57.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7fedad068640 1 -- start start 2023-10-30T13:40:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7fedad068640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda80ca0e0 0x7feda8158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7fedad068640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda80cf160 0x7feda80ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7fedad068640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feda81538a0 0x7feda80cead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7fedad068640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda81f36b0 con 0x7feda80ca0e0 2023-10-30T13:40:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7fedad068640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda8157a90 con 0x7feda80cf160 2023-10-30T13:40:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7fedad068640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda814d790 con 0x7feda81538a0 2023-10-30T13:40:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7feda6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda80ca0e0 0x7feda8158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7feda6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda80ca0e0 0x7feda8158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55156/0 (socket says 172.21.15.46:55156) 2023-10-30T13:40:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7feda6d9d640 1 -- 172.21.15.46:0/2350643995 learned_addr learned my addr 172.21.15.46:0/2350643995 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:40:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7feda659c640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda80cf160 0x7feda80ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7feda759e640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feda81538a0 0x7feda80cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7feda6d9d640 1 -- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feda81538a0 msgr2=0x7feda80cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7feda6d9d640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feda81538a0 0x7feda80cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7feda6d9d640 1 -- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda80cf160 msgr2=0x7feda80ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:57.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7feda6d9d640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda80cf160 0x7feda80ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7feda6d9d640 1 -- 172.21.15.46:0/2350643995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda0009b30 con 0x7feda80ca0e0 2023-10-30T13:40:57.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7feda659c640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda80cf160 0x7feda80ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:57.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7feda6d9d640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda80ca0e0 0x7feda8158910 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fed9801d320 tx=0x7fed9801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:57.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7fed8f7fe640 1 -- 172.21.15.46:0/2350643995 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed980bb640 con 0x7feda80ca0e0 2023-10-30T13:40:57.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7fed8f7fe640 1 -- 172.21.15.46:0/2350643995 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed980bb7c0 con 0x7feda80ca0e0 2023-10-30T13:40:57.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7fed8f7fe640 1 -- 172.21.15.46:0/2350643995 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed980bbb70 con 0x7feda80ca0e0 2023-10-30T13:40:57.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7fedad068640 1 -- 172.21.15.46:0/2350643995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed98009420 con 0x7feda80ca0e0 2023-10-30T13:40:57.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.386+0000 7fedad068640 1 -- 172.21.15.46:0/2350643995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda81f2d10 con 0x7feda80ca0e0 2023-10-30T13:40:57.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.390+0000 7fedad068640 1 -- 172.21.15.46:0/2350643995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda80b2a50 con 0x7feda80ca0e0 2023-10-30T13:40:57.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.390+0000 7fed8f7fe640 1 -- 172.21.15.46:0/2350643995 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed980b7040 con 0x7feda80ca0e0 2023-10-30T13:40:57.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.390+0000 7fed8f7fe640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed940664a0 0x7fed94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:57.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.390+0000 7fed8f7fe640 1 -- 172.21.15.46:0/2350643995 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fed98122460 con 0x7feda80ca0e0 2023-10-30T13:40:57.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.390+0000 7feda659c640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed940664a0 0x7fed94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:57.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.394+0000 7feda659c640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed940664a0 0x7fed94068960 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7feda814e1e0 tx=0x7fed9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:57.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.394+0000 7fed8f7fe640 1 -- 172.21.15.46:0/2350643995 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fed980ef5d0 con 0x7feda80ca0e0 2023-10-30T13:40:57.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.594+0000 7fedad068640 1 -- 172.21.15.46:0/2350643995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feda804e170 con 0x7feda80ca0e0 2023-10-30T13:40:57.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.594+0000 7fed8f7fe640 1 -- 172.21.15.46:0/2350643995 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6537 (secure 0 0 0) 0x7fed980f2bf0 con 0x7feda80ca0e0 2023-10-30T13:40:57.600 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:40:57.600 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6201,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:40:29.124030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 89 2023-10-30T13:40:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.598+0000 7fedad068640 1 -- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed940664a0 msgr2=0x7fed94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.598+0000 7fedad068640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed940664a0 0x7fed94068960 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7feda814e1e0 tx=0x7fed9c009040 comp rx=0 tx=0).stop 2023-10-30T13:40:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.598+0000 7fedad068640 1 -- 172.21.15.46:0/2350643995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda80ca0e0 msgr2=0x7feda8158910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.598+0000 7fedad068640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda80ca0e0 0x7feda8158910 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fed9801d320 tx=0x7fed9801ba10 comp rx=0 tx=0).stop 2023-10-30T13:40:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.598+0000 7fedad068640 1 -- 172.21.15.46:0/2350643995 shutdown_connections 2023-10-30T13:40:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.598+0000 7fedad068640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed940664a0 0x7fed94068960 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.598+0000 7fedad068640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feda81538a0 0x7feda80cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.598+0000 7fedad068640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda80cf160 0x7feda80ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.598+0000 7fedad068640 1 --2- 172.21.15.46:0/2350643995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda80ca0e0 0x7feda8158910 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.598+0000 7fedad068640 1 -- 172.21.15.46:0/2350643995 >> 172.21.15.46:0/2350643995 conn(0x7feda806bfe0 msgr2=0x7feda81510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.598+0000 7fedad068640 1 -- 172.21.15.46:0/2350643995 shutdown_connections 2023-10-30T13:40:57.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.598+0000 7fedad068640 1 -- 172.21.15.46:0/2350643995 wait complete. 2023-10-30T13:40:57.621 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T13:40:57.621 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:40:57.622 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:40:57.641 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:40:57.641 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:57.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0eee3c4640 1 Processor -- start 2023-10-30T13:40:57.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0eee3c4640 1 -- start start 2023-10-30T13:40:57.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0eee3c4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0ee80c05d0 0x7f0ee813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:57.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0eee3c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee80c6f60 0x7f0ee813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:57.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0eee3c4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee80c0a50 con 0x7f0ee80c6b80 2023-10-30T13:40:57.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0eee3c4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee806f6e0 con 0x7f0ee80c05d0 2023-10-30T13:40:57.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0eee3c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee806f9c0 con 0x7f0ee80c6f60 2023-10-30T13:40:57.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0ee77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0ee80c05d0 0x7f0ee813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:57.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0ee77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0ee80c05d0 0x7f0ee813ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55166/0 (socket says 172.21.15.46:55166) 2023-10-30T13:40:57.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0ee77fe640 1 -- 172.21.15.46:0/3244244588 learned_addr learned my addr 172.21.15.46:0/3244244588 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:40:57.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0eec961640 1 --2- 172.21.15.46:0/3244244588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee80c6f60 0x7f0ee813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:57.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0ee77fe640 1 -- 172.21.15.46:0/3244244588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee80c6f60 msgr2=0x7f0ee813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:57.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.826+0000 7f0ee77fe640 1 --2- 172.21.15.46:0/3244244588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee80c6f60 0x7f0ee813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0ee77fe640 1 -- 172.21.15.46:0/3244244588 >> v1:172.21.15.154:6789/0 conn(0x7f0ee80c6b80 legacy=0x7f0ee80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:57.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0ee77fe640 1 -- 172.21.15.46:0/3244244588 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee806fca0 con 0x7f0ee80c05d0 2023-10-30T13:40:57.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0ee77fe640 1 --2- 172.21.15.46:0/3244244588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee80c05d0 0x7f0ee813ed90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009870 tx=0x7f0edc01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af13cc433d421524 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:57.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0ee6ffd640 1 -- 172.21.15.46:0/3244244588 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0edc0ab650 con 0x7f0ee80c05d0 2023-10-30T13:40:57.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0ee6ffd640 1 -- 172.21.15.46:0/3244244588 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0edc0ab7b0 con 0x7f0ee80c05d0 2023-10-30T13:40:57.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0ee6ffd640 1 -- 172.21.15.46:0/3244244588 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0edc0abb40 con 0x7f0ee80c05d0 2023-10-30T13:40:57.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3244244588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee80c05d0 msgr2=0x7f0ee813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:57.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0eee3c4640 1 --2- 172.21.15.46:0/3244244588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee80c05d0 0x7f0ee813ed90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009870 tx=0x7f0edc01c540 comp rx=0 tx=0).stop 2023-10-30T13:40:57.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3244244588 shutdown_connections 2023-10-30T13:40:57.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0eee3c4640 1 --2- 172.21.15.46:0/3244244588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee80c6f60 0x7f0ee813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0eee3c4640 1 --2- 172.21.15.46:0/3244244588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee80c05d0 0x7f0ee813ed90 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3244244588 >> 172.21.15.46:0/3244244588 conn(0x7f0ee80c7910 msgr2=0x7f0ee80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:57.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3244244588 shutdown_connections 2023-10-30T13:40:57.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.830+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3244244588 wait complete. 2023-10-30T13:40:57.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0eee3c4640 1 Processor -- start 2023-10-30T13:40:57.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0eee3c4640 1 -- start start 2023-10-30T13:40:57.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0eee3c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee80c05d0 0x7f0ee8138920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:57.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0eee3c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee80c6b80 0x7f0ee813d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:57.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0eee3c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee80c6f60 0x7f0ee8152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:57.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0eee3c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee81e9310 con 0x7f0ee80c05d0 2023-10-30T13:40:57.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0eee3c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee81e7c70 con 0x7f0ee80c6f60 2023-10-30T13:40:57.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0eee3c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee8158290 con 0x7f0ee80c6b80 2023-10-30T13:40:57.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee80c05d0 0x7f0ee8138920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:57.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee80c05d0 0x7f0ee8138920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55176/0 (socket says 172.21.15.46:55176) 2023-10-30T13:40:57.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee7fff640 1 -- 172.21.15.46:0/3377147068 learned_addr learned my addr 172.21.15.46:0/3377147068 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:40:57.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee77fe640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee80c6b80 0x7f0ee813d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:57.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0eec961640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee80c6f60 0x7f0ee8152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:57.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee7fff640 1 -- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee80c6b80 msgr2=0x7f0ee813d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:57.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee7fff640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee80c6b80 0x7f0ee813d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee7fff640 1 -- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee80c6f60 msgr2=0x7f0ee8152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:57.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee7fff640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee80c6f60 0x7f0ee8152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:57.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee7fff640 1 -- 172.21.15.46:0/3377147068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0edc0094d0 con 0x7f0ee80c05d0 2023-10-30T13:40:57.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee77fe640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee80c6b80 0x7f0ee813d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:57.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0eec961640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee80c6f60 0x7f0ee8152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:57.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee7fff640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee80c05d0 0x7f0ee8138920 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0ed800c500 tx=0x7f0ed800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:57.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee4ff9640 1 -- 172.21.15.46:0/3377147068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8005220 con 0x7f0ee80c05d0 2023-10-30T13:40:57.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee4ff9640 1 -- 172.21.15.46:0/3377147068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed8005380 con 0x7f0ee80c05d0 2023-10-30T13:40:57.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0ee4ff9640 1 -- 172.21.15.46:0/3377147068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8007270 con 0x7f0ee80c05d0 2023-10-30T13:40:57.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.834+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3377147068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee8158570 con 0x7f0ee80c05d0 2023-10-30T13:40:57.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.838+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3377147068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee8158b30 con 0x7f0ee80c05d0 2023-10-30T13:40:57.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.838+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3377147068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee8158d20 con 0x7f0ee80c05d0 2023-10-30T13:40:57.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.838+0000 7f0ee4ff9640 1 -- 172.21.15.46:0/3377147068 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ed80073d0 con 0x7f0ee80c05d0 2023-10-30T13:40:57.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.838+0000 7f0ee4ff9640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ed40664a0 0x7f0ed4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:57.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.838+0000 7f0ee4ff9640 1 -- 172.21.15.46:0/3377147068 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f0ed8083bd0 con 0x7f0ee80c05d0 2023-10-30T13:40:57.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.838+0000 7f0ee77fe640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ed40664a0 0x7f0ed4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:57.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.842+0000 7f0ee77fe640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ed40664a0 0x7f0ed4068960 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0edc0af000 tx=0x7f0edc0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:57.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:57.842+0000 7f0ee4ff9640 1 -- 172.21.15.46:0/3377147068 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0ed8050c20 con 0x7f0ee80c05d0 2023-10-30T13:40:58.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.034+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3377147068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ee804e170 con 0x7f0ee80c05d0 2023-10-30T13:40:58.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.034+0000 7f0ee4ff9640 1 -- 172.21.15.46:0/3377147068 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6537 (secure 0 0 0) 0x7f0ed8054240 con 0x7f0ee80c05d0 2023-10-30T13:40:58.038 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:40:58.039 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6201,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:40:29.124030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:58.039 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 89 2023-10-30T13:40:58.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.038+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ed40664a0 msgr2=0x7f0ed4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.038+0000 7f0eee3c4640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ed40664a0 0x7f0ed4068960 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0edc0af000 tx=0x7f0edc0a7000 comp rx=0 tx=0).stop 2023-10-30T13:40:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.038+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3377147068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee80c05d0 msgr2=0x7f0ee8138920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.038+0000 7f0eee3c4640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee80c05d0 0x7f0ee8138920 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0ed800c500 tx=0x7f0ed800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.038+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3377147068 shutdown_connections 2023-10-30T13:40:58.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.038+0000 7f0eee3c4640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ed40664a0 0x7f0ed4068960 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.038+0000 7f0eee3c4640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee80c6f60 0x7f0ee8152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.038+0000 7f0eee3c4640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee80c6b80 0x7f0ee813d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.038+0000 7f0eee3c4640 1 --2- 172.21.15.46:0/3377147068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee80c05d0 0x7f0ee8138920 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.038+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3377147068 >> 172.21.15.46:0/3377147068 conn(0x7f0ee80c7910 msgr2=0x7f0ee8139310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:58.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.038+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3377147068 shutdown_connections 2023-10-30T13:40:58.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:40:58.038+0000 7f0eee3c4640 1 -- 172.21.15.46:0/3377147068 wait complete. 2023-10-30T13:40:58.060 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:41:00.060 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5ca854640 1 Processor -- start 2023-10-30T13:41:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5ca854640 1 -- start start 2023-10-30T13:41:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5ca854640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5c406f7a0 0x7fe5c406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5ca854640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5c40c6bd0 0x7fe5c4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5ca854640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe5c4071110 0x7fe5c4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5ca854640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c40700c0 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5ca854640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c406a3d0 con 0x7fe5c4071110 2023-10-30T13:41:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5ca854640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c4066cf0 con 0x7fe5c406f7a0 2023-10-30T13:41:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5c37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5c40c6bd0 0x7fe5c4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5c37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5c40c6bd0 0x7fe5c4069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55178/0 (socket says 172.21.15.46:55178) 2023-10-30T13:41:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5c37fe640 1 -- 172.21.15.46:0/634017214 learned_addr learned my addr 172.21.15.46:0/634017214 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5c8df1640 1 --2- 172.21.15.46:0/634017214 >> v2:172.21.15.154:3300/0 conn(0x7fe5c4071110 0x7fe5c4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5c3fff640 1 --2- 172.21.15.46:0/634017214 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5c406f7a0 0x7fe5c406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5c37fe640 1 -- 172.21.15.46:0/634017214 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5c406f7a0 msgr2=0x7fe5c406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5c37fe640 1 --2- 172.21.15.46:0/634017214 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5c406f7a0 0x7fe5c406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5c37fe640 1 -- 172.21.15.46:0/634017214 >> v2:172.21.15.154:3300/0 conn(0x7fe5c4071110 msgr2=0x7fe5c4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5c37fe640 1 --2- 172.21.15.46:0/634017214 >> v2:172.21.15.154:3300/0 conn(0x7fe5c4071110 0x7fe5c4158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5c37fe640 1 -- 172.21.15.46:0/634017214 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c40c70d0 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.210+0000 7fe5c8df1640 1 --2- 172.21.15.46:0/634017214 >> v2:172.21.15.154:3300/0 conn(0x7fe5c4071110 0x7fe5c4158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5c37fe640 1 --2- 172.21.15.46:0/634017214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40c6bd0 0x7fe5c4069e90 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe5c406c7d0 tx=0x7fe5b801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc46ecacb7cb579 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5c2ffd640 1 -- 172.21.15.46:0/634017214 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b80a89b0 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5c2ffd640 1 -- 172.21.15.46:0/634017214 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b80a8b10 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5c2ffd640 1 -- 172.21.15.46:0/634017214 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b80a8ea0 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5ca854640 1 -- 172.21.15.46:0/634017214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40c6bd0 msgr2=0x7fe5c4069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5ca854640 1 --2- 172.21.15.46:0/634017214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40c6bd0 0x7fe5c4069e90 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe5c406c7d0 tx=0x7fe5b801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5ca854640 1 -- 172.21.15.46:0/634017214 shutdown_connections 2023-10-30T13:41:00.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5ca854640 1 --2- 172.21.15.46:0/634017214 >> v2:172.21.15.154:3300/0 conn(0x7fe5c4071110 0x7fe5c4158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5ca854640 1 --2- 172.21.15.46:0/634017214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40c6bd0 0x7fe5c4069e90 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5ca854640 1 --2- 172.21.15.46:0/634017214 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5c406f7a0 0x7fe5c406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5ca854640 1 -- 172.21.15.46:0/634017214 >> 172.21.15.46:0/634017214 conn(0x7fe5c40c7910 msgr2=0x7fe5c40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:00.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5ca854640 1 -- 172.21.15.46:0/634017214 shutdown_connections 2023-10-30T13:41:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.214+0000 7fe5ca854640 1 -- 172.21.15.46:0/634017214 wait complete. 2023-10-30T13:41:00.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5ca854640 1 Processor -- start 2023-10-30T13:41:00.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5ca854640 1 -- start start 2023-10-30T13:41:00.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5ca854640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5c406f7a0 0x7fe5c4145d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:00.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5ca854640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5c4071110 0x7fe5c414d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:00.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5ca854640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40c6bd0 0x7fe5c414adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:00.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5ca854640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c41f00f0 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5ca854640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c41f0300 con 0x7fe5c406f7a0 2023-10-30T13:41:00.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5ca854640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c41f05e0 con 0x7fe5c4071110 2023-10-30T13:41:00.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c8df1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40c6bd0 0x7fe5c414adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:00.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c8df1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40c6bd0 0x7fe5c414adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55182/0 (socket says 172.21.15.46:55182) 2023-10-30T13:41:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c8df1640 1 -- 172.21.15.46:0/1086759890 learned_addr learned my addr 172.21.15.46:0/1086759890 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c3fff640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5c406f7a0 0x7fe5c4145d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c37fe640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5c4071110 0x7fe5c414d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c8df1640 1 -- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5c4071110 msgr2=0x7fe5c414d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c8df1640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5c4071110 0x7fe5c414d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c8df1640 1 -- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5c406f7a0 msgr2=0x7fe5c4145d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c8df1640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5c406f7a0 0x7fe5c4145d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c8df1640 1 -- 172.21.15.46:0/1086759890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b80094d0 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c8df1640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40c6bd0 0x7fe5c414adb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc007df0 tx=0x7fe5bc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c0ff9640 1 -- 172.21.15.46:0/1086759890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc0039a0 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c0ff9640 1 -- 172.21.15.46:0/1086759890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5bc00f040 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5c0ff9640 1 -- 172.21.15.46:0/1086759890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc003d40 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.218+0000 7fe5ca854640 1 -- 172.21.15.46:0/1086759890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c41f08c0 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.222+0000 7fe5ca854640 1 -- 172.21.15.46:0/1086759890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c41462a0 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.222+0000 7fe5ca854640 1 -- 172.21.15.46:0/1086759890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c41f0d70 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.222+0000 7fe5c0ff9640 1 -- 172.21.15.46:0/1086759890 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5bc004030 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.222+0000 7fe5c0ff9640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe5b00664a0 0x7fe5b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.222+0000 7fe5c0ff9640 1 -- 172.21.15.46:0/1086759890 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fe5bc084580 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.222+0000 7fe5c3fff640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe5b00664a0 0x7fe5b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.226+0000 7fe5c3fff640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe5b00664a0 0x7fe5b0068960 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40045f0 tx=0x7fe5b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:00.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.226+0000 7fe5c0ff9640 1 -- 172.21.15.46:0/1086759890 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5c41f0d70 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 -- 172.21.15.46:0/1086759890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5c41f0d70 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5c0ff9640 1 -- 172.21.15.46:0/1086759890 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6537 (secure 0 0 0) 0x7fe5c41f0d70 con 0x7fe5c40c6bd0 2023-10-30T13:41:00.422 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:00.422 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6201,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:40:29.124030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:00.422 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 89 2023-10-30T13:41:00.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 -- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe5b00664a0 msgr2=0x7fe5b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:00.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe5b00664a0 0x7fe5b0068960 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40045f0 tx=0x7fe5b4008040 comp rx=0 tx=0).stop 2023-10-30T13:41:00.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 -- 172.21.15.46:0/1086759890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40c6bd0 msgr2=0x7fe5c414adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:00.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40c6bd0 0x7fe5c414adb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc007df0 tx=0x7fe5bc00d4a0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 -- 172.21.15.46:0/1086759890 shutdown_connections 2023-10-30T13:41:00.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe5b00664a0 0x7fe5b0068960 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40c6bd0 0x7fe5c414adb0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5c4071110 0x7fe5c414d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 --2- 172.21.15.46:0/1086759890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5c406f7a0 0x7fe5c4145d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:00.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 -- 172.21.15.46:0/1086759890 >> 172.21.15.46:0/1086759890 conn(0x7fe5c40c7910 msgr2=0x7fe5c40c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:00.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 -- 172.21.15.46:0/1086759890 shutdown_connections 2023-10-30T13:41:00.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:00.418+0000 7fe5ca854640 1 -- 172.21.15.46:0/1086759890 wait complete. 2023-10-30T13:41:00.443 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:41:02.444 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:02.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e8cea640 1 Processor -- start 2023-10-30T13:41:02.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e8cea640 1 -- start start 2023-10-30T13:41:02.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e8cea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc5e41538a0 0x7fc5e415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:02.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e8cea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5e40ca0e0 0x7fc5e415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:02.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e8cea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e40cb800 con 0x7fc5e40cf160 2023-10-30T13:41:02.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e8cea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e415dc30 con 0x7fc5e41538a0 2023-10-30T13:41:02.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e8cea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40cb330 con 0x7fc5e40ca0e0 2023-10-30T13:41:02.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e259c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc5e41538a0 0x7fc5e415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:02.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e2d9d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fc5e40cf160 0x7fc5e40cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:52154/0 (socket says 172.21.15.46:52154) 2023-10-30T13:41:02.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e2d9d640 1 -- 172.21.15.46:0/2072722382 learned_addr learned my addr 172.21.15.46:0/2072722382 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:41:02.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e1d9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5e40ca0e0 0x7fc5e415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:02.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e259c640 1 -- 172.21.15.46:0/2072722382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5e40ca0e0 msgr2=0x7fc5e415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:02.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e259c640 1 --2- 172.21.15.46:0/2072722382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5e40ca0e0 0x7fc5e415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:02.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e259c640 1 -- 172.21.15.46:0/2072722382 >> v1:172.21.15.154:6789/0 conn(0x7fc5e40cf160 legacy=0x7fc5e40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:02.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e259c640 1 -- 172.21.15.46:0/2072722382 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e40cad70 con 0x7fc5e41538a0 2023-10-30T13:41:02.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.594+0000 7fc5e1d9b640 1 --2- 172.21.15.46:0/2072722382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5e40ca0e0 0x7fc5e415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:41:02.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.598+0000 7fc5e259c640 1 --2- 172.21.15.46:0/2072722382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e41538a0 0x7fc5e415d6f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40097c0 tx=0x7fc5d401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be52b2dbd5711802 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:02.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.598+0000 7fc5e159a640 1 -- 172.21.15.46:0/2072722382 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40a8970 con 0x7fc5e41538a0 2023-10-30T13:41:02.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.598+0000 7fc5e159a640 1 -- 172.21.15.46:0/2072722382 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d40a8ad0 con 0x7fc5e41538a0 2023-10-30T13:41:02.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.598+0000 7fc5e159a640 1 -- 172.21.15.46:0/2072722382 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40a8e60 con 0x7fc5e41538a0 2023-10-30T13:41:02.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.598+0000 7fc5e8cea640 1 -- 172.21.15.46:0/2072722382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e41538a0 msgr2=0x7fc5e415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:02.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.598+0000 7fc5e8cea640 1 --2- 172.21.15.46:0/2072722382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e41538a0 0x7fc5e415d6f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40097c0 tx=0x7fc5d401ec70 comp rx=0 tx=0).stop 2023-10-30T13:41:02.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 -- 172.21.15.46:0/2072722382 shutdown_connections 2023-10-30T13:41:02.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 --2- 172.21.15.46:0/2072722382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5e40ca0e0 0x7fc5e415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:02.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 --2- 172.21.15.46:0/2072722382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e41538a0 0x7fc5e415d6f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:02.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 -- 172.21.15.46:0/2072722382 >> 172.21.15.46:0/2072722382 conn(0x7fc5e406bfe0 msgr2=0x7fc5e406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:02.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 -- 172.21.15.46:0/2072722382 shutdown_connections 2023-10-30T13:41:02.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 -- 172.21.15.46:0/2072722382 wait complete. 2023-10-30T13:41:02.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 Processor -- start 2023-10-30T13:41:02.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 -- start start 2023-10-30T13:41:02.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e40ca0e0 0x7fc5e41591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:02.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5e40cf160 0x7fc5e4157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:02.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5e41538a0 0x7fc5e414e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:02.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e41f3530 con 0x7fc5e41538a0 2023-10-30T13:41:02.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e41f2990 con 0x7fc5e40ca0e0 2023-10-30T13:41:02.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e8cea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e41f2c00 con 0x7fc5e40cf160 2023-10-30T13:41:02.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5e41538a0 0x7fc5e414e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:02.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5e41538a0 0x7fc5e414e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55192/0 (socket says 172.21.15.46:55192) 2023-10-30T13:41:02.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e2d9d640 1 -- 172.21.15.46:0/1537412586 learned_addr learned my addr 172.21.15.46:0/1537412586 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:02.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e259c640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e40ca0e0 0x7fc5e41591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:02.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e1d9b640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5e40cf160 0x7fc5e4157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:02.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e2d9d640 1 -- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5e40cf160 msgr2=0x7fc5e4157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:02.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e2d9d640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5e40cf160 0x7fc5e4157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:02.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e2d9d640 1 -- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e40ca0e0 msgr2=0x7fc5e41591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:02.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e2d9d640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e40ca0e0 0x7fc5e41591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:02.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.602+0000 7fc5e2d9d640 1 -- 172.21.15.46:0/1537412586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8009420 con 0x7fc5e41538a0 2023-10-30T13:41:02.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.606+0000 7fc5e2d9d640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5e41538a0 0x7fc5e414e0f0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00ca70 tx=0x7fc5dc00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:02.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.606+0000 7fc5d2ffd640 1 -- 172.21.15.46:0/1537412586 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc00abd0 con 0x7fc5e41538a0 2023-10-30T13:41:02.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.606+0000 7fc5d2ffd640 1 -- 172.21.15.46:0/1537412586 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5dc00ad30 con 0x7fc5e41538a0 2023-10-30T13:41:02.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.606+0000 7fc5d2ffd640 1 -- 172.21.15.46:0/1537412586 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc012600 con 0x7fc5e41538a0 2023-10-30T13:41:02.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.606+0000 7fc5e8cea640 1 -- 172.21.15.46:0/1537412586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d4009420 con 0x7fc5e41538a0 2023-10-30T13:41:02.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.606+0000 7fc5e8cea640 1 -- 172.21.15.46:0/1537412586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e41f31c0 con 0x7fc5e41538a0 2023-10-30T13:41:02.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.606+0000 7fc5e8cea640 1 -- 172.21.15.46:0/1537412586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e40b2a50 con 0x7fc5e41538a0 2023-10-30T13:41:02.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.606+0000 7fc5d2ffd640 1 -- 172.21.15.46:0/1537412586 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc5dc007070 con 0x7fc5e41538a0 2023-10-30T13:41:02.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.606+0000 7fc5d2ffd640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5c40664a0 0x7fc5c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:02.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.606+0000 7fc5d2ffd640 1 -- 172.21.15.46:0/1537412586 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fc5dc084080 con 0x7fc5e41538a0 2023-10-30T13:41:02.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.606+0000 7fc5e259c640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5c40664a0 0x7fc5c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:02.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.610+0000 7fc5e259c640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5c40664a0 0x7fc5c4068960 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40b2000 tx=0x7fc5d40aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:02.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.610+0000 7fc5d2ffd640 1 -- 172.21.15.46:0/1537412586 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc5dc0b5090 con 0x7fc5e41538a0 2023-10-30T13:41:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 -- 172.21.15.46:0/1537412586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5e404e170 con 0x7fc5e41538a0 2023-10-30T13:41:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5d2ffd640 1 -- 172.21.15.46:0/1537412586 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6537 (secure 0 0 0) 0x7fc5dc017070 con 0x7fc5e41538a0 2023-10-30T13:41:02.806 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:02.806 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6201,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:40:29.124030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:02.807 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 89 2023-10-30T13:41:02.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 -- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5c40664a0 msgr2=0x7fc5c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:02.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5c40664a0 0x7fc5c4068960 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40b2000 tx=0x7fc5d40aa070 comp rx=0 tx=0).stop 2023-10-30T13:41:02.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 -- 172.21.15.46:0/1537412586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5e41538a0 msgr2=0x7fc5e414e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:02.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5e41538a0 0x7fc5e414e0f0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00ca70 tx=0x7fc5dc00cf30 comp rx=0 tx=0).stop 2023-10-30T13:41:02.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 -- 172.21.15.46:0/1537412586 shutdown_connections 2023-10-30T13:41:02.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc5c40664a0 0x7fc5c4068960 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:02.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5e41538a0 0x7fc5e414e0f0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:02.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5e40cf160 0x7fc5e4157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:02.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 --2- 172.21.15.46:0/1537412586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e40ca0e0 0x7fc5e41591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:02.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 -- 172.21.15.46:0/1537412586 >> 172.21.15.46:0/1537412586 conn(0x7fc5e406bfe0 msgr2=0x7fc5e406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:02.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 -- 172.21.15.46:0/1537412586 shutdown_connections 2023-10-30T13:41:02.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:02.802+0000 7fc5e8cea640 1 -- 172.21.15.46:0/1537412586 wait complete. 2023-10-30T13:41:02.827 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:41:04.828 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.978+0000 7f078ced6640 1 Processor -- start 2023-10-30T13:41:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.978+0000 7f078ced6640 1 -- start start 2023-10-30T13:41:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f078ced6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0788156f80 0x7f0788159880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f078ced6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0788070c90 0x7f0788164120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f078ced6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07880c2f20 con 0x7f0788070650 2023-10-30T13:41:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f078ced6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0788157360 con 0x7f0788070c90 2023-10-30T13:41:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f078ced6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07880c85c0 con 0x7f0788156f80 2023-10-30T13:41:04.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f0786d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0788070c90 0x7f0788164120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:04.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f0786d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0788070c90 0x7f0788164120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55204/0 (socket says 172.21.15.46:55204) 2023-10-30T13:41:04.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f0786d9d640 1 -- 172.21.15.46:0/2513198310 learned_addr learned my addr 172.21.15.46:0/2513198310 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:04.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f078659c640 1 --2- 172.21.15.46:0/2513198310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0788156f80 0x7f0788159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:04.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f0786d9d640 1 -- 172.21.15.46:0/2513198310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0788156f80 msgr2=0x7f0788159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:04.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f0786d9d640 1 --2- 172.21.15.46:0/2513198310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0788156f80 0x7f0788159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:04.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f0786d9d640 1 -- 172.21.15.46:0/2513198310 >> v1:172.21.15.154:6789/0 conn(0x7f0788070650 legacy=0x7f0788071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:04.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f0786d9d640 1 -- 172.21.15.46:0/2513198310 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0788071230 con 0x7f0788070c90 2023-10-30T13:41:04.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f0786d9d640 1 --2- 172.21.15.46:0/2513198310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0788070c90 0x7f0788164120 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0780009e50 tx=0x7f078001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d9e26751d97f21 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:04.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f0785d9b640 1 -- 172.21.15.46:0/2513198310 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07800a58a0 con 0x7f0788070c90 2023-10-30T13:41:04.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f0785d9b640 1 -- 172.21.15.46:0/2513198310 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07800a5a00 con 0x7f0788070c90 2023-10-30T13:41:04.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.982+0000 7f0785d9b640 1 -- 172.21.15.46:0/2513198310 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07800a5d90 con 0x7f0788070c90 2023-10-30T13:41:04.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 -- 172.21.15.46:0/2513198310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0788070c90 msgr2=0x7f0788164120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:04.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 --2- 172.21.15.46:0/2513198310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0788070c90 0x7f0788164120 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f0780009e50 tx=0x7f078001ca10 comp rx=0 tx=0).stop 2023-10-30T13:41:04.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 -- 172.21.15.46:0/2513198310 shutdown_connections 2023-10-30T13:41:04.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 --2- 172.21.15.46:0/2513198310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0788070c90 0x7f0788164120 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:04.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 --2- 172.21.15.46:0/2513198310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0788156f80 0x7f0788159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:04.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 -- 172.21.15.46:0/2513198310 >> 172.21.15.46:0/2513198310 conn(0x7f07880cbeb0 msgr2=0x7f07880cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:04.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 -- 172.21.15.46:0/2513198310 shutdown_connections 2023-10-30T13:41:04.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 -- 172.21.15.46:0/2513198310 wait complete. 2023-10-30T13:41:04.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 Processor -- start 2023-10-30T13:41:04.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 -- start start 2023-10-30T13:41:04.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0788070650 0x7f07881562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:04.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0788070c90 0x7f078806b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:04.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0788156f80 0x7f07881464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:04.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0788159b80 con 0x7f0788070650 2023-10-30T13:41:04.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0788157b30 con 0x7f0788156f80 2023-10-30T13:41:04.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078ced6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07881576a0 con 0x7f0788070c90 2023-10-30T13:41:04.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0788070650 0x7f07881562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:04.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0788070650 0x7f07881562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55218/0 (socket says 172.21.15.46:55218) 2023-10-30T13:41:04.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f078659c640 1 -- 172.21.15.46:0/2601697436 learned_addr learned my addr 172.21.15.46:0/2601697436 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:04.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f0786d9d640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0788156f80 0x7f07881464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:04.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.986+0000 7f077ffff640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0788070c90 0x7f078806b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:04.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f078659c640 1 -- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0788070c90 msgr2=0x7f078806b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:04.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f078659c640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0788070c90 0x7f078806b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:04.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f078659c640 1 -- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0788156f80 msgr2=0x7f07881464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:04.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f078659c640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0788156f80 0x7f07881464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:04.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f078659c640 1 -- 172.21.15.46:0/2601697436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0780009b30 con 0x7f0788070650 2023-10-30T13:41:04.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f078659c640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0788070650 0x7f07881562e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f077800d800 tx=0x7f077800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f077effd640 1 -- 172.21.15.46:0/2601697436 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077800b6c0 con 0x7f0788070650 2023-10-30T13:41:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f077effd640 1 -- 172.21.15.46:0/2601697436 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077800b820 con 0x7f0788070650 2023-10-30T13:41:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f077effd640 1 -- 172.21.15.46:0/2601697436 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077800bb90 con 0x7f0788070650 2023-10-30T13:41:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f078ced6640 1 -- 172.21.15.46:0/2601697436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0788155640 con 0x7f0788070650 2023-10-30T13:41:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f078ced6640 1 -- 172.21.15.46:0/2601697436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07881595c0 con 0x7f0788070650 2023-10-30T13:41:04.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f078ced6640 1 -- 172.21.15.46:0/2601697436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078804e170 con 0x7f0788070650 2023-10-30T13:41:04.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f077effd640 1 -- 172.21.15.46:0/2601697436 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f077800bcf0 con 0x7f0788070650 2023-10-30T13:41:04.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f077effd640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07680664a0 0x7f0768068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:04.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.990+0000 7f077effd640 1 -- 172.21.15.46:0/2601697436 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f0778084770 con 0x7f0788070650 2023-10-30T13:41:04.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.994+0000 7f077ffff640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07680664a0 0x7f0768068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:04.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.994+0000 7f077ffff640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07680664a0 0x7f0768068960 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f07880ca3c0 tx=0x7f0770009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:05.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:04.998+0000 7f077effd640 1 -- 172.21.15.46:0/2601697436 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07780517b0 con 0x7f0788070650 2023-10-30T13:41:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.190+0000 7f078ced6640 1 -- 172.21.15.46:0/2601697436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f078806ab40 con 0x7f0788070650 2023-10-30T13:41:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.190+0000 7f077effd640 1 -- 172.21.15.46:0/2601697436 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6537 (secure 0 0 0) 0x7f0778054dd0 con 0x7f0788070650 2023-10-30T13:41:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:05.195 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6201,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:40:29.124030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 89 2023-10-30T13:41:05.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.194+0000 7f078ced6640 1 -- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07680664a0 msgr2=0x7f0768068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:05.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.194+0000 7f078ced6640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07680664a0 0x7f0768068960 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f07880ca3c0 tx=0x7f0770009040 comp rx=0 tx=0).stop 2023-10-30T13:41:05.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.194+0000 7f078ced6640 1 -- 172.21.15.46:0/2601697436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0788070650 msgr2=0x7f07881562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:05.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.194+0000 7f078ced6640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0788070650 0x7f07881562e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f077800d800 tx=0x7f077800dcc0 comp rx=0 tx=0).stop 2023-10-30T13:41:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.194+0000 7f078ced6640 1 -- 172.21.15.46:0/2601697436 shutdown_connections 2023-10-30T13:41:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.194+0000 7f078ced6640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07680664a0 0x7f0768068960 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.194+0000 7f078ced6640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0788156f80 0x7f07881464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.194+0000 7f078ced6640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0788070c90 0x7f078806b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.194+0000 7f078ced6640 1 --2- 172.21.15.46:0/2601697436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0788070650 0x7f07881562e0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.194+0000 7f078ced6640 1 -- 172.21.15.46:0/2601697436 >> 172.21.15.46:0/2601697436 conn(0x7f07880cbeb0 msgr2=0x7f078806a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.194+0000 7f078ced6640 1 -- 172.21.15.46:0/2601697436 shutdown_connections 2023-10-30T13:41:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:05.194+0000 7f078ced6640 1 -- 172.21.15.46:0/2601697436 wait complete. 2023-10-30T13:41:05.216 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:41:07.217 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37bbc4640 1 Processor -- start 2023-10-30T13:41:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37bbc4640 1 -- start start 2023-10-30T13:41:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37bbc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3741538a0 0x7fa37415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37bbc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa3740cf160 0x7fa3740cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37bbc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3740cb800 con 0x7fa3740ca0e0 2023-10-30T13:41:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37bbc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37415dc30 con 0x7fa3740cf160 2023-10-30T13:41:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37bbc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3740cb330 con 0x7fa3741538a0 2023-10-30T13:41:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37a161640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa3740cf160 0x7fa3740cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa379960640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3741538a0 0x7fa37415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37915f640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa3740ca0e0 0x7fa37415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:50520/0 (socket says 172.21.15.46:50520) 2023-10-30T13:41:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37a161640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa3740cf160 0x7fa3740cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:34422/0 (socket says 172.21.15.46:34422) 2023-10-30T13:41:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa379960640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3741538a0 0x7fa37415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:42596/0 (socket says 172.21.15.46:42596) 2023-10-30T13:41:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37915f640 1 -- 172.21.15.46:0/2760615046 learned_addr learned my addr 172.21.15.46:0/2760615046 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:41:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37a161640 1 -- 172.21.15.46:0/2760615046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3741538a0 msgr2=0x7fa37415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37a161640 1 --2- 172.21.15.46:0/2760615046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3741538a0 0x7fa37415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37a161640 1 -- 172.21.15.46:0/2760615046 >> v1:172.21.15.154:6789/0 conn(0x7fa3740ca0e0 legacy=0x7fa37415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa37a161640 1 -- 172.21.15.46:0/2760615046 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3740cad70 con 0x7fa3740cf160 2023-10-30T13:41:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.366+0000 7fa379960640 1 --2- 172.21.15.46:0/2760615046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3741538a0 0x7fa37415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:41:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.370+0000 7fa37a161640 1 --2- 172.21.15.46:0/2760615046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3740cf160 0x7fa3740cf590 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009d90 tx=0x7fa36c01f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b479003616407a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.370+0000 7fa37895e640 1 -- 172.21.15.46:0/2760615046 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c0af600 con 0x7fa3740cf160 2023-10-30T13:41:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.370+0000 7fa37895e640 1 -- 172.21.15.46:0/2760615046 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36c0ab050 con 0x7fa3740cf160 2023-10-30T13:41:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.370+0000 7fa37895e640 1 -- 172.21.15.46:0/2760615046 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c0af980 con 0x7fa3740cf160 2023-10-30T13:41:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.370+0000 7fa37bbc4640 1 -- 172.21.15.46:0/2760615046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3740cf160 msgr2=0x7fa3740cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.370+0000 7fa37bbc4640 1 --2- 172.21.15.46:0/2760615046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3740cf160 0x7fa3740cf590 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009d90 tx=0x7fa36c01f130 comp rx=0 tx=0).stop 2023-10-30T13:41:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.370+0000 7fa37bbc4640 1 -- 172.21.15.46:0/2760615046 shutdown_connections 2023-10-30T13:41:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.370+0000 7fa37bbc4640 1 --2- 172.21.15.46:0/2760615046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3740cf160 0x7fa3740cf590 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.370+0000 7fa37bbc4640 1 --2- 172.21.15.46:0/2760615046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3741538a0 0x7fa37415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.370+0000 7fa37bbc4640 1 -- 172.21.15.46:0/2760615046 >> 172.21.15.46:0/2760615046 conn(0x7fa37406bfe0 msgr2=0x7fa37406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.370+0000 7fa37bbc4640 1 -- 172.21.15.46:0/2760615046 shutdown_connections 2023-10-30T13:41:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.370+0000 7fa37bbc4640 1 -- 172.21.15.46:0/2760615046 wait complete. 2023-10-30T13:41:07.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37bbc4640 1 Processor -- start 2023-10-30T13:41:07.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37bbc4640 1 -- start start 2023-10-30T13:41:07.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37bbc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3740ca0e0 0x7fa3741e4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:07.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37bbc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3740cf160 0x7fa3741ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:07.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37bbc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3741538a0 0x7fa3741efdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:07.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37bbc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3741ef400 con 0x7fa3740cf160 2023-10-30T13:41:07.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37bbc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3741f3670 con 0x7fa3740ca0e0 2023-10-30T13:41:07.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37bbc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3741f2850 con 0x7fa3741538a0 2023-10-30T13:41:07.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa379960640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3740ca0e0 0x7fa3741e4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:07.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37915f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3740cf160 0x7fa3741ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:07.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa379960640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3740ca0e0 0x7fa3741e4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:34426/0 (socket says 172.21.15.46:34426) 2023-10-30T13:41:07.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa379960640 1 -- 172.21.15.46:0/1903782463 learned_addr learned my addr 172.21.15.46:0/1903782463 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:07.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37a161640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3741538a0 0x7fa3741efdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37915f640 1 -- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3741538a0 msgr2=0x7fa3741efdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37915f640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3741538a0 0x7fa3741efdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37915f640 1 -- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3740ca0e0 msgr2=0x7fa3741e4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37915f640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3740ca0e0 0x7fa3741e4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37915f640 1 -- 172.21.15.46:0/1903782463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa370009420 con 0x7fa3740cf160 2023-10-30T13:41:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37915f640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3740cf160 0x7fa3741ef890 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa36400c390 tx=0x7fa36400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa36a7fc640 1 -- 172.21.15.46:0/1903782463 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364005cf0 con 0x7fa3740cf160 2023-10-30T13:41:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa36a7fc640 1 -- 172.21.15.46:0/1903782463 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa364005e50 con 0x7fa3740cf160 2023-10-30T13:41:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa36a7fc640 1 -- 172.21.15.46:0/1903782463 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3640072f0 con 0x7fa3740cf160 2023-10-30T13:41:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37bbc4640 1 -- 172.21.15.46:0/1903782463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36c009a70 con 0x7fa3740cf160 2023-10-30T13:41:07.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.374+0000 7fa37bbc4640 1 -- 172.21.15.46:0/1903782463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3741f2af0 con 0x7fa3740cf160 2023-10-30T13:41:07.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.378+0000 7fa37bbc4640 1 -- 172.21.15.46:0/1903782463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3740b2a50 con 0x7fa3740cf160 2023-10-30T13:41:07.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.378+0000 7fa36a7fc640 1 -- 172.21.15.46:0/1903782463 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa364007450 con 0x7fa3740cf160 2023-10-30T13:41:07.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.378+0000 7fa36a7fc640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3580664a0 0x7fa358068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:07.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.378+0000 7fa36a7fc640 1 -- 172.21.15.46:0/1903782463 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fa3640836c0 con 0x7fa3740cf160 2023-10-30T13:41:07.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.378+0000 7fa379960640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3580664a0 0x7fa358068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:07.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.378+0000 7fa379960640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3580664a0 0x7fa358068960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa370009d50 tx=0x7fa370009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:07.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.382+0000 7fa36a7fc640 1 -- 172.21.15.46:0/1903782463 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa364050840 con 0x7fa3740cf160 2023-10-30T13:41:07.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 -- 172.21.15.46:0/1903782463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa37404e170 con 0x7fa3740cf160 2023-10-30T13:41:07.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa36a7fc640 1 -- 172.21.15.46:0/1903782463 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6537 (secure 0 0 0) 0x7fa364053e60 con 0x7fa3740cf160 2023-10-30T13:41:07.598 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:07.599 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6201,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:40:29.124030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:07.599 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 89 2023-10-30T13:41:07.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 -- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3580664a0 msgr2=0x7fa358068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:07.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3580664a0 0x7fa358068960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa370009d50 tx=0x7fa370009190 comp rx=0 tx=0).stop 2023-10-30T13:41:07.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 -- 172.21.15.46:0/1903782463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3740cf160 msgr2=0x7fa3741ef890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:07.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3740cf160 0x7fa3741ef890 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa36400c390 tx=0x7fa36400c850 comp rx=0 tx=0).stop 2023-10-30T13:41:07.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 -- 172.21.15.46:0/1903782463 shutdown_connections 2023-10-30T13:41:07.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3580664a0 0x7fa358068960 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:07.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3741538a0 0x7fa3741efdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:07.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3740cf160 0x7fa3741ef890 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:07.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 --2- 172.21.15.46:0/1903782463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3740ca0e0 0x7fa3741e4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:07.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 -- 172.21.15.46:0/1903782463 >> 172.21.15.46:0/1903782463 conn(0x7fa37406bfe0 msgr2=0x7fa374149840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:07.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 -- 172.21.15.46:0/1903782463 shutdown_connections 2023-10-30T13:41:07.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:07.594+0000 7fa37bbc4640 1 -- 172.21.15.46:0/1903782463 wait complete. 2023-10-30T13:41:07.620 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:41:09.620 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:09.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.770+0000 7fb97b5d3640 1 Processor -- start 2023-10-30T13:41:09.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.770+0000 7fb97b5d3640 1 -- start start 2023-10-30T13:41:09.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb97b5d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb9741580e0 0x7fb974070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:09.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb97b5d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb97415e700 0x7fb97406a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:09.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb97b5d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9740c5810 con 0x7fb9740c6540 2023-10-30T13:41:09.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb97b5d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9740c6990 con 0x7fb9741580e0 2023-10-30T13:41:09.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb97b5d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97415eb90 con 0x7fb97415e700 2023-10-30T13:41:09.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb97936f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb9741580e0 0x7fb974070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:09.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb97936f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb9741580e0 0x7fb974070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53376/0 (socket says 172.21.15.46:53376) 2023-10-30T13:41:09.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb97936f640 1 -- 172.21.15.46:0/656677771 learned_addr learned my addr 172.21.15.46:0/656677771 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:09.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb979b70640 1 --2- 172.21.15.46:0/656677771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb97415e700 0x7fb97406a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:09.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb97936f640 1 -- 172.21.15.46:0/656677771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb97415e700 msgr2=0x7fb97406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:09.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb97936f640 1 --2- 172.21.15.46:0/656677771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb97415e700 0x7fb97406a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:09.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb97936f640 1 -- 172.21.15.46:0/656677771 >> v1:172.21.15.46:6789/0 conn(0x7fb9740c6540 legacy=0x7fb974069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:09.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.774+0000 7fb97936f640 1 -- 172.21.15.46:0/656677771 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9740c6bb0 con 0x7fb9741580e0 2023-10-30T13:41:09.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb97936f640 1 --2- 172.21.15.46:0/656677771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9741580e0 0x7fb974070740 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb970009870 tx=0x7fb97001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde724f5acbe67b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:09.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb96bfff640 1 -- 172.21.15.46:0/656677771 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9700aa670 con 0x7fb9741580e0 2023-10-30T13:41:09.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb96bfff640 1 -- 172.21.15.46:0/656677771 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9700aa7d0 con 0x7fb9741580e0 2023-10-30T13:41:09.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb96bfff640 1 -- 172.21.15.46:0/656677771 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9700aab60 con 0x7fb9741580e0 2023-10-30T13:41:09.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb97b5d3640 1 -- 172.21.15.46:0/656677771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9741580e0 msgr2=0x7fb974070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:09.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb97b5d3640 1 --2- 172.21.15.46:0/656677771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9741580e0 0x7fb974070740 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb970009870 tx=0x7fb97001c510 comp rx=0 tx=0).stop 2023-10-30T13:41:09.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb97b5d3640 1 -- 172.21.15.46:0/656677771 shutdown_connections 2023-10-30T13:41:09.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb97b5d3640 1 --2- 172.21.15.46:0/656677771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb97415e700 0x7fb97406a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:09.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb97b5d3640 1 --2- 172.21.15.46:0/656677771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9741580e0 0x7fb974070740 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:09.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb97b5d3640 1 -- 172.21.15.46:0/656677771 >> 172.21.15.46:0/656677771 conn(0x7fb9740c7910 msgr2=0x7fb9740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:09.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb97b5d3640 1 -- 172.21.15.46:0/656677771 shutdown_connections 2023-10-30T13:41:09.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb97b5d3640 1 -- 172.21.15.46:0/656677771 wait complete. 2023-10-30T13:41:09.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb97b5d3640 1 Processor -- start 2023-10-30T13:41:09.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.778+0000 7fb97b5d3640 1 -- start start 2023-10-30T13:41:09.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97b5d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740c6540 0x7fb9740c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:09.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97b5d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9741580e0 0x7fb9741479c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:09.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97b5d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb97415e700 0x7fb974147f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:09.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97b5d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740c0f20 con 0x7fb9740c6540 2023-10-30T13:41:09.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97b5d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb974145e40 con 0x7fb97415e700 2023-10-30T13:41:09.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97b5d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740c1720 con 0x7fb9741580e0 2023-10-30T13:41:09.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97936f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740c6540 0x7fb9740c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:09.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97936f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740c6540 0x7fb9740c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53386/0 (socket says 172.21.15.46:53386) 2023-10-30T13:41:09.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97936f640 1 -- 172.21.15.46:0/1284149528 learned_addr learned my addr 172.21.15.46:0/1284149528 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:09.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb978b6e640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9741580e0 0x7fb9741479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:09.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb979b70640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb97415e700 0x7fb974147f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:09.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97936f640 1 -- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9741580e0 msgr2=0x7fb9741479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:09.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97936f640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9741580e0 0x7fb9741479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:09.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97936f640 1 -- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb97415e700 msgr2=0x7fb974147f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:09.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97936f640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb97415e700 0x7fb974147f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:09.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97936f640 1 -- 172.21.15.46:0/1284149528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9700094d0 con 0x7fb9740c6540 2023-10-30T13:41:09.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb979b70640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb97415e700 0x7fb974147f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:41:09.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97936f640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740c6540 0x7fb9740c8e70 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb9700ac040 tx=0x7fb97001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:09.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb969ffb640 1 -- 172.21.15.46:0/1284149528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9700b3800 con 0x7fb9740c6540 2023-10-30T13:41:09.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb969ffb640 1 -- 172.21.15.46:0/1284149528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9700b3980 con 0x7fb9740c6540 2023-10-30T13:41:09.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97b5d3640 1 -- 172.21.15.46:0/1284149528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb974149690 con 0x7fb9740c6540 2023-10-30T13:41:09.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb969ffb640 1 -- 172.21.15.46:0/1284149528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9700b3d30 con 0x7fb9740c6540 2023-10-30T13:41:09.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97b5d3640 1 -- 172.21.15.46:0/1284149528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9740c19c0 con 0x7fb9740c6540 2023-10-30T13:41:09.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb97b5d3640 1 -- 172.21.15.46:0/1284149528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97404e170 con 0x7fb9740c6540 2023-10-30T13:41:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.782+0000 7fb969ffb640 1 -- 172.21.15.46:0/1284149528 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9700c6390 con 0x7fb9740c6540 2023-10-30T13:41:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.786+0000 7fb969ffb640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9580664d0 0x7fb958068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.786+0000 7fb969ffb640 1 -- 172.21.15.46:0/1284149528 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fb97011ef30 con 0x7fb9740c6540 2023-10-30T13:41:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.786+0000 7fb978b6e640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9580664d0 0x7fb958068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:09.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.786+0000 7fb978b6e640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9580664d0 0x7fb958068990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb97414eb00 tx=0x7fb964009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:09.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.790+0000 7fb969ffb640 1 -- 172.21.15.46:0/1284149528 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9700ebee0 con 0x7fb9740c6540 2023-10-30T13:41:09.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.978+0000 7fb97b5d3640 1 -- 172.21.15.46:0/1284149528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb97406bfe0 con 0x7fb9740c6540 2023-10-30T13:41:09.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.978+0000 7fb969ffb640 1 -- 172.21.15.46:0/1284149528 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6537 (secure 0 0 0) 0x7fb9700ef500 con 0x7fb9740c6540 2023-10-30T13:41:09.983 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:09.983 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6201,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:40:29.124030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 89 2023-10-30T13:41:09.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.982+0000 7fb97b5d3640 1 -- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9580664d0 msgr2=0x7fb958068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:09.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.982+0000 7fb97b5d3640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9580664d0 0x7fb958068990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb97414eb00 tx=0x7fb964009040 comp rx=0 tx=0).stop 2023-10-30T13:41:09.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.982+0000 7fb97b5d3640 1 -- 172.21.15.46:0/1284149528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740c6540 msgr2=0x7fb9740c8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:09.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.982+0000 7fb97b5d3640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740c6540 0x7fb9740c8e70 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb9700ac040 tx=0x7fb97001ad50 comp rx=0 tx=0).stop 2023-10-30T13:41:09.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.982+0000 7fb97b5d3640 1 -- 172.21.15.46:0/1284149528 shutdown_connections 2023-10-30T13:41:09.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.982+0000 7fb97b5d3640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9580664d0 0x7fb958068990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:09.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.982+0000 7fb97b5d3640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb97415e700 0x7fb974147f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:09.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.982+0000 7fb97b5d3640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9741580e0 0x7fb9741479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:09.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.982+0000 7fb97b5d3640 1 --2- 172.21.15.46:0/1284149528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740c6540 0x7fb9740c8e70 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:09.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.982+0000 7fb97b5d3640 1 -- 172.21.15.46:0/1284149528 >> 172.21.15.46:0/1284149528 conn(0x7fb9740c7910 msgr2=0x7fb974153c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:09.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.982+0000 7fb97b5d3640 1 -- 172.21.15.46:0/1284149528 shutdown_connections 2023-10-30T13:41:09.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:09.982+0000 7fb97b5d3640 1 -- 172.21.15.46:0/1284149528 wait complete. 2023-10-30T13:41:10.004 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:41:12.006 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:12.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a95d65640 1 Processor -- start 2023-10-30T13:41:12.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a95d65640 1 -- start start 2023-10-30T13:41:12.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a95d65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a901538a0 0x7f1a9015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:12.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a95d65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a900ca0e0 0x7f1a9015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:12.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a95d65640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a900cf160 0x7f1a900cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:12.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a95d65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a900cb800 con 0x7f1a900ca0e0 2023-10-30T13:41:12.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a95d65640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a9015dc30 con 0x7f1a900cf160 2023-10-30T13:41:12.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a95d65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a900cb330 con 0x7f1a901538a0 2023-10-30T13:41:12.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a8effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a900ca0e0 0x7f1a9015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a8effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a900ca0e0 0x7f1a9015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53392/0 (socket says 172.21.15.46:53392) 2023-10-30T13:41:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a8effd640 1 -- 172.21.15.46:0/1245673418 learned_addr learned my addr 172.21.15.46:0/1245673418 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a8ffff640 1 --2- 172.21.15.46:0/1245673418 >> v2:172.21.15.154:3300/0 conn(0x7f1a900cf160 0x7f1a900cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a8f7fe640 1 --2- 172.21.15.46:0/1245673418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a901538a0 0x7f1a9015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a8effd640 1 -- 172.21.15.46:0/1245673418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a901538a0 msgr2=0x7f1a9015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a8effd640 1 --2- 172.21.15.46:0/1245673418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a901538a0 0x7f1a9015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a8effd640 1 -- 172.21.15.46:0/1245673418 >> v2:172.21.15.154:3300/0 conn(0x7f1a900cf160 msgr2=0x7f1a900cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:12.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a8effd640 1 --2- 172.21.15.46:0/1245673418 >> v2:172.21.15.154:3300/0 conn(0x7f1a900cf160 0x7f1a900cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a8effd640 1 -- 172.21.15.46:0/1245673418 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a900cad70 con 0x7f1a900ca0e0 2023-10-30T13:41:12.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.158+0000 7f1a8f7fe640 1 --2- 172.21.15.46:0/1245673418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a901538a0 0x7f1a9015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:12.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a8effd640 1 --2- 172.21.15.46:0/1245673418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a900ca0e0 0x7f1a9015a090 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1a9006cb00 tx=0x7f1a8401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03b92c4ad566062 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a8e7fc640 1 -- 172.21.15.46:0/1245673418 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a840a89b0 con 0x7f1a900ca0e0 2023-10-30T13:41:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a8e7fc640 1 -- 172.21.15.46:0/1245673418 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a840a8b10 con 0x7f1a900ca0e0 2023-10-30T13:41:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a8e7fc640 1 -- 172.21.15.46:0/1245673418 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a840a8ea0 con 0x7f1a900ca0e0 2023-10-30T13:41:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a95d65640 1 -- 172.21.15.46:0/1245673418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a900ca0e0 msgr2=0x7f1a9015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a95d65640 1 --2- 172.21.15.46:0/1245673418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a900ca0e0 0x7f1a9015a090 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1a9006cb00 tx=0x7f1a8401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a95d65640 1 -- 172.21.15.46:0/1245673418 shutdown_connections 2023-10-30T13:41:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a95d65640 1 --2- 172.21.15.46:0/1245673418 >> v2:172.21.15.154:3300/0 conn(0x7f1a900cf160 0x7f1a900cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a95d65640 1 --2- 172.21.15.46:0/1245673418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a900ca0e0 0x7f1a9015a090 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a95d65640 1 --2- 172.21.15.46:0/1245673418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a901538a0 0x7f1a9015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a95d65640 1 -- 172.21.15.46:0/1245673418 >> 172.21.15.46:0/1245673418 conn(0x7f1a9006bfe0 msgr2=0x7f1a9006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:12.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a95d65640 1 -- 172.21.15.46:0/1245673418 shutdown_connections 2023-10-30T13:41:12.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.162+0000 7f1a95d65640 1 -- 172.21.15.46:0/1245673418 wait complete. 2023-10-30T13:41:12.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a95d65640 1 Processor -- start 2023-10-30T13:41:12.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a95d65640 1 -- start start 2023-10-30T13:41:12.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a95d65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a900ca0e0 0x7f1a901efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:12.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a95d65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a900cf160 0x7f1a900ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:12.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a95d65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a901538a0 0x7f1a901e5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:12.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a95d65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a90142670 con 0x7f1a900ca0e0 2023-10-30T13:41:12.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a95d65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9015bf20 con 0x7f1a901538a0 2023-10-30T13:41:12.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a95d65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a901f0000 con 0x7f1a900cf160 2023-10-30T13:41:12.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a8f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a900ca0e0 0x7f1a901efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:12.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a8f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a900ca0e0 0x7f1a901efac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53396/0 (socket says 172.21.15.46:53396) 2023-10-30T13:41:12.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a8f7fe640 1 -- 172.21.15.46:0/3350288883 learned_addr learned my addr 172.21.15.46:0/3350288883 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:12.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a8ffff640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a901538a0 0x7f1a901e5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:12.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a8effd640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a900cf160 0x7f1a900ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:12.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a8f7fe640 1 -- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a900cf160 msgr2=0x7f1a900ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:12.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a8f7fe640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a900cf160 0x7f1a900ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a8f7fe640 1 -- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a901538a0 msgr2=0x7f1a901e5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:12.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a8f7fe640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a901538a0 0x7f1a901e5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a8f7fe640 1 -- 172.21.15.46:0/3350288883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a840094d0 con 0x7f1a900ca0e0 2023-10-30T13:41:12.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a8ffff640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a901538a0 0x7f1a901e5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:12.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a8f7fe640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a900ca0e0 0x7f1a901efac0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000d800 tx=0x7f1a8000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:12.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a77fff640 1 -- 172.21.15.46:0/3350288883 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a800096c0 con 0x7f1a900ca0e0 2023-10-30T13:41:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a77fff640 1 -- 172.21.15.46:0/3350288883 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a80009820 con 0x7f1a900ca0e0 2023-10-30T13:41:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a77fff640 1 -- 172.21.15.46:0/3350288883 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80009bd0 con 0x7f1a900ca0e0 2023-10-30T13:41:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a95d65640 1 -- 172.21.15.46:0/3350288883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a901e6450 con 0x7f1a900ca0e0 2023-10-30T13:41:12.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.166+0000 7f1a95d65640 1 -- 172.21.15.46:0/3350288883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a901ef070 con 0x7f1a900ca0e0 2023-10-30T13:41:12.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.170+0000 7f1a95d65640 1 -- 172.21.15.46:0/3350288883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a900b2a50 con 0x7f1a900ca0e0 2023-10-30T13:41:12.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.170+0000 7f1a77fff640 1 -- 172.21.15.46:0/3350288883 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1a80009d30 con 0x7f1a900ca0e0 2023-10-30T13:41:12.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.170+0000 7f1a77fff640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a7c0664a0 0x7f1a7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:12.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.170+0000 7f1a77fff640 1 -- 172.21.15.46:0/3350288883 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f1a80084a50 con 0x7f1a900ca0e0 2023-10-30T13:41:12.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.170+0000 7f1a8effd640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a7c0664a0 0x7f1a7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:12.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.170+0000 7f1a8effd640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a7c0664a0 0x7f1a7c068960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1a840b2000 tx=0x7f1a84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:12.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.174+0000 7f1a77fff640 1 -- 172.21.15.46:0/3350288883 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a80051a00 con 0x7f1a900ca0e0 2023-10-30T13:41:12.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.362+0000 7f1a95d65640 1 -- 172.21.15.46:0/3350288883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a900b1dd0 con 0x7f1a900ca0e0 2023-10-30T13:41:12.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.362+0000 7f1a77fff640 1 -- 172.21.15.46:0/3350288883 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6537 (secure 0 0 0) 0x7f1a80055020 con 0x7f1a900ca0e0 2023-10-30T13:41:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:12.369 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6201,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:40:29.124030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5721,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6833/4044310929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4044310929},{"type":"v1","addr":"172.21.15.46:6833","nonce":4044310929}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 89 2023-10-30T13:41:12.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.366+0000 7f1a95d65640 1 -- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a7c0664a0 msgr2=0x7f1a7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:12.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.366+0000 7f1a95d65640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a7c0664a0 0x7f1a7c068960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1a840b2000 tx=0x7f1a84008040 comp rx=0 tx=0).stop 2023-10-30T13:41:12.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.366+0000 7f1a95d65640 1 -- 172.21.15.46:0/3350288883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a900ca0e0 msgr2=0x7f1a901efac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.366+0000 7f1a95d65640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a900ca0e0 0x7f1a901efac0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000d800 tx=0x7f1a8000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.366+0000 7f1a95d65640 1 -- 172.21.15.46:0/3350288883 shutdown_connections 2023-10-30T13:41:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.366+0000 7f1a95d65640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a7c0664a0 0x7f1a7c068960 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.366+0000 7f1a95d65640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a901538a0 0x7f1a901e5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.366+0000 7f1a95d65640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a900cf160 0x7f1a900ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.366+0000 7f1a95d65640 1 --2- 172.21.15.46:0/3350288883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a900ca0e0 0x7f1a901efac0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:12.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.366+0000 7f1a95d65640 1 -- 172.21.15.46:0/3350288883 >> 172.21.15.46:0/3350288883 conn(0x7f1a9006bfe0 msgr2=0x7f1a901525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:12.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.366+0000 7f1a95d65640 1 -- 172.21.15.46:0/3350288883 shutdown_connections 2023-10-30T13:41:12.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:12.366+0000 7f1a95d65640 1 -- 172.21.15.46:0/3350288883 wait complete. 2023-10-30T13:41:12.389 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:41:14.391 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedfd0ea640 1 Processor -- start 2023-10-30T13:41:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedfd0ea640 1 -- start start 2023-10-30T13:41:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedfd0ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fedf81538a0 0x7fedf815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedfd0ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf80ca0e0 0x7fedf815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedfd0ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf80cb800 con 0x7fedf80cf160 2023-10-30T13:41:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedfd0ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf815dc30 con 0x7fedf81538a0 2023-10-30T13:41:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedfd0ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80cb330 con 0x7fedf80ca0e0 2023-10-30T13:41:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf759e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fedf80cf160 0x7fedf80cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33430/0 (socket says 172.21.15.46:33430) 2023-10-30T13:41:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf759e640 1 -- 172.21.15.46:0/1878178140 learned_addr learned my addr 172.21.15.46:0/1878178140 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:41:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf6d9d640 1 --2- 172.21.15.46:0/1878178140 >> v2:172.21.15.154:3300/0 conn(0x7fedf81538a0 0x7fedf815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedeffff640 1 --2- 172.21.15.46:0/1878178140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf80ca0e0 0x7fedf815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257910478 0 0) 0x7fedf80cb800 con 0x7fedf80cf160 2023-10-30T13:41:14.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede4003530 con 0x7fedf80cf160 2023-10-30T13:41:14.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2724342499 0 0) 0x7fede4003530 con 0x7fedf80cf160 2023-10-30T13:41:14.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf80ca0e0 msgr2=0x7fedf815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:14.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 --2- 172.21.15.46:0/1878178140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf80ca0e0 0x7fedf815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 >> v2:172.21.15.154:3300/0 conn(0x7fedf81538a0 msgr2=0x7fedf815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:14.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 --2- 172.21.15.46:0/1878178140 >> v2:172.21.15.154:3300/0 conn(0x7fedf81538a0 0x7fedf815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80cad70 con 0x7fedf80cf160 2023-10-30T13:41:14.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fedf0001a00 con 0x7fedf80cf160 2023-10-30T13:41:14.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 >> v1:172.21.15.46:6789/0 conn(0x7fedf80cf160 legacy=0x7fedf80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:14.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 --2- 172.21.15.46:0/1878178140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fede4003980 0x7fede4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 --2- 172.21.15.46:0/1878178140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf80ca0e0 0x7fede40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.542+0000 7fedf659c640 1 --2- 172.21.15.46:0/1878178140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede4004d10 0x7fede4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf815dc30 con 0x7fede4004d10 2023-10-30T13:41:14.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80cb330 con 0x7fede4003980 2023-10-30T13:41:14.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede400b990 con 0x7fedf80ca0e0 2023-10-30T13:41:14.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf6d9d640 1 --2- 172.21.15.46:0/1878178140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fede4003980 0x7fede4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf759e640 1 --2- 172.21.15.46:0/1878178140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede4004d10 0x7fede4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedeffff640 1 --2- 172.21.15.46:0/1878178140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf80ca0e0 0x7fede40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf759e640 1 -- 172.21.15.46:0/1878178140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf80ca0e0 msgr2=0x7fede40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:14.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf759e640 1 --2- 172.21.15.46:0/1878178140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf80ca0e0 0x7fede40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf759e640 1 -- 172.21.15.46:0/1878178140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fede4003980 msgr2=0x7fede4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:14.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf759e640 1 --2- 172.21.15.46:0/1878178140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fede4003980 0x7fede4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf759e640 1 -- 172.21.15.46:0/1878178140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede4003530 con 0x7fede4004d10 2023-10-30T13:41:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf759e640 1 --2- 172.21.15.46:0/1878178140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede4004d10 0x7fede4005190 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fedf000a860 tx=0x7fedf001d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333f5b863c27003f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf00ad630 con 0x7fede4004d10 2023-10-30T13:41:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf00a8070 con 0x7fede4004d10 2023-10-30T13:41:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.546+0000 7fedf659c640 1 -- 172.21.15.46:0/1878178140 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf00ad9f0 con 0x7fede4004d10 2023-10-30T13:41:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 -- 172.21.15.46:0/1878178140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede4004d10 msgr2=0x7fede4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 --2- 172.21.15.46:0/1878178140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede4004d10 0x7fede4005190 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fedf000a860 tx=0x7fedf001d360 comp rx=0 tx=0).stop 2023-10-30T13:41:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedf759e640 1 -- 172.21.15.46:0/1878178140 reap_dead start 2023-10-30T13:41:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 -- 172.21.15.46:0/1878178140 shutdown_connections 2023-10-30T13:41:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 -- 172.21.15.46:0/1878178140 >> 172.21.15.46:0/1878178140 conn(0x7fedf806bfe0 msgr2=0x7fedf806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 -- 172.21.15.46:0/1878178140 shutdown_connections 2023-10-30T13:41:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 -- 172.21.15.46:0/1878178140 wait complete. 2023-10-30T13:41:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 Processor -- start 2023-10-30T13:41:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 -- start start 2023-10-30T13:41:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede4004d10 0x7fedf815ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf815b460 0x7fedf8157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedf814d6f0 0x7fedf8158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf8159730 con 0x7fede4004d10 2023-10-30T13:41:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf8157730 con 0x7fedf814d6f0 2023-10-30T13:41:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedfd0ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf8158c10 con 0x7fedf815b460 2023-10-30T13:41:14.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedf6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede4004d10 0x7fedf815ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedf6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede4004d10 0x7fedf815ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53418/0 (socket says 172.21.15.46:53418) 2023-10-30T13:41:14.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedf6d9d640 1 -- 172.21.15.46:0/614944280 learned_addr learned my addr 172.21.15.46:0/614944280 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:14.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedf759e640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedf814d6f0 0x7fedf8158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedeffff640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf815b460 0x7fedf8157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedf6d9d640 1 -- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf815b460 msgr2=0x7fedf8157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:14.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedf6d9d640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf815b460 0x7fedf8157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedf6d9d640 1 -- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedf814d6f0 msgr2=0x7fedf8158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:14.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedf6d9d640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedf814d6f0 0x7fedf8158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.550+0000 7fedf6d9d640 1 -- 172.21.15.46:0/614944280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf8158e90 con 0x7fede4004d10 2023-10-30T13:41:14.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.554+0000 7fedf6d9d640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede4004d10 0x7fedf815ab10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fede80119d0 tx=0x7fede8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:14.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.554+0000 7fedef7fe640 1 -- 172.21.15.46:0/614944280 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede800fb50 con 0x7fede4004d10 2023-10-30T13:41:14.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.554+0000 7fedef7fe640 1 -- 172.21.15.46:0/614944280 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede800fcb0 con 0x7fede4004d10 2023-10-30T13:41:14.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.554+0000 7fedef7fe640 1 -- 172.21.15.46:0/614944280 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8014440 con 0x7fede4004d10 2023-10-30T13:41:14.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.554+0000 7fedfd0ea640 1 -- 172.21.15.46:0/614944280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf8159170 con 0x7fede4004d10 2023-10-30T13:41:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.554+0000 7fedfd0ea640 1 -- 172.21.15.46:0/614944280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf80b4b70 con 0x7fede4004d10 2023-10-30T13:41:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.554+0000 7fedfd0ea640 1 -- 172.21.15.46:0/614944280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf8159870 con 0x7fede4004d10 2023-10-30T13:41:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.554+0000 7fedef7fe640 1 -- 172.21.15.46:0/614944280 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fede80145a0 con 0x7fede4004d10 2023-10-30T13:41:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.554+0000 7fedef7fe640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fedd80664d0 0x7fedd8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.554+0000 7fedef7fe640 1 -- 172.21.15.46:0/614944280 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fede8084c30 con 0x7fede4004d10 2023-10-30T13:41:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.554+0000 7fedeffff640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fedd80664d0 0x7fedd8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.558+0000 7fedeffff640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fedd80664d0 0x7fedd8068990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fedf815dc00 tx=0x7fede0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.558+0000 7fedef7fe640 1 -- 172.21.15.46:0/614944280 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fedf8159870 con 0x7fede4004d10 2023-10-30T13:41:14.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 -- 172.21.15.46:0/614944280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedf8159870 con 0x7fede4004d10 2023-10-30T13:41:14.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedef7fe640 1 -- 172.21.15.46:0/614944280 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fedf8159870 con 0x7fede4004d10 2023-10-30T13:41:14.758 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:14.758 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:14.000593+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:14.759 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 90 2023-10-30T13:41:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 -- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fedd80664d0 msgr2=0x7fedd8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fedd80664d0 0x7fedd8068990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fedf815dc00 tx=0x7fede0008040 comp rx=0 tx=0).stop 2023-10-30T13:41:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 -- 172.21.15.46:0/614944280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede4004d10 msgr2=0x7fedf815ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede4004d10 0x7fedf815ab10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fede80119d0 tx=0x7fede8011e90 comp rx=0 tx=0).stop 2023-10-30T13:41:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 -- 172.21.15.46:0/614944280 shutdown_connections 2023-10-30T13:41:14.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fedd80664d0 0x7fedd8068990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedf814d6f0 0x7fedf8158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedf815b460 0x7fedf8157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 --2- 172.21.15.46:0/614944280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fede4004d10 0x7fedf815ab10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 -- 172.21.15.46:0/614944280 >> 172.21.15.46:0/614944280 conn(0x7fedf806bfe0 msgr2=0x7fedf8149270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:14.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 -- 172.21.15.46:0/614944280 shutdown_connections 2023-10-30T13:41:14.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.754+0000 7fedfd0ea640 1 -- 172.21.15.46:0/614944280 wait complete. 2023-10-30T13:41:14.779 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:41:14.779 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:41:14.780 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:14.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.918+0000 7fa1e41aa640 1 Processor -- start 2023-10-30T13:41:14.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.918+0000 7fa1e41aa640 1 -- start start 2023-10-30T13:41:14.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e41aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1dc069e90 0x7fa1dc15a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e41aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa1dc06cb80 0x7fa1dc06d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e41aa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1dc06d550 con 0x7fa1dc06a270 2023-10-30T13:41:14.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e41aa640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1dc0c8570 con 0x7fa1dc06cb80 2023-10-30T13:41:14.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e41aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc0c6c10 con 0x7fa1dc069e90 2023-10-30T13:41:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e2747640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa1dc06cb80 0x7fa1dc06d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e2747640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa1dc06cb80 0x7fa1dc06d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53428/0 (socket says 172.21.15.46:53428) 2023-10-30T13:41:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e2747640 1 -- 172.21.15.46:0/3252720280 learned_addr learned my addr 172.21.15.46:0/3252720280 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:14.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e1f46640 1 --2- 172.21.15.46:0/3252720280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1dc069e90 0x7fa1dc15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e2747640 1 -- 172.21.15.46:0/3252720280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1dc069e90 msgr2=0x7fa1dc15a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e2747640 1 --2- 172.21.15.46:0/3252720280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1dc069e90 0x7fa1dc15a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e2747640 1 -- 172.21.15.46:0/3252720280 >> v1:172.21.15.46:6789/0 conn(0x7fa1dc06a270 legacy=0x7fa1dc0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:14.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e2747640 1 -- 172.21.15.46:0/3252720280 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc0c6ef0 con 0x7fa1dc06cb80 2023-10-30T13:41:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e2747640 1 --2- 172.21.15.46:0/3252720280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1dc06cb80 0x7fa1dc06d010 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4009e50 tx=0x7fa1d401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47933e6909bb09f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.922+0000 7fa1e0f44640 1 -- 172.21.15.46:0/3252720280 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d40a57d0 con 0x7fa1dc06cb80 2023-10-30T13:41:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e0f44640 1 -- 172.21.15.46:0/3252720280 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d40a5930 con 0x7fa1dc06cb80 2023-10-30T13:41:14.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e0f44640 1 -- 172.21.15.46:0/3252720280 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d40a5cc0 con 0x7fa1dc06cb80 2023-10-30T13:41:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 -- 172.21.15.46:0/3252720280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1dc06cb80 msgr2=0x7fa1dc06d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 --2- 172.21.15.46:0/3252720280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1dc06cb80 0x7fa1dc06d010 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4009e50 tx=0x7fa1d401ca10 comp rx=0 tx=0).stop 2023-10-30T13:41:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 -- 172.21.15.46:0/3252720280 shutdown_connections 2023-10-30T13:41:14.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 --2- 172.21.15.46:0/3252720280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1dc06cb80 0x7fa1dc06d010 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 --2- 172.21.15.46:0/3252720280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1dc069e90 0x7fa1dc15a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 -- 172.21.15.46:0/3252720280 >> 172.21.15.46:0/3252720280 conn(0x7fa1dc06dfd0 msgr2=0x7fa1dc06e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 -- 172.21.15.46:0/3252720280 shutdown_connections 2023-10-30T13:41:14.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 -- 172.21.15.46:0/3252720280 wait complete. 2023-10-30T13:41:14.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 Processor -- start 2023-10-30T13:41:14.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 -- start start 2023-10-30T13:41:14.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1dc069e90 0x7fa1dc156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1dc06a270 0x7fa1dc14d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1dc06cb80 0x7fa1dc14de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc15d6b0 con 0x7fa1dc06a270 2023-10-30T13:41:14.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc15c130 con 0x7fa1dc069e90 2023-10-30T13:41:14.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e41aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc15d970 con 0x7fa1dc06cb80 2023-10-30T13:41:14.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e1745640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1dc06a270 0x7fa1dc14d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e1745640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1dc06a270 0x7fa1dc14d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53440/0 (socket says 172.21.15.46:53440) 2023-10-30T13:41:14.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e1745640 1 -- 172.21.15.46:0/4059777983 learned_addr learned my addr 172.21.15.46:0/4059777983 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:14.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e2747640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1dc06cb80 0x7fa1dc14de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.926+0000 7fa1e1f46640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1dc069e90 0x7fa1dc156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1e1745640 1 -- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1dc06cb80 msgr2=0x7fa1dc14de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:14.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1e1745640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1dc06cb80 0x7fa1dc14de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1e1745640 1 -- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1dc069e90 msgr2=0x7fa1dc156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:14.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1e1745640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1dc069e90 0x7fa1dc156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:14.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1e1745640 1 -- 172.21.15.46:0/4059777983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d4009b30 con 0x7fa1dc06a270 2023-10-30T13:41:14.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1e1f46640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1dc069e90 0x7fa1dc156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:14.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1e1745640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1dc06a270 0x7fa1dc14d8c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc00c500 tx=0x7fa1cc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:14.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1d27fc640 1 -- 172.21.15.46:0/4059777983 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc005220 con 0x7fa1dc06a270 2023-10-30T13:41:14.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1d27fc640 1 -- 172.21.15.46:0/4059777983 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc005380 con 0x7fa1dc06a270 2023-10-30T13:41:14.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1d27fc640 1 -- 172.21.15.46:0/4059777983 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc007270 con 0x7fa1dc06a270 2023-10-30T13:41:14.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1e41aa640 1 -- 172.21.15.46:0/4059777983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1dc15b890 con 0x7fa1dc06a270 2023-10-30T13:41:14.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1e41aa640 1 -- 172.21.15.46:0/4059777983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1dc15be50 con 0x7fa1dc06a270 2023-10-30T13:41:14.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1e41aa640 1 -- 172.21.15.46:0/4059777983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1dc04e170 con 0x7fa1dc06a270 2023-10-30T13:41:14.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1d27fc640 1 -- 172.21.15.46:0/4059777983 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1cc0073d0 con 0x7fa1dc06a270 2023-10-30T13:41:14.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1d27fc640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1c00664d0 0x7fa1c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:14.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.930+0000 7fa1d27fc640 1 -- 172.21.15.46:0/4059777983 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fa1cc083b00 con 0x7fa1dc06a270 2023-10-30T13:41:14.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.934+0000 7fa1e1f46640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1c00664d0 0x7fa1c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:14.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.934+0000 7fa1e1f46640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1c00664d0 0x7fa1c0068990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8007a70 tx=0x7fa1d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:14.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:14.934+0000 7fa1d27fc640 1 -- 172.21.15.46:0/4059777983 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa1cc085e60 con 0x7fa1dc06a270 2023-10-30T13:41:15.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.126+0000 7fa1e41aa640 1 -- 172.21.15.46:0/4059777983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1dc1571e0 con 0x7fa1dc06a270 2023-10-30T13:41:15.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.126+0000 7fa1d27fc640 1 -- 172.21.15.46:0/4059777983 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fa1cc050b00 con 0x7fa1dc06a270 2023-10-30T13:41:15.132 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:15.132 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:14.000593+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:15.132 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 90 2023-10-30T13:41:15.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.130+0000 7fa1e41aa640 1 -- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1c00664d0 msgr2=0x7fa1c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:15.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.130+0000 7fa1e41aa640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1c00664d0 0x7fa1c0068990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8007a70 tx=0x7fa1d8008040 comp rx=0 tx=0).stop 2023-10-30T13:41:15.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.130+0000 7fa1e41aa640 1 -- 172.21.15.46:0/4059777983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1dc06a270 msgr2=0x7fa1dc14d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:15.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.130+0000 7fa1e41aa640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1dc06a270 0x7fa1dc14d8c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc00c500 tx=0x7fa1cc00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.130+0000 7fa1e41aa640 1 -- 172.21.15.46:0/4059777983 shutdown_connections 2023-10-30T13:41:15.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.130+0000 7fa1e41aa640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1c00664d0 0x7fa1c0068990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.130+0000 7fa1e41aa640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1dc06cb80 0x7fa1dc14de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.130+0000 7fa1e41aa640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1dc06a270 0x7fa1dc14d8c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.130+0000 7fa1e41aa640 1 --2- 172.21.15.46:0/4059777983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1dc069e90 0x7fa1dc156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:15.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.130+0000 7fa1e41aa640 1 -- 172.21.15.46:0/4059777983 >> 172.21.15.46:0/4059777983 conn(0x7fa1dc06dfd0 msgr2=0x7fa1dc071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:15.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.130+0000 7fa1e41aa640 1 -- 172.21.15.46:0/4059777983 shutdown_connections 2023-10-30T13:41:15.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:15.130+0000 7fa1e41aa640 1 -- 172.21.15.46:0/4059777983 wait complete. 2023-10-30T13:41:15.154 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 90, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 90, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 28 }, "gid_6036": { "addr": "172.21.15.46:6836/3037060424", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3037060424, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3037060424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6036, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 37 }, "gid_6201": { "addr": "172.21.15.46:6837/2546436963", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2546436963, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2546436963, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6201, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:41:14.000593+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6201, "mds_1": 6036, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:41:17.155 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:17.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.306+0000 7f9061809640 1 Processor -- start 2023-10-30T13:41:17.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.306+0000 7f9061809640 1 -- start start 2023-10-30T13:41:17.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9061809640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f905c0c6b80 0x7f905c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:17.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9061809640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f905c0c6f60 0x7f905c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:17.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9061809640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905c0c0a50 con 0x7f905c0c8430 2023-10-30T13:41:17.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9061809640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905c06f6e0 con 0x7f905c0c6f60 2023-10-30T13:41:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9061809640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905c06f9c0 con 0x7f905c0c6b80 2023-10-30T13:41:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9053fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f905c0c6f60 0x7f905c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9053fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f905c0c6f60 0x7f905c0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50014/0 (socket says 172.21.15.46:50014) 2023-10-30T13:41:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9053fff640 1 -- 172.21.15.46:0/4252490008 learned_addr learned my addr 172.21.15.46:0/4252490008 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:17.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f905affd640 1 --2- 172.21.15.46:0/4252490008 >> v2:172.21.15.154:3300/0 conn(0x7f905c0c6b80 0x7f905c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:17.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9053fff640 1 -- 172.21.15.46:0/4252490008 >> v2:172.21.15.154:3300/0 conn(0x7f905c0c6b80 msgr2=0x7f905c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:17.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9053fff640 1 --2- 172.21.15.46:0/4252490008 >> v2:172.21.15.154:3300/0 conn(0x7f905c0c6b80 0x7f905c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9053fff640 1 -- 172.21.15.46:0/4252490008 >> v1:172.21.15.46:6789/0 conn(0x7f905c0c8430 legacy=0x7f905c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:17.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9053fff640 1 -- 172.21.15.46:0/4252490008 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c06fca0 con 0x7f905c0c6f60 2023-10-30T13:41:17.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f9053fff640 1 --2- 172.21.15.46:0/4252490008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c0c6f60 0x7f905c0c0400 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f905c06f1b0 tx=0x7f904401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0db4de7f8471d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:17.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.310+0000 7f905a7fc640 1 -- 172.21.15.46:0/4252490008 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90440ab610 con 0x7f905c0c6f60 2023-10-30T13:41:17.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f905a7fc640 1 -- 172.21.15.46:0/4252490008 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90440ab770 con 0x7f905c0c6f60 2023-10-30T13:41:17.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f905a7fc640 1 -- 172.21.15.46:0/4252490008 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90440abb00 con 0x7f905c0c6f60 2023-10-30T13:41:17.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 -- 172.21.15.46:0/4252490008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c0c6f60 msgr2=0x7f905c0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:17.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 --2- 172.21.15.46:0/4252490008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c0c6f60 0x7f905c0c0400 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f905c06f1b0 tx=0x7f904401c500 comp rx=0 tx=0).stop 2023-10-30T13:41:17.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 -- 172.21.15.46:0/4252490008 shutdown_connections 2023-10-30T13:41:17.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 --2- 172.21.15.46:0/4252490008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c0c6f60 0x7f905c0c0400 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 --2- 172.21.15.46:0/4252490008 >> v2:172.21.15.154:3300/0 conn(0x7f905c0c6b80 0x7f905c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 -- 172.21.15.46:0/4252490008 >> 172.21.15.46:0/4252490008 conn(0x7f905c0c7910 msgr2=0x7f905c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:17.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 -- 172.21.15.46:0/4252490008 shutdown_connections 2023-10-30T13:41:17.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 -- 172.21.15.46:0/4252490008 wait complete. 2023-10-30T13:41:17.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 Processor -- start 2023-10-30T13:41:17.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 -- start start 2023-10-30T13:41:17.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f905c0c6b80 0x7f905c154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:17.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c0c6f60 0x7f905c139e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:17.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f905c0c8430 0x7f905c0c2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:17.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905c0c3980 con 0x7f905c0c6f60 2023-10-30T13:41:17.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905c1f19d0 con 0x7f905c0c8430 2023-10-30T13:41:17.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9061809640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905c0c4590 con 0x7f905c0c6b80 2023-10-30T13:41:17.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9053fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c0c6f60 0x7f905c139e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:17.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9053fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c0c6f60 0x7f905c139e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50026/0 (socket says 172.21.15.46:50026) 2023-10-30T13:41:17.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f9053fff640 1 -- 172.21.15.46:0/1151726139 learned_addr learned my addr 172.21.15.46:0/1151726139 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:17.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f905affd640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f905c0c6b80 0x7f905c154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:17.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.314+0000 7f905b7fe640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f905c0c8430 0x7f905c0c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:17.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9053fff640 1 -- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f905c0c6b80 msgr2=0x7f905c154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:17.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9053fff640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f905c0c6b80 0x7f905c154480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9053fff640 1 -- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f905c0c8430 msgr2=0x7f905c0c2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:17.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9053fff640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f905c0c8430 0x7f905c0c2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9053fff640 1 -- 172.21.15.46:0/1151726139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90440094d0 con 0x7f905c0c6f60 2023-10-30T13:41:17.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9053fff640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c0c6f60 0x7f905c139e30 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9044019110 tx=0x7f904401ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:17.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9058ff9640 1 -- 172.21.15.46:0/1151726139 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90440b4800 con 0x7f905c0c6f60 2023-10-30T13:41:17.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9058ff9640 1 -- 172.21.15.46:0/1151726139 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90440b4960 con 0x7f905c0c6f60 2023-10-30T13:41:17.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9061809640 1 -- 172.21.15.46:0/1151726139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905c0c4870 con 0x7f905c0c6f60 2023-10-30T13:41:17.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9058ff9640 1 -- 172.21.15.46:0/1151726139 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90440b4d10 con 0x7f905c0c6f60 2023-10-30T13:41:17.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9061809640 1 -- 172.21.15.46:0/1151726139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f905c1f1f70 con 0x7f905c0c6f60 2023-10-30T13:41:17.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9061809640 1 -- 172.21.15.46:0/1151726139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f905c04e170 con 0x7f905c0c6f60 2023-10-30T13:41:17.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9058ff9640 1 -- 172.21.15.46:0/1151726139 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f90440b2410 con 0x7f905c0c6f60 2023-10-30T13:41:17.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9058ff9640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f903c0664d0 0x7f903c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:17.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f9058ff9640 1 -- 172.21.15.46:0/1151726139 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f904411eeb0 con 0x7f905c0c6f60 2023-10-30T13:41:17.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.318+0000 7f905affd640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f903c0664d0 0x7f903c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:17.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.322+0000 7f905affd640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f903c0664d0 0x7f903c068990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f904c00b9b0 tx=0x7f904c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:17.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.322+0000 7f9058ff9640 1 -- 172.21.15.46:0/1151726139 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f90440ebea0 con 0x7f905c0c6f60 2023-10-30T13:41:17.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 -- 172.21.15.46:0/1151726139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f905c0c60f0 con 0x7f905c0c6f60 2023-10-30T13:41:17.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9058ff9640 1 -- 172.21.15.46:0/1151726139 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f90440ef4c0 con 0x7f905c0c6f60 2023-10-30T13:41:17.518 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:17.518 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:14.000593+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:17.519 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 90 2023-10-30T13:41:17.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 -- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f903c0664d0 msgr2=0x7f903c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:17.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f903c0664d0 0x7f903c068990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f904c00b9b0 tx=0x7f904c008040 comp rx=0 tx=0).stop 2023-10-30T13:41:17.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 -- 172.21.15.46:0/1151726139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c0c6f60 msgr2=0x7f905c139e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:17.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c0c6f60 0x7f905c139e30 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9044019110 tx=0x7f904401ad40 comp rx=0 tx=0).stop 2023-10-30T13:41:17.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 -- 172.21.15.46:0/1151726139 shutdown_connections 2023-10-30T13:41:17.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f903c0664d0 0x7f903c068990 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f905c0c8430 0x7f905c0c2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c0c6f60 0x7f905c139e30 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 --2- 172.21.15.46:0/1151726139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f905c0c6b80 0x7f905c154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:17.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 -- 172.21.15.46:0/1151726139 >> 172.21.15.46:0/1151726139 conn(0x7f905c0c7910 msgr2=0x7f905c0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:17.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 -- 172.21.15.46:0/1151726139 shutdown_connections 2023-10-30T13:41:17.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:17.514+0000 7f9061809640 1 -- 172.21.15.46:0/1151726139 wait complete. 2023-10-30T13:41:17.539 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:18.479 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:41:18.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:41:19.541 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:19.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e39d3640 1 Processor -- start 2023-10-30T13:41:19.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e39d3640 1 -- start start 2023-10-30T13:41:19.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e39d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62dc15d6f0 0x7f62dc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:19.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e39d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62dc0cf160 0x7f62dc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:19.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e39d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62dc0cb800 con 0x7f62dc1538a0 2023-10-30T13:41:19.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e39d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62dc0cb330 con 0x7f62dc15d6f0 2023-10-30T13:41:19.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e39d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc0cad70 con 0x7f62dc0cf160 2023-10-30T13:41:19.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e0f6e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62dc15d6f0 0x7f62dc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:19.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e0f6e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62dc15d6f0 0x7f62dc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50032/0 (socket says 172.21.15.46:50032) 2023-10-30T13:41:19.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e0f6e640 1 -- 172.21.15.46:0/4054388774 learned_addr learned my addr 172.21.15.46:0/4054388774 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e1f70640 1 --2- 172.21.15.46:0/4054388774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62dc0cf160 0x7f62dc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e0f6e640 1 -- 172.21.15.46:0/4054388774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62dc0cf160 msgr2=0x7f62dc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e0f6e640 1 --2- 172.21.15.46:0/4054388774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62dc0cf160 0x7f62dc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:19.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e0f6e640 1 -- 172.21.15.46:0/4054388774 >> v1:172.21.15.46:6789/0 conn(0x7f62dc1538a0 legacy=0x7f62dc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:19.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.686+0000 7f62e0f6e640 1 -- 172.21.15.46:0/4054388774 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc0cb050 con 0x7f62dc15d6f0 2023-10-30T13:41:19.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.690+0000 7f62e0f6e640 1 --2- 172.21.15.46:0/4054388774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc15d6f0 0x7f62dc15a090 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009870 tx=0x7f62cc01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6193d3dada25b8fe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:19.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.690+0000 7f62d3fff640 1 -- 172.21.15.46:0/4054388774 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc0a8a30 con 0x7f62dc15d6f0 2023-10-30T13:41:19.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.690+0000 7f62d3fff640 1 -- 172.21.15.46:0/4054388774 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62cc0a8b90 con 0x7f62dc15d6f0 2023-10-30T13:41:19.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.690+0000 7f62d3fff640 1 -- 172.21.15.46:0/4054388774 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc0b0440 con 0x7f62dc15d6f0 2023-10-30T13:41:19.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.690+0000 7f62e39d3640 1 -- 172.21.15.46:0/4054388774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc15d6f0 msgr2=0x7f62dc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:19.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.690+0000 7f62e39d3640 1 --2- 172.21.15.46:0/4054388774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc15d6f0 0x7f62dc15a090 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009870 tx=0x7f62cc01ed30 comp rx=0 tx=0).stop 2023-10-30T13:41:19.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.690+0000 7f62e39d3640 1 -- 172.21.15.46:0/4054388774 shutdown_connections 2023-10-30T13:41:19.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.690+0000 7f62e39d3640 1 --2- 172.21.15.46:0/4054388774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62dc0cf160 0x7f62dc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:19.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.690+0000 7f62e39d3640 1 --2- 172.21.15.46:0/4054388774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc15d6f0 0x7f62dc15a090 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:19.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.690+0000 7f62e39d3640 1 -- 172.21.15.46:0/4054388774 >> 172.21.15.46:0/4054388774 conn(0x7f62dc06bfe0 msgr2=0x7f62dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:19.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.694+0000 7f62e39d3640 1 -- 172.21.15.46:0/4054388774 shutdown_connections 2023-10-30T13:41:19.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.694+0000 7f62e39d3640 1 -- 172.21.15.46:0/4054388774 wait complete. 2023-10-30T13:41:19.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.694+0000 7f62e39d3640 1 Processor -- start 2023-10-30T13:41:19.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.694+0000 7f62e39d3640 1 -- start start 2023-10-30T13:41:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e39d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0cf160 0x7f62dc159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e39d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62dc1538a0 0x7f62dc0c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e39d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62dc15d6f0 0x7f62dc1e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e39d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc1e3bf0 con 0x7f62dc0cf160 2023-10-30T13:41:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e39d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc1580f0 con 0x7f62dc15d6f0 2023-10-30T13:41:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e39d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc0c59a0 con 0x7f62dc1538a0 2023-10-30T13:41:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e176f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0cf160 0x7f62dc159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e176f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0cf160 0x7f62dc159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50040/0 (socket says 172.21.15.46:50040) 2023-10-30T13:41:19.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e176f640 1 -- 172.21.15.46:0/3561498740 learned_addr learned my addr 172.21.15.46:0/3561498740 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:19.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e0f6e640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62dc1538a0 0x7f62dc0c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:19.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e1f70640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62dc15d6f0 0x7f62dc1e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:19.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e176f640 1 -- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62dc1538a0 msgr2=0x7f62dc0c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:19.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e176f640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62dc1538a0 0x7f62dc0c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:19.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e176f640 1 -- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62dc15d6f0 msgr2=0x7f62dc1e2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:19.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e176f640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62dc15d6f0 0x7f62dc1e2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:19.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e176f640 1 -- 172.21.15.46:0/3561498740 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc0094d0 con 0x7f62dc0cf160 2023-10-30T13:41:19.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e1f70640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62dc15d6f0 0x7f62dc1e2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:19.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e176f640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0cf160 0x7f62dc159300 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f62d800c500 tx=0x7f62d800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62d1ffb640 1 -- 172.21.15.46:0/3561498740 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d8005220 con 0x7f62dc0cf160 2023-10-30T13:41:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62d1ffb640 1 -- 172.21.15.46:0/3561498740 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d8005380 con 0x7f62dc0cf160 2023-10-30T13:41:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62d1ffb640 1 -- 172.21.15.46:0/3561498740 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d8007270 con 0x7f62dc0cf160 2023-10-30T13:41:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e39d3640 1 -- 172.21.15.46:0/3561498740 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62dc1584d0 con 0x7f62dc0cf160 2023-10-30T13:41:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e39d3640 1 -- 172.21.15.46:0/3561498740 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc0c4f60 con 0x7f62dc0cf160 2023-10-30T13:41:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.698+0000 7f62e39d3640 1 -- 172.21.15.46:0/3561498740 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62dc0b2a50 con 0x7f62dc0cf160 2023-10-30T13:41:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.702+0000 7f62d1ffb640 1 -- 172.21.15.46:0/3561498740 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f62d80073d0 con 0x7f62dc0cf160 2023-10-30T13:41:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.702+0000 7f62d1ffb640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62c00664d0 0x7f62c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.702+0000 7f62d1ffb640 1 -- 172.21.15.46:0/3561498740 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f62d8083a60 con 0x7f62dc0cf160 2023-10-30T13:41:19.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.702+0000 7f62e0f6e640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62c00664d0 0x7f62c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:19.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.702+0000 7f62e0f6e640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62c00664d0 0x7f62c0068990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0b2000 tx=0x7f62cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:19.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.706+0000 7f62d1ffb640 1 -- 172.21.15.46:0/3561498740 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f62d8050b00 con 0x7f62dc0cf160 2023-10-30T13:41:19.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.918+0000 7f62e39d3640 1 -- 172.21.15.46:0/3561498740 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62dc04e170 con 0x7f62dc0cf160 2023-10-30T13:41:19.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.918+0000 7f62d1ffb640 1 -- 172.21.15.46:0/3561498740 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f62d8054120 con 0x7f62dc0cf160 2023-10-30T13:41:19.923 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:19.924 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:14.000593+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:19.924 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 90 2023-10-30T13:41:19.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.922+0000 7f62e39d3640 1 -- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62c00664d0 msgr2=0x7f62c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:19.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.922+0000 7f62e39d3640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62c00664d0 0x7f62c0068990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0b2000 tx=0x7f62cc009040 comp rx=0 tx=0).stop 2023-10-30T13:41:19.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.922+0000 7f62e39d3640 1 -- 172.21.15.46:0/3561498740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0cf160 msgr2=0x7f62dc159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:19.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.922+0000 7f62e39d3640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0cf160 0x7f62dc159300 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f62d800c500 tx=0x7f62d800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:41:19.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.922+0000 7f62e39d3640 1 -- 172.21.15.46:0/3561498740 shutdown_connections 2023-10-30T13:41:19.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.922+0000 7f62e39d3640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62c00664d0 0x7f62c0068990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:19.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.922+0000 7f62e39d3640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62dc15d6f0 0x7f62dc1e2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:19.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.922+0000 7f62e39d3640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62dc1538a0 0x7f62dc0c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:19.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.922+0000 7f62e39d3640 1 --2- 172.21.15.46:0/3561498740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62dc0cf160 0x7f62dc159300 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:19.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.922+0000 7f62e39d3640 1 -- 172.21.15.46:0/3561498740 >> 172.21.15.46:0/3561498740 conn(0x7f62dc06bfe0 msgr2=0x7f62dc14ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:19.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.922+0000 7f62e39d3640 1 -- 172.21.15.46:0/3561498740 shutdown_connections 2023-10-30T13:41:19.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:19.922+0000 7f62e39d3640 1 -- 172.21.15.46:0/3561498740 wait complete. 2023-10-30T13:41:19.945 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:21.946 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:22.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.086+0000 7f79f3d3e640 1 Processor -- start 2023-10-30T13:41:22.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.086+0000 7f79f3d3e640 1 -- start start 2023-10-30T13:41:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f3d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79ec1538a0 0x7f79ec15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f3d3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f79ec0cf160 0x7f79ec0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f3d3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ec0cb800 con 0x7f79ec0ca0e0 2023-10-30T13:41:22.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f3d3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79ec15dc30 con 0x7f79ec0cf160 2023-10-30T13:41:22.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f3d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79ec0cb330 con 0x7f79ec1538a0 2023-10-30T13:41:22.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f1ada640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79ec1538a0 0x7f79ec15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:22.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f12d9640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f79ec0ca0e0 0x7f79ec15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:36844/0 (socket says 172.21.15.46:36844) 2023-10-30T13:41:22.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f12d9640 1 -- 172.21.15.46:0/2285716524 learned_addr learned my addr 172.21.15.46:0/2285716524 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:41:22.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f22db640 1 --2- 172.21.15.46:0/2285716524 >> v2:172.21.15.154:3300/0 conn(0x7f79ec0cf160 0x7f79ec0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:22.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f1ada640 1 -- 172.21.15.46:0/2285716524 >> v2:172.21.15.154:3300/0 conn(0x7f79ec0cf160 msgr2=0x7f79ec0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:22.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f1ada640 1 --2- 172.21.15.46:0/2285716524 >> v2:172.21.15.154:3300/0 conn(0x7f79ec0cf160 0x7f79ec0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f1ada640 1 -- 172.21.15.46:0/2285716524 >> v1:172.21.15.154:6789/0 conn(0x7f79ec0ca0e0 legacy=0x7f79ec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:22.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f1ada640 1 -- 172.21.15.46:0/2285716524 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ec0cad70 con 0x7f79ec1538a0 2023-10-30T13:41:22.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.090+0000 7f79f0ad8640 1 -- 172.21.15.46:0/2285716524 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811585347 0 0) 0x7f79ec0cb800 con 0x7f79ec0ca0e0 2023-10-30T13:41:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f1ada640 1 --2- 172.21.15.46:0/2285716524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79ec1538a0 0x7f79ec15d6f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f79e80097c0 tx=0x7f79e801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ce6152357550f1f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f0ad8640 1 -- 172.21.15.46:0/2285716524 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e80a8970 con 0x7f79ec1538a0 2023-10-30T13:41:22.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f0ad8640 1 -- 172.21.15.46:0/2285716524 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e80a8ad0 con 0x7f79ec1538a0 2023-10-30T13:41:22.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f0ad8640 1 -- 172.21.15.46:0/2285716524 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e80a8e60 con 0x7f79ec1538a0 2023-10-30T13:41:22.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2285716524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79ec1538a0 msgr2=0x7f79ec15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:22.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f3d3e640 1 --2- 172.21.15.46:0/2285716524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79ec1538a0 0x7f79ec15d6f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f79e80097c0 tx=0x7f79e801ec70 comp rx=0 tx=0).stop 2023-10-30T13:41:22.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2285716524 shutdown_connections 2023-10-30T13:41:22.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f3d3e640 1 --2- 172.21.15.46:0/2285716524 >> v2:172.21.15.154:3300/0 conn(0x7f79ec0cf160 0x7f79ec0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f3d3e640 1 --2- 172.21.15.46:0/2285716524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79ec1538a0 0x7f79ec15d6f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2285716524 >> 172.21.15.46:0/2285716524 conn(0x7f79ec06bfe0 msgr2=0x7f79ec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:22.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2285716524 shutdown_connections 2023-10-30T13:41:22.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2285716524 wait complete. 2023-10-30T13:41:22.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.094+0000 7f79f3d3e640 1 Processor -- start 2023-10-30T13:41:22.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f3d3e640 1 -- start start 2023-10-30T13:41:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f3d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79ec0ca0e0 0x7f79ec0b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f3d3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79ec0cf160 0x7f79ec1f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f3d3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ec1538a0 0x7f79ec0b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f3d3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ec0c82e0 con 0x7f79ec1538a0 2023-10-30T13:41:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f3d3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ec0c69c0 con 0x7f79ec0cf160 2023-10-30T13:41:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f3d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79ec0c6ca0 con 0x7f79ec0ca0e0 2023-10-30T13:41:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f22db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ec1538a0 0x7f79ec0b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f22db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ec1538a0 0x7f79ec0b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50044/0 (socket says 172.21.15.46:50044) 2023-10-30T13:41:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f22db640 1 -- 172.21.15.46:0/2259841616 learned_addr learned my addr 172.21.15.46:0/2259841616 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f1ada640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79ec0ca0e0 0x7f79ec0b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f12d9640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79ec0cf160 0x7f79ec1f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f22db640 1 -- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79ec0ca0e0 msgr2=0x7f79ec0b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f22db640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79ec0ca0e0 0x7f79ec0b3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f22db640 1 -- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79ec0cf160 msgr2=0x7f79ec1f0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:22.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f22db640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79ec0cf160 0x7f79ec1f0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f22db640 1 -- 172.21.15.46:0/2259841616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e8009420 con 0x7f79ec1538a0 2023-10-30T13:41:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f12d9640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79ec0cf160 0x7f79ec1f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f22db640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ec1538a0 0x7f79ec0b4b00 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f79e400efc0 tx=0x7f79e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79e27fc640 1 -- 172.21.15.46:0/2259841616 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4009b10 con 0x7f79ec1538a0 2023-10-30T13:41:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79e27fc640 1 -- 172.21.15.46:0/2259841616 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e400f040 con 0x7f79ec1538a0 2023-10-30T13:41:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79e27fc640 1 -- 172.21.15.46:0/2259841616 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e40146c0 con 0x7f79ec1538a0 2023-10-30T13:41:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2259841616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79ec0b3240 con 0x7f79ec1538a0 2023-10-30T13:41:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2259841616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79ec0b3800 con 0x7f79ec1538a0 2023-10-30T13:41:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.098+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2259841616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79ec0b2a50 con 0x7f79ec1538a0 2023-10-30T13:41:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.102+0000 7f79e27fc640 1 -- 172.21.15.46:0/2259841616 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f79e4014820 con 0x7f79ec1538a0 2023-10-30T13:41:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.102+0000 7f79e27fc640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f79d00664d0 0x7f79d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.102+0000 7f79e27fc640 1 -- 172.21.15.46:0/2259841616 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f79e4084cf0 con 0x7f79ec1538a0 2023-10-30T13:41:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.102+0000 7f79f1ada640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f79d00664d0 0x7f79d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:22.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.102+0000 7f79f1ada640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f79d00664d0 0x7f79d0068990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f79e80b2000 tx=0x7f79e80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:22.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.106+0000 7f79e27fc640 1 -- 172.21.15.46:0/2259841616 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f79e4051d70 con 0x7f79ec1538a0 2023-10-30T13:41:22.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.314+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2259841616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79ec0b3a20 con 0x7f79ec1538a0 2023-10-30T13:41:22.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.314+0000 7f79e27fc640 1 -- 172.21.15.46:0/2259841616 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f79e4018020 con 0x7f79ec1538a0 2023-10-30T13:41:22.317 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:22.318 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:14.000593+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:22.318 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 90 2023-10-30T13:41:22.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.314+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f79d00664d0 msgr2=0x7f79d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:22.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.314+0000 7f79f3d3e640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f79d00664d0 0x7f79d0068990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f79e80b2000 tx=0x7f79e80aa070 comp rx=0 tx=0).stop 2023-10-30T13:41:22.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.314+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2259841616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ec1538a0 msgr2=0x7f79ec0b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:22.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.314+0000 7f79f3d3e640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ec1538a0 0x7f79ec0b4b00 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f79e400efc0 tx=0x7f79e400c490 comp rx=0 tx=0).stop 2023-10-30T13:41:22.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.318+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2259841616 shutdown_connections 2023-10-30T13:41:22.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.318+0000 7f79f3d3e640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f79d00664d0 0x7f79d0068990 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.318+0000 7f79f3d3e640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ec1538a0 0x7f79ec0b4b00 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.318+0000 7f79f3d3e640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79ec0cf160 0x7f79ec1f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.318+0000 7f79f3d3e640 1 --2- 172.21.15.46:0/2259841616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79ec0ca0e0 0x7f79ec0b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:22.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.318+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2259841616 >> 172.21.15.46:0/2259841616 conn(0x7f79ec06bfe0 msgr2=0x7f79ec0cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:22.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.318+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2259841616 shutdown_connections 2023-10-30T13:41:22.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:22.318+0000 7f79f3d3e640 1 -- 172.21.15.46:0/2259841616 wait complete. 2023-10-30T13:41:22.341 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:24.342 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:24.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff617e9e640 1 Processor -- start 2023-10-30T13:41:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff617e9e640 1 -- start start 2023-10-30T13:41:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff617e9e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff610153d70 0x7ff61006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff617e9e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff61006e470 0x7ff6100cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff617e9e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61006afa0 con 0x7ff6100c8490 2023-10-30T13:41:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff617e9e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff61006d780 con 0x7ff61006e470 2023-10-30T13:41:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff617e9e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff610154150 con 0x7ff610153d70 2023-10-30T13:41:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff615439640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff61006e470 0x7ff6100cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff615439640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff61006e470 0x7ff6100cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50048/0 (socket says 172.21.15.46:50048) 2023-10-30T13:41:24.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff615439640 1 -- 172.21.15.46:0/1125955175 learned_addr learned my addr 172.21.15.46:0/1125955175 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:24.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff615c3a640 1 --2- 172.21.15.46:0/1125955175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff610153d70 0x7ff61006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:24.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff615439640 1 -- 172.21.15.46:0/1125955175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff610153d70 msgr2=0x7ff61006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:24.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff615439640 1 --2- 172.21.15.46:0/1125955175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff610153d70 0x7ff61006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:24.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff615439640 1 -- 172.21.15.46:0/1125955175 >> v1:172.21.15.154:6789/0 conn(0x7ff6100c8490 legacy=0x7ff6100c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:24.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.494+0000 7ff615439640 1 -- 172.21.15.46:0/1125955175 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61006e030 con 0x7ff61006e470 2023-10-30T13:41:24.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.498+0000 7ff615439640 1 --2- 172.21.15.46:0/1125955175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff61006e470 0x7ff6100cce00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7ff600009870 tx=0x7ff60001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f0d01274165466 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:24.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.498+0000 7ff614c38640 1 -- 172.21.15.46:0/1125955175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6000aa670 con 0x7ff61006e470 2023-10-30T13:41:24.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.498+0000 7ff614c38640 1 -- 172.21.15.46:0/1125955175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6000aa7d0 con 0x7ff61006e470 2023-10-30T13:41:24.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.498+0000 7ff614c38640 1 -- 172.21.15.46:0/1125955175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6000aab60 con 0x7ff61006e470 2023-10-30T13:41:24.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.498+0000 7ff617e9e640 1 -- 172.21.15.46:0/1125955175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff61006e470 msgr2=0x7ff6100cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:24.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.498+0000 7ff617e9e640 1 --2- 172.21.15.46:0/1125955175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff61006e470 0x7ff6100cce00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7ff600009870 tx=0x7ff60001c510 comp rx=0 tx=0).stop 2023-10-30T13:41:24.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.498+0000 7ff617e9e640 1 -- 172.21.15.46:0/1125955175 shutdown_connections 2023-10-30T13:41:24.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.498+0000 7ff617e9e640 1 --2- 172.21.15.46:0/1125955175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff61006e470 0x7ff6100cce00 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:24.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.498+0000 7ff617e9e640 1 --2- 172.21.15.46:0/1125955175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff610153d70 0x7ff61006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:24.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.498+0000 7ff617e9e640 1 -- 172.21.15.46:0/1125955175 >> 172.21.15.46:0/1125955175 conn(0x7ff610071ea0 msgr2=0x7ff610065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:24.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.498+0000 7ff617e9e640 1 -- 172.21.15.46:0/1125955175 shutdown_connections 2023-10-30T13:41:24.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.498+0000 7ff617e9e640 1 -- 172.21.15.46:0/1125955175 wait complete. 2023-10-30T13:41:24.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff617e9e640 1 Processor -- start 2023-10-30T13:41:24.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff617e9e640 1 -- start start 2023-10-30T13:41:24.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff617e9e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff61006e470 0x7ff610154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:24.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff617e9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6100c8490 0x7ff610155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:24.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff617e9e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff610153d70 0x7ff6101f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff617e9e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6100b0210 con 0x7ff6100c8490 2023-10-30T13:41:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff617e9e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6101542f0 con 0x7ff61006e470 2023-10-30T13:41:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff617e9e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff610154870 con 0x7ff610153d70 2023-10-30T13:41:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff615439640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6100c8490 0x7ff610155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff615439640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6100c8490 0x7ff610155240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50058/0 (socket says 172.21.15.46:50058) 2023-10-30T13:41:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff615439640 1 -- 172.21.15.46:0/2731232801 learned_addr learned my addr 172.21.15.46:0/2731232801 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff615c3a640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff61006e470 0x7ff610154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff61643b640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff610153d70 0x7ff6101f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:24.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff615439640 1 -- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff610153d70 msgr2=0x7ff6101f60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:24.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff615439640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff610153d70 0x7ff6101f60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:24.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff615439640 1 -- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff61006e470 msgr2=0x7ff610154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:24.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff615439640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff61006e470 0x7ff610154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:24.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff615439640 1 -- 172.21.15.46:0/2731232801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6000094d0 con 0x7ff6100c8490 2023-10-30T13:41:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff615439640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6100c8490 0x7ff610155240 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff6000ac040 tx=0x7ff60001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff6067fc640 1 -- 172.21.15.46:0/2731232801 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6000b3800 con 0x7ff6100c8490 2023-10-30T13:41:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff6067fc640 1 -- 172.21.15.46:0/2731232801 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6000b3980 con 0x7ff6100c8490 2023-10-30T13:41:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff6067fc640 1 -- 172.21.15.46:0/2731232801 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6000b3d30 con 0x7ff6100c8490 2023-10-30T13:41:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff617e9e640 1 -- 172.21.15.46:0/2731232801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6100b17c0 con 0x7ff6100c8490 2023-10-30T13:41:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff617e9e640 1 -- 172.21.15.46:0/2731232801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff610156f90 con 0x7ff6100c8490 2023-10-30T13:41:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.502+0000 7ff617e9e640 1 -- 172.21.15.46:0/2731232801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff610157120 con 0x7ff6100c8490 2023-10-30T13:41:24.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.506+0000 7ff6067fc640 1 -- 172.21.15.46:0/2731232801 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff6000b2380 con 0x7ff6100c8490 2023-10-30T13:41:24.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.506+0000 7ff6067fc640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5f40664a0 0x7ff5f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:24.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.506+0000 7ff6067fc640 1 -- 172.21.15.46:0/2731232801 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7ff60011eb50 con 0x7ff6100c8490 2023-10-30T13:41:24.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.506+0000 7ff615c3a640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5f40664a0 0x7ff5f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:24.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.506+0000 7ff615c3a640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5f40664a0 0x7ff5f4068960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0079c0 tx=0x7ff60c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:24.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.510+0000 7ff6067fc640 1 -- 172.21.15.46:0/2731232801 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6000ebbf0 con 0x7ff6100c8490 2023-10-30T13:41:24.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 -- 172.21.15.46:0/2731232801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff61004e170 con 0x7ff6100c8490 2023-10-30T13:41:24.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff6067fc640 1 -- 172.21.15.46:0/2731232801 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5744 (secure 0 0 0) 0x7ff6000c3090 con 0x7ff6100c8490 2023-10-30T13:41:24.721 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:24.721 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:22.762462+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:resolve","state_seq":27,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:24.722 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 91 2023-10-30T13:41:24.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 -- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5f40664a0 msgr2=0x7ff5f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:24.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5f40664a0 0x7ff5f4068960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0079c0 tx=0x7ff60c008040 comp rx=0 tx=0).stop 2023-10-30T13:41:24.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 -- 172.21.15.46:0/2731232801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6100c8490 msgr2=0x7ff610155240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:24.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6100c8490 0x7ff610155240 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff6000ac040 tx=0x7ff60001ad50 comp rx=0 tx=0).stop 2023-10-30T13:41:24.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 -- 172.21.15.46:0/2731232801 shutdown_connections 2023-10-30T13:41:24.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5f40664a0 0x7ff5f4068960 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:24.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff610153d70 0x7ff6101f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:24.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6100c8490 0x7ff610155240 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:24.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 --2- 172.21.15.46:0/2731232801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff61006e470 0x7ff610154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:24.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 -- 172.21.15.46:0/2731232801 >> 172.21.15.46:0/2731232801 conn(0x7ff610071ea0 msgr2=0x7ff6100c7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:24.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 -- 172.21.15.46:0/2731232801 shutdown_connections 2023-10-30T13:41:24.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:24.718+0000 7ff617e9e640 1 -- 172.21.15.46:0/2731232801 wait complete. 2023-10-30T13:41:24.742 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:26.743 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:26.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.894+0000 7f2ed52c9640 1 Processor -- start 2023-10-30T13:41:26.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.894+0000 7f2ed52c9640 1 -- start start 2023-10-30T13:41:26.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ed52c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed015d6f0 0x7f2ed015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:26.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ed52c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed00cb800 con 0x7f2ed01538a0 2023-10-30T13:41:26.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ed52c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed00cb330 con 0x7f2ed00cf160 2023-10-30T13:41:26.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ed52c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed00cad70 con 0x7f2ed015d6f0 2023-10-30T13:41:26.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2eceffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2ed01538a0 0x7f2ed00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60188/0 (socket says 172.21.15.46:60188) 2023-10-30T13:41:26.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2eceffd640 1 -- 172.21.15.46:0/2442672291 learned_addr learned my addr 172.21.15.46:0/2442672291 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:41:26.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ec7fff640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed015d6f0 0x7f2ed015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:26.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410435004 0 0) 0x7f2ed00cb800 con 0x7f2ed01538a0 2023-10-30T13:41:26.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ebc003530 con 0x7f2ed01538a0 2023-10-30T13:41:26.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419227732 0 0) 0x7f2ed00cb330 con 0x7f2ed00cf160 2023-10-30T13:41:26.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed00cb800 con 0x7f2ed00cf160 2023-10-30T13:41:26.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 872969182 0 0) 0x7f2ebc003530 con 0x7f2ed01538a0 2023-10-30T13:41:26.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed015d6f0 msgr2=0x7f2ed015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:26.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed015d6f0 0x7f2ed015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:26.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 >> v1:172.21.15.154:6789/0 conn(0x7f2ed00cf160 legacy=0x7f2ed0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:26.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed00cb050 con 0x7f2ed01538a0 2023-10-30T13:41:26.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f2ec0001680 con 0x7f2ed01538a0 2023-10-30T13:41:26.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 >> v1:172.21.15.46:6789/0 conn(0x7f2ed01538a0 legacy=0x7f2ed00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:26.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed015d6f0 0x7f2ebc003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:26.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ebc0040a0 0x7f2ebc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:26.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ebc004d10 0x7f2ebc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:26.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed00cad70 con 0x7f2ebc0040a0 2023-10-30T13:41:26.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc00ba70 con 0x7f2ebc004d10 2023-10-30T13:41:26.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc00bc30 con 0x7f2ed015d6f0 2023-10-30T13:41:26.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ec7fff640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ebc0040a0 0x7f2ebc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:26.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2eceffd640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed015d6f0 0x7f2ebc003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:26.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ecf7fe640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ebc004d10 0x7f2ebc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:26.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ec7fff640 1 -- 172.21.15.46:0/2442672291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed015d6f0 msgr2=0x7f2ebc003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:26.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ec7fff640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed015d6f0 0x7f2ebc003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:26.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ec7fff640 1 -- 172.21.15.46:0/2442672291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ebc004d10 msgr2=0x7f2ebc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:26.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ec7fff640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ebc004d10 0x7f2ebc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:26.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ec7fff640 1 -- 172.21.15.46:0/2442672291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc003530 con 0x7f2ebc0040a0 2023-10-30T13:41:26.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.898+0000 7f2ecf7fe640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ebc004d10 0x7f2ebc005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:26.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ec7fff640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ebc0040a0 0x7f2ebc0047d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009ae0 tx=0x7f2eb801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3fc868ba019d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:26.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0007ea0 con 0x7f2ebc0040a0 2023-10-30T13:41:26.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb80a5530 con 0x7f2ebc0040a0 2023-10-30T13:41:26.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ece7fc640 1 -- 172.21.15.46:0/2442672291 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb80a58e0 con 0x7f2ebc0040a0 2023-10-30T13:41:26.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ed52c9640 1 -- 172.21.15.46:0/2442672291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ebc0040a0 msgr2=0x7f2ebc0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ed52c9640 1 --2- 172.21.15.46:0/2442672291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ebc0040a0 0x7f2ebc0047d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009ae0 tx=0x7f2eb801ca50 comp rx=0 tx=0).stop 2023-10-30T13:41:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ecf7fe640 1 -- 172.21.15.46:0/2442672291 reap_dead start 2023-10-30T13:41:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ed52c9640 1 -- 172.21.15.46:0/2442672291 shutdown_connections 2023-10-30T13:41:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ed52c9640 1 -- 172.21.15.46:0/2442672291 >> 172.21.15.46:0/2442672291 conn(0x7f2ed006bfe0 msgr2=0x7f2ed006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ed52c9640 1 -- 172.21.15.46:0/2442672291 shutdown_connections 2023-10-30T13:41:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ed52c9640 1 -- 172.21.15.46:0/2442672291 wait complete. 2023-10-30T13:41:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ed52c9640 1 Processor -- start 2023-10-30T13:41:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.902+0000 7f2ed52c9640 1 -- start start 2023-10-30T13:41:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ed52c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ebc0040a0 0x7f2ed01f2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ed52c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ed014d690 0x7f2ed00cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ed52c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed00b4e70 0x7f2ed00cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ed52c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed014f9e0 con 0x7f2ebc0040a0 2023-10-30T13:41:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ed52c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed00cd3b0 con 0x7f2ed014d690 2023-10-30T13:41:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ed52c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed01f3510 con 0x7f2ed00b4e70 2023-10-30T13:41:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2eceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ebc0040a0 0x7f2ed01f2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:26.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2eceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ebc0040a0 0x7f2ed01f2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59128/0 (socket says 172.21.15.46:59128) 2023-10-30T13:41:26.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2eceffd640 1 -- 172.21.15.46:0/265976873 learned_addr learned my addr 172.21.15.46:0/265976873 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:26.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ec7fff640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ed014d690 0x7f2ed00cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:26.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ecf7fe640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed00b4e70 0x7f2ed00cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:26.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2eceffd640 1 -- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed00b4e70 msgr2=0x7f2ed00cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:26.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2eceffd640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed00b4e70 0x7f2ed00cdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:26.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2eceffd640 1 -- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ed014d690 msgr2=0x7f2ed00cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:26.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2eceffd640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ed014d690 0x7f2ed00cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:26.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2eceffd640 1 -- 172.21.15.46:0/265976873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed014eb30 con 0x7f2ebc0040a0 2023-10-30T13:41:26.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2eceffd640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ebc0040a0 0x7f2ed01f2bb0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0012570 tx=0x7f2ec0012a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:26.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ec77fe640 1 -- 172.21.15.46:0/265976873 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0014810 con 0x7f2ebc0040a0 2023-10-30T13:41:26.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ec77fe640 1 -- 172.21.15.46:0/265976873 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec00149b0 con 0x7f2ebc0040a0 2023-10-30T13:41:26.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ec77fe640 1 -- 172.21.15.46:0/265976873 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0014d60 con 0x7f2ebc0040a0 2023-10-30T13:41:26.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ed52c9640 1 -- 172.21.15.46:0/265976873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed014ee10 con 0x7f2ebc0040a0 2023-10-30T13:41:26.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ed52c9640 1 -- 172.21.15.46:0/265976873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed0150b30 con 0x7f2ebc0040a0 2023-10-30T13:41:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ed52c9640 1 -- 172.21.15.46:0/265976873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed014f2c0 con 0x7f2ebc0040a0 2023-10-30T13:41:26.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.906+0000 7f2ec77fe640 1 -- 172.21.15.46:0/265976873 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ec00153e0 con 0x7f2ebc0040a0 2023-10-30T13:41:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.910+0000 7f2ec77fe640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2eb00664d0 0x7f2eb0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.910+0000 7f2ec77fe640 1 -- 172.21.15.46:0/265976873 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f2ec0086bf0 con 0x7f2ebc0040a0 2023-10-30T13:41:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.910+0000 7f2ec7fff640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2eb00664d0 0x7f2eb0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.910+0000 7f2ec7fff640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2eb00664d0 0x7f2eb0068990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80af000 tx=0x7f2eb80046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:26.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:26.914+0000 7f2ec77fe640 1 -- 172.21.15.46:0/265976873 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ed014f2c0 con 0x7f2ebc0040a0 2023-10-30T13:41:27.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.106+0000 7f2ed52c9640 1 -- 172.21.15.46:0/265976873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ed014f2c0 con 0x7f2ebc0040a0 2023-10-30T13:41:27.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.106+0000 7f2ec77fe640 1 -- 172.21.15.46:0/265976873 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f2ed014f2c0 con 0x7f2ebc0040a0 2023-10-30T13:41:27.111 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:27.111 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:22.762462+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:resolve","state_seq":27,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:27.111 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 91 2023-10-30T13:41:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.110+0000 7f2ed52c9640 1 -- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2eb00664d0 msgr2=0x7f2eb0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.110+0000 7f2ed52c9640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2eb00664d0 0x7f2eb0068990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80af000 tx=0x7f2eb80046d0 comp rx=0 tx=0).stop 2023-10-30T13:41:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.110+0000 7f2ed52c9640 1 -- 172.21.15.46:0/265976873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ebc0040a0 msgr2=0x7f2ed01f2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.110+0000 7f2ed52c9640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ebc0040a0 0x7f2ed01f2bb0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0012570 tx=0x7f2ec0012a30 comp rx=0 tx=0).stop 2023-10-30T13:41:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.110+0000 7f2ed52c9640 1 -- 172.21.15.46:0/265976873 shutdown_connections 2023-10-30T13:41:27.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.110+0000 7f2ed52c9640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2eb00664d0 0x7f2eb0068990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:27.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.110+0000 7f2ed52c9640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ed00b4e70 0x7f2ed00cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:27.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.110+0000 7f2ed52c9640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ed014d690 0x7f2ed00cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:27.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.110+0000 7f2ed52c9640 1 --2- 172.21.15.46:0/265976873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ebc0040a0 0x7f2ed01f2bb0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:27.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.110+0000 7f2ed52c9640 1 -- 172.21.15.46:0/265976873 >> 172.21.15.46:0/265976873 conn(0x7f2ed006bfe0 msgr2=0x7f2ed0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:27.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.110+0000 7f2ed52c9640 1 -- 172.21.15.46:0/265976873 shutdown_connections 2023-10-30T13:41:27.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:27.110+0000 7f2ed52c9640 1 -- 172.21.15.46:0/265976873 wait complete. 2023-10-30T13:41:27.133 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 91, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 28 }, "gid_6036": { "addr": "172.21.15.46:6836/3037060424", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3037060424, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3037060424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6036, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 37 }, "gid_6201": { "addr": "172.21.15.46:6837/2546436963", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2546436963, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2546436963, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6201, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:resolve", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:41:22.762462+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6201, "mds_1": 6036, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:41:29.134 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.290+0000 7f2d2553e640 1 Processor -- start 2023-10-30T13:41:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.290+0000 7f2d2553e640 1 -- start start 2023-10-30T13:41:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.290+0000 7f2d2553e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d2015d6f0 0x7f2d2015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.290+0000 7f2d2553e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d200cf160 0x7f2d200cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.290+0000 7f2d2553e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d200cb800 con 0x7f2d201538a0 2023-10-30T13:41:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.290+0000 7f2d2553e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d200cb330 con 0x7f2d200cf160 2023-10-30T13:41:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.290+0000 7f2d2553e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d200cad70 con 0x7f2d2015d6f0 2023-10-30T13:41:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.290+0000 7f2d1f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d200cf160 0x7f2d200cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.294+0000 7f2d1f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d200cf160 0x7f2d200cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59142/0 (socket says 172.21.15.46:59142) 2023-10-30T13:41:29.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.294+0000 7f2d1f7fe640 1 -- 172.21.15.46:0/4144090100 learned_addr learned my addr 172.21.15.46:0/4144090100 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:29.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.294+0000 7f2d17fff640 1 --2- 172.21.15.46:0/4144090100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d2015d6f0 0x7f2d2015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:29.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.294+0000 7f2d1f7fe640 1 -- 172.21.15.46:0/4144090100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d2015d6f0 msgr2=0x7f2d2015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:29.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.294+0000 7f2d1f7fe640 1 --2- 172.21.15.46:0/4144090100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d2015d6f0 0x7f2d2015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.294+0000 7f2d1f7fe640 1 -- 172.21.15.46:0/4144090100 >> v1:172.21.15.154:6789/0 conn(0x7f2d201538a0 legacy=0x7f2d200ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:29.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.294+0000 7f2d1f7fe640 1 -- 172.21.15.46:0/4144090100 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d200cb050 con 0x7f2d200cf160 2023-10-30T13:41:29.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.294+0000 7f2d1f7fe640 1 --2- 172.21.15.46:0/4144090100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d200cf160 0x7f2d200cf590 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009e50 tx=0x7f2d1801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc5b870088a69a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.294+0000 7f2d1e7fc640 1 -- 172.21.15.46:0/4144090100 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d180af400 con 0x7f2d200cf160 2023-10-30T13:41:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.294+0000 7f2d1e7fc640 1 -- 172.21.15.46:0/4144090100 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d180ab050 con 0x7f2d200cf160 2023-10-30T13:41:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.294+0000 7f2d1e7fc640 1 -- 172.21.15.46:0/4144090100 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d180af780 con 0x7f2d200cf160 2023-10-30T13:41:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 -- 172.21.15.46:0/4144090100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d200cf160 msgr2=0x7f2d200cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:29.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 --2- 172.21.15.46:0/4144090100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d200cf160 0x7f2d200cf590 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009e50 tx=0x7f2d1801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 -- 172.21.15.46:0/4144090100 shutdown_connections 2023-10-30T13:41:29.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 --2- 172.21.15.46:0/4144090100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d200cf160 0x7f2d200cf590 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 --2- 172.21.15.46:0/4144090100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d2015d6f0 0x7f2d2015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 -- 172.21.15.46:0/4144090100 >> 172.21.15.46:0/4144090100 conn(0x7f2d2006bfe0 msgr2=0x7f2d2006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:29.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 -- 172.21.15.46:0/4144090100 shutdown_connections 2023-10-30T13:41:29.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 -- 172.21.15.46:0/4144090100 wait complete. 2023-10-30T13:41:29.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 Processor -- start 2023-10-30T13:41:29.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 -- start start 2023-10-30T13:41:29.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d200cf160 0x7f2d201e49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:29.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d201538a0 0x7f2d200b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:29.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d2015d6f0 0x7f2d201f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:29.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d201e3180 con 0x7f2d201538a0 2023-10-30T13:41:29.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d201e2a10 con 0x7f2d200cf160 2023-10-30T13:41:29.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d2553e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d201e2cf0 con 0x7f2d2015d6f0 2023-10-30T13:41:29.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d201538a0 0x7f2d200b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:29.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d201538a0 0x7f2d200b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59156/0 (socket says 172.21.15.46:59156) 2023-10-30T13:41:29.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d17fff640 1 -- 172.21.15.46:0/3521538804 learned_addr learned my addr 172.21.15.46:0/3521538804 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:29.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d1effd640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d200cf160 0x7f2d201e49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:29.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d1f7fe640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d2015d6f0 0x7f2d201f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:29.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d17fff640 1 -- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d2015d6f0 msgr2=0x7f2d201f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:29.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d17fff640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d2015d6f0 0x7f2d201f2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d17fff640 1 -- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d200cf160 msgr2=0x7f2d201e49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:29.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d17fff640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d200cf160 0x7f2d201e49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d17fff640 1 -- 172.21.15.46:0/3521538804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18009b30 con 0x7f2d201538a0 2023-10-30T13:41:29.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.298+0000 7f2d1effd640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d200cf160 0x7f2d201e49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:29.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.302+0000 7f2d17fff640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d201538a0 0x7f2d200b31a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800e870 tx=0x7f2d0800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:29.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.302+0000 7f2d177fe640 1 -- 172.21.15.46:0/3521538804 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d08009510 con 0x7f2d201538a0 2023-10-30T13:41:29.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.302+0000 7f2d177fe640 1 -- 172.21.15.46:0/3521538804 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0800eec0 con 0x7f2d201538a0 2023-10-30T13:41:29.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.302+0000 7f2d177fe640 1 -- 172.21.15.46:0/3521538804 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0800cb40 con 0x7f2d201538a0 2023-10-30T13:41:29.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.302+0000 7f2d2553e640 1 -- 172.21.15.46:0/3521538804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d200b4a90 con 0x7f2d201538a0 2023-10-30T13:41:29.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.302+0000 7f2d2553e640 1 -- 172.21.15.46:0/3521538804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d200b4d30 con 0x7f2d201538a0 2023-10-30T13:41:29.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.302+0000 7f2d2553e640 1 -- 172.21.15.46:0/3521538804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d200b2a50 con 0x7f2d201538a0 2023-10-30T13:41:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.302+0000 7f2d177fe640 1 -- 172.21.15.46:0/3521538804 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2d0800cca0 con 0x7f2d201538a0 2023-10-30T13:41:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.302+0000 7f2d177fe640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2d000664a0 0x7f2d00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.302+0000 7f2d177fe640 1 -- 172.21.15.46:0/3521538804 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f2d080847e0 con 0x7f2d201538a0 2023-10-30T13:41:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.302+0000 7f2d1effd640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2d000664a0 0x7f2d00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:29.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.306+0000 7f2d1effd640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2d000664a0 0x7f2d00068960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2d10000f70 tx=0x7f2d10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:29.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.306+0000 7f2d177fe640 1 -- 172.21.15.46:0/3521538804 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2d08051740 con 0x7f2d201538a0 2023-10-30T13:41:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.498+0000 7f2d2553e640 1 -- 172.21.15.46:0/3521538804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d201e2e60 con 0x7f2d201538a0 2023-10-30T13:41:29.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.498+0000 7f2d177fe640 1 -- 172.21.15.46:0/3521538804 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f2d08054d60 con 0x7f2d201538a0 2023-10-30T13:41:29.503 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:29.504 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:28.201967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:29.504 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 92 2023-10-30T13:41:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.502+0000 7f2d2553e640 1 -- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2d000664a0 msgr2=0x7f2d00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.502+0000 7f2d2553e640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2d000664a0 0x7f2d00068960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2d10000f70 tx=0x7f2d10009040 comp rx=0 tx=0).stop 2023-10-30T13:41:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.502+0000 7f2d2553e640 1 -- 172.21.15.46:0/3521538804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d201538a0 msgr2=0x7f2d200b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.502+0000 7f2d2553e640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d201538a0 0x7f2d200b31a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800e870 tx=0x7f2d0800ec70 comp rx=0 tx=0).stop 2023-10-30T13:41:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.502+0000 7f2d2553e640 1 -- 172.21.15.46:0/3521538804 shutdown_connections 2023-10-30T13:41:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.502+0000 7f2d2553e640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2d000664a0 0x7f2d00068960 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.502+0000 7f2d2553e640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d2015d6f0 0x7f2d201f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.502+0000 7f2d2553e640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d201538a0 0x7f2d200b31a0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.502+0000 7f2d2553e640 1 --2- 172.21.15.46:0/3521538804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d200cf160 0x7f2d201e49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:29.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.502+0000 7f2d2553e640 1 -- 172.21.15.46:0/3521538804 >> 172.21.15.46:0/3521538804 conn(0x7f2d2006bfe0 msgr2=0x7f2d20151cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:29.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.502+0000 7f2d2553e640 1 -- 172.21.15.46:0/3521538804 shutdown_connections 2023-10-30T13:41:29.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:29.502+0000 7f2d2553e640 1 -- 172.21.15.46:0/3521538804 wait complete. 2023-10-30T13:41:29.524 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:31.525 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:31.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.674+0000 7f9684d3e640 1 Processor -- start 2023-10-30T13:41:31.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.674+0000 7f9684d3e640 1 -- start start 2023-10-30T13:41:31.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f9684d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9680069e90 0x7f968006a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:31.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f9684d3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9680153d80 con 0x7f9680070a00 2023-10-30T13:41:31.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f9684d3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96800c5c50 con 0x7f96801539a0 2023-10-30T13:41:31.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f9684d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968006fbb0 con 0x7f9680069e90 2023-10-30T13:41:31.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f9677fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9680070a00 0x7f9680070e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60198/0 (socket says 172.21.15.46:60198) 2023-10-30T13:41:31.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f9677fff640 1 -- 172.21.15.46:0/2905234141 learned_addr learned my addr 172.21.15.46:0/2905234141 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:41:31.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f967ed9d640 1 --2- 172.21.15.46:0/2905234141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9680069e90 0x7f968006a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:31.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f967dd9b640 1 -- 172.21.15.46:0/2905234141 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565627568 0 0) 0x7f9680153d80 con 0x7f9680070a00 2023-10-30T13:41:31.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f967dd9b640 1 -- 172.21.15.46:0/2905234141 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f966c003530 con 0x7f9680070a00 2023-10-30T13:41:31.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f967dd9b640 1 -- 172.21.15.46:0/2905234141 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101420964 0 0) 0x7f96800c5c50 con 0x7f96801539a0 2023-10-30T13:41:31.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f967dd9b640 1 -- 172.21.15.46:0/2905234141 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9680153d80 con 0x7f96801539a0 2023-10-30T13:41:31.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f967dd9b640 1 -- 172.21.15.46:0/2905234141 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1465049937 0 0) 0x7f966c003530 con 0x7f9680070a00 2023-10-30T13:41:31.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f967ed9d640 1 -- 172.21.15.46:0/2905234141 >> v1:172.21.15.154:6789/0 conn(0x7f96801539a0 legacy=0x7f968015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:31.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f967ed9d640 1 -- 172.21.15.46:0/2905234141 >> v1:172.21.15.46:6789/0 conn(0x7f9680070a00 legacy=0x7f9680070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:31.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.678+0000 7f967ed9d640 1 -- 172.21.15.46:0/2905234141 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96800c5f30 con 0x7f9680069e90 2023-10-30T13:41:31.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.682+0000 7f967ed9d640 1 --2- 172.21.15.46:0/2905234141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9680069e90 0x7f968006a340 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9678009dc0 tx=0x7f967801c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba7247b45fd8548 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:31.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.682+0000 7f967dd9b640 1 -- 172.21.15.46:0/2905234141 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9668001680 con 0x7f9680069e90 2023-10-30T13:41:31.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.682+0000 7f967dd9b640 1 -- 172.21.15.46:0/2905234141 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96780a5520 con 0x7f9680069e90 2023-10-30T13:41:31.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.682+0000 7f967dd9b640 1 -- 172.21.15.46:0/2905234141 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96780a58b0 con 0x7f9680069e90 2023-10-30T13:41:31.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.682+0000 7f9684d3e640 1 -- 172.21.15.46:0/2905234141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9680069e90 msgr2=0x7f968006a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:31.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.682+0000 7f9684d3e640 1 --2- 172.21.15.46:0/2905234141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9680069e90 0x7f968006a340 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9678009dc0 tx=0x7f967801c7d0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.682+0000 7f9684d3e640 1 -- 172.21.15.46:0/2905234141 shutdown_connections 2023-10-30T13:41:31.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.682+0000 7f9684d3e640 1 --2- 172.21.15.46:0/2905234141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9680069e90 0x7f968006a340 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.682+0000 7f9684d3e640 1 -- 172.21.15.46:0/2905234141 >> 172.21.15.46:0/2905234141 conn(0x7f96800c7910 msgr2=0x7f96800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:31.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.682+0000 7f9684d3e640 1 -- 172.21.15.46:0/2905234141 shutdown_connections 2023-10-30T13:41:31.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.682+0000 7f9684d3e640 1 -- 172.21.15.46:0/2905234141 wait complete. 2023-10-30T13:41:31.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9684d3e640 1 Processor -- start 2023-10-30T13:41:31.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9684d3e640 1 -- start start 2023-10-30T13:41:31.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9684d3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9680069e90 0x7f9680147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:31.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9684d3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9680070a00 0x7f9680144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:31.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9684d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96801539a0 0x7f96800c2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:31.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9684d3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96800c4e80 con 0x7f9680070a00 2023-10-30T13:41:31.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9684d3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96801f64e0 con 0x7f9680069e90 2023-10-30T13:41:31.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9684d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96801f6a00 con 0x7f96801539a0 2023-10-30T13:41:31.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9677fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9680070a00 0x7f9680144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:31.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9677fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9680070a00 0x7f9680144ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59158/0 (socket says 172.21.15.46:59158) 2023-10-30T13:41:31.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9677fff640 1 -- 172.21.15.46:0/2447904379 learned_addr learned my addr 172.21.15.46:0/2447904379 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:31.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f967ed9d640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96801539a0 0x7f96800c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:31.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f967e59c640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9680069e90 0x7f9680147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:31.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9677fff640 1 -- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96801539a0 msgr2=0x7f96800c2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:31.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9677fff640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96801539a0 0x7f96800c2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9677fff640 1 -- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9680069e90 msgr2=0x7f9680147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:31.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9677fff640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9680069e90 0x7f9680147660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9677fff640 1 -- 172.21.15.46:0/2447904379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9678009a70 con 0x7f9680070a00 2023-10-30T13:41:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f967ed9d640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96801539a0 0x7f96800c2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9677fff640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9680070a00 0x7f9680144ca0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f966800c780 tx=0x7f966800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9676ffd640 1 -- 172.21.15.46:0/2447904379 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9668005cf0 con 0x7f9680070a00 2023-10-30T13:41:31.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9676ffd640 1 -- 172.21.15.46:0/2447904379 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9668005e50 con 0x7f9680070a00 2023-10-30T13:41:31.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9676ffd640 1 -- 172.21.15.46:0/2447904379 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96680072b0 con 0x7f9680070a00 2023-10-30T13:41:31.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9684d3e640 1 -- 172.21.15.46:0/2447904379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96801f6740 con 0x7f9680070a00 2023-10-30T13:41:31.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9684d3e640 1 -- 172.21.15.46:0/2447904379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96800c4830 con 0x7f9680070a00 2023-10-30T13:41:31.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.686+0000 7f9684d3e640 1 -- 172.21.15.46:0/2447904379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96800c4ff0 con 0x7f9680070a00 2023-10-30T13:41:31.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.690+0000 7f9676ffd640 1 -- 172.21.15.46:0/2447904379 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9668007410 con 0x7f9680070a00 2023-10-30T13:41:31.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.690+0000 7f9676ffd640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96600664a0 0x7f9660068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:31.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.690+0000 7f9676ffd640 1 -- 172.21.15.46:0/2447904379 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f9668083b30 con 0x7f9680070a00 2023-10-30T13:41:31.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.690+0000 7f967e59c640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96600664a0 0x7f9660068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:31.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.690+0000 7f967e59c640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96600664a0 0x7f9660068960 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f96700011f0 tx=0x7f9670009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:31.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.694+0000 7f9676ffd640 1 -- 172.21.15.46:0/2447904379 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f96680545b0 con 0x7f9680070a00 2023-10-30T13:41:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.882+0000 7f9684d3e640 1 -- 172.21.15.46:0/2447904379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f968004e170 con 0x7f9680070a00 2023-10-30T13:41:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.882+0000 7f9676ffd640 1 -- 172.21.15.46:0/2447904379 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f9668017020 con 0x7f9680070a00 2023-10-30T13:41:31.887 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:31.887 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:28.201967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:31.889 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 92 2023-10-30T13:41:31.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.886+0000 7f9684d3e640 1 -- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96600664a0 msgr2=0x7f9660068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:31.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.886+0000 7f9684d3e640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96600664a0 0x7f9660068960 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f96700011f0 tx=0x7f9670009040 comp rx=0 tx=0).stop 2023-10-30T13:41:31.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.886+0000 7f9684d3e640 1 -- 172.21.15.46:0/2447904379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9680070a00 msgr2=0x7f9680144ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.886+0000 7f9684d3e640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9680070a00 0x7f9680144ca0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f966800c780 tx=0x7f966800cc40 comp rx=0 tx=0).stop 2023-10-30T13:41:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.886+0000 7f9684d3e640 1 -- 172.21.15.46:0/2447904379 shutdown_connections 2023-10-30T13:41:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.886+0000 7f9684d3e640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f96600664a0 0x7f9660068960 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.886+0000 7f9684d3e640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96801539a0 0x7f96800c2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.886+0000 7f9684d3e640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9680070a00 0x7f9680144ca0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.886+0000 7f9684d3e640 1 --2- 172.21.15.46:0/2447904379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9680069e90 0x7f9680147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.886+0000 7f9684d3e640 1 -- 172.21.15.46:0/2447904379 >> 172.21.15.46:0/2447904379 conn(0x7f96800c7910 msgr2=0x7f9680145f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.886+0000 7f9684d3e640 1 -- 172.21.15.46:0/2447904379 shutdown_connections 2023-10-30T13:41:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:31.886+0000 7f9684d3e640 1 -- 172.21.15.46:0/2447904379 wait complete. 2023-10-30T13:41:31.908 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:33.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:34.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f199ddc5640 1 Processor -- start 2023-10-30T13:41:34.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f199ddc5640 1 -- start start 2023-10-30T13:41:34.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f199ddc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f199815d6f0 0x7f199815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:34.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f199ddc5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f19980cf160 0x7f19980cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:34.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f199ddc5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19980cb800 con 0x7f19981538a0 2023-10-30T13:41:34.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f199ddc5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19980cb330 con 0x7f19980cf160 2023-10-30T13:41:34.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f199ddc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19980cad70 con 0x7f199815d6f0 2023-10-30T13:41:34.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f19977fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f19981538a0 0x7f19980ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60206/0 (socket says 172.21.15.46:60206) 2023-10-30T13:41:34.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f19977fe640 1 -- 172.21.15.46:0/1455675088 learned_addr learned my addr 172.21.15.46:0/1455675088 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:41:34.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f1997fff640 1 --2- 172.21.15.46:0/1455675088 >> v2:172.21.15.46:3300/0 conn(0x7f19980cf160 0x7f19980cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:34.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f1996ffd640 1 --2- 172.21.15.46:0/1455675088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f199815d6f0 0x7f199815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:34.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f1997fff640 1 -- 172.21.15.46:0/1455675088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f199815d6f0 msgr2=0x7f199815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:34.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f1997fff640 1 --2- 172.21.15.46:0/1455675088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f199815d6f0 0x7f199815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:34.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f1997fff640 1 -- 172.21.15.46:0/1455675088 >> v1:172.21.15.46:6789/0 conn(0x7f19981538a0 legacy=0x7f19980ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:34.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.062+0000 7f1997fff640 1 -- 172.21.15.46:0/1455675088 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19980cb050 con 0x7f19980cf160 2023-10-30T13:41:34.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.066+0000 7f1997fff640 1 --2- 172.21.15.46:0/1455675088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19980cf160 0x7f19980cf590 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f1990009d90 tx=0x7f199001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5407aaf7b6b5b07b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:34.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.066+0000 7f19967fc640 1 -- 172.21.15.46:0/1455675088 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19900af600 con 0x7f19980cf160 2023-10-30T13:41:34.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.066+0000 7f19967fc640 1 -- 172.21.15.46:0/1455675088 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19900ab050 con 0x7f19980cf160 2023-10-30T13:41:34.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.066+0000 7f19967fc640 1 -- 172.21.15.46:0/1455675088 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19900af980 con 0x7f19980cf160 2023-10-30T13:41:34.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.066+0000 7f199ddc5640 1 -- 172.21.15.46:0/1455675088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19980cf160 msgr2=0x7f19980cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:34.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.066+0000 7f199ddc5640 1 --2- 172.21.15.46:0/1455675088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19980cf160 0x7f19980cf590 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f1990009d90 tx=0x7f199001f130 comp rx=0 tx=0).stop 2023-10-30T13:41:34.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.066+0000 7f199ddc5640 1 -- 172.21.15.46:0/1455675088 shutdown_connections 2023-10-30T13:41:34.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.066+0000 7f199ddc5640 1 --2- 172.21.15.46:0/1455675088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19980cf160 0x7f19980cf590 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:34.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.066+0000 7f199ddc5640 1 --2- 172.21.15.46:0/1455675088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f199815d6f0 0x7f199815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:34.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.066+0000 7f199ddc5640 1 -- 172.21.15.46:0/1455675088 >> 172.21.15.46:0/1455675088 conn(0x7f199806bfe0 msgr2=0x7f199806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:34.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.066+0000 7f199ddc5640 1 -- 172.21.15.46:0/1455675088 shutdown_connections 2023-10-30T13:41:34.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.066+0000 7f199ddc5640 1 -- 172.21.15.46:0/1455675088 wait complete. 2023-10-30T13:41:34.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f199ddc5640 1 Processor -- start 2023-10-30T13:41:34.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f199ddc5640 1 -- start start 2023-10-30T13:41:34.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f199ddc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19980cf160 0x7f19981581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:34.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f199ddc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19981538a0 0x7f19981e2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:34.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f199ddc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f199815d6f0 0x7f19981f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:34.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f199ddc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19981587b0 con 0x7f199815d6f0 2023-10-30T13:41:34.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f199ddc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1998157ec0 con 0x7f19980cf160 2023-10-30T13:41:34.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f199ddc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19981f2490 con 0x7f19981538a0 2023-10-30T13:41:34.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f1997fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f199815d6f0 0x7f19981f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:34.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f1997fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f199815d6f0 0x7f19981f29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59174/0 (socket says 172.21.15.46:59174) 2023-10-30T13:41:34.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f1997fff640 1 -- 172.21.15.46:0/2554053181 learned_addr learned my addr 172.21.15.46:0/2554053181 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:34.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f19977fe640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19980cf160 0x7f19981581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:34.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f1996ffd640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19981538a0 0x7f19981e2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:34.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f1997fff640 1 -- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19981538a0 msgr2=0x7f19981e2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f1997fff640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19981538a0 0x7f19981e2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f1997fff640 1 -- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19980cf160 msgr2=0x7f19981581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f1997fff640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19980cf160 0x7f19981581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f1997fff640 1 -- 172.21.15.46:0/2554053181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990009a70 con 0x7f199815d6f0 2023-10-30T13:41:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f19977fe640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19980cf160 0x7f19981581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f1997fff640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f199815d6f0 0x7f19981f29b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f199001cd60 tx=0x7f199001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f197ffff640 1 -- 172.21.15.46:0/2554053181 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990009450 con 0x7f199815d6f0 2023-10-30T13:41:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f197ffff640 1 -- 172.21.15.46:0/2554053181 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19900afda0 con 0x7f199815d6f0 2023-10-30T13:41:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f199ddc5640 1 -- 172.21.15.46:0/2554053181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19981595c0 con 0x7f199815d6f0 2023-10-30T13:41:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f197ffff640 1 -- 172.21.15.46:0/2554053181 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19900ba660 con 0x7f199815d6f0 2023-10-30T13:41:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f199ddc5640 1 -- 172.21.15.46:0/2554053181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19980ceb10 con 0x7f199815d6f0 2023-10-30T13:41:34.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.070+0000 7f199ddc5640 1 -- 172.21.15.46:0/2554053181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19980b2a50 con 0x7f199815d6f0 2023-10-30T13:41:34.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.074+0000 7f197ffff640 1 -- 172.21.15.46:0/2554053181 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f19900ba7c0 con 0x7f199815d6f0 2023-10-30T13:41:34.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.074+0000 7f197ffff640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19840664d0 0x7f1984068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:34.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.074+0000 7f197ffff640 1 -- 172.21.15.46:0/2554053181 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f1990122af0 con 0x7f199815d6f0 2023-10-30T13:41:34.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.074+0000 7f19977fe640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19840664d0 0x7f1984068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:34.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.074+0000 7f19977fe640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19840664d0 0x7f1984068990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1988000f70 tx=0x7f1988009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:34.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.078+0000 7f197ffff640 1 -- 172.21.15.46:0/2554053181 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f19900efb90 con 0x7f199815d6f0 2023-10-30T13:41:34.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.270+0000 7f199ddc5640 1 -- 172.21.15.46:0/2554053181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1998157d20 con 0x7f199815d6f0 2023-10-30T13:41:34.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.270+0000 7f197ffff640 1 -- 172.21.15.46:0/2554053181 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f1998157d20 con 0x7f199815d6f0 2023-10-30T13:41:34.274 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:34.275 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:28.201967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:34.275 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 92 2023-10-30T13:41:34.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.270+0000 7f199ddc5640 1 -- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19840664d0 msgr2=0x7f1984068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:34.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.270+0000 7f199ddc5640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19840664d0 0x7f1984068990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1988000f70 tx=0x7f1988009040 comp rx=0 tx=0).stop 2023-10-30T13:41:34.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.270+0000 7f199ddc5640 1 -- 172.21.15.46:0/2554053181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f199815d6f0 msgr2=0x7f19981f29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:34.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.270+0000 7f199ddc5640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f199815d6f0 0x7f19981f29b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f199001cd60 tx=0x7f199001bed0 comp rx=0 tx=0).stop 2023-10-30T13:41:34.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.270+0000 7f199ddc5640 1 -- 172.21.15.46:0/2554053181 shutdown_connections 2023-10-30T13:41:34.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.270+0000 7f199ddc5640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19840664d0 0x7f1984068990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:34.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.270+0000 7f199ddc5640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f199815d6f0 0x7f19981f29b0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:34.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.270+0000 7f199ddc5640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19981538a0 0x7f19981e2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:34.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.270+0000 7f199ddc5640 1 --2- 172.21.15.46:0/2554053181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19980cf160 0x7f19981581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:34.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.270+0000 7f199ddc5640 1 -- 172.21.15.46:0/2554053181 >> 172.21.15.46:0/2554053181 conn(0x7f199806bfe0 msgr2=0x7f199806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:34.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.274+0000 7f199ddc5640 1 -- 172.21.15.46:0/2554053181 shutdown_connections 2023-10-30T13:41:34.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:34.274+0000 7f199ddc5640 1 -- 172.21.15.46:0/2554053181 wait complete. 2023-10-30T13:41:34.296 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:36.297 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:36.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b8759e640 1 Processor -- start 2023-10-30T13:41:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b8759e640 1 -- start start 2023-10-30T13:41:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b8759e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0b881538f0 0x7f0b88159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b8759e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b8815cd50 0x7f0b880c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b8759e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8815d300 con 0x7f0b880cefa0 2023-10-30T13:41:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b8759e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b8815d5e0 con 0x7f0b8815cd50 2023-10-30T13:41:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b8759e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b8815d8c0 con 0x7f0b881538f0 2023-10-30T13:41:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b85d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b8815cd50 0x7f0b880c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:36.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b85d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b8815cd50 0x7f0b880c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51384/0 (socket says 172.21.15.46:51384) 2023-10-30T13:41:36.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b85d9b640 1 -- 172.21.15.46:0/4245980452 learned_addr learned my addr 172.21.15.46:0/4245980452 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:36.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b8659c640 1 --2- 172.21.15.46:0/4245980452 >> v2:172.21.15.154:3300/0 conn(0x7f0b881538f0 0x7f0b88159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:36.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b85d9b640 1 -- 172.21.15.46:0/4245980452 >> v2:172.21.15.154:3300/0 conn(0x7f0b881538f0 msgr2=0x7f0b88159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:36.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b85d9b640 1 --2- 172.21.15.46:0/4245980452 >> v2:172.21.15.154:3300/0 conn(0x7f0b881538f0 0x7f0b88159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:36.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b85d9b640 1 -- 172.21.15.46:0/4245980452 >> v1:172.21.15.154:6789/0 conn(0x7f0b880cefa0 legacy=0x7f0b880cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:36.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.450+0000 7f0b85d9b640 1 -- 172.21.15.46:0/4245980452 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8815dba0 con 0x7f0b8815cd50 2023-10-30T13:41:36.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.454+0000 7f0b85d9b640 1 --2- 172.21.15.46:0/4245980452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b8815cd50 0x7f0b880c9f90 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f0b880c9cb0 tx=0x7f0b7c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0438602214cd82 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:36.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.454+0000 7f0b8559a640 1 -- 172.21.15.46:0/4245980452 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c0a89f0 con 0x7f0b8815cd50 2023-10-30T13:41:36.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.454+0000 7f0b8559a640 1 -- 172.21.15.46:0/4245980452 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7c0a8b50 con 0x7f0b8815cd50 2023-10-30T13:41:36.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.454+0000 7f0b8559a640 1 -- 172.21.15.46:0/4245980452 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c0b0440 con 0x7f0b8815cd50 2023-10-30T13:41:36.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.454+0000 7f0b8759e640 1 -- 172.21.15.46:0/4245980452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b8815cd50 msgr2=0x7f0b880c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:36.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.454+0000 7f0b8759e640 1 --2- 172.21.15.46:0/4245980452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b8815cd50 0x7f0b880c9f90 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f0b880c9cb0 tx=0x7f0b7c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:41:36.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.454+0000 7f0b8759e640 1 -- 172.21.15.46:0/4245980452 shutdown_connections 2023-10-30T13:41:36.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.454+0000 7f0b8759e640 1 --2- 172.21.15.46:0/4245980452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b8815cd50 0x7f0b880c9f90 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:36.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.454+0000 7f0b8759e640 1 --2- 172.21.15.46:0/4245980452 >> v2:172.21.15.154:3300/0 conn(0x7f0b881538f0 0x7f0b88159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:36.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.454+0000 7f0b8759e640 1 -- 172.21.15.46:0/4245980452 >> 172.21.15.46:0/4245980452 conn(0x7f0b8806a640 msgr2=0x7f0b88065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:36.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.454+0000 7f0b8759e640 1 -- 172.21.15.46:0/4245980452 shutdown_connections 2023-10-30T13:41:36.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.454+0000 7f0b8759e640 1 -- 172.21.15.46:0/4245980452 wait complete. 2023-10-30T13:41:36.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b8759e640 1 Processor -- start 2023-10-30T13:41:36.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b8759e640 1 -- start start 2023-10-30T13:41:36.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b8759e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b880cefa0 0x7f0b8813c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:36.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b8759e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b881538f0 0x7f0b881eeef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:36.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b8759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b8815cd50 0x7f0b88141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:36.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b8759e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880b4cc0 con 0x7f0b8815cd50 2023-10-30T13:41:36.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b8759e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b88142a90 con 0x7f0b881538f0 2023-10-30T13:41:36.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b8759e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880b2fa0 con 0x7f0b880cefa0 2023-10-30T13:41:36.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b86d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b8815cd50 0x7f0b88141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:36.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b86d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b8815cd50 0x7f0b88141cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51396/0 (socket says 172.21.15.46:51396) 2023-10-30T13:41:36.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b86d9d640 1 -- 172.21.15.46:0/907162034 learned_addr learned my addr 172.21.15.46:0/907162034 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:36.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b85d9b640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b881538f0 0x7f0b881eeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:36.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b8659c640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b880cefa0 0x7f0b8813c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:36.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b86d9d640 1 -- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b880cefa0 msgr2=0x7f0b8813c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:36.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b86d9d640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b880cefa0 0x7f0b8813c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:36.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b86d9d640 1 -- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b881538f0 msgr2=0x7f0b881eeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:36.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b86d9d640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b881538f0 0x7f0b881eeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:36.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b86d9d640 1 -- 172.21.15.46:0/907162034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c0094d0 con 0x7f0b8815cd50 2023-10-30T13:41:36.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b86d9d640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b8815cd50 0x7f0b88141cd0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000c770 tx=0x7f0b8000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:36.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b76ffd640 1 -- 172.21.15.46:0/907162034 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80003b80 con 0x7f0b8815cd50 2023-10-30T13:41:36.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b76ffd640 1 -- 172.21.15.46:0/907162034 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b80003ce0 con 0x7f0b8815cd50 2023-10-30T13:41:36.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b8759e640 1 -- 172.21.15.46:0/907162034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b880b3840 con 0x7f0b8815cd50 2023-10-30T13:41:36.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b76ffd640 1 -- 172.21.15.46:0/907162034 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8000aa90 con 0x7f0b8815cd50 2023-10-30T13:41:36.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b8759e640 1 -- 172.21.15.46:0/907162034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8813c870 con 0x7f0b8815cd50 2023-10-30T13:41:36.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.458+0000 7f0b8759e640 1 -- 172.21.15.46:0/907162034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b880b39d0 con 0x7f0b8815cd50 2023-10-30T13:41:36.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.462+0000 7f0b76ffd640 1 -- 172.21.15.46:0/907162034 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0b8000abf0 con 0x7f0b8815cd50 2023-10-30T13:41:36.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.462+0000 7f0b76ffd640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0b680664a0 0x7f0b68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:36.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.462+0000 7f0b76ffd640 1 -- 172.21.15.46:0/907162034 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f0b80084040 con 0x7f0b8815cd50 2023-10-30T13:41:36.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.462+0000 7f0b8659c640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0b680664a0 0x7f0b68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:36.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.462+0000 7f0b8659c640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0b680664a0 0x7f0b68068960 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0b7800b9d0 tx=0x7f0b78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:36.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.466+0000 7f0b76ffd640 1 -- 172.21.15.46:0/907162034 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0b800b5090 con 0x7f0b8815cd50 2023-10-30T13:41:36.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 -- 172.21.15.46:0/907162034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b8804e170 con 0x7f0b8815cd50 2023-10-30T13:41:36.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b76ffd640 1 -- 172.21.15.46:0/907162034 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f0b800510c0 con 0x7f0b8815cd50 2023-10-30T13:41:36.662 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:36.662 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:34.199097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:36.662 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 93 2023-10-30T13:41:36.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 -- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0b680664a0 msgr2=0x7f0b68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:36.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0b680664a0 0x7f0b68068960 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0b7800b9d0 tx=0x7f0b78008040 comp rx=0 tx=0).stop 2023-10-30T13:41:36.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 -- 172.21.15.46:0/907162034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b8815cd50 msgr2=0x7f0b88141cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:36.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b8815cd50 0x7f0b88141cd0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000c770 tx=0x7f0b8000cc30 comp rx=0 tx=0).stop 2023-10-30T13:41:36.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 -- 172.21.15.46:0/907162034 shutdown_connections 2023-10-30T13:41:36.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0b680664a0 0x7f0b68068960 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b8815cd50 0x7f0b88141cd0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b881538f0 0x7f0b881eeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 --2- 172.21.15.46:0/907162034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0b880cefa0 0x7f0b8813c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 -- 172.21.15.46:0/907162034 >> 172.21.15.46:0/907162034 conn(0x7f0b8806a640 msgr2=0x7f0b880c4990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:36.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 -- 172.21.15.46:0/907162034 shutdown_connections 2023-10-30T13:41:36.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:36.658+0000 7f0b8759e640 1 -- 172.21.15.46:0/907162034 wait complete. 2023-10-30T13:41:36.682 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:38.684 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:38.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.834+0000 7f3210fc5640 1 Processor -- start 2023-10-30T13:41:38.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.834+0000 7f3210fc5640 1 -- start start 2023-10-30T13:41:38.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.834+0000 7f3210fc5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f320c1539a0 0x7f320c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:38.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.834+0000 7f3210fc5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f320c070a00 0x7f320c070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:38.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.834+0000 7f3210fc5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320c15a8c0 con 0x7f320c069e90 2023-10-30T13:41:38.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.834+0000 7f3210fc5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320c06fbb0 con 0x7f320c1539a0 2023-10-30T13:41:38.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.834+0000 7f3210fc5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320c0713b0 con 0x7f320c070a00 2023-10-30T13:41:38.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.838+0000 7f320a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f320c1539a0 0x7f320c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:38.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.838+0000 7f320a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f320c1539a0 0x7f320c15a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51398/0 (socket says 172.21.15.46:51398) 2023-10-30T13:41:38.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.838+0000 7f320a59c640 1 -- 172.21.15.46:0/23448821 learned_addr learned my addr 172.21.15.46:0/23448821 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:38.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.838+0000 7f3209d9b640 1 --2- 172.21.15.46:0/23448821 >> v2:172.21.15.154:3300/0 conn(0x7f320c070a00 0x7f320c070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:38.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.838+0000 7f320a59c640 1 -- 172.21.15.46:0/23448821 >> v2:172.21.15.154:3300/0 conn(0x7f320c070a00 msgr2=0x7f320c070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:38.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.838+0000 7f320a59c640 1 --2- 172.21.15.46:0/23448821 >> v2:172.21.15.154:3300/0 conn(0x7f320c070a00 0x7f320c070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:38.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.838+0000 7f320a59c640 1 -- 172.21.15.46:0/23448821 >> v1:172.21.15.154:6789/0 conn(0x7f320c069e90 legacy=0x7f320c06a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:38.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.838+0000 7f320a59c640 1 -- 172.21.15.46:0/23448821 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c153d80 con 0x7f320c1539a0 2023-10-30T13:41:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.838+0000 7f320a59c640 1 --2- 172.21.15.46:0/23448821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f320c1539a0 0x7f320c15a380 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f320c072640 tx=0x7f31fc01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332a83458b77ff6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.838+0000 7f320959a640 1 -- 172.21.15.46:0/23448821 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0ab640 con 0x7f320c1539a0 2023-10-30T13:41:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.838+0000 7f320959a640 1 -- 172.21.15.46:0/23448821 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc0a7070 con 0x7f320c1539a0 2023-10-30T13:41:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.838+0000 7f320959a640 1 -- 172.21.15.46:0/23448821 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0ab9c0 con 0x7f320c1539a0 2023-10-30T13:41:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 -- 172.21.15.46:0/23448821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f320c1539a0 msgr2=0x7f320c15a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 --2- 172.21.15.46:0/23448821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f320c1539a0 0x7f320c15a380 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f320c072640 tx=0x7f31fc01c440 comp rx=0 tx=0).stop 2023-10-30T13:41:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 -- 172.21.15.46:0/23448821 shutdown_connections 2023-10-30T13:41:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 --2- 172.21.15.46:0/23448821 >> v2:172.21.15.154:3300/0 conn(0x7f320c070a00 0x7f320c070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 --2- 172.21.15.46:0/23448821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f320c1539a0 0x7f320c15a380 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 -- 172.21.15.46:0/23448821 >> 172.21.15.46:0/23448821 conn(0x7f320c0c7910 msgr2=0x7f320c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 -- 172.21.15.46:0/23448821 shutdown_connections 2023-10-30T13:41:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 -- 172.21.15.46:0/23448821 wait complete. 2023-10-30T13:41:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 Processor -- start 2023-10-30T13:41:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 -- start start 2023-10-30T13:41:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f320c069e90 0x7f320c1f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f320c070a00 0x7f320c14f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f320c1539a0 0x7f320c0ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c1f3090 con 0x7f320c069e90 2023-10-30T13:41:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c1f21e0 con 0x7f320c1539a0 2023-10-30T13:41:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3210fc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c1f2480 con 0x7f320c070a00 2023-10-30T13:41:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f320a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f320c069e90 0x7f320c1f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f320a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f320c069e90 0x7f320c1f61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51412/0 (socket says 172.21.15.46:51412) 2023-10-30T13:41:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f320a59c640 1 -- 172.21.15.46:0/790970104 learned_addr learned my addr 172.21.15.46:0/790970104 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f3209d9b640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f320c070a00 0x7f320c14f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.842+0000 7f320ad9d640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f320c1539a0 0x7f320c0ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f320a59c640 1 -- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f320c070a00 msgr2=0x7f320c14f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f320a59c640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f320c070a00 0x7f320c14f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f320a59c640 1 -- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f320c1539a0 msgr2=0x7f320c0ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f320a59c640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f320c1539a0 0x7f320c0ca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f320a59c640 1 -- 172.21.15.46:0/790970104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31fc0094d0 con 0x7f320c069e90 2023-10-30T13:41:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f320ad9d640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f320c1539a0 0x7f320c0ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f320a59c640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f320c069e90 0x7f320c1f61a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f31fc019050 tx=0x7f31fc01ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f31faffd640 1 -- 172.21.15.46:0/790970104 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0be3f0 con 0x7f320c069e90 2023-10-30T13:41:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f31faffd640 1 -- 172.21.15.46:0/790970104 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc0be550 con 0x7f320c069e90 2023-10-30T13:41:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f31faffd640 1 -- 172.21.15.46:0/790970104 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0be900 con 0x7f320c069e90 2023-10-30T13:41:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f3210fc5640 1 -- 172.21.15.46:0/790970104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320c1f2760 con 0x7f320c069e90 2023-10-30T13:41:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f3210fc5640 1 -- 172.21.15.46:0/790970104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320c1f2d20 con 0x7f320c069e90 2023-10-30T13:41:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f3210fc5640 1 -- 172.21.15.46:0/790970104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320c04e170 con 0x7f320c069e90 2023-10-30T13:41:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f31faffd640 1 -- 172.21.15.46:0/790970104 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f31fc0b43f0 con 0x7f320c069e90 2023-10-30T13:41:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.846+0000 7f31faffd640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31ec0664a0 0x7f31ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.850+0000 7f31faffd640 1 -- 172.21.15.46:0/790970104 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f31fc11e630 con 0x7f320c069e90 2023-10-30T13:41:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.850+0000 7f3209d9b640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31ec0664a0 0x7f31ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.850+0000 7f3209d9b640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31ec0664a0 0x7f31ec068960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f320c1593b0 tx=0x7f3200008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:38.850+0000 7f31faffd640 1 -- 172.21.15.46:0/790970104 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31fc0eb640 con 0x7f320c069e90 2023-10-30T13:41:39.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.042+0000 7f3210fc5640 1 -- 172.21.15.46:0/790970104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f320c14da30 con 0x7f320c069e90 2023-10-30T13:41:39.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.042+0000 7f31faffd640 1 -- 172.21.15.46:0/790970104 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f31fc0eec60 con 0x7f320c069e90 2023-10-30T13:41:39.047 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:39.047 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:34.199097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:39.047 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 93 2023-10-30T13:41:39.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.042+0000 7f3210fc5640 1 -- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31ec0664a0 msgr2=0x7f31ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:39.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.042+0000 7f3210fc5640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31ec0664a0 0x7f31ec068960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f320c1593b0 tx=0x7f3200008040 comp rx=0 tx=0).stop 2023-10-30T13:41:39.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.042+0000 7f3210fc5640 1 -- 172.21.15.46:0/790970104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f320c069e90 msgr2=0x7f320c1f61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:39.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.042+0000 7f3210fc5640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f320c069e90 0x7f320c1f61a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f31fc019050 tx=0x7f31fc01ac80 comp rx=0 tx=0).stop 2023-10-30T13:41:39.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.046+0000 7f3210fc5640 1 -- 172.21.15.46:0/790970104 shutdown_connections 2023-10-30T13:41:39.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.046+0000 7f3210fc5640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31ec0664a0 0x7f31ec068960 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:39.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.046+0000 7f3210fc5640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f320c1539a0 0x7f320c0ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.046+0000 7f3210fc5640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f320c070a00 0x7f320c14f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.046+0000 7f3210fc5640 1 --2- 172.21.15.46:0/790970104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f320c069e90 0x7f320c1f61a0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.046+0000 7f3210fc5640 1 -- 172.21.15.46:0/790970104 >> 172.21.15.46:0/790970104 conn(0x7f320c0c7910 msgr2=0x7f320c14de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.046+0000 7f3210fc5640 1 -- 172.21.15.46:0/790970104 shutdown_connections 2023-10-30T13:41:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:39.046+0000 7f3210fc5640 1 -- 172.21.15.46:0/790970104 wait complete. 2023-10-30T13:41:39.071 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 28 }, "gid_6036": { "addr": "172.21.15.46:6836/3037060424", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3037060424, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3037060424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6036, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 37 }, "gid_6201": { "addr": "172.21.15.46:6837/2546436963", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2546436963, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2546436963, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6201, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:41:34.199097+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6201, "mds_1": 6036, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:41:41.072 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.222+0000 7f7aac71d640 1 Processor -- start 2023-10-30T13:41:41.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.222+0000 7f7aac71d640 1 -- start start 2023-10-30T13:41:41.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.222+0000 7f7aac71d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa4153d70 0x7f7aa406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:41.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.222+0000 7f7aac71d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa406afa0 con 0x7f7aa40c8490 2023-10-30T13:41:41.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aac71d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa4154150 con 0x7f7aa406e470 2023-10-30T13:41:41.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aac71d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa406d780 con 0x7f7aa4153d70 2023-10-30T13:41:41.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aaacba640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7aa40c8490 0x7f7aa40c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47898/0 (socket says 172.21.15.46:47898) 2023-10-30T13:41:41.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aaacba640 1 -- 172.21.15.46:0/1804840834 learned_addr learned my addr 172.21.15.46:0/1804840834 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:41:41.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aaa4b9640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa4153d70 0x7f7aa406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:41.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368329108 0 0) 0x7f7aa406afa0 con 0x7f7aa40c8490 2023-10-30T13:41:41.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a90003530 con 0x7f7aa40c8490 2023-10-30T13:41:41.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421652722 0 0) 0x7f7aa4154150 con 0x7f7aa406e470 2023-10-30T13:41:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aa406afa0 con 0x7f7aa406e470 2023-10-30T13:41:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2746102217 0 0) 0x7f7a90003530 con 0x7f7aa40c8490 2023-10-30T13:41:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa4153d70 msgr2=0x7f7aa406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa4153d70 0x7f7aa406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 >> v1:172.21.15.154:6789/0 conn(0x7f7aa406e470 legacy=0x7f7aa40cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa406e030 con 0x7f7aa40c8490 2023-10-30T13:41:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f7a9c001a00 con 0x7f7aa40c8490 2023-10-30T13:41:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aaa4b9640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa4153d70 0x7f7aa406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:41:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 >> v1:172.21.15.46:6789/0 conn(0x7f7aa40c8490 legacy=0x7f7aa40c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a90003b40 0x7f7a90003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa4153d70 0x7f7a900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a90004d10 0x7f7a90005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa406d780 con 0x7f7a90003b40 2023-10-30T13:41:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9000ba40 con 0x7f7a90004d10 2023-10-30T13:41:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9000bb80 con 0x7f7aa4153d70 2023-10-30T13:41:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aaa4b9640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a90003b40 0x7f7a90003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aaacba640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a90004d10 0x7f7a90005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aa9cb8640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa4153d70 0x7f7a900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aaa4b9640 1 -- 172.21.15.46:0/1804840834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa4153d70 msgr2=0x7f7a900047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aaa4b9640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa4153d70 0x7f7a900047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aaa4b9640 1 -- 172.21.15.46:0/1804840834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a90004d10 msgr2=0x7f7a90005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aaa4b9640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a90004d10 0x7f7a90005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.226+0000 7f7aaa4b9640 1 -- 172.21.15.46:0/1804840834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa0009420 con 0x7f7a90003b40 2023-10-30T13:41:41.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aaa4b9640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a90003b40 0x7f7a90003f40 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009ce0 tx=0x7f7aa001cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9761b418c1930dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:41.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00a5840 con 0x7f7a90003b40 2023-10-30T13:41:41.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa00a59a0 con 0x7f7a90003b40 2023-10-30T13:41:41.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aa94b7640 1 -- 172.21.15.46:0/1804840834 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00a5d50 con 0x7f7a90003b40 2023-10-30T13:41:41.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aac71d640 1 -- 172.21.15.46:0/1804840834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a90003b40 msgr2=0x7f7a90003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:41.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aac71d640 1 --2- 172.21.15.46:0/1804840834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a90003b40 0x7f7a90003f40 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009ce0 tx=0x7f7aa001cba0 comp rx=0 tx=0).stop 2023-10-30T13:41:41.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aaacba640 1 -- 172.21.15.46:0/1804840834 reap_dead start 2023-10-30T13:41:41.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aac71d640 1 -- 172.21.15.46:0/1804840834 shutdown_connections 2023-10-30T13:41:41.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aac71d640 1 -- 172.21.15.46:0/1804840834 >> 172.21.15.46:0/1804840834 conn(0x7f7aa4071ea0 msgr2=0x7f7aa4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:41.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aac71d640 1 -- 172.21.15.46:0/1804840834 shutdown_connections 2023-10-30T13:41:41.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aac71d640 1 -- 172.21.15.46:0/1804840834 wait complete. 2023-10-30T13:41:41.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aac71d640 1 Processor -- start 2023-10-30T13:41:41.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.230+0000 7f7aac71d640 1 -- start start 2023-10-30T13:41:41.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aac71d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a90003b40 0x7f7aa4161f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:41.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aac71d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7aa414d8f0 0x7f7aa414a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:41.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aac71d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa41601d0 0x7f7aa4156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aac71d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa414d460 con 0x7f7a90003b40 2023-10-30T13:41:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aac71d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa414c5e0 con 0x7f7aa414d8f0 2023-10-30T13:41:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aac71d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa414aa70 con 0x7f7aa41601d0 2023-10-30T13:41:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aaa4b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a90003b40 0x7f7aa4161f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:41.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aaa4b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a90003b40 0x7f7aa4161f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51422/0 (socket says 172.21.15.46:51422) 2023-10-30T13:41:41.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aaa4b9640 1 -- 172.21.15.46:0/3475716273 learned_addr learned my addr 172.21.15.46:0/3475716273 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:41.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aaacba640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa41601d0 0x7f7aa4156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:41.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aa9cb8640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7aa414d8f0 0x7f7aa414a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:41.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aaa4b9640 1 -- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa41601d0 msgr2=0x7f7aa4156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:41.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aaa4b9640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa41601d0 0x7f7aa4156760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:41.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aaa4b9640 1 -- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7aa414d8f0 msgr2=0x7f7aa414a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:41.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aaa4b9640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7aa414d8f0 0x7f7aa414a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:41.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aaa4b9640 1 -- 172.21.15.46:0/3475716273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa414f4b0 con 0x7f7a90003b40 2023-10-30T13:41:41.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aaa4b9640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a90003b40 0x7f7aa4161f60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009ce0 tx=0x7f7a90005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:41.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7a9affd640 1 -- 172.21.15.46:0/3475716273 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00ad440 con 0x7f7a90003b40 2023-10-30T13:41:41.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7a9affd640 1 -- 172.21.15.46:0/3475716273 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa00ad5a0 con 0x7f7a90003b40 2023-10-30T13:41:41.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7a9affd640 1 -- 172.21.15.46:0/3475716273 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00ad930 con 0x7f7a90003b40 2023-10-30T13:41:41.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aac71d640 1 -- 172.21.15.46:0/3475716273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa414c850 con 0x7f7a90003b40 2023-10-30T13:41:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aac71d640 1 -- 172.21.15.46:0/3475716273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa414ce10 con 0x7f7a90003b40 2023-10-30T13:41:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7aac71d640 1 -- 172.21.15.46:0/3475716273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa404e170 con 0x7f7a90003b40 2023-10-30T13:41:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.234+0000 7f7a9affd640 1 -- 172.21.15.46:0/3475716273 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7aa00b4040 con 0x7f7a90003b40 2023-10-30T13:41:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.238+0000 7f7a9affd640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7a880664a0 0x7f7a88068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.238+0000 7f7a9affd640 1 -- 172.21.15.46:0/3475716273 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f7aa011ea70 con 0x7f7a90003b40 2023-10-30T13:41:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.238+0000 7f7aa9cb8640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7a880664a0 0x7f7a88068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.238+0000 7f7aa9cb8640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7a880664a0 0x7f7a88068960 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7a94007af0 tx=0x7f7a94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.242+0000 7f7a9affd640 1 -- 172.21.15.46:0/3475716273 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7aa00ebbb0 con 0x7f7a90003b40 2023-10-30T13:41:41.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 -- 172.21.15.46:0/3475716273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7aa406c2a0 con 0x7f7a90003b40 2023-10-30T13:41:41.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7a9affd640 1 -- 172.21.15.46:0/3475716273 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5749 (secure 0 0 0) 0x7f7aa00ef1d0 con 0x7f7a90003b40 2023-10-30T13:41:41.434 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:41.434 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:40.224427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:41.435 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 94 2023-10-30T13:41:41.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 -- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7a880664a0 msgr2=0x7f7a88068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:41.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7a880664a0 0x7f7a88068960 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7a94007af0 tx=0x7f7a94009040 comp rx=0 tx=0).stop 2023-10-30T13:41:41.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 -- 172.21.15.46:0/3475716273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a90003b40 msgr2=0x7f7aa4161f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:41.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a90003b40 0x7f7aa4161f60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009ce0 tx=0x7f7a90005800 comp rx=0 tx=0).stop 2023-10-30T13:41:41.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 -- 172.21.15.46:0/3475716273 shutdown_connections 2023-10-30T13:41:41.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7a880664a0 0x7f7a88068960 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:41.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7aa41601d0 0x7f7aa4156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:41.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7aa414d8f0 0x7f7aa414a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:41.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 --2- 172.21.15.46:0/3475716273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a90003b40 0x7f7aa4161f60 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:41.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 -- 172.21.15.46:0/3475716273 >> 172.21.15.46:0/3475716273 conn(0x7f7aa4071ea0 msgr2=0x7f7aa414eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:41.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 -- 172.21.15.46:0/3475716273 shutdown_connections 2023-10-30T13:41:41.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:41.430+0000 7f7aac71d640 1 -- 172.21.15.46:0/3475716273 wait complete. 2023-10-30T13:41:41.455 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:43.457 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:43.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.606+0000 7f0aded3e640 1 Processor -- start 2023-10-30T13:41:43.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.606+0000 7f0aded3e640 1 -- start start 2023-10-30T13:41:43.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0aded3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad806f7a0 0x7f0ad806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:43.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0aded3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0ad80c6bd0 0x7f0ad8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:43.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0aded3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad806a3d0 con 0x7f0ad8071110 2023-10-30T13:41:43.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0aded3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad80700c0 con 0x7f0ad80c6bd0 2023-10-30T13:41:43.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0aded3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad8066cf0 con 0x7f0ad806f7a0 2023-10-30T13:41:43.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0adcada640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad806f7a0 0x7f0ad806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:43.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0ad7fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0ad80c6bd0 0x7f0ad8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:43.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0add2db640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f0ad8071110 0x7f0ad8158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:51914/0 (socket says 172.21.15.46:51914) 2023-10-30T13:41:43.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0add2db640 1 -- 172.21.15.46:0/1652052927 learned_addr learned my addr 172.21.15.46:0/1652052927 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:41:43.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0adcada640 1 -- 172.21.15.46:0/1652052927 >> v2:172.21.15.154:3300/0 conn(0x7f0ad80c6bd0 msgr2=0x7f0ad8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:43.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0adcada640 1 --2- 172.21.15.46:0/1652052927 >> v2:172.21.15.154:3300/0 conn(0x7f0ad80c6bd0 0x7f0ad8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:43.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0adcada640 1 -- 172.21.15.46:0/1652052927 >> v1:172.21.15.154:6789/0 conn(0x7f0ad8071110 legacy=0x7f0ad8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:43.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0adcada640 1 -- 172.21.15.46:0/1652052927 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad80c70d0 con 0x7f0ad806f7a0 2023-10-30T13:41:43.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.610+0000 7f0ad7fff640 1 --2- 172.21.15.46:0/1652052927 >> v2:172.21.15.154:3300/0 conn(0x7f0ad80c6bd0 0x7f0ad8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:41:43.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0adcada640 1 --2- 172.21.15.46:0/1652052927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad806f7a0 0x7f0ad806fb80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0097c0 tx=0x7f0acc01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=685341bd42b66ae3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:43.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0ad77fe640 1 -- 172.21.15.46:0/1652052927 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc0a8970 con 0x7f0ad806f7a0 2023-10-30T13:41:43.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0ad77fe640 1 -- 172.21.15.46:0/1652052927 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0acc0a8ad0 con 0x7f0ad806f7a0 2023-10-30T13:41:43.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0ad77fe640 1 -- 172.21.15.46:0/1652052927 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc0a8e60 con 0x7f0ad806f7a0 2023-10-30T13:41:43.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0aded3e640 1 -- 172.21.15.46:0/1652052927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad806f7a0 msgr2=0x7f0ad806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:43.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0aded3e640 1 --2- 172.21.15.46:0/1652052927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad806f7a0 0x7f0ad806fb80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0097c0 tx=0x7f0acc01ec70 comp rx=0 tx=0).stop 2023-10-30T13:41:43.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0aded3e640 1 -- 172.21.15.46:0/1652052927 shutdown_connections 2023-10-30T13:41:43.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0aded3e640 1 --2- 172.21.15.46:0/1652052927 >> v2:172.21.15.154:3300/0 conn(0x7f0ad80c6bd0 0x7f0ad8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:43.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0aded3e640 1 --2- 172.21.15.46:0/1652052927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad806f7a0 0x7f0ad806fb80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:43.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0aded3e640 1 -- 172.21.15.46:0/1652052927 >> 172.21.15.46:0/1652052927 conn(0x7f0ad80c7910 msgr2=0x7f0ad80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:43.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0aded3e640 1 -- 172.21.15.46:0/1652052927 shutdown_connections 2023-10-30T13:41:43.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0aded3e640 1 -- 172.21.15.46:0/1652052927 wait complete. 2023-10-30T13:41:43.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0aded3e640 1 Processor -- start 2023-10-30T13:41:43.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.614+0000 7f0aded3e640 1 -- start start 2023-10-30T13:41:43.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0aded3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad806f7a0 0x7f0ad80ae530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:43.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0aded3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad8071110 0x7f0ad80aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:43.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0aded3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad80c6bd0 0x7f0ad80c0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:43.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0aded3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad81f3530 con 0x7f0ad8071110 2023-10-30T13:41:43.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0aded3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad80afc70 con 0x7f0ad806f7a0 2023-10-30T13:41:43.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0aded3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad815ac10 con 0x7f0ad80c6bd0 2023-10-30T13:41:43.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad8071110 0x7f0ad80aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:43.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad8071110 0x7f0ad80aea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51430/0 (socket says 172.21.15.46:51430) 2023-10-30T13:41:43.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad7fff640 1 -- 172.21.15.46:0/1836104694 learned_addr learned my addr 172.21.15.46:0/1836104694 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:43.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0adcada640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad806f7a0 0x7f0ad80ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:43.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0add2db640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad80c6bd0 0x7f0ad80c0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:43.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad7fff640 1 -- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad80c6bd0 msgr2=0x7f0ad80c0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:43.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad7fff640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad80c6bd0 0x7f0ad80c0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:43.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad7fff640 1 -- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad806f7a0 msgr2=0x7f0ad80ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:43.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad7fff640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad806f7a0 0x7f0ad80ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:43.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad7fff640 1 -- 172.21.15.46:0/1836104694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac4009420 con 0x7f0ad8071110 2023-10-30T13:41:43.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0add2db640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad80c6bd0 0x7f0ad80c0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:43.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad7fff640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad8071110 0x7f0ad80aea70 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400e9b0 tx=0x7f0ac400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:43.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad57fa640 1 -- 172.21.15.46:0/1836104694 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac400ca00 con 0x7f0ad8071110 2023-10-30T13:41:43.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad57fa640 1 -- 172.21.15.46:0/1836104694 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac400cb60 con 0x7f0ad8071110 2023-10-30T13:41:43.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad57fa640 1 -- 172.21.15.46:0/1836104694 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4013470 con 0x7f0ad8071110 2023-10-30T13:41:43.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0aded3e640 1 -- 172.21.15.46:0/1836104694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0acc009420 con 0x7f0ad8071110 2023-10-30T13:41:43.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0aded3e640 1 -- 172.21.15.46:0/1836104694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad80c00f0 con 0x7f0ad8071110 2023-10-30T13:41:43.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0aded3e640 1 -- 172.21.15.46:0/1836104694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad80c0280 con 0x7f0ad8071110 2023-10-30T13:41:43.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.618+0000 7f0ad57fa640 1 -- 172.21.15.46:0/1836104694 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ac40135d0 con 0x7f0ad8071110 2023-10-30T13:41:43.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.622+0000 7f0ad57fa640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0abc0664d0 0x7f0abc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:43.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.622+0000 7f0ad57fa640 1 -- 172.21.15.46:0/1836104694 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f0ac4084480 con 0x7f0ad8071110 2023-10-30T13:41:43.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.622+0000 7f0adcada640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0abc0664d0 0x7f0abc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:43.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.622+0000 7f0adcada640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0abc0664d0 0x7f0abc068990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0b2000 tx=0x7f0acc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:43.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.626+0000 7f0ad57fa640 1 -- 172.21.15.46:0/1836104694 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0ac4051520 con 0x7f0ad8071110 2023-10-30T13:41:43.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.818+0000 7f0aded3e640 1 -- 172.21.15.46:0/1836104694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ad804e170 con 0x7f0ad8071110 2023-10-30T13:41:43.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.818+0000 7f0ad57fa640 1 -- 172.21.15.46:0/1836104694 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5749 (secure 0 0 0) 0x7f0ac4054b40 con 0x7f0ad8071110 2023-10-30T13:41:43.823 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:43.823 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:40.224427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:43.824 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 94 2023-10-30T13:41:43.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.822+0000 7f0aded3e640 1 -- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0abc0664d0 msgr2=0x7f0abc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:43.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.822+0000 7f0aded3e640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0abc0664d0 0x7f0abc068990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0b2000 tx=0x7f0acc008040 comp rx=0 tx=0).stop 2023-10-30T13:41:43.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.822+0000 7f0aded3e640 1 -- 172.21.15.46:0/1836104694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad8071110 msgr2=0x7f0ad80aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:43.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.822+0000 7f0aded3e640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad8071110 0x7f0ad80aea70 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400e9b0 tx=0x7f0ac400ee70 comp rx=0 tx=0).stop 2023-10-30T13:41:43.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.822+0000 7f0aded3e640 1 -- 172.21.15.46:0/1836104694 shutdown_connections 2023-10-30T13:41:43.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.822+0000 7f0aded3e640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0abc0664d0 0x7f0abc068990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:43.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.822+0000 7f0aded3e640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad80c6bd0 0x7f0ad80c0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:43.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.822+0000 7f0aded3e640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad8071110 0x7f0ad80aea70 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:43.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.822+0000 7f0aded3e640 1 --2- 172.21.15.46:0/1836104694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad806f7a0 0x7f0ad80ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.822+0000 7f0aded3e640 1 -- 172.21.15.46:0/1836104694 >> 172.21.15.46:0/1836104694 conn(0x7f0ad80c7910 msgr2=0x7f0ad80c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.822+0000 7f0aded3e640 1 -- 172.21.15.46:0/1836104694 shutdown_connections 2023-10-30T13:41:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:43.822+0000 7f0aded3e640 1 -- 172.21.15.46:0/1836104694 wait complete. 2023-10-30T13:41:43.844 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:45.845 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:46.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc77d3640 1 Processor -- start 2023-10-30T13:41:46.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc77d3640 1 -- start start 2023-10-30T13:41:46.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc77d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6fc015d6f0 0x7f6fc015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:46.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc77d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc00cf160 0x7f6fc00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:46.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc77d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc00cb800 con 0x7f6fc01538a0 2023-10-30T13:41:46.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc77d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc00cb330 con 0x7f6fc015d6f0 2023-10-30T13:41:46.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc77d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc00cad70 con 0x7f6fc00cf160 2023-10-30T13:41:46.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc4d6e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6fc015d6f0 0x7f6fc015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:46.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc4d6e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6fc015d6f0 0x7f6fc015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51446/0 (socket says 172.21.15.46:51446) 2023-10-30T13:41:46.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc4d6e640 1 -- 172.21.15.46:0/191331035 learned_addr learned my addr 172.21.15.46:0/191331035 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:46.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc5d70640 1 --2- 172.21.15.46:0/191331035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc00cf160 0x7f6fc00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc4d6e640 1 -- 172.21.15.46:0/191331035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc00cf160 msgr2=0x7f6fc00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc4d6e640 1 --2- 172.21.15.46:0/191331035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc00cf160 0x7f6fc00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc4d6e640 1 -- 172.21.15.46:0/191331035 >> v1:172.21.15.154:6789/0 conn(0x7f6fc01538a0 legacy=0x7f6fc00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:46.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:45.998+0000 7f6fc4d6e640 1 -- 172.21.15.46:0/191331035 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc00cb050 con 0x7f6fc015d6f0 2023-10-30T13:41:46.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.002+0000 7f6fc4d6e640 1 --2- 172.21.15.46:0/191331035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc015d6f0 0x7f6fc015a090 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0009870 tx=0x7f6fb001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689fefe896010645 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.002+0000 7f6fb7fff640 1 -- 172.21.15.46:0/191331035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb00a8a30 con 0x7f6fc015d6f0 2023-10-30T13:41:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.002+0000 7f6fb7fff640 1 -- 172.21.15.46:0/191331035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb00a8b90 con 0x7f6fc015d6f0 2023-10-30T13:41:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.002+0000 7f6fb7fff640 1 -- 172.21.15.46:0/191331035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb00b0440 con 0x7f6fc015d6f0 2023-10-30T13:41:46.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.002+0000 7f6fc77d3640 1 -- 172.21.15.46:0/191331035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc015d6f0 msgr2=0x7f6fc015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:46.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.002+0000 7f6fc77d3640 1 --2- 172.21.15.46:0/191331035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc015d6f0 0x7f6fc015a090 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0009870 tx=0x7f6fb001ed30 comp rx=0 tx=0).stop 2023-10-30T13:41:46.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.002+0000 7f6fc77d3640 1 -- 172.21.15.46:0/191331035 shutdown_connections 2023-10-30T13:41:46.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.002+0000 7f6fc77d3640 1 --2- 172.21.15.46:0/191331035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc00cf160 0x7f6fc00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:46.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.002+0000 7f6fc77d3640 1 --2- 172.21.15.46:0/191331035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc015d6f0 0x7f6fc015a090 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:46.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.002+0000 7f6fc77d3640 1 -- 172.21.15.46:0/191331035 >> 172.21.15.46:0/191331035 conn(0x7f6fc006bfe0 msgr2=0x7f6fc006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:46.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.002+0000 7f6fc77d3640 1 -- 172.21.15.46:0/191331035 shutdown_connections 2023-10-30T13:41:46.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.002+0000 7f6fc77d3640 1 -- 172.21.15.46:0/191331035 wait complete. 2023-10-30T13:41:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc77d3640 1 Processor -- start 2023-10-30T13:41:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc77d3640 1 -- start start 2023-10-30T13:41:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc77d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc00cf160 0x7f6fc0159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc77d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc01538a0 0x7f6fc00c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc77d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc015d6f0 0x7f6fc01e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc77d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc01e3bf0 con 0x7f6fc00cf160 2023-10-30T13:41:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc77d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc01580f0 con 0x7f6fc01538a0 2023-10-30T13:41:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc77d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc00c59a0 con 0x7f6fc015d6f0 2023-10-30T13:41:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc556f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc00cf160 0x7f6fc0159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc556f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc00cf160 0x7f6fc0159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51458/0 (socket says 172.21.15.46:51458) 2023-10-30T13:41:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc556f640 1 -- 172.21.15.46:0/1698726046 learned_addr learned my addr 172.21.15.46:0/1698726046 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc4d6e640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc01538a0 0x7f6fc00c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc5d70640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc015d6f0 0x7f6fc01e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc556f640 1 -- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc015d6f0 msgr2=0x7f6fc01e2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc556f640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc015d6f0 0x7f6fc01e2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc556f640 1 -- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc01538a0 msgr2=0x7f6fc00c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc556f640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc01538a0 0x7f6fc00c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc556f640 1 -- 172.21.15.46:0/1698726046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb00094d0 con 0x7f6fc00cf160 2023-10-30T13:41:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc4d6e640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc01538a0 0x7f6fc00c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc556f640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc00cf160 0x7f6fc0159300 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00c500 tx=0x7f6fbc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fb5ffb640 1 -- 172.21.15.46:0/1698726046 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc005220 con 0x7f6fc00cf160 2023-10-30T13:41:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fb5ffb640 1 -- 172.21.15.46:0/1698726046 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc005380 con 0x7f6fc00cf160 2023-10-30T13:41:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fb5ffb640 1 -- 172.21.15.46:0/1698726046 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc007270 con 0x7f6fc00cf160 2023-10-30T13:41:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc77d3640 1 -- 172.21.15.46:0/1698726046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc01584d0 con 0x7f6fc00cf160 2023-10-30T13:41:46.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.006+0000 7f6fc77d3640 1 -- 172.21.15.46:0/1698726046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc00c4f60 con 0x7f6fc00cf160 2023-10-30T13:41:46.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.010+0000 7f6fc77d3640 1 -- 172.21.15.46:0/1698726046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc00b2a50 con 0x7f6fc00cf160 2023-10-30T13:41:46.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.010+0000 7f6fb5ffb640 1 -- 172.21.15.46:0/1698726046 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6fbc0073d0 con 0x7f6fc00cf160 2023-10-30T13:41:46.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.010+0000 7f6fb5ffb640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6fa40664d0 0x7f6fa4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:46.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.010+0000 7f6fb5ffb640 1 -- 172.21.15.46:0/1698726046 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f6fbc083c30 con 0x7f6fc00cf160 2023-10-30T13:41:46.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.010+0000 7f6fc4d6e640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6fa40664d0 0x7f6fa4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:46.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.010+0000 7f6fc4d6e640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6fa40664d0 0x7f6fa4068990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00b2000 tx=0x7f6fb0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:46.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.014+0000 7f6fb5ffb640 1 -- 172.21.15.46:0/1698726046 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6fbc050c30 con 0x7f6fc00cf160 2023-10-30T13:41:46.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 -- 172.21.15.46:0/1698726046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fc004e170 con 0x7f6fc00cf160 2023-10-30T13:41:46.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fb5ffb640 1 -- 172.21.15.46:0/1698726046 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5749 (secure 0 0 0) 0x7f6fbc054250 con 0x7f6fc00cf160 2023-10-30T13:41:46.282 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:46.282 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:40.224427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:46.283 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 94 2023-10-30T13:41:46.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 -- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6fa40664d0 msgr2=0x7f6fa4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:46.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6fa40664d0 0x7f6fa4068990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00b2000 tx=0x7f6fb0008040 comp rx=0 tx=0).stop 2023-10-30T13:41:46.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 -- 172.21.15.46:0/1698726046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc00cf160 msgr2=0x7f6fc0159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:46.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc00cf160 0x7f6fc0159300 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00c500 tx=0x7f6fbc00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:41:46.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 -- 172.21.15.46:0/1698726046 shutdown_connections 2023-10-30T13:41:46.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6fa40664d0 0x7f6fa4068990 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc015d6f0 0x7f6fc01e2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc01538a0 0x7f6fc00c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 --2- 172.21.15.46:0/1698726046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc00cf160 0x7f6fc0159300 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 -- 172.21.15.46:0/1698726046 >> 172.21.15.46:0/1698726046 conn(0x7f6fc006bfe0 msgr2=0x7f6fc014ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 -- 172.21.15.46:0/1698726046 shutdown_connections 2023-10-30T13:41:46.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:46.278+0000 7f6fc77d3640 1 -- 172.21.15.46:0/1698726046 wait complete. 2023-10-30T13:41:46.303 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:41:48.304 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:48.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.450+0000 7fe1bde79640 1 Processor -- start 2023-10-30T13:41:48.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.450+0000 7fe1bde79640 1 -- start start 2023-10-30T13:41:48.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.450+0000 7fe1bde79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b80c05d0 0x7fe1b813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:48.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.450+0000 7fe1bde79640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1b80c6f60 0x7fe1b813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:48.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.450+0000 7fe1bde79640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b80c0a50 con 0x7fe1b80c6b80 2023-10-30T13:41:48.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.450+0000 7fe1bde79640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b806f6e0 con 0x7fe1b80c6f60 2023-10-30T13:41:48.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.450+0000 7fe1bde79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b806f9c0 con 0x7fe1b80c05d0 2023-10-30T13:41:48.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.450+0000 7fe1b7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1b80c6f60 0x7fe1b813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:48.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.450+0000 7fe1b7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1b80c6f60 0x7fe1b813f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36568/0 (socket says 172.21.15.46:36568) 2023-10-30T13:41:48.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.450+0000 7fe1b7fff640 1 -- 172.21.15.46:0/876964698 learned_addr learned my addr 172.21.15.46:0/876964698 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.450+0000 7fe1b6ffd640 1 --2- 172.21.15.46:0/876964698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b80c05d0 0x7fe1b813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.454+0000 7fe1b7fff640 1 -- 172.21.15.46:0/876964698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b80c05d0 msgr2=0x7fe1b813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.454+0000 7fe1b7fff640 1 --2- 172.21.15.46:0/876964698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b80c05d0 0x7fe1b813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:48.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.454+0000 7fe1b7fff640 1 -- 172.21.15.46:0/876964698 >> v1:172.21.15.154:6789/0 conn(0x7fe1b80c6b80 legacy=0x7fe1b80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:48.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.454+0000 7fe1b7fff640 1 -- 172.21.15.46:0/876964698 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b806fca0 con 0x7fe1b80c6f60 2023-10-30T13:41:48.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.454+0000 7fe1b6ffd640 1 --2- 172.21.15.46:0/876964698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b80c05d0 0x7fe1b813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:48.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.454+0000 7fe1b7fff640 1 --2- 172.21.15.46:0/876964698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b80c6f60 0x7fe1b813f2d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009e50 tx=0x7fe1b001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a158aedb051f4eff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:48.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.454+0000 7fe1b67fc640 1 -- 172.21.15.46:0/876964698 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b00a57f0 con 0x7fe1b80c6f60 2023-10-30T13:41:48.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.454+0000 7fe1b67fc640 1 -- 172.21.15.46:0/876964698 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b00a5950 con 0x7fe1b80c6f60 2023-10-30T13:41:48.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.454+0000 7fe1b67fc640 1 -- 172.21.15.46:0/876964698 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b00a5ce0 con 0x7fe1b80c6f60 2023-10-30T13:41:48.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.454+0000 7fe1bde79640 1 -- 172.21.15.46:0/876964698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b80c6f60 msgr2=0x7fe1b813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:48.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.454+0000 7fe1bde79640 1 --2- 172.21.15.46:0/876964698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b80c6f60 0x7fe1b813f2d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009e50 tx=0x7fe1b001ca10 comp rx=0 tx=0).stop 2023-10-30T13:41:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 -- 172.21.15.46:0/876964698 shutdown_connections 2023-10-30T13:41:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 --2- 172.21.15.46:0/876964698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b80c6f60 0x7fe1b813f2d0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 --2- 172.21.15.46:0/876964698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b80c05d0 0x7fe1b813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 -- 172.21.15.46:0/876964698 >> 172.21.15.46:0/876964698 conn(0x7fe1b80c7910 msgr2=0x7fe1b80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 -- 172.21.15.46:0/876964698 shutdown_connections 2023-10-30T13:41:48.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 -- 172.21.15.46:0/876964698 wait complete. 2023-10-30T13:41:48.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 Processor -- start 2023-10-30T13:41:48.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 -- start start 2023-10-30T13:41:48.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b80c05d0 0x7fe1b8136e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:48.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b80c6b80 0x7fe1b81f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:48.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b80c6f60 0x7fe1b8153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:48.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b8154010 con 0x7fe1b80c6b80 2023-10-30T13:41:48.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b81f3500 con 0x7fe1b80c6f60 2023-10-30T13:41:48.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1bde79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b80c3d60 con 0x7fe1b80c05d0 2023-10-30T13:41:48.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1b6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b80c6b80 0x7fe1b81f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:48.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1b6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b80c6b80 0x7fe1b81f3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36580/0 (socket says 172.21.15.46:36580) 2023-10-30T13:41:48.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1b6ffd640 1 -- 172.21.15.46:0/3023513960 learned_addr learned my addr 172.21.15.46:0/3023513960 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:48.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1b77fe640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b80c05d0 0x7fe1b8136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:48.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1b7fff640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b80c6f60 0x7fe1b8153ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1b6ffd640 1 -- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b80c05d0 msgr2=0x7fe1b8136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1b6ffd640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b80c05d0 0x7fe1b8136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1b6ffd640 1 -- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b80c6f60 msgr2=0x7fe1b8153ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1b6ffd640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b80c6f60 0x7fe1b8153ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:48.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1b6ffd640 1 -- 172.21.15.46:0/3023513960 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b0009b30 con 0x7fe1b80c6b80 2023-10-30T13:41:48.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.458+0000 7fe1b6ffd640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b80c6b80 0x7fe1b81f3990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00e810 tx=0x7fe1ac00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:48.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.462+0000 7fe19ffff640 1 -- 172.21.15.46:0/3023513960 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac00c870 con 0x7fe1b80c6b80 2023-10-30T13:41:48.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.462+0000 7fe19ffff640 1 -- 172.21.15.46:0/3023513960 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac00c9d0 con 0x7fe1b80c6b80 2023-10-30T13:41:48.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.462+0000 7fe19ffff640 1 -- 172.21.15.46:0/3023513960 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac00cd60 con 0x7fe1b80c6b80 2023-10-30T13:41:48.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.462+0000 7fe1bde79640 1 -- 172.21.15.46:0/3023513960 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b80c4550 con 0x7fe1b80c6b80 2023-10-30T13:41:48.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.462+0000 7fe1bde79640 1 -- 172.21.15.46:0/3023513960 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b80c4b10 con 0x7fe1b80c6b80 2023-10-30T13:41:48.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.462+0000 7fe1bde79640 1 -- 172.21.15.46:0/3023513960 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b804e170 con 0x7fe1b80c6b80 2023-10-30T13:41:48.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.462+0000 7fe19ffff640 1 -- 172.21.15.46:0/3023513960 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe1ac013410 con 0x7fe1b80c6b80 2023-10-30T13:41:48.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.462+0000 7fe19ffff640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe1a40664a0 0x7fe1a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:48.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.462+0000 7fe19ffff640 1 -- 172.21.15.46:0/3023513960 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fe1ac084060 con 0x7fe1b80c6b80 2023-10-30T13:41:48.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.462+0000 7fe1b77fe640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe1a40664a0 0x7fe1a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:48.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.466+0000 7fe1b77fe640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe1a40664a0 0x7fe1a4068960 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80ca340 tx=0x7fe1a8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:48.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.466+0000 7fe19ffff640 1 -- 172.21.15.46:0/3023513960 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe1ac051070 con 0x7fe1b80c6b80 2023-10-30T13:41:48.542 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:41:48.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:41:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.658+0000 7fe1bde79640 1 -- 172.21.15.46:0/3023513960 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1b80c64f0 con 0x7fe1b80c6b80 2023-10-30T13:41:48.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.658+0000 7fe19ffff640 1 -- 172.21.15.46:0/3023513960 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5743 (secure 0 0 0) 0x7fe1ac021090 con 0x7fe1b80c6b80 2023-10-30T13:41:48.663 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:48.664 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:46.201293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 95 2023-10-30T13:41:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.662+0000 7fe1bde79640 1 -- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe1a40664a0 msgr2=0x7fe1a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.662+0000 7fe1bde79640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe1a40664a0 0x7fe1a4068960 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80ca340 tx=0x7fe1a8009040 comp rx=0 tx=0).stop 2023-10-30T13:41:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.662+0000 7fe1bde79640 1 -- 172.21.15.46:0/3023513960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b80c6b80 msgr2=0x7fe1b81f3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.662+0000 7fe1bde79640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b80c6b80 0x7fe1b81f3990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00e810 tx=0x7fe1ac00ecd0 comp rx=0 tx=0).stop 2023-10-30T13:41:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.662+0000 7fe1bde79640 1 -- 172.21.15.46:0/3023513960 shutdown_connections 2023-10-30T13:41:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.662+0000 7fe1bde79640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe1a40664a0 0x7fe1a4068960 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.662+0000 7fe1bde79640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b80c6f60 0x7fe1b8153ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.662+0000 7fe1bde79640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b80c6b80 0x7fe1b81f3990 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.662+0000 7fe1bde79640 1 --2- 172.21.15.46:0/3023513960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b80c05d0 0x7fe1b8136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.662+0000 7fe1bde79640 1 -- 172.21.15.46:0/3023513960 >> 172.21.15.46:0/3023513960 conn(0x7fe1b80c7910 msgr2=0x7fe1b80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.662+0000 7fe1bde79640 1 -- 172.21.15.46:0/3023513960 shutdown_connections 2023-10-30T13:41:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:48.662+0000 7fe1bde79640 1 -- 172.21.15.46:0/3023513960 wait complete. 2023-10-30T13:41:48.685 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=5721 2023-10-30T13:41:48.686 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.a 2023-10-30T13:41:50.686 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:41:50.687 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:41:50.687 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:41:50.692 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:41:50.692 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:50.794 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-10-30T13:41:50.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f723386b640 1 Processor -- start 2023-10-30T13:41:50.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f723386b640 1 -- start start 2023-10-30T13:41:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f723386b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f722c070a00 0x7f722c070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f723386b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f722c069e90 0x7f722c06a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f723386b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f722c0713b0 con 0x7f722c1539a0 2023-10-30T13:41:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f723386b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722c06fbb0 con 0x7f722c070a00 2023-10-30T13:41:50.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f723386b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722c153d80 con 0x7f722c069e90 2023-10-30T13:41:50.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f7230e06640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f722c070a00 0x7f722c070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:50.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f7230e06640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f722c070a00 0x7f722c070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36602/0 (socket says 172.21.15.46:36602) 2023-10-30T13:41:50.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f7230e06640 1 -- 172.21.15.46:0/916502006 learned_addr learned my addr 172.21.15.46:0/916502006 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:50.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f7231e08640 1 --2- 172.21.15.46:0/916502006 >> v2:172.21.15.154:3300/0 conn(0x7f722c069e90 0x7f722c06a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:50.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f7230e06640 1 -- 172.21.15.46:0/916502006 >> v2:172.21.15.154:3300/0 conn(0x7f722c069e90 msgr2=0x7f722c06a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:50.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f7230e06640 1 --2- 172.21.15.46:0/916502006 >> v2:172.21.15.154:3300/0 conn(0x7f722c069e90 0x7f722c06a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:50.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f7230e06640 1 -- 172.21.15.46:0/916502006 >> v1:172.21.15.154:6789/0 conn(0x7f722c1539a0 legacy=0x7f722c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:50.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.849+0000 7f7230e06640 1 -- 172.21.15.46:0/916502006 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c0c5c50 con 0x7f722c070a00 2023-10-30T13:41:50.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.853+0000 7f7230e06640 1 --2- 172.21.15.46:0/916502006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f722c070a00 0x7f722c070e70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f722c0b1080 tx=0x7f721c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad458f652e7ee11e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:50.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.853+0000 7f7223fff640 1 -- 172.21.15.46:0/916502006 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721c004230 con 0x7f722c070a00 2023-10-30T13:41:50.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.853+0000 7f7223fff640 1 -- 172.21.15.46:0/916502006 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721c0a7050 con 0x7f722c070a00 2023-10-30T13:41:50.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.853+0000 7f7223fff640 1 -- 172.21.15.46:0/916502006 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721c003620 con 0x7f722c070a00 2023-10-30T13:41:50.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.853+0000 7f723386b640 1 -- 172.21.15.46:0/916502006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f722c070a00 msgr2=0x7f722c070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:50.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.853+0000 7f723386b640 1 --2- 172.21.15.46:0/916502006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f722c070a00 0x7f722c070e70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f722c0b1080 tx=0x7f721c01c440 comp rx=0 tx=0).stop 2023-10-30T13:41:50.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.853+0000 7f723386b640 1 -- 172.21.15.46:0/916502006 shutdown_connections 2023-10-30T13:41:50.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.853+0000 7f723386b640 1 --2- 172.21.15.46:0/916502006 >> v2:172.21.15.154:3300/0 conn(0x7f722c069e90 0x7f722c06a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:50.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.853+0000 7f723386b640 1 --2- 172.21.15.46:0/916502006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f722c070a00 0x7f722c070e70 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:50.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.853+0000 7f723386b640 1 -- 172.21.15.46:0/916502006 >> 172.21.15.46:0/916502006 conn(0x7f722c0c7910 msgr2=0x7f722c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:50.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.853+0000 7f723386b640 1 -- 172.21.15.46:0/916502006 shutdown_connections 2023-10-30T13:41:50.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.853+0000 7f723386b640 1 -- 172.21.15.46:0/916502006 wait complete. 2023-10-30T13:41:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f723386b640 1 Processor -- start 2023-10-30T13:41:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f723386b640 1 -- start start 2023-10-30T13:41:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f723386b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f722c069e90 0x7f722c157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f723386b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f722c070a00 0x7f722c0ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:50.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f723386b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f722c1539a0 0x7f722c13e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:50.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f723386b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c0c03b0 con 0x7f722c1539a0 2023-10-30T13:41:50.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f723386b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c0bfec0 con 0x7f722c070a00 2023-10-30T13:41:50.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f723386b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f722c0c8c10 con 0x7f722c069e90 2023-10-30T13:41:50.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7231e08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f722c1539a0 0x7f722c13e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:50.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7231e08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f722c1539a0 0x7f722c13e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36614/0 (socket says 172.21.15.46:36614) 2023-10-30T13:41:50.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7231e08640 1 -- 172.21.15.46:0/3727121893 learned_addr learned my addr 172.21.15.46:0/3727121893 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:50.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7230e06640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f722c070a00 0x7f722c0ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:50.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7231607640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f722c069e90 0x7f722c157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:50.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7231e08640 1 -- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f722c069e90 msgr2=0x7f722c157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:50.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7231e08640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f722c069e90 0x7f722c157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:50.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7231e08640 1 -- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f722c070a00 msgr2=0x7f722c0ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:50.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7231e08640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f722c070a00 0x7f722c0ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:50.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7231e08640 1 -- 172.21.15.46:0/3727121893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c0094d0 con 0x7f722c1539a0 2023-10-30T13:41:50.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7231e08640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f722c1539a0 0x7f722c13e460 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7224007e40 tx=0x7f722400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:50.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7221ffb640 1 -- 172.21.15.46:0/3727121893 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224014620 con 0x7f722c1539a0 2023-10-30T13:41:50.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7221ffb640 1 -- 172.21.15.46:0/3727121893 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722400ee90 con 0x7f722c1539a0 2023-10-30T13:41:50.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f7221ffb640 1 -- 172.21.15.46:0/3727121893 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72240149e0 con 0x7f722c1539a0 2023-10-30T13:41:50.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f723386b640 1 -- 172.21.15.46:0/3727121893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722c0c00d0 con 0x7f722c1539a0 2023-10-30T13:41:50.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f723386b640 1 -- 172.21.15.46:0/3727121893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722c1441d0 con 0x7f722c1539a0 2023-10-30T13:41:50.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.857+0000 7f723386b640 1 -- 172.21.15.46:0/3727121893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722c143d50 con 0x7f722c1539a0 2023-10-30T13:41:50.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.861+0000 7f7221ffb640 1 -- 172.21.15.46:0/3727121893 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7224014b40 con 0x7f722c1539a0 2023-10-30T13:41:50.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.861+0000 7f7221ffb640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72100664a0 0x7f7210068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:50.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.861+0000 7f7221ffb640 1 -- 172.21.15.46:0/3727121893 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f7224084ba0 con 0x7f722c1539a0 2023-10-30T13:41:50.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.861+0000 7f7231607640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72100664a0 0x7f7210068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:50.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.861+0000 7f7231607640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72100664a0 0x7f7210068960 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7228000f70 tx=0x7f7228009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:50.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:50.865+0000 7f7221ffb640 1 -- 172.21.15.46:0/3727121893 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f722c143d50 con 0x7f722c1539a0 2023-10-30T13:41:51.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.057+0000 7f723386b640 1 -- 172.21.15.46:0/3727121893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f722c143d50 con 0x7f722c1539a0 2023-10-30T13:41:51.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.057+0000 7f7221ffb640 1 -- 172.21.15.46:0/3727121893 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f722c143d50 con 0x7f722c1539a0 2023-10-30T13:41:51.063 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:51.064 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:46.201293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:51.064 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 95 2023-10-30T13:41:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.061+0000 7f723386b640 1 -- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72100664a0 msgr2=0x7f7210068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.061+0000 7f723386b640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72100664a0 0x7f7210068960 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7228000f70 tx=0x7f7228009040 comp rx=0 tx=0).stop 2023-10-30T13:41:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.061+0000 7f723386b640 1 -- 172.21.15.46:0/3727121893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f722c1539a0 msgr2=0x7f722c13e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.061+0000 7f723386b640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f722c1539a0 0x7f722c13e460 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7224007e40 tx=0x7f722400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:41:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.061+0000 7f723386b640 1 -- 172.21.15.46:0/3727121893 shutdown_connections 2023-10-30T13:41:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.061+0000 7f723386b640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72100664a0 0x7f7210068960 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.061+0000 7f723386b640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f722c1539a0 0x7f722c13e460 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.061+0000 7f723386b640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f722c070a00 0x7f722c0ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.061+0000 7f723386b640 1 --2- 172.21.15.46:0/3727121893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f722c069e90 0x7f722c157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.061+0000 7f723386b640 1 -- 172.21.15.46:0/3727121893 >> 172.21.15.46:0/3727121893 conn(0x7f722c0c7910 msgr2=0x7f722c14fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.061+0000 7f723386b640 1 -- 172.21.15.46:0/3727121893 shutdown_connections 2023-10-30T13:41:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:51.061+0000 7f723386b640 1 -- 172.21.15.46:0/3727121893 wait complete. 2023-10-30T13:41:51.084 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T13:41:52.243 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T13:41:52.237+0000 7f3d2c233640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:41:52.243 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T13:41:52.237+0000 7f3d2c233640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:41:52.243 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T13:41:52.237+0000 7f3d2c233640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:41:53.085 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:41:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.237+0000 7efcfe71d640 1 Processor -- start 2023-10-30T13:41:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.237+0000 7efcfe71d640 1 -- start start 2023-10-30T13:41:53.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.237+0000 7efcfe71d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efcf80c6fe0 0x7efcf8071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:53.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.237+0000 7efcfe71d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efcf80c5f60 0x7efcf80cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:53.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.237+0000 7efcfe71d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf80700b0 con 0x7efcf8153bf0 2023-10-30T13:41:53.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.237+0000 7efcfe71d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf80c6810 con 0x7efcf80c6fe0 2023-10-30T13:41:53.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.237+0000 7efcfe71d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf8159900 con 0x7efcf80c5f60 2023-10-30T13:41:53.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.237+0000 7efcf77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efcf80c6fe0 0x7efcf8071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:53.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.237+0000 7efcf77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efcf80c6fe0 0x7efcf8071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36616/0 (socket says 172.21.15.46:36616) 2023-10-30T13:41:53.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.237+0000 7efcf77fe640 1 -- 172.21.15.46:0/1999353321 learned_addr learned my addr 172.21.15.46:0/1999353321 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:53.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.237+0000 7efcfccba640 1 --2- 172.21.15.46:0/1999353321 >> v2:172.21.15.154:3300/0 conn(0x7efcf80c5f60 0x7efcf80cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:53.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcf77fe640 1 -- 172.21.15.46:0/1999353321 >> v2:172.21.15.154:3300/0 conn(0x7efcf80c5f60 msgr2=0x7efcf80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:53.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcf77fe640 1 --2- 172.21.15.46:0/1999353321 >> v2:172.21.15.154:3300/0 conn(0x7efcf80c5f60 0x7efcf80cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:53.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcf77fe640 1 -- 172.21.15.46:0/1999353321 >> v1:172.21.15.154:6789/0 conn(0x7efcf8153bf0 legacy=0x7efcf80c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:41:53.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcf77fe640 1 -- 172.21.15.46:0/1999353321 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf80c6530 con 0x7efcf80c6fe0 2023-10-30T13:41:53.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcf77fe640 1 --2- 172.21.15.46:0/1999353321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcf80c6fe0 0x7efcf8071260 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7efcf806ce90 tx=0x7efcec01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a2a0e7f95e30a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:53.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcf6ffd640 1 -- 172.21.15.46:0/1999353321 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec0a89f0 con 0x7efcf80c6fe0 2023-10-30T13:41:53.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcf6ffd640 1 -- 172.21.15.46:0/1999353321 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcec0a8b50 con 0x7efcf80c6fe0 2023-10-30T13:41:53.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcf6ffd640 1 -- 172.21.15.46:0/1999353321 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec0b0440 con 0x7efcf80c6fe0 2023-10-30T13:41:53.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcfe71d640 1 -- 172.21.15.46:0/1999353321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcf80c6fe0 msgr2=0x7efcf8071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:53.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcfe71d640 1 --2- 172.21.15.46:0/1999353321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcf80c6fe0 0x7efcf8071260 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7efcf806ce90 tx=0x7efcec01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:41:53.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcfe71d640 1 -- 172.21.15.46:0/1999353321 shutdown_connections 2023-10-30T13:41:53.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcfe71d640 1 --2- 172.21.15.46:0/1999353321 >> v2:172.21.15.154:3300/0 conn(0x7efcf80c5f60 0x7efcf80cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:53.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcfe71d640 1 --2- 172.21.15.46:0/1999353321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcf80c6fe0 0x7efcf8071260 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:53.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.241+0000 7efcfe71d640 1 -- 172.21.15.46:0/1999353321 >> 172.21.15.46:0/1999353321 conn(0x7efcf806c6e0 msgr2=0x7efcf8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:53.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcfe71d640 1 -- 172.21.15.46:0/1999353321 shutdown_connections 2023-10-30T13:41:53.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcfe71d640 1 -- 172.21.15.46:0/1999353321 wait complete. 2023-10-30T13:41:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcfe71d640 1 Processor -- start 2023-10-30T13:41:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcfe71d640 1 -- start start 2023-10-30T13:41:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcfe71d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcf80c5f60 0x7efcf8154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcfe71d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcf80c6fe0 0x7efcf814c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcfe71d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efcf8153bf0 0x7efcf8155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:53.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcfe71d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf814e860 con 0x7efcf80c6fe0 2023-10-30T13:41:53.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcfe71d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf8152fa0 con 0x7efcf80c5f60 2023-10-30T13:41:53.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcfe71d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf80c5740 con 0x7efcf8153bf0 2023-10-30T13:41:53.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcf77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcf80c6fe0 0x7efcf814c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:53.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcf77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcf80c6fe0 0x7efcf814c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36632/0 (socket says 172.21.15.46:36632) 2023-10-30T13:41:53.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcf77fe640 1 -- 172.21.15.46:0/3688174709 learned_addr learned my addr 172.21.15.46:0/3688174709 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:41:53.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcf7fff640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcf80c5f60 0x7efcf8154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:53.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcfccba640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efcf8153bf0 0x7efcf8155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:53.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcf77fe640 1 -- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efcf8153bf0 msgr2=0x7efcf8155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:53.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcf77fe640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efcf8153bf0 0x7efcf8155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:53.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcf77fe640 1 -- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcf80c5f60 msgr2=0x7efcf8154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcf77fe640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcf80c5f60 0x7efcf8154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcf77fe640 1 -- 172.21.15.46:0/3688174709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0094d0 con 0x7efcf80c6fe0 2023-10-30T13:41:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcf7fff640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcf80c5f60 0x7efcf8154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:41:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.245+0000 7efcf77fe640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcf80c6fe0 0x7efcf814c000 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7efcec01b900 tx=0x7efcec01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:53.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.249+0000 7efcf4ff9640 1 -- 172.21.15.46:0/3688174709 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec0ba600 con 0x7efcf80c6fe0 2023-10-30T13:41:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.249+0000 7efcf4ff9640 1 -- 172.21.15.46:0/3688174709 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcec0ba760 con 0x7efcf80c6fe0 2023-10-30T13:41:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.249+0000 7efcf4ff9640 1 -- 172.21.15.46:0/3688174709 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec0bab10 con 0x7efcf80c6fe0 2023-10-30T13:41:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.249+0000 7efcfe71d640 1 -- 172.21.15.46:0/3688174709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf814eaa0 con 0x7efcf80c6fe0 2023-10-30T13:41:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.249+0000 7efcfe71d640 1 -- 172.21.15.46:0/3688174709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf814f060 con 0x7efcf80c6fe0 2023-10-30T13:41:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.249+0000 7efcfe71d640 1 -- 172.21.15.46:0/3688174709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf804e170 con 0x7efcf80c6fe0 2023-10-30T13:41:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.249+0000 7efcf4ff9640 1 -- 172.21.15.46:0/3688174709 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efcec0bac70 con 0x7efcf80c6fe0 2023-10-30T13:41:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.249+0000 7efcf4ff9640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efce40664a0 0x7efce4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:41:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.249+0000 7efcf4ff9640 1 -- 172.21.15.46:0/3688174709 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7efcec1228d0 con 0x7efcf80c6fe0 2023-10-30T13:41:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.249+0000 7efcf7fff640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efce40664a0 0x7efce4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:41:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.253+0000 7efcf7fff640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efce40664a0 0x7efce4068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efce8000f70 tx=0x7efce8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:41:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.253+0000 7efcf4ff9640 1 -- 172.21.15.46:0/3688174709 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efcec0ef8c0 con 0x7efcf80c6fe0 2023-10-30T13:41:53.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.453+0000 7efcfe71d640 1 -- 172.21.15.46:0/3688174709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcf806fe90 con 0x7efcf80c6fe0 2023-10-30T13:41:53.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.453+0000 7efcf4ff9640 1 -- 172.21.15.46:0/3688174709 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6531 (secure 0 0 0) 0x7efcec0b5090 con 0x7efcf80c6fe0 2023-10-30T13:41:53.458 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:41:53.458 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:46.201293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:41:53.459 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 96 2023-10-30T13:41:53.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.453+0000 7efcfe71d640 1 -- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efce40664a0 msgr2=0x7efce4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:53.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.453+0000 7efcfe71d640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efce40664a0 0x7efce4068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efce8000f70 tx=0x7efce8009040 comp rx=0 tx=0).stop 2023-10-30T13:41:53.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.453+0000 7efcfe71d640 1 -- 172.21.15.46:0/3688174709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcf80c6fe0 msgr2=0x7efcf814c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:41:53.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.453+0000 7efcfe71d640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcf80c6fe0 0x7efcf814c000 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7efcec01b900 tx=0x7efcec01ba90 comp rx=0 tx=0).stop 2023-10-30T13:41:53.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.453+0000 7efcfe71d640 1 -- 172.21.15.46:0/3688174709 shutdown_connections 2023-10-30T13:41:53.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.453+0000 7efcfe71d640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efce40664a0 0x7efce4068960 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:53.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.453+0000 7efcfe71d640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efcf8153bf0 0x7efcf8155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:53.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.453+0000 7efcfe71d640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcf80c6fe0 0x7efcf814c000 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:53.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.453+0000 7efcfe71d640 1 --2- 172.21.15.46:0/3688174709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcf80c5f60 0x7efcf8154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:41:53.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.453+0000 7efcfe71d640 1 -- 172.21.15.46:0/3688174709 >> 172.21.15.46:0/3688174709 conn(0x7efcf806c6e0 msgr2=0x7efcf8158560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:41:53.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.457+0000 7efcfe71d640 1 -- 172.21.15.46:0/3688174709 shutdown_connections 2023-10-30T13:41:53.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:41:53.457+0000 7efcfe71d640 1 -- 172.21.15.46:0/3688174709 wait complete. 2023-10-30T13:41:53.480 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:41:53.480 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 10 secs before thrashing 2023-10-30T13:42:03.481 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:03.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.629+0000 7fb43833e640 1 Processor -- start 2023-10-30T13:42:03.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.629+0000 7fb43833e640 1 -- start start 2023-10-30T13:42:03.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.629+0000 7fb43833e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4300c05d0 0x7fb43013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:03.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.629+0000 7fb43833e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb4300c6f60 0x7fb43013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.629+0000 7fb43833e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4300c0a50 con 0x7fb4300c6b80 2023-10-30T13:42:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.629+0000 7fb43833e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43006f6e0 con 0x7fb4300c05d0 2023-10-30T13:42:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.629+0000 7fb43833e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43006f9c0 con 0x7fb4300c6f60 2023-10-30T13:42:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.633+0000 7fb4358d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4300c05d0 0x7fb43013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.633+0000 7fb4358d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4300c05d0 0x7fb43013ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32886/0 (socket says 172.21.15.46:32886) 2023-10-30T13:42:03.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.633+0000 7fb4358d9640 1 -- 172.21.15.46:0/3600021730 learned_addr learned my addr 172.21.15.46:0/3600021730 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.633+0000 7fb4368db640 1 --2- 172.21.15.46:0/3600021730 >> v2:172.21.15.154:3300/0 conn(0x7fb4300c6f60 0x7fb43013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.633+0000 7fb4358d9640 1 -- 172.21.15.46:0/3600021730 >> v2:172.21.15.154:3300/0 conn(0x7fb4300c6f60 msgr2=0x7fb43013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:03.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.633+0000 7fb4358d9640 1 --2- 172.21.15.46:0/3600021730 >> v2:172.21.15.154:3300/0 conn(0x7fb4300c6f60 0x7fb43013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.633+0000 7fb4358d9640 1 -- 172.21.15.46:0/3600021730 >> v1:172.21.15.154:6789/0 conn(0x7fb4300c6b80 legacy=0x7fb4300bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.633+0000 7fb4358d9640 1 -- 172.21.15.46:0/3600021730 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43006fca0 con 0x7fb4300c05d0 2023-10-30T13:42:03.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.633+0000 7fb4358d9640 1 --2- 172.21.15.46:0/3600021730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4300c05d0 0x7fb43013ed90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb4300c89b0 tx=0x7fb42001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb7648c5977a955 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.633+0000 7fb4350d8640 1 -- 172.21.15.46:0/3600021730 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4200ab610 con 0x7fb4300c05d0 2023-10-30T13:42:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.633+0000 7fb4350d8640 1 -- 172.21.15.46:0/3600021730 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4200ab770 con 0x7fb4300c05d0 2023-10-30T13:42:03.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.633+0000 7fb4350d8640 1 -- 172.21.15.46:0/3600021730 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4200abb00 con 0x7fb4300c05d0 2023-10-30T13:42:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 -- 172.21.15.46:0/3600021730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4300c05d0 msgr2=0x7fb43013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 --2- 172.21.15.46:0/3600021730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4300c05d0 0x7fb43013ed90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb4300c89b0 tx=0x7fb42001c500 comp rx=0 tx=0).stop 2023-10-30T13:42:03.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 -- 172.21.15.46:0/3600021730 shutdown_connections 2023-10-30T13:42:03.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 --2- 172.21.15.46:0/3600021730 >> v2:172.21.15.154:3300/0 conn(0x7fb4300c6f60 0x7fb43013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:03.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 --2- 172.21.15.46:0/3600021730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4300c05d0 0x7fb43013ed90 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:03.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 -- 172.21.15.46:0/3600021730 >> 172.21.15.46:0/3600021730 conn(0x7fb4300c7910 msgr2=0x7fb4300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:03.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 -- 172.21.15.46:0/3600021730 shutdown_connections 2023-10-30T13:42:03.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 -- 172.21.15.46:0/3600021730 wait complete. 2023-10-30T13:42:03.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 Processor -- start 2023-10-30T13:42:03.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 -- start start 2023-10-30T13:42:03.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4300c05d0 0x7fb4301494e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:03.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4300c6b80 0x7fb43014ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:03.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4300c6f60 0x7fb43014f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:03.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4301589f0 con 0x7fb4300c05d0 2023-10-30T13:42:03.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb430131670 con 0x7fb4300c6f60 2023-10-30T13:42:03.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb43833e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4300c1b30 con 0x7fb4300c6b80 2023-10-30T13:42:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb4360da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4300c05d0 0x7fb4301494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb4360da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4300c05d0 0x7fb4301494e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32888/0 (socket says 172.21.15.46:32888) 2023-10-30T13:42:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb4360da640 1 -- 172.21.15.46:0/3262617665 learned_addr learned my addr 172.21.15.46:0/3262617665 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb4358d9640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4300c6b80 0x7fb43014ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:03.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.637+0000 7fb4368db640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4300c6f60 0x7fb43014f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb4360da640 1 -- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4300c6b80 msgr2=0x7fb43014ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb4360da640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4300c6b80 0x7fb43014ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb4360da640 1 -- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4300c6f60 msgr2=0x7fb43014f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb4360da640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4300c6f60 0x7fb43014f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:03.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb4360da640 1 -- 172.21.15.46:0/3262617665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4200094d0 con 0x7fb4300c05d0 2023-10-30T13:42:03.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb4360da640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4300c05d0 0x7fb4301494e0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00c500 tx=0x7fb42c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:03.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb426ffd640 1 -- 172.21.15.46:0/3262617665 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c005220 con 0x7fb4300c05d0 2023-10-30T13:42:03.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb426ffd640 1 -- 172.21.15.46:0/3262617665 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c005380 con 0x7fb4300c05d0 2023-10-30T13:42:03.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb43833e640 1 -- 172.21.15.46:0/3262617665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb430131910 con 0x7fb4300c05d0 2023-10-30T13:42:03.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb426ffd640 1 -- 172.21.15.46:0/3262617665 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c007270 con 0x7fb4300c05d0 2023-10-30T13:42:03.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb43833e640 1 -- 172.21.15.46:0/3262617665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4301f3400 con 0x7fb4300c05d0 2023-10-30T13:42:03.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb43833e640 1 -- 172.21.15.46:0/3262617665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43004e170 con 0x7fb4300c05d0 2023-10-30T13:42:03.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb426ffd640 1 -- 172.21.15.46:0/3262617665 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb42c0073d0 con 0x7fb4300c05d0 2023-10-30T13:42:03.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb426ffd640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4140664d0 0x7fb414068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:03.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb426ffd640 1 -- 172.21.15.46:0/3262617665 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fb42c083b00 con 0x7fb4300c05d0 2023-10-30T13:42:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.641+0000 7fb4358d9640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4140664d0 0x7fb414068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.645+0000 7fb4358d9640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4140664d0 0x7fb414068990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb4200af000 tx=0x7fb4200a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.645+0000 7fb426ffd640 1 -- 172.21.15.46:0/3262617665 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb42c050ba0 con 0x7fb4300c05d0 2023-10-30T13:42:03.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 -- 172.21.15.46:0/3262617665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb430131090 con 0x7fb4300c05d0 2023-10-30T13:42:03.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb426ffd640 1 -- 172.21.15.46:0/3262617665 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fb42c0541c0 con 0x7fb4300c05d0 2023-10-30T13:42:03.843 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:03.843 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:41:46.201293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 96 2023-10-30T13:42:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 -- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4140664d0 msgr2=0x7fb414068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:03.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4140664d0 0x7fb414068990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb4200af000 tx=0x7fb4200a7000 comp rx=0 tx=0).stop 2023-10-30T13:42:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 -- 172.21.15.46:0/3262617665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4300c05d0 msgr2=0x7fb4301494e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4300c05d0 0x7fb4301494e0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00c500 tx=0x7fb42c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:42:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 -- 172.21.15.46:0/3262617665 shutdown_connections 2023-10-30T13:42:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4140664d0 0x7fb414068990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:03.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4300c6f60 0x7fb43014f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4300c6b80 0x7fb43014ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 --2- 172.21.15.46:0/3262617665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4300c05d0 0x7fb4301494e0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 -- 172.21.15.46:0/3262617665 >> 172.21.15.46:0/3262617665 conn(0x7fb4300c7910 msgr2=0x7fb4301302f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 -- 172.21.15.46:0/3262617665 shutdown_connections 2023-10-30T13:42:03.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:03.837+0000 7fb43833e640 1 -- 172.21.15.46:0/3262617665 wait complete. 2023-10-30T13:42:03.863 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 3 -> 2 2023-10-30T13:42:03.863 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2023-10-30T13:42:04.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e8217640 1 Processor -- start 2023-10-30T13:42:04.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e8217640 1 -- start start 2023-10-30T13:42:04.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e8217640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f02e01508c0 0x7f02e00c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:04.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e8217640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02e00722a0 0x7f02e00c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:04.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e8217640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e00c0a30 con 0x7f02e00bfec0 2023-10-30T13:42:04.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e8217640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e00c85b0 con 0x7f02e00722a0 2023-10-30T13:42:04.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e8217640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e006f650 con 0x7f02e01508c0 2023-10-30T13:42:04.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e57b2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02e00722a0 0x7f02e00c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:04.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e57b2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02e00722a0 0x7f02e00c6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32900/0 (socket says 172.21.15.46:32900) 2023-10-30T13:42:04.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e57b2640 1 -- 172.21.15.46:0/2779458230 learned_addr learned my addr 172.21.15.46:0/2779458230 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:04.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e5fb3640 1 --2- 172.21.15.46:0/2779458230 >> v2:172.21.15.154:3300/0 conn(0x7f02e01508c0 0x7f02e00c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:04.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e57b2640 1 -- 172.21.15.46:0/2779458230 >> v2:172.21.15.154:3300/0 conn(0x7f02e01508c0 msgr2=0x7f02e00c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:04.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e57b2640 1 --2- 172.21.15.46:0/2779458230 >> v2:172.21.15.154:3300/0 conn(0x7f02e01508c0 0x7f02e00c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e57b2640 1 -- 172.21.15.46:0/2779458230 >> v1:172.21.15.154:6789/0 conn(0x7f02e00bfec0 legacy=0x7f02e0156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:04.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.005+0000 7f02e57b2640 1 -- 172.21.15.46:0/2779458230 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e006f930 con 0x7f02e00722a0 2023-10-30T13:42:04.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.009+0000 7f02e57b2640 1 --2- 172.21.15.46:0/2779458230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02e00722a0 0x7f02e00c6e60 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f02e00c7540 tx=0x7f02d001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce04ca3f00e41706 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:04.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.009+0000 7f02e4fb1640 1 -- 172.21.15.46:0/2779458230 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d00ad660 con 0x7f02e00722a0 2023-10-30T13:42:04.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.009+0000 7f02e4fb1640 1 -- 172.21.15.46:0/2779458230 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d00ad7c0 con 0x7f02e00722a0 2023-10-30T13:42:04.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.009+0000 7f02e4fb1640 1 -- 172.21.15.46:0/2779458230 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d00adb50 con 0x7f02e00722a0 2023-10-30T13:42:04.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.009+0000 7f02e8217640 1 -- 172.21.15.46:0/2779458230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02e00722a0 msgr2=0x7f02e00c6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:04.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.009+0000 7f02e8217640 1 --2- 172.21.15.46:0/2779458230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02e00722a0 0x7f02e00c6e60 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f02e00c7540 tx=0x7f02d001c470 comp rx=0 tx=0).stop 2023-10-30T13:42:04.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.009+0000 7f02e8217640 1 -- 172.21.15.46:0/2779458230 shutdown_connections 2023-10-30T13:42:04.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.009+0000 7f02e8217640 1 --2- 172.21.15.46:0/2779458230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02e00722a0 0x7f02e00c6e60 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.009+0000 7f02e8217640 1 --2- 172.21.15.46:0/2779458230 >> v2:172.21.15.154:3300/0 conn(0x7f02e01508c0 0x7f02e00c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.009+0000 7f02e8217640 1 -- 172.21.15.46:0/2779458230 >> 172.21.15.46:0/2779458230 conn(0x7f02e00c7910 msgr2=0x7f02e00c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:04.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.009+0000 7f02e8217640 1 -- 172.21.15.46:0/2779458230 shutdown_connections 2023-10-30T13:42:04.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.009+0000 7f02e8217640 1 -- 172.21.15.46:0/2779458230 wait complete. 2023-10-30T13:42:04.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e8217640 1 Processor -- start 2023-10-30T13:42:04.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e8217640 1 -- start start 2023-10-30T13:42:04.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e8217640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02e00722a0 0x7f02e01f42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:04.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e8217640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02e00bfec0 0x7f02e0153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:04.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e8217640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e01508c0 0x7f02e00c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:04.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e8217640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e01f4810 con 0x7f02e00722a0 2023-10-30T13:42:04.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e8217640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e014c2b0 con 0x7f02e01508c0 2023-10-30T13:42:04.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e8217640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e014c570 con 0x7f02e00bfec0 2023-10-30T13:42:04.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e5fb3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02e00722a0 0x7f02e01f42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:04.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e5fb3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02e00722a0 0x7f02e01f42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32914/0 (socket says 172.21.15.46:32914) 2023-10-30T13:42:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e5fb3640 1 -- 172.21.15.46:0/825418592 learned_addr learned my addr 172.21.15.46:0/825418592 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e57b2640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02e00bfec0 0x7f02e0153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e67b4640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e01508c0 0x7f02e00c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:04.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e5fb3640 1 -- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02e00bfec0 msgr2=0x7f02e0153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:04.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e5fb3640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02e00bfec0 0x7f02e0153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e5fb3640 1 -- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e01508c0 msgr2=0x7f02e00c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:04.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e5fb3640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e01508c0 0x7f02e00c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:04.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e5fb3640 1 -- 172.21.15.46:0/825418592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d00094d0 con 0x7f02e00722a0 2023-10-30T13:42:04.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e5fb3640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02e00722a0 0x7f02e01f42d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00e870 tx=0x7f02dc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:04.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02d67fc640 1 -- 172.21.15.46:0/825418592 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc0095d0 con 0x7f02e00722a0 2023-10-30T13:42:04.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02d67fc640 1 -- 172.21.15.46:0/825418592 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02dc00c940 con 0x7f02e00722a0 2023-10-30T13:42:04.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02d67fc640 1 -- 172.21.15.46:0/825418592 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc00ccb0 con 0x7f02e00722a0 2023-10-30T13:42:04.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e8217640 1 -- 172.21.15.46:0/825418592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e014c850 con 0x7f02e00722a0 2023-10-30T13:42:04.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.013+0000 7f02e8217640 1 -- 172.21.15.46:0/825418592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e014b670 con 0x7f02e00722a0 2023-10-30T13:42:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.017+0000 7f02e8217640 1 -- 172.21.15.46:0/825418592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e014b1f0 con 0x7f02e00722a0 2023-10-30T13:42:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.017+0000 7f02d67fc640 1 -- 172.21.15.46:0/825418592 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02dc013470 con 0x7f02e00722a0 2023-10-30T13:42:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.017+0000 7f02d67fc640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02c40664d0 0x7f02c4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.017+0000 7f02d67fc640 1 -- 172.21.15.46:0/825418592 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f02dc084890 con 0x7f02e00722a0 2023-10-30T13:42:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.017+0000 7f02e57b2640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02c40664d0 0x7f02c4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.017+0000 7f02e57b2640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02c40664d0 0x7f02c4068990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f02d00af000 tx=0x7f02d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.021+0000 7f02d67fc640 1 -- 172.21.15.46:0/825418592 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f02e014b1f0 con 0x7f02e00722a0 2023-10-30T13:42:04.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:04.217+0000 7f02e8217640 1 -- 172.21.15.46:0/825418592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f02e014b1f0 con 0x7f02e00722a0 2023-10-30T13:42:05.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.181+0000 7f02d67fc640 1 -- 172.21.15.46:0/825418592 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v97) v1 ==== 105+0+0 (secure 0 0 0) 0x7f02e014b1f0 con 0x7f02e00722a0 2023-10-30T13:42:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.185+0000 7f02e8217640 1 -- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02c40664d0 msgr2=0x7f02c4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.185+0000 7f02e8217640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02c40664d0 0x7f02c4068990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f02d00af000 tx=0x7f02d0008040 comp rx=0 tx=0).stop 2023-10-30T13:42:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.185+0000 7f02e8217640 1 -- 172.21.15.46:0/825418592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02e00722a0 msgr2=0x7f02e01f42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.185+0000 7f02e8217640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02e00722a0 0x7f02e01f42d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00e870 tx=0x7f02dc00ed30 comp rx=0 tx=0).stop 2023-10-30T13:42:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.185+0000 7f02e8217640 1 -- 172.21.15.46:0/825418592 shutdown_connections 2023-10-30T13:42:05.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.185+0000 7f02e8217640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f02c40664d0 0x7f02c4068990 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:05.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.185+0000 7f02e8217640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e01508c0 0x7f02e00c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:05.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.185+0000 7f02e8217640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02e00bfec0 0x7f02e0153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:05.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.185+0000 7f02e8217640 1 --2- 172.21.15.46:0/825418592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02e00722a0 0x7f02e01f42d0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:05.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.185+0000 7f02e8217640 1 -- 172.21.15.46:0/825418592 >> 172.21.15.46:0/825418592 conn(0x7f02e00c7910 msgr2=0x7f02e006d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:05.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.185+0000 7f02e8217640 1 -- 172.21.15.46:0/825418592 shutdown_connections 2023-10-30T13:42:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.185+0000 7f02e8217640 1 -- 172.21.15.46:0/825418592 wait complete. 2023-10-30T13:42:05.213 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:42:05.214 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:05.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.357+0000 7f2478d65640 1 Processor -- start 2023-10-30T13:42:05.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.357+0000 7f2478d65640 1 -- start start 2023-10-30T13:42:05.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.357+0000 7f2478d65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24741538a0 0x7f247415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:05.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.357+0000 7f2478d65640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24740cb800 con 0x7f24740ca0e0 2023-10-30T13:42:05.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.357+0000 7f2478d65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247415dc30 con 0x7f24740cf160 2023-10-30T13:42:05.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.357+0000 7f2478d65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24740cb330 con 0x7f24741538a0 2023-10-30T13:42:05.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.357+0000 7f247259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24741538a0 0x7f247415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:05.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.357+0000 7f247259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24741538a0 0x7f247415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32922/0 (socket says 172.21.15.46:32922) 2023-10-30T13:42:05.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.357+0000 7f247259c640 1 -- 172.21.15.46:0/788937605 learned_addr learned my addr 172.21.15.46:0/788937605 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:05.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.357+0000 7f247259c640 1 -- 172.21.15.46:0/788937605 >> v1:172.21.15.154:6789/0 conn(0x7f24740cf160 legacy=0x7f2474164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:05.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.357+0000 7f247259c640 1 -- 172.21.15.46:0/788937605 >> v1:172.21.15.46:6789/0 conn(0x7f24740ca0e0 legacy=0x7f247415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:05.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.357+0000 7f247259c640 1 -- 172.21.15.46:0/788937605 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24740cad70 con 0x7f24741538a0 2023-10-30T13:42:05.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.361+0000 7f247259c640 1 --2- 172.21.15.46:0/788937605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24741538a0 0x7f247415d6f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f2464009870 tx=0x7f246401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0253161333fb32 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.361+0000 7f2471d9b640 1 -- 172.21.15.46:0/788937605 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24640a89f0 con 0x7f24741538a0 2023-10-30T13:42:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.361+0000 7f2471d9b640 1 -- 172.21.15.46:0/788937605 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24640a8b50 con 0x7f24741538a0 2023-10-30T13:42:05.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.361+0000 7f2471d9b640 1 -- 172.21.15.46:0/788937605 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24640b0440 con 0x7f24741538a0 2023-10-30T13:42:05.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.361+0000 7f2478d65640 1 -- 172.21.15.46:0/788937605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24741538a0 msgr2=0x7f247415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:05.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.361+0000 7f2478d65640 1 --2- 172.21.15.46:0/788937605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24741538a0 0x7f247415d6f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f2464009870 tx=0x7f246401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:42:05.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.361+0000 7f2478d65640 1 -- 172.21.15.46:0/788937605 shutdown_connections 2023-10-30T13:42:05.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.361+0000 7f2478d65640 1 --2- 172.21.15.46:0/788937605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24741538a0 0x7f247415d6f0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:05.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.361+0000 7f2478d65640 1 -- 172.21.15.46:0/788937605 >> 172.21.15.46:0/788937605 conn(0x7f247406bfe0 msgr2=0x7f247406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:05.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.361+0000 7f2478d65640 1 -- 172.21.15.46:0/788937605 shutdown_connections 2023-10-30T13:42:05.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.361+0000 7f2478d65640 1 -- 172.21.15.46:0/788937605 wait complete. 2023-10-30T13:42:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f2478d65640 1 Processor -- start 2023-10-30T13:42:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f2478d65640 1 -- start start 2023-10-30T13:42:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f2478d65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24740ca0e0 0x7f24741f3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f2478d65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24740cf160 0x7f24741e2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:05.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f2478d65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24741538a0 0x7f24741e5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:05.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f2478d65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741e5970 con 0x7f24740ca0e0 2023-10-30T13:42:05.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f2478d65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741e3b90 con 0x7f24741538a0 2023-10-30T13:42:05.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f2478d65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24741590f0 con 0x7f24740cf160 2023-10-30T13:42:05.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f247259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24740ca0e0 0x7f24741f3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:05.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f247259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24740ca0e0 0x7f24741f3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32932/0 (socket says 172.21.15.46:32932) 2023-10-30T13:42:05.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f247259c640 1 -- 172.21.15.46:0/3664878884 learned_addr learned my addr 172.21.15.46:0/3664878884 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:05.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f246bfff640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24740cf160 0x7f24741e2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:05.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f2472d9d640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24741538a0 0x7f24741e5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:05.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f247259c640 1 -- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24740cf160 msgr2=0x7f24741e2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:05.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f247259c640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24740cf160 0x7f24741e2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:05.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f247259c640 1 -- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24741538a0 msgr2=0x7f24741e5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:05.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f247259c640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24741538a0 0x7f24741e5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:05.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f247259c640 1 -- 172.21.15.46:0/3664878884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24640094d0 con 0x7f24740ca0e0 2023-10-30T13:42:05.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f247259c640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24740ca0e0 0x7f24741f3b10 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f24640b3040 tx=0x7f246401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:05.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f246affd640 1 -- 172.21.15.46:0/3664878884 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24640a89f0 con 0x7f24740ca0e0 2023-10-30T13:42:05.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f246affd640 1 -- 172.21.15.46:0/3664878884 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24640bb480 con 0x7f24740ca0e0 2023-10-30T13:42:05.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f2478d65640 1 -- 172.21.15.46:0/3664878884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24741f3070 con 0x7f24740ca0e0 2023-10-30T13:42:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f246affd640 1 -- 172.21.15.46:0/3664878884 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24640bb850 con 0x7f24740ca0e0 2023-10-30T13:42:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.365+0000 7f2478d65640 1 -- 172.21.15.46:0/3664878884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24741f3310 con 0x7f24740ca0e0 2023-10-30T13:42:05.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.369+0000 7f2478d65640 1 -- 172.21.15.46:0/3664878884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247404e170 con 0x7f24740ca0e0 2023-10-30T13:42:05.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.369+0000 7f246affd640 1 -- 172.21.15.46:0/3664878884 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f24640bb9b0 con 0x7f24740ca0e0 2023-10-30T13:42:05.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.369+0000 7f246affd640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24540664a0 0x7f2454068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:05.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.369+0000 7f246affd640 1 -- 172.21.15.46:0/3664878884 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f24641228d0 con 0x7f24740ca0e0 2023-10-30T13:42:05.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.369+0000 7f246bfff640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24540664a0 0x7f2454068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:05.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.369+0000 7f246bfff640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24540664a0 0x7f2454068960 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f24740c5ad0 tx=0x7f245c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:05.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.373+0000 7f246affd640 1 -- 172.21.15.46:0/3664878884 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f24640ef8c0 con 0x7f24740ca0e0 2023-10-30T13:42:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 -- 172.21.15.46:0/3664878884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f247415dc30 con 0x7f24740ca0e0 2023-10-30T13:42:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f246affd640 1 -- 172.21.15.46:0/3664878884 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7f247415dc30 con 0x7f24740ca0e0 2023-10-30T13:42:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:05.570 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:05.570 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:05.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 -- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24540664a0 msgr2=0x7f2454068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:05.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24540664a0 0x7f2454068960 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f24740c5ad0 tx=0x7f245c009040 comp rx=0 tx=0).stop 2023-10-30T13:42:05.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 -- 172.21.15.46:0/3664878884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24740ca0e0 msgr2=0x7f24741f3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:05.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24740ca0e0 0x7f24741f3b10 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f24640b3040 tx=0x7f246401ba90 comp rx=0 tx=0).stop 2023-10-30T13:42:05.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 -- 172.21.15.46:0/3664878884 shutdown_connections 2023-10-30T13:42:05.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24540664a0 0x7f2454068960 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:05.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24741538a0 0x7f24741e5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:05.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24740cf160 0x7f24741e2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:05.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 --2- 172.21.15.46:0/3664878884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24740ca0e0 0x7f24741f3b10 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:05.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 -- 172.21.15.46:0/3664878884 >> 172.21.15.46:0/3664878884 conn(0x7f247406bfe0 msgr2=0x7f2474065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:05.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 -- 172.21.15.46:0/3664878884 shutdown_connections 2023-10-30T13:42:05.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:05.565+0000 7f2478d65640 1 -- 172.21.15.46:0/3664878884 wait complete. 2023-10-30T13:42:05.598 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:42:05.598 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 98, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 98, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:stopping", "state_seq": 28 }, "gid_6036": { "addr": "172.21.15.46:6836/3037060424", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3037060424, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3037060424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6036, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 37 }, "gid_6201": { "addr": "172.21.15.46:6837/2546436963", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2546436963, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2546436963, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6201, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:42:05.189550+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6201, "mds_1": 6036, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.46:6833/190134548", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 190134548, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 190134548, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6366, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:42:07.600 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:07.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.749+0000 7fde18c09640 1 Processor -- start 2023-10-30T13:42:07.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde18c09640 1 -- start start 2023-10-30T13:42:07.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde18c09640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde140c05d0 0x7fde1413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:07.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde18c09640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde140c6f60 0x7fde1413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:07.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde18c09640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde140c0a50 con 0x7fde140c6b80 2023-10-30T13:42:07.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde18c09640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1406f6e0 con 0x7fde140c05d0 2023-10-30T13:42:07.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde18c09640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1406f9c0 con 0x7fde140c6f60 2023-10-30T13:42:07.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde11d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde140c05d0 0x7fde1413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:07.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde1259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fde140c6b80 0x7fde140bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35598/0 (socket says 172.21.15.46:35598) 2023-10-30T13:42:07.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde1259c640 1 -- 172.21.15.46:0/81200328 learned_addr learned my addr 172.21.15.46:0/81200328 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:07.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde12d9d640 1 --2- 172.21.15.46:0/81200328 >> v2:172.21.15.154:3300/0 conn(0x7fde140c6f60 0x7fde1413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:07.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde11d9b640 1 -- 172.21.15.46:0/81200328 >> v2:172.21.15.154:3300/0 conn(0x7fde140c6f60 msgr2=0x7fde1413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:07.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde11d9b640 1 --2- 172.21.15.46:0/81200328 >> v2:172.21.15.154:3300/0 conn(0x7fde140c6f60 0x7fde1413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:07.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde11d9b640 1 -- 172.21.15.46:0/81200328 >> v1:172.21.15.46:6789/0 conn(0x7fde140c6b80 legacy=0x7fde140bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:07.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde11d9b640 1 -- 172.21.15.46:0/81200328 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1406fca0 con 0x7fde140c05d0 2023-10-30T13:42:07.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde1159a640 1 -- 172.21.15.46:0/81200328 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659681406 0 0) 0x7fde140c0a50 con 0x7fde140c6b80 2023-10-30T13:42:07.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.753+0000 7fde12d9d640 1 --2- 172.21.15.46:0/81200328 >> v2:172.21.15.154:3300/0 conn(0x7fde140c6f60 0x7fde1413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:07.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.757+0000 7fde11d9b640 1 --2- 172.21.15.46:0/81200328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde140c05d0 0x7fde1413ed90 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fde140c89b0 tx=0x7fde0801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a71f0c6532744c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:07.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.757+0000 7fde1159a640 1 -- 172.21.15.46:0/81200328 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde080ab600 con 0x7fde140c05d0 2023-10-30T13:42:07.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.757+0000 7fde1159a640 1 -- 172.21.15.46:0/81200328 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde080a7050 con 0x7fde140c05d0 2023-10-30T13:42:07.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.757+0000 7fde1159a640 1 -- 172.21.15.46:0/81200328 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde080ab980 con 0x7fde140c05d0 2023-10-30T13:42:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.757+0000 7fde18c09640 1 -- 172.21.15.46:0/81200328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde140c05d0 msgr2=0x7fde1413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.757+0000 7fde18c09640 1 --2- 172.21.15.46:0/81200328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde140c05d0 0x7fde1413ed90 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fde140c89b0 tx=0x7fde0801c460 comp rx=0 tx=0).stop 2023-10-30T13:42:07.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.757+0000 7fde18c09640 1 -- 172.21.15.46:0/81200328 shutdown_connections 2023-10-30T13:42:07.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.757+0000 7fde18c09640 1 --2- 172.21.15.46:0/81200328 >> v2:172.21.15.154:3300/0 conn(0x7fde140c6f60 0x7fde1413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:07.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.757+0000 7fde18c09640 1 --2- 172.21.15.46:0/81200328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde140c05d0 0x7fde1413ed90 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.757+0000 7fde18c09640 1 -- 172.21.15.46:0/81200328 >> 172.21.15.46:0/81200328 conn(0x7fde140c7910 msgr2=0x7fde140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.757+0000 7fde18c09640 1 -- 172.21.15.46:0/81200328 shutdown_connections 2023-10-30T13:42:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.757+0000 7fde18c09640 1 -- 172.21.15.46:0/81200328 wait complete. 2023-10-30T13:42:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde18c09640 1 Processor -- start 2023-10-30T13:42:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde18c09640 1 -- start start 2023-10-30T13:42:07.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde18c09640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde140c05d0 0x7fde14153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:07.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde18c09640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde140c6b80 0x7fde140c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:07.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde18c09640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde140c6f60 0x7fde14069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:07.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde18c09640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde140c92a0 con 0x7fde140c6f60 2023-10-30T13:42:07.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde18c09640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde140c5fd0 con 0x7fde140c6b80 2023-10-30T13:42:07.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde18c09640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde14153590 con 0x7fde140c05d0 2023-10-30T13:42:07.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde12d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde140c6f60 0x7fde14069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:07.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde12d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde140c6f60 0x7fde14069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37304/0 (socket says 172.21.15.46:37304) 2023-10-30T13:42:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde12d9d640 1 -- 172.21.15.46:0/2076145732 learned_addr learned my addr 172.21.15.46:0/2076145732 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde11d9b640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde140c6b80 0x7fde140c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde1259c640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde140c05d0 0x7fde14153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde12d9d640 1 -- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde140c05d0 msgr2=0x7fde14153a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:07.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde12d9d640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde140c05d0 0x7fde14153a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:07.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde12d9d640 1 -- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde140c6b80 msgr2=0x7fde140c61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:07.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde12d9d640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde140c6b80 0x7fde140c61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:07.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde12d9d640 1 -- 172.21.15.46:0/2076145732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde08009420 con 0x7fde140c6f60 2023-10-30T13:42:07.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde12d9d640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde140c6f60 0x7fde14069e90 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fde0c007df0 tx=0x7fde0c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:07.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde02ffd640 1 -- 172.21.15.46:0/2076145732 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c0039a0 con 0x7fde140c6f60 2023-10-30T13:42:07.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde02ffd640 1 -- 172.21.15.46:0/2076145732 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c00f040 con 0x7fde140c6f60 2023-10-30T13:42:07.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde02ffd640 1 -- 172.21.15.46:0/2076145732 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c003d40 con 0x7fde140c6f60 2023-10-30T13:42:07.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde18c09640 1 -- 172.21.15.46:0/2076145732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde14152e30 con 0x7fde140c6f60 2023-10-30T13:42:07.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde18c09640 1 -- 172.21.15.46:0/2076145732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde140c54b0 con 0x7fde140c6f60 2023-10-30T13:42:07.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.761+0000 7fde18c09640 1 -- 172.21.15.46:0/2076145732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1413c3b0 con 0x7fde140c6f60 2023-10-30T13:42:07.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.765+0000 7fde02ffd640 1 -- 172.21.15.46:0/2076145732 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fde0c004030 con 0x7fde140c6f60 2023-10-30T13:42:07.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.765+0000 7fde02ffd640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddf40664a0 0x7fddf4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:07.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.765+0000 7fde02ffd640 1 -- 172.21.15.46:0/2076145732 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fde0c084540 con 0x7fde140c6f60 2023-10-30T13:42:07.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.765+0000 7fde1259c640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddf40664a0 0x7fddf4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:07.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.765+0000 7fde1259c640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddf40664a0 0x7fddf4068960 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fde04007fd0 tx=0x7fde04007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:07.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.769+0000 7fde02ffd640 1 -- 172.21.15.46:0/2076145732 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fde0c0515e0 con 0x7fde140c6f60 2023-10-30T13:42:07.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.957+0000 7fde18c09640 1 -- 172.21.15.46:0/2076145732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde141575c0 con 0x7fde140c6f60 2023-10-30T13:42:07.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde02ffd640 1 -- 172.21.15.46:0/2076145732 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7fde0c054c00 con 0x7fde140c6f60 2023-10-30T13:42:07.965 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:07.965 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:07.965 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde18c09640 1 -- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddf40664a0 msgr2=0x7fddf4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde18c09640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddf40664a0 0x7fddf4068960 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fde04007fd0 tx=0x7fde04007d60 comp rx=0 tx=0).stop 2023-10-30T13:42:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde18c09640 1 -- 172.21.15.46:0/2076145732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde140c6f60 msgr2=0x7fde14069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde18c09640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde140c6f60 0x7fde14069e90 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fde0c007df0 tx=0x7fde0c00d4a0 comp rx=0 tx=0).stop 2023-10-30T13:42:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde18c09640 1 -- 172.21.15.46:0/2076145732 shutdown_connections 2023-10-30T13:42:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde18c09640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddf40664a0 0x7fddf4068960 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde18c09640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde140c6f60 0x7fde14069e90 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde18c09640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde140c6b80 0x7fde140c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde18c09640 1 --2- 172.21.15.46:0/2076145732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde140c05d0 0x7fde14153a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde18c09640 1 -- 172.21.15.46:0/2076145732 >> 172.21.15.46:0/2076145732 conn(0x7fde140c7910 msgr2=0x7fde140c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde18c09640 1 -- 172.21.15.46:0/2076145732 shutdown_connections 2023-10-30T13:42:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:07.961+0000 7fde18c09640 1 -- 172.21.15.46:0/2076145732 wait complete. 2023-10-30T13:42:07.986 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:09.987 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:10.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.145+0000 7febe3d2a640 1 Processor -- start 2023-10-30T13:42:10.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.145+0000 7febe3d2a640 1 -- start start 2023-10-30T13:42:10.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe3d2a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febdc070a00 0x7febdc070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:10.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe3d2a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7febdc069e90 0x7febdc06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:10.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe3d2a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc071380 con 0x7febdc1539a0 2023-10-30T13:42:10.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe3d2a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febdc06fbb0 con 0x7febdc069e90 2023-10-30T13:42:10.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe3d2a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febdc153d80 con 0x7febdc070a00 2023-10-30T13:42:10.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe1ac6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7febdc1539a0 0x7febdc15a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35608/0 (socket says 172.21.15.46:35608) 2023-10-30T13:42:10.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe22c7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7febdc069e90 0x7febdc06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:10.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe1ac6640 1 -- 172.21.15.46:0/2443755850 learned_addr learned my addr 172.21.15.46:0/2443755850 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:10.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe12c5640 1 --2- 172.21.15.46:0/2443755850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febdc070a00 0x7febdc070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:10.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe22c7640 1 -- 172.21.15.46:0/2443755850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febdc070a00 msgr2=0x7febdc070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe22c7640 1 --2- 172.21.15.46:0/2443755850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febdc070a00 0x7febdc070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe22c7640 1 -- 172.21.15.46:0/2443755850 >> v1:172.21.15.46:6789/0 conn(0x7febdc1539a0 legacy=0x7febdc15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe22c7640 1 -- 172.21.15.46:0/2443755850 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc0c5c50 con 0x7febdc069e90 2023-10-30T13:42:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.149+0000 7febe0ac4640 1 -- 172.21.15.46:0/2443755850 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205805179 0 0) 0x7febdc071380 con 0x7febdc1539a0 2023-10-30T13:42:10.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.153+0000 7febe22c7640 1 --2- 172.21.15.46:0/2443755850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febdc069e90 0x7febdc06a300 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7febd40097c0 tx=0x7febd401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee07bf28c51f10a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:10.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.153+0000 7febe0ac4640 1 -- 172.21.15.46:0/2443755850 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd40ab600 con 0x7febdc069e90 2023-10-30T13:42:10.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.153+0000 7febe0ac4640 1 -- 172.21.15.46:0/2443755850 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd40a7050 con 0x7febdc069e90 2023-10-30T13:42:10.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.153+0000 7febe0ac4640 1 -- 172.21.15.46:0/2443755850 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd40ab980 con 0x7febdc069e90 2023-10-30T13:42:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.153+0000 7febe3d2a640 1 -- 172.21.15.46:0/2443755850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febdc069e90 msgr2=0x7febdc06a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.153+0000 7febe3d2a640 1 --2- 172.21.15.46:0/2443755850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febdc069e90 0x7febdc06a300 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7febd40097c0 tx=0x7febd401c3c0 comp rx=0 tx=0).stop 2023-10-30T13:42:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.153+0000 7febe3d2a640 1 -- 172.21.15.46:0/2443755850 shutdown_connections 2023-10-30T13:42:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.153+0000 7febe3d2a640 1 --2- 172.21.15.46:0/2443755850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febdc069e90 0x7febdc06a300 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.153+0000 7febe3d2a640 1 --2- 172.21.15.46:0/2443755850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febdc070a00 0x7febdc070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.153+0000 7febe3d2a640 1 -- 172.21.15.46:0/2443755850 >> 172.21.15.46:0/2443755850 conn(0x7febdc0c7910 msgr2=0x7febdc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.153+0000 7febe3d2a640 1 -- 172.21.15.46:0/2443755850 shutdown_connections 2023-10-30T13:42:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.153+0000 7febe3d2a640 1 -- 172.21.15.46:0/2443755850 wait complete. 2023-10-30T13:42:10.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe3d2a640 1 Processor -- start 2023-10-30T13:42:10.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe3d2a640 1 -- start start 2023-10-30T13:42:10.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe3d2a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febdc069e90 0x7febdc1f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:10.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe3d2a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febdc070a00 0x7febdc0ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:10.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe3d2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febdc1539a0 0x7febdc141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:10.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe3d2a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febdc0c17a0 con 0x7febdc1539a0 2023-10-30T13:42:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe3d2a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febdc0ca010 con 0x7febdc070a00 2023-10-30T13:42:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe3d2a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febdc1491a0 con 0x7febdc069e90 2023-10-30T13:42:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe22c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febdc1539a0 0x7febdc141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe22c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febdc1539a0 0x7febdc141520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37324/0 (socket says 172.21.15.46:37324) 2023-10-30T13:42:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe22c7640 1 -- 172.21.15.46:0/2873297525 learned_addr learned my addr 172.21.15.46:0/2873297525 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe12c5640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febdc070a00 0x7febdc0ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe1ac6640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febdc069e90 0x7febdc1f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe22c7640 1 -- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febdc069e90 msgr2=0x7febdc1f52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe22c7640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febdc069e90 0x7febdc1f52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe22c7640 1 -- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febdc070a00 msgr2=0x7febdc0ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe22c7640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febdc070a00 0x7febdc0ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:10.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe22c7640 1 -- 172.21.15.46:0/2873297525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd4009420 con 0x7febdc1539a0 2023-10-30T13:42:10.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe22c7640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febdc1539a0 0x7febdc141520 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7febd40ae040 tx=0x7febd4019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:10.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febd27fc640 1 -- 172.21.15.46:0/2873297525 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4006e20 con 0x7febdc1539a0 2023-10-30T13:42:10.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febd27fc640 1 -- 172.21.15.46:0/2873297525 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd40b4410 con 0x7febdc1539a0 2023-10-30T13:42:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe3d2a640 1 -- 172.21.15.46:0/2873297525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febdc0c1120 con 0x7febdc1539a0 2023-10-30T13:42:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febd27fc640 1 -- 172.21.15.46:0/2873297525 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd40b4780 con 0x7febdc1539a0 2023-10-30T13:42:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.157+0000 7febe3d2a640 1 -- 172.21.15.46:0/2873297525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febdc149440 con 0x7febdc1539a0 2023-10-30T13:42:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.161+0000 7febe3d2a640 1 -- 172.21.15.46:0/2873297525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febdc0c18e0 con 0x7febdc1539a0 2023-10-30T13:42:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.161+0000 7febd27fc640 1 -- 172.21.15.46:0/2873297525 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7febd40b2040 con 0x7febdc1539a0 2023-10-30T13:42:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.161+0000 7febd27fc640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7febc00664d0 0x7febc0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:10.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.161+0000 7febd27fc640 1 -- 172.21.15.46:0/2873297525 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7febd411ea00 con 0x7febdc1539a0 2023-10-30T13:42:10.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.161+0000 7febe1ac6640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7febc00664d0 0x7febc0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:10.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.165+0000 7febe1ac6640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7febc00664d0 0x7febc0068990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7febd8007fd0 tx=0x7febd8007ee0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:10.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.165+0000 7febd27fc640 1 -- 172.21.15.46:0/2873297525 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7febd40eb9e0 con 0x7febdc1539a0 2023-10-30T13:42:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 -- 172.21.15.46:0/2873297525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febdc0c18e0 con 0x7febdc1539a0 2023-10-30T13:42:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febd27fc640 1 -- 172.21.15.46:0/2873297525 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7febdc0c18e0 con 0x7febdc1539a0 2023-10-30T13:42:10.462 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:10.462 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 -- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7febc00664d0 msgr2=0x7febc0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7febc00664d0 0x7febc0068990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7febd8007fd0 tx=0x7febd8007ee0 comp rx=0 tx=0).stop 2023-10-30T13:42:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 -- 172.21.15.46:0/2873297525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febdc1539a0 msgr2=0x7febdc141520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febdc1539a0 0x7febdc141520 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7febd40ae040 tx=0x7febd4019160 comp rx=0 tx=0).stop 2023-10-30T13:42:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 -- 172.21.15.46:0/2873297525 shutdown_connections 2023-10-30T13:42:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7febc00664d0 0x7febc0068990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febdc1539a0 0x7febdc141520 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febdc070a00 0x7febdc0ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 --2- 172.21.15.46:0/2873297525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febdc069e90 0x7febdc1f52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 -- 172.21.15.46:0/2873297525 >> 172.21.15.46:0/2873297525 conn(0x7febdc0c7910 msgr2=0x7febdc1524e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 -- 172.21.15.46:0/2873297525 shutdown_connections 2023-10-30T13:42:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:10.457+0000 7febe3d2a640 1 -- 172.21.15.46:0/2873297525 wait complete. 2023-10-30T13:42:10.483 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:12.483 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:12.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.621+0000 7f7907dc4640 1 Processor -- start 2023-10-30T13:42:12.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.621+0000 7f7907dc4640 1 -- start start 2023-10-30T13:42:12.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7907dc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000c93d0 0x7f790006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:12.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7907dc4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790006afa0 con 0x7f7900153d70 2023-10-30T13:42:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7907dc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7900154150 con 0x7f790006e470 2023-10-30T13:42:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7907dc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790006d780 con 0x7f79000c93d0 2023-10-30T13:42:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7905b60640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7900153d70 0x7f79000cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35622/0 (socket says 172.21.15.46:35622) 2023-10-30T13:42:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7905b60640 1 -- 172.21.15.46:0/3629306658 learned_addr learned my addr 172.21.15.46:0/3629306658 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:12.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7906361640 1 --2- 172.21.15.46:0/3629306658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000c93d0 0x7f790006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:12.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758747150 0 0) 0x7f790006afa0 con 0x7f7900153d70 2023-10-30T13:42:12.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78ec003530 con 0x7f7900153d70 2023-10-30T13:42:12.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3464569483 0 0) 0x7f78ec003530 con 0x7f7900153d70 2023-10-30T13:42:12.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000c93d0 msgr2=0x7f790006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:12.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 --2- 172.21.15.46:0/3629306658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000c93d0 0x7f790006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:12.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 >> v1:172.21.15.154:6789/0 conn(0x7f790006e470 legacy=0x7f79000c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:12.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790006e030 con 0x7f7900153d70 2023-10-30T13:42:12.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f78fc001680 con 0x7f7900153d70 2023-10-30T13:42:12.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 >> v1:172.21.15.46:6789/0 conn(0x7f7900153d70 legacy=0x7f79000cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:12.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 --2- 172.21.15.46:0/3629306658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78ec0039b0 0x7f78ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:12.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 --2- 172.21.15.46:0/3629306658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78ec004360 0x7f78ec004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:12.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 --2- 172.21.15.46:0/3629306658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000c93d0 0x7f78ec005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:12.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790006d780 con 0x7f78ec0039b0 2023-10-30T13:42:12.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ec00b990 con 0x7f78ec004360 2023-10-30T13:42:12.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ec00bb50 con 0x7f79000c93d0 2023-10-30T13:42:12.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7905b60640 1 --2- 172.21.15.46:0/3629306658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78ec0039b0 0x7f78ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:12.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f790535f640 1 --2- 172.21.15.46:0/3629306658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78ec004360 0x7f78ec004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:12.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7906361640 1 --2- 172.21.15.46:0/3629306658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000c93d0 0x7f78ec005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:12.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7905b60640 1 -- 172.21.15.46:0/3629306658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000c93d0 msgr2=0x7f78ec005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:12.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7905b60640 1 --2- 172.21.15.46:0/3629306658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000c93d0 0x7f78ec005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:12.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7905b60640 1 -- 172.21.15.46:0/3629306658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78ec004360 msgr2=0x7f78ec004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:12.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7905b60640 1 --2- 172.21.15.46:0/3629306658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78ec004360 0x7f78ec004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:12.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.625+0000 7f7905b60640 1 -- 172.21.15.46:0/3629306658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ec003530 con 0x7f78ec0039b0 2023-10-30T13:42:12.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.629+0000 7f7905b60640 1 --2- 172.21.15.46:0/3629306658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78ec0039b0 0x7f78ec003d90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00c210 tx=0x7f78fc01cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9c2db6048b22b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:12.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.629+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc0a5b10 con 0x7f78ec0039b0 2023-10-30T13:42:12.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.629+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc0a5c90 con 0x7f78ec0039b0 2023-10-30T13:42:12.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.629+0000 7f7904b5e640 1 -- 172.21.15.46:0/3629306658 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc0058e0 con 0x7f78ec0039b0 2023-10-30T13:42:12.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.629+0000 7f7907dc4640 1 -- 172.21.15.46:0/3629306658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78ec0039b0 msgr2=0x7f78ec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:12.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.629+0000 7f7907dc4640 1 --2- 172.21.15.46:0/3629306658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78ec0039b0 0x7f78ec003d90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00c210 tx=0x7f78fc01cc30 comp rx=0 tx=0).stop 2023-10-30T13:42:12.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.629+0000 7f7906361640 1 -- 172.21.15.46:0/3629306658 reap_dead start 2023-10-30T13:42:12.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.629+0000 7f7907dc4640 1 -- 172.21.15.46:0/3629306658 shutdown_connections 2023-10-30T13:42:12.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.629+0000 7f7907dc4640 1 -- 172.21.15.46:0/3629306658 >> 172.21.15.46:0/3629306658 conn(0x7f7900071ea0 msgr2=0x7f7900065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:12.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.629+0000 7f7907dc4640 1 -- 172.21.15.46:0/3629306658 shutdown_connections 2023-10-30T13:42:12.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.629+0000 7f7907dc4640 1 -- 172.21.15.46:0/3629306658 wait complete. 2023-10-30T13:42:12.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f7907dc4640 1 Processor -- start 2023-10-30T13:42:12.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f7907dc4640 1 -- start start 2023-10-30T13:42:12.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f7907dc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78ec0039b0 0x7f790014c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f7907dc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79001f3cd0 0x7f7900149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f7907dc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7900147e30 0x7f79001482c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f7907dc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790014cca0 con 0x7f79001f3cd0 2023-10-30T13:42:12.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f7907dc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7900148800 con 0x7f7900147e30 2023-10-30T13:42:12.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f7907dc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790015a110 con 0x7f78ec0039b0 2023-10-30T13:42:12.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f790535f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79001f3cd0 0x7f7900149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:12.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f790535f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79001f3cd0 0x7f7900149eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37332/0 (socket says 172.21.15.46:37332) 2023-10-30T13:42:12.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f790535f640 1 -- 172.21.15.46:0/2853124084 learned_addr learned my addr 172.21.15.46:0/2853124084 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:12.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f7905b60640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78ec0039b0 0x7f790014c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:12.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f7906361640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7900147e30 0x7f79001482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:12.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f790535f640 1 -- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78ec0039b0 msgr2=0x7f790014c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:12.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f790535f640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78ec0039b0 0x7f790014c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:12.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f790535f640 1 -- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7900147e30 msgr2=0x7f79001482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:12.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f790535f640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7900147e30 0x7f79001482c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:12.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f790535f640 1 -- 172.21.15.46:0/2853124084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790015a370 con 0x7f79001f3cd0 2023-10-30T13:42:12.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f790535f640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79001f3cd0 0x7f7900149eb0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f78f000e510 tx=0x7f78f000e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:12.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f78f67fc640 1 -- 172.21.15.46:0/2853124084 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f00052e0 con 0x7f79001f3cd0 2023-10-30T13:42:12.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f78f67fc640 1 -- 172.21.15.46:0/2853124084 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f0005440 con 0x7f79001f3cd0 2023-10-30T13:42:12.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f7907dc4640 1 -- 172.21.15.46:0/2853124084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7900162ef0 con 0x7f79001f3cd0 2023-10-30T13:42:12.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f78f67fc640 1 -- 172.21.15.46:0/2853124084 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f0003840 con 0x7f79001f3cd0 2023-10-30T13:42:12.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.633+0000 7f7907dc4640 1 -- 172.21.15.46:0/2853124084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790014b2d0 con 0x7f79001f3cd0 2023-10-30T13:42:12.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.637+0000 7f7907dc4640 1 -- 172.21.15.46:0/2853124084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790004e170 con 0x7f79001f3cd0 2023-10-30T13:42:12.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.637+0000 7f78f67fc640 1 -- 172.21.15.46:0/2853124084 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f78f00039a0 con 0x7f79001f3cd0 2023-10-30T13:42:12.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.637+0000 7f78f67fc640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f78e40664a0 0x7f78e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:12.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.637+0000 7f78f67fc640 1 -- 172.21.15.46:0/2853124084 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f78f0083ec0 con 0x7f79001f3cd0 2023-10-30T13:42:12.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.637+0000 7f7905b60640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f78e40664a0 0x7f78e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:12.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.637+0000 7f7905b60640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f78e40664a0 0x7f78e4068960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00c210 tx=0x7f78fc0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:12.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.641+0000 7f78f67fc640 1 -- 172.21.15.46:0/2853124084 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f78f0051000 con 0x7f79001f3cd0 2023-10-30T13:42:12.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 -- 172.21.15.46:0/2853124084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f790006c610 con 0x7f79001f3cd0 2023-10-30T13:42:12.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f78f67fc640 1 -- 172.21.15.46:0/2853124084 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7f78f0054620 con 0x7f79001f3cd0 2023-10-30T13:42:12.837 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:12.838 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:12.838 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:12.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 -- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f78e40664a0 msgr2=0x7f78e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:12.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f78e40664a0 0x7f78e4068960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00c210 tx=0x7f78fc0a7070 comp rx=0 tx=0).stop 2023-10-30T13:42:12.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 -- 172.21.15.46:0/2853124084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79001f3cd0 msgr2=0x7f7900149eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:12.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79001f3cd0 0x7f7900149eb0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f78f000e510 tx=0x7f78f000e9d0 comp rx=0 tx=0).stop 2023-10-30T13:42:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 -- 172.21.15.46:0/2853124084 shutdown_connections 2023-10-30T13:42:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f78e40664a0 0x7f78e4068960 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7900147e30 0x7f79001482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79001f3cd0 0x7f7900149eb0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 --2- 172.21.15.46:0/2853124084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78ec0039b0 0x7f790014c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:12.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 -- 172.21.15.46:0/2853124084 >> 172.21.15.46:0/2853124084 conn(0x7f7900071ea0 msgr2=0x7f790014f880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:12.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 -- 172.21.15.46:0/2853124084 shutdown_connections 2023-10-30T13:42:12.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:12.833+0000 7f7907dc4640 1 -- 172.21.15.46:0/2853124084 wait complete. 2023-10-30T13:42:12.858 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:14.858 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.009+0000 7f0d03b0f640 1 Processor -- start 2023-10-30T13:42:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.009+0000 7f0d03b0f640 1 -- start start 2023-10-30T13:42:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.009+0000 7f0d03b0f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0cfc1538a0 0x7f0cfc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.009+0000 7f0d03b0f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cfc0cb800 con 0x7f0cfc0cf160 2023-10-30T13:42:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.009+0000 7f0d03b0f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cfc15dc30 con 0x7f0cfc0ca0e0 2023-10-30T13:42:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.009+0000 7f0d03b0f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cfc0cb330 con 0x7f0cfc1538a0 2023-10-30T13:42:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.009+0000 7f0d020ac640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0cfc0cf160 0x7f0cfc164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35632/0 (socket says 172.21.15.46:35632) 2023-10-30T13:42:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.009+0000 7f0d020ac640 1 -- 172.21.15.46:0/3143443150 learned_addr learned my addr 172.21.15.46:0/3143443150 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d018ab640 1 --2- 172.21.15.46:0/3143443150 >> v2:172.21.15.154:3300/0 conn(0x7f0cfc1538a0 0x7f0cfc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457212079 0 0) 0x7f0cfc0cb800 con 0x7f0cfc0cf160 2023-10-30T13:42:15.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce8003530 con 0x7f0cfc0cf160 2023-10-30T13:42:15.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2009461309 0 0) 0x7f0ce8003530 con 0x7f0cfc0cf160 2023-10-30T13:42:15.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 >> v2:172.21.15.154:3300/0 conn(0x7f0cfc1538a0 msgr2=0x7f0cfc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:15.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 --2- 172.21.15.46:0/3143443150 >> v2:172.21.15.154:3300/0 conn(0x7f0cfc1538a0 0x7f0cfc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:15.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 >> v1:172.21.15.154:6789/0 conn(0x7f0cfc0ca0e0 legacy=0x7f0cfc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc0cad70 con 0x7f0cfc0cf160 2023-10-30T13:42:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f0cf4001a00 con 0x7f0cfc0cf160 2023-10-30T13:42:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 >> v1:172.21.15.46:6789/0 conn(0x7f0cfc0cf160 legacy=0x7f0cfc164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 --2- 172.21.15.46:0/3143443150 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce8003980 0x7f0ce8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 --2- 172.21.15.46:0/3143443150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ce80047d0 0x7f0ce8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 --2- 172.21.15.46:0/3143443150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce8005190 0x7f0ce800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:15.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc0cb330 con 0x7f0ce80047d0 2023-10-30T13:42:15.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce800c040 con 0x7f0ce8005190 2023-10-30T13:42:15.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce800c200 con 0x7f0ce8003980 2023-10-30T13:42:15.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d010aa640 1 --2- 172.21.15.46:0/3143443150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ce80047d0 0x7f0ce8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:15.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d020ac640 1 --2- 172.21.15.46:0/3143443150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce8005190 0x7f0ce800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d018ab640 1 --2- 172.21.15.46:0/3143443150 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce8003980 0x7f0ce8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d010aa640 1 -- 172.21.15.46:0/3143443150 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce8003980 msgr2=0x7f0ce8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d010aa640 1 --2- 172.21.15.46:0/3143443150 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ce8003980 0x7f0ce8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d010aa640 1 -- 172.21.15.46:0/3143443150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce8005190 msgr2=0x7f0ce800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d010aa640 1 --2- 172.21.15.46:0/3143443150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce8005190 0x7f0ce800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d010aa640 1 -- 172.21.15.46:0/3143443150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8003530 con 0x7f0ce80047d0 2023-10-30T13:42:15.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.013+0000 7f0d020ac640 1 -- 172.21.15.46:0/3143443150 reap_dead start 2023-10-30T13:42:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d010aa640 1 --2- 172.21.15.46:0/3143443150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ce80047d0 0x7f0ce8004c50 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f0cec0015e0 tx=0x7f0cec01c240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c6478f7ff175d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec0052d0 con 0x7f0ce80047d0 2023-10-30T13:42:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec0a7050 con 0x7f0ce80047d0 2023-10-30T13:42:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d008a9640 1 -- 172.21.15.46:0/3143443150 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec003600 con 0x7f0ce80047d0 2023-10-30T13:42:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d03b0f640 1 -- 172.21.15.46:0/3143443150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ce80047d0 msgr2=0x7f0ce8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:15.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d03b0f640 1 --2- 172.21.15.46:0/3143443150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ce80047d0 0x7f0ce8004c50 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f0cec0015e0 tx=0x7f0cec01c240 comp rx=0 tx=0).stop 2023-10-30T13:42:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d03b0f640 1 -- 172.21.15.46:0/3143443150 shutdown_connections 2023-10-30T13:42:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d03b0f640 1 --2- 172.21.15.46:0/3143443150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ce80047d0 0x7f0ce8004c50 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d03b0f640 1 -- 172.21.15.46:0/3143443150 >> 172.21.15.46:0/3143443150 conn(0x7f0cfc06bfe0 msgr2=0x7f0cfc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d03b0f640 1 -- 172.21.15.46:0/3143443150 shutdown_connections 2023-10-30T13:42:15.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d03b0f640 1 -- 172.21.15.46:0/3143443150 wait complete. 2023-10-30T13:42:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d03b0f640 1 Processor -- start 2023-10-30T13:42:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.017+0000 7f0d03b0f640 1 -- start start 2023-10-30T13:42:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d03b0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce80047d0 0x7f0cfc065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d03b0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0cfc0b4d20 0x7f0cfc1e41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d03b0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cfc1e46e0 0x7f0cfc1f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:15.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d03b0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc150900 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d03b0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc0b4890 con 0x7f0ce80047d0 2023-10-30T13:42:15.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d03b0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc159340 con 0x7f0cfc0b4d20 2023-10-30T13:42:15.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d020ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cfc1e46e0 0x7f0cfc1f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:15.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d020ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cfc1e46e0 0x7f0cfc1f2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37354/0 (socket says 172.21.15.46:37354) 2023-10-30T13:42:15.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d020ac640 1 -- 172.21.15.46:0/4012107006 learned_addr learned my addr 172.21.15.46:0/4012107006 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:15.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d018ab640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce80047d0 0x7f0cfc065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:15.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d010aa640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0cfc0b4d20 0x7f0cfc1e41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:15.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d020ac640 1 -- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0cfc0b4d20 msgr2=0x7f0cfc1e41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:15.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d020ac640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0cfc0b4d20 0x7f0cfc1e41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:15.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d020ac640 1 -- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce80047d0 msgr2=0x7f0cfc065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:15.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d020ac640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce80047d0 0x7f0cfc065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:15.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d020ac640 1 -- 172.21.15.46:0/4012107006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc0b3e50 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d020ac640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cfc1e46e0 0x7f0cfc1f2fd0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc06cb00 tx=0x7f0cf400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:15.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0cf27fc640 1 -- 172.21.15.46:0/4012107006 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf400ebc0 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0cf27fc640 1 -- 172.21.15.46:0/4012107006 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf400ed20 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0cf27fc640 1 -- 172.21.15.46:0/4012107006 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4014670 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d03b0f640 1 -- 172.21.15.46:0/4012107006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cfc159600 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d03b0f640 1 -- 172.21.15.46:0/4012107006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cfc0b30a0 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0d03b0f640 1 -- 172.21.15.46:0/4012107006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cfc04e170 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.021+0000 7f0cf27fc640 1 -- 172.21.15.46:0/4012107006 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0cf40147d0 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.025+0000 7f0cf27fc640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ce00664d0 0x7f0ce0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:15.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.025+0000 7f0cf27fc640 1 -- 172.21.15.46:0/4012107006 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f0cf4083fd0 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.025+0000 7f0d018ab640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ce00664d0 0x7f0ce0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:15.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.025+0000 7f0d018ab640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ce00664d0 0x7f0ce0068990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc069010 tx=0x7f0cf8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:15.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.029+0000 7f0cf27fc640 1 -- 172.21.15.46:0/4012107006 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0cf4050fc0 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 -- 172.21.15.46:0/4012107006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cfc0b1dd0 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0cf27fc640 1 -- 172.21.15.46:0/4012107006 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7f0cf40545e0 con 0x7f0cfc1e46e0 2023-10-30T13:42:15.226 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:15.226 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:15.227 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:15.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 -- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ce00664d0 msgr2=0x7f0ce0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:15.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ce00664d0 0x7f0ce0068990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc069010 tx=0x7f0cf8008040 comp rx=0 tx=0).stop 2023-10-30T13:42:15.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 -- 172.21.15.46:0/4012107006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cfc1e46e0 msgr2=0x7f0cfc1f2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:15.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cfc1e46e0 0x7f0cfc1f2fd0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc06cb00 tx=0x7f0cf400e530 comp rx=0 tx=0).stop 2023-10-30T13:42:15.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 -- 172.21.15.46:0/4012107006 shutdown_connections 2023-10-30T13:42:15.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ce00664d0 0x7f0ce0068990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:15.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cfc1e46e0 0x7f0cfc1f2fd0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:15.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0cfc0b4d20 0x7f0cfc1e41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:15.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 --2- 172.21.15.46:0/4012107006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ce80047d0 0x7f0cfc065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:15.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 -- 172.21.15.46:0/4012107006 >> 172.21.15.46:0/4012107006 conn(0x7f0cfc06bfe0 msgr2=0x7f0cfc1519a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:15.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 -- 172.21.15.46:0/4012107006 shutdown_connections 2023-10-30T13:42:15.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:15.221+0000 7f0d03b0f640 1 -- 172.21.15.46:0/4012107006 wait complete. 2023-10-30T13:42:15.247 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:17.248 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.417+0000 7f9b1ede4640 1 Processor -- start 2023-10-30T13:42:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.417+0000 7f9b1ede4640 1 -- start start 2023-10-30T13:42:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.417+0000 7f9b1ede4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b180ccdc0 0x7f9b1806f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.417+0000 7f9b1ede4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9b180cd1a0 0x7f9b1815d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1ede4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9b180c2490 0x7f9b180c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1ede4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b180c2e60 con 0x7f9b180c2490 2023-10-30T13:42:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1ede4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b1815da50 con 0x7f9b180cd1a0 2023-10-30T13:42:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1ede4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b18070050 con 0x7f9b180ccdc0 2023-10-30T13:42:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1e5e3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9b180c2490 0x7f9b180c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1e5e3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9b180c2490 0x7f9b180c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44372/0 (socket says 172.21.15.46:44372) 2023-10-30T13:42:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1e5e3640 1 -- 172.21.15.46:0/3737342560 learned_addr learned my addr 172.21.15.46:0/3737342560 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1d5e1640 1 --2- 172.21.15.46:0/3737342560 >> v2:172.21.15.154:3300/0 conn(0x7f9b180cd1a0 0x7f9b1815d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1dde2640 1 --2- 172.21.15.46:0/3737342560 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b180ccdc0 0x7f9b1806f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1e5e3640 1 -- 172.21.15.46:0/3737342560 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b180ccdc0 msgr2=0x7f9b1806f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1e5e3640 1 --2- 172.21.15.46:0/3737342560 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b180ccdc0 0x7f9b1806f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1e5e3640 1 -- 172.21.15.46:0/3737342560 >> v2:172.21.15.154:3300/0 conn(0x7f9b180cd1a0 msgr2=0x7f9b1815d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1e5e3640 1 --2- 172.21.15.46:0/3737342560 >> v2:172.21.15.154:3300/0 conn(0x7f9b180cd1a0 0x7f9b1815d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1e5e3640 1 -- 172.21.15.46:0/3737342560 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b18159860 con 0x7f9b180c2490 2023-10-30T13:42:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.421+0000 7f9b1e5e3640 1 --2- 172.21.15.46:0/3737342560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b180c2490 0x7f9b180c2920 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f9b180cdef0 tx=0x7f9b1001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84161c5b7f15f6d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1cde0640 1 -- 172.21.15.46:0/3737342560 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b100af400 con 0x7f9b180c2490 2023-10-30T13:42:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1cde0640 1 -- 172.21.15.46:0/3737342560 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b100ab050 con 0x7f9b180c2490 2023-10-30T13:42:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1cde0640 1 -- 172.21.15.46:0/3737342560 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b100af780 con 0x7f9b180c2490 2023-10-30T13:42:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3737342560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b180c2490 msgr2=0x7f9b180c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 --2- 172.21.15.46:0/3737342560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b180c2490 0x7f9b180c2920 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f9b180cdef0 tx=0x7f9b1001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3737342560 shutdown_connections 2023-10-30T13:42:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 --2- 172.21.15.46:0/3737342560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b180c2490 0x7f9b180c2920 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 --2- 172.21.15.46:0/3737342560 >> v2:172.21.15.154:3300/0 conn(0x7f9b180cd1a0 0x7f9b1815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 --2- 172.21.15.46:0/3737342560 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b180ccdc0 0x7f9b1806f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3737342560 >> 172.21.15.46:0/3737342560 conn(0x7f9b1806dfc0 msgr2=0x7f9b18071710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3737342560 shutdown_connections 2023-10-30T13:42:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3737342560 wait complete. 2023-10-30T13:42:17.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 Processor -- start 2023-10-30T13:42:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 -- start start 2023-10-30T13:42:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b180c2490 0x7f9b1813fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b180ccdc0 0x7f9b18157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b180cd1a0 0x7f9b18141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1814e4b0 con 0x7f9b180cd1a0 2023-10-30T13:42:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b181f3260 con 0x7f9b180ccdc0 2023-10-30T13:42:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1ede4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b18150450 con 0x7f9b180c2490 2023-10-30T13:42:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1e5e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b180cd1a0 0x7f9b18141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1e5e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b180cd1a0 0x7f9b18141820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44382/0 (socket says 172.21.15.46:44382) 2023-10-30T13:42:17.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1e5e3640 1 -- 172.21.15.46:0/3352044686 learned_addr learned my addr 172.21.15.46:0/3352044686 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:17.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.425+0000 7f9b1d5e1640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b180ccdc0 0x7f9b18157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b1dde2640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b180c2490 0x7f9b1813fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b1e5e3640 1 -- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b180c2490 msgr2=0x7f9b1813fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b1e5e3640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b180c2490 0x7f9b1813fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b1e5e3640 1 -- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b180ccdc0 msgr2=0x7f9b18157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b1e5e3640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b180ccdc0 0x7f9b18157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b1e5e3640 1 -- 172.21.15.46:0/3352044686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b10009b30 con 0x7f9b180cd1a0 2023-10-30T13:42:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b1e5e3640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b180cd1a0 0x7f9b18141820 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9b10009f10 tx=0x7f9b1001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b0e7fc640 1 -- 172.21.15.46:0/3352044686 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b100afc60 con 0x7f9b180cd1a0 2023-10-30T13:42:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b0e7fc640 1 -- 172.21.15.46:0/3352044686 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b100afdc0 con 0x7f9b180cd1a0 2023-10-30T13:42:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b0e7fc640 1 -- 172.21.15.46:0/3352044686 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b100b8640 con 0x7f9b180cd1a0 2023-10-30T13:42:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3352044686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b181f23b0 con 0x7f9b180cd1a0 2023-10-30T13:42:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3352044686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b181f2930 con 0x7f9b180cd1a0 2023-10-30T13:42:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3352044686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b1804e170 con 0x7f9b180cd1a0 2023-10-30T13:42:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b0e7fc640 1 -- 172.21.15.46:0/3352044686 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9b100b6040 con 0x7f9b180cd1a0 2023-10-30T13:42:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b0e7fc640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9afc0664d0 0x7f9afc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.429+0000 7f9b0e7fc640 1 -- 172.21.15.46:0/3352044686 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f9b10122100 con 0x7f9b180cd1a0 2023-10-30T13:42:17.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.433+0000 7f9b1dde2640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9afc0664d0 0x7f9afc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.433+0000 7f9b1dde2640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9afc0664d0 0x7f9afc068990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9b14009ae0 tx=0x7f9b14009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:17.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.433+0000 7f9b0e7fc640 1 -- 172.21.15.46:0/3352044686 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9b100ef270 con 0x7f9b180cd1a0 2023-10-30T13:42:17.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.621+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3352044686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b1814d710 con 0x7f9b180cd1a0 2023-10-30T13:42:17.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b0e7fc640 1 -- 172.21.15.46:0/3352044686 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7f9b100b4020 con 0x7f9b180cd1a0 2023-10-30T13:42:17.629 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:17.629 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:17.630 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:17.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9afc0664d0 msgr2=0x7f9afc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b1ede4640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9afc0664d0 0x7f9afc068990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9b14009ae0 tx=0x7f9b14009190 comp rx=0 tx=0).stop 2023-10-30T13:42:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3352044686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b180cd1a0 msgr2=0x7f9b18141820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b1ede4640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b180cd1a0 0x7f9b18141820 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9b10009f10 tx=0x7f9b1001bf60 comp rx=0 tx=0).stop 2023-10-30T13:42:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3352044686 shutdown_connections 2023-10-30T13:42:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b1ede4640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9afc0664d0 0x7f9afc068990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b1ede4640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b180cd1a0 0x7f9b18141820 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b1ede4640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b180ccdc0 0x7f9b18157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b1ede4640 1 --2- 172.21.15.46:0/3352044686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b180c2490 0x7f9b1813fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3352044686 >> 172.21.15.46:0/3352044686 conn(0x7f9b1806dfc0 msgr2=0x7f9b1806ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:17.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3352044686 shutdown_connections 2023-10-30T13:42:17.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:17.625+0000 7f9b1ede4640 1 -- 172.21.15.46:0/3352044686 wait complete. 2023-10-30T13:42:17.650 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:42:17.651 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 98, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 98, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:stopping", "state_seq": 28 }, "gid_6036": { "addr": "172.21.15.46:6836/3037060424", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3037060424, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3037060424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6036, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 37 }, "gid_6201": { "addr": "172.21.15.46:6837/2546436963", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2546436963, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2546436963, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6201, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:42:05.189550+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6201, "mds_1": 6036, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.46:6833/190134548", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 190134548, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 190134548, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6366, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:42:18.689 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:42:18.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:42:19.652 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:19.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.801+0000 7f6785417640 1 Processor -- start 2023-10-30T13:42:19.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.801+0000 7f6785417640 1 -- start start 2023-10-30T13:42:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f6785417640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67800c6b80 0x7f67800bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f6785417640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f678013f4a0 0x7f67800c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f6785417640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67800c0a50 con 0x7f67800c6f60 2023-10-30T13:42:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f6785417640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f678006f6e0 con 0x7f678013f4a0 2023-10-30T13:42:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f6785417640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678006f9c0 con 0x7f67800c6b80 2023-10-30T13:42:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f677f7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f678013f4a0 0x7f67800c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f677effd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67800c6b80 0x7f67800bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f677f7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f678013f4a0 0x7f67800c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:46862/0 (socket says 172.21.15.46:46862) 2023-10-30T13:42:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f677f7fe640 1 -- 172.21.15.46:0/46277611 learned_addr learned my addr 172.21.15.46:0/46277611 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f677effd640 1 -- 172.21.15.46:0/46277611 >> v2:172.21.15.154:3300/0 conn(0x7f678013f4a0 msgr2=0x7f67800c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f677effd640 1 --2- 172.21.15.46:0/46277611 >> v2:172.21.15.154:3300/0 conn(0x7f678013f4a0 0x7f67800c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f677effd640 1 -- 172.21.15.46:0/46277611 >> v1:172.21.15.154:6789/0 conn(0x7f67800c6f60 legacy=0x7f678013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f677effd640 1 -- 172.21.15.46:0/46277611 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678006fca0 con 0x7f67800c6b80 2023-10-30T13:42:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.805+0000 7f677f7fe640 1 --2- 172.21.15.46:0/46277611 >> v2:172.21.15.154:3300/0 conn(0x7f678013f4a0 0x7f67800c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:42:19.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f677effd640 1 --2- 172.21.15.46:0/46277611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67800c6b80 0x7f67800bfec0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f67700097c0 tx=0x7f677001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4933ad949afb9e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f677e7fc640 1 -- 172.21.15.46:0/46277611 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67700ad640 con 0x7f67800c6b80 2023-10-30T13:42:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f677e7fc640 1 -- 172.21.15.46:0/46277611 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67700ad7a0 con 0x7f67800c6b80 2023-10-30T13:42:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f677e7fc640 1 -- 172.21.15.46:0/46277611 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67700adb30 con 0x7f67800c6b80 2023-10-30T13:42:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f6785417640 1 -- 172.21.15.46:0/46277611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67800c6b80 msgr2=0x7f67800bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f6785417640 1 --2- 172.21.15.46:0/46277611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67800c6b80 0x7f67800bfec0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f67700097c0 tx=0x7f677001c480 comp rx=0 tx=0).stop 2023-10-30T13:42:19.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f6785417640 1 -- 172.21.15.46:0/46277611 shutdown_connections 2023-10-30T13:42:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f6785417640 1 --2- 172.21.15.46:0/46277611 >> v2:172.21.15.154:3300/0 conn(0x7f678013f4a0 0x7f67800c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f6785417640 1 --2- 172.21.15.46:0/46277611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67800c6b80 0x7f67800bfec0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f6785417640 1 -- 172.21.15.46:0/46277611 >> 172.21.15.46:0/46277611 conn(0x7f67800c7910 msgr2=0x7f67800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:19.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f6785417640 1 -- 172.21.15.46:0/46277611 shutdown_connections 2023-10-30T13:42:19.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f6785417640 1 -- 172.21.15.46:0/46277611 wait complete. 2023-10-30T13:42:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f6785417640 1 Processor -- start 2023-10-30T13:42:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f6785417640 1 -- start start 2023-10-30T13:42:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f6785417640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67800c6b80 0x7f67801f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:19.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.809+0000 7f6785417640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67800c6f60 0x7f67801f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6785417640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f678013f4a0 0x7f67800c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6785417640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67801564f0 con 0x7f67800c6f60 2023-10-30T13:42:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6785417640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6780155fa0 con 0x7f67800c6b80 2023-10-30T13:42:19.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6785417640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67800c8db0 con 0x7f678013f4a0 2023-10-30T13:42:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6777fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67800c6f60 0x7f67801f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6777fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67800c6f60 0x7f67801f60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44386/0 (socket says 172.21.15.46:44386) 2023-10-30T13:42:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6777fff640 1 -- 172.21.15.46:0/862802385 learned_addr learned my addr 172.21.15.46:0/862802385 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f677effd640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67800c6b80 0x7f67801f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f677f7fe640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f678013f4a0 0x7f67800c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6777fff640 1 -- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f678013f4a0 msgr2=0x7f67800c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6777fff640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f678013f4a0 0x7f67800c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6777fff640 1 -- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67800c6b80 msgr2=0x7f67801f5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6777fff640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67800c6b80 0x7f67801f5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6777fff640 1 -- 172.21.15.46:0/862802385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6778009b30 con 0x7f67800c6f60 2023-10-30T13:42:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6777fff640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67800c6f60 0x7f67801f60b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f676800c430 tx=0x7f676800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f67777fe640 1 -- 172.21.15.46:0/862802385 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67680052d0 con 0x7f67800c6f60 2023-10-30T13:42:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f67777fe640 1 -- 172.21.15.46:0/862802385 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6768005430 con 0x7f67800c6f60 2023-10-30T13:42:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f67777fe640 1 -- 172.21.15.46:0/862802385 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6768007260 con 0x7f67800c6f60 2023-10-30T13:42:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6785417640 1 -- 172.21.15.46:0/862802385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6770009420 con 0x7f67800c6f60 2023-10-30T13:42:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6785417640 1 -- 172.21.15.46:0/862802385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67800c9050 con 0x7f67800c6f60 2023-10-30T13:42:19.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f6785417640 1 -- 172.21.15.46:0/862802385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67800c8c10 con 0x7f67800c6f60 2023-10-30T13:42:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.813+0000 7f67777fe640 1 -- 172.21.15.46:0/862802385 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67680073c0 con 0x7f67800c6f60 2023-10-30T13:42:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.817+0000 7f67777fe640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f67600664a0 0x7f6760068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.817+0000 7f67777fe640 1 -- 172.21.15.46:0/862802385 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f6768083670 con 0x7f67800c6f60 2023-10-30T13:42:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.817+0000 7f677effd640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f67600664a0 0x7f6760068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.817+0000 7f677effd640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f67600664a0 0x7f6760068960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f67700af000 tx=0x7f67700a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:19.821+0000 7f67777fe640 1 -- 172.21.15.46:0/862802385 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f67800c8c10 con 0x7f67800c6f60 2023-10-30T13:42:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 -- 172.21.15.46:0/862802385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67800c8c10 con 0x7f67800c6f60 2023-10-30T13:42:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f67777fe640 1 -- 172.21.15.46:0/862802385 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7f67800c8c10 con 0x7f67800c6f60 2023-10-30T13:42:20.014 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:20.014 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:20.014 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:20.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 -- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f67600664a0 msgr2=0x7f6760068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:20.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f67600664a0 0x7f6760068960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f67700af000 tx=0x7f67700a7070 comp rx=0 tx=0).stop 2023-10-30T13:42:20.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 -- 172.21.15.46:0/862802385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67800c6f60 msgr2=0x7f67801f60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:20.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67800c6f60 0x7f67801f60b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f676800c430 tx=0x7f676800c8f0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 -- 172.21.15.46:0/862802385 shutdown_connections 2023-10-30T13:42:20.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f67600664a0 0x7f6760068960 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f678013f4a0 0x7f67800c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67800c6f60 0x7f67801f60b0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 --2- 172.21.15.46:0/862802385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67800c6b80 0x7f67801f5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 -- 172.21.15.46:0/862802385 >> 172.21.15.46:0/862802385 conn(0x7f67800c7910 msgr2=0x7f6780069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 -- 172.21.15.46:0/862802385 shutdown_connections 2023-10-30T13:42:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:20.009+0000 7f6785417640 1 -- 172.21.15.46:0/862802385 wait complete. 2023-10-30T13:42:20.035 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:22.036 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:22.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.173+0000 7f6e4bcea640 1 Processor -- start 2023-10-30T13:42:22.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.173+0000 7f6e4bcea640 1 -- start start 2023-10-30T13:42:22.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e4bcea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e440cf160 0x7f6e440cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:22.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e4bcea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e440cb800 con 0x7f6e441538a0 2023-10-30T13:42:22.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e4bcea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e440cb330 con 0x7f6e4415d6f0 2023-10-30T13:42:22.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e4bcea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e440cad70 con 0x7f6e440cf160 2023-10-30T13:42:22.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e49a86640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6e441538a0 0x7f6e440ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60054/0 (socket says 172.21.15.46:60054) 2023-10-30T13:42:22.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e49a86640 1 -- 172.21.15.46:0/3599150598 learned_addr learned my addr 172.21.15.46:0/3599150598 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:22.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e4a287640 1 --2- 172.21.15.46:0/3599150598 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e440cf160 0x7f6e440cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:22.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e48a84640 1 -- 172.21.15.46:0/3599150598 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564334705 0 0) 0x7f6e440cb800 con 0x7f6e441538a0 2023-10-30T13:42:22.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e48a84640 1 -- 172.21.15.46:0/3599150598 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e30003530 con 0x7f6e441538a0 2023-10-30T13:42:22.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e48a84640 1 -- 172.21.15.46:0/3599150598 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648565200 0 0) 0x7f6e440cb330 con 0x7f6e4415d6f0 2023-10-30T13:42:22.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e48a84640 1 -- 172.21.15.46:0/3599150598 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e440cb800 con 0x7f6e4415d6f0 2023-10-30T13:42:22.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e48a84640 1 -- 172.21.15.46:0/3599150598 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3765949764 0 0) 0x7f6e30003530 con 0x7f6e441538a0 2023-10-30T13:42:22.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e4a287640 1 -- 172.21.15.46:0/3599150598 >> v1:172.21.15.154:6789/0 conn(0x7f6e4415d6f0 legacy=0x7f6e4415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e4a287640 1 -- 172.21.15.46:0/3599150598 >> v1:172.21.15.46:6789/0 conn(0x7f6e441538a0 legacy=0x7f6e440ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.177+0000 7f6e4a287640 1 -- 172.21.15.46:0/3599150598 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e440cb050 con 0x7f6e440cf160 2023-10-30T13:42:22.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.181+0000 7f6e4a287640 1 --2- 172.21.15.46:0/3599150598 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e440cf160 0x7f6e440cf590 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c009dc0 tx=0x7f6e3c01f080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e71a9aa56ad72388 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:22.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.181+0000 7f6e48a84640 1 -- 172.21.15.46:0/3599150598 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e40001680 con 0x7f6e440cf160 2023-10-30T13:42:22.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.181+0000 7f6e48a84640 1 -- 172.21.15.46:0/3599150598 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c0a8c10 con 0x7f6e440cf160 2023-10-30T13:42:22.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.181+0000 7f6e48a84640 1 -- 172.21.15.46:0/3599150598 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c0b0440 con 0x7f6e440cf160 2023-10-30T13:42:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.181+0000 7f6e4bcea640 1 -- 172.21.15.46:0/3599150598 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e440cf160 msgr2=0x7f6e440cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.181+0000 7f6e4bcea640 1 --2- 172.21.15.46:0/3599150598 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e440cf160 0x7f6e440cf590 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c009dc0 tx=0x7f6e3c01f080 comp rx=0 tx=0).stop 2023-10-30T13:42:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 -- 172.21.15.46:0/3599150598 shutdown_connections 2023-10-30T13:42:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 --2- 172.21.15.46:0/3599150598 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e440cf160 0x7f6e440cf590 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 -- 172.21.15.46:0/3599150598 >> 172.21.15.46:0/3599150598 conn(0x7f6e4406bfe0 msgr2=0x7f6e4406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 -- 172.21.15.46:0/3599150598 shutdown_connections 2023-10-30T13:42:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 -- 172.21.15.46:0/3599150598 wait complete. 2023-10-30T13:42:22.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 Processor -- start 2023-10-30T13:42:22.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 -- start start 2023-10-30T13:42:22.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e440cf160 0x7f6e4415b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:22.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e441538a0 0x7f6e44157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:22.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e4415d6f0 0x7f6e44158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:22.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e440c5670 con 0x7f6e441538a0 2023-10-30T13:42:22.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e441f3330 con 0x7f6e440cf160 2023-10-30T13:42:22.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e4bcea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e440b4c80 con 0x7f6e4415d6f0 2023-10-30T13:42:22.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.185+0000 7f6e49285640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e441538a0 0x7f6e44157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:22.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e49285640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e441538a0 0x7f6e44157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44392/0 (socket says 172.21.15.46:44392) 2023-10-30T13:42:22.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e49285640 1 -- 172.21.15.46:0/4071965463 learned_addr learned my addr 172.21.15.46:0/4071965463 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:22.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e4a287640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e4415d6f0 0x7f6e44158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:22.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e49a86640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e440cf160 0x7f6e4415b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:22.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e49285640 1 -- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e4415d6f0 msgr2=0x7f6e44158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e49285640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e4415d6f0 0x7f6e44158370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e49285640 1 -- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e440cf160 msgr2=0x7f6e4415b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e49285640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e440cf160 0x7f6e4415b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e49285640 1 -- 172.21.15.46:0/4071965463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3c009a70 con 0x7f6e441538a0 2023-10-30T13:42:22.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e49a86640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e440cf160 0x7f6e4415b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:22.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e49285640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e441538a0 0x7f6e44157e30 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400c490 tx=0x7f6e3400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:22.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e3a7fc640 1 -- 172.21.15.46:0/4071965463 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e340052d0 con 0x7f6e441538a0 2023-10-30T13:42:22.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e3a7fc640 1 -- 172.21.15.46:0/4071965463 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e34005430 con 0x7f6e441538a0 2023-10-30T13:42:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e4bcea640 1 -- 172.21.15.46:0/4071965463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e440b4f60 con 0x7f6e441538a0 2023-10-30T13:42:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e3a7fc640 1 -- 172.21.15.46:0/4071965463 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e34012870 con 0x7f6e441538a0 2023-10-30T13:42:22.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e4bcea640 1 -- 172.21.15.46:0/4071965463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e441588b0 con 0x7f6e441538a0 2023-10-30T13:42:22.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e4bcea640 1 -- 172.21.15.46:0/4071965463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e440b51a0 con 0x7f6e441538a0 2023-10-30T13:42:22.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.189+0000 7f6e3a7fc640 1 -- 172.21.15.46:0/4071965463 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6e340129d0 con 0x7f6e441538a0 2023-10-30T13:42:22.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.193+0000 7f6e3a7fc640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e280664a0 0x7f6e28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:22.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.193+0000 7f6e3a7fc640 1 -- 172.21.15.46:0/4071965463 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f6e34083e20 con 0x7f6e441538a0 2023-10-30T13:42:22.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.193+0000 7f6e49a86640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e280664a0 0x7f6e28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:22.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.193+0000 7f6e49a86640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e280664a0 0x7f6e28068960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6e40001ec0 tx=0x7f6e40009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:22.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.197+0000 7f6e3a7fc640 1 -- 172.21.15.46:0/4071965463 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6e440b51a0 con 0x7f6e441538a0 2023-10-30T13:42:22.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 -- 172.21.15.46:0/4071965463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e44157770 con 0x7f6e441538a0 2023-10-30T13:42:22.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e3a7fc640 1 -- 172.21.15.46:0/4071965463 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7f6e44157770 con 0x7f6e441538a0 2023-10-30T13:42:22.438 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:22.438 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:22.439 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:22.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 -- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e280664a0 msgr2=0x7f6e28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e280664a0 0x7f6e28068960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6e40001ec0 tx=0x7f6e40009040 comp rx=0 tx=0).stop 2023-10-30T13:42:22.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 -- 172.21.15.46:0/4071965463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e441538a0 msgr2=0x7f6e44157e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e441538a0 0x7f6e44157e30 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400c490 tx=0x7f6e3400c950 comp rx=0 tx=0).stop 2023-10-30T13:42:22.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 -- 172.21.15.46:0/4071965463 shutdown_connections 2023-10-30T13:42:22.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e280664a0 0x7f6e28068960 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e4415d6f0 0x7f6e44158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e441538a0 0x7f6e44157e30 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 --2- 172.21.15.46:0/4071965463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e440cf160 0x7f6e4415b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 -- 172.21.15.46:0/4071965463 >> 172.21.15.46:0/4071965463 conn(0x7f6e4406bfe0 msgr2=0x7f6e440ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:22.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 -- 172.21.15.46:0/4071965463 shutdown_connections 2023-10-30T13:42:22.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:22.433+0000 7f6e4bcea640 1 -- 172.21.15.46:0/4071965463 wait complete. 2023-10-30T13:42:22.458 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:24.459 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:24.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.609+0000 7f3955a54640 1 Processor -- start 2023-10-30T13:42:24.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.609+0000 7f3955a54640 1 -- start start 2023-10-30T13:42:24.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.609+0000 7f3955a54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3950153d70 0x7f395006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.609+0000 7f3955a54640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f395006e470 0x7f39500cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.609+0000 7f3955a54640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395006afa0 con 0x7f39500c8490 2023-10-30T13:42:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.609+0000 7f3955a54640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f395006d780 con 0x7f395006e470 2023-10-30T13:42:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.609+0000 7f3955a54640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3950154150 con 0x7f3950153d70 2023-10-30T13:42:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.609+0000 7f3947fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f395006e470 0x7f39500cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:24.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.609+0000 7f3947fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f395006e470 0x7f39500cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44402/0 (socket says 172.21.15.46:44402) 2023-10-30T13:42:24.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.609+0000 7f3947fff640 1 -- 172.21.15.46:0/3300384700 learned_addr learned my addr 172.21.15.46:0/3300384700 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:24.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.609+0000 7f394effd640 1 --2- 172.21.15.46:0/3300384700 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3950153d70 0x7f395006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:24.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.613+0000 7f3947fff640 1 -- 172.21.15.46:0/3300384700 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3950153d70 msgr2=0x7f395006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:24.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.613+0000 7f3947fff640 1 --2- 172.21.15.46:0/3300384700 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3950153d70 0x7f395006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.613+0000 7f3947fff640 1 -- 172.21.15.46:0/3300384700 >> v1:172.21.15.154:6789/0 conn(0x7f39500c8490 legacy=0x7f39500c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:24.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.613+0000 7f3947fff640 1 -- 172.21.15.46:0/3300384700 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395006e030 con 0x7f395006e470 2023-10-30T13:42:24.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.613+0000 7f3947fff640 1 --2- 172.21.15.46:0/3300384700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395006e470 0x7f39500cce00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f3938009870 tx=0x7f393801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269be6e670395d08 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:24.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.613+0000 7f394e7fc640 1 -- 172.21.15.46:0/3300384700 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39380aa670 con 0x7f395006e470 2023-10-30T13:42:24.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.613+0000 7f394e7fc640 1 -- 172.21.15.46:0/3300384700 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39380aa7d0 con 0x7f395006e470 2023-10-30T13:42:24.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.613+0000 7f394e7fc640 1 -- 172.21.15.46:0/3300384700 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39380aab60 con 0x7f395006e470 2023-10-30T13:42:24.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.613+0000 7f3955a54640 1 -- 172.21.15.46:0/3300384700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395006e470 msgr2=0x7f39500cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:24.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.613+0000 7f3955a54640 1 --2- 172.21.15.46:0/3300384700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395006e470 0x7f39500cce00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f3938009870 tx=0x7f393801c510 comp rx=0 tx=0).stop 2023-10-30T13:42:24.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 -- 172.21.15.46:0/3300384700 shutdown_connections 2023-10-30T13:42:24.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 --2- 172.21.15.46:0/3300384700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395006e470 0x7f39500cce00 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 --2- 172.21.15.46:0/3300384700 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3950153d70 0x7f395006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 -- 172.21.15.46:0/3300384700 >> 172.21.15.46:0/3300384700 conn(0x7f3950071ea0 msgr2=0x7f3950065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:24.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 -- 172.21.15.46:0/3300384700 shutdown_connections 2023-10-30T13:42:24.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 -- 172.21.15.46:0/3300384700 wait complete. 2023-10-30T13:42:24.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 Processor -- start 2023-10-30T13:42:24.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 -- start start 2023-10-30T13:42:24.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395006e470 0x7f3950154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:24.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39500c8490 0x7f3950155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:24.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3950153d70 0x7f39501f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:24.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39500b0210 con 0x7f395006e470 2023-10-30T13:42:24.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39501542f0 con 0x7f39500c8490 2023-10-30T13:42:24.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3950154870 con 0x7f3950153d70 2023-10-30T13:42:24.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395006e470 0x7f3950154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:24.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395006e470 0x7f3950154d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44416/0 (socket says 172.21.15.46:44416) 2023-10-30T13:42:24.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394effd640 1 -- 172.21.15.46:0/2886737956 learned_addr learned my addr 172.21.15.46:0/2886737956 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:24.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3947fff640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39500c8490 0x7f3950155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394f7fe640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3950153d70 0x7f39501f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394effd640 1 -- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3950153d70 msgr2=0x7f39501f60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394effd640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3950153d70 0x7f39501f60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394effd640 1 -- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39500c8490 msgr2=0x7f3950155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394effd640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39500c8490 0x7f3950155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394effd640 1 -- 172.21.15.46:0/2886737956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39380094d0 con 0x7f395006e470 2023-10-30T13:42:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394effd640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395006e470 0x7f3950154d00 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f394000e870 tx=0x7f394000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394cff9640 1 -- 172.21.15.46:0/2886737956 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3940009510 con 0x7f395006e470 2023-10-30T13:42:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394cff9640 1 -- 172.21.15.46:0/2886737956 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394000eec0 con 0x7f395006e470 2023-10-30T13:42:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f3955a54640 1 -- 172.21.15.46:0/2886737956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39500b17c0 con 0x7f395006e470 2023-10-30T13:42:24.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.617+0000 7f394cff9640 1 -- 172.21.15.46:0/2886737956 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394000cb40 con 0x7f395006e470 2023-10-30T13:42:24.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.621+0000 7f3955a54640 1 -- 172.21.15.46:0/2886737956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3950156f90 con 0x7f395006e470 2023-10-30T13:42:24.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.621+0000 7f3955a54640 1 -- 172.21.15.46:0/2886737956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395004e170 con 0x7f395006e470 2023-10-30T13:42:24.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.621+0000 7f394cff9640 1 -- 172.21.15.46:0/2886737956 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f394000cca0 con 0x7f395006e470 2023-10-30T13:42:24.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.621+0000 7f394cff9640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f39300664a0 0x7f3930068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:24.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.621+0000 7f394cff9640 1 -- 172.21.15.46:0/2886737956 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f3940084460 con 0x7f395006e470 2023-10-30T13:42:24.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.621+0000 7f3947fff640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f39300664a0 0x7f3930068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:24.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.625+0000 7f3947fff640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f39300664a0 0x7f3930068960 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f39380ae000 tx=0x7f39380a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:24.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.625+0000 7f394cff9640 1 -- 172.21.15.46:0/2886737956 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3940051450 con 0x7f395006e470 2023-10-30T13:42:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 -- 172.21.15.46:0/2886737956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3950157180 con 0x7f395006e470 2023-10-30T13:42:24.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f394cff9640 1 -- 172.21.15.46:0/2886737956 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7f3950157180 con 0x7f395006e470 2023-10-30T13:42:24.821 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:24.822 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 -- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f39300664a0 msgr2=0x7f3930068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f39300664a0 0x7f3930068960 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f39380ae000 tx=0x7f39380a7000 comp rx=0 tx=0).stop 2023-10-30T13:42:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 -- 172.21.15.46:0/2886737956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395006e470 msgr2=0x7f3950154d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395006e470 0x7f3950154d00 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f394000e870 tx=0x7f394000ec70 comp rx=0 tx=0).stop 2023-10-30T13:42:24.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 -- 172.21.15.46:0/2886737956 shutdown_connections 2023-10-30T13:42:24.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f39300664a0 0x7f3930068960 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3950153d70 0x7f39501f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39500c8490 0x7f3950155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 --2- 172.21.15.46:0/2886737956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395006e470 0x7f3950154d00 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 -- 172.21.15.46:0/2886737956 >> 172.21.15.46:0/2886737956 conn(0x7f3950071ea0 msgr2=0x7f39500c7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:24.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 -- 172.21.15.46:0/2886737956 shutdown_connections 2023-10-30T13:42:24.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:24.817+0000 7f3955a54640 1 -- 172.21.15.46:0/2886737956 wait complete. 2023-10-30T13:42:24.842 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:26.843 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:26.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.981+0000 7feb795d3640 1 Processor -- start 2023-10-30T13:42:26.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.981+0000 7feb795d3640 1 -- start start 2023-10-30T13:42:26.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb795d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feb740c05d0 0x7feb7413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:26.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb795d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb740c6f60 0x7feb7413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:26.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb795d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb740c0a50 con 0x7feb740c6b80 2023-10-30T13:42:26.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb795d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb7406f6e0 con 0x7feb740c6f60 2023-10-30T13:42:26.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb795d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb7406f9c0 con 0x7feb740c05d0 2023-10-30T13:42:26.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb737fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb740c6f60 0x7feb7413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:26.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb737fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb740c6f60 0x7feb7413f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37548/0 (socket says 172.21.15.46:37548) 2023-10-30T13:42:26.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb737fe640 1 -- 172.21.15.46:0/3269505090 learned_addr learned my addr 172.21.15.46:0/3269505090 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:26.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb6bfff640 1 --2- 172.21.15.46:0/3269505090 >> v2:172.21.15.154:3300/0 conn(0x7feb740c05d0 0x7feb7413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:26.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb737fe640 1 -- 172.21.15.46:0/3269505090 >> v2:172.21.15.154:3300/0 conn(0x7feb740c05d0 msgr2=0x7feb7413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:26.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb737fe640 1 --2- 172.21.15.46:0/3269505090 >> v2:172.21.15.154:3300/0 conn(0x7feb740c05d0 0x7feb7413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:26.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb737fe640 1 -- 172.21.15.46:0/3269505090 >> v1:172.21.15.154:6789/0 conn(0x7feb740c6b80 legacy=0x7feb740bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:26.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.985+0000 7feb737fe640 1 -- 172.21.15.46:0/3269505090 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7406fca0 con 0x7feb740c6f60 2023-10-30T13:42:26.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb737fe640 1 --2- 172.21.15.46:0/3269505090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb740c6f60 0x7feb7413f2d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7feb7406c7e0 tx=0x7feb6c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332acd6231793c1c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:26.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb727fc640 1 -- 172.21.15.46:0/3269505090 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0a57a0 con 0x7feb740c6f60 2023-10-30T13:42:26.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb727fc640 1 -- 172.21.15.46:0/3269505090 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c0a5900 con 0x7feb740c6f60 2023-10-30T13:42:26.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb727fc640 1 -- 172.21.15.46:0/3269505090 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0a5c90 con 0x7feb740c6f60 2023-10-30T13:42:26.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb795d3640 1 -- 172.21.15.46:0/3269505090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb740c6f60 msgr2=0x7feb7413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:26.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb795d3640 1 --2- 172.21.15.46:0/3269505090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb740c6f60 0x7feb7413f2d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7feb7406c7e0 tx=0x7feb6c01c9d0 comp rx=0 tx=0).stop 2023-10-30T13:42:26.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb795d3640 1 -- 172.21.15.46:0/3269505090 shutdown_connections 2023-10-30T13:42:26.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb795d3640 1 --2- 172.21.15.46:0/3269505090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb740c6f60 0x7feb7413f2d0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:26.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb795d3640 1 --2- 172.21.15.46:0/3269505090 >> v2:172.21.15.154:3300/0 conn(0x7feb740c05d0 0x7feb7413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:26.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb795d3640 1 -- 172.21.15.46:0/3269505090 >> 172.21.15.46:0/3269505090 conn(0x7feb740c7910 msgr2=0x7feb740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:26.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb795d3640 1 -- 172.21.15.46:0/3269505090 shutdown_connections 2023-10-30T13:42:26.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb795d3640 1 -- 172.21.15.46:0/3269505090 wait complete. 2023-10-30T13:42:26.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb795d3640 1 Processor -- start 2023-10-30T13:42:26.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.989+0000 7feb795d3640 1 -- start start 2023-10-30T13:42:26.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb795d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb740c05d0 0x7feb74154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:26.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb795d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb740c6b80 0x7feb74155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:26.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb795d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb740c6f60 0x7feb74137b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:26.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb795d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7413dcc0 con 0x7feb740c6f60 2023-10-30T13:42:26.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb795d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb741549b0 con 0x7feb740c6b80 2023-10-30T13:42:26.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb795d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7413e7b0 con 0x7feb740c05d0 2023-10-30T13:42:26.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb737fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb740c6f60 0x7feb74137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:26.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb737fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb740c6f60 0x7feb74137b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37556/0 (socket says 172.21.15.46:37556) 2023-10-30T13:42:26.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb737fe640 1 -- 172.21.15.46:0/1462297088 learned_addr learned my addr 172.21.15.46:0/1462297088 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:27.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb6bfff640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb740c6b80 0x7feb74155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:27.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb72ffd640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb740c05d0 0x7feb74154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:27.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb737fe640 1 -- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb740c05d0 msgr2=0x7feb74154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:27.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb737fe640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb740c05d0 0x7feb74154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb737fe640 1 -- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb740c6b80 msgr2=0x7feb74155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb737fe640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb740c6b80 0x7feb74155380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb737fe640 1 -- 172.21.15.46:0/1462297088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c009b30 con 0x7feb740c6f60 2023-10-30T13:42:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb6bfff640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb740c6b80 0x7feb74155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb737fe640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb740c6f60 0x7feb74137b10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7feb6c01a600 tx=0x7feb6c019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb6b7fe640 1 -- 172.21.15.46:0/1462297088 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0b6890 con 0x7feb740c6f60 2023-10-30T13:42:27.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb6b7fe640 1 -- 172.21.15.46:0/1462297088 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c0b6a30 con 0x7feb740c6f60 2023-10-30T13:42:27.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb6b7fe640 1 -- 172.21.15.46:0/1462297088 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0b6de0 con 0x7feb740c6f60 2023-10-30T13:42:27.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb795d3640 1 -- 172.21.15.46:0/1462297088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7413df10 con 0x7feb740c6f60 2023-10-30T13:42:27.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb795d3640 1 -- 172.21.15.46:0/1462297088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7413e4d0 con 0x7feb740c6f60 2023-10-30T13:42:27.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb795d3640 1 -- 172.21.15.46:0/1462297088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7404e170 con 0x7feb740c6f60 2023-10-30T13:42:27.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.993+0000 7feb6b7fe640 1 -- 172.21.15.46:0/1462297088 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feb6c0c1460 con 0x7feb740c6f60 2023-10-30T13:42:27.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.997+0000 7feb6b7fe640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb540664d0 0x7feb54068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:27.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.997+0000 7feb6b7fe640 1 -- 172.21.15.46:0/1462297088 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7feb6c11f5f0 con 0x7feb740c6f60 2023-10-30T13:42:27.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.997+0000 7feb72ffd640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb540664d0 0x7feb54068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:27.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:26.997+0000 7feb72ffd640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb540664d0 0x7feb54068990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7feb64000f70 tx=0x7feb64009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:27.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.001+0000 7feb6b7fe640 1 -- 172.21.15.46:0/1462297088 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feb6c0ec790 con 0x7feb740c6f60 2023-10-30T13:42:27.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 -- 172.21.15.46:0/1462297088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb74154af0 con 0x7feb740c6f60 2023-10-30T13:42:27.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb6b7fe640 1 -- 172.21.15.46:0/1462297088 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7feb6c0efdb0 con 0x7feb740c6f60 2023-10-30T13:42:27.197 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:27.197 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:27.198 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 -- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb540664d0 msgr2=0x7feb54068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:27.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb540664d0 0x7feb54068990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7feb64000f70 tx=0x7feb64009040 comp rx=0 tx=0).stop 2023-10-30T13:42:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 -- 172.21.15.46:0/1462297088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb740c6f60 msgr2=0x7feb74137b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb740c6f60 0x7feb74137b10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7feb6c01a600 tx=0x7feb6c019770 comp rx=0 tx=0).stop 2023-10-30T13:42:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 -- 172.21.15.46:0/1462297088 shutdown_connections 2023-10-30T13:42:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb540664d0 0x7feb54068990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb740c6f60 0x7feb74137b10 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb740c6b80 0x7feb74155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 --2- 172.21.15.46:0/1462297088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb740c05d0 0x7feb74154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 -- 172.21.15.46:0/1462297088 >> 172.21.15.46:0/1462297088 conn(0x7feb740c7910 msgr2=0x7feb74139260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:27.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 -- 172.21.15.46:0/1462297088 shutdown_connections 2023-10-30T13:42:27.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:27.193+0000 7feb795d3640 1 -- 172.21.15.46:0/1462297088 wait complete. 2023-10-30T13:42:27.219 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:29.220 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:29.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.369+0000 7fd1e18ea640 1 Processor -- start 2023-10-30T13:42:29.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.369+0000 7fd1e18ea640 1 -- start start 2023-10-30T13:42:29.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.369+0000 7fd1e18ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1dc06e470 0x7fd1dc06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:29.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.369+0000 7fd1e18ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd1dc0c8490 0x7fd1dc0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:29.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.369+0000 7fd1e18ea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1dc06afa0 con 0x7fd1dc153d70 2023-10-30T13:42:29.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.369+0000 7fd1e18ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1dc06d780 con 0x7fd1dc0c8490 2023-10-30T13:42:29.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.369+0000 7fd1e18ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc154150 con 0x7fd1dc06e470 2023-10-30T13:42:29.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.373+0000 7fd1db7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd1dc0c8490 0x7fd1dc0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:29.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.373+0000 7fd1d3fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1dc06e470 0x7fd1dc06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:29.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.373+0000 7fd1daffd640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd1dc153d70 0x7fd1dc0cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:35644/0 (socket says 172.21.15.46:35644) 2023-10-30T13:42:29.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.373+0000 7fd1daffd640 1 -- 172.21.15.46:0/1064227832 learned_addr learned my addr 172.21.15.46:0/1064227832 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:29.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.373+0000 7fd1db7fe640 1 -- 172.21.15.46:0/1064227832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1dc06e470 msgr2=0x7fd1dc06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:29.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.373+0000 7fd1db7fe640 1 --2- 172.21.15.46:0/1064227832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1dc06e470 0x7fd1dc06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.373+0000 7fd1db7fe640 1 -- 172.21.15.46:0/1064227832 >> v1:172.21.15.154:6789/0 conn(0x7fd1dc153d70 legacy=0x7fd1dc0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:29.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.373+0000 7fd1db7fe640 1 -- 172.21.15.46:0/1064227832 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc06e030 con 0x7fd1dc0c8490 2023-10-30T13:42:29.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.373+0000 7fd1d3fff640 1 --2- 172.21.15.46:0/1064227832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1dc06e470 0x7fd1dc06ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:42:29.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.373+0000 7fd1db7fe640 1 --2- 172.21.15.46:0/1064227832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc0c8490 0x7fd1dc0c8cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4009d90 tx=0x7fd1d401c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecdf2e11ad1a0aaa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:29.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1da7fc640 1 -- 172.21.15.46:0/1064227832 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d40a59e0 con 0x7fd1dc0c8490 2023-10-30T13:42:29.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1da7fc640 1 -- 172.21.15.46:0/1064227832 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d40a5b40 con 0x7fd1dc0c8490 2023-10-30T13:42:29.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1da7fc640 1 -- 172.21.15.46:0/1064227832 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d40ad440 con 0x7fd1dc0c8490 2023-10-30T13:42:29.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 -- 172.21.15.46:0/1064227832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc0c8490 msgr2=0x7fd1dc0c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:29.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 --2- 172.21.15.46:0/1064227832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc0c8490 0x7fd1dc0c8cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4009d90 tx=0x7fd1d401c920 comp rx=0 tx=0).stop 2023-10-30T13:42:29.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 -- 172.21.15.46:0/1064227832 shutdown_connections 2023-10-30T13:42:29.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 --2- 172.21.15.46:0/1064227832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc0c8490 0x7fd1dc0c8cc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 --2- 172.21.15.46:0/1064227832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1dc06e470 0x7fd1dc06ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 -- 172.21.15.46:0/1064227832 >> 172.21.15.46:0/1064227832 conn(0x7fd1dc071ea0 msgr2=0x7fd1dc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:29.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 -- 172.21.15.46:0/1064227832 shutdown_connections 2023-10-30T13:42:29.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 -- 172.21.15.46:0/1064227832 wait complete. 2023-10-30T13:42:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 Processor -- start 2023-10-30T13:42:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 -- start start 2023-10-30T13:42:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1dc06e470 0x7fd1dc1524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc0c8490 0x7fd1dc14bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc153d70 0x7fd1dc14c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc14aa00 con 0x7fd1dc0c8490 2023-10-30T13:42:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc14e280 con 0x7fd1dc153d70 2023-10-30T13:42:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.377+0000 7fd1e18ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc14ac50 con 0x7fd1dc06e470 2023-10-30T13:42:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc0c8490 0x7fd1dc14bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc0c8490 0x7fd1dc14bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37570/0 (socket says 172.21.15.46:37570) 2023-10-30T13:42:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1d3fff640 1 -- 172.21.15.46:0/3216529084 learned_addr learned my addr 172.21.15.46:0/3216529084 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1db7fe640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc153d70 0x7fd1dc14c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1daffd640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1dc06e470 0x7fd1dc1524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1d3fff640 1 -- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1dc06e470 msgr2=0x7fd1dc1524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1d3fff640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1dc06e470 0x7fd1dc1524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1d3fff640 1 -- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc153d70 msgr2=0x7fd1dc14c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:29.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1d3fff640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc153d70 0x7fd1dc14c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1d3fff640 1 -- 172.21.15.46:0/3216529084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c4009420 con 0x7fd1dc0c8490 2023-10-30T13:42:29.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1d3fff640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc0c8490 0x7fd1dc14bf60 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400ebe0 tx=0x7fd1c400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:29.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1d8ff9640 1 -- 172.21.15.46:0/3216529084 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c400eee0 con 0x7fd1dc0c8490 2023-10-30T13:42:29.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1d8ff9640 1 -- 172.21.15.46:0/3216529084 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c400cc30 con 0x7fd1dc0c8490 2023-10-30T13:42:29.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1e18ea640 1 -- 172.21.15.46:0/3216529084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d4009a70 con 0x7fd1dc0c8490 2023-10-30T13:42:29.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1d8ff9640 1 -- 172.21.15.46:0/3216529084 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4013470 con 0x7fd1dc0c8490 2023-10-30T13:42:29.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1e18ea640 1 -- 172.21.15.46:0/3216529084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1dc14b210 con 0x7fd1dc0c8490 2023-10-30T13:42:29.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.381+0000 7fd1e18ea640 1 -- 172.21.15.46:0/3216529084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1dc067280 con 0x7fd1dc0c8490 2023-10-30T13:42:29.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.385+0000 7fd1d8ff9640 1 -- 172.21.15.46:0/3216529084 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd1c40135d0 con 0x7fd1dc0c8490 2023-10-30T13:42:29.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.385+0000 7fd1d8ff9640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1bc0664a0 0x7fd1bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:29.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.385+0000 7fd1d8ff9640 1 -- 172.21.15.46:0/3216529084 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fd1c4084950 con 0x7fd1dc0c8490 2023-10-30T13:42:29.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.385+0000 7fd1daffd640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1bc0664a0 0x7fd1bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:29.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.385+0000 7fd1daffd640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1bc0664a0 0x7fd1bc068960 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc005890 tx=0x7fd1cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:29.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.389+0000 7fd1d8ff9640 1 -- 172.21.15.46:0/3216529084 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd1c40519f0 con 0x7fd1dc0c8490 2023-10-30T13:42:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.585+0000 7fd1e18ea640 1 -- 172.21.15.46:0/3216529084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1dc04e170 con 0x7fd1dc0c8490 2023-10-30T13:42:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.585+0000 7fd1d8ff9640 1 -- 172.21.15.46:0/3216529084 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7fd1c4055010 con 0x7fd1dc0c8490 2023-10-30T13:42:29.591 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:29.591 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:29.592 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:29.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.589+0000 7fd1e18ea640 1 -- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1bc0664a0 msgr2=0x7fd1bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:29.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.589+0000 7fd1e18ea640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1bc0664a0 0x7fd1bc068960 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc005890 tx=0x7fd1cc009040 comp rx=0 tx=0).stop 2023-10-30T13:42:29.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.589+0000 7fd1e18ea640 1 -- 172.21.15.46:0/3216529084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc0c8490 msgr2=0x7fd1dc14bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:29.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.589+0000 7fd1e18ea640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc0c8490 0x7fd1dc14bf60 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400ebe0 tx=0x7fd1c400c520 comp rx=0 tx=0).stop 2023-10-30T13:42:29.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.589+0000 7fd1e18ea640 1 -- 172.21.15.46:0/3216529084 shutdown_connections 2023-10-30T13:42:29.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.589+0000 7fd1e18ea640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1bc0664a0 0x7fd1bc068960 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.589+0000 7fd1e18ea640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc153d70 0x7fd1dc14c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.589+0000 7fd1e18ea640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc0c8490 0x7fd1dc14bf60 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.589+0000 7fd1e18ea640 1 --2- 172.21.15.46:0/3216529084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1dc06e470 0x7fd1dc1524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.589+0000 7fd1e18ea640 1 -- 172.21.15.46:0/3216529084 >> 172.21.15.46:0/3216529084 conn(0x7fd1dc071ea0 msgr2=0x7fd1dc06f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:29.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.589+0000 7fd1e18ea640 1 -- 172.21.15.46:0/3216529084 shutdown_connections 2023-10-30T13:42:29.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:29.589+0000 7fd1e18ea640 1 -- 172.21.15.46:0/3216529084 wait complete. 2023-10-30T13:42:29.612 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:42:29.612 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 98, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 98, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:stopping", "state_seq": 28 }, "gid_6036": { "addr": "172.21.15.46:6836/3037060424", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3037060424, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3037060424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6036, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 37 }, "gid_6201": { "addr": "172.21.15.46:6837/2546436963", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2546436963, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2546436963, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6201, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:42:05.189550+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6201, "mds_1": 6036, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.46:6833/190134548", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 190134548, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 190134548, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6366, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:42:31.614 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.765+0000 7fc2af668640 1 Processor -- start 2023-10-30T13:42:31.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.765+0000 7fc2af668640 1 -- start start 2023-10-30T13:42:31.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.765+0000 7fc2af668640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2a81580e0 0x7fc2a8070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.765+0000 7fc2af668640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a80c5810 con 0x7fc2a80c6540 2023-10-30T13:42:31.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.765+0000 7fc2af668640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a815eb90 con 0x7fc2a815e700 2023-10-30T13:42:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.765+0000 7fc2af668640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a806acb0 con 0x7fc2a81580e0 2023-10-30T13:42:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.765+0000 7fc2acc03640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc2a80c6540 0x7fc2a8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49872/0 (socket says 172.21.15.46:49872) 2023-10-30T13:42:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.765+0000 7fc2acc03640 1 -- 172.21.15.46:0/3630432002 learned_addr learned my addr 172.21.15.46:0/3630432002 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.765+0000 7fc2ad404640 1 --2- 172.21.15.46:0/3630432002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2a81580e0 0x7fc2a8070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.765+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679550429 0 0) 0x7fc2a80c5810 con 0x7fc2a80c6540 2023-10-30T13:42:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.765+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc294003530 con 0x7fc2a80c6540 2023-10-30T13:42:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549467322 0 0) 0x7fc2a815eb90 con 0x7fc2a815e700 2023-10-30T13:42:31.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a80c5810 con 0x7fc2a815e700 2023-10-30T13:42:31.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2629627537 0 0) 0x7fc294003530 con 0x7fc2a80c6540 2023-10-30T13:42:31.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2a81580e0 msgr2=0x7fc2a8070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:31.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 --2- 172.21.15.46:0/3630432002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2a81580e0 0x7fc2a8070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 >> v1:172.21.15.154:6789/0 conn(0x7fc2a815e700 legacy=0x7fc2a806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a80c6960 con 0x7fc2a80c6540 2023-10-30T13:42:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fc298001680 con 0x7fc2a80c6540 2023-10-30T13:42:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 >> v1:172.21.15.46:6789/0 conn(0x7fc2a80c6540 legacy=0x7fc2a8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 --2- 172.21.15.46:0/3630432002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc294003b40 0x7fc294003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 --2- 172.21.15.46:0/3630432002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2940047d0 0x7fc294004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 --2- 172.21.15.46:0/3630432002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2a81580e0 0x7fc294005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29400ba40 con 0x7fc2940047d0 2023-10-30T13:42:31.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29400bb80 con 0x7fc294003b40 2023-10-30T13:42:31.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29400bd10 con 0x7fc2a81580e0 2023-10-30T13:42:31.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc2acc03640 1 --2- 172.21.15.46:0/3630432002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2940047d0 0x7fc294004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc2ad404640 1 --2- 172.21.15.46:0/3630432002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc294003b40 0x7fc294003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc2adc05640 1 --2- 172.21.15.46:0/3630432002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2a81580e0 0x7fc294005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc2acc03640 1 -- 172.21.15.46:0/3630432002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2a81580e0 msgr2=0x7fc294005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:31.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc2acc03640 1 --2- 172.21.15.46:0/3630432002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2a81580e0 0x7fc294005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:31.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc2acc03640 1 -- 172.21.15.46:0/3630432002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc294003b40 msgr2=0x7fc294003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:31.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc2acc03640 1 --2- 172.21.15.46:0/3630432002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc294003b40 0x7fc294003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:31.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc2acc03640 1 -- 172.21.15.46:0/3630432002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc294003530 con 0x7fc2940047d0 2023-10-30T13:42:31.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.769+0000 7fc2acc03640 1 --2- 172.21.15.46:0/3630432002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2940047d0 0x7fc294004c50 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc29800c190 tx=0x7fc29801cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e460a20d76e93b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:31.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2980058e0 con 0x7fc2940047d0 2023-10-30T13:42:31.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc298005a40 con 0x7fc2940047d0 2023-10-30T13:42:31.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc29ffff640 1 -- 172.21.15.46:0/3630432002 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298005e10 con 0x7fc2940047d0 2023-10-30T13:42:31.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc2af668640 1 -- 172.21.15.46:0/3630432002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2940047d0 msgr2=0x7fc294004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:31.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc2af668640 1 --2- 172.21.15.46:0/3630432002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2940047d0 0x7fc294004c50 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc29800c190 tx=0x7fc29801cb40 comp rx=0 tx=0).stop 2023-10-30T13:42:31.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc2adc05640 1 -- 172.21.15.46:0/3630432002 reap_dead start 2023-10-30T13:42:31.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc2af668640 1 -- 172.21.15.46:0/3630432002 shutdown_connections 2023-10-30T13:42:31.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc2af668640 1 -- 172.21.15.46:0/3630432002 >> 172.21.15.46:0/3630432002 conn(0x7fc2a80c7910 msgr2=0x7fc2a80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:31.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc2af668640 1 -- 172.21.15.46:0/3630432002 shutdown_connections 2023-10-30T13:42:31.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc2af668640 1 -- 172.21.15.46:0/3630432002 wait complete. 2023-10-30T13:42:31.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc2af668640 1 Processor -- start 2023-10-30T13:42:31.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc2af668640 1 -- start start 2023-10-30T13:42:31.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.773+0000 7fc2af668640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2940047d0 0x7fc2a80717d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2af668640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2a80c20b0 0x7fc2a81f6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2af668640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2a81f7530 0x7fc2a81f37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2af668640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a81f68c0 con 0x7fc2a80c20b0 2023-10-30T13:42:31.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2af668640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8151780 con 0x7fc2a81f7530 2023-10-30T13:42:31.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2af668640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8072420 con 0x7fc2940047d0 2023-10-30T13:42:31.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2acc03640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2a80c20b0 0x7fc2a81f6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2acc03640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2a80c20b0 0x7fc2a81f6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37598/0 (socket says 172.21.15.46:37598) 2023-10-30T13:42:31.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2acc03640 1 -- 172.21.15.46:0/2690230794 learned_addr learned my addr 172.21.15.46:0/2690230794 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:31.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2ad404640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2940047d0 0x7fc2a80717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2adc05640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2a81f7530 0x7fc2a81f37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2acc03640 1 -- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2940047d0 msgr2=0x7fc2a80717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:31.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2acc03640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2940047d0 0x7fc2a80717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:31.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2acc03640 1 -- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2a81f7530 msgr2=0x7fc2a81f37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:31.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2acc03640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2a81f7530 0x7fc2a81f37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:31.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2acc03640 1 -- 172.21.15.46:0/2690230794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a8151e80 con 0x7fc2a80c20b0 2023-10-30T13:42:31.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2acc03640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2a80c20b0 0x7fc2a81f6ff0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc29800c190 tx=0x7fc294005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:31.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc29dffb640 1 -- 172.21.15.46:0/2690230794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2980a5d10 con 0x7fc2a80c20b0 2023-10-30T13:42:31.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc29dffb640 1 -- 172.21.15.46:0/2690230794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2980a5e70 con 0x7fc2a80c20b0 2023-10-30T13:42:31.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc29dffb640 1 -- 172.21.15.46:0/2690230794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2980049f0 con 0x7fc2a80c20b0 2023-10-30T13:42:31.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2af668640 1 -- 172.21.15.46:0/2690230794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a80c2c80 con 0x7fc2a80c20b0 2023-10-30T13:42:31.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2af668640 1 -- 172.21.15.46:0/2690230794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a81f6ad0 con 0x7fc2a80c20b0 2023-10-30T13:42:31.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc2af668640 1 -- 172.21.15.46:0/2690230794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a804e170 con 0x7fc2a80c20b0 2023-10-30T13:42:31.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.777+0000 7fc29dffb640 1 -- 172.21.15.46:0/2690230794 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc298004b50 con 0x7fc2a80c20b0 2023-10-30T13:42:31.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.781+0000 7fc29dffb640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc28c066520 0x7fc28c0689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.781+0000 7fc29dffb640 1 -- 172.21.15.46:0/2690230794 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fc29811dae0 con 0x7fc2a80c20b0 2023-10-30T13:42:31.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.781+0000 7fc2ad404640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc28c066520 0x7fc28c0689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.781+0000 7fc2ad404640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc28c066520 0x7fc28c0689e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4009a40 tx=0x7fc2a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:31.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.785+0000 7fc29dffb640 1 -- 172.21.15.46:0/2690230794 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc2980eaac0 con 0x7fc2a80c20b0 2023-10-30T13:42:31.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.973+0000 7fc2af668640 1 -- 172.21.15.46:0/2690230794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2a806aed0 con 0x7fc2a80c20b0 2023-10-30T13:42:31.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.973+0000 7fc29dffb640 1 -- 172.21.15.46:0/2690230794 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7fc2980ee0e0 con 0x7fc2a80c20b0 2023-10-30T13:42:31.979 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:31.980 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:31.980 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.977+0000 7fc2af668640 1 -- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc28c066520 msgr2=0x7fc28c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.977+0000 7fc2af668640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc28c066520 0x7fc28c0689e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4009a40 tx=0x7fc2a4008040 comp rx=0 tx=0).stop 2023-10-30T13:42:31.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.977+0000 7fc2af668640 1 -- 172.21.15.46:0/2690230794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2a80c20b0 msgr2=0x7fc2a81f6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:31.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.977+0000 7fc2af668640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2a80c20b0 0x7fc2a81f6ff0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc29800c190 tx=0x7fc294005800 comp rx=0 tx=0).stop 2023-10-30T13:42:31.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.977+0000 7fc2af668640 1 -- 172.21.15.46:0/2690230794 shutdown_connections 2023-10-30T13:42:31.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.977+0000 7fc2af668640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc28c066520 0x7fc28c0689e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:31.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.977+0000 7fc2af668640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2a81f7530 0x7fc2a81f37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:31.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.977+0000 7fc2af668640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2a80c20b0 0x7fc2a81f6ff0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:31.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.977+0000 7fc2af668640 1 --2- 172.21.15.46:0/2690230794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2940047d0 0x7fc2a80717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:31.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.977+0000 7fc2af668640 1 -- 172.21.15.46:0/2690230794 >> 172.21.15.46:0/2690230794 conn(0x7fc2a80c7910 msgr2=0x7fc2a81566b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:31.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.977+0000 7fc2af668640 1 -- 172.21.15.46:0/2690230794 shutdown_connections 2023-10-30T13:42:31.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:31.977+0000 7fc2af668640 1 -- 172.21.15.46:0/2690230794 wait complete. 2023-10-30T13:42:32.000 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:34.001 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:34.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.149+0000 7ff18aec9640 1 Processor -- start 2023-10-30T13:42:34.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.149+0000 7ff18aec9640 1 -- start start 2023-10-30T13:42:34.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff18aec9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18415a410 0x7ff18406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:34.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff18aec9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff18406cb80 0x7ff18406d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:34.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff18aec9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18406d550 con 0x7ff184069e90 2023-10-30T13:42:34.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff18aec9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1840c8570 con 0x7ff18406cb80 2023-10-30T13:42:34.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff18aec9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18415a850 con 0x7ff18415a410 2023-10-30T13:42:34.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff188c65640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff184069e90 0x7ff1840c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49884/0 (socket says 172.21.15.46:49884) 2023-10-30T13:42:34.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff188c65640 1 -- 172.21.15.46:0/3742903979 learned_addr learned my addr 172.21.15.46:0/3742903979 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff189466640 1 --2- 172.21.15.46:0/3742903979 >> v2:172.21.15.154:3300/0 conn(0x7ff18406cb80 0x7ff18406d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff183fff640 1 --2- 172.21.15.46:0/3742903979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18415a410 0x7ff18406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff1837fe640 1 -- 172.21.15.46:0/3742903979 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563794263 0 0) 0x7ff18406d550 con 0x7ff184069e90 2023-10-30T13:42:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff1837fe640 1 -- 172.21.15.46:0/3742903979 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff174003530 con 0x7ff184069e90 2023-10-30T13:42:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff189466640 1 -- 172.21.15.46:0/3742903979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18415a410 msgr2=0x7ff18406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff189466640 1 --2- 172.21.15.46:0/3742903979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18415a410 0x7ff18406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff1837fe640 1 -- 172.21.15.46:0/3742903979 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1465962006 0 0) 0x7ff174003530 con 0x7ff184069e90 2023-10-30T13:42:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff189466640 1 -- 172.21.15.46:0/3742903979 >> v1:172.21.15.46:6789/0 conn(0x7ff184069e90 legacy=0x7ff1840c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.153+0000 7ff189466640 1 -- 172.21.15.46:0/3742903979 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840c6c10 con 0x7ff18406cb80 2023-10-30T13:42:34.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff189466640 1 --2- 172.21.15.46:0/3742903979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff18406cb80 0x7ff18406d010 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009d90 tx=0x7ff17c01c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f77c3e921df29c32 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff1837fe640 1 -- 172.21.15.46:0/3742903979 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178001680 con 0x7ff18406cb80 2023-10-30T13:42:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff1837fe640 1 -- 172.21.15.46:0/3742903979 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c0a6480 con 0x7ff18406cb80 2023-10-30T13:42:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff1837fe640 1 -- 172.21.15.46:0/3742903979 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0a67f0 con 0x7ff18406cb80 2023-10-30T13:42:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff18aec9640 1 -- 172.21.15.46:0/3742903979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff18406cb80 msgr2=0x7ff18406d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff18aec9640 1 --2- 172.21.15.46:0/3742903979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff18406cb80 0x7ff18406d010 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009d90 tx=0x7ff17c01c870 comp rx=0 tx=0).stop 2023-10-30T13:42:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff18aec9640 1 -- 172.21.15.46:0/3742903979 shutdown_connections 2023-10-30T13:42:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff18aec9640 1 --2- 172.21.15.46:0/3742903979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff18406cb80 0x7ff18406d010 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff18aec9640 1 --2- 172.21.15.46:0/3742903979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18415a410 0x7ff18406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff18aec9640 1 -- 172.21.15.46:0/3742903979 >> 172.21.15.46:0/3742903979 conn(0x7ff18406dfd0 msgr2=0x7ff18406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff18aec9640 1 -- 172.21.15.46:0/3742903979 shutdown_connections 2023-10-30T13:42:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff18aec9640 1 -- 172.21.15.46:0/3742903979 wait complete. 2023-10-30T13:42:34.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff18aec9640 1 Processor -- start 2023-10-30T13:42:34.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.157+0000 7ff18aec9640 1 -- start start 2023-10-30T13:42:34.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff18aec9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff184069e90 0x7ff1840c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:34.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff18aec9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18406cb80 0x7ff1841f5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:34.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff18aec9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18415a410 0x7ff1841f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:34.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff18aec9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1841537e0 con 0x7ff18415a410 2023-10-30T13:42:34.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff18aec9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1841f55e0 con 0x7ff184069e90 2023-10-30T13:42:34.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff18aec9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840c3480 con 0x7ff18406cb80 2023-10-30T13:42:34.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff189466640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18415a410 0x7ff1841f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff189466640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18415a410 0x7ff1841f1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37606/0 (socket says 172.21.15.46:37606) 2023-10-30T13:42:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff189466640 1 -- 172.21.15.46:0/2045530377 learned_addr learned my addr 172.21.15.46:0/2045530377 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff188c65640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff184069e90 0x7ff1840c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff183fff640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18406cb80 0x7ff1841f5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff189466640 1 -- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18406cb80 msgr2=0x7ff1841f5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff189466640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18406cb80 0x7ff1841f5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff189466640 1 -- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff184069e90 msgr2=0x7ff1840c5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff189466640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff184069e90 0x7ff1840c5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff189466640 1 -- 172.21.15.46:0/2045530377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c009a70 con 0x7ff18415a410 2023-10-30T13:42:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff189466640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18415a410 0x7ff1841f1a20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff17c01af20 tx=0x7ff17c019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff1817fa640 1 -- 172.21.15.46:0/2045530377 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0b7070 con 0x7ff18415a410 2023-10-30T13:42:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff1817fa640 1 -- 172.21.15.46:0/2045530377 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c004250 con 0x7ff18415a410 2023-10-30T13:42:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff18aec9640 1 -- 172.21.15.46:0/2045530377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff184153a30 con 0x7ff18415a410 2023-10-30T13:42:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff1817fa640 1 -- 172.21.15.46:0/2045530377 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0045e0 con 0x7ff18415a410 2023-10-30T13:42:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff18aec9640 1 -- 172.21.15.46:0/2045530377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1841f4f60 con 0x7ff18415a410 2023-10-30T13:42:34.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.161+0000 7ff18aec9640 1 -- 172.21.15.46:0/2045530377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18404e170 con 0x7ff18415a410 2023-10-30T13:42:34.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.165+0000 7ff1817fa640 1 -- 172.21.15.46:0/2045530377 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff17c004740 con 0x7ff18415a410 2023-10-30T13:42:34.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.165+0000 7ff1817fa640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1680664a0 0x7ff168068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:34.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.165+0000 7ff1817fa640 1 -- 172.21.15.46:0/2045530377 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7ff17c11e410 con 0x7ff18415a410 2023-10-30T13:42:34.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.165+0000 7ff188c65640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1680664a0 0x7ff168068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:34.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.165+0000 7ff188c65640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1680664a0 0x7ff168068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff178006b60 tx=0x7ff178001dd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:34.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.169+0000 7ff1817fa640 1 -- 172.21.15.46:0/2045530377 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff17c0eef30 con 0x7ff18415a410 2023-10-30T13:42:34.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.361+0000 7ff18aec9640 1 -- 172.21.15.46:0/2045530377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1841f5720 con 0x7ff18415a410 2023-10-30T13:42:34.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.361+0000 7ff1817fa640 1 -- 172.21.15.46:0/2045530377 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6533 (secure 0 0 0) 0x7ff17c0eeac0 con 0x7ff18415a410 2023-10-30T13:42:34.368 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:34.368 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:05.189550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 98 2023-10-30T13:42:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.365+0000 7ff18aec9640 1 -- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1680664a0 msgr2=0x7ff168068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.365+0000 7ff18aec9640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1680664a0 0x7ff168068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff178006b60 tx=0x7ff178001dd0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.365+0000 7ff18aec9640 1 -- 172.21.15.46:0/2045530377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18415a410 msgr2=0x7ff1841f1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.365+0000 7ff18aec9640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18415a410 0x7ff1841f1a20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff17c01af20 tx=0x7ff17c019610 comp rx=0 tx=0).stop 2023-10-30T13:42:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.365+0000 7ff18aec9640 1 -- 172.21.15.46:0/2045530377 shutdown_connections 2023-10-30T13:42:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.365+0000 7ff18aec9640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1680664a0 0x7ff168068960 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.365+0000 7ff18aec9640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff18415a410 0x7ff1841f1a20 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.365+0000 7ff18aec9640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18406cb80 0x7ff1841f5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.365+0000 7ff18aec9640 1 --2- 172.21.15.46:0/2045530377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff184069e90 0x7ff1840c5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.365+0000 7ff18aec9640 1 -- 172.21.15.46:0/2045530377 >> 172.21.15.46:0/2045530377 conn(0x7ff18406dfd0 msgr2=0x7ff18406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.365+0000 7ff18aec9640 1 -- 172.21.15.46:0/2045530377 shutdown_connections 2023-10-30T13:42:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:34.365+0000 7ff18aec9640 1 -- 172.21.15.46:0/2045530377 wait complete. 2023-10-30T13:42:34.389 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:36.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:36.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f7405088640 1 Processor -- start 2023-10-30T13:42:36.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f7405088640 1 -- start start 2023-10-30T13:42:36.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f7405088640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74001538f0 0x7f7400159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:36.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f7405088640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74000cefa0 0x7f74000cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:36.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f7405088640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74000ca6a0 con 0x7f740015cd50 2023-10-30T13:42:36.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f7405088640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f740015d300 con 0x7f74000cefa0 2023-10-30T13:42:36.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f7405088640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740015d5e0 con 0x7f74001538f0 2023-10-30T13:42:36.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f73ff7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f740015cd50 0x7f74000c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54814/0 (socket says 172.21.15.46:54814) 2023-10-30T13:42:36.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f73ff7fe640 1 -- 172.21.15.46:0/2488786433 learned_addr learned my addr 172.21.15.46:0/2488786433 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:36.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f7404887640 1 --2- 172.21.15.46:0/2488786433 >> v2:172.21.15.154:3300/0 conn(0x7f74000cefa0 0x7f74000cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:36.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f73fffff640 1 --2- 172.21.15.46:0/2488786433 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74001538f0 0x7f7400159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:36.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f73feffd640 1 -- 172.21.15.46:0/2488786433 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755102695 0 0) 0x7f74000ca6a0 con 0x7f740015cd50 2023-10-30T13:42:36.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f73feffd640 1 -- 172.21.15.46:0/2488786433 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e8003530 con 0x7f740015cd50 2023-10-30T13:42:36.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f7404887640 1 -- 172.21.15.46:0/2488786433 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74001538f0 msgr2=0x7f7400159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:36.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f7404887640 1 --2- 172.21.15.46:0/2488786433 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74001538f0 0x7f7400159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f73feffd640 1 -- 172.21.15.46:0/2488786433 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2575788619 0 0) 0x7f73e8003530 con 0x7f740015cd50 2023-10-30T13:42:36.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f7404887640 1 -- 172.21.15.46:0/2488786433 >> v1:172.21.15.46:6789/0 conn(0x7f740015cd50 legacy=0x7f74000c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:36.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.541+0000 7f7404887640 1 -- 172.21.15.46:0/2488786433 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740015d8c0 con 0x7f74000cefa0 2023-10-30T13:42:36.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.545+0000 7f7404887640 1 --2- 172.21.15.46:0/2488786433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74000cefa0 0x7f74000cf3f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009d90 tx=0x7f73f801f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=235edc054402d006 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.545+0000 7f73feffd640 1 -- 172.21.15.46:0/2488786433 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4001680 con 0x7f74000cefa0 2023-10-30T13:42:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.545+0000 7f73feffd640 1 -- 172.21.15.46:0/2488786433 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f80a8bb0 con 0x7f74000cefa0 2023-10-30T13:42:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.545+0000 7f73feffd640 1 -- 172.21.15.46:0/2488786433 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f80b0440 con 0x7f74000cefa0 2023-10-30T13:42:36.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.545+0000 7f7405088640 1 -- 172.21.15.46:0/2488786433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74000cefa0 msgr2=0x7f74000cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.545+0000 7f7405088640 1 --2- 172.21.15.46:0/2488786433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74000cefa0 0x7f74000cf3f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009d90 tx=0x7f73f801f060 comp rx=0 tx=0).stop 2023-10-30T13:42:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.545+0000 7f7405088640 1 -- 172.21.15.46:0/2488786433 shutdown_connections 2023-10-30T13:42:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.545+0000 7f7405088640 1 --2- 172.21.15.46:0/2488786433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74000cefa0 0x7f74000cf3f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.545+0000 7f7405088640 1 --2- 172.21.15.46:0/2488786433 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74001538f0 0x7f7400159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.545+0000 7f7405088640 1 -- 172.21.15.46:0/2488786433 >> 172.21.15.46:0/2488786433 conn(0x7f740006a640 msgr2=0x7f7400065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:36.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.545+0000 7f7405088640 1 -- 172.21.15.46:0/2488786433 shutdown_connections 2023-10-30T13:42:36.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.545+0000 7f7405088640 1 -- 172.21.15.46:0/2488786433 wait complete. 2023-10-30T13:42:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7405088640 1 Processor -- start 2023-10-30T13:42:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7405088640 1 -- start start 2023-10-30T13:42:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7405088640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74000cefa0 0x7f740015c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7405088640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74001538f0 0x7f74000ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:36.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7405088640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f740015cd50 0x7f74001f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:36.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7405088640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74000b41e0 con 0x7f740015cd50 2023-10-30T13:42:36.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7405088640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74000b4c50 con 0x7f74000cefa0 2023-10-30T13:42:36.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7405088640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74001f2360 con 0x7f74001538f0 2023-10-30T13:42:36.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7404887640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f740015cd50 0x7f74001f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:36.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7404887640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f740015cd50 0x7f74001f27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43368/0 (socket says 172.21.15.46:43368) 2023-10-30T13:42:36.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7404887640 1 -- 172.21.15.46:0/2727162358 learned_addr learned my addr 172.21.15.46:0/2727162358 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:36.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f73fffff640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74000cefa0 0x7f740015c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:36.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f73ff7fe640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74001538f0 0x7f74000ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7404887640 1 -- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74001538f0 msgr2=0x7f74000ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7404887640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74001538f0 0x7f74000ce6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7404887640 1 -- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74000cefa0 msgr2=0x7f740015c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7404887640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74000cefa0 0x7f740015c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7404887640 1 -- 172.21.15.46:0/2727162358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f8009a70 con 0x7f740015cd50 2023-10-30T13:42:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7404887640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f740015cd50 0x7f74001f27f0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f73f801f900 tx=0x7f73f801be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f73fcff9640 1 -- 172.21.15.46:0/2727162358 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8009450 con 0x7f740015cd50 2023-10-30T13:42:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f73fcff9640 1 -- 172.21.15.46:0/2727162358 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f80bc450 con 0x7f740015cd50 2023-10-30T13:42:36.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7405088640 1 -- 172.21.15.46:0/2727162358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74000b43f0 con 0x7f740015cd50 2023-10-30T13:42:36.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f73fcff9640 1 -- 172.21.15.46:0/2727162358 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f80bc7e0 con 0x7f740015cd50 2023-10-30T13:42:36.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.549+0000 7f7405088640 1 -- 172.21.15.46:0/2727162358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f740015cad0 con 0x7f740015cd50 2023-10-30T13:42:36.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.553+0000 7f7405088640 1 -- 172.21.15.46:0/2727162358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f740004e170 con 0x7f740015cd50 2023-10-30T13:42:36.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.553+0000 7f73fcff9640 1 -- 172.21.15.46:0/2727162358 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f73f80bc940 con 0x7f740015cd50 2023-10-30T13:42:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.553+0000 7f73fcff9640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f73ec0664a0 0x7f73ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.553+0000 7f73fcff9640 1 -- 172.21.15.46:0/2727162358 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f73f8122c70 con 0x7f740015cd50 2023-10-30T13:42:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.553+0000 7f73fffff640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f73ec0664a0 0x7f73ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.553+0000 7f73fffff640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f73ec0664a0 0x7f73ec068960 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009b60 tx=0x7f73f0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.557+0000 7f73fcff9640 1 -- 172.21.15.46:0/2727162358 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f73f80efc90 con 0x7f740015cd50 2023-10-30T13:42:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.749+0000 7f7405088640 1 -- 172.21.15.46:0/2727162358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74001f24a0 con 0x7f740015cd50 2023-10-30T13:42:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.749+0000 7f73fcff9640 1 -- 172.21.15.46:0/2727162358 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f73f80f32b0 con 0x7f740015cd50 2023-10-30T13:42:36.756 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:36.756 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:34.173048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:36.756 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 99 2023-10-30T13:42:36.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.753+0000 7f7405088640 1 -- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f73ec0664a0 msgr2=0x7f73ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.753+0000 7f7405088640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f73ec0664a0 0x7f73ec068960 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009b60 tx=0x7f73f0009190 comp rx=0 tx=0).stop 2023-10-30T13:42:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.753+0000 7f7405088640 1 -- 172.21.15.46:0/2727162358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f740015cd50 msgr2=0x7f74001f27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.753+0000 7f7405088640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f740015cd50 0x7f74001f27f0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f73f801f900 tx=0x7f73f801be00 comp rx=0 tx=0).stop 2023-10-30T13:42:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.753+0000 7f7405088640 1 -- 172.21.15.46:0/2727162358 shutdown_connections 2023-10-30T13:42:36.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.753+0000 7f7405088640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f73ec0664a0 0x7f73ec068960 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.753+0000 7f7405088640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f740015cd50 0x7f74001f27f0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.753+0000 7f7405088640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74001538f0 0x7f74000ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.753+0000 7f7405088640 1 --2- 172.21.15.46:0/2727162358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74000cefa0 0x7f740015c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.753+0000 7f7405088640 1 -- 172.21.15.46:0/2727162358 >> 172.21.15.46:0/2727162358 conn(0x7f740006a640 msgr2=0x7f74000c4350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.753+0000 7f7405088640 1 -- 172.21.15.46:0/2727162358 shutdown_connections 2023-10-30T13:42:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:36.753+0000 7f7405088640 1 -- 172.21.15.46:0/2727162358 wait complete. 2023-10-30T13:42:36.777 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:38.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:38.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d423f640 1 Processor -- start 2023-10-30T13:42:38.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d423f640 1 -- start start 2023-10-30T13:42:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d423f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc7cc1580e0 0x7fc7cc070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d423f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7cc0c6540 0x7fc7cc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d423f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7cc0c5810 con 0x7fc7cc069e90 2023-10-30T13:42:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d423f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7cc0c6990 con 0x7fc7cc0c6540 2023-10-30T13:42:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d423f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7cc0c6bb0 con 0x7fc7cc1580e0 2023-10-30T13:42:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d17da640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7cc0c6540 0x7fc7cc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d17da640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7cc0c6540 0x7fc7cc15e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43382/0 (socket says 172.21.15.46:43382) 2023-10-30T13:42:38.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d17da640 1 -- 172.21.15.46:0/10865645 learned_addr learned my addr 172.21.15.46:0/10865645 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:38.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d27dc640 1 --1- 172.21.15.46:0/10865645 >> v1:172.21.15.46:6789/0 conn(0x7fc7cc069e90 0x7fc7cc06a320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54822/0 (socket says 172.21.15.46:54822) 2023-10-30T13:42:38.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d1fdb640 1 --2- 172.21.15.46:0/10865645 >> v2:172.21.15.154:3300/0 conn(0x7fc7cc1580e0 0x7fc7cc070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:38.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d17da640 1 -- 172.21.15.46:0/10865645 >> v2:172.21.15.154:3300/0 conn(0x7fc7cc1580e0 msgr2=0x7fc7cc070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:38.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d17da640 1 --2- 172.21.15.46:0/10865645 >> v2:172.21.15.154:3300/0 conn(0x7fc7cc1580e0 0x7fc7cc070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:38.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d17da640 1 -- 172.21.15.46:0/10865645 >> v1:172.21.15.46:6789/0 conn(0x7fc7cc069e90 legacy=0x7fc7cc06a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:38.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d17da640 1 -- 172.21.15.46:0/10865645 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc0c6e70 con 0x7fc7cc0c6540 2023-10-30T13:42:38.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d0fd9640 1 -- 172.21.15.46:0/10865645 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165101292 0 0) 0x7fc7cc0c5810 con 0x7fc7cc069e90 2023-10-30T13:42:38.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.929+0000 7fc7d1fdb640 1 --2- 172.21.15.46:0/10865645 >> v2:172.21.15.154:3300/0 conn(0x7fc7cc1580e0 0x7fc7cc070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:38.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.933+0000 7fc7d17da640 1 --2- 172.21.15.46:0/10865645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7cc0c6540 0x7fc7cc15e700 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc06eea0 tx=0x7fc7bc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d284fabee83f29 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:38.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.933+0000 7fc7d0fd9640 1 -- 172.21.15.46:0/10865645 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc0aa660 con 0x7fc7cc0c6540 2023-10-30T13:42:38.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.933+0000 7fc7d0fd9640 1 -- 172.21.15.46:0/10865645 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7bc0aa7c0 con 0x7fc7cc0c6540 2023-10-30T13:42:38.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.933+0000 7fc7d0fd9640 1 -- 172.21.15.46:0/10865645 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc0aab50 con 0x7fc7cc0c6540 2023-10-30T13:42:38.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.933+0000 7fc7d423f640 1 -- 172.21.15.46:0/10865645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7cc0c6540 msgr2=0x7fc7cc15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:38.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.933+0000 7fc7d423f640 1 --2- 172.21.15.46:0/10865645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7cc0c6540 0x7fc7cc15e700 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc06eea0 tx=0x7fc7bc01c500 comp rx=0 tx=0).stop 2023-10-30T13:42:38.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.933+0000 7fc7d423f640 1 -- 172.21.15.46:0/10865645 shutdown_connections 2023-10-30T13:42:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.933+0000 7fc7d423f640 1 --2- 172.21.15.46:0/10865645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7cc0c6540 0x7fc7cc15e700 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.933+0000 7fc7d423f640 1 --2- 172.21.15.46:0/10865645 >> v2:172.21.15.154:3300/0 conn(0x7fc7cc1580e0 0x7fc7cc070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.933+0000 7fc7d423f640 1 -- 172.21.15.46:0/10865645 >> 172.21.15.46:0/10865645 conn(0x7fc7cc0c7910 msgr2=0x7fc7cc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.933+0000 7fc7d423f640 1 -- 172.21.15.46:0/10865645 shutdown_connections 2023-10-30T13:42:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.933+0000 7fc7d423f640 1 -- 172.21.15.46:0/10865645 wait complete. 2023-10-30T13:42:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d423f640 1 Processor -- start 2023-10-30T13:42:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d423f640 1 -- start start 2023-10-30T13:42:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d423f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7cc069e90 0x7fc7cc1f9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:38.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d423f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7cc0c6540 0x7fc7cc149e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:38.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d423f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7cc1580e0 0x7fc7cc140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:38.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d423f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc142b30 con 0x7fc7cc0c6540 2023-10-30T13:42:38.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d423f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc1fa520 con 0x7fc7cc069e90 2023-10-30T13:42:38.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d423f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc1493f0 con 0x7fc7cc1580e0 2023-10-30T13:42:38.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d17da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7cc0c6540 0x7fc7cc149e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:38.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d17da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7cc0c6540 0x7fc7cc149e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43390/0 (socket says 172.21.15.46:43390) 2023-10-30T13:42:38.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d17da640 1 -- 172.21.15.46:0/3082662384 learned_addr learned my addr 172.21.15.46:0/3082662384 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:38.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d1fdb640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7cc069e90 0x7fc7cc1f9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:38.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d27dc640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7cc1580e0 0x7fc7cc140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d17da640 1 -- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7cc1580e0 msgr2=0x7fc7cc140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d17da640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7cc1580e0 0x7fc7cc140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d17da640 1 -- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7cc069e90 msgr2=0x7fc7cc1f9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d17da640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7cc069e90 0x7fc7cc1f9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d17da640 1 -- 172.21.15.46:0/3082662384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc0094d0 con 0x7fc7cc0c6540 2023-10-30T13:42:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d17da640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7cc0c6540 0x7fc7cc149e40 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc0ac040 tx=0x7fc7bc01ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7c27fc640 1 -- 172.21.15.46:0/3082662384 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc0aad40 con 0x7fc7cc0c6540 2023-10-30T13:42:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7c27fc640 1 -- 172.21.15.46:0/3082662384 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7bc0b7070 con 0x7fc7cc0c6540 2023-10-30T13:42:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7c27fc640 1 -- 172.21.15.46:0/3082662384 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc0aaea0 con 0x7fc7cc0c6540 2023-10-30T13:42:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d423f640 1 -- 172.21.15.46:0/3082662384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7cc1499b0 con 0x7fc7cc0c6540 2023-10-30T13:42:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.937+0000 7fc7d423f640 1 -- 172.21.15.46:0/3082662384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7cc0c90c0 con 0x7fc7cc0c6540 2023-10-30T13:42:38.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.941+0000 7fc7d423f640 1 -- 172.21.15.46:0/3082662384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7cc149b40 con 0x7fc7cc0c6540 2023-10-30T13:42:38.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.941+0000 7fc7c27fc640 1 -- 172.21.15.46:0/3082662384 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7bc004220 con 0x7fc7cc0c6540 2023-10-30T13:42:38.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.941+0000 7fc7c27fc640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7b00664a0 0x7fc7b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:38.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.941+0000 7fc7c27fc640 1 -- 172.21.15.46:0/3082662384 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fc7bc0a6040 con 0x7fc7cc0c6540 2023-10-30T13:42:38.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.941+0000 7fc7d1fdb640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7b00664a0 0x7fc7b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:38.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.941+0000 7fc7d1fdb640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7b00664a0 0x7fc7b0068960 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc7c80045f0 tx=0x7fc7c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:38.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:38.945+0000 7fc7c27fc640 1 -- 172.21.15.46:0/3082662384 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7bc0eb440 con 0x7fc7cc0c6540 2023-10-30T13:42:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 -- 172.21.15.46:0/3082662384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7cc04e170 con 0x7fc7cc0c6540 2023-10-30T13:42:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7c27fc640 1 -- 172.21.15.46:0/3082662384 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fc7bc0eea60 con 0x7fc7cc0c6540 2023-10-30T13:42:39.142 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:39.142 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:34.173048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:39.142 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 99 2023-10-30T13:42:39.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 -- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7b00664a0 msgr2=0x7fc7b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:39.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7b00664a0 0x7fc7b0068960 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc7c80045f0 tx=0x7fc7c8008040 comp rx=0 tx=0).stop 2023-10-30T13:42:39.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 -- 172.21.15.46:0/3082662384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7cc0c6540 msgr2=0x7fc7cc149e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7cc0c6540 0x7fc7cc149e40 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc0ac040 tx=0x7fc7bc01ad40 comp rx=0 tx=0).stop 2023-10-30T13:42:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 -- 172.21.15.46:0/3082662384 shutdown_connections 2023-10-30T13:42:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7b00664a0 0x7fc7b0068960 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7cc1580e0 0x7fc7cc140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7cc0c6540 0x7fc7cc149e40 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 --2- 172.21.15.46:0/3082662384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7cc069e90 0x7fc7cc1f9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 -- 172.21.15.46:0/3082662384 >> 172.21.15.46:0/3082662384 conn(0x7fc7cc0c7910 msgr2=0x7fc7cc06a320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 -- 172.21.15.46:0/3082662384 shutdown_connections 2023-10-30T13:42:39.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:39.137+0000 7fc7d423f640 1 -- 172.21.15.46:0/3082662384 wait complete. 2023-10-30T13:42:39.163 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:41.163 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:41.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27d0d3e640 1 Processor -- start 2023-10-30T13:42:41.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27d0d3e640 1 -- start start 2023-10-30T13:42:41.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27d0d3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27cc069e90 0x7f27cc15a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:41.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27d0d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc06cb80 0x7f27cc06d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:41.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27d0d3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc06d550 con 0x7f27cc06a270 2023-10-30T13:42:41.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27d0d3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27cc0c8570 con 0x7f27cc069e90 2023-10-30T13:42:41.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27d0d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0c6c10 con 0x7f27cc06cb80 2023-10-30T13:42:41.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27ca59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27cc069e90 0x7f27cc15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:41.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27c9d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f27cc06a270 0x7f27cc0c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54826/0 (socket says 172.21.15.46:54826) 2023-10-30T13:42:41.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27c9d9b640 1 -- 172.21.15.46:0/1416260508 learned_addr learned my addr 172.21.15.46:0/1416260508 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:41.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27cad9d640 1 --2- 172.21.15.46:0/1416260508 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc06cb80 0x7f27cc06d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:41.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27c959a640 1 -- 172.21.15.46:0/1416260508 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264205022 0 0) 0x7f27cc06d550 con 0x7f27cc06a270 2023-10-30T13:42:41.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27c959a640 1 -- 172.21.15.46:0/1416260508 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b4003530 con 0x7f27cc06a270 2023-10-30T13:42:41.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27ca59c640 1 -- 172.21.15.46:0/1416260508 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc06cb80 msgr2=0x7f27cc06d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:41.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27ca59c640 1 --2- 172.21.15.46:0/1416260508 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc06cb80 0x7f27cc06d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27ca59c640 1 -- 172.21.15.46:0/1416260508 >> v1:172.21.15.46:6789/0 conn(0x7f27cc06a270 legacy=0x7f27cc0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:41.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27ca59c640 1 -- 172.21.15.46:0/1416260508 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc0c6ef0 con 0x7f27cc069e90 2023-10-30T13:42:41.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.317+0000 7f27cad9d640 1 --2- 172.21.15.46:0/1416260508 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc06cb80 0x7f27cc06d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:41.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.321+0000 7f27ca59c640 1 --2- 172.21.15.46:0/1416260508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc069e90 0x7f27cc15a410 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0097c0 tx=0x7f27bc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7153d65c28eec715 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:41.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.321+0000 7f27c959a640 1 -- 172.21.15.46:0/1416260508 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc0ab600 con 0x7f27cc069e90 2023-10-30T13:42:41.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.321+0000 7f27c959a640 1 -- 172.21.15.46:0/1416260508 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc0a7050 con 0x7f27cc069e90 2023-10-30T13:42:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.321+0000 7f27c959a640 1 -- 172.21.15.46:0/1416260508 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc0ab980 con 0x7f27cc069e90 2023-10-30T13:42:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.321+0000 7f27d0d3e640 1 -- 172.21.15.46:0/1416260508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc069e90 msgr2=0x7f27cc15a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.321+0000 7f27d0d3e640 1 --2- 172.21.15.46:0/1416260508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc069e90 0x7f27cc15a410 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0097c0 tx=0x7f27bc01c480 comp rx=0 tx=0).stop 2023-10-30T13:42:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.321+0000 7f27d0d3e640 1 -- 172.21.15.46:0/1416260508 shutdown_connections 2023-10-30T13:42:41.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.321+0000 7f27d0d3e640 1 --2- 172.21.15.46:0/1416260508 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc06cb80 0x7f27cc06d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.321+0000 7f27d0d3e640 1 --2- 172.21.15.46:0/1416260508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc069e90 0x7f27cc15a410 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.321+0000 7f27d0d3e640 1 -- 172.21.15.46:0/1416260508 >> 172.21.15.46:0/1416260508 conn(0x7f27cc06dfd0 msgr2=0x7f27cc06e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:41.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.321+0000 7f27d0d3e640 1 -- 172.21.15.46:0/1416260508 shutdown_connections 2023-10-30T13:42:41.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.321+0000 7f27d0d3e640 1 -- 172.21.15.46:0/1416260508 wait complete. 2023-10-30T13:42:41.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27d0d3e640 1 Processor -- start 2023-10-30T13:42:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27d0d3e640 1 -- start start 2023-10-30T13:42:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27d0d3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc069e90 0x7f27cc1f4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27d0d3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc06a270 0x7f27cc14d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27d0d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc06cb80 0x7f27cc14dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27d0d3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0c4850 con 0x7f27cc069e90 2023-10-30T13:42:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27d0d3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0c43c0 con 0x7f27cc06a270 2023-10-30T13:42:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27d0d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0cb770 con 0x7f27cc06cb80 2023-10-30T13:42:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27ca59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc069e90 0x7f27cc1f4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27ca59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc069e90 0x7f27cc1f4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43410/0 (socket says 172.21.15.46:43410) 2023-10-30T13:42:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27ca59c640 1 -- 172.21.15.46:0/317361347 learned_addr learned my addr 172.21.15.46:0/317361347 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27c9d9b640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc06a270 0x7f27cc14d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27cad9d640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc06cb80 0x7f27cc14dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27ca59c640 1 -- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc06cb80 msgr2=0x7f27cc14dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27ca59c640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc06cb80 0x7f27cc14dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27ca59c640 1 -- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc06a270 msgr2=0x7f27cc14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27ca59c640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc06a270 0x7f27cc14d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27ca59c640 1 -- 172.21.15.46:0/317361347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc009420 con 0x7f27cc069e90 2023-10-30T13:42:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27ca59c640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc069e90 0x7f27cc1f4e70 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f27bc01a0b0 tx=0x7f27bc01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27baffd640 1 -- 172.21.15.46:0/317361347 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc0b7450 con 0x7f27cc069e90 2023-10-30T13:42:41.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27baffd640 1 -- 172.21.15.46:0/317361347 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc0b75b0 con 0x7f27cc069e90 2023-10-30T13:42:41.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27baffd640 1 -- 172.21.15.46:0/317361347 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc0b7960 con 0x7f27cc069e90 2023-10-30T13:42:41.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27d0d3e640 1 -- 172.21.15.46:0/317361347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc0c59f0 con 0x7f27cc069e90 2023-10-30T13:42:41.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.325+0000 7f27d0d3e640 1 -- 172.21.15.46:0/317361347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc0c3a90 con 0x7f27cc069e90 2023-10-30T13:42:41.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.329+0000 7f27d0d3e640 1 -- 172.21.15.46:0/317361347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27cc04e170 con 0x7f27cc069e90 2023-10-30T13:42:41.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.329+0000 7f27baffd640 1 -- 172.21.15.46:0/317361347 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f27bc0b7ac0 con 0x7f27cc069e90 2023-10-30T13:42:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.329+0000 7f27baffd640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27ac0664a0 0x7f27ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.329+0000 7f27baffd640 1 -- 172.21.15.46:0/317361347 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f27bc11eb20 con 0x7f27cc069e90 2023-10-30T13:42:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.329+0000 7f27c9d9b640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27ac0664a0 0x7f27ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.329+0000 7f27c9d9b640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27ac0664a0 0x7f27ac068960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f27cc14d330 tx=0x7f27c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:41.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.333+0000 7f27baffd640 1 -- 172.21.15.46:0/317361347 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f27bc0ebc30 con 0x7f27cc069e90 2023-10-30T13:42:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.521+0000 7f27d0d3e640 1 -- 172.21.15.46:0/317361347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27cc06ef60 con 0x7f27cc069e90 2023-10-30T13:42:41.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.521+0000 7f27baffd640 1 -- 172.21.15.46:0/317361347 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f27bc0b0090 con 0x7f27cc069e90 2023-10-30T13:42:41.528 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:41.528 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:34.173048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:41.529 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 99 2023-10-30T13:42:41.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.525+0000 7f27d0d3e640 1 -- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27ac0664a0 msgr2=0x7f27ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:41.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.525+0000 7f27d0d3e640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27ac0664a0 0x7f27ac068960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f27cc14d330 tx=0x7f27c0009040 comp rx=0 tx=0).stop 2023-10-30T13:42:41.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.525+0000 7f27d0d3e640 1 -- 172.21.15.46:0/317361347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc069e90 msgr2=0x7f27cc1f4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:41.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.525+0000 7f27d0d3e640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc069e90 0x7f27cc1f4e70 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f27bc01a0b0 tx=0x7f27bc01acc0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.525+0000 7f27d0d3e640 1 -- 172.21.15.46:0/317361347 shutdown_connections 2023-10-30T13:42:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.525+0000 7f27d0d3e640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27ac0664a0 0x7f27ac068960 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.525+0000 7f27d0d3e640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc06cb80 0x7f27cc14dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.525+0000 7f27d0d3e640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc06a270 0x7f27cc14d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.525+0000 7f27d0d3e640 1 --2- 172.21.15.46:0/317361347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc069e90 0x7f27cc1f4e70 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.525+0000 7f27d0d3e640 1 -- 172.21.15.46:0/317361347 >> 172.21.15.46:0/317361347 conn(0x7f27cc06dfd0 msgr2=0x7f27cc071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.525+0000 7f27d0d3e640 1 -- 172.21.15.46:0/317361347 shutdown_connections 2023-10-30T13:42:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:41.525+0000 7f27d0d3e640 1 -- 172.21.15.46:0/317361347 wait complete. 2023-10-30T13:42:41.549 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:42:41.550 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 99, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 99, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5830": { "addr": "172.21.15.154:6835/1224164391", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 1224164391, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 1224164391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5830, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:stopping", "state_seq": 28 }, "gid_6036": { "addr": "172.21.15.46:6836/3037060424", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3037060424, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3037060424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6036, "incarnation": 81, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 37 }, "gid_6201": { "addr": "172.21.15.46:6837/2546436963", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2546436963, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2546436963, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6201, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:42:34.173048+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6201, "mds_1": 6036, "mds_2": 5830 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.46:6833/190134548", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 190134548, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 190134548, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6366, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:42:43.551 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:43.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f584cb63640 1 Processor -- start 2023-10-30T13:42:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f584cb63640 1 -- start start 2023-10-30T13:42:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f584cb63640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f58481538f0 0x7f5848159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f584cb63640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58480ca6a0 con 0x7f584815cd50 2023-10-30T13:42:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f584cb63640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584815d300 con 0x7f58480cefa0 2023-10-30T13:42:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f584cb63640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f584815d5e0 con 0x7f58481538f0 2023-10-30T13:42:43.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f5846ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f584815cd50 0x7f58480c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54830/0 (socket says 172.21.15.46:54830) 2023-10-30T13:42:43.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f5846ffd640 1 -- 172.21.15.46:0/3012505978 learned_addr learned my addr 172.21.15.46:0/3012505978 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:43.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f58477fe640 1 --2- 172.21.15.46:0/3012505978 >> v2:172.21.15.154:3300/0 conn(0x7f58481538f0 0x7f5848159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:43.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77245876 0 0) 0x7f58480ca6a0 con 0x7f584815cd50 2023-10-30T13:42:43.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5830003530 con 0x7f584815cd50 2023-10-30T13:42:43.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3878311521 0 0) 0x7f5830003530 con 0x7f584815cd50 2023-10-30T13:42:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 >> v2:172.21.15.154:3300/0 conn(0x7f58481538f0 msgr2=0x7f5848159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f58467fc640 1 --2- 172.21.15.46:0/3012505978 >> v2:172.21.15.154:3300/0 conn(0x7f58481538f0 0x7f5848159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 >> v1:172.21.15.154:6789/0 conn(0x7f58480cefa0 legacy=0x7f58481640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584815d8c0 con 0x7f584815cd50 2023-10-30T13:42:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.701+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f583c001680 con 0x7f584815cd50 2023-10-30T13:42:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 >> v1:172.21.15.46:6789/0 conn(0x7f584815cd50 legacy=0x7f58480c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f58467fc640 1 --2- 172.21.15.46:0/3012505978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5830003980 0x7f5830003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f58467fc640 1 --2- 172.21.15.46:0/3012505978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58300047d0 0x7f5830004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f58467fc640 1 --2- 172.21.15.46:0/3012505978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5830005190 0x7f583000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584815d5e0 con 0x7f58300047d0 2023-10-30T13:42:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583000c040 con 0x7f5830003980 2023-10-30T13:42:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583000c200 con 0x7f5830005190 2023-10-30T13:42:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f5846ffd640 1 --2- 172.21.15.46:0/3012505978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58300047d0 0x7f5830004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f58477fe640 1 --2- 172.21.15.46:0/3012505978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5830003980 0x7f5830003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f5847fff640 1 --2- 172.21.15.46:0/3012505978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5830005190 0x7f583000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f5846ffd640 1 -- 172.21.15.46:0/3012505978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5830005190 msgr2=0x7f583000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f5846ffd640 1 --2- 172.21.15.46:0/3012505978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5830005190 0x7f583000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f5846ffd640 1 -- 172.21.15.46:0/3012505978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5830003980 msgr2=0x7f5830003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f5846ffd640 1 --2- 172.21.15.46:0/3012505978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5830003980 0x7f5830003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f5846ffd640 1 -- 172.21.15.46:0/3012505978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5830003530 con 0x7f58300047d0 2023-10-30T13:42:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f5847fff640 1 -- 172.21.15.46:0/3012505978 reap_dead start 2023-10-30T13:42:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f5846ffd640 1 --2- 172.21.15.46:0/3012505978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58300047d0 0x7f5830004c50 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f583c00c240 tx=0x7f583c01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f124d4d58d9626d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c0a5b40 con 0x7f58300047d0 2023-10-30T13:42:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583c0a5ca0 con 0x7f58300047d0 2023-10-30T13:42:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.705+0000 7f58467fc640 1 -- 172.21.15.46:0/3012505978 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c005b20 con 0x7f58300047d0 2023-10-30T13:42:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 -- 172.21.15.46:0/3012505978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58300047d0 msgr2=0x7f5830004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 --2- 172.21.15.46:0/3012505978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58300047d0 0x7f5830004c50 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f583c00c240 tx=0x7f583c01cc40 comp rx=0 tx=0).stop 2023-10-30T13:42:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 -- 172.21.15.46:0/3012505978 shutdown_connections 2023-10-30T13:42:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 --2- 172.21.15.46:0/3012505978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58300047d0 0x7f5830004c50 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 -- 172.21.15.46:0/3012505978 >> 172.21.15.46:0/3012505978 conn(0x7f584806a640 msgr2=0x7f5848065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 -- 172.21.15.46:0/3012505978 shutdown_connections 2023-10-30T13:42:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 -- 172.21.15.46:0/3012505978 wait complete. 2023-10-30T13:42:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 Processor -- start 2023-10-30T13:42:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 -- start start 2023-10-30T13:42:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58300047d0 0x7f584815b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58481417b0 0x7f58480b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5848148150 0x7f584813f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58480b3da0 con 0x7f58481417b0 2023-10-30T13:42:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5848140a80 con 0x7f5848148150 2023-10-30T13:42:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f584cb63640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58480b4f80 con 0x7f58300047d0 2023-10-30T13:42:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f5846ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58481417b0 0x7f58480b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f5846ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58481417b0 0x7f58480b3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43440/0 (socket says 172.21.15.46:43440) 2023-10-30T13:42:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.709+0000 7f5846ffd640 1 -- 172.21.15.46:0/2215061794 learned_addr learned my addr 172.21.15.46:0/2215061794 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f58477fe640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58300047d0 0x7f584815b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f5847fff640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5848148150 0x7f584813f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f5846ffd640 1 -- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58300047d0 msgr2=0x7f584815b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f5846ffd640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58300047d0 0x7f584815b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:43.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f5846ffd640 1 -- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5848148150 msgr2=0x7f584813f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:43.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f5846ffd640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5848148150 0x7f584813f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:43.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f5846ffd640 1 -- 172.21.15.46:0/2215061794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584815abc0 con 0x7f58481417b0 2023-10-30T13:42:43.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f5847fff640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5848148150 0x7f584813f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:43.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f5846ffd640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58481417b0 0x7f58480b3860 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f583c001940 tx=0x7f5830005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:43.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f582ffff640 1 -- 172.21.15.46:0/2215061794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c0b3070 con 0x7f58481417b0 2023-10-30T13:42:43.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f582ffff640 1 -- 172.21.15.46:0/2215061794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583c000ee0 con 0x7f58481417b0 2023-10-30T13:42:43.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f584cb63640 1 -- 172.21.15.46:0/2215061794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584815a7c0 con 0x7f58481417b0 2023-10-30T13:42:43.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f582ffff640 1 -- 172.21.15.46:0/2215061794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c001660 con 0x7f58481417b0 2023-10-30T13:42:43.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f584cb63640 1 -- 172.21.15.46:0/2215061794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5848140cd0 con 0x7f58481417b0 2023-10-30T13:42:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f584cb63640 1 -- 172.21.15.46:0/2215061794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f584804e170 con 0x7f58481417b0 2023-10-30T13:42:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.713+0000 7f582ffff640 1 -- 172.21.15.46:0/2215061794 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f583c007140 con 0x7f58481417b0 2023-10-30T13:42:43.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.717+0000 7f582ffff640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58340664d0 0x7f5834068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:43.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.717+0000 7f582ffff640 1 -- 172.21.15.46:0/2215061794 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f583c11e0d0 con 0x7f58481417b0 2023-10-30T13:42:43.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.717+0000 7f58477fe640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58340664d0 0x7f5834068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:43.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.717+0000 7f58477fe640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58340664d0 0x7f5834068990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5838009c60 tx=0x7f5838009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:43.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.717+0000 7f582ffff640 1 -- 172.21.15.46:0/2215061794 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f583c0eb160 con 0x7f58481417b0 2023-10-30T13:42:43.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.905+0000 7f584cb63640 1 -- 172.21.15.46:0/2215061794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58481463b0 con 0x7f58481417b0 2023-10-30T13:42:43.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.905+0000 7f582ffff640 1 -- 172.21.15.46:0/2215061794 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f583c0ee780 con 0x7f58481417b0 2023-10-30T13:42:43.912 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:43.913 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:34.173048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:43.913 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 99 2023-10-30T13:42:43.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.909+0000 7f584cb63640 1 -- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58340664d0 msgr2=0x7f5834068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:43.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.909+0000 7f584cb63640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58340664d0 0x7f5834068990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5838009c60 tx=0x7f5838009190 comp rx=0 tx=0).stop 2023-10-30T13:42:43.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.909+0000 7f584cb63640 1 -- 172.21.15.46:0/2215061794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58481417b0 msgr2=0x7f58480b3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:43.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.909+0000 7f584cb63640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58481417b0 0x7f58480b3860 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f583c001940 tx=0x7f5830005800 comp rx=0 tx=0).stop 2023-10-30T13:42:43.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.909+0000 7f584cb63640 1 -- 172.21.15.46:0/2215061794 shutdown_connections 2023-10-30T13:42:43.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.909+0000 7f584cb63640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58340664d0 0x7f5834068990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:43.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.909+0000 7f584cb63640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5848148150 0x7f584813f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:43.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.909+0000 7f584cb63640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58481417b0 0x7f58480b3860 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:43.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.909+0000 7f584cb63640 1 --2- 172.21.15.46:0/2215061794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58300047d0 0x7f584815b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:43.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.909+0000 7f584cb63640 1 -- 172.21.15.46:0/2215061794 >> 172.21.15.46:0/2215061794 conn(0x7f584806a640 msgr2=0x7f58480c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:43.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.909+0000 7f584cb63640 1 -- 172.21.15.46:0/2215061794 shutdown_connections 2023-10-30T13:42:43.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:43.909+0000 7f584cb63640 1 -- 172.21.15.46:0/2215061794 wait complete. 2023-10-30T13:42:43.933 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:45.934 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:46.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d4d8c9640 1 Processor -- start 2023-10-30T13:42:46.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d4d8c9640 1 -- start start 2023-10-30T13:42:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d4d8c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d480c6b80 0x7f4d480bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d4d8c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d4813f4a0 0x7f4d480c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d4d8c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d480c0a50 con 0x7f4d480c6f60 2023-10-30T13:42:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d4d8c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4806f6e0 con 0x7f4d4813f4a0 2023-10-30T13:42:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d4d8c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4806f9c0 con 0x7f4d480c6b80 2023-10-30T13:42:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d477fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d4813f4a0 0x7f4d480c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d477fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d4813f4a0 0x7f4d480c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43444/0 (socket says 172.21.15.46:43444) 2023-10-30T13:42:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d477fe640 1 -- 172.21.15.46:0/1229080029 learned_addr learned my addr 172.21.15.46:0/1229080029 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d46ffd640 1 --2- 172.21.15.46:0/1229080029 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d480c6b80 0x7f4d480bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d477fe640 1 -- 172.21.15.46:0/1229080029 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d480c6b80 msgr2=0x7f4d480bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d477fe640 1 --2- 172.21.15.46:0/1229080029 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d480c6b80 0x7f4d480bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d477fe640 1 -- 172.21.15.46:0/1229080029 >> v1:172.21.15.154:6789/0 conn(0x7f4d480c6f60 legacy=0x7f4d4813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.085+0000 7f4d477fe640 1 -- 172.21.15.46:0/1229080029 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4806fca0 con 0x7f4d4813f4a0 2023-10-30T13:42:46.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.089+0000 7f4d477fe640 1 --2- 172.21.15.46:0/1229080029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d4813f4a0 0x7f4d480c0400 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009e50 tx=0x7f4d4001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01fd751346678db server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:46.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.089+0000 7f4d467fc640 1 -- 172.21.15.46:0/1229080029 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d400a57f0 con 0x7f4d4813f4a0 2023-10-30T13:42:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.089+0000 7f4d467fc640 1 -- 172.21.15.46:0/1229080029 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d400a5950 con 0x7f4d4813f4a0 2023-10-30T13:42:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.089+0000 7f4d467fc640 1 -- 172.21.15.46:0/1229080029 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d400a5ce0 con 0x7f4d4813f4a0 2023-10-30T13:42:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.089+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/1229080029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d4813f4a0 msgr2=0x7f4d480c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.089+0000 7f4d4d8c9640 1 --2- 172.21.15.46:0/1229080029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d4813f4a0 0x7f4d480c0400 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009e50 tx=0x7f4d4001ca10 comp rx=0 tx=0).stop 2023-10-30T13:42:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.089+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/1229080029 shutdown_connections 2023-10-30T13:42:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.089+0000 7f4d4d8c9640 1 --2- 172.21.15.46:0/1229080029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d4813f4a0 0x7f4d480c0400 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.089+0000 7f4d4d8c9640 1 --2- 172.21.15.46:0/1229080029 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d480c6b80 0x7f4d480bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.089+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/1229080029 >> 172.21.15.46:0/1229080029 conn(0x7f4d480c7910 msgr2=0x7f4d480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.089+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/1229080029 shutdown_connections 2023-10-30T13:42:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.089+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/1229080029 wait complete. 2023-10-30T13:42:46.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d4d8c9640 1 Processor -- start 2023-10-30T13:42:46.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d4d8c9640 1 -- start start 2023-10-30T13:42:46.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d4d8c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d480c6b80 0x7f4d480c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d4d8c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d480c6f60 0x7f4d480ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d4d8c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d4813f4a0 0x7f4d48155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d4d8c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d481f5ec0 con 0x7f4d4813f4a0 2023-10-30T13:42:46.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d4d8c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d48154ee0 con 0x7f4d480c6b80 2023-10-30T13:42:46.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d4d8c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d481f63d0 con 0x7f4d480c6f60 2023-10-30T13:42:46.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d477fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d4813f4a0 0x7f4d48155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d477fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d4813f4a0 0x7f4d48155370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43460/0 (socket says 172.21.15.46:43460) 2023-10-30T13:42:46.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d477fe640 1 -- 172.21.15.46:0/2156862227 learned_addr learned my addr 172.21.15.46:0/2156862227 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:46.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d46ffd640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d480c6b80 0x7f4d480c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d3e7fc640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d480c6f60 0x7f4d480ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d477fe640 1 -- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d480c6f60 msgr2=0x7f4d480ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d477fe640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d480c6f60 0x7f4d480ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d477fe640 1 -- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d480c6b80 msgr2=0x7f4d480c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d477fe640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d480c6b80 0x7f4d480c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d477fe640 1 -- 172.21.15.46:0/2156862227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d40009b30 con 0x7f4d4813f4a0 2023-10-30T13:42:46.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d477fe640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d4813f4a0 0x7f4d48155370 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4d4001a640 tx=0x7f4d400197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:46.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.093+0000 7f4d44ff9640 1 -- 172.21.15.46:0/2156862227 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d400b8880 con 0x7f4d4813f4a0 2023-10-30T13:42:46.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.097+0000 7f4d44ff9640 1 -- 172.21.15.46:0/2156862227 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d400b89e0 con 0x7f4d4813f4a0 2023-10-30T13:42:46.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.097+0000 7f4d44ff9640 1 -- 172.21.15.46:0/2156862227 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d400b8d90 con 0x7f4d4813f4a0 2023-10-30T13:42:46.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.097+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/2156862227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d481f60f0 con 0x7f4d4813f4a0 2023-10-30T13:42:46.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.097+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/2156862227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d48154890 con 0x7f4d4813f4a0 2023-10-30T13:42:46.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.097+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/2156862227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d48155090 con 0x7f4d4813f4a0 2023-10-30T13:42:46.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.097+0000 7f4d44ff9640 1 -- 172.21.15.46:0/2156862227 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4d400c3460 con 0x7f4d4813f4a0 2023-10-30T13:42:46.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.097+0000 7f4d44ff9640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4d280664a0 0x7f4d28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.097+0000 7f4d44ff9640 1 -- 172.21.15.46:0/2156862227 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f4d4011fa90 con 0x7f4d4813f4a0 2023-10-30T13:42:46.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.097+0000 7f4d46ffd640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4d280664a0 0x7f4d28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.101+0000 7f4d46ffd640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4d280664a0 0x7f4d28068960 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4d30007a60 tx=0x7f4d30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:46.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.101+0000 7f4d44ff9640 1 -- 172.21.15.46:0/2156862227 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4d48155090 con 0x7f4d4813f4a0 2023-10-30T13:42:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.293+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/2156862227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d48155090 con 0x7f4d4813f4a0 2023-10-30T13:42:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.293+0000 7f4d44ff9640 1 -- 172.21.15.46:0/2156862227 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f4d48155090 con 0x7f4d4813f4a0 2023-10-30T13:42:46.299 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:46.300 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:34.173048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 99 2023-10-30T13:42:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.297+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4d280664a0 msgr2=0x7f4d28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.297+0000 7f4d4d8c9640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4d280664a0 0x7f4d28068960 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4d30007a60 tx=0x7f4d30008040 comp rx=0 tx=0).stop 2023-10-30T13:42:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.297+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/2156862227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d4813f4a0 msgr2=0x7f4d48155370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.297+0000 7f4d4d8c9640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d4813f4a0 0x7f4d48155370 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4d4001a640 tx=0x7f4d400197b0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.297+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/2156862227 shutdown_connections 2023-10-30T13:42:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.297+0000 7f4d4d8c9640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4d280664a0 0x7f4d28068960 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.297+0000 7f4d4d8c9640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d4813f4a0 0x7f4d48155370 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.297+0000 7f4d4d8c9640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d480c6f60 0x7f4d480ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.297+0000 7f4d4d8c9640 1 --2- 172.21.15.46:0/2156862227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d480c6b80 0x7f4d480c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.297+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/2156862227 >> 172.21.15.46:0/2156862227 conn(0x7f4d480c7910 msgr2=0x7f4d481532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.297+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/2156862227 shutdown_connections 2023-10-30T13:42:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:46.297+0000 7f4d4d8c9640 1 -- 172.21.15.46:0/2156862227 wait complete. 2023-10-30T13:42:46.320 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:48.321 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.469+0000 7f3462046640 1 Processor -- start 2023-10-30T13:42:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.469+0000 7f3462046640 1 -- start start 2023-10-30T13:42:48.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f3462046640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f345c0c6b80 0x7f345c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f3462046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345c13f4a0 0x7f345c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f3462046640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345c0c0a50 con 0x7f345c0c6f60 2023-10-30T13:42:48.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f3462046640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f345c06f6e0 con 0x7f345c0c6b80 2023-10-30T13:42:48.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f3462046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c06f9c0 con 0x7f345c13f4a0 2023-10-30T13:42:48.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f345b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f345c0c6b80 0x7f345c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f345b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f345c0c6b80 0x7f345c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43856/0 (socket says 172.21.15.46:43856) 2023-10-30T13:42:48.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f345b7fe640 1 -- 172.21.15.46:0/209166611 learned_addr learned my addr 172.21.15.46:0/209166611 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f345affd640 1 --1- 172.21.15.46:0/209166611 >> v1:172.21.15.46:6789/0 conn(0x7f345c0c6f60 0x7f345c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33950/0 (socket says 172.21.15.46:33950) 2023-10-30T13:42:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f345bfff640 1 --2- 172.21.15.46:0/209166611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345c13f4a0 0x7f345c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f345b7fe640 1 -- 172.21.15.46:0/209166611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345c13f4a0 msgr2=0x7f345c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f345b7fe640 1 --2- 172.21.15.46:0/209166611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345c13f4a0 0x7f345c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f345b7fe640 1 -- 172.21.15.46:0/209166611 >> v1:172.21.15.46:6789/0 conn(0x7f345c0c6f60 legacy=0x7f345c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:48.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.473+0000 7f345b7fe640 1 -- 172.21.15.46:0/209166611 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c06fca0 con 0x7f345c0c6b80 2023-10-30T13:42:48.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f345b7fe640 1 --2- 172.21.15.46:0/209166611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f345c0c6b80 0x7f345c0bfec0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f344c009870 tx=0x7f344c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d329635337de01a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:48.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f345a7fc640 1 -- 172.21.15.46:0/209166611 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c0ab650 con 0x7f345c0c6b80 2023-10-30T13:42:48.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f345a7fc640 1 -- 172.21.15.46:0/209166611 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344c0ab7b0 con 0x7f345c0c6b80 2023-10-30T13:42:48.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f345a7fc640 1 -- 172.21.15.46:0/209166611 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c0abb40 con 0x7f345c0c6b80 2023-10-30T13:42:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f3462046640 1 -- 172.21.15.46:0/209166611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f345c0c6b80 msgr2=0x7f345c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f3462046640 1 --2- 172.21.15.46:0/209166611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f345c0c6b80 0x7f345c0bfec0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f344c009870 tx=0x7f344c01c540 comp rx=0 tx=0).stop 2023-10-30T13:42:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f3462046640 1 -- 172.21.15.46:0/209166611 shutdown_connections 2023-10-30T13:42:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f3462046640 1 --2- 172.21.15.46:0/209166611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345c13f4a0 0x7f345c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f3462046640 1 --2- 172.21.15.46:0/209166611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f345c0c6b80 0x7f345c0bfec0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f3462046640 1 -- 172.21.15.46:0/209166611 >> 172.21.15.46:0/209166611 conn(0x7f345c0c7910 msgr2=0x7f345c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f3462046640 1 -- 172.21.15.46:0/209166611 shutdown_connections 2023-10-30T13:42:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f3462046640 1 -- 172.21.15.46:0/209166611 wait complete. 2023-10-30T13:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f3462046640 1 Processor -- start 2023-10-30T13:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.477+0000 7f3462046640 1 -- start start 2023-10-30T13:42:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3462046640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f345c0c6b80 0x7f345c0c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3462046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345c0c6f60 0x7f345c0c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3462046640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f345c13f4a0 0x7f345c13a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3462046640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c158120 con 0x7f345c0c6b80 2023-10-30T13:42:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3462046640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c1589c0 con 0x7f345c13f4a0 2023-10-30T13:42:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3462046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c0c9300 con 0x7f345c0c6f60 2023-10-30T13:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f345b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f345c0c6b80 0x7f345c0c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f345b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f345c0c6b80 0x7f345c0c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43868/0 (socket says 172.21.15.46:43868) 2023-10-30T13:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f345b7fe640 1 -- 172.21.15.46:0/2238954697 learned_addr learned my addr 172.21.15.46:0/2238954697 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f345affd640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345c0c6f60 0x7f345c0c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f345bfff640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f345c13f4a0 0x7f345c13a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f345b7fe640 1 -- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345c0c6f60 msgr2=0x7f345c0c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:48.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f345b7fe640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345c0c6f60 0x7f345c0c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f345b7fe640 1 -- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f345c13f4a0 msgr2=0x7f345c13a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:48.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f345b7fe640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f345c13f4a0 0x7f345c13a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f345b7fe640 1 -- 172.21.15.46:0/2238954697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c0094d0 con 0x7f345c0c6b80 2023-10-30T13:42:48.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f345b7fe640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f345c0c6b80 0x7f345c0c8dc0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f344c01abf0 tx=0x7f344c01ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3443fff640 1 -- 172.21.15.46:0/2238954697 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c0aa8c0 con 0x7f345c0c6b80 2023-10-30T13:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3443fff640 1 -- 172.21.15.46:0/2238954697 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344c0aaa20 con 0x7f345c0c6b80 2023-10-30T13:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3443fff640 1 -- 172.21.15.46:0/2238954697 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c0aadd0 con 0x7f345c0c6b80 2023-10-30T13:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3462046640 1 -- 172.21.15.46:0/2238954697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345c157950 con 0x7f345c0c6b80 2023-10-30T13:42:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3462046640 1 -- 172.21.15.46:0/2238954697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f345c158300 con 0x7f345c0c6b80 2023-10-30T13:42:48.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3462046640 1 -- 172.21.15.46:0/2238954697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345c04e170 con 0x7f345c0c6b80 2023-10-30T13:42:48.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.481+0000 7f3443fff640 1 -- 172.21.15.46:0/2238954697 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f344c0be3f0 con 0x7f345c0c6b80 2023-10-30T13:42:48.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.485+0000 7f3443fff640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f34480664d0 0x7f3448068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.485+0000 7f3443fff640 1 -- 172.21.15.46:0/2238954697 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f344c11eff0 con 0x7f345c0c6b80 2023-10-30T13:42:48.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.485+0000 7f345affd640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f34480664d0 0x7f3448068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.485+0000 7f345affd640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f34480664d0 0x7f3448068990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f345c13bb60 tx=0x7f3450007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.489+0000 7f3443fff640 1 -- 172.21.15.46:0/2238954697 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f344c0ec020 con 0x7f345c0c6b80 2023-10-30T13:42:48.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.677+0000 7f3462046640 1 -- 172.21.15.46:0/2238954697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f345c13b410 con 0x7f345c0c6b80 2023-10-30T13:42:48.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.677+0000 7f3443fff640 1 -- 172.21.15.46:0/2238954697 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f344c0c4020 con 0x7f345c0c6b80 2023-10-30T13:42:48.682 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:48.683 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:34.173048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":6201,"mds_1":6036,"mds_2":5830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5830":{"gid":5830,"name":"b","rank":2,"incarnation":72,"state":"up:stopping","state_seq":28,"addr":"172.21.15.154:6835/1224164391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1224164391},{"type":"v1","addr":"172.21.15.154:6835","nonce":1224164391}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:48.683 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 99 2023-10-30T13:42:48.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.677+0000 7f3462046640 1 -- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f34480664d0 msgr2=0x7f3448068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:48.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.677+0000 7f3462046640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f34480664d0 0x7f3448068990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f345c13bb60 tx=0x7f3450007c90 comp rx=0 tx=0).stop 2023-10-30T13:42:48.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.681+0000 7f3462046640 1 -- 172.21.15.46:0/2238954697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f345c0c6b80 msgr2=0x7f345c0c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:48.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.681+0000 7f3462046640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f345c0c6b80 0x7f345c0c8dc0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f344c01abf0 tx=0x7f344c01ad80 comp rx=0 tx=0).stop 2023-10-30T13:42:48.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.681+0000 7f3462046640 1 -- 172.21.15.46:0/2238954697 shutdown_connections 2023-10-30T13:42:48.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.681+0000 7f3462046640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f34480664d0 0x7f3448068990 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.681+0000 7f3462046640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f345c13f4a0 0x7f345c13a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.681+0000 7f3462046640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345c0c6f60 0x7f345c0c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.681+0000 7f3462046640 1 --2- 172.21.15.46:0/2238954697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f345c0c6b80 0x7f345c0c8dc0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.681+0000 7f3462046640 1 -- 172.21.15.46:0/2238954697 >> 172.21.15.46:0/2238954697 conn(0x7f345c0c7910 msgr2=0x7f345c13ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:48.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.681+0000 7f3462046640 1 -- 172.21.15.46:0/2238954697 shutdown_connections 2023-10-30T13:42:48.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:48.681+0000 7f3462046640 1 -- 172.21.15.46:0/2238954697 wait complete. 2023-10-30T13:42:48.704 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:48.761 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:42:48.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:42:50.704 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:50.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.853+0000 7f1ef1009640 1 Processor -- start 2023-10-30T13:42:50.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.853+0000 7f1ef1009640 1 -- start start 2023-10-30T13:42:50.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1ef1009640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eec153bf0 0x7f1eec071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:50.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1ef1009640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1eec0cb550 0x7f1eec0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:50.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1ef1009640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec0c7b90 con 0x7f1eec0c6fe0 2023-10-30T13:42:50.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1ef1009640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec15d870 con 0x7f1eec0cb550 2023-10-30T13:42:50.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1ef1009640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec0c60e0 con 0x7f1eec153bf0 2023-10-30T13:42:50.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1eead9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eec153bf0 0x7f1eec071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1eeb59e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1eec0cb550 0x7f1eec0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1ee3fff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f1eec0c6fe0 0x7f1eec0c7480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:35352/0 (socket says 172.21.15.46:35352) 2023-10-30T13:42:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1ee3fff640 1 -- 172.21.15.46:0/3723619975 learned_addr learned my addr 172.21.15.46:0/3723619975 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1eead9d640 1 -- 172.21.15.46:0/3723619975 >> v2:172.21.15.154:3300/0 conn(0x7f1eec0cb550 msgr2=0x7f1eec0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1eead9d640 1 --2- 172.21.15.46:0/3723619975 >> v2:172.21.15.154:3300/0 conn(0x7f1eec0cb550 0x7f1eec0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:50.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1eead9d640 1 -- 172.21.15.46:0/3723619975 >> v1:172.21.15.154:6789/0 conn(0x7f1eec0c6fe0 legacy=0x7f1eec0c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:50.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1eead9d640 1 -- 172.21.15.46:0/3723619975 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec0c63a0 con 0x7f1eec153bf0 2023-10-30T13:42:50.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.857+0000 7f1eeb59e640 1 --2- 172.21.15.46:0/3723619975 >> v2:172.21.15.154:3300/0 conn(0x7f1eec0cb550 0x7f1eec0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:42:50.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1eead9d640 1 --2- 172.21.15.46:0/3723619975 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eec153bf0 0x7f1eec071260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0097c0 tx=0x7f1edc01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bee18aef974dca5d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:50.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1eea59c640 1 -- 172.21.15.46:0/3723619975 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc0ab650 con 0x7f1eec153bf0 2023-10-30T13:42:50.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1eea59c640 1 -- 172.21.15.46:0/3723619975 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1edc0ab7b0 con 0x7f1eec153bf0 2023-10-30T13:42:50.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1eea59c640 1 -- 172.21.15.46:0/3723619975 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc0abb40 con 0x7f1eec153bf0 2023-10-30T13:42:50.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1ef1009640 1 -- 172.21.15.46:0/3723619975 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eec153bf0 msgr2=0x7f1eec071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:50.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1ef1009640 1 --2- 172.21.15.46:0/3723619975 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eec153bf0 0x7f1eec071260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0097c0 tx=0x7f1edc01c450 comp rx=0 tx=0).stop 2023-10-30T13:42:50.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1ef1009640 1 -- 172.21.15.46:0/3723619975 shutdown_connections 2023-10-30T13:42:50.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1ef1009640 1 --2- 172.21.15.46:0/3723619975 >> v2:172.21.15.154:3300/0 conn(0x7f1eec0cb550 0x7f1eec0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:50.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1ef1009640 1 --2- 172.21.15.46:0/3723619975 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eec153bf0 0x7f1eec071260 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:50.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1ef1009640 1 -- 172.21.15.46:0/3723619975 >> 172.21.15.46:0/3723619975 conn(0x7f1eec06c6e0 msgr2=0x7f1eec069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:50.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1ef1009640 1 -- 172.21.15.46:0/3723619975 shutdown_connections 2023-10-30T13:42:50.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1ef1009640 1 -- 172.21.15.46:0/3723619975 wait complete. 2023-10-30T13:42:50.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.861+0000 7f1ef1009640 1 Processor -- start 2023-10-30T13:42:50.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ef1009640 1 -- start start 2023-10-30T13:42:50.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ef1009640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eec0c6fe0 0x7f1eec1f57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:50.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ef1009640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eec0cb550 0x7f1eec13e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:50.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ef1009640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eec153bf0 0x7f1eec13fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:50.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ef1009640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec15c990 con 0x7f1eec153bf0 2023-10-30T13:42:50.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ef1009640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec0c3dd0 con 0x7f1eec0c6fe0 2023-10-30T13:42:50.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ef1009640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec1f5d20 con 0x7f1eec0cb550 2023-10-30T13:42:50.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1eeb59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eec153bf0 0x7f1eec13fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:50.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1eead9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eec0c6fe0 0x7f1eec1f57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:50.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1eeb59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eec153bf0 0x7f1eec13fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43870/0 (socket says 172.21.15.46:43870) 2023-10-30T13:42:50.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ee3fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eec0cb550 0x7f1eec13e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:50.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1eeb59e640 1 -- 172.21.15.46:0/997057254 learned_addr learned my addr 172.21.15.46:0/997057254 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:50.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1eeb59e640 1 -- 172.21.15.46:0/997057254 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eec0cb550 msgr2=0x7f1eec13e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:50.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1eeb59e640 1 --2- 172.21.15.46:0/997057254 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eec0cb550 0x7f1eec13e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:50.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1eeb59e640 1 -- 172.21.15.46:0/997057254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eec0c6fe0 msgr2=0x7f1eec1f57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:50.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1eeb59e640 1 --2- 172.21.15.46:0/997057254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eec0c6fe0 0x7f1eec1f57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:50.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1eeb59e640 1 -- 172.21.15.46:0/997057254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee4009a70 con 0x7f1eec153bf0 2023-10-30T13:42:50.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1eeb59e640 1 --2- 172.21.15.46:0/997057254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eec153bf0 0x7f1eec13fcf0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400efc0 tx=0x7f1ee4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:50.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ee37fe640 1 -- 172.21.15.46:0/997057254 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4009d70 con 0x7f1eec153bf0 2023-10-30T13:42:50.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ee37fe640 1 -- 172.21.15.46:0/997057254 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee400f040 con 0x7f1eec153bf0 2023-10-30T13:42:50.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ee37fe640 1 -- 172.21.15.46:0/997057254 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee40039a0 con 0x7f1eec153bf0 2023-10-30T13:42:50.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ef1009640 1 -- 172.21.15.46:0/997057254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1edc009420 con 0x7f1eec153bf0 2023-10-30T13:42:50.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ef1009640 1 -- 172.21.15.46:0/997057254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eec15b8f0 con 0x7f1eec153bf0 2023-10-30T13:42:50.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.865+0000 7f1ef1009640 1 -- 172.21.15.46:0/997057254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eec04e170 con 0x7f1eec153bf0 2023-10-30T13:42:50.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.869+0000 7f1ee37fe640 1 -- 172.21.15.46:0/997057254 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ee40144a0 con 0x7f1eec153bf0 2023-10-30T13:42:50.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.869+0000 7f1ee37fe640 1 --2- 172.21.15.46:0/997057254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1ecc0664a0 0x7f1ecc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:50.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.869+0000 7f1ee37fe640 1 -- 172.21.15.46:0/997057254 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f1ee4084ea0 con 0x7f1eec153bf0 2023-10-30T13:42:50.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.869+0000 7f1eead9d640 1 --2- 172.21.15.46:0/997057254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1ecc0664a0 0x7f1ecc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:50.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.869+0000 7f1eead9d640 1 --2- 172.21.15.46:0/997057254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1ecc0664a0 0x7f1ecc068960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0af000 tx=0x7f1edc0b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:50.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:50.873+0000 7f1ee37fe640 1 -- 172.21.15.46:0/997057254 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ee4051eb0 con 0x7f1eec153bf0 2023-10-30T13:42:51.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.065+0000 7f1ef1009640 1 -- 172.21.15.46:0/997057254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1eec164040 con 0x7f1eec153bf0 2023-10-30T13:42:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.065+0000 7f1ee37fe640 1 -- 172.21.15.46:0/997057254 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f1ee4018020 con 0x7f1eec153bf0 2023-10-30T13:42:51.070 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:51.070 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:50.198737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6036},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:51.071 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 100 2023-10-30T13:42:51.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.069+0000 7f1ef1009640 1 -- 172.21.15.46:0/997057254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1ecc0664a0 msgr2=0x7f1ecc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.069+0000 7f1ef1009640 1 --2- 172.21.15.46:0/997057254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1ecc0664a0 0x7f1ecc068960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0af000 tx=0x7f1edc0b6040 comp rx=0 tx=0).stop 2023-10-30T13:42:51.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.069+0000 7f1ef1009640 1 -- 172.21.15.46:0/997057254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eec153bf0 msgr2=0x7f1eec13fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.069+0000 7f1ef1009640 1 --2- 172.21.15.46:0/997057254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eec153bf0 0x7f1eec13fcf0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400efc0 tx=0x7f1ee4009f90 comp rx=0 tx=0).stop 2023-10-30T13:42:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.069+0000 7f1ef1009640 1 -- 172.21.15.46:0/997057254 shutdown_connections 2023-10-30T13:42:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.069+0000 7f1ef1009640 1 --2- 172.21.15.46:0/997057254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1ecc0664a0 0x7f1ecc068960 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.069+0000 7f1ef1009640 1 --2- 172.21.15.46:0/997057254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eec153bf0 0x7f1eec13fcf0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.069+0000 7f1ef1009640 1 --2- 172.21.15.46:0/997057254 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eec0cb550 0x7f1eec13e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.069+0000 7f1ef1009640 1 --2- 172.21.15.46:0/997057254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eec0c6fe0 0x7f1eec1f57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.069+0000 7f1ef1009640 1 -- 172.21.15.46:0/997057254 >> 172.21.15.46:0/997057254 conn(0x7f1eec06c6e0 msgr2=0x7f1eec14a260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:51.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.069+0000 7f1ef1009640 1 -- 172.21.15.46:0/997057254 shutdown_connections 2023-10-30T13:42:51.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.069+0000 7f1ef1009640 1 -- 172.21.15.46:0/997057254 wait complete. 2023-10-30T13:42:51.095 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 2 alive and active, now stable! 2023-10-30T13:42:51.095 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=2) 2023-10-30T13:42:51.095 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T13:42:51.096 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:42:51.112 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T13:42:51.112 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:51.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.249+0000 7f12c6ad6640 1 Processor -- start 2023-10-30T13:42:51.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.249+0000 7f12c6ad6640 1 -- start start 2023-10-30T13:42:51.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c6ad6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12c00c6b80 0x7f12c00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c6ad6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12c013f4a0 0x7f12c00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c6ad6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c00c0a50 con 0x7f12c00c6f60 2023-10-30T13:42:51.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c6ad6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c006f6e0 con 0x7f12c00c6b80 2023-10-30T13:42:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c6ad6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c006f9c0 con 0x7f12c013f4a0 2023-10-30T13:42:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c4872640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12c00c6b80 0x7f12c00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12bffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f12c00c6f60 0x7f12c013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33956/0 (socket says 172.21.15.46:33956) 2023-10-30T13:42:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c4872640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12c00c6b80 0x7f12c00bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43874/0 (socket says 172.21.15.46:43874) 2023-10-30T13:42:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12bffff640 1 -- 172.21.15.46:0/3778378820 learned_addr learned my addr 172.21.15.46:0/3778378820 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:51.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c5073640 1 --2- 172.21.15.46:0/3778378820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12c013f4a0 0x7f12c00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c4872640 1 -- 172.21.15.46:0/3778378820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12c013f4a0 msgr2=0x7f12c00c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c4872640 1 --2- 172.21.15.46:0/3778378820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12c013f4a0 0x7f12c00c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c4872640 1 -- 172.21.15.46:0/3778378820 >> v1:172.21.15.46:6789/0 conn(0x7f12c00c6f60 legacy=0x7f12c013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12bf7fe640 1 -- 172.21.15.46:0/3778378820 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106318216 0 0) 0x7f12c00c0a50 con 0x7f12c00c6f60 2023-10-30T13:42:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c4872640 1 -- 172.21.15.46:0/3778378820 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c006fca0 con 0x7f12c00c6b80 2023-10-30T13:42:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.253+0000 7f12c5073640 1 --2- 172.21.15.46:0/3778378820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12c013f4a0 0x7f12c00c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12c4872640 1 --2- 172.21.15.46:0/3778378820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c00c6b80 0x7f12c00bfec0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f12b40097c0 tx=0x7f12b401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6413abf7feb14c96 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12bf7fe640 1 -- 172.21.15.46:0/3778378820 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b40ad640 con 0x7f12c00c6b80 2023-10-30T13:42:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12bf7fe640 1 -- 172.21.15.46:0/3778378820 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b40ad7a0 con 0x7f12c00c6b80 2023-10-30T13:42:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12bf7fe640 1 -- 172.21.15.46:0/3778378820 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b40adb30 con 0x7f12c00c6b80 2023-10-30T13:42:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12c6ad6640 1 -- 172.21.15.46:0/3778378820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c00c6b80 msgr2=0x7f12c00bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12c6ad6640 1 --2- 172.21.15.46:0/3778378820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c00c6b80 0x7f12c00bfec0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f12b40097c0 tx=0x7f12b401c480 comp rx=0 tx=0).stop 2023-10-30T13:42:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12c6ad6640 1 -- 172.21.15.46:0/3778378820 shutdown_connections 2023-10-30T13:42:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12c6ad6640 1 --2- 172.21.15.46:0/3778378820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12c013f4a0 0x7f12c00c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12c6ad6640 1 --2- 172.21.15.46:0/3778378820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c00c6b80 0x7f12c00bfec0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12c6ad6640 1 -- 172.21.15.46:0/3778378820 >> 172.21.15.46:0/3778378820 conn(0x7f12c00c7910 msgr2=0x7f12c00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12c6ad6640 1 -- 172.21.15.46:0/3778378820 shutdown_connections 2023-10-30T13:42:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12c6ad6640 1 -- 172.21.15.46:0/3778378820 wait complete. 2023-10-30T13:42:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12c6ad6640 1 Processor -- start 2023-10-30T13:42:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.257+0000 7f12c6ad6640 1 -- start start 2023-10-30T13:42:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c6ad6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12c00c6b80 0x7f12c01f53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c6ad6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12c00c6f60 0x7f12c0158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c6ad6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c013f4a0 0x7f12c0135480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c6ad6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c013bcb0 con 0x7f12c013f4a0 2023-10-30T13:42:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c6ad6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c0154a90 con 0x7f12c00c6b80 2023-10-30T13:42:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c6ad6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c0154540 con 0x7f12c00c6f60 2023-10-30T13:42:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c5073640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c013f4a0 0x7f12c0135480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c5073640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c013f4a0 0x7f12c0135480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43888/0 (socket says 172.21.15.46:43888) 2023-10-30T13:42:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c5073640 1 -- 172.21.15.46:0/2324683154 learned_addr learned my addr 172.21.15.46:0/2324683154 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c4872640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12c00c6b80 0x7f12c01f53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12bffff640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12c00c6f60 0x7f12c0158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c5073640 1 -- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12c00c6f60 msgr2=0x7f12c0158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c5073640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12c00c6f60 0x7f12c0158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c5073640 1 -- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12c00c6b80 msgr2=0x7f12c01f53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c5073640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12c00c6b80 0x7f12c01f53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c5073640 1 -- 172.21.15.46:0/2324683154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4009420 con 0x7f12c013f4a0 2023-10-30T13:42:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c5073640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c013f4a0 0x7f12c0135480 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f12b8007e50 tx=0x7f12b800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12bd7fa640 1 -- 172.21.15.46:0/2324683154 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b80039a0 con 0x7f12c013f4a0 2023-10-30T13:42:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12bd7fa640 1 -- 172.21.15.46:0/2324683154 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b800f040 con 0x7f12c013f4a0 2023-10-30T13:42:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c6ad6640 1 -- 172.21.15.46:0/2324683154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c01f58e0 con 0x7f12c013f4a0 2023-10-30T13:42:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12bd7fa640 1 -- 172.21.15.46:0/2324683154 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8003d40 con 0x7f12c013f4a0 2023-10-30T13:42:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c6ad6640 1 -- 172.21.15.46:0/2324683154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c01344d0 con 0x7f12c013f4a0 2023-10-30T13:42:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12c6ad6640 1 -- 172.21.15.46:0/2324683154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c004e170 con 0x7f12c013f4a0 2023-10-30T13:42:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.261+0000 7f12bd7fa640 1 -- 172.21.15.46:0/2324683154 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f12b8004030 con 0x7f12c013f4a0 2023-10-30T13:42:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.265+0000 7f12bd7fa640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12a40664d0 0x7f12a4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.265+0000 7f12bd7fa640 1 -- 172.21.15.46:0/2324683154 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f12b8084460 con 0x7f12c013f4a0 2023-10-30T13:42:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.265+0000 7f12c4872640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12a40664d0 0x7f12a4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.265+0000 7f12c4872640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12a40664d0 0x7f12a4068990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f12b40af000 tx=0x7f12b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.269+0000 7f12bd7fa640 1 -- 172.21.15.46:0/2324683154 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f12b8051590 con 0x7f12c013f4a0 2023-10-30T13:42:51.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.529+0000 7f12c6ad6640 1 -- 172.21.15.46:0/2324683154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12c006c5d0 con 0x7f12c013f4a0 2023-10-30T13:42:51.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.529+0000 7f12bd7fa640 1 -- 172.21.15.46:0/2324683154 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f12b8054bb0 con 0x7f12c013f4a0 2023-10-30T13:42:51.533 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:51.534 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:50.198737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6036},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:51.534 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 100 2023-10-30T13:42:51.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.529+0000 7f12c6ad6640 1 -- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12a40664d0 msgr2=0x7f12a4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.529+0000 7f12c6ad6640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12a40664d0 0x7f12a4068990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f12b40af000 tx=0x7f12b4008040 comp rx=0 tx=0).stop 2023-10-30T13:42:51.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.529+0000 7f12c6ad6640 1 -- 172.21.15.46:0/2324683154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c013f4a0 msgr2=0x7f12c0135480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.529+0000 7f12c6ad6640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c013f4a0 0x7f12c0135480 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f12b8007e50 tx=0x7f12b800d4a0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.533+0000 7f12c6ad6640 1 -- 172.21.15.46:0/2324683154 shutdown_connections 2023-10-30T13:42:51.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.533+0000 7f12c6ad6640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12a40664d0 0x7f12a4068990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.533+0000 7f12c6ad6640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12c013f4a0 0x7f12c0135480 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.533+0000 7f12c6ad6640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12c00c6f60 0x7f12c0158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.533+0000 7f12c6ad6640 1 --2- 172.21.15.46:0/2324683154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12c00c6b80 0x7f12c01f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.533+0000 7f12c6ad6640 1 -- 172.21.15.46:0/2324683154 >> 172.21.15.46:0/2324683154 conn(0x7f12c00c7910 msgr2=0x7f12c00b0920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:51.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.533+0000 7f12c6ad6640 1 -- 172.21.15.46:0/2324683154 shutdown_connections 2023-10-30T13:42:51.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.533+0000 7f12c6ad6640 1 -- 172.21.15.46:0/2324683154 wait complete. 2023-10-30T13:42:51.557 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-30T13:42:51.557 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:42:51.558 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:51.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.713+0000 7fb2a9805640 1 Processor -- start 2023-10-30T13:42:51.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.713+0000 7fb2a9805640 1 -- start start 2023-10-30T13:42:51.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.713+0000 7fb2a9805640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2a40c8cf0 0x7fb2a40c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.713+0000 7fb2a9805640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb2a40c90d0 0x7fb2a40ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.713+0000 7fb2a9805640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a415ce30 con 0x7fb2a40cef90 2023-10-30T13:42:51.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.713+0000 7fb2a9805640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a4159880 con 0x7fb2a40c8cf0 2023-10-30T13:42:51.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.713+0000 7fb2a9805640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a4159ae0 con 0x7fb2a40c90d0 2023-10-30T13:42:51.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.713+0000 7fb2a2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2a40c8cf0 0x7fb2a40c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.713+0000 7fb2a37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb2a40cef90 0x7fb2a415d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33966/0 (socket says 172.21.15.46:33966) 2023-10-30T13:42:51.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.713+0000 7fb2a37fe640 1 -- 172.21.15.46:0/1562633865 learned_addr learned my addr 172.21.15.46:0/1562633865 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:42:51.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.713+0000 7fb2a27fc640 1 --2- 172.21.15.46:0/1562633865 >> v2:172.21.15.154:3300/0 conn(0x7fb2a40c90d0 0x7fb2a40ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.717+0000 7fb2a2ffd640 1 -- 172.21.15.46:0/1562633865 >> v2:172.21.15.154:3300/0 conn(0x7fb2a40c90d0 msgr2=0x7fb2a40ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:51.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.717+0000 7fb2a2ffd640 1 --2- 172.21.15.46:0/1562633865 >> v2:172.21.15.154:3300/0 conn(0x7fb2a40c90d0 0x7fb2a40ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.717+0000 7fb2a2ffd640 1 -- 172.21.15.46:0/1562633865 >> v1:172.21.15.46:6789/0 conn(0x7fb2a40cef90 legacy=0x7fb2a415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.717+0000 7fb2a2ffd640 1 -- 172.21.15.46:0/1562633865 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a4159dc0 con 0x7fb2a40c8cf0 2023-10-30T13:42:51.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.717+0000 7fb2a2ffd640 1 --2- 172.21.15.46:0/1562633865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40c8cf0 0x7fb2a40c9e00 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb2a406cb60 tx=0x7fb29401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b03a501ac96f5e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:51.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.717+0000 7fb2a1ffb640 1 -- 172.21.15.46:0/1562633865 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2940a7070 con 0x7fb2a40c8cf0 2023-10-30T13:42:51.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.717+0000 7fb2a1ffb640 1 -- 172.21.15.46:0/1562633865 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2940ab430 con 0x7fb2a40c8cf0 2023-10-30T13:42:51.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.717+0000 7fb2a1ffb640 1 -- 172.21.15.46:0/1562633865 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2940ab7c0 con 0x7fb2a40c8cf0 2023-10-30T13:42:51.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.721+0000 7fb2a9805640 1 -- 172.21.15.46:0/1562633865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40c8cf0 msgr2=0x7fb2a40c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.721+0000 7fb2a9805640 1 --2- 172.21.15.46:0/1562633865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40c8cf0 0x7fb2a40c9e00 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb2a406cb60 tx=0x7fb29401c430 comp rx=0 tx=0).stop 2023-10-30T13:42:51.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.721+0000 7fb2a9805640 1 -- 172.21.15.46:0/1562633865 shutdown_connections 2023-10-30T13:42:51.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.721+0000 7fb2a9805640 1 --2- 172.21.15.46:0/1562633865 >> v2:172.21.15.154:3300/0 conn(0x7fb2a40c90d0 0x7fb2a40ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.721+0000 7fb2a9805640 1 --2- 172.21.15.46:0/1562633865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40c8cf0 0x7fb2a40c9e00 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.721+0000 7fb2a9805640 1 -- 172.21.15.46:0/1562633865 >> 172.21.15.46:0/1562633865 conn(0x7fb2a406be40 msgr2=0x7fb2a406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:51.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.721+0000 7fb2a9805640 1 -- 172.21.15.46:0/1562633865 shutdown_connections 2023-10-30T13:42:51.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.721+0000 7fb2a9805640 1 -- 172.21.15.46:0/1562633865 wait complete. 2023-10-30T13:42:51.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.721+0000 7fb2a9805640 1 Processor -- start 2023-10-30T13:42:51.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.721+0000 7fb2a9805640 1 -- start start 2023-10-30T13:42:51.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a9805640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2a40c8cf0 0x7fb2a414cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a9805640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40c90d0 0x7fb2a41462e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a9805640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2a40cef90 0x7fb2a4146820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a9805640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a415b3d0 con 0x7fb2a40c90d0 2023-10-30T13:42:51.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a9805640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a415c690 con 0x7fb2a40cef90 2023-10-30T13:42:51.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a9805640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a40c8120 con 0x7fb2a40c8cf0 2023-10-30T13:42:51.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40c90d0 0x7fb2a41462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40c90d0 0x7fb2a41462e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43902/0 (socket says 172.21.15.46:43902) 2023-10-30T13:42:51.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a27fc640 1 -- 172.21.15.46:0/286413784 learned_addr learned my addr 172.21.15.46:0/286413784 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:42:51.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a37fe640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2a40cef90 0x7fb2a4146820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a2ffd640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2a40c8cf0 0x7fb2a414cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a27fc640 1 -- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2a40c8cf0 msgr2=0x7fb2a414cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a27fc640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2a40c8cf0 0x7fb2a414cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a27fc640 1 -- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2a40cef90 msgr2=0x7fb2a4146820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a27fc640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2a40cef90 0x7fb2a4146820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a27fc640 1 -- 172.21.15.46:0/286413784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb294009420 con 0x7fb2a40c90d0 2023-10-30T13:42:51.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a27fc640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40c90d0 0x7fb2a41462e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb29800d5f0 tx=0x7fb29800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:51.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb28bfff640 1 -- 172.21.15.46:0/286413784 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2980125f0 con 0x7fb2a40c90d0 2023-10-30T13:42:51.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb28bfff640 1 -- 172.21.15.46:0/286413784 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb298012750 con 0x7fb2a40c90d0 2023-10-30T13:42:51.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a9805640 1 -- 172.21.15.46:0/286413784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a415aaa0 con 0x7fb2a40c90d0 2023-10-30T13:42:51.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb28bfff640 1 -- 172.21.15.46:0/286413784 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb298012ae0 con 0x7fb2a40c90d0 2023-10-30T13:42:51.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a9805640 1 -- 172.21.15.46:0/286413784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a415b060 con 0x7fb2a40c90d0 2023-10-30T13:42:51.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.725+0000 7fb2a9805640 1 -- 172.21.15.46:0/286413784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a41f5b80 con 0x7fb2a40c90d0 2023-10-30T13:42:51.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.729+0000 7fb28bfff640 1 -- 172.21.15.46:0/286413784 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb298012c40 con 0x7fb2a40c90d0 2023-10-30T13:42:51.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.729+0000 7fb28bfff640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb2900664a0 0x7fb290068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.729+0000 7fb28bfff640 1 -- 172.21.15.46:0/286413784 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fb298085080 con 0x7fb2a40c90d0 2023-10-30T13:42:51.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.729+0000 7fb2a2ffd640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb2900664a0 0x7fb290068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.729+0000 7fb2a2ffd640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb2900664a0 0x7fb290068960 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb2940af000 tx=0x7fb2940a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:51.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.733+0000 7fb28bfff640 1 -- 172.21.15.46:0/286413784 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb298052090 con 0x7fb2a40c90d0 2023-10-30T13:42:51.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.917+0000 7fb2a9805640 1 -- 172.21.15.46:0/286413784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2a41f5b80 con 0x7fb2a40c90d0 2023-10-30T13:42:51.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb28bfff640 1 -- 172.21.15.46:0/286413784 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fb2a41f5b80 con 0x7fb2a40c90d0 2023-10-30T13:42:51.925 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:42:51.925 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:50.198737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6036},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:51.926 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 100 2023-10-30T13:42:51.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb2a9805640 1 -- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb2900664a0 msgr2=0x7fb290068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb2a9805640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb2900664a0 0x7fb290068960 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb2940af000 tx=0x7fb2940a7000 comp rx=0 tx=0).stop 2023-10-30T13:42:51.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb2a9805640 1 -- 172.21.15.46:0/286413784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40c90d0 msgr2=0x7fb2a41462e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb2a9805640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40c90d0 0x7fb2a41462e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb29800d5f0 tx=0x7fb29800dab0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb2a9805640 1 -- 172.21.15.46:0/286413784 shutdown_connections 2023-10-30T13:42:51.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb2a9805640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb2900664a0 0x7fb290068960 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb2a9805640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2a40cef90 0x7fb2a4146820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb2a9805640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a40c90d0 0x7fb2a41462e0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb2a9805640 1 --2- 172.21.15.46:0/286413784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2a40c8cf0 0x7fb2a414cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb2a9805640 1 -- 172.21.15.46:0/286413784 >> 172.21.15.46:0/286413784 conn(0x7fb2a406be40 msgr2=0x7fb2a4153760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:51.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb2a9805640 1 -- 172.21.15.46:0/286413784 shutdown_connections 2023-10-30T13:42:51.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:42:51.921+0000 7fb2a9805640 1 -- 172.21.15.46:0/286413784 wait complete. 2023-10-30T13:42:51.946 INFO:tasks.mds_thrash.fs.[cephfs]:cluster has 2 actives (max_mds is 2), no MDS can replace rank 2 2023-10-30T13:42:51.946 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.b 2023-10-30T13:43:00.947 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-30T13:43:00.948 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T13:43:00.948 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T13:43:00.951 INFO:tasks.ceph.mds.b:Started 2023-10-30T13:43:00.951 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:01.028 INFO:tasks.ceph.mds.b.smithi154.stdout:starting mds.b at 2023-10-30T13:43:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc976e46640 1 Processor -- start 2023-10-30T13:43:01.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc976e46640 1 -- start start 2023-10-30T13:43:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc976e46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc970153d70 0x7fc97006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc976e46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97006afa0 con 0x7fc9700c8490 2023-10-30T13:43:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc976e46640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc970154150 con 0x7fc97006e470 2023-10-30T13:43:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc976e46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97006d780 con 0x7fc970153d70 2023-10-30T13:43:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc974be2640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc970153d70 0x7fc97006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc9753e3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc9700c8490 0x7fc9700c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59698/0 (socket says 172.21.15.46:59698) 2023-10-30T13:43:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc9753e3640 1 -- 172.21.15.46:0/893629742 learned_addr learned my addr 172.21.15.46:0/893629742 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:43:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc974be2640 1 -- 172.21.15.46:0/893629742 >> v1:172.21.15.154:6789/0 conn(0x7fc97006e470 legacy=0x7fc9700cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc96f7fe640 1 -- 172.21.15.46:0/893629742 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942398965 0 0) 0x7fc970154150 con 0x7fc97006e470 2023-10-30T13:43:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc974be2640 1 -- 172.21.15.46:0/893629742 >> v1:172.21.15.46:6789/0 conn(0x7fc9700c8490 legacy=0x7fc9700c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.093+0000 7fc974be2640 1 -- 172.21.15.46:0/893629742 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97006e030 con 0x7fc970153d70 2023-10-30T13:43:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.097+0000 7fc974be2640 1 --2- 172.21.15.46:0/893629742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc970153d70 0x7fc97006ca80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc9640097f0 tx=0x7fc96401c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aca8d673ce8e17d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.097+0000 7fc96f7fe640 1 -- 172.21.15.46:0/893629742 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9640ab600 con 0x7fc970153d70 2023-10-30T13:43:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.097+0000 7fc96f7fe640 1 -- 172.21.15.46:0/893629742 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9640a7050 con 0x7fc970153d70 2023-10-30T13:43:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.097+0000 7fc96f7fe640 1 -- 172.21.15.46:0/893629742 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9640ab980 con 0x7fc970153d70 2023-10-30T13:43:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.097+0000 7fc976e46640 1 -- 172.21.15.46:0/893629742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc970153d70 msgr2=0x7fc97006ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.097+0000 7fc976e46640 1 --2- 172.21.15.46:0/893629742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc970153d70 0x7fc97006ca80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc9640097f0 tx=0x7fc96401c350 comp rx=0 tx=0).stop 2023-10-30T13:43:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.097+0000 7fc976e46640 1 -- 172.21.15.46:0/893629742 shutdown_connections 2023-10-30T13:43:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.097+0000 7fc976e46640 1 --2- 172.21.15.46:0/893629742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc970153d70 0x7fc97006ca80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.097+0000 7fc976e46640 1 -- 172.21.15.46:0/893629742 >> 172.21.15.46:0/893629742 conn(0x7fc970071ea0 msgr2=0x7fc970065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.097+0000 7fc976e46640 1 -- 172.21.15.46:0/893629742 shutdown_connections 2023-10-30T13:43:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.097+0000 7fc976e46640 1 -- 172.21.15.46:0/893629742 wait complete. 2023-10-30T13:43:01.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc976e46640 1 Processor -- start 2023-10-30T13:43:01.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc976e46640 1 -- start start 2023-10-30T13:43:01.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc976e46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc97006e470 0x7fc970156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:01.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc976e46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9700c8490 0x7fc9701575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:01.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc976e46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc970153d70 0x7fc970149740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:01.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc976e46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc970159270 con 0x7fc9700c8490 2023-10-30T13:43:01.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc976e46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9701ef430 con 0x7fc970153d70 2023-10-30T13:43:01.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc976e46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9701f0400 con 0x7fc97006e470 2023-10-30T13:43:01.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc96ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9700c8490 0x7fc9701575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:01.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc96ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9700c8490 0x7fc9701575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45364/0 (socket says 172.21.15.46:45364) 2023-10-30T13:43:01.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc96ffff640 1 -- 172.21.15.46:0/3469826284 learned_addr learned my addr 172.21.15.46:0/3469826284 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:43:01.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc974be2640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc97006e470 0x7fc970156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:01.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc9753e3640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc970153d70 0x7fc970149740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:01.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc96ffff640 1 -- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc97006e470 msgr2=0x7fc970156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc96ffff640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc97006e470 0x7fc970156760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc96ffff640 1 -- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc970153d70 msgr2=0x7fc970149740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc96ffff640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc970153d70 0x7fc970149740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc96ffff640 1 -- 172.21.15.46:0/3469826284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc964009420 con 0x7fc9700c8490 2023-10-30T13:43:01.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc974be2640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc97006e470 0x7fc970156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:43:01.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc96ffff640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9700c8490 0x7fc9701575b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fc95c001ec0 tx=0x7fc95c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:01.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc96d7fa640 1 -- 172.21.15.46:0/3469826284 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c0054e0 con 0x7fc9700c8490 2023-10-30T13:43:01.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc96d7fa640 1 -- 172.21.15.46:0/3469826284 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95c003600 con 0x7fc9700c8490 2023-10-30T13:43:01.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc96d7fa640 1 -- 172.21.15.46:0/3469826284 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c0039b0 con 0x7fc9700c8490 2023-10-30T13:43:01.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc976e46640 1 -- 172.21.15.46:0/3469826284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9701eff70 con 0x7fc9700c8490 2023-10-30T13:43:01.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.101+0000 7fc976e46640 1 -- 172.21.15.46:0/3469826284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9701ef640 con 0x7fc9700c8490 2023-10-30T13:43:01.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.105+0000 7fc976e46640 1 -- 172.21.15.46:0/3469826284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97004e170 con 0x7fc9700c8490 2023-10-30T13:43:01.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.105+0000 7fc96d7fa640 1 -- 172.21.15.46:0/3469826284 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc95c003b10 con 0x7fc9700c8490 2023-10-30T13:43:01.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.105+0000 7fc96d7fa640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9540664a0 0x7fc954068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:01.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.105+0000 7fc96d7fa640 1 -- 172.21.15.46:0/3469826284 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fc95c0838b0 con 0x7fc9700c8490 2023-10-30T13:43:01.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.105+0000 7fc974be2640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9540664a0 0x7fc954068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:01.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.105+0000 7fc974be2640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9540664a0 0x7fc954068960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc9640ad000 tx=0x7fc9640a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:01.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.109+0000 7fc96d7fa640 1 -- 172.21.15.46:0/3469826284 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc95c0508c0 con 0x7fc9700c8490 2023-10-30T13:43:01.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.297+0000 7fc976e46640 1 -- 172.21.15.46:0/3469826284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc97014e1c0 con 0x7fc9700c8490 2023-10-30T13:43:01.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.297+0000 7fc96d7fa640 1 -- 172.21.15.46:0/3469826284 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fc95c017070 con 0x7fc9700c8490 2023-10-30T13:43:01.304 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:43:01.304 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:42:57.183163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6036},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:01.304 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 102 2023-10-30T13:43:01.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.301+0000 7fc976e46640 1 -- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9540664a0 msgr2=0x7fc954068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.301+0000 7fc976e46640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9540664a0 0x7fc954068960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc9640ad000 tx=0x7fc9640a6000 comp rx=0 tx=0).stop 2023-10-30T13:43:01.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.301+0000 7fc976e46640 1 -- 172.21.15.46:0/3469826284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9700c8490 msgr2=0x7fc9701575b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.301+0000 7fc976e46640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9700c8490 0x7fc9701575b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fc95c001ec0 tx=0x7fc95c00c8e0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.301+0000 7fc976e46640 1 -- 172.21.15.46:0/3469826284 shutdown_connections 2023-10-30T13:43:01.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.301+0000 7fc976e46640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9540664a0 0x7fc954068960 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.301+0000 7fc976e46640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc970153d70 0x7fc970149740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.301+0000 7fc976e46640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9700c8490 0x7fc9701575b0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.301+0000 7fc976e46640 1 --2- 172.21.15.46:0/3469826284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc97006e470 0x7fc970156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.301+0000 7fc976e46640 1 -- 172.21.15.46:0/3469826284 >> 172.21.15.46:0/3469826284 conn(0x7fc970071ea0 msgr2=0x7fc9700c7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:01.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.301+0000 7fc976e46640 1 -- 172.21.15.46:0/3469826284 shutdown_connections 2023-10-30T13:43:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:43:01.301+0000 7fc976e46640 1 -- 172.21.15.46:0/3469826284 wait complete. 2023-10-30T13:43:01.325 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-30T13:43:01.325 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 82 secs before thrashing 2023-10-30T13:43:18.815 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:43:18.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:43:48.555 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 0% (98/12861) Updating files: 1% (129/12861) Updating files: 1% (158/12861) Updating files: 1% (234/12861) Updating files: 2% (258/12861) Updating files: 2% (297/12861) Updating files: 2% (347/12861) Updating files: 3% (386/12861) Updating files: 3% (444/12861) Updating files: 3% (490/12861) Updating files: 4% (515/12861) Updating files: 4% (600/12861) Updating files: 4% (627/12861) Updating files: 5% (644/12861) Updating files: 5% (670/12861) Updating files: 5% (738/12861) Updating files: 5% (766/12861) Updating files: 6% (772/12861) Updating files: 6% (802/12861) Updating files: 6% (824/12861) Updating files: 7% (901/12861) Updating files: 7% (912/12861) Updating files: 7% (935/12861) Updating files: 7% (958/12861) Updating files: 7% (970/12861) Updating files: 7% (980/12861) Updating files: 7% (1005/12861) Updating files: 8% (1029/12861) Updating files: 8% (1055/12861) Updating files: 8% (1097/12861) Updating files: 8% (1144/12861) Updating files: 8% (1149/12861) Updating files: 9% (1158/12861) Updating files: 9% (1165/12861) Updating files: 9% (1167/12861) Updating files: 10% (1287/12861) Updating files: 10% (1307/12861) Updating files: 10% (1355/12861) Updating files: 10% (1414/12861) Updating files: 11% (1415/12861) Updating files: 11% (1509/12861) Updating files: 11% (1522/12861) Updating files: 12% (1544/12861) Updating files: 12% (1583/12861) Updating files: 12% (1601/12861) Updating files: 12% (1613/12861) Updating files: 12% (1636/12861) Updating files: 12% (1663/12861) Updating files: 13% (1672/12861) Updating files: 13% (1750/12861) Updating files: 14% (1801/12861) Updating files: 14% (1872/12861) Updating files: 14% (1880/12861) Updating files: 15% (1930/12861) Updating files: 15% (1964/12861) Updating files: 15% (2008/12861) Updating files: 16% (2058/12861) Updating files: 16% (2061/12861) Updating files: 16% (2128/12861) Updating files: 16% (2170/12861) Updating files: 16% (2172/12861) Updating files: 17% (2187/12861) Updating files: 17% (2235/12861) Updating files: 17% (2271/12861) Updating files: 17% (2277/12861) Updating files: 17% (2298/12861) Updating files: 18% (2315/12861) Updating files: 18% (2348/12861) Updating files: 19% (2444/12861) Updating files: 19% (2452/12861) Updating files: 19% (2457/12861) Updating files: 19% (2480/12861) Updating files: 19% (2504/12861) Updating files: 19% (2505/12861) Updating files: 19% (2514/12861) Updating files: 20% (2573/12861) Updating files: 20% (2575/12861) Updating files: 20% (2601/12861) Updating files: 20% (2670/12861) Updating files: 20% (2682/12861) Updating files: 21% (2701/12861) Updating files: 21% (2720/12861) Updating files: 21% (2761/12861) Updating files: 22% (2830/12861) Updating files: 22% (2861/12861) Updating files: 23% (2959/12861) Updating files: 23% (3004/12861) Updating files: 23% (3084/12861) Updating files: 24% (3087/12861) Updating files: 25% (3216/12861) Updating files: 25% (3244/12861) Updating files: 25% (3283/12861) Updating files: 26% (3344/12861) Updating files: 26% (3347/12861) Updating files: 27% (3473/12861) Updating files: 27% (3475/12861) Updating files: 27% (3500/12861) Updating files: 27% (3513/12861) Updating files: 27% (3552/12861) Updating files: 27% (3562/12861) Updating files: 28% (3602/12861) Updating files: 28% (3641/12861) Updating files: 28% (3726/12861) Updating files: 29% (3730/12861) Updating files: 29% (3753/12861) Updating files: 29% (3824/12861) Updating files: 30% (3859/12861) Updating files: 30% (3886/12861) Updating files: 30% (3959/12861) Updating files: 30% (3980/12861) Updating files: 31% (3987/12861) Updating files: 31% (4003/12861) Updating files: 31% (4013/12861) Updating files: 31% (4031/12861) Updating files: 32% (4116/12861) Updating files: 32% (4168/12861) Updating files: 33% (4245/12861) Updating files: 33% (4259/12861) Updating files: 33% (4301/12861) Updating files: 33% (4342/12861) Updating files: 33% (4356/12861) Updating files: 34% (4373/12861) Updating files: 34% (4385/12861) Updating files: 34% (4451/12861) Updating files: 35% (4502/12861) Updating files: 35% (4603/12861) Updating files: 35% (4620/12861) Updating files: 36% (4630/12861) Updating files: 36% (4669/12861) Updating files: 36% (4752/12861) Updating files: 37% (4759/12861) Updating files: 37% (4805/12861) Updating files: 38% (4888/12861) Updating files: 38% (4940/12861) Updating files: 38% (4987/12861) Updating files: 39% (5016/12861) Updating files: 39% (5099/12861) Updating files: 39% (5120/12861) Updating files: 40% (5145/12861) Updating files: 40% (5164/12861) Updating files: 40% (5170/12861) Updating files: 40% (5206/12861) Updating files: 41% (5274/12861) Updating files: 41% (5335/12861) Updating files: 42% (5402/12861) Updating files: 42% (5463/12861) Updating files: 42% (5479/12861) Updating files: 43% (5531/12861) Updating files: 43% (5610/12861) Updating files: 44% (5659/12861) Updating files: 44% (5695/12861) Updating files: 45% (5788/12861) Updating files: 45% (5895/12861) Updating files: 46% (5917/12861) Updating files: 46% (5969/12861) Updating files: 46% (6004/12861) Updating files: 47% (6045/12861) Updating files: 47% (6061/12861) Updating files: 47% (6083/12861) Updating files: 47% (6167/12861) Updating files: 48% (6174/12861) Updating files: 48% (6183/12861) Updating files: 48% (6241/12861) Updating files: 49% (6302/12861) Updating files: 49% (6319/12861) Updating files: 49% (6333/12861) Updating files: 49% (6416/12861) Updating files: 49% (6426/12861) Updating files: 50% (6431/12861) Updating files: 50% (6481/12861) Updating files: 50% (6501/12861) Updating files: 51% (6560/12861) Updating files: 51% (6561/12861) Updating files: 51% (6594/12861) Updating files: 51% (6681/12861) Updating files: 52% (6688/12861) Updating files: 52% (6713/12861) Updating files: 53% (6817/12861) Updating files: 53% (6845/12861) Updating files: 54% (6945/12861) Updating files: 54% (6986/12861) Updating files: 54% (7040/12861) Updating files: 54% (7053/12861) Updating files: 55% (7074/12861) Updating files: 55% (7178/12861) Updating files: 56% (7203/12861) Updating files: 56% (7240/12861) Updating files: 57% (7331/12861) Updating files: 58% (7460/12861) Updating files: 58% (7475/12861) Updating files: 58% (7573/12861) Updating files: 59% (7588/12861) Updating files: 59% (7629/12861) Updating files: 59% (7676/12861) Updating files: 60% (7717/12861) Updating files: 60% (7785/12861) Updating files: 61% (7846/12861) Updating files: 61% (7918/12861) Updating files: 62% (7974/12861) Updating files: 62% (8034/12861) Updating files: 62% (8090/12861) Updating files: 63% (8103/12861) Updating files: 63% (8122/12861) Updating files: 63% (8132/12861) Updating files: 64% (8232/12861) Updating files: 64% (8245/12861) Updating files: 64% (8279/12861) Updating files: 64% (8303/12861) Updating files: 64% (8336/12861) Updating files: 65% (8360/12861) Updating files: 65% (8386/12861) Updating files: 65% (8447/12861) Updating files: 65% (8484/12861) Updating files: 66% (8489/12861) Updating files: 66% (8490/12861) Updating files: 66% (8570/12861) Updating files: 66% (8587/12861) Updating files: 66% (8595/12861) Updating files: 67% (8617/12861) Updating files: 67% (8680/12861) Updating files: 68% (8746/12861) Updating files: 68% (8798/12861) Updating files: 69% (8875/12861) Updating files: 69% (8923/12861) Updating files: 69% (8963/12861) Updating files: 69% (9000/12861) Updating files: 70% (9003/12861) Updating files: 70% (9090/12861) Updating files: 71% (9132/12861) Updating files: 71% (9203/12861) Updating files: 71% (9223/12861) Updating files: 71% (9242/12861) Updating files: 72% (9260/12861) Updating files: 72% (9275/12861) Updating files: 72% (9287/12861) Updating files: 72% (9289/12861) Updating files: 73% (9389/12861) Updating files: 73% (9455/12861) Updating files: 73% (9480/12861) Updating files: 73% (9514/12861) Updating files: 74% (9518/12861) Updating files: 74% (9581/12861) Updating files: 74% (9613/12861) Updating files: 74% (9639/12861) Updating files: 75% (9646/12861) Updating files: 75% (9655/12861) Updating files: 75% (9749/12861) Updating files: 76% (9775/12861) Updating files: 76% (9859/12861) Updating files: 77% (9903/12861) Updating files: 77% (10030/12861) Updating files: 78% (10032/12861) Updating files: 78% (10052/12861) Updating files: 78% (10110/12861) Updating files: 78% (10124/12861) Updating files: 79% (10161/12861) Updating files: 79% (10190/12861) Updating files: 79% (10208/12861) Updating files: 79% (10283/12861) Updating files: 80% (10289/12861) Updating files: 80% (10407/12861) Updating files: 81% (10418/12861) Updating files: 81% (10533/12861) Updating files: 82% (10547/12861) Updating files: 83% (10675/12861) Updating files: 83% (10776/12861) Updating files: 84% (10804/12861) Updating files: 84% (10847/12861) Updating files: 84% (10892/12861) Updating files: 85% (10932/12861) Updating files: 85% (10983/12861) Updating files: 86% (11061/12861) Updating files: 86% (11083/12861) Updating files: 86% (11135/12861) Updating files: 86% (11175/12861) Updating files: 87% (11190/12861) Updating files: 87% (11192/12861) Updating files: 88% (11318/12861) Updating files: 88% (11409/12861) Updating files: 89% (11447/12861) Updating files: 89% (11461/12861) Updating files: 89% (11545/12861) Updating files: 90% (11575/12861) Updating files: 90% (11670/12861) Updating files: 91% (11704/12861) Updating files: 91% (11779/12861) Updating files: 92% (11833/12861) Updating files: 92% (11875/12861) Updating files: 93% (11961/12861) Updating files: 94% (12090/12861) Updating files: 95% (12218/12861) Updating files: 96% (12347/12861) Updating files: 96% (12348/12861) Updating files: 96% (12354/12861) Updating files: 96% (12377/12861) Updating files: 96% (12422/12861) Updating files: 96% (12435/12861) Updating files: 97% (12476/12861) Updating files: 97% (12548/12861) Updating files: 98% (12604/12861) Updating files: 98% (12676/12861) Updating files: 99% (12733/12861) Updating files: 99% (12775/12861) Updating files: 99% (12842/12861) Updating files: 100% (12861/12861) Updating files: 100% (12861/12861), done. 2023-10-30T13:43:48.872 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:43:48.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:43:48.901 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 204c175eea3 msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2023-10-30T13:43:48.910 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.1 2023-10-30T13:43:49.010 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 2 -eq 48 ] 2023-10-30T13:43:49.010 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.2 2023-10-30T13:43:49.011 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.2 2023-10-30T13:43:49.011 INFO:tasks.workunit.client.0.smithi046.stdout:v0.2 2023-10-30T13:43:49.011 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.2 2023-10-30T13:43:50.345 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 45% (101/222) Updating files: 46% (103/222) Updating files: 47% (105/222) Updating files: 48% (107/222) Updating files: 49% (109/222) Updating files: 50% (111/222) Updating files: 51% (114/222) Updating files: 52% (116/222) Updating files: 53% (118/222) Updating files: 54% (120/222) Updating files: 55% (123/222) Updating files: 56% (125/222) Updating files: 57% (127/222) Updating files: 58% (129/222) Updating files: 59% (131/222) Updating files: 60% (134/222) Updating files: 61% (136/222) Updating files: 62% (138/222) Updating files: 63% (140/222) Updating files: 64% (143/222) Updating files: 65% (145/222) Updating files: 66% (147/222) Updating files: 67% (149/222) Updating files: 68% (151/222) Updating files: 69% (154/222) Updating files: 70% (156/222) Updating files: 71% (158/222) Updating files: 72% (160/222) Updating files: 73% (163/222) Updating files: 74% (165/222) Updating files: 75% (167/222) Updating files: 76% (169/222) Updating files: 77% (171/222) Updating files: 78% (174/222) Updating files: 79% (176/222) Updating files: 80% (178/222) Updating files: 81% (180/222) Updating files: 82% (183/222) Updating files: 83% (185/222) Updating files: 84% (187/222) Updating files: 85% (189/222) Updating files: 86% (191/222) Updating files: 87% (194/222) Updating files: 88% (196/222) Updating files: 89% (198/222) Updating files: 90% (200/222) Updating files: 91% (203/222) Updating files: 92% (205/222) Updating files: 93% (207/222) Updating files: 94% (209/222) Updating files: 95% (211/222) Updating files: 96% (214/222) Updating files: 97% (216/222) Updating files: 98% (218/222) Updating files: 99% (220/222) Updating files: 100% (222/222) Updating files: 100% (222/222), done. 2023-10-30T13:43:50.444 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 2b5f76f48e9 readme 2023-10-30T13:43:50.450 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.2 2023-10-30T13:43:50.608 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 3 -eq 48 ] 2023-10-30T13:43:50.608 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.3 2023-10-30T13:43:50.608 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.3 2023-10-30T13:43:50.609 INFO:tasks.workunit.client.0.smithi046.stdout:v0.3 2023-10-30T13:43:50.609 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.3 2023-10-30T13:43:56.532 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 8% (24/295) Updating files: 9% (27/295) Updating files: 10% (30/295) Updating files: 11% (33/295) Updating files: 12% (36/295) Updating files: 13% (39/295) Updating files: 14% (42/295) Updating files: 15% (45/295) Updating files: 16% (48/295) Updating files: 17% (51/295) Updating files: 18% (54/295) Updating files: 19% (57/295) Updating files: 20% (59/295) Updating files: 21% (62/295) Updating files: 22% (65/295) Updating files: 23% (68/295) Updating files: 24% (71/295) Updating files: 25% (74/295) Updating files: 26% (77/295) Updating files: 27% (80/295) Updating files: 28% (83/295) Updating files: 29% (86/295) Updating files: 30% (89/295) Updating files: 31% (92/295) Updating files: 32% (95/295) Updating files: 33% (98/295) Updating files: 34% (101/295) Updating files: 35% (104/295) Updating files: 36% (107/295) Updating files: 37% (110/295) Updating files: 38% (113/295) Updating files: 39% (116/295) Updating files: 40% (118/295) Updating files: 41% (121/295) Updating files: 42% (124/295) Updating files: 43% (127/295) Updating files: 44% (130/295) Updating files: 45% (133/295) Updating files: 46% (136/295) Updating files: 47% (139/295) Updating files: 48% (142/295) Updating files: 49% (145/295) Updating files: 50% (148/295) Updating files: 51% (151/295) Updating files: 52% (154/295) Updating files: 53% (157/295) Updating files: 54% (160/295) Updating files: 55% (163/295) Updating files: 56% (166/295) Updating files: 57% (169/295) Updating files: 58% (172/295) Updating files: 59% (175/295) Updating files: 60% (177/295) Updating files: 61% (180/295) Updating files: 62% (183/295) Updating files: 63% (186/295) Updating files: 64% (189/295) Updating files: 65% (192/295) Updating files: 66% (195/295) Updating files: 67% (198/295) Updating files: 68% (201/295) Updating files: 69% (204/295) Updating files: 70% (207/295) Updating files: 71% (210/295) Updating files: 72% (213/295) Updating files: 73% (216/295) Updating files: 74% (219/295) Updating files: 75% (222/295) Updating files: 76% (225/295) Updating files: 77% (228/295) Updating files: 78% (231/295) Updating files: 79% (234/295) Updating files: 80% (236/295) Updating files: 81% (239/295) Updating files: 82% (242/295) Updating files: 83% (245/295) Updating files: 84% (248/295) Updating files: 85% (251/295) Updating files: 86% (254/295) Updating files: 87% (257/295) Updating files: 88% (260/295) Updating files: 89% (263/295) Updating files: 90% (266/295) Updating files: 91% (269/295) Updating files: 92% (272/295) Updating files: 93% (275/295) Updating files: 94% (278/295) Updating files: 95% (281/295) Updating files: 96% (284/295) Updating files: 97% (287/295) Updating files: 98% (290/295) Updating files: 99% (293/295) Updating files: 100% (295/295) Updating files: 100% (295/295), done. 2023-10-30T13:43:56.807 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 2e986fe9ddf include ceph.spec in configure.ac 2023-10-30T13:43:56.815 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.3 2023-10-30T13:43:57.890 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 4 -eq 48 ] 2023-10-30T13:43:57.890 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.4 2023-10-30T13:43:57.890 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.4 2023-10-30T13:43:57.890 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.4 2023-10-30T13:43:57.891 INFO:tasks.workunit.client.0.smithi046.stdout:v0.4 2023-10-30T13:44:05.089 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 6% (16/247) Updating files: 7% (18/247) Updating files: 8% (20/247) Updating files: 9% (23/247) Updating files: 10% (25/247) Updating files: 11% (28/247) Updating files: 12% (30/247) Updating files: 13% (33/247) Updating files: 14% (35/247) Updating files: 15% (38/247) Updating files: 16% (40/247) Updating files: 17% (42/247) Updating files: 18% (45/247) Updating files: 19% (47/247) Updating files: 20% (50/247) Updating files: 21% (52/247) Updating files: 22% (55/247) Updating files: 23% (57/247) Updating files: 24% (60/247) Updating files: 25% (62/247) Updating files: 26% (65/247) Updating files: 27% (67/247) Updating files: 28% (70/247) Updating files: 29% (72/247) Updating files: 30% (75/247) Updating files: 31% (77/247) Updating files: 31% (78/247) Updating files: 32% (80/247) Updating files: 33% (82/247) Updating files: 34% (84/247) Updating files: 35% (87/247) Updating files: 36% (89/247) Updating files: 37% (92/247) Updating files: 38% (94/247) Updating files: 39% (97/247) Updating files: 40% (99/247) Updating files: 41% (102/247) Updating files: 42% (104/247) Updating files: 43% (107/247) Updating files: 44% (109/247) Updating files: 45% (112/247) Updating files: 46% (114/247) Updating files: 47% (117/247) Updating files: 48% (119/247) Updating files: 49% (122/247) Updating files: 50% (124/247) Updating files: 51% (126/247) Updating files: 52% (129/247) Updating files: 53% (131/247) Updating files: 54% (134/247) Updating files: 55% (136/247) Updating files: 56% (139/247) Updating files: 57% (141/247) Updating files: 58% (144/247) Updating files: 59% (146/247) Updating files: 60% (149/247) Updating files: 60% (150/247) Updating files: 61% (151/247) Updating files: 62% (154/247) Updating files: 63% (156/247) Updating files: 64% (159/247) Updating files: 65% (161/247) Updating files: 66% (164/247) Updating files: 67% (166/247) Updating files: 68% (168/247) Updating files: 69% (171/247) Updating files: 70% (173/247) Updating files: 71% (176/247) Updating files: 72% (178/247) Updating files: 73% (181/247) Updating files: 74% (183/247) Updating files: 75% (186/247) Updating files: 76% (188/247) Updating files: 76% (189/247) Updating files: 77% (191/247) Updating files: 78% (193/247) Updating files: 79% (196/247) Updating files: 80% (198/247) Updating files: 81% (201/247) Updating files: 82% (203/247) Updating files: 83% (206/247) Updating files: 84% (208/247) Updating files: 85% (210/247) Updating files: 86% (213/247) Updating files: 87% (215/247) Updating files: 88% (218/247) Updating files: 89% (220/247) Updating files: 90% (223/247) Updating files: 91% (225/247) Updating files: 92% (228/247) Updating files: 93% (230/247) Updating files: 94% (233/247) Updating files: 95% (235/247) Updating files: 96% (238/247) Updating files: 97% (240/247) Updating files: 98% (243/247) Updating files: 99% (245/247) Updating files: 100% (247/247) Updating files: 100% (247/247), done. 2023-10-30T13:44:08.437 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 13b1bf7c492 kclient: initializing kaddr (merge fix) 2023-10-30T13:44:08.448 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.4 2023-10-30T13:44:09.408 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 5 -eq 48 ] 2023-10-30T13:44:09.408 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.5 2023-10-30T13:44:09.408 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.5 2023-10-30T13:44:09.409 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.5 2023-10-30T13:44:09.409 INFO:tasks.workunit.client.0.smithi046.stdout:v0.5 2023-10-30T13:44:12.404 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 66% (163/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 70% (173/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2023-10-30T13:44:12.453 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 1dd420957f9 journal: detect size of raw block devices properly 2023-10-30T13:44:12.459 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.5 2023-10-30T13:44:12.612 INFO:tasks.workunit.client.0.smithi046.stdout:v0.6 2023-10-30T13:44:12.612 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 6 -eq 48 ] 2023-10-30T13:44:12.613 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.6 2023-10-30T13:44:12.613 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.6 2023-10-30T13:44:12.613 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.6 2023-10-30T13:44:15.694 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 7% (17/227) Updating files: 8% (19/227) Updating files: 9% (21/227) Updating files: 10% (23/227) Updating files: 11% (25/227) Updating files: 12% (28/227) Updating files: 13% (30/227) Updating files: 14% (32/227) Updating files: 15% (35/227) Updating files: 16% (37/227) Updating files: 17% (39/227) Updating files: 18% (41/227) Updating files: 19% (44/227) Updating files: 20% (46/227) Updating files: 21% (48/227) Updating files: 22% (50/227) Updating files: 23% (53/227) Updating files: 24% (55/227) Updating files: 25% (57/227) Updating files: 26% (60/227) Updating files: 27% (62/227) Updating files: 28% (64/227) Updating files: 29% (66/227) Updating files: 30% (69/227) Updating files: 31% (71/227) Updating files: 32% (73/227) Updating files: 33% (75/227) Updating files: 34% (78/227) Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 70% (161/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2023-10-30T13:44:15.759 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at a4c752df713 v0.6 2023-10-30T13:44:15.764 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.6 2023-10-30T13:44:16.027 INFO:tasks.workunit.client.0.smithi046.stdout:v0.7 2023-10-30T13:44:16.027 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 7 -eq 48 ] 2023-10-30T13:44:16.028 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.7 2023-10-30T13:44:16.028 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.7 2023-10-30T13:44:16.028 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.7 2023-10-30T13:44:18.977 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:44:18.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:44:23.326 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:23.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f76419d3640 1 Processor -- start 2023-10-30T13:44:23.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f76419d3640 1 -- start start 2023-10-30T13:44:23.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f76419d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f763c0cf160 0x7f763c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f76419d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f763c0cb800 con 0x7f763c15d6f0 2023-10-30T13:44:23.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f76419d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f763c0cb330 con 0x7f763c1538a0 2023-10-30T13:44:23.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f76419d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f763c0cad70 con 0x7f763c0cf160 2023-10-30T13:44:23.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f7633fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f763c15d6f0 0x7f763c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57076/0 (socket says 172.21.15.46:57076) 2023-10-30T13:44:23.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f7633fff640 1 -- 172.21.15.46:0/133373026 learned_addr learned my addr 172.21.15.46:0/133373026 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:44:23.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f763b7fe640 1 --2- 172.21.15.46:0/133373026 >> v2:172.21.15.154:3300/0 conn(0x7f763c0cf160 0x7f763c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655043250 0 0) 0x7f763c0cb800 con 0x7f763c15d6f0 2023-10-30T13:44:23.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7628003530 con 0x7f763c15d6f0 2023-10-30T13:44:23.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3529139250 0 0) 0x7f7628003530 con 0x7f763c15d6f0 2023-10-30T13:44:23.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 >> v2:172.21.15.154:3300/0 conn(0x7f763c0cf160 msgr2=0x7f763c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:23.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f763a7fc640 1 --2- 172.21.15.46:0/133373026 >> v2:172.21.15.154:3300/0 conn(0x7f763c0cf160 0x7f763c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 >> v1:172.21.15.154:6789/0 conn(0x7f763c1538a0 legacy=0x7f763c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763c0cb050 con 0x7f763c15d6f0 2023-10-30T13:44:23.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f7624001680 con 0x7f763c15d6f0 2023-10-30T13:44:23.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 >> v1:172.21.15.46:6789/0 conn(0x7f763c15d6f0 legacy=0x7f763c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.472+0000 7f763a7fc640 1 --2- 172.21.15.46:0/133373026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7628003980 0x7f7628003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763a7fc640 1 --2- 172.21.15.46:0/133373026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76280047d0 0x7f7628004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763a7fc640 1 --2- 172.21.15.46:0/133373026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 0x7f762800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763c0cad70 con 0x7f7628005190 2023-10-30T13:44:23.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762800c040 con 0x7f7628003980 2023-10-30T13:44:23.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762800c200 con 0x7f76280047d0 2023-10-30T13:44:23.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763b7fe640 1 --2- 172.21.15.46:0/133373026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 0x7f762800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f7633fff640 1 --2- 172.21.15.46:0/133373026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76280047d0 0x7f7628004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763affd640 1 --2- 172.21.15.46:0/133373026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7628003980 0x7f7628003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763b7fe640 1 -- 172.21.15.46:0/133373026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76280047d0 msgr2=0x7f7628004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:23.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763b7fe640 1 --2- 172.21.15.46:0/133373026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76280047d0 0x7f7628004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763b7fe640 1 -- 172.21.15.46:0/133373026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7628003980 msgr2=0x7f7628003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:23.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763b7fe640 1 --2- 172.21.15.46:0/133373026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7628003980 0x7f7628003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763b7fe640 1 -- 172.21.15.46:0/133373026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7628003530 con 0x7f7628005190 2023-10-30T13:44:23.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763b7fe640 1 -- 172.21.15.46:0/133373026 reap_dead start 2023-10-30T13:44:23.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763b7fe640 1 --2- 172.21.15.46:0/133373026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 0x7f762800b990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f7634009fc0 tx=0x7f763401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0cf09aa0dce9249 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:23.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76340a5cb0 con 0x7f7628005190 2023-10-30T13:44:23.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763400f040 con 0x7f7628005190 2023-10-30T13:44:23.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.476+0000 7f763a7fc640 1 -- 172.21.15.46:0/133373026 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7634003be0 con 0x7f7628005190 2023-10-30T13:44:23.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 -- 172.21.15.46:0/133373026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 msgr2=0x7f762800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 --2- 172.21.15.46:0/133373026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 0x7f762800b990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f7634009fc0 tx=0x7f763401cca0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 -- 172.21.15.46:0/133373026 shutdown_connections 2023-10-30T13:44:23.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 --2- 172.21.15.46:0/133373026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 0x7f762800b990 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 -- 172.21.15.46:0/133373026 >> 172.21.15.46:0/133373026 conn(0x7f763c06bfe0 msgr2=0x7f763c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:23.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 -- 172.21.15.46:0/133373026 shutdown_connections 2023-10-30T13:44:23.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 -- 172.21.15.46:0/133373026 wait complete. 2023-10-30T13:44:23.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 Processor -- start 2023-10-30T13:44:23.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 -- start start 2023-10-30T13:44:23.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 0x7f763c1ef3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f763c14d350 0x7f763c0c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f763c148a00 0x7f763c148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763c1e3120 con 0x7f7628005190 2023-10-30T13:44:23.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763c1e3610 con 0x7f763c14d350 2023-10-30T13:44:23.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f76419d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763c1e38b0 con 0x7f763c148a00 2023-10-30T13:44:23.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f763affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 0x7f763c1ef3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f763affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 0x7f763c1ef3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53246/0 (socket says 172.21.15.46:53246) 2023-10-30T13:44:23.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f763affd640 1 -- 172.21.15.46:0/3804580608 learned_addr learned my addr 172.21.15.46:0/3804580608 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:23.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f7633fff640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f763c14d350 0x7f763c0c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f763b7fe640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f763c148a00 0x7f763c148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f763affd640 1 -- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f763c148a00 msgr2=0x7f763c148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f763affd640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f763c148a00 0x7f763c148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f763affd640 1 -- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f763c14d350 msgr2=0x7f763c0c4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f763affd640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f763c14d350 0x7f763c0c4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.480+0000 7f763affd640 1 -- 172.21.15.46:0/3804580608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763c1e3b90 con 0x7f7628005190 2023-10-30T13:44:23.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.484+0000 7f763affd640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 0x7f763c1ef3c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f762c00e4d0 tx=0x7f762c00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:23.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.484+0000 7f7638ff9640 1 -- 172.21.15.46:0/3804580608 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c0052e0 con 0x7f7628005190 2023-10-30T13:44:23.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.484+0000 7f7638ff9640 1 -- 172.21.15.46:0/3804580608 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762c005440 con 0x7f7628005190 2023-10-30T13:44:23.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.484+0000 7f7638ff9640 1 -- 172.21.15.46:0/3804580608 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c003820 con 0x7f7628005190 2023-10-30T13:44:23.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.484+0000 7f76419d3640 1 -- 172.21.15.46:0/3804580608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f763c1eebc0 con 0x7f7628005190 2023-10-30T13:44:23.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.484+0000 7f76419d3640 1 -- 172.21.15.46:0/3804580608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f763c0c6ff0 con 0x7f7628005190 2023-10-30T13:44:23.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.484+0000 7f76419d3640 1 -- 172.21.15.46:0/3804580608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f763c04e170 con 0x7f7628005190 2023-10-30T13:44:23.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.484+0000 7f7638ff9640 1 -- 172.21.15.46:0/3804580608 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f762c003980 con 0x7f7628005190 2023-10-30T13:44:23.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.484+0000 7f7638ff9640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f761c066520 0x7f761c0689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.484+0000 7f7638ff9640 1 -- 172.21.15.46:0/3804580608 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f762c083d20 con 0x7f7628005190 2023-10-30T13:44:23.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.484+0000 7f7633fff640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f761c066520 0x7f761c0689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.488+0000 7f7633fff640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f761c066520 0x7f761c0689e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7624001060 tx=0x7f7624009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:23.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.488+0000 7f7638ff9640 1 -- 172.21.15.46:0/3804580608 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f762c050d00 con 0x7f7628005190 2023-10-30T13:44:23.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.680+0000 7f76419d3640 1 -- 172.21.15.46:0/3804580608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f763c0b1dd0 con 0x7f7628005190 2023-10-30T13:44:23.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.680+0000 7f7638ff9640 1 -- 172.21.15.46:0/3804580608 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f762c054320 con 0x7f7628005190 2023-10-30T13:44:23.688 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:23.688 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:43:29.173143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6036},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:23.689 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 103 2023-10-30T13:44:23.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.684+0000 7f76419d3640 1 -- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f761c066520 msgr2=0x7f761c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.684+0000 7f76419d3640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f761c066520 0x7f761c0689e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7624001060 tx=0x7f7624009040 comp rx=0 tx=0).stop 2023-10-30T13:44:23.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.684+0000 7f76419d3640 1 -- 172.21.15.46:0/3804580608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 msgr2=0x7f763c1ef3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.684+0000 7f76419d3640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 0x7f763c1ef3c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f762c00e4d0 tx=0x7f762c00e990 comp rx=0 tx=0).stop 2023-10-30T13:44:23.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.684+0000 7f76419d3640 1 -- 172.21.15.46:0/3804580608 shutdown_connections 2023-10-30T13:44:23.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.684+0000 7f76419d3640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f761c066520 0x7f761c0689e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.684+0000 7f76419d3640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f763c148a00 0x7f763c148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.684+0000 7f76419d3640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f763c14d350 0x7f763c0c4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.684+0000 7f76419d3640 1 --2- 172.21.15.46:0/3804580608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7628005190 0x7f763c1ef3c0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.684+0000 7f76419d3640 1 -- 172.21.15.46:0/3804580608 >> 172.21.15.46:0/3804580608 conn(0x7f763c06bfe0 msgr2=0x7f763c073690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:23.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.684+0000 7f76419d3640 1 -- 172.21.15.46:0/3804580608 shutdown_connections 2023-10-30T13:44:23.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.684+0000 7f76419d3640 1 -- 172.21.15.46:0/3804580608 wait complete. 2023-10-30T13:44:23.709 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=1) 2023-10-30T13:44:23.710 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:44:23.710 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:44:23.726 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:44:23.726 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.904+0000 7fac143d3640 1 Processor -- start 2023-10-30T13:44:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.904+0000 7fac143d3640 1 -- start start 2023-10-30T13:44:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac143d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fac0c0b0dc0 0x7fac0c0c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac143d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac0c0c5e70 0x7fac0c0c8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac143d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0c0c74f0 con 0x7fac0c153b10 2023-10-30T13:44:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac143d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac0c0c1f20 con 0x7fac0c0c5e70 2023-10-30T13:44:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac143d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac0c0c77b0 con 0x7fac0c0b0dc0 2023-10-30T13:44:23.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac1216f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fac0c153b10 0x7fac0c15ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57090/0 (socket says 172.21.15.46:57090) 2023-10-30T13:44:23.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac1216f640 1 -- 172.21.15.46:0/143021682 learned_addr learned my addr 172.21.15.46:0/143021682 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:44:23.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac12970640 1 --2- 172.21.15.46:0/143021682 >> v2:172.21.15.46:3300/0 conn(0x7fac0c0c5e70 0x7fac0c0c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac1196e640 1 --2- 172.21.15.46:0/143021682 >> v2:172.21.15.154:3300/0 conn(0x7fac0c0b0dc0 0x7fac0c0c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac1116d640 1 -- 172.21.15.46:0/143021682 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248289206 0 0) 0x7fac0c0c74f0 con 0x7fac0c153b10 2023-10-30T13:44:23.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac12970640 1 -- 172.21.15.46:0/143021682 >> v2:172.21.15.154:3300/0 conn(0x7fac0c0b0dc0 msgr2=0x7fac0c0c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:23.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac12970640 1 --2- 172.21.15.46:0/143021682 >> v2:172.21.15.154:3300/0 conn(0x7fac0c0b0dc0 0x7fac0c0c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac12970640 1 -- 172.21.15.46:0/143021682 >> v1:172.21.15.46:6789/0 conn(0x7fac0c153b10 legacy=0x7fac0c15ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.908+0000 7fac12970640 1 -- 172.21.15.46:0/143021682 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c159b80 con 0x7fac0c0c5e70 2023-10-30T13:44:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac12970640 1 --2- 172.21.15.46:0/143021682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac0c0c5e70 0x7fac0c0c8390 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fac0c06b850 tx=0x7fac0401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b605ddd5a1c950 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac1116d640 1 -- 172.21.15.46:0/143021682 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac040b0440 con 0x7fac0c0c5e70 2023-10-30T13:44:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac1116d640 1 -- 172.21.15.46:0/143021682 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac040b05a0 con 0x7fac0c0c5e70 2023-10-30T13:44:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac1116d640 1 -- 172.21.15.46:0/143021682 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac040b0930 con 0x7fac0c0c5e70 2023-10-30T13:44:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 -- 172.21.15.46:0/143021682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac0c0c5e70 msgr2=0x7fac0c0c8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 --2- 172.21.15.46:0/143021682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac0c0c5e70 0x7fac0c0c8390 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fac0c06b850 tx=0x7fac0401f0f0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 -- 172.21.15.46:0/143021682 shutdown_connections 2023-10-30T13:44:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 --2- 172.21.15.46:0/143021682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac0c0c5e70 0x7fac0c0c8390 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 --2- 172.21.15.46:0/143021682 >> v2:172.21.15.154:3300/0 conn(0x7fac0c0b0dc0 0x7fac0c0c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 -- 172.21.15.46:0/143021682 >> 172.21.15.46:0/143021682 conn(0x7fac0c06d770 msgr2=0x7fac0c06db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 -- 172.21.15.46:0/143021682 shutdown_connections 2023-10-30T13:44:23.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 -- 172.21.15.46:0/143021682 wait complete. 2023-10-30T13:44:23.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 Processor -- start 2023-10-30T13:44:23.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 -- start start 2023-10-30T13:44:23.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac0c0b0dc0 0x7fac0c071930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac0c0c5e70 0x7fac0c1488d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac0c153b10 0x7fac0c1f1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c14d4f0 con 0x7fac0c0c5e70 2023-10-30T13:44:23.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0c71f0 con 0x7fac0c153b10 2023-10-30T13:44:23.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.912+0000 7fac143d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c149500 con 0x7fac0c0b0dc0 2023-10-30T13:44:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac1196e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac0c0c5e70 0x7fac0c1488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac1196e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac0c0c5e70 0x7fac0c1488d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53268/0 (socket says 172.21.15.46:53268) 2023-10-30T13:44:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac1196e640 1 -- 172.21.15.46:0/1138190126 learned_addr learned my addr 172.21.15.46:0/1138190126 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac1216f640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac0c0b0dc0 0x7fac0c071930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac12970640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac0c153b10 0x7fac0c1f1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac1196e640 1 -- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac0c0b0dc0 msgr2=0x7fac0c071930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac1196e640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac0c0b0dc0 0x7fac0c071930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac1196e640 1 -- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac0c153b10 msgr2=0x7fac0c1f1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac1196e640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac0c153b10 0x7fac0c1f1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac1196e640 1 -- 172.21.15.46:0/1138190126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac04009a70 con 0x7fac0c0c5e70 2023-10-30T13:44:23.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac1196e640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac0c0c5e70 0x7fac0c1488d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00d800 tx=0x7fabfc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:23.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac02ffd640 1 -- 172.21.15.46:0/1138190126 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc00b660 con 0x7fac0c0c5e70 2023-10-30T13:44:23.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac02ffd640 1 -- 172.21.15.46:0/1138190126 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc00b7c0 con 0x7fac0c0c5e70 2023-10-30T13:44:23.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac143d3640 1 -- 172.21.15.46:0/1138190126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0c1f31b0 con 0x7fac0c0c5e70 2023-10-30T13:44:23.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac02ffd640 1 -- 172.21.15.46:0/1138190126 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc00bb30 con 0x7fac0c0c5e70 2023-10-30T13:44:23.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac143d3640 1 -- 172.21.15.46:0/1138190126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0c146fd0 con 0x7fac0c0c5e70 2023-10-30T13:44:23.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac143d3640 1 -- 172.21.15.46:0/1138190126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0c074250 con 0x7fac0c0c5e70 2023-10-30T13:44:23.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.916+0000 7fac02ffd640 1 -- 172.21.15.46:0/1138190126 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fabfc00bc90 con 0x7fac0c0c5e70 2023-10-30T13:44:23.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.920+0000 7fac02ffd640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fabf00664a0 0x7fabf0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.920+0000 7fac02ffd640 1 -- 172.21.15.46:0/1138190126 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fabfc084900 con 0x7fac0c0c5e70 2023-10-30T13:44:23.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.920+0000 7fac1216f640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fabf00664a0 0x7fabf0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.920+0000 7fac1216f640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fabf00664a0 0x7fabf0068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fac08000f70 tx=0x7fac08007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:23.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:23.924+0000 7fac02ffd640 1 -- 172.21.15.46:0/1138190126 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fabfc051860 con 0x7fac0c0c5e70 2023-10-30T13:44:24.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.112+0000 7fac143d3640 1 -- 172.21.15.46:0/1138190126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac0c04e170 con 0x7fac0c0c5e70 2023-10-30T13:44:24.120 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:24.120 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:43:29.173143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6036},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:24.120 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 103 2023-10-30T13:44:24.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.112+0000 7fac02ffd640 1 -- 172.21.15.46:0/1138190126 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6520 (secure 0 0 0) 0x7fabfc011340 con 0x7fac0c0c5e70 2023-10-30T13:44:24.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.116+0000 7fac143d3640 1 -- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fabf00664a0 msgr2=0x7fabf0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:24.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.116+0000 7fac143d3640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fabf00664a0 0x7fabf0068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fac08000f70 tx=0x7fac08007c10 comp rx=0 tx=0).stop 2023-10-30T13:44:24.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.116+0000 7fac143d3640 1 -- 172.21.15.46:0/1138190126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac0c0c5e70 msgr2=0x7fac0c1488d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:24.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.116+0000 7fac143d3640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac0c0c5e70 0x7fac0c1488d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00d800 tx=0x7fabfc00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.116+0000 7fac143d3640 1 -- 172.21.15.46:0/1138190126 shutdown_connections 2023-10-30T13:44:24.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.116+0000 7fac143d3640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fabf00664a0 0x7fabf0068960 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.116+0000 7fac143d3640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac0c153b10 0x7fac0c1f1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.116+0000 7fac143d3640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac0c0c5e70 0x7fac0c1488d0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.116+0000 7fac143d3640 1 --2- 172.21.15.46:0/1138190126 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac0c0b0dc0 0x7fac0c071930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.116+0000 7fac143d3640 1 -- 172.21.15.46:0/1138190126 >> 172.21.15.46:0/1138190126 conn(0x7fac0c06d770 msgr2=0x7fac0c0c1110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:24.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.116+0000 7fac143d3640 1 -- 172.21.15.46:0/1138190126 shutdown_connections 2023-10-30T13:44:24.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:24.116+0000 7fac143d3640 1 -- 172.21.15.46:0/1138190126 wait complete. 2023-10-30T13:44:24.328 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:44:26.329 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.476+0000 7f2864e91640 1 Processor -- start 2023-10-30T13:44:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.476+0000 7f2864e91640 1 -- start start 2023-10-30T13:44:26.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f2864e91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600c05d0 0x7f286013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f2864e91640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600c0a50 con 0x7f28600c6b80 2023-10-30T13:44:26.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f2864e91640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286006f6e0 con 0x7f28600c6f60 2023-10-30T13:44:26.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f2864e91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286006f9c0 con 0x7f28600c05d0 2023-10-30T13:44:26.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285e59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f28600c6b80 0x7f28600bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45992/0 (socket says 172.21.15.46:45992) 2023-10-30T13:44:26.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285e59c640 1 -- 172.21.15.46:0/890849892 learned_addr learned my addr 172.21.15.46:0/890849892 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:44:26.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f2857fff640 1 --2- 172.21.15.46:0/890849892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600c05d0 0x7f286013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466126731 0 0) 0x7f28600c0a50 con 0x7f28600c6b80 2023-10-30T13:44:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f284c005740 con 0x7f28600c6b80 2023-10-30T13:44:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441997905 0 0) 0x7f286006f6e0 con 0x7f28600c6f60 2023-10-30T13:44:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28600c0a50 con 0x7f28600c6f60 2023-10-30T13:44:26.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4156096786 0 0) 0x7f284c005740 con 0x7f28600c6b80 2023-10-30T13:44:26.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600c05d0 msgr2=0x7f286013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:26.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 --2- 172.21.15.46:0/890849892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600c05d0 0x7f286013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 >> v1:172.21.15.154:6789/0 conn(0x7f28600c6f60 legacy=0x7f28601566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286006fca0 con 0x7f28600c6b80 2023-10-30T13:44:26.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f2850001680 con 0x7f28600c6b80 2023-10-30T13:44:26.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 >> v1:172.21.15.46:6789/0 conn(0x7f28600c6b80 legacy=0x7f28600bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 --2- 172.21.15.46:0/890849892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f284c005d50 0x7f284c006150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 --2- 172.21.15.46:0/890849892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600c05d0 0x7f284c0069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 --2- 172.21.15.46:0/890849892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c006f20 0x7f284c0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286006f9c0 con 0x7f284c005d50 2023-10-30T13:44:26.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c00dc50 con 0x7f284c006f20 2023-10-30T13:44:26.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284c00de10 con 0x7f28600c05d0 2023-10-30T13:44:26.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285e59c640 1 --2- 172.21.15.46:0/890849892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f284c005d50 0x7f284c006150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f2857fff640 1 --2- 172.21.15.46:0/890849892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600c05d0 0x7f284c0069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285ed9d640 1 --2- 172.21.15.46:0/890849892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c006f20 0x7f284c0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285e59c640 1 -- 172.21.15.46:0/890849892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600c05d0 msgr2=0x7f284c0069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:26.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285e59c640 1 --2- 172.21.15.46:0/890849892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600c05d0 0x7f284c0069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285e59c640 1 -- 172.21.15.46:0/890849892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c006f20 msgr2=0x7f284c0073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:26.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285e59c640 1 --2- 172.21.15.46:0/890849892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c006f20 0x7f284c0073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.480+0000 7f285e59c640 1 -- 172.21.15.46:0/890849892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c005740 con 0x7f284c005d50 2023-10-30T13:44:26.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.484+0000 7f285e59c640 1 --2- 172.21.15.46:0/890849892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f284c005d50 0x7f284c006150 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f285000c160 tx=0x7f285001cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49848731917ebad1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:26.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.484+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28500058e0 con 0x7f284c005d50 2023-10-30T13:44:26.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.484+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2850005a60 con 0x7f284c005d50 2023-10-30T13:44:26.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.484+0000 7f285dd9b640 1 -- 172.21.15.46:0/890849892 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850005e10 con 0x7f284c005d50 2023-10-30T13:44:26.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.484+0000 7f2864e91640 1 -- 172.21.15.46:0/890849892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f284c005d50 msgr2=0x7f284c006150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.484+0000 7f2864e91640 1 --2- 172.21.15.46:0/890849892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f284c005d50 0x7f284c006150 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f285000c160 tx=0x7f285001cbc0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.484+0000 7f285ed9d640 1 -- 172.21.15.46:0/890849892 reap_dead start 2023-10-30T13:44:26.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.484+0000 7f2864e91640 1 -- 172.21.15.46:0/890849892 shutdown_connections 2023-10-30T13:44:26.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.484+0000 7f2864e91640 1 -- 172.21.15.46:0/890849892 >> 172.21.15.46:0/890849892 conn(0x7f28600c7910 msgr2=0x7f28600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:26.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.484+0000 7f2864e91640 1 -- 172.21.15.46:0/890849892 shutdown_connections 2023-10-30T13:44:26.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.484+0000 7f2864e91640 1 -- 172.21.15.46:0/890849892 wait complete. 2023-10-30T13:44:26.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2864e91640 1 Processor -- start 2023-10-30T13:44:26.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2864e91640 1 -- start start 2023-10-30T13:44:26.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2864e91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c005d50 0x7f28600c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2864e91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600bfec0 0x7f28601f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2864e91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28601f2a50 0x7f28601544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2864e91640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28600c37d0 con 0x7f28601f2a50 2023-10-30T13:44:26.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2864e91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28601f6630 con 0x7f284c005d50 2023-10-30T13:44:26.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2864e91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28601f2ed0 con 0x7f28600bfec0 2023-10-30T13:44:26.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f285ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28601f2a50 0x7f28601544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f285ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28601f2a50 0x7f28601544a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51096/0 (socket says 172.21.15.46:51096) 2023-10-30T13:44:26.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f285ed9d640 1 -- 172.21.15.46:0/297294683 learned_addr learned my addr 172.21.15.46:0/297294683 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:26.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f285e59c640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c005d50 0x7f28600c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2857fff640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600bfec0 0x7f28601f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f285ed9d640 1 -- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600bfec0 msgr2=0x7f28601f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f285ed9d640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600bfec0 0x7f28601f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f285ed9d640 1 -- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c005d50 msgr2=0x7f28600c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f285ed9d640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c005d50 0x7f28600c4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f285ed9d640 1 -- 172.21.15.46:0/297294683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28600c1520 con 0x7f28601f2a50 2023-10-30T13:44:26.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f285e59c640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c005d50 0x7f28600c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:26.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f285ed9d640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28601f2a50 0x7f28601544a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f285800e690 tx=0x7f285800eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:26.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2856ffd640 1 -- 172.21.15.46:0/297294683 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858003980 con 0x7f28601f2a50 2023-10-30T13:44:26.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2856ffd640 1 -- 172.21.15.46:0/297294683 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2858003ae0 con 0x7f28601f2a50 2023-10-30T13:44:26.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2864e91640 1 -- 172.21.15.46:0/297294683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2860156a40 con 0x7f28601f2a50 2023-10-30T13:44:26.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2856ffd640 1 -- 172.21.15.46:0/297294683 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858003e90 con 0x7f28601f2a50 2023-10-30T13:44:26.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.488+0000 7f2864e91640 1 -- 172.21.15.46:0/297294683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28601f5d00 con 0x7f28601f2a50 2023-10-30T13:44:26.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.492+0000 7f2864e91640 1 -- 172.21.15.46:0/297294683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286004e170 con 0x7f28601f2a50 2023-10-30T13:44:26.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.492+0000 7f2856ffd640 1 -- 172.21.15.46:0/297294683 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f28580050d0 con 0x7f28601f2a50 2023-10-30T13:44:26.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.492+0000 7f2856ffd640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28400664d0 0x7f2840068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.492+0000 7f2856ffd640 1 -- 172.21.15.46:0/297294683 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f2858084150 con 0x7f28601f2a50 2023-10-30T13:44:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.492+0000 7f285e59c640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28400664d0 0x7f2840068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.492+0000 7f285e59c640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28400664d0 0x7f2840068990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f28500aa000 tx=0x7f2850001a00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:26.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.496+0000 7f2856ffd640 1 -- 172.21.15.46:0/297294683 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2858054b40 con 0x7f28601f2a50 2023-10-30T13:44:26.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.684+0000 7f2864e91640 1 -- 172.21.15.46:0/297294683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f286013c3f0 con 0x7f28601f2a50 2023-10-30T13:44:26.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2856ffd640 1 -- 172.21.15.46:0/297294683 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f28580546d0 con 0x7f28601f2a50 2023-10-30T13:44:26.693 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:26.693 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:43:29.173143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6036},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 103 2023-10-30T13:44:26.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2864e91640 1 -- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28400664d0 msgr2=0x7f2840068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2864e91640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28400664d0 0x7f2840068990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f28500aa000 tx=0x7f2850001a00 comp rx=0 tx=0).stop 2023-10-30T13:44:26.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2864e91640 1 -- 172.21.15.46:0/297294683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28601f2a50 msgr2=0x7f28601544a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2864e91640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28601f2a50 0x7f28601544a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f285800e690 tx=0x7f285800eb50 comp rx=0 tx=0).stop 2023-10-30T13:44:26.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2864e91640 1 -- 172.21.15.46:0/297294683 shutdown_connections 2023-10-30T13:44:26.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2864e91640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28400664d0 0x7f2840068990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2864e91640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28601f2a50 0x7f28601544a0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2864e91640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28600bfec0 0x7f28601f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2864e91640 1 --2- 172.21.15.46:0/297294683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f284c005d50 0x7f28600c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2864e91640 1 -- 172.21.15.46:0/297294683 >> 172.21.15.46:0/297294683 conn(0x7f28600c7910 msgr2=0x7f28600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:26.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2864e91640 1 -- 172.21.15.46:0/297294683 shutdown_connections 2023-10-30T13:44:26.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:26.688+0000 7f2864e91640 1 -- 172.21.15.46:0/297294683 wait complete. 2023-10-30T13:44:26.714 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:44:28.715 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.864+0000 7f00cc365640 1 Processor -- start 2023-10-30T13:44:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.864+0000 7f00cc365640 1 -- start start 2023-10-30T13:44:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.864+0000 7f00cc365640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00c40c05d0 0x7f00c413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.864+0000 7f00cc365640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c40c6f60 0x7f00c413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.864+0000 7f00cc365640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c40c0a50 con 0x7f00c40c6b80 2023-10-30T13:44:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.864+0000 7f00cc365640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00c406f6e0 con 0x7f00c40c05d0 2023-10-30T13:44:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.864+0000 7f00cc365640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c406f9c0 con 0x7f00c40c6f60 2023-10-30T13:44:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.864+0000 7f00c9900640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00c40c05d0 0x7f00c413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.864+0000 7f00ca101640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f00c40c6b80 0x7f00c40bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46004/0 (socket says 172.21.15.46:46004) 2023-10-30T13:44:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.864+0000 7f00ca101640 1 -- 172.21.15.46:0/788914774 learned_addr learned my addr 172.21.15.46:0/788914774 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:44:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.864+0000 7f00ca902640 1 --2- 172.21.15.46:0/788914774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c40c6f60 0x7f00c413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:28.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00c9900640 1 -- 172.21.15.46:0/788914774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c40c6f60 msgr2=0x7f00c413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:28.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00c9900640 1 --2- 172.21.15.46:0/788914774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c40c6f60 0x7f00c413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00c9900640 1 -- 172.21.15.46:0/788914774 >> v1:172.21.15.46:6789/0 conn(0x7f00c40c6b80 legacy=0x7f00c40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:28.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00c90ff640 1 -- 172.21.15.46:0/788914774 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666228953 0 0) 0x7f00c40c0a50 con 0x7f00c40c6b80 2023-10-30T13:44:28.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00c9900640 1 -- 172.21.15.46:0/788914774 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c406fca0 con 0x7f00c40c05d0 2023-10-30T13:44:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00c9900640 1 --2- 172.21.15.46:0/788914774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40c05d0 0x7f00c413ed90 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f00b40097c0 tx=0x7f00b401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0dcd1fe9f70988 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00c90ff640 1 -- 172.21.15.46:0/788914774 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40ad640 con 0x7f00c40c05d0 2023-10-30T13:44:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00c90ff640 1 -- 172.21.15.46:0/788914774 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b40ad7a0 con 0x7f00c40c05d0 2023-10-30T13:44:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00c90ff640 1 -- 172.21.15.46:0/788914774 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40adb30 con 0x7f00c40c05d0 2023-10-30T13:44:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00cc365640 1 -- 172.21.15.46:0/788914774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40c05d0 msgr2=0x7f00c413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00cc365640 1 --2- 172.21.15.46:0/788914774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40c05d0 0x7f00c413ed90 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f00b40097c0 tx=0x7f00b401c480 comp rx=0 tx=0).stop 2023-10-30T13:44:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00cc365640 1 -- 172.21.15.46:0/788914774 shutdown_connections 2023-10-30T13:44:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00cc365640 1 --2- 172.21.15.46:0/788914774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c40c6f60 0x7f00c413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00cc365640 1 --2- 172.21.15.46:0/788914774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40c05d0 0x7f00c413ed90 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.868+0000 7f00cc365640 1 -- 172.21.15.46:0/788914774 >> 172.21.15.46:0/788914774 conn(0x7f00c40c7910 msgr2=0x7f00c40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:28.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00cc365640 1 -- 172.21.15.46:0/788914774 shutdown_connections 2023-10-30T13:44:28.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00cc365640 1 -- 172.21.15.46:0/788914774 wait complete. 2023-10-30T13:44:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00cc365640 1 Processor -- start 2023-10-30T13:44:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00cc365640 1 -- start start 2023-10-30T13:44:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00cc365640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c40c05d0 0x7f00c41379f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00cc365640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00c40c6b80 0x7f00c40c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00cc365640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40c6f60 0x7f00c40afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00cc365640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c41563e0 con 0x7f00c40c6f60 2023-10-30T13:44:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00cc365640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c413dc60 con 0x7f00c40c6b80 2023-10-30T13:44:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00cc365640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c4155b40 con 0x7f00c40c05d0 2023-10-30T13:44:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00ca902640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40c6f60 0x7f00c40afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00ca902640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40c6f60 0x7f00c40afe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51118/0 (socket says 172.21.15.46:51118) 2023-10-30T13:44:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00ca902640 1 -- 172.21.15.46:0/889524905 learned_addr learned my addr 172.21.15.46:0/889524905 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00c9900640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00c40c6b80 0x7f00c40c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:28.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00ca101640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c40c05d0 0x7f00c41379f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:28.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00ca902640 1 -- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c40c05d0 msgr2=0x7f00c41379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:28.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00ca902640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c40c05d0 0x7f00c41379f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00ca902640 1 -- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00c40c6b80 msgr2=0x7f00c40c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:28.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00ca902640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00c40c6b80 0x7f00c40c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00ca902640 1 -- 172.21.15.46:0/889524905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b4009420 con 0x7f00c40c6f60 2023-10-30T13:44:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00ca902640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40c6f60 0x7f00c40afe70 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f00bc007e60 tx=0x7f00bc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00baffd640 1 -- 172.21.15.46:0/889524905 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc014420 con 0x7f00c40c6f60 2023-10-30T13:44:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00baffd640 1 -- 172.21.15.46:0/889524905 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00bc0145a0 con 0x7f00c40c6f60 2023-10-30T13:44:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00baffd640 1 -- 172.21.15.46:0/889524905 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc014950 con 0x7f00c40c6f60 2023-10-30T13:44:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.872+0000 7f00cc365640 1 -- 172.21.15.46:0/889524905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c4155e20 con 0x7f00c40c6f60 2023-10-30T13:44:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.876+0000 7f00cc365640 1 -- 172.21.15.46:0/889524905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c413a260 con 0x7f00c40c6f60 2023-10-30T13:44:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.876+0000 7f00cc365640 1 -- 172.21.15.46:0/889524905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c40afb90 con 0x7f00c40c6f60 2023-10-30T13:44:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.876+0000 7f00baffd640 1 -- 172.21.15.46:0/889524905 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f00bc014ab0 con 0x7f00c40c6f60 2023-10-30T13:44:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.876+0000 7f00baffd640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00a80664a0 0x7f00a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.876+0000 7f00baffd640 1 -- 172.21.15.46:0/889524905 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f00bc084c30 con 0x7f00c40c6f60 2023-10-30T13:44:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.876+0000 7f00ca101640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00a80664a0 0x7f00a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.880+0000 7f00ca101640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00a80664a0 0x7f00a8068960 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f00c0000f70 tx=0x7f00c0007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:28.880+0000 7f00baffd640 1 -- 172.21.15.46:0/889524905 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f00c40afb90 con 0x7f00c40c6f60 2023-10-30T13:44:29.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.072+0000 7f00cc365640 1 -- 172.21.15.46:0/889524905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00c40afb90 con 0x7f00c40c6f60 2023-10-30T13:44:29.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.072+0000 7f00baffd640 1 -- 172.21.15.46:0/889524905 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f00c40afb90 con 0x7f00c40c6f60 2023-10-30T13:44:29.078 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:29.079 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:43:29.173143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6036},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 103 2023-10-30T13:44:29.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.072+0000 7f00cc365640 1 -- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00a80664a0 msgr2=0x7f00a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:29.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.072+0000 7f00cc365640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00a80664a0 0x7f00a8068960 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f00c0000f70 tx=0x7f00c0007b60 comp rx=0 tx=0).stop 2023-10-30T13:44:29.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.072+0000 7f00cc365640 1 -- 172.21.15.46:0/889524905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40c6f60 msgr2=0x7f00c40afe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:29.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.072+0000 7f00cc365640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40c6f60 0x7f00c40afe70 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f00bc007e60 tx=0x7f00bc00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.076+0000 7f00cc365640 1 -- 172.21.15.46:0/889524905 shutdown_connections 2023-10-30T13:44:29.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.076+0000 7f00cc365640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00a80664a0 0x7f00a8068960 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.076+0000 7f00cc365640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40c6f60 0x7f00c40afe70 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.076+0000 7f00cc365640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00c40c6b80 0x7f00c40c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.076+0000 7f00cc365640 1 --2- 172.21.15.46:0/889524905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c40c05d0 0x7f00c41379f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:29.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.076+0000 7f00cc365640 1 -- 172.21.15.46:0/889524905 >> 172.21.15.46:0/889524905 conn(0x7f00c40c7910 msgr2=0x7f00c413ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:29.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.076+0000 7f00cc365640 1 -- 172.21.15.46:0/889524905 shutdown_connections 2023-10-30T13:44:29.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:29.076+0000 7f00cc365640 1 -- 172.21.15.46:0/889524905 wait complete. 2023-10-30T13:44:29.099 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:44:31.101 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:31.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8b5965640 1 Processor -- start 2023-10-30T13:44:31.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8b5965640 1 -- start start 2023-10-30T13:44:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8b5965640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8b006d290 0x7fb8b006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8b5965640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b00c6c10 con 0x7fb8b0069e90 2023-10-30T13:44:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8b5965640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b00c6ef0 con 0x7fb8b015a410 2023-10-30T13:44:31.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8b5965640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8b00c8570 con 0x7fb8b006d290 2023-10-30T13:44:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8af7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8b006d290 0x7fb8b006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8aeffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb8b0069e90 0x7fb8b00c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46012/0 (socket says 172.21.15.46:46012) 2023-10-30T13:44:31.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8aeffd640 1 -- 172.21.15.46:0/2187385552 learned_addr learned my addr 172.21.15.46:0/2187385552 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:44:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8af7fe640 1 -- 172.21.15.46:0/2187385552 >> v1:172.21.15.154:6789/0 conn(0x7fb8b015a410 legacy=0x7fb8b006cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8af7fe640 1 -- 172.21.15.46:0/2187385552 >> v1:172.21.15.46:6789/0 conn(0x7fb8b0069e90 legacy=0x7fb8b00c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8adffb640 1 -- 172.21.15.46:0/2187385552 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906685858 0 0) 0x7fb8b00c6c10 con 0x7fb8b0069e90 2023-10-30T13:44:31.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.252+0000 7fb8af7fe640 1 -- 172.21.15.46:0/2187385552 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b00c71d0 con 0x7fb8b006d290 2023-10-30T13:44:31.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.256+0000 7fb8af7fe640 1 --2- 172.21.15.46:0/2187385552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b006d290 0x7fb8b006a270 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8009d90 tx=0x7fb8a801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d70fd73b45cb3eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:31.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.256+0000 7fb8adffb640 1 -- 172.21.15.46:0/2187385552 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a80a5810 con 0x7fb8b006d290 2023-10-30T13:44:31.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.256+0000 7fb8adffb640 1 -- 172.21.15.46:0/2187385552 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a80a5970 con 0x7fb8b006d290 2023-10-30T13:44:31.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.256+0000 7fb8adffb640 1 -- 172.21.15.46:0/2187385552 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a80a5d00 con 0x7fb8b006d290 2023-10-30T13:44:31.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.256+0000 7fb8b5965640 1 -- 172.21.15.46:0/2187385552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b006d290 msgr2=0x7fb8b006a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:31.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.256+0000 7fb8b5965640 1 --2- 172.21.15.46:0/2187385552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b006d290 0x7fb8b006a270 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8009d90 tx=0x7fb8a801c950 comp rx=0 tx=0).stop 2023-10-30T13:44:31.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.256+0000 7fb8b5965640 1 -- 172.21.15.46:0/2187385552 shutdown_connections 2023-10-30T13:44:31.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.256+0000 7fb8b5965640 1 --2- 172.21.15.46:0/2187385552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b006d290 0x7fb8b006a270 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:31.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.256+0000 7fb8b5965640 1 -- 172.21.15.46:0/2187385552 >> 172.21.15.46:0/2187385552 conn(0x7fb8b006dfd0 msgr2=0x7fb8b006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:31.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.256+0000 7fb8b5965640 1 -- 172.21.15.46:0/2187385552 shutdown_connections 2023-10-30T13:44:31.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.256+0000 7fb8b5965640 1 -- 172.21.15.46:0/2187385552 wait complete. 2023-10-30T13:44:31.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.256+0000 7fb8b5965640 1 Processor -- start 2023-10-30T13:44:31.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8b5965640 1 -- start start 2023-10-30T13:44:31.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8b5965640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b0069e90 0x7fb8b01497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:31.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8b5965640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b006d290 0x7fb8b0145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:31.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8b5965640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b015a410 0x7fb8b014b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:31.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8b5965640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b0071eb0 con 0x7fb8b006d290 2023-10-30T13:44:31.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8b5965640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b0071bd0 con 0x7fb8b015a410 2023-10-30T13:44:31.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8b5965640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b015aaf0 con 0x7fb8b0069e90 2023-10-30T13:44:31.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8ae7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b006d290 0x7fb8b0145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:31.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8ae7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b006d290 0x7fb8b0145bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51134/0 (socket says 172.21.15.46:51134) 2023-10-30T13:44:31.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8ae7fc640 1 -- 172.21.15.46:0/965596219 learned_addr learned my addr 172.21.15.46:0/965596219 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:31.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8aeffd640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b0069e90 0x7fb8b01497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:31.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8af7fe640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b015a410 0x7fb8b014b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:31.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8ae7fc640 1 -- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b0069e90 msgr2=0x7fb8b01497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:31.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8ae7fc640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b0069e90 0x7fb8b01497e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:31.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8ae7fc640 1 -- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b015a410 msgr2=0x7fb8b014b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:31.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8ae7fc640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b015a410 0x7fb8b014b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:31.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8ae7fc640 1 -- 172.21.15.46:0/965596219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a8009a70 con 0x7fb8b006d290 2023-10-30T13:44:31.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8af7fe640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b015a410 0x7fb8b014b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:31.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8ae7fc640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b006d290 0x7fb8b0145bc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400c3c0 tx=0x7fb8a400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:31.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb897fff640 1 -- 172.21.15.46:0/965596219 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a40052d0 con 0x7fb8b006d290 2023-10-30T13:44:31.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb897fff640 1 -- 172.21.15.46:0/965596219 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a4005430 con 0x7fb8b006d290 2023-10-30T13:44:31.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb897fff640 1 -- 172.21.15.46:0/965596219 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4003840 con 0x7fb8b006d290 2023-10-30T13:44:31.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8b5965640 1 -- 172.21.15.46:0/965596219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b015add0 con 0x7fb8b006d290 2023-10-30T13:44:31.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8b5965640 1 -- 172.21.15.46:0/965596219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b015c1f0 con 0x7fb8b006d290 2023-10-30T13:44:31.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.260+0000 7fb8b5965640 1 -- 172.21.15.46:0/965596219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b01f2010 con 0x7fb8b006d290 2023-10-30T13:44:31.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.264+0000 7fb897fff640 1 -- 172.21.15.46:0/965596219 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb8a40039a0 con 0x7fb8b006d290 2023-10-30T13:44:31.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.264+0000 7fb897fff640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb89c0664a0 0x7fb89c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:31.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.264+0000 7fb897fff640 1 -- 172.21.15.46:0/965596219 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fb8a40837a0 con 0x7fb8b006d290 2023-10-30T13:44:31.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.264+0000 7fb8aeffd640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb89c0664a0 0x7fb89c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:31.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.264+0000 7fb8aeffd640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb89c0664a0 0x7fb89c068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0007fd0 tx=0x7fb8a0007ba0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:31.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.268+0000 7fb897fff640 1 -- 172.21.15.46:0/965596219 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8b01f2010 con 0x7fb8b006d290 2023-10-30T13:44:31.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.456+0000 7fb8b5965640 1 -- 172.21.15.46:0/965596219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8b01f2010 con 0x7fb8b006d290 2023-10-30T13:44:31.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.456+0000 7fb897fff640 1 -- 172.21.15.46:0/965596219 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6520 (secure 0 0 0) 0x7fb8b01f2010 con 0x7fb8b006d290 2023-10-30T13:44:31.463 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:31.464 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:43:29.173143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6036},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:31.464 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 103 2023-10-30T13:44:31.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.460+0000 7fb8b5965640 1 -- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb89c0664a0 msgr2=0x7fb89c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:31.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.460+0000 7fb8b5965640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb89c0664a0 0x7fb89c068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0007fd0 tx=0x7fb8a0007ba0 comp rx=0 tx=0).stop 2023-10-30T13:44:31.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.460+0000 7fb8b5965640 1 -- 172.21.15.46:0/965596219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b006d290 msgr2=0x7fb8b0145bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:31.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.460+0000 7fb8b5965640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b006d290 0x7fb8b0145bc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400c3c0 tx=0x7fb8a400c880 comp rx=0 tx=0).stop 2023-10-30T13:44:31.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.460+0000 7fb8b5965640 1 -- 172.21.15.46:0/965596219 shutdown_connections 2023-10-30T13:44:31.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.460+0000 7fb8b5965640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb89c0664a0 0x7fb89c068960 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:31.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.460+0000 7fb8b5965640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b015a410 0x7fb8b014b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:31.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.460+0000 7fb8b5965640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b006d290 0x7fb8b0145bc0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:31.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.460+0000 7fb8b5965640 1 --2- 172.21.15.46:0/965596219 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b0069e90 0x7fb8b01497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:31.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.460+0000 7fb8b5965640 1 -- 172.21.15.46:0/965596219 >> 172.21.15.46:0/965596219 conn(0x7fb8b006dfd0 msgr2=0x7fb8b014f050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:31.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.460+0000 7fb8b5965640 1 -- 172.21.15.46:0/965596219 shutdown_connections 2023-10-30T13:44:31.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:31.460+0000 7fb8b5965640 1 -- 172.21.15.46:0/965596219 wait complete. 2023-10-30T13:44:31.484 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:44:33.486 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:33.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.628+0000 7f97adc46640 1 Processor -- start 2023-10-30T13:44:33.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.628+0000 7f97adc46640 1 -- start start 2023-10-30T13:44:33.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.628+0000 7f97adc46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97a80c05d0 0x7f97a813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.628+0000 7f97adc46640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97a80c6f60 0x7f97a813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.628+0000 7f97adc46640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a80c0a50 con 0x7f97a80c6b80 2023-10-30T13:44:33.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.628+0000 7f97adc46640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a806f6e0 con 0x7f97a80c6f60 2023-10-30T13:44:33.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.628+0000 7f97adc46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a806f9c0 con 0x7f97a80c05d0 2023-10-30T13:44:33.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.632+0000 7f97a7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97a80c6f60 0x7f97a813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.632+0000 7f97a7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97a80c6f60 0x7f97a813f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51140/0 (socket says 172.21.15.46:51140) 2023-10-30T13:44:33.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.632+0000 7f97a7fff640 1 -- 172.21.15.46:0/4269508500 learned_addr learned my addr 172.21.15.46:0/4269508500 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:33.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.632+0000 7f97a6ffd640 1 --2- 172.21.15.46:0/4269508500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97a80c05d0 0x7f97a813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.632+0000 7f97a7fff640 1 -- 172.21.15.46:0/4269508500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97a80c05d0 msgr2=0x7f97a813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:33.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.632+0000 7f97a7fff640 1 --2- 172.21.15.46:0/4269508500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97a80c05d0 0x7f97a813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.632+0000 7f97a7fff640 1 -- 172.21.15.46:0/4269508500 >> v1:172.21.15.154:6789/0 conn(0x7f97a80c6b80 legacy=0x7f97a80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.632+0000 7f97a7fff640 1 -- 172.21.15.46:0/4269508500 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a806fca0 con 0x7f97a80c6f60 2023-10-30T13:44:33.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.632+0000 7f97a7fff640 1 --2- 172.21.15.46:0/4269508500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a80c6f60 0x7f97a813f2d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009e50 tx=0x7f97a001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdac2033e85a6cd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:33.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.632+0000 7f97a67fc640 1 -- 172.21.15.46:0/4269508500 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a00a57f0 con 0x7f97a80c6f60 2023-10-30T13:44:33.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.632+0000 7f97a67fc640 1 -- 172.21.15.46:0/4269508500 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a00a5950 con 0x7f97a80c6f60 2023-10-30T13:44:33.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.632+0000 7f97a67fc640 1 -- 172.21.15.46:0/4269508500 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a00a5ce0 con 0x7f97a80c6f60 2023-10-30T13:44:33.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 -- 172.21.15.46:0/4269508500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a80c6f60 msgr2=0x7f97a813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:33.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 --2- 172.21.15.46:0/4269508500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a80c6f60 0x7f97a813f2d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009e50 tx=0x7f97a001ca10 comp rx=0 tx=0).stop 2023-10-30T13:44:33.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 -- 172.21.15.46:0/4269508500 shutdown_connections 2023-10-30T13:44:33.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 --2- 172.21.15.46:0/4269508500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a80c6f60 0x7f97a813f2d0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 --2- 172.21.15.46:0/4269508500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97a80c05d0 0x7f97a813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 -- 172.21.15.46:0/4269508500 >> 172.21.15.46:0/4269508500 conn(0x7f97a80c7910 msgr2=0x7f97a80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:33.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 -- 172.21.15.46:0/4269508500 shutdown_connections 2023-10-30T13:44:33.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 -- 172.21.15.46:0/4269508500 wait complete. 2023-10-30T13:44:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 Processor -- start 2023-10-30T13:44:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 -- start start 2023-10-30T13:44:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97a80c05d0 0x7f97a813df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a80c6b80 0x7f97a81f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97a80c6f60 0x7f97a81f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a81f5590 con 0x7f97a80c6b80 2023-10-30T13:44:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a8160d90 con 0x7f97a80c05d0 2023-10-30T13:44:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97adc46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a81547c0 con 0x7f97a80c6f60 2023-10-30T13:44:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97a6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a80c6b80 0x7f97a81f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97a6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a80c6b80 0x7f97a81f5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51152/0 (socket says 172.21.15.46:51152) 2023-10-30T13:44:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97a6ffd640 1 -- 172.21.15.46:0/3151852667 learned_addr learned my addr 172.21.15.46:0/3151852667 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97a77fe640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97a80c05d0 0x7f97a813df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97a7fff640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97a80c6f60 0x7f97a81f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97a6ffd640 1 -- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97a80c6f60 msgr2=0x7f97a81f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97a6ffd640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97a80c6f60 0x7f97a81f26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97a6ffd640 1 -- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97a80c05d0 msgr2=0x7f97a813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97a6ffd640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97a80c05d0 0x7f97a813df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.636+0000 7f97a6ffd640 1 -- 172.21.15.46:0/3151852667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a0009b30 con 0x7f97a80c6b80 2023-10-30T13:44:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.640+0000 7f97a6ffd640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a80c6b80 0x7f97a81f5cc0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f979c00d990 tx=0x7f979c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:33.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.640+0000 7f978ffff640 1 -- 172.21.15.46:0/3151852667 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c007b10 con 0x7f97a80c6b80 2023-10-30T13:44:33.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.640+0000 7f978ffff640 1 -- 172.21.15.46:0/3151852667 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979c007c70 con 0x7f97a80c6b80 2023-10-30T13:44:33.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.640+0000 7f978ffff640 1 -- 172.21.15.46:0/3151852667 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c010470 con 0x7f97a80c6b80 2023-10-30T13:44:33.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.640+0000 7f97adc46640 1 -- 172.21.15.46:0/3151852667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a80caa60 con 0x7f97a80c6b80 2023-10-30T13:44:33.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.640+0000 7f97adc46640 1 -- 172.21.15.46:0/3151852667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a8154d60 con 0x7f97a80c6b80 2023-10-30T13:44:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.640+0000 7f97adc46640 1 -- 172.21.15.46:0/3151852667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a8154f80 con 0x7f97a80c6b80 2023-10-30T13:44:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.640+0000 7f978ffff640 1 -- 172.21.15.46:0/3151852667 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f979c0105d0 con 0x7f97a80c6b80 2023-10-30T13:44:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.640+0000 7f978ffff640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97940664a0 0x7f9794068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.640+0000 7f978ffff640 1 -- 172.21.15.46:0/3151852667 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f979c0847b0 con 0x7f97a80c6b80 2023-10-30T13:44:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.644+0000 7f97a77fe640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97940664a0 0x7f9794068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.644+0000 7f97a77fe640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97940664a0 0x7f9794068960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9798000f70 tx=0x7f9798009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.644+0000 7f978ffff640 1 -- 172.21.15.46:0/3151852667 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f979c0517a0 con 0x7f97a80c6b80 2023-10-30T13:44:33.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.836+0000 7f97adc46640 1 -- 172.21.15.46:0/3151852667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97a804e170 con 0x7f97a80c6b80 2023-10-30T13:44:33.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.836+0000 7f978ffff640 1 -- 172.21.15.46:0/3151852667 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f979c054dc0 con 0x7f97a80c6b80 2023-10-30T13:44:33.843 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:33.843 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:43:29.173143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6036},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:33.843 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 103 2023-10-30T13:44:33.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.840+0000 7f97adc46640 1 -- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97940664a0 msgr2=0x7f9794068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:33.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.840+0000 7f97adc46640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97940664a0 0x7f9794068960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9798000f70 tx=0x7f9798009040 comp rx=0 tx=0).stop 2023-10-30T13:44:33.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.840+0000 7f97adc46640 1 -- 172.21.15.46:0/3151852667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a80c6b80 msgr2=0x7f97a81f5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:33.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.840+0000 7f97adc46640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a80c6b80 0x7f97a81f5cc0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f979c00d990 tx=0x7f979c00dd90 comp rx=0 tx=0).stop 2023-10-30T13:44:33.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.840+0000 7f97adc46640 1 -- 172.21.15.46:0/3151852667 shutdown_connections 2023-10-30T13:44:33.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.840+0000 7f97adc46640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97940664a0 0x7f9794068960 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.840+0000 7f97adc46640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97a80c6f60 0x7f97a81f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.840+0000 7f97adc46640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a80c6b80 0x7f97a81f5cc0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.840+0000 7f97adc46640 1 --2- 172.21.15.46:0/3151852667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97a80c05d0 0x7f97a813df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.840+0000 7f97adc46640 1 -- 172.21.15.46:0/3151852667 >> 172.21.15.46:0/3151852667 conn(0x7f97a80c7910 msgr2=0x7f97a81352c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:33.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.840+0000 7f97adc46640 1 -- 172.21.15.46:0/3151852667 shutdown_connections 2023-10-30T13:44:33.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:33.840+0000 7f97adc46640 1 -- 172.21.15.46:0/3151852667 wait complete. 2023-10-30T13:44:33.864 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:44:35.864 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:36.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.012+0000 7f11f6532640 1 Processor -- start 2023-10-30T13:44:36.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.012+0000 7f11f6532640 1 -- start start 2023-10-30T13:44:36.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f6532640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f0157c80 0x7f11f0161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f6532640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f11f006ab20 0x7f11f0070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f6532640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f0161f60 con 0x7f11f006fef0 2023-10-30T13:44:36.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f6532640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f015ef30 con 0x7f11f006ab20 2023-10-30T13:44:36.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f6532640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f0160e50 con 0x7f11f0157c80 2023-10-30T13:44:36.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f5530640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f0157c80 0x7f11f0161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f5d31640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f11f006ab20 0x7f11f0070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f5d31640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f11f006ab20 0x7f11f0070c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:52526/0 (socket says 172.21.15.46:52526) 2023-10-30T13:44:36.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f5d31640 1 -- 172.21.15.46:0/2716271367 learned_addr learned my addr 172.21.15.46:0/2716271367 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:36.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f5530640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f0157c80 0x7f11f0161030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:39204/0 (socket says 172.21.15.46:39204) 2023-10-30T13:44:36.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f4d2f640 1 --1- 172.21.15.46:0/2716271367 >> v1:172.21.15.154:6789/0 conn(0x7f11f006fef0 0x7f11f006f340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:48888/0 (socket says 172.21.15.46:48888) 2023-10-30T13:44:36.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f5530640 1 -- 172.21.15.46:0/2716271367 >> v2:172.21.15.154:3300/0 conn(0x7f11f006ab20 msgr2=0x7f11f0070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:36.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f5530640 1 --2- 172.21.15.46:0/2716271367 >> v2:172.21.15.154:3300/0 conn(0x7f11f006ab20 0x7f11f0070c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f5530640 1 -- 172.21.15.46:0/2716271367 >> v1:172.21.15.154:6789/0 conn(0x7f11f006fef0 legacy=0x7f11f006f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.016+0000 7f11f5530640 1 -- 172.21.15.46:0/2716271367 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f01616c0 con 0x7f11f0157c80 2023-10-30T13:44:36.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11f5530640 1 --2- 172.21.15.46:0/2716271367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f0157c80 0x7f11f0161030 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0097a0 tx=0x7f11ec01ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33729292001fcc46 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:36.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11e7fff640 1 -- 172.21.15.46:0/2716271367 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec0a8950 con 0x7f11f0157c80 2023-10-30T13:44:36.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11e7fff640 1 -- 172.21.15.46:0/2716271367 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec0a8ab0 con 0x7f11f0157c80 2023-10-30T13:44:36.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11e7fff640 1 -- 172.21.15.46:0/2716271367 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec0a8e40 con 0x7f11f0157c80 2023-10-30T13:44:36.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11f6532640 1 -- 172.21.15.46:0/2716271367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f0157c80 msgr2=0x7f11f0161030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11f6532640 1 --2- 172.21.15.46:0/2716271367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f0157c80 0x7f11f0161030 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0097a0 tx=0x7f11ec01ec50 comp rx=0 tx=0).stop 2023-10-30T13:44:36.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11f6532640 1 -- 172.21.15.46:0/2716271367 shutdown_connections 2023-10-30T13:44:36.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11f6532640 1 --2- 172.21.15.46:0/2716271367 >> v2:172.21.15.154:3300/0 conn(0x7f11f006ab20 0x7f11f0070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11f6532640 1 --2- 172.21.15.46:0/2716271367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f0157c80 0x7f11f0161030 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11f6532640 1 -- 172.21.15.46:0/2716271367 >> 172.21.15.46:0/2716271367 conn(0x7f11f00c75d0 msgr2=0x7f11f00c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:36.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11f6532640 1 -- 172.21.15.46:0/2716271367 shutdown_connections 2023-10-30T13:44:36.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11f6532640 1 -- 172.21.15.46:0/2716271367 wait complete. 2023-10-30T13:44:36.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11f6532640 1 Processor -- start 2023-10-30T13:44:36.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.020+0000 7f11f6532640 1 -- start start 2023-10-30T13:44:36.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f6532640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11f006ab20 0x7f11f01ea6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f6532640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f006fef0 0x7f11f0158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f6532640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f0157c80 0x7f11f01f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f6532640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f01f7070 con 0x7f11f006fef0 2023-10-30T13:44:36.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f6532640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f00c6ac0 con 0x7f11f006ab20 2023-10-30T13:44:36.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f6532640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f00c6da0 con 0x7f11f0157c80 2023-10-30T13:44:36.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f4d2f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f006fef0 0x7f11f0158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f4d2f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f006fef0 0x7f11f0158160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51154/0 (socket says 172.21.15.46:51154) 2023-10-30T13:44:36.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f4d2f640 1 -- 172.21.15.46:0/1879316893 learned_addr learned my addr 172.21.15.46:0/1879316893 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:36.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f5530640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11f006ab20 0x7f11f01ea6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f5d31640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f0157c80 0x7f11f01f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f4d2f640 1 -- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f0157c80 msgr2=0x7f11f01f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f4d2f640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f0157c80 0x7f11f01f6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f4d2f640 1 -- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11f006ab20 msgr2=0x7f11f01ea6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f4d2f640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11f006ab20 0x7f11f01ea6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f4d2f640 1 -- 172.21.15.46:0/1879316893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ec0093b0 con 0x7f11f006fef0 2023-10-30T13:44:36.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f4d2f640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f006fef0 0x7f11f0158160 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f11e000c500 tx=0x7f11e000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:36.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11e5ffb640 1 -- 172.21.15.46:0/1879316893 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e00074e0 con 0x7f11f006fef0 2023-10-30T13:44:36.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11e5ffb640 1 -- 172.21.15.46:0/1879316893 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e0007640 con 0x7f11f006fef0 2023-10-30T13:44:36.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11e5ffb640 1 -- 172.21.15.46:0/1879316893 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e00049f0 con 0x7f11f006fef0 2023-10-30T13:44:36.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f6532640 1 -- 172.21.15.46:0/1879316893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f00c7080 con 0x7f11f006fef0 2023-10-30T13:44:36.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f6532640 1 -- 172.21.15.46:0/1879316893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f00c9040 con 0x7f11f006fef0 2023-10-30T13:44:36.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11f6532640 1 -- 172.21.15.46:0/1879316893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f01e8bb0 con 0x7f11f006fef0 2023-10-30T13:44:36.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.024+0000 7f11e5ffb640 1 -- 172.21.15.46:0/1879316893 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f11e0004b50 con 0x7f11f006fef0 2023-10-30T13:44:36.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.028+0000 7f11e5ffb640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11d40664a0 0x7f11d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.028+0000 7f11e5ffb640 1 -- 172.21.15.46:0/1879316893 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f11e0083b70 con 0x7f11f006fef0 2023-10-30T13:44:36.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.028+0000 7f11f5530640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11d40664a0 0x7f11d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.028+0000 7f11f5530640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11d40664a0 0x7f11d4068960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0b2000 tx=0x7f11ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:36.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.032+0000 7f11e5ffb640 1 -- 172.21.15.46:0/1879316893 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f11f01e8bb0 con 0x7f11f006fef0 2023-10-30T13:44:36.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.220+0000 7f11f6532640 1 -- 172.21.15.46:0/1879316893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11f01e8bb0 con 0x7f11f006fef0 2023-10-30T13:44:36.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11e5ffb640 1 -- 172.21.15.46:0/1879316893 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f11f01e8bb0 con 0x7f11f006fef0 2023-10-30T13:44:36.229 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:36.229 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:43:29.173143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6036},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:36.229 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 103 2023-10-30T13:44:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11f6532640 1 -- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11d40664a0 msgr2=0x7f11d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11f6532640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11d40664a0 0x7f11d4068960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0b2000 tx=0x7f11ec008040 comp rx=0 tx=0).stop 2023-10-30T13:44:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11f6532640 1 -- 172.21.15.46:0/1879316893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f006fef0 msgr2=0x7f11f0158160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11f6532640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f006fef0 0x7f11f0158160 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f11e000c500 tx=0x7f11e000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11f6532640 1 -- 172.21.15.46:0/1879316893 shutdown_connections 2023-10-30T13:44:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11f6532640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11d40664a0 0x7f11d4068960 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11f6532640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f0157c80 0x7f11f01f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11f6532640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f006fef0 0x7f11f0158160 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11f6532640 1 --2- 172.21.15.46:0/1879316893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11f006ab20 0x7f11f01ea6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11f6532640 1 -- 172.21.15.46:0/1879316893 >> 172.21.15.46:0/1879316893 conn(0x7f11f00c75d0 msgr2=0x7f11f00c8930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11f6532640 1 -- 172.21.15.46:0/1879316893 shutdown_connections 2023-10-30T13:44:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:36.224+0000 7f11f6532640 1 -- 172.21.15.46:0/1879316893 wait complete. 2023-10-30T13:44:36.249 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:44:38.251 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:38.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f1768cc9640 1 Processor -- start 2023-10-30T13:44:38.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f1768cc9640 1 -- start start 2023-10-30T13:44:38.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f1768cc9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f17640c05d0 0x7f176413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:38.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f1768cc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17640c6f60 0x7f176413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:38.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f1768cc9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17640c0a50 con 0x7f17640c6b80 2023-10-30T13:44:38.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f1768cc9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f176406f6e0 con 0x7f17640c05d0 2023-10-30T13:44:38.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f1768cc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176406f9c0 con 0x7f17640c6f60 2023-10-30T13:44:38.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f175bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f17640c05d0 0x7f176413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:38.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f175bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f17640c05d0 0x7f176413ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33070/0 (socket says 172.21.15.46:33070) 2023-10-30T13:44:38.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f175bfff640 1 -- 172.21.15.46:0/3403823909 learned_addr learned my addr 172.21.15.46:0/3403823909 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:38.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f176259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f17640c6b80 0x7f17640bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40180/0 (socket says 172.21.15.46:40180) 2023-10-30T13:44:38.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f1762d9d640 1 --2- 172.21.15.46:0/3403823909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17640c6f60 0x7f176413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:38.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f175bfff640 1 -- 172.21.15.46:0/3403823909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17640c6f60 msgr2=0x7f176413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:38.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f175bfff640 1 --2- 172.21.15.46:0/3403823909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17640c6f60 0x7f176413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f175bfff640 1 -- 172.21.15.46:0/3403823909 >> v1:172.21.15.46:6789/0 conn(0x7f17640c6b80 legacy=0x7f17640bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f175bfff640 1 -- 172.21.15.46:0/3403823909 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176406fca0 con 0x7f17640c05d0 2023-10-30T13:44:38.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.400+0000 7f1761d9b640 1 -- 172.21.15.46:0/3403823909 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386407069 0 0) 0x7f17640c0a50 con 0x7f17640c6b80 2023-10-30T13:44:38.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.404+0000 7f175bfff640 1 --2- 172.21.15.46:0/3403823909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640c05d0 0x7f176413ed90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f174c00bb10 tx=0x7f174c01ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac4aeb462413ac9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:38.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.404+0000 7f1761d9b640 1 -- 172.21.15.46:0/3403823909 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c0a7bf0 con 0x7f17640c05d0 2023-10-30T13:44:38.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.404+0000 7f1761d9b640 1 -- 172.21.15.46:0/3403823909 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c0a7d50 con 0x7f17640c05d0 2023-10-30T13:44:38.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.404+0000 7f1761d9b640 1 -- 172.21.15.46:0/3403823909 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c0af660 con 0x7f17640c05d0 2023-10-30T13:44:38.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.404+0000 7f1768cc9640 1 -- 172.21.15.46:0/3403823909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640c05d0 msgr2=0x7f176413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.404+0000 7f1768cc9640 1 --2- 172.21.15.46:0/3403823909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640c05d0 0x7f176413ed90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f174c00bb10 tx=0x7f174c01ec10 comp rx=0 tx=0).stop 2023-10-30T13:44:38.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.404+0000 7f1768cc9640 1 -- 172.21.15.46:0/3403823909 shutdown_connections 2023-10-30T13:44:38.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.404+0000 7f1768cc9640 1 --2- 172.21.15.46:0/3403823909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17640c6f60 0x7f176413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.404+0000 7f1768cc9640 1 --2- 172.21.15.46:0/3403823909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640c05d0 0x7f176413ed90 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.404+0000 7f1768cc9640 1 -- 172.21.15.46:0/3403823909 >> 172.21.15.46:0/3403823909 conn(0x7f17640c7910 msgr2=0x7f17640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:38.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.404+0000 7f1768cc9640 1 -- 172.21.15.46:0/3403823909 shutdown_connections 2023-10-30T13:44:38.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.404+0000 7f1768cc9640 1 -- 172.21.15.46:0/3403823909 wait complete. 2023-10-30T13:44:38.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f1768cc9640 1 Processor -- start 2023-10-30T13:44:38.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f1768cc9640 1 -- start start 2023-10-30T13:44:38.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f1768cc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17640c05d0 0x7f17641ee750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:38.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f1768cc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640c6b80 0x7f176414de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:38.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f1768cc9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17640c6f60 0x7f1764069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:38.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f1768cc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17641eec90 con 0x7f17640c6b80 2023-10-30T13:44:38.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f1768cc9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176414d530 con 0x7f17640c6f60 2023-10-30T13:44:38.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f1768cc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176414c500 con 0x7f17640c05d0 2023-10-30T13:44:38.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f175bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640c6b80 0x7f176414de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:38.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f175bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640c6b80 0x7f176414de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33076/0 (socket says 172.21.15.46:33076) 2023-10-30T13:44:38.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f175bfff640 1 -- 172.21.15.46:0/79712821 learned_addr learned my addr 172.21.15.46:0/79712821 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:38.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f176259c640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17640c05d0 0x7f17641ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:38.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f1762d9d640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17640c6f60 0x7f1764069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:38.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f175bfff640 1 -- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17640c05d0 msgr2=0x7f17641ee750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f175bfff640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17640c05d0 0x7f17641ee750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f175bfff640 1 -- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17640c6f60 msgr2=0x7f1764069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f175bfff640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17640c6f60 0x7f1764069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f175bfff640 1 -- 172.21.15.46:0/79712821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c00b770 con 0x7f17640c6b80 2023-10-30T13:44:38.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f175bfff640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640c6b80 0x7f176414de50 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f174c00b2c0 tx=0x7f174c01b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:38.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f175affd640 1 -- 172.21.15.46:0/79712821 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c0ba660 con 0x7f17640c6b80 2023-10-30T13:44:38.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f175affd640 1 -- 172.21.15.46:0/79712821 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c0ba7c0 con 0x7f17640c6b80 2023-10-30T13:44:38.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f1768cc9640 1 -- 172.21.15.46:0/79712821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1764130060 con 0x7f17640c6b80 2023-10-30T13:44:38.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f175affd640 1 -- 172.21.15.46:0/79712821 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c0bab70 con 0x7f17640c6b80 2023-10-30T13:44:38.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.408+0000 7f1768cc9640 1 -- 172.21.15.46:0/79712821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176414ca50 con 0x7f17640c6b80 2023-10-30T13:44:38.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.412+0000 7f1768cc9640 1 -- 172.21.15.46:0/79712821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176404e170 con 0x7f17640c6b80 2023-10-30T13:44:38.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.412+0000 7f175affd640 1 -- 172.21.15.46:0/79712821 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f174c0c33b0 con 0x7f17640c6b80 2023-10-30T13:44:38.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.412+0000 7f175affd640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17440664a0 0x7f1744068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:38.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.412+0000 7f175affd640 1 -- 172.21.15.46:0/79712821 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f174c11f350 con 0x7f17640c6b80 2023-10-30T13:44:38.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.412+0000 7f176259c640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17440664a0 0x7f1744068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:38.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.412+0000 7f176259c640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17440664a0 0x7f1744068960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1754000f70 tx=0x7f1754007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:38.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.416+0000 7f175affd640 1 -- 172.21.15.46:0/79712821 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f174c0ec340 con 0x7f17640c6b80 2023-10-30T13:44:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.604+0000 7f1768cc9640 1 -- 172.21.15.46:0/79712821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17641390f0 con 0x7f17640c6b80 2023-10-30T13:44:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.604+0000 7f175affd640 1 -- 172.21.15.46:0/79712821 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f174c0ef960 con 0x7f17640c6b80 2023-10-30T13:44:38.612 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:38.613 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:43:29.173143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6036},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6036":{"gid":6036,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6836/3037060424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3037060424},{"type":"v1","addr":"172.21.15.46:6836","nonce":3037060424}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 103 2023-10-30T13:44:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.608+0000 7f1768cc9640 1 -- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17440664a0 msgr2=0x7f1744068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.608+0000 7f1768cc9640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17440664a0 0x7f1744068960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1754000f70 tx=0x7f1754007b60 comp rx=0 tx=0).stop 2023-10-30T13:44:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.608+0000 7f1768cc9640 1 -- 172.21.15.46:0/79712821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640c6b80 msgr2=0x7f176414de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.608+0000 7f1768cc9640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640c6b80 0x7f176414de50 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f174c00b2c0 tx=0x7f174c01b9b0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.608+0000 7f1768cc9640 1 -- 172.21.15.46:0/79712821 shutdown_connections 2023-10-30T13:44:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.608+0000 7f1768cc9640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17440664a0 0x7f1744068960 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.608+0000 7f1768cc9640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17640c6f60 0x7f1764069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.608+0000 7f1768cc9640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640c6b80 0x7f176414de50 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.608+0000 7f1768cc9640 1 --2- 172.21.15.46:0/79712821 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17640c05d0 0x7f17641ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.608+0000 7f1768cc9640 1 -- 172.21.15.46:0/79712821 >> 172.21.15.46:0/79712821 conn(0x7f17640c7910 msgr2=0x7f1764139310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.608+0000 7f1768cc9640 1 -- 172.21.15.46:0/79712821 shutdown_connections 2023-10-30T13:44:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:38.608+0000 7f1768cc9640 1 -- 172.21.15.46:0/79712821 wait complete. 2023-10-30T13:44:38.633 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:44:40.634 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:40.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb953479640 1 Processor -- start 2023-10-30T13:44:40.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb953479640 1 -- start start 2023-10-30T13:44:40.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb953479640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb94c0cc0b0 0x7fb94c06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:40.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb953479640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb94c0b1ea0 0x7fb94c0cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:40.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb953479640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c06ade0 con 0x7fb94c153c30 2023-10-30T13:44:40.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb953479640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c1596e0 con 0x7fb94c0cc0b0 2023-10-30T13:44:40.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb953479640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c1598f0 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb950a14640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb94c0cc0b0 0x7fb94c06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:40.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb950a14640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb94c0cc0b0 0x7fb94c06a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33086/0 (socket says 172.21.15.46:33086) 2023-10-30T13:44:40.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb950a14640 1 -- 172.21.15.46:0/2213867189 learned_addr learned my addr 172.21.15.46:0/2213867189 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:40.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb951a16640 1 --2- 172.21.15.46:0/2213867189 >> v2:172.21.15.154:3300/0 conn(0x7fb94c0b1ea0 0x7fb94c0cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:40.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb950a14640 1 -- 172.21.15.46:0/2213867189 >> v2:172.21.15.154:3300/0 conn(0x7fb94c0b1ea0 msgr2=0x7fb94c0cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:40.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb950a14640 1 --2- 172.21.15.46:0/2213867189 >> v2:172.21.15.154:3300/0 conn(0x7fb94c0b1ea0 0x7fb94c0cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb950a14640 1 -- 172.21.15.46:0/2213867189 >> v1:172.21.15.154:6789/0 conn(0x7fb94c153c30 legacy=0x7fb94c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.784+0000 7fb950a14640 1 -- 172.21.15.46:0/2213867189 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c159b90 con 0x7fb94c0cc0b0 2023-10-30T13:44:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.788+0000 7fb950a14640 1 --2- 172.21.15.46:0/2213867189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c0cc0b0 0x7fb94c06a8a0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0b4610 tx=0x7fb93c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44552713d39d92d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:40.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.788+0000 7fb943fff640 1 -- 172.21.15.46:0/2213867189 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0a89f0 con 0x7fb94c0cc0b0 2023-10-30T13:44:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.788+0000 7fb943fff640 1 -- 172.21.15.46:0/2213867189 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c0a8b50 con 0x7fb94c0cc0b0 2023-10-30T13:44:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.788+0000 7fb943fff640 1 -- 172.21.15.46:0/2213867189 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0b0440 con 0x7fb94c0cc0b0 2023-10-30T13:44:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.788+0000 7fb953479640 1 -- 172.21.15.46:0/2213867189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c0cc0b0 msgr2=0x7fb94c06a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.788+0000 7fb953479640 1 --2- 172.21.15.46:0/2213867189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c0cc0b0 0x7fb94c06a8a0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0b4610 tx=0x7fb93c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.788+0000 7fb953479640 1 -- 172.21.15.46:0/2213867189 shutdown_connections 2023-10-30T13:44:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.788+0000 7fb953479640 1 --2- 172.21.15.46:0/2213867189 >> v2:172.21.15.154:3300/0 conn(0x7fb94c0b1ea0 0x7fb94c0cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.788+0000 7fb953479640 1 --2- 172.21.15.46:0/2213867189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c0cc0b0 0x7fb94c06a8a0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.788+0000 7fb953479640 1 -- 172.21.15.46:0/2213867189 >> 172.21.15.46:0/2213867189 conn(0x7fb94c06bf40 msgr2=0x7fb94c06c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.788+0000 7fb953479640 1 -- 172.21.15.46:0/2213867189 shutdown_connections 2023-10-30T13:44:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.788+0000 7fb953479640 1 -- 172.21.15.46:0/2213867189 wait complete. 2023-10-30T13:44:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb953479640 1 Processor -- start 2023-10-30T13:44:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb953479640 1 -- start start 2023-10-30T13:44:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb953479640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c0b1ea0 0x7fb94c13ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb953479640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb94c0cc0b0 0x7fb94c0c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb953479640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb94c153c30 0x7fb94c147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb953479640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0b3d90 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb953479640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c1404a0 con 0x7fb94c0cc0b0 2023-10-30T13:44:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb953479640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0b3fc0 con 0x7fb94c153c30 2023-10-30T13:44:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb951215640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c0b1ea0 0x7fb94c13ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb951215640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c0b1ea0 0x7fb94c13ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33090/0 (socket says 172.21.15.46:33090) 2023-10-30T13:44:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb951215640 1 -- 172.21.15.46:0/2610138032 learned_addr learned my addr 172.21.15.46:0/2610138032 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb950a14640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb94c0cc0b0 0x7fb94c0c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:40.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb951a16640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb94c153c30 0x7fb94c147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:40.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb951215640 1 -- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb94c153c30 msgr2=0x7fb94c147a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:40.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb951215640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb94c153c30 0x7fb94c147a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb951215640 1 -- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb94c0cc0b0 msgr2=0x7fb94c0c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:40.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb951215640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb94c0cc0b0 0x7fb94c0c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb951215640 1 -- 172.21.15.46:0/2610138032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93c009470 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb951a16640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb94c153c30 0x7fb94c147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:40.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb951215640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c0b1ea0 0x7fb94c13ff60 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb94800c3c0 tx=0x7fb94800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:40.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb941ffb640 1 -- 172.21.15.46:0/2610138032 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9480052d0 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb941ffb640 1 -- 172.21.15.46:0/2610138032 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb948005430 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb941ffb640 1 -- 172.21.15.46:0/2610138032 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948003840 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb953479640 1 -- 172.21.15.46:0/2610138032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c0b4280 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb953479640 1 -- 172.21.15.46:0/2610138032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c144980 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.792+0000 7fb953479640 1 -- 172.21.15.46:0/2610138032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94c0b2620 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.796+0000 7fb941ffb640 1 -- 172.21.15.46:0/2610138032 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9480039a0 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.796+0000 7fb941ffb640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9300664a0 0x7fb930068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:40.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.796+0000 7fb941ffb640 1 -- 172.21.15.46:0/2610138032 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fb9480836c0 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.796+0000 7fb950a14640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9300664a0 0x7fb930068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:40.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.796+0000 7fb950a14640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9300664a0 0x7fb930068960 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0b2000 tx=0x7fb93c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:40.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.800+0000 7fb941ffb640 1 -- 172.21.15.46:0/2610138032 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb94c0b2620 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.988+0000 7fb953479640 1 -- 172.21.15.46:0/2610138032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb94c06ade0 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.988+0000 7fb941ffb640 1 -- 172.21.15.46:0/2610138032 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5728 (secure 0 0 0) 0x7fb94c06ade0 con 0x7fb94c0b1ea0 2023-10-30T13:44:40.996 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:40.996 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:44:39.066162+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 104 2023-10-30T13:44:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.992+0000 7fb953479640 1 -- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9300664a0 msgr2=0x7fb930068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.992+0000 7fb953479640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9300664a0 0x7fb930068960 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0b2000 tx=0x7fb93c009040 comp rx=0 tx=0).stop 2023-10-30T13:44:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.992+0000 7fb953479640 1 -- 172.21.15.46:0/2610138032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c0b1ea0 msgr2=0x7fb94c13ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.992+0000 7fb953479640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c0b1ea0 0x7fb94c13ff60 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb94800c3c0 tx=0x7fb94800c880 comp rx=0 tx=0).stop 2023-10-30T13:44:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.992+0000 7fb953479640 1 -- 172.21.15.46:0/2610138032 shutdown_connections 2023-10-30T13:44:40.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.992+0000 7fb953479640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9300664a0 0x7fb930068960 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.992+0000 7fb953479640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb94c153c30 0x7fb94c147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.992+0000 7fb953479640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb94c0cc0b0 0x7fb94c0c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.992+0000 7fb953479640 1 --2- 172.21.15.46:0/2610138032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb94c0b1ea0 0x7fb94c13ff60 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.992+0000 7fb953479640 1 -- 172.21.15.46:0/2610138032 >> 172.21.15.46:0/2610138032 conn(0x7fb94c06bf40 msgr2=0x7fb94c0caad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:41.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.992+0000 7fb953479640 1 -- 172.21.15.46:0/2610138032 shutdown_connections 2023-10-30T13:44:41.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:40.992+0000 7fb953479640 1 -- 172.21.15.46:0/2610138032 wait complete. 2023-10-30T13:44:41.017 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:44:41.017 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:44:41.017 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f883b194640 1 Processor -- start 2023-10-30T13:44:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f883b194640 1 -- start start 2023-10-30T13:44:41.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f883b194640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8834069e90 0x7f883415a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:41.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f883b194640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f883406a270 0x7f88340c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:41.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f883b194640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88340c6c10 con 0x7f883406cb80 2023-10-30T13:44:41.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f883b194640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88340c8570 con 0x7f883406a270 2023-10-30T13:44:41.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f883b194640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88340c6ef0 con 0x7f8834069e90 2023-10-30T13:44:41.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f8833fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f883406a270 0x7f88340c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:41.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f8833fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f883406a270 0x7f88340c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33100/0 (socket says 172.21.15.46:33100) 2023-10-30T13:44:41.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f8839731640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f883406cb80 0x7f883406d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40196/0 (socket says 172.21.15.46:40196) 2023-10-30T13:44:41.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f8833fff640 1 -- 172.21.15.46:0/1686369449 learned_addr learned my addr 172.21.15.46:0/1686369449 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:41.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f8838f30640 1 --2- 172.21.15.46:0/1686369449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8834069e90 0x7f883415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:41.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f8833fff640 1 -- 172.21.15.46:0/1686369449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8834069e90 msgr2=0x7f883415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:41.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f8833fff640 1 --2- 172.21.15.46:0/1686369449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8834069e90 0x7f883415a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f8833fff640 1 -- 172.21.15.46:0/1686369449 >> v1:172.21.15.46:6789/0 conn(0x7f883406cb80 legacy=0x7f883406d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:41.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f8833fff640 1 -- 172.21.15.46:0/1686369449 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88340c71d0 con 0x7f883406a270 2023-10-30T13:44:41.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.156+0000 7f88337fe640 1 -- 172.21.15.46:0/1686369449 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965312150 0 0) 0x7f88340c6c10 con 0x7f883406cb80 2023-10-30T13:44:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.160+0000 7f8833fff640 1 --2- 172.21.15.46:0/1686369449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883406a270 0x7f88340c9cb0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f8820009870 tx=0x7f882001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc990694729d46c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.160+0000 7f88337fe640 1 -- 172.21.15.46:0/1686369449 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88200a7070 con 0x7f883406a270 2023-10-30T13:44:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.160+0000 7f88337fe640 1 -- 172.21.15.46:0/1686369449 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88200ab430 con 0x7f883406a270 2023-10-30T13:44:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.160+0000 7f88337fe640 1 -- 172.21.15.46:0/1686369449 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88200ab7c0 con 0x7f883406a270 2023-10-30T13:44:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.160+0000 7f883b194640 1 -- 172.21.15.46:0/1686369449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883406a270 msgr2=0x7f88340c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.160+0000 7f883b194640 1 --2- 172.21.15.46:0/1686369449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883406a270 0x7f88340c9cb0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f8820009870 tx=0x7f882001c540 comp rx=0 tx=0).stop 2023-10-30T13:44:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.160+0000 7f883b194640 1 -- 172.21.15.46:0/1686369449 shutdown_connections 2023-10-30T13:44:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.160+0000 7f883b194640 1 --2- 172.21.15.46:0/1686369449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883406a270 0x7f88340c9cb0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.160+0000 7f883b194640 1 --2- 172.21.15.46:0/1686369449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8834069e90 0x7f883415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.160+0000 7f883b194640 1 -- 172.21.15.46:0/1686369449 >> 172.21.15.46:0/1686369449 conn(0x7f883406dfd0 msgr2=0x7f883406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:41.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.160+0000 7f883b194640 1 -- 172.21.15.46:0/1686369449 shutdown_connections 2023-10-30T13:44:41.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.160+0000 7f883b194640 1 -- 172.21.15.46:0/1686369449 wait complete. 2023-10-30T13:44:41.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f883b194640 1 Processor -- start 2023-10-30T13:44:41.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f883b194640 1 -- start start 2023-10-30T13:44:41.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f883b194640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8834069e90 0x7f883414c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:41.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f883b194640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f883406a270 0x7f88341568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:41.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f883b194640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883406cb80 0x7f8834156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:41.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f883b194640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88341f29e0 con 0x7f883406cb80 2023-10-30T13:44:41.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f883b194640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88341f1e60 con 0x7f8834069e90 2023-10-30T13:44:41.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f883b194640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88341f2140 con 0x7f883406a270 2023-10-30T13:44:41.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8839731640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883406cb80 0x7f8834156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:41.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8839731640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883406cb80 0x7f8834156de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33106/0 (socket says 172.21.15.46:33106) 2023-10-30T13:44:41.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8839731640 1 -- 172.21.15.46:0/3403095196 learned_addr learned my addr 172.21.15.46:0/3403095196 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:41.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8833fff640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f883406a270 0x7f88341568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:41.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8838f30640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8834069e90 0x7f883414c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:41.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8839731640 1 -- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f883406a270 msgr2=0x7f88341568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8839731640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f883406a270 0x7f88341568a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8839731640 1 -- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8834069e90 msgr2=0x7f883414c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8839731640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8834069e90 0x7f883414c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8839731640 1 -- 172.21.15.46:0/3403095196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88200094d0 con 0x7f883406cb80 2023-10-30T13:44:41.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8838f30640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8834069e90 0x7f883414c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:44:41.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8833fff640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f883406a270 0x7f88341568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:44:41.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f8839731640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883406cb80 0x7f8834156de0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f882c00c780 tx=0x7f882c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:41.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f88317fa640 1 -- 172.21.15.46:0/3403095196 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c00ce90 con 0x7f883406cb80 2023-10-30T13:44:41.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f88317fa640 1 -- 172.21.15.46:0/3403095196 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882c00a640 con 0x7f883406cb80 2023-10-30T13:44:41.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f88317fa640 1 -- 172.21.15.46:0/3403095196 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c00a9d0 con 0x7f883406cb80 2023-10-30T13:44:41.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f883b194640 1 -- 172.21.15.46:0/3403095196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88341f2420 con 0x7f883406cb80 2023-10-30T13:44:41.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.164+0000 7f883b194640 1 -- 172.21.15.46:0/3403095196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8834156380 con 0x7f883406cb80 2023-10-30T13:44:41.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.168+0000 7f883b194640 1 -- 172.21.15.46:0/3403095196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883406a6b0 con 0x7f883406cb80 2023-10-30T13:44:41.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.168+0000 7f88317fa640 1 -- 172.21.15.46:0/3403095196 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f882c00ab30 con 0x7f883406cb80 2023-10-30T13:44:41.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.168+0000 7f88317fa640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88180664a0 0x7f8818068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:41.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.168+0000 7f88317fa640 1 -- 172.21.15.46:0/3403095196 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f882c083760 con 0x7f883406cb80 2023-10-30T13:44:41.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.168+0000 7f8838f30640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88180664a0 0x7f8818068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:41.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.168+0000 7f8838f30640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88180664a0 0x7f8818068960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8828009ae0 tx=0x7f8828008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:41.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.172+0000 7f88317fa640 1 -- 172.21.15.46:0/3403095196 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f882c050830 con 0x7f883406cb80 2023-10-30T13:44:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.360+0000 7f883b194640 1 -- 172.21.15.46:0/3403095196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88340ca1c0 con 0x7f883406cb80 2023-10-30T13:44:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.360+0000 7f88317fa640 1 -- 172.21.15.46:0/3403095196 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5728 (secure 0 0 0) 0x7f882c018090 con 0x7f883406cb80 2023-10-30T13:44:41.368 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:41.369 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:44:39.066162+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:41.370 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 104 2023-10-30T13:44:41.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.364+0000 7f883b194640 1 -- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88180664a0 msgr2=0x7f8818068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:41.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.364+0000 7f883b194640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88180664a0 0x7f8818068960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8828009ae0 tx=0x7f8828008040 comp rx=0 tx=0).stop 2023-10-30T13:44:41.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.364+0000 7f883b194640 1 -- 172.21.15.46:0/3403095196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883406cb80 msgr2=0x7f8834156de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:41.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.364+0000 7f883b194640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883406cb80 0x7f8834156de0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f882c00c780 tx=0x7f882c00cc40 comp rx=0 tx=0).stop 2023-10-30T13:44:41.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.364+0000 7f883b194640 1 -- 172.21.15.46:0/3403095196 shutdown_connections 2023-10-30T13:44:41.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.364+0000 7f883b194640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88180664a0 0x7f8818068960 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.364+0000 7f883b194640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f883406cb80 0x7f8834156de0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.364+0000 7f883b194640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f883406a270 0x7f88341568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.364+0000 7f883b194640 1 --2- 172.21.15.46:0/3403095196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8834069e90 0x7f883414c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.364+0000 7f883b194640 1 -- 172.21.15.46:0/3403095196 >> 172.21.15.46:0/3403095196 conn(0x7f883406dfd0 msgr2=0x7f8834071b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.364+0000 7f883b194640 1 -- 172.21.15.46:0/3403095196 shutdown_connections 2023-10-30T13:44:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:41.364+0000 7f883b194640 1 -- 172.21.15.46:0/3403095196 wait complete. 2023-10-30T13:44:41.390 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 104, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 104, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6201": { "addr": "172.21.15.46:6837/2546436963", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2546436963, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2546436963, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 6201, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_6366": { "addr": "172.21.15.46:6833/190134548", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 190134548, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 190134548, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6366, "incarnation": 104, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:44:39.066162+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6201, "mds_1": 6366 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:44:43.391 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c9cbd3640 1 Processor -- start 2023-10-30T13:44:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c9cbd3640 1 -- start start 2023-10-30T13:44:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c9cbd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c981538a0 0x7f4c9815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c9cbd3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4c980cf160 0x7f4c980cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c9cbd3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c980cb800 con 0x7f4c980ca0e0 2023-10-30T13:44:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c9cbd3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c9815dc30 con 0x7f4c980cf160 2023-10-30T13:44:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c9cbd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c980cb330 con 0x7f4c981538a0 2023-10-30T13:44:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c96d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4c980cf160 0x7f4c980cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c96d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4c980cf160 0x7f4c980cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33110/0 (socket says 172.21.15.46:33110) 2023-10-30T13:44:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c96d9d640 1 -- 172.21.15.46:0/4183983250 learned_addr learned my addr 172.21.15.46:0/4183983250 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c9659c640 1 --2- 172.21.15.46:0/4183983250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c981538a0 0x7f4c9815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c96d9d640 1 -- 172.21.15.46:0/4183983250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c981538a0 msgr2=0x7f4c9815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c96d9d640 1 --2- 172.21.15.46:0/4183983250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c981538a0 0x7f4c9815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c96d9d640 1 -- 172.21.15.46:0/4183983250 >> v1:172.21.15.154:6789/0 conn(0x7f4c980ca0e0 legacy=0x7f4c9815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c96d9d640 1 -- 172.21.15.46:0/4183983250 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c980cad70 con 0x7f4c980cf160 2023-10-30T13:44:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.540+0000 7f4c9659c640 1 --2- 172.21.15.46:0/4183983250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c981538a0 0x7f4c9815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.544+0000 7f4c96d9d640 1 --2- 172.21.15.46:0/4183983250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c980cf160 0x7f4c980cf590 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009e50 tx=0x7f4c9001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18f3f48e0ce0d8c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.544+0000 7f4c95d9b640 1 -- 172.21.15.46:0/4183983250 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c900af400 con 0x7f4c980cf160 2023-10-30T13:44:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.544+0000 7f4c95d9b640 1 -- 172.21.15.46:0/4183983250 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c900ab050 con 0x7f4c980cf160 2023-10-30T13:44:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.544+0000 7f4c95d9b640 1 -- 172.21.15.46:0/4183983250 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c900af780 con 0x7f4c980cf160 2023-10-30T13:44:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.544+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/4183983250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c980cf160 msgr2=0x7f4c980cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.544+0000 7f4c9cbd3640 1 --2- 172.21.15.46:0/4183983250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c980cf160 0x7f4c980cf590 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009e50 tx=0x7f4c9001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.544+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/4183983250 shutdown_connections 2023-10-30T13:44:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.544+0000 7f4c9cbd3640 1 --2- 172.21.15.46:0/4183983250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c980cf160 0x7f4c980cf590 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.544+0000 7f4c9cbd3640 1 --2- 172.21.15.46:0/4183983250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c981538a0 0x7f4c9815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.544+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/4183983250 >> 172.21.15.46:0/4183983250 conn(0x7f4c9806bfe0 msgr2=0x7f4c9806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:43.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.544+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/4183983250 shutdown_connections 2023-10-30T13:44:43.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.544+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/4183983250 wait complete. 2023-10-30T13:44:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9cbd3640 1 Processor -- start 2023-10-30T13:44:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9cbd3640 1 -- start start 2023-10-30T13:44:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9cbd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c980ca0e0 0x7f4c9815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9cbd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c980cf160 0x7f4c9815b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9cbd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c981538a0 0x7f4c9815b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9cbd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981e3960 con 0x7f4c980ca0e0 2023-10-30T13:44:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9cbd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981e4750 con 0x7f4c981538a0 2023-10-30T13:44:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9cbd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981e3ba0 con 0x7f4c980cf160 2023-10-30T13:44:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c980ca0e0 0x7f4c9815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c980ca0e0 0x7f4c9815a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33114/0 (socket says 172.21.15.46:33114) 2023-10-30T13:44:43.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9659c640 1 -- 172.21.15.46:0/752092625 learned_addr learned my addr 172.21.15.46:0/752092625 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:43.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c96d9d640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c981538a0 0x7f4c9815b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c8ffff640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c980cf160 0x7f4c9815b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9659c640 1 -- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c980cf160 msgr2=0x7f4c9815b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9659c640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c980cf160 0x7f4c9815b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9659c640 1 -- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c981538a0 msgr2=0x7f4c9815b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9659c640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c981538a0 0x7f4c9815b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9659c640 1 -- 172.21.15.46:0/752092625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c90009b30 con 0x7f4c980ca0e0 2023-10-30T13:44:43.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c8ffff640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c980cf160 0x7f4c9815b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:43.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9659c640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c980ca0e0 0x7f4c9815a8b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800d800 tx=0x7f4c8800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:43.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c8effd640 1 -- 172.21.15.46:0/752092625 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c880096c0 con 0x7f4c980ca0e0 2023-10-30T13:44:43.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c8effd640 1 -- 172.21.15.46:0/752092625 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c88009820 con 0x7f4c980ca0e0 2023-10-30T13:44:43.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c8effd640 1 -- 172.21.15.46:0/752092625 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88009bd0 con 0x7f4c980ca0e0 2023-10-30T13:44:43.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/752092625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c981e3430 con 0x7f4c980ca0e0 2023-10-30T13:44:43.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.548+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/752092625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c981e3db0 con 0x7f4c980ca0e0 2023-10-30T13:44:43.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.552+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/752092625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c981e49c0 con 0x7f4c980ca0e0 2023-10-30T13:44:43.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.552+0000 7f4c8effd640 1 -- 172.21.15.46:0/752092625 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4c88009d30 con 0x7f4c980ca0e0 2023-10-30T13:44:43.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.552+0000 7f4c8effd640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4c780664a0 0x7f4c78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.552+0000 7f4c8effd640 1 -- 172.21.15.46:0/752092625 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f4c880849e0 con 0x7f4c980ca0e0 2023-10-30T13:44:43.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.552+0000 7f4c8ffff640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4c780664a0 0x7f4c78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.556+0000 7f4c8ffff640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4c780664a0 0x7f4c78068960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4c9815a130 tx=0x7f4c80009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:43.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.556+0000 7f4c8effd640 1 -- 172.21.15.46:0/752092625 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4c880519a0 con 0x7f4c980ca0e0 2023-10-30T13:44:43.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.748+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/752092625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c980b2a50 con 0x7f4c980ca0e0 2023-10-30T13:44:43.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.748+0000 7f4c8effd640 1 -- 172.21.15.46:0/752092625 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5728 (secure 0 0 0) 0x7f4c88054fc0 con 0x7f4c980ca0e0 2023-10-30T13:44:43.754 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:43.755 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:44:39.066162+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:43.755 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 104 2023-10-30T13:44:43.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.752+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4c780664a0 msgr2=0x7f4c78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.752+0000 7f4c9cbd3640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4c780664a0 0x7f4c78068960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4c9815a130 tx=0x7f4c80009040 comp rx=0 tx=0).stop 2023-10-30T13:44:43.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.752+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/752092625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c980ca0e0 msgr2=0x7f4c9815a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.752+0000 7f4c9cbd3640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c980ca0e0 0x7f4c9815a8b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800d800 tx=0x7f4c8800dcc0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.752+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/752092625 shutdown_connections 2023-10-30T13:44:43.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.752+0000 7f4c9cbd3640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4c780664a0 0x7f4c78068960 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.752+0000 7f4c9cbd3640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c981538a0 0x7f4c9815b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.752+0000 7f4c9cbd3640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c980cf160 0x7f4c9815b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.752+0000 7f4c9cbd3640 1 --2- 172.21.15.46:0/752092625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c980ca0e0 0x7f4c9815a8b0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.752+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/752092625 >> 172.21.15.46:0/752092625 conn(0x7f4c9806bfe0 msgr2=0x7f4c980c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:43.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.752+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/752092625 shutdown_connections 2023-10-30T13:44:43.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:43.752+0000 7f4c9cbd3640 1 -- 172.21.15.46:0/752092625 wait complete. 2023-10-30T13:44:43.776 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:45.776 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:45.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7d4ed6640 1 Processor -- start 2023-10-30T13:44:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7d4ed6640 1 -- start start 2023-10-30T13:44:45.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7d4ed6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d006f7a0 0x7fb7d006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:45.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7d4ed6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb7d00c6bd0 0x7fb7d0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:45.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7d4ed6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d006a3d0 con 0x7fb7d0071110 2023-10-30T13:44:45.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7d4ed6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d00700c0 con 0x7fb7d00c6bd0 2023-10-30T13:44:45.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7d4ed6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0066cf0 con 0x7fb7d006f7a0 2023-10-30T13:44:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7cdd9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb7d00c6bd0 0x7fb7d0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7ce59c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d006f7a0 0x7fb7d006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7ce59c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d006f7a0 0x7fb7d006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:48220/0 (socket says 172.21.15.46:48220) 2023-10-30T13:44:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7ce59c640 1 -- 172.21.15.46:0/1854476844 learned_addr learned my addr 172.21.15.46:0/1854476844 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:45.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7cdd9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb7d00c6bd0 0x7fb7d0069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:56666/0 (socket says 172.21.15.46:56666) 2023-10-30T13:44:45.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7ce59c640 1 -- 172.21.15.46:0/1854476844 >> v2:172.21.15.154:3300/0 conn(0x7fb7d00c6bd0 msgr2=0x7fb7d0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:45.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7ce59c640 1 --2- 172.21.15.46:0/1854476844 >> v2:172.21.15.154:3300/0 conn(0x7fb7d00c6bd0 0x7fb7d0069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7ce59c640 1 -- 172.21.15.46:0/1854476844 >> v1:172.21.15.154:6789/0 conn(0x7fb7d0071110 legacy=0x7fb7d0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:45.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.920+0000 7fb7ce59c640 1 -- 172.21.15.46:0/1854476844 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d00c70d0 con 0x7fb7d006f7a0 2023-10-30T13:44:45.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.924+0000 7fb7ce59c640 1 --2- 172.21.15.46:0/1854476844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d006f7a0 0x7fb7d006fb80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0009870 tx=0x7fb7c001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab62ead428c51a45 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.924+0000 7fb7cd59a640 1 -- 172.21.15.46:0/1854476844 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c00a8a30 con 0x7fb7d006f7a0 2023-10-30T13:44:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.924+0000 7fb7cd59a640 1 -- 172.21.15.46:0/1854476844 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c00a8b90 con 0x7fb7d006f7a0 2023-10-30T13:44:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.924+0000 7fb7cd59a640 1 -- 172.21.15.46:0/1854476844 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c00b0440 con 0x7fb7d006f7a0 2023-10-30T13:44:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.924+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1854476844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d006f7a0 msgr2=0x7fb7d006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.924+0000 7fb7d4ed6640 1 --2- 172.21.15.46:0/1854476844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d006f7a0 0x7fb7d006fb80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0009870 tx=0x7fb7c001ed30 comp rx=0 tx=0).stop 2023-10-30T13:44:45.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.924+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1854476844 shutdown_connections 2023-10-30T13:44:45.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.924+0000 7fb7d4ed6640 1 --2- 172.21.15.46:0/1854476844 >> v2:172.21.15.154:3300/0 conn(0x7fb7d00c6bd0 0x7fb7d0069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.924+0000 7fb7d4ed6640 1 --2- 172.21.15.46:0/1854476844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d006f7a0 0x7fb7d006fb80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.924+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1854476844 >> 172.21.15.46:0/1854476844 conn(0x7fb7d00c7910 msgr2=0x7fb7d00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.924+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1854476844 shutdown_connections 2023-10-30T13:44:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.924+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1854476844 wait complete. 2023-10-30T13:44:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7d4ed6640 1 Processor -- start 2023-10-30T13:44:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7d4ed6640 1 -- start start 2023-10-30T13:44:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7d4ed6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d006f7a0 0x7fb7d01eefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7d4ed6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d0071110 0x7fb7d01f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7d4ed6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7d00c6bd0 0x7fb7d00c0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:45.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7d4ed6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d01f2b80 con 0x7fb7d006f7a0 2023-10-30T13:44:45.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7d4ed6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d01f2550 con 0x7fb7d00c6bd0 2023-10-30T13:44:45.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7d4ed6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0154e20 con 0x7fb7d0071110 2023-10-30T13:44:45.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7ce59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d006f7a0 0x7fb7d01eefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:45.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7ce59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d006f7a0 0x7fb7d01eefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33124/0 (socket says 172.21.15.46:33124) 2023-10-30T13:44:45.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7ce59c640 1 -- 172.21.15.46:0/1889288557 learned_addr learned my addr 172.21.15.46:0/1889288557 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:45.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7ced9d640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7d00c6bd0 0x7fb7d00c0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:45.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7cdd9b640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d0071110 0x7fb7d01f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:45.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7ce59c640 1 -- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d0071110 msgr2=0x7fb7d01f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:45.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7ce59c640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d0071110 0x7fb7d01f3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7ce59c640 1 -- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7d00c6bd0 msgr2=0x7fb7d00c0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:45.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7ce59c640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7d00c6bd0 0x7fb7d00c0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7ce59c640 1 -- 172.21.15.46:0/1889288557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c00094d0 con 0x7fb7d006f7a0 2023-10-30T13:44:45.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7cdd9b640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d0071110 0x7fb7d01f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:45.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7ce59c640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d006f7a0 0x7fb7d01eefe0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00b3040 tx=0x7fb7c001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:45.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7beffd640 1 -- 172.21.15.46:0/1889288557 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c00b0dd0 con 0x7fb7d006f7a0 2023-10-30T13:44:45.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7beffd640 1 -- 172.21.15.46:0/1889288557 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c00bb460 con 0x7fb7d006f7a0 2023-10-30T13:44:45.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1889288557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d00adcf0 con 0x7fb7d006f7a0 2023-10-30T13:44:45.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7beffd640 1 -- 172.21.15.46:0/1889288557 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c00bb810 con 0x7fb7d006f7a0 2023-10-30T13:44:45.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.928+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1889288557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d00ae2b0 con 0x7fb7d006f7a0 2023-10-30T13:44:45.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.932+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1889288557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d004e170 con 0x7fb7d006f7a0 2023-10-30T13:44:45.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.932+0000 7fb7beffd640 1 -- 172.21.15.46:0/1889288557 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb7c00b83f0 con 0x7fb7d006f7a0 2023-10-30T13:44:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.932+0000 7fb7beffd640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7b00664d0 0x7fb7b0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.932+0000 7fb7beffd640 1 -- 172.21.15.46:0/1889288557 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fb7c0121bd0 con 0x7fb7d006f7a0 2023-10-30T13:44:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.932+0000 7fb7cdd9b640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7b00664d0 0x7fb7b0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.936+0000 7fb7cdd9b640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7b00664d0 0x7fb7b0068990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb7d015bd70 tx=0x7fb7c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:45.936+0000 7fb7beffd640 1 -- 172.21.15.46:0/1889288557 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb7c00eec40 con 0x7fb7d006f7a0 2023-10-30T13:44:46.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.128+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1889288557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7d006caf0 con 0x7fb7d006f7a0 2023-10-30T13:44:46.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.128+0000 7fb7beffd640 1 -- 172.21.15.46:0/1889288557 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5728 (secure 0 0 0) 0x7fb7c00c0090 con 0x7fb7d006f7a0 2023-10-30T13:44:46.135 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:46.136 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:44:39.066162+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:46.136 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 104 2023-10-30T13:44:46.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.132+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7b00664d0 msgr2=0x7fb7b0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:46.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.132+0000 7fb7d4ed6640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7b00664d0 0x7fb7b0068990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb7d015bd70 tx=0x7fb7c4008040 comp rx=0 tx=0).stop 2023-10-30T13:44:46.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.132+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1889288557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d006f7a0 msgr2=0x7fb7d01eefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:46.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.132+0000 7fb7d4ed6640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d006f7a0 0x7fb7d01eefe0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00b3040 tx=0x7fb7c001bad0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.132+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1889288557 shutdown_connections 2023-10-30T13:44:46.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.132+0000 7fb7d4ed6640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7b00664d0 0x7fb7b0068990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.132+0000 7fb7d4ed6640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7d00c6bd0 0x7fb7d00c0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.132+0000 7fb7d4ed6640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7d0071110 0x7fb7d01f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.132+0000 7fb7d4ed6640 1 --2- 172.21.15.46:0/1889288557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d006f7a0 0x7fb7d01eefe0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.132+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1889288557 >> 172.21.15.46:0/1889288557 conn(0x7fb7d00c7910 msgr2=0x7fb7d00c7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.132+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1889288557 shutdown_connections 2023-10-30T13:44:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:46.132+0000 7fb7d4ed6640 1 -- 172.21.15.46:0/1889288557 wait complete. 2023-10-30T13:44:46.157 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:48.158 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:48.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa79246b640 1 Processor -- start 2023-10-30T13:44:48.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa79246b640 1 -- start start 2023-10-30T13:44:48.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa79246b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa78c1538a0 0x7fa78c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:48.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa79246b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa78c0ca0e0 0x7fa78c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:48.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa79246b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78c0cb800 con 0x7fa78c0cf160 2023-10-30T13:44:48.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa79246b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa78c15dc30 con 0x7fa78c0ca0e0 2023-10-30T13:44:48.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa79246b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa78c0cb330 con 0x7fa78c1538a0 2023-10-30T13:44:48.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa78b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa78c0ca0e0 0x7fa78c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:48.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa790a08640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa78c0cf160 0x7fa78c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47362/0 (socket says 172.21.15.46:47362) 2023-10-30T13:44:48.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa790a08640 1 -- 172.21.15.46:0/718383174 learned_addr learned my addr 172.21.15.46:0/718383174 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:44:48.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa78b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa78c0ca0e0 0x7fa78c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43632/0 (socket says 172.21.15.46:43632) 2023-10-30T13:44:48.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa78bfff640 1 --2- 172.21.15.46:0/718383174 >> v2:172.21.15.154:3300/0 conn(0x7fa78c1538a0 0x7fa78c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:48.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa78b7fe640 1 -- 172.21.15.46:0/718383174 >> v2:172.21.15.154:3300/0 conn(0x7fa78c1538a0 msgr2=0x7fa78c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:48.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa78b7fe640 1 --2- 172.21.15.46:0/718383174 >> v2:172.21.15.154:3300/0 conn(0x7fa78c1538a0 0x7fa78c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa78b7fe640 1 -- 172.21.15.46:0/718383174 >> v1:172.21.15.46:6789/0 conn(0x7fa78c0cf160 legacy=0x7fa78c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.308+0000 7fa78b7fe640 1 -- 172.21.15.46:0/718383174 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c0cad70 con 0x7fa78c0ca0e0 2023-10-30T13:44:48.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.312+0000 7fa78b7fe640 1 --2- 172.21.15.46:0/718383174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa78c0ca0e0 0x7fa78c15a090 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fa78c0cfd10 tx=0x7fa78001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b62dbeca11e47aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:48.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.312+0000 7fa78affd640 1 -- 172.21.15.46:0/718383174 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7800a8950 con 0x7fa78c0ca0e0 2023-10-30T13:44:48.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.312+0000 7fa78affd640 1 -- 172.21.15.46:0/718383174 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7800a8ab0 con 0x7fa78c0ca0e0 2023-10-30T13:44:48.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.312+0000 7fa78affd640 1 -- 172.21.15.46:0/718383174 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7800a8e40 con 0x7fa78c0ca0e0 2023-10-30T13:44:48.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.312+0000 7fa79246b640 1 -- 172.21.15.46:0/718383174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa78c0ca0e0 msgr2=0x7fa78c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.312+0000 7fa79246b640 1 --2- 172.21.15.46:0/718383174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa78c0ca0e0 0x7fa78c15a090 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fa78c0cfd10 tx=0x7fa78001ec50 comp rx=0 tx=0).stop 2023-10-30T13:44:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.312+0000 7fa79246b640 1 -- 172.21.15.46:0/718383174 shutdown_connections 2023-10-30T13:44:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.312+0000 7fa79246b640 1 --2- 172.21.15.46:0/718383174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa78c0ca0e0 0x7fa78c15a090 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.312+0000 7fa79246b640 1 --2- 172.21.15.46:0/718383174 >> v2:172.21.15.154:3300/0 conn(0x7fa78c1538a0 0x7fa78c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.312+0000 7fa79246b640 1 -- 172.21.15.46:0/718383174 >> 172.21.15.46:0/718383174 conn(0x7fa78c06bfe0 msgr2=0x7fa78c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.312+0000 7fa79246b640 1 -- 172.21.15.46:0/718383174 shutdown_connections 2023-10-30T13:44:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.312+0000 7fa79246b640 1 -- 172.21.15.46:0/718383174 wait complete. 2023-10-30T13:44:48.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa79246b640 1 Processor -- start 2023-10-30T13:44:48.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa79246b640 1 -- start start 2023-10-30T13:44:48.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa79246b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa78c0ca0e0 0x7fa78c0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:48.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa79246b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa78c0cf160 0x7fa78c142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:48.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa79246b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1538a0 0x7fa78c1ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:48.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa79246b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c1584d0 con 0x7fa78c0cf160 2023-10-30T13:44:48.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa79246b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c1ef390 con 0x7fa78c1538a0 2023-10-30T13:44:48.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa79246b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c0cec20 con 0x7fa78c0ca0e0 2023-10-30T13:44:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa78b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa78c0cf160 0x7fa78c142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa78b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa78c0cf160 0x7fa78c142550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43646/0 (socket says 172.21.15.46:43646) 2023-10-30T13:44:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa78b7fe640 1 -- 172.21.15.46:0/2396868103 learned_addr learned my addr 172.21.15.46:0/2396868103 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa78bfff640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa78c0ca0e0 0x7fa78c0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa790a08640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1538a0 0x7fa78c1ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa78b7fe640 1 -- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa78c0ca0e0 msgr2=0x7fa78c0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa78b7fe640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa78c0ca0e0 0x7fa78c0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa78b7fe640 1 -- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1538a0 msgr2=0x7fa78c1ef7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa78b7fe640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1538a0 0x7fa78c1ef7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa78b7fe640 1 -- 172.21.15.46:0/2396868103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa780009420 con 0x7fa78c0cf160 2023-10-30T13:44:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa78b7fe640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa78c0cf160 0x7fa78c142550 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fa7800b3040 tx=0x7fa78001b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa788ff9640 1 -- 172.21.15.46:0/2396868103 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7800b9630 con 0x7fa78c0cf160 2023-10-30T13:44:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa788ff9640 1 -- 172.21.15.46:0/2396868103 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7800b97b0 con 0x7fa78c0cf160 2023-10-30T13:44:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa788ff9640 1 -- 172.21.15.46:0/2396868103 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7800b9b60 con 0x7fa78c0cf160 2023-10-30T13:44:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa79246b640 1 -- 172.21.15.46:0/2396868103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78c0ceec0 con 0x7fa78c0cf160 2023-10-30T13:44:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.316+0000 7fa79246b640 1 -- 172.21.15.46:0/2396868103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78c149080 con 0x7fa78c0cf160 2023-10-30T13:44:48.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.320+0000 7fa79246b640 1 -- 172.21.15.46:0/2396868103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78c0b2a50 con 0x7fa78c0cf160 2023-10-30T13:44:48.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.320+0000 7fa788ff9640 1 -- 172.21.15.46:0/2396868103 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7800b7040 con 0x7fa78c0cf160 2023-10-30T13:44:48.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.320+0000 7fa788ff9640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7780664a0 0x7fa778068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:48.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.320+0000 7fa788ff9640 1 -- 172.21.15.46:0/2396868103 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fa780122310 con 0x7fa78c0cf160 2023-10-30T13:44:48.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.320+0000 7fa78bfff640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7780664a0 0x7fa778068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:48.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.320+0000 7fa78bfff640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7780664a0 0x7fa778068960 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa77c00b930 tx=0x7fa77c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:48.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.324+0000 7fa788ff9640 1 -- 172.21.15.46:0/2396868103 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7800ef350 con 0x7fa78c0cf160 2023-10-30T13:44:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.512+0000 7fa79246b640 1 -- 172.21.15.46:0/2396868103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa78c04e170 con 0x7fa78c0cf160 2023-10-30T13:44:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.512+0000 7fa788ff9640 1 -- 172.21.15.46:0/2396868103 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fa7800f2970 con 0x7fa78c0cf160 2023-10-30T13:44:48.520 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:48.520 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:44:46.321020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:resolve","state_seq":45,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:48.521 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 105 2023-10-30T13:44:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.516+0000 7fa79246b640 1 -- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7780664a0 msgr2=0x7fa778068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.516+0000 7fa79246b640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7780664a0 0x7fa778068960 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa77c00b930 tx=0x7fa77c008040 comp rx=0 tx=0).stop 2023-10-30T13:44:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.516+0000 7fa79246b640 1 -- 172.21.15.46:0/2396868103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa78c0cf160 msgr2=0x7fa78c142550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.516+0000 7fa79246b640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa78c0cf160 0x7fa78c142550 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fa7800b3040 tx=0x7fa78001b9f0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.516+0000 7fa79246b640 1 -- 172.21.15.46:0/2396868103 shutdown_connections 2023-10-30T13:44:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.516+0000 7fa79246b640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7780664a0 0x7fa778068960 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.516+0000 7fa79246b640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1538a0 0x7fa78c1ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.516+0000 7fa79246b640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa78c0cf160 0x7fa78c142550 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.516+0000 7fa79246b640 1 --2- 172.21.15.46:0/2396868103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa78c0ca0e0 0x7fa78c0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.516+0000 7fa79246b640 1 -- 172.21.15.46:0/2396868103 >> 172.21.15.46:0/2396868103 conn(0x7fa78c06bfe0 msgr2=0x7fa78c152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:48.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.516+0000 7fa79246b640 1 -- 172.21.15.46:0/2396868103 shutdown_connections 2023-10-30T13:44:48.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:48.516+0000 7fa79246b640 1 -- 172.21.15.46:0/2396868103 wait complete. 2023-10-30T13:44:48.541 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:49.043 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:44:49.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:44:50.542 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:50.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.692+0000 7fd578ac9640 1 Processor -- start 2023-10-30T13:44:50.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.692+0000 7fd578ac9640 1 -- start start 2023-10-30T13:44:50.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.692+0000 7fd578ac9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd57406f7a0 0x7fd57406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.692+0000 7fd578ac9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd574066cf0 con 0x7fd574071110 2023-10-30T13:44:50.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.692+0000 7fd578ac9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5740c70d0 con 0x7fd5740c6bd0 2023-10-30T13:44:50.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.692+0000 7fd578ac9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5740700c0 con 0x7fd57406f7a0 2023-10-30T13:44:50.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.692+0000 7fd57259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd57406f7a0 0x7fd57406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.692+0000 7fd57259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd57406f7a0 0x7fd57406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43662/0 (socket says 172.21.15.46:43662) 2023-10-30T13:44:50.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.692+0000 7fd57259c640 1 -- 172.21.15.46:0/2642491590 learned_addr learned my addr 172.21.15.46:0/2642491590 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:50.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.692+0000 7fd57259c640 1 -- 172.21.15.46:0/2642491590 >> v1:172.21.15.154:6789/0 conn(0x7fd5740c6bd0 legacy=0x7fd574069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.692+0000 7fd57259c640 1 -- 172.21.15.46:0/2642491590 >> v1:172.21.15.46:6789/0 conn(0x7fd574071110 legacy=0x7fd574158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.692+0000 7fd57259c640 1 -- 172.21.15.46:0/2642491590 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5740c7390 con 0x7fd57406f7a0 2023-10-30T13:44:50.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.696+0000 7fd57259c640 1 --2- 172.21.15.46:0/2642491590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd57406f7a0 0x7fd57406fb80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd564009870 tx=0x7fd56401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3261b9c692513f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:50.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.696+0000 7fd571d9b640 1 -- 172.21.15.46:0/2642491590 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5640a8a30 con 0x7fd57406f7a0 2023-10-30T13:44:50.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.696+0000 7fd571d9b640 1 -- 172.21.15.46:0/2642491590 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5640a8b90 con 0x7fd57406f7a0 2023-10-30T13:44:50.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.696+0000 7fd571d9b640 1 -- 172.21.15.46:0/2642491590 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5640b0440 con 0x7fd57406f7a0 2023-10-30T13:44:50.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.696+0000 7fd578ac9640 1 -- 172.21.15.46:0/2642491590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd57406f7a0 msgr2=0x7fd57406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.696+0000 7fd578ac9640 1 --2- 172.21.15.46:0/2642491590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd57406f7a0 0x7fd57406fb80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd564009870 tx=0x7fd56401ed30 comp rx=0 tx=0).stop 2023-10-30T13:44:50.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.696+0000 7fd578ac9640 1 -- 172.21.15.46:0/2642491590 shutdown_connections 2023-10-30T13:44:50.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.696+0000 7fd578ac9640 1 --2- 172.21.15.46:0/2642491590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd57406f7a0 0x7fd57406fb80 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.696+0000 7fd578ac9640 1 -- 172.21.15.46:0/2642491590 >> 172.21.15.46:0/2642491590 conn(0x7fd5740c7910 msgr2=0x7fd5740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:50.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.696+0000 7fd578ac9640 1 -- 172.21.15.46:0/2642491590 shutdown_connections 2023-10-30T13:44:50.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.696+0000 7fd578ac9640 1 -- 172.21.15.46:0/2642491590 wait complete. 2023-10-30T13:44:50.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd578ac9640 1 Processor -- start 2023-10-30T13:44:50.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd578ac9640 1 -- start start 2023-10-30T13:44:50.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd578ac9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd57406f7a0 0x7fd5741464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd578ac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd574071110 0x7fd5741e2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd578ac9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5740c6bd0 0x7fd574140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd578ac9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5741f2620 con 0x7fd57406f7a0 2023-10-30T13:44:50.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd578ac9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd574153ee0 con 0x7fd5740c6bd0 2023-10-30T13:44:50.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd578ac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd574154150 con 0x7fd574071110 2023-10-30T13:44:50.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd57259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd57406f7a0 0x7fd5741464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd57259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd57406f7a0 0x7fd5741464b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43666/0 (socket says 172.21.15.46:43666) 2023-10-30T13:44:50.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd57259c640 1 -- 172.21.15.46:0/2132021079 learned_addr learned my addr 172.21.15.46:0/2132021079 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:50.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd56bfff640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd574071110 0x7fd5741e2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd572d9d640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5740c6bd0 0x7fd574140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd57259c640 1 -- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd574071110 msgr2=0x7fd5741e2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd57259c640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd574071110 0x7fd5741e2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd57259c640 1 -- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5740c6bd0 msgr2=0x7fd574140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd57259c640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5740c6bd0 0x7fd574140af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd57259c640 1 -- 172.21.15.46:0/2132021079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5640094d0 con 0x7fd57406f7a0 2023-10-30T13:44:50.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd57259c640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd57406f7a0 0x7fd5741464b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd5640b3040 tx=0x7fd56401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:50.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd56affd640 1 -- 172.21.15.46:0/2132021079 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5640be070 con 0x7fd57406f7a0 2023-10-30T13:44:50.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd56affd640 1 -- 172.21.15.46:0/2132021079 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5640ba400 con 0x7fd57406f7a0 2023-10-30T13:44:50.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd56affd640 1 -- 172.21.15.46:0/2132021079 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5640ba7b0 con 0x7fd57406f7a0 2023-10-30T13:44:50.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd578ac9640 1 -- 172.21.15.46:0/2132021079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd574154430 con 0x7fd57406f7a0 2023-10-30T13:44:50.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.700+0000 7fd578ac9640 1 -- 172.21.15.46:0/2132021079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5741e3570 con 0x7fd57406f7a0 2023-10-30T13:44:50.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.704+0000 7fd578ac9640 1 -- 172.21.15.46:0/2132021079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd574154290 con 0x7fd57406f7a0 2023-10-30T13:44:50.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.704+0000 7fd56affd640 1 -- 172.21.15.46:0/2132021079 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd5640ba910 con 0x7fd57406f7a0 2023-10-30T13:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.704+0000 7fd56affd640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5540664a0 0x7fd554068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.704+0000 7fd56affd640 1 -- 172.21.15.46:0/2132021079 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fd564122170 con 0x7fd57406f7a0 2023-10-30T13:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.704+0000 7fd56bfff640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5540664a0 0x7fd554068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.704+0000 7fd56bfff640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5540664a0 0x7fd554068960 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd574146f10 tx=0x7fd55c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:50.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.708+0000 7fd56affd640 1 -- 172.21.15.46:0/2132021079 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd574154290 con 0x7fd57406f7a0 2023-10-30T13:44:50.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.900+0000 7fd578ac9640 1 -- 172.21.15.46:0/2132021079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd574154290 con 0x7fd57406f7a0 2023-10-30T13:44:50.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.900+0000 7fd56affd640 1 -- 172.21.15.46:0/2132021079 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fd574154290 con 0x7fd57406f7a0 2023-10-30T13:44:50.907 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:50.907 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:44:46.321020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:resolve","state_seq":45,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:50.908 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 105 2023-10-30T13:44:50.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.904+0000 7fd578ac9640 1 -- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5540664a0 msgr2=0x7fd554068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.904+0000 7fd578ac9640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5540664a0 0x7fd554068960 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd574146f10 tx=0x7fd55c009040 comp rx=0 tx=0).stop 2023-10-30T13:44:50.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.904+0000 7fd578ac9640 1 -- 172.21.15.46:0/2132021079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd57406f7a0 msgr2=0x7fd5741464b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.904+0000 7fd578ac9640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd57406f7a0 0x7fd5741464b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd5640b3040 tx=0x7fd56401bad0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.904+0000 7fd578ac9640 1 -- 172.21.15.46:0/2132021079 shutdown_connections 2023-10-30T13:44:50.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.904+0000 7fd578ac9640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5540664a0 0x7fd554068960 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.904+0000 7fd578ac9640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5740c6bd0 0x7fd574140af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.904+0000 7fd578ac9640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd574071110 0x7fd5741e2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.904+0000 7fd578ac9640 1 --2- 172.21.15.46:0/2132021079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd57406f7a0 0x7fd5741464b0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.904+0000 7fd578ac9640 1 -- 172.21.15.46:0/2132021079 >> 172.21.15.46:0/2132021079 conn(0x7fd5740c7910 msgr2=0x7fd5740c6380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:50.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.904+0000 7fd578ac9640 1 -- 172.21.15.46:0/2132021079 shutdown_connections 2023-10-30T13:44:50.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:50.904+0000 7fd578ac9640 1 -- 172.21.15.46:0/2132021079 wait complete. 2023-10-30T13:44:50.929 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:52.930 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.076+0000 7fc353bf5640 1 Processor -- start 2023-10-30T13:44:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.076+0000 7fc353bf5640 1 -- start start 2023-10-30T13:44:53.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc353bf5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc34c1580e0 0x7fc34c070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc353bf5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc34c15e700 0x7fc34c06a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc353bf5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc34c0c5810 con 0x7fc34c0c6540 2023-10-30T13:44:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc353bf5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc34c15eb90 con 0x7fc34c15e700 2023-10-30T13:44:53.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc353bf5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c0c6960 con 0x7fc34c1580e0 2023-10-30T13:44:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc351190640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc34c0c6540 0x7fc34c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47376/0 (socket says 172.21.15.46:47376) 2023-10-30T13:44:53.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc351190640 1 -- 172.21.15.46:0/3828813443 learned_addr learned my addr 172.21.15.46:0/3828813443 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:44:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc352192640 1 --2- 172.21.15.46:0/3828813443 >> v2:172.21.15.46:3300/0 conn(0x7fc34c15e700 0x7fc34c06a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc351991640 1 --2- 172.21.15.46:0/3828813443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc34c1580e0 0x7fc34c070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc35098f640 1 -- 172.21.15.46:0/3828813443 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924377150 0 0) 0x7fc34c0c5810 con 0x7fc34c0c6540 2023-10-30T13:44:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc352192640 1 -- 172.21.15.46:0/3828813443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc34c1580e0 msgr2=0x7fc34c070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:53.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc352192640 1 --2- 172.21.15.46:0/3828813443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc34c1580e0 0x7fc34c070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc352192640 1 -- 172.21.15.46:0/3828813443 >> v1:172.21.15.46:6789/0 conn(0x7fc34c0c6540 legacy=0x7fc34c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.080+0000 7fc352192640 1 -- 172.21.15.46:0/3828813443 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34c0c6bb0 con 0x7fc34c15e700 2023-10-30T13:44:53.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.084+0000 7fc352192640 1 --2- 172.21.15.46:0/3828813443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc34c15e700 0x7fc34c06a5a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc344009d90 tx=0x7fc34401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a9242d4869286d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.084+0000 7fc35098f640 1 -- 172.21.15.46:0/3828813443 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3440a59e0 con 0x7fc34c15e700 2023-10-30T13:44:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.084+0000 7fc35098f640 1 -- 172.21.15.46:0/3828813443 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3440a5b40 con 0x7fc34c15e700 2023-10-30T13:44:53.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.084+0000 7fc35098f640 1 -- 172.21.15.46:0/3828813443 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3440ad440 con 0x7fc34c15e700 2023-10-30T13:44:53.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.084+0000 7fc353bf5640 1 -- 172.21.15.46:0/3828813443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc34c15e700 msgr2=0x7fc34c06a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.084+0000 7fc353bf5640 1 --2- 172.21.15.46:0/3828813443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc34c15e700 0x7fc34c06a5a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc344009d90 tx=0x7fc34401c920 comp rx=0 tx=0).stop 2023-10-30T13:44:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.084+0000 7fc353bf5640 1 -- 172.21.15.46:0/3828813443 shutdown_connections 2023-10-30T13:44:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.084+0000 7fc353bf5640 1 --2- 172.21.15.46:0/3828813443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc34c15e700 0x7fc34c06a5a0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.084+0000 7fc353bf5640 1 --2- 172.21.15.46:0/3828813443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc34c1580e0 0x7fc34c070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.084+0000 7fc353bf5640 1 -- 172.21.15.46:0/3828813443 >> 172.21.15.46:0/3828813443 conn(0x7fc34c0c7910 msgr2=0x7fc34c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.084+0000 7fc353bf5640 1 -- 172.21.15.46:0/3828813443 shutdown_connections 2023-10-30T13:44:53.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.084+0000 7fc353bf5640 1 -- 172.21.15.46:0/3828813443 wait complete. 2023-10-30T13:44:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc353bf5640 1 Processor -- start 2023-10-30T13:44:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc353bf5640 1 -- start start 2023-10-30T13:44:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc353bf5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc34c0c6540 0x7fc34c0c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc353bf5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c1580e0 0x7fc34c1f82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:53.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc353bf5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc34c15e700 0x7fc34c1f87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc353bf5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c160b90 con 0x7fc34c15e700 2023-10-30T13:44:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc353bf5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c1619f0 con 0x7fc34c1580e0 2023-10-30T13:44:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc353bf5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c15f730 con 0x7fc34c0c6540 2023-10-30T13:44:53.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc352192640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc34c15e700 0x7fc34c1f87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc352192640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc34c15e700 0x7fc34c1f87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43684/0 (socket says 172.21.15.46:43684) 2023-10-30T13:44:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc352192640 1 -- 172.21.15.46:0/1908274709 learned_addr learned my addr 172.21.15.46:0/1908274709 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc351991640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc34c0c6540 0x7fc34c0c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc351190640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c1580e0 0x7fc34c1f82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc352192640 1 -- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc34c0c6540 msgr2=0x7fc34c0c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:53.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc352192640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc34c0c6540 0x7fc34c0c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:53.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc352192640 1 -- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c1580e0 msgr2=0x7fc34c1f82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:53.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc352192640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c1580e0 0x7fc34c1f82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:53.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc352192640 1 -- 172.21.15.46:0/1908274709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc344009a70 con 0x7fc34c15e700 2023-10-30T13:44:53.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc352192640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc34c15e700 0x7fc34c1f87f0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc344009d90 tx=0x7fc3440196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:53.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc3427fc640 1 -- 172.21.15.46:0/1908274709 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344009450 con 0x7fc34c15e700 2023-10-30T13:44:53.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc3427fc640 1 -- 172.21.15.46:0/1908274709 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3440adc60 con 0x7fc34c15e700 2023-10-30T13:44:53.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc353bf5640 1 -- 172.21.15.46:0/1908274709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc34c1f7210 con 0x7fc34c15e700 2023-10-30T13:44:53.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc3427fc640 1 -- 172.21.15.46:0/1908274709 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3440b8420 con 0x7fc34c15e700 2023-10-30T13:44:53.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.088+0000 7fc353bf5640 1 -- 172.21.15.46:0/1908274709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34c1f77d0 con 0x7fc34c15e700 2023-10-30T13:44:53.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.092+0000 7fc353bf5640 1 -- 172.21.15.46:0/1908274709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc34c066c80 con 0x7fc34c15e700 2023-10-30T13:44:53.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.092+0000 7fc3427fc640 1 -- 172.21.15.46:0/1908274709 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3440b8580 con 0x7fc34c15e700 2023-10-30T13:44:53.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.092+0000 7fc3427fc640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3300664a0 0x7fc330068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:53.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.092+0000 7fc3427fc640 1 -- 172.21.15.46:0/1908274709 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fc34411fd00 con 0x7fc34c15e700 2023-10-30T13:44:53.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.092+0000 7fc351991640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3300664a0 0x7fc330068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:53.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.092+0000 7fc351991640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3300664a0 0x7fc330068960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc348007a60 tx=0x7fc348008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:53.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.100+0000 7fc3427fc640 1 -- 172.21.15.46:0/1908274709 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc3440ece00 con 0x7fc34c15e700 2023-10-30T13:44:53.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.296+0000 7fc353bf5640 1 -- 172.21.15.46:0/1908274709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc34c04e170 con 0x7fc34c15e700 2023-10-30T13:44:53.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.296+0000 7fc3427fc640 1 -- 172.21.15.46:0/1908274709 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fc3440c6090 con 0x7fc34c15e700 2023-10-30T13:44:53.304 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:53.305 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:44:52.327152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:reconnect","state_seq":47,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:53.306 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 106 2023-10-30T13:44:53.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.300+0000 7fc353bf5640 1 -- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3300664a0 msgr2=0x7fc330068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:53.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.300+0000 7fc353bf5640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3300664a0 0x7fc330068960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc348007a60 tx=0x7fc348008040 comp rx=0 tx=0).stop 2023-10-30T13:44:53.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.300+0000 7fc353bf5640 1 -- 172.21.15.46:0/1908274709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc34c15e700 msgr2=0x7fc34c1f87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:53.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.300+0000 7fc353bf5640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc34c15e700 0x7fc34c1f87f0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc344009d90 tx=0x7fc3440196c0 comp rx=0 tx=0).stop 2023-10-30T13:44:53.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.300+0000 7fc353bf5640 1 -- 172.21.15.46:0/1908274709 shutdown_connections 2023-10-30T13:44:53.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.300+0000 7fc353bf5640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3300664a0 0x7fc330068960 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:53.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.300+0000 7fc353bf5640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc34c15e700 0x7fc34c1f87f0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:53.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.300+0000 7fc353bf5640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c1580e0 0x7fc34c1f82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:53.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.300+0000 7fc353bf5640 1 --2- 172.21.15.46:0/1908274709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc34c0c6540 0x7fc34c0c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:53.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.300+0000 7fc353bf5640 1 -- 172.21.15.46:0/1908274709 >> 172.21.15.46:0/1908274709 conn(0x7fc34c0c7910 msgr2=0x7fc34c155740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:53.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.300+0000 7fc353bf5640 1 -- 172.21.15.46:0/1908274709 shutdown_connections 2023-10-30T13:44:53.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:53.300+0000 7fc353bf5640 1 -- 172.21.15.46:0/1908274709 wait complete. 2023-10-30T13:44:53.325 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 106, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 106, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6201": { "addr": "172.21.15.46:6837/2546436963", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2546436963, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2546436963, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 6201, "incarnation": 90, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 35 }, "gid_6366": { "addr": "172.21.15.46:6833/190134548", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 190134548, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 190134548, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6366, "incarnation": 104, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:reconnect", "state_seq": 47 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:44:52.327152+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6201, "mds_1": 6366 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:44:55.326 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:55.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97d7a54640 1 Processor -- start 2023-10-30T13:44:55.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97d7a54640 1 -- start start 2023-10-30T13:44:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97d7a54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d00c6f60 0x7f97d01566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97d7a54640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d00c0a50 con 0x7f97d00c6b80 2023-10-30T13:44:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97d7a54640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d006f6e0 con 0x7f97d00c05d0 2023-10-30T13:44:55.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97d7a54640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d006f9c0 con 0x7f97d00c6f60 2023-10-30T13:44:55.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97d57f0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f97d00c6b80 0x7f97d00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47392/0 (socket says 172.21.15.46:47392) 2023-10-30T13:44:55.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97d57f0640 1 -- 172.21.15.46:0/3673363485 learned_addr learned my addr 172.21.15.46:0/3673363485 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:44:55.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97d5ff1640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d00c6f60 0x7f97d01566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506748830 0 0) 0x7f97d00c0a50 con 0x7f97d00c6b80 2023-10-30T13:44:55.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97bc003530 con 0x7f97d00c6b80 2023-10-30T13:44:55.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 46091586 0 0) 0x7f97bc003530 con 0x7f97d00c6b80 2023-10-30T13:44:55.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d00c6f60 msgr2=0x7f97d01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:55.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97c7fff640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d00c6f60 0x7f97d01566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.472+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 >> v1:172.21.15.154:6789/0 conn(0x7f97d00c05d0 legacy=0x7f97d013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d006fca0 con 0x7f97d00c6b80 2023-10-30T13:44:55.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f97cc001680 con 0x7f97d00c6b80 2023-10-30T13:44:55.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 >> v1:172.21.15.46:6789/0 conn(0x7f97d00c6b80 legacy=0x7f97d00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97c7fff640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97bc0039b0 0x7f97bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97c7fff640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d00c6f60 0x7f97bc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97c7fff640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97bc004d10 0x7f97bc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d006f9c0 con 0x7f97bc0039b0 2023-10-30T13:44:55.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc00dba0 con 0x7f97bc004d10 2023-10-30T13:44:55.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc00dd60 con 0x7f97d00c6f60 2023-10-30T13:44:55.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97d57f0640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97bc0039b0 0x7f97bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97d4fef640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d00c6f60 0x7f97bc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97d5ff1640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97bc004d10 0x7f97bc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97d57f0640 1 -- 172.21.15.46:0/3673363485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d00c6f60 msgr2=0x7f97bc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:55.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97d57f0640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d00c6f60 0x7f97bc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97d57f0640 1 -- 172.21.15.46:0/3673363485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97bc004d10 msgr2=0x7f97bc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:55.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97d57f0640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97bc004d10 0x7f97bc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97d57f0640 1 -- 172.21.15.46:0/3673363485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc003530 con 0x7f97bc0039b0 2023-10-30T13:44:55.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97d4fef640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d00c6f60 0x7f97bc0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:55.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97d57f0640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97bc0039b0 0x7f97bc003d90 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00c210 tx=0x7f97cc01cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b59df6ccd4f705be server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:55.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc005ae0 con 0x7f97bc0039b0 2023-10-30T13:44:55.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97cc005c60 con 0x7f97bc0039b0 2023-10-30T13:44:55.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.476+0000 7f97c7fff640 1 -- 172.21.15.46:0/3673363485 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc0a5790 con 0x7f97bc0039b0 2023-10-30T13:44:55.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 -- 172.21.15.46:0/3673363485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97bc0039b0 msgr2=0x7f97bc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 --2- 172.21.15.46:0/3673363485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97bc0039b0 0x7f97bc003d90 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00c210 tx=0x7f97cc01cba0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d5ff1640 1 -- 172.21.15.46:0/3673363485 reap_dead start 2023-10-30T13:44:55.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 -- 172.21.15.46:0/3673363485 shutdown_connections 2023-10-30T13:44:55.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 -- 172.21.15.46:0/3673363485 >> 172.21.15.46:0/3673363485 conn(0x7f97d00c7910 msgr2=0x7f97d00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:55.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 -- 172.21.15.46:0/3673363485 shutdown_connections 2023-10-30T13:44:55.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 -- 172.21.15.46:0/3673363485 wait complete. 2023-10-30T13:44:55.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 Processor -- start 2023-10-30T13:44:55.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 -- start start 2023-10-30T13:44:55.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97bc0039b0 0x7f97d01f5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d0130c40 0x7f97d013dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97d00c7cf0 0x7f97d012f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00c17a0 con 0x7f97d00c7cf0 2023-10-30T13:44:55.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d006ad00 con 0x7f97bc0039b0 2023-10-30T13:44:55.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d7a54640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d01307b0 con 0x7f97d0130c40 2023-10-30T13:44:55.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d5ff1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97d00c7cf0 0x7f97d012f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d5ff1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97d00c7cf0 0x7f97d012f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43700/0 (socket says 172.21.15.46:43700) 2023-10-30T13:44:55.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d5ff1640 1 -- 172.21.15.46:0/1145059649 learned_addr learned my addr 172.21.15.46:0/1145059649 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:55.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d4fef640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d0130c40 0x7f97d013dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.480+0000 7f97d57f0640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97bc0039b0 0x7f97d01f5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97d5ff1640 1 -- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d0130c40 msgr2=0x7f97d013dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97d5ff1640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d0130c40 0x7f97d013dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97d5ff1640 1 -- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97bc0039b0 msgr2=0x7f97d01f5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97d5ff1640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97bc0039b0 0x7f97d01f5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97d5ff1640 1 -- 172.21.15.46:0/1145059649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d006a1c0 con 0x7f97d00c7cf0 2023-10-30T13:44:55.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97d4fef640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d0130c40 0x7f97d013dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:55.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97d5ff1640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97d00c7cf0 0x7f97d012f590 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f97d0159b90 tx=0x7f97c800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:55.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97c5ffb640 1 -- 172.21.15.46:0/1145059649 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c800e040 con 0x7f97d00c7cf0 2023-10-30T13:44:55.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97c5ffb640 1 -- 172.21.15.46:0/1145059649 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c800f040 con 0x7f97d00c7cf0 2023-10-30T13:44:55.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97c5ffb640 1 -- 172.21.15.46:0/1145059649 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c8015430 con 0x7f97d00c7cf0 2023-10-30T13:44:55.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97d7a54640 1 -- 172.21.15.46:0/1145059649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d006a460 con 0x7f97d00c7cf0 2023-10-30T13:44:55.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97d7a54640 1 -- 172.21.15.46:0/1145059649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d006aa20 con 0x7f97d00c7cf0 2023-10-30T13:44:55.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97d7a54640 1 -- 172.21.15.46:0/1145059649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d0130960 con 0x7f97d00c7cf0 2023-10-30T13:44:55.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.484+0000 7f97c5ffb640 1 -- 172.21.15.46:0/1145059649 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f97c8015590 con 0x7f97d00c7cf0 2023-10-30T13:44:55.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.488+0000 7f97c5ffb640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97b40664d0 0x7f97b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.488+0000 7f97c5ffb640 1 -- 172.21.15.46:0/1145059649 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f97c80848f0 con 0x7f97d00c7cf0 2023-10-30T13:44:55.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.488+0000 7f97d57f0640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97b40664d0 0x7f97b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.488+0000 7f97d57f0640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97b40664d0 0x7f97b4068990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f97cc0ab000 tx=0x7f97cc0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:55.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.492+0000 7f97c5ffb640 1 -- 172.21.15.46:0/1145059649 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97d0130960 con 0x7f97d00c7cf0 2023-10-30T13:44:55.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 -- 172.21.15.46:0/1145059649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97d004e170 con 0x7f97d00c7cf0 2023-10-30T13:44:55.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97c5ffb640 1 -- 172.21.15.46:0/1145059649 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f97c8051930 con 0x7f97d00c7cf0 2023-10-30T13:44:55.689 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:55.689 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:44:52.327152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:reconnect","state_seq":47,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:55.690 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 106 2023-10-30T13:44:55.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 -- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97b40664d0 msgr2=0x7f97b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97b40664d0 0x7f97b4068990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f97cc0ab000 tx=0x7f97cc0a7070 comp rx=0 tx=0).stop 2023-10-30T13:44:55.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 -- 172.21.15.46:0/1145059649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97d00c7cf0 msgr2=0x7f97d012f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97d00c7cf0 0x7f97d012f590 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f97d0159b90 tx=0x7f97c800f580 comp rx=0 tx=0).stop 2023-10-30T13:44:55.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 -- 172.21.15.46:0/1145059649 shutdown_connections 2023-10-30T13:44:55.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97b40664d0 0x7f97b4068990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97d00c7cf0 0x7f97d012f590 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97d0130c40 0x7f97d013dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 --2- 172.21.15.46:0/1145059649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97bc0039b0 0x7f97d01f5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 -- 172.21.15.46:0/1145059649 >> 172.21.15.46:0/1145059649 conn(0x7f97d00c7910 msgr2=0x7f97d0136340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:55.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 -- 172.21.15.46:0/1145059649 shutdown_connections 2023-10-30T13:44:55.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:55.684+0000 7f97d7a54640 1 -- 172.21.15.46:0/1145059649 wait complete. 2023-10-30T13:44:55.710 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:57.711 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:57.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.860+0000 7ff890c46640 1 Processor -- start 2023-10-30T13:44:57.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.860+0000 7ff890c46640 1 -- start start 2023-10-30T13:44:57.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.860+0000 7ff890c46640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff88c070a00 0x7ff88c070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.860+0000 7ff890c46640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff88c069e90 0x7ff88c06a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.860+0000 7ff890c46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88c0713b0 con 0x7ff88c1539a0 2023-10-30T13:44:57.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.860+0000 7ff890c46640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c06fbb0 con 0x7ff88c070a00 2023-10-30T13:44:57.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.860+0000 7ff890c46640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c153d80 con 0x7ff88c069e90 2023-10-30T13:44:57.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.860+0000 7ff883fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff88c070a00 0x7ff88c070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.860+0000 7ff88a59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff88c1539a0 0x7ff88c15a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49516/0 (socket says 172.21.15.46:49516) 2023-10-30T13:44:57.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.860+0000 7ff88a59c640 1 -- 172.21.15.46:0/4217438874 learned_addr learned my addr 172.21.15.46:0/4217438874 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:44:57.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff88ad9d640 1 --2- 172.21.15.46:0/4217438874 >> v2:172.21.15.154:3300/0 conn(0x7ff88c069e90 0x7ff88c06a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff883fff640 1 -- 172.21.15.46:0/4217438874 >> v2:172.21.15.154:3300/0 conn(0x7ff88c069e90 msgr2=0x7ff88c06a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:57.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff883fff640 1 --2- 172.21.15.46:0/4217438874 >> v2:172.21.15.154:3300/0 conn(0x7ff88c069e90 0x7ff88c06a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff883fff640 1 -- 172.21.15.46:0/4217438874 >> v1:172.21.15.46:6789/0 conn(0x7ff88c1539a0 legacy=0x7ff88c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:57.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff883fff640 1 -- 172.21.15.46:0/4217438874 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c0c5c50 con 0x7ff88c070a00 2023-10-30T13:44:57.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff889d9b640 1 -- 172.21.15.46:0/4217438874 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439790351 0 0) 0x7ff88c0713b0 con 0x7ff88c1539a0 2023-10-30T13:44:57.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff883fff640 1 --2- 172.21.15.46:0/4217438874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c070a00 0x7ff88c070e70 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0b1080 tx=0x7ff87401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f7ed08d5f9f11e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:57.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff889d9b640 1 -- 172.21.15.46:0/4217438874 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8740ab600 con 0x7ff88c070a00 2023-10-30T13:44:57.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff889d9b640 1 -- 172.21.15.46:0/4217438874 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8740a7050 con 0x7ff88c070a00 2023-10-30T13:44:57.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff889d9b640 1 -- 172.21.15.46:0/4217438874 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8740ab980 con 0x7ff88c070a00 2023-10-30T13:44:57.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff890c46640 1 -- 172.21.15.46:0/4217438874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c070a00 msgr2=0x7ff88c070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:57.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff890c46640 1 --2- 172.21.15.46:0/4217438874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c070a00 0x7ff88c070e70 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0b1080 tx=0x7ff87401c3a0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff890c46640 1 -- 172.21.15.46:0/4217438874 shutdown_connections 2023-10-30T13:44:57.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff890c46640 1 --2- 172.21.15.46:0/4217438874 >> v2:172.21.15.154:3300/0 conn(0x7ff88c069e90 0x7ff88c06a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff890c46640 1 --2- 172.21.15.46:0/4217438874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c070a00 0x7ff88c070e70 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.864+0000 7ff890c46640 1 -- 172.21.15.46:0/4217438874 >> 172.21.15.46:0/4217438874 conn(0x7ff88c0c7910 msgr2=0x7ff88c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:57.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff890c46640 1 -- 172.21.15.46:0/4217438874 shutdown_connections 2023-10-30T13:44:57.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff890c46640 1 -- 172.21.15.46:0/4217438874 wait complete. 2023-10-30T13:44:57.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff890c46640 1 Processor -- start 2023-10-30T13:44:57.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff890c46640 1 -- start start 2023-10-30T13:44:57.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff890c46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c069e90 0x7ff88c0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff890c46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c070a00 0x7ff88c1f5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff890c46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c1539a0 0x7ff88c0c0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff890c46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c1f6810 con 0x7ff88c070a00 2023-10-30T13:44:57.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff890c46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c1f5f70 con 0x7ff88c069e90 2023-10-30T13:44:57.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff890c46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c1f6250 con 0x7ff88c1539a0 2023-10-30T13:44:57.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff883fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c070a00 0x7ff88c1f5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff883fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c070a00 0x7ff88c1f5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41794/0 (socket says 172.21.15.46:41794) 2023-10-30T13:44:57.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff883fff640 1 -- 172.21.15.46:0/1964882260 learned_addr learned my addr 172.21.15.46:0/1964882260 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:44:57.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff88a59c640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c069e90 0x7ff88c0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff88ad9d640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c1539a0 0x7ff88c0c0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff883fff640 1 -- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c1539a0 msgr2=0x7ff88c0c0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:57.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff883fff640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c1539a0 0x7ff88c0c0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff883fff640 1 -- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c069e90 msgr2=0x7ff88c0c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:57.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff883fff640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c069e90 0x7ff88c0c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff883fff640 1 -- 172.21.15.46:0/1964882260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff874009420 con 0x7ff88c070a00 2023-10-30T13:44:57.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff88ad9d640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c1539a0 0x7ff88c0c0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:57.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.868+0000 7ff883fff640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c070a00 0x7ff88c1f5930 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7ff8740ae040 tx=0x7ff874019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.872+0000 7ff882ffd640 1 -- 172.21.15.46:0/1964882260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8740b4430 con 0x7ff88c070a00 2023-10-30T13:44:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.872+0000 7ff882ffd640 1 -- 172.21.15.46:0/1964882260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8740b45b0 con 0x7ff88c070a00 2023-10-30T13:44:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.872+0000 7ff882ffd640 1 -- 172.21.15.46:0/1964882260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8740b4960 con 0x7ff88c070a00 2023-10-30T13:44:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.872+0000 7ff890c46640 1 -- 172.21.15.46:0/1964882260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88c1f6530 con 0x7ff88c070a00 2023-10-30T13:44:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.872+0000 7ff890c46640 1 -- 172.21.15.46:0/1964882260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88c0c0eb0 con 0x7ff88c070a00 2023-10-30T13:44:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.872+0000 7ff890c46640 1 -- 172.21.15.46:0/1964882260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88c0c10a0 con 0x7ff88c070a00 2023-10-30T13:44:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.872+0000 7ff882ffd640 1 -- 172.21.15.46:0/1964882260 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff8740b2040 con 0x7ff88c070a00 2023-10-30T13:44:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.872+0000 7ff882ffd640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff86c0664a0 0x7ff86c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.872+0000 7ff882ffd640 1 -- 172.21.15.46:0/1964882260 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7ff87411e230 con 0x7ff88c070a00 2023-10-30T13:44:57.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.872+0000 7ff88a59c640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff86c0664a0 0x7ff86c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.876+0000 7ff88a59c640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff86c0664a0 0x7ff86c068960 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff87c000f70 tx=0x7ff87c007df0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:57.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:57.876+0000 7ff882ffd640 1 -- 172.21.15.46:0/1964882260 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff8740eb1f0 con 0x7ff88c070a00 2023-10-30T13:44:58.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.072+0000 7ff890c46640 1 -- 172.21.15.46:0/1964882260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff88c04e170 con 0x7ff88c070a00 2023-10-30T13:44:58.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.072+0000 7ff882ffd640 1 -- 172.21.15.46:0/1964882260 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5732 (secure 0 0 0) 0x7ff8740ee810 con 0x7ff88c070a00 2023-10-30T13:44:58.078 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:44:58.078 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:44:52.327152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:reconnect","state_seq":47,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:58.079 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 106 2023-10-30T13:44:58.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.072+0000 7ff890c46640 1 -- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff86c0664a0 msgr2=0x7ff86c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:58.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.072+0000 7ff890c46640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff86c0664a0 0x7ff86c068960 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff87c000f70 tx=0x7ff87c007df0 comp rx=0 tx=0).stop 2023-10-30T13:44:58.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.076+0000 7ff890c46640 1 -- 172.21.15.46:0/1964882260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c070a00 msgr2=0x7ff88c1f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:58.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.076+0000 7ff890c46640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c070a00 0x7ff88c1f5930 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7ff8740ae040 tx=0x7ff874019140 comp rx=0 tx=0).stop 2023-10-30T13:44:58.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.076+0000 7ff890c46640 1 -- 172.21.15.46:0/1964882260 shutdown_connections 2023-10-30T13:44:58.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.076+0000 7ff890c46640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff86c0664a0 0x7ff86c068960 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:58.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.076+0000 7ff890c46640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c1539a0 0x7ff88c0c0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:58.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.076+0000 7ff890c46640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c070a00 0x7ff88c1f5930 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:58.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.076+0000 7ff890c46640 1 --2- 172.21.15.46:0/1964882260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c069e90 0x7ff88c0c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:58.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.076+0000 7ff890c46640 1 -- 172.21.15.46:0/1964882260 >> 172.21.15.46:0/1964882260 conn(0x7ff88c0c7910 msgr2=0x7ff88c152a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:58.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.076+0000 7ff890c46640 1 -- 172.21.15.46:0/1964882260 shutdown_connections 2023-10-30T13:44:58.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:44:58.076+0000 7ff890c46640 1 -- 172.21.15.46:0/1964882260 wait complete. 2023-10-30T13:44:58.099 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:45:00.100 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:00.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.248+0000 7f64a3e17640 1 Processor -- start 2023-10-30T13:45:00.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.248+0000 7f64a3e17640 1 -- start start 2023-10-30T13:45:00.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.252+0000 7f64a3e17640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f649c071110 0x7f649c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:00.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.252+0000 7f64a3e17640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c066cf0 con 0x7f649c0c6bd0 2023-10-30T13:45:00.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.252+0000 7f64a3e17640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c0c70d0 con 0x7f649c06f7a0 2023-10-30T13:45:00.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.252+0000 7f64a3e17640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f649c0700c0 con 0x7f649c071110 2023-10-30T13:45:00.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.252+0000 7f64a23b4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f649c071110 0x7f649c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:00.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.252+0000 7f64a23b4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f649c071110 0x7f649c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41806/0 (socket says 172.21.15.46:41806) 2023-10-30T13:45:00.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.252+0000 7f64a23b4640 1 -- 172.21.15.46:0/2328025963 learned_addr learned my addr 172.21.15.46:0/2328025963 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:45:00.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.252+0000 7f64a13b2640 1 --1- 172.21.15.46:0/2328025963 >> v1:172.21.15.46:6789/0 conn(0x7f649c0c6bd0 0x7f649c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49520/0 (socket says 172.21.15.46:49520) 2023-10-30T13:45:00.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.252+0000 7f64a23b4640 1 -- 172.21.15.46:0/2328025963 >> v1:172.21.15.154:6789/0 conn(0x7f649c06f7a0 legacy=0x7f649c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:00.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.252+0000 7f64a23b4640 1 -- 172.21.15.46:0/2328025963 >> v1:172.21.15.46:6789/0 conn(0x7f649c0c6bd0 legacy=0x7f649c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:00.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.252+0000 7f64a23b4640 1 -- 172.21.15.46:0/2328025963 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c0c7390 con 0x7f649c071110 2023-10-30T13:45:00.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a23b4640 1 --2- 172.21.15.46:0/2328025963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c071110 0x7f649c06fb80 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f6494009e50 tx=0x7f649401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc15bc89fdcc554 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:00.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a0bb1640 1 -- 172.21.15.46:0/2328025963 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64940af400 con 0x7f649c071110 2023-10-30T13:45:00.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a0bb1640 1 -- 172.21.15.46:0/2328025963 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64940ab050 con 0x7f649c071110 2023-10-30T13:45:00.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a0bb1640 1 -- 172.21.15.46:0/2328025963 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64940af780 con 0x7f649c071110 2023-10-30T13:45:00.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a3e17640 1 -- 172.21.15.46:0/2328025963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c071110 msgr2=0x7f649c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:00.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a3e17640 1 --2- 172.21.15.46:0/2328025963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c071110 0x7f649c06fb80 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f6494009e50 tx=0x7f649401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:45:00.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a3e17640 1 -- 172.21.15.46:0/2328025963 shutdown_connections 2023-10-30T13:45:00.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a3e17640 1 --2- 172.21.15.46:0/2328025963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c071110 0x7f649c06fb80 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a3e17640 1 -- 172.21.15.46:0/2328025963 >> 172.21.15.46:0/2328025963 conn(0x7f649c0c7910 msgr2=0x7f649c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a3e17640 1 -- 172.21.15.46:0/2328025963 shutdown_connections 2023-10-30T13:45:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a3e17640 1 -- 172.21.15.46:0/2328025963 wait complete. 2023-10-30T13:45:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a3e17640 1 Processor -- start 2023-10-30T13:45:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.256+0000 7f64a3e17640 1 -- start start 2023-10-30T13:45:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a3e17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f649c06f7a0 0x7f649c0c06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a3e17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c071110 0x7f649c143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:00.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a3e17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f649c0c6bd0 0x7f649c150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:00.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a3e17640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c0b0ae0 con 0x7f649c071110 2023-10-30T13:45:00.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a3e17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c14df20 con 0x7f649c06f7a0 2023-10-30T13:45:00.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a3e17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649c1f3970 con 0x7f649c0c6bd0 2023-10-30T13:45:00.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a13b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c071110 0x7f649c143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:00.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a13b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c071110 0x7f649c143910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41822/0 (socket says 172.21.15.46:41822) 2023-10-30T13:45:00.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a13b2640 1 -- 172.21.15.46:0/2150973468 learned_addr learned my addr 172.21.15.46:0/2150973468 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:45:00.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a1bb3640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f649c06f7a0 0x7f649c0c06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:00.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a23b4640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f649c0c6bd0 0x7f649c150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:00.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a13b2640 1 -- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f649c0c6bd0 msgr2=0x7f649c150410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:00.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a13b2640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f649c0c6bd0 0x7f649c150410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:00.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a13b2640 1 -- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f649c06f7a0 msgr2=0x7f649c0c06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:00.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a13b2640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f649c06f7a0 0x7f649c0c06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:00.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a13b2640 1 -- 172.21.15.46:0/2150973468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6494009b30 con 0x7f649c071110 2023-10-30T13:45:00.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a23b4640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f649c0c6bd0 0x7f649c150410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:00.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a13b2640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c071110 0x7f649c143910 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f648c00c500 tx=0x7f648c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:00.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64927fc640 1 -- 172.21.15.46:0/2150973468 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c007500 con 0x7f649c071110 2023-10-30T13:45:00.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64927fc640 1 -- 172.21.15.46:0/2150973468 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648c007660 con 0x7f649c071110 2023-10-30T13:45:00.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64927fc640 1 -- 172.21.15.46:0/2150973468 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c004a10 con 0x7f649c071110 2023-10-30T13:45:00.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a3e17640 1 -- 172.21.15.46:0/2150973468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649c14e1c0 con 0x7f649c071110 2023-10-30T13:45:00.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a3e17640 1 -- 172.21.15.46:0/2150973468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649c14e780 con 0x7f649c071110 2023-10-30T13:45:00.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64a3e17640 1 -- 172.21.15.46:0/2150973468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649c04e170 con 0x7f649c071110 2023-10-30T13:45:00.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.260+0000 7f64927fc640 1 -- 172.21.15.46:0/2150973468 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f648c004b70 con 0x7f649c071110 2023-10-30T13:45:00.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.264+0000 7f64927fc640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64800664a0 0x7f6480068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:00.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.264+0000 7f64927fc640 1 -- 172.21.15.46:0/2150973468 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f648c0834c0 con 0x7f649c071110 2023-10-30T13:45:00.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.264+0000 7f64a1bb3640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64800664a0 0x7f6480068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:00.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.264+0000 7f64a1bb3640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64800664a0 0x7f6480068960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6498007fd0 tx=0x7f6498009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:00.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.268+0000 7f64927fc640 1 -- 172.21.15.46:0/2150973468 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f648c0505c0 con 0x7f649c071110 2023-10-30T13:45:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 -- 172.21.15.46:0/2150973468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f649c06fc20 con 0x7f649c071110 2023-10-30T13:45:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64927fc640 1 -- 172.21.15.46:0/2150973468 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f648c053be0 con 0x7f649c071110 2023-10-30T13:45:00.462 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:45:00.462 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:44:58.099346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:rejoin","state_seq":49,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 107 2023-10-30T13:45:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 -- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64800664a0 msgr2=0x7f6480068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64800664a0 0x7f6480068960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6498007fd0 tx=0x7f6498009040 comp rx=0 tx=0).stop 2023-10-30T13:45:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 -- 172.21.15.46:0/2150973468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c071110 msgr2=0x7f649c143910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c071110 0x7f649c143910 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f648c00c500 tx=0x7f648c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:45:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 -- 172.21.15.46:0/2150973468 shutdown_connections 2023-10-30T13:45:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64800664a0 0x7f6480068960 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f649c0c6bd0 0x7f649c150410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c071110 0x7f649c143910 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 --2- 172.21.15.46:0/2150973468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f649c06f7a0 0x7f649c0c06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 -- 172.21.15.46:0/2150973468 >> 172.21.15.46:0/2150973468 conn(0x7f649c0c7910 msgr2=0x7f649c0c7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 -- 172.21.15.46:0/2150973468 shutdown_connections 2023-10-30T13:45:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:00.456+0000 7f64a3e17640 1 -- 172.21.15.46:0/2150973468 wait complete. 2023-10-30T13:45:00.483 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:45:02.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.632+0000 7f5e4f50e640 1 Processor -- start 2023-10-30T13:45:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.632+0000 7f5e4f50e640 1 -- start start 2023-10-30T13:45:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.632+0000 7f5e4f50e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5e4806f7a0 0x7f5e4806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.632+0000 7f5e4f50e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5e48071110 0x7f5e48158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.632+0000 7f5e4f50e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e48066cf0 con 0x7f5e480c6bd0 2023-10-30T13:45:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.632+0000 7f5e4f50e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e480700c0 con 0x7f5e4806f7a0 2023-10-30T13:45:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.632+0000 7f5e4f50e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e480c70d0 con 0x7f5e48071110 2023-10-30T13:45:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.632+0000 7f5e4d2aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5e4806f7a0 0x7f5e4806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.632+0000 7f5e4d2aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5e4806f7a0 0x7f5e4806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41838/0 (socket says 172.21.15.46:41838) 2023-10-30T13:45:02.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.632+0000 7f5e4d2aa640 1 -- 172.21.15.46:0/1151672619 learned_addr learned my addr 172.21.15.46:0/1151672619 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:45:02.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.636+0000 7f5e4daab640 1 --2- 172.21.15.46:0/1151672619 >> v2:172.21.15.154:3300/0 conn(0x7f5e48071110 0x7f5e48158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.636+0000 7f5e4d2aa640 1 -- 172.21.15.46:0/1151672619 >> v2:172.21.15.154:3300/0 conn(0x7f5e48071110 msgr2=0x7f5e48158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:02.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.636+0000 7f5e4d2aa640 1 --2- 172.21.15.46:0/1151672619 >> v2:172.21.15.154:3300/0 conn(0x7f5e48071110 0x7f5e48158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.636+0000 7f5e4d2aa640 1 -- 172.21.15.46:0/1151672619 >> v1:172.21.15.46:6789/0 conn(0x7f5e480c6bd0 legacy=0x7f5e48069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.636+0000 7f5e4d2aa640 1 -- 172.21.15.46:0/1151672619 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e480c7390 con 0x7f5e4806f7a0 2023-10-30T13:45:02.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.636+0000 7f5e4d2aa640 1 --2- 172.21.15.46:0/1151672619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e4806f7a0 0x7f5e4806fb80 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f5e4806cdd0 tx=0x7f5e4401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d32eb389f4b6d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:02.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.636+0000 7f5e3ffff640 1 -- 172.21.15.46:0/1151672619 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e440a89f0 con 0x7f5e4806f7a0 2023-10-30T13:45:02.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.636+0000 7f5e3ffff640 1 -- 172.21.15.46:0/1151672619 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e440a8b50 con 0x7f5e4806f7a0 2023-10-30T13:45:02.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.636+0000 7f5e3ffff640 1 -- 172.21.15.46:0/1151672619 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e440b0440 con 0x7f5e4806f7a0 2023-10-30T13:45:02.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.636+0000 7f5e4f50e640 1 -- 172.21.15.46:0/1151672619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e4806f7a0 msgr2=0x7f5e4806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.636+0000 7f5e4f50e640 1 --2- 172.21.15.46:0/1151672619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e4806f7a0 0x7f5e4806fb80 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f5e4806cdd0 tx=0x7f5e4401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 -- 172.21.15.46:0/1151672619 shutdown_connections 2023-10-30T13:45:02.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 --2- 172.21.15.46:0/1151672619 >> v2:172.21.15.154:3300/0 conn(0x7f5e48071110 0x7f5e48158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 --2- 172.21.15.46:0/1151672619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e4806f7a0 0x7f5e4806fb80 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 -- 172.21.15.46:0/1151672619 >> 172.21.15.46:0/1151672619 conn(0x7f5e480c7910 msgr2=0x7f5e480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 -- 172.21.15.46:0/1151672619 shutdown_connections 2023-10-30T13:45:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 -- 172.21.15.46:0/1151672619 wait complete. 2023-10-30T13:45:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 Processor -- start 2023-10-30T13:45:02.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 -- start start 2023-10-30T13:45:02.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e4806f7a0 0x7f5e480c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e48071110 0x7f5e481f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e480c6bd0 0x7f5e481f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e48156170 con 0x7f5e480c6bd0 2023-10-30T13:45:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4806c4d0 con 0x7f5e4806f7a0 2023-10-30T13:45:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4f50e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e481550c0 con 0x7f5e48071110 2023-10-30T13:45:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4daab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e480c6bd0 0x7f5e481f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4daab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e480c6bd0 0x7f5e481f26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41840/0 (socket says 172.21.15.46:41840) 2023-10-30T13:45:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4daab640 1 -- 172.21.15.46:0/114794800 learned_addr learned my addr 172.21.15.46:0/114794800 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:45:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4d2aa640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e4806f7a0 0x7f5e480c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4caa9640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e48071110 0x7f5e481f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4daab640 1 -- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e48071110 msgr2=0x7f5e481f21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4daab640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e48071110 0x7f5e481f21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4daab640 1 -- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e4806f7a0 msgr2=0x7f5e480c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4daab640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e4806f7a0 0x7f5e480c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4daab640 1 -- 172.21.15.46:0/114794800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e440094d0 con 0x7f5e480c6bd0 2023-10-30T13:45:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4d2aa640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e4806f7a0 0x7f5e480c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:02.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.640+0000 7f5e4daab640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e480c6bd0 0x7f5e481f26e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f5e40007e90 tx=0x7f5e4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:02.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.644+0000 7f5e3dffb640 1 -- 172.21.15.46:0/114794800 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e40014420 con 0x7f5e480c6bd0 2023-10-30T13:45:02.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.644+0000 7f5e3dffb640 1 -- 172.21.15.46:0/114794800 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e40014580 con 0x7f5e480c6bd0 2023-10-30T13:45:02.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.644+0000 7f5e4f50e640 1 -- 172.21.15.46:0/114794800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e48154de0 con 0x7f5e480c6bd0 2023-10-30T13:45:02.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.644+0000 7f5e3dffb640 1 -- 172.21.15.46:0/114794800 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e40014930 con 0x7f5e480c6bd0 2023-10-30T13:45:02.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.644+0000 7f5e4f50e640 1 -- 172.21.15.46:0/114794800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e48155b20 con 0x7f5e480c6bd0 2023-10-30T13:45:02.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.644+0000 7f5e4f50e640 1 -- 172.21.15.46:0/114794800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e48070330 con 0x7f5e480c6bd0 2023-10-30T13:45:02.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.644+0000 7f5e3dffb640 1 -- 172.21.15.46:0/114794800 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5e40014a90 con 0x7f5e480c6bd0 2023-10-30T13:45:02.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.644+0000 7f5e3dffb640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e2c0664a0 0x7f5e2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.644+0000 7f5e3dffb640 1 -- 172.21.15.46:0/114794800 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f5e40084c60 con 0x7f5e480c6bd0 2023-10-30T13:45:02.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.644+0000 7f5e4d2aa640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e2c0664a0 0x7f5e2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.648+0000 7f5e4d2aa640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e2c0664a0 0x7f5e2c068960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5e440b2000 tx=0x7f5e44008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.648+0000 7f5e3dffb640 1 -- 172.21.15.46:0/114794800 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5e48070330 con 0x7f5e480c6bd0 2023-10-30T13:45:02.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.840+0000 7f5e4f50e640 1 -- 172.21.15.46:0/114794800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e48070330 con 0x7f5e480c6bd0 2023-10-30T13:45:02.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.840+0000 7f5e3dffb640 1 -- 172.21.15.46:0/114794800 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f5e48070330 con 0x7f5e480c6bd0 2023-10-30T13:45:02.847 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:45:02.847 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:44:58.099346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:rejoin","state_seq":49,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:02.848 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 107 2023-10-30T13:45:02.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.844+0000 7f5e4f50e640 1 -- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e2c0664a0 msgr2=0x7f5e2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.844+0000 7f5e4f50e640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e2c0664a0 0x7f5e2c068960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5e440b2000 tx=0x7f5e44008040 comp rx=0 tx=0).stop 2023-10-30T13:45:02.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.844+0000 7f5e4f50e640 1 -- 172.21.15.46:0/114794800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e480c6bd0 msgr2=0x7f5e481f26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.844+0000 7f5e4f50e640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e480c6bd0 0x7f5e481f26e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f5e40007e90 tx=0x7f5e4000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.844+0000 7f5e4f50e640 1 -- 172.21.15.46:0/114794800 shutdown_connections 2023-10-30T13:45:02.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.844+0000 7f5e4f50e640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e2c0664a0 0x7f5e2c068960 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.844+0000 7f5e4f50e640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e480c6bd0 0x7f5e481f26e0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.844+0000 7f5e4f50e640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e48071110 0x7f5e481f21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.844+0000 7f5e4f50e640 1 --2- 172.21.15.46:0/114794800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e4806f7a0 0x7f5e480c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.844+0000 7f5e4f50e640 1 -- 172.21.15.46:0/114794800 >> 172.21.15.46:0/114794800 conn(0x7f5e480c7910 msgr2=0x7f5e4806a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:02.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.844+0000 7f5e4f50e640 1 -- 172.21.15.46:0/114794800 shutdown_connections 2023-10-30T13:45:02.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:02.844+0000 7f5e4f50e640 1 -- 172.21.15.46:0/114794800 wait complete. 2023-10-30T13:45:02.868 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:45:04.870 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:05.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f854013e640 1 Processor -- start 2023-10-30T13:45:05.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f854013e640 1 -- start start 2023-10-30T13:45:05.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f854013e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f853815d6f0 0x7f853815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f854013e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85380cf160 0x7f85380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f854013e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85380cb800 con 0x7f85381538a0 2023-10-30T13:45:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f854013e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85380cb330 con 0x7f853815d6f0 2023-10-30T13:45:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f854013e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85380cad70 con 0x7f85380cf160 2023-10-30T13:45:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f853d6d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f853815d6f0 0x7f853815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f853d6d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f853815d6f0 0x7f853815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41854/0 (socket says 172.21.15.46:41854) 2023-10-30T13:45:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f853d6d9640 1 -- 172.21.15.46:0/2951911850 learned_addr learned my addr 172.21.15.46:0/2951911850 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:45:05.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f853e6db640 1 --2- 172.21.15.46:0/2951911850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85380cf160 0x7f85380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f853d6d9640 1 -- 172.21.15.46:0/2951911850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85380cf160 msgr2=0x7f85380cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:05.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f853d6d9640 1 --2- 172.21.15.46:0/2951911850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85380cf160 0x7f85380cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f853d6d9640 1 -- 172.21.15.46:0/2951911850 >> v1:172.21.15.154:6789/0 conn(0x7f85381538a0 legacy=0x7f85380ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:05.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.020+0000 7f853d6d9640 1 -- 172.21.15.46:0/2951911850 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85380cb050 con 0x7f853815d6f0 2023-10-30T13:45:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.024+0000 7f853d6d9640 1 --2- 172.21.15.46:0/2951911850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f853815d6f0 0x7f853815a090 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8528009870 tx=0x7f852801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5abfd0e1e3576b03 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.024+0000 7f853ced8640 1 -- 172.21.15.46:0/2951911850 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85280a8a30 con 0x7f853815d6f0 2023-10-30T13:45:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.024+0000 7f853ced8640 1 -- 172.21.15.46:0/2951911850 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85280a8b90 con 0x7f853815d6f0 2023-10-30T13:45:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.024+0000 7f853ced8640 1 -- 172.21.15.46:0/2951911850 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85280b0440 con 0x7f853815d6f0 2023-10-30T13:45:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.024+0000 7f854013e640 1 -- 172.21.15.46:0/2951911850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f853815d6f0 msgr2=0x7f853815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.024+0000 7f854013e640 1 --2- 172.21.15.46:0/2951911850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f853815d6f0 0x7f853815a090 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8528009870 tx=0x7f852801ed30 comp rx=0 tx=0).stop 2023-10-30T13:45:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.024+0000 7f854013e640 1 -- 172.21.15.46:0/2951911850 shutdown_connections 2023-10-30T13:45:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.024+0000 7f854013e640 1 --2- 172.21.15.46:0/2951911850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85380cf160 0x7f85380cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.024+0000 7f854013e640 1 --2- 172.21.15.46:0/2951911850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f853815d6f0 0x7f853815a090 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.024+0000 7f854013e640 1 -- 172.21.15.46:0/2951911850 >> 172.21.15.46:0/2951911850 conn(0x7f853806bfe0 msgr2=0x7f853806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:05.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.024+0000 7f854013e640 1 -- 172.21.15.46:0/2951911850 shutdown_connections 2023-10-30T13:45:05.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.024+0000 7f854013e640 1 -- 172.21.15.46:0/2951911850 wait complete. 2023-10-30T13:45:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f854013e640 1 Processor -- start 2023-10-30T13:45:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f854013e640 1 -- start start 2023-10-30T13:45:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f854013e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85380cf160 0x7f8538159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f854013e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85381538a0 0x7f85380c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f854013e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f853815d6f0 0x7f85381e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f854013e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85381e3bf0 con 0x7f85380cf160 2023-10-30T13:45:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f854013e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85381580f0 con 0x7f85381538a0 2023-10-30T13:45:05.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f854013e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85380c59a0 con 0x7f853815d6f0 2023-10-30T13:45:05.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f853deda640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85380cf160 0x7f8538159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f853deda640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85380cf160 0x7f8538159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41862/0 (socket says 172.21.15.46:41862) 2023-10-30T13:45:05.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f853deda640 1 -- 172.21.15.46:0/1616907926 learned_addr learned my addr 172.21.15.46:0/1616907926 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:45:05.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f853d6d9640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85381538a0 0x7f85380c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f853e6db640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f853815d6f0 0x7f85381e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f853deda640 1 -- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f853815d6f0 msgr2=0x7f85381e2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:05.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f853deda640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f853815d6f0 0x7f85381e2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f853deda640 1 -- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85381538a0 msgr2=0x7f85380c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f853deda640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85381538a0 0x7f85380c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f853deda640 1 -- 172.21.15.46:0/1616907926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85280094d0 con 0x7f85380cf160 2023-10-30T13:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f853deda640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85380cf160 0x7f8538159300 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f853400c500 tx=0x7f853400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f852e7fc640 1 -- 172.21.15.46:0/1616907926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534005220 con 0x7f85380cf160 2023-10-30T13:45:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f852e7fc640 1 -- 172.21.15.46:0/1616907926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8534005380 con 0x7f85380cf160 2023-10-30T13:45:05.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f852e7fc640 1 -- 172.21.15.46:0/1616907926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534007270 con 0x7f85380cf160 2023-10-30T13:45:05.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f854013e640 1 -- 172.21.15.46:0/1616907926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85381584d0 con 0x7f85380cf160 2023-10-30T13:45:05.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f854013e640 1 -- 172.21.15.46:0/1616907926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85380c4f60 con 0x7f85380cf160 2023-10-30T13:45:05.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.028+0000 7f854013e640 1 -- 172.21.15.46:0/1616907926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85380b2a50 con 0x7f85380cf160 2023-10-30T13:45:05.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.032+0000 7f852e7fc640 1 -- 172.21.15.46:0/1616907926 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f85340073d0 con 0x7f85380cf160 2023-10-30T13:45:05.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.032+0000 7f852e7fc640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f851c0664d0 0x7f851c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.032+0000 7f852e7fc640 1 -- 172.21.15.46:0/1616907926 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f8534083a80 con 0x7f85380cf160 2023-10-30T13:45:05.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.032+0000 7f853d6d9640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f851c0664d0 0x7f851c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.032+0000 7f853d6d9640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f851c0664d0 0x7f851c068990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f85280b2000 tx=0x7f8528009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:05.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.036+0000 7f852e7fc640 1 -- 172.21.15.46:0/1616907926 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8534050b00 con 0x7f85380cf160 2023-10-30T13:45:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.224+0000 7f854013e640 1 -- 172.21.15.46:0/1616907926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f853804e170 con 0x7f85380cf160 2023-10-30T13:45:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.224+0000 7f852e7fc640 1 -- 172.21.15.46:0/1616907926 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f8534054120 con 0x7f85380cf160 2023-10-30T13:45:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:45:05.232 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:45:03.361779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 108 2023-10-30T13:45:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.228+0000 7f854013e640 1 -- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f851c0664d0 msgr2=0x7f851c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.228+0000 7f854013e640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f851c0664d0 0x7f851c068990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f85280b2000 tx=0x7f8528009040 comp rx=0 tx=0).stop 2023-10-30T13:45:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.228+0000 7f854013e640 1 -- 172.21.15.46:0/1616907926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85380cf160 msgr2=0x7f8538159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:05.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.228+0000 7f854013e640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85380cf160 0x7f8538159300 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f853400c500 tx=0x7f853400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.228+0000 7f854013e640 1 -- 172.21.15.46:0/1616907926 shutdown_connections 2023-10-30T13:45:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.228+0000 7f854013e640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f851c0664d0 0x7f851c068990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.228+0000 7f854013e640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f853815d6f0 0x7f85381e2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.228+0000 7f854013e640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85381538a0 0x7f85380c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.228+0000 7f854013e640 1 --2- 172.21.15.46:0/1616907926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85380cf160 0x7f8538159300 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.228+0000 7f854013e640 1 -- 172.21.15.46:0/1616907926 >> 172.21.15.46:0/1616907926 conn(0x7f853806bfe0 msgr2=0x7f853814ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.228+0000 7f854013e640 1 -- 172.21.15.46:0/1616907926 shutdown_connections 2023-10-30T13:45:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:05.228+0000 7f854013e640 1 -- 172.21.15.46:0/1616907926 wait complete. 2023-10-30T13:45:05.253 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=6036 2023-10-30T13:45:05.253 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.c 2023-10-30T13:45:12.249 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 8% (18/201) Updating files: 9% (19/201) Updating files: 10% (21/201) Updating files: 11% (23/201) Updating files: 12% (25/201) Updating files: 13% (27/201) Updating files: 14% (29/201) Updating files: 15% (31/201) Updating files: 16% (33/201) Updating files: 17% (35/201) Updating files: 18% (37/201) Updating files: 19% (39/201) Updating files: 20% (41/201) Updating files: 21% (43/201) Updating files: 22% (45/201) Updating files: 23% (47/201) Updating files: 24% (49/201) Updating files: 25% (51/201) Updating files: 26% (53/201) Updating files: 27% (55/201) Updating files: 28% (57/201) Updating files: 29% (59/201) Updating files: 30% (61/201) Updating files: 31% (63/201) Updating files: 32% (65/201) Updating files: 33% (67/201) Updating files: 34% (69/201) Updating files: 35% (71/201) Updating files: 36% (73/201) Updating files: 37% (75/201) Updating files: 38% (77/201) Updating files: 39% (79/201) Updating files: 40% (81/201) Updating files: 41% (83/201) Updating files: 42% (85/201) Updating files: 43% (87/201) Updating files: 44% (89/201) Updating files: 45% (91/201) Updating files: 46% (93/201) Updating files: 47% (95/201) Updating files: 48% (97/201) Updating files: 49% (99/201) Updating files: 50% (101/201) Updating files: 51% (103/201) Updating files: 52% (105/201) Updating files: 53% (107/201) Updating files: 54% (109/201) Updating files: 55% (111/201) Updating files: 56% (113/201) Updating files: 57% (115/201) Updating files: 58% (117/201) Updating files: 59% (119/201) Updating files: 60% (121/201) Updating files: 61% (123/201) Updating files: 62% (125/201) Updating files: 63% (127/201) Updating files: 64% (129/201) Updating files: 65% (131/201) Updating files: 66% (133/201) Updating files: 67% (135/201) Updating files: 68% (137/201) Updating files: 69% (139/201) Updating files: 70% (141/201) Updating files: 71% (143/201) Updating files: 72% (145/201) Updating files: 73% (147/201) Updating files: 74% (149/201) Updating files: 75% (151/201) Updating files: 76% (153/201) Updating files: 77% (155/201) Updating files: 78% (157/201) Updating files: 79% (159/201) Updating files: 80% (161/201) Updating files: 81% (163/201) Updating files: 82% (165/201) Updating files: 83% (167/201) Updating files: 84% (169/201) Updating files: 85% (171/201) Updating files: 86% (173/201) Updating files: 87% (175/201) Updating files: 88% (177/201) Updating files: 89% (179/201) Updating files: 90% (181/201) Updating files: 91% (183/201) Updating files: 92% (185/201) Updating files: 93% (187/201) Updating files: 94% (189/201) Updating files: 95% (191/201) Updating files: 96% (193/201) Updating files: 97% (195/201) Updating files: 98% (197/201) Updating files: 99% (199/201) Updating files: 100% (201/201) Updating files: 100% (201/201), done. 2023-10-30T13:45:12.565 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at c46e1b320c6 mkcephfs: btrfsctl -a between mkfs and mount 2023-10-30T13:45:12.570 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.7 2023-10-30T13:45:13.319 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 8 -eq 48 ] 2023-10-30T13:45:13.319 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.8 2023-10-30T13:45:13.320 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.8 2023-10-30T13:45:13.320 INFO:tasks.workunit.client.0.smithi046.stdout:v0.8 2023-10-30T13:45:13.320 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.8 2023-10-30T13:45:14.255 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:45:14.255 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:45:14.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:45:14.259 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:45:14.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:14.365 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-10-30T13:45:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d6a6b640 1 Processor -- start 2023-10-30T13:45:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d6a6b640 1 -- start start 2023-10-30T13:45:14.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d6a6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d01538a0 0x7fb2d015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:14.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d6a6b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb2d00cf160 0x7fb2d00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:14.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d6a6b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d00cb800 con 0x7fb2d00ca0e0 2023-10-30T13:45:14.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d6a6b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d015dc30 con 0x7fb2d00cf160 2023-10-30T13:45:14.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d6a6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00cb330 con 0x7fb2d01538a0 2023-10-30T13:45:14.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb2d00ca0e0 0x7fb2d015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50450/0 (socket says 172.21.15.46:50450) 2023-10-30T13:45:14.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cffff640 1 -- 172.21.15.46:0/783928414 learned_addr learned my addr 172.21.15.46:0/783928414 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:45:14.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d5008640 1 --2- 172.21.15.46:0/783928414 >> v2:172.21.15.154:3300/0 conn(0x7fb2d00cf160 0x7fb2d00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:14.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d4807640 1 --2- 172.21.15.46:0/783928414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d01538a0 0x7fb2d015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:14.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210816792 0 0) 0x7fb2d00cb800 con 0x7fb2d00ca0e0 2023-10-30T13:45:14.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c0003530 con 0x7fb2d00ca0e0 2023-10-30T13:45:14.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 714378765 0 0) 0x7fb2c0003530 con 0x7fb2d00ca0e0 2023-10-30T13:45:14.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d01538a0 msgr2=0x7fb2d015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:14.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 --2- 172.21.15.46:0/783928414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d01538a0 0x7fb2d015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:14.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 >> v2:172.21.15.154:3300/0 conn(0x7fb2d00cf160 msgr2=0x7fb2d00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:14.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 --2- 172.21.15.46:0/783928414 >> v2:172.21.15.154:3300/0 conn(0x7fb2d00cf160 0x7fb2d00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:14.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d00cad70 con 0x7fb2d00ca0e0 2023-10-30T13:45:14.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fb2bc001680 con 0x7fb2d00ca0e0 2023-10-30T13:45:14.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 >> v1:172.21.15.46:6789/0 conn(0x7fb2d00ca0e0 legacy=0x7fb2d015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:14.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 --2- 172.21.15.46:0/783928414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d01538a0 0x7fb2c0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:14.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 --2- 172.21.15.46:0/783928414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2c0003f00 0x7fb2c00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:14.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 --2- 172.21.15.46:0/783928414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c0004d10 0x7fb2c0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d015dc30 con 0x7fb2c0004d10 2023-10-30T13:45:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00cb330 con 0x7fb2c0003f00 2023-10-30T13:45:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c000b9c0 con 0x7fb2d01538a0 2023-10-30T13:45:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d5008640 1 --2- 172.21.15.46:0/783928414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c0004d10 0x7fb2c0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2cffff640 1 --2- 172.21.15.46:0/783928414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2c0003f00 0x7fb2c00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d4807640 1 --2- 172.21.15.46:0/783928414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d01538a0 0x7fb2c0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:14.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d5008640 1 -- 172.21.15.46:0/783928414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d01538a0 msgr2=0x7fb2c0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:14.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d5008640 1 --2- 172.21.15.46:0/783928414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d01538a0 0x7fb2c0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:14.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d5008640 1 -- 172.21.15.46:0/783928414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2c0003f00 msgr2=0x7fb2c00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:14.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d5008640 1 --2- 172.21.15.46:0/783928414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2c0003f00 0x7fb2c00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:14.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.420+0000 7fb2d5008640 1 -- 172.21.15.46:0/783928414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c0003530 con 0x7fb2c0004d10 2023-10-30T13:45:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.424+0000 7fb2d5008640 1 --2- 172.21.15.46:0/783928414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c0004d10 0x7fb2c0005190 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009fc0 tx=0x7fb2c801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ad183c41737418 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:14.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.424+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c80a5cb0 con 0x7fb2c0004d10 2023-10-30T13:45:14.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.424+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c800f040 con 0x7fb2c0004d10 2023-10-30T13:45:14.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.424+0000 7fb2cf7fe640 1 -- 172.21.15.46:0/783928414 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c80ad680 con 0x7fb2c0004d10 2023-10-30T13:45:14.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.424+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/783928414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c0004d10 msgr2=0x7fb2c0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:14.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.424+0000 7fb2d6a6b640 1 --2- 172.21.15.46:0/783928414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c0004d10 0x7fb2c0005190 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009fc0 tx=0x7fb2c801cca0 comp rx=0 tx=0).stop 2023-10-30T13:45:14.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.424+0000 7fb2d5008640 1 -- 172.21.15.46:0/783928414 reap_dead start 2023-10-30T13:45:14.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.424+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/783928414 shutdown_connections 2023-10-30T13:45:14.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.424+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/783928414 >> 172.21.15.46:0/783928414 conn(0x7fb2d006bfe0 msgr2=0x7fb2d006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:14.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.424+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/783928414 shutdown_connections 2023-10-30T13:45:14.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.424+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/783928414 wait complete. 2023-10-30T13:45:14.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d6a6b640 1 Processor -- start 2023-10-30T13:45:14.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d6a6b640 1 -- start start 2023-10-30T13:45:14.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d6a6b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2c0004d10 0x7fb2d00c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:14.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d6a6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d00ce770 0x7fb2d015be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:14.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d6a6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d00cebb0 0x7fb2d015c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:14.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d6a6b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d015ba70 con 0x7fb2d00ce770 2023-10-30T13:45:14.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d6a6b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d015a8b0 con 0x7fb2c0004d10 2023-10-30T13:45:14.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d6a6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00c7420 con 0x7fb2d00cebb0 2023-10-30T13:45:14.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d00ce770 0x7fb2d015be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:14.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d00ce770 0x7fb2d015be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41856/0 (socket says 172.21.15.46:41856) 2023-10-30T13:45:14.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2cffff640 1 -- 172.21.15.46:0/129460601 learned_addr learned my addr 172.21.15.46:0/129460601 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:45:14.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d4807640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2c0004d10 0x7fb2d00c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:14.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d5008640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d00cebb0 0x7fb2d015c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:14.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2cffff640 1 -- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d00cebb0 msgr2=0x7fb2d015c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:14.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2cffff640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d00cebb0 0x7fb2d015c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:14.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2cffff640 1 -- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2c0004d10 msgr2=0x7fb2d00c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:14.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2cffff640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2c0004d10 0x7fb2d00c6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:14.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2cffff640 1 -- 172.21.15.46:0/129460601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d015b440 con 0x7fb2d00ce770 2023-10-30T13:45:14.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d5008640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d00cebb0 0x7fb2d015c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:14.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2cffff640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d00ce770 0x7fb2d015be90 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc007310 tx=0x7fb2bc007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:14.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2cd7fa640 1 -- 172.21.15.46:0/129460601 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc00f9c0 con 0x7fb2d00ce770 2023-10-30T13:45:14.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2cd7fa640 1 -- 172.21.15.46:0/129460601 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2bc00fb60 con 0x7fb2d00ce770 2023-10-30T13:45:14.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2cd7fa640 1 -- 172.21.15.46:0/129460601 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc0133f0 con 0x7fb2d00ce770 2023-10-30T13:45:14.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/129460601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d015aae0 con 0x7fb2d00ce770 2023-10-30T13:45:14.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.428+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/129460601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d015b0a0 con 0x7fb2d00ce770 2023-10-30T13:45:14.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.432+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/129460601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d015b260 con 0x7fb2d00ce770 2023-10-30T13:45:14.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.432+0000 7fb2cd7fa640 1 -- 172.21.15.46:0/129460601 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb2bc013550 con 0x7fb2d00ce770 2023-10-30T13:45:14.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.432+0000 7fb2cd7fa640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb2b40664a0 0x7fb2b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:14.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.432+0000 7fb2cd7fa640 1 -- 172.21.15.46:0/129460601 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fb2bc0845e0 con 0x7fb2d00ce770 2023-10-30T13:45:14.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.432+0000 7fb2d4807640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb2b40664a0 0x7fb2b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:14.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.432+0000 7fb2d4807640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb2b40664a0 0x7fb2b4068960 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4009be0 tx=0x7fb2c4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:14.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.436+0000 7fb2cd7fa640 1 -- 172.21.15.46:0/129460601 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb2bc051510 con 0x7fb2d00ce770 2023-10-30T13:45:14.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/129460601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2d004e170 con 0x7fb2d00ce770 2023-10-30T13:45:14.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2cd7fa640 1 -- 172.21.15.46:0/129460601 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fb2bc054b30 con 0x7fb2d00ce770 2023-10-30T13:45:14.638 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:45:14.638 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:45:03.361779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:14.639 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 108 2023-10-30T13:45:14.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb2b40664a0 msgr2=0x7fb2b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:14.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb2b40664a0 0x7fb2b4068960 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4009be0 tx=0x7fb2c4009190 comp rx=0 tx=0).stop 2023-10-30T13:45:14.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/129460601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d00ce770 msgr2=0x7fb2d015be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:14.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d00ce770 0x7fb2d015be90 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc007310 tx=0x7fb2bc007710 comp rx=0 tx=0).stop 2023-10-30T13:45:14.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/129460601 shutdown_connections 2023-10-30T13:45:14.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb2b40664a0 0x7fb2b4068960 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:14.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2d00cebb0 0x7fb2d015c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:14.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d00ce770 0x7fb2d015be90 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:14.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 --2- 172.21.15.46:0/129460601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2c0004d10 0x7fb2d00c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:14.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/129460601 >> 172.21.15.46:0/129460601 conn(0x7fb2d006bfe0 msgr2=0x7fb2d00663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:14.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/129460601 shutdown_connections 2023-10-30T13:45:14.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:14.632+0000 7fb2d6a6b640 1 -- 172.21.15.46:0/129460601 wait complete. 2023-10-30T13:45:14.658 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:45:16.659 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:16.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0ecb4bb640 1 Processor -- start 2023-10-30T13:45:16.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0ecb4bb640 1 -- start start 2023-10-30T13:45:16.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0ecb4bb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0ec41538f0 0x7f0ec4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:16.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0ecb4bb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ec415cd50 0x7f0ec40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:16.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0ecb4bb640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec415d300 con 0x7f0ec40cefa0 2023-10-30T13:45:16.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0ecb4bb640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec415d5e0 con 0x7f0ec41538f0 2023-10-30T13:45:16.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0ecb4bb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec415d8c0 con 0x7f0ec415cd50 2023-10-30T13:45:16.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0eca4b9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0ec41538f0 0x7f0ec4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:16.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0eca4b9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0ec41538f0 0x7f0ec4159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42298/0 (socket says 172.21.15.46:42298) 2023-10-30T13:45:16.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0eca4b9640 1 -- 172.21.15.46:0/4167449498 learned_addr learned my addr 172.21.15.46:0/4167449498 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:45:16.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0ec9cb8640 1 --2- 172.21.15.46:0/4167449498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ec415cd50 0x7f0ec40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:16.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0eca4b9640 1 -- 172.21.15.46:0/4167449498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ec415cd50 msgr2=0x7f0ec40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:16.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0eca4b9640 1 --2- 172.21.15.46:0/4167449498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ec415cd50 0x7f0ec40c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:16.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0eca4b9640 1 -- 172.21.15.46:0/4167449498 >> v1:172.21.15.154:6789/0 conn(0x7f0ec40cefa0 legacy=0x7f0ec40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:16.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.800+0000 7f0eca4b9640 1 -- 172.21.15.46:0/4167449498 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec415dba0 con 0x7f0ec41538f0 2023-10-30T13:45:16.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0eca4b9640 1 --2- 172.21.15.46:0/4167449498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec41538f0 0x7f0ec4159fb0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0009870 tx=0x7f0ec001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1dded70baaadd6d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ec94b7640 1 -- 172.21.15.46:0/4167449498 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec00a8a30 con 0x7f0ec41538f0 2023-10-30T13:45:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ec94b7640 1 -- 172.21.15.46:0/4167449498 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec00a8b90 con 0x7f0ec41538f0 2023-10-30T13:45:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ec94b7640 1 -- 172.21.15.46:0/4167449498 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec00b0440 con 0x7f0ec41538f0 2023-10-30T13:45:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/4167449498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec41538f0 msgr2=0x7f0ec4159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ecb4bb640 1 --2- 172.21.15.46:0/4167449498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec41538f0 0x7f0ec4159fb0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0009870 tx=0x7f0ec001ed30 comp rx=0 tx=0).stop 2023-10-30T13:45:16.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/4167449498 shutdown_connections 2023-10-30T13:45:16.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ecb4bb640 1 --2- 172.21.15.46:0/4167449498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ec415cd50 0x7f0ec40c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:16.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ecb4bb640 1 --2- 172.21.15.46:0/4167449498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec41538f0 0x7f0ec4159fb0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:16.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/4167449498 >> 172.21.15.46:0/4167449498 conn(0x7f0ec406a640 msgr2=0x7f0ec4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:16.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/4167449498 shutdown_connections 2023-10-30T13:45:16.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/4167449498 wait complete. 2023-10-30T13:45:16.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ecb4bb640 1 Processor -- start 2023-10-30T13:45:16.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.804+0000 7f0ecb4bb640 1 -- start start 2023-10-30T13:45:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecb4bb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ec40cefa0 0x7f0ec40b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecb4bb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ec41538f0 0x7f0ec41e3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecb4bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec415cd50 0x7f0ec41f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecb4bb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec40b48f0 con 0x7f0ec415cd50 2023-10-30T13:45:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecb4bb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec4148180 con 0x7f0ec40cefa0 2023-10-30T13:45:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecb4bb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec41f2780 con 0x7f0ec41538f0 2023-10-30T13:45:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecacba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec415cd50 0x7f0ec41f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecacba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec415cd50 0x7f0ec41f2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42312/0 (socket says 172.21.15.46:42312) 2023-10-30T13:45:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecacba640 1 -- 172.21.15.46:0/2960482788 learned_addr learned my addr 172.21.15.46:0/2960482788 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:45:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ec9cb8640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ec41538f0 0x7f0ec41e3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:16.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0eca4b9640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ec40cefa0 0x7f0ec40b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:16.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecacba640 1 -- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ec41538f0 msgr2=0x7f0ec41e3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:16.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecacba640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ec41538f0 0x7f0ec41e3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:16.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecacba640 1 -- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ec40cefa0 msgr2=0x7f0ec40b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:16.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecacba640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ec40cefa0 0x7f0ec40b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:16.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecacba640 1 -- 172.21.15.46:0/2960482788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec00094d0 con 0x7f0ec415cd50 2023-10-30T13:45:16.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0eca4b9640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ec40cefa0 0x7f0ec40b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:16.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecacba640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec415cd50 0x7f0ec41f2c10 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00c7a0 tx=0x7f0ebc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:16.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ebaffd640 1 -- 172.21.15.46:0/2960482788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc003b80 con 0x7f0ec415cd50 2023-10-30T13:45:16.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ebaffd640 1 -- 172.21.15.46:0/2960482788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ebc003ce0 con 0x7f0ec415cd50 2023-10-30T13:45:16.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ebaffd640 1 -- 172.21.15.46:0/2960482788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc00aa90 con 0x7f0ec415cd50 2023-10-30T13:45:16.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/2960482788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec41483b0 con 0x7f0ec415cd50 2023-10-30T13:45:16.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/2960482788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec41e5510 con 0x7f0ec415cd50 2023-10-30T13:45:16.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.808+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/2960482788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec41f2930 con 0x7f0ec415cd50 2023-10-30T13:45:16.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.812+0000 7f0ebaffd640 1 -- 172.21.15.46:0/2960482788 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ebc00abf0 con 0x7f0ec415cd50 2023-10-30T13:45:16.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.812+0000 7f0ebaffd640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ea80664d0 0x7f0ea8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:16.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.812+0000 7f0ebaffd640 1 -- 172.21.15.46:0/2960482788 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f0ebc084120 con 0x7f0ec415cd50 2023-10-30T13:45:16.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.812+0000 7f0eca4b9640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ea80664d0 0x7f0ea8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:16.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.812+0000 7f0eca4b9640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ea80664d0 0x7f0ea8068990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00b2000 tx=0x7f0ec0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:16.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:16.816+0000 7f0ebaffd640 1 -- 172.21.15.46:0/2960482788 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0ec41f2930 con 0x7f0ec415cd50 2023-10-30T13:45:17.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.008+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/2960482788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ec41f28f0 con 0x7f0ec415cd50 2023-10-30T13:45:17.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.008+0000 7f0ebaffd640 1 -- 172.21.15.46:0/2960482788 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+6521 (secure 0 0 0) 0x7f0ebc0510e0 con 0x7f0ec415cd50 2023-10-30T13:45:17.015 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:45:17.016 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6702,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:45:03.361779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:17.016 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 109 2023-10-30T13:45:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.012+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ea80664d0 msgr2=0x7f0ea8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.012+0000 7f0ecb4bb640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ea80664d0 0x7f0ea8068990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00b2000 tx=0x7f0ec0008040 comp rx=0 tx=0).stop 2023-10-30T13:45:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.012+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/2960482788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec415cd50 msgr2=0x7f0ec41f2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.012+0000 7f0ecb4bb640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec415cd50 0x7f0ec41f2c10 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00c7a0 tx=0x7f0ebc00cc60 comp rx=0 tx=0).stop 2023-10-30T13:45:17.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.012+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/2960482788 shutdown_connections 2023-10-30T13:45:17.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.012+0000 7f0ecb4bb640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ea80664d0 0x7f0ea8068990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:17.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.012+0000 7f0ecb4bb640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec415cd50 0x7f0ec41f2c10 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:17.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.012+0000 7f0ecb4bb640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ec41538f0 0x7f0ec41e3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:17.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.012+0000 7f0ecb4bb640 1 --2- 172.21.15.46:0/2960482788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ec40cefa0 0x7f0ec40b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:17.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.012+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/2960482788 >> 172.21.15.46:0/2960482788 conn(0x7f0ec406a640 msgr2=0x7f0ec41489b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:17.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.012+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/2960482788 shutdown_connections 2023-10-30T13:45:17.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:45:17.012+0000 7f0ecb4bb640 1 -- 172.21.15.46:0/2960482788 wait complete. 2023-10-30T13:45:17.038 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:45:17.039 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2023-10-30T13:45:19.111 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:45:19.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:45:19.841 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 26% (60/225) Updating files: 27% (61/225) Updating files: 28% (63/225) Updating files: 29% (66/225) Updating files: 30% (68/225) Updating files: 31% (70/225) Updating files: 32% (72/225) Updating files: 33% (75/225) Updating files: 34% (77/225) Updating files: 35% (79/225) Updating files: 36% (81/225) Updating files: 37% (84/225) Updating files: 37% (85/225) Updating files: 38% (86/225) Updating files: 39% (88/225) Updating files: 40% (90/225) Updating files: 41% (93/225) Updating files: 42% (95/225) Updating files: 43% (97/225) Updating files: 44% (99/225) Updating files: 45% (102/225) Updating files: 46% (104/225) Updating files: 47% (106/225) Updating files: 48% (108/225) Updating files: 49% (111/225) Updating files: 50% (113/225) Updating files: 51% (115/225) Updating files: 52% (117/225) Updating files: 53% (120/225) Updating files: 54% (122/225) Updating files: 55% (124/225) Updating files: 56% (126/225) Updating files: 57% (129/225) Updating files: 58% (131/225) Updating files: 59% (133/225) Updating files: 60% (135/225) Updating files: 61% (138/225) Updating files: 62% (140/225) Updating files: 63% (142/225) Updating files: 64% (144/225) Updating files: 65% (147/225) Updating files: 66% (149/225) Updating files: 67% (151/225) Updating files: 68% (153/225) Updating files: 69% (156/225) Updating files: 70% (158/225) Updating files: 71% (160/225) Updating files: 72% (162/225) Updating files: 73% (165/225) Updating files: 74% (167/225) Updating files: 75% (169/225) Updating files: 76% (171/225) Updating files: 76% (173/225) Updating files: 77% (174/225) Updating files: 78% (176/225) Updating files: 79% (178/225) Updating files: 80% (180/225) Updating files: 81% (183/225) Updating files: 82% (185/225) Updating files: 83% (187/225) Updating files: 84% (189/225) Updating files: 85% (192/225) Updating files: 86% (194/225) Updating files: 87% (196/225) Updating files: 88% (198/225) Updating files: 89% (201/225) Updating files: 90% (203/225) Updating files: 91% (205/225) Updating files: 92% (207/225) Updating files: 93% (210/225) Updating files: 94% (212/225) Updating files: 95% (214/225) Updating files: 96% (216/225) Updating files: 97% (219/225) Updating files: 98% (221/225) Updating files: 99% (223/225) Updating files: 100% (225/225) Updating files: 100% (225/225), done. 2023-10-30T13:45:19.860 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 284161c00ab kclient: kill dput/d_drop debug cruft 2023-10-30T13:45:19.866 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.8 2023-10-30T13:45:19.912 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 9 -eq 48 ] 2023-10-30T13:45:19.912 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.9 2023-10-30T13:45:19.912 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.9 2023-10-30T13:45:19.913 INFO:tasks.workunit.client.0.smithi046.stdout:v0.9 2023-10-30T13:45:19.913 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.9 2023-10-30T13:45:21.364 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 68% (133/193) Updating files: 69% (134/193) Updating files: 70% (136/193) Updating files: 71% (138/193) Updating files: 72% (139/193) Updating files: 73% (141/193) Updating files: 74% (143/193) Updating files: 75% (145/193) Updating files: 76% (147/193) Updating files: 77% (149/193) Updating files: 78% (151/193) Updating files: 79% (153/193) Updating files: 80% (155/193) Updating files: 81% (157/193) Updating files: 82% (159/193) Updating files: 83% (161/193) Updating files: 84% (163/193) Updating files: 85% (165/193) Updating files: 86% (166/193) Updating files: 87% (168/193) Updating files: 88% (170/193) Updating files: 89% (172/193) Updating files: 90% (174/193) Updating files: 91% (176/193) Updating files: 92% (178/193) Updating files: 93% (180/193) Updating files: 94% (182/193) Updating files: 95% (184/193) Updating files: 96% (186/193) Updating files: 97% (188/193) Updating files: 98% (190/193) Updating files: 99% (192/193) Updating files: 100% (193/193) Updating files: 100% (193/193), done. 2023-10-30T13:45:21.644 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 41481853924 kclient: strip out kernel version compatibility cruft 2023-10-30T13:45:21.650 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.9 2023-10-30T13:45:21.736 INFO:tasks.workunit.client.0.smithi046.stdout:v0.10 2023-10-30T13:45:21.736 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 10 -eq 48 ] 2023-10-30T13:45:21.737 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.10 2023-10-30T13:45:21.737 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.10 2023-10-30T13:45:21.737 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.10 2023-10-30T13:45:26.105 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 8% (14/164) Updating files: 9% (15/164) Updating files: 10% (17/164) Updating files: 11% (19/164) Updating files: 12% (20/164) Updating files: 13% (22/164) Updating files: 14% (23/164) Updating files: 15% (25/164) Updating files: 16% (27/164) Updating files: 17% (28/164) Updating files: 18% (30/164) Updating files: 19% (32/164) Updating files: 20% (33/164) Updating files: 21% (35/164) Updating files: 22% (37/164) Updating files: 23% (38/164) Updating files: 24% (40/164) Updating files: 25% (41/164) Updating files: 26% (43/164) Updating files: 27% (45/164) Updating files: 28% (46/164) Updating files: 29% (48/164) Updating files: 30% (50/164) Updating files: 31% (51/164) Updating files: 32% (53/164) Updating files: 33% (55/164) Updating files: 34% (56/164) Updating files: 35% (58/164) Updating files: 36% (60/164) Updating files: 37% (61/164) Updating files: 38% (63/164) Updating files: 39% (64/164) Updating files: 40% (66/164) Updating files: 41% (68/164) Updating files: 42% (69/164) Updating files: 43% (71/164) Updating files: 44% (73/164) Updating files: 45% (74/164) Updating files: 46% (76/164) Updating files: 47% (78/164) Updating files: 48% (79/164) Updating files: 49% (81/164) Updating files: 50% (82/164) Updating files: 51% (84/164) Updating files: 51% (85/164) Updating files: 52% (86/164) Updating files: 53% (87/164) Updating files: 54% (89/164) Updating files: 55% (91/164) Updating files: 56% (92/164) Updating files: 57% (94/164) Updating files: 58% (96/164) Updating files: 59% (97/164) Updating files: 60% (99/164) Updating files: 61% (101/164) Updating files: 62% (102/164) Updating files: 63% (104/164) Updating files: 64% (105/164) Updating files: 65% (107/164) Updating files: 66% (109/164) Updating files: 67% (110/164) Updating files: 68% (112/164) Updating files: 69% (114/164) Updating files: 70% (115/164) Updating files: 71% (117/164) Updating files: 72% (119/164) Updating files: 73% (120/164) Updating files: 74% (122/164) Updating files: 75% (123/164) Updating files: 76% (125/164) Updating files: 77% (127/164) Updating files: 78% (128/164) Updating files: 79% (130/164) Updating files: 80% (132/164) Updating files: 81% (133/164) Updating files: 82% (135/164) Updating files: 83% (137/164) Updating files: 84% (138/164) Updating files: 85% (140/164) Updating files: 86% (142/164) Updating files: 87% (143/164) Updating files: 88% (145/164) Updating files: 89% (146/164) Updating files: 90% (148/164) Updating files: 91% (150/164) Updating files: 92% (151/164) Updating files: 93% (153/164) Updating files: 94% (155/164) Updating files: 95% (156/164) Updating files: 96% (158/164) Updating files: 97% (160/164) Updating files: 98% (161/164) Updating files: 99% (163/164) Updating files: 100% (164/164) Updating files: 100% (164/164), done. 2023-10-30T13:45:26.142 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 007313fe844 v0.10 2023-10-30T13:45:26.148 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.10 2023-10-30T13:45:26.203 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 11 -eq 48 ] 2023-10-30T13:45:26.203 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.11 2023-10-30T13:45:26.203 INFO:tasks.workunit.client.0.smithi046.stdout:v0.11 2023-10-30T13:45:26.204 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.11 2023-10-30T13:45:26.204 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.11 2023-10-30T13:45:28.098 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 95% (66/69) Updating files: 97% (67/69) Updating files: 98% (68/69) Updating files: 100% (69/69) Updating files: 100% (69/69), done. 2023-10-30T13:45:28.132 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 9fa6f5e7d09 v0.11 2023-10-30T13:45:28.136 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.11 2023-10-30T13:45:28.184 INFO:tasks.workunit.client.0.smithi046.stdout:v0.12 2023-10-30T13:45:28.184 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 12 -eq 48 ] 2023-10-30T13:45:28.184 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.12 2023-10-30T13:45:28.184 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.12 2023-10-30T13:45:28.185 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.12 2023-10-30T13:45:44.422 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 5% (4/75) Updating files: 6% (5/75) Updating files: 8% (6/75) Updating files: 9% (7/75) Updating files: 10% (8/75) Updating files: 12% (9/75) Updating files: 13% (10/75) Updating files: 14% (11/75) Updating files: 16% (12/75) Updating files: 17% (13/75) Updating files: 18% (14/75) Updating files: 20% (15/75) Updating files: 21% (16/75) Updating files: 22% (17/75) Updating files: 24% (18/75) Updating files: 25% (19/75) Updating files: 26% (20/75) Updating files: 28% (21/75) Updating files: 29% (22/75) Updating files: 30% (23/75) Updating files: 32% (24/75) Updating files: 33% (25/75) Updating files: 34% (26/75) Updating files: 36% (27/75) Updating files: 37% (28/75) Updating files: 38% (29/75) Updating files: 40% (30/75) Updating files: 41% (31/75) Updating files: 42% (32/75) Updating files: 44% (33/75) Updating files: 45% (34/75) Updating files: 46% (35/75) Updating files: 48% (36/75) Updating files: 49% (37/75) Updating files: 50% (38/75) Updating files: 52% (39/75) Updating files: 53% (40/75) Updating files: 54% (41/75) Updating files: 56% (42/75) Updating files: 57% (43/75) Updating files: 58% (44/75) Updating files: 60% (45/75) Updating files: 61% (46/75) Updating files: 62% (47/75) Updating files: 64% (48/75) Updating files: 65% (49/75) Updating files: 66% (50/75) Updating files: 68% (51/75) Updating files: 69% (52/75) Updating files: 70% (53/75) Updating files: 72% (54/75) Updating files: 73% (55/75) Updating files: 74% (56/75) Updating files: 76% (57/75) Updating files: 77% (58/75) Updating files: 78% (59/75) Updating files: 80% (60/75) Updating files: 81% (61/75) Updating files: 82% (62/75) Updating files: 84% (63/75) Updating files: 85% (64/75) Updating files: 86% (65/75) Updating files: 88% (66/75) Updating files: 89% (67/75) Updating files: 90% (68/75) Updating files: 92% (69/75) Updating files: 93% (70/75) Updating files: 94% (71/75) Updating files: 96% (72/75) Updating files: 97% (73/75) Updating files: 98% (74/75) Updating files: 100% (75/75) Updating files: 100% (75/75), done. 2023-10-30T13:45:44.452 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at f524150822a v0.12 2023-10-30T13:45:44.456 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.12 2023-10-30T13:45:44.487 INFO:tasks.workunit.client.0.smithi046.stdout:v0.13 2023-10-30T13:45:44.488 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 13 -eq 48 ] 2023-10-30T13:45:44.488 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.13 2023-10-30T13:45:44.488 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.13 2023-10-30T13:45:44.488 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.13 2023-10-30T13:45:48.069 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 19% (26/133) Updating files: 20% (27/133) Updating files: 21% (28/133) Updating files: 22% (30/133) Updating files: 23% (31/133) Updating files: 24% (32/133) Updating files: 25% (34/133) Updating files: 26% (35/133) Updating files: 27% (36/133) Updating files: 28% (38/133) Updating files: 29% (39/133) Updating files: 30% (40/133) Updating files: 31% (42/133) Updating files: 32% (43/133) Updating files: 33% (44/133) Updating files: 34% (46/133) Updating files: 35% (47/133) Updating files: 36% (48/133) Updating files: 37% (50/133) Updating files: 38% (51/133) Updating files: 39% (52/133) Updating files: 40% (54/133) Updating files: 41% (55/133) Updating files: 42% (56/133) Updating files: 43% (58/133) Updating files: 44% (59/133) Updating files: 45% (60/133) Updating files: 46% (62/133) Updating files: 47% (63/133) Updating files: 48% (64/133) Updating files: 49% (66/133) Updating files: 50% (67/133) Updating files: 51% (68/133) Updating files: 52% (70/133) Updating files: 53% (71/133) Updating files: 54% (72/133) Updating files: 55% (74/133) Updating files: 56% (75/133) Updating files: 57% (76/133) Updating files: 58% (78/133) Updating files: 59% (79/133) Updating files: 60% (80/133) Updating files: 61% (82/133) Updating files: 62% (83/133) Updating files: 63% (84/133) Updating files: 64% (86/133) Updating files: 65% (87/133) Updating files: 66% (88/133) Updating files: 67% (90/133) Updating files: 68% (91/133) Updating files: 69% (92/133) Updating files: 70% (94/133) Updating files: 71% (95/133) Updating files: 72% (96/133) Updating files: 73% (98/133) Updating files: 74% (99/133) Updating files: 75% (100/133) Updating files: 76% (102/133) Updating files: 77% (103/133) Updating files: 78% (104/133) Updating files: 79% (106/133) Updating files: 80% (107/133) Updating files: 81% (108/133) Updating files: 82% (110/133) Updating files: 83% (111/133) Updating files: 84% (112/133) Updating files: 85% (114/133) Updating files: 86% (115/133) Updating files: 87% (116/133) Updating files: 88% (118/133) Updating files: 89% (119/133) Updating files: 90% (120/133) Updating files: 91% (122/133) Updating files: 92% (123/133) Updating files: 93% (124/133) Updating files: 94% (126/133) Updating files: 95% (127/133) Updating files: 96% (128/133) Updating files: 97% (130/133) Updating files: 98% (131/133) Updating files: 99% (132/133) Updating files: 100% (133/133) Updating files: 100% (133/133), done. 2023-10-30T13:45:48.100 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 2b32297267d v0.13 2023-10-30T13:45:48.105 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.13 2023-10-30T13:45:48.362 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 14 -eq 48 ] 2023-10-30T13:45:48.363 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.14 2023-10-30T13:45:48.363 INFO:tasks.workunit.client.0.smithi046.stdout:v0.14 2023-10-30T13:45:48.363 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.14 2023-10-30T13:45:48.364 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.14 2023-10-30T13:45:49.341 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:45:49.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:45:49.968 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 83% (126/151) Updating files: 84% (127/151) Updating files: 85% (129/151) Updating files: 86% (130/151) Updating files: 87% (132/151) Updating files: 88% (133/151) Updating files: 89% (135/151) Updating files: 90% (136/151) Updating files: 91% (138/151) Updating files: 92% (139/151) Updating files: 93% (141/151) Updating files: 94% (142/151) Updating files: 95% (144/151) Updating files: 96% (145/151) Updating files: 97% (147/151) Updating files: 98% (148/151) Updating files: 99% (150/151) Updating files: 100% (151/151) Updating files: 100% (151/151), done. 2023-10-30T13:45:50.000 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 1ec1e8d6fb6 v0.14 2023-10-30T13:45:50.005 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.14 2023-10-30T13:45:50.165 INFO:tasks.workunit.client.0.smithi046.stdout:v0.15 2023-10-30T13:45:50.166 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 15 -eq 48 ] 2023-10-30T13:45:50.166 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.15 2023-10-30T13:45:50.166 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.15 2023-10-30T13:45:50.166 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.15 2023-10-30T13:46:07.982 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 67% (76/112) Updating files: 68% (77/112) Updating files: 69% (78/112) Updating files: 70% (79/112) Updating files: 71% (80/112) Updating files: 72% (81/112) Updating files: 73% (82/112) Updating files: 74% (83/112) Updating files: 75% (84/112) Updating files: 76% (86/112) Updating files: 77% (87/112) Updating files: 78% (88/112) Updating files: 79% (89/112) Updating files: 80% (90/112) Updating files: 81% (91/112) Updating files: 82% (92/112) Updating files: 83% (93/112) Updating files: 84% (95/112) Updating files: 85% (96/112) Updating files: 86% (97/112) Updating files: 87% (98/112) Updating files: 88% (99/112) Updating files: 89% (100/112) Updating files: 90% (101/112) Updating files: 91% (102/112) Updating files: 92% (104/112) Updating files: 93% (105/112) Updating files: 94% (106/112) Updating files: 95% (107/112) Updating files: 96% (108/112) Updating files: 97% (109/112) Updating files: 98% (110/112) Updating files: 99% (111/112) Updating files: 100% (112/112) Updating files: 100% (112/112), done. 2023-10-30T13:46:08.015 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at ab3c421eb95 v0.15 2023-10-30T13:46:08.018 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.15 2023-10-30T13:46:08.292 INFO:tasks.workunit.client.0.smithi046.stdout:v0.16 2023-10-30T13:46:08.293 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 16 -eq 48 ] 2023-10-30T13:46:08.293 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.16 2023-10-30T13:46:08.293 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.16 2023-10-30T13:46:08.293 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.16 2023-10-30T13:46:09.335 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 38dbaa5175c v0.16 2023-10-30T13:46:09.340 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.16 2023-10-30T13:46:09.403 INFO:tasks.workunit.client.0.smithi046.stdout:v0.17 2023-10-30T13:46:09.403 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 17 -eq 48 ] 2023-10-30T13:46:09.404 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.17 2023-10-30T13:46:09.404 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.17 2023-10-30T13:46:09.404 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.17 2023-10-30T13:46:19.608 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:46:19.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:46:27.318 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 53% (116/217) Updating files: 54% (118/217) Updating files: 55% (120/217) Updating files: 56% (122/217) Updating files: 57% (124/217) Updating files: 58% (126/217) Updating files: 59% (129/217) Updating files: 60% (131/217) Updating files: 61% (133/217) Updating files: 62% (135/217) Updating files: 63% (137/217) Updating files: 64% (139/217) Updating files: 65% (142/217) Updating files: 66% (144/217) Updating files: 67% (146/217) Updating files: 68% (148/217) Updating files: 69% (150/217) Updating files: 70% (152/217) Updating files: 71% (155/217) Updating files: 72% (157/217) Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2023-10-30T13:46:27.598 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at dac3a8c35fb v0.17 2023-10-30T13:46:27.601 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.17 2023-10-30T13:46:28.444 INFO:tasks.workunit.client.0.smithi046.stdout:v0.18 2023-10-30T13:46:28.445 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 18 -eq 48 ] 2023-10-30T13:46:28.445 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.18 2023-10-30T13:46:28.445 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.18 2023-10-30T13:46:28.446 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.18 2023-10-30T13:46:36.791 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 5% (46/779) Updating files: 6% (47/779) Updating files: 6% (51/779) Updating files: 7% (55/779) Updating files: 8% (63/779) Updating files: 9% (71/779) Updating files: 9% (72/779) Updating files: 10% (78/779) Updating files: 11% (86/779) Updating files: 12% (94/779) Updating files: 13% (102/779) Updating files: 14% (110/779) Updating files: 15% (117/779) Updating files: 16% (125/779) Updating files: 17% (133/779) Updating files: 17% (136/779) Updating files: 18% (141/779) Updating files: 19% (149/779) Updating files: 20% (156/779) Updating files: 21% (164/779) Updating files: 22% (172/779) Updating files: 23% (180/779) Updating files: 24% (187/779) Updating files: 25% (195/779) Updating files: 26% (203/779) Updating files: 27% (211/779) Updating files: 28% (219/779) Updating files: 29% (226/779) Updating files: 30% (234/779) Updating files: 31% (242/779) Updating files: 32% (250/779) Updating files: 32% (255/779) Updating files: 33% (258/779) Updating files: 34% (265/779) Updating files: 35% (273/779) Updating files: 36% (281/779) Updating files: 37% (289/779) Updating files: 38% (297/779) Updating files: 39% (304/779) Updating files: 40% (312/779) Updating files: 41% (320/779) Updating files: 42% (328/779) Updating files: 43% (335/779) Updating files: 44% (343/779) Updating files: 45% (351/779) Updating files: 46% (359/779) Updating files: 47% (367/779) Updating files: 48% (374/779) Updating files: 49% (382/779) Updating files: 50% (390/779) Updating files: 51% (398/779) Updating files: 51% (403/779) Updating files: 52% (406/779) Updating files: 53% (413/779) Updating files: 54% (421/779) Updating files: 55% (429/779) Updating files: 56% (437/779) Updating files: 57% (445/779) Updating files: 58% (452/779) Updating files: 59% (460/779) Updating files: 60% (468/779) Updating files: 61% (476/779) Updating files: 62% (483/779) Updating files: 63% (491/779) Updating files: 64% (499/779) Updating files: 65% (507/779) Updating files: 66% (515/779) Updating files: 67% (522/779) Updating files: 68% (530/779) Updating files: 69% (538/779) Updating files: 70% (546/779) Updating files: 71% (554/779) Updating files: 72% (561/779) Updating files: 73% (569/779) Updating files: 74% (577/779) Updating files: 74% (580/779) Updating files: 75% (585/779) Updating files: 76% (593/779) Updating files: 77% (600/779) Updating files: 78% (608/779) Updating files: 79% (616/779) Updating files: 80% (624/779) Updating files: 81% (631/779) Updating files: 82% (639/779) Updating files: 83% (647/779) Updating files: 84% (655/779) Updating files: 85% (663/779) Updating files: 86% (670/779) Updating files: 87% (678/779) Updating files: 88% (686/779) Updating files: 89% (694/779) Updating files: 90% (702/779) Updating files: 91% (709/779) Updating files: 92% (717/779) Updating files: 93% (725/779) Updating files: 94% (733/779) Updating files: 95% (741/779) Updating files: 96% (748/779) Updating files: 97% (756/779) Updating files: 98% (764/779) Updating files: 99% (772/779) Updating files: 100% (779/779) Updating files: 100% (779/779), done. 2023-10-30T13:46:36.821 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 2f10137986a v0.18 2023-10-30T13:46:36.826 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.18 2023-10-30T13:46:37.102 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 19 -eq 48 ] 2023-10-30T13:46:37.102 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.19 2023-10-30T13:46:37.103 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.19 2023-10-30T13:46:37.103 INFO:tasks.workunit.client.0.smithi046.stdout:v0.19 2023-10-30T13:46:37.103 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.19 2023-10-30T13:46:42.142 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 46% (101/217) Updating files: 47% (102/217) Updating files: 48% (105/217) Updating files: 49% (107/217) Updating files: 50% (109/217) Updating files: 51% (111/217) Updating files: 52% (113/217) Updating files: 53% (116/217) Updating files: 54% (118/217) Updating files: 55% (120/217) Updating files: 56% (122/217) Updating files: 57% (124/217) Updating files: 58% (126/217) Updating files: 59% (129/217) Updating files: 60% (131/217) Updating files: 61% (133/217) Updating files: 62% (135/217) Updating files: 63% (137/217) Updating files: 64% (139/217) Updating files: 65% (142/217) Updating files: 66% (144/217) Updating files: 67% (146/217) Updating files: 68% (148/217) Updating files: 69% (150/217) Updating files: 70% (152/217) Updating files: 71% (155/217) Updating files: 72% (157/217) Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2023-10-30T13:46:42.168 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 98f5be53f85 v0.19 2023-10-30T13:46:42.172 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.19 2023-10-30T13:46:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 20 -eq 48 ] 2023-10-30T13:46:42.290 INFO:tasks.workunit.client.0.smithi046.stdout:v0.20 2023-10-30T13:46:42.290 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.20 2023-10-30T13:46:42.291 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.20 2023-10-30T13:46:42.291 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.20 2023-10-30T13:46:46.869 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 29% (72/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 83% (205/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2023-10-30T13:46:47.152 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 266d2076b6c debian: fix up debian scripts 2023-10-30T13:46:47.155 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.20 2023-10-30T13:46:47.598 INFO:tasks.workunit.client.0.smithi046.stdout:v0.21 2023-10-30T13:46:47.599 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 21 -eq 48 ] 2023-10-30T13:46:47.599 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.21 2023-10-30T13:46:47.599 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.21 2023-10-30T13:46:47.599 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.21 2023-10-30T13:46:49.718 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:46:49.722 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:47:08.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.191+0000 7fd17c046640 1 Processor -- start 2023-10-30T13:47:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.191+0000 7fd17c046640 1 -- start start 2023-10-30T13:47:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd17c046640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd17414fd50 0x7fd1740c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd17c046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd174071960 0x7fd1740cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd17c046640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1741560a0 con 0x7fd1740cb550 2023-10-30T13:47:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd17c046640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd174156380 con 0x7fd17414fd50 2023-10-30T13:47:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd17c046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd174156660 con 0x7fd174071960 2023-10-30T13:47:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd179de2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd17414fd50 0x7fd1740c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd179de2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd17414fd50 0x7fd1740c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:46120/0 (socket says 172.21.15.46:46120) 2023-10-30T13:47:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd179de2640 1 -- 172.21.15.46:0/3347002558 learned_addr learned my addr 172.21.15.46:0/3347002558 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd17a5e3640 1 --2- 172.21.15.46:0/3347002558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd174071960 0x7fd1740cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd179de2640 1 -- 172.21.15.46:0/3347002558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd174071960 msgr2=0x7fd1740cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:08.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd179de2640 1 --2- 172.21.15.46:0/3347002558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd174071960 0x7fd1740cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd179de2640 1 -- 172.21.15.46:0/3347002558 >> v1:172.21.15.154:6789/0 conn(0x7fd1740cb550 legacy=0x7fd174071250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd179de2640 1 -- 172.21.15.46:0/3347002558 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd174156940 con 0x7fd17414fd50 2023-10-30T13:47:08.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.195+0000 7fd17a5e3640 1 --2- 172.21.15.46:0/3347002558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd174071960 0x7fd1740cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:47:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.199+0000 7fd179de2640 1 --2- 172.21.15.46:0/3347002558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17414fd50 0x7fd1740c65d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd17000aa10 tx=0x7fd17001d8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdadbb083af1404e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.199+0000 7fd178de0640 1 -- 172.21.15.46:0/3347002558 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1700a6900 con 0x7fd17414fd50 2023-10-30T13:47:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.199+0000 7fd178de0640 1 -- 172.21.15.46:0/3347002558 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1700a6a60 con 0x7fd17414fd50 2023-10-30T13:47:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.199+0000 7fd178de0640 1 -- 172.21.15.46:0/3347002558 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1700a6df0 con 0x7fd17414fd50 2023-10-30T13:47:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.199+0000 7fd17c046640 1 -- 172.21.15.46:0/3347002558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17414fd50 msgr2=0x7fd1740c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.199+0000 7fd17c046640 1 --2- 172.21.15.46:0/3347002558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17414fd50 0x7fd1740c65d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd17000aa10 tx=0x7fd17001d8e0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.199+0000 7fd17c046640 1 -- 172.21.15.46:0/3347002558 shutdown_connections 2023-10-30T13:47:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.199+0000 7fd17c046640 1 --2- 172.21.15.46:0/3347002558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd174071960 0x7fd1740cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.199+0000 7fd17c046640 1 --2- 172.21.15.46:0/3347002558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17414fd50 0x7fd1740c65d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.199+0000 7fd17c046640 1 -- 172.21.15.46:0/3347002558 >> 172.21.15.46:0/3347002558 conn(0x7fd17406c6e0 msgr2=0x7fd174069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.199+0000 7fd17c046640 1 -- 172.21.15.46:0/3347002558 shutdown_connections 2023-10-30T13:47:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.199+0000 7fd17c046640 1 -- 172.21.15.46:0/3347002558 wait complete. 2023-10-30T13:47:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17c046640 1 Processor -- start 2023-10-30T13:47:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17c046640 1 -- start start 2023-10-30T13:47:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17c046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd174071960 0x7fd1741f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17c046640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1740cb550 0x7fd1740c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17c046640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd17414fd50 0x7fd1741f5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17c046640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741f2340 con 0x7fd17414fd50 2023-10-30T13:47:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17c046640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741f48f0 con 0x7fd1740cb550 2023-10-30T13:47:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17c046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd174152d80 con 0x7fd174071960 2023-10-30T13:47:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17a5e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd17414fd50 0x7fd1741f5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17a5e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd17414fd50 0x7fd1741f5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40148/0 (socket says 172.21.15.46:40148) 2023-10-30T13:47:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17a5e3640 1 -- 172.21.15.46:0/2058093396 learned_addr learned my addr 172.21.15.46:0/2058093396 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd1795e1640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1740cb550 0x7fd1740c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd179de2640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd174071960 0x7fd1741f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17a5e3640 1 -- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd174071960 msgr2=0x7fd1741f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17a5e3640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd174071960 0x7fd1741f1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17a5e3640 1 -- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1740cb550 msgr2=0x7fd1740c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17a5e3640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1740cb550 0x7fd1740c79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17a5e3640 1 -- 172.21.15.46:0/2058093396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c009a70 con 0x7fd17414fd50 2023-10-30T13:47:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17a5e3640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd17414fd50 0x7fd1741f5060 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00efc0 tx=0x7fd16c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd16a7fc640 1 -- 172.21.15.46:0/2058093396 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c009cd0 con 0x7fd17414fd50 2023-10-30T13:47:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd16a7fc640 1 -- 172.21.15.46:0/2058093396 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c00f040 con 0x7fd17414fd50 2023-10-30T13:47:08.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17c046640 1 -- 172.21.15.46:0/2058093396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17000a670 con 0x7fd17414fd50 2023-10-30T13:47:08.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd16a7fc640 1 -- 172.21.15.46:0/2058093396 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c0039a0 con 0x7fd17414fd50 2023-10-30T13:47:08.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17c046640 1 -- 172.21.15.46:0/2058093396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd174153020 con 0x7fd17414fd50 2023-10-30T13:47:08.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.203+0000 7fd17c046640 1 -- 172.21.15.46:0/2058093396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17404e170 con 0x7fd17414fd50 2023-10-30T13:47:08.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.207+0000 7fd16a7fc640 1 -- 172.21.15.46:0/2058093396 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd16c014480 con 0x7fd17414fd50 2023-10-30T13:47:08.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.207+0000 7fd16a7fc640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1580664a0 0x7fd158068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.207+0000 7fd16a7fc640 1 -- 172.21.15.46:0/2058093396 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fd16c084e50 con 0x7fd17414fd50 2023-10-30T13:47:08.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.207+0000 7fd179de2640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1580664a0 0x7fd158068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.207+0000 7fd179de2640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1580664a0 0x7fd158068960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd1700b0000 tx=0x7fd170008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:08.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.211+0000 7fd16a7fc640 1 -- 172.21.15.46:0/2058093396 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd16c051e30 con 0x7fd17414fd50 2023-10-30T13:47:08.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.399+0000 7fd17c046640 1 -- 172.21.15.46:0/2058093396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1740c7180 con 0x7fd17414fd50 2023-10-30T13:47:08.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd16a7fc640 1 -- 172.21.15.46:0/2058093396 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6522 (secure 0 0 0) 0x7fd16c018090 con 0x7fd17414fd50 2023-10-30T13:47:08.404 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:08.404 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6702,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:45:44.251897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:08.405 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 110 2023-10-30T13:47:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd17c046640 1 -- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1580664a0 msgr2=0x7fd158068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd17c046640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1580664a0 0x7fd158068960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd1700b0000 tx=0x7fd170008040 comp rx=0 tx=0).stop 2023-10-30T13:47:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd17c046640 1 -- 172.21.15.46:0/2058093396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd17414fd50 msgr2=0x7fd1741f5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd17c046640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd17414fd50 0x7fd1741f5060 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00efc0 tx=0x7fd16c00c490 comp rx=0 tx=0).stop 2023-10-30T13:47:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd17c046640 1 -- 172.21.15.46:0/2058093396 shutdown_connections 2023-10-30T13:47:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd17c046640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1580664a0 0x7fd158068960 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd17c046640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd17414fd50 0x7fd1741f5060 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd17c046640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1740cb550 0x7fd1740c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd17c046640 1 --2- 172.21.15.46:0/2058093396 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd174071960 0x7fd1741f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd17c046640 1 -- 172.21.15.46:0/2058093396 >> 172.21.15.46:0/2058093396 conn(0x7fd17406c6e0 msgr2=0x7fd17415fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd17c046640 1 -- 172.21.15.46:0/2058093396 shutdown_connections 2023-10-30T13:47:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.403+0000 7fd17c046640 1 -- 172.21.15.46:0/2058093396 wait complete. 2023-10-30T13:47:08.427 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T13:47:08.427 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:47:08.427 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:47:08.448 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:47:08.448 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa64ea79640 1 Processor -- start 2023-10-30T13:47:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa64ea79640 1 -- start start 2023-10-30T13:47:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa64ea79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa64806e470 0x7fa64806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa64ea79640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64806afa0 con 0x7fa6480c8490 2023-10-30T13:47:08.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa64ea79640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa648154150 con 0x7fa648153d70 2023-10-30T13:47:08.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa64ea79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64806d780 con 0x7fa64806e470 2023-10-30T13:47:08.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa64d016640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa6480c8490 0x7fa6480c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38508/0 (socket says 172.21.15.46:38508) 2023-10-30T13:47:08.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa64d016640 1 -- 172.21.15.46:0/495437616 learned_addr learned my addr 172.21.15.46:0/495437616 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:47:08.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa647fff640 1 --2- 172.21.15.46:0/495437616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa64806e470 0x7fa64806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580076076 0 0) 0x7fa64806afa0 con 0x7fa6480c8490 2023-10-30T13:47:08.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa638003530 con 0x7fa6480c8490 2023-10-30T13:47:08.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827576184 0 0) 0x7fa648154150 con 0x7fa648153d70 2023-10-30T13:47:08.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa64806afa0 con 0x7fa648153d70 2023-10-30T13:47:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 583478327 0 0) 0x7fa638003530 con 0x7fa6480c8490 2023-10-30T13:47:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa64806e470 msgr2=0x7fa64806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa6477fe640 1 --2- 172.21.15.46:0/495437616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa64806e470 0x7fa64806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 >> v1:172.21.15.154:6789/0 conn(0x7fa648153d70 legacy=0x7fa6480cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64806e030 con 0x7fa6480c8490 2023-10-30T13:47:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.631+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fa640001a00 con 0x7fa6480c8490 2023-10-30T13:47:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 >> v1:172.21.15.46:6789/0 conn(0x7fa6480c8490 legacy=0x7fa6480c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa6477fe640 1 --2- 172.21.15.46:0/495437616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa64806e470 0x7fa638003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa6477fe640 1 --2- 172.21.15.46:0/495437616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6380040f0 0x7fa6380047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa6477fe640 1 --2- 172.21.15.46:0/495437616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d10 0x7fa638005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64806d780 con 0x7fa638004d10 2023-10-30T13:47:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63800bad0 con 0x7fa6380040f0 2023-10-30T13:47:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63800bc90 con 0x7fa64806e470 2023-10-30T13:47:08.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa64d016640 1 --2- 172.21.15.46:0/495437616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d10 0x7fa638005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa64c815640 1 --2- 172.21.15.46:0/495437616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa64806e470 0x7fa638003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa647fff640 1 --2- 172.21.15.46:0/495437616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6380040f0 0x7fa6380047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa64d016640 1 -- 172.21.15.46:0/495437616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa64806e470 msgr2=0x7fa638003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa64d016640 1 --2- 172.21.15.46:0/495437616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa64806e470 0x7fa638003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa64d016640 1 -- 172.21.15.46:0/495437616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6380040f0 msgr2=0x7fa6380047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa64d016640 1 --2- 172.21.15.46:0/495437616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6380040f0 0x7fa6380047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa64d016640 1 -- 172.21.15.46:0/495437616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa638003530 con 0x7fa638004d10 2023-10-30T13:47:08.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa64d016640 1 --2- 172.21.15.46:0/495437616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d10 0x7fa638005190 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fa64000a710 tx=0x7fa64001f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd7673250136dce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:08.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6400ac610 con 0x7fa638004d10 2023-10-30T13:47:08.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6400ac770 con 0x7fa638004d10 2023-10-30T13:47:08.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.635+0000 7fa6477fe640 1 -- 172.21.15.46:0/495437616 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6400acba0 con 0x7fa638004d10 2023-10-30T13:47:08.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 -- 172.21.15.46:0/495437616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d10 msgr2=0x7fa638005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 --2- 172.21.15.46:0/495437616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d10 0x7fa638005190 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fa64000a710 tx=0x7fa64001f960 comp rx=0 tx=0).stop 2023-10-30T13:47:08.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64d016640 1 -- 172.21.15.46:0/495437616 reap_dead start 2023-10-30T13:47:08.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 -- 172.21.15.46:0/495437616 shutdown_connections 2023-10-30T13:47:08.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 -- 172.21.15.46:0/495437616 >> 172.21.15.46:0/495437616 conn(0x7fa648071ea0 msgr2=0x7fa648065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 -- 172.21.15.46:0/495437616 shutdown_connections 2023-10-30T13:47:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 -- 172.21.15.46:0/495437616 wait complete. 2023-10-30T13:47:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 Processor -- start 2023-10-30T13:47:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 -- start start 2023-10-30T13:47:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa638004d10 0x7fa6480ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6481ef760 0x7fa6480cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa64814cfe0 0x7fa6480b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64814c500 con 0x7fa64814cfe0 2023-10-30T13:47:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6480ad970 con 0x7fa6481ef760 2023-10-30T13:47:08.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64ea79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64814dba0 con 0x7fa638004d10 2023-10-30T13:47:08.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64d016640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa64814cfe0 0x7fa6480b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64d016640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa64814cfe0 0x7fa6480b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40166/0 (socket says 172.21.15.46:40166) 2023-10-30T13:47:08.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64d016640 1 -- 172.21.15.46:0/1952418972 learned_addr learned my addr 172.21.15.46:0/1952418972 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:08.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa64c815640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa638004d10 0x7fa6480ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.639+0000 7fa647fff640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6481ef760 0x7fa6480cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa64d016640 1 -- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa638004d10 msgr2=0x7fa6480ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa64d016640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa638004d10 0x7fa6480ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa64d016640 1 -- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6481ef760 msgr2=0x7fa6480cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa64d016640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6481ef760 0x7fa6480cc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa64d016640 1 -- 172.21.15.46:0/1952418972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64814c7a0 con 0x7fa64814cfe0 2023-10-30T13:47:08.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa647fff640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6481ef760 0x7fa6480cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:08.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa64d016640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa64814cfe0 0x7fa6480b0d00 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa64000a710 tx=0x7fa638005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:08.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa6457fa640 1 -- 172.21.15.46:0/1952418972 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa640001210 con 0x7fa64814cfe0 2023-10-30T13:47:08.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa6457fa640 1 -- 172.21.15.46:0/1952418972 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6400ac410 con 0x7fa64814cfe0 2023-10-30T13:47:08.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa6457fa640 1 -- 172.21.15.46:0/1952418972 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6400ac580 con 0x7fa64814cfe0 2023-10-30T13:47:08.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa64ea79640 1 -- 172.21.15.46:0/1952418972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64814ca80 con 0x7fa64814cfe0 2023-10-30T13:47:08.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa64ea79640 1 -- 172.21.15.46:0/1952418972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64814cd60 con 0x7fa64814cfe0 2023-10-30T13:47:08.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa64ea79640 1 -- 172.21.15.46:0/1952418972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64804e170 con 0x7fa64814cfe0 2023-10-30T13:47:08.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa6457fa640 1 -- 172.21.15.46:0/1952418972 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa6400be070 con 0x7fa64814cfe0 2023-10-30T13:47:08.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa6457fa640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa62c0664a0 0x7fa62c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:08.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa6457fa640 1 -- 172.21.15.46:0/1952418972 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fa640122830 con 0x7fa64814cfe0 2023-10-30T13:47:08.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.643+0000 7fa64c815640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa62c0664a0 0x7fa62c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:08.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.647+0000 7fa64c815640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa62c0664a0 0x7fa62c068960 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa63c000c10 tx=0x7fa63c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:08.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.647+0000 7fa6457fa640 1 -- 172.21.15.46:0/1952418972 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6400ef760 con 0x7fa64814cfe0 2023-10-30T13:47:08.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 -- 172.21.15.46:0/1952418972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa648150020 con 0x7fa64814cfe0 2023-10-30T13:47:08.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa6457fa640 1 -- 172.21.15.46:0/1952418972 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6522 (secure 0 0 0) 0x7fa6400f2d80 con 0x7fa64814cfe0 2023-10-30T13:47:08.841 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:08.842 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6702,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:45:44.251897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:08.842 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 110 2023-10-30T13:47:08.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 -- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa62c0664a0 msgr2=0x7fa62c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa62c0664a0 0x7fa62c068960 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa63c000c10 tx=0x7fa63c009040 comp rx=0 tx=0).stop 2023-10-30T13:47:08.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 -- 172.21.15.46:0/1952418972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa64814cfe0 msgr2=0x7fa6480b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:08.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa64814cfe0 0x7fa6480b0d00 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa64000a710 tx=0x7fa638005800 comp rx=0 tx=0).stop 2023-10-30T13:47:08.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 -- 172.21.15.46:0/1952418972 shutdown_connections 2023-10-30T13:47:08.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa62c0664a0 0x7fa62c068960 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa64814cfe0 0x7fa6480b0d00 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6481ef760 0x7fa6480cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 --2- 172.21.15.46:0/1952418972 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa638004d10 0x7fa6480ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:08.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 -- 172.21.15.46:0/1952418972 >> 172.21.15.46:0/1952418972 conn(0x7fa648071ea0 msgr2=0x7fa6480c9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:08.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 -- 172.21.15.46:0/1952418972 shutdown_connections 2023-10-30T13:47:08.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:08.839+0000 7fa64ea79640 1 -- 172.21.15.46:0/1952418972 wait complete. 2023-10-30T13:47:08.863 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:47:10.863 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:11.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.015+0000 7f12dea67640 1 Processor -- start 2023-10-30T13:47:11.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.015+0000 7f12dea67640 1 -- start start 2023-10-30T13:47:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dea67640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12d81538f0 0x7f12d8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dea67640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f12d815cd50 0x7f12d80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dea67640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d815d300 con 0x7f12d80cefa0 2023-10-30T13:47:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dea67640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d815d5e0 con 0x7f12d81538f0 2023-10-30T13:47:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dea67640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d815d8c0 con 0x7f12d815cd50 2023-10-30T13:47:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dda65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12d81538f0 0x7f12d8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dda65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12d81538f0 0x7f12d8159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40176/0 (socket says 172.21.15.46:40176) 2023-10-30T13:47:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12de266640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f12d80cefa0 0x7f12d80cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38516/0 (socket says 172.21.15.46:38516) 2023-10-30T13:47:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dda65640 1 -- 172.21.15.46:0/3956755989 learned_addr learned my addr 172.21.15.46:0/3956755989 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dd264640 1 --2- 172.21.15.46:0/3956755989 >> v2:172.21.15.154:3300/0 conn(0x7f12d815cd50 0x7f12d80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dda65640 1 -- 172.21.15.46:0/3956755989 >> v2:172.21.15.154:3300/0 conn(0x7f12d815cd50 msgr2=0x7f12d80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dda65640 1 --2- 172.21.15.46:0/3956755989 >> v2:172.21.15.154:3300/0 conn(0x7f12d815cd50 0x7f12d80c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dda65640 1 -- 172.21.15.46:0/3956755989 >> v1:172.21.15.46:6789/0 conn(0x7f12d80cefa0 legacy=0x7f12d80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dda65640 1 -- 172.21.15.46:0/3956755989 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d815dba0 con 0x7f12d81538f0 2023-10-30T13:47:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.019+0000 7f12dca63640 1 -- 172.21.15.46:0/3956755989 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255735731 0 0) 0x7f12d815d300 con 0x7f12d80cefa0 2023-10-30T13:47:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dda65640 1 --2- 172.21.15.46:0/3956755989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d81538f0 0x7f12d8159fb0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f12d8072e90 tx=0x7f12d401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a37d027eba6c28f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dca63640 1 -- 172.21.15.46:0/3956755989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d40a89f0 con 0x7f12d81538f0 2023-10-30T13:47:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dca63640 1 -- 172.21.15.46:0/3956755989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d40a8b50 con 0x7f12d81538f0 2023-10-30T13:47:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dca63640 1 -- 172.21.15.46:0/3956755989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d40b0440 con 0x7f12d81538f0 2023-10-30T13:47:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dea67640 1 -- 172.21.15.46:0/3956755989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d81538f0 msgr2=0x7f12d8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dea67640 1 --2- 172.21.15.46:0/3956755989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d81538f0 0x7f12d8159fb0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f12d8072e90 tx=0x7f12d401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dea67640 1 -- 172.21.15.46:0/3956755989 shutdown_connections 2023-10-30T13:47:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dea67640 1 --2- 172.21.15.46:0/3956755989 >> v2:172.21.15.154:3300/0 conn(0x7f12d815cd50 0x7f12d80c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dea67640 1 --2- 172.21.15.46:0/3956755989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d81538f0 0x7f12d8159fb0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dea67640 1 -- 172.21.15.46:0/3956755989 >> 172.21.15.46:0/3956755989 conn(0x7f12d806a640 msgr2=0x7f12d8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:11.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dea67640 1 -- 172.21.15.46:0/3956755989 shutdown_connections 2023-10-30T13:47:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dea67640 1 -- 172.21.15.46:0/3956755989 wait complete. 2023-10-30T13:47:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.023+0000 7f12dea67640 1 Processor -- start 2023-10-30T13:47:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dea67640 1 -- start start 2023-10-30T13:47:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dea67640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d80cefa0 0x7f12d8146820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dea67640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d81538f0 0x7f12d80b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:11.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dea67640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12d815cd50 0x7f12d80b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:11.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dea67640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d815bea0 con 0x7f12d80cefa0 2023-10-30T13:47:11.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dea67640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81487e0 con 0x7f12d815cd50 2023-10-30T13:47:11.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dea67640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81eece0 con 0x7f12d81538f0 2023-10-30T13:47:11.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dda65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d80cefa0 0x7f12d8146820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:11.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dda65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d80cefa0 0x7f12d8146820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40188/0 (socket says 172.21.15.46:40188) 2023-10-30T13:47:11.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dda65640 1 -- 172.21.15.46:0/3920848151 learned_addr learned my addr 172.21.15.46:0/3920848151 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:11.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dd264640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d81538f0 0x7f12d80b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:11.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12de266640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12d815cd50 0x7f12d80b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:11.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dda65640 1 -- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d81538f0 msgr2=0x7f12d80b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:11.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dda65640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d81538f0 0x7f12d80b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dda65640 1 -- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12d815cd50 msgr2=0x7f12d80b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:11.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dda65640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12d815cd50 0x7f12d80b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dda65640 1 -- 172.21.15.46:0/3920848151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d40094d0 con 0x7f12d80cefa0 2023-10-30T13:47:11.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dda65640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d80cefa0 0x7f12d8146820 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f12d401b900 tx=0x7f12d401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:11.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12ce7fc640 1 -- 172.21.15.46:0/3920848151 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d40ba600 con 0x7f12d80cefa0 2023-10-30T13:47:11.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12ce7fc640 1 -- 172.21.15.46:0/3920848151 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d40ba760 con 0x7f12d80cefa0 2023-10-30T13:47:11.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12ce7fc640 1 -- 172.21.15.46:0/3920848151 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d40bab10 con 0x7f12d80cefa0 2023-10-30T13:47:11.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dea67640 1 -- 172.21.15.46:0/3920848151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d8148500 con 0x7f12d80cefa0 2023-10-30T13:47:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dea67640 1 -- 172.21.15.46:0/3920848151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d81ef260 con 0x7f12d80cefa0 2023-10-30T13:47:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.027+0000 7f12dea67640 1 -- 172.21.15.46:0/3920848151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d804e170 con 0x7f12d80cefa0 2023-10-30T13:47:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.031+0000 7f12ce7fc640 1 -- 172.21.15.46:0/3920848151 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f12d40bac70 con 0x7f12d80cefa0 2023-10-30T13:47:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.031+0000 7f12ce7fc640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12bc0664a0 0x7f12bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:11.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.031+0000 7f12ce7fc640 1 -- 172.21.15.46:0/3920848151 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f12d41229c0 con 0x7f12d80cefa0 2023-10-30T13:47:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.031+0000 7f12dd264640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12bc0664a0 0x7f12bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.031+0000 7f12dd264640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12bc0664a0 0x7f12bc068960 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f12d8157c30 tx=0x7f12c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.035+0000 7f12ce7fc640 1 -- 172.21.15.46:0/3920848151 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f12d40ef8f0 con 0x7f12d80cefa0 2023-10-30T13:47:11.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.223+0000 7f12dea67640 1 -- 172.21.15.46:0/3920848151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12d80b2290 con 0x7f12d80cefa0 2023-10-30T13:47:11.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12ce7fc640 1 -- 172.21.15.46:0/3920848151 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f12d40b5090 con 0x7f12d80cefa0 2023-10-30T13:47:11.229 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:11.229 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6702,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:45:44.251897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:11.229 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 110 2023-10-30T13:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12dea67640 1 -- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12bc0664a0 msgr2=0x7f12bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12dea67640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12bc0664a0 0x7f12bc068960 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f12d8157c30 tx=0x7f12c8008040 comp rx=0 tx=0).stop 2023-10-30T13:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12dea67640 1 -- 172.21.15.46:0/3920848151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d80cefa0 msgr2=0x7f12d8146820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12dea67640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d80cefa0 0x7f12d8146820 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f12d401b900 tx=0x7f12d401ba90 comp rx=0 tx=0).stop 2023-10-30T13:47:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12dea67640 1 -- 172.21.15.46:0/3920848151 shutdown_connections 2023-10-30T13:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12dea67640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f12bc0664a0 0x7f12bc068960 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12dea67640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12d815cd50 0x7f12d80b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12dea67640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12d81538f0 0x7f12d80b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12dea67640 1 --2- 172.21.15.46:0/3920848151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d80cefa0 0x7f12d8146820 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12dea67640 1 -- 172.21.15.46:0/3920848151 >> 172.21.15.46:0/3920848151 conn(0x7f12d806a640 msgr2=0x7f12d8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:11.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12dea67640 1 -- 172.21.15.46:0/3920848151 shutdown_connections 2023-10-30T13:47:11.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:11.227+0000 7f12dea67640 1 -- 172.21.15.46:0/3920848151 wait complete. 2023-10-30T13:47:11.250 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:47:13.251 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:13.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.403+0000 7f5f25165640 1 Processor -- start 2023-10-30T13:47:13.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.403+0000 7f5f25165640 1 -- start start 2023-10-30T13:47:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f25165640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5f20153bf0 0x7f5f20071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f25165640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f200c6fe0 0x7f5f200c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f25165640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f200700b0 con 0x7f5f200c79e0 2023-10-30T13:47:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f25165640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f200c6810 con 0x7f5f200c6fe0 2023-10-30T13:47:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f25165640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f20159900 con 0x7f5f20153bf0 2023-10-30T13:47:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f17fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f200c6fe0 0x7f5f200c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f17fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f200c6fe0 0x7f5f200c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40190/0 (socket says 172.21.15.46:40190) 2023-10-30T13:47:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f17fff640 1 -- 172.21.15.46:0/3832414244 learned_addr learned my addr 172.21.15.46:0/3832414244 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f1ed9d640 1 --2- 172.21.15.46:0/3832414244 >> v2:172.21.15.154:3300/0 conn(0x7f5f20153bf0 0x7f5f20071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f17fff640 1 -- 172.21.15.46:0/3832414244 >> v2:172.21.15.154:3300/0 conn(0x7f5f20153bf0 msgr2=0x7f5f20071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:13.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f17fff640 1 --2- 172.21.15.46:0/3832414244 >> v2:172.21.15.154:3300/0 conn(0x7f5f20153bf0 0x7f5f20071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f17fff640 1 -- 172.21.15.46:0/3832414244 >> v1:172.21.15.154:6789/0 conn(0x7f5f200c79e0 legacy=0x7f5f200cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.407+0000 7f5f17fff640 1 -- 172.21.15.46:0/3832414244 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f200c6530 con 0x7f5f200c6fe0 2023-10-30T13:47:13.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f17fff640 1 --2- 172.21.15.46:0/3832414244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f200c6fe0 0x7f5f200c74a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f5f200c9150 tx=0x7f5f0801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fdb9e87b79ce4a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:13.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f1e59c640 1 -- 172.21.15.46:0/3832414244 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f080a89f0 con 0x7f5f200c6fe0 2023-10-30T13:47:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f1e59c640 1 -- 172.21.15.46:0/3832414244 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f080a8b50 con 0x7f5f200c6fe0 2023-10-30T13:47:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f1e59c640 1 -- 172.21.15.46:0/3832414244 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f080b0440 con 0x7f5f200c6fe0 2023-10-30T13:47:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f25165640 1 -- 172.21.15.46:0/3832414244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f200c6fe0 msgr2=0x7f5f200c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f25165640 1 --2- 172.21.15.46:0/3832414244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f200c6fe0 0x7f5f200c74a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f5f200c9150 tx=0x7f5f0801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f25165640 1 -- 172.21.15.46:0/3832414244 shutdown_connections 2023-10-30T13:47:13.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f25165640 1 --2- 172.21.15.46:0/3832414244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f200c6fe0 0x7f5f200c74a0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f25165640 1 --2- 172.21.15.46:0/3832414244 >> v2:172.21.15.154:3300/0 conn(0x7f5f20153bf0 0x7f5f20071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f25165640 1 -- 172.21.15.46:0/3832414244 >> 172.21.15.46:0/3832414244 conn(0x7f5f2006c6e0 msgr2=0x7f5f20069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:13.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f25165640 1 -- 172.21.15.46:0/3832414244 shutdown_connections 2023-10-30T13:47:13.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f25165640 1 -- 172.21.15.46:0/3832414244 wait complete. 2023-10-30T13:47:13.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f25165640 1 Processor -- start 2023-10-30T13:47:13.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.411+0000 7f5f25165640 1 -- start start 2023-10-30T13:47:13.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f25165640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f200c6fe0 0x7f5f200c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f25165640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f200c79e0 0x7f5f20149b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f25165640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f20153bf0 0x7f5f20144250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f25165640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f201ef790 con 0x7f5f200c79e0 2023-10-30T13:47:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f25165640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f201ee930 con 0x7f5f200c6fe0 2023-10-30T13:47:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f25165640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f201eec10 con 0x7f5f20153bf0 2023-10-30T13:47:13.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f200c79e0 0x7f5f20149b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f200c79e0 0x7f5f20149b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40202/0 (socket says 172.21.15.46:40202) 2023-10-30T13:47:13.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f17fff640 1 -- 172.21.15.46:0/190745422 learned_addr learned my addr 172.21.15.46:0/190745422 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:13.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f1ed9d640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f200c6fe0 0x7f5f200c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f1f59e640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f20153bf0 0x7f5f20144250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f17fff640 1 -- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f20153bf0 msgr2=0x7f5f20144250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f17fff640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f20153bf0 0x7f5f20144250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f17fff640 1 -- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f200c6fe0 msgr2=0x7f5f200c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f17fff640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f200c6fe0 0x7f5f200c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f17fff640 1 -- 172.21.15.46:0/190745422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f080094d0 con 0x7f5f200c79e0 2023-10-30T13:47:13.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f1ed9d640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f200c6fe0 0x7f5f200c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:13.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f17fff640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f200c79e0 0x7f5f20149b40 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f5f080b3040 tx=0x7f5f0801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:13.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f177fe640 1 -- 172.21.15.46:0/190745422 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f080ba600 con 0x7f5f200c79e0 2023-10-30T13:47:13.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f177fe640 1 -- 172.21.15.46:0/190745422 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f080ba760 con 0x7f5f200c79e0 2023-10-30T13:47:13.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f177fe640 1 -- 172.21.15.46:0/190745422 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f080bab10 con 0x7f5f200c79e0 2023-10-30T13:47:13.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f25165640 1 -- 172.21.15.46:0/190745422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f201eeef0 con 0x7f5f200c79e0 2023-10-30T13:47:13.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f25165640 1 -- 172.21.15.46:0/190745422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f201ef4b0 con 0x7f5f200c79e0 2023-10-30T13:47:13.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.415+0000 7f5f25165640 1 -- 172.21.15.46:0/190745422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2004e170 con 0x7f5f200c79e0 2023-10-30T13:47:13.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.419+0000 7f5f177fe640 1 -- 172.21.15.46:0/190745422 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5f080bac70 con 0x7f5f200c79e0 2023-10-30T13:47:13.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.419+0000 7f5f177fe640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f000664a0 0x7f5f00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:13.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.419+0000 7f5f177fe640 1 -- 172.21.15.46:0/190745422 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f5f08122870 con 0x7f5f200c79e0 2023-10-30T13:47:13.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.419+0000 7f5f1ed9d640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f000664a0 0x7f5f00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:13.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.419+0000 7f5f1ed9d640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f000664a0 0x7f5f00068960 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009b10 tx=0x7f5f10008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:13.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.423+0000 7f5f177fe640 1 -- 172.21.15.46:0/190745422 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5f080ef9e0 con 0x7f5f200c79e0 2023-10-30T13:47:13.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.611+0000 7f5f25165640 1 -- 172.21.15.46:0/190745422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f201517f0 con 0x7f5f200c79e0 2023-10-30T13:47:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.611+0000 7f5f177fe640 1 -- 172.21.15.46:0/190745422 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f5f080f3000 con 0x7f5f200c79e0 2023-10-30T13:47:13.615 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:13.615 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6702,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:45:44.251897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 110 2023-10-30T13:47:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.615+0000 7f5f25165640 1 -- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f000664a0 msgr2=0x7f5f00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.615+0000 7f5f25165640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f000664a0 0x7f5f00068960 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009b10 tx=0x7f5f10008040 comp rx=0 tx=0).stop 2023-10-30T13:47:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.615+0000 7f5f25165640 1 -- 172.21.15.46:0/190745422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f200c79e0 msgr2=0x7f5f20149b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:13.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.615+0000 7f5f25165640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f200c79e0 0x7f5f20149b40 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f5f080b3040 tx=0x7f5f0801ba90 comp rx=0 tx=0).stop 2023-10-30T13:47:13.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.615+0000 7f5f25165640 1 -- 172.21.15.46:0/190745422 shutdown_connections 2023-10-30T13:47:13.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.615+0000 7f5f25165640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f000664a0 0x7f5f00068960 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.615+0000 7f5f25165640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f20153bf0 0x7f5f20144250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.615+0000 7f5f25165640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f200c79e0 0x7f5f20149b40 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.615+0000 7f5f25165640 1 --2- 172.21.15.46:0/190745422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f200c6fe0 0x7f5f200c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:13.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.615+0000 7f5f25165640 1 -- 172.21.15.46:0/190745422 >> 172.21.15.46:0/190745422 conn(0x7f5f2006c6e0 msgr2=0x7f5f20158ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:13.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.615+0000 7f5f25165640 1 -- 172.21.15.46:0/190745422 shutdown_connections 2023-10-30T13:47:13.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:13.615+0000 7f5f25165640 1 -- 172.21.15.46:0/190745422 wait complete. 2023-10-30T13:47:13.637 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:47:15.638 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:15.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf3e9a1640 1 Processor -- start 2023-10-30T13:47:15.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf3e9a1640 1 -- start start 2023-10-30T13:47:15.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf3e9a1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf38151800 0x7fdf3806e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:15.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf3e9a1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdf380c1ee0 0x7fdf380c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:15.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf3e9a1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3806e090 con 0x7fdf380c9230 2023-10-30T13:47:15.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf3e9a1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3806dc70 con 0x7fdf38151800 2023-10-30T13:47:15.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf3e9a1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf380c8a50 con 0x7fdf380c1ee0 2023-10-30T13:47:15.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf37fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf38151800 0x7fdf3806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:15.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf37fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf38151800 0x7fdf3806e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40216/0 (socket says 172.21.15.46:40216) 2023-10-30T13:47:15.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf37fff640 1 -- 172.21.15.46:0/801886398 learned_addr learned my addr 172.21.15.46:0/801886398 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:15.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf377fe640 1 --2- 172.21.15.46:0/801886398 >> v2:172.21.15.154:3300/0 conn(0x7fdf380c1ee0 0x7fdf380c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf37fff640 1 -- 172.21.15.46:0/801886398 >> v2:172.21.15.154:3300/0 conn(0x7fdf380c1ee0 msgr2=0x7fdf380c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf37fff640 1 --2- 172.21.15.46:0/801886398 >> v2:172.21.15.154:3300/0 conn(0x7fdf380c1ee0 0x7fdf380c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf37fff640 1 -- 172.21.15.46:0/801886398 >> v1:172.21.15.154:6789/0 conn(0x7fdf380c9230 legacy=0x7fdf380cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.783+0000 7fdf37fff640 1 -- 172.21.15.46:0/801886398 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf380c3b10 con 0x7fdf38151800 2023-10-30T13:47:15.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.787+0000 7fdf37fff640 1 --2- 172.21.15.46:0/801886398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf38151800 0x7fdf3806e470 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fdf3806e9b0 tx=0x7fdf2801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa0a38d86226505 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.787+0000 7fdf36ffd640 1 -- 172.21.15.46:0/801886398 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf280aa430 con 0x7fdf38151800 2023-10-30T13:47:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.787+0000 7fdf36ffd640 1 -- 172.21.15.46:0/801886398 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf280aa590 con 0x7fdf38151800 2023-10-30T13:47:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.787+0000 7fdf36ffd640 1 -- 172.21.15.46:0/801886398 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf280aa920 con 0x7fdf38151800 2023-10-30T13:47:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.787+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/801886398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf38151800 msgr2=0x7fdf3806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.787+0000 7fdf3e9a1640 1 --2- 172.21.15.46:0/801886398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf38151800 0x7fdf3806e470 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fdf3806e9b0 tx=0x7fdf2801c4d0 comp rx=0 tx=0).stop 2023-10-30T13:47:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.787+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/801886398 shutdown_connections 2023-10-30T13:47:15.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.787+0000 7fdf3e9a1640 1 --2- 172.21.15.46:0/801886398 >> v2:172.21.15.154:3300/0 conn(0x7fdf380c1ee0 0x7fdf380c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:15.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.787+0000 7fdf3e9a1640 1 --2- 172.21.15.46:0/801886398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf38151800 0x7fdf3806e470 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:15.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.787+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/801886398 >> 172.21.15.46:0/801886398 conn(0x7fdf38071ea0 msgr2=0x7fdf38065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:15.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.787+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/801886398 shutdown_connections 2023-10-30T13:47:15.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.787+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/801886398 wait complete. 2023-10-30T13:47:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3e9a1640 1 Processor -- start 2023-10-30T13:47:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3e9a1640 1 -- start start 2023-10-30T13:47:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3e9a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf380c1ee0 0x7fdf380b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3e9a1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf380c9230 0x7fdf38156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3e9a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf38151800 0x7fdf3806b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:15.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3e9a1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf381523a0 con 0x7fdf38151800 2023-10-30T13:47:15.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3e9a1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380b2210 con 0x7fdf380c9230 2023-10-30T13:47:15.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3e9a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf381f7560 con 0x7fdf380c1ee0 2023-10-30T13:47:15.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3cf3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf38151800 0x7fdf3806b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:15.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3cf3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf38151800 0x7fdf3806b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40226/0 (socket says 172.21.15.46:40226) 2023-10-30T13:47:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3cf3e640 1 -- 172.21.15.46:0/3845122775 learned_addr learned my addr 172.21.15.46:0/3845122775 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf37fff640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf380c1ee0 0x7fdf380b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf377fe640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf380c9230 0x7fdf38156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3cf3e640 1 -- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf380c1ee0 msgr2=0x7fdf380b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3cf3e640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf380c1ee0 0x7fdf380b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3cf3e640 1 -- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf380c9230 msgr2=0x7fdf38156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3cf3e640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf380c9230 0x7fdf38156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3cf3e640 1 -- 172.21.15.46:0/3845122775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf280094d0 con 0x7fdf38151800 2023-10-30T13:47:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3cf3e640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf38151800 0x7fdf3806b030 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000c760 tx=0x7fdf3000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf34ff9640 1 -- 172.21.15.46:0/3845122775 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf30003ba0 con 0x7fdf38151800 2023-10-30T13:47:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf34ff9640 1 -- 172.21.15.46:0/3845122775 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf30003d00 con 0x7fdf38151800 2023-10-30T13:47:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/3845122775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf380b2430 con 0x7fdf38151800 2023-10-30T13:47:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf34ff9640 1 -- 172.21.15.46:0/3845122775 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3000aa90 con 0x7fdf38151800 2023-10-30T13:47:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.791+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/3845122775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf381f6c30 con 0x7fdf38151800 2023-10-30T13:47:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.795+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/3845122775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3804e170 con 0x7fdf38151800 2023-10-30T13:47:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.795+0000 7fdf34ff9640 1 -- 172.21.15.46:0/3845122775 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdf3000abf0 con 0x7fdf38151800 2023-10-30T13:47:15.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.795+0000 7fdf34ff9640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf240664a0 0x7fdf24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:15.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.795+0000 7fdf34ff9640 1 -- 172.21.15.46:0/3845122775 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fdf30084120 con 0x7fdf38151800 2023-10-30T13:47:15.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.795+0000 7fdf37fff640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf240664a0 0x7fdf24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:15.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.795+0000 7fdf37fff640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf240664a0 0x7fdf24068960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdf280ae000 tx=0x7fdf280a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:15.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.799+0000 7fdf34ff9640 1 -- 172.21.15.46:0/3845122775 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdf300510e0 con 0x7fdf38151800 2023-10-30T13:47:15.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/3845122775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf381f76a0 con 0x7fdf38151800 2023-10-30T13:47:15.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf34ff9640 1 -- 172.21.15.46:0/3845122775 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6522 (secure 0 0 0) 0x7fdf30054700 con 0x7fdf38151800 2023-10-30T13:47:15.993 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:15.994 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6702,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:45:44.251897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:15.994 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 110 2023-10-30T13:47:15.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf240664a0 msgr2=0x7fdf24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:15.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf240664a0 0x7fdf24068960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdf280ae000 tx=0x7fdf280a7000 comp rx=0 tx=0).stop 2023-10-30T13:47:15.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/3845122775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf38151800 msgr2=0x7fdf3806b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:15.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf38151800 0x7fdf3806b030 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000c760 tx=0x7fdf3000cc20 comp rx=0 tx=0).stop 2023-10-30T13:47:15.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/3845122775 shutdown_connections 2023-10-30T13:47:15.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf240664a0 0x7fdf24068960 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:15.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf38151800 0x7fdf3806b030 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf380c9230 0x7fdf38156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 --2- 172.21.15.46:0/3845122775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf380c1ee0 0x7fdf380b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/3845122775 >> 172.21.15.46:0/3845122775 conn(0x7fdf38071ea0 msgr2=0x7fdf3806f920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/3845122775 shutdown_connections 2023-10-30T13:47:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:15.991+0000 7fdf3e9a1640 1 -- 172.21.15.46:0/3845122775 wait complete. 2023-10-30T13:47:16.014 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:47:18.016 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:18.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.166+0000 7f7e8fd0f640 1 Processor -- start 2023-10-30T13:47:18.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.166+0000 7f7e8fd0f640 1 -- start start 2023-10-30T13:47:18.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8fd0f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e8814fd50 0x7f7e880c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8fd0f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7e880cb550 0x7f7e880cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8fd0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e88071250 0x7f7e880716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8fd0f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e88071be0 con 0x7f7e8814fd50 2023-10-30T13:47:18.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8fd0f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e881560a0 con 0x7f7e880cb550 2023-10-30T13:47:18.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8fd0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e88156380 con 0x7f7e88071250 2023-10-30T13:47:18.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8daab640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e8814fd50 0x7f7e880c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8daab640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e8814fd50 0x7f7e880c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49422/0 (socket says 172.21.15.46:49422) 2023-10-30T13:47:18.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8daab640 1 -- 172.21.15.46:0/3288568906 learned_addr learned my addr 172.21.15.46:0/3288568906 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:18.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8d2aa640 1 --2- 172.21.15.46:0/3288568906 >> v2:172.21.15.154:3300/0 conn(0x7f7e880cb550 0x7f7e880cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8e2ac640 1 --2- 172.21.15.46:0/3288568906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e88071250 0x7f7e880716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8daab640 1 -- 172.21.15.46:0/3288568906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e88071250 msgr2=0x7f7e880716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:18.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8daab640 1 --2- 172.21.15.46:0/3288568906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e88071250 0x7f7e880716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8daab640 1 -- 172.21.15.46:0/3288568906 >> v2:172.21.15.154:3300/0 conn(0x7f7e880cb550 msgr2=0x7f7e880cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:18.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8daab640 1 --2- 172.21.15.46:0/3288568906 >> v2:172.21.15.154:3300/0 conn(0x7f7e880cb550 0x7f7e880cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8daab640 1 -- 172.21.15.46:0/3288568906 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e88156660 con 0x7f7e8814fd50 2023-10-30T13:47:18.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8e2ac640 1 --2- 172.21.15.46:0/3288568906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e88071250 0x7f7e880716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:47:18.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.170+0000 7f7e8daab640 1 --2- 172.21.15.46:0/3288568906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8814fd50 0x7f7e880c65d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f7e8806ac30 tx=0x7f7e8401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9b83510d2928b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8caa9640 1 -- 172.21.15.46:0/3288568906 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e840ab610 con 0x7f7e8814fd50 2023-10-30T13:47:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8caa9640 1 -- 172.21.15.46:0/3288568906 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e840ab770 con 0x7f7e8814fd50 2023-10-30T13:47:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8caa9640 1 -- 172.21.15.46:0/3288568906 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e840abb00 con 0x7f7e8814fd50 2023-10-30T13:47:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/3288568906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8814fd50 msgr2=0x7f7e880c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 --2- 172.21.15.46:0/3288568906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8814fd50 0x7f7e880c65d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f7e8806ac30 tx=0x7f7e8401c4d0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/3288568906 shutdown_connections 2023-10-30T13:47:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 --2- 172.21.15.46:0/3288568906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e88071250 0x7f7e880716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 --2- 172.21.15.46:0/3288568906 >> v2:172.21.15.154:3300/0 conn(0x7f7e880cb550 0x7f7e880cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 --2- 172.21.15.46:0/3288568906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e8814fd50 0x7f7e880c65d0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/3288568906 >> 172.21.15.46:0/3288568906 conn(0x7f7e8806c6e0 msgr2=0x7f7e88069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:18.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/3288568906 shutdown_connections 2023-10-30T13:47:18.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/3288568906 wait complete. 2023-10-30T13:47:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 Processor -- start 2023-10-30T13:47:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 -- start start 2023-10-30T13:47:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e88071250 0x7f7e881f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e880cb550 0x7f7e8815fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e8814fd50 0x7f7e881f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e881452f0 con 0x7f7e88071250 2023-10-30T13:47:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e881f1a40 con 0x7f7e880cb550 2023-10-30T13:47:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8fd0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e88160820 con 0x7f7e8814fd50 2023-10-30T13:47:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.174+0000 7f7e8daab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e88071250 0x7f7e881f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8daab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e88071250 0x7f7e881f2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49424/0 (socket says 172.21.15.46:49424) 2023-10-30T13:47:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8daab640 1 -- 172.21.15.46:0/1479643691 learned_addr learned my addr 172.21.15.46:0/1479643691 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8d2aa640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e880cb550 0x7f7e8815fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8e2ac640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e8814fd50 0x7f7e881f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8daab640 1 -- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e8814fd50 msgr2=0x7f7e881f4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8daab640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e8814fd50 0x7f7e881f4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8daab640 1 -- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e880cb550 msgr2=0x7f7e8815fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8daab640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e880cb550 0x7f7e8815fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8daab640 1 -- 172.21.15.46:0/1479643691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e840094d0 con 0x7f7e88071250 2023-10-30T13:47:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8d2aa640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e880cb550 0x7f7e8815fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:47:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8daab640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e88071250 0x7f7e881f2200 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f7e8401ab80 tx=0x7f7e8401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e7e7fc640 1 -- 172.21.15.46:0/1479643691 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e840b4800 con 0x7f7e88071250 2023-10-30T13:47:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e7e7fc640 1 -- 172.21.15.46:0/1479643691 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e840b4980 con 0x7f7e88071250 2023-10-30T13:47:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/1479643691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e88145540 con 0x7f7e88071250 2023-10-30T13:47:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e7e7fc640 1 -- 172.21.15.46:0/1479643691 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e840b4d30 con 0x7f7e88071250 2023-10-30T13:47:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/1479643691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e88145820 con 0x7f7e88071250 2023-10-30T13:47:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/1479643691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8804e170 con 0x7f7e88071250 2023-10-30T13:47:18.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e7e7fc640 1 -- 172.21.15.46:0/1479643691 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7e840c7390 con 0x7f7e88071250 2023-10-30T13:47:18.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e7e7fc640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7e6c0664d0 0x7f7e6c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:18.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.178+0000 7f7e7e7fc640 1 -- 172.21.15.46:0/1479643691 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f7e8411ece0 con 0x7f7e88071250 2023-10-30T13:47:18.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.182+0000 7f7e8d2aa640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7e6c0664d0 0x7f7e6c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:18.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.182+0000 7f7e8d2aa640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7e6c0664d0 0x7f7e6c068990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7e8806a270 tx=0x7f7e78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:18.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.182+0000 7f7e7e7fc640 1 -- 172.21.15.46:0/1479643691 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7e840ebca0 con 0x7f7e88071250 2023-10-30T13:47:18.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/1479643691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e880cbb70 con 0x7f7e88071250 2023-10-30T13:47:18.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e7e7fc640 1 -- 172.21.15.46:0/1479643691 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f7e8411e020 con 0x7f7e88071250 2023-10-30T13:47:18.378 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:18.378 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6702,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:45:44.251897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:18.379 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 110 2023-10-30T13:47:18.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7e6c0664d0 msgr2=0x7f7e6c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:18.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7e6c0664d0 0x7f7e6c068990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7e8806a270 tx=0x7f7e78008040 comp rx=0 tx=0).stop 2023-10-30T13:47:18.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/1479643691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e88071250 msgr2=0x7f7e881f2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:18.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e88071250 0x7f7e881f2200 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f7e8401ab80 tx=0x7f7e8401ad10 comp rx=0 tx=0).stop 2023-10-30T13:47:18.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/1479643691 shutdown_connections 2023-10-30T13:47:18.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7e6c0664d0 0x7f7e6c068990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e8814fd50 0x7f7e881f4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e880cb550 0x7f7e8815fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 --2- 172.21.15.46:0/1479643691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e88071250 0x7f7e881f2200 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:18.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/1479643691 >> 172.21.15.46:0/1479643691 conn(0x7f7e8806c6e0 msgr2=0x7f7e880ca0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:18.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/1479643691 shutdown_connections 2023-10-30T13:47:18.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:18.374+0000 7f7e8fd0f640 1 -- 172.21.15.46:0/1479643691 wait complete. 2023-10-30T13:47:18.399 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:47:19.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:47:19.774 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:47:20.400 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:20.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.550+0000 7f5642068640 1 Processor -- start 2023-10-30T13:47:20.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.550+0000 7f5642068640 1 -- start start 2023-10-30T13:47:20.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.550+0000 7f5642068640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f563c0c6b80 0x7f563c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:20.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.550+0000 7f5642068640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563c0c0a50 con 0x7f563c0c6f60 2023-10-30T13:47:20.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.550+0000 7f5642068640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563c06f6e0 con 0x7f563c13f4a0 2023-10-30T13:47:20.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.550+0000 7f5642068640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563c06f9c0 con 0x7f563c0c6b80 2023-10-30T13:47:20.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.554+0000 7f563b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f563c0c6b80 0x7f563c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:20.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.554+0000 7f563b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f563c0c6b80 0x7f563c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49430/0 (socket says 172.21.15.46:49430) 2023-10-30T13:47:20.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.554+0000 7f563b7fe640 1 -- 172.21.15.46:0/558247518 learned_addr learned my addr 172.21.15.46:0/558247518 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:20.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.554+0000 7f563b7fe640 1 -- 172.21.15.46:0/558247518 >> v1:172.21.15.154:6789/0 conn(0x7f563c13f4a0 legacy=0x7f563c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:20.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.554+0000 7f563b7fe640 1 -- 172.21.15.46:0/558247518 >> v1:172.21.15.46:6789/0 conn(0x7f563c0c6f60 legacy=0x7f563c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:20.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.554+0000 7f563b7fe640 1 -- 172.21.15.46:0/558247518 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c06fca0 con 0x7f563c0c6b80 2023-10-30T13:47:20.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.554+0000 7f563a7fc640 1 -- 172.21.15.46:0/558247518 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923668283 0 0) 0x7f563c0c0a50 con 0x7f563c0c6f60 2023-10-30T13:47:20.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.554+0000 7f563b7fe640 1 --2- 172.21.15.46:0/558247518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f563c0c6b80 0x7f563c0bfec0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f562c009870 tx=0x7f562c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b483a6318c28b5f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:20.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.554+0000 7f563a7fc640 1 -- 172.21.15.46:0/558247518 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c0ab650 con 0x7f563c0c6b80 2023-10-30T13:47:20.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.554+0000 7f563a7fc640 1 -- 172.21.15.46:0/558247518 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562c0ab7b0 con 0x7f563c0c6b80 2023-10-30T13:47:20.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.554+0000 7f563a7fc640 1 -- 172.21.15.46:0/558247518 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c0abb40 con 0x7f563c0c6b80 2023-10-30T13:47:20.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 -- 172.21.15.46:0/558247518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f563c0c6b80 msgr2=0x7f563c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:20.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 --2- 172.21.15.46:0/558247518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f563c0c6b80 0x7f563c0bfec0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f562c009870 tx=0x7f562c01c540 comp rx=0 tx=0).stop 2023-10-30T13:47:20.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 -- 172.21.15.46:0/558247518 shutdown_connections 2023-10-30T13:47:20.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 --2- 172.21.15.46:0/558247518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f563c0c6b80 0x7f563c0bfec0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:20.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 -- 172.21.15.46:0/558247518 >> 172.21.15.46:0/558247518 conn(0x7f563c0c7910 msgr2=0x7f563c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:20.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 -- 172.21.15.46:0/558247518 shutdown_connections 2023-10-30T13:47:20.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 -- 172.21.15.46:0/558247518 wait complete. 2023-10-30T13:47:20.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 Processor -- start 2023-10-30T13:47:20.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 -- start start 2023-10-30T13:47:20.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f563c0c6b80 0x7f563c0c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:20.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f563c0c6f60 0x7f563c1552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:20.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f563c13f4a0 0x7f563c1557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:20.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c1f32c0 con 0x7f563c0c6b80 2023-10-30T13:47:20.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c139270 con 0x7f563c0c6f60 2023-10-30T13:47:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f5642068640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c154e20 con 0x7f563c13f4a0 2023-10-30T13:47:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f563b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f563c0c6b80 0x7f563c0c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f563b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f563c0c6b80 0x7f563c0c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49432/0 (socket says 172.21.15.46:49432) 2023-10-30T13:47:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f563b7fe640 1 -- 172.21.15.46:0/4133779331 learned_addr learned my addr 172.21.15.46:0/4133779331 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f563affd640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f563c0c6f60 0x7f563c1552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.558+0000 7f563bfff640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f563c13f4a0 0x7f563c1557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:20.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f563b7fe640 1 -- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f563c13f4a0 msgr2=0x7f563c1557f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:20.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f563b7fe640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f563c13f4a0 0x7f563c1557f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:20.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f563b7fe640 1 -- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f563c0c6f60 msgr2=0x7f563c1552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:20.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f563b7fe640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f563c0c6f60 0x7f563c1552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:20.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f563b7fe640 1 -- 172.21.15.46:0/4133779331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c0094d0 con 0x7f563c0c6b80 2023-10-30T13:47:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f563b7fe640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f563c0c6b80 0x7f563c0c4fa0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f562c019150 tx=0x7f562c01ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:20.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f5623fff640 1 -- 172.21.15.46:0/4133779331 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c0b48c0 con 0x7f563c0c6b80 2023-10-30T13:47:20.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f5623fff640 1 -- 172.21.15.46:0/4133779331 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562c0b4a40 con 0x7f563c0c6b80 2023-10-30T13:47:20.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f5623fff640 1 -- 172.21.15.46:0/4133779331 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c0b4df0 con 0x7f563c0c6b80 2023-10-30T13:47:20.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f5642068640 1 -- 172.21.15.46:0/4133779331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563c0c54e0 con 0x7f563c0c6b80 2023-10-30T13:47:20.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f5642068640 1 -- 172.21.15.46:0/4133779331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563c1f2fe0 con 0x7f563c0c6b80 2023-10-30T13:47:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f5642068640 1 -- 172.21.15.46:0/4133779331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563c04e170 con 0x7f563c0c6b80 2023-10-30T13:47:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f5623fff640 1 -- 172.21.15.46:0/4133779331 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f562c0b3380 con 0x7f563c0c6b80 2023-10-30T13:47:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f5623fff640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56280664d0 0x7f5628068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.562+0000 7f5623fff640 1 -- 172.21.15.46:0/4133779331 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f562c11ec40 con 0x7f563c0c6b80 2023-10-30T13:47:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.566+0000 7f563affd640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56280664d0 0x7f5628068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.566+0000 7f563affd640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56280664d0 0x7f5628068990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f563c157970 tx=0x7f5630009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:20.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.570+0000 7f5623fff640 1 -- 172.21.15.46:0/4133779331 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f562c0ebdc0 con 0x7f563c0c6b80 2023-10-30T13:47:20.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.758+0000 7f5642068640 1 -- 172.21.15.46:0/4133779331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f563c13a210 con 0x7f563c0c6b80 2023-10-30T13:47:20.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.758+0000 7f5623fff640 1 -- 172.21.15.46:0/4133779331 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f562c0c4090 con 0x7f563c0c6b80 2023-10-30T13:47:20.763 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:20.764 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6702,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:45:44.251897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:20.764 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 110 2023-10-30T13:47:20.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.762+0000 7f5642068640 1 -- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56280664d0 msgr2=0x7f5628068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:20.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.762+0000 7f5642068640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56280664d0 0x7f5628068990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f563c157970 tx=0x7f5630009040 comp rx=0 tx=0).stop 2023-10-30T13:47:20.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.762+0000 7f5642068640 1 -- 172.21.15.46:0/4133779331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f563c0c6b80 msgr2=0x7f563c0c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:20.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.762+0000 7f5642068640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f563c0c6b80 0x7f563c0c4fa0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f562c019150 tx=0x7f562c01ad80 comp rx=0 tx=0).stop 2023-10-30T13:47:20.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.762+0000 7f5642068640 1 -- 172.21.15.46:0/4133779331 shutdown_connections 2023-10-30T13:47:20.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.762+0000 7f5642068640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56280664d0 0x7f5628068990 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:20.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.762+0000 7f5642068640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f563c13f4a0 0x7f563c1557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:20.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.762+0000 7f5642068640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f563c0c6f60 0x7f563c1552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:20.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.762+0000 7f5642068640 1 --2- 172.21.15.46:0/4133779331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f563c0c6b80 0x7f563c0c4fa0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:20.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.762+0000 7f5642068640 1 -- 172.21.15.46:0/4133779331 >> 172.21.15.46:0/4133779331 conn(0x7f563c0c7910 msgr2=0x7f563c0c5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:20.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.762+0000 7f5642068640 1 -- 172.21.15.46:0/4133779331 shutdown_connections 2023-10-30T13:47:20.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:20.762+0000 7f5642068640 1 -- 172.21.15.46:0/4133779331 wait complete. 2023-10-30T13:47:20.784 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:47:22.785 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce4673e640 1 Processor -- start 2023-10-30T13:47:22.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce4673e640 1 -- start start 2023-10-30T13:47:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce4673e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fce400cf160 0x7fce400cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce4673e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce400cb800 con 0x7fce401538a0 2023-10-30T13:47:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce4673e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce400cb330 con 0x7fce4015d6f0 2023-10-30T13:47:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce4673e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce400cad70 con 0x7fce400cf160 2023-10-30T13:47:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce3ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fce401538a0 0x7fce400ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46384/0 (socket says 172.21.15.46:46384) 2023-10-30T13:47:22.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce44cdb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fce400cf160 0x7fce400cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:22.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce3ffff640 1 -- 172.21.15.46:0/2390792708 learned_addr learned my addr 172.21.15.46:0/2390792708 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:47:22.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce44cdb640 1 -- 172.21.15.46:0/2390792708 >> v1:172.21.15.154:6789/0 conn(0x7fce4015d6f0 legacy=0x7fce4015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:22.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce44cdb640 1 -- 172.21.15.46:0/2390792708 >> v1:172.21.15.46:6789/0 conn(0x7fce401538a0 legacy=0x7fce400ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:22.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce3effd640 1 -- 172.21.15.46:0/2390792708 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466948154 0 0) 0x7fce400cb800 con 0x7fce401538a0 2023-10-30T13:47:22.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.938+0000 7fce44cdb640 1 -- 172.21.15.46:0/2390792708 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce400cb050 con 0x7fce400cf160 2023-10-30T13:47:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.942+0000 7fce44cdb640 1 --2- 172.21.15.46:0/2390792708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce400cf160 0x7fce400cf590 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fce38009d90 tx=0x7fce3801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0f9c8b99a256b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.942+0000 7fce3effd640 1 -- 172.21.15.46:0/2390792708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce380af600 con 0x7fce400cf160 2023-10-30T13:47:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.942+0000 7fce3effd640 1 -- 172.21.15.46:0/2390792708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce380ab050 con 0x7fce400cf160 2023-10-30T13:47:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.942+0000 7fce3effd640 1 -- 172.21.15.46:0/2390792708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce380af980 con 0x7fce400cf160 2023-10-30T13:47:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.942+0000 7fce4673e640 1 -- 172.21.15.46:0/2390792708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce400cf160 msgr2=0x7fce400cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.942+0000 7fce4673e640 1 --2- 172.21.15.46:0/2390792708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce400cf160 0x7fce400cf590 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fce38009d90 tx=0x7fce3801f130 comp rx=0 tx=0).stop 2023-10-30T13:47:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.942+0000 7fce4673e640 1 -- 172.21.15.46:0/2390792708 shutdown_connections 2023-10-30T13:47:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.942+0000 7fce4673e640 1 --2- 172.21.15.46:0/2390792708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce400cf160 0x7fce400cf590 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.942+0000 7fce4673e640 1 -- 172.21.15.46:0/2390792708 >> 172.21.15.46:0/2390792708 conn(0x7fce4006bfe0 msgr2=0x7fce4006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.942+0000 7fce4673e640 1 -- 172.21.15.46:0/2390792708 shutdown_connections 2023-10-30T13:47:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.942+0000 7fce4673e640 1 -- 172.21.15.46:0/2390792708 wait complete. 2023-10-30T13:47:22.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce4673e640 1 Processor -- start 2023-10-30T13:47:22.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce4673e640 1 -- start start 2023-10-30T13:47:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce4673e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce400cf160 0x7fce4015bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce4673e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce401538a0 0x7fce40140cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce4673e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce4015d6f0 0x7fce40141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce4673e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce400ceb40 con 0x7fce4015d6f0 2023-10-30T13:47:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce4673e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce40159210 con 0x7fce401538a0 2023-10-30T13:47:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce4673e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce401594f0 con 0x7fce400cf160 2023-10-30T13:47:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce44cdb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce4015d6f0 0x7fce40141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce44cdb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce4015d6f0 0x7fce40141210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49458/0 (socket says 172.21.15.46:49458) 2023-10-30T13:47:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce44cdb640 1 -- 172.21.15.46:0/871619283 learned_addr learned my addr 172.21.15.46:0/871619283 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:22.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce3ffff640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce400cf160 0x7fce4015bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:22.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce3f7fe640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce401538a0 0x7fce40140cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:22.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce44cdb640 1 -- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce400cf160 msgr2=0x7fce4015bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:22.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce44cdb640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce400cf160 0x7fce4015bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:22.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce44cdb640 1 -- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce401538a0 msgr2=0x7fce40140cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:22.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce44cdb640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce401538a0 0x7fce40140cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:22.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce44cdb640 1 -- 172.21.15.46:0/871619283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce38009a70 con 0x7fce4015d6f0 2023-10-30T13:47:22.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce44cdb640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce4015d6f0 0x7fce40141210 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fce380b2040 tx=0x7fce3801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:22.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce3cff9640 1 -- 172.21.15.46:0/871619283 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce380b9410 con 0x7fce4015d6f0 2023-10-30T13:47:22.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce3cff9640 1 -- 172.21.15.46:0/871619283 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce380b9570 con 0x7fce4015d6f0 2023-10-30T13:47:22.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce3cff9640 1 -- 172.21.15.46:0/871619283 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce380b9920 con 0x7fce4015d6f0 2023-10-30T13:47:22.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce4673e640 1 -- 172.21.15.46:0/871619283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce400ced70 con 0x7fce4015d6f0 2023-10-30T13:47:22.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.946+0000 7fce4673e640 1 -- 172.21.15.46:0/871619283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce401464e0 con 0x7fce4015d6f0 2023-10-30T13:47:22.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.950+0000 7fce4673e640 1 -- 172.21.15.46:0/871619283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4004e170 con 0x7fce4015d6f0 2023-10-30T13:47:22.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.950+0000 7fce3cff9640 1 -- 172.21.15.46:0/871619283 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fce380b6050 con 0x7fce4015d6f0 2023-10-30T13:47:22.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.950+0000 7fce3cff9640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fce2c0664d0 0x7fce2c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:22.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.950+0000 7fce3cff9640 1 -- 172.21.15.46:0/871619283 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fce381228e0 con 0x7fce4015d6f0 2023-10-30T13:47:22.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.950+0000 7fce3ffff640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fce2c0664d0 0x7fce2c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:22.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.950+0000 7fce3ffff640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fce2c0664d0 0x7fce2c068990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fce30000f70 tx=0x7fce30007240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:22.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:22.954+0000 7fce3cff9640 1 -- 172.21.15.46:0/871619283 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fce380efaa0 con 0x7fce4015d6f0 2023-10-30T13:47:23.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.146+0000 7fce4673e640 1 -- 172.21.15.46:0/871619283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce400ca5d0 con 0x7fce4015d6f0 2023-10-30T13:47:23.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.146+0000 7fce3cff9640 1 -- 172.21.15.46:0/871619283 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6522 (secure 0 0 0) 0x7fce380f30c0 con 0x7fce4015d6f0 2023-10-30T13:47:23.150 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:23.150 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6702,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:45:44.251897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6201,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6201":{"gid":6201,"name":"e","rank":0,"incarnation":90,"state":"up:active","state_seq":35,"addr":"172.21.15.46:6837/2546436963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2546436963},{"type":"v1","addr":"172.21.15.46:6837","nonce":2546436963}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:23.151 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 110 2023-10-30T13:47:23.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.150+0000 7fce4673e640 1 -- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fce2c0664d0 msgr2=0x7fce2c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:23.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.150+0000 7fce4673e640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fce2c0664d0 0x7fce2c068990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fce30000f70 tx=0x7fce30007240 comp rx=0 tx=0).stop 2023-10-30T13:47:23.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.150+0000 7fce4673e640 1 -- 172.21.15.46:0/871619283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce4015d6f0 msgr2=0x7fce40141210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:23.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.150+0000 7fce4673e640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce4015d6f0 0x7fce40141210 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fce380b2040 tx=0x7fce3801bed0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.150+0000 7fce4673e640 1 -- 172.21.15.46:0/871619283 shutdown_connections 2023-10-30T13:47:23.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.150+0000 7fce4673e640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fce2c0664d0 0x7fce2c068990 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.150+0000 7fce4673e640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce4015d6f0 0x7fce40141210 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.150+0000 7fce4673e640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce401538a0 0x7fce40140cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.150+0000 7fce4673e640 1 --2- 172.21.15.46:0/871619283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce400cf160 0x7fce4015bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:23.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.150+0000 7fce4673e640 1 -- 172.21.15.46:0/871619283 >> 172.21.15.46:0/871619283 conn(0x7fce4006bfe0 msgr2=0x7fce40151b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:23.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.150+0000 7fce4673e640 1 -- 172.21.15.46:0/871619283 shutdown_connections 2023-10-30T13:47:23.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:23.150+0000 7fce4673e640 1 -- 172.21.15.46:0/871619283 wait complete. 2023-10-30T13:47:23.172 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:47:25.173 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:25.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.326+0000 7f604f617640 1 Processor -- start 2023-10-30T13:47:25.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604f617640 1 -- start start 2023-10-30T13:47:25.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604f617640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60480c6b80 0x7f60480bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604f617640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60480c6f60 0x7f60480c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604f617640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60480c0a50 con 0x7f60480c8430 2023-10-30T13:47:25.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604f617640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604806f6e0 con 0x7f60480c6f60 2023-10-30T13:47:25.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604f617640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604806f9c0 con 0x7f60480c6b80 2023-10-30T13:47:25.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604cbb2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60480c6f60 0x7f60480c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604dbb4640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f60480c8430 0x7f604813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46400/0 (socket says 172.21.15.46:46400) 2023-10-30T13:47:25.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604dbb4640 1 -- 172.21.15.46:0/2579400861 learned_addr learned my addr 172.21.15.46:0/2579400861 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:47:25.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604d3b3640 1 --2- 172.21.15.46:0/2579400861 >> v2:172.21.15.154:3300/0 conn(0x7f60480c6b80 0x7f60480bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f603ffff640 1 -- 172.21.15.46:0/2579400861 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246501278 0 0) 0x7f60480c0a50 con 0x7f60480c8430 2023-10-30T13:47:25.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f603ffff640 1 -- 172.21.15.46:0/2579400861 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6034003530 con 0x7f60480c8430 2023-10-30T13:47:25.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604cbb2640 1 -- 172.21.15.46:0/2579400861 >> v2:172.21.15.154:3300/0 conn(0x7f60480c6b80 msgr2=0x7f60480bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:25.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604cbb2640 1 --2- 172.21.15.46:0/2579400861 >> v2:172.21.15.154:3300/0 conn(0x7f60480c6b80 0x7f60480bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604cbb2640 1 -- 172.21.15.46:0/2579400861 >> v1:172.21.15.46:6789/0 conn(0x7f60480c8430 legacy=0x7f604813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.330+0000 7f604cbb2640 1 -- 172.21.15.46:0/2579400861 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604806fca0 con 0x7f60480c6f60 2023-10-30T13:47:25.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f604cbb2640 1 --2- 172.21.15.46:0/2579400861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60480c6f60 0x7f60480c0400 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f604806f1b0 tx=0x7f603801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1429890e95246006 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:25.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f603ffff640 1 -- 172.21.15.46:0/2579400861 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60380ab600 con 0x7f60480c6f60 2023-10-30T13:47:25.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f603ffff640 1 -- 172.21.15.46:0/2579400861 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60380a7050 con 0x7f60480c6f60 2023-10-30T13:47:25.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f603ffff640 1 -- 172.21.15.46:0/2579400861 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60380ab980 con 0x7f60480c6f60 2023-10-30T13:47:25.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f604f617640 1 -- 172.21.15.46:0/2579400861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60480c6f60 msgr2=0x7f60480c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f604f617640 1 --2- 172.21.15.46:0/2579400861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60480c6f60 0x7f60480c0400 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f604806f1b0 tx=0x7f603801c460 comp rx=0 tx=0).stop 2023-10-30T13:47:25.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f604f617640 1 -- 172.21.15.46:0/2579400861 shutdown_connections 2023-10-30T13:47:25.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f604f617640 1 --2- 172.21.15.46:0/2579400861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60480c6f60 0x7f60480c0400 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f604f617640 1 --2- 172.21.15.46:0/2579400861 >> v2:172.21.15.154:3300/0 conn(0x7f60480c6b80 0x7f60480bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f604f617640 1 -- 172.21.15.46:0/2579400861 >> 172.21.15.46:0/2579400861 conn(0x7f60480c7910 msgr2=0x7f60480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:25.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f604f617640 1 -- 172.21.15.46:0/2579400861 shutdown_connections 2023-10-30T13:47:25.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f604f617640 1 -- 172.21.15.46:0/2579400861 wait complete. 2023-10-30T13:47:25.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f604f617640 1 Processor -- start 2023-10-30T13:47:25.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.334+0000 7f604f617640 1 -- start start 2023-10-30T13:47:25.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604f617640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60480c6b80 0x7f60480c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604f617640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60480c6f60 0x7f60481f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604f617640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60480c8430 0x7f60480c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604f617640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60480c1230 con 0x7f60480c6b80 2023-10-30T13:47:25.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604f617640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604813e2f0 con 0x7f60480c6f60 2023-10-30T13:47:25.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604f617640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604813e030 con 0x7f60480c8430 2023-10-30T13:47:25.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604d3b3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60480c6b80 0x7f60480c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604d3b3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60480c6b80 0x7f60480c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49470/0 (socket says 172.21.15.46:49470) 2023-10-30T13:47:25.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604d3b3640 1 -- 172.21.15.46:0/289567713 learned_addr learned my addr 172.21.15.46:0/289567713 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:25.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604cbb2640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60480c6f60 0x7f60481f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604dbb4640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60480c8430 0x7f60480c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604d3b3640 1 -- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60480c8430 msgr2=0x7f60480c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604d3b3640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60480c8430 0x7f60480c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604d3b3640 1 -- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60480c6f60 msgr2=0x7f60481f0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604d3b3640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60480c6f60 0x7f60481f0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604d3b3640 1 -- 172.21.15.46:0/289567713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6038009420 con 0x7f60480c6b80 2023-10-30T13:47:25.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604d3b3640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60480c6b80 0x7f60480c8c10 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f604400d800 tx=0x7f604400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:25.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f603dffb640 1 -- 172.21.15.46:0/289567713 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604400b6c0 con 0x7f60480c6b80 2023-10-30T13:47:25.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f603dffb640 1 -- 172.21.15.46:0/289567713 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604400b820 con 0x7f60480c6b80 2023-10-30T13:47:25.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604f617640 1 -- 172.21.15.46:0/289567713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60480ca740 con 0x7f60480c6b80 2023-10-30T13:47:25.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f603dffb640 1 -- 172.21.15.46:0/289567713 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604400bb90 con 0x7f60480c6b80 2023-10-30T13:47:25.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604f617640 1 -- 172.21.15.46:0/289567713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60480caa00 con 0x7f60480c6b80 2023-10-30T13:47:25.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.338+0000 7f604f617640 1 -- 172.21.15.46:0/289567713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60481f0150 con 0x7f60480c6b80 2023-10-30T13:47:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.342+0000 7f603dffb640 1 -- 172.21.15.46:0/289567713 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f604400bcf0 con 0x7f60480c6b80 2023-10-30T13:47:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.342+0000 7f603dffb640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f602c0664d0 0x7f602c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.342+0000 7f603dffb640 1 -- 172.21.15.46:0/289567713 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f60440848e0 con 0x7f60480c6b80 2023-10-30T13:47:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.342+0000 7f604cbb2640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f602c0664d0 0x7f602c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.342+0000 7f604cbb2640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f602c0664d0 0x7f602c068990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f60380ad000 tx=0x7f6038008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:25.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.346+0000 7f603dffb640 1 -- 172.21.15.46:0/289567713 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6044051820 con 0x7f60480c6b80 2023-10-30T13:47:25.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 -- 172.21.15.46:0/289567713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f604804e170 con 0x7f60480c6b80 2023-10-30T13:47:25.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f603dffb640 1 -- 172.21.15.46:0/289567713 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f6044054e40 con 0x7f60480c6b80 2023-10-30T13:47:25.537 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:25.537 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:24.113529+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:25.538 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 111 2023-10-30T13:47:25.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 -- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f602c0664d0 msgr2=0x7f602c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f602c0664d0 0x7f602c068990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f60380ad000 tx=0x7f6038008040 comp rx=0 tx=0).stop 2023-10-30T13:47:25.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 -- 172.21.15.46:0/289567713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60480c6b80 msgr2=0x7f60480c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60480c6b80 0x7f60480c8c10 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f604400d800 tx=0x7f604400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 -- 172.21.15.46:0/289567713 shutdown_connections 2023-10-30T13:47:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f602c0664d0 0x7f602c068990 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60480c8430 0x7f60480c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60480c6f60 0x7f60481f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 --2- 172.21.15.46:0/289567713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60480c6b80 0x7f60480c8c10 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 -- 172.21.15.46:0/289567713 >> 172.21.15.46:0/289567713 conn(0x7f60480c7910 msgr2=0x7f60480c1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 -- 172.21.15.46:0/289567713 shutdown_connections 2023-10-30T13:47:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.534+0000 7f604f617640 1 -- 172.21.15.46:0/289567713 wait complete. 2023-10-30T13:47:25.558 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T13:47:25.559 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:47:25.559 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd56aa79640 1 Processor -- start 2023-10-30T13:47:25.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd56aa79640 1 -- start start 2023-10-30T13:47:25.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd56aa79640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd564156f80 0x7fd564159880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd56aa79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd564070650 0x7fd564071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd56aa79640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5640c2f20 con 0x7fd564070c70 2023-10-30T13:47:25.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd56aa79640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd564157360 con 0x7fd564156f80 2023-10-30T13:47:25.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd56aa79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5640c85c0 con 0x7fd564070650 2023-10-30T13:47:25.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd568815640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd564156f80 0x7fd564159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd568815640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd564156f80 0x7fd564159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49472/0 (socket says 172.21.15.46:49472) 2023-10-30T13:47:25.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd568815640 1 -- 172.21.15.46:0/2135071746 learned_addr learned my addr 172.21.15.46:0/2135071746 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:25.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd563fff640 1 --2- 172.21.15.46:0/2135071746 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd564070650 0x7fd564071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd568815640 1 -- 172.21.15.46:0/2135071746 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd564070650 msgr2=0x7fd564071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:25.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd568815640 1 --2- 172.21.15.46:0/2135071746 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd564070650 0x7fd564071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd568815640 1 -- 172.21.15.46:0/2135071746 >> v1:172.21.15.154:6789/0 conn(0x7fd564070c70 legacy=0x7fd56415cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd568815640 1 -- 172.21.15.46:0/2135071746 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd564071230 con 0x7fd564156f80 2023-10-30T13:47:25.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.698+0000 7fd563fff640 1 --2- 172.21.15.46:0/2135071746 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd564070650 0x7fd564071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:25.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.702+0000 7fd568815640 1 --2- 172.21.15.46:0/2135071746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564156f80 0x7fd564159880 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd558009870 tx=0x7fd55801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c6b84431bca844 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:25.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.702+0000 7fd5637fe640 1 -- 172.21.15.46:0/2135071746 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5580aa660 con 0x7fd564156f80 2023-10-30T13:47:25.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.702+0000 7fd5637fe640 1 -- 172.21.15.46:0/2135071746 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5580aa7c0 con 0x7fd564156f80 2023-10-30T13:47:25.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.702+0000 7fd5637fe640 1 -- 172.21.15.46:0/2135071746 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5580aab50 con 0x7fd564156f80 2023-10-30T13:47:25.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.702+0000 7fd56aa79640 1 -- 172.21.15.46:0/2135071746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564156f80 msgr2=0x7fd564159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.702+0000 7fd56aa79640 1 --2- 172.21.15.46:0/2135071746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564156f80 0x7fd564159880 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd558009870 tx=0x7fd55801c540 comp rx=0 tx=0).stop 2023-10-30T13:47:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.702+0000 7fd56aa79640 1 -- 172.21.15.46:0/2135071746 shutdown_connections 2023-10-30T13:47:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.702+0000 7fd56aa79640 1 --2- 172.21.15.46:0/2135071746 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd564070650 0x7fd564071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.702+0000 7fd56aa79640 1 --2- 172.21.15.46:0/2135071746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564156f80 0x7fd564159880 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.702+0000 7fd56aa79640 1 -- 172.21.15.46:0/2135071746 >> 172.21.15.46:0/2135071746 conn(0x7fd5640cbeb0 msgr2=0x7fd5640cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.702+0000 7fd56aa79640 1 -- 172.21.15.46:0/2135071746 shutdown_connections 2023-10-30T13:47:25.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.702+0000 7fd56aa79640 1 -- 172.21.15.46:0/2135071746 wait complete. 2023-10-30T13:47:25.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd56aa79640 1 Processor -- start 2023-10-30T13:47:25.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd56aa79640 1 -- start start 2023-10-30T13:47:25.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd56aa79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd564070650 0x7fd5640aec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd56aa79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd564070c70 0x7fd5640caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd56aa79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564156f80 0x7fd5641f5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd56aa79640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5640c4ab0 con 0x7fd564156f80 2023-10-30T13:47:25.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd56aa79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56415cea0 con 0x7fd564070650 2023-10-30T13:47:25.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd56aa79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5640c6600 con 0x7fd564070c70 2023-10-30T13:47:25.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd569016640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564156f80 0x7fd5641f5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd569016640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564156f80 0x7fd5641f5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49480/0 (socket says 172.21.15.46:49480) 2023-10-30T13:47:25.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd569016640 1 -- 172.21.15.46:0/2602239764 learned_addr learned my addr 172.21.15.46:0/2602239764 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:25.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd568815640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd564070650 0x7fd5640aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd563fff640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd564070c70 0x7fd5640caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd569016640 1 -- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd564070c70 msgr2=0x7fd5640caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd569016640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd564070c70 0x7fd5640caa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd569016640 1 -- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd564070650 msgr2=0x7fd5640aec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd569016640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd564070650 0x7fd5640aec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd569016640 1 -- 172.21.15.46:0/2602239764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5580094d0 con 0x7fd564156f80 2023-10-30T13:47:25.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd569016640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564156f80 0x7fd5641f5880 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00c770 tx=0x7fd55c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd5617fa640 1 -- 172.21.15.46:0/2602239764 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c003b80 con 0x7fd564156f80 2023-10-30T13:47:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd5617fa640 1 -- 172.21.15.46:0/2602239764 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55c003ce0 con 0x7fd564156f80 2023-10-30T13:47:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd5617fa640 1 -- 172.21.15.46:0/2602239764 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c00aa90 con 0x7fd564156f80 2023-10-30T13:47:25.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.706+0000 7fd56aa79640 1 -- 172.21.15.46:0/2602239764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56415d110 con 0x7fd564156f80 2023-10-30T13:47:25.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.710+0000 7fd56aa79640 1 -- 172.21.15.46:0/2602239764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5640c5e00 con 0x7fd564156f80 2023-10-30T13:47:25.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.710+0000 7fd56aa79640 1 -- 172.21.15.46:0/2602239764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5640c5980 con 0x7fd564156f80 2023-10-30T13:47:25.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.710+0000 7fd5617fa640 1 -- 172.21.15.46:0/2602239764 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd55c00abf0 con 0x7fd564156f80 2023-10-30T13:47:25.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.710+0000 7fd5617fa640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5480664d0 0x7fd548068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:25.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.710+0000 7fd5617fa640 1 -- 172.21.15.46:0/2602239764 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7fd55c084070 con 0x7fd564156f80 2023-10-30T13:47:25.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.710+0000 7fd568815640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5480664d0 0x7fd548068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:25.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.714+0000 7fd568815640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5480664d0 0x7fd548068990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd5580ae000 tx=0x7fd5580b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:25.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.714+0000 7fd5617fa640 1 -- 172.21.15.46:0/2602239764 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd55c0b5090 con 0x7fd564156f80 2023-10-30T13:47:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 -- 172.21.15.46:0/2602239764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd56404e170 con 0x7fd564156f80 2023-10-30T13:47:25.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd5617fa640 1 -- 172.21.15.46:0/2602239764 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fd5640c5980 con 0x7fd564156f80 2023-10-30T13:47:25.909 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:25.909 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:24.113529+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:25.909 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 111 2023-10-30T13:47:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 -- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5480664d0 msgr2=0x7fd548068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5480664d0 0x7fd548068990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd5580ae000 tx=0x7fd5580b5040 comp rx=0 tx=0).stop 2023-10-30T13:47:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 -- 172.21.15.46:0/2602239764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564156f80 msgr2=0x7fd5641f5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564156f80 0x7fd5641f5880 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00c770 tx=0x7fd55c00cc30 comp rx=0 tx=0).stop 2023-10-30T13:47:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 -- 172.21.15.46:0/2602239764 shutdown_connections 2023-10-30T13:47:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5480664d0 0x7fd548068990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564156f80 0x7fd5641f5880 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd564070c70 0x7fd5640caa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 --2- 172.21.15.46:0/2602239764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd564070650 0x7fd5640aec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 -- 172.21.15.46:0/2602239764 >> 172.21.15.46:0/2602239764 conn(0x7fd5640cbeb0 msgr2=0x7fd5640b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 -- 172.21.15.46:0/2602239764 shutdown_connections 2023-10-30T13:47:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:25.906+0000 7fd56aa79640 1 -- 172.21.15.46:0/2602239764 wait complete. 2023-10-30T13:47:25.931 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 111, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 111, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6366": { "addr": "172.21.15.46:6833/190134548", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 190134548, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 190134548, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6366, "incarnation": 104, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 51 }, "gid_6702": { "addr": "172.21.15.46:6836/2197729559", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 2197729559, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 2197729559, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6702, "incarnation": 111, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 69, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:47:24.113529+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6702, "mds_1": 6366 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:47:27.932 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.082+0000 7fb3371f8640 1 Processor -- start 2023-10-30T13:47:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.082+0000 7fb3371f8640 1 -- start start 2023-10-30T13:47:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb3371f8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb330069e90 0x7fb33015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb3371f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb33006a270 0x7fb3300c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb3371f8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb33006cb80 0x7fb33006d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb3371f8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3300c8570 con 0x7fb33006cb80 2023-10-30T13:47:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb3371f8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33006d550 con 0x7fb330069e90 2023-10-30T13:47:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb3371f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3300c6c10 con 0x7fb33006a270 2023-10-30T13:47:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb335795640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb33006cb80 0x7fb33006d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb335795640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb33006cb80 0x7fb33006d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51238/0 (socket says 172.21.15.46:51238) 2023-10-30T13:47:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb335795640 1 -- 172.21.15.46:0/3954593671 learned_addr learned my addr 172.21.15.46:0/3954593671 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb32ffff640 1 --2- 172.21.15.46:0/3954593671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb33006a270 0x7fb3300c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb334f94640 1 --2- 172.21.15.46:0/3954593671 >> v2:172.21.15.154:3300/0 conn(0x7fb330069e90 0x7fb33015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb335795640 1 -- 172.21.15.46:0/3954593671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb33006a270 msgr2=0x7fb3300c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb335795640 1 --2- 172.21.15.46:0/3954593671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb33006a270 0x7fb3300c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb335795640 1 -- 172.21.15.46:0/3954593671 >> v2:172.21.15.154:3300/0 conn(0x7fb330069e90 msgr2=0x7fb33015a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb335795640 1 --2- 172.21.15.46:0/3954593671 >> v2:172.21.15.154:3300/0 conn(0x7fb330069e90 0x7fb33015a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb335795640 1 -- 172.21.15.46:0/3954593671 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3300c6ef0 con 0x7fb33006cb80 2023-10-30T13:47:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.086+0000 7fb335795640 1 --2- 172.21.15.46:0/3954593671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33006cb80 0x7fb33006d010 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb33006ed20 tx=0x7fb32801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1f1e66778ef54e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb32f7fe640 1 -- 172.21.15.46:0/3954593671 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3280a5740 con 0x7fb33006cb80 2023-10-30T13:47:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb32f7fe640 1 -- 172.21.15.46:0/3954593671 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3280a58a0 con 0x7fb33006cb80 2023-10-30T13:47:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb32f7fe640 1 -- 172.21.15.46:0/3954593671 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3280a5c30 con 0x7fb33006cb80 2023-10-30T13:47:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb3371f8640 1 -- 172.21.15.46:0/3954593671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33006cb80 msgr2=0x7fb33006d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb3371f8640 1 --2- 172.21.15.46:0/3954593671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33006cb80 0x7fb33006d010 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb33006ed20 tx=0x7fb32801c9d0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb3371f8640 1 -- 172.21.15.46:0/3954593671 shutdown_connections 2023-10-30T13:47:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb3371f8640 1 --2- 172.21.15.46:0/3954593671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33006cb80 0x7fb33006d010 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb3371f8640 1 --2- 172.21.15.46:0/3954593671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb33006a270 0x7fb3300c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb3371f8640 1 --2- 172.21.15.46:0/3954593671 >> v2:172.21.15.154:3300/0 conn(0x7fb330069e90 0x7fb33015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb3371f8640 1 -- 172.21.15.46:0/3954593671 >> 172.21.15.46:0/3954593671 conn(0x7fb33006dfd0 msgr2=0x7fb33006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb3371f8640 1 -- 172.21.15.46:0/3954593671 shutdown_connections 2023-10-30T13:47:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb3371f8640 1 -- 172.21.15.46:0/3954593671 wait complete. 2023-10-30T13:47:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb3371f8640 1 Processor -- start 2023-10-30T13:47:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb3371f8640 1 -- start start 2023-10-30T13:47:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.090+0000 7fb3371f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb330069e90 0x7fb3301eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb3371f8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33006a270 0x7fb3301f2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb3371f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb33006cb80 0x7fb33015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb3371f8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb330156830 con 0x7fb33006a270 2023-10-30T13:47:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb3371f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb330157280 con 0x7fb33006cb80 2023-10-30T13:47:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb3371f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3301f2e80 con 0x7fb330069e90 2023-10-30T13:47:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33006a270 0x7fb3301f2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33006a270 0x7fb3301f2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51248/0 (socket says 172.21.15.46:51248) 2023-10-30T13:47:28.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32ffff640 1 -- 172.21.15.46:0/1368284189 learned_addr learned my addr 172.21.15.46:0/1368284189 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb334f94640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb330069e90 0x7fb3301eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb335795640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb33006cb80 0x7fb33015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32ffff640 1 -- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb330069e90 msgr2=0x7fb3301eb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32ffff640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb330069e90 0x7fb3301eb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32ffff640 1 -- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb33006cb80 msgr2=0x7fb33015ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:28.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32ffff640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb33006cb80 0x7fb33015ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32ffff640 1 -- 172.21.15.46:0/1368284189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb328009b30 con 0x7fb33006a270 2023-10-30T13:47:28.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32ffff640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33006a270 0x7fb3301f2940 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00d830 tx=0x7fb31c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:28.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32d7fa640 1 -- 172.21.15.46:0/1368284189 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c00b6c0 con 0x7fb33006a270 2023-10-30T13:47:28.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32d7fa640 1 -- 172.21.15.46:0/1368284189 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c00b820 con 0x7fb33006a270 2023-10-30T13:47:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32d7fa640 1 -- 172.21.15.46:0/1368284189 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c00bb90 con 0x7fb33006a270 2023-10-30T13:47:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb3371f8640 1 -- 172.21.15.46:0/1368284189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb330156d00 con 0x7fb33006a270 2023-10-30T13:47:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb3371f8640 1 -- 172.21.15.46:0/1368284189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb330156fa0 con 0x7fb33006a270 2023-10-30T13:47:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb3371f8640 1 -- 172.21.15.46:0/1368284189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3301f4a40 con 0x7fb33006a270 2023-10-30T13:47:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.094+0000 7fb32d7fa640 1 -- 172.21.15.46:0/1368284189 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb31c01e460 con 0x7fb33006a270 2023-10-30T13:47:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.098+0000 7fb32d7fa640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3140664d0 0x7fb314068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.098+0000 7fb32d7fa640 1 -- 172.21.15.46:0/1368284189 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7fb31c084ae0 con 0x7fb33006a270 2023-10-30T13:47:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.098+0000 7fb334f94640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3140664d0 0x7fb314068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.098+0000 7fb334f94640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3140664d0 0x7fb314068990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb3240045f0 tx=0x7fb324008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.098+0000 7fb32d7fa640 1 -- 172.21.15.46:0/1368284189 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb3301f4a40 con 0x7fb33006a270 2023-10-30T13:47:28.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 -- 172.21.15.46:0/1368284189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3301f4a40 con 0x7fb33006a270 2023-10-30T13:47:28.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb32d7fa640 1 -- 172.21.15.46:0/1368284189 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fb3301f4a40 con 0x7fb33006a270 2023-10-30T13:47:28.293 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:28.294 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:24.113529+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:28.294 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 111 2023-10-30T13:47:28.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 -- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3140664d0 msgr2=0x7fb314068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:28.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3140664d0 0x7fb314068990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb3240045f0 tx=0x7fb324008040 comp rx=0 tx=0).stop 2023-10-30T13:47:28.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 -- 172.21.15.46:0/1368284189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33006a270 msgr2=0x7fb3301f2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:28.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33006a270 0x7fb3301f2940 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00d830 tx=0x7fb31c00dcf0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 -- 172.21.15.46:0/1368284189 shutdown_connections 2023-10-30T13:47:28.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3140664d0 0x7fb314068990 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb33006cb80 0x7fb33015ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33006a270 0x7fb3301f2940 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 --2- 172.21.15.46:0/1368284189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb330069e90 0x7fb3301eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:28.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 -- 172.21.15.46:0/1368284189 >> 172.21.15.46:0/1368284189 conn(0x7fb33006dfd0 msgr2=0x7fb33006e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:28.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 -- 172.21.15.46:0/1368284189 shutdown_connections 2023-10-30T13:47:28.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:28.290+0000 7fb3371f8640 1 -- 172.21.15.46:0/1368284189 wait complete. 2023-10-30T13:47:28.315 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:30.315 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:30.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.458+0000 7ffa0adf8640 1 Processor -- start 2023-10-30T13:47:30.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.458+0000 7ffa0adf8640 1 -- start start 2023-10-30T13:47:30.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa0adf8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0406f7a0 0x7ffa0406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa0adf8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa04066cf0 con 0x7ffa040c6bd0 2023-10-30T13:47:30.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa0adf8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa040c70d0 con 0x7ffa04071110 2023-10-30T13:47:30.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa0adf8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa040700c0 con 0x7ffa0406f7a0 2023-10-30T13:47:30.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa03fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffa040c6bd0 0x7ffa04069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55298/0 (socket says 172.21.15.46:55298) 2023-10-30T13:47:30.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa03fff640 1 -- 172.21.15.46:0/1816338867 learned_addr learned my addr 172.21.15.46:0/1816338867 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:47:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa08b94640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0406f7a0 0x7ffa0406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304272013 0 0) 0x7ffa04066cf0 con 0x7ffa040c6bd0 2023-10-30T13:47:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f4003530 con 0x7ffa040c6bd0 2023-10-30T13:47:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2638163541 0 0) 0x7ff9f4003530 con 0x7ffa040c6bd0 2023-10-30T13:47:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0406f7a0 msgr2=0x7ffa0406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:30.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0406f7a0 0x7ffa0406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 >> v1:172.21.15.154:6789/0 conn(0x7ffa04071110 legacy=0x7ffa04158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa040c7390 con 0x7ffa040c6bd0 2023-10-30T13:47:30.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7ff9f0001680 con 0x7ffa040c6bd0 2023-10-30T13:47:30.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa08b94640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0406f7a0 0x7ffa0406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:47:30.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 >> v1:172.21.15.46:6789/0 conn(0x7ffa040c6bd0 legacy=0x7ffa04069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0406f7a0 0x7ff9f40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9f4003ef0 0x7ff9f40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4004d10 0x7ff9f4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa040700c0 con 0x7ff9f4004d10 2023-10-30T13:47:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f400b9f0 con 0x7ff9f4003ef0 2023-10-30T13:47:30.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f400bbb0 con 0x7ffa0406f7a0 2023-10-30T13:47:30.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa09395640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4004d10 0x7ff9f4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa03fff640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9f4003ef0 0x7ff9f40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa08b94640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0406f7a0 0x7ff9f40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa09395640 1 -- 172.21.15.46:0/1816338867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0406f7a0 msgr2=0x7ff9f40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:30.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa09395640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0406f7a0 0x7ff9f40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa09395640 1 -- 172.21.15.46:0/1816338867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9f4003ef0 msgr2=0x7ff9f40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:30.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa09395640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9f4003ef0 0x7ff9f40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.462+0000 7ffa09395640 1 -- 172.21.15.46:0/1816338867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f8009420 con 0x7ff9f4004d10 2023-10-30T13:47:30.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.466+0000 7ffa09395640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4004d10 0x7ff9f4005190 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00ed20 tx=0x7ff9fc020b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac09182062368ec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.466+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc00e2d0 con 0x7ff9f4004d10 2023-10-30T13:47:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.466+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9fc0a9980 con 0x7ff9f4004d10 2023-10-30T13:47:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.466+0000 7ffa037fe640 1 -- 172.21.15.46:0/1816338867 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc0a9d30 con 0x7ff9f4004d10 2023-10-30T13:47:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.466+0000 7ffa0adf8640 1 -- 172.21.15.46:0/1816338867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4004d10 msgr2=0x7ff9f4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.466+0000 7ffa0adf8640 1 --2- 172.21.15.46:0/1816338867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f4004d10 0x7ff9f4005190 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00ed20 tx=0x7ff9fc020b70 comp rx=0 tx=0).stop 2023-10-30T13:47:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.466+0000 7ffa09395640 1 -- 172.21.15.46:0/1816338867 reap_dead start 2023-10-30T13:47:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.466+0000 7ffa0adf8640 1 -- 172.21.15.46:0/1816338867 shutdown_connections 2023-10-30T13:47:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.466+0000 7ffa0adf8640 1 -- 172.21.15.46:0/1816338867 >> 172.21.15.46:0/1816338867 conn(0x7ffa040c7910 msgr2=0x7ffa040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.466+0000 7ffa0adf8640 1 -- 172.21.15.46:0/1816338867 shutdown_connections 2023-10-30T13:47:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.466+0000 7ffa0adf8640 1 -- 172.21.15.46:0/1816338867 wait complete. 2023-10-30T13:47:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.470+0000 7ffa0adf8640 1 Processor -- start 2023-10-30T13:47:30.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.470+0000 7ffa0adf8640 1 -- start start 2023-10-30T13:47:30.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.470+0000 7ffa0adf8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9f4004d10 0x7ffa04139340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.470+0000 7ffa0adf8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa040c8cf0 0x7ffa04150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.470+0000 7ffa0adf8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa041eac70 0x7ffa041eb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.470+0000 7ffa0adf8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0414ee70 con 0x7ffa040c8cf0 2023-10-30T13:47:30.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.470+0000 7ffa0adf8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa041ea560 con 0x7ff9f4004d10 2023-10-30T13:47:30.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.470+0000 7ffa0adf8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0414b560 con 0x7ffa041eac70 2023-10-30T13:47:30.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa03fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa040c8cf0 0x7ffa04150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa03fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa040c8cf0 0x7ffa04150760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51254/0 (socket says 172.21.15.46:51254) 2023-10-30T13:47:30.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa03fff640 1 -- 172.21.15.46:0/2655196243 learned_addr learned my addr 172.21.15.46:0/2655196243 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:30.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa08b94640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9f4004d10 0x7ffa04139340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa09395640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa041eac70 0x7ffa041eb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa03fff640 1 -- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa041eac70 msgr2=0x7ffa041eb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa03fff640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa041eac70 0x7ffa041eb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa03fff640 1 -- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9f4004d10 msgr2=0x7ffa04139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa03fff640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9f4004d10 0x7ffa04139340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa03fff640 1 -- 172.21.15.46:0/2655196243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa041ea7e0 con 0x7ffa040c8cf0 2023-10-30T13:47:30.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa03fff640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa040c8cf0 0x7ffa04150760 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7ff9f00072f0 tx=0x7ff9f00077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:30.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa017fa640 1 -- 172.21.15.46:0/2655196243 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00014f0 con 0x7ffa040c8cf0 2023-10-30T13:47:30.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa017fa640 1 -- 172.21.15.46:0/2655196243 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f000f9c0 con 0x7ffa040c8cf0 2023-10-30T13:47:30.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa017fa640 1 -- 172.21.15.46:0/2655196243 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f000fd50 con 0x7ffa040c8cf0 2023-10-30T13:47:30.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa0adf8640 1 -- 172.21.15.46:0/2655196243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa041edf10 con 0x7ffa040c8cf0 2023-10-30T13:47:30.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa0adf8640 1 -- 172.21.15.46:0/2655196243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa041ee4d0 con 0x7ffa040c8cf0 2023-10-30T13:47:30.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.474+0000 7ffa0adf8640 1 -- 172.21.15.46:0/2655196243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa041ea970 con 0x7ffa040c8cf0 2023-10-30T13:47:30.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.478+0000 7ffa017fa640 1 -- 172.21.15.46:0/2655196243 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff9f001d360 con 0x7ffa040c8cf0 2023-10-30T13:47:30.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.478+0000 7ffa017fa640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9e80664d0 0x7ff9e8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:30.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.478+0000 7ffa017fa640 1 -- 172.21.15.46:0/2655196243 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7ff9f0083c30 con 0x7ffa040c8cf0 2023-10-30T13:47:30.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.478+0000 7ffa08b94640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9e80664d0 0x7ff9e8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:30.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.478+0000 7ffa08b94640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9e80664d0 0x7ff9e8068990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009cb0 tx=0x7ff9f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:30.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.482+0000 7ffa017fa640 1 -- 172.21.15.46:0/2655196243 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9f0050c00 con 0x7ffa040c8cf0 2023-10-30T13:47:30.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.674+0000 7ffa0adf8640 1 -- 172.21.15.46:0/2655196243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa0404e170 con 0x7ffa040c8cf0 2023-10-30T13:47:30.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.674+0000 7ffa017fa640 1 -- 172.21.15.46:0/2655196243 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5729 (secure 0 0 0) 0x7ff9f0054220 con 0x7ffa040c8cf0 2023-10-30T13:47:30.679 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:30.679 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:24.113529+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:30.679 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 111 2023-10-30T13:47:30.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.678+0000 7ffa0adf8640 1 -- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9e80664d0 msgr2=0x7ff9e8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.678+0000 7ffa0adf8640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9e80664d0 0x7ff9e8068990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009cb0 tx=0x7ff9f8008040 comp rx=0 tx=0).stop 2023-10-30T13:47:30.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.678+0000 7ffa0adf8640 1 -- 172.21.15.46:0/2655196243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa040c8cf0 msgr2=0x7ffa04150760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:30.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.678+0000 7ffa0adf8640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa040c8cf0 0x7ffa04150760 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7ff9f00072f0 tx=0x7ff9f00077b0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.678+0000 7ffa0adf8640 1 -- 172.21.15.46:0/2655196243 shutdown_connections 2023-10-30T13:47:30.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.678+0000 7ffa0adf8640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9e80664d0 0x7ff9e8068990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.678+0000 7ffa0adf8640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa041eac70 0x7ffa041eb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.678+0000 7ffa0adf8640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa040c8cf0 0x7ffa04150760 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.678+0000 7ffa0adf8640 1 --2- 172.21.15.46:0/2655196243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9f4004d10 0x7ffa04139340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:30.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.678+0000 7ffa0adf8640 1 -- 172.21.15.46:0/2655196243 >> 172.21.15.46:0/2655196243 conn(0x7ffa040c7910 msgr2=0x7ffa04147a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:30.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.678+0000 7ffa0adf8640 1 -- 172.21.15.46:0/2655196243 shutdown_connections 2023-10-30T13:47:30.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:30.678+0000 7ffa0adf8640 1 -- 172.21.15.46:0/2655196243 wait complete. 2023-10-30T13:47:30.702 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:32.703 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:32.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.846+0000 7fde599d3640 1 Processor -- start 2023-10-30T13:47:32.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.846+0000 7fde599d3640 1 -- start start 2023-10-30T13:47:32.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde599d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde541538a0 0x7fde5415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:32.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde599d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde540ca0e0 0x7fde5415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:32.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde599d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde540cf160 0x7fde540cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:32.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde599d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde540cb800 con 0x7fde541538a0 2023-10-30T13:47:32.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde599d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde5415dc30 con 0x7fde540cf160 2023-10-30T13:47:32.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde599d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde540cb330 con 0x7fde540ca0e0 2023-10-30T13:47:32.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde52ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde541538a0 0x7fde5415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:32.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde52ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde541538a0 0x7fde5415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51256/0 (socket says 172.21.15.46:51256) 2023-10-30T13:47:32.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde52ffd640 1 -- 172.21.15.46:0/1152279410 learned_addr learned my addr 172.21.15.46:0/1152279410 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:32.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde527fc640 1 --2- 172.21.15.46:0/1152279410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde540ca0e0 0x7fde5415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde537fe640 1 --2- 172.21.15.46:0/1152279410 >> v2:172.21.15.154:3300/0 conn(0x7fde540cf160 0x7fde540cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde52ffd640 1 -- 172.21.15.46:0/1152279410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde540ca0e0 msgr2=0x7fde5415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde52ffd640 1 --2- 172.21.15.46:0/1152279410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde540ca0e0 0x7fde5415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde52ffd640 1 -- 172.21.15.46:0/1152279410 >> v2:172.21.15.154:3300/0 conn(0x7fde540cf160 msgr2=0x7fde540cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde52ffd640 1 --2- 172.21.15.46:0/1152279410 >> v2:172.21.15.154:3300/0 conn(0x7fde540cf160 0x7fde540cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde52ffd640 1 -- 172.21.15.46:0/1152279410 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde540cad70 con 0x7fde541538a0 2023-10-30T13:47:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde527fc640 1 --2- 172.21.15.46:0/1152279410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde540ca0e0 0x7fde5415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:47:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde537fe640 1 --2- 172.21.15.46:0/1152279410 >> v2:172.21.15.154:3300/0 conn(0x7fde540cf160 0x7fde540cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:47:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.850+0000 7fde52ffd640 1 --2- 172.21.15.46:0/1152279410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde541538a0 0x7fde5415d6f0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fde5406cb00 tx=0x7fde4401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07d5ff3ac5b1fb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde51ffb640 1 -- 172.21.15.46:0/1152279410 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde440a89b0 con 0x7fde541538a0 2023-10-30T13:47:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde51ffb640 1 -- 172.21.15.46:0/1152279410 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde440a8b10 con 0x7fde541538a0 2023-10-30T13:47:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde51ffb640 1 -- 172.21.15.46:0/1152279410 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde440a8ea0 con 0x7fde541538a0 2023-10-30T13:47:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 -- 172.21.15.46:0/1152279410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde541538a0 msgr2=0x7fde5415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 --2- 172.21.15.46:0/1152279410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde541538a0 0x7fde5415d6f0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fde5406cb00 tx=0x7fde4401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:47:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 -- 172.21.15.46:0/1152279410 shutdown_connections 2023-10-30T13:47:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 --2- 172.21.15.46:0/1152279410 >> v2:172.21.15.154:3300/0 conn(0x7fde540cf160 0x7fde540cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 --2- 172.21.15.46:0/1152279410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde540ca0e0 0x7fde5415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 --2- 172.21.15.46:0/1152279410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde541538a0 0x7fde5415d6f0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 -- 172.21.15.46:0/1152279410 >> 172.21.15.46:0/1152279410 conn(0x7fde5406bfe0 msgr2=0x7fde5406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 -- 172.21.15.46:0/1152279410 shutdown_connections 2023-10-30T13:47:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 -- 172.21.15.46:0/1152279410 wait complete. 2023-10-30T13:47:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 Processor -- start 2023-10-30T13:47:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 -- start start 2023-10-30T13:47:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde540ca0e0 0x7fde540ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde540cf160 0x7fde541e3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde541538a0 0x7fde5414cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde541e2e60 con 0x7fde540ca0e0 2023-10-30T13:47:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde541e58c0 con 0x7fde540cf160 2023-10-30T13:47:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde599d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5415b600 con 0x7fde541538a0 2023-10-30T13:47:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.854+0000 7fde52ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde540ca0e0 0x7fde540ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde52ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde540ca0e0 0x7fde540ce390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51264/0 (socket says 172.21.15.46:51264) 2023-10-30T13:47:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde52ffd640 1 -- 172.21.15.46:0/731308695 learned_addr learned my addr 172.21.15.46:0/731308695 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde527fc640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde540cf160 0x7fde541e3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde537fe640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde541538a0 0x7fde5414cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde52ffd640 1 -- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde541538a0 msgr2=0x7fde5414cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde52ffd640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde541538a0 0x7fde5414cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde52ffd640 1 -- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde540cf160 msgr2=0x7fde541e3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde52ffd640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde540cf160 0x7fde541e3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde52ffd640 1 -- 172.21.15.46:0/731308695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde440094d0 con 0x7fde540ca0e0 2023-10-30T13:47:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde537fe640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde541538a0 0x7fde5414cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde527fc640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde540cf160 0x7fde541e3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde52ffd640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde540ca0e0 0x7fde540ce390 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fde4401d3a0 tx=0x7fde4401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde3bfff640 1 -- 172.21.15.46:0/731308695 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde440ba600 con 0x7fde540ca0e0 2023-10-30T13:47:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde3bfff640 1 -- 172.21.15.46:0/731308695 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde440ba780 con 0x7fde540ca0e0 2023-10-30T13:47:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde3bfff640 1 -- 172.21.15.46:0/731308695 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde440bab30 con 0x7fde540ca0e0 2023-10-30T13:47:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde599d3640 1 -- 172.21.15.46:0/731308695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde541f2770 con 0x7fde540ca0e0 2023-10-30T13:47:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde599d3640 1 -- 172.21.15.46:0/731308695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde541f2a10 con 0x7fde540ca0e0 2023-10-30T13:47:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde599d3640 1 -- 172.21.15.46:0/731308695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde540b2a50 con 0x7fde540ca0e0 2023-10-30T13:47:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.858+0000 7fde3bfff640 1 -- 172.21.15.46:0/731308695 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fde440bac90 con 0x7fde540ca0e0 2023-10-30T13:47:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.862+0000 7fde3bfff640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde400664d0 0x7fde40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.862+0000 7fde3bfff640 1 -- 172.21.15.46:0/731308695 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7fde44122ae0 con 0x7fde540ca0e0 2023-10-30T13:47:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.862+0000 7fde527fc640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde400664d0 0x7fde40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.862+0000 7fde527fc640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde400664d0 0x7fde40068990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fde540d0140 tx=0x7fde48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:32.866+0000 7fde3bfff640 1 -- 172.21.15.46:0/731308695 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fde440ef990 con 0x7fde540ca0e0 2023-10-30T13:47:33.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.058+0000 7fde599d3640 1 -- 172.21.15.46:0/731308695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde54159b50 con 0x7fde540ca0e0 2023-10-30T13:47:33.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.058+0000 7fde3bfff640 1 -- 172.21.15.46:0/731308695 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fde54159b50 con 0x7fde540ca0e0 2023-10-30T13:47:33.061 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:33.061 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:24.113529+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:33.062 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 111 2023-10-30T13:47:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.058+0000 7fde599d3640 1 -- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde400664d0 msgr2=0x7fde40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.058+0000 7fde599d3640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde400664d0 0x7fde40068990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fde540d0140 tx=0x7fde48008040 comp rx=0 tx=0).stop 2023-10-30T13:47:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.058+0000 7fde599d3640 1 -- 172.21.15.46:0/731308695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde540ca0e0 msgr2=0x7fde540ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.058+0000 7fde599d3640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde540ca0e0 0x7fde540ce390 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fde4401d3a0 tx=0x7fde4401ba90 comp rx=0 tx=0).stop 2023-10-30T13:47:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.058+0000 7fde599d3640 1 -- 172.21.15.46:0/731308695 shutdown_connections 2023-10-30T13:47:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.058+0000 7fde599d3640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde400664d0 0x7fde40068990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:33.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.058+0000 7fde599d3640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde541538a0 0x7fde5414cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:33.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.058+0000 7fde599d3640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde540cf160 0x7fde541e3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:33.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.058+0000 7fde599d3640 1 --2- 172.21.15.46:0/731308695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde540ca0e0 0x7fde540ce390 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:33.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.058+0000 7fde599d3640 1 -- 172.21.15.46:0/731308695 >> 172.21.15.46:0/731308695 conn(0x7fde5406bfe0 msgr2=0x7fde5414db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:33.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.062+0000 7fde599d3640 1 -- 172.21.15.46:0/731308695 shutdown_connections 2023-10-30T13:47:33.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:33.062+0000 7fde599d3640 1 -- 172.21.15.46:0/731308695 wait complete. 2023-10-30T13:47:33.084 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:35.084 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:35.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1b50d6640 1 Processor -- start 2023-10-30T13:47:35.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1b50d6640 1 -- start start 2023-10-30T13:47:35.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1b50d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1b006e470 0x7fe1b006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:35.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1b50d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe1b00c8490 0x7fe1b00c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:35.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1b50d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b006afa0 con 0x7fe1b0153d70 2023-10-30T13:47:35.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1b50d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b006d780 con 0x7fe1b006e470 2023-10-30T13:47:35.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1b50d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b0154150 con 0x7fe1b00c8490 2023-10-30T13:47:35.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1ae59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1b006e470 0x7fe1b006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1ae59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1b006e470 0x7fe1b006ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51272/0 (socket says 172.21.15.46:51272) 2023-10-30T13:47:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1ae59c640 1 -- 172.21.15.46:0/1694762526 learned_addr learned my addr 172.21.15.46:0/1694762526 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1af59e640 1 --2- 172.21.15.46:0/1694762526 >> v2:172.21.15.154:3300/0 conn(0x7fe1b00c8490 0x7fe1b00c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1ae59c640 1 -- 172.21.15.46:0/1694762526 >> v2:172.21.15.154:3300/0 conn(0x7fe1b00c8490 msgr2=0x7fe1b00c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:35.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1ae59c640 1 --2- 172.21.15.46:0/1694762526 >> v2:172.21.15.154:3300/0 conn(0x7fe1b00c8490 0x7fe1b00c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1ae59c640 1 -- 172.21.15.46:0/1694762526 >> v1:172.21.15.154:6789/0 conn(0x7fe1b0153d70 legacy=0x7fe1b00cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:35.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.226+0000 7fe1ae59c640 1 -- 172.21.15.46:0/1694762526 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b006e030 con 0x7fe1b006e470 2023-10-30T13:47:35.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.230+0000 7fe1ae59c640 1 --2- 172.21.15.46:0/1694762526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b006e470 0x7fe1b006ca80 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00cddf0 tx=0x7fe1a401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75392c162448bbc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:35.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.230+0000 7fe1add9b640 1 -- 172.21.15.46:0/1694762526 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a40aa5f0 con 0x7fe1b006e470 2023-10-30T13:47:35.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.230+0000 7fe1add9b640 1 -- 172.21.15.46:0/1694762526 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a40aa750 con 0x7fe1b006e470 2023-10-30T13:47:35.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.230+0000 7fe1add9b640 1 -- 172.21.15.46:0/1694762526 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a40aaae0 con 0x7fe1b006e470 2023-10-30T13:47:35.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.230+0000 7fe1b50d6640 1 -- 172.21.15.46:0/1694762526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b006e470 msgr2=0x7fe1b006ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:35.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.230+0000 7fe1b50d6640 1 --2- 172.21.15.46:0/1694762526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b006e470 0x7fe1b006ca80 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00cddf0 tx=0x7fe1a401c490 comp rx=0 tx=0).stop 2023-10-30T13:47:35.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.230+0000 7fe1b50d6640 1 -- 172.21.15.46:0/1694762526 shutdown_connections 2023-10-30T13:47:35.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.230+0000 7fe1b50d6640 1 --2- 172.21.15.46:0/1694762526 >> v2:172.21.15.154:3300/0 conn(0x7fe1b00c8490 0x7fe1b00c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.230+0000 7fe1b50d6640 1 --2- 172.21.15.46:0/1694762526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b006e470 0x7fe1b006ca80 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.230+0000 7fe1b50d6640 1 -- 172.21.15.46:0/1694762526 >> 172.21.15.46:0/1694762526 conn(0x7fe1b0071ea0 msgr2=0x7fe1b0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:35.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.230+0000 7fe1b50d6640 1 -- 172.21.15.46:0/1694762526 shutdown_connections 2023-10-30T13:47:35.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.230+0000 7fe1b50d6640 1 -- 172.21.15.46:0/1694762526 wait complete. 2023-10-30T13:47:35.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1b50d6640 1 Processor -- start 2023-10-30T13:47:35.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1b50d6640 1 -- start start 2023-10-30T13:47:35.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1b50d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b006e470 0x7fe1b01454c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:35.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1b50d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b00c8490 0x7fe1b0141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:35.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1b50d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b0153d70 0x7fe1b0140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:35.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1b50d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b01f3960 con 0x7fe1b0153d70 2023-10-30T13:47:35.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1b50d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b01432b0 con 0x7fe1b00c8490 2023-10-30T13:47:35.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1b50d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b014a630 con 0x7fe1b006e470 2023-10-30T13:47:35.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1af59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b0153d70 0x7fe1b0140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:35.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1af59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b0153d70 0x7fe1b0140b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51284/0 (socket says 172.21.15.46:51284) 2023-10-30T13:47:35.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1af59e640 1 -- 172.21.15.46:0/4275828357 learned_addr learned my addr 172.21.15.46:0/4275828357 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:35.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1ae59c640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b00c8490 0x7fe1b0141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:35.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1aed9d640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b006e470 0x7fe1b01454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:35.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1af59e640 1 -- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b006e470 msgr2=0x7fe1b01454c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:35.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1af59e640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b006e470 0x7fe1b01454c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1af59e640 1 -- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b00c8490 msgr2=0x7fe1b0141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:35.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1af59e640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b00c8490 0x7fe1b0141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1af59e640 1 -- 172.21.15.46:0/4275828357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a40094d0 con 0x7fe1b0153d70 2023-10-30T13:47:35.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1aed9d640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b006e470 0x7fe1b01454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:35.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1af59e640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b0153d70 0x7fe1b0140b70 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8007e40 tx=0x7fe1a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:35.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1977fe640 1 -- 172.21.15.46:0/4275828357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8014620 con 0x7fe1b0153d70 2023-10-30T13:47:35.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1977fe640 1 -- 172.21.15.46:0/4275828357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a800ee90 con 0x7fe1b0153d70 2023-10-30T13:47:35.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1977fe640 1 -- 172.21.15.46:0/4275828357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a80149e0 con 0x7fe1b0153d70 2023-10-30T13:47:35.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1b50d6640 1 -- 172.21.15.46:0/4275828357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b01f2fe0 con 0x7fe1b0153d70 2023-10-30T13:47:35.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1b50d6640 1 -- 172.21.15.46:0/4275828357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b01f32a0 con 0x7fe1b0153d70 2023-10-30T13:47:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.234+0000 7fe1b50d6640 1 -- 172.21.15.46:0/4275828357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b01f3aa0 con 0x7fe1b0153d70 2023-10-30T13:47:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.238+0000 7fe1977fe640 1 -- 172.21.15.46:0/4275828357 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe1a8014b40 con 0x7fe1b0153d70 2023-10-30T13:47:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.238+0000 7fe1977fe640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe19c0664a0 0x7fe19c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.238+0000 7fe1977fe640 1 -- 172.21.15.46:0/4275828357 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7fe1a8084c20 con 0x7fe1b0153d70 2023-10-30T13:47:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.238+0000 7fe1aed9d640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe19c0664a0 0x7fe19c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:35.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.238+0000 7fe1aed9d640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe19c0664a0 0x7fe19c068960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0003ec0 tx=0x7fe1a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.242+0000 7fe1977fe640 1 -- 172.21.15.46:0/4275828357 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe1b01f3aa0 con 0x7fe1b0153d70 2023-10-30T13:47:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.430+0000 7fe1b50d6640 1 -- 172.21.15.46:0/4275828357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1b01f3aa0 con 0x7fe1b0153d70 2023-10-30T13:47:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.430+0000 7fe1977fe640 1 -- 172.21.15.46:0/4275828357 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5731 (secure 0 0 0) 0x7fe1b01f3aa0 con 0x7fe1b0153d70 2023-10-30T13:47:35.434 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:35.434 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:34.895246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:resolve","state_seq":37,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 112 2023-10-30T13:47:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.434+0000 7fe1b50d6640 1 -- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe19c0664a0 msgr2=0x7fe19c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.434+0000 7fe1b50d6640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe19c0664a0 0x7fe19c068960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0003ec0 tx=0x7fe1a0009040 comp rx=0 tx=0).stop 2023-10-30T13:47:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.434+0000 7fe1b50d6640 1 -- 172.21.15.46:0/4275828357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b0153d70 msgr2=0x7fe1b0140b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.434+0000 7fe1b50d6640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b0153d70 0x7fe1b0140b70 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8007e40 tx=0x7fe1a800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.434+0000 7fe1b50d6640 1 -- 172.21.15.46:0/4275828357 shutdown_connections 2023-10-30T13:47:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.434+0000 7fe1b50d6640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe19c0664a0 0x7fe19c068960 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.434+0000 7fe1b50d6640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b0153d70 0x7fe1b0140b70 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.434+0000 7fe1b50d6640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b00c8490 0x7fe1b0141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.434+0000 7fe1b50d6640 1 --2- 172.21.15.46:0/4275828357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b006e470 0x7fe1b01454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.434+0000 7fe1b50d6640 1 -- 172.21.15.46:0/4275828357 >> 172.21.15.46:0/4275828357 conn(0x7fe1b0071ea0 msgr2=0x7fe1b0152c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.434+0000 7fe1b50d6640 1 -- 172.21.15.46:0/4275828357 shutdown_connections 2023-10-30T13:47:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:35.434+0000 7fe1b50d6640 1 -- 172.21.15.46:0/4275828357 wait complete. 2023-10-30T13:47:35.455 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:37.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:37.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.606+0000 7f40e1446640 1 Processor -- start 2023-10-30T13:47:37.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.606+0000 7f40e1446640 1 -- start start 2023-10-30T13:47:37.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.606+0000 7f40e1446640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40dc0c6f60 0x7f40dc1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:37.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.606+0000 7f40e1446640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40dc0c0a50 con 0x7f40dc0c6b80 2023-10-30T13:47:37.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.606+0000 7f40e1446640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40dc06f6e0 con 0x7f40dc0c05d0 2023-10-30T13:47:37.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.606+0000 7f40e1446640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc06f9c0 con 0x7f40dc0c6f60 2023-10-30T13:47:37.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.606+0000 7f40daffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f40dc0c6b80 0x7f40dc0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52410/0 (socket says 172.21.15.46:52410) 2023-10-30T13:47:37.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.610+0000 7f40daffd640 1 -- 172.21.15.46:0/1978541275 learned_addr learned my addr 172.21.15.46:0/1978541275 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:47:37.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.610+0000 7f40db7fe640 1 --2- 172.21.15.46:0/1978541275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40dc0c6f60 0x7f40dc1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:37.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.610+0000 7f40da7fc640 1 -- 172.21.15.46:0/1978541275 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233306436 0 0) 0x7f40dc0c0a50 con 0x7f40dc0c6b80 2023-10-30T13:47:37.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.610+0000 7f40da7fc640 1 -- 172.21.15.46:0/1978541275 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c8005740 con 0x7f40dc0c6b80 2023-10-30T13:47:37.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.610+0000 7f40da7fc640 1 -- 172.21.15.46:0/1978541275 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104729210 0 0) 0x7f40dc06f6e0 con 0x7f40dc0c05d0 2023-10-30T13:47:37.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.610+0000 7f40da7fc640 1 -- 172.21.15.46:0/1978541275 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40dc0c0a50 con 0x7f40dc0c05d0 2023-10-30T13:47:37.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.610+0000 7f40db7fe640 1 -- 172.21.15.46:0/1978541275 >> v1:172.21.15.154:6789/0 conn(0x7f40dc0c05d0 legacy=0x7f40dc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.610+0000 7f40da7fc640 1 -- 172.21.15.46:0/1978541275 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3669199424 0 0) 0x7f40c8005740 con 0x7f40dc0c6b80 2023-10-30T13:47:37.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.610+0000 7f40db7fe640 1 -- 172.21.15.46:0/1978541275 >> v1:172.21.15.46:6789/0 conn(0x7f40dc0c6b80 legacy=0x7f40dc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.610+0000 7f40db7fe640 1 -- 172.21.15.46:0/1978541275 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc06fca0 con 0x7f40dc0c6f60 2023-10-30T13:47:37.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.610+0000 7f40db7fe640 1 --2- 172.21.15.46:0/1978541275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40dc0c6f60 0x7f40dc1566a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f40d4009dc0 tx=0x7f40d401c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=645b4af7b9c2013a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:37.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40da7fc640 1 -- 172.21.15.46:0/1978541275 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc001680 con 0x7f40dc0c6f60 2023-10-30T13:47:37.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40da7fc640 1 -- 172.21.15.46:0/1978541275 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d40a64f0 con 0x7f40dc0c6f60 2023-10-30T13:47:37.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40da7fc640 1 -- 172.21.15.46:0/1978541275 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d40a6880 con 0x7f40dc0c6f60 2023-10-30T13:47:37.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 -- 172.21.15.46:0/1978541275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40dc0c6f60 msgr2=0x7f40dc1566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 --2- 172.21.15.46:0/1978541275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40dc0c6f60 0x7f40dc1566a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f40d4009dc0 tx=0x7f40d401c890 comp rx=0 tx=0).stop 2023-10-30T13:47:37.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 -- 172.21.15.46:0/1978541275 shutdown_connections 2023-10-30T13:47:37.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 --2- 172.21.15.46:0/1978541275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40dc0c6f60 0x7f40dc1566a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 -- 172.21.15.46:0/1978541275 >> 172.21.15.46:0/1978541275 conn(0x7f40dc0c7910 msgr2=0x7f40dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:37.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 -- 172.21.15.46:0/1978541275 shutdown_connections 2023-10-30T13:47:37.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 -- 172.21.15.46:0/1978541275 wait complete. 2023-10-30T13:47:37.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 Processor -- start 2023-10-30T13:47:37.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 -- start start 2023-10-30T13:47:37.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40dc0c05d0 0x7f40dc0ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:37.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40dc0c6b80 0x7f40dc158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:37.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc0c6f60 0x7f40dc13c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:37.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc0ca040 con 0x7f40dc0c6f60 2023-10-30T13:47:37.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc13e190 con 0x7f40dc0c6b80 2023-10-30T13:47:37.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40e1446640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc14a7e0 con 0x7f40dc0c05d0 2023-10-30T13:47:37.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40db7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc0c6f60 0x7f40dc13c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:37.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40db7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc0c6f60 0x7f40dc13c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42210/0 (socket says 172.21.15.46:42210) 2023-10-30T13:47:37.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.614+0000 7f40db7fe640 1 -- 172.21.15.46:0/2409834247 learned_addr learned my addr 172.21.15.46:0/2409834247 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:37.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40daffd640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40dc0c05d0 0x7f40dc0ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:37.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40d3fff640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40dc0c6b80 0x7f40dc158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:37.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40db7fe640 1 -- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40dc0c05d0 msgr2=0x7f40dc0ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40db7fe640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40dc0c05d0 0x7f40dc0ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40db7fe640 1 -- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40dc0c6b80 msgr2=0x7f40dc158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40db7fe640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40dc0c6b80 0x7f40dc158760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40db7fe640 1 -- 172.21.15.46:0/2409834247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4009a70 con 0x7f40dc0c6f60 2023-10-30T13:47:37.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40daffd640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40dc0c05d0 0x7f40dc0ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:37.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40db7fe640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc0c6f60 0x7f40dc13c820 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f40d40b0040 tx=0x7f40d4019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:37.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40d37fe640 1 -- 172.21.15.46:0/2409834247 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d4003ba0 con 0x7f40dc0c6f60 2023-10-30T13:47:37.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40d37fe640 1 -- 172.21.15.46:0/2409834247 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d4003d20 con 0x7f40dc0c6f60 2023-10-30T13:47:37.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40d37fe640 1 -- 172.21.15.46:0/2409834247 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d4004270 con 0x7f40dc0c6f60 2023-10-30T13:47:37.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40e1446640 1 -- 172.21.15.46:0/2409834247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc06ac60 con 0x7f40dc0c6f60 2023-10-30T13:47:37.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40e1446640 1 -- 172.21.15.46:0/2409834247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc13db60 con 0x7f40dc0c6f60 2023-10-30T13:47:37.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40e1446640 1 -- 172.21.15.46:0/2409834247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc04e170 con 0x7f40dc0c6f60 2023-10-30T13:47:37.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40d37fe640 1 -- 172.21.15.46:0/2409834247 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f40d40043d0 con 0x7f40dc0c6f60 2023-10-30T13:47:37.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40d37fe640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f40bc0664a0 0x7f40bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:37.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.618+0000 7f40d37fe640 1 -- 172.21.15.46:0/2409834247 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f40d40aa070 con 0x7f40dc0c6f60 2023-10-30T13:47:37.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.622+0000 7f40daffd640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f40bc0664a0 0x7f40bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:37.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.622+0000 7f40daffd640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f40bc0664a0 0x7f40bc068960 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f40cc000f70 tx=0x7f40cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:37.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.622+0000 7f40d37fe640 1 -- 172.21.15.46:0/2409834247 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f40d4122030 con 0x7f40dc0c6f60 2023-10-30T13:47:37.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 -- 172.21.15.46:0/2409834247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40dc06b0d0 con 0x7f40dc0c6f60 2023-10-30T13:47:37.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40d37fe640 1 -- 172.21.15.46:0/2409834247 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f40d40ef8f0 con 0x7f40dc0c6f60 2023-10-30T13:47:37.818 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:37.818 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:34.895246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:resolve","state_seq":37,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:37.818 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 112 2023-10-30T13:47:37.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 -- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f40bc0664a0 msgr2=0x7f40bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f40bc0664a0 0x7f40bc068960 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f40cc000f70 tx=0x7f40cc009040 comp rx=0 tx=0).stop 2023-10-30T13:47:37.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 -- 172.21.15.46:0/2409834247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc0c6f60 msgr2=0x7f40dc13c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:37.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc0c6f60 0x7f40dc13c820 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f40d40b0040 tx=0x7f40d4019630 comp rx=0 tx=0).stop 2023-10-30T13:47:37.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 -- 172.21.15.46:0/2409834247 shutdown_connections 2023-10-30T13:47:37.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f40bc0664a0 0x7f40bc068960 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40dc0c6f60 0x7f40dc13c820 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40dc0c6b80 0x7f40dc158760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 --2- 172.21.15.46:0/2409834247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40dc0c05d0 0x7f40dc0ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:37.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 -- 172.21.15.46:0/2409834247 >> 172.21.15.46:0/2409834247 conn(0x7f40dc0c7910 msgr2=0x7f40dc0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:37.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 -- 172.21.15.46:0/2409834247 shutdown_connections 2023-10-30T13:47:37.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:37.814+0000 7f40e1446640 1 -- 172.21.15.46:0/2409834247 wait complete. 2023-10-30T13:47:37.839 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 112, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 112, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6366": { "addr": "172.21.15.46:6833/190134548", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 190134548, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 190134548, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6366, "incarnation": 104, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 51 }, "gid_6702": { "addr": "172.21.15.46:6836/2197729559", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 2197729559, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 2197729559, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6702, "incarnation": 111, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:resolve", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 69, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:47:34.895246+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6702, "mds_1": 6366 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:47:39.840 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:39.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.990+0000 7f1713a91640 1 Processor -- start 2023-10-30T13:47:39.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.990+0000 7f1713a91640 1 -- start start 2023-10-30T13:47:39.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.990+0000 7f1713a91640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f170c070740 0x7f170c0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:39.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.990+0000 7f1713a91640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f170c15e700 0x7f170c06a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:39.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.990+0000 7f1713a91640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170c0c5810 con 0x7f170c1580e0 2023-10-30T13:47:39.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.990+0000 7f1713a91640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f170c15eb70 con 0x7f170c070740 2023-10-30T13:47:39.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.990+0000 7f1713a91640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f170c0c6bb0 con 0x7f170c15e700 2023-10-30T13:47:39.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.994+0000 7f171102c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f170c070740 0x7f170c0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:39.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.994+0000 7f171102c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f170c070740 0x7f170c0c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42224/0 (socket says 172.21.15.46:42224) 2023-10-30T13:47:39.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.994+0000 7f171102c640 1 -- 172.21.15.46:0/2306884307 learned_addr learned my addr 172.21.15.46:0/2306884307 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:39.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.994+0000 7f171202e640 1 --2- 172.21.15.46:0/2306884307 >> v2:172.21.15.154:3300/0 conn(0x7f170c15e700 0x7f170c06a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:39.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.994+0000 7f171102c640 1 -- 172.21.15.46:0/2306884307 >> v2:172.21.15.154:3300/0 conn(0x7f170c15e700 msgr2=0x7f170c06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:39.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.994+0000 7f171102c640 1 --2- 172.21.15.46:0/2306884307 >> v2:172.21.15.154:3300/0 conn(0x7f170c15e700 0x7f170c06a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:39.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.994+0000 7f171102c640 1 -- 172.21.15.46:0/2306884307 >> v1:172.21.15.154:6789/0 conn(0x7f170c1580e0 legacy=0x7f170c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.994+0000 7f171102c640 1 -- 172.21.15.46:0/2306884307 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c0c6e70 con 0x7f170c070740 2023-10-30T13:47:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.994+0000 7f171102c640 1 --2- 172.21.15.46:0/2306884307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f170c070740 0x7f170c0c6540 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f170c0cbeb0 tx=0x7f16fc01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7996f388644303ec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.994+0000 7f171082b640 1 -- 172.21.15.46:0/2306884307 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc0ac410 con 0x7f170c070740 2023-10-30T13:47:40.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.994+0000 7f171082b640 1 -- 172.21.15.46:0/2306884307 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc0ac570 con 0x7f170c070740 2023-10-30T13:47:40.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.994+0000 7f171082b640 1 -- 172.21.15.46:0/2306884307 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc0ac900 con 0x7f170c070740 2023-10-30T13:47:40.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 -- 172.21.15.46:0/2306884307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f170c070740 msgr2=0x7f170c0c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:40.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 --2- 172.21.15.46:0/2306884307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f170c070740 0x7f170c0c6540 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f170c0cbeb0 tx=0x7f16fc01c4c0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 -- 172.21.15.46:0/2306884307 shutdown_connections 2023-10-30T13:47:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 --2- 172.21.15.46:0/2306884307 >> v2:172.21.15.154:3300/0 conn(0x7f170c15e700 0x7f170c06a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 --2- 172.21.15.46:0/2306884307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f170c070740 0x7f170c0c6540 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 -- 172.21.15.46:0/2306884307 >> 172.21.15.46:0/2306884307 conn(0x7f170c0c7910 msgr2=0x7f170c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 -- 172.21.15.46:0/2306884307 shutdown_connections 2023-10-30T13:47:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 -- 172.21.15.46:0/2306884307 wait complete. 2023-10-30T13:47:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 Processor -- start 2023-10-30T13:47:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 -- start start 2023-10-30T13:47:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f170c070740 0x7f170c1f6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f170c1580e0 0x7f170c149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f170c15e700 0x7f170c070f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0c8d90 con 0x7f170c070740 2023-10-30T13:47:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0c9030 con 0x7f170c15e700 2023-10-30T13:47:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f1713a91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170c1f6380 con 0x7f170c1580e0 2023-10-30T13:47:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f171182d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f170c070740 0x7f170c1f6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f171182d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f170c070740 0x7f170c1f6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42226/0 (socket says 172.21.15.46:42226) 2023-10-30T13:47:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f171182d640 1 -- 172.21.15.46:0/1124785401 learned_addr learned my addr 172.21.15.46:0/1124785401 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f171102c640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f170c1580e0 0x7f170c149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f171202e640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f170c15e700 0x7f170c070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f171182d640 1 -- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f170c1580e0 msgr2=0x7f170c149f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f171182d640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f170c1580e0 0x7f170c149f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f171182d640 1 -- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f170c15e700 msgr2=0x7f170c070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f171182d640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f170c15e700 0x7f170c070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f171182d640 1 -- 172.21.15.46:0/1124785401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16fc0094d0 con 0x7f170c070740 2023-10-30T13:47:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:39.998+0000 7f171202e640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f170c15e700 0x7f170c070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.002+0000 7f171182d640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f170c070740 0x7f170c1f6e60 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f170800c450 tx=0x7f170800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.002+0000 7f17027fc640 1 -- 172.21.15.46:0/1124785401 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1708012830 con 0x7f170c070740 2023-10-30T13:47:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.002+0000 7f17027fc640 1 -- 172.21.15.46:0/1124785401 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17080129b0 con 0x7f170c070740 2023-10-30T13:47:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.002+0000 7f1713a91640 1 -- 172.21.15.46:0/1124785401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170c1f6660 con 0x7f170c070740 2023-10-30T13:47:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.002+0000 7f17027fc640 1 -- 172.21.15.46:0/1124785401 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1708012d60 con 0x7f170c070740 2023-10-30T13:47:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.002+0000 7f1713a91640 1 -- 172.21.15.46:0/1124785401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170c071460 con 0x7f170c070740 2023-10-30T13:47:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.002+0000 7f1713a91640 1 -- 172.21.15.46:0/1124785401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170c04e170 con 0x7f170c070740 2023-10-30T13:47:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.002+0000 7f17027fc640 1 -- 172.21.15.46:0/1124785401 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f170801c460 con 0x7f170c070740 2023-10-30T13:47:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.002+0000 7f17027fc640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16f00664d0 0x7f16f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.002+0000 7f17027fc640 1 -- 172.21.15.46:0/1124785401 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f1708084210 con 0x7f170c070740 2023-10-30T13:47:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.002+0000 7f171102c640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16f00664d0 0x7f16f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.006+0000 7f171102c640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16f00664d0 0x7f16f0068990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f16fc0ae000 tx=0x7f16fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.010+0000 7f17027fc640 1 -- 172.21.15.46:0/1124785401 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f17080510c0 con 0x7f170c070740 2023-10-30T13:47:40.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.198+0000 7f1713a91640 1 -- 172.21.15.46:0/1124785401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f170c1683e0 con 0x7f170c070740 2023-10-30T13:47:40.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.198+0000 7f17027fc640 1 -- 172.21.15.46:0/1124785401 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f17080546e0 con 0x7f170c070740 2023-10-30T13:47:40.202 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:40.202 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:34.895246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:resolve","state_seq":37,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 112 2023-10-30T13:47:40.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.198+0000 7f1713a91640 1 -- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16f00664d0 msgr2=0x7f16f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:40.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.198+0000 7f1713a91640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16f00664d0 0x7f16f0068990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f16fc0ae000 tx=0x7f16fc009040 comp rx=0 tx=0).stop 2023-10-30T13:47:40.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.198+0000 7f1713a91640 1 -- 172.21.15.46:0/1124785401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f170c070740 msgr2=0x7f170c1f6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:40.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.198+0000 7f1713a91640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f170c070740 0x7f170c1f6e60 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f170800c450 tx=0x7f170800c910 comp rx=0 tx=0).stop 2023-10-30T13:47:40.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.198+0000 7f1713a91640 1 -- 172.21.15.46:0/1124785401 shutdown_connections 2023-10-30T13:47:40.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.198+0000 7f1713a91640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f16f00664d0 0x7f16f0068990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.198+0000 7f1713a91640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f170c15e700 0x7f170c070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.198+0000 7f1713a91640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f170c1580e0 0x7f170c149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.198+0000 7f1713a91640 1 --2- 172.21.15.46:0/1124785401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f170c070740 0x7f170c1f6e60 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:40.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.198+0000 7f1713a91640 1 -- 172.21.15.46:0/1124785401 >> 172.21.15.46:0/1124785401 conn(0x7f170c0c7910 msgr2=0x7f170c06bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:40.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.202+0000 7f1713a91640 1 -- 172.21.15.46:0/1124785401 shutdown_connections 2023-10-30T13:47:40.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:40.202+0000 7f1713a91640 1 -- 172.21.15.46:0/1124785401 wait complete. 2023-10-30T13:47:40.222 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:42.224 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:42.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.374+0000 7f1214417640 1 Processor -- start 2023-10-30T13:47:42.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.374+0000 7f1214417640 1 -- start start 2023-10-30T13:47:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f1214417640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f120c153bf0 0x7f120c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f1214417640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f120c0c5f60 0x7f120c0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f1214417640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c0700b0 con 0x7f120c0c6fe0 2023-10-30T13:47:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f1214417640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f120c0c6810 con 0x7f120c0c5f60 2023-10-30T13:47:42.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f1214417640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120c159900 con 0x7f120c153bf0 2023-10-30T13:47:42.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f12129b4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f120c0c5f60 0x7f120c0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:42.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f12129b4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f120c0c5f60 0x7f120c0cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42230/0 (socket says 172.21.15.46:42230) 2023-10-30T13:47:42.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f12129b4640 1 -- 172.21.15.46:0/592924617 learned_addr learned my addr 172.21.15.46:0/592924617 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:42.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f12121b3640 1 --2- 172.21.15.46:0/592924617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f120c153bf0 0x7f120c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:42.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f12129b4640 1 -- 172.21.15.46:0/592924617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f120c153bf0 msgr2=0x7f120c071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:42.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f12129b4640 1 --2- 172.21.15.46:0/592924617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f120c153bf0 0x7f120c071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:42.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f12129b4640 1 -- 172.21.15.46:0/592924617 >> v1:172.21.15.154:6789/0 conn(0x7f120c0c6fe0 legacy=0x7f120c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:42.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.378+0000 7f12129b4640 1 -- 172.21.15.46:0/592924617 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c0c6530 con 0x7f120c0c5f60 2023-10-30T13:47:42.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f12129b4640 1 --2- 172.21.15.46:0/592924617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f120c0c5f60 0x7f120c0cb570 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f1204009e50 tx=0x7f120401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=681ecc4adb72e014 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:42.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f12111b1640 1 -- 172.21.15.46:0/592924617 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12040af400 con 0x7f120c0c5f60 2023-10-30T13:47:42.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f12111b1640 1 -- 172.21.15.46:0/592924617 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12040ab050 con 0x7f120c0c5f60 2023-10-30T13:47:42.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f12111b1640 1 -- 172.21.15.46:0/592924617 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12040af780 con 0x7f120c0c5f60 2023-10-30T13:47:42.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f1214417640 1 -- 172.21.15.46:0/592924617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f120c0c5f60 msgr2=0x7f120c0cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:42.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f1214417640 1 --2- 172.21.15.46:0/592924617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f120c0c5f60 0x7f120c0cb570 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f1204009e50 tx=0x7f120401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:47:42.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f1214417640 1 -- 172.21.15.46:0/592924617 shutdown_connections 2023-10-30T13:47:42.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f1214417640 1 --2- 172.21.15.46:0/592924617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f120c0c5f60 0x7f120c0cb570 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:42.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f1214417640 1 --2- 172.21.15.46:0/592924617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f120c153bf0 0x7f120c071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:42.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f1214417640 1 -- 172.21.15.46:0/592924617 >> 172.21.15.46:0/592924617 conn(0x7f120c06c6e0 msgr2=0x7f120c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:42.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f1214417640 1 -- 172.21.15.46:0/592924617 shutdown_connections 2023-10-30T13:47:42.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f1214417640 1 -- 172.21.15.46:0/592924617 wait complete. 2023-10-30T13:47:42.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f1214417640 1 Processor -- start 2023-10-30T13:47:42.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.382+0000 7f1214417640 1 -- start start 2023-10-30T13:47:42.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1214417640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f120c0c5f60 0x7f120c148390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:42.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1214417640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f120c0c6fe0 0x7f120c1f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:42.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1214417640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f120c153bf0 0x7f120c0c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:42.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1214417640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c1f2850 con 0x7f120c0c6fe0 2023-10-30T13:47:42.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1214417640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c1570a0 con 0x7f120c0c5f60 2023-10-30T13:47:42.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1214417640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120c0ca040 con 0x7f120c153bf0 2023-10-30T13:47:42.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f12119b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f120c0c6fe0 0x7f120c1f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:42.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f12119b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f120c0c6fe0 0x7f120c1f2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42238/0 (socket says 172.21.15.46:42238) 2023-10-30T13:47:42.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f12119b2640 1 -- 172.21.15.46:0/1047166858 learned_addr learned my addr 172.21.15.46:0/1047166858 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:42.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f12121b3640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f120c0c5f60 0x7f120c148390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:42.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f12129b4640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f120c153bf0 0x7f120c0c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:42.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f12119b2640 1 -- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f120c153bf0 msgr2=0x7f120c0c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:42.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f12119b2640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f120c153bf0 0x7f120c0c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:42.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f12119b2640 1 -- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f120c0c5f60 msgr2=0x7f120c148390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:42.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f12119b2640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f120c0c5f60 0x7f120c148390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:42.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f12119b2640 1 -- 172.21.15.46:0/1047166858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1204009b30 con 0x7f120c0c6fe0 2023-10-30T13:47:42.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f12119b2640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f120c0c6fe0 0x7f120c1f2ce0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f11fc00c500 tx=0x7f11fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:42.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1202ffd640 1 -- 172.21.15.46:0/1047166858 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc005220 con 0x7f120c0c6fe0 2023-10-30T13:47:42.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1202ffd640 1 -- 172.21.15.46:0/1047166858 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc005380 con 0x7f120c0c6fe0 2023-10-30T13:47:42.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1214417640 1 -- 172.21.15.46:0/1047166858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120c1f2220 con 0x7f120c0c6fe0 2023-10-30T13:47:42.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1202ffd640 1 -- 172.21.15.46:0/1047166858 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc007270 con 0x7f120c0c6fe0 2023-10-30T13:47:42.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1214417640 1 -- 172.21.15.46:0/1047166858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120c0c4130 con 0x7f120c0c6fe0 2023-10-30T13:47:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1214417640 1 -- 172.21.15.46:0/1047166858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120c04e170 con 0x7f120c0c6fe0 2023-10-30T13:47:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.386+0000 7f1202ffd640 1 -- 172.21.15.46:0/1047166858 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f11fc0073d0 con 0x7f120c0c6fe0 2023-10-30T13:47:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.390+0000 7f1202ffd640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11f00664a0 0x7f11f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.390+0000 7f1202ffd640 1 -- 172.21.15.46:0/1047166858 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f11fc083b70 con 0x7f120c0c6fe0 2023-10-30T13:47:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.390+0000 7f12121b3640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11f00664a0 0x7f11f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.390+0000 7f12121b3640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11f00664a0 0x7f11f0068960 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f12080079c0 tx=0x7f1208008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.394+0000 7f1202ffd640 1 -- 172.21.15.46:0/1047166858 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f11fc050b70 con 0x7f120c0c6fe0 2023-10-30T13:47:42.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.586+0000 7f1214417640 1 -- 172.21.15.46:0/1047166858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f120c06fe90 con 0x7f120c0c6fe0 2023-10-30T13:47:42.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.586+0000 7f1202ffd640 1 -- 172.21.15.46:0/1047166858 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f11fc054190 con 0x7f120c0c6fe0 2023-10-30T13:47:42.591 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:42.591 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:40.203380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:reconnect","state_seq":39,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:42.591 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 113 2023-10-30T13:47:42.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.590+0000 7f1214417640 1 -- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11f00664a0 msgr2=0x7f11f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:42.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.590+0000 7f1214417640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11f00664a0 0x7f11f0068960 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f12080079c0 tx=0x7f1208008040 comp rx=0 tx=0).stop 2023-10-30T13:47:42.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.590+0000 7f1214417640 1 -- 172.21.15.46:0/1047166858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f120c0c6fe0 msgr2=0x7f120c1f2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:42.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.590+0000 7f1214417640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f120c0c6fe0 0x7f120c1f2ce0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f11fc00c500 tx=0x7f11fc00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:47:42.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.590+0000 7f1214417640 1 -- 172.21.15.46:0/1047166858 shutdown_connections 2023-10-30T13:47:42.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.590+0000 7f1214417640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11f00664a0 0x7f11f0068960 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:42.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.590+0000 7f1214417640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f120c153bf0 0x7f120c0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:42.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.590+0000 7f1214417640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f120c0c6fe0 0x7f120c1f2ce0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:42.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.590+0000 7f1214417640 1 --2- 172.21.15.46:0/1047166858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f120c0c5f60 0x7f120c148390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:42.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.590+0000 7f1214417640 1 -- 172.21.15.46:0/1047166858 >> 172.21.15.46:0/1047166858 conn(0x7f120c06c6e0 msgr2=0x7f120c157d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:42.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.590+0000 7f1214417640 1 -- 172.21.15.46:0/1047166858 shutdown_connections 2023-10-30T13:47:42.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:42.590+0000 7f1214417640 1 -- 172.21.15.46:0/1047166858 wait complete. 2023-10-30T13:47:42.611 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:44.612 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.762+0000 7f8262adc640 1 Processor -- start 2023-10-30T13:47:44.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.762+0000 7f8262adc640 1 -- start start 2023-10-30T13:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f8262adc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f825c0717b0 0x7f825c15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f8262adc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f825c0c6290 0x7f825c161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f8262adc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c15b060 con 0x7f825c071b90 2023-10-30T13:47:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f8262adc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f825c157530 con 0x7f825c0717b0 2023-10-30T13:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f8262adc640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f825c0c4ab0 con 0x7f825c0c6290 2023-10-30T13:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f8261ada640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f825c0717b0 0x7f825c15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f8261ada640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f825c0717b0 0x7f825c15ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42244/0 (socket says 172.21.15.46:42244) 2023-10-30T13:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f8261ada640 1 -- 172.21.15.46:0/2631272896 learned_addr learned my addr 172.21.15.46:0/2631272896 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f82612d9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f825c071b90 0x7f825c1513c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52424/0 (socket says 172.21.15.46:52424) 2023-10-30T13:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f82622db640 1 --2- 172.21.15.46:0/2631272896 >> v2:172.21.15.154:3300/0 conn(0x7f825c0c6290 0x7f825c161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f8261ada640 1 -- 172.21.15.46:0/2631272896 >> v2:172.21.15.154:3300/0 conn(0x7f825c0c6290 msgr2=0x7f825c161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f8261ada640 1 --2- 172.21.15.46:0/2631272896 >> v2:172.21.15.154:3300/0 conn(0x7f825c0c6290 0x7f825c161eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f8261ada640 1 -- 172.21.15.46:0/2631272896 >> v1:172.21.15.46:6789/0 conn(0x7f825c071b90 legacy=0x7f825c1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.766+0000 7f8261ada640 1 -- 172.21.15.46:0/2631272896 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c0c4d90 con 0x7f825c0717b0 2023-10-30T13:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8261ada640 1 --2- 172.21.15.46:0/2631272896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f825c0717b0 0x7f825c15ab20 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f825c06c3d0 tx=0x7f825801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9baa54b5289744f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8260ad8640 1 -- 172.21.15.46:0/2631272896 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82580a89f0 con 0x7f825c0717b0 2023-10-30T13:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8260ad8640 1 -- 172.21.15.46:0/2631272896 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82580a8b50 con 0x7f825c0717b0 2023-10-30T13:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8260ad8640 1 -- 172.21.15.46:0/2631272896 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82580b0440 con 0x7f825c0717b0 2023-10-30T13:47:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8262adc640 1 -- 172.21.15.46:0/2631272896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f825c0717b0 msgr2=0x7f825c15ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8262adc640 1 --2- 172.21.15.46:0/2631272896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f825c0717b0 0x7f825c15ab20 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f825c06c3d0 tx=0x7f825801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8262adc640 1 -- 172.21.15.46:0/2631272896 shutdown_connections 2023-10-30T13:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8262adc640 1 --2- 172.21.15.46:0/2631272896 >> v2:172.21.15.154:3300/0 conn(0x7f825c0c6290 0x7f825c161eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8262adc640 1 --2- 172.21.15.46:0/2631272896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f825c0717b0 0x7f825c15ab20 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8262adc640 1 -- 172.21.15.46:0/2631272896 >> 172.21.15.46:0/2631272896 conn(0x7f825c0c75d0 msgr2=0x7f825c0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8262adc640 1 -- 172.21.15.46:0/2631272896 shutdown_connections 2023-10-30T13:47:44.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8262adc640 1 -- 172.21.15.46:0/2631272896 wait complete. 2023-10-30T13:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8262adc640 1 Processor -- start 2023-10-30T13:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8262adc640 1 -- start start 2023-10-30T13:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8262adc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f825c0717b0 0x7f825c156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.770+0000 7f8262adc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f825c071b90 0x7f825c1ef380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:44.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f8262adc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f825c0c6290 0x7f825c1ef8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:44.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f8262adc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c1f34c0 con 0x7f825c071b90 2023-10-30T13:47:44.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f8262adc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c1f2840 con 0x7f825c0c6290 2023-10-30T13:47:44.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f8262adc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825c1f2b20 con 0x7f825c0717b0 2023-10-30T13:47:44.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82612d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f825c071b90 0x7f825c1ef380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82612d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f825c071b90 0x7f825c1ef380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42248/0 (socket says 172.21.15.46:42248) 2023-10-30T13:47:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82612d9640 1 -- 172.21.15.46:0/2040254184 learned_addr learned my addr 172.21.15.46:0/2040254184 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82622db640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f825c0c6290 0x7f825c1ef8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f8261ada640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f825c0717b0 0x7f825c156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82612d9640 1 -- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f825c0717b0 msgr2=0x7f825c156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82612d9640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f825c0717b0 0x7f825c156560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82612d9640 1 -- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f825c0c6290 msgr2=0x7f825c1ef8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82612d9640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f825c0c6290 0x7f825c1ef8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82612d9640 1 -- 172.21.15.46:0/2040254184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8258009470 con 0x7f825c071b90 2023-10-30T13:47:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82612d9640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f825c071b90 0x7f825c1ef380 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f824c00c430 tx=0x7f824c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:44.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82527fc640 1 -- 172.21.15.46:0/2040254184 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824c0074e0 con 0x7f825c071b90 2023-10-30T13:47:44.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82527fc640 1 -- 172.21.15.46:0/2040254184 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824c007640 con 0x7f825c071b90 2023-10-30T13:47:44.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82527fc640 1 -- 172.21.15.46:0/2040254184 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824c004a10 con 0x7f825c071b90 2023-10-30T13:47:44.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f8262adc640 1 -- 172.21.15.46:0/2040254184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825c1f2e00 con 0x7f825c071b90 2023-10-30T13:47:44.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f8262adc640 1 -- 172.21.15.46:0/2040254184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f825c1f2550 con 0x7f825c071b90 2023-10-30T13:47:44.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f8262adc640 1 -- 172.21.15.46:0/2040254184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f825c04e170 con 0x7f825c071b90 2023-10-30T13:47:44.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.774+0000 7f82527fc640 1 -- 172.21.15.46:0/2040254184 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f824c004b70 con 0x7f825c071b90 2023-10-30T13:47:44.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.778+0000 7f82527fc640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82400664a0 0x7f8240068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:44.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.778+0000 7f82527fc640 1 -- 172.21.15.46:0/2040254184 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f824c083e60 con 0x7f825c071b90 2023-10-30T13:47:44.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.778+0000 7f8261ada640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82400664a0 0x7f8240068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:44.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.778+0000 7f8261ada640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82400664a0 0x7f8240068960 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f82580b2000 tx=0x7f8258008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:44.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.782+0000 7f82527fc640 1 -- 172.21.15.46:0/2040254184 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f824c050d10 con 0x7f825c071b90 2023-10-30T13:47:44.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.970+0000 7f8262adc640 1 -- 172.21.15.46:0/2040254184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f825c1579d0 con 0x7f825c071b90 2023-10-30T13:47:44.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.970+0000 7f82527fc640 1 -- 172.21.15.46:0/2040254184 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f824c054330 con 0x7f825c071b90 2023-10-30T13:47:44.975 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:44.976 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:40.203380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:reconnect","state_seq":39,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:44.976 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 113 2023-10-30T13:47:44.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.974+0000 7f8262adc640 1 -- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82400664a0 msgr2=0x7f8240068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:44.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.974+0000 7f8262adc640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82400664a0 0x7f8240068960 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f82580b2000 tx=0x7f8258008040 comp rx=0 tx=0).stop 2023-10-30T13:47:44.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.974+0000 7f8262adc640 1 -- 172.21.15.46:0/2040254184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f825c071b90 msgr2=0x7f825c1ef380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:44.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.974+0000 7f8262adc640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f825c071b90 0x7f825c1ef380 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f824c00c430 tx=0x7f824c00c8f0 comp rx=0 tx=0).stop 2023-10-30T13:47:44.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.974+0000 7f8262adc640 1 -- 172.21.15.46:0/2040254184 shutdown_connections 2023-10-30T13:47:44.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.974+0000 7f8262adc640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82400664a0 0x7f8240068960 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:44.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.974+0000 7f8262adc640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f825c0c6290 0x7f825c1ef8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:44.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.974+0000 7f8262adc640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f825c071b90 0x7f825c1ef380 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:44.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.974+0000 7f8262adc640 1 --2- 172.21.15.46:0/2040254184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f825c0717b0 0x7f825c156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:44.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.974+0000 7f8262adc640 1 -- 172.21.15.46:0/2040254184 >> 172.21.15.46:0/2040254184 conn(0x7f825c0c75d0 msgr2=0x7f825c06d330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:44.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.974+0000 7f8262adc640 1 -- 172.21.15.46:0/2040254184 shutdown_connections 2023-10-30T13:47:44.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:44.974+0000 7f8262adc640 1 -- 172.21.15.46:0/2040254184 wait complete. 2023-10-30T13:47:44.996 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:46.997 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.146+0000 7fea8bb72640 1 Processor -- start 2023-10-30T13:47:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.146+0000 7fea8bb72640 1 -- start start 2023-10-30T13:47:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.150+0000 7fea8bb72640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fea841580e0 0x7fea84070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:47.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.150+0000 7fea8bb72640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea840c5810 con 0x7fea840c6540 2023-10-30T13:47:47.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.150+0000 7fea8bb72640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea840c6990 con 0x7fea8415e700 2023-10-30T13:47:47.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.150+0000 7fea8bb72640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea8406acb0 con 0x7fea841580e0 2023-10-30T13:47:47.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.150+0000 7fea8910d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fea840c6540 0x7fea84069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45892/0 (socket says 172.21.15.46:45892) 2023-10-30T13:47:47.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.150+0000 7fea8910d640 1 -- 172.21.15.46:0/1490333336 learned_addr learned my addr 172.21.15.46:0/1490333336 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:47:47.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.150+0000 7fea8990e640 1 --2- 172.21.15.46:0/1490333336 >> v2:172.21.15.154:3300/0 conn(0x7fea841580e0 0x7fea84070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:47.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.150+0000 7fea8890c640 1 -- 172.21.15.46:0/1490333336 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815948314 0 0) 0x7fea840c5810 con 0x7fea840c6540 2023-10-30T13:47:47.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.150+0000 7fea8890c640 1 -- 172.21.15.46:0/1490333336 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea70003530 con 0x7fea840c6540 2023-10-30T13:47:47.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.150+0000 7fea8990e640 1 -- 172.21.15.46:0/1490333336 >> v1:172.21.15.154:6789/0 conn(0x7fea8415e700 legacy=0x7fea8406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:47.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.150+0000 7fea8990e640 1 -- 172.21.15.46:0/1490333336 >> v1:172.21.15.46:6789/0 conn(0x7fea840c6540 legacy=0x7fea84069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:47.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.150+0000 7fea8990e640 1 -- 172.21.15.46:0/1490333336 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8415eb90 con 0x7fea841580e0 2023-10-30T13:47:47.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8990e640 1 --2- 172.21.15.46:0/1490333336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea841580e0 0x7fea84070740 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fea800097c0 tx=0x7fea8001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c43935adff04e810 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:47.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8890c640 1 -- 172.21.15.46:0/1490333336 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea800ab680 con 0x7fea841580e0 2023-10-30T13:47:47.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8890c640 1 -- 172.21.15.46:0/1490333336 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea800ab7e0 con 0x7fea841580e0 2023-10-30T13:47:47.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8890c640 1 -- 172.21.15.46:0/1490333336 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea800abb70 con 0x7fea841580e0 2023-10-30T13:47:47.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8bb72640 1 -- 172.21.15.46:0/1490333336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea841580e0 msgr2=0x7fea84070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:47.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8bb72640 1 --2- 172.21.15.46:0/1490333336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea841580e0 0x7fea84070740 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fea800097c0 tx=0x7fea8001c480 comp rx=0 tx=0).stop 2023-10-30T13:47:47.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8bb72640 1 -- 172.21.15.46:0/1490333336 shutdown_connections 2023-10-30T13:47:47.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8bb72640 1 --2- 172.21.15.46:0/1490333336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea841580e0 0x7fea84070740 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:47.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8bb72640 1 -- 172.21.15.46:0/1490333336 >> 172.21.15.46:0/1490333336 conn(0x7fea840c7910 msgr2=0x7fea840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:47.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8bb72640 1 -- 172.21.15.46:0/1490333336 shutdown_connections 2023-10-30T13:47:47.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8bb72640 1 -- 172.21.15.46:0/1490333336 wait complete. 2023-10-30T13:47:47.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8bb72640 1 Processor -- start 2023-10-30T13:47:47.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.154+0000 7fea8bb72640 1 -- start start 2023-10-30T13:47:47.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8bb72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c6540 0x7fea84153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8bb72640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea841580e0 0x7fea841f6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8bb72640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea8415e700 0x7fea840c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8bb72640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea840c3af0 con 0x7fea840c6540 2023-10-30T13:47:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8bb72640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea841fa0a0 con 0x7fea841580e0 2023-10-30T13:47:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8bb72640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea840714b0 con 0x7fea8415e700 2023-10-30T13:47:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8990e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c6540 0x7fea84153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8990e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c6540 0x7fea84153cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50420/0 (socket says 172.21.15.46:50420) 2023-10-30T13:47:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8990e640 1 -- 172.21.15.46:0/4279176387 learned_addr learned my addr 172.21.15.46:0/4279176387 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8910d640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea841580e0 0x7fea841f6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8a10f640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea8415e700 0x7fea840c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8990e640 1 -- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea8415e700 msgr2=0x7fea840c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8990e640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea8415e700 0x7fea840c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8990e640 1 -- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea841580e0 msgr2=0x7fea841f6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8990e640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea841580e0 0x7fea841f6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8990e640 1 -- 172.21.15.46:0/4279176387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea80009420 con 0x7fea840c6540 2023-10-30T13:47:47.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8910d640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea841580e0 0x7fea841f6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:47:47.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8a10f640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea8415e700 0x7fea840c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:47:47.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8990e640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c6540 0x7fea84153cc0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fea80019090 tx=0x7fea8001acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:47.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea7a7fc640 1 -- 172.21.15.46:0/4279176387 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea800b5810 con 0x7fea840c6540 2023-10-30T13:47:47.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea7a7fc640 1 -- 172.21.15.46:0/4279176387 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea800b5990 con 0x7fea840c6540 2023-10-30T13:47:47.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea7a7fc640 1 -- 172.21.15.46:0/4279176387 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea800b5d40 con 0x7fea840c6540 2023-10-30T13:47:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8bb72640 1 -- 172.21.15.46:0/4279176387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea840c46c0 con 0x7fea840c6540 2023-10-30T13:47:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8bb72640 1 -- 172.21.15.46:0/4279176387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea841f9770 con 0x7fea840c6540 2023-10-30T13:47:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea8bb72640 1 -- 172.21.15.46:0/4279176387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea8404e170 con 0x7fea840c6540 2023-10-30T13:47:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.158+0000 7fea7a7fc640 1 -- 172.21.15.46:0/4279176387 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fea800aa460 con 0x7fea840c6540 2023-10-30T13:47:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.162+0000 7fea7a7fc640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea680664a0 0x7fea68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.162+0000 7fea7a7fc640 1 -- 172.21.15.46:0/4279176387 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7fea8011f1f0 con 0x7fea840c6540 2023-10-30T13:47:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.162+0000 7fea8910d640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea680664a0 0x7fea68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.162+0000 7fea8910d640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea680664a0 0x7fea68068960 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fea84155ee0 tx=0x7fea74009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.166+0000 7fea7a7fc640 1 -- 172.21.15.46:0/4279176387 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fea800ec1b0 con 0x7fea840c6540 2023-10-30T13:47:47.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 -- 172.21.15.46:0/4279176387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea84070200 con 0x7fea840c6540 2023-10-30T13:47:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea7a7fc640 1 -- 172.21.15.46:0/4279176387 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fea800ef7d0 con 0x7fea840c6540 2023-10-30T13:47:47.361 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:47.361 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:46.149322+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 114 2023-10-30T13:47:47.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 -- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea680664a0 msgr2=0x7fea68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:47.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea680664a0 0x7fea68068960 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fea84155ee0 tx=0x7fea74009040 comp rx=0 tx=0).stop 2023-10-30T13:47:47.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 -- 172.21.15.46:0/4279176387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c6540 msgr2=0x7fea84153cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:47.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c6540 0x7fea84153cc0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fea80019090 tx=0x7fea8001acc0 comp rx=0 tx=0).stop 2023-10-30T13:47:47.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 -- 172.21.15.46:0/4279176387 shutdown_connections 2023-10-30T13:47:47.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea680664a0 0x7fea68068960 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:47.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea8415e700 0x7fea840c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea841580e0 0x7fea841f6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 --2- 172.21.15.46:0/4279176387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea840c6540 0x7fea84153cc0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 -- 172.21.15.46:0/4279176387 >> 172.21.15.46:0/4279176387 conn(0x7fea840c7910 msgr2=0x7fea841569a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 -- 172.21.15.46:0/4279176387 shutdown_connections 2023-10-30T13:47:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:47.358+0000 7fea8bb72640 1 -- 172.21.15.46:0/4279176387 wait complete. 2023-10-30T13:47:47.382 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:47:49.384 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:49.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.534+0000 7fed41fc5640 1 Processor -- start 2023-10-30T13:47:49.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.534+0000 7fed41fc5640 1 -- start start 2023-10-30T13:47:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.534+0000 7fed41fc5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fed3c0c6b80 0x7fed3c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.534+0000 7fed41fc5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fed3c0c6f60 0x7fed3c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.534+0000 7fed41fc5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3c0c0a50 con 0x7fed3c0c8430 2023-10-30T13:47:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.534+0000 7fed41fc5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed3c06f6e0 con 0x7fed3c0c6f60 2023-10-30T13:47:49.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.534+0000 7fed41fc5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed3c06f9c0 con 0x7fed3c0c6b80 2023-10-30T13:47:49.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.534+0000 7fed3affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fed3c0c6f60 0x7fed3c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:49.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.534+0000 7fed3affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fed3c0c6f60 0x7fed3c0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50430/0 (socket says 172.21.15.46:50430) 2023-10-30T13:47:49.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.534+0000 7fed3affd640 1 -- 172.21.15.46:0/2018559854 learned_addr learned my addr 172.21.15.46:0/2018559854 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:49.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.534+0000 7fed3b7fe640 1 --2- 172.21.15.46:0/2018559854 >> v2:172.21.15.154:3300/0 conn(0x7fed3c0c6b80 0x7fed3c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:49.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed3affd640 1 -- 172.21.15.46:0/2018559854 >> v2:172.21.15.154:3300/0 conn(0x7fed3c0c6b80 msgr2=0x7fed3c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:49.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed3affd640 1 --2- 172.21.15.46:0/2018559854 >> v2:172.21.15.154:3300/0 conn(0x7fed3c0c6b80 0x7fed3c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed3affd640 1 -- 172.21.15.46:0/2018559854 >> v1:172.21.15.154:6789/0 conn(0x7fed3c0c8430 legacy=0x7fed3c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:49.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed3affd640 1 -- 172.21.15.46:0/2018559854 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c06fca0 con 0x7fed3c0c6f60 2023-10-30T13:47:49.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed3affd640 1 --2- 172.21.15.46:0/2018559854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c0c6f60 0x7fed3c0c0400 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fed3c06f1b0 tx=0x7fed3001eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f69de8c418e8f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:49.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed3a7fc640 1 -- 172.21.15.46:0/2018559854 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed300a7b70 con 0x7fed3c0c6f60 2023-10-30T13:47:49.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed3a7fc640 1 -- 172.21.15.46:0/2018559854 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed300a7cd0 con 0x7fed3c0c6f60 2023-10-30T13:47:49.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed3a7fc640 1 -- 172.21.15.46:0/2018559854 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed300af660 con 0x7fed3c0c6f60 2023-10-30T13:47:49.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed41fc5640 1 -- 172.21.15.46:0/2018559854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c0c6f60 msgr2=0x7fed3c0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:49.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed41fc5640 1 --2- 172.21.15.46:0/2018559854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c0c6f60 0x7fed3c0c0400 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fed3c06f1b0 tx=0x7fed3001eb90 comp rx=0 tx=0).stop 2023-10-30T13:47:49.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed41fc5640 1 -- 172.21.15.46:0/2018559854 shutdown_connections 2023-10-30T13:47:49.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed41fc5640 1 --2- 172.21.15.46:0/2018559854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c0c6f60 0x7fed3c0c0400 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed41fc5640 1 --2- 172.21.15.46:0/2018559854 >> v2:172.21.15.154:3300/0 conn(0x7fed3c0c6b80 0x7fed3c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed41fc5640 1 -- 172.21.15.46:0/2018559854 >> 172.21.15.46:0/2018559854 conn(0x7fed3c0c7910 msgr2=0x7fed3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.538+0000 7fed41fc5640 1 -- 172.21.15.46:0/2018559854 shutdown_connections 2023-10-30T13:47:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed41fc5640 1 -- 172.21.15.46:0/2018559854 wait complete. 2023-10-30T13:47:49.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed41fc5640 1 Processor -- start 2023-10-30T13:47:49.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed41fc5640 1 -- start start 2023-10-30T13:47:49.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed41fc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c0c6b80 0x7fed3c134d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:49.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed41fc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed3c0c6f60 0x7fed3c136150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:49.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed41fc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed3c0c8430 0x7fed3c1500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:49.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed41fc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c13a860 con 0x7fed3c0c6b80 2023-10-30T13:47:49.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed41fc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c13ace0 con 0x7fed3c0c8430 2023-10-30T13:47:49.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed41fc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c13cec0 con 0x7fed3c0c6f60 2023-10-30T13:47:49.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c0c6b80 0x7fed3c134d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:49.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c0c6b80 0x7fed3c134d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50440/0 (socket says 172.21.15.46:50440) 2023-10-30T13:47:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3b7fe640 1 -- 172.21.15.46:0/3403360575 learned_addr learned my addr 172.21.15.46:0/3403360575 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3affd640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed3c0c6f60 0x7fed3c136150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3bfff640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed3c0c8430 0x7fed3c1500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3b7fe640 1 -- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed3c0c6f60 msgr2=0x7fed3c136150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3b7fe640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed3c0c6f60 0x7fed3c136150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3b7fe640 1 -- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed3c0c8430 msgr2=0x7fed3c1500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3b7fe640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed3c0c8430 0x7fed3c1500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3b7fe640 1 -- 172.21.15.46:0/3403360575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3000b770 con 0x7fed3c0c6b80 2023-10-30T13:47:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3bfff640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed3c0c8430 0x7fed3c1500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:47:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3affd640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed3c0c6f60 0x7fed3c136150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:47:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed3b7fe640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c0c6b80 0x7fed3c134d00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fed2c00d800 tx=0x7fed2c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed23fff640 1 -- 172.21.15.46:0/3403360575 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c00b6c0 con 0x7fed3c0c6b80 2023-10-30T13:47:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed23fff640 1 -- 172.21.15.46:0/3403360575 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2c00b820 con 0x7fed3c0c6b80 2023-10-30T13:47:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed23fff640 1 -- 172.21.15.46:0/3403360575 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c00bb90 con 0x7fed3c0c6b80 2023-10-30T13:47:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.542+0000 7fed41fc5640 1 -- 172.21.15.46:0/3403360575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3c13cc00 con 0x7fed3c0c6b80 2023-10-30T13:47:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.546+0000 7fed41fc5640 1 -- 172.21.15.46:0/3403360575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3c13a580 con 0x7fed3c0c6b80 2023-10-30T13:47:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.546+0000 7fed41fc5640 1 -- 172.21.15.46:0/3403360575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3c04e170 con 0x7fed3c0c6b80 2023-10-30T13:47:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.546+0000 7fed23fff640 1 -- 172.21.15.46:0/3403360575 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed2c00bcf0 con 0x7fed3c0c6b80 2023-10-30T13:47:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.546+0000 7fed23fff640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed280664a0 0x7fed28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.546+0000 7fed23fff640 1 -- 172.21.15.46:0/3403360575 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7fed2c0848c0 con 0x7fed3c0c6b80 2023-10-30T13:47:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.546+0000 7fed3affd640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed280664a0 0x7fed28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:49.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.550+0000 7fed3affd640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed280664a0 0x7fed28068960 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fed300b1000 tx=0x7fed30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.550+0000 7fed23fff640 1 -- 172.21.15.46:0/3403360575 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fed2c051890 con 0x7fed3c0c6b80 2023-10-30T13:47:49.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.746+0000 7fed41fc5640 1 -- 172.21.15.46:0/3403360575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed3c13aab0 con 0x7fed3c0c6b80 2023-10-30T13:47:49.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.746+0000 7fed23fff640 1 -- 172.21.15.46:0/3403360575 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fed2c011340 con 0x7fed3c0c6b80 2023-10-30T13:47:49.751 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:46.149322+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:49.751 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 114 2023-10-30T13:47:49.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.750+0000 7fed41fc5640 1 -- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed280664a0 msgr2=0x7fed28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:49.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.750+0000 7fed41fc5640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed280664a0 0x7fed28068960 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fed300b1000 tx=0x7fed30008040 comp rx=0 tx=0).stop 2023-10-30T13:47:49.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.750+0000 7fed41fc5640 1 -- 172.21.15.46:0/3403360575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c0c6b80 msgr2=0x7fed3c134d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:49.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.750+0000 7fed41fc5640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c0c6b80 0x7fed3c134d00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fed2c00d800 tx=0x7fed2c00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.750+0000 7fed41fc5640 1 -- 172.21.15.46:0/3403360575 shutdown_connections 2023-10-30T13:47:49.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.750+0000 7fed41fc5640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed280664a0 0x7fed28068960 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.750+0000 7fed41fc5640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed3c0c8430 0x7fed3c1500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.750+0000 7fed41fc5640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed3c0c6f60 0x7fed3c136150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.750+0000 7fed41fc5640 1 --2- 172.21.15.46:0/3403360575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c0c6b80 0x7fed3c134d00 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.750+0000 7fed41fc5640 1 -- 172.21.15.46:0/3403360575 >> 172.21.15.46:0/3403360575 conn(0x7fed3c0c7910 msgr2=0x7fed3c06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.750+0000 7fed41fc5640 1 -- 172.21.15.46:0/3403360575 shutdown_connections 2023-10-30T13:47:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:49.750+0000 7fed41fc5640 1 -- 172.21.15.46:0/3403360575 wait complete. 2023-10-30T13:47:49.772 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 114, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 114, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6366": { "addr": "172.21.15.46:6833/190134548", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 190134548, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 190134548, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6366, "incarnation": 104, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 51 }, "gid_6702": { "addr": "172.21.15.46:6836/2197729559", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 2197729559, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 2197729559, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6702, "incarnation": 111, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 69, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:47:46.149322+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6702, "mds_1": 6366 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:47:50.304 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:47:50.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:47:51.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:51.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.918+0000 7f15e3bc5640 1 Processor -- start 2023-10-30T13:47:51.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.918+0000 7f15e3bc5640 1 -- start start 2023-10-30T13:47:51.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e3bc5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f15dc1580e0 0x7f15dc070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:51.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e3bc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15dc0c6540 0x7f15dc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:51.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e3bc5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15dc0c5810 con 0x7f15dc069e90 2023-10-30T13:47:51.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e3bc5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15dc0c6960 con 0x7f15dc1580e0 2023-10-30T13:47:51.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e3bc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc0c6bb0 con 0x7f15dc0c6540 2023-10-30T13:47:51.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e2162640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f15dc069e90 0x7f15dc06a320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45908/0 (socket says 172.21.15.46:45908) 2023-10-30T13:47:51.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e2162640 1 -- 172.21.15.46:0/4016475766 learned_addr learned my addr 172.21.15.46:0/4016475766 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:47:51.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e1160640 1 --2- 172.21.15.46:0/4016475766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15dc0c6540 0x7f15dc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:51.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e1961640 1 --2- 172.21.15.46:0/4016475766 >> v2:172.21.15.154:3300/0 conn(0x7f15dc1580e0 0x7f15dc070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:51.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e095f640 1 -- 172.21.15.46:0/4016475766 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329647239 0 0) 0x7f15dc0c5810 con 0x7f15dc069e90 2023-10-30T13:47:51.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e095f640 1 -- 172.21.15.46:0/4016475766 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c8003530 con 0x7f15dc069e90 2023-10-30T13:47:51.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e095f640 1 -- 172.21.15.46:0/4016475766 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2466685322 0 0) 0x7f15c8003530 con 0x7f15dc069e90 2023-10-30T13:47:51.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e1160640 1 -- 172.21.15.46:0/4016475766 >> v2:172.21.15.154:3300/0 conn(0x7f15dc1580e0 msgr2=0x7f15dc070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:51.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e1160640 1 --2- 172.21.15.46:0/4016475766 >> v2:172.21.15.154:3300/0 conn(0x7f15dc1580e0 0x7f15dc070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:51.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e1160640 1 -- 172.21.15.46:0/4016475766 >> v1:172.21.15.46:6789/0 conn(0x7f15dc069e90 legacy=0x7f15dc06a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:51.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e1160640 1 -- 172.21.15.46:0/4016475766 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc0c6e70 con 0x7f15dc0c6540 2023-10-30T13:47:51.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.922+0000 7f15e1961640 1 --2- 172.21.15.46:0/4016475766 >> v2:172.21.15.154:3300/0 conn(0x7f15dc1580e0 0x7f15dc070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:47:51.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e1160640 1 --2- 172.21.15.46:0/4016475766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15dc0c6540 0x7f15dc15e700 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f15cc0097c0 tx=0x7f15cc01c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bd568001fe96462 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:51.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e095f640 1 -- 172.21.15.46:0/4016475766 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4001a00 con 0x7f15dc0c6540 2023-10-30T13:47:51.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e095f640 1 -- 172.21.15.46:0/4016475766 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15cc0a7050 con 0x7f15dc0c6540 2023-10-30T13:47:51.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e095f640 1 -- 172.21.15.46:0/4016475766 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc0ab600 con 0x7f15dc0c6540 2023-10-30T13:47:51.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e3bc5640 1 -- 172.21.15.46:0/4016475766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15dc0c6540 msgr2=0x7f15dc15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:51.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e3bc5640 1 --2- 172.21.15.46:0/4016475766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15dc0c6540 0x7f15dc15e700 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f15cc0097c0 tx=0x7f15cc01c3a0 comp rx=0 tx=0).stop 2023-10-30T13:47:51.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e3bc5640 1 -- 172.21.15.46:0/4016475766 shutdown_connections 2023-10-30T13:47:51.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e3bc5640 1 --2- 172.21.15.46:0/4016475766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15dc0c6540 0x7f15dc15e700 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:51.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e3bc5640 1 --2- 172.21.15.46:0/4016475766 >> v2:172.21.15.154:3300/0 conn(0x7f15dc1580e0 0x7f15dc070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:51.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e3bc5640 1 -- 172.21.15.46:0/4016475766 >> 172.21.15.46:0/4016475766 conn(0x7f15dc0c7910 msgr2=0x7f15dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:51.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e3bc5640 1 -- 172.21.15.46:0/4016475766 shutdown_connections 2023-10-30T13:47:51.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e3bc5640 1 -- 172.21.15.46:0/4016475766 wait complete. 2023-10-30T13:47:51.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e3bc5640 1 Processor -- start 2023-10-30T13:47:51.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.926+0000 7f15e3bc5640 1 -- start start 2023-10-30T13:47:51.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e3bc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15dc069e90 0x7f15dc1493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:51.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e3bc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15dc0c6540 0x7f15dc1f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:51.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e3bc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15dc1580e0 0x7f15dc15fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:51.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e3bc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc1f5ea0 con 0x7f15dc1580e0 2023-10-30T13:47:51.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e3bc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc0724d0 con 0x7f15dc0c6540 2023-10-30T13:47:51.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e3bc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc1f6df0 con 0x7f15dc069e90 2023-10-30T13:47:51.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e2162640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15dc1580e0 0x7f15dc15fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:51.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e2162640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15dc1580e0 0x7f15dc15fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50456/0 (socket says 172.21.15.46:50456) 2023-10-30T13:47:51.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e2162640 1 -- 172.21.15.46:0/503695722 learned_addr learned my addr 172.21.15.46:0/503695722 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:51.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e1160640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15dc0c6540 0x7f15dc1f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:51.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e1961640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15dc069e90 0x7f15dc1493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:51.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e2162640 1 -- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15dc069e90 msgr2=0x7f15dc1493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e2162640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15dc069e90 0x7f15dc1493d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e2162640 1 -- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15dc0c6540 msgr2=0x7f15dc1f68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e2162640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15dc0c6540 0x7f15dc1f68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e2162640 1 -- 172.21.15.46:0/503695722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d8009420 con 0x7f15dc1580e0 2023-10-30T13:47:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e1961640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15dc069e90 0x7f15dc1493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e2162640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15dc1580e0 0x7f15dc15fae0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f15d4001840 tx=0x7f15d400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:51.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15d27fc640 1 -- 172.21.15.46:0/503695722 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4012400 con 0x7f15dc1580e0 2023-10-30T13:47:51.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15d27fc640 1 -- 172.21.15.46:0/503695722 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d4012560 con 0x7f15dc1580e0 2023-10-30T13:47:51.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15d27fc640 1 -- 172.21.15.46:0/503695722 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4012910 con 0x7f15dc1580e0 2023-10-30T13:47:51.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e3bc5640 1 -- 172.21.15.46:0/503695722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15cc009420 con 0x7f15dc1580e0 2023-10-30T13:47:51.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e3bc5640 1 -- 172.21.15.46:0/503695722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15dc071f10 con 0x7f15dc1580e0 2023-10-30T13:47:51.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.930+0000 7f15e3bc5640 1 -- 172.21.15.46:0/503695722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15dc04e170 con 0x7f15dc1580e0 2023-10-30T13:47:51.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.934+0000 7f15d27fc640 1 -- 172.21.15.46:0/503695722 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15d4012a70 con 0x7f15dc1580e0 2023-10-30T13:47:51.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.934+0000 7f15d27fc640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f15c00664d0 0x7f15c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:51.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.934+0000 7f15d27fc640 1 -- 172.21.15.46:0/503695722 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f15d4084b70 con 0x7f15dc1580e0 2023-10-30T13:47:51.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.934+0000 7f15e1961640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f15c00664d0 0x7f15c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:51.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.934+0000 7f15e1961640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f15c00664d0 0x7f15c0068990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009ca0 tx=0x7f15d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:51.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:51.938+0000 7f15d27fc640 1 -- 172.21.15.46:0/503695722 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15d4051ab0 con 0x7f15dc1580e0 2023-10-30T13:47:52.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.130+0000 7f15e3bc5640 1 -- 172.21.15.46:0/503695722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15dc15f8c0 con 0x7f15dc1580e0 2023-10-30T13:47:52.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.130+0000 7f15d27fc640 1 -- 172.21.15.46:0/503695722 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f15d40550d0 con 0x7f15dc1580e0 2023-10-30T13:47:52.134 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:52.135 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:51.253577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:52.135 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 115 2023-10-30T13:47:52.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.134+0000 7f15e3bc5640 1 -- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f15c00664d0 msgr2=0x7f15c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:52.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.134+0000 7f15e3bc5640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f15c00664d0 0x7f15c0068990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009ca0 tx=0x7f15d8008040 comp rx=0 tx=0).stop 2023-10-30T13:47:52.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.134+0000 7f15e3bc5640 1 -- 172.21.15.46:0/503695722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15dc1580e0 msgr2=0x7f15dc15fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:52.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.134+0000 7f15e3bc5640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15dc1580e0 0x7f15dc15fae0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f15d4001840 tx=0x7f15d400a430 comp rx=0 tx=0).stop 2023-10-30T13:47:52.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.134+0000 7f15e3bc5640 1 -- 172.21.15.46:0/503695722 shutdown_connections 2023-10-30T13:47:52.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.134+0000 7f15e3bc5640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f15c00664d0 0x7f15c0068990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:52.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.134+0000 7f15e3bc5640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15dc1580e0 0x7f15dc15fae0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:52.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.134+0000 7f15e3bc5640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15dc0c6540 0x7f15dc1f68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:52.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.134+0000 7f15e3bc5640 1 --2- 172.21.15.46:0/503695722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15dc069e90 0x7f15dc1493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:52.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.134+0000 7f15e3bc5640 1 -- 172.21.15.46:0/503695722 >> 172.21.15.46:0/503695722 conn(0x7f15dc0c7910 msgr2=0x7f15dc156510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:52.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.134+0000 7f15e3bc5640 1 -- 172.21.15.46:0/503695722 shutdown_connections 2023-10-30T13:47:52.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:52.134+0000 7f15e3bc5640 1 -- 172.21.15.46:0/503695722 wait complete. 2023-10-30T13:47:52.155 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=6201 2023-10-30T13:47:52.156 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.e 2023-10-30T13:47:57.575 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 20% (108/537) Updating files: 21% (113/537) Updating files: 22% (119/537) Updating files: 23% (124/537) Updating files: 24% (129/537) Updating files: 25% (135/537) Updating files: 26% (140/537) Updating files: 27% (145/537) Updating files: 28% (151/537) Updating files: 29% (156/537) Updating files: 30% (162/537) Updating files: 31% (167/537) Updating files: 32% (172/537) Updating files: 33% (178/537) Updating files: 34% (183/537) Updating files: 35% (188/537) Updating files: 36% (194/537) Updating files: 37% (199/537) Updating files: 38% (205/537) Updating files: 39% (210/537) Updating files: 40% (215/537) Updating files: 41% (221/537) Updating files: 42% (226/537) Updating files: 43% (231/537) Updating files: 44% (237/537) Updating files: 45% (242/537) Updating files: 45% (247/537) Updating files: 46% (248/537) Updating files: 47% (253/537) Updating files: 47% (256/537) Updating files: 48% (258/537) Updating files: 49% (264/537) Updating files: 50% (269/537) Updating files: 50% (273/537) Updating files: 51% (274/537) Updating files: 52% (280/537) Updating files: 53% (285/537) Updating files: 54% (290/537) Updating files: 55% (296/537) Updating files: 56% (301/537) Updating files: 57% (307/537) Updating files: 58% (312/537) Updating files: 59% (317/537) Updating files: 60% (323/537) Updating files: 61% (328/537) Updating files: 62% (333/537) Updating files: 63% (339/537) Updating files: 64% (344/537) Updating files: 65% (350/537) Updating files: 66% (355/537) Updating files: 67% (360/537) Updating files: 68% (366/537) Updating files: 69% (371/537) Updating files: 70% (376/537) Updating files: 71% (382/537) Updating files: 72% (387/537) Updating files: 73% (393/537) Updating files: 74% (398/537) Updating files: 75% (403/537) Updating files: 76% (409/537) Updating files: 77% (414/537) Updating files: 78% (419/537) Updating files: 78% (423/537) Updating files: 79% (425/537) Updating files: 80% (430/537) Updating files: 81% (435/537) Updating files: 82% (441/537) Updating files: 83% (446/537) Updating files: 83% (449/537) Updating files: 84% (452/537) Updating files: 85% (457/537) Updating files: 86% (462/537) Updating files: 87% (468/537) Updating files: 88% (473/537) Updating files: 89% (478/537) Updating files: 90% (484/537) Updating files: 91% (489/537) Updating files: 91% (491/537) Updating files: 92% (495/537) Updating files: 93% (500/537) Updating files: 94% (505/537) Updating files: 95% (511/537) Updating files: 96% (516/537) Updating files: 97% (521/537) Updating files: 98% (527/537) Updating files: 99% (532/537) Updating files: 100% (537/537) Updating files: 100% (537/537), done. 2023-10-30T13:47:57.644 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 090436f56ac v0.21 2023-10-30T13:47:57.650 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.21 2023-10-30T13:47:57.694 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 22 -eq 48 ] 2023-10-30T13:47:57.694 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.22 2023-10-30T13:47:57.694 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.22 2023-10-30T13:47:57.694 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.22 2023-10-30T13:47:57.695 INFO:tasks.workunit.client.0.smithi046.stdout:v0.22 2023-10-30T13:47:58.157 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T13:47:58.157 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:47:58.157 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:47:58.161 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:47:58.162 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:58.259 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T13:47:58.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.310+0000 7f745fec9640 1 Processor -- start 2023-10-30T13:47:58.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.310+0000 7f745fec9640 1 -- start start 2023-10-30T13:47:58.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.314+0000 7f745fec9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f745815a410 0x7f745806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.314+0000 7f745fec9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74580c6c10 con 0x7f7458069e90 2023-10-30T13:47:58.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.314+0000 7f745fec9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74580c6ef0 con 0x7f745806cb80 2023-10-30T13:47:58.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.314+0000 7f745fec9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74580c8570 con 0x7f745815a410 2023-10-30T13:47:58.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.314+0000 7f745dc65640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7458069e90 0x7f74580c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57530/0 (socket says 172.21.15.46:57530) 2023-10-30T13:47:58.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.314+0000 7f745dc65640 1 -- 172.21.15.46:0/737821000 learned_addr learned my addr 172.21.15.46:0/737821000 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:47:58.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.314+0000 7f745d464640 1 --2- 172.21.15.46:0/737821000 >> v2:172.21.15.154:3300/0 conn(0x7f745815a410 0x7f745806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.314+0000 7f745cc63640 1 -- 172.21.15.46:0/737821000 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549744031 0 0) 0x7f74580c6c10 con 0x7f7458069e90 2023-10-30T13:47:58.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.314+0000 7f745cc63640 1 -- 172.21.15.46:0/737821000 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7444003530 con 0x7f7458069e90 2023-10-30T13:47:58.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.314+0000 7f745d464640 1 -- 172.21.15.46:0/737821000 >> v1:172.21.15.154:6789/0 conn(0x7f745806cb80 legacy=0x7f745806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.314+0000 7f745d464640 1 -- 172.21.15.46:0/737821000 >> v1:172.21.15.46:6789/0 conn(0x7f7458069e90 legacy=0x7f74580c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.314+0000 7f745d464640 1 -- 172.21.15.46:0/737821000 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74580c71d0 con 0x7f745815a410 2023-10-30T13:47:58.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.318+0000 7f745d464640 1 --2- 172.21.15.46:0/737821000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f745815a410 0x7f745806a270 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f74480097c0 tx=0x7f744801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e669c6193af5ccb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:58.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.326+0000 7f745cc63640 1 -- 172.21.15.46:0/737821000 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480ab600 con 0x7f745815a410 2023-10-30T13:47:58.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.326+0000 7f745cc63640 1 -- 172.21.15.46:0/737821000 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74480a7050 con 0x7f745815a410 2023-10-30T13:47:58.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.326+0000 7f745cc63640 1 -- 172.21.15.46:0/737821000 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480ab960 con 0x7f745815a410 2023-10-30T13:47:58.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.326+0000 7f745fec9640 1 -- 172.21.15.46:0/737821000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f745815a410 msgr2=0x7f745806a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.326+0000 7f745fec9640 1 --2- 172.21.15.46:0/737821000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f745815a410 0x7f745806a270 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f74480097c0 tx=0x7f744801c480 comp rx=0 tx=0).stop 2023-10-30T13:47:58.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.326+0000 7f745fec9640 1 -- 172.21.15.46:0/737821000 shutdown_connections 2023-10-30T13:47:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.326+0000 7f745fec9640 1 --2- 172.21.15.46:0/737821000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f745815a410 0x7f745806a270 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.326+0000 7f745fec9640 1 -- 172.21.15.46:0/737821000 >> 172.21.15.46:0/737821000 conn(0x7f745806dfd0 msgr2=0x7f745806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.326+0000 7f745fec9640 1 -- 172.21.15.46:0/737821000 shutdown_connections 2023-10-30T13:47:58.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.326+0000 7f745fec9640 1 -- 172.21.15.46:0/737821000 wait complete. 2023-10-30T13:47:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745fec9640 1 Processor -- start 2023-10-30T13:47:58.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745fec9640 1 -- start start 2023-10-30T13:47:58.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745fec9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7458069e90 0x7f74581458f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745fec9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f745806cb80 0x7f7458145e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745fec9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f745815a410 0x7f7458149bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745fec9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74580c4220 con 0x7f745806cb80 2023-10-30T13:47:58.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745fec9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7458159930 con 0x7f7458069e90 2023-10-30T13:47:58.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745fec9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74580c3600 con 0x7f745815a410 2023-10-30T13:47:58.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745d464640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f745806cb80 0x7f7458145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745d464640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f745806cb80 0x7f7458145e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40102/0 (socket says 172.21.15.46:40102) 2023-10-30T13:47:58.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745d464640 1 -- 172.21.15.46:0/260626654 learned_addr learned my addr 172.21.15.46:0/260626654 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:47:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745dc65640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7458069e90 0x7f74581458f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745e466640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f745815a410 0x7f7458149bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745d464640 1 -- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f745815a410 msgr2=0x7f7458149bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745d464640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f745815a410 0x7f7458149bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745d464640 1 -- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7458069e90 msgr2=0x7f74581458f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745d464640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7458069e90 0x7f74581458f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745d464640 1 -- 172.21.15.46:0/260626654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7448009420 con 0x7f745806cb80 2023-10-30T13:47:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745e466640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f745815a410 0x7f7458149bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745d464640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f745806cb80 0x7f7458145e30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f7448019090 tx=0x7f7448019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f744e7fc640 1 -- 172.21.15.46:0/260626654 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7448003af0 con 0x7f745806cb80 2023-10-30T13:47:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f744e7fc640 1 -- 172.21.15.46:0/260626654 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7448003c50 con 0x7f745806cb80 2023-10-30T13:47:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f744e7fc640 1 -- 172.21.15.46:0/260626654 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480b4400 con 0x7f745806cb80 2023-10-30T13:47:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745fec9640 1 -- 172.21.15.46:0/260626654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74580c4810 con 0x7f745806cb80 2023-10-30T13:47:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.330+0000 7f745fec9640 1 -- 172.21.15.46:0/260626654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74580c3320 con 0x7f745806cb80 2023-10-30T13:47:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.334+0000 7f745fec9640 1 -- 172.21.15.46:0/260626654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f745804e170 con 0x7f745806cb80 2023-10-30T13:47:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.334+0000 7f744e7fc640 1 -- 172.21.15.46:0/260626654 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f74480b2040 con 0x7f745806cb80 2023-10-30T13:47:58.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.334+0000 7f744e7fc640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f743c0664a0 0x7f743c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.334+0000 7f744e7fc640 1 -- 172.21.15.46:0/260626654 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f744811e320 con 0x7f745806cb80 2023-10-30T13:47:58.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.334+0000 7f745dc65640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f743c0664a0 0x7f743c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.334+0000 7f745dc65640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f743c0664a0 0x7f743c068960 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f74540011f0 tx=0x7f7454007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:58.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.338+0000 7f744e7fc640 1 -- 172.21.15.46:0/260626654 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f74480eb2f0 con 0x7f745806cb80 2023-10-30T13:47:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.554+0000 7f745fec9640 1 -- 172.21.15.46:0/260626654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74580c4810 con 0x7f745806cb80 2023-10-30T13:47:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.554+0000 7f744e7fc640 1 -- 172.21.15.46:0/260626654 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f74580c4810 con 0x7f745806cb80 2023-10-30T13:47:58.559 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:47:58.559 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6858,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:51.253577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 116 2023-10-30T13:47:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.558+0000 7f745fec9640 1 -- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f743c0664a0 msgr2=0x7f743c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.558+0000 7f745fec9640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f743c0664a0 0x7f743c068960 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f74540011f0 tx=0x7f7454007d40 comp rx=0 tx=0).stop 2023-10-30T13:47:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.558+0000 7f745fec9640 1 -- 172.21.15.46:0/260626654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f745806cb80 msgr2=0x7f7458145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.558+0000 7f745fec9640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f745806cb80 0x7f7458145e30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f7448019090 tx=0x7f7448019220 comp rx=0 tx=0).stop 2023-10-30T13:47:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.558+0000 7f745fec9640 1 -- 172.21.15.46:0/260626654 shutdown_connections 2023-10-30T13:47:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.558+0000 7f745fec9640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f743c0664a0 0x7f743c068960 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.558+0000 7f745fec9640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f745815a410 0x7f7458149bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.558+0000 7f745fec9640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f745806cb80 0x7f7458145e30 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.558+0000 7f745fec9640 1 --2- 172.21.15.46:0/260626654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7458069e90 0x7f74581458f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.558+0000 7f745fec9640 1 -- 172.21.15.46:0/260626654 >> 172.21.15.46:0/260626654 conn(0x7f745806dfd0 msgr2=0x7f745806afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.558+0000 7f745fec9640 1 -- 172.21.15.46:0/260626654 shutdown_connections 2023-10-30T13:47:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:47:58.558+0000 7f745fec9640 1 -- 172.21.15.46:0/260626654 wait complete. 2023-10-30T13:47:58.580 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T13:47:58.580 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 110 secs before thrashing 2023-10-30T13:48:00.104 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T13:48:00.102+0000 7f326d637640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:48:00.105 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T13:48:00.102+0000 7f326d637640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:48:04.003 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 21% (43/204) Updating files: 22% (45/204) Updating files: 23% (47/204) Updating files: 24% (49/204) Updating files: 25% (51/204) Updating files: 26% (54/204) Updating files: 27% (56/204) Updating files: 28% (58/204) Updating files: 29% (60/204) Updating files: 30% (62/204) Updating files: 31% (64/204) Updating files: 32% (66/204) Updating files: 33% (68/204) Updating files: 34% (70/204) Updating files: 35% (72/204) Updating files: 36% (74/204) Updating files: 37% (76/204) Updating files: 38% (78/204) Updating files: 39% (80/204) Updating files: 40% (82/204) Updating files: 41% (84/204) Updating files: 42% (86/204) Updating files: 43% (88/204) Updating files: 44% (90/204) Updating files: 45% (92/204) Updating files: 46% (94/204) Updating files: 47% (96/204) Updating files: 48% (98/204) Updating files: 49% (100/204) Updating files: 50% (102/204) Updating files: 50% (104/204) Updating files: 51% (105/204) Updating files: 52% (107/204) Updating files: 53% (109/204) Updating files: 54% (111/204) Updating files: 55% (113/204) Updating files: 56% (115/204) Updating files: 57% (117/204) Updating files: 58% (119/204) Updating files: 59% (121/204) Updating files: 60% (123/204) Updating files: 61% (125/204) Updating files: 62% (127/204) Updating files: 63% (129/204) Updating files: 64% (131/204) Updating files: 65% (133/204) Updating files: 66% (135/204) Updating files: 67% (137/204) Updating files: 68% (139/204) Updating files: 69% (141/204) Updating files: 70% (143/204) Updating files: 71% (145/204) Updating files: 72% (147/204) Updating files: 73% (149/204) Updating files: 74% (151/204) Updating files: 75% (153/204) Updating files: 76% (156/204) Updating files: 77% (158/204) Updating files: 78% (160/204) Updating files: 79% (162/204) Updating files: 80% (164/204) Updating files: 81% (166/204) Updating files: 82% (168/204) Updating files: 83% (170/204) Updating files: 84% (172/204) Updating files: 85% (174/204) Updating files: 86% (176/204) Updating files: 87% (178/204) Updating files: 88% (180/204) Updating files: 89% (182/204) Updating files: 90% (184/204) Updating files: 91% (186/204) Updating files: 92% (188/204) Updating files: 93% (190/204) Updating files: 94% (192/204) Updating files: 95% (194/204) Updating files: 96% (196/204) Updating files: 97% (198/204) Updating files: 98% (200/204) Updating files: 99% (202/204) Updating files: 100% (204/204) Updating files: 100% (204/204), done. 2023-10-30T13:48:08.600 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 8a7c95f60ad v0.22 2023-10-30T13:48:08.605 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.22 2023-10-30T13:48:08.996 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 23 -eq 48 ] 2023-10-30T13:48:08.996 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.23 2023-10-30T13:48:08.996 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.23 2023-10-30T13:48:08.997 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.23 2023-10-30T13:48:08.997 INFO:tasks.workunit.client.0.smithi046.stdout:v0.23 2023-10-30T13:48:13.777 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 12% (20/163) Updating files: 13% (22/163) Updating files: 14% (23/163) Updating files: 15% (25/163) Updating files: 16% (27/163) Updating files: 17% (28/163) Updating files: 18% (30/163) Updating files: 19% (31/163) Updating files: 20% (33/163) Updating files: 21% (35/163) Updating files: 22% (36/163) Updating files: 23% (38/163) Updating files: 24% (40/163) Updating files: 25% (41/163) Updating files: 26% (43/163) Updating files: 27% (45/163) Updating files: 28% (46/163) Updating files: 29% (48/163) Updating files: 30% (49/163) Updating files: 31% (51/163) Updating files: 32% (53/163) Updating files: 33% (54/163) Updating files: 34% (56/163) Updating files: 35% (58/163) Updating files: 36% (59/163) Updating files: 37% (61/163) Updating files: 38% (62/163) Updating files: 39% (64/163) Updating files: 40% (66/163) Updating files: 41% (67/163) Updating files: 42% (69/163) Updating files: 43% (71/163) Updating files: 44% (72/163) Updating files: 45% (74/163) Updating files: 46% (75/163) Updating files: 47% (77/163) Updating files: 48% (79/163) Updating files: 49% (80/163) Updating files: 50% (82/163) Updating files: 51% (84/163) Updating files: 52% (85/163) Updating files: 53% (87/163) Updating files: 54% (89/163) Updating files: 55% (90/163) Updating files: 56% (92/163) Updating files: 57% (93/163) Updating files: 58% (95/163) Updating files: 59% (97/163) Updating files: 60% (98/163) Updating files: 61% (100/163) Updating files: 62% (102/163) Updating files: 63% (103/163) Updating files: 64% (105/163) Updating files: 65% (106/163) Updating files: 66% (108/163) Updating files: 67% (110/163) Updating files: 68% (111/163) Updating files: 69% (113/163) Updating files: 70% (115/163) Updating files: 71% (116/163) Updating files: 72% (118/163) Updating files: 73% (119/163) Updating files: 74% (121/163) Updating files: 75% (123/163) Updating files: 76% (124/163) Updating files: 77% (126/163) Updating files: 78% (128/163) Updating files: 79% (129/163) Updating files: 80% (131/163) Updating files: 81% (133/163) Updating files: 82% (134/163) Updating files: 83% (136/163) Updating files: 84% (137/163) Updating files: 85% (139/163) Updating files: 86% (141/163) Updating files: 87% (142/163) Updating files: 88% (144/163) Updating files: 89% (146/163) Updating files: 90% (147/163) Updating files: 91% (149/163) Updating files: 92% (150/163) Updating files: 93% (152/163) Updating files: 94% (154/163) Updating files: 95% (155/163) Updating files: 96% (157/163) Updating files: 97% (159/163) Updating files: 98% (160/163) Updating files: 99% (162/163) Updating files: 100% (163/163) Updating files: 100% (163/163), done. 2023-10-30T13:48:14.064 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 5d1d8d0c460 v0.23 2023-10-30T13:48:14.068 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.23 2023-10-30T13:48:14.353 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 24 -eq 48 ] 2023-10-30T13:48:14.354 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.24 2023-10-30T13:48:14.354 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.24 2023-10-30T13:48:14.354 INFO:tasks.workunit.client.0.smithi046.stdout:v0.24 2023-10-30T13:48:14.355 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.24 2023-10-30T13:48:15.844 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 36% (67/185) Updating files: 37% (69/185) Updating files: 38% (71/185) Updating files: 39% (73/185) Updating files: 40% (74/185) Updating files: 41% (76/185) Updating files: 42% (78/185) Updating files: 43% (80/185) Updating files: 44% (82/185) Updating files: 45% (84/185) Updating files: 46% (86/185) Updating files: 47% (87/185) Updating files: 48% (89/185) Updating files: 49% (91/185) Updating files: 50% (93/185) Updating files: 51% (95/185) Updating files: 52% (97/185) Updating files: 53% (99/185) Updating files: 54% (100/185) Updating files: 55% (102/185) Updating files: 56% (104/185) Updating files: 57% (106/185) Updating files: 58% (108/185) Updating files: 59% (110/185) Updating files: 60% (111/185) Updating files: 61% (113/185) Updating files: 62% (115/185) Updating files: 63% (117/185) Updating files: 64% (119/185) Updating files: 65% (121/185) Updating files: 66% (123/185) Updating files: 67% (124/185) Updating files: 68% (126/185) Updating files: 69% (128/185) Updating files: 70% (130/185) Updating files: 71% (132/185) Updating files: 72% (134/185) Updating files: 73% (136/185) Updating files: 74% (137/185) Updating files: 75% (139/185) Updating files: 76% (141/185) Updating files: 77% (143/185) Updating files: 78% (145/185) Updating files: 79% (147/185) Updating files: 80% (148/185) Updating files: 81% (150/185) Updating files: 82% (152/185) Updating files: 83% (154/185) Updating files: 84% (156/185) Updating files: 85% (158/185) Updating files: 86% (160/185) Updating files: 87% (161/185) Updating files: 88% (163/185) Updating files: 89% (165/185) Updating files: 90% (167/185) Updating files: 91% (169/185) Updating files: 92% (171/185) Updating files: 93% (173/185) Updating files: 94% (174/185) Updating files: 95% (176/185) Updating files: 96% (178/185) Updating files: 97% (180/185) Updating files: 98% (182/185) Updating files: 99% (184/185) Updating files: 100% (185/185) Updating files: 100% (185/185), done. 2023-10-30T13:48:15.878 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 180a4176035 v0.24 2023-10-30T13:48:15.883 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.24 2023-10-30T13:48:15.989 INFO:tasks.workunit.client.0.smithi046.stdout:v0.25 2023-10-30T13:48:15.989 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 25 -eq 48 ] 2023-10-30T13:48:15.989 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.25 2023-10-30T13:48:15.990 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.25 2023-10-30T13:48:15.990 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.25 2023-10-30T13:48:20.360 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:20.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:31.865 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 15% (98/622) Updating files: 16% (100/622) Updating files: 17% (106/622) Updating files: 18% (112/622) Updating files: 19% (119/622) Updating files: 20% (125/622) Updating files: 21% (131/622) Updating files: 22% (137/622) Updating files: 23% (144/622) Updating files: 24% (150/622) Updating files: 25% (156/622) Updating files: 26% (162/622) Updating files: 27% (168/622) Updating files: 28% (175/622) Updating files: 29% (181/622) Updating files: 30% (187/622) Updating files: 31% (193/622) Updating files: 32% (200/622) Updating files: 33% (206/622) Updating files: 34% (212/622) Updating files: 35% (218/622) Updating files: 36% (224/622) Updating files: 36% (230/622) Updating files: 37% (231/622) Updating files: 38% (237/622) Updating files: 39% (243/622) Updating files: 40% (249/622) Updating files: 41% (256/622) Updating files: 42% (262/622) Updating files: 43% (268/622) Updating files: 44% (274/622) Updating files: 45% (280/622) Updating files: 46% (287/622) Updating files: 47% (293/622) Updating files: 48% (299/622) Updating files: 49% (305/622) Updating files: 50% (311/622) Updating files: 51% (318/622) Updating files: 52% (324/622) Updating files: 53% (330/622) Updating files: 54% (336/622) Updating files: 55% (343/622) Updating files: 56% (349/622) Updating files: 57% (355/622) Updating files: 58% (361/622) Updating files: 58% (365/622) Updating files: 59% (367/622) Updating files: 60% (374/622) Updating files: 61% (380/622) Updating files: 62% (386/622) Updating files: 63% (392/622) Updating files: 64% (399/622) Updating files: 65% (405/622) Updating files: 66% (411/622) Updating files: 67% (417/622) Updating files: 68% (423/622) Updating files: 69% (430/622) Updating files: 70% (436/622) Updating files: 71% (442/622) Updating files: 72% (448/622) Updating files: 73% (455/622) Updating files: 74% (461/622) Updating files: 75% (467/622) Updating files: 76% (473/622) Updating files: 77% (479/622) Updating files: 78% (486/622) Updating files: 79% (492/622) Updating files: 80% (498/622) Updating files: 81% (504/622) Updating files: 82% (511/622) Updating files: 83% (517/622) Updating files: 84% (523/622) Updating files: 84% (525/622) Updating files: 85% (529/622) Updating files: 86% (535/622) Updating files: 87% (542/622) Updating files: 88% (548/622) Updating files: 89% (554/622) Updating files: 90% (560/622) Updating files: 91% (567/622) Updating files: 92% (573/622) Updating files: 93% (579/622) Updating files: 94% (585/622) Updating files: 95% (591/622) Updating files: 96% (598/622) Updating files: 97% (604/622) Updating files: 98% (610/622) Updating files: 99% (616/622) Updating files: 100% (622/622) Updating files: 100% (622/622), done. 2023-10-30T13:48:31.895 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 3fa121bf0a8 v0.25 2023-10-30T13:48:31.902 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.25 2023-10-30T13:48:32.186 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 26 -eq 48 ] 2023-10-30T13:48:32.186 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.26 2023-10-30T13:48:32.186 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.26 2023-10-30T13:48:32.187 INFO:tasks.workunit.client.0.smithi046.stdout:v0.26 2023-10-30T13:48:32.187 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.26 2023-10-30T13:48:35.042 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 24% (59/239) Updating files: 25% (60/239) Updating files: 26% (63/239) Updating files: 27% (65/239) Updating files: 28% (67/239) Updating files: 29% (70/239) Updating files: 30% (72/239) Updating files: 31% (75/239) Updating files: 32% (77/239) Updating files: 33% (79/239) Updating files: 34% (82/239) Updating files: 35% (84/239) Updating files: 36% (87/239) Updating files: 37% (89/239) Updating files: 38% (91/239) Updating files: 39% (94/239) Updating files: 40% (96/239) Updating files: 41% (98/239) Updating files: 42% (101/239) Updating files: 43% (103/239) Updating files: 44% (106/239) Updating files: 45% (108/239) Updating files: 46% (110/239) Updating files: 47% (113/239) Updating files: 48% (115/239) Updating files: 49% (118/239) Updating files: 50% (120/239) Updating files: 51% (122/239) Updating files: 52% (125/239) Updating files: 53% (127/239) Updating files: 54% (130/239) Updating files: 55% (132/239) Updating files: 56% (134/239) Updating files: 57% (137/239) Updating files: 58% (139/239) Updating files: 59% (142/239) Updating files: 60% (144/239) Updating files: 61% (146/239) Updating files: 62% (149/239) Updating files: 62% (150/239) Updating files: 63% (151/239) Updating files: 64% (153/239) Updating files: 65% (156/239) Updating files: 66% (158/239) Updating files: 67% (161/239) Updating files: 68% (163/239) Updating files: 69% (165/239) Updating files: 70% (168/239) Updating files: 71% (170/239) Updating files: 72% (173/239) Updating files: 73% (175/239) Updating files: 74% (177/239) Updating files: 75% (180/239) Updating files: 76% (182/239) Updating files: 77% (185/239) Updating files: 78% (187/239) Updating files: 79% (189/239) Updating files: 80% (192/239) Updating files: 81% (194/239) Updating files: 82% (196/239) Updating files: 83% (199/239) Updating files: 84% (201/239) Updating files: 85% (204/239) Updating files: 86% (206/239) Updating files: 87% (208/239) Updating files: 88% (211/239) Updating files: 89% (213/239) Updating files: 90% (216/239) Updating files: 91% (218/239) Updating files: 92% (220/239) Updating files: 93% (223/239) Updating files: 94% (225/239) Updating files: 95% (228/239) Updating files: 96% (230/239) Updating files: 97% (232/239) Updating files: 98% (235/239) Updating files: 99% (237/239) Updating files: 100% (239/239) Updating files: 100% (239/239), done. 2023-10-30T13:48:35.085 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 9981ff90968 v0.26 2023-10-30T13:48:35.090 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.26 2023-10-30T13:48:35.378 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 27 -eq 48 ] 2023-10-30T13:48:35.379 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.27 2023-10-30T13:48:35.379 INFO:tasks.workunit.client.0.smithi046.stdout:v0.27 2023-10-30T13:48:35.379 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.27 2023-10-30T13:48:35.379 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.27 2023-10-30T13:48:50.505 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:50.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:55.570 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 11% (29/253) Updating files: 12% (31/253) Updating files: 13% (33/253) Updating files: 14% (36/253) Updating files: 15% (38/253) Updating files: 15% (40/253) Updating files: 16% (41/253) Updating files: 17% (44/253) Updating files: 18% (46/253) Updating files: 19% (49/253) Updating files: 20% (51/253) Updating files: 21% (54/253) Updating files: 22% (56/253) Updating files: 23% (59/253) Updating files: 24% (61/253) Updating files: 24% (62/253) Updating files: 25% (64/253) Updating files: 26% (66/253) Updating files: 27% (69/253) Updating files: 27% (70/253) Updating files: 28% (71/253) Updating files: 29% (74/253) Updating files: 30% (76/253) Updating files: 31% (79/253) Updating files: 32% (81/253) Updating files: 33% (84/253) Updating files: 34% (87/253) Updating files: 35% (89/253) Updating files: 36% (92/253) Updating files: 37% (94/253) Updating files: 38% (97/253) Updating files: 39% (99/253) Updating files: 40% (102/253) Updating files: 41% (104/253) Updating files: 42% (107/253) Updating files: 43% (109/253) Updating files: 44% (112/253) Updating files: 45% (114/253) Updating files: 46% (117/253) Updating files: 47% (119/253) Updating files: 48% (122/253) Updating files: 49% (124/253) Updating files: 50% (127/253) Updating files: 51% (130/253) Updating files: 52% (132/253) Updating files: 53% (135/253) Updating files: 54% (137/253) Updating files: 55% (140/253) Updating files: 56% (142/253) Updating files: 57% (145/253) Updating files: 58% (147/253) Updating files: 59% (150/253) Updating files: 60% (152/253) Updating files: 61% (155/253) Updating files: 62% (157/253) Updating files: 63% (160/253) Updating files: 64% (162/253) Updating files: 65% (165/253) Updating files: 66% (167/253) Updating files: 67% (170/253) Updating files: 68% (173/253) Updating files: 69% (175/253) Updating files: 70% (178/253) Updating files: 71% (180/253) Updating files: 72% (183/253) Updating files: 72% (184/253) Updating files: 73% (185/253) Updating files: 74% (188/253) Updating files: 75% (190/253) Updating files: 76% (193/253) Updating files: 77% (195/253) Updating files: 78% (198/253) Updating files: 78% (199/253) Updating files: 79% (200/253) Updating files: 80% (203/253) Updating files: 81% (205/253) Updating files: 82% (208/253) Updating files: 83% (210/253) Updating files: 84% (213/253) Updating files: 85% (216/253) Updating files: 86% (218/253) Updating files: 87% (221/253) Updating files: 88% (223/253) Updating files: 89% (226/253) Updating files: 90% (228/253) Updating files: 91% (231/253) Updating files: 92% (233/253) Updating files: 93% (236/253) Updating files: 94% (238/253) Updating files: 95% (241/253) Updating files: 96% (243/253) Updating files: 97% (246/253) Updating files: 98% (248/253) Updating files: 99% (251/253) Updating files: 100% (253/253) Updating files: 100% (253/253), done. 2023-10-30T13:48:55.618 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 793034c62c8 v0.27 2023-10-30T13:48:55.623 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.27 2023-10-30T13:48:55.689 INFO:tasks.workunit.client.0.smithi046.stdout:v0.28 2023-10-30T13:48:55.689 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 28 -eq 48 ] 2023-10-30T13:48:55.689 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.28 2023-10-30T13:48:55.690 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.28 2023-10-30T13:48:55.690 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.28 2023-10-30T13:48:59.156 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 89% (204/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 93% (213/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2023-10-30T13:49:00.064 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 071881d7e55 v0.28 2023-10-30T13:49:00.070 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.28 2023-10-30T13:49:00.155 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 29 -eq 48 ] 2023-10-30T13:49:00.155 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.29 2023-10-30T13:49:00.155 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.29 2023-10-30T13:49:00.156 INFO:tasks.workunit.client.0.smithi046.stdout:v0.29 2023-10-30T13:49:00.156 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.29 2023-10-30T13:49:14.786 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 20% (32/159) Updating files: 21% (34/159) Updating files: 22% (35/159) Updating files: 23% (37/159) Updating files: 24% (39/159) Updating files: 25% (40/159) Updating files: 26% (42/159) Updating files: 27% (43/159) Updating files: 28% (45/159) Updating files: 29% (47/159) Updating files: 30% (48/159) Updating files: 31% (50/159) Updating files: 32% (51/159) Updating files: 33% (53/159) Updating files: 34% (55/159) Updating files: 35% (56/159) Updating files: 36% (58/159) Updating files: 37% (59/159) Updating files: 38% (61/159) Updating files: 39% (63/159) Updating files: 40% (64/159) Updating files: 41% (66/159) Updating files: 42% (67/159) Updating files: 43% (69/159) Updating files: 44% (70/159) Updating files: 45% (72/159) Updating files: 46% (74/159) Updating files: 47% (75/159) Updating files: 48% (77/159) Updating files: 49% (78/159) Updating files: 50% (80/159) Updating files: 51% (82/159) Updating files: 52% (83/159) Updating files: 53% (85/159) Updating files: 54% (86/159) Updating files: 54% (87/159) Updating files: 55% (88/159) Updating files: 56% (90/159) Updating files: 57% (91/159) Updating files: 58% (93/159) Updating files: 59% (94/159) Updating files: 60% (96/159) Updating files: 61% (97/159) Updating files: 61% (98/159) Updating files: 62% (99/159) Updating files: 63% (101/159) Updating files: 64% (102/159) Updating files: 65% (104/159) Updating files: 66% (105/159) Updating files: 67% (107/159) Updating files: 68% (109/159) Updating files: 69% (110/159) Updating files: 70% (112/159) Updating files: 71% (113/159) Updating files: 72% (115/159) Updating files: 73% (117/159) Updating files: 74% (118/159) Updating files: 75% (120/159) Updating files: 76% (121/159) Updating files: 77% (123/159) Updating files: 78% (125/159) Updating files: 79% (126/159) Updating files: 80% (128/159) Updating files: 81% (129/159) Updating files: 82% (131/159) Updating files: 83% (132/159) Updating files: 84% (134/159) Updating files: 85% (136/159) Updating files: 86% (137/159) Updating files: 87% (139/159) Updating files: 88% (140/159) Updating files: 89% (142/159) Updating files: 90% (144/159) Updating files: 91% (145/159) Updating files: 92% (147/159) Updating files: 93% (148/159) Updating files: 94% (150/159) Updating files: 95% (152/159) Updating files: 96% (153/159) Updating files: 97% (155/159) Updating files: 98% (156/159) Updating files: 99% (158/159) Updating files: 100% (159/159) Updating files: 100% (159/159), done. 2023-10-30T13:49:14.821 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 8e69c39f699 v0.29 2023-10-30T13:49:14.827 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.29 2023-10-30T13:49:15.631 INFO:tasks.workunit.client.0.smithi046.stdout:v0.30 2023-10-30T13:49:15.632 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 30 -eq 48 ] 2023-10-30T13:49:15.632 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.30 2023-10-30T13:49:15.632 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.30 2023-10-30T13:49:15.632 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.30 2023-10-30T13:49:20.617 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:49:20.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:49:34.424 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 36% (104/285) Updating files: 37% (106/285) Updating files: 38% (109/285) Updating files: 39% (112/285) Updating files: 40% (114/285) Updating files: 41% (117/285) Updating files: 42% (120/285) Updating files: 43% (123/285) Updating files: 44% (126/285) Updating files: 45% (129/285) Updating files: 46% (132/285) Updating files: 47% (134/285) Updating files: 48% (137/285) Updating files: 49% (140/285) Updating files: 49% (141/285) Updating files: 50% (143/285) Updating files: 51% (146/285) Updating files: 52% (149/285) Updating files: 53% (152/285) Updating files: 54% (154/285) Updating files: 55% (157/285) Updating files: 56% (160/285) Updating files: 57% (163/285) Updating files: 58% (166/285) Updating files: 59% (169/285) Updating files: 60% (171/285) Updating files: 61% (174/285) Updating files: 62% (177/285) Updating files: 63% (180/285) Updating files: 64% (183/285) Updating files: 65% (186/285) Updating files: 66% (189/285) Updating files: 67% (191/285) Updating files: 68% (194/285) Updating files: 69% (197/285) Updating files: 70% (200/285) Updating files: 71% (203/285) Updating files: 72% (206/285) Updating files: 73% (209/285) Updating files: 74% (211/285) Updating files: 75% (214/285) Updating files: 76% (217/285) Updating files: 77% (220/285) Updating files: 78% (223/285) Updating files: 79% (226/285) Updating files: 80% (228/285) Updating files: 81% (231/285) Updating files: 82% (234/285) Updating files: 83% (237/285) Updating files: 84% (240/285) Updating files: 85% (243/285) Updating files: 85% (245/285) Updating files: 86% (246/285) Updating files: 87% (248/285) Updating files: 88% (251/285) Updating files: 89% (254/285) Updating files: 90% (257/285) Updating files: 91% (260/285) Updating files: 91% (261/285) Updating files: 92% (263/285) Updating files: 93% (266/285) Updating files: 94% (268/285) Updating files: 95% (271/285) Updating files: 96% (274/285) Updating files: 97% (277/285) Updating files: 98% (280/285) Updating files: 99% (283/285) Updating files: 100% (285/285) Updating files: 100% (285/285), done. 2023-10-30T13:49:34.461 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 64b1b2c70f0 v0.30 2023-10-30T13:49:34.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.30 2023-10-30T13:49:34.514 INFO:tasks.workunit.client.0.smithi046.stdout:v0.31 2023-10-30T13:49:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 31 -eq 48 ] 2023-10-30T13:49:34.514 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.31 2023-10-30T13:49:34.515 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.31 2023-10-30T13:49:34.515 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.31 2023-10-30T13:49:41.604 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 32% (130/398) Updating files: 33% (132/398) Updating files: 34% (136/398) Updating files: 35% (140/398) Updating files: 36% (144/398) Updating files: 37% (148/398) Updating files: 37% (150/398) Updating files: 38% (152/398) Updating files: 39% (156/398) Updating files: 40% (160/398) Updating files: 41% (164/398) Updating files: 41% (166/398) Updating files: 42% (168/398) Updating files: 43% (172/398) Updating files: 43% (173/398) Updating files: 44% (176/398) Updating files: 45% (180/398) Updating files: 46% (184/398) Updating files: 47% (188/398) Updating files: 48% (192/398) Updating files: 49% (196/398) Updating files: 50% (199/398) Updating files: 51% (203/398) Updating files: 52% (207/398) Updating files: 53% (211/398) Updating files: 54% (215/398) Updating files: 55% (219/398) Updating files: 56% (223/398) Updating files: 57% (227/398) Updating files: 58% (231/398) Updating files: 59% (235/398) Updating files: 60% (239/398) Updating files: 61% (243/398) Updating files: 62% (247/398) Updating files: 63% (251/398) Updating files: 64% (255/398) Updating files: 65% (259/398) Updating files: 66% (263/398) Updating files: 67% (267/398) Updating files: 68% (271/398) Updating files: 69% (275/398) Updating files: 70% (279/398) Updating files: 71% (283/398) Updating files: 71% (285/398) Updating files: 72% (287/398) Updating files: 73% (291/398) Updating files: 74% (295/398) Updating files: 75% (299/398) Updating files: 76% (303/398) Updating files: 77% (307/398) Updating files: 78% (311/398) Updating files: 79% (315/398) Updating files: 80% (319/398) Updating files: 81% (323/398) Updating files: 82% (327/398) Updating files: 83% (331/398) Updating files: 84% (335/398) Updating files: 85% (339/398) Updating files: 86% (343/398) Updating files: 87% (347/398) Updating files: 88% (351/398) Updating files: 89% (355/398) Updating files: 90% (359/398) Updating files: 91% (363/398) Updating files: 92% (367/398) Updating files: 93% (371/398) Updating files: 94% (375/398) Updating files: 95% (379/398) Updating files: 96% (383/398) Updating files: 97% (387/398) Updating files: 98% (391/398) Updating files: 99% (395/398) Updating files: 100% (398/398) Updating files: 100% (398/398), done. 2023-10-30T13:49:41.671 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 9019c6ce640 v0.31 2023-10-30T13:49:41.675 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.31 2023-10-30T13:49:41.718 INFO:tasks.workunit.client.0.smithi046.stdout:v0.32 2023-10-30T13:49:41.718 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 32 -eq 48 ] 2023-10-30T13:49:41.718 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.32 2023-10-30T13:49:41.719 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.32 2023-10-30T13:49:41.719 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.32 2023-10-30T13:49:44.504 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 51% (100/195) Updating files: 52% (102/195) Updating files: 53% (104/195) Updating files: 54% (106/195) Updating files: 55% (108/195) Updating files: 56% (110/195) Updating files: 57% (112/195) Updating files: 58% (114/195) Updating files: 59% (116/195) Updating files: 60% (117/195) Updating files: 61% (119/195) Updating files: 62% (121/195) Updating files: 63% (123/195) Updating files: 64% (125/195) Updating files: 65% (127/195) Updating files: 66% (129/195) Updating files: 67% (131/195) Updating files: 68% (133/195) Updating files: 69% (135/195) Updating files: 69% (136/195) Updating files: 70% (137/195) Updating files: 71% (139/195) Updating files: 72% (141/195) Updating files: 73% (143/195) Updating files: 74% (145/195) Updating files: 75% (147/195) Updating files: 76% (149/195) Updating files: 77% (151/195) Updating files: 78% (153/195) Updating files: 79% (155/195) Updating files: 80% (156/195) Updating files: 81% (158/195) Updating files: 82% (160/195) Updating files: 83% (162/195) Updating files: 84% (164/195) Updating files: 85% (166/195) Updating files: 86% (168/195) Updating files: 87% (170/195) Updating files: 88% (172/195) Updating files: 89% (174/195) Updating files: 90% (176/195) Updating files: 91% (178/195) Updating files: 92% (180/195) Updating files: 93% (182/195) Updating files: 94% (184/195) Updating files: 95% (186/195) Updating files: 96% (188/195) Updating files: 97% (190/195) Updating files: 98% (192/195) Updating files: 99% (194/195) Updating files: 100% (195/195) Updating files: 100% (195/195), done. 2023-10-30T13:49:44.540 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at c08d08baa6a v0.32 2023-10-30T13:49:44.544 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.32 2023-10-30T13:49:44.619 INFO:tasks.workunit.client.0.smithi046.stdout:v0.33 2023-10-30T13:49:44.619 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 33 -eq 48 ] 2023-10-30T13:49:44.619 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.33 2023-10-30T13:49:44.620 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.33 2023-10-30T13:49:44.620 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.33 2023-10-30T13:49:48.581 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:48.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.729+0000 7f08171d3640 1 Processor -- start 2023-10-30T13:49:48.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f08171d3640 1 -- start start 2023-10-30T13:49:48.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f08171d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08100c05d0 0x7f081013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f08171d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f08100c6f60 0x7f081013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f08171d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08100c0a50 con 0x7f08100c6b80 2023-10-30T13:49:48.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f08171d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f081006f6e0 con 0x7f08100c6f60 2023-10-30T13:49:48.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f08171d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081006f9c0 con 0x7f08100c05d0 2023-10-30T13:49:48.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f0814f6f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f08100c6b80 0x7f08100bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39376/0 (socket says 172.21.15.46:39376) 2023-10-30T13:49:48.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f0814f6f640 1 -- 172.21.15.46:0/2695059160 learned_addr learned my addr 172.21.15.46:0/2695059160 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:49:48.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f0815770640 1 --2- 172.21.15.46:0/2695059160 >> v2:172.21.15.154:3300/0 conn(0x7f08100c6f60 0x7f081013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f080ffff640 1 --2- 172.21.15.46:0/2695059160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08100c05d0 0x7f081013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f080f7fe640 1 -- 172.21.15.46:0/2695059160 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378739909 0 0) 0x7f08100c0a50 con 0x7f08100c6b80 2023-10-30T13:49:48.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f080f7fe640 1 -- 172.21.15.46:0/2695059160 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0800003530 con 0x7f08100c6b80 2023-10-30T13:49:48.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f0815770640 1 -- 172.21.15.46:0/2695059160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08100c05d0 msgr2=0x7f081013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:48.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f0815770640 1 --2- 172.21.15.46:0/2695059160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08100c05d0 0x7f081013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f080f7fe640 1 -- 172.21.15.46:0/2695059160 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2522979799 0 0) 0x7f0800003530 con 0x7f08100c6b80 2023-10-30T13:49:48.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f0815770640 1 -- 172.21.15.46:0/2695059160 >> v1:172.21.15.46:6789/0 conn(0x7f08100c6b80 legacy=0x7f08100bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.733+0000 7f0815770640 1 -- 172.21.15.46:0/2695059160 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081006fca0 con 0x7f08100c6f60 2023-10-30T13:49:48.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.737+0000 7f0815770640 1 --2- 172.21.15.46:0/2695059160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08100c6f60 0x7f081013f2d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0808009d90 tx=0x7f080801c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ca741cd3f074813 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:48.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.737+0000 7f080f7fe640 1 -- 172.21.15.46:0/2695059160 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804001680 con 0x7f08100c6f60 2023-10-30T13:49:48.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.737+0000 7f080f7fe640 1 -- 172.21.15.46:0/2695059160 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08080a5580 con 0x7f08100c6f60 2023-10-30T13:49:48.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.737+0000 7f080f7fe640 1 -- 172.21.15.46:0/2695059160 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08080a58f0 con 0x7f08100c6f60 2023-10-30T13:49:48.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.737+0000 7f08171d3640 1 -- 172.21.15.46:0/2695059160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08100c6f60 msgr2=0x7f081013f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.737+0000 7f08171d3640 1 --2- 172.21.15.46:0/2695059160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08100c6f60 0x7f081013f2d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0808009d90 tx=0x7f080801c870 comp rx=0 tx=0).stop 2023-10-30T13:49:48.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.737+0000 7f08171d3640 1 -- 172.21.15.46:0/2695059160 shutdown_connections 2023-10-30T13:49:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.737+0000 7f08171d3640 1 --2- 172.21.15.46:0/2695059160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08100c6f60 0x7f081013f2d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.737+0000 7f08171d3640 1 --2- 172.21.15.46:0/2695059160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08100c05d0 0x7f081013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.737+0000 7f08171d3640 1 -- 172.21.15.46:0/2695059160 >> 172.21.15.46:0/2695059160 conn(0x7f08100c7910 msgr2=0x7f08100c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.737+0000 7f08171d3640 1 -- 172.21.15.46:0/2695059160 shutdown_connections 2023-10-30T13:49:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.737+0000 7f08171d3640 1 -- 172.21.15.46:0/2695059160 wait complete. 2023-10-30T13:49:48.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f08171d3640 1 Processor -- start 2023-10-30T13:49:48.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f08171d3640 1 -- start start 2023-10-30T13:49:48.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f08171d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08100c05d0 0x7f08100af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f08171d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08100c6b80 0x7f08100c0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f08171d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08100c6f60 0x7f08100c1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f08171d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08101f26a0 con 0x7f08100c6f60 2023-10-30T13:49:48.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f08171d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08101f5ad0 con 0x7f08100c05d0 2023-10-30T13:49:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f08171d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08101f5d70 con 0x7f08100c6b80 2023-10-30T13:49:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f0815770640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08100c6f60 0x7f08100c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f0815770640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08100c6f60 0x7f08100c1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39942/0 (socket says 172.21.15.46:39942) 2023-10-30T13:49:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f0815770640 1 -- 172.21.15.46:0/4040776172 learned_addr learned my addr 172.21.15.46:0/4040776172 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:49:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f0814f6f640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08100c05d0 0x7f08100af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f080ffff640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08100c6b80 0x7f08100c0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f0815770640 1 -- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08100c6b80 msgr2=0x7f08100c0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f0815770640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08100c6b80 0x7f08100c0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f0815770640 1 -- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08100c05d0 msgr2=0x7f08100af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f0815770640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08100c05d0 0x7f08100af560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f0815770640 1 -- 172.21.15.46:0/4040776172 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0808009a70 con 0x7f08100c6f60 2023-10-30T13:49:48.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f0815770640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08100c6f60 0x7f08100c1270 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f080801af20 tx=0x7f0808019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:48.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f080d7fa640 1 -- 172.21.15.46:0/4040776172 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808009450 con 0x7f08100c6f60 2023-10-30T13:49:48.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f080d7fa640 1 -- 172.21.15.46:0/4040776172 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0808003e60 con 0x7f08100c6f60 2023-10-30T13:49:48.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f080d7fa640 1 -- 172.21.15.46:0/4040776172 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808004040 con 0x7f08100c6f60 2023-10-30T13:49:48.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f08171d3640 1 -- 172.21.15.46:0/4040776172 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08101f1d00 con 0x7f08100c6f60 2023-10-30T13:49:48.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.741+0000 7f08171d3640 1 -- 172.21.15.46:0/4040776172 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08101f22c0 con 0x7f08100c6f60 2023-10-30T13:49:48.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.745+0000 7f08171d3640 1 -- 172.21.15.46:0/4040776172 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08101f2480 con 0x7f08100c6f60 2023-10-30T13:49:48.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.745+0000 7f080d7fa640 1 -- 172.21.15.46:0/4040776172 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f08080041a0 con 0x7f08100c6f60 2023-10-30T13:49:48.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.745+0000 7f080d7fa640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07f40664a0 0x7f07f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:48.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.745+0000 7f080d7fa640 1 -- 172.21.15.46:0/4040776172 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f080811f810 con 0x7f08100c6f60 2023-10-30T13:49:48.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.745+0000 7f0814f6f640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07f40664a0 0x7f07f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:48.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.745+0000 7f0814f6f640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07f40664a0 0x7f07f4068960 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f08040011f0 tx=0x7f08040016e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:48.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.749+0000 7f080d7fa640 1 -- 172.21.15.46:0/4040776172 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f08080ec770 con 0x7f08100c6f60 2023-10-30T13:49:48.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.941+0000 7f08171d3640 1 -- 172.21.15.46:0/4040776172 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08101f27e0 con 0x7f08100c6f60 2023-10-30T13:49:48.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.941+0000 7f080d7fa640 1 -- 172.21.15.46:0/4040776172 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f08101f27e0 con 0x7f08100c6f60 2023-10-30T13:49:48.947 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:49:48.947 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6858,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:51.253577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:48.947 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 116 2023-10-30T13:49:48.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.945+0000 7f08171d3640 1 -- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07f40664a0 msgr2=0x7f07f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.945+0000 7f08171d3640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07f40664a0 0x7f07f4068960 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f08040011f0 tx=0x7f08040016e0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.945+0000 7f08171d3640 1 -- 172.21.15.46:0/4040776172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08100c6f60 msgr2=0x7f08100c1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:48.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.945+0000 7f08171d3640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08100c6f60 0x7f08100c1270 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f080801af20 tx=0x7f0808019610 comp rx=0 tx=0).stop 2023-10-30T13:49:48.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.945+0000 7f08171d3640 1 -- 172.21.15.46:0/4040776172 shutdown_connections 2023-10-30T13:49:48.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.945+0000 7f08171d3640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f07f40664a0 0x7f07f4068960 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.945+0000 7f08171d3640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08100c6f60 0x7f08100c1270 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.945+0000 7f08171d3640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08100c6b80 0x7f08100c0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.945+0000 7f08171d3640 1 --2- 172.21.15.46:0/4040776172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08100c05d0 0x7f08100af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:48.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.945+0000 7f08171d3640 1 -- 172.21.15.46:0/4040776172 >> 172.21.15.46:0/4040776172 conn(0x7f08100c7910 msgr2=0x7f08100c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:48.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.945+0000 7f08171d3640 1 -- 172.21.15.46:0/4040776172 shutdown_connections 2023-10-30T13:49:48.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:48.945+0000 7f08171d3640 1 -- 172.21.15.46:0/4040776172 wait complete. 2023-10-30T13:49:48.968 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=1) 2023-10-30T13:49:48.968 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:49:48.968 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:49:48.990 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:49:48.991 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:49.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.169+0000 7f5350fa0640 1 Processor -- start 2023-10-30T13:49:49.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.169+0000 7f5350fa0640 1 -- start start 2023-10-30T13:49:49.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.169+0000 7f5350fa0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f534c0cefa0 0x7f534c0cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:49.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.169+0000 7f5350fa0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534c15a6c0 con 0x7f534c1538f0 2023-10-30T13:49:49.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.169+0000 7f5350fa0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534c0ca6a0 con 0x7f534c15cd50 2023-10-30T13:49:49.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.169+0000 7f5350fa0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f534c0cf8f0 con 0x7f534c0cefa0 2023-10-30T13:49:49.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.169+0000 7f534b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f534c1538f0 0x7f534c159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39382/0 (socket says 172.21.15.46:39382) 2023-10-30T13:49:49.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.173+0000 7f534b7fe640 1 -- 172.21.15.46:0/392386529 learned_addr learned my addr 172.21.15.46:0/392386529 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:49:49.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.169+0000 7f534bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f534c0cefa0 0x7f534c0cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:49.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.173+0000 7f534bfff640 1 -- 172.21.15.46:0/392386529 >> v1:172.21.15.154:6789/0 conn(0x7f534c15cd50 legacy=0x7f534c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:49.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.173+0000 7f534bfff640 1 -- 172.21.15.46:0/392386529 >> v1:172.21.15.46:6789/0 conn(0x7f534c1538f0 legacy=0x7f534c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:49.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.173+0000 7f534bfff640 1 -- 172.21.15.46:0/392386529 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c15d300 con 0x7f534c0cefa0 2023-10-30T13:49:49.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.173+0000 7f534bfff640 1 --2- 172.21.15.46:0/392386529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f534c0cefa0 0x7f534c0cf3b0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f5344009d90 tx=0x7f534401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1de89ceeadf2e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:49.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.173+0000 7f534affd640 1 -- 172.21.15.46:0/392386529 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53440af600 con 0x7f534c0cefa0 2023-10-30T13:49:49.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.177+0000 7f534affd640 1 -- 172.21.15.46:0/392386529 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53440ab050 con 0x7f534c0cefa0 2023-10-30T13:49:49.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.177+0000 7f534affd640 1 -- 172.21.15.46:0/392386529 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53440af980 con 0x7f534c0cefa0 2023-10-30T13:49:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.177+0000 7f5350fa0640 1 -- 172.21.15.46:0/392386529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f534c0cefa0 msgr2=0x7f534c0cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.177+0000 7f5350fa0640 1 --2- 172.21.15.46:0/392386529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f534c0cefa0 0x7f534c0cf3b0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f5344009d90 tx=0x7f534401f130 comp rx=0 tx=0).stop 2023-10-30T13:49:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.177+0000 7f5350fa0640 1 -- 172.21.15.46:0/392386529 shutdown_connections 2023-10-30T13:49:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.177+0000 7f5350fa0640 1 --2- 172.21.15.46:0/392386529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f534c0cefa0 0x7f534c0cf3b0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.177+0000 7f5350fa0640 1 -- 172.21.15.46:0/392386529 >> 172.21.15.46:0/392386529 conn(0x7f534c06a640 msgr2=0x7f534c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:49.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.177+0000 7f5350fa0640 1 -- 172.21.15.46:0/392386529 shutdown_connections 2023-10-30T13:49:49.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.177+0000 7f5350fa0640 1 -- 172.21.15.46:0/392386529 wait complete. 2023-10-30T13:49:49.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.177+0000 7f5350fa0640 1 Processor -- start 2023-10-30T13:49:49.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.181+0000 7f5350fa0640 1 -- start start 2023-10-30T13:49:49.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.181+0000 7f5350fa0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f534c0cefa0 0x7f534c1efca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:49.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.181+0000 7f5350fa0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f534c1538f0 0x7f534c0b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:49.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.181+0000 7f5350fa0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f534c15cd50 0x7f534c0b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:49.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.181+0000 7f5350fa0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c1459a0 con 0x7f534c1538f0 2023-10-30T13:49:49.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.181+0000 7f5350fa0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c0b47a0 con 0x7f534c15cd50 2023-10-30T13:49:49.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.181+0000 7f5350fa0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534c1ef810 con 0x7f534c0cefa0 2023-10-30T13:49:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.185+0000 7f534b7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f534c0cefa0 0x7f534c1efca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.185+0000 7f534b7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f534c0cefa0 0x7f534c1efca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:43100/0 (socket says 172.21.15.46:43100) 2023-10-30T13:49:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.185+0000 7f534b7fe640 1 -- 172.21.15.46:0/1819276350 learned_addr learned my addr 172.21.15.46:0/1819276350 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:49:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.185+0000 7f534bfff640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f534c15cd50 0x7f534c0b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.189+0000 7f534b7fe640 1 -- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f534c15cd50 msgr2=0x7f534c0b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.189+0000 7f534b7fe640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f534c15cd50 0x7f534c0b4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:49.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.189+0000 7f534b7fe640 1 -- 172.21.15.46:0/1819276350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f534c1538f0 msgr2=0x7f534c0b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:49.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.189+0000 7f534b7fe640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f534c1538f0 0x7f534c0b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:49.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.189+0000 7f534b7fe640 1 -- 172.21.15.46:0/1819276350 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5344009a70 con 0x7f534c0cefa0 2023-10-30T13:49:49.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.197+0000 7f534b7fe640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f534c0cefa0 0x7f534c1efca0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f533c00c500 tx=0x7f533c00c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:49.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.201+0000 7f5348ff9640 1 -- 172.21.15.46:0/1819276350 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533c005cf0 con 0x7f534c0cefa0 2023-10-30T13:49:49.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.201+0000 7f5348ff9640 1 -- 172.21.15.46:0/1819276350 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533c005e50 con 0x7f534c0cefa0 2023-10-30T13:49:49.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.201+0000 7f5348ff9640 1 -- 172.21.15.46:0/1819276350 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533c0072f0 con 0x7f534c0cefa0 2023-10-30T13:49:49.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.205+0000 7f5350fa0640 1 -- 172.21.15.46:0/1819276350 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534c0b49c0 con 0x7f534c0cefa0 2023-10-30T13:49:49.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.205+0000 7f5350fa0640 1 -- 172.21.15.46:0/1819276350 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f534c0b4f80 con 0x7f534c0cefa0 2023-10-30T13:49:49.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.205+0000 7f5350fa0640 1 -- 172.21.15.46:0/1819276350 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5330005260 con 0x7f534c0cefa0 2023-10-30T13:49:49.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.217+0000 7f5348ff9640 1 -- 172.21.15.46:0/1819276350 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f533c007450 con 0x7f534c0cefa0 2023-10-30T13:49:49.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.217+0000 7f5348ff9640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f532c0664f0 0x7f532c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:49.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.217+0000 7f5343fff640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f532c0664f0 0x7f532c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:49.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.221+0000 7f5343fff640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f532c0664f0 0x7f532c0689b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5334000f70 tx=0x7f53340079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:49.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.241+0000 7f5348ff9640 1 -- 172.21.15.46:0/1819276350 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f533c054a20 con 0x7f534c0cefa0 2023-10-30T13:49:49.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.257+0000 7f5348ff9640 1 -- 172.21.15.46:0/1819276350 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f533c0547f0 con 0x7f534c0cefa0 2023-10-30T13:49:49.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 -- 172.21.15.46:0/1819276350 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53300057e0 con 0x7f534c0cefa0 2023-10-30T13:49:49.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5348ff9640 1 -- 172.21.15.46:0/1819276350 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f533c017020 con 0x7f534c0cefa0 2023-10-30T13:49:49.453 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:49:49.454 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6858,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:51.253577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:49.454 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 116 2023-10-30T13:49:49.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 -- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f532c0664f0 msgr2=0x7f532c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:49.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f532c0664f0 0x7f532c0689b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5334000f70 tx=0x7f53340079c0 comp rx=0 tx=0).stop 2023-10-30T13:49:49.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 -- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f534c0cefa0 msgr2=0x7f534c1efca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:49.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f534c0cefa0 0x7f534c1efca0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f533c00c500 tx=0x7f533c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:49:49.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 -- 172.21.15.46:0/1819276350 shutdown_connections 2023-10-30T13:49:49.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f532c0664f0 0x7f532c0689b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:49.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f534c15cd50 0x7f534c0b4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:49.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f534c1538f0 0x7f534c0b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:49.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 --2- 172.21.15.46:0/1819276350 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f534c0cefa0 0x7f534c1efca0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:49.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 -- 172.21.15.46:0/1819276350 >> 172.21.15.46:0/1819276350 conn(0x7f534c06a640 msgr2=0x7f534c0cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:49.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 -- 172.21.15.46:0/1819276350 shutdown_connections 2023-10-30T13:49:49.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:49.449+0000 7f5350fa0640 1 -- 172.21.15.46:0/1819276350 wait complete. 2023-10-30T13:49:49.476 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:49:50.748 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:49:50.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:49:51.477 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe8a1901640 1 Processor -- start 2023-10-30T13:49:51.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe8a1901640 1 -- start start 2023-10-30T13:49:51.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe8a1901640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe89c0c6f60 0x7fe89c1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:51.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe8a1901640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0c0a50 con 0x7fe89c0c6b80 2023-10-30T13:49:51.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe8a1901640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c06f6e0 con 0x7fe89c0c05d0 2023-10-30T13:49:51.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe8a1901640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c06f9c0 con 0x7fe89c0c6f60 2023-10-30T13:49:51.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe89affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe89c0c6b80 0x7fe89c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39396/0 (socket says 172.21.15.46:39396) 2023-10-30T13:49:51.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe89affd640 1 -- 172.21.15.46:0/1063292726 learned_addr learned my addr 172.21.15.46:0/1063292726 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:49:51.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe89b7fe640 1 --2- 172.21.15.46:0/1063292726 >> v2:172.21.15.46:3300/0 conn(0x7fe89c0c6f60 0x7fe89c1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:51.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe89a7fc640 1 -- 172.21.15.46:0/1063292726 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332727827 0 0) 0x7fe89c0c0a50 con 0x7fe89c0c6b80 2023-10-30T13:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe89a7fc640 1 -- 172.21.15.46:0/1063292726 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe888003530 con 0x7fe89c0c6b80 2023-10-30T13:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe89b7fe640 1 -- 172.21.15.46:0/1063292726 >> v1:172.21.15.154:6789/0 conn(0x7fe89c0c05d0 legacy=0x7fe89c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe89b7fe640 1 -- 172.21.15.46:0/1063292726 >> v1:172.21.15.46:6789/0 conn(0x7fe89c0c6b80 legacy=0x7fe89c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:51.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.633+0000 7fe89b7fe640 1 -- 172.21.15.46:0/1063292726 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c06fca0 con 0x7fe89c0c6f60 2023-10-30T13:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.637+0000 7fe89b7fe640 1 --2- 172.21.15.46:0/1063292726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0c6f60 0x7fe89c1566a0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fe894009d90 tx=0x7fe89401eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca04d98b60bca489 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.637+0000 7fe89a7fc640 1 -- 172.21.15.46:0/1063292726 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8940a7a00 con 0x7fe89c0c6f60 2023-10-30T13:49:51.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.637+0000 7fe89a7fc640 1 -- 172.21.15.46:0/1063292726 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8940a7b60 con 0x7fe89c0c6f60 2023-10-30T13:49:51.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.637+0000 7fe89a7fc640 1 -- 172.21.15.46:0/1063292726 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8940af440 con 0x7fe89c0c6f60 2023-10-30T13:49:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.637+0000 7fe8a1901640 1 -- 172.21.15.46:0/1063292726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0c6f60 msgr2=0x7fe89c1566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.637+0000 7fe8a1901640 1 --2- 172.21.15.46:0/1063292726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0c6f60 0x7fe89c1566a0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fe894009d90 tx=0x7fe89401eb20 comp rx=0 tx=0).stop 2023-10-30T13:49:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.637+0000 7fe8a1901640 1 -- 172.21.15.46:0/1063292726 shutdown_connections 2023-10-30T13:49:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.637+0000 7fe8a1901640 1 --2- 172.21.15.46:0/1063292726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0c6f60 0x7fe89c1566a0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:51.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.637+0000 7fe8a1901640 1 -- 172.21.15.46:0/1063292726 >> 172.21.15.46:0/1063292726 conn(0x7fe89c0c7910 msgr2=0x7fe89c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.637+0000 7fe8a1901640 1 -- 172.21.15.46:0/1063292726 shutdown_connections 2023-10-30T13:49:51.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.637+0000 7fe8a1901640 1 -- 172.21.15.46:0/1063292726 wait complete. 2023-10-30T13:49:51.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8a1901640 1 Processor -- start 2023-10-30T13:49:51.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8a1901640 1 -- start start 2023-10-30T13:49:51.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8a1901640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe89c0c05d0 0x7fe89c0aec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:51.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8a1901640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0c6b80 0x7fe89c0c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:51.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8a1901640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89c0c6f60 0x7fe89c0c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:51.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8a1901640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c1f2b10 con 0x7fe89c0c6b80 2023-10-30T13:49:51.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8a1901640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c1f3000 con 0x7fe89c0c05d0 2023-10-30T13:49:51.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8a1901640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0c9170 con 0x7fe89c0c6f60 2023-10-30T13:49:51.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8927fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0c6b80 0x7fe89c0c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:51.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8927fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0c6b80 0x7fe89c0c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39992/0 (socket says 172.21.15.46:39992) 2023-10-30T13:49:51.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8927fc640 1 -- 172.21.15.46:0/2803871775 learned_addr learned my addr 172.21.15.46:0/2803871775 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:49:51.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe89affd640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe89c0c05d0 0x7fe89c0aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:51.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe89b7fe640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89c0c6f60 0x7fe89c0c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:51.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8927fc640 1 -- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89c0c6f60 msgr2=0x7fe89c0c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:51.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8927fc640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89c0c6f60 0x7fe89c0c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:51.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8927fc640 1 -- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe89c0c05d0 msgr2=0x7fe89c0aec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:51.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8927fc640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe89c0c05d0 0x7fe89c0aec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:51.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8927fc640 1 -- 172.21.15.46:0/2803871775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe894009a70 con 0x7fe89c0c6b80 2023-10-30T13:49:51.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe89affd640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe89c0c05d0 0x7fe89c0aec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:51.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8927fc640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0c6b80 0x7fe89c0c9930 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fe88400c410 tx=0x7fe88400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:51.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe898ff9640 1 -- 172.21.15.46:0/2803871775 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884011790 con 0x7fe89c0c6b80 2023-10-30T13:49:51.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe898ff9640 1 -- 172.21.15.46:0/2803871775 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8840118f0 con 0x7fe89c0c6b80 2023-10-30T13:49:51.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe898ff9640 1 -- 172.21.15.46:0/2803871775 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884011ce0 con 0x7fe89c0c6b80 2023-10-30T13:49:51.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8a1901640 1 -- 172.21.15.46:0/2803871775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c0c1d80 con 0x7fe89c0c6b80 2023-10-30T13:49:51.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8a1901640 1 -- 172.21.15.46:0/2803871775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c0c9410 con 0x7fe89c0c6b80 2023-10-30T13:49:51.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.641+0000 7fe8a1901640 1 -- 172.21.15.46:0/2803871775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c04e170 con 0x7fe89c0c6b80 2023-10-30T13:49:51.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.645+0000 7fe898ff9640 1 -- 172.21.15.46:0/2803871775 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe88401c460 con 0x7fe89c0c6b80 2023-10-30T13:49:51.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.645+0000 7fe898ff9640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:51.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.645+0000 7fe898ff9640 1 -- 172.21.15.46:0/2803871775 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7fe884084100 con 0x7fe89c0c6b80 2023-10-30T13:49:51.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.645+0000 7fe89affd640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:51.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.645+0000 7fe89affd640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe87c0664a0 0x7fe87c068960 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0011f0 tx=0x7fe88c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:51.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.649+0000 7fe898ff9640 1 -- 172.21.15.46:0/2803871775 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe884051040 con 0x7fe89c0c6b80 2023-10-30T13:49:51.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.841+0000 7fe8a1901640 1 -- 172.21.15.46:0/2803871775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe89c0c7fc0 con 0x7fe89c0c6b80 2023-10-30T13:49:51.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.841+0000 7fe898ff9640 1 -- 172.21.15.46:0/2803871775 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+6522 (secure 0 0 0) 0x7fe884054660 con 0x7fe89c0c6b80 2023-10-30T13:49:51.847 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:49:51.847 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6858,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:51.253577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:51.848 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 116 2023-10-30T13:49:51.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.845+0000 7fe8a1901640 1 -- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe87c0664a0 msgr2=0x7fe87c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:51.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.845+0000 7fe8a1901640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe87c0664a0 0x7fe87c068960 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0011f0 tx=0x7fe88c009040 comp rx=0 tx=0).stop 2023-10-30T13:49:51.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.845+0000 7fe8a1901640 1 -- 172.21.15.46:0/2803871775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0c6b80 msgr2=0x7fe89c0c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:51.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.845+0000 7fe8a1901640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0c6b80 0x7fe89c0c9930 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fe88400c410 tx=0x7fe88400c8d0 comp rx=0 tx=0).stop 2023-10-30T13:49:51.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.845+0000 7fe8a1901640 1 -- 172.21.15.46:0/2803871775 shutdown_connections 2023-10-30T13:49:51.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.845+0000 7fe8a1901640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:51.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.845+0000 7fe8a1901640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89c0c6f60 0x7fe89c0c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:51.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.845+0000 7fe8a1901640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0c6b80 0x7fe89c0c9930 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:51.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.845+0000 7fe8a1901640 1 --2- 172.21.15.46:0/2803871775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe89c0c05d0 0x7fe89c0aec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:51.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.845+0000 7fe8a1901640 1 -- 172.21.15.46:0/2803871775 >> 172.21.15.46:0/2803871775 conn(0x7fe89c0c7910 msgr2=0x7fe89c1388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:51.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.845+0000 7fe8a1901640 1 -- 172.21.15.46:0/2803871775 shutdown_connections 2023-10-30T13:49:51.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:51.845+0000 7fe8a1901640 1 -- 172.21.15.46:0/2803871775 wait complete. 2023-10-30T13:49:51.869 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:49:53.870 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:54.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4953d3e640 1 Processor -- start 2023-10-30T13:49:54.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4953d3e640 1 -- start start 2023-10-30T13:49:54.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4953d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f494c06f7a0 0x7f494c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4953d3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c066cf0 con 0x7f494c071110 2023-10-30T13:49:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4953d3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c0c70d0 con 0x7f494c0c6bd0 2023-10-30T13:49:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4953d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494c0700c0 con 0x7f494c06f7a0 2023-10-30T13:49:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f49522db640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f494c071110 0x7f494c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39398/0 (socket says 172.21.15.46:39398) 2023-10-30T13:49:54.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f49522db640 1 -- 172.21.15.46:0/3112095061 learned_addr learned my addr 172.21.15.46:0/3112095061 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:49:54.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4951ada640 1 --2- 172.21.15.46:0/3112095061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f494c06f7a0 0x7f494c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599873096 0 0) 0x7f494c066cf0 con 0x7f494c071110 2023-10-30T13:49:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4938003530 con 0x7f494c071110 2023-10-30T13:49:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 387701476 0 0) 0x7f4938003530 con 0x7f494c071110 2023-10-30T13:49:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f494c06f7a0 msgr2=0x7f494c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 --2- 172.21.15.46:0/3112095061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f494c06f7a0 0x7f494c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 >> v1:172.21.15.154:6789/0 conn(0x7f494c0c6bd0 legacy=0x7f494c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:54.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c0c7390 con 0x7f494c071110 2023-10-30T13:49:54.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f4944001a00 con 0x7f494c071110 2023-10-30T13:49:54.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 >> v1:172.21.15.46:6789/0 conn(0x7f494c071110 legacy=0x7f494c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:54.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 --2- 172.21.15.46:0/3112095061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49380039b0 0x7f4938003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 --2- 172.21.15.46:0/3112095061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49380047d0 0x7f4938004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 --2- 172.21.15.46:0/3112095061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f494c06f7a0 0x7f4938005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c0700c0 con 0x7f49380047d0 2023-10-30T13:49:54.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493800bad0 con 0x7f49380039b0 2023-10-30T13:49:54.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493800bc90 con 0x7f494c06f7a0 2023-10-30T13:49:54.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f49512d9640 1 --2- 172.21.15.46:0/3112095061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49380047d0 0x7f4938004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f4951ada640 1 --2- 172.21.15.46:0/3112095061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49380039b0 0x7f4938003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f49522db640 1 --2- 172.21.15.46:0/3112095061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f494c06f7a0 0x7f4938005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f49512d9640 1 -- 172.21.15.46:0/3112095061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f494c06f7a0 msgr2=0x7f4938005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:54.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f49512d9640 1 --2- 172.21.15.46:0/3112095061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f494c06f7a0 0x7f4938005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.021+0000 7f49512d9640 1 -- 172.21.15.46:0/3112095061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49380039b0 msgr2=0x7f4938003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f49512d9640 1 --2- 172.21.15.46:0/3112095061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49380039b0 0x7f4938003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f49512d9640 1 -- 172.21.15.46:0/3112095061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4938003530 con 0x7f49380047d0 2023-10-30T13:49:54.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f49512d9640 1 --2- 172.21.15.46:0/3112095061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49380047d0 0x7f4938004c50 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f493c001750 tx=0x7f493c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff079d471db2e4e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:54.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c0052d0 con 0x7f49380047d0 2023-10-30T13:49:54.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493c0a7050 con 0x7f49380047d0 2023-10-30T13:49:54.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f4950ad8640 1 -- 172.21.15.46:0/3112095061 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c003600 con 0x7f49380047d0 2023-10-30T13:49:54.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f4953d3e640 1 -- 172.21.15.46:0/3112095061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49380047d0 msgr2=0x7f4938004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:54.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f4953d3e640 1 --2- 172.21.15.46:0/3112095061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49380047d0 0x7f4938004c50 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f493c001750 tx=0x7f493c01c3a0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f49522db640 1 -- 172.21.15.46:0/3112095061 reap_dead start 2023-10-30T13:49:54.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f4953d3e640 1 -- 172.21.15.46:0/3112095061 shutdown_connections 2023-10-30T13:49:54.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f4953d3e640 1 -- 172.21.15.46:0/3112095061 >> 172.21.15.46:0/3112095061 conn(0x7f494c0c7910 msgr2=0x7f494c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:54.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f4953d3e640 1 -- 172.21.15.46:0/3112095061 shutdown_connections 2023-10-30T13:49:54.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.025+0000 7f4953d3e640 1 -- 172.21.15.46:0/3112095061 wait complete. 2023-10-30T13:49:54.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f4953d3e640 1 Processor -- start 2023-10-30T13:49:54.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f4953d3e640 1 -- start start 2023-10-30T13:49:54.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f4953d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49380047d0 0x7f494c1eae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f4953d3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f494c0b0a80 0x7f494c1508e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f4953d3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f494c159140 0x7f494c1595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f4953d3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c15a410 con 0x7f494c0b0a80 2023-10-30T13:49:54.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f4953d3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c159b00 con 0x7f494c159140 2023-10-30T13:49:54.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f4953d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494c1eb860 con 0x7f49380047d0 2023-10-30T13:49:54.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49512d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f494c0b0a80 0x7f494c1508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49512d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f494c0b0a80 0x7f494c1508e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40004/0 (socket says 172.21.15.46:40004) 2023-10-30T13:49:54.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49512d9640 1 -- 172.21.15.46:0/1481768177 learned_addr learned my addr 172.21.15.46:0/1481768177 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:49:54.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f4951ada640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49380047d0 0x7f494c1eae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49522db640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f494c159140 0x7f494c1595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49512d9640 1 -- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49380047d0 msgr2=0x7f494c1eae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:54.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49512d9640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49380047d0 0x7f494c1eae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49512d9640 1 -- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f494c159140 msgr2=0x7f494c1595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:54.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49512d9640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f494c159140 0x7f494c1595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49512d9640 1 -- 172.21.15.46:0/1481768177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c158cb0 con 0x7f494c0b0a80 2023-10-30T13:49:54.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49512d9640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f494c0b0a80 0x7f494c1508e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f493c001750 tx=0x7f4938005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:54.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49427fc640 1 -- 172.21.15.46:0/1481768177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c007a40 con 0x7f494c0b0a80 2023-10-30T13:49:54.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49427fc640 1 -- 172.21.15.46:0/1481768177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493c007680 con 0x7f494c0b0a80 2023-10-30T13:49:54.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f49427fc640 1 -- 172.21.15.46:0/1481768177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c006a00 con 0x7f494c0b0a80 2023-10-30T13:49:54.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f4953d3e640 1 -- 172.21.15.46:0/1481768177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494c1eb3b0 con 0x7f494c0b0a80 2023-10-30T13:49:54.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.029+0000 7f4953d3e640 1 -- 172.21.15.46:0/1481768177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494c15b220 con 0x7f494c0b0a80 2023-10-30T13:49:54.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.033+0000 7f4953d3e640 1 -- 172.21.15.46:0/1481768177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f494c158e60 con 0x7f494c0b0a80 2023-10-30T13:49:54.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.033+0000 7f49427fc640 1 -- 172.21.15.46:0/1481768177 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f493c006b60 con 0x7f494c0b0a80 2023-10-30T13:49:54.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.033+0000 7f49427fc640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49300664a0 0x7f4930068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.033+0000 7f49427fc640 1 -- 172.21.15.46:0/1481768177 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f493c11e8e0 con 0x7f494c0b0a80 2023-10-30T13:49:54.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.033+0000 7f4951ada640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49300664a0 0x7f4930068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.033+0000 7f4951ada640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49300664a0 0x7f4930068960 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4948009a30 tx=0x7f4948008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:54.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.037+0000 7f49427fc640 1 -- 172.21.15.46:0/1481768177 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f494c158e60 con 0x7f494c0b0a80 2023-10-30T13:49:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.229+0000 7f4953d3e640 1 -- 172.21.15.46:0/1481768177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f494c1eb3b0 con 0x7f494c0b0a80 2023-10-30T13:49:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.229+0000 7f49427fc640 1 -- 172.21.15.46:0/1481768177 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f494c1eb3b0 con 0x7f494c0b0a80 2023-10-30T13:49:54.235 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:49:54.235 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6858,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:51.253577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:54.235 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 116 2023-10-30T13:49:54.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.233+0000 7f4953d3e640 1 -- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49300664a0 msgr2=0x7f4930068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:54.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.233+0000 7f4953d3e640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49300664a0 0x7f4930068960 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4948009a30 tx=0x7f4948008040 comp rx=0 tx=0).stop 2023-10-30T13:49:54.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.233+0000 7f4953d3e640 1 -- 172.21.15.46:0/1481768177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f494c0b0a80 msgr2=0x7f494c1508e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:54.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.233+0000 7f4953d3e640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f494c0b0a80 0x7f494c1508e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f493c001750 tx=0x7f4938005800 comp rx=0 tx=0).stop 2023-10-30T13:49:54.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.233+0000 7f4953d3e640 1 -- 172.21.15.46:0/1481768177 shutdown_connections 2023-10-30T13:49:54.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.233+0000 7f4953d3e640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49300664a0 0x7f4930068960 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.233+0000 7f4953d3e640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f494c159140 0x7f494c1595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.233+0000 7f4953d3e640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f494c0b0a80 0x7f494c1508e0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.233+0000 7f4953d3e640 1 --2- 172.21.15.46:0/1481768177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49380047d0 0x7f494c1eae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.233+0000 7f4953d3e640 1 -- 172.21.15.46:0/1481768177 >> 172.21.15.46:0/1481768177 conn(0x7f494c0c7910 msgr2=0x7f494c0c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:54.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.233+0000 7f4953d3e640 1 -- 172.21.15.46:0/1481768177 shutdown_connections 2023-10-30T13:49:54.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:54.233+0000 7f4953d3e640 1 -- 172.21.15.46:0/1481768177 wait complete. 2023-10-30T13:49:54.256 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:49:56.258 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:56.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.421+0000 7f213926b640 1 Processor -- start 2023-10-30T13:49:56.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.421+0000 7f213926b640 1 -- start start 2023-10-30T13:49:56.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f213926b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21341538a0 0x7f213415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:56.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f213926b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f21340cf160 0x7f21340cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f213926b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21340cb800 con 0x7f21340ca0e0 2023-10-30T13:49:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f213926b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213415dc30 con 0x7f21340cf160 2023-10-30T13:49:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f213926b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21340cb330 con 0x7f21341538a0 2023-10-30T13:49:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f2132ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21341538a0 0x7f213415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f21337fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f21340cf160 0x7f21340cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:56.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f21337fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f21340cf160 0x7f21340cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:48730/0 (socket says 172.21.15.46:48730) 2023-10-30T13:49:56.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f2132ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21341538a0 0x7f213415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:38318/0 (socket says 172.21.15.46:38318) 2023-10-30T13:49:56.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f21337fe640 1 -- 172.21.15.46:0/3281027388 learned_addr learned my addr 172.21.15.46:0/3281027388 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:49:56.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f21327fc640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f21340ca0e0 0x7f213415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:55690/0 (socket says 172.21.15.46:55690) 2023-10-30T13:49:56.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f21337fe640 1 -- 172.21.15.46:0/3281027388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21341538a0 msgr2=0x7f213415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:56.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f21337fe640 1 --2- 172.21.15.46:0/3281027388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21341538a0 0x7f213415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:56.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f21337fe640 1 -- 172.21.15.46:0/3281027388 >> v1:172.21.15.154:6789/0 conn(0x7f21340ca0e0 legacy=0x7f213415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:56.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f21337fe640 1 -- 172.21.15.46:0/3281027388 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21340cad70 con 0x7f21340cf160 2023-10-30T13:49:56.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.425+0000 7f2132ffd640 1 --2- 172.21.15.46:0/3281027388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21341538a0 0x7f213415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:49:56.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.429+0000 7f21337fe640 1 --2- 172.21.15.46:0/3281027388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21340cf160 0x7f21340cf590 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f212c009e50 tx=0x7f212c01f1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6b941049dd5e803 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:56.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.429+0000 7f2131ffb640 1 -- 172.21.15.46:0/3281027388 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c0af400 con 0x7f21340cf160 2023-10-30T13:49:56.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.429+0000 7f2131ffb640 1 -- 172.21.15.46:0/3281027388 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c0ab050 con 0x7f21340cf160 2023-10-30T13:49:56.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.429+0000 7f2131ffb640 1 -- 172.21.15.46:0/3281027388 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c0af780 con 0x7f21340cf160 2023-10-30T13:49:56.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.429+0000 7f213926b640 1 -- 172.21.15.46:0/3281027388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21340cf160 msgr2=0x7f21340cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:56.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.429+0000 7f213926b640 1 --2- 172.21.15.46:0/3281027388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21340cf160 0x7f21340cf590 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f212c009e50 tx=0x7f212c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:49:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.429+0000 7f213926b640 1 -- 172.21.15.46:0/3281027388 shutdown_connections 2023-10-30T13:49:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.429+0000 7f213926b640 1 --2- 172.21.15.46:0/3281027388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21340cf160 0x7f21340cf590 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.429+0000 7f213926b640 1 --2- 172.21.15.46:0/3281027388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21341538a0 0x7f213415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.429+0000 7f213926b640 1 -- 172.21.15.46:0/3281027388 >> 172.21.15.46:0/3281027388 conn(0x7f213406bfe0 msgr2=0x7f213406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:56.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.429+0000 7f213926b640 1 -- 172.21.15.46:0/3281027388 shutdown_connections 2023-10-30T13:49:56.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.429+0000 7f213926b640 1 -- 172.21.15.46:0/3281027388 wait complete. 2023-10-30T13:49:56.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f213926b640 1 Processor -- start 2023-10-30T13:49:56.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f213926b640 1 -- start start 2023-10-30T13:49:56.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f213926b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21340ca0e0 0x7f21340ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:56.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f213926b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21340cf160 0x7f213415c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:56.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f213926b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21341538a0 0x7f2134144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:56.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f213926b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21341442e0 con 0x7f21341538a0 2023-10-30T13:49:56.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f213926b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2134143fe0 con 0x7f21340cf160 2023-10-30T13:49:56.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f213926b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213414c570 con 0x7f21340ca0e0 2023-10-30T13:49:56.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f21337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21341538a0 0x7f2134144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:56.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f21337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21341538a0 0x7f2134144770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42102/0 (socket says 172.21.15.46:42102) 2023-10-30T13:49:56.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f21337fe640 1 -- 172.21.15.46:0/3330281451 learned_addr learned my addr 172.21.15.46:0/3330281451 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:49:56.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f2132ffd640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21340ca0e0 0x7f21340ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:56.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f21327fc640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21340cf160 0x7f213415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:56.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f21337fe640 1 -- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21340ca0e0 msgr2=0x7f21340ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f21337fe640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21340ca0e0 0x7f21340ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f21337fe640 1 -- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21340cf160 msgr2=0x7f213415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f21337fe640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21340cf160 0x7f213415c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f21337fe640 1 -- 172.21.15.46:0/3330281451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2124009420 con 0x7f21341538a0 2023-10-30T13:49:56.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f21337fe640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21341538a0 0x7f2134144770 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f212c0b2040 tx=0x7f212c01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:56.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f211b7fe640 1 -- 172.21.15.46:0/3330281451 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c0afc80 con 0x7f21341538a0 2023-10-30T13:49:56.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f211b7fe640 1 -- 172.21.15.46:0/3330281451 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c0afde0 con 0x7f21341538a0 2023-10-30T13:49:56.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f211b7fe640 1 -- 172.21.15.46:0/3330281451 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c0ba680 con 0x7f21341538a0 2023-10-30T13:49:56.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f213926b640 1 -- 172.21.15.46:0/3330281451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f212c009b30 con 0x7f21341538a0 2023-10-30T13:49:56.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.433+0000 7f213926b640 1 -- 172.21.15.46:0/3330281451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213414cb30 con 0x7f21341538a0 2023-10-30T13:49:56.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.437+0000 7f213926b640 1 -- 172.21.15.46:0/3330281451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21340b2a50 con 0x7f21341538a0 2023-10-30T13:49:56.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.437+0000 7f211b7fe640 1 -- 172.21.15.46:0/3330281451 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f212c0b6040 con 0x7f21341538a0 2023-10-30T13:49:56.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.437+0000 7f211b7fe640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21200664a0 0x7f2120068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:56.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.437+0000 7f211b7fe640 1 -- 172.21.15.46:0/3330281451 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f212c122260 con 0x7f21341538a0 2023-10-30T13:49:56.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.441+0000 7f2132ffd640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21200664a0 0x7f2120068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:56.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.441+0000 7f2132ffd640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21200664a0 0x7f2120068960 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2124009c50 tx=0x7f2124009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:56.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.445+0000 7f211b7fe640 1 -- 172.21.15.46:0/3330281451 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f212c0ef420 con 0x7f21341538a0 2023-10-30T13:49:56.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.645+0000 7f213926b640 1 -- 172.21.15.46:0/3330281451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2134144490 con 0x7f21341538a0 2023-10-30T13:49:56.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.645+0000 7f211b7fe640 1 -- 172.21.15.46:0/3330281451 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f2134144490 con 0x7f21341538a0 2023-10-30T13:49:56.651 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:49:56.651 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6858,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:51.253577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:56.652 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 116 2023-10-30T13:49:56.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.649+0000 7f213926b640 1 -- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21200664a0 msgr2=0x7f2120068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:56.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.649+0000 7f213926b640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21200664a0 0x7f2120068960 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2124009c50 tx=0x7f2124009190 comp rx=0 tx=0).stop 2023-10-30T13:49:56.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.649+0000 7f213926b640 1 -- 172.21.15.46:0/3330281451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21341538a0 msgr2=0x7f2134144770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:56.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.649+0000 7f213926b640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21341538a0 0x7f2134144770 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f212c0b2040 tx=0x7f212c01bf90 comp rx=0 tx=0).stop 2023-10-30T13:49:56.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.649+0000 7f213926b640 1 -- 172.21.15.46:0/3330281451 shutdown_connections 2023-10-30T13:49:56.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.649+0000 7f213926b640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21200664a0 0x7f2120068960 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:56.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.649+0000 7f213926b640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21341538a0 0x7f2134144770 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.649+0000 7f213926b640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21340cf160 0x7f213415c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.649+0000 7f213926b640 1 --2- 172.21.15.46:0/3330281451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21340ca0e0 0x7f21340ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.649+0000 7f213926b640 1 -- 172.21.15.46:0/3330281451 >> 172.21.15.46:0/3330281451 conn(0x7f213406bfe0 msgr2=0x7f2134151320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.649+0000 7f213926b640 1 -- 172.21.15.46:0/3330281451 shutdown_connections 2023-10-30T13:49:56.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:56.649+0000 7f213926b640 1 -- 172.21.15.46:0/3330281451 wait complete. 2023-10-30T13:49:56.673 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:49:58.674 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:58.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b379c5640 1 Processor -- start 2023-10-30T13:49:58.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b379c5640 1 -- start start 2023-10-30T13:49:58.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b379c5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7b30071110 0x7f7b3006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:58.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b379c5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b30066cf0 con 0x7f7b300c6bd0 2023-10-30T13:49:58.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b379c5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b300c70d0 con 0x7f7b3006f7a0 2023-10-30T13:49:58.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b379c5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b300700c0 con 0x7f7b30071110 2023-10-30T13:49:58.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b34f60640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7b300c6bd0 0x7f7b30158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57720/0 (socket says 172.21.15.46:57720) 2023-10-30T13:49:58.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b34f60640 1 -- 172.21.15.46:0/2750982424 learned_addr learned my addr 172.21.15.46:0/2750982424 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:49:58.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b35f62640 1 --2- 172.21.15.46:0/2750982424 >> v2:172.21.15.154:3300/0 conn(0x7f7b30071110 0x7f7b3006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:58.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b27fff640 1 -- 172.21.15.46:0/2750982424 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484862017 0 0) 0x7f7b30066cf0 con 0x7f7b300c6bd0 2023-10-30T13:49:58.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b27fff640 1 -- 172.21.15.46:0/2750982424 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b1c003530 con 0x7f7b300c6bd0 2023-10-30T13:49:58.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b35f62640 1 -- 172.21.15.46:0/2750982424 >> v1:172.21.15.154:6789/0 conn(0x7f7b3006f7a0 legacy=0x7f7b30069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:58.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b35f62640 1 -- 172.21.15.46:0/2750982424 >> v1:172.21.15.46:6789/0 conn(0x7f7b300c6bd0 legacy=0x7f7b30158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:58.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.825+0000 7f7b35f62640 1 -- 172.21.15.46:0/2750982424 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b300c7390 con 0x7f7b30071110 2023-10-30T13:49:58.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.829+0000 7f7b35f62640 1 --2- 172.21.15.46:0/2750982424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b30071110 0x7f7b3006fb80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009d90 tx=0x7f7b2801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da695c280ca08c16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:58.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.829+0000 7f7b27fff640 1 -- 172.21.15.46:0/2750982424 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b280af600 con 0x7f7b30071110 2023-10-30T13:49:58.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.829+0000 7f7b27fff640 1 -- 172.21.15.46:0/2750982424 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b280ab050 con 0x7f7b30071110 2023-10-30T13:49:58.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.829+0000 7f7b27fff640 1 -- 172.21.15.46:0/2750982424 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b280af980 con 0x7f7b30071110 2023-10-30T13:49:58.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.829+0000 7f7b379c5640 1 -- 172.21.15.46:0/2750982424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b30071110 msgr2=0x7f7b3006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:58.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.829+0000 7f7b379c5640 1 --2- 172.21.15.46:0/2750982424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b30071110 0x7f7b3006fb80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009d90 tx=0x7f7b2801f130 comp rx=0 tx=0).stop 2023-10-30T13:49:58.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.829+0000 7f7b379c5640 1 -- 172.21.15.46:0/2750982424 shutdown_connections 2023-10-30T13:49:58.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.829+0000 7f7b379c5640 1 --2- 172.21.15.46:0/2750982424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b30071110 0x7f7b3006fb80 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:58.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.829+0000 7f7b379c5640 1 -- 172.21.15.46:0/2750982424 >> 172.21.15.46:0/2750982424 conn(0x7f7b300c7910 msgr2=0x7f7b300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:58.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.829+0000 7f7b379c5640 1 -- 172.21.15.46:0/2750982424 shutdown_connections 2023-10-30T13:49:58.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.829+0000 7f7b379c5640 1 -- 172.21.15.46:0/2750982424 wait complete. 2023-10-30T13:49:58.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b379c5640 1 Processor -- start 2023-10-30T13:49:58.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b379c5640 1 -- start start 2023-10-30T13:49:58.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b379c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3006f7a0 0x7f7b301efde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:58.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b379c5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b30071110 0x7f7b301f0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:58.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b379c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b300c6bd0 0x7f7b30157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:58.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b379c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3014dd40 con 0x7f7b3006f7a0 2023-10-30T13:49:58.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b379c5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3014f3d0 con 0x7f7b30071110 2023-10-30T13:49:58.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b379c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b300c9020 con 0x7f7b300c6bd0 2023-10-30T13:49:58.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b35761640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3006f7a0 0x7f7b301efde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:58.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b35761640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3006f7a0 0x7f7b301efde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42104/0 (socket says 172.21.15.46:42104) 2023-10-30T13:49:58.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b35761640 1 -- 172.21.15.46:0/1754434495 learned_addr learned my addr 172.21.15.46:0/1754434495 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:49:58.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b34f60640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b30071110 0x7f7b301f0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:58.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b35f62640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b300c6bd0 0x7f7b30157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:58.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b35761640 1 -- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b300c6bd0 msgr2=0x7f7b30157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:58.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b35761640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b300c6bd0 0x7f7b30157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:58.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b35761640 1 -- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b30071110 msgr2=0x7f7b301f0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:58.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b35761640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b30071110 0x7f7b301f0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:58.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b35761640 1 -- 172.21.15.46:0/1754434495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b28009a70 con 0x7f7b3006f7a0 2023-10-30T13:49:58.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b35761640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3006f7a0 0x7f7b301efde0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00c3c0 tx=0x7f7b2c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:58.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b25ffb640 1 -- 172.21.15.46:0/1754434495 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c0052d0 con 0x7f7b3006f7a0 2023-10-30T13:49:58.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b25ffb640 1 -- 172.21.15.46:0/1754434495 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2c005430 con 0x7f7b3006f7a0 2023-10-30T13:49:58.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b25ffb640 1 -- 172.21.15.46:0/1754434495 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c003840 con 0x7f7b3006f7a0 2023-10-30T13:49:58.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.833+0000 7f7b379c5640 1 -- 172.21.15.46:0/1754434495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3014da80 con 0x7f7b3006f7a0 2023-10-30T13:49:58.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.837+0000 7f7b379c5640 1 -- 172.21.15.46:0/1754434495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3014ed10 con 0x7f7b3006f7a0 2023-10-30T13:49:58.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.837+0000 7f7b379c5640 1 -- 172.21.15.46:0/1754434495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3014f510 con 0x7f7b3006f7a0 2023-10-30T13:49:58.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.837+0000 7f7b25ffb640 1 -- 172.21.15.46:0/1754434495 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7b2c0039a0 con 0x7f7b3006f7a0 2023-10-30T13:49:58.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.837+0000 7f7b25ffb640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b14066520 0x7f7b140689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:58.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.837+0000 7f7b25ffb640 1 -- 172.21.15.46:0/1754434495 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f7b2c083870 con 0x7f7b3006f7a0 2023-10-30T13:49:58.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.837+0000 7f7b34f60640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b14066520 0x7f7b140689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:58.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.841+0000 7f7b34f60640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b14066520 0x7f7b140689e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7b3006fe10 tx=0x7f7b20009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:58.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:58.841+0000 7f7b25ffb640 1 -- 172.21.15.46:0/1754434495 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7b3014f510 con 0x7f7b3006f7a0 2023-10-30T13:49:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 -- 172.21.15.46:0/1754434495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b3004e170 con 0x7f7b3006f7a0 2023-10-30T13:49:59.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b25ffb640 1 -- 172.21.15.46:0/1754434495 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f7b2c050830 con 0x7f7b3006f7a0 2023-10-30T13:49:59.037 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:49:59.037 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6858,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:51.253577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:59.038 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 116 2023-10-30T13:49:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 -- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b14066520 msgr2=0x7f7b140689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:59.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b14066520 0x7f7b140689e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7b3006fe10 tx=0x7f7b20009040 comp rx=0 tx=0).stop 2023-10-30T13:49:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 -- 172.21.15.46:0/1754434495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3006f7a0 msgr2=0x7f7b301efde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3006f7a0 0x7f7b301efde0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00c3c0 tx=0x7f7b2c00c880 comp rx=0 tx=0).stop 2023-10-30T13:49:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 -- 172.21.15.46:0/1754434495 shutdown_connections 2023-10-30T13:49:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b14066520 0x7f7b140689e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b300c6bd0 0x7f7b30157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b30071110 0x7f7b301f0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 --2- 172.21.15.46:0/1754434495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3006f7a0 0x7f7b301efde0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 -- 172.21.15.46:0/1754434495 >> 172.21.15.46:0/1754434495 conn(0x7f7b300c7910 msgr2=0x7f7b300c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 -- 172.21.15.46:0/1754434495 shutdown_connections 2023-10-30T13:49:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:49:59.033+0000 7f7b379c5640 1 -- 172.21.15.46:0/1754434495 wait complete. 2023-10-30T13:49:59.059 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:50:01.060 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:01.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.213+0000 7f25168ac640 1 Processor -- start 2023-10-30T13:50:01.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.213+0000 7f25168ac640 1 -- start start 2023-10-30T13:50:01.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25168ac640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25101538f0 0x7f2510159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:01.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25168ac640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f251015cd50 0x7f25100c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:01.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25168ac640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251015d300 con 0x7f25100cefa0 2023-10-30T13:50:01.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25168ac640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251015d5e0 con 0x7f25101538f0 2023-10-30T13:50:01.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25168ac640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251015d8c0 con 0x7f251015cd50 2023-10-30T13:50:01.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25158aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25101538f0 0x7f2510159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:01.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25160ab640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f25100cefa0 0x7f25100cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57736/0 (socket says 172.21.15.46:57736) 2023-10-30T13:50:01.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25160ab640 1 -- 172.21.15.46:0/5289185 learned_addr learned my addr 172.21.15.46:0/5289185 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:50:01.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25150a9640 1 --2- 172.21.15.46:0/5289185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f251015cd50 0x7f25100c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:01.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25158aa640 1 -- 172.21.15.46:0/5289185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f251015cd50 msgr2=0x7f25100c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:01.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25158aa640 1 --2- 172.21.15.46:0/5289185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f251015cd50 0x7f25100c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:01.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25158aa640 1 -- 172.21.15.46:0/5289185 >> v1:172.21.15.46:6789/0 conn(0x7f25100cefa0 legacy=0x7f25100cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:01.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25148a8640 1 -- 172.21.15.46:0/5289185 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094141550 0 0) 0x7f251015d300 con 0x7f25100cefa0 2023-10-30T13:50:01.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.217+0000 7f25158aa640 1 -- 172.21.15.46:0/5289185 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251015dba0 con 0x7f25101538f0 2023-10-30T13:50:01.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25158aa640 1 --2- 172.21.15.46:0/5289185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25101538f0 0x7f2510159fb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f250c009d90 tx=0x7f250c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884b1dbd55f0a6fa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:01.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25148a8640 1 -- 172.21.15.46:0/5289185 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c0af600 con 0x7f25101538f0 2023-10-30T13:50:01.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25148a8640 1 -- 172.21.15.46:0/5289185 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c0ab050 con 0x7f25101538f0 2023-10-30T13:50:01.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25148a8640 1 -- 172.21.15.46:0/5289185 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c0af980 con 0x7f25101538f0 2023-10-30T13:50:01.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 -- 172.21.15.46:0/5289185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25101538f0 msgr2=0x7f2510159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:01.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 --2- 172.21.15.46:0/5289185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25101538f0 0x7f2510159fb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f250c009d90 tx=0x7f250c01f130 comp rx=0 tx=0).stop 2023-10-30T13:50:01.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 -- 172.21.15.46:0/5289185 shutdown_connections 2023-10-30T13:50:01.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 --2- 172.21.15.46:0/5289185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f251015cd50 0x7f25100c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:01.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 --2- 172.21.15.46:0/5289185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25101538f0 0x7f2510159fb0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:01.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 -- 172.21.15.46:0/5289185 >> 172.21.15.46:0/5289185 conn(0x7f251006a640 msgr2=0x7f2510065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:01.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 -- 172.21.15.46:0/5289185 shutdown_connections 2023-10-30T13:50:01.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 -- 172.21.15.46:0/5289185 wait complete. 2023-10-30T13:50:01.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 Processor -- start 2023-10-30T13:50:01.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 -- start start 2023-10-30T13:50:01.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25100cefa0 0x7f25100b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:01.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25101538f0 0x7f25101efb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:01.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f251015cd50 0x7f2510147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101f3480 con 0x7f25101538f0 2023-10-30T13:50:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101ef0d0 con 0x7f251015cd50 2023-10-30T13:50:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.221+0000 7f25168ac640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25101e6410 con 0x7f25100cefa0 2023-10-30T13:50:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25150a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25101538f0 0x7f25101efb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25150a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25101538f0 0x7f25101efb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42126/0 (socket says 172.21.15.46:42126) 2023-10-30T13:50:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25150a9640 1 -- 172.21.15.46:0/3249582896 learned_addr learned my addr 172.21.15.46:0/3249582896 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25158aa640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25100cefa0 0x7f25100b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25160ab640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f251015cd50 0x7f2510147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25150a9640 1 -- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25100cefa0 msgr2=0x7f25100b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25150a9640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25100cefa0 0x7f25100b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25150a9640 1 -- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f251015cd50 msgr2=0x7f2510147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:01.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25150a9640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f251015cd50 0x7f2510147bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:01.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25150a9640 1 -- 172.21.15.46:0/3249582896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c009a70 con 0x7f25101538f0 2023-10-30T13:50:01.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25150a9640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25101538f0 0x7f25101efb90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f250000e870 tx=0x7f250000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:01.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25067fc640 1 -- 172.21.15.46:0/3249582896 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2500009510 con 0x7f25101538f0 2023-10-30T13:50:01.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25067fc640 1 -- 172.21.15.46:0/3249582896 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250000eec0 con 0x7f25101538f0 2023-10-30T13:50:01.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25067fc640 1 -- 172.21.15.46:0/3249582896 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250000cb40 con 0x7f25101538f0 2023-10-30T13:50:01.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25168ac640 1 -- 172.21.15.46:0/3249582896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25101e66f0 con 0x7f25101538f0 2023-10-30T13:50:01.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25168ac640 1 -- 172.21.15.46:0/3249582896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25101ef390 con 0x7f25101538f0 2023-10-30T13:50:01.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25168ac640 1 -- 172.21.15.46:0/3249582896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251004e170 con 0x7f25101538f0 2023-10-30T13:50:01.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.225+0000 7f25067fc640 1 -- 172.21.15.46:0/3249582896 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f250000cca0 con 0x7f25101538f0 2023-10-30T13:50:01.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.229+0000 7f25067fc640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24f40664a0 0x7f24f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:01.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.229+0000 7f25067fc640 1 -- 172.21.15.46:0/3249582896 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f25000843b0 con 0x7f25101538f0 2023-10-30T13:50:01.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.229+0000 7f25158aa640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24f40664a0 0x7f24f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:01.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.229+0000 7f25158aa640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24f40664a0 0x7f24f4068960 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f250c0b1000 tx=0x7f250c0aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:01.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.233+0000 7f25067fc640 1 -- 172.21.15.46:0/3249582896 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2500051430 con 0x7f25101538f0 2023-10-30T13:50:01.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.421+0000 7f25168ac640 1 -- 172.21.15.46:0/3249582896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2510159a60 con 0x7f25101538f0 2023-10-30T13:50:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25067fc640 1 -- 172.21.15.46:0/3249582896 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f2500054a50 con 0x7f25101538f0 2023-10-30T13:50:01.428 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:01.428 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6858,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:51.253577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:01.429 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 116 2023-10-30T13:50:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25168ac640 1 -- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24f40664a0 msgr2=0x7f24f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25168ac640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24f40664a0 0x7f24f4068960 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f250c0b1000 tx=0x7f250c0aa000 comp rx=0 tx=0).stop 2023-10-30T13:50:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25168ac640 1 -- 172.21.15.46:0/3249582896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25101538f0 msgr2=0x7f25101efb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25168ac640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25101538f0 0x7f25101efb90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f250000e870 tx=0x7f250000ec70 comp rx=0 tx=0).stop 2023-10-30T13:50:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25168ac640 1 -- 172.21.15.46:0/3249582896 shutdown_connections 2023-10-30T13:50:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25168ac640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24f40664a0 0x7f24f4068960 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25168ac640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f251015cd50 0x7f2510147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25168ac640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25101538f0 0x7f25101efb90 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25168ac640 1 --2- 172.21.15.46:0/3249582896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25100cefa0 0x7f25100b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25168ac640 1 -- 172.21.15.46:0/3249582896 >> 172.21.15.46:0/3249582896 conn(0x7f251006a640 msgr2=0x7f25100caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25168ac640 1 -- 172.21.15.46:0/3249582896 shutdown_connections 2023-10-30T13:50:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:01.425+0000 7f25168ac640 1 -- 172.21.15.46:0/3249582896 wait complete. 2023-10-30T13:50:01.449 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:50:03.451 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:03.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57eb65640 1 Processor -- start 2023-10-30T13:50:03.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57eb65640 1 -- start start 2023-10-30T13:50:03.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57eb65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5781538a0 0x7ff57815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:03.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57eb65640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff5780cf160 0x7ff5780cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:03.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57eb65640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5780cb800 con 0x7ff5780ca0e0 2023-10-30T13:50:03.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57eb65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57815dc30 con 0x7ff5781538a0 2023-10-30T13:50:03.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57eb65640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5780cb330 con 0x7ff5780cf160 2023-10-30T13:50:03.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57c901640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5781538a0 0x7ff57815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:03.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff577fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff5780ca0e0 0x7ff57815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57746/0 (socket says 172.21.15.46:57746) 2023-10-30T13:50:03.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff577fff640 1 -- 172.21.15.46:0/3304894356 learned_addr learned my addr 172.21.15.46:0/3304894356 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:50:03.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57d102640 1 --2- 172.21.15.46:0/3304894356 >> v2:172.21.15.154:3300/0 conn(0x7ff5780cf160 0x7ff5780cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:03.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57c901640 1 -- 172.21.15.46:0/3304894356 >> v2:172.21.15.154:3300/0 conn(0x7ff5780cf160 msgr2=0x7ff5780cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:03.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57c901640 1 --2- 172.21.15.46:0/3304894356 >> v2:172.21.15.154:3300/0 conn(0x7ff5780cf160 0x7ff5780cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57c901640 1 -- 172.21.15.46:0/3304894356 >> v1:172.21.15.46:6789/0 conn(0x7ff5780ca0e0 legacy=0x7ff57815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:03.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff57c901640 1 -- 172.21.15.46:0/3304894356 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5780cad70 con 0x7ff5781538a0 2023-10-30T13:50:03.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.601+0000 7ff5777fe640 1 -- 172.21.15.46:0/3304894356 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111728577 0 0) 0x7ff5780cb800 con 0x7ff5780ca0e0 2023-10-30T13:50:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.605+0000 7ff57c901640 1 --2- 172.21.15.46:0/3304894356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5781538a0 0x7ff57815d6f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7ff5780b5330 tx=0x7ff56c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260ab148f7d0446c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.605+0000 7ff5777fe640 1 -- 172.21.15.46:0/3304894356 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c0a8950 con 0x7ff5781538a0 2023-10-30T13:50:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.605+0000 7ff5777fe640 1 -- 172.21.15.46:0/3304894356 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c0a8ab0 con 0x7ff5781538a0 2023-10-30T13:50:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.605+0000 7ff5777fe640 1 -- 172.21.15.46:0/3304894356 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c0a8e40 con 0x7ff5781538a0 2023-10-30T13:50:03.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.605+0000 7ff57eb65640 1 -- 172.21.15.46:0/3304894356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5781538a0 msgr2=0x7ff57815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:03.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.605+0000 7ff57eb65640 1 --2- 172.21.15.46:0/3304894356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5781538a0 0x7ff57815d6f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7ff5780b5330 tx=0x7ff56c01ec50 comp rx=0 tx=0).stop 2023-10-30T13:50:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.605+0000 7ff57eb65640 1 -- 172.21.15.46:0/3304894356 shutdown_connections 2023-10-30T13:50:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.605+0000 7ff57eb65640 1 --2- 172.21.15.46:0/3304894356 >> v2:172.21.15.154:3300/0 conn(0x7ff5780cf160 0x7ff5780cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.605+0000 7ff57eb65640 1 --2- 172.21.15.46:0/3304894356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5781538a0 0x7ff57815d6f0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.605+0000 7ff57eb65640 1 -- 172.21.15.46:0/3304894356 >> 172.21.15.46:0/3304894356 conn(0x7ff57806bfe0 msgr2=0x7ff57806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.605+0000 7ff57eb65640 1 -- 172.21.15.46:0/3304894356 shutdown_connections 2023-10-30T13:50:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.605+0000 7ff57eb65640 1 -- 172.21.15.46:0/3304894356 wait complete. 2023-10-30T13:50:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57eb65640 1 Processor -- start 2023-10-30T13:50:03.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57eb65640 1 -- start start 2023-10-30T13:50:03.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57eb65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780ca0e0 0x7ff57815b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57eb65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5780cf160 0x7ff5780b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57eb65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5781538a0 0x7ff5781ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57eb65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57814ecd0 con 0x7ff5780ca0e0 2023-10-30T13:50:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57eb65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57814e750 con 0x7ff5781538a0 2023-10-30T13:50:03.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57eb65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5781f3380 con 0x7ff5780cf160 2023-10-30T13:50:03.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57c901640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780ca0e0 0x7ff57815b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:03.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57c901640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780ca0e0 0x7ff57815b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42152/0 (socket says 172.21.15.46:42152) 2023-10-30T13:50:03.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57c901640 1 -- 172.21.15.46:0/327050835 learned_addr learned my addr 172.21.15.46:0/327050835 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:03.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff577fff640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5780cf160 0x7ff5780b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:03.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57d102640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5781538a0 0x7ff5781ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:03.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57c901640 1 -- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5780cf160 msgr2=0x7ff5780b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:03.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57c901640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5780cf160 0x7ff5780b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57c901640 1 -- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5781538a0 msgr2=0x7ff5781ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:03.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57c901640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5781538a0 0x7ff5781ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57c901640 1 -- 172.21.15.46:0/327050835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c009420 con 0x7ff5780ca0e0 2023-10-30T13:50:03.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff577fff640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5780cf160 0x7ff5780b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:03.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57c901640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780ca0e0 0x7ff57815b060 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0b3040 tx=0x7ff56c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:03.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff5757fa640 1 -- 172.21.15.46:0/327050835 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c0b9630 con 0x7ff5780ca0e0 2023-10-30T13:50:03.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff5757fa640 1 -- 172.21.15.46:0/327050835 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c0b97b0 con 0x7ff5780ca0e0 2023-10-30T13:50:03.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff5757fa640 1 -- 172.21.15.46:0/327050835 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c0b9b60 con 0x7ff5780ca0e0 2023-10-30T13:50:03.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57eb65640 1 -- 172.21.15.46:0/327050835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57814e9f0 con 0x7ff5780ca0e0 2023-10-30T13:50:03.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.609+0000 7ff57eb65640 1 -- 172.21.15.46:0/327050835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5780b1b60 con 0x7ff5780ca0e0 2023-10-30T13:50:03.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.613+0000 7ff57eb65640 1 -- 172.21.15.46:0/327050835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5780b2a50 con 0x7ff5780ca0e0 2023-10-30T13:50:03.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.613+0000 7ff5757fa640 1 -- 172.21.15.46:0/327050835 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff56c0b7040 con 0x7ff5780ca0e0 2023-10-30T13:50:03.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.613+0000 7ff5757fa640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff55c0664a0 0x7ff55c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.613+0000 7ff5757fa640 1 -- 172.21.15.46:0/327050835 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6886+0+0 (secure 0 0 0) 0x7ff56c1223d0 con 0x7ff5780ca0e0 2023-10-30T13:50:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.613+0000 7ff577fff640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff55c0664a0 0x7ff55c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.613+0000 7ff577fff640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff55c0664a0 0x7ff55c068960 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff5781597c0 tx=0x7ff564007cb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:03.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.617+0000 7ff5757fa640 1 -- 172.21.15.46:0/327050835 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff56c0ef430 con 0x7ff5780ca0e0 2023-10-30T13:50:03.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 -- 172.21.15.46:0/327050835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5781ef280 con 0x7ff5780ca0e0 2023-10-30T13:50:03.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff5757fa640 1 -- 172.21.15.46:0/327050835 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+6522 (secure 0 0 0) 0x7ff5781ef280 con 0x7ff5780ca0e0 2023-10-30T13:50:03.813 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:03.813 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6858,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:47:51.253577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6366},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6366":{"gid":6366,"name":"a","rank":1,"incarnation":104,"state":"up:active","state_seq":51,"addr":"172.21.15.46:6833/190134548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":190134548},{"type":"v1","addr":"172.21.15.46:6833","nonce":190134548}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:03.814 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 116 2023-10-30T13:50:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 -- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff55c0664a0 msgr2=0x7ff55c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff55c0664a0 0x7ff55c068960 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff5781597c0 tx=0x7ff564007cb0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 -- 172.21.15.46:0/327050835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780ca0e0 msgr2=0x7ff57815b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:03.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780ca0e0 0x7ff57815b060 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0b3040 tx=0x7ff56c01b9f0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 -- 172.21.15.46:0/327050835 shutdown_connections 2023-10-30T13:50:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff55c0664a0 0x7ff55c068960 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5781538a0 0x7ff5781ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5780cf160 0x7ff5780b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 --2- 172.21.15.46:0/327050835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780ca0e0 0x7ff57815b060 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 -- 172.21.15.46:0/327050835 >> 172.21.15.46:0/327050835 conn(0x7ff57806bfe0 msgr2=0x7ff578065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 -- 172.21.15.46:0/327050835 shutdown_connections 2023-10-30T13:50:03.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:03.809+0000 7ff57eb65640 1 -- 172.21.15.46:0/327050835 wait complete. 2023-10-30T13:50:03.836 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:50:05.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f8912e91640 1 Processor -- start 2023-10-30T13:50:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f8912e91640 1 -- start start 2023-10-30T13:50:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f8912e91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f890c06e470 0x7f890c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:05.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f8912e91640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f890c0c8490 0x7f890c0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:05.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f8912e91640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890c06afa0 con 0x7f890c153d70 2023-10-30T13:50:05.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f8912e91640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890c06d780 con 0x7f890c0c8490 2023-10-30T13:50:05.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f8912e91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c154150 con 0x7f890c06e470 2023-10-30T13:50:05.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f891142e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f890c0c8490 0x7f890c0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:05.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f890bfff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f890c06e470 0x7f890c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:05.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f8910c2d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f890c153d70 0x7f890c0cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:55710/0 (socket says 172.21.15.46:55710) 2023-10-30T13:50:05.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f8910c2d640 1 -- 172.21.15.46:0/1451097924 learned_addr learned my addr 172.21.15.46:0/1451097924 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:50:05.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f891142e640 1 --2- 172.21.15.46:0/1451097924 >> v2:172.21.15.154:3300/0 conn(0x7f890c0c8490 0x7f890c0c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:48786/0 (socket says 172.21.15.46:48786) 2023-10-30T13:50:05.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f891142e640 1 -- 172.21.15.46:0/1451097924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f890c06e470 msgr2=0x7f890c06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:05.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f891142e640 1 --2- 172.21.15.46:0/1451097924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f890c06e470 0x7f890c06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:05.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f891142e640 1 -- 172.21.15.46:0/1451097924 >> v1:172.21.15.154:6789/0 conn(0x7f890c153d70 legacy=0x7f890c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:05.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f891142e640 1 -- 172.21.15.46:0/1451097924 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c06e030 con 0x7f890c0c8490 2023-10-30T13:50:05.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.989+0000 7f890b7fe640 1 -- 172.21.15.46:0/1451097924 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034559782 0 0) 0x7f890c06afa0 con 0x7f890c153d70 2023-10-30T13:50:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.993+0000 7f891142e640 1 --2- 172.21.15.46:0/1451097924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f890c0c8490 0x7f890c0c8cc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8904009d90 tx=0x7f890401c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66d187b87106568b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.993+0000 7f890b7fe640 1 -- 172.21.15.46:0/1451097924 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89040a59e0 con 0x7f890c0c8490 2023-10-30T13:50:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.993+0000 7f890b7fe640 1 -- 172.21.15.46:0/1451097924 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89040a5b40 con 0x7f890c0c8490 2023-10-30T13:50:05.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.993+0000 7f890b7fe640 1 -- 172.21.15.46:0/1451097924 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89040ad440 con 0x7f890c0c8490 2023-10-30T13:50:06.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.993+0000 7f8912e91640 1 -- 172.21.15.46:0/1451097924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f890c0c8490 msgr2=0x7f890c0c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:06.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.993+0000 7f8912e91640 1 --2- 172.21.15.46:0/1451097924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f890c0c8490 0x7f890c0c8cc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8904009d90 tx=0x7f890401c920 comp rx=0 tx=0).stop 2023-10-30T13:50:06.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.993+0000 7f8912e91640 1 -- 172.21.15.46:0/1451097924 shutdown_connections 2023-10-30T13:50:06.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.993+0000 7f8912e91640 1 --2- 172.21.15.46:0/1451097924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f890c0c8490 0x7f890c0c8cc0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.993+0000 7f8912e91640 1 --2- 172.21.15.46:0/1451097924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f890c06e470 0x7f890c06ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.993+0000 7f8912e91640 1 -- 172.21.15.46:0/1451097924 >> 172.21.15.46:0/1451097924 conn(0x7f890c071ea0 msgr2=0x7f890c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:06.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.993+0000 7f8912e91640 1 -- 172.21.15.46:0/1451097924 shutdown_connections 2023-10-30T13:50:06.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.993+0000 7f8912e91640 1 -- 172.21.15.46:0/1451097924 wait complete. 2023-10-30T13:50:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f8912e91640 1 Processor -- start 2023-10-30T13:50:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f8912e91640 1 -- start start 2023-10-30T13:50:06.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f8912e91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f890c06e470 0x7f890c14c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:06.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f8912e91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f890c0c8490 0x7f890c0cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:06.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f8912e91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c153d70 0x7f890c147970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:06.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f8912e91640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c0adca0 con 0x7f890c153d70 2023-10-30T13:50:06.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f8912e91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c14cab0 con 0x7f890c06e470 2023-10-30T13:50:06.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f8912e91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c14ac40 con 0x7f890c0c8490 2023-10-30T13:50:06.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f891142e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c153d70 0x7f890c147970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:06.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f891142e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c153d70 0x7f890c147970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42166/0 (socket says 172.21.15.46:42166) 2023-10-30T13:50:06.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f891142e640 1 -- 172.21.15.46:0/413402052 learned_addr learned my addr 172.21.15.46:0/413402052 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:06.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f8910c2d640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f890c06e470 0x7f890c14c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:06.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f890bfff640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f890c0c8490 0x7f890c0cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:06.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f891142e640 1 -- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f890c0c8490 msgr2=0x7f890c0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:06.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f891142e640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f890c0c8490 0x7f890c0cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f891142e640 1 -- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f890c06e470 msgr2=0x7f890c14c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:06.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f891142e640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f890c06e470 0x7f890c14c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f891142e640 1 -- 172.21.15.46:0/413402052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8904009a70 con 0x7f890c153d70 2023-10-30T13:50:06.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f891142e640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c153d70 0x7f890c147970 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f890401a550 tx=0x7f89040196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:06.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f89097fa640 1 -- 172.21.15.46:0/413402052 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8904009450 con 0x7f890c153d70 2023-10-30T13:50:06.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f89097fa640 1 -- 172.21.15.46:0/413402052 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89040adb70 con 0x7f890c153d70 2023-10-30T13:50:06.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f8912e91640 1 -- 172.21.15.46:0/413402052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890c14af20 con 0x7f890c153d70 2023-10-30T13:50:06.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:05.997+0000 7f89097fa640 1 -- 172.21.15.46:0/413402052 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89040b8420 con 0x7f890c153d70 2023-10-30T13:50:06.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.001+0000 7f8912e91640 1 -- 172.21.15.46:0/413402052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890c14b4e0 con 0x7f890c153d70 2023-10-30T13:50:06.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.001+0000 7f8912e91640 1 -- 172.21.15.46:0/413402052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890c04e170 con 0x7f890c153d70 2023-10-30T13:50:06.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.001+0000 7f89097fa640 1 -- 172.21.15.46:0/413402052 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f89040b8580 con 0x7f890c153d70 2023-10-30T13:50:06.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.001+0000 7f89097fa640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88f00664d0 0x7f88f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:06.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.001+0000 7f89097fa640 1 -- 172.21.15.46:0/413402052 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f890411f950 con 0x7f890c153d70 2023-10-30T13:50:06.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.001+0000 7f8910c2d640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88f00664d0 0x7f88f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:06.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.005+0000 7f8910c2d640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88f00664d0 0x7f88f0068990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8900000f70 tx=0x7f8900007df0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:06.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.005+0000 7f89097fa640 1 -- 172.21.15.46:0/413402052 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f89040eca10 con 0x7f890c153d70 2023-10-30T13:50:06.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.197+0000 7f8912e91640 1 -- 172.21.15.46:0/413402052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f890c06c2a0 con 0x7f890c153d70 2023-10-30T13:50:06.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.197+0000 7f89097fa640 1 -- 172.21.15.46:0/413402052 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f89040f0030 con 0x7f890c153d70 2023-10-30T13:50:06.202 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:06.203 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:04.191491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:06.203 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 117 2023-10-30T13:50:06.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.201+0000 7f8912e91640 1 -- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88f00664d0 msgr2=0x7f88f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:06.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.201+0000 7f8912e91640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88f00664d0 0x7f88f0068990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8900000f70 tx=0x7f8900007df0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.201+0000 7f8912e91640 1 -- 172.21.15.46:0/413402052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c153d70 msgr2=0x7f890c147970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:06.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.201+0000 7f8912e91640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c153d70 0x7f890c147970 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f890401a550 tx=0x7f89040196c0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.201+0000 7f8912e91640 1 -- 172.21.15.46:0/413402052 shutdown_connections 2023-10-30T13:50:06.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.201+0000 7f8912e91640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f88f00664d0 0x7f88f0068990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.201+0000 7f8912e91640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c153d70 0x7f890c147970 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.201+0000 7f8912e91640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f890c0c8490 0x7f890c0cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.201+0000 7f8912e91640 1 --2- 172.21.15.46:0/413402052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f890c06e470 0x7f890c14c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.201+0000 7f8912e91640 1 -- 172.21.15.46:0/413402052 >> 172.21.15.46:0/413402052 conn(0x7f890c071ea0 msgr2=0x7f890c151500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:06.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.201+0000 7f8912e91640 1 -- 172.21.15.46:0/413402052 shutdown_connections 2023-10-30T13:50:06.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.201+0000 7f8912e91640 1 -- 172.21.15.46:0/413402052 wait complete. 2023-10-30T13:50:06.224 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:50:06.224 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:50:06.225 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:06.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e0349c640 1 Processor -- start 2023-10-30T13:50:06.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e0349c640 1 -- start start 2023-10-30T13:50:06.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e0349c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3dfc070740 0x7f3dfc0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:06.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e0349c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3dfc15e700 0x7f3dfc06a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:06.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e0349c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dfc0c5810 con 0x7f3dfc1580e0 2023-10-30T13:50:06.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e0349c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dfc15eb70 con 0x7f3dfc15e700 2023-10-30T13:50:06.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e0349c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dfc0c6bb0 con 0x7f3dfc070740 2023-10-30T13:50:06.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e01a39640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3dfc15e700 0x7f3dfc06a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:06.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e01a39640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3dfc15e700 0x7f3dfc06a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42170/0 (socket says 172.21.15.46:42170) 2023-10-30T13:50:06.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e01a39640 1 -- 172.21.15.46:0/3782714070 learned_addr learned my addr 172.21.15.46:0/3782714070 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:06.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e00a37640 1 --2- 172.21.15.46:0/3782714070 >> v2:172.21.15.154:3300/0 conn(0x7f3dfc070740 0x7f3dfc0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:06.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e01a39640 1 -- 172.21.15.46:0/3782714070 >> v2:172.21.15.154:3300/0 conn(0x7f3dfc070740 msgr2=0x7f3dfc0c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:06.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e01a39640 1 --2- 172.21.15.46:0/3782714070 >> v2:172.21.15.154:3300/0 conn(0x7f3dfc070740 0x7f3dfc0c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e01a39640 1 -- 172.21.15.46:0/3782714070 >> v1:172.21.15.46:6789/0 conn(0x7f3dfc1580e0 legacy=0x7f3dfc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:06.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3e01a39640 1 -- 172.21.15.46:0/3782714070 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc0c6e70 con 0x7f3dfc15e700 2023-10-30T13:50:06.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.373+0000 7f3df3fff640 1 -- 172.21.15.46:0/3782714070 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642532322 0 0) 0x7f3dfc0c5810 con 0x7f3dfc1580e0 2023-10-30T13:50:06.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.377+0000 7f3e01a39640 1 --2- 172.21.15.46:0/3782714070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dfc15e700 0x7f3dfc06a5a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc06b1a0 tx=0x7f3df401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e4cfa07e6693d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:06.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.377+0000 7f3df3fff640 1 -- 172.21.15.46:0/3782714070 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df40a5780 con 0x7f3dfc15e700 2023-10-30T13:50:06.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.377+0000 7f3df3fff640 1 -- 172.21.15.46:0/3782714070 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df40a58e0 con 0x7f3dfc15e700 2023-10-30T13:50:06.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.377+0000 7f3df3fff640 1 -- 172.21.15.46:0/3782714070 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df40a5c70 con 0x7f3dfc15e700 2023-10-30T13:50:06.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.377+0000 7f3e0349c640 1 -- 172.21.15.46:0/3782714070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dfc15e700 msgr2=0x7f3dfc06a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:06.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.377+0000 7f3e0349c640 1 --2- 172.21.15.46:0/3782714070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dfc15e700 0x7f3dfc06a5a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc06b1a0 tx=0x7f3df401c9d0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.377+0000 7f3e0349c640 1 -- 172.21.15.46:0/3782714070 shutdown_connections 2023-10-30T13:50:06.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.377+0000 7f3e0349c640 1 --2- 172.21.15.46:0/3782714070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dfc15e700 0x7f3dfc06a5a0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.377+0000 7f3e0349c640 1 --2- 172.21.15.46:0/3782714070 >> v2:172.21.15.154:3300/0 conn(0x7f3dfc070740 0x7f3dfc0c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.377+0000 7f3e0349c640 1 -- 172.21.15.46:0/3782714070 >> 172.21.15.46:0/3782714070 conn(0x7f3dfc0c7910 msgr2=0x7f3dfc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:06.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.377+0000 7f3e0349c640 1 -- 172.21.15.46:0/3782714070 shutdown_connections 2023-10-30T13:50:06.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.377+0000 7f3e0349c640 1 -- 172.21.15.46:0/3782714070 wait complete. 2023-10-30T13:50:06.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e0349c640 1 Processor -- start 2023-10-30T13:50:06.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e0349c640 1 -- start start 2023-10-30T13:50:06.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e0349c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dfc070740 0x7f3dfc0c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:06.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e0349c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dfc1580e0 0x7f3dfc0ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:06.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e0349c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3dfc15e700 0x7f3dfc143d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:06.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e0349c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc142530 con 0x7f3dfc070740 2023-10-30T13:50:06.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e0349c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0c9b40 con 0x7f3dfc1580e0 2023-10-30T13:50:06.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e0349c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc14a1a0 con 0x7f3dfc15e700 2023-10-30T13:50:06.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e01238640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dfc070740 0x7f3dfc0c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:06.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e01238640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dfc070740 0x7f3dfc0c9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42184/0 (socket says 172.21.15.46:42184) 2023-10-30T13:50:06.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e01238640 1 -- 172.21.15.46:0/849847735 learned_addr learned my addr 172.21.15.46:0/849847735 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:06.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e00a37640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dfc1580e0 0x7f3dfc0ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:06.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e01a39640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3dfc15e700 0x7f3dfc143d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:06.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e01238640 1 -- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3dfc15e700 msgr2=0x7f3dfc143d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:06.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e01238640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3dfc15e700 0x7f3dfc143d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e01238640 1 -- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dfc1580e0 msgr2=0x7f3dfc0ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:06.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e01238640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dfc1580e0 0x7f3dfc0ca530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e01238640 1 -- 172.21.15.46:0/849847735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df4009b30 con 0x7f3dfc070740 2023-10-30T13:50:06.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e01a39640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3dfc15e700 0x7f3dfc143d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:06.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e01238640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dfc070740 0x7f3dfc0c9ff0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f3df800c4d0 tx=0x7f3df800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:06.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3df1ffb640 1 -- 172.21.15.46:0/849847735 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df8005480 con 0x7f3dfc070740 2023-10-30T13:50:06.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3df1ffb640 1 -- 172.21.15.46:0/849847735 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df8003600 con 0x7f3dfc070740 2023-10-30T13:50:06.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3df1ffb640 1 -- 172.21.15.46:0/849847735 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df80039b0 con 0x7f3dfc070740 2023-10-30T13:50:06.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e0349c640 1 -- 172.21.15.46:0/849847735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dfc148570 con 0x7f3dfc070740 2023-10-30T13:50:06.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e0349c640 1 -- 172.21.15.46:0/849847735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dfc148b30 con 0x7f3dfc070740 2023-10-30T13:50:06.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.381+0000 7f3e0349c640 1 -- 172.21.15.46:0/849847735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dfc04e170 con 0x7f3dfc070740 2023-10-30T13:50:06.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.385+0000 7f3df1ffb640 1 -- 172.21.15.46:0/849847735 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3df8003b10 con 0x7f3dfc070740 2023-10-30T13:50:06.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.385+0000 7f3df1ffb640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3de00664a0 0x7f3de0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:06.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.385+0000 7f3df1ffb640 1 -- 172.21.15.46:0/849847735 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f3df8083a20 con 0x7f3dfc070740 2023-10-30T13:50:06.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.385+0000 7f3e00a37640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3de00664a0 0x7f3de0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:06.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.385+0000 7f3e00a37640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3de00664a0 0x7f3de0068960 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc141560 tx=0x7f3dec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:06.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.389+0000 7f3df1ffb640 1 -- 172.21.15.46:0/849847735 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3df80509f0 con 0x7f3dfc070740 2023-10-30T13:50:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 -- 172.21.15.46:0/849847735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3dfc1683e0 con 0x7f3dfc070740 2023-10-30T13:50:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3df1ffb640 1 -- 172.21.15.46:0/849847735 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f3df8017070 con 0x7f3dfc070740 2023-10-30T13:50:06.585 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:06.585 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:04.191491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 117 2023-10-30T13:50:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 -- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3de00664a0 msgr2=0x7f3de0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3de00664a0 0x7f3de0068960 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc141560 tx=0x7f3dec008040 comp rx=0 tx=0).stop 2023-10-30T13:50:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 -- 172.21.15.46:0/849847735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dfc070740 msgr2=0x7f3dfc0c9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:06.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dfc070740 0x7f3dfc0c9ff0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f3df800c4d0 tx=0x7f3df800c990 comp rx=0 tx=0).stop 2023-10-30T13:50:06.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 -- 172.21.15.46:0/849847735 shutdown_connections 2023-10-30T13:50:06.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3de00664a0 0x7f3de0068960 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3dfc15e700 0x7f3dfc143d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dfc1580e0 0x7f3dfc0ca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 --2- 172.21.15.46:0/849847735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3dfc070740 0x7f3dfc0c9ff0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:06.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 -- 172.21.15.46:0/849847735 >> 172.21.15.46:0/849847735 conn(0x7f3dfc0c7910 msgr2=0x7f3dfc1561a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:06.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 -- 172.21.15.46:0/849847735 shutdown_connections 2023-10-30T13:50:06.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:06.581+0000 7f3e0349c640 1 -- 172.21.15.46:0/849847735 wait complete. 2023-10-30T13:50:06.607 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 117, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 117, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6702": { "addr": "172.21.15.46:6836/2197729559", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 2197729559, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 2197729559, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6702, "incarnation": 111, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 43 }, "gid_6858": { "addr": "172.21.15.46:6837/2572113428", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2572113428, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2572113428, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6858, "incarnation": 117, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:50:04.191491+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6702, "mds_1": 6858 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:50:08.608 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:08.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0f151d640 1 Processor -- start 2023-10-30T13:50:08.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0f151d640 1 -- start start 2023-10-30T13:50:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0f151d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0ec1538a0 0x7fd0ec15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0f151d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0ec0ca0e0 0x7fd0ec15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0f151d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd0ec0cf160 0x7fd0ec0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0f151d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ec0cb800 con 0x7fd0ec1538a0 2023-10-30T13:50:08.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0f151d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ec15dc30 con 0x7fd0ec0cf160 2023-10-30T13:50:08.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0f151d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0cb330 con 0x7fd0ec0ca0e0 2023-10-30T13:50:08.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0eaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0ec1538a0 0x7fd0ec15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:08.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0eaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0ec1538a0 0x7fd0ec15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55420/0 (socket says 172.21.15.46:55420) 2023-10-30T13:50:08.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0eaffd640 1 -- 172.21.15.46:0/2642274721 learned_addr learned my addr 172.21.15.46:0/2642274721 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:08.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0eb7fe640 1 --2- 172.21.15.46:0/2642274721 >> v2:172.21.15.154:3300/0 conn(0x7fd0ec0cf160 0x7fd0ec0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:08.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0e3fff640 1 --2- 172.21.15.46:0/2642274721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0ec0ca0e0 0x7fd0ec15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:08.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0eaffd640 1 -- 172.21.15.46:0/2642274721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0ec0ca0e0 msgr2=0x7fd0ec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:08.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0eaffd640 1 --2- 172.21.15.46:0/2642274721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0ec0ca0e0 0x7fd0ec15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0eaffd640 1 -- 172.21.15.46:0/2642274721 >> v2:172.21.15.154:3300/0 conn(0x7fd0ec0cf160 msgr2=0x7fd0ec0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:08.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0eaffd640 1 --2- 172.21.15.46:0/2642274721 >> v2:172.21.15.154:3300/0 conn(0x7fd0ec0cf160 0x7fd0ec0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.761+0000 7fd0eaffd640 1 -- 172.21.15.46:0/2642274721 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec0cad70 con 0x7fd0ec1538a0 2023-10-30T13:50:08.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0eaffd640 1 --2- 172.21.15.46:0/2642274721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ec1538a0 0x7fd0ec15d6f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec06cb00 tx=0x7fd0dc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2886fbc3c9da7de1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:08.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0ea7fc640 1 -- 172.21.15.46:0/2642274721 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc0a89b0 con 0x7fd0ec1538a0 2023-10-30T13:50:08.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0ea7fc640 1 -- 172.21.15.46:0/2642274721 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc0a8b10 con 0x7fd0ec1538a0 2023-10-30T13:50:08.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0ea7fc640 1 -- 172.21.15.46:0/2642274721 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc0a8ea0 con 0x7fd0ec1538a0 2023-10-30T13:50:08.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0f151d640 1 -- 172.21.15.46:0/2642274721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ec1538a0 msgr2=0x7fd0ec15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:08.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0f151d640 1 --2- 172.21.15.46:0/2642274721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ec1538a0 0x7fd0ec15d6f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec06cb00 tx=0x7fd0dc01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0f151d640 1 -- 172.21.15.46:0/2642274721 shutdown_connections 2023-10-30T13:50:08.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0f151d640 1 --2- 172.21.15.46:0/2642274721 >> v2:172.21.15.154:3300/0 conn(0x7fd0ec0cf160 0x7fd0ec0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0f151d640 1 --2- 172.21.15.46:0/2642274721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0ec0ca0e0 0x7fd0ec15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0f151d640 1 --2- 172.21.15.46:0/2642274721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ec1538a0 0x7fd0ec15d6f0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0f151d640 1 -- 172.21.15.46:0/2642274721 >> 172.21.15.46:0/2642274721 conn(0x7fd0ec06bfe0 msgr2=0x7fd0ec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:08.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0f151d640 1 -- 172.21.15.46:0/2642274721 shutdown_connections 2023-10-30T13:50:08.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.765+0000 7fd0f151d640 1 -- 172.21.15.46:0/2642274721 wait complete. 2023-10-30T13:50:08.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0f151d640 1 Processor -- start 2023-10-30T13:50:08.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0f151d640 1 -- start start 2023-10-30T13:50:08.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0f151d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0ec0ca0e0 0x7fd0ec0b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:08.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0f151d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0ec0cf160 0x7fd0ec14fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:08.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0f151d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ec1538a0 0x7fd0ec0b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:08.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0f151d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec1f36d0 con 0x7fd0ec1538a0 2023-10-30T13:50:08.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0f151d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec1efd60 con 0x7fd0ec0ca0e0 2023-10-30T13:50:08.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0f151d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec15cc90 con 0x7fd0ec0cf160 2023-10-30T13:50:08.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0eb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ec1538a0 0x7fd0ec0b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:08.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0eb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ec1538a0 0x7fd0ec0b35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55430/0 (socket says 172.21.15.46:55430) 2023-10-30T13:50:08.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0eb7fe640 1 -- 172.21.15.46:0/3998205525 learned_addr learned my addr 172.21.15.46:0/3998205525 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:08.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0e3fff640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0ec0cf160 0x7fd0ec14fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:08.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0eaffd640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0ec0ca0e0 0x7fd0ec0b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:08.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0eb7fe640 1 -- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0ec0cf160 msgr2=0x7fd0ec14fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:08.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0eb7fe640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0ec0cf160 0x7fd0ec14fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0eb7fe640 1 -- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0ec0ca0e0 msgr2=0x7fd0ec0b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:08.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0eb7fe640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0ec0ca0e0 0x7fd0ec0b4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0eb7fe640 1 -- 172.21.15.46:0/3998205525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc0094d0 con 0x7fd0ec1538a0 2023-10-30T13:50:08.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0e3fff640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0ec0cf160 0x7fd0ec14fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:08.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0eb7fe640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ec1538a0 0x7fd0ec0b35f0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4007dc0 tx=0x7fd0e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:08.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0e37fe640 1 -- 172.21.15.46:0/3998205525 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4014420 con 0x7fd0ec1538a0 2023-10-30T13:50:08.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0e37fe640 1 -- 172.21.15.46:0/3998205525 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e40145a0 con 0x7fd0ec1538a0 2023-10-30T13:50:08.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0e37fe640 1 -- 172.21.15.46:0/3998205525 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4014950 con 0x7fd0ec1538a0 2023-10-30T13:50:08.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0f151d640 1 -- 172.21.15.46:0/3998205525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ec1f0180 con 0x7fd0ec1538a0 2023-10-30T13:50:08.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.769+0000 7fd0f151d640 1 -- 172.21.15.46:0/3998205525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ec1ef3c0 con 0x7fd0ec1538a0 2023-10-30T13:50:08.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.773+0000 7fd0f151d640 1 -- 172.21.15.46:0/3998205525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ec0b2a50 con 0x7fd0ec1538a0 2023-10-30T13:50:08.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.773+0000 7fd0e37fe640 1 -- 172.21.15.46:0/3998205525 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0e4014ab0 con 0x7fd0ec1538a0 2023-10-30T13:50:08.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.773+0000 7fd0e37fe640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0cc0664a0 0x7fd0cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:08.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.773+0000 7fd0e37fe640 1 -- 172.21.15.46:0/3998205525 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fd0e4084c10 con 0x7fd0ec1538a0 2023-10-30T13:50:08.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.773+0000 7fd0eaffd640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0cc0664a0 0x7fd0cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:08.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.773+0000 7fd0eaffd640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0cc0664a0 0x7fd0cc068960 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0b2000 tx=0x7fd0dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:08.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.777+0000 7fd0e37fe640 1 -- 172.21.15.46:0/3998205525 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0e4051b00 con 0x7fd0ec1538a0 2023-10-30T13:50:08.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.965+0000 7fd0f151d640 1 -- 172.21.15.46:0/3998205525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0ec1efea0 con 0x7fd0ec1538a0 2023-10-30T13:50:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.965+0000 7fd0e37fe640 1 -- 172.21.15.46:0/3998205525 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5731 (secure 0 0 0) 0x7fd0ec1efea0 con 0x7fd0ec1538a0 2023-10-30T13:50:08.971 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:04.191491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 117 2023-10-30T13:50:08.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.969+0000 7fd0f151d640 1 -- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0cc0664a0 msgr2=0x7fd0cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:08.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.969+0000 7fd0f151d640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0cc0664a0 0x7fd0cc068960 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0b2000 tx=0x7fd0dc009040 comp rx=0 tx=0).stop 2023-10-30T13:50:08.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.969+0000 7fd0f151d640 1 -- 172.21.15.46:0/3998205525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ec1538a0 msgr2=0x7fd0ec0b35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:08.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.969+0000 7fd0f151d640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ec1538a0 0x7fd0ec0b35f0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4007dc0 tx=0x7fd0e400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.969+0000 7fd0f151d640 1 -- 172.21.15.46:0/3998205525 shutdown_connections 2023-10-30T13:50:08.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.969+0000 7fd0f151d640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0cc0664a0 0x7fd0cc068960 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.969+0000 7fd0f151d640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ec1538a0 0x7fd0ec0b35f0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.969+0000 7fd0f151d640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0ec0cf160 0x7fd0ec14fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.969+0000 7fd0f151d640 1 --2- 172.21.15.46:0/3998205525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0ec0ca0e0 0x7fd0ec0b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.969+0000 7fd0f151d640 1 -- 172.21.15.46:0/3998205525 >> 172.21.15.46:0/3998205525 conn(0x7fd0ec06bfe0 msgr2=0x7fd0ec151f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.969+0000 7fd0f151d640 1 -- 172.21.15.46:0/3998205525 shutdown_connections 2023-10-30T13:50:08.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:08.969+0000 7fd0f151d640 1 -- 172.21.15.46:0/3998205525 wait complete. 2023-10-30T13:50:08.993 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:50:10.994 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:11.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7fb6b3e640 1 Processor -- start 2023-10-30T13:50:11.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7fb6b3e640 1 -- start start 2023-10-30T13:50:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7fb6b3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7fb00c6b80 0x7f7fb00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7fb6b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fb00c6f60 0x7f7fb00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7fb6b3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb00c0a50 con 0x7f7fb00c8430 2023-10-30T13:50:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7fb6b3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb006f6e0 con 0x7f7fb00c6b80 2023-10-30T13:50:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7fb6b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb006f9c0 con 0x7f7fb00c6f60 2023-10-30T13:50:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7fb50db640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7fb00c8430 0x7f7fb013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37906/0 (socket says 172.21.15.46:37906) 2023-10-30T13:50:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7fb50db640 1 -- 172.21.15.46:0/2800509460 learned_addr learned my addr 172.21.15.46:0/2800509460 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:50:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7fb48da640 1 --2- 172.21.15.46:0/2800509460 >> v2:172.21.15.154:3300/0 conn(0x7f7fb00c6b80 0x7f7fb00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7faffff640 1 --2- 172.21.15.46:0/2800509460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fb00c6f60 0x7f7fb00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:11.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7faf7fe640 1 -- 172.21.15.46:0/2800509460 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117592636 0 0) 0x7f7fb00c0a50 con 0x7f7fb00c8430 2023-10-30T13:50:11.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.137+0000 7f7faf7fe640 1 -- 172.21.15.46:0/2800509460 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa0003530 con 0x7f7fb00c8430 2023-10-30T13:50:11.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.141+0000 7f7fb48da640 1 -- 172.21.15.46:0/2800509460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fb00c6f60 msgr2=0x7f7fb00c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:11.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.141+0000 7f7fb48da640 1 --2- 172.21.15.46:0/2800509460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fb00c6f60 0x7f7fb00c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:11.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.141+0000 7f7fb48da640 1 -- 172.21.15.46:0/2800509460 >> v1:172.21.15.46:6789/0 conn(0x7f7fb00c8430 legacy=0x7f7fb013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:11.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.141+0000 7f7fb48da640 1 -- 172.21.15.46:0/2800509460 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb006fca0 con 0x7f7fb00c6b80 2023-10-30T13:50:11.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.141+0000 7f7fb48da640 1 --2- 172.21.15.46:0/2800509460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fb00c6b80 0x7f7fb00bfec0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40097c0 tx=0x7f7fa401c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=321fcc49c2bba21b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:11.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.141+0000 7f7faf7fe640 1 -- 172.21.15.46:0/2800509460 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa40ab600 con 0x7f7fb00c6b80 2023-10-30T13:50:11.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.141+0000 7f7faf7fe640 1 -- 172.21.15.46:0/2800509460 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa40a7050 con 0x7f7fb00c6b80 2023-10-30T13:50:11.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.141+0000 7f7faf7fe640 1 -- 172.21.15.46:0/2800509460 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa40ab960 con 0x7f7fb00c6b80 2023-10-30T13:50:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2800509460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fb00c6b80 msgr2=0x7f7fb00bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 --2- 172.21.15.46:0/2800509460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fb00c6b80 0x7f7fb00bfec0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40097c0 tx=0x7f7fa401c3f0 comp rx=0 tx=0).stop 2023-10-30T13:50:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2800509460 shutdown_connections 2023-10-30T13:50:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 --2- 172.21.15.46:0/2800509460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fb00c6f60 0x7f7fb00c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 --2- 172.21.15.46:0/2800509460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fb00c6b80 0x7f7fb00bfec0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2800509460 >> 172.21.15.46:0/2800509460 conn(0x7f7fb00c7910 msgr2=0x7f7fb00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:11.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2800509460 shutdown_connections 2023-10-30T13:50:11.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2800509460 wait complete. 2023-10-30T13:50:11.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 Processor -- start 2023-10-30T13:50:11.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 -- start start 2023-10-30T13:50:11.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fb00c6b80 0x7f7fb01368f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:11.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb00c6f60 0x7f7fb0136e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:11.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fb00c8430 0x7f7fb0157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:11.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb0139310 con 0x7f7fb00c6f60 2023-10-30T13:50:11.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb013af60 con 0x7f7fb00c8430 2023-10-30T13:50:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb6b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb0139560 con 0x7f7fb00c6b80 2023-10-30T13:50:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7faffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb00c6f60 0x7f7fb0136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7faffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb00c6f60 0x7f7fb0136e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55436/0 (socket says 172.21.15.46:55436) 2023-10-30T13:50:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7faffff640 1 -- 172.21.15.46:0/2791528143 learned_addr learned my addr 172.21.15.46:0/2791528143 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb48da640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fb00c6b80 0x7f7fb01368f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.145+0000 7f7fb50db640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fb00c8430 0x7f7fb0157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7faffff640 1 -- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fb00c6b80 msgr2=0x7f7fb01368f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7faffff640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fb00c6b80 0x7f7fb01368f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7faffff640 1 -- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fb00c8430 msgr2=0x7f7fb0157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7faffff640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fb00c8430 0x7f7fb0157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7faffff640 1 -- 172.21.15.46:0/2791528143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa4009420 con 0x7f7fb00c6f60 2023-10-30T13:50:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7faffff640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb00c6f60 0x7f7fb0136e30 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00ed40 tx=0x7f7f9c00d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7fad7fa640 1 -- 172.21.15.46:0/2791528143 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c00de30 con 0x7f7fb00c6f60 2023-10-30T13:50:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7fad7fa640 1 -- 172.21.15.46:0/2791528143 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9c010650 con 0x7f7fb00c6f60 2023-10-30T13:50:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7fad7fa640 1 -- 172.21.15.46:0/2791528143 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c0109c0 con 0x7f7fb00c6f60 2023-10-30T13:50:11.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2791528143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb013b200 con 0x7f7fb00c6f60 2023-10-30T13:50:11.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2791528143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb0139840 con 0x7f7fb00c6f60 2023-10-30T13:50:11.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2791528143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb0157820 con 0x7f7fb00c6f60 2023-10-30T13:50:11.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.149+0000 7f7fad7fa640 1 -- 172.21.15.46:0/2791528143 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7f9c010b20 con 0x7f7fb00c6f60 2023-10-30T13:50:11.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.153+0000 7f7fad7fa640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f940664a0 0x7f7f94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:11.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.153+0000 7f7fad7fa640 1 -- 172.21.15.46:0/2791528143 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f7f9c084810 con 0x7f7fb00c6f60 2023-10-30T13:50:11.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.153+0000 7f7fb48da640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f940664a0 0x7f7f94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:11.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.153+0000 7f7fb48da640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f940664a0 0x7f7f94068960 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40ad000 tx=0x7f7fa40a4f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:11.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.157+0000 7f7fad7fa640 1 -- 172.21.15.46:0/2791528143 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7f9c051840 con 0x7f7fb00c6f60 2023-10-30T13:50:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.349+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2791528143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fb004e170 con 0x7f7fb00c6f60 2023-10-30T13:50:11.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fad7fa640 1 -- 172.21.15.46:0/2791528143 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f7f9c054e60 con 0x7f7fb00c6f60 2023-10-30T13:50:11.356 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:11.356 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:04.191491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 117 2023-10-30T13:50:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f940664a0 msgr2=0x7f7f94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fb6b3e640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f940664a0 0x7f7f94068960 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40ad000 tx=0x7f7fa40a4f90 comp rx=0 tx=0).stop 2023-10-30T13:50:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2791528143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb00c6f60 msgr2=0x7f7fb0136e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fb6b3e640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb00c6f60 0x7f7fb0136e30 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00ed40 tx=0x7f7f9c00d720 comp rx=0 tx=0).stop 2023-10-30T13:50:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2791528143 shutdown_connections 2023-10-30T13:50:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fb6b3e640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f940664a0 0x7f7f94068960 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fb6b3e640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fb00c8430 0x7f7fb0157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fb6b3e640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fb00c6f60 0x7f7fb0136e30 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fb6b3e640 1 --2- 172.21.15.46:0/2791528143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fb00c6b80 0x7f7fb01368f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2791528143 >> 172.21.15.46:0/2791528143 conn(0x7f7fb00c7910 msgr2=0x7f7fb013b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2791528143 shutdown_connections 2023-10-30T13:50:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:11.353+0000 7f7fb6b3e640 1 -- 172.21.15.46:0/2791528143 wait complete. 2023-10-30T13:50:11.378 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:50:13.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:13.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.517+0000 7f17ecc9c640 1 Processor -- start 2023-10-30T13:50:13.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.517+0000 7f17ecc9c640 1 -- start start 2023-10-30T13:50:13.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17ecc9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f17e80c05d0 0x7f17e813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:13.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17ecc9c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f17e80c6f60 0x7f17e813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:13.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17ecc9c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e80c0a50 con 0x7f17e80c6b80 2023-10-30T13:50:13.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17ecc9c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e806f6e0 con 0x7f17e80c05d0 2023-10-30T13:50:13.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17ecc9c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e806f9c0 con 0x7f17e80c6f60 2023-10-30T13:50:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17e5d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f17e80c05d0 0x7f17e813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17e5d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f17e80c05d0 0x7f17e813ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55438/0 (socket says 172.21.15.46:55438) 2023-10-30T13:50:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17e5d9b640 1 -- 172.21.15.46:0/2794151487 learned_addr learned my addr 172.21.15.46:0/2794151487 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:13.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17e6d9d640 1 --2- 172.21.15.46:0/2794151487 >> v2:172.21.15.154:3300/0 conn(0x7f17e80c6f60 0x7f17e813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17e5d9b640 1 -- 172.21.15.46:0/2794151487 >> v2:172.21.15.154:3300/0 conn(0x7f17e80c6f60 msgr2=0x7f17e813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17e5d9b640 1 --2- 172.21.15.46:0/2794151487 >> v2:172.21.15.154:3300/0 conn(0x7f17e80c6f60 0x7f17e813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17e5d9b640 1 -- 172.21.15.46:0/2794151487 >> v1:172.21.15.154:6789/0 conn(0x7f17e80c6b80 legacy=0x7f17e80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17e5d9b640 1 -- 172.21.15.46:0/2794151487 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e806fca0 con 0x7f17e80c05d0 2023-10-30T13:50:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.521+0000 7f17e6d9d640 1 --2- 172.21.15.46:0/2794151487 >> v2:172.21.15.154:3300/0 conn(0x7f17e80c6f60 0x7f17e813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:13.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17e5d9b640 1 --2- 172.21.15.46:0/2794151487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17e80c05d0 0x7f17e813ed90 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f17e80c89b0 tx=0x7f17dc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a341e249e7fbd5b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17e559a640 1 -- 172.21.15.46:0/2794151487 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc0ab610 con 0x7f17e80c05d0 2023-10-30T13:50:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17e559a640 1 -- 172.21.15.46:0/2794151487 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17dc0ab770 con 0x7f17e80c05d0 2023-10-30T13:50:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17e559a640 1 -- 172.21.15.46:0/2794151487 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc0abb00 con 0x7f17e80c05d0 2023-10-30T13:50:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17ecc9c640 1 -- 172.21.15.46:0/2794151487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17e80c05d0 msgr2=0x7f17e813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17ecc9c640 1 --2- 172.21.15.46:0/2794151487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17e80c05d0 0x7f17e813ed90 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f17e80c89b0 tx=0x7f17dc01c500 comp rx=0 tx=0).stop 2023-10-30T13:50:13.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17ecc9c640 1 -- 172.21.15.46:0/2794151487 shutdown_connections 2023-10-30T13:50:13.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17ecc9c640 1 --2- 172.21.15.46:0/2794151487 >> v2:172.21.15.154:3300/0 conn(0x7f17e80c6f60 0x7f17e813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:13.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17ecc9c640 1 --2- 172.21.15.46:0/2794151487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17e80c05d0 0x7f17e813ed90 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:13.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17ecc9c640 1 -- 172.21.15.46:0/2794151487 >> 172.21.15.46:0/2794151487 conn(0x7f17e80c7910 msgr2=0x7f17e80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:13.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17ecc9c640 1 -- 172.21.15.46:0/2794151487 shutdown_connections 2023-10-30T13:50:13.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17ecc9c640 1 -- 172.21.15.46:0/2794151487 wait complete. 2023-10-30T13:50:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17ecc9c640 1 Processor -- start 2023-10-30T13:50:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.525+0000 7f17ecc9c640 1 -- start start 2023-10-30T13:50:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17ecc9c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17e80c05d0 0x7f17e8150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17ecc9c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17e80c6b80 0x7f17e8158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17ecc9c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17e80c6f60 0x7f17e8159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17ecc9c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80c9500 con 0x7f17e80c05d0 2023-10-30T13:50:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17ecc9c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e81586c0 con 0x7f17e80c6b80 2023-10-30T13:50:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17ecc9c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e8157830 con 0x7f17e80c6f60 2023-10-30T13:50:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17e659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17e80c05d0 0x7f17e8150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17e659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17e80c05d0 0x7f17e8150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55442/0 (socket says 172.21.15.46:55442) 2023-10-30T13:50:13.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17e659c640 1 -- 172.21.15.46:0/3381342264 learned_addr learned my addr 172.21.15.46:0/3381342264 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:13.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17e6d9d640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17e80c6f60 0x7f17e8159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:13.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17e5d9b640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17e80c6b80 0x7f17e8158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:13.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17e659c640 1 -- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17e80c6f60 msgr2=0x7f17e8159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:13.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17e659c640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17e80c6f60 0x7f17e8159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:13.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17e659c640 1 -- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17e80c6b80 msgr2=0x7f17e8158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:13.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17e659c640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17e80c6b80 0x7f17e8158b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:13.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17e659c640 1 -- 172.21.15.46:0/3381342264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc0094d0 con 0x7f17e80c05d0 2023-10-30T13:50:13.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17e659c640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17e80c05d0 0x7f17e8150c80 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f17d800c500 tx=0x7f17d800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:13.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17d6ffd640 1 -- 172.21.15.46:0/3381342264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d8005220 con 0x7f17e80c05d0 2023-10-30T13:50:13.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17d6ffd640 1 -- 172.21.15.46:0/3381342264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d8005380 con 0x7f17e80c05d0 2023-10-30T13:50:13.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17ecc9c640 1 -- 172.21.15.46:0/3381342264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e8157b10 con 0x7f17e80c05d0 2023-10-30T13:50:13.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17d6ffd640 1 -- 172.21.15.46:0/3381342264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d8007270 con 0x7f17e80c05d0 2023-10-30T13:50:13.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17ecc9c640 1 -- 172.21.15.46:0/3381342264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e806ab90 con 0x7f17e80c05d0 2023-10-30T13:50:13.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.529+0000 7f17ecc9c640 1 -- 172.21.15.46:0/3381342264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e81584b0 con 0x7f17e80c05d0 2023-10-30T13:50:13.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.533+0000 7f17d6ffd640 1 -- 172.21.15.46:0/3381342264 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f17d80073d0 con 0x7f17e80c05d0 2023-10-30T13:50:13.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.533+0000 7f17d6ffd640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17c80664d0 0x7f17c8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:13.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.533+0000 7f17d6ffd640 1 -- 172.21.15.46:0/3381342264 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f17d8083b50 con 0x7f17e80c05d0 2023-10-30T13:50:13.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.533+0000 7f17e5d9b640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17c80664d0 0x7f17c8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:13.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.533+0000 7f17e5d9b640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17c80664d0 0x7f17c8068990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0af000 tx=0x7f17dc0b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:13.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.537+0000 7f17d6ffd640 1 -- 172.21.15.46:0/3381342264 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f17d8050b00 con 0x7f17e80c05d0 2023-10-30T13:50:13.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 -- 172.21.15.46:0/3381342264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17e804e170 con 0x7f17e80c05d0 2023-10-30T13:50:13.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17d6ffd640 1 -- 172.21.15.46:0/3381342264 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f17d8054120 con 0x7f17e80c05d0 2023-10-30T13:50:13.729 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:13.729 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:13.113459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:resolve","state_seq":35,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:13.730 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 118 2023-10-30T13:50:13.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 -- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17c80664d0 msgr2=0x7f17c8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:13.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17c80664d0 0x7f17c8068990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0af000 tx=0x7f17dc0b6040 comp rx=0 tx=0).stop 2023-10-30T13:50:13.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 -- 172.21.15.46:0/3381342264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17e80c05d0 msgr2=0x7f17e8150c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:13.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17e80c05d0 0x7f17e8150c80 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f17d800c500 tx=0x7f17d800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:50:13.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 -- 172.21.15.46:0/3381342264 shutdown_connections 2023-10-30T13:50:13.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17c80664d0 0x7f17c8068990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:13.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17e80c6f60 0x7f17e8159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:13.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17e80c6b80 0x7f17e8158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:13.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 --2- 172.21.15.46:0/3381342264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17e80c05d0 0x7f17e8150c80 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:13.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 -- 172.21.15.46:0/3381342264 >> 172.21.15.46:0/3381342264 conn(0x7f17e80c7910 msgr2=0x7f17e813a350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:13.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 -- 172.21.15.46:0/3381342264 shutdown_connections 2023-10-30T13:50:13.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:13.725+0000 7f17ecc9c640 1 -- 172.21.15.46:0/3381342264 wait complete. 2023-10-30T13:50:13.751 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:50:15.752 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:15.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.901+0000 7fd7ab50e640 1 Processor -- start 2023-10-30T13:50:15.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.901+0000 7fd7ab50e640 1 -- start start 2023-10-30T13:50:15.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.901+0000 7fd7ab50e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7a4150920 0x7fd7a406d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:15.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.901+0000 7fd7ab50e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd7a40c2570 0x7fd7a40cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:15.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.901+0000 7fd7ab50e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40c68b0 con 0x7fd7a40c29d0 2023-10-30T13:50:15.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.901+0000 7fd7ab50e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a4150d00 con 0x7fd7a4150920 2023-10-30T13:50:15.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd7ab50e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a406d960 con 0x7fd7a40c2570 2023-10-30T13:50:15.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd7a92aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7a4150920 0x7fd7a406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:15.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd7a92aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7a4150920 0x7fd7a406d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55454/0 (socket says 172.21.15.46:55454) 2023-10-30T13:50:15.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd7a92aa640 1 -- 172.21.15.46:0/905614359 learned_addr learned my addr 172.21.15.46:0/905614359 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:15.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd7a8aa9640 1 --2- 172.21.15.46:0/905614359 >> v2:172.21.15.154:3300/0 conn(0x7fd7a40c2570 0x7fd7a40cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:15.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd7a92aa640 1 -- 172.21.15.46:0/905614359 >> v2:172.21.15.154:3300/0 conn(0x7fd7a40c2570 msgr2=0x7fd7a40cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:15.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd7a92aa640 1 --2- 172.21.15.46:0/905614359 >> v2:172.21.15.154:3300/0 conn(0x7fd7a40c2570 0x7fd7a40cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:15.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd7a92aa640 1 -- 172.21.15.46:0/905614359 >> v1:172.21.15.154:6789/0 conn(0x7fd7a40c29d0 legacy=0x7fd7a4156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:15.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd7a92aa640 1 -- 172.21.15.46:0/905614359 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a40c2e90 con 0x7fd7a4150920 2023-10-30T13:50:15.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd7a92aa640 1 --2- 172.21.15.46:0/905614359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7a4150920 0x7fd7a406d420 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fd7a406dfa0 tx=0x7fd7a001c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6842ff02ebddf30f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:15.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd79bfff640 1 -- 172.21.15.46:0/905614359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a00ac410 con 0x7fd7a4150920 2023-10-30T13:50:15.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd79bfff640 1 -- 172.21.15.46:0/905614359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a00ac570 con 0x7fd7a4150920 2023-10-30T13:50:15.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.905+0000 7fd79bfff640 1 -- 172.21.15.46:0/905614359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a00ac900 con 0x7fd7a4150920 2023-10-30T13:50:15.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 -- 172.21.15.46:0/905614359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7a4150920 msgr2=0x7fd7a406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:15.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 --2- 172.21.15.46:0/905614359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7a4150920 0x7fd7a406d420 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fd7a406dfa0 tx=0x7fd7a001c4c0 comp rx=0 tx=0).stop 2023-10-30T13:50:15.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 -- 172.21.15.46:0/905614359 shutdown_connections 2023-10-30T13:50:15.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 --2- 172.21.15.46:0/905614359 >> v2:172.21.15.154:3300/0 conn(0x7fd7a40c2570 0x7fd7a40cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:15.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 --2- 172.21.15.46:0/905614359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7a4150920 0x7fd7a406d420 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:15.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 -- 172.21.15.46:0/905614359 >> 172.21.15.46:0/905614359 conn(0x7fd7a406dfd0 msgr2=0x7fd7a406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:15.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 -- 172.21.15.46:0/905614359 shutdown_connections 2023-10-30T13:50:15.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 -- 172.21.15.46:0/905614359 wait complete. 2023-10-30T13:50:15.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 Processor -- start 2023-10-30T13:50:15.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 -- start start 2023-10-30T13:50:15.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7a40c2570 0x7fd7a4155c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:15.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7a40c29d0 0x7fd7a414ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:15.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7a4150920 0x7fd7a414e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:15.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a406bd70 con 0x7fd7a4150920 2023-10-30T13:50:15.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a41f2c50 con 0x7fd7a40c2570 2023-10-30T13:50:15.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7ab50e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a41f2360 con 0x7fd7a40c29d0 2023-10-30T13:50:15.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7a9aab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7a4150920 0x7fd7a414e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:15.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7a9aab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7a4150920 0x7fd7a414e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55462/0 (socket says 172.21.15.46:55462) 2023-10-30T13:50:15.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7a9aab640 1 -- 172.21.15.46:0/3750539660 learned_addr learned my addr 172.21.15.46:0/3750539660 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:15.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7a92aa640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7a40c2570 0x7fd7a4155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:15.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7a8aa9640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7a40c29d0 0x7fd7a414ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:15.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7a9aab640 1 -- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7a40c29d0 msgr2=0x7fd7a414ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:15.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7a9aab640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7a40c29d0 0x7fd7a414ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:15.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7a9aab640 1 -- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7a40c2570 msgr2=0x7fd7a4155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:15.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7a9aab640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7a40c2570 0x7fd7a4155c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:15.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7a9aab640 1 -- 172.21.15.46:0/3750539660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a00094d0 con 0x7fd7a4150920 2023-10-30T13:50:15.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.909+0000 7fd7a8aa9640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7a40c29d0 0x7fd7a414ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:15.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.913+0000 7fd7a9aab640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7a4150920 0x7fd7a414e970 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00c7b0 tx=0x7fd79c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:15.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.913+0000 7fd799ffb640 1 -- 172.21.15.46:0/3750539660 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79c00a9f0 con 0x7fd7a4150920 2023-10-30T13:50:15.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.913+0000 7fd799ffb640 1 -- 172.21.15.46:0/3750539660 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79c00ab50 con 0x7fd7a4150920 2023-10-30T13:50:15.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.913+0000 7fd7ab50e640 1 -- 172.21.15.46:0/3750539660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a414fe10 con 0x7fd7a4150920 2023-10-30T13:50:15.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.913+0000 7fd799ffb640 1 -- 172.21.15.46:0/3750539660 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79c012420 con 0x7fd7a4150920 2023-10-30T13:50:15.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.913+0000 7fd7ab50e640 1 -- 172.21.15.46:0/3750539660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a41f28e0 con 0x7fd7a4150920 2023-10-30T13:50:15.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.913+0000 7fd7ab50e640 1 -- 172.21.15.46:0/3750539660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a41f2e00 con 0x7fd7a4150920 2023-10-30T13:50:15.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.913+0000 7fd799ffb640 1 -- 172.21.15.46:0/3750539660 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd79c012580 con 0x7fd7a4150920 2023-10-30T13:50:15.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.913+0000 7fd799ffb640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd7880664a0 0x7fd788068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:15.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.913+0000 7fd799ffb640 1 -- 172.21.15.46:0/3750539660 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fd79c084510 con 0x7fd7a4150920 2023-10-30T13:50:15.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.913+0000 7fd7a92aa640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd7880664a0 0x7fd788068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:15.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.917+0000 7fd7a92aa640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd7880664a0 0x7fd788068960 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd7a00ae000 tx=0x7fd7a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:15.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:15.917+0000 7fd799ffb640 1 -- 172.21.15.46:0/3750539660 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd7a41f2e00 con 0x7fd7a4150920 2023-10-30T13:50:16.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 -- 172.21.15.46:0/3750539660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7a41f2e00 con 0x7fd7a4150920 2023-10-30T13:50:16.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd799ffb640 1 -- 172.21.15.46:0/3750539660 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fd7a41f2e00 con 0x7fd7a4150920 2023-10-30T13:50:16.113 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:16.113 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:13.113459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:resolve","state_seq":35,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:16.113 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 118 2023-10-30T13:50:16.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 -- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd7880664a0 msgr2=0x7fd788068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:16.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd7880664a0 0x7fd788068960 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd7a00ae000 tx=0x7fd7a0009040 comp rx=0 tx=0).stop 2023-10-30T13:50:16.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 -- 172.21.15.46:0/3750539660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7a4150920 msgr2=0x7fd7a414e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:16.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7a4150920 0x7fd7a414e970 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00c7b0 tx=0x7fd79c00cc70 comp rx=0 tx=0).stop 2023-10-30T13:50:16.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 -- 172.21.15.46:0/3750539660 shutdown_connections 2023-10-30T13:50:16.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd7880664a0 0x7fd788068960 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:16.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7a4150920 0x7fd7a414e970 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:16.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7a40c29d0 0x7fd7a414ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:16.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 --2- 172.21.15.46:0/3750539660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7a40c2570 0x7fd7a4155c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:16.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 -- 172.21.15.46:0/3750539660 >> 172.21.15.46:0/3750539660 conn(0x7fd7a406dfd0 msgr2=0x7fd7a415f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:16.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 -- 172.21.15.46:0/3750539660 shutdown_connections 2023-10-30T13:50:16.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:16.109+0000 7fd7ab50e640 1 -- 172.21.15.46:0/3750539660 wait complete. 2023-10-30T13:50:16.134 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:50:18.135 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:18.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.289+0000 7f235a6d6640 1 Processor -- start 2023-10-30T13:50:18.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.289+0000 7f235a6d6640 1 -- start start 2023-10-30T13:50:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f235a6d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2354153d70 0x7f235406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f235a6d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23540c8490 0x7f23540c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f235a6d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235406afa0 con 0x7f235406e470 2023-10-30T13:50:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f235a6d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f235406d780 con 0x7f2354153d70 2023-10-30T13:50:18.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f235a6d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2354154150 con 0x7f23540c8490 2023-10-30T13:50:18.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f23537fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f235406e470 0x7f23540cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34884/0 (socket says 172.21.15.46:34884) 2023-10-30T13:50:18.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f23537fe640 1 -- 172.21.15.46:0/2547989227 learned_addr learned my addr 172.21.15.46:0/2547989227 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:50:18.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f2353fff640 1 --2- 172.21.15.46:0/2547989227 >> v2:172.21.15.154:3300/0 conn(0x7f2354153d70 0x7f235406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f2358c73640 1 --2- 172.21.15.46:0/2547989227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23540c8490 0x7f23540c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f2352ffd640 1 -- 172.21.15.46:0/2547989227 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835563276 0 0) 0x7f235406afa0 con 0x7f235406e470 2023-10-30T13:50:18.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f2352ffd640 1 -- 172.21.15.46:0/2547989227 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f233c003530 con 0x7f235406e470 2023-10-30T13:50:18.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f2353fff640 1 -- 172.21.15.46:0/2547989227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23540c8490 msgr2=0x7f23540c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:18.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f2353fff640 1 --2- 172.21.15.46:0/2547989227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23540c8490 0x7f23540c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:18.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f2353fff640 1 -- 172.21.15.46:0/2547989227 >> v1:172.21.15.46:6789/0 conn(0x7f235406e470 legacy=0x7f23540cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:18.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.293+0000 7f2353fff640 1 -- 172.21.15.46:0/2547989227 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235406e030 con 0x7f2354153d70 2023-10-30T13:50:18.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.297+0000 7f2353fff640 1 --2- 172.21.15.46:0/2547989227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354153d70 0x7f235406ca80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f23440097c0 tx=0x7f234401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48dbad0279099b8e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:18.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.297+0000 7f2352ffd640 1 -- 172.21.15.46:0/2547989227 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23440ab650 con 0x7f2354153d70 2023-10-30T13:50:18.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.297+0000 7f2352ffd640 1 -- 172.21.15.46:0/2547989227 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23440ab7b0 con 0x7f2354153d70 2023-10-30T13:50:18.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.297+0000 7f2352ffd640 1 -- 172.21.15.46:0/2547989227 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23440abb20 con 0x7f2354153d70 2023-10-30T13:50:18.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.297+0000 7f235a6d6640 1 -- 172.21.15.46:0/2547989227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354153d70 msgr2=0x7f235406ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.297+0000 7f235a6d6640 1 --2- 172.21.15.46:0/2547989227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354153d70 0x7f235406ca80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f23440097c0 tx=0x7f234401c450 comp rx=0 tx=0).stop 2023-10-30T13:50:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.297+0000 7f235a6d6640 1 -- 172.21.15.46:0/2547989227 shutdown_connections 2023-10-30T13:50:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.297+0000 7f235a6d6640 1 --2- 172.21.15.46:0/2547989227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23540c8490 0x7f23540c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.297+0000 7f235a6d6640 1 --2- 172.21.15.46:0/2547989227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354153d70 0x7f235406ca80 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.297+0000 7f235a6d6640 1 -- 172.21.15.46:0/2547989227 >> 172.21.15.46:0/2547989227 conn(0x7f2354071ea0 msgr2=0x7f2354065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:18.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.297+0000 7f235a6d6640 1 -- 172.21.15.46:0/2547989227 shutdown_connections 2023-10-30T13:50:18.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.297+0000 7f235a6d6640 1 -- 172.21.15.46:0/2547989227 wait complete. 2023-10-30T13:50:18.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f235a6d6640 1 Processor -- start 2023-10-30T13:50:18.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f235a6d6640 1 -- start start 2023-10-30T13:50:18.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f235a6d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f235406e470 0x7f2354157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:18.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f235a6d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23540c8490 0x7f235414d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:18.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f235a6d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2354153d70 0x7f2354152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:18.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f235a6d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235414c0d0 con 0x7f235406e470 2023-10-30T13:50:18.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f235a6d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235414ba90 con 0x7f23540c8490 2023-10-30T13:50:18.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f235a6d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23541f54d0 con 0x7f2354153d70 2023-10-30T13:50:18.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f2353fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f235406e470 0x7f2354157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:18.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f2353fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f235406e470 0x7f2354157360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55524/0 (socket says 172.21.15.46:55524) 2023-10-30T13:50:18.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f2353fff640 1 -- 172.21.15.46:0/559968035 learned_addr learned my addr 172.21.15.46:0/559968035 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:18.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f23537fe640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23540c8490 0x7f235414d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:18.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f2358c73640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2354153d70 0x7f2354152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:18.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f2353fff640 1 -- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2354153d70 msgr2=0x7f2354152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:18.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f2353fff640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2354153d70 0x7f2354152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:18.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f2353fff640 1 -- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23540c8490 msgr2=0x7f235414d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:18.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f2353fff640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23540c8490 0x7f235414d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:18.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f2353fff640 1 -- 172.21.15.46:0/559968035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2344009420 con 0x7f235406e470 2023-10-30T13:50:18.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f23537fe640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23540c8490 0x7f235414d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:18.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.301+0000 7f2358c73640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2354153d70 0x7f2354152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:18.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.305+0000 7f2353fff640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f235406e470 0x7f2354157360 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f23440ad040 tx=0x7f23440191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:18.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.305+0000 7f2350ff9640 1 -- 172.21.15.46:0/559968035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2344004450 con 0x7f235406e470 2023-10-30T13:50:18.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.305+0000 7f2350ff9640 1 -- 172.21.15.46:0/559968035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23440ab920 con 0x7f235406e470 2023-10-30T13:50:18.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.305+0000 7f235a6d6640 1 -- 172.21.15.46:0/559968035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235414af30 con 0x7f235406e470 2023-10-30T13:50:18.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.305+0000 7f2350ff9640 1 -- 172.21.15.46:0/559968035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23440abc90 con 0x7f235406e470 2023-10-30T13:50:18.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.305+0000 7f235a6d6640 1 -- 172.21.15.46:0/559968035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235414b4d0 con 0x7f235406e470 2023-10-30T13:50:18.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.305+0000 7f235a6d6640 1 -- 172.21.15.46:0/559968035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f235404e170 con 0x7f235406e470 2023-10-30T13:50:18.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.305+0000 7f2350ff9640 1 -- 172.21.15.46:0/559968035 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2344003620 con 0x7f235406e470 2023-10-30T13:50:18.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.309+0000 7f2350ff9640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23400664a0 0x7f2340068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:18.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.309+0000 7f2350ff9640 1 -- 172.21.15.46:0/559968035 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f234411e550 con 0x7f235406e470 2023-10-30T13:50:18.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.309+0000 7f23537fe640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23400664a0 0x7f2340068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:18.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.309+0000 7f23537fe640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23400664a0 0x7f2340068960 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f23541efb90 tx=0x7f2348009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:18.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.317+0000 7f2350ff9640 1 -- 172.21.15.46:0/559968035 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23440eb4e0 con 0x7f235406e470 2023-10-30T13:50:18.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.521+0000 7f235a6d6640 1 -- 172.21.15.46:0/559968035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f235406c2a0 con 0x7f235406e470 2023-10-30T13:50:18.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.521+0000 7f2350ff9640 1 -- 172.21.15.46:0/559968035 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f23440eeb00 con 0x7f235406e470 2023-10-30T13:50:18.526 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:18.526 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:13.113459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:resolve","state_seq":35,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:18.526 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 118 2023-10-30T13:50:18.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.521+0000 7f235a6d6640 1 -- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23400664a0 msgr2=0x7f2340068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:18.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.521+0000 7f235a6d6640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23400664a0 0x7f2340068960 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f23541efb90 tx=0x7f2348009040 comp rx=0 tx=0).stop 2023-10-30T13:50:18.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.521+0000 7f235a6d6640 1 -- 172.21.15.46:0/559968035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f235406e470 msgr2=0x7f2354157360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:18.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.521+0000 7f235a6d6640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f235406e470 0x7f2354157360 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f23440ad040 tx=0x7f23440191f0 comp rx=0 tx=0).stop 2023-10-30T13:50:18.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.525+0000 7f235a6d6640 1 -- 172.21.15.46:0/559968035 shutdown_connections 2023-10-30T13:50:18.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.525+0000 7f235a6d6640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23400664a0 0x7f2340068960 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:18.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.525+0000 7f235a6d6640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2354153d70 0x7f2354152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:18.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.525+0000 7f235a6d6640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23540c8490 0x7f235414d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:18.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.525+0000 7f235a6d6640 1 --2- 172.21.15.46:0/559968035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f235406e470 0x7f2354157360 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:18.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.525+0000 7f235a6d6640 1 -- 172.21.15.46:0/559968035 >> 172.21.15.46:0/559968035 conn(0x7f2354071ea0 msgr2=0x7f235414e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:18.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.525+0000 7f235a6d6640 1 -- 172.21.15.46:0/559968035 shutdown_connections 2023-10-30T13:50:18.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:18.525+0000 7f235a6d6640 1 -- 172.21.15.46:0/559968035 wait complete. 2023-10-30T13:50:18.548 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 118, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 118, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6702": { "addr": "172.21.15.46:6836/2197729559", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 2197729559, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 2197729559, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6702, "incarnation": 111, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 43 }, "gid_6858": { "addr": "172.21.15.46:6837/2572113428", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2572113428, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2572113428, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6858, "incarnation": 117, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:resolve", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:50:13.113459+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6702, "mds_1": 6858 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:50:20.549 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:20.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.697+0000 7fc45253e640 1 Processor -- start 2023-10-30T13:50:20.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc45253e640 1 -- start start 2023-10-30T13:50:20.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc45253e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc44c1539a0 0x7fc44c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:20.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc45253e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc44c070a00 0x7fc44c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:20.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc45253e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc44c15a8c0 con 0x7fc44c069e90 2023-10-30T13:50:20.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc45253e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc44c06fbb0 con 0x7fc44c070a00 2023-10-30T13:50:20.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc45253e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c071380 con 0x7fc44c1539a0 2023-10-30T13:50:20.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc44b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc44c070a00 0x7fc44c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:20.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc44b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc44c070a00 0x7fc44c070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55540/0 (socket says 172.21.15.46:55540) 2023-10-30T13:50:20.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc44b7fe640 1 -- 172.21.15.46:0/3419484771 learned_addr learned my addr 172.21.15.46:0/3419484771 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:20.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc44bfff640 1 --2- 172.21.15.46:0/3419484771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc44c1539a0 0x7fc44c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:20.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc44b7fe640 1 -- 172.21.15.46:0/3419484771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc44c1539a0 msgr2=0x7fc44c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:20.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc44b7fe640 1 --2- 172.21.15.46:0/3419484771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc44c1539a0 0x7fc44c15a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:20.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc44b7fe640 1 -- 172.21.15.46:0/3419484771 >> v1:172.21.15.154:6789/0 conn(0x7fc44c069e90 legacy=0x7fc44c06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:20.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc44b7fe640 1 -- 172.21.15.46:0/3419484771 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c153d80 con 0x7fc44c070a00 2023-10-30T13:50:20.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.701+0000 7fc44bfff640 1 --2- 172.21.15.46:0/3419484771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc44c1539a0 0x7fc44c15a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:20.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc44b7fe640 1 --2- 172.21.15.46:0/3419484771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44c070a00 0x7fc44c070e40 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc440009870 tx=0x7fc44001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b25fbbb68cd6cda server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:20.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc44affd640 1 -- 172.21.15.46:0/3419484771 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4400ab680 con 0x7fc44c070a00 2023-10-30T13:50:20.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc44affd640 1 -- 172.21.15.46:0/3419484771 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4400ab7e0 con 0x7fc44c070a00 2023-10-30T13:50:20.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc44affd640 1 -- 172.21.15.46:0/3419484771 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4400abb70 con 0x7fc44c070a00 2023-10-30T13:50:20.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc45253e640 1 -- 172.21.15.46:0/3419484771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44c070a00 msgr2=0x7fc44c070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:20.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc45253e640 1 --2- 172.21.15.46:0/3419484771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44c070a00 0x7fc44c070e40 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc440009870 tx=0x7fc44001c480 comp rx=0 tx=0).stop 2023-10-30T13:50:20.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc45253e640 1 -- 172.21.15.46:0/3419484771 shutdown_connections 2023-10-30T13:50:20.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc45253e640 1 --2- 172.21.15.46:0/3419484771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44c070a00 0x7fc44c070e40 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:20.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc45253e640 1 --2- 172.21.15.46:0/3419484771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc44c1539a0 0x7fc44c15a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:20.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc45253e640 1 -- 172.21.15.46:0/3419484771 >> 172.21.15.46:0/3419484771 conn(0x7fc44c0c7910 msgr2=0x7fc44c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:20.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc45253e640 1 -- 172.21.15.46:0/3419484771 shutdown_connections 2023-10-30T13:50:20.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc45253e640 1 -- 172.21.15.46:0/3419484771 wait complete. 2023-10-30T13:50:20.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc45253e640 1 Processor -- start 2023-10-30T13:50:20.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.705+0000 7fc45253e640 1 -- start start 2023-10-30T13:50:20.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc45253e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44c069e90 0x7fc44c146890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:20.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc45253e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc44c070a00 0x7fc44c0c4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:20.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc45253e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc44c1539a0 0x7fc44c0c4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:20.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc45253e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c0c2210 con 0x7fc44c069e90 2023-10-30T13:50:20.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc45253e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c1590b0 con 0x7fc44c070a00 2023-10-30T13:50:20.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc45253e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c0ca020 con 0x7fc44c1539a0 2023-10-30T13:50:20.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc44bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44c069e90 0x7fc44c146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:20.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc44bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44c069e90 0x7fc44c146890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55542/0 (socket says 172.21.15.46:55542) 2023-10-30T13:50:20.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc44bfff640 1 -- 172.21.15.46:0/236315814 learned_addr learned my addr 172.21.15.46:0/236315814 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:20.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc44b7fe640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc44c070a00 0x7fc44c0c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:20.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc450adb640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc44c1539a0 0x7fc44c0c4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:20.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc44bfff640 1 -- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc44c1539a0 msgr2=0x7fc44c0c4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:20.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc44bfff640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc44c1539a0 0x7fc44c0c4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:20.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc44bfff640 1 -- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc44c070a00 msgr2=0x7fc44c0c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:20.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc44bfff640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc44c070a00 0x7fc44c0c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:20.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc44bfff640 1 -- 172.21.15.46:0/236315814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4400094d0 con 0x7fc44c069e90 2023-10-30T13:50:20.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc44b7fe640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc44c070a00 0x7fc44c0c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:20.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc44bfff640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44c069e90 0x7fc44c146890 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00d800 tx=0x7fc43c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:20.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc448ff9640 1 -- 172.21.15.46:0/236315814 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c0096c0 con 0x7fc44c069e90 2023-10-30T13:50:20.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc448ff9640 1 -- 172.21.15.46:0/236315814 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43c009820 con 0x7fc44c069e90 2023-10-30T13:50:20.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc448ff9640 1 -- 172.21.15.46:0/236315814 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c009bd0 con 0x7fc44c069e90 2023-10-30T13:50:20.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc45253e640 1 -- 172.21.15.46:0/236315814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c159300 con 0x7fc44c069e90 2023-10-30T13:50:20.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc45253e640 1 -- 172.21.15.46:0/236315814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44c0c2420 con 0x7fc44c069e90 2023-10-30T13:50:20.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.709+0000 7fc45253e640 1 -- 172.21.15.46:0/236315814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44c04e170 con 0x7fc44c069e90 2023-10-30T13:50:20.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.713+0000 7fc448ff9640 1 -- 172.21.15.46:0/236315814 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc43c009d30 con 0x7fc44c069e90 2023-10-30T13:50:20.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.713+0000 7fc448ff9640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc4380664d0 0x7fc438068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:20.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.713+0000 7fc448ff9640 1 -- 172.21.15.46:0/236315814 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fc43c084ac0 con 0x7fc44c069e90 2023-10-30T13:50:20.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.713+0000 7fc44b7fe640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc4380664d0 0x7fc438068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:20.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.713+0000 7fc44b7fe640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc4380664d0 0x7fc438068990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc4400af000 tx=0x7fc4400a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:20.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.717+0000 7fc448ff9640 1 -- 172.21.15.46:0/236315814 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc43c0519b0 con 0x7fc44c069e90 2023-10-30T13:50:20.805 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:50:20.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:50:20.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.905+0000 7fc45253e640 1 -- 172.21.15.46:0/236315814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc44c0c72f0 con 0x7fc44c069e90 2023-10-30T13:50:20.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.905+0000 7fc448ff9640 1 -- 172.21.15.46:0/236315814 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+5735 (secure 0 0 0) 0x7fc43c054fd0 con 0x7fc44c069e90 2023-10-30T13:50:20.911 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:20.911 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:18.339102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:20.912 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 119 2023-10-30T13:50:20.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.909+0000 7fc45253e640 1 -- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc4380664d0 msgr2=0x7fc438068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:20.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.909+0000 7fc45253e640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc4380664d0 0x7fc438068990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc4400af000 tx=0x7fc4400a7000 comp rx=0 tx=0).stop 2023-10-30T13:50:20.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.909+0000 7fc45253e640 1 -- 172.21.15.46:0/236315814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44c069e90 msgr2=0x7fc44c146890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:20.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.909+0000 7fc45253e640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44c069e90 0x7fc44c146890 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00d800 tx=0x7fc43c00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:50:20.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.909+0000 7fc45253e640 1 -- 172.21.15.46:0/236315814 shutdown_connections 2023-10-30T13:50:20.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.909+0000 7fc45253e640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc4380664d0 0x7fc438068990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:20.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.909+0000 7fc45253e640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc44c1539a0 0x7fc44c0c4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:20.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.909+0000 7fc45253e640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc44c070a00 0x7fc44c0c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:20.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.909+0000 7fc45253e640 1 --2- 172.21.15.46:0/236315814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44c069e90 0x7fc44c146890 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:20.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.909+0000 7fc45253e640 1 -- 172.21.15.46:0/236315814 >> 172.21.15.46:0/236315814 conn(0x7fc44c0c7910 msgr2=0x7fc44c073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:20.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.909+0000 7fc45253e640 1 -- 172.21.15.46:0/236315814 shutdown_connections 2023-10-30T13:50:20.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:20.909+0000 7fc45253e640 1 -- 172.21.15.46:0/236315814 wait complete. 2023-10-30T13:50:20.932 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:50:22.933 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:23.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.085+0000 7fd0b5468640 1 Processor -- start 2023-10-30T13:50:23.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.085+0000 7fd0b5468640 1 -- start start 2023-10-30T13:50:23.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.085+0000 7fd0b5468640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0b01538a0 0x7fd0b015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:23.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.085+0000 7fd0b5468640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd0b00ca0e0 0x7fd0b015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:23.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.085+0000 7fd0b5468640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b00cb800 con 0x7fd0b00cf160 2023-10-30T13:50:23.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.085+0000 7fd0b5468640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b015dc30 con 0x7fd0b00ca0e0 2023-10-30T13:50:23.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.085+0000 7fd0b5468640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00cb330 con 0x7fd0b01538a0 2023-10-30T13:50:23.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.085+0000 7fd0aeffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0b01538a0 0x7fd0b015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:23.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.085+0000 7fd0a7fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd0b00ca0e0 0x7fd0b015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.089+0000 7fd0aeffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0b01538a0 0x7fd0b015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:38388/0 (socket says 172.21.15.46:38388) 2023-10-30T13:50:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.089+0000 7fd0aeffd640 1 -- 172.21.15.46:0/3680442409 learned_addr learned my addr 172.21.15.46:0/3680442409 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.089+0000 7fd0af7fe640 1 --1- 172.21.15.46:0/3680442409 >> v1:172.21.15.154:6789/0 conn(0x7fd0b00cf160 0x7fd0b00cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:44424/0 (socket says 172.21.15.46:44424) 2023-10-30T13:50:23.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.089+0000 7fd0a7fff640 1 -- 172.21.15.46:0/3680442409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0b01538a0 msgr2=0x7fd0b015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:23.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.089+0000 7fd0a7fff640 1 --2- 172.21.15.46:0/3680442409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0b01538a0 0x7fd0b015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:23.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.089+0000 7fd0a7fff640 1 -- 172.21.15.46:0/3680442409 >> v1:172.21.15.154:6789/0 conn(0x7fd0b00cf160 legacy=0x7fd0b00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:23.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.089+0000 7fd0a7fff640 1 -- 172.21.15.46:0/3680442409 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b00cad70 con 0x7fd0b00ca0e0 2023-10-30T13:50:23.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.089+0000 7fd0aeffd640 1 --2- 172.21.15.46:0/3680442409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0b01538a0 0x7fd0b015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:50:23.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.089+0000 7fd0a7fff640 1 --2- 172.21.15.46:0/3680442409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b00ca0e0 0x7fd0b015a090 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd0980097c0 tx=0x7fd09801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71fa11e3e9729510 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:23.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.089+0000 7fd0ae7fc640 1 -- 172.21.15.46:0/3680442409 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0980a8970 con 0x7fd0b00ca0e0 2023-10-30T13:50:23.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.089+0000 7fd0ae7fc640 1 -- 172.21.15.46:0/3680442409 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0980a8ad0 con 0x7fd0b00ca0e0 2023-10-30T13:50:23.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.089+0000 7fd0ae7fc640 1 -- 172.21.15.46:0/3680442409 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0980a8e60 con 0x7fd0b00ca0e0 2023-10-30T13:50:23.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 -- 172.21.15.46:0/3680442409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b00ca0e0 msgr2=0x7fd0b015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:23.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 --2- 172.21.15.46:0/3680442409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b00ca0e0 0x7fd0b015a090 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd0980097c0 tx=0x7fd09801ec70 comp rx=0 tx=0).stop 2023-10-30T13:50:23.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 -- 172.21.15.46:0/3680442409 shutdown_connections 2023-10-30T13:50:23.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 --2- 172.21.15.46:0/3680442409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b00ca0e0 0x7fd0b015a090 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:23.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 --2- 172.21.15.46:0/3680442409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0b01538a0 0x7fd0b015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:23.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 -- 172.21.15.46:0/3680442409 >> 172.21.15.46:0/3680442409 conn(0x7fd0b006bfe0 msgr2=0x7fd0b006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:23.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 -- 172.21.15.46:0/3680442409 shutdown_connections 2023-10-30T13:50:23.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 -- 172.21.15.46:0/3680442409 wait complete. 2023-10-30T13:50:23.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 Processor -- start 2023-10-30T13:50:23.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 -- start start 2023-10-30T13:50:23.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b00ca0e0 0x7fd0b0147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:23.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0b00cf160 0x7fd0b015c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:23.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b01538a0 0x7fd0b015ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:23.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b01f0c50 con 0x7fd0b01538a0 2023-10-30T13:50:23.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b01436f0 con 0x7fd0b00ca0e0 2023-10-30T13:50:23.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0b5468640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b015acf0 con 0x7fd0b00cf160 2023-10-30T13:50:23.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0af7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b01538a0 0x7fd0b015ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:23.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0af7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b01538a0 0x7fd0b015ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55552/0 (socket says 172.21.15.46:55552) 2023-10-30T13:50:23.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0af7fe640 1 -- 172.21.15.46:0/12655761 learned_addr learned my addr 172.21.15.46:0/12655761 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:23.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0aeffd640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b00ca0e0 0x7fd0b0147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:23.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.093+0000 7fd0a7fff640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0b00cf160 0x7fd0b015c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:23.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0af7fe640 1 -- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0b00cf160 msgr2=0x7fd0b015c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:23.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0af7fe640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0b00cf160 0x7fd0b015c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:23.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0af7fe640 1 -- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b00ca0e0 msgr2=0x7fd0b0147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:23.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0af7fe640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b00ca0e0 0x7fd0b0147e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:23.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0af7fe640 1 -- 172.21.15.46:0/12655761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a00094d0 con 0x7fd0b01538a0 2023-10-30T13:50:23.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0af7fe640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b01538a0 0x7fd0b015ca40 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800c730 tx=0x7fd0a800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:23.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0a77fe640 1 -- 172.21.15.46:0/12655761 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a800ce40 con 0x7fd0b01538a0 2023-10-30T13:50:23.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0a77fe640 1 -- 172.21.15.46:0/12655761 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a800a5e0 con 0x7fd0b01538a0 2023-10-30T13:50:23.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0a77fe640 1 -- 172.21.15.46:0/12655761 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a800a950 con 0x7fd0b01538a0 2023-10-30T13:50:23.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0b5468640 1 -- 172.21.15.46:0/12655761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd098009420 con 0x7fd0b01538a0 2023-10-30T13:50:23.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0b5468640 1 -- 172.21.15.46:0/12655761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b015b2b0 con 0x7fd0b01538a0 2023-10-30T13:50:23.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0b5468640 1 -- 172.21.15.46:0/12655761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b00b2a50 con 0x7fd0b01538a0 2023-10-30T13:50:23.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0a77fe640 1 -- 172.21.15.46:0/12655761 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0a800aab0 con 0x7fd0b01538a0 2023-10-30T13:50:23.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0a77fe640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0900664a0 0x7fd090068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:23.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0a77fe640 1 -- 172.21.15.46:0/12655761 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fd0a8083ae0 con 0x7fd0b01538a0 2023-10-30T13:50:23.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.097+0000 7fd0aeffd640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0900664a0 0x7fd090068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:23.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.101+0000 7fd0aeffd640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0900664a0 0x7fd090068960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0006cb0 tx=0x7fd0a0009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:23.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.101+0000 7fd0a77fe640 1 -- 172.21.15.46:0/12655761 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0a80509f0 con 0x7fd0b01538a0 2023-10-30T13:50:23.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.289+0000 7fd0b5468640 1 -- 172.21.15.46:0/12655761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0b01f0d90 con 0x7fd0b01538a0 2023-10-30T13:50:23.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0a77fe640 1 -- 172.21.15.46:0/12655761 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+5735 (secure 0 0 0) 0x7fd0b01f0d90 con 0x7fd0b01538a0 2023-10-30T13:50:23.296 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:23.296 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:18.339102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:23.296 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 119 2023-10-30T13:50:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0b5468640 1 -- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0900664a0 msgr2=0x7fd090068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0b5468640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0900664a0 0x7fd090068960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0006cb0 tx=0x7fd0a0009240 comp rx=0 tx=0).stop 2023-10-30T13:50:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0b5468640 1 -- 172.21.15.46:0/12655761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b01538a0 msgr2=0x7fd0b015ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:23.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0b5468640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b01538a0 0x7fd0b015ca40 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800c730 tx=0x7fd0a800cbf0 comp rx=0 tx=0).stop 2023-10-30T13:50:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0b5468640 1 -- 172.21.15.46:0/12655761 shutdown_connections 2023-10-30T13:50:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0b5468640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0900664a0 0x7fd090068960 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0b5468640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0b01538a0 0x7fd0b015ca40 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0b5468640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0b00cf160 0x7fd0b015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:23.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0b5468640 1 --2- 172.21.15.46:0/12655761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b00ca0e0 0x7fd0b0147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:23.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0b5468640 1 -- 172.21.15.46:0/12655761 >> 172.21.15.46:0/12655761 conn(0x7fd0b006bfe0 msgr2=0x7fd0b00cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:23.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0b5468640 1 -- 172.21.15.46:0/12655761 shutdown_connections 2023-10-30T13:50:23.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:23.293+0000 7fd0b5468640 1 -- 172.21.15.46:0/12655761 wait complete. 2023-10-30T13:50:23.317 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:50:25.318 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:25.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d184ea640 1 Processor -- start 2023-10-30T13:50:25.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d184ea640 1 -- start start 2023-10-30T13:50:25.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d184ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7d1015d6f0 0x7f7d1015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:25.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d184ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7d100cf160 0x7f7d100cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:25.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d184ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d100cb800 con 0x7f7d101538a0 2023-10-30T13:50:25.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d184ea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d100cb330 con 0x7f7d100cf160 2023-10-30T13:50:25.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d184ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d100cad70 con 0x7f7d1015d6f0 2023-10-30T13:50:25.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d16286640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7d101538a0 0x7f7d100ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34898/0 (socket says 172.21.15.46:34898) 2023-10-30T13:50:25.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d16286640 1 -- 172.21.15.46:0/417955311 learned_addr learned my addr 172.21.15.46:0/417955311 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:50:25.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d16a87640 1 --2- 172.21.15.46:0/417955311 >> v2:172.21.15.46:3300/0 conn(0x7f7d100cf160 0x7f7d100cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:25.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d15a85640 1 --2- 172.21.15.46:0/417955311 >> v2:172.21.15.154:3300/0 conn(0x7f7d1015d6f0 0x7f7d1015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:25.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d16a87640 1 -- 172.21.15.46:0/417955311 >> v2:172.21.15.154:3300/0 conn(0x7f7d1015d6f0 msgr2=0x7f7d1015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:25.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d16a87640 1 --2- 172.21.15.46:0/417955311 >> v2:172.21.15.154:3300/0 conn(0x7f7d1015d6f0 0x7f7d1015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:25.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d15284640 1 -- 172.21.15.46:0/417955311 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668157834 0 0) 0x7f7d100cb800 con 0x7f7d101538a0 2023-10-30T13:50:25.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d16a87640 1 -- 172.21.15.46:0/417955311 >> v1:172.21.15.46:6789/0 conn(0x7f7d101538a0 legacy=0x7f7d100ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:25.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.469+0000 7f7d16a87640 1 -- 172.21.15.46:0/417955311 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d100cb050 con 0x7f7d100cf160 2023-10-30T13:50:25.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.473+0000 7f7d16a87640 1 --2- 172.21.15.46:0/417955311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d100cf160 0x7f7d100cf590 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f7d100b30a0 tx=0x7f7d0801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6513a16043778af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.473+0000 7f7d15284640 1 -- 172.21.15.46:0/417955311 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d080b0440 con 0x7f7d100cf160 2023-10-30T13:50:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.473+0000 7f7d15284640 1 -- 172.21.15.46:0/417955311 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d080b05a0 con 0x7f7d100cf160 2023-10-30T13:50:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.473+0000 7f7d15284640 1 -- 172.21.15.46:0/417955311 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d080b0930 con 0x7f7d100cf160 2023-10-30T13:50:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.473+0000 7f7d184ea640 1 -- 172.21.15.46:0/417955311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d100cf160 msgr2=0x7f7d100cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.473+0000 7f7d184ea640 1 --2- 172.21.15.46:0/417955311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d100cf160 0x7f7d100cf590 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f7d100b30a0 tx=0x7f7d0801f0f0 comp rx=0 tx=0).stop 2023-10-30T13:50:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.473+0000 7f7d184ea640 1 -- 172.21.15.46:0/417955311 shutdown_connections 2023-10-30T13:50:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.473+0000 7f7d184ea640 1 --2- 172.21.15.46:0/417955311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d100cf160 0x7f7d100cf590 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.473+0000 7f7d184ea640 1 --2- 172.21.15.46:0/417955311 >> v2:172.21.15.154:3300/0 conn(0x7f7d1015d6f0 0x7f7d1015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.473+0000 7f7d184ea640 1 -- 172.21.15.46:0/417955311 >> 172.21.15.46:0/417955311 conn(0x7f7d1006bfe0 msgr2=0x7f7d1006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.473+0000 7f7d184ea640 1 -- 172.21.15.46:0/417955311 shutdown_connections 2023-10-30T13:50:25.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.473+0000 7f7d184ea640 1 -- 172.21.15.46:0/417955311 wait complete. 2023-10-30T13:50:25.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d184ea640 1 Processor -- start 2023-10-30T13:50:25.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d184ea640 1 -- start start 2023-10-30T13:50:25.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d184ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d100cf160 0x7f7d101542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:25.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d184ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d101538a0 0x7f7d100b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:25.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d184ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d1015d6f0 0x7f7d101ef880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:25.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d184ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d10156c20 con 0x7f7d101538a0 2023-10-30T13:50:25.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d184ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d101f3690 con 0x7f7d1015d6f0 2023-10-30T13:50:25.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d184ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d101f3110 con 0x7f7d100cf160 2023-10-30T13:50:25.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d15a85640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d101538a0 0x7f7d100b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:25.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d15a85640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d101538a0 0x7f7d100b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55560/0 (socket says 172.21.15.46:55560) 2023-10-30T13:50:25.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d15a85640 1 -- 172.21.15.46:0/1831386524 learned_addr learned my addr 172.21.15.46:0/1831386524 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:25.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d16a87640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d1015d6f0 0x7f7d101ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:25.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d16286640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d100cf160 0x7f7d101542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:25.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d15a85640 1 -- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d100cf160 msgr2=0x7f7d101542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:25.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d15a85640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d100cf160 0x7f7d101542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:25.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d15a85640 1 -- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d1015d6f0 msgr2=0x7f7d101ef880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:25.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d15a85640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d1015d6f0 0x7f7d101ef880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:25.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d15a85640 1 -- 172.21.15.46:0/1831386524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d08009a70 con 0x7f7d101538a0 2023-10-30T13:50:25.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d15a85640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d101538a0 0x7f7d100b4580 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000d5f0 tx=0x7f7d0000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:25.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d06ffd640 1 -- 172.21.15.46:0/1831386524 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d000125f0 con 0x7f7d101538a0 2023-10-30T13:50:25.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d06ffd640 1 -- 172.21.15.46:0/1831386524 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d00012750 con 0x7f7d101538a0 2023-10-30T13:50:25.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d06ffd640 1 -- 172.21.15.46:0/1831386524 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d00012ae0 con 0x7f7d101538a0 2023-10-30T13:50:25.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d184ea640 1 -- 172.21.15.46:0/1831386524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d10154800 con 0x7f7d101538a0 2023-10-30T13:50:25.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.477+0000 7f7d184ea640 1 -- 172.21.15.46:0/1831386524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d101f33b0 con 0x7f7d101538a0 2023-10-30T13:50:25.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.481+0000 7f7d184ea640 1 -- 172.21.15.46:0/1831386524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d100b2a50 con 0x7f7d101538a0 2023-10-30T13:50:25.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.481+0000 7f7d06ffd640 1 -- 172.21.15.46:0/1831386524 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7d00012c40 con 0x7f7d101538a0 2023-10-30T13:50:25.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.481+0000 7f7d06ffd640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7cf40664a0 0x7f7cf4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:25.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.481+0000 7f7d06ffd640 1 -- 172.21.15.46:0/1831386524 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f7d00084910 con 0x7f7d101538a0 2023-10-30T13:50:25.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.481+0000 7f7d16286640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7cf40664a0 0x7f7cf4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:25.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.481+0000 7f7d16286640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7cf40664a0 0x7f7cf4068960 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c000f70 tx=0x7f7d0c007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:25.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.485+0000 7f7d06ffd640 1 -- 172.21.15.46:0/1831386524 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7d00051800 con 0x7f7d101538a0 2023-10-30T13:50:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 -- 172.21.15.46:0/1831386524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d1004e170 con 0x7f7d101538a0 2023-10-30T13:50:25.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d06ffd640 1 -- 172.21.15.46:0/1831386524 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f7d00007380 con 0x7f7d101538a0 2023-10-30T13:50:25.681 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:25.681 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:24.329817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:25.682 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 120 2023-10-30T13:50:25.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 -- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7cf40664a0 msgr2=0x7f7cf4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:25.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7cf40664a0 0x7f7cf4068960 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c000f70 tx=0x7f7d0c007b60 comp rx=0 tx=0).stop 2023-10-30T13:50:25.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 -- 172.21.15.46:0/1831386524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d101538a0 msgr2=0x7f7d100b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:25.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d101538a0 0x7f7d100b4580 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000d5f0 tx=0x7f7d0000dab0 comp rx=0 tx=0).stop 2023-10-30T13:50:25.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 -- 172.21.15.46:0/1831386524 shutdown_connections 2023-10-30T13:50:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7cf40664a0 0x7f7cf4068960 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d1015d6f0 0x7f7d101ef880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d101538a0 0x7f7d100b4580 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 --2- 172.21.15.46:0/1831386524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d100cf160 0x7f7d101542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:25.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 -- 172.21.15.46:0/1831386524 >> 172.21.15.46:0/1831386524 conn(0x7f7d1006bfe0 msgr2=0x7f7d1015aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:25.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 -- 172.21.15.46:0/1831386524 shutdown_connections 2023-10-30T13:50:25.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:25.677+0000 7f7d184ea640 1 -- 172.21.15.46:0/1831386524 wait complete. 2023-10-30T13:50:25.702 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:50:27.703 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:27.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.853+0000 7ff57533e640 1 Processor -- start 2023-10-30T13:50:27.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.853+0000 7ff57533e640 1 -- start start 2023-10-30T13:50:27.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.853+0000 7ff57533e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff570153d70 0x7ff57006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:27.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.853+0000 7ff57533e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff5700c8490 0x7ff5700c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:27.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.853+0000 7ff57533e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57006afa0 con 0x7ff57006e470 2023-10-30T13:50:27.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.853+0000 7ff57533e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57006d780 con 0x7ff570153d70 2023-10-30T13:50:27.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.853+0000 7ff57533e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff570154150 con 0x7ff5700c8490 2023-10-30T13:50:27.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.853+0000 7ff56effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff570153d70 0x7ff57006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:27.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.857+0000 7ff56effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff570153d70 0x7ff57006ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55886/0 (socket says 172.21.15.46:55886) 2023-10-30T13:50:27.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.857+0000 7ff56effd640 1 -- 172.21.15.46:0/1372835649 learned_addr learned my addr 172.21.15.46:0/1372835649 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:27.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.857+0000 7ff56f7fe640 1 --2- 172.21.15.46:0/1372835649 >> v2:172.21.15.154:3300/0 conn(0x7ff5700c8490 0x7ff5700c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:27.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.857+0000 7ff56effd640 1 -- 172.21.15.46:0/1372835649 >> v2:172.21.15.154:3300/0 conn(0x7ff5700c8490 msgr2=0x7ff5700c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:27.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.857+0000 7ff56effd640 1 --2- 172.21.15.46:0/1372835649 >> v2:172.21.15.154:3300/0 conn(0x7ff5700c8490 0x7ff5700c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:27.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.857+0000 7ff56effd640 1 -- 172.21.15.46:0/1372835649 >> v1:172.21.15.154:6789/0 conn(0x7ff57006e470 legacy=0x7ff5700cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:27.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.857+0000 7ff56effd640 1 -- 172.21.15.46:0/1372835649 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57006e030 con 0x7ff570153d70 2023-10-30T13:50:27.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.857+0000 7ff56effd640 1 --2- 172.21.15.46:0/1372835649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff570153d70 0x7ff57006ca80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7ff5700cddf0 tx=0x7ff56001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4abb5e8d7bc3afd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:27.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.857+0000 7ff56e7fc640 1 -- 172.21.15.46:0/1372835649 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5600aa5f0 con 0x7ff570153d70 2023-10-30T13:50:27.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.857+0000 7ff56e7fc640 1 -- 172.21.15.46:0/1372835649 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5600aa750 con 0x7ff570153d70 2023-10-30T13:50:27.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.857+0000 7ff56e7fc640 1 -- 172.21.15.46:0/1372835649 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5600aaae0 con 0x7ff570153d70 2023-10-30T13:50:27.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 -- 172.21.15.46:0/1372835649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff570153d70 msgr2=0x7ff57006ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:27.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 --2- 172.21.15.46:0/1372835649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff570153d70 0x7ff57006ca80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7ff5700cddf0 tx=0x7ff56001c490 comp rx=0 tx=0).stop 2023-10-30T13:50:27.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 -- 172.21.15.46:0/1372835649 shutdown_connections 2023-10-30T13:50:27.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 --2- 172.21.15.46:0/1372835649 >> v2:172.21.15.154:3300/0 conn(0x7ff5700c8490 0x7ff5700c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:27.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 --2- 172.21.15.46:0/1372835649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff570153d70 0x7ff57006ca80 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:27.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 -- 172.21.15.46:0/1372835649 >> 172.21.15.46:0/1372835649 conn(0x7ff570071ea0 msgr2=0x7ff570065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:27.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 -- 172.21.15.46:0/1372835649 shutdown_connections 2023-10-30T13:50:27.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 -- 172.21.15.46:0/1372835649 wait complete. 2023-10-30T13:50:27.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 Processor -- start 2023-10-30T13:50:27.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 -- start start 2023-10-30T13:50:27.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57006e470 0x7ff570145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:27.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5700c8490 0x7ff5700cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:27.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff570153d70 0x7ff5700cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:27.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff570146940 con 0x7ff57006e470 2023-10-30T13:50:27.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff570149b00 con 0x7ff5700c8490 2023-10-30T13:50:27.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff57533e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff570154b30 con 0x7ff570153d70 2023-10-30T13:50:27.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff56effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57006e470 0x7ff570145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff56effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57006e470 0x7ff570145cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55888/0 (socket says 172.21.15.46:55888) 2023-10-30T13:50:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff56effd640 1 -- 172.21.15.46:0/1324259923 learned_addr learned my addr 172.21.15.46:0/1324259923 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff567fff640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5700c8490 0x7ff5700cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff56f7fe640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff570153d70 0x7ff5700cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:27.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff56effd640 1 -- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff570153d70 msgr2=0x7ff5700cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff56effd640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff570153d70 0x7ff5700cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff56effd640 1 -- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5700c8490 msgr2=0x7ff5700cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff56effd640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5700c8490 0x7ff5700cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff56effd640 1 -- 172.21.15.46:0/1324259923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5600094d0 con 0x7ff57006e470 2023-10-30T13:50:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.861+0000 7ff56f7fe640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff570153d70 0x7ff5700cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:50:27.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.865+0000 7ff56effd640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57006e470 0x7ff570145cb0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7ff56001a0c0 tx=0x7ff56001acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:27.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.865+0000 7ff5677fe640 1 -- 172.21.15.46:0/1324259923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5600aa5f0 con 0x7ff57006e470 2023-10-30T13:50:27.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.865+0000 7ff5677fe640 1 -- 172.21.15.46:0/1324259923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5600b3800 con 0x7ff57006e470 2023-10-30T13:50:27.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.865+0000 7ff5677fe640 1 -- 172.21.15.46:0/1324259923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5600b3b90 con 0x7ff57006e470 2023-10-30T13:50:27.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.865+0000 7ff57533e640 1 -- 172.21.15.46:0/1324259923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff570149790 con 0x7ff57006e470 2023-10-30T13:50:27.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.865+0000 7ff57533e640 1 -- 172.21.15.46:0/1324259923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5701453c0 con 0x7ff57006e470 2023-10-30T13:50:27.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.865+0000 7ff57533e640 1 -- 172.21.15.46:0/1324259923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff570149e00 con 0x7ff57006e470 2023-10-30T13:50:27.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.865+0000 7ff5677fe640 1 -- 172.21.15.46:0/1324259923 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5600b3cf0 con 0x7ff57006e470 2023-10-30T13:50:27.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.865+0000 7ff5677fe640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5500664a0 0x7ff550068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:27.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.865+0000 7ff5677fe640 1 -- 172.21.15.46:0/1324259923 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7ff5600a7070 con 0x7ff57006e470 2023-10-30T13:50:27.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.865+0000 7ff567fff640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5500664a0 0x7ff550068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:27.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.869+0000 7ff567fff640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5500664a0 0x7ff550068960 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff5700c8410 tx=0x7ff558008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:27.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:27.869+0000 7ff5677fe640 1 -- 172.21.15.46:0/1324259923 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5600eb8d0 con 0x7ff57006e470 2023-10-30T13:50:28.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.057+0000 7ff57533e640 1 -- 172.21.15.46:0/1324259923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff57004e170 con 0x7ff57006e470 2023-10-30T13:50:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff5677fe640 1 -- 172.21.15.46:0/1324259923 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5732 (secure 0 0 0) 0x7ff5600eeef0 con 0x7ff57006e470 2023-10-30T13:50:28.064 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:28.064 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:24.329817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 120 2023-10-30T13:50:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff57533e640 1 -- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5500664a0 msgr2=0x7ff550068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff57533e640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5500664a0 0x7ff550068960 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff5700c8410 tx=0x7ff558008040 comp rx=0 tx=0).stop 2023-10-30T13:50:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff57533e640 1 -- 172.21.15.46:0/1324259923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57006e470 msgr2=0x7ff570145cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff57533e640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57006e470 0x7ff570145cb0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7ff56001a0c0 tx=0x7ff56001acd0 comp rx=0 tx=0).stop 2023-10-30T13:50:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff57533e640 1 -- 172.21.15.46:0/1324259923 shutdown_connections 2023-10-30T13:50:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff57533e640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5500664a0 0x7ff550068960 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff57533e640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff570153d70 0x7ff5700cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff57533e640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5700c8490 0x7ff5700cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff57533e640 1 --2- 172.21.15.46:0/1324259923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57006e470 0x7ff570145cb0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff57533e640 1 -- 172.21.15.46:0/1324259923 >> 172.21.15.46:0/1324259923 conn(0x7ff570071ea0 msgr2=0x7ff570151ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:28.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff57533e640 1 -- 172.21.15.46:0/1324259923 shutdown_connections 2023-10-30T13:50:28.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:28.061+0000 7ff57533e640 1 -- 172.21.15.46:0/1324259923 wait complete. 2023-10-30T13:50:28.085 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:50:30.086 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:30.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.245+0000 7fcf543aa640 1 Processor -- start 2023-10-30T13:50:30.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.245+0000 7fcf543aa640 1 -- start start 2023-10-30T13:50:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.245+0000 7fcf543aa640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcf4c06f7a0 0x7fcf4c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.245+0000 7fcf543aa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c066cf0 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.245+0000 7fcf543aa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c0c70d0 con 0x7fcf4c071110 2023-10-30T13:50:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.245+0000 7fcf543aa640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c0700c0 con 0x7fcf4c06f7a0 2023-10-30T13:50:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.245+0000 7fcf52146640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcf4c06f7a0 0x7fcf4c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.245+0000 7fcf52146640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcf4c06f7a0 0x7fcf4c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:51356/0 (socket says 172.21.15.46:51356) 2023-10-30T13:50:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.245+0000 7fcf52146640 1 -- 172.21.15.46:0/493235171 learned_addr learned my addr 172.21.15.46:0/493235171 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:30.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.249+0000 7fcf51144640 1 -- 172.21.15.46:0/493235171 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745482119 0 0) 0x7fcf4c0c70d0 con 0x7fcf4c071110 2023-10-30T13:50:30.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.249+0000 7fcf51144640 1 -- 172.21.15.46:0/493235171 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf38003530 con 0x7fcf4c071110 2023-10-30T13:50:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.249+0000 7fcf51144640 1 -- 172.21.15.46:0/493235171 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989625194 0 0) 0x7fcf4c066cf0 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.249+0000 7fcf51144640 1 -- 172.21.15.46:0/493235171 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf4c0c70d0 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.249+0000 7fcf52146640 1 -- 172.21.15.46:0/493235171 >> v1:172.21.15.154:6789/0 conn(0x7fcf4c071110 legacy=0x7fcf4c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.249+0000 7fcf52146640 1 -- 172.21.15.46:0/493235171 >> v1:172.21.15.46:6789/0 conn(0x7fcf4c0c6bd0 legacy=0x7fcf4c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.249+0000 7fcf52146640 1 -- 172.21.15.46:0/493235171 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf4c0c7390 con 0x7fcf4c06f7a0 2023-10-30T13:50:30.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.253+0000 7fcf52146640 1 --2- 172.21.15.46:0/493235171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c06f7a0 0x7fcf4c06fb80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009870 tx=0x7fcf4801ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f09f917b359bad7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:30.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.253+0000 7fcf51144640 1 -- 172.21.15.46:0/493235171 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf480a8a30 con 0x7fcf4c06f7a0 2023-10-30T13:50:30.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.253+0000 7fcf51144640 1 -- 172.21.15.46:0/493235171 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf480a8b90 con 0x7fcf4c06f7a0 2023-10-30T13:50:30.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.253+0000 7fcf51144640 1 -- 172.21.15.46:0/493235171 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf480b0440 con 0x7fcf4c06f7a0 2023-10-30T13:50:30.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.253+0000 7fcf543aa640 1 -- 172.21.15.46:0/493235171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c06f7a0 msgr2=0x7fcf4c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:30.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.253+0000 7fcf543aa640 1 --2- 172.21.15.46:0/493235171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c06f7a0 0x7fcf4c06fb80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009870 tx=0x7fcf4801ed50 comp rx=0 tx=0).stop 2023-10-30T13:50:30.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.253+0000 7fcf543aa640 1 -- 172.21.15.46:0/493235171 shutdown_connections 2023-10-30T13:50:30.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.253+0000 7fcf543aa640 1 --2- 172.21.15.46:0/493235171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c06f7a0 0x7fcf4c06fb80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:30.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.253+0000 7fcf543aa640 1 -- 172.21.15.46:0/493235171 >> 172.21.15.46:0/493235171 conn(0x7fcf4c0c7910 msgr2=0x7fcf4c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:30.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.253+0000 7fcf543aa640 1 -- 172.21.15.46:0/493235171 shutdown_connections 2023-10-30T13:50:30.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.253+0000 7fcf543aa640 1 -- 172.21.15.46:0/493235171 wait complete. 2023-10-30T13:50:30.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf543aa640 1 Processor -- start 2023-10-30T13:50:30.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf543aa640 1 -- start start 2023-10-30T13:50:30.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf543aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf4c06f7a0 0x7fcf4c141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:30.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf543aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c071110 0x7fcf4c1396d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:30.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf543aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf4c0c6bd0 0x7fcf4c139c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:30.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf543aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c0c1a70 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf543aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c141cc0 con 0x7fcf4c071110 2023-10-30T13:50:30.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf543aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c0c1ca0 con 0x7fcf4c06f7a0 2023-10-30T13:50:30.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf52947640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf4c0c6bd0 0x7fcf4c139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:30.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf52947640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf4c0c6bd0 0x7fcf4c139c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55890/0 (socket says 172.21.15.46:55890) 2023-10-30T13:50:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf52947640 1 -- 172.21.15.46:0/2173554225 learned_addr learned my addr 172.21.15.46:0/2173554225 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf51945640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c071110 0x7fcf4c1396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf52947640 1 -- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf4c06f7a0 msgr2=0x7fcf4c141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf52947640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf4c06f7a0 0x7fcf4c141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf52947640 1 -- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c071110 msgr2=0x7fcf4c1396d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:30.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf52947640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c071110 0x7fcf4c1396d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:30.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf52947640 1 -- 172.21.15.46:0/2173554225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf480094d0 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf52947640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf4c0c6bd0 0x7fcf4c139c10 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400c700 tx=0x7fcf4400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:30.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf42ffd640 1 -- 172.21.15.46:0/2173554225 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44003b80 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf42ffd640 1 -- 172.21.15.46:0/2173554225 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf44003ce0 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf42ffd640 1 -- 172.21.15.46:0/2173554225 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4400aa70 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf543aa640 1 -- 172.21.15.46:0/2173554225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf4c0c1f80 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.257+0000 7fcf543aa640 1 -- 172.21.15.46:0/2173554225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf4c0c0ec0 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.261+0000 7fcf543aa640 1 -- 172.21.15.46:0/2173554225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf4c04e170 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.261+0000 7fcf42ffd640 1 -- 172.21.15.46:0/2173554225 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcf4400abd0 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.261+0000 7fcf42ffd640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcf300664a0 0x7fcf30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:30.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.261+0000 7fcf42ffd640 1 -- 172.21.15.46:0/2173554225 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fcf440840d0 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.261+0000 7fcf52146640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcf300664a0 0x7fcf30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:30.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.265+0000 7fcf52146640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcf300664a0 0x7fcf30068960 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcf480b2000 tx=0x7fcf48004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:30.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.265+0000 7fcf42ffd640 1 -- 172.21.15.46:0/2173554225 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcf44051060 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.457+0000 7fcf543aa640 1 -- 172.21.15.46:0/2173554225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf4c0c8100 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.457+0000 7fcf42ffd640 1 -- 172.21.15.46:0/2173554225 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fcf44054680 con 0x7fcf4c0c6bd0 2023-10-30T13:50:30.463 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:30.463 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:24.329817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:30.463 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 120 2023-10-30T13:50:30.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.461+0000 7fcf543aa640 1 -- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcf300664a0 msgr2=0x7fcf30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:30.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.461+0000 7fcf543aa640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcf300664a0 0x7fcf30068960 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcf480b2000 tx=0x7fcf48004740 comp rx=0 tx=0).stop 2023-10-30T13:50:30.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.461+0000 7fcf543aa640 1 -- 172.21.15.46:0/2173554225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf4c0c6bd0 msgr2=0x7fcf4c139c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:30.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.461+0000 7fcf543aa640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf4c0c6bd0 0x7fcf4c139c10 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400c700 tx=0x7fcf4400cbc0 comp rx=0 tx=0).stop 2023-10-30T13:50:30.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.461+0000 7fcf543aa640 1 -- 172.21.15.46:0/2173554225 shutdown_connections 2023-10-30T13:50:30.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.461+0000 7fcf543aa640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcf300664a0 0x7fcf30068960 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:30.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.461+0000 7fcf543aa640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf4c0c6bd0 0x7fcf4c139c10 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:30.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.461+0000 7fcf543aa640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf4c071110 0x7fcf4c1396d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:30.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.461+0000 7fcf543aa640 1 --2- 172.21.15.46:0/2173554225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf4c06f7a0 0x7fcf4c141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:30.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.461+0000 7fcf543aa640 1 -- 172.21.15.46:0/2173554225 >> 172.21.15.46:0/2173554225 conn(0x7fcf4c0c7910 msgr2=0x7fcf4c154b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:30.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.461+0000 7fcf543aa640 1 -- 172.21.15.46:0/2173554225 shutdown_connections 2023-10-30T13:50:30.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:30.461+0000 7fcf543aa640 1 -- 172.21.15.46:0/2173554225 wait complete. 2023-10-30T13:50:30.485 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 120, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 120, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6702": { "addr": "172.21.15.46:6836/2197729559", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 2197729559, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 2197729559, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6702, "incarnation": 111, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 43 }, "gid_6858": { "addr": "172.21.15.46:6837/2572113428", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2572113428, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2572113428, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6858, "incarnation": 117, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 39 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:50:24.329817+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6702, "mds_1": 6858 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:50:32.485 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:32.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fed00b0f640 1 Processor -- start 2023-10-30T13:50:32.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fed00b0f640 1 -- start start 2023-10-30T13:50:32.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fed00b0f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fecfc15d6f0 0x7fecfc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:32.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fed00b0f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fecfc0cf160 0x7fecfc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:32.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fed00b0f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecfc0cb800 con 0x7fecfc1538a0 2023-10-30T13:50:32.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fed00b0f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecfc0cb330 con 0x7fecfc15d6f0 2023-10-30T13:50:32.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fed00b0f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecfc0cad70 con 0x7fecfc0cf160 2023-10-30T13:50:32.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fecf3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fecfc15d6f0 0x7fecfc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:32.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fecf3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fecfc15d6f0 0x7fecfc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55906/0 (socket says 172.21.15.46:55906) 2023-10-30T13:50:32.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fecf3fff640 1 -- 172.21.15.46:0/793402217 learned_addr learned my addr 172.21.15.46:0/793402217 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:32.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fecfad9d640 1 --2- 172.21.15.46:0/793402217 >> v2:172.21.15.154:3300/0 conn(0x7fecfc0cf160 0x7fecfc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:32.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fecf3fff640 1 -- 172.21.15.46:0/793402217 >> v2:172.21.15.154:3300/0 conn(0x7fecfc0cf160 msgr2=0x7fecfc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:32.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fecf3fff640 1 --2- 172.21.15.46:0/793402217 >> v2:172.21.15.154:3300/0 conn(0x7fecfc0cf160 0x7fecfc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fecf3fff640 1 -- 172.21.15.46:0/793402217 >> v1:172.21.15.154:6789/0 conn(0x7fecfc1538a0 legacy=0x7fecfc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fecf3fff640 1 -- 172.21.15.46:0/793402217 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0cb050 con 0x7fecfc15d6f0 2023-10-30T13:50:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.637+0000 7fecfad9d640 1 --2- 172.21.15.46:0/793402217 >> v2:172.21.15.154:3300/0 conn(0x7fecfc0cf160 0x7fecfc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.641+0000 7fecf3fff640 1 --2- 172.21.15.46:0/793402217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecfc15d6f0 0x7fecfc15a090 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fecfc0b5330 tx=0x7fece401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e28fc96711bd0a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.641+0000 7fecf9d9b640 1 -- 172.21.15.46:0/793402217 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece40a89f0 con 0x7fecfc15d6f0 2023-10-30T13:50:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.641+0000 7fecf9d9b640 1 -- 172.21.15.46:0/793402217 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece40a8b50 con 0x7fecfc15d6f0 2023-10-30T13:50:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.641+0000 7fecf9d9b640 1 -- 172.21.15.46:0/793402217 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece40b0440 con 0x7fecfc15d6f0 2023-10-30T13:50:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.641+0000 7fed00b0f640 1 -- 172.21.15.46:0/793402217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecfc15d6f0 msgr2=0x7fecfc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.641+0000 7fed00b0f640 1 --2- 172.21.15.46:0/793402217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecfc15d6f0 0x7fecfc15a090 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fecfc0b5330 tx=0x7fece401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.641+0000 7fed00b0f640 1 -- 172.21.15.46:0/793402217 shutdown_connections 2023-10-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.641+0000 7fed00b0f640 1 --2- 172.21.15.46:0/793402217 >> v2:172.21.15.154:3300/0 conn(0x7fecfc0cf160 0x7fecfc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.641+0000 7fed00b0f640 1 --2- 172.21.15.46:0/793402217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecfc15d6f0 0x7fecfc15a090 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:32.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.641+0000 7fed00b0f640 1 -- 172.21.15.46:0/793402217 >> 172.21.15.46:0/793402217 conn(0x7fecfc06bfe0 msgr2=0x7fecfc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:32.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.641+0000 7fed00b0f640 1 -- 172.21.15.46:0/793402217 shutdown_connections 2023-10-30T13:50:32.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.641+0000 7fed00b0f640 1 -- 172.21.15.46:0/793402217 wait complete. 2023-10-30T13:50:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fed00b0f640 1 Processor -- start 2023-10-30T13:50:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fed00b0f640 1 -- start start 2023-10-30T13:50:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fed00b0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecfc0cf160 0x7fecfc15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fed00b0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecfc1538a0 0x7fecfc0b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fed00b0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecfc15d6f0 0x7fecfc0b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fed00b0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc15ca10 con 0x7fecfc15d6f0 2023-10-30T13:50:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fed00b0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc0b4d90 con 0x7fecfc1538a0 2023-10-30T13:50:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fed00b0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc0b31b0 con 0x7fecfc0cf160 2023-10-30T13:50:32.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecfad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecfc15d6f0 0x7fecfc0b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:32.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecfad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecfc15d6f0 0x7fecfc0b44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55920/0 (socket says 172.21.15.46:55920) 2023-10-30T13:50:32.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecfad9d640 1 -- 172.21.15.46:0/489764053 learned_addr learned my addr 172.21.15.46:0/489764053 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:32.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecfa59c640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecfc0cf160 0x7fecfc15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:32.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecf3fff640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecfc1538a0 0x7fecfc0b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:32.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecfad9d640 1 -- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecfc0cf160 msgr2=0x7fecfc15bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:32.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecfad9d640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecfc0cf160 0x7fecfc15bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:32.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecfad9d640 1 -- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecfc1538a0 msgr2=0x7fecfc0b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:32.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecfad9d640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecfc1538a0 0x7fecfc0b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:32.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecfad9d640 1 -- 172.21.15.46:0/489764053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece40094d0 con 0x7fecfc15d6f0 2023-10-30T13:50:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecfad9d640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecfc15d6f0 0x7fecfc0b44b0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fecf4007de0 tx=0x7fecf400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecf2ffd640 1 -- 172.21.15.46:0/489764053 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf40039a0 con 0x7fecfc15d6f0 2023-10-30T13:50:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecf2ffd640 1 -- 172.21.15.46:0/489764053 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf400f040 con 0x7fecfc15d6f0 2023-10-30T13:50:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fecf2ffd640 1 -- 172.21.15.46:0/489764053 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4003d40 con 0x7fecfc15d6f0 2023-10-30T13:50:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fed00b0f640 1 -- 172.21.15.46:0/489764053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecfc0b3490 con 0x7fecfc15d6f0 2023-10-30T13:50:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.645+0000 7fed00b0f640 1 -- 172.21.15.46:0/489764053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc0b3a50 con 0x7fecfc15d6f0 2023-10-30T13:50:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.649+0000 7fed00b0f640 1 -- 172.21.15.46:0/489764053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecfc0b2a50 con 0x7fecfc15d6f0 2023-10-30T13:50:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.649+0000 7fecf2ffd640 1 -- 172.21.15.46:0/489764053 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fecf4004030 con 0x7fecfc15d6f0 2023-10-30T13:50:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.649+0000 7fecf2ffd640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecdc0664d0 0x7fecdc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.649+0000 7fecf2ffd640 1 -- 172.21.15.46:0/489764053 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fecf4084650 con 0x7fecfc15d6f0 2023-10-30T13:50:32.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.649+0000 7fecfa59c640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecdc0664d0 0x7fecdc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:32.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.649+0000 7fecfa59c640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecdc0664d0 0x7fecdc068990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fecec007fd0 tx=0x7fecec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:32.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.653+0000 7fecf2ffd640 1 -- 172.21.15.46:0/489764053 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fecf4051690 con 0x7fecfc15d6f0 2023-10-30T13:50:32.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 -- 172.21.15.46:0/489764053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecfc04e170 con 0x7fecfc15d6f0 2023-10-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fecf2ffd640 1 -- 172.21.15.46:0/489764053 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fecf4054cb0 con 0x7fecfc15d6f0 2023-10-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:32.849 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:30.306260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:32.850 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 121 2023-10-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 -- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecdc0664d0 msgr2=0x7fecdc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecdc0664d0 0x7fecdc068990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fecec007fd0 tx=0x7fecec009040 comp rx=0 tx=0).stop 2023-10-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 -- 172.21.15.46:0/489764053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecfc15d6f0 msgr2=0x7fecfc0b44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecfc15d6f0 0x7fecfc0b44b0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fecf4007de0 tx=0x7fecf400d4a0 comp rx=0 tx=0).stop 2023-10-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 -- 172.21.15.46:0/489764053 shutdown_connections 2023-10-30T13:50:32.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecdc0664d0 0x7fecdc068990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecfc15d6f0 0x7fecfc0b44b0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecfc1538a0 0x7fecfc0b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 --2- 172.21.15.46:0/489764053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecfc0cf160 0x7fecfc15bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 -- 172.21.15.46:0/489764053 >> 172.21.15.46:0/489764053 conn(0x7fecfc06bfe0 msgr2=0x7fecfc0cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:32.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 -- 172.21.15.46:0/489764053 shutdown_connections 2023-10-30T13:50:32.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:32.845+0000 7fed00b0f640 1 -- 172.21.15.46:0/489764053 wait complete. 2023-10-30T13:50:32.870 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=1, replacing gid=6366 2023-10-30T13:50:32.871 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.a 2023-10-30T13:50:37.872 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:50:37.872 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:50:37.872 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:50:37.876 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:50:37.876 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:37.969 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-10-30T13:50:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.017+0000 7f4f64b3e640 1 Processor -- start 2023-10-30T13:50:38.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.017+0000 7f4f64b3e640 1 -- start start 2023-10-30T13:50:38.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.017+0000 7f4f64b3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f600c6b80 0x7f4f600bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:38.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.017+0000 7f4f64b3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4f600c6f60 0x7f4f600c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:38.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.017+0000 7f4f64b3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f600c0a50 con 0x7f4f600c8430 2023-10-30T13:50:38.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f64b3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6006f6e0 con 0x7f4f600c6b80 2023-10-30T13:50:38.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f64b3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6006f9c0 con 0x7f4f600c6f60 2023-10-30T13:50:38.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f5e59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f600c6b80 0x7f4f600bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:38.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f5ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4f600c8430 0x7f4f6013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52374/0 (socket says 172.21.15.46:52374) 2023-10-30T13:50:38.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f5ed9d640 1 -- 172.21.15.46:0/381032906 learned_addr learned my addr 172.21.15.46:0/381032906 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:50:38.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f5dd9b640 1 --2- 172.21.15.46:0/381032906 >> v2:172.21.15.154:3300/0 conn(0x7f4f600c6f60 0x7f4f600c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:38.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f5d59a640 1 -- 172.21.15.46:0/381032906 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973339237 0 0) 0x7f4f600c0a50 con 0x7f4f600c8430 2023-10-30T13:50:38.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f5d59a640 1 -- 172.21.15.46:0/381032906 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f48003530 con 0x7f4f600c8430 2023-10-30T13:50:38.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f5e59c640 1 -- 172.21.15.46:0/381032906 >> v2:172.21.15.154:3300/0 conn(0x7f4f600c6f60 msgr2=0x7f4f600c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:38.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f5e59c640 1 --2- 172.21.15.46:0/381032906 >> v2:172.21.15.154:3300/0 conn(0x7f4f600c6f60 0x7f4f600c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f5e59c640 1 -- 172.21.15.46:0/381032906 >> v1:172.21.15.46:6789/0 conn(0x7f4f600c8430 legacy=0x7f4f6013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f5e59c640 1 -- 172.21.15.46:0/381032906 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6006fca0 con 0x7f4f600c6b80 2023-10-30T13:50:38.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f5e59c640 1 --2- 172.21.15.46:0/381032906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f600c6b80 0x7f4f600bfec0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f4f60159b90 tx=0x7f4f5001e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0bb6d9cdf51b8c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:38.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.021+0000 7f4f5d59a640 1 -- 172.21.15.46:0/381032906 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500a7870 con 0x7f4f600c6b80 2023-10-30T13:50:38.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f5d59a640 1 -- 172.21.15.46:0/381032906 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f500a79d0 con 0x7f4f600c6b80 2023-10-30T13:50:38.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f5d59a640 1 -- 172.21.15.46:0/381032906 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500a7d60 con 0x7f4f600c6b80 2023-10-30T13:50:38.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 -- 172.21.15.46:0/381032906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f600c6b80 msgr2=0x7f4f600bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:38.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 --2- 172.21.15.46:0/381032906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f600c6b80 0x7f4f600bfec0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f4f60159b90 tx=0x7f4f5001e890 comp rx=0 tx=0).stop 2023-10-30T13:50:38.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 -- 172.21.15.46:0/381032906 shutdown_connections 2023-10-30T13:50:38.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 --2- 172.21.15.46:0/381032906 >> v2:172.21.15.154:3300/0 conn(0x7f4f600c6f60 0x7f4f600c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:38.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 --2- 172.21.15.46:0/381032906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f600c6b80 0x7f4f600bfec0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:38.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 -- 172.21.15.46:0/381032906 >> 172.21.15.46:0/381032906 conn(0x7f4f600c7910 msgr2=0x7f4f600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:38.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 -- 172.21.15.46:0/381032906 shutdown_connections 2023-10-30T13:50:38.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 -- 172.21.15.46:0/381032906 wait complete. 2023-10-30T13:50:38.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 Processor -- start 2023-10-30T13:50:38.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 -- start start 2023-10-30T13:50:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f600c6b80 0x7f4f600ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:38.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f600c6f60 0x7f4f600c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:38.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f600c8430 0x7f4f60154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:38.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f60139500 con 0x7f4f600c6b80 2023-10-30T13:50:38.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f601546b0 con 0x7f4f600c6f60 2023-10-30T13:50:38.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f64b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6013dd40 con 0x7f4f600c8430 2023-10-30T13:50:38.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f5e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f600c6b80 0x7f4f600ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:38.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f5e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f600c6b80 0x7f4f600ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55028/0 (socket says 172.21.15.46:55028) 2023-10-30T13:50:38.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.025+0000 7f4f5e59c640 1 -- 172.21.15.46:0/2611942179 learned_addr learned my addr 172.21.15.46:0/2611942179 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:38.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f5ed9d640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f600c8430 0x7f4f60154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:38.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f5dd9b640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f600c6f60 0x7f4f600c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:38.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f5e59c640 1 -- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f600c8430 msgr2=0x7f4f60154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:38.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f5e59c640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f600c8430 0x7f4f60154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:38.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f5e59c640 1 -- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f600c6f60 msgr2=0x7f4f600c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:38.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f5e59c640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f600c6f60 0x7f4f600c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:38.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f5e59c640 1 -- 172.21.15.46:0/2611942179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f50009420 con 0x7f4f600c6b80 2023-10-30T13:50:38.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f5e59c640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f600c6b80 0x7f4f600ca760 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f4f500b2040 tx=0x7f4f5001b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:38.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f4effd640 1 -- 172.21.15.46:0/2611942179 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500af8d0 con 0x7f4f600c6b80 2023-10-30T13:50:38.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f4effd640 1 -- 172.21.15.46:0/2611942179 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f500afa30 con 0x7f4f600c6b80 2023-10-30T13:50:38.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f64b3e640 1 -- 172.21.15.46:0/2611942179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f60155790 con 0x7f4f600c6b80 2023-10-30T13:50:38.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f4effd640 1 -- 172.21.15.46:0/2611942179 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500afde0 con 0x7f4f600c6b80 2023-10-30T13:50:38.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f64b3e640 1 -- 172.21.15.46:0/2611942179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f60155d30 con 0x7f4f600c6b80 2023-10-30T13:50:38.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f64b3e640 1 -- 172.21.15.46:0/2611942179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6004e170 con 0x7f4f600c6b80 2023-10-30T13:50:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.029+0000 7f4f4effd640 1 -- 172.21.15.46:0/2611942179 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4f500b6040 con 0x7f4f600c6b80 2023-10-30T13:50:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.033+0000 7f4f4effd640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f400664a0 0x7f4f40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.033+0000 7f4f4effd640 1 -- 172.21.15.46:0/2611942179 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f4f50120bb0 con 0x7f4f600c6b80 2023-10-30T13:50:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.033+0000 7f4f5dd9b640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f400664a0 0x7f4f40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:38.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.033+0000 7f4f5dd9b640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f400664a0 0x7f4f40068960 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4f6013ea10 tx=0x7f4f54008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:38.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.037+0000 7f4f4effd640 1 -- 172.21.15.46:0/2611942179 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4f500edb40 con 0x7f4f600c6b80 2023-10-30T13:50:38.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.245+0000 7f4f64b3e640 1 -- 172.21.15.46:0/2611942179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f6013f250 con 0x7f4f600c6b80 2023-10-30T13:50:38.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f4effd640 1 -- 172.21.15.46:0/2611942179 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f4f500b4090 con 0x7f4f600c6b80 2023-10-30T13:50:38.252 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:30.306260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:38.252 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 121 2023-10-30T13:50:38.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f64b3e640 1 -- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f400664a0 msgr2=0x7f4f40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:38.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f64b3e640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f400664a0 0x7f4f40068960 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4f6013ea10 tx=0x7f4f54008040 comp rx=0 tx=0).stop 2023-10-30T13:50:38.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f64b3e640 1 -- 172.21.15.46:0/2611942179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f600c6b80 msgr2=0x7f4f600ca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:38.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f64b3e640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f600c6b80 0x7f4f600ca760 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f4f500b2040 tx=0x7f4f5001b630 comp rx=0 tx=0).stop 2023-10-30T13:50:38.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f64b3e640 1 -- 172.21.15.46:0/2611942179 shutdown_connections 2023-10-30T13:50:38.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f64b3e640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f400664a0 0x7f4f40068960 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:38.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f64b3e640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f600c8430 0x7f4f60154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:38.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f64b3e640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f600c6f60 0x7f4f600c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:38.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f64b3e640 1 --2- 172.21.15.46:0/2611942179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f600c6b80 0x7f4f600ca760 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:38.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f64b3e640 1 -- 172.21.15.46:0/2611942179 >> 172.21.15.46:0/2611942179 conn(0x7f4f600c7910 msgr2=0x7f4f600c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:38.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f64b3e640 1 -- 172.21.15.46:0/2611942179 shutdown_connections 2023-10-30T13:50:38.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:38.249+0000 7f4f64b3e640 1 -- 172.21.15.46:0/2611942179 wait complete. 2023-10-30T13:50:38.272 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T13:50:39.033 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 1% (3/160) Updating files: 2% (4/160) Updating files: 3% (5/160) Updating files: 3% (6/160) Updating files: 4% (7/160) Updating files: 5% (8/160) Updating files: 6% (10/160) Updating files: 7% (12/160) Updating files: 8% (13/160) Updating files: 9% (15/160) Updating files: 10% (16/160) Updating files: 11% (18/160) Updating files: 12% (20/160) Updating files: 13% (21/160) Updating files: 14% (23/160) Updating files: 15% (24/160) Updating files: 16% (26/160) Updating files: 16% (27/160) Updating files: 17% (28/160) Updating files: 18% (29/160) Updating files: 19% (31/160) Updating files: 20% (32/160) Updating files: 21% (34/160) Updating files: 22% (36/160) Updating files: 23% (37/160) Updating files: 24% (39/160) Updating files: 25% (40/160) Updating files: 26% (42/160) Updating files: 27% (44/160) Updating files: 28% (45/160) Updating files: 29% (47/160) Updating files: 30% (48/160) Updating files: 31% (50/160) Updating files: 32% (52/160) Updating files: 33% (53/160) Updating files: 34% (55/160) Updating files: 35% (56/160) Updating files: 36% (58/160) Updating files: 37% (60/160) Updating files: 38% (61/160) Updating files: 39% (63/160) Updating files: 40% (64/160) Updating files: 41% (66/160) Updating files: 42% (68/160) Updating files: 43% (69/160) Updating files: 44% (71/160) Updating files: 45% (72/160) Updating files: 46% (74/160) Updating files: 46% (75/160) Updating files: 47% (76/160) Updating files: 48% (77/160) Updating files: 49% (79/160) Updating files: 50% (80/160) Updating files: 51% (82/160) Updating files: 52% (84/160) Updating files: 53% (85/160) Updating files: 54% (87/160) Updating files: 55% (88/160) Updating files: 56% (90/160) Updating files: 57% (92/160) Updating files: 58% (93/160) Updating files: 59% (95/160) Updating files: 60% (96/160) Updating files: 61% (98/160) Updating files: 62% (100/160) Updating files: 63% (101/160) Updating files: 64% (103/160) Updating files: 65% (104/160) Updating files: 66% (106/160) Updating files: 67% (108/160) Updating files: 68% (109/160) Updating files: 69% (111/160) Updating files: 70% (112/160) Updating files: 71% (114/160) Updating files: 72% (116/160) Updating files: 73% (117/160) Updating files: 74% (119/160) Updating files: 75% (120/160) Updating files: 76% (122/160) Updating files: 77% (124/160) Updating files: 78% (125/160) Updating files: 79% (127/160) Updating files: 80% (128/160) Updating files: 81% (130/160) Updating files: 82% (132/160) Updating files: 83% (133/160) Updating files: 84% (135/160) Updating files: 85% (136/160) Updating files: 86% (138/160) Updating files: 87% (140/160) Updating files: 88% (141/160) Updating files: 89% (143/160) Updating files: 90% (144/160) Updating files: 91% (146/160) Updating files: 92% (148/160) Updating files: 93% (149/160) Updating files: 94% (151/160) Updating files: 95% (152/160) Updating files: 96% (154/160) Updating files: 97% (156/160) Updating files: 98% (157/160) Updating files: 99% (159/160) Updating files: 100% (160/160) Updating files: 100% (160/160), done. 2023-10-30T13:50:39.663 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 340b4348bf6 v0.33 2023-10-30T13:50:39.667 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.33 2023-10-30T13:50:39.706 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 34 -eq 48 ] 2023-10-30T13:50:39.706 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.34 2023-10-30T13:50:39.706 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.34 2023-10-30T13:50:39.707 INFO:tasks.workunit.client.0.smithi046.stdout:v0.34 2023-10-30T13:50:39.707 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.34 2023-10-30T13:50:40.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:40.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f410f640 1 Processor -- start 2023-10-30T13:50:40.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f410f640 1 -- start start 2023-10-30T13:50:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f410f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd3ec0cf160 0x7fd3ec0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f410f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ec0cb800 con 0x7fd3ec1538a0 2023-10-30T13:50:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f410f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ec0cb330 con 0x7fd3ec15d6f0 2023-10-30T13:50:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f410f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ec0cad70 con 0x7fd3ec0cf160 2023-10-30T13:50:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f1eab640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd3ec1538a0 0x7fd3ec0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52386/0 (socket says 172.21.15.46:52386) 2023-10-30T13:50:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f1eab640 1 -- 172.21.15.46:0/820941605 learned_addr learned my addr 172.21.15.46:0/820941605 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:50:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f26ac640 1 --2- 172.21.15.46:0/820941605 >> v2:172.21.15.154:3300/0 conn(0x7fd3ec0cf160 0x7fd3ec0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884566698 0 0) 0x7fd3ec0cb800 con 0x7fd3ec1538a0 2023-10-30T13:50:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d8003530 con 0x7fd3ec1538a0 2023-10-30T13:50:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2534112862 0 0) 0x7fd3d8003530 con 0x7fd3ec1538a0 2023-10-30T13:50:40.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 >> v2:172.21.15.154:3300/0 conn(0x7fd3ec0cf160 msgr2=0x7fd3ec0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:40.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 --2- 172.21.15.46:0/820941605 >> v2:172.21.15.154:3300/0 conn(0x7fd3ec0cf160 0x7fd3ec0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:40.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 >> v1:172.21.15.154:6789/0 conn(0x7fd3ec15d6f0 legacy=0x7fd3ec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:40.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec0cb050 con 0x7fd3ec1538a0 2023-10-30T13:50:40.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fd3e8001680 con 0x7fd3ec1538a0 2023-10-30T13:50:40.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 >> v1:172.21.15.46:6789/0 conn(0x7fd3ec1538a0 legacy=0x7fd3ec0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:40.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 --2- 172.21.15.46:0/820941605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 0x7fd3d8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:40.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 --2- 172.21.15.46:0/820941605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3d80047d0 0x7fd3d8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:40.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 --2- 172.21.15.46:0/820941605 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3d8005190 0x7fd3d800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:40.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec0cad70 con 0x7fd3d8003980 2023-10-30T13:50:40.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d800c040 con 0x7fd3d80047d0 2023-10-30T13:50:40.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d800c200 con 0x7fd3d8005190 2023-10-30T13:50:40.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f1eab640 1 --2- 172.21.15.46:0/820941605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 0x7fd3d8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:40.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f16aa640 1 --2- 172.21.15.46:0/820941605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3d80047d0 0x7fd3d8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:40.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.421+0000 7fd3f26ac640 1 --2- 172.21.15.46:0/820941605 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3d8005190 0x7fd3d800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:40.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f1eab640 1 -- 172.21.15.46:0/820941605 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3d8005190 msgr2=0x7fd3d800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:40.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f1eab640 1 --2- 172.21.15.46:0/820941605 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3d8005190 0x7fd3d800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f1eab640 1 -- 172.21.15.46:0/820941605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3d80047d0 msgr2=0x7fd3d8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f1eab640 1 --2- 172.21.15.46:0/820941605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3d80047d0 0x7fd3d8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f1eab640 1 -- 172.21.15.46:0/820941605 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d8003530 con 0x7fd3d8003980 2023-10-30T13:50:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f26ac640 1 -- 172.21.15.46:0/820941605 reap_dead start 2023-10-30T13:50:40.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f1eab640 1 --2- 172.21.15.46:0/820941605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 0x7fd3d8003da0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800c240 tx=0x7fd3e801f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eebf043942642a81 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:40.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8005ae0 con 0x7fd3d8003980 2023-10-30T13:50:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e80ab070 con 0x7fd3d8003980 2023-10-30T13:50:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f0ea9640 1 -- 172.21.15.46:0/820941605 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8007070 con 0x7fd3d8003980 2023-10-30T13:50:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f410f640 1 -- 172.21.15.46:0/820941605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 msgr2=0x7fd3d8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f410f640 1 --2- 172.21.15.46:0/820941605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 0x7fd3d8003da0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800c240 tx=0x7fd3e801f330 comp rx=0 tx=0).stop 2023-10-30T13:50:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f410f640 1 -- 172.21.15.46:0/820941605 shutdown_connections 2023-10-30T13:50:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f410f640 1 --2- 172.21.15.46:0/820941605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 0x7fd3d8003da0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f410f640 1 -- 172.21.15.46:0/820941605 >> 172.21.15.46:0/820941605 conn(0x7fd3ec06bfe0 msgr2=0x7fd3ec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f410f640 1 -- 172.21.15.46:0/820941605 shutdown_connections 2023-10-30T13:50:40.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.425+0000 7fd3f410f640 1 -- 172.21.15.46:0/820941605 wait complete. 2023-10-30T13:50:40.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f410f640 1 Processor -- start 2023-10-30T13:50:40.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f410f640 1 -- start start 2023-10-30T13:50:40.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f410f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 0x7fd3ec146420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:40.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f410f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3ec0b3a40 0x7fd3ec15c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:40.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f410f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3ec1ef0f0 0x7fd3ec1ef580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:40.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f410f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec0b4070 con 0x7fd3d8003980 2023-10-30T13:50:40.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f410f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec1f35c0 con 0x7fd3ec1ef0f0 2023-10-30T13:50:40.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f410f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec1f3130 con 0x7fd3ec0b3a40 2023-10-30T13:50:40.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f1eab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 0x7fd3ec146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:40.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f1eab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 0x7fd3ec146420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55050/0 (socket says 172.21.15.46:55050) 2023-10-30T13:50:40.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f1eab640 1 -- 172.21.15.46:0/3613177176 learned_addr learned my addr 172.21.15.46:0/3613177176 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:50:40.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f26ac640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3ec1ef0f0 0x7fd3ec1ef580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:40.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f16aa640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3ec0b3a40 0x7fd3ec15c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:40.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f1eab640 1 -- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3ec0b3a40 msgr2=0x7fd3ec15c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:40.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f1eab640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3ec0b3a40 0x7fd3ec15c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:40.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f1eab640 1 -- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3ec1ef0f0 msgr2=0x7fd3ec1ef580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:40.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f1eab640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3ec1ef0f0 0x7fd3ec1ef580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:40.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f1eab640 1 -- 172.21.15.46:0/3613177176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec146ad0 con 0x7fd3d8003980 2023-10-30T13:50:40.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f1eab640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 0x7fd3ec146420 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fd3e801f160 tx=0x7fd3d8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:40.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3e27fc640 1 -- 172.21.15.46:0/3613177176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8001680 con 0x7fd3d8003980 2023-10-30T13:50:40.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3e27fc640 1 -- 172.21.15.46:0/3613177176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e8007070 con 0x7fd3d8003980 2023-10-30T13:50:40.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f410f640 1 -- 172.21.15.46:0/3613177176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ec1f2800 con 0x7fd3d8003980 2023-10-30T13:50:40.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3e27fc640 1 -- 172.21.15.46:0/3613177176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8007400 con 0x7fd3d8003980 2023-10-30T13:50:40.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.429+0000 7fd3f410f640 1 -- 172.21.15.46:0/3613177176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ec1f2dc0 con 0x7fd3d8003980 2023-10-30T13:50:40.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.433+0000 7fd3f410f640 1 -- 172.21.15.46:0/3613177176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ec04e170 con 0x7fd3d8003980 2023-10-30T13:50:40.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.433+0000 7fd3e27fc640 1 -- 172.21.15.46:0/3613177176 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd3e8007560 con 0x7fd3d8003980 2023-10-30T13:50:40.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.433+0000 7fd3e27fc640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3d00664a0 0x7fd3d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:40.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.433+0000 7fd3e27fc640 1 -- 172.21.15.46:0/3613177176 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fd3e80b7070 con 0x7fd3d8003980 2023-10-30T13:50:40.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.433+0000 7fd3f16aa640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3d00664a0 0x7fd3d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:40.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.433+0000 7fd3f16aa640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3d00664a0 0x7fd3d0068960 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec159790 tx=0x7fd3dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:40.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.437+0000 7fd3e27fc640 1 -- 172.21.15.46:0/3613177176 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd3e80edfd0 con 0x7fd3d8003980 2023-10-30T13:50:40.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.653+0000 7fd3f410f640 1 -- 172.21.15.46:0/3613177176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3ec0b1dd0 con 0x7fd3d8003980 2023-10-30T13:50:40.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.653+0000 7fd3e27fc640 1 -- 172.21.15.46:0/3613177176 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+6524 (secure 0 0 0) 0x7fd3e80eff00 con 0x7fd3d8003980 2023-10-30T13:50:40.658 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:50:40.658 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7002,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:50:30.306260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:40.658 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 122 2023-10-30T13:50:40.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.657+0000 7fd3f410f640 1 -- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3d00664a0 msgr2=0x7fd3d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:40.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.657+0000 7fd3f410f640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3d00664a0 0x7fd3d0068960 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec159790 tx=0x7fd3dc009040 comp rx=0 tx=0).stop 2023-10-30T13:50:40.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.657+0000 7fd3f410f640 1 -- 172.21.15.46:0/3613177176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 msgr2=0x7fd3ec146420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:40.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.657+0000 7fd3f410f640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 0x7fd3ec146420 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fd3e801f160 tx=0x7fd3d8005800 comp rx=0 tx=0).stop 2023-10-30T13:50:40.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.657+0000 7fd3f410f640 1 -- 172.21.15.46:0/3613177176 shutdown_connections 2023-10-30T13:50:40.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.657+0000 7fd3f410f640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3d00664a0 0x7fd3d0068960 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:40.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.657+0000 7fd3f410f640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3ec1ef0f0 0x7fd3ec1ef580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:40.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.657+0000 7fd3f410f640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3ec0b3a40 0x7fd3ec15c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:40.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.657+0000 7fd3f410f640 1 --2- 172.21.15.46:0/3613177176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8003980 0x7fd3ec146420 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:40.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.657+0000 7fd3f410f640 1 -- 172.21.15.46:0/3613177176 >> 172.21.15.46:0/3613177176 conn(0x7fd3ec06bfe0 msgr2=0x7fd3ec0c4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:40.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.657+0000 7fd3f410f640 1 -- 172.21.15.46:0/3613177176 shutdown_connections 2023-10-30T13:50:40.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:50:40.657+0000 7fd3f410f640 1 -- 172.21.15.46:0/3613177176 wait complete. 2023-10-30T13:50:40.679 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:50:40.680 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-10-30T13:50:48.356 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 15% (20/129) Updating files: 16% (21/129) Updating files: 17% (22/129) Updating files: 18% (24/129) Updating files: 19% (25/129) Updating files: 20% (26/129) Updating files: 21% (28/129) Updating files: 22% (29/129) Updating files: 23% (30/129) Updating files: 24% (31/129) Updating files: 25% (33/129) Updating files: 26% (34/129) Updating files: 27% (35/129) Updating files: 28% (37/129) Updating files: 29% (38/129) Updating files: 30% (39/129) Updating files: 31% (40/129) Updating files: 32% (42/129) Updating files: 33% (43/129) Updating files: 34% (44/129) Updating files: 35% (46/129) Updating files: 36% (47/129) Updating files: 37% (48/129) Updating files: 38% (50/129) Updating files: 39% (51/129) Updating files: 40% (52/129) Updating files: 41% (53/129) Updating files: 42% (55/129) Updating files: 43% (56/129) Updating files: 44% (57/129) Updating files: 45% (59/129) Updating files: 46% (60/129) Updating files: 47% (61/129) Updating files: 48% (62/129) Updating files: 49% (64/129) Updating files: 50% (65/129) Updating files: 51% (66/129) Updating files: 52% (68/129) Updating files: 53% (69/129) Updating files: 54% (70/129) Updating files: 55% (71/129) Updating files: 56% (73/129) Updating files: 57% (74/129) Updating files: 58% (75/129) Updating files: 59% (77/129) Updating files: 60% (78/129) Updating files: 61% (79/129) Updating files: 62% (80/129) Updating files: 63% (82/129) Updating files: 64% (83/129) Updating files: 65% (84/129) Updating files: 66% (86/129) Updating files: 67% (87/129) Updating files: 68% (88/129) Updating files: 69% (90/129) Updating files: 70% (91/129) Updating files: 71% (92/129) Updating files: 72% (93/129) Updating files: 73% (95/129) Updating files: 74% (96/129) Updating files: 75% (97/129) Updating files: 76% (99/129) Updating files: 77% (100/129) Updating files: 78% (101/129) Updating files: 79% (102/129) Updating files: 80% (104/129) Updating files: 81% (105/129) Updating files: 82% (106/129) Updating files: 83% (108/129) Updating files: 84% (109/129) Updating files: 85% (110/129) Updating files: 86% (111/129) Updating files: 87% (113/129) Updating files: 88% (114/129) Updating files: 89% (115/129) Updating files: 90% (117/129) Updating files: 91% (118/129) Updating files: 92% (119/129) Updating files: 93% (120/129) Updating files: 94% (122/129) Updating files: 95% (123/129) Updating files: 96% (124/129) Updating files: 97% (126/129) Updating files: 98% (127/129) Updating files: 99% (128/129) Updating files: 100% (129/129) Updating files: 100% (129/129), done. 2023-10-30T13:50:48.445 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 2f039eeeb74 v0.34 2023-10-30T13:50:48.448 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.34 2023-10-30T13:50:48.517 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 35 -eq 48 ] 2023-10-30T13:50:48.518 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.35 2023-10-30T13:50:48.518 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.35 2023-10-30T13:50:48.518 INFO:tasks.workunit.client.0.smithi046.stdout:v0.35 2023-10-30T13:50:48.519 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.35 2023-10-30T13:50:50.933 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:50:50.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:50:50.960 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 81% (272/334) Updating files: 82% (274/334) Updating files: 83% (278/334) Updating files: 84% (281/334) Updating files: 85% (284/334) Updating files: 86% (288/334) Updating files: 87% (291/334) Updating files: 88% (294/334) Updating files: 89% (298/334) Updating files: 90% (301/334) Updating files: 91% (304/334) Updating files: 92% (308/334) Updating files: 92% (310/334) Updating files: 93% (311/334) Updating files: 94% (314/334) Updating files: 95% (318/334) Updating files: 96% (321/334) Updating files: 97% (324/334) Updating files: 98% (328/334) Updating files: 99% (331/334) Updating files: 100% (334/334) Updating files: 100% (334/334), done. 2023-10-30T13:50:51.002 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 0afda378124 v0.35 2023-10-30T13:50:51.005 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.35 2023-10-30T13:50:51.169 INFO:tasks.workunit.client.0.smithi046.stdout:v0.36 2023-10-30T13:50:51.169 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 36 -eq 48 ] 2023-10-30T13:50:51.169 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.36 2023-10-30T13:50:51.169 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.36 2023-10-30T13:50:51.170 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.36 2023-10-30T13:50:53.932 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 47% (115/244) Updating files: 48% (118/244) Updating files: 49% (120/244) Updating files: 50% (122/244) Updating files: 51% (125/244) Updating files: 52% (127/244) Updating files: 52% (129/244) Updating files: 53% (130/244) Updating files: 54% (132/244) Updating files: 55% (135/244) Updating files: 56% (137/244) Updating files: 57% (140/244) Updating files: 58% (142/244) Updating files: 59% (144/244) Updating files: 60% (147/244) Updating files: 61% (149/244) Updating files: 62% (152/244) Updating files: 63% (154/244) Updating files: 64% (157/244) Updating files: 65% (159/244) Updating files: 66% (162/244) Updating files: 67% (164/244) Updating files: 68% (166/244) Updating files: 69% (169/244) Updating files: 70% (171/244) Updating files: 71% (174/244) Updating files: 72% (176/244) Updating files: 73% (179/244) Updating files: 74% (181/244) Updating files: 75% (183/244) Updating files: 76% (186/244) Updating files: 77% (188/244) Updating files: 78% (191/244) Updating files: 79% (193/244) Updating files: 80% (196/244) Updating files: 81% (198/244) Updating files: 82% (201/244) Updating files: 83% (203/244) Updating files: 84% (205/244) Updating files: 85% (208/244) Updating files: 86% (210/244) Updating files: 87% (213/244) Updating files: 88% (215/244) Updating files: 89% (218/244) Updating files: 90% (220/244) Updating files: 91% (223/244) Updating files: 92% (225/244) Updating files: 93% (227/244) Updating files: 94% (230/244) Updating files: 95% (232/244) Updating files: 96% (235/244) Updating files: 97% (237/244) Updating files: 98% (240/244) Updating files: 99% (242/244) Updating files: 100% (244/244) Updating files: 100% (244/244), done. 2023-10-30T13:50:53.969 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 877cacbbdbb v0.36 2023-10-30T13:50:53.971 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.36 2023-10-30T13:50:54.507 INFO:tasks.workunit.client.0.smithi046.stdout:v0.37 2023-10-30T13:50:54.508 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 37 -eq 48 ] 2023-10-30T13:50:54.508 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.37 2023-10-30T13:50:54.508 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.37 2023-10-30T13:50:54.508 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.37 2023-10-30T13:51:13.229 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 7% (14/181) Updating files: 8% (15/181) Updating files: 9% (17/181) Updating files: 10% (19/181) Updating files: 11% (20/181) Updating files: 12% (22/181) Updating files: 13% (24/181) Updating files: 14% (26/181) Updating files: 15% (28/181) Updating files: 16% (29/181) Updating files: 17% (31/181) Updating files: 18% (33/181) Updating files: 19% (35/181) Updating files: 20% (37/181) Updating files: 21% (39/181) Updating files: 22% (40/181) Updating files: 23% (42/181) Updating files: 24% (44/181) Updating files: 25% (46/181) Updating files: 26% (48/181) Updating files: 27% (49/181) Updating files: 28% (51/181) Updating files: 29% (53/181) Updating files: 30% (55/181) Updating files: 31% (57/181) Updating files: 32% (58/181) Updating files: 33% (60/181) Updating files: 34% (62/181) Updating files: 35% (64/181) Updating files: 36% (66/181) Updating files: 37% (67/181) Updating files: 38% (69/181) Updating files: 39% (71/181) Updating files: 40% (73/181) Updating files: 41% (75/181) Updating files: 42% (77/181) Updating files: 43% (78/181) Updating files: 44% (80/181) Updating files: 45% (82/181) Updating files: 46% (84/181) Updating files: 47% (86/181) Updating files: 48% (87/181) Updating files: 49% (89/181) Updating files: 50% (91/181) Updating files: 51% (93/181) Updating files: 52% (95/181) Updating files: 53% (96/181) Updating files: 54% (98/181) Updating files: 55% (100/181) Updating files: 56% (102/181) Updating files: 57% (104/181) Updating files: 58% (105/181) Updating files: 59% (107/181) Updating files: 60% (109/181) Updating files: 61% (111/181) Updating files: 62% (113/181) Updating files: 63% (115/181) Updating files: 64% (116/181) Updating files: 65% (118/181) Updating files: 66% (120/181) Updating files: 67% (122/181) Updating files: 68% (124/181) Updating files: 69% (125/181) Updating files: 70% (127/181) Updating files: 71% (129/181) Updating files: 72% (131/181) Updating files: 73% (133/181) Updating files: 74% (134/181) Updating files: 75% (136/181) Updating files: 76% (138/181) Updating files: 77% (140/181) Updating files: 78% (142/181) Updating files: 79% (143/181) Updating files: 80% (145/181) Updating files: 81% (147/181) Updating files: 82% (149/181) Updating files: 83% (151/181) Updating files: 84% (153/181) Updating files: 85% (154/181) Updating files: 86% (156/181) Updating files: 87% (158/181) Updating files: 88% (160/181) Updating files: 89% (162/181) Updating files: 90% (163/181) Updating files: 91% (165/181) Updating files: 92% (167/181) Updating files: 93% (169/181) Updating files: 94% (171/181) Updating files: 95% (172/181) Updating files: 95% (173/181) Updating files: 96% (174/181) Updating files: 97% (176/181) Updating files: 98% (178/181) Updating files: 99% (180/181) Updating files: 100% (181/181) Updating files: 100% (181/181), done. 2023-10-30T13:51:13.286 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at a6f3bbb744a v0.37 2023-10-30T13:51:13.289 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.37 2023-10-30T13:51:13.591 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 38 -eq 48 ] 2023-10-30T13:51:13.591 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.38 2023-10-30T13:51:13.591 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.38 2023-10-30T13:51:13.592 INFO:tasks.workunit.client.0.smithi046.stdout:v0.38 2023-10-30T13:51:13.592 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.38 2023-10-30T13:51:17.268 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 19% (34/171) Updating files: 20% (35/171) Updating files: 21% (36/171) Updating files: 22% (38/171) Updating files: 23% (40/171) Updating files: 24% (42/171) Updating files: 25% (43/171) Updating files: 26% (45/171) Updating files: 27% (47/171) Updating files: 28% (48/171) Updating files: 29% (50/171) Updating files: 30% (52/171) Updating files: 31% (54/171) Updating files: 32% (55/171) Updating files: 33% (57/171) Updating files: 34% (59/171) Updating files: 35% (60/171) Updating files: 36% (62/171) Updating files: 37% (64/171) Updating files: 38% (65/171) Updating files: 39% (67/171) Updating files: 40% (69/171) Updating files: 41% (71/171) Updating files: 42% (72/171) Updating files: 43% (74/171) Updating files: 44% (76/171) Updating files: 45% (77/171) Updating files: 46% (79/171) Updating files: 47% (81/171) Updating files: 48% (83/171) Updating files: 49% (84/171) Updating files: 50% (86/171) Updating files: 50% (87/171) Updating files: 51% (88/171) Updating files: 52% (89/171) Updating files: 53% (91/171) Updating files: 54% (93/171) Updating files: 55% (95/171) Updating files: 56% (96/171) Updating files: 57% (98/171) Updating files: 58% (100/171) Updating files: 59% (101/171) Updating files: 60% (103/171) Updating files: 61% (105/171) Updating files: 62% (107/171) Updating files: 63% (108/171) Updating files: 64% (110/171) Updating files: 65% (112/171) Updating files: 66% (113/171) Updating files: 67% (115/171) Updating files: 68% (117/171) Updating files: 69% (118/171) Updating files: 70% (120/171) Updating files: 70% (121/171) Updating files: 71% (122/171) Updating files: 72% (124/171) Updating files: 73% (125/171) Updating files: 74% (127/171) Updating files: 75% (129/171) Updating files: 76% (130/171) Updating files: 77% (132/171) Updating files: 78% (134/171) Updating files: 79% (136/171) Updating files: 80% (137/171) Updating files: 81% (139/171) Updating files: 82% (141/171) Updating files: 83% (142/171) Updating files: 84% (144/171) Updating files: 85% (146/171) Updating files: 86% (148/171) Updating files: 87% (149/171) Updating files: 88% (151/171) Updating files: 89% (153/171) Updating files: 90% (154/171) Updating files: 91% (156/171) Updating files: 92% (158/171) Updating files: 93% (160/171) Updating files: 94% (161/171) Updating files: 95% (163/171) Updating files: 96% (165/171) Updating files: 97% (166/171) Updating files: 98% (168/171) Updating files: 99% (170/171) Updating files: 100% (171/171) Updating files: 100% (171/171), done. 2023-10-30T13:51:17.310 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at b600ec2ac7c v0.38 2023-10-30T13:51:17.313 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.38 2023-10-30T13:51:17.883 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 39 -eq 48 ] 2023-10-30T13:51:17.883 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.39 2023-10-30T13:51:17.883 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.39 2023-10-30T13:51:17.884 INFO:tasks.workunit.client.0.smithi046.stdout:v0.39 2023-10-30T13:51:17.884 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.39 2023-10-30T13:51:19.681 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.832+0000 7fa67d632640 1 Processor -- start 2023-10-30T13:51:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.832+0000 7fa67d632640 1 -- start start 2023-10-30T13:51:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa67d632640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa67806fd50 0x7fa67815ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa67d632640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa67806afa0 0x7fa67815d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa67d632640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa678157340 con 0x7fa678156f60 2023-10-30T13:51:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa67d632640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa67806bea0 con 0x7fa67806fd50 2023-10-30T13:51:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa67d632640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6780caa80 con 0x7fa67806afa0 2023-10-30T13:51:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa676ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa678156f60 0x7fa678071b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42266/0 (socket says 172.21.15.46:42266) 2023-10-30T13:51:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa676ffd640 1 -- 172.21.15.46:0/3415201566 learned_addr learned my addr 172.21.15.46:0/3415201566 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:51:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa66ffff640 1 --2- 172.21.15.46:0/3415201566 >> v2:172.21.15.46:3300/0 conn(0x7fa67806fd50 0x7fa67815ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa6777fe640 1 --2- 172.21.15.46:0/3415201566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa67806afa0 0x7fa67815d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa6767fc640 1 -- 172.21.15.46:0/3415201566 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917973000 0 0) 0x7fa678157340 con 0x7fa678156f60 2023-10-30T13:51:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa6767fc640 1 -- 172.21.15.46:0/3415201566 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa664003530 con 0x7fa678156f60 2023-10-30T13:51:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa66ffff640 1 -- 172.21.15.46:0/3415201566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa67806afa0 msgr2=0x7fa67815d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa66ffff640 1 --2- 172.21.15.46:0/3415201566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa67806afa0 0x7fa67815d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa66ffff640 1 -- 172.21.15.46:0/3415201566 >> v1:172.21.15.46:6789/0 conn(0x7fa678156f60 legacy=0x7fa678071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.836+0000 7fa66ffff640 1 -- 172.21.15.46:0/3415201566 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67806ca30 con 0x7fa67806fd50 2023-10-30T13:51:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa66ffff640 1 --2- 172.21.15.46:0/3415201566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa67806fd50 0x7fa67815ce10 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fa6600097c0 tx=0x7fa66001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e470613b8c70d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:19.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa6767fc640 1 -- 172.21.15.46:0/3415201566 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6600ab660 con 0x7fa67806fd50 2023-10-30T13:51:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa6767fc640 1 -- 172.21.15.46:0/3415201566 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6600ab7c0 con 0x7fa67806fd50 2023-10-30T13:51:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa6767fc640 1 -- 172.21.15.46:0/3415201566 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6600abb50 con 0x7fa67806fd50 2023-10-30T13:51:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa67d632640 1 -- 172.21.15.46:0/3415201566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa67806fd50 msgr2=0x7fa67815ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa67d632640 1 --2- 172.21.15.46:0/3415201566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa67806fd50 0x7fa67815ce10 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fa6600097c0 tx=0x7fa66001c460 comp rx=0 tx=0).stop 2023-10-30T13:51:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa67d632640 1 -- 172.21.15.46:0/3415201566 shutdown_connections 2023-10-30T13:51:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa67d632640 1 --2- 172.21.15.46:0/3415201566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa67806afa0 0x7fa67815d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa67d632640 1 --2- 172.21.15.46:0/3415201566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa67806fd50 0x7fa67815ce10 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa67d632640 1 -- 172.21.15.46:0/3415201566 >> 172.21.15.46:0/3415201566 conn(0x7fa678069e90 msgr2=0x7fa67806a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:19.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa67d632640 1 -- 172.21.15.46:0/3415201566 shutdown_connections 2023-10-30T13:51:19.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa67d632640 1 -- 172.21.15.46:0/3415201566 wait complete. 2023-10-30T13:51:19.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa67d632640 1 Processor -- start 2023-10-30T13:51:19.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.840+0000 7fa67d632640 1 -- start start 2023-10-30T13:51:19.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa67d632640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa67806afa0 0x7fa678153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa67d632640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa67806fd50 0x7fa678157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa67d632640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa678156f60 0x7fa678158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa67d632640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa678150c10 con 0x7fa678156f60 2023-10-30T13:51:19.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa67d632640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa678151140 con 0x7fa67806fd50 2023-10-30T13:51:19.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa67d632640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6780cb230 con 0x7fa67806afa0 2023-10-30T13:51:19.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa6777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa678156f60 0x7fa678158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa6777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa678156f60 0x7fa678158ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44030/0 (socket says 172.21.15.46:44030) 2023-10-30T13:51:19.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa6777fe640 1 -- 172.21.15.46:0/1274680717 learned_addr learned my addr 172.21.15.46:0/1274680717 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:19.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa66ffff640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa67806fd50 0x7fa678157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa676ffd640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa67806afa0 0x7fa678153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa6777fe640 1 -- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa67806afa0 msgr2=0x7fa678153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa6777fe640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa67806afa0 0x7fa678153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa6777fe640 1 -- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa67806fd50 msgr2=0x7fa678157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa6777fe640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa67806fd50 0x7fa678157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa6777fe640 1 -- 172.21.15.46:0/1274680717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa660009420 con 0x7fa678156f60 2023-10-30T13:51:19.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa6777fe640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa678156f60 0x7fa678158ab0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fa67000efc0 tx=0x7fa67000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:19.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa66f7fe640 1 -- 172.21.15.46:0/1274680717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa670009b20 con 0x7fa678156f60 2023-10-30T13:51:19.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa66f7fe640 1 -- 172.21.15.46:0/1274680717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67000f040 con 0x7fa678156f60 2023-10-30T13:51:19.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa66f7fe640 1 -- 172.21.15.46:0/1274680717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700146c0 con 0x7fa678156f60 2023-10-30T13:51:19.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa67d632640 1 -- 172.21.15.46:0/1274680717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa678150e60 con 0x7fa678156f60 2023-10-30T13:51:19.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa67d632640 1 -- 172.21.15.46:0/1274680717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa678153a70 con 0x7fa678156f60 2023-10-30T13:51:19.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa67d632640 1 -- 172.21.15.46:0/1274680717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67804e170 con 0x7fa678156f60 2023-10-30T13:51:19.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.844+0000 7fa66f7fe640 1 -- 172.21.15.46:0/1274680717 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa670014820 con 0x7fa678156f60 2023-10-30T13:51:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.848+0000 7fa66f7fe640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa6580664f0 0x7fa6580689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.848+0000 7fa66f7fe640 1 -- 172.21.15.46:0/1274680717 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fa6700850f0 con 0x7fa678156f60 2023-10-30T13:51:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.848+0000 7fa676ffd640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa6580664f0 0x7fa6580689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.848+0000 7fa676ffd640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa6580664f0 0x7fa6580689b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa6680011f0 tx=0x7fa668009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:19.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:19.852+0000 7fa66f7fe640 1 -- 172.21.15.46:0/1274680717 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6700520f0 con 0x7fa678156f60 2023-10-30T13:51:20.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 -- 172.21.15.46:0/1274680717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa67806fb30 con 0x7fa678156f60 2023-10-30T13:51:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa66f7fe640 1 -- 172.21.15.46:0/1274680717 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+6526 (secure 0 0 0) 0x7fa670009e60 con 0x7fa678156f60 2023-10-30T13:51:20.045 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:20.045 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7002,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:06.414877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2023-10-30T13:51:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 -- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa6580664f0 msgr2=0x7fa6580689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa6580664f0 0x7fa6580689b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa6680011f0 tx=0x7fa668009040 comp rx=0 tx=0).stop 2023-10-30T13:51:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 -- 172.21.15.46:0/1274680717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa678156f60 msgr2=0x7fa678158ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa678156f60 0x7fa678158ab0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fa67000efc0 tx=0x7fa67000c490 comp rx=0 tx=0).stop 2023-10-30T13:51:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 -- 172.21.15.46:0/1274680717 shutdown_connections 2023-10-30T13:51:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa6580664f0 0x7fa6580689b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa678156f60 0x7fa678158ab0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa67806fd50 0x7fa678157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 --2- 172.21.15.46:0/1274680717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa67806afa0 0x7fa678153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 -- 172.21.15.46:0/1274680717 >> 172.21.15.46:0/1274680717 conn(0x7fa678069e90 msgr2=0x7fa6780c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 -- 172.21.15.46:0/1274680717 shutdown_connections 2023-10-30T13:51:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.040+0000 7fa67d632640 1 -- 172.21.15.46:0/1274680717 wait complete. 2023-10-30T13:51:20.066 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T13:51:20.067 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:51:20.067 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:51:20.092 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:51:20.092 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:20.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea73d1d640 1 Processor -- start 2023-10-30T13:51:20.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea73d1d640 1 -- start start 2023-10-30T13:51:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea73d1d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea6c1538a0 0x7fea6c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea73d1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c0cb800 con 0x7fea6c0ca0e0 2023-10-30T13:51:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea73d1d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c15dc30 con 0x7fea6c0cf160 2023-10-30T13:51:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea73d1d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6c0cb330 con 0x7fea6c1538a0 2023-10-30T13:51:20.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea71ab9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea6c1538a0 0x7fea6c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:20.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea712b8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fea6c0ca0e0 0x7fea6c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42280/0 (socket says 172.21.15.46:42280) 2023-10-30T13:51:20.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea712b8640 1 -- 172.21.15.46:0/1135446601 learned_addr learned my addr 172.21.15.46:0/1135446601 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:51:20.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea71ab9640 1 -- 172.21.15.46:0/1135446601 >> v1:172.21.15.154:6789/0 conn(0x7fea6c0cf160 legacy=0x7fea6c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:20.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea71ab9640 1 -- 172.21.15.46:0/1135446601 >> v1:172.21.15.46:6789/0 conn(0x7fea6c0ca0e0 legacy=0x7fea6c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:20.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea70ab7640 1 -- 172.21.15.46:0/1135446601 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657568661 0 0) 0x7fea6c0cb800 con 0x7fea6c0ca0e0 2023-10-30T13:51:20.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.272+0000 7fea71ab9640 1 -- 172.21.15.46:0/1135446601 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c0cad70 con 0x7fea6c1538a0 2023-10-30T13:51:20.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.276+0000 7fea71ab9640 1 --2- 172.21.15.46:0/1135446601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea6c1538a0 0x7fea6c15d6f0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fea680097c0 tx=0x7fea6801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e408bada62274d0e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:20.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.276+0000 7fea70ab7640 1 -- 172.21.15.46:0/1135446601 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea680a8970 con 0x7fea6c1538a0 2023-10-30T13:51:20.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.276+0000 7fea70ab7640 1 -- 172.21.15.46:0/1135446601 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea680a8ad0 con 0x7fea6c1538a0 2023-10-30T13:51:20.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.276+0000 7fea70ab7640 1 -- 172.21.15.46:0/1135446601 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea680a8e60 con 0x7fea6c1538a0 2023-10-30T13:51:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.276+0000 7fea73d1d640 1 -- 172.21.15.46:0/1135446601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea6c1538a0 msgr2=0x7fea6c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:20.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.276+0000 7fea73d1d640 1 --2- 172.21.15.46:0/1135446601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea6c1538a0 0x7fea6c15d6f0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fea680097c0 tx=0x7fea6801ec70 comp rx=0 tx=0).stop 2023-10-30T13:51:20.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.276+0000 7fea73d1d640 1 -- 172.21.15.46:0/1135446601 shutdown_connections 2023-10-30T13:51:20.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.276+0000 7fea73d1d640 1 --2- 172.21.15.46:0/1135446601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea6c1538a0 0x7fea6c15d6f0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.276+0000 7fea73d1d640 1 -- 172.21.15.46:0/1135446601 >> 172.21.15.46:0/1135446601 conn(0x7fea6c06bfe0 msgr2=0x7fea6c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:20.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.276+0000 7fea73d1d640 1 -- 172.21.15.46:0/1135446601 shutdown_connections 2023-10-30T13:51:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.276+0000 7fea73d1d640 1 -- 172.21.15.46:0/1135446601 wait complete. 2023-10-30T13:51:20.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea73d1d640 1 Processor -- start 2023-10-30T13:51:20.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea73d1d640 1 -- start start 2023-10-30T13:51:20.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea73d1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea6c0ca0e0 0x7fea6c0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:20.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea73d1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea6c0cf160 0x7fea6c148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:20.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea73d1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea6c1538a0 0x7fea6c1448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:20.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea73d1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c1e5480 con 0x7fea6c0cf160 2023-10-30T13:51:20.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea73d1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c0b4d10 con 0x7fea6c0ca0e0 2023-10-30T13:51:20.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea73d1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c144010 con 0x7fea6c1538a0 2023-10-30T13:51:20.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea712b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea6c0cf160 0x7fea6c148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:20.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea712b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea6c0cf160 0x7fea6c148080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44054/0 (socket says 172.21.15.46:44054) 2023-10-30T13:51:20.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea712b8640 1 -- 172.21.15.46:0/3537343989 learned_addr learned my addr 172.21.15.46:0/3537343989 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:20.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea71ab9640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea6c0ca0e0 0x7fea6c0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:20.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea722ba640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea6c1538a0 0x7fea6c1448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:20.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea712b8640 1 -- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea6c1538a0 msgr2=0x7fea6c1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:20.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea712b8640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea6c1538a0 0x7fea6c1448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea712b8640 1 -- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea6c0ca0e0 msgr2=0x7fea6c0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea712b8640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea6c0ca0e0 0x7fea6c0b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea712b8640 1 -- 172.21.15.46:0/3537343989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea68009420 con 0x7fea6c0cf160 2023-10-30T13:51:20.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea712b8640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea6c0cf160 0x7fea6c148080 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fea5c00c410 tx=0x7fea5c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:20.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea627fc640 1 -- 172.21.15.46:0/3537343989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c0074e0 con 0x7fea6c0cf160 2023-10-30T13:51:20.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea627fc640 1 -- 172.21.15.46:0/3537343989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c007640 con 0x7fea6c0cf160 2023-10-30T13:51:20.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea73d1d640 1 -- 172.21.15.46:0/3537343989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6c144420 con 0x7fea6c0cf160 2023-10-30T13:51:20.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea627fc640 1 -- 172.21.15.46:0/3537343989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c004a30 con 0x7fea6c0cf160 2023-10-30T13:51:20.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.280+0000 7fea73d1d640 1 -- 172.21.15.46:0/3537343989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6c1e49e0 con 0x7fea6c0cf160 2023-10-30T13:51:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.284+0000 7fea73d1d640 1 -- 172.21.15.46:0/3537343989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6c04e170 con 0x7fea6c0cf160 2023-10-30T13:51:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.284+0000 7fea627fc640 1 -- 172.21.15.46:0/3537343989 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fea5c004b90 con 0x7fea6c0cf160 2023-10-30T13:51:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.284+0000 7fea627fc640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea500664a0 0x7fea50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:20.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.284+0000 7fea627fc640 1 -- 172.21.15.46:0/3537343989 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fea5c083ff0 con 0x7fea6c0cf160 2023-10-30T13:51:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.284+0000 7fea71ab9640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea500664a0 0x7fea50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.284+0000 7fea71ab9640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea500664a0 0x7fea50068960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fea680b2000 tx=0x7fea68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.288+0000 7fea627fc640 1 -- 172.21.15.46:0/3537343989 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fea5c050fd0 con 0x7fea6c0cf160 2023-10-30T13:51:20.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.480+0000 7fea73d1d640 1 -- 172.21.15.46:0/3537343989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea6c0666a0 con 0x7fea6c0cf160 2023-10-30T13:51:20.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.480+0000 7fea627fc640 1 -- 172.21.15.46:0/3537343989 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+6526 (secure 0 0 0) 0x7fea5c0545f0 con 0x7fea6c0cf160 2023-10-30T13:51:20.487 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:20.487 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7002,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:06.414877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:20.488 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2023-10-30T13:51:20.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.484+0000 7fea73d1d640 1 -- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea500664a0 msgr2=0x7fea50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:20.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.484+0000 7fea73d1d640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea500664a0 0x7fea50068960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fea680b2000 tx=0x7fea68008040 comp rx=0 tx=0).stop 2023-10-30T13:51:20.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.484+0000 7fea73d1d640 1 -- 172.21.15.46:0/3537343989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea6c0cf160 msgr2=0x7fea6c148080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:20.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.484+0000 7fea73d1d640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea6c0cf160 0x7fea6c148080 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fea5c00c410 tx=0x7fea5c00c8d0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.484+0000 7fea73d1d640 1 -- 172.21.15.46:0/3537343989 shutdown_connections 2023-10-30T13:51:20.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.484+0000 7fea73d1d640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fea500664a0 0x7fea50068960 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.484+0000 7fea73d1d640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea6c1538a0 0x7fea6c1448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.484+0000 7fea73d1d640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea6c0cf160 0x7fea6c148080 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.484+0000 7fea73d1d640 1 --2- 172.21.15.46:0/3537343989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea6c0ca0e0 0x7fea6c0b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.484+0000 7fea73d1d640 1 -- 172.21.15.46:0/3537343989 >> 172.21.15.46:0/3537343989 conn(0x7fea6c06bfe0 msgr2=0x7fea6c142980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:20.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.484+0000 7fea73d1d640 1 -- 172.21.15.46:0/3537343989 shutdown_connections 2023-10-30T13:51:20.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:20.484+0000 7fea73d1d640 1 -- 172.21.15.46:0/3537343989 wait complete. 2023-10-30T13:51:20.508 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:51:21.080 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:21.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:22.509 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:22.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6b0bb640 1 Processor -- start 2023-10-30T13:51:22.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6b0bb640 1 -- start start 2023-10-30T13:51:22.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6b0bb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fee641538f0 0x7fee64159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6b0bb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee640cefa0 0x7fee640cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6b0bb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee640ca6a0 con 0x7fee6415cd50 2023-10-30T13:51:22.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6b0bb640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6415d300 con 0x7fee641538f0 2023-10-30T13:51:22.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6b0bb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6415d5e0 con 0x7fee640cefa0 2023-10-30T13:51:22.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee698b8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fee6415cd50 0x7fee640c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42292/0 (socket says 172.21.15.46:42292) 2023-10-30T13:51:22.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee698b8640 1 -- 172.21.15.46:0/1205263093 learned_addr learned my addr 172.21.15.46:0/1205263093 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:51:22.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6a0b9640 1 --2- 172.21.15.46:0/1205263093 >> v2:172.21.15.154:3300/0 conn(0x7fee641538f0 0x7fee64159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6a8ba640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee640cefa0 0x7fee640cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25846736 0 0) 0x7fee640ca6a0 con 0x7fee6415cd50 2023-10-30T13:51:22.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee50003530 con 0x7fee6415cd50 2023-10-30T13:51:22.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1264273321 0 0) 0x7fee50003530 con 0x7fee6415cd50 2023-10-30T13:51:22.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee640cefa0 msgr2=0x7fee640cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:22.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee640cefa0 0x7fee640cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 >> v2:172.21.15.154:3300/0 conn(0x7fee641538f0 msgr2=0x7fee64159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:22.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 --2- 172.21.15.46:0/1205263093 >> v2:172.21.15.154:3300/0 conn(0x7fee641538f0 0x7fee64159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6415d8c0 con 0x7fee6415cd50 2023-10-30T13:51:22.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fee54001680 con 0x7fee6415cd50 2023-10-30T13:51:22.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 >> v1:172.21.15.46:6789/0 conn(0x7fee6415cd50 legacy=0x7fee640c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee50003980 0x7fee50003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500047d0 0x7fee50004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee640cefa0 0x7fee50005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6415d300 con 0x7fee50003980 2023-10-30T13:51:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6415d5e0 con 0x7fee500047d0 2023-10-30T13:51:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5000b990 con 0x7fee640cefa0 2023-10-30T13:51:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6a0b9640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee50003980 0x7fee50003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee698b8640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500047d0 0x7fee50004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6a8ba640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee640cefa0 0x7fee50005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6a0b9640 1 -- 172.21.15.46:0/1205263093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee640cefa0 msgr2=0x7fee50005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6a0b9640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee640cefa0 0x7fee50005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6a0b9640 1 -- 172.21.15.46:0/1205263093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500047d0 msgr2=0x7fee50004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6a0b9640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500047d0 0x7fee50004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6a0b9640 1 -- 172.21.15.46:0/1205263093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee50003530 con 0x7fee50003980 2023-10-30T13:51:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.660+0000 7fee6a8ba640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee640cefa0 0x7fee50005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:51:22.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.664+0000 7fee6a0b9640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee50003980 0x7fee50003da0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fee60009ae0 tx=0x7fee6001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd6cb36c01755ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:22.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.664+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee600a64a0 con 0x7fee50003980 2023-10-30T13:51:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.664+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee600a6600 con 0x7fee50003980 2023-10-30T13:51:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.664+0000 7fee690b7640 1 -- 172.21.15.46:0/1205263093 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee600a69b0 con 0x7fee50003980 2023-10-30T13:51:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.664+0000 7fee6b0bb640 1 -- 172.21.15.46:0/1205263093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee50003980 msgr2=0x7fee50003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.664+0000 7fee6b0bb640 1 --2- 172.21.15.46:0/1205263093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee50003980 0x7fee50003da0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fee60009ae0 tx=0x7fee6001ca90 comp rx=0 tx=0).stop 2023-10-30T13:51:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.664+0000 7fee6a8ba640 1 -- 172.21.15.46:0/1205263093 reap_dead start 2023-10-30T13:51:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.664+0000 7fee6b0bb640 1 -- 172.21.15.46:0/1205263093 shutdown_connections 2023-10-30T13:51:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.664+0000 7fee6b0bb640 1 -- 172.21.15.46:0/1205263093 >> 172.21.15.46:0/1205263093 conn(0x7fee6406a640 msgr2=0x7fee64065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.664+0000 7fee6b0bb640 1 -- 172.21.15.46:0/1205263093 shutdown_connections 2023-10-30T13:51:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.664+0000 7fee6b0bb640 1 -- 172.21.15.46:0/1205263093 wait complete. 2023-10-30T13:51:22.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6b0bb640 1 Processor -- start 2023-10-30T13:51:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6b0bb640 1 -- start start 2023-10-30T13:51:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6b0bb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee50003980 0x7fee640c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6b0bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee64152ce0 0x7fee64158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6b0bb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee64158d60 0x7fee6413e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6b0bb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee640c64b0 con 0x7fee64152ce0 2023-10-30T13:51:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6b0bb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee640c7790 con 0x7fee50003980 2023-10-30T13:51:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6b0bb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee640b4f10 con 0x7fee64158d60 2023-10-30T13:51:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee698b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee64152ce0 0x7fee64158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee698b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee64152ce0 0x7fee64158820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44070/0 (socket says 172.21.15.46:44070) 2023-10-30T13:51:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee698b8640 1 -- 172.21.15.46:0/2967752535 learned_addr learned my addr 172.21.15.46:0/2967752535 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6a0b9640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee50003980 0x7fee640c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6a8ba640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee64158d60 0x7fee6413e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee698b8640 1 -- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee64158d60 msgr2=0x7fee6413e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee698b8640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee64158d60 0x7fee6413e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee698b8640 1 -- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee50003980 msgr2=0x7fee640c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee698b8640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee50003980 0x7fee640c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee698b8640 1 -- 172.21.15.46:0/2967752535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6415a9c0 con 0x7fee64152ce0 2023-10-30T13:51:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6a8ba640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee64158d60 0x7fee6413e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee698b8640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee64152ce0 0x7fee64158820 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fee54011840 tx=0x7fee54011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee5affd640 1 -- 172.21.15.46:0/2967752535 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5400fb60 con 0x7fee64152ce0 2023-10-30T13:51:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee5affd640 1 -- 172.21.15.46:0/2967752535 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5400fcc0 con 0x7fee64152ce0 2023-10-30T13:51:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee5affd640 1 -- 172.21.15.46:0/2967752535 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee54015650 con 0x7fee64152ce0 2023-10-30T13:51:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6b0bb640 1 -- 172.21.15.46:0/2967752535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6415af20 con 0x7fee64152ce0 2023-10-30T13:51:22.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.668+0000 7fee6b0bb640 1 -- 172.21.15.46:0/2967752535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee640b4c30 con 0x7fee64152ce0 2023-10-30T13:51:22.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.672+0000 7fee6b0bb640 1 -- 172.21.15.46:0/2967752535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee640c65f0 con 0x7fee64152ce0 2023-10-30T13:51:22.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.672+0000 7fee5affd640 1 -- 172.21.15.46:0/2967752535 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee540157b0 con 0x7fee64152ce0 2023-10-30T13:51:22.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.672+0000 7fee5affd640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fee480664a0 0x7fee48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.672+0000 7fee5affd640 1 -- 172.21.15.46:0/2967752535 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fee54087630 con 0x7fee64152ce0 2023-10-30T13:51:22.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.672+0000 7fee6a0b9640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fee480664a0 0x7fee48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.676+0000 7fee6a0b9640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fee480664a0 0x7fee48068960 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fee600ad000 tx=0x7fee600b3040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:22.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.676+0000 7fee5affd640 1 -- 172.21.15.46:0/2967752535 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee640c65f0 con 0x7fee64152ce0 2023-10-30T13:51:22.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.864+0000 7fee6b0bb640 1 -- 172.21.15.46:0/2967752535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee640c65f0 con 0x7fee64152ce0 2023-10-30T13:51:22.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.864+0000 7fee5affd640 1 -- 172.21.15.46:0/2967752535 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+6526 (secure 0 0 0) 0x7fee640c65f0 con 0x7fee64152ce0 2023-10-30T13:51:22.871 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:22.871 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7002,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:06.414877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:22.872 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2023-10-30T13:51:22.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.868+0000 7fee6b0bb640 1 -- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fee480664a0 msgr2=0x7fee48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.868+0000 7fee6b0bb640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fee480664a0 0x7fee48068960 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fee600ad000 tx=0x7fee600b3040 comp rx=0 tx=0).stop 2023-10-30T13:51:22.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.868+0000 7fee6b0bb640 1 -- 172.21.15.46:0/2967752535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee64152ce0 msgr2=0x7fee64158820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.868+0000 7fee6b0bb640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee64152ce0 0x7fee64158820 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fee54011840 tx=0x7fee54011d00 comp rx=0 tx=0).stop 2023-10-30T13:51:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.868+0000 7fee6b0bb640 1 -- 172.21.15.46:0/2967752535 shutdown_connections 2023-10-30T13:51:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.868+0000 7fee6b0bb640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fee480664a0 0x7fee48068960 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.868+0000 7fee6b0bb640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee64158d60 0x7fee6413e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.868+0000 7fee6b0bb640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee64152ce0 0x7fee64158820 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.868+0000 7fee6b0bb640 1 --2- 172.21.15.46:0/2967752535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee50003980 0x7fee640c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.868+0000 7fee6b0bb640 1 -- 172.21.15.46:0/2967752535 >> 172.21.15.46:0/2967752535 conn(0x7fee6406a640 msgr2=0x7fee640c9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.868+0000 7fee6b0bb640 1 -- 172.21.15.46:0/2967752535 shutdown_connections 2023-10-30T13:51:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:22.868+0000 7fee6b0bb640 1 -- 172.21.15.46:0/2967752535 wait complete. 2023-10-30T13:51:22.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:51:24.908 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f26459a0640 1 Processor -- start 2023-10-30T13:51:25.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f26459a0640 1 -- start start 2023-10-30T13:51:25.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f26459a0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401538f0 0x7f2640159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:25.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f26459a0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26400ca6a0 con 0x7f264015cd50 2023-10-30T13:51:25.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f26459a0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264015d300 con 0x7f26400cefa0 2023-10-30T13:51:25.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f26459a0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264015d5e0 con 0x7f26401538f0 2023-10-30T13:51:25.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f264015cd50 0x7f26400c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42308/0 (socket says 172.21.15.46:42308) 2023-10-30T13:51:25.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263ffff640 1 -- 172.21.15.46:0/3580889458 learned_addr learned my addr 172.21.15.46:0/3580889458 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:51:25.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f264499e640 1 --2- 172.21.15.46:0/3580889458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401538f0 0x7f2640159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:25.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735084382 0 0) 0x7f26400ca6a0 con 0x7f264015cd50 2023-10-30T13:51:25.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2630003530 con 0x7f264015cd50 2023-10-30T13:51:25.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3726699128 0 0) 0x7f2630003530 con 0x7f264015cd50 2023-10-30T13:51:25.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401538f0 msgr2=0x7f2640159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:25.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 --2- 172.21.15.46:0/3580889458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401538f0 0x7f2640159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:25.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 >> v1:172.21.15.154:6789/0 conn(0x7f26400cefa0 legacy=0x7f26401640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:25.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264015d8c0 con 0x7f264015cd50 2023-10-30T13:51:25.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f262c001680 con 0x7f264015cd50 2023-10-30T13:51:25.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 >> v1:172.21.15.46:6789/0 conn(0x7f264015cd50 legacy=0x7f26400c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:25.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 --2- 172.21.15.46:0/3580889458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401538f0 0x7f26300039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:25.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 --2- 172.21.15.46:0/3580889458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2630003ef0 0x7f26300047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:25.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 --2- 172.21.15.46:0/3580889458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2630004d10 0x7f2630005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:25.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264015d5e0 con 0x7f2630004d10 2023-10-30T13:51:25.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263000b9f0 con 0x7f2630003ef0 2023-10-30T13:51:25.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263000bbb0 con 0x7f26401538f0 2023-10-30T13:51:25.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f264519f640 1 --2- 172.21.15.46:0/3580889458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2630004d10 0x7f2630005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:25.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f264499e640 1 --2- 172.21.15.46:0/3580889458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401538f0 0x7f26300039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:25.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f263ffff640 1 --2- 172.21.15.46:0/3580889458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2630003ef0 0x7f26300047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f264519f640 1 -- 172.21.15.46:0/3580889458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401538f0 msgr2=0x7f26300039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f264519f640 1 --2- 172.21.15.46:0/3580889458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401538f0 0x7f26300039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f264519f640 1 -- 172.21.15.46:0/3580889458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2630003ef0 msgr2=0x7f26300047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f264519f640 1 --2- 172.21.15.46:0/3580889458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2630003ef0 0x7f26300047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:25.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.060+0000 7f264519f640 1 -- 172.21.15.46:0/3580889458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2630003530 con 0x7f2630004d10 2023-10-30T13:51:25.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.064+0000 7f264519f640 1 --2- 172.21.15.46:0/3580889458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2630004d10 0x7f2630005190 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2638001f60 tx=0x7f263801c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfeb49e900d8fe1d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:25.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.064+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26380a5750 con 0x7f2630004d10 2023-10-30T13:51:25.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.064+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26380a58b0 con 0x7f2630004d10 2023-10-30T13:51:25.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.064+0000 7f263f7fe640 1 -- 172.21.15.46:0/3580889458 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26380a5ca0 con 0x7f2630004d10 2023-10-30T13:51:25.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.064+0000 7f26459a0640 1 -- 172.21.15.46:0/3580889458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2630004d10 msgr2=0x7f2630005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:25.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.064+0000 7f26459a0640 1 --2- 172.21.15.46:0/3580889458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2630004d10 0x7f2630005190 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2638001f60 tx=0x7f263801c860 comp rx=0 tx=0).stop 2023-10-30T13:51:25.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.064+0000 7f264519f640 1 -- 172.21.15.46:0/3580889458 reap_dead start 2023-10-30T13:51:25.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.064+0000 7f26459a0640 1 -- 172.21.15.46:0/3580889458 shutdown_connections 2023-10-30T13:51:25.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.064+0000 7f26459a0640 1 -- 172.21.15.46:0/3580889458 >> 172.21.15.46:0/3580889458 conn(0x7f264006a640 msgr2=0x7f2640065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:25.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.064+0000 7f26459a0640 1 -- 172.21.15.46:0/3580889458 shutdown_connections 2023-10-30T13:51:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.064+0000 7f26459a0640 1 -- 172.21.15.46:0/3580889458 wait complete. 2023-10-30T13:51:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f26459a0640 1 Processor -- start 2023-10-30T13:51:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f26459a0640 1 -- start start 2023-10-30T13:51:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f26459a0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2630004d10 0x7f26401585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:25.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f26459a0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401573a0 0x7f264015c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f26459a0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26401510f0 0x7f2640151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f26459a0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26401f0080 con 0x7f2630004d10 2023-10-30T13:51:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f26459a0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2640158af0 con 0x7f26401510f0 2023-10-30T13:51:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f26459a0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26400ca330 con 0x7f26401573a0 2023-10-30T13:51:25.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f264499e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2630004d10 0x7f26401585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:25.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f264499e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2630004d10 0x7f26401585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44090/0 (socket says 172.21.15.46:44090) 2023-10-30T13:51:25.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f264499e640 1 -- 172.21.15.46:0/3413691630 learned_addr learned my addr 172.21.15.46:0/3413691630 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:25.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f263ffff640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401573a0 0x7f264015c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:25.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f264519f640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26401510f0 0x7f2640151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:25.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f264499e640 1 -- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401573a0 msgr2=0x7f264015c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:25.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f264499e640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401573a0 0x7f264015c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:25.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f264499e640 1 -- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26401510f0 msgr2=0x7f2640151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:25.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f264499e640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26401510f0 0x7f2640151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:25.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f264499e640 1 -- 172.21.15.46:0/3413691630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26401521f0 con 0x7f2630004d10 2023-10-30T13:51:25.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f263ffff640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401573a0 0x7f264015c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:51:25.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f264519f640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26401510f0 0x7f2640151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:51:25.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f264499e640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2630004d10 0x7f26401585b0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f26340119d0 tx=0x7f2634011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:25.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f263d7fa640 1 -- 172.21.15.46:0/3413691630 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263400fb50 con 0x7f2630004d10 2023-10-30T13:51:25.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f263d7fa640 1 -- 172.21.15.46:0/3413691630 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263400fcb0 con 0x7f2630004d10 2023-10-30T13:51:25.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f263d7fa640 1 -- 172.21.15.46:0/3413691630 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2634014440 con 0x7f2630004d10 2023-10-30T13:51:25.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f26459a0640 1 -- 172.21.15.46:0/3413691630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26401524d0 con 0x7f2630004d10 2023-10-30T13:51:25.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.068+0000 7f26459a0640 1 -- 172.21.15.46:0/3413691630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2640152a90 con 0x7f2630004d10 2023-10-30T13:51:25.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.072+0000 7f26459a0640 1 -- 172.21.15.46:0/3413691630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264004e170 con 0x7f2630004d10 2023-10-30T13:51:25.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.072+0000 7f263d7fa640 1 -- 172.21.15.46:0/3413691630 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f26340145a0 con 0x7f2630004d10 2023-10-30T13:51:25.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.072+0000 7f263d7fa640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f26240664a0 0x7f2624068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:25.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.072+0000 7f263d7fa640 1 -- 172.21.15.46:0/3413691630 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f2634084c40 con 0x7f2630004d10 2023-10-30T13:51:25.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.072+0000 7f263ffff640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f26240664a0 0x7f2624068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:25.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.072+0000 7f263ffff640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f26240664a0 0x7f2624068960 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f262c0017b0 tx=0x7f262c000ee0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:25.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.076+0000 7f263d7fa640 1 -- 172.21.15.46:0/3413691630 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2634051c80 con 0x7f2630004d10 2023-10-30T13:51:25.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.264+0000 7f26459a0640 1 -- 172.21.15.46:0/3413691630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2640152c20 con 0x7f2630004d10 2023-10-30T13:51:25.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f263d7fa640 1 -- 172.21.15.46:0/3413691630 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+6526 (secure 0 0 0) 0x7f2634022090 con 0x7f2630004d10 2023-10-30T13:51:25.272 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:25.272 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7002,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:06.414877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:25.273 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2023-10-30T13:51:25.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f26459a0640 1 -- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f26240664a0 msgr2=0x7f2624068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:25.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f26459a0640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f26240664a0 0x7f2624068960 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f262c0017b0 tx=0x7f262c000ee0 comp rx=0 tx=0).stop 2023-10-30T13:51:25.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f26459a0640 1 -- 172.21.15.46:0/3413691630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2630004d10 msgr2=0x7f26401585b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:25.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f26459a0640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2630004d10 0x7f26401585b0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f26340119d0 tx=0x7f2634011e90 comp rx=0 tx=0).stop 2023-10-30T13:51:25.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f26459a0640 1 -- 172.21.15.46:0/3413691630 shutdown_connections 2023-10-30T13:51:25.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f26459a0640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f26240664a0 0x7f2624068960 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:25.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f26459a0640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26401510f0 0x7f2640151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:25.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f26459a0640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26401573a0 0x7f264015c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:25.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f26459a0640 1 --2- 172.21.15.46:0/3413691630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2630004d10 0x7f26401585b0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:25.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f26459a0640 1 -- 172.21.15.46:0/3413691630 >> 172.21.15.46:0/3413691630 conn(0x7f264006a640 msgr2=0x7f26400caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:25.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f26459a0640 1 -- 172.21.15.46:0/3413691630 shutdown_connections 2023-10-30T13:51:25.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:25.268+0000 7f26459a0640 1 -- 172.21.15.46:0/3413691630 wait complete. 2023-10-30T13:51:25.293 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:51:27.294 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:27.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.444+0000 7f50d209e640 1 Processor -- start 2023-10-30T13:51:27.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.444+0000 7f50d209e640 1 -- start start 2023-10-30T13:51:27.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.444+0000 7f50d209e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50cc0c05d0 0x7f50cc13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.444+0000 7f50d209e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc0c0a50 con 0x7f50cc0c6b80 2023-10-30T13:51:27.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.444+0000 7f50d209e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc06f6e0 con 0x7f50cc0c6f60 2023-10-30T13:51:27.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.444+0000 7f50d209e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50cc06f9c0 con 0x7f50cc0c05d0 2023-10-30T13:51:27.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.444+0000 7f50caffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50cc0c05d0 0x7f50cc13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.444+0000 7f50caffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50cc0c05d0 0x7f50cc13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54840/0 (socket says 172.21.15.46:54840) 2023-10-30T13:51:27.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.444+0000 7f50caffd640 1 -- 172.21.15.46:0/1686910761 learned_addr learned my addr 172.21.15.46:0/1686910761 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:27.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50caffd640 1 -- 172.21.15.46:0/1686910761 >> v1:172.21.15.154:6789/0 conn(0x7f50cc0c6f60 legacy=0x7f50cc1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50caffd640 1 -- 172.21.15.46:0/1686910761 >> v1:172.21.15.46:6789/0 conn(0x7f50cc0c6b80 legacy=0x7f50cc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50caffd640 1 -- 172.21.15.46:0/1686910761 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc06fca0 con 0x7f50cc0c05d0 2023-10-30T13:51:27.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50ca7fc640 1 -- 172.21.15.46:0/1686910761 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487375405 0 0) 0x7f50cc0c0a50 con 0x7f50cc0c6b80 2023-10-30T13:51:27.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50caffd640 1 --2- 172.21.15.46:0/1686910761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50cc0c05d0 0x7f50cc13ed90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009870 tx=0x7f50c001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7100113132d2c751 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:27.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50ca7fc640 1 -- 172.21.15.46:0/1686910761 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c00ab650 con 0x7f50cc0c05d0 2023-10-30T13:51:27.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50ca7fc640 1 -- 172.21.15.46:0/1686910761 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c00ab7b0 con 0x7f50cc0c05d0 2023-10-30T13:51:27.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50ca7fc640 1 -- 172.21.15.46:0/1686910761 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c00abb40 con 0x7f50cc0c05d0 2023-10-30T13:51:27.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50d209e640 1 -- 172.21.15.46:0/1686910761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50cc0c05d0 msgr2=0x7f50cc13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50d209e640 1 --2- 172.21.15.46:0/1686910761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50cc0c05d0 0x7f50cc13ed90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009870 tx=0x7f50c001c540 comp rx=0 tx=0).stop 2023-10-30T13:51:27.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50d209e640 1 -- 172.21.15.46:0/1686910761 shutdown_connections 2023-10-30T13:51:27.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50d209e640 1 --2- 172.21.15.46:0/1686910761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50cc0c05d0 0x7f50cc13ed90 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50d209e640 1 -- 172.21.15.46:0/1686910761 >> 172.21.15.46:0/1686910761 conn(0x7f50cc0c7910 msgr2=0x7f50cc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:27.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.448+0000 7f50d209e640 1 -- 172.21.15.46:0/1686910761 shutdown_connections 2023-10-30T13:51:27.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50d209e640 1 -- 172.21.15.46:0/1686910761 wait complete. 2023-10-30T13:51:27.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50d209e640 1 Processor -- start 2023-10-30T13:51:27.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50d209e640 1 -- start start 2023-10-30T13:51:27.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50d209e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50cc0c05d0 0x7f50cc0c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50d209e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50cc0c6b80 0x7f50cc0c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50d209e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50cc0c6f60 0x7f50cc0c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50d209e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc159310 con 0x7f50cc0c05d0 2023-10-30T13:51:27.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50d209e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc1595f0 con 0x7f50cc0c6b80 2023-10-30T13:51:27.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50d209e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0ca1f0 con 0x7f50cc0c6f60 2023-10-30T13:51:27.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50cc0c05d0 0x7f50cc0c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50cc0c05d0 0x7f50cc0c59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54842/0 (socket says 172.21.15.46:54842) 2023-10-30T13:51:27.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50cb7fe640 1 -- 172.21.15.46:0/4095941723 learned_addr learned my addr 172.21.15.46:0/4095941723 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:27.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50caffd640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50cc0c6b80 0x7f50cc0c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50cbfff640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50cc0c6f60 0x7f50cc0c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50cb7fe640 1 -- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50cc0c6f60 msgr2=0x7f50cc0c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50cb7fe640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50cc0c6f60 0x7f50cc0c95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50cb7fe640 1 -- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50cc0c6b80 msgr2=0x7f50cc0c9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50cb7fe640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50cc0c6b80 0x7f50cc0c9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50cb7fe640 1 -- 172.21.15.46:0/4095941723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c00094d0 con 0x7f50cc0c05d0 2023-10-30T13:51:27.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.452+0000 7f50cb7fe640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50cc0c05d0 0x7f50cc0c59e0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00c480 tx=0x7f50bc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:27.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.456+0000 7f50b3fff640 1 -- 172.21.15.46:0/4095941723 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc005480 con 0x7f50cc0c05d0 2023-10-30T13:51:27.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.456+0000 7f50b3fff640 1 -- 172.21.15.46:0/4095941723 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc003600 con 0x7f50cc0c05d0 2023-10-30T13:51:27.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.456+0000 7f50b3fff640 1 -- 172.21.15.46:0/4095941723 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc0039b0 con 0x7f50cc0c05d0 2023-10-30T13:51:27.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.456+0000 7f50d209e640 1 -- 172.21.15.46:0/4095941723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50cc0ca4d0 con 0x7f50cc0c05d0 2023-10-30T13:51:27.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.456+0000 7f50d209e640 1 -- 172.21.15.46:0/4095941723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50cc13d070 con 0x7f50cc0c05d0 2023-10-30T13:51:27.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.456+0000 7f50d209e640 1 -- 172.21.15.46:0/4095941723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50cc0ca050 con 0x7f50cc0c05d0 2023-10-30T13:51:27.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.456+0000 7f50b3fff640 1 -- 172.21.15.46:0/4095941723 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f50bc003b10 con 0x7f50cc0c05d0 2023-10-30T13:51:27.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.456+0000 7f50b3fff640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50b80664a0 0x7f50b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.456+0000 7f50b3fff640 1 -- 172.21.15.46:0/4095941723 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f50bc083f10 con 0x7f50cc0c05d0 2023-10-30T13:51:27.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.456+0000 7f50caffd640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50b80664a0 0x7f50b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.460+0000 7f50caffd640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50b80664a0 0x7f50b8068960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f50c00af000 tx=0x7f50c00a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:27.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.460+0000 7f50b3fff640 1 -- 172.21.15.46:0/4095941723 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f50cc0ca050 con 0x7f50cc0c05d0 2023-10-30T13:51:27.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.652+0000 7f50d209e640 1 -- 172.21.15.46:0/4095941723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50cc0ca050 con 0x7f50cc0c05d0 2023-10-30T13:51:27.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.652+0000 7f50b3fff640 1 -- 172.21.15.46:0/4095941723 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+6526 (secure 0 0 0) 0x7f50cc0ca050 con 0x7f50cc0c05d0 2023-10-30T13:51:27.658 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:27.658 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7002,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:06.414877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:27.659 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2023-10-30T13:51:27.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.656+0000 7f50d209e640 1 -- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50b80664a0 msgr2=0x7f50b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.656+0000 7f50d209e640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50b80664a0 0x7f50b8068960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f50c00af000 tx=0x7f50c00a7000 comp rx=0 tx=0).stop 2023-10-30T13:51:27.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.656+0000 7f50d209e640 1 -- 172.21.15.46:0/4095941723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50cc0c05d0 msgr2=0x7f50cc0c59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.656+0000 7f50d209e640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50cc0c05d0 0x7f50cc0c59e0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00c480 tx=0x7f50bc00c940 comp rx=0 tx=0).stop 2023-10-30T13:51:27.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.656+0000 7f50d209e640 1 -- 172.21.15.46:0/4095941723 shutdown_connections 2023-10-30T13:51:27.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.656+0000 7f50d209e640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50b80664a0 0x7f50b8068960 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.656+0000 7f50d209e640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50cc0c6f60 0x7f50cc0c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.656+0000 7f50d209e640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50cc0c6b80 0x7f50cc0c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.656+0000 7f50d209e640 1 --2- 172.21.15.46:0/4095941723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50cc0c05d0 0x7f50cc0c59e0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.656+0000 7f50d209e640 1 -- 172.21.15.46:0/4095941723 >> 172.21.15.46:0/4095941723 conn(0x7f50cc0c7910 msgr2=0x7f50cc13be90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:27.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.656+0000 7f50d209e640 1 -- 172.21.15.46:0/4095941723 shutdown_connections 2023-10-30T13:51:27.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:27.656+0000 7f50d209e640 1 -- 172.21.15.46:0/4095941723 wait complete. 2023-10-30T13:51:27.680 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:51:29.681 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.832+0000 7fa94ee17640 1 Processor -- start 2023-10-30T13:51:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.832+0000 7fa94ee17640 1 -- start start 2023-10-30T13:51:29.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.832+0000 7fa94ee17640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa9480cb550 0x7fa9480cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.832+0000 7fa94ee17640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa948071250 0x7fa948071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.832+0000 7fa94ee17640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9481560a0 con 0x7fa94814fd50 2023-10-30T13:51:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.832+0000 7fa94ee17640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa948156380 con 0x7fa9480cb550 2023-10-30T13:51:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.832+0000 7fa94ee17640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa948156660 con 0x7fa948071250 2023-10-30T13:51:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.832+0000 7fa947fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa9480cb550 0x7fa9480cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.836+0000 7fa94cbb3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa94814fd50 0x7fa9480c65d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46370/0 (socket says 172.21.15.46:46370) 2023-10-30T13:51:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.836+0000 7fa94cbb3640 1 -- 172.21.15.46:0/3571158526 learned_addr learned my addr 172.21.15.46:0/3571158526 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:51:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.836+0000 7fa94d3b4640 1 --2- 172.21.15.46:0/3571158526 >> v2:172.21.15.154:3300/0 conn(0x7fa948071250 0x7fa948071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.836+0000 7fa947fff640 1 -- 172.21.15.46:0/3571158526 >> v2:172.21.15.154:3300/0 conn(0x7fa948071250 msgr2=0x7fa948071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.836+0000 7fa947fff640 1 --2- 172.21.15.46:0/3571158526 >> v2:172.21.15.154:3300/0 conn(0x7fa948071250 0x7fa948071650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.836+0000 7fa947fff640 1 -- 172.21.15.46:0/3571158526 >> v1:172.21.15.46:6789/0 conn(0x7fa94814fd50 legacy=0x7fa9480c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.836+0000 7fa947fff640 1 -- 172.21.15.46:0/3571158526 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa948156940 con 0x7fa9480cb550 2023-10-30T13:51:29.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.836+0000 7fa9477fe640 1 -- 172.21.15.46:0/3571158526 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133247898 0 0) 0x7fa9481560a0 con 0x7fa94814fd50 2023-10-30T13:51:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.836+0000 7fa947fff640 1 --2- 172.21.15.46:0/3571158526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9480cb550 0x7fa9480cb9e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fa9480c9010 tx=0x7fa93401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b976097582f8a7e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.836+0000 7fa9477fe640 1 -- 172.21.15.46:0/3571158526 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9340ab620 con 0x7fa9480cb550 2023-10-30T13:51:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.836+0000 7fa9477fe640 1 -- 172.21.15.46:0/3571158526 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9340ab780 con 0x7fa9480cb550 2023-10-30T13:51:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.836+0000 7fa9477fe640 1 -- 172.21.15.46:0/3571158526 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9340abb10 con 0x7fa9480cb550 2023-10-30T13:51:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 -- 172.21.15.46:0/3571158526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9480cb550 msgr2=0x7fa9480cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 --2- 172.21.15.46:0/3571158526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9480cb550 0x7fa9480cb9e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fa9480c9010 tx=0x7fa93401c460 comp rx=0 tx=0).stop 2023-10-30T13:51:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 -- 172.21.15.46:0/3571158526 shutdown_connections 2023-10-30T13:51:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 --2- 172.21.15.46:0/3571158526 >> v2:172.21.15.154:3300/0 conn(0x7fa948071250 0x7fa948071650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 --2- 172.21.15.46:0/3571158526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9480cb550 0x7fa9480cb9e0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 -- 172.21.15.46:0/3571158526 >> 172.21.15.46:0/3571158526 conn(0x7fa94806c6e0 msgr2=0x7fa948069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:29.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 -- 172.21.15.46:0/3571158526 shutdown_connections 2023-10-30T13:51:29.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 -- 172.21.15.46:0/3571158526 wait complete. 2023-10-30T13:51:29.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 Processor -- start 2023-10-30T13:51:29.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 -- start start 2023-10-30T13:51:29.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa948071250 0x7fa9481f3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9480cb550 0x7fa948143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa94814fd50 0x7fa9481efc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa948141750 con 0x7fa94814fd50 2023-10-30T13:51:29.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa948141a10 con 0x7fa9480cb550 2023-10-30T13:51:29.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94ee17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa948141cf0 con 0x7fa948071250 2023-10-30T13:51:29.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94d3b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa94814fd50 0x7fa9481efc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94d3b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa94814fd50 0x7fa9481efc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54862/0 (socket says 172.21.15.46:54862) 2023-10-30T13:51:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94d3b4640 1 -- 172.21.15.46:0/293258873 learned_addr learned my addr 172.21.15.46:0/293258873 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94cbb3640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa948071250 0x7fa9481f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa947fff640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9480cb550 0x7fa948143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94d3b4640 1 -- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa948071250 msgr2=0x7fa9481f3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94d3b4640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa948071250 0x7fa9481f3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94d3b4640 1 -- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9480cb550 msgr2=0x7fa948143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94d3b4640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9480cb550 0x7fa948143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94d3b4640 1 -- 172.21.15.46:0/293258873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa934009420 con 0x7fa94814fd50 2023-10-30T13:51:29.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa94cbb3640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa948071250 0x7fa9481f3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:29.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.840+0000 7fa947fff640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9480cb550 0x7fa948143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:29.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.844+0000 7fa94d3b4640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa94814fd50 0x7fa9481efc40 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fa940007e60 tx=0x7fa94000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:29.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.844+0000 7fa9457fa640 1 -- 172.21.15.46:0/293258873 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa940014420 con 0x7fa94814fd50 2023-10-30T13:51:29.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.844+0000 7fa9457fa640 1 -- 172.21.15.46:0/293258873 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9400145a0 con 0x7fa94814fd50 2023-10-30T13:51:29.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.844+0000 7fa9457fa640 1 -- 172.21.15.46:0/293258873 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa940014950 con 0x7fa94814fd50 2023-10-30T13:51:29.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.844+0000 7fa94ee17640 1 -- 172.21.15.46:0/293258873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa948141fd0 con 0x7fa94814fd50 2023-10-30T13:51:29.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.844+0000 7fa94ee17640 1 -- 172.21.15.46:0/293258873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9481f2e60 con 0x7fa94814fd50 2023-10-30T13:51:29.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.844+0000 7fa94ee17640 1 -- 172.21.15.46:0/293258873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94804e170 con 0x7fa94814fd50 2023-10-30T13:51:29.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.844+0000 7fa9457fa640 1 -- 172.21.15.46:0/293258873 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa940014ab0 con 0x7fa94814fd50 2023-10-30T13:51:29.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.844+0000 7fa9457fa640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa92c0664a0 0x7fa92c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.844+0000 7fa9457fa640 1 -- 172.21.15.46:0/293258873 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fa940084d30 con 0x7fa94814fd50 2023-10-30T13:51:29.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.844+0000 7fa94cbb3640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa92c0664a0 0x7fa92c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.848+0000 7fa94cbb3640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa92c0664a0 0x7fa92c068960 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa93c007fd0 tx=0x7fa93c007c20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:29.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:29.848+0000 7fa9457fa640 1 -- 172.21.15.46:0/293258873 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa940051c20 con 0x7fa94814fd50 2023-10-30T13:51:30.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.040+0000 7fa94ee17640 1 -- 172.21.15.46:0/293258873 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9481604f0 con 0x7fa94814fd50 2023-10-30T13:51:30.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.040+0000 7fa9457fa640 1 -- 172.21.15.46:0/293258873 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+6526 (secure 0 0 0) 0x7fa940055240 con 0x7fa94814fd50 2023-10-30T13:51:30.046 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:30.047 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7002,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:06.414877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:30.047 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2023-10-30T13:51:30.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.044+0000 7fa94ee17640 1 -- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa92c0664a0 msgr2=0x7fa92c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:30.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.044+0000 7fa94ee17640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa92c0664a0 0x7fa92c068960 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa93c007fd0 tx=0x7fa93c007c20 comp rx=0 tx=0).stop 2023-10-30T13:51:30.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.044+0000 7fa94ee17640 1 -- 172.21.15.46:0/293258873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa94814fd50 msgr2=0x7fa9481efc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:30.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.044+0000 7fa94ee17640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa94814fd50 0x7fa9481efc40 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fa940007e60 tx=0x7fa94000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:51:30.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.044+0000 7fa94ee17640 1 -- 172.21.15.46:0/293258873 shutdown_connections 2023-10-30T13:51:30.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.044+0000 7fa94ee17640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa92c0664a0 0x7fa92c068960 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:30.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.044+0000 7fa94ee17640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa94814fd50 0x7fa9481efc40 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:30.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.044+0000 7fa94ee17640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9480cb550 0x7fa948143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:30.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.044+0000 7fa94ee17640 1 --2- 172.21.15.46:0/293258873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa948071250 0x7fa9481f3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:30.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.044+0000 7fa94ee17640 1 -- 172.21.15.46:0/293258873 >> 172.21.15.46:0/293258873 conn(0x7fa94806c6e0 msgr2=0x7fa9480c9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:30.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.044+0000 7fa94ee17640 1 -- 172.21.15.46:0/293258873 shutdown_connections 2023-10-30T13:51:30.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:30.044+0000 7fa94ee17640 1 -- 172.21.15.46:0/293258873 wait complete. 2023-10-30T13:51:30.067 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:51:32.069 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:32.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff13ada1640 1 Processor -- start 2023-10-30T13:51:32.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff13ada1640 1 -- start start 2023-10-30T13:51:32.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff13ada1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff134069e90 0x7ff13415a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:32.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff13ada1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff13406a270 0x7ff1340c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:32.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff13ada1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff13406cb80 0x7ff13406d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:32.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff13ada1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1340c8570 con 0x7ff13406a270 2023-10-30T13:51:32.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff13ada1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff13406d550 con 0x7ff134069e90 2023-10-30T13:51:32.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff13ada1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1340c6c10 con 0x7ff13406cb80 2023-10-30T13:51:32.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff133fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff13406a270 0x7ff1340c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:32.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff133fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff13406a270 0x7ff1340c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54870/0 (socket says 172.21.15.46:54870) 2023-10-30T13:51:32.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff133fff640 1 -- 172.21.15.46:0/2926656194 learned_addr learned my addr 172.21.15.46:0/2926656194 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:32.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff138b3d640 1 --2- 172.21.15.46:0/2926656194 >> v2:172.21.15.154:3300/0 conn(0x7ff134069e90 0x7ff13415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:32.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff13933e640 1 --2- 172.21.15.46:0/2926656194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff13406cb80 0x7ff13406d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:32.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff133fff640 1 -- 172.21.15.46:0/2926656194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff13406cb80 msgr2=0x7ff13406d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:32.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff133fff640 1 --2- 172.21.15.46:0/2926656194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff13406cb80 0x7ff13406d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:32.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff133fff640 1 -- 172.21.15.46:0/2926656194 >> v2:172.21.15.154:3300/0 conn(0x7ff134069e90 msgr2=0x7ff13415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:32.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff133fff640 1 --2- 172.21.15.46:0/2926656194 >> v2:172.21.15.154:3300/0 conn(0x7ff134069e90 0x7ff13415a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:32.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.220+0000 7ff133fff640 1 -- 172.21.15.46:0/2926656194 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1340c6ef0 con 0x7ff13406a270 2023-10-30T13:51:32.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff133fff640 1 --2- 172.21.15.46:0/2926656194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff13406a270 0x7ff1340c9cb0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7ff13406ed20 tx=0x7ff12001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1ea5a0059f2ae8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:32.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff1337fe640 1 -- 172.21.15.46:0/2926656194 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1200ac410 con 0x7ff13406a270 2023-10-30T13:51:32.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff1337fe640 1 -- 172.21.15.46:0/2926656194 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1200ac570 con 0x7ff13406a270 2023-10-30T13:51:32.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff1337fe640 1 -- 172.21.15.46:0/2926656194 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1200ac900 con 0x7ff13406a270 2023-10-30T13:51:32.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff13ada1640 1 -- 172.21.15.46:0/2926656194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff13406a270 msgr2=0x7ff1340c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:32.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff13ada1640 1 --2- 172.21.15.46:0/2926656194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff13406a270 0x7ff1340c9cb0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7ff13406ed20 tx=0x7ff12001c500 comp rx=0 tx=0).stop 2023-10-30T13:51:32.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff13ada1640 1 -- 172.21.15.46:0/2926656194 shutdown_connections 2023-10-30T13:51:32.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff13ada1640 1 --2- 172.21.15.46:0/2926656194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff13406cb80 0x7ff13406d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:32.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff13ada1640 1 --2- 172.21.15.46:0/2926656194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff13406a270 0x7ff1340c9cb0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:32.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff13ada1640 1 --2- 172.21.15.46:0/2926656194 >> v2:172.21.15.154:3300/0 conn(0x7ff134069e90 0x7ff13415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:32.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff13ada1640 1 -- 172.21.15.46:0/2926656194 >> 172.21.15.46:0/2926656194 conn(0x7ff13406dfd0 msgr2=0x7ff13406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:32.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff13ada1640 1 -- 172.21.15.46:0/2926656194 shutdown_connections 2023-10-30T13:51:32.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.224+0000 7ff13ada1640 1 -- 172.21.15.46:0/2926656194 wait complete. 2023-10-30T13:51:32.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff13ada1640 1 Processor -- start 2023-10-30T13:51:32.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff13ada1640 1 -- start start 2023-10-30T13:51:32.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff13ada1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff134069e90 0x7ff134154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:32.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff13ada1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff13406a270 0x7ff134158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:32.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff13ada1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff13406cb80 0x7ff13414e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff13ada1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff134158030 con 0x7ff13406a270 2023-10-30T13:51:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff13ada1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1341f5f50 con 0x7ff134069e90 2023-10-30T13:51:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff13ada1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff134157440 con 0x7ff13406cb80 2023-10-30T13:51:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff133fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff13406a270 0x7ff134158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff133fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff13406a270 0x7ff134158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54872/0 (socket says 172.21.15.46:54872) 2023-10-30T13:51:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff133fff640 1 -- 172.21.15.46:0/1858446203 learned_addr learned my addr 172.21.15.46:0/1858446203 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff138b3d640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff134069e90 0x7ff134154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff13933e640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff13406cb80 0x7ff13414e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff133fff640 1 -- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff13406cb80 msgr2=0x7ff13414e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff133fff640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff13406cb80 0x7ff13414e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff133fff640 1 -- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff134069e90 msgr2=0x7ff134154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:32.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff133fff640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff134069e90 0x7ff134154210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:32.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff133fff640 1 -- 172.21.15.46:0/1858446203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1200094d0 con 0x7ff13406a270 2023-10-30T13:51:32.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff133fff640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff13406a270 0x7ff134158450 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7ff12001abb0 tx=0x7ff1200192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:32.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff1317fa640 1 -- 172.21.15.46:0/1858446203 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1200ac410 con 0x7ff13406a270 2023-10-30T13:51:32.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff1317fa640 1 -- 172.21.15.46:0/1858446203 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1200b45c0 con 0x7ff13406a270 2023-10-30T13:51:32.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff13ada1640 1 -- 172.21.15.46:0/1858446203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1341f6140 con 0x7ff13406a270 2023-10-30T13:51:32.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff1317fa640 1 -- 172.21.15.46:0/1858446203 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1200b4990 con 0x7ff13406a270 2023-10-30T13:51:32.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.228+0000 7ff13ada1640 1 -- 172.21.15.46:0/1858446203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff134157690 con 0x7ff13406a270 2023-10-30T13:51:32.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.232+0000 7ff13ada1640 1 -- 172.21.15.46:0/1858446203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff134158170 con 0x7ff13406a270 2023-10-30T13:51:32.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.232+0000 7ff1317fa640 1 -- 172.21.15.46:0/1858446203 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1200b4af0 con 0x7ff13406a270 2023-10-30T13:51:32.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.232+0000 7ff1317fa640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1180664a0 0x7ff118068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:32.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.232+0000 7ff1317fa640 1 -- 172.21.15.46:0/1858446203 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7ff12011e9c0 con 0x7ff13406a270 2023-10-30T13:51:32.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.232+0000 7ff138b3d640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1180664a0 0x7ff118068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:32.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.232+0000 7ff138b3d640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1180664a0 0x7ff118068960 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff12800ba30 tx=0x7ff128008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:32.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.236+0000 7ff1317fa640 1 -- 172.21.15.46:0/1858446203 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff134158170 con 0x7ff13406a270 2023-10-30T13:51:32.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 -- 172.21.15.46:0/1858446203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff134158170 con 0x7ff13406a270 2023-10-30T13:51:32.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff1317fa640 1 -- 172.21.15.46:0/1858446203 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+6526 (secure 0 0 0) 0x7ff134158170 con 0x7ff13406a270 2023-10-30T13:51:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:32.433 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7002,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:06.414877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:32.433 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2023-10-30T13:51:32.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 -- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1180664a0 msgr2=0x7ff118068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:32.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1180664a0 0x7ff118068960 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff12800ba30 tx=0x7ff128008040 comp rx=0 tx=0).stop 2023-10-30T13:51:32.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 -- 172.21.15.46:0/1858446203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff13406a270 msgr2=0x7ff134158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:32.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff13406a270 0x7ff134158450 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7ff12001abb0 tx=0x7ff1200192a0 comp rx=0 tx=0).stop 2023-10-30T13:51:32.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 -- 172.21.15.46:0/1858446203 shutdown_connections 2023-10-30T13:51:32.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff1180664a0 0x7ff118068960 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:32.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff13406cb80 0x7ff13414e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:32.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff13406a270 0x7ff134158450 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:32.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 --2- 172.21.15.46:0/1858446203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff134069e90 0x7ff134154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:32.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 -- 172.21.15.46:0/1858446203 >> 172.21.15.46:0/1858446203 conn(0x7ff13406dfd0 msgr2=0x7ff13415abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:32.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 -- 172.21.15.46:0/1858446203 shutdown_connections 2023-10-30T13:51:32.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:32.428+0000 7ff13ada1640 1 -- 172.21.15.46:0/1858446203 wait complete. 2023-10-30T13:51:32.454 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:51:34.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:34.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.604+0000 7f3244409640 1 Processor -- start 2023-10-30T13:51:34.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.604+0000 7f3244409640 1 -- start start 2023-10-30T13:51:34.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f3244409640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f323c070a00 0x7f323c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f3244409640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f323c069e90 0x7f323c06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f3244409640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323c071380 con 0x7f323c1539a0 2023-10-30T13:51:34.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f3244409640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f323c06fbb0 con 0x7f323c069e90 2023-10-30T13:51:34.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f3244409640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323c153d80 con 0x7f323c070a00 2023-10-30T13:51:34.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f32429a6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f323c069e90 0x7f323c06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f32429a6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f323c069e90 0x7f323c06a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54886/0 (socket says 172.21.15.46:54886) 2023-10-30T13:51:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f32429a6640 1 -- 172.21.15.46:0/595673286 learned_addr learned my addr 172.21.15.46:0/595673286 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f32419a4640 1 --2- 172.21.15.46:0/595673286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f323c070a00 0x7f323c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f32429a6640 1 -- 172.21.15.46:0/595673286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f323c070a00 msgr2=0x7f323c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f32429a6640 1 --2- 172.21.15.46:0/595673286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f323c070a00 0x7f323c070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f32429a6640 1 -- 172.21.15.46:0/595673286 >> v1:172.21.15.154:6789/0 conn(0x7f323c1539a0 legacy=0x7f323c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:34.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.608+0000 7f32429a6640 1 -- 172.21.15.46:0/595673286 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c0c5c50 con 0x7f323c069e90 2023-10-30T13:51:34.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f32429a6640 1 --2- 172.21.15.46:0/595673286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f323c069e90 0x7f323c06a300 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3234009e50 tx=0x7f323401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9c9fed091e8e4c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:34.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f32411a3640 1 -- 172.21.15.46:0/595673286 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32340a5720 con 0x7f323c069e90 2023-10-30T13:51:34.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f32411a3640 1 -- 172.21.15.46:0/595673286 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32340a5880 con 0x7f323c069e90 2023-10-30T13:51:34.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f32411a3640 1 -- 172.21.15.46:0/595673286 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32340a5c10 con 0x7f323c069e90 2023-10-30T13:51:34.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 -- 172.21.15.46:0/595673286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f323c069e90 msgr2=0x7f323c06a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:34.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 --2- 172.21.15.46:0/595673286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f323c069e90 0x7f323c06a300 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3234009e50 tx=0x7f323401c950 comp rx=0 tx=0).stop 2023-10-30T13:51:34.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 -- 172.21.15.46:0/595673286 shutdown_connections 2023-10-30T13:51:34.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 --2- 172.21.15.46:0/595673286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f323c069e90 0x7f323c06a300 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 --2- 172.21.15.46:0/595673286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f323c070a00 0x7f323c070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 -- 172.21.15.46:0/595673286 >> 172.21.15.46:0/595673286 conn(0x7f323c0c7910 msgr2=0x7f323c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:34.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 -- 172.21.15.46:0/595673286 shutdown_connections 2023-10-30T13:51:34.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 -- 172.21.15.46:0/595673286 wait complete. 2023-10-30T13:51:34.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 Processor -- start 2023-10-30T13:51:34.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 -- start start 2023-10-30T13:51:34.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f323c069e90 0x7f323c0c0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f323c070a00 0x7f323c158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f323c1539a0 0x7f323c150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0c16b0 con 0x7f323c070a00 2023-10-30T13:51:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0c12d0 con 0x7f323c069e90 2023-10-30T13:51:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.612+0000 7f3244409640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323c146fe0 con 0x7f323c1539a0 2023-10-30T13:51:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f32419a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f323c070a00 0x7f323c158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f32419a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f323c070a00 0x7f323c158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54902/0 (socket says 172.21.15.46:54902) 2023-10-30T13:51:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f32419a4640 1 -- 172.21.15.46:0/3462881089 learned_addr learned my addr 172.21.15.46:0/3462881089 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f32421a5640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f323c069e90 0x7f323c0c0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f32429a6640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f323c1539a0 0x7f323c150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f32419a4640 1 -- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f323c1539a0 msgr2=0x7f323c150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f32419a4640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f323c1539a0 0x7f323c150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f32419a4640 1 -- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f323c069e90 msgr2=0x7f323c0c0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f32419a4640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f323c069e90 0x7f323c0c0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f32419a4640 1 -- 172.21.15.46:0/3462881089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3234009b30 con 0x7f323c070a00 2023-10-30T13:51:34.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f32421a5640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f323c069e90 0x7f323c0c0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:34.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f32419a4640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f323c070a00 0x7f323c158560 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f322c00d800 tx=0x7f322c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:34.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f3232ffd640 1 -- 172.21.15.46:0/3462881089 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c00b6c0 con 0x7f323c070a00 2023-10-30T13:51:34.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f3232ffd640 1 -- 172.21.15.46:0/3462881089 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c00b820 con 0x7f323c070a00 2023-10-30T13:51:34.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f3232ffd640 1 -- 172.21.15.46:0/3462881089 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c00bb90 con 0x7f323c070a00 2023-10-30T13:51:34.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f3244409640 1 -- 172.21.15.46:0/3462881089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323c0c1970 con 0x7f323c070a00 2023-10-30T13:51:34.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f3244409640 1 -- 172.21.15.46:0/3462881089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323c0c0240 con 0x7f323c070a00 2023-10-30T13:51:34.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f3244409640 1 -- 172.21.15.46:0/3462881089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323c04e170 con 0x7f323c070a00 2023-10-30T13:51:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f3232ffd640 1 -- 172.21.15.46:0/3462881089 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f322c00bcf0 con 0x7f323c070a00 2023-10-30T13:51:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.616+0000 7f3232ffd640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32200664a0 0x7f3220068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.620+0000 7f3232ffd640 1 -- 172.21.15.46:0/3462881089 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f322c084830 con 0x7f323c070a00 2023-10-30T13:51:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.620+0000 7f32421a5640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32200664a0 0x7f3220068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.620+0000 7f32421a5640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32200664a0 0x7f3220068960 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3238000f70 tx=0x7f3238009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.624+0000 7f3232ffd640 1 -- 172.21.15.46:0/3462881089 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f322c0517b0 con 0x7f323c070a00 2023-10-30T13:51:34.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.808+0000 7f3244409640 1 -- 172.21.15.46:0/3462881089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f323c159ac0 con 0x7f323c070a00 2023-10-30T13:51:34.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3232ffd640 1 -- 172.21.15.46:0/3462881089 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+6526 (secure 0 0 0) 0x7f322c054dd0 con 0x7f323c070a00 2023-10-30T13:51:34.816 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:34.816 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7002,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:06.414877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:34.817 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2023-10-30T13:51:34.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3244409640 1 -- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32200664a0 msgr2=0x7f3220068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:34.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3244409640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32200664a0 0x7f3220068960 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3238000f70 tx=0x7f3238009040 comp rx=0 tx=0).stop 2023-10-30T13:51:34.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3244409640 1 -- 172.21.15.46:0/3462881089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f323c070a00 msgr2=0x7f323c158560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:34.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3244409640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f323c070a00 0x7f323c158560 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f322c00d800 tx=0x7f322c00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3244409640 1 -- 172.21.15.46:0/3462881089 shutdown_connections 2023-10-30T13:51:34.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3244409640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32200664a0 0x7f3220068960 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3244409640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f323c1539a0 0x7f323c150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3244409640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f323c070a00 0x7f323c158560 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3244409640 1 --2- 172.21.15.46:0/3462881089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f323c069e90 0x7f323c0c0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:34.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3244409640 1 -- 172.21.15.46:0/3462881089 >> 172.21.15.46:0/3462881089 conn(0x7f323c0c7910 msgr2=0x7f323c1525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:34.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3244409640 1 -- 172.21.15.46:0/3462881089 shutdown_connections 2023-10-30T13:51:34.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:34.812+0000 7f3244409640 1 -- 172.21.15.46:0/3462881089 wait complete. 2023-10-30T13:51:34.837 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:51:36.838 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:36.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.988+0000 7f5f628d6640 1 Processor -- start 2023-10-30T13:51:36.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.988+0000 7f5f628d6640 1 -- start start 2023-10-30T13:51:36.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f628d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f5c0c6b80 0x7f5f5c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:36.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f628d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f5c13f4a0 0x7f5f5c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:36.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f628d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c0c0a50 con 0x7f5f5c0c6f60 2023-10-30T13:51:36.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f628d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f5c06f6e0 con 0x7f5f5c13f4a0 2023-10-30T13:51:36.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f628d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c06f9c0 con 0x7f5f5c0c6b80 2023-10-30T13:51:36.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f60e73640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f5c13f4a0 0x7f5f5c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:36.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f60e73640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f5c13f4a0 0x7f5f5c0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53880/0 (socket says 172.21.15.46:53880) 2023-10-30T13:51:36.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f60e73640 1 -- 172.21.15.46:0/632407722 learned_addr learned my addr 172.21.15.46:0/632407722 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:36.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f5bfff640 1 --2- 172.21.15.46:0/632407722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f5c0c6b80 0x7f5f5c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:36.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f60e73640 1 -- 172.21.15.46:0/632407722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f5c0c6b80 msgr2=0x7f5f5c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:36.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f60e73640 1 --2- 172.21.15.46:0/632407722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f5c0c6b80 0x7f5f5c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:36.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f60e73640 1 -- 172.21.15.46:0/632407722 >> v1:172.21.15.154:6789/0 conn(0x7f5f5c0c6f60 legacy=0x7f5f5c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:36.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.992+0000 7f5f60e73640 1 -- 172.21.15.46:0/632407722 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c06fca0 con 0x7f5f5c13f4a0 2023-10-30T13:51:37.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f60e73640 1 --2- 172.21.15.46:0/632407722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c13f4a0 0x7f5f5c0c0400 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400d150 tx=0x7f5f5401ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86520f6cd1d5adb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:37.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f5affd640 1 -- 172.21.15.46:0/632407722 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5400b2d0 con 0x7f5f5c13f4a0 2023-10-30T13:51:37.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f5affd640 1 -- 172.21.15.46:0/632407722 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5400b430 con 0x7f5f5c13f4a0 2023-10-30T13:51:37.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f5affd640 1 -- 172.21.15.46:0/632407722 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f540a7c10 con 0x7f5f5c13f4a0 2023-10-30T13:51:37.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 -- 172.21.15.46:0/632407722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c13f4a0 msgr2=0x7f5f5c0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:37.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 --2- 172.21.15.46:0/632407722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c13f4a0 0x7f5f5c0c0400 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400d150 tx=0x7f5f5401ec10 comp rx=0 tx=0).stop 2023-10-30T13:51:37.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 -- 172.21.15.46:0/632407722 shutdown_connections 2023-10-30T13:51:37.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 --2- 172.21.15.46:0/632407722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c13f4a0 0x7f5f5c0c0400 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:37.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 --2- 172.21.15.46:0/632407722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f5c0c6b80 0x7f5f5c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:37.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 -- 172.21.15.46:0/632407722 >> 172.21.15.46:0/632407722 conn(0x7f5f5c0c7910 msgr2=0x7f5f5c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:37.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 -- 172.21.15.46:0/632407722 shutdown_connections 2023-10-30T13:51:37.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 -- 172.21.15.46:0/632407722 wait complete. 2023-10-30T13:51:37.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 Processor -- start 2023-10-30T13:51:37.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 -- start start 2023-10-30T13:51:37.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f5c0c6b80 0x7f5f5c137980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:37.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c0c6f60 0x7f5f5c1f5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:37.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f5c13f4a0 0x7f5f5c06b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:37.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c15ec30 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c1f2e40 con 0x7f5f5c13f4a0 2023-10-30T13:51:37.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:36.996+0000 7f5f628d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c1f3400 con 0x7f5f5c0c6b80 2023-10-30T13:51:37.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f5b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c0c6f60 0x7f5f5c1f5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:37.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f5b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c0c6f60 0x7f5f5c1f5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53894/0 (socket says 172.21.15.46:53894) 2023-10-30T13:51:37.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f5b7fe640 1 -- 172.21.15.46:0/3688448802 learned_addr learned my addr 172.21.15.46:0/3688448802 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:37.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f60e73640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f5c13f4a0 0x7f5f5c06b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:37.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f5bfff640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f5c0c6b80 0x7f5f5c137980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:37.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f5b7fe640 1 -- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f5c0c6b80 msgr2=0x7f5f5c137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:37.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f5b7fe640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f5c0c6b80 0x7f5f5c137980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:37.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f5b7fe640 1 -- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f5c13f4a0 msgr2=0x7f5f5c06b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:37.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f5b7fe640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f5c13f4a0 0x7f5f5c06b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:37.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f5b7fe640 1 -- 172.21.15.46:0/3688448802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5400bd40 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f5b7fe640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c0c6f60 0x7f5f5c1f5900 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000c500 tx=0x7f5f5000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:37.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f58ff9640 1 -- 172.21.15.46:0/3688448802 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f500052d0 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f58ff9640 1 -- 172.21.15.46:0/3688448802 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f50005430 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f628d6640 1 -- 172.21.15.46:0/3688448802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5c137ec0 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f58ff9640 1 -- 172.21.15.46:0/3688448802 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50012830 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f628d6640 1 -- 172.21.15.46:0/3688448802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5c15fbc0 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f628d6640 1 -- 172.21.15.46:0/3688448802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5c04e170 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.000+0000 7f5f58ff9640 1 -- 172.21.15.46:0/3688448802 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5f50012990 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.004+0000 7f5f58ff9640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f480664a0 0x7f5f48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:37.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.004+0000 7f5f58ff9640 1 -- 172.21.15.46:0/3688448802 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f5f50083dc0 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.004+0000 7f5f5bfff640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f480664a0 0x7f5f48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:37.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.004+0000 7f5f5bfff640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f480664a0 0x7f5f48068960 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c009b60 tx=0x7f5f4c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:37.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.008+0000 7f5f58ff9640 1 -- 172.21.15.46:0/3688448802 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5f50050d70 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.196+0000 7f5f628d6640 1 -- 172.21.15.46:0/3688448802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f5c0b0b40 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.196+0000 7f5f58ff9640 1 -- 172.21.15.46:0/3688448802 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+6526 (secure 0 0 0) 0x7f5f50054390 con 0x7f5f5c0c6f60 2023-10-30T13:51:37.203 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:37.203 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7002,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:06.414877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6702,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6702":{"gid":6702,"name":"c","rank":0,"incarnation":111,"state":"up:active","state_seq":43,"addr":"172.21.15.46:6836/2197729559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2197729559},{"type":"v1","addr":"172.21.15.46:6836","nonce":2197729559}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:37.203 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2023-10-30T13:51:37.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.200+0000 7f5f628d6640 1 -- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f480664a0 msgr2=0x7f5f48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:37.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.200+0000 7f5f628d6640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f480664a0 0x7f5f48068960 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c009b60 tx=0x7f5f4c009190 comp rx=0 tx=0).stop 2023-10-30T13:51:37.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.200+0000 7f5f628d6640 1 -- 172.21.15.46:0/3688448802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c0c6f60 msgr2=0x7f5f5c1f5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:37.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.200+0000 7f5f628d6640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c0c6f60 0x7f5f5c1f5900 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000c500 tx=0x7f5f5000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:51:37.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.200+0000 7f5f628d6640 1 -- 172.21.15.46:0/3688448802 shutdown_connections 2023-10-30T13:51:37.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.200+0000 7f5f628d6640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f480664a0 0x7f5f48068960 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:37.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.200+0000 7f5f628d6640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f5c13f4a0 0x7f5f5c06b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:37.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.200+0000 7f5f628d6640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c0c6f60 0x7f5f5c1f5900 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:37.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.200+0000 7f5f628d6640 1 --2- 172.21.15.46:0/3688448802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f5c0c6b80 0x7f5f5c137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:37.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.200+0000 7f5f628d6640 1 -- 172.21.15.46:0/3688448802 >> 172.21.15.46:0/3688448802 conn(0x7f5f5c0c7910 msgr2=0x7f5f5c0c5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:37.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.200+0000 7f5f628d6640 1 -- 172.21.15.46:0/3688448802 shutdown_connections 2023-10-30T13:51:37.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:37.200+0000 7f5f628d6640 1 -- 172.21.15.46:0/3688448802 wait complete. 2023-10-30T13:51:37.226 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:51:39.227 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.376+0000 7eff21fd3640 1 Processor -- start 2023-10-30T13:51:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.376+0000 7eff21fd3640 1 -- start start 2023-10-30T13:51:39.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff21fd3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7eff1c0c5f60 0x7eff1c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff21fd3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1c0700b0 con 0x7eff1c0c6fe0 2023-10-30T13:51:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff21fd3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1c0c6530 con 0x7eff1c153bf0 2023-10-30T13:51:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff21fd3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff1c0c6810 con 0x7eff1c0c5f60 2023-10-30T13:51:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7eff1c0c6fe0 0x7eff1c0cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48782/0 (socket says 172.21.15.46:48782) 2023-10-30T13:51:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1affd640 1 -- 172.21.15.46:0/4029609800 learned_addr learned my addr 172.21.15.46:0/4029609800 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:51:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1bfff640 1 --2- 172.21.15.46:0/4029609800 >> v2:172.21.15.154:3300/0 conn(0x7eff1c0c5f60 0x7eff1c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677905736 0 0) 0x7eff1c0700b0 con 0x7eff1c0c6fe0 2023-10-30T13:51:39.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff04003530 con 0x7eff1c0c6fe0 2023-10-30T13:51:39.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3094584197 0 0) 0x7eff04003530 con 0x7eff1c0c6fe0 2023-10-30T13:51:39.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 >> v2:172.21.15.154:3300/0 conn(0x7eff1c0c5f60 msgr2=0x7eff1c071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:39.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 --2- 172.21.15.46:0/4029609800 >> v2:172.21.15.154:3300/0 conn(0x7eff1c0c5f60 0x7eff1c071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 >> v1:172.21.15.154:6789/0 conn(0x7eff1c153bf0 legacy=0x7eff1c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c159b10 con 0x7eff1c0c6fe0 2023-10-30T13:51:39.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7eff10001680 con 0x7eff1c0c6fe0 2023-10-30T13:51:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 >> v1:172.21.15.46:6789/0 conn(0x7eff1c0c6fe0 legacy=0x7eff1c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 --2- 172.21.15.46:0/4029609800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff04003980 0x7eff04003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 --2- 172.21.15.46:0/4029609800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 0x7eff04004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 --2- 172.21.15.46:0/4029609800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff04005190 0x7eff0400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c0c6810 con 0x7eff040047d0 2023-10-30T13:51:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0400c040 con 0x7eff04005190 2023-10-30T13:51:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0400c200 con 0x7eff04003980 2023-10-30T13:51:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1affd640 1 --2- 172.21.15.46:0/4029609800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 0x7eff04004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1b7fe640 1 --2- 172.21.15.46:0/4029609800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff04003980 0x7eff04003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1bfff640 1 --2- 172.21.15.46:0/4029609800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff04005190 0x7eff0400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1affd640 1 -- 172.21.15.46:0/4029609800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff04003980 msgr2=0x7eff04003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1affd640 1 --2- 172.21.15.46:0/4029609800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff04003980 0x7eff04003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1affd640 1 -- 172.21.15.46:0/4029609800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff04005190 msgr2=0x7eff0400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:39.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1affd640 1 --2- 172.21.15.46:0/4029609800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff04005190 0x7eff0400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1affd640 1 -- 172.21.15.46:0/4029609800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff04003530 con 0x7eff040047d0 2023-10-30T13:51:39.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.380+0000 7eff1bfff640 1 -- 172.21.15.46:0/4029609800 reap_dead start 2023-10-30T13:51:39.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff1affd640 1 --2- 172.21.15.46:0/4029609800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 0x7eff04004c50 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7eff1000c1c0 tx=0x7eff1001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12cde20bca2fd069 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:39.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10005ae0 con 0x7eff040047d0 2023-10-30T13:51:39.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff10005c40 con 0x7eff040047d0 2023-10-30T13:51:39.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff1a7fc640 1 -- 172.21.15.46:0/4029609800 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff100a5b20 con 0x7eff040047d0 2023-10-30T13:51:39.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff21fd3640 1 -- 172.21.15.46:0/4029609800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 msgr2=0x7eff04004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff21fd3640 1 --2- 172.21.15.46:0/4029609800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 0x7eff04004c50 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7eff1000c1c0 tx=0x7eff1001cbe0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff21fd3640 1 -- 172.21.15.46:0/4029609800 shutdown_connections 2023-10-30T13:51:39.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff21fd3640 1 --2- 172.21.15.46:0/4029609800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 0x7eff04004c50 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff21fd3640 1 -- 172.21.15.46:0/4029609800 >> 172.21.15.46:0/4029609800 conn(0x7eff1c06c6e0 msgr2=0x7eff1c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:39.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff21fd3640 1 -- 172.21.15.46:0/4029609800 shutdown_connections 2023-10-30T13:51:39.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff21fd3640 1 -- 172.21.15.46:0/4029609800 wait complete. 2023-10-30T13:51:39.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff21fd3640 1 Processor -- start 2023-10-30T13:51:39.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.384+0000 7eff21fd3640 1 -- start start 2023-10-30T13:51:39.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff21fd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 0x7eff1c155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff21fd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff1c1547a0 0x7eff1c158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff21fd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0c4f30 0x7eff1c1f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff21fd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c159910 con 0x7eff040047d0 2023-10-30T13:51:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff21fd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c156f10 con 0x7eff1c1547a0 2023-10-30T13:51:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff21fd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c1593e0 con 0x7eff1c0c4f30 2023-10-30T13:51:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff1b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 0x7eff1c155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff1b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 0x7eff1c155670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53908/0 (socket says 172.21.15.46:53908) 2023-10-30T13:51:39.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff1b7fe640 1 -- 172.21.15.46:0/2109789816 learned_addr learned my addr 172.21.15.46:0/2109789816 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:39.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff1affd640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff1c1547a0 0x7eff1c158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff1bfff640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0c4f30 0x7eff1c1f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff1b7fe640 1 -- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0c4f30 msgr2=0x7eff1c1f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff1b7fe640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0c4f30 0x7eff1c1f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff1b7fe640 1 -- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff1c1547a0 msgr2=0x7eff1c158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff1b7fe640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff1c1547a0 0x7eff1c158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff1b7fe640 1 -- 172.21.15.46:0/2109789816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c157140 con 0x7eff040047d0 2023-10-30T13:51:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff1bfff640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0c4f30 0x7eff1c1f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff1b7fe640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 0x7eff1c155670 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00f5f0 tx=0x7eff0c00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff03fff640 1 -- 172.21.15.46:0/2109789816 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c0052e0 con 0x7eff040047d0 2023-10-30T13:51:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff03fff640 1 -- 172.21.15.46:0/2109789816 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0c005440 con 0x7eff040047d0 2023-10-30T13:51:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff03fff640 1 -- 172.21.15.46:0/2109789816 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c003820 con 0x7eff040047d0 2023-10-30T13:51:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff21fd3640 1 -- 172.21.15.46:0/2109789816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1c159630 con 0x7eff040047d0 2023-10-30T13:51:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff21fd3640 1 -- 172.21.15.46:0/2109789816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff1c152b20 con 0x7eff040047d0 2023-10-30T13:51:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.388+0000 7eff21fd3640 1 -- 172.21.15.46:0/2109789816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1c0c4870 con 0x7eff040047d0 2023-10-30T13:51:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.392+0000 7eff03fff640 1 -- 172.21.15.46:0/2109789816 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7eff0c003980 con 0x7eff040047d0 2023-10-30T13:51:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.392+0000 7eff03fff640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7eff080664a0 0x7eff08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.392+0000 7eff03fff640 1 -- 172.21.15.46:0/2109789816 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7eff0c0841f0 con 0x7eff040047d0 2023-10-30T13:51:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.392+0000 7eff1affd640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7eff080664a0 0x7eff08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.392+0000 7eff1affd640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7eff080664a0 0x7eff08068960 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7eff100ab000 tx=0x7eff100b1040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.396+0000 7eff03fff640 1 -- 172.21.15.46:0/2109789816 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7eff1c0c4870 con 0x7eff040047d0 2023-10-30T13:51:39.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.584+0000 7eff21fd3640 1 -- 172.21.15.46:0/2109789816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff1c0c4870 con 0x7eff040047d0 2023-10-30T13:51:39.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.584+0000 7eff03fff640 1 -- 172.21.15.46:0/2109789816 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5732 (secure 0 0 0) 0x7eff1c0c4870 con 0x7eff040047d0 2023-10-30T13:51:39.590 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:39.590 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:39.216908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:39.591 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 125 2023-10-30T13:51:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.588+0000 7eff21fd3640 1 -- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7eff080664a0 msgr2=0x7eff08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.588+0000 7eff21fd3640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7eff080664a0 0x7eff08068960 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7eff100ab000 tx=0x7eff100b1040 comp rx=0 tx=0).stop 2023-10-30T13:51:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.588+0000 7eff21fd3640 1 -- 172.21.15.46:0/2109789816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 msgr2=0x7eff1c155670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.588+0000 7eff21fd3640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 0x7eff1c155670 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00f5f0 tx=0x7eff0c00fab0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.588+0000 7eff21fd3640 1 -- 172.21.15.46:0/2109789816 shutdown_connections 2023-10-30T13:51:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.588+0000 7eff21fd3640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7eff080664a0 0x7eff08068960 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.588+0000 7eff21fd3640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0c4f30 0x7eff1c1f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.588+0000 7eff21fd3640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff1c1547a0 0x7eff1c158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.588+0000 7eff21fd3640 1 --2- 172.21.15.46:0/2109789816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff040047d0 0x7eff1c155670 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.588+0000 7eff21fd3640 1 -- 172.21.15.46:0/2109789816 >> 172.21.15.46:0/2109789816 conn(0x7eff1c06c6e0 msgr2=0x7eff1c150dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.588+0000 7eff21fd3640 1 -- 172.21.15.46:0/2109789816 shutdown_connections 2023-10-30T13:51:39.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.588+0000 7eff21fd3640 1 -- 172.21.15.46:0/2109789816 wait complete. 2023-10-30T13:51:39.611 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:51:39.612 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:51:39.612 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:39.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa89a46640 1 Processor -- start 2023-10-30T13:51:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa89a46640 1 -- start start 2023-10-30T13:51:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa89a46640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffa840c8cf0 0x7ffa840c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa89a46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa840ca340 0x7ffa8415d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa89a46640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa8415ce30 con 0x7ffa840c90d0 2023-10-30T13:51:39.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa89a46640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa84159880 con 0x7ffa840c8cf0 2023-10-30T13:51:39.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa89a46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa84159ae0 con 0x7ffa840ca340 2023-10-30T13:51:39.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa82ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffa840c8cf0 0x7ffa840c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa82ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffa840c8cf0 0x7ffa840c9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53922/0 (socket says 172.21.15.46:53922) 2023-10-30T13:51:39.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa82ffd640 1 -- 172.21.15.46:0/2134407667 learned_addr learned my addr 172.21.15.46:0/2134407667 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:39.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa837fe640 1 --2- 172.21.15.46:0/2134407667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa840ca340 0x7ffa8415d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa82ffd640 1 -- 172.21.15.46:0/2134407667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa840ca340 msgr2=0x7ffa8415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:39.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa82ffd640 1 --2- 172.21.15.46:0/2134407667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa840ca340 0x7ffa8415d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa82ffd640 1 -- 172.21.15.46:0/2134407667 >> v1:172.21.15.154:6789/0 conn(0x7ffa840c90d0 legacy=0x7ffa840cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.752+0000 7ffa82ffd640 1 -- 172.21.15.46:0/2134407667 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa84159dc0 con 0x7ffa840c8cf0 2023-10-30T13:51:39.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa82ffd640 1 --2- 172.21.15.46:0/2134407667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa840c8cf0 0x7ffa840c9e00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009870 tx=0x7ffa7401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=978382b917eadb6f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:39.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa81ffb640 1 -- 172.21.15.46:0/2134407667 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa740ab620 con 0x7ffa840c8cf0 2023-10-30T13:51:39.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa81ffb640 1 -- 172.21.15.46:0/2134407667 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa740ab780 con 0x7ffa840c8cf0 2023-10-30T13:51:39.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa81ffb640 1 -- 172.21.15.46:0/2134407667 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa740abb10 con 0x7ffa840c8cf0 2023-10-30T13:51:39.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa89a46640 1 -- 172.21.15.46:0/2134407667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa840c8cf0 msgr2=0x7ffa840c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa89a46640 1 --2- 172.21.15.46:0/2134407667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa840c8cf0 0x7ffa840c9e00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009870 tx=0x7ffa7401c510 comp rx=0 tx=0).stop 2023-10-30T13:51:39.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa89a46640 1 -- 172.21.15.46:0/2134407667 shutdown_connections 2023-10-30T13:51:39.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa89a46640 1 --2- 172.21.15.46:0/2134407667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa840ca340 0x7ffa8415d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa89a46640 1 --2- 172.21.15.46:0/2134407667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa840c8cf0 0x7ffa840c9e00 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa89a46640 1 -- 172.21.15.46:0/2134407667 >> 172.21.15.46:0/2134407667 conn(0x7ffa8406be40 msgr2=0x7ffa8406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:39.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa89a46640 1 -- 172.21.15.46:0/2134407667 shutdown_connections 2023-10-30T13:51:39.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa89a46640 1 -- 172.21.15.46:0/2134407667 wait complete. 2023-10-30T13:51:39.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.756+0000 7ffa89a46640 1 Processor -- start 2023-10-30T13:51:39.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa89a46640 1 -- start start 2023-10-30T13:51:39.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa89a46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa840c8cf0 0x7ffa84151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa89a46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa840c90d0 0x7ffa840cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa89a46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa840ca340 0x7ffa841f5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa89a46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa840c4890 con 0x7ffa840ca340 2023-10-30T13:51:39.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa89a46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8415c790 con 0x7ffa840c8cf0 2023-10-30T13:51:39.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa89a46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8415c240 con 0x7ffa840c90d0 2023-10-30T13:51:39.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa840ca340 0x7ffa841f5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa840ca340 0x7ffa841f5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53938/0 (socket says 172.21.15.46:53938) 2023-10-30T13:51:39.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa837fe640 1 -- 172.21.15.46:0/855779991 learned_addr learned my addr 172.21.15.46:0/855779991 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:39.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa827fc640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa840c90d0 0x7ffa840cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa82ffd640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa840c8cf0 0x7ffa84151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa837fe640 1 -- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa840c90d0 msgr2=0x7ffa840cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa837fe640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa840c90d0 0x7ffa840cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa837fe640 1 -- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa840c8cf0 msgr2=0x7ffa84151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa837fe640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa840c8cf0 0x7ffa84151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa837fe640 1 -- 172.21.15.46:0/855779991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa740094d0 con 0x7ffa840ca340 2023-10-30T13:51:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa82ffd640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa840c8cf0 0x7ffa84151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa827fc640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa840c90d0 0x7ffa840cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa837fe640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa840ca340 0x7ffa841f5500 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009dd0 tx=0x7ffa7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa6bfff640 1 -- 172.21.15.46:0/855779991 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c014480 con 0x7ffa840ca340 2023-10-30T13:51:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa6bfff640 1 -- 172.21.15.46:0/855779991 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c00f040 con 0x7ffa840ca340 2023-10-30T13:51:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa89a46640 1 -- 172.21.15.46:0/855779991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8415c4b0 con 0x7ffa840ca340 2023-10-30T13:51:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa6bfff640 1 -- 172.21.15.46:0/855779991 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c014840 con 0x7ffa840ca340 2023-10-30T13:51:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa89a46640 1 -- 172.21.15.46:0/855779991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa84150d00 con 0x7ffa840ca340 2023-10-30T13:51:39.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.760+0000 7ffa89a46640 1 -- 172.21.15.46:0/855779991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa841f5220 con 0x7ffa840ca340 2023-10-30T13:51:39.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.764+0000 7ffa6bfff640 1 -- 172.21.15.46:0/855779991 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffa7c0149a0 con 0x7ffa840ca340 2023-10-30T13:51:39.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.764+0000 7ffa6bfff640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffa700664a0 0x7ffa70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:39.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.764+0000 7ffa6bfff640 1 -- 172.21.15.46:0/855779991 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7ffa7c085320 con 0x7ffa840ca340 2023-10-30T13:51:39.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.764+0000 7ffa82ffd640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffa700664a0 0x7ffa70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:39.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.764+0000 7ffa82ffd640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffa700664a0 0x7ffa70068960 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ffa740af000 tx=0x7ffa740a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:39.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.768+0000 7ffa6bfff640 1 -- 172.21.15.46:0/855779991 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffa841f5220 con 0x7ffa840ca340 2023-10-30T13:51:39.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.960+0000 7ffa89a46640 1 -- 172.21.15.46:0/855779991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa841f5220 con 0x7ffa840ca340 2023-10-30T13:51:39.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.960+0000 7ffa6bfff640 1 -- 172.21.15.46:0/855779991 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5732 (secure 0 0 0) 0x7ffa841f5220 con 0x7ffa840ca340 2023-10-30T13:51:39.967 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:39.967 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:39.216908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 125 2023-10-30T13:51:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.964+0000 7ffa89a46640 1 -- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffa700664a0 msgr2=0x7ffa70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.964+0000 7ffa89a46640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffa700664a0 0x7ffa70068960 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ffa740af000 tx=0x7ffa740a7000 comp rx=0 tx=0).stop 2023-10-30T13:51:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.964+0000 7ffa89a46640 1 -- 172.21.15.46:0/855779991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa840ca340 msgr2=0x7ffa841f5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.964+0000 7ffa89a46640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa840ca340 0x7ffa841f5500 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009dd0 tx=0x7ffa7c00c490 comp rx=0 tx=0).stop 2023-10-30T13:51:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.964+0000 7ffa89a46640 1 -- 172.21.15.46:0/855779991 shutdown_connections 2023-10-30T13:51:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.964+0000 7ffa89a46640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffa700664a0 0x7ffa70068960 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.964+0000 7ffa89a46640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa840ca340 0x7ffa841f5500 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.964+0000 7ffa89a46640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa840c90d0 0x7ffa840cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.964+0000 7ffa89a46640 1 --2- 172.21.15.46:0/855779991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa840c8cf0 0x7ffa84151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.964+0000 7ffa89a46640 1 -- 172.21.15.46:0/855779991 >> 172.21.15.46:0/855779991 conn(0x7ffa8406be40 msgr2=0x7ffa840b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.964+0000 7ffa89a46640 1 -- 172.21.15.46:0/855779991 shutdown_connections 2023-10-30T13:51:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:39.964+0000 7ffa89a46640 1 -- 172.21.15.46:0/855779991 wait complete. 2023-10-30T13:51:39.989 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 125, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 125, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6858": { "addr": "172.21.15.46:6837/2572113428", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2572113428, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2572113428, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6858, "incarnation": 117, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_7002": { "addr": "172.21.15.46:6833/2807435195", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2807435195, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2807435195, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7002, "incarnation": 125, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:51:39.216908+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7002, "mds_1": 6858 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:51:41.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.140+0000 7fb0aca68640 1 Processor -- start 2023-10-30T13:51:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.140+0000 7fb0aca68640 1 -- start start 2023-10-30T13:51:42.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0aca68640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb0a8153d70 0x7fb0a806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:42.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0aca68640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb0a80c8490 0x7fb0a80c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:42.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0aca68640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a806afa0 con 0x7fb0a806e470 2023-10-30T13:51:42.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0aca68640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a806d780 con 0x7fb0a8153d70 2023-10-30T13:51:42.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0aca68640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a8154150 con 0x7fb0a80c8490 2023-10-30T13:51:42.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0a659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb0a8153d70 0x7fb0a806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:42.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0a659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb0a8153d70 0x7fb0a806ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53952/0 (socket says 172.21.15.46:53952) 2023-10-30T13:51:42.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0a659c640 1 -- 172.21.15.46:0/1416116923 learned_addr learned my addr 172.21.15.46:0/1416116923 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:42.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0a6d9d640 1 --2- 172.21.15.46:0/1416116923 >> v2:172.21.15.154:3300/0 conn(0x7fb0a80c8490 0x7fb0a80c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:42.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0a659c640 1 -- 172.21.15.46:0/1416116923 >> v2:172.21.15.154:3300/0 conn(0x7fb0a80c8490 msgr2=0x7fb0a80c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:42.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0a659c640 1 --2- 172.21.15.46:0/1416116923 >> v2:172.21.15.154:3300/0 conn(0x7fb0a80c8490 0x7fb0a80c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:42.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0a659c640 1 -- 172.21.15.46:0/1416116923 >> v1:172.21.15.154:6789/0 conn(0x7fb0a806e470 legacy=0x7fb0a80cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:42.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.144+0000 7fb0a659c640 1 -- 172.21.15.46:0/1416116923 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a806e030 con 0x7fb0a8153d70 2023-10-30T13:51:42.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0a659c640 1 --2- 172.21.15.46:0/1416116923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0a8153d70 0x7fb0a806ca80 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80cddf0 tx=0x7fb09801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c984e6ad4281cf9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:42.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0a5d9b640 1 -- 172.21.15.46:0/1416116923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0980aa5f0 con 0x7fb0a8153d70 2023-10-30T13:51:42.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0a5d9b640 1 -- 172.21.15.46:0/1416116923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0980aa750 con 0x7fb0a8153d70 2023-10-30T13:51:42.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0a5d9b640 1 -- 172.21.15.46:0/1416116923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0980aaae0 con 0x7fb0a8153d70 2023-10-30T13:51:42.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0aca68640 1 -- 172.21.15.46:0/1416116923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0a8153d70 msgr2=0x7fb0a806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:42.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0aca68640 1 --2- 172.21.15.46:0/1416116923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0a8153d70 0x7fb0a806ca80 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80cddf0 tx=0x7fb09801c490 comp rx=0 tx=0).stop 2023-10-30T13:51:42.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0aca68640 1 -- 172.21.15.46:0/1416116923 shutdown_connections 2023-10-30T13:51:42.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0aca68640 1 --2- 172.21.15.46:0/1416116923 >> v2:172.21.15.154:3300/0 conn(0x7fb0a80c8490 0x7fb0a80c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:42.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0aca68640 1 --2- 172.21.15.46:0/1416116923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0a8153d70 0x7fb0a806ca80 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:42.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0aca68640 1 -- 172.21.15.46:0/1416116923 >> 172.21.15.46:0/1416116923 conn(0x7fb0a8071ea0 msgr2=0x7fb0a8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:42.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0aca68640 1 -- 172.21.15.46:0/1416116923 shutdown_connections 2023-10-30T13:51:42.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0aca68640 1 -- 172.21.15.46:0/1416116923 wait complete. 2023-10-30T13:51:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0aca68640 1 Processor -- start 2023-10-30T13:51:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.148+0000 7fb0aca68640 1 -- start start 2023-10-30T13:51:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb0aca68640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0a806e470 0x7fb0a8145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb0aca68640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0a80c8490 0x7fb0a80cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb0aca68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0a8153d70 0x7fb0a80cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb0aca68640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a8146940 con 0x7fb0a80c8490 2023-10-30T13:51:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb0aca68640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a8149b00 con 0x7fb0a8153d70 2023-10-30T13:51:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb0aca68640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a8154b30 con 0x7fb0a806e470 2023-10-30T13:51:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0a80c8490 0x7fb0a80cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0a80c8490 0x7fb0a80cac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53958/0 (socket says 172.21.15.46:53958) 2023-10-30T13:51:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09ffff640 1 -- 172.21.15.46:0/720743037 learned_addr learned my addr 172.21.15.46:0/720743037 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb0a659c640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0a806e470 0x7fb0a8145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb0a6d9d640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0a8153d70 0x7fb0a80cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09ffff640 1 -- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0a806e470 msgr2=0x7fb0a8145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09ffff640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0a806e470 0x7fb0a8145cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09ffff640 1 -- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0a8153d70 msgr2=0x7fb0a80cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09ffff640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0a8153d70 0x7fb0a80cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09ffff640 1 -- 172.21.15.46:0/720743037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0980094d0 con 0x7fb0a80c8490 2023-10-30T13:51:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09ffff640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0a80c8490 0x7fb0a80cac70 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb09000b410 tx=0x7fb09000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09effd640 1 -- 172.21.15.46:0/720743037 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090004020 con 0x7fb0a80c8490 2023-10-30T13:51:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09effd640 1 -- 172.21.15.46:0/720743037 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb090004180 con 0x7fb0a80c8490 2023-10-30T13:51:42.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb0aca68640 1 -- 172.21.15.46:0/720743037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a8149790 con 0x7fb0a80c8490 2023-10-30T13:51:42.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09effd640 1 -- 172.21.15.46:0/720743037 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090011630 con 0x7fb0a80c8490 2023-10-30T13:51:42.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb0aca68640 1 -- 172.21.15.46:0/720743037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a81453c0 con 0x7fb0a80c8490 2023-10-30T13:51:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb0aca68640 1 -- 172.21.15.46:0/720743037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a8149c40 con 0x7fb0a80c8490 2023-10-30T13:51:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.152+0000 7fb09effd640 1 -- 172.21.15.46:0/720743037 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb090011790 con 0x7fb0a80c8490 2023-10-30T13:51:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.156+0000 7fb09effd640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0880664a0 0x7fb088068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.156+0000 7fb09effd640 1 -- 172.21.15.46:0/720743037 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fb0900842b0 con 0x7fb0a80c8490 2023-10-30T13:51:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.156+0000 7fb0a659c640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0880664a0 0x7fb088068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.156+0000 7fb0a659c640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0880664a0 0x7fb088068960 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb0980ae000 tx=0x7fb0980a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.160+0000 7fb09effd640 1 -- 172.21.15.46:0/720743037 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb0a8149c40 con 0x7fb0a80c8490 2023-10-30T13:51:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb0aca68640 1 -- 172.21.15.46:0/720743037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0a8149c40 con 0x7fb0a80c8490 2023-10-30T13:51:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb09effd640 1 -- 172.21.15.46:0/720743037 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fb0a8149c40 con 0x7fb0a80c8490 2023-10-30T13:51:42.361 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:42.362 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:39.216908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 125 2023-10-30T13:51:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb0aca68640 1 -- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0880664a0 msgr2=0x7fb088068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb0aca68640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0880664a0 0x7fb088068960 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb0980ae000 tx=0x7fb0980a7000 comp rx=0 tx=0).stop 2023-10-30T13:51:42.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb0aca68640 1 -- 172.21.15.46:0/720743037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0a80c8490 msgr2=0x7fb0a80cac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:42.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb0aca68640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0a80c8490 0x7fb0a80cac70 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb09000b410 tx=0x7fb09000b8d0 comp rx=0 tx=0).stop 2023-10-30T13:51:42.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb0aca68640 1 -- 172.21.15.46:0/720743037 shutdown_connections 2023-10-30T13:51:42.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb0aca68640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0880664a0 0x7fb088068960 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:42.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb0aca68640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0a8153d70 0x7fb0a80cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:42.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb0aca68640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0a80c8490 0x7fb0a80cac70 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:42.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb0aca68640 1 --2- 172.21.15.46:0/720743037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0a806e470 0x7fb0a8145cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:42.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb0aca68640 1 -- 172.21.15.46:0/720743037 >> 172.21.15.46:0/720743037 conn(0x7fb0a8071ea0 msgr2=0x7fb0a8151ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:42.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.356+0000 7fb0aca68640 1 -- 172.21.15.46:0/720743037 shutdown_connections 2023-10-30T13:51:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:42.360+0000 7fb0aca68640 1 -- 172.21.15.46:0/720743037 wait complete. 2023-10-30T13:51:42.383 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:44.384 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:44.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c2ab65640 1 Processor -- start 2023-10-30T13:51:44.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c2ab65640 1 -- start start 2023-10-30T13:51:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c2ab65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c241538a0 0x7f9c2415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c2ab65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c240cf160 0x7f9c240cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c2ab65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c240cb800 con 0x7f9c240ca0e0 2023-10-30T13:51:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c2ab65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2415dc30 con 0x7f9c241538a0 2023-10-30T13:51:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c2ab65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c240cb330 con 0x7f9c240cf160 2023-10-30T13:51:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c28901640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c241538a0 0x7f9c2415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c28901640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c241538a0 0x7f9c2415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53964/0 (socket says 172.21.15.46:53964) 2023-10-30T13:51:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c28901640 1 -- 172.21.15.46:0/3288902213 learned_addr learned my addr 172.21.15.46:0/3288902213 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c29102640 1 --2- 172.21.15.46:0/3288902213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c240cf160 0x7f9c240cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c28901640 1 -- 172.21.15.46:0/3288902213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c240cf160 msgr2=0x7f9c240cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c28901640 1 --2- 172.21.15.46:0/3288902213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c240cf160 0x7f9c240cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c28901640 1 -- 172.21.15.46:0/3288902213 >> v1:172.21.15.154:6789/0 conn(0x7f9c240ca0e0 legacy=0x7f9c2415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.532+0000 7f9c28901640 1 -- 172.21.15.46:0/3288902213 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c240cad70 con 0x7f9c241538a0 2023-10-30T13:51:44.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.536+0000 7f9c28901640 1 --2- 172.21.15.46:0/3288902213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c241538a0 0x7f9c2415d6f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f9c18009870 tx=0x7f9c1801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca23418278ba58c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:44.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.536+0000 7f9c237fe640 1 -- 172.21.15.46:0/3288902213 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c180a89b0 con 0x7f9c241538a0 2023-10-30T13:51:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.536+0000 7f9c237fe640 1 -- 172.21.15.46:0/3288902213 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c180a8b10 con 0x7f9c241538a0 2023-10-30T13:51:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.536+0000 7f9c237fe640 1 -- 172.21.15.46:0/3288902213 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c180a8ea0 con 0x7f9c241538a0 2023-10-30T13:51:44.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.536+0000 7f9c2ab65640 1 -- 172.21.15.46:0/3288902213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c241538a0 msgr2=0x7f9c2415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:44.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.536+0000 7f9c2ab65640 1 --2- 172.21.15.46:0/3288902213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c241538a0 0x7f9c2415d6f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f9c18009870 tx=0x7f9c1801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:51:44.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.536+0000 7f9c2ab65640 1 -- 172.21.15.46:0/3288902213 shutdown_connections 2023-10-30T13:51:44.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.536+0000 7f9c2ab65640 1 --2- 172.21.15.46:0/3288902213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c240cf160 0x7f9c240cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:44.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.536+0000 7f9c2ab65640 1 --2- 172.21.15.46:0/3288902213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c241538a0 0x7f9c2415d6f0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:44.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.536+0000 7f9c2ab65640 1 -- 172.21.15.46:0/3288902213 >> 172.21.15.46:0/3288902213 conn(0x7f9c2406bfe0 msgr2=0x7f9c2406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:44.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.536+0000 7f9c2ab65640 1 -- 172.21.15.46:0/3288902213 shutdown_connections 2023-10-30T13:51:44.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.536+0000 7f9c2ab65640 1 -- 172.21.15.46:0/3288902213 wait complete. 2023-10-30T13:51:44.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.540+0000 7f9c2ab65640 1 Processor -- start 2023-10-30T13:51:44.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.540+0000 7f9c2ab65640 1 -- start start 2023-10-30T13:51:44.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c2ab65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c240ca0e0 0x7f9c2415d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:44.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c2ab65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c240cf160 0x7f9c2415a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:44.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c2ab65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c241538a0 0x7f9c2414cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:44.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c2ab65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2415bc50 con 0x7f9c240ca0e0 2023-10-30T13:51:44.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c2ab65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c241efb00 con 0x7f9c240cf160 2023-10-30T13:51:44.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c2ab65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c241ef240 con 0x7f9c241538a0 2023-10-30T13:51:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c28901640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c240ca0e0 0x7f9c2415d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c28901640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c240ca0e0 0x7f9c2415d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53972/0 (socket says 172.21.15.46:53972) 2023-10-30T13:51:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c28901640 1 -- 172.21.15.46:0/2655203016 learned_addr learned my addr 172.21.15.46:0/2655203016 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c29102640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c241538a0 0x7f9c2414cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:44.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c23fff640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c240cf160 0x7f9c2415a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:44.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c28901640 1 -- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c241538a0 msgr2=0x7f9c2414cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:44.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c28901640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c241538a0 0x7f9c2414cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:44.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c28901640 1 -- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c240cf160 msgr2=0x7f9c2415a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:44.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c28901640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c240cf160 0x7f9c2415a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:44.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c28901640 1 -- 172.21.15.46:0/2655203016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c180094d0 con 0x7f9c240ca0e0 2023-10-30T13:51:44.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c23fff640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c240cf160 0x7f9c2415a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:44.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c28901640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c240ca0e0 0x7f9c2415d380 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f9c1801c920 tx=0x7f9c1801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:44.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c217fa640 1 -- 172.21.15.46:0/2655203016 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c180b1df0 con 0x7f9c240ca0e0 2023-10-30T13:51:44.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c217fa640 1 -- 172.21.15.46:0/2655203016 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c180bb470 con 0x7f9c240ca0e0 2023-10-30T13:51:44.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c2ab65640 1 -- 172.21.15.46:0/2655203016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c241e4a90 con 0x7f9c240ca0e0 2023-10-30T13:51:44.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c217fa640 1 -- 172.21.15.46:0/2655203016 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c180bb7e0 con 0x7f9c240ca0e0 2023-10-30T13:51:44.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c2ab65640 1 -- 172.21.15.46:0/2655203016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c241ef790 con 0x7f9c240ca0e0 2023-10-30T13:51:44.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.544+0000 7f9c2ab65640 1 -- 172.21.15.46:0/2655203016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c240b2a50 con 0x7f9c240ca0e0 2023-10-30T13:51:44.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.548+0000 7f9c217fa640 1 -- 172.21.15.46:0/2655203016 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9c180b7070 con 0x7f9c240ca0e0 2023-10-30T13:51:44.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.548+0000 7f9c217fa640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9c080664a0 0x7f9c08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:44.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.548+0000 7f9c217fa640 1 -- 172.21.15.46:0/2655203016 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f9c181223d0 con 0x7f9c240ca0e0 2023-10-30T13:51:44.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.548+0000 7f9c23fff640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9c080664a0 0x7f9c08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:44.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.548+0000 7f9c23fff640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9c080664a0 0x7f9c08068960 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9c240cea60 tx=0x7f9c10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:44.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.552+0000 7f9c217fa640 1 -- 172.21.15.46:0/2655203016 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9c180ef310 con 0x7f9c240ca0e0 2023-10-30T13:51:44.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.792+0000 7f9c2ab65640 1 -- 172.21.15.46:0/2655203016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c240b2a50 con 0x7f9c240ca0e0 2023-10-30T13:51:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.792+0000 7f9c217fa640 1 -- 172.21.15.46:0/2655203016 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f9c180f7e60 con 0x7f9c240ca0e0 2023-10-30T13:51:44.798 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:44.798 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:39.216908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:44.799 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 125 2023-10-30T13:51:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.796+0000 7f9c2ab65640 1 -- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9c080664a0 msgr2=0x7f9c08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.796+0000 7f9c2ab65640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9c080664a0 0x7f9c08068960 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9c240cea60 tx=0x7f9c10009040 comp rx=0 tx=0).stop 2023-10-30T13:51:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.796+0000 7f9c2ab65640 1 -- 172.21.15.46:0/2655203016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c240ca0e0 msgr2=0x7f9c2415d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.796+0000 7f9c2ab65640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c240ca0e0 0x7f9c2415d380 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f9c1801c920 tx=0x7f9c1801ba90 comp rx=0 tx=0).stop 2023-10-30T13:51:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.796+0000 7f9c2ab65640 1 -- 172.21.15.46:0/2655203016 shutdown_connections 2023-10-30T13:51:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.796+0000 7f9c2ab65640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9c080664a0 0x7f9c08068960 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.796+0000 7f9c2ab65640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c241538a0 0x7f9c2414cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.796+0000 7f9c2ab65640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c240cf160 0x7f9c2415a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.796+0000 7f9c2ab65640 1 --2- 172.21.15.46:0/2655203016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c240ca0e0 0x7f9c2415d380 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.796+0000 7f9c2ab65640 1 -- 172.21.15.46:0/2655203016 >> 172.21.15.46:0/2655203016 conn(0x7f9c2406bfe0 msgr2=0x7f9c24151cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.796+0000 7f9c2ab65640 1 -- 172.21.15.46:0/2655203016 shutdown_connections 2023-10-30T13:51:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:44.796+0000 7f9c2ab65640 1 -- 172.21.15.46:0/2655203016 wait complete. 2023-10-30T13:51:44.828 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:46.829 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:46.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.972+0000 7f95b92ea640 1 Processor -- start 2023-10-30T13:51:46.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.972+0000 7f95b92ea640 1 -- start start 2023-10-30T13:51:46.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.972+0000 7f95b92ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95b41538a0 0x7f95b415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.972+0000 7f95b92ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95b40ca0e0 0x7f95b415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b92ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f95b40cf160 0x7f95b40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b92ea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95b40cb800 con 0x7f95b41538a0 2023-10-30T13:51:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b92ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95b415dc30 con 0x7f95b40cf160 2023-10-30T13:51:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b92ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b40cb330 con 0x7f95b40ca0e0 2023-10-30T13:51:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95b41538a0 0x7f95b415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95b41538a0 0x7f95b415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33488/0 (socket says 172.21.15.46:33488) 2023-10-30T13:51:46.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b2ffd640 1 -- 172.21.15.46:0/269551526 learned_addr learned my addr 172.21.15.46:0/269551526 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:46.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b37fe640 1 --2- 172.21.15.46:0/269551526 >> v2:172.21.15.154:3300/0 conn(0x7f95b40cf160 0x7f95b40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95abfff640 1 --2- 172.21.15.46:0/269551526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95b40ca0e0 0x7f95b415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b2ffd640 1 -- 172.21.15.46:0/269551526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95b40ca0e0 msgr2=0x7f95b415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:46.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b2ffd640 1 --2- 172.21.15.46:0/269551526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95b40ca0e0 0x7f95b415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b2ffd640 1 -- 172.21.15.46:0/269551526 >> v2:172.21.15.154:3300/0 conn(0x7f95b40cf160 msgr2=0x7f95b40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:46.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b2ffd640 1 --2- 172.21.15.46:0/269551526 >> v2:172.21.15.154:3300/0 conn(0x7f95b40cf160 0x7f95b40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.976+0000 7f95b2ffd640 1 -- 172.21.15.46:0/269551526 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b40cad70 con 0x7f95b41538a0 2023-10-30T13:51:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b2ffd640 1 --2- 172.21.15.46:0/269551526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95b41538a0 0x7f95b415d6f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f95b406cb00 tx=0x7f95a401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ccaac1ca8c0e05 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b27fc640 1 -- 172.21.15.46:0/269551526 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a40a89b0 con 0x7f95b41538a0 2023-10-30T13:51:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b27fc640 1 -- 172.21.15.46:0/269551526 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a40a8b10 con 0x7f95b41538a0 2023-10-30T13:51:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b27fc640 1 -- 172.21.15.46:0/269551526 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a40a8ea0 con 0x7f95b41538a0 2023-10-30T13:51:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b92ea640 1 -- 172.21.15.46:0/269551526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95b41538a0 msgr2=0x7f95b415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b92ea640 1 --2- 172.21.15.46:0/269551526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95b41538a0 0x7f95b415d6f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f95b406cb00 tx=0x7f95a401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b92ea640 1 -- 172.21.15.46:0/269551526 shutdown_connections 2023-10-30T13:51:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b92ea640 1 --2- 172.21.15.46:0/269551526 >> v2:172.21.15.154:3300/0 conn(0x7f95b40cf160 0x7f95b40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b92ea640 1 --2- 172.21.15.46:0/269551526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95b40ca0e0 0x7f95b415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b92ea640 1 --2- 172.21.15.46:0/269551526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95b41538a0 0x7f95b415d6f0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b92ea640 1 -- 172.21.15.46:0/269551526 >> 172.21.15.46:0/269551526 conn(0x7f95b406bfe0 msgr2=0x7f95b406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b92ea640 1 -- 172.21.15.46:0/269551526 shutdown_connections 2023-10-30T13:51:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b92ea640 1 -- 172.21.15.46:0/269551526 wait complete. 2023-10-30T13:51:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b92ea640 1 Processor -- start 2023-10-30T13:51:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b92ea640 1 -- start start 2023-10-30T13:51:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.980+0000 7f95b92ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95b40ca0e0 0x7f95b40b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95b92ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95b40cf160 0x7f95b414fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95b92ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95b41538a0 0x7f95b40b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95b92ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b41f36d0 con 0x7f95b40cf160 2023-10-30T13:51:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95b92ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b41efd60 con 0x7f95b40ca0e0 2023-10-30T13:51:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95b92ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b415cc90 con 0x7f95b41538a0 2023-10-30T13:51:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95abfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95b40cf160 0x7f95b414fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95abfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95b40cf160 0x7f95b414fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33494/0 (socket says 172.21.15.46:33494) 2023-10-30T13:51:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95abfff640 1 -- 172.21.15.46:0/252889701 learned_addr learned my addr 172.21.15.46:0/252889701 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95b2ffd640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95b40ca0e0 0x7f95b40b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95b37fe640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95b41538a0 0x7f95b40b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95abfff640 1 -- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95b41538a0 msgr2=0x7f95b40b35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95abfff640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95b41538a0 0x7f95b40b35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95abfff640 1 -- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95b40ca0e0 msgr2=0x7f95b40b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95abfff640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95b40ca0e0 0x7f95b40b4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95abfff640 1 -- 172.21.15.46:0/252889701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a40094d0 con 0x7f95b40cf160 2023-10-30T13:51:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95abfff640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95b40cf160 0x7f95b414fde0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f959c00d830 tx=0x7f959c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95ab7fe640 1 -- 172.21.15.46:0/252889701 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c009640 con 0x7f95b40cf160 2023-10-30T13:51:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95ab7fe640 1 -- 172.21.15.46:0/252889701 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959c0097a0 con 0x7f95b40cf160 2023-10-30T13:51:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95b92ea640 1 -- 172.21.15.46:0/252889701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b41f0180 con 0x7f95b40cf160 2023-10-30T13:51:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95ab7fe640 1 -- 172.21.15.46:0/252889701 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c009b10 con 0x7f95b40cf160 2023-10-30T13:51:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95b92ea640 1 -- 172.21.15.46:0/252889701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b41ef3c0 con 0x7f95b40cf160 2023-10-30T13:51:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95b92ea640 1 -- 172.21.15.46:0/252889701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b41effd0 con 0x7f95b40cf160 2023-10-30T13:51:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.984+0000 7f95ab7fe640 1 -- 172.21.15.46:0/252889701 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f959c009c70 con 0x7f95b40cf160 2023-10-30T13:51:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.988+0000 7f95ab7fe640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f95940664a0 0x7f9594068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.988+0000 7f95ab7fe640 1 -- 172.21.15.46:0/252889701 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f959c084a60 con 0x7f95b40cf160 2023-10-30T13:51:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.988+0000 7f95b2ffd640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f95940664a0 0x7f9594068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.988+0000 7f95b2ffd640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f95940664a0 0x7f9594068960 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f95a40b2000 tx=0x7f95a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:46.992+0000 7f95ab7fe640 1 -- 172.21.15.46:0/252889701 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f959c0519a0 con 0x7f95b40cf160 2023-10-30T13:51:47.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.180+0000 7f95b92ea640 1 -- 172.21.15.46:0/252889701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95b40b2a50 con 0x7f95b40cf160 2023-10-30T13:51:47.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95ab7fe640 1 -- 172.21.15.46:0/252889701 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f959c054fc0 con 0x7f95b40cf160 2023-10-30T13:51:47.188 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:47.188 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:39.216908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:47.188 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 125 2023-10-30T13:51:47.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95b92ea640 1 -- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f95940664a0 msgr2=0x7f9594068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:47.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95b92ea640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f95940664a0 0x7f9594068960 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f95a40b2000 tx=0x7f95a4009040 comp rx=0 tx=0).stop 2023-10-30T13:51:47.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95b92ea640 1 -- 172.21.15.46:0/252889701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95b40cf160 msgr2=0x7f95b414fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:47.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95b92ea640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95b40cf160 0x7f95b414fde0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f959c00d830 tx=0x7f959c00dcf0 comp rx=0 tx=0).stop 2023-10-30T13:51:47.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95b92ea640 1 -- 172.21.15.46:0/252889701 shutdown_connections 2023-10-30T13:51:47.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95b92ea640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f95940664a0 0x7f9594068960 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:47.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95b92ea640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95b41538a0 0x7f95b40b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:47.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95b92ea640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95b40cf160 0x7f95b414fde0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:47.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95b92ea640 1 --2- 172.21.15.46:0/252889701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95b40ca0e0 0x7f95b40b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:47.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95b92ea640 1 -- 172.21.15.46:0/252889701 >> 172.21.15.46:0/252889701 conn(0x7f95b406bfe0 msgr2=0x7f95b4151f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:47.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95b92ea640 1 -- 172.21.15.46:0/252889701 shutdown_connections 2023-10-30T13:51:47.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:47.184+0000 7f95b92ea640 1 -- 172.21.15.46:0/252889701 wait complete. 2023-10-30T13:51:47.210 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:49.211 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:49.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.360+0000 7f56d071d640 1 Processor -- start 2023-10-30T13:51:49.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.360+0000 7f56d071d640 1 -- start start 2023-10-30T13:51:49.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56d071d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56c81538a0 0x7f56c815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:49.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56d071d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f56c80ca0e0 0x7f56c815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:49.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56d071d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c80cb800 con 0x7f56c80cf160 2023-10-30T13:51:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56d071d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c815dc30 con 0x7f56c81538a0 2023-10-30T13:51:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56d071d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c80cb330 con 0x7f56c80ca0e0 2023-10-30T13:51:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56ce4b9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56c81538a0 0x7f56c815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56cecba640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f56c80cf160 0x7f56c80cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51432/0 (socket says 172.21.15.46:51432) 2023-10-30T13:51:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56cecba640 1 -- 172.21.15.46:0/2588435406 learned_addr learned my addr 172.21.15.46:0/2588435406 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:51:49.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56cdcb8640 1 --2- 172.21.15.46:0/2588435406 >> v2:172.21.15.154:3300/0 conn(0x7f56c80ca0e0 0x7f56c815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:49.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56ce4b9640 1 -- 172.21.15.46:0/2588435406 >> v2:172.21.15.154:3300/0 conn(0x7f56c80ca0e0 msgr2=0x7f56c815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:49.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56ce4b9640 1 --2- 172.21.15.46:0/2588435406 >> v2:172.21.15.154:3300/0 conn(0x7f56c80ca0e0 0x7f56c815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:49.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56ce4b9640 1 -- 172.21.15.46:0/2588435406 >> v1:172.21.15.46:6789/0 conn(0x7f56c80cf160 legacy=0x7f56c80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:49.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.364+0000 7f56ce4b9640 1 -- 172.21.15.46:0/2588435406 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c80cad70 con 0x7f56c81538a0 2023-10-30T13:51:49.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56ce4b9640 1 --2- 172.21.15.46:0/2588435406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c81538a0 0x7f56c815d6f0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f56c80b30a0 tx=0x7f56c401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a648dd98da77cf56 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:49.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56cd4b7640 1 -- 172.21.15.46:0/2588435406 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c40a8950 con 0x7f56c81538a0 2023-10-30T13:51:49.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56cd4b7640 1 -- 172.21.15.46:0/2588435406 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c40a8ab0 con 0x7f56c81538a0 2023-10-30T13:51:49.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56cd4b7640 1 -- 172.21.15.46:0/2588435406 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c40a8e40 con 0x7f56c81538a0 2023-10-30T13:51:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56d071d640 1 -- 172.21.15.46:0/2588435406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c81538a0 msgr2=0x7f56c815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56d071d640 1 --2- 172.21.15.46:0/2588435406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c81538a0 0x7f56c815d6f0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f56c80b30a0 tx=0x7f56c401ec50 comp rx=0 tx=0).stop 2023-10-30T13:51:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56d071d640 1 -- 172.21.15.46:0/2588435406 shutdown_connections 2023-10-30T13:51:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56d071d640 1 --2- 172.21.15.46:0/2588435406 >> v2:172.21.15.154:3300/0 conn(0x7f56c80ca0e0 0x7f56c815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56d071d640 1 --2- 172.21.15.46:0/2588435406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c81538a0 0x7f56c815d6f0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56d071d640 1 -- 172.21.15.46:0/2588435406 >> 172.21.15.46:0/2588435406 conn(0x7f56c806bfe0 msgr2=0x7f56c806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56d071d640 1 -- 172.21.15.46:0/2588435406 shutdown_connections 2023-10-30T13:51:49.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56d071d640 1 -- 172.21.15.46:0/2588435406 wait complete. 2023-10-30T13:51:49.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56d071d640 1 Processor -- start 2023-10-30T13:51:49.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.368+0000 7f56d071d640 1 -- start start 2023-10-30T13:51:49.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56d071d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c80ca0e0 0x7f56c8158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:49.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56d071d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c80cf160 0x7f56c80c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56d071d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56c81538a0 0x7f56c81ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56d071d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c81e4db0 con 0x7f56c80ca0e0 2023-10-30T13:51:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56d071d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c81ef520 con 0x7f56c80cf160 2023-10-30T13:51:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56d071d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c81f2960 con 0x7f56c81538a0 2023-10-30T13:51:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56ce4b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c80ca0e0 0x7f56c8158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56ce4b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c80ca0e0 0x7f56c8158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33508/0 (socket says 172.21.15.46:33508) 2023-10-30T13:51:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56ce4b9640 1 -- 172.21.15.46:0/758463592 learned_addr learned my addr 172.21.15.46:0/758463592 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56cecba640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56c81538a0 0x7f56c81ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56cdcb8640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c80cf160 0x7f56c80c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56ce4b9640 1 -- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56c81538a0 msgr2=0x7f56c81ef9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56ce4b9640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56c81538a0 0x7f56c81ef9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56ce4b9640 1 -- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c80cf160 msgr2=0x7f56c80c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:49.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56ce4b9640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c80cf160 0x7f56c80c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:49.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56ce4b9640 1 -- 172.21.15.46:0/758463592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c4009420 con 0x7f56c80ca0e0 2023-10-30T13:51:49.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56ce4b9640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c80ca0e0 0x7f56c8158440 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f56c40b3040 tx=0x7f56c401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:49.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56beffd640 1 -- 172.21.15.46:0/758463592 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c40b9630 con 0x7f56c80ca0e0 2023-10-30T13:51:49.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56beffd640 1 -- 172.21.15.46:0/758463592 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c40b97b0 con 0x7f56c80ca0e0 2023-10-30T13:51:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56beffd640 1 -- 172.21.15.46:0/758463592 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c40b9b60 con 0x7f56c80ca0e0 2023-10-30T13:51:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56d071d640 1 -- 172.21.15.46:0/758463592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c81f2c20 con 0x7f56c80ca0e0 2023-10-30T13:51:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56d071d640 1 -- 172.21.15.46:0/758463592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c81f31e0 con 0x7f56c80ca0e0 2023-10-30T13:51:49.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56bcff9640 1 -- 172.21.15.46:0/758463592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c80b2a50 con 0x7f56c80ca0e0 2023-10-30T13:51:49.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.372+0000 7f56beffd640 1 -- 172.21.15.46:0/758463592 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f56c40b7040 con 0x7f56c80ca0e0 2023-10-30T13:51:49.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.376+0000 7f56beffd640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56ac0664d0 0x7f56ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:49.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.376+0000 7f56beffd640 1 -- 172.21.15.46:0/758463592 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f56c41223d0 con 0x7f56c80ca0e0 2023-10-30T13:51:49.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.376+0000 7f56cdcb8640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56ac0664d0 0x7f56ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:49.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.376+0000 7f56cdcb8640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56ac0664d0 0x7f56ac068990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f56c80cf0d0 tx=0x7f56b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:49.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.380+0000 7f56beffd640 1 -- 172.21.15.46:0/758463592 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f56c40ef260 con 0x7f56c80ca0e0 2023-10-30T13:51:49.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 -- 172.21.15.46:0/758463592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56c8159850 con 0x7f56c80ca0e0 2023-10-30T13:51:49.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56beffd640 1 -- 172.21.15.46:0/758463592 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f56c40f2880 con 0x7f56c80ca0e0 2023-10-30T13:51:49.605 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:49.606 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:39.216908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:49.606 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 125 2023-10-30T13:51:49.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 -- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56ac0664d0 msgr2=0x7f56ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:49.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56ac0664d0 0x7f56ac068990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f56c80cf0d0 tx=0x7f56b8008040 comp rx=0 tx=0).stop 2023-10-30T13:51:49.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 -- 172.21.15.46:0/758463592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c80ca0e0 msgr2=0x7f56c8158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:49.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c80ca0e0 0x7f56c8158440 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f56c40b3040 tx=0x7f56c401b9f0 comp rx=0 tx=0).stop 2023-10-30T13:51:49.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 -- 172.21.15.46:0/758463592 shutdown_connections 2023-10-30T13:51:49.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56ac0664d0 0x7f56ac068990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:49.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56c81538a0 0x7f56c81ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:49.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c80cf160 0x7f56c80c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:49.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 --2- 172.21.15.46:0/758463592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c80ca0e0 0x7f56c8158440 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:49.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 -- 172.21.15.46:0/758463592 >> 172.21.15.46:0/758463592 conn(0x7f56c806bfe0 msgr2=0x7f56c8151f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:49.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 -- 172.21.15.46:0/758463592 shutdown_connections 2023-10-30T13:51:49.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:49.600+0000 7f56bcff9640 1 -- 172.21.15.46:0/758463592 wait complete. 2023-10-30T13:51:49.627 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:51.143 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:51.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:51.628 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:51.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f71185b8640 1 Processor -- start 2023-10-30T13:51:51.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f71185b8640 1 -- start start 2023-10-30T13:51:51.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f71185b8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71101538a0 0x7f711015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:51.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f71185b8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71100cf160 0x7f71100cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:51.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f71185b8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71100cb800 con 0x7f71100ca0e0 2023-10-30T13:51:51.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f71185b8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f711015dc30 con 0x7f71101538a0 2023-10-30T13:51:51.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f71185b8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71100cb330 con 0x7f71100cf160 2023-10-30T13:51:51.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f7116354640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71101538a0 0x7f711015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:51.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f7116354640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71101538a0 0x7f711015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33516/0 (socket says 172.21.15.46:33516) 2023-10-30T13:51:51.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f7116354640 1 -- 172.21.15.46:0/369592585 learned_addr learned my addr 172.21.15.46:0/369592585 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:51.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f7116b55640 1 --2- 172.21.15.46:0/369592585 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71100cf160 0x7f71100cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:51.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f7116354640 1 -- 172.21.15.46:0/369592585 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71100cf160 msgr2=0x7f71100cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:51.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f7116354640 1 --2- 172.21.15.46:0/369592585 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71100cf160 0x7f71100cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f7116354640 1 -- 172.21.15.46:0/369592585 >> v1:172.21.15.46:6789/0 conn(0x7f71100ca0e0 legacy=0x7f711015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:51.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f7116354640 1 -- 172.21.15.46:0/369592585 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71100cad70 con 0x7f71101538a0 2023-10-30T13:51:51.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.780+0000 7f7115352640 1 -- 172.21.15.46:0/369592585 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601611222 0 0) 0x7f71100cb800 con 0x7f71100ca0e0 2023-10-30T13:51:51.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.784+0000 7f7116354640 1 --2- 172.21.15.46:0/369592585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71101538a0 0x7f711015d6f0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f710c009870 tx=0x7f710c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=825505e72e9e0d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:51.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.784+0000 7f7115352640 1 -- 172.21.15.46:0/369592585 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c0a8a30 con 0x7f71101538a0 2023-10-30T13:51:51.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.784+0000 7f7115352640 1 -- 172.21.15.46:0/369592585 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710c0a8b90 con 0x7f71101538a0 2023-10-30T13:51:51.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.784+0000 7f7115352640 1 -- 172.21.15.46:0/369592585 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c0b0440 con 0x7f71101538a0 2023-10-30T13:51:51.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.784+0000 7f71185b8640 1 -- 172.21.15.46:0/369592585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71101538a0 msgr2=0x7f711015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:51.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.784+0000 7f71185b8640 1 --2- 172.21.15.46:0/369592585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71101538a0 0x7f711015d6f0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f710c009870 tx=0x7f710c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:51:51.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.784+0000 7f71185b8640 1 -- 172.21.15.46:0/369592585 shutdown_connections 2023-10-30T13:51:51.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.784+0000 7f71185b8640 1 --2- 172.21.15.46:0/369592585 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71100cf160 0x7f71100cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.784+0000 7f71185b8640 1 --2- 172.21.15.46:0/369592585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71101538a0 0x7f711015d6f0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.784+0000 7f71185b8640 1 -- 172.21.15.46:0/369592585 >> 172.21.15.46:0/369592585 conn(0x7f711006bfe0 msgr2=0x7f711006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:51.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.784+0000 7f71185b8640 1 -- 172.21.15.46:0/369592585 shutdown_connections 2023-10-30T13:51:51.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.784+0000 7f71185b8640 1 -- 172.21.15.46:0/369592585 wait complete. 2023-10-30T13:51:51.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f71185b8640 1 Processor -- start 2023-10-30T13:51:51.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f71185b8640 1 -- start start 2023-10-30T13:51:51.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f71185b8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71100ca0e0 0x7f711015aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:51.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f71185b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71100cf160 0x7f71100ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:51.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f71185b8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71101538a0 0x7f71101ef320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:51.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f71185b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71101f3160 con 0x7f71100cf160 2023-10-30T13:51:51.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f71185b8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711015b3e0 con 0x7f71101538a0 2023-10-30T13:51:51.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f71185b8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71101f2890 con 0x7f71100ca0e0 2023-10-30T13:51:51.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7115b53640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71100cf160 0x7f71100ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:51.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7115b53640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71100cf160 0x7f71100ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33530/0 (socket says 172.21.15.46:33530) 2023-10-30T13:51:51.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7115b53640 1 -- 172.21.15.46:0/1007969355 learned_addr learned my addr 172.21.15.46:0/1007969355 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:51.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7116354640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71100ca0e0 0x7f711015aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:51.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7116b55640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71101538a0 0x7f71101ef320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:51.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7115b53640 1 -- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71100ca0e0 msgr2=0x7f711015aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:51.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7115b53640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71100ca0e0 0x7f711015aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7115b53640 1 -- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71101538a0 msgr2=0x7f71101ef320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:51.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7115b53640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71101538a0 0x7f71101ef320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7115b53640 1 -- 172.21.15.46:0/1007969355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c0094d0 con 0x7f71100cf160 2023-10-30T13:51:51.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7116b55640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71101538a0 0x7f71101ef320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:51.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7115b53640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71100cf160 0x7f71100ce930 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f710000c4d0 tx=0x7f710000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:51.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7106ffd640 1 -- 172.21.15.46:0/1007969355 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7100005480 con 0x7f71100cf160 2023-10-30T13:51:51.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7106ffd640 1 -- 172.21.15.46:0/1007969355 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7100003600 con 0x7f71100cf160 2023-10-30T13:51:51.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f71185b8640 1 -- 172.21.15.46:0/1007969355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71100b51f0 con 0x7f71100cf160 2023-10-30T13:51:51.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f7106ffd640 1 -- 172.21.15.46:0/1007969355 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000039b0 con 0x7f71100cf160 2023-10-30T13:51:51.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.788+0000 7f71185b8640 1 -- 172.21.15.46:0/1007969355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71101f2aa0 con 0x7f71100cf160 2023-10-30T13:51:51.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.792+0000 7f71185b8640 1 -- 172.21.15.46:0/1007969355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71100b2a50 con 0x7f71100cf160 2023-10-30T13:51:51.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.792+0000 7f7106ffd640 1 -- 172.21.15.46:0/1007969355 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7100003b10 con 0x7f71100cf160 2023-10-30T13:51:51.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.792+0000 7f7106ffd640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f70f40664a0 0x7f70f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:51.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.792+0000 7f7106ffd640 1 -- 172.21.15.46:0/1007969355 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f71000839c0 con 0x7f71100cf160 2023-10-30T13:51:51.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.792+0000 7f7116354640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f70f40664a0 0x7f70f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:51.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.796+0000 7f7116354640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f70f40664a0 0x7f70f4068960 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f710c0b2000 tx=0x7f710c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:51.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.796+0000 7f7106ffd640 1 -- 172.21.15.46:0/1007969355 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f71000509c0 con 0x7f71100cf160 2023-10-30T13:51:51.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.988+0000 7f71185b8640 1 -- 172.21.15.46:0/1007969355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71100b37a0 con 0x7f71100cf160 2023-10-30T13:51:51.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.988+0000 7f7106ffd640 1 -- 172.21.15.46:0/1007969355 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5734 (secure 0 0 0) 0x7f71100b37a0 con 0x7f71100cf160 2023-10-30T13:51:51.995 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:51.996 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:50.764205+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:resolve","state_seq":20,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:51.996 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 126 2023-10-30T13:51:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.992+0000 7f71185b8640 1 -- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f70f40664a0 msgr2=0x7f70f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:51.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.992+0000 7f71185b8640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f70f40664a0 0x7f70f4068960 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f710c0b2000 tx=0x7f710c009040 comp rx=0 tx=0).stop 2023-10-30T13:51:51.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.992+0000 7f71185b8640 1 -- 172.21.15.46:0/1007969355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71100cf160 msgr2=0x7f71100ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:51.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.992+0000 7f71185b8640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71100cf160 0x7f71100ce930 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f710000c4d0 tx=0x7f710000c990 comp rx=0 tx=0).stop 2023-10-30T13:51:51.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.992+0000 7f71185b8640 1 -- 172.21.15.46:0/1007969355 shutdown_connections 2023-10-30T13:51:51.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.992+0000 7f71185b8640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f70f40664a0 0x7f70f4068960 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.992+0000 7f71185b8640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71101538a0 0x7f71101ef320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.992+0000 7f71185b8640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71100cf160 0x7f71100ce930 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.992+0000 7f71185b8640 1 --2- 172.21.15.46:0/1007969355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71100ca0e0 0x7f711015aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:51.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.992+0000 7f71185b8640 1 -- 172.21.15.46:0/1007969355 >> 172.21.15.46:0/1007969355 conn(0x7f711006bfe0 msgr2=0x7f711014db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:51.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.992+0000 7f71185b8640 1 -- 172.21.15.46:0/1007969355 shutdown_connections 2023-10-30T13:51:52.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:51.992+0000 7f71185b8640 1 -- 172.21.15.46:0/1007969355 wait complete. 2023-10-30T13:51:52.017 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 126, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 126, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6858": { "addr": "172.21.15.46:6837/2572113428", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2572113428, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2572113428, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6858, "incarnation": 117, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_7002": { "addr": "172.21.15.46:6833/2807435195", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2807435195, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2807435195, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7002, "incarnation": 125, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:resolve", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:51:50.764205+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7002, "mds_1": 6858 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:51:54.018 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:54.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f289166b640 1 Processor -- start 2023-10-30T13:51:54.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f289166b640 1 -- start start 2023-10-30T13:51:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f289166b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f288c15d6f0 0x7f288c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f289166b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f288c0cf160 0x7f288c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f289166b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288c0cb800 con 0x7f288c1538a0 2023-10-30T13:51:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f289166b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f288c0cb330 con 0x7f288c15d6f0 2023-10-30T13:51:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f289166b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f288c0cad70 con 0x7f288c0cf160 2023-10-30T13:51:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f2883fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f288c15d6f0 0x7f288c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f288affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f288c1538a0 0x7f288c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51442/0 (socket says 172.21.15.46:51442) 2023-10-30T13:51:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f2883fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f288c15d6f0 0x7f288c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33536/0 (socket says 172.21.15.46:33536) 2023-10-30T13:51:54.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f288affd640 1 -- 172.21.15.46:0/2855435887 learned_addr learned my addr 172.21.15.46:0/2855435887 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:51:54.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f288b7fe640 1 --2- 172.21.15.46:0/2855435887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f288c0cf160 0x7f288c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:54.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f2883fff640 1 -- 172.21.15.46:0/2855435887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f288c0cf160 msgr2=0x7f288c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:54.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f2883fff640 1 --2- 172.21.15.46:0/2855435887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f288c0cf160 0x7f288c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:54.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f2883fff640 1 -- 172.21.15.46:0/2855435887 >> v1:172.21.15.46:6789/0 conn(0x7f288c1538a0 legacy=0x7f288c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:54.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f2883fff640 1 -- 172.21.15.46:0/2855435887 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c0cb050 con 0x7f288c15d6f0 2023-10-30T13:51:54.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.168+0000 7f288a7fc640 1 -- 172.21.15.46:0/2855435887 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793393525 0 0) 0x7f288c0cb800 con 0x7f288c1538a0 2023-10-30T13:51:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.172+0000 7f2883fff640 1 --2- 172.21.15.46:0/2855435887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f288c15d6f0 0x7f288c15a090 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f28740097c0 tx=0x7f287401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898e112af5dc5012 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.172+0000 7f288a7fc640 1 -- 172.21.15.46:0/2855435887 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28740a8970 con 0x7f288c15d6f0 2023-10-30T13:51:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.172+0000 7f288a7fc640 1 -- 172.21.15.46:0/2855435887 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28740a8ad0 con 0x7f288c15d6f0 2023-10-30T13:51:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.172+0000 7f288a7fc640 1 -- 172.21.15.46:0/2855435887 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28740a8e60 con 0x7f288c15d6f0 2023-10-30T13:51:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.172+0000 7f289166b640 1 -- 172.21.15.46:0/2855435887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f288c15d6f0 msgr2=0x7f288c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.172+0000 7f289166b640 1 --2- 172.21.15.46:0/2855435887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f288c15d6f0 0x7f288c15a090 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f28740097c0 tx=0x7f287401ec70 comp rx=0 tx=0).stop 2023-10-30T13:51:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.172+0000 7f289166b640 1 -- 172.21.15.46:0/2855435887 shutdown_connections 2023-10-30T13:51:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.172+0000 7f289166b640 1 --2- 172.21.15.46:0/2855435887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f288c0cf160 0x7f288c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.172+0000 7f289166b640 1 --2- 172.21.15.46:0/2855435887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f288c15d6f0 0x7f288c15a090 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.172+0000 7f289166b640 1 -- 172.21.15.46:0/2855435887 >> 172.21.15.46:0/2855435887 conn(0x7f288c06bfe0 msgr2=0x7f288c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:54.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.172+0000 7f289166b640 1 -- 172.21.15.46:0/2855435887 shutdown_connections 2023-10-30T13:51:54.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.172+0000 7f289166b640 1 -- 172.21.15.46:0/2855435887 wait complete. 2023-10-30T13:51:54.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f289166b640 1 Processor -- start 2023-10-30T13:51:54.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f289166b640 1 -- start start 2023-10-30T13:51:54.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f289166b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f288c0cf160 0x7f288c1577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:54.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f289166b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f288c1538a0 0x7f288c1ef950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:54.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f289166b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f288c15d6f0 0x7f288c14d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:54.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f289166b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288c1596e0 con 0x7f288c0cf160 2023-10-30T13:51:54.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f289166b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288c14f560 con 0x7f288c15d6f0 2023-10-30T13:51:54.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f289166b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f288c1efe90 con 0x7f288c1538a0 2023-10-30T13:51:54.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f288affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f288c0cf160 0x7f288c1577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f288affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f288c0cf160 0x7f288c1577c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33550/0 (socket says 172.21.15.46:33550) 2023-10-30T13:51:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f288affd640 1 -- 172.21.15.46:0/198077347 learned_addr learned my addr 172.21.15.46:0/198077347 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f2883fff640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f288c1538a0 0x7f288c1ef950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f288b7fe640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f288c15d6f0 0x7f288c14d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f288affd640 1 -- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f288c1538a0 msgr2=0x7f288c1ef950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f288affd640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f288c1538a0 0x7f288c1ef950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f288affd640 1 -- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f288c15d6f0 msgr2=0x7f288c14d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f288affd640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f288c15d6f0 0x7f288c14d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f288affd640 1 -- 172.21.15.46:0/198077347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2874009420 con 0x7f288c0cf160 2023-10-30T13:51:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f2883fff640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f288c1538a0 0x7f288c1ef950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f288affd640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f288c0cf160 0x7f288c1577c0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f287c0011f0 tx=0x7f287c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f28837fe640 1 -- 172.21.15.46:0/198077347 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c0074e0 con 0x7f288c0cf160 2023-10-30T13:51:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f28837fe640 1 -- 172.21.15.46:0/198077347 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287c007640 con 0x7f288c0cf160 2023-10-30T13:51:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f28837fe640 1 -- 172.21.15.46:0/198077347 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c004a10 con 0x7f288c0cf160 2023-10-30T13:51:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f289166b640 1 -- 172.21.15.46:0/198077347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288c14fcf0 con 0x7f288c0cf160 2023-10-30T13:51:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.176+0000 7f289166b640 1 -- 172.21.15.46:0/198077347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288c1f2bc0 con 0x7f288c0cf160 2023-10-30T13:51:54.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.180+0000 7f289166b640 1 -- 172.21.15.46:0/198077347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288c0b2a50 con 0x7f288c0cf160 2023-10-30T13:51:54.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.180+0000 7f28837fe640 1 -- 172.21.15.46:0/198077347 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f287c004b70 con 0x7f288c0cf160 2023-10-30T13:51:54.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.180+0000 7f28837fe640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f286c0664a0 0x7f286c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:54.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.180+0000 7f28837fe640 1 -- 172.21.15.46:0/198077347 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f287c083660 con 0x7f288c0cf160 2023-10-30T13:51:54.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.180+0000 7f2883fff640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f286c0664a0 0x7f286c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:54.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.180+0000 7f2883fff640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f286c0664a0 0x7f286c068960 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f28740b2000 tx=0x7f2874008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:54.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.184+0000 7f28837fe640 1 -- 172.21.15.46:0/198077347 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f287c050600 con 0x7f288c0cf160 2023-10-30T13:51:54.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.372+0000 7f289166b640 1 -- 172.21.15.46:0/198077347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f288c04e170 con 0x7f288c0cf160 2023-10-30T13:51:54.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.372+0000 7f28837fe640 1 -- 172.21.15.46:0/198077347 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5734 (secure 0 0 0) 0x7f287c018090 con 0x7f288c0cf160 2023-10-30T13:51:54.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:54.380 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:50.764205+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:resolve","state_seq":20,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:54.380 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 126 2023-10-30T13:51:54.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.376+0000 7f289166b640 1 -- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f286c0664a0 msgr2=0x7f286c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:54.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.376+0000 7f289166b640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f286c0664a0 0x7f286c068960 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f28740b2000 tx=0x7f2874008040 comp rx=0 tx=0).stop 2023-10-30T13:51:54.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.376+0000 7f289166b640 1 -- 172.21.15.46:0/198077347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f288c0cf160 msgr2=0x7f288c1577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:54.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.376+0000 7f289166b640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f288c0cf160 0x7f288c1577c0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f287c0011f0 tx=0x7f287c00c9b0 comp rx=0 tx=0).stop 2023-10-30T13:51:54.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.376+0000 7f289166b640 1 -- 172.21.15.46:0/198077347 shutdown_connections 2023-10-30T13:51:54.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.376+0000 7f289166b640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f286c0664a0 0x7f286c068960 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:54.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.376+0000 7f289166b640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f288c15d6f0 0x7f288c14d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.376+0000 7f289166b640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f288c1538a0 0x7f288c1ef950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.376+0000 7f289166b640 1 --2- 172.21.15.46:0/198077347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f288c0cf160 0x7f288c1577c0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.376+0000 7f289166b640 1 -- 172.21.15.46:0/198077347 >> 172.21.15.46:0/198077347 conn(0x7f288c06bfe0 msgr2=0x7f288c150ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.376+0000 7f289166b640 1 -- 172.21.15.46:0/198077347 shutdown_connections 2023-10-30T13:51:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:54.376+0000 7f289166b640 1 -- 172.21.15.46:0/198077347 wait complete. 2023-10-30T13:51:54.400 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:56.401 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:56.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29ff046640 1 Processor -- start 2023-10-30T13:51:56.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29ff046640 1 -- start start 2023-10-30T13:51:56.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29ff046640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29f81538a0 0x7f29f815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29ff046640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f29f80ca0e0 0x7f29f815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29ff046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 0x7f29f80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29ff046640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f80cb800 con 0x7f29f81538a0 2023-10-30T13:51:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29ff046640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f815dc30 con 0x7f29f80ca0e0 2023-10-30T13:51:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29ff046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f80cb330 con 0x7f29f80cf160 2023-10-30T13:51:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29fcde2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29f81538a0 0x7f29f815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29fcde2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29f81538a0 0x7f29f815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44194/0 (socket says 172.21.15.46:44194) 2023-10-30T13:51:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29fcde2640 1 -- 172.21.15.46:0/3080382593 learned_addr learned my addr 172.21.15.46:0/3080382593 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29f7fff640 1 --2- 172.21.15.46:0/3080382593 >> v2:172.21.15.154:3300/0 conn(0x7f29f80ca0e0 0x7f29f815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29fd5e3640 1 --2- 172.21.15.46:0/3080382593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 0x7f29f80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29fcde2640 1 -- 172.21.15.46:0/3080382593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 msgr2=0x7f29f80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29fcde2640 1 --2- 172.21.15.46:0/3080382593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 0x7f29f80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29fcde2640 1 -- 172.21.15.46:0/3080382593 >> v2:172.21.15.154:3300/0 conn(0x7f29f80ca0e0 msgr2=0x7f29f815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29fcde2640 1 --2- 172.21.15.46:0/3080382593 >> v2:172.21.15.154:3300/0 conn(0x7f29f80ca0e0 0x7f29f815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.544+0000 7f29fcde2640 1 -- 172.21.15.46:0/3080382593 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f80cad70 con 0x7f29f81538a0 2023-10-30T13:51:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29fcde2640 1 --2- 172.21.15.46:0/3080382593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 0x7f29f815d6f0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f29f80b30a0 tx=0x7f29ec01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91bc09955944a6bf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29f77fe640 1 -- 172.21.15.46:0/3080382593 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0a89b0 con 0x7f29f81538a0 2023-10-30T13:51:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29f77fe640 1 -- 172.21.15.46:0/3080382593 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec0a8b10 con 0x7f29f81538a0 2023-10-30T13:51:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29f77fe640 1 -- 172.21.15.46:0/3080382593 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0a8ea0 con 0x7f29f81538a0 2023-10-30T13:51:56.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29ff046640 1 -- 172.21.15.46:0/3080382593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 msgr2=0x7f29f815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29ff046640 1 --2- 172.21.15.46:0/3080382593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 0x7f29f815d6f0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f29f80b30a0 tx=0x7f29ec01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:51:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29ff046640 1 -- 172.21.15.46:0/3080382593 shutdown_connections 2023-10-30T13:51:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29ff046640 1 --2- 172.21.15.46:0/3080382593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 0x7f29f80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29ff046640 1 --2- 172.21.15.46:0/3080382593 >> v2:172.21.15.154:3300/0 conn(0x7f29f80ca0e0 0x7f29f815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29ff046640 1 --2- 172.21.15.46:0/3080382593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 0x7f29f815d6f0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29ff046640 1 -- 172.21.15.46:0/3080382593 >> 172.21.15.46:0/3080382593 conn(0x7f29f806bfe0 msgr2=0x7f29f806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29ff046640 1 -- 172.21.15.46:0/3080382593 shutdown_connections 2023-10-30T13:51:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.548+0000 7f29ff046640 1 -- 172.21.15.46:0/3080382593 wait complete. 2023-10-30T13:51:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29ff046640 1 Processor -- start 2023-10-30T13:51:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29ff046640 1 -- start start 2023-10-30T13:51:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29ff046640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f80ca0e0 0x7f29f815b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29ff046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 0x7f29f81f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29ff046640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f81538a0 0x7f29f80ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29ff046640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f815aec0 con 0x7f29f80ca0e0 2023-10-30T13:51:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29ff046640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f8150dc0 con 0x7f29f81538a0 2023-10-30T13:51:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29ff046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f80b46b0 con 0x7f29f80cf160 2023-10-30T13:51:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29fcde2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f80ca0e0 0x7f29f815b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29fcde2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f80ca0e0 0x7f29f815b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44202/0 (socket says 172.21.15.46:44202) 2023-10-30T13:51:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29fcde2640 1 -- 172.21.15.46:0/812821812 learned_addr learned my addr 172.21.15.46:0/812821812 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29f7fff640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 0x7f29f81f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29fd5e3640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f81538a0 0x7f29f80ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29fcde2640 1 -- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 msgr2=0x7f29f81f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29fcde2640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 0x7f29f81f2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29fcde2640 1 -- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f81538a0 msgr2=0x7f29f80ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29fcde2640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f81538a0 0x7f29f80ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29fcde2640 1 -- 172.21.15.46:0/812821812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec0094d0 con 0x7f29f80ca0e0 2023-10-30T13:51:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29fcde2640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f80ca0e0 0x7f29f815b350 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0b3040 tx=0x7f29ec01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29f57fa640 1 -- 172.21.15.46:0/812821812 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0ba600 con 0x7f29f80ca0e0 2023-10-30T13:51:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29f57fa640 1 -- 172.21.15.46:0/812821812 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec0ba760 con 0x7f29f80ca0e0 2023-10-30T13:51:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29ff046640 1 -- 172.21.15.46:0/812821812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f815a8b0 con 0x7f29f80ca0e0 2023-10-30T13:51:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29f57fa640 1 -- 172.21.15.46:0/812821812 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0bab10 con 0x7f29f80ca0e0 2023-10-30T13:51:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.552+0000 7f29ff046640 1 -- 172.21.15.46:0/812821812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f80b4c30 con 0x7f29f80ca0e0 2023-10-30T13:51:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.556+0000 7f29ff046640 1 -- 172.21.15.46:0/812821812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f80b2a50 con 0x7f29f80ca0e0 2023-10-30T13:51:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.556+0000 7f29f57fa640 1 -- 172.21.15.46:0/812821812 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29ec0bac70 con 0x7f29f80ca0e0 2023-10-30T13:51:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.556+0000 7f29f57fa640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29dc0664a0 0x7f29dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.556+0000 7f29f57fa640 1 -- 172.21.15.46:0/812821812 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f29ec122ad0 con 0x7f29f80ca0e0 2023-10-30T13:51:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.556+0000 7f29f7fff640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29dc0664a0 0x7f29dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.556+0000 7f29f7fff640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29dc0664a0 0x7f29dc068960 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f29f8146180 tx=0x7f29e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.560+0000 7f29f57fa640 1 -- 172.21.15.46:0/812821812 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29ec0ef9f0 con 0x7f29f80ca0e0 2023-10-30T13:51:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.748+0000 7f29ff046640 1 -- 172.21.15.46:0/812821812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29f804e170 con 0x7f29f80ca0e0 2023-10-30T13:51:56.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.748+0000 7f29f57fa640 1 -- 172.21.15.46:0/812821812 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5734 (secure 0 0 0) 0x7f29ec0b5020 con 0x7f29f80ca0e0 2023-10-30T13:51:56.754 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:56.755 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:50.764205+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:resolve","state_seq":20,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:56.755 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 126 2023-10-30T13:51:56.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.752+0000 7f29ff046640 1 -- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29dc0664a0 msgr2=0x7f29dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:56.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.752+0000 7f29ff046640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29dc0664a0 0x7f29dc068960 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f29f8146180 tx=0x7f29e4008040 comp rx=0 tx=0).stop 2023-10-30T13:51:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.752+0000 7f29ff046640 1 -- 172.21.15.46:0/812821812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f80ca0e0 msgr2=0x7f29f815b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.752+0000 7f29ff046640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f80ca0e0 0x7f29f815b350 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0b3040 tx=0x7f29ec01ba90 comp rx=0 tx=0).stop 2023-10-30T13:51:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.752+0000 7f29ff046640 1 -- 172.21.15.46:0/812821812 shutdown_connections 2023-10-30T13:51:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.752+0000 7f29ff046640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29dc0664a0 0x7f29dc068960 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:56.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.752+0000 7f29ff046640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f81538a0 0x7f29f80ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:56.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.752+0000 7f29ff046640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 0x7f29f81f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:56.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.752+0000 7f29ff046640 1 --2- 172.21.15.46:0/812821812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f80ca0e0 0x7f29f815b350 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:56.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.752+0000 7f29ff046640 1 -- 172.21.15.46:0/812821812 >> 172.21.15.46:0/812821812 conn(0x7f29f806bfe0 msgr2=0x7f29f81524c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:56.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.752+0000 7f29ff046640 1 -- 172.21.15.46:0/812821812 shutdown_connections 2023-10-30T13:51:56.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:56.752+0000 7f29ff046640 1 -- 172.21.15.46:0/812821812 wait complete. 2023-10-30T13:51:56.775 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:58.777 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:58.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.924+0000 7f6b3c491640 1 Processor -- start 2023-10-30T13:51:58.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.924+0000 7f6b3c491640 1 -- start start 2023-10-30T13:51:58.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b3c491640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b340b2300 0x7f6b34071b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:58.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b3c491640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6b3406f370 0x7f6b34072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:58.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b3c491640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3406b3b0 con 0x7f6b341570f0 2023-10-30T13:51:58.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b3c491640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b340c38a0 con 0x7f6b340b2300 2023-10-30T13:51:58.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b3c491640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b340c3b20 con 0x7f6b3406f370 2023-10-30T13:51:58.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b3a22d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6b341570f0 0x7f6b3406ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51402/0 (socket says 172.21.15.46:51402) 2023-10-30T13:51:58.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b3a22d640 1 -- 172.21.15.46:0/3880300843 learned_addr learned my addr 172.21.15.46:0/3880300843 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:51:58.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b39a2c640 1 --2- 172.21.15.46:0/3880300843 >> v2:172.21.15.46:3300/0 conn(0x7f6b340b2300 0x7f6b34071b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:58.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b3aa2e640 1 --2- 172.21.15.46:0/3880300843 >> v2:172.21.15.154:3300/0 conn(0x7f6b3406f370 0x7f6b34072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:58.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b3922b640 1 -- 172.21.15.46:0/3880300843 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019719071 0 0) 0x7f6b3406b3b0 con 0x7f6b341570f0 2023-10-30T13:51:58.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b3922b640 1 -- 172.21.15.46:0/3880300843 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b20003530 con 0x7f6b341570f0 2023-10-30T13:51:58.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b39a2c640 1 -- 172.21.15.46:0/3880300843 >> v2:172.21.15.154:3300/0 conn(0x7f6b3406f370 msgr2=0x7f6b34072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:58.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b39a2c640 1 --2- 172.21.15.46:0/3880300843 >> v2:172.21.15.154:3300/0 conn(0x7f6b3406f370 0x7f6b34072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:58.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b39a2c640 1 -- 172.21.15.46:0/3880300843 >> v1:172.21.15.46:6789/0 conn(0x7f6b341570f0 legacy=0x7f6b3406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:58.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.928+0000 7f6b39a2c640 1 -- 172.21.15.46:0/3880300843 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b340ca9f0 con 0x7f6b340b2300 2023-10-30T13:51:58.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b39a2c640 1 --2- 172.21.15.46:0/3880300843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b340b2300 0x7f6b34071b30 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f6b340c4110 tx=0x7f6b2401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf868caa56e34e25 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:58.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3922b640 1 -- 172.21.15.46:0/3880300843 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24004230 con 0x7f6b340b2300 2023-10-30T13:51:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3922b640 1 -- 172.21.15.46:0/3880300843 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b240a7050 con 0x7f6b340b2300 2023-10-30T13:51:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3922b640 1 -- 172.21.15.46:0/3880300843 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24003620 con 0x7f6b340b2300 2023-10-30T13:51:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3c491640 1 -- 172.21.15.46:0/3880300843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b340b2300 msgr2=0x7f6b34071b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3c491640 1 --2- 172.21.15.46:0/3880300843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b340b2300 0x7f6b34071b30 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f6b340c4110 tx=0x7f6b2401c460 comp rx=0 tx=0).stop 2023-10-30T13:51:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3c491640 1 -- 172.21.15.46:0/3880300843 shutdown_connections 2023-10-30T13:51:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3c491640 1 --2- 172.21.15.46:0/3880300843 >> v2:172.21.15.154:3300/0 conn(0x7f6b3406f370 0x7f6b34072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3c491640 1 --2- 172.21.15.46:0/3880300843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b340b2300 0x7f6b34071b30 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3c491640 1 -- 172.21.15.46:0/3880300843 >> 172.21.15.46:0/3880300843 conn(0x7f6b34069e90 msgr2=0x7f6b3406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3c491640 1 -- 172.21.15.46:0/3880300843 shutdown_connections 2023-10-30T13:51:58.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3c491640 1 -- 172.21.15.46:0/3880300843 wait complete. 2023-10-30T13:51:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3c491640 1 Processor -- start 2023-10-30T13:51:58.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.932+0000 7f6b3c491640 1 -- start start 2023-10-30T13:51:58.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b3c491640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b3406f370 0x7f6b3414d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:58.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b3c491640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b340b2300 0x7f6b3414c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:58.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b3c491640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b341570f0 0x7f6b34141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:58.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b3c491640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b34141ac0 con 0x7f6b340b2300 2023-10-30T13:51:58.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b3c491640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b341439a0 con 0x7f6b341570f0 2023-10-30T13:51:58.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b3c491640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b341f2c70 con 0x7f6b3406f370 2023-10-30T13:51:58.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b39a2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b340b2300 0x7f6b3414c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:58.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b39a2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b340b2300 0x7f6b3414c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44218/0 (socket says 172.21.15.46:44218) 2023-10-30T13:51:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b39a2c640 1 -- 172.21.15.46:0/3481951843 learned_addr learned my addr 172.21.15.46:0/3481951843 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:51:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b3aa2e640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b341570f0 0x7f6b34141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b3a22d640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b3406f370 0x7f6b3414d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b39a2c640 1 -- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b3406f370 msgr2=0x7f6b3414d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b39a2c640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b3406f370 0x7f6b3414d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b39a2c640 1 -- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b341570f0 msgr2=0x7f6b34141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b39a2c640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b341570f0 0x7f6b34141ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b39a2c640 1 -- 172.21.15.46:0/3481951843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b24009420 con 0x7f6b340b2300 2023-10-30T13:51:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b39a2c640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b340b2300 0x7f6b3414c590 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f6b240ae040 tx=0x7f6b24019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b2affd640 1 -- 172.21.15.46:0/3481951843 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24003af0 con 0x7f6b340b2300 2023-10-30T13:51:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b2affd640 1 -- 172.21.15.46:0/3481951843 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b24003c50 con 0x7f6b340b2300 2023-10-30T13:51:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b2affd640 1 -- 172.21.15.46:0/3481951843 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b240ab440 con 0x7f6b340b2300 2023-10-30T13:51:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b3c491640 1 -- 172.21.15.46:0/3481951843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b341f20f0 con 0x7f6b340b2300 2023-10-30T13:51:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b3c491640 1 -- 172.21.15.46:0/3481951843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b341f26b0 con 0x7f6b340b2300 2023-10-30T13:51:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b3c491640 1 -- 172.21.15.46:0/3481951843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b34141c00 con 0x7f6b340b2300 2023-10-30T13:51:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.936+0000 7f6b2affd640 1 -- 172.21.15.46:0/3481951843 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b240b2040 con 0x7f6b340b2300 2023-10-30T13:51:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.940+0000 7f6b2affd640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b180664d0 0x7f6b18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.940+0000 7f6b2affd640 1 -- 172.21.15.46:0/3481951843 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f6b2411e2d0 con 0x7f6b340b2300 2023-10-30T13:51:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.940+0000 7f6b3a22d640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b180664d0 0x7f6b18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.940+0000 7f6b3a22d640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b180664d0 0x7f6b18068990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6b300011f0 tx=0x7f6b30009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:58.944+0000 7f6b2affd640 1 -- 172.21.15.46:0/3481951843 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6b34141c00 con 0x7f6b340b2300 2023-10-30T13:51:59.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 -- 172.21.15.46:0/3481951843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b34141c00 con 0x7f6b340b2300 2023-10-30T13:51:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b2affd640 1 -- 172.21.15.46:0/3481951843 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f6b34141c00 con 0x7f6b340b2300 2023-10-30T13:51:59.141 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:51:59.142 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:56.420713+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 127 2023-10-30T13:51:59.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 -- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b180664d0 msgr2=0x7f6b18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:59.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b180664d0 0x7f6b18068990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6b300011f0 tx=0x7f6b30009040 comp rx=0 tx=0).stop 2023-10-30T13:51:59.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 -- 172.21.15.46:0/3481951843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b340b2300 msgr2=0x7f6b3414c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:59.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b340b2300 0x7f6b3414c590 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f6b240ae040 tx=0x7f6b24019200 comp rx=0 tx=0).stop 2023-10-30T13:51:59.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 -- 172.21.15.46:0/3481951843 shutdown_connections 2023-10-30T13:51:59.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b180664d0 0x7f6b18068990 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:59.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b341570f0 0x7f6b34141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:59.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b340b2300 0x7f6b3414c590 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:59.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 --2- 172.21.15.46:0/3481951843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b3406f370 0x7f6b3414d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:59.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 -- 172.21.15.46:0/3481951843 >> 172.21.15.46:0/3481951843 conn(0x7f6b34069e90 msgr2=0x7f6b340c1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:59.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 -- 172.21.15.46:0/3481951843 shutdown_connections 2023-10-30T13:51:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:51:59.136+0000 7f6b3c491640 1 -- 172.21.15.46:0/3481951843 wait complete. 2023-10-30T13:51:59.163 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:52:01.164 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce8ed6640 1 Processor -- start 2023-10-30T13:52:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce8ed6640 1 -- start start 2023-10-30T13:52:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce8ed6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6ce41538a0 0x7f6ce415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce8ed6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ce40ca0e0 0x7f6ce415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce8ed6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce40cf160 0x7f6ce40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce8ed6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce40cb800 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce8ed6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce415dc30 con 0x7f6ce41538a0 2023-10-30T13:52:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce8ed6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce40cb330 con 0x7f6ce40cf160 2023-10-30T13:52:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce1d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ce40ca0e0 0x7f6ce415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce1d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ce40ca0e0 0x7f6ce415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44234/0 (socket says 172.21.15.46:44234) 2023-10-30T13:52:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce1d9b640 1 -- 172.21.15.46:0/1430333604 learned_addr learned my addr 172.21.15.46:0/1430333604 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce2d9d640 1 --2- 172.21.15.46:0/1430333604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce40cf160 0x7f6ce40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce259c640 1 --2- 172.21.15.46:0/1430333604 >> v2:172.21.15.154:3300/0 conn(0x7f6ce41538a0 0x7f6ce415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce1d9b640 1 -- 172.21.15.46:0/1430333604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce40cf160 msgr2=0x7f6ce40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce1d9b640 1 --2- 172.21.15.46:0/1430333604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce40cf160 0x7f6ce40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce1d9b640 1 -- 172.21.15.46:0/1430333604 >> v2:172.21.15.154:3300/0 conn(0x7f6ce41538a0 msgr2=0x7f6ce415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce1d9b640 1 --2- 172.21.15.46:0/1430333604 >> v2:172.21.15.154:3300/0 conn(0x7f6ce41538a0 0x7f6ce415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.316+0000 7f6ce1d9b640 1 -- 172.21.15.46:0/1430333604 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce40cad70 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce1d9b640 1 --2- 172.21.15.46:0/1430333604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce40ca0e0 0x7f6ce415a090 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40cfd10 tx=0x7f6cd801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2fdfce515d6fd93 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:01.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce159a640 1 -- 172.21.15.46:0/1430333604 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd80a89b0 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce159a640 1 -- 172.21.15.46:0/1430333604 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd80a8b10 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce159a640 1 -- 172.21.15.46:0/1430333604 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd80a8ea0 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/1430333604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce40ca0e0 msgr2=0x7f6ce415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce8ed6640 1 --2- 172.21.15.46:0/1430333604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce40ca0e0 0x7f6ce415a090 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40cfd10 tx=0x7f6cd801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/1430333604 shutdown_connections 2023-10-30T13:52:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce8ed6640 1 --2- 172.21.15.46:0/1430333604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce40cf160 0x7f6ce40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce8ed6640 1 --2- 172.21.15.46:0/1430333604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce40ca0e0 0x7f6ce415a090 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce8ed6640 1 --2- 172.21.15.46:0/1430333604 >> v2:172.21.15.154:3300/0 conn(0x7f6ce41538a0 0x7f6ce415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/1430333604 >> 172.21.15.46:0/1430333604 conn(0x7f6ce406bfe0 msgr2=0x7f6ce406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/1430333604 shutdown_connections 2023-10-30T13:52:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.320+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/1430333604 wait complete. 2023-10-30T13:52:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce8ed6640 1 Processor -- start 2023-10-30T13:52:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce8ed6640 1 -- start start 2023-10-30T13:52:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce8ed6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce40ca0e0 0x7f6ce41e2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:01.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce8ed6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ce40cf160 0x7f6ce41e3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:01.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce8ed6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce41538a0 0x7f6ce4142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:01.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce8ed6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce4151a30 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce8ed6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce41f0230 con 0x7f6ce40cf160 2023-10-30T13:52:01.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce8ed6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce41e32c0 con 0x7f6ce41538a0 2023-10-30T13:52:01.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce40ca0e0 0x7f6ce41e2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:01.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce40ca0e0 0x7f6ce41e2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44248/0 (socket says 172.21.15.46:44248) 2023-10-30T13:52:01.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce259c640 1 -- 172.21.15.46:0/778942223 learned_addr learned my addr 172.21.15.46:0/778942223 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:01.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce2d9d640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce41538a0 0x7f6ce4142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:01.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce1d9b640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ce40cf160 0x7f6ce41e3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce259c640 1 -- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce41538a0 msgr2=0x7f6ce4142bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce259c640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce41538a0 0x7f6ce4142bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce259c640 1 -- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ce40cf160 msgr2=0x7f6ce41e3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce259c640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ce40cf160 0x7f6ce41e3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce259c640 1 -- 172.21.15.46:0/778942223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd80094d0 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce259c640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce40ca0e0 0x7f6ce41e2a80 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400d620 tx=0x7f6cd400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6cd2ffd640 1 -- 172.21.15.46:0/778942223 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd400b620 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6cd2ffd640 1 -- 172.21.15.46:0/778942223 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd400b780 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6cd2ffd640 1 -- 172.21.15.46:0/778942223 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd400bb10 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.324+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/778942223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce41f2de0 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.328+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/778942223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce41e3510 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.328+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/778942223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce40b2a50 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.328+0000 7f6cd2ffd640 1 -- 172.21.15.46:0/778942223 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6cd400bc70 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.328+0000 7f6cd2ffd640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6cc40664d0 0x7f6cc4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:01.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.328+0000 7f6cd2ffd640 1 -- 172.21.15.46:0/778942223 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f6cd4084770 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.328+0000 7f6ce1d9b640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6cc40664d0 0x7f6cc4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:01.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.332+0000 7f6ce1d9b640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6cc40664d0 0x7f6cc4068990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6cd80b2000 tx=0x7f6cd8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:01.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.332+0000 7f6cd2ffd640 1 -- 172.21.15.46:0/778942223 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6cd4051740 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.524+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/778942223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ce404e170 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.524+0000 7f6cd2ffd640 1 -- 172.21.15.46:0/778942223 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f6cd4054d60 con 0x7f6ce40ca0e0 2023-10-30T13:52:01.530 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:01.530 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:51:56.420713+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:01.531 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 127 2023-10-30T13:52:01.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.528+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6cc40664d0 msgr2=0x7f6cc4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:01.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.528+0000 7f6ce8ed6640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6cc40664d0 0x7f6cc4068990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6cd80b2000 tx=0x7f6cd8009040 comp rx=0 tx=0).stop 2023-10-30T13:52:01.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.528+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/778942223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce40ca0e0 msgr2=0x7f6ce41e2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:01.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.528+0000 7f6ce8ed6640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce40ca0e0 0x7f6ce41e2a80 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400d620 tx=0x7f6cd400dae0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.528+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/778942223 shutdown_connections 2023-10-30T13:52:01.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.528+0000 7f6ce8ed6640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6cc40664d0 0x7f6cc4068990 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.528+0000 7f6ce8ed6640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce41538a0 0x7f6ce4142bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.528+0000 7f6ce8ed6640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ce40cf160 0x7f6ce41e3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.528+0000 7f6ce8ed6640 1 --2- 172.21.15.46:0/778942223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ce40ca0e0 0x7f6ce41e2a80 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:01.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.528+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/778942223 >> 172.21.15.46:0/778942223 conn(0x7f6ce406bfe0 msgr2=0x7f6ce406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:01.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.528+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/778942223 shutdown_connections 2023-10-30T13:52:01.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:01.528+0000 7f6ce8ed6640 1 -- 172.21.15.46:0/778942223 wait complete. 2023-10-30T13:52:01.552 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:52:03.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:03.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.700+0000 7f89ce054640 1 Processor -- start 2023-10-30T13:52:03.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.700+0000 7f89ce054640 1 -- start start 2023-10-30T13:52:03.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89ce054640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f89c806f7a0 0x7f89c806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:03.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89ce054640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89c80c6bd0 0x7f89c8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:03.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89ce054640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c806a3d0 con 0x7f89c8071110 2023-10-30T13:52:03.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89ce054640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89c80700c0 con 0x7f89c806f7a0 2023-10-30T13:52:03.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89ce054640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c8066cf0 con 0x7f89c80c6bd0 2023-10-30T13:52:03.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89c6ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89c80c6bd0 0x7f89c8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:03.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89c77fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f89c806f7a0 0x7f89c806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:03.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89c7fff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f89c8071110 0x7f89c8158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:44740/0 (socket says 172.21.15.46:44740) 2023-10-30T13:52:03.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89c7fff640 1 -- 172.21.15.46:0/3062956849 learned_addr learned my addr 172.21.15.46:0/3062956849 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:52:03.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89c77fe640 1 -- 172.21.15.46:0/3062956849 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89c80c6bd0 msgr2=0x7f89c8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:03.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89c77fe640 1 --2- 172.21.15.46:0/3062956849 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89c80c6bd0 0x7f89c8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:03.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89c77fe640 1 -- 172.21.15.46:0/3062956849 >> v1:172.21.15.154:6789/0 conn(0x7f89c8071110 legacy=0x7f89c8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:03.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89c77fe640 1 -- 172.21.15.46:0/3062956849 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c80c70d0 con 0x7f89c806f7a0 2023-10-30T13:52:03.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.704+0000 7f89c6ffd640 1 --2- 172.21.15.46:0/3062956849 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89c80c6bd0 0x7f89c8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:52:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89c77fe640 1 --2- 172.21.15.46:0/3062956849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89c806f7a0 0x7f89c806fb80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f89b80097c0 tx=0x7f89b801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75a41fc67db583cb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89c67fc640 1 -- 172.21.15.46:0/3062956849 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b80a8970 con 0x7f89c806f7a0 2023-10-30T13:52:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89c67fc640 1 -- 172.21.15.46:0/3062956849 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b80a8ad0 con 0x7f89c806f7a0 2023-10-30T13:52:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89c67fc640 1 -- 172.21.15.46:0/3062956849 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b80a8e60 con 0x7f89c806f7a0 2023-10-30T13:52:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89ce054640 1 -- 172.21.15.46:0/3062956849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89c806f7a0 msgr2=0x7f89c806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89ce054640 1 --2- 172.21.15.46:0/3062956849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89c806f7a0 0x7f89c806fb80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f89b80097c0 tx=0x7f89b801ec70 comp rx=0 tx=0).stop 2023-10-30T13:52:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89ce054640 1 -- 172.21.15.46:0/3062956849 shutdown_connections 2023-10-30T13:52:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89ce054640 1 --2- 172.21.15.46:0/3062956849 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89c80c6bd0 0x7f89c8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89ce054640 1 --2- 172.21.15.46:0/3062956849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89c806f7a0 0x7f89c806fb80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89ce054640 1 -- 172.21.15.46:0/3062956849 >> 172.21.15.46:0/3062956849 conn(0x7f89c80c7910 msgr2=0x7f89c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:03.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89ce054640 1 -- 172.21.15.46:0/3062956849 shutdown_connections 2023-10-30T13:52:03.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89ce054640 1 -- 172.21.15.46:0/3062956849 wait complete. 2023-10-30T13:52:03.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89ce054640 1 Processor -- start 2023-10-30T13:52:03.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.708+0000 7f89ce054640 1 -- start start 2023-10-30T13:52:03.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89ce054640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89c806f7a0 0x7f89c81ed6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:03.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89ce054640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89c8071110 0x7f89c80c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:03.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89ce054640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89c80c6bd0 0x7f89c8146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:03.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89ce054640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c80c84d0 con 0x7f89c806f7a0 2023-10-30T13:52:03.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89ce054640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c81edc00 con 0x7f89c8071110 2023-10-30T13:52:03.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89ce054640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c81ecc40 con 0x7f89c80c6bd0 2023-10-30T13:52:03.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89c806f7a0 0x7f89c81ed6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:03.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89c806f7a0 0x7f89c81ed6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44260/0 (socket says 172.21.15.46:44260) 2023-10-30T13:52:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89c77fe640 1 -- 172.21.15.46:0/1582840389 learned_addr learned my addr 172.21.15.46:0/1582840389 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89c6ffd640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89c8071110 0x7f89c80c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89c7fff640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89c80c6bd0 0x7f89c8146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89c77fe640 1 -- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89c80c6bd0 msgr2=0x7f89c8146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89c77fe640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89c80c6bd0 0x7f89c8146690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89c77fe640 1 -- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89c8071110 msgr2=0x7f89c80c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89c77fe640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89c8071110 0x7f89c80c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89c77fe640 1 -- 172.21.15.46:0/1582840389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc009420 con 0x7f89c806f7a0 2023-10-30T13:52:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89c7fff640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89c80c6bd0 0x7f89c8146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89c77fe640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89c806f7a0 0x7f89c81ed6c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f89b80b3040 tx=0x7f89b801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89affff640 1 -- 172.21.15.46:0/1582840389 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b80b9630 con 0x7f89c806f7a0 2023-10-30T13:52:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89affff640 1 -- 172.21.15.46:0/1582840389 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b80b97b0 con 0x7f89c806f7a0 2023-10-30T13:52:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89affff640 1 -- 172.21.15.46:0/1582840389 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b80b9b60 con 0x7f89c806f7a0 2023-10-30T13:52:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89ce054640 1 -- 172.21.15.46:0/1582840389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b8009420 con 0x7f89c806f7a0 2023-10-30T13:52:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89ce054640 1 -- 172.21.15.46:0/1582840389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c814d060 con 0x7f89c806f7a0 2023-10-30T13:52:03.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.712+0000 7f89ce054640 1 -- 172.21.15.46:0/1582840389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c804e170 con 0x7f89c806f7a0 2023-10-30T13:52:03.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.716+0000 7f89affff640 1 -- 172.21.15.46:0/1582840389 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f89b80b7040 con 0x7f89c806f7a0 2023-10-30T13:52:03.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.716+0000 7f89affff640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89b40664a0 0x7f89b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.716+0000 7f89affff640 1 -- 172.21.15.46:0/1582840389 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f89b8122220 con 0x7f89c806f7a0 2023-10-30T13:52:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.716+0000 7f89c6ffd640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89b40664a0 0x7f89b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.716+0000 7f89c6ffd640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89b40664a0 0x7f89b4068960 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f89c80c4980 tx=0x7f89bc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:03.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.720+0000 7f89affff640 1 -- 172.21.15.46:0/1582840389 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f89b80ef270 con 0x7f89c806f7a0 2023-10-30T13:52:03.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.908+0000 7f89ce054640 1 -- 172.21.15.46:0/1582840389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89c8070b50 con 0x7f89c806f7a0 2023-10-30T13:52:03.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.908+0000 7f89affff640 1 -- 172.21.15.46:0/1582840389 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f89b80f2890 con 0x7f89c806f7a0 2023-10-30T13:52:03.913 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:03.914 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:02.256838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:03.914 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 128 2023-10-30T13:52:03.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.908+0000 7f89ce054640 1 -- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89b40664a0 msgr2=0x7f89b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:03.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.908+0000 7f89ce054640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89b40664a0 0x7f89b4068960 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f89c80c4980 tx=0x7f89bc009190 comp rx=0 tx=0).stop 2023-10-30T13:52:03.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.908+0000 7f89ce054640 1 -- 172.21.15.46:0/1582840389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89c806f7a0 msgr2=0x7f89c81ed6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:03.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.908+0000 7f89ce054640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89c806f7a0 0x7f89c81ed6c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f89b80b3040 tx=0x7f89b801ba10 comp rx=0 tx=0).stop 2023-10-30T13:52:03.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.908+0000 7f89ce054640 1 -- 172.21.15.46:0/1582840389 shutdown_connections 2023-10-30T13:52:03.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.908+0000 7f89ce054640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89b40664a0 0x7f89b4068960 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:03.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.908+0000 7f89ce054640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89c80c6bd0 0x7f89c8146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:03.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.908+0000 7f89ce054640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89c8071110 0x7f89c80c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:03.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.908+0000 7f89ce054640 1 --2- 172.21.15.46:0/1582840389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89c806f7a0 0x7f89c81ed6c0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:03.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.908+0000 7f89ce054640 1 -- 172.21.15.46:0/1582840389 >> 172.21.15.46:0/1582840389 conn(0x7f89c80c7910 msgr2=0x7f89c80c3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:03.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.912+0000 7f89ce054640 1 -- 172.21.15.46:0/1582840389 shutdown_connections 2023-10-30T13:52:03.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:03.912+0000 7f89ce054640 1 -- 172.21.15.46:0/1582840389 wait complete. 2023-10-30T13:52:03.936 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 128, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 128, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6858": { "addr": "172.21.15.46:6837/2572113428", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2572113428, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2572113428, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6858, "incarnation": 117, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_7002": { "addr": "172.21.15.46:6833/2807435195", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2807435195, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2807435195, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7002, "incarnation": 125, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:52:02.256838+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7002, "mds_1": 6858 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:52:05.937 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:06.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf22c9c640 1 Processor -- start 2023-10-30T13:52:06.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf22c9c640 1 -- start start 2023-10-30T13:52:06.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf22c9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf1c1538a0 0x7fdf1c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:06.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf22c9c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf1c0ca0e0 0x7fdf1c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:06.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf22c9c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf1c0cb800 con 0x7fdf1c0cf160 2023-10-30T13:52:06.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf22c9c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf1c15dc30 con 0x7fdf1c1538a0 2023-10-30T13:52:06.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf22c9c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c0cb330 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf20a38640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf1c1538a0 0x7fdf1c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:06.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf20a38640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf1c1538a0 0x7fdf1c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44272/0 (socket says 172.21.15.46:44272) 2023-10-30T13:52:06.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf20a38640 1 -- 172.21.15.46:0/1497561747 learned_addr learned my addr 172.21.15.46:0/1497561747 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:06.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf1bfff640 1 --2- 172.21.15.46:0/1497561747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf1c0ca0e0 0x7fdf1c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:06.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf20a38640 1 -- 172.21.15.46:0/1497561747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf1c0ca0e0 msgr2=0x7fdf1c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:06.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf20a38640 1 --2- 172.21.15.46:0/1497561747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf1c0ca0e0 0x7fdf1c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:06.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf20a38640 1 -- 172.21.15.46:0/1497561747 >> v1:172.21.15.154:6789/0 conn(0x7fdf1c0cf160 legacy=0x7fdf1c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:06.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf20a38640 1 -- 172.21.15.46:0/1497561747 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c0cad70 con 0x7fdf1c1538a0 2023-10-30T13:52:06.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.088+0000 7fdf1bfff640 1 --2- 172.21.15.46:0/1497561747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf1c0ca0e0 0x7fdf1c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:06.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.092+0000 7fdf20a38640 1 --2- 172.21.15.46:0/1497561747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf1c1538a0 0x7fdf1c15d6f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009870 tx=0x7fdf1001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d5abbdc8a947ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:06.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.092+0000 7fdf1b7fe640 1 -- 172.21.15.46:0/1497561747 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100a8a30 con 0x7fdf1c1538a0 2023-10-30T13:52:06.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.092+0000 7fdf1b7fe640 1 -- 172.21.15.46:0/1497561747 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf100a8b90 con 0x7fdf1c1538a0 2023-10-30T13:52:06.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.092+0000 7fdf1b7fe640 1 -- 172.21.15.46:0/1497561747 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100b0440 con 0x7fdf1c1538a0 2023-10-30T13:52:06.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.092+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1497561747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf1c1538a0 msgr2=0x7fdf1c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:06.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.092+0000 7fdf22c9c640 1 --2- 172.21.15.46:0/1497561747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf1c1538a0 0x7fdf1c15d6f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009870 tx=0x7fdf1001ed30 comp rx=0 tx=0).stop 2023-10-30T13:52:06.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.092+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1497561747 shutdown_connections 2023-10-30T13:52:06.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.092+0000 7fdf22c9c640 1 --2- 172.21.15.46:0/1497561747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf1c0ca0e0 0x7fdf1c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:06.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.092+0000 7fdf22c9c640 1 --2- 172.21.15.46:0/1497561747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf1c1538a0 0x7fdf1c15d6f0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:06.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.092+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1497561747 >> 172.21.15.46:0/1497561747 conn(0x7fdf1c06bfe0 msgr2=0x7fdf1c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:06.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.092+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1497561747 shutdown_connections 2023-10-30T13:52:06.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.092+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1497561747 wait complete. 2023-10-30T13:52:06.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf22c9c640 1 Processor -- start 2023-10-30T13:52:06.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf22c9c640 1 -- start start 2023-10-30T13:52:06.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf22c9c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf1c0ca0e0 0x7fdf1c0ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:06.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf22c9c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf1c0cf160 0x7fdf1c158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:06.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf22c9c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf1c1538a0 0x7fdf1c14cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:06.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf22c9c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c1ef540 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf22c9c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c1f0510 con 0x7fdf1c1538a0 2023-10-30T13:52:06.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf22c9c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c0666b0 con 0x7fdf1c0cf160 2023-10-30T13:52:06.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf20a38640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf1c0ca0e0 0x7fdf1c0ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:06.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf20a38640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf1c0ca0e0 0x7fdf1c0ce390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44288/0 (socket says 172.21.15.46:44288) 2023-10-30T13:52:06.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf20a38640 1 -- 172.21.15.46:0/1654458618 learned_addr learned my addr 172.21.15.46:0/1654458618 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:06.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf1bfff640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf1c0cf160 0x7fdf1c158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:06.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf21239640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf1c1538a0 0x7fdf1c14cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:06.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf20a38640 1 -- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf1c0cf160 msgr2=0x7fdf1c158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:06.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf20a38640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf1c0cf160 0x7fdf1c158250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:06.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf20a38640 1 -- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf1c1538a0 msgr2=0x7fdf1c14cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:06.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf20a38640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf1c1538a0 0x7fdf1c14cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:06.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf20a38640 1 -- 172.21.15.46:0/1654458618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf100094d0 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf20a38640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf1c0ca0e0 0x7fdf1c0ce390 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fdf100b3040 tx=0x7fdf1001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:06.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf197fa640 1 -- 172.21.15.46:0/1654458618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100be070 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf197fa640 1 -- 172.21.15.46:0/1654458618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf100ba400 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf197fa640 1 -- 172.21.15.46:0/1654458618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100ba7d0 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1654458618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1c1ef750 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1654458618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c1efd10 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.096+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1654458618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c0b2a50 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.100+0000 7fdf197fa640 1 -- 172.21.15.46:0/1654458618 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdf100c3410 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.100+0000 7fdf197fa640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf000664a0 0x7fdf00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:06.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.100+0000 7fdf197fa640 1 -- 172.21.15.46:0/1654458618 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fdf10121f80 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.100+0000 7fdf1bfff640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf000664a0 0x7fdf00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:06.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.100+0000 7fdf1bfff640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf000664a0 0x7fdf00068960 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c15d9a0 tx=0x7fdf08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:06.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.104+0000 7fdf197fa640 1 -- 172.21.15.46:0/1654458618 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdf100eeea0 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.292+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1654458618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf1c04e170 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.292+0000 7fdf197fa640 1 -- 172.21.15.46:0/1654458618 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fdf100f24c0 con 0x7fdf1c0ca0e0 2023-10-30T13:52:06.300 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:06.300 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:02.256838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:06.300 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 128 2023-10-30T13:52:06.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.296+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf000664a0 msgr2=0x7fdf00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:06.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.296+0000 7fdf22c9c640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf000664a0 0x7fdf00068960 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c15d9a0 tx=0x7fdf08008040 comp rx=0 tx=0).stop 2023-10-30T13:52:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.296+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1654458618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf1c0ca0e0 msgr2=0x7fdf1c0ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.296+0000 7fdf22c9c640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf1c0ca0e0 0x7fdf1c0ce390 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fdf100b3040 tx=0x7fdf1001bad0 comp rx=0 tx=0).stop 2023-10-30T13:52:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.296+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1654458618 shutdown_connections 2023-10-30T13:52:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.296+0000 7fdf22c9c640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf000664a0 0x7fdf00068960 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.296+0000 7fdf22c9c640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf1c1538a0 0x7fdf1c14cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.296+0000 7fdf22c9c640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf1c0cf160 0x7fdf1c158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.296+0000 7fdf22c9c640 1 --2- 172.21.15.46:0/1654458618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf1c0ca0e0 0x7fdf1c0ce390 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.296+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1654458618 >> 172.21.15.46:0/1654458618 conn(0x7fdf1c06bfe0 msgr2=0x7fdf1c14db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.296+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1654458618 shutdown_connections 2023-10-30T13:52:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:06.296+0000 7fdf22c9c640 1 -- 172.21.15.46:0/1654458618 wait complete. 2023-10-30T13:52:06.321 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:52:08.322 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:08.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.468+0000 7f478ea79640 1 Processor -- start 2023-10-30T13:52:08.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.468+0000 7f478ea79640 1 -- start start 2023-10-30T13:52:08.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f478ea79640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f478806f7a0 0x7f478806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:08.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f478ea79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47880c6bd0 0x7f4788069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:08.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f478ea79640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478806a3d0 con 0x7f4788071110 2023-10-30T13:52:08.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f478ea79640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47880700c0 con 0x7f478806f7a0 2023-10-30T13:52:08.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f478ea79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4788066cf0 con 0x7f47880c6bd0 2023-10-30T13:52:08.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f478c815640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f478806f7a0 0x7f478806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:08.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f478c815640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f478806f7a0 0x7f478806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42674/0 (socket says 172.21.15.46:42674) 2023-10-30T13:52:08.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f478c815640 1 -- 172.21.15.46:0/77067902 learned_addr learned my addr 172.21.15.46:0/77067902 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:08.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f4787fff640 1 --2- 172.21.15.46:0/77067902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47880c6bd0 0x7f4788069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:08.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f478c815640 1 -- 172.21.15.46:0/77067902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47880c6bd0 msgr2=0x7f4788069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:08.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f478c815640 1 --2- 172.21.15.46:0/77067902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47880c6bd0 0x7f4788069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:08.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f478c815640 1 -- 172.21.15.46:0/77067902 >> v1:172.21.15.46:6789/0 conn(0x7f4788071110 legacy=0x7f4788158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:08.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.472+0000 7f478c815640 1 -- 172.21.15.46:0/77067902 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47880c70d0 con 0x7f478806f7a0 2023-10-30T13:52:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f478c815640 1 --2- 172.21.15.46:0/77067902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478806f7a0 0x7f478806fb80 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f477c009870 tx=0x7f477c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e1a5768ce7256d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f47877fe640 1 -- 172.21.15.46:0/77067902 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477c0a8a30 con 0x7f478806f7a0 2023-10-30T13:52:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f47877fe640 1 -- 172.21.15.46:0/77067902 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477c0a8b90 con 0x7f478806f7a0 2023-10-30T13:52:08.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f47877fe640 1 -- 172.21.15.46:0/77067902 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477c0b0440 con 0x7f478806f7a0 2023-10-30T13:52:08.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f478ea79640 1 -- 172.21.15.46:0/77067902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478806f7a0 msgr2=0x7f478806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:08.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f478ea79640 1 --2- 172.21.15.46:0/77067902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478806f7a0 0x7f478806fb80 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f477c009870 tx=0x7f477c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:52:08.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f478ea79640 1 -- 172.21.15.46:0/77067902 shutdown_connections 2023-10-30T13:52:08.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f478ea79640 1 --2- 172.21.15.46:0/77067902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47880c6bd0 0x7f4788069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f478ea79640 1 --2- 172.21.15.46:0/77067902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478806f7a0 0x7f478806fb80 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f478ea79640 1 -- 172.21.15.46:0/77067902 >> 172.21.15.46:0/77067902 conn(0x7f47880c7910 msgr2=0x7f47880c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f478ea79640 1 -- 172.21.15.46:0/77067902 shutdown_connections 2023-10-30T13:52:08.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f478ea79640 1 -- 172.21.15.46:0/77067902 wait complete. 2023-10-30T13:52:08.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f478ea79640 1 Processor -- start 2023-10-30T13:52:08.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.476+0000 7f478ea79640 1 -- start start 2023-10-30T13:52:08.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f478ea79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478806f7a0 0x7f47880c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:08.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f478ea79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4788071110 0x7f47880c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f478ea79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47880c6bd0 0x7f47880bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f478ea79640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47880ca1a0 con 0x7f4788071110 2023-10-30T13:52:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f478ea79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478814b430 con 0x7f47880c6bd0 2023-10-30T13:52:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f478ea79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47880ae9f0 con 0x7f478806f7a0 2023-10-30T13:52:08.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f4787fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4788071110 0x7f47880c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f4787fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4788071110 0x7f47880c17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42688/0 (socket says 172.21.15.46:42688) 2023-10-30T13:52:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f4787fff640 1 -- 172.21.15.46:0/3106950691 learned_addr learned my addr 172.21.15.46:0/3106950691 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f478c815640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478806f7a0 0x7f47880c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:08.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f478d016640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47880c6bd0 0x7f47880bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f4787fff640 1 -- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478806f7a0 msgr2=0x7f47880c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f4787fff640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478806f7a0 0x7f47880c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f4787fff640 1 -- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47880c6bd0 msgr2=0x7f47880bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:08.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f4787fff640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47880c6bd0 0x7f47880bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:08.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f4787fff640 1 -- 172.21.15.46:0/3106950691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c0094d0 con 0x7f4788071110 2023-10-30T13:52:08.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f478d016640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47880c6bd0 0x7f47880bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:08.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f4787fff640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4788071110 0x7f47880c17f0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f477400d5f0 tx=0x7f477400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:08.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f47857fa640 1 -- 172.21.15.46:0/3106950691 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740125f0 con 0x7f4788071110 2023-10-30T13:52:08.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f47857fa640 1 -- 172.21.15.46:0/3106950691 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4774012750 con 0x7f4788071110 2023-10-30T13:52:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f47857fa640 1 -- 172.21.15.46:0/3106950691 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774012ae0 con 0x7f4788071110 2023-10-30T13:52:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f478ea79640 1 -- 172.21.15.46:0/3106950691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47880af570 con 0x7f4788071110 2023-10-30T13:52:08.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f478ea79640 1 -- 172.21.15.46:0/3106950691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47880aef20 con 0x7f4788071110 2023-10-30T13:52:08.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f478ea79640 1 -- 172.21.15.46:0/3106950691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47880af700 con 0x7f4788071110 2023-10-30T13:52:08.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.480+0000 7f47857fa640 1 -- 172.21.15.46:0/3106950691 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4774012c40 con 0x7f4788071110 2023-10-30T13:52:08.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.484+0000 7f47857fa640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f476c0664d0 0x7f476c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:08.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.484+0000 7f47857fa640 1 -- 172.21.15.46:0/3106950691 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f47740847d0 con 0x7f4788071110 2023-10-30T13:52:08.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.484+0000 7f478c815640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f476c0664d0 0x7f476c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:08.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.484+0000 7f478c815640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f476c0664d0 0x7f476c068990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f477c0b2000 tx=0x7f477c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:08.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.488+0000 7f47857fa640 1 -- 172.21.15.46:0/3106950691 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f47740516f0 con 0x7f4788071110 2023-10-30T13:52:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.676+0000 7f478ea79640 1 -- 172.21.15.46:0/3106950691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f478804e170 con 0x7f4788071110 2023-10-30T13:52:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.676+0000 7f47857fa640 1 -- 172.21.15.46:0/3106950691 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f4774007380 con 0x7f4788071110 2023-10-30T13:52:08.683 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:08.683 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:07.401786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 129 2023-10-30T13:52:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.680+0000 7f478ea79640 1 -- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f476c0664d0 msgr2=0x7f476c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.680+0000 7f478ea79640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f476c0664d0 0x7f476c068990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f477c0b2000 tx=0x7f477c009040 comp rx=0 tx=0).stop 2023-10-30T13:52:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.680+0000 7f478ea79640 1 -- 172.21.15.46:0/3106950691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4788071110 msgr2=0x7f47880c17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.680+0000 7f478ea79640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4788071110 0x7f47880c17f0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f477400d5f0 tx=0x7f477400dab0 comp rx=0 tx=0).stop 2023-10-30T13:52:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.680+0000 7f478ea79640 1 -- 172.21.15.46:0/3106950691 shutdown_connections 2023-10-30T13:52:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.680+0000 7f478ea79640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f476c0664d0 0x7f476c068990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.680+0000 7f478ea79640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47880c6bd0 0x7f47880bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:08.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.680+0000 7f478ea79640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4788071110 0x7f47880c17f0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:08.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.680+0000 7f478ea79640 1 --2- 172.21.15.46:0/3106950691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478806f7a0 0x7f47880c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:08.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.680+0000 7f478ea79640 1 -- 172.21.15.46:0/3106950691 >> 172.21.15.46:0/3106950691 conn(0x7f47880c7910 msgr2=0x7f47880c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:08.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.680+0000 7f478ea79640 1 -- 172.21.15.46:0/3106950691 shutdown_connections 2023-10-30T13:52:08.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:08.680+0000 7f478ea79640 1 -- 172.21.15.46:0/3106950691 wait complete. 2023-10-30T13:52:08.704 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6702 2023-10-30T13:52:08.704 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.c 2023-10-30T13:52:10.706 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:52:10.706 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:52:10.706 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:52:10.710 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:52:10.711 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:10.806 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-10-30T13:52:10.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f132a9f5640 1 Processor -- start 2023-10-30T13:52:10.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f132a9f5640 1 -- start start 2023-10-30T13:52:10.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f132a9f5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13241538a0 0x7f132415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:10.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f132a9f5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13240cb800 con 0x7f13240cf160 2023-10-30T13:52:10.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f132a9f5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132415dc30 con 0x7f13240ca0e0 2023-10-30T13:52:10.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f132a9f5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13240cb330 con 0x7f13241538a0 2023-10-30T13:52:10.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f1328f92640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f13240cf160 0x7f1324164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48586/0 (socket says 172.21.15.46:48586) 2023-10-30T13:52:10.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f1328f92640 1 -- 172.21.15.46:0/1547228268 learned_addr learned my addr 172.21.15.46:0/1547228268 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:52:10.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f1323fff640 1 --2- 172.21.15.46:0/1547228268 >> v2:172.21.15.46:3300/0 conn(0x7f13241538a0 0x7f132415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:10.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f1322ffd640 1 -- 172.21.15.46:0/1547228268 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035748820 0 0) 0x7f13240cb800 con 0x7f13240cf160 2023-10-30T13:52:10.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f1323fff640 1 -- 172.21.15.46:0/1547228268 >> v1:172.21.15.154:6789/0 conn(0x7f13240ca0e0 legacy=0x7f132415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:10.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f1323fff640 1 -- 172.21.15.46:0/1547228268 >> v1:172.21.15.46:6789/0 conn(0x7f13240cf160 legacy=0x7f1324164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:10.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.860+0000 7f1323fff640 1 -- 172.21.15.46:0/1547228268 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13240cad70 con 0x7f13241538a0 2023-10-30T13:52:10.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.864+0000 7f1323fff640 1 --2- 172.21.15.46:0/1547228268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13241538a0 0x7f132415d6f0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f13140097c0 tx=0x7f131401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12394581931fb047 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.864+0000 7f1322ffd640 1 -- 172.21.15.46:0/1547228268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13140a8970 con 0x7f13241538a0 2023-10-30T13:52:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.864+0000 7f1322ffd640 1 -- 172.21.15.46:0/1547228268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13140a8ad0 con 0x7f13241538a0 2023-10-30T13:52:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.864+0000 7f1322ffd640 1 -- 172.21.15.46:0/1547228268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13140a8e60 con 0x7f13241538a0 2023-10-30T13:52:10.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.864+0000 7f132a9f5640 1 -- 172.21.15.46:0/1547228268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13241538a0 msgr2=0x7f132415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:10.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.864+0000 7f132a9f5640 1 --2- 172.21.15.46:0/1547228268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13241538a0 0x7f132415d6f0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f13140097c0 tx=0x7f131401ec70 comp rx=0 tx=0).stop 2023-10-30T13:52:10.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.864+0000 7f132a9f5640 1 -- 172.21.15.46:0/1547228268 shutdown_connections 2023-10-30T13:52:10.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.864+0000 7f132a9f5640 1 --2- 172.21.15.46:0/1547228268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13241538a0 0x7f132415d6f0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:10.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.864+0000 7f132a9f5640 1 -- 172.21.15.46:0/1547228268 >> 172.21.15.46:0/1547228268 conn(0x7f132406bfe0 msgr2=0x7f132406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:10.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.864+0000 7f132a9f5640 1 -- 172.21.15.46:0/1547228268 shutdown_connections 2023-10-30T13:52:10.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.864+0000 7f132a9f5640 1 -- 172.21.15.46:0/1547228268 wait complete. 2023-10-30T13:52:10.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f132a9f5640 1 Processor -- start 2023-10-30T13:52:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f132a9f5640 1 -- start start 2023-10-30T13:52:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f132a9f5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13240ca0e0 0x7f13240c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f132a9f5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240cf160 0x7f13241507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f132a9f5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13241538a0 0x7f13240b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f132a9f5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1324157830 con 0x7f13240cf160 2023-10-30T13:52:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f132a9f5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13241575c0 con 0x7f13240ca0e0 2023-10-30T13:52:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f132a9f5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13240b3d90 con 0x7f13241538a0 2023-10-30T13:52:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f13237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240cf160 0x7f13241507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f13237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240cf160 0x7f13241507d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42730/0 (socket says 172.21.15.46:42730) 2023-10-30T13:52:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f13237fe640 1 -- 172.21.15.46:0/3473175525 learned_addr learned my addr 172.21.15.46:0/3473175525 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f1323fff640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13240ca0e0 0x7f13240c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f1328f92640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13241538a0 0x7f13240b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f13237fe640 1 -- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13241538a0 msgr2=0x7f13240b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f13237fe640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13241538a0 0x7f13240b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f13237fe640 1 -- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13240ca0e0 msgr2=0x7f13240c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f13237fe640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13240ca0e0 0x7f13240c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f13237fe640 1 -- 172.21.15.46:0/3473175525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1314009420 con 0x7f13240cf160 2023-10-30T13:52:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f13237fe640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240cf160 0x7f13241507d0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f131800c3c0 tx=0x7f131800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:10.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f1320ff9640 1 -- 172.21.15.46:0/3473175525 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13180052d0 con 0x7f13240cf160 2023-10-30T13:52:10.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f1320ff9640 1 -- 172.21.15.46:0/3473175525 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1318005430 con 0x7f13240cf160 2023-10-30T13:52:10.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f1320ff9640 1 -- 172.21.15.46:0/3473175525 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1318003840 con 0x7f13240cf160 2023-10-30T13:52:10.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f132a9f5640 1 -- 172.21.15.46:0/3473175525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1324159040 con 0x7f13240cf160 2023-10-30T13:52:10.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.868+0000 7f132a9f5640 1 -- 172.21.15.46:0/3473175525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13240b4010 con 0x7f13240cf160 2023-10-30T13:52:10.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.872+0000 7f132a9f5640 1 -- 172.21.15.46:0/3473175525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f132404e170 con 0x7f13240cf160 2023-10-30T13:52:10.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.872+0000 7f1320ff9640 1 -- 172.21.15.46:0/3473175525 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13180039a0 con 0x7f13240cf160 2023-10-30T13:52:10.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.872+0000 7f1320ff9640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13100664a0 0x7f1310068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.872+0000 7f1320ff9640 1 -- 172.21.15.46:0/3473175525 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f1318083780 con 0x7f13240cf160 2023-10-30T13:52:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.872+0000 7f1323fff640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13100664a0 0x7f1310068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.872+0000 7f1323fff640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13100664a0 0x7f1310068960 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f13140b2000 tx=0x7f1314008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:10.876+0000 7f1320ff9640 1 -- 172.21.15.46:0/3473175525 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f13180506c0 con 0x7f13240cf160 2023-10-30T13:52:11.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.068+0000 7f132a9f5640 1 -- 172.21.15.46:0/3473175525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1324159b40 con 0x7f13240cf160 2023-10-30T13:52:11.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.068+0000 7f1320ff9640 1 -- 172.21.15.46:0/3473175525 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f1318053ce0 con 0x7f13240cf160 2023-10-30T13:52:11.074 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:11.074 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:07.401786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:11.074 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 129 2023-10-30T13:52:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.072+0000 7f132a9f5640 1 -- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13100664a0 msgr2=0x7f1310068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.072+0000 7f132a9f5640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13100664a0 0x7f1310068960 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f13140b2000 tx=0x7f1314008040 comp rx=0 tx=0).stop 2023-10-30T13:52:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.072+0000 7f132a9f5640 1 -- 172.21.15.46:0/3473175525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240cf160 msgr2=0x7f13241507d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.072+0000 7f132a9f5640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240cf160 0x7f13241507d0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f131800c3c0 tx=0x7f131800c880 comp rx=0 tx=0).stop 2023-10-30T13:52:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.072+0000 7f132a9f5640 1 -- 172.21.15.46:0/3473175525 shutdown_connections 2023-10-30T13:52:11.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.072+0000 7f132a9f5640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13100664a0 0x7f1310068960 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:11.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.072+0000 7f132a9f5640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13241538a0 0x7f13240b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:11.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.072+0000 7f132a9f5640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13240cf160 0x7f13241507d0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:11.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.072+0000 7f132a9f5640 1 --2- 172.21.15.46:0/3473175525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13240ca0e0 0x7f13240c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:11.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.072+0000 7f132a9f5640 1 -- 172.21.15.46:0/3473175525 >> 172.21.15.46:0/3473175525 conn(0x7f132406bfe0 msgr2=0x7f1324152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:11.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.072+0000 7f132a9f5640 1 -- 172.21.15.46:0/3473175525 shutdown_connections 2023-10-30T13:52:11.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:11.072+0000 7f132a9f5640 1 -- 172.21.15.46:0/3473175525 wait complete. 2023-10-30T13:52:11.095 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:52:13.096 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:13.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.244+0000 7f47ea10e640 1 Processor -- start 2023-10-30T13:52:13.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.244+0000 7f47ea10e640 1 -- start start 2023-10-30T13:52:13.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47ea10e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f47e406f7a0 0x7f47e406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:13.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47ea10e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47e40c6bd0 0x7f47e4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:13.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47ea10e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e406a3d0 con 0x7f47e4071110 2023-10-30T13:52:13.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47ea10e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e40700c0 con 0x7f47e406f7a0 2023-10-30T13:52:13.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47ea10e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e4066cf0 con 0x7f47e40c6bd0 2023-10-30T13:52:13.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47e2ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47e40c6bd0 0x7f47e4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:13.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47e37fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f47e406f7a0 0x7f47e406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:13.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47e2ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47e40c6bd0 0x7f47e4069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:39394/0 (socket says 172.21.15.46:39394) 2023-10-30T13:52:13.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47e3fff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f47e4071110 0x7f47e4158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:59190/0 (socket says 172.21.15.46:59190) 2023-10-30T13:52:13.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47e2ffd640 1 -- 172.21.15.46:0/2631444682 learned_addr learned my addr 172.21.15.46:0/2631444682 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:13.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47e2ffd640 1 -- 172.21.15.46:0/2631444682 >> v2:172.21.15.154:3300/0 conn(0x7f47e406f7a0 msgr2=0x7f47e406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:13.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47e2ffd640 1 --2- 172.21.15.46:0/2631444682 >> v2:172.21.15.154:3300/0 conn(0x7f47e406f7a0 0x7f47e406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:13.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47e2ffd640 1 -- 172.21.15.46:0/2631444682 >> v1:172.21.15.154:6789/0 conn(0x7f47e4071110 legacy=0x7f47e4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:13.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47e2ffd640 1 -- 172.21.15.46:0/2631444682 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e40c70d0 con 0x7f47e40c6bd0 2023-10-30T13:52:13.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.248+0000 7f47e27fc640 1 -- 172.21.15.46:0/2631444682 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437942397 0 0) 0x7f47e406a3d0 con 0x7f47e4071110 2023-10-30T13:52:13.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.252+0000 7f47e2ffd640 1 --2- 172.21.15.46:0/2631444682 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47e40c6bd0 0x7f47e4069e90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f47d8009870 tx=0x7f47d801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=160cd18f6ef15a40 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:13.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.252+0000 7f47e27fc640 1 -- 172.21.15.46:0/2631444682 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d80a8a30 con 0x7f47e40c6bd0 2023-10-30T13:52:13.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.252+0000 7f47e27fc640 1 -- 172.21.15.46:0/2631444682 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d80a8b90 con 0x7f47e40c6bd0 2023-10-30T13:52:13.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.252+0000 7f47e27fc640 1 -- 172.21.15.46:0/2631444682 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d80b0440 con 0x7f47e40c6bd0 2023-10-30T13:52:13.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.252+0000 7f47ea10e640 1 -- 172.21.15.46:0/2631444682 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47e40c6bd0 msgr2=0x7f47e4069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:13.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.252+0000 7f47ea10e640 1 --2- 172.21.15.46:0/2631444682 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47e40c6bd0 0x7f47e4069e90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f47d8009870 tx=0x7f47d801ed30 comp rx=0 tx=0).stop 2023-10-30T13:52:13.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.252+0000 7f47ea10e640 1 -- 172.21.15.46:0/2631444682 shutdown_connections 2023-10-30T13:52:13.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.252+0000 7f47ea10e640 1 --2- 172.21.15.46:0/2631444682 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47e40c6bd0 0x7f47e4069e90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:13.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.252+0000 7f47ea10e640 1 --2- 172.21.15.46:0/2631444682 >> v2:172.21.15.154:3300/0 conn(0x7f47e406f7a0 0x7f47e406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:13.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.252+0000 7f47ea10e640 1 -- 172.21.15.46:0/2631444682 >> 172.21.15.46:0/2631444682 conn(0x7f47e40c7910 msgr2=0x7f47e40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:13.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.252+0000 7f47ea10e640 1 -- 172.21.15.46:0/2631444682 shutdown_connections 2023-10-30T13:52:13.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.252+0000 7f47ea10e640 1 -- 172.21.15.46:0/2631444682 wait complete. 2023-10-30T13:52:13.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47ea10e640 1 Processor -- start 2023-10-30T13:52:13.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47ea10e640 1 -- start start 2023-10-30T13:52:13.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47ea10e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47e406f7a0 0x7f47e41efd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47ea10e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47e4071110 0x7f47e4150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47ea10e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47e40c6bd0 0x7f47e4150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47ea10e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e4144080 con 0x7f47e40c6bd0 2023-10-30T13:52:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47ea10e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e415a630 con 0x7f47e4071110 2023-10-30T13:52:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47ea10e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e4159b20 con 0x7f47e406f7a0 2023-10-30T13:52:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47e40c6bd0 0x7f47e4150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47e40c6bd0 0x7f47e4150670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42738/0 (socket says 172.21.15.46:42738) 2023-10-30T13:52:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47e3fff640 1 -- 172.21.15.46:0/3203572082 learned_addr learned my addr 172.21.15.46:0/3203572082 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47e37fe640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47e406f7a0 0x7f47e41efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:13.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47e2ffd640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47e4071110 0x7f47e4150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47e3fff640 1 -- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47e406f7a0 msgr2=0x7f47e41efd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47e3fff640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47e406f7a0 0x7f47e41efd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47e3fff640 1 -- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47e4071110 msgr2=0x7f47e4150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47e3fff640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47e4071110 0x7f47e4150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47e3fff640 1 -- 172.21.15.46:0/3203572082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d80094d0 con 0x7f47e40c6bd0 2023-10-30T13:52:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47e3fff640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47e40c6bd0 0x7f47e4150670 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00c780 tx=0x7f47dc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47cbfff640 1 -- 172.21.15.46:0/3203572082 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc00ce90 con 0x7f47e40c6bd0 2023-10-30T13:52:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47cbfff640 1 -- 172.21.15.46:0/3203572082 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc00a640 con 0x7f47e40c6bd0 2023-10-30T13:52:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47cbfff640 1 -- 172.21.15.46:0/3203572082 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc00a9d0 con 0x7f47e40c6bd0 2023-10-30T13:52:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47ea10e640 1 -- 172.21.15.46:0/3203572082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e41498e0 con 0x7f47e40c6bd0 2023-10-30T13:52:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.256+0000 7f47ea10e640 1 -- 172.21.15.46:0/3203572082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e4159020 con 0x7f47e40c6bd0 2023-10-30T13:52:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.260+0000 7f47ea10e640 1 -- 172.21.15.46:0/3203572082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e41591b0 con 0x7f47e40c6bd0 2023-10-30T13:52:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.260+0000 7f47cbfff640 1 -- 172.21.15.46:0/3203572082 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f47dc00ab30 con 0x7f47e40c6bd0 2023-10-30T13:52:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.260+0000 7f47cbfff640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f47d00664a0 0x7f47d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.260+0000 7f47cbfff640 1 -- 172.21.15.46:0/3203572082 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f47dc0838f0 con 0x7f47e40c6bd0 2023-10-30T13:52:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.260+0000 7f47e37fe640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f47d00664a0 0x7f47d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.260+0000 7f47e37fe640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f47d00664a0 0x7f47d0068960 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009ac0 tx=0x7f47d4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.264+0000 7f47cbfff640 1 -- 172.21.15.46:0/3203572082 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f47dc050830 con 0x7f47e40c6bd0 2023-10-30T13:52:13.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47ea10e640 1 -- 172.21.15.46:0/3203572082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47e404e170 con 0x7f47e40c6bd0 2023-10-30T13:52:13.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47cbfff640 1 -- 172.21.15.46:0/3203572082 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f47dc018020 con 0x7f47e40c6bd0 2023-10-30T13:52:13.529 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:13.529 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:07.401786+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:13.529 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 130 2023-10-30T13:52:13.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47ea10e640 1 -- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f47d00664a0 msgr2=0x7f47d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:13.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47ea10e640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f47d00664a0 0x7f47d0068960 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009ac0 tx=0x7f47d4009190 comp rx=0 tx=0).stop 2023-10-30T13:52:13.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47ea10e640 1 -- 172.21.15.46:0/3203572082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47e40c6bd0 msgr2=0x7f47e4150670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:13.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47ea10e640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47e40c6bd0 0x7f47e4150670 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00c780 tx=0x7f47dc00cc40 comp rx=0 tx=0).stop 2023-10-30T13:52:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47ea10e640 1 -- 172.21.15.46:0/3203572082 shutdown_connections 2023-10-30T13:52:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47ea10e640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f47d00664a0 0x7f47d0068960 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47ea10e640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47e40c6bd0 0x7f47e4150670 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47ea10e640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47e4071110 0x7f47e4150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:13.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47ea10e640 1 --2- 172.21.15.46:0/3203572082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f47e406f7a0 0x7f47e41efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47ea10e640 1 -- 172.21.15.46:0/3203572082 >> 172.21.15.46:0/3203572082 conn(0x7f47e40c7910 msgr2=0x7f47e40c2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.524+0000 7f47ea10e640 1 -- 172.21.15.46:0/3203572082 shutdown_connections 2023-10-30T13:52:13.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:13.528+0000 7f47ea10e640 1 -- 172.21.15.46:0/3203572082 wait complete. 2023-10-30T13:52:13.561 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:52:13.561 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 29 secs before thrashing 2023-10-30T13:52:18.295 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-10-30T13:52:18.292+0000 7f3940f23640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:52:18.295 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-10-30T13:52:18.292+0000 7f3940f23640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:52:21.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:52:21.277 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:52:26.856 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 26% (68/258) Updating files: 27% (70/258) Updating files: 28% (73/258) Updating files: 29% (75/258) Updating files: 30% (78/258) Updating files: 31% (80/258) Updating files: 32% (83/258) Updating files: 33% (86/258) Updating files: 34% (88/258) Updating files: 35% (91/258) Updating files: 36% (93/258) Updating files: 37% (96/258) Updating files: 38% (99/258) Updating files: 39% (101/258) Updating files: 40% (104/258) Updating files: 41% (106/258) Updating files: 42% (109/258) Updating files: 43% (111/258) Updating files: 44% (114/258) Updating files: 45% (117/258) Updating files: 46% (119/258) Updating files: 47% (122/258) Updating files: 48% (124/258) Updating files: 49% (127/258) Updating files: 50% (129/258) Updating files: 51% (132/258) Updating files: 52% (135/258) Updating files: 53% (137/258) Updating files: 54% (140/258) Updating files: 55% (142/258) Updating files: 55% (144/258) Updating files: 56% (145/258) Updating files: 57% (148/258) Updating files: 58% (150/258) Updating files: 59% (153/258) Updating files: 60% (155/258) Updating files: 61% (158/258) Updating files: 62% (160/258) Updating files: 63% (163/258) Updating files: 64% (166/258) Updating files: 65% (168/258) Updating files: 66% (171/258) Updating files: 66% (172/258) Updating files: 67% (173/258) Updating files: 68% (176/258) Updating files: 69% (179/258) Updating files: 70% (181/258) Updating files: 71% (184/258) Updating files: 72% (186/258) Updating files: 73% (189/258) Updating files: 74% (191/258) Updating files: 75% (194/258) Updating files: 76% (197/258) Updating files: 77% (199/258) Updating files: 78% (202/258) Updating files: 79% (204/258) Updating files: 80% (207/258) Updating files: 81% (209/258) Updating files: 82% (212/258) Updating files: 83% (215/258) Updating files: 84% (217/258) Updating files: 85% (220/258) Updating files: 86% (222/258) Updating files: 87% (225/258) Updating files: 88% (228/258) Updating files: 89% (230/258) Updating files: 90% (233/258) Updating files: 91% (235/258) Updating files: 92% (238/258) Updating files: 93% (240/258) Updating files: 94% (243/258) Updating files: 95% (246/258) Updating files: 96% (248/258) Updating files: 97% (251/258) Updating files: 98% (253/258) Updating files: 99% (256/258) Updating files: 100% (258/258) Updating files: 100% (258/258), done. 2023-10-30T13:52:26.964 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 321ecdaba2c v0.39 2023-10-30T13:52:26.967 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.39 2023-10-30T13:52:27.271 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 40 -eq 48 ] 2023-10-30T13:52:27.271 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.40 2023-10-30T13:52:27.271 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.40 2023-10-30T13:52:27.272 INFO:tasks.workunit.client.0.smithi046.stdout:v0.40 2023-10-30T13:52:27.272 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.40 2023-10-30T13:52:34.186 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 88% (199/226) Updating files: 89% (202/226) Updating files: 90% (204/226) Updating files: 91% (206/226) Updating files: 92% (208/226) Updating files: 93% (211/226) Updating files: 94% (213/226) Updating files: 95% (215/226) Updating files: 96% (217/226) Updating files: 97% (220/226) Updating files: 98% (222/226) Updating files: 99% (224/226) Updating files: 100% (226/226) Updating files: 100% (226/226), done. 2023-10-30T13:52:34.572 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 7eea40ea37f v0.40 2023-10-30T13:52:34.577 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.40 2023-10-30T13:52:35.402 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 41 -eq 48 ] 2023-10-30T13:52:35.402 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.41 2023-10-30T13:52:35.402 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.41 2023-10-30T13:52:35.403 INFO:tasks.workunit.client.0.smithi046.stdout:v0.41 2023-10-30T13:52:35.403 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.41 2023-10-30T13:52:36.838 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 43% (38/87) Updating files: 44% (39/87) Updating files: 45% (40/87) Updating files: 47% (41/87) Updating files: 48% (42/87) Updating files: 49% (43/87) Updating files: 50% (44/87) Updating files: 51% (45/87) Updating files: 52% (46/87) Updating files: 54% (47/87) Updating files: 55% (48/87) Updating files: 56% (49/87) Updating files: 57% (50/87) Updating files: 58% (51/87) Updating files: 59% (52/87) Updating files: 60% (53/87) Updating files: 62% (54/87) Updating files: 63% (55/87) Updating files: 64% (56/87) Updating files: 65% (57/87) Updating files: 66% (58/87) Updating files: 67% (59/87) Updating files: 68% (60/87) Updating files: 70% (61/87) Updating files: 71% (62/87) Updating files: 72% (63/87) Updating files: 73% (64/87) Updating files: 74% (65/87) Updating files: 75% (66/87) Updating files: 77% (67/87) Updating files: 78% (68/87) Updating files: 79% (69/87) Updating files: 80% (70/87) Updating files: 81% (71/87) Updating files: 82% (72/87) Updating files: 83% (73/87) Updating files: 85% (74/87) Updating files: 86% (75/87) Updating files: 87% (76/87) Updating files: 88% (77/87) Updating files: 89% (78/87) Updating files: 90% (79/87) Updating files: 91% (80/87) Updating files: 93% (81/87) Updating files: 94% (82/87) Updating files: 95% (83/87) Updating files: 96% (84/87) Updating files: 97% (85/87) Updating files: 98% (86/87) Updating files: 100% (87/87) Updating files: 100% (87/87), done. 2023-10-30T13:52:36.878 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at c1345f7136a v0.41 2023-10-30T13:52:36.881 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.41 2023-10-30T13:52:36.965 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 42 -eq 48 ] 2023-10-30T13:52:36.965 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.42 2023-10-30T13:52:36.966 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.42 2023-10-30T13:52:36.966 INFO:tasks.workunit.client.0.smithi046.stdout:v0.42 2023-10-30T13:52:36.966 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.42 2023-10-30T13:52:41.196 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 2% (6/250) Updating files: 3% (8/250) Updating files: 4% (10/250) Updating files: 5% (13/250) Updating files: 6% (15/250) Updating files: 7% (18/250) Updating files: 8% (20/250) Updating files: 9% (23/250) Updating files: 10% (25/250) Updating files: 11% (28/250) Updating files: 12% (30/250) Updating files: 13% (33/250) Updating files: 14% (35/250) Updating files: 15% (38/250) Updating files: 16% (40/250) Updating files: 17% (43/250) Updating files: 18% (45/250) Updating files: 19% (48/250) Updating files: 20% (50/250) Updating files: 21% (53/250) Updating files: 22% (55/250) Updating files: 23% (58/250) Updating files: 24% (60/250) Updating files: 25% (63/250) Updating files: 26% (65/250) Updating files: 27% (68/250) Updating files: 28% (70/250) Updating files: 29% (73/250) Updating files: 30% (75/250) Updating files: 31% (78/250) Updating files: 32% (80/250) Updating files: 33% (83/250) Updating files: 34% (85/250) Updating files: 35% (88/250) Updating files: 35% (89/250) Updating files: 36% (90/250) Updating files: 37% (93/250) Updating files: 38% (95/250) Updating files: 39% (98/250) Updating files: 40% (100/250) Updating files: 41% (103/250) Updating files: 42% (105/250) Updating files: 43% (108/250) Updating files: 44% (110/250) Updating files: 45% (113/250) Updating files: 46% (115/250) Updating files: 47% (118/250) Updating files: 48% (120/250) Updating files: 49% (123/250) Updating files: 50% (125/250) Updating files: 51% (128/250) Updating files: 52% (130/250) Updating files: 53% (133/250) Updating files: 53% (134/250) Updating files: 54% (135/250) Updating files: 55% (138/250) Updating files: 56% (140/250) Updating files: 57% (143/250) Updating files: 58% (145/250) Updating files: 59% (148/250) Updating files: 60% (150/250) Updating files: 61% (153/250) Updating files: 62% (155/250) Updating files: 63% (158/250) Updating files: 64% (160/250) Updating files: 65% (163/250) Updating files: 66% (165/250) Updating files: 67% (168/250) Updating files: 68% (170/250) Updating files: 69% (173/250) Updating files: 70% (175/250) Updating files: 71% (178/250) Updating files: 72% (180/250) Updating files: 73% (183/250) Updating files: 74% (185/250) Updating files: 75% (188/250) Updating files: 76% (190/250) Updating files: 77% (193/250) Updating files: 78% (195/250) Updating files: 79% (198/250) Updating files: 80% (200/250) Updating files: 81% (203/250) Updating files: 82% (205/250) Updating files: 83% (208/250) Updating files: 84% (210/250) Updating files: 85% (213/250) Updating files: 86% (215/250) Updating files: 87% (218/250) Updating files: 88% (220/250) Updating files: 89% (223/250) Updating files: 90% (225/250) Updating files: 91% (228/250) Updating files: 92% (230/250) Updating files: 93% (233/250) Updating files: 94% (235/250) Updating files: 95% (238/250) Updating files: 96% (240/250) Updating files: 97% (243/250) Updating files: 98% (245/250) Updating files: 99% (248/250) Updating files: 100% (250/250) Updating files: 100% (250/250), done. 2023-10-30T13:52:41.235 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at b205c64c687 v0.42 2023-10-30T13:52:41.239 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.42 2023-10-30T13:52:41.333 INFO:tasks.workunit.client.0.smithi046.stdout:v0.43 2023-10-30T13:52:41.333 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 43 -eq 48 ] 2023-10-30T13:52:41.333 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.43 2023-10-30T13:52:41.334 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.43 2023-10-30T13:52:41.334 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.43 2023-10-30T13:52:42.106 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 9fa8781c014 v0.43 2023-10-30T13:52:42.109 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.43 2023-10-30T13:52:42.446 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 44 -eq 48 ] 2023-10-30T13:52:42.446 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.44 2023-10-30T13:52:42.446 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.44 2023-10-30T13:52:42.447 INFO:tasks.workunit.client.0.smithi046.stdout:v0.44 2023-10-30T13:52:42.447 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.44 2023-10-30T13:52:42.562 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.711+0000 7f92f7501640 1 Processor -- start 2023-10-30T13:52:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.711+0000 7f92f7501640 1 -- start start 2023-10-30T13:52:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.711+0000 7f92f7501640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92f0069e90 0x7f92f015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.711+0000 7f92f7501640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f92f006cb80 0x7f92f006d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.711+0000 7f92f7501640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f006d550 con 0x7f92f006a270 2023-10-30T13:52:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.711+0000 7f92f7501640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f00c8570 con 0x7f92f006cb80 2023-10-30T13:52:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.711+0000 7f92f7501640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00c6c10 con 0x7f92f0069e90 2023-10-30T13:52:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.711+0000 7f92f5a9e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f92f006cb80 0x7f92f006d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.711+0000 7f92f5a9e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f92f006cb80 0x7f92f006d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40446/0 (socket says 172.21.15.46:40446) 2023-10-30T13:52:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.711+0000 7f92f5a9e640 1 -- 172.21.15.46:0/298936015 learned_addr learned my addr 172.21.15.46:0/298936015 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.711+0000 7f92f529d640 1 --2- 172.21.15.46:0/298936015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92f0069e90 0x7f92f015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f5a9e640 1 -- 172.21.15.46:0/298936015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92f0069e90 msgr2=0x7f92f015a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f5a9e640 1 --2- 172.21.15.46:0/298936015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92f0069e90 0x7f92f015a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f5a9e640 1 -- 172.21.15.46:0/298936015 >> v1:172.21.15.154:6789/0 conn(0x7f92f006a270 legacy=0x7f92f00c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f5a9e640 1 -- 172.21.15.46:0/298936015 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f00c6ef0 con 0x7f92f006cb80 2023-10-30T13:52:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f5a9e640 1 --2- 172.21.15.46:0/298936015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f006cb80 0x7f92f006d010 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009e50 tx=0x7f92e801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25766ce8edf76014 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92e7fff640 1 -- 172.21.15.46:0/298936015 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e80a5740 con 0x7f92f006cb80 2023-10-30T13:52:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92e7fff640 1 -- 172.21.15.46:0/298936015 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e80a58a0 con 0x7f92f006cb80 2023-10-30T13:52:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92e7fff640 1 -- 172.21.15.46:0/298936015 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e80a5c30 con 0x7f92f006cb80 2023-10-30T13:52:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f7501640 1 -- 172.21.15.46:0/298936015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f006cb80 msgr2=0x7f92f006d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f7501640 1 --2- 172.21.15.46:0/298936015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f006cb80 0x7f92f006d010 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009e50 tx=0x7f92e801c9d0 comp rx=0 tx=0).stop 2023-10-30T13:52:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f7501640 1 -- 172.21.15.46:0/298936015 shutdown_connections 2023-10-30T13:52:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f7501640 1 --2- 172.21.15.46:0/298936015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f006cb80 0x7f92f006d010 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f7501640 1 --2- 172.21.15.46:0/298936015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92f0069e90 0x7f92f015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f7501640 1 -- 172.21.15.46:0/298936015 >> 172.21.15.46:0/298936015 conn(0x7f92f006dfd0 msgr2=0x7f92f006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f7501640 1 -- 172.21.15.46:0/298936015 shutdown_connections 2023-10-30T13:52:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.715+0000 7f92f7501640 1 -- 172.21.15.46:0/298936015 wait complete. 2023-10-30T13:52:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f7501640 1 Processor -- start 2023-10-30T13:52:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f7501640 1 -- start start 2023-10-30T13:52:42.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f7501640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f0069e90 0x7f92f0151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:42.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f7501640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92f006a270 0x7f92f014a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:42.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f7501640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92f006cb80 0x7f92f014c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:42.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f7501640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f014d2a0 con 0x7f92f0069e90 2023-10-30T13:52:42.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f7501640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f01551f0 con 0x7f92f006a270 2023-10-30T13:52:42.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f7501640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f0072030 con 0x7f92f006cb80 2023-10-30T13:52:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f529d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f0069e90 0x7f92f0151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f529d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f0069e90 0x7f92f0151350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40462/0 (socket says 172.21.15.46:40462) 2023-10-30T13:52:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f529d640 1 -- 172.21.15.46:0/3321900389 learned_addr learned my addr 172.21.15.46:0/3321900389 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f5a9e640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92f006cb80 0x7f92f014c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f4a9c640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92f006a270 0x7f92f014a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f529d640 1 -- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92f006cb80 msgr2=0x7f92f014c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f529d640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92f006cb80 0x7f92f014c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f529d640 1 -- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92f006a270 msgr2=0x7f92f014a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f529d640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92f006a270 0x7f92f014a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f529d640 1 -- 172.21.15.46:0/3321900389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e8009b30 con 0x7f92f0069e90 2023-10-30T13:52:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f529d640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f0069e90 0x7f92f0151350 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00d800 tx=0x7f92ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92e5ffb640 1 -- 172.21.15.46:0/3321900389 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec00b6c0 con 0x7f92f0069e90 2023-10-30T13:52:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92e5ffb640 1 -- 172.21.15.46:0/3321900389 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec00b820 con 0x7f92f0069e90 2023-10-30T13:52:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92e5ffb640 1 -- 172.21.15.46:0/3321900389 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec00bb90 con 0x7f92f0069e90 2023-10-30T13:52:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f7501640 1 -- 172.21.15.46:0/3321900389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f014cfc0 con 0x7f92f0069e90 2023-10-30T13:52:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.719+0000 7f92f7501640 1 -- 172.21.15.46:0/3321900389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f0152ea0 con 0x7f92f0069e90 2023-10-30T13:52:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.723+0000 7f92f7501640 1 -- 172.21.15.46:0/3321900389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f006a690 con 0x7f92f0069e90 2023-10-30T13:52:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.723+0000 7f92e5ffb640 1 -- 172.21.15.46:0/3321900389 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f92ec00bcf0 con 0x7f92f0069e90 2023-10-30T13:52:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.723+0000 7f92e5ffb640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92d40664a0 0x7f92d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.723+0000 7f92e5ffb640 1 -- 172.21.15.46:0/3321900389 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f92ec0849a0 con 0x7f92f0069e90 2023-10-30T13:52:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.723+0000 7f92f4a9c640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92d40664a0 0x7f92d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.727+0000 7f92f4a9c640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92d40664a0 0x7f92d4068960 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f92f0149c00 tx=0x7f92e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.727+0000 7f92e5ffb640 1 -- 172.21.15.46:0/3321900389 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92f006a690 con 0x7f92f0069e90 2023-10-30T13:52:42.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 -- 172.21.15.46:0/3321900389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92f006a690 con 0x7f92f0069e90 2023-10-30T13:52:42.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92e5ffb640 1 -- 172.21.15.46:0/3321900389 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f92f006a690 con 0x7f92f0069e90 2023-10-30T13:52:42.925 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:42.925 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:33.434373+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:42.925 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 131 2023-10-30T13:52:42.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 -- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92d40664a0 msgr2=0x7f92d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:42.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92d40664a0 0x7f92d4068960 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f92f0149c00 tx=0x7f92e0009040 comp rx=0 tx=0).stop 2023-10-30T13:52:42.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 -- 172.21.15.46:0/3321900389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f0069e90 msgr2=0x7f92f0151350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:42.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f0069e90 0x7f92f0151350 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00d800 tx=0x7f92ec00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:52:42.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 -- 172.21.15.46:0/3321900389 shutdown_connections 2023-10-30T13:52:42.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92d40664a0 0x7f92d4068960 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:42.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92f006cb80 0x7f92f014c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:42.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92f006a270 0x7f92f014a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:42.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 --2- 172.21.15.46:0/3321900389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f0069e90 0x7f92f0151350 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:42.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 -- 172.21.15.46:0/3321900389 >> 172.21.15.46:0/3321900389 conn(0x7f92f006dfd0 msgr2=0x7f92f00c9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:42.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 -- 172.21.15.46:0/3321900389 shutdown_connections 2023-10-30T13:52:42.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:42.919+0000 7f92f7501640 1 -- 172.21.15.46:0/3321900389 wait complete. 2023-10-30T13:52:42.946 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=1) 2023-10-30T13:52:42.947 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:52:42.947 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:52:42.969 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:52:42.969 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe8a031d640 1 Processor -- start 2023-10-30T13:52:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe8a031d640 1 -- start start 2023-10-30T13:52:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe8a031d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89815d6f0 0x7fe89815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe8a031d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8980cb800 con 0x7fe8980cf160 2023-10-30T13:52:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe8a031d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8980cb330 con 0x7fe8981538a0 2023-10-30T13:52:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe8a031d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8980cad70 con 0x7fe89815d6f0 2023-10-30T13:52:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89e8ba640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe8980cf160 0x7fe898164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33166/0 (socket says 172.21.15.46:33166) 2023-10-30T13:52:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89e8ba640 1 -- 172.21.15.46:0/2726292221 learned_addr learned my addr 172.21.15.46:0/2726292221 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:52:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89d8b8640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89815d6f0 0x7fe89815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475471448 0 0) 0x7fe8980cb800 con 0x7fe8980cf160 2023-10-30T13:52:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe884003530 con 0x7fe8980cf160 2023-10-30T13:52:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649453349 0 0) 0x7fe8980cb330 con 0x7fe8981538a0 2023-10-30T13:52:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8980cb800 con 0x7fe8981538a0 2023-10-30T13:52:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2701947517 0 0) 0x7fe884003530 con 0x7fe8980cf160 2023-10-30T13:52:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89815d6f0 msgr2=0x7fe89815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89d0b7640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89815d6f0 0x7fe89815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 >> v1:172.21.15.154:6789/0 conn(0x7fe8981538a0 legacy=0x7fe8980ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980cb050 con 0x7fe8980cf160 2023-10-30T13:52:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fe890001a00 con 0x7fe8980cf160 2023-10-30T13:52:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.151+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 >> v1:172.21.15.46:6789/0 conn(0x7fe8980cf160 legacy=0x7fe898164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89d0b7640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe884003b40 0x7fe884003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89d0b7640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89815d6f0 0x7fe8840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89d0b7640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe884004d10 0x7fe884005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8980cad70 con 0x7fe884003b40 2023-10-30T13:52:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88400ba40 con 0x7fe884004d10 2023-10-30T13:52:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88400bc00 con 0x7fe89815d6f0 2023-10-30T13:52:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89e0b9640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe884003b40 0x7fe884003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89d8b8640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89815d6f0 0x7fe8840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89e8ba640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe884004d10 0x7fe884005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:43.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89e0b9640 1 -- 172.21.15.46:0/2726292221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89815d6f0 msgr2=0x7fe8840047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:43.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89e0b9640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe89815d6f0 0x7fe8840047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:43.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89e0b9640 1 -- 172.21.15.46:0/2726292221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe884004d10 msgr2=0x7fe884005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:43.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89e0b9640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe884004d10 0x7fe884005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:43.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89e0b9640 1 -- 172.21.15.46:0/2726292221 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe884003530 con 0x7fe884003b40 2023-10-30T13:52:43.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89e8ba640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe884004d10 0x7fe884005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:43.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89e0b9640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe884003b40 0x7fe884003f40 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fe8940018d0 tx=0x7fe89401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e559e9314e7bf18 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:43.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8940052d0 con 0x7fe884003b40 2023-10-30T13:52:43.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe894005450 con 0x7fe884003b40 2023-10-30T13:52:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.155+0000 7fe89d0b7640 1 -- 172.21.15.46:0/2726292221 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8940a8930 con 0x7fe884003b40 2023-10-30T13:52:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 -- 172.21.15.46:0/2726292221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe884003b40 msgr2=0x7fe884003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 --2- 172.21.15.46:0/2726292221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe884003b40 0x7fe884003f40 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fe8940018d0 tx=0x7fe89401ecd0 comp rx=0 tx=0).stop 2023-10-30T13:52:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe89e8ba640 1 -- 172.21.15.46:0/2726292221 reap_dead start 2023-10-30T13:52:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 -- 172.21.15.46:0/2726292221 shutdown_connections 2023-10-30T13:52:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 -- 172.21.15.46:0/2726292221 >> 172.21.15.46:0/2726292221 conn(0x7fe89806bfe0 msgr2=0x7fe89806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:43.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 -- 172.21.15.46:0/2726292221 shutdown_connections 2023-10-30T13:52:43.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 -- 172.21.15.46:0/2726292221 wait complete. 2023-10-30T13:52:43.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 Processor -- start 2023-10-30T13:52:43.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 -- start start 2023-10-30T13:52:43.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe884003b40 0x7fe8980ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8980b4110 0x7fe898158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89814f6c0 0x7fe89814fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8981f23e0 con 0x7fe89814f6c0 2023-10-30T13:52:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8981f2c80 con 0x7fe884003b40 2023-10-30T13:52:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe8a031d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8980ceb20 con 0x7fe8980b4110 2023-10-30T13:52:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe89e8ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89814f6c0 0x7fe89814fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe89e8ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89814f6c0 0x7fe89814fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40468/0 (socket says 172.21.15.46:40468) 2023-10-30T13:52:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe89e8ba640 1 -- 172.21.15.46:0/3677432399 learned_addr learned my addr 172.21.15.46:0/3677432399 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe89d8b8640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8980b4110 0x7fe898158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.159+0000 7fe89e0b9640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe884003b40 0x7fe8980ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe89e8ba640 1 -- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8980b4110 msgr2=0x7fe898158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe89e8ba640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8980b4110 0x7fe898158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe89e8ba640 1 -- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe884003b40 msgr2=0x7fe8980ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:43.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe89e8ba640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe884003b40 0x7fe8980ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:43.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe89e8ba640 1 -- 172.21.15.46:0/3677432399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980cede0 con 0x7fe89814f6c0 2023-10-30T13:52:43.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe89e8ba640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89814f6c0 0x7fe89814fb40 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fe890001340 tx=0x7fe890007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:43.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe88effd640 1 -- 172.21.15.46:0/3677432399 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89000eb00 con 0x7fe89814f6c0 2023-10-30T13:52:43.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe88effd640 1 -- 172.21.15.46:0/3677432399 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89000ec80 con 0x7fe89814f6c0 2023-10-30T13:52:43.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe88effd640 1 -- 172.21.15.46:0/3677432399 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890013420 con 0x7fe89814f6c0 2023-10-30T13:52:43.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe8a031d640 1 -- 172.21.15.46:0/3677432399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8981f2630 con 0x7fe89814f6c0 2023-10-30T13:52:43.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe8a031d640 1 -- 172.21.15.46:0/3677432399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe898066340 con 0x7fe89814f6c0 2023-10-30T13:52:43.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe8a031d640 1 -- 172.21.15.46:0/3677432399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89804e170 con 0x7fe89814f6c0 2023-10-30T13:52:43.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe88effd640 1 -- 172.21.15.46:0/3677432399 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8900135a0 con 0x7fe89814f6c0 2023-10-30T13:52:43.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe88effd640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:43.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe88effd640 1 -- 172.21.15.46:0/3677432399 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fe890084830 con 0x7fe89814f6c0 2023-10-30T13:52:43.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.163+0000 7fe89e0b9640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:43.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.167+0000 7fe89e0b9640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe87c0664a0 0x7fe87c068960 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe8940b0000 tx=0x7fe8940aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:43.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.167+0000 7fe88effd640 1 -- 172.21.15.46:0/3677432399 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe8900516c0 con 0x7fe89814f6c0 2023-10-30T13:52:43.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.359+0000 7fe8a031d640 1 -- 172.21.15.46:0/3677432399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8980b1dd0 con 0x7fe89814f6c0 2023-10-30T13:52:43.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.359+0000 7fe88effd640 1 -- 172.21.15.46:0/3677432399 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fe890054ce0 con 0x7fe89814f6c0 2023-10-30T13:52:43.366 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:43.366 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:33.434373+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 131 2023-10-30T13:52:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.363+0000 7fe8a031d640 1 -- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe87c0664a0 msgr2=0x7fe87c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.363+0000 7fe8a031d640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe87c0664a0 0x7fe87c068960 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe8940b0000 tx=0x7fe8940aa070 comp rx=0 tx=0).stop 2023-10-30T13:52:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.363+0000 7fe8a031d640 1 -- 172.21.15.46:0/3677432399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89814f6c0 msgr2=0x7fe89814fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.363+0000 7fe8a031d640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89814f6c0 0x7fe89814fb40 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fe890001340 tx=0x7fe890007bc0 comp rx=0 tx=0).stop 2023-10-30T13:52:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.363+0000 7fe8a031d640 1 -- 172.21.15.46:0/3677432399 shutdown_connections 2023-10-30T13:52:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.363+0000 7fe8a031d640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.363+0000 7fe8a031d640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89814f6c0 0x7fe89814fb40 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.363+0000 7fe8a031d640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8980b4110 0x7fe898158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.363+0000 7fe8a031d640 1 --2- 172.21.15.46:0/3677432399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe884003b40 0x7fe8980ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.363+0000 7fe8a031d640 1 -- 172.21.15.46:0/3677432399 >> 172.21.15.46:0/3677432399 conn(0x7fe89806bfe0 msgr2=0x7fe8980736b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.363+0000 7fe8a031d640 1 -- 172.21.15.46:0/3677432399 shutdown_connections 2023-10-30T13:52:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:43.363+0000 7fe8a031d640 1 -- 172.21.15.46:0/3677432399 wait complete. 2023-10-30T13:52:43.387 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:52:45.388 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa106565640 1 Processor -- start 2023-10-30T13:52:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa106565640 1 -- start start 2023-10-30T13:52:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa106565640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa10015d6f0 0x7fa10015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa106565640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa1000cf160 0x7fa1000cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa106565640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1000cb800 con 0x7fa1001538a0 2023-10-30T13:52:45.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa106565640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1000cb330 con 0x7fa10015d6f0 2023-10-30T13:52:45.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa106565640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1000cad70 con 0x7fa1000cf160 2023-10-30T13:52:45.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa0ff7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa10015d6f0 0x7fa10015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:45.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa0ff7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa10015d6f0 0x7fa10015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40478/0 (socket says 172.21.15.46:40478) 2023-10-30T13:52:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa0ff7fe640 1 -- 172.21.15.46:0/3044511479 learned_addr learned my addr 172.21.15.46:0/3044511479 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa104b02640 1 --2- 172.21.15.46:0/3044511479 >> v2:172.21.15.154:3300/0 conn(0x7fa1000cf160 0x7fa1000cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa0ff7fe640 1 -- 172.21.15.46:0/3044511479 >> v2:172.21.15.154:3300/0 conn(0x7fa1000cf160 msgr2=0x7fa1000cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa0ff7fe640 1 --2- 172.21.15.46:0/3044511479 >> v2:172.21.15.154:3300/0 conn(0x7fa1000cf160 0x7fa1000cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa0ff7fe640 1 -- 172.21.15.46:0/3044511479 >> v1:172.21.15.46:6789/0 conn(0x7fa1001538a0 legacy=0x7fa1000ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa0ff7fe640 1 -- 172.21.15.46:0/3044511479 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1000cb050 con 0x7fa10015d6f0 2023-10-30T13:52:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.539+0000 7fa0feffd640 1 -- 172.21.15.46:0/3044511479 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407085811 0 0) 0x7fa1000cb800 con 0x7fa1001538a0 2023-10-30T13:52:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.543+0000 7fa0ff7fe640 1 --2- 172.21.15.46:0/3044511479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10015d6f0 0x7fa10015a090 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa1000b5330 tx=0x7fa0f401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1330f6139a137179 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.543+0000 7fa0feffd640 1 -- 172.21.15.46:0/3044511479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f40a89f0 con 0x7fa10015d6f0 2023-10-30T13:52:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.543+0000 7fa0feffd640 1 -- 172.21.15.46:0/3044511479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f40a8b50 con 0x7fa10015d6f0 2023-10-30T13:52:45.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.543+0000 7fa0feffd640 1 -- 172.21.15.46:0/3044511479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f40b0440 con 0x7fa10015d6f0 2023-10-30T13:52:45.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.543+0000 7fa106565640 1 -- 172.21.15.46:0/3044511479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10015d6f0 msgr2=0x7fa10015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:45.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.543+0000 7fa106565640 1 --2- 172.21.15.46:0/3044511479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10015d6f0 0x7fa10015a090 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa1000b5330 tx=0x7fa0f401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:52:45.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.543+0000 7fa106565640 1 -- 172.21.15.46:0/3044511479 shutdown_connections 2023-10-30T13:52:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.543+0000 7fa106565640 1 --2- 172.21.15.46:0/3044511479 >> v2:172.21.15.154:3300/0 conn(0x7fa1000cf160 0x7fa1000cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.543+0000 7fa106565640 1 --2- 172.21.15.46:0/3044511479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10015d6f0 0x7fa10015a090 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.543+0000 7fa106565640 1 -- 172.21.15.46:0/3044511479 >> 172.21.15.46:0/3044511479 conn(0x7fa10006bfe0 msgr2=0x7fa10006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.543+0000 7fa106565640 1 -- 172.21.15.46:0/3044511479 shutdown_connections 2023-10-30T13:52:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.543+0000 7fa106565640 1 -- 172.21.15.46:0/3044511479 wait complete. 2023-10-30T13:52:45.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa106565640 1 Processor -- start 2023-10-30T13:52:45.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa106565640 1 -- start start 2023-10-30T13:52:45.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa106565640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1000cf160 0x7fa1001e5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:45.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa106565640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1001538a0 0x7fa100158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:45.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa106565640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa10015d6f0 0x7fa1000b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:45.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa106565640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10014c9d0 con 0x7fa1000cf160 2023-10-30T13:52:45.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa106565640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1001e42b0 con 0x7fa10015d6f0 2023-10-30T13:52:45.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa106565640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa100158620 con 0x7fa1001538a0 2023-10-30T13:52:45.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1000cf160 0x7fa1001e5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:45.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1000cf160 0x7fa1001e5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40488/0 (socket says 172.21.15.46:40488) 2023-10-30T13:52:45.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0fffff640 1 -- 172.21.15.46:0/2037979745 learned_addr learned my addr 172.21.15.46:0/2037979745 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:45.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0ff7fe640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1001538a0 0x7fa100158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:45.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa104b02640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa10015d6f0 0x7fa1000b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:45.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0fffff640 1 -- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1001538a0 msgr2=0x7fa100158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:45.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0fffff640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1001538a0 0x7fa100158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:45.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0fffff640 1 -- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa10015d6f0 msgr2=0x7fa1000b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:45.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0fffff640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa10015d6f0 0x7fa1000b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:45.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0fffff640 1 -- 172.21.15.46:0/2037979745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f40094d0 con 0x7fa1000cf160 2023-10-30T13:52:45.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0fffff640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1000cf160 0x7fa1001e5470 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000c4d0 tx=0x7fa0f000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:45.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0fcff9640 1 -- 172.21.15.46:0/2037979745 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0005480 con 0x7fa1000cf160 2023-10-30T13:52:45.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0fcff9640 1 -- 172.21.15.46:0/2037979745 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f0003600 con 0x7fa1000cf160 2023-10-30T13:52:45.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa0fcff9640 1 -- 172.21.15.46:0/2037979745 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f00039b0 con 0x7fa1000cf160 2023-10-30T13:52:45.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa106565640 1 -- 172.21.15.46:0/2037979745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1001588c0 con 0x7fa1000cf160 2023-10-30T13:52:45.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa106565640 1 -- 172.21.15.46:0/2037979745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1001e3750 con 0x7fa1000cf160 2023-10-30T13:52:45.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.547+0000 7fa106565640 1 -- 172.21.15.46:0/2037979745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1000b2a50 con 0x7fa1000cf160 2023-10-30T13:52:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.551+0000 7fa0fcff9640 1 -- 172.21.15.46:0/2037979745 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa0f0003b10 con 0x7fa1000cf160 2023-10-30T13:52:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.551+0000 7fa0fcff9640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0ec0664a0 0x7fa0ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.551+0000 7fa0fcff9640 1 -- 172.21.15.46:0/2037979745 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fa0f0083a50 con 0x7fa1000cf160 2023-10-30T13:52:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.551+0000 7fa0ff7fe640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0ec0664a0 0x7fa0ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.551+0000 7fa0ff7fe640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0ec0664a0 0x7fa0ec068960 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40b2000 tx=0x7fa0f40aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:45.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.555+0000 7fa0fcff9640 1 -- 172.21.15.46:0/2037979745 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa0f00509a0 con 0x7fa1000cf160 2023-10-30T13:52:45.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.747+0000 7fa106565640 1 -- 172.21.15.46:0/2037979745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa10004e170 con 0x7fa1000cf160 2023-10-30T13:52:45.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.747+0000 7fa0fcff9640 1 -- 172.21.15.46:0/2037979745 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fa0f0017020 con 0x7fa1000cf160 2023-10-30T13:52:45.751 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:45.751 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:33.434373+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:45.751 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 131 2023-10-30T13:52:45.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.751+0000 7fa106565640 1 -- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0ec0664a0 msgr2=0x7fa0ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:45.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.751+0000 7fa106565640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0ec0664a0 0x7fa0ec068960 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40b2000 tx=0x7fa0f40aa070 comp rx=0 tx=0).stop 2023-10-30T13:52:45.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.751+0000 7fa106565640 1 -- 172.21.15.46:0/2037979745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1000cf160 msgr2=0x7fa1001e5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:45.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.751+0000 7fa106565640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1000cf160 0x7fa1001e5470 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000c4d0 tx=0x7fa0f000c990 comp rx=0 tx=0).stop 2023-10-30T13:52:45.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.751+0000 7fa106565640 1 -- 172.21.15.46:0/2037979745 shutdown_connections 2023-10-30T13:52:45.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.751+0000 7fa106565640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0ec0664a0 0x7fa0ec068960 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:45.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.751+0000 7fa106565640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa10015d6f0 0x7fa1000b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:45.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.751+0000 7fa106565640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1001538a0 0x7fa100158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:45.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.751+0000 7fa106565640 1 --2- 172.21.15.46:0/2037979745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1000cf160 0x7fa1001e5470 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:45.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.751+0000 7fa106565640 1 -- 172.21.15.46:0/2037979745 >> 172.21.15.46:0/2037979745 conn(0x7fa10006bfe0 msgr2=0x7fa10006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:45.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.751+0000 7fa106565640 1 -- 172.21.15.46:0/2037979745 shutdown_connections 2023-10-30T13:52:45.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:45.751+0000 7fa106565640 1 -- 172.21.15.46:0/2037979745 wait complete. 2023-10-30T13:52:45.772 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:52:47.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:47.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.923+0000 7f43e7ec9640 1 Processor -- start 2023-10-30T13:52:47.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.923+0000 7f43e7ec9640 1 -- start start 2023-10-30T13:52:47.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e7ec9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43e01580e0 0x7f43e0070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:47.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e7ec9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43e00c6540 0x7f43e015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:47.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e7ec9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f43e0069e90 0x7f43e006a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:47.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e7ec9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e00c5810 con 0x7f43e01580e0 2023-10-30T13:52:47.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e7ec9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e00c6960 con 0x7f43e0069e90 2023-10-30T13:52:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e7ec9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e00c6bb0 con 0x7f43e00c6540 2023-10-30T13:52:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e5c65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43e01580e0 0x7f43e0070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e5c65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43e01580e0 0x7f43e0070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58758/0 (socket says 172.21.15.46:58758) 2023-10-30T13:52:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e5c65640 1 -- 172.21.15.46:0/1986464358 learned_addr learned my addr 172.21.15.46:0/1986464358 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e5464640 1 --2- 172.21.15.46:0/1986464358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43e00c6540 0x7f43e015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:47.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e6466640 1 --2- 172.21.15.46:0/1986464358 >> v2:172.21.15.154:3300/0 conn(0x7f43e0069e90 0x7f43e006a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:47.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e5c65640 1 -- 172.21.15.46:0/1986464358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43e00c6540 msgr2=0x7f43e015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:47.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e5c65640 1 --2- 172.21.15.46:0/1986464358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43e00c6540 0x7f43e015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:47.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e5c65640 1 -- 172.21.15.46:0/1986464358 >> v2:172.21.15.154:3300/0 conn(0x7f43e0069e90 msgr2=0x7f43e006a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:47.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e5c65640 1 --2- 172.21.15.46:0/1986464358 >> v2:172.21.15.154:3300/0 conn(0x7f43e0069e90 0x7f43e006a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:47.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.927+0000 7f43e5c65640 1 -- 172.21.15.46:0/1986464358 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e00c6e70 con 0x7f43e01580e0 2023-10-30T13:52:47.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e5c65640 1 --2- 172.21.15.46:0/1986464358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43e01580e0 0x7f43e0070740 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f43e00c54b0 tx=0x7f43dc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f5392c4b27132c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:47.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e4c63640 1 -- 172.21.15.46:0/1986464358 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc0ac630 con 0x7f43e01580e0 2023-10-30T13:52:47.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e4c63640 1 -- 172.21.15.46:0/1986464358 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43dc0ac790 con 0x7f43e01580e0 2023-10-30T13:52:47.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e4c63640 1 -- 172.21.15.46:0/1986464358 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc0acb20 con 0x7f43e01580e0 2023-10-30T13:52:47.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e7ec9640 1 -- 172.21.15.46:0/1986464358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43e01580e0 msgr2=0x7f43e0070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:47.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e7ec9640 1 --2- 172.21.15.46:0/1986464358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43e01580e0 0x7f43e0070740 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f43e00c54b0 tx=0x7f43dc01c500 comp rx=0 tx=0).stop 2023-10-30T13:52:47.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e7ec9640 1 -- 172.21.15.46:0/1986464358 shutdown_connections 2023-10-30T13:52:47.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e7ec9640 1 --2- 172.21.15.46:0/1986464358 >> v2:172.21.15.154:3300/0 conn(0x7f43e0069e90 0x7f43e006a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e7ec9640 1 --2- 172.21.15.46:0/1986464358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43e00c6540 0x7f43e015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e7ec9640 1 --2- 172.21.15.46:0/1986464358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43e01580e0 0x7f43e0070740 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e7ec9640 1 -- 172.21.15.46:0/1986464358 >> 172.21.15.46:0/1986464358 conn(0x7f43e00c7910 msgr2=0x7f43e00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e7ec9640 1 -- 172.21.15.46:0/1986464358 shutdown_connections 2023-10-30T13:52:47.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e7ec9640 1 -- 172.21.15.46:0/1986464358 wait complete. 2023-10-30T13:52:47.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e7ec9640 1 Processor -- start 2023-10-30T13:52:47.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.931+0000 7f43e7ec9640 1 -- start start 2023-10-30T13:52:47.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e7ec9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43e0069e90 0x7f43e00c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:47.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e7ec9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e00c6540 0x7f43e0155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:47.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e7ec9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43e01580e0 0x7f43e01561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:47.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e7ec9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e01578d0 con 0x7f43e01580e0 2023-10-30T13:52:47.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e7ec9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e01f6710 con 0x7f43e00c6540 2023-10-30T13:52:47.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e7ec9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e01f69d0 con 0x7f43e0069e90 2023-10-30T13:52:47.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e6466640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43e01580e0 0x7f43e01561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:47.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e6466640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43e01580e0 0x7f43e01561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58768/0 (socket says 172.21.15.46:58768) 2023-10-30T13:52:47.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e6466640 1 -- 172.21.15.46:0/3697769058 learned_addr learned my addr 172.21.15.46:0/3697769058 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:47.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e5c65640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43e0069e90 0x7f43e00c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:47.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e5464640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e00c6540 0x7f43e0155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:47.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e6466640 1 -- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43e0069e90 msgr2=0x7f43e00c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:47.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e6466640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43e0069e90 0x7f43e00c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:47.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e6466640 1 -- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e00c6540 msgr2=0x7f43e0155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:47.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e6466640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e00c6540 0x7f43e0155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:47.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e6466640 1 -- 172.21.15.46:0/3697769058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43dc0094d0 con 0x7f43e01580e0 2023-10-30T13:52:47.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e6466640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43e01580e0 0x7f43e01561f0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f43d8007e60 tx=0x7f43d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:47.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43d67fc640 1 -- 172.21.15.46:0/3697769058 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d8014620 con 0x7f43e01580e0 2023-10-30T13:52:47.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43d67fc640 1 -- 172.21.15.46:0/3697769058 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d800ee90 con 0x7f43e01580e0 2023-10-30T13:52:47.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43d67fc640 1 -- 172.21.15.46:0/3697769058 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d80149e0 con 0x7f43e01580e0 2023-10-30T13:52:47.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e7ec9640 1 -- 172.21.15.46:0/3697769058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e01f6cb0 con 0x7f43e01580e0 2023-10-30T13:52:47.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e7ec9640 1 -- 172.21.15.46:0/3697769058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e01f7270 con 0x7f43e01580e0 2023-10-30T13:52:47.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43e7ec9640 1 -- 172.21.15.46:0/3697769058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e004e170 con 0x7f43e01580e0 2023-10-30T13:52:47.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.935+0000 7f43d67fc640 1 -- 172.21.15.46:0/3697769058 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f43d8014b40 con 0x7f43e01580e0 2023-10-30T13:52:47.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.939+0000 7f43d67fc640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43c40664a0 0x7f43c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:47.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.939+0000 7f43d67fc640 1 -- 172.21.15.46:0/3697769058 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f43d8084db0 con 0x7f43e01580e0 2023-10-30T13:52:47.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.939+0000 7f43e5c65640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43c40664a0 0x7f43c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:47.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.939+0000 7f43e5c65640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43c40664a0 0x7f43c4068960 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0ae000 tx=0x7f43dc0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:47.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:47.943+0000 7f43d67fc640 1 -- 172.21.15.46:0/3697769058 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f43d8089070 con 0x7f43e01580e0 2023-10-30T13:52:48.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.143+0000 7f43e7ec9640 1 -- 172.21.15.46:0/3697769058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43e01683e0 con 0x7f43e01580e0 2023-10-30T13:52:48.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43d67fc640 1 -- 172.21.15.46:0/3697769058 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f43d8051c40 con 0x7f43e01580e0 2023-10-30T13:52:48.148 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:48.149 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:33.434373+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:48.150 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 131 2023-10-30T13:52:48.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43e7ec9640 1 -- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43c40664a0 msgr2=0x7f43c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:48.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43e7ec9640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43c40664a0 0x7f43c4068960 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0ae000 tx=0x7f43dc0a7000 comp rx=0 tx=0).stop 2023-10-30T13:52:48.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43e7ec9640 1 -- 172.21.15.46:0/3697769058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43e01580e0 msgr2=0x7f43e01561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:48.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43e7ec9640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43e01580e0 0x7f43e01561f0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f43d8007e60 tx=0x7f43d800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:52:48.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43e7ec9640 1 -- 172.21.15.46:0/3697769058 shutdown_connections 2023-10-30T13:52:48.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43e7ec9640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43c40664a0 0x7f43c4068960 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:48.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43e7ec9640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43e01580e0 0x7f43e01561f0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:48.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43e7ec9640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e00c6540 0x7f43e0155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:48.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43e7ec9640 1 --2- 172.21.15.46:0/3697769058 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43e0069e90 0x7f43e00c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:48.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43e7ec9640 1 -- 172.21.15.46:0/3697769058 >> 172.21.15.46:0/3697769058 conn(0x7f43e00c7910 msgr2=0x7f43e014dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:48.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43e7ec9640 1 -- 172.21.15.46:0/3697769058 shutdown_connections 2023-10-30T13:52:48.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:48.147+0000 7f43e7ec9640 1 -- 172.21.15.46:0/3697769058 wait complete. 2023-10-30T13:52:48.169 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:52:50.170 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:50.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae3c79640 1 Processor -- start 2023-10-30T13:52:50.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae3c79640 1 -- start start 2023-10-30T13:52:50.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae3c79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faadc0cf160 0x7faadc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:50.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae3c79640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faadc0cb800 con 0x7faadc1538a0 2023-10-30T13:52:50.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae3c79640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faadc0cb330 con 0x7faadc15d6f0 2023-10-30T13:52:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae3c79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faadc0cad70 con 0x7faadc0cf160 2023-10-30T13:52:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae1a15640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faadc1538a0 0x7faadc0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34278/0 (socket says 172.21.15.46:34278) 2023-10-30T13:52:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae1a15640 1 -- 172.21.15.46:0/1120671550 learned_addr learned my addr 172.21.15.46:0/1120671550 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:52:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae2216640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faadc0cf160 0x7faadc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657774797 0 0) 0x7faadc0cb800 con 0x7faadc1538a0 2023-10-30T13:52:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faac8003530 con 0x7faadc1538a0 2023-10-30T13:52:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1468542574 0 0) 0x7faac8003530 con 0x7faadc1538a0 2023-10-30T13:52:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faadc0cf160 msgr2=0x7faadc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faadc0cf160 0x7faadc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 >> v1:172.21.15.154:6789/0 conn(0x7faadc15d6f0 legacy=0x7faadc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc0cb050 con 0x7faadc1538a0 2023-10-30T13:52:50.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7faad8001680 con 0x7faadc1538a0 2023-10-30T13:52:50.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 >> v1:172.21.15.46:6789/0 conn(0x7faadc1538a0 legacy=0x7faadc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:50.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faac80039b0 0x7faac8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:50.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faac80047d0 0x7faac8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:50.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faadc0cf160 0x7faac8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:50.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc0cad70 con 0x7faac80039b0 2023-10-30T13:52:50.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac800bad0 con 0x7faac80047d0 2023-10-30T13:52:50.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac800bc90 con 0x7faadc0cf160 2023-10-30T13:52:50.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae1a15640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faac80039b0 0x7faac8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:50.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae1214640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faac80047d0 0x7faac8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:50.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae2216640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faadc0cf160 0x7faac8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:50.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae1a15640 1 -- 172.21.15.46:0/1120671550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faadc0cf160 msgr2=0x7faac8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:50.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae1a15640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faadc0cf160 0x7faac8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:50.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae1a15640 1 -- 172.21.15.46:0/1120671550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faac80047d0 msgr2=0x7faac8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:50.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae1a15640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faac80047d0 0x7faac8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:50.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae1a15640 1 -- 172.21.15.46:0/1120671550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac8003530 con 0x7faac80039b0 2023-10-30T13:52:50.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.315+0000 7faae1214640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faac80047d0 0x7faac8004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:50.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.319+0000 7faae1a15640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faac80039b0 0x7faac8003dd0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7faad800c210 tx=0x7faad801f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9990ca6a7b716b59 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:50.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.319+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8005ae0 con 0x7faac80039b0 2023-10-30T13:52:50.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.319+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad80ab070 con 0x7faac80039b0 2023-10-30T13:52:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.319+0000 7faae0a13640 1 -- 172.21.15.46:0/1120671550 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8007070 con 0x7faac80039b0 2023-10-30T13:52:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.319+0000 7faae3c79640 1 -- 172.21.15.46:0/1120671550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faac80039b0 msgr2=0x7faac8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.319+0000 7faae3c79640 1 --2- 172.21.15.46:0/1120671550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faac80039b0 0x7faac8003dd0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7faad800c210 tx=0x7faad801f350 comp rx=0 tx=0).stop 2023-10-30T13:52:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.319+0000 7faae2216640 1 -- 172.21.15.46:0/1120671550 reap_dead start 2023-10-30T13:52:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.319+0000 7faae3c79640 1 -- 172.21.15.46:0/1120671550 shutdown_connections 2023-10-30T13:52:50.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.319+0000 7faae3c79640 1 -- 172.21.15.46:0/1120671550 >> 172.21.15.46:0/1120671550 conn(0x7faadc06bfe0 msgr2=0x7faadc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:50.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.319+0000 7faae3c79640 1 -- 172.21.15.46:0/1120671550 shutdown_connections 2023-10-30T13:52:50.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.319+0000 7faae3c79640 1 -- 172.21.15.46:0/1120671550 wait complete. 2023-10-30T13:52:50.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae3c79640 1 Processor -- start 2023-10-30T13:52:50.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae3c79640 1 -- start start 2023-10-30T13:52:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae3c79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faac80039b0 0x7faadc1440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae3c79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faadc1eea10 0x7faadc1eee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae3c79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faadc15b2c0 0x7faadc15b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae3c79640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc1f2910 con 0x7faadc15b2c0 2023-10-30T13:52:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae3c79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc0b4fa0 con 0x7faadc1eea10 2023-10-30T13:52:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae3c79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faadc1ef3b0 con 0x7faac80039b0 2023-10-30T13:52:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae2216640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faadc15b2c0 0x7faadc15b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae2216640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faadc15b2c0 0x7faadc15b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58780/0 (socket says 172.21.15.46:58780) 2023-10-30T13:52:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae2216640 1 -- 172.21.15.46:0/3939624647 learned_addr learned my addr 172.21.15.46:0/3939624647 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae1214640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faadc1eea10 0x7faadc1eee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:50.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae1a15640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faac80039b0 0x7faadc1440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:50.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae2216640 1 -- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faac80039b0 msgr2=0x7faadc1440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:50.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae2216640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faac80039b0 0x7faadc1440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:50.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae2216640 1 -- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faadc1eea10 msgr2=0x7faadc1eee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:50.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae2216640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faadc1eea10 0x7faadc1eee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae2216640 1 -- 172.21.15.46:0/3939624647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc1f2e40 con 0x7faadc15b2c0 2023-10-30T13:52:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.323+0000 7faae2216640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faadc15b2c0 0x7faadc15b740 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7faadc072ce0 tx=0x7faad40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.327+0000 7faad27fc640 1 -- 172.21.15.46:0/3939624647 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad400e070 con 0x7faadc15b2c0 2023-10-30T13:52:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.327+0000 7faad27fc640 1 -- 172.21.15.46:0/3939624647 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad4011de0 con 0x7faadc15b2c0 2023-10-30T13:52:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.327+0000 7faad27fc640 1 -- 172.21.15.46:0/3939624647 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad40039a0 con 0x7faadc15b2c0 2023-10-30T13:52:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.327+0000 7faae3c79640 1 -- 172.21.15.46:0/3939624647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc0b32b0 con 0x7faadc15b2c0 2023-10-30T13:52:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.327+0000 7faae3c79640 1 -- 172.21.15.46:0/3939624647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc0b4c30 con 0x7faadc15b2c0 2023-10-30T13:52:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.327+0000 7faae3c79640 1 -- 172.21.15.46:0/3939624647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc0b4dc0 con 0x7faadc15b2c0 2023-10-30T13:52:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.327+0000 7faad27fc640 1 -- 172.21.15.46:0/3939624647 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faad4003b00 con 0x7faadc15b2c0 2023-10-30T13:52:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.327+0000 7faad27fc640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faac00664a0 0x7faac0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.327+0000 7faad27fc640 1 -- 172.21.15.46:0/3939624647 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7faad4084ab0 con 0x7faadc15b2c0 2023-10-30T13:52:50.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.327+0000 7faae1a15640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faac00664a0 0x7faac0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:50.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.331+0000 7faae1a15640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faac00664a0 0x7faac0068960 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7faad80af000 tx=0x7faad80aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:50.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.331+0000 7faad27fc640 1 -- 172.21.15.46:0/3939624647 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faad40519f0 con 0x7faadc15b2c0 2023-10-30T13:52:50.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.523+0000 7faae3c79640 1 -- 172.21.15.46:0/3939624647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faadc04e170 con 0x7faadc15b2c0 2023-10-30T13:52:50.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.523+0000 7faad27fc640 1 -- 172.21.15.46:0/3939624647 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6523 (secure 0 0 0) 0x7faad4055010 con 0x7faadc15b2c0 2023-10-30T13:52:50.526 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:50.526 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:33.434373+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:50.526 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 131 2023-10-30T13:52:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.527+0000 7faae3c79640 1 -- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faac00664a0 msgr2=0x7faac0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.527+0000 7faae3c79640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faac00664a0 0x7faac0068960 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7faad80af000 tx=0x7faad80aa000 comp rx=0 tx=0).stop 2023-10-30T13:52:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.527+0000 7faae3c79640 1 -- 172.21.15.46:0/3939624647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faadc15b2c0 msgr2=0x7faadc15b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.527+0000 7faae3c79640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faadc15b2c0 0x7faadc15b740 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7faadc072ce0 tx=0x7faad40115a0 comp rx=0 tx=0).stop 2023-10-30T13:52:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.527+0000 7faae3c79640 1 -- 172.21.15.46:0/3939624647 shutdown_connections 2023-10-30T13:52:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.527+0000 7faae3c79640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faac00664a0 0x7faac0068960 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.527+0000 7faae3c79640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faadc15b2c0 0x7faadc15b740 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.527+0000 7faae3c79640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faadc1eea10 0x7faadc1eee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.527+0000 7faae3c79640 1 --2- 172.21.15.46:0/3939624647 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faac80039b0 0x7faadc1440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.527+0000 7faae3c79640 1 -- 172.21.15.46:0/3939624647 >> 172.21.15.46:0/3939624647 conn(0x7faadc06bfe0 msgr2=0x7faadc15aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.527+0000 7faae3c79640 1 -- 172.21.15.46:0/3939624647 shutdown_connections 2023-10-30T13:52:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:50.527+0000 7faae3c79640 1 -- 172.21.15.46:0/3939624647 wait complete. 2023-10-30T13:52:50.547 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:52:51.479 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:52:51.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:52:52.548 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:52.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.699+0000 7f9f7b046640 1 Processor -- start 2023-10-30T13:52:52.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.699+0000 7f9f7b046640 1 -- start start 2023-10-30T13:52:52.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f7b046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f74071110 0x7f9f7406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:52.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f7b046640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f74066cf0 con 0x7f9f7406f7a0 2023-10-30T13:52:52.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f7b046640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f740c70d0 con 0x7f9f740c6bd0 2023-10-30T13:52:52.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f7b046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f740700c0 con 0x7f9f74071110 2023-10-30T13:52:52.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f78de2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9f7406f7a0 0x7f9f74069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34294/0 (socket says 172.21.15.46:34294) 2023-10-30T13:52:52.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f78de2640 1 -- 172.21.15.46:0/1452601086 learned_addr learned my addr 172.21.15.46:0/1452601086 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:52:52.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f795e3640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f74071110 0x7f9f7406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957640979 0 0) 0x7f9f74066cf0 con 0x7f9f7406f7a0 2023-10-30T13:52:52.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f64003530 con 0x7f9f7406f7a0 2023-10-30T13:52:52.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996432721 0 0) 0x7f9f740c70d0 con 0x7f9f740c6bd0 2023-10-30T13:52:52.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f74066cf0 con 0x7f9f740c6bd0 2023-10-30T13:52:52.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1129446037 0 0) 0x7f9f64003530 con 0x7f9f7406f7a0 2023-10-30T13:52:52.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f74071110 msgr2=0x7f9f7406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:52.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f74071110 0x7f9f7406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:52.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 >> v1:172.21.15.154:6789/0 conn(0x7f9f740c6bd0 legacy=0x7f9f74158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:52.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f740c7390 con 0x7f9f7406f7a0 2023-10-30T13:52:52.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f9f68001680 con 0x7f9f7406f7a0 2023-10-30T13:52:52.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 >> v1:172.21.15.46:6789/0 conn(0x7f9f7406f7a0 legacy=0x7f9f74069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:52.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f74071110 0x7f9f64003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:52.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f640040f0 0x7f9f640047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:52.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f64004d10 0x7f9f64005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f740700c0 con 0x7f9f64004d10 2023-10-30T13:52:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6400bad0 con 0x7f9f640040f0 2023-10-30T13:52:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6400bc90 con 0x7f9f74071110 2023-10-30T13:52:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f795e3640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f64004d10 0x7f9f64005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:52.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f78de2640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f74071110 0x7f9f64003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:52.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f73fff640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f640040f0 0x7f9f640047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:52.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f795e3640 1 -- 172.21.15.46:0/1452601086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f74071110 msgr2=0x7f9f64003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:52.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f795e3640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f74071110 0x7f9f64003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:52.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f795e3640 1 -- 172.21.15.46:0/1452601086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f640040f0 msgr2=0x7f9f640047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:52.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f795e3640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f640040f0 0x7f9f640047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:52.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f795e3640 1 -- 172.21.15.46:0/1452601086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f64003530 con 0x7f9f64004d10 2023-10-30T13:52:52.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.703+0000 7f9f78de2640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f74071110 0x7f9f64003b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.707+0000 7f9f795e3640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f64004d10 0x7f9f64005190 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c009fc0 tx=0x7f9f6c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20cbb205b6faeb07 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.707+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c0a5c70 con 0x7f9f64004d10 2023-10-30T13:52:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.707+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c00f040 con 0x7f9f64004d10 2023-10-30T13:52:52.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.707+0000 7f9f737fe640 1 -- 172.21.15.46:0/1452601086 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c0ad440 con 0x7f9f64004d10 2023-10-30T13:52:52.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.707+0000 7f9f7b046640 1 -- 172.21.15.46:0/1452601086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f64004d10 msgr2=0x7f9f64005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:52.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.707+0000 7f9f7b046640 1 --2- 172.21.15.46:0/1452601086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f64004d10 0x7f9f64005190 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c009fc0 tx=0x7f9f6c01cc60 comp rx=0 tx=0).stop 2023-10-30T13:52:52.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.707+0000 7f9f795e3640 1 -- 172.21.15.46:0/1452601086 reap_dead start 2023-10-30T13:52:52.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.707+0000 7f9f7b046640 1 -- 172.21.15.46:0/1452601086 shutdown_connections 2023-10-30T13:52:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.707+0000 7f9f7b046640 1 -- 172.21.15.46:0/1452601086 >> 172.21.15.46:0/1452601086 conn(0x7f9f740c7910 msgr2=0x7f9f740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.707+0000 7f9f7b046640 1 -- 172.21.15.46:0/1452601086 shutdown_connections 2023-10-30T13:52:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.707+0000 7f9f7b046640 1 -- 172.21.15.46:0/1452601086 wait complete. 2023-10-30T13:52:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f7b046640 1 Processor -- start 2023-10-30T13:52:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f7b046640 1 -- start start 2023-10-30T13:52:52.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f7b046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f64004d10 0x7f9f740c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:52.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f7b046640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f740bfec0 0x7f9f7414f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:52.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f7b046640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f7414fe80 0x7f9f740c0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:52.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f7b046640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7414f520 con 0x7f9f740bfec0 2023-10-30T13:52:52.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f7b046640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f740c1f50 con 0x7f9f7414fe80 2023-10-30T13:52:52.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f7b046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7414ec10 con 0x7f9f64004d10 2023-10-30T13:52:52.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f73fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f740bfec0 0x7f9f7414f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:52.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f73fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f740bfec0 0x7f9f7414f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58798/0 (socket says 172.21.15.46:58798) 2023-10-30T13:52:52.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f73fff640 1 -- 172.21.15.46:0/3929591339 learned_addr learned my addr 172.21.15.46:0/3929591339 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:52.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f78de2640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f64004d10 0x7f9f740c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:52.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f795e3640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f7414fe80 0x7f9f740c0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:52.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f73fff640 1 -- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f64004d10 msgr2=0x7f9f740c1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:52.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f73fff640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f64004d10 0x7f9f740c1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:52.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f73fff640 1 -- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f7414fe80 msgr2=0x7f9f740c0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f73fff640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f7414fe80 0x7f9f740c0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f73fff640 1 -- 172.21.15.46:0/3929591339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f74150300 con 0x7f9f740bfec0 2023-10-30T13:52:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f73fff640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f740bfec0 0x7f9f7414f940 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f9f6000e600 tx=0x7f9f6000eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f717fa640 1 -- 172.21.15.46:0/3929591339 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f600052e0 con 0x7f9f740bfec0 2023-10-30T13:52:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f717fa640 1 -- 172.21.15.46:0/3929591339 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f60005460 con 0x7f9f740bfec0 2023-10-30T13:52:52.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f717fa640 1 -- 172.21.15.46:0/3929591339 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f60003840 con 0x7f9f740bfec0 2023-10-30T13:52:52.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f7b046640 1 -- 172.21.15.46:0/3929591339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f741ef460 con 0x7f9f740bfec0 2023-10-30T13:52:52.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f7b046640 1 -- 172.21.15.46:0/3929591339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7414ee60 con 0x7f9f740bfec0 2023-10-30T13:52:52.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.711+0000 7f9f7b046640 1 -- 172.21.15.46:0/3929591339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7414f660 con 0x7f9f740bfec0 2023-10-30T13:52:52.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.715+0000 7f9f717fa640 1 -- 172.21.15.46:0/3929591339 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9f600039a0 con 0x7f9f740bfec0 2023-10-30T13:52:52.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.715+0000 7f9f717fa640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9f580664d0 0x7f9f58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:52.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.715+0000 7f9f717fa640 1 -- 172.21.15.46:0/3929591339 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f9f600842d0 con 0x7f9f740bfec0 2023-10-30T13:52:52.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.715+0000 7f9f78de2640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9f580664d0 0x7f9f58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:52.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.715+0000 7f9f78de2640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9f580664d0 0x7f9f58068990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9f68001000 tx=0x7f9f68009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:52.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.719+0000 7f9f717fa640 1 -- 172.21.15.46:0/3929591339 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9f7414f660 con 0x7f9f740bfec0 2023-10-30T13:52:52.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.907+0000 7f9f7b046640 1 -- 172.21.15.46:0/3929591339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f7414f660 con 0x7f9f740bfec0 2023-10-30T13:52:52.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.907+0000 7f9f717fa640 1 -- 172.21.15.46:0/3929591339 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f9f7414f660 con 0x7f9f740bfec0 2023-10-30T13:52:52.911 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:52.911 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:33.434373+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:52.912 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 131 2023-10-30T13:52:52.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.911+0000 7f9f7b046640 1 -- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9f580664d0 msgr2=0x7f9f58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:52.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.911+0000 7f9f7b046640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9f580664d0 0x7f9f58068990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9f68001000 tx=0x7f9f68009040 comp rx=0 tx=0).stop 2023-10-30T13:52:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.911+0000 7f9f7b046640 1 -- 172.21.15.46:0/3929591339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f740bfec0 msgr2=0x7f9f7414f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.911+0000 7f9f7b046640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f740bfec0 0x7f9f7414f940 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f9f6000e600 tx=0x7f9f6000eac0 comp rx=0 tx=0).stop 2023-10-30T13:52:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.911+0000 7f9f7b046640 1 -- 172.21.15.46:0/3929591339 shutdown_connections 2023-10-30T13:52:52.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.911+0000 7f9f7b046640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9f580664d0 0x7f9f58068990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:52.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.911+0000 7f9f7b046640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f7414fe80 0x7f9f740c0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:52.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.911+0000 7f9f7b046640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f740bfec0 0x7f9f7414f940 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:52.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.911+0000 7f9f7b046640 1 --2- 172.21.15.46:0/3929591339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f64004d10 0x7f9f740c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:52.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.911+0000 7f9f7b046640 1 -- 172.21.15.46:0/3929591339 >> 172.21.15.46:0/3929591339 conn(0x7f9f740c7910 msgr2=0x7f9f740c8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:52.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.911+0000 7f9f7b046640 1 -- 172.21.15.46:0/3929591339 shutdown_connections 2023-10-30T13:52:52.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:52.911+0000 7f9f7b046640 1 -- 172.21.15.46:0/3929591339 wait complete. 2023-10-30T13:52:52.932 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:52:54.934 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.087+0000 7fc14c26b640 1 Processor -- start 2023-10-30T13:52:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.087+0000 7fc14c26b640 1 -- start start 2023-10-30T13:52:55.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.087+0000 7fc14c26b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc144153d70 0x7fc14406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:55.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.087+0000 7fc14c26b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc1440c8490 0x7fc1440c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.087+0000 7fc14c26b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc14406afa0 con 0x7fc14406e470 2023-10-30T13:52:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.087+0000 7fc14c26b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc14406d780 con 0x7fc144153d70 2023-10-30T13:52:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.087+0000 7fc14c26b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc144154150 con 0x7fc1440c8490 2023-10-30T13:52:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.087+0000 7fc14a007640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc144153d70 0x7fc14406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.087+0000 7fc149806640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc14406e470 0x7fc1440cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34306/0 (socket says 172.21.15.46:34306) 2023-10-30T13:52:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.087+0000 7fc149806640 1 -- 172.21.15.46:0/3000607065 learned_addr learned my addr 172.21.15.46:0/3000607065 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:52:55.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc14a808640 1 --2- 172.21.15.46:0/3000607065 >> v2:172.21.15.154:3300/0 conn(0x7fc1440c8490 0x7fc1440c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:55.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc14a007640 1 -- 172.21.15.46:0/3000607065 >> v2:172.21.15.154:3300/0 conn(0x7fc1440c8490 msgr2=0x7fc1440c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:55.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc14a007640 1 --2- 172.21.15.46:0/3000607065 >> v2:172.21.15.154:3300/0 conn(0x7fc1440c8490 0x7fc1440c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:55.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc14a007640 1 -- 172.21.15.46:0/3000607065 >> v1:172.21.15.46:6789/0 conn(0x7fc14406e470 legacy=0x7fc1440cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:55.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc14a007640 1 -- 172.21.15.46:0/3000607065 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14406e030 con 0x7fc144153d70 2023-10-30T13:52:55.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc149005640 1 -- 172.21.15.46:0/3000607065 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422547115 0 0) 0x7fc14406afa0 con 0x7fc14406e470 2023-10-30T13:52:55.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc14a808640 1 --2- 172.21.15.46:0/3000607065 >> v2:172.21.15.154:3300/0 conn(0x7fc1440c8490 0x7fc1440c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:52:55.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc14a007640 1 --2- 172.21.15.46:0/3000607065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc144153d70 0x7fc14406ca80 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fc1440cddf0 tx=0x7fc14001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54edb20a2efafa6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:55.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc149005640 1 -- 172.21.15.46:0/3000607065 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1400a7070 con 0x7fc144153d70 2023-10-30T13:52:55.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc149005640 1 -- 172.21.15.46:0/3000607065 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1400ab430 con 0x7fc144153d70 2023-10-30T13:52:55.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc149005640 1 -- 172.21.15.46:0/3000607065 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1400ab7c0 con 0x7fc144153d70 2023-10-30T13:52:55.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc14c26b640 1 -- 172.21.15.46:0/3000607065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc144153d70 msgr2=0x7fc14406ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:55.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.091+0000 7fc14c26b640 1 --2- 172.21.15.46:0/3000607065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc144153d70 0x7fc14406ca80 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fc1440cddf0 tx=0x7fc14001c430 comp rx=0 tx=0).stop 2023-10-30T13:52:55.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 -- 172.21.15.46:0/3000607065 shutdown_connections 2023-10-30T13:52:55.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 --2- 172.21.15.46:0/3000607065 >> v2:172.21.15.154:3300/0 conn(0x7fc1440c8490 0x7fc1440c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:55.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 --2- 172.21.15.46:0/3000607065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc144153d70 0x7fc14406ca80 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:55.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 -- 172.21.15.46:0/3000607065 >> 172.21.15.46:0/3000607065 conn(0x7fc144071ea0 msgr2=0x7fc144065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:55.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 -- 172.21.15.46:0/3000607065 shutdown_connections 2023-10-30T13:52:55.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 -- 172.21.15.46:0/3000607065 wait complete. 2023-10-30T13:52:55.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 Processor -- start 2023-10-30T13:52:55.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 -- start start 2023-10-30T13:52:55.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc14406e470 0x7fc1441524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:55.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1440c8490 0x7fc144141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:55.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc144153d70 0x7fc1440cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:55.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1441f67e0 con 0x7fc1440c8490 2023-10-30T13:52:55.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc144142420 con 0x7fc14406e470 2023-10-30T13:52:55.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14c26b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1441f6d00 con 0x7fc144153d70 2023-10-30T13:52:55.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc149806640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1440c8490 0x7fc144141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:55.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc149806640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1440c8490 0x7fc144141280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58816/0 (socket says 172.21.15.46:58816) 2023-10-30T13:52:55.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc149806640 1 -- 172.21.15.46:0/3856747869 learned_addr learned my addr 172.21.15.46:0/3856747869 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:55.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14a007640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc14406e470 0x7fc1441524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:55.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14a808640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc144153d70 0x7fc1440cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:55.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc149806640 1 -- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc144153d70 msgr2=0x7fc1440cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:55.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc149806640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc144153d70 0x7fc1440cbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:55.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc149806640 1 -- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc14406e470 msgr2=0x7fc1441524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:55.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc149806640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc14406e470 0x7fc1441524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:55.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc149806640 1 -- 172.21.15.46:0/3856747869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc140009420 con 0x7fc1440c8490 2023-10-30T13:52:55.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc14a808640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc144153d70 0x7fc1440cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:55.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.095+0000 7fc149806640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1440c8490 0x7fc144141280 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc1340011f0 tx=0x7fc13400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:55.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.099+0000 7fc13affd640 1 -- 172.21.15.46:0/3856747869 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1340074e0 con 0x7fc1440c8490 2023-10-30T13:52:55.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.099+0000 7fc13affd640 1 -- 172.21.15.46:0/3856747869 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc134007640 con 0x7fc1440c8490 2023-10-30T13:52:55.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.099+0000 7fc13affd640 1 -- 172.21.15.46:0/3856747869 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc134004a10 con 0x7fc1440c8490 2023-10-30T13:52:55.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.099+0000 7fc14c26b640 1 -- 172.21.15.46:0/3856747869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1441f2a30 con 0x7fc1440c8490 2023-10-30T13:52:55.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.099+0000 7fc14c26b640 1 -- 172.21.15.46:0/3856747869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1441f2cd0 con 0x7fc1440c8490 2023-10-30T13:52:55.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.099+0000 7fc14c26b640 1 -- 172.21.15.46:0/3856747869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc144067280 con 0x7fc1440c8490 2023-10-30T13:52:55.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.099+0000 7fc13affd640 1 -- 172.21.15.46:0/3856747869 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc134004b70 con 0x7fc1440c8490 2023-10-30T13:52:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.099+0000 7fc13affd640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc1280664d0 0x7fc128068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.099+0000 7fc13affd640 1 -- 172.21.15.46:0/3856747869 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fc1340835a0 con 0x7fc1440c8490 2023-10-30T13:52:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.099+0000 7fc14a007640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc1280664d0 0x7fc128068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:55.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.103+0000 7fc14a007640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc1280664d0 0x7fc128068990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc1400af000 tx=0x7fc1400a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:55.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.103+0000 7fc13affd640 1 -- 172.21.15.46:0/3856747869 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc1340504d0 con 0x7fc1440c8490 2023-10-30T13:52:55.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.291+0000 7fc14c26b640 1 -- 172.21.15.46:0/3856747869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1441f2890 con 0x7fc1440c8490 2023-10-30T13:52:55.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc13affd640 1 -- 172.21.15.46:0/3856747869 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fc1441f2890 con 0x7fc1440c8490 2023-10-30T13:52:55.296 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:55.296 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:33.434373+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:55.296 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 131 2023-10-30T13:52:55.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc14c26b640 1 -- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc1280664d0 msgr2=0x7fc128068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:55.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc14c26b640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc1280664d0 0x7fc128068990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc1400af000 tx=0x7fc1400a7000 comp rx=0 tx=0).stop 2023-10-30T13:52:55.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc14c26b640 1 -- 172.21.15.46:0/3856747869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1440c8490 msgr2=0x7fc144141280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:55.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc14c26b640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1440c8490 0x7fc144141280 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc1340011f0 tx=0x7fc13400c9b0 comp rx=0 tx=0).stop 2023-10-30T13:52:55.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc14c26b640 1 -- 172.21.15.46:0/3856747869 shutdown_connections 2023-10-30T13:52:55.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc14c26b640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc1280664d0 0x7fc128068990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:55.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc14c26b640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc144153d70 0x7fc1440cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:55.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc14c26b640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1440c8490 0x7fc144141280 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:55.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc14c26b640 1 --2- 172.21.15.46:0/3856747869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc14406e470 0x7fc1441524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:55.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc14c26b640 1 -- 172.21.15.46:0/3856747869 >> 172.21.15.46:0/3856747869 conn(0x7fc144071ea0 msgr2=0x7fc1440c78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:55.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc14c26b640 1 -- 172.21.15.46:0/3856747869 shutdown_connections 2023-10-30T13:52:55.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:55.295+0000 7fc14c26b640 1 -- 172.21.15.46:0/3856747869 wait complete. 2023-10-30T13:52:55.317 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:52:57.318 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:57.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.467+0000 7fca67417640 1 Processor -- start 2023-10-30T13:52:57.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca67417640 1 -- start start 2023-10-30T13:52:57.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca67417640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca60153bf0 0x7fca60071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:57.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca67417640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca600c5f60 0x7fca600cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:57.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca67417640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca600700b0 con 0x7fca600c6fe0 2023-10-30T13:52:57.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca67417640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca600c6810 con 0x7fca60153bf0 2023-10-30T13:52:57.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca67417640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca60159900 con 0x7fca600c5f60 2023-10-30T13:52:57.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca651b3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca60153bf0 0x7fca60071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:57.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca651b3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca60153bf0 0x7fca60071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54466/0 (socket says 172.21.15.46:54466) 2023-10-30T13:52:57.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca651b3640 1 -- 172.21.15.46:0/2214406747 learned_addr learned my addr 172.21.15.46:0/2214406747 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:57.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca659b4640 1 --2- 172.21.15.46:0/2214406747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca600c5f60 0x7fca600cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:57.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca651b3640 1 -- 172.21.15.46:0/2214406747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca600c5f60 msgr2=0x7fca600cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:57.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca651b3640 1 --2- 172.21.15.46:0/2214406747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca600c5f60 0x7fca600cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca651b3640 1 -- 172.21.15.46:0/2214406747 >> v1:172.21.15.46:6789/0 conn(0x7fca600c6fe0 legacy=0x7fca600c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:57.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.471+0000 7fca651b3640 1 -- 172.21.15.46:0/2214406747 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca600c6530 con 0x7fca60153bf0 2023-10-30T13:52:57.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca651b3640 1 --2- 172.21.15.46:0/2214406747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca60153bf0 0x7fca60071260 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009870 tx=0x7fca5c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b43cfa0ecd98bb68 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:57.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca57fff640 1 -- 172.21.15.46:0/2214406747 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c0a8a30 con 0x7fca60153bf0 2023-10-30T13:52:57.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca57fff640 1 -- 172.21.15.46:0/2214406747 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca5c0a8b90 con 0x7fca60153bf0 2023-10-30T13:52:57.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca57fff640 1 -- 172.21.15.46:0/2214406747 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c0b0440 con 0x7fca60153bf0 2023-10-30T13:52:57.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca67417640 1 -- 172.21.15.46:0/2214406747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca60153bf0 msgr2=0x7fca60071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:57.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca67417640 1 --2- 172.21.15.46:0/2214406747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca60153bf0 0x7fca60071260 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009870 tx=0x7fca5c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:52:57.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca67417640 1 -- 172.21.15.46:0/2214406747 shutdown_connections 2023-10-30T13:52:57.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca67417640 1 --2- 172.21.15.46:0/2214406747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca600c5f60 0x7fca600cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca67417640 1 --2- 172.21.15.46:0/2214406747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca60153bf0 0x7fca60071260 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca67417640 1 -- 172.21.15.46:0/2214406747 >> 172.21.15.46:0/2214406747 conn(0x7fca6006c6e0 msgr2=0x7fca60069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:57.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca67417640 1 -- 172.21.15.46:0/2214406747 shutdown_connections 2023-10-30T13:52:57.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca67417640 1 -- 172.21.15.46:0/2214406747 wait complete. 2023-10-30T13:52:57.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca67417640 1 Processor -- start 2023-10-30T13:52:57.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.475+0000 7fca67417640 1 -- start start 2023-10-30T13:52:57.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca67417640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca600c5f60 0x7fca6014f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:57.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca67417640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca600c6fe0 0x7fca6014ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:57.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca67417640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca60153bf0 0x7fca60152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:57.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca67417640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6014f240 con 0x7fca60153bf0 2023-10-30T13:52:57.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca67417640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca601efbf0 con 0x7fca600c6fe0 2023-10-30T13:52:57.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca67417640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6014df40 con 0x7fca600c5f60 2023-10-30T13:52:57.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca659b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca60153bf0 0x7fca60152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:57.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca659b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca60153bf0 0x7fca60152d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54468/0 (socket says 172.21.15.46:54468) 2023-10-30T13:52:57.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca659b4640 1 -- 172.21.15.46:0/2108264606 learned_addr learned my addr 172.21.15.46:0/2108264606 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:57.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca651b3640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca600c5f60 0x7fca6014f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:57.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca649b2640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca600c6fe0 0x7fca6014ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:57.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca659b4640 1 -- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca600c5f60 msgr2=0x7fca6014f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:57.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca659b4640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca600c5f60 0x7fca6014f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca659b4640 1 -- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca600c6fe0 msgr2=0x7fca6014ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:57.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca659b4640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca600c6fe0 0x7fca6014ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca659b4640 1 -- 172.21.15.46:0/2108264606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c0094d0 con 0x7fca60153bf0 2023-10-30T13:52:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca649b2640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca600c6fe0 0x7fca6014ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca659b4640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca60153bf0 0x7fca60152d30 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fca58007e60 tx=0x7fca5800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca55ffb640 1 -- 172.21.15.46:0/2108264606 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca58014420 con 0x7fca60153bf0 2023-10-30T13:52:57.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca55ffb640 1 -- 172.21.15.46:0/2108264606 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca580145a0 con 0x7fca60153bf0 2023-10-30T13:52:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca55ffb640 1 -- 172.21.15.46:0/2108264606 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca58014950 con 0x7fca60153bf0 2023-10-30T13:52:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca67417640 1 -- 172.21.15.46:0/2108264606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca6014e220 con 0x7fca60153bf0 2023-10-30T13:52:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca67417640 1 -- 172.21.15.46:0/2108264606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6014e7e0 con 0x7fca60153bf0 2023-10-30T13:52:57.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.479+0000 7fca67417640 1 -- 172.21.15.46:0/2108264606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6004e170 con 0x7fca60153bf0 2023-10-30T13:52:57.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.483+0000 7fca55ffb640 1 -- 172.21.15.46:0/2108264606 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fca58014ab0 con 0x7fca60153bf0 2023-10-30T13:52:57.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.483+0000 7fca55ffb640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca440664a0 0x7fca44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:57.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.483+0000 7fca55ffb640 1 -- 172.21.15.46:0/2108264606 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fca58084d40 con 0x7fca60153bf0 2023-10-30T13:52:57.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.483+0000 7fca651b3640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca440664a0 0x7fca44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:57.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.483+0000 7fca651b3640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca440664a0 0x7fca44068960 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fca5c0b2000 tx=0x7fca5c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:57.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.487+0000 7fca55ffb640 1 -- 172.21.15.46:0/2108264606 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fca58051bd0 con 0x7fca60153bf0 2023-10-30T13:52:57.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.675+0000 7fca67417640 1 -- 172.21.15.46:0/2108264606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca6006fe90 con 0x7fca60153bf0 2023-10-30T13:52:57.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca55ffb640 1 -- 172.21.15.46:0/2108264606 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fca580551f0 con 0x7fca60153bf0 2023-10-30T13:52:57.680 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:52:57.680 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:33.434373+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":6858},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6858":{"gid":6858,"name":"e","rank":1,"incarnation":117,"state":"up:active","state_seq":41,"addr":"172.21.15.46:6837/2572113428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2572113428},{"type":"v1","addr":"172.21.15.46:6837","nonce":2572113428}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:52:57.681 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 131 2023-10-30T13:52:57.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca67417640 1 -- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca440664a0 msgr2=0x7fca44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:57.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca67417640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca440664a0 0x7fca44068960 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fca5c0b2000 tx=0x7fca5c008040 comp rx=0 tx=0).stop 2023-10-30T13:52:57.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca67417640 1 -- 172.21.15.46:0/2108264606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca60153bf0 msgr2=0x7fca60152d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:57.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca67417640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca60153bf0 0x7fca60152d30 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fca58007e60 tx=0x7fca5800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca67417640 1 -- 172.21.15.46:0/2108264606 shutdown_connections 2023-10-30T13:52:57.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca67417640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca440664a0 0x7fca44068960 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca67417640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca60153bf0 0x7fca60152d30 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca67417640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca600c6fe0 0x7fca6014ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca67417640 1 --2- 172.21.15.46:0/2108264606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca600c5f60 0x7fca6014f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca67417640 1 -- 172.21.15.46:0/2108264606 >> 172.21.15.46:0/2108264606 conn(0x7fca6006c6e0 msgr2=0x7fca60158e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:57.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca67417640 1 -- 172.21.15.46:0/2108264606 shutdown_connections 2023-10-30T13:52:57.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:57.679+0000 7fca67417640 1 -- 172.21.15.46:0/2108264606 wait complete. 2023-10-30T13:52:57.701 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:52:59.702 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:52:59.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.855+0000 7f76160c9640 1 Processor -- start 2023-10-30T13:52:59.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.855+0000 7f76160c9640 1 -- start start 2023-10-30T13:52:59.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.855+0000 7f76160c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f76100c6fe0 0x7f7610071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:59.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.855+0000 7f76160c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76100c5f60 0x7f76100cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:59.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f76160c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100700b0 con 0x7f7610153bf0 2023-10-30T13:52:59.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f76160c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76100c6810 con 0x7f76100c5f60 2023-10-30T13:52:59.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f76160c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7610159900 con 0x7f76100c6fe0 2023-10-30T13:52:59.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f760ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76100c5f60 0x7f76100cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:59.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f760ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76100c5f60 0x7f76100cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54482/0 (socket says 172.21.15.46:54482) 2023-10-30T13:52:59.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f760ffff640 1 -- 172.21.15.46:0/684744020 learned_addr learned my addr 172.21.15.46:0/684744020 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:59.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f760effd640 1 --2- 172.21.15.46:0/684744020 >> v2:172.21.15.154:3300/0 conn(0x7f76100c6fe0 0x7f7610071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:59.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f760ffff640 1 -- 172.21.15.46:0/684744020 >> v2:172.21.15.154:3300/0 conn(0x7f76100c6fe0 msgr2=0x7f7610071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:59.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f760ffff640 1 --2- 172.21.15.46:0/684744020 >> v2:172.21.15.154:3300/0 conn(0x7f76100c6fe0 0x7f7610071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:59.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f760ffff640 1 -- 172.21.15.46:0/684744020 >> v1:172.21.15.154:6789/0 conn(0x7f7610153bf0 legacy=0x7f76100c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:59.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f760ffff640 1 -- 172.21.15.46:0/684744020 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76100c6530 con 0x7f76100c5f60 2023-10-30T13:52:59.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f760ffff640 1 --2- 172.21.15.46:0/684744020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76100c5f60 0x7f76100cb570 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f76100723d0 tx=0x7f760801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ede1efa56386c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:59.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f760e7fc640 1 -- 172.21.15.46:0/684744020 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76080af400 con 0x7f76100c5f60 2023-10-30T13:52:59.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f760e7fc640 1 -- 172.21.15.46:0/684744020 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76080ab050 con 0x7f76100c5f60 2023-10-30T13:52:59.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.859+0000 7f760e7fc640 1 -- 172.21.15.46:0/684744020 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76080af780 con 0x7f76100c5f60 2023-10-30T13:52:59.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 -- 172.21.15.46:0/684744020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76100c5f60 msgr2=0x7f76100cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:59.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 --2- 172.21.15.46:0/684744020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76100c5f60 0x7f76100cb570 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f76100723d0 tx=0x7f760801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:52:59.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 -- 172.21.15.46:0/684744020 shutdown_connections 2023-10-30T13:52:59.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 --2- 172.21.15.46:0/684744020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76100c5f60 0x7f76100cb570 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:59.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 --2- 172.21.15.46:0/684744020 >> v2:172.21.15.154:3300/0 conn(0x7f76100c6fe0 0x7f7610071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:59.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 -- 172.21.15.46:0/684744020 >> 172.21.15.46:0/684744020 conn(0x7f761006c6e0 msgr2=0x7f7610069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:59.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 -- 172.21.15.46:0/684744020 shutdown_connections 2023-10-30T13:52:59.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 -- 172.21.15.46:0/684744020 wait complete. 2023-10-30T13:52:59.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 Processor -- start 2023-10-30T13:52:59.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 -- start start 2023-10-30T13:52:59.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76100c5f60 0x7f761014f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:59.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76100c6fe0 0x7f76101485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:59.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7610153bf0 0x7f7610148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:59.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7610070520 con 0x7f76100c5f60 2023-10-30T13:52:59.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76100712c0 con 0x7f76100c6fe0 2023-10-30T13:52:59.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f76160c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7610070770 con 0x7f7610153bf0 2023-10-30T13:52:59.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f760f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76100c5f60 0x7f761014f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:59.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f760f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76100c5f60 0x7f761014f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54484/0 (socket says 172.21.15.46:54484) 2023-10-30T13:52:59.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f760f7fe640 1 -- 172.21.15.46:0/723993862 learned_addr learned my addr 172.21.15.46:0/723993862 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:52:59.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f760effd640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76100c6fe0 0x7f76101485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:59.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.863+0000 7f760ffff640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7610153bf0 0x7f7610148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:59.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f760f7fe640 1 -- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7610153bf0 msgr2=0x7f7610148b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:59.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f760f7fe640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7610153bf0 0x7f7610148b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:59.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f760f7fe640 1 -- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76100c6fe0 msgr2=0x7f76101485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:59.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f760f7fe640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76100c6fe0 0x7f76101485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:59.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f760f7fe640 1 -- 172.21.15.46:0/723993862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7608009b30 con 0x7f76100c5f60 2023-10-30T13:52:59.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f760f7fe640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76100c5f60 0x7f761014f1a0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f760000c3c0 tx=0x7f760000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:59.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f75f7fff640 1 -- 172.21.15.46:0/723993862 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76000052d0 con 0x7f76100c5f60 2023-10-30T13:52:59.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f75f7fff640 1 -- 172.21.15.46:0/723993862 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7600005430 con 0x7f76100c5f60 2023-10-30T13:52:59.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f75f7fff640 1 -- 172.21.15.46:0/723993862 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600003840 con 0x7f76100c5f60 2023-10-30T13:52:59.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f76160c9640 1 -- 172.21.15.46:0/723993862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7610070a50 con 0x7f76100c5f60 2023-10-30T13:52:59.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f76160c9640 1 -- 172.21.15.46:0/723993862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761014e7d0 con 0x7f76100c5f60 2023-10-30T13:52:59.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f76160c9640 1 -- 172.21.15.46:0/723993862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76100715a0 con 0x7f76100c5f60 2023-10-30T13:52:59.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f75f7fff640 1 -- 172.21.15.46:0/723993862 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f76000039a0 con 0x7f76100c5f60 2023-10-30T13:52:59.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f75f7fff640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75fc0664a0 0x7f75fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:59.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f75f7fff640 1 -- 172.21.15.46:0/723993862 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f7600083a90 con 0x7f76100c5f60 2023-10-30T13:52:59.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.867+0000 7f760effd640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75fc0664a0 0x7f75fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:59.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.871+0000 7f760effd640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75fc0664a0 0x7f75fc068960 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7610144bc0 tx=0x7f7604008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:59.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:52:59.871+0000 7f75f7fff640 1 -- 172.21.15.46:0/723993862 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7600050840 con 0x7f76100c5f60 2023-10-30T13:53:00.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 -- 172.21.15.46:0/723993862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f761004e170 con 0x7f76100c5f60 2023-10-30T13:53:00.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f75f7fff640 1 -- 172.21.15.46:0/723993862 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f7600017090 con 0x7f76100c5f60 2023-10-30T13:53:00.065 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:00.065 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:59.238247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:00.065 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 132 2023-10-30T13:53:00.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 -- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75fc0664a0 msgr2=0x7f75fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:00.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75fc0664a0 0x7f75fc068960 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7610144bc0 tx=0x7f7604008040 comp rx=0 tx=0).stop 2023-10-30T13:53:00.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 -- 172.21.15.46:0/723993862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76100c5f60 msgr2=0x7f761014f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:00.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76100c5f60 0x7f761014f1a0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f760000c3c0 tx=0x7f760000c880 comp rx=0 tx=0).stop 2023-10-30T13:53:00.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 -- 172.21.15.46:0/723993862 shutdown_connections 2023-10-30T13:53:00.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75fc0664a0 0x7f75fc068960 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:00.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7610153bf0 0x7f7610148b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:00.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76100c6fe0 0x7f76101485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:00.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 --2- 172.21.15.46:0/723993862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76100c5f60 0x7f761014f1a0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:00.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 -- 172.21.15.46:0/723993862 >> 172.21.15.46:0/723993862 conn(0x7f761006c6e0 msgr2=0x7f76101581b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:00.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 -- 172.21.15.46:0/723993862 shutdown_connections 2023-10-30T13:53:00.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.063+0000 7f76160c9640 1 -- 172.21.15.46:0/723993862 wait complete. 2023-10-30T13:53:00.085 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T13:53:00.086 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:53:00.086 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.227+0000 7fb3596c9640 1 Processor -- start 2023-10-30T13:53:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.227+0000 7fb3596c9640 1 -- start start 2023-10-30T13:53:00.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.227+0000 7fb3596c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb35406fd50 0x7fb35415ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:00.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.227+0000 7fb3596c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb354157340 con 0x7fb35406afa0 2023-10-30T13:53:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.227+0000 7fb3596c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35406ca30 con 0x7fb354156f60 2023-10-30T13:53:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.227+0000 7fb3596c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35406bea0 con 0x7fb35406fd50 2023-10-30T13:53:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.227+0000 7fb3537fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb35406afa0 0x7fb354164160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35758/0 (socket says 172.21.15.46:35758) 2023-10-30T13:53:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.227+0000 7fb3537fe640 1 -- 172.21.15.46:0/1700150148 learned_addr learned my addr 172.21.15.46:0/1700150148 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:53:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.227+0000 7fb34bfff640 1 --2- 172.21.15.46:0/1700150148 >> v2:172.21.15.154:3300/0 conn(0x7fb35406fd50 0x7fb35415ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:00.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.227+0000 7fb3527fc640 1 -- 172.21.15.46:0/1700150148 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637872118 0 0) 0x7fb354157340 con 0x7fb35406afa0 2023-10-30T13:53:00.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.227+0000 7fb3527fc640 1 -- 172.21.15.46:0/1700150148 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb340003530 con 0x7fb35406afa0 2023-10-30T13:53:00.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.231+0000 7fb34bfff640 1 -- 172.21.15.46:0/1700150148 >> v1:172.21.15.154:6789/0 conn(0x7fb354156f60 legacy=0x7fb354071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:00.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.231+0000 7fb3527fc640 1 -- 172.21.15.46:0/1700150148 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370583319 0 0) 0x7fb35406ca30 con 0x7fb354156f60 2023-10-30T13:53:00.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.231+0000 7fb34bfff640 1 -- 172.21.15.46:0/1700150148 >> v1:172.21.15.46:6789/0 conn(0x7fb35406afa0 legacy=0x7fb354164160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:00.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.231+0000 7fb34bfff640 1 -- 172.21.15.46:0/1700150148 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35406cf80 con 0x7fb35406fd50 2023-10-30T13:53:00.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.231+0000 7fb34bfff640 1 --2- 172.21.15.46:0/1700150148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35406fd50 0x7fb35415ce10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0097c0 tx=0x7fb33c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=481377c4cbeb7cd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:00.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.231+0000 7fb3527fc640 1 -- 172.21.15.46:0/1700150148 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c0ab600 con 0x7fb35406fd50 2023-10-30T13:53:00.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.231+0000 7fb3527fc640 1 -- 172.21.15.46:0/1700150148 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c0a7050 con 0x7fb35406fd50 2023-10-30T13:53:00.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.231+0000 7fb3527fc640 1 -- 172.21.15.46:0/1700150148 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c0ab960 con 0x7fb35406fd50 2023-10-30T13:53:00.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 -- 172.21.15.46:0/1700150148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35406fd50 msgr2=0x7fb35415ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:00.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 --2- 172.21.15.46:0/1700150148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35406fd50 0x7fb35415ce10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0097c0 tx=0x7fb33c01c480 comp rx=0 tx=0).stop 2023-10-30T13:53:00.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 -- 172.21.15.46:0/1700150148 shutdown_connections 2023-10-30T13:53:00.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 --2- 172.21.15.46:0/1700150148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35406fd50 0x7fb35415ce10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:00.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 -- 172.21.15.46:0/1700150148 >> 172.21.15.46:0/1700150148 conn(0x7fb354069e90 msgr2=0x7fb35406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:00.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 -- 172.21.15.46:0/1700150148 shutdown_connections 2023-10-30T13:53:00.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 -- 172.21.15.46:0/1700150148 wait complete. 2023-10-30T13:53:00.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 Processor -- start 2023-10-30T13:53:00.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 -- start start 2023-10-30T13:53:00.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35406afa0 0x7fb35415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:00.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb35406fd50 0x7fb354158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:00.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb354156f60 0x7fb3541587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:00.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb354149640 con 0x7fb354156f60 2023-10-30T13:53:00.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb354157560 con 0x7fb35406afa0 2023-10-30T13:53:00.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3596c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb354157e00 con 0x7fb35406fd50 2023-10-30T13:53:00.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3537fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb354156f60 0x7fb3541587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:00.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3537fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb354156f60 0x7fb3541587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54488/0 (socket says 172.21.15.46:54488) 2023-10-30T13:53:00.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3537fe640 1 -- 172.21.15.46:0/1296796892 learned_addr learned my addr 172.21.15.46:0/1296796892 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:00.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb34bfff640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb35406fd50 0x7fb354158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:00.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb352ffd640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35406afa0 0x7fb35415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:00.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3537fe640 1 -- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb35406fd50 msgr2=0x7fb354158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:00.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3537fe640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb35406fd50 0x7fb354158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:00.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3537fe640 1 -- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35406afa0 msgr2=0x7fb35415ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:00.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3537fe640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35406afa0 0x7fb35415ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:00.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3537fe640 1 -- 172.21.15.46:0/1296796892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c009420 con 0x7fb354156f60 2023-10-30T13:53:00.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.235+0000 7fb3537fe640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb354156f60 0x7fb3541587d0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00cad0 tx=0x7fb34c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:00.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.239+0000 7fb34b7fe640 1 -- 172.21.15.46:0/1296796892 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c00abd0 con 0x7fb354156f60 2023-10-30T13:53:00.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.239+0000 7fb34b7fe640 1 -- 172.21.15.46:0/1296796892 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34c00ad30 con 0x7fb354156f60 2023-10-30T13:53:00.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.239+0000 7fb34b7fe640 1 -- 172.21.15.46:0/1296796892 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c00aea0 con 0x7fb354156f60 2023-10-30T13:53:00.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.239+0000 7fb3596c9640 1 -- 172.21.15.46:0/1296796892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35415b6c0 con 0x7fb354156f60 2023-10-30T13:53:00.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.239+0000 7fb3596c9640 1 -- 172.21.15.46:0/1296796892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3541577b0 con 0x7fb354156f60 2023-10-30T13:53:00.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.239+0000 7fb3596c9640 1 -- 172.21.15.46:0/1296796892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35404e170 con 0x7fb354156f60 2023-10-30T13:53:00.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.239+0000 7fb34b7fe640 1 -- 172.21.15.46:0/1296796892 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb34c007070 con 0x7fb354156f60 2023-10-30T13:53:00.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.239+0000 7fb34b7fe640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3340664a0 0x7fb334068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:00.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.239+0000 7fb34b7fe640 1 -- 172.21.15.46:0/1296796892 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7fb34c0842e0 con 0x7fb354156f60 2023-10-30T13:53:00.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.239+0000 7fb352ffd640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3340664a0 0x7fb334068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:00.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.243+0000 7fb352ffd640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3340664a0 0x7fb334068960 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb344000f70 tx=0x7fb344009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:00.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.243+0000 7fb34b7fe640 1 -- 172.21.15.46:0/1296796892 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb34c051260 con 0x7fb354156f60 2023-10-30T13:53:00.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 -- 172.21.15.46:0/1296796892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb35406f180 con 0x7fb354156f60 2023-10-30T13:53:00.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb34b7fe640 1 -- 172.21.15.46:0/1296796892 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fb34c054880 con 0x7fb354156f60 2023-10-30T13:53:00.444 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:00.444 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:59.238247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:00.445 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 132 2023-10-30T13:53:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 -- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3340664a0 msgr2=0x7fb334068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3340664a0 0x7fb334068960 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb344000f70 tx=0x7fb344009040 comp rx=0 tx=0).stop 2023-10-30T13:53:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 -- 172.21.15.46:0/1296796892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb354156f60 msgr2=0x7fb3541587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb354156f60 0x7fb3541587d0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00cad0 tx=0x7fb34c00cf90 comp rx=0 tx=0).stop 2023-10-30T13:53:00.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 -- 172.21.15.46:0/1296796892 shutdown_connections 2023-10-30T13:53:00.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3340664a0 0x7fb334068960 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:00.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb354156f60 0x7fb3541587d0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:00.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb35406fd50 0x7fb354158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:00.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 --2- 172.21.15.46:0/1296796892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35406afa0 0x7fb35415ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:00.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 -- 172.21.15.46:0/1296796892 >> 172.21.15.46:0/1296796892 conn(0x7fb354069e90 msgr2=0x7fb3540c9220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:00.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 -- 172.21.15.46:0/1296796892 shutdown_connections 2023-10-30T13:53:00.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:00.443+0000 7fb3596c9640 1 -- 172.21.15.46:0/1296796892 wait complete. 2023-10-30T13:53:00.466 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 132, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 132, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7002": { "addr": "172.21.15.46:6833/2807435195", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2807435195, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2807435195, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7002, "incarnation": 125, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_7188": { "addr": "172.21.15.46:6836/711135361", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 711135361, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 711135361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7188, "incarnation": 132, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 72, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:52:59.238247+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7002, "mds_1": 7188 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:53:02.467 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:02.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.619+0000 7f4725ad6640 1 Processor -- start 2023-10-30T13:53:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.619+0000 7f4725ad6640 1 -- start start 2023-10-30T13:53:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.623+0000 7f4725ad6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4720071960 0x7f47200cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.623+0000 7f4725ad6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47201560a0 con 0x7f472014fd50 2023-10-30T13:53:02.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.623+0000 7f4725ad6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4720156380 con 0x7f47200cb550 2023-10-30T13:53:02.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.623+0000 7f4725ad6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4720156660 con 0x7f4720071960 2023-10-30T13:53:02.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.623+0000 7f471ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4720071960 0x7f47200cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:02.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.623+0000 7f471ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4720071960 0x7f47200cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54490/0 (socket says 172.21.15.46:54490) 2023-10-30T13:53:02.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.623+0000 7f471ffff640 1 -- 172.21.15.46:0/3258590148 learned_addr learned my addr 172.21.15.46:0/3258590148 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.623+0000 7f471f7fe640 1 --1- 172.21.15.46:0/3258590148 >> v1:172.21.15.46:6789/0 conn(0x7f472014fd50 0x7f47200c65d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35770/0 (socket says 172.21.15.46:35770) 2023-10-30T13:53:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.623+0000 7f471ffff640 1 -- 172.21.15.46:0/3258590148 >> v1:172.21.15.154:6789/0 conn(0x7f47200cb550 legacy=0x7f4720071250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.623+0000 7f471ffff640 1 -- 172.21.15.46:0/3258590148 >> v1:172.21.15.46:6789/0 conn(0x7f472014fd50 legacy=0x7f47200c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:02.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.623+0000 7f471ffff640 1 -- 172.21.15.46:0/3258590148 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4720156940 con 0x7f4720071960 2023-10-30T13:53:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f471ffff640 1 --2- 172.21.15.46:0/3258590148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4720071960 0x7f47200cb9e0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f471800d150 tx=0x7f471801eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6597d29b0ea9e242 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:02.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f471e7fc640 1 -- 172.21.15.46:0/3258590148 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47180a7b10 con 0x7f4720071960 2023-10-30T13:53:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f471e7fc640 1 -- 172.21.15.46:0/3258590148 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47180ab070 con 0x7f4720071960 2023-10-30T13:53:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f471e7fc640 1 -- 172.21.15.46:0/3258590148 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47180a7e90 con 0x7f4720071960 2023-10-30T13:53:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 -- 172.21.15.46:0/3258590148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4720071960 msgr2=0x7f47200cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 --2- 172.21.15.46:0/3258590148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4720071960 0x7f47200cb9e0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f471800d150 tx=0x7f471801eb20 comp rx=0 tx=0).stop 2023-10-30T13:53:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 -- 172.21.15.46:0/3258590148 shutdown_connections 2023-10-30T13:53:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 --2- 172.21.15.46:0/3258590148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4720071960 0x7f47200cb9e0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 -- 172.21.15.46:0/3258590148 >> 172.21.15.46:0/3258590148 conn(0x7f472006c6e0 msgr2=0x7f4720069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 -- 172.21.15.46:0/3258590148 shutdown_connections 2023-10-30T13:53:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 -- 172.21.15.46:0/3258590148 wait complete. 2023-10-30T13:53:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 Processor -- start 2023-10-30T13:53:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 -- start start 2023-10-30T13:53:02.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720071960 0x7f4720160ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:02.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47200cb550 0x7f47200c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:02.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f472014fd50 0x7f47200c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:02.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4720155830 con 0x7f47200cb550 2023-10-30T13:53:02.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4720144240 con 0x7f4720071960 2023-10-30T13:53:02.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.627+0000 7f4725ad6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4720144520 con 0x7f472014fd50 2023-10-30T13:53:02.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f471effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47200cb550 0x7f47200c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:02.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f471effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47200cb550 0x7f47200c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54498/0 (socket says 172.21.15.46:54498) 2023-10-30T13:53:02.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f471effd640 1 -- 172.21.15.46:0/220388727 learned_addr learned my addr 172.21.15.46:0/220388727 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:02.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f471ffff640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f472014fd50 0x7f47200c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f471f7fe640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720071960 0x7f4720160ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f471effd640 1 -- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f472014fd50 msgr2=0x7f47200c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f471effd640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f472014fd50 0x7f47200c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f471effd640 1 -- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720071960 msgr2=0x7f4720160ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:02.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f471effd640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720071960 0x7f4720160ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f471effd640 1 -- 172.21.15.46:0/220388727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471800bd40 con 0x7f47200cb550 2023-10-30T13:53:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f471f7fe640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720071960 0x7f4720160ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f471effd640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47200cb550 0x7f47200c83f0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f471400c3c0 tx=0x7f471400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f4707fff640 1 -- 172.21.15.46:0/220388727 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47140052d0 con 0x7f47200cb550 2023-10-30T13:53:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f4707fff640 1 -- 172.21.15.46:0/220388727 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4714005430 con 0x7f47200cb550 2023-10-30T13:53:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f4707fff640 1 -- 172.21.15.46:0/220388727 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4714003840 con 0x7f47200cb550 2023-10-30T13:53:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f4725ad6640 1 -- 172.21.15.46:0/220388727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47201f1b20 con 0x7f47200cb550 2023-10-30T13:53:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f4725ad6640 1 -- 172.21.15.46:0/220388727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4720144800 con 0x7f47200cb550 2023-10-30T13:53:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f4725ad6640 1 -- 172.21.15.46:0/220388727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472004e170 con 0x7f47200cb550 2023-10-30T13:53:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.631+0000 7f4707fff640 1 -- 172.21.15.46:0/220388727 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f47140039a0 con 0x7f47200cb550 2023-10-30T13:53:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.635+0000 7f4707fff640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f470c0664a0 0x7f470c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.635+0000 7f4707fff640 1 -- 172.21.15.46:0/220388727 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f47140838c0 con 0x7f47200cb550 2023-10-30T13:53:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.635+0000 7f471f7fe640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f470c0664a0 0x7f470c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.635+0000 7f471f7fe640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f470c0664a0 0x7f470c068960 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4710000f70 tx=0x7f4710007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.639+0000 7f4707fff640 1 -- 172.21.15.46:0/220388727 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4714050850 con 0x7f47200cb550 2023-10-30T13:53:02.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 -- 172.21.15.46:0/220388727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47201536b0 con 0x7f47200cb550 2023-10-30T13:53:02.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4707fff640 1 -- 172.21.15.46:0/220388727 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f4714053e70 con 0x7f47200cb550 2023-10-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:02.829 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:59.238247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 132 2023-10-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 -- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f470c0664a0 msgr2=0x7f470c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f470c0664a0 0x7f470c068960 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4710000f70 tx=0x7f4710007d50 comp rx=0 tx=0).stop 2023-10-30T13:53:02.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 -- 172.21.15.46:0/220388727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47200cb550 msgr2=0x7f47200c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47200cb550 0x7f47200c83f0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f471400c3c0 tx=0x7f471400c880 comp rx=0 tx=0).stop 2023-10-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 -- 172.21.15.46:0/220388727 shutdown_connections 2023-10-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f470c0664a0 0x7f470c068960 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:02.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f472014fd50 0x7f47200c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47200cb550 0x7f47200c83f0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 --2- 172.21.15.46:0/220388727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720071960 0x7f4720160ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 -- 172.21.15.46:0/220388727 >> 172.21.15.46:0/220388727 conn(0x7f472006c6e0 msgr2=0x7f4720071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 -- 172.21.15.46:0/220388727 shutdown_connections 2023-10-30T13:53:02.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:02.827+0000 7f4725ad6640 1 -- 172.21.15.46:0/220388727 wait complete. 2023-10-30T13:53:02.849 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:53:04.851 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.995+0000 7f13eb8d6640 1 Processor -- start 2023-10-30T13:53:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.995+0000 7f13eb8d6640 1 -- start start 2023-10-30T13:53:04.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13eb8d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4159e60 0x7f13e406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:05.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13eb8d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f13e40bfec0 0x7f13e40c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:05.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13eb8d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e406fa80 con 0x7f13e406a720 2023-10-30T13:53:05.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13eb8d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e4071290 con 0x7f13e40bfec0 2023-10-30T13:53:05.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13eb8d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e4071570 con 0x7f13e4159e60 2023-10-30T13:53:05.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13e8e71640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f13e40bfec0 0x7f13e40c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:05.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13e9672640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4159e60 0x7f13e406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:05.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13e9672640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4159e60 0x7f13e406f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:43876/0 (socket says 172.21.15.46:43876) 2023-10-30T13:53:05.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13e9672640 1 -- 172.21.15.46:0/3626490231 learned_addr learned my addr 172.21.15.46:0/3626490231 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:05.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13e9e73640 1 --1- 172.21.15.46:0/3626490231 >> v1:172.21.15.154:6789/0 conn(0x7f13e406a720 0x7f13e4155d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:40658/0 (socket says 172.21.15.46:40658) 2023-10-30T13:53:05.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13e8e71640 1 -- 172.21.15.46:0/3626490231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4159e60 msgr2=0x7f13e406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:05.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13e8e71640 1 --2- 172.21.15.46:0/3626490231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4159e60 0x7f13e406f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:05.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13e8e71640 1 -- 172.21.15.46:0/3626490231 >> v1:172.21.15.154:6789/0 conn(0x7f13e406a720 legacy=0x7f13e4155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:05.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13e8e71640 1 -- 172.21.15.46:0/3626490231 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e4071850 con 0x7f13e40bfec0 2023-10-30T13:53:05.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:04.999+0000 7f13e9672640 1 --2- 172.21.15.46:0/3626490231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4159e60 0x7f13e406f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:53:05.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.003+0000 7f13e8e71640 1 --2- 172.21.15.46:0/3626490231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e40bfec0 0x7f13e40c1a40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f13d40097c0 tx=0x7f13d401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b93abb9c990922d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:05.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.003+0000 7f13dbfff640 1 -- 172.21.15.46:0/3626490231 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d40a8970 con 0x7f13e40bfec0 2023-10-30T13:53:05.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.003+0000 7f13dbfff640 1 -- 172.21.15.46:0/3626490231 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d40a8ad0 con 0x7f13e40bfec0 2023-10-30T13:53:05.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.003+0000 7f13dbfff640 1 -- 172.21.15.46:0/3626490231 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d40a8e60 con 0x7f13e40bfec0 2023-10-30T13:53:05.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.003+0000 7f13eb8d6640 1 -- 172.21.15.46:0/3626490231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e40bfec0 msgr2=0x7f13e40c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:05.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.003+0000 7f13eb8d6640 1 --2- 172.21.15.46:0/3626490231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e40bfec0 0x7f13e40c1a40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f13d40097c0 tx=0x7f13d401ec70 comp rx=0 tx=0).stop 2023-10-30T13:53:05.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.007+0000 7f13eb8d6640 1 -- 172.21.15.46:0/3626490231 shutdown_connections 2023-10-30T13:53:05.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.007+0000 7f13eb8d6640 1 --2- 172.21.15.46:0/3626490231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e40bfec0 0x7f13e40c1a40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:05.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.007+0000 7f13eb8d6640 1 --2- 172.21.15.46:0/3626490231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4159e60 0x7f13e406f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:05.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.007+0000 7f13eb8d6640 1 -- 172.21.15.46:0/3626490231 >> 172.21.15.46:0/3626490231 conn(0x7f13e40c7910 msgr2=0x7f13e40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:05.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.007+0000 7f13eb8d6640 1 -- 172.21.15.46:0/3626490231 shutdown_connections 2023-10-30T13:53:05.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.007+0000 7f13eb8d6640 1 -- 172.21.15.46:0/3626490231 wait complete. 2023-10-30T13:53:05.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.007+0000 7f13eb8d6640 1 Processor -- start 2023-10-30T13:53:05.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.007+0000 7f13eb8d6640 1 -- start start 2023-10-30T13:53:05.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13eb8d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e406a720 0x7f13e40c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:05.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13eb8d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e40bfec0 0x7f13e41eea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:05.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13eb8d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e4159e60 0x7f13e41eef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:05.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13eb8d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e414e390 con 0x7f13e406a720 2023-10-30T13:53:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13eb8d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e41f2f20 con 0x7f13e4159e60 2023-10-30T13:53:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13eb8d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e414e0b0 con 0x7f13e40bfec0 2023-10-30T13:53:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13e9672640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e406a720 0x7f13e40c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13e9672640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e406a720 0x7f13e40c67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54504/0 (socket says 172.21.15.46:54504) 2023-10-30T13:53:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13e9672640 1 -- 172.21.15.46:0/2930524071 learned_addr learned my addr 172.21.15.46:0/2930524071 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13e8e71640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e40bfec0 0x7f13e41eea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13e9e73640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e4159e60 0x7f13e41eef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13e9672640 1 -- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e40bfec0 msgr2=0x7f13e41eea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13e9672640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e40bfec0 0x7f13e41eea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13e9672640 1 -- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e4159e60 msgr2=0x7f13e41eef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13e9672640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e4159e60 0x7f13e41eef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13e9672640 1 -- 172.21.15.46:0/2930524071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e000a670 con 0x7f13e406a720 2023-10-30T13:53:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13e9672640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e406a720 0x7f13e40c67b0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f13e000e830 tx=0x7f13e000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13d9ffb640 1 -- 172.21.15.46:0/2930524071 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e000cc40 con 0x7f13e406a720 2023-10-30T13:53:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13d9ffb640 1 -- 172.21.15.46:0/2930524071 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e000cda0 con 0x7f13e406a720 2023-10-30T13:53:05.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13d9ffb640 1 -- 172.21.15.46:0/2930524071 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e0004230 con 0x7f13e406a720 2023-10-30T13:53:05.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13eb8d6640 1 -- 172.21.15.46:0/2930524071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d4009420 con 0x7f13e406a720 2023-10-30T13:53:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13eb8d6640 1 -- 172.21.15.46:0/2930524071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e41f2440 con 0x7f13e406a720 2023-10-30T13:53:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.011+0000 7f13eb8d6640 1 -- 172.21.15.46:0/2930524071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e404e170 con 0x7f13e406a720 2023-10-30T13:53:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.015+0000 7f13d9ffb640 1 -- 172.21.15.46:0/2930524071 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13e0004390 con 0x7f13e406a720 2023-10-30T13:53:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.015+0000 7f13d9ffb640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13c80664a0 0x7f13c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:05.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.015+0000 7f13d9ffb640 1 -- 172.21.15.46:0/2930524071 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f13e0084140 con 0x7f13e406a720 2023-10-30T13:53:05.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.015+0000 7f13e8e71640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13c80664a0 0x7f13c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:05.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.015+0000 7f13e8e71640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13c80664a0 0x7f13c8068960 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f13d40b2000 tx=0x7f13d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:05.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.019+0000 7f13d9ffb640 1 -- 172.21.15.46:0/2930524071 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f13e0050f90 con 0x7f13e406a720 2023-10-30T13:53:05.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.215+0000 7f13eb8d6640 1 -- 172.21.15.46:0/2930524071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13e406ae00 con 0x7f13e406a720 2023-10-30T13:53:05.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.215+0000 7f13d9ffb640 1 -- 172.21.15.46:0/2930524071 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f13e001f090 con 0x7f13e406a720 2023-10-30T13:53:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:05.218 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:59.238247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:05.219 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 132 2023-10-30T13:53:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.219+0000 7f13eb8d6640 1 -- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13c80664a0 msgr2=0x7f13c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.219+0000 7f13eb8d6640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13c80664a0 0x7f13c8068960 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f13d40b2000 tx=0x7f13d4008040 comp rx=0 tx=0).stop 2023-10-30T13:53:05.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.219+0000 7f13eb8d6640 1 -- 172.21.15.46:0/2930524071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e406a720 msgr2=0x7f13e40c67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:05.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.219+0000 7f13eb8d6640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e406a720 0x7f13e40c67b0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f13e000e830 tx=0x7f13e000ecf0 comp rx=0 tx=0).stop 2023-10-30T13:53:05.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.219+0000 7f13eb8d6640 1 -- 172.21.15.46:0/2930524071 shutdown_connections 2023-10-30T13:53:05.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.219+0000 7f13eb8d6640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f13c80664a0 0x7f13c8068960 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:05.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.219+0000 7f13eb8d6640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e4159e60 0x7f13e41eef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:05.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.219+0000 7f13eb8d6640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e40bfec0 0x7f13e41eea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:05.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.219+0000 7f13eb8d6640 1 --2- 172.21.15.46:0/2930524071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13e406a720 0x7f13e40c67b0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:05.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.219+0000 7f13eb8d6640 1 -- 172.21.15.46:0/2930524071 >> 172.21.15.46:0/2930524071 conn(0x7f13e40c7910 msgr2=0x7f13e41e42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:05.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.219+0000 7f13eb8d6640 1 -- 172.21.15.46:0/2930524071 shutdown_connections 2023-10-30T13:53:05.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:05.219+0000 7f13eb8d6640 1 -- 172.21.15.46:0/2930524071 wait complete. 2023-10-30T13:53:05.240 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:53:07.240 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:07.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.387+0000 7f761c079640 1 Processor -- start 2023-10-30T13:53:07.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.387+0000 7f761c079640 1 -- start start 2023-10-30T13:53:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f761c079640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76140c6b80 0x7f76140bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f761c079640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76140c6f60 0x7f76140c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f761c079640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76140c0a50 con 0x7f76140c8430 2023-10-30T13:53:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f761c079640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761406f6e0 con 0x7f76140c6b80 2023-10-30T13:53:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f761c079640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761406f9c0 con 0x7f76140c6f60 2023-10-30T13:53:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f7619e15640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76140c6b80 0x7f76140bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f7619e15640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76140c6b80 0x7f76140bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53290/0 (socket says 172.21.15.46:53290) 2023-10-30T13:53:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f7619e15640 1 -- 172.21.15.46:0/762102918 learned_addr learned my addr 172.21.15.46:0/762102918 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f7619614640 1 --2- 172.21.15.46:0/762102918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76140c6f60 0x7f76140c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f7619e15640 1 -- 172.21.15.46:0/762102918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76140c6f60 msgr2=0x7f76140c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f7619e15640 1 --2- 172.21.15.46:0/762102918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76140c6f60 0x7f76140c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f7619e15640 1 -- 172.21.15.46:0/762102918 >> v1:172.21.15.154:6789/0 conn(0x7f76140c8430 legacy=0x7f761413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:07.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.391+0000 7f7619e15640 1 -- 172.21.15.46:0/762102918 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761406fca0 con 0x7f76140c6b80 2023-10-30T13:53:07.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f7619e15640 1 --2- 172.21.15.46:0/762102918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76140c6b80 0x7f76140bfec0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f761000bb10 tx=0x7f761001eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870cd9f49d41743f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:07.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f7618e13640 1 -- 172.21.15.46:0/762102918 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76100a7b60 con 0x7f76140c6b80 2023-10-30T13:53:07.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f7618e13640 1 -- 172.21.15.46:0/762102918 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76100a7cc0 con 0x7f76140c6b80 2023-10-30T13:53:07.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f7618e13640 1 -- 172.21.15.46:0/762102918 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76100af660 con 0x7f76140c6b80 2023-10-30T13:53:07.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f761c079640 1 -- 172.21.15.46:0/762102918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76140c6b80 msgr2=0x7f76140bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:07.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f761c079640 1 --2- 172.21.15.46:0/762102918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76140c6b80 0x7f76140bfec0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f761000bb10 tx=0x7f761001eb80 comp rx=0 tx=0).stop 2023-10-30T13:53:07.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f761c079640 1 -- 172.21.15.46:0/762102918 shutdown_connections 2023-10-30T13:53:07.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f761c079640 1 --2- 172.21.15.46:0/762102918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76140c6f60 0x7f76140c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f761c079640 1 --2- 172.21.15.46:0/762102918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76140c6b80 0x7f76140bfec0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f761c079640 1 -- 172.21.15.46:0/762102918 >> 172.21.15.46:0/762102918 conn(0x7f76140c7910 msgr2=0x7f76140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f761c079640 1 -- 172.21.15.46:0/762102918 shutdown_connections 2023-10-30T13:53:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f761c079640 1 -- 172.21.15.46:0/762102918 wait complete. 2023-10-30T13:53:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f761c079640 1 Processor -- start 2023-10-30T13:53:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.395+0000 7f761c079640 1 -- start start 2023-10-30T13:53:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f761c079640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76140c6b80 0x7f7614160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f761c079640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76140c6f60 0x7f761413aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f761c079640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76140c8430 0x7f761413ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f761c079640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76140c9790 con 0x7f76140c6b80 2023-10-30T13:53:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f761c079640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7614132000 con 0x7f76140c8430 2023-10-30T13:53:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f761c079640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7614157790 con 0x7f76140c6f60 2023-10-30T13:53:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f7619e15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76140c6b80 0x7f7614160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f7619e15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76140c6b80 0x7f7614160600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53304/0 (socket says 172.21.15.46:53304) 2023-10-30T13:53:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f7619e15640 1 -- 172.21.15.46:0/692972527 learned_addr learned my addr 172.21.15.46:0/692972527 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f761a616640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76140c8430 0x7f761413ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f7619614640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76140c6f60 0x7f761413aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f7619e15640 1 -- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76140c6f60 msgr2=0x7f761413aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f7619e15640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76140c6f60 0x7f761413aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f7619e15640 1 -- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76140c8430 msgr2=0x7f761413ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f7619e15640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76140c8430 0x7f761413ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f7619e15640 1 -- 172.21.15.46:0/692972527 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761000b770 con 0x7f76140c6b80 2023-10-30T13:53:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f7619e15640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76140c6b80 0x7f7614160600 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f761000b2c0 tx=0x7f761001d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f760a7fc640 1 -- 172.21.15.46:0/692972527 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76100ba680 con 0x7f76140c6b80 2023-10-30T13:53:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f760a7fc640 1 -- 172.21.15.46:0/692972527 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76100ba800 con 0x7f76140c6b80 2023-10-30T13:53:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f760a7fc640 1 -- 172.21.15.46:0/692972527 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76100babb0 con 0x7f76140c6b80 2023-10-30T13:53:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f761c079640 1 -- 172.21.15.46:0/692972527 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7614158a80 con 0x7f76140c6b80 2023-10-30T13:53:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f761c079640 1 -- 172.21.15.46:0/692972527 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7614158d20 con 0x7f76140c6b80 2023-10-30T13:53:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.399+0000 7f761c079640 1 -- 172.21.15.46:0/692972527 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761404e170 con 0x7f76140c6b80 2023-10-30T13:53:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.403+0000 7f760a7fc640 1 -- 172.21.15.46:0/692972527 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f76100bad10 con 0x7f76140c6b80 2023-10-30T13:53:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.403+0000 7f760a7fc640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75f80664a0 0x7f75f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.403+0000 7f760a7fc640 1 -- 172.21.15.46:0/692972527 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f761011fe40 con 0x7f76140c6b80 2023-10-30T13:53:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.403+0000 7f7619614640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75f80664a0 0x7f75f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.403+0000 7f7619614640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75f80664a0 0x7f75f8068960 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7614153ec0 tx=0x7f7604008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.411+0000 7f760a7fc640 1 -- 172.21.15.46:0/692972527 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f76100ecbf0 con 0x7f76140c6b80 2023-10-30T13:53:07.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.623+0000 7f761c079640 1 -- 172.21.15.46:0/692972527 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7614135450 con 0x7f76140c6b80 2023-10-30T13:53:07.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.623+0000 7f760a7fc640 1 -- 172.21.15.46:0/692972527 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f76100b4030 con 0x7f76140c6b80 2023-10-30T13:53:07.627 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:07.627 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:52:59.238247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:07.628 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 132 2023-10-30T13:53:07.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.627+0000 7f761c079640 1 -- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75f80664a0 msgr2=0x7f75f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:07.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.627+0000 7f761c079640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75f80664a0 0x7f75f8068960 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7614153ec0 tx=0x7f7604008040 comp rx=0 tx=0).stop 2023-10-30T13:53:07.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.627+0000 7f761c079640 1 -- 172.21.15.46:0/692972527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76140c6b80 msgr2=0x7f7614160600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:07.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.627+0000 7f761c079640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76140c6b80 0x7f7614160600 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f761000b2c0 tx=0x7f761001d3c0 comp rx=0 tx=0).stop 2023-10-30T13:53:07.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.627+0000 7f761c079640 1 -- 172.21.15.46:0/692972527 shutdown_connections 2023-10-30T13:53:07.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.627+0000 7f761c079640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75f80664a0 0x7f75f8068960 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:07.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.627+0000 7f761c079640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76140c8430 0x7f761413ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:07.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.627+0000 7f761c079640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76140c6f60 0x7f761413aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:07.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.627+0000 7f761c079640 1 --2- 172.21.15.46:0/692972527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76140c6b80 0x7f7614160600 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:07.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.627+0000 7f761c079640 1 -- 172.21.15.46:0/692972527 >> 172.21.15.46:0/692972527 conn(0x7f76140c7910 msgr2=0x7f76140bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:07.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.627+0000 7f761c079640 1 -- 172.21.15.46:0/692972527 shutdown_connections 2023-10-30T13:53:07.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:07.627+0000 7f761c079640 1 -- 172.21.15.46:0/692972527 wait complete. 2023-10-30T13:53:07.661 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:53:09.662 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:09.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f9f3e640 1 Processor -- start 2023-10-30T13:53:09.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f9f3e640 1 -- start start 2023-10-30T13:53:09.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f9f3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82f40cf160 0x7f82f40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:09.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f9f3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f40cb800 con 0x7f82f41538a0 2023-10-30T13:53:09.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f9f3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f40cb330 con 0x7f82f415d6f0 2023-10-30T13:53:09.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f9f3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f40cad70 con 0x7f82f40cf160 2023-10-30T13:53:09.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f82f41538a0 0x7f82f40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52212/0 (socket says 172.21.15.46:52212) 2023-10-30T13:53:09.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f37fe640 1 -- 172.21.15.46:0/1276898398 learned_addr learned my addr 172.21.15.46:0/1276898398 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:53:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f3fff640 1 --2- 172.21.15.46:0/1276898398 >> v2:172.21.15.46:3300/0 conn(0x7f82f40cf160 0x7f82f40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f3fff640 1 -- 172.21.15.46:0/1276898398 >> v1:172.21.15.154:6789/0 conn(0x7f82f415d6f0 legacy=0x7f82f415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f3fff640 1 -- 172.21.15.46:0/1276898398 >> v1:172.21.15.46:6789/0 conn(0x7f82f41538a0 legacy=0x7f82f40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:09.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f3fff640 1 -- 172.21.15.46:0/1276898398 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f40cb050 con 0x7f82f40cf160 2023-10-30T13:53:09.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.815+0000 7f82f27fc640 1 -- 172.21.15.46:0/1276898398 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300501393 0 0) 0x7f82f40cb800 con 0x7f82f41538a0 2023-10-30T13:53:09.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.819+0000 7f82f3fff640 1 --2- 172.21.15.46:0/1276898398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f40cf160 0x7f82f40cf590 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009d90 tx=0x7f82ec01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2e2f9265b27bf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:09.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.819+0000 7f82f27fc640 1 -- 172.21.15.46:0/1276898398 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0af600 con 0x7f82f40cf160 2023-10-30T13:53:09.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.819+0000 7f82f27fc640 1 -- 172.21.15.46:0/1276898398 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec0ab050 con 0x7f82f40cf160 2023-10-30T13:53:09.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.819+0000 7f82f27fc640 1 -- 172.21.15.46:0/1276898398 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0af980 con 0x7f82f40cf160 2023-10-30T13:53:09.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.819+0000 7f82f9f3e640 1 -- 172.21.15.46:0/1276898398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f40cf160 msgr2=0x7f82f40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:09.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.819+0000 7f82f9f3e640 1 --2- 172.21.15.46:0/1276898398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f40cf160 0x7f82f40cf590 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009d90 tx=0x7f82ec01f130 comp rx=0 tx=0).stop 2023-10-30T13:53:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.819+0000 7f82f9f3e640 1 -- 172.21.15.46:0/1276898398 shutdown_connections 2023-10-30T13:53:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.819+0000 7f82f9f3e640 1 --2- 172.21.15.46:0/1276898398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f40cf160 0x7f82f40cf590 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.819+0000 7f82f9f3e640 1 -- 172.21.15.46:0/1276898398 >> 172.21.15.46:0/1276898398 conn(0x7f82f406bfe0 msgr2=0x7f82f406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.819+0000 7f82f9f3e640 1 -- 172.21.15.46:0/1276898398 shutdown_connections 2023-10-30T13:53:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.819+0000 7f82f9f3e640 1 -- 172.21.15.46:0/1276898398 wait complete. 2023-10-30T13:53:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f9f3e640 1 Processor -- start 2023-10-30T13:53:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f9f3e640 1 -- start start 2023-10-30T13:53:09.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f9f3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82f40cf160 0x7f82f415bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:09.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f9f3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f41538a0 0x7f82f4140cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:09.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f9f3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82f415d6f0 0x7f82f4141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:09.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f9f3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40ceb40 con 0x7f82f41538a0 2023-10-30T13:53:09.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f9f3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4159210 con 0x7f82f40cf160 2023-10-30T13:53:09.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f9f3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41594f0 con 0x7f82f415d6f0 2023-10-30T13:53:09.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f41538a0 0x7f82f4140cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:09.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f41538a0 0x7f82f4140cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53332/0 (socket says 172.21.15.46:53332) 2023-10-30T13:53:09.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f2ffd640 1 -- 172.21.15.46:0/4065079332 learned_addr learned my addr 172.21.15.46:0/4065079332 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:09.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f3fff640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82f415d6f0 0x7f82f4141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:09.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f37fe640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82f40cf160 0x7f82f415bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:09.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f2ffd640 1 -- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82f415d6f0 msgr2=0x7f82f4141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:09.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f2ffd640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82f415d6f0 0x7f82f4141210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f2ffd640 1 -- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82f40cf160 msgr2=0x7f82f415bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f2ffd640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82f40cf160 0x7f82f415bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f2ffd640 1 -- 172.21.15.46:0/4065079332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec009a70 con 0x7f82f41538a0 2023-10-30T13:53:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f37fe640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82f40cf160 0x7f82f415bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f2ffd640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f41538a0 0x7f82f4140cd0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f82e800c3c0 tx=0x7f82e800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:09.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82dbfff640 1 -- 172.21.15.46:0/4065079332 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e80052d0 con 0x7f82f41538a0 2023-10-30T13:53:09.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82dbfff640 1 -- 172.21.15.46:0/4065079332 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e8005430 con 0x7f82f41538a0 2023-10-30T13:53:09.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82dbfff640 1 -- 172.21.15.46:0/4065079332 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8003840 con 0x7f82f41538a0 2023-10-30T13:53:09.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f9f3e640 1 -- 172.21.15.46:0/4065079332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f40ced70 con 0x7f82f41538a0 2023-10-30T13:53:09.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.823+0000 7f82f9f3e640 1 -- 172.21.15.46:0/4065079332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f41464e0 con 0x7f82f41538a0 2023-10-30T13:53:09.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.827+0000 7f82f9f3e640 1 -- 172.21.15.46:0/4065079332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f4159660 con 0x7f82f41538a0 2023-10-30T13:53:09.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.827+0000 7f82dbfff640 1 -- 172.21.15.46:0/4065079332 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82e80039a0 con 0x7f82f41538a0 2023-10-30T13:53:09.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.827+0000 7f82dbfff640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82e00664d0 0x7f82e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:09.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.827+0000 7f82dbfff640 1 -- 172.21.15.46:0/4065079332 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f82e80837d0 con 0x7f82f41538a0 2023-10-30T13:53:09.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.827+0000 7f82f37fe640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82e00664d0 0x7f82e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:09.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.827+0000 7f82f37fe640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82e00664d0 0x7f82e0068990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f82e4000f70 tx=0x7f82e4007c60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:09.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:09.831+0000 7f82dbfff640 1 -- 172.21.15.46:0/4065079332 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82e80507f0 con 0x7f82f41538a0 2023-10-30T13:53:10.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.023+0000 7f82f9f3e640 1 -- 172.21.15.46:0/4065079332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82f404e170 con 0x7f82f41538a0 2023-10-30T13:53:10.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.023+0000 7f82dbfff640 1 -- 172.21.15.46:0/4065079332 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f82e8053e10 con 0x7f82f41538a0 2023-10-30T13:53:10.027 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:10.027 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:08.425371+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:resolve","state_seq":16,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:10.028 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 133 2023-10-30T13:53:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.027+0000 7f82f9f3e640 1 -- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82e00664d0 msgr2=0x7f82e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.027+0000 7f82f9f3e640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82e00664d0 0x7f82e0068990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f82e4000f70 tx=0x7f82e4007c60 comp rx=0 tx=0).stop 2023-10-30T13:53:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.027+0000 7f82f9f3e640 1 -- 172.21.15.46:0/4065079332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f41538a0 msgr2=0x7f82f4140cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.027+0000 7f82f9f3e640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f41538a0 0x7f82f4140cd0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f82e800c3c0 tx=0x7f82e800c880 comp rx=0 tx=0).stop 2023-10-30T13:53:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.027+0000 7f82f9f3e640 1 -- 172.21.15.46:0/4065079332 shutdown_connections 2023-10-30T13:53:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.027+0000 7f82f9f3e640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82e00664d0 0x7f82e0068990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.027+0000 7f82f9f3e640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82f415d6f0 0x7f82f4141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.027+0000 7f82f9f3e640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f41538a0 0x7f82f4140cd0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.027+0000 7f82f9f3e640 1 --2- 172.21.15.46:0/4065079332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82f40cf160 0x7f82f415bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:10.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.027+0000 7f82f9f3e640 1 -- 172.21.15.46:0/4065079332 >> 172.21.15.46:0/4065079332 conn(0x7f82f406bfe0 msgr2=0x7f82f4151b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:10.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.027+0000 7f82f9f3e640 1 -- 172.21.15.46:0/4065079332 shutdown_connections 2023-10-30T13:53:10.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:10.027+0000 7f82f9f3e640 1 -- 172.21.15.46:0/4065079332 wait complete. 2023-10-30T13:53:10.048 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:53:12.048 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:12.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e69b80640 1 Processor -- start 2023-10-30T13:53:12.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e69b80640 1 -- start start 2023-10-30T13:53:12.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e69b80640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e6406f7a0 0x7f6e6406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:12.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e69b80640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6e640c6bd0 0x7f6e64069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:12.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e69b80640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6406a3d0 con 0x7f6e64071110 2023-10-30T13:53:12.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e69b80640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e640700c0 con 0x7f6e640c6bd0 2023-10-30T13:53:12.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e69b80640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e64066cf0 con 0x7f6e6406f7a0 2023-10-30T13:53:12.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e63fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6e64071110 0x7f6e64158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52214/0 (socket says 172.21.15.46:52214) 2023-10-30T13:53:12.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e63fff640 1 -- 172.21.15.46:0/4287175738 learned_addr learned my addr 172.21.15.46:0/4287175738 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:53:12.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e637fe640 1 --2- 172.21.15.46:0/4287175738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e6406f7a0 0x7f6e6406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:12.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e62ffd640 1 --2- 172.21.15.46:0/4287175738 >> v2:172.21.15.154:3300/0 conn(0x7f6e640c6bd0 0x7f6e64069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:12.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478206017 0 0) 0x7f6e6406a3d0 con 0x7f6e64071110 2023-10-30T13:53:12.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c003530 con 0x7f6e64071110 2023-10-30T13:53:12.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1109742851 0 0) 0x7f6e4c003530 con 0x7f6e64071110 2023-10-30T13:53:12.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e6406f7a0 msgr2=0x7f6e6406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:12.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 --2- 172.21.15.46:0/4287175738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e6406f7a0 0x7f6e6406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:12.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 >> v2:172.21.15.154:3300/0 conn(0x7f6e640c6bd0 msgr2=0x7f6e64069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:12.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 --2- 172.21.15.46:0/4287175738 >> v2:172.21.15.154:3300/0 conn(0x7f6e640c6bd0 0x7f6e64069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:12.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e640c70d0 con 0x7f6e64071110 2023-10-30T13:53:12.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f6e5c001a00 con 0x7f6e64071110 2023-10-30T13:53:12.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 >> v1:172.21.15.46:6789/0 conn(0x7f6e64071110 legacy=0x7f6e64158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:12.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 --2- 172.21.15.46:0/4287175738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e4c003980 0x7f6e4c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:12.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 --2- 172.21.15.46:0/4287175738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e6406f7a0 0x7f6e4c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:12.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 --2- 172.21.15.46:0/4287175738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e4c004d10 0x7f6e4c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:12.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640700c0 con 0x7f6e4c004d10 2023-10-30T13:53:12.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e64066cf0 con 0x7f6e4c003980 2023-10-30T13:53:12.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.203+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c00b990 con 0x7f6e6406f7a0 2023-10-30T13:53:12.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.207+0000 7f6e63fff640 1 --2- 172.21.15.46:0/4287175738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e4c004d10 0x7f6e4c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:12.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.207+0000 7f6e62ffd640 1 --2- 172.21.15.46:0/4287175738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e6406f7a0 0x7f6e4c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:12.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.207+0000 7f6e637fe640 1 --2- 172.21.15.46:0/4287175738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e4c003980 0x7f6e4c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:12.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.207+0000 7f6e63fff640 1 -- 172.21.15.46:0/4287175738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e6406f7a0 msgr2=0x7f6e4c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:12.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.207+0000 7f6e63fff640 1 --2- 172.21.15.46:0/4287175738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e6406f7a0 0x7f6e4c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:12.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.207+0000 7f6e63fff640 1 -- 172.21.15.46:0/4287175738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e4c003980 msgr2=0x7f6e4c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:12.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.207+0000 7f6e63fff640 1 --2- 172.21.15.46:0/4287175738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e4c003980 0x7f6e4c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:12.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.207+0000 7f6e63fff640 1 -- 172.21.15.46:0/4287175738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c003530 con 0x7f6e4c004d10 2023-10-30T13:53:12.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.207+0000 7f6e63fff640 1 --2- 172.21.15.46:0/4287175738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e4c004d10 0x7f6e4c005190 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00a860 tx=0x7f6e5c01d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84175bbe24e26 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.207+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c0ad630 con 0x7f6e4c004d10 2023-10-30T13:53:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.207+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c0a8070 con 0x7f6e4c004d10 2023-10-30T13:53:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.207+0000 7f6e627fc640 1 -- 172.21.15.46:0/4287175738 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c0ad9f0 con 0x7f6e4c004d10 2023-10-30T13:53:12.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 -- 172.21.15.46:0/4287175738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e4c004d10 msgr2=0x7f6e4c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:12.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 --2- 172.21.15.46:0/4287175738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e4c004d10 0x7f6e4c005190 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00a860 tx=0x7f6e5c01d360 comp rx=0 tx=0).stop 2023-10-30T13:53:12.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e63fff640 1 -- 172.21.15.46:0/4287175738 reap_dead start 2023-10-30T13:53:12.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 -- 172.21.15.46:0/4287175738 shutdown_connections 2023-10-30T13:53:12.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 -- 172.21.15.46:0/4287175738 >> 172.21.15.46:0/4287175738 conn(0x7f6e640c7910 msgr2=0x7f6e640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:12.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 -- 172.21.15.46:0/4287175738 shutdown_connections 2023-10-30T13:53:12.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 -- 172.21.15.46:0/4287175738 wait complete. 2023-10-30T13:53:12.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 Processor -- start 2023-10-30T13:53:12.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 -- start start 2023-10-30T13:53:12.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e4c004d10 0x7f6e640c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:12.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e641ef230 0x7f6e64155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:12.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e641f22b0 0x7f6e641f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:12.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e641f1e20 con 0x7f6e641ef230 2023-10-30T13:53:12.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e641582c0 con 0x7f6e641f22b0 2023-10-30T13:53:12.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e69b80640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e641f2c70 con 0x7f6e4c004d10 2023-10-30T13:53:12.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e62ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e641ef230 0x7f6e64155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e62ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e641ef230 0x7f6e64155fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53350/0 (socket says 172.21.15.46:53350) 2023-10-30T13:53:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e62ffd640 1 -- 172.21.15.46:0/3235314722 learned_addr learned my addr 172.21.15.46:0/3235314722 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e637fe640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e4c004d10 0x7f6e640c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.211+0000 7f6e63fff640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e641f22b0 0x7f6e641f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e62ffd640 1 -- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e4c004d10 msgr2=0x7f6e640c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:12.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e62ffd640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e4c004d10 0x7f6e640c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:12.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e62ffd640 1 -- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e641f22b0 msgr2=0x7f6e641f2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:12.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e62ffd640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e641f22b0 0x7f6e641f2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:12.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e62ffd640 1 -- 172.21.15.46:0/3235314722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e64158540 con 0x7f6e641ef230 2023-10-30T13:53:12.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e637fe640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e4c004d10 0x7f6e640c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:12.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e62ffd640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e641ef230 0x7f6e64155fd0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f6e58011ad0 tx=0x7f6e58011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:12.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e4bfff640 1 -- 172.21.15.46:0/3235314722 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5800fbb0 con 0x7f6e641ef230 2023-10-30T13:53:12.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e4bfff640 1 -- 172.21.15.46:0/3235314722 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5800fd10 con 0x7f6e641ef230 2023-10-30T13:53:12.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e4bfff640 1 -- 172.21.15.46:0/3235314722 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e58014680 con 0x7f6e641ef230 2023-10-30T13:53:12.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e69b80640 1 -- 172.21.15.46:0/3235314722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e640c05a0 con 0x7f6e641ef230 2023-10-30T13:53:12.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e69b80640 1 -- 172.21.15.46:0/3235314722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e641ef6b0 con 0x7f6e641ef230 2023-10-30T13:53:12.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e69b80640 1 -- 172.21.15.46:0/3235314722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e641f1f90 con 0x7f6e641ef230 2023-10-30T13:53:12.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e4bfff640 1 -- 172.21.15.46:0/3235314722 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6e580147e0 con 0x7f6e641ef230 2023-10-30T13:53:12.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e4bfff640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e500664d0 0x7f6e50068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:12.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e4bfff640 1 -- 172.21.15.46:0/3235314722 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f6e58084e90 con 0x7f6e641ef230 2023-10-30T13:53:12.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.215+0000 7f6e637fe640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e500664d0 0x7f6e50068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:12.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.219+0000 7f6e637fe640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e500664d0 0x7f6e50068990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009b60 tx=0x7f6e54008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:12.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.219+0000 7f6e4bfff640 1 -- 172.21.15.46:0/3235314722 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6e58051c40 con 0x7f6e641ef230 2023-10-30T13:53:12.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.411+0000 7f6e69b80640 1 -- 172.21.15.46:0/3235314722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e6404e170 con 0x7f6e641ef230 2023-10-30T13:53:12.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.411+0000 7f6e4bfff640 1 -- 172.21.15.46:0/3235314722 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f6e58086090 con 0x7f6e641ef230 2023-10-30T13:53:12.413 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:12.414 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:08.425371+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:resolve","state_seq":16,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:12.414 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 133 2023-10-30T13:53:12.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.411+0000 7f6e69b80640 1 -- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e500664d0 msgr2=0x7f6e50068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:12.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.411+0000 7f6e69b80640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e500664d0 0x7f6e50068990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009b60 tx=0x7f6e54008040 comp rx=0 tx=0).stop 2023-10-30T13:53:12.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.411+0000 7f6e69b80640 1 -- 172.21.15.46:0/3235314722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e641ef230 msgr2=0x7f6e64155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:12.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.411+0000 7f6e69b80640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e641ef230 0x7f6e64155fd0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f6e58011ad0 tx=0x7f6e58011f90 comp rx=0 tx=0).stop 2023-10-30T13:53:12.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.415+0000 7f6e69b80640 1 -- 172.21.15.46:0/3235314722 shutdown_connections 2023-10-30T13:53:12.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.415+0000 7f6e69b80640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e500664d0 0x7f6e50068990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:12.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.415+0000 7f6e69b80640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e641f22b0 0x7f6e641f2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:12.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.415+0000 7f6e69b80640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e641ef230 0x7f6e64155fd0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:12.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.415+0000 7f6e69b80640 1 --2- 172.21.15.46:0/3235314722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e4c004d10 0x7f6e640c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:12.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.415+0000 7f6e69b80640 1 -- 172.21.15.46:0/3235314722 >> 172.21.15.46:0/3235314722 conn(0x7f6e640c7910 msgr2=0x7f6e640c8130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:12.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.415+0000 7f6e69b80640 1 -- 172.21.15.46:0/3235314722 shutdown_connections 2023-10-30T13:53:12.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:12.415+0000 7f6e69b80640 1 -- 172.21.15.46:0/3235314722 wait complete. 2023-10-30T13:53:12.435 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 133, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 133, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7002": { "addr": "172.21.15.46:6833/2807435195", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2807435195, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2807435195, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7002, "incarnation": 125, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_7188": { "addr": "172.21.15.46:6836/711135361", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 711135361, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 711135361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7188, "incarnation": 132, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:resolve", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 72, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:53:08.425371+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7002, "mds_1": 7188 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:53:14.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b34d3e640 1 Processor -- start 2023-10-30T13:53:14.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b34d3e640 1 -- start start 2023-10-30T13:53:14.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b34d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3015d6f0 0x7f2b3015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:14.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b34d3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2b300cf160 0x7f2b300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:14.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b34d3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b300cb800 con 0x7f2b301538a0 2023-10-30T13:53:14.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b34d3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b300cb330 con 0x7f2b300cf160 2023-10-30T13:53:14.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b34d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b300cad70 con 0x7f2b3015d6f0 2023-10-30T13:53:14.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b2ed9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2b300cf160 0x7f2b300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:14.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b27fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3015d6f0 0x7f2b3015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:14.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b2e59c640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f2b301538a0 0x7f2b300ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:38154/0 (socket says 172.21.15.46:38154) 2023-10-30T13:53:14.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b2e59c640 1 -- 172.21.15.46:0/2660850052 learned_addr learned my addr 172.21.15.46:0/2660850052 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:53:14.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b27fff640 1 -- 172.21.15.46:0/2660850052 >> v2:172.21.15.154:3300/0 conn(0x7f2b300cf160 msgr2=0x7f2b300cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:14.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.587+0000 7f2b27fff640 1 --2- 172.21.15.46:0/2660850052 >> v2:172.21.15.154:3300/0 conn(0x7f2b300cf160 0x7f2b300cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:14.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.591+0000 7f2b27fff640 1 -- 172.21.15.46:0/2660850052 >> v1:172.21.15.154:6789/0 conn(0x7f2b301538a0 legacy=0x7f2b300ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:14.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.591+0000 7f2b27fff640 1 -- 172.21.15.46:0/2660850052 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b300cb050 con 0x7f2b3015d6f0 2023-10-30T13:53:14.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.591+0000 7f2b27fff640 1 --2- 172.21.15.46:0/2660850052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3015d6f0 0x7f2b3015a090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2b180097c0 tx=0x7f2b1801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=109da68479c8427f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:14.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.591+0000 7f2b2dd9b640 1 -- 172.21.15.46:0/2660850052 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b180a8970 con 0x7f2b3015d6f0 2023-10-30T13:53:14.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.591+0000 7f2b2dd9b640 1 -- 172.21.15.46:0/2660850052 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b180a8ad0 con 0x7f2b3015d6f0 2023-10-30T13:53:14.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.591+0000 7f2b2dd9b640 1 -- 172.21.15.46:0/2660850052 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b180a8e60 con 0x7f2b3015d6f0 2023-10-30T13:53:14.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.591+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2660850052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3015d6f0 msgr2=0x7f2b3015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:14.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.591+0000 7f2b34d3e640 1 --2- 172.21.15.46:0/2660850052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3015d6f0 0x7f2b3015a090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2b180097c0 tx=0x7f2b1801ec70 comp rx=0 tx=0).stop 2023-10-30T13:53:14.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.591+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2660850052 shutdown_connections 2023-10-30T13:53:14.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.591+0000 7f2b34d3e640 1 --2- 172.21.15.46:0/2660850052 >> v2:172.21.15.154:3300/0 conn(0x7f2b300cf160 0x7f2b300cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:14.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.591+0000 7f2b34d3e640 1 --2- 172.21.15.46:0/2660850052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3015d6f0 0x7f2b3015a090 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:14.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.591+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2660850052 >> 172.21.15.46:0/2660850052 conn(0x7f2b3006bfe0 msgr2=0x7f2b3006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:14.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2660850052 shutdown_connections 2023-10-30T13:53:14.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2660850052 wait complete. 2023-10-30T13:53:14.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b34d3e640 1 Processor -- start 2023-10-30T13:53:14.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b34d3e640 1 -- start start 2023-10-30T13:53:14.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b34d3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b300cf160 0x7f2b300b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:14.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b34d3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b301538a0 0x7f2b300c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:14.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b34d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3015d6f0 0x7f2b3014ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:14.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b34d3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b300c7d50 con 0x7f2b300cf160 2023-10-30T13:53:14.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b34d3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b300c92c0 con 0x7f2b301538a0 2023-10-30T13:53:14.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b34d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3014ea70 con 0x7f2b3015d6f0 2023-10-30T13:53:14.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b2e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b300cf160 0x7f2b300b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:14.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b2e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b300cf160 0x7f2b300b43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53364/0 (socket says 172.21.15.46:53364) 2023-10-30T13:53:14.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b2e59c640 1 -- 172.21.15.46:0/2383554204 learned_addr learned my addr 172.21.15.46:0/2383554204 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:14.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b2ed9d640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3015d6f0 0x7f2b3014ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:14.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b27fff640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b301538a0 0x7f2b300c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:14.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b2e59c640 1 -- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3015d6f0 msgr2=0x7f2b3014ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:14.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b2e59c640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3015d6f0 0x7f2b3014ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:14.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b2e59c640 1 -- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b301538a0 msgr2=0x7f2b300c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:14.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b2e59c640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b301538a0 0x7f2b300c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:14.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b2e59c640 1 -- 172.21.15.46:0/2383554204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b18009420 con 0x7f2b300cf160 2023-10-30T13:53:14.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b2ed9d640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3015d6f0 0x7f2b3014ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:14.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.595+0000 7f2b2e59c640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b300cf160 0x7f2b300b43c0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000c450 tx=0x7f2b2000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:14.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.599+0000 7f2b26ffd640 1 -- 172.21.15.46:0/2383554204 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b20005cf0 con 0x7f2b300cf160 2023-10-30T13:53:14.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.599+0000 7f2b26ffd640 1 -- 172.21.15.46:0/2383554204 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b20005e50 con 0x7f2b300cf160 2023-10-30T13:53:14.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.599+0000 7f2b26ffd640 1 -- 172.21.15.46:0/2383554204 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b20007290 con 0x7f2b300cf160 2023-10-30T13:53:14.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.599+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2383554204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b300c9510 con 0x7f2b300cf160 2023-10-30T13:53:14.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.599+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2383554204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b300c7a70 con 0x7f2b300cf160 2023-10-30T13:53:14.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.599+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2383554204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b300b2a50 con 0x7f2b300cf160 2023-10-30T13:53:14.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.599+0000 7f2b26ffd640 1 -- 172.21.15.46:0/2383554204 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2b200073f0 con 0x7f2b300cf160 2023-10-30T13:53:14.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.599+0000 7f2b26ffd640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b100664d0 0x7f2b10068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:14.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.599+0000 7f2b26ffd640 1 -- 172.21.15.46:0/2383554204 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f2b20083a70 con 0x7f2b300cf160 2023-10-30T13:53:14.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.599+0000 7f2b27fff640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b100664d0 0x7f2b10068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:14.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.603+0000 7f2b27fff640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b100664d0 0x7f2b10068990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2b180b2000 tx=0x7f2b180aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:14.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.603+0000 7f2b26ffd640 1 -- 172.21.15.46:0/2383554204 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2b20050820 con 0x7f2b300cf160 2023-10-30T13:53:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2383554204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b3004e170 con 0x7f2b300cf160 2023-10-30T13:53:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b26ffd640 1 -- 172.21.15.46:0/2383554204 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f2b20017070 con 0x7f2b300cf160 2023-10-30T13:53:14.797 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:14.797 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:08.425371+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:resolve","state_seq":16,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 133 2023-10-30T13:53:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b100664d0 msgr2=0x7f2b10068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b100664d0 0x7f2b10068990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2b180b2000 tx=0x7f2b180aa070 comp rx=0 tx=0).stop 2023-10-30T13:53:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2383554204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b300cf160 msgr2=0x7f2b300b43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b300cf160 0x7f2b300b43c0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000c450 tx=0x7f2b2000c910 comp rx=0 tx=0).stop 2023-10-30T13:53:14.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2383554204 shutdown_connections 2023-10-30T13:53:14.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b100664d0 0x7f2b10068990 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:14.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3015d6f0 0x7f2b3014ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:14.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b301538a0 0x7f2b300c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:14.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 --2- 172.21.15.46:0/2383554204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b300cf160 0x7f2b300b43c0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:14.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2383554204 >> 172.21.15.46:0/2383554204 conn(0x7f2b3006bfe0 msgr2=0x7f2b300cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:14.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2383554204 shutdown_connections 2023-10-30T13:53:14.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:14.795+0000 7f2b34d3e640 1 -- 172.21.15.46:0/2383554204 wait complete. 2023-10-30T13:53:14.818 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:53:16.819 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:16.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.971+0000 7fd84ad3e640 1 Processor -- start 2023-10-30T13:53:16.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.971+0000 7fd84ad3e640 1 -- start start 2023-10-30T13:53:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.971+0000 7fd84ad3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd844070a00 0x7fd844070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.971+0000 7fd84ad3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd844069e90 0x7fd84406a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.971+0000 7fd84ad3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8440713b0 con 0x7fd8441539a0 2023-10-30T13:53:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.971+0000 7fd84ad3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84406fbb0 con 0x7fd844070a00 2023-10-30T13:53:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.971+0000 7fd84ad3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd844153d80 con 0x7fd844069e90 2023-10-30T13:53:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.971+0000 7fd843fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd844070a00 0x7fd844070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.971+0000 7fd843fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd844070a00 0x7fd844070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47758/0 (socket says 172.21.15.46:47758) 2023-10-30T13:53:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.971+0000 7fd843fff640 1 -- 172.21.15.46:0/3111577637 learned_addr learned my addr 172.21.15.46:0/3111577637 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.971+0000 7fd8492db640 1 --2- 172.21.15.46:0/3111577637 >> v2:172.21.15.154:3300/0 conn(0x7fd844069e90 0x7fd84406a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:16.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd843fff640 1 -- 172.21.15.46:0/3111577637 >> v2:172.21.15.154:3300/0 conn(0x7fd844069e90 msgr2=0x7fd84406a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:16.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd843fff640 1 --2- 172.21.15.46:0/3111577637 >> v2:172.21.15.154:3300/0 conn(0x7fd844069e90 0x7fd84406a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd843fff640 1 -- 172.21.15.46:0/3111577637 >> v1:172.21.15.154:6789/0 conn(0x7fd8441539a0 legacy=0x7fd84415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd843fff640 1 -- 172.21.15.46:0/3111577637 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8440c5c50 con 0x7fd844070a00 2023-10-30T13:53:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd843fff640 1 --2- 172.21.15.46:0/3111577637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd844070a00 0x7fd844070e70 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fd8440b1080 tx=0x7fd83001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4444fcc50c8987 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd8437fe640 1 -- 172.21.15.46:0/3111577637 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd830004230 con 0x7fd844070a00 2023-10-30T13:53:16.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd8437fe640 1 -- 172.21.15.46:0/3111577637 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8300a7050 con 0x7fd844070a00 2023-10-30T13:53:16.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd8437fe640 1 -- 172.21.15.46:0/3111577637 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd830003620 con 0x7fd844070a00 2023-10-30T13:53:16.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd84ad3e640 1 -- 172.21.15.46:0/3111577637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd844070a00 msgr2=0x7fd844070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:16.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd84ad3e640 1 --2- 172.21.15.46:0/3111577637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd844070a00 0x7fd844070e70 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fd8440b1080 tx=0x7fd83001c440 comp rx=0 tx=0).stop 2023-10-30T13:53:16.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd84ad3e640 1 -- 172.21.15.46:0/3111577637 shutdown_connections 2023-10-30T13:53:16.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd84ad3e640 1 --2- 172.21.15.46:0/3111577637 >> v2:172.21.15.154:3300/0 conn(0x7fd844069e90 0x7fd84406a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:16.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd84ad3e640 1 --2- 172.21.15.46:0/3111577637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd844070a00 0x7fd844070e70 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:16.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd84ad3e640 1 -- 172.21.15.46:0/3111577637 >> 172.21.15.46:0/3111577637 conn(0x7fd8440c7910 msgr2=0x7fd8440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:16.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd84ad3e640 1 -- 172.21.15.46:0/3111577637 shutdown_connections 2023-10-30T13:53:16.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.975+0000 7fd84ad3e640 1 -- 172.21.15.46:0/3111577637 wait complete. 2023-10-30T13:53:16.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd84ad3e640 1 Processor -- start 2023-10-30T13:53:16.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd84ad3e640 1 -- start start 2023-10-30T13:53:16.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd84ad3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd844069e90 0x7fd844157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:16.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd84ad3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd844070a00 0x7fd8440ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:16.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd84ad3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8441539a0 0x7fd84413e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:16.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd84ad3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8440c03b0 con 0x7fd844069e90 2023-10-30T13:53:16.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd84ad3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8440bfec0 con 0x7fd8441539a0 2023-10-30T13:53:16.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd84ad3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8440c8c10 con 0x7fd844070a00 2023-10-30T13:53:16.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd848ada640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd844069e90 0x7fd844157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:16.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd848ada640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd844069e90 0x7fd844157770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47766/0 (socket says 172.21.15.46:47766) 2023-10-30T13:53:16.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd848ada640 1 -- 172.21.15.46:0/1826563677 learned_addr learned my addr 172.21.15.46:0/1826563677 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:16.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd8492db640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8441539a0 0x7fd84413e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:16.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd843fff640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd844070a00 0x7fd8440ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:16.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd848ada640 1 -- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd844070a00 msgr2=0x7fd8440ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:16.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd848ada640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd844070a00 0x7fd8440ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:16.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd848ada640 1 -- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8441539a0 msgr2=0x7fd84413e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:16.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd848ada640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8441539a0 0x7fd84413e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:16.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd848ada640 1 -- 172.21.15.46:0/1826563677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8300094d0 con 0x7fd844069e90 2023-10-30T13:53:16.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd843fff640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd844070a00 0x7fd8440ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd848ada640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd844069e90 0x7fd844157770 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fd83800c3c0 tx=0x7fd83800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.979+0000 7fd8417fa640 1 -- 172.21.15.46:0/1826563677 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8380052d0 con 0x7fd844069e90 2023-10-30T13:53:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.983+0000 7fd8417fa640 1 -- 172.21.15.46:0/1826563677 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd838005430 con 0x7fd844069e90 2023-10-30T13:53:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.983+0000 7fd8417fa640 1 -- 172.21.15.46:0/1826563677 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838003840 con 0x7fd844069e90 2023-10-30T13:53:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.983+0000 7fd84ad3e640 1 -- 172.21.15.46:0/1826563677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8440c00d0 con 0x7fd844069e90 2023-10-30T13:53:16.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.983+0000 7fd84ad3e640 1 -- 172.21.15.46:0/1826563677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8441441d0 con 0x7fd844069e90 2023-10-30T13:53:16.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.983+0000 7fd84ad3e640 1 -- 172.21.15.46:0/1826563677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd844143d50 con 0x7fd844069e90 2023-10-30T13:53:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.983+0000 7fd8417fa640 1 -- 172.21.15.46:0/1826563677 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd8380039a0 con 0x7fd844069e90 2023-10-30T13:53:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.983+0000 7fd8417fa640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8280664a0 0x7fd828068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.983+0000 7fd8417fa640 1 -- 172.21.15.46:0/1826563677 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7fd8380838a0 con 0x7fd844069e90 2023-10-30T13:53:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.983+0000 7fd843fff640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8280664a0 0x7fd828068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.987+0000 7fd843fff640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8280664a0 0x7fd828068960 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd8300ad000 tx=0x7fd830009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:16.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:16.987+0000 7fd8417fa640 1 -- 172.21.15.46:0/1826563677 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd844143d50 con 0x7fd844069e90 2023-10-30T13:53:17.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.175+0000 7fd84ad3e640 1 -- 172.21.15.46:0/1826563677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd844143d50 con 0x7fd844069e90 2023-10-30T13:53:17.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd8417fa640 1 -- 172.21.15.46:0/1826563677 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fd844143d50 con 0x7fd844069e90 2023-10-30T13:53:17.180 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:17.180 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:14.377230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:17.180 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 134 2023-10-30T13:53:17.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd84ad3e640 1 -- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8280664a0 msgr2=0x7fd828068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:17.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd84ad3e640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8280664a0 0x7fd828068960 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd8300ad000 tx=0x7fd830009040 comp rx=0 tx=0).stop 2023-10-30T13:53:17.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd84ad3e640 1 -- 172.21.15.46:0/1826563677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd844069e90 msgr2=0x7fd844157770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:17.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd84ad3e640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd844069e90 0x7fd844157770 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fd83800c3c0 tx=0x7fd83800c880 comp rx=0 tx=0).stop 2023-10-30T13:53:17.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd84ad3e640 1 -- 172.21.15.46:0/1826563677 shutdown_connections 2023-10-30T13:53:17.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd84ad3e640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8280664a0 0x7fd828068960 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:17.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd84ad3e640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8441539a0 0x7fd84413e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:17.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd84ad3e640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd844070a00 0x7fd8440ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:17.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd84ad3e640 1 --2- 172.21.15.46:0/1826563677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd844069e90 0x7fd844157770 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:17.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd84ad3e640 1 -- 172.21.15.46:0/1826563677 >> 172.21.15.46:0/1826563677 conn(0x7fd8440c7910 msgr2=0x7fd84414fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:17.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd84ad3e640 1 -- 172.21.15.46:0/1826563677 shutdown_connections 2023-10-30T13:53:17.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:17.179+0000 7fd84ad3e640 1 -- 172.21.15.46:0/1826563677 wait complete. 2023-10-30T13:53:17.201 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:53:19.202 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:19.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.355+0000 7f8ef9046640 1 Processor -- start 2023-10-30T13:53:19.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef9046640 1 -- start start 2023-10-30T13:53:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef9046640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8ef40cf160 0x7f8ef40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef9046640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef40cb800 con 0x7f8ef41538a0 2023-10-30T13:53:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef9046640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef40cb330 con 0x7f8ef415d6f0 2023-10-30T13:53:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef9046640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef40cad70 con 0x7f8ef40cf160 2023-10-30T13:53:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef2d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8ef41538a0 0x7f8ef40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51170/0 (socket says 172.21.15.46:51170) 2023-10-30T13:53:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef2d9d640 1 -- 172.21.15.46:0/1906569741 learned_addr learned my addr 172.21.15.46:0/1906569741 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:53:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef359e640 1 --2- 172.21.15.46:0/1906569741 >> v2:172.21.15.46:3300/0 conn(0x7f8ef40cf160 0x7f8ef40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef259c640 1 -- 172.21.15.46:0/1906569741 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151248541 0 0) 0x7f8ef40cb800 con 0x7f8ef41538a0 2023-10-30T13:53:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef259c640 1 -- 172.21.15.46:0/1906569741 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee0003530 con 0x7f8ef41538a0 2023-10-30T13:53:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef359e640 1 -- 172.21.15.46:0/1906569741 >> v1:172.21.15.154:6789/0 conn(0x7f8ef415d6f0 legacy=0x7f8ef415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef359e640 1 -- 172.21.15.46:0/1906569741 >> v1:172.21.15.46:6789/0 conn(0x7f8ef41538a0 legacy=0x7f8ef40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.359+0000 7f8ef359e640 1 -- 172.21.15.46:0/1906569741 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef40cb050 con 0x7f8ef40cf160 2023-10-30T13:53:19.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.363+0000 7f8ef359e640 1 --2- 172.21.15.46:0/1906569741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef40cf160 0x7f8ef40cf590 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009d90 tx=0x7f8eec01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e4b97d5f0b3c13 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:19.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.363+0000 7f8ef259c640 1 -- 172.21.15.46:0/1906569741 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0af600 con 0x7f8ef40cf160 2023-10-30T13:53:19.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.363+0000 7f8ef259c640 1 -- 172.21.15.46:0/1906569741 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec0ab050 con 0x7f8ef40cf160 2023-10-30T13:53:19.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.363+0000 7f8ef259c640 1 -- 172.21.15.46:0/1906569741 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0af980 con 0x7f8ef40cf160 2023-10-30T13:53:19.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.363+0000 7f8ef9046640 1 -- 172.21.15.46:0/1906569741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef40cf160 msgr2=0x7f8ef40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:19.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.363+0000 7f8ef9046640 1 --2- 172.21.15.46:0/1906569741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef40cf160 0x7f8ef40cf590 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009d90 tx=0x7f8eec01f130 comp rx=0 tx=0).stop 2023-10-30T13:53:19.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.363+0000 7f8ef9046640 1 -- 172.21.15.46:0/1906569741 shutdown_connections 2023-10-30T13:53:19.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.363+0000 7f8ef9046640 1 --2- 172.21.15.46:0/1906569741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef40cf160 0x7f8ef40cf590 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.363+0000 7f8ef9046640 1 -- 172.21.15.46:0/1906569741 >> 172.21.15.46:0/1906569741 conn(0x7f8ef406bfe0 msgr2=0x7f8ef406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.363+0000 7f8ef9046640 1 -- 172.21.15.46:0/1906569741 shutdown_connections 2023-10-30T13:53:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.363+0000 7f8ef9046640 1 -- 172.21.15.46:0/1906569741 wait complete. 2023-10-30T13:53:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.363+0000 7f8ef9046640 1 Processor -- start 2023-10-30T13:53:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef9046640 1 -- start start 2023-10-30T13:53:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef9046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ef40cf160 0x7f8ef40ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef9046640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef41538a0 0x7f8ef40b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:19.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef9046640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef415d6f0 0x7f8ef4148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:19.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef9046640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef4149a30 con 0x7f8ef415d6f0 2023-10-30T13:53:19.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef9046640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef40ce3a0 con 0x7f8ef41538a0 2023-10-30T13:53:19.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef9046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef4149cd0 con 0x7f8ef40cf160 2023-10-30T13:53:19.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef415d6f0 0x7f8ef4148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:19.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef415d6f0 0x7f8ef4148690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47798/0 (socket says 172.21.15.46:47798) 2023-10-30T13:53:19.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef359e640 1 -- 172.21.15.46:0/2413658899 learned_addr learned my addr 172.21.15.46:0/2413658899 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:19.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef2d9d640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ef40cf160 0x7f8ef40ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:19.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8eebfff640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef41538a0 0x7f8ef40b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:19.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef359e640 1 -- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ef40cf160 msgr2=0x7f8ef40ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:19.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef359e640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ef40cf160 0x7f8ef40ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:19.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef359e640 1 -- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef41538a0 msgr2=0x7f8ef40b3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:19.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef359e640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef41538a0 0x7f8ef40b3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:19.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef359e640 1 -- 172.21.15.46:0/2413658899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec009a70 con 0x7f8ef415d6f0 2023-10-30T13:53:19.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef359e640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef415d6f0 0x7f8ef4148690 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f8eec01cd60 tx=0x7f8eec01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:19.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8eeb7fe640 1 -- 172.21.15.46:0/2413658899 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0afb70 con 0x7f8ef415d6f0 2023-10-30T13:53:19.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8eeb7fe640 1 -- 172.21.15.46:0/2413658899 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec0afcd0 con 0x7f8ef415d6f0 2023-10-30T13:53:19.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8eeb7fe640 1 -- 172.21.15.46:0/2413658899 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0ba640 con 0x7f8ef415d6f0 2023-10-30T13:53:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef9046640 1 -- 172.21.15.46:0/2413658899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef4149fb0 con 0x7f8ef415d6f0 2023-10-30T13:53:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef9046640 1 -- 172.21.15.46:0/2413658899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef40ce680 con 0x7f8ef415d6f0 2023-10-30T13:53:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.367+0000 7f8ef9046640 1 -- 172.21.15.46:0/2413658899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef404e170 con 0x7f8ef415d6f0 2023-10-30T13:53:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.371+0000 7f8eeb7fe640 1 -- 172.21.15.46:0/2413658899 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8eec0ba7a0 con 0x7f8ef415d6f0 2023-10-30T13:53:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.371+0000 7f8eeb7fe640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ed40664a0 0x7f8ed4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.371+0000 7f8eeb7fe640 1 -- 172.21.15.46:0/2413658899 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f8eec122cb0 con 0x7f8ef415d6f0 2023-10-30T13:53:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.371+0000 7f8ef2d9d640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ed40664a0 0x7f8ed4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.371+0000 7f8ef2d9d640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ed40664a0 0x7f8ed4068960 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40011f0 tx=0x7f8ee4007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:19.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.375+0000 7f8eeb7fe640 1 -- 172.21.15.46:0/2413658899 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8eec0efb20 con 0x7f8ef415d6f0 2023-10-30T13:53:19.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.567+0000 7f8ef9046640 1 -- 172.21.15.46:0/2413658899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ef4148340 con 0x7f8ef415d6f0 2023-10-30T13:53:19.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.567+0000 7f8eeb7fe640 1 -- 172.21.15.46:0/2413658899 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f8eec0f3140 con 0x7f8ef415d6f0 2023-10-30T13:53:19.570 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:19.570 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:14.377230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:19.571 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 134 2023-10-30T13:53:19.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.571+0000 7f8ef9046640 1 -- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ed40664a0 msgr2=0x7f8ed4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:19.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.571+0000 7f8ef9046640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ed40664a0 0x7f8ed4068960 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40011f0 tx=0x7f8ee4007d40 comp rx=0 tx=0).stop 2023-10-30T13:53:19.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.571+0000 7f8ef9046640 1 -- 172.21.15.46:0/2413658899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef415d6f0 msgr2=0x7f8ef4148690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:19.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.571+0000 7f8ef9046640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef415d6f0 0x7f8ef4148690 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f8eec01cd60 tx=0x7f8eec01bed0 comp rx=0 tx=0).stop 2023-10-30T13:53:19.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.571+0000 7f8ef9046640 1 -- 172.21.15.46:0/2413658899 shutdown_connections 2023-10-30T13:53:19.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.571+0000 7f8ef9046640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ed40664a0 0x7f8ed4068960 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:19.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.571+0000 7f8ef9046640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef415d6f0 0x7f8ef4148690 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:19.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.571+0000 7f8ef9046640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef41538a0 0x7f8ef40b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:19.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.571+0000 7f8ef9046640 1 --2- 172.21.15.46:0/2413658899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ef40cf160 0x7f8ef40ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:19.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.571+0000 7f8ef9046640 1 -- 172.21.15.46:0/2413658899 >> 172.21.15.46:0/2413658899 conn(0x7f8ef406bfe0 msgr2=0x7f8ef40c5610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:19.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.571+0000 7f8ef9046640 1 -- 172.21.15.46:0/2413658899 shutdown_connections 2023-10-30T13:53:19.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:19.571+0000 7f8ef9046640 1 -- 172.21.15.46:0/2413658899 wait complete. 2023-10-30T13:53:19.591 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:53:21.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:21.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:21.592 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:21.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41c7254640 1 Processor -- start 2023-10-30T13:53:21.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41c7254640 1 -- start start 2023-10-30T13:53:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41c7254640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41c00c6b80 0x7f41c00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41c7254640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f41c00c6f60 0x7f41c00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41c7254640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f41c00c8430 0x7f41c013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41c7254640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c00c0a50 con 0x7f41c00c6f60 2023-10-30T13:53:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41c7254640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c006f6e0 con 0x7f41c00c8430 2023-10-30T13:53:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41c7254640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c006f9c0 con 0x7f41c00c6b80 2023-10-30T13:53:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41bffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f41c00c6f60 0x7f41c00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41bffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f41c00c6f60 0x7f41c00c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47808/0 (socket says 172.21.15.46:47808) 2023-10-30T13:53:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41bffff640 1 -- 172.21.15.46:0/3966948347 learned_addr learned my addr 172.21.15.46:0/3966948347 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41c4ff0640 1 --2- 172.21.15.46:0/3966948347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41c00c6b80 0x7f41c00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:21.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41c57f1640 1 --2- 172.21.15.46:0/3966948347 >> v2:172.21.15.154:3300/0 conn(0x7f41c00c8430 0x7f41c013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:21.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41bffff640 1 -- 172.21.15.46:0/3966948347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41c00c6b80 msgr2=0x7f41c00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:21.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41bffff640 1 --2- 172.21.15.46:0/3966948347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41c00c6b80 0x7f41c00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:21.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41bffff640 1 -- 172.21.15.46:0/3966948347 >> v2:172.21.15.154:3300/0 conn(0x7f41c00c8430 msgr2=0x7f41c013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:21.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41bffff640 1 --2- 172.21.15.46:0/3966948347 >> v2:172.21.15.154:3300/0 conn(0x7f41c00c8430 0x7f41c013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:21.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.743+0000 7f41bffff640 1 -- 172.21.15.46:0/3966948347 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c006fca0 con 0x7f41c00c6f60 2023-10-30T13:53:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41bffff640 1 --2- 172.21.15.46:0/3966948347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c00c6f60 0x7f41c00c0400 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f41c00b0860 tx=0x7f41ac01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb36851183f553c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41bf7fe640 1 -- 172.21.15.46:0/3966948347 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac0ac630 con 0x7f41c00c6f60 2023-10-30T13:53:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41bf7fe640 1 -- 172.21.15.46:0/3966948347 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac0ac790 con 0x7f41c00c6f60 2023-10-30T13:53:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41bf7fe640 1 -- 172.21.15.46:0/3966948347 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac0acb20 con 0x7f41c00c6f60 2023-10-30T13:53:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41c7254640 1 -- 172.21.15.46:0/3966948347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c00c6f60 msgr2=0x7f41c00c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41c7254640 1 --2- 172.21.15.46:0/3966948347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c00c6f60 0x7f41c00c0400 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f41c00b0860 tx=0x7f41ac01c500 comp rx=0 tx=0).stop 2023-10-30T13:53:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41c7254640 1 -- 172.21.15.46:0/3966948347 shutdown_connections 2023-10-30T13:53:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41c7254640 1 --2- 172.21.15.46:0/3966948347 >> v2:172.21.15.154:3300/0 conn(0x7f41c00c8430 0x7f41c013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41c7254640 1 --2- 172.21.15.46:0/3966948347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c00c6f60 0x7f41c00c0400 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41c7254640 1 --2- 172.21.15.46:0/3966948347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41c00c6b80 0x7f41c00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41c7254640 1 -- 172.21.15.46:0/3966948347 >> 172.21.15.46:0/3966948347 conn(0x7f41c00c7910 msgr2=0x7f41c00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41c7254640 1 -- 172.21.15.46:0/3966948347 shutdown_connections 2023-10-30T13:53:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.747+0000 7f41c7254640 1 -- 172.21.15.46:0/3966948347 wait complete. 2023-10-30T13:53:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41c7254640 1 Processor -- start 2023-10-30T13:53:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41c7254640 1 -- start start 2023-10-30T13:53:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41c7254640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41c00c6b80 0x7f41c0137930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41c7254640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c00c6f60 0x7f41c00c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:21.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41c7254640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41c00c8430 0x7f41c0150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:21.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41c7254640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c0132b10 con 0x7f41c00c6f60 2023-10-30T13:53:21.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41c7254640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c0134400 con 0x7f41c00c8430 2023-10-30T13:53:21.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41c7254640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c01350b0 con 0x7f41c00c6b80 2023-10-30T13:53:21.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41bffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c00c6f60 0x7f41c00c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:21.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41bffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c00c6f60 0x7f41c00c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47822/0 (socket says 172.21.15.46:47822) 2023-10-30T13:53:21.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41bffff640 1 -- 172.21.15.46:0/566260196 learned_addr learned my addr 172.21.15.46:0/566260196 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:21.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41c4ff0640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41c00c6b80 0x7f41c0137930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:21.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41c57f1640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41c00c8430 0x7f41c0150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:21.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41bffff640 1 -- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41c00c6b80 msgr2=0x7f41c0137930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:21.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41bffff640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41c00c6b80 0x7f41c0137930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:21.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41bffff640 1 -- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41c00c8430 msgr2=0x7f41c0150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:21.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41bffff640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41c00c8430 0x7f41c0150560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:21.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41bffff640 1 -- 172.21.15.46:0/566260196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac0094d0 con 0x7f41c00c6f60 2023-10-30T13:53:21.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41bffff640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c00c6f60 0x7f41c00c8c10 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f41ac01abb0 tx=0x7f41ac0192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:21.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41bd7fa640 1 -- 172.21.15.46:0/566260196 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac0ab7e0 con 0x7f41c00c6f60 2023-10-30T13:53:21.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41bd7fa640 1 -- 172.21.15.46:0/566260196 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac0ab940 con 0x7f41c00c6f60 2023-10-30T13:53:21.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41bd7fa640 1 -- 172.21.15.46:0/566260196 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac0abcb0 con 0x7f41c00c6f60 2023-10-30T13:53:21.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41c7254640 1 -- 172.21.15.46:0/566260196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c0132d40 con 0x7f41c00c6f60 2023-10-30T13:53:21.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.751+0000 7f41c7254640 1 -- 172.21.15.46:0/566260196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c0134a60 con 0x7f41c00c6f60 2023-10-30T13:53:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.755+0000 7f41c7254640 1 -- 172.21.15.46:0/566260196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c004e170 con 0x7f41c00c6f60 2023-10-30T13:53:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.755+0000 7f41bd7fa640 1 -- 172.21.15.46:0/566260196 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41ac0b3070 con 0x7f41c00c6f60 2023-10-30T13:53:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.755+0000 7f41bd7fa640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41a4066520 0x7f41a40689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.755+0000 7f41bd7fa640 1 -- 172.21.15.46:0/566260196 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f41ac11e6b0 con 0x7f41c00c6f60 2023-10-30T13:53:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.755+0000 7f41c4ff0640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41a4066520 0x7f41a40689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.755+0000 7f41c4ff0640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41a4066520 0x7f41a40689e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f41b40045f0 tx=0x7f41b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.759+0000 7f41bd7fa640 1 -- 172.21.15.46:0/566260196 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41ac0eb5a0 con 0x7f41c00c6f60 2023-10-30T13:53:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.951+0000 7f41c7254640 1 -- 172.21.15.46:0/566260196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41c01351f0 con 0x7f41c00c6f60 2023-10-30T13:53:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41bd7fa640 1 -- 172.21.15.46:0/566260196 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f41ac0eebc0 con 0x7f41c00c6f60 2023-10-30T13:53:21.956 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:21.956 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:20.353943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 135 2023-10-30T13:53:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41c7254640 1 -- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41a4066520 msgr2=0x7f41a40689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41c7254640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41a4066520 0x7f41a40689e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f41b40045f0 tx=0x7f41b4008040 comp rx=0 tx=0).stop 2023-10-30T13:53:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41c7254640 1 -- 172.21.15.46:0/566260196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c00c6f60 msgr2=0x7f41c00c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41c7254640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c00c6f60 0x7f41c00c8c10 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f41ac01abb0 tx=0x7f41ac0192a0 comp rx=0 tx=0).stop 2023-10-30T13:53:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41c7254640 1 -- 172.21.15.46:0/566260196 shutdown_connections 2023-10-30T13:53:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41c7254640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41a4066520 0x7f41a40689e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41c7254640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41c00c8430 0x7f41c0150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41c7254640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c00c6f60 0x7f41c00c8c10 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41c7254640 1 --2- 172.21.15.46:0/566260196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41c00c6b80 0x7f41c0137930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41c7254640 1 -- 172.21.15.46:0/566260196 >> 172.21.15.46:0/566260196 conn(0x7f41c00c7910 msgr2=0x7f41c006b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41c7254640 1 -- 172.21.15.46:0/566260196 shutdown_connections 2023-10-30T13:53:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:21.955+0000 7f41c7254640 1 -- 172.21.15.46:0/566260196 wait complete. 2023-10-30T13:53:21.977 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:53:23.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.131+0000 7f6b3ff9d640 1 Processor -- start 2023-10-30T13:53:24.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.131+0000 7f6b3ff9d640 1 -- start start 2023-10-30T13:53:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3ff9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b381580e0 0x7f6b38070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3ff9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b380c6540 0x7f6b3815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3ff9d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b380c5810 con 0x7f6b38069e90 2023-10-30T13:53:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3ff9d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b380c6960 con 0x7f6b381580e0 2023-10-30T13:53:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3ff9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380c6bb0 con 0x7f6b380c6540 2023-10-30T13:53:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3dd39640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b381580e0 0x7f6b38070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3dd39640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b381580e0 0x7f6b38070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47834/0 (socket says 172.21.15.46:47834) 2023-10-30T13:53:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3e53a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6b38069e90 0x7f6b3806a320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51186/0 (socket says 172.21.15.46:51186) 2023-10-30T13:53:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3dd39640 1 -- 172.21.15.46:0/2757341143 learned_addr learned my addr 172.21.15.46:0/2757341143 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3d538640 1 --2- 172.21.15.46:0/2757341143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b380c6540 0x7f6b3815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3dd39640 1 -- 172.21.15.46:0/2757341143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b380c6540 msgr2=0x7f6b3815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3dd39640 1 --2- 172.21.15.46:0/2757341143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b380c6540 0x7f6b3815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3dd39640 1 -- 172.21.15.46:0/2757341143 >> v1:172.21.15.46:6789/0 conn(0x7f6b38069e90 legacy=0x7f6b3806a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3dd39640 1 -- 172.21.15.46:0/2757341143 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b380c6e70 con 0x7f6b381580e0 2023-10-30T13:53:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.135+0000 7f6b3dd39640 1 --2- 172.21.15.46:0/2757341143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b381580e0 0x7f6b38070740 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009870 tx=0x7f6b3401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd9c6fec7ca6f25 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3cd37640 1 -- 172.21.15.46:0/2757341143 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b340a9400 con 0x7f6b381580e0 2023-10-30T13:53:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3cd37640 1 -- 172.21.15.46:0/2757341143 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b340a9560 con 0x7f6b381580e0 2023-10-30T13:53:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3cd37640 1 -- 172.21.15.46:0/2757341143 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b340a98f0 con 0x7f6b381580e0 2023-10-30T13:53:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/2757341143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b381580e0 msgr2=0x7f6b38070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 --2- 172.21.15.46:0/2757341143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b381580e0 0x7f6b38070740 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009870 tx=0x7f6b3401c540 comp rx=0 tx=0).stop 2023-10-30T13:53:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/2757341143 shutdown_connections 2023-10-30T13:53:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 --2- 172.21.15.46:0/2757341143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b380c6540 0x7f6b3815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 --2- 172.21.15.46:0/2757341143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b381580e0 0x7f6b38070740 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/2757341143 >> 172.21.15.46:0/2757341143 conn(0x7f6b380c7910 msgr2=0x7f6b380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/2757341143 shutdown_connections 2023-10-30T13:53:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/2757341143 wait complete. 2023-10-30T13:53:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 Processor -- start 2023-10-30T13:53:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 -- start start 2023-10-30T13:53:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b38069e90 0x7f6b381fa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380c6540 0x7f6b38071f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b381580e0 0x7f6b381613a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3815faf0 con 0x7f6b38069e90 2023-10-30T13:53:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b381faa30 con 0x7f6b380c6540 2023-10-30T13:53:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.139+0000 7f6b3ff9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b38070760 con 0x7f6b381580e0 2023-10-30T13:53:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3d538640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380c6540 0x7f6b38071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3d538640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380c6540 0x7f6b38071f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39462/0 (socket says 172.21.15.46:39462) 2023-10-30T13:53:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3d538640 1 -- 172.21.15.46:0/1483637017 learned_addr learned my addr 172.21.15.46:0/1483637017 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3e53a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b381580e0 0x7f6b381613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3dd39640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b38069e90 0x7f6b381fa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:24.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3dd39640 1 -- 172.21.15.46:0/1483637017 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b381580e0 msgr2=0x7f6b381613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:24.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3dd39640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b381580e0 0x7f6b381613a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:24.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3dd39640 1 -- 172.21.15.46:0/1483637017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380c6540 msgr2=0x7f6b38071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:24.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3dd39640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380c6540 0x7f6b38071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:24.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3dd39640 1 -- 172.21.15.46:0/1483637017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b340094d0 con 0x7f6b38069e90 2023-10-30T13:53:24.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3dd39640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b38069e90 0x7f6b381fa4f0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f6b340aa000 tx=0x7f6b340192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:24.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b2e7fc640 1 -- 172.21.15.46:0/1483637017 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b340a9400 con 0x7f6b38069e90 2023-10-30T13:53:24.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b2e7fc640 1 -- 172.21.15.46:0/1483637017 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b340b5070 con 0x7f6b38069e90 2023-10-30T13:53:24.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b2e7fc640 1 -- 172.21.15.46:0/1483637017 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b340b1a60 con 0x7f6b38069e90 2023-10-30T13:53:24.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/1483637017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b38072440 con 0x7f6b38069e90 2023-10-30T13:53:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/1483637017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3815f1c0 con 0x7f6b38069e90 2023-10-30T13:53:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/1483637017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3804e170 con 0x7f6b38069e90 2023-10-30T13:53:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b2e7fc640 1 -- 172.21.15.46:0/1483637017 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b340b1bc0 con 0x7f6b38069e90 2023-10-30T13:53:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b2e7fc640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b1c0664a0 0x7f6b1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.143+0000 7f6b2e7fc640 1 -- 172.21.15.46:0/1483637017 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f6b340b7070 con 0x7f6b38069e90 2023-10-30T13:53:24.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.147+0000 7f6b3d538640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b1c0664a0 0x7f6b1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:24.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.147+0000 7f6b3d538640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b1c0664a0 0x7f6b1c068960 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6b3814e570 tx=0x7f6b28009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:24.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.147+0000 7f6b2e7fc640 1 -- 172.21.15.46:0/1483637017 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6b340eb780 con 0x7f6b38069e90 2023-10-30T13:53:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.339+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/1483637017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b38153e90 con 0x7f6b38069e90 2023-10-30T13:53:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b2e7fc640 1 -- 172.21.15.46:0/1483637017 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f6b340a6020 con 0x7f6b38069e90 2023-10-30T13:53:24.344 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:24.344 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:20.353943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:24.344 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 135 2023-10-30T13:53:24.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/1483637017 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b1c0664a0 msgr2=0x7f6b1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:24.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b3ff9d640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b1c0664a0 0x7f6b1c068960 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6b3814e570 tx=0x7f6b28009190 comp rx=0 tx=0).stop 2023-10-30T13:53:24.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/1483637017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b38069e90 msgr2=0x7f6b381fa4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:24.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b3ff9d640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b38069e90 0x7f6b381fa4f0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f6b340aa000 tx=0x7f6b340192e0 comp rx=0 tx=0).stop 2023-10-30T13:53:24.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/1483637017 shutdown_connections 2023-10-30T13:53:24.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b3ff9d640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b1c0664a0 0x7f6b1c068960 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:24.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b3ff9d640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b381580e0 0x7f6b381613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:24.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b3ff9d640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380c6540 0x7f6b38071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:24.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b3ff9d640 1 --2- 172.21.15.46:0/1483637017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b38069e90 0x7f6b381fa4f0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:24.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/1483637017 >> 172.21.15.46:0/1483637017 conn(0x7f6b380c7910 msgr2=0x7f6b3806a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:24.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/1483637017 shutdown_connections 2023-10-30T13:53:24.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:24.343+0000 7f6b3ff9d640 1 -- 172.21.15.46:0/1483637017 wait complete. 2023-10-30T13:53:24.365 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 135, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 135, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7002": { "addr": "172.21.15.46:6833/2807435195", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2807435195, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2807435195, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7002, "incarnation": 125, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_7188": { "addr": "172.21.15.46:6836/711135361", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 711135361, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 711135361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7188, "incarnation": 132, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:rejoin", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 72, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:53:20.353943+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7002, "mds_1": 7188 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:53:26.367 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:26.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b98ea640 1 Processor -- start 2023-10-30T13:53:26.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b98ea640 1 -- start start 2023-10-30T13:53:26.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b98ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f45b41580e0 0x7f45b4070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:26.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b98ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45b415e700 0x7f45b406a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:26.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b98ea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b40c5810 con 0x7f45b40c6540 2023-10-30T13:53:26.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b98ea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b40c6990 con 0x7f45b415e700 2023-10-30T13:53:26.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b98ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b415eb90 con 0x7f45b41580e0 2023-10-30T13:53:26.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45b415e700 0x7f45b406a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:26.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45b415e700 0x7f45b406a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46672/0 (socket says 172.21.15.46:46672) 2023-10-30T13:53:26.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b37fe640 1 -- 172.21.15.46:0/687808438 learned_addr learned my addr 172.21.15.46:0/687808438 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:26.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b2ffd640 1 --2- 172.21.15.46:0/687808438 >> v2:172.21.15.154:3300/0 conn(0x7f45b41580e0 0x7f45b4070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:26.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b37fe640 1 -- 172.21.15.46:0/687808438 >> v2:172.21.15.154:3300/0 conn(0x7f45b41580e0 msgr2=0x7f45b4070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:26.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b37fe640 1 --2- 172.21.15.46:0/687808438 >> v2:172.21.15.154:3300/0 conn(0x7f45b41580e0 0x7f45b4070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:26.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b37fe640 1 -- 172.21.15.46:0/687808438 >> v1:172.21.15.154:6789/0 conn(0x7f45b40c6540 legacy=0x7f45b4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:26.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.511+0000 7f45b37fe640 1 -- 172.21.15.46:0/687808438 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b40c6bb0 con 0x7f45b415e700 2023-10-30T13:53:26.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.515+0000 7f45b37fe640 1 --2- 172.21.15.46:0/687808438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b415e700 0x7f45b406a5a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f45b406eea0 tx=0x7f45ac01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18af7973eddfd281 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:26.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.515+0000 7f45b27fc640 1 -- 172.21.15.46:0/687808438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac0a5860 con 0x7f45b415e700 2023-10-30T13:53:26.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.515+0000 7f45b27fc640 1 -- 172.21.15.46:0/687808438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac0a59c0 con 0x7f45b415e700 2023-10-30T13:53:26.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.515+0000 7f45b27fc640 1 -- 172.21.15.46:0/687808438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac0a5d50 con 0x7f45b415e700 2023-10-30T13:53:26.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.515+0000 7f45b98ea640 1 -- 172.21.15.46:0/687808438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b415e700 msgr2=0x7f45b406a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:26.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.515+0000 7f45b98ea640 1 --2- 172.21.15.46:0/687808438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b415e700 0x7f45b406a5a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f45b406eea0 tx=0x7f45ac01c9a0 comp rx=0 tx=0).stop 2023-10-30T13:53:26.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.515+0000 7f45b98ea640 1 -- 172.21.15.46:0/687808438 shutdown_connections 2023-10-30T13:53:26.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.515+0000 7f45b98ea640 1 --2- 172.21.15.46:0/687808438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b415e700 0x7f45b406a5a0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:26.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.515+0000 7f45b98ea640 1 --2- 172.21.15.46:0/687808438 >> v2:172.21.15.154:3300/0 conn(0x7f45b41580e0 0x7f45b4070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:26.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.515+0000 7f45b98ea640 1 -- 172.21.15.46:0/687808438 >> 172.21.15.46:0/687808438 conn(0x7f45b40c7910 msgr2=0x7f45b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.515+0000 7f45b98ea640 1 -- 172.21.15.46:0/687808438 shutdown_connections 2023-10-30T13:53:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.515+0000 7f45b98ea640 1 -- 172.21.15.46:0/687808438 wait complete. 2023-10-30T13:53:26.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b98ea640 1 Processor -- start 2023-10-30T13:53:26.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b98ea640 1 -- start start 2023-10-30T13:53:26.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b98ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b40c6540 0x7f45b40c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:26.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b98ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b41580e0 0x7f45b40c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:26.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b98ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b415e700 0x7f45b40c2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:26.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b98ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4148670 con 0x7f45b40c6540 2023-10-30T13:53:26.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b98ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4148400 con 0x7f45b415e700 2023-10-30T13:53:26.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b98ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b40c20b0 con 0x7f45b41580e0 2023-10-30T13:53:26.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b40c6540 0x7f45b40c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:26.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b40c6540 0x7f45b40c8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46688/0 (socket says 172.21.15.46:46688) 2023-10-30T13:53:26.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b2ffd640 1 -- 172.21.15.46:0/4015409178 learned_addr learned my addr 172.21.15.46:0/4015409178 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:26.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45abfff640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b41580e0 0x7f45b40c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:26.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b37fe640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b415e700 0x7f45b40c2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:26.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b2ffd640 1 -- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b41580e0 msgr2=0x7f45b40c25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:26.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b2ffd640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b41580e0 0x7f45b40c25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:26.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b2ffd640 1 -- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b415e700 msgr2=0x7f45b40c2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:26.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b2ffd640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b415e700 0x7f45b40c2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:26.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b2ffd640 1 -- 172.21.15.46:0/4015409178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac009b30 con 0x7f45b40c6540 2023-10-30T13:53:26.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b2ffd640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b40c6540 0x7f45b40c8f20 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f45a400d800 tx=0x7f45a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:26.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b0ff9640 1 -- 172.21.15.46:0/4015409178 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a400b6c0 con 0x7f45b40c6540 2023-10-30T13:53:26.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b0ff9640 1 -- 172.21.15.46:0/4015409178 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a400b820 con 0x7f45b40c6540 2023-10-30T13:53:26.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b0ff9640 1 -- 172.21.15.46:0/4015409178 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a400bb90 con 0x7f45b40c6540 2023-10-30T13:53:26.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b98ea640 1 -- 172.21.15.46:0/4015409178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b41493b0 con 0x7f45b40c6540 2023-10-30T13:53:26.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.519+0000 7f45b98ea640 1 -- 172.21.15.46:0/4015409178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b4149970 con 0x7f45b40c6540 2023-10-30T13:53:26.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.523+0000 7f45b98ea640 1 -- 172.21.15.46:0/4015409178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b4149b00 con 0x7f45b40c6540 2023-10-30T13:53:26.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.523+0000 7f45b0ff9640 1 -- 172.21.15.46:0/4015409178 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f45a400bcf0 con 0x7f45b40c6540 2023-10-30T13:53:26.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.523+0000 7f45b0ff9640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45940664a0 0x7f4594068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:26.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.523+0000 7f45b0ff9640 1 -- 172.21.15.46:0/4015409178 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f45a4084ae0 con 0x7f45b40c6540 2023-10-30T13:53:26.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.523+0000 7f45abfff640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45940664a0 0x7f4594068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:26.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.523+0000 7f45abfff640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45940664a0 0x7f4594068960 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f45b4072210 tx=0x7f459c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:26.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.527+0000 7f45b0ff9640 1 -- 172.21.15.46:0/4015409178 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f45a4051890 con 0x7f45b40c6540 2023-10-30T13:53:26.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.715+0000 7f45b98ea640 1 -- 172.21.15.46:0/4015409178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45b404e170 con 0x7f45b40c6540 2023-10-30T13:53:26.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b0ff9640 1 -- 172.21.15.46:0/4015409178 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f45a4011340 con 0x7f45b40c6540 2023-10-30T13:53:26.719 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:26.720 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:20.353943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:26.720 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 135 2023-10-30T13:53:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b98ea640 1 -- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45940664a0 msgr2=0x7f4594068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b98ea640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45940664a0 0x7f4594068960 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f45b4072210 tx=0x7f459c009040 comp rx=0 tx=0).stop 2023-10-30T13:53:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b98ea640 1 -- 172.21.15.46:0/4015409178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b40c6540 msgr2=0x7f45b40c8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b98ea640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b40c6540 0x7f45b40c8f20 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f45a400d800 tx=0x7f45a400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:53:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b98ea640 1 -- 172.21.15.46:0/4015409178 shutdown_connections 2023-10-30T13:53:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b98ea640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45940664a0 0x7f4594068960 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b98ea640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b415e700 0x7f45b40c2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b98ea640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b41580e0 0x7f45b40c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b98ea640 1 --2- 172.21.15.46:0/4015409178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b40c6540 0x7f45b40c8f20 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b98ea640 1 -- 172.21.15.46:0/4015409178 >> 172.21.15.46:0/4015409178 conn(0x7f45b40c7910 msgr2=0x7f45b4156eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b98ea640 1 -- 172.21.15.46:0/4015409178 shutdown_connections 2023-10-30T13:53:26.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:26.719+0000 7f45b98ea640 1 -- 172.21.15.46:0/4015409178 wait complete. 2023-10-30T13:53:26.741 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:53:28.299 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-10-30T13:53:28.295+0000 7f3940f23640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:53:28.742 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:28.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a9c98d640 1 Processor -- start 2023-10-30T13:53:28.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a9c98d640 1 -- start start 2023-10-30T13:53:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a9c98d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a980c6bd0 0x7f1a9806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a9c98d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a98071110 0x7f1a98158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a9c98d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a98066cf0 con 0x7f1a9806f7a0 2023-10-30T13:53:28.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a9c98d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a980700c0 con 0x7f1a980c6bd0 2023-10-30T13:53:28.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a9c98d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a980c70d0 con 0x7f1a98071110 2023-10-30T13:53:28.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a8ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a980c6bd0 0x7f1a9806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a9659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1a9806f7a0 0x7f1a98069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47910/0 (socket says 172.21.15.46:47910) 2023-10-30T13:53:28.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a9659c640 1 -- 172.21.15.46:0/3722244974 learned_addr learned my addr 172.21.15.46:0/3722244974 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:53:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a96d9d640 1 --2- 172.21.15.46:0/3722244974 >> v2:172.21.15.154:3300/0 conn(0x7f1a98071110 0x7f1a98158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a8ffff640 1 -- 172.21.15.46:0/3722244974 >> v2:172.21.15.154:3300/0 conn(0x7f1a98071110 msgr2=0x7f1a98158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a8ffff640 1 --2- 172.21.15.46:0/3722244974 >> v2:172.21.15.154:3300/0 conn(0x7f1a98071110 0x7f1a98158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a8ffff640 1 -- 172.21.15.46:0/3722244974 >> v1:172.21.15.46:6789/0 conn(0x7f1a9806f7a0 legacy=0x7f1a98069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a8ffff640 1 -- 172.21.15.46:0/3722244974 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a980c7390 con 0x7f1a980c6bd0 2023-10-30T13:53:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.895+0000 7f1a95d9b640 1 -- 172.21.15.46:0/3722244974 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116823169 0 0) 0x7f1a98066cf0 con 0x7f1a9806f7a0 2023-10-30T13:53:28.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.899+0000 7f1a8ffff640 1 --2- 172.21.15.46:0/3722244974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a980c6bd0 0x7f1a9806fb80 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f1a9806cdd0 tx=0x7f1a8001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9a8be90e46660c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:28.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.899+0000 7f1a95d9b640 1 -- 172.21.15.46:0/3722244974 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a800a8950 con 0x7f1a980c6bd0 2023-10-30T13:53:28.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.899+0000 7f1a95d9b640 1 -- 172.21.15.46:0/3722244974 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a800a8ab0 con 0x7f1a980c6bd0 2023-10-30T13:53:28.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.899+0000 7f1a95d9b640 1 -- 172.21.15.46:0/3722244974 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a800a8e40 con 0x7f1a980c6bd0 2023-10-30T13:53:28.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.899+0000 7f1a9c98d640 1 -- 172.21.15.46:0/3722244974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a980c6bd0 msgr2=0x7f1a9806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:28.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.899+0000 7f1a9c98d640 1 --2- 172.21.15.46:0/3722244974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a980c6bd0 0x7f1a9806fb80 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f1a9806cdd0 tx=0x7f1a8001ec50 comp rx=0 tx=0).stop 2023-10-30T13:53:28.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.899+0000 7f1a9c98d640 1 -- 172.21.15.46:0/3722244974 shutdown_connections 2023-10-30T13:53:28.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.899+0000 7f1a9c98d640 1 --2- 172.21.15.46:0/3722244974 >> v2:172.21.15.154:3300/0 conn(0x7f1a98071110 0x7f1a98158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.899+0000 7f1a9c98d640 1 --2- 172.21.15.46:0/3722244974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a980c6bd0 0x7f1a9806fb80 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.899+0000 7f1a9c98d640 1 -- 172.21.15.46:0/3722244974 >> 172.21.15.46:0/3722244974 conn(0x7f1a980c7910 msgr2=0x7f1a980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.899+0000 7f1a9c98d640 1 -- 172.21.15.46:0/3722244974 shutdown_connections 2023-10-30T13:53:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.899+0000 7f1a9c98d640 1 -- 172.21.15.46:0/3722244974 wait complete. 2023-10-30T13:53:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9c98d640 1 Processor -- start 2023-10-30T13:53:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9c98d640 1 -- start start 2023-10-30T13:53:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9c98d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a9806f7a0 0x7f1a98156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9c98d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a98071110 0x7f1a981f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9c98d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a980c6bd0 0x7f1a9814f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9c98d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a981f1f00 con 0x7f1a9806f7a0 2023-10-30T13:53:28.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9c98d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a980afbf0 con 0x7f1a98071110 2023-10-30T13:53:28.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9c98d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a981eedc0 con 0x7f1a980c6bd0 2023-10-30T13:53:28.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a9806f7a0 0x7f1a98156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a9806f7a0 0x7f1a98156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46712/0 (socket says 172.21.15.46:46712) 2023-10-30T13:53:28.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9659c640 1 -- 172.21.15.46:0/1370615772 learned_addr learned my addr 172.21.15.46:0/1370615772 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a96d9d640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a980c6bd0 0x7f1a9814f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a8ffff640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a98071110 0x7f1a981f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9659c640 1 -- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a980c6bd0 msgr2=0x7f1a9814f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9659c640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a980c6bd0 0x7f1a9814f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9659c640 1 -- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a98071110 msgr2=0x7f1a981f2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9659c640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a98071110 0x7f1a981f2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9659c640 1 -- 172.21.15.46:0/1370615772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a80009420 con 0x7f1a9806f7a0 2023-10-30T13:53:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9659c640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a9806f7a0 0x7f1a98156a30 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800c480 tx=0x7f1a8800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a8effd640 1 -- 172.21.15.46:0/1370615772 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a880074e0 con 0x7f1a9806f7a0 2023-10-30T13:53:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a8effd640 1 -- 172.21.15.46:0/1370615772 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a88007660 con 0x7f1a9806f7a0 2023-10-30T13:53:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9c98d640 1 -- 172.21.15.46:0/1370615772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a981ef0a0 con 0x7f1a9806f7a0 2023-10-30T13:53:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a8effd640 1 -- 172.21.15.46:0/1370615772 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a88004a30 con 0x7f1a9806f7a0 2023-10-30T13:53:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.903+0000 7f1a9c98d640 1 -- 172.21.15.46:0/1370615772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a981ef660 con 0x7f1a9806f7a0 2023-10-30T13:53:28.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.907+0000 7f1a9c98d640 1 -- 172.21.15.46:0/1370615772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a9804e170 con 0x7f1a9806f7a0 2023-10-30T13:53:28.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.907+0000 7f1a8effd640 1 -- 172.21.15.46:0/1370615772 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1a88004b90 con 0x7f1a9806f7a0 2023-10-30T13:53:28.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.907+0000 7f1a8effd640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a780664d0 0x7f1a78068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:28.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.907+0000 7f1a8effd640 1 -- 172.21.15.46:0/1370615772 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f1a88083870 con 0x7f1a9806f7a0 2023-10-30T13:53:28.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.907+0000 7f1a8ffff640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a780664d0 0x7f1a78068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:28.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.907+0000 7f1a8ffff640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a780664d0 0x7f1a78068990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1a800b2000 tx=0x7f1a800aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:28.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:28.911+0000 7f1a8effd640 1 -- 172.21.15.46:0/1370615772 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a88054280 con 0x7f1a9806f7a0 2023-10-30T13:53:29.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.099+0000 7f1a9c98d640 1 -- 172.21.15.46:0/1370615772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a981510d0 con 0x7f1a9806f7a0 2023-10-30T13:53:29.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.099+0000 7f1a8effd640 1 -- 172.21.15.46:0/1370615772 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f1a88053e10 con 0x7f1a9806f7a0 2023-10-30T13:53:29.103 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:29.103 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:26.395496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:29.103 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 136 2023-10-30T13:53:29.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.103+0000 7f1a9c98d640 1 -- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a780664d0 msgr2=0x7f1a78068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:29.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.103+0000 7f1a9c98d640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a780664d0 0x7f1a78068990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1a800b2000 tx=0x7f1a800aa070 comp rx=0 tx=0).stop 2023-10-30T13:53:29.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.103+0000 7f1a9c98d640 1 -- 172.21.15.46:0/1370615772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a9806f7a0 msgr2=0x7f1a98156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:29.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.103+0000 7f1a9c98d640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a9806f7a0 0x7f1a98156a30 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800c480 tx=0x7f1a8800c940 comp rx=0 tx=0).stop 2023-10-30T13:53:29.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.103+0000 7f1a9c98d640 1 -- 172.21.15.46:0/1370615772 shutdown_connections 2023-10-30T13:53:29.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.103+0000 7f1a9c98d640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a780664d0 0x7f1a78068990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.103+0000 7f1a9c98d640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a980c6bd0 0x7f1a9814f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.103+0000 7f1a9c98d640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a98071110 0x7f1a981f2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.103+0000 7f1a9c98d640 1 --2- 172.21.15.46:0/1370615772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a9806f7a0 0x7f1a98156a30 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:29.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.103+0000 7f1a9c98d640 1 -- 172.21.15.46:0/1370615772 >> 172.21.15.46:0/1370615772 conn(0x7f1a980c7910 msgr2=0x7f1a980c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:29.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.103+0000 7f1a9c98d640 1 -- 172.21.15.46:0/1370615772 shutdown_connections 2023-10-30T13:53:29.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:29.103+0000 7f1a9c98d640 1 -- 172.21.15.46:0/1370615772 wait complete. 2023-10-30T13:53:29.124 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=6858 2023-10-30T13:53:29.124 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.e 2023-10-30T13:53:31.125 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T13:53:31.125 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:53:31.126 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:53:31.130 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:53:31.130 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:31.228 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T13:53:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.287+0000 7fe4fae17640 1 Processor -- start 2023-10-30T13:53:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.287+0000 7fe4fae17640 1 -- start start 2023-10-30T13:53:31.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.287+0000 7fe4fae17640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe4f40cf160 0x7fe4f40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:31.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.287+0000 7fe4fae17640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f40cb800 con 0x7fe4f415d6f0 2023-10-30T13:53:31.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.287+0000 7fe4fae17640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f40cb330 con 0x7fe4f41538a0 2023-10-30T13:53:31.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.287+0000 7fe4fae17640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f40cad70 con 0x7fe4f40cf160 2023-10-30T13:53:31.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.287+0000 7fe4f3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe4f415d6f0 0x7fe4f415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47916/0 (socket says 172.21.15.46:47916) 2023-10-30T13:53:31.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.287+0000 7fe4f3fff640 1 -- 172.21.15.46:0/2327876028 learned_addr learned my addr 172.21.15.46:0/2327876028 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:53:31.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.287+0000 7fe4f93b4640 1 --2- 172.21.15.46:0/2327876028 >> v2:172.21.15.154:3300/0 conn(0x7fe4f40cf160 0x7fe4f40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.291+0000 7fe4f37fe640 1 -- 172.21.15.46:0/2327876028 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942861289 0 0) 0x7fe4f40cb800 con 0x7fe4f415d6f0 2023-10-30T13:53:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.291+0000 7fe4f37fe640 1 -- 172.21.15.46:0/2327876028 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e4003530 con 0x7fe4f415d6f0 2023-10-30T13:53:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.291+0000 7fe4f93b4640 1 -- 172.21.15.46:0/2327876028 >> v1:172.21.15.154:6789/0 conn(0x7fe4f41538a0 legacy=0x7fe4f40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.291+0000 7fe4f93b4640 1 -- 172.21.15.46:0/2327876028 >> v1:172.21.15.46:6789/0 conn(0x7fe4f415d6f0 legacy=0x7fe4f415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.291+0000 7fe4f93b4640 1 -- 172.21.15.46:0/2327876028 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f40cb050 con 0x7fe4f40cf160 2023-10-30T13:53:31.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.291+0000 7fe4f93b4640 1 --2- 172.21.15.46:0/2327876028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f40cf160 0x7fe4f40cf590 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec009d90 tx=0x7fe4ec01f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7de46ac146a9a9d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:31.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.291+0000 7fe4f37fe640 1 -- 172.21.15.46:0/2327876028 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec0af600 con 0x7fe4f40cf160 2023-10-30T13:53:31.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.291+0000 7fe4f37fe640 1 -- 172.21.15.46:0/2327876028 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec0ab050 con 0x7fe4f40cf160 2023-10-30T13:53:31.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.291+0000 7fe4f37fe640 1 -- 172.21.15.46:0/2327876028 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec0af960 con 0x7fe4f40cf160 2023-10-30T13:53:31.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 -- 172.21.15.46:0/2327876028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f40cf160 msgr2=0x7fe4f40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:31.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 --2- 172.21.15.46:0/2327876028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f40cf160 0x7fe4f40cf590 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec009d90 tx=0x7fe4ec01f130 comp rx=0 tx=0).stop 2023-10-30T13:53:31.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 -- 172.21.15.46:0/2327876028 shutdown_connections 2023-10-30T13:53:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 --2- 172.21.15.46:0/2327876028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f40cf160 0x7fe4f40cf590 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 -- 172.21.15.46:0/2327876028 >> 172.21.15.46:0/2327876028 conn(0x7fe4f406bfe0 msgr2=0x7fe4f406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 -- 172.21.15.46:0/2327876028 shutdown_connections 2023-10-30T13:53:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 -- 172.21.15.46:0/2327876028 wait complete. 2023-10-30T13:53:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 Processor -- start 2023-10-30T13:53:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 -- start start 2023-10-30T13:53:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4f40cf160 0x7fe4f41503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f41538a0 0x7fe4f40c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4f415d6f0 0x7fe4f40c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f40c98b0 con 0x7fe4f40cf160 2023-10-30T13:53:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f40c64d0 con 0x7fe4f41538a0 2023-10-30T13:53:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4fae17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f414efd0 con 0x7fe4f415d6f0 2023-10-30T13:53:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4f8bb3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4f40cf160 0x7fe4f41503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4f8bb3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4f40cf160 0x7fe4f41503f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46726/0 (socket says 172.21.15.46:46726) 2023-10-30T13:53:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4f8bb3640 1 -- 172.21.15.46:0/3507704639 learned_addr learned my addr 172.21.15.46:0/3507704639 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:53:31.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4f3fff640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f41538a0 0x7fe4f40c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:31.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4f93b4640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4f415d6f0 0x7fe4f40c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:31.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4f8bb3640 1 -- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4f415d6f0 msgr2=0x7fe4f40c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:31.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4f8bb3640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4f415d6f0 0x7fe4f40c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4f8bb3640 1 -- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f41538a0 msgr2=0x7fe4f40c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:31.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4f8bb3640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f41538a0 0x7fe4f40c8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.295+0000 7fe4f8bb3640 1 -- 172.21.15.46:0/3507704639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec009a70 con 0x7fe4f40cf160 2023-10-30T13:53:31.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.299+0000 7fe4f8bb3640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4f40cf160 0x7fe4f41503f0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fe4e800c420 tx=0x7fe4e800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:31.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.299+0000 7fe4f17fa640 1 -- 172.21.15.46:0/3507704639 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e80052d0 con 0x7fe4f40cf160 2023-10-30T13:53:31.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.299+0000 7fe4f17fa640 1 -- 172.21.15.46:0/3507704639 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e8005430 con 0x7fe4f40cf160 2023-10-30T13:53:31.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.299+0000 7fe4fae17640 1 -- 172.21.15.46:0/3507704639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f41596a0 con 0x7fe4f40cf160 2023-10-30T13:53:31.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.299+0000 7fe4f17fa640 1 -- 172.21.15.46:0/3507704639 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8007260 con 0x7fe4f40cf160 2023-10-30T13:53:31.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.299+0000 7fe4fae17640 1 -- 172.21.15.46:0/3507704639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f414f630 con 0x7fe4f40cf160 2023-10-30T13:53:31.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.299+0000 7fe4fae17640 1 -- 172.21.15.46:0/3507704639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f404e170 con 0x7fe4f40cf160 2023-10-30T13:53:31.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.299+0000 7fe4f17fa640 1 -- 172.21.15.46:0/3507704639 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe4e80073c0 con 0x7fe4f40cf160 2023-10-30T13:53:31.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.299+0000 7fe4f17fa640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4d80664d0 0x7fe4d8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:31.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.303+0000 7fe4f17fa640 1 -- 172.21.15.46:0/3507704639 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7fe4e8083830 con 0x7fe4f40cf160 2023-10-30T13:53:31.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.303+0000 7fe4f3fff640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4d80664d0 0x7fe4d8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:31.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.303+0000 7fe4f3fff640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4d80664d0 0x7fe4d8068990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe4f415cde0 tx=0x7fe4e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:31.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.307+0000 7fe4f17fa640 1 -- 172.21.15.46:0/3507704639 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe4e8050670 con 0x7fe4f40cf160 2023-10-30T13:53:31.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.495+0000 7fe4fae17640 1 -- 172.21.15.46:0/3507704639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4f414d6e0 con 0x7fe4f40cf160 2023-10-30T13:53:31.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4f17fa640 1 -- 172.21.15.46:0/3507704639 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+6522 (secure 0 0 0) 0x7fe4e8053c90 con 0x7fe4f40cf160 2023-10-30T13:53:31.500 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:53:31.500 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7356,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:26.395496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:31.501 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 137 2023-10-30T13:53:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4fae17640 1 -- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4d80664d0 msgr2=0x7fe4d8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4fae17640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4d80664d0 0x7fe4d8068990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe4f415cde0 tx=0x7fe4e0009040 comp rx=0 tx=0).stop 2023-10-30T13:53:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4fae17640 1 -- 172.21.15.46:0/3507704639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4f40cf160 msgr2=0x7fe4f41503f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4fae17640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4f40cf160 0x7fe4f41503f0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fe4e800c420 tx=0x7fe4e800c8e0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4fae17640 1 -- 172.21.15.46:0/3507704639 shutdown_connections 2023-10-30T13:53:31.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4fae17640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4d80664d0 0x7fe4d8068990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4fae17640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4f415d6f0 0x7fe4f40c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4fae17640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f41538a0 0x7fe4f40c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4fae17640 1 --2- 172.21.15.46:0/3507704639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4f40cf160 0x7fe4f41503f0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:31.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4fae17640 1 -- 172.21.15.46:0/3507704639 >> 172.21.15.46:0/3507704639 conn(0x7fe4f406bfe0 msgr2=0x7fe4f415b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:31.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4fae17640 1 -- 172.21.15.46:0/3507704639 shutdown_connections 2023-10-30T13:53:31.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:53:31.499+0000 7fe4fae17640 1 -- 172.21.15.46:0/3507704639 wait complete. 2023-10-30T13:53:31.521 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T13:53:31.521 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 51 secs before thrashing 2023-10-30T13:53:45.071 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 24% (45/186) Updating files: 25% (47/186) Updating files: 26% (49/186) Updating files: 27% (51/186) Updating files: 28% (53/186) Updating files: 29% (54/186) Updating files: 30% (56/186) Updating files: 31% (58/186) Updating files: 32% (60/186) Updating files: 33% (62/186) Updating files: 34% (64/186) Updating files: 35% (66/186) Updating files: 36% (67/186) Updating files: 37% (69/186) Updating files: 38% (71/186) Updating files: 39% (73/186) Updating files: 40% (75/186) Updating files: 41% (77/186) Updating files: 42% (79/186) Updating files: 43% (80/186) Updating files: 44% (82/186) Updating files: 45% (84/186) Updating files: 46% (86/186) Updating files: 47% (88/186) Updating files: 48% (90/186) Updating files: 49% (92/186) Updating files: 50% (93/186) Updating files: 51% (95/186) Updating files: 52% (97/186) Updating files: 53% (99/186) Updating files: 54% (101/186) Updating files: 55% (103/186) Updating files: 56% (105/186) Updating files: 57% (107/186) Updating files: 58% (108/186) Updating files: 59% (110/186) Updating files: 60% (112/186) Updating files: 61% (114/186) Updating files: 62% (116/186) Updating files: 63% (118/186) Updating files: 64% (120/186) Updating files: 65% (121/186) Updating files: 66% (123/186) Updating files: 67% (125/186) Updating files: 68% (127/186) Updating files: 69% (129/186) Updating files: 70% (131/186) Updating files: 71% (133/186) Updating files: 72% (134/186) Updating files: 73% (136/186) Updating files: 74% (138/186) Updating files: 75% (140/186) Updating files: 76% (142/186) Updating files: 77% (144/186) Updating files: 78% (146/186) Updating files: 79% (147/186) Updating files: 80% (149/186) Updating files: 81% (151/186) Updating files: 82% (153/186) Updating files: 83% (155/186) Updating files: 84% (157/186) Updating files: 85% (159/186) Updating files: 86% (160/186) Updating files: 87% (162/186) Updating files: 88% (164/186) Updating files: 89% (166/186) Updating files: 90% (168/186) Updating files: 91% (170/186) Updating files: 92% (172/186) Updating files: 93% (173/186) Updating files: 94% (175/186) Updating files: 95% (177/186) Updating files: 96% (179/186) Updating files: 97% (181/186) Updating files: 98% (183/186) Updating files: 99% (185/186) Updating files: 100% (186/186) Updating files: 100% (186/186), done. 2023-10-30T13:53:45.206 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at cdd5298d276 v0.44 2023-10-30T13:53:45.209 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.44 2023-10-30T13:53:45.235 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 45 -eq 48 ] 2023-10-30T13:53:45.235 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.45 2023-10-30T13:53:45.235 INFO:tasks.workunit.client.0.smithi046.stdout:v0.45 2023-10-30T13:53:45.236 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.45 2023-10-30T13:53:45.236 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.45 2023-10-30T13:53:50.149 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 25% (39/154) Updating files: 26% (41/154) Updating files: 27% (42/154) Updating files: 28% (44/154) Updating files: 29% (45/154) Updating files: 30% (47/154) Updating files: 31% (48/154) Updating files: 32% (50/154) Updating files: 33% (51/154) Updating files: 34% (53/154) Updating files: 35% (54/154) Updating files: 36% (56/154) Updating files: 37% (57/154) Updating files: 38% (59/154) Updating files: 39% (61/154) Updating files: 40% (62/154) Updating files: 41% (64/154) Updating files: 42% (65/154) Updating files: 43% (67/154) Updating files: 44% (68/154) Updating files: 45% (70/154) Updating files: 46% (71/154) Updating files: 47% (73/154) Updating files: 48% (74/154) Updating files: 49% (76/154) Updating files: 50% (77/154) Updating files: 51% (79/154) Updating files: 52% (81/154) Updating files: 53% (82/154) Updating files: 54% (84/154) Updating files: 55% (85/154) Updating files: 56% (87/154) Updating files: 57% (88/154) Updating files: 58% (90/154) Updating files: 59% (91/154) Updating files: 60% (93/154) Updating files: 61% (94/154) Updating files: 62% (96/154) Updating files: 63% (98/154) Updating files: 64% (99/154) Updating files: 65% (101/154) Updating files: 66% (102/154) Updating files: 67% (104/154) Updating files: 68% (105/154) Updating files: 69% (107/154) Updating files: 70% (108/154) Updating files: 71% (110/154) Updating files: 72% (111/154) Updating files: 73% (113/154) Updating files: 74% (114/154) Updating files: 75% (116/154) Updating files: 76% (118/154) Updating files: 77% (119/154) Updating files: 78% (121/154) Updating files: 79% (122/154) Updating files: 80% (124/154) Updating files: 81% (125/154) Updating files: 82% (127/154) Updating files: 83% (128/154) Updating files: 84% (130/154) Updating files: 85% (131/154) Updating files: 86% (133/154) Updating files: 87% (134/154) Updating files: 88% (136/154) Updating files: 89% (138/154) Updating files: 90% (139/154) Updating files: 91% (141/154) Updating files: 92% (142/154) Updating files: 93% (144/154) Updating files: 94% (145/154) Updating files: 95% (147/154) Updating files: 96% (148/154) Updating files: 97% (150/154) Updating files: 98% (151/154) Updating files: 99% (153/154) Updating files: 100% (154/154) Updating files: 100% (154/154), done. 2023-10-30T13:53:50.190 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 0aea1cb1df5 v0.45 2023-10-30T13:53:50.193 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.45 2023-10-30T13:53:51.584 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 46 -eq 48 ] 2023-10-30T13:53:51.584 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.46 2023-10-30T13:53:51.585 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.46 2023-10-30T13:53:51.585 INFO:tasks.workunit.client.0.smithi046.stdout:v0.46 2023-10-30T13:53:51.585 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.46 2023-10-30T13:53:51.647 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:51.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:55.792 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 45% (141/309) Updating files: 46% (143/309) Updating files: 47% (146/309) Updating files: 48% (149/309) Updating files: 49% (152/309) Updating files: 50% (155/309) Updating files: 51% (158/309) Updating files: 52% (161/309) Updating files: 53% (164/309) Updating files: 54% (167/309) Updating files: 55% (170/309) Updating files: 56% (174/309) Updating files: 57% (177/309) Updating files: 58% (180/309) Updating files: 59% (183/309) Updating files: 60% (186/309) Updating files: 61% (189/309) Updating files: 62% (192/309) Updating files: 63% (195/309) Updating files: 64% (198/309) Updating files: 65% (201/309) Updating files: 66% (204/309) Updating files: 67% (208/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 70% (219/309) Updating files: 71% (220/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 78% (242/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 89% (276/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2023-10-30T13:53:55.819 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at cb7f1c9c752 v0.46 2023-10-30T13:53:55.823 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.46 2023-10-30T13:53:56.081 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 47 -eq 48 ] 2023-10-30T13:53:56.081 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.47 2023-10-30T13:53:56.082 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.47 2023-10-30T13:53:56.082 INFO:tasks.workunit.client.0.smithi046.stdout:v0.47 2023-10-30T13:53:56.082 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.47 2023-10-30T13:53:58.186 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 32% (71/219) Updating files: 33% (73/219) Updating files: 34% (75/219) Updating files: 35% (77/219) Updating files: 36% (79/219) Updating files: 37% (82/219) Updating files: 38% (84/219) Updating files: 39% (86/219) Updating files: 40% (88/219) Updating files: 41% (90/219) Updating files: 42% (92/219) Updating files: 43% (95/219) Updating files: 44% (97/219) Updating files: 45% (99/219) Updating files: 46% (101/219) Updating files: 47% (103/219) Updating files: 48% (106/219) Updating files: 49% (108/219) Updating files: 50% (110/219) Updating files: 51% (112/219) Updating files: 52% (114/219) Updating files: 53% (117/219) Updating files: 54% (119/219) Updating files: 55% (121/219) Updating files: 56% (123/219) Updating files: 57% (125/219) Updating files: 58% (128/219) Updating files: 59% (130/219) Updating files: 60% (132/219) Updating files: 61% (134/219) Updating files: 62% (136/219) Updating files: 63% (138/219) Updating files: 64% (141/219) Updating files: 65% (143/219) Updating files: 66% (145/219) Updating files: 67% (147/219) Updating files: 68% (149/219) Updating files: 69% (152/219) Updating files: 70% (154/219) Updating files: 71% (156/219) Updating files: 72% (158/219) Updating files: 73% (160/219) Updating files: 74% (163/219) Updating files: 75% (165/219) Updating files: 76% (167/219) Updating files: 77% (169/219) Updating files: 78% (171/219) Updating files: 79% (174/219) Updating files: 80% (176/219) Updating files: 81% (178/219) Updating files: 82% (180/219) Updating files: 83% (182/219) Updating files: 84% (184/219) Updating files: 85% (187/219) Updating files: 86% (189/219) Updating files: 87% (191/219) Updating files: 88% (193/219) Updating files: 89% (195/219) Updating files: 90% (198/219) Updating files: 91% (200/219) Updating files: 92% (202/219) Updating files: 93% (204/219) Updating files: 94% (206/219) Updating files: 95% (209/219) Updating files: 96% (211/219) Updating files: 96% (212/219) Updating files: 97% (213/219) Updating files: 98% (215/219) Updating files: 99% (217/219) Updating files: 100% (219/219) Updating files: 100% (219/219), done. 2023-10-30T13:53:58.231 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at ad663d58c89 v0.47 2023-10-30T13:53:58.234 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.47 2023-10-30T13:54:00.343 INFO:tasks.workunit.client.0.smithi046.stdout:v0.49 2023-10-30T13:54:00.343 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 48 -eq 48 ] 2023-10-30T13:54:00.344 INFO:tasks.workunit.client.0.smithi046.stderr:+ continue 2023-10-30T13:54:00.344 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 49 -eq 48 ] 2023-10-30T13:54:00.344 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.49 2023-10-30T13:54:00.344 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.49 2023-10-30T13:54:00.344 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.49 2023-10-30T13:54:16.078 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 36% (175/481) Updating files: 37% (178/481) Updating files: 38% (183/481) Updating files: 39% (188/481) Updating files: 40% (193/481) Updating files: 41% (198/481) Updating files: 42% (203/481) Updating files: 42% (204/481) Updating files: 42% (206/481) Updating files: 43% (207/481) Updating files: 44% (212/481) Updating files: 45% (217/481) Updating files: 46% (222/481) Updating files: 47% (227/481) Updating files: 48% (231/481) Updating files: 49% (236/481) Updating files: 50% (241/481) Updating files: 51% (246/481) Updating files: 52% (251/481) Updating files: 53% (255/481) Updating files: 54% (260/481) Updating files: 55% (265/481) Updating files: 56% (270/481) Updating files: 57% (275/481) Updating files: 58% (279/481) Updating files: 59% (284/481) Updating files: 60% (289/481) Updating files: 61% (294/481) Updating files: 62% (299/481) Updating files: 62% (301/481) Updating files: 63% (304/481) Updating files: 64% (308/481) Updating files: 65% (313/481) Updating files: 66% (318/481) Updating files: 67% (323/481) Updating files: 68% (328/481) Updating files: 69% (332/481) Updating files: 70% (337/481) Updating files: 71% (342/481) Updating files: 72% (347/481) Updating files: 73% (352/481) Updating files: 74% (356/481) Updating files: 75% (361/481) Updating files: 76% (366/481) Updating files: 77% (371/481) Updating files: 78% (376/481) Updating files: 79% (380/481) Updating files: 80% (385/481) Updating files: 81% (390/481) Updating files: 82% (395/481) Updating files: 83% (400/481) Updating files: 84% (405/481) Updating files: 85% (409/481) Updating files: 86% (414/481) Updating files: 87% (419/481) Updating files: 88% (424/481) Updating files: 89% (429/481) Updating files: 90% (433/481) Updating files: 91% (438/481) Updating files: 92% (443/481) Updating files: 93% (448/481) Updating files: 94% (453/481) Updating files: 95% (457/481) Updating files: 96% (462/481) Updating files: 97% (467/481) Updating files: 98% (472/481) Updating files: 99% (477/481) Updating files: 100% (481/481) Updating files: 100% (481/481), done. 2023-10-30T13:54:16.119 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at ca6265d0f4d v0.49 2023-10-30T13:54:16.124 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.49 2023-10-30T13:54:16.411 INFO:tasks.workunit.client.0.smithi046.stdout:v0.50 2023-10-30T13:54:16.411 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 50 -eq 48 ] 2023-10-30T13:54:16.411 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.50 2023-10-30T13:54:16.411 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.50 2023-10-30T13:54:16.412 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.50 2023-10-30T13:54:20.272 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 82% (113/137) Updating files: 83% (114/137) Updating files: 84% (116/137) Updating files: 85% (117/137) Updating files: 86% (118/137) Updating files: 87% (120/137) Updating files: 88% (121/137) Updating files: 89% (122/137) Updating files: 90% (124/137) Updating files: 91% (125/137) Updating files: 92% (127/137) Updating files: 93% (128/137) Updating files: 94% (129/137) Updating files: 95% (131/137) Updating files: 96% (132/137) Updating files: 97% (133/137) Updating files: 98% (135/137) Updating files: 99% (136/137) Updating files: 100% (137/137) Updating files: 100% (137/137), done. 2023-10-30T13:54:20.504 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at a4428bd0e19 v0.50 2023-10-30T13:54:20.507 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.50 2023-10-30T13:54:21.369 INFO:tasks.workunit.client.0.smithi046.stdout:v0.51 2023-10-30T13:54:21.369 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 51 -eq 48 ] 2023-10-30T13:54:21.369 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.51 2023-10-30T13:54:21.370 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.51 2023-10-30T13:54:21.370 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.51 2023-10-30T13:54:21.762 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:21.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:22.522 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:22.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.667+0000 7f3c8508f640 1 Processor -- start 2023-10-30T13:54:22.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.667+0000 7f3c8508f640 1 -- start start 2023-10-30T13:54:22.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.667+0000 7f3c8508f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c8014ff10 0x7f3c800c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:22.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.667+0000 7f3c8508f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c80156220 con 0x7f3c8006fc90 2023-10-30T13:54:22.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.667+0000 7f3c8508f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c80155bf0 con 0x7f3c801566b0 2023-10-30T13:54:22.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.667+0000 7f3c8508f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c80070110 con 0x7f3c8014ff10 2023-10-30T13:54:22.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.667+0000 7f3c7ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c8014ff10 0x7f3c800c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.667+0000 7f3c77fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3c8006fc90 0x7f3c800c1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36246/0 (socket says 172.21.15.46:36246) 2023-10-30T13:54:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.667+0000 7f3c77fff640 1 -- 172.21.15.46:0/3151234218 learned_addr learned my addr 172.21.15.46:0/3151234218 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:22.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.671+0000 7f3c7ffff640 1 -- 172.21.15.46:0/3151234218 >> v1:172.21.15.154:6789/0 conn(0x7f3c801566b0 legacy=0x7f3c8006a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.671+0000 7f3c7ffff640 1 -- 172.21.15.46:0/3151234218 >> v1:172.21.15.46:6789/0 conn(0x7f3c8006fc90 legacy=0x7f3c800c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:22.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.671+0000 7f3c7ffff640 1 -- 172.21.15.46:0/3151234218 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c800703f0 con 0x7f3c8014ff10 2023-10-30T13:54:22.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.671+0000 7f3c7ffff640 1 --2- 172.21.15.46:0/3151234218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c8014ff10 0x7f3c800c54d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000ba40 tx=0x7f3c7001eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7687be4cb9a02b16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:22.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.671+0000 7f3c7f7fe640 1 -- 172.21.15.46:0/3151234218 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c700a7a70 con 0x7f3c8014ff10 2023-10-30T13:54:22.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.671+0000 7f3c7f7fe640 1 -- 172.21.15.46:0/3151234218 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c700a7bd0 con 0x7f3c8014ff10 2023-10-30T13:54:22.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.671+0000 7f3c7f7fe640 1 -- 172.21.15.46:0/3151234218 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c700af440 con 0x7f3c8014ff10 2023-10-30T13:54:22.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.671+0000 7f3c8508f640 1 -- 172.21.15.46:0/3151234218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c8014ff10 msgr2=0x7f3c800c54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:22.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.671+0000 7f3c8508f640 1 --2- 172.21.15.46:0/3151234218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c8014ff10 0x7f3c800c54d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000ba40 tx=0x7f3c7001eb30 comp rx=0 tx=0).stop 2023-10-30T13:54:22.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 -- 172.21.15.46:0/3151234218 shutdown_connections 2023-10-30T13:54:22.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 --2- 172.21.15.46:0/3151234218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c8014ff10 0x7f3c800c54d0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:22.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 -- 172.21.15.46:0/3151234218 >> 172.21.15.46:0/3151234218 conn(0x7f3c800c75d0 msgr2=0x7f3c800c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:22.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 -- 172.21.15.46:0/3151234218 shutdown_connections 2023-10-30T13:54:22.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 -- 172.21.15.46:0/3151234218 wait complete. 2023-10-30T13:54:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 Processor -- start 2023-10-30T13:54:22.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 -- start start 2023-10-30T13:54:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c8006fc90 0x7f3c801507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c8014ff10 0x7f3c800c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c801566b0 0x7f3c80149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:22.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c801518a0 con 0x7f3c8014ff10 2023-10-30T13:54:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8014a120 con 0x7f3c8006fc90 2023-10-30T13:54:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8508f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c80152180 con 0x7f3c801566b0 2023-10-30T13:54:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c77fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c8014ff10 0x7f3c800c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c77fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c8014ff10 0x7f3c800c4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56602/0 (socket says 172.21.15.46:56602) 2023-10-30T13:54:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c77fff640 1 -- 172.21.15.46:0/54608514 learned_addr learned my addr 172.21.15.46:0/54608514 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:22.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c7ffff640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c8006fc90 0x7f3c801507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:22.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c8488e640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c801566b0 0x7f3c80149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:22.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c77fff640 1 -- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c801566b0 msgr2=0x7f3c80149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:22.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c77fff640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c801566b0 0x7f3c80149340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:22.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c77fff640 1 -- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c8006fc90 msgr2=0x7f3c801507c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:22.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c77fff640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c8006fc90 0x7f3c801507c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c77fff640 1 -- 172.21.15.46:0/54608514 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7000b650 con 0x7f3c8014ff10 2023-10-30T13:54:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.675+0000 7f3c77fff640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c8014ff10 0x7f3c800c4ba0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800c400 tx=0x7f3c6800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.679+0000 7f3c7d7fa640 1 -- 172.21.15.46:0/54608514 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68005cf0 con 0x7f3c8014ff10 2023-10-30T13:54:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.679+0000 7f3c7d7fa640 1 -- 172.21.15.46:0/54608514 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c68005e50 con 0x7f3c8014ff10 2023-10-30T13:54:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.679+0000 7f3c7d7fa640 1 -- 172.21.15.46:0/54608514 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c680072d0 con 0x7f3c8014ff10 2023-10-30T13:54:22.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.679+0000 7f3c8508f640 1 -- 172.21.15.46:0/54608514 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c800c4710 con 0x7f3c8014ff10 2023-10-30T13:54:22.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.679+0000 7f3c8508f640 1 -- 172.21.15.46:0/54608514 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c800c3de0 con 0x7f3c8014ff10 2023-10-30T13:54:22.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.679+0000 7f3c8508f640 1 -- 172.21.15.46:0/54608514 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c800c48a0 con 0x7f3c8014ff10 2023-10-30T13:54:22.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.679+0000 7f3c7d7fa640 1 -- 172.21.15.46:0/54608514 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3c68007430 con 0x7f3c8014ff10 2023-10-30T13:54:22.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.679+0000 7f3c7d7fa640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3c600664a0 0x7f3c60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:22.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.679+0000 7f3c7d7fa640 1 -- 172.21.15.46:0/54608514 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f3c680838e0 con 0x7f3c8014ff10 2023-10-30T13:54:22.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.679+0000 7f3c7ffff640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3c600664a0 0x7f3c60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:22.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.683+0000 7f3c7ffff640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3c600664a0 0x7f3c60068960 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3c700b1000 tx=0x7f3c70008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:22.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.683+0000 7f3c7d7fa640 1 -- 172.21.15.46:0/54608514 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3c680507d0 con 0x7f3c8014ff10 2023-10-30T13:54:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.871+0000 7f3c8508f640 1 -- 172.21.15.46:0/54608514 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c800c3860 con 0x7f3c8014ff10 2023-10-30T13:54:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c7d7fa640 1 -- 172.21.15.46:0/54608514 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f3c68053df0 con 0x7f3c8014ff10 2023-10-30T13:54:22.875 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:22.876 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7356,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:37.414104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:22.876 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2023-10-30T13:54:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c8508f640 1 -- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3c600664a0 msgr2=0x7f3c60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c8508f640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3c600664a0 0x7f3c60068960 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3c700b1000 tx=0x7f3c70008040 comp rx=0 tx=0).stop 2023-10-30T13:54:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c8508f640 1 -- 172.21.15.46:0/54608514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c8014ff10 msgr2=0x7f3c800c4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:22.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c8508f640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c8014ff10 0x7f3c800c4ba0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800c400 tx=0x7f3c6800c8c0 comp rx=0 tx=0).stop 2023-10-30T13:54:22.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c8508f640 1 -- 172.21.15.46:0/54608514 shutdown_connections 2023-10-30T13:54:22.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c8508f640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3c600664a0 0x7f3c60068960 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:22.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c8508f640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c801566b0 0x7f3c80149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c8508f640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c8014ff10 0x7f3c800c4ba0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c8508f640 1 --2- 172.21.15.46:0/54608514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c8006fc90 0x7f3c801507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c8508f640 1 -- 172.21.15.46:0/54608514 >> 172.21.15.46:0/54608514 conn(0x7f3c800c75d0 msgr2=0x7f3c8015fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c8508f640 1 -- 172.21.15.46:0/54608514 shutdown_connections 2023-10-30T13:54:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:22.875+0000 7f3c8508f640 1 -- 172.21.15.46:0/54608514 wait complete. 2023-10-30T13:54:22.896 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T13:54:22.896 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:54:22.897 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:22.923 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:54:22.923 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:23.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabf5b3e640 1 Processor -- start 2023-10-30T13:54:23.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabf5b3e640 1 -- start start 2023-10-30T13:54:23.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabf5b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabf015d6f0 0x7fabf015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:23.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabf5b3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fabf00cf160 0x7fabf00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:23.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabf5b3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf00cb800 con 0x7fabf01538a0 2023-10-30T13:54:23.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabf5b3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf00cb330 con 0x7fabf00cf160 2023-10-30T13:54:23.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabf5b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00cad70 con 0x7fabf015d6f0 2023-10-30T13:54:23.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabef7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fabf01538a0 0x7fabf00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36250/0 (socket says 172.21.15.46:36250) 2023-10-30T13:54:23.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabef7fe640 1 -- 172.21.15.46:0/1116534957 learned_addr learned my addr 172.21.15.46:0/1116534957 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabeeffd640 1 --2- 172.21.15.46:0/1116534957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabf015d6f0 0x7fabf015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabeffff640 1 --2- 172.21.15.46:0/1116534957 >> v2:172.21.15.154:3300/0 conn(0x7fabf00cf160 0x7fabf00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabee7fc640 1 -- 172.21.15.46:0/1116534957 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445201035 0 0) 0x7fabf00cb800 con 0x7fabf01538a0 2023-10-30T13:54:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabee7fc640 1 -- 172.21.15.46:0/1116534957 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd8003530 con 0x7fabf01538a0 2023-10-30T13:54:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabee7fc640 1 -- 172.21.15.46:0/1116534957 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3673821401 0 0) 0x7fabd8003530 con 0x7fabf01538a0 2023-10-30T13:54:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabeeffd640 1 -- 172.21.15.46:0/1116534957 >> v2:172.21.15.154:3300/0 conn(0x7fabf00cf160 msgr2=0x7fabf00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabeeffd640 1 --2- 172.21.15.46:0/1116534957 >> v2:172.21.15.154:3300/0 conn(0x7fabf00cf160 0x7fabf00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabeeffd640 1 -- 172.21.15.46:0/1116534957 >> v1:172.21.15.46:6789/0 conn(0x7fabf01538a0 legacy=0x7fabf00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.111+0000 7fabeeffd640 1 -- 172.21.15.46:0/1116534957 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf00cb050 con 0x7fabf015d6f0 2023-10-30T13:54:23.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabeeffd640 1 --2- 172.21.15.46:0/1116534957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabf015d6f0 0x7fabf015a090 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fabe40097c0 tx=0x7fabe401eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c42257b844d66d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabee7fc640 1 -- 172.21.15.46:0/1116534957 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0001680 con 0x7fabf015d6f0 2023-10-30T13:54:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabee7fc640 1 -- 172.21.15.46:0/1116534957 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe40a85d0 con 0x7fabf015d6f0 2023-10-30T13:54:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabee7fc640 1 -- 172.21.15.46:0/1116534957 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe40a8940 con 0x7fabf015d6f0 2023-10-30T13:54:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabf5b3e640 1 -- 172.21.15.46:0/1116534957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabf015d6f0 msgr2=0x7fabf015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabf5b3e640 1 --2- 172.21.15.46:0/1116534957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabf015d6f0 0x7fabf015a090 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fabe40097c0 tx=0x7fabe401eb90 comp rx=0 tx=0).stop 2023-10-30T13:54:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabf5b3e640 1 -- 172.21.15.46:0/1116534957 shutdown_connections 2023-10-30T13:54:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabf5b3e640 1 --2- 172.21.15.46:0/1116534957 >> v2:172.21.15.154:3300/0 conn(0x7fabf00cf160 0x7fabf00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabf5b3e640 1 --2- 172.21.15.46:0/1116534957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabf015d6f0 0x7fabf015a090 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabf5b3e640 1 -- 172.21.15.46:0/1116534957 >> 172.21.15.46:0/1116534957 conn(0x7fabf006bfe0 msgr2=0x7fabf006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabf5b3e640 1 -- 172.21.15.46:0/1116534957 shutdown_connections 2023-10-30T13:54:23.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabf5b3e640 1 -- 172.21.15.46:0/1116534957 wait complete. 2023-10-30T13:54:23.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.115+0000 7fabf5b3e640 1 Processor -- start 2023-10-30T13:54:23.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabf5b3e640 1 -- start start 2023-10-30T13:54:23.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabf5b3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabf00cf160 0x7fabf0141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:23.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabf5b3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabf01538a0 0x7fabf0157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:23.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabf5b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabf015d6f0 0x7fabf0149450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:23.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabf5b3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00ce740 con 0x7fabf01538a0 2023-10-30T13:54:23.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabf5b3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf00cecc0 con 0x7fabf00cf160 2023-10-30T13:54:23.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabf5b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf01594b0 con 0x7fabf015d6f0 2023-10-30T13:54:23.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabeeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabf01538a0 0x7fabf0157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:23.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabeeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabf01538a0 0x7fabf0157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56618/0 (socket says 172.21.15.46:56618) 2023-10-30T13:54:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabeeffd640 1 -- 172.21.15.46:0/2893794473 learned_addr learned my addr 172.21.15.46:0/2893794473 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabeffff640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabf015d6f0 0x7fabf0149450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabef7fe640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabf00cf160 0x7fabf0141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabeeffd640 1 -- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabf015d6f0 msgr2=0x7fabf0149450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabeeffd640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabf015d6f0 0x7fabf0149450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:23.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabeeffd640 1 -- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabf00cf160 msgr2=0x7fabf0141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:23.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabeeffd640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabf00cf160 0x7fabf0141030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:23.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabeeffd640 1 -- 172.21.15.46:0/2893794473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe4009420 con 0x7fabf01538a0 2023-10-30T13:54:23.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabeeffd640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabf01538a0 0x7fabf0157620 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fabe401f430 tx=0x7fabe401b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:23.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabd7fff640 1 -- 172.21.15.46:0/2893794473 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe40bf070 con 0x7fabf01538a0 2023-10-30T13:54:23.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabd7fff640 1 -- 172.21.15.46:0/2893794473 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe40a8950 con 0x7fabf01538a0 2023-10-30T13:54:23.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabd7fff640 1 -- 172.21.15.46:0/2893794473 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe40a8cc0 con 0x7fabf01538a0 2023-10-30T13:54:23.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabf5b3e640 1 -- 172.21.15.46:0/2893794473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf0158f90 con 0x7fabf01538a0 2023-10-30T13:54:23.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabf5b3e640 1 -- 172.21.15.46:0/2893794473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf0146c10 con 0x7fabf01538a0 2023-10-30T13:54:23.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.119+0000 7fabf5b3e640 1 -- 172.21.15.46:0/2893794473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf00b2a50 con 0x7fabf01538a0 2023-10-30T13:54:23.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.123+0000 7fabd7fff640 1 -- 172.21.15.46:0/2893794473 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fabe4003620 con 0x7fabf01538a0 2023-10-30T13:54:23.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.123+0000 7fabd7fff640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fabdc0664a0 0x7fabdc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:23.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.123+0000 7fabd7fff640 1 -- 172.21.15.46:0/2893794473 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7fabe4121970 con 0x7fabf01538a0 2023-10-30T13:54:23.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.123+0000 7fabef7fe640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fabdc0664a0 0x7fabdc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:23.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.123+0000 7fabef7fe640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fabdc0664a0 0x7fabdc068960 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fabe0007a10 tx=0x7fabe0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:23.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.127+0000 7fabd7fff640 1 -- 172.21.15.46:0/2893794473 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fabe40ee8f0 con 0x7fabf01538a0 2023-10-30T13:54:23.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 -- 172.21.15.46:0/2893794473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabf004e170 con 0x7fabf01538a0 2023-10-30T13:54:23.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabd7fff640 1 -- 172.21.15.46:0/2893794473 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fabe40b5020 con 0x7fabf01538a0 2023-10-30T13:54:23.321 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:23.321 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7356,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:37.414104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:23.321 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2023-10-30T13:54:23.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 -- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fabdc0664a0 msgr2=0x7fabdc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:23.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fabdc0664a0 0x7fabdc068960 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fabe0007a10 tx=0x7fabe0009040 comp rx=0 tx=0).stop 2023-10-30T13:54:23.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 -- 172.21.15.46:0/2893794473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabf01538a0 msgr2=0x7fabf0157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:23.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabf01538a0 0x7fabf0157620 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fabe401f430 tx=0x7fabe401b930 comp rx=0 tx=0).stop 2023-10-30T13:54:23.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 -- 172.21.15.46:0/2893794473 shutdown_connections 2023-10-30T13:54:23.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fabdc0664a0 0x7fabdc068960 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:23.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabf015d6f0 0x7fabf0149450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:23.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabf01538a0 0x7fabf0157620 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:23.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 --2- 172.21.15.46:0/2893794473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabf00cf160 0x7fabf0141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:23.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 -- 172.21.15.46:0/2893794473 >> 172.21.15.46:0/2893794473 conn(0x7fabf006bfe0 msgr2=0x7fabf00c49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:23.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 -- 172.21.15.46:0/2893794473 shutdown_connections 2023-10-30T13:54:23.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:23.319+0000 7fabf5b3e640 1 -- 172.21.15.46:0/2893794473 wait complete. 2023-10-30T13:54:23.342 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:54:25.343 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:25.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.494+0000 7f422b394640 1 Processor -- start 2023-10-30T13:54:25.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f422b394640 1 -- start start 2023-10-30T13:54:25.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f422b394640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4224158a20 0x7f422406fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:25.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f422b394640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4224158eb0 con 0x7f42241518a0 2023-10-30T13:54:25.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f422b394640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4224151c80 con 0x7f42241581b0 2023-10-30T13:54:25.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f422b394640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42240c30c0 con 0x7f4224158a20 2023-10-30T13:54:25.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f4229130640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f42241518a0 0x7f42240ccfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36254/0 (socket says 172.21.15.46:36254) 2023-10-30T13:54:25.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f4229130640 1 -- 172.21.15.46:0/3842378252 learned_addr learned my addr 172.21.15.46:0/3842378252 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:25.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f4229931640 1 --2- 172.21.15.46:0/3842378252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4224158a20 0x7f422406fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:25.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293640189 0 0) 0x7f4224158eb0 con 0x7f42241518a0 2023-10-30T13:54:25.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4210003530 con 0x7f42241518a0 2023-10-30T13:54:25.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1142982660 0 0) 0x7f4210003530 con 0x7f42241518a0 2023-10-30T13:54:25.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4224158a20 msgr2=0x7f422406fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:25.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 --2- 172.21.15.46:0/3842378252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4224158a20 0x7f422406fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:25.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 >> v1:172.21.15.154:6789/0 conn(0x7f42241581b0 legacy=0x7f422415fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:25.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42240c4120 con 0x7f42241518a0 2023-10-30T13:54:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f4220001680 con 0x7f42241518a0 2023-10-30T13:54:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 >> v1:172.21.15.46:6789/0 conn(0x7f42241518a0 legacy=0x7f42240ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 --2- 172.21.15.46:0/3842378252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4224158a20 0x7f42100039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 --2- 172.21.15.46:0/3842378252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4210003f50 0x7f42100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 --2- 172.21.15.46:0/3842378252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4210004d10 0x7f4210005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42240c30c0 con 0x7f4210004d10 2023-10-30T13:54:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421000bb30 con 0x7f4210003f50 2023-10-30T13:54:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421000bcf0 con 0x7f4224158a20 2023-10-30T13:54:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f4229931640 1 --2- 172.21.15.46:0/3842378252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4210004d10 0x7f4210005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f4229130640 1 --2- 172.21.15.46:0/3842378252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4224158a20 0x7f42100039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f422892f640 1 --2- 172.21.15.46:0/3842378252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4210003f50 0x7f42100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:25.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f4229931640 1 -- 172.21.15.46:0/3842378252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4224158a20 msgr2=0x7f42100039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:25.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f4229931640 1 --2- 172.21.15.46:0/3842378252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4224158a20 0x7f42100039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:25.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f4229931640 1 -- 172.21.15.46:0/3842378252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4210003f50 msgr2=0x7f42100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:25.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f4229931640 1 --2- 172.21.15.46:0/3842378252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4210003f50 0x7f42100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:25.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.498+0000 7f4229931640 1 -- 172.21.15.46:0/3842378252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4210003530 con 0x7f4210004d10 2023-10-30T13:54:25.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.502+0000 7f4229931640 1 --2- 172.21.15.46:0/3842378252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4210004d10 0x7f4210005190 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f421c009f90 tx=0x7f421c01cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ac9d5d1a6d5edf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:25.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.502+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c0a5c30 con 0x7f4210004d10 2023-10-30T13:54:25.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.502+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421c00f040 con 0x7f4210004d10 2023-10-30T13:54:25.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.502+0000 7f421bfff640 1 -- 172.21.15.46:0/3842378252 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c0ad440 con 0x7f4210004d10 2023-10-30T13:54:25.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 -- 172.21.15.46:0/3842378252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4210004d10 msgr2=0x7f4210005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:25.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 --2- 172.21.15.46:0/3842378252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4210004d10 0x7f4210005190 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f421c009f90 tx=0x7f421c01cc20 comp rx=0 tx=0).stop 2023-10-30T13:54:25.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f4229931640 1 -- 172.21.15.46:0/3842378252 reap_dead start 2023-10-30T13:54:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 -- 172.21.15.46:0/3842378252 shutdown_connections 2023-10-30T13:54:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 -- 172.21.15.46:0/3842378252 >> 172.21.15.46:0/3842378252 conn(0x7f4224069e90 msgr2=0x7f422406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 -- 172.21.15.46:0/3842378252 shutdown_connections 2023-10-30T13:54:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 -- 172.21.15.46:0/3842378252 wait complete. 2023-10-30T13:54:25.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 Processor -- start 2023-10-30T13:54:25.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 -- start start 2023-10-30T13:54:25.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4210004d10 0x7f42240c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:25.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4224156640 0x7f42240c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:25.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42240c63e0 0x7f422406bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:25.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42240c5a10 con 0x7f4224156640 2023-10-30T13:54:25.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42240c5150 con 0x7f42240c63e0 2023-10-30T13:54:25.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422b394640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42240c6860 con 0x7f4210004d10 2023-10-30T13:54:25.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422892f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4224156640 0x7f42240c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:25.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422892f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4224156640 0x7f42240c5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56642/0 (socket says 172.21.15.46:56642) 2023-10-30T13:54:25.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422892f640 1 -- 172.21.15.46:0/2260883669 learned_addr learned my addr 172.21.15.46:0/2260883669 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:25.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f4229130640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4210004d10 0x7f42240c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:25.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f4229931640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42240c63e0 0x7f422406bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:25.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422892f640 1 -- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4210004d10 msgr2=0x7f42240c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:25.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.506+0000 7f422892f640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4210004d10 0x7f42240c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:25.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f422892f640 1 -- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42240c63e0 msgr2=0x7f422406bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:25.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f422892f640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42240c63e0 0x7f422406bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:25.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f422892f640 1 -- 172.21.15.46:0/2260883669 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42240ca490 con 0x7f4224156640 2023-10-30T13:54:25.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f422892f640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4224156640 0x7f42240c5ea0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f421400e550 tx=0x7f421400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:25.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f4219ffb640 1 -- 172.21.15.46:0/2260883669 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42140052e0 con 0x7f4224156640 2023-10-30T13:54:25.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f4219ffb640 1 -- 172.21.15.46:0/2260883669 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4214005440 con 0x7f4224156640 2023-10-30T13:54:25.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f4219ffb640 1 -- 172.21.15.46:0/2260883669 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4214003820 con 0x7f4224156640 2023-10-30T13:54:25.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f422b394640 1 -- 172.21.15.46:0/2260883669 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42241eadf0 con 0x7f4224156640 2023-10-30T13:54:25.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f422b394640 1 -- 172.21.15.46:0/2260883669 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42240c53c0 con 0x7f4224156640 2023-10-30T13:54:25.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f422b394640 1 -- 172.21.15.46:0/2260883669 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42240c5b80 con 0x7f4224156640 2023-10-30T13:54:25.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f4219ffb640 1 -- 172.21.15.46:0/2260883669 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4214003980 con 0x7f4224156640 2023-10-30T13:54:25.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f4219ffb640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42080664a0 0x7f4208068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:25.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f4219ffb640 1 -- 172.21.15.46:0/2260883669 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f4214084220 con 0x7f4224156640 2023-10-30T13:54:25.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.510+0000 7f4229130640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42080664a0 0x7f4208068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:25.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.514+0000 7f4229130640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42080664a0 0x7f4208068960 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4220001c90 tx=0x7f4220009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:25.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.514+0000 7f4219ffb640 1 -- 172.21.15.46:0/2260883669 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f42140510f0 con 0x7f4224156640 2023-10-30T13:54:25.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.706+0000 7f422b394640 1 -- 172.21.15.46:0/2260883669 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f422404e170 con 0x7f4224156640 2023-10-30T13:54:25.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.706+0000 7f4219ffb640 1 -- 172.21.15.46:0/2260883669 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f4214054710 con 0x7f4224156640 2023-10-30T13:54:25.710 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:25.710 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7356,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:37.414104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:25.710 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2023-10-30T13:54:25.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.710+0000 7f422b394640 1 -- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42080664a0 msgr2=0x7f4208068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:25.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.710+0000 7f422b394640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42080664a0 0x7f4208068960 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4220001c90 tx=0x7f4220009040 comp rx=0 tx=0).stop 2023-10-30T13:54:25.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.710+0000 7f422b394640 1 -- 172.21.15.46:0/2260883669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4224156640 msgr2=0x7f42240c5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:25.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.710+0000 7f422b394640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4224156640 0x7f42240c5ea0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f421400e550 tx=0x7f421400ea10 comp rx=0 tx=0).stop 2023-10-30T13:54:25.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.710+0000 7f422b394640 1 -- 172.21.15.46:0/2260883669 shutdown_connections 2023-10-30T13:54:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.710+0000 7f422b394640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f42080664a0 0x7f4208068960 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.710+0000 7f422b394640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42240c63e0 0x7f422406bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.710+0000 7f422b394640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4224156640 0x7f42240c5ea0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.710+0000 7f422b394640 1 --2- 172.21.15.46:0/2260883669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4210004d10 0x7f42240c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:25.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.710+0000 7f422b394640 1 -- 172.21.15.46:0/2260883669 >> 172.21.15.46:0/2260883669 conn(0x7f4224069e90 msgr2=0x7f42240c7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:25.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.710+0000 7f422b394640 1 -- 172.21.15.46:0/2260883669 shutdown_connections 2023-10-30T13:54:25.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:25.710+0000 7f422b394640 1 -- 172.21.15.46:0/2260883669 wait complete. 2023-10-30T13:54:25.731 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:54:27.732 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.874+0000 7f8662df5640 1 Processor -- start 2023-10-30T13:54:27.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.874+0000 7f8662df5640 1 -- start start 2023-10-30T13:54:27.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.874+0000 7f8662df5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f865c06f7a0 0x7f865c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.874+0000 7f8662df5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f865c066cf0 con 0x7f865c071110 2023-10-30T13:54:27.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.874+0000 7f8662df5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f865c0c70d0 con 0x7f865c0c6bd0 2023-10-30T13:54:27.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.874+0000 7f8662df5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865c0700c0 con 0x7f865c06f7a0 2023-10-30T13:54:27.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.874+0000 7f8661392640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f865c071110 0x7f865c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36194/0 (socket says 172.21.15.46:36194) 2023-10-30T13:54:27.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.874+0000 7f8661392640 1 -- 172.21.15.46:0/784150253 learned_addr learned my addr 172.21.15.46:0/784150253 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:27.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.874+0000 7f8660b91640 1 --2- 172.21.15.46:0/784150253 >> v2:172.21.15.154:3300/0 conn(0x7f865c06f7a0 0x7f865c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649012858 0 0) 0x7f865c066cf0 con 0x7f865c071110 2023-10-30T13:54:27.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f864c003530 con 0x7f865c071110 2023-10-30T13:54:27.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3235572628 0 0) 0x7f864c003530 con 0x7f865c071110 2023-10-30T13:54:27.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 >> v2:172.21.15.154:3300/0 conn(0x7f865c06f7a0 msgr2=0x7f865c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:27.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 --2- 172.21.15.46:0/784150253 >> v2:172.21.15.154:3300/0 conn(0x7f865c06f7a0 0x7f865c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 >> v1:172.21.15.154:6789/0 conn(0x7f865c0c6bd0 legacy=0x7f865c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:27.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c0c7390 con 0x7f865c071110 2023-10-30T13:54:27.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f8654001a00 con 0x7f865c071110 2023-10-30T13:54:27.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 >> v1:172.21.15.46:6789/0 conn(0x7f865c071110 legacy=0x7f865c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:27.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 --2- 172.21.15.46:0/784150253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c003980 0x7f864c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 --2- 172.21.15.46:0/784150253 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f864c0047d0 0x7f864c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 --2- 172.21.15.46:0/784150253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f864c005190 0x7f864c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c0700c0 con 0x7f864c003980 2023-10-30T13:54:27.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c00c040 con 0x7f864c005190 2023-10-30T13:54:27.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c00c200 con 0x7f864c0047d0 2023-10-30T13:54:27.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f8660b91640 1 --2- 172.21.15.46:0/784150253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c003980 0x7f864c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f865bfff640 1 --2- 172.21.15.46:0/784150253 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f864c0047d0 0x7f864c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f8661392640 1 --2- 172.21.15.46:0/784150253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f864c005190 0x7f864c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f8660b91640 1 -- 172.21.15.46:0/784150253 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f864c0047d0 msgr2=0x7f864c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:27.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f8660b91640 1 --2- 172.21.15.46:0/784150253 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f864c0047d0 0x7f864c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f8660b91640 1 -- 172.21.15.46:0/784150253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f864c005190 msgr2=0x7f864c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:27.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f8660b91640 1 --2- 172.21.15.46:0/784150253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f864c005190 0x7f864c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f8660b91640 1 -- 172.21.15.46:0/784150253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c003530 con 0x7f864c003980 2023-10-30T13:54:27.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.878+0000 7f8661392640 1 -- 172.21.15.46:0/784150253 reap_dead start 2023-10-30T13:54:27.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8660b91640 1 --2- 172.21.15.46:0/784150253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c003980 0x7f864c003da0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f8650009ae0 tx=0x7f865001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb10def1d4f9a8bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:27.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86500a5530 con 0x7f864c003980 2023-10-30T13:54:27.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86500a5690 con 0x7f864c003980 2023-10-30T13:54:27.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f865b7fe640 1 -- 172.21.15.46:0/784150253 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86500a5a40 con 0x7f864c003980 2023-10-30T13:54:27.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 -- 172.21.15.46:0/784150253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c003980 msgr2=0x7f864c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:27.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 --2- 172.21.15.46:0/784150253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c003980 0x7f864c003da0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f8650009ae0 tx=0x7f865001ca50 comp rx=0 tx=0).stop 2023-10-30T13:54:27.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 -- 172.21.15.46:0/784150253 shutdown_connections 2023-10-30T13:54:27.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 --2- 172.21.15.46:0/784150253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c003980 0x7f864c003da0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 -- 172.21.15.46:0/784150253 >> 172.21.15.46:0/784150253 conn(0x7f865c0c7910 msgr2=0x7f865c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:27.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 -- 172.21.15.46:0/784150253 shutdown_connections 2023-10-30T13:54:27.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 -- 172.21.15.46:0/784150253 wait complete. 2023-10-30T13:54:27.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 Processor -- start 2023-10-30T13:54:27.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 -- start start 2023-10-30T13:54:27.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f864c003980 0x7f865c159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f865c158140 0x7f865c159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c14f0b0 0x7f865c14f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c155050 con 0x7f865c14f0b0 2023-10-30T13:54:27.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c15b4e0 con 0x7f865c158140 2023-10-30T13:54:27.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.882+0000 7f8662df5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865c15b7a0 con 0x7f864c003980 2023-10-30T13:54:27.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8661392640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c14f0b0 0x7f865c14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8660b91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f864c003980 0x7f865c159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f865bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f865c158140 0x7f865c159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8661392640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c14f0b0 0x7f865c14f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33718/0 (socket says 172.21.15.46:33718) 2023-10-30T13:54:27.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8661392640 1 -- 172.21.15.46:0/2268110344 learned_addr learned my addr 172.21.15.46:0/2268110344 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:27.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8661392640 1 -- 172.21.15.46:0/2268110344 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f864c003980 msgr2=0x7f865c159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:27.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8661392640 1 --2- 172.21.15.46:0/2268110344 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f864c003980 0x7f865c159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8661392640 1 -- 172.21.15.46:0/2268110344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f865c158140 msgr2=0x7f865c159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:27.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8661392640 1 --2- 172.21.15.46:0/2268110344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f865c158140 0x7f865c159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:27.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8661392640 1 -- 172.21.15.46:0/2268110344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c158940 con 0x7f865c14f0b0 2023-10-30T13:54:27.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8661392640 1 --2- 172.21.15.46:0/2268110344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c14f0b0 0x7f865c14f530 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f865c0c6fb0 tx=0x7f8654011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:27.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f86597fa640 1 -- 172.21.15.46:0/2268110344 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865400f920 con 0x7f865c14f0b0 2023-10-30T13:54:27.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f86597fa640 1 -- 172.21.15.46:0/2268110344 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865400faa0 con 0x7f865c14f0b0 2023-10-30T13:54:27.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f86597fa640 1 -- 172.21.15.46:0/2268110344 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865400fe50 con 0x7f865c14f0b0 2023-10-30T13:54:27.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8662df5640 1 -- 172.21.15.46:0/2268110344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865c158c20 con 0x7f865c14f0b0 2023-10-30T13:54:27.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8662df5640 1 -- 172.21.15.46:0/2268110344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f865c1591e0 con 0x7f865c14f0b0 2023-10-30T13:54:27.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f8662df5640 1 -- 172.21.15.46:0/2268110344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865c04e170 con 0x7f865c14f0b0 2023-10-30T13:54:27.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.886+0000 7f86597fa640 1 -- 172.21.15.46:0/2268110344 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f865401f460 con 0x7f865c14f0b0 2023-10-30T13:54:27.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.890+0000 7f86597fa640 1 --2- 172.21.15.46:0/2268110344 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f86400664f0 0x7f86400689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:27.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.890+0000 7f86597fa640 1 -- 172.21.15.46:0/2268110344 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f8654087270 con 0x7f865c14f0b0 2023-10-30T13:54:27.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.890+0000 7f8660b91640 1 --2- 172.21.15.46:0/2268110344 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f86400664f0 0x7f86400689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:27.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.890+0000 7f8660b91640 1 --2- 172.21.15.46:0/2268110344 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f86400664f0 0x7f86400689b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f86500af000 tx=0x7f86500046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:27.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:27.894+0000 7f86597fa640 1 -- 172.21.15.46:0/2268110344 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8654054270 con 0x7f865c14f0b0 2023-10-30T13:54:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.082+0000 7f8662df5640 1 -- 172.21.15.46:0/2268110344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f865c0c4600 con 0x7f865c14f0b0 2023-10-30T13:54:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.082+0000 7f86597fa640 1 -- 172.21.15.46:0/2268110344 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f8654057890 con 0x7f865c14f0b0 2023-10-30T13:54:28.087 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:28.087 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7356,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:37.414104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2023-10-30T13:54:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.086+0000 7f8662df5640 1 -- 172.21.15.46:0/2268110344 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f86400664f0 msgr2=0x7f86400689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.086+0000 7f8662df5640 1 --2- 172.21.15.46:0/2268110344 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f86400664f0 0x7f86400689b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f86500af000 tx=0x7f86500046d0 comp rx=0 tx=0).stop 2023-10-30T13:54:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.086+0000 7f8662df5640 1 -- 172.21.15.46:0/2268110344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c14f0b0 msgr2=0x7f865c14f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.086+0000 7f8662df5640 1 --2- 172.21.15.46:0/2268110344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c14f0b0 0x7f865c14f530 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f865c0c6fb0 tx=0x7f8654011c20 comp rx=0 tx=0).stop 2023-10-30T13:54:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.086+0000 7f8662df5640 1 -- 172.21.15.46:0/2268110344 shutdown_connections 2023-10-30T13:54:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.086+0000 7f8662df5640 1 --2- 172.21.15.46:0/2268110344 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f86400664f0 0x7f86400689b0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.086+0000 7f8662df5640 1 --2- 172.21.15.46:0/2268110344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f865c14f0b0 0x7f865c14f530 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.086+0000 7f8662df5640 1 --2- 172.21.15.46:0/2268110344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f865c158140 0x7f865c159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.086+0000 7f8662df5640 1 --2- 172.21.15.46:0/2268110344 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f864c003980 0x7f865c159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.086+0000 7f8662df5640 1 -- 172.21.15.46:0/2268110344 >> 172.21.15.46:0/2268110344 conn(0x7f865c0c7910 msgr2=0x7f865c0c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.086+0000 7f8662df5640 1 -- 172.21.15.46:0/2268110344 shutdown_connections 2023-10-30T13:54:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:28.086+0000 7f8662df5640 1 -- 172.21.15.46:0/2268110344 wait complete. 2023-10-30T13:54:28.108 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:54:30.110 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:30.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.258+0000 7f45beb0e640 1 Processor -- start 2023-10-30T13:54:30.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.258+0000 7f45beb0e640 1 -- start start 2023-10-30T13:54:30.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45beb0e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b80c6f60 0x7f45b81566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45beb0e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b80c0a50 con 0x7f45b80c6b80 2023-10-30T13:54:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45beb0e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b806f6e0 con 0x7f45b80c05d0 2023-10-30T13:54:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45beb0e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b806f9c0 con 0x7f45b80c6f60 2023-10-30T13:54:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45bc8aa640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f45b80c6b80 0x7f45b80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36200/0 (socket says 172.21.15.46:36200) 2023-10-30T13:54:30.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45bc8aa640 1 -- 172.21.15.46:0/445282315 learned_addr learned my addr 172.21.15.46:0/445282315 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:30.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45bd0ab640 1 --2- 172.21.15.46:0/445282315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b80c6f60 0x7f45b81566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:30.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45b77fe640 1 -- 172.21.15.46:0/445282315 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606938119 0 0) 0x7f45b80c0a50 con 0x7f45b80c6b80 2023-10-30T13:54:30.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45b77fe640 1 -- 172.21.15.46:0/445282315 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a8003530 con 0x7f45b80c6b80 2023-10-30T13:54:30.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45bd0ab640 1 -- 172.21.15.46:0/445282315 >> v1:172.21.15.154:6789/0 conn(0x7f45b80c05d0 legacy=0x7f45b813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:30.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45b77fe640 1 -- 172.21.15.46:0/445282315 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1204287186 0 0) 0x7f45a8003530 con 0x7f45b80c6b80 2023-10-30T13:54:30.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45bd0ab640 1 -- 172.21.15.46:0/445282315 >> v1:172.21.15.46:6789/0 conn(0x7f45b80c6b80 legacy=0x7f45b80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:30.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.262+0000 7f45bd0ab640 1 -- 172.21.15.46:0/445282315 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b806fca0 con 0x7f45b80c6f60 2023-10-30T13:54:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45bd0ab640 1 --2- 172.21.15.46:0/445282315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b80c6f60 0x7f45b81566a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009dc0 tx=0x7f45b001c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7d1e5e42b537d5c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:30.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45b77fe640 1 -- 172.21.15.46:0/445282315 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac001680 con 0x7f45b80c6f60 2023-10-30T13:54:30.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45b77fe640 1 -- 172.21.15.46:0/445282315 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b00a64a0 con 0x7f45b80c6f60 2023-10-30T13:54:30.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45b77fe640 1 -- 172.21.15.46:0/445282315 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b00a6810 con 0x7f45b80c6f60 2023-10-30T13:54:30.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45beb0e640 1 -- 172.21.15.46:0/445282315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b80c6f60 msgr2=0x7f45b81566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:30.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45beb0e640 1 --2- 172.21.15.46:0/445282315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b80c6f60 0x7f45b81566a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009dc0 tx=0x7f45b001c740 comp rx=0 tx=0).stop 2023-10-30T13:54:30.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45beb0e640 1 -- 172.21.15.46:0/445282315 shutdown_connections 2023-10-30T13:54:30.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45beb0e640 1 --2- 172.21.15.46:0/445282315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b80c6f60 0x7f45b81566a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:30.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45beb0e640 1 -- 172.21.15.46:0/445282315 >> 172.21.15.46:0/445282315 conn(0x7f45b80c7910 msgr2=0x7f45b80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:30.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45beb0e640 1 -- 172.21.15.46:0/445282315 shutdown_connections 2023-10-30T13:54:30.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45beb0e640 1 -- 172.21.15.46:0/445282315 wait complete. 2023-10-30T13:54:30.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45beb0e640 1 Processor -- start 2023-10-30T13:54:30.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.266+0000 7f45beb0e640 1 -- start start 2023-10-30T13:54:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45beb0e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80c05d0 0x7f45b80ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:30.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45beb0e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b80c6b80 0x7f45b81f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:30.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45beb0e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b80c6f60 0x7f45b81591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:30.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45beb0e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b81f2770 con 0x7f45b80c05d0 2023-10-30T13:54:30.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45beb0e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b80cabe0 con 0x7f45b80c6b80 2023-10-30T13:54:30.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45beb0e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b8156120 con 0x7f45b80c6f60 2023-10-30T13:54:30.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45bc8aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80c05d0 0x7f45b80ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:30.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45bc8aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80c05d0 0x7f45b80ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33722/0 (socket says 172.21.15.46:33722) 2023-10-30T13:54:30.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45bc8aa640 1 -- 172.21.15.46:0/1153782954 learned_addr learned my addr 172.21.15.46:0/1153782954 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:30.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45bd0ab640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b80c6f60 0x7f45b81591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:30.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45b7fff640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b80c6b80 0x7f45b81f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:30.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45bc8aa640 1 -- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b80c6f60 msgr2=0x7f45b81591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:30.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45bc8aa640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b80c6f60 0x7f45b81591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:30.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45bc8aa640 1 -- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b80c6b80 msgr2=0x7f45b81f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:30.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45bc8aa640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b80c6b80 0x7f45b81f2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:30.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45bc8aa640 1 -- 172.21.15.46:0/1153782954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b0009a70 con 0x7f45b80c05d0 2023-10-30T13:54:30.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45b7fff640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b80c6b80 0x7f45b81f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:30.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45bc8aa640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80c05d0 0x7f45b80ca6a0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00c750 tx=0x7f45ac00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:30.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45b57fa640 1 -- 172.21.15.46:0/1153782954 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac0072d0 con 0x7f45b80c05d0 2023-10-30T13:54:30.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45b57fa640 1 -- 172.21.15.46:0/1153782954 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac007430 con 0x7f45b80c05d0 2023-10-30T13:54:30.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45beb0e640 1 -- 172.21.15.46:0/1153782954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b81f2440 con 0x7f45b80c05d0 2023-10-30T13:54:30.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45b57fa640 1 -- 172.21.15.46:0/1153782954 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac0047f0 con 0x7f45b80c05d0 2023-10-30T13:54:30.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45beb0e640 1 -- 172.21.15.46:0/1153782954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b8155b60 con 0x7f45b80c05d0 2023-10-30T13:54:30.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45beb0e640 1 -- 172.21.15.46:0/1153782954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b81f22a0 con 0x7f45b80c05d0 2023-10-30T13:54:30.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.270+0000 7f45b57fa640 1 -- 172.21.15.46:0/1153782954 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f45ac004950 con 0x7f45b80c05d0 2023-10-30T13:54:30.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.274+0000 7f45b57fa640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f459c0664a0 0x7f459c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:30.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.274+0000 7f45b57fa640 1 -- 172.21.15.46:0/1153782954 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f45ac083690 con 0x7f45b80c05d0 2023-10-30T13:54:30.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.274+0000 7f45b7fff640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f459c0664a0 0x7f459c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:30.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.274+0000 7f45b7fff640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f459c0664a0 0x7f459c068960 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f45b81efed0 tx=0x7f45a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:30.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.278+0000 7f45b57fa640 1 -- 172.21.15.46:0/1153782954 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f45b81f22a0 con 0x7f45b80c05d0 2023-10-30T13:54:30.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.470+0000 7f45beb0e640 1 -- 172.21.15.46:0/1153782954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45b81f22a0 con 0x7f45b80c05d0 2023-10-30T13:54:30.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.470+0000 7f45b57fa640 1 -- 172.21.15.46:0/1153782954 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f45b81f22a0 con 0x7f45b80c05d0 2023-10-30T13:54:30.473 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:30.474 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7356,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:37.414104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2023-10-30T13:54:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.470+0000 7f45beb0e640 1 -- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f459c0664a0 msgr2=0x7f459c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.470+0000 7f45beb0e640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f459c0664a0 0x7f459c068960 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f45b81efed0 tx=0x7f45a4009040 comp rx=0 tx=0).stop 2023-10-30T13:54:30.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.470+0000 7f45beb0e640 1 -- 172.21.15.46:0/1153782954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80c05d0 msgr2=0x7f45b80ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:30.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.470+0000 7f45beb0e640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80c05d0 0x7f45b80ca6a0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00c750 tx=0x7f45ac00cc10 comp rx=0 tx=0).stop 2023-10-30T13:54:30.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.474+0000 7f45beb0e640 1 -- 172.21.15.46:0/1153782954 shutdown_connections 2023-10-30T13:54:30.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.474+0000 7f45beb0e640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f459c0664a0 0x7f459c068960 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:30.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.474+0000 7f45beb0e640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b80c6f60 0x7f45b81591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:30.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.474+0000 7f45beb0e640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b80c6b80 0x7f45b81f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:30.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.474+0000 7f45beb0e640 1 --2- 172.21.15.46:0/1153782954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80c05d0 0x7f45b80ca6a0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:30.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.474+0000 7f45beb0e640 1 -- 172.21.15.46:0/1153782954 >> 172.21.15.46:0/1153782954 conn(0x7f45b80c7910 msgr2=0x7f45b806a810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:30.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.474+0000 7f45beb0e640 1 -- 172.21.15.46:0/1153782954 shutdown_connections 2023-10-30T13:54:30.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:30.474+0000 7f45beb0e640 1 -- 172.21.15.46:0/1153782954 wait complete. 2023-10-30T13:54:30.495 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:54:32.496 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:32.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3d017640 1 Processor -- start 2023-10-30T13:54:32.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3d017640 1 -- start start 2023-10-30T13:54:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3d017640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38159e60 0x7fdd3806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3d017640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3806fa80 con 0x7fdd3806a720 2023-10-30T13:54:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3d017640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd38071290 con 0x7fdd380bfec0 2023-10-30T13:54:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3d017640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd38071570 con 0x7fdd38159e60 2023-10-30T13:54:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3759e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdd3806a720 0x7fdd38160610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36214/0 (socket says 172.21.15.46:36214) 2023-10-30T13:54:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3759e640 1 -- 172.21.15.46:0/1858610752 learned_addr learned my addr 172.21.15.46:0/1858610752 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd36d9d640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38159e60 0x7fdd3806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544360026 0 0) 0x7fdd3806fa80 con 0x7fdd3806a720 2023-10-30T13:54:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd24003530 con 0x7fdd3806a720 2023-10-30T13:54:32.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 685339440 0 0) 0x7fdd24003530 con 0x7fdd3806a720 2023-10-30T13:54:32.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38159e60 msgr2=0x7fdd3806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:32.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3659c640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38159e60 0x7fdd3806f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:32.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 >> v1:172.21.15.154:6789/0 conn(0x7fdd380bfec0 legacy=0x7fdd380c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd38071850 con 0x7fdd3806a720 2023-10-30T13:54:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fdd30001a00 con 0x7fdd3806a720 2023-10-30T13:54:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd36d9d640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38159e60 0x7fdd3806f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:54:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.646+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 >> v1:172.21.15.46:6789/0 conn(0x7fdd3806a720 legacy=0x7fdd38160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd3659c640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd240039b0 0x7fdd24003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd3659c640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38159e60 0x7fdd240047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd3659c640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd24004d10 0x7fdd24005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd38071570 con 0x7fdd240039b0 2023-10-30T13:54:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2400b990 con 0x7fdd24004d10 2023-10-30T13:54:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2400bb00 con 0x7fdd38159e60 2023-10-30T13:54:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd36d9d640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd240039b0 0x7fdd24003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd2ffff640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38159e60 0x7fdd240047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd3759e640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd24004d10 0x7fdd24005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd36d9d640 1 -- 172.21.15.46:0/1858610752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38159e60 msgr2=0x7fdd240047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd36d9d640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38159e60 0x7fdd240047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd36d9d640 1 -- 172.21.15.46:0/1858610752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd24004d10 msgr2=0x7fdd24005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:32.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd36d9d640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd24004d10 0x7fdd24005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:32.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd36d9d640 1 -- 172.21.15.46:0/1858610752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd28009420 con 0x7fdd240039b0 2023-10-30T13:54:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd36d9d640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd240039b0 0x7fdd24003d90 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009cb0 tx=0x7fdd2801ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4def98b1a2664e59 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd280a7d20 con 0x7fdd240039b0 2023-10-30T13:54:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd280a7e80 con 0x7fdd240039b0 2023-10-30T13:54:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.650+0000 7fdd3659c640 1 -- 172.21.15.46:0/1858610752 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd280af660 con 0x7fdd240039b0 2023-10-30T13:54:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 -- 172.21.15.46:0/1858610752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd240039b0 msgr2=0x7fdd24003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 --2- 172.21.15.46:0/1858610752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd240039b0 0x7fdd24003d90 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009cb0 tx=0x7fdd2801ee60 comp rx=0 tx=0).stop 2023-10-30T13:54:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3759e640 1 -- 172.21.15.46:0/1858610752 reap_dead start 2023-10-30T13:54:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 -- 172.21.15.46:0/1858610752 shutdown_connections 2023-10-30T13:54:32.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 -- 172.21.15.46:0/1858610752 >> 172.21.15.46:0/1858610752 conn(0x7fdd380c7910 msgr2=0x7fdd380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:32.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 -- 172.21.15.46:0/1858610752 shutdown_connections 2023-10-30T13:54:32.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 -- 172.21.15.46:0/1858610752 wait complete. 2023-10-30T13:54:32.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 Processor -- start 2023-10-30T13:54:32.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 -- start start 2023-10-30T13:54:32.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd240039b0 0x7fdd381e19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd3814fe00 0x7fdd380ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd3814f8b0 0x7fdd380adf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd38153780 con 0x7fdd240039b0 2023-10-30T13:54:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd380b0ce0 con 0x7fdd3814fe00 2023-10-30T13:54:32.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3d017640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3806f9e0 con 0x7fdd3814f8b0 2023-10-30T13:54:32.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd36d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd240039b0 0x7fdd381e19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:32.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd36d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd240039b0 0x7fdd381e19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33746/0 (socket says 172.21.15.46:33746) 2023-10-30T13:54:32.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd36d9d640 1 -- 172.21.15.46:0/3412141822 learned_addr learned my addr 172.21.15.46:0/3412141822 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:32.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd3759e640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd3814f8b0 0x7fdd380adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.654+0000 7fdd2ffff640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd3814fe00 0x7fdd380ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd36d9d640 1 -- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd3814f8b0 msgr2=0x7fdd380adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd36d9d640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd3814f8b0 0x7fdd380adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd36d9d640 1 -- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd3814fe00 msgr2=0x7fdd380ad9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:32.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd36d9d640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd3814fe00 0x7fdd380ad9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd36d9d640 1 -- 172.21.15.46:0/3412141822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd380afba0 con 0x7fdd240039b0 2023-10-30T13:54:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd36d9d640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd240039b0 0x7fdd381e19a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fdd2801ec90 tx=0x7fdd24005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd2f7fe640 1 -- 172.21.15.46:0/3412141822 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28004220 con 0x7fdd240039b0 2023-10-30T13:54:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd2f7fe640 1 -- 172.21.15.46:0/3412141822 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd28004380 con 0x7fdd240039b0 2023-10-30T13:54:32.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd2f7fe640 1 -- 172.21.15.46:0/3412141822 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28003840 con 0x7fdd240039b0 2023-10-30T13:54:32.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd3d017640 1 -- 172.21.15.46:0/3412141822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd380b0060 con 0x7fdd240039b0 2023-10-30T13:54:32.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd3d017640 1 -- 172.21.15.46:0/3412141822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd380b0620 con 0x7fdd240039b0 2023-10-30T13:54:32.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd3d017640 1 -- 172.21.15.46:0/3412141822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3804e170 con 0x7fdd240039b0 2023-10-30T13:54:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd2f7fe640 1 -- 172.21.15.46:0/3412141822 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdd280a7d30 con 0x7fdd240039b0 2023-10-30T13:54:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.658+0000 7fdd2f7fe640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdd180664d0 0x7fdd18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:32.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.662+0000 7fdd2f7fe640 1 -- 172.21.15.46:0/3412141822 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7fdd2811eb40 con 0x7fdd240039b0 2023-10-30T13:54:32.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.662+0000 7fdd2ffff640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdd180664d0 0x7fdd18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:32.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.662+0000 7fdd2ffff640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdd180664d0 0x7fdd18068990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdd3806fbe0 tx=0x7fdd20007cd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:32.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.666+0000 7fdd2f7fe640 1 -- 172.21.15.46:0/3412141822 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdd280eb8f0 con 0x7fdd240039b0 2023-10-30T13:54:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.854+0000 7fdd3d017640 1 -- 172.21.15.46:0/3412141822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd3806fd90 con 0x7fdd240039b0 2023-10-30T13:54:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.854+0000 7fdd2f7fe640 1 -- 172.21.15.46:0/3412141822 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fdd280eef10 con 0x7fdd240039b0 2023-10-30T13:54:32.858 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:32.859 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7356,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:37.414104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2023-10-30T13:54:32.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.858+0000 7fdd3d017640 1 -- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdd180664d0 msgr2=0x7fdd18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:32.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.858+0000 7fdd3d017640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdd180664d0 0x7fdd18068990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdd3806fbe0 tx=0x7fdd20007cd0 comp rx=0 tx=0).stop 2023-10-30T13:54:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.858+0000 7fdd3d017640 1 -- 172.21.15.46:0/3412141822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd240039b0 msgr2=0x7fdd381e19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.858+0000 7fdd3d017640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd240039b0 0x7fdd381e19a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fdd2801ec90 tx=0x7fdd24005800 comp rx=0 tx=0).stop 2023-10-30T13:54:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.858+0000 7fdd3d017640 1 -- 172.21.15.46:0/3412141822 shutdown_connections 2023-10-30T13:54:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.858+0000 7fdd3d017640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdd180664d0 0x7fdd18068990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.858+0000 7fdd3d017640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd3814f8b0 0x7fdd380adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.858+0000 7fdd3d017640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd3814fe00 0x7fdd380ad9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.858+0000 7fdd3d017640 1 --2- 172.21.15.46:0/3412141822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd240039b0 0x7fdd381e19a0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.858+0000 7fdd3d017640 1 -- 172.21.15.46:0/3412141822 >> 172.21.15.46:0/3412141822 conn(0x7fdd380c7910 msgr2=0x7fdd38158100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.858+0000 7fdd3d017640 1 -- 172.21.15.46:0/3412141822 shutdown_connections 2023-10-30T13:54:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:32.858+0000 7fdd3d017640 1 -- 172.21.15.46:0/3412141822 wait complete. 2023-10-30T13:54:32.880 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:54:34.881 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:35.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.030+0000 7f03a66c9640 1 Processor -- start 2023-10-30T13:54:35.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.030+0000 7f03a66c9640 1 -- start start 2023-10-30T13:54:35.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f03a66c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f03a00c05d0 0x7f03a013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:35.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f03a66c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f03a00c6f60 0x7f03a013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:35.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f03a66c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a00c0a50 con 0x7f03a00c6b80 2023-10-30T13:54:35.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f03a66c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a006f6e0 con 0x7f03a00c6f60 2023-10-30T13:54:35.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f03a66c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a006f9c0 con 0x7f03a00c05d0 2023-10-30T13:54:35.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f03a4c66640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f03a00c6f60 0x7f03a013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:35.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f039ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f03a00c6b80 0x7f03a00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36222/0 (socket says 172.21.15.46:36222) 2023-10-30T13:54:35.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f039ffff640 1 -- 172.21.15.46:0/1770583824 learned_addr learned my addr 172.21.15.46:0/1770583824 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:35.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f039f7fe640 1 --2- 172.21.15.46:0/1770583824 >> v2:172.21.15.154:3300/0 conn(0x7f03a00c05d0 0x7f03a013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:35.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f03a4c66640 1 -- 172.21.15.46:0/1770583824 >> v2:172.21.15.154:3300/0 conn(0x7f03a00c05d0 msgr2=0x7f03a013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:35.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f03a4c66640 1 --2- 172.21.15.46:0/1770583824 >> v2:172.21.15.154:3300/0 conn(0x7f03a00c05d0 0x7f03a013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:35.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f03a4c66640 1 -- 172.21.15.46:0/1770583824 >> v1:172.21.15.46:6789/0 conn(0x7f03a00c6b80 legacy=0x7f03a00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:35.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f03a4c66640 1 -- 172.21.15.46:0/1770583824 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a006fca0 con 0x7f03a00c6f60 2023-10-30T13:54:35.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f039effd640 1 -- 172.21.15.46:0/1770583824 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153488770 0 0) 0x7f03a00c0a50 con 0x7f03a00c6b80 2023-10-30T13:54:35.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.034+0000 7f039f7fe640 1 --2- 172.21.15.46:0/1770583824 >> v2:172.21.15.154:3300/0 conn(0x7f03a00c05d0 0x7f03a013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:35.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a4c66640 1 --2- 172.21.15.46:0/1770583824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a00c6f60 0x7f03a013f2d0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f03a006c7e0 tx=0x7f039801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991fa69a7569a0c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:35.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f039effd640 1 -- 172.21.15.46:0/1770583824 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03980a58e0 con 0x7f03a00c6f60 2023-10-30T13:54:35.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f039effd640 1 -- 172.21.15.46:0/1770583824 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03980a5a40 con 0x7f03a00c6f60 2023-10-30T13:54:35.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f039effd640 1 -- 172.21.15.46:0/1770583824 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03980a5dd0 con 0x7f03a00c6f60 2023-10-30T13:54:35.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 -- 172.21.15.46:0/1770583824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a00c6f60 msgr2=0x7f03a013f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:35.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 --2- 172.21.15.46:0/1770583824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a00c6f60 0x7f03a013f2d0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f03a006c7e0 tx=0x7f039801c910 comp rx=0 tx=0).stop 2023-10-30T13:54:35.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 -- 172.21.15.46:0/1770583824 shutdown_connections 2023-10-30T13:54:35.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 --2- 172.21.15.46:0/1770583824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a00c6f60 0x7f03a013f2d0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:35.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 --2- 172.21.15.46:0/1770583824 >> v2:172.21.15.154:3300/0 conn(0x7f03a00c05d0 0x7f03a013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:35.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 -- 172.21.15.46:0/1770583824 >> 172.21.15.46:0/1770583824 conn(0x7f03a00c7910 msgr2=0x7f03a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:35.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 -- 172.21.15.46:0/1770583824 shutdown_connections 2023-10-30T13:54:35.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 -- 172.21.15.46:0/1770583824 wait complete. 2023-10-30T13:54:35.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 Processor -- start 2023-10-30T13:54:35.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 -- start start 2023-10-30T13:54:35.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a00c05d0 0x7f03a0159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:35.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03a00c6b80 0x7f03a013c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:35.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a00c6f60 0x7f03a014ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:35.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a00c5420 con 0x7f03a00c6f60 2023-10-30T13:54:35.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a013db20 con 0x7f03a00c05d0 2023-10-30T13:54:35.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.038+0000 7f03a66c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a01597a0 con 0x7f03a00c6b80 2023-10-30T13:54:35.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f03a4c66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a00c6f60 0x7f03a014ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:35.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f03a4c66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a00c6f60 0x7f03a014ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33776/0 (socket says 172.21.15.46:33776) 2023-10-30T13:54:35.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f03a4c66640 1 -- 172.21.15.46:0/538547633 learned_addr learned my addr 172.21.15.46:0/538547633 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:35.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f039ffff640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a00c05d0 0x7f03a0159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:35.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f039f7fe640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03a00c6b80 0x7f03a013c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:35.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f03a4c66640 1 -- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03a00c6b80 msgr2=0x7f03a013c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:35.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f03a4c66640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03a00c6b80 0x7f03a013c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f03a4c66640 1 -- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a00c05d0 msgr2=0x7f03a0159260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f03a4c66640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a00c05d0 0x7f03a0159260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f03a4c66640 1 -- 172.21.15.46:0/538547633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0398009a70 con 0x7f03a00c6f60 2023-10-30T13:54:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f03a4c66640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a00c6f60 0x7f03a014ee90 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f0398098400 tx=0x7f03980196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:35.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f039cff9640 1 -- 172.21.15.46:0/538547633 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03980bf070 con 0x7f03a00c6f60 2023-10-30T13:54:35.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f039cff9640 1 -- 172.21.15.46:0/538547633 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03980ae820 con 0x7f03a00c6f60 2023-10-30T13:54:35.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f039cff9640 1 -- 172.21.15.46:0/538547633 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03980aebb0 con 0x7f03a00c6f60 2023-10-30T13:54:35.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f03a66c9640 1 -- 172.21.15.46:0/538547633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a00c8c10 con 0x7f03a00c6f60 2023-10-30T13:54:35.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f03a66c9640 1 -- 172.21.15.46:0/538547633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a00c4df0 con 0x7f03a00c6f60 2023-10-30T13:54:35.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f03a66c9640 1 -- 172.21.15.46:0/538547633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a004e170 con 0x7f03a00c6f60 2023-10-30T13:54:35.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.042+0000 7f039cff9640 1 -- 172.21.15.46:0/538547633 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f03980b4050 con 0x7f03a00c6f60 2023-10-30T13:54:35.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.046+0000 7f039cff9640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f038c0664d0 0x7f038c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:35.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.046+0000 7f039cff9640 1 -- 172.21.15.46:0/538547633 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f039811e7a0 con 0x7f03a00c6f60 2023-10-30T13:54:35.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.046+0000 7f039ffff640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f038c0664d0 0x7f038c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:35.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.046+0000 7f039ffff640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f038c0664d0 0x7f038c068990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0390007fd0 tx=0x7f0390007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:35.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.050+0000 7f039cff9640 1 -- 172.21.15.46:0/538547633 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f03980eb720 con 0x7f03a00c6f60 2023-10-30T13:54:35.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.238+0000 7f03a66c9640 1 -- 172.21.15.46:0/538547633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03a013b210 con 0x7f03a00c6f60 2023-10-30T13:54:35.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.238+0000 7f039cff9640 1 -- 172.21.15.46:0/538547633 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f03980eed40 con 0x7f03a00c6f60 2023-10-30T13:54:35.243 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:35.243 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7356,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:37.414104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:35.244 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2023-10-30T13:54:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.242+0000 7f03a66c9640 1 -- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f038c0664d0 msgr2=0x7f038c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.242+0000 7f03a66c9640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f038c0664d0 0x7f038c068990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0390007fd0 tx=0x7f0390007d60 comp rx=0 tx=0).stop 2023-10-30T13:54:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.242+0000 7f03a66c9640 1 -- 172.21.15.46:0/538547633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a00c6f60 msgr2=0x7f03a014ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.242+0000 7f03a66c9640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a00c6f60 0x7f03a014ee90 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f0398098400 tx=0x7f03980196b0 comp rx=0 tx=0).stop 2023-10-30T13:54:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.242+0000 7f03a66c9640 1 -- 172.21.15.46:0/538547633 shutdown_connections 2023-10-30T13:54:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.242+0000 7f03a66c9640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f038c0664d0 0x7f038c068990 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.242+0000 7f03a66c9640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a00c6f60 0x7f03a014ee90 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.242+0000 7f03a66c9640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03a00c6b80 0x7f03a013c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:35.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.242+0000 7f03a66c9640 1 --2- 172.21.15.46:0/538547633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a00c05d0 0x7f03a0159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:35.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.242+0000 7f03a66c9640 1 -- 172.21.15.46:0/538547633 >> 172.21.15.46:0/538547633 conn(0x7f03a00c7910 msgr2=0x7f03a00c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:35.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.242+0000 7f03a66c9640 1 -- 172.21.15.46:0/538547633 shutdown_connections 2023-10-30T13:54:35.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:35.242+0000 7f03a66c9640 1 -- 172.21.15.46:0/538547633 wait complete. 2023-10-30T13:54:35.265 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:54:37.265 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:37.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.430+0000 7fbfcce91640 1 Processor -- start 2023-10-30T13:54:37.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.430+0000 7fbfcce91640 1 -- start start 2023-10-30T13:54:37.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.430+0000 7fbfcce91640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbfc80cf160 0x7fbfc80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:37.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.430+0000 7fbfcce91640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc80cb800 con 0x7fbfc81538a0 2023-10-30T13:54:37.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.430+0000 7fbfcce91640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc80cb330 con 0x7fbfc815d6f0 2023-10-30T13:54:37.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.430+0000 7fbfcce91640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc80cad70 con 0x7fbfc80cf160 2023-10-30T13:54:37.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.430+0000 7fbfc659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbfc81538a0 0x7fbfc80ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60794/0 (socket says 172.21.15.46:60794) 2023-10-30T13:54:37.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.430+0000 7fbfc659c640 1 -- 172.21.15.46:0/2770091598 learned_addr learned my addr 172.21.15.46:0/2770091598 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:37.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.430+0000 7fbfc6d9d640 1 --2- 172.21.15.46:0/2770091598 >> v2:172.21.15.154:3300/0 conn(0x7fbfc80cf160 0x7fbfc80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:37.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989516487 0 0) 0x7fbfc80cb800 con 0x7fbfc81538a0 2023-10-30T13:54:37.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfb0003530 con 0x7fbfc81538a0 2023-10-30T13:54:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1747373222 0 0) 0x7fbfb0003530 con 0x7fbfc81538a0 2023-10-30T13:54:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 >> v2:172.21.15.154:3300/0 conn(0x7fbfc80cf160 msgr2=0x7fbfc80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 --2- 172.21.15.46:0/2770091598 >> v2:172.21.15.154:3300/0 conn(0x7fbfc80cf160 0x7fbfc80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 >> v1:172.21.15.154:6789/0 conn(0x7fbfc815d6f0 legacy=0x7fbfc815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc80cb050 con 0x7fbfc81538a0 2023-10-30T13:54:37.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fbfb8001660 con 0x7fbfc81538a0 2023-10-30T13:54:37.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 >> v1:172.21.15.46:6789/0 conn(0x7fbfc81538a0 legacy=0x7fbfc80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:37.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 --2- 172.21.15.46:0/2770091598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb0003980 0x7fbfb0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:37.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 --2- 172.21.15.46:0/2770091598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfb00047d0 0x7fbfb0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:37.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 --2- 172.21.15.46:0/2770091598 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfb0005190 0x7fbfb000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:37.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc80cad70 con 0x7fbfb00047d0 2023-10-30T13:54:37.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb000c0a0 con 0x7fbfb0003980 2023-10-30T13:54:37.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb000c260 con 0x7fbfb0005190 2023-10-30T13:54:37.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc5d9b640 1 --2- 172.21.15.46:0/2770091598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfb00047d0 0x7fbfb0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:37.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc5d9b640 1 -- 172.21.15.46:0/2770091598 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfb0005190 msgr2=0x7fbfb000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:37.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc5d9b640 1 --2- 172.21.15.46:0/2770091598 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfb0005190 0x7fbfb000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc5d9b640 1 -- 172.21.15.46:0/2770091598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb0003980 msgr2=0x7fbfb0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc5d9b640 1 --2- 172.21.15.46:0/2770091598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb0003980 0x7fbfb0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc5d9b640 1 -- 172.21.15.46:0/2770091598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb0003530 con 0x7fbfb00047d0 2023-10-30T13:54:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.434+0000 7fbfc6d9d640 1 -- 172.21.15.46:0/2770091598 reap_dead start 2023-10-30T13:54:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.438+0000 7fbfc5d9b640 1 --2- 172.21.15.46:0/2770091598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfb00047d0 0x7fbfb0004c50 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc001780 tx=0x7fbfbc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbd097eb482d6ff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.438+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc0052d0 con 0x7fbfb00047d0 2023-10-30T13:54:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.438+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc0a7050 con 0x7fbfb00047d0 2023-10-30T13:54:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.438+0000 7fbfc559a640 1 -- 172.21.15.46:0/2770091598 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc003600 con 0x7fbfb00047d0 2023-10-30T13:54:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.438+0000 7fbfcce91640 1 -- 172.21.15.46:0/2770091598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfb00047d0 msgr2=0x7fbfb0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.438+0000 7fbfcce91640 1 --2- 172.21.15.46:0/2770091598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfb00047d0 0x7fbfb0004c50 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc001780 tx=0x7fbfbc01c3e0 comp rx=0 tx=0).stop 2023-10-30T13:54:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.438+0000 7fbfcce91640 1 -- 172.21.15.46:0/2770091598 shutdown_connections 2023-10-30T13:54:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.438+0000 7fbfcce91640 1 --2- 172.21.15.46:0/2770091598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfb00047d0 0x7fbfb0004c50 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.438+0000 7fbfcce91640 1 -- 172.21.15.46:0/2770091598 >> 172.21.15.46:0/2770091598 conn(0x7fbfc806bfe0 msgr2=0x7fbfc806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.438+0000 7fbfcce91640 1 -- 172.21.15.46:0/2770091598 shutdown_connections 2023-10-30T13:54:37.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.438+0000 7fbfcce91640 1 -- 172.21.15.46:0/2770091598 wait complete. 2023-10-30T13:54:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfcce91640 1 Processor -- start 2023-10-30T13:54:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfcce91640 1 -- start start 2023-10-30T13:54:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfcce91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb00047d0 0x7fbfc80b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfcce91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc80c5080 0x7fbfc81405c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfcce91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc80c5500 0x7fbfc8140b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:37.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfcce91640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc80cede0 con 0x7fbfc80c5080 2023-10-30T13:54:37.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfcce91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc80ce380 con 0x7fbfb00047d0 2023-10-30T13:54:37.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfcce91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc80ce660 con 0x7fbfc80c5500 2023-10-30T13:54:37.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfc5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc80c5080 0x7fbfc81405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:37.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfc5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc80c5080 0x7fbfc81405c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35938/0 (socket says 172.21.15.46:35938) 2023-10-30T13:54:37.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfc5d9b640 1 -- 172.21.15.46:0/2372724439 learned_addr learned my addr 172.21.15.46:0/2372724439 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:37.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfc659c640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb00047d0 0x7fbfc80b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:37.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfc6d9d640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc80c5500 0x7fbfc8140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:37.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfc5d9b640 1 -- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc80c5500 msgr2=0x7fbfc8140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:37.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfc5d9b640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc80c5500 0x7fbfc8140b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:37.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfc5d9b640 1 -- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb00047d0 msgr2=0x7fbfc80b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:37.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfc5d9b640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb00047d0 0x7fbfc80b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:37.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfc5d9b640 1 -- 172.21.15.46:0/2372724439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc81e2b40 con 0x7fbfc80c5080 2023-10-30T13:54:37.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfc5d9b640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc80c5080 0x7fbfc81405c0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc01c210 tx=0x7fbfb0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:37.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfb6ffd640 1 -- 172.21.15.46:0/2372724439 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc007440 con 0x7fbfc80c5080 2023-10-30T13:54:37.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfb6ffd640 1 -- 172.21.15.46:0/2372724439 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc0075c0 con 0x7fbfc80c5080 2023-10-30T13:54:37.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfcce91640 1 -- 172.21.15.46:0/2372724439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc81e2e20 con 0x7fbfc80c5080 2023-10-30T13:54:37.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfb6ffd640 1 -- 172.21.15.46:0/2372724439 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc006a20 con 0x7fbfc80c5080 2023-10-30T13:54:37.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.442+0000 7fbfcce91640 1 -- 172.21.15.46:0/2372724439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc81e4780 con 0x7fbfc80c5080 2023-10-30T13:54:37.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.446+0000 7fbfcce91640 1 -- 172.21.15.46:0/2372724439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc804e170 con 0x7fbfc80c5080 2023-10-30T13:54:37.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.446+0000 7fbfb6ffd640 1 -- 172.21.15.46:0/2372724439 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbfbc006b80 con 0x7fbfc80c5080 2023-10-30T13:54:37.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.446+0000 7fbfb6ffd640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbfa80664d0 0x7fbfa8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:37.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.446+0000 7fbfb6ffd640 1 -- 172.21.15.46:0/2372724439 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7144+0+0 (secure 0 0 0) 0x7fbfbc11e550 con 0x7fbfc80c5080 2023-10-30T13:54:37.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.446+0000 7fbfc659c640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbfa80664d0 0x7fbfa8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:37.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.446+0000 7fbfc659c640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbfa80664d0 0x7fbfa8068990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80011f0 tx=0x7fbfb8000bd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:37.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.450+0000 7fbfb6ffd640 1 -- 172.21.15.46:0/2372724439 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbfbc0eb3b0 con 0x7fbfc80c5080 2023-10-30T13:54:37.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.650+0000 7fbfcce91640 1 -- 172.21.15.46:0/2372724439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfc815cea0 con 0x7fbfc80c5080 2023-10-30T13:54:37.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfb6ffd640 1 -- 172.21.15.46:0/2372724439 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fbfbc0ee9d0 con 0x7fbfc80c5080 2023-10-30T13:54:37.655 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:37.656 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7356,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:53:37.414104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7002,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":125,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/2807435195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2807435195},{"type":"v1","addr":"172.21.15.46:6833","nonce":2807435195}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:37.656 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2023-10-30T13:54:37.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfcce91640 1 -- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbfa80664d0 msgr2=0x7fbfa8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:37.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfcce91640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbfa80664d0 0x7fbfa8068990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80011f0 tx=0x7fbfb8000bd0 comp rx=0 tx=0).stop 2023-10-30T13:54:37.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfcce91640 1 -- 172.21.15.46:0/2372724439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc80c5080 msgr2=0x7fbfc81405c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:37.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfcce91640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc80c5080 0x7fbfc81405c0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc01c210 tx=0x7fbfb0005800 comp rx=0 tx=0).stop 2023-10-30T13:54:37.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfcce91640 1 -- 172.21.15.46:0/2372724439 shutdown_connections 2023-10-30T13:54:37.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfcce91640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbfa80664d0 0x7fbfa8068990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:37.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfcce91640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbfc80c5500 0x7fbfc8140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:37.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfcce91640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc80c5080 0x7fbfc81405c0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:37.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfcce91640 1 --2- 172.21.15.46:0/2372724439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb00047d0 0x7fbfc80b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:37.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfcce91640 1 -- 172.21.15.46:0/2372724439 >> 172.21.15.46:0/2372724439 conn(0x7fbfc806bfe0 msgr2=0x7fbfc8151f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:37.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfcce91640 1 -- 172.21.15.46:0/2372724439 shutdown_connections 2023-10-30T13:54:37.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:37.654+0000 7fbfcce91640 1 -- 172.21.15.46:0/2372724439 wait complete. 2023-10-30T13:54:37.677 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:54:39.678 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:39.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.830+0000 7f3f47854640 1 Processor -- start 2023-10-30T13:54:39.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f47854640 1 -- start start 2023-10-30T13:54:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f47854640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3f4015d6f0 0x7f3f4015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f47854640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f400cf160 0x7f3f400cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:39.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f47854640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f400cb800 con 0x7f3f401538a0 2023-10-30T13:54:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f47854640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f400cb330 con 0x7f3f4015d6f0 2023-10-30T13:54:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f47854640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f400cad70 con 0x7f3f400cf160 2023-10-30T13:54:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f455f0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3f401538a0 0x7f3f400ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60810/0 (socket says 172.21.15.46:60810) 2023-10-30T13:54:39.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f455f0640 1 -- 172.21.15.46:0/452198232 learned_addr learned my addr 172.21.15.46:0/452198232 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f45df1640 1 --2- 172.21.15.46:0/452198232 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f400cf160 0x7f3f400cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f44def640 1 --2- 172.21.15.46:0/452198232 >> v2:172.21.15.154:3300/0 conn(0x7f3f4015d6f0 0x7f3f4015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424437917 0 0) 0x7f3f400cb800 con 0x7f3f401538a0 2023-10-30T13:54:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f2c003530 con 0x7f3f401538a0 2023-10-30T13:54:39.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 538696915 0 0) 0x7f3f2c003530 con 0x7f3f401538a0 2023-10-30T13:54:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f400cf160 msgr2=0x7f3f400cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 --2- 172.21.15.46:0/452198232 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f400cf160 0x7f3f400cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 >> v2:172.21.15.154:3300/0 conn(0x7f3f4015d6f0 msgr2=0x7f3f4015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:39.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 --2- 172.21.15.46:0/452198232 >> v2:172.21.15.154:3300/0 conn(0x7f3f4015d6f0 0x7f3f4015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f400cb050 con 0x7f3f401538a0 2023-10-30T13:54:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f3f3c001680 con 0x7f3f401538a0 2023-10-30T13:54:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 >> v1:172.21.15.46:6789/0 conn(0x7f3f401538a0 legacy=0x7f3f400ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 --2- 172.21.15.46:0/452198232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c003980 0x7f3f2c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:39.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 --2- 172.21.15.46:0/452198232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f2c0047d0 0x7f3f2c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:39.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 --2- 172.21.15.46:0/452198232 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f400cf160 0x7f3f2c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:39.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f400cb330 con 0x7f3f2c0047d0 2023-10-30T13:54:39.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f400cad70 con 0x7f3f2c003980 2023-10-30T13:54:39.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c00b990 con 0x7f3f400cf160 2023-10-30T13:54:39.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f44def640 1 --2- 172.21.15.46:0/452198232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f2c0047d0 0x7f3f2c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:39.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f455f0640 1 --2- 172.21.15.46:0/452198232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c003980 0x7f3f2c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:39.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.834+0000 7f3f45df1640 1 --2- 172.21.15.46:0/452198232 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f400cf160 0x7f3f2c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:39.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f44def640 1 -- 172.21.15.46:0/452198232 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f400cf160 msgr2=0x7f3f2c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:39.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f44def640 1 --2- 172.21.15.46:0/452198232 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f400cf160 0x7f3f2c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:39.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f44def640 1 -- 172.21.15.46:0/452198232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c003980 msgr2=0x7f3f2c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:39.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f44def640 1 --2- 172.21.15.46:0/452198232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c003980 0x7f3f2c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:39.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f44def640 1 -- 172.21.15.46:0/452198232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c003530 con 0x7f3f2c0047d0 2023-10-30T13:54:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f44def640 1 --2- 172.21.15.46:0/452198232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f2c0047d0 0x7f3f2c004c50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f3f30009ae0 tx=0x7f3f3001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6492d99437cc153 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f300a55a0 con 0x7f3f2c0047d0 2023-10-30T13:54:39.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f300a5700 con 0x7f3f2c0047d0 2023-10-30T13:54:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f37fff640 1 -- 172.21.15.46:0/452198232 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f300a5ab0 con 0x7f3f2c0047d0 2023-10-30T13:54:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f47854640 1 -- 172.21.15.46:0/452198232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f2c0047d0 msgr2=0x7f3f2c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f47854640 1 --2- 172.21.15.46:0/452198232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f2c0047d0 0x7f3f2c004c50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f3f30009ae0 tx=0x7f3f3001ca90 comp rx=0 tx=0).stop 2023-10-30T13:54:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f45df1640 1 -- 172.21.15.46:0/452198232 reap_dead start 2023-10-30T13:54:39.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.838+0000 7f3f47854640 1 -- 172.21.15.46:0/452198232 shutdown_connections 2023-10-30T13:54:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f47854640 1 -- 172.21.15.46:0/452198232 >> 172.21.15.46:0/452198232 conn(0x7f3f4006bfe0 msgr2=0x7f3f4006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f47854640 1 -- 172.21.15.46:0/452198232 shutdown_connections 2023-10-30T13:54:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f47854640 1 -- 172.21.15.46:0/452198232 wait complete. 2023-10-30T13:54:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f47854640 1 Processor -- start 2023-10-30T13:54:39.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f47854640 1 -- start start 2023-10-30T13:54:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f47854640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f2c0047d0 0x7f3f4015bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f47854640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f4015c260 0x7f3f40158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f47854640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f40159060 0x7f3f401f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f47854640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f400cd0b0 con 0x7f3f2c0047d0 2023-10-30T13:54:39.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f47854640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4015b2c0 con 0x7f3f40159060 2023-10-30T13:54:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f47854640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f401efd80 con 0x7f3f4015c260 2023-10-30T13:54:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f455f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f2c0047d0 0x7f3f4015bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f455f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f2c0047d0 0x7f3f4015bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35960/0 (socket says 172.21.15.46:35960) 2023-10-30T13:54:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f455f0640 1 -- 172.21.15.46:0/4216616782 learned_addr learned my addr 172.21.15.46:0/4216616782 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f44def640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f4015c260 0x7f3f40158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f45df1640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f40159060 0x7f3f401f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:39.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f455f0640 1 -- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f4015c260 msgr2=0x7f3f40158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:39.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f455f0640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f4015c260 0x7f3f40158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:39.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f455f0640 1 -- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f40159060 msgr2=0x7f3f401f2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:39.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f455f0640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f40159060 0x7f3f401f2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:39.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f455f0640 1 -- 172.21.15.46:0/4216616782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4015a920 con 0x7f3f2c0047d0 2023-10-30T13:54:39.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f45df1640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f40159060 0x7f3f401f2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:39.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f455f0640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f2c0047d0 0x7f3f4015bd20 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c00e850 tx=0x7f3f3c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:39.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f35ffb640 1 -- 172.21.15.46:0/4216616782 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c00c8d0 con 0x7f3f2c0047d0 2023-10-30T13:54:39.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f35ffb640 1 -- 172.21.15.46:0/4216616782 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c00ca30 con 0x7f3f2c0047d0 2023-10-30T13:54:39.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f35ffb640 1 -- 172.21.15.46:0/4216616782 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c00cde0 con 0x7f3f2c0047d0 2023-10-30T13:54:39.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.842+0000 7f3f47854640 1 -- 172.21.15.46:0/4216616782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4015c6a0 con 0x7f3f2c0047d0 2023-10-30T13:54:39.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.846+0000 7f3f47854640 1 -- 172.21.15.46:0/4216616782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4015b510 con 0x7f3f2c0047d0 2023-10-30T13:54:39.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.846+0000 7f3f47854640 1 -- 172.21.15.46:0/4216616782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4015b700 con 0x7f3f2c0047d0 2023-10-30T13:54:39.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.846+0000 7f3f35ffb640 1 -- 172.21.15.46:0/4216616782 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3f3c01d460 con 0x7f3f2c0047d0 2023-10-30T13:54:39.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.846+0000 7f3f35ffb640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f24066520 0x7f3f240689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:39.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.846+0000 7f3f35ffb640 1 -- 172.21.15.46:0/4216616782 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f3f3c0844d0 con 0x7f3f2c0047d0 2023-10-30T13:54:39.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.846+0000 7f3f44def640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f24066520 0x7f3f240689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:39.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.850+0000 7f3f44def640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f24066520 0x7f3f240689e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3f300af000 tx=0x7f3f300046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:39.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:39.850+0000 7f3f35ffb640 1 -- 172.21.15.46:0/4216616782 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3f3c051340 con 0x7f3f2c0047d0 2023-10-30T13:54:40.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.042+0000 7f3f47854640 1 -- 172.21.15.46:0/4216616782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f4004e170 con 0x7f3f2c0047d0 2023-10-30T13:54:40.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.042+0000 7f3f35ffb640 1 -- 172.21.15.46:0/4216616782 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f3f3c054960 con 0x7f3f2c0047d0 2023-10-30T13:54:40.047 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:40.047 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:54:39.265084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 139 2023-10-30T13:54:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.046+0000 7f3f47854640 1 -- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f24066520 msgr2=0x7f3f240689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.046+0000 7f3f47854640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f24066520 0x7f3f240689e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3f300af000 tx=0x7f3f300046d0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.046+0000 7f3f47854640 1 -- 172.21.15.46:0/4216616782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f2c0047d0 msgr2=0x7f3f4015bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.046+0000 7f3f47854640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f2c0047d0 0x7f3f4015bd20 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c00e850 tx=0x7f3f3c00ed10 comp rx=0 tx=0).stop 2023-10-30T13:54:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.046+0000 7f3f47854640 1 -- 172.21.15.46:0/4216616782 shutdown_connections 2023-10-30T13:54:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.046+0000 7f3f47854640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f24066520 0x7f3f240689e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.046+0000 7f3f47854640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f40159060 0x7f3f401f2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.046+0000 7f3f47854640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f4015c260 0x7f3f40158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.046+0000 7f3f47854640 1 --2- 172.21.15.46:0/4216616782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f2c0047d0 0x7f3f4015bd20 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.046+0000 7f3f47854640 1 -- 172.21.15.46:0/4216616782 >> 172.21.15.46:0/4216616782 conn(0x7f3f4006bfe0 msgr2=0x7f3f40151870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:40.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.046+0000 7f3f47854640 1 -- 172.21.15.46:0/4216616782 shutdown_connections 2023-10-30T13:54:40.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.046+0000 7f3f47854640 1 -- 172.21.15.46:0/4216616782 wait complete. 2023-10-30T13:54:40.067 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:54:40.068 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:54:40.068 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:40.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41e2dd3640 1 Processor -- start 2023-10-30T13:54:40.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41e2dd3640 1 -- start start 2023-10-30T13:54:40.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41e2dd3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f41dc06f7a0 0x7f41dc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:40.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41e2dd3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc066cf0 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41e2dd3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc0c70d0 con 0x7f41dc071110 2023-10-30T13:54:40.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41e2dd3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41dc0700c0 con 0x7f41dc06f7a0 2023-10-30T13:54:40.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41e0b6f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f41dc06f7a0 0x7f41dc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:40.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41dbfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f41dc0c6bd0 0x7f41dc069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60822/0 (socket says 172.21.15.46:60822) 2023-10-30T13:54:40.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41dbfff640 1 -- 172.21.15.46:0/26820824 learned_addr learned my addr 172.21.15.46:0/26820824 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:40.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41e0b6f640 1 -- 172.21.15.46:0/26820824 >> v1:172.21.15.154:6789/0 conn(0x7f41dc071110 legacy=0x7f41dc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:40.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41e0b6f640 1 -- 172.21.15.46:0/26820824 >> v1:172.21.15.46:6789/0 conn(0x7f41dc0c6bd0 legacy=0x7f41dc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:40.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41e0b6f640 1 -- 172.21.15.46:0/26820824 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc0c7390 con 0x7f41dc06f7a0 2023-10-30T13:54:40.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.210+0000 7f41db7fe640 1 -- 172.21.15.46:0/26820824 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782151348 0 0) 0x7f41dc066cf0 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.214+0000 7f41e0b6f640 1 --2- 172.21.15.46:0/26820824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc06f7a0 0x7f41dc06fb80 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f41d00097c0 tx=0x7f41d001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5874471d12024a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:40.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.214+0000 7f41db7fe640 1 -- 172.21.15.46:0/26820824 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d00a8970 con 0x7f41dc06f7a0 2023-10-30T13:54:40.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.214+0000 7f41db7fe640 1 -- 172.21.15.46:0/26820824 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d00a8ad0 con 0x7f41dc06f7a0 2023-10-30T13:54:40.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.214+0000 7f41db7fe640 1 -- 172.21.15.46:0/26820824 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d00a8e60 con 0x7f41dc06f7a0 2023-10-30T13:54:40.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.214+0000 7f41e2dd3640 1 -- 172.21.15.46:0/26820824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc06f7a0 msgr2=0x7f41dc06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:40.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.214+0000 7f41e2dd3640 1 --2- 172.21.15.46:0/26820824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc06f7a0 0x7f41dc06fb80 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f41d00097c0 tx=0x7f41d001ec70 comp rx=0 tx=0).stop 2023-10-30T13:54:40.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.214+0000 7f41e2dd3640 1 -- 172.21.15.46:0/26820824 shutdown_connections 2023-10-30T13:54:40.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.214+0000 7f41e2dd3640 1 --2- 172.21.15.46:0/26820824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc06f7a0 0x7f41dc06fb80 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.214+0000 7f41e2dd3640 1 -- 172.21.15.46:0/26820824 >> 172.21.15.46:0/26820824 conn(0x7f41dc0c7910 msgr2=0x7f41dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:40.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.214+0000 7f41e2dd3640 1 -- 172.21.15.46:0/26820824 shutdown_connections 2023-10-30T13:54:40.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.214+0000 7f41e2dd3640 1 -- 172.21.15.46:0/26820824 wait complete. 2023-10-30T13:54:40.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e2dd3640 1 Processor -- start 2023-10-30T13:54:40.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e2dd3640 1 -- start start 2023-10-30T13:54:40.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e2dd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41dc06f7a0 0x7f41dc152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:40.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e2dd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41dc071110 0x7f41dc1de540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:40.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e2dd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc0c6bd0 0x7f41dc1ed010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:40.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e2dd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc1ecbc0 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e2dd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc1f02b0 con 0x7f41dc06f7a0 2023-10-30T13:54:40.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e2dd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc1f0590 con 0x7f41dc071110 2023-10-30T13:54:40.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e1370640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc0c6bd0 0x7f41dc1ed010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:40.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e1370640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc0c6bd0 0x7f41dc1ed010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35974/0 (socket says 172.21.15.46:35974) 2023-10-30T13:54:40.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e1370640 1 -- 172.21.15.46:0/3301084151 learned_addr learned my addr 172.21.15.46:0/3301084151 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:40.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e0b6f640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41dc06f7a0 0x7f41dc152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:40.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41dbfff640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41dc071110 0x7f41dc1de540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e1370640 1 -- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41dc071110 msgr2=0x7f41dc1de540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e1370640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41dc071110 0x7f41dc1de540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e1370640 1 -- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41dc06f7a0 msgr2=0x7f41dc152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e1370640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41dc06f7a0 0x7f41dc152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e1370640 1 -- 172.21.15.46:0/3301084151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d0009420 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e1370640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc0c6bd0 0x7f41dc1ed010 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f41d400c730 tx=0x7f41d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41d97fa640 1 -- 172.21.15.46:0/3301084151 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d4003b80 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41d97fa640 1 -- 172.21.15.46:0/3301084151 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d4003ce0 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41d97fa640 1 -- 172.21.15.46:0/3301084151 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d400aa30 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e2dd3640 1 -- 172.21.15.46:0/3301084151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc1f0870 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.218+0000 7f41e2dd3640 1 -- 172.21.15.46:0/3301084151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc1f0e30 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.222+0000 7f41e2dd3640 1 -- 172.21.15.46:0/3301084151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41dc1ecd30 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.222+0000 7f41d97fa640 1 -- 172.21.15.46:0/3301084151 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41d400ab90 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.222+0000 7f41d97fa640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41c00664a0 0x7f41c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:40.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.222+0000 7f41d97fa640 1 -- 172.21.15.46:0/3301084151 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f41d40841a0 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.222+0000 7f41e0b6f640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41c00664a0 0x7f41c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.222+0000 7f41e0b6f640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41c00664a0 0x7f41c0068960 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f41d00b2000 tx=0x7f41d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:40.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.226+0000 7f41d97fa640 1 -- 172.21.15.46:0/3301084151 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41dc1ecd30 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.418+0000 7f41e2dd3640 1 -- 172.21.15.46:0/3301084151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41dc1ecd00 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.418+0000 7f41d97fa640 1 -- 172.21.15.46:0/3301084151 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f41d4051020 con 0x7f41dc0c6bd0 2023-10-30T13:54:40.422 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:40.422 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:54:39.265084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 139 2023-10-30T13:54:40.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.422+0000 7f41e2dd3640 1 -- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41c00664a0 msgr2=0x7f41c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:40.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.422+0000 7f41e2dd3640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41c00664a0 0x7f41c0068960 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f41d00b2000 tx=0x7f41d0008040 comp rx=0 tx=0).stop 2023-10-30T13:54:40.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.422+0000 7f41e2dd3640 1 -- 172.21.15.46:0/3301084151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc0c6bd0 msgr2=0x7f41dc1ed010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:40.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.422+0000 7f41e2dd3640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc0c6bd0 0x7f41dc1ed010 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f41d400c730 tx=0x7f41d400cbf0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.422+0000 7f41e2dd3640 1 -- 172.21.15.46:0/3301084151 shutdown_connections 2023-10-30T13:54:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.422+0000 7f41e2dd3640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41c00664a0 0x7f41c0068960 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.422+0000 7f41e2dd3640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41dc0c6bd0 0x7f41dc1ed010 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.422+0000 7f41e2dd3640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41dc071110 0x7f41dc1de540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.422+0000 7f41e2dd3640 1 --2- 172.21.15.46:0/3301084151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41dc06f7a0 0x7f41dc152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.422+0000 7f41e2dd3640 1 -- 172.21.15.46:0/3301084151 >> 172.21.15.46:0/3301084151 conn(0x7f41dc0c7910 msgr2=0x7f41dc0c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.422+0000 7f41e2dd3640 1 -- 172.21.15.46:0/3301084151 shutdown_connections 2023-10-30T13:54:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:40.422+0000 7f41e2dd3640 1 -- 172.21.15.46:0/3301084151 wait complete. 2023-10-30T13:54:40.444 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 139, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 139, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7188": { "addr": "172.21.15.46:6836/711135361", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 711135361, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 711135361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7188, "incarnation": 132, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 22 }, "gid_7356": { "addr": "172.21.15.46:6837/3112266825", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3112266825, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3112266825, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7356, "incarnation": 139, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:54:39.265084+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7356, "mds_1": 7188 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:54:42.444 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:42.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.590+0000 7f63d373e640 1 Processor -- start 2023-10-30T13:54:42.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.590+0000 7f63d373e640 1 -- start start 2023-10-30T13:54:42.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d373e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f63cc0cb550 0x7f63cc0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:42.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d373e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f63cc071250 0x7f63cc071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:42.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d373e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc1560a0 con 0x7f63cc14fd50 2023-10-30T13:54:42.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d373e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc156380 con 0x7f63cc0cb550 2023-10-30T13:54:42.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d373e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc156660 con 0x7f63cc071250 2023-10-30T13:54:42.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d0cd9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f63cc0cb550 0x7f63cc0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:42.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d0cd9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f63cc0cb550 0x7f63cc0cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35982/0 (socket says 172.21.15.46:35982) 2023-10-30T13:54:42.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d0cd9640 1 -- 172.21.15.46:0/1675961909 learned_addr learned my addr 172.21.15.46:0/1675961909 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:42.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d1cdb640 1 --2- 172.21.15.46:0/1675961909 >> v2:172.21.15.154:3300/0 conn(0x7f63cc071250 0x7f63cc071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:42.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d0cd9640 1 -- 172.21.15.46:0/1675961909 >> v2:172.21.15.154:3300/0 conn(0x7f63cc071250 msgr2=0x7f63cc071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:42.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d0cd9640 1 --2- 172.21.15.46:0/1675961909 >> v2:172.21.15.154:3300/0 conn(0x7f63cc071250 0x7f63cc071650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:42.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d0cd9640 1 -- 172.21.15.46:0/1675961909 >> v1:172.21.15.154:6789/0 conn(0x7f63cc14fd50 legacy=0x7f63cc0c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:42.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.594+0000 7f63d0cd9640 1 -- 172.21.15.46:0/1675961909 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc156940 con 0x7f63cc0cb550 2023-10-30T13:54:42.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63d0cd9640 1 --2- 172.21.15.46:0/1675961909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63cc0cb550 0x7f63cc0cb9e0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0c9010 tx=0x7f63bc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b02de8ca6b78da7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:42.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63c3fff640 1 -- 172.21.15.46:0/1675961909 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0ab600 con 0x7f63cc0cb550 2023-10-30T13:54:42.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63c3fff640 1 -- 172.21.15.46:0/1675961909 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc0a7050 con 0x7f63cc0cb550 2023-10-30T13:54:42.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63c3fff640 1 -- 172.21.15.46:0/1675961909 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0ab980 con 0x7f63cc0cb550 2023-10-30T13:54:42.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63d373e640 1 -- 172.21.15.46:0/1675961909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63cc0cb550 msgr2=0x7f63cc0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:42.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63d373e640 1 --2- 172.21.15.46:0/1675961909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63cc0cb550 0x7f63cc0cb9e0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0c9010 tx=0x7f63bc01c500 comp rx=0 tx=0).stop 2023-10-30T13:54:42.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63d373e640 1 -- 172.21.15.46:0/1675961909 shutdown_connections 2023-10-30T13:54:42.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63d373e640 1 --2- 172.21.15.46:0/1675961909 >> v2:172.21.15.154:3300/0 conn(0x7f63cc071250 0x7f63cc071650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:42.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63d373e640 1 --2- 172.21.15.46:0/1675961909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63cc0cb550 0x7f63cc0cb9e0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:42.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63d373e640 1 -- 172.21.15.46:0/1675961909 >> 172.21.15.46:0/1675961909 conn(0x7f63cc06c6e0 msgr2=0x7f63cc069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:42.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63d373e640 1 -- 172.21.15.46:0/1675961909 shutdown_connections 2023-10-30T13:54:42.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63d373e640 1 -- 172.21.15.46:0/1675961909 wait complete. 2023-10-30T13:54:42.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63d373e640 1 Processor -- start 2023-10-30T13:54:42.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.598+0000 7f63d373e640 1 -- start start 2023-10-30T13:54:42.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d373e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63cc071250 0x7f63cc154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:42.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d373e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63cc0cb550 0x7f63cc1f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:42.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d373e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63cc14fd50 0x7f63cc1f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:42.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d373e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1f18d0 con 0x7f63cc071250 2023-10-30T13:54:42.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d373e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1f5a30 con 0x7f63cc0cb550 2023-10-30T13:54:42.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d373e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc14d520 con 0x7f63cc14fd50 2023-10-30T13:54:42.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d14da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63cc071250 0x7f63cc154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:42.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d14da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63cc071250 0x7f63cc154ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35996/0 (socket says 172.21.15.46:35996) 2023-10-30T13:54:42.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d14da640 1 -- 172.21.15.46:0/3035852046 learned_addr learned my addr 172.21.15.46:0/3035852046 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:42.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d1cdb640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63cc14fd50 0x7f63cc1f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:42.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d0cd9640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63cc0cb550 0x7f63cc1f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:42.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d14da640 1 -- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63cc14fd50 msgr2=0x7f63cc1f2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:42.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d14da640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63cc14fd50 0x7f63cc1f2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:42.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d14da640 1 -- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63cc0cb550 msgr2=0x7f63cc1f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:42.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d14da640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63cc0cb550 0x7f63cc1f1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:42.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d14da640 1 -- 172.21.15.46:0/3035852046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc0094d0 con 0x7f63cc071250 2023-10-30T13:54:42.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d14da640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63cc071250 0x7f63cc154ca0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f63c800c500 tx=0x7f63c800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:42.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63c1ffb640 1 -- 172.21.15.46:0/3035852046 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8005220 con 0x7f63cc071250 2023-10-30T13:54:42.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63c1ffb640 1 -- 172.21.15.46:0/3035852046 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c8005380 con 0x7f63cc071250 2023-10-30T13:54:42.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d373e640 1 -- 172.21.15.46:0/3035852046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc14d800 con 0x7f63cc071250 2023-10-30T13:54:42.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63c1ffb640 1 -- 172.21.15.46:0/3035852046 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8007270 con 0x7f63cc071250 2023-10-30T13:54:42.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d373e640 1 -- 172.21.15.46:0/3035852046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc1f5700 con 0x7f63cc071250 2023-10-30T13:54:42.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63d373e640 1 -- 172.21.15.46:0/3035852046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc04e170 con 0x7f63cc071250 2023-10-30T13:54:42.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.602+0000 7f63c1ffb640 1 -- 172.21.15.46:0/3035852046 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f63c80073d0 con 0x7f63cc071250 2023-10-30T13:54:42.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.606+0000 7f63c1ffb640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63b00664d0 0x7f63b0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:42.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.606+0000 7f63c1ffb640 1 -- 172.21.15.46:0/3035852046 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f63c8083da0 con 0x7f63cc071250 2023-10-30T13:54:42.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.606+0000 7f63d0cd9640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63b00664d0 0x7f63b0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:42.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.606+0000 7f63d0cd9640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63b00664d0 0x7f63b0068990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0ad000 tx=0x7f63bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:42.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.610+0000 7f63c1ffb640 1 -- 172.21.15.46:0/3035852046 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f63c8050ba0 con 0x7f63cc071250 2023-10-30T13:54:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.798+0000 7f63d373e640 1 -- 172.21.15.46:0/3035852046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63cc0c8300 con 0x7f63cc071250 2023-10-30T13:54:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.798+0000 7f63c1ffb640 1 -- 172.21.15.46:0/3035852046 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f63c80541c0 con 0x7f63cc071250 2023-10-30T13:54:42.803 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:42.803 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:54:39.265084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 139 2023-10-30T13:54:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.802+0000 7f63d373e640 1 -- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63b00664d0 msgr2=0x7f63b0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.802+0000 7f63d373e640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63b00664d0 0x7f63b0068990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0ad000 tx=0x7f63bc008040 comp rx=0 tx=0).stop 2023-10-30T13:54:42.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.802+0000 7f63d373e640 1 -- 172.21.15.46:0/3035852046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63cc071250 msgr2=0x7f63cc154ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.802+0000 7f63d373e640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63cc071250 0x7f63cc154ca0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f63c800c500 tx=0x7f63c800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:54:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.802+0000 7f63d373e640 1 -- 172.21.15.46:0/3035852046 shutdown_connections 2023-10-30T13:54:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.802+0000 7f63d373e640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63b00664d0 0x7f63b0068990 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.802+0000 7f63d373e640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63cc14fd50 0x7f63cc1f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.802+0000 7f63d373e640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63cc0cb550 0x7f63cc1f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:42.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.802+0000 7f63d373e640 1 --2- 172.21.15.46:0/3035852046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63cc071250 0x7f63cc154ca0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:42.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.802+0000 7f63d373e640 1 -- 172.21.15.46:0/3035852046 >> 172.21.15.46:0/3035852046 conn(0x7f63cc06c6e0 msgr2=0x7f63cc15fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:42.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.802+0000 7f63d373e640 1 -- 172.21.15.46:0/3035852046 shutdown_connections 2023-10-30T13:54:42.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:42.802+0000 7f63d373e640 1 -- 172.21.15.46:0/3035852046 wait complete. 2023-10-30T13:54:42.824 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:54:44.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:44.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.970+0000 7f1e1d93e640 1 Processor -- start 2023-10-30T13:54:44.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.970+0000 7f1e1d93e640 1 -- start start 2023-10-30T13:54:44.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.970+0000 7f1e1d93e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1e181538a0 0x7f1e1815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:44.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.970+0000 7f1e1d93e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e180cf160 0x7f1e180cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:44.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.970+0000 7f1e1d93e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e180cb800 con 0x7f1e180ca0e0 2023-10-30T13:54:44.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.970+0000 7f1e1d93e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1815dc30 con 0x7f1e181538a0 2023-10-30T13:54:44.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.970+0000 7f1e1d93e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180cb330 con 0x7f1e180cf160 2023-10-30T13:54:44.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.974+0000 7f1e16ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1e181538a0 0x7f1e1815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:44.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.974+0000 7f1e16ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1e181538a0 0x7f1e1815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35998/0 (socket says 172.21.15.46:35998) 2023-10-30T13:54:44.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.974+0000 7f1e16ffd640 1 -- 172.21.15.46:0/239849916 learned_addr learned my addr 172.21.15.46:0/239849916 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:44.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.974+0000 7f1e177fe640 1 --2- 172.21.15.46:0/239849916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e180cf160 0x7f1e180cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:44.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.974+0000 7f1e16ffd640 1 -- 172.21.15.46:0/239849916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e180cf160 msgr2=0x7f1e180cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:44.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.974+0000 7f1e16ffd640 1 --2- 172.21.15.46:0/239849916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e180cf160 0x7f1e180cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:44.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.974+0000 7f1e16ffd640 1 -- 172.21.15.46:0/239849916 >> v1:172.21.15.46:6789/0 conn(0x7f1e180ca0e0 legacy=0x7f1e1815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:44.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.974+0000 7f1e16ffd640 1 -- 172.21.15.46:0/239849916 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e180cad70 con 0x7f1e181538a0 2023-10-30T13:54:44.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.974+0000 7f1e177fe640 1 --2- 172.21.15.46:0/239849916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e180cf160 0x7f1e180cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.974+0000 7f1e16ffd640 1 --2- 172.21.15.46:0/239849916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e181538a0 0x7f1e1815d6f0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009870 tx=0x7f1e0801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83f044b416d4ea1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.974+0000 7f1e15ffb640 1 -- 172.21.15.46:0/239849916 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e080a8a30 con 0x7f1e181538a0 2023-10-30T13:54:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.974+0000 7f1e15ffb640 1 -- 172.21.15.46:0/239849916 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e080a8b90 con 0x7f1e181538a0 2023-10-30T13:54:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e15ffb640 1 -- 172.21.15.46:0/239849916 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e080b0440 con 0x7f1e181538a0 2023-10-30T13:54:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 -- 172.21.15.46:0/239849916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e181538a0 msgr2=0x7f1e1815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 --2- 172.21.15.46:0/239849916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e181538a0 0x7f1e1815d6f0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009870 tx=0x7f1e0801ed30 comp rx=0 tx=0).stop 2023-10-30T13:54:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 -- 172.21.15.46:0/239849916 shutdown_connections 2023-10-30T13:54:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 --2- 172.21.15.46:0/239849916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e180cf160 0x7f1e180cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 --2- 172.21.15.46:0/239849916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e181538a0 0x7f1e1815d6f0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 -- 172.21.15.46:0/239849916 >> 172.21.15.46:0/239849916 conn(0x7f1e1806bfe0 msgr2=0x7f1e1806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 -- 172.21.15.46:0/239849916 shutdown_connections 2023-10-30T13:54:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 -- 172.21.15.46:0/239849916 wait complete. 2023-10-30T13:54:44.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 Processor -- start 2023-10-30T13:54:44.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 -- start start 2023-10-30T13:54:44.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e180ca0e0 0x7f1e181f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:44.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e180cf160 0x7f1e18066180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:44.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e181538a0 0x7f1e1814cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:44.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e181e2f00 con 0x7f1e180cf160 2023-10-30T13:54:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e181e31e0 con 0x7f1e181538a0 2023-10-30T13:54:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e1d93e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e181e34c0 con 0x7f1e180ca0e0 2023-10-30T13:54:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e167fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e180cf160 0x7f1e18066180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e167fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e180cf160 0x7f1e18066180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36006/0 (socket says 172.21.15.46:36006) 2023-10-30T13:54:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.978+0000 7f1e167fc640 1 -- 172.21.15.46:0/161762153 learned_addr learned my addr 172.21.15.46:0/161762153 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1e16ffd640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e180ca0e0 0x7f1e181f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1e177fe640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e181538a0 0x7f1e1814cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1e167fc640 1 -- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e180ca0e0 msgr2=0x7f1e181f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1e167fc640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e180ca0e0 0x7f1e181f2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1e167fc640 1 -- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e181538a0 msgr2=0x7f1e1814cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1e167fc640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e181538a0 0x7f1e1814cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1e167fc640 1 -- 172.21.15.46:0/161762153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e080094d0 con 0x7f1e180cf160 2023-10-30T13:54:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1e177fe640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e181538a0 0x7f1e1814cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1e167fc640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e180cf160 0x7f1e18066180 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00c500 tx=0x7f1e0c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:44.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1dfffff640 1 -- 172.21.15.46:0/161762153 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c005220 con 0x7f1e180cf160 2023-10-30T13:54:44.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1dfffff640 1 -- 172.21.15.46:0/161762153 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0c005380 con 0x7f1e180cf160 2023-10-30T13:54:44.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1e1d93e640 1 -- 172.21.15.46:0/161762153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e181f2740 con 0x7f1e180cf160 2023-10-30T13:54:44.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1dfffff640 1 -- 172.21.15.46:0/161762153 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c007270 con 0x7f1e180cf160 2023-10-30T13:54:44.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1e1d93e640 1 -- 172.21.15.46:0/161762153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e181f3480 con 0x7f1e180cf160 2023-10-30T13:54:44.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.982+0000 7f1e1d93e640 1 -- 172.21.15.46:0/161762153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e180b2a50 con 0x7f1e180cf160 2023-10-30T13:54:44.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.986+0000 7f1dfffff640 1 -- 172.21.15.46:0/161762153 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1e0c0073d0 con 0x7f1e180cf160 2023-10-30T13:54:44.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.986+0000 7f1dfffff640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1e040664a0 0x7f1e04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:44.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.986+0000 7f1dfffff640 1 -- 172.21.15.46:0/161762153 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f1e0c083db0 con 0x7f1e180cf160 2023-10-30T13:54:44.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.986+0000 7f1e16ffd640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1e040664a0 0x7f1e04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:44.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.986+0000 7f1e16ffd640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1e040664a0 0x7f1e04068960 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1e080b2000 tx=0x7f1e08009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:44.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:44.990+0000 7f1dfffff640 1 -- 172.21.15.46:0/161762153 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1e0c088070 con 0x7f1e180cf160 2023-10-30T13:54:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.186+0000 7f1e1d93e640 1 -- 172.21.15.46:0/161762153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e1804e170 con 0x7f1e180cf160 2023-10-30T13:54:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.186+0000 7f1dfffff640 1 -- 172.21.15.46:0/161762153 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f1e0c050bb0 con 0x7f1e180cf160 2023-10-30T13:54:45.191 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:45.191 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:54:39.265084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:45.191 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 139 2023-10-30T13:54:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.190+0000 7f1e1d93e640 1 -- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1e040664a0 msgr2=0x7f1e04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.190+0000 7f1e1d93e640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1e040664a0 0x7f1e04068960 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1e080b2000 tx=0x7f1e08009040 comp rx=0 tx=0).stop 2023-10-30T13:54:45.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.190+0000 7f1e1d93e640 1 -- 172.21.15.46:0/161762153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e180cf160 msgr2=0x7f1e18066180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.190+0000 7f1e1d93e640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e180cf160 0x7f1e18066180 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00c500 tx=0x7f1e0c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:54:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.190+0000 7f1e1d93e640 1 -- 172.21.15.46:0/161762153 shutdown_connections 2023-10-30T13:54:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.190+0000 7f1e1d93e640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1e040664a0 0x7f1e04068960 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.190+0000 7f1e1d93e640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e181538a0 0x7f1e1814cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:45.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.190+0000 7f1e1d93e640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e180cf160 0x7f1e18066180 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.190+0000 7f1e1d93e640 1 --2- 172.21.15.46:0/161762153 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e180ca0e0 0x7f1e181f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.190+0000 7f1e1d93e640 1 -- 172.21.15.46:0/161762153 >> 172.21.15.46:0/161762153 conn(0x7f1e1806bfe0 msgr2=0x7f1e18151940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.190+0000 7f1e1d93e640 1 -- 172.21.15.46:0/161762153 shutdown_connections 2023-10-30T13:54:45.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:45.190+0000 7f1e1d93e640 1 -- 172.21.15.46:0/161762153 wait complete. 2023-10-30T13:54:45.213 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:54:47.215 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:47.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.358+0000 7fd6e7891640 1 Processor -- start 2023-10-30T13:54:47.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.362+0000 7fd6e7891640 1 -- start start 2023-10-30T13:54:47.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.362+0000 7fd6e7891640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6e015d6f0 0x7fd6e015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:47.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.362+0000 7fd6e7891640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e00cb800 con 0x7fd6e00cf160 2023-10-30T13:54:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.362+0000 7fd6e7891640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e00cb330 con 0x7fd6e01538a0 2023-10-30T13:54:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.362+0000 7fd6e7891640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e00cad70 con 0x7fd6e015d6f0 2023-10-30T13:54:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.362+0000 7fd6e4e2c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6e015d6f0 0x7fd6e015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.362+0000 7fd6e562d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd6e01538a0 0x7fd6e00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:53836/0 (socket says 172.21.15.46:53836) 2023-10-30T13:54:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.362+0000 7fd6e562d640 1 -- 172.21.15.46:0/3165433491 learned_addr learned my addr 172.21.15.46:0/3165433491 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.362+0000 7fd6e4e2c640 1 -- 172.21.15.46:0/3165433491 >> v1:172.21.15.154:6789/0 conn(0x7fd6e01538a0 legacy=0x7fd6e00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.362+0000 7fd6e4e2c640 1 -- 172.21.15.46:0/3165433491 >> v1:172.21.15.46:6789/0 conn(0x7fd6e00cf160 legacy=0x7fd6e0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.362+0000 7fd6d7fff640 1 -- 172.21.15.46:0/3165433491 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418017711 0 0) 0x7fd6e00cb330 con 0x7fd6e01538a0 2023-10-30T13:54:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.362+0000 7fd6e4e2c640 1 -- 172.21.15.46:0/3165433491 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e00cb050 con 0x7fd6e015d6f0 2023-10-30T13:54:47.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.366+0000 7fd6e4e2c640 1 --2- 172.21.15.46:0/3165433491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e015d6f0 0x7fd6e015a090 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00097c0 tx=0x7fd6d001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24eb2798013ce31b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:47.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.366+0000 7fd6d7fff640 1 -- 172.21.15.46:0/3165433491 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00a8970 con 0x7fd6e015d6f0 2023-10-30T13:54:47.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.366+0000 7fd6d7fff640 1 -- 172.21.15.46:0/3165433491 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d00a8ad0 con 0x7fd6e015d6f0 2023-10-30T13:54:47.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.366+0000 7fd6d7fff640 1 -- 172.21.15.46:0/3165433491 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00a8e60 con 0x7fd6e015d6f0 2023-10-30T13:54:47.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.370+0000 7fd6e7891640 1 -- 172.21.15.46:0/3165433491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e015d6f0 msgr2=0x7fd6e015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:47.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.370+0000 7fd6e7891640 1 --2- 172.21.15.46:0/3165433491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e015d6f0 0x7fd6e015a090 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00097c0 tx=0x7fd6d001ec90 comp rx=0 tx=0).stop 2023-10-30T13:54:47.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.370+0000 7fd6e7891640 1 -- 172.21.15.46:0/3165433491 shutdown_connections 2023-10-30T13:54:47.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.370+0000 7fd6e7891640 1 --2- 172.21.15.46:0/3165433491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e015d6f0 0x7fd6e015a090 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:47.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.370+0000 7fd6e7891640 1 -- 172.21.15.46:0/3165433491 >> 172.21.15.46:0/3165433491 conn(0x7fd6e006bfe0 msgr2=0x7fd6e006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:47.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.370+0000 7fd6e7891640 1 -- 172.21.15.46:0/3165433491 shutdown_connections 2023-10-30T13:54:47.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.370+0000 7fd6e7891640 1 -- 172.21.15.46:0/3165433491 wait complete. 2023-10-30T13:54:47.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.370+0000 7fd6e7891640 1 Processor -- start 2023-10-30T13:54:47.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.370+0000 7fd6e7891640 1 -- start start 2023-10-30T13:54:47.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e7891640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6e00cf160 0x7fd6e014cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:47.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e7891640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e01538a0 0x7fd6e01e4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:47.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e7891640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e015d6f0 0x7fd6e01e4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:47.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e7891640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e01ef350 con 0x7fd6e01538a0 2023-10-30T13:54:47.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e7891640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e0159600 con 0x7fd6e015d6f0 2023-10-30T13:54:47.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e7891640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e014f220 con 0x7fd6e00cf160 2023-10-30T13:54:47.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e4e2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e01538a0 0x7fd6e01e4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e4e2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e01538a0 0x7fd6e01e4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37196/0 (socket says 172.21.15.46:37196) 2023-10-30T13:54:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e4e2c640 1 -- 172.21.15.46:0/1260976300 learned_addr learned my addr 172.21.15.46:0/1260976300 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e562d640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6e00cf160 0x7fd6e014cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e5e2e640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e015d6f0 0x7fd6e01e4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e4e2c640 1 -- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6e00cf160 msgr2=0x7fd6e014cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:47.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e4e2c640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6e00cf160 0x7fd6e014cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:47.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e4e2c640 1 -- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e015d6f0 msgr2=0x7fd6e01e4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:47.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e4e2c640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e015d6f0 0x7fd6e01e4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:47.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e4e2c640 1 -- 172.21.15.46:0/1260976300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d0009420 con 0x7fd6e01538a0 2023-10-30T13:54:47.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e5e2e640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e015d6f0 0x7fd6e01e4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e4e2c640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e01538a0 0x7fd6e01e4680 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00b3040 tx=0x7fd6d001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:47.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6d5ffb640 1 -- 172.21.15.46:0/1260976300 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00b9630 con 0x7fd6e01538a0 2023-10-30T13:54:47.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6d5ffb640 1 -- 172.21.15.46:0/1260976300 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d00b97b0 con 0x7fd6e01538a0 2023-10-30T13:54:47.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6d5ffb640 1 -- 172.21.15.46:0/1260976300 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00b9b60 con 0x7fd6e01538a0 2023-10-30T13:54:47.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e7891640 1 -- 172.21.15.46:0/1260976300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e014f500 con 0x7fd6e01538a0 2023-10-30T13:54:47.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.374+0000 7fd6e7891640 1 -- 172.21.15.46:0/1260976300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e014fac0 con 0x7fd6e01538a0 2023-10-30T13:54:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.378+0000 7fd6e7891640 1 -- 172.21.15.46:0/1260976300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e01e3d00 con 0x7fd6e01538a0 2023-10-30T13:54:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.378+0000 7fd6d5ffb640 1 -- 172.21.15.46:0/1260976300 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6d00b7040 con 0x7fd6e01538a0 2023-10-30T13:54:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.378+0000 7fd6d5ffb640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6c40664a0 0x7fd6c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.378+0000 7fd6d5ffb640 1 -- 172.21.15.46:0/1260976300 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7fd6d0122540 con 0x7fd6e01538a0 2023-10-30T13:54:47.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.378+0000 7fd6e562d640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6c40664a0 0x7fd6c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:47.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.382+0000 7fd6e562d640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6c40664a0 0x7fd6c4068960 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc007fd0 tx=0x7fd6dc007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.382+0000 7fd6d5ffb640 1 -- 172.21.15.46:0/1260976300 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6e01e3d00 con 0x7fd6e01538a0 2023-10-30T13:54:47.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 -- 172.21.15.46:0/1260976300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6e01e3d00 con 0x7fd6e01538a0 2023-10-30T13:54:47.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6d5ffb640 1 -- 172.21.15.46:0/1260976300 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fd6e01e3d00 con 0x7fd6e01538a0 2023-10-30T13:54:47.588 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:47.589 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:54:39.265084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 139 2023-10-30T13:54:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 -- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6c40664a0 msgr2=0x7fd6c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6c40664a0 0x7fd6c4068960 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc007fd0 tx=0x7fd6dc007d60 comp rx=0 tx=0).stop 2023-10-30T13:54:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 -- 172.21.15.46:0/1260976300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e01538a0 msgr2=0x7fd6e01e4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e01538a0 0x7fd6e01e4680 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00b3040 tx=0x7fd6d001ba30 comp rx=0 tx=0).stop 2023-10-30T13:54:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 -- 172.21.15.46:0/1260976300 shutdown_connections 2023-10-30T13:54:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6c40664a0 0x7fd6c4068960 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e015d6f0 0x7fd6e01e4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e01538a0 0x7fd6e01e4680 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 --2- 172.21.15.46:0/1260976300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6e00cf160 0x7fd6e014cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 -- 172.21.15.46:0/1260976300 >> 172.21.15.46:0/1260976300 conn(0x7fd6e006bfe0 msgr2=0x7fd6e00cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 -- 172.21.15.46:0/1260976300 shutdown_connections 2023-10-30T13:54:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:47.586+0000 7fd6e7891640 1 -- 172.21.15.46:0/1260976300 wait complete. 2023-10-30T13:54:47.612 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:54:49.613 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:49.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d8691d640 1 Processor -- start 2023-10-30T13:54:49.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d8691d640 1 -- start start 2023-10-30T13:54:49.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d8691d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d801538a0 0x7f5d8015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:49.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d8691d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d800ca0e0 0x7f5d8015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:49.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d8691d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5d800cf160 0x7f5d800cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d8691d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d800cb800 con 0x7f5d801538a0 2023-10-30T13:54:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d8691d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d8015dc30 con 0x7f5d800cf160 2023-10-30T13:54:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d8691d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d800cb330 con 0x7f5d800ca0e0 2023-10-30T13:54:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d7ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d801538a0 0x7f5d8015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d7ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d801538a0 0x7f5d8015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37202/0 (socket says 172.21.15.46:37202) 2023-10-30T13:54:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d7ffff640 1 -- 172.21.15.46:0/176812947 learned_addr learned my addr 172.21.15.46:0/176812947 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d7f7fe640 1 --2- 172.21.15.46:0/176812947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d800ca0e0 0x7f5d8015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d84eba640 1 --2- 172.21.15.46:0/176812947 >> v2:172.21.15.154:3300/0 conn(0x7f5d800cf160 0x7f5d800cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d7ffff640 1 -- 172.21.15.46:0/176812947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d800ca0e0 msgr2=0x7f5d8015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d7ffff640 1 --2- 172.21.15.46:0/176812947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d800ca0e0 0x7f5d8015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d7ffff640 1 -- 172.21.15.46:0/176812947 >> v2:172.21.15.154:3300/0 conn(0x7f5d800cf160 msgr2=0x7f5d800cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d7ffff640 1 --2- 172.21.15.46:0/176812947 >> v2:172.21.15.154:3300/0 conn(0x7f5d800cf160 0x7f5d800cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d7ffff640 1 -- 172.21.15.46:0/176812947 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d800cad70 con 0x7f5d801538a0 2023-10-30T13:54:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.758+0000 7f5d84eba640 1 --2- 172.21.15.46:0/176812947 >> v2:172.21.15.154:3300/0 conn(0x7f5d800cf160 0x7f5d800cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:54:49.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.762+0000 7f5d7ffff640 1 --2- 172.21.15.46:0/176812947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d801538a0 0x7f5d8015d6f0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5d8006cb00 tx=0x7f5d7001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93de208369246c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:49.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d7effd640 1 -- 172.21.15.46:0/176812947 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d700a89b0 con 0x7f5d801538a0 2023-10-30T13:54:49.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d7effd640 1 -- 172.21.15.46:0/176812947 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d700a8b10 con 0x7f5d801538a0 2023-10-30T13:54:49.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d7effd640 1 -- 172.21.15.46:0/176812947 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d700a8ea0 con 0x7f5d801538a0 2023-10-30T13:54:49.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d8691d640 1 -- 172.21.15.46:0/176812947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d801538a0 msgr2=0x7f5d8015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:49.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d8691d640 1 --2- 172.21.15.46:0/176812947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d801538a0 0x7f5d8015d6f0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5d8006cb00 tx=0x7f5d7001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d8691d640 1 -- 172.21.15.46:0/176812947 shutdown_connections 2023-10-30T13:54:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d8691d640 1 --2- 172.21.15.46:0/176812947 >> v2:172.21.15.154:3300/0 conn(0x7f5d800cf160 0x7f5d800cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d8691d640 1 --2- 172.21.15.46:0/176812947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d800ca0e0 0x7f5d8015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d8691d640 1 --2- 172.21.15.46:0/176812947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d801538a0 0x7f5d8015d6f0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d8691d640 1 -- 172.21.15.46:0/176812947 >> 172.21.15.46:0/176812947 conn(0x7f5d8006bfe0 msgr2=0x7f5d8006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d8691d640 1 -- 172.21.15.46:0/176812947 shutdown_connections 2023-10-30T13:54:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d8691d640 1 -- 172.21.15.46:0/176812947 wait complete. 2023-10-30T13:54:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d8691d640 1 Processor -- start 2023-10-30T13:54:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.766+0000 7f5d8691d640 1 -- start start 2023-10-30T13:54:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d8691d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d800ca0e0 0x7f5d8014ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d8691d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d800cf160 0x7f5d800cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d8691d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d801538a0 0x7f5d801f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d8691d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d80157170 con 0x7f5d800cf160 2023-10-30T13:54:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d8691d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d80146780 con 0x7f5d800ca0e0 2023-10-30T13:54:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d8691d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d800cd440 con 0x7f5d801538a0 2023-10-30T13:54:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d800cf160 0x7f5d800cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d800cf160 0x7f5d800cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37208/0 (socket says 172.21.15.46:37208) 2023-10-30T13:54:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7f7fe640 1 -- 172.21.15.46:0/3820646266 learned_addr learned my addr 172.21.15.46:0/3820646266 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7ffff640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d800ca0e0 0x7f5d8014ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d84eba640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d801538a0 0x7f5d801f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:49.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7f7fe640 1 -- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d801538a0 msgr2=0x7f5d801f24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:49.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7f7fe640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d801538a0 0x7f5d801f24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7f7fe640 1 -- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d800ca0e0 msgr2=0x7f5d8014ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:49.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7f7fe640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d800ca0e0 0x7f5d8014ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7f7fe640 1 -- 172.21.15.46:0/3820646266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d700094d0 con 0x7f5d800cf160 2023-10-30T13:54:49.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d84eba640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d801538a0 0x7f5d801f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:49.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7f7fe640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d800cf160 0x7f5d800cd8d0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f5d7400d830 tx=0x7f5d7400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:49.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7cff9640 1 -- 172.21.15.46:0/3820646266 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d740096a0 con 0x7f5d800cf160 2023-10-30T13:54:49.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7cff9640 1 -- 172.21.15.46:0/3820646266 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d74009800 con 0x7f5d800cf160 2023-10-30T13:54:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d8691d640 1 -- 172.21.15.46:0/3820646266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d80144e10 con 0x7f5d800cf160 2023-10-30T13:54:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d7cff9640 1 -- 172.21.15.46:0/3820646266 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74009b70 con 0x7f5d800cf160 2023-10-30T13:54:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.770+0000 7f5d8691d640 1 -- 172.21.15.46:0/3820646266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d800ceb10 con 0x7f5d800cf160 2023-10-30T13:54:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.774+0000 7f5d8691d640 1 -- 172.21.15.46:0/3820646266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d800b2a50 con 0x7f5d800cf160 2023-10-30T13:54:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.774+0000 7f5d7cff9640 1 -- 172.21.15.46:0/3820646266 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5d74014410 con 0x7f5d800cf160 2023-10-30T13:54:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.774+0000 7f5d7cff9640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d6c0664a0 0x7f5d6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.774+0000 7f5d7cff9640 1 -- 172.21.15.46:0/3820646266 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f5d74084ce0 con 0x7f5d800cf160 2023-10-30T13:54:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.774+0000 7f5d7ffff640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d6c0664a0 0x7f5d6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.778+0000 7f5d7ffff640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d6c0664a0 0x7f5d6c068960 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5d700b2000 tx=0x7f5d70009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.778+0000 7f5d7cff9640 1 -- 172.21.15.46:0/3820646266 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5d74051be0 con 0x7f5d800cf160 2023-10-30T13:54:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.990+0000 7f5d8691d640 1 -- 172.21.15.46:0/3820646266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d800ce970 con 0x7f5d800cf160 2023-10-30T13:54:49.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.990+0000 7f5d7cff9640 1 -- 172.21.15.46:0/3820646266 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f5d800ce970 con 0x7f5d800cf160 2023-10-30T13:54:49.995 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:49.995 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:54:39.265084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:49.995 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 139 2023-10-30T13:54:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.994+0000 7f5d8691d640 1 -- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d6c0664a0 msgr2=0x7f5d6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:49.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.994+0000 7f5d8691d640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d6c0664a0 0x7f5d6c068960 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5d700b2000 tx=0x7f5d70009040 comp rx=0 tx=0).stop 2023-10-30T13:54:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.994+0000 7f5d8691d640 1 -- 172.21.15.46:0/3820646266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d800cf160 msgr2=0x7f5d800cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.994+0000 7f5d8691d640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d800cf160 0x7f5d800cd8d0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f5d7400d830 tx=0x7f5d7400dcf0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.994+0000 7f5d8691d640 1 -- 172.21.15.46:0/3820646266 shutdown_connections 2023-10-30T13:54:49.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.994+0000 7f5d8691d640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d6c0664a0 0x7f5d6c068960 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.994+0000 7f5d8691d640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d801538a0 0x7f5d801f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.994+0000 7f5d8691d640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d800cf160 0x7f5d800cd8d0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.994+0000 7f5d8691d640 1 --2- 172.21.15.46:0/3820646266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d800ca0e0 0x7f5d8014ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.994+0000 7f5d8691d640 1 -- 172.21.15.46:0/3820646266 >> 172.21.15.46:0/3820646266 conn(0x7f5d8006bfe0 msgr2=0x7f5d800c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:49.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.994+0000 7f5d8691d640 1 -- 172.21.15.46:0/3820646266 shutdown_connections 2023-10-30T13:54:49.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:49.994+0000 7f5d8691d640 1 -- 172.21.15.46:0/3820646266 wait complete. 2023-10-30T13:54:50.016 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:54:51.907 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:51.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:52.017 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.166+0000 7f69edbaa640 1 Processor -- start 2023-10-30T13:54:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.166+0000 7f69edbaa640 1 -- start start 2023-10-30T13:54:52.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69edbaa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69e806f7a0 0x7f69e806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:52.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69edbaa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e80c6bd0 0x7f69e8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:52.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69edbaa640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f69e8071110 0x7f69e8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69edbaa640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e80700c0 con 0x7f69e806f7a0 2023-10-30T13:54:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69edbaa640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e806a3d0 con 0x7f69e8071110 2023-10-30T13:54:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69edbaa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e8066cf0 con 0x7f69e80c6bd0 2023-10-30T13:54:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69e77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69e806f7a0 0x7f69e806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69e77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69e806f7a0 0x7f69e806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37210/0 (socket says 172.21.15.46:37210) 2023-10-30T13:54:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69e77fe640 1 -- 172.21.15.46:0/3621174727 learned_addr learned my addr 172.21.15.46:0/3621174727 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69e7fff640 1 --2- 172.21.15.46:0/3621174727 >> v2:172.21.15.154:3300/0 conn(0x7f69e8071110 0x7f69e8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69e6ffd640 1 --2- 172.21.15.46:0/3621174727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e80c6bd0 0x7f69e8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69e77fe640 1 -- 172.21.15.46:0/3621174727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e80c6bd0 msgr2=0x7f69e8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69e77fe640 1 --2- 172.21.15.46:0/3621174727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e80c6bd0 0x7f69e8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69e77fe640 1 -- 172.21.15.46:0/3621174727 >> v2:172.21.15.154:3300/0 conn(0x7f69e8071110 msgr2=0x7f69e8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69e77fe640 1 --2- 172.21.15.46:0/3621174727 >> v2:172.21.15.154:3300/0 conn(0x7f69e8071110 0x7f69e8158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69e77fe640 1 -- 172.21.15.46:0/3621174727 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e80c70d0 con 0x7f69e806f7a0 2023-10-30T13:54:52.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.170+0000 7f69e77fe640 1 --2- 172.21.15.46:0/3621174727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e806f7a0 0x7f69e806fb80 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f69e806c7d0 tx=0x7f69d801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b8c015f9b6382d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:52.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69e67fc640 1 -- 172.21.15.46:0/3621174727 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d80a89b0 con 0x7f69e806f7a0 2023-10-30T13:54:52.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69e67fc640 1 -- 172.21.15.46:0/3621174727 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d80a8b10 con 0x7f69e806f7a0 2023-10-30T13:54:52.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69e67fc640 1 -- 172.21.15.46:0/3621174727 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d80a8ea0 con 0x7f69e806f7a0 2023-10-30T13:54:52.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 -- 172.21.15.46:0/3621174727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e806f7a0 msgr2=0x7f69e806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:52.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 --2- 172.21.15.46:0/3621174727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e806f7a0 0x7f69e806fb80 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f69e806c7d0 tx=0x7f69d801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:54:52.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 -- 172.21.15.46:0/3621174727 shutdown_connections 2023-10-30T13:54:52.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 --2- 172.21.15.46:0/3621174727 >> v2:172.21.15.154:3300/0 conn(0x7f69e8071110 0x7f69e8158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 --2- 172.21.15.46:0/3621174727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e80c6bd0 0x7f69e8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 --2- 172.21.15.46:0/3621174727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e806f7a0 0x7f69e806fb80 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 -- 172.21.15.46:0/3621174727 >> 172.21.15.46:0/3621174727 conn(0x7f69e80c7910 msgr2=0x7f69e80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 -- 172.21.15.46:0/3621174727 shutdown_connections 2023-10-30T13:54:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 -- 172.21.15.46:0/3621174727 wait complete. 2023-10-30T13:54:52.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 Processor -- start 2023-10-30T13:54:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 -- start start 2023-10-30T13:54:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e806f7a0 0x7f69e8156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e8071110 0x7f69e814e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e80c6bd0 0x7f69e814e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:52.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e815a430 con 0x7f69e806f7a0 2023-10-30T13:54:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e80c8d40 con 0x7f69e80c6bd0 2023-10-30T13:54:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69edbaa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e815a7b0 con 0x7f69e8071110 2023-10-30T13:54:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69e77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e806f7a0 0x7f69e8156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:52.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69e77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e806f7a0 0x7f69e8156010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37226/0 (socket says 172.21.15.46:37226) 2023-10-30T13:54:52.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.174+0000 7f69e77fe640 1 -- 172.21.15.46:0/2260292087 learned_addr learned my addr 172.21.15.46:0/2260292087 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:52.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69e6ffd640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e8071110 0x7f69e814e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:52.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69e7fff640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e80c6bd0 0x7f69e814e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:52.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69e77fe640 1 -- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e8071110 msgr2=0x7f69e814e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:52.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69e77fe640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e8071110 0x7f69e814e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:52.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69e77fe640 1 -- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e80c6bd0 msgr2=0x7f69e814e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:52.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69e77fe640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e80c6bd0 0x7f69e814e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:52.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69e77fe640 1 -- 172.21.15.46:0/2260292087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d80094d0 con 0x7f69e806f7a0 2023-10-30T13:54:52.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69e77fe640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e806f7a0 0x7f69e8156010 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f69d801d3a0 tx=0x7f69d801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:52.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69cffff640 1 -- 172.21.15.46:0/2260292087 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d80ba600 con 0x7f69e806f7a0 2023-10-30T13:54:52.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69cffff640 1 -- 172.21.15.46:0/2260292087 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d80ba760 con 0x7f69e806f7a0 2023-10-30T13:54:52.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69cffff640 1 -- 172.21.15.46:0/2260292087 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d80bab10 con 0x7f69e806f7a0 2023-10-30T13:54:52.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69edbaa640 1 -- 172.21.15.46:0/2260292087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e815aa90 con 0x7f69e806f7a0 2023-10-30T13:54:52.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69edbaa640 1 -- 172.21.15.46:0/2260292087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e815b050 con 0x7f69e806f7a0 2023-10-30T13:54:52.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69edbaa640 1 -- 172.21.15.46:0/2260292087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e804e170 con 0x7f69e806f7a0 2023-10-30T13:54:52.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69cffff640 1 -- 172.21.15.46:0/2260292087 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f69d80bac70 con 0x7f69e806f7a0 2023-10-30T13:54:52.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.178+0000 7f69cffff640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69d40664a0 0x7f69d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:52.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.182+0000 7f69cffff640 1 -- 172.21.15.46:0/2260292087 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f69d8122a40 con 0x7f69e806f7a0 2023-10-30T13:54:52.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.182+0000 7f69e6ffd640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69d40664a0 0x7f69d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.182+0000 7f69e6ffd640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69d40664a0 0x7f69d4068960 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f69e80c6900 tx=0x7f69dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.186+0000 7f69cffff640 1 -- 172.21.15.46:0/2260292087 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f69d80ef850 con 0x7f69e806f7a0 2023-10-30T13:54:52.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.374+0000 7f69edbaa640 1 -- 172.21.15.46:0/2260292087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69e80c8400 con 0x7f69e806f7a0 2023-10-30T13:54:52.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.374+0000 7f69cffff640 1 -- 172.21.15.46:0/2260292087 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f69d80b5020 con 0x7f69e806f7a0 2023-10-30T13:54:52.378 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:52.379 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:54:49.948093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:resolve","state_seq":21,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:52.379 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 140 2023-10-30T13:54:52.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.378+0000 7f69edbaa640 1 -- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69d40664a0 msgr2=0x7f69d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:52.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.378+0000 7f69edbaa640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69d40664a0 0x7f69d4068960 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f69e80c6900 tx=0x7f69dc008040 comp rx=0 tx=0).stop 2023-10-30T13:54:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.378+0000 7f69edbaa640 1 -- 172.21.15.46:0/2260292087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e806f7a0 msgr2=0x7f69e8156010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.378+0000 7f69edbaa640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e806f7a0 0x7f69e8156010 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f69d801d3a0 tx=0x7f69d801ba90 comp rx=0 tx=0).stop 2023-10-30T13:54:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.378+0000 7f69edbaa640 1 -- 172.21.15.46:0/2260292087 shutdown_connections 2023-10-30T13:54:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.378+0000 7f69edbaa640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69d40664a0 0x7f69d4068960 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.378+0000 7f69edbaa640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e80c6bd0 0x7f69e814e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.378+0000 7f69edbaa640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e8071110 0x7f69e814e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:52.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.378+0000 7f69edbaa640 1 --2- 172.21.15.46:0/2260292087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69e806f7a0 0x7f69e8156010 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:52.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.378+0000 7f69edbaa640 1 -- 172.21.15.46:0/2260292087 >> 172.21.15.46:0/2260292087 conn(0x7f69e80c7910 msgr2=0x7f69e80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:52.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.378+0000 7f69edbaa640 1 -- 172.21.15.46:0/2260292087 shutdown_connections 2023-10-30T13:54:52.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:52.378+0000 7f69edbaa640 1 -- 172.21.15.46:0/2260292087 wait complete. 2023-10-30T13:54:52.402 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 140, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 140, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7188": { "addr": "172.21.15.46:6836/711135361", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 711135361, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 711135361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7188, "incarnation": 132, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 22 }, "gid_7356": { "addr": "172.21.15.46:6837/3112266825", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3112266825, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3112266825, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7356, "incarnation": 139, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:resolve", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:54:49.948093+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7356, "mds_1": 7188 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:54:54.404 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.546+0000 7f7145002640 1 Processor -- start 2023-10-30T13:54:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.546+0000 7f7145002640 1 -- start start 2023-10-30T13:54:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f7145002640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71400c8cf0 0x7f71400c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f7145002640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71400ca340 0x7f714015d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f7145002640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714015ce30 con 0x7f71400c90d0 2023-10-30T13:54:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f7145002640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7140159880 con 0x7f71400c8cf0 2023-10-30T13:54:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f7145002640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7140159ae0 con 0x7f71400ca340 2023-10-30T13:54:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f713ed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71400c8cf0 0x7f71400c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f7137fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f71400c90d0 0x7f71400cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38930/0 (socket says 172.21.15.46:38930) 2023-10-30T13:54:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f7137fff640 1 -- 172.21.15.46:0/1936577649 learned_addr learned my addr 172.21.15.46:0/1936577649 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f713f59e640 1 --2- 172.21.15.46:0/1936577649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71400ca340 0x7f714015d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f713ed9d640 1 -- 172.21.15.46:0/1936577649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71400ca340 msgr2=0x7f714015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f713ed9d640 1 --2- 172.21.15.46:0/1936577649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71400ca340 0x7f714015d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:54.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f713ed9d640 1 -- 172.21.15.46:0/1936577649 >> v1:172.21.15.46:6789/0 conn(0x7f71400c90d0 legacy=0x7f71400cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:54.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f713ed9d640 1 -- 172.21.15.46:0/1936577649 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7140159dc0 con 0x7f71400c8cf0 2023-10-30T13:54:54.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.550+0000 7f713e59c640 1 -- 172.21.15.46:0/1936577649 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121070651 0 0) 0x7f714015ce30 con 0x7f71400c90d0 2023-10-30T13:54:54.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f713ed9d640 1 --2- 172.21.15.46:0/1936577649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71400c8cf0 0x7f71400c9e00 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f71300097c0 tx=0x7f713001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252e1125d217cec6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f713e59c640 1 -- 172.21.15.46:0/1936577649 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71300ad660 con 0x7f71400c8cf0 2023-10-30T13:54:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f713e59c640 1 -- 172.21.15.46:0/1936577649 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71300ad7c0 con 0x7f71400c8cf0 2023-10-30T13:54:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f713e59c640 1 -- 172.21.15.46:0/1936577649 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71300adb50 con 0x7f71400c8cf0 2023-10-30T13:54:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 -- 172.21.15.46:0/1936577649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71400c8cf0 msgr2=0x7f71400c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 --2- 172.21.15.46:0/1936577649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71400c8cf0 0x7f71400c9e00 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f71300097c0 tx=0x7f713001c450 comp rx=0 tx=0).stop 2023-10-30T13:54:54.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 -- 172.21.15.46:0/1936577649 shutdown_connections 2023-10-30T13:54:54.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 --2- 172.21.15.46:0/1936577649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71400ca340 0x7f714015d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:54.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 --2- 172.21.15.46:0/1936577649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71400c8cf0 0x7f71400c9e00 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:54.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 -- 172.21.15.46:0/1936577649 >> 172.21.15.46:0/1936577649 conn(0x7f714006be40 msgr2=0x7f714006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:54.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 -- 172.21.15.46:0/1936577649 shutdown_connections 2023-10-30T13:54:54.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 -- 172.21.15.46:0/1936577649 wait complete. 2023-10-30T13:54:54.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 Processor -- start 2023-10-30T13:54:54.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 -- start start 2023-10-30T13:54:54.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71400c8cf0 0x7f7140150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:54.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71400c90d0 0x7f71401509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71400ca340 0x7f7140154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7140156700 con 0x7f71400c90d0 2023-10-30T13:54:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71401541f0 con 0x7f71400c8cf0 2023-10-30T13:54:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.554+0000 7f7145002640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71401547b0 con 0x7f71400ca340 2023-10-30T13:54:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f7137fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71400c90d0 0x7f71401509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f7137fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71400c90d0 0x7f71401509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37252/0 (socket says 172.21.15.46:37252) 2023-10-30T13:54:54.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f7137fff640 1 -- 172.21.15.46:0/3972566568 learned_addr learned my addr 172.21.15.46:0/3972566568 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:54.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f713f59e640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71400ca340 0x7f7140154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:54.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f713ed9d640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71400c8cf0 0x7f7140150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:54.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f7137fff640 1 -- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71400ca340 msgr2=0x7f7140154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:54.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f7137fff640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71400ca340 0x7f7140154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:54.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f7137fff640 1 -- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71400c8cf0 msgr2=0x7f7140150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:54.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f7137fff640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71400c8cf0 0x7f7140150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:54.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f7137fff640 1 -- 172.21.15.46:0/3972566568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7130009420 con 0x7f71400c90d0 2023-10-30T13:54:54.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f713f59e640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71400ca340 0x7f7140154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:54.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f713ed9d640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71400c8cf0 0x7f7140150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:54.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f7137fff640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71400c90d0 0x7f71401509a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f71280011f0 tx=0x7f712800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:54.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f71377fe640 1 -- 172.21.15.46:0/3972566568 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71280074e0 con 0x7f71400c90d0 2023-10-30T13:54:54.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f71377fe640 1 -- 172.21.15.46:0/3972566568 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7128007640 con 0x7f71400c90d0 2023-10-30T13:54:54.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f7145002640 1 -- 172.21.15.46:0/3972566568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71401573c0 con 0x7f71400c90d0 2023-10-30T13:54:54.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f71377fe640 1 -- 172.21.15.46:0/3972566568 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7128004a10 con 0x7f71400c90d0 2023-10-30T13:54:54.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f7145002640 1 -- 172.21.15.46:0/3972566568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7140156bf0 con 0x7f71400c90d0 2023-10-30T13:54:54.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f7145002640 1 -- 172.21.15.46:0/3972566568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7140154960 con 0x7f71400c90d0 2023-10-30T13:54:54.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.558+0000 7f71377fe640 1 -- 172.21.15.46:0/3972566568 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7128004b70 con 0x7f71400c90d0 2023-10-30T13:54:54.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.562+0000 7f71377fe640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f71200664a0 0x7f7120068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:54.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.562+0000 7f71377fe640 1 -- 172.21.15.46:0/3972566568 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f7128083780 con 0x7f71400c90d0 2023-10-30T13:54:54.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.562+0000 7f713ed9d640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f71200664a0 0x7f7120068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:54.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.562+0000 7f713ed9d640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f71200664a0 0x7f7120068960 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f71300af000 tx=0x7f71300a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:54.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.566+0000 7f71377fe640 1 -- 172.21.15.46:0/3972566568 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7140154960 con 0x7f71400c90d0 2023-10-30T13:54:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.754+0000 7f7145002640 1 -- 172.21.15.46:0/3972566568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f714004e170 con 0x7f71400c90d0 2023-10-30T13:54:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.754+0000 7f71377fe640 1 -- 172.21.15.46:0/3972566568 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f7128018090 con 0x7f71400c90d0 2023-10-30T13:54:54.759 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:54.759 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:54:49.948093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:resolve","state_seq":21,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 140 2023-10-30T13:54:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.758+0000 7f7145002640 1 -- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f71200664a0 msgr2=0x7f7120068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.758+0000 7f7145002640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f71200664a0 0x7f7120068960 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f71300af000 tx=0x7f71300a7000 comp rx=0 tx=0).stop 2023-10-30T13:54:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.758+0000 7f7145002640 1 -- 172.21.15.46:0/3972566568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71400c90d0 msgr2=0x7f71401509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.758+0000 7f7145002640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71400c90d0 0x7f71401509a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f71280011f0 tx=0x7f712800c9b0 comp rx=0 tx=0).stop 2023-10-30T13:54:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.758+0000 7f7145002640 1 -- 172.21.15.46:0/3972566568 shutdown_connections 2023-10-30T13:54:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.758+0000 7f7145002640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f71200664a0 0x7f7120068960 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.758+0000 7f7145002640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71400ca340 0x7f7140154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.758+0000 7f7145002640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71400c90d0 0x7f71401509a0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.758+0000 7f7145002640 1 --2- 172.21.15.46:0/3972566568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71400c8cf0 0x7f7140150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.758+0000 7f7145002640 1 -- 172.21.15.46:0/3972566568 >> 172.21.15.46:0/3972566568 conn(0x7f714006be40 msgr2=0x7f71400cba90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.758+0000 7f7145002640 1 -- 172.21.15.46:0/3972566568 shutdown_connections 2023-10-30T13:54:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:54.758+0000 7f7145002640 1 -- 172.21.15.46:0/3972566568 wait complete. 2023-10-30T13:54:54.780 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:54:56.780 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:56.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e8ac09640 1 Processor -- start 2023-10-30T13:54:56.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e8ac09640 1 -- start start 2023-10-30T13:54:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e8ac09640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4e8406f7a0 0x7f4e8406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:56.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e8ac09640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4e84071110 0x7f4e84158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:56.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e8ac09640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e84066cf0 con 0x7f4e840c6bd0 2023-10-30T13:54:56.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e8ac09640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e840700c0 con 0x7f4e84071110 2023-10-30T13:54:56.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e8ac09640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e840c70d0 con 0x7f4e8406f7a0 2023-10-30T13:54:56.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e891a6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4e84071110 0x7f4e84158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:56.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e891a6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4e84071110 0x7f4e84158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44684/0 (socket says 172.21.15.46:44684) 2023-10-30T13:54:56.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e891a6640 1 -- 172.21.15.46:0/3108554766 learned_addr learned my addr 172.21.15.46:0/3108554766 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:56.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e889a5640 1 --2- 172.21.15.46:0/3108554766 >> v2:172.21.15.154:3300/0 conn(0x7f4e8406f7a0 0x7f4e8406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:56.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e891a6640 1 -- 172.21.15.46:0/3108554766 >> v2:172.21.15.154:3300/0 conn(0x7f4e8406f7a0 msgr2=0x7f4e8406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:56.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e891a6640 1 --2- 172.21.15.46:0/3108554766 >> v2:172.21.15.154:3300/0 conn(0x7f4e8406f7a0 0x7f4e8406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:56.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e891a6640 1 -- 172.21.15.46:0/3108554766 >> v1:172.21.15.154:6789/0 conn(0x7f4e840c6bd0 legacy=0x7f4e84069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:56.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.934+0000 7f4e891a6640 1 -- 172.21.15.46:0/3108554766 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e840c7390 con 0x7f4e84071110 2023-10-30T13:54:56.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.938+0000 7f4e891a6640 1 --2- 172.21.15.46:0/3108554766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e84071110 0x7f4e84158110 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f4e8406c830 tx=0x7f4e7c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c025fde0fc0179e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:56.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.938+0000 7f4e837fe640 1 -- 172.21.15.46:0/3108554766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c0af400 con 0x7f4e84071110 2023-10-30T13:54:56.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.938+0000 7f4e837fe640 1 -- 172.21.15.46:0/3108554766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7c0ab050 con 0x7f4e84071110 2023-10-30T13:54:56.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.938+0000 7f4e837fe640 1 -- 172.21.15.46:0/3108554766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c0af780 con 0x7f4e84071110 2023-10-30T13:54:56.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.938+0000 7f4e8ac09640 1 -- 172.21.15.46:0/3108554766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e84071110 msgr2=0x7f4e84158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.938+0000 7f4e8ac09640 1 --2- 172.21.15.46:0/3108554766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e84071110 0x7f4e84158110 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f4e8406c830 tx=0x7f4e7c01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:54:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.938+0000 7f4e8ac09640 1 -- 172.21.15.46:0/3108554766 shutdown_connections 2023-10-30T13:54:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.938+0000 7f4e8ac09640 1 --2- 172.21.15.46:0/3108554766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e84071110 0x7f4e84158110 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.938+0000 7f4e8ac09640 1 --2- 172.21.15.46:0/3108554766 >> v2:172.21.15.154:3300/0 conn(0x7f4e8406f7a0 0x7f4e8406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.938+0000 7f4e8ac09640 1 -- 172.21.15.46:0/3108554766 >> 172.21.15.46:0/3108554766 conn(0x7f4e840c7910 msgr2=0x7f4e840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:56.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.938+0000 7f4e8ac09640 1 -- 172.21.15.46:0/3108554766 shutdown_connections 2023-10-30T13:54:56.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.938+0000 7f4e8ac09640 1 -- 172.21.15.46:0/3108554766 wait complete. 2023-10-30T13:54:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e8ac09640 1 Processor -- start 2023-10-30T13:54:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e8ac09640 1 -- start start 2023-10-30T13:54:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e8ac09640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e8406f7a0 0x7f4e841f0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:56.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e8ac09640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e84071110 0x7f4e841ed850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:56.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e8ac09640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e840c6bd0 0x7f4e841491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:56.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e8ac09640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e841df390 con 0x7f4e84071110 2023-10-30T13:54:56.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e8ac09640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e841ec3e0 con 0x7f4e8406f7a0 2023-10-30T13:54:56.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e8ac09640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e841e4720 con 0x7f4e840c6bd0 2023-10-30T13:54:56.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e83fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e84071110 0x7f4e841ed850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:56.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e83fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e84071110 0x7f4e841ed850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44688/0 (socket says 172.21.15.46:44688) 2023-10-30T13:54:56.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e83fff640 1 -- 172.21.15.46:0/4178705992 learned_addr learned my addr 172.21.15.46:0/4178705992 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:56.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e889a5640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e8406f7a0 0x7f4e841f0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:56.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e891a6640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e840c6bd0 0x7f4e841491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:56.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e83fff640 1 -- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e840c6bd0 msgr2=0x7f4e841491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:56.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e83fff640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e840c6bd0 0x7f4e841491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:56.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e83fff640 1 -- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e8406f7a0 msgr2=0x7f4e841f0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:56.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e83fff640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e8406f7a0 0x7f4e841f0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:56.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e83fff640 1 -- 172.21.15.46:0/4178705992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c009b30 con 0x7f4e84071110 2023-10-30T13:54:56.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e83fff640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e84071110 0x7f4e841ed850 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000c500 tx=0x7f4e7000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:56.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e817fa640 1 -- 172.21.15.46:0/4178705992 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e700052d0 con 0x7f4e84071110 2023-10-30T13:54:56.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e817fa640 1 -- 172.21.15.46:0/4178705992 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e70005430 con 0x7f4e84071110 2023-10-30T13:54:56.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e8ac09640 1 -- 172.21.15.46:0/4178705992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e84155150 con 0x7f4e84071110 2023-10-30T13:54:56.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e817fa640 1 -- 172.21.15.46:0/4178705992 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70012830 con 0x7f4e84071110 2023-10-30T13:54:56.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.942+0000 7f4e8ac09640 1 -- 172.21.15.46:0/4178705992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e841f08a0 con 0x7f4e84071110 2023-10-30T13:54:56.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.946+0000 7f4e8ac09640 1 -- 172.21.15.46:0/4178705992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8404e170 con 0x7f4e84071110 2023-10-30T13:54:56.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.946+0000 7f4e817fa640 1 -- 172.21.15.46:0/4178705992 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e70012990 con 0x7f4e84071110 2023-10-30T13:54:56.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.946+0000 7f4e817fa640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e680664a0 0x7f4e68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:56.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.946+0000 7f4e817fa640 1 -- 172.21.15.46:0/4178705992 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f4e70083d50 con 0x7f4e84071110 2023-10-30T13:54:56.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.946+0000 7f4e889a5640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e680664a0 0x7f4e68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:56.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.946+0000 7f4e889a5640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e680664a0 0x7f4e68068960 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4e78009ac0 tx=0x7f4e78009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:56.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:56.950+0000 7f4e817fa640 1 -- 172.21.15.46:0/4178705992 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e70050c00 con 0x7f4e84071110 2023-10-30T13:54:57.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.142+0000 7f4e8ac09640 1 -- 172.21.15.46:0/4178705992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e841f08a0 con 0x7f4e84071110 2023-10-30T13:54:57.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.142+0000 7f4e817fa640 1 -- 172.21.15.46:0/4178705992 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f4e841f08a0 con 0x7f4e84071110 2023-10-30T13:54:57.147 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:57.147 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:54:55.395162+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:57.147 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 141 2023-10-30T13:54:57.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.146+0000 7f4e8ac09640 1 -- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e680664a0 msgr2=0x7f4e68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:57.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.146+0000 7f4e8ac09640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e680664a0 0x7f4e68068960 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4e78009ac0 tx=0x7f4e78009190 comp rx=0 tx=0).stop 2023-10-30T13:54:57.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.146+0000 7f4e8ac09640 1 -- 172.21.15.46:0/4178705992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e84071110 msgr2=0x7f4e841ed850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:57.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.146+0000 7f4e8ac09640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e84071110 0x7f4e841ed850 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000c500 tx=0x7f4e7000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:54:57.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.146+0000 7f4e8ac09640 1 -- 172.21.15.46:0/4178705992 shutdown_connections 2023-10-30T13:54:57.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.146+0000 7f4e8ac09640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e680664a0 0x7f4e68068960 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:57.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.146+0000 7f4e8ac09640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e840c6bd0 0x7f4e841491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:57.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.146+0000 7f4e8ac09640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e84071110 0x7f4e841ed850 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:57.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.146+0000 7f4e8ac09640 1 --2- 172.21.15.46:0/4178705992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e8406f7a0 0x7f4e841f0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:57.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.146+0000 7f4e8ac09640 1 -- 172.21.15.46:0/4178705992 >> 172.21.15.46:0/4178705992 conn(0x7f4e840c7910 msgr2=0x7f4e840c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:57.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.146+0000 7f4e8ac09640 1 -- 172.21.15.46:0/4178705992 shutdown_connections 2023-10-30T13:54:57.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:57.146+0000 7f4e8ac09640 1 -- 172.21.15.46:0/4178705992 wait complete. 2023-10-30T13:54:57.168 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:54:59.169 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:54:59.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.322+0000 7f504970e640 1 Processor -- start 2023-10-30T13:54:59.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.322+0000 7f504970e640 1 -- start start 2023-10-30T13:54:59.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.326+0000 7f504970e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50440c5660 0x7f50441561a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:59.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.326+0000 7f504970e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5044155ce0 con 0x7f50440c18c0 2023-10-30T13:54:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.326+0000 7f504970e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50440c3f60 con 0x7f5044150130 2023-10-30T13:54:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.326+0000 7f504970e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5044150510 con 0x7f50440c5660 2023-10-30T13:54:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.326+0000 7f503bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f50440c18c0 0x7f5044069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55880/0 (socket says 172.21.15.46:55880) 2023-10-30T13:54:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.326+0000 7f503bfff640 1 -- 172.21.15.46:0/2467494825 learned_addr learned my addr 172.21.15.46:0/2467494825 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:54:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.326+0000 7f50437fe640 1 --2- 172.21.15.46:0/2467494825 >> v2:172.21.15.46:3300/0 conn(0x7f50440c5660 0x7f50441561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:59.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.326+0000 7f50437fe640 1 -- 172.21.15.46:0/2467494825 >> v1:172.21.15.154:6789/0 conn(0x7f5044150130 legacy=0x7f50440c20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:59.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.326+0000 7f50437fe640 1 -- 172.21.15.46:0/2467494825 >> v1:172.21.15.46:6789/0 conn(0x7f50440c18c0 legacy=0x7f5044069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:59.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.326+0000 7f50437fe640 1 -- 172.21.15.46:0/2467494825 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50440c4240 con 0x7f50440c5660 2023-10-30T13:54:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.326+0000 7f50437fe640 1 --2- 172.21.15.46:0/2467494825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50440c5660 0x7f50441561a0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f503c009d90 tx=0x7f503c01d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826477439a14554e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f50427fc640 1 -- 172.21.15.46:0/2467494825 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c0a6a50 con 0x7f50440c5660 2023-10-30T13:54:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f50427fc640 1 -- 172.21.15.46:0/2467494825 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c0a6bb0 con 0x7f50440c5660 2023-10-30T13:54:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f50427fc640 1 -- 172.21.15.46:0/2467494825 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c0ae440 con 0x7f50440c5660 2023-10-30T13:54:59.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 -- 172.21.15.46:0/2467494825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50440c5660 msgr2=0x7f50441561a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:59.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 --2- 172.21.15.46:0/2467494825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50440c5660 0x7f50441561a0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f503c009d90 tx=0x7f503c01d9d0 comp rx=0 tx=0).stop 2023-10-30T13:54:59.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 -- 172.21.15.46:0/2467494825 shutdown_connections 2023-10-30T13:54:59.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 --2- 172.21.15.46:0/2467494825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50440c5660 0x7f50441561a0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:59.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 -- 172.21.15.46:0/2467494825 >> 172.21.15.46:0/2467494825 conn(0x7f50440cace0 msgr2=0x7f50440cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:59.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 -- 172.21.15.46:0/2467494825 shutdown_connections 2023-10-30T13:54:59.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 -- 172.21.15.46:0/2467494825 wait complete. 2023-10-30T13:54:59.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 Processor -- start 2023-10-30T13:54:59.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 -- start start 2023-10-30T13:54:59.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50440c18c0 0x7f50441f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:59.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50440c5660 0x7f50440c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:59.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5044150130 0x7f50440aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:59.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504414da50 con 0x7f5044150130 2023-10-30T13:54:59.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50441f27a0 con 0x7f50440c5660 2023-10-30T13:54:59.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.330+0000 7f504970e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50441f2a60 con 0x7f50440c18c0 2023-10-30T13:54:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f50437fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5044150130 0x7f50440aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f50437fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5044150130 0x7f50440aff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44718/0 (socket says 172.21.15.46:44718) 2023-10-30T13:54:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f50437fe640 1 -- 172.21.15.46:0/1985611270 learned_addr learned my addr 172.21.15.46:0/1985611270 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:54:59.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f5042ffd640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50440c18c0 0x7f50441f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f503bfff640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50440c5660 0x7f50440c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f50437fe640 1 -- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50440c18c0 msgr2=0x7f50441f21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f50437fe640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50440c18c0 0x7f50441f21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:59.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f50437fe640 1 -- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50440c5660 msgr2=0x7f50440c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f50437fe640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50440c5660 0x7f50440c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f50437fe640 1 -- 172.21.15.46:0/1985611270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c009a70 con 0x7f5044150130 2023-10-30T13:54:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f50437fe640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5044150130 0x7f50440aff20 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f503c01b600 tx=0x7f503c01c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:59.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f503b7fe640 1 -- 172.21.15.46:0/1985611270 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c0aed10 con 0x7f5044150130 2023-10-30T13:54:59.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f503b7fe640 1 -- 172.21.15.46:0/1985611270 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c0aee70 con 0x7f5044150130 2023-10-30T13:54:59.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f504970e640 1 -- 172.21.15.46:0/1985611270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50441f2d40 con 0x7f5044150130 2023-10-30T13:54:59.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f503b7fe640 1 -- 172.21.15.46:0/1985611270 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c0b9670 con 0x7f5044150130 2023-10-30T13:54:59.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f504970e640 1 -- 172.21.15.46:0/1985611270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504414d400 con 0x7f5044150130 2023-10-30T13:54:59.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f504970e640 1 -- 172.21.15.46:0/1985611270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f504414dbc0 con 0x7f5044150130 2023-10-30T13:54:59.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f503b7fe640 1 -- 172.21.15.46:0/1985611270 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f503c0b5070 con 0x7f5044150130 2023-10-30T13:54:59.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.334+0000 7f503b7fe640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50240664a0 0x7f5024068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:59.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.338+0000 7f503b7fe640 1 -- 172.21.15.46:0/1985611270 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f503c11f680 con 0x7f5044150130 2023-10-30T13:54:59.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.338+0000 7f5042ffd640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50240664a0 0x7f5024068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:59.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.338+0000 7f5042ffd640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50240664a0 0x7f5024068960 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5034007fd0 tx=0x7f5034009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:59.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.338+0000 7f503b7fe640 1 -- 172.21.15.46:0/1985611270 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f503c0ec640 con 0x7f5044150130 2023-10-30T13:54:59.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 -- 172.21.15.46:0/1985611270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f504404e170 con 0x7f5044150130 2023-10-30T13:54:59.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f503b7fe640 1 -- 172.21.15.46:0/1985611270 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f503c0efc60 con 0x7f5044150130 2023-10-30T13:54:59.532 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:54:59.533 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:54:55.395162+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:54:59.533 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 141 2023-10-30T13:54:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 -- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50240664a0 msgr2=0x7f5024068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50240664a0 0x7f5024068960 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5034007fd0 tx=0x7f5034009040 comp rx=0 tx=0).stop 2023-10-30T13:54:59.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 -- 172.21.15.46:0/1985611270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5044150130 msgr2=0x7f50440aff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:59.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5044150130 0x7f50440aff20 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f503c01b600 tx=0x7f503c01c210 comp rx=0 tx=0).stop 2023-10-30T13:54:59.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 -- 172.21.15.46:0/1985611270 shutdown_connections 2023-10-30T13:54:59.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50240664a0 0x7f5024068960 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:59.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5044150130 0x7f50440aff20 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50440c5660 0x7f50440c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 --2- 172.21.15.46:0/1985611270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50440c18c0 0x7f50441f21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 -- 172.21.15.46:0/1985611270 >> 172.21.15.46:0/1985611270 conn(0x7f50440cace0 msgr2=0x7f50440c0830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 -- 172.21.15.46:0/1985611270 shutdown_connections 2023-10-30T13:54:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:54:59.530+0000 7f504970e640 1 -- 172.21.15.46:0/1985611270 wait complete. 2023-10-30T13:54:59.553 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:55:01.554 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.706+0000 7f5f1e2dc640 1 Processor -- start 2023-10-30T13:55:01.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.706+0000 7f5f1e2dc640 1 -- start start 2023-10-30T13:55:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.706+0000 7f5f1e2dc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f181538f0 0x7f5f18159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:01.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.706+0000 7f5f1e2dc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f180ca6a0 con 0x7f5f180cefa0 2023-10-30T13:55:01.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.706+0000 7f5f1e2dc640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1815d300 con 0x7f5f1815cd50 2023-10-30T13:55:01.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.706+0000 7f5f1e2dc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1815d5e0 con 0x7f5f181538f0 2023-10-30T13:55:01.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.706+0000 7f5f1dadb640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5f180cefa0 0x7f5f181640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55884/0 (socket says 172.21.15.46:55884) 2023-10-30T13:55:01.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.706+0000 7f5f1dadb640 1 -- 172.21.15.46:0/1000588624 learned_addr learned my addr 172.21.15.46:0/1000588624 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:55:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.706+0000 7f5f1d2da640 1 --2- 172.21.15.46:0/1000588624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f181538f0 0x7f5f18159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692691115 0 0) 0x7f5f180ca6a0 con 0x7f5f180cefa0 2023-10-30T13:55:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f04003530 con 0x7f5f180cefa0 2023-10-30T13:55:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791846370 0 0) 0x7f5f1815d300 con 0x7f5f1815cd50 2023-10-30T13:55:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f180ca6a0 con 0x7f5f1815cd50 2023-10-30T13:55:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2715251682 0 0) 0x7f5f04003530 con 0x7f5f180cefa0 2023-10-30T13:55:01.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f181538f0 msgr2=0x7f5f18159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 --2- 172.21.15.46:0/1000588624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f181538f0 0x7f5f18159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 >> v1:172.21.15.154:6789/0 conn(0x7f5f1815cd50 legacy=0x7f5f180c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1815d8c0 con 0x7f5f180cefa0 2023-10-30T13:55:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f5f10001980 con 0x7f5f180cefa0 2023-10-30T13:55:01.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 >> v1:172.21.15.46:6789/0 conn(0x7f5f180cefa0 legacy=0x7f5f181640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 --2- 172.21.15.46:0/1000588624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f181538f0 0x7f5f04003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 --2- 172.21.15.46:0/1000588624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040040f0 0x7f5f040047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 --2- 172.21.15.46:0/1000588624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f04004d10 0x7f5f04005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1815d5e0 con 0x7f5f040040f0 2023-10-30T13:55:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0400bad0 con 0x7f5f04004d10 2023-10-30T13:55:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0400bc90 con 0x7f5f181538f0 2023-10-30T13:55:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f1cad9640 1 --2- 172.21.15.46:0/1000588624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040040f0 0x7f5f040047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f1d2da640 1 --2- 172.21.15.46:0/1000588624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f181538f0 0x7f5f04003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f1dadb640 1 --2- 172.21.15.46:0/1000588624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f04004d10 0x7f5f04005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f1cad9640 1 -- 172.21.15.46:0/1000588624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f181538f0 msgr2=0x7f5f04003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f1cad9640 1 --2- 172.21.15.46:0/1000588624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f181538f0 0x7f5f04003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f1cad9640 1 -- 172.21.15.46:0/1000588624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f04004d10 msgr2=0x7f5f04005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f1cad9640 1 --2- 172.21.15.46:0/1000588624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f04004d10 0x7f5f04005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.710+0000 7f5f1cad9640 1 -- 172.21.15.46:0/1000588624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f04003530 con 0x7f5f040040f0 2023-10-30T13:55:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1cad9640 1 --2- 172.21.15.46:0/1000588624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040040f0 0x7f5f040047d0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f5f080018d0 tx=0x7f5f0801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3bd08a010111365 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f080052d0 con 0x7f5f040040f0 2023-10-30T13:55:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f08005450 con 0x7f5f040040f0 2023-10-30T13:55:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f0ffff640 1 -- 172.21.15.46:0/1000588624 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f08003840 con 0x7f5f040040f0 2023-10-30T13:55:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1000588624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040040f0 msgr2=0x7f5f040047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 --2- 172.21.15.46:0/1000588624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f040040f0 0x7f5f040047d0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f5f080018d0 tx=0x7f5f0801c4a0 comp rx=0 tx=0).stop 2023-10-30T13:55:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1dadb640 1 -- 172.21.15.46:0/1000588624 reap_dead start 2023-10-30T13:55:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1000588624 shutdown_connections 2023-10-30T13:55:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1000588624 >> 172.21.15.46:0/1000588624 conn(0x7f5f1806a640 msgr2=0x7f5f18065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1000588624 shutdown_connections 2023-10-30T13:55:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1000588624 wait complete. 2023-10-30T13:55:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 Processor -- start 2023-10-30T13:55:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 -- start start 2023-10-30T13:55:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f040040f0 0x7f5f181528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f18151030 0x7f5f181502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f1814e9f0 0x7f5f181ef430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1814de30 con 0x7f5f18151030 2023-10-30T13:55:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f181f3900 con 0x7f5f1814e9f0 2023-10-30T13:55:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1e2dc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f18152ec0 con 0x7f5f040040f0 2023-10-30T13:55:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1cad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f18151030 0x7f5f181502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1cad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f18151030 0x7f5f181502b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44736/0 (socket says 172.21.15.46:44736) 2023-10-30T13:55:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.714+0000 7f5f1cad9640 1 -- 172.21.15.46:0/1861008549 learned_addr learned my addr 172.21.15.46:0/1861008549 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1d2da640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f040040f0 0x7f5f181528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1dadb640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f1814e9f0 0x7f5f181ef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1cad9640 1 -- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f040040f0 msgr2=0x7f5f181528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1cad9640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f040040f0 0x7f5f181528d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1cad9640 1 -- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f1814e9f0 msgr2=0x7f5f181ef430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1cad9640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f1814e9f0 0x7f5f181ef430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1cad9640 1 -- 172.21.15.46:0/1861008549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f181f2b40 con 0x7f5f18151030 2023-10-30T13:55:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1dadb640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f1814e9f0 0x7f5f181ef430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1d2da640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f040040f0 0x7f5f181528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1cad9640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f18151030 0x7f5f181502b0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f5f0801c2d0 tx=0x7f5f04005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f0dffb640 1 -- 172.21.15.46:0/1861008549 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f080052d0 con 0x7f5f18151030 2023-10-30T13:55:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f0dffb640 1 -- 172.21.15.46:0/1861008549 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f080ac430 con 0x7f5f18151030 2023-10-30T13:55:01.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1861008549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f181f2e20 con 0x7f5f18151030 2023-10-30T13:55:01.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f0dffb640 1 -- 172.21.15.46:0/1861008549 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f080ac7e0 con 0x7f5f18151030 2023-10-30T13:55:01.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1861008549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f181f33e0 con 0x7f5f18151030 2023-10-30T13:55:01.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1861008549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1804e170 con 0x7f5f18151030 2023-10-30T13:55:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f0dffb640 1 -- 172.21.15.46:0/1861008549 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5f080ac940 con 0x7f5f18151030 2023-10-30T13:55:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.718+0000 7f5f0dffb640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5efc0664d0 0x7f5efc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.722+0000 7f5f0dffb640 1 -- 172.21.15.46:0/1861008549 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f5f0811ec90 con 0x7f5f18151030 2023-10-30T13:55:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.722+0000 7f5f1d2da640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5efc0664d0 0x7f5efc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.722+0000 7f5f1d2da640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5efc0664d0 0x7f5efc068990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009a10 tx=0x7f5f14008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.722+0000 7f5f0dffb640 1 -- 172.21.15.46:0/1861008549 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5f080eba80 con 0x7f5f18151030 2023-10-30T13:55:01.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.914+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1861008549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f1814d750 con 0x7f5f18151030 2023-10-30T13:55:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.914+0000 7f5f0dffb640 1 -- 172.21.15.46:0/1861008549 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f5f080ef0a0 con 0x7f5f18151030 2023-10-30T13:55:01.918 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:01.918 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:01.329015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:01.919 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 142 2023-10-30T13:55:01.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.918+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5efc0664d0 msgr2=0x7f5efc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:01.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.918+0000 7f5f1e2dc640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5efc0664d0 0x7f5efc068990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009a10 tx=0x7f5f14008040 comp rx=0 tx=0).stop 2023-10-30T13:55:01.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.918+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1861008549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f18151030 msgr2=0x7f5f181502b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:01.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.918+0000 7f5f1e2dc640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f18151030 0x7f5f181502b0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f5f0801c2d0 tx=0x7f5f04005800 comp rx=0 tx=0).stop 2023-10-30T13:55:01.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.918+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1861008549 shutdown_connections 2023-10-30T13:55:01.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.918+0000 7f5f1e2dc640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5efc0664d0 0x7f5efc068990 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:01.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.918+0000 7f5f1e2dc640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f1814e9f0 0x7f5f181ef430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:01.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.918+0000 7f5f1e2dc640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f18151030 0x7f5f181502b0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:01.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.918+0000 7f5f1e2dc640 1 --2- 172.21.15.46:0/1861008549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f040040f0 0x7f5f181528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:01.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.918+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1861008549 >> 172.21.15.46:0/1861008549 conn(0x7f5f1806a640 msgr2=0x7f5f180cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:01.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.918+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1861008549 shutdown_connections 2023-10-30T13:55:01.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:01.918+0000 7f5f1e2dc640 1 -- 172.21.15.46:0/1861008549 wait complete. 2023-10-30T13:55:01.939 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:55:03.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:04.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.090+0000 7f634c93e640 1 Processor -- start 2023-10-30T13:55:04.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.090+0000 7f634c93e640 1 -- start start 2023-10-30T13:55:04.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.090+0000 7f634c93e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f63480c6b80 0x7f63480bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:04.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.090+0000 7f634c93e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63480c6f60 0x7f63480c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:04.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.090+0000 7f634c93e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63480c0a50 con 0x7f63480c8430 2023-10-30T13:55:04.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.090+0000 7f634c93e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f634806f6e0 con 0x7f63480c6b80 2023-10-30T13:55:04.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.090+0000 7f634c93e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634806f9c0 con 0x7f63480c6f60 2023-10-30T13:55:04.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.090+0000 7f634659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f63480c6b80 0x7f63480bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:04.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.090+0000 7f634659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f63480c6b80 0x7f63480bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44750/0 (socket says 172.21.15.46:44750) 2023-10-30T13:55:04.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.090+0000 7f634659c640 1 -- 172.21.15.46:0/2698411422 learned_addr learned my addr 172.21.15.46:0/2698411422 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:04.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.090+0000 7f633ffff640 1 --2- 172.21.15.46:0/2698411422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63480c6f60 0x7f63480c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:04.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f634659c640 1 -- 172.21.15.46:0/2698411422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63480c6f60 msgr2=0x7f63480c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:04.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f634659c640 1 --2- 172.21.15.46:0/2698411422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63480c6f60 0x7f63480c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:04.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f634659c640 1 -- 172.21.15.46:0/2698411422 >> v1:172.21.15.154:6789/0 conn(0x7f63480c8430 legacy=0x7f634813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:04.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f634659c640 1 -- 172.21.15.46:0/2698411422 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634806fca0 con 0x7f63480c6b80 2023-10-30T13:55:04.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f633ffff640 1 --2- 172.21.15.46:0/2698411422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63480c6f60 0x7f63480c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:04.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f634659c640 1 --2- 172.21.15.46:0/2698411422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63480c6b80 0x7f63480bfec0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f6338009870 tx=0x7f633801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec68b743ec108a4a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:04.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f6345d9b640 1 -- 172.21.15.46:0/2698411422 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63380ac620 con 0x7f63480c6b80 2023-10-30T13:55:04.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f6345d9b640 1 -- 172.21.15.46:0/2698411422 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63380ac780 con 0x7f63480c6b80 2023-10-30T13:55:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f6345d9b640 1 -- 172.21.15.46:0/2698411422 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63380acb10 con 0x7f63480c6b80 2023-10-30T13:55:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f634c93e640 1 -- 172.21.15.46:0/2698411422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63480c6b80 msgr2=0x7f63480bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f634c93e640 1 --2- 172.21.15.46:0/2698411422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63480c6b80 0x7f63480bfec0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f6338009870 tx=0x7f633801c4b0 comp rx=0 tx=0).stop 2023-10-30T13:55:04.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f634c93e640 1 -- 172.21.15.46:0/2698411422 shutdown_connections 2023-10-30T13:55:04.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f634c93e640 1 --2- 172.21.15.46:0/2698411422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63480c6f60 0x7f63480c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:04.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f634c93e640 1 --2- 172.21.15.46:0/2698411422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63480c6b80 0x7f63480bfec0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:04.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f634c93e640 1 -- 172.21.15.46:0/2698411422 >> 172.21.15.46:0/2698411422 conn(0x7f63480c7910 msgr2=0x7f63480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:04.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.094+0000 7f634c93e640 1 -- 172.21.15.46:0/2698411422 shutdown_connections 2023-10-30T13:55:04.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634c93e640 1 -- 172.21.15.46:0/2698411422 wait complete. 2023-10-30T13:55:04.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634c93e640 1 Processor -- start 2023-10-30T13:55:04.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634c93e640 1 -- start start 2023-10-30T13:55:04.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634c93e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63480c6b80 0x7f6348158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:04.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634c93e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63480c6f60 0x7f634814c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:04.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634c93e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63480c8430 0x7f63481588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:04.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634c93e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634814b220 con 0x7f63480c6b80 2023-10-30T13:55:04.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634c93e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63481f4360 con 0x7f63480c8430 2023-10-30T13:55:04.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634c93e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634814af60 con 0x7f63480c6f60 2023-10-30T13:55:04.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63480c6b80 0x7f6348158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:04.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63480c6b80 0x7f6348158170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44766/0 (socket says 172.21.15.46:44766) 2023-10-30T13:55:04.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634659c640 1 -- 172.21.15.46:0/2961411209 learned_addr learned my addr 172.21.15.46:0/2961411209 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:04.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f633ffff640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63480c6f60 0x7f634814c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:04.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f6346d9d640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63480c8430 0x7f63481588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:04.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634659c640 1 -- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63480c6f60 msgr2=0x7f634814c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:04.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634659c640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63480c6f60 0x7f634814c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:04.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634659c640 1 -- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63480c8430 msgr2=0x7f63481588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:04.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634659c640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63480c8430 0x7f63481588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:04.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634659c640 1 -- 172.21.15.46:0/2961411209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63380094d0 con 0x7f63480c6b80 2023-10-30T13:55:04.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634659c640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63480c6b80 0x7f6348158170 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f63380af040 tx=0x7f633801acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:04.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f633effd640 1 -- 172.21.15.46:0/2961411209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63380b7830 con 0x7f63480c6b80 2023-10-30T13:55:04.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f633effd640 1 -- 172.21.15.46:0/2961411209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63380b7990 con 0x7f63480c6b80 2023-10-30T13:55:04.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f633effd640 1 -- 172.21.15.46:0/2961411209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63380b7d40 con 0x7f63480c6b80 2023-10-30T13:55:04.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.098+0000 7f634c93e640 1 -- 172.21.15.46:0/2961411209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63481f35a0 con 0x7f63480c6b80 2023-10-30T13:55:04.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.102+0000 7f634c93e640 1 -- 172.21.15.46:0/2961411209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63481f3b60 con 0x7f63480c6b80 2023-10-30T13:55:04.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.102+0000 7f634c93e640 1 -- 172.21.15.46:0/2961411209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f634804e170 con 0x7f63480c6b80 2023-10-30T13:55:04.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.102+0000 7f633effd640 1 -- 172.21.15.46:0/2961411209 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f63380b3040 con 0x7f63480c6b80 2023-10-30T13:55:04.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.102+0000 7f633effd640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63280664d0 0x7f6328068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:04.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.102+0000 7f633effd640 1 -- 172.21.15.46:0/2961411209 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f633811ec30 con 0x7f63480c6b80 2023-10-30T13:55:04.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.102+0000 7f633ffff640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63280664d0 0x7f6328068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:04.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.106+0000 7f633ffff640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63280664d0 0x7f6328068990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f634814f6b0 tx=0x7f6330008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:04.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.106+0000 7f633effd640 1 -- 172.21.15.46:0/2961411209 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f63380eba20 con 0x7f63480c6b80 2023-10-30T13:55:04.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.298+0000 7f634c93e640 1 -- 172.21.15.46:0/2961411209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63480719e0 con 0x7f63480c6b80 2023-10-30T13:55:04.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.298+0000 7f633effd640 1 -- 172.21.15.46:0/2961411209 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f63380ac5e0 con 0x7f63480c6b80 2023-10-30T13:55:04.302 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:04.302 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:01.329015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 142 2023-10-30T13:55:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.302+0000 7f634c93e640 1 -- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63280664d0 msgr2=0x7f6328068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.302+0000 7f634c93e640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63280664d0 0x7f6328068990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f634814f6b0 tx=0x7f6330008040 comp rx=0 tx=0).stop 2023-10-30T13:55:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.302+0000 7f634c93e640 1 -- 172.21.15.46:0/2961411209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63480c6b80 msgr2=0x7f6348158170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.302+0000 7f634c93e640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63480c6b80 0x7f6348158170 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f63380af040 tx=0x7f633801acf0 comp rx=0 tx=0).stop 2023-10-30T13:55:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.302+0000 7f634c93e640 1 -- 172.21.15.46:0/2961411209 shutdown_connections 2023-10-30T13:55:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.302+0000 7f634c93e640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63280664d0 0x7f6328068990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.302+0000 7f634c93e640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63480c8430 0x7f63481588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.302+0000 7f634c93e640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63480c6f60 0x7f634814c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:04.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.302+0000 7f634c93e640 1 --2- 172.21.15.46:0/2961411209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63480c6b80 0x7f6348158170 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:04.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.302+0000 7f634c93e640 1 -- 172.21.15.46:0/2961411209 >> 172.21.15.46:0/2961411209 conn(0x7f63480c7910 msgr2=0x7f63480c19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:04.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.302+0000 7f634c93e640 1 -- 172.21.15.46:0/2961411209 shutdown_connections 2023-10-30T13:55:04.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:04.302+0000 7f634c93e640 1 -- 172.21.15.46:0/2961411209 wait complete. 2023-10-30T13:55:04.323 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 142, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 142, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7188": { "addr": "172.21.15.46:6836/711135361", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 711135361, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 711135361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7188, "incarnation": 132, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 22 }, "gid_7356": { "addr": "172.21.15.46:6837/3112266825", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3112266825, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3112266825, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7356, "incarnation": 139, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:55:01.329015+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7356, "mds_1": 7188 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:55:06.324 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:06.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.486+0000 7f55d2793640 1 Processor -- start 2023-10-30T13:55:06.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.486+0000 7f55d2793640 1 -- start start 2023-10-30T13:55:06.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d2793640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc06fef0 0x7f55cc161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d2793640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55cc161f60 con 0x7f55cc157c80 2023-10-30T13:55:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d2793640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55cc1616c0 con 0x7f55cc06ab20 2023-10-30T13:55:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d2793640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc15ef30 con 0x7f55cc06fef0 2023-10-30T13:55:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d1791640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f55cc157c80 0x7f55cc06f340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33320/0 (socket says 172.21.15.46:33320) 2023-10-30T13:55:06.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d1791640 1 -- 172.21.15.46:0/3297462257 learned_addr learned my addr 172.21.15.46:0/3297462257 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:55:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d0f90640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc06fef0 0x7f55cc161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057022687 0 0) 0x7f55cc161f60 con 0x7f55cc157c80 2023-10-30T13:55:06.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b8003530 con 0x7f55cc157c80 2023-10-30T13:55:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2782363597 0 0) 0x7f55b8003530 con 0x7f55cc157c80 2023-10-30T13:55:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc06fef0 msgr2=0x7f55cc161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc06fef0 0x7f55cc161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:06.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 >> v1:172.21.15.154:6789/0 conn(0x7f55cc06ab20 legacy=0x7f55cc070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc1619a0 con 0x7f55cc157c80 2023-10-30T13:55:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f55c8001800 con 0x7f55cc157c80 2023-10-30T13:55:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 >> v1:172.21.15.46:6789/0 conn(0x7f55cc157c80 legacy=0x7f55cc06f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55b80039b0 0x7f55b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc06fef0 0x7f55b80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55b8004d10 0x7f55b8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b800b990 con 0x7f55b80039b0 2023-10-30T13:55:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b800bad0 con 0x7f55b8004d10 2023-10-30T13:55:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b800bc90 con 0x7f55cc06fef0 2023-10-30T13:55:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d1791640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55b80039b0 0x7f55b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d0f90640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc06fef0 0x7f55b80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d1f92640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55b8004d10 0x7f55b8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d1791640 1 -- 172.21.15.46:0/3297462257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc06fef0 msgr2=0x7f55b80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d1791640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc06fef0 0x7f55b80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:06.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d1791640 1 -- 172.21.15.46:0/3297462257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55b8004d10 msgr2=0x7f55b8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:06.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d1791640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55b8004d10 0x7f55b8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:06.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d1791640 1 -- 172.21.15.46:0/3297462257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b8003530 con 0x7f55b80039b0 2023-10-30T13:55:06.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.490+0000 7f55d1f92640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55b8004d10 0x7f55b8005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:55:06.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.494+0000 7f55d1791640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55b80039b0 0x7f55b8003d90 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c1c0 tx=0x7f55c801cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37fd14e1b0d86c0f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:06.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.494+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c8007270 con 0x7f55b80039b0 2023-10-30T13:55:06.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.494+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c80073d0 con 0x7f55b80039b0 2023-10-30T13:55:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.494+0000 7f55c3fff640 1 -- 172.21.15.46:0/3297462257 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c80a5880 con 0x7f55b80039b0 2023-10-30T13:55:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.494+0000 7f55d2793640 1 -- 172.21.15.46:0/3297462257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55b80039b0 msgr2=0x7f55b8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.494+0000 7f55d2793640 1 --2- 172.21.15.46:0/3297462257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55b80039b0 0x7f55b8003d90 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c1c0 tx=0x7f55c801cb80 comp rx=0 tx=0).stop 2023-10-30T13:55:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.494+0000 7f55d1f92640 1 -- 172.21.15.46:0/3297462257 reap_dead start 2023-10-30T13:55:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.494+0000 7f55d2793640 1 -- 172.21.15.46:0/3297462257 shutdown_connections 2023-10-30T13:55:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.494+0000 7f55d2793640 1 -- 172.21.15.46:0/3297462257 >> 172.21.15.46:0/3297462257 conn(0x7f55cc0c75d0 msgr2=0x7f55cc0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.494+0000 7f55d2793640 1 -- 172.21.15.46:0/3297462257 shutdown_connections 2023-10-30T13:55:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.494+0000 7f55d2793640 1 -- 172.21.15.46:0/3297462257 wait complete. 2023-10-30T13:55:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d2793640 1 Processor -- start 2023-10-30T13:55:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d2793640 1 -- start start 2023-10-30T13:55:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d2793640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55b80039b0 0x7f55cc1f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d2793640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55cc156fd0 0x7f55cc15a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d2793640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc0ca580 0x7f55cc0c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d2793640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0c6ef0 con 0x7f55cc156fd0 2023-10-30T13:55:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d2793640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc15e920 con 0x7f55b80039b0 2023-10-30T13:55:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d2793640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc15e3a0 con 0x7f55cc0ca580 2023-10-30T13:55:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d0f90640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55cc156fd0 0x7f55cc15a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d0f90640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55cc156fd0 0x7f55cc15a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37084/0 (socket says 172.21.15.46:37084) 2023-10-30T13:55:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d0f90640 1 -- 172.21.15.46:0/127093665 learned_addr learned my addr 172.21.15.46:0/127093665 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d1f92640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc0ca580 0x7f55cc0c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:06.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d1791640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55b80039b0 0x7f55cc1f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d0f90640 1 -- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc0ca580 msgr2=0x7f55cc0c3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d0f90640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc0ca580 0x7f55cc0c3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d0f90640 1 -- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55b80039b0 msgr2=0x7f55cc1f6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d0f90640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55b80039b0 0x7f55cc1f6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d0f90640 1 -- 172.21.15.46:0/127093665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc159cf0 con 0x7f55cc156fd0 2023-10-30T13:55:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d1f92640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc0ca580 0x7f55cc0c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d1791640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55b80039b0 0x7f55cc1f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d0f90640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55cc156fd0 0x7f55cc15a4b0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f55bc011590 tx=0x7f55bc011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55c1ffb640 1 -- 172.21.15.46:0/127093665 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc004220 con 0x7f55cc156fd0 2023-10-30T13:55:06.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55c1ffb640 1 -- 172.21.15.46:0/127093665 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc004380 con 0x7f55cc156fd0 2023-10-30T13:55:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d2793640 1 -- 172.21.15.46:0/127093665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55cc15e640 con 0x7f55cc156fd0 2023-10-30T13:55:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55c1ffb640 1 -- 172.21.15.46:0/127093665 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc015870 con 0x7f55cc156fd0 2023-10-30T13:55:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.498+0000 7f55d2793640 1 -- 172.21.15.46:0/127093665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55cc0c5d90 con 0x7f55cc156fd0 2023-10-30T13:55:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.502+0000 7f55d2793640 1 -- 172.21.15.46:0/127093665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55cc0674d0 con 0x7f55cc156fd0 2023-10-30T13:55:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.502+0000 7f55c1ffb640 1 -- 172.21.15.46:0/127093665 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f55bc0159d0 con 0x7f55cc156fd0 2023-10-30T13:55:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.502+0000 7f55c1ffb640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f55b00664a0 0x7f55b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.502+0000 7f55c1ffb640 1 -- 172.21.15.46:0/127093665 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f55bc0846d0 con 0x7f55cc156fd0 2023-10-30T13:55:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.502+0000 7f55d1791640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f55b00664a0 0x7f55b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.502+0000 7f55d1791640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f55b00664a0 0x7f55b0068960 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f55c80ab000 tx=0x7f55c80a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:06.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.506+0000 7f55c1ffb640 1 -- 172.21.15.46:0/127093665 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f55bc051430 con 0x7f55cc156fd0 2023-10-30T13:55:06.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.694+0000 7f55d2793640 1 -- 172.21.15.46:0/127093665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55cc159b50 con 0x7f55cc156fd0 2023-10-30T13:55:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.694+0000 7f55c1ffb640 1 -- 172.21.15.46:0/127093665 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f55cc159b50 con 0x7f55cc156fd0 2023-10-30T13:55:06.698 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:06.699 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:01.329015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 142 2023-10-30T13:55:06.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.698+0000 7f55d2793640 1 -- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f55b00664a0 msgr2=0x7f55b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:06.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.698+0000 7f55d2793640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f55b00664a0 0x7f55b0068960 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f55c80ab000 tx=0x7f55c80a7070 comp rx=0 tx=0).stop 2023-10-30T13:55:06.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.698+0000 7f55d2793640 1 -- 172.21.15.46:0/127093665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55cc156fd0 msgr2=0x7f55cc15a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.698+0000 7f55d2793640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55cc156fd0 0x7f55cc15a4b0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f55bc011590 tx=0x7f55bc011a50 comp rx=0 tx=0).stop 2023-10-30T13:55:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.698+0000 7f55d2793640 1 -- 172.21.15.46:0/127093665 shutdown_connections 2023-10-30T13:55:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.698+0000 7f55d2793640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f55b00664a0 0x7f55b0068960 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.698+0000 7f55d2793640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55cc0ca580 0x7f55cc0c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.698+0000 7f55d2793640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55cc156fd0 0x7f55cc15a4b0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:06.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.698+0000 7f55d2793640 1 --2- 172.21.15.46:0/127093665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55b80039b0 0x7f55cc1f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:06.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.698+0000 7f55d2793640 1 -- 172.21.15.46:0/127093665 >> 172.21.15.46:0/127093665 conn(0x7f55cc0c75d0 msgr2=0x7f55cc06f680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:06.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.698+0000 7f55d2793640 1 -- 172.21.15.46:0/127093665 shutdown_connections 2023-10-30T13:55:06.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:06.698+0000 7f55d2793640 1 -- 172.21.15.46:0/127093665 wait complete. 2023-10-30T13:55:06.719 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:55:08.720 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:08.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb97a809640 1 Processor -- start 2023-10-30T13:55:08.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb97a809640 1 -- start start 2023-10-30T13:55:08.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb97a809640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9741538a0 0x7fb97415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:08.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb97a809640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb9740ca0e0 0x7fb97415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:08.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb97a809640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9740cb800 con 0x7fb9740cf160 2023-10-30T13:55:08.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb97a809640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97415dc30 con 0x7fb9740ca0e0 2023-10-30T13:55:08.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb97a809640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740cb330 con 0x7fb9741538a0 2023-10-30T13:55:08.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb973fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9741538a0 0x7fb97415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:08.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb973fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9741538a0 0x7fb97415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:57046/0 (socket says 172.21.15.46:57046) 2023-10-30T13:55:08.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb973fff640 1 -- 172.21.15.46:0/1936476248 learned_addr learned my addr 172.21.15.46:0/1936476248 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:08.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb9737fe640 1 --2- 172.21.15.46:0/1936476248 >> v2:172.21.15.154:3300/0 conn(0x7fb9740ca0e0 0x7fb97415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:08.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb973fff640 1 -- 172.21.15.46:0/1936476248 >> v2:172.21.15.154:3300/0 conn(0x7fb9740ca0e0 msgr2=0x7fb97415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:08.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb973fff640 1 --2- 172.21.15.46:0/1936476248 >> v2:172.21.15.154:3300/0 conn(0x7fb9740ca0e0 0x7fb97415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:08.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb973fff640 1 -- 172.21.15.46:0/1936476248 >> v1:172.21.15.154:6789/0 conn(0x7fb9740cf160 legacy=0x7fb9740cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:08.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb973fff640 1 -- 172.21.15.46:0/1936476248 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9740cad70 con 0x7fb9741538a0 2023-10-30T13:55:08.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.874+0000 7fb9737fe640 1 --2- 172.21.15.46:0/1936476248 >> v2:172.21.15.154:3300/0 conn(0x7fb9740ca0e0 0x7fb97415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:55:08.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.878+0000 7fb973fff640 1 --2- 172.21.15.46:0/1936476248 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9741538a0 0x7fb97415d6f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb964009870 tx=0x7fb96401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdbc14e1971185bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:08.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.878+0000 7fb972ffd640 1 -- 172.21.15.46:0/1936476248 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9640a8a30 con 0x7fb9741538a0 2023-10-30T13:55:08.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.878+0000 7fb972ffd640 1 -- 172.21.15.46:0/1936476248 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9640a8b90 con 0x7fb9741538a0 2023-10-30T13:55:08.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.878+0000 7fb972ffd640 1 -- 172.21.15.46:0/1936476248 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9640b0440 con 0x7fb9741538a0 2023-10-30T13:55:08.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.878+0000 7fb97a809640 1 -- 172.21.15.46:0/1936476248 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9741538a0 msgr2=0x7fb97415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:08.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.878+0000 7fb97a809640 1 --2- 172.21.15.46:0/1936476248 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9741538a0 0x7fb97415d6f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb964009870 tx=0x7fb96401ed30 comp rx=0 tx=0).stop 2023-10-30T13:55:08.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.878+0000 7fb97a809640 1 -- 172.21.15.46:0/1936476248 shutdown_connections 2023-10-30T13:55:08.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.878+0000 7fb97a809640 1 --2- 172.21.15.46:0/1936476248 >> v2:172.21.15.154:3300/0 conn(0x7fb9740ca0e0 0x7fb97415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:08.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.878+0000 7fb97a809640 1 --2- 172.21.15.46:0/1936476248 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9741538a0 0x7fb97415d6f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:08.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.878+0000 7fb97a809640 1 -- 172.21.15.46:0/1936476248 >> 172.21.15.46:0/1936476248 conn(0x7fb97406bfe0 msgr2=0x7fb97406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:08.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.878+0000 7fb97a809640 1 -- 172.21.15.46:0/1936476248 shutdown_connections 2023-10-30T13:55:08.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.878+0000 7fb97a809640 1 -- 172.21.15.46:0/1936476248 wait complete. 2023-10-30T13:55:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb97a809640 1 Processor -- start 2023-10-30T13:55:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb97a809640 1 -- start start 2023-10-30T13:55:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb97a809640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740ca0e0 0x7fb9740ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:08.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb97a809640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9740cf160 0x7fb9740b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb97a809640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9741538a0 0x7fb9740b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb97a809640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9741efcc0 con 0x7fb9740ca0e0 2023-10-30T13:55:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb97a809640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9741f01f0 con 0x7fb9741538a0 2023-10-30T13:55:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb97a809640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9741eff10 con 0x7fb9740cf160 2023-10-30T13:55:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb973fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740ca0e0 0x7fb9740ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb973fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740ca0e0 0x7fb9740ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37094/0 (socket says 172.21.15.46:37094) 2023-10-30T13:55:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb973fff640 1 -- 172.21.15.46:0/2203861760 learned_addr learned my addr 172.21.15.46:0/2203861760 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb9737fe640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9740cf160 0x7fb9740b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb978da6640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9741538a0 0x7fb9740b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb973fff640 1 -- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9740cf160 msgr2=0x7fb9740b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb973fff640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9740cf160 0x7fb9740b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb973fff640 1 -- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9741538a0 msgr2=0x7fb9740b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb973fff640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9741538a0 0x7fb9740b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb973fff640 1 -- 172.21.15.46:0/2203861760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb968009420 con 0x7fb9740ca0e0 2023-10-30T13:55:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb973fff640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740ca0e0 0x7fb9740ce260 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb9640b3040 tx=0x7fb96401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb970ff9640 1 -- 172.21.15.46:0/2203861760 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9640b9620 con 0x7fb9740ca0e0 2023-10-30T13:55:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb97a809640 1 -- 172.21.15.46:0/2203861760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9640094d0 con 0x7fb9740ca0e0 2023-10-30T13:55:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb970ff9640 1 -- 172.21.15.46:0/2203861760 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9640b97a0 con 0x7fb9740ca0e0 2023-10-30T13:55:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb970ff9640 1 -- 172.21.15.46:0/2203861760 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9640b9b50 con 0x7fb9740ca0e0 2023-10-30T13:55:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.882+0000 7fb97a809640 1 -- 172.21.15.46:0/2203861760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9740b35b0 con 0x7fb9740ca0e0 2023-10-30T13:55:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.886+0000 7fb97a809640 1 -- 172.21.15.46:0/2203861760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9740b2a50 con 0x7fb9740ca0e0 2023-10-30T13:55:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.886+0000 7fb970ff9640 1 -- 172.21.15.46:0/2203861760 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9640b7070 con 0x7fb9740ca0e0 2023-10-30T13:55:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.886+0000 7fb970ff9640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9600664d0 0x7fb960068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.886+0000 7fb970ff9640 1 -- 172.21.15.46:0/2203861760 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7fb964122200 con 0x7fb9740ca0e0 2023-10-30T13:55:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.886+0000 7fb9737fe640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9600664d0 0x7fb960068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:08.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.890+0000 7fb9737fe640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9600664d0 0x7fb960068990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb9741f3870 tx=0x7fb968009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:08.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:08.890+0000 7fb970ff9640 1 -- 172.21.15.46:0/2203861760 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9640ef0b0 con 0x7fb9740ca0e0 2023-10-30T13:55:09.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.078+0000 7fb97a809640 1 -- 172.21.15.46:0/2203861760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb97404e170 con 0x7fb9740ca0e0 2023-10-30T13:55:09.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.078+0000 7fb970ff9640 1 -- 172.21.15.46:0/2203861760 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fb9640f26d0 con 0x7fb9740ca0e0 2023-10-30T13:55:09.083 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:09.083 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:06.472633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:09.084 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 143 2023-10-30T13:55:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.082+0000 7fb97a809640 1 -- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9600664d0 msgr2=0x7fb960068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.082+0000 7fb97a809640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9600664d0 0x7fb960068990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb9741f3870 tx=0x7fb968009190 comp rx=0 tx=0).stop 2023-10-30T13:55:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.082+0000 7fb97a809640 1 -- 172.21.15.46:0/2203861760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740ca0e0 msgr2=0x7fb9740ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:09.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.082+0000 7fb97a809640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740ca0e0 0x7fb9740ce260 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb9640b3040 tx=0x7fb96401bad0 comp rx=0 tx=0).stop 2023-10-30T13:55:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.082+0000 7fb97a809640 1 -- 172.21.15.46:0/2203861760 shutdown_connections 2023-10-30T13:55:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.082+0000 7fb97a809640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9600664d0 0x7fb960068990 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.082+0000 7fb97a809640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9741538a0 0x7fb9740b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.082+0000 7fb97a809640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9740cf160 0x7fb9740b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:09.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.082+0000 7fb97a809640 1 --2- 172.21.15.46:0/2203861760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9740ca0e0 0x7fb9740ce260 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:09.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.082+0000 7fb97a809640 1 -- 172.21.15.46:0/2203861760 >> 172.21.15.46:0/2203861760 conn(0x7fb97406bfe0 msgr2=0x7fb974152870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:09.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.082+0000 7fb97a809640 1 -- 172.21.15.46:0/2203861760 shutdown_connections 2023-10-30T13:55:09.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:09.082+0000 7fb97a809640 1 -- 172.21.15.46:0/2203861760 wait complete. 2023-10-30T13:55:09.104 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=7002 2023-10-30T13:55:09.104 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.a 2023-10-30T13:55:16.334 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T13:55:16.330+0000 7f1fec58c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:55:16.334 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T13:55:16.330+0000 7f1fec58c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:55:16.414 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 2% (3/105) Updating files: 3% (4/105) Updating files: 4% (5/105) Updating files: 5% (6/105) Updating files: 6% (7/105) Updating files: 7% (8/105) Updating files: 8% (9/105) Updating files: 9% (10/105) Updating files: 10% (11/105) Updating files: 11% (12/105) Updating files: 12% (13/105) Updating files: 13% (14/105) Updating files: 14% (15/105) Updating files: 15% (16/105) Updating files: 16% (17/105) Updating files: 17% (18/105) Updating files: 18% (19/105) Updating files: 19% (20/105) Updating files: 20% (21/105) Updating files: 21% (23/105) Updating files: 22% (24/105) Updating files: 23% (25/105) Updating files: 24% (26/105) Updating files: 25% (27/105) Updating files: 26% (28/105) Updating files: 27% (29/105) Updating files: 28% (30/105) Updating files: 29% (31/105) Updating files: 30% (32/105) Updating files: 31% (33/105) Updating files: 32% (34/105) Updating files: 33% (35/105) Updating files: 34% (36/105) Updating files: 35% (37/105) Updating files: 36% (38/105) Updating files: 37% (39/105) Updating files: 38% (40/105) Updating files: 39% (41/105) Updating files: 40% (42/105) Updating files: 41% (44/105) Updating files: 42% (45/105) Updating files: 43% (46/105) Updating files: 44% (47/105) Updating files: 45% (48/105) Updating files: 46% (49/105) Updating files: 47% (50/105) Updating files: 48% (51/105) Updating files: 49% (52/105) Updating files: 50% (53/105) Updating files: 51% (54/105) Updating files: 52% (55/105) Updating files: 53% (56/105) Updating files: 54% (57/105) Updating files: 55% (58/105) Updating files: 56% (59/105) Updating files: 57% (60/105) Updating files: 58% (61/105) Updating files: 59% (62/105) Updating files: 60% (63/105) Updating files: 61% (65/105) Updating files: 62% (66/105) Updating files: 63% (67/105) Updating files: 64% (68/105) Updating files: 65% (69/105) Updating files: 66% (70/105) Updating files: 67% (71/105) Updating files: 68% (72/105) Updating files: 69% (73/105) Updating files: 70% (74/105) Updating files: 71% (75/105) Updating files: 72% (76/105) Updating files: 73% (77/105) Updating files: 74% (78/105) Updating files: 75% (79/105) Updating files: 76% (80/105) Updating files: 77% (81/105) Updating files: 78% (82/105) Updating files: 79% (83/105) Updating files: 80% (84/105) Updating files: 81% (86/105) Updating files: 82% (87/105) Updating files: 83% (88/105) Updating files: 84% (89/105) Updating files: 85% (90/105) Updating files: 86% (91/105) Updating files: 87% (92/105) Updating files: 88% (93/105) Updating files: 89% (94/105) Updating files: 90% (95/105) Updating files: 91% (96/105) Updating files: 92% (97/105) Updating files: 93% (98/105) Updating files: 94% (99/105) Updating files: 95% (100/105) Updating files: 96% (101/105) Updating files: 97% (102/105) Updating files: 98% (103/105) Updating files: 99% (104/105) Updating files: 100% (105/105) Updating files: 100% (105/105), done. 2023-10-30T13:55:17.259 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at c03ca95d235 v0.51 2023-10-30T13:55:17.262 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.51 2023-10-30T13:55:17.300 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 52 -eq 48 ] 2023-10-30T13:55:17.300 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.52 2023-10-30T13:55:17.300 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.52 2023-10-30T13:55:17.301 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.52 2023-10-30T13:55:17.301 INFO:tasks.workunit.client.0.smithi046.stdout:v0.52 2023-10-30T13:55:18.106 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:55:18.106 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:55:18.107 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:55:18.111 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:55:18.111 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:18.200 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-10-30T13:55:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.254+0000 7f04c21d3640 1 Processor -- start 2023-10-30T13:55:18.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.254+0000 7f04c21d3640 1 -- start start 2023-10-30T13:55:18.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.254+0000 7f04c21d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f04bc1538a0 0x7f04bc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:18.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.254+0000 7f04c21d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04bc0cf160 0x7f04bc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:18.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.258+0000 7f04c21d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04bc0cb800 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.258+0000 7f04c21d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04bc15dc30 con 0x7f04bc1538a0 2023-10-30T13:55:18.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.258+0000 7f04c21d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0cb330 con 0x7f04bc0cf160 2023-10-30T13:55:18.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.258+0000 7f04bb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f04bc1538a0 0x7f04bc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:18.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.258+0000 7f04bb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f04bc1538a0 0x7f04bc15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46392/0 (socket says 172.21.15.46:46392) 2023-10-30T13:55:18.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.258+0000 7f04bb7fe640 1 -- 172.21.15.46:0/2254965096 learned_addr learned my addr 172.21.15.46:0/2254965096 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:18.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.258+0000 7f04bbfff640 1 --2- 172.21.15.46:0/2254965096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04bc0cf160 0x7f04bc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:18.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.258+0000 7f04bb7fe640 1 -- 172.21.15.46:0/2254965096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04bc0cf160 msgr2=0x7f04bc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:18.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.258+0000 7f04bb7fe640 1 --2- 172.21.15.46:0/2254965096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04bc0cf160 0x7f04bc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:18.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.258+0000 7f04bb7fe640 1 -- 172.21.15.46:0/2254965096 >> v1:172.21.15.154:6789/0 conn(0x7f04bc0ca0e0 legacy=0x7f04bc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:18.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.258+0000 7f04bb7fe640 1 -- 172.21.15.46:0/2254965096 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc0cad70 con 0x7f04bc1538a0 2023-10-30T13:55:18.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.258+0000 7f04bb7fe640 1 --2- 172.21.15.46:0/2254965096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc1538a0 0x7f04bc15d6f0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009870 tx=0x7f04ac01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aaf3f15e3f2e892 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:18.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04ba7fc640 1 -- 172.21.15.46:0/2254965096 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac0a8a30 con 0x7f04bc1538a0 2023-10-30T13:55:18.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04ba7fc640 1 -- 172.21.15.46:0/2254965096 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac0a8b90 con 0x7f04bc1538a0 2023-10-30T13:55:18.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04ba7fc640 1 -- 172.21.15.46:0/2254965096 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac0b0440 con 0x7f04bc1538a0 2023-10-30T13:55:18.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 -- 172.21.15.46:0/2254965096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc1538a0 msgr2=0x7f04bc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:18.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 --2- 172.21.15.46:0/2254965096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc1538a0 0x7f04bc15d6f0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009870 tx=0x7f04ac01ed30 comp rx=0 tx=0).stop 2023-10-30T13:55:18.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 -- 172.21.15.46:0/2254965096 shutdown_connections 2023-10-30T13:55:18.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 --2- 172.21.15.46:0/2254965096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04bc0cf160 0x7f04bc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:18.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 --2- 172.21.15.46:0/2254965096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc1538a0 0x7f04bc15d6f0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:18.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 -- 172.21.15.46:0/2254965096 >> 172.21.15.46:0/2254965096 conn(0x7f04bc06bfe0 msgr2=0x7f04bc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:18.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 -- 172.21.15.46:0/2254965096 shutdown_connections 2023-10-30T13:55:18.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 -- 172.21.15.46:0/2254965096 wait complete. 2023-10-30T13:55:18.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 Processor -- start 2023-10-30T13:55:18.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 -- start start 2023-10-30T13:55:18.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc0ca0e0 0x7f04bc1e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:18.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04bc0cf160 0x7f04bc15b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:18.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04bc1538a0 0x7f04bc15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:18.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc14d4f0 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc1e4d40 con 0x7f04bc1538a0 2023-10-30T13:55:18.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04c21d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc15b2a0 con 0x7f04bc0cf160 2023-10-30T13:55:18.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04bb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc0ca0e0 0x7f04bc1e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:18.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04bb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc0ca0e0 0x7f04bc1e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46402/0 (socket says 172.21.15.46:46402) 2023-10-30T13:55:18.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.262+0000 7f04bb7fe640 1 -- 172.21.15.46:0/2937762147 learned_addr learned my addr 172.21.15.46:0/2937762147 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:18.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04bbfff640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04bc1538a0 0x7f04bc15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:18.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04baffd640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04bc0cf160 0x7f04bc15b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:18.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04bb7fe640 1 -- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04bc0cf160 msgr2=0x7f04bc15b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:18.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04bb7fe640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04bc0cf160 0x7f04bc15b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:18.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04bb7fe640 1 -- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04bc1538a0 msgr2=0x7f04bc15bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:18.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04bb7fe640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04bc1538a0 0x7f04bc15bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:18.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04bb7fe640 1 -- 172.21.15.46:0/2937762147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac0094d0 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04bb7fe640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc0ca0e0 0x7f04bc1e4090 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0b3040 tx=0x7f04ac01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:18.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04a3fff640 1 -- 172.21.15.46:0/2937762147 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac0b0dd0 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04a3fff640 1 -- 172.21.15.46:0/2937762147 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac0bb460 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04a3fff640 1 -- 172.21.15.46:0/2937762147 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac0bb810 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04c21d3640 1 -- 172.21.15.46:0/2937762147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04bc0cef10 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04c21d3640 1 -- 172.21.15.46:0/2937762147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04bc0c50b0 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04c21d3640 1 -- 172.21.15.46:0/2937762147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04bc0b2a50 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04a3fff640 1 -- 172.21.15.46:0/2937762147 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f04ac0b83f0 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.266+0000 7f04a3fff640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04a80664a0 0x7f04a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:18.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.270+0000 7f04a3fff640 1 -- 172.21.15.46:0/2937762147 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f04ac121cb0 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.270+0000 7f04baffd640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04a80664a0 0x7f04a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:18.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.270+0000 7f04baffd640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04a80664a0 0x7f04a8068960 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f04bc0c4660 tx=0x7f04b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:18.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.274+0000 7f04a3fff640 1 -- 172.21.15.46:0/2937762147 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f04ac0eeb40 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.506+0000 7f04c21d3640 1 -- 172.21.15.46:0/2937762147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04bc151940 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.506+0000 7f04a3fff640 1 -- 172.21.15.46:0/2937762147 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f04ac0b5090 con 0x7f04bc0ca0e0 2023-10-30T13:55:18.511 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:18.511 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7518,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:17.419200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:18.511 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 144 2023-10-30T13:55:18.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.510+0000 7f04c21d3640 1 -- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04a80664a0 msgr2=0x7f04a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:18.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.510+0000 7f04c21d3640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04a80664a0 0x7f04a8068960 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f04bc0c4660 tx=0x7f04b0009040 comp rx=0 tx=0).stop 2023-10-30T13:55:18.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.510+0000 7f04c21d3640 1 -- 172.21.15.46:0/2937762147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc0ca0e0 msgr2=0x7f04bc1e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:18.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.510+0000 7f04c21d3640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc0ca0e0 0x7f04bc1e4090 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0b3040 tx=0x7f04ac01bad0 comp rx=0 tx=0).stop 2023-10-30T13:55:18.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.510+0000 7f04c21d3640 1 -- 172.21.15.46:0/2937762147 shutdown_connections 2023-10-30T13:55:18.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.510+0000 7f04c21d3640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04a80664a0 0x7f04a8068960 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:18.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.510+0000 7f04c21d3640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04bc1538a0 0x7f04bc15bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.510+0000 7f04c21d3640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04bc0cf160 0x7f04bc15b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.510+0000 7f04c21d3640 1 --2- 172.21.15.46:0/2937762147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04bc0ca0e0 0x7f04bc1e4090 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.510+0000 7f04c21d3640 1 -- 172.21.15.46:0/2937762147 >> 172.21.15.46:0/2937762147 conn(0x7f04bc06bfe0 msgr2=0x7f04bc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.510+0000 7f04c21d3640 1 -- 172.21.15.46:0/2937762147 shutdown_connections 2023-10-30T13:55:18.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:18.510+0000 7f04c21d3640 1 -- 172.21.15.46:0/2937762147 wait complete. 2023-10-30T13:55:18.541 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:55:18.542 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before thrashing 2023-10-30T13:55:22.003 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:55:22.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:55:26.543 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc8ef7640 1 Processor -- start 2023-10-30T13:55:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc8ef7640 1 -- start start 2023-10-30T13:55:26.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc8ef7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cc4150830 0x7f5cc40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:26.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc8ef7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cc40c29e0 0x7f5cc4159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:26.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc8ef7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc4156700 con 0x7f5cc40c85d0 2023-10-30T13:55:26.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc8ef7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc41576f0 con 0x7f5cc4150830 2023-10-30T13:55:26.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc8ef7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc4157260 con 0x7f5cc40c29e0 2023-10-30T13:55:26.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cc4150830 0x7f5cc40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:26.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cc4150830 0x7f5cc40c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57158/0 (socket says 172.21.15.46:57158) 2023-10-30T13:55:26.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc259c640 1 -- 172.21.15.46:0/3592146059 learned_addr learned my addr 172.21.15.46:0/3592146059 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:26.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cbbfff640 1 --2- 172.21.15.46:0/3592146059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cc40c29e0 0x7f5cc4159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:26.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc259c640 1 -- 172.21.15.46:0/3592146059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cc40c29e0 msgr2=0x7f5cc4159cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:26.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc259c640 1 --2- 172.21.15.46:0/3592146059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cc40c29e0 0x7f5cc4159cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:26.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc259c640 1 -- 172.21.15.46:0/3592146059 >> v1:172.21.15.154:6789/0 conn(0x7f5cc40c85d0 legacy=0x7f5cc40c4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:26.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.690+0000 7f5cc259c640 1 -- 172.21.15.46:0/3592146059 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc40c58b0 con 0x7f5cc4150830 2023-10-30T13:55:26.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.694+0000 7f5cc259c640 1 --2- 172.21.15.46:0/3592146059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc4150830 0x7f5cc40c51e0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009870 tx=0x7f5cb401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97f0442dde8f40 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:26.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.694+0000 7f5cc1d9b640 1 -- 172.21.15.46:0/3592146059 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40ab620 con 0x7f5cc4150830 2023-10-30T13:55:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.694+0000 7f5cc1d9b640 1 -- 172.21.15.46:0/3592146059 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb40ab780 con 0x7f5cc4150830 2023-10-30T13:55:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.694+0000 7f5cc1d9b640 1 -- 172.21.15.46:0/3592146059 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40abb10 con 0x7f5cc4150830 2023-10-30T13:55:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.694+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/3592146059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc4150830 msgr2=0x7f5cc40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.694+0000 7f5cc8ef7640 1 --2- 172.21.15.46:0/3592146059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc4150830 0x7f5cc40c51e0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009870 tx=0x7f5cb401c510 comp rx=0 tx=0).stop 2023-10-30T13:55:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.694+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/3592146059 shutdown_connections 2023-10-30T13:55:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.694+0000 7f5cc8ef7640 1 --2- 172.21.15.46:0/3592146059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cc40c29e0 0x7f5cc4159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.694+0000 7f5cc8ef7640 1 --2- 172.21.15.46:0/3592146059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc4150830 0x7f5cc40c51e0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.694+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/3592146059 >> 172.21.15.46:0/3592146059 conn(0x7f5cc406c6e0 msgr2=0x7f5cc4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.694+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/3592146059 shutdown_connections 2023-10-30T13:55:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.694+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/3592146059 wait complete. 2023-10-30T13:55:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc8ef7640 1 Processor -- start 2023-10-30T13:55:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc8ef7640 1 -- start start 2023-10-30T13:55:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc8ef7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40c29e0 0x7f5cc41f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc8ef7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cc40c85d0 0x7f5cc4144c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:26.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc8ef7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cc4150830 0x7f5cc4143e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc8ef7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc41468a0 con 0x7f5cc40c29e0 2023-10-30T13:55:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc8ef7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc41589c0 con 0x7f5cc40c85d0 2023-10-30T13:55:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc8ef7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc41f6700 con 0x7f5cc4150830 2023-10-30T13:55:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40c29e0 0x7f5cc41f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:26.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40c29e0 0x7f5cc41f61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57166/0 (socket says 172.21.15.46:57166) 2023-10-30T13:55:26.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc259c640 1 -- 172.21.15.46:0/4280099372 learned_addr learned my addr 172.21.15.46:0/4280099372 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:26.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cbbfff640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cc40c85d0 0x7f5cc4144c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:26.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc2d9d640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cc4150830 0x7f5cc4143e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:26.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc259c640 1 -- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cc4150830 msgr2=0x7f5cc4143e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:26.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc259c640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cc4150830 0x7f5cc4143e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc259c640 1 -- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cc40c85d0 msgr2=0x7f5cc4144c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc259c640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cc40c85d0 0x7f5cc4144c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc259c640 1 -- 172.21.15.46:0/4280099372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb40094d0 con 0x7f5cc40c29e0 2023-10-30T13:55:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc259c640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40c29e0 0x7f5cc41f61c0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40ad040 tx=0x7f5cb401ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cbaffd640 1 -- 172.21.15.46:0/4280099372 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40be3f0 con 0x7f5cc40c29e0 2023-10-30T13:55:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cbaffd640 1 -- 172.21.15.46:0/4280099372 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb40be570 con 0x7f5cc40c29e0 2023-10-30T13:55:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/4280099372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc4070d60 con 0x7f5cc40c29e0 2023-10-30T13:55:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.698+0000 7f5cbaffd640 1 -- 172.21.15.46:0/4280099372 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40be920 con 0x7f5cc40c29e0 2023-10-30T13:55:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.702+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/4280099372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc4145c20 con 0x7f5cc40c29e0 2023-10-30T13:55:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.702+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/4280099372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc41469e0 con 0x7f5cc40c29e0 2023-10-30T13:55:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.702+0000 7f5cbaffd640 1 -- 172.21.15.46:0/4280099372 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5cb40b43f0 con 0x7f5cc40c29e0 2023-10-30T13:55:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.702+0000 7f5cbaffd640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ca40664d0 0x7f5ca4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.702+0000 7f5cbaffd640 1 -- 172.21.15.46:0/4280099372 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f5cb411eac0 con 0x7f5cc40c29e0 2023-10-30T13:55:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.702+0000 7f5cbbfff640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ca40664d0 0x7f5ca4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.706+0000 7f5cbbfff640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ca40664d0 0x7f5ca4068990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4162140 tx=0x7f5cac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.706+0000 7f5cbaffd640 1 -- 172.21.15.46:0/4280099372 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5cc41469e0 con 0x7f5cc40c29e0 2023-10-30T13:55:26.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/4280099372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cc41469e0 con 0x7f5cc40c29e0 2023-10-30T13:55:26.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cbaffd640 1 -- 172.21.15.46:0/4280099372 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f5cc41469e0 con 0x7f5cc40c29e0 2023-10-30T13:55:26.904 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:26.905 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7518,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:17.419200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:26.905 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 144 2023-10-30T13:55:26.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ca40664d0 msgr2=0x7f5ca4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:26.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ca40664d0 0x7f5ca4068990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4162140 tx=0x7f5cac008040 comp rx=0 tx=0).stop 2023-10-30T13:55:26.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/4280099372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40c29e0 msgr2=0x7f5cc41f61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:26.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40c29e0 0x7f5cc41f61c0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40ad040 tx=0x7f5cb401ad50 comp rx=0 tx=0).stop 2023-10-30T13:55:26.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/4280099372 shutdown_connections 2023-10-30T13:55:26.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5ca40664d0 0x7f5ca4068990 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:26.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cc4150830 0x7f5cc4143e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:26.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cc40c85d0 0x7f5cc4144c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:26.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 --2- 172.21.15.46:0/4280099372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40c29e0 0x7f5cc41f61c0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:26.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/4280099372 >> 172.21.15.46:0/4280099372 conn(0x7f5cc406c6e0 msgr2=0x7f5cc40cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:26.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/4280099372 shutdown_connections 2023-10-30T13:55:26.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:26.902+0000 7f5cc8ef7640 1 -- 172.21.15.46:0/4280099372 wait complete. 2023-10-30T13:55:26.926 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=1) 2023-10-30T13:55:26.926 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:55:26.927 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:55:26.953 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:55:26.954 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:27.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.138+0000 7fcddd09c640 1 Processor -- start 2023-10-30T13:55:27.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.138+0000 7fcddd09c640 1 -- start start 2023-10-30T13:55:27.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.142+0000 7fcddd09c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcdd806f7a0 0x7fcdd806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:27.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.142+0000 7fcddd09c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd8066cf0 con 0x7fcdd8071110 2023-10-30T13:55:27.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.142+0000 7fcddd09c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd80c70d0 con 0x7fcdd80c6bd0 2023-10-30T13:55:27.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.142+0000 7fcddd09c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd80700c0 con 0x7fcdd806f7a0 2023-10-30T13:55:27.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.142+0000 7fcdd6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcdd806f7a0 0x7fcdd806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:27.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.142+0000 7fcdd6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcdd806f7a0 0x7fcdd806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57178/0 (socket says 172.21.15.46:57178) 2023-10-30T13:55:27.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.142+0000 7fcdd6d9d640 1 -- 172.21.15.46:0/1986245457 learned_addr learned my addr 172.21.15.46:0/1986245457 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:27.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.142+0000 7fcdd6d9d640 1 -- 172.21.15.46:0/1986245457 >> v1:172.21.15.154:6789/0 conn(0x7fcdd80c6bd0 legacy=0x7fcdd8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:27.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.142+0000 7fcdd6d9d640 1 -- 172.21.15.46:0/1986245457 >> v1:172.21.15.46:6789/0 conn(0x7fcdd8071110 legacy=0x7fcdd8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:27.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.142+0000 7fcdd6d9d640 1 -- 172.21.15.46:0/1986245457 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd80c7390 con 0x7fcdd806f7a0 2023-10-30T13:55:27.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcdd6d9d640 1 --2- 172.21.15.46:0/1986245457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdd806f7a0 0x7fcdd806fb80 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009870 tx=0x7fcdc801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dbdf2caec26c1e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:27.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcdd5d9b640 1 -- 172.21.15.46:0/1986245457 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc80a8a30 con 0x7fcdd806f7a0 2023-10-30T13:55:27.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcdd5d9b640 1 -- 172.21.15.46:0/1986245457 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc80a8b90 con 0x7fcdd806f7a0 2023-10-30T13:55:27.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcdd5d9b640 1 -- 172.21.15.46:0/1986245457 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc80b0440 con 0x7fcdd806f7a0 2023-10-30T13:55:27.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcddd09c640 1 -- 172.21.15.46:0/1986245457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdd806f7a0 msgr2=0x7fcdd806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:27.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcddd09c640 1 --2- 172.21.15.46:0/1986245457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdd806f7a0 0x7fcdd806fb80 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009870 tx=0x7fcdc801ed30 comp rx=0 tx=0).stop 2023-10-30T13:55:27.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcddd09c640 1 -- 172.21.15.46:0/1986245457 shutdown_connections 2023-10-30T13:55:27.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcddd09c640 1 --2- 172.21.15.46:0/1986245457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdd806f7a0 0x7fcdd806fb80 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:27.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcddd09c640 1 -- 172.21.15.46:0/1986245457 >> 172.21.15.46:0/1986245457 conn(0x7fcdd80c7910 msgr2=0x7fcdd80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:27.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcddd09c640 1 -- 172.21.15.46:0/1986245457 shutdown_connections 2023-10-30T13:55:27.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcddd09c640 1 -- 172.21.15.46:0/1986245457 wait complete. 2023-10-30T13:55:27.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcddd09c640 1 Processor -- start 2023-10-30T13:55:27.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.146+0000 7fcddd09c640 1 -- start start 2023-10-30T13:55:27.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcddd09c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcdd806f7a0 0x7fcdd80c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:27.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcddd09c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdd8071110 0x7fcdd81eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:27.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcddd09c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd80c6bd0 0x7fcdd814de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:27.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcddd09c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd8155f60 con 0x7fcdd8071110 2023-10-30T13:55:27.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcddd09c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd8155710 con 0x7fcdd80c6bd0 2023-10-30T13:55:27.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcddd09c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd81f2160 con 0x7fcdd806f7a0 2023-10-30T13:55:27.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdd659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdd8071110 0x7fcdd81eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:27.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdd659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdd8071110 0x7fcdd81eeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57192/0 (socket says 172.21.15.46:57192) 2023-10-30T13:55:27.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdd659c640 1 -- 172.21.15.46:0/4202309807 learned_addr learned my addr 172.21.15.46:0/4202309807 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:27.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdd6d9d640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcdd806f7a0 0x7fcdd80c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:27.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdd759e640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd80c6bd0 0x7fcdd814de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:27.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdd659c640 1 -- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcdd806f7a0 msgr2=0x7fcdd80c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:27.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdd659c640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcdd806f7a0 0x7fcdd80c5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:27.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdd659c640 1 -- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd80c6bd0 msgr2=0x7fcdd814de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:27.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdd659c640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd80c6bd0 0x7fcdd814de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:27.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdd659c640 1 -- 172.21.15.46:0/4202309807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc80094d0 con 0x7fcdd8071110 2023-10-30T13:55:27.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdd659c640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdd8071110 0x7fcdd81eeb50 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00c4d0 tx=0x7fcdcc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:27.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdbf7fe640 1 -- 172.21.15.46:0/4202309807 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0052d0 con 0x7fcdd8071110 2023-10-30T13:55:27.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdbf7fe640 1 -- 172.21.15.46:0/4202309807 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc005430 con 0x7fcdd8071110 2023-10-30T13:55:27.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdbf7fe640 1 -- 172.21.15.46:0/4202309807 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc012830 con 0x7fcdd8071110 2023-10-30T13:55:27.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcddd09c640 1 -- 172.21.15.46:0/4202309807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd81f2440 con 0x7fcdd8071110 2023-10-30T13:55:27.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcddd09c640 1 -- 172.21.15.46:0/4202309807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd81f2a00 con 0x7fcdd8071110 2023-10-30T13:55:27.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcddd09c640 1 -- 172.21.15.46:0/4202309807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd804e170 con 0x7fcdd8071110 2023-10-30T13:55:27.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.150+0000 7fcdbf7fe640 1 -- 172.21.15.46:0/4202309807 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcdcc012990 con 0x7fcdd8071110 2023-10-30T13:55:27.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.154+0000 7fcdbf7fe640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcdc40664a0 0x7fcdc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:27.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.154+0000 7fcdbf7fe640 1 -- 172.21.15.46:0/4202309807 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7fcdcc0840c0 con 0x7fcdd8071110 2023-10-30T13:55:27.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.154+0000 7fcdd6d9d640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcdc40664a0 0x7fcdc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:27.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.154+0000 7fcdd6d9d640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcdc40664a0 0x7fcdc4068960 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80b2000 tx=0x7fcdc8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:27.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.158+0000 7fcdbf7fe640 1 -- 172.21.15.46:0/4202309807 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcdcc050e30 con 0x7fcdd8071110 2023-10-30T13:55:27.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.346+0000 7fcddd09c640 1 -- 172.21.15.46:0/4202309807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdd80c27e0 con 0x7fcdd8071110 2023-10-30T13:55:27.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcdbf7fe640 1 -- 172.21.15.46:0/4202309807 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+6520 (secure 0 0 0) 0x7fcdcc054450 con 0x7fcdd8071110 2023-10-30T13:55:27.351 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:27.352 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7518,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:17.419200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:27.352 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 144 2023-10-30T13:55:27.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcddd09c640 1 -- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcdc40664a0 msgr2=0x7fcdc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:27.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcddd09c640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcdc40664a0 0x7fcdc4068960 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80b2000 tx=0x7fcdc8008040 comp rx=0 tx=0).stop 2023-10-30T13:55:27.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcddd09c640 1 -- 172.21.15.46:0/4202309807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdd8071110 msgr2=0x7fcdd81eeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:27.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcddd09c640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdd8071110 0x7fcdd81eeb50 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00c4d0 tx=0x7fcdcc00c990 comp rx=0 tx=0).stop 2023-10-30T13:55:27.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcddd09c640 1 -- 172.21.15.46:0/4202309807 shutdown_connections 2023-10-30T13:55:27.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcddd09c640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcdc40664a0 0x7fcdc4068960 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:27.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcddd09c640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd80c6bd0 0x7fcdd814de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:27.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcddd09c640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdd8071110 0x7fcdd81eeb50 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:27.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcddd09c640 1 --2- 172.21.15.46:0/4202309807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcdd806f7a0 0x7fcdd80c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:27.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcddd09c640 1 -- 172.21.15.46:0/4202309807 >> 172.21.15.46:0/4202309807 conn(0x7fcdd80c7910 msgr2=0x7fcdd814f650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:27.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcddd09c640 1 -- 172.21.15.46:0/4202309807 shutdown_connections 2023-10-30T13:55:27.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:27.350+0000 7fcddd09c640 1 -- 172.21.15.46:0/4202309807 wait complete. 2023-10-30T13:55:27.373 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:55:29.374 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.522+0000 7f08f550f640 1 Processor -- start 2023-10-30T13:55:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.522+0000 7f08f550f640 1 -- start start 2023-10-30T13:55:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08f550f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f0070a00 0x7f08f0070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08f550f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f08f0069e90 0x7f08f006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08f550f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f0071380 con 0x7f08f01539a0 2023-10-30T13:55:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08f550f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f006fbb0 con 0x7f08f0069e90 2023-10-30T13:55:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08f550f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f0153d80 con 0x7f08f0070a00 2023-10-30T13:55:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08eeffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f08f01539a0 0x7f08f015a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48076/0 (socket says 172.21.15.46:48076) 2023-10-30T13:55:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08eeffd640 1 -- 172.21.15.46:0/1737248826 learned_addr learned my addr 172.21.15.46:0/1737248826 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:55:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08e7fff640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f0070a00 0x7f08f0070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ef7fe640 1 --2- 172.21.15.46:0/1737248826 >> v2:172.21.15.154:3300/0 conn(0x7f08f0069e90 0x7f08f006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369415878 0 0) 0x7f08f0071380 con 0x7f08f01539a0 2023-10-30T13:55:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08dc003530 con 0x7f08f01539a0 2023-10-30T13:55:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3990241783 0 0) 0x7f08dc003530 con 0x7f08f01539a0 2023-10-30T13:55:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f0070a00 msgr2=0x7f08f0070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f0070a00 0x7f08f0070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 >> v2:172.21.15.154:3300/0 conn(0x7f08f0069e90 msgr2=0x7f08f006a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 --2- 172.21.15.46:0/1737248826 >> v2:172.21.15.154:3300/0 conn(0x7f08f0069e90 0x7f08f006a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f00c5c50 con 0x7f08f01539a0 2023-10-30T13:55:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f08e0001680 con 0x7f08f01539a0 2023-10-30T13:55:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 >> v1:172.21.15.46:6789/0 conn(0x7f08f01539a0 legacy=0x7f08f015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08dc003980 0x7f08dc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08dc0047d0 0x7f08dc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f0070a00 0x7f08dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f006fbb0 con 0x7f08dc0047d0 2023-10-30T13:55:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f0153d80 con 0x7f08dc003980 2023-10-30T13:55:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08dc00b990 con 0x7f08f0070a00 2023-10-30T13:55:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08e7fff640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08dc0047d0 0x7f08dc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08eeffd640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08dc003980 0x7f08dc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08ef7fe640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f0070a00 0x7f08dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08e7fff640 1 -- 172.21.15.46:0/1737248826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f0070a00 msgr2=0x7f08dc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08e7fff640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f0070a00 0x7f08dc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08e7fff640 1 -- 172.21.15.46:0/1737248826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08dc003980 msgr2=0x7f08dc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08e7fff640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08dc003980 0x7f08dc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08e7fff640 1 -- 172.21.15.46:0/1737248826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08dc003530 con 0x7f08dc0047d0 2023-10-30T13:55:29.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.526+0000 7f08eeffd640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08dc003980 0x7f08dc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:29.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.530+0000 7f08e7fff640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08dc0047d0 0x7f08dc004c50 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009b10 tx=0x7f08d801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81f39fe93b7bf4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:29.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.530+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d80a6430 con 0x7f08dc0047d0 2023-10-30T13:55:29.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.530+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d80a6590 con 0x7f08dc0047d0 2023-10-30T13:55:29.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.530+0000 7f08ee7fc640 1 -- 172.21.15.46:0/1737248826 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d80a6940 con 0x7f08dc0047d0 2023-10-30T13:55:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.530+0000 7f08f550f640 1 -- 172.21.15.46:0/1737248826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08dc0047d0 msgr2=0x7f08dc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.530+0000 7f08f550f640 1 --2- 172.21.15.46:0/1737248826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08dc0047d0 0x7f08dc004c50 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009b10 tx=0x7f08d801ca20 comp rx=0 tx=0).stop 2023-10-30T13:55:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.530+0000 7f08ef7fe640 1 -- 172.21.15.46:0/1737248826 reap_dead start 2023-10-30T13:55:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.530+0000 7f08f550f640 1 -- 172.21.15.46:0/1737248826 shutdown_connections 2023-10-30T13:55:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.530+0000 7f08f550f640 1 -- 172.21.15.46:0/1737248826 >> 172.21.15.46:0/1737248826 conn(0x7f08f00c7910 msgr2=0x7f08f00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.530+0000 7f08f550f640 1 -- 172.21.15.46:0/1737248826 shutdown_connections 2023-10-30T13:55:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.530+0000 7f08f550f640 1 -- 172.21.15.46:0/1737248826 wait complete. 2023-10-30T13:55:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08f550f640 1 Processor -- start 2023-10-30T13:55:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08f550f640 1 -- start start 2023-10-30T13:55:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08f550f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08dc0047d0 0x7f08f015ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08f550f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08f0150ad0 0x7f08f015f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08f550f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f00c8d10 0x7f08f00c9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08f550f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f00c96d0 con 0x7f08dc0047d0 2023-10-30T13:55:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08f550f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f00ca6a0 con 0x7f08f0150ad0 2023-10-30T13:55:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08f550f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f00c0970 con 0x7f08f00c8d10 2023-10-30T13:55:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08eeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08dc0047d0 0x7f08f015ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08eeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08dc0047d0 0x7f08f015ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57214/0 (socket says 172.21.15.46:57214) 2023-10-30T13:55:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08eeffd640 1 -- 172.21.15.46:0/2720460071 learned_addr learned my addr 172.21.15.46:0/2720460071 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08ef7fe640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f00c8d10 0x7f08f00c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08e7fff640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08f0150ad0 0x7f08f015f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08eeffd640 1 -- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f00c8d10 msgr2=0x7f08f00c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08eeffd640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f00c8d10 0x7f08f00c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08eeffd640 1 -- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08f0150ad0 msgr2=0x7f08f015f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08eeffd640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08f0150ad0 0x7f08f015f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08eeffd640 1 -- 172.21.15.46:0/2720460071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f00c0c50 con 0x7f08dc0047d0 2023-10-30T13:55:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08eeffd640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08dc0047d0 0x7f08f015ec20 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f08e0001000 tx=0x7f08e000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08e77fe640 1 -- 172.21.15.46:0/2720460071 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e000c840 con 0x7f08dc0047d0 2023-10-30T13:55:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08e77fe640 1 -- 172.21.15.46:0/2720460071 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e000c9a0 con 0x7f08dc0047d0 2023-10-30T13:55:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08e77fe640 1 -- 172.21.15.46:0/2720460071 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e000cd10 con 0x7f08dc0047d0 2023-10-30T13:55:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08f550f640 1 -- 172.21.15.46:0/2720460071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f00c0f30 con 0x7f08dc0047d0 2023-10-30T13:55:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08f550f640 1 -- 172.21.15.46:0/2720460071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f00ca960 con 0x7f08dc0047d0 2023-10-30T13:55:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.534+0000 7f08f550f640 1 -- 172.21.15.46:0/2720460071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f00c1a60 con 0x7f08dc0047d0 2023-10-30T13:55:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.538+0000 7f08e77fe640 1 -- 172.21.15.46:0/2720460071 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f08e001d460 con 0x7f08dc0047d0 2023-10-30T13:55:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.538+0000 7f08e77fe640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f08d00664a0 0x7f08d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.538+0000 7f08e77fe640 1 -- 172.21.15.46:0/2720460071 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f08e0084570 con 0x7f08dc0047d0 2023-10-30T13:55:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.538+0000 7f08e7fff640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f08d00664a0 0x7f08d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:29.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.538+0000 7f08e7fff640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f08d00664a0 0x7f08d0068960 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f08d80ad000 tx=0x7f08d80b2040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:29.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.542+0000 7f08e77fe640 1 -- 172.21.15.46:0/2720460071 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f08f00c1a60 con 0x7f08dc0047d0 2023-10-30T13:55:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 -- 172.21.15.46:0/2720460071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08f00c1a60 con 0x7f08dc0047d0 2023-10-30T13:55:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08e77fe640 1 -- 172.21.15.46:0/2720460071 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f08f00c1a60 con 0x7f08dc0047d0 2023-10-30T13:55:29.736 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:29.736 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7518,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:17.419200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:29.736 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 144 2023-10-30T13:55:29.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 -- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f08d00664a0 msgr2=0x7f08d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:29.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f08d00664a0 0x7f08d0068960 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f08d80ad000 tx=0x7f08d80b2040 comp rx=0 tx=0).stop 2023-10-30T13:55:29.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 -- 172.21.15.46:0/2720460071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08dc0047d0 msgr2=0x7f08f015ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:29.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08dc0047d0 0x7f08f015ec20 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f08e0001000 tx=0x7f08e000ec80 comp rx=0 tx=0).stop 2023-10-30T13:55:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 -- 172.21.15.46:0/2720460071 shutdown_connections 2023-10-30T13:55:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f08d00664a0 0x7f08d0068960 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08f00c8d10 0x7f08f00c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08f0150ad0 0x7f08f015f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 --2- 172.21.15.46:0/2720460071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08dc0047d0 0x7f08f015ec20 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:29.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 -- 172.21.15.46:0/2720460071 >> 172.21.15.46:0/2720460071 conn(0x7f08f00c7910 msgr2=0x7f08f006b340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:29.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 -- 172.21.15.46:0/2720460071 shutdown_connections 2023-10-30T13:55:29.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:29.734+0000 7f08f550f640 1 -- 172.21.15.46:0/2720460071 wait complete. 2023-10-30T13:55:29.757 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:55:31.758 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:31.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.910+0000 7fdebd70f640 1 Processor -- start 2023-10-30T13:55:31.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.910+0000 7fdebd70f640 1 -- start start 2023-10-30T13:55:31.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.910+0000 7fdebd70f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdeb80c05d0 0x7fdeb813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:31.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.910+0000 7fdebd70f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdeb80c6f60 0x7fdeb813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:31.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.910+0000 7fdebd70f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb80c0a50 con 0x7fdeb80c6b80 2023-10-30T13:55:31.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.910+0000 7fdebd70f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb806f6e0 con 0x7fdeb80c6f60 2023-10-30T13:55:31.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.910+0000 7fdebd70f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb806f9c0 con 0x7fdeb80c05d0 2023-10-30T13:55:31.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.910+0000 7fdeb77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdeb80c6f60 0x7fdeb813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:31.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.914+0000 7fdeb77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdeb80c6f60 0x7fdeb813f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57226/0 (socket says 172.21.15.46:57226) 2023-10-30T13:55:31.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.914+0000 7fdeb77fe640 1 -- 172.21.15.46:0/4082299228 learned_addr learned my addr 172.21.15.46:0/4082299228 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:31.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.914+0000 7fdeb67fc640 1 --2- 172.21.15.46:0/4082299228 >> v2:172.21.15.154:3300/0 conn(0x7fdeb80c05d0 0x7fdeb813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:31.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.914+0000 7fdeb77fe640 1 -- 172.21.15.46:0/4082299228 >> v2:172.21.15.154:3300/0 conn(0x7fdeb80c05d0 msgr2=0x7fdeb813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:31.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.914+0000 7fdeb77fe640 1 --2- 172.21.15.46:0/4082299228 >> v2:172.21.15.154:3300/0 conn(0x7fdeb80c05d0 0x7fdeb813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:31.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.914+0000 7fdeb77fe640 1 -- 172.21.15.46:0/4082299228 >> v1:172.21.15.154:6789/0 conn(0x7fdeb80c6b80 legacy=0x7fdeb80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:31.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.914+0000 7fdeb77fe640 1 -- 172.21.15.46:0/4082299228 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb806fca0 con 0x7fdeb80c6f60 2023-10-30T13:55:31.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.914+0000 7fdeb67fc640 1 --2- 172.21.15.46:0/4082299228 >> v2:172.21.15.154:3300/0 conn(0x7fdeb80c05d0 0x7fdeb813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:31.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.914+0000 7fdeb77fe640 1 --2- 172.21.15.46:0/4082299228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdeb80c6f60 0x7fdeb813f2d0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fdeb806c7e0 tx=0x7fdeb001c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c1f80ef4021ff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:31.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.914+0000 7fdeb5ffb640 1 -- 172.21.15.46:0/4082299228 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb00a5760 con 0x7fdeb80c6f60 2023-10-30T13:55:31.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.914+0000 7fdeb5ffb640 1 -- 172.21.15.46:0/4082299228 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb00a58c0 con 0x7fdeb80c6f60 2023-10-30T13:55:31.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.914+0000 7fdeb5ffb640 1 -- 172.21.15.46:0/4082299228 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb00a5c50 con 0x7fdeb80c6f60 2023-10-30T13:55:31.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 -- 172.21.15.46:0/4082299228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdeb80c6f60 msgr2=0x7fdeb813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:31.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 --2- 172.21.15.46:0/4082299228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdeb80c6f60 0x7fdeb813f2d0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fdeb806c7e0 tx=0x7fdeb001c990 comp rx=0 tx=0).stop 2023-10-30T13:55:31.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 -- 172.21.15.46:0/4082299228 shutdown_connections 2023-10-30T13:55:31.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 --2- 172.21.15.46:0/4082299228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdeb80c6f60 0x7fdeb813f2d0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:31.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 --2- 172.21.15.46:0/4082299228 >> v2:172.21.15.154:3300/0 conn(0x7fdeb80c05d0 0x7fdeb813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:31.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 -- 172.21.15.46:0/4082299228 >> 172.21.15.46:0/4082299228 conn(0x7fdeb80c7910 msgr2=0x7fdeb80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:31.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 -- 172.21.15.46:0/4082299228 shutdown_connections 2023-10-30T13:55:31.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 -- 172.21.15.46:0/4082299228 wait complete. 2023-10-30T13:55:31.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 Processor -- start 2023-10-30T13:55:31.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 -- start start 2023-10-30T13:55:31.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdeb80c05d0 0x7fdeb81f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:31.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdeb80c6b80 0x7fdeb8133700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:31.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdeb80c6f60 0x7fdeb81510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:31.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb81f55b0 con 0x7fdeb80c6f60 2023-10-30T13:55:31.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80c7e10 con 0x7fdeb80c6b80 2023-10-30T13:55:31.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdebd70f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80c80d0 con 0x7fdeb80c05d0 2023-10-30T13:55:31.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdeb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdeb80c6f60 0x7fdeb81510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:31.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdeb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdeb80c6f60 0x7fdeb81510d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57228/0 (socket says 172.21.15.46:57228) 2023-10-30T13:55:31.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdeb77fe640 1 -- 172.21.15.46:0/281216247 learned_addr learned my addr 172.21.15.46:0/281216247 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:31.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdeb67fc640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdeb80c6b80 0x7fdeb8133700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:31.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdeb6ffd640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdeb80c05d0 0x7fdeb81f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:31.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdeb77fe640 1 -- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdeb80c05d0 msgr2=0x7fdeb81f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:31.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdeb77fe640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdeb80c05d0 0x7fdeb81f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:31.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdeb77fe640 1 -- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdeb80c6b80 msgr2=0x7fdeb8133700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:31.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdeb77fe640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdeb80c6b80 0x7fdeb8133700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:31.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdeb77fe640 1 -- 172.21.15.46:0/281216247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb0009b30 con 0x7fdeb80c6f60 2023-10-30T13:55:31.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdeb67fc640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdeb80c6b80 0x7fdeb8133700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:31.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.918+0000 7fdeb6ffd640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdeb80c05d0 0x7fdeb81f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:31.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.922+0000 7fdeb77fe640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdeb80c6f60 0x7fdeb81510d0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fdeb001a5c0 tx=0x7fdeb0019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:31.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.922+0000 7fde9f7fe640 1 -- 172.21.15.46:0/281216247 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb00b8880 con 0x7fdeb80c6f60 2023-10-30T13:55:31.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.922+0000 7fde9f7fe640 1 -- 172.21.15.46:0/281216247 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb00b89e0 con 0x7fdeb80c6f60 2023-10-30T13:55:31.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.922+0000 7fdebd70f640 1 -- 172.21.15.46:0/281216247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb80cabe0 con 0x7fdeb80c6f60 2023-10-30T13:55:31.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.922+0000 7fde9f7fe640 1 -- 172.21.15.46:0/281216247 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb00b8d90 con 0x7fdeb80c6f60 2023-10-30T13:55:31.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.922+0000 7fdebd70f640 1 -- 172.21.15.46:0/281216247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb80ca900 con 0x7fdeb80c6f60 2023-10-30T13:55:31.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.922+0000 7fdebd70f640 1 -- 172.21.15.46:0/281216247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb81f5720 con 0x7fdeb80c6f60 2023-10-30T13:55:31.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.922+0000 7fde9f7fe640 1 -- 172.21.15.46:0/281216247 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdeb00c3460 con 0x7fdeb80c6f60 2023-10-30T13:55:31.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.922+0000 7fde9f7fe640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdea40664a0 0x7fdea4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:31.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.922+0000 7fde9f7fe640 1 -- 172.21.15.46:0/281216247 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7fdeb011fd80 con 0x7fdeb80c6f60 2023-10-30T13:55:31.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.922+0000 7fdeb6ffd640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdea40664a0 0x7fdea4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:31.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.926+0000 7fdeb6ffd640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdea40664a0 0x7fdea4068960 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fdea8003ec0 tx=0x7fdea8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:31.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:31.926+0000 7fde9f7fe640 1 -- 172.21.15.46:0/281216247 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdeb00ecca0 con 0x7fdeb80c6f60 2023-10-30T13:55:32.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 -- 172.21.15.46:0/281216247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdeb804e170 con 0x7fdeb80c6f60 2023-10-30T13:55:32.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fde9f7fe640 1 -- 172.21.15.46:0/281216247 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+6520 (secure 0 0 0) 0x7fdeb0009510 con 0x7fdeb80c6f60 2023-10-30T13:55:32.120 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:32.120 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7518,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:17.419200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:32.120 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 144 2023-10-30T13:55:32.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 -- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdea40664a0 msgr2=0x7fdea4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:32.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdea40664a0 0x7fdea4068960 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fdea8003ec0 tx=0x7fdea8009040 comp rx=0 tx=0).stop 2023-10-30T13:55:32.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 -- 172.21.15.46:0/281216247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdeb80c6f60 msgr2=0x7fdeb81510d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:32.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdeb80c6f60 0x7fdeb81510d0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fdeb001a5c0 tx=0x7fdeb0019730 comp rx=0 tx=0).stop 2023-10-30T13:55:32.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 -- 172.21.15.46:0/281216247 shutdown_connections 2023-10-30T13:55:32.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdea40664a0 0x7fdea4068960 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:32.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdeb80c6f60 0x7fdeb81510d0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:32.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdeb80c6b80 0x7fdeb8133700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:32.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 --2- 172.21.15.46:0/281216247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdeb80c05d0 0x7fdeb81f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:32.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 -- 172.21.15.46:0/281216247 >> 172.21.15.46:0/281216247 conn(0x7fdeb80c7910 msgr2=0x7fdeb8131ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:32.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 -- 172.21.15.46:0/281216247 shutdown_connections 2023-10-30T13:55:32.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:32.118+0000 7fdebd70f640 1 -- 172.21.15.46:0/281216247 wait complete. 2023-10-30T13:55:32.142 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:55:34.143 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:34.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.294+0000 7f8b3bcea640 1 Processor -- start 2023-10-30T13:55:34.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.294+0000 7f8b3bcea640 1 -- start start 2023-10-30T13:55:34.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b3bcea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8b3406f7a0 0x7f8b3406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:34.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b3bcea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 0x7f8b34158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:34.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b3bcea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b34066cf0 con 0x7f8b340c6bd0 2023-10-30T13:55:34.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b3bcea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b340700c0 con 0x7f8b3406f7a0 2023-10-30T13:55:34.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b3bcea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340c70d0 con 0x7f8b34071110 2023-10-30T13:55:34.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b3a287640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 0x7f8b34158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:34.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b39a86640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8b3406f7a0 0x7f8b3406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:34.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b3a287640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 0x7f8b34158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:56754/0 (socket says 172.21.15.46:56754) 2023-10-30T13:55:34.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b3a287640 1 -- 172.21.15.46:0/4114726048 learned_addr learned my addr 172.21.15.46:0/4114726048 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:34.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b39285640 1 --1- 172.21.15.46:0/4114726048 >> v1:172.21.15.154:6789/0 conn(0x7f8b340c6bd0 0x7f8b34069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:42562/0 (socket says 172.21.15.46:42562) 2023-10-30T13:55:34.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b3a287640 1 -- 172.21.15.46:0/4114726048 >> v2:172.21.15.154:3300/0 conn(0x7f8b3406f7a0 msgr2=0x7f8b3406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:34.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b3a287640 1 --2- 172.21.15.46:0/4114726048 >> v2:172.21.15.154:3300/0 conn(0x7f8b3406f7a0 0x7f8b3406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:34.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b3a287640 1 -- 172.21.15.46:0/4114726048 >> v1:172.21.15.154:6789/0 conn(0x7f8b340c6bd0 legacy=0x7f8b34069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:34.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.298+0000 7f8b3a287640 1 -- 172.21.15.46:0/4114726048 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b340c7390 con 0x7f8b34071110 2023-10-30T13:55:34.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b3a287640 1 --2- 172.21.15.46:0/4114726048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 0x7f8b34158110 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009e50 tx=0x7f8b2c01f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f699fbf01484244 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:34.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b38a84640 1 -- 172.21.15.46:0/4114726048 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c0af400 con 0x7f8b34071110 2023-10-30T13:55:34.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b38a84640 1 -- 172.21.15.46:0/4114726048 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c0ab050 con 0x7f8b34071110 2023-10-30T13:55:34.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b38a84640 1 -- 172.21.15.46:0/4114726048 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c0af780 con 0x7f8b34071110 2023-10-30T13:55:34.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b3bcea640 1 -- 172.21.15.46:0/4114726048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 msgr2=0x7f8b34158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:34.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b3bcea640 1 --2- 172.21.15.46:0/4114726048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 0x7f8b34158110 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009e50 tx=0x7f8b2c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:55:34.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b3bcea640 1 -- 172.21.15.46:0/4114726048 shutdown_connections 2023-10-30T13:55:34.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b3bcea640 1 --2- 172.21.15.46:0/4114726048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 0x7f8b34158110 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:34.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b3bcea640 1 --2- 172.21.15.46:0/4114726048 >> v2:172.21.15.154:3300/0 conn(0x7f8b3406f7a0 0x7f8b3406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:34.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b3bcea640 1 -- 172.21.15.46:0/4114726048 >> 172.21.15.46:0/4114726048 conn(0x7f8b340c7910 msgr2=0x7f8b340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:34.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b3bcea640 1 -- 172.21.15.46:0/4114726048 shutdown_connections 2023-10-30T13:55:34.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b3bcea640 1 -- 172.21.15.46:0/4114726048 wait complete. 2023-10-30T13:55:34.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b3bcea640 1 Processor -- start 2023-10-30T13:55:34.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.302+0000 7f8b3bcea640 1 -- start start 2023-10-30T13:55:34.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b3bcea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b3406f7a0 0x7f8b340c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:34.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b3bcea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 0x7f8b341ecf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:34.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b3bcea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b340c6bd0 0x7f8b341f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b3bcea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b341ef360 con 0x7f8b3406f7a0 2023-10-30T13:55:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b3bcea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b34154a50 con 0x7f8b340c6bd0 2023-10-30T13:55:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b3bcea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b34152680 con 0x7f8b34071110 2023-10-30T13:55:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b39a86640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b3406f7a0 0x7f8b340c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b39a86640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b3406f7a0 0x7f8b340c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57242/0 (socket says 172.21.15.46:57242) 2023-10-30T13:55:34.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b39a86640 1 -- 172.21.15.46:0/3815170083 learned_addr learned my addr 172.21.15.46:0/3815170083 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:34.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b39285640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 0x7f8b341ecf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:34.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b3a287640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b340c6bd0 0x7f8b341f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:34.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b39a86640 1 -- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 msgr2=0x7f8b341ecf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:34.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b39a86640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 0x7f8b341ecf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:34.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b39a86640 1 -- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b340c6bd0 msgr2=0x7f8b341f2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:34.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b39a86640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b340c6bd0 0x7f8b341f2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:34.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b39a86640 1 -- 172.21.15.46:0/3815170083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c009b30 con 0x7f8b3406f7a0 2023-10-30T13:55:34.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b39285640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 0x7f8b341ecf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:34.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b39a86640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b3406f7a0 0x7f8b340c12a0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000e870 tx=0x7f8b3000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:34.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b2a7fc640 1 -- 172.21.15.46:0/3815170083 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30009510 con 0x7f8b3406f7a0 2023-10-30T13:55:34.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b2a7fc640 1 -- 172.21.15.46:0/3815170083 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3000eec0 con 0x7f8b3406f7a0 2023-10-30T13:55:34.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b2a7fc640 1 -- 172.21.15.46:0/3815170083 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3000cb40 con 0x7f8b3406f7a0 2023-10-30T13:55:34.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b3bcea640 1 -- 172.21.15.46:0/3815170083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b34152960 con 0x7f8b3406f7a0 2023-10-30T13:55:34.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b3bcea640 1 -- 172.21.15.46:0/3815170083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b34152f20 con 0x7f8b3406f7a0 2023-10-30T13:55:34.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.306+0000 7f8b3bcea640 1 -- 172.21.15.46:0/3815170083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3404e170 con 0x7f8b3406f7a0 2023-10-30T13:55:34.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.310+0000 7f8b2a7fc640 1 -- 172.21.15.46:0/3815170083 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8b3000ccc0 con 0x7f8b3406f7a0 2023-10-30T13:55:34.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.310+0000 7f8b2a7fc640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8b180664a0 0x7f8b18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:34.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.310+0000 7f8b2a7fc640 1 -- 172.21.15.46:0/3815170083 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f8b300845f0 con 0x7f8b3406f7a0 2023-10-30T13:55:34.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.310+0000 7f8b39285640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8b180664a0 0x7f8b18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:34.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.310+0000 7f8b39285640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8b180664a0 0x7f8b18068960 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8b24000f70 tx=0x7f8b24007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:34.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.314+0000 7f8b2a7fc640 1 -- 172.21.15.46:0/3815170083 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8b300513e0 con 0x7f8b3406f7a0 2023-10-30T13:55:34.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.502+0000 7f8b3bcea640 1 -- 172.21.15.46:0/3815170083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b340b0420 con 0x7f8b3406f7a0 2023-10-30T13:55:34.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.502+0000 7f8b2a7fc640 1 -- 172.21.15.46:0/3815170083 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f8b30054a00 con 0x7f8b3406f7a0 2023-10-30T13:55:34.507 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:34.507 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7518,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:17.419200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:34.507 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 144 2023-10-30T13:55:34.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.506+0000 7f8b3bcea640 1 -- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8b180664a0 msgr2=0x7f8b18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:34.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.506+0000 7f8b3bcea640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8b180664a0 0x7f8b18068960 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8b24000f70 tx=0x7f8b24007d50 comp rx=0 tx=0).stop 2023-10-30T13:55:34.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.506+0000 7f8b3bcea640 1 -- 172.21.15.46:0/3815170083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b3406f7a0 msgr2=0x7f8b340c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:34.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.506+0000 7f8b3bcea640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b3406f7a0 0x7f8b340c12a0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000e870 tx=0x7f8b3000ec70 comp rx=0 tx=0).stop 2023-10-30T13:55:34.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.506+0000 7f8b3bcea640 1 -- 172.21.15.46:0/3815170083 shutdown_connections 2023-10-30T13:55:34.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.506+0000 7f8b3bcea640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8b180664a0 0x7f8b18068960 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:34.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.506+0000 7f8b3bcea640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b340c6bd0 0x7f8b341f2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.506+0000 7f8b3bcea640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b34071110 0x7f8b341ecf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.506+0000 7f8b3bcea640 1 --2- 172.21.15.46:0/3815170083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b3406f7a0 0x7f8b340c12a0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.506+0000 7f8b3bcea640 1 -- 172.21.15.46:0/3815170083 >> 172.21.15.46:0/3815170083 conn(0x7f8b340c7910 msgr2=0x7f8b340ca230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.506+0000 7f8b3bcea640 1 -- 172.21.15.46:0/3815170083 shutdown_connections 2023-10-30T13:55:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:34.506+0000 7f8b3bcea640 1 -- 172.21.15.46:0/3815170083 wait complete. 2023-10-30T13:55:34.527 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:55:36.529 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:36.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c2088640 1 Processor -- start 2023-10-30T13:55:36.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c2088640 1 -- start start 2023-10-30T13:55:36.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c2088640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f58bc1538f0 0x7f58bc159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:36.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c2088640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58bc0cefa0 0x7f58bc0cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:36.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c2088640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc0ca6a0 con 0x7f58bc15cd50 2023-10-30T13:55:36.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c2088640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58bc0cf8f0 con 0x7f58bc0cefa0 2023-10-30T13:55:36.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c2088640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58bc15d300 con 0x7f58bc1538f0 2023-10-30T13:55:36.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c1887640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58bc0cefa0 0x7f58bc0cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:36.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c1887640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58bc0cefa0 0x7f58bc0cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52440/0 (socket says 172.21.15.46:52440) 2023-10-30T13:55:36.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c1887640 1 -- 172.21.15.46:0/852694142 learned_addr learned my addr 172.21.15.46:0/852694142 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:36.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c0885640 1 --1- 172.21.15.46:0/852694142 >> v1:172.21.15.46:6789/0 conn(0x7f58bc15cd50 0x7f58bc0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55820/0 (socket says 172.21.15.46:55820) 2023-10-30T13:55:36.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c1086640 1 --2- 172.21.15.46:0/852694142 >> v2:172.21.15.154:3300/0 conn(0x7f58bc1538f0 0x7f58bc159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:36.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c1887640 1 -- 172.21.15.46:0/852694142 >> v2:172.21.15.154:3300/0 conn(0x7f58bc1538f0 msgr2=0x7f58bc159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:36.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c1887640 1 --2- 172.21.15.46:0/852694142 >> v2:172.21.15.154:3300/0 conn(0x7f58bc1538f0 0x7f58bc159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:36.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c1887640 1 -- 172.21.15.46:0/852694142 >> v1:172.21.15.46:6789/0 conn(0x7f58bc15cd50 legacy=0x7f58bc0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:36.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58c1887640 1 -- 172.21.15.46:0/852694142 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc15d5e0 con 0x7f58bc0cefa0 2023-10-30T13:55:36.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.682+0000 7f58b3fff640 1 -- 172.21.15.46:0/852694142 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746214414 0 0) 0x7f58bc0ca6a0 con 0x7f58bc15cd50 2023-10-30T13:55:36.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.686+0000 7f58c1887640 1 --2- 172.21.15.46:0/852694142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58bc0cefa0 0x7f58bc0cf3b0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0c9cb0 tx=0x7f58b401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76e1af19da0cbc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:36.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.686+0000 7f58b3fff640 1 -- 172.21.15.46:0/852694142 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b40af400 con 0x7f58bc0cefa0 2023-10-30T13:55:36.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.686+0000 7f58b3fff640 1 -- 172.21.15.46:0/852694142 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b40ab050 con 0x7f58bc0cefa0 2023-10-30T13:55:36.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.686+0000 7f58b3fff640 1 -- 172.21.15.46:0/852694142 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b40af780 con 0x7f58bc0cefa0 2023-10-30T13:55:36.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.686+0000 7f58c2088640 1 -- 172.21.15.46:0/852694142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58bc0cefa0 msgr2=0x7f58bc0cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:36.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.686+0000 7f58c2088640 1 --2- 172.21.15.46:0/852694142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58bc0cefa0 0x7f58bc0cf3b0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0c9cb0 tx=0x7f58b401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:55:36.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.686+0000 7f58c2088640 1 -- 172.21.15.46:0/852694142 shutdown_connections 2023-10-30T13:55:36.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.686+0000 7f58c2088640 1 --2- 172.21.15.46:0/852694142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58bc0cefa0 0x7f58bc0cf3b0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:36.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.686+0000 7f58c2088640 1 --2- 172.21.15.46:0/852694142 >> v2:172.21.15.154:3300/0 conn(0x7f58bc1538f0 0x7f58bc159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:36.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.686+0000 7f58c2088640 1 -- 172.21.15.46:0/852694142 >> 172.21.15.46:0/852694142 conn(0x7f58bc06a640 msgr2=0x7f58bc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:36.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.686+0000 7f58c2088640 1 -- 172.21.15.46:0/852694142 shutdown_connections 2023-10-30T13:55:36.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.686+0000 7f58c2088640 1 -- 172.21.15.46:0/852694142 wait complete. 2023-10-30T13:55:36.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c2088640 1 Processor -- start 2023-10-30T13:55:36.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c2088640 1 -- start start 2023-10-30T13:55:36.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c2088640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58bc0cefa0 0x7f58bc147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:36.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c2088640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58bc1538f0 0x7f58bc1efba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:36.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c2088640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58bc15cd50 0x7f58bc13fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:36.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c2088640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1456a0 con 0x7f58bc1538f0 2023-10-30T13:55:36.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c2088640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc145370 con 0x7f58bc0cefa0 2023-10-30T13:55:36.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c2088640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc144b20 con 0x7f58bc15cd50 2023-10-30T13:55:36.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c0885640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58bc1538f0 0x7f58bc1efba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:36.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c0885640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58bc1538f0 0x7f58bc1efba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52456/0 (socket says 172.21.15.46:52456) 2023-10-30T13:55:36.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c0885640 1 -- 172.21.15.46:0/2981817067 learned_addr learned my addr 172.21.15.46:0/2981817067 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:36.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c1086640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58bc0cefa0 0x7f58bc147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:36.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c1887640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58bc15cd50 0x7f58bc13fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:36.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c0885640 1 -- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58bc15cd50 msgr2=0x7f58bc13fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:36.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c0885640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58bc15cd50 0x7f58bc13fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:36.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c0885640 1 -- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58bc0cefa0 msgr2=0x7f58bc147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:36.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c0885640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58bc0cefa0 0x7f58bc147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:36.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c0885640 1 -- 172.21.15.46:0/2981817067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b4009b30 con 0x7f58bc1538f0 2023-10-30T13:55:36.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c1086640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58bc0cefa0 0x7f58bc147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:36.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c0885640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58bc1538f0 0x7f58bc1efba0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00c480 tx=0x7f58ac00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:36.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58b1ffb640 1 -- 172.21.15.46:0/2981817067 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac0074e0 con 0x7f58bc1538f0 2023-10-30T13:55:36.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58b1ffb640 1 -- 172.21.15.46:0/2981817067 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac007660 con 0x7f58bc1538f0 2023-10-30T13:55:36.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58b1ffb640 1 -- 172.21.15.46:0/2981817067 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac004a30 con 0x7f58bc1538f0 2023-10-30T13:55:36.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c2088640 1 -- 172.21.15.46:0/2981817067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc0b4d90 con 0x7f58bc1538f0 2023-10-30T13:55:36.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.690+0000 7f58c2088640 1 -- 172.21.15.46:0/2981817067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc0b4a90 con 0x7f58bc1538f0 2023-10-30T13:55:36.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.694+0000 7f58c2088640 1 -- 172.21.15.46:0/2981817067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc0b52b0 con 0x7f58bc1538f0 2023-10-30T13:55:36.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.694+0000 7f58b1ffb640 1 -- 172.21.15.46:0/2981817067 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f58ac004b90 con 0x7f58bc1538f0 2023-10-30T13:55:36.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.694+0000 7f58b1ffb640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58a00664a0 0x7f58a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:36.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.694+0000 7f58b1ffb640 1 -- 172.21.15.46:0/2981817067 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f58ac0835c0 con 0x7f58bc1538f0 2023-10-30T13:55:36.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.694+0000 7f58c1086640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58a00664a0 0x7f58a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:36.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.694+0000 7f58c1086640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58a00664a0 0x7f58a0068960 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f58b8007a70 tx=0x7f58b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:36.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.698+0000 7f58b1ffb640 1 -- 172.21.15.46:0/2981817067 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f58ac053f80 con 0x7f58bc1538f0 2023-10-30T13:55:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 -- 172.21.15.46:0/2981817067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58bc04e170 con 0x7f58bc1538f0 2023-10-30T13:55:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58b1ffb640 1 -- 172.21.15.46:0/2981817067 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f58ac053b10 con 0x7f58bc1538f0 2023-10-30T13:55:36.893 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:36.893 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7518,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:17.419200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:36.893 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 144 2023-10-30T13:55:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 -- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58a00664a0 msgr2=0x7f58a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58a00664a0 0x7f58a0068960 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f58b8007a70 tx=0x7f58b8008040 comp rx=0 tx=0).stop 2023-10-30T13:55:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 -- 172.21.15.46:0/2981817067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58bc1538f0 msgr2=0x7f58bc1efba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58bc1538f0 0x7f58bc1efba0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00c480 tx=0x7f58ac00c940 comp rx=0 tx=0).stop 2023-10-30T13:55:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 -- 172.21.15.46:0/2981817067 shutdown_connections 2023-10-30T13:55:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f58a00664a0 0x7f58a0068960 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58bc15cd50 0x7f58bc13fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58bc1538f0 0x7f58bc1efba0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 --2- 172.21.15.46:0/2981817067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58bc0cefa0 0x7f58bc147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 -- 172.21.15.46:0/2981817067 >> 172.21.15.46:0/2981817067 conn(0x7f58bc06a640 msgr2=0x7f58bc0c3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 -- 172.21.15.46:0/2981817067 shutdown_connections 2023-10-30T13:55:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:36.890+0000 7f58c2088640 1 -- 172.21.15.46:0/2981817067 wait complete. 2023-10-30T13:55:36.914 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:55:38.915 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:39.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.066+0000 7fa43cd0f640 1 Processor -- start 2023-10-30T13:55:39.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.066+0000 7fa43cd0f640 1 -- start start 2023-10-30T13:55:39.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa43cd0f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa4380cb7e0 0x7fa438156680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:39.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa43cd0f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4380c68b0 con 0x7fa4380c2570 2023-10-30T13:55:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa43cd0f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa438150d00 con 0x7fa438150920 2023-10-30T13:55:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa43cd0f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4380c2e90 con 0x7fa4380cb7e0 2023-10-30T13:55:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa42ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa4380c2570 0x7fa4380cb0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55836/0 (socket says 172.21.15.46:55836) 2023-10-30T13:55:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa42ffff640 1 -- 172.21.15.46:0/3170266621 learned_addr learned my addr 172.21.15.46:0/3170266621 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:55:39.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa436d9d640 1 --2- 172.21.15.46:0/3170266621 >> v2:172.21.15.154:3300/0 conn(0x7fa4380cb7e0 0x7fa438156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:39.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa435d9b640 1 -- 172.21.15.46:0/3170266621 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953537090 0 0) 0x7fa4380c68b0 con 0x7fa4380c2570 2023-10-30T13:55:39.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa435d9b640 1 -- 172.21.15.46:0/3170266621 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa424003530 con 0x7fa4380c2570 2023-10-30T13:55:39.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa436d9d640 1 -- 172.21.15.46:0/3170266621 >> v1:172.21.15.154:6789/0 conn(0x7fa438150920 legacy=0x7fa43806d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:39.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa436d9d640 1 -- 172.21.15.46:0/3170266621 >> v1:172.21.15.46:6789/0 conn(0x7fa4380c2570 legacy=0x7fa4380cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:39.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa435d9b640 1 -- 172.21.15.46:0/3170266621 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3937717446 0 0) 0x7fa424003530 con 0x7fa4380c2570 2023-10-30T13:55:39.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.070+0000 7fa436d9d640 1 -- 172.21.15.46:0/3170266621 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4380c6b00 con 0x7fa4380cb7e0 2023-10-30T13:55:39.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa436d9d640 1 --2- 172.21.15.46:0/3170266621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4380cb7e0 0x7fa438156680 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa430009d90 tx=0x7fa43001c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d4ef4bca01037d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:39.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa435d9b640 1 -- 172.21.15.46:0/3170266621 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa420001680 con 0x7fa4380cb7e0 2023-10-30T13:55:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa435d9b640 1 -- 172.21.15.46:0/3170266621 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4300a55a0 con 0x7fa4380cb7e0 2023-10-30T13:55:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa435d9b640 1 -- 172.21.15.46:0/3170266621 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4300a5910 con 0x7fa4380cb7e0 2023-10-30T13:55:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa43cd0f640 1 -- 172.21.15.46:0/3170266621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4380cb7e0 msgr2=0x7fa438156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa43cd0f640 1 --2- 172.21.15.46:0/3170266621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4380cb7e0 0x7fa438156680 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa430009d90 tx=0x7fa43001c870 comp rx=0 tx=0).stop 2023-10-30T13:55:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa43cd0f640 1 -- 172.21.15.46:0/3170266621 shutdown_connections 2023-10-30T13:55:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa43cd0f640 1 --2- 172.21.15.46:0/3170266621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4380cb7e0 0x7fa438156680 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:39.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa43cd0f640 1 -- 172.21.15.46:0/3170266621 >> 172.21.15.46:0/3170266621 conn(0x7fa43806dfd0 msgr2=0x7fa43806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa43cd0f640 1 -- 172.21.15.46:0/3170266621 shutdown_connections 2023-10-30T13:55:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa43cd0f640 1 -- 172.21.15.46:0/3170266621 wait complete. 2023-10-30T13:55:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa43cd0f640 1 Processor -- start 2023-10-30T13:55:39.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.074+0000 7fa43cd0f640 1 -- start start 2023-10-30T13:55:39.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43cd0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4380c2570 0x7fa4381f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43cd0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4380cb7e0 0x7fa43814f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43cd0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa438150920 0x7fa438149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43cd0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4381f2910 con 0x7fa4380c2570 2023-10-30T13:55:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43cd0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa438148fd0 con 0x7fa4380cb7e0 2023-10-30T13:55:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43cd0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa438148ca0 con 0x7fa438150920 2023-10-30T13:55:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4380c2570 0x7fa4381f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4380c2570 0x7fa4381f23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52468/0 (socket says 172.21.15.46:52468) 2023-10-30T13:55:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43659c640 1 -- 172.21.15.46:0/2001198762 learned_addr learned my addr 172.21.15.46:0/2001198762 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa42ffff640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4380cb7e0 0x7fa43814f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:39.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa436d9d640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa438150920 0x7fa438149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:39.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43659c640 1 -- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa438150920 msgr2=0x7fa438149460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:39.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43659c640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa438150920 0x7fa438149460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:39.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43659c640 1 -- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4380cb7e0 msgr2=0x7fa43814f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:39.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43659c640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4380cb7e0 0x7fa43814f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:39.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43659c640 1 -- 172.21.15.46:0/2001198762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa430009a70 con 0x7fa4380c2570 2023-10-30T13:55:39.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa436d9d640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa438150920 0x7fa438149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:39.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43659c640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4380c2570 0x7fa4381f23d0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fa42800c3c0 tx=0x7fa42800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:39.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa42effd640 1 -- 172.21.15.46:0/2001198762 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4280052d0 con 0x7fa4380c2570 2023-10-30T13:55:39.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa42effd640 1 -- 172.21.15.46:0/2001198762 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa428005430 con 0x7fa4380c2570 2023-10-30T13:55:39.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa42effd640 1 -- 172.21.15.46:0/2001198762 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa428003840 con 0x7fa4380c2570 2023-10-30T13:55:39.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43cd0f640 1 -- 172.21.15.46:0/2001198762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa438148310 con 0x7fa4380c2570 2023-10-30T13:55:39.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43cd0f640 1 -- 172.21.15.46:0/2001198762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa438147d50 con 0x7fa4380c2570 2023-10-30T13:55:39.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa43cd0f640 1 -- 172.21.15.46:0/2001198762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa43804e170 con 0x7fa4380c2570 2023-10-30T13:55:39.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.078+0000 7fa42effd640 1 -- 172.21.15.46:0/2001198762 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4280039a0 con 0x7fa4380c2570 2023-10-30T13:55:39.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.082+0000 7fa42effd640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4180664a0 0x7fa418068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:39.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.082+0000 7fa42effd640 1 -- 172.21.15.46:0/2001198762 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7fa428083910 con 0x7fa4380c2570 2023-10-30T13:55:39.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.082+0000 7fa42ffff640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4180664a0 0x7fa418068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:39.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.082+0000 7fa42ffff640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4180664a0 0x7fa418068960 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa420001ad0 tx=0x7fa420001a60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:39.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.086+0000 7fa42effd640 1 -- 172.21.15.46:0/2001198762 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa428050840 con 0x7fa4380c2570 2023-10-30T13:55:39.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.278+0000 7fa43cd0f640 1 -- 172.21.15.46:0/2001198762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4380b1b20 con 0x7fa4380c2570 2023-10-30T13:55:39.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.278+0000 7fa42effd640 1 -- 172.21.15.46:0/2001198762 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+6520 (secure 0 0 0) 0x7fa428053e60 con 0x7fa4380c2570 2023-10-30T13:55:39.283 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:39.283 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7518,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:17.419200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:39.283 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 144 2023-10-30T13:55:39.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.282+0000 7fa43cd0f640 1 -- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4180664a0 msgr2=0x7fa418068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:39.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.282+0000 7fa43cd0f640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4180664a0 0x7fa418068960 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa420001ad0 tx=0x7fa420001a60 comp rx=0 tx=0).stop 2023-10-30T13:55:39.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.282+0000 7fa43cd0f640 1 -- 172.21.15.46:0/2001198762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4380c2570 msgr2=0x7fa4381f23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:39.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.282+0000 7fa43cd0f640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4380c2570 0x7fa4381f23d0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fa42800c3c0 tx=0x7fa42800c880 comp rx=0 tx=0).stop 2023-10-30T13:55:39.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.282+0000 7fa43cd0f640 1 -- 172.21.15.46:0/2001198762 shutdown_connections 2023-10-30T13:55:39.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.282+0000 7fa43cd0f640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4180664a0 0x7fa418068960 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:39.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.282+0000 7fa43cd0f640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa438150920 0x7fa438149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:39.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.282+0000 7fa43cd0f640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4380cb7e0 0x7fa43814f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:39.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.282+0000 7fa43cd0f640 1 --2- 172.21.15.46:0/2001198762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4380c2570 0x7fa4381f23d0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:39.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.282+0000 7fa43cd0f640 1 -- 172.21.15.46:0/2001198762 >> 172.21.15.46:0/2001198762 conn(0x7fa43806dfd0 msgr2=0x7fa438070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:39.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.282+0000 7fa43cd0f640 1 -- 172.21.15.46:0/2001198762 shutdown_connections 2023-10-30T13:55:39.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:39.282+0000 7fa43cd0f640 1 -- 172.21.15.46:0/2001198762 wait complete. 2023-10-30T13:55:39.305 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:55:41.306 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:41.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.458+0000 7f29208c9640 1 Processor -- start 2023-10-30T13:55:41.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.458+0000 7f29208c9640 1 -- start start 2023-10-30T13:55:41.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.458+0000 7f29208c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f291c1539a0 0x7f291c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:41.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.458+0000 7f29208c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291c15a8c0 con 0x7f291c069e90 2023-10-30T13:55:41.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.458+0000 7f29208c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291c153d80 con 0x7f291c070a00 2023-10-30T13:55:41.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.458+0000 7f29208c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291c06fbb0 con 0x7f291c1539a0 2023-10-30T13:55:41.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.458+0000 7f291a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f291c1539a0 0x7f291c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:41.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.458+0000 7f291ad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f291c069e90 0x7f291c06a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55842/0 (socket says 172.21.15.46:55842) 2023-10-30T13:55:41.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.458+0000 7f291ad9d640 1 -- 172.21.15.46:0/3736278934 learned_addr learned my addr 172.21.15.46:0/3736278934 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:55:41.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.462+0000 7f291a59c640 1 -- 172.21.15.46:0/3736278934 >> v1:172.21.15.154:6789/0 conn(0x7f291c070a00 legacy=0x7f291c070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:41.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.462+0000 7f291a59c640 1 -- 172.21.15.46:0/3736278934 >> v1:172.21.15.46:6789/0 conn(0x7f291c069e90 legacy=0x7f291c06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:41.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.462+0000 7f2919d9b640 1 -- 172.21.15.46:0/3736278934 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305031907 0 0) 0x7f291c15a8c0 con 0x7f291c069e90 2023-10-30T13:55:41.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.462+0000 7f291a59c640 1 -- 172.21.15.46:0/3736278934 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c0c5c50 con 0x7f291c1539a0 2023-10-30T13:55:41.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.462+0000 7f291a59c640 1 --2- 172.21.15.46:0/3736278934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f291c1539a0 0x7f291c15a380 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f290c0097c0 tx=0x7f290c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884048dbc7dfb940 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:41.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.462+0000 7f2919d9b640 1 -- 172.21.15.46:0/3736278934 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c0ac620 con 0x7f291c1539a0 2023-10-30T13:55:41.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.462+0000 7f2919d9b640 1 -- 172.21.15.46:0/3736278934 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c0ac780 con 0x7f291c1539a0 2023-10-30T13:55:41.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.462+0000 7f2919d9b640 1 -- 172.21.15.46:0/3736278934 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c0acb10 con 0x7f291c1539a0 2023-10-30T13:55:41.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.462+0000 7f29208c9640 1 -- 172.21.15.46:0/3736278934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f291c1539a0 msgr2=0x7f291c15a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:41.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.462+0000 7f29208c9640 1 --2- 172.21.15.46:0/3736278934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f291c1539a0 0x7f291c15a380 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f290c0097c0 tx=0x7f290c01c3c0 comp rx=0 tx=0).stop 2023-10-30T13:55:41.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 -- 172.21.15.46:0/3736278934 shutdown_connections 2023-10-30T13:55:41.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 --2- 172.21.15.46:0/3736278934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f291c1539a0 0x7f291c15a380 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:41.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 -- 172.21.15.46:0/3736278934 >> 172.21.15.46:0/3736278934 conn(0x7f291c0c7910 msgr2=0x7f291c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:41.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 -- 172.21.15.46:0/3736278934 shutdown_connections 2023-10-30T13:55:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 -- 172.21.15.46:0/3736278934 wait complete. 2023-10-30T13:55:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 Processor -- start 2023-10-30T13:55:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 -- start start 2023-10-30T13:55:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f291c069e90 0x7f291c159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f291c070a00 0x7f291c144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f291c1539a0 0x7f291c157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c155cc0 con 0x7f291c1539a0 2023-10-30T13:55:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c157770 con 0x7f291c070a00 2023-10-30T13:55:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f29208c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291c155f10 con 0x7f291c069e90 2023-10-30T13:55:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f291ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f291c1539a0 0x7f291c157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f291ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f291c1539a0 0x7f291c157ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52500/0 (socket says 172.21.15.46:52500) 2023-10-30T13:55:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f291ad9d640 1 -- 172.21.15.46:0/3106008751 learned_addr learned my addr 172.21.15.46:0/3106008751 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f291a59c640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f291c069e90 0x7f291c159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f2913fff640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f291c070a00 0x7f291c144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:41.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f291ad9d640 1 -- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f291c069e90 msgr2=0x7f291c159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:41.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f291ad9d640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f291c069e90 0x7f291c159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:41.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f291ad9d640 1 -- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f291c070a00 msgr2=0x7f291c144c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:41.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f291ad9d640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f291c070a00 0x7f291c144c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:41.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.466+0000 7f291ad9d640 1 -- 172.21.15.46:0/3106008751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c009420 con 0x7f291c1539a0 2023-10-30T13:55:41.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.470+0000 7f291ad9d640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f291c1539a0 0x7f291c157ee0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f2914001570 tx=0x7f291400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:41.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.470+0000 7f2912ffd640 1 -- 172.21.15.46:0/3106008751 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291400a640 con 0x7f291c1539a0 2023-10-30T13:55:41.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.470+0000 7f2912ffd640 1 -- 172.21.15.46:0/3106008751 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291400a7c0 con 0x7f291c1539a0 2023-10-30T13:55:41.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.470+0000 7f2912ffd640 1 -- 172.21.15.46:0/3106008751 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291400ab70 con 0x7f291c1539a0 2023-10-30T13:55:41.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.470+0000 7f29208c9640 1 -- 172.21.15.46:0/3106008751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291c1561f0 con 0x7f291c1539a0 2023-10-30T13:55:41.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.470+0000 7f29208c9640 1 -- 172.21.15.46:0/3106008751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c1579c0 con 0x7f291c1539a0 2023-10-30T13:55:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.470+0000 7f29208c9640 1 -- 172.21.15.46:0/3106008751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c04e170 con 0x7f291c1539a0 2023-10-30T13:55:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.470+0000 7f2912ffd640 1 -- 172.21.15.46:0/3106008751 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f291400acd0 con 0x7f291c1539a0 2023-10-30T13:55:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.470+0000 7f2912ffd640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28fc0664a0 0x7f28fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.470+0000 7f2912ffd640 1 -- 172.21.15.46:0/3106008751 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f2914083a50 con 0x7f291c1539a0 2023-10-30T13:55:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.470+0000 7f291a59c640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28fc0664a0 0x7f28fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.474+0000 7f291a59c640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28fc0664a0 0x7f28fc068960 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f290c0ae000 tx=0x7f290c0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:41.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.474+0000 7f2912ffd640 1 -- 172.21.15.46:0/3106008751 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29140508f0 con 0x7f291c1539a0 2023-10-30T13:55:41.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.666+0000 7f29208c9640 1 -- 172.21.15.46:0/3106008751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f291c0c72f0 con 0x7f291c1539a0 2023-10-30T13:55:41.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.666+0000 7f2912ffd640 1 -- 172.21.15.46:0/3106008751 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f2914018020 con 0x7f291c1539a0 2023-10-30T13:55:41.669 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:41.670 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7518,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:17.419200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:41.670 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 144 2023-10-30T13:55:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.670+0000 7f29208c9640 1 -- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28fc0664a0 msgr2=0x7f28fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.670+0000 7f29208c9640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28fc0664a0 0x7f28fc068960 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f290c0ae000 tx=0x7f290c0a7000 comp rx=0 tx=0).stop 2023-10-30T13:55:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.670+0000 7f29208c9640 1 -- 172.21.15.46:0/3106008751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f291c1539a0 msgr2=0x7f291c157ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.670+0000 7f29208c9640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f291c1539a0 0x7f291c157ee0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f2914001570 tx=0x7f291400cca0 comp rx=0 tx=0).stop 2023-10-30T13:55:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.670+0000 7f29208c9640 1 -- 172.21.15.46:0/3106008751 shutdown_connections 2023-10-30T13:55:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.670+0000 7f29208c9640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28fc0664a0 0x7f28fc068960 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.670+0000 7f29208c9640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f291c1539a0 0x7f291c157ee0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.670+0000 7f29208c9640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f291c070a00 0x7f291c144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.670+0000 7f29208c9640 1 --2- 172.21.15.46:0/3106008751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f291c069e90 0x7f291c159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.670+0000 7f29208c9640 1 -- 172.21.15.46:0/3106008751 >> 172.21.15.46:0/3106008751 conn(0x7f291c0c7910 msgr2=0x7f291c14e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.670+0000 7f29208c9640 1 -- 172.21.15.46:0/3106008751 shutdown_connections 2023-10-30T13:55:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:41.670+0000 7f29208c9640 1 -- 172.21.15.46:0/3106008751 wait complete. 2023-10-30T13:55:41.691 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:55:43.691 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:43.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6db4f3e640 1 Processor -- start 2023-10-30T13:55:43.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6db4f3e640 1 -- start start 2023-10-30T13:55:43.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6db4f3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6db00cf160 0x7f6db00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6db4f3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db00cb800 con 0x7f6db01538a0 2023-10-30T13:55:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6db4f3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db00cb330 con 0x7f6db015d6f0 2023-10-30T13:55:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6db4f3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db00cad70 con 0x7f6db00cf160 2023-10-30T13:55:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6dae59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6db01538a0 0x7f6db00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55850/0 (socket says 172.21.15.46:55850) 2023-10-30T13:55:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6dae59c640 1 -- 172.21.15.46:0/2858980350 learned_addr learned my addr 172.21.15.46:0/2858980350 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:55:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6daed9d640 1 --2- 172.21.15.46:0/2858980350 >> v2:172.21.15.46:3300/0 conn(0x7f6db00cf160 0x7f6db00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6daed9d640 1 -- 172.21.15.46:0/2858980350 >> v1:172.21.15.154:6789/0 conn(0x7f6db015d6f0 legacy=0x7f6db015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6daed9d640 1 -- 172.21.15.46:0/2858980350 >> v1:172.21.15.46:6789/0 conn(0x7f6db01538a0 legacy=0x7f6db00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6dadd9b640 1 -- 172.21.15.46:0/2858980350 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091716810 0 0) 0x7f6db00cb800 con 0x7f6db01538a0 2023-10-30T13:55:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.846+0000 7f6daed9d640 1 -- 172.21.15.46:0/2858980350 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db00cb050 con 0x7f6db00cf160 2023-10-30T13:55:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.850+0000 7f6daed9d640 1 --2- 172.21.15.46:0/2858980350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db00cf160 0x7f6db00cf590 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f6da8009d90 tx=0x7f6da801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7797c2f73c704f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.850+0000 7f6dadd9b640 1 -- 172.21.15.46:0/2858980350 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da80af600 con 0x7f6db00cf160 2023-10-30T13:55:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.850+0000 7f6dadd9b640 1 -- 172.21.15.46:0/2858980350 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da80ab050 con 0x7f6db00cf160 2023-10-30T13:55:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.850+0000 7f6dadd9b640 1 -- 172.21.15.46:0/2858980350 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da80af980 con 0x7f6db00cf160 2023-10-30T13:55:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.850+0000 7f6db4f3e640 1 -- 172.21.15.46:0/2858980350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db00cf160 msgr2=0x7f6db00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.850+0000 7f6db4f3e640 1 --2- 172.21.15.46:0/2858980350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db00cf160 0x7f6db00cf590 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f6da8009d90 tx=0x7f6da801f130 comp rx=0 tx=0).stop 2023-10-30T13:55:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.850+0000 7f6db4f3e640 1 -- 172.21.15.46:0/2858980350 shutdown_connections 2023-10-30T13:55:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.850+0000 7f6db4f3e640 1 --2- 172.21.15.46:0/2858980350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db00cf160 0x7f6db00cf590 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.850+0000 7f6db4f3e640 1 -- 172.21.15.46:0/2858980350 >> 172.21.15.46:0/2858980350 conn(0x7f6db006bfe0 msgr2=0x7f6db006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.850+0000 7f6db4f3e640 1 -- 172.21.15.46:0/2858980350 shutdown_connections 2023-10-30T13:55:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.850+0000 7f6db4f3e640 1 -- 172.21.15.46:0/2858980350 wait complete. 2023-10-30T13:55:43.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6db4f3e640 1 Processor -- start 2023-10-30T13:55:43.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6db4f3e640 1 -- start start 2023-10-30T13:55:43.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6db4f3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6db00cf160 0x7f6db015bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:43.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6db4f3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6db01538a0 0x7f6db0140cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:43.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6db4f3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db015d6f0 0x7f6db0141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:43.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6db4f3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00ceb40 con 0x7f6db015d6f0 2023-10-30T13:55:43.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6db4f3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db0159210 con 0x7f6db01538a0 2023-10-30T13:55:43.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6db4f3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db01594f0 con 0x7f6db00cf160 2023-10-30T13:55:43.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6daed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db015d6f0 0x7f6db0141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:43.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6daed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db015d6f0 0x7f6db0141210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52524/0 (socket says 172.21.15.46:52524) 2023-10-30T13:55:43.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6daed9d640 1 -- 172.21.15.46:0/428123028 learned_addr learned my addr 172.21.15.46:0/428123028 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:43.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6da7fff640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6db01538a0 0x7f6db0140cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:43.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6dae59c640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6db00cf160 0x7f6db015bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:43.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6daed9d640 1 -- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6db00cf160 msgr2=0x7f6db015bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:43.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6daed9d640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6db00cf160 0x7f6db015bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:43.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6daed9d640 1 -- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6db01538a0 msgr2=0x7f6db0140cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:43.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6daed9d640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6db01538a0 0x7f6db0140cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:43.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6daed9d640 1 -- 172.21.15.46:0/428123028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da8009a70 con 0x7f6db015d6f0 2023-10-30T13:55:43.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6da7fff640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6db01538a0 0x7f6db0140cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:43.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6daed9d640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db015d6f0 0x7f6db0141210 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f6da80b2040 tx=0x7f6da801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:43.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6da6ffd640 1 -- 172.21.15.46:0/428123028 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da80b9410 con 0x7f6db015d6f0 2023-10-30T13:55:43.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6da6ffd640 1 -- 172.21.15.46:0/428123028 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da80b9570 con 0x7f6db015d6f0 2023-10-30T13:55:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6da6ffd640 1 -- 172.21.15.46:0/428123028 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da80b9920 con 0x7f6db015d6f0 2023-10-30T13:55:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6db4f3e640 1 -- 172.21.15.46:0/428123028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db00ced70 con 0x7f6db015d6f0 2023-10-30T13:55:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.854+0000 7f6db4f3e640 1 -- 172.21.15.46:0/428123028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db01464e0 con 0x7f6db015d6f0 2023-10-30T13:55:43.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.858+0000 7f6db4f3e640 1 -- 172.21.15.46:0/428123028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db004e170 con 0x7f6db015d6f0 2023-10-30T13:55:43.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.858+0000 7f6da6ffd640 1 -- 172.21.15.46:0/428123028 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6da80b6050 con 0x7f6db015d6f0 2023-10-30T13:55:43.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.858+0000 7f6da6ffd640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d900664d0 0x7f6d90068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:43.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.858+0000 7f6da6ffd640 1 -- 172.21.15.46:0/428123028 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f6da8122b60 con 0x7f6db015d6f0 2023-10-30T13:55:43.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.858+0000 7f6dae59c640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d900664d0 0x7f6d90068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:43.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.858+0000 7f6dae59c640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d900664d0 0x7f6d90068990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6da0007fd0 tx=0x7f6da0007ba0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:43.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:43.862+0000 7f6da6ffd640 1 -- 172.21.15.46:0/428123028 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6da80efaa0 con 0x7f6db015d6f0 2023-10-30T13:55:44.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.050+0000 7f6db4f3e640 1 -- 172.21.15.46:0/428123028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6db0159630 con 0x7f6db015d6f0 2023-10-30T13:55:44.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.050+0000 7f6da6ffd640 1 -- 172.21.15.46:0/428123028 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f6da80f30c0 con 0x7f6db015d6f0 2023-10-30T13:55:44.053 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:44.054 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7518,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:17.419200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7188},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7188":{"gid":7188,"name":"c","rank":1,"incarnation":132,"state":"up:active","state_seq":22,"addr":"172.21.15.46:6836/711135361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":711135361},{"type":"v1","addr":"172.21.15.46:6836","nonce":711135361}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:44.054 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 144 2023-10-30T13:55:44.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.050+0000 7f6db4f3e640 1 -- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d900664d0 msgr2=0x7f6d90068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:44.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.050+0000 7f6db4f3e640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d900664d0 0x7f6d90068990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6da0007fd0 tx=0x7f6da0007ba0 comp rx=0 tx=0).stop 2023-10-30T13:55:44.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.050+0000 7f6db4f3e640 1 -- 172.21.15.46:0/428123028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db015d6f0 msgr2=0x7f6db0141210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:44.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.054+0000 7f6db4f3e640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db015d6f0 0x7f6db0141210 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f6da80b2040 tx=0x7f6da801bed0 comp rx=0 tx=0).stop 2023-10-30T13:55:44.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.054+0000 7f6db4f3e640 1 -- 172.21.15.46:0/428123028 shutdown_connections 2023-10-30T13:55:44.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.054+0000 7f6db4f3e640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d900664d0 0x7f6d90068990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:44.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.054+0000 7f6db4f3e640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db015d6f0 0x7f6db0141210 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:44.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.054+0000 7f6db4f3e640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6db01538a0 0x7f6db0140cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:44.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.054+0000 7f6db4f3e640 1 --2- 172.21.15.46:0/428123028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6db00cf160 0x7f6db015bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:44.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.054+0000 7f6db4f3e640 1 -- 172.21.15.46:0/428123028 >> 172.21.15.46:0/428123028 conn(0x7f6db006bfe0 msgr2=0x7f6db0151b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:44.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.054+0000 7f6db4f3e640 1 -- 172.21.15.46:0/428123028 shutdown_connections 2023-10-30T13:55:44.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:44.054+0000 7f6db4f3e640 1 -- 172.21.15.46:0/428123028 wait complete. 2023-10-30T13:55:44.074 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:55:46.076 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:46.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac44668640 1 Processor -- start 2023-10-30T13:55:46.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac44668640 1 -- start start 2023-10-30T13:55:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac44668640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fac3c1538a0 0x7fac3c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac44668640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac3c0ca0e0 0x7fac3c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac44668640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3c0cb800 con 0x7fac3c0cf160 2023-10-30T13:55:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac44668640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac3c15dc30 con 0x7fac3c0ca0e0 2023-10-30T13:55:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac44668640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac3c0cb330 con 0x7fac3c1538a0 2023-10-30T13:55:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac42c05640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fac3c0cf160 0x7fac3c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55858/0 (socket says 172.21.15.46:55858) 2023-10-30T13:55:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac42c05640 1 -- 172.21.15.46:0/3495991874 learned_addr learned my addr 172.21.15.46:0/3495991874 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:55:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac41c03640 1 --2- 172.21.15.46:0/3495991874 >> v2:172.21.15.46:3300/0 conn(0x7fac3c0ca0e0 0x7fac3c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac42404640 1 --2- 172.21.15.46:0/3495991874 >> v2:172.21.15.154:3300/0 conn(0x7fac3c1538a0 0x7fac3c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac41402640 1 -- 172.21.15.46:0/3495991874 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936190294 0 0) 0x7fac3c0cb800 con 0x7fac3c0cf160 2023-10-30T13:55:46.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac41402640 1 -- 172.21.15.46:0/3495991874 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac28003530 con 0x7fac3c0cf160 2023-10-30T13:55:46.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac41c03640 1 -- 172.21.15.46:0/3495991874 >> v2:172.21.15.154:3300/0 conn(0x7fac3c1538a0 msgr2=0x7fac3c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:46.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac41c03640 1 --2- 172.21.15.46:0/3495991874 >> v2:172.21.15.154:3300/0 conn(0x7fac3c1538a0 0x7fac3c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac41c03640 1 -- 172.21.15.46:0/3495991874 >> v1:172.21.15.46:6789/0 conn(0x7fac3c0cf160 legacy=0x7fac3c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.226+0000 7fac41c03640 1 -- 172.21.15.46:0/3495991874 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c0cad70 con 0x7fac3c0ca0e0 2023-10-30T13:55:46.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.230+0000 7fac41c03640 1 --2- 172.21.15.46:0/3495991874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac3c0ca0e0 0x7fac3c15a090 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0cfd10 tx=0x7fac2c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ead731d0028686 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:46.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.230+0000 7fac41402640 1 -- 172.21.15.46:0/3495991874 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0a8950 con 0x7fac3c0ca0e0 2023-10-30T13:55:46.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.230+0000 7fac41402640 1 -- 172.21.15.46:0/3495991874 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c0a8ab0 con 0x7fac3c0ca0e0 2023-10-30T13:55:46.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.230+0000 7fac41402640 1 -- 172.21.15.46:0/3495991874 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0a8e40 con 0x7fac3c0ca0e0 2023-10-30T13:55:46.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.230+0000 7fac44668640 1 -- 172.21.15.46:0/3495991874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac3c0ca0e0 msgr2=0x7fac3c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.230+0000 7fac44668640 1 --2- 172.21.15.46:0/3495991874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac3c0ca0e0 0x7fac3c15a090 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0cfd10 tx=0x7fac2c01ec50 comp rx=0 tx=0).stop 2023-10-30T13:55:46.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.230+0000 7fac44668640 1 -- 172.21.15.46:0/3495991874 shutdown_connections 2023-10-30T13:55:46.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.230+0000 7fac44668640 1 --2- 172.21.15.46:0/3495991874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac3c0ca0e0 0x7fac3c15a090 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.230+0000 7fac44668640 1 --2- 172.21.15.46:0/3495991874 >> v2:172.21.15.154:3300/0 conn(0x7fac3c1538a0 0x7fac3c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.230+0000 7fac44668640 1 -- 172.21.15.46:0/3495991874 >> 172.21.15.46:0/3495991874 conn(0x7fac3c06bfe0 msgr2=0x7fac3c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:46.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.230+0000 7fac44668640 1 -- 172.21.15.46:0/3495991874 shutdown_connections 2023-10-30T13:55:46.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.230+0000 7fac44668640 1 -- 172.21.15.46:0/3495991874 wait complete. 2023-10-30T13:55:46.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac44668640 1 Processor -- start 2023-10-30T13:55:46.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac44668640 1 -- start start 2023-10-30T13:55:46.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac44668640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac3c0ca0e0 0x7fac3c0ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:46.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac44668640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac3c0cf160 0x7fac3c0ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:46.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac44668640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac3c1538a0 0x7fac3c157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:46.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac44668640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c1e4af0 con 0x7fac3c0cf160 2023-10-30T13:55:46.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac44668640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c15da80 con 0x7fac3c0ca0e0 2023-10-30T13:55:46.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac44668640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c1e5910 con 0x7fac3c1538a0 2023-10-30T13:55:46.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac41c03640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac3c0cf160 0x7fac3c0ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:46.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac41c03640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac3c0cf160 0x7fac3c0ce540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52538/0 (socket says 172.21.15.46:52538) 2023-10-30T13:55:46.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac41c03640 1 -- 172.21.15.46:0/71574479 learned_addr learned my addr 172.21.15.46:0/71574479 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:46.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac42c05640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac3c1538a0 0x7fac3c157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:46.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac42404640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac3c0ca0e0 0x7fac3c0ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:46.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac41c03640 1 -- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac3c1538a0 msgr2=0x7fac3c157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac41c03640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac3c1538a0 0x7fac3c157860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac41c03640 1 -- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac3c0ca0e0 msgr2=0x7fac3c0ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac41c03640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac3c0ca0e0 0x7fac3c0ce000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac41c03640 1 -- 172.21.15.46:0/71574479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c009420 con 0x7fac3c0cf160 2023-10-30T13:55:46.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac41c03640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac3c0cf160 0x7fac3c0ce540 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0b3040 tx=0x7fac2c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:46.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac32ffd640 1 -- 172.21.15.46:0/71574479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0b0d20 con 0x7fac3c0cf160 2023-10-30T13:55:46.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac32ffd640 1 -- 172.21.15.46:0/71574479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c0b9430 con 0x7fac3c0cf160 2023-10-30T13:55:46.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac32ffd640 1 -- 172.21.15.46:0/71574479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0b0ea0 con 0x7fac3c0cf160 2023-10-30T13:55:46.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac44668640 1 -- 172.21.15.46:0/71574479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac3c1e4550 con 0x7fac3c0cf160 2023-10-30T13:55:46.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.234+0000 7fac44668640 1 -- 172.21.15.46:0/71574479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac3c1e2c80 con 0x7fac3c0cf160 2023-10-30T13:55:46.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.238+0000 7fac44668640 1 -- 172.21.15.46:0/71574479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3c0b2a50 con 0x7fac3c0cf160 2023-10-30T13:55:46.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.238+0000 7fac32ffd640 1 -- 172.21.15.46:0/71574479 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fac2c0b7040 con 0x7fac3c0cf160 2023-10-30T13:55:46.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.238+0000 7fac32ffd640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fac200664d0 0x7fac20068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.238+0000 7fac32ffd640 1 -- 172.21.15.46:0/71574479 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fac2c121ff0 con 0x7fac3c0cf160 2023-10-30T13:55:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.238+0000 7fac42404640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fac200664d0 0x7fac20068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.238+0000 7fac42404640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fac200664d0 0x7fac20068990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fac38009a50 tx=0x7fac38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.242+0000 7fac32ffd640 1 -- 172.21.15.46:0/71574479 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fac2c0eeda0 con 0x7fac3c0cf160 2023-10-30T13:55:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.434+0000 7fac44668640 1 -- 172.21.15.46:0/71574479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac3c1e2ae0 con 0x7fac3c0cf160 2023-10-30T13:55:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.434+0000 7fac32ffd640 1 -- 172.21.15.46:0/71574479 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fac3c1e2ae0 con 0x7fac3c0cf160 2023-10-30T13:55:46.438 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:46.438 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:44.283045+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 145 2023-10-30T13:55:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.438+0000 7fac44668640 1 -- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fac200664d0 msgr2=0x7fac20068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.438+0000 7fac44668640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fac200664d0 0x7fac20068990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fac38009a50 tx=0x7fac38008040 comp rx=0 tx=0).stop 2023-10-30T13:55:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.438+0000 7fac44668640 1 -- 172.21.15.46:0/71574479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac3c0cf160 msgr2=0x7fac3c0ce540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.438+0000 7fac44668640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac3c0cf160 0x7fac3c0ce540 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0b3040 tx=0x7fac2c01b9f0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.438+0000 7fac44668640 1 -- 172.21.15.46:0/71574479 shutdown_connections 2023-10-30T13:55:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.438+0000 7fac44668640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fac200664d0 0x7fac20068990 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.438+0000 7fac44668640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac3c1538a0 0x7fac3c157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.438+0000 7fac44668640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac3c0cf160 0x7fac3c0ce540 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.438+0000 7fac44668640 1 --2- 172.21.15.46:0/71574479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac3c0ca0e0 0x7fac3c0ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.438+0000 7fac44668640 1 -- 172.21.15.46:0/71574479 >> 172.21.15.46:0/71574479 conn(0x7fac3c06bfe0 msgr2=0x7fac3c148c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:46.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.438+0000 7fac44668640 1 -- 172.21.15.46:0/71574479 shutdown_connections 2023-10-30T13:55:46.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.438+0000 7fac44668640 1 -- 172.21.15.46:0/71574479 wait complete. 2023-10-30T13:55:46.459 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:55:46.460 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:55:46.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:46.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed13e17640 1 Processor -- start 2023-10-30T13:55:46.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed13e17640 1 -- start start 2023-10-30T13:55:46.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed13e17640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fed0c0c6b80 0x7fed0c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:46.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed13e17640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c0c0a50 con 0x7fed0c13f4a0 2023-10-30T13:55:46.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed13e17640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c06f6e0 con 0x7fed0c0c6f60 2023-10-30T13:55:46.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed13e17640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0c06f9c0 con 0x7fed0c0c6b80 2023-10-30T13:55:46.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed11bb3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fed0c0c6b80 0x7fed0c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:46.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed11bb3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fed0c0c6b80 0x7fed0c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55738/0 (socket says 172.21.15.46:55738) 2023-10-30T13:55:46.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed11bb3640 1 -- 172.21.15.46:0/3464482751 learned_addr learned my addr 172.21.15.46:0/3464482751 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed11bb3640 1 -- 172.21.15.46:0/3464482751 >> v1:172.21.15.154:6789/0 conn(0x7fed0c0c6f60 legacy=0x7fed0c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed10bb1640 1 -- 172.21.15.46:0/3464482751 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527450749 0 0) 0x7fed0c0c0a50 con 0x7fed0c13f4a0 2023-10-30T13:55:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed11bb3640 1 -- 172.21.15.46:0/3464482751 >> v1:172.21.15.46:6789/0 conn(0x7fed0c13f4a0 legacy=0x7fed0c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.602+0000 7fed11bb3640 1 -- 172.21.15.46:0/3464482751 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c06fca0 con 0x7fed0c0c6b80 2023-10-30T13:55:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.606+0000 7fed11bb3640 1 --2- 172.21.15.46:0/3464482751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0c0c6b80 0x7fed0c0bfec0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fed08009870 tx=0x7fed0801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3ef88c8d661a8a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.606+0000 7fed10bb1640 1 -- 172.21.15.46:0/3464482751 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed080ab650 con 0x7fed0c0c6b80 2023-10-30T13:55:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.606+0000 7fed10bb1640 1 -- 172.21.15.46:0/3464482751 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed080ab7b0 con 0x7fed0c0c6b80 2023-10-30T13:55:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.606+0000 7fed10bb1640 1 -- 172.21.15.46:0/3464482751 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed080abb40 con 0x7fed0c0c6b80 2023-10-30T13:55:46.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.606+0000 7fed13e17640 1 -- 172.21.15.46:0/3464482751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0c0c6b80 msgr2=0x7fed0c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.606+0000 7fed13e17640 1 --2- 172.21.15.46:0/3464482751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0c0c6b80 0x7fed0c0bfec0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fed08009870 tx=0x7fed0801c540 comp rx=0 tx=0).stop 2023-10-30T13:55:46.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.606+0000 7fed13e17640 1 -- 172.21.15.46:0/3464482751 shutdown_connections 2023-10-30T13:55:46.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.606+0000 7fed13e17640 1 --2- 172.21.15.46:0/3464482751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0c0c6b80 0x7fed0c0bfec0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.606+0000 7fed13e17640 1 -- 172.21.15.46:0/3464482751 >> 172.21.15.46:0/3464482751 conn(0x7fed0c0c7910 msgr2=0x7fed0c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:46.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.606+0000 7fed13e17640 1 -- 172.21.15.46:0/3464482751 shutdown_connections 2023-10-30T13:55:46.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.606+0000 7fed13e17640 1 -- 172.21.15.46:0/3464482751 wait complete. 2023-10-30T13:55:46.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed13e17640 1 Processor -- start 2023-10-30T13:55:46.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed13e17640 1 -- start start 2023-10-30T13:55:46.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed13e17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed0c0c6b80 0x7fed0c158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:46.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed13e17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed0c0c6f60 0x7fed0c1f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:46.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed13e17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0c13f4a0 0x7fed0c1f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:46.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed13e17640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0c9ee0 con 0x7fed0c13f4a0 2023-10-30T13:55:46.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed13e17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c13be50 con 0x7fed0c0c6f60 2023-10-30T13:55:46.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed13e17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1f5a90 con 0x7fed0c0c6b80 2023-10-30T13:55:46.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed123b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0c13f4a0 0x7fed0c1f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:46.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed11bb3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed0c0c6b80 0x7fed0c158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:46.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed123b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0c13f4a0 0x7fed0c1f64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55744/0 (socket says 172.21.15.46:55744) 2023-10-30T13:55:46.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed123b4640 1 -- 172.21.15.46:0/2092530059 learned_addr learned my addr 172.21.15.46:0/2092530059 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:46.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed113b2640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed0c0c6f60 0x7fed0c1f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:46.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed123b4640 1 -- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed0c0c6b80 msgr2=0x7fed0c158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed123b4640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed0c0c6b80 0x7fed0c158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed123b4640 1 -- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed0c0c6f60 msgr2=0x7fed0c1f5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed123b4640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed0c0c6f60 0x7fed0c1f5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed123b4640 1 -- 172.21.15.46:0/2092530059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed080094d0 con 0x7fed0c13f4a0 2023-10-30T13:55:46.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed123b4640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0c13f4a0 0x7fed0c1f64d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fed04002240 tx=0x7fed0400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:46.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed027fc640 1 -- 172.21.15.46:0/2092530059 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04003db0 con 0x7fed0c13f4a0 2023-10-30T13:55:46.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed027fc640 1 -- 172.21.15.46:0/2092530059 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0400ce30 con 0x7fed0c13f4a0 2023-10-30T13:55:46.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed13e17640 1 -- 172.21.15.46:0/2092530059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c13ca60 con 0x7fed0c13f4a0 2023-10-30T13:55:46.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed027fc640 1 -- 172.21.15.46:0/2092530059 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04005270 con 0x7fed0c13f4a0 2023-10-30T13:55:46.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed13e17640 1 -- 172.21.15.46:0/2092530059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c13c3d0 con 0x7fed0c13f4a0 2023-10-30T13:55:46.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.610+0000 7fed13e17640 1 -- 172.21.15.46:0/2092530059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0c04e170 con 0x7fed0c13f4a0 2023-10-30T13:55:46.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.614+0000 7fed027fc640 1 -- 172.21.15.46:0/2092530059 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed040053d0 con 0x7fed0c13f4a0 2023-10-30T13:55:46.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.614+0000 7fed027fc640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecf00664a0 0x7fecf0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:46.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.614+0000 7fed027fc640 1 -- 172.21.15.46:0/2092530059 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fed04083bf0 con 0x7fed0c13f4a0 2023-10-30T13:55:46.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.614+0000 7fed11bb3640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecf00664a0 0x7fecf0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:46.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.614+0000 7fed11bb3640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecf00664a0 0x7fecf0068960 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fed080af000 tx=0x7fed080b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:46.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.618+0000 7fed027fc640 1 -- 172.21.15.46:0/2092530059 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fed04050a30 con 0x7fed0c13f4a0 2023-10-30T13:55:46.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 -- 172.21.15.46:0/2092530059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed0c13b980 con 0x7fed0c13f4a0 2023-10-30T13:55:46.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed027fc640 1 -- 172.21.15.46:0/2092530059 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fed04017020 con 0x7fed0c13f4a0 2023-10-30T13:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:44.283045+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:46.813 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 145 2023-10-30T13:55:46.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 -- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecf00664a0 msgr2=0x7fecf0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecf00664a0 0x7fecf0068960 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fed080af000 tx=0x7fed080b6040 comp rx=0 tx=0).stop 2023-10-30T13:55:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 -- 172.21.15.46:0/2092530059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0c13f4a0 msgr2=0x7fed0c1f64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0c13f4a0 0x7fed0c1f64d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fed04002240 tx=0x7fed0400cbe0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 -- 172.21.15.46:0/2092530059 shutdown_connections 2023-10-30T13:55:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecf00664a0 0x7fecf0068960 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0c13f4a0 0x7fed0c1f64d0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed0c0c6f60 0x7fed0c1f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 --2- 172.21.15.46:0/2092530059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed0c0c6b80 0x7fed0c158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 -- 172.21.15.46:0/2092530059 >> 172.21.15.46:0/2092530059 conn(0x7fed0c0c7910 msgr2=0x7fed0c0c19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 -- 172.21.15.46:0/2092530059 shutdown_connections 2023-10-30T13:55:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:46.810+0000 7fed13e17640 1 -- 172.21.15.46:0/2092530059 wait complete. 2023-10-30T13:55:46.835 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 145, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 145, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7356": { "addr": "172.21.15.46:6837/3112266825", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3112266825, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3112266825, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7356, "incarnation": 139, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 27 }, "gid_7518": { "addr": "172.21.15.46:6833/157453225", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 157453225, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 157453225, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7518, "incarnation": 145, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:55:44.283045+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7356, "mds_1": 7518 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:55:48.836 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:48.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.986+0000 7f011e794640 1 Processor -- start 2023-10-30T13:55:48.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.986+0000 7f011e794640 1 -- start start 2023-10-30T13:55:48.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f011e794640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0118070650 0x7f0118071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:48.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f011e794640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0118070c90 0x7f01180720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:48.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f011e794640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01180c2f20 con 0x7f0118156f80 2023-10-30T13:55:48.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f011e794640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0118157360 con 0x7f0118070c90 2023-10-30T13:55:48.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f011e794640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01180c85c0 con 0x7f0118070650 2023-10-30T13:55:48.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f01177fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0118070650 0x7f0118071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:48.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f011cd31640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0118070c90 0x7f01180720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:48.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f011cd31640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0118070c90 0x7f01180720f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39086/0 (socket says 172.21.15.46:39086) 2023-10-30T13:55:48.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f011cd31640 1 -- 172.21.15.46:0/4232832325 learned_addr learned my addr 172.21.15.46:0/4232832325 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:48.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f0117fff640 1 --1- 172.21.15.46:0/4232832325 >> v1:172.21.15.154:6789/0 conn(0x7f0118156f80 0x7f0118159880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:32882/0 (socket says 172.21.15.46:32882) 2023-10-30T13:55:48.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f011cd31640 1 -- 172.21.15.46:0/4232832325 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0118070650 msgr2=0x7f0118071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:48.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f011cd31640 1 --2- 172.21.15.46:0/4232832325 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0118070650 0x7f0118071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:48.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f011cd31640 1 -- 172.21.15.46:0/4232832325 >> v1:172.21.15.154:6789/0 conn(0x7f0118156f80 legacy=0x7f0118159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:48.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f011cd31640 1 -- 172.21.15.46:0/4232832325 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0118071230 con 0x7f0118070c90 2023-10-30T13:55:48.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.990+0000 7f01177fe640 1 --2- 172.21.15.46:0/4232832325 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0118070650 0x7f0118071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:55:48.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f011cd31640 1 --2- 172.21.15.46:0/4232832325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0118070c90 0x7f01180720f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0110009e50 tx=0x7f011001ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34a21550a7c3140e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:48.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f0116ffd640 1 -- 172.21.15.46:0/4232832325 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100a5870 con 0x7f0118070c90 2023-10-30T13:55:48.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f0116ffd640 1 -- 172.21.15.46:0/4232832325 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01100a59d0 con 0x7f0118070c90 2023-10-30T13:55:48.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f0116ffd640 1 -- 172.21.15.46:0/4232832325 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100a5d60 con 0x7f0118070c90 2023-10-30T13:55:48.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f011e794640 1 -- 172.21.15.46:0/4232832325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0118070c90 msgr2=0x7f01180720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:48.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f011e794640 1 --2- 172.21.15.46:0/4232832325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0118070c90 0x7f01180720f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0110009e50 tx=0x7f011001ca10 comp rx=0 tx=0).stop 2023-10-30T13:55:48.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f011e794640 1 -- 172.21.15.46:0/4232832325 shutdown_connections 2023-10-30T13:55:48.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f011e794640 1 --2- 172.21.15.46:0/4232832325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0118070c90 0x7f01180720f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:48.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f011e794640 1 --2- 172.21.15.46:0/4232832325 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0118070650 0x7f0118071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:48.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f011e794640 1 -- 172.21.15.46:0/4232832325 >> 172.21.15.46:0/4232832325 conn(0x7f01180cbeb0 msgr2=0x7f01180cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:48.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f011e794640 1 -- 172.21.15.46:0/4232832325 shutdown_connections 2023-10-30T13:55:49.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f011e794640 1 -- 172.21.15.46:0/4232832325 wait complete. 2023-10-30T13:55:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.994+0000 7f011e794640 1 Processor -- start 2023-10-30T13:55:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f011e794640 1 -- start start 2023-10-30T13:55:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f011e794640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0118070650 0x7f0118155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f011e794640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0118070c90 0x7f01180afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:49.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f011e794640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0118156f80 0x7f01180b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:49.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f011e794640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01181f2a60 con 0x7f0118070c90 2023-10-30T13:55:49.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f011e794640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01181f1e50 con 0x7f0118156f80 2023-10-30T13:55:49.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f011e794640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01181f2130 con 0x7f0118070650 2023-10-30T13:55:49.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f01177fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0118070c90 0x7f01180afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:49.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f01177fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0118070c90 0x7f01180afe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55750/0 (socket says 172.21.15.46:55750) 2023-10-30T13:55:49.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f0117fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0118070650 0x7f0118155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:49.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f01177fe640 1 -- 172.21.15.46:0/944762550 learned_addr learned my addr 172.21.15.46:0/944762550 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:49.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f011cd31640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0118156f80 0x7f01180b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:49.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f01177fe640 1 -- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0118070650 msgr2=0x7f0118155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f01177fe640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0118070650 0x7f0118155830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f01177fe640 1 -- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0118156f80 msgr2=0x7f01180b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f01177fe640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0118156f80 0x7f01180b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f01177fe640 1 -- 172.21.15.46:0/944762550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010c009420 con 0x7f0118070c90 2023-10-30T13:55:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f01177fe640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0118070c90 0x7f01180afe00 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f010c00e930 tx=0x7f010c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f0114ff9640 1 -- 172.21.15.46:0/944762550 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c00ca00 con 0x7f0118070c90 2023-10-30T13:55:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f0114ff9640 1 -- 172.21.15.46:0/944762550 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c00cb60 con 0x7f0118070c90 2023-10-30T13:55:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f011e794640 1 -- 172.21.15.46:0/944762550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0110009b30 con 0x7f0118070c90 2023-10-30T13:55:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f0114ff9640 1 -- 172.21.15.46:0/944762550 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c013470 con 0x7f0118070c90 2023-10-30T13:55:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f011e794640 1 -- 172.21.15.46:0/944762550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01181f26f0 con 0x7f0118070c90 2023-10-30T13:55:49.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:48.998+0000 7f011e794640 1 -- 172.21.15.46:0/944762550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01181f2bd0 con 0x7f0118070c90 2023-10-30T13:55:49.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.002+0000 7f0114ff9640 1 -- 172.21.15.46:0/944762550 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f010c0135d0 con 0x7f0118070c90 2023-10-30T13:55:49.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.002+0000 7f0114ff9640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01040664d0 0x7f0104068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:49.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.002+0000 7f0114ff9640 1 -- 172.21.15.46:0/944762550 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f010c084680 con 0x7f0118070c90 2023-10-30T13:55:49.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.002+0000 7f0117fff640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01040664d0 0x7f0104068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:49.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.002+0000 7f0117fff640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01040664d0 0x7f0104068990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0108000f70 tx=0x7f0108007e50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:49.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.006+0000 7f0114ff9640 1 -- 172.21.15.46:0/944762550 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f010c0514c0 con 0x7f0118070c90 2023-10-30T13:55:49.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.194+0000 7f011e794640 1 -- 172.21.15.46:0/944762550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f011804e170 con 0x7f0118070c90 2023-10-30T13:55:49.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.194+0000 7f0114ff9640 1 -- 172.21.15.46:0/944762550 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f010c054ae0 con 0x7f0118070c90 2023-10-30T13:55:49.199 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:49.199 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:44.283045+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:49.199 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 145 2023-10-30T13:55:49.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.198+0000 7f011e794640 1 -- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01040664d0 msgr2=0x7f0104068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:49.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.198+0000 7f011e794640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01040664d0 0x7f0104068990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0108000f70 tx=0x7f0108007e50 comp rx=0 tx=0).stop 2023-10-30T13:55:49.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.198+0000 7f011e794640 1 -- 172.21.15.46:0/944762550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0118070c90 msgr2=0x7f01180afe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:49.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.198+0000 7f011e794640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0118070c90 0x7f01180afe00 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f010c00e930 tx=0x7f010c00edf0 comp rx=0 tx=0).stop 2023-10-30T13:55:49.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.198+0000 7f011e794640 1 -- 172.21.15.46:0/944762550 shutdown_connections 2023-10-30T13:55:49.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.198+0000 7f011e794640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01040664d0 0x7f0104068990 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:49.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.198+0000 7f011e794640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0118156f80 0x7f01180b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:49.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.198+0000 7f011e794640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0118070c90 0x7f01180afe00 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:49.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.198+0000 7f011e794640 1 --2- 172.21.15.46:0/944762550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0118070650 0x7f0118155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:49.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.198+0000 7f011e794640 1 -- 172.21.15.46:0/944762550 >> 172.21.15.46:0/944762550 conn(0x7f01180cbeb0 msgr2=0x7f01180b0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:49.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.198+0000 7f011e794640 1 -- 172.21.15.46:0/944762550 shutdown_connections 2023-10-30T13:55:49.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:49.198+0000 7f011e794640 1 -- 172.21.15.46:0/944762550 wait complete. 2023-10-30T13:55:49.220 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:55:51.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:51.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.366+0000 7fa5cc9ab640 1 Processor -- start 2023-10-30T13:55:51.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.366+0000 7fa5cc9ab640 1 -- start start 2023-10-30T13:55:51.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5cc9ab640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa5c81538a0 0x7fa5c815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:51.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5cc9ab640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c80cf160 0x7fa5c80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:51.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5cc9ab640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c80cb800 con 0x7fa5c80ca0e0 2023-10-30T13:55:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5cc9ab640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c815dc30 con 0x7fa5c81538a0 2023-10-30T13:55:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5cc9ab640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c80cb330 con 0x7fa5c80cf160 2023-10-30T13:55:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5c5d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa5c80ca0e0 0x7fa5c815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35048/0 (socket says 172.21.15.46:35048) 2023-10-30T13:55:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5c5d9b640 1 -- 172.21.15.46:0/205426855 learned_addr learned my addr 172.21.15.46:0/205426855 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:55:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5c6d9d640 1 --2- 172.21.15.46:0/205426855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c80cf160 0x7fa5c80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5c659c640 1 --2- 172.21.15.46:0/205426855 >> v2:172.21.15.154:3300/0 conn(0x7fa5c81538a0 0x7fa5c815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:51.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5c559a640 1 -- 172.21.15.46:0/205426855 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066751523 0 0) 0x7fa5c80cb800 con 0x7fa5c80ca0e0 2023-10-30T13:55:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5c559a640 1 -- 172.21.15.46:0/205426855 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b0003530 con 0x7fa5c80ca0e0 2023-10-30T13:55:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5c659c640 1 -- 172.21.15.46:0/205426855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c80cf160 msgr2=0x7fa5c80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5c659c640 1 --2- 172.21.15.46:0/205426855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c80cf160 0x7fa5c80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5c659c640 1 -- 172.21.15.46:0/205426855 >> v1:172.21.15.46:6789/0 conn(0x7fa5c80ca0e0 legacy=0x7fa5c815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:51.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.370+0000 7fa5c659c640 1 -- 172.21.15.46:0/205426855 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c80cad70 con 0x7fa5c81538a0 2023-10-30T13:55:51.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.374+0000 7fa5c659c640 1 --2- 172.21.15.46:0/205426855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c81538a0 0x7fa5c815d6f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa5b80097c0 tx=0x7fa5b801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36d011ac5132653 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:51.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.374+0000 7fa5c559a640 1 -- 172.21.15.46:0/205426855 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b80a8970 con 0x7fa5c81538a0 2023-10-30T13:55:51.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.374+0000 7fa5c559a640 1 -- 172.21.15.46:0/205426855 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b80a8ad0 con 0x7fa5c81538a0 2023-10-30T13:55:51.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.374+0000 7fa5c559a640 1 -- 172.21.15.46:0/205426855 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b80a8e60 con 0x7fa5c81538a0 2023-10-30T13:55:51.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.374+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/205426855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c81538a0 msgr2=0x7fa5c815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:51.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.374+0000 7fa5cc9ab640 1 --2- 172.21.15.46:0/205426855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c81538a0 0x7fa5c815d6f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa5b80097c0 tx=0x7fa5b801ec70 comp rx=0 tx=0).stop 2023-10-30T13:55:51.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.374+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/205426855 shutdown_connections 2023-10-30T13:55:51.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.374+0000 7fa5cc9ab640 1 --2- 172.21.15.46:0/205426855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c80cf160 0x7fa5c80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:51.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.374+0000 7fa5cc9ab640 1 --2- 172.21.15.46:0/205426855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c81538a0 0x7fa5c815d6f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:51.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.374+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/205426855 >> 172.21.15.46:0/205426855 conn(0x7fa5c806bfe0 msgr2=0x7fa5c806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:51.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.374+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/205426855 shutdown_connections 2023-10-30T13:55:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.374+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/205426855 wait complete. 2023-10-30T13:55:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5cc9ab640 1 Processor -- start 2023-10-30T13:55:51.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5cc9ab640 1 -- start start 2023-10-30T13:55:51.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5cc9ab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c80ca0e0 0x7fa5c8145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:51.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5cc9ab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c80cf160 0x7fa5c8146010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:51.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5cc9ab640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c81538a0 0x7fa5c81e3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:51.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5cc9ab640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81e49e0 con 0x7fa5c80cf160 2023-10-30T13:55:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5cc9ab640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81e2ae0 con 0x7fa5c80ca0e0 2023-10-30T13:55:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5cc9ab640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81e3660 con 0x7fa5c81538a0 2023-10-30T13:55:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5c5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c80cf160 0x7fa5c8146010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5c5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c80cf160 0x7fa5c8146010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55754/0 (socket says 172.21.15.46:55754) 2023-10-30T13:55:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5c5d9b640 1 -- 172.21.15.46:0/783367848 learned_addr learned my addr 172.21.15.46:0/783367848 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:51.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5c659c640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c80ca0e0 0x7fa5c8145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:51.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5c6d9d640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c81538a0 0x7fa5c81e3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:51.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5c5d9b640 1 -- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c81538a0 msgr2=0x7fa5c81e3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:51.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5c5d9b640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c81538a0 0x7fa5c81e3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:51.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5c5d9b640 1 -- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c80ca0e0 msgr2=0x7fa5c8145ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:51.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5c5d9b640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c80ca0e0 0x7fa5c8145ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:51.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5c5d9b640 1 -- 172.21.15.46:0/783367848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b8009420 con 0x7fa5c80cf160 2023-10-30T13:55:51.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5c6d9d640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c81538a0 0x7fa5c81e3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:55:51.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5c5d9b640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c80cf160 0x7fa5c8146010 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc00c4d0 tx=0x7fa5bc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:51.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5b6ffd640 1 -- 172.21.15.46:0/783367848 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc005cf0 con 0x7fa5c80cf160 2023-10-30T13:55:51.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5b6ffd640 1 -- 172.21.15.46:0/783367848 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc005e50 con 0x7fa5c80cf160 2023-10-30T13:55:51.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5b6ffd640 1 -- 172.21.15.46:0/783367848 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc007270 con 0x7fa5c80cf160 2023-10-30T13:55:51.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/783367848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c81efce0 con 0x7fa5c80cf160 2023-10-30T13:55:51.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/783367848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c81e3010 con 0x7fa5c80cf160 2023-10-30T13:55:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.378+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/783367848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c80b2a50 con 0x7fa5c80cf160 2023-10-30T13:55:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.382+0000 7fa5b6ffd640 1 -- 172.21.15.46:0/783367848 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa5bc0073d0 con 0x7fa5c80cf160 2023-10-30T13:55:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.382+0000 7fa5b6ffd640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5a80664a0 0x7fa5a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:51.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.382+0000 7fa5b6ffd640 1 -- 172.21.15.46:0/783367848 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fa5bc083b20 con 0x7fa5c80cf160 2023-10-30T13:55:51.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.382+0000 7fa5c659c640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5a80664a0 0x7fa5a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:51.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.382+0000 7fa5c659c640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5a80664a0 0x7fa5a8068960 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa5b80b2000 tx=0x7fa5b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:51.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.386+0000 7fa5b6ffd640 1 -- 172.21.15.46:0/783367848 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa5bc0508d0 con 0x7fa5c80cf160 2023-10-30T13:55:51.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.582+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/783367848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5c804e170 con 0x7fa5c80cf160 2023-10-30T13:55:51.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.582+0000 7fa5b6ffd640 1 -- 172.21.15.46:0/783367848 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fa5bc017020 con 0x7fa5c80cf160 2023-10-30T13:55:51.587 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:51.587 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:44.283045+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:51.587 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 145 2023-10-30T13:55:51.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.586+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5a80664a0 msgr2=0x7fa5a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:51.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.586+0000 7fa5cc9ab640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5a80664a0 0x7fa5a8068960 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa5b80b2000 tx=0x7fa5b8008040 comp rx=0 tx=0).stop 2023-10-30T13:55:51.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.586+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/783367848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c80cf160 msgr2=0x7fa5c8146010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:51.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.586+0000 7fa5cc9ab640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c80cf160 0x7fa5c8146010 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc00c4d0 tx=0x7fa5bc00c990 comp rx=0 tx=0).stop 2023-10-30T13:55:51.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.586+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/783367848 shutdown_connections 2023-10-30T13:55:51.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.586+0000 7fa5cc9ab640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5a80664a0 0x7fa5a8068960 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:51.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.586+0000 7fa5cc9ab640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c81538a0 0x7fa5c81e3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:51.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.586+0000 7fa5cc9ab640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c80cf160 0x7fa5c8146010 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:51.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.586+0000 7fa5cc9ab640 1 --2- 172.21.15.46:0/783367848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c80ca0e0 0x7fa5c8145ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:51.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.586+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/783367848 >> 172.21.15.46:0/783367848 conn(0x7fa5c806bfe0 msgr2=0x7fa5c8151c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:51.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.586+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/783367848 shutdown_connections 2023-10-30T13:55:51.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:51.586+0000 7fa5cc9ab640 1 -- 172.21.15.46:0/783367848 wait complete. 2023-10-30T13:55:51.608 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:55:52.072 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:55:52.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:55:53.609 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.758+0000 7f549aa46640 1 Processor -- start 2023-10-30T13:55:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.758+0000 7f549aa46640 1 -- start start 2023-10-30T13:55:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f549aa46640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f54940c93d0 0x7f549406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f549aa46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549406afa0 con 0x7f5494153d70 2023-10-30T13:55:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f549aa46640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5494154150 con 0x7f549406e470 2023-10-30T13:55:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f549aa46640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f549406d780 con 0x7f54940c93d0 2023-10-30T13:55:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f5493fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5494153d70 0x7f54940cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35064/0 (socket says 172.21.15.46:35064) 2023-10-30T13:55:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f5493fff640 1 -- 172.21.15.46:0/216600081 learned_addr learned my addr 172.21.15.46:0/216600081 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:55:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f5498fe3640 1 --2- 172.21.15.46:0/216600081 >> v2:172.21.15.154:3300/0 conn(0x7f54940c93d0 0x7f549406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810818236 0 0) 0x7f549406afa0 con 0x7f5494153d70 2023-10-30T13:55:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5480003530 con 0x7f5494153d70 2023-10-30T13:55:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947279899 0 0) 0x7f5494154150 con 0x7f549406e470 2023-10-30T13:55:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f549406afa0 con 0x7f549406e470 2023-10-30T13:55:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3939289241 0 0) 0x7f5480003530 con 0x7f5494153d70 2023-10-30T13:55:53.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 >> v2:172.21.15.154:3300/0 conn(0x7f54940c93d0 msgr2=0x7f549406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:53.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 --2- 172.21.15.46:0/216600081 >> v2:172.21.15.154:3300/0 conn(0x7f54940c93d0 0x7f549406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 >> v1:172.21.15.154:6789/0 conn(0x7f549406e470 legacy=0x7f54940c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549406e030 con 0x7f5494153d70 2023-10-30T13:55:53.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f5484001680 con 0x7f5494153d70 2023-10-30T13:55:53.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f5498fe3640 1 --2- 172.21.15.46:0/216600081 >> v2:172.21.15.154:3300/0 conn(0x7f54940c93d0 0x7f549406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:55:53.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 >> v1:172.21.15.46:6789/0 conn(0x7f5494153d70 legacy=0x7f54940cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 --2- 172.21.15.46:0/216600081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5480003b10 0x7f5480003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 --2- 172.21.15.46:0/216600081 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54800047d0 0x7f5480004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 --2- 172.21.15.46:0/216600081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5480005190 0x7f548000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549406d780 con 0x7f5480005190 2023-10-30T13:55:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548000c010 con 0x7f5480003b10 2023-10-30T13:55:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548000c1d0 con 0x7f54800047d0 2023-10-30T13:55:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f5498fe3640 1 --2- 172.21.15.46:0/216600081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5480005190 0x7f548000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f5493fff640 1 --2- 172.21.15.46:0/216600081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5480003b10 0x7f5480003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f548bfff640 1 --2- 172.21.15.46:0/216600081 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54800047d0 0x7f5480004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f5498fe3640 1 -- 172.21.15.46:0/216600081 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54800047d0 msgr2=0x7f5480004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:53.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f5498fe3640 1 --2- 172.21.15.46:0/216600081 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54800047d0 0x7f5480004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f5498fe3640 1 -- 172.21.15.46:0/216600081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5480003b10 msgr2=0x7f5480003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:53.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f5498fe3640 1 --2- 172.21.15.46:0/216600081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5480003b10 0x7f5480003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.762+0000 7f5498fe3640 1 -- 172.21.15.46:0/216600081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5480003530 con 0x7f5480005190 2023-10-30T13:55:53.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.766+0000 7f5498fe3640 1 -- 172.21.15.46:0/216600081 reap_dead start 2023-10-30T13:55:53.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.766+0000 7f5498fe3640 1 --2- 172.21.15.46:0/216600081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5480005190 0x7f548000b990 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f548c00e350 tx=0x7f548c01ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d70b6c1901c350f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:53.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.770+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484007ea0 con 0x7f5480005190 2023-10-30T13:55:53.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.770+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548c00f040 con 0x7f5480005190 2023-10-30T13:55:53.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.770+0000 7f54937fe640 1 -- 172.21.15.46:0/216600081 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c0039a0 con 0x7f5480005190 2023-10-30T13:55:53.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.770+0000 7f549aa46640 1 -- 172.21.15.46:0/216600081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5480005190 msgr2=0x7f548000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.770+0000 7f549aa46640 1 --2- 172.21.15.46:0/216600081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5480005190 0x7f548000b990 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f548c00e350 tx=0x7f548c01ce60 comp rx=0 tx=0).stop 2023-10-30T13:55:53.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.770+0000 7f549aa46640 1 -- 172.21.15.46:0/216600081 shutdown_connections 2023-10-30T13:55:53.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.770+0000 7f549aa46640 1 --2- 172.21.15.46:0/216600081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5480005190 0x7f548000b990 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.770+0000 7f549aa46640 1 -- 172.21.15.46:0/216600081 >> 172.21.15.46:0/216600081 conn(0x7f5494071ea0 msgr2=0x7f5494065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:53.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.770+0000 7f549aa46640 1 -- 172.21.15.46:0/216600081 shutdown_connections 2023-10-30T13:55:53.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.770+0000 7f549aa46640 1 -- 172.21.15.46:0/216600081 wait complete. 2023-10-30T13:55:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.770+0000 7f549aa46640 1 Processor -- start 2023-10-30T13:55:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.770+0000 7f549aa46640 1 -- start start 2023-10-30T13:55:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f549aa46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5480005190 0x7f5494146250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f549aa46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f549414e670 0x7f549414ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f549aa46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549414aa60 0x7f549414aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f549aa46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549414cc50 con 0x7f549414e670 2023-10-30T13:55:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f549aa46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54941f0160 con 0x7f549414aa60 2023-10-30T13:55:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f549aa46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549414cea0 con 0x7f5480005190 2023-10-30T13:55:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f548bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f549414e670 0x7f549414ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f548bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f549414e670 0x7f549414ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55764/0 (socket says 172.21.15.46:55764) 2023-10-30T13:55:53.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f548bfff640 1 -- 172.21.15.46:0/2654010566 learned_addr learned my addr 172.21.15.46:0/2654010566 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:53.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f5493fff640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5480005190 0x7f5494146250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f5498fe3640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549414aa60 0x7f549414aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f548bfff640 1 -- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5480005190 msgr2=0x7f5494146250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f548bfff640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5480005190 0x7f5494146250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f548bfff640 1 -- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549414aa60 msgr2=0x7f549414aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f548bfff640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549414aa60 0x7f549414aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f548bfff640 1 -- 172.21.15.46:0/2654010566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54941ef550 con 0x7f549414e670 2023-10-30T13:55:53.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f548bfff640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f549414e670 0x7f549414ea90 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f547c00e530 tx=0x7f547c00e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f54917fa640 1 -- 172.21.15.46:0/2654010566 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c0052e0 con 0x7f549414e670 2023-10-30T13:55:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f54917fa640 1 -- 172.21.15.46:0/2654010566 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c005440 con 0x7f549414e670 2023-10-30T13:55:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f54917fa640 1 -- 172.21.15.46:0/2654010566 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c003840 con 0x7f549414e670 2023-10-30T13:55:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f549aa46640 1 -- 172.21.15.46:0/2654010566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54941ef830 con 0x7f549414e670 2023-10-30T13:55:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f549aa46640 1 -- 172.21.15.46:0/2654010566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54941efdf0 con 0x7f549414e670 2023-10-30T13:55:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f549aa46640 1 -- 172.21.15.46:0/2654010566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549404e170 con 0x7f549414e670 2023-10-30T13:55:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.774+0000 7f54917fa640 1 -- 172.21.15.46:0/2654010566 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f547c0039a0 con 0x7f549414e670 2023-10-30T13:55:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.778+0000 7f54917fa640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f54740664d0 0x7f5474068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.778+0000 7f54917fa640 1 -- 172.21.15.46:0/2654010566 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f547c0842c0 con 0x7f549414e670 2023-10-30T13:55:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.778+0000 7f5493fff640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f54740664d0 0x7f5474068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:53.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.778+0000 7f5493fff640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f54740664d0 0x7f5474068990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5484000c10 tx=0x7f5484009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:53.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.782+0000 7f54917fa640 1 -- 172.21.15.46:0/2654010566 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f547c051060 con 0x7f549414e670 2023-10-30T13:55:53.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 -- 172.21.15.46:0/2654010566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f549414dbc0 con 0x7f549414e670 2023-10-30T13:55:53.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f54917fa640 1 -- 172.21.15.46:0/2654010566 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f547c054680 con 0x7f549414e670 2023-10-30T13:55:53.984 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:53.984 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:44.283045+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:53.985 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 145 2023-10-30T13:55:53.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 -- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f54740664d0 msgr2=0x7f5474068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f54740664d0 0x7f5474068990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5484000c10 tx=0x7f5484009040 comp rx=0 tx=0).stop 2023-10-30T13:55:53.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 -- 172.21.15.46:0/2654010566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f549414e670 msgr2=0x7f549414ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:53.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f549414e670 0x7f549414ea90 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f547c00e530 tx=0x7f547c00e9f0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 -- 172.21.15.46:0/2654010566 shutdown_connections 2023-10-30T13:55:53.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f54740664d0 0x7f5474068990 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549414aa60 0x7f549414aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f549414e670 0x7f549414ea90 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 --2- 172.21.15.46:0/2654010566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5480005190 0x7f5494146250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:53.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 -- 172.21.15.46:0/2654010566 >> 172.21.15.46:0/2654010566 conn(0x7f5494071ea0 msgr2=0x7f54940cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:53.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 -- 172.21.15.46:0/2654010566 shutdown_connections 2023-10-30T13:55:53.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:53.982+0000 7f549aa46640 1 -- 172.21.15.46:0/2654010566 wait complete. 2023-10-30T13:55:54.014 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:55:56.016 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.166+0000 7f8fd32d6640 1 Processor -- start 2023-10-30T13:55:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.166+0000 7f8fd32d6640 1 -- start start 2023-10-30T13:55:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.166+0000 7f8fd32d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8fcc06f7a0 0x7f8fcc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.166+0000 7f8fd32d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8fcc071110 0x7f8fcc158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.166+0000 7f8fd32d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fcc066cf0 con 0x7f8fcc0c6bd0 2023-10-30T13:55:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.166+0000 7f8fd32d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0700c0 con 0x7f8fcc071110 2023-10-30T13:55:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.166+0000 7f8fd32d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0c70d0 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.166+0000 7f8fd1873640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8fcc071110 0x7f8fcc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.170+0000 7f8fd1873640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8fcc071110 0x7f8fcc158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55776/0 (socket says 172.21.15.46:55776) 2023-10-30T13:55:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.170+0000 7f8fd1873640 1 -- 172.21.15.46:0/535514544 learned_addr learned my addr 172.21.15.46:0/535514544 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.170+0000 7f8fd1072640 1 --2- 172.21.15.46:0/535514544 >> v2:172.21.15.154:3300/0 conn(0x7f8fcc06f7a0 0x7f8fcc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.170+0000 7f8fd1873640 1 -- 172.21.15.46:0/535514544 >> v2:172.21.15.154:3300/0 conn(0x7f8fcc06f7a0 msgr2=0x7f8fcc06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.170+0000 7f8fd1873640 1 --2- 172.21.15.46:0/535514544 >> v2:172.21.15.154:3300/0 conn(0x7f8fcc06f7a0 0x7f8fcc06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:56.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.170+0000 7f8fd1873640 1 -- 172.21.15.46:0/535514544 >> v1:172.21.15.154:6789/0 conn(0x7f8fcc0c6bd0 legacy=0x7f8fcc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.170+0000 7f8fd1873640 1 -- 172.21.15.46:0/535514544 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0c7390 con 0x7f8fcc071110 2023-10-30T13:55:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.170+0000 7f8fd1873640 1 --2- 172.21.15.46:0/535514544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fcc071110 0x7f8fcc158110 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc06c830 tx=0x7f8fc401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c6d60dad052a80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.170+0000 7f8fc3fff640 1 -- 172.21.15.46:0/535514544 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40af400 con 0x7f8fcc071110 2023-10-30T13:55:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.170+0000 7f8fc3fff640 1 -- 172.21.15.46:0/535514544 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc40ab050 con 0x7f8fcc071110 2023-10-30T13:55:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.170+0000 7f8fc3fff640 1 -- 172.21.15.46:0/535514544 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40af780 con 0x7f8fcc071110 2023-10-30T13:55:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 -- 172.21.15.46:0/535514544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fcc071110 msgr2=0x7f8fcc158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 --2- 172.21.15.46:0/535514544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fcc071110 0x7f8fcc158110 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc06c830 tx=0x7f8fc401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:55:56.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 -- 172.21.15.46:0/535514544 shutdown_connections 2023-10-30T13:55:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 --2- 172.21.15.46:0/535514544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fcc071110 0x7f8fcc158110 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 --2- 172.21.15.46:0/535514544 >> v2:172.21.15.154:3300/0 conn(0x7f8fcc06f7a0 0x7f8fcc06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 -- 172.21.15.46:0/535514544 >> 172.21.15.46:0/535514544 conn(0x7f8fcc0c7910 msgr2=0x7f8fcc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 -- 172.21.15.46:0/535514544 shutdown_connections 2023-10-30T13:55:56.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 -- 172.21.15.46:0/535514544 wait complete. 2023-10-30T13:55:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 Processor -- start 2023-10-30T13:55:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 -- start start 2023-10-30T13:55:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fcc06f7a0 0x7f8fcc1575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fcc071110 0x7f8fcc152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fcc0c6bd0 0x7f8fcc14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc06c560 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0c20b0 con 0x7f8fcc071110 2023-10-30T13:55:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd32d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0c9340 con 0x7f8fcc0c6bd0 2023-10-30T13:55:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd1072640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fcc06f7a0 0x7f8fcc1575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd1072640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fcc06f7a0 0x7f8fcc1575a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55784/0 (socket says 172.21.15.46:55784) 2023-10-30T13:55:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd1072640 1 -- 172.21.15.46:0/2154340732 learned_addr learned my addr 172.21.15.46:0/2154340732 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd1873640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fcc0c6bd0 0x7f8fcc14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd0871640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fcc071110 0x7f8fcc152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd1072640 1 -- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fcc0c6bd0 msgr2=0x7f8fcc14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd1072640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fcc0c6bd0 0x7f8fcc14efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd1072640 1 -- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fcc071110 msgr2=0x7f8fcc152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd1072640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fcc071110 0x7f8fcc152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.174+0000 7f8fd1072640 1 -- 172.21.15.46:0/2154340732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc4009b30 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.178+0000 7f8fd1072640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fcc06f7a0 0x7f8fcc1575a0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800e870 tx=0x7f8fc800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.178+0000 7f8fc1ffb640 1 -- 172.21.15.46:0/2154340732 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8009510 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.178+0000 7f8fc1ffb640 1 -- 172.21.15.46:0/2154340732 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc800eec0 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.178+0000 7f8fc1ffb640 1 -- 172.21.15.46:0/2154340732 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc800cb40 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.178+0000 7f8fd32d6640 1 -- 172.21.15.46:0/2154340732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc154b40 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.178+0000 7f8fd32d6640 1 -- 172.21.15.46:0/2154340732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc0c9060 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.178+0000 7f8fd32d6640 1 -- 172.21.15.46:0/2154340732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc070330 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.178+0000 7f8fc1ffb640 1 -- 172.21.15.46:0/2154340732 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8fc800cca0 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.178+0000 7f8fc1ffb640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8fb00664a0 0x7f8fb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.178+0000 7f8fc1ffb640 1 -- 172.21.15.46:0/2154340732 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f8fc80849c0 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.178+0000 7f8fd0871640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8fb00664a0 0x7f8fb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.182+0000 7f8fd0871640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8fb00664a0 0x7f8fb0068960 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc1e58f0 tx=0x7f8fbc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.182+0000 7f8fc1ffb640 1 -- 172.21.15.46:0/2154340732 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8fcc070330 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.374+0000 7f8fd32d6640 1 -- 172.21.15.46:0/2154340732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fcc070330 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.374+0000 7f8fc1ffb640 1 -- 172.21.15.46:0/2154340732 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f8fcc070330 con 0x7f8fcc06f7a0 2023-10-30T13:55:56.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:56.379 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:54.369067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:resolve","state_seq":11,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 146 2023-10-30T13:55:56.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.378+0000 7f8fd32d6640 1 -- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8fb00664a0 msgr2=0x7f8fb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:56.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.378+0000 7f8fd32d6640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8fb00664a0 0x7f8fb0068960 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc1e58f0 tx=0x7f8fbc009040 comp rx=0 tx=0).stop 2023-10-30T13:55:56.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.378+0000 7f8fd32d6640 1 -- 172.21.15.46:0/2154340732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fcc06f7a0 msgr2=0x7f8fcc1575a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:56.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.378+0000 7f8fd32d6640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fcc06f7a0 0x7f8fcc1575a0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800e870 tx=0x7f8fc800ec70 comp rx=0 tx=0).stop 2023-10-30T13:55:56.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.378+0000 7f8fd32d6640 1 -- 172.21.15.46:0/2154340732 shutdown_connections 2023-10-30T13:55:56.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.378+0000 7f8fd32d6640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8fb00664a0 0x7f8fb0068960 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:56.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.378+0000 7f8fd32d6640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fcc0c6bd0 0x7f8fcc14efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:56.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.378+0000 7f8fd32d6640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fcc071110 0x7f8fcc152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:56.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.378+0000 7f8fd32d6640 1 --2- 172.21.15.46:0/2154340732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fcc06f7a0 0x7f8fcc1575a0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:56.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.378+0000 7f8fd32d6640 1 -- 172.21.15.46:0/2154340732 >> 172.21.15.46:0/2154340732 conn(0x7f8fcc0c7910 msgr2=0x7f8fcc0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:56.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.378+0000 7f8fd32d6640 1 -- 172.21.15.46:0/2154340732 shutdown_connections 2023-10-30T13:55:56.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:56.378+0000 7f8fd32d6640 1 -- 172.21.15.46:0/2154340732 wait complete. 2023-10-30T13:55:56.401 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:55:58.402 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:55:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.550+0000 7fe44aa91640 1 Processor -- start 2023-10-30T13:55:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.550+0000 7fe44aa91640 1 -- start start 2023-10-30T13:55:58.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe44aa91640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4440c6b80 0x7fe4440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:58.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe44aa91640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe4440c6f60 0x7fe4440c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:58.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe44aa91640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440c0a50 con 0x7fe4440c8430 2023-10-30T13:55:58.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe44aa91640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe44406f6e0 con 0x7fe4440c6b80 2023-10-30T13:55:58.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe44aa91640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe44406f9c0 con 0x7fe4440c6f60 2023-10-30T13:55:58.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe44882d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4440c6b80 0x7fe4440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:58.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe44882d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4440c6b80 0x7fe4440bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52162/0 (socket says 172.21.15.46:52162) 2023-10-30T13:55:58.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe44882d640 1 -- 172.21.15.46:0/2071136249 learned_addr learned my addr 172.21.15.46:0/2071136249 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:58.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe443fff640 1 --2- 172.21.15.46:0/2071136249 >> v2:172.21.15.154:3300/0 conn(0x7fe4440c6f60 0x7fe4440c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe44882d640 1 -- 172.21.15.46:0/2071136249 >> v2:172.21.15.154:3300/0 conn(0x7fe4440c6f60 msgr2=0x7fe4440c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:55:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe44882d640 1 --2- 172.21.15.46:0/2071136249 >> v2:172.21.15.154:3300/0 conn(0x7fe4440c6f60 0x7fe4440c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe44882d640 1 -- 172.21.15.46:0/2071136249 >> v1:172.21.15.154:6789/0 conn(0x7fe4440c8430 legacy=0x7fe44413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.554+0000 7fe44882d640 1 -- 172.21.15.46:0/2071136249 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44406fca0 con 0x7fe4440c6b80 2023-10-30T13:55:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe44882d640 1 --2- 172.21.15.46:0/2071136249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4440c6b80 0x7fe4440bfec0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fe444159b90 tx=0x7fe43801ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b522789effc7029 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe4437fe640 1 -- 172.21.15.46:0/2071136249 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4380a7bb0 con 0x7fe4440c6b80 2023-10-30T13:55:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe4437fe640 1 -- 172.21.15.46:0/2071136249 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4380a7d10 con 0x7fe4440c6b80 2023-10-30T13:55:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe4437fe640 1 -- 172.21.15.46:0/2071136249 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4380af660 con 0x7fe4440c6b80 2023-10-30T13:55:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe44aa91640 1 -- 172.21.15.46:0/2071136249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4440c6b80 msgr2=0x7fe4440bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe44aa91640 1 --2- 172.21.15.46:0/2071136249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4440c6b80 0x7fe4440bfec0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fe444159b90 tx=0x7fe43801ebd0 comp rx=0 tx=0).stop 2023-10-30T13:55:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe44aa91640 1 -- 172.21.15.46:0/2071136249 shutdown_connections 2023-10-30T13:55:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe44aa91640 1 --2- 172.21.15.46:0/2071136249 >> v2:172.21.15.154:3300/0 conn(0x7fe4440c6f60 0x7fe4440c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe44aa91640 1 --2- 172.21.15.46:0/2071136249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4440c6b80 0x7fe4440bfec0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe44aa91640 1 -- 172.21.15.46:0/2071136249 >> 172.21.15.46:0/2071136249 conn(0x7fe4440c7910 msgr2=0x7fe4440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe44aa91640 1 -- 172.21.15.46:0/2071136249 shutdown_connections 2023-10-30T13:55:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe44aa91640 1 -- 172.21.15.46:0/2071136249 wait complete. 2023-10-30T13:55:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe44aa91640 1 Processor -- start 2023-10-30T13:55:58.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe44aa91640 1 -- start start 2023-10-30T13:55:58.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.558+0000 7fe44aa91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4440c6b80 0x7fe4440c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:58.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44aa91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4440c6f60 0x7fe4440c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:58.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44aa91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4440c8430 0x7fe444152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:58.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44aa91640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe444157750 con 0x7fe4440c6b80 2023-10-30T13:55:58.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44aa91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44413e690 con 0x7fe4440c6f60 2023-10-30T13:55:58.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44aa91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe44413e0d0 con 0x7fe4440c8430 2023-10-30T13:55:58.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44882d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4440c6b80 0x7fe4440c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:58.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44882d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4440c6b80 0x7fe4440c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52164/0 (socket says 172.21.15.46:52164) 2023-10-30T13:55:58.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44882d640 1 -- 172.21.15.46:0/337118547 learned_addr learned my addr 172.21.15.46:0/337118547 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:55:58.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe443fff640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4440c6f60 0x7fe4440c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:58.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44902e640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4440c8430 0x7fe444152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:58.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44882d640 1 -- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4440c8430 msgr2=0x7fe444152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:58.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44882d640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4440c8430 0x7fe444152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:58.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44882d640 1 -- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4440c6f60 msgr2=0x7fe4440c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:58.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44882d640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4440c6f60 0x7fe4440c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:58.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44882d640 1 -- 172.21.15.46:0/337118547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43800b770 con 0x7fe4440c6b80 2023-10-30T13:55:58.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44882d640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4440c6b80 0x7fe4440c8c10 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe43800b2c0 tx=0x7fe43801b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:58.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe4417fa640 1 -- 172.21.15.46:0/337118547 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe438006ee0 con 0x7fe4440c6b80 2023-10-30T13:55:58.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe4417fa640 1 -- 172.21.15.46:0/337118547 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4380ba620 con 0x7fe4440c6b80 2023-10-30T13:55:58.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe4417fa640 1 -- 172.21.15.46:0/337118547 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4380ba9b0 con 0x7fe4440c6b80 2023-10-30T13:55:58.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44aa91640 1 -- 172.21.15.46:0/337118547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe44413e3b0 con 0x7fe4440c6b80 2023-10-30T13:55:58.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44aa91640 1 -- 172.21.15.46:0/337118547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe444135b40 con 0x7fe4440c6b80 2023-10-30T13:55:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe44aa91640 1 -- 172.21.15.46:0/337118547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44413ca90 con 0x7fe4440c6b80 2023-10-30T13:55:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe4417fa640 1 -- 172.21.15.46:0/337118547 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe4380b6040 con 0x7fe4440c6b80 2023-10-30T13:55:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.562+0000 7fe4417fa640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4280664a0 0x7fe428068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:55:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.566+0000 7fe4417fa640 1 -- 172.21.15.46:0/337118547 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fe4381219d0 con 0x7fe4440c6b80 2023-10-30T13:55:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.566+0000 7fe443fff640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4280664a0 0x7fe428068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:55:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.566+0000 7fe443fff640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4280664a0 0x7fe428068960 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe4440ca8f0 tx=0x7fe430008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:55:58.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.570+0000 7fe4417fa640 1 -- 172.21.15.46:0/337118547 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe44413ca90 con 0x7fe4440c6b80 2023-10-30T13:55:58.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 -- 172.21.15.46:0/337118547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe44413cf10 con 0x7fe4440c6b80 2023-10-30T13:55:58.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe4417fa640 1 -- 172.21.15.46:0/337118547 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+5731 (secure 0 0 0) 0x7fe44413cf10 con 0x7fe4440c6b80 2023-10-30T13:55:58.760 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:55:58.760 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:54.369067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:resolve","state_seq":11,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:55:58.761 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 146 2023-10-30T13:55:58.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 -- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4280664a0 msgr2=0x7fe428068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:58.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4280664a0 0x7fe428068960 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe4440ca8f0 tx=0x7fe430008040 comp rx=0 tx=0).stop 2023-10-30T13:55:58.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 -- 172.21.15.46:0/337118547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4440c6b80 msgr2=0x7fe4440c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:55:58.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4440c6b80 0x7fe4440c8c10 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe43800b2c0 tx=0x7fe43801b970 comp rx=0 tx=0).stop 2023-10-30T13:55:58.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 -- 172.21.15.46:0/337118547 shutdown_connections 2023-10-30T13:55:58.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe4280664a0 0x7fe428068960 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:58.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4440c8430 0x7fe444152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:58.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4440c6f60 0x7fe4440c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:58.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 --2- 172.21.15.46:0/337118547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4440c6b80 0x7fe4440c8c10 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:55:58.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 -- 172.21.15.46:0/337118547 >> 172.21.15.46:0/337118547 conn(0x7fe4440c7910 msgr2=0x7fe4440c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:55:58.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 -- 172.21.15.46:0/337118547 shutdown_connections 2023-10-30T13:55:58.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:55:58.758+0000 7fe44aa91640 1 -- 172.21.15.46:0/337118547 wait complete. 2023-10-30T13:55:58.781 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 146, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 146, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7356": { "addr": "172.21.15.46:6837/3112266825", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3112266825, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3112266825, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7356, "incarnation": 139, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 27 }, "gid_7518": { "addr": "172.21.15.46:6833/157453225", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 157453225, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 157453225, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7518, "incarnation": 145, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:resolve", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:55:54.369067+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7356, "mds_1": 7518 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:56:00.782 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:00.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc28ac54640 1 Processor -- start 2023-10-30T13:56:00.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc28ac54640 1 -- start start 2023-10-30T13:56:00.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc28ac54640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc2840c8460 0x7fc2840715c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc28ac54640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc284070650 0x7fc284159810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc28ac54640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc284159d50 con 0x7fc284157010 2023-10-30T13:56:00.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc28ac54640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2840c21b0 con 0x7fc2840c8460 2023-10-30T13:56:00.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc28ac54640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2840c1530 con 0x7fc284070650 2023-10-30T13:56:00.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc283fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc2840c8460 0x7fc2840715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc2889f0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc284157010 0x7fc284071bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44770/0 (socket says 172.21.15.46:44770) 2023-10-30T13:56:00.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc2889f0640 1 -- 172.21.15.46:0/1274633540 learned_addr learned my addr 172.21.15.46:0/1274633540 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:56:00.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc2891f1640 1 --2- 172.21.15.46:0/1274633540 >> v2:172.21.15.154:3300/0 conn(0x7fc284070650 0x7fc284159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc283fff640 1 -- 172.21.15.46:0/1274633540 >> v2:172.21.15.154:3300/0 conn(0x7fc284070650 msgr2=0x7fc284159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:00.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc283fff640 1 --2- 172.21.15.46:0/1274633540 >> v2:172.21.15.154:3300/0 conn(0x7fc284070650 0x7fc284159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc283fff640 1 -- 172.21.15.46:0/1274633540 >> v1:172.21.15.46:6789/0 conn(0x7fc284157010 legacy=0x7fc284071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:00.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc283fff640 1 -- 172.21.15.46:0/1274633540 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2840c0f70 con 0x7fc2840c8460 2023-10-30T13:56:00.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.934+0000 7fc2837fe640 1 -- 172.21.15.46:0/1274633540 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593050964 0 0) 0x7fc284159d50 con 0x7fc284157010 2023-10-30T13:56:00.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.938+0000 7fc283fff640 1 --2- 172.21.15.46:0/1274633540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2840c8460 0x7fc2840715c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fc2840ccaf0 tx=0x7fc27001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147f5881971a309c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:00.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.938+0000 7fc2837fe640 1 -- 172.21.15.46:0/1274633540 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc270004230 con 0x7fc2840c8460 2023-10-30T13:56:00.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.938+0000 7fc2837fe640 1 -- 172.21.15.46:0/1274633540 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2700a7050 con 0x7fc2840c8460 2023-10-30T13:56:00.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.938+0000 7fc2837fe640 1 -- 172.21.15.46:0/1274633540 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc270003620 con 0x7fc2840c8460 2023-10-30T13:56:00.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.938+0000 7fc28ac54640 1 -- 172.21.15.46:0/1274633540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2840c8460 msgr2=0x7fc2840715c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:00.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.938+0000 7fc28ac54640 1 --2- 172.21.15.46:0/1274633540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2840c8460 0x7fc2840715c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fc2840ccaf0 tx=0x7fc27001c460 comp rx=0 tx=0).stop 2023-10-30T13:56:00.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.938+0000 7fc28ac54640 1 -- 172.21.15.46:0/1274633540 shutdown_connections 2023-10-30T13:56:00.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.938+0000 7fc28ac54640 1 --2- 172.21.15.46:0/1274633540 >> v2:172.21.15.154:3300/0 conn(0x7fc284070650 0x7fc284159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.938+0000 7fc28ac54640 1 --2- 172.21.15.46:0/1274633540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2840c8460 0x7fc2840715c0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.938+0000 7fc28ac54640 1 -- 172.21.15.46:0/1274633540 >> 172.21.15.46:0/1274633540 conn(0x7fc2840cbeb0 msgr2=0x7fc2840cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:00.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.938+0000 7fc28ac54640 1 -- 172.21.15.46:0/1274633540 shutdown_connections 2023-10-30T13:56:00.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.938+0000 7fc28ac54640 1 -- 172.21.15.46:0/1274633540 wait complete. 2023-10-30T13:56:00.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc28ac54640 1 Processor -- start 2023-10-30T13:56:00.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc28ac54640 1 -- start start 2023-10-30T13:56:00.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc28ac54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc284070650 0x7fc28406b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc28ac54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2840c8460 0x7fc2840af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc28ac54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc284157010 0x7fc2840af710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc28ac54640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2840adb70 con 0x7fc284157010 2023-10-30T13:56:00.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc28ac54640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2840aed80 con 0x7fc2840c8460 2023-10-30T13:56:00.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc28ac54640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2841f1b80 con 0x7fc284070650 2023-10-30T13:56:00.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2891f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc284157010 0x7fc2840af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2891f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc284157010 0x7fc2840af710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52192/0 (socket says 172.21.15.46:52192) 2023-10-30T13:56:00.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2891f1640 1 -- 172.21.15.46:0/3483412336 learned_addr learned my addr 172.21.15.46:0/3483412336 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:00.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2889f0640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc284070650 0x7fc28406b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc283fff640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2840c8460 0x7fc2840af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2891f1640 1 -- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc284070650 msgr2=0x7fc28406b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:00.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2891f1640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc284070650 0x7fc28406b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2891f1640 1 -- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2840c8460 msgr2=0x7fc2840af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:00.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2891f1640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2840c8460 0x7fc2840af1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2891f1640 1 -- 172.21.15.46:0/3483412336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc270009420 con 0x7fc284157010 2023-10-30T13:56:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2891f1640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc284157010 0x7fc2840af710 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fc27c007e60 tx=0x7fc27c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2817fa640 1 -- 172.21.15.46:0/3483412336 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27c014420 con 0x7fc284157010 2023-10-30T13:56:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2817fa640 1 -- 172.21.15.46:0/3483412336 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27c0145a0 con 0x7fc284157010 2023-10-30T13:56:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc2817fa640 1 -- 172.21.15.46:0/3483412336 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27c014950 con 0x7fc284157010 2023-10-30T13:56:00.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc28ac54640 1 -- 172.21.15.46:0/3483412336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2840ae5a0 con 0x7fc284157010 2023-10-30T13:56:00.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc28ac54640 1 -- 172.21.15.46:0/3483412336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2841f2120 con 0x7fc284157010 2023-10-30T13:56:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.942+0000 7fc28ac54640 1 -- 172.21.15.46:0/3483412336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2840aeef0 con 0x7fc284157010 2023-10-30T13:56:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.946+0000 7fc2817fa640 1 -- 172.21.15.46:0/3483412336 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc27c014ab0 con 0x7fc284157010 2023-10-30T13:56:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.946+0000 7fc2817fa640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc2680664a0 0x7fc268068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.946+0000 7fc2817fa640 1 -- 172.21.15.46:0/3483412336 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fc27c084cd0 con 0x7fc284157010 2023-10-30T13:56:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.946+0000 7fc2889f0640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc2680664a0 0x7fc268068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:00.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.946+0000 7fc2889f0640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc2680664a0 0x7fc268068960 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc278007fd0 tx=0x7fc278007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:00.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:00.950+0000 7fc2817fa640 1 -- 172.21.15.46:0/3483412336 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc2840aeef0 con 0x7fc284157010 2023-10-30T13:56:01.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.142+0000 7fc28ac54640 1 -- 172.21.15.46:0/3483412336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2840aeef0 con 0x7fc284157010 2023-10-30T13:56:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.142+0000 7fc2817fa640 1 -- 172.21.15.46:0/3483412336 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+5731 (secure 0 0 0) 0x7fc2840aeef0 con 0x7fc284157010 2023-10-30T13:56:01.146 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:56:01.146 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:55:54.369067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:resolve","state_seq":11,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 146 2023-10-30T13:56:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.146+0000 7fc28ac54640 1 -- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc2680664a0 msgr2=0x7fc268068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.146+0000 7fc28ac54640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc2680664a0 0x7fc268068960 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc278007fd0 tx=0x7fc278007d60 comp rx=0 tx=0).stop 2023-10-30T13:56:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.146+0000 7fc28ac54640 1 -- 172.21.15.46:0/3483412336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc284157010 msgr2=0x7fc2840af710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.146+0000 7fc28ac54640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc284157010 0x7fc2840af710 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fc27c007e60 tx=0x7fc27c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:56:01.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.146+0000 7fc28ac54640 1 -- 172.21.15.46:0/3483412336 shutdown_connections 2023-10-30T13:56:01.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.146+0000 7fc28ac54640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc2680664a0 0x7fc268068960 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:01.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.146+0000 7fc28ac54640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc284157010 0x7fc2840af710 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:01.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.146+0000 7fc28ac54640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2840c8460 0x7fc2840af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:01.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.146+0000 7fc28ac54640 1 --2- 172.21.15.46:0/3483412336 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc284070650 0x7fc28406b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:01.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.146+0000 7fc28ac54640 1 -- 172.21.15.46:0/3483412336 >> 172.21.15.46:0/3483412336 conn(0x7fc2840cbeb0 msgr2=0x7fc284152420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:01.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.146+0000 7fc28ac54640 1 -- 172.21.15.46:0/3483412336 shutdown_connections 2023-10-30T13:56:01.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:01.146+0000 7fc28ac54640 1 -- 172.21.15.46:0/3483412336 wait complete. 2023-10-30T13:56:01.167 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:03.167 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:03.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.318+0000 7f34b8ec9640 1 Processor -- start 2023-10-30T13:56:03.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.318+0000 7f34b8ec9640 1 -- start start 2023-10-30T13:56:03.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b8ec9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34b406f7a0 0x7f34b406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:03.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b8ec9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34b4071110 0x7f34b4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:03.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b8ec9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b4066cf0 con 0x7f34b40c6bd0 2023-10-30T13:56:03.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b8ec9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b40700c0 con 0x7f34b406f7a0 2023-10-30T13:56:03.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b8ec9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40c70d0 con 0x7f34b4071110 2023-10-30T13:56:03.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34b406f7a0 0x7f34b406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:03.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34abfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f34b40c6bd0 0x7f34b4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44782/0 (socket says 172.21.15.46:44782) 2023-10-30T13:56:03.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34b406f7a0 0x7f34b406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52198/0 (socket says 172.21.15.46:52198) 2023-10-30T13:56:03.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34abfff640 1 -- 172.21.15.46:0/366935077 learned_addr learned my addr 172.21.15.46:0/366935077 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:56:03.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b2d9d640 1 --2- 172.21.15.46:0/366935077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34b4071110 0x7f34b4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:03.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b259c640 1 -- 172.21.15.46:0/366935077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34b4071110 msgr2=0x7f34b4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:03.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b259c640 1 --2- 172.21.15.46:0/366935077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34b4071110 0x7f34b4158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b259c640 1 -- 172.21.15.46:0/366935077 >> v1:172.21.15.46:6789/0 conn(0x7f34b40c6bd0 legacy=0x7f34b4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:03.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b1d9b640 1 -- 172.21.15.46:0/366935077 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175342759 0 0) 0x7f34b4066cf0 con 0x7f34b40c6bd0 2023-10-30T13:56:03.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.322+0000 7f34b259c640 1 -- 172.21.15.46:0/366935077 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b40c7390 con 0x7f34b406f7a0 2023-10-30T13:56:03.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b259c640 1 --2- 172.21.15.46:0/366935077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b406f7a0 0x7f34b406fb80 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f34a40097c0 tx=0x7f34a401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3655cb699c3f8e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:03.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b1d9b640 1 -- 172.21.15.46:0/366935077 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a40a8970 con 0x7f34b406f7a0 2023-10-30T13:56:03.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b1d9b640 1 -- 172.21.15.46:0/366935077 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a40a8ad0 con 0x7f34b406f7a0 2023-10-30T13:56:03.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b1d9b640 1 -- 172.21.15.46:0/366935077 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a40a8e60 con 0x7f34b406f7a0 2023-10-30T13:56:03.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b8ec9640 1 -- 172.21.15.46:0/366935077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b406f7a0 msgr2=0x7f34b406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:03.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b8ec9640 1 --2- 172.21.15.46:0/366935077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b406f7a0 0x7f34b406fb80 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f34a40097c0 tx=0x7f34a401ec70 comp rx=0 tx=0).stop 2023-10-30T13:56:03.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b8ec9640 1 -- 172.21.15.46:0/366935077 shutdown_connections 2023-10-30T13:56:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b8ec9640 1 --2- 172.21.15.46:0/366935077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34b4071110 0x7f34b4158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b8ec9640 1 --2- 172.21.15.46:0/366935077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b406f7a0 0x7f34b406fb80 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b8ec9640 1 -- 172.21.15.46:0/366935077 >> 172.21.15.46:0/366935077 conn(0x7f34b40c7910 msgr2=0x7f34b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:03.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b8ec9640 1 -- 172.21.15.46:0/366935077 shutdown_connections 2023-10-30T13:56:03.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b8ec9640 1 -- 172.21.15.46:0/366935077 wait complete. 2023-10-30T13:56:03.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b8ec9640 1 Processor -- start 2023-10-30T13:56:03.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.326+0000 7f34b8ec9640 1 -- start start 2023-10-30T13:56:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b8ec9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34b406f7a0 0x7f34b40c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b8ec9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34b4071110 0x7f34b41f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b8ec9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40c6bd0 0x7f34b40c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b8ec9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b41f1ba0 con 0x7f34b40c6bd0 2023-10-30T13:56:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b8ec9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b41f2b30 con 0x7f34b4071110 2023-10-30T13:56:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b8ec9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40aec90 con 0x7f34b406f7a0 2023-10-30T13:56:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40c6bd0 0x7f34b40c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:03.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40c6bd0 0x7f34b40c38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52212/0 (socket says 172.21.15.46:52212) 2023-10-30T13:56:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b2d9d640 1 -- 172.21.15.46:0/2750489388 learned_addr learned my addr 172.21.15.46:0/2750489388 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b259c640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34b406f7a0 0x7f34b40c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34abfff640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34b4071110 0x7f34b41f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:03.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b2d9d640 1 -- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34b406f7a0 msgr2=0x7f34b40c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b2d9d640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34b406f7a0 0x7f34b40c9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b2d9d640 1 -- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34b4071110 msgr2=0x7f34b41f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b2d9d640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34b4071110 0x7f34b41f25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b2d9d640 1 -- 172.21.15.46:0/2750489388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a4009420 con 0x7f34b40c6bd0 2023-10-30T13:56:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34abfff640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34b4071110 0x7f34b41f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b2d9d640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40c6bd0 0x7f34b40c38b0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f34ac007e70 tx=0x7f34ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34aaffd640 1 -- 172.21.15.46:0/2750489388 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac014620 con 0x7f34b40c6bd0 2023-10-30T13:56:03.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34aaffd640 1 -- 172.21.15.46:0/2750489388 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ac0147a0 con 0x7f34b40c6bd0 2023-10-30T13:56:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34aaffd640 1 -- 172.21.15.46:0/2750489388 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac014b50 con 0x7f34b40c6bd0 2023-10-30T13:56:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b8ec9640 1 -- 172.21.15.46:0/2750489388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b40ca4e0 con 0x7f34b40c6bd0 2023-10-30T13:56:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b8ec9640 1 -- 172.21.15.46:0/2750489388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b40caaa0 con 0x7f34b40c6bd0 2023-10-30T13:56:03.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34b8ec9640 1 -- 172.21.15.46:0/2750489388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b404e170 con 0x7f34b40c6bd0 2023-10-30T13:56:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.330+0000 7f34aaffd640 1 -- 172.21.15.46:0/2750489388 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34ac014cb0 con 0x7f34b40c6bd0 2023-10-30T13:56:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.334+0000 7f34aaffd640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f34940664a0 0x7f3494068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.334+0000 7f34aaffd640 1 -- 172.21.15.46:0/2750489388 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f34ac084db0 con 0x7f34b40c6bd0 2023-10-30T13:56:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.334+0000 7f34b259c640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f34940664a0 0x7f3494068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:03.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.334+0000 7f34b259c640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f34940664a0 0x7f3494068960 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f34a40b2000 tx=0x7f34a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:03.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.334+0000 7f34aaffd640 1 -- 172.21.15.46:0/2750489388 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f34ac089070 con 0x7f34b40c6bd0 2023-10-30T13:56:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.526+0000 7f34b8ec9640 1 -- 172.21.15.46:0/2750489388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34b40c4340 con 0x7f34b40c6bd0 2023-10-30T13:56:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.526+0000 7f34aaffd640 1 -- 172.21.15.46:0/2750489388 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f34ac051bd0 con 0x7f34b40c6bd0 2023-10-30T13:56:03.530 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:56:03.530 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:00.466836+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 147 2023-10-30T13:56:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.530+0000 7f34b8ec9640 1 -- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f34940664a0 msgr2=0x7f3494068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.530+0000 7f34b8ec9640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f34940664a0 0x7f3494068960 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f34a40b2000 tx=0x7f34a4008040 comp rx=0 tx=0).stop 2023-10-30T13:56:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.530+0000 7f34b8ec9640 1 -- 172.21.15.46:0/2750489388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40c6bd0 msgr2=0x7f34b40c38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.530+0000 7f34b8ec9640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40c6bd0 0x7f34b40c38b0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f34ac007e70 tx=0x7f34ac00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.530+0000 7f34b8ec9640 1 -- 172.21.15.46:0/2750489388 shutdown_connections 2023-10-30T13:56:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.530+0000 7f34b8ec9640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f34940664a0 0x7f3494068960 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.530+0000 7f34b8ec9640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40c6bd0 0x7f34b40c38b0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.530+0000 7f34b8ec9640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34b4071110 0x7f34b41f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.530+0000 7f34b8ec9640 1 --2- 172.21.15.46:0/2750489388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34b406f7a0 0x7f34b40c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.530+0000 7f34b8ec9640 1 -- 172.21.15.46:0/2750489388 >> 172.21.15.46:0/2750489388 conn(0x7f34b40c7910 msgr2=0x7f34b40c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.530+0000 7f34b8ec9640 1 -- 172.21.15.46:0/2750489388 shutdown_connections 2023-10-30T13:56:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:03.530+0000 7f34b8ec9640 1 -- 172.21.15.46:0/2750489388 wait complete. 2023-10-30T13:56:03.551 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:05.552 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:05.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.702+0000 7f318b3d3640 1 Processor -- start 2023-10-30T13:56:05.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.702+0000 7f318b3d3640 1 -- start start 2023-10-30T13:56:05.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318b3d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3184069e90 0x7f318415a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:05.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318b3d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f318406a270 0x7f31840c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:05.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318b3d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31840c6c10 con 0x7f318406cb80 2023-10-30T13:56:05.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318b3d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31840c8570 con 0x7f318406a270 2023-10-30T13:56:05.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318b3d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31840c6ef0 con 0x7f3184069e90 2023-10-30T13:56:05.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318896e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f318406a270 0x7f31840c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:05.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318896e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f318406a270 0x7f31840c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52224/0 (socket says 172.21.15.46:52224) 2023-10-30T13:56:05.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318896e640 1 -- 172.21.15.46:0/3568817022 learned_addr learned my addr 172.21.15.46:0/3568817022 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:05.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318916f640 1 --2- 172.21.15.46:0/3568817022 >> v2:172.21.15.154:3300/0 conn(0x7f3184069e90 0x7f318415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:05.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318896e640 1 -- 172.21.15.46:0/3568817022 >> v2:172.21.15.154:3300/0 conn(0x7f3184069e90 msgr2=0x7f318415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:05.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318896e640 1 --2- 172.21.15.46:0/3568817022 >> v2:172.21.15.154:3300/0 conn(0x7f3184069e90 0x7f318415a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318896e640 1 -- 172.21.15.46:0/3568817022 >> v1:172.21.15.154:6789/0 conn(0x7f318406cb80 legacy=0x7f318406cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:05.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.706+0000 7f318896e640 1 -- 172.21.15.46:0/3568817022 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31840c71d0 con 0x7f318406a270 2023-10-30T13:56:05.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f318896e640 1 --2- 172.21.15.46:0/3568817022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f318406a270 0x7f31840c9cb0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f318406ed20 tx=0x7f317401c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d108cf2dee976b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:05.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f317bfff640 1 -- 172.21.15.46:0/3568817022 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31740ac410 con 0x7f318406a270 2023-10-30T13:56:05.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f317bfff640 1 -- 172.21.15.46:0/3568817022 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31740ac570 con 0x7f318406a270 2023-10-30T13:56:05.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f317bfff640 1 -- 172.21.15.46:0/3568817022 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31740ac900 con 0x7f318406a270 2023-10-30T13:56:05.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f318b3d3640 1 -- 172.21.15.46:0/3568817022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f318406a270 msgr2=0x7f31840c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:05.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f318b3d3640 1 --2- 172.21.15.46:0/3568817022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f318406a270 0x7f31840c9cb0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f318406ed20 tx=0x7f317401c4c0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f318b3d3640 1 -- 172.21.15.46:0/3568817022 shutdown_connections 2023-10-30T13:56:05.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f318b3d3640 1 --2- 172.21.15.46:0/3568817022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f318406a270 0x7f31840c9cb0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f318b3d3640 1 --2- 172.21.15.46:0/3568817022 >> v2:172.21.15.154:3300/0 conn(0x7f3184069e90 0x7f318415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f318b3d3640 1 -- 172.21.15.46:0/3568817022 >> 172.21.15.46:0/3568817022 conn(0x7f318406dfd0 msgr2=0x7f318406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f318b3d3640 1 -- 172.21.15.46:0/3568817022 shutdown_connections 2023-10-30T13:56:05.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f318b3d3640 1 -- 172.21.15.46:0/3568817022 wait complete. 2023-10-30T13:56:05.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f318b3d3640 1 Processor -- start 2023-10-30T13:56:05.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.710+0000 7f318b3d3640 1 -- start start 2023-10-30T13:56:05.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318b3d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3184069e90 0x7f31841f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:05.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318b3d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f318406a270 0x7f31841492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318b3d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f318406cb80 0x7f3184071690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318b3d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318415b250 con 0x7f318406a270 2023-10-30T13:56:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318b3d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3184152520 con 0x7f318406cb80 2023-10-30T13:56:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318b3d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3184151220 con 0x7f3184069e90 2023-10-30T13:56:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318896e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f318406a270 0x7f31841492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:05.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318896e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f318406a270 0x7f31841492b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52226/0 (socket says 172.21.15.46:52226) 2023-10-30T13:56:05.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318896e640 1 -- 172.21.15.46:0/3069503938 learned_addr learned my addr 172.21.15.46:0/3069503938 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:05.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318916f640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3184069e90 0x7f31841f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:05.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f3189970640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f318406cb80 0x7f3184071690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:05.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318896e640 1 -- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3184069e90 msgr2=0x7f31841f2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:05.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318896e640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3184069e90 0x7f31841f2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318896e640 1 -- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f318406cb80 msgr2=0x7f3184071690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:05.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318896e640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f318406cb80 0x7f3184071690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318896e640 1 -- 172.21.15.46:0/3069503938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31740094d0 con 0x7f318406a270 2023-10-30T13:56:05.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318896e640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f318406a270 0x7f31841492b0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f317401a0f0 tx=0x7f3174019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:05.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f3179ffb640 1 -- 172.21.15.46:0/3069503938 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31740ac410 con 0x7f318406a270 2023-10-30T13:56:05.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f3179ffb640 1 -- 172.21.15.46:0/3069503938 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31740b65f0 con 0x7f318406a270 2023-10-30T13:56:05.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318b3d3640 1 -- 172.21.15.46:0/3069503938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3184152770 con 0x7f318406a270 2023-10-30T13:56:05.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f3179ffb640 1 -- 172.21.15.46:0/3069503938 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31740b69c0 con 0x7f318406a270 2023-10-30T13:56:05.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318b3d3640 1 -- 172.21.15.46:0/3069503938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3184150eb0 con 0x7f318406a270 2023-10-30T13:56:05.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f318b3d3640 1 -- 172.21.15.46:0/3069503938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3184152c90 con 0x7f318406a270 2023-10-30T13:56:05.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.714+0000 7f3179ffb640 1 -- 172.21.15.46:0/3069503938 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f31740b6b20 con 0x7f318406a270 2023-10-30T13:56:05.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.718+0000 7f3179ffb640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31680664a0 0x7f3168068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:05.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.718+0000 7f3179ffb640 1 -- 172.21.15.46:0/3069503938 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f317411ef10 con 0x7f318406a270 2023-10-30T13:56:05.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.718+0000 7f318916f640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31680664a0 0x7f3168068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:05.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.718+0000 7f318916f640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31680664a0 0x7f3168068960 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3180007a60 tx=0x7f3180008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:05.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.722+0000 7f3179ffb640 1 -- 172.21.15.46:0/3069503938 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31740ebe50 con 0x7f318406a270 2023-10-30T13:56:05.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.910+0000 7f318b3d3640 1 -- 172.21.15.46:0/3069503938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3184151390 con 0x7f318406a270 2023-10-30T13:56:05.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.910+0000 7f3179ffb640 1 -- 172.21.15.46:0/3069503938 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f31740b1090 con 0x7f318406a270 2023-10-30T13:56:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:56:05.913 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:00.466836+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:05.914 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 147 2023-10-30T13:56:05.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.910+0000 7f318b3d3640 1 -- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31680664a0 msgr2=0x7f3168068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:05.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.910+0000 7f318b3d3640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31680664a0 0x7f3168068960 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3180007a60 tx=0x7f3180008040 comp rx=0 tx=0).stop 2023-10-30T13:56:05.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.910+0000 7f318b3d3640 1 -- 172.21.15.46:0/3069503938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f318406a270 msgr2=0x7f31841492b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:05.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.910+0000 7f318b3d3640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f318406a270 0x7f31841492b0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f317401a0f0 tx=0x7f3174019260 comp rx=0 tx=0).stop 2023-10-30T13:56:05.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.914+0000 7f318b3d3640 1 -- 172.21.15.46:0/3069503938 shutdown_connections 2023-10-30T13:56:05.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.914+0000 7f318b3d3640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31680664a0 0x7f3168068960 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.914+0000 7f318b3d3640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f318406cb80 0x7f3184071690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.914+0000 7f318b3d3640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f318406a270 0x7f31841492b0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.914+0000 7f318b3d3640 1 --2- 172.21.15.46:0/3069503938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3184069e90 0x7f31841f2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:05.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.914+0000 7f318b3d3640 1 -- 172.21.15.46:0/3069503938 >> 172.21.15.46:0/3069503938 conn(0x7f318406dfd0 msgr2=0x7f318414f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:05.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.914+0000 7f318b3d3640 1 -- 172.21.15.46:0/3069503938 shutdown_connections 2023-10-30T13:56:05.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:05.914+0000 7f318b3d3640 1 -- 172.21.15.46:0/3069503938 wait complete. 2023-10-30T13:56:05.935 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:07.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:08.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.090+0000 7f6340fd3640 1 Processor -- start 2023-10-30T13:56:08.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.090+0000 7f6340fd3640 1 -- start start 2023-10-30T13:56:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f6340fd3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f633c153bf0 0x7f633c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f6340fd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f633c0c6fe0 0x7f633c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f6340fd3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f633c0c79e0 0x7f633c0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f6340fd3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f633c0c6810 con 0x7f633c0c79e0 2023-10-30T13:56:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f6340fd3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f633c159900 con 0x7f633c153bf0 2023-10-30T13:56:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f6340fd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633c0700b0 con 0x7f633c0c6fe0 2023-10-30T13:56:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f633ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f633c0c79e0 0x7f633c0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f633ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f633c0c79e0 0x7f633c0cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60508/0 (socket says 172.21.15.46:60508) 2023-10-30T13:56:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f633ad9d640 1 -- 172.21.15.46:0/1774748917 learned_addr learned my addr 172.21.15.46:0/1774748917 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f633a59c640 1 --2- 172.21.15.46:0/1774748917 >> v2:172.21.15.154:3300/0 conn(0x7f633c153bf0 0x7f633c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f6333fff640 1 --2- 172.21.15.46:0/1774748917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f633c0c6fe0 0x7f633c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f633ad9d640 1 -- 172.21.15.46:0/1774748917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f633c0c6fe0 msgr2=0x7f633c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f633ad9d640 1 --2- 172.21.15.46:0/1774748917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f633c0c6fe0 0x7f633c0c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f633ad9d640 1 -- 172.21.15.46:0/1774748917 >> v2:172.21.15.154:3300/0 conn(0x7f633c153bf0 msgr2=0x7f633c071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f633ad9d640 1 --2- 172.21.15.46:0/1774748917 >> v2:172.21.15.154:3300/0 conn(0x7f633c153bf0 0x7f633c071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f633ad9d640 1 -- 172.21.15.46:0/1774748917 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c0c6530 con 0x7f633c0c79e0 2023-10-30T13:56:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f633a59c640 1 --2- 172.21.15.46:0/1774748917 >> v2:172.21.15.154:3300/0 conn(0x7f633c153bf0 0x7f633c071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.094+0000 7f633ad9d640 1 --2- 172.21.15.46:0/1774748917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633c0c79e0 0x7f633c0cb570 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f633c0c9150 tx=0x7f633401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6992ae7b93b156bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6339d9b640 1 -- 172.21.15.46:0/1774748917 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63340af400 con 0x7f633c0c79e0 2023-10-30T13:56:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6339d9b640 1 -- 172.21.15.46:0/1774748917 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63340ab050 con 0x7f633c0c79e0 2023-10-30T13:56:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6339d9b640 1 -- 172.21.15.46:0/1774748917 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63340af780 con 0x7f633c0c79e0 2023-10-30T13:56:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 -- 172.21.15.46:0/1774748917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633c0c79e0 msgr2=0x7f633c0cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 --2- 172.21.15.46:0/1774748917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633c0c79e0 0x7f633c0cb570 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f633c0c9150 tx=0x7f633401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 -- 172.21.15.46:0/1774748917 shutdown_connections 2023-10-30T13:56:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 --2- 172.21.15.46:0/1774748917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633c0c79e0 0x7f633c0cb570 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 --2- 172.21.15.46:0/1774748917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f633c0c6fe0 0x7f633c0c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 --2- 172.21.15.46:0/1774748917 >> v2:172.21.15.154:3300/0 conn(0x7f633c153bf0 0x7f633c071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 -- 172.21.15.46:0/1774748917 >> 172.21.15.46:0/1774748917 conn(0x7f633c06c6e0 msgr2=0x7f633c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 -- 172.21.15.46:0/1774748917 shutdown_connections 2023-10-30T13:56:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 -- 172.21.15.46:0/1774748917 wait complete. 2023-10-30T13:56:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 Processor -- start 2023-10-30T13:56:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 -- start start 2023-10-30T13:56:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f633c0c6fe0 0x7f633c1561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633c0c79e0 0x7f633c14dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f633c153bf0 0x7f633c150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c154580 con 0x7f633c0c79e0 2023-10-30T13:56:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c154310 con 0x7f633c0c6fe0 2023-10-30T13:56:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.098+0000 7f6340fd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633c14fe20 con 0x7f633c153bf0 2023-10-30T13:56:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6333fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633c0c79e0 0x7f633c14dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6333fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633c0c79e0 0x7f633c14dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60514/0 (socket says 172.21.15.46:60514) 2023-10-30T13:56:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6333fff640 1 -- 172.21.15.46:0/3723437379 learned_addr learned my addr 172.21.15.46:0/3723437379 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f633a59c640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f633c0c6fe0 0x7f633c1561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f633ad9d640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f633c153bf0 0x7f633c150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6333fff640 1 -- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f633c153bf0 msgr2=0x7f633c150be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6333fff640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f633c153bf0 0x7f633c150be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6333fff640 1 -- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f633c0c6fe0 msgr2=0x7f633c1561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6333fff640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f633c0c6fe0 0x7f633c1561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6333fff640 1 -- 172.21.15.46:0/3723437379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6334009b30 con 0x7f633c0c79e0 2023-10-30T13:56:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6333fff640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633c0c79e0 0x7f633c14dda0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f632400d830 tx=0x7f632400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6332ffd640 1 -- 172.21.15.46:0/3723437379 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63240096a0 con 0x7f633c0c79e0 2023-10-30T13:56:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6332ffd640 1 -- 172.21.15.46:0/3723437379 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6324009800 con 0x7f633c0c79e0 2023-10-30T13:56:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6332ffd640 1 -- 172.21.15.46:0/3723437379 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324009b70 con 0x7f633c0c79e0 2023-10-30T13:56:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6340fd3640 1 -- 172.21.15.46:0/3723437379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633c150100 con 0x7f633c0c79e0 2023-10-30T13:56:08.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6340fd3640 1 -- 172.21.15.46:0/3723437379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633c1506c0 con 0x7f633c0c79e0 2023-10-30T13:56:08.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6340fd3640 1 -- 172.21.15.46:0/3723437379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633c04e170 con 0x7f633c0c79e0 2023-10-30T13:56:08.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.102+0000 7f6332ffd640 1 -- 172.21.15.46:0/3723437379 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6324014410 con 0x7f633c0c79e0 2023-10-30T13:56:08.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.106+0000 7f6332ffd640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f631c0664d0 0x7f631c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.106+0000 7f6332ffd640 1 -- 172.21.15.46:0/3723437379 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f6324084cd0 con 0x7f633c0c79e0 2023-10-30T13:56:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.106+0000 7f633a59c640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f631c0664d0 0x7f631c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.106+0000 7f633a59c640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f631c0664d0 0x7f631c068990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f632c0045f0 tx=0x7f632c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.110+0000 7f6332ffd640 1 -- 172.21.15.46:0/3723437379 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6324051ba0 con 0x7f633c0c79e0 2023-10-30T13:56:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 -- 172.21.15.46:0/3723437379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f633c06fe90 con 0x7f633c0c79e0 2023-10-30T13:56:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6332ffd640 1 -- 172.21.15.46:0/3723437379 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f63240551c0 con 0x7f633c0c79e0 2023-10-30T13:56:08.301 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:56:08.301 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:06.421046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 148 2023-10-30T13:56:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 -- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f631c0664d0 msgr2=0x7f631c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f631c0664d0 0x7f631c068990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f632c0045f0 tx=0x7f632c008040 comp rx=0 tx=0).stop 2023-10-30T13:56:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 -- 172.21.15.46:0/3723437379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633c0c79e0 msgr2=0x7f633c14dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633c0c79e0 0x7f633c14dda0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f632400d830 tx=0x7f632400dcf0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 -- 172.21.15.46:0/3723437379 shutdown_connections 2023-10-30T13:56:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f631c0664d0 0x7f631c068990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f633c153bf0 0x7f633c150be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633c0c79e0 0x7f633c14dda0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 --2- 172.21.15.46:0/3723437379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f633c0c6fe0 0x7f633c1561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 -- 172.21.15.46:0/3723437379 >> 172.21.15.46:0/3723437379 conn(0x7f633c06c6e0 msgr2=0x7f633c06e290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:08.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 -- 172.21.15.46:0/3723437379 shutdown_connections 2023-10-30T13:56:08.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:08.298+0000 7f6340fd3640 1 -- 172.21.15.46:0/3723437379 wait complete. 2023-10-30T13:56:08.322 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:10.323 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:10.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.474+0000 7f0536965640 1 Processor -- start 2023-10-30T13:56:10.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.474+0000 7f0536965640 1 -- start start 2023-10-30T13:56:10.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f0536965640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f05300c6b80 0x7f05300bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:10.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f0536965640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05300c0a50 con 0x7f05300c6f60 2023-10-30T13:56:10.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f0536965640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053006f6e0 con 0x7f053013f4a0 2023-10-30T13:56:10.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f0536965640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f053006f9c0 con 0x7f05300c6b80 2023-10-30T13:56:10.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f052f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f05300c6f60 0x7f053013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42452/0 (socket says 172.21.15.46:42452) 2023-10-30T13:56:10.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f052f7fe640 1 -- 172.21.15.46:0/3871835369 learned_addr learned my addr 172.21.15.46:0/3871835369 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:56:10.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f052ffff640 1 --2- 172.21.15.46:0/3871835369 >> v2:172.21.15.154:3300/0 conn(0x7f05300c6b80 0x7f05300bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:10.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f052effd640 1 -- 172.21.15.46:0/3871835369 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872038825 0 0) 0x7f05300c0a50 con 0x7f05300c6f60 2023-10-30T13:56:10.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f052effd640 1 -- 172.21.15.46:0/3871835369 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0518003530 con 0x7f05300c6f60 2023-10-30T13:56:10.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f052effd640 1 -- 172.21.15.46:0/3871835369 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3275477001 0 0) 0x7f0518003530 con 0x7f05300c6f60 2023-10-30T13:56:10.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f052ffff640 1 -- 172.21.15.46:0/3871835369 >> v1:172.21.15.154:6789/0 conn(0x7f053013f4a0 legacy=0x7f05301566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:10.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f052ffff640 1 -- 172.21.15.46:0/3871835369 >> v1:172.21.15.46:6789/0 conn(0x7f05300c6f60 legacy=0x7f053013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:10.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.478+0000 7f052ffff640 1 -- 172.21.15.46:0/3871835369 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053006fca0 con 0x7f05300c6b80 2023-10-30T13:56:10.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f052ffff640 1 --2- 172.21.15.46:0/3871835369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05300c6b80 0x7f05300bfec0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f05200097c0 tx=0x7f052001c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13a5f9ad1cbe08d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:10.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f052effd640 1 -- 172.21.15.46:0/3871835369 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524001680 con 0x7f05300c6b80 2023-10-30T13:56:10.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f052effd640 1 -- 172.21.15.46:0/3871835369 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05200a7050 con 0x7f05300c6b80 2023-10-30T13:56:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f052effd640 1 -- 172.21.15.46:0/3871835369 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05200ab600 con 0x7f05300c6b80 2023-10-30T13:56:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f0536965640 1 -- 172.21.15.46:0/3871835369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05300c6b80 msgr2=0x7f05300bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f0536965640 1 --2- 172.21.15.46:0/3871835369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05300c6b80 0x7f05300bfec0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f05200097c0 tx=0x7f052001c3a0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f0536965640 1 -- 172.21.15.46:0/3871835369 shutdown_connections 2023-10-30T13:56:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f0536965640 1 --2- 172.21.15.46:0/3871835369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05300c6b80 0x7f05300bfec0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f0536965640 1 -- 172.21.15.46:0/3871835369 >> 172.21.15.46:0/3871835369 conn(0x7f05300c7910 msgr2=0x7f05300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f0536965640 1 -- 172.21.15.46:0/3871835369 shutdown_connections 2023-10-30T13:56:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f0536965640 1 -- 172.21.15.46:0/3871835369 wait complete. 2023-10-30T13:56:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f0536965640 1 Processor -- start 2023-10-30T13:56:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.482+0000 7f0536965640 1 -- start start 2023-10-30T13:56:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f0536965640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05300c6b80 0x7f053013d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f0536965640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300c6f60 0x7f05300c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f0536965640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f053013f4a0 0x7f05300c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:10.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f0536965640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053013bab0 con 0x7f05300c6f60 2023-10-30T13:56:10.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f0536965640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053013c350 con 0x7f053013f4a0 2023-10-30T13:56:10.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f0536965640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053013a100 con 0x7f05300c6b80 2023-10-30T13:56:10.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300c6f60 0x7f05300c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:10.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300c6f60 0x7f05300c5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60524/0 (socket says 172.21.15.46:60524) 2023-10-30T13:56:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052f7fe640 1 -- 172.21.15.46:0/933141323 learned_addr learned my addr 172.21.15.46:0/933141323 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052ffff640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05300c6b80 0x7f053013d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f0534f02640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f053013f4a0 0x7f05300c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052f7fe640 1 -- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05300c6b80 msgr2=0x7f053013d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052f7fe640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05300c6b80 0x7f053013d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052f7fe640 1 -- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f053013f4a0 msgr2=0x7f05300c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:10.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052f7fe640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f053013f4a0 0x7f05300c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052f7fe640 1 -- 172.21.15.46:0/933141323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0520009420 con 0x7f05300c6f60 2023-10-30T13:56:10.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f0534f02640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f053013f4a0 0x7f05300c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:10.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052f7fe640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300c6f60 0x7f05300c5a70 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f052400c760 tx=0x7f052400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:10.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052cff9640 1 -- 172.21.15.46:0/933141323 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524005ae0 con 0x7f05300c6f60 2023-10-30T13:56:10.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052cff9640 1 -- 172.21.15.46:0/933141323 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0524005c40 con 0x7f05300c6f60 2023-10-30T13:56:10.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f0536965640 1 -- 172.21.15.46:0/933141323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0530136890 con 0x7f05300c6f60 2023-10-30T13:56:10.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f052cff9640 1 -- 172.21.15.46:0/933141323 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524007070 con 0x7f05300c6f60 2023-10-30T13:56:10.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f0536965640 1 -- 172.21.15.46:0/933141323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053013bd00 con 0x7f05300c6f60 2023-10-30T13:56:10.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.486+0000 7f0536965640 1 -- 172.21.15.46:0/933141323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053013c4c0 con 0x7f05300c6f60 2023-10-30T13:56:10.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.490+0000 7f052cff9640 1 -- 172.21.15.46:0/933141323 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f05240071d0 con 0x7f05300c6f60 2023-10-30T13:56:10.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.490+0000 7f052cff9640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f051c0664d0 0x7f051c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:10.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.490+0000 7f052cff9640 1 -- 172.21.15.46:0/933141323 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f05240841a0 con 0x7f05300c6f60 2023-10-30T13:56:10.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.490+0000 7f052ffff640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f051c0664d0 0x7f051c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:10.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.490+0000 7f052ffff640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f051c0664d0 0x7f051c068990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f05200ad000 tx=0x7f05200a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:10.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.494+0000 7f052cff9640 1 -- 172.21.15.46:0/933141323 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0524054af0 con 0x7f05300c6f60 2023-10-30T13:56:10.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 -- 172.21.15.46:0/933141323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f053004e170 con 0x7f05300c6f60 2023-10-30T13:56:10.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f052cff9640 1 -- 172.21.15.46:0/933141323 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f0524054680 con 0x7f05300c6f60 2023-10-30T13:56:10.685 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:56:10.685 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:06.421046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:10.687 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 148 2023-10-30T13:56:10.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 -- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f051c0664d0 msgr2=0x7f051c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:10.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f051c0664d0 0x7f051c068990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f05200ad000 tx=0x7f05200a6000 comp rx=0 tx=0).stop 2023-10-30T13:56:10.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 -- 172.21.15.46:0/933141323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300c6f60 msgr2=0x7f05300c5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:10.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300c6f60 0x7f05300c5a70 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f052400c760 tx=0x7f052400cc20 comp rx=0 tx=0).stop 2023-10-30T13:56:10.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 -- 172.21.15.46:0/933141323 shutdown_connections 2023-10-30T13:56:10.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f051c0664d0 0x7f051c068990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f053013f4a0 0x7f05300c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05300c6f60 0x7f05300c5a70 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 --2- 172.21.15.46:0/933141323 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05300c6b80 0x7f053013d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:10.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 -- 172.21.15.46:0/933141323 >> 172.21.15.46:0/933141323 conn(0x7f05300c7910 msgr2=0x7f05300c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:10.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 -- 172.21.15.46:0/933141323 shutdown_connections 2023-10-30T13:56:10.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:10.682+0000 7f0536965640 1 -- 172.21.15.46:0/933141323 wait complete. 2023-10-30T13:56:10.706 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 148, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 148, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7356": { "addr": "172.21.15.46:6837/3112266825", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3112266825, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3112266825, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7356, "incarnation": 139, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 27 }, "gid_7518": { "addr": "172.21.15.46:6833/157453225", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 157453225, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 157453225, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7518, "incarnation": 145, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:rejoin", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:56:06.421046+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7356, "mds_1": 7518 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:56:12.708 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:12.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.857+0000 7f24cdfaa640 1 Processor -- start 2023-10-30T13:56:12.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24cdfaa640 1 -- start start 2023-10-30T13:56:12.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24cdfaa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24c806f7a0 0x7f24c806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:12.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24cdfaa640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f24c80c6bd0 0x7f24c8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:12.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24cdfaa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24c8071110 0x7f24c8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:12.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24cdfaa640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c80700c0 con 0x7f24c8071110 2023-10-30T13:56:12.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24cdfaa640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c806a3d0 con 0x7f24c80c6bd0 2023-10-30T13:56:12.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24cdfaa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c8066cf0 con 0x7f24c806f7a0 2023-10-30T13:56:12.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24c7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24c8071110 0x7f24c8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:12.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24c7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24c8071110 0x7f24c8158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60532/0 (socket says 172.21.15.46:60532) 2023-10-30T13:56:12.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24c7fff640 1 -- 172.21.15.46:0/514600293 learned_addr learned my addr 172.21.15.46:0/514600293 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:12.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24c77fe640 1 --2- 172.21.15.46:0/514600293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24c806f7a0 0x7f24c806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:12.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24c6ffd640 1 --2- 172.21.15.46:0/514600293 >> v2:172.21.15.154:3300/0 conn(0x7f24c80c6bd0 0x7f24c8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:12.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24c7fff640 1 -- 172.21.15.46:0/514600293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24c806f7a0 msgr2=0x7f24c806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:12.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24c7fff640 1 --2- 172.21.15.46:0/514600293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24c806f7a0 0x7f24c806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24c7fff640 1 -- 172.21.15.46:0/514600293 >> v2:172.21.15.154:3300/0 conn(0x7f24c80c6bd0 msgr2=0x7f24c8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:12.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24c7fff640 1 --2- 172.21.15.46:0/514600293 >> v2:172.21.15.154:3300/0 conn(0x7f24c80c6bd0 0x7f24c8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.861+0000 7f24c7fff640 1 -- 172.21.15.46:0/514600293 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c80c70d0 con 0x7f24c8071110 2023-10-30T13:56:12.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24c7fff640 1 --2- 172.21.15.46:0/514600293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c8071110 0x7f24c8158110 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f24c80b1210 tx=0x7f24c001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9745a38a5d6f77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:12.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24c67fc640 1 -- 172.21.15.46:0/514600293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00af400 con 0x7f24c8071110 2023-10-30T13:56:12.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24c67fc640 1 -- 172.21.15.46:0/514600293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c00ab050 con 0x7f24c8071110 2023-10-30T13:56:12.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24c67fc640 1 -- 172.21.15.46:0/514600293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00af780 con 0x7f24c8071110 2023-10-30T13:56:12.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24cdfaa640 1 -- 172.21.15.46:0/514600293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c8071110 msgr2=0x7f24c8158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:12.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24cdfaa640 1 --2- 172.21.15.46:0/514600293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c8071110 0x7f24c8158110 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f24c80b1210 tx=0x7f24c001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24cdfaa640 1 -- 172.21.15.46:0/514600293 shutdown_connections 2023-10-30T13:56:12.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24cdfaa640 1 --2- 172.21.15.46:0/514600293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c8071110 0x7f24c8158110 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24cdfaa640 1 --2- 172.21.15.46:0/514600293 >> v2:172.21.15.154:3300/0 conn(0x7f24c80c6bd0 0x7f24c8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24cdfaa640 1 --2- 172.21.15.46:0/514600293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24c806f7a0 0x7f24c806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24cdfaa640 1 -- 172.21.15.46:0/514600293 >> 172.21.15.46:0/514600293 conn(0x7f24c80c7910 msgr2=0x7f24c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:12.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24cdfaa640 1 -- 172.21.15.46:0/514600293 shutdown_connections 2023-10-30T13:56:12.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24cdfaa640 1 -- 172.21.15.46:0/514600293 wait complete. 2023-10-30T13:56:12.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.865+0000 7f24cdfaa640 1 Processor -- start 2023-10-30T13:56:12.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24cdfaa640 1 -- start start 2023-10-30T13:56:12.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24cdfaa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c806f7a0 0x7f24c80c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:12.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24cdfaa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24c8071110 0x7f24c81f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24cdfaa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24c80c6bd0 0x7f24c80c1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24cdfaa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c81ef3f0 con 0x7f24c806f7a0 2023-10-30T13:56:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24cdfaa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c80ada50 con 0x7f24c80c6bd0 2023-10-30T13:56:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24cdfaa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c80adcf0 con 0x7f24c8071110 2023-10-30T13:56:12.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c806f7a0 0x7f24c80c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c806f7a0 0x7f24c80c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60540/0 (socket says 172.21.15.46:60540) 2023-10-30T13:56:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24c77fe640 1 -- 172.21.15.46:0/3463167920 learned_addr learned my addr 172.21.15.46:0/3463167920 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24c6ffd640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24c8071110 0x7f24c81f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24c7fff640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24c80c6bd0 0x7f24c80c1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:12.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24c77fe640 1 -- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24c8071110 msgr2=0x7f24c81f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:12.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24c77fe640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24c8071110 0x7f24c81f2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24c77fe640 1 -- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24c80c6bd0 msgr2=0x7f24c80c1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:12.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24c77fe640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24c80c6bd0 0x7f24c80c1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:12.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24c77fe640 1 -- 172.21.15.46:0/3463167920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c0009b30 con 0x7f24c806f7a0 2023-10-30T13:56:12.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24c7fff640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24c80c6bd0 0x7f24c80c1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:12.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24c77fe640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c806f7a0 0x7f24c80c33e0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f24b800e870 tx=0x7f24b800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:12.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24affff640 1 -- 172.21.15.46:0/3463167920 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b800ee60 con 0x7f24c806f7a0 2023-10-30T13:56:12.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24affff640 1 -- 172.21.15.46:0/3463167920 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b800c940 con 0x7f24c806f7a0 2023-10-30T13:56:12.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24affff640 1 -- 172.21.15.46:0/3463167920 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b800ccf0 con 0x7f24c806f7a0 2023-10-30T13:56:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24cdfaa640 1 -- 172.21.15.46:0/3463167920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c80adfd0 con 0x7f24c806f7a0 2023-10-30T13:56:12.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24cdfaa640 1 -- 172.21.15.46:0/3463167920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c81ef080 con 0x7f24c806f7a0 2023-10-30T13:56:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.869+0000 7f24cdfaa640 1 -- 172.21.15.46:0/3463167920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c804e170 con 0x7f24c806f7a0 2023-10-30T13:56:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.873+0000 7f24affff640 1 -- 172.21.15.46:0/3463167920 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f24b8013470 con 0x7f24c806f7a0 2023-10-30T13:56:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.873+0000 7f24affff640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24b40664d0 0x7f24b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.873+0000 7f24affff640 1 -- 172.21.15.46:0/3463167920 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f24b8084540 con 0x7f24c806f7a0 2023-10-30T13:56:12.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.873+0000 7f24c6ffd640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24b40664d0 0x7f24b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:12.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.873+0000 7f24c6ffd640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24b40664d0 0x7f24b4068990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f24c80706b0 tx=0x7f24bc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:12.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:12.877+0000 7f24affff640 1 -- 172.21.15.46:0/3463167920 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f24b8051390 con 0x7f24c806f7a0 2023-10-30T13:56:13.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.065+0000 7f24cdfaa640 1 -- 172.21.15.46:0/3463167920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24c80c7d90 con 0x7f24c806f7a0 2023-10-30T13:56:13.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.065+0000 7f24affff640 1 -- 172.21.15.46:0/3463167920 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f24b80549b0 con 0x7f24c806f7a0 2023-10-30T13:56:13.071 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:56:13.071 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":149,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:11.645197+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:13.071 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 149 2023-10-30T13:56:13.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.069+0000 7f24cdfaa640 1 -- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24b40664d0 msgr2=0x7f24b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:13.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.069+0000 7f24cdfaa640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24b40664d0 0x7f24b4068990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f24c80706b0 tx=0x7f24bc009190 comp rx=0 tx=0).stop 2023-10-30T13:56:13.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.069+0000 7f24cdfaa640 1 -- 172.21.15.46:0/3463167920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c806f7a0 msgr2=0x7f24c80c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:13.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.069+0000 7f24cdfaa640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c806f7a0 0x7f24c80c33e0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f24b800e870 tx=0x7f24b800ed30 comp rx=0 tx=0).stop 2023-10-30T13:56:13.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.069+0000 7f24cdfaa640 1 -- 172.21.15.46:0/3463167920 shutdown_connections 2023-10-30T13:56:13.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.069+0000 7f24cdfaa640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24b40664d0 0x7f24b4068990 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:13.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.069+0000 7f24cdfaa640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24c80c6bd0 0x7f24c80c1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:13.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.069+0000 7f24cdfaa640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24c8071110 0x7f24c81f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:13.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.069+0000 7f24cdfaa640 1 --2- 172.21.15.46:0/3463167920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c806f7a0 0x7f24c80c33e0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:13.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.069+0000 7f24cdfaa640 1 -- 172.21.15.46:0/3463167920 >> 172.21.15.46:0/3463167920 conn(0x7f24c80c7910 msgr2=0x7f24c80c4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:13.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.069+0000 7f24cdfaa640 1 -- 172.21.15.46:0/3463167920 shutdown_connections 2023-10-30T13:56:13.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:13.069+0000 7f24cdfaa640 1 -- 172.21.15.46:0/3463167920 wait complete. 2023-10-30T13:56:13.092 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:15.093 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1ae994640 1 Processor -- start 2023-10-30T13:56:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1ae994640 1 -- start start 2023-10-30T13:56:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1ae994640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd1a81538a0 0x7fd1a815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1ae994640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd1a80cf160 0x7fd1a80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1ae994640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a80cb800 con 0x7fd1a80ca0e0 2023-10-30T13:56:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1ae994640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a815dc30 con 0x7fd1a80cf160 2023-10-30T13:56:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1ae994640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a80cb330 con 0x7fd1a81538a0 2023-10-30T13:56:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1acf31640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd1a80cf160 0x7fd1a80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1acf31640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd1a80cf160 0x7fd1a80cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60550/0 (socket says 172.21.15.46:60550) 2023-10-30T13:56:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1acf31640 1 -- 172.21.15.46:0/1782567830 learned_addr learned my addr 172.21.15.46:0/1782567830 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1a7fff640 1 --2- 172.21.15.46:0/1782567830 >> v2:172.21.15.154:3300/0 conn(0x7fd1a81538a0 0x7fd1a815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:15.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1acf31640 1 -- 172.21.15.46:0/1782567830 >> v2:172.21.15.154:3300/0 conn(0x7fd1a81538a0 msgr2=0x7fd1a815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:15.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1acf31640 1 --2- 172.21.15.46:0/1782567830 >> v2:172.21.15.154:3300/0 conn(0x7fd1a81538a0 0x7fd1a815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1acf31640 1 -- 172.21.15.46:0/1782567830 >> v1:172.21.15.154:6789/0 conn(0x7fd1a80ca0e0 legacy=0x7fd1a815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:15.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1acf31640 1 -- 172.21.15.46:0/1782567830 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a80cad70 con 0x7fd1a80cf160 2023-10-30T13:56:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.245+0000 7fd1a7fff640 1 --2- 172.21.15.46:0/1782567830 >> v2:172.21.15.154:3300/0 conn(0x7fd1a81538a0 0x7fd1a815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.249+0000 7fd1acf31640 1 --2- 172.21.15.46:0/1782567830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a80cf160 0x7fd1a80cf590 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fd1a80cfd10 tx=0x7fd1a001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383d202f09241792 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.249+0000 7fd1a6ffd640 1 -- 172.21.15.46:0/1782567830 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a00af400 con 0x7fd1a80cf160 2023-10-30T13:56:15.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.249+0000 7fd1a6ffd640 1 -- 172.21.15.46:0/1782567830 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a00ab050 con 0x7fd1a80cf160 2023-10-30T13:56:15.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.249+0000 7fd1a6ffd640 1 -- 172.21.15.46:0/1782567830 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a00af780 con 0x7fd1a80cf160 2023-10-30T13:56:15.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.249+0000 7fd1ae994640 1 -- 172.21.15.46:0/1782567830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a80cf160 msgr2=0x7fd1a80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:15.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.249+0000 7fd1ae994640 1 --2- 172.21.15.46:0/1782567830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a80cf160 0x7fd1a80cf590 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fd1a80cfd10 tx=0x7fd1a001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.249+0000 7fd1ae994640 1 -- 172.21.15.46:0/1782567830 shutdown_connections 2023-10-30T13:56:15.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.249+0000 7fd1ae994640 1 --2- 172.21.15.46:0/1782567830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a80cf160 0x7fd1a80cf590 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.249+0000 7fd1ae994640 1 --2- 172.21.15.46:0/1782567830 >> v2:172.21.15.154:3300/0 conn(0x7fd1a81538a0 0x7fd1a815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.249+0000 7fd1ae994640 1 -- 172.21.15.46:0/1782567830 >> 172.21.15.46:0/1782567830 conn(0x7fd1a806bfe0 msgr2=0x7fd1a806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:15.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.249+0000 7fd1ae994640 1 -- 172.21.15.46:0/1782567830 shutdown_connections 2023-10-30T13:56:15.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.249+0000 7fd1ae994640 1 -- 172.21.15.46:0/1782567830 wait complete. 2023-10-30T13:56:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1ae994640 1 Processor -- start 2023-10-30T13:56:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1ae994640 1 -- start start 2023-10-30T13:56:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1ae994640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1a80ca0e0 0x7fd1a8143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1ae994640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1a80cf160 0x7fd1a81f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1ae994640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a81538a0 0x7fd1a80c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1ae994640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a8144510 con 0x7fd1a81538a0 2023-10-30T13:56:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1ae994640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a81f3800 con 0x7fd1a80ca0e0 2023-10-30T13:56:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1ae994640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a80b34d0 con 0x7fd1a80cf160 2023-10-30T13:56:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1acf31640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a81538a0 0x7fd1a80c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1acf31640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a81538a0 0x7fd1a80c4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60566/0 (socket says 172.21.15.46:60566) 2023-10-30T13:56:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1acf31640 1 -- 172.21.15.46:0/2598439484 learned_addr learned my addr 172.21.15.46:0/2598439484 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1a77fe640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1a80cf160 0x7fd1a81f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1a7fff640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1a80ca0e0 0x7fd1a8143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1acf31640 1 -- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1a80cf160 msgr2=0x7fd1a81f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1acf31640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1a80cf160 0x7fd1a81f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1acf31640 1 -- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1a80ca0e0 msgr2=0x7fd1a8143fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1acf31640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1a80ca0e0 0x7fd1a8143fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1acf31640 1 -- 172.21.15.46:0/2598439484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a0009b30 con 0x7fd1a81538a0 2023-10-30T13:56:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1acf31640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a81538a0 0x7fd1a80c4c80 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fd1a00b2040 tx=0x7fd1a001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1a4ff9640 1 -- 172.21.15.46:0/2598439484 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a00afd30 con 0x7fd1a81538a0 2023-10-30T13:56:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1a4ff9640 1 -- 172.21.15.46:0/2598439484 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a0009510 con 0x7fd1a81538a0 2023-10-30T13:56:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1ae994640 1 -- 172.21.15.46:0/2598439484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a81ef340 con 0x7fd1a81538a0 2023-10-30T13:56:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1a4ff9640 1 -- 172.21.15.46:0/2598439484 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a00ba660 con 0x7fd1a81538a0 2023-10-30T13:56:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.253+0000 7fd1ae994640 1 -- 172.21.15.46:0/2598439484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a81ef8c0 con 0x7fd1a81538a0 2023-10-30T13:56:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.257+0000 7fd1ae994640 1 -- 172.21.15.46:0/2598439484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a80b2a50 con 0x7fd1a81538a0 2023-10-30T13:56:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.257+0000 7fd1a4ff9640 1 -- 172.21.15.46:0/2598439484 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd1a00b6040 con 0x7fd1a81538a0 2023-10-30T13:56:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.257+0000 7fd1a4ff9640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1940664a0 0x7fd194068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.257+0000 7fd1a4ff9640 1 -- 172.21.15.46:0/2598439484 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fd1a01227d0 con 0x7fd1a81538a0 2023-10-30T13:56:15.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.257+0000 7fd1a7fff640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1940664a0 0x7fd194068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:15.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.261+0000 7fd1a7fff640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1940664a0 0x7fd194068960 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd19800c900 tx=0x7fd198008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:15.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.261+0000 7fd1a4ff9640 1 -- 172.21.15.46:0/2598439484 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd1a00ef590 con 0x7fd1a81538a0 2023-10-30T13:56:15.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 -- 172.21.15.46:0/2598439484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1a80b3680 con 0x7fd1a81538a0 2023-10-30T13:56:15.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1a4ff9640 1 -- 172.21.15.46:0/2598439484 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+5736 (secure 0 0 0) 0x7fd1a80b3680 con 0x7fd1a81538a0 2023-10-30T13:56:15.460 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:56:15.460 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":149,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:11.645197+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:15.460 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 149 2023-10-30T13:56:15.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 -- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1940664a0 msgr2=0x7fd194068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:15.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1940664a0 0x7fd194068960 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd19800c900 tx=0x7fd198008040 comp rx=0 tx=0).stop 2023-10-30T13:56:15.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 -- 172.21.15.46:0/2598439484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a81538a0 msgr2=0x7fd1a80c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:15.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a81538a0 0x7fd1a80c4c80 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fd1a00b2040 tx=0x7fd1a001bf60 comp rx=0 tx=0).stop 2023-10-30T13:56:15.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 -- 172.21.15.46:0/2598439484 shutdown_connections 2023-10-30T13:56:15.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1940664a0 0x7fd194068960 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a81538a0 0x7fd1a80c4c80 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1a80cf160 0x7fd1a81f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 --2- 172.21.15.46:0/2598439484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1a80ca0e0 0x7fd1a8143fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:15.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 -- 172.21.15.46:0/2598439484 >> 172.21.15.46:0/2598439484 conn(0x7fd1a806bfe0 msgr2=0x7fd1a81513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:15.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 -- 172.21.15.46:0/2598439484 shutdown_connections 2023-10-30T13:56:15.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:15.457+0000 7fd1ae994640 1 -- 172.21.15.46:0/2598439484 wait complete. 2023-10-30T13:56:15.481 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:17.482 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:17.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99b49c4640 1 Processor -- start 2023-10-30T13:56:17.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99b49c4640 1 -- start start 2023-10-30T13:56:17.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99b49c4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f99b015d6f0 0x7f99b015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:17.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99b49c4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f99b00cf160 0x7f99b00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:17.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99b49c4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b00cb800 con 0x7f99b01538a0 2023-10-30T13:56:17.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99b49c4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b00cb330 con 0x7f99b015d6f0 2023-10-30T13:56:17.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99b49c4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b00cad70 con 0x7f99b00cf160 2023-10-30T13:56:17.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99add9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f99b015d6f0 0x7f99b015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:17.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99add9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f99b015d6f0 0x7f99b015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55568/0 (socket says 172.21.15.46:55568) 2023-10-30T13:56:17.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99add9b640 1 -- 172.21.15.46:0/862814044 learned_addr learned my addr 172.21.15.46:0/862814044 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:17.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99aed9d640 1 --2- 172.21.15.46:0/862814044 >> v2:172.21.15.154:3300/0 conn(0x7f99b00cf160 0x7f99b00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:17.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99add9b640 1 -- 172.21.15.46:0/862814044 >> v2:172.21.15.154:3300/0 conn(0x7f99b00cf160 msgr2=0x7f99b00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:17.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99add9b640 1 --2- 172.21.15.46:0/862814044 >> v2:172.21.15.154:3300/0 conn(0x7f99b00cf160 0x7f99b00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99add9b640 1 -- 172.21.15.46:0/862814044 >> v1:172.21.15.46:6789/0 conn(0x7f99b01538a0 legacy=0x7f99b00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99add9b640 1 -- 172.21.15.46:0/862814044 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b00cb050 con 0x7f99b015d6f0 2023-10-30T13:56:17.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99ad59a640 1 -- 172.21.15.46:0/862814044 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930577475 0 0) 0x7f99b00cb800 con 0x7f99b01538a0 2023-10-30T13:56:17.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.645+0000 7f99aed9d640 1 --2- 172.21.15.46:0/862814044 >> v2:172.21.15.154:3300/0 conn(0x7f99b00cf160 0x7f99b00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:56:17.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.649+0000 7f99add9b640 1 --2- 172.21.15.46:0/862814044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b015d6f0 0x7f99b015a090 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f99b00b5330 tx=0x7f99a401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c6e17044db2d11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:17.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.649+0000 7f99ad59a640 1 -- 172.21.15.46:0/862814044 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a40a89f0 con 0x7f99b015d6f0 2023-10-30T13:56:17.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.649+0000 7f99ad59a640 1 -- 172.21.15.46:0/862814044 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a40a8b50 con 0x7f99b015d6f0 2023-10-30T13:56:17.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.649+0000 7f99ad59a640 1 -- 172.21.15.46:0/862814044 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a40b0440 con 0x7f99b015d6f0 2023-10-30T13:56:17.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.649+0000 7f99b49c4640 1 -- 172.21.15.46:0/862814044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b015d6f0 msgr2=0x7f99b015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.649+0000 7f99b49c4640 1 --2- 172.21.15.46:0/862814044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b015d6f0 0x7f99b015a090 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f99b00b5330 tx=0x7f99a401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.649+0000 7f99b49c4640 1 -- 172.21.15.46:0/862814044 shutdown_connections 2023-10-30T13:56:17.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.649+0000 7f99b49c4640 1 --2- 172.21.15.46:0/862814044 >> v2:172.21.15.154:3300/0 conn(0x7f99b00cf160 0x7f99b00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.649+0000 7f99b49c4640 1 --2- 172.21.15.46:0/862814044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b015d6f0 0x7f99b015a090 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.649+0000 7f99b49c4640 1 -- 172.21.15.46:0/862814044 >> 172.21.15.46:0/862814044 conn(0x7f99b006bfe0 msgr2=0x7f99b006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:17.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.649+0000 7f99b49c4640 1 -- 172.21.15.46:0/862814044 shutdown_connections 2023-10-30T13:56:17.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.649+0000 7f99b49c4640 1 -- 172.21.15.46:0/862814044 wait complete. 2023-10-30T13:56:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99b49c4640 1 Processor -- start 2023-10-30T13:56:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99b49c4640 1 -- start start 2023-10-30T13:56:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99b49c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b00cf160 0x7f99b00cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:17.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99b49c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b01538a0 0x7f99b00b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:17.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99b49c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99b015d6f0 0x7f99b00b3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:17.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99b49c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b0158a00 con 0x7f99b01538a0 2023-10-30T13:56:17.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99b49c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b0157690 con 0x7f99b015d6f0 2023-10-30T13:56:17.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99b49c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b0158c70 con 0x7f99b00cf160 2023-10-30T13:56:17.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99add9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b01538a0 0x7f99b00b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:17.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99add9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b01538a0 0x7f99b00b2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55580/0 (socket says 172.21.15.46:55580) 2023-10-30T13:56:17.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99add9b640 1 -- 172.21.15.46:0/332787720 learned_addr learned my addr 172.21.15.46:0/332787720 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:17.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99aed9d640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99b015d6f0 0x7f99b00b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:17.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99ae59c640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b00cf160 0x7f99b00cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:17.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99add9b640 1 -- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b00cf160 msgr2=0x7f99b00cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99add9b640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b00cf160 0x7f99b00cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99add9b640 1 -- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99b015d6f0 msgr2=0x7f99b00b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99add9b640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99b015d6f0 0x7f99b00b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99add9b640 1 -- 172.21.15.46:0/332787720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a40094d0 con 0x7f99b01538a0 2023-10-30T13:56:17.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99add9b640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b01538a0 0x7f99b00b2ff0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f99a401b900 tx=0x7f99a401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:17.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f999effd640 1 -- 172.21.15.46:0/332787720 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a40ba600 con 0x7f99b01538a0 2023-10-30T13:56:17.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f999effd640 1 -- 172.21.15.46:0/332787720 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a40ba780 con 0x7f99b01538a0 2023-10-30T13:56:17.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f999effd640 1 -- 172.21.15.46:0/332787720 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a40bab30 con 0x7f99b01538a0 2023-10-30T13:56:17.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99b49c4640 1 -- 172.21.15.46:0/332787720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b0157930 con 0x7f99b01538a0 2023-10-30T13:56:17.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.653+0000 7f99b49c4640 1 -- 172.21.15.46:0/332787720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b00ce230 con 0x7f99b01538a0 2023-10-30T13:56:17.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.657+0000 7f99b49c4640 1 -- 172.21.15.46:0/332787720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b00b2a50 con 0x7f99b01538a0 2023-10-30T13:56:17.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.657+0000 7f999effd640 1 -- 172.21.15.46:0/332787720 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f99a40bac90 con 0x7f99b01538a0 2023-10-30T13:56:17.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.657+0000 7f999effd640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99900664a0 0x7f9990068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:17.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.657+0000 7f999effd640 1 -- 172.21.15.46:0/332787720 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f99a4122c00 con 0x7f99b01538a0 2023-10-30T13:56:17.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.657+0000 7f99ae59c640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99900664a0 0x7f9990068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:17.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.661+0000 7f99ae59c640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99900664a0 0x7f9990068960 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f99a0000f70 tx=0x7f99a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.661+0000 7f999effd640 1 -- 172.21.15.46:0/332787720 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f99a40ef990 con 0x7f99b01538a0 2023-10-30T13:56:17.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.917+0000 7f99b49c4640 1 -- 172.21.15.46:0/332787720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99b004e170 con 0x7f99b01538a0 2023-10-30T13:56:17.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f999effd640 1 -- 172.21.15.46:0/332787720 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f99a4120e60 con 0x7f99b01538a0 2023-10-30T13:56:17.923 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:56:17.924 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":149,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:11.645197+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:17.924 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 149 2023-10-30T13:56:17.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f99b49c4640 1 -- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99900664a0 msgr2=0x7f9990068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f99b49c4640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99900664a0 0x7f9990068960 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f99a0000f70 tx=0x7f99a0009040 comp rx=0 tx=0).stop 2023-10-30T13:56:17.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f99b49c4640 1 -- 172.21.15.46:0/332787720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b01538a0 msgr2=0x7f99b00b2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:17.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f99b49c4640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b01538a0 0x7f99b00b2ff0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f99a401b900 tx=0x7f99a401ba90 comp rx=0 tx=0).stop 2023-10-30T13:56:17.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f99b49c4640 1 -- 172.21.15.46:0/332787720 shutdown_connections 2023-10-30T13:56:17.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f99b49c4640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99900664a0 0x7f9990068960 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f99b49c4640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99b015d6f0 0x7f99b00b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f99b49c4640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99b01538a0 0x7f99b00b2ff0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f99b49c4640 1 --2- 172.21.15.46:0/332787720 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b00cf160 0x7f99b00cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:17.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f99b49c4640 1 -- 172.21.15.46:0/332787720 >> 172.21.15.46:0/332787720 conn(0x7f99b006bfe0 msgr2=0x7f99b00c49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:17.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f99b49c4640 1 -- 172.21.15.46:0/332787720 shutdown_connections 2023-10-30T13:56:17.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:17.921+0000 7f99b49c4640 1 -- 172.21.15.46:0/332787720 wait complete. 2023-10-30T13:56:17.944 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:56:19.945 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:20.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.093+0000 7f69389c3640 1 Processor -- start 2023-10-30T13:56:20.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f69389c3640 1 -- start start 2023-10-30T13:56:20.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f69389c3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f693415cd50 0x7f69340c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:20.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f69389c3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693415a6c0 con 0x7f69340cefa0 2023-10-30T13:56:20.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f69389c3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693415d300 con 0x7f69341538f0 2023-10-30T13:56:20.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f69389c3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693415d5e0 con 0x7f693415cd50 2023-10-30T13:56:20.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f693359e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f69340cefa0 0x7f69341640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57140/0 (socket says 172.21.15.46:57140) 2023-10-30T13:56:20.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f693359e640 1 -- 172.21.15.46:0/970011287 learned_addr learned my addr 172.21.15.46:0/970011287 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:56:20.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f693259c640 1 --2- 172.21.15.46:0/970011287 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f693415cd50 0x7f69340c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:20.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f6931d9b640 1 -- 172.21.15.46:0/970011287 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971659969 0 0) 0x7f693415a6c0 con 0x7f69340cefa0 2023-10-30T13:56:20.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f6931d9b640 1 -- 172.21.15.46:0/970011287 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c003530 con 0x7f69340cefa0 2023-10-30T13:56:20.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f6931d9b640 1 -- 172.21.15.46:0/970011287 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803159988 0 0) 0x7f693415d300 con 0x7f69341538f0 2023-10-30T13:56:20.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f6931d9b640 1 -- 172.21.15.46:0/970011287 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f693415a6c0 con 0x7f69341538f0 2023-10-30T13:56:20.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f693259c640 1 -- 172.21.15.46:0/970011287 >> v1:172.21.15.154:6789/0 conn(0x7f69341538f0 legacy=0x7f6934159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:20.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f6931d9b640 1 -- 172.21.15.46:0/970011287 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2221071617 0 0) 0x7f691c003530 con 0x7f69340cefa0 2023-10-30T13:56:20.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f693259c640 1 -- 172.21.15.46:0/970011287 >> v1:172.21.15.46:6789/0 conn(0x7f69340cefa0 legacy=0x7f69341640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:20.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.097+0000 7f693259c640 1 -- 172.21.15.46:0/970011287 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693415d8c0 con 0x7f693415cd50 2023-10-30T13:56:20.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.101+0000 7f693259c640 1 --2- 172.21.15.46:0/970011287 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f693415cd50 0x7f69340c9f90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f69280097f0 tx=0x7f692801ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b3358f7235ebb99 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:20.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.101+0000 7f6931d9b640 1 -- 172.21.15.46:0/970011287 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c001a00 con 0x7f693415cd50 2023-10-30T13:56:20.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.101+0000 7f6931d9b640 1 -- 172.21.15.46:0/970011287 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69280a8650 con 0x7f693415cd50 2023-10-30T13:56:20.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.101+0000 7f6931d9b640 1 -- 172.21.15.46:0/970011287 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69280a89e0 con 0x7f693415cd50 2023-10-30T13:56:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.101+0000 7f69389c3640 1 -- 172.21.15.46:0/970011287 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f693415cd50 msgr2=0x7f69340c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.101+0000 7f69389c3640 1 --2- 172.21.15.46:0/970011287 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f693415cd50 0x7f69340c9f90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f69280097f0 tx=0x7f692801ebd0 comp rx=0 tx=0).stop 2023-10-30T13:56:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.101+0000 7f69389c3640 1 -- 172.21.15.46:0/970011287 shutdown_connections 2023-10-30T13:56:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.101+0000 7f69389c3640 1 --2- 172.21.15.46:0/970011287 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f693415cd50 0x7f69340c9f90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.101+0000 7f69389c3640 1 -- 172.21.15.46:0/970011287 >> 172.21.15.46:0/970011287 conn(0x7f693406a640 msgr2=0x7f6934065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.101+0000 7f69389c3640 1 -- 172.21.15.46:0/970011287 shutdown_connections 2023-10-30T13:56:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.101+0000 7f69389c3640 1 -- 172.21.15.46:0/970011287 wait complete. 2023-10-30T13:56:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f69389c3640 1 Processor -- start 2023-10-30T13:56:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f69389c3640 1 -- start start 2023-10-30T13:56:20.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f69389c3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69340cefa0 0x7f6934152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:20.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f69389c3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69341538f0 0x7f69340b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f69389c3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693415cd50 0x7f693415b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f69389c3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6934158a40 con 0x7f693415cd50 2023-10-30T13:56:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f69389c3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6934158ed0 con 0x7f69340cefa0 2023-10-30T13:56:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f69389c3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6934154880 con 0x7f69341538f0 2023-10-30T13:56:20.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f693359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693415cd50 0x7f693415b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:20.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f693359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693415cd50 0x7f693415b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55594/0 (socket says 172.21.15.46:55594) 2023-10-30T13:56:20.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f693359e640 1 -- 172.21.15.46:0/2280649390 learned_addr learned my addr 172.21.15.46:0/2280649390 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:20.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f6932d9d640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69340cefa0 0x7f6934152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:20.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f693259c640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69341538f0 0x7f69340b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:20.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f693359e640 1 -- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69341538f0 msgr2=0x7f69340b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:20.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f693359e640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69341538f0 0x7f69340b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f693359e640 1 -- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69340cefa0 msgr2=0x7f6934152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f693359e640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69340cefa0 0x7f6934152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f693359e640 1 -- 172.21.15.46:0/2280649390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6928009420 con 0x7f693415cd50 2023-10-30T13:56:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f6932d9d640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69340cefa0 0x7f6934152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f693259c640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69341538f0 0x7f69340b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:20.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f693359e640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693415cd50 0x7f693415b970 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f692c002030 tx=0x7f692c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f691b7fe640 1 -- 172.21.15.46:0/2280649390 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c012400 con 0x7f693415cd50 2023-10-30T13:56:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f691b7fe640 1 -- 172.21.15.46:0/2280649390 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692c0125a0 con 0x7f693415cd50 2023-10-30T13:56:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f691b7fe640 1 -- 172.21.15.46:0/2280649390 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c012950 con 0x7f693415cd50 2023-10-30T13:56:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f69389c3640 1 -- 172.21.15.46:0/2280649390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6934157ed0 con 0x7f693415cd50 2023-10-30T13:56:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.105+0000 7f69389c3640 1 -- 172.21.15.46:0/2280649390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69341f2670 con 0x7f693415cd50 2023-10-30T13:56:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.109+0000 7f69389c3640 1 -- 172.21.15.46:0/2280649390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6934158bf0 con 0x7f693415cd50 2023-10-30T13:56:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.109+0000 7f691b7fe640 1 -- 172.21.15.46:0/2280649390 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f692c012ab0 con 0x7f693415cd50 2023-10-30T13:56:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.109+0000 7f691b7fe640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69200664d0 0x7f6920068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.109+0000 7f691b7fe640 1 -- 172.21.15.46:0/2280649390 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f692c084af0 con 0x7f693415cd50 2023-10-30T13:56:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.109+0000 7f6932d9d640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69200664d0 0x7f6920068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.109+0000 7f6932d9d640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69200664d0 0x7f6920068990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f69240011f0 tx=0x7f6924009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:20.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.113+0000 7f691b7fe640 1 -- 172.21.15.46:0/2280649390 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6934158bf0 con 0x7f693415cd50 2023-10-30T13:56:20.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 -- 172.21.15.46:0/2280649390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6934157ed0 con 0x7f693415cd50 2023-10-30T13:56:20.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f691b7fe640 1 -- 172.21.15.46:0/2280649390 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f6934157ed0 con 0x7f693415cd50 2023-10-30T13:56:20.308 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:56:20.308 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:17.573566+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:20.309 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 150 2023-10-30T13:56:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 -- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69200664d0 msgr2=0x7f6920068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69200664d0 0x7f6920068990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f69240011f0 tx=0x7f6924009040 comp rx=0 tx=0).stop 2023-10-30T13:56:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 -- 172.21.15.46:0/2280649390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693415cd50 msgr2=0x7f693415b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693415cd50 0x7f693415b970 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f692c002030 tx=0x7f692c00a430 comp rx=0 tx=0).stop 2023-10-30T13:56:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 -- 172.21.15.46:0/2280649390 shutdown_connections 2023-10-30T13:56:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69200664d0 0x7f6920068990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693415cd50 0x7f693415b970 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69341538f0 0x7f69340b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 --2- 172.21.15.46:0/2280649390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69340cefa0 0x7f6934152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 -- 172.21.15.46:0/2280649390 >> 172.21.15.46:0/2280649390 conn(0x7f693406a640 msgr2=0x7f69340cd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 -- 172.21.15.46:0/2280649390 shutdown_connections 2023-10-30T13:56:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:20.305+0000 7f69389c3640 1 -- 172.21.15.46:0/2280649390 wait complete. 2023-10-30T13:56:20.330 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=7188 2023-10-30T13:56:20.330 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.c 2023-10-30T13:56:21.338 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T13:56:21.333+0000 7f1fec58c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:56:22.905 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:56:22.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:56:27.331 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:56:27.331 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:56:27.332 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:56:27.336 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:56:27.336 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:27.428 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-10-30T13:56:27.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.481+0000 7f5e41c9a640 1 Processor -- start 2023-10-30T13:56:27.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.481+0000 7f5e41c9a640 1 -- start start 2023-10-30T13:56:27.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.481+0000 7f5e41c9a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c1539a0 0x7f5e3c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:27.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.485+0000 7f5e41c9a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c15a8c0 con 0x7f5e3c069e90 2023-10-30T13:56:27.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.485+0000 7f5e41c9a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c153d80 con 0x7f5e3c070a00 2023-10-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.485+0000 7f5e41c9a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c06fbb0 con 0x7f5e3c1539a0 2023-10-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.485+0000 7f5e3bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5e3c069e90 0x7f5e3c06a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47008/0 (socket says 172.21.15.46:47008) 2023-10-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.485+0000 7f5e3bfff640 1 -- 172.21.15.46:0/3180760048 learned_addr learned my addr 172.21.15.46:0/3180760048 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.485+0000 7f5e3b7fe640 1 --2- 172.21.15.46:0/3180760048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c1539a0 0x7f5e3c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.485+0000 7f5e3a7fc640 1 -- 172.21.15.46:0/3180760048 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668633786 0 0) 0x7f5e3c15a8c0 con 0x7f5e3c069e90 2023-10-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.485+0000 7f5e3a7fc640 1 -- 172.21.15.46:0/3180760048 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e24003530 con 0x7f5e3c069e90 2023-10-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.485+0000 7f5e3a7fc640 1 -- 172.21.15.46:0/3180760048 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1517123704 0 0) 0x7f5e24003530 con 0x7f5e3c069e90 2023-10-30T13:56:27.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.485+0000 7f5e3b7fe640 1 -- 172.21.15.46:0/3180760048 >> v1:172.21.15.154:6789/0 conn(0x7f5e3c070a00 legacy=0x7f5e3c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:27.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.485+0000 7f5e3b7fe640 1 -- 172.21.15.46:0/3180760048 >> v1:172.21.15.46:6789/0 conn(0x7f5e3c069e90 legacy=0x7f5e3c06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:27.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.485+0000 7f5e3b7fe640 1 -- 172.21.15.46:0/3180760048 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c0c5c50 con 0x7f5e3c1539a0 2023-10-30T13:56:27.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e3b7fe640 1 --2- 172.21.15.46:0/3180760048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c1539a0 0x7f5e3c15a380 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0097f0 tx=0x7f5e2c01c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d115a7c78520fd20 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:27.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e3a7fc640 1 -- 172.21.15.46:0/3180760048 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34001a00 con 0x7f5e3c1539a0 2023-10-30T13:56:27.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e3a7fc640 1 -- 172.21.15.46:0/3180760048 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2c0a7050 con 0x7f5e3c1539a0 2023-10-30T13:56:27.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e3a7fc640 1 -- 172.21.15.46:0/3180760048 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c0ab600 con 0x7f5e3c1539a0 2023-10-30T13:56:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e41c9a640 1 -- 172.21.15.46:0/3180760048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c1539a0 msgr2=0x7f5e3c15a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e41c9a640 1 --2- 172.21.15.46:0/3180760048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c1539a0 0x7f5e3c15a380 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0097f0 tx=0x7f5e2c01c320 comp rx=0 tx=0).stop 2023-10-30T13:56:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e41c9a640 1 -- 172.21.15.46:0/3180760048 shutdown_connections 2023-10-30T13:56:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e41c9a640 1 --2- 172.21.15.46:0/3180760048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c1539a0 0x7f5e3c15a380 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e41c9a640 1 -- 172.21.15.46:0/3180760048 >> 172.21.15.46:0/3180760048 conn(0x7f5e3c0c7910 msgr2=0x7f5e3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:27.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e41c9a640 1 -- 172.21.15.46:0/3180760048 shutdown_connections 2023-10-30T13:56:27.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e41c9a640 1 -- 172.21.15.46:0/3180760048 wait complete. 2023-10-30T13:56:27.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e41c9a640 1 Processor -- start 2023-10-30T13:56:27.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.489+0000 7f5e41c9a640 1 -- start start 2023-10-30T13:56:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e41c9a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c069e90 0x7f5e3c1f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e41c9a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c070a00 0x7f5e3c1f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e41c9a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e3c1539a0 0x7f5e3c0ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e41c9a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0c4ab0 con 0x7f5e3c1539a0 2023-10-30T13:56:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e41c9a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c1f1c00 con 0x7f5e3c070a00 2023-10-30T13:56:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e41c9a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c14e500 con 0x7f5e3c069e90 2023-10-30T13:56:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e3bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e3c1539a0 0x7f5e3c0ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e3b7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c069e90 0x7f5e3c1f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e3bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e3c1539a0 0x7f5e3c0ca210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48826/0 (socket says 172.21.15.46:48826) 2023-10-30T13:56:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e3bfff640 1 -- 172.21.15.46:0/2873496622 learned_addr learned my addr 172.21.15.46:0/2873496622 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e3affd640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c070a00 0x7f5e3c1f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e3bfff640 1 -- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c069e90 msgr2=0x7f5e3c1f20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e3bfff640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c069e90 0x7f5e3c1f20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e3bfff640 1 -- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c070a00 msgr2=0x7f5e3c1f2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e3bfff640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c070a00 0x7f5e3c1f2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e3bfff640 1 -- 172.21.15.46:0/2873496622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c009420 con 0x7f5e3c1539a0 2023-10-30T13:56:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e3bfff640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e3c1539a0 0x7f5e3c0ca210 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f5e34001870 tx=0x7f5e3400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e23fff640 1 -- 172.21.15.46:0/2873496622 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34012400 con 0x7f5e3c1539a0 2023-10-30T13:56:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e23fff640 1 -- 172.21.15.46:0/2873496622 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e34012560 con 0x7f5e3c1539a0 2023-10-30T13:56:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e23fff640 1 -- 172.21.15.46:0/2873496622 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34012910 con 0x7f5e3c1539a0 2023-10-30T13:56:27.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e41c9a640 1 -- 172.21.15.46:0/2873496622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c0c4d00 con 0x7f5e3c1539a0 2023-10-30T13:56:27.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e41c9a640 1 -- 172.21.15.46:0/2873496622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c142800 con 0x7f5e3c1539a0 2023-10-30T13:56:27.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e41c9a640 1 -- 172.21.15.46:0/2873496622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c04e170 con 0x7f5e3c1539a0 2023-10-30T13:56:27.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.493+0000 7f5e23fff640 1 -- 172.21.15.46:0/2873496622 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5e34012a70 con 0x7f5e3c1539a0 2023-10-30T13:56:27.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.497+0000 7f5e23fff640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e280664d0 0x7f5e28068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:27.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.497+0000 7f5e23fff640 1 -- 172.21.15.46:0/2873496622 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f5e34084ca0 con 0x7f5e3c1539a0 2023-10-30T13:56:27.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.497+0000 7f5e3b7fe640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e280664d0 0x7f5e28068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:27.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.497+0000 7f5e3b7fe640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e280664d0 0x7f5e28068990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0ad000 tx=0x7f5e2c0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:27.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.501+0000 7f5e23fff640 1 -- 172.21.15.46:0/2873496622 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5e34051a50 con 0x7f5e3c1539a0 2023-10-30T13:56:27.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.689+0000 7f5e41c9a640 1 -- 172.21.15.46:0/2873496622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e3c14da30 con 0x7f5e3c1539a0 2023-10-30T13:56:27.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.689+0000 7f5e23fff640 1 -- 172.21.15.46:0/2873496622 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f5e34055070 con 0x7f5e3c1539a0 2023-10-30T13:56:27.695 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:56:27.695 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:17.573566+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:27.696 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 150 2023-10-30T13:56:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.693+0000 7f5e41c9a640 1 -- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e280664d0 msgr2=0x7f5e28068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.693+0000 7f5e41c9a640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e280664d0 0x7f5e28068990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0ad000 tx=0x7f5e2c0a6000 comp rx=0 tx=0).stop 2023-10-30T13:56:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.693+0000 7f5e41c9a640 1 -- 172.21.15.46:0/2873496622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e3c1539a0 msgr2=0x7f5e3c0ca210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.693+0000 7f5e41c9a640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e3c1539a0 0x7f5e3c0ca210 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f5e34001870 tx=0x7f5e3400a430 comp rx=0 tx=0).stop 2023-10-30T13:56:27.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.693+0000 7f5e41c9a640 1 -- 172.21.15.46:0/2873496622 shutdown_connections 2023-10-30T13:56:27.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.693+0000 7f5e41c9a640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e280664d0 0x7f5e28068990 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:27.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.693+0000 7f5e41c9a640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e3c1539a0 0x7f5e3c0ca210 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:27.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.693+0000 7f5e41c9a640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c070a00 0x7f5e3c1f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:27.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.693+0000 7f5e41c9a640 1 --2- 172.21.15.46:0/2873496622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c069e90 0x7f5e3c1f20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.693+0000 7f5e41c9a640 1 -- 172.21.15.46:0/2873496622 >> 172.21.15.46:0/2873496622 conn(0x7f5e3c0c7910 msgr2=0x7f5e3c1452a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.693+0000 7f5e41c9a640 1 -- 172.21.15.46:0/2873496622 shutdown_connections 2023-10-30T13:56:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:27.693+0000 7f5e41c9a640 1 -- 172.21.15.46:0/2873496622 wait complete. 2023-10-30T13:56:27.716 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:56:28.716 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 10% (19/183) Updating files: 11% (21/183) Updating files: 12% (22/183) Updating files: 12% (23/183) Updating files: 13% (24/183) Updating files: 14% (26/183) Updating files: 15% (28/183) Updating files: 16% (30/183) Updating files: 17% (32/183) Updating files: 18% (33/183) Updating files: 19% (35/183) Updating files: 20% (37/183) Updating files: 21% (39/183) Updating files: 22% (41/183) Updating files: 23% (43/183) Updating files: 24% (44/183) Updating files: 25% (46/183) Updating files: 26% (48/183) Updating files: 27% (50/183) Updating files: 28% (52/183) Updating files: 29% (54/183) Updating files: 30% (55/183) Updating files: 31% (57/183) Updating files: 32% (59/183) Updating files: 33% (61/183) Updating files: 34% (63/183) Updating files: 35% (65/183) Updating files: 36% (66/183) Updating files: 37% (68/183) Updating files: 38% (70/183) Updating files: 39% (72/183) Updating files: 40% (74/183) Updating files: 41% (76/183) Updating files: 42% (77/183) Updating files: 43% (79/183) Updating files: 44% (81/183) Updating files: 45% (83/183) Updating files: 46% (85/183) Updating files: 47% (87/183) Updating files: 48% (88/183) Updating files: 49% (90/183) Updating files: 50% (92/183) Updating files: 51% (94/183) Updating files: 52% (96/183) Updating files: 53% (97/183) Updating files: 54% (99/183) Updating files: 55% (101/183) Updating files: 56% (103/183) Updating files: 57% (105/183) Updating files: 58% (107/183) Updating files: 59% (108/183) Updating files: 60% (110/183) Updating files: 61% (112/183) Updating files: 62% (114/183) Updating files: 63% (116/183) Updating files: 64% (118/183) Updating files: 65% (119/183) Updating files: 65% (120/183) Updating files: 66% (121/183) Updating files: 67% (123/183) Updating files: 68% (125/183) Updating files: 69% (127/183) Updating files: 70% (129/183) Updating files: 71% (130/183) Updating files: 72% (132/183) Updating files: 73% (134/183) Updating files: 74% (136/183) Updating files: 75% (138/183) Updating files: 76% (140/183) Updating files: 77% (141/183) Updating files: 78% (143/183) Updating files: 79% (145/183) Updating files: 80% (147/183) Updating files: 81% (149/183) Updating files: 82% (151/183) Updating files: 83% (152/183) Updating files: 84% (154/183) Updating files: 85% (156/183) Updating files: 86% (158/183) Updating files: 87% (160/183) Updating files: 88% (162/183) Updating files: 89% (163/183) Updating files: 90% (165/183) Updating files: 91% (167/183) Updating files: 92% (169/183) Updating files: 93% (171/183) Updating files: 94% (173/183) Updating files: 95% (174/183) Updating files: 96% (176/183) Updating files: 97% (178/183) Updating files: 98% (180/183) Updating files: 99% (182/183) Updating files: 100% (183/183) Updating files: 100% (183/183), done. 2023-10-30T13:56:28.894 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at e48859474c4 v0.52 2023-10-30T13:56:28.899 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.52 2023-10-30T13:56:28.940 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 53 -eq 48 ] 2023-10-30T13:56:28.940 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.53 2023-10-30T13:56:28.941 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.53 2023-10-30T13:56:28.941 INFO:tasks.workunit.client.0.smithi046.stdout:v0.53 2023-10-30T13:56:28.941 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.53 2023-10-30T13:56:29.716 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:56:29.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152fcf1640 1 Processor -- start 2023-10-30T13:56:29.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152fcf1640 1 -- start start 2023-10-30T13:56:29.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152fcf1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f15281538a0 0x7f152815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:29.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152fcf1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f15280cf160 0x7f15280cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:29.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152fcf1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15280cb800 con 0x7f15280ca0e0 2023-10-30T13:56:29.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152fcf1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f152815dc30 con 0x7f15280cf160 2023-10-30T13:56:29.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152fcf1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15280cb330 con 0x7f15281538a0 2023-10-30T13:56:29.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152e28e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f15280cf160 0x7f15280cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:29.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152d28c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f15280ca0e0 0x7f152815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47010/0 (socket says 172.21.15.46:47010) 2023-10-30T13:56:29.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152d28c640 1 -- 172.21.15.46:0/3415672196 learned_addr learned my addr 172.21.15.46:0/3415672196 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:56:29.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152da8d640 1 --2- 172.21.15.46:0/3415672196 >> v2:172.21.15.154:3300/0 conn(0x7f15281538a0 0x7f152815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:29.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152e28e640 1 -- 172.21.15.46:0/3415672196 >> v2:172.21.15.154:3300/0 conn(0x7f15281538a0 msgr2=0x7f152815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:56:29.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152e28e640 1 --2- 172.21.15.46:0/3415672196 >> v2:172.21.15.154:3300/0 conn(0x7f15281538a0 0x7f152815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152e28e640 1 -- 172.21.15.46:0/3415672196 >> v1:172.21.15.46:6789/0 conn(0x7f15280ca0e0 legacy=0x7f152815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:29.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152e28e640 1 -- 172.21.15.46:0/3415672196 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15280cad70 con 0x7f15280cf160 2023-10-30T13:56:29.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.861+0000 7f152ca8b640 1 -- 172.21.15.46:0/3415672196 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419230105 0 0) 0x7f15280cb800 con 0x7f15280ca0e0 2023-10-30T13:56:29.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.865+0000 7f152e28e640 1 --2- 172.21.15.46:0/3415672196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15280cf160 0x7f15280cf590 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f15280cfd10 tx=0x7f152001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31ca9d9f1366bee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:29.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.865+0000 7f152ca8b640 1 -- 172.21.15.46:0/3415672196 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15200b0440 con 0x7f15280cf160 2023-10-30T13:56:29.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.865+0000 7f152ca8b640 1 -- 172.21.15.46:0/3415672196 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15200b05a0 con 0x7f15280cf160 2023-10-30T13:56:29.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.865+0000 7f152ca8b640 1 -- 172.21.15.46:0/3415672196 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15200b0930 con 0x7f15280cf160 2023-10-30T13:56:29.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.865+0000 7f152fcf1640 1 -- 172.21.15.46:0/3415672196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15280cf160 msgr2=0x7f15280cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:29.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.865+0000 7f152fcf1640 1 --2- 172.21.15.46:0/3415672196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15280cf160 0x7f15280cf590 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f15280cfd10 tx=0x7f152001f0f0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.865+0000 7f152fcf1640 1 -- 172.21.15.46:0/3415672196 shutdown_connections 2023-10-30T13:56:29.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.865+0000 7f152fcf1640 1 --2- 172.21.15.46:0/3415672196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15280cf160 0x7f15280cf590 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.865+0000 7f152fcf1640 1 --2- 172.21.15.46:0/3415672196 >> v2:172.21.15.154:3300/0 conn(0x7f15281538a0 0x7f152815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.865+0000 7f152fcf1640 1 -- 172.21.15.46:0/3415672196 >> 172.21.15.46:0/3415672196 conn(0x7f152806bfe0 msgr2=0x7f152806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:29.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.865+0000 7f152fcf1640 1 -- 172.21.15.46:0/3415672196 shutdown_connections 2023-10-30T13:56:29.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.865+0000 7f152fcf1640 1 -- 172.21.15.46:0/3415672196 wait complete. 2023-10-30T13:56:29.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152fcf1640 1 Processor -- start 2023-10-30T13:56:29.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152fcf1640 1 -- start start 2023-10-30T13:56:29.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152fcf1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15280ca0e0 0x7f15280b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:29.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152fcf1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15280cf160 0x7f15280ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:29.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152fcf1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15281538a0 0x7f15280b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:29.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152fcf1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15280c5a00 con 0x7f15281538a0 2023-10-30T13:56:29.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152fcf1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1528158200 con 0x7f15280cf160 2023-10-30T13:56:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152fcf1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1528157470 con 0x7f15280ca0e0 2023-10-30T13:56:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152e28e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15281538a0 0x7f15280b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152e28e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15281538a0 0x7f15280b4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48852/0 (socket says 172.21.15.46:48852) 2023-10-30T13:56:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152e28e640 1 -- 172.21.15.46:0/2322528327 learned_addr learned my addr 172.21.15.46:0/2322528327 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:56:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152da8d640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15280ca0e0 0x7f15280b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152d28c640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15280cf160 0x7f15280ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:29.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152e28e640 1 -- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15280ca0e0 msgr2=0x7f15280b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:29.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152e28e640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15280ca0e0 0x7f15280b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152e28e640 1 -- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15280cf160 msgr2=0x7f15280ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:29.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152e28e640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15280cf160 0x7f15280ce650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:29.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152e28e640 1 -- 172.21.15.46:0/2322528327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1520009a70 con 0x7f15281538a0 2023-10-30T13:56:29.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.869+0000 7f152da8d640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15280ca0e0 0x7f15280b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:56:29.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.873+0000 7f152e28e640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15281538a0 0x7f15280b4d30 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f152001bd00 tx=0x7f152001be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.873+0000 7f151e7fc640 1 -- 172.21.15.46:0/2322528327 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15200b0440 con 0x7f15281538a0 2023-10-30T13:56:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.873+0000 7f151e7fc640 1 -- 172.21.15.46:0/2322528327 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15200bc860 con 0x7f15281538a0 2023-10-30T13:56:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.873+0000 7f151e7fc640 1 -- 172.21.15.46:0/2322528327 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15200bcc10 con 0x7f15281538a0 2023-10-30T13:56:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.873+0000 7f152fcf1640 1 -- 172.21.15.46:0/2322528327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15281efb50 con 0x7f15281538a0 2023-10-30T13:56:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.873+0000 7f152fcf1640 1 -- 172.21.15.46:0/2322528327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15281eed90 con 0x7f15281538a0 2023-10-30T13:56:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.873+0000 7f152fcf1640 1 -- 172.21.15.46:0/2322528327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15280b2a50 con 0x7f15281538a0 2023-10-30T13:56:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.873+0000 7f151e7fc640 1 -- 172.21.15.46:0/2322528327 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15200c7460 con 0x7f15281538a0 2023-10-30T13:56:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.873+0000 7f151e7fc640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f150c0664a0 0x7f150c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:56:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.873+0000 7f151e7fc640 1 -- 172.21.15.46:0/2322528327 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f15201232a0 con 0x7f15281538a0 2023-10-30T13:56:29.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.873+0000 7f152da8d640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f150c0664a0 0x7f150c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:56:29.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.877+0000 7f152da8d640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f150c0664a0 0x7f150c068960 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f1524007a70 tx=0x7f1524008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:56:29.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:29.881+0000 7f151e7fc640 1 -- 172.21.15.46:0/2322528327 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15200effa0 con 0x7f15281538a0 2023-10-30T13:56:30.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.081+0000 7f152fcf1640 1 -- 172.21.15.46:0/2322528327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f152804e170 con 0x7f15281538a0 2023-10-30T13:56:30.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.081+0000 7f151e7fc640 1 -- 172.21.15.46:0/2322528327 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f1520009450 con 0x7f15281538a0 2023-10-30T13:56:30.086 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:56:30.086 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:28.365952+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:56:30.087 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 151 2023-10-30T13:56:30.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.085+0000 7f152fcf1640 1 -- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f150c0664a0 msgr2=0x7f150c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:30.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.085+0000 7f152fcf1640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f150c0664a0 0x7f150c068960 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f1524007a70 tx=0x7f1524008040 comp rx=0 tx=0).stop 2023-10-30T13:56:30.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.085+0000 7f152fcf1640 1 -- 172.21.15.46:0/2322528327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15281538a0 msgr2=0x7f15280b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:56:30.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.085+0000 7f152fcf1640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15281538a0 0x7f15280b4d30 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f152001bd00 tx=0x7f152001be90 comp rx=0 tx=0).stop 2023-10-30T13:56:30.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.085+0000 7f152fcf1640 1 -- 172.21.15.46:0/2322528327 shutdown_connections 2023-10-30T13:56:30.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.085+0000 7f152fcf1640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f150c0664a0 0x7f150c068960 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:30.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.085+0000 7f152fcf1640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15281538a0 0x7f15280b4d30 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:30.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.085+0000 7f152fcf1640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15280cf160 0x7f15280ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:30.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.085+0000 7f152fcf1640 1 --2- 172.21.15.46:0/2322528327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15280ca0e0 0x7f15280b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:56:30.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.085+0000 7f152fcf1640 1 -- 172.21.15.46:0/2322528327 >> 172.21.15.46:0/2322528327 conn(0x7f152806bfe0 msgr2=0x7f1528066310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:56:30.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.085+0000 7f152fcf1640 1 -- 172.21.15.46:0/2322528327 shutdown_connections 2023-10-30T13:56:30.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:56:30.085+0000 7f152fcf1640 1 -- 172.21.15.46:0/2322528327 wait complete. 2023-10-30T13:56:30.106 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:56:30.106 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2023-10-30T13:56:43.475 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 15% (58/386) Updating files: 16% (62/386) Updating files: 17% (66/386) Updating files: 18% (70/386) Updating files: 19% (74/386) Updating files: 20% (78/386) Updating files: 21% (82/386) Updating files: 22% (85/386) Updating files: 23% (89/386) Updating files: 24% (93/386) Updating files: 25% (97/386) Updating files: 26% (101/386) Updating files: 27% (105/386) Updating files: 28% (109/386) Updating files: 29% (112/386) Updating files: 30% (116/386) Updating files: 31% (120/386) Updating files: 32% (124/386) Updating files: 33% (128/386) Updating files: 34% (132/386) Updating files: 35% (136/386) Updating files: 36% (139/386) Updating files: 37% (143/386) Updating files: 38% (147/386) Updating files: 39% (151/386) Updating files: 39% (154/386) Updating files: 40% (155/386) Updating files: 41% (159/386) Updating files: 42% (163/386) Updating files: 43% (166/386) Updating files: 44% (170/386) Updating files: 45% (174/386) Updating files: 46% (178/386) Updating files: 47% (182/386) Updating files: 48% (186/386) Updating files: 49% (190/386) Updating files: 50% (193/386) Updating files: 51% (197/386) Updating files: 52% (201/386) Updating files: 53% (205/386) Updating files: 54% (209/386) Updating files: 55% (213/386) Updating files: 56% (217/386) Updating files: 57% (221/386) Updating files: 58% (224/386) Updating files: 59% (228/386) Updating files: 60% (232/386) Updating files: 61% (236/386) Updating files: 62% (240/386) Updating files: 63% (244/386) Updating files: 63% (246/386) Updating files: 64% (248/386) Updating files: 65% (251/386) Updating files: 66% (255/386) Updating files: 67% (259/386) Updating files: 68% (263/386) Updating files: 69% (267/386) Updating files: 70% (271/386) Updating files: 71% (275/386) Updating files: 72% (278/386) Updating files: 73% (282/386) Updating files: 73% (284/386) Updating files: 74% (286/386) Updating files: 75% (290/386) Updating files: 76% (294/386) Updating files: 77% (298/386) Updating files: 78% (302/386) Updating files: 79% (305/386) Updating files: 80% (309/386) Updating files: 81% (313/386) Updating files: 82% (317/386) Updating files: 83% (321/386) Updating files: 84% (325/386) Updating files: 85% (329/386) Updating files: 86% (332/386) Updating files: 87% (336/386) Updating files: 88% (340/386) Updating files: 89% (344/386) Updating files: 90% (348/386) Updating files: 91% (352/386) Updating files: 92% (356/386) Updating files: 93% (359/386) Updating files: 94% (363/386) Updating files: 95% (367/386) Updating files: 96% (371/386) Updating files: 97% (375/386) Updating files: 98% (379/386) Updating files: 99% (383/386) Updating files: 100% (386/386) Updating files: 100% (386/386), done. 2023-10-30T13:56:43.544 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 2528b5ee105 v0.43 2023-10-30T13:56:43.549 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.53 2023-10-30T13:56:43.602 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 54 -eq 48 ] 2023-10-30T13:56:43.602 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.54 2023-10-30T13:56:43.602 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.54 2023-10-30T13:56:43.603 INFO:tasks.workunit.client.0.smithi046.stdout:v0.54 2023-10-30T13:56:43.604 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.54 2023-10-30T13:56:51.646 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 53% (113/212) Updating files: 54% (115/212) Updating files: 55% (117/212) Updating files: 56% (119/212) Updating files: 57% (121/212) Updating files: 58% (123/212) Updating files: 59% (126/212) Updating files: 60% (128/212) Updating files: 61% (130/212) Updating files: 62% (132/212) Updating files: 63% (134/212) Updating files: 64% (136/212) Updating files: 65% (138/212) Updating files: 66% (140/212) Updating files: 67% (143/212) Updating files: 68% (145/212) Updating files: 69% (147/212) Updating files: 70% (149/212) Updating files: 71% (151/212) Updating files: 72% (153/212) Updating files: 73% (155/212) Updating files: 74% (157/212) Updating files: 75% (159/212) Updating files: 76% (162/212) Updating files: 77% (164/212) Updating files: 78% (166/212) Updating files: 79% (168/212) Updating files: 80% (170/212) Updating files: 81% (172/212) Updating files: 82% (174/212) Updating files: 83% (176/212) Updating files: 84% (179/212) Updating files: 85% (181/212) Updating files: 86% (183/212) Updating files: 87% (185/212) Updating files: 88% (187/212) Updating files: 89% (189/212) Updating files: 90% (191/212) Updating files: 91% (193/212) Updating files: 92% (196/212) Updating files: 93% (198/212) Updating files: 94% (200/212) Updating files: 95% (202/212) Updating files: 96% (204/212) Updating files: 97% (206/212) Updating files: 98% (208/212) Updating files: 99% (210/212) Updating files: 100% (212/212) Updating files: 100% (212/212), done. 2023-10-30T13:56:51.727 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 60b84b095b1 v0.54 2023-10-30T13:56:51.735 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.54 2023-10-30T13:56:51.956 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 55 -eq 48 ] 2023-10-30T13:56:51.956 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.55 2023-10-30T13:56:51.957 INFO:tasks.workunit.client.0.smithi046.stdout:v0.55 2023-10-30T13:56:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.55 2023-10-30T13:56:51.957 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.55 2023-10-30T13:56:52.963 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:56:52.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:56:59.325 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 12% (68/540) Updating files: 13% (71/540) Updating files: 14% (76/540) Updating files: 15% (81/540) Updating files: 16% (87/540) Updating files: 17% (92/540) Updating files: 18% (98/540) Updating files: 19% (103/540) Updating files: 20% (108/540) Updating files: 21% (114/540) Updating files: 22% (119/540) Updating files: 23% (125/540) Updating files: 24% (130/540) Updating files: 25% (135/540) Updating files: 26% (141/540) Updating files: 27% (146/540) Updating files: 28% (152/540) Updating files: 29% (157/540) Updating files: 30% (162/540) Updating files: 31% (168/540) Updating files: 32% (173/540) Updating files: 33% (179/540) Updating files: 34% (184/540) Updating files: 35% (189/540) Updating files: 36% (195/540) Updating files: 37% (200/540) Updating files: 37% (201/540) Updating files: 38% (206/540) Updating files: 39% (211/540) Updating files: 40% (216/540) Updating files: 41% (222/540) Updating files: 42% (227/540) Updating files: 43% (233/540) Updating files: 44% (238/540) Updating files: 45% (243/540) Updating files: 46% (249/540) Updating files: 47% (254/540) Updating files: 48% (260/540) Updating files: 49% (265/540) Updating files: 50% (270/540) Updating files: 51% (276/540) Updating files: 52% (281/540) Updating files: 53% (287/540) Updating files: 54% (292/540) Updating files: 55% (297/540) Updating files: 55% (300/540) Updating files: 56% (303/540) Updating files: 57% (308/540) Updating files: 58% (314/540) Updating files: 59% (319/540) Updating files: 60% (324/540) Updating files: 61% (330/540) Updating files: 61% (334/540) Updating files: 62% (335/540) Updating files: 62% (337/540) Updating files: 63% (341/540) Updating files: 64% (346/540) Updating files: 65% (351/540) Updating files: 66% (357/540) Updating files: 67% (362/540) Updating files: 68% (368/540) Updating files: 69% (373/540) Updating files: 69% (376/540) Updating files: 70% (378/540) Updating files: 71% (384/540) Updating files: 72% (389/540) Updating files: 73% (395/540) Updating files: 74% (400/540) Updating files: 75% (405/540) Updating files: 76% (411/540) Updating files: 77% (416/540) Updating files: 78% (422/540) Updating files: 79% (427/540) Updating files: 80% (432/540) Updating files: 81% (438/540) Updating files: 82% (443/540) Updating files: 83% (449/540) Updating files: 84% (454/540) Updating files: 85% (459/540) Updating files: 86% (465/540) Updating files: 87% (470/540) Updating files: 88% (476/540) Updating files: 89% (481/540) Updating files: 90% (486/540) Updating files: 91% (492/540) Updating files: 92% (497/540) Updating files: 93% (503/540) Updating files: 94% (508/540) Updating files: 95% (513/540) Updating files: 96% (519/540) Updating files: 97% (524/540) Updating files: 98% (530/540) Updating files: 99% (535/540) Updating files: 100% (540/540) Updating files: 100% (540/540), done. 2023-10-30T13:56:59.441 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 690f8175606 v0.55 2023-10-30T13:56:59.449 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.55 2023-10-30T13:56:59.517 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 56 -eq 48 ] 2023-10-30T13:56:59.517 INFO:tasks.workunit.client.0.smithi046.stdout:v0.56 2023-10-30T13:56:59.517 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.56 2023-10-30T13:56:59.518 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.56 2023-10-30T13:56:59.518 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.56 2023-10-30T13:57:07.172 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 29% (73/246) Updating files: 30% (74/246) Updating files: 31% (77/246) Updating files: 32% (79/246) Updating files: 33% (82/246) Updating files: 34% (84/246) Updating files: 35% (87/246) Updating files: 36% (89/246) Updating files: 37% (92/246) Updating files: 38% (94/246) Updating files: 39% (96/246) Updating files: 40% (99/246) Updating files: 41% (101/246) Updating files: 42% (104/246) Updating files: 43% (106/246) Updating files: 44% (109/246) Updating files: 45% (111/246) Updating files: 46% (114/246) Updating files: 47% (116/246) Updating files: 48% (119/246) Updating files: 49% (121/246) Updating files: 50% (123/246) Updating files: 51% (126/246) Updating files: 52% (128/246) Updating files: 53% (131/246) Updating files: 54% (133/246) Updating files: 55% (136/246) Updating files: 56% (138/246) Updating files: 57% (141/246) Updating files: 58% (143/246) Updating files: 59% (146/246) Updating files: 60% (148/246) Updating files: 61% (151/246) Updating files: 62% (153/246) Updating files: 63% (155/246) Updating files: 64% (158/246) Updating files: 65% (160/246) Updating files: 66% (163/246) Updating files: 67% (165/246) Updating files: 67% (166/246) Updating files: 68% (168/246) Updating files: 69% (170/246) Updating files: 70% (173/246) Updating files: 71% (175/246) Updating files: 72% (178/246) Updating files: 73% (180/246) Updating files: 74% (183/246) Updating files: 75% (185/246) Updating files: 76% (187/246) Updating files: 76% (189/246) Updating files: 77% (190/246) Updating files: 78% (192/246) Updating files: 79% (195/246) Updating files: 80% (197/246) Updating files: 81% (200/246) Updating files: 82% (202/246) Updating files: 83% (205/246) Updating files: 84% (207/246) Updating files: 85% (210/246) Updating files: 86% (212/246) Updating files: 87% (215/246) Updating files: 88% (217/246) Updating files: 89% (219/246) Updating files: 90% (222/246) Updating files: 91% (224/246) Updating files: 92% (227/246) Updating files: 93% (229/246) Updating files: 94% (232/246) Updating files: 95% (234/246) Updating files: 96% (237/246) Updating files: 97% (239/246) Updating files: 98% (242/246) Updating files: 99% (244/246) Updating files: 100% (246/246) Updating files: 100% (246/246), done. 2023-10-30T13:57:07.263 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 1a32f0a0b42 v0.56 2023-10-30T13:57:07.270 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.56 2023-10-30T13:57:07.551 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 57 -eq 48 ] 2023-10-30T13:57:07.552 INFO:tasks.workunit.client.0.smithi046.stdout:v0.57 2023-10-30T13:57:07.552 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.57 2023-10-30T13:57:07.552 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.57 2023-10-30T13:57:07.552 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.57 2023-10-30T13:57:15.089 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 32% (99/304) Updating files: 33% (101/304) Updating files: 34% (104/304) Updating files: 35% (107/304) Updating files: 36% (110/304) Updating files: 37% (113/304) Updating files: 38% (116/304) Updating files: 39% (119/304) Updating files: 40% (122/304) Updating files: 41% (125/304) Updating files: 42% (128/304) Updating files: 43% (131/304) Updating files: 44% (134/304) Updating files: 45% (137/304) Updating files: 46% (140/304) Updating files: 47% (143/304) Updating files: 48% (146/304) Updating files: 49% (149/304) Updating files: 50% (152/304) Updating files: 51% (156/304) Updating files: 52% (159/304) Updating files: 53% (162/304) Updating files: 54% (165/304) Updating files: 55% (168/304) Updating files: 56% (171/304) Updating files: 57% (174/304) Updating files: 58% (177/304) Updating files: 58% (178/304) Updating files: 59% (180/304) Updating files: 59% (181/304) Updating files: 60% (183/304) Updating files: 61% (186/304) Updating files: 62% (189/304) Updating files: 63% (192/304) Updating files: 64% (195/304) Updating files: 65% (198/304) Updating files: 66% (201/304) Updating files: 67% (204/304) Updating files: 68% (207/304) Updating files: 69% (210/304) Updating files: 70% (213/304) Updating files: 71% (216/304) Updating files: 71% (217/304) Updating files: 72% (219/304) Updating files: 73% (222/304) Updating files: 74% (225/304) Updating files: 75% (228/304) Updating files: 76% (232/304) Updating files: 77% (235/304) Updating files: 78% (238/304) Updating files: 79% (241/304) Updating files: 80% (244/304) Updating files: 81% (247/304) Updating files: 82% (250/304) Updating files: 83% (253/304) Updating files: 84% (256/304) Updating files: 85% (259/304) Updating files: 86% (262/304) Updating files: 87% (265/304) Updating files: 88% (268/304) Updating files: 89% (271/304) Updating files: 90% (274/304) Updating files: 91% (277/304) Updating files: 92% (280/304) Updating files: 93% (283/304) Updating files: 94% (286/304) Updating files: 94% (287/304) Updating files: 95% (289/304) Updating files: 96% (292/304) Updating files: 97% (295/304) Updating files: 98% (298/304) Updating files: 99% (301/304) Updating files: 100% (304/304) Updating files: 100% (304/304), done. 2023-10-30T13:57:15.174 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 9a7a9d06c06 v0.57 2023-10-30T13:57:15.181 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.57 2023-10-30T13:57:15.246 INFO:tasks.workunit.client.0.smithi046.stdout:v0.58 2023-10-30T13:57:15.247 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 58 -eq 48 ] 2023-10-30T13:57:15.247 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.58 2023-10-30T13:57:15.247 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.58 2023-10-30T13:57:15.247 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.58 2023-10-30T13:57:22.851 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 8% (23/268) Updating files: 9% (25/268) Updating files: 10% (27/268) Updating files: 11% (30/268) Updating files: 12% (33/268) Updating files: 13% (35/268) Updating files: 14% (38/268) Updating files: 15% (41/268) Updating files: 16% (43/268) Updating files: 17% (46/268) Updating files: 18% (49/268) Updating files: 19% (51/268) Updating files: 20% (54/268) Updating files: 21% (57/268) Updating files: 22% (59/268) Updating files: 23% (62/268) Updating files: 23% (63/268) Updating files: 24% (65/268) Updating files: 25% (67/268) Updating files: 26% (70/268) Updating files: 27% (73/268) Updating files: 28% (76/268) Updating files: 29% (78/268) Updating files: 30% (81/268) Updating files: 31% (84/268) Updating files: 32% (86/268) Updating files: 33% (89/268) Updating files: 34% (92/268) Updating files: 35% (94/268) Updating files: 36% (97/268) Updating files: 37% (100/268) Updating files: 38% (102/268) Updating files: 39% (105/268) Updating files: 40% (108/268) Updating files: 41% (110/268) Updating files: 42% (113/268) Updating files: 43% (116/268) Updating files: 44% (118/268) Updating files: 45% (121/268) Updating files: 46% (124/268) Updating files: 47% (126/268) Updating files: 48% (129/268) Updating files: 49% (132/268) Updating files: 50% (134/268) Updating files: 51% (137/268) Updating files: 52% (140/268) Updating files: 53% (143/268) Updating files: 54% (145/268) Updating files: 55% (148/268) Updating files: 56% (151/268) Updating files: 57% (153/268) Updating files: 58% (156/268) Updating files: 59% (159/268) Updating files: 60% (161/268) Updating files: 61% (164/268) Updating files: 62% (167/268) Updating files: 63% (169/268) Updating files: 64% (172/268) Updating files: 65% (175/268) Updating files: 66% (177/268) Updating files: 67% (180/268) Updating files: 68% (183/268) Updating files: 69% (185/268) Updating files: 70% (188/268) Updating files: 70% (189/268) Updating files: 71% (191/268) Updating files: 72% (193/268) Updating files: 73% (196/268) Updating files: 74% (199/268) Updating files: 74% (200/268) Updating files: 75% (201/268) Updating files: 76% (204/268) Updating files: 77% (207/268) Updating files: 78% (210/268) Updating files: 79% (212/268) Updating files: 80% (215/268) Updating files: 81% (218/268) Updating files: 82% (220/268) Updating files: 83% (223/268) Updating files: 84% (226/268) Updating files: 85% (228/268) Updating files: 86% (231/268) Updating files: 87% (234/268) Updating files: 88% (236/268) Updating files: 89% (239/268) Updating files: 90% (242/268) Updating files: 91% (244/268) Updating files: 92% (247/268) Updating files: 93% (250/268) Updating files: 94% (252/268) Updating files: 95% (255/268) Updating files: 96% (258/268) Updating files: 97% (260/268) Updating files: 97% (262/268) Updating files: 98% (263/268) Updating files: 99% (266/268) Updating files: 100% (268/268) Updating files: 100% (268/268), done. 2023-10-30T13:57:23.066 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:57:23.070 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:57:23.452 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at ba3f91e7504 v0.58 2023-10-30T13:57:23.466 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.58 2023-10-30T13:57:23.525 INFO:tasks.workunit.client.0.smithi046.stdout:v0.59 2023-10-30T13:57:23.526 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 59 -eq 48 ] 2023-10-30T13:57:23.526 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.59 2023-10-30T13:57:23.526 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.59 2023-10-30T13:57:23.526 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.59 2023-10-30T13:57:28.107 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:28.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc3acc9640 1 Processor -- start 2023-10-30T13:57:28.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc3acc9640 1 -- start start 2023-10-30T13:57:28.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc3acc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415d6f0 0x7fdc3415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc3acc9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdc340cf160 0x7fdc340cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc3acc9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc340cb800 con 0x7fdc341538a0 2023-10-30T13:57:28.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc3acc9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc340cb330 con 0x7fdc340cf160 2023-10-30T13:57:28.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc3acc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc340cad70 con 0x7fdc3415d6f0 2023-10-30T13:57:28.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc38a65640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdc341538a0 0x7fdc340ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60042/0 (socket says 172.21.15.46:60042) 2023-10-30T13:57:28.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc38a65640 1 -- 172.21.15.46:0/3959818017 learned_addr learned my addr 172.21.15.46:0/3959818017 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:57:28.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc33fff640 1 --2- 172.21.15.46:0/3959818017 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415d6f0 0x7fdc3415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc39266640 1 --2- 172.21.15.46:0/3959818017 >> v2:172.21.15.154:3300/0 conn(0x7fdc340cf160 0x7fdc340cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057240006 0 0) 0x7fdc340cb800 con 0x7fdc341538a0 2023-10-30T13:57:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc24003530 con 0x7fdc341538a0 2023-10-30T13:57:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3307251241 0 0) 0x7fdc24003530 con 0x7fdc341538a0 2023-10-30T13:57:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415d6f0 msgr2=0x7fdc3415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 --2- 172.21.15.46:0/3959818017 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415d6f0 0x7fdc3415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 >> v2:172.21.15.154:3300/0 conn(0x7fdc340cf160 msgr2=0x7fdc340cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 --2- 172.21.15.46:0/3959818017 >> v2:172.21.15.154:3300/0 conn(0x7fdc340cf160 0x7fdc340cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc340cb050 con 0x7fdc341538a0 2023-10-30T13:57:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fdc28001680 con 0x7fdc341538a0 2023-10-30T13:57:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 >> v1:172.21.15.46:6789/0 conn(0x7fdc341538a0 legacy=0x7fdc340ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 --2- 172.21.15.46:0/3959818017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc24003980 0x7fdc24003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 --2- 172.21.15.46:0/3959818017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc240047d0 0x7fdc24004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 --2- 172.21.15.46:0/3959818017 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415d6f0 0x7fdc24005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc340cb330 con 0x7fdc24003980 2023-10-30T13:57:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc340cad70 con 0x7fdc240047d0 2023-10-30T13:57:28.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2400b9c0 con 0x7fdc3415d6f0 2023-10-30T13:57:28.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.249+0000 7fdc38a65640 1 --2- 172.21.15.46:0/3959818017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc24003980 0x7fdc24003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.253+0000 7fdc33fff640 1 --2- 172.21.15.46:0/3959818017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc240047d0 0x7fdc24004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.253+0000 7fdc39266640 1 --2- 172.21.15.46:0/3959818017 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415d6f0 0x7fdc24005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.253+0000 7fdc38a65640 1 -- 172.21.15.46:0/3959818017 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415d6f0 msgr2=0x7fdc24005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:28.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.253+0000 7fdc38a65640 1 --2- 172.21.15.46:0/3959818017 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415d6f0 0x7fdc24005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.253+0000 7fdc38a65640 1 -- 172.21.15.46:0/3959818017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc240047d0 msgr2=0x7fdc24004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:28.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.253+0000 7fdc38a65640 1 --2- 172.21.15.46:0/3959818017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc240047d0 0x7fdc24004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.253+0000 7fdc38a65640 1 -- 172.21.15.46:0/3959818017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc24003530 con 0x7fdc24003980 2023-10-30T13:57:28.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.253+0000 7fdc38a65640 1 --2- 172.21.15.46:0/3959818017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc24003980 0x7fdc24003da0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800c210 tx=0x7fdc2801f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3807837d0a75b5e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:28.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.253+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc280a8cb0 con 0x7fdc24003980 2023-10-30T13:57:28.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.253+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc280a8e10 con 0x7fdc24003980 2023-10-30T13:57:28.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.253+0000 7fdc337fe640 1 -- 172.21.15.46:0/3959818017 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc28005b20 con 0x7fdc24003980 2023-10-30T13:57:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 -- 172.21.15.46:0/3959818017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc24003980 msgr2=0x7fdc24003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 --2- 172.21.15.46:0/3959818017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc24003980 0x7fdc24003da0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800c210 tx=0x7fdc2801f390 comp rx=0 tx=0).stop 2023-10-30T13:57:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc39266640 1 -- 172.21.15.46:0/3959818017 reap_dead start 2023-10-30T13:57:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 -- 172.21.15.46:0/3959818017 shutdown_connections 2023-10-30T13:57:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 -- 172.21.15.46:0/3959818017 >> 172.21.15.46:0/3959818017 conn(0x7fdc3406bfe0 msgr2=0x7fdc3406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:28.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 -- 172.21.15.46:0/3959818017 shutdown_connections 2023-10-30T13:57:28.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 -- 172.21.15.46:0/3959818017 wait complete. 2023-10-30T13:57:28.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 Processor -- start 2023-10-30T13:57:28.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 -- start start 2023-10-30T13:57:28.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc24003980 0x7fdc341f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415bb30 0x7fdc340ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc3414f150 0x7fdc3414f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc34158e40 con 0x7fdc3414f150 2023-10-30T13:57:28.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc34150110 con 0x7fdc24003980 2023-10-30T13:57:28.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc3acc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341f2ab0 con 0x7fdc3415bb30 2023-10-30T13:57:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc39266640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc3414f150 0x7fdc3414f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc39266640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc3414f150 0x7fdc3414f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52320/0 (socket says 172.21.15.46:52320) 2023-10-30T13:57:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc39266640 1 -- 172.21.15.46:0/2700082777 learned_addr learned my addr 172.21.15.46:0/2700082777 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc38a65640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc24003980 0x7fdc341f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc33fff640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415bb30 0x7fdc340ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc39266640 1 -- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415bb30 msgr2=0x7fdc340ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc39266640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415bb30 0x7fdc340ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc39266640 1 -- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc24003980 msgr2=0x7fdc341f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc39266640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc24003980 0x7fdc341f2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.257+0000 7fdc39266640 1 -- 172.21.15.46:0/2700082777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3414fea0 con 0x7fdc3414f150 2023-10-30T13:57:28.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.261+0000 7fdc39266640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc3414f150 0x7fdc3414f600 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fdc340b1a70 tx=0x7fdc2c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:28.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.261+0000 7fdc317fa640 1 -- 172.21.15.46:0/2700082777 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c00e070 con 0x7fdc3414f150 2023-10-30T13:57:28.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.261+0000 7fdc317fa640 1 -- 172.21.15.46:0/2700082777 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2c011ed0 con 0x7fdc3414f150 2023-10-30T13:57:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.261+0000 7fdc317fa640 1 -- 172.21.15.46:0/2700082777 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c0039a0 con 0x7fdc3414f150 2023-10-30T13:57:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.261+0000 7fdc3acc9640 1 -- 172.21.15.46:0/2700082777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc340b51b0 con 0x7fdc3414f150 2023-10-30T13:57:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.261+0000 7fdc3acc9640 1 -- 172.21.15.46:0/2700082777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc340b4ed0 con 0x7fdc3414f150 2023-10-30T13:57:28.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.261+0000 7fdc3acc9640 1 -- 172.21.15.46:0/2700082777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc341f2c60 con 0x7fdc3414f150 2023-10-30T13:57:28.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.261+0000 7fdc317fa640 1 -- 172.21.15.46:0/2700082777 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdc2c003b00 con 0x7fdc3414f150 2023-10-30T13:57:28.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.261+0000 7fdc317fa640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdc180664a0 0x7fdc18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.261+0000 7fdc317fa640 1 -- 172.21.15.46:0/2700082777 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fdc2c0849a0 con 0x7fdc3414f150 2023-10-30T13:57:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.265+0000 7fdc38a65640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdc180664a0 0x7fdc18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.265+0000 7fdc38a65640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdc180664a0 0x7fdc18068960 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fdc280ad000 tx=0x7fdc28000ff0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.269+0000 7fdc317fa640 1 -- 172.21.15.46:0/2700082777 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdc341f2c60 con 0x7fdc3414f150 2023-10-30T13:57:28.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.517+0000 7fdc3acc9640 1 -- 172.21.15.46:0/2700082777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc341f2c60 con 0x7fdc3414f150 2023-10-30T13:57:28.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.517+0000 7fdc317fa640 1 -- 172.21.15.46:0/2700082777 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fdc341f2c60 con 0x7fdc3414f150 2023-10-30T13:57:28.522 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:28.522 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:28.365952+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:28.522 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 151 2023-10-30T13:57:28.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.521+0000 7fdc3acc9640 1 -- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdc180664a0 msgr2=0x7fdc18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.521+0000 7fdc3acc9640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdc180664a0 0x7fdc18068960 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fdc280ad000 tx=0x7fdc28000ff0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.521+0000 7fdc3acc9640 1 -- 172.21.15.46:0/2700082777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc3414f150 msgr2=0x7fdc3414f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.521+0000 7fdc3acc9640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc3414f150 0x7fdc3414f600 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fdc340b1a70 tx=0x7fdc2c0115a0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.521+0000 7fdc3acc9640 1 -- 172.21.15.46:0/2700082777 shutdown_connections 2023-10-30T13:57:28.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.521+0000 7fdc3acc9640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdc180664a0 0x7fdc18068960 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.521+0000 7fdc3acc9640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc3414f150 0x7fdc3414f600 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.521+0000 7fdc3acc9640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc3415bb30 0x7fdc340ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.521+0000 7fdc3acc9640 1 --2- 172.21.15.46:0/2700082777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc24003980 0x7fdc341f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.521+0000 7fdc3acc9640 1 -- 172.21.15.46:0/2700082777 >> 172.21.15.46:0/2700082777 conn(0x7fdc3406bfe0 msgr2=0x7fdc340c4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.521+0000 7fdc3acc9640 1 -- 172.21.15.46:0/2700082777 shutdown_connections 2023-10-30T13:57:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.521+0000 7fdc3acc9640 1 -- 172.21.15.46:0/2700082777 wait complete. 2023-10-30T13:57:28.550 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T13:57:28.550 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:57:28.551 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:57:28.580 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:57:28.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:28.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.761+0000 7fa103c68640 1 Processor -- start 2023-10-30T13:57:28.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.761+0000 7fa103c68640 1 -- start start 2023-10-30T13:57:28.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.761+0000 7fa103c68640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0fc1538a0 0x7fa0fc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.761+0000 7fa103c68640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa0fc0cf160 0x7fa0fc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.761+0000 7fa103c68640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc0cb800 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.761+0000 7fa103c68640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0fc15dc30 con 0x7fa0fc0cf160 2023-10-30T13:57:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.761+0000 7fa103c68640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0fc0cb330 con 0x7fa0fc1538a0 2023-10-30T13:57:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.761+0000 7fa101a04640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0fc1538a0 0x7fa0fc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.761+0000 7fa102205640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa0fc0cf160 0x7fa0fc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.761+0000 7fa102205640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa0fc0cf160 0x7fa0fc0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:42252/0 (socket says 172.21.15.46:42252) 2023-10-30T13:57:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.761+0000 7fa102205640 1 -- 172.21.15.46:0/3917549026 learned_addr learned my addr 172.21.15.46:0/3917549026 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.761+0000 7fa101203640 1 --1- 172.21.15.46:0/3917549026 >> v1:172.21.15.154:6789/0 conn(0x7fa0fc0ca0e0 0x7fa0fc15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:51882/0 (socket says 172.21.15.46:51882) 2023-10-30T13:57:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.765+0000 7fa101a04640 1 -- 172.21.15.46:0/3917549026 >> v2:172.21.15.154:3300/0 conn(0x7fa0fc0cf160 msgr2=0x7fa0fc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.765+0000 7fa101a04640 1 --2- 172.21.15.46:0/3917549026 >> v2:172.21.15.154:3300/0 conn(0x7fa0fc0cf160 0x7fa0fc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.765+0000 7fa101a04640 1 -- 172.21.15.46:0/3917549026 >> v1:172.21.15.154:6789/0 conn(0x7fa0fc0ca0e0 legacy=0x7fa0fc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.765+0000 7fa101a04640 1 -- 172.21.15.46:0/3917549026 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0fc0cad70 con 0x7fa0fc1538a0 2023-10-30T13:57:28.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.765+0000 7fa102205640 1 --2- 172.21.15.46:0/3917549026 >> v2:172.21.15.154:3300/0 conn(0x7fa0fc0cf160 0x7fa0fc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:57:28.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.765+0000 7fa101a04640 1 --2- 172.21.15.46:0/3917549026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0fc1538a0 0x7fa0fc15d6f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80097c0 tx=0x7fa0f801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35c62908f248320f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:28.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.765+0000 7fa100a02640 1 -- 172.21.15.46:0/3917549026 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80a8970 con 0x7fa0fc1538a0 2023-10-30T13:57:28.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.765+0000 7fa100a02640 1 -- 172.21.15.46:0/3917549026 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f80a8ad0 con 0x7fa0fc1538a0 2023-10-30T13:57:28.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.765+0000 7fa100a02640 1 -- 172.21.15.46:0/3917549026 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80a8e60 con 0x7fa0fc1538a0 2023-10-30T13:57:28.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.765+0000 7fa103c68640 1 -- 172.21.15.46:0/3917549026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0fc1538a0 msgr2=0x7fa0fc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.765+0000 7fa103c68640 1 --2- 172.21.15.46:0/3917549026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0fc1538a0 0x7fa0fc15d6f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80097c0 tx=0x7fa0f801ec70 comp rx=0 tx=0).stop 2023-10-30T13:57:28.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 -- 172.21.15.46:0/3917549026 shutdown_connections 2023-10-30T13:57:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 --2- 172.21.15.46:0/3917549026 >> v2:172.21.15.154:3300/0 conn(0x7fa0fc0cf160 0x7fa0fc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 --2- 172.21.15.46:0/3917549026 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0fc1538a0 0x7fa0fc15d6f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 -- 172.21.15.46:0/3917549026 >> 172.21.15.46:0/3917549026 conn(0x7fa0fc06bfe0 msgr2=0x7fa0fc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 -- 172.21.15.46:0/3917549026 shutdown_connections 2023-10-30T13:57:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 -- 172.21.15.46:0/3917549026 wait complete. 2023-10-30T13:57:28.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 Processor -- start 2023-10-30T13:57:28.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 -- start start 2023-10-30T13:57:28.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0fc0ca0e0 0x7fa0fc15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0fc0cf160 0x7fa0fc14e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0fc1538a0 0x7fa0fc1eefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0fc15b360 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0fc15b010 con 0x7fa0fc1538a0 2023-10-30T13:57:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa103c68640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0fc1f2320 con 0x7fa0fc0cf160 2023-10-30T13:57:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa101a04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0fc0ca0e0 0x7fa0fc15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa101a04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0fc0ca0e0 0x7fa0fc15a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52324/0 (socket says 172.21.15.46:52324) 2023-10-30T13:57:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa101a04640 1 -- 172.21.15.46:0/1586459217 learned_addr learned my addr 172.21.15.46:0/1586459217 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa101203640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0fc0cf160 0x7fa0fc14e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.769+0000 7fa102205640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0fc1538a0 0x7fa0fc1eefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa101a04640 1 -- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0fc0cf160 msgr2=0x7fa0fc14e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa101a04640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0fc0cf160 0x7fa0fc14e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa101a04640 1 -- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0fc1538a0 msgr2=0x7fa0fc1eefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa101a04640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0fc1538a0 0x7fa0fc1eefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa101a04640 1 -- 172.21.15.46:0/1586459217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f4009b30 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa101a04640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0fc0ca0e0 0x7fa0fc15a8b0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fa0f801c8a0 tx=0x7fa0f801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:28.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa0f27fc640 1 -- 172.21.15.46:0/1586459217 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80bb640 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa0f27fc640 1 -- 172.21.15.46:0/1586459217 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f80bb7c0 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa103c68640 1 -- 172.21.15.46:0/1586459217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f8009420 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa0f27fc640 1 -- 172.21.15.46:0/1586459217 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80bbb70 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa103c68640 1 -- 172.21.15.46:0/1586459217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0fc1f2810 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa103c68640 1 -- 172.21.15.46:0/1586459217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0fc0b2a50 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa0f27fc640 1 -- 172.21.15.46:0/1586459217 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa0f80b7040 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa0f27fc640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0e00664a0 0x7fa0e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.773+0000 7fa0f27fc640 1 -- 172.21.15.46:0/1586459217 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fa0f8122790 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.777+0000 7fa101203640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0e00664a0 0x7fa0e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.777+0000 7fa101203640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0e00664a0 0x7fa0e0068960 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc1e4590 tx=0x7fa0ec007e50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:28.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.781+0000 7fa0f27fc640 1 -- 172.21.15.46:0/1586459217 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa0f80ef540 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 -- 172.21.15.46:0/1586459217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0fc1eee00 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa0f27fc640 1 -- 172.21.15.46:0/1586459217 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fa0f80f2b60 con 0x7fa0fc0ca0e0 2023-10-30T13:57:28.973 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:28.973 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:28.365952+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:28.974 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 151 2023-10-30T13:57:28.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 -- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0e00664a0 msgr2=0x7fa0e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0e00664a0 0x7fa0e0068960 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc1e4590 tx=0x7fa0ec007e50 comp rx=0 tx=0).stop 2023-10-30T13:57:28.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 -- 172.21.15.46:0/1586459217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0fc0ca0e0 msgr2=0x7fa0fc15a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:28.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0fc0ca0e0 0x7fa0fc15a8b0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fa0f801c8a0 tx=0x7fa0f801ba10 comp rx=0 tx=0).stop 2023-10-30T13:57:28.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 -- 172.21.15.46:0/1586459217 shutdown_connections 2023-10-30T13:57:28.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0e00664a0 0x7fa0e0068960 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0fc1538a0 0x7fa0fc1eefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0fc0cf160 0x7fa0fc14e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 --2- 172.21.15.46:0/1586459217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0fc0ca0e0 0x7fa0fc15a8b0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:28.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 -- 172.21.15.46:0/1586459217 >> 172.21.15.46:0/1586459217 conn(0x7fa0fc06bfe0 msgr2=0x7fa0fc0cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:28.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 -- 172.21.15.46:0/1586459217 shutdown_connections 2023-10-30T13:57:28.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:28.969+0000 7fa103c68640 1 -- 172.21.15.46:0/1586459217 wait complete. 2023-10-30T13:57:28.994 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:57:30.996 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:31.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.145+0000 7f62863c4640 1 Processor -- start 2023-10-30T13:57:31.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.145+0000 7f62863c4640 1 -- start start 2023-10-30T13:57:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f62863c4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f62801539a0 0x7f628015a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f62863c4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6280069e90 0x7f628006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:31.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f62863c4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628015a8c0 con 0x7f6280070a00 2023-10-30T13:57:31.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f62863c4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628006fbb0 con 0x7f6280069e90 2023-10-30T13:57:31.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f62863c4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6280153d80 con 0x7f62801539a0 2023-10-30T13:57:31.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f6284961640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6280069e90 0x7f628006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:31.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f6284961640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6280069e90 0x7f628006a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52326/0 (socket says 172.21.15.46:52326) 2023-10-30T13:57:31.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f6284961640 1 -- 172.21.15.46:0/687925576 learned_addr learned my addr 172.21.15.46:0/687925576 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:31.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f627ffff640 1 --2- 172.21.15.46:0/687925576 >> v2:172.21.15.154:3300/0 conn(0x7f62801539a0 0x7f628015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:31.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f6284961640 1 -- 172.21.15.46:0/687925576 >> v2:172.21.15.154:3300/0 conn(0x7f62801539a0 msgr2=0x7f628015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:31.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f6284961640 1 --2- 172.21.15.46:0/687925576 >> v2:172.21.15.154:3300/0 conn(0x7f62801539a0 0x7f628015a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f6284961640 1 -- 172.21.15.46:0/687925576 >> v1:172.21.15.154:6789/0 conn(0x7f6280070a00 legacy=0x7f6280070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:31.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.149+0000 7f6284961640 1 -- 172.21.15.46:0/687925576 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62800c5c50 con 0x7f6280069e90 2023-10-30T13:57:31.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f6284961640 1 --2- 172.21.15.46:0/687925576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6280069e90 0x7f628006a300 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f62800c8950 tx=0x7f627801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e199cb2ce974545 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:31.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f627effd640 1 -- 172.21.15.46:0/687925576 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62780a56a0 con 0x7f6280069e90 2023-10-30T13:57:31.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f627effd640 1 -- 172.21.15.46:0/687925576 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62780a5800 con 0x7f6280069e90 2023-10-30T13:57:31.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f627effd640 1 -- 172.21.15.46:0/687925576 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62780a5b90 con 0x7f6280069e90 2023-10-30T13:57:31.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f62863c4640 1 -- 172.21.15.46:0/687925576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6280069e90 msgr2=0x7f628006a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:31.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f62863c4640 1 --2- 172.21.15.46:0/687925576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6280069e90 0x7f628006a300 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f62800c8950 tx=0x7f627801c8d0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f62863c4640 1 -- 172.21.15.46:0/687925576 shutdown_connections 2023-10-30T13:57:31.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f62863c4640 1 --2- 172.21.15.46:0/687925576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6280069e90 0x7f628006a300 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f62863c4640 1 --2- 172.21.15.46:0/687925576 >> v2:172.21.15.154:3300/0 conn(0x7f62801539a0 0x7f628015a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f62863c4640 1 -- 172.21.15.46:0/687925576 >> 172.21.15.46:0/687925576 conn(0x7f62800c7910 msgr2=0x7f62800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:31.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f62863c4640 1 -- 172.21.15.46:0/687925576 shutdown_connections 2023-10-30T13:57:31.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f62863c4640 1 -- 172.21.15.46:0/687925576 wait complete. 2023-10-30T13:57:31.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f62863c4640 1 Processor -- start 2023-10-30T13:57:31.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.153+0000 7f62863c4640 1 -- start start 2023-10-30T13:57:31.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f62863c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6280069e90 0x7f62800c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:31.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f62863c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6280070a00 0x7f6280149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:31.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f62863c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62801539a0 0x7f6280142d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:31.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f62863c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62801474f0 con 0x7f6280069e90 2023-10-30T13:57:31.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f62863c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62800bfec0 con 0x7f6280070a00 2023-10-30T13:57:31.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f62863c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62800c00f0 con 0x7f62801539a0 2023-10-30T13:57:31.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6280069e90 0x7f62800c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:31.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6280069e90 0x7f62800c0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52330/0 (socket says 172.21.15.46:52330) 2023-10-30T13:57:31.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627ffff640 1 -- 172.21.15.46:0/2074580625 learned_addr learned my addr 172.21.15.46:0/2074580625 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:31.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f6284961640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62801539a0 0x7f6280142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:31.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627f7fe640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6280070a00 0x7f6280149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:31.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627ffff640 1 -- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62801539a0 msgr2=0x7f6280142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:31.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627ffff640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62801539a0 0x7f6280142d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627ffff640 1 -- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6280070a00 msgr2=0x7f6280149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:31.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627ffff640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6280070a00 0x7f6280149170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627ffff640 1 -- 172.21.15.46:0/2074580625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6278009b30 con 0x7f6280069e90 2023-10-30T13:57:31.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627ffff640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6280069e90 0x7f62800c0f90 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f627000d620 tx=0x7f627000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:31.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627cff9640 1 -- 172.21.15.46:0/2074580625 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627000b620 con 0x7f6280069e90 2023-10-30T13:57:31.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627cff9640 1 -- 172.21.15.46:0/2074580625 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627000b780 con 0x7f6280069e90 2023-10-30T13:57:31.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627cff9640 1 -- 172.21.15.46:0/2074580625 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627000bb10 con 0x7f6280069e90 2023-10-30T13:57:31.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f62863c4640 1 -- 172.21.15.46:0/2074580625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62800c03d0 con 0x7f6280069e90 2023-10-30T13:57:31.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f62863c4640 1 -- 172.21.15.46:0/2074580625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6280158c00 con 0x7f6280069e90 2023-10-30T13:57:31.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f62863c4640 1 -- 172.21.15.46:0/2074580625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628004e170 con 0x7f6280069e90 2023-10-30T13:57:31.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.157+0000 7f627cff9640 1 -- 172.21.15.46:0/2074580625 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f627000bc70 con 0x7f6280069e90 2023-10-30T13:57:31.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.161+0000 7f627cff9640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f626c0664a0 0x7f626c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:31.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.161+0000 7f627cff9640 1 -- 172.21.15.46:0/2074580625 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f6270084900 con 0x7f6280069e90 2023-10-30T13:57:31.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.161+0000 7f627f7fe640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f626c0664a0 0x7f626c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:31.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.161+0000 7f627f7fe640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f626c0664a0 0x7f626c068960 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6280144620 tx=0x7f6274007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:31.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.165+0000 7f627cff9640 1 -- 172.21.15.46:0/2074580625 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6270051740 con 0x7f6280069e90 2023-10-30T13:57:31.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.353+0000 7f62863c4640 1 -- 172.21.15.46:0/2074580625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62800c72f0 con 0x7f6280069e90 2023-10-30T13:57:31.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.353+0000 7f627cff9640 1 -- 172.21.15.46:0/2074580625 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f6270054d60 con 0x7f6280069e90 2023-10-30T13:57:31.358 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:31.359 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:28.365952+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:31.359 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 151 2023-10-30T13:57:31.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.357+0000 7f62863c4640 1 -- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f626c0664a0 msgr2=0x7f626c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:31.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.357+0000 7f62863c4640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f626c0664a0 0x7f626c068960 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6280144620 tx=0x7f6274007e40 comp rx=0 tx=0).stop 2023-10-30T13:57:31.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.357+0000 7f62863c4640 1 -- 172.21.15.46:0/2074580625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6280069e90 msgr2=0x7f62800c0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:31.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.357+0000 7f62863c4640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6280069e90 0x7f62800c0f90 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f627000d620 tx=0x7f627000dae0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.357+0000 7f62863c4640 1 -- 172.21.15.46:0/2074580625 shutdown_connections 2023-10-30T13:57:31.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.357+0000 7f62863c4640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f626c0664a0 0x7f626c068960 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.357+0000 7f62863c4640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62801539a0 0x7f6280142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.357+0000 7f62863c4640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6280070a00 0x7f6280149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.357+0000 7f62863c4640 1 --2- 172.21.15.46:0/2074580625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6280069e90 0x7f62800c0f90 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:31.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.357+0000 7f62863c4640 1 -- 172.21.15.46:0/2074580625 >> 172.21.15.46:0/2074580625 conn(0x7f62800c7910 msgr2=0x7f628014f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:31.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.357+0000 7f62863c4640 1 -- 172.21.15.46:0/2074580625 shutdown_connections 2023-10-30T13:57:31.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:31.357+0000 7f62863c4640 1 -- 172.21.15.46:0/2074580625 wait complete. 2023-10-30T13:57:31.381 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:57:33.382 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:33.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.545+0000 7fecf48d6640 1 Processor -- start 2023-10-30T13:57:33.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.545+0000 7fecf48d6640 1 -- start start 2023-10-30T13:57:33.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.545+0000 7fecf48d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fecf00c6f60 0x7fecf01566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.545+0000 7fecf48d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf00c0a50 con 0x7fecf00c05d0 2023-10-30T13:57:33.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.545+0000 7fecf48d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf006f6e0 con 0x7fecf00c6b80 2023-10-30T13:57:33.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.545+0000 7fecf48d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf006f9c0 con 0x7fecf00c6f60 2023-10-30T13:57:33.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.545+0000 7fecedd9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fecf00c05d0 0x7fecf013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60048/0 (socket says 172.21.15.46:60048) 2023-10-30T13:57:33.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.545+0000 7fecedd9b640 1 -- 172.21.15.46:0/953795889 learned_addr learned my addr 172.21.15.46:0/953795889 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:57:33.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.545+0000 7feceed9d640 1 --2- 172.21.15.46:0/953795889 >> v2:172.21.15.154:3300/0 conn(0x7fecf00c6f60 0x7fecf01566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.545+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274528228 0 0) 0x7fecf00c0a50 con 0x7fecf00c05d0 2023-10-30T13:57:33.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd8003530 con 0x7fecf00c05d0 2023-10-30T13:57:33.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4026415727 0 0) 0x7fecd8003530 con 0x7fecf00c05d0 2023-10-30T13:57:33.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 >> v2:172.21.15.154:3300/0 conn(0x7fecf00c6f60 msgr2=0x7fecf01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:33.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 --2- 172.21.15.46:0/953795889 >> v2:172.21.15.154:3300/0 conn(0x7fecf00c6f60 0x7fecf01566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 >> v1:172.21.15.154:6789/0 conn(0x7fecf00c6b80 legacy=0x7fecf00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:33.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf006fca0 con 0x7fecf00c05d0 2023-10-30T13:57:33.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fece4001680 con 0x7fecf00c05d0 2023-10-30T13:57:33.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 >> v1:172.21.15.46:6789/0 conn(0x7fecf00c05d0 legacy=0x7fecf013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:33.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 --2- 172.21.15.46:0/953795889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd8003980 0x7fecd8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 --2- 172.21.15.46:0/953795889 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecd80047d0 0x7fecd8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 --2- 172.21.15.46:0/953795889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecd8005190 0x7fecd800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf006f9c0 con 0x7fecd8003980 2023-10-30T13:57:33.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd800c030 con 0x7fecd8005190 2023-10-30T13:57:33.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd800c1f0 con 0x7fecd80047d0 2023-10-30T13:57:33.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7fecee59c640 1 --2- 172.21.15.46:0/953795889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd8003980 0x7fecd8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feceed9d640 1 --2- 172.21.15.46:0/953795889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecd8005190 0x7fecd800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7fecedd9b640 1 --2- 172.21.15.46:0/953795889 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecd80047d0 0x7fecd8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7fecee59c640 1 -- 172.21.15.46:0/953795889 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecd80047d0 msgr2=0x7fecd8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:33.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7fecee59c640 1 --2- 172.21.15.46:0/953795889 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecd80047d0 0x7fecd8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7fecee59c640 1 -- 172.21.15.46:0/953795889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecd8005190 msgr2=0x7fecd800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:33.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7fecee59c640 1 --2- 172.21.15.46:0/953795889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecd8005190 0x7fecd800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7fecee59c640 1 -- 172.21.15.46:0/953795889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd8003530 con 0x7fecd8003980 2023-10-30T13:57:33.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7feceed9d640 1 -- 172.21.15.46:0/953795889 reap_dead start 2023-10-30T13:57:33.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.549+0000 7fecee59c640 1 --2- 172.21.15.46:0/953795889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd8003980 0x7fecd8003da0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fece0001360 tx=0x7fece001e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60fcb013d7268bdd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:33.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece00a9070 con 0x7fecd8003980 2023-10-30T13:57:33.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece00050d0 con 0x7fecd8003980 2023-10-30T13:57:33.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7feced59a640 1 -- 172.21.15.46:0/953795889 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece00054a0 con 0x7fecd8003980 2023-10-30T13:57:33.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7fecf48d6640 1 -- 172.21.15.46:0/953795889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd8003980 msgr2=0x7fecd8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:33.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7fecf48d6640 1 --2- 172.21.15.46:0/953795889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd8003980 0x7fecd8003da0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fece0001360 tx=0x7fece001e5f0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7fecf48d6640 1 -- 172.21.15.46:0/953795889 shutdown_connections 2023-10-30T13:57:33.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7fecf48d6640 1 --2- 172.21.15.46:0/953795889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd8003980 0x7fecd8003da0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7fecf48d6640 1 -- 172.21.15.46:0/953795889 >> 172.21.15.46:0/953795889 conn(0x7fecf00c7910 msgr2=0x7fecf00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:33.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7fecf48d6640 1 -- 172.21.15.46:0/953795889 shutdown_connections 2023-10-30T13:57:33.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7fecf48d6640 1 -- 172.21.15.46:0/953795889 wait complete. 2023-10-30T13:57:33.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7fecf48d6640 1 Processor -- start 2023-10-30T13:57:33.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7fecf48d6640 1 -- start start 2023-10-30T13:57:33.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7fecf48d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecd8003980 0x7fecf00ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.553+0000 7fecf48d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecf0159620 0x7fecf00aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecf48d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecf00aef50 0x7fecf0137b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecf48d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf00c9f70 con 0x7fecf0159620 2023-10-30T13:57:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecf48d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf0158480 con 0x7fecd8003980 2023-10-30T13:57:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecf48d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf01f3790 con 0x7fecf00aef50 2023-10-30T13:57:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecedd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecf0159620 0x7fecf00aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecedd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecf0159620 0x7fecf00aea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52344/0 (socket says 172.21.15.46:52344) 2023-10-30T13:57:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecedd9b640 1 -- 172.21.15.46:0/2711610240 learned_addr learned my addr 172.21.15.46:0/2711610240 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7feceed9d640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecf00aef50 0x7fecf0137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecee59c640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecd8003980 0x7fecf00ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecedd9b640 1 -- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecf00aef50 msgr2=0x7fecf0137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecedd9b640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecf00aef50 0x7fecf0137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecedd9b640 1 -- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecd8003980 msgr2=0x7fecf00ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecedd9b640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecd8003980 0x7fecf00ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecedd9b640 1 -- 172.21.15.46:0/2711610240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf0158720 con 0x7fecf0159620 2023-10-30T13:57:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecee59c640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecd8003980 0x7fecf00ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecedd9b640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecf0159620 0x7fecf00aea10 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fece4000f70 tx=0x7fece400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecdeffd640 1 -- 172.21.15.46:0/2711610240 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece400c6c0 con 0x7fecf0159620 2023-10-30T13:57:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecdeffd640 1 -- 172.21.15.46:0/2711610240 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece400c840 con 0x7fecf0159620 2023-10-30T13:57:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecdeffd640 1 -- 172.21.15.46:0/2711610240 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece400cbf0 con 0x7fecf0159620 2023-10-30T13:57:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecf48d6640 1 -- 172.21.15.46:0/2711610240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf00ad970 con 0x7fecf0159620 2023-10-30T13:57:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecf48d6640 1 -- 172.21.15.46:0/2711610240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf00adf30 con 0x7fecf0159620 2023-10-30T13:57:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecf48d6640 1 -- 172.21.15.46:0/2711610240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf004e170 con 0x7fecf0159620 2023-10-30T13:57:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.557+0000 7fecdeffd640 1 -- 172.21.15.46:0/2711610240 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fece400cd50 con 0x7fecf0159620 2023-10-30T13:57:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.561+0000 7fecdeffd640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecd00664a0 0x7fecd0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.561+0000 7fecdeffd640 1 -- 172.21.15.46:0/2711610240 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fece40843f0 con 0x7fecf0159620 2023-10-30T13:57:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.561+0000 7fecee59c640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecd00664a0 0x7fecd0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:33.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.561+0000 7fecee59c640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecd00664a0 0x7fecd0068960 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fece0001750 tx=0x7fece0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:33.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.561+0000 7fecdeffd640 1 -- 172.21.15.46:0/2711610240 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fece4051210 con 0x7fecf0159620 2023-10-30T13:57:33.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.773+0000 7fecf48d6640 1 -- 172.21.15.46:0/2711610240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecf00c7d90 con 0x7fecf0159620 2023-10-30T13:57:33.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.773+0000 7fecdeffd640 1 -- 172.21.15.46:0/2711610240 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fece4054830 con 0x7fecf0159620 2023-10-30T13:57:33.777 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:33.777 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:28.365952+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:33.778 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 151 2023-10-30T13:57:33.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.773+0000 7fecf48d6640 1 -- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecd00664a0 msgr2=0x7fecd0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:33.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.773+0000 7fecf48d6640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecd00664a0 0x7fecd0068960 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fece0001750 tx=0x7fece0009040 comp rx=0 tx=0).stop 2023-10-30T13:57:33.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.773+0000 7fecf48d6640 1 -- 172.21.15.46:0/2711610240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecf0159620 msgr2=0x7fecf00aea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.773+0000 7fecf48d6640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecf0159620 0x7fecf00aea10 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fece4000f70 tx=0x7fece400eba0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.773+0000 7fecf48d6640 1 -- 172.21.15.46:0/2711610240 shutdown_connections 2023-10-30T13:57:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.773+0000 7fecf48d6640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fecd00664a0 0x7fecd0068960 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.773+0000 7fecf48d6640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fecf00aef50 0x7fecf0137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.773+0000 7fecf48d6640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecf0159620 0x7fecf00aea10 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.773+0000 7fecf48d6640 1 --2- 172.21.15.46:0/2711610240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fecd8003980 0x7fecf00ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.773+0000 7fecf48d6640 1 -- 172.21.15.46:0/2711610240 >> 172.21.15.46:0/2711610240 conn(0x7fecf00c7910 msgr2=0x7fecf00c60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.777+0000 7fecf48d6640 1 -- 172.21.15.46:0/2711610240 shutdown_connections 2023-10-30T13:57:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:33.777+0000 7fecf48d6640 1 -- 172.21.15.46:0/2711610240 wait complete. 2023-10-30T13:57:33.798 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:57:35.799 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:35.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b5d365640 1 Processor -- start 2023-10-30T13:57:35.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b5d365640 1 -- start start 2023-10-30T13:57:35.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b5d365640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b581538a0 0x7f5b5815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:35.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b5d365640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5b580cf160 0x7f5b580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:35.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b5d365640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b580cb800 con 0x7f5b580ca0e0 2023-10-30T13:57:35.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b5d365640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b5815dc30 con 0x7f5b580cf160 2023-10-30T13:57:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b5d365640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b580cb330 con 0x7f5b581538a0 2023-10-30T13:57:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b577fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5b580cf160 0x7f5b580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b577fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5b580cf160 0x7f5b580cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52360/0 (socket says 172.21.15.46:52360) 2023-10-30T13:57:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b577fe640 1 -- 172.21.15.46:0/1845132237 learned_addr learned my addr 172.21.15.46:0/1845132237 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b56ffd640 1 --2- 172.21.15.46:0/1845132237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b581538a0 0x7f5b5815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b577fe640 1 -- 172.21.15.46:0/1845132237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b581538a0 msgr2=0x7f5b5815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b577fe640 1 --2- 172.21.15.46:0/1845132237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b581538a0 0x7f5b5815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b577fe640 1 -- 172.21.15.46:0/1845132237 >> v1:172.21.15.154:6789/0 conn(0x7f5b580ca0e0 legacy=0x7f5b5815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.949+0000 7f5b577fe640 1 -- 172.21.15.46:0/1845132237 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b580cad70 con 0x7f5b580cf160 2023-10-30T13:57:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.953+0000 7f5b577fe640 1 --2- 172.21.15.46:0/1845132237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b580cf160 0x7f5b580cf590 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009e50 tx=0x7f5b5001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276229b99dbaa7d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.953+0000 7f5b567fc640 1 -- 172.21.15.46:0/1845132237 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b500af400 con 0x7f5b580cf160 2023-10-30T13:57:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.953+0000 7f5b567fc640 1 -- 172.21.15.46:0/1845132237 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b500ab050 con 0x7f5b580cf160 2023-10-30T13:57:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.953+0000 7f5b567fc640 1 -- 172.21.15.46:0/1845132237 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b500af780 con 0x7f5b580cf160 2023-10-30T13:57:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.953+0000 7f5b5d365640 1 -- 172.21.15.46:0/1845132237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b580cf160 msgr2=0x7f5b580cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.953+0000 7f5b5d365640 1 --2- 172.21.15.46:0/1845132237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b580cf160 0x7f5b580cf590 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009e50 tx=0x7f5b5001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:57:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.953+0000 7f5b5d365640 1 -- 172.21.15.46:0/1845132237 shutdown_connections 2023-10-30T13:57:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.953+0000 7f5b5d365640 1 --2- 172.21.15.46:0/1845132237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b580cf160 0x7f5b580cf590 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.953+0000 7f5b5d365640 1 --2- 172.21.15.46:0/1845132237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b581538a0 0x7f5b5815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.953+0000 7f5b5d365640 1 -- 172.21.15.46:0/1845132237 >> 172.21.15.46:0/1845132237 conn(0x7f5b5806bfe0 msgr2=0x7f5b5806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.953+0000 7f5b5d365640 1 -- 172.21.15.46:0/1845132237 shutdown_connections 2023-10-30T13:57:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.953+0000 7f5b5d365640 1 -- 172.21.15.46:0/1845132237 wait complete. 2023-10-30T13:57:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b5d365640 1 Processor -- start 2023-10-30T13:57:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b5d365640 1 -- start start 2023-10-30T13:57:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b5d365640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b580ca0e0 0x7f5b5815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b5d365640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b580cf160 0x7f5b5815b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b5d365640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b581538a0 0x7f5b5815b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b5d365640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581e3960 con 0x7f5b580ca0e0 2023-10-30T13:57:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b5d365640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581e4750 con 0x7f5b580cf160 2023-10-30T13:57:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b5d365640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581e3ba0 con 0x7f5b581538a0 2023-10-30T13:57:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b56ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b580ca0e0 0x7f5b5815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b56ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b580ca0e0 0x7f5b5815a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52364/0 (socket says 172.21.15.46:52364) 2023-10-30T13:57:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b56ffd640 1 -- 172.21.15.46:0/513148771 learned_addr learned my addr 172.21.15.46:0/513148771 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b577fe640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b581538a0 0x7f5b5815b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b4e7fc640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b580cf160 0x7f5b5815b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b56ffd640 1 -- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b581538a0 msgr2=0x7f5b5815b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b56ffd640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b581538a0 0x7f5b5815b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b56ffd640 1 -- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b580cf160 msgr2=0x7f5b5815b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b56ffd640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b580cf160 0x7f5b5815b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b56ffd640 1 -- 172.21.15.46:0/513148771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b50009b30 con 0x7f5b580ca0e0 2023-10-30T13:57:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b4e7fc640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b580cf160 0x7f5b5815b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b56ffd640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b580ca0e0 0x7f5b5815a8b0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000e870 tx=0x7f5b4000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b4ffff640 1 -- 172.21.15.46:0/513148771 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b40009510 con 0x7f5b580ca0e0 2023-10-30T13:57:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b4ffff640 1 -- 172.21.15.46:0/513148771 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4000eec0 con 0x7f5b580ca0e0 2023-10-30T13:57:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b5d365640 1 -- 172.21.15.46:0/513148771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b581e3430 con 0x7f5b580ca0e0 2023-10-30T13:57:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b4ffff640 1 -- 172.21.15.46:0/513148771 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4000cb40 con 0x7f5b580ca0e0 2023-10-30T13:57:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.957+0000 7f5b5d365640 1 -- 172.21.15.46:0/513148771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b581e3db0 con 0x7f5b580ca0e0 2023-10-30T13:57:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.961+0000 7f5b5d365640 1 -- 172.21.15.46:0/513148771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b581e49c0 con 0x7f5b580ca0e0 2023-10-30T13:57:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.961+0000 7f5b4ffff640 1 -- 172.21.15.46:0/513148771 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5b4000cca0 con 0x7f5b580ca0e0 2023-10-30T13:57:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.961+0000 7f5b4ffff640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5b380664a0 0x7f5b38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.961+0000 7f5b4ffff640 1 -- 172.21.15.46:0/513148771 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f5b40084640 con 0x7f5b580ca0e0 2023-10-30T13:57:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.961+0000 7f5b4e7fc640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5b380664a0 0x7f5b38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.961+0000 7f5b4e7fc640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5b380664a0 0x7f5b38068960 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5b5815a130 tx=0x7f5b48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:35.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:35.965+0000 7f5b4ffff640 1 -- 172.21.15.46:0/513148771 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5b40051480 con 0x7f5b580ca0e0 2023-10-30T13:57:36.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.157+0000 7f5b5d365640 1 -- 172.21.15.46:0/513148771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b580b2a50 con 0x7f5b580ca0e0 2023-10-30T13:57:36.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.157+0000 7f5b4ffff640 1 -- 172.21.15.46:0/513148771 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f5b40054aa0 con 0x7f5b580ca0e0 2023-10-30T13:57:36.162 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:36.162 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:28.365952+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:36.162 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 151 2023-10-30T13:57:36.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.161+0000 7f5b5d365640 1 -- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5b380664a0 msgr2=0x7f5b38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:36.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.161+0000 7f5b5d365640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5b380664a0 0x7f5b38068960 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5b5815a130 tx=0x7f5b48009040 comp rx=0 tx=0).stop 2023-10-30T13:57:36.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.161+0000 7f5b5d365640 1 -- 172.21.15.46:0/513148771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b580ca0e0 msgr2=0x7f5b5815a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:36.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.161+0000 7f5b5d365640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b580ca0e0 0x7f5b5815a8b0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000e870 tx=0x7f5b4000ec70 comp rx=0 tx=0).stop 2023-10-30T13:57:36.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.161+0000 7f5b5d365640 1 -- 172.21.15.46:0/513148771 shutdown_connections 2023-10-30T13:57:36.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.161+0000 7f5b5d365640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5b380664a0 0x7f5b38068960 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:36.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.161+0000 7f5b5d365640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b581538a0 0x7f5b5815b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:36.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.161+0000 7f5b5d365640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b580cf160 0x7f5b5815b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:36.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.161+0000 7f5b5d365640 1 --2- 172.21.15.46:0/513148771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b580ca0e0 0x7f5b5815a8b0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:36.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.161+0000 7f5b5d365640 1 -- 172.21.15.46:0/513148771 >> 172.21.15.46:0/513148771 conn(0x7f5b5806bfe0 msgr2=0x7f5b580c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:36.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.161+0000 7f5b5d365640 1 -- 172.21.15.46:0/513148771 shutdown_connections 2023-10-30T13:57:36.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:36.161+0000 7f5b5d365640 1 -- 172.21.15.46:0/513148771 wait complete. 2023-10-30T13:57:36.183 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:57:38.184 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:38.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.337+0000 7fc9206d6640 1 Processor -- start 2023-10-30T13:57:38.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.337+0000 7fc9206d6640 1 -- start start 2023-10-30T13:57:38.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.337+0000 7fc9206d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9180c6f60 0x7fc9181566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:38.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.337+0000 7fc9206d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9180c0a50 con 0x7fc9180c6b80 2023-10-30T13:57:38.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.337+0000 7fc9206d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc91806f6e0 con 0x7fc9180c05d0 2023-10-30T13:57:38.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.337+0000 7fc9206d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc91806f9c0 con 0x7fc9180c6f60 2023-10-30T13:57:38.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.337+0000 7fc91e472640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc9180c6b80 0x7fc9180bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50680/0 (socket says 172.21.15.46:50680) 2023-10-30T13:57:38.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.337+0000 7fc91e472640 1 -- 172.21.15.46:0/293232871 learned_addr learned my addr 172.21.15.46:0/293232871 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:57:38.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.341+0000 7fc91ec73640 1 --2- 172.21.15.46:0/293232871 >> v2:172.21.15.154:3300/0 conn(0x7fc9180c6f60 0x7fc9181566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:38.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.341+0000 7fc91d470640 1 -- 172.21.15.46:0/293232871 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492567640 0 0) 0x7fc9180c0a50 con 0x7fc9180c6b80 2023-10-30T13:57:38.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.341+0000 7fc91d470640 1 -- 172.21.15.46:0/293232871 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc904003530 con 0x7fc9180c6b80 2023-10-30T13:57:38.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.341+0000 7fc91d470640 1 -- 172.21.15.46:0/293232871 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 55412445 0 0) 0x7fc904003530 con 0x7fc9180c6b80 2023-10-30T13:57:38.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.341+0000 7fc91ec73640 1 -- 172.21.15.46:0/293232871 >> v1:172.21.15.154:6789/0 conn(0x7fc9180c05d0 legacy=0x7fc91813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.341+0000 7fc91ec73640 1 -- 172.21.15.46:0/293232871 >> v1:172.21.15.46:6789/0 conn(0x7fc9180c6b80 legacy=0x7fc9180bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.341+0000 7fc91ec73640 1 -- 172.21.15.46:0/293232871 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91806fca0 con 0x7fc9180c6f60 2023-10-30T13:57:38.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.341+0000 7fc91ec73640 1 --2- 172.21.15.46:0/293232871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9180c6f60 0x7fc9181566a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc910009d90 tx=0x7fc91001c720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd5d904f9de7768f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:38.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.341+0000 7fc91d470640 1 -- 172.21.15.46:0/293232871 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914001680 con 0x7fc9180c6f60 2023-10-30T13:57:38.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc91d470640 1 -- 172.21.15.46:0/293232871 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9100a64c0 con 0x7fc9180c6f60 2023-10-30T13:57:38.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc91d470640 1 -- 172.21.15.46:0/293232871 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9100a6850 con 0x7fc9180c6f60 2023-10-30T13:57:38.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 -- 172.21.15.46:0/293232871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9180c6f60 msgr2=0x7fc9181566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 --2- 172.21.15.46:0/293232871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9180c6f60 0x7fc9181566a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc910009d90 tx=0x7fc91001c720 comp rx=0 tx=0).stop 2023-10-30T13:57:38.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 -- 172.21.15.46:0/293232871 shutdown_connections 2023-10-30T13:57:38.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 --2- 172.21.15.46:0/293232871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9180c6f60 0x7fc9181566a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 -- 172.21.15.46:0/293232871 >> 172.21.15.46:0/293232871 conn(0x7fc9180c7910 msgr2=0x7fc9180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:38.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 -- 172.21.15.46:0/293232871 shutdown_connections 2023-10-30T13:57:38.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 -- 172.21.15.46:0/293232871 wait complete. 2023-10-30T13:57:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 Processor -- start 2023-10-30T13:57:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 -- start start 2023-10-30T13:57:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9180c05d0 0x7fc91813e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9180c6b80 0x7fc9181553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:38.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9180c6f60 0x7fc918155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:38.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc918154f50 con 0x7fc9180c6f60 2023-10-30T13:57:38.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9180c9f00 con 0x7fc9180c05d0 2023-10-30T13:57:38.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc9206d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9180c9960 con 0x7fc9180c6b80 2023-10-30T13:57:38.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc91ec73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9180c6f60 0x7fc918155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:38.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc91ec73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9180c6f60 0x7fc918155920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35774/0 (socket says 172.21.15.46:35774) 2023-10-30T13:57:38.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc91ec73640 1 -- 172.21.15.46:0/2560021718 learned_addr learned my addr 172.21.15.46:0/2560021718 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:38.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc91e472640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9180c05d0 0x7fc91813e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:38.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.345+0000 7fc91dc71640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9180c6b80 0x7fc9181553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:38.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc91ec73640 1 -- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9180c6b80 msgr2=0x7fc9181553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc91ec73640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9180c6b80 0x7fc9181553e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc91ec73640 1 -- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9180c05d0 msgr2=0x7fc91813e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc91ec73640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9180c05d0 0x7fc91813e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc91ec73640 1 -- 172.21.15.46:0/2560021718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc910009a70 con 0x7fc9180c6f60 2023-10-30T13:57:38.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc91ec73640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9180c6f60 0x7fc918155920 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fc910009d90 tx=0x7fc9100194c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:38.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc90effd640 1 -- 172.21.15.46:0/2560021718 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910003ba0 con 0x7fc9180c6f60 2023-10-30T13:57:38.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc90effd640 1 -- 172.21.15.46:0/2560021718 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc910003d00 con 0x7fc9180c6f60 2023-10-30T13:57:38.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc90effd640 1 -- 172.21.15.46:0/2560021718 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910004270 con 0x7fc9180c6f60 2023-10-30T13:57:38.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc9206d6640 1 -- 172.21.15.46:0/2560021718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9180c9c20 con 0x7fc9180c6f60 2023-10-30T13:57:38.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc9206d6640 1 -- 172.21.15.46:0/2560021718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc918154be0 con 0x7fc9180c6f60 2023-10-30T13:57:38.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc9206d6640 1 -- 172.21.15.46:0/2560021718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc918155100 con 0x7fc9180c6f60 2023-10-30T13:57:38.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.349+0000 7fc90effd640 1 -- 172.21.15.46:0/2560021718 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc9100043d0 con 0x7fc9180c6f60 2023-10-30T13:57:38.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.353+0000 7fc90effd640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8fc0664d0 0x7fc8fc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:38.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.353+0000 7fc90effd640 1 -- 172.21.15.46:0/2560021718 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fc9100aa070 con 0x7fc9180c6f60 2023-10-30T13:57:38.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.353+0000 7fc91e472640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8fc0664d0 0x7fc8fc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:38.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.353+0000 7fc91e472640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8fc0664d0 0x7fc8fc068990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc9140017b0 tx=0x7fc914001240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:38.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.357+0000 7fc90effd640 1 -- 172.21.15.46:0/2560021718 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc918155100 con 0x7fc9180c6f60 2023-10-30T13:57:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 -- 172.21.15.46:0/2560021718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc918155100 con 0x7fc9180c6f60 2023-10-30T13:57:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc90effd640 1 -- 172.21.15.46:0/2560021718 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fc918155100 con 0x7fc9180c6f60 2023-10-30T13:57:38.616 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:38.616 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:28.365952+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 151 2023-10-30T13:57:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 -- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8fc0664d0 msgr2=0x7fc8fc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8fc0664d0 0x7fc8fc068990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc9140017b0 tx=0x7fc914001240 comp rx=0 tx=0).stop 2023-10-30T13:57:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 -- 172.21.15.46:0/2560021718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9180c6f60 msgr2=0x7fc918155920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9180c6f60 0x7fc918155920 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fc910009d90 tx=0x7fc9100194c0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 -- 172.21.15.46:0/2560021718 shutdown_connections 2023-10-30T13:57:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8fc0664d0 0x7fc8fc068990 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9180c6f60 0x7fc918155920 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9180c6b80 0x7fc9181553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 --2- 172.21.15.46:0/2560021718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9180c05d0 0x7fc91813e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 -- 172.21.15.46:0/2560021718 >> 172.21.15.46:0/2560021718 conn(0x7fc9180c7910 msgr2=0x7fc9180c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 -- 172.21.15.46:0/2560021718 shutdown_connections 2023-10-30T13:57:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:38.613+0000 7fc9206d6640 1 -- 172.21.15.46:0/2560021718 wait complete. 2023-10-30T13:57:38.637 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:57:40.638 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.789+0000 7f8071f9d640 1 Processor -- start 2023-10-30T13:57:40.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.789+0000 7f8071f9d640 1 -- start start 2023-10-30T13:57:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.789+0000 7f8071f9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f806c0715c0 0x7f806c070650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.789+0000 7f8071f9d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f806c0c1530 con 0x7f806c157010 2023-10-30T13:57:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.789+0000 7f8071f9d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f806c0c0f70 con 0x7f806c0c8460 2023-10-30T13:57:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.789+0000 7f8071f9d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806c0c21b0 con 0x7f806c0715c0 2023-10-30T13:57:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.789+0000 7f806b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f806c157010 0x7f806c071bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50696/0 (socket says 172.21.15.46:50696) 2023-10-30T13:57:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.789+0000 7f806b7fe640 1 -- 172.21.15.46:0/2029863236 learned_addr learned my addr 172.21.15.46:0/2029863236 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:57:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.793+0000 7f806bfff640 1 --2- 172.21.15.46:0/2029863236 >> v2:172.21.15.154:3300/0 conn(0x7f806c0715c0 0x7f806c070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.793+0000 7f806a7fc640 1 -- 172.21.15.46:0/2029863236 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953080623 0 0) 0x7f806c0c1530 con 0x7f806c157010 2023-10-30T13:57:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.793+0000 7f806a7fc640 1 -- 172.21.15.46:0/2029863236 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8054003530 con 0x7f806c157010 2023-10-30T13:57:40.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.793+0000 7f806bfff640 1 -- 172.21.15.46:0/2029863236 >> v1:172.21.15.154:6789/0 conn(0x7f806c0c8460 legacy=0x7f806c1620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:40.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.793+0000 7f806a7fc640 1 -- 172.21.15.46:0/2029863236 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3324486754 0 0) 0x7f8054003530 con 0x7f806c157010 2023-10-30T13:57:40.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.793+0000 7f806bfff640 1 -- 172.21.15.46:0/2029863236 >> v1:172.21.15.46:6789/0 conn(0x7f806c157010 legacy=0x7f806c071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:40.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.793+0000 7f806bfff640 1 -- 172.21.15.46:0/2029863236 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806c0c1250 con 0x7f806c0715c0 2023-10-30T13:57:40.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.793+0000 7f806bfff640 1 --2- 172.21.15.46:0/2029863236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0715c0 0x7f806c070650 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8064009d90 tx=0x7f806401c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc50bab3fee38682 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:40.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f806a7fc640 1 -- 172.21.15.46:0/2029863236 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c001680 con 0x7f806c0715c0 2023-10-30T13:57:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f806a7fc640 1 -- 172.21.15.46:0/2029863236 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80640a5680 con 0x7f806c0715c0 2023-10-30T13:57:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f806a7fc640 1 -- 172.21.15.46:0/2029863236 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80640a59f0 con 0x7f806c0715c0 2023-10-30T13:57:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 -- 172.21.15.46:0/2029863236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0715c0 msgr2=0x7f806c070650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:40.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 --2- 172.21.15.46:0/2029863236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0715c0 0x7f806c070650 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8064009d90 tx=0x7f806401c840 comp rx=0 tx=0).stop 2023-10-30T13:57:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 -- 172.21.15.46:0/2029863236 shutdown_connections 2023-10-30T13:57:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 --2- 172.21.15.46:0/2029863236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0715c0 0x7f806c070650 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 -- 172.21.15.46:0/2029863236 >> 172.21.15.46:0/2029863236 conn(0x7f806c0cbeb0 msgr2=0x7f806c0cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 -- 172.21.15.46:0/2029863236 shutdown_connections 2023-10-30T13:57:40.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 -- 172.21.15.46:0/2029863236 wait complete. 2023-10-30T13:57:40.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 Processor -- start 2023-10-30T13:57:40.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 -- start start 2023-10-30T13:57:40.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f806c0715c0 0x7f806c157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:40.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0c8460 0x7f806c158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:40.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f806c157010 0x7f806c14b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:40.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c0cbae0 con 0x7f806c157010 2023-10-30T13:57:40.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c157b90 con 0x7f806c0c8460 2023-10-30T13:57:40.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.797+0000 7f8071f9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806c158a80 con 0x7f806c0715c0 2023-10-30T13:57:40.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f806bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f806c157010 0x7f806c14b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:40.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f806bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f806c157010 0x7f806c14b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35776/0 (socket says 172.21.15.46:35776) 2023-10-30T13:57:40.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f806bfff640 1 -- 172.21.15.46:0/344446750 learned_addr learned my addr 172.21.15.46:0/344446750 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:40.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f806b7fe640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f806c0715c0 0x7f806c157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:40.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f806affd640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0c8460 0x7f806c158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:40.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f806bfff640 1 -- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f806c0715c0 msgr2=0x7f806c157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:40.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f806bfff640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f806c0715c0 0x7f806c157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:40.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f806bfff640 1 -- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0c8460 msgr2=0x7f806c158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:40.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f806bfff640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0c8460 0x7f806c158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:40.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f806bfff640 1 -- 172.21.15.46:0/344446750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8064009a70 con 0x7f806c157010 2023-10-30T13:57:40.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f806b7fe640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f806c0715c0 0x7f806c157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:40.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f806bfff640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f806c157010 0x7f806c14b440 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f806401a470 tx=0x7f80640195e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:40.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f8053fff640 1 -- 172.21.15.46:0/344446750 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80640a5a00 con 0x7f806c157010 2023-10-30T13:57:40.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f8053fff640 1 -- 172.21.15.46:0/344446750 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80640a5b60 con 0x7f806c157010 2023-10-30T13:57:40.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f8071f9d640 1 -- 172.21.15.46:0/344446750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806c1592c0 con 0x7f806c157010 2023-10-30T13:57:40.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f8053fff640 1 -- 172.21.15.46:0/344446750 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8064004040 con 0x7f806c157010 2023-10-30T13:57:40.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f8071f9d640 1 -- 172.21.15.46:0/344446750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806c0cafd0 con 0x7f806c157010 2023-10-30T13:57:40.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f8071f9d640 1 -- 172.21.15.46:0/344446750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806c0cbc20 con 0x7f806c157010 2023-10-30T13:57:40.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f8053fff640 1 -- 172.21.15.46:0/344446750 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f80640041a0 con 0x7f806c157010 2023-10-30T13:57:40.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.801+0000 7f8053fff640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f80580664a0 0x7f8058068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:40.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.805+0000 7f8053fff640 1 -- 172.21.15.46:0/344446750 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f806411f240 con 0x7f806c157010 2023-10-30T13:57:40.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.805+0000 7f806b7fe640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f80580664a0 0x7f8058068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:40.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.805+0000 7f806b7fe640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f80580664a0 0x7f8058068960 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f805c005890 tx=0x7f805c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:40.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.805+0000 7f8053fff640 1 -- 172.21.15.46:0/344446750 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f806c0cbc20 con 0x7f806c157010 2023-10-30T13:57:41.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.997+0000 7f8071f9d640 1 -- 172.21.15.46:0/344446750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f806c0cbc20 con 0x7f806c157010 2023-10-30T13:57:41.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:40.997+0000 7f8053fff640 1 -- 172.21.15.46:0/344446750 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f806c0cbc20 con 0x7f806c157010 2023-10-30T13:57:41.003 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:41.003 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:28.365952+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:41.003 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 151 2023-10-30T13:57:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:41.001+0000 7f8071f9d640 1 -- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f80580664a0 msgr2=0x7f8058068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:41.001+0000 7f8071f9d640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f80580664a0 0x7f8058068960 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f805c005890 tx=0x7f805c009040 comp rx=0 tx=0).stop 2023-10-30T13:57:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:41.001+0000 7f8071f9d640 1 -- 172.21.15.46:0/344446750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f806c157010 msgr2=0x7f806c14b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:41.001+0000 7f8071f9d640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f806c157010 0x7f806c14b440 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f806401a470 tx=0x7f80640195e0 comp rx=0 tx=0).stop 2023-10-30T13:57:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:41.001+0000 7f8071f9d640 1 -- 172.21.15.46:0/344446750 shutdown_connections 2023-10-30T13:57:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:41.001+0000 7f8071f9d640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f80580664a0 0x7f8058068960 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:41.001+0000 7f8071f9d640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f806c157010 0x7f806c14b440 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:41.001+0000 7f8071f9d640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0c8460 0x7f806c158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:41.001+0000 7f8071f9d640 1 --2- 172.21.15.46:0/344446750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f806c0715c0 0x7f806c157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:41.001+0000 7f8071f9d640 1 -- 172.21.15.46:0/344446750 >> 172.21.15.46:0/344446750 conn(0x7f806c0cbeb0 msgr2=0x7f806c152570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:41.001+0000 7f8071f9d640 1 -- 172.21.15.46:0/344446750 shutdown_connections 2023-10-30T13:57:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:41.001+0000 7f8071f9d640 1 -- 172.21.15.46:0/344446750 wait complete. 2023-10-30T13:57:41.024 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:57:43.025 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.177+0000 7f3ff12ad640 1 Processor -- start 2023-10-30T13:57:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.177+0000 7f3ff12ad640 1 -- start start 2023-10-30T13:57:43.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3ff12ad640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fec071b90 0x7f3fec15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3ff12ad640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3fec0c6290 0x7f3fec161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3ff12ad640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fec15b060 con 0x7f3fec0717b0 2023-10-30T13:57:43.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3ff12ad640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fec157530 con 0x7f3fec071b90 2023-10-30T13:57:43.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3ff12ad640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fec0c4ab0 con 0x7f3fec0c6290 2023-10-30T13:57:43.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3feb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fec071b90 0x7f3fec15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3febfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3fec0717b0 0x7f3fec1513c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50698/0 (socket says 172.21.15.46:50698) 2023-10-30T13:57:43.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3febfff640 1 -- 172.21.15.46:0/2452502105 learned_addr learned my addr 172.21.15.46:0/2452502105 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:57:43.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3ff0aac640 1 --2- 172.21.15.46:0/2452502105 >> v2:172.21.15.154:3300/0 conn(0x7f3fec0c6290 0x7f3fec161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3feb7fe640 1 -- 172.21.15.46:0/2452502105 >> v2:172.21.15.154:3300/0 conn(0x7f3fec0c6290 msgr2=0x7f3fec161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:43.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3feb7fe640 1 --2- 172.21.15.46:0/2452502105 >> v2:172.21.15.154:3300/0 conn(0x7f3fec0c6290 0x7f3fec161eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3feb7fe640 1 -- 172.21.15.46:0/2452502105 >> v1:172.21.15.46:6789/0 conn(0x7f3fec0717b0 legacy=0x7f3fec1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3feb7fe640 1 -- 172.21.15.46:0/2452502105 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec0c4d90 con 0x7f3fec071b90 2023-10-30T13:57:43.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3ff0aac640 1 --2- 172.21.15.46:0/2452502105 >> v2:172.21.15.154:3300/0 conn(0x7f3fec0c6290 0x7f3fec161eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:57:43.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3feaffd640 1 -- 172.21.15.46:0/2452502105 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289119757 0 0) 0x7f3fec15b060 con 0x7f3fec0717b0 2023-10-30T13:57:43.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3feb7fe640 1 --2- 172.21.15.46:0/2452502105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec071b90 0x7f3fec15ab20 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f3fec06c3d0 tx=0x7f3fe001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1b79616d105905 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:43.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.181+0000 7f3feaffd640 1 -- 172.21.15.46:0/2452502105 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe00a8950 con 0x7f3fec071b90 2023-10-30T13:57:43.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3feaffd640 1 -- 172.21.15.46:0/2452502105 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe00a8ab0 con 0x7f3fec071b90 2023-10-30T13:57:43.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3feaffd640 1 -- 172.21.15.46:0/2452502105 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe00a8e40 con 0x7f3fec071b90 2023-10-30T13:57:43.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2452502105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec071b90 msgr2=0x7f3fec15ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 --2- 172.21.15.46:0/2452502105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec071b90 0x7f3fec15ab20 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f3fec06c3d0 tx=0x7f3fe001ec50 comp rx=0 tx=0).stop 2023-10-30T13:57:43.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2452502105 shutdown_connections 2023-10-30T13:57:43.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 --2- 172.21.15.46:0/2452502105 >> v2:172.21.15.154:3300/0 conn(0x7f3fec0c6290 0x7f3fec161eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 --2- 172.21.15.46:0/2452502105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec071b90 0x7f3fec15ab20 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2452502105 >> 172.21.15.46:0/2452502105 conn(0x7f3fec0c75d0 msgr2=0x7f3fec0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:43.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2452502105 shutdown_connections 2023-10-30T13:57:43.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2452502105 wait complete. 2023-10-30T13:57:43.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 Processor -- start 2023-10-30T13:57:43.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 -- start start 2023-10-30T13:57:43.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec0717b0 0x7f3fec14de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec071b90 0x7f3fec14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3fec0c6290 0x7f3fec14f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec1522b0 con 0x7f3fec0717b0 2023-10-30T13:57:43.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec154b00 con 0x7f3fec071b90 2023-10-30T13:57:43.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff12ad640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec154de0 con 0x7f3fec0c6290 2023-10-30T13:57:43.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3febfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec0717b0 0x7f3fec14de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3febfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec0717b0 0x7f3fec14de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35788/0 (socket says 172.21.15.46:35788) 2023-10-30T13:57:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3febfff640 1 -- 172.21.15.46:0/2454223894 learned_addr learned my addr 172.21.15.46:0/2454223894 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.185+0000 7f3ff0aac640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3fec0c6290 0x7f3fec14f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3feb7fe640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec071b90 0x7f3fec14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3febfff640 1 -- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3fec0c6290 msgr2=0x7f3fec14f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3febfff640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3fec0c6290 0x7f3fec14f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3febfff640 1 -- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec071b90 msgr2=0x7f3fec14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3febfff640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec071b90 0x7f3fec14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3febfff640 1 -- 172.21.15.46:0/2454223894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe0009420 con 0x7f3fec0717b0 2023-10-30T13:57:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3feb7fe640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec071b90 0x7f3fec14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3febfff640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec0717b0 0x7f3fec14de80 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00c430 tx=0x7f3fdc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3fe8ff9640 1 -- 172.21.15.46:0/2454223894 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc0074e0 con 0x7f3fec0717b0 2023-10-30T13:57:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3fe8ff9640 1 -- 172.21.15.46:0/2454223894 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fdc007640 con 0x7f3fec0717b0 2023-10-30T13:57:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3fe8ff9640 1 -- 172.21.15.46:0/2454223894 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc0059d0 con 0x7f3fec0717b0 2023-10-30T13:57:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2454223894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fec1550c0 con 0x7f3fec0717b0 2023-10-30T13:57:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2454223894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fec1f3e80 con 0x7f3fec0717b0 2023-10-30T13:57:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2454223894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fec04e170 con 0x7f3fec0717b0 2023-10-30T13:57:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.189+0000 7f3fe8ff9640 1 -- 172.21.15.46:0/2454223894 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3fdc005b30 con 0x7f3fec0717b0 2023-10-30T13:57:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.193+0000 7f3fe8ff9640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3fd80664a0 0x7f3fd8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.193+0000 7f3fe8ff9640 1 -- 172.21.15.46:0/2454223894 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f3fdc083a60 con 0x7f3fec0717b0 2023-10-30T13:57:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.193+0000 7f3feb7fe640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3fd80664a0 0x7f3fd8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.193+0000 7f3feb7fe640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3fd80664a0 0x7f3fd8068960 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00b2000 tx=0x7f3fe0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.193+0000 7f3fe8ff9640 1 -- 172.21.15.46:0/2454223894 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3fdc050a00 con 0x7f3fec0717b0 2023-10-30T13:57:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.389+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2454223894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fec0c3410 con 0x7f3fec0717b0 2023-10-30T13:57:43.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.389+0000 7f3fe8ff9640 1 -- 172.21.15.46:0/2454223894 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f3fdc017090 con 0x7f3fec0717b0 2023-10-30T13:57:43.394 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:43.394 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":7704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:56:28.365952+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7356,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7356":{"gid":7356,"name":"e","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.46:6837/3112266825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3112266825},{"type":"v1","addr":"172.21.15.46:6837","nonce":3112266825}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:43.395 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 151 2023-10-30T13:57:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.393+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3fd80664a0 msgr2=0x7f3fd8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.393+0000 7f3ff12ad640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3fd80664a0 0x7f3fd8068960 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00b2000 tx=0x7f3fe0008040 comp rx=0 tx=0).stop 2023-10-30T13:57:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.393+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2454223894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec0717b0 msgr2=0x7f3fec14de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.393+0000 7f3ff12ad640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec0717b0 0x7f3fec14de80 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00c430 tx=0x7f3fdc00c8f0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.393+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2454223894 shutdown_connections 2023-10-30T13:57:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.393+0000 7f3ff12ad640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3fd80664a0 0x7f3fd8068960 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.393+0000 7f3ff12ad640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3fec0c6290 0x7f3fec14f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.393+0000 7f3ff12ad640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec071b90 0x7f3fec14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.393+0000 7f3ff12ad640 1 --2- 172.21.15.46:0/2454223894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec0717b0 0x7f3fec14de80 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:43.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.393+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2454223894 >> 172.21.15.46:0/2454223894 conn(0x7f3fec0c75d0 msgr2=0x7f3fec06cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:43.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.393+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2454223894 shutdown_connections 2023-10-30T13:57:43.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:43.393+0000 7f3ff12ad640 1 -- 172.21.15.46:0/2454223894 wait complete. 2023-10-30T13:57:43.415 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:57:45.417 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d0eff8640 1 Processor -- start 2023-10-30T13:57:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d0eff8640 1 -- start start 2023-10-30T13:57:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d0eff8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6d0815d6f0 0x7f6d0815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d0eff8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d080cb800 con 0x7f6d081538a0 2023-10-30T13:57:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d0eff8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d080cb330 con 0x7f6d080cf160 2023-10-30T13:57:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d0eff8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d080cad70 con 0x7f6d0815d6f0 2023-10-30T13:57:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d0cd94640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6d081538a0 0x7f6d080ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50706/0 (socket says 172.21.15.46:50706) 2023-10-30T13:57:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d0cd94640 1 -- 172.21.15.46:0/2130145364 learned_addr learned my addr 172.21.15.46:0/2130145364 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:57:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d07fff640 1 --2- 172.21.15.46:0/2130145364 >> v2:172.21.15.154:3300/0 conn(0x7f6d0815d6f0 0x7f6d0815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354039596 0 0) 0x7f6d080cb800 con 0x7f6d081538a0 2023-10-30T13:57:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cf8003530 con 0x7f6d081538a0 2023-10-30T13:57:45.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2706478596 0 0) 0x7f6cf8003530 con 0x7f6d081538a0 2023-10-30T13:57:45.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 >> v2:172.21.15.154:3300/0 conn(0x7f6d0815d6f0 msgr2=0x7f6d0815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:45.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d077fe640 1 --2- 172.21.15.46:0/2130145364 >> v2:172.21.15.154:3300/0 conn(0x7f6d0815d6f0 0x7f6d0815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 >> v1:172.21.15.154:6789/0 conn(0x7f6d080cf160 legacy=0x7f6d08164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:45.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d080cb050 con 0x7f6d081538a0 2023-10-30T13:57:45.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f6cfc001680 con 0x7f6d081538a0 2023-10-30T13:57:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.569+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 >> v1:172.21.15.46:6789/0 conn(0x7f6d081538a0 legacy=0x7f6d080ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d077fe640 1 --2- 172.21.15.46:0/2130145364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf8003980 0x7f6cf8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d077fe640 1 --2- 172.21.15.46:0/2130145364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80047d0 0x7f6cf8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d077fe640 1 --2- 172.21.15.46:0/2130145364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf8005190 0x7f6cf800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080cad70 con 0x7f6cf80047d0 2023-10-30T13:57:45.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf800c040 con 0x7f6cf8003980 2023-10-30T13:57:45.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf800c200 con 0x7f6cf8005190 2023-10-30T13:57:45.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d07fff640 1 --2- 172.21.15.46:0/2130145364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80047d0 0x7f6cf8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d0cd94640 1 --2- 172.21.15.46:0/2130145364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf8003980 0x7f6cf8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d0d595640 1 --2- 172.21.15.46:0/2130145364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf8005190 0x7f6cf800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d07fff640 1 -- 172.21.15.46:0/2130145364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf8005190 msgr2=0x7f6cf800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d07fff640 1 --2- 172.21.15.46:0/2130145364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf8005190 0x7f6cf800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d07fff640 1 -- 172.21.15.46:0/2130145364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf8003980 msgr2=0x7f6cf8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d07fff640 1 --2- 172.21.15.46:0/2130145364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf8003980 0x7f6cf8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d07fff640 1 -- 172.21.15.46:0/2130145364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf8003530 con 0x7f6cf80047d0 2023-10-30T13:57:45.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d0d595640 1 -- 172.21.15.46:0/2130145364 reap_dead start 2023-10-30T13:57:45.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d07fff640 1 --2- 172.21.15.46:0/2130145364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80047d0 0x7f6cf8004c50 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009ae0 tx=0x7f6cf401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee39c95d6ec2d406 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40a5660 con 0x7f6cf80047d0 2023-10-30T13:57:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf40a57c0 con 0x7f6cf80047d0 2023-10-30T13:57:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.573+0000 7f6d077fe640 1 -- 172.21.15.46:0/2130145364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40a5b70 con 0x7f6cf80047d0 2023-10-30T13:57:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 -- 172.21.15.46:0/2130145364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80047d0 msgr2=0x7f6cf8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 --2- 172.21.15.46:0/2130145364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80047d0 0x7f6cf8004c50 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009ae0 tx=0x7f6cf401ca70 comp rx=0 tx=0).stop 2023-10-30T13:57:45.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 -- 172.21.15.46:0/2130145364 shutdown_connections 2023-10-30T13:57:45.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 --2- 172.21.15.46:0/2130145364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80047d0 0x7f6cf8004c50 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 -- 172.21.15.46:0/2130145364 >> 172.21.15.46:0/2130145364 conn(0x7f6d0806bfe0 msgr2=0x7f6d0806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:45.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 -- 172.21.15.46:0/2130145364 shutdown_connections 2023-10-30T13:57:45.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 -- 172.21.15.46:0/2130145364 wait complete. 2023-10-30T13:57:45.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 Processor -- start 2023-10-30T13:57:45.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 -- start start 2023-10-30T13:57:45.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80047d0 0x7f6d081efcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d0815b240 0x7f6d081e4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d0815b680 0x7f6d081e4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080ccfb0 con 0x7f6d0815b240 2023-10-30T13:57:45.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081e4480 con 0x7f6cf80047d0 2023-10-30T13:57:45.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0eff8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081f0200 con 0x7f6d0815b680 2023-10-30T13:57:45.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d0815b240 0x7f6d081e4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d0815b240 0x7f6d081e4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35808/0 (socket says 172.21.15.46:35808) 2023-10-30T13:57:45.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d07fff640 1 -- 172.21.15.46:0/3909838371 learned_addr learned my addr 172.21.15.46:0/3909838371 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:45.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0cd94640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80047d0 0x7f6d081efcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.577+0000 7f6d0d595640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d0815b680 0x7f6d081e4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d07fff640 1 -- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d0815b680 msgr2=0x7f6d081e4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:45.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d07fff640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d0815b680 0x7f6d081e4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d07fff640 1 -- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80047d0 msgr2=0x7f6d081efcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:45.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d07fff640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80047d0 0x7f6d081efcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d07fff640 1 -- 172.21.15.46:0/3909838371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d081e2e50 con 0x7f6d0815b240 2023-10-30T13:57:45.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d07fff640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d0815b240 0x7f6d081e4910 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f6cf401c8a0 tx=0x7f6cf4019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:45.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d057fa640 1 -- 172.21.15.46:0/3909838371 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40bd070 con 0x7f6d0815b240 2023-10-30T13:57:45.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d057fa640 1 -- 172.21.15.46:0/3909838371 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf4004020 con 0x7f6d0815b240 2023-10-30T13:57:45.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d057fa640 1 -- 172.21.15.46:0/3909838371 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40043d0 con 0x7f6d0815b240 2023-10-30T13:57:45.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d0eff8640 1 -- 172.21.15.46:0/3909838371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d081e30f0 con 0x7f6d0815b240 2023-10-30T13:57:45.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d0eff8640 1 -- 172.21.15.46:0/3909838371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d081e36b0 con 0x7f6d0815b240 2023-10-30T13:57:45.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d0eff8640 1 -- 172.21.15.46:0/3909838371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d0804e170 con 0x7f6d0815b240 2023-10-30T13:57:45.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d057fa640 1 -- 172.21.15.46:0/3909838371 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6cf40b4070 con 0x7f6d0815b240 2023-10-30T13:57:45.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.581+0000 7f6d057fa640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6cec0664a0 0x7f6cec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.585+0000 7f6d057fa640 1 -- 172.21.15.46:0/3909838371 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f6cf411e550 con 0x7f6d0815b240 2023-10-30T13:57:45.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.585+0000 7f6d0cd94640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6cec0664a0 0x7f6cec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.585+0000 7f6d0cd94640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6cec0664a0 0x7f6cec068960 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc007db0 tx=0x7f6cfc001be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:45.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.589+0000 7f6d057fa640 1 -- 172.21.15.46:0/3909838371 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6cf40eb3d0 con 0x7f6d0815b240 2023-10-30T13:57:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 -- 172.21.15.46:0/3909838371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d080b1dd0 con 0x7f6d0815b240 2023-10-30T13:57:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d057fa640 1 -- 172.21.15.46:0/3909838371 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f6cf40ee9f0 con 0x7f6d0815b240 2023-10-30T13:57:45.784 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:45.784 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:57:44.339842+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 152 2023-10-30T13:57:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 -- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6cec0664a0 msgr2=0x7f6cec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6cec0664a0 0x7f6cec068960 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc007db0 tx=0x7f6cfc001be0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 -- 172.21.15.46:0/3909838371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d0815b240 msgr2=0x7f6d081e4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d0815b240 0x7f6d081e4910 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f6cf401c8a0 tx=0x7f6cf4019810 comp rx=0 tx=0).stop 2023-10-30T13:57:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 -- 172.21.15.46:0/3909838371 shutdown_connections 2023-10-30T13:57:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6cec0664a0 0x7f6cec068960 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d0815b680 0x7f6d081e4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d0815b240 0x7f6d081e4910 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 --2- 172.21.15.46:0/3909838371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80047d0 0x7f6d081efcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 -- 172.21.15.46:0/3909838371 >> 172.21.15.46:0/3909838371 conn(0x7f6d0806bfe0 msgr2=0x7f6d08065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 -- 172.21.15.46:0/3909838371 shutdown_connections 2023-10-30T13:57:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.781+0000 7f6d0eff8640 1 -- 172.21.15.46:0/3909838371 wait complete. 2023-10-30T13:57:45.805 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T13:57:45.806 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:57:45.806 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:45.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.945+0000 7f24d133e640 1 Processor -- start 2023-10-30T13:57:45.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.945+0000 7f24d133e640 1 -- start start 2023-10-30T13:57:45.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.945+0000 7f24d133e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f24cc153d70 0x7f24cc06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.945+0000 7f24d133e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24cc06e470 0x7f24cc0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.945+0000 7f24d133e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24cc06afa0 con 0x7f24cc0c8490 2023-10-30T13:57:45.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.945+0000 7f24d133e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24cc06d780 con 0x7f24cc153d70 2023-10-30T13:57:45.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.945+0000 7f24d133e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc154150 con 0x7f24cc06e470 2023-10-30T13:57:45.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.949+0000 7f24c3fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24cc06e470 0x7f24cc0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.949+0000 7f24caffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f24cc153d70 0x7f24cc06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.949+0000 7f24c3fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24cc06e470 0x7f24cc0cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:49866/0 (socket says 172.21.15.46:49866) 2023-10-30T13:57:45.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.949+0000 7f24c3fff640 1 -- 172.21.15.46:0/2930797526 learned_addr learned my addr 172.21.15.46:0/2930797526 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:45.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.949+0000 7f24cb7fe640 1 --1- 172.21.15.46:0/2930797526 >> v1:172.21.15.154:6789/0 conn(0x7f24cc0c8490 0x7f24cc0c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:60642/0 (socket says 172.21.15.46:60642) 2023-10-30T13:57:45.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.949+0000 7f24c3fff640 1 -- 172.21.15.46:0/2930797526 >> v2:172.21.15.154:3300/0 conn(0x7f24cc153d70 msgr2=0x7f24cc06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:45.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.949+0000 7f24c3fff640 1 --2- 172.21.15.46:0/2930797526 >> v2:172.21.15.154:3300/0 conn(0x7f24cc153d70 0x7f24cc06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.949+0000 7f24c3fff640 1 -- 172.21.15.46:0/2930797526 >> v1:172.21.15.154:6789/0 conn(0x7f24cc0c8490 legacy=0x7f24cc0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:45.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.949+0000 7f24c3fff640 1 -- 172.21.15.46:0/2930797526 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc06e030 con 0x7f24cc06e470 2023-10-30T13:57:45.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.949+0000 7f24ca7fc640 1 -- 172.21.15.46:0/2930797526 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583859245 0 0) 0x7f24cc06afa0 con 0x7f24cc0c8490 2023-10-30T13:57:45.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.949+0000 7f24c3fff640 1 --2- 172.21.15.46:0/2930797526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24cc06e470 0x7f24cc0cce00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009870 tx=0x7f24b401c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ba555bbf8ab5089 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:45.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24ca7fc640 1 -- 172.21.15.46:0/2930797526 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b40aa670 con 0x7f24cc06e470 2023-10-30T13:57:45.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24ca7fc640 1 -- 172.21.15.46:0/2930797526 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b40aa7d0 con 0x7f24cc06e470 2023-10-30T13:57:45.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24ca7fc640 1 -- 172.21.15.46:0/2930797526 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b40aab60 con 0x7f24cc06e470 2023-10-30T13:57:45.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 -- 172.21.15.46:0/2930797526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24cc06e470 msgr2=0x7f24cc0cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:45.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 --2- 172.21.15.46:0/2930797526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24cc06e470 0x7f24cc0cce00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009870 tx=0x7f24b401c510 comp rx=0 tx=0).stop 2023-10-30T13:57:45.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 -- 172.21.15.46:0/2930797526 shutdown_connections 2023-10-30T13:57:45.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 --2- 172.21.15.46:0/2930797526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24cc06e470 0x7f24cc0cce00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 --2- 172.21.15.46:0/2930797526 >> v2:172.21.15.154:3300/0 conn(0x7f24cc153d70 0x7f24cc06ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 -- 172.21.15.46:0/2930797526 >> 172.21.15.46:0/2930797526 conn(0x7f24cc071ea0 msgr2=0x7f24cc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:45.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 -- 172.21.15.46:0/2930797526 shutdown_connections 2023-10-30T13:57:45.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 -- 172.21.15.46:0/2930797526 wait complete. 2023-10-30T13:57:45.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 Processor -- start 2023-10-30T13:57:45.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 -- start start 2023-10-30T13:57:45.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24cc06e470 0x7f24cc0ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24cc0c8490 0x7f24cc156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc153d70 0x7f24cc1f53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc1f3390 con 0x7f24cc153d70 2023-10-30T13:57:45.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc1f38c0 con 0x7f24cc0c8490 2023-10-30T13:57:45.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24d133e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc14b6b0 con 0x7f24cc06e470 2023-10-30T13:57:45.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.953+0000 7f24cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc153d70 0x7f24cc1f53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc153d70 0x7f24cc1f53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35816/0 (socket says 172.21.15.46:35816) 2023-10-30T13:57:45.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24c3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24cc0c8490 0x7f24cc156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24cb7fe640 1 -- 172.21.15.46:0/2932567275 learned_addr learned my addr 172.21.15.46:0/2932567275 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:45.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24caffd640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24cc06e470 0x7f24cc0ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24cb7fe640 1 -- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24cc06e470 msgr2=0x7f24cc0ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:45.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24cb7fe640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24cc06e470 0x7f24cc0ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24cb7fe640 1 -- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24cc0c8490 msgr2=0x7f24cc156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:45.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24cb7fe640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24cc0c8490 0x7f24cc156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24cb7fe640 1 -- 172.21.15.46:0/2932567275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b40094d0 con 0x7f24cc153d70 2023-10-30T13:57:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24cb7fe640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc153d70 0x7f24cc1f53f0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f24c400c7a0 tx=0x7f24c400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24c37fe640 1 -- 172.21.15.46:0/2932567275 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c400a640 con 0x7f24cc153d70 2023-10-30T13:57:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24c37fe640 1 -- 172.21.15.46:0/2932567275 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c400a7a0 con 0x7f24cc153d70 2023-10-30T13:57:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24d133e640 1 -- 172.21.15.46:0/2932567275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24cc1f35e0 con 0x7f24cc153d70 2023-10-30T13:57:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24c37fe640 1 -- 172.21.15.46:0/2932567275 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c400abd0 con 0x7f24cc153d70 2023-10-30T13:57:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24d133e640 1 -- 172.21.15.46:0/2932567275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24cc156870 con 0x7f24cc153d70 2023-10-30T13:57:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24d133e640 1 -- 172.21.15.46:0/2932567275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24cc156a60 con 0x7f24cc153d70 2023-10-30T13:57:45.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.957+0000 7f24c37fe640 1 -- 172.21.15.46:0/2932567275 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f24c400ad30 con 0x7f24cc153d70 2023-10-30T13:57:45.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.961+0000 7f24c37fe640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24ac0664d0 0x7f24ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:45.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.961+0000 7f24c37fe640 1 -- 172.21.15.46:0/2932567275 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f24c4083c50 con 0x7f24cc153d70 2023-10-30T13:57:45.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.961+0000 7f24caffd640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24ac0664d0 0x7f24ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:45.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.961+0000 7f24caffd640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24ac0664d0 0x7f24ac068990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009b60 tx=0x7f24bc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:45.965+0000 7f24c37fe640 1 -- 172.21.15.46:0/2932567275 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f24c4050ad0 con 0x7f24cc153d70 2023-10-30T13:57:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 -- 172.21.15.46:0/2932567275 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24cc04e170 con 0x7f24cc153d70 2023-10-30T13:57:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24c37fe640 1 -- 172.21.15.46:0/2932567275 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f24c4018090 con 0x7f24cc153d70 2023-10-30T13:57:46.157 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:46.157 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:57:44.339842+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 152 2023-10-30T13:57:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 -- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24ac0664d0 msgr2=0x7f24ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24ac0664d0 0x7f24ac068990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009b60 tx=0x7f24bc009190 comp rx=0 tx=0).stop 2023-10-30T13:57:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 -- 172.21.15.46:0/2932567275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc153d70 msgr2=0x7f24cc1f53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc153d70 0x7f24cc1f53f0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f24c400c7a0 tx=0x7f24c400cc60 comp rx=0 tx=0).stop 2023-10-30T13:57:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 -- 172.21.15.46:0/2932567275 shutdown_connections 2023-10-30T13:57:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f24ac0664d0 0x7f24ac068990 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc153d70 0x7f24cc1f53f0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24cc0c8490 0x7f24cc156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 --2- 172.21.15.46:0/2932567275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24cc06e470 0x7f24cc0ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 -- 172.21.15.46:0/2932567275 >> 172.21.15.46:0/2932567275 conn(0x7f24cc071ea0 msgr2=0x7f24cc14e920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 -- 172.21.15.46:0/2932567275 shutdown_connections 2023-10-30T13:57:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:46.153+0000 7f24d133e640 1 -- 172.21.15.46:0/2932567275 wait complete. 2023-10-30T13:57:46.178 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 152, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 152, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7518": { "addr": "172.21.15.46:6833/157453225", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 157453225, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 157453225, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7518, "incarnation": 145, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_7704": { "addr": "172.21.15.46:6836/3734066164", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3734066164, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3734066164, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7704, "incarnation": 152, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:57:44.339842+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7704, "mds_1": 7518 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:57:48.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:48.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f951710e640 1 Processor -- start 2023-10-30T13:57:48.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f951710e640 1 -- start start 2023-10-30T13:57:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f951710e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100c6b80 0x7f95100bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f951710e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f951013f4a0 0x7f95100c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f951710e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95100c0a50 con 0x7f95100c6f60 2023-10-30T13:57:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f951710e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f951006f6e0 con 0x7f951013f4a0 2023-10-30T13:57:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f951710e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951006f9c0 con 0x7f95100c6b80 2023-10-30T13:57:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f95100c6f60 0x7f951013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56162/0 (socket says 172.21.15.46:56162) 2023-10-30T13:57:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950ffff640 1 -- 172.21.15.46:0/3953559427 learned_addr learned my addr 172.21.15.46:0/3953559427 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:57:48.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f9514eaa640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100c6b80 0x7f95100bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f95156ab640 1 --2- 172.21.15.46:0/3953559427 >> v2:172.21.15.154:3300/0 conn(0x7f951013f4a0 0x7f95100c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832153011 0 0) 0x7f95100c0a50 con 0x7f95100c6f60 2023-10-30T13:57:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9500003530 con 0x7f95100c6f60 2023-10-30T13:57:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 779317980 0 0) 0x7f9500003530 con 0x7f95100c6f60 2023-10-30T13:57:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100c6b80 msgr2=0x7f95100bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100c6b80 0x7f95100bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 >> v2:172.21.15.154:3300/0 conn(0x7f951013f4a0 msgr2=0x7f95100c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 --2- 172.21.15.46:0/3953559427 >> v2:172.21.15.154:3300/0 conn(0x7f951013f4a0 0x7f95100c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951006fca0 con 0x7f95100c6f60 2023-10-30T13:57:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f94fc001680 con 0x7f95100c6f60 2023-10-30T13:57:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 >> v1:172.21.15.46:6789/0 conn(0x7f95100c6f60 legacy=0x7f951013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:48.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9500003980 0x7f9500003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9500004370 0x7f9500004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9500004ec0 0x7f950000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951006f6e0 con 0x7f9500004ec0 2023-10-30T13:57:48.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951006f9c0 con 0x7f9500004370 2023-10-30T13:57:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950000bed0 con 0x7f9500003980 2023-10-30T13:57:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f9514eaa640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100c6b80 0x7f95100bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:57:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.329+0000 7f95156ab640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9500004ec0 0x7f950000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f9514eaa640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9500003980 0x7f9500003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f950ffff640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9500004370 0x7f9500004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f95156ab640 1 -- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9500003980 msgr2=0x7f9500003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:48.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f95156ab640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9500003980 0x7f9500003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f95156ab640 1 -- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9500004370 msgr2=0x7f9500004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:48.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f95156ab640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9500004370 0x7f9500004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f95156ab640 1 -- 172.21.15.46:0/3953559427 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9500003530 con 0x7f9500004ec0 2023-10-30T13:57:48.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f9514eaa640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9500003980 0x7f9500003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f95156ab640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9500004ec0 0x7f950000b990 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f951006f4f0 tx=0x7f950801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83be9f4ea0a18adf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95080a5c50 con 0x7f9500004ec0 2023-10-30T13:57:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950800f040 con 0x7f9500004ec0 2023-10-30T13:57:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f950f7fe640 1 -- 172.21.15.46:0/3953559427 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95080ad440 con 0x7f9500004ec0 2023-10-30T13:57:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f951710e640 1 -- 172.21.15.46:0/3953559427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9500004ec0 msgr2=0x7f950000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f951710e640 1 --2- 172.21.15.46:0/3953559427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9500004ec0 0x7f950000b990 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f951006f4f0 tx=0x7f950801cc40 comp rx=0 tx=0).stop 2023-10-30T13:57:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.333+0000 7f95156ab640 1 -- 172.21.15.46:0/3953559427 reap_dead start 2023-10-30T13:57:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f951710e640 1 -- 172.21.15.46:0/3953559427 shutdown_connections 2023-10-30T13:57:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f951710e640 1 -- 172.21.15.46:0/3953559427 >> 172.21.15.46:0/3953559427 conn(0x7f95100c7910 msgr2=0x7f95100c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f951710e640 1 -- 172.21.15.46:0/3953559427 shutdown_connections 2023-10-30T13:57:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f951710e640 1 -- 172.21.15.46:0/3953559427 wait complete. 2023-10-30T13:57:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f951710e640 1 Processor -- start 2023-10-30T13:57:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f951710e640 1 -- start start 2023-10-30T13:57:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f951710e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510158c50 0x7f95101f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f951710e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f951013c780 0x7f951014f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f951710e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510159030 0x7f951014fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f951710e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9510161d30 con 0x7f9510158c50 2023-10-30T13:57:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f951710e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9510139430 con 0x7f9510159030 2023-10-30T13:57:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f951710e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95101f24a0 con 0x7f951013c780 2023-10-30T13:57:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f9514eaa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510158c50 0x7f95101f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f9514eaa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510158c50 0x7f95101f2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55540/0 (socket says 172.21.15.46:55540) 2023-10-30T13:57:48.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f9514eaa640 1 -- 172.21.15.46:0/4028937344 learned_addr learned my addr 172.21.15.46:0/4028937344 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:48.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f950ffff640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f951013c780 0x7f951014f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f95156ab640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510159030 0x7f951014fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f9514eaa640 1 -- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f951013c780 msgr2=0x7f951014f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f9514eaa640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f951013c780 0x7f951014f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f9514eaa640 1 -- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510159030 msgr2=0x7f951014fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f9514eaa640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510159030 0x7f951014fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.337+0000 7f9514eaa640 1 -- 172.21.15.46:0/4028937344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9510138e70 con 0x7f9510158c50 2023-10-30T13:57:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.341+0000 7f9514eaa640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510158c50 0x7f95101f2c60 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f9504011680 tx=0x7f9504011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.341+0000 7f950d7fa640 1 -- 172.21.15.46:0/4028937344 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950400f960 con 0x7f9510158c50 2023-10-30T13:57:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.341+0000 7f950d7fa640 1 -- 172.21.15.46:0/4028937344 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950400fac0 con 0x7f9510158c50 2023-10-30T13:57:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.341+0000 7f950d7fa640 1 -- 172.21.15.46:0/4028937344 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950400fe30 con 0x7f9510158c50 2023-10-30T13:57:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.341+0000 7f951710e640 1 -- 172.21.15.46:0/4028937344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9510139150 con 0x7f9510158c50 2023-10-30T13:57:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.341+0000 7f951710e640 1 -- 172.21.15.46:0/4028937344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95101f31a0 con 0x7f9510158c50 2023-10-30T13:57:48.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.341+0000 7f951710e640 1 -- 172.21.15.46:0/4028937344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95101f2300 con 0x7f9510158c50 2023-10-30T13:57:48.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.341+0000 7f950d7fa640 1 -- 172.21.15.46:0/4028937344 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9504015460 con 0x7f9510158c50 2023-10-30T13:57:48.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.341+0000 7f950d7fa640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94f40664a0 0x7f94f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:48.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.341+0000 7f950d7fa640 1 -- 172.21.15.46:0/4028937344 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f9504084fe0 con 0x7f9510158c50 2023-10-30T13:57:48.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.341+0000 7f950ffff640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94f40664a0 0x7f94f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:48.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.345+0000 7f950ffff640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94f40664a0 0x7f94f4068960 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f94fc001ec0 tx=0x7f94fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:48.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.345+0000 7f950d7fa640 1 -- 172.21.15.46:0/4028937344 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f95101f2300 con 0x7f9510158c50 2023-10-30T13:57:48.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.537+0000 7f951710e640 1 -- 172.21.15.46:0/4028937344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95101f2300 con 0x7f9510158c50 2023-10-30T13:57:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.537+0000 7f950d7fa640 1 -- 172.21.15.46:0/4028937344 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f95101f2300 con 0x7f9510158c50 2023-10-30T13:57:48.543 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:48.543 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:57:44.339842+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 152 2023-10-30T13:57:48.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.541+0000 7f951710e640 1 -- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94f40664a0 msgr2=0x7f94f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:48.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.541+0000 7f951710e640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94f40664a0 0x7f94f4068960 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f94fc001ec0 tx=0x7f94fc009040 comp rx=0 tx=0).stop 2023-10-30T13:57:48.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.541+0000 7f951710e640 1 -- 172.21.15.46:0/4028937344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510158c50 msgr2=0x7f95101f2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:48.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.541+0000 7f951710e640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510158c50 0x7f95101f2c60 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f9504011680 tx=0x7f9504011b40 comp rx=0 tx=0).stop 2023-10-30T13:57:48.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.541+0000 7f951710e640 1 -- 172.21.15.46:0/4028937344 shutdown_connections 2023-10-30T13:57:48.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.541+0000 7f951710e640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94f40664a0 0x7f94f4068960 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.541+0000 7f951710e640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510159030 0x7f951014fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.541+0000 7f951710e640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f951013c780 0x7f951014f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.541+0000 7f951710e640 1 --2- 172.21.15.46:0/4028937344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510158c50 0x7f95101f2c60 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:48.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.541+0000 7f951710e640 1 -- 172.21.15.46:0/4028937344 >> 172.21.15.46:0/4028937344 conn(0x7f95100c7910 msgr2=0x7f95100c1870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:48.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.541+0000 7f951710e640 1 -- 172.21.15.46:0/4028937344 shutdown_connections 2023-10-30T13:57:48.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:48.541+0000 7f951710e640 1 -- 172.21.15.46:0/4028937344 wait complete. 2023-10-30T13:57:48.565 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:57:50.566 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:50.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c11571640 1 Processor -- start 2023-10-30T13:57:50.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c11571640 1 -- start start 2023-10-30T13:57:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c11571640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c0c1538f0 0x7f0c0c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c11571640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0c0c0cefa0 0x7f0c0c0cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c11571640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0c0ca6a0 con 0x7f0c0c15cd50 2023-10-30T13:57:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c11571640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0c0cf8f0 con 0x7f0c0c1538f0 2023-10-30T13:57:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c11571640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0c15d300 con 0x7f0c0c0cefa0 2023-10-30T13:57:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c0b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0c0c15cd50 0x7f0c0c0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56172/0 (socket says 172.21.15.46:56172) 2023-10-30T13:57:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c0b7fe640 1 -- 172.21.15.46:0/3156541977 learned_addr learned my addr 172.21.15.46:0/3156541977 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:57:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c0bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c0c1538f0 0x7f0c0c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c10d70640 1 --2- 172.21.15.46:0/3156541977 >> v2:172.21.15.154:3300/0 conn(0x7f0c0c0cefa0 0x7f0c0c0cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c0bfff640 1 -- 172.21.15.46:0/3156541977 >> v2:172.21.15.154:3300/0 conn(0x7f0c0c0cefa0 msgr2=0x7f0c0c0cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:50.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c0bfff640 1 --2- 172.21.15.46:0/3156541977 >> v2:172.21.15.154:3300/0 conn(0x7f0c0c0cefa0 0x7f0c0c0cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c0affd640 1 -- 172.21.15.46:0/3156541977 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668781038 0 0) 0x7f0c0c0ca6a0 con 0x7f0c0c15cd50 2023-10-30T13:57:50.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c0bfff640 1 -- 172.21.15.46:0/3156541977 >> v1:172.21.15.46:6789/0 conn(0x7f0c0c15cd50 legacy=0x7f0c0c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:50.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c0bfff640 1 -- 172.21.15.46:0/3156541977 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c15d5e0 con 0x7f0c0c1538f0 2023-10-30T13:57:50.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.713+0000 7f0c10d70640 1 --2- 172.21.15.46:0/3156541977 >> v2:172.21.15.154:3300/0 conn(0x7f0c0c0cefa0 0x7f0c0c0cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.717+0000 7f0c0bfff640 1 --2- 172.21.15.46:0/3156541977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c0c1538f0 0x7f0c0c159fb0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c066a40 tx=0x7f0bfc01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4973f23f90b6905 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.717+0000 7f0c0affd640 1 -- 172.21.15.46:0/3156541977 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc0a8950 con 0x7f0c0c1538f0 2023-10-30T13:57:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.717+0000 7f0c0affd640 1 -- 172.21.15.46:0/3156541977 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc0a8ab0 con 0x7f0c0c1538f0 2023-10-30T13:57:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.717+0000 7f0c0affd640 1 -- 172.21.15.46:0/3156541977 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc0a8e40 con 0x7f0c0c1538f0 2023-10-30T13:57:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.717+0000 7f0c11571640 1 -- 172.21.15.46:0/3156541977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c0c1538f0 msgr2=0x7f0c0c159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.717+0000 7f0c11571640 1 --2- 172.21.15.46:0/3156541977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c0c1538f0 0x7f0c0c159fb0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c066a40 tx=0x7f0bfc01ec50 comp rx=0 tx=0).stop 2023-10-30T13:57:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.717+0000 7f0c11571640 1 -- 172.21.15.46:0/3156541977 shutdown_connections 2023-10-30T13:57:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.717+0000 7f0c11571640 1 --2- 172.21.15.46:0/3156541977 >> v2:172.21.15.154:3300/0 conn(0x7f0c0c0cefa0 0x7f0c0c0cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.717+0000 7f0c11571640 1 --2- 172.21.15.46:0/3156541977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c0c1538f0 0x7f0c0c159fb0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.717+0000 7f0c11571640 1 -- 172.21.15.46:0/3156541977 >> 172.21.15.46:0/3156541977 conn(0x7f0c0c06a640 msgr2=0x7f0c0c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:50.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.717+0000 7f0c11571640 1 -- 172.21.15.46:0/3156541977 shutdown_connections 2023-10-30T13:57:50.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.717+0000 7f0c11571640 1 -- 172.21.15.46:0/3156541977 wait complete. 2023-10-30T13:57:50.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.721+0000 7f0c11571640 1 Processor -- start 2023-10-30T13:57:50.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.721+0000 7f0c11571640 1 -- start start 2023-10-30T13:57:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.721+0000 7f0c11571640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c0c0cefa0 0x7f0c0c15bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.721+0000 7f0c11571640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c0c1538f0 0x7f0c0c13e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.721+0000 7f0c11571640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c0c15cd50 0x7f0c0c13eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.721+0000 7f0c11571640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0b34b0 con 0x7f0c0c15cd50 2023-10-30T13:57:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.721+0000 7f0c11571640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0b4bf0 con 0x7f0c0c1538f0 2023-10-30T13:57:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.721+0000 7f0c11571640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0b4ed0 con 0x7f0c0c0cefa0 2023-10-30T13:57:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.721+0000 7f0c10d70640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c0c15cd50 0x7f0c0c13eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c10d70640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c0c15cd50 0x7f0c0c13eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55552/0 (socket says 172.21.15.46:55552) 2023-10-30T13:57:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c10d70640 1 -- 172.21.15.46:0/995459633 learned_addr learned my addr 172.21.15.46:0/995459633 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c0b7fe640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c0c1538f0 0x7f0c0c13e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c0bfff640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c0c0cefa0 0x7f0c0c15bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c10d70640 1 -- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c0c0cefa0 msgr2=0x7f0c0c15bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c10d70640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c0c0cefa0 0x7f0c0c15bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c10d70640 1 -- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c0c1538f0 msgr2=0x7f0c0c13e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c10d70640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c0c1538f0 0x7f0c0c13e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c10d70640 1 -- 172.21.15.46:0/995459633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc009420 con 0x7f0c0c15cd50 2023-10-30T13:57:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c0bfff640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c0c0cefa0 0x7f0c0c15bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c0b7fe640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c0c1538f0 0x7f0c0c13e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c10d70640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c0c15cd50 0x7f0c0c13eea0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f0c04007ef0 tx=0x7f0c0400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c08ff9640 1 -- 172.21.15.46:0/995459633 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04003ba0 con 0x7f0c0c15cd50 2023-10-30T13:57:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c08ff9640 1 -- 172.21.15.46:0/995459633 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0400f040 con 0x7f0c0c15cd50 2023-10-30T13:57:50.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c11571640 1 -- 172.21.15.46:0/995459633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0c0b3750 con 0x7f0c0c15cd50 2023-10-30T13:57:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c08ff9640 1 -- 172.21.15.46:0/995459633 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0400de80 con 0x7f0c0c15cd50 2023-10-30T13:57:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c11571640 1 -- 172.21.15.46:0/995459633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0c144c10 con 0x7f0c0c15cd50 2023-10-30T13:57:50.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.725+0000 7f0c11571640 1 -- 172.21.15.46:0/995459633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0c0b3c70 con 0x7f0c0c15cd50 2023-10-30T13:57:50.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.729+0000 7f0c08ff9640 1 -- 172.21.15.46:0/995459633 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c04004030 con 0x7f0c0c15cd50 2023-10-30T13:57:50.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.729+0000 7f0c08ff9640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0bf80664d0 0x7f0bf8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:50.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.729+0000 7f0c08ff9640 1 -- 172.21.15.46:0/995459633 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f0c04084700 con 0x7f0c0c15cd50 2023-10-30T13:57:50.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.729+0000 7f0c0bfff640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0bf80664d0 0x7f0bf8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:50.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.729+0000 7f0c0bfff640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0bf80664d0 0x7f0bf8068990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0b2000 tx=0x7f0bfc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:50.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.733+0000 7f0c08ff9640 1 -- 172.21.15.46:0/995459633 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0c0c0b3c70 con 0x7f0c0c15cd50 2023-10-30T13:57:50.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.925+0000 7f0c11571640 1 -- 172.21.15.46:0/995459633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c0c0b3c70 con 0x7f0c0c15cd50 2023-10-30T13:57:50.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.925+0000 7f0c08ff9640 1 -- 172.21.15.46:0/995459633 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f0c0c0b3c70 con 0x7f0c0c15cd50 2023-10-30T13:57:50.929 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:50.930 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:57:44.339842+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:50.930 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 152 2023-10-30T13:57:50.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.929+0000 7f0c11571640 1 -- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0bf80664d0 msgr2=0x7f0bf8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:50.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.929+0000 7f0c11571640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0bf80664d0 0x7f0bf8068990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0b2000 tx=0x7f0bfc008040 comp rx=0 tx=0).stop 2023-10-30T13:57:50.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.929+0000 7f0c11571640 1 -- 172.21.15.46:0/995459633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c0c15cd50 msgr2=0x7f0c0c13eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:50.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.929+0000 7f0c11571640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c0c15cd50 0x7f0c0c13eea0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f0c04007ef0 tx=0x7f0c0400d4a0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.929+0000 7f0c11571640 1 -- 172.21.15.46:0/995459633 shutdown_connections 2023-10-30T13:57:50.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.929+0000 7f0c11571640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0bf80664d0 0x7f0bf8068990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.929+0000 7f0c11571640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c0c15cd50 0x7f0c0c13eea0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.929+0000 7f0c11571640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c0c1538f0 0x7f0c0c13e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.929+0000 7f0c11571640 1 --2- 172.21.15.46:0/995459633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c0c0cefa0 0x7f0c0c15bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:50.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.929+0000 7f0c11571640 1 -- 172.21.15.46:0/995459633 >> 172.21.15.46:0/995459633 conn(0x7f0c0c06a640 msgr2=0x7f0c0c1491c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:50.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.929+0000 7f0c11571640 1 -- 172.21.15.46:0/995459633 shutdown_connections 2023-10-30T13:57:50.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:50.929+0000 7f0c11571640 1 -- 172.21.15.46:0/995459633 wait complete. 2023-10-30T13:57:50.953 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:57:52.954 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:53.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.141+0000 7f6841365640 1 Processor -- start 2023-10-30T13:57:53.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.141+0000 7f6841365640 1 -- start start 2023-10-30T13:57:53.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.141+0000 7f6841365640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f683c15d6f0 0x7f683c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.141+0000 7f6841365640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f683c0cf160 0x7f683c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.141+0000 7f6841365640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f683c0cb800 con 0x7f683c1538a0 2023-10-30T13:57:53.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f6841365640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f683c0cb330 con 0x7f683c0cf160 2023-10-30T13:57:53.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f6841365640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f683c0cad70 con 0x7f683c15d6f0 2023-10-30T13:57:53.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f683affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f683c1538a0 0x7f683c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56188/0 (socket says 172.21.15.46:56188) 2023-10-30T13:57:53.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f683affd640 1 -- 172.21.15.46:0/1601583060 learned_addr learned my addr 172.21.15.46:0/1601583060 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:57:53.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f683b7fe640 1 --2- 172.21.15.46:0/1601583060 >> v2:172.21.15.46:3300/0 conn(0x7f683c0cf160 0x7f683c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f683a7fc640 1 --2- 172.21.15.46:0/1601583060 >> v2:172.21.15.154:3300/0 conn(0x7f683c15d6f0 0x7f683c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f6839ffb640 1 -- 172.21.15.46:0/1601583060 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542740201 0 0) 0x7f683c0cb800 con 0x7f683c1538a0 2023-10-30T13:57:53.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f6839ffb640 1 -- 172.21.15.46:0/1601583060 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6824003530 con 0x7f683c1538a0 2023-10-30T13:57:53.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f683b7fe640 1 -- 172.21.15.46:0/1601583060 >> v2:172.21.15.154:3300/0 conn(0x7f683c15d6f0 msgr2=0x7f683c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:53.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f683b7fe640 1 --2- 172.21.15.46:0/1601583060 >> v2:172.21.15.154:3300/0 conn(0x7f683c15d6f0 0x7f683c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f683b7fe640 1 -- 172.21.15.46:0/1601583060 >> v1:172.21.15.46:6789/0 conn(0x7f683c1538a0 legacy=0x7f683c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:53.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f683b7fe640 1 -- 172.21.15.46:0/1601583060 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683c0cb050 con 0x7f683c0cf160 2023-10-30T13:57:53.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.145+0000 7f683b7fe640 1 --2- 172.21.15.46:0/1601583060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f683c0cf160 0x7f683c0cf590 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f683c0b30a0 tx=0x7f683401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a37b595b607b2e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:53.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6839ffb640 1 -- 172.21.15.46:0/1601583060 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68340b0440 con 0x7f683c0cf160 2023-10-30T13:57:53.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6839ffb640 1 -- 172.21.15.46:0/1601583060 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68340b05a0 con 0x7f683c0cf160 2023-10-30T13:57:53.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6839ffb640 1 -- 172.21.15.46:0/1601583060 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68340b0930 con 0x7f683c0cf160 2023-10-30T13:57:53.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 -- 172.21.15.46:0/1601583060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f683c0cf160 msgr2=0x7f683c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:53.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 --2- 172.21.15.46:0/1601583060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f683c0cf160 0x7f683c0cf590 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f683c0b30a0 tx=0x7f683401f0f0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 -- 172.21.15.46:0/1601583060 shutdown_connections 2023-10-30T13:57:53.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 --2- 172.21.15.46:0/1601583060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f683c0cf160 0x7f683c0cf590 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 --2- 172.21.15.46:0/1601583060 >> v2:172.21.15.154:3300/0 conn(0x7f683c15d6f0 0x7f683c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 -- 172.21.15.46:0/1601583060 >> 172.21.15.46:0/1601583060 conn(0x7f683c06bfe0 msgr2=0x7f683c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:53.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 -- 172.21.15.46:0/1601583060 shutdown_connections 2023-10-30T13:57:53.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 -- 172.21.15.46:0/1601583060 wait complete. 2023-10-30T13:57:53.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 Processor -- start 2023-10-30T13:57:53.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 -- start start 2023-10-30T13:57:53.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f683c0cf160 0x7f683c1efeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f683c1538a0 0x7f683c157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f683c15d6f0 0x7f683c0b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c1e4e50 con 0x7f683c0cf160 2023-10-30T13:57:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c1f03f0 con 0x7f683c15d6f0 2023-10-30T13:57:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.149+0000 7f6841365640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f683c159000 con 0x7f683c1538a0 2023-10-30T13:57:53.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f683affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f683c0cf160 0x7f683c1efeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f683affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f683c0cf160 0x7f683c1efeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55576/0 (socket says 172.21.15.46:55576) 2023-10-30T13:57:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f683affd640 1 -- 172.21.15.46:0/1683992213 learned_addr learned my addr 172.21.15.46:0/1683992213 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f683b7fe640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f683c15d6f0 0x7f683c0b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f683a7fc640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f683c1538a0 0x7f683c157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f683affd640 1 -- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f683c1538a0 msgr2=0x7f683c157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f683affd640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f683c1538a0 0x7f683c157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f683affd640 1 -- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f683c15d6f0 msgr2=0x7f683c0b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f683affd640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f683c15d6f0 0x7f683c0b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f683affd640 1 -- 172.21.15.46:0/1683992213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6834009a70 con 0x7f683c0cf160 2023-10-30T13:57:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f683affd640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f683c0cf160 0x7f683c1efeb0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f682c00c4d0 tx=0x7f682c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:53.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f68237fe640 1 -- 172.21.15.46:0/1683992213 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c005cf0 con 0x7f683c0cf160 2023-10-30T13:57:53.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f68237fe640 1 -- 172.21.15.46:0/1683992213 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682c005e50 con 0x7f683c0cf160 2023-10-30T13:57:53.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f6841365640 1 -- 172.21.15.46:0/1683992213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683c157a60 con 0x7f683c0cf160 2023-10-30T13:57:53.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f68237fe640 1 -- 172.21.15.46:0/1683992213 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682c007270 con 0x7f683c0cf160 2023-10-30T13:57:53.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f6841365640 1 -- 172.21.15.46:0/1683992213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683c1592c0 con 0x7f683c0cf160 2023-10-30T13:57:53.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f6841365640 1 -- 172.21.15.46:0/1683992213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f683c0b2a50 con 0x7f683c0cf160 2023-10-30T13:57:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.153+0000 7f68237fe640 1 -- 172.21.15.46:0/1683992213 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f682c0073d0 con 0x7f683c0cf160 2023-10-30T13:57:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.157+0000 7f68237fe640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68280664a0 0x7f6828068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.157+0000 7f68237fe640 1 -- 172.21.15.46:0/1683992213 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f682c083ba0 con 0x7f683c0cf160 2023-10-30T13:57:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.157+0000 7f683a7fc640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68280664a0 0x7f6828068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:53.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.157+0000 7f683a7fc640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68280664a0 0x7f6828068960 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f683c1589c0 tx=0x7f6830008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:53.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.161+0000 7f68237fe640 1 -- 172.21.15.46:0/1683992213 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f682c050900 con 0x7f683c0cf160 2023-10-30T13:57:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 -- 172.21.15.46:0/1683992213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f683c04e170 con 0x7f683c0cf160 2023-10-30T13:57:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f68237fe640 1 -- 172.21.15.46:0/1683992213 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f682c017020 con 0x7f683c0cf160 2023-10-30T13:57:53.361 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:53.361 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:57:44.339842+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 152 2023-10-30T13:57:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 -- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68280664a0 msgr2=0x7f6828068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68280664a0 0x7f6828068960 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f683c1589c0 tx=0x7f6830008040 comp rx=0 tx=0).stop 2023-10-30T13:57:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 -- 172.21.15.46:0/1683992213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f683c0cf160 msgr2=0x7f683c1efeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f683c0cf160 0x7f683c1efeb0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f682c00c4d0 tx=0x7f682c00c990 comp rx=0 tx=0).stop 2023-10-30T13:57:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 -- 172.21.15.46:0/1683992213 shutdown_connections 2023-10-30T13:57:53.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68280664a0 0x7f6828068960 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f683c15d6f0 0x7f683c0b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f683c1538a0 0x7f683c157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 --2- 172.21.15.46:0/1683992213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f683c0cf160 0x7f683c1efeb0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:53.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 -- 172.21.15.46:0/1683992213 >> 172.21.15.46:0/1683992213 conn(0x7f683c06bfe0 msgr2=0x7f683c15aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:53.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 -- 172.21.15.46:0/1683992213 shutdown_connections 2023-10-30T13:57:53.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:53.357+0000 7f6841365640 1 -- 172.21.15.46:0/1683992213 wait complete. 2023-10-30T13:57:53.390 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:57:53.711 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:57:53.716 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:57:55.391 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.532+0000 7f9d6566b640 1 Processor -- start 2023-10-30T13:57:55.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.532+0000 7f9d6566b640 1 -- start start 2023-10-30T13:57:55.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.532+0000 7f9d6566b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d60153d70 0x7f9d6006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.532+0000 7f9d6566b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6006afa0 con 0x7f9d6006e470 2023-10-30T13:57:55.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.532+0000 7f9d6566b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d60154150 con 0x7f9d600c8490 2023-10-30T13:57:55.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.532+0000 7f9d6566b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6006d780 con 0x7f9d60153d70 2023-10-30T13:57:55.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.532+0000 7f9d567fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9d6006e470 0x7f9d600cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56200/0 (socket says 172.21.15.46:56200) 2023-10-30T13:57:55.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.532+0000 7f9d567fc640 1 -- 172.21.15.46:0/2455042856 learned_addr learned my addr 172.21.15.46:0/2455042856 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:57:55.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.532+0000 7f9d5effd640 1 --2- 172.21.15.46:0/2455042856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d60153d70 0x7f9d6006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.532+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143353060 0 0) 0x7f9d6006afa0 con 0x7f9d6006e470 2023-10-30T13:57:55.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d4c003530 con 0x7f9d6006e470 2023-10-30T13:57:55.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1754995534 0 0) 0x7f9d4c003530 con 0x7f9d6006e470 2023-10-30T13:57:55.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d60153d70 msgr2=0x7f9d6006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:55.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 --2- 172.21.15.46:0/2455042856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d60153d70 0x7f9d6006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 >> v1:172.21.15.154:6789/0 conn(0x7f9d600c8490 legacy=0x7f9d600c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:55.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6006e030 con 0x7f9d6006e470 2023-10-30T13:57:55.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f9d50001680 con 0x7f9d6006e470 2023-10-30T13:57:55.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 >> v1:172.21.15.46:6789/0 conn(0x7f9d6006e470 legacy=0x7f9d600cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:55.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 --2- 172.21.15.46:0/2455042856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0039b0 0x7f9d4c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 --2- 172.21.15.46:0/2455042856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d60153d70 0x7f9d4c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 --2- 172.21.15.46:0/2455042856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d4c004d10 0x7f9d4c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6006d780 con 0x7f9d4c0039b0 2023-10-30T13:57:55.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c00b990 con 0x7f9d4c004d10 2023-10-30T13:57:55.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c00bb50 con 0x7f9d60153d70 2023-10-30T13:57:55.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5effd640 1 --2- 172.21.15.46:0/2455042856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0039b0 0x7f9d4c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d567fc640 1 --2- 172.21.15.46:0/2455042856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d60153d70 0x7f9d4c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5f7fe640 1 --2- 172.21.15.46:0/2455042856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d4c004d10 0x7f9d4c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5effd640 1 -- 172.21.15.46:0/2455042856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d60153d70 msgr2=0x7f9d4c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:55.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5effd640 1 --2- 172.21.15.46:0/2455042856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d60153d70 0x7f9d4c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5effd640 1 -- 172.21.15.46:0/2455042856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d4c004d10 msgr2=0x7f9d4c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:55.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5effd640 1 --2- 172.21.15.46:0/2455042856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d4c004d10 0x7f9d4c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.536+0000 7f9d5effd640 1 -- 172.21.15.46:0/2455042856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c003530 con 0x7f9d4c0039b0 2023-10-30T13:57:55.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d5effd640 1 --2- 172.21.15.46:0/2455042856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0039b0 0x7f9d4c003d90 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009ae0 tx=0x7f9d4801c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=138d51e977c47fb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:55.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d480a5580 con 0x7f9d4c0039b0 2023-10-30T13:57:55.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d480a56e0 con 0x7f9d4c0039b0 2023-10-30T13:57:55.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d5e7fc640 1 -- 172.21.15.46:0/2455042856 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d480a5a90 con 0x7f9d4c0039b0 2023-10-30T13:57:55.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 -- 172.21.15.46:0/2455042856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0039b0 msgr2=0x7f9d4c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:55.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 --2- 172.21.15.46:0/2455042856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0039b0 0x7f9d4c003d90 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009ae0 tx=0x7f9d4801c990 comp rx=0 tx=0).stop 2023-10-30T13:57:55.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d5f7fe640 1 -- 172.21.15.46:0/2455042856 reap_dead start 2023-10-30T13:57:55.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 -- 172.21.15.46:0/2455042856 shutdown_connections 2023-10-30T13:57:55.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 -- 172.21.15.46:0/2455042856 >> 172.21.15.46:0/2455042856 conn(0x7f9d60071ea0 msgr2=0x7f9d60065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:55.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 -- 172.21.15.46:0/2455042856 shutdown_connections 2023-10-30T13:57:55.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 -- 172.21.15.46:0/2455042856 wait complete. 2023-10-30T13:57:55.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 Processor -- start 2023-10-30T13:57:55.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 -- start start 2023-10-30T13:57:55.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d4c0039b0 0x7f9d601efad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d60156de0 0x7f9d601e3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60154330 0x7f9d6006eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d601551a0 con 0x7f9d60156de0 2023-10-30T13:57:55.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d601e5780 con 0x7f9d60154330 2023-10-30T13:57:55.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.540+0000 7f9d6566b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d600cc3b0 con 0x7f9d4c0039b0 2023-10-30T13:57:55.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d567fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d60156de0 0x7f9d601e3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d567fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d60156de0 0x7f9d601e3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55598/0 (socket says 172.21.15.46:55598) 2023-10-30T13:57:55.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d567fc640 1 -- 172.21.15.46:0/1930235667 learned_addr learned my addr 172.21.15.46:0/1930235667 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:55.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d5effd640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d4c0039b0 0x7f9d601efad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d5f7fe640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60154330 0x7f9d6006eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d567fc640 1 -- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d4c0039b0 msgr2=0x7f9d601efad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:55.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d567fc640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d4c0039b0 0x7f9d601efad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d567fc640 1 -- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60154330 msgr2=0x7f9d6006eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:55.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d567fc640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60154330 0x7f9d6006eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d567fc640 1 -- 172.21.15.46:0/1930235667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d600cc690 con 0x7f9d60156de0 2023-10-30T13:57:55.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d5f7fe640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60154330 0x7f9d6006eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:57:55.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d567fc640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d60156de0 0x7f9d601e3730 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9d50000f70 tx=0x7f9d5000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:55.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d57fff640 1 -- 172.21.15.46:0/1930235667 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5000c8a0 con 0x7f9d60156de0 2023-10-30T13:57:55.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d57fff640 1 -- 172.21.15.46:0/1930235667 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5000ca00 con 0x7f9d60156de0 2023-10-30T13:57:55.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d6566b640 1 -- 172.21.15.46:0/1930235667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d600cc970 con 0x7f9d60156de0 2023-10-30T13:57:55.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d57fff640 1 -- 172.21.15.46:0/1930235667 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5000cdb0 con 0x7f9d60156de0 2023-10-30T13:57:55.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d6566b640 1 -- 172.21.15.46:0/1930235667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d601e2f30 con 0x7f9d60156de0 2023-10-30T13:57:55.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d6566b640 1 -- 172.21.15.46:0/1930235667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6004e170 con 0x7f9d60156de0 2023-10-30T13:57:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d57fff640 1 -- 172.21.15.46:0/1930235667 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9d5001d460 con 0x7f9d60156de0 2023-10-30T13:57:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.544+0000 7f9d57fff640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9d400664a0 0x7f9d40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.548+0000 7f9d57fff640 1 -- 172.21.15.46:0/1930235667 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f9d50084530 con 0x7f9d60156de0 2023-10-30T13:57:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.548+0000 7f9d5effd640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9d400664a0 0x7f9d40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.548+0000 7f9d5effd640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9d400664a0 0x7f9d40068960 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9d480af000 tx=0x7f9d480046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.552+0000 7f9d57fff640 1 -- 172.21.15.46:0/1930235667 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9d500513c0 con 0x7f9d60156de0 2023-10-30T13:57:55.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.740+0000 7f9d6566b640 1 -- 172.21.15.46:0/1930235667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d60157360 con 0x7f9d60156de0 2023-10-30T13:57:55.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.740+0000 7f9d57fff640 1 -- 172.21.15.46:0/1930235667 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f9d500549e0 con 0x7f9d60156de0 2023-10-30T13:57:55.747 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:55.747 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:57:55.140760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:resolve","state_seq":23,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:55.747 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 153 2023-10-30T13:57:55.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.744+0000 7f9d6566b640 1 -- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9d400664a0 msgr2=0x7f9d40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:55.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.744+0000 7f9d6566b640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9d400664a0 0x7f9d40068960 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9d480af000 tx=0x7f9d480046d0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.744+0000 7f9d6566b640 1 -- 172.21.15.46:0/1930235667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d60156de0 msgr2=0x7f9d601e3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:55.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.744+0000 7f9d6566b640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d60156de0 0x7f9d601e3730 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9d50000f70 tx=0x7f9d5000ed50 comp rx=0 tx=0).stop 2023-10-30T13:57:55.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.744+0000 7f9d6566b640 1 -- 172.21.15.46:0/1930235667 shutdown_connections 2023-10-30T13:57:55.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.744+0000 7f9d6566b640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9d400664a0 0x7f9d40068960 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.744+0000 7f9d6566b640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60154330 0x7f9d6006eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.744+0000 7f9d6566b640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d60156de0 0x7f9d601e3730 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.744+0000 7f9d6566b640 1 --2- 172.21.15.46:0/1930235667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d4c0039b0 0x7f9d601efad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:55.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.744+0000 7f9d6566b640 1 -- 172.21.15.46:0/1930235667 >> 172.21.15.46:0/1930235667 conn(0x7f9d60071ea0 msgr2=0x7f9d600c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:55.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.744+0000 7f9d6566b640 1 -- 172.21.15.46:0/1930235667 shutdown_connections 2023-10-30T13:57:55.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:55.744+0000 7f9d6566b640 1 -- 172.21.15.46:0/1930235667 wait complete. 2023-10-30T13:57:55.767 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:57:57.769 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:57:57.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f498a9a7640 1 Processor -- start 2023-10-30T13:57:57.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f498a9a7640 1 -- start start 2023-10-30T13:57:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f498a9a7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49841538f0 0x7f4984159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f498a9a7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f498415cd50 0x7f49840c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f498a9a7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498415d300 con 0x7f49840cefa0 2023-10-30T13:57:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f498a9a7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f498415d5e0 con 0x7f498415cd50 2023-10-30T13:57:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f498a9a7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498415d8c0 con 0x7f49841538f0 2023-10-30T13:57:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f49891a4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f498415cd50 0x7f49840c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f49891a4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f498415cd50 0x7f49840c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50440/0 (socket says 172.21.15.46:50440) 2023-10-30T13:57:57.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f49891a4640 1 -- 172.21.15.46:0/1847715487 learned_addr learned my addr 172.21.15.46:0/1847715487 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:57.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f49899a5640 1 --2- 172.21.15.46:0/1847715487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49841538f0 0x7f4984159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:57.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f49891a4640 1 -- 172.21.15.46:0/1847715487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49841538f0 msgr2=0x7f4984159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:57:57.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f49891a4640 1 --2- 172.21.15.46:0/1847715487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49841538f0 0x7f4984159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:57.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f49891a4640 1 -- 172.21.15.46:0/1847715487 >> v1:172.21.15.154:6789/0 conn(0x7f49840cefa0 legacy=0x7f49840cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.920+0000 7f49891a4640 1 -- 172.21.15.46:0/1847715487 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498415dba0 con 0x7f498415cd50 2023-10-30T13:57:57.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.924+0000 7f49891a4640 1 --2- 172.21.15.46:0/1847715487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f498415cd50 0x7f49840c9f90 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f4974009870 tx=0x7f497401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f995c087c6e4c34 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:57.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.924+0000 7f49889a3640 1 -- 172.21.15.46:0/1847715487 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49740a8a30 con 0x7f498415cd50 2023-10-30T13:57:57.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.924+0000 7f49889a3640 1 -- 172.21.15.46:0/1847715487 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49740a8b90 con 0x7f498415cd50 2023-10-30T13:57:57.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.924+0000 7f49889a3640 1 -- 172.21.15.46:0/1847715487 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49740b0440 con 0x7f498415cd50 2023-10-30T13:57:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.924+0000 7f498a9a7640 1 -- 172.21.15.46:0/1847715487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f498415cd50 msgr2=0x7f49840c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.924+0000 7f498a9a7640 1 --2- 172.21.15.46:0/1847715487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f498415cd50 0x7f49840c9f90 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f4974009870 tx=0x7f497401ed30 comp rx=0 tx=0).stop 2023-10-30T13:57:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.924+0000 7f498a9a7640 1 -- 172.21.15.46:0/1847715487 shutdown_connections 2023-10-30T13:57:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.924+0000 7f498a9a7640 1 --2- 172.21.15.46:0/1847715487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f498415cd50 0x7f49840c9f90 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.924+0000 7f498a9a7640 1 --2- 172.21.15.46:0/1847715487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49841538f0 0x7f4984159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.924+0000 7f498a9a7640 1 -- 172.21.15.46:0/1847715487 >> 172.21.15.46:0/1847715487 conn(0x7f498406a640 msgr2=0x7f4984065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.924+0000 7f498a9a7640 1 -- 172.21.15.46:0/1847715487 shutdown_connections 2023-10-30T13:57:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.924+0000 7f498a9a7640 1 -- 172.21.15.46:0/1847715487 wait complete. 2023-10-30T13:57:57.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a9a7640 1 Processor -- start 2023-10-30T13:57:57.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a9a7640 1 -- start start 2023-10-30T13:57:57.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a9a7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49840cefa0 0x7f49841452e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:57.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a9a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49841538f0 0x7f49841f02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:57.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a9a7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f498415cd50 0x7f4984148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:57.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a9a7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498415b110 con 0x7f49841538f0 2023-10-30T13:57:57.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a9a7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49841f0830 con 0x7f49840cefa0 2023-10-30T13:57:57.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a9a7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49840b4260 con 0x7f498415cd50 2023-10-30T13:57:57.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f49891a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49841538f0 0x7f49841f02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:57.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f49891a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49841538f0 0x7f49841f02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50442/0 (socket says 172.21.15.46:50442) 2023-10-30T13:57:57.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f49891a4640 1 -- 172.21.15.46:0/646206221 learned_addr learned my addr 172.21.15.46:0/646206221 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:57:57.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f49899a5640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49840cefa0 0x7f49841452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a1a6640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f498415cd50 0x7f4984148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f49891a4640 1 -- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f498415cd50 msgr2=0x7f4984148a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f49891a4640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f498415cd50 0x7f4984148a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:57.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f49891a4640 1 -- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49840cefa0 msgr2=0x7f49841452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f49891a4640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49840cefa0 0x7f49841452e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f49891a4640 1 -- 172.21.15.46:0/646206221 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49740094d0 con 0x7f49841538f0 2023-10-30T13:57:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a1a6640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f498415cd50 0x7f4984148a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:57:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f49891a4640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49841538f0 0x7f49841f02f0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f49740b3040 tx=0x7f497401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f497a7fc640 1 -- 172.21.15.46:0/646206221 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49740b9620 con 0x7f49841538f0 2023-10-30T13:57:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f497a7fc640 1 -- 172.21.15.46:0/646206221 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49740b97a0 con 0x7f49841538f0 2023-10-30T13:57:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f497a7fc640 1 -- 172.21.15.46:0/646206221 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49740b9b50 con 0x7f49841538f0 2023-10-30T13:57:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a9a7640 1 -- 172.21.15.46:0/646206221 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49840b5230 con 0x7f49841538f0 2023-10-30T13:57:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a9a7640 1 -- 172.21.15.46:0/646206221 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498415aac0 con 0x7f49841538f0 2023-10-30T13:57:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.928+0000 7f498a9a7640 1 -- 172.21.15.46:0/646206221 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498415a920 con 0x7f49841538f0 2023-10-30T13:57:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.932+0000 7f497a7fc640 1 -- 172.21.15.46:0/646206221 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f49740b7070 con 0x7f49841538f0 2023-10-30T13:57:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.932+0000 7f497a7fc640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49680664d0 0x7f4968068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:57:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.932+0000 7f497a7fc640 1 -- 172.21.15.46:0/646206221 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f49741223f0 con 0x7f49841538f0 2023-10-30T13:57:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.932+0000 7f49899a5640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49680664d0 0x7f4968068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:57:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.932+0000 7f49899a5640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49680664d0 0x7f4968068990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f498000b9d0 tx=0x7f4980008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:57:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:57.936+0000 7f497a7fc640 1 -- 172.21.15.46:0/646206221 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f498415a920 con 0x7f49841538f0 2023-10-30T13:57:58.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.124+0000 7f498a9a7640 1 -- 172.21.15.46:0/646206221 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f498415a920 con 0x7f49841538f0 2023-10-30T13:57:58.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.124+0000 7f497a7fc640 1 -- 172.21.15.46:0/646206221 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f498415a920 con 0x7f49841538f0 2023-10-30T13:57:58.130 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:57:58.130 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:57:55.140760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:resolve","state_seq":23,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:57:58.130 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 153 2023-10-30T13:57:58.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.128+0000 7f498a9a7640 1 -- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49680664d0 msgr2=0x7f4968068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:58.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.128+0000 7f498a9a7640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49680664d0 0x7f4968068990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f498000b9d0 tx=0x7f4980008040 comp rx=0 tx=0).stop 2023-10-30T13:57:58.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.128+0000 7f498a9a7640 1 -- 172.21.15.46:0/646206221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49841538f0 msgr2=0x7f49841f02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:57:58.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.128+0000 7f498a9a7640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49841538f0 0x7f49841f02f0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f49740b3040 tx=0x7f497401bad0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.128+0000 7f498a9a7640 1 -- 172.21.15.46:0/646206221 shutdown_connections 2023-10-30T13:57:58.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.128+0000 7f498a9a7640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49680664d0 0x7f4968068990 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.128+0000 7f498a9a7640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f498415cd50 0x7f4984148a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.128+0000 7f498a9a7640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49841538f0 0x7f49841f02f0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.128+0000 7f498a9a7640 1 --2- 172.21.15.46:0/646206221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49840cefa0 0x7f49841452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:57:58.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.128+0000 7f498a9a7640 1 -- 172.21.15.46:0/646206221 >> 172.21.15.46:0/646206221 conn(0x7f498406a640 msgr2=0x7f49840ca090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:57:58.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.128+0000 7f498a9a7640 1 -- 172.21.15.46:0/646206221 shutdown_connections 2023-10-30T13:57:58.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:57:58.128+0000 7f498a9a7640 1 -- 172.21.15.46:0/646206221 wait complete. 2023-10-30T13:57:58.164 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 153, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 153, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7518": { "addr": "172.21.15.46:6833/157453225", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 157453225, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 157453225, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7518, "incarnation": 145, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_7704": { "addr": "172.21.15.46:6836/3734066164", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3734066164, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3734066164, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7704, "incarnation": 152, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:resolve", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:57:55.140760+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7704, "mds_1": 7518 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:58:00.165 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:00.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.316+0000 7f2a08f0f640 1 Processor -- start 2023-10-30T13:58:00.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.316+0000 7f2a08f0f640 1 -- start start 2023-10-30T13:58:00.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a08f0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a040c5f60 0x7f2a04071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a08f0f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a040700b0 con 0x7f2a04153bf0 2023-10-30T13:58:00.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a08f0f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a040c6530 con 0x7f2a040c6fe0 2023-10-30T13:58:00.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a08f0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a040c6810 con 0x7f2a040c5f60 2023-10-30T13:58:00.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a0259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2a04153bf0 0x7f2a040c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60542/0 (socket says 172.21.15.46:60542) 2023-10-30T13:58:00.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a0259c640 1 -- 172.21.15.46:0/3697002453 learned_addr learned my addr 172.21.15.46:0/3697002453 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:58:00.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a02d9d640 1 --2- 172.21.15.46:0/3697002453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a040c5f60 0x7f2a04071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781139302 0 0) 0x7f2a040700b0 con 0x7f2a04153bf0 2023-10-30T13:58:00.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f0003530 con 0x7f2a04153bf0 2023-10-30T13:58:00.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3780934084 0 0) 0x7f29f0003530 con 0x7f2a04153bf0 2023-10-30T13:58:00.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a040c5f60 msgr2=0x7f2a04071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:00.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 --2- 172.21.15.46:0/3697002453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a040c5f60 0x7f2a04071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 >> v1:172.21.15.154:6789/0 conn(0x7f2a040c6fe0 legacy=0x7f2a040cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a04159b10 con 0x7f2a04153bf0 2023-10-30T13:58:00.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f29f4001680 con 0x7f2a04153bf0 2023-10-30T13:58:00.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 >> v1:172.21.15.46:6789/0 conn(0x7f2a04153bf0 legacy=0x7f2a040c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 --2- 172.21.15.46:0/3697002453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f00039b0 0x7f29f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 --2- 172.21.15.46:0/3697002453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f00047d0 0x7f29f0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 --2- 172.21.15.46:0/3697002453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a040c5f60 0x7f29f0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a040c6810 con 0x7f29f00047d0 2023-10-30T13:58:00.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f000bad0 con 0x7f29f00039b0 2023-10-30T13:58:00.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f000bc90 con 0x7f2a040c5f60 2023-10-30T13:58:00.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f29fbfff640 1 --2- 172.21.15.46:0/3697002453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f00047d0 0x7f29f0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a0259c640 1 --2- 172.21.15.46:0/3697002453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f00039b0 0x7f29f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f2a02d9d640 1 --2- 172.21.15.46:0/3697002453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a040c5f60 0x7f29f0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f29fbfff640 1 -- 172.21.15.46:0/3697002453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a040c5f60 msgr2=0x7f29f0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:00.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f29fbfff640 1 --2- 172.21.15.46:0/3697002453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a040c5f60 0x7f29f0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f29fbfff640 1 -- 172.21.15.46:0/3697002453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f00039b0 msgr2=0x7f29f0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:00.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f29fbfff640 1 --2- 172.21.15.46:0/3697002453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f00039b0 0x7f29f0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.320+0000 7f29fbfff640 1 -- 172.21.15.46:0/3697002453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f0003530 con 0x7f29f00047d0 2023-10-30T13:58:00.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f29fbfff640 1 --2- 172.21.15.46:0/3697002453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f00047d0 0x7f29f0004c50 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f29ec001750 tx=0x7f29ec01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f70ae6bbfdc044 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:00.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0052d0 con 0x7f29f00047d0 2023-10-30T13:58:00.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec0a7050 con 0x7f29f00047d0 2023-10-30T13:58:00.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f2a01d9b640 1 -- 172.21.15.46:0/3697002453 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec003600 con 0x7f29f00047d0 2023-10-30T13:58:00.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f2a08f0f640 1 -- 172.21.15.46:0/3697002453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f00047d0 msgr2=0x7f29f0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f2a08f0f640 1 --2- 172.21.15.46:0/3697002453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f00047d0 0x7f29f0004c50 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f29ec001750 tx=0x7f29ec01c3a0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f2a02d9d640 1 -- 172.21.15.46:0/3697002453 reap_dead start 2023-10-30T13:58:00.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f2a08f0f640 1 -- 172.21.15.46:0/3697002453 shutdown_connections 2023-10-30T13:58:00.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f2a08f0f640 1 -- 172.21.15.46:0/3697002453 >> 172.21.15.46:0/3697002453 conn(0x7f2a0406c6e0 msgr2=0x7f2a04069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:00.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f2a08f0f640 1 -- 172.21.15.46:0/3697002453 shutdown_connections 2023-10-30T13:58:00.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f2a08f0f640 1 -- 172.21.15.46:0/3697002453 wait complete. 2023-10-30T13:58:00.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f2a08f0f640 1 Processor -- start 2023-10-30T13:58:00.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.324+0000 7f2a08f0f640 1 -- start start 2023-10-30T13:58:00.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f2a08f0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f00047d0 0x7f2a0414f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f2a08f0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0414b370 0x7f2a041e4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f2a08f0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a041e5140 0x7f2a040c3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f2a08f0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a040c3570 con 0x7f2a0414b370 2023-10-30T13:58:00.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f2a08f0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0414bd30 con 0x7f2a041e5140 2023-10-30T13:58:00.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f2a08f0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0414fee0 con 0x7f29f00047d0 2023-10-30T13:58:00.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f29fbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0414b370 0x7f2a041e4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f29fbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0414b370 0x7f2a041e4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50464/0 (socket says 172.21.15.46:50464) 2023-10-30T13:58:00.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f29fbfff640 1 -- 172.21.15.46:0/2884574438 learned_addr learned my addr 172.21.15.46:0/2884574438 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:00.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f2a0259c640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f00047d0 0x7f2a0414f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f2a02d9d640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a041e5140 0x7f2a040c3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f29fbfff640 1 -- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f00047d0 msgr2=0x7f2a0414f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f29fbfff640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f00047d0 0x7f2a0414f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f29fbfff640 1 -- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a041e5140 msgr2=0x7f2a040c3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f29fbfff640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a041e5140 0x7f2a040c3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f29fbfff640 1 -- 172.21.15.46:0/2884574438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0414fc40 con 0x7f2a0414b370 2023-10-30T13:58:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f2a0259c640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f00047d0 0x7f2a0414f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:58:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f29fbfff640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0414b370 0x7f2a041e4c00 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f29ec001750 tx=0x7f29f0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f29faffd640 1 -- 172.21.15.46:0/2884574438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec007a40 con 0x7f2a0414b370 2023-10-30T13:58:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f29faffd640 1 -- 172.21.15.46:0/2884574438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec007680 con 0x7f2a0414b370 2023-10-30T13:58:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f2a08f0f640 1 -- 172.21.15.46:0/2884574438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a041e5960 con 0x7f2a0414b370 2023-10-30T13:58:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f29faffd640 1 -- 172.21.15.46:0/2884574438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec006a20 con 0x7f2a0414b370 2023-10-30T13:58:00.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f2a08f0f640 1 -- 172.21.15.46:0/2884574438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a040c4c00 con 0x7f2a0414b370 2023-10-30T13:58:00.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.328+0000 7f2a08f0f640 1 -- 172.21.15.46:0/2884574438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a040c36b0 con 0x7f2a0414b370 2023-10-30T13:58:00.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.332+0000 7f29faffd640 1 -- 172.21.15.46:0/2884574438 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29ec006b80 con 0x7f2a0414b370 2023-10-30T13:58:00.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.332+0000 7f29faffd640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29e40664a0 0x7f29e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:00.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.332+0000 7f29faffd640 1 -- 172.21.15.46:0/2884574438 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f29ec11ed70 con 0x7f2a0414b370 2023-10-30T13:58:00.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.332+0000 7f2a0259c640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29e40664a0 0x7f29e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.332+0000 7f2a0259c640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29e40664a0 0x7f29e4068960 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f29f4000fc0 tx=0x7f29f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:00.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.336+0000 7f29faffd640 1 -- 172.21.15.46:0/2884574438 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2a040c36b0 con 0x7f2a0414b370 2023-10-30T13:58:00.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.532+0000 7f2a08f0f640 1 -- 172.21.15.46:0/2884574438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a040c36b0 con 0x7f2a0414b370 2023-10-30T13:58:00.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.532+0000 7f29faffd640 1 -- 172.21.15.46:0/2884574438 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f2a040c36b0 con 0x7f2a0414b370 2023-10-30T13:58:00.539 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:58:00.539 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:57:55.140760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:resolve","state_seq":23,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:00.539 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 153 2023-10-30T13:58:00.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.536+0000 7f2a08f0f640 1 -- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29e40664a0 msgr2=0x7f29e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.536+0000 7f2a08f0f640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29e40664a0 0x7f29e4068960 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f29f4000fc0 tx=0x7f29f4009040 comp rx=0 tx=0).stop 2023-10-30T13:58:00.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.536+0000 7f2a08f0f640 1 -- 172.21.15.46:0/2884574438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0414b370 msgr2=0x7f2a041e4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:00.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.536+0000 7f2a08f0f640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0414b370 0x7f2a041e4c00 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f29ec001750 tx=0x7f29f0005800 comp rx=0 tx=0).stop 2023-10-30T13:58:00.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.536+0000 7f2a08f0f640 1 -- 172.21.15.46:0/2884574438 shutdown_connections 2023-10-30T13:58:00.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.536+0000 7f2a08f0f640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29e40664a0 0x7f29e4068960 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.536+0000 7f2a08f0f640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a041e5140 0x7f2a040c3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.536+0000 7f2a08f0f640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a0414b370 0x7f2a041e4c00 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.536+0000 7f2a08f0f640 1 --2- 172.21.15.46:0/2884574438 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f00047d0 0x7f2a0414f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:00.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.536+0000 7f2a08f0f640 1 -- 172.21.15.46:0/2884574438 >> 172.21.15.46:0/2884574438 conn(0x7f2a0406c6e0 msgr2=0x7f2a040c7450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:00.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.536+0000 7f2a08f0f640 1 -- 172.21.15.46:0/2884574438 shutdown_connections 2023-10-30T13:58:00.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:00.536+0000 7f2a08f0f640 1 -- 172.21.15.46:0/2884574438 wait complete. 2023-10-30T13:58:00.559 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:58:02.560 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.708+0000 7ff37f11e640 1 Processor -- start 2023-10-30T13:58:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37f11e640 1 -- start start 2023-10-30T13:58:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37f11e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff37815cd50 0x7ff3780c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:02.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37f11e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff3780cefa0 0x7ff3780cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:02.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37f11e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37815a6c0 con 0x7ff3781538f0 2023-10-30T13:58:02.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37f11e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3780cf8f0 con 0x7ff37815cd50 2023-10-30T13:58:02.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37f11e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff37815d300 con 0x7ff3780cefa0 2023-10-30T13:58:02.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37e11c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff3781538f0 0x7ff378159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60550/0 (socket says 172.21.15.46:60550) 2023-10-30T13:58:02.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37e11c640 1 -- 172.21.15.46:0/858713670 learned_addr learned my addr 172.21.15.46:0/858713670 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:58:02.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37d91b640 1 --2- 172.21.15.46:0/858713670 >> v2:172.21.15.46:3300/0 conn(0x7ff37815cd50 0x7ff3780c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:02.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37e91d640 1 --2- 172.21.15.46:0/858713670 >> v2:172.21.15.154:3300/0 conn(0x7ff3780cefa0 0x7ff3780cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:02.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37d11a640 1 -- 172.21.15.46:0/858713670 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605109747 0 0) 0x7ff37815a6c0 con 0x7ff3781538f0 2023-10-30T13:58:02.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37d11a640 1 -- 172.21.15.46:0/858713670 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff364003530 con 0x7ff3781538f0 2023-10-30T13:58:02.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37d91b640 1 -- 172.21.15.46:0/858713670 >> v2:172.21.15.154:3300/0 conn(0x7ff3780cefa0 msgr2=0x7ff3780cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:02.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37d91b640 1 --2- 172.21.15.46:0/858713670 >> v2:172.21.15.154:3300/0 conn(0x7ff3780cefa0 0x7ff3780cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37d91b640 1 -- 172.21.15.46:0/858713670 >> v1:172.21.15.46:6789/0 conn(0x7ff3781538f0 legacy=0x7ff378159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:02.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.712+0000 7ff37d91b640 1 -- 172.21.15.46:0/858713670 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37815d5e0 con 0x7ff37815cd50 2023-10-30T13:58:02.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37d91b640 1 --2- 172.21.15.46:0/858713670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff37815cd50 0x7ff3780c9f90 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7ff378066a40 tx=0x7ff36801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5d124b5f449b14 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:02.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37d11a640 1 -- 172.21.15.46:0/858713670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3680a8950 con 0x7ff37815cd50 2023-10-30T13:58:02.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37d11a640 1 -- 172.21.15.46:0/858713670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3680a8ab0 con 0x7ff37815cd50 2023-10-30T13:58:02.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37d11a640 1 -- 172.21.15.46:0/858713670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3680a8e40 con 0x7ff37815cd50 2023-10-30T13:58:02.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37f11e640 1 -- 172.21.15.46:0/858713670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff37815cd50 msgr2=0x7ff3780c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:02.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37f11e640 1 --2- 172.21.15.46:0/858713670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff37815cd50 0x7ff3780c9f90 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7ff378066a40 tx=0x7ff36801ec50 comp rx=0 tx=0).stop 2023-10-30T13:58:02.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37f11e640 1 -- 172.21.15.46:0/858713670 shutdown_connections 2023-10-30T13:58:02.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37f11e640 1 --2- 172.21.15.46:0/858713670 >> v2:172.21.15.154:3300/0 conn(0x7ff3780cefa0 0x7ff3780cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37f11e640 1 --2- 172.21.15.46:0/858713670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff37815cd50 0x7ff3780c9f90 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37f11e640 1 -- 172.21.15.46:0/858713670 >> 172.21.15.46:0/858713670 conn(0x7ff37806a640 msgr2=0x7ff378065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:02.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37f11e640 1 -- 172.21.15.46:0/858713670 shutdown_connections 2023-10-30T13:58:02.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37f11e640 1 -- 172.21.15.46:0/858713670 wait complete. 2023-10-30T13:58:02.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.716+0000 7ff37f11e640 1 Processor -- start 2023-10-30T13:58:02.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37f11e640 1 -- start start 2023-10-30T13:58:02.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37f11e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3780cefa0 0x7ff37815be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:02.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37f11e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3781538f0 0x7ff3780b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:02.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37f11e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff37815cd50 0x7ff3780b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:02.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37f11e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3780b4c60 con 0x7ff3780cefa0 2023-10-30T13:58:02.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37f11e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff378157440 con 0x7ff37815cd50 2023-10-30T13:58:02.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37f11e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff378157720 con 0x7ff3781538f0 2023-10-30T13:58:02.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37e11c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3780cefa0 0x7ff37815be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:02.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37e11c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3780cefa0 0x7ff37815be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50478/0 (socket says 172.21.15.46:50478) 2023-10-30T13:58:02.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37e11c640 1 -- 172.21.15.46:0/3779565872 learned_addr learned my addr 172.21.15.46:0/3779565872 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:02.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37d91b640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3781538f0 0x7ff3780b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:02.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37e91d640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff37815cd50 0x7ff3780b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:02.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37e11c640 1 -- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3781538f0 msgr2=0x7ff3780b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:02.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37e11c640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3781538f0 0x7ff3780b34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37e11c640 1 -- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff37815cd50 msgr2=0x7ff3780b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:02.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37e11c640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff37815cd50 0x7ff3780b39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37e11c640 1 -- 172.21.15.46:0/3779565872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff368009420 con 0x7ff3780cefa0 2023-10-30T13:58:02.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37e91d640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff37815cd50 0x7ff3780b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:58:02.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37e11c640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3780cefa0 0x7ff37815be20 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7ff37400c4d0 tx=0x7ff37400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:02.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff36effd640 1 -- 172.21.15.46:0/3779565872 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374005cf0 con 0x7ff3780cefa0 2023-10-30T13:58:02.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff36effd640 1 -- 172.21.15.46:0/3779565872 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff374005e50 con 0x7ff3780cefa0 2023-10-30T13:58:02.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff36effd640 1 -- 172.21.15.46:0/3779565872 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374007270 con 0x7ff3780cefa0 2023-10-30T13:58:02.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37f11e640 1 -- 172.21.15.46:0/3779565872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff378157a00 con 0x7ff3780cefa0 2023-10-30T13:58:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37f11e640 1 -- 172.21.15.46:0/3779565872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff378157fc0 con 0x7ff3780cefa0 2023-10-30T13:58:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.720+0000 7ff37f11e640 1 -- 172.21.15.46:0/3779565872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3780b31d0 con 0x7ff3780cefa0 2023-10-30T13:58:02.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.724+0000 7ff36effd640 1 -- 172.21.15.46:0/3779565872 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff3740073d0 con 0x7ff3780cefa0 2023-10-30T13:58:02.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.724+0000 7ff36effd640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff35c0664a0 0x7ff35c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:02.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.724+0000 7ff36effd640 1 -- 172.21.15.46:0/3779565872 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7ff374083a70 con 0x7ff3780cefa0 2023-10-30T13:58:02.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.724+0000 7ff37d91b640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff35c0664a0 0x7ff35c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.724+0000 7ff37d91b640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff35c0664a0 0x7ff35c068960 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff3680b2000 tx=0x7ff368008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.728+0000 7ff36effd640 1 -- 172.21.15.46:0/3779565872 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff3780b31d0 con 0x7ff3780cefa0 2023-10-30T13:58:02.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 -- 172.21.15.46:0/3779565872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3780b31d0 con 0x7ff3780cefa0 2023-10-30T13:58:02.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff36effd640 1 -- 172.21.15.46:0/3779565872 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5733 (secure 0 0 0) 0x7ff3780b31d0 con 0x7ff3780cefa0 2023-10-30T13:58:02.924 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:58:02.924 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:00.459468+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:02.925 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 154 2023-10-30T13:58:02.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 -- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff35c0664a0 msgr2=0x7ff35c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:02.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff35c0664a0 0x7ff35c068960 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff3680b2000 tx=0x7ff368008040 comp rx=0 tx=0).stop 2023-10-30T13:58:02.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 -- 172.21.15.46:0/3779565872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3780cefa0 msgr2=0x7ff37815be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:02.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3780cefa0 0x7ff37815be20 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7ff37400c4d0 tx=0x7ff37400c990 comp rx=0 tx=0).stop 2023-10-30T13:58:02.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 -- 172.21.15.46:0/3779565872 shutdown_connections 2023-10-30T13:58:02.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff35c0664a0 0x7ff35c068960 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff37815cd50 0x7ff3780b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3781538f0 0x7ff3780b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 --2- 172.21.15.46:0/3779565872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3780cefa0 0x7ff37815be20 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:02.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 -- 172.21.15.46:0/3779565872 >> 172.21.15.46:0/3779565872 conn(0x7ff37806a640 msgr2=0x7ff378148920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:02.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 -- 172.21.15.46:0/3779565872 shutdown_connections 2023-10-30T13:58:02.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:02.920+0000 7ff37f11e640 1 -- 172.21.15.46:0/3779565872 wait complete. 2023-10-30T13:58:02.945 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:58:04.947 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:05.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.096+0000 7f75a93d3640 1 Processor -- start 2023-10-30T13:58:05.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.096+0000 7f75a93d3640 1 -- start start 2023-10-30T13:58:05.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.096+0000 7f75a93d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f75a406f7a0 0x7f75a406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:05.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.096+0000 7f75a93d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75a4071110 0x7f75a4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:05.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.096+0000 7f75a93d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a4066cf0 con 0x7f75a40c6bd0 2023-10-30T13:58:05.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.096+0000 7f75a93d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a40700c0 con 0x7f75a406f7a0 2023-10-30T13:58:05.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.096+0000 7f75a93d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40c70d0 con 0x7f75a4071110 2023-10-30T13:58:05.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f75a2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f75a406f7a0 0x7f75a406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:05.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f759bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f75a40c6bd0 0x7f75a4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60560/0 (socket says 172.21.15.46:60560) 2023-10-30T13:58:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f759bfff640 1 -- 172.21.15.46:0/2302059071 learned_addr learned my addr 172.21.15.46:0/2302059071 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:58:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f75a37fe640 1 --2- 172.21.15.46:0/2302059071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75a4071110 0x7f75a4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f75a2ffd640 1 -- 172.21.15.46:0/2302059071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75a4071110 msgr2=0x7f75a4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f75a2ffd640 1 --2- 172.21.15.46:0/2302059071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75a4071110 0x7f75a4158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f75a2ffd640 1 -- 172.21.15.46:0/2302059071 >> v1:172.21.15.46:6789/0 conn(0x7f75a40c6bd0 legacy=0x7f75a4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f75a2ffd640 1 -- 172.21.15.46:0/2302059071 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a40c7390 con 0x7f75a406f7a0 2023-10-30T13:58:05.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f75a27fc640 1 -- 172.21.15.46:0/2302059071 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612978203 0 0) 0x7f75a4066cf0 con 0x7f75a40c6bd0 2023-10-30T13:58:05.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f75a2ffd640 1 --2- 172.21.15.46:0/2302059071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75a406f7a0 0x7f75a406fb80 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f75940097c0 tx=0x7f759401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8542db373b82d722 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:05.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f75a27fc640 1 -- 172.21.15.46:0/2302059071 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940a8970 con 0x7f75a406f7a0 2023-10-30T13:58:05.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f75a27fc640 1 -- 172.21.15.46:0/2302059071 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75940a8ad0 con 0x7f75a406f7a0 2023-10-30T13:58:05.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.100+0000 7f75a27fc640 1 -- 172.21.15.46:0/2302059071 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940a8e60 con 0x7f75a406f7a0 2023-10-30T13:58:05.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 -- 172.21.15.46:0/2302059071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75a406f7a0 msgr2=0x7f75a406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 --2- 172.21.15.46:0/2302059071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75a406f7a0 0x7f75a406fb80 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f75940097c0 tx=0x7f759401ec70 comp rx=0 tx=0).stop 2023-10-30T13:58:05.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 -- 172.21.15.46:0/2302059071 shutdown_connections 2023-10-30T13:58:05.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 --2- 172.21.15.46:0/2302059071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75a4071110 0x7f75a4158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 --2- 172.21.15.46:0/2302059071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75a406f7a0 0x7f75a406fb80 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 -- 172.21.15.46:0/2302059071 >> 172.21.15.46:0/2302059071 conn(0x7f75a40c7910 msgr2=0x7f75a40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:05.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 -- 172.21.15.46:0/2302059071 shutdown_connections 2023-10-30T13:58:05.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 -- 172.21.15.46:0/2302059071 wait complete. 2023-10-30T13:58:05.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 Processor -- start 2023-10-30T13:58:05.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 -- start start 2023-10-30T13:58:05.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75a406f7a0 0x7f75a41f1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:05.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75a4071110 0x7f75a40c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:05.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75a40c6bd0 0x7f75a414f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:05.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40c10a0 con 0x7f75a4071110 2023-10-30T13:58:05.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40c1330 con 0x7f75a40c6bd0 2023-10-30T13:58:05.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a93d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40c1b90 con 0x7f75a406f7a0 2023-10-30T13:58:05.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f759bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75a4071110 0x7f75a40c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:05.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f759bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75a4071110 0x7f75a40c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50492/0 (socket says 172.21.15.46:50492) 2023-10-30T13:58:05.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f759bfff640 1 -- 172.21.15.46:0/1596214562 learned_addr learned my addr 172.21.15.46:0/1596214562 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:05.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a2ffd640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75a406f7a0 0x7f75a41f1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:05.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a37fe640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75a40c6bd0 0x7f75a414f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:05.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f759bfff640 1 -- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75a406f7a0 msgr2=0x7f75a41f1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f759bfff640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75a406f7a0 0x7f75a41f1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f759bfff640 1 -- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75a40c6bd0 msgr2=0x7f75a414f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f759bfff640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75a40c6bd0 0x7f75a414f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f759bfff640 1 -- 172.21.15.46:0/1596214562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7594009420 con 0x7f75a4071110 2023-10-30T13:58:05.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.104+0000 7f75a2ffd640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75a406f7a0 0x7f75a41f1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:58:05.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.108+0000 7f759bfff640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75a4071110 0x7f75a40c5760 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f758c00c4b0 tx=0x7f758c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:05.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.108+0000 7f759b7fe640 1 -- 172.21.15.46:0/1596214562 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c0074e0 con 0x7f75a4071110 2023-10-30T13:58:05.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.108+0000 7f759b7fe640 1 -- 172.21.15.46:0/1596214562 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758c007660 con 0x7f75a4071110 2023-10-30T13:58:05.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.108+0000 7f759b7fe640 1 -- 172.21.15.46:0/1596214562 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c004a30 con 0x7f75a4071110 2023-10-30T13:58:05.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.108+0000 7f75a93d3640 1 -- 172.21.15.46:0/1596214562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a40c0ae0 con 0x7f75a4071110 2023-10-30T13:58:05.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.108+0000 7f75a93d3640 1 -- 172.21.15.46:0/1596214562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a40c1540 con 0x7f75a4071110 2023-10-30T13:58:05.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.108+0000 7f75a93d3640 1 -- 172.21.15.46:0/1596214562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a40c1d20 con 0x7f75a4071110 2023-10-30T13:58:05.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.108+0000 7f759b7fe640 1 -- 172.21.15.46:0/1596214562 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f758c004b90 con 0x7f75a4071110 2023-10-30T13:58:05.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.108+0000 7f759b7fe640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75840664a0 0x7f7584068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:05.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.108+0000 7f759b7fe640 1 -- 172.21.15.46:0/1596214562 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f758c0838e0 con 0x7f75a4071110 2023-10-30T13:58:05.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.108+0000 7f75a2ffd640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75840664a0 0x7f7584068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:05.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.112+0000 7f75a2ffd640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75840664a0 0x7f7584068960 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f75940b2000 tx=0x7f7594008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:05.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.112+0000 7f759b7fe640 1 -- 172.21.15.46:0/1596214562 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f758c050650 con 0x7f75a4071110 2023-10-30T13:58:05.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.300+0000 7f75a93d3640 1 -- 172.21.15.46:0/1596214562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75a404e170 con 0x7f75a4071110 2023-10-30T13:58:05.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f759b7fe640 1 -- 172.21.15.46:0/1596214562 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f758c018070 con 0x7f75a4071110 2023-10-30T13:58:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:58:05.308 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:00.459468+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:05.308 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 154 2023-10-30T13:58:05.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f75a93d3640 1 -- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75840664a0 msgr2=0x7f7584068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f75a93d3640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75840664a0 0x7f7584068960 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f75940b2000 tx=0x7f7594008040 comp rx=0 tx=0).stop 2023-10-30T13:58:05.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f75a93d3640 1 -- 172.21.15.46:0/1596214562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75a4071110 msgr2=0x7f75a40c5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:05.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f75a93d3640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75a4071110 0x7f75a40c5760 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f758c00c4b0 tx=0x7f758c00c970 comp rx=0 tx=0).stop 2023-10-30T13:58:05.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f75a93d3640 1 -- 172.21.15.46:0/1596214562 shutdown_connections 2023-10-30T13:58:05.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f75a93d3640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75840664a0 0x7f7584068960 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f75a93d3640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75a40c6bd0 0x7f75a414f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f75a93d3640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75a4071110 0x7f75a40c5760 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f75a93d3640 1 --2- 172.21.15.46:0/1596214562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75a406f7a0 0x7f75a41f1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:05.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f75a93d3640 1 -- 172.21.15.46:0/1596214562 >> 172.21.15.46:0/1596214562 conn(0x7f75a40c7910 msgr2=0x7f75a40c6380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:05.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f75a93d3640 1 -- 172.21.15.46:0/1596214562 shutdown_connections 2023-10-30T13:58:05.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:05.304+0000 7f75a93d3640 1 -- 172.21.15.46:0/1596214562 wait complete. 2023-10-30T13:58:05.329 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:58:07.330 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:07.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.480+0000 7f2cbad3e640 1 Processor -- start 2023-10-30T13:58:07.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.480+0000 7f2cbad3e640 1 -- start start 2023-10-30T13:58:07.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.480+0000 7f2cbad3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2cb40c6b80 0x7f2cb40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:07.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.480+0000 7f2cbad3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb413f4a0 0x7f2cb40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:07.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.480+0000 7f2cbad3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb40c0a50 con 0x7f2cb40c6f60 2023-10-30T13:58:07.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.480+0000 7f2cbad3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb406f6e0 con 0x7f2cb40c6b80 2023-10-30T13:58:07.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.480+0000 7f2cbad3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb406f9c0 con 0x7f2cb413f4a0 2023-10-30T13:58:07.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.484+0000 7f2cb8ada640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2cb40c6b80 0x7f2cb40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:07.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.484+0000 7f2cb8ada640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2cb40c6b80 0x7f2cb40bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54114/0 (socket says 172.21.15.46:54114) 2023-10-30T13:58:07.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.484+0000 7f2cb8ada640 1 -- 172.21.15.46:0/1247283924 learned_addr learned my addr 172.21.15.46:0/1247283924 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:07.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.484+0000 7f2cb92db640 1 --2- 172.21.15.46:0/1247283924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb413f4a0 0x7f2cb40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:07.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.484+0000 7f2cb8ada640 1 -- 172.21.15.46:0/1247283924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb413f4a0 msgr2=0x7f2cb40c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:07.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.484+0000 7f2cb8ada640 1 --2- 172.21.15.46:0/1247283924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb413f4a0 0x7f2cb40c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.484+0000 7f2cb8ada640 1 -- 172.21.15.46:0/1247283924 >> v1:172.21.15.46:6789/0 conn(0x7f2cb40c6f60 legacy=0x7f2cb413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.484+0000 7f2cb8ada640 1 -- 172.21.15.46:0/1247283924 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb406fca0 con 0x7f2cb40c6b80 2023-10-30T13:58:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.484+0000 7f2cb92db640 1 --2- 172.21.15.46:0/1247283924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb413f4a0 0x7f2cb40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:58:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.484+0000 7f2cb37fe640 1 -- 172.21.15.46:0/1247283924 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173091480 0 0) 0x7f2cb40c0a50 con 0x7f2cb40c6f60 2023-10-30T13:58:07.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.484+0000 7f2cb8ada640 1 --2- 172.21.15.46:0/1247283924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cb40c6b80 0x7f2cb40bfec0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009870 tx=0x7f2ca801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50450e288a7166f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:07.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cb37fe640 1 -- 172.21.15.46:0/1247283924 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca80ab650 con 0x7f2cb40c6b80 2023-10-30T13:58:07.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cb37fe640 1 -- 172.21.15.46:0/1247283924 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca80ab7b0 con 0x7f2cb40c6b80 2023-10-30T13:58:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cb37fe640 1 -- 172.21.15.46:0/1247283924 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca80abb40 con 0x7f2cb40c6b80 2023-10-30T13:58:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 -- 172.21.15.46:0/1247283924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cb40c6b80 msgr2=0x7f2cb40bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:07.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 --2- 172.21.15.46:0/1247283924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cb40c6b80 0x7f2cb40bfec0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009870 tx=0x7f2ca801c540 comp rx=0 tx=0).stop 2023-10-30T13:58:07.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 -- 172.21.15.46:0/1247283924 shutdown_connections 2023-10-30T13:58:07.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 --2- 172.21.15.46:0/1247283924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb413f4a0 0x7f2cb40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 --2- 172.21.15.46:0/1247283924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cb40c6b80 0x7f2cb40bfec0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 -- 172.21.15.46:0/1247283924 >> 172.21.15.46:0/1247283924 conn(0x7f2cb40c7910 msgr2=0x7f2cb40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:07.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 -- 172.21.15.46:0/1247283924 shutdown_connections 2023-10-30T13:58:07.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 -- 172.21.15.46:0/1247283924 wait complete. 2023-10-30T13:58:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 Processor -- start 2023-10-30T13:58:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 -- start start 2023-10-30T13:58:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cb40c6b80 0x7f2cb4155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb40c6f60 0x7f2cb4158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cb413f4a0 0x7f2cb41f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb4153ad0 con 0x7f2cb413f4a0 2023-10-30T13:58:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb41f1450 con 0x7f2cb40c6b80 2023-10-30T13:58:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.488+0000 7f2cbad3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb4156160 con 0x7f2cb40c6f60 2023-10-30T13:58:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb92db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cb413f4a0 0x7f2cb41f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb92db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cb413f4a0 0x7f2cb41f1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54116/0 (socket says 172.21.15.46:54116) 2023-10-30T13:58:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb92db640 1 -- 172.21.15.46:0/4286139800 learned_addr learned my addr 172.21.15.46:0/4286139800 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:07.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb3fff640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb40c6f60 0x7f2cb4158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:07.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb8ada640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cb40c6b80 0x7f2cb4155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:07.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb92db640 1 -- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb40c6f60 msgr2=0x7f2cb4158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:07.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb92db640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb40c6f60 0x7f2cb4158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb92db640 1 -- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cb40c6b80 msgr2=0x7f2cb4155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:07.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb92db640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cb40c6b80 0x7f2cb4155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb92db640 1 -- 172.21.15.46:0/4286139800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca80094d0 con 0x7f2cb413f4a0 2023-10-30T13:58:07.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb3fff640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb40c6f60 0x7f2cb4158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:58:07.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb92db640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cb413f4a0 0x7f2cb41f1c10 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f2cac007df0 tx=0x7f2cac00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:07.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb17fa640 1 -- 172.21.15.46:0/4286139800 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac0039a0 con 0x7f2cb413f4a0 2023-10-30T13:58:07.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb17fa640 1 -- 172.21.15.46:0/4286139800 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cac00f040 con 0x7f2cb413f4a0 2023-10-30T13:58:07.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cbad3e640 1 -- 172.21.15.46:0/4286139800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb41545d0 con 0x7f2cb413f4a0 2023-10-30T13:58:07.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb17fa640 1 -- 172.21.15.46:0/4286139800 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac003d40 con 0x7f2cb413f4a0 2023-10-30T13:58:07.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cbad3e640 1 -- 172.21.15.46:0/4286139800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb4154b90 con 0x7f2cb413f4a0 2023-10-30T13:58:07.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cbad3e640 1 -- 172.21.15.46:0/4286139800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb404e170 con 0x7f2cb413f4a0 2023-10-30T13:58:07.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.492+0000 7f2cb17fa640 1 -- 172.21.15.46:0/4286139800 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2cac004030 con 0x7f2cb413f4a0 2023-10-30T13:58:07.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.496+0000 7f2cb17fa640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c980664a0 0x7f2c98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:07.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.496+0000 7f2cb17fa640 1 -- 172.21.15.46:0/4286139800 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f2cac084760 con 0x7f2cb413f4a0 2023-10-30T13:58:07.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.496+0000 7f2cb8ada640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c980664a0 0x7f2c98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:07.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.496+0000 7f2cb8ada640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c980664a0 0x7f2c98068960 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2ca80af000 tx=0x7f2ca80b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:07.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.500+0000 7f2cb17fa640 1 -- 172.21.15.46:0/4286139800 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2cac0515e0 con 0x7f2cb413f4a0 2023-10-30T13:58:07.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 -- 172.21.15.46:0/4286139800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cb413b890 con 0x7f2cb413f4a0 2023-10-30T13:58:07.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cb17fa640 1 -- 172.21.15.46:0/4286139800 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f2cac054c00 con 0x7f2cb413f4a0 2023-10-30T13:58:07.700 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:58:07.700 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:06.439313+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:07.701 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 155 2023-10-30T13:58:07.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 -- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c980664a0 msgr2=0x7f2c98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:07.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c980664a0 0x7f2c98068960 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2ca80af000 tx=0x7f2ca80b6040 comp rx=0 tx=0).stop 2023-10-30T13:58:07.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 -- 172.21.15.46:0/4286139800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cb413f4a0 msgr2=0x7f2cb41f1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cb413f4a0 0x7f2cb41f1c10 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f2cac007df0 tx=0x7f2cac00d4a0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 -- 172.21.15.46:0/4286139800 shutdown_connections 2023-10-30T13:58:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2c980664a0 0x7f2c98068960 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cb413f4a0 0x7f2cb41f1c10 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb40c6f60 0x7f2cb4158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 --2- 172.21.15.46:0/4286139800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cb40c6b80 0x7f2cb4155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 -- 172.21.15.46:0/4286139800 >> 172.21.15.46:0/4286139800 conn(0x7f2cb40c7910 msgr2=0x7f2cb40c0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 -- 172.21.15.46:0/4286139800 shutdown_connections 2023-10-30T13:58:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:07.696+0000 7f2cbad3e640 1 -- 172.21.15.46:0/4286139800 wait complete. 2023-10-30T13:58:07.722 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:58:09.723 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.872+0000 7fb431cea640 1 Processor -- start 2023-10-30T13:58:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.872+0000 7fb431cea640 1 -- start start 2023-10-30T13:58:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb431cea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb42c0c6b80 0x7fb42c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb431cea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c13f4a0 0x7fb42c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb431cea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c0c0a50 con 0x7fb42c0c6f60 2023-10-30T13:58:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb431cea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42c06f6e0 con 0x7fb42c0c6b80 2023-10-30T13:58:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb431cea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c06f9c0 con 0x7fb42c13f4a0 2023-10-30T13:58:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb42c0c6f60 0x7fb42c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53976/0 (socket says 172.21.15.46:53976) 2023-10-30T13:58:09.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42affd640 1 -- 172.21.15.46:0/1097114009 learned_addr learned my addr 172.21.15.46:0/1097114009 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:58:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42b7fe640 1 --2- 172.21.15.46:0/1097114009 >> v2:172.21.15.154:3300/0 conn(0x7fb42c0c6b80 0x7fb42c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42bfff640 1 --2- 172.21.15.46:0/1097114009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c13f4a0 0x7fb42c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167229790 0 0) 0x7fb42c0c0a50 con 0x7fb42c0c6f60 2023-10-30T13:58:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb414003530 con 0x7fb42c0c6f60 2023-10-30T13:58:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2953391896 0 0) 0x7fb414003530 con 0x7fb42c0c6f60 2023-10-30T13:58:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c13f4a0 msgr2=0x7fb42c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 --2- 172.21.15.46:0/1097114009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c13f4a0 0x7fb42c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:09.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 >> v2:172.21.15.154:3300/0 conn(0x7fb42c0c6b80 msgr2=0x7fb42c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:09.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 --2- 172.21.15.46:0/1097114009 >> v2:172.21.15.154:3300/0 conn(0x7fb42c0c6b80 0x7fb42c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:09.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c06fca0 con 0x7fb42c0c6f60 2023-10-30T13:58:09.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fb420001680 con 0x7fb42c0c6f60 2023-10-30T13:58:09.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 >> v1:172.21.15.46:6789/0 conn(0x7fb42c0c6f60 legacy=0x7fb42c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:09.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 --2- 172.21.15.46:0/1097114009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb414003980 0x7fb414003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:09.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 --2- 172.21.15.46:0/1097114009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4140047d0 0x7fb414004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:09.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 --2- 172.21.15.46:0/1097114009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c13f4a0 0x7fb414005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:09.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c06f6e0 con 0x7fb414003980 2023-10-30T13:58:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c06f9c0 con 0x7fb4140047d0 2023-10-30T13:58:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41400dba0 con 0x7fb42c13f4a0 2023-10-30T13:58:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42b7fe640 1 --2- 172.21.15.46:0/1097114009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb414003980 0x7fb414003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42affd640 1 --2- 172.21.15.46:0/1097114009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4140047d0 0x7fb414004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42bfff640 1 --2- 172.21.15.46:0/1097114009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c13f4a0 0x7fb414005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42b7fe640 1 -- 172.21.15.46:0/1097114009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c13f4a0 msgr2=0x7fb414005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42b7fe640 1 --2- 172.21.15.46:0/1097114009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c13f4a0 0x7fb414005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42b7fe640 1 -- 172.21.15.46:0/1097114009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4140047d0 msgr2=0x7fb414004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42b7fe640 1 --2- 172.21.15.46:0/1097114009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4140047d0 0x7fb414004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.876+0000 7fb42b7fe640 1 -- 172.21.15.46:0/1097114009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb414003530 con 0x7fb414003980 2023-10-30T13:58:09.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.880+0000 7fb42b7fe640 1 --2- 172.21.15.46:0/1097114009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb414003980 0x7fb414003da0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009ae0 tx=0x7fb41c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78a5af4da8d79df server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:09.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.880+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c0a64a0 con 0x7fb414003980 2023-10-30T13:58:09.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.880+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c0a6600 con 0x7fb414003980 2023-10-30T13:58:09.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.880+0000 7fb42a7fc640 1 -- 172.21.15.46:0/1097114009 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c0a69b0 con 0x7fb414003980 2023-10-30T13:58:09.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.880+0000 7fb431cea640 1 -- 172.21.15.46:0/1097114009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb414003980 msgr2=0x7fb414003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.880+0000 7fb431cea640 1 --2- 172.21.15.46:0/1097114009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb414003980 0x7fb414003da0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009ae0 tx=0x7fb41c01ca90 comp rx=0 tx=0).stop 2023-10-30T13:58:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.880+0000 7fb42bfff640 1 -- 172.21.15.46:0/1097114009 reap_dead start 2023-10-30T13:58:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.880+0000 7fb431cea640 1 -- 172.21.15.46:0/1097114009 shutdown_connections 2023-10-30T13:58:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.880+0000 7fb431cea640 1 -- 172.21.15.46:0/1097114009 >> 172.21.15.46:0/1097114009 conn(0x7fb42c0c7910 msgr2=0x7fb42c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.880+0000 7fb431cea640 1 -- 172.21.15.46:0/1097114009 shutdown_connections 2023-10-30T13:58:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.880+0000 7fb431cea640 1 -- 172.21.15.46:0/1097114009 wait complete. 2023-10-30T13:58:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb431cea640 1 Processor -- start 2023-10-30T13:58:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb431cea640 1 -- start start 2023-10-30T13:58:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb431cea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb414003980 0x7fb42c13da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb431cea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c1e9a60 0x7fb42c0c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb431cea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb42c1f57a0 0x7fb42c1f5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb431cea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c13dfd0 con 0x7fb42c1f57a0 2023-10-30T13:58:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb431cea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c1f6160 con 0x7fb414003980 2023-10-30T13:58:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb431cea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c1f6410 con 0x7fb42c1e9a60 2023-10-30T13:58:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb42bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb42c1f57a0 0x7fb42c1f5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb42bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb42c1f57a0 0x7fb42c1f5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54126/0 (socket says 172.21.15.46:54126) 2023-10-30T13:58:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb42bfff640 1 -- 172.21.15.46:0/3696888064 learned_addr learned my addr 172.21.15.46:0/3696888064 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb42b7fe640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb414003980 0x7fb42c13da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb42affd640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c1e9a60 0x7fb42c0c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb42bfff640 1 -- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c1e9a60 msgr2=0x7fb42c0c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb42bfff640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c1e9a60 0x7fb42c0c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb42bfff640 1 -- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb414003980 msgr2=0x7fb42c13da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb42bfff640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb414003980 0x7fb42c13da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb42bfff640 1 -- 172.21.15.46:0/3696888064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c13a3e0 con 0x7fb42c1f57a0 2023-10-30T13:58:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb42b7fe640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb414003980 0x7fb42c13da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:58:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb42bfff640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb42c1f57a0 0x7fb42c1f5c20 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fb42c06f4d0 tx=0x7fb42400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb413fff640 1 -- 172.21.15.46:0/3696888064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42400e040 con 0x7fb42c1f57a0 2023-10-30T13:58:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb413fff640 1 -- 172.21.15.46:0/3696888064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42400f040 con 0x7fb42c1f57a0 2023-10-30T13:58:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb431cea640 1 -- 172.21.15.46:0/3696888064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c13a6c0 con 0x7fb42c1f57a0 2023-10-30T13:58:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb413fff640 1 -- 172.21.15.46:0/3696888064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb424015670 con 0x7fb42c1f57a0 2023-10-30T13:58:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb431cea640 1 -- 172.21.15.46:0/3696888064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c13e560 con 0x7fb42c1f57a0 2023-10-30T13:58:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.884+0000 7fb431cea640 1 -- 172.21.15.46:0/3696888064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c04e170 con 0x7fb42c1f57a0 2023-10-30T13:58:09.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.888+0000 7fb413fff640 1 -- 172.21.15.46:0/3696888064 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb4240157d0 con 0x7fb42c1f57a0 2023-10-30T13:58:09.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.888+0000 7fb413fff640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4180664a0 0x7fb418068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:09.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.888+0000 7fb413fff640 1 -- 172.21.15.46:0/3696888064 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7fb424084e30 con 0x7fb42c1f57a0 2023-10-30T13:58:09.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.888+0000 7fb42b7fe640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4180664a0 0x7fb418068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:09.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.888+0000 7fb42b7fe640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4180664a0 0x7fb418068960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0ad000 tx=0x7fb41c0b2040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:09.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:09.892+0000 7fb413fff640 1 -- 172.21.15.46:0/3696888064 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb424051ae0 con 0x7fb42c1f57a0 2023-10-30T13:58:10.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.084+0000 7fb431cea640 1 -- 172.21.15.46:0/3696888064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb42c0c7ec0 con 0x7fb42c1f57a0 2023-10-30T13:58:10.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.084+0000 7fb413fff640 1 -- 172.21.15.46:0/3696888064 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fb424055100 con 0x7fb42c1f57a0 2023-10-30T13:58:10.090 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:58:10.090 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:06.439313+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 155 2023-10-30T13:58:10.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.088+0000 7fb431cea640 1 -- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4180664a0 msgr2=0x7fb418068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:10.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.088+0000 7fb431cea640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4180664a0 0x7fb418068960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0ad000 tx=0x7fb41c0b2040 comp rx=0 tx=0).stop 2023-10-30T13:58:10.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.088+0000 7fb431cea640 1 -- 172.21.15.46:0/3696888064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb42c1f57a0 msgr2=0x7fb42c1f5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:10.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.088+0000 7fb431cea640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb42c1f57a0 0x7fb42c1f5c20 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fb42c06f4d0 tx=0x7fb42400f580 comp rx=0 tx=0).stop 2023-10-30T13:58:10.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.088+0000 7fb431cea640 1 -- 172.21.15.46:0/3696888064 shutdown_connections 2023-10-30T13:58:10.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.088+0000 7fb431cea640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4180664a0 0x7fb418068960 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.088+0000 7fb431cea640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb42c1f57a0 0x7fb42c1f5c20 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.088+0000 7fb431cea640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb42c1e9a60 0x7fb42c0c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.088+0000 7fb431cea640 1 --2- 172.21.15.46:0/3696888064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb414003980 0x7fb42c13da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:10.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.088+0000 7fb431cea640 1 -- 172.21.15.46:0/3696888064 >> 172.21.15.46:0/3696888064 conn(0x7fb42c0c7910 msgr2=0x7fb42c0afb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:10.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.088+0000 7fb431cea640 1 -- 172.21.15.46:0/3696888064 shutdown_connections 2023-10-30T13:58:10.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:10.088+0000 7fb431cea640 1 -- 172.21.15.46:0/3696888064 wait complete. 2023-10-30T13:58:10.112 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 155, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 155, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7518": { "addr": "172.21.15.46:6833/157453225", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 157453225, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 157453225, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7518, "incarnation": 145, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_7704": { "addr": "172.21.15.46:6836/3734066164", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3734066164, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3734066164, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7704, "incarnation": 152, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:58:06.439313+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7704, "mds_1": 7518 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:58:12.114 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:12.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f5aad640 1 Processor -- start 2023-10-30T13:58:12.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f5aad640 1 -- start start 2023-10-30T13:58:12.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f5aad640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83f01538f0 0x7f83f0159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f5aad640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f83f00cefa0 0x7f83f00cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f5aad640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f00ca6a0 con 0x7f83f015cd50 2023-10-30T13:58:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f5aad640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f00cf8f0 con 0x7f83f01538f0 2023-10-30T13:58:12.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f5aad640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f015d300 con 0x7f83f00cefa0 2023-10-30T13:58:12.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f4aab640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83f01538f0 0x7f83f0159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:12.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f4aab640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83f01538f0 0x7f83f0159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54142/0 (socket says 172.21.15.46:54142) 2023-10-30T13:58:12.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83effff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f83f015cd50 0x7f83f00c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53992/0 (socket says 172.21.15.46:53992) 2023-10-30T13:58:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f4aab640 1 -- 172.21.15.46:0/1359845159 learned_addr learned my addr 172.21.15.46:0/1359845159 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f52ac640 1 --2- 172.21.15.46:0/1359845159 >> v2:172.21.15.154:3300/0 conn(0x7f83f00cefa0 0x7f83f00cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f4aab640 1 -- 172.21.15.46:0/1359845159 >> v2:172.21.15.154:3300/0 conn(0x7f83f00cefa0 msgr2=0x7f83f00cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f4aab640 1 --2- 172.21.15.46:0/1359845159 >> v2:172.21.15.154:3300/0 conn(0x7f83f00cefa0 0x7f83f00cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f4aab640 1 -- 172.21.15.46:0/1359845159 >> v1:172.21.15.46:6789/0 conn(0x7f83f015cd50 legacy=0x7f83f00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83ef7fe640 1 -- 172.21.15.46:0/1359845159 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948659131 0 0) 0x7f83f00ca6a0 con 0x7f83f015cd50 2023-10-30T13:58:12.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.264+0000 7f83f4aab640 1 -- 172.21.15.46:0/1359845159 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f015d5e0 con 0x7f83f01538f0 2023-10-30T13:58:12.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.268+0000 7f83f4aab640 1 --2- 172.21.15.46:0/1359845159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83f01538f0 0x7f83f0159fb0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f83f0066a40 tx=0x7f83e401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361d25d88ecb74e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:12.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.268+0000 7f83ef7fe640 1 -- 172.21.15.46:0/1359845159 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e40a89f0 con 0x7f83f01538f0 2023-10-30T13:58:12.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.268+0000 7f83ef7fe640 1 -- 172.21.15.46:0/1359845159 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e40a8b50 con 0x7f83f01538f0 2023-10-30T13:58:12.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.268+0000 7f83ef7fe640 1 -- 172.21.15.46:0/1359845159 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e40b0440 con 0x7f83f01538f0 2023-10-30T13:58:12.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.268+0000 7f83f5aad640 1 -- 172.21.15.46:0/1359845159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83f01538f0 msgr2=0x7f83f0159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:12.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.268+0000 7f83f5aad640 1 --2- 172.21.15.46:0/1359845159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83f01538f0 0x7f83f0159fb0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f83f0066a40 tx=0x7f83e401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.268+0000 7f83f5aad640 1 -- 172.21.15.46:0/1359845159 shutdown_connections 2023-10-30T13:58:12.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.268+0000 7f83f5aad640 1 --2- 172.21.15.46:0/1359845159 >> v2:172.21.15.154:3300/0 conn(0x7f83f00cefa0 0x7f83f00cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.268+0000 7f83f5aad640 1 --2- 172.21.15.46:0/1359845159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83f01538f0 0x7f83f0159fb0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.268+0000 7f83f5aad640 1 -- 172.21.15.46:0/1359845159 >> 172.21.15.46:0/1359845159 conn(0x7f83f006a640 msgr2=0x7f83f0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:12.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.268+0000 7f83f5aad640 1 -- 172.21.15.46:0/1359845159 shutdown_connections 2023-10-30T13:58:12.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.268+0000 7f83f5aad640 1 -- 172.21.15.46:0/1359845159 wait complete. 2023-10-30T13:58:12.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f5aad640 1 Processor -- start 2023-10-30T13:58:12.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f5aad640 1 -- start start 2023-10-30T13:58:12.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f5aad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83f00cefa0 0x7f83f015bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:12.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f5aad640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83f01538f0 0x7f83f01f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:12.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f5aad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f015cd50 0x7f83f01f2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:12.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f5aad640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f01597c0 con 0x7f83f00cefa0 2023-10-30T13:58:12.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f5aad640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f00b3df0 con 0x7f83f015cd50 2023-10-30T13:58:12.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f5aad640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f01432b0 con 0x7f83f01538f0 2023-10-30T13:58:12.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f4aab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83f00cefa0 0x7f83f015bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:12.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f4aab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83f00cefa0 0x7f83f015bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54154/0 (socket says 172.21.15.46:54154) 2023-10-30T13:58:12.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f4aab640 1 -- 172.21.15.46:0/1982292072 learned_addr learned my addr 172.21.15.46:0/1982292072 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:12.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83effff640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83f01538f0 0x7f83f01f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:12.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f52ac640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f015cd50 0x7f83f01f2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:12.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f4aab640 1 -- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83f01538f0 msgr2=0x7f83f01f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:12.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f4aab640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83f01538f0 0x7f83f01f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f4aab640 1 -- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f015cd50 msgr2=0x7f83f01f2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:12.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f4aab640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f015cd50 0x7f83f01f2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f4aab640 1 -- 172.21.15.46:0/1982292072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e40094d0 con 0x7f83f00cefa0 2023-10-30T13:58:12.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f4aab640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83f00cefa0 0x7f83f015bbd0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f83e401b900 tx=0x7f83e401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:12.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83ed7fa640 1 -- 172.21.15.46:0/1982292072 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e40ba600 con 0x7f83f00cefa0 2023-10-30T13:58:12.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83ed7fa640 1 -- 172.21.15.46:0/1982292072 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e40ba760 con 0x7f83f00cefa0 2023-10-30T13:58:12.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83ed7fa640 1 -- 172.21.15.46:0/1982292072 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e40bab10 con 0x7f83f00cefa0 2023-10-30T13:58:12.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f5aad640 1 -- 172.21.15.46:0/1982292072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f00b3830 con 0x7f83f00cefa0 2023-10-30T13:58:12.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.272+0000 7f83f5aad640 1 -- 172.21.15.46:0/1982292072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f0159460 con 0x7f83f00cefa0 2023-10-30T13:58:12.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.276+0000 7f83f5aad640 1 -- 172.21.15.46:0/1982292072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f00b3690 con 0x7f83f00cefa0 2023-10-30T13:58:12.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.276+0000 7f83ed7fa640 1 -- 172.21.15.46:0/1982292072 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83e40bac70 con 0x7f83f00cefa0 2023-10-30T13:58:12.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.276+0000 7f83ed7fa640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83d40664a0 0x7f83d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:12.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.276+0000 7f83ed7fa640 1 -- 172.21.15.46:0/1982292072 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f83e4122b60 con 0x7f83f00cefa0 2023-10-30T13:58:12.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.276+0000 7f83effff640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83d40664a0 0x7f83d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:12.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.280+0000 7f83effff640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83d40664a0 0x7f83d4068960 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f83f014d790 tx=0x7f83dc007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:12.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.280+0000 7f83ed7fa640 1 -- 172.21.15.46:0/1982292072 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f83f00b3690 con 0x7f83f00cefa0 2023-10-30T13:58:12.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.504+0000 7f83f5aad640 1 -- 172.21.15.46:0/1982292072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83f00b3690 con 0x7f83f00cefa0 2023-10-30T13:58:12.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.504+0000 7f83ed7fa640 1 -- 172.21.15.46:0/1982292072 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f83f00b3690 con 0x7f83f00cefa0 2023-10-30T13:58:12.511 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:58:12.511 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:06.439313+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:12.511 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 155 2023-10-30T13:58:12.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.508+0000 7f83f5aad640 1 -- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83d40664a0 msgr2=0x7f83d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:12.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.508+0000 7f83f5aad640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83d40664a0 0x7f83d4068960 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f83f014d790 tx=0x7f83dc007c10 comp rx=0 tx=0).stop 2023-10-30T13:58:12.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.508+0000 7f83f5aad640 1 -- 172.21.15.46:0/1982292072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83f00cefa0 msgr2=0x7f83f015bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:12.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.508+0000 7f83f5aad640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83f00cefa0 0x7f83f015bbd0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f83e401b900 tx=0x7f83e401ba90 comp rx=0 tx=0).stop 2023-10-30T13:58:12.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.508+0000 7f83f5aad640 1 -- 172.21.15.46:0/1982292072 shutdown_connections 2023-10-30T13:58:12.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.508+0000 7f83f5aad640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83d40664a0 0x7f83d4068960 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.508+0000 7f83f5aad640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f015cd50 0x7f83f01f2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.508+0000 7f83f5aad640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83f01538f0 0x7f83f01f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.508+0000 7f83f5aad640 1 --2- 172.21.15.46:0/1982292072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83f00cefa0 0x7f83f015bbd0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:12.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.508+0000 7f83f5aad640 1 -- 172.21.15.46:0/1982292072 >> 172.21.15.46:0/1982292072 conn(0x7f83f006a640 msgr2=0x7f83f01492c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:12.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.508+0000 7f83f5aad640 1 -- 172.21.15.46:0/1982292072 shutdown_connections 2023-10-30T13:58:12.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:12.508+0000 7f83f5aad640 1 -- 172.21.15.46:0/1982292072 wait complete. 2023-10-30T13:58:12.532 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:58:14.533 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:14.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82dced6640 1 Processor -- start 2023-10-30T13:58:14.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82dced6640 1 -- start start 2023-10-30T13:58:14.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82dced6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82d815d6f0 0x7f82d815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:14.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82dced6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82d80cf160 0x7f82d80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:14.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82dced6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d80cb800 con 0x7f82d81538a0 2023-10-30T13:58:14.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82dced6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d80cb330 con 0x7f82d80cf160 2023-10-30T13:58:14.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82dced6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d80cad70 con 0x7f82d815d6f0 2023-10-30T13:58:14.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82d6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82d80cf160 0x7f82d80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:14.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82d6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82d80cf160 0x7f82d80cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54170/0 (socket says 172.21.15.46:54170) 2023-10-30T13:58:14.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82d6d9d640 1 -- 172.21.15.46:0/962273592 learned_addr learned my addr 172.21.15.46:0/962273592 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:14.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82d5d9b640 1 --2- 172.21.15.46:0/962273592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82d815d6f0 0x7f82d815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:14.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82d6d9d640 1 -- 172.21.15.46:0/962273592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82d815d6f0 msgr2=0x7f82d815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:14.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82d6d9d640 1 --2- 172.21.15.46:0/962273592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82d815d6f0 0x7f82d815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82d6d9d640 1 -- 172.21.15.46:0/962273592 >> v1:172.21.15.154:6789/0 conn(0x7f82d81538a0 legacy=0x7f82d80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:14.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82d6d9d640 1 -- 172.21.15.46:0/962273592 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d80cb050 con 0x7f82d80cf160 2023-10-30T13:58:14.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.684+0000 7f82d5d9b640 1 --2- 172.21.15.46:0/962273592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82d815d6f0 0x7f82d815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:58:14.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.688+0000 7f82d6d9d640 1 --2- 172.21.15.46:0/962273592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d80cf160 0x7f82d80cf590 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009e50 tx=0x7f82d001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cab9b9cda3d13ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.688+0000 7f82d559a640 1 -- 172.21.15.46:0/962273592 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d00af400 con 0x7f82d80cf160 2023-10-30T13:58:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.688+0000 7f82d559a640 1 -- 172.21.15.46:0/962273592 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d00ab050 con 0x7f82d80cf160 2023-10-30T13:58:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.688+0000 7f82d559a640 1 -- 172.21.15.46:0/962273592 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d00af780 con 0x7f82d80cf160 2023-10-30T13:58:14.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.688+0000 7f82dced6640 1 -- 172.21.15.46:0/962273592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d80cf160 msgr2=0x7f82d80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:14.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.688+0000 7f82dced6640 1 --2- 172.21.15.46:0/962273592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d80cf160 0x7f82d80cf590 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009e50 tx=0x7f82d001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.688+0000 7f82dced6640 1 -- 172.21.15.46:0/962273592 shutdown_connections 2023-10-30T13:58:14.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.688+0000 7f82dced6640 1 --2- 172.21.15.46:0/962273592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d80cf160 0x7f82d80cf590 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.688+0000 7f82dced6640 1 --2- 172.21.15.46:0/962273592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82d815d6f0 0x7f82d815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.688+0000 7f82dced6640 1 -- 172.21.15.46:0/962273592 >> 172.21.15.46:0/962273592 conn(0x7f82d806bfe0 msgr2=0x7f82d806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:14.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.688+0000 7f82dced6640 1 -- 172.21.15.46:0/962273592 shutdown_connections 2023-10-30T13:58:14.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.688+0000 7f82dced6640 1 -- 172.21.15.46:0/962273592 wait complete. 2023-10-30T13:58:14.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82dced6640 1 Processor -- start 2023-10-30T13:58:14.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82dced6640 1 -- start start 2023-10-30T13:58:14.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82dced6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82d80cf160 0x7f82d80c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:14.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82dced6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d81538a0 0x7f82d80cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:14.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82dced6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82d815d6f0 0x7f82d80cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:14.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82dced6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d80c6c30 con 0x7f82d81538a0 2023-10-30T13:58:14.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82dced6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d8159130 con 0x7f82d80cf160 2023-10-30T13:58:14.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82dced6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d80ccfb0 con 0x7f82d815d6f0 2023-10-30T13:58:14.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82d5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d81538a0 0x7f82d80cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:14.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82d5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d81538a0 0x7f82d80cd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54172/0 (socket says 172.21.15.46:54172) 2023-10-30T13:58:14.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82d5d9b640 1 -- 172.21.15.46:0/453706193 learned_addr learned my addr 172.21.15.46:0/453706193 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:14.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82d6d9d640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82d815d6f0 0x7f82d80cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:14.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82d659c640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82d80cf160 0x7f82d80c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:14.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82d5d9b640 1 -- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82d815d6f0 msgr2=0x7f82d80cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:14.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82d5d9b640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82d815d6f0 0x7f82d80cdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82d5d9b640 1 -- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82d80cf160 msgr2=0x7f82d80c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:14.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82d5d9b640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82d80cf160 0x7f82d80c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82d5d9b640 1 -- 172.21.15.46:0/453706193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d0009b30 con 0x7f82d81538a0 2023-10-30T13:58:14.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82d659c640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82d80cf160 0x7f82d80c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:58:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82d5d9b640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d81538a0 0x7f82d80cd9c0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00d800 tx=0x7f82cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82c6ffd640 1 -- 172.21.15.46:0/453706193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc009660 con 0x7f82d81538a0 2023-10-30T13:58:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82c6ffd640 1 -- 172.21.15.46:0/453706193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82cc0097c0 con 0x7f82d81538a0 2023-10-30T13:58:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82c6ffd640 1 -- 172.21.15.46:0/453706193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc009b70 con 0x7f82d81538a0 2023-10-30T13:58:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82dced6640 1 -- 172.21.15.46:0/453706193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d81588d0 con 0x7f82d81538a0 2023-10-30T13:58:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.692+0000 7f82dced6640 1 -- 172.21.15.46:0/453706193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d8158e50 con 0x7f82d81538a0 2023-10-30T13:58:14.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.696+0000 7f82dced6640 1 -- 172.21.15.46:0/453706193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d80b18e0 con 0x7f82d81538a0 2023-10-30T13:58:14.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.696+0000 7f82c6ffd640 1 -- 172.21.15.46:0/453706193 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82cc009cd0 con 0x7f82d81538a0 2023-10-30T13:58:14.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.696+0000 7f82c6ffd640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82b80664d0 0x7f82b8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:14.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.696+0000 7f82c6ffd640 1 -- 172.21.15.46:0/453706193 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f82cc084bb0 con 0x7f82d81538a0 2023-10-30T13:58:14.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.696+0000 7f82d659c640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82b80664d0 0x7f82b8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:14.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.696+0000 7f82d659c640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82b80664d0 0x7f82b8068990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f82c8000f70 tx=0x7f82c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:14.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.700+0000 7f82c6ffd640 1 -- 172.21.15.46:0/453706193 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82cc051980 con 0x7f82d81538a0 2023-10-30T13:58:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 -- 172.21.15.46:0/453706193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82d80b2a50 con 0x7f82d81538a0 2023-10-30T13:58:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82c6ffd640 1 -- 172.21.15.46:0/453706193 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f82cc054fa0 con 0x7f82d81538a0 2023-10-30T13:58:14.897 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:58:14.897 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:13.334794+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 156 2023-10-30T13:58:14.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 -- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82b80664d0 msgr2=0x7f82b8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:14.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82b80664d0 0x7f82b8068990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f82c8000f70 tx=0x7f82c8009040 comp rx=0 tx=0).stop 2023-10-30T13:58:14.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 -- 172.21.15.46:0/453706193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d81538a0 msgr2=0x7f82d80cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:14.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d81538a0 0x7f82d80cd9c0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00d800 tx=0x7f82cc00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 -- 172.21.15.46:0/453706193 shutdown_connections 2023-10-30T13:58:14.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82b80664d0 0x7f82b8068990 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82d815d6f0 0x7f82d80cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d81538a0 0x7f82d80cd9c0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 --2- 172.21.15.46:0/453706193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82d80cf160 0x7f82d80c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:14.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 -- 172.21.15.46:0/453706193 >> 172.21.15.46:0/453706193 conn(0x7f82d806bfe0 msgr2=0x7f82d815b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:14.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 -- 172.21.15.46:0/453706193 shutdown_connections 2023-10-30T13:58:14.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:14.892+0000 7f82dced6640 1 -- 172.21.15.46:0/453706193 wait complete. 2023-10-30T13:58:14.918 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=7356 2023-10-30T13:58:14.918 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.e 2023-10-30T13:58:18.919 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T13:58:18.920 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:58:18.920 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:58:18.924 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:58:18.924 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:19.028 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T13:58:19.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.100+0000 7f7d3b165640 1 Processor -- start 2023-10-30T13:58:19.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.100+0000 7f7d3b165640 1 -- start start 2023-10-30T13:58:19.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.104+0000 7f7d3b165640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7d341538a0 0x7f7d3415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:19.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.104+0000 7f7d3b165640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d340cb800 con 0x7f7d340cf160 2023-10-30T13:58:19.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.104+0000 7f7d3b165640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3415dc30 con 0x7f7d340ca0e0 2023-10-30T13:58:19.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.104+0000 7f7d3b165640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d340cb330 con 0x7f7d341538a0 2023-10-30T13:58:19.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.104+0000 7f7d39702640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7d340cf160 0x7f7d34164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38782/0 (socket says 172.21.15.46:38782) 2023-10-30T13:58:19.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.104+0000 7f7d39702640 1 -- 172.21.15.46:0/3968240440 learned_addr learned my addr 172.21.15.46:0/3968240440 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T13:58:19.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.104+0000 7f7d38f01640 1 --2- 172.21.15.46:0/3968240440 >> v2:172.21.15.46:3300/0 conn(0x7f7d341538a0 0x7f7d3415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:19.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.104+0000 7f7d337fe640 1 -- 172.21.15.46:0/3968240440 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509586579 0 0) 0x7f7d340cb800 con 0x7f7d340cf160 2023-10-30T13:58:19.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.104+0000 7f7d337fe640 1 -- 172.21.15.46:0/3968240440 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d24003530 con 0x7f7d340cf160 2023-10-30T13:58:19.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.104+0000 7f7d38f01640 1 -- 172.21.15.46:0/3968240440 >> v1:172.21.15.154:6789/0 conn(0x7f7d340ca0e0 legacy=0x7f7d3415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:19.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.104+0000 7f7d38f01640 1 -- 172.21.15.46:0/3968240440 >> v1:172.21.15.46:6789/0 conn(0x7f7d340cf160 legacy=0x7f7d34164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:19.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.104+0000 7f7d38f01640 1 -- 172.21.15.46:0/3968240440 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d340cad70 con 0x7f7d341538a0 2023-10-30T13:58:19.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.108+0000 7f7d38f01640 1 --2- 172.21.15.46:0/3968240440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d341538a0 0x7f7d3415d6f0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f7d280097c0 tx=0x7f7d2801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52fefcccf91355fd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:19.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.108+0000 7f7d337fe640 1 -- 172.21.15.46:0/3968240440 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d280a8970 con 0x7f7d341538a0 2023-10-30T13:58:19.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.108+0000 7f7d337fe640 1 -- 172.21.15.46:0/3968240440 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d280a8ad0 con 0x7f7d341538a0 2023-10-30T13:58:19.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.108+0000 7f7d337fe640 1 -- 172.21.15.46:0/3968240440 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d280a8e60 con 0x7f7d341538a0 2023-10-30T13:58:19.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.108+0000 7f7d3b165640 1 -- 172.21.15.46:0/3968240440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d341538a0 msgr2=0x7f7d3415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:19.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.108+0000 7f7d3b165640 1 --2- 172.21.15.46:0/3968240440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d341538a0 0x7f7d3415d6f0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f7d280097c0 tx=0x7f7d2801ec70 comp rx=0 tx=0).stop 2023-10-30T13:58:19.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.108+0000 7f7d3b165640 1 -- 172.21.15.46:0/3968240440 shutdown_connections 2023-10-30T13:58:19.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.108+0000 7f7d3b165640 1 --2- 172.21.15.46:0/3968240440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d341538a0 0x7f7d3415d6f0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:19.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.108+0000 7f7d3b165640 1 -- 172.21.15.46:0/3968240440 >> 172.21.15.46:0/3968240440 conn(0x7f7d3406bfe0 msgr2=0x7f7d3406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:19.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.108+0000 7f7d3b165640 1 -- 172.21.15.46:0/3968240440 shutdown_connections 2023-10-30T13:58:19.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.108+0000 7f7d3b165640 1 -- 172.21.15.46:0/3968240440 wait complete. 2023-10-30T13:58:19.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d3b165640 1 Processor -- start 2023-10-30T13:58:19.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d3b165640 1 -- start start 2023-10-30T13:58:19.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d3b165640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d340ca0e0 0x7f7d340ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:19.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d3b165640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d340cf160 0x7f7d34148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:19.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d3b165640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d341538a0 0x7f7d341ef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:19.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d3b165640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d34146720 con 0x7f7d340cf160 2023-10-30T13:58:19.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d3b165640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d340b31a0 con 0x7f7d341538a0 2023-10-30T13:58:19.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d3b165640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d340b42d0 con 0x7f7d340ca0e0 2023-10-30T13:58:19.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d33fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d340cf160 0x7f7d34148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:19.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d33fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d340cf160 0x7f7d34148f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46910/0 (socket says 172.21.15.46:46910) 2023-10-30T13:58:19.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d33fff640 1 -- 172.21.15.46:0/4084303190 learned_addr learned my addr 172.21.15.46:0/4084303190 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:19.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d38f01640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d340ca0e0 0x7f7d340ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:19.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.112+0000 7f7d39702640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d341538a0 0x7f7d341ef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:19.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d33fff640 1 -- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d340ca0e0 msgr2=0x7f7d340ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:19.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d33fff640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d340ca0e0 0x7f7d340ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:19.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d33fff640 1 -- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d341538a0 msgr2=0x7f7d341ef3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:19.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d33fff640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d341538a0 0x7f7d341ef3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:19.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d33fff640 1 -- 172.21.15.46:0/4084303190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d28009420 con 0x7f7d340cf160 2023-10-30T13:58:19.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d33fff640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d340cf160 0x7f7d34148f60 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000c3f0 tx=0x7f7d2000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:19.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d317fa640 1 -- 172.21.15.46:0/4084303190 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d200052d0 con 0x7f7d340cf160 2023-10-30T13:58:19.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d317fa640 1 -- 172.21.15.46:0/4084303190 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d20005430 con 0x7f7d340cf160 2023-10-30T13:58:19.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d3b165640 1 -- 172.21.15.46:0/4084303190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d341f36b0 con 0x7f7d340cf160 2023-10-30T13:58:19.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d317fa640 1 -- 172.21.15.46:0/4084303190 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d20003840 con 0x7f7d340cf160 2023-10-30T13:58:19.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d3b165640 1 -- 172.21.15.46:0/4084303190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d340b33f0 con 0x7f7d340cf160 2023-10-30T13:58:19.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d3b165640 1 -- 172.21.15.46:0/4084303190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d340b4460 con 0x7f7d340cf160 2023-10-30T13:58:19.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.116+0000 7f7d317fa640 1 -- 172.21.15.46:0/4084303190 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7d200039a0 con 0x7f7d340cf160 2023-10-30T13:58:19.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.120+0000 7f7d317fa640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7d180664d0 0x7f7d18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:19.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.120+0000 7f7d317fa640 1 -- 172.21.15.46:0/4084303190 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f7d200838d0 con 0x7f7d340cf160 2023-10-30T13:58:19.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.120+0000 7f7d38f01640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7d180664d0 0x7f7d18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:19.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.124+0000 7f7d38f01640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7d180664d0 0x7f7d18068990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7d280b2000 tx=0x7f7d28008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:19.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.128+0000 7f7d317fa640 1 -- 172.21.15.46:0/4084303190 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7d200506c0 con 0x7f7d340cf160 2023-10-30T13:58:19.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.344+0000 7f7d3b165640 1 -- 172.21.15.46:0/4084303190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d3415dc30 con 0x7f7d340cf160 2023-10-30T13:58:19.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.344+0000 7f7d317fa640 1 -- 172.21.15.46:0/4084303190 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f7d3415dc30 con 0x7f7d340cf160 2023-10-30T13:58:19.349 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:58:19.350 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:13.334794+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:19.350 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 156 2023-10-30T13:58:19.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.348+0000 7f7d3b165640 1 -- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7d180664d0 msgr2=0x7f7d18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:19.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.348+0000 7f7d3b165640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7d180664d0 0x7f7d18068990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7d280b2000 tx=0x7f7d28008040 comp rx=0 tx=0).stop 2023-10-30T13:58:19.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.348+0000 7f7d3b165640 1 -- 172.21.15.46:0/4084303190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d340cf160 msgr2=0x7f7d34148f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:19.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.348+0000 7f7d3b165640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d340cf160 0x7f7d34148f60 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000c3f0 tx=0x7f7d2000c8b0 comp rx=0 tx=0).stop 2023-10-30T13:58:19.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.348+0000 7f7d3b165640 1 -- 172.21.15.46:0/4084303190 shutdown_connections 2023-10-30T13:58:19.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.348+0000 7f7d3b165640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7d180664d0 0x7f7d18068990 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:19.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.348+0000 7f7d3b165640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d341538a0 0x7f7d341ef3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:19.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.348+0000 7f7d3b165640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d340cf160 0x7f7d34148f60 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:19.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.348+0000 7f7d3b165640 1 --2- 172.21.15.46:0/4084303190 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d340ca0e0 0x7f7d340ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:19.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.348+0000 7f7d3b165640 1 -- 172.21.15.46:0/4084303190 >> 172.21.15.46:0/4084303190 conn(0x7f7d3406bfe0 msgr2=0x7f7d340c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:19.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.348+0000 7f7d3b165640 1 -- 172.21.15.46:0/4084303190 shutdown_connections 2023-10-30T13:58:19.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:19.348+0000 7f7d3b165640 1 -- 172.21.15.46:0/4084303190 wait complete. 2023-10-30T13:58:19.380 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T13:58:19.429 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 19% (42/211) Updating files: 20% (43/211) Updating files: 21% (45/211) Updating files: 22% (47/211) Updating files: 23% (49/211) Updating files: 24% (51/211) Updating files: 25% (53/211) Updating files: 26% (55/211) Updating files: 27% (57/211) Updating files: 28% (60/211) Updating files: 29% (62/211) Updating files: 30% (64/211) Updating files: 31% (66/211) Updating files: 32% (68/211) Updating files: 33% (70/211) Updating files: 34% (72/211) Updating files: 35% (74/211) Updating files: 36% (76/211) Updating files: 37% (79/211) Updating files: 38% (81/211) Updating files: 39% (83/211) Updating files: 40% (85/211) Updating files: 41% (87/211) Updating files: 42% (89/211) Updating files: 43% (91/211) Updating files: 44% (93/211) Updating files: 45% (95/211) Updating files: 46% (98/211) Updating files: 47% (100/211) Updating files: 48% (102/211) Updating files: 49% (104/211) Updating files: 50% (106/211) Updating files: 51% (108/211) Updating files: 52% (110/211) Updating files: 53% (112/211) Updating files: 54% (114/211) Updating files: 55% (117/211) Updating files: 56% (119/211) Updating files: 57% (121/211) Updating files: 58% (123/211) Updating files: 59% (125/211) Updating files: 60% (127/211) Updating files: 61% (129/211) Updating files: 62% (131/211) Updating files: 63% (133/211) Updating files: 64% (136/211) Updating files: 65% (138/211) Updating files: 66% (140/211) Updating files: 67% (142/211) Updating files: 68% (144/211) Updating files: 69% (146/211) Updating files: 70% (148/211) Updating files: 71% (150/211) Updating files: 72% (152/211) Updating files: 73% (155/211) Updating files: 74% (157/211) Updating files: 75% (159/211) Updating files: 76% (161/211) Updating files: 77% (163/211) Updating files: 78% (165/211) Updating files: 79% (167/211) Updating files: 80% (169/211) Updating files: 81% (171/211) Updating files: 81% (172/211) Updating files: 82% (174/211) Updating files: 83% (176/211) Updating files: 84% (178/211) Updating files: 85% (180/211) Updating files: 86% (182/211) Updating files: 87% (184/211) Updating files: 88% (186/211) Updating files: 88% (187/211) Updating files: 89% (188/211) Updating files: 89% (189/211) Updating files: 90% (190/211) Updating files: 91% (193/211) Updating files: 92% (195/211) Updating files: 92% (196/211) Updating files: 93% (197/211) Updating files: 94% (199/211) Updating files: 95% (201/211) Updating files: 96% (203/211) Updating files: 97% (205/211) Updating files: 98% (207/211) Updating files: 99% (209/211) Updating files: 100% (211/211) Updating files: 100% (211/211), done. 2023-10-30T13:58:20.210 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at cbae6a435c6 v0.59 2023-10-30T13:58:20.213 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.59 2023-10-30T13:58:20.231 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 60 -eq 48 ] 2023-10-30T13:58:20.231 INFO:tasks.workunit.client.0.smithi046.stdout:v0.60 2023-10-30T13:58:20.232 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.60 2023-10-30T13:58:20.232 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.60 2023-10-30T13:58:20.232 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.60 2023-10-30T13:58:21.381 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:58:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.528+0000 7f51dee88640 1 Processor -- start 2023-10-30T13:58:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.528+0000 7f51dee88640 1 -- start start 2023-10-30T13:58:21.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.528+0000 7f51dee88640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f51d815cd50 0x7f51d80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:21.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.528+0000 7f51dee88640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51d80cefa0 0x7f51d80cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:21.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.528+0000 7f51dee88640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d815a6c0 con 0x7f51d81538f0 2023-10-30T13:58:21.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.528+0000 7f51dee88640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d815d300 con 0x7f51d815cd50 2023-10-30T13:58:21.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.528+0000 7f51dee88640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d815d5e0 con 0x7f51d80cefa0 2023-10-30T13:58:21.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.532+0000 7f51de687640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51d80cefa0 0x7f51d80cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:21.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.532+0000 7f51dd685640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f51d815cd50 0x7f51d80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.532+0000 7f51de687640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51d80cefa0 0x7f51d80cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:54096/0 (socket says 172.21.15.46:54096) 2023-10-30T13:58:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.532+0000 7f51de687640 1 -- 172.21.15.46:0/2268573784 learned_addr learned my addr 172.21.15.46:0/2268573784 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:21.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.532+0000 7f51de687640 1 -- 172.21.15.46:0/2268573784 >> v2:172.21.15.154:3300/0 conn(0x7f51d815cd50 msgr2=0x7f51d80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:58:21.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.532+0000 7f51de687640 1 --2- 172.21.15.46:0/2268573784 >> v2:172.21.15.154:3300/0 conn(0x7f51d815cd50 0x7f51d80c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:21.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.532+0000 7f51de687640 1 -- 172.21.15.46:0/2268573784 >> v1:172.21.15.154:6789/0 conn(0x7f51d81538f0 legacy=0x7f51d8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:21.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.532+0000 7f51de687640 1 -- 172.21.15.46:0/2268573784 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d815d8c0 con 0x7f51d80cefa0 2023-10-30T13:58:21.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.532+0000 7f51de687640 1 --2- 172.21.15.46:0/2268573784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51d80cefa0 0x7f51d80cf3d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009e50 tx=0x7f51d001f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53879bae89dea838 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:21.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dce84640 1 -- 172.21.15.46:0/2268573784 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d00af400 con 0x7f51d80cefa0 2023-10-30T13:58:21.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dce84640 1 -- 172.21.15.46:0/2268573784 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d00ab050 con 0x7f51d80cefa0 2023-10-30T13:58:21.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dce84640 1 -- 172.21.15.46:0/2268573784 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d00af780 con 0x7f51d80cefa0 2023-10-30T13:58:21.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 -- 172.21.15.46:0/2268573784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51d80cefa0 msgr2=0x7f51d80cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:21.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 --2- 172.21.15.46:0/2268573784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51d80cefa0 0x7f51d80cf3d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009e50 tx=0x7f51d001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:58:21.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 -- 172.21.15.46:0/2268573784 shutdown_connections 2023-10-30T13:58:21.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 --2- 172.21.15.46:0/2268573784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51d80cefa0 0x7f51d80cf3d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:21.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 --2- 172.21.15.46:0/2268573784 >> v2:172.21.15.154:3300/0 conn(0x7f51d815cd50 0x7f51d80c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:21.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 -- 172.21.15.46:0/2268573784 >> 172.21.15.46:0/2268573784 conn(0x7f51d806a640 msgr2=0x7f51d8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:21.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 -- 172.21.15.46:0/2268573784 shutdown_connections 2023-10-30T13:58:21.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 -- 172.21.15.46:0/2268573784 wait complete. 2023-10-30T13:58:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 Processor -- start 2023-10-30T13:58:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 -- start start 2023-10-30T13:58:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d80cefa0 0x7f51d80b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51d81538f0 0x7f51d813ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51d815cd50 0x7f51d815c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d8145db0 con 0x7f51d80cefa0 2023-10-30T13:58:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80b42b0 con 0x7f51d815cd50 2023-10-30T13:58:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dee88640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80b35d0 con 0x7f51d81538f0 2023-10-30T13:58:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.536+0000 7f51dde86640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d80cefa0 0x7f51d80b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51dde86640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d80cefa0 0x7f51d80b3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46926/0 (socket says 172.21.15.46:46926) 2023-10-30T13:58:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51dde86640 1 -- 172.21.15.46:0/2013659216 learned_addr learned my addr 172.21.15.46:0/2013659216 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T13:58:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51de687640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51d815cd50 0x7f51d815c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51dd685640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51d81538f0 0x7f51d813ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:21.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51dde86640 1 -- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51d81538f0 msgr2=0x7f51d813ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:21.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51dde86640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51d81538f0 0x7f51d813ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:21.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51dde86640 1 -- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51d815cd50 msgr2=0x7f51d815c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:21.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51dde86640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51d815cd50 0x7f51d815c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:21.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51dde86640 1 -- 172.21.15.46:0/2013659216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d0009b30 con 0x7f51d80cefa0 2023-10-30T13:58:21.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51dde86640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d80cefa0 0x7f51d80b3a60 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f51d400c500 tx=0x7f51d400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:21.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51ce7fc640 1 -- 172.21.15.46:0/2013659216 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d4005220 con 0x7f51d80cefa0 2023-10-30T13:58:21.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51ce7fc640 1 -- 172.21.15.46:0/2013659216 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d4005380 con 0x7f51d80cefa0 2023-10-30T13:58:21.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51dee88640 1 -- 172.21.15.46:0/2013659216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d815b2e0 con 0x7f51d80cefa0 2023-10-30T13:58:21.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51ce7fc640 1 -- 172.21.15.46:0/2013659216 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d4007270 con 0x7f51d80cefa0 2023-10-30T13:58:21.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51dee88640 1 -- 172.21.15.46:0/2013659216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d815b540 con 0x7f51d80cefa0 2023-10-30T13:58:21.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51dee88640 1 -- 172.21.15.46:0/2013659216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d804e170 con 0x7f51d80cefa0 2023-10-30T13:58:21.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.540+0000 7f51ce7fc640 1 -- 172.21.15.46:0/2013659216 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51d40073d0 con 0x7f51d80cefa0 2023-10-30T13:58:21.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.544+0000 7f51ce7fc640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51bc0664a0 0x7f51bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:58:21.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.544+0000 7f51ce7fc640 1 -- 172.21.15.46:0/2013659216 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f51d4083db0 con 0x7f51d80cefa0 2023-10-30T13:58:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.544+0000 7f51dd685640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51bc0664a0 0x7f51bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:58:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.544+0000 7f51dd685640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51bc0664a0 0x7f51bc068960 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f51d81497b0 tx=0x7f51c8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:58:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.548+0000 7f51ce7fc640 1 -- 172.21.15.46:0/2013659216 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51d4088070 con 0x7f51d80cefa0 2023-10-30T13:58:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.756+0000 7f51dee88640 1 -- 172.21.15.46:0/2013659216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51d80b18c0 con 0x7f51d80cefa0 2023-10-30T13:58:21.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.756+0000 7f51ce7fc640 1 -- 172.21.15.46:0/2013659216 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f51d4050bb0 con 0x7f51d80cefa0 2023-10-30T13:58:21.763 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T13:58:21.763 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6809,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:13.334794+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:58:21.764 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 157 2023-10-30T13:58:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.760+0000 7f51dee88640 1 -- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51bc0664a0 msgr2=0x7f51bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.760+0000 7f51dee88640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51bc0664a0 0x7f51bc068960 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f51d81497b0 tx=0x7f51c8009190 comp rx=0 tx=0).stop 2023-10-30T13:58:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.760+0000 7f51dee88640 1 -- 172.21.15.46:0/2013659216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d80cefa0 msgr2=0x7f51d80b3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:58:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.760+0000 7f51dee88640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d80cefa0 0x7f51d80b3a60 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f51d400c500 tx=0x7f51d400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:58:21.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.760+0000 7f51dee88640 1 -- 172.21.15.46:0/2013659216 shutdown_connections 2023-10-30T13:58:21.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.760+0000 7f51dee88640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51bc0664a0 0x7f51bc068960 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:21.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.760+0000 7f51dee88640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51d815cd50 0x7f51d815c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:21.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.760+0000 7f51dee88640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51d81538f0 0x7f51d813ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:21.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.760+0000 7f51dee88640 1 --2- 172.21.15.46:0/2013659216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51d80cefa0 0x7f51d80b3a60 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:58:21.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.760+0000 7f51dee88640 1 -- 172.21.15.46:0/2013659216 >> 172.21.15.46:0/2013659216 conn(0x7f51d806a640 msgr2=0x7f51d80c3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:58:21.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.760+0000 7f51dee88640 1 -- 172.21.15.46:0/2013659216 shutdown_connections 2023-10-30T13:58:21.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T13:58:21.760+0000 7f51dee88640 1 -- 172.21.15.46:0/2013659216 wait complete. 2023-10-30T13:58:21.788 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T13:58:21.788 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 115 secs before thrashing 2023-10-30T13:58:23.402 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T13:58:23.396+0000 7fdd5c91c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:58:23.403 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T13:58:23.396+0000 7fdd5c91c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:58:23.761 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:58:23.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:58:43.764 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 33% (87/256) Updating files: 34% (88/256) Updating files: 35% (90/256) Updating files: 36% (93/256) Updating files: 37% (95/256) Updating files: 38% (98/256) Updating files: 39% (100/256) Updating files: 40% (103/256) Updating files: 41% (105/256) Updating files: 42% (108/256) Updating files: 43% (111/256) Updating files: 43% (112/256) Updating files: 44% (113/256) Updating files: 45% (116/256) Updating files: 46% (118/256) Updating files: 47% (121/256) Updating files: 48% (123/256) Updating files: 49% (126/256) Updating files: 50% (128/256) Updating files: 51% (131/256) Updating files: 52% (134/256) Updating files: 53% (136/256) Updating files: 54% (139/256) Updating files: 55% (141/256) Updating files: 56% (144/256) Updating files: 57% (146/256) Updating files: 58% (149/256) Updating files: 59% (152/256) Updating files: 60% (154/256) Updating files: 61% (157/256) Updating files: 62% (159/256) Updating files: 63% (162/256) Updating files: 64% (164/256) Updating files: 65% (167/256) Updating files: 66% (169/256) Updating files: 67% (172/256) Updating files: 68% (175/256) Updating files: 69% (177/256) Updating files: 70% (180/256) Updating files: 71% (182/256) Updating files: 72% (185/256) Updating files: 73% (187/256) Updating files: 74% (190/256) Updating files: 75% (192/256) Updating files: 76% (195/256) Updating files: 77% (198/256) Updating files: 78% (200/256) Updating files: 79% (203/256) Updating files: 80% (205/256) Updating files: 81% (208/256) Updating files: 82% (210/256) Updating files: 83% (213/256) Updating files: 84% (216/256) Updating files: 85% (218/256) Updating files: 86% (221/256) Updating files: 87% (223/256) Updating files: 88% (226/256) Updating files: 89% (228/256) Updating files: 90% (231/256) Updating files: 91% (233/256) Updating files: 92% (236/256) Updating files: 93% (239/256) Updating files: 94% (241/256) Updating files: 95% (244/256) Updating files: 96% (246/256) Updating files: 97% (249/256) Updating files: 98% (251/256) Updating files: 99% (254/256) Updating files: 100% (256/256) Updating files: 100% (256/256), done. 2023-10-30T13:58:45.091 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at f26f7a39021 v0.60 2023-10-30T13:58:45.099 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.60 2023-10-30T13:58:45.134 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 61 -eq 48 ] 2023-10-30T13:58:45.134 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.61 2023-10-30T13:58:45.134 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.61 2023-10-30T13:58:45.134 INFO:tasks.workunit.client.0.smithi046.stdout:v0.61 2023-10-30T13:58:45.135 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.61 2023-10-30T13:58:53.732 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 10% (40/376) Updating files: 11% (42/376) Updating files: 12% (46/376) Updating files: 13% (49/376) Updating files: 14% (53/376) Updating files: 15% (57/376) Updating files: 16% (61/376) Updating files: 17% (64/376) Updating files: 18% (68/376) Updating files: 19% (72/376) Updating files: 20% (76/376) Updating files: 21% (79/376) Updating files: 22% (83/376) Updating files: 23% (87/376) Updating files: 24% (91/376) Updating files: 25% (94/376) Updating files: 26% (98/376) Updating files: 27% (102/376) Updating files: 28% (106/376) Updating files: 29% (110/376) Updating files: 30% (113/376) Updating files: 31% (117/376) Updating files: 32% (121/376) Updating files: 33% (125/376) Updating files: 34% (128/376) Updating files: 35% (132/376) Updating files: 36% (136/376) Updating files: 37% (140/376) Updating files: 38% (143/376) Updating files: 39% (147/376) Updating files: 40% (151/376) Updating files: 41% (155/376) Updating files: 42% (158/376) Updating files: 43% (162/376) Updating files: 44% (166/376) Updating files: 45% (170/376) Updating files: 46% (173/376) Updating files: 47% (177/376) Updating files: 48% (181/376) Updating files: 49% (185/376) Updating files: 50% (188/376) Updating files: 51% (192/376) Updating files: 52% (196/376) Updating files: 53% (200/376) Updating files: 54% (204/376) Updating files: 55% (207/376) Updating files: 56% (211/376) Updating files: 57% (215/376) Updating files: 58% (219/376) Updating files: 59% (222/376) Updating files: 60% (226/376) Updating files: 61% (230/376) Updating files: 62% (234/376) Updating files: 63% (237/376) Updating files: 64% (241/376) Updating files: 65% (245/376) Updating files: 66% (249/376) Updating files: 67% (252/376) Updating files: 68% (256/376) Updating files: 69% (260/376) Updating files: 69% (262/376) Updating files: 70% (264/376) Updating files: 71% (267/376) Updating files: 72% (271/376) Updating files: 73% (275/376) Updating files: 74% (279/376) Updating files: 75% (282/376) Updating files: 76% (286/376) Updating files: 77% (290/376) Updating files: 78% (294/376) Updating files: 78% (296/376) Updating files: 79% (298/376) Updating files: 80% (301/376) Updating files: 81% (305/376) Updating files: 82% (309/376) Updating files: 83% (313/376) Updating files: 84% (316/376) Updating files: 85% (320/376) Updating files: 86% (324/376) Updating files: 87% (328/376) Updating files: 88% (331/376) Updating files: 89% (335/376) Updating files: 90% (339/376) Updating files: 91% (343/376) Updating files: 92% (346/376) Updating files: 93% (350/376) Updating files: 94% (354/376) Updating files: 95% (358/376) Updating files: 96% (361/376) Updating files: 97% (365/376) Updating files: 98% (369/376) Updating files: 99% (373/376) Updating files: 100% (376/376) Updating files: 100% (376/376), done. 2023-10-30T13:58:53.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:58:53.894 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:58:54.287 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 237f3f1e8d8 v0.61 2023-10-30T13:58:54.296 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.61 2023-10-30T13:58:54.432 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 62 -eq 48 ] 2023-10-30T13:58:54.432 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.62 2023-10-30T13:58:54.432 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.62 2023-10-30T13:58:54.433 INFO:tasks.workunit.client.0.smithi046.stdout:v0.62 2023-10-30T13:58:54.433 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.62 2023-10-30T13:58:57.595 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 9382379cc6b v0.62 2023-10-30T13:58:57.598 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.62 2023-10-30T13:58:57.941 INFO:tasks.workunit.client.0.smithi046.stdout:v0.63 2023-10-30T13:58:57.941 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 63 -eq 48 ] 2023-10-30T13:58:57.942 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.63 2023-10-30T13:58:57.942 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.63 2023-10-30T13:58:57.942 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.63 2023-10-30T13:59:01.437 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 30% (45/146) Updating files: 31% (46/146) Updating files: 32% (47/146) Updating files: 33% (49/146) Updating files: 34% (50/146) Updating files: 35% (52/146) Updating files: 36% (53/146) Updating files: 37% (55/146) Updating files: 38% (56/146) Updating files: 39% (57/146) Updating files: 40% (59/146) Updating files: 41% (60/146) Updating files: 42% (62/146) Updating files: 43% (63/146) Updating files: 44% (65/146) Updating files: 45% (66/146) Updating files: 46% (68/146) Updating files: 47% (69/146) Updating files: 48% (71/146) Updating files: 49% (72/146) Updating files: 50% (73/146) Updating files: 51% (75/146) Updating files: 52% (76/146) Updating files: 53% (78/146) Updating files: 54% (79/146) Updating files: 55% (81/146) Updating files: 56% (82/146) Updating files: 57% (84/146) Updating files: 58% (85/146) Updating files: 59% (87/146) Updating files: 60% (88/146) Updating files: 61% (90/146) Updating files: 62% (91/146) Updating files: 63% (92/146) Updating files: 64% (94/146) Updating files: 65% (95/146) Updating files: 66% (97/146) Updating files: 67% (98/146) Updating files: 68% (100/146) Updating files: 69% (101/146) Updating files: 70% (103/146) Updating files: 71% (104/146) Updating files: 72% (106/146) Updating files: 73% (107/146) Updating files: 74% (109/146) Updating files: 75% (110/146) Updating files: 76% (111/146) Updating files: 77% (113/146) Updating files: 78% (114/146) Updating files: 79% (116/146) Updating files: 80% (117/146) Updating files: 81% (119/146) Updating files: 82% (120/146) Updating files: 83% (122/146) Updating files: 84% (123/146) Updating files: 85% (125/146) Updating files: 86% (126/146) Updating files: 87% (128/146) Updating files: 88% (129/146) Updating files: 89% (130/146) Updating files: 90% (132/146) Updating files: 91% (133/146) Updating files: 92% (135/146) Updating files: 93% (136/146) Updating files: 94% (138/146) Updating files: 95% (139/146) Updating files: 96% (141/146) Updating files: 97% (142/146) Updating files: 98% (144/146) Updating files: 99% (145/146) Updating files: 100% (146/146) Updating files: 100% (146/146), done. 2023-10-30T13:59:01.677 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 054e96cf79e v0.63 2023-10-30T13:59:01.684 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.63 2023-10-30T13:59:01.722 INFO:tasks.workunit.client.0.smithi046.stdout:v0.64 2023-10-30T13:59:01.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 64 -eq 48 ] 2023-10-30T13:59:01.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.64 2023-10-30T13:59:01.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.64 2023-10-30T13:59:01.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.64 2023-10-30T13:59:21.857 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 33% (48/143) Updating files: 34% (49/143) Updating files: 35% (51/143) Updating files: 36% (52/143) Updating files: 37% (53/143) Updating files: 38% (55/143) Updating files: 39% (56/143) Updating files: 40% (58/143) Updating files: 41% (59/143) Updating files: 42% (61/143) Updating files: 43% (62/143) Updating files: 44% (63/143) Updating files: 44% (64/143) Updating files: 45% (65/143) Updating files: 46% (66/143) Updating files: 47% (68/143) Updating files: 48% (69/143) Updating files: 49% (71/143) Updating files: 50% (72/143) Updating files: 51% (73/143) Updating files: 52% (75/143) Updating files: 53% (76/143) Updating files: 54% (78/143) Updating files: 55% (79/143) Updating files: 56% (81/143) Updating files: 57% (82/143) Updating files: 58% (83/143) Updating files: 59% (85/143) Updating files: 60% (86/143) Updating files: 61% (88/143) Updating files: 62% (89/143) Updating files: 63% (91/143) Updating files: 64% (92/143) Updating files: 65% (93/143) Updating files: 66% (95/143) Updating files: 67% (96/143) Updating files: 68% (98/143) Updating files: 69% (99/143) Updating files: 70% (101/143) Updating files: 71% (102/143) Updating files: 72% (103/143) Updating files: 73% (105/143) Updating files: 74% (106/143) Updating files: 75% (108/143) Updating files: 76% (109/143) Updating files: 77% (111/143) Updating files: 78% (112/143) Updating files: 79% (113/143) Updating files: 80% (115/143) Updating files: 81% (116/143) Updating files: 82% (118/143) Updating files: 83% (119/143) Updating files: 84% (121/143) Updating files: 85% (122/143) Updating files: 86% (123/143) Updating files: 87% (125/143) Updating files: 88% (126/143) Updating files: 89% (128/143) Updating files: 90% (129/143) Updating files: 91% (131/143) Updating files: 92% (132/143) Updating files: 93% (133/143) Updating files: 94% (135/143) Updating files: 95% (136/143) Updating files: 96% (138/143) Updating files: 97% (139/143) Updating files: 98% (141/143) Updating files: 99% (142/143) Updating files: 100% (143/143) Updating files: 100% (143/143), done. 2023-10-30T13:59:21.947 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 42e06c12db6 v0.64 2023-10-30T13:59:21.954 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.64 2023-10-30T13:59:22.012 INFO:tasks.workunit.client.0.smithi046.stdout:v0.65 2023-10-30T13:59:22.012 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 65 -eq 48 ] 2023-10-30T13:59:22.013 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.65 2023-10-30T13:59:22.013 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.65 2023-10-30T13:59:22.013 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.65 2023-10-30T13:59:24.011 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:59:24.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:59:29.271 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 8% (17/203) Updating files: 9% (19/203) Updating files: 10% (21/203) Updating files: 11% (23/203) Updating files: 12% (25/203) Updating files: 13% (27/203) Updating files: 14% (29/203) Updating files: 15% (31/203) Updating files: 16% (33/203) Updating files: 17% (35/203) Updating files: 18% (37/203) Updating files: 19% (39/203) Updating files: 20% (41/203) Updating files: 21% (43/203) Updating files: 22% (45/203) Updating files: 23% (47/203) Updating files: 24% (49/203) Updating files: 25% (51/203) Updating files: 26% (53/203) Updating files: 27% (55/203) Updating files: 28% (57/203) Updating files: 29% (59/203) Updating files: 30% (61/203) Updating files: 31% (63/203) Updating files: 32% (65/203) Updating files: 33% (67/203) Updating files: 34% (70/203) Updating files: 35% (72/203) Updating files: 36% (74/203) Updating files: 37% (76/203) Updating files: 38% (78/203) Updating files: 39% (80/203) Updating files: 40% (82/203) Updating files: 41% (84/203) Updating files: 42% (86/203) Updating files: 43% (88/203) Updating files: 44% (90/203) Updating files: 45% (92/203) Updating files: 46% (94/203) Updating files: 47% (96/203) Updating files: 48% (98/203) Updating files: 49% (100/203) Updating files: 50% (102/203) Updating files: 51% (104/203) Updating files: 52% (106/203) Updating files: 53% (108/203) Updating files: 54% (110/203) Updating files: 55% (112/203) Updating files: 56% (114/203) Updating files: 57% (116/203) Updating files: 58% (118/203) Updating files: 59% (120/203) Updating files: 60% (122/203) Updating files: 61% (124/203) Updating files: 62% (126/203) Updating files: 63% (128/203) Updating files: 63% (129/203) Updating files: 64% (130/203) Updating files: 65% (132/203) Updating files: 66% (134/203) Updating files: 67% (137/203) Updating files: 68% (139/203) Updating files: 69% (141/203) Updating files: 70% (143/203) Updating files: 71% (145/203) Updating files: 72% (147/203) Updating files: 73% (149/203) Updating files: 74% (151/203) Updating files: 75% (153/203) Updating files: 76% (155/203) Updating files: 77% (157/203) Updating files: 78% (159/203) Updating files: 79% (161/203) Updating files: 80% (163/203) Updating files: 81% (165/203) Updating files: 82% (167/203) Updating files: 83% (169/203) Updating files: 84% (171/203) Updating files: 85% (173/203) Updating files: 86% (175/203) Updating files: 87% (177/203) Updating files: 88% (179/203) Updating files: 89% (181/203) Updating files: 90% (183/203) Updating files: 91% (185/203) Updating files: 92% (187/203) Updating files: 93% (189/203) Updating files: 94% (191/203) Updating files: 95% (193/203) Updating files: 96% (195/203) Updating files: 97% (197/203) Updating files: 98% (199/203) Updating files: 98% (200/203) Updating files: 99% (201/203) Updating files: 100% (203/203) Updating files: 100% (203/203), done. 2023-10-30T13:59:29.365 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at c2d517ef966 v0.65 2023-10-30T13:59:29.371 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.65 2023-10-30T13:59:29.429 INFO:tasks.workunit.client.0.smithi046.stdout:v0.66 2023-10-30T13:59:29.429 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 66 -eq 48 ] 2023-10-30T13:59:29.429 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.66 2023-10-30T13:59:29.429 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.66 2023-10-30T13:59:29.430 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.66 2023-10-30T13:59:41.843 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 38% (59/152) Updating files: 39% (60/152) Updating files: 40% (61/152) Updating files: 41% (63/152) Updating files: 42% (64/152) Updating files: 43% (66/152) Updating files: 44% (67/152) Updating files: 45% (69/152) Updating files: 46% (70/152) Updating files: 47% (72/152) Updating files: 48% (73/152) Updating files: 49% (75/152) Updating files: 50% (76/152) Updating files: 51% (78/152) Updating files: 52% (80/152) Updating files: 53% (81/152) Updating files: 54% (83/152) Updating files: 55% (84/152) Updating files: 56% (86/152) Updating files: 57% (87/152) Updating files: 58% (89/152) Updating files: 59% (90/152) Updating files: 60% (92/152) Updating files: 61% (93/152) Updating files: 62% (95/152) Updating files: 63% (96/152) Updating files: 64% (98/152) Updating files: 65% (99/152) Updating files: 66% (101/152) Updating files: 67% (102/152) Updating files: 68% (104/152) Updating files: 69% (105/152) Updating files: 70% (107/152) Updating files: 71% (108/152) Updating files: 72% (110/152) Updating files: 73% (111/152) Updating files: 74% (113/152) Updating files: 75% (114/152) Updating files: 76% (116/152) Updating files: 77% (118/152) Updating files: 78% (119/152) Updating files: 79% (121/152) Updating files: 80% (122/152) Updating files: 81% (124/152) Updating files: 82% (125/152) Updating files: 83% (127/152) Updating files: 84% (128/152) Updating files: 85% (130/152) Updating files: 86% (131/152) Updating files: 87% (133/152) Updating files: 88% (134/152) Updating files: 89% (136/152) Updating files: 90% (137/152) Updating files: 91% (139/152) Updating files: 92% (140/152) Updating files: 93% (142/152) Updating files: 94% (143/152) Updating files: 95% (145/152) Updating files: 96% (146/152) Updating files: 97% (148/152) Updating files: 98% (149/152) Updating files: 99% (151/152) Updating files: 100% (152/152) Updating files: 100% (152/152), done. 2023-10-30T13:59:42.177 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at b6b48dbefad v0.66 2023-10-30T13:59:42.183 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.66 2023-10-30T13:59:42.709 INFO:tasks.workunit.client.0.smithi046.stdout:v0.67 2023-10-30T13:59:42.710 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 67 -eq 48 ] 2023-10-30T13:59:42.710 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.67 2023-10-30T13:59:42.710 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.67 2023-10-30T13:59:42.710 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.67 2023-10-30T13:59:51.636 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 16% (59/355) Updating files: 17% (61/355) Updating files: 18% (64/355) Updating files: 19% (68/355) Updating files: 20% (71/355) Updating files: 21% (75/355) Updating files: 22% (79/355) Updating files: 23% (82/355) Updating files: 24% (86/355) Updating files: 25% (89/355) Updating files: 26% (93/355) Updating files: 27% (96/355) Updating files: 28% (100/355) Updating files: 29% (103/355) Updating files: 29% (105/355) Updating files: 30% (107/355) Updating files: 31% (111/355) Updating files: 32% (114/355) Updating files: 33% (118/355) Updating files: 34% (121/355) Updating files: 35% (125/355) Updating files: 36% (128/355) Updating files: 37% (132/355) Updating files: 38% (135/355) Updating files: 39% (139/355) Updating files: 40% (142/355) Updating files: 41% (146/355) Updating files: 42% (150/355) Updating files: 43% (153/355) Updating files: 44% (157/355) Updating files: 45% (160/355) Updating files: 46% (164/355) Updating files: 47% (167/355) Updating files: 48% (171/355) Updating files: 49% (174/355) Updating files: 50% (178/355) Updating files: 50% (180/355) Updating files: 51% (182/355) Updating files: 52% (185/355) Updating files: 53% (189/355) Updating files: 54% (192/355) Updating files: 55% (196/355) Updating files: 56% (199/355) Updating files: 57% (203/355) Updating files: 58% (206/355) Updating files: 59% (210/355) Updating files: 60% (213/355) Updating files: 61% (217/355) Updating files: 62% (221/355) Updating files: 63% (224/355) Updating files: 64% (228/355) Updating files: 65% (231/355) Updating files: 66% (235/355) Updating files: 67% (238/355) Updating files: 67% (239/355) Updating files: 68% (242/355) Updating files: 69% (245/355) Updating files: 70% (249/355) Updating files: 71% (253/355) Updating files: 72% (256/355) Updating files: 73% (260/355) Updating files: 74% (263/355) Updating files: 75% (267/355) Updating files: 76% (270/355) Updating files: 77% (274/355) Updating files: 78% (277/355) Updating files: 78% (279/355) Updating files: 79% (281/355) Updating files: 80% (284/355) Updating files: 81% (288/355) Updating files: 82% (292/355) Updating files: 82% (293/355) Updating files: 83% (295/355) Updating files: 84% (299/355) Updating files: 85% (302/355) Updating files: 86% (306/355) Updating files: 87% (309/355) Updating files: 88% (313/355) Updating files: 89% (316/355) Updating files: 90% (320/355) Updating files: 91% (324/355) Updating files: 92% (327/355) Updating files: 93% (331/355) Updating files: 94% (334/355) Updating files: 95% (338/355) Updating files: 95% (340/355) Updating files: 96% (341/355) Updating files: 97% (345/355) Updating files: 98% (348/355) Updating files: 99% (352/355) Updating files: 100% (355/355) Updating files: 100% (355/355), done. 2023-10-30T13:59:52.742 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at e3b7bc5bce8 v0.67 2023-10-30T13:59:52.750 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.67 2023-10-30T13:59:52.817 INFO:tasks.workunit.client.0.smithi046.stdout:v0.68 2023-10-30T13:59:52.818 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 68 -eq 48 ] 2023-10-30T13:59:52.818 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.68 2023-10-30T13:59:52.818 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.68 2023-10-30T13:59:52.818 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.68 2023-10-30T13:59:54.066 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:59:54.070 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:59:58.726 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 25% (61/241) Updating files: 26% (63/241) Updating files: 27% (66/241) Updating files: 28% (68/241) Updating files: 29% (70/241) Updating files: 30% (73/241) Updating files: 31% (75/241) Updating files: 32% (78/241) Updating files: 33% (80/241) Updating files: 34% (82/241) Updating files: 35% (85/241) Updating files: 36% (87/241) Updating files: 37% (90/241) Updating files: 38% (92/241) Updating files: 39% (94/241) Updating files: 40% (97/241) Updating files: 41% (99/241) Updating files: 42% (102/241) Updating files: 43% (104/241) Updating files: 43% (105/241) Updating files: 44% (107/241) Updating files: 45% (109/241) Updating files: 46% (111/241) Updating files: 47% (114/241) Updating files: 48% (116/241) Updating files: 49% (119/241) Updating files: 50% (121/241) Updating files: 51% (123/241) Updating files: 52% (126/241) Updating files: 53% (128/241) Updating files: 54% (131/241) Updating files: 55% (133/241) Updating files: 56% (135/241) Updating files: 57% (138/241) Updating files: 58% (140/241) Updating files: 59% (143/241) Updating files: 60% (145/241) Updating files: 61% (148/241) Updating files: 62% (150/241) Updating files: 63% (152/241) Updating files: 64% (155/241) Updating files: 65% (157/241) Updating files: 66% (160/241) Updating files: 67% (162/241) Updating files: 68% (164/241) Updating files: 69% (167/241) Updating files: 70% (169/241) Updating files: 71% (172/241) Updating files: 72% (174/241) Updating files: 73% (176/241) Updating files: 74% (179/241) Updating files: 75% (181/241) Updating files: 76% (184/241) Updating files: 77% (186/241) Updating files: 78% (188/241) Updating files: 79% (191/241) Updating files: 80% (193/241) Updating files: 81% (196/241) Updating files: 82% (198/241) Updating files: 83% (201/241) Updating files: 84% (203/241) Updating files: 85% (205/241) Updating files: 86% (208/241) Updating files: 87% (210/241) Updating files: 88% (213/241) Updating files: 89% (215/241) Updating files: 89% (216/241) Updating files: 90% (217/241) Updating files: 91% (220/241) Updating files: 92% (222/241) Updating files: 93% (225/241) Updating files: 94% (227/241) Updating files: 95% (229/241) Updating files: 96% (232/241) Updating files: 97% (234/241) Updating files: 98% (237/241) Updating files: 99% (239/241) Updating files: 100% (241/241) Updating files: 100% (241/241), done. 2023-10-30T13:59:59.451 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at b4cf0f2574b v0.68 2023-10-30T13:59:59.457 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.68 2023-10-30T13:59:59.530 INFO:tasks.workunit.client.0.smithi046.stdout:v0.69 2023-10-30T13:59:59.530 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 69 -eq 48 ] 2023-10-30T13:59:59.531 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.69 2023-10-30T13:59:59.531 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.69 2023-10-30T13:59:59.531 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.69 2023-10-30T14:00:16.195 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 77% (120/155) Updating files: 78% (121/155) Updating files: 79% (123/155) Updating files: 80% (124/155) Updating files: 81% (126/155) Updating files: 82% (128/155) Updating files: 83% (129/155) Updating files: 84% (131/155) Updating files: 85% (132/155) Updating files: 86% (134/155) Updating files: 87% (135/155) Updating files: 88% (137/155) Updating files: 89% (138/155) Updating files: 90% (140/155) Updating files: 91% (142/155) Updating files: 92% (143/155) Updating files: 93% (145/155) Updating files: 94% (146/155) Updating files: 95% (148/155) Updating files: 96% (149/155) Updating files: 97% (151/155) Updating files: 98% (152/155) Updating files: 99% (154/155) Updating files: 100% (155/155) Updating files: 100% (155/155), done. 2023-10-30T14:00:16.789 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:16.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.939+0000 7f27c8ac9640 1 Processor -- start 2023-10-30T14:00:16.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.939+0000 7f27c8ac9640 1 -- start start 2023-10-30T14:00:16.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c8ac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27c406f7a0 0x7f27c406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c8ac9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f27c40c6bd0 0x7f27c4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c8ac9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27c4071110 0x7f27c4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c8ac9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c40700c0 con 0x7f27c4071110 2023-10-30T14:00:16.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c8ac9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c406a3d0 con 0x7f27c40c6bd0 2023-10-30T14:00:16.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c8ac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c4066cf0 con 0x7f27c406f7a0 2023-10-30T14:00:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27c4071110 0x7f27c4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27c4071110 0x7f27c4158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35422/0 (socket says 172.21.15.46:35422) 2023-10-30T14:00:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c2d9d640 1 -- 172.21.15.46:0/1132813102 learned_addr learned my addr 172.21.15.46:0/1132813102 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:16.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27bbfff640 1 --2- 172.21.15.46:0/1132813102 >> v2:172.21.15.154:3300/0 conn(0x7f27c40c6bd0 0x7f27c4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c259c640 1 --2- 172.21.15.46:0/1132813102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27c406f7a0 0x7f27c406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c2d9d640 1 -- 172.21.15.46:0/1132813102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27c406f7a0 msgr2=0x7f27c406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c2d9d640 1 --2- 172.21.15.46:0/1132813102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27c406f7a0 0x7f27c406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c2d9d640 1 -- 172.21.15.46:0/1132813102 >> v2:172.21.15.154:3300/0 conn(0x7f27c40c6bd0 msgr2=0x7f27c4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:16.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c2d9d640 1 --2- 172.21.15.46:0/1132813102 >> v2:172.21.15.154:3300/0 conn(0x7f27c40c6bd0 0x7f27c4069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.943+0000 7f27c2d9d640 1 -- 172.21.15.46:0/1132813102 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c40c70d0 con 0x7f27c4071110 2023-10-30T14:00:16.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c2d9d640 1 --2- 172.21.15.46:0/1132813102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27c4071110 0x7f27c4158110 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f27c40b1210 tx=0x7f27bc01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd556368ac9d3a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:16.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c1d9b640 1 -- 172.21.15.46:0/1132813102 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc0af400 con 0x7f27c4071110 2023-10-30T14:00:16.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c1d9b640 1 -- 172.21.15.46:0/1132813102 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc0ab050 con 0x7f27c4071110 2023-10-30T14:00:16.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c1d9b640 1 -- 172.21.15.46:0/1132813102 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc0af780 con 0x7f27c4071110 2023-10-30T14:00:16.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 -- 172.21.15.46:0/1132813102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27c4071110 msgr2=0x7f27c4158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:16.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 --2- 172.21.15.46:0/1132813102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27c4071110 0x7f27c4158110 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f27c40b1210 tx=0x7f27bc01f1c0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 -- 172.21.15.46:0/1132813102 shutdown_connections 2023-10-30T14:00:16.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 --2- 172.21.15.46:0/1132813102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27c4071110 0x7f27c4158110 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 --2- 172.21.15.46:0/1132813102 >> v2:172.21.15.154:3300/0 conn(0x7f27c40c6bd0 0x7f27c4069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 --2- 172.21.15.46:0/1132813102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27c406f7a0 0x7f27c406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 -- 172.21.15.46:0/1132813102 >> 172.21.15.46:0/1132813102 conn(0x7f27c40c7910 msgr2=0x7f27c40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:16.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 -- 172.21.15.46:0/1132813102 shutdown_connections 2023-10-30T14:00:16.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 -- 172.21.15.46:0/1132813102 wait complete. 2023-10-30T14:00:16.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 Processor -- start 2023-10-30T14:00:16.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 -- start start 2023-10-30T14:00:16.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27c406f7a0 0x7f27c40ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c4071110 0x7f27c40c1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27c40c6bd0 0x7f27c41eefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c40c01f0 con 0x7f27c406f7a0 2023-10-30T14:00:16.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c40c0dc0 con 0x7f27c4071110 2023-10-30T14:00:16.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.947+0000 7f27c8ac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c40c1380 con 0x7f27c40c6bd0 2023-10-30T14:00:16.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27c406f7a0 0x7f27c40ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27c406f7a0 0x7f27c40ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35428/0 (socket says 172.21.15.46:35428) 2023-10-30T14:00:16.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c259c640 1 -- 172.21.15.46:0/360475999 learned_addr learned my addr 172.21.15.46:0/360475999 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:16.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c2d9d640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27c40c6bd0 0x7f27c41eefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27bbfff640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c4071110 0x7f27c40c1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c259c640 1 -- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27c40c6bd0 msgr2=0x7f27c41eefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:16.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c259c640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27c40c6bd0 0x7f27c41eefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c259c640 1 -- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c4071110 msgr2=0x7f27c40c1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:16.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c259c640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c4071110 0x7f27c40c1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:16.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c259c640 1 -- 172.21.15.46:0/360475999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc009b30 con 0x7f27c406f7a0 2023-10-30T14:00:16.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27bbfff640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c4071110 0x7f27c40c1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:00:16.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c259c640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27c406f7a0 0x7f27c40ca7d0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f27b400d830 tx=0x7f27b400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:16.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27baffd640 1 -- 172.21.15.46:0/360475999 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b400be10 con 0x7f27c406f7a0 2023-10-30T14:00:16.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27baffd640 1 -- 172.21.15.46:0/360475999 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b40098b0 con 0x7f27c406f7a0 2023-10-30T14:00:16.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27baffd640 1 -- 172.21.15.46:0/360475999 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4009a40 con 0x7f27c406f7a0 2023-10-30T14:00:16.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c8ac9640 1 -- 172.21.15.46:0/360475999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c41efbc0 con 0x7f27c406f7a0 2023-10-30T14:00:16.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c8ac9640 1 -- 172.21.15.46:0/360475999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c41f3030 con 0x7f27c406f7a0 2023-10-30T14:00:16.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27c8ac9640 1 -- 172.21.15.46:0/360475999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c404e170 con 0x7f27c406f7a0 2023-10-30T14:00:16.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.951+0000 7f27baffd640 1 -- 172.21.15.46:0/360475999 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f27b4009ba0 con 0x7f27c406f7a0 2023-10-30T14:00:16.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.955+0000 7f27baffd640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27a40664a0 0x7f27a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:16.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.955+0000 7f27baffd640 1 -- 172.21.15.46:0/360475999 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f27b4084810 con 0x7f27c406f7a0 2023-10-30T14:00:16.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.955+0000 7f27bbfff640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27a40664a0 0x7f27a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:16.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.955+0000 7f27bbfff640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27a40664a0 0x7f27a4068960 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f27c41e9630 tx=0x7f27ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:16.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:16.959+0000 7f27baffd640 1 -- 172.21.15.46:0/360475999 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f27b40515f0 con 0x7f27c406f7a0 2023-10-30T14:00:17.138 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 6ca6f2f9f75 v0.69 2023-10-30T14:00:17.151 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.69 2023-10-30T14:00:17.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.159+0000 7f27c8ac9640 1 -- 172.21.15.46:0/360475999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27c41510d0 con 0x7f27c406f7a0 2023-10-30T14:00:17.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.159+0000 7f27baffd640 1 -- 172.21.15.46:0/360475999 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f27b4054c10 con 0x7f27c406f7a0 2023-10-30T14:00:17.161 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:17.161 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6809,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:23.718831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:17.161 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 158 2023-10-30T14:00:17.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.163+0000 7f27c8ac9640 1 -- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27a40664a0 msgr2=0x7f27a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:17.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.163+0000 7f27c8ac9640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27a40664a0 0x7f27a4068960 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f27c41e9630 tx=0x7f27ac008040 comp rx=0 tx=0).stop 2023-10-30T14:00:17.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.163+0000 7f27c8ac9640 1 -- 172.21.15.46:0/360475999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27c406f7a0 msgr2=0x7f27c40ca7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:17.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.163+0000 7f27c8ac9640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27c406f7a0 0x7f27c40ca7d0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f27b400d830 tx=0x7f27b400dcf0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.163+0000 7f27c8ac9640 1 -- 172.21.15.46:0/360475999 shutdown_connections 2023-10-30T14:00:17.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.163+0000 7f27c8ac9640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27a40664a0 0x7f27a4068960 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.163+0000 7f27c8ac9640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27c40c6bd0 0x7f27c41eefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.163+0000 7f27c8ac9640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c4071110 0x7f27c40c1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.163+0000 7f27c8ac9640 1 --2- 172.21.15.46:0/360475999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27c406f7a0 0x7f27c40ca7d0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.163+0000 7f27c8ac9640 1 -- 172.21.15.46:0/360475999 >> 172.21.15.46:0/360475999 conn(0x7f27c40c7910 msgr2=0x7f27c40c7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.163+0000 7f27c8ac9640 1 -- 172.21.15.46:0/360475999 shutdown_connections 2023-10-30T14:00:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.163+0000 7f27c8ac9640 1 -- 172.21.15.46:0/360475999 wait complete. 2023-10-30T14:00:17.193 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=1) 2023-10-30T14:00:17.194 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T14:00:17.194 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:00:17.220 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T14:00:17.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:17.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.403+0000 7fde859d3640 1 Processor -- start 2023-10-30T14:00:17.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.403+0000 7fde859d3640 1 -- start start 2023-10-30T14:00:17.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.403+0000 7fde859d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8006f7a0 0x7fde8006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:17.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.403+0000 7fde859d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde800c6bd0 0x7fde80069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:17.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.403+0000 7fde859d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde80071110 0x7fde80158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:17.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.403+0000 7fde859d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde800700c0 con 0x7fde80071110 2023-10-30T14:00:17.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.403+0000 7fde859d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde8006a3d0 con 0x7fde800c6bd0 2023-10-30T14:00:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.403+0000 7fde859d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde80066cf0 con 0x7fde8006f7a0 2023-10-30T14:00:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.403+0000 7fde7f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde80071110 0x7fde80158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.403+0000 7fde7f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde80071110 0x7fde80158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35440/0 (socket says 172.21.15.46:35440) 2023-10-30T14:00:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.403+0000 7fde7f7fe640 1 -- 172.21.15.46:0/2589451058 learned_addr learned my addr 172.21.15.46:0/2589451058 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:17.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.403+0000 7fde7effd640 1 --2- 172.21.15.46:0/2589451058 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8006f7a0 0x7fde8006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:17.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde77fff640 1 --2- 172.21.15.46:0/2589451058 >> v2:172.21.15.154:3300/0 conn(0x7fde800c6bd0 0x7fde80069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:17.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde7f7fe640 1 -- 172.21.15.46:0/2589451058 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8006f7a0 msgr2=0x7fde8006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:17.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde7f7fe640 1 --2- 172.21.15.46:0/2589451058 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8006f7a0 0x7fde8006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde7f7fe640 1 -- 172.21.15.46:0/2589451058 >> v2:172.21.15.154:3300/0 conn(0x7fde800c6bd0 msgr2=0x7fde80069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:17.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde7f7fe640 1 --2- 172.21.15.46:0/2589451058 >> v2:172.21.15.154:3300/0 conn(0x7fde800c6bd0 0x7fde80069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde7f7fe640 1 -- 172.21.15.46:0/2589451058 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800c70d0 con 0x7fde80071110 2023-10-30T14:00:17.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde7f7fe640 1 --2- 172.21.15.46:0/2589451058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde80071110 0x7fde80158110 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fde800b1210 tx=0x7fde7801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b3b7898cde99200 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:17.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde7e7fc640 1 -- 172.21.15.46:0/2589451058 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde780af400 con 0x7fde80071110 2023-10-30T14:00:17.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde7e7fc640 1 -- 172.21.15.46:0/2589451058 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde780ab050 con 0x7fde80071110 2023-10-30T14:00:17.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde7e7fc640 1 -- 172.21.15.46:0/2589451058 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde780af780 con 0x7fde80071110 2023-10-30T14:00:17.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde859d3640 1 -- 172.21.15.46:0/2589451058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde80071110 msgr2=0x7fde80158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:17.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde859d3640 1 --2- 172.21.15.46:0/2589451058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde80071110 0x7fde80158110 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fde800b1210 tx=0x7fde7801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde859d3640 1 -- 172.21.15.46:0/2589451058 shutdown_connections 2023-10-30T14:00:17.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde859d3640 1 --2- 172.21.15.46:0/2589451058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde80071110 0x7fde80158110 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde859d3640 1 --2- 172.21.15.46:0/2589451058 >> v2:172.21.15.154:3300/0 conn(0x7fde800c6bd0 0x7fde80069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde859d3640 1 --2- 172.21.15.46:0/2589451058 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8006f7a0 0x7fde8006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.407+0000 7fde859d3640 1 -- 172.21.15.46:0/2589451058 >> 172.21.15.46:0/2589451058 conn(0x7fde800c7910 msgr2=0x7fde800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:17.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde859d3640 1 -- 172.21.15.46:0/2589451058 shutdown_connections 2023-10-30T14:00:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde859d3640 1 -- 172.21.15.46:0/2589451058 wait complete. 2023-10-30T14:00:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde859d3640 1 Processor -- start 2023-10-30T14:00:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde859d3640 1 -- start start 2023-10-30T14:00:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde859d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde8006f7a0 0x7fde801f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde859d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde80071110 0x7fde800c1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde859d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde800c6bd0 0x7fde800c18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde859d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800c0ee0 con 0x7fde8006f7a0 2023-10-30T14:00:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde859d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800af210 con 0x7fde800c6bd0 2023-10-30T14:00:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde859d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde800c02d0 con 0x7fde80071110 2023-10-30T14:00:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde8006f7a0 0x7fde801f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde8006f7a0 0x7fde801f2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35448/0 (socket says 172.21.15.46:35448) 2023-10-30T14:00:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7effd640 1 -- 172.21.15.46:0/2417932264 learned_addr learned my addr 172.21.15.46:0/2417932264 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde77fff640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde80071110 0x7fde800c1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7f7fe640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde800c6bd0 0x7fde800c18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7effd640 1 -- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde80071110 msgr2=0x7fde800c1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7effd640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde80071110 0x7fde800c1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7effd640 1 -- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde800c6bd0 msgr2=0x7fde800c18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7effd640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde800c6bd0 0x7fde800c18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7effd640 1 -- 172.21.15.46:0/2417932264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde78009b30 con 0x7fde8006f7a0 2023-10-30T14:00:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7effd640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde8006f7a0 0x7fde801f2380 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fde7000d830 tx=0x7fde7000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7cff9640 1 -- 172.21.15.46:0/2417932264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7000be10 con 0x7fde8006f7a0 2023-10-30T14:00:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7cff9640 1 -- 172.21.15.46:0/2417932264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde700098b0 con 0x7fde8006f7a0 2023-10-30T14:00:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde7cff9640 1 -- 172.21.15.46:0/2417932264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde70009a40 con 0x7fde8006f7a0 2023-10-30T14:00:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde859d3640 1 -- 172.21.15.46:0/2417932264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde800c05b0 con 0x7fde8006f7a0 2023-10-30T14:00:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.411+0000 7fde859d3640 1 -- 172.21.15.46:0/2417932264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde800c0b70 con 0x7fde8006f7a0 2023-10-30T14:00:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.415+0000 7fde859d3640 1 -- 172.21.15.46:0/2417932264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8004e170 con 0x7fde8006f7a0 2023-10-30T14:00:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.415+0000 7fde7cff9640 1 -- 172.21.15.46:0/2417932264 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fde70009ba0 con 0x7fde8006f7a0 2023-10-30T14:00:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.415+0000 7fde7cff9640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde600664a0 0x7fde60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.415+0000 7fde7cff9640 1 -- 172.21.15.46:0/2417932264 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7fde70084a10 con 0x7fde8006f7a0 2023-10-30T14:00:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.415+0000 7fde77fff640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde600664a0 0x7fde60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.415+0000 7fde77fff640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde600664a0 0x7fde60068960 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fde801f3180 tx=0x7fde68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:17.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.419+0000 7fde7cff9640 1 -- 172.21.15.46:0/2417932264 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fde700516c0 con 0x7fde8006f7a0 2023-10-30T14:00:17.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 -- 172.21.15.46:0/2417932264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde801510d0 con 0x7fde8006f7a0 2023-10-30T14:00:17.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde7cff9640 1 -- 172.21.15.46:0/2417932264 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fde70054ce0 con 0x7fde8006f7a0 2023-10-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6809,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:23.718831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:17.613 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 158 2023-10-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 -- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde600664a0 msgr2=0x7fde60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde600664a0 0x7fde60068960 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fde801f3180 tx=0x7fde68008040 comp rx=0 tx=0).stop 2023-10-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 -- 172.21.15.46:0/2417932264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde8006f7a0 msgr2=0x7fde801f2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde8006f7a0 0x7fde801f2380 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fde7000d830 tx=0x7fde7000dcf0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 -- 172.21.15.46:0/2417932264 shutdown_connections 2023-10-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fde600664a0 0x7fde60068960 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde800c6bd0 0x7fde800c18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde80071110 0x7fde800c1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 --2- 172.21.15.46:0/2417932264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde8006f7a0 0x7fde801f2380 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:17.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 -- 172.21.15.46:0/2417932264 >> 172.21.15.46:0/2417932264 conn(0x7fde800c7910 msgr2=0x7fde800c7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 -- 172.21.15.46:0/2417932264 shutdown_connections 2023-10-30T14:00:17.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:17.611+0000 7fde859d3640 1 -- 172.21.15.46:0/2417932264 wait complete. 2023-10-30T14:00:17.634 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:00:19.636 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:19.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.787+0000 7f764bbd3640 1 Processor -- start 2023-10-30T14:00:19.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.787+0000 7f764bbd3640 1 -- start start 2023-10-30T14:00:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764bbd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7644156f80 0x7f7644159880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764bbd3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76440c2f20 con 0x7f7644070650 2023-10-30T14:00:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764bbd3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76440c85c0 con 0x7f7644070c90 2023-10-30T14:00:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764bbd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7644157360 con 0x7f7644156f80 2023-10-30T14:00:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764916e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7644070650 0x7f7644071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53424/0 (socket says 172.21.15.46:53424) 2023-10-30T14:00:19.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764916e640 1 -- 172.21.15.46:0/768061242 learned_addr learned my addr 172.21.15.46:0/768061242 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:00:19.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764996f640 1 --2- 172.21.15.46:0/768061242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7644156f80 0x7f7644159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:19.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789917661 0 0) 0x7f76440c2f20 con 0x7f7644070650 2023-10-30T14:00:19.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7630003530 con 0x7f7644070650 2023-10-30T14:00:19.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4111305494 0 0) 0x7f7630003530 con 0x7f7644070650 2023-10-30T14:00:19.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7644156f80 msgr2=0x7f7644159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:19.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 --2- 172.21.15.46:0/768061242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7644156f80 0x7f7644159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:19.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 >> v1:172.21.15.154:6789/0 conn(0x7f7644070c90 legacy=0x7f7644164120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7644071230 con 0x7f7644070650 2023-10-30T14:00:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f7634001680 con 0x7f7644070650 2023-10-30T14:00:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 >> v1:172.21.15.46:6789/0 conn(0x7f7644070650 legacy=0x7f7644071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 --2- 172.21.15.46:0/768061242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76300039b0 0x7f7630003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:19.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 --2- 172.21.15.46:0/768061242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76300047d0 0x7f7630004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:19.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 --2- 172.21.15.46:0/768061242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7644156f80 0x7f7630005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:19.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7644157360 con 0x7f76300039b0 2023-10-30T14:00:19.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763000ba40 con 0x7f76300047d0 2023-10-30T14:00:19.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763000bc00 con 0x7f7644156f80 2023-10-30T14:00:19.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764996f640 1 --2- 172.21.15.46:0/768061242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76300039b0 0x7f7630003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764a170640 1 --2- 172.21.15.46:0/768061242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7644156f80 0x7f7630005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764916e640 1 --2- 172.21.15.46:0/768061242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76300047d0 0x7f7630004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764996f640 1 -- 172.21.15.46:0/768061242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7644156f80 msgr2=0x7f7630005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764996f640 1 --2- 172.21.15.46:0/768061242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7644156f80 0x7f7630005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764996f640 1 -- 172.21.15.46:0/768061242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76300047d0 msgr2=0x7f7630004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764996f640 1 --2- 172.21.15.46:0/768061242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76300047d0 0x7f7630004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.791+0000 7f764996f640 1 -- 172.21.15.46:0/768061242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7630003530 con 0x7f76300039b0 2023-10-30T14:00:19.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.795+0000 7f764996f640 1 --2- 172.21.15.46:0/768061242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76300039b0 0x7f7630003dd0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f7640009ab0 tx=0x7f764001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d904e8eb3e9f45ac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:19.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.795+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76400a5570 con 0x7f76300039b0 2023-10-30T14:00:19.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.795+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76400a56d0 con 0x7f76300039b0 2023-10-30T14:00:19.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.795+0000 7f764896d640 1 -- 172.21.15.46:0/768061242 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76400a5a80 con 0x7f76300039b0 2023-10-30T14:00:19.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.795+0000 7f764bbd3640 1 -- 172.21.15.46:0/768061242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76300039b0 msgr2=0x7f7630003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:19.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.795+0000 7f764bbd3640 1 --2- 172.21.15.46:0/768061242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76300039b0 0x7f7630003dd0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f7640009ab0 tx=0x7f764001ca10 comp rx=0 tx=0).stop 2023-10-30T14:00:19.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.795+0000 7f764a170640 1 -- 172.21.15.46:0/768061242 reap_dead start 2023-10-30T14:00:19.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.795+0000 7f764bbd3640 1 -- 172.21.15.46:0/768061242 shutdown_connections 2023-10-30T14:00:19.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.795+0000 7f764bbd3640 1 -- 172.21.15.46:0/768061242 >> 172.21.15.46:0/768061242 conn(0x7f76440cbeb0 msgr2=0x7f76440cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.795+0000 7f764bbd3640 1 -- 172.21.15.46:0/768061242 shutdown_connections 2023-10-30T14:00:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.795+0000 7f764bbd3640 1 -- 172.21.15.46:0/768061242 wait complete. 2023-10-30T14:00:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764bbd3640 1 Processor -- start 2023-10-30T14:00:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764bbd3640 1 -- start start 2023-10-30T14:00:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764bbd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76300039b0 0x7f76441502d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764bbd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f764414f9c0 0x7f76441ead60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764bbd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76441558e0 0x7f76441eb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764bbd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7644151020 con 0x7f76441558e0 2023-10-30T14:00:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764bbd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76441547b0 con 0x7f764414f9c0 2023-10-30T14:00:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764bbd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76441f3180 con 0x7f76300039b0 2023-10-30T14:00:19.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764a170640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76441558e0 0x7f76441eb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:19.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764a170640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76441558e0 0x7f76441eb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35476/0 (socket says 172.21.15.46:35476) 2023-10-30T14:00:19.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764a170640 1 -- 172.21.15.46:0/4291026855 learned_addr learned my addr 172.21.15.46:0/4291026855 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:19.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764996f640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76300039b0 0x7f76441502d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:19.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764916e640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f764414f9c0 0x7f76441ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:19.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764a170640 1 -- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76300039b0 msgr2=0x7f76441502d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:19.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764a170640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76300039b0 0x7f76441502d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:19.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764a170640 1 -- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f764414f9c0 msgr2=0x7f76441ead60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:19.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764a170640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f764414f9c0 0x7f76441ead60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:19.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764a170640 1 -- 172.21.15.46:0/4291026855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7644154a00 con 0x7f76441558e0 2023-10-30T14:00:19.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764916e640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f764414f9c0 0x7f76441ead60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:00:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764a170640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76441558e0 0x7f76441eb2a0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f763c000f90 tx=0x7f763c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f763a7fc640 1 -- 172.21.15.46:0/4291026855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c006b60 con 0x7f76441558e0 2023-10-30T14:00:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f763a7fc640 1 -- 172.21.15.46:0/4291026855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c006ce0 con 0x7f76441558e0 2023-10-30T14:00:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f763a7fc640 1 -- 172.21.15.46:0/4291026855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c003bc0 con 0x7f76441558e0 2023-10-30T14:00:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764bbd3640 1 -- 172.21.15.46:0/4291026855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7644154ce0 con 0x7f76441558e0 2023-10-30T14:00:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764bbd3640 1 -- 172.21.15.46:0/4291026855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76441552a0 con 0x7f76441558e0 2023-10-30T14:00:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.799+0000 7f764bbd3640 1 -- 172.21.15.46:0/4291026855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764404e170 con 0x7f76441558e0 2023-10-30T14:00:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.803+0000 7f763a7fc640 1 -- 172.21.15.46:0/4291026855 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f763c003d20 con 0x7f76441558e0 2023-10-30T14:00:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.803+0000 7f763a7fc640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76280664a0 0x7f7628068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.803+0000 7f763a7fc640 1 -- 172.21.15.46:0/4291026855 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f763c083d50 con 0x7f76441558e0 2023-10-30T14:00:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.803+0000 7f764996f640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76280664a0 0x7f7628068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.803+0000 7f764996f640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76280664a0 0x7f7628068960 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f76400af000 tx=0x7f76400046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.807+0000 7f763a7fc640 1 -- 172.21.15.46:0/4291026855 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f763c050b40 con 0x7f76441558e0 2023-10-30T14:00:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.995+0000 7f764bbd3640 1 -- 172.21.15.46:0/4291026855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76440c8220 con 0x7f76441558e0 2023-10-30T14:00:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.995+0000 7f763a7fc640 1 -- 172.21.15.46:0/4291026855 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f763c054160 con 0x7f76441558e0 2023-10-30T14:00:19.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:19.999 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6809,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:23.718831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 158 2023-10-30T14:00:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.999+0000 7f764bbd3640 1 -- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76280664a0 msgr2=0x7f7628068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.999+0000 7f764bbd3640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76280664a0 0x7f7628068960 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f76400af000 tx=0x7f76400046d0 comp rx=0 tx=0).stop 2023-10-30T14:00:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.999+0000 7f764bbd3640 1 -- 172.21.15.46:0/4291026855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76441558e0 msgr2=0x7f76441eb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.999+0000 7f764bbd3640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76441558e0 0x7f76441eb2a0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f763c000f90 tx=0x7f763c00ec10 comp rx=0 tx=0).stop 2023-10-30T14:00:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.999+0000 7f764bbd3640 1 -- 172.21.15.46:0/4291026855 shutdown_connections 2023-10-30T14:00:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.999+0000 7f764bbd3640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76280664a0 0x7f7628068960 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.999+0000 7f764bbd3640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76441558e0 0x7f76441eb2a0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.999+0000 7f764bbd3640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f764414f9c0 0x7f76441ead60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.999+0000 7f764bbd3640 1 --2- 172.21.15.46:0/4291026855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76300039b0 0x7f76441502d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.999+0000 7f764bbd3640 1 -- 172.21.15.46:0/4291026855 >> 172.21.15.46:0/4291026855 conn(0x7f76440cbeb0 msgr2=0x7f76440c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.999+0000 7f764bbd3640 1 -- 172.21.15.46:0/4291026855 shutdown_connections 2023-10-30T14:00:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:19.999+0000 7f764bbd3640 1 -- 172.21.15.46:0/4291026855 wait complete. 2023-10-30T14:00:20.020 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:00:22.021 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:22.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.175+0000 7f1f26da7640 1 Processor -- start 2023-10-30T14:00:22.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.175+0000 7f1f26da7640 1 -- start start 2023-10-30T14:00:22.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.175+0000 7f1f26da7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f201538f0 0x7f1f20159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:22.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.175+0000 7f1f26da7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1f2015cd50 0x7f1f200c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:22.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.175+0000 7f1f26da7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2015d300 con 0x7f1f200cefa0 2023-10-30T14:00:22.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.175+0000 7f1f26da7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2015d5e0 con 0x7f1f2015cd50 2023-10-30T14:00:22.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.175+0000 7f1f26da7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2015d8c0 con 0x7f1f201538f0 2023-10-30T14:00:22.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.175+0000 7f1f265a6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1f200cefa0 0x7f1f200cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53432/0 (socket says 172.21.15.46:53432) 2023-10-30T14:00:22.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.175+0000 7f1f265a6640 1 -- 172.21.15.46:0/2361694035 learned_addr learned my addr 172.21.15.46:0/2361694035 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:00:22.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.179+0000 7f1f25da5640 1 --2- 172.21.15.46:0/2361694035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f201538f0 0x7f1f20159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:22.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.179+0000 7f1f255a4640 1 --2- 172.21.15.46:0/2361694035 >> v2:172.21.15.154:3300/0 conn(0x7f1f2015cd50 0x7f1f200c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:22.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.179+0000 7f1f24da3640 1 -- 172.21.15.46:0/2361694035 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508872899 0 0) 0x7f1f2015d300 con 0x7f1f200cefa0 2023-10-30T14:00:22.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.179+0000 7f1f24da3640 1 -- 172.21.15.46:0/2361694035 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f0c003530 con 0x7f1f200cefa0 2023-10-30T14:00:22.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.179+0000 7f1f24da3640 1 -- 172.21.15.46:0/2361694035 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3346503169 0 0) 0x7f1f0c003530 con 0x7f1f200cefa0 2023-10-30T14:00:22.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.179+0000 7f1f25da5640 1 -- 172.21.15.46:0/2361694035 >> v2:172.21.15.154:3300/0 conn(0x7f1f2015cd50 msgr2=0x7f1f200c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:22.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.179+0000 7f1f25da5640 1 --2- 172.21.15.46:0/2361694035 >> v2:172.21.15.154:3300/0 conn(0x7f1f2015cd50 0x7f1f200c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:22.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.179+0000 7f1f25da5640 1 -- 172.21.15.46:0/2361694035 >> v1:172.21.15.46:6789/0 conn(0x7f1f200cefa0 legacy=0x7f1f200cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:22.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.179+0000 7f1f25da5640 1 -- 172.21.15.46:0/2361694035 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2015dba0 con 0x7f1f201538f0 2023-10-30T14:00:22.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.179+0000 7f1f25da5640 1 --2- 172.21.15.46:0/2361694035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f201538f0 0x7f1f20159fb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0097c0 tx=0x7f1f1c01eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78519e330b8d9767 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:22.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f24da3640 1 -- 172.21.15.46:0/2361694035 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18001a00 con 0x7f1f201538f0 2023-10-30T14:00:22.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f24da3640 1 -- 172.21.15.46:0/2361694035 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c0a85d0 con 0x7f1f201538f0 2023-10-30T14:00:22.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f24da3640 1 -- 172.21.15.46:0/2361694035 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c0a8940 con 0x7f1f201538f0 2023-10-30T14:00:22.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 -- 172.21.15.46:0/2361694035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f201538f0 msgr2=0x7f1f20159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:22.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 --2- 172.21.15.46:0/2361694035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f201538f0 0x7f1f20159fb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0097c0 tx=0x7f1f1c01eb90 comp rx=0 tx=0).stop 2023-10-30T14:00:22.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 -- 172.21.15.46:0/2361694035 shutdown_connections 2023-10-30T14:00:22.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 --2- 172.21.15.46:0/2361694035 >> v2:172.21.15.154:3300/0 conn(0x7f1f2015cd50 0x7f1f200c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:22.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 --2- 172.21.15.46:0/2361694035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f201538f0 0x7f1f20159fb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:22.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 -- 172.21.15.46:0/2361694035 >> 172.21.15.46:0/2361694035 conn(0x7f1f2006a640 msgr2=0x7f1f20065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:22.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 -- 172.21.15.46:0/2361694035 shutdown_connections 2023-10-30T14:00:22.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 -- 172.21.15.46:0/2361694035 wait complete. 2023-10-30T14:00:22.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 Processor -- start 2023-10-30T14:00:22.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 -- start start 2023-10-30T14:00:22.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f200cefa0 0x7f1f201ef1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:22.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f201538f0 0x7f1f200b4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:22.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f2015cd50 0x7f1f20157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:22.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f201573a0 con 0x7f1f201538f0 2023-10-30T14:00:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f20157680 con 0x7f1f200cefa0 2023-10-30T14:00:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f26da7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f20146af0 con 0x7f1f2015cd50 2023-10-30T14:00:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f255a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f201538f0 0x7f1f200b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f255a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f201538f0 0x7f1f200b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35490/0 (socket says 172.21.15.46:35490) 2023-10-30T14:00:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.183+0000 7f1f255a4640 1 -- 172.21.15.46:0/4145941676 learned_addr learned my addr 172.21.15.46:0/4145941676 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:22.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f25da5640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f200cefa0 0x7f1f201ef1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f265a6640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f2015cd50 0x7f1f20157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f255a4640 1 -- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f2015cd50 msgr2=0x7f1f20157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f255a4640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f2015cd50 0x7f1f20157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f255a4640 1 -- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f200cefa0 msgr2=0x7f1f201ef1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:22.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f255a4640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f200cefa0 0x7f1f201ef1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:22.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f255a4640 1 -- 172.21.15.46:0/4145941676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c009420 con 0x7f1f201538f0 2023-10-30T14:00:22.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f255a4640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f201538f0 0x7f1f200b4720 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000e8a0 tx=0x7f1f1000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:22.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f167fc640 1 -- 172.21.15.46:0/4145941676 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10009510 con 0x7f1f201538f0 2023-10-30T14:00:22.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f167fc640 1 -- 172.21.15.46:0/4145941676 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1000c940 con 0x7f1f201538f0 2023-10-30T14:00:22.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f26da7640 1 -- 172.21.15.46:0/4145941676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f20146dd0 con 0x7f1f201538f0 2023-10-30T14:00:22.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f167fc640 1 -- 172.21.15.46:0/4145941676 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1000ccb0 con 0x7f1f201538f0 2023-10-30T14:00:22.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f26da7640 1 -- 172.21.15.46:0/4145941676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f20147390 con 0x7f1f201538f0 2023-10-30T14:00:22.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f26da7640 1 -- 172.21.15.46:0/4145941676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f20147580 con 0x7f1f201538f0 2023-10-30T14:00:22.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.187+0000 7f1f167fc640 1 -- 172.21.15.46:0/4145941676 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1f10013470 con 0x7f1f201538f0 2023-10-30T14:00:22.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.191+0000 7f1f167fc640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1f040664a0 0x7f1f04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:22.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.191+0000 7f1f167fc640 1 -- 172.21.15.46:0/4145941676 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f1f100846d0 con 0x7f1f201538f0 2023-10-30T14:00:22.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.191+0000 7f1f25da5640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1f040664a0 0x7f1f04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:22.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.191+0000 7f1f25da5640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1f040664a0 0x7f1f04068960 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0b2000 tx=0x7f1f1c0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:22.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.195+0000 7f1f167fc640 1 -- 172.21.15.46:0/4145941676 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1f10051430 con 0x7f1f201538f0 2023-10-30T14:00:22.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.383+0000 7f1f26da7640 1 -- 172.21.15.46:0/4145941676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f2004e170 con 0x7f1f201538f0 2023-10-30T14:00:22.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.383+0000 7f1f167fc640 1 -- 172.21.15.46:0/4145941676 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f1f10054a50 con 0x7f1f201538f0 2023-10-30T14:00:22.386 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:22.386 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6809,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:23.718831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:22.386 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 158 2023-10-30T14:00:22.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.383+0000 7f1f26da7640 1 -- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1f040664a0 msgr2=0x7f1f04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:22.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.383+0000 7f1f26da7640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1f040664a0 0x7f1f04068960 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0b2000 tx=0x7f1f1c0aa070 comp rx=0 tx=0).stop 2023-10-30T14:00:22.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.383+0000 7f1f26da7640 1 -- 172.21.15.46:0/4145941676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f201538f0 msgr2=0x7f1f200b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:22.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.383+0000 7f1f26da7640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f201538f0 0x7f1f200b4720 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000e8a0 tx=0x7f1f1000eca0 comp rx=0 tx=0).stop 2023-10-30T14:00:22.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.387+0000 7f1f26da7640 1 -- 172.21.15.46:0/4145941676 shutdown_connections 2023-10-30T14:00:22.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.387+0000 7f1f26da7640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1f040664a0 0x7f1f04068960 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:22.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.387+0000 7f1f26da7640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1f2015cd50 0x7f1f20157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:22.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.387+0000 7f1f26da7640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f201538f0 0x7f1f200b4720 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:22.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.387+0000 7f1f26da7640 1 --2- 172.21.15.46:0/4145941676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f200cefa0 0x7f1f201ef1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:22.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.387+0000 7f1f26da7640 1 -- 172.21.15.46:0/4145941676 >> 172.21.15.46:0/4145941676 conn(0x7f1f2006a640 msgr2=0x7f1f200c50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:22.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.387+0000 7f1f26da7640 1 -- 172.21.15.46:0/4145941676 shutdown_connections 2023-10-30T14:00:22.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:22.387+0000 7f1f26da7640 1 -- 172.21.15.46:0/4145941676 wait complete. 2023-10-30T14:00:22.407 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:00:24.130 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:00:24.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:00:24.408 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.559+0000 7fdf8a93e640 1 Processor -- start 2023-10-30T14:00:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf8a93e640 1 -- start start 2023-10-30T14:00:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf8a93e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf8415d6f0 0x7fdf8415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf8a93e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf840cb800 con 0x7fdf840cf160 2023-10-30T14:00:24.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf8a93e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf840cb330 con 0x7fdf841538a0 2023-10-30T14:00:24.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf8a93e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf840cad70 con 0x7fdf8415d6f0 2023-10-30T14:00:24.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf837fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf8415d6f0 0x7fdf8415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:24.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf837fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf8415d6f0 0x7fdf8415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35504/0 (socket says 172.21.15.46:35504) 2023-10-30T14:00:24.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf837fe640 1 -- 172.21.15.46:0/3756609030 learned_addr learned my addr 172.21.15.46:0/3756609030 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:24.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf88edb640 1 --1- 172.21.15.46:0/3756609030 >> v1:172.21.15.46:6789/0 conn(0x7fdf840cf160 0x7fdf84164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53448/0 (socket says 172.21.15.46:53448) 2023-10-30T14:00:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf837fe640 1 -- 172.21.15.46:0/3756609030 >> v1:172.21.15.154:6789/0 conn(0x7fdf841538a0 legacy=0x7fdf840ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf837fe640 1 -- 172.21.15.46:0/3756609030 >> v1:172.21.15.46:6789/0 conn(0x7fdf840cf160 legacy=0x7fdf84164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf82ffd640 1 -- 172.21.15.46:0/3756609030 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982398127 0 0) 0x7fdf840cb800 con 0x7fdf840cf160 2023-10-30T14:00:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.563+0000 7fdf837fe640 1 -- 172.21.15.46:0/3756609030 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf840cb050 con 0x7fdf8415d6f0 2023-10-30T14:00:24.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.567+0000 7fdf837fe640 1 --2- 172.21.15.46:0/3756609030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf8415d6f0 0x7fdf8415a090 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009870 tx=0x7fdf7801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3daa1a4e5ea288be server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:24.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.567+0000 7fdf82ffd640 1 -- 172.21.15.46:0/3756609030 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf780a8a30 con 0x7fdf8415d6f0 2023-10-30T14:00:24.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.567+0000 7fdf82ffd640 1 -- 172.21.15.46:0/3756609030 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf780a8b90 con 0x7fdf8415d6f0 2023-10-30T14:00:24.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.567+0000 7fdf82ffd640 1 -- 172.21.15.46:0/3756609030 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf780b0440 con 0x7fdf8415d6f0 2023-10-30T14:00:24.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.567+0000 7fdf8a93e640 1 -- 172.21.15.46:0/3756609030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf8415d6f0 msgr2=0x7fdf8415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:24.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.567+0000 7fdf8a93e640 1 --2- 172.21.15.46:0/3756609030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf8415d6f0 0x7fdf8415a090 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009870 tx=0x7fdf7801ed30 comp rx=0 tx=0).stop 2023-10-30T14:00:24.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.567+0000 7fdf8a93e640 1 -- 172.21.15.46:0/3756609030 shutdown_connections 2023-10-30T14:00:24.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.567+0000 7fdf8a93e640 1 --2- 172.21.15.46:0/3756609030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf8415d6f0 0x7fdf8415a090 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:24.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.567+0000 7fdf8a93e640 1 -- 172.21.15.46:0/3756609030 >> 172.21.15.46:0/3756609030 conn(0x7fdf8406bfe0 msgr2=0x7fdf8406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:24.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.567+0000 7fdf8a93e640 1 -- 172.21.15.46:0/3756609030 shutdown_connections 2023-10-30T14:00:24.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.567+0000 7fdf8a93e640 1 -- 172.21.15.46:0/3756609030 wait complete. 2023-10-30T14:00:24.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.567+0000 7fdf8a93e640 1 Processor -- start 2023-10-30T14:00:24.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf8a93e640 1 -- start start 2023-10-30T14:00:24.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf8a93e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf840cf160 0x7fdf8414ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:24.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf8a93e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf841538a0 0x7fdf840ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:24.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf8a93e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf8415d6f0 0x7fdf840c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:24.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf8a93e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf841ee880 con 0x7fdf840cf160 2023-10-30T14:00:24.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf8a93e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8414d380 con 0x7fdf841538a0 2023-10-30T14:00:24.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf8a93e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8415cfa0 con 0x7fdf8415d6f0 2023-10-30T14:00:24.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf83fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf840cf160 0x7fdf8414ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:24.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf83fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf840cf160 0x7fdf8414ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35506/0 (socket says 172.21.15.46:35506) 2023-10-30T14:00:24.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf83fff640 1 -- 172.21.15.46:0/1478694596 learned_addr learned my addr 172.21.15.46:0/1478694596 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:24.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf88edb640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf8415d6f0 0x7fdf840c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:24.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf837fe640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf841538a0 0x7fdf840ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:24.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf83fff640 1 -- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf8415d6f0 msgr2=0x7fdf840c4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:24.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf83fff640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf8415d6f0 0x7fdf840c4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:24.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf83fff640 1 -- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf841538a0 msgr2=0x7fdf840ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:24.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf83fff640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf841538a0 0x7fdf840ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:24.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf83fff640 1 -- 172.21.15.46:0/1478694596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf780094d0 con 0x7fdf840cf160 2023-10-30T14:00:24.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf83fff640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf840cf160 0x7fdf8414ce40 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400c530 tx=0x7fdf7400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:24.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf80ff9640 1 -- 172.21.15.46:0/1478694596 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74005220 con 0x7fdf840cf160 2023-10-30T14:00:24.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf80ff9640 1 -- 172.21.15.46:0/1478694596 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf74005380 con 0x7fdf840cf160 2023-10-30T14:00:24.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf8a93e640 1 -- 172.21.15.46:0/1478694596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf841eea70 con 0x7fdf840cf160 2023-10-30T14:00:24.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf80ff9640 1 -- 172.21.15.46:0/1478694596 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74007270 con 0x7fdf840cf160 2023-10-30T14:00:24.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.571+0000 7fdf8a93e640 1 -- 172.21.15.46:0/1478694596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf841ef030 con 0x7fdf840cf160 2023-10-30T14:00:24.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.575+0000 7fdf8a93e640 1 -- 172.21.15.46:0/1478694596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8404e170 con 0x7fdf840cf160 2023-10-30T14:00:24.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.575+0000 7fdf80ff9640 1 -- 172.21.15.46:0/1478694596 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdf740073d0 con 0x7fdf840cf160 2023-10-30T14:00:24.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.575+0000 7fdf80ff9640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf700664a0 0x7fdf70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:24.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.575+0000 7fdf80ff9640 1 -- 172.21.15.46:0/1478694596 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7fdf74083d00 con 0x7fdf840cf160 2023-10-30T14:00:24.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.575+0000 7fdf837fe640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf700664a0 0x7fdf70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:24.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.575+0000 7fdf837fe640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf700664a0 0x7fdf70068960 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fdf780b2000 tx=0x7fdf78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:24.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.579+0000 7fdf80ff9640 1 -- 172.21.15.46:0/1478694596 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdf74050b80 con 0x7fdf840cf160 2023-10-30T14:00:24.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.771+0000 7fdf8a93e640 1 -- 172.21.15.46:0/1478694596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf841f1e70 con 0x7fdf840cf160 2023-10-30T14:00:24.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.771+0000 7fdf80ff9640 1 -- 172.21.15.46:0/1478694596 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fdf740541a0 con 0x7fdf840cf160 2023-10-30T14:00:24.774 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:24.775 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6809,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:23.718831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:24.775 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 158 2023-10-30T14:00:24.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.775+0000 7fdf8a93e640 1 -- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf700664a0 msgr2=0x7fdf70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:24.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.775+0000 7fdf8a93e640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf700664a0 0x7fdf70068960 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fdf780b2000 tx=0x7fdf78009040 comp rx=0 tx=0).stop 2023-10-30T14:00:24.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.775+0000 7fdf8a93e640 1 -- 172.21.15.46:0/1478694596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf840cf160 msgr2=0x7fdf8414ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:24.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.775+0000 7fdf8a93e640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf840cf160 0x7fdf8414ce40 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400c530 tx=0x7fdf7400c9f0 comp rx=0 tx=0).stop 2023-10-30T14:00:24.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.775+0000 7fdf8a93e640 1 -- 172.21.15.46:0/1478694596 shutdown_connections 2023-10-30T14:00:24.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.775+0000 7fdf8a93e640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdf700664a0 0x7fdf70068960 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:24.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.775+0000 7fdf8a93e640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf8415d6f0 0x7fdf840c4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:24.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.775+0000 7fdf8a93e640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf841538a0 0x7fdf840ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:24.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.775+0000 7fdf8a93e640 1 --2- 172.21.15.46:0/1478694596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf840cf160 0x7fdf8414ce40 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:24.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.775+0000 7fdf8a93e640 1 -- 172.21.15.46:0/1478694596 >> 172.21.15.46:0/1478694596 conn(0x7fdf8406bfe0 msgr2=0x7fdf84152a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:24.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.775+0000 7fdf8a93e640 1 -- 172.21.15.46:0/1478694596 shutdown_connections 2023-10-30T14:00:24.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:24.775+0000 7fdf8a93e640 1 -- 172.21.15.46:0/1478694596 wait complete. 2023-10-30T14:00:24.796 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:00:26.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:27.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.035+0000 7f23930c9640 1 Processor -- start 2023-10-30T14:00:27.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.035+0000 7f23930c9640 1 -- start start 2023-10-30T14:00:27.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.035+0000 7f23930c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f238c15d6f0 0x7f238c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:27.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.035+0000 7f23930c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f238c0cf160 0x7f238c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:27.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.035+0000 7f23930c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c0cb800 con 0x7f238c1538a0 2023-10-30T14:00:27.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.035+0000 7f23930c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f238c0cb330 con 0x7f238c0cf160 2023-10-30T14:00:27.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.035+0000 7f23930c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0cad70 con 0x7f238c15d6f0 2023-10-30T14:00:27.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.035+0000 7f238bfff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f238c15d6f0 0x7f238c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:27.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.035+0000 7f2390e65640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f238c1538a0 0x7f238c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:48430/0 (socket says 172.21.15.46:48430) 2023-10-30T14:00:27.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.035+0000 7f2390e65640 1 -- 172.21.15.46:0/2828052315 learned_addr learned my addr 172.21.15.46:0/2828052315 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:00:27.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.035+0000 7f2391666640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f238c0cf160 0x7f238c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:27.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.039+0000 7f238b7fe640 1 -- 172.21.15.46:0/2828052315 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845217979 0 0) 0x7f238c0cb800 con 0x7f238c1538a0 2023-10-30T14:00:27.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.039+0000 7f238bfff640 1 -- 172.21.15.46:0/2828052315 >> v2:172.21.15.46:3300/0 conn(0x7f238c0cf160 msgr2=0x7f238c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:27.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.039+0000 7f238bfff640 1 --2- 172.21.15.46:0/2828052315 >> v2:172.21.15.46:3300/0 conn(0x7f238c0cf160 0x7f238c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:27.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.039+0000 7f238bfff640 1 -- 172.21.15.46:0/2828052315 >> v1:172.21.15.154:6789/0 conn(0x7f238c1538a0 legacy=0x7f238c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:27.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.039+0000 7f238bfff640 1 -- 172.21.15.46:0/2828052315 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c0cb050 con 0x7f238c15d6f0 2023-10-30T14:00:27.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.039+0000 7f2391666640 1 --2- 172.21.15.46:0/2828052315 >> v2:172.21.15.46:3300/0 conn(0x7f238c0cf160 0x7f238c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:00:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.039+0000 7f238bfff640 1 --2- 172.21.15.46:0/2828052315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f238c15d6f0 0x7f238c15a090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f23780097c0 tx=0x7f237801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9245057b99c4ad2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.039+0000 7f238b7fe640 1 -- 172.21.15.46:0/2828052315 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23780a8970 con 0x7f238c15d6f0 2023-10-30T14:00:27.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.039+0000 7f238b7fe640 1 -- 172.21.15.46:0/2828052315 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23780a8ad0 con 0x7f238c15d6f0 2023-10-30T14:00:27.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.039+0000 7f238b7fe640 1 -- 172.21.15.46:0/2828052315 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23780a8e60 con 0x7f238c15d6f0 2023-10-30T14:00:27.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 -- 172.21.15.46:0/2828052315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f238c15d6f0 msgr2=0x7f238c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:27.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 --2- 172.21.15.46:0/2828052315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f238c15d6f0 0x7f238c15a090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f23780097c0 tx=0x7f237801ec70 comp rx=0 tx=0).stop 2023-10-30T14:00:27.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 -- 172.21.15.46:0/2828052315 shutdown_connections 2023-10-30T14:00:27.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 --2- 172.21.15.46:0/2828052315 >> v2:172.21.15.46:3300/0 conn(0x7f238c0cf160 0x7f238c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 --2- 172.21.15.46:0/2828052315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f238c15d6f0 0x7f238c15a090 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 -- 172.21.15.46:0/2828052315 >> 172.21.15.46:0/2828052315 conn(0x7f238c06bfe0 msgr2=0x7f238c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 -- 172.21.15.46:0/2828052315 shutdown_connections 2023-10-30T14:00:27.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 -- 172.21.15.46:0/2828052315 wait complete. 2023-10-30T14:00:27.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 Processor -- start 2023-10-30T14:00:27.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 -- start start 2023-10-30T14:00:27.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f238c0cf160 0x7f238c0b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f238c1538a0 0x7f238c14c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238c15d6f0 0x7f238c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c14c950 con 0x7f238c15d6f0 2023-10-30T14:00:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c14f130 con 0x7f238c1538a0 2023-10-30T14:00:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f23930c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c14dd80 con 0x7f238c0cf160 2023-10-30T14:00:27.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f2390e65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f238c0cf160 0x7f238c0b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:27.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f2391666640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238c15d6f0 0x7f238c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:27.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f2391666640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238c15d6f0 0x7f238c0ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59298/0 (socket says 172.21.15.46:59298) 2023-10-30T14:00:27.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f2391666640 1 -- 172.21.15.46:0/2943657509 learned_addr learned my addr 172.21.15.46:0/2943657509 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:27.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f238bfff640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f238c1538a0 0x7f238c14c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f2391666640 1 -- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f238c0cf160 msgr2=0x7f238c0b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f2391666640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f238c0cf160 0x7f238c0b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f2391666640 1 -- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f238c1538a0 msgr2=0x7f238c14c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f2391666640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f238c1538a0 0x7f238c14c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f2391666640 1 -- 172.21.15.46:0/2943657509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2384009a70 con 0x7f238c15d6f0 2023-10-30T14:00:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.043+0000 7f238bfff640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f238c1538a0 0x7f238c14c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:00:27.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.047+0000 7f2391666640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238c15d6f0 0x7f238c0ccfb0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f238400efc0 tx=0x7f238400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.047+0000 7f23897fa640 1 -- 172.21.15.46:0/2943657509 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384009cc0 con 0x7f238c15d6f0 2023-10-30T14:00:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.047+0000 7f23897fa640 1 -- 172.21.15.46:0/2943657509 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238400f040 con 0x7f238c15d6f0 2023-10-30T14:00:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.047+0000 7f23897fa640 1 -- 172.21.15.46:0/2943657509 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23840039a0 con 0x7f238c15d6f0 2023-10-30T14:00:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.047+0000 7f23930c9640 1 -- 172.21.15.46:0/2943657509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2378009420 con 0x7f238c15d6f0 2023-10-30T14:00:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.047+0000 7f23930c9640 1 -- 172.21.15.46:0/2943657509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c14e900 con 0x7f238c15d6f0 2023-10-30T14:00:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.047+0000 7f23930c9640 1 -- 172.21.15.46:0/2943657509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c0b2a50 con 0x7f238c15d6f0 2023-10-30T14:00:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.047+0000 7f23897fa640 1 -- 172.21.15.46:0/2943657509 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2384014480 con 0x7f238c15d6f0 2023-10-30T14:00:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.051+0000 7f23897fa640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23700664d0 0x7f2370068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.051+0000 7f23897fa640 1 -- 172.21.15.46:0/2943657509 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f2384084f70 con 0x7f238c15d6f0 2023-10-30T14:00:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.051+0000 7f2390e65640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23700664d0 0x7f2370068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.051+0000 7f2390e65640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23700664d0 0x7f2370068990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f2380007fd0 tx=0x7f2380007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:27.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.055+0000 7f23897fa640 1 -- 172.21.15.46:0/2943657509 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2384051df0 con 0x7f238c15d6f0 2023-10-30T14:00:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.247+0000 7f23930c9640 1 -- 172.21.15.46:0/2943657509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f238c14ea90 con 0x7f238c15d6f0 2023-10-30T14:00:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23897fa640 1 -- 172.21.15.46:0/2943657509 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f2384018090 con 0x7f238c15d6f0 2023-10-30T14:00:27.252 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:27.252 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":6809,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T13:58:23.718831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":7518},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7518":{"gid":7518,"name":"a","rank":1,"incarnation":145,"state":"up:active","state_seq":19,"addr":"172.21.15.46:6833/157453225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":157453225},{"type":"v1","addr":"172.21.15.46:6833","nonce":157453225}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:27.252 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 158 2023-10-30T14:00:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23930c9640 1 -- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23700664d0 msgr2=0x7f2370068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23930c9640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23700664d0 0x7f2370068990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f2380007fd0 tx=0x7f2380007d60 comp rx=0 tx=0).stop 2023-10-30T14:00:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23930c9640 1 -- 172.21.15.46:0/2943657509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238c15d6f0 msgr2=0x7f238c0ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:27.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23930c9640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238c15d6f0 0x7f238c0ccfb0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f238400efc0 tx=0x7f238400c490 comp rx=0 tx=0).stop 2023-10-30T14:00:27.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23930c9640 1 -- 172.21.15.46:0/2943657509 shutdown_connections 2023-10-30T14:00:27.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23930c9640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23700664d0 0x7f2370068990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:27.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23930c9640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f238c15d6f0 0x7f238c0ccfb0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:27.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23930c9640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f238c1538a0 0x7f238c14c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:27.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23930c9640 1 --2- 172.21.15.46:0/2943657509 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f238c0cf160 0x7f238c0b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:27.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23930c9640 1 -- 172.21.15.46:0/2943657509 >> 172.21.15.46:0/2943657509 conn(0x7f238c06bfe0 msgr2=0x7f238c0c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:27.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23930c9640 1 -- 172.21.15.46:0/2943657509 shutdown_connections 2023-10-30T14:00:27.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:27.251+0000 7f23930c9640 1 -- 172.21.15.46:0/2943657509 wait complete. 2023-10-30T14:00:27.273 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:00:29.275 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:29.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.435+0000 7f994d288640 1 Processor -- start 2023-10-30T14:00:29.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.435+0000 7f994d288640 1 -- start start 2023-10-30T14:00:29.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f994d288640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f99481538f0 0x7f9948159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f994d288640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f994815cd50 0x7f99480c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f994d288640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cefa0 0x7f99480cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f994d288640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994815d300 con 0x7f994815cd50 2023-10-30T14:00:29.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f994d288640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994815d5e0 con 0x7f99481538f0 2023-10-30T14:00:29.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f994d288640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994815d8c0 con 0x7f99480cefa0 2023-10-30T14:00:29.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f99477fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f994815cd50 0x7f99480c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f99477fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f994815cd50 0x7f99480c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59302/0 (socket says 172.21.15.46:59302) 2023-10-30T14:00:29.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f99477fe640 1 -- 172.21.15.46:0/2351320050 learned_addr learned my addr 172.21.15.46:0/2351320050 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:29.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f9947fff640 1 --2- 172.21.15.46:0/2351320050 >> v2:172.21.15.154:3300/0 conn(0x7f99481538f0 0x7f9948159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f994ca87640 1 --2- 172.21.15.46:0/2351320050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cefa0 0x7f99480cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f99477fe640 1 -- 172.21.15.46:0/2351320050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cefa0 msgr2=0x7f99480cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:29.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f99477fe640 1 --2- 172.21.15.46:0/2351320050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cefa0 0x7f99480cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f99477fe640 1 -- 172.21.15.46:0/2351320050 >> v2:172.21.15.154:3300/0 conn(0x7f99481538f0 msgr2=0x7f9948159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:29.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f99477fe640 1 --2- 172.21.15.46:0/2351320050 >> v2:172.21.15.154:3300/0 conn(0x7f99481538f0 0x7f9948159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f99477fe640 1 -- 172.21.15.46:0/2351320050 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994815dba0 con 0x7f994815cd50 2023-10-30T14:00:29.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f9947fff640 1 --2- 172.21.15.46:0/2351320050 >> v2:172.21.15.154:3300/0 conn(0x7f99481538f0 0x7f9948159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:00:29.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.439+0000 7f99477fe640 1 --2- 172.21.15.46:0/2351320050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f994815cd50 0x7f99480c9f90 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f99480c9cb0 tx=0x7f993c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ef25465f343622 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:29.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f9946ffd640 1 -- 172.21.15.46:0/2351320050 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c0a89b0 con 0x7f994815cd50 2023-10-30T14:00:29.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f9946ffd640 1 -- 172.21.15.46:0/2351320050 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c0a8b10 con 0x7f994815cd50 2023-10-30T14:00:29.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f9946ffd640 1 -- 172.21.15.46:0/2351320050 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c0a8ea0 con 0x7f994815cd50 2023-10-30T14:00:29.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 -- 172.21.15.46:0/2351320050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f994815cd50 msgr2=0x7f99480c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:29.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 --2- 172.21.15.46:0/2351320050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f994815cd50 0x7f99480c9f90 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f99480c9cb0 tx=0x7f993c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 -- 172.21.15.46:0/2351320050 shutdown_connections 2023-10-30T14:00:29.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 --2- 172.21.15.46:0/2351320050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99480cefa0 0x7f99480cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 --2- 172.21.15.46:0/2351320050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f994815cd50 0x7f99480c9f90 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 --2- 172.21.15.46:0/2351320050 >> v2:172.21.15.154:3300/0 conn(0x7f99481538f0 0x7f9948159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 -- 172.21.15.46:0/2351320050 >> 172.21.15.46:0/2351320050 conn(0x7f994806a640 msgr2=0x7f9948065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:29.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 -- 172.21.15.46:0/2351320050 shutdown_connections 2023-10-30T14:00:29.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 -- 172.21.15.46:0/2351320050 wait complete. 2023-10-30T14:00:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 Processor -- start 2023-10-30T14:00:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 -- start start 2023-10-30T14:00:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99480cefa0 0x7f99480b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99481538f0 0x7f99480b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994815cd50 0x7f9948147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480c6680 con 0x7f99480cefa0 2023-10-30T14:00:29.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994815a7f0 con 0x7f99481538f0 2023-10-30T14:00:29.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f994d288640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99480c5820 con 0x7f994815cd50 2023-10-30T14:00:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f9947fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99480cefa0 0x7f99480b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f9947fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99480cefa0 0x7f99480b31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59310/0 (socket says 172.21.15.46:59310) 2023-10-30T14:00:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.443+0000 7f9947fff640 1 -- 172.21.15.46:0/3548143564 learned_addr learned my addr 172.21.15.46:0/3548143564 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f99477fe640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99481538f0 0x7f99480b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f994ca87640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994815cd50 0x7f9948147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f9947fff640 1 -- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994815cd50 msgr2=0x7f9948147aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:29.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f9947fff640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994815cd50 0x7f9948147aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f9947fff640 1 -- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99481538f0 msgr2=0x7f99480b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:29.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f9947fff640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99481538f0 0x7f99480b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f9947fff640 1 -- 172.21.15.46:0/3548143564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c0094d0 con 0x7f99480cefa0 2023-10-30T14:00:29.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f9947fff640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99480cefa0 0x7f99480b31c0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f993800c7c0 tx=0x7f993800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:29.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f9944ff9640 1 -- 172.21.15.46:0/3548143564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938007a30 con 0x7f99480cefa0 2023-10-30T14:00:29.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f9944ff9640 1 -- 172.21.15.46:0/3548143564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9938007b90 con 0x7f99480cefa0 2023-10-30T14:00:29.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f9944ff9640 1 -- 172.21.15.46:0/3548143564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993800ce70 con 0x7f99480cefa0 2023-10-30T14:00:29.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f994d288640 1 -- 172.21.15.46:0/3548143564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994815aab0 con 0x7f99480cefa0 2023-10-30T14:00:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f994d288640 1 -- 172.21.15.46:0/3548143564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994815b070 con 0x7f99480cefa0 2023-10-30T14:00:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f994d288640 1 -- 172.21.15.46:0/3548143564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99480cb730 con 0x7f99480cefa0 2023-10-30T14:00:29.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.447+0000 7f9944ff9640 1 -- 172.21.15.46:0/3548143564 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f99380113f0 con 0x7f99480cefa0 2023-10-30T14:00:29.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.451+0000 7f9944ff9640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99340664f0 0x7f99340689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.451+0000 7f9944ff9640 1 -- 172.21.15.46:0/3548143564 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f99380849b0 con 0x7f99480cefa0 2023-10-30T14:00:29.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.451+0000 7f99477fe640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99340664f0 0x7f99340689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.451+0000 7f99477fe640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99340664f0 0x7f99340689b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f993c0b2000 tx=0x7f993c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:29.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.455+0000 7f9944ff9640 1 -- 172.21.15.46:0/3548143564 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f99480cb730 con 0x7f99480cefa0 2023-10-30T14:00:29.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.643+0000 7f994d288640 1 -- 172.21.15.46:0/3548143564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99480cb730 con 0x7f99480cefa0 2023-10-30T14:00:29.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.643+0000 7f9944ff9640 1 -- 172.21.15.46:0/3548143564 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f99480cb730 con 0x7f99480cefa0 2023-10-30T14:00:29.646 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:29.646 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:29.382715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:29.646 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 159 2023-10-30T14:00:29.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.647+0000 7f994d288640 1 -- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99340664f0 msgr2=0x7f99340689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:29.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.647+0000 7f994d288640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99340664f0 0x7f99340689b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f993c0b2000 tx=0x7f993c009040 comp rx=0 tx=0).stop 2023-10-30T14:00:29.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.647+0000 7f994d288640 1 -- 172.21.15.46:0/3548143564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99480cefa0 msgr2=0x7f99480b31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:29.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.647+0000 7f994d288640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99480cefa0 0x7f99480b31c0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f993800c7c0 tx=0x7f993800cc80 comp rx=0 tx=0).stop 2023-10-30T14:00:29.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.647+0000 7f994d288640 1 -- 172.21.15.46:0/3548143564 shutdown_connections 2023-10-30T14:00:29.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.647+0000 7f994d288640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f99340664f0 0x7f99340689b0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.647+0000 7f994d288640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994815cd50 0x7f9948147aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.647+0000 7f994d288640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99481538f0 0x7f99480b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.647+0000 7f994d288640 1 --2- 172.21.15.46:0/3548143564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99480cefa0 0x7f99480b31c0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.647+0000 7f994d288640 1 -- 172.21.15.46:0/3548143564 >> 172.21.15.46:0/3548143564 conn(0x7f994806a640 msgr2=0x7f99480cf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:29.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.647+0000 7f994d288640 1 -- 172.21.15.46:0/3548143564 shutdown_connections 2023-10-30T14:00:29.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.647+0000 7f994d288640 1 -- 172.21.15.46:0/3548143564 wait complete. 2023-10-30T14:00:29.667 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T14:00:29.667 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:00:29.667 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:29.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.807+0000 7fe75fbc5640 1 Processor -- start 2023-10-30T14:00:29.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.807+0000 7fe75fbc5640 1 -- start start 2023-10-30T14:00:29.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75fbc5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe7580c6b80 0x7fe7580bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75fbc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe75813f4a0 0x7fe7580c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75fbc5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7580c0a50 con 0x7fe7580c6f60 2023-10-30T14:00:29.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75fbc5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75806f6e0 con 0x7fe7580c6b80 2023-10-30T14:00:29.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75fbc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75806f9c0 con 0x7fe75813f4a0 2023-10-30T14:00:29.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75e162640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe75813f4a0 0x7fe7580c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75d961640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe7580c6b80 0x7fe7580bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75e162640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe75813f4a0 0x7fe7580c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:41514/0 (socket says 172.21.15.46:41514) 2023-10-30T14:00:29.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75e162640 1 -- 172.21.15.46:0/2822237111 learned_addr learned my addr 172.21.15.46:0/2822237111 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:29.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75d961640 1 -- 172.21.15.46:0/2822237111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe75813f4a0 msgr2=0x7fe7580c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:29.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75d961640 1 --2- 172.21.15.46:0/2822237111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe75813f4a0 0x7fe7580c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75d961640 1 -- 172.21.15.46:0/2822237111 >> v1:172.21.15.154:6789/0 conn(0x7fe7580c6f60 legacy=0x7fe75813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:29.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.811+0000 7fe75d961640 1 -- 172.21.15.46:0/2822237111 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75806fca0 con 0x7fe7580c6b80 2023-10-30T14:00:29.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75d961640 1 --2- 172.21.15.46:0/2822237111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7580c6b80 0x7fe7580bfec0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe7540097c0 tx=0x7fe75401e690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30cf43dc2ba3e6f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:29.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75c95f640 1 -- 172.21.15.46:0/2822237111 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7540ab670 con 0x7fe7580c6b80 2023-10-30T14:00:29.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75c95f640 1 -- 172.21.15.46:0/2822237111 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7540ab7d0 con 0x7fe7580c6b80 2023-10-30T14:00:29.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75c95f640 1 -- 172.21.15.46:0/2822237111 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7540abb60 con 0x7fe7580c6b80 2023-10-30T14:00:29.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75fbc5640 1 -- 172.21.15.46:0/2822237111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7580c6b80 msgr2=0x7fe7580bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:29.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75fbc5640 1 --2- 172.21.15.46:0/2822237111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7580c6b80 0x7fe7580bfec0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe7540097c0 tx=0x7fe75401e690 comp rx=0 tx=0).stop 2023-10-30T14:00:29.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75fbc5640 1 -- 172.21.15.46:0/2822237111 shutdown_connections 2023-10-30T14:00:29.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75fbc5640 1 --2- 172.21.15.46:0/2822237111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe75813f4a0 0x7fe7580c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75fbc5640 1 --2- 172.21.15.46:0/2822237111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7580c6b80 0x7fe7580bfec0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75fbc5640 1 -- 172.21.15.46:0/2822237111 >> 172.21.15.46:0/2822237111 conn(0x7fe7580c7910 msgr2=0x7fe7580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:29.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75fbc5640 1 -- 172.21.15.46:0/2822237111 shutdown_connections 2023-10-30T14:00:29.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75fbc5640 1 -- 172.21.15.46:0/2822237111 wait complete. 2023-10-30T14:00:29.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75fbc5640 1 Processor -- start 2023-10-30T14:00:29.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.815+0000 7fe75fbc5640 1 -- start start 2023-10-30T14:00:29.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75fbc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7580c6b80 0x7fe75813a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75fbc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7580c6f60 0x7fe7581586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75fbc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe75813f4a0 0x7fe758158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75fbc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe758158210 con 0x7fe7580c6f60 2023-10-30T14:00:29.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75fbc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe758160bb0 con 0x7fe7580c6b80 2023-10-30T14:00:29.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75fbc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580c8010 con 0x7fe75813f4a0 2023-10-30T14:00:29.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75d160640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7580c6f60 0x7fe7581586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75d160640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7580c6f60 0x7fe7581586a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59326/0 (socket says 172.21.15.46:59326) 2023-10-30T14:00:29.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75d160640 1 -- 172.21.15.46:0/3270815544 learned_addr learned my addr 172.21.15.46:0/3270815544 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:29.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75e162640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe75813f4a0 0x7fe758158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75d961640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7580c6b80 0x7fe75813a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75d160640 1 -- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe75813f4a0 msgr2=0x7fe758158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:29.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75d160640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe75813f4a0 0x7fe758158be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75d160640 1 -- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7580c6b80 msgr2=0x7fe75813a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:29.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75d160640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7580c6b80 0x7fe75813a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:29.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75d160640 1 -- 172.21.15.46:0/3270815544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe754009420 con 0x7fe7580c6f60 2023-10-30T14:00:29.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75d160640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7580c6f60 0x7fe7581586a0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c500 tx=0x7fe74800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:29.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe74e7fc640 1 -- 172.21.15.46:0/3270815544 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748005220 con 0x7fe7580c6f60 2023-10-30T14:00:29.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe74e7fc640 1 -- 172.21.15.46:0/3270815544 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe748005380 con 0x7fe7580c6f60 2023-10-30T14:00:29.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75fbc5640 1 -- 172.21.15.46:0/3270815544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7581608f0 con 0x7fe7580c6f60 2023-10-30T14:00:29.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe74e7fc640 1 -- 172.21.15.46:0/3270815544 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748007270 con 0x7fe7580c6f60 2023-10-30T14:00:29.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75fbc5640 1 -- 172.21.15.46:0/3270815544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe758157ba0 con 0x7fe7580c6f60 2023-10-30T14:00:29.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.819+0000 7fe75fbc5640 1 -- 172.21.15.46:0/3270815544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7581583c0 con 0x7fe7580c6f60 2023-10-30T14:00:29.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.823+0000 7fe74e7fc640 1 -- 172.21.15.46:0/3270815544 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe7480073d0 con 0x7fe7580c6f60 2023-10-30T14:00:29.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.823+0000 7fe74e7fc640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe73c0664d0 0x7fe73c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:29.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.823+0000 7fe74e7fc640 1 -- 172.21.15.46:0/3270815544 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fe748083d50 con 0x7fe7580c6f60 2023-10-30T14:00:29.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.823+0000 7fe75d961640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe73c0664d0 0x7fe73c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:29.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.823+0000 7fe75d961640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe73c0664d0 0x7fe73c068990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe7540b0000 tx=0x7fe7540a9000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:29.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:29.827+0000 7fe74e7fc640 1 -- 172.21.15.46:0/3270815544 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe7581583c0 con 0x7fe7580c6f60 2023-10-30T14:00:30.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 -- 172.21.15.46:0/3270815544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7581583c0 con 0x7fe7580c6f60 2023-10-30T14:00:30.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe74e7fc640 1 -- 172.21.15.46:0/3270815544 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fe7581583c0 con 0x7fe7580c6f60 2023-10-30T14:00:30.017 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:30.017 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:29.382715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:30.018 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 159 2023-10-30T14:00:30.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 -- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe73c0664d0 msgr2=0x7fe73c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:30.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe73c0664d0 0x7fe73c068990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe7540b0000 tx=0x7fe7540a9000 comp rx=0 tx=0).stop 2023-10-30T14:00:30.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 -- 172.21.15.46:0/3270815544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7580c6f60 msgr2=0x7fe7581586a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:30.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7580c6f60 0x7fe7581586a0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c500 tx=0x7fe74800c9c0 comp rx=0 tx=0).stop 2023-10-30T14:00:30.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 -- 172.21.15.46:0/3270815544 shutdown_connections 2023-10-30T14:00:30.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe73c0664d0 0x7fe73c068990 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:30.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe75813f4a0 0x7fe758158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:30.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7580c6f60 0x7fe7581586a0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:30.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 --2- 172.21.15.46:0/3270815544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7580c6b80 0x7fe75813a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:30.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 -- 172.21.15.46:0/3270815544 >> 172.21.15.46:0/3270815544 conn(0x7fe7580c7910 msgr2=0x7fe75813ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:30.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 -- 172.21.15.46:0/3270815544 shutdown_connections 2023-10-30T14:00:30.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:30.015+0000 7fe75fbc5640 1 -- 172.21.15.46:0/3270815544 wait complete. 2023-10-30T14:00:30.039 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 159, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 159, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6809": { "addr": "172.21.15.46:6837/1135482564", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 1135482564, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 1135482564, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6809, "incarnation": 159, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_7704": { "addr": "172.21.15.46:6836/3734066164", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3734066164, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3734066164, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7704, "incarnation": 152, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:00:29.382715+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7704, "mds_1": 6809 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:00:32.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:32.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.191+0000 7f160d93e640 1 Processor -- start 2023-10-30T14:00:32.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f160d93e640 1 -- start start 2023-10-30T14:00:32.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f160d93e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16080c93d0 0x7f160806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:32.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f160d93e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160806afa0 con 0x7f160806e470 2023-10-30T14:00:32.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f160d93e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1608154150 con 0x7f1608153d70 2023-10-30T14:00:32.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f160d93e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160806d780 con 0x7f16080c93d0 2023-10-30T14:00:32.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f16067fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f160806e470 0x7f16080c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42232/0 (socket says 172.21.15.46:42232) 2023-10-30T14:00:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f16067fc640 1 -- 172.21.15.46:0/898641259 learned_addr learned my addr 172.21.15.46:0/898641259 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:00:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f16077fe640 1 --2- 172.21.15.46:0/898641259 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16080c93d0 0x7f160806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:32.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907674791 0 0) 0x7f160806afa0 con 0x7f160806e470 2023-10-30T14:00:32.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f0003530 con 0x7f160806e470 2023-10-30T14:00:32.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378396821 0 0) 0x7f1608154150 con 0x7f1608153d70 2023-10-30T14:00:32.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f160806afa0 con 0x7f1608153d70 2023-10-30T14:00:32.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 324212270 0 0) 0x7f15f0003530 con 0x7f160806e470 2023-10-30T14:00:32.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16080c93d0 msgr2=0x7f160806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:32.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 --2- 172.21.15.46:0/898641259 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16080c93d0 0x7f160806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:32.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 >> v1:172.21.15.154:6789/0 conn(0x7f1608153d70 legacy=0x7f16080cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:32.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160806e030 con 0x7f160806e470 2023-10-30T14:00:32.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f15fc001680 con 0x7f160806e470 2023-10-30T14:00:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 >> v1:172.21.15.46:6789/0 conn(0x7f160806e470 legacy=0x7f16080c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 --2- 172.21.15.46:0/898641259 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16080c93d0 0x7f15f0003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 --2- 172.21.15.46:0/898641259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f00040f0 0x7f15f00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 --2- 172.21.15.46:0/898641259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f0004d10 0x7f15f0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160806d780 con 0x7f15f0004d10 2023-10-30T14:00:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f000bad0 con 0x7f15f00040f0 2023-10-30T14:00:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f000bc90 con 0x7f16080c93d0 2023-10-30T14:00:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f16077fe640 1 --2- 172.21.15.46:0/898641259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f0004d10 0x7f15f0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f16067fc640 1 --2- 172.21.15.46:0/898641259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f00040f0 0x7f15f00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f1606ffd640 1 --2- 172.21.15.46:0/898641259 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16080c93d0 0x7f15f0003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f16077fe640 1 -- 172.21.15.46:0/898641259 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16080c93d0 msgr2=0x7f15f0003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f16077fe640 1 --2- 172.21.15.46:0/898641259 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16080c93d0 0x7f15f0003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f16077fe640 1 -- 172.21.15.46:0/898641259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f00040f0 msgr2=0x7f15f00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:32.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f16077fe640 1 --2- 172.21.15.46:0/898641259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f00040f0 0x7f15f00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:32.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.195+0000 7f16077fe640 1 -- 172.21.15.46:0/898641259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f0003530 con 0x7f15f0004d10 2023-10-30T14:00:32.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.199+0000 7f16077fe640 1 --2- 172.21.15.46:0/898641259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f0004d10 0x7f15f0005190 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f1600009fc0 tx=0x7f160001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01f582ed331fa60 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:32.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.199+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16000a5b60 con 0x7f15f0004d10 2023-10-30T14:00:32.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.199+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160000f040 con 0x7f15f0004d10 2023-10-30T14:00:32.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.199+0000 7f1605ffb640 1 -- 172.21.15.46:0/898641259 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16000ad440 con 0x7f15f0004d10 2023-10-30T14:00:32.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.199+0000 7f160d93e640 1 -- 172.21.15.46:0/898641259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f0004d10 msgr2=0x7f15f0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:32.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.199+0000 7f160d93e640 1 --2- 172.21.15.46:0/898641259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f0004d10 0x7f15f0005190 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f1600009fc0 tx=0x7f160001cc60 comp rx=0 tx=0).stop 2023-10-30T14:00:32.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.199+0000 7f16077fe640 1 -- 172.21.15.46:0/898641259 reap_dead start 2023-10-30T14:00:32.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.199+0000 7f160d93e640 1 -- 172.21.15.46:0/898641259 shutdown_connections 2023-10-30T14:00:32.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.199+0000 7f160d93e640 1 -- 172.21.15.46:0/898641259 >> 172.21.15.46:0/898641259 conn(0x7f1608071ea0 msgr2=0x7f1608065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:32.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.199+0000 7f160d93e640 1 -- 172.21.15.46:0/898641259 shutdown_connections 2023-10-30T14:00:32.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.199+0000 7f160d93e640 1 -- 172.21.15.46:0/898641259 wait complete. 2023-10-30T14:00:32.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f160d93e640 1 Processor -- start 2023-10-30T14:00:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f160d93e640 1 -- start start 2023-10-30T14:00:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f160d93e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f0004d10 0x7f16080cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f160d93e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f160814aaa0 0x7f160814b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f160d93e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f160814be30 0x7f160814c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f160d93e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16081effc0 con 0x7f15f0004d10 2023-10-30T14:00:32.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f160d93e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16081f0260 con 0x7f160814be30 2023-10-30T14:00:32.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f160d93e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160814d200 con 0x7f160814aaa0 2023-10-30T14:00:32.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f1606ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f0004d10 0x7f16080cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:32.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f1606ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f0004d10 0x7f16080cad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59352/0 (socket says 172.21.15.46:59352) 2023-10-30T14:00:32.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f1606ffd640 1 -- 172.21.15.46:0/2671464088 learned_addr learned my addr 172.21.15.46:0/2671464088 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:32.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f16067fc640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f160814aaa0 0x7f160814b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:32.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f16077fe640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f160814be30 0x7f160814c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:32.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f1606ffd640 1 -- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f160814aaa0 msgr2=0x7f160814b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:32.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f1606ffd640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f160814aaa0 0x7f160814b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:32.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f1606ffd640 1 -- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f160814be30 msgr2=0x7f160814c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:32.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f1606ffd640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f160814be30 0x7f160814c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:32.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f1606ffd640 1 -- 172.21.15.46:0/2671464088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16081544d0 con 0x7f15f0004d10 2023-10-30T14:00:32.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f1606ffd640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f0004d10 0x7f16080cad20 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f15f800f770 tx=0x7f15f800fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:32.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f15effff640 1 -- 172.21.15.46:0/2671464088 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f80014b0 con 0x7f15f0004d10 2023-10-30T14:00:32.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f15effff640 1 -- 172.21.15.46:0/2671464088 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f800fe80 con 0x7f15f0004d10 2023-10-30T14:00:32.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f160d93e640 1 -- 172.21.15.46:0/2671464088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16081549f0 con 0x7f15f0004d10 2023-10-30T14:00:32.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f15effff640 1 -- 172.21.15.46:0/2671464088 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f800d730 con 0x7f15f0004d10 2023-10-30T14:00:32.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.203+0000 7f160d93e640 1 -- 172.21.15.46:0/2671464088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16080cb260 con 0x7f15f0004d10 2023-10-30T14:00:32.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.207+0000 7f160d93e640 1 -- 172.21.15.46:0/2671464088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16080cb480 con 0x7f15f0004d10 2023-10-30T14:00:32.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.207+0000 7f15effff640 1 -- 172.21.15.46:0/2671464088 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15f800d890 con 0x7f15f0004d10 2023-10-30T14:00:32.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.207+0000 7f15effff640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f15f40664d0 0x7f15f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:32.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.207+0000 7f15effff640 1 -- 172.21.15.46:0/2671464088 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f15f8084700 con 0x7f15f0004d10 2023-10-30T14:00:32.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.207+0000 7f16067fc640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f15f40664d0 0x7f15f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:32.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.207+0000 7f16067fc640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f15f40664d0 0x7f15f4068990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f15fc001b90 tx=0x7f15fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:32.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.211+0000 7f15effff640 1 -- 172.21.15.46:0/2671464088 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f16080cb480 con 0x7f15f0004d10 2023-10-30T14:00:32.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 -- 172.21.15.46:0/2671464088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1608154330 con 0x7f15f0004d10 2023-10-30T14:00:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f15effff640 1 -- 172.21.15.46:0/2671464088 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f1608154330 con 0x7f15f0004d10 2023-10-30T14:00:32.404 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:32.404 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:29.382715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 159 2023-10-30T14:00:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 -- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f15f40664d0 msgr2=0x7f15f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f15f40664d0 0x7f15f4068990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f15fc001b90 tx=0x7f15fc009040 comp rx=0 tx=0).stop 2023-10-30T14:00:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 -- 172.21.15.46:0/2671464088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f0004d10 msgr2=0x7f16080cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:32.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f0004d10 0x7f16080cad20 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f15f800f770 tx=0x7f15f800fc30 comp rx=0 tx=0).stop 2023-10-30T14:00:32.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 -- 172.21.15.46:0/2671464088 shutdown_connections 2023-10-30T14:00:32.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f15f40664d0 0x7f15f4068990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:32.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f160814be30 0x7f160814c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:32.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f160814aaa0 0x7f160814b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:32.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 --2- 172.21.15.46:0/2671464088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15f0004d10 0x7f16080cad20 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:32.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 -- 172.21.15.46:0/2671464088 >> 172.21.15.46:0/2671464088 conn(0x7f1608071ea0 msgr2=0x7f16080c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:32.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 -- 172.21.15.46:0/2671464088 shutdown_connections 2023-10-30T14:00:32.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:32.403+0000 7f160d93e640 1 -- 172.21.15.46:0/2671464088 wait complete. 2023-10-30T14:00:32.425 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:00:34.427 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:34.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.579+0000 7f33d9a68640 1 Processor -- start 2023-10-30T14:00:34.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.579+0000 7f33d9a68640 1 -- start start 2023-10-30T14:00:34.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.583+0000 7f33d9a68640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33d40cf160 0x7f33d40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:34.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.583+0000 7f33d9a68640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d40cb800 con 0x7f33d41538a0 2023-10-30T14:00:34.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.583+0000 7f33d9a68640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d40cb330 con 0x7f33d415d6f0 2023-10-30T14:00:34.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.583+0000 7f33d9a68640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d40cad70 con 0x7f33d40cf160 2023-10-30T14:00:34.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.583+0000 7f33d37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f33d41538a0 0x7f33d40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42238/0 (socket says 172.21.15.46:42238) 2023-10-30T14:00:34.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.583+0000 7f33d37fe640 1 -- 172.21.15.46:0/2090719356 learned_addr learned my addr 172.21.15.46:0/2090719356 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:00:34.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.583+0000 7f33d3fff640 1 --2- 172.21.15.46:0/2090719356 >> v2:172.21.15.46:3300/0 conn(0x7f33d40cf160 0x7f33d40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:34.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.583+0000 7f33d27fc640 1 -- 172.21.15.46:0/2090719356 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113043920 0 0) 0x7f33d40cb800 con 0x7f33d41538a0 2023-10-30T14:00:34.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.583+0000 7f33d27fc640 1 -- 172.21.15.46:0/2090719356 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33bc003530 con 0x7f33d41538a0 2023-10-30T14:00:34.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.583+0000 7f33d3fff640 1 -- 172.21.15.46:0/2090719356 >> v1:172.21.15.154:6789/0 conn(0x7f33d415d6f0 legacy=0x7f33d415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:34.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.583+0000 7f33d3fff640 1 -- 172.21.15.46:0/2090719356 >> v1:172.21.15.46:6789/0 conn(0x7f33d41538a0 legacy=0x7f33d40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:34.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.583+0000 7f33d3fff640 1 -- 172.21.15.46:0/2090719356 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d40cb050 con 0x7f33d40cf160 2023-10-30T14:00:34.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.587+0000 7f33d3fff640 1 --2- 172.21.15.46:0/2090719356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33d40cf160 0x7f33d40cf590 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009d90 tx=0x7f33cc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f765211a45c9a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:34.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.587+0000 7f33d27fc640 1 -- 172.21.15.46:0/2090719356 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc0af600 con 0x7f33d40cf160 2023-10-30T14:00:34.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.587+0000 7f33d27fc640 1 -- 172.21.15.46:0/2090719356 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc0ab050 con 0x7f33d40cf160 2023-10-30T14:00:34.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.587+0000 7f33d27fc640 1 -- 172.21.15.46:0/2090719356 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc0af980 con 0x7f33d40cf160 2023-10-30T14:00:34.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.587+0000 7f33d9a68640 1 -- 172.21.15.46:0/2090719356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33d40cf160 msgr2=0x7f33d40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:34.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.587+0000 7f33d9a68640 1 --2- 172.21.15.46:0/2090719356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33d40cf160 0x7f33d40cf590 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009d90 tx=0x7f33cc01f130 comp rx=0 tx=0).stop 2023-10-30T14:00:34.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.587+0000 7f33d9a68640 1 -- 172.21.15.46:0/2090719356 shutdown_connections 2023-10-30T14:00:34.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.587+0000 7f33d9a68640 1 --2- 172.21.15.46:0/2090719356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33d40cf160 0x7f33d40cf590 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:34.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.587+0000 7f33d9a68640 1 -- 172.21.15.46:0/2090719356 >> 172.21.15.46:0/2090719356 conn(0x7f33d406bfe0 msgr2=0x7f33d406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:34.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.587+0000 7f33d9a68640 1 -- 172.21.15.46:0/2090719356 shutdown_connections 2023-10-30T14:00:34.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.587+0000 7f33d9a68640 1 -- 172.21.15.46:0/2090719356 wait complete. 2023-10-30T14:00:34.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.587+0000 7f33d9a68640 1 Processor -- start 2023-10-30T14:00:34.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d9a68640 1 -- start start 2023-10-30T14:00:34.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d9a68640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33d40cf160 0x7f33d40ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:34.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d9a68640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33d41538a0 0x7f33d40b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:34.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d9a68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33d415d6f0 0x7f33d4148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:34.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d9a68640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d4149a30 con 0x7f33d41538a0 2023-10-30T14:00:34.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d9a68640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d40ce3a0 con 0x7f33d415d6f0 2023-10-30T14:00:34.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d9a68640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d4149cd0 con 0x7f33d40cf160 2023-10-30T14:00:34.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33d41538a0 0x7f33d40b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33d41538a0 0x7f33d40b3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59370/0 (socket says 172.21.15.46:59370) 2023-10-30T14:00:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d2ffd640 1 -- 172.21.15.46:0/2739528880 learned_addr learned my addr 172.21.15.46:0/2739528880 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d3fff640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33d415d6f0 0x7f33d4148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d37fe640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33d40cf160 0x7f33d40ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d2ffd640 1 -- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33d40cf160 msgr2=0x7f33d40ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d2ffd640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33d40cf160 0x7f33d40ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:34.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d2ffd640 1 -- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33d415d6f0 msgr2=0x7f33d4148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:34.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d2ffd640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33d415d6f0 0x7f33d4148690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:34.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d2ffd640 1 -- 172.21.15.46:0/2739528880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc009a70 con 0x7f33d41538a0 2023-10-30T14:00:34.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d37fe640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33d40cf160 0x7f33d40ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:00:34.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d2ffd640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33d41538a0 0x7f33d40b3400 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f33c800c3c0 tx=0x7f33c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:34.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33bbfff640 1 -- 172.21.15.46:0/2739528880 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c80052d0 con 0x7f33d41538a0 2023-10-30T14:00:34.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33bbfff640 1 -- 172.21.15.46:0/2739528880 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c8005430 con 0x7f33d41538a0 2023-10-30T14:00:34.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33bbfff640 1 -- 172.21.15.46:0/2739528880 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c8003840 con 0x7f33d41538a0 2023-10-30T14:00:34.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d9a68640 1 -- 172.21.15.46:0/2739528880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d4149fb0 con 0x7f33d41538a0 2023-10-30T14:00:34.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d9a68640 1 -- 172.21.15.46:0/2739528880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d40ce680 con 0x7f33d41538a0 2023-10-30T14:00:34.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.591+0000 7f33d9a68640 1 -- 172.21.15.46:0/2739528880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d404e170 con 0x7f33d41538a0 2023-10-30T14:00:34.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.595+0000 7f33bbfff640 1 -- 172.21.15.46:0/2739528880 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33c80039a0 con 0x7f33d41538a0 2023-10-30T14:00:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.595+0000 7f33bbfff640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33c00664a0 0x7f33c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.595+0000 7f33bbfff640 1 -- 172.21.15.46:0/2739528880 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f33c8083b40 con 0x7f33d41538a0 2023-10-30T14:00:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.595+0000 7f33d37fe640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33c00664a0 0x7f33c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.595+0000 7f33d37fe640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33c00664a0 0x7f33c0068960 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f33c40011f0 tx=0x7f33c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:34.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.603+0000 7f33bbfff640 1 -- 172.21.15.46:0/2739528880 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33c8050850 con 0x7f33d41538a0 2023-10-30T14:00:34.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.815+0000 7f33d9a68640 1 -- 172.21.15.46:0/2739528880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33d40b1dd0 con 0x7f33d41538a0 2023-10-30T14:00:34.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.815+0000 7f33bbfff640 1 -- 172.21.15.46:0/2739528880 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f33c8053e70 con 0x7f33d41538a0 2023-10-30T14:00:34.819 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:34.819 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:29.382715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:34.819 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 159 2023-10-30T14:00:34.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.819+0000 7f33d9a68640 1 -- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33c00664a0 msgr2=0x7f33c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:34.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.819+0000 7f33d9a68640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33c00664a0 0x7f33c0068960 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f33c40011f0 tx=0x7f33c4009040 comp rx=0 tx=0).stop 2023-10-30T14:00:34.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.819+0000 7f33d9a68640 1 -- 172.21.15.46:0/2739528880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33d41538a0 msgr2=0x7f33d40b3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:34.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.819+0000 7f33d9a68640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33d41538a0 0x7f33d40b3400 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f33c800c3c0 tx=0x7f33c800c880 comp rx=0 tx=0).stop 2023-10-30T14:00:34.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.819+0000 7f33d9a68640 1 -- 172.21.15.46:0/2739528880 shutdown_connections 2023-10-30T14:00:34.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.819+0000 7f33d9a68640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33c00664a0 0x7f33c0068960 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:34.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.819+0000 7f33d9a68640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33d415d6f0 0x7f33d4148690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:34.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.819+0000 7f33d9a68640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33d41538a0 0x7f33d40b3400 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:34.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.819+0000 7f33d9a68640 1 --2- 172.21.15.46:0/2739528880 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33d40cf160 0x7f33d40ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:34.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.819+0000 7f33d9a68640 1 -- 172.21.15.46:0/2739528880 >> 172.21.15.46:0/2739528880 conn(0x7f33d406bfe0 msgr2=0x7f33d40c5610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:34.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.819+0000 7f33d9a68640 1 -- 172.21.15.46:0/2739528880 shutdown_connections 2023-10-30T14:00:34.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:34.819+0000 7f33d9a68640 1 -- 172.21.15.46:0/2739528880 wait complete. 2023-10-30T14:00:34.841 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:00:36.842 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:37.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.003+0000 7fb55533e640 1 Processor -- start 2023-10-30T14:00:37.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb55533e640 1 -- start start 2023-10-30T14:00:37.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb55533e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb55006f7a0 0x7fb55006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:37.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb55533e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb550066cf0 con 0x7fb550071110 2023-10-30T14:00:37.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb55533e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5500c70d0 con 0x7fb5500c6bd0 2023-10-30T14:00:37.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb55533e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5500700c0 con 0x7fb55006f7a0 2023-10-30T14:00:37.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb550071110 0x7fb550158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36808/0 (socket says 172.21.15.46:36808) 2023-10-30T14:00:37.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54f7fe640 1 -- 172.21.15.46:0/2850043374 learned_addr learned my addr 172.21.15.46:0/2850043374 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:00:37.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54effd640 1 --2- 172.21.15.46:0/2850043374 >> v2:172.21.15.154:3300/0 conn(0x7fb55006f7a0 0x7fb55006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:37.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549786896 0 0) 0x7fb550066cf0 con 0x7fb550071110 2023-10-30T14:00:37.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb538003530 con 0x7fb550071110 2023-10-30T14:00:37.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2834242847 0 0) 0x7fb538003530 con 0x7fb550071110 2023-10-30T14:00:37.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 >> v2:172.21.15.154:3300/0 conn(0x7fb55006f7a0 msgr2=0x7fb55006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:37.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 --2- 172.21.15.46:0/2850043374 >> v2:172.21.15.154:3300/0 conn(0x7fb55006f7a0 0x7fb55006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:37.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 >> v1:172.21.15.154:6789/0 conn(0x7fb5500c6bd0 legacy=0x7fb550069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:37.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5500c7390 con 0x7fb550071110 2023-10-30T14:00:37.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fb548001a00 con 0x7fb550071110 2023-10-30T14:00:37.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 >> v1:172.21.15.46:6789/0 conn(0x7fb550071110 legacy=0x7fb550158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:37.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 --2- 172.21.15.46:0/2850043374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb538003980 0x7fb538003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:37.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 --2- 172.21.15.46:0/2850043374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5380047d0 0x7fb538004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:37.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 --2- 172.21.15.46:0/2850043374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 0x7fb53800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:37.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5500700c0 con 0x7fb538005190 2023-10-30T14:00:37.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53800bfe0 con 0x7fb5380047d0 2023-10-30T14:00:37.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53800c1a0 con 0x7fb538003980 2023-10-30T14:00:37.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54f7fe640 1 --2- 172.21.15.46:0/2850043374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 0x7fb53800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:37.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54effd640 1 --2- 172.21.15.46:0/2850043374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb538003980 0x7fb538003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:37.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54e7fc640 1 --2- 172.21.15.46:0/2850043374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5380047d0 0x7fb538004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:37.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54f7fe640 1 -- 172.21.15.46:0/2850043374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb538003980 msgr2=0x7fb538003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:37.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54f7fe640 1 --2- 172.21.15.46:0/2850043374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb538003980 0x7fb538003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:37.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54f7fe640 1 -- 172.21.15.46:0/2850043374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5380047d0 msgr2=0x7fb538004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:37.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54f7fe640 1 --2- 172.21.15.46:0/2850043374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5380047d0 0x7fb538004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:37.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.007+0000 7fb54f7fe640 1 -- 172.21.15.46:0/2850043374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5500c7390 con 0x7fb538005190 2023-10-30T14:00:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.011+0000 7fb54f7fe640 1 -- 172.21.15.46:0/2850043374 reap_dead start 2023-10-30T14:00:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.011+0000 7fb54f7fe640 1 --2- 172.21.15.46:0/2850043374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 0x7fb53800b990 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb54800a900 tx=0x7fb54801d400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234f09a226c75658 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.011+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5480072d0 con 0x7fb538005190 2023-10-30T14:00:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.011+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5480a8070 con 0x7fb538005190 2023-10-30T14:00:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.011+0000 7fb54dffb640 1 -- 172.21.15.46:0/2850043374 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548007650 con 0x7fb538005190 2023-10-30T14:00:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.011+0000 7fb55533e640 1 -- 172.21.15.46:0/2850043374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 msgr2=0x7fb53800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.011+0000 7fb55533e640 1 --2- 172.21.15.46:0/2850043374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 0x7fb53800b990 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb54800a900 tx=0x7fb54801d400 comp rx=0 tx=0).stop 2023-10-30T14:00:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.011+0000 7fb55533e640 1 -- 172.21.15.46:0/2850043374 shutdown_connections 2023-10-30T14:00:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.011+0000 7fb55533e640 1 --2- 172.21.15.46:0/2850043374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 0x7fb53800b990 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.011+0000 7fb55533e640 1 -- 172.21.15.46:0/2850043374 >> 172.21.15.46:0/2850043374 conn(0x7fb5500c7910 msgr2=0x7fb5500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.011+0000 7fb55533e640 1 -- 172.21.15.46:0/2850043374 shutdown_connections 2023-10-30T14:00:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.011+0000 7fb55533e640 1 -- 172.21.15.46:0/2850043374 wait complete. 2023-10-30T14:00:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb55533e640 1 Processor -- start 2023-10-30T14:00:37.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb55533e640 1 -- start start 2023-10-30T14:00:37.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb55533e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 0x7fb5500c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:37.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb55533e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5501f2040 0x7fb55015a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:37.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb55533e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb550158c30 0x7fb5501590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:37.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb55533e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5501587a0 con 0x7fb538005190 2023-10-30T14:00:37.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb55533e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb550159620 con 0x7fb5501f2040 2023-10-30T14:00:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb55533e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb550158470 con 0x7fb550158c30 2023-10-30T14:00:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb54effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 0x7fb5500c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb54e7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5501f2040 0x7fb55015a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb54effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 0x7fb5500c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48312/0 (socket says 172.21.15.46:48312) 2023-10-30T14:00:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb54effd640 1 -- 172.21.15.46:0/22329932 learned_addr learned my addr 172.21.15.46:0/22329932 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb54f7fe640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb550158c30 0x7fb5501590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb54effd640 1 -- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb550158c30 msgr2=0x7fb5501590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb54effd640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb550158c30 0x7fb5501590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb54effd640 1 -- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5501f2040 msgr2=0x7fb55015a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb54effd640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5501f2040 0x7fb55015a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb54effd640 1 -- 172.21.15.46:0/22329932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5500c6400 con 0x7fb538005190 2023-10-30T14:00:37.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.015+0000 7fb54effd640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 0x7fb5500c5760 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fb5400115c0 tx=0x7fb540011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:37.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.019+0000 7fb5377fe640 1 -- 172.21.15.46:0/22329932 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540015870 con 0x7fb538005190 2023-10-30T14:00:37.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.019+0000 7fb5377fe640 1 -- 172.21.15.46:0/22329932 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5400159d0 con 0x7fb538005190 2023-10-30T14:00:37.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.019+0000 7fb5377fe640 1 -- 172.21.15.46:0/22329932 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540015d40 con 0x7fb538005190 2023-10-30T14:00:37.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.019+0000 7fb55533e640 1 -- 172.21.15.46:0/22329932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55015acc0 con 0x7fb538005190 2023-10-30T14:00:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.019+0000 7fb55533e640 1 -- 172.21.15.46:0/22329932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55015b280 con 0x7fb538005190 2023-10-30T14:00:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.019+0000 7fb55533e640 1 -- 172.21.15.46:0/22329932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55004e170 con 0x7fb538005190 2023-10-30T14:00:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.019+0000 7fb5377fe640 1 -- 172.21.15.46:0/22329932 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb54000f460 con 0x7fb538005190 2023-10-30T14:00:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.019+0000 7fb5377fe640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb53c0664d0 0x7fb53c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.019+0000 7fb5377fe640 1 -- 172.21.15.46:0/22329932 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fb540084910 con 0x7fb538005190 2023-10-30T14:00:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.023+0000 7fb54e7fc640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb53c0664d0 0x7fb53c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.023+0000 7fb54e7fc640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb53c0664d0 0x7fb53c068990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb544007fd0 tx=0x7fb544009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:37.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.023+0000 7fb5377fe640 1 -- 172.21.15.46:0/22329932 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb540051640 con 0x7fb538005190 2023-10-30T14:00:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.231+0000 7fb55533e640 1 -- 172.21.15.46:0/22329932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5500c7fd0 con 0x7fb538005190 2023-10-30T14:00:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.231+0000 7fb5377fe640 1 -- 172.21.15.46:0/22329932 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fb540054c60 con 0x7fb538005190 2023-10-30T14:00:37.234 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:37.234 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:29.382715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 159 2023-10-30T14:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.235+0000 7fb55533e640 1 -- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb53c0664d0 msgr2=0x7fb53c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.235+0000 7fb55533e640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb53c0664d0 0x7fb53c068990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb544007fd0 tx=0x7fb544009040 comp rx=0 tx=0).stop 2023-10-30T14:00:37.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.235+0000 7fb55533e640 1 -- 172.21.15.46:0/22329932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 msgr2=0x7fb5500c5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:37.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.235+0000 7fb55533e640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 0x7fb5500c5760 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fb5400115c0 tx=0x7fb540011a80 comp rx=0 tx=0).stop 2023-10-30T14:00:37.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.235+0000 7fb55533e640 1 -- 172.21.15.46:0/22329932 shutdown_connections 2023-10-30T14:00:37.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.235+0000 7fb55533e640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb53c0664d0 0x7fb53c068990 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:37.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.235+0000 7fb55533e640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb550158c30 0x7fb5501590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:37.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.235+0000 7fb55533e640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5501f2040 0x7fb55015a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:37.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.235+0000 7fb55533e640 1 --2- 172.21.15.46:0/22329932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb538005190 0x7fb5500c5760 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:37.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.235+0000 7fb55533e640 1 -- 172.21.15.46:0/22329932 >> 172.21.15.46:0/22329932 conn(0x7fb5500c7910 msgr2=0x7fb55006a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:37.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.235+0000 7fb55533e640 1 -- 172.21.15.46:0/22329932 shutdown_connections 2023-10-30T14:00:37.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:37.235+0000 7fb55533e640 1 -- 172.21.15.46:0/22329932 wait complete. 2023-10-30T14:00:37.256 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:00:39.257 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:39.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.399+0000 7f689ef65640 1 Processor -- start 2023-10-30T14:00:39.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.399+0000 7f689ef65640 1 -- start start 2023-10-30T14:00:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.399+0000 7f689ef65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68981538a0 0x7f689815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:39.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.399+0000 7f689ef65640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68980cb800 con 0x7f68980ca0e0 2023-10-30T14:00:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.399+0000 7f689ef65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689815dc30 con 0x7f68980cf160 2023-10-30T14:00:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.403+0000 7f689ef65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68980cb330 con 0x7f68981538a0 2023-10-30T14:00:39.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.403+0000 7f6897fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f68980ca0e0 0x7f689815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36810/0 (socket says 172.21.15.46:36810) 2023-10-30T14:00:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.403+0000 7f6897fff640 1 -- 172.21.15.46:0/2682813333 learned_addr learned my addr 172.21.15.46:0/2682813333 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:00:39.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.403+0000 7f689cd01640 1 --2- 172.21.15.46:0/2682813333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68981538a0 0x7f689815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.403+0000 7f68977fe640 1 -- 172.21.15.46:0/2682813333 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678325011 0 0) 0x7f68980cb800 con 0x7f68980ca0e0 2023-10-30T14:00:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.403+0000 7f68977fe640 1 -- 172.21.15.46:0/2682813333 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6888003530 con 0x7f68980ca0e0 2023-10-30T14:00:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.403+0000 7f68977fe640 1 -- 172.21.15.46:0/2682813333 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 759415822 0 0) 0x7f6888003530 con 0x7f68980ca0e0 2023-10-30T14:00:39.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.403+0000 7f689cd01640 1 -- 172.21.15.46:0/2682813333 >> v1:172.21.15.154:6789/0 conn(0x7f68980cf160 legacy=0x7f6898164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.403+0000 7f689cd01640 1 -- 172.21.15.46:0/2682813333 >> v1:172.21.15.46:6789/0 conn(0x7f68980ca0e0 legacy=0x7f689815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.403+0000 7f689cd01640 1 -- 172.21.15.46:0/2682813333 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68980cad70 con 0x7f68981538a0 2023-10-30T14:00:39.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.407+0000 7f689cd01640 1 --2- 172.21.15.46:0/2682813333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68981538a0 0x7f689815d6f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f688c0097f0 tx=0x7f688c01ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64a1686a2f181c1b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:39.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f68977fe640 1 -- 172.21.15.46:0/2682813333 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6884001680 con 0x7f68981538a0 2023-10-30T14:00:39.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f68977fe640 1 -- 172.21.15.46:0/2682813333 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688c0a8650 con 0x7f68981538a0 2023-10-30T14:00:39.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f68977fe640 1 -- 172.21.15.46:0/2682813333 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c0a89e0 con 0x7f68981538a0 2023-10-30T14:00:39.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f689ef65640 1 -- 172.21.15.46:0/2682813333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68981538a0 msgr2=0x7f689815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:39.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f689ef65640 1 --2- 172.21.15.46:0/2682813333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68981538a0 0x7f689815d6f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f688c0097f0 tx=0x7f688c01ebd0 comp rx=0 tx=0).stop 2023-10-30T14:00:39.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f689ef65640 1 -- 172.21.15.46:0/2682813333 shutdown_connections 2023-10-30T14:00:39.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f689ef65640 1 --2- 172.21.15.46:0/2682813333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68981538a0 0x7f689815d6f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:39.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f689ef65640 1 -- 172.21.15.46:0/2682813333 >> 172.21.15.46:0/2682813333 conn(0x7f689806bfe0 msgr2=0x7f689806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:39.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f689ef65640 1 -- 172.21.15.46:0/2682813333 shutdown_connections 2023-10-30T14:00:39.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f689ef65640 1 -- 172.21.15.46:0/2682813333 wait complete. 2023-10-30T14:00:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f689ef65640 1 Processor -- start 2023-10-30T14:00:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f689ef65640 1 -- start start 2023-10-30T14:00:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.411+0000 7f689ef65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68980ca0e0 0x7f68980cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689ef65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68980cf160 0x7f68981e2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689ef65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68981538a0 0x7f68981efe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689ef65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68980ccfb0 con 0x7f68980ca0e0 2023-10-30T14:00:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689ef65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68981ef6e0 con 0x7f68980cf160 2023-10-30T14:00:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689ef65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6898147bc0 con 0x7f68981538a0 2023-10-30T14:00:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689cd01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68980ca0e0 0x7f68980cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689cd01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68980ca0e0 0x7f68980cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48324/0 (socket says 172.21.15.46:48324) 2023-10-30T14:00:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689cd01640 1 -- 172.21.15.46:0/679984151 learned_addr learned my addr 172.21.15.46:0/679984151 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f6897fff640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68980cf160 0x7f68981e2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689d502640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68981538a0 0x7f68981efe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689cd01640 1 -- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68981538a0 msgr2=0x7f68981efe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689cd01640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68981538a0 0x7f68981efe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689cd01640 1 -- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68980cf160 msgr2=0x7f68981e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689cd01640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68980cf160 0x7f68981e2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689cd01640 1 -- 172.21.15.46:0/679984151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c009420 con 0x7f68980ca0e0 2023-10-30T14:00:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689cd01640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68980ca0e0 0x7f68980cea20 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f688c0097f0 tx=0x7f688c01b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f68957fa640 1 -- 172.21.15.46:0/679984151 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c0b0850 con 0x7f68980ca0e0 2023-10-30T14:00:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f68957fa640 1 -- 172.21.15.46:0/679984151 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688c0b09b0 con 0x7f68980ca0e0 2023-10-30T14:00:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689ef65640 1 -- 172.21.15.46:0/679984151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68981592c0 con 0x7f68980ca0e0 2023-10-30T14:00:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f68957fa640 1 -- 172.21.15.46:0/679984151 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c0b0d20 con 0x7f68980ca0e0 2023-10-30T14:00:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689ef65640 1 -- 172.21.15.46:0/679984151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68980ce740 con 0x7f68980ca0e0 2023-10-30T14:00:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f689ef65640 1 -- 172.21.15.46:0/679984151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f689804e170 con 0x7f68980ca0e0 2023-10-30T14:00:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.415+0000 7f68957fa640 1 -- 172.21.15.46:0/679984151 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f688c0b7040 con 0x7f68980ca0e0 2023-10-30T14:00:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.419+0000 7f68957fa640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f687c0664d0 0x7f687c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.419+0000 7f68957fa640 1 -- 172.21.15.46:0/679984151 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f688c121220 con 0x7f68980ca0e0 2023-10-30T14:00:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.419+0000 7f6897fff640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f687c0664d0 0x7f687c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.419+0000 7f6897fff640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f687c0664d0 0x7f687c068990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6898148270 tx=0x7f6884009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:39.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.423+0000 7f68957fa640 1 -- 172.21.15.46:0/679984151 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f688c0edea0 con 0x7f68980ca0e0 2023-10-30T14:00:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.611+0000 7f689ef65640 1 -- 172.21.15.46:0/679984151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f689814d930 con 0x7f68980ca0e0 2023-10-30T14:00:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.611+0000 7f68957fa640 1 -- 172.21.15.46:0/679984151 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+5734 (secure 0 0 0) 0x7f688c0b5090 con 0x7f68980ca0e0 2023-10-30T14:00:39.615 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:39.615 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:39.205930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:resolve","state_seq":37,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 160 2023-10-30T14:00:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.615+0000 7f689ef65640 1 -- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f687c0664d0 msgr2=0x7f687c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.615+0000 7f689ef65640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f687c0664d0 0x7f687c068990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6898148270 tx=0x7f6884009040 comp rx=0 tx=0).stop 2023-10-30T14:00:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.615+0000 7f689ef65640 1 -- 172.21.15.46:0/679984151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68980ca0e0 msgr2=0x7f68980cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.615+0000 7f689ef65640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68980ca0e0 0x7f68980cea20 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f688c0097f0 tx=0x7f688c01b970 comp rx=0 tx=0).stop 2023-10-30T14:00:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.615+0000 7f689ef65640 1 -- 172.21.15.46:0/679984151 shutdown_connections 2023-10-30T14:00:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.615+0000 7f689ef65640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f687c0664d0 0x7f687c068990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.615+0000 7f689ef65640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68981538a0 0x7f68981efe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.615+0000 7f689ef65640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68980cf160 0x7f68981e2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.615+0000 7f689ef65640 1 --2- 172.21.15.46:0/679984151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68980ca0e0 0x7f68980cea20 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.615+0000 7f689ef65640 1 -- 172.21.15.46:0/679984151 >> 172.21.15.46:0/679984151 conn(0x7f689806bfe0 msgr2=0x7f689815a090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.615+0000 7f689ef65640 1 -- 172.21.15.46:0/679984151 shutdown_connections 2023-10-30T14:00:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:39.615+0000 7f689ef65640 1 -- 172.21.15.46:0/679984151 wait complete. 2023-10-30T14:00:39.636 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:00:41.637 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:41.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1fb0f640 1 Processor -- start 2023-10-30T14:00:41.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1fb0f640 1 -- start start 2023-10-30T14:00:41.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1fb0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc181538a0 0x7fdc1815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:41.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1fb0f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdc180cf160 0x7fdc180cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:41.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1fb0f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc180cb800 con 0x7fdc180ca0e0 2023-10-30T14:00:41.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1fb0f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1815dc30 con 0x7fdc180cf160 2023-10-30T14:00:41.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1fb0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180cb330 con 0x7fdc181538a0 2023-10-30T14:00:41.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1d0aa640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdc180ca0e0 0x7fdc1815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36820/0 (socket says 172.21.15.46:36820) 2023-10-30T14:00:41.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1d0aa640 1 -- 172.21.15.46:0/514345260 learned_addr learned my addr 172.21.15.46:0/514345260 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:00:41.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1d8ab640 1 --2- 172.21.15.46:0/514345260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc181538a0 0x7fdc1815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:41.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1e0ac640 1 --2- 172.21.15.46:0/514345260 >> v2:172.21.15.154:3300/0 conn(0x7fdc180cf160 0x7fdc180cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:41.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1c8a9640 1 -- 172.21.15.46:0/514345260 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534534668 0 0) 0x7fdc180cb800 con 0x7fdc180ca0e0 2023-10-30T14:00:41.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1c8a9640 1 -- 172.21.15.46:0/514345260 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc04003530 con 0x7fdc180ca0e0 2023-10-30T14:00:41.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1d8ab640 1 -- 172.21.15.46:0/514345260 >> v2:172.21.15.154:3300/0 conn(0x7fdc180cf160 msgr2=0x7fdc180cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:41.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1d8ab640 1 --2- 172.21.15.46:0/514345260 >> v2:172.21.15.154:3300/0 conn(0x7fdc180cf160 0x7fdc180cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:41.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1d8ab640 1 -- 172.21.15.46:0/514345260 >> v1:172.21.15.46:6789/0 conn(0x7fdc180ca0e0 legacy=0x7fdc1815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:41.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1c8a9640 1 -- 172.21.15.46:0/514345260 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3973520512 0 0) 0x7fdc04003530 con 0x7fdc180ca0e0 2023-10-30T14:00:41.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.791+0000 7fdc1d8ab640 1 -- 172.21.15.46:0/514345260 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc180cad70 con 0x7fdc181538a0 2023-10-30T14:00:41.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.795+0000 7fdc1d8ab640 1 --2- 172.21.15.46:0/514345260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc181538a0 0x7fdc1815d6f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdc140097c0 tx=0x7fdc1401eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f5bfb33a224a80a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:41.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.795+0000 7fdc1c8a9640 1 -- 172.21.15.46:0/514345260 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08001680 con 0x7fdc181538a0 2023-10-30T14:00:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.795+0000 7fdc1c8a9640 1 -- 172.21.15.46:0/514345260 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc140a85d0 con 0x7fdc181538a0 2023-10-30T14:00:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.795+0000 7fdc1c8a9640 1 -- 172.21.15.46:0/514345260 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc140a8940 con 0x7fdc181538a0 2023-10-30T14:00:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.795+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/514345260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc181538a0 msgr2=0x7fdc1815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.795+0000 7fdc1fb0f640 1 --2- 172.21.15.46:0/514345260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc181538a0 0x7fdc1815d6f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdc140097c0 tx=0x7fdc1401eb90 comp rx=0 tx=0).stop 2023-10-30T14:00:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.795+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/514345260 shutdown_connections 2023-10-30T14:00:41.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.795+0000 7fdc1fb0f640 1 --2- 172.21.15.46:0/514345260 >> v2:172.21.15.154:3300/0 conn(0x7fdc180cf160 0x7fdc180cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:41.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.795+0000 7fdc1fb0f640 1 --2- 172.21.15.46:0/514345260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc181538a0 0x7fdc1815d6f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:41.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.795+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/514345260 >> 172.21.15.46:0/514345260 conn(0x7fdc1806bfe0 msgr2=0x7fdc1806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:41.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.795+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/514345260 shutdown_connections 2023-10-30T14:00:41.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.795+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/514345260 wait complete. 2023-10-30T14:00:41.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1fb0f640 1 Processor -- start 2023-10-30T14:00:41.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1fb0f640 1 -- start start 2023-10-30T14:00:41.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1fb0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc180ca0e0 0x7fdc1815b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:41.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1fb0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc180cf160 0x7fdc181f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:41.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1fb0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc181538a0 0x7fdc181e6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:41.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1fb0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc181f32e0 con 0x7fdc181538a0 2023-10-30T14:00:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1fb0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1815a8b0 con 0x7fdc180cf160 2023-10-30T14:00:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1fb0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1815d580 con 0x7fdc180ca0e0 2023-10-30T14:00:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1e0ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc181538a0 0x7fdc181e6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1e0ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc181538a0 0x7fdc181e6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48332/0 (socket says 172.21.15.46:48332) 2023-10-30T14:00:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1e0ac640 1 -- 172.21.15.46:0/1250711061 learned_addr learned my addr 172.21.15.46:0/1250711061 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1d8ab640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc180ca0e0 0x7fdc1815b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1d0aa640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc180cf160 0x7fdc181f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1e0ac640 1 -- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc180ca0e0 msgr2=0x7fdc1815b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1e0ac640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc180ca0e0 0x7fdc1815b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1e0ac640 1 -- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc180cf160 msgr2=0x7fdc181f2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1e0ac640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc180cf160 0x7fdc181f2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1e0ac640 1 -- 172.21.15.46:0/1250711061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc14009420 con 0x7fdc181538a0 2023-10-30T14:00:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1e0ac640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc181538a0 0x7fdc181e6380 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000efc0 tx=0x7fdc1000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc0e7fc640 1 -- 172.21.15.46:0/1250711061 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1000ced0 con 0x7fdc181538a0 2023-10-30T14:00:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc0e7fc640 1 -- 172.21.15.46:0/1250711061 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1000f040 con 0x7fdc181538a0 2023-10-30T14:00:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc0e7fc640 1 -- 172.21.15.46:0/1250711061 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc100146c0 con 0x7fdc181538a0 2023-10-30T14:00:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/1250711061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1815d860 con 0x7fdc181538a0 2023-10-30T14:00:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.799+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/1250711061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1815ade0 con 0x7fdc181538a0 2023-10-30T14:00:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.803+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/1250711061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc1815bb20 con 0x7fdc181538a0 2023-10-30T14:00:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.803+0000 7fdc0e7fc640 1 -- 172.21.15.46:0/1250711061 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdc10014820 con 0x7fdc181538a0 2023-10-30T14:00:41.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.803+0000 7fdc0e7fc640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbfc0664d0 0x7fdbfc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:41.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.803+0000 7fdc0e7fc640 1 -- 172.21.15.46:0/1250711061 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fdc10084f80 con 0x7fdc181538a0 2023-10-30T14:00:41.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.803+0000 7fdc1d8ab640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbfc0664d0 0x7fdbfc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:41.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.803+0000 7fdc1d8ab640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbfc0664d0 0x7fdbfc068990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fdc140b2000 tx=0x7fdc140aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:41.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.807+0000 7fdc0e7fc640 1 -- 172.21.15.46:0/1250711061 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdc100b6090 con 0x7fdc181538a0 2023-10-30T14:00:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.995+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/1250711061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc180b2a50 con 0x7fdc181538a0 2023-10-30T14:00:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.995+0000 7fdc0e7fc640 1 -- 172.21.15.46:0/1250711061 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+5734 (secure 0 0 0) 0x7fdc10018020 con 0x7fdc181538a0 2023-10-30T14:00:41.998 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:41.998 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:39.205930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:resolve","state_seq":37,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:41.998 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 160 2023-10-30T14:00:41.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.999+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbfc0664d0 msgr2=0x7fdbfc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:42.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.999+0000 7fdc1fb0f640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbfc0664d0 0x7fdbfc068990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fdc140b2000 tx=0x7fdc140aa070 comp rx=0 tx=0).stop 2023-10-30T14:00:42.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.999+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/1250711061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc181538a0 msgr2=0x7fdc181e6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:42.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.999+0000 7fdc1fb0f640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc181538a0 0x7fdc181e6380 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000efc0 tx=0x7fdc1000c490 comp rx=0 tx=0).stop 2023-10-30T14:00:42.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.999+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/1250711061 shutdown_connections 2023-10-30T14:00:42.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.999+0000 7fdc1fb0f640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbfc0664d0 0x7fdbfc068990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:42.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.999+0000 7fdc1fb0f640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc181538a0 0x7fdc181e6380 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:42.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.999+0000 7fdc1fb0f640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc180cf160 0x7fdc181f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:42.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.999+0000 7fdc1fb0f640 1 --2- 172.21.15.46:0/1250711061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc180ca0e0 0x7fdc1815b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:42.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.999+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/1250711061 >> 172.21.15.46:0/1250711061 conn(0x7fdc1806bfe0 msgr2=0x7fdc18151420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:42.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.999+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/1250711061 shutdown_connections 2023-10-30T14:00:42.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:41.999+0000 7fdc1fb0f640 1 -- 172.21.15.46:0/1250711061 wait complete. 2023-10-30T14:00:42.019 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 160, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 160, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6809": { "addr": "172.21.15.46:6837/1135482564", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 1135482564, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 1135482564, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6809, "incarnation": 159, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:resolve", "state_seq": 37 }, "gid_7704": { "addr": "172.21.15.46:6836/3734066164", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3734066164, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3734066164, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7704, "incarnation": 152, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:00:39.205930+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7704, "mds_1": 6809 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:00:44.020 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:44.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.171+0000 7f189ecc9640 1 Processor -- start 2023-10-30T14:00:44.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.171+0000 7f189ecc9640 1 -- start start 2023-10-30T14:00:44.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189ecc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1898071110 0x7f189806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:44.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189ecc9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1898066cf0 con 0x7f189806f7a0 2023-10-30T14:00:44.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189ecc9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18980c70d0 con 0x7f18980c6bd0 2023-10-30T14:00:44.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189ecc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18980700c0 con 0x7f1898071110 2023-10-30T14:00:44.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189ca65640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f189806f7a0 0x7f1898069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36830/0 (socket says 172.21.15.46:36830) 2023-10-30T14:00:44.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189ca65640 1 -- 172.21.15.46:0/2437733806 learned_addr learned my addr 172.21.15.46:0/2437733806 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:00:44.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189d266640 1 --2- 172.21.15.46:0/2437733806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1898071110 0x7f189806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:44.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574193423 0 0) 0x7f1898066cf0 con 0x7f189806f7a0 2023-10-30T14:00:44.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1888003530 con 0x7f189806f7a0 2023-10-30T14:00:44.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2558616082 0 0) 0x7f1888003530 con 0x7f189806f7a0 2023-10-30T14:00:44.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1898071110 msgr2=0x7f189806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:44.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 --2- 172.21.15.46:0/2437733806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1898071110 0x7f189806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:44.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 >> v1:172.21.15.154:6789/0 conn(0x7f18980c6bd0 legacy=0x7f1898158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:44.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18980c7390 con 0x7f189806f7a0 2023-10-30T14:00:44.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f188c001680 con 0x7f189806f7a0 2023-10-30T14:00:44.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 >> v1:172.21.15.46:6789/0 conn(0x7f189806f7a0 legacy=0x7f1898069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:44.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 --2- 172.21.15.46:0/2437733806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18880039b0 0x7f1888003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:44.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 --2- 172.21.15.46:0/2437733806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1898071110 0x7f18880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:44.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 --2- 172.21.15.46:0/2437733806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1888004d10 0x7f1888005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:44.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18980700c0 con 0x7f1888004d10 2023-10-30T14:00:44.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188800b990 con 0x7f18880039b0 2023-10-30T14:00:44.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f188800bb50 con 0x7f1898071110 2023-10-30T14:00:44.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189d266640 1 --2- 172.21.15.46:0/2437733806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1888004d10 0x7f1888005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:44.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189ca65640 1 --2- 172.21.15.46:0/2437733806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18880039b0 0x7f1888003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:44.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f1897fff640 1 --2- 172.21.15.46:0/2437733806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1898071110 0x7f18880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:44.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189d266640 1 -- 172.21.15.46:0/2437733806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1898071110 msgr2=0x7f18880047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:44.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189d266640 1 --2- 172.21.15.46:0/2437733806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1898071110 0x7f18880047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:44.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189d266640 1 -- 172.21.15.46:0/2437733806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18880039b0 msgr2=0x7f1888003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:44.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189d266640 1 --2- 172.21.15.46:0/2437733806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18880039b0 0x7f1888003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:44.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.175+0000 7f189d266640 1 -- 172.21.15.46:0/2437733806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1888003530 con 0x7f1888004d10 2023-10-30T14:00:44.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f189d266640 1 --2- 172.21.15.46:0/2437733806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1888004d10 0x7f1888005190 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f1890009fc0 tx=0x7f189001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8875ff69d48a4f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:44.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18900a5c70 con 0x7f1888004d10 2023-10-30T14:00:44.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189000f040 con 0x7f1888004d10 2023-10-30T14:00:44.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f18977fe640 1 -- 172.21.15.46:0/2437733806 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18900ad440 con 0x7f1888004d10 2023-10-30T14:00:44.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f189ecc9640 1 -- 172.21.15.46:0/2437733806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1888004d10 msgr2=0x7f1888005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:44.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f189ecc9640 1 --2- 172.21.15.46:0/2437733806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1888004d10 0x7f1888005190 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f1890009fc0 tx=0x7f189001cc60 comp rx=0 tx=0).stop 2023-10-30T14:00:44.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f189d266640 1 -- 172.21.15.46:0/2437733806 reap_dead start 2023-10-30T14:00:44.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f189ecc9640 1 -- 172.21.15.46:0/2437733806 shutdown_connections 2023-10-30T14:00:44.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f189ecc9640 1 -- 172.21.15.46:0/2437733806 >> 172.21.15.46:0/2437733806 conn(0x7f18980c7910 msgr2=0x7f18980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:44.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f189ecc9640 1 -- 172.21.15.46:0/2437733806 shutdown_connections 2023-10-30T14:00:44.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f189ecc9640 1 -- 172.21.15.46:0/2437733806 wait complete. 2023-10-30T14:00:44.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f189ecc9640 1 Processor -- start 2023-10-30T14:00:44.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.179+0000 7f189ecc9640 1 -- start start 2023-10-30T14:00:44.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ecc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1888004d10 0x7f18980c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:44.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ecc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f189814d9b0 0x7f189814f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:44.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ecc9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f189814b650 0x7f189814fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:44.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ecc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189814ccc0 con 0x7f1888004d10 2023-10-30T14:00:44.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ecc9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189814e7b0 con 0x7f189814b650 2023-10-30T14:00:44.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ecc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189814df10 con 0x7f189814d9b0 2023-10-30T14:00:44.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ca65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1888004d10 0x7f18980c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:44.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ca65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1888004d10 0x7f18980c5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48346/0 (socket says 172.21.15.46:48346) 2023-10-30T14:00:44.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ca65640 1 -- 172.21.15.46:0/19845227 learned_addr learned my addr 172.21.15.46:0/19845227 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:44.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f1897fff640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f189814d9b0 0x7f189814f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:44.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189d266640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f189814b650 0x7f189814fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:44.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ca65640 1 -- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f189814d9b0 msgr2=0x7f189814f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:44.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ca65640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f189814d9b0 0x7f189814f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:44.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ca65640 1 -- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f189814b650 msgr2=0x7f189814fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:44.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ca65640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f189814b650 0x7f189814fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ca65640 1 -- 172.21.15.46:0/19845227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189814e1f0 con 0x7f1888004d10 2023-10-30T14:00:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189d266640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f189814b650 0x7f189814fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:00:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ca65640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1888004d10 0x7f18980c5d20 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f188c007290 tx=0x7f188c007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f18957fa640 1 -- 172.21.15.46:0/19845227 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c00f940 con 0x7f1888004d10 2023-10-30T14:00:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f18957fa640 1 -- 172.21.15.46:0/19845227 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188c00fac0 con 0x7f1888004d10 2023-10-30T14:00:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ecc9640 1 -- 172.21.15.46:0/19845227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f189814e4d0 con 0x7f1888004d10 2023-10-30T14:00:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f18957fa640 1 -- 172.21.15.46:0/19845227 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c00fe70 con 0x7f1888004d10 2023-10-30T14:00:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ecc9640 1 -- 172.21.15.46:0/19845227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f189814efc0 con 0x7f1888004d10 2023-10-30T14:00:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.183+0000 7f189ecc9640 1 -- 172.21.15.46:0/19845227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f189804e170 con 0x7f1888004d10 2023-10-30T14:00:44.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.187+0000 7f18957fa640 1 -- 172.21.15.46:0/19845227 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f188c01d460 con 0x7f1888004d10 2023-10-30T14:00:44.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.187+0000 7f18957fa640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f187c0664a0 0x7f187c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:44.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.187+0000 7f18957fa640 1 -- 172.21.15.46:0/19845227 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f188c0847e0 con 0x7f1888004d10 2023-10-30T14:00:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.187+0000 7f1897fff640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f187c0664a0 0x7f187c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.187+0000 7f1897fff640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f187c0664a0 0x7f187c068960 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f18981e01b0 tx=0x7f1884009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.191+0000 7f18957fa640 1 -- 172.21.15.46:0/19845227 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f188c0515b0 con 0x7f1888004d10 2023-10-30T14:00:44.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f189ecc9640 1 -- 172.21.15.46:0/19845227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18980c55a0 con 0x7f1888004d10 2023-10-30T14:00:44.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f18957fa640 1 -- 172.21.15.46:0/19845227 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+5734 (secure 0 0 0) 0x7f188c054bd0 con 0x7f1888004d10 2023-10-30T14:00:44.385 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:44.385 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:39.205930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:resolve","state_seq":37,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 160 2023-10-30T14:00:44.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f189ecc9640 1 -- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f187c0664a0 msgr2=0x7f187c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:44.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f189ecc9640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f187c0664a0 0x7f187c068960 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f18981e01b0 tx=0x7f1884009040 comp rx=0 tx=0).stop 2023-10-30T14:00:44.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f189ecc9640 1 -- 172.21.15.46:0/19845227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1888004d10 msgr2=0x7f18980c5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:44.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f189ecc9640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1888004d10 0x7f18980c5d20 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f188c007290 tx=0x7f188c007750 comp rx=0 tx=0).stop 2023-10-30T14:00:44.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f189ecc9640 1 -- 172.21.15.46:0/19845227 shutdown_connections 2023-10-30T14:00:44.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f189ecc9640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f187c0664a0 0x7f187c068960 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:44.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f189ecc9640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f189814b650 0x7f189814fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f189ecc9640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f189814d9b0 0x7f189814f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f189ecc9640 1 --2- 172.21.15.46:0/19845227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1888004d10 0x7f18980c5d20 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f189ecc9640 1 -- 172.21.15.46:0/19845227 >> 172.21.15.46:0/19845227 conn(0x7f18980c7910 msgr2=0x7f18980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.383+0000 7f189ecc9640 1 -- 172.21.15.46:0/19845227 shutdown_connections 2023-10-30T14:00:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:44.387+0000 7f189ecc9640 1 -- 172.21.15.46:0/19845227 wait complete. 2023-10-30T14:00:44.407 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:00:46.408 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:46.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe066bc5640 1 Processor -- start 2023-10-30T14:00:46.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe066bc5640 1 -- start start 2023-10-30T14:00:46.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe066bc5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe0601538a0 0x7fe06015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:46.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe066bc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0600cf160 0x7fe0600cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:46.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe066bc5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0600cb800 con 0x7fe0600ca0e0 2023-10-30T14:00:46.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe066bc5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06015dc30 con 0x7fe0601538a0 2023-10-30T14:00:46.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe066bc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600cb330 con 0x7fe0600cf160 2023-10-30T14:00:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe064961640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe0601538a0 0x7fe06015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe064961640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe0601538a0 0x7fe06015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37616/0 (socket says 172.21.15.46:37616) 2023-10-30T14:00:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe05ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe0600ca0e0 0x7fe06015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51454/0 (socket says 172.21.15.46:51454) 2023-10-30T14:00:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe064961640 1 -- 172.21.15.46:0/665903732 learned_addr learned my addr 172.21.15.46:0/665903732 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:46.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe065162640 1 --2- 172.21.15.46:0/665903732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0600cf160 0x7fe0600cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:46.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe064961640 1 -- 172.21.15.46:0/665903732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0600cf160 msgr2=0x7fe0600cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:46.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe064961640 1 --2- 172.21.15.46:0/665903732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0600cf160 0x7fe0600cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:46.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe064961640 1 -- 172.21.15.46:0/665903732 >> v1:172.21.15.46:6789/0 conn(0x7fe0600ca0e0 legacy=0x7fe06015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:46.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe05f7fe640 1 -- 172.21.15.46:0/665903732 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321040697 0 0) 0x7fe0600cb800 con 0x7fe0600ca0e0 2023-10-30T14:00:46.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.562+0000 7fe064961640 1 -- 172.21.15.46:0/665903732 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0600cad70 con 0x7fe0601538a0 2023-10-30T14:00:46.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.566+0000 7fe064961640 1 --2- 172.21.15.46:0/665903732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0601538a0 0x7fe06015d6f0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe054009870 tx=0x7fe05401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb37902a2cee98e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:46.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.566+0000 7fe05f7fe640 1 -- 172.21.15.46:0/665903732 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0540a8a30 con 0x7fe0601538a0 2023-10-30T14:00:46.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.566+0000 7fe05f7fe640 1 -- 172.21.15.46:0/665903732 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0540a8b90 con 0x7fe0601538a0 2023-10-30T14:00:46.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.566+0000 7fe05f7fe640 1 -- 172.21.15.46:0/665903732 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0540b0440 con 0x7fe0601538a0 2023-10-30T14:00:46.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.566+0000 7fe066bc5640 1 -- 172.21.15.46:0/665903732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0601538a0 msgr2=0x7fe06015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:46.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.566+0000 7fe066bc5640 1 --2- 172.21.15.46:0/665903732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0601538a0 0x7fe06015d6f0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe054009870 tx=0x7fe05401ed30 comp rx=0 tx=0).stop 2023-10-30T14:00:46.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.566+0000 7fe066bc5640 1 -- 172.21.15.46:0/665903732 shutdown_connections 2023-10-30T14:00:46.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.566+0000 7fe066bc5640 1 --2- 172.21.15.46:0/665903732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0600cf160 0x7fe0600cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:46.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.566+0000 7fe066bc5640 1 --2- 172.21.15.46:0/665903732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0601538a0 0x7fe06015d6f0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.566+0000 7fe066bc5640 1 -- 172.21.15.46:0/665903732 >> 172.21.15.46:0/665903732 conn(0x7fe06006bfe0 msgr2=0x7fe06006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.566+0000 7fe066bc5640 1 -- 172.21.15.46:0/665903732 shutdown_connections 2023-10-30T14:00:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.566+0000 7fe066bc5640 1 -- 172.21.15.46:0/665903732 wait complete. 2023-10-30T14:00:46.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe066bc5640 1 Processor -- start 2023-10-30T14:00:46.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe066bc5640 1 -- start start 2023-10-30T14:00:46.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe066bc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0600ca0e0 0x7fe06015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:46.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe066bc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0600cf160 0x7fe06015d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:46.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe066bc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0601538a0 0x7fe0601ef350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:46.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe066bc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06015bbc0 con 0x7fe0601538a0 2023-10-30T14:00:46.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe066bc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06015bea0 con 0x7fe0600ca0e0 2023-10-30T14:00:46.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe066bc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06015c180 con 0x7fe0600cf160 2023-10-30T14:00:46.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe065162640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0601538a0 0x7fe0601ef350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:46.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe065162640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0601538a0 0x7fe0601ef350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37628/0 (socket says 172.21.15.46:37628) 2023-10-30T14:00:46.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe065162640 1 -- 172.21.15.46:0/2460728222 learned_addr learned my addr 172.21.15.46:0/2460728222 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:46.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe05ffff640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0600cf160 0x7fe06015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:46.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe064961640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0600ca0e0 0x7fe06015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:46.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe065162640 1 -- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0600cf160 msgr2=0x7fe06015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:46.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe065162640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0600cf160 0x7fe06015d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:46.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe065162640 1 -- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0600ca0e0 msgr2=0x7fe06015b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:46.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe065162640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0600ca0e0 0x7fe06015b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:46.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe065162640 1 -- 172.21.15.46:0/2460728222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0540094d0 con 0x7fe0601538a0 2023-10-30T14:00:46.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe065162640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0601538a0 0x7fe0601ef350 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe058007d90 tx=0x7fe05800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:46.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe05d7fa640 1 -- 172.21.15.46:0/2460728222 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058014420 con 0x7fe0601538a0 2023-10-30T14:00:46.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe05d7fa640 1 -- 172.21.15.46:0/2460728222 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0580145a0 con 0x7fe0601538a0 2023-10-30T14:00:46.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe05d7fa640 1 -- 172.21.15.46:0/2460728222 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058014950 con 0x7fe0601538a0 2023-10-30T14:00:46.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe066bc5640 1 -- 172.21.15.46:0/2460728222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe06015c460 con 0x7fe0601538a0 2023-10-30T14:00:46.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe066bc5640 1 -- 172.21.15.46:0/2460728222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0600d0060 con 0x7fe0601538a0 2023-10-30T14:00:46.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.570+0000 7fe066bc5640 1 -- 172.21.15.46:0/2460728222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0600b2a50 con 0x7fe0601538a0 2023-10-30T14:00:46.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.574+0000 7fe05d7fa640 1 -- 172.21.15.46:0/2460728222 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe058014ab0 con 0x7fe0601538a0 2023-10-30T14:00:46.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.574+0000 7fe05d7fa640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0440664d0 0x7fe044068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:46.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.574+0000 7fe05d7fa640 1 -- 172.21.15.46:0/2460728222 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fe058084c60 con 0x7fe0601538a0 2023-10-30T14:00:46.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.574+0000 7fe064961640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0440664d0 0x7fe044068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:46.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.574+0000 7fe064961640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0440664d0 0x7fe044068990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe0540b2000 tx=0x7fe054009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:46.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.578+0000 7fe05d7fa640 1 -- 172.21.15.46:0/2460728222 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe05808a020 con 0x7fe0601538a0 2023-10-30T14:00:46.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.770+0000 7fe066bc5640 1 -- 172.21.15.46:0/2460728222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe06004e170 con 0x7fe0601538a0 2023-10-30T14:00:46.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.770+0000 7fe05d7fa640 1 -- 172.21.15.46:0/2460728222 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+5736 (secure 0 0 0) 0x7fe0580555c0 con 0x7fe0601538a0 2023-10-30T14:00:46.774 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:46.774 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:44.472269+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:reconnect","state_seq":39,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:46.774 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 161 2023-10-30T14:00:46.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.774+0000 7fe066bc5640 1 -- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0440664d0 msgr2=0x7fe044068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:46.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.774+0000 7fe066bc5640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0440664d0 0x7fe044068990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe0540b2000 tx=0x7fe054009040 comp rx=0 tx=0).stop 2023-10-30T14:00:46.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.774+0000 7fe066bc5640 1 -- 172.21.15.46:0/2460728222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0601538a0 msgr2=0x7fe0601ef350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:46.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.774+0000 7fe066bc5640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0601538a0 0x7fe0601ef350 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe058007d90 tx=0x7fe05800e4b0 comp rx=0 tx=0).stop 2023-10-30T14:00:46.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.774+0000 7fe066bc5640 1 -- 172.21.15.46:0/2460728222 shutdown_connections 2023-10-30T14:00:46.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.774+0000 7fe066bc5640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0440664d0 0x7fe044068990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:46.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.774+0000 7fe066bc5640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0601538a0 0x7fe0601ef350 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:46.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.774+0000 7fe066bc5640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0600cf160 0x7fe06015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:46.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.774+0000 7fe066bc5640 1 --2- 172.21.15.46:0/2460728222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0600ca0e0 0x7fe06015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:46.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.774+0000 7fe066bc5640 1 -- 172.21.15.46:0/2460728222 >> 172.21.15.46:0/2460728222 conn(0x7fe06006bfe0 msgr2=0x7fe0601527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:46.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.774+0000 7fe066bc5640 1 -- 172.21.15.46:0/2460728222 shutdown_connections 2023-10-30T14:00:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:46.774+0000 7fe066bc5640 1 -- 172.21.15.46:0/2460728222 wait complete. 2023-10-30T14:00:46.795 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:00:48.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:48.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a3be68640 1 Processor -- start 2023-10-30T14:00:48.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a3be68640 1 -- start start 2023-10-30T14:00:48.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a3be68640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340c6bd0 0x7f0a3406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:48.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a3be68640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0a34071110 0x7f0a34158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:48.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a3be68640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a34066cf0 con 0x7f0a3406f7a0 2023-10-30T14:00:48.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a3be68640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a340700c0 con 0x7f0a34071110 2023-10-30T14:00:48.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a3be68640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340c70d0 con 0x7f0a340c6bd0 2023-10-30T14:00:48.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a39c04640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0a3406f7a0 0x7f0a34069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51456/0 (socket says 172.21.15.46:51456) 2023-10-30T14:00:48.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a39c04640 1 -- 172.21.15.46:0/808298994 learned_addr learned my addr 172.21.15.46:0/808298994 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:00:48.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a3a405640 1 --2- 172.21.15.46:0/808298994 >> v2:172.21.15.154:3300/0 conn(0x7f0a34071110 0x7f0a34158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:48.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a39403640 1 --2- 172.21.15.46:0/808298994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340c6bd0 0x7f0a3406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:48.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137901244 0 0) 0x7f0a34066cf0 con 0x7f0a3406f7a0 2023-10-30T14:00:48.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a20003530 con 0x7f0a3406f7a0 2023-10-30T14:00:48.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3575236286 0 0) 0x7f0a20003530 con 0x7f0a3406f7a0 2023-10-30T14:00:48.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340c6bd0 msgr2=0x7f0a3406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:48.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 --2- 172.21.15.46:0/808298994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340c6bd0 0x7f0a3406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:48.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 >> v2:172.21.15.154:3300/0 conn(0x7f0a34071110 msgr2=0x7f0a34158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:48.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 --2- 172.21.15.46:0/808298994 >> v2:172.21.15.154:3300/0 conn(0x7f0a34071110 0x7f0a34158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:48.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a340c7390 con 0x7f0a3406f7a0 2023-10-30T14:00:48.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f0a30001680 con 0x7f0a3406f7a0 2023-10-30T14:00:48.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 >> v1:172.21.15.46:6789/0 conn(0x7f0a3406f7a0 legacy=0x7f0a34069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:48.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 --2- 172.21.15.46:0/808298994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340c6bd0 0x7f0a20003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:48.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 --2- 172.21.15.46:0/808298994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a20003f00 0x7f0a200047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:48.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 --2- 172.21.15.46:0/808298994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20004d10 0x7f0a20005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:48.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340700c0 con 0x7f0a20004d10 2023-10-30T14:00:48.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340c70d0 con 0x7f0a20003f00 2023-10-30T14:00:48.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2000b9c0 con 0x7f0a340c6bd0 2023-10-30T14:00:48.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a3a405640 1 --2- 172.21.15.46:0/808298994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20004d10 0x7f0a20005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:48.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a39c04640 1 --2- 172.21.15.46:0/808298994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340c6bd0 0x7f0a20003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:48.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a39403640 1 --2- 172.21.15.46:0/808298994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a20003f00 0x7f0a200047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:48.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a3a405640 1 -- 172.21.15.46:0/808298994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340c6bd0 msgr2=0x7f0a20003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:48.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.950+0000 7f0a3a405640 1 --2- 172.21.15.46:0/808298994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340c6bd0 0x7f0a20003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:48.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.954+0000 7f0a3a405640 1 -- 172.21.15.46:0/808298994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a20003f00 msgr2=0x7f0a200047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:48.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.954+0000 7f0a3a405640 1 --2- 172.21.15.46:0/808298994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a20003f00 0x7f0a200047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:48.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.954+0000 7f0a3a405640 1 -- 172.21.15.46:0/808298994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a20003530 con 0x7f0a20004d10 2023-10-30T14:00:48.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.954+0000 7f0a3a405640 1 --2- 172.21.15.46:0/808298994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20004d10 0x7f0a20005190 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009fc0 tx=0x7f0a2c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bb28b3a5f1687a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:48.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.954+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c0a5cb0 con 0x7f0a20004d10 2023-10-30T14:00:48.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.954+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2c00f040 con 0x7f0a20004d10 2023-10-30T14:00:48.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.954+0000 7f0a38c02640 1 -- 172.21.15.46:0/808298994 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c0ad680 con 0x7f0a20004d10 2023-10-30T14:00:48.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.954+0000 7f0a3be68640 1 -- 172.21.15.46:0/808298994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20004d10 msgr2=0x7f0a20005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:48.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.954+0000 7f0a3be68640 1 --2- 172.21.15.46:0/808298994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20004d10 0x7f0a20005190 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009fc0 tx=0x7f0a2c01cca0 comp rx=0 tx=0).stop 2023-10-30T14:00:48.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.954+0000 7f0a3a405640 1 -- 172.21.15.46:0/808298994 reap_dead start 2023-10-30T14:00:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.954+0000 7f0a3be68640 1 -- 172.21.15.46:0/808298994 shutdown_connections 2023-10-30T14:00:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.954+0000 7f0a3be68640 1 -- 172.21.15.46:0/808298994 >> 172.21.15.46:0/808298994 conn(0x7f0a340c7910 msgr2=0x7f0a340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a3be68640 1 -- 172.21.15.46:0/808298994 shutdown_connections 2023-10-30T14:00:48.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a3be68640 1 -- 172.21.15.46:0/808298994 wait complete. 2023-10-30T14:00:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a3be68640 1 Processor -- start 2023-10-30T14:00:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a3be68640 1 -- start start 2023-10-30T14:00:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a3be68640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20004d10 0x7f0a340ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a3be68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a341588a0 0x7f0a34156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a3be68640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340ae700 0x7f0a341ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a3be68640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340bff60 con 0x7f0a20004d10 2023-10-30T14:00:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a3be68640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a341f26f0 con 0x7f0a341588a0 2023-10-30T14:00:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a3be68640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a340c0200 con 0x7f0a340ae700 2023-10-30T14:00:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a39c04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20004d10 0x7f0a340ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a39c04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20004d10 0x7f0a340ae050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37634/0 (socket says 172.21.15.46:37634) 2023-10-30T14:00:48.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a39c04640 1 -- 172.21.15.46:0/28485557 learned_addr learned my addr 172.21.15.46:0/28485557 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:48.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a3a405640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340ae700 0x7f0a341ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:48.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a39403640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a341588a0 0x7f0a34156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:48.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a39c04640 1 -- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340ae700 msgr2=0x7f0a341ee5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:48.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a39c04640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340ae700 0x7f0a341ee5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:48.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a39c04640 1 -- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a341588a0 msgr2=0x7f0a34156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:48.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a39c04640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a341588a0 0x7f0a34156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:48.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a39c04640 1 -- 172.21.15.46:0/28485557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a340c04e0 con 0x7f0a20004d10 2023-10-30T14:00:48.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.958+0000 7f0a39c04640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20004d10 0x7f0a340ae050 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000e8c0 tx=0x7f0a3000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:48.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.962+0000 7f0a2a7fc640 1 -- 172.21.15.46:0/28485557 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3000c8d0 con 0x7f0a20004d10 2023-10-30T14:00:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.962+0000 7f0a2a7fc640 1 -- 172.21.15.46:0/28485557 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3000ca70 con 0x7f0a20004d10 2023-10-30T14:00:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.962+0000 7f0a2a7fc640 1 -- 172.21.15.46:0/28485557 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3000ce20 con 0x7f0a20004d10 2023-10-30T14:00:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.962+0000 7f0a3be68640 1 -- 172.21.15.46:0/28485557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a340c07c0 con 0x7f0a20004d10 2023-10-30T14:00:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.962+0000 7f0a3be68640 1 -- 172.21.15.46:0/28485557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a340c0ce0 con 0x7f0a20004d10 2023-10-30T14:00:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.962+0000 7f0a3be68640 1 -- 172.21.15.46:0/28485557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a340c09e0 con 0x7f0a20004d10 2023-10-30T14:00:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.962+0000 7f0a2a7fc640 1 -- 172.21.15.46:0/28485557 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0a3001d460 con 0x7f0a20004d10 2023-10-30T14:00:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.962+0000 7f0a2a7fc640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a180664d0 0x7f0a18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:48.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.962+0000 7f0a2a7fc640 1 -- 172.21.15.46:0/28485557 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f0a30084690 con 0x7f0a20004d10 2023-10-30T14:00:48.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.962+0000 7f0a39403640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a180664d0 0x7f0a18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:48.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.966+0000 7f0a39403640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a180664d0 0x7f0a18068990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0a340b0620 tx=0x7f0a24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:48.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:48.966+0000 7f0a2a7fc640 1 -- 172.21.15.46:0/28485557 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0a300513a0 con 0x7f0a20004d10 2023-10-30T14:00:49.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.158+0000 7f0a3be68640 1 -- 172.21.15.46:0/28485557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a3404e170 con 0x7f0a20004d10 2023-10-30T14:00:49.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.158+0000 7f0a2a7fc640 1 -- 172.21.15.46:0/28485557 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f0a300549c0 con 0x7f0a20004d10 2023-10-30T14:00:49.162 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:49.163 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:44.472269+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:reconnect","state_seq":39,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:49.163 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 161 2023-10-30T14:00:49.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.162+0000 7f0a3be68640 1 -- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a180664d0 msgr2=0x7f0a18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:49.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.162+0000 7f0a3be68640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a180664d0 0x7f0a18068990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0a340b0620 tx=0x7f0a24008040 comp rx=0 tx=0).stop 2023-10-30T14:00:49.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.162+0000 7f0a3be68640 1 -- 172.21.15.46:0/28485557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20004d10 msgr2=0x7f0a340ae050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:49.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.162+0000 7f0a3be68640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20004d10 0x7f0a340ae050 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000e8c0 tx=0x7f0a3000ed80 comp rx=0 tx=0).stop 2023-10-30T14:00:49.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.162+0000 7f0a3be68640 1 -- 172.21.15.46:0/28485557 shutdown_connections 2023-10-30T14:00:49.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.162+0000 7f0a3be68640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a180664d0 0x7f0a18068990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:49.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.162+0000 7f0a3be68640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a340ae700 0x7f0a341ee5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:49.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.162+0000 7f0a3be68640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a341588a0 0x7f0a34156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:49.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.162+0000 7f0a3be68640 1 --2- 172.21.15.46:0/28485557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20004d10 0x7f0a340ae050 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:49.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.162+0000 7f0a3be68640 1 -- 172.21.15.46:0/28485557 >> 172.21.15.46:0/28485557 conn(0x7f0a340c7910 msgr2=0x7f0a3406a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:49.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.162+0000 7f0a3be68640 1 -- 172.21.15.46:0/28485557 shutdown_connections 2023-10-30T14:00:49.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:49.162+0000 7f0a3be68640 1 -- 172.21.15.46:0/28485557 wait complete. 2023-10-30T14:00:49.183 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:00:51.185 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:51.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f944926b640 1 Processor -- start 2023-10-30T14:00:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f944926b640 1 -- start start 2023-10-30T14:00:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f944926b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94440c8cf0 0x7f94440c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f944926b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f94440ca340 0x7f944415d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f944926b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944415ce30 con 0x7f94440c90d0 2023-10-30T14:00:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f944926b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9444159880 con 0x7f94440c8cf0 2023-10-30T14:00:51.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f944926b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9444159ae0 con 0x7f94440ca340 2023-10-30T14:00:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f9442ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94440c8cf0 0x7f94440c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f9442ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94440c8cf0 0x7f94440c9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37636/0 (socket says 172.21.15.46:37636) 2023-10-30T14:00:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f9442ffd640 1 -- 172.21.15.46:0/3659531606 learned_addr learned my addr 172.21.15.46:0/3659531606 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f94437fe640 1 --2- 172.21.15.46:0/3659531606 >> v2:172.21.15.154:3300/0 conn(0x7f94440ca340 0x7f944415d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f9442ffd640 1 -- 172.21.15.46:0/3659531606 >> v2:172.21.15.154:3300/0 conn(0x7f94440ca340 msgr2=0x7f944415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f9442ffd640 1 --2- 172.21.15.46:0/3659531606 >> v2:172.21.15.154:3300/0 conn(0x7f94440ca340 0x7f944415d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f9442ffd640 1 -- 172.21.15.46:0/3659531606 >> v1:172.21.15.154:6789/0 conn(0x7f94440c90d0 legacy=0x7f94440cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:51.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.338+0000 7f9442ffd640 1 -- 172.21.15.46:0/3659531606 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9444159dc0 con 0x7f94440c8cf0 2023-10-30T14:00:51.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.342+0000 7f9442ffd640 1 --2- 172.21.15.46:0/3659531606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94440c8cf0 0x7f94440c9e00 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f94440c60c0 tx=0x7f943401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4935e5d7be5d588 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:51.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.342+0000 7f9441ffb640 1 -- 172.21.15.46:0/3659531606 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94340ac640 con 0x7f94440c8cf0 2023-10-30T14:00:51.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.342+0000 7f9441ffb640 1 -- 172.21.15.46:0/3659531606 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94340ac7a0 con 0x7f94440c8cf0 2023-10-30T14:00:51.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.342+0000 7f9441ffb640 1 -- 172.21.15.46:0/3659531606 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94340acb30 con 0x7f94440c8cf0 2023-10-30T14:00:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.342+0000 7f944926b640 1 -- 172.21.15.46:0/3659531606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94440c8cf0 msgr2=0x7f94440c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.342+0000 7f944926b640 1 --2- 172.21.15.46:0/3659531606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94440c8cf0 0x7f94440c9e00 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f94440c60c0 tx=0x7f943401c4d0 comp rx=0 tx=0).stop 2023-10-30T14:00:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.342+0000 7f944926b640 1 -- 172.21.15.46:0/3659531606 shutdown_connections 2023-10-30T14:00:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.342+0000 7f944926b640 1 --2- 172.21.15.46:0/3659531606 >> v2:172.21.15.154:3300/0 conn(0x7f94440ca340 0x7f944415d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:51.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.342+0000 7f944926b640 1 --2- 172.21.15.46:0/3659531606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94440c8cf0 0x7f94440c9e00 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:51.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.342+0000 7f944926b640 1 -- 172.21.15.46:0/3659531606 >> 172.21.15.46:0/3659531606 conn(0x7f944406be40 msgr2=0x7f944406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:51.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.342+0000 7f944926b640 1 -- 172.21.15.46:0/3659531606 shutdown_connections 2023-10-30T14:00:51.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.342+0000 7f944926b640 1 -- 172.21.15.46:0/3659531606 wait complete. 2023-10-30T14:00:51.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f944926b640 1 Processor -- start 2023-10-30T14:00:51.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f944926b640 1 -- start start 2023-10-30T14:00:51.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f944926b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94440c8cf0 0x7f94441f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:51.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f944926b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94440c90d0 0x7f9444154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:51.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f944926b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94440ca340 0x7f94440c7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:51.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f944926b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94440c8850 con 0x7f94440c90d0 2023-10-30T14:00:51.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f944926b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94440c82b0 con 0x7f94440c8cf0 2023-10-30T14:00:51.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f944926b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94440c8570 con 0x7f94440ca340 2023-10-30T14:00:51.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f94427fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94440c90d0 0x7f9444154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:51.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f94427fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94440c90d0 0x7f9444154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37642/0 (socket says 172.21.15.46:37642) 2023-10-30T14:00:51.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f94427fc640 1 -- 172.21.15.46:0/407366512 learned_addr learned my addr 172.21.15.46:0/407366512 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:51.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f94437fe640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94440ca340 0x7f94440c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f9442ffd640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94440c8cf0 0x7f94441f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f94427fc640 1 -- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94440ca340 msgr2=0x7f94440c7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f94427fc640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94440ca340 0x7f94440c7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f94427fc640 1 -- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94440c8cf0 msgr2=0x7f94441f1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f94427fc640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94440c8cf0 0x7f94441f1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f94427fc640 1 -- 172.21.15.46:0/407366512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94340094d0 con 0x7f94440c90d0 2023-10-30T14:00:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f9442ffd640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94440c8cf0 0x7f94441f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:00:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f94427fc640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94440c90d0 0x7f9444154450 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f943800c500 tx=0x7f943800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f942b7fe640 1 -- 172.21.15.46:0/407366512 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438005220 con 0x7f94440c90d0 2023-10-30T14:00:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f942b7fe640 1 -- 172.21.15.46:0/407366512 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9438005380 con 0x7f94440c90d0 2023-10-30T14:00:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f942b7fe640 1 -- 172.21.15.46:0/407366512 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438007270 con 0x7f94440c90d0 2023-10-30T14:00:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f944926b640 1 -- 172.21.15.46:0/407366512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94441f5430 con 0x7f94440c90d0 2023-10-30T14:00:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.346+0000 7f944926b640 1 -- 172.21.15.46:0/407366512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94441f59f0 con 0x7f94440c90d0 2023-10-30T14:00:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.350+0000 7f944926b640 1 -- 172.21.15.46:0/407366512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944404e170 con 0x7f94440c90d0 2023-10-30T14:00:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.350+0000 7f942b7fe640 1 -- 172.21.15.46:0/407366512 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94380073d0 con 0x7f94440c90d0 2023-10-30T14:00:51.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.350+0000 7f942b7fe640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94300664a0 0x7f9430068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:51.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.350+0000 7f942b7fe640 1 -- 172.21.15.46:0/407366512 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f9438083e10 con 0x7f94440c90d0 2023-10-30T14:00:51.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.350+0000 7f9442ffd640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94300664a0 0x7f9430068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.350+0000 7f9442ffd640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94300664a0 0x7f9430068960 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f94340ae000 tx=0x7f9434008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:51.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.354+0000 7f942b7fe640 1 -- 172.21.15.46:0/407366512 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9438087030 con 0x7f94440c90d0 2023-10-30T14:00:51.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.542+0000 7f944926b640 1 -- 172.21.15.46:0/407366512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94440cf810 con 0x7f94440c90d0 2023-10-30T14:00:51.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.542+0000 7f942b7fe640 1 -- 172.21.15.46:0/407366512 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f9438050b30 con 0x7f94440c90d0 2023-10-30T14:00:51.547 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:51.547 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:50.461676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:51.547 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 162 2023-10-30T14:00:51.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.546+0000 7f944926b640 1 -- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94300664a0 msgr2=0x7f9430068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:51.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.546+0000 7f944926b640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94300664a0 0x7f9430068960 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f94340ae000 tx=0x7f9434008040 comp rx=0 tx=0).stop 2023-10-30T14:00:51.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.546+0000 7f944926b640 1 -- 172.21.15.46:0/407366512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94440c90d0 msgr2=0x7f9444154450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:51.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.546+0000 7f944926b640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94440c90d0 0x7f9444154450 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f943800c500 tx=0x7f943800c9c0 comp rx=0 tx=0).stop 2023-10-30T14:00:51.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.546+0000 7f944926b640 1 -- 172.21.15.46:0/407366512 shutdown_connections 2023-10-30T14:00:51.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.546+0000 7f944926b640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94300664a0 0x7f9430068960 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:51.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.546+0000 7f944926b640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94440ca340 0x7f94440c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:51.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.546+0000 7f944926b640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94440c90d0 0x7f9444154450 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:51.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.546+0000 7f944926b640 1 --2- 172.21.15.46:0/407366512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94440c8cf0 0x7f94441f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:51.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.546+0000 7f944926b640 1 -- 172.21.15.46:0/407366512 >> 172.21.15.46:0/407366512 conn(0x7f944406be40 msgr2=0x7f94440cba90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:51.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.546+0000 7f944926b640 1 -- 172.21.15.46:0/407366512 shutdown_connections 2023-10-30T14:00:51.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:51.546+0000 7f944926b640 1 -- 172.21.15.46:0/407366512 wait complete. 2023-10-30T14:00:51.568 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:00:53.569 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:53.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.722+0000 7fda9c33e640 1 Processor -- start 2023-10-30T14:00:53.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.722+0000 7fda9c33e640 1 -- start start 2023-10-30T14:00:53.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda9c33e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda9406f7a0 0x7fda9406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:53.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda9c33e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda940c6bd0 0x7fda94069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:53.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda9c33e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9406a3d0 con 0x7fda94071110 2023-10-30T14:00:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda9c33e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda940700c0 con 0x7fda940c6bd0 2023-10-30T14:00:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda9c33e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda94066cf0 con 0x7fda9406f7a0 2023-10-30T14:00:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda998d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda940c6bd0 0x7fda94069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:53.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda9a8db640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fda94071110 0x7fda94158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51464/0 (socket says 172.21.15.46:51464) 2023-10-30T14:00:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda9a8db640 1 -- 172.21.15.46:0/1935913332 learned_addr learned my addr 172.21.15.46:0/1935913332 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:00:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda9a0da640 1 --2- 172.21.15.46:0/1935913332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda9406f7a0 0x7fda9406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda998d9640 1 -- 172.21.15.46:0/1935913332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda9406f7a0 msgr2=0x7fda9406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda998d9640 1 --2- 172.21.15.46:0/1935913332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda9406f7a0 0x7fda9406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:53.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda998d9640 1 -- 172.21.15.46:0/1935913332 >> v1:172.21.15.46:6789/0 conn(0x7fda94071110 legacy=0x7fda94158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda998d9640 1 -- 172.21.15.46:0/1935913332 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda940c70d0 con 0x7fda940c6bd0 2023-10-30T14:00:53.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda990d8640 1 -- 172.21.15.46:0/1935913332 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179216528 0 0) 0x7fda9406a3d0 con 0x7fda94071110 2023-10-30T14:00:53.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda998d9640 1 --2- 172.21.15.46:0/1935913332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda940c6bd0 0x7fda94069e90 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fda840097c0 tx=0x7fda8401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591af5110d33ebe9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:53.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.726+0000 7fda990d8640 1 -- 172.21.15.46:0/1935913332 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda840a8970 con 0x7fda940c6bd0 2023-10-30T14:00:53.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda990d8640 1 -- 172.21.15.46:0/1935913332 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda840a8ad0 con 0x7fda940c6bd0 2023-10-30T14:00:53.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda990d8640 1 -- 172.21.15.46:0/1935913332 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda840a8e60 con 0x7fda940c6bd0 2023-10-30T14:00:53.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 -- 172.21.15.46:0/1935913332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda940c6bd0 msgr2=0x7fda94069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:53.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 --2- 172.21.15.46:0/1935913332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda940c6bd0 0x7fda94069e90 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fda840097c0 tx=0x7fda8401ec70 comp rx=0 tx=0).stop 2023-10-30T14:00:53.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 -- 172.21.15.46:0/1935913332 shutdown_connections 2023-10-30T14:00:53.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 --2- 172.21.15.46:0/1935913332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda940c6bd0 0x7fda94069e90 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:53.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 --2- 172.21.15.46:0/1935913332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda9406f7a0 0x7fda9406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:53.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 -- 172.21.15.46:0/1935913332 >> 172.21.15.46:0/1935913332 conn(0x7fda940c7910 msgr2=0x7fda940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:53.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 -- 172.21.15.46:0/1935913332 shutdown_connections 2023-10-30T14:00:53.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 -- 172.21.15.46:0/1935913332 wait complete. 2023-10-30T14:00:53.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 Processor -- start 2023-10-30T14:00:53.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 -- start start 2023-10-30T14:00:53.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9406f7a0 0x7fda941592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:53.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda94071110 0x7fda940c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:53.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda940c6bd0 0x7fda940c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:53.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda941ed6d0 con 0x7fda940c6bd0 2023-10-30T14:00:53.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda940cab10 con 0x7fda9406f7a0 2023-10-30T14:00:53.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9c33e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda941ec800 con 0x7fda94071110 2023-10-30T14:00:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9a8db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda940c6bd0 0x7fda940c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9a8db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda940c6bd0 0x7fda940c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37664/0 (socket says 172.21.15.46:37664) 2023-10-30T14:00:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9a8db640 1 -- 172.21.15.46:0/2506268959 learned_addr learned my addr 172.21.15.46:0/2506268959 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda998d9640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda94071110 0x7fda940c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:53.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.730+0000 7fda9a0da640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9406f7a0 0x7fda941592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:53.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda9a8db640 1 -- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda94071110 msgr2=0x7fda940c3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:53.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda9a8db640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda94071110 0x7fda940c3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:53.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda9a8db640 1 -- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9406f7a0 msgr2=0x7fda941592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:53.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda9a8db640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9406f7a0 0x7fda941592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:53.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda9a8db640 1 -- 172.21.15.46:0/2506268959 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda84009420 con 0x7fda940c6bd0 2023-10-30T14:00:53.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda9a8db640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda940c6bd0 0x7fda940c9ec0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00c780 tx=0x7fda8c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:53.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda8affd640 1 -- 172.21.15.46:0/2506268959 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c00ce90 con 0x7fda940c6bd0 2023-10-30T14:00:53.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda8affd640 1 -- 172.21.15.46:0/2506268959 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8c00a640 con 0x7fda940c6bd0 2023-10-30T14:00:53.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda8affd640 1 -- 172.21.15.46:0/2506268959 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c00a9d0 con 0x7fda940c6bd0 2023-10-30T14:00:53.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda9c33e640 1 -- 172.21.15.46:0/2506268959 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda941ecac0 con 0x7fda940c6bd0 2023-10-30T14:00:53.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda9c33e640 1 -- 172.21.15.46:0/2506268959 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda941ed080 con 0x7fda940c6bd0 2023-10-30T14:00:53.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda9c33e640 1 -- 172.21.15.46:0/2506268959 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda94070330 con 0x7fda940c6bd0 2023-10-30T14:00:53.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda8affd640 1 -- 172.21.15.46:0/2506268959 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fda8c00ab30 con 0x7fda940c6bd0 2023-10-30T14:00:53.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda8affd640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda780664f0 0x7fda780689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:53.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.734+0000 7fda8affd640 1 -- 172.21.15.46:0/2506268959 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fda8c083e10 con 0x7fda940c6bd0 2023-10-30T14:00:53.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.738+0000 7fda9a0da640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda780664f0 0x7fda780689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:53.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.738+0000 7fda9a0da640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda780664f0 0x7fda780689b0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fda9000b930 tx=0x7fda90008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:53.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.738+0000 7fda8affd640 1 -- 172.21.15.46:0/2506268959 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fda94070330 con 0x7fda940c6bd0 2023-10-30T14:00:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.930+0000 7fda9c33e640 1 -- 172.21.15.46:0/2506268959 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda9415aca0 con 0x7fda940c6bd0 2023-10-30T14:00:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.930+0000 7fda8affd640 1 -- 172.21.15.46:0/2506268959 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fda8c018030 con 0x7fda940c6bd0 2023-10-30T14:00:53.934 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:53.934 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:50.461676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:53.934 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 162 2023-10-30T14:00:53.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.934+0000 7fda9c33e640 1 -- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda780664f0 msgr2=0x7fda780689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:53.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.934+0000 7fda9c33e640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda780664f0 0x7fda780689b0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fda9000b930 tx=0x7fda90008040 comp rx=0 tx=0).stop 2023-10-30T14:00:53.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.934+0000 7fda9c33e640 1 -- 172.21.15.46:0/2506268959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda940c6bd0 msgr2=0x7fda940c9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:53.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.934+0000 7fda9c33e640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda940c6bd0 0x7fda940c9ec0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00c780 tx=0x7fda8c00cc40 comp rx=0 tx=0).stop 2023-10-30T14:00:53.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.934+0000 7fda9c33e640 1 -- 172.21.15.46:0/2506268959 shutdown_connections 2023-10-30T14:00:53.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.934+0000 7fda9c33e640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda780664f0 0x7fda780689b0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:53.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.934+0000 7fda9c33e640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda940c6bd0 0x7fda940c9ec0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:53.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.934+0000 7fda9c33e640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda94071110 0x7fda940c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:53.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.934+0000 7fda9c33e640 1 --2- 172.21.15.46:0/2506268959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9406f7a0 0x7fda941592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:53.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.934+0000 7fda9c33e640 1 -- 172.21.15.46:0/2506268959 >> 172.21.15.46:0/2506268959 conn(0x7fda940c7910 msgr2=0x7fda940c5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:53.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.934+0000 7fda9c33e640 1 -- 172.21.15.46:0/2506268959 shutdown_connections 2023-10-30T14:00:53.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:53.934+0000 7fda9c33e640 1 -- 172.21.15.46:0/2506268959 wait complete. 2023-10-30T14:00:53.956 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 162, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 162, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6809": { "addr": "172.21.15.46:6837/1135482564", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 1135482564, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 1135482564, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6809, "incarnation": 159, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 41 }, "gid_7704": { "addr": "172.21.15.46:6836/3734066164", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3734066164, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3734066164, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7704, "incarnation": 152, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:00:50.461676+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7704, "mds_1": 6809 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:00:54.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:00:54.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:00:55.957 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.098+0000 7ffb76fd3640 1 Processor -- start 2023-10-30T14:00:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.098+0000 7ffb76fd3640 1 -- start start 2023-10-30T14:00:56.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb76fd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb7006f7a0 0x7ffb7006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:56.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb76fd3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb70066cf0 con 0x7ffb700c6bd0 2023-10-30T14:00:56.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb76fd3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb700c70d0 con 0x7ffb70071110 2023-10-30T14:00:56.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb76fd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb700700c0 con 0x7ffb7006f7a0 2023-10-30T14:00:56.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffb700c6bd0 0x7ffb70069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51468/0 (socket says 172.21.15.46:51468) 2023-10-30T14:00:56.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6ffff640 1 -- 172.21.15.46:0/2850567572 learned_addr learned my addr 172.21.15.46:0/2850567572 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:00:56.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb74d6f640 1 --2- 172.21.15.46:0/2850567572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb7006f7a0 0x7ffb7006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:56.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242338448 0 0) 0x7ffb70066cf0 con 0x7ffb700c6bd0 2023-10-30T14:00:56.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb60003530 con 0x7ffb700c6bd0 2023-10-30T14:00:56.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3539881185 0 0) 0x7ffb60003530 con 0x7ffb700c6bd0 2023-10-30T14:00:56.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb7006f7a0 msgr2=0x7ffb7006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:56.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 --2- 172.21.15.46:0/2850567572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb7006f7a0 0x7ffb7006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:56.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 >> v1:172.21.15.154:6789/0 conn(0x7ffb70071110 legacy=0x7ffb70158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:56.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb700c7390 con 0x7ffb700c6bd0 2023-10-30T14:00:56.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7ffb5c001680 con 0x7ffb700c6bd0 2023-10-30T14:00:56.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 >> v1:172.21.15.46:6789/0 conn(0x7ffb700c6bd0 legacy=0x7ffb70069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:56.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 --2- 172.21.15.46:0/2850567572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb600039b0 0x7ffb60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:56.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 --2- 172.21.15.46:0/2850567572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb7006f7a0 0x7ffb600047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:56.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 --2- 172.21.15.46:0/2850567572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb60004d10 0x7ffb60005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:56.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb700700c0 con 0x7ffb60004d10 2023-10-30T14:00:56.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6000b990 con 0x7ffb600039b0 2023-10-30T14:00:56.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6000bb50 con 0x7ffb7006f7a0 2023-10-30T14:00:56.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb75570640 1 --2- 172.21.15.46:0/2850567572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb60004d10 0x7ffb60005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:56.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb74d6f640 1 --2- 172.21.15.46:0/2850567572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb600039b0 0x7ffb60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:56.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb6ffff640 1 --2- 172.21.15.46:0/2850567572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb7006f7a0 0x7ffb600047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:56.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb75570640 1 -- 172.21.15.46:0/2850567572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb7006f7a0 msgr2=0x7ffb600047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:56.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb75570640 1 --2- 172.21.15.46:0/2850567572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb7006f7a0 0x7ffb600047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:56.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb75570640 1 -- 172.21.15.46:0/2850567572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb600039b0 msgr2=0x7ffb60003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:56.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb75570640 1 --2- 172.21.15.46:0/2850567572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb600039b0 0x7ffb60003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:56.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.102+0000 7ffb75570640 1 -- 172.21.15.46:0/2850567572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb60003530 con 0x7ffb60004d10 2023-10-30T14:00:56.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.106+0000 7ffb75570640 1 --2- 172.21.15.46:0/2850567572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb60004d10 0x7ffb60005190 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7ffb68002010 tx=0x7ffb6801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3324232846709af7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:56.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.106+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb680a57b0 con 0x7ffb60004d10 2023-10-30T14:00:56.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.106+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb680a5930 con 0x7ffb60004d10 2023-10-30T14:00:56.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.106+0000 7ffb6f7fe640 1 -- 172.21.15.46:0/2850567572 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb680a5ce0 con 0x7ffb60004d10 2023-10-30T14:00:56.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.106+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2850567572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb60004d10 msgr2=0x7ffb60005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:56.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.106+0000 7ffb76fd3640 1 --2- 172.21.15.46:0/2850567572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb60004d10 0x7ffb60005190 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7ffb68002010 tx=0x7ffb6801c8c0 comp rx=0 tx=0).stop 2023-10-30T14:00:56.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.106+0000 7ffb75570640 1 -- 172.21.15.46:0/2850567572 reap_dead start 2023-10-30T14:00:56.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.106+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2850567572 shutdown_connections 2023-10-30T14:00:56.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.106+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2850567572 >> 172.21.15.46:0/2850567572 conn(0x7ffb700c7910 msgr2=0x7ffb700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:56.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.106+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2850567572 shutdown_connections 2023-10-30T14:00:56.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.106+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2850567572 wait complete. 2023-10-30T14:00:56.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb76fd3640 1 Processor -- start 2023-10-30T14:00:56.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb76fd3640 1 -- start start 2023-10-30T14:00:56.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb76fd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb60004d10 0x7ffb701ed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:56.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb76fd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb70154690 0x7ffb700c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:56.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb76fd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb701efc20 0x7ffb7014f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:56.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb76fd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb701f00a0 con 0x7ffb60004d10 2023-10-30T14:00:56.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb76fd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb700ca600 con 0x7ffb701efc20 2023-10-30T14:00:56.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb76fd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb700ca040 con 0x7ffb70154690 2023-10-30T14:00:56.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb74d6f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb60004d10 0x7ffb701ed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:56.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb74d6f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb60004d10 0x7ffb701ed2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37676/0 (socket says 172.21.15.46:37676) 2023-10-30T14:00:56.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb74d6f640 1 -- 172.21.15.46:0/2365021738 learned_addr learned my addr 172.21.15.46:0/2365021738 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:56.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb6ffff640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb70154690 0x7ffb700c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:56.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb75570640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb701efc20 0x7ffb7014f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:56.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb74d6f640 1 -- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb70154690 msgr2=0x7ffb700c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:56.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb74d6f640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb70154690 0x7ffb700c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:56.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb74d6f640 1 -- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb701efc20 msgr2=0x7ffb7014f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:56.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb74d6f640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb701efc20 0x7ffb7014f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:56.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb74d6f640 1 -- 172.21.15.46:0/2365021738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb700ca320 con 0x7ffb60004d10 2023-10-30T14:00:56.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb6ffff640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb70154690 0x7ffb700c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:00:56.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb74d6f640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb60004d10 0x7ffb701ed2a0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7ffb640119d0 tx=0x7ffb64011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:56.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb6d7fa640 1 -- 172.21.15.46:0/2365021738 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6400fb50 con 0x7ffb60004d10 2023-10-30T14:00:56.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb6d7fa640 1 -- 172.21.15.46:0/2365021738 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6400fcb0 con 0x7ffb60004d10 2023-10-30T14:00:56.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb6d7fa640 1 -- 172.21.15.46:0/2365021738 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64014440 con 0x7ffb60004d10 2023-10-30T14:00:56.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2365021738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb701ec7c0 con 0x7ffb60004d10 2023-10-30T14:00:56.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2365021738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb701ecd80 con 0x7ffb60004d10 2023-10-30T14:00:56.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.110+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2365021738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb70154a70 con 0x7ffb60004d10 2023-10-30T14:00:56.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.114+0000 7ffb6d7fa640 1 -- 172.21.15.46:0/2365021738 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffb640145a0 con 0x7ffb60004d10 2023-10-30T14:00:56.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.114+0000 7ffb6d7fa640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffb540664d0 0x7ffb54068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:56.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.114+0000 7ffb6d7fa640 1 -- 172.21.15.46:0/2365021738 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7ffb64084e70 con 0x7ffb60004d10 2023-10-30T14:00:56.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.114+0000 7ffb6ffff640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffb540664d0 0x7ffb54068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:56.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.114+0000 7ffb6ffff640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffb540664d0 0x7ffb54068990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ffb701f0d00 tx=0x7ffb5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:56.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.118+0000 7ffb6d7fa640 1 -- 172.21.15.46:0/2365021738 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffb64051c30 con 0x7ffb60004d10 2023-10-30T14:00:56.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.306+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2365021738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb7004e170 con 0x7ffb60004d10 2023-10-30T14:00:56.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.306+0000 7ffb6d7fa640 1 -- 172.21.15.46:0/2365021738 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+5739 (secure 0 0 0) 0x7ffb64022070 con 0x7ffb60004d10 2023-10-30T14:00:56.310 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:56.311 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:55.609732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:clientreplay","state_seq":43,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:56.311 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 163 2023-10-30T14:00:56.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.310+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffb540664d0 msgr2=0x7ffb54068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:56.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.310+0000 7ffb76fd3640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffb540664d0 0x7ffb54068990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ffb701f0d00 tx=0x7ffb5c009040 comp rx=0 tx=0).stop 2023-10-30T14:00:56.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.310+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2365021738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb60004d10 msgr2=0x7ffb701ed2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:56.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.310+0000 7ffb76fd3640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb60004d10 0x7ffb701ed2a0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7ffb640119d0 tx=0x7ffb64011e90 comp rx=0 tx=0).stop 2023-10-30T14:00:56.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.310+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2365021738 shutdown_connections 2023-10-30T14:00:56.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.310+0000 7ffb76fd3640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffb540664d0 0x7ffb54068990 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:56.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.310+0000 7ffb76fd3640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb701efc20 0x7ffb7014f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:56.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.310+0000 7ffb76fd3640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb70154690 0x7ffb700c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:56.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.310+0000 7ffb76fd3640 1 --2- 172.21.15.46:0/2365021738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb60004d10 0x7ffb701ed2a0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:56.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.310+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2365021738 >> 172.21.15.46:0/2365021738 conn(0x7ffb700c7910 msgr2=0x7ffb7014fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:56.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.310+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2365021738 shutdown_connections 2023-10-30T14:00:56.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:56.310+0000 7ffb76fd3640 1 -- 172.21.15.46:0/2365021738 wait complete. 2023-10-30T14:00:56.331 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:00:58.333 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:00:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb31e5f5640 1 Processor -- start 2023-10-30T14:00:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb31e5f5640 1 -- start start 2023-10-30T14:00:58.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb31e5f5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3181538a0 0x7fb31815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:58.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb31e5f5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3180ca0e0 0x7fb31815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:58.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb31e5f5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3180cb800 con 0x7fb3180cf160 2023-10-30T14:00:58.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb31e5f5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31815dc30 con 0x7fb3181538a0 2023-10-30T14:00:58.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb31e5f5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180cb330 con 0x7fb3180ca0e0 2023-10-30T14:00:58.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb317fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3181538a0 0x7fb31815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:58.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb317fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3181538a0 0x7fb31815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39008/0 (socket says 172.21.15.46:39008) 2023-10-30T14:00:58.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb317fff640 1 -- 172.21.15.46:0/248560629 learned_addr learned my addr 172.21.15.46:0/248560629 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:58.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb3177fe640 1 --2- 172.21.15.46:0/248560629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3180ca0e0 0x7fb31815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:58.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb317fff640 1 -- 172.21.15.46:0/248560629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3180ca0e0 msgr2=0x7fb31815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:00:58.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb317fff640 1 --2- 172.21.15.46:0/248560629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3180ca0e0 0x7fb31815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:58.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb317fff640 1 -- 172.21.15.46:0/248560629 >> v1:172.21.15.46:6789/0 conn(0x7fb3180cf160 legacy=0x7fb3180cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:58.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb317fff640 1 -- 172.21.15.46:0/248560629 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3180cad70 con 0x7fb3181538a0 2023-10-30T14:00:58.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.486+0000 7fb316ffd640 1 -- 172.21.15.46:0/248560629 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305298869 0 0) 0x7fb3180cb800 con 0x7fb3180cf160 2023-10-30T14:00:58.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.490+0000 7fb317fff640 1 --2- 172.21.15.46:0/248560629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3181538a0 0x7fb31815d6f0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fb308009870 tx=0x7fb30801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3667cefe58eecc2c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:58.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.490+0000 7fb316ffd640 1 -- 172.21.15.46:0/248560629 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3080a8a30 con 0x7fb3181538a0 2023-10-30T14:00:58.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.490+0000 7fb316ffd640 1 -- 172.21.15.46:0/248560629 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3080a8b90 con 0x7fb3181538a0 2023-10-30T14:00:58.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.490+0000 7fb316ffd640 1 -- 172.21.15.46:0/248560629 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3080b0440 con 0x7fb3181538a0 2023-10-30T14:00:58.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.490+0000 7fb31e5f5640 1 -- 172.21.15.46:0/248560629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3181538a0 msgr2=0x7fb31815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:58.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.490+0000 7fb31e5f5640 1 --2- 172.21.15.46:0/248560629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3181538a0 0x7fb31815d6f0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fb308009870 tx=0x7fb30801ed30 comp rx=0 tx=0).stop 2023-10-30T14:00:58.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.490+0000 7fb31e5f5640 1 -- 172.21.15.46:0/248560629 shutdown_connections 2023-10-30T14:00:58.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.490+0000 7fb31e5f5640 1 --2- 172.21.15.46:0/248560629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3180ca0e0 0x7fb31815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:58.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.490+0000 7fb31e5f5640 1 --2- 172.21.15.46:0/248560629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3181538a0 0x7fb31815d6f0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:58.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.490+0000 7fb31e5f5640 1 -- 172.21.15.46:0/248560629 >> 172.21.15.46:0/248560629 conn(0x7fb31806bfe0 msgr2=0x7fb31806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:58.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.490+0000 7fb31e5f5640 1 -- 172.21.15.46:0/248560629 shutdown_connections 2023-10-30T14:00:58.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.490+0000 7fb31e5f5640 1 -- 172.21.15.46:0/248560629 wait complete. 2023-10-30T14:00:58.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb31e5f5640 1 Processor -- start 2023-10-30T14:00:58.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb31e5f5640 1 -- start start 2023-10-30T14:00:58.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb31e5f5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3180ca0e0 0x7fb318157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:58.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb31e5f5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3180cf160 0x7fb3181e5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:58.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb31e5f5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3181538a0 0x7fb3180b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:58.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb31e5f5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb318158d70 con 0x7fb3180ca0e0 2023-10-30T14:00:58.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb31e5f5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180b1a00 con 0x7fb3180cf160 2023-10-30T14:00:58.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb31e5f5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3181e5550 con 0x7fb3181538a0 2023-10-30T14:00:58.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb317fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3180ca0e0 0x7fb318157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:58.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb317fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3180ca0e0 0x7fb318157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39010/0 (socket says 172.21.15.46:39010) 2023-10-30T14:00:58.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb317fff640 1 -- 172.21.15.46:0/1818609373 learned_addr learned my addr 172.21.15.46:0/1818609373 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:00:58.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb3177fe640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3180cf160 0x7fb3181e5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb31cb92640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3181538a0 0x7fb3180b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb317fff640 1 -- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3181538a0 msgr2=0x7fb3180b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb317fff640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3181538a0 0x7fb3180b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb317fff640 1 -- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3180cf160 msgr2=0x7fb3181e5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:58.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb317fff640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3180cf160 0x7fb3181e5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb317fff640 1 -- 172.21.15.46:0/1818609373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3080094d0 con 0x7fb3180ca0e0 2023-10-30T14:00:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb31cb92640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3181538a0 0x7fb3180b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:00:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb317fff640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3180ca0e0 0x7fb318157fd0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fb3080b3040 tx=0x7fb30801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb314ff9640 1 -- 172.21.15.46:0/1818609373 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3080be070 con 0x7fb3180ca0e0 2023-10-30T14:00:58.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb314ff9640 1 -- 172.21.15.46:0/1818609373 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3080ba400 con 0x7fb3180ca0e0 2023-10-30T14:00:58.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb31e5f5640 1 -- 172.21.15.46:0/1818609373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3180b4e20 con 0x7fb3180ca0e0 2023-10-30T14:00:58.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb314ff9640 1 -- 172.21.15.46:0/1818609373 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3080ba7d0 con 0x7fb3180ca0e0 2023-10-30T14:00:58.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.494+0000 7fb31e5f5640 1 -- 172.21.15.46:0/1818609373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3181592a0 con 0x7fb3180ca0e0 2023-10-30T14:00:58.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.498+0000 7fb31e5f5640 1 -- 172.21.15.46:0/1818609373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb318158510 con 0x7fb3180ca0e0 2023-10-30T14:00:58.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.498+0000 7fb314ff9640 1 -- 172.21.15.46:0/1818609373 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb3080c3410 con 0x7fb3180ca0e0 2023-10-30T14:00:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.498+0000 7fb314ff9640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3040664d0 0x7fb304068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:00:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.498+0000 7fb314ff9640 1 -- 172.21.15.46:0/1818609373 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fb3081220a0 con 0x7fb3180ca0e0 2023-10-30T14:00:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.498+0000 7fb3177fe640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3040664d0 0x7fb304068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:00:58.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.498+0000 7fb3177fe640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3040664d0 0x7fb304068990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb318071590 tx=0x7fb30c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:00:58.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.502+0000 7fb314ff9640 1 -- 172.21.15.46:0/1818609373 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb3080eee40 con 0x7fb3180ca0e0 2023-10-30T14:00:58.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb31e5f5640 1 -- 172.21.15.46:0/1818609373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3180b2a50 con 0x7fb3180ca0e0 2023-10-30T14:00:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb314ff9640 1 -- 172.21.15.46:0/1818609373 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+5739 (secure 0 0 0) 0x7fb3080f2460 con 0x7fb3180ca0e0 2023-10-30T14:00:58.701 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:00:58.701 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:55.609732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:clientreplay","state_seq":43,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:00:58.702 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 163 2023-10-30T14:00:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb31e5f5640 1 -- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3040664d0 msgr2=0x7fb304068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb31e5f5640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3040664d0 0x7fb304068990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb318071590 tx=0x7fb30c008040 comp rx=0 tx=0).stop 2023-10-30T14:00:58.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb31e5f5640 1 -- 172.21.15.46:0/1818609373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3180ca0e0 msgr2=0x7fb318157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:00:58.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb31e5f5640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3180ca0e0 0x7fb318157fd0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fb3080b3040 tx=0x7fb30801bad0 comp rx=0 tx=0).stop 2023-10-30T14:00:58.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb31e5f5640 1 -- 172.21.15.46:0/1818609373 shutdown_connections 2023-10-30T14:00:58.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb31e5f5640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3040664d0 0x7fb304068990 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:58.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb31e5f5640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3181538a0 0x7fb3180b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:58.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb31e5f5640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3180cf160 0x7fb3181e5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:58.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb31e5f5640 1 --2- 172.21.15.46:0/1818609373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3180ca0e0 0x7fb318157fd0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:00:58.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb31e5f5640 1 -- 172.21.15.46:0/1818609373 >> 172.21.15.46:0/1818609373 conn(0x7fb31806bfe0 msgr2=0x7fb318151d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:00:58.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.698+0000 7fb31e5f5640 1 -- 172.21.15.46:0/1818609373 shutdown_connections 2023-10-30T14:00:58.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:00:58.702+0000 7fb31e5f5640 1 -- 172.21.15.46:0/1818609373 wait complete. 2023-10-30T14:00:58.722 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:01:00.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:00.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.910+0000 7fb59eb0f640 1 Processor -- start 2023-10-30T14:01:00.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.910+0000 7fb59eb0f640 1 -- start start 2023-10-30T14:01:00.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb59eb0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb59806f7a0 0x7fb59806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb59eb0f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb5980c6bd0 0x7fb598069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb59eb0f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59806a3d0 con 0x7fb598071110 2023-10-30T14:01:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb59eb0f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5980700c0 con 0x7fb5980c6bd0 2023-10-30T14:01:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb59eb0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb598066cf0 con 0x7fb59806f7a0 2023-10-30T14:01:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb59d0ac640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb598071110 0x7fb598158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59754/0 (socket says 172.21.15.46:59754) 2023-10-30T14:01:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb59d0ac640 1 -- 172.21.15.46:0/1325236035 learned_addr learned my addr 172.21.15.46:0/1325236035 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:01:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb59c8ab640 1 --2- 172.21.15.46:0/1325236035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb59806f7a0 0x7fb59806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb597fff640 1 --2- 172.21.15.46:0/1325236035 >> v2:172.21.15.154:3300/0 conn(0x7fb5980c6bd0 0x7fb598069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:00.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb5977fe640 1 -- 172.21.15.46:0/1325236035 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049220634 0 0) 0x7fb59806a3d0 con 0x7fb598071110 2023-10-30T14:01:00.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb5977fe640 1 -- 172.21.15.46:0/1325236035 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb580003530 con 0x7fb598071110 2023-10-30T14:01:00.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb5977fe640 1 -- 172.21.15.46:0/1325236035 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3958288300 0 0) 0x7fb580003530 con 0x7fb598071110 2023-10-30T14:01:00.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb597fff640 1 -- 172.21.15.46:0/1325236035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb59806f7a0 msgr2=0x7fb59806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:00.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb597fff640 1 --2- 172.21.15.46:0/1325236035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb59806f7a0 0x7fb59806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:00.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb597fff640 1 -- 172.21.15.46:0/1325236035 >> v1:172.21.15.46:6789/0 conn(0x7fb598071110 legacy=0x7fb598158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:00.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb597fff640 1 -- 172.21.15.46:0/1325236035 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5980c70d0 con 0x7fb5980c6bd0 2023-10-30T14:01:00.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.914+0000 7fb59c8ab640 1 --2- 172.21.15.46:0/1325236035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb59806f7a0 0x7fb59806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:01:00.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.918+0000 7fb597fff640 1 --2- 172.21.15.46:0/1325236035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5980c6bd0 0x7fb598069e90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb5880097c0 tx=0x7fb58801eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbcdba968ff5c0d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:00.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.922+0000 7fb5977fe640 1 -- 172.21.15.46:0/1325236035 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb590001a00 con 0x7fb5980c6bd0 2023-10-30T14:01:00.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.922+0000 7fb5977fe640 1 -- 172.21.15.46:0/1325236035 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5880a85d0 con 0x7fb5980c6bd0 2023-10-30T14:01:00.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.922+0000 7fb5977fe640 1 -- 172.21.15.46:0/1325236035 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5880a8960 con 0x7fb5980c6bd0 2023-10-30T14:01:00.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.922+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1325236035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5980c6bd0 msgr2=0x7fb598069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:00.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.922+0000 7fb59eb0f640 1 --2- 172.21.15.46:0/1325236035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5980c6bd0 0x7fb598069e90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb5880097c0 tx=0x7fb58801eb90 comp rx=0 tx=0).stop 2023-10-30T14:01:00.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.922+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1325236035 shutdown_connections 2023-10-30T14:01:00.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.922+0000 7fb59eb0f640 1 --2- 172.21.15.46:0/1325236035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5980c6bd0 0x7fb598069e90 secure :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb5880097c0 tx=0x7fb58801eb90 comp rx=0 tx=0).stop 2023-10-30T14:01:00.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.922+0000 7fb59eb0f640 1 --2- 172.21.15.46:0/1325236035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb59806f7a0 0x7fb59806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:00.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.922+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1325236035 >> 172.21.15.46:0/1325236035 conn(0x7fb5980c7910 msgr2=0x7fb5980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:00.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.922+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1325236035 shutdown_connections 2023-10-30T14:01:00.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.922+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1325236035 wait complete. 2023-10-30T14:01:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb59eb0f640 1 Processor -- start 2023-10-30T14:01:00.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb59eb0f640 1 -- start start 2023-10-30T14:01:00.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb59eb0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb59806f7a0 0x7fb5980c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb59eb0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb598071110 0x7fb5980c1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb59eb0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5981efbd0 0x7fb5980c1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb59eb0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5980c1e60 con 0x7fb598071110 2023-10-30T14:01:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb59eb0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59814dba0 con 0x7fb5981efbd0 2023-10-30T14:01:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb59eb0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5981f00a0 con 0x7fb59806f7a0 2023-10-30T14:01:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb59d0ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5981efbd0 0x7fb5980c1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb59c8ab640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb59806f7a0 0x7fb5980c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:00.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb59c8ab640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb59806f7a0 0x7fb5980c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:45582/0 (socket says 172.21.15.46:45582) 2023-10-30T14:01:00.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb59c8ab640 1 -- 172.21.15.46:0/1502020269 learned_addr learned my addr 172.21.15.46:0/1502020269 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:00.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.926+0000 7fb597fff640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb598071110 0x7fb5980c1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:00.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.930+0000 7fb59d0ac640 1 -- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb59806f7a0 msgr2=0x7fb5980c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:00.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.930+0000 7fb59d0ac640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb59806f7a0 0x7fb5980c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:00.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.930+0000 7fb59d0ac640 1 -- 172.21.15.46:0/1502020269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb598071110 msgr2=0x7fb5980c1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:00.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.930+0000 7fb59d0ac640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb598071110 0x7fb5980c1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:00.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.930+0000 7fb59d0ac640 1 -- 172.21.15.46:0/1502020269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58c009420 con 0x7fb5981efbd0 2023-10-30T14:01:00.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.930+0000 7fb59d0ac640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5981efbd0 0x7fb5980c1640 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb590001710 tx=0x7fb59000a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:00.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.930+0000 7fb5957fa640 1 -- 172.21.15.46:0/1502020269 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59000ad40 con 0x7fb5981efbd0 2023-10-30T14:01:00.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.930+0000 7fb5957fa640 1 -- 172.21.15.46:0/1502020269 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb590012400 con 0x7fb5981efbd0 2023-10-30T14:01:00.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.930+0000 7fb5957fa640 1 -- 172.21.15.46:0/1502020269 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59000aea0 con 0x7fb5981efbd0 2023-10-30T14:01:00.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.930+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1502020269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb588009420 con 0x7fb5981efbd0 2023-10-30T14:01:00.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.930+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1502020269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59814d8c0 con 0x7fb5981efbd0 2023-10-30T14:01:00.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.930+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1502020269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59804e170 con 0x7fb5981efbd0 2023-10-30T14:01:00.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.934+0000 7fb5957fa640 1 -- 172.21.15.46:0/1502020269 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb5900127b0 con 0x7fb5981efbd0 2023-10-30T14:01:00.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.934+0000 7fb5957fa640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5840664a0 0x7fb584068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:00.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.934+0000 7fb5957fa640 1 -- 172.21.15.46:0/1502020269 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fb5900856d0 con 0x7fb5981efbd0 2023-10-30T14:01:00.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.934+0000 7fb59c8ab640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5840664a0 0x7fb584068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:00.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.934+0000 7fb59c8ab640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5840664a0 0x7fb584068960 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb58c009d50 tx=0x7fb58c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:00.946+0000 7fb5957fa640 1 -- 172.21.15.46:0/1502020269 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb590055e70 con 0x7fb5981efbd0 2023-10-30T14:01:01.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.198+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1502020269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5980b1210 con 0x7fb5981efbd0 2023-10-30T14:01:01.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.198+0000 7fb5957fa640 1 -- 172.21.15.46:0/1502020269 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+5739 (secure 0 0 0) 0x7fb590055a00 con 0x7fb5981efbd0 2023-10-30T14:01:01.202 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:01.202 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:00:55.609732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:clientreplay","state_seq":43,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 163 2023-10-30T14:01:01.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.202+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5840664a0 msgr2=0x7fb584068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:01.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.202+0000 7fb59eb0f640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5840664a0 0x7fb584068960 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb58c009d50 tx=0x7fb58c009190 comp rx=0 tx=0).stop 2023-10-30T14:01:01.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.202+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5981efbd0 msgr2=0x7fb5980c1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:01.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.202+0000 7fb59eb0f640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5981efbd0 0x7fb5980c1640 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb590001710 tx=0x7fb59000a430 comp rx=0 tx=0).stop 2023-10-30T14:01:01.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.202+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1502020269 shutdown_connections 2023-10-30T14:01:01.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.202+0000 7fb59eb0f640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb5840664a0 0x7fb584068960 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:01.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.202+0000 7fb59eb0f640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5981efbd0 0x7fb5980c1640 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.202+0000 7fb59eb0f640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb598071110 0x7fb5980c1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.202+0000 7fb59eb0f640 1 --2- 172.21.15.46:0/1502020269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb59806f7a0 0x7fb5980c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.202+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1502020269 >> 172.21.15.46:0/1502020269 conn(0x7fb5980c7910 msgr2=0x7fb5980c3770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.202+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1502020269 shutdown_connections 2023-10-30T14:01:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:01.202+0000 7fb59eb0f640 1 -- 172.21.15.46:0/1502020269 wait complete. 2023-10-30T14:01:01.223 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:01:03.224 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.374+0000 7fadf746b640 1 Processor -- start 2023-10-30T14:01:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.374+0000 7fadf746b640 1 -- start start 2023-10-30T14:01:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf746b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fadf0151800 0x7fadf006e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf746b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fadf00c1ee0 0x7fadf00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf746b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf00c9230 0x7fadf00cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf746b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf006dc70 con 0x7fadf0151800 2023-10-30T14:01:03.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf746b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf006e090 con 0x7fadf00c1ee0 2023-10-30T14:01:03.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf746b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf00c8a50 con 0x7fadf00c9230 2023-10-30T14:01:03.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf5207640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fadf0151800 0x7fadf006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:03.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf5207640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fadf0151800 0x7fadf006e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39032/0 (socket says 172.21.15.46:39032) 2023-10-30T14:01:03.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf5207640 1 -- 172.21.15.46:0/3359267428 learned_addr learned my addr 172.21.15.46:0/3359267428 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:03.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf5a08640 1 --2- 172.21.15.46:0/3359267428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf00c9230 0x7fadf00cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:03.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf4a06640 1 --2- 172.21.15.46:0/3359267428 >> v2:172.21.15.154:3300/0 conn(0x7fadf00c1ee0 0x7fadf00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:03.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf5207640 1 -- 172.21.15.46:0/3359267428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf00c9230 msgr2=0x7fadf00cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:03.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf5207640 1 --2- 172.21.15.46:0/3359267428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf00c9230 0x7fadf00cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:03.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf5207640 1 -- 172.21.15.46:0/3359267428 >> v2:172.21.15.154:3300/0 conn(0x7fadf00c1ee0 msgr2=0x7fadf00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:03.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf5207640 1 --2- 172.21.15.46:0/3359267428 >> v2:172.21.15.154:3300/0 conn(0x7fadf00c1ee0 0x7fadf00c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:03.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf5207640 1 -- 172.21.15.46:0/3359267428 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf00c3b10 con 0x7fadf0151800 2023-10-30T14:01:03.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.378+0000 7fadf4a06640 1 --2- 172.21.15.46:0/3359267428 >> v2:172.21.15.154:3300/0 conn(0x7fadf00c1ee0 0x7fadf00c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:01:03.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fadf5207640 1 --2- 172.21.15.46:0/3359267428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf0151800 0x7fadf006e470 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fadf006e9b0 tx=0x7fadec01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d9cdb50739bc5e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:03.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fade7fff640 1 -- 172.21.15.46:0/3359267428 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec0aa5f0 con 0x7fadf0151800 2023-10-30T14:01:03.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fade7fff640 1 -- 172.21.15.46:0/3359267428 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec0aa750 con 0x7fadf0151800 2023-10-30T14:01:03.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fade7fff640 1 -- 172.21.15.46:0/3359267428 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec0aaae0 con 0x7fadf0151800 2023-10-30T14:01:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fadf746b640 1 -- 172.21.15.46:0/3359267428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf0151800 msgr2=0x7fadf006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fadf746b640 1 --2- 172.21.15.46:0/3359267428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf0151800 0x7fadf006e470 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fadf006e9b0 tx=0x7fadec01c4d0 comp rx=0 tx=0).stop 2023-10-30T14:01:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fadf746b640 1 -- 172.21.15.46:0/3359267428 shutdown_connections 2023-10-30T14:01:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fadf746b640 1 --2- 172.21.15.46:0/3359267428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf00c9230 0x7fadf00cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:03.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fadf746b640 1 --2- 172.21.15.46:0/3359267428 >> v2:172.21.15.154:3300/0 conn(0x7fadf00c1ee0 0x7fadf00c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fadf746b640 1 --2- 172.21.15.46:0/3359267428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf0151800 0x7fadf006e470 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fadf746b640 1 -- 172.21.15.46:0/3359267428 >> 172.21.15.46:0/3359267428 conn(0x7fadf0071ea0 msgr2=0x7fadf0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fadf746b640 1 -- 172.21.15.46:0/3359267428 shutdown_connections 2023-10-30T14:01:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fadf746b640 1 -- 172.21.15.46:0/3359267428 wait complete. 2023-10-30T14:01:03.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fadf746b640 1 Processor -- start 2023-10-30T14:01:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.382+0000 7fadf746b640 1 -- start start 2023-10-30T14:01:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf746b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf00c1ee0 0x7fadf0152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf746b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf00c9230 0x7fadf00cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:03.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf746b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadf0151800 0x7fadf00cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf746b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf0153fa0 con 0x7fadf00c1ee0 2023-10-30T14:01:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf746b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf01f69b0 con 0x7fadf0151800 2023-10-30T14:01:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf746b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf014a590 con 0x7fadf00c9230 2023-10-30T14:01:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf5207640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf00c1ee0 0x7fadf0152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf5207640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf00c1ee0 0x7fadf0152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39040/0 (socket says 172.21.15.46:39040) 2023-10-30T14:01:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf5207640 1 -- 172.21.15.46:0/2323803176 learned_addr learned my addr 172.21.15.46:0/2323803176 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf4a06640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf00c9230 0x7fadf00cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf5a08640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadf0151800 0x7fadf00cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf5207640 1 -- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf00c9230 msgr2=0x7fadf00cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:03.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf5207640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf00c9230 0x7fadf00cc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf5207640 1 -- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadf0151800 msgr2=0x7fadf00cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf5207640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadf0151800 0x7fadf00cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf5207640 1 -- 172.21.15.46:0/2323803176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec0094d0 con 0x7fadf00c1ee0 2023-10-30T14:01:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf4a06640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf00c9230 0x7fadf00cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:01:03.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf5a08640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadf0151800 0x7fadf00cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:01:03.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf5207640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf00c1ee0 0x7fadf0152960 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fadec01ab80 tx=0x7fadec01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:03.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fade5ffb640 1 -- 172.21.15.46:0/2323803176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec0b3800 con 0x7fadf00c1ee0 2023-10-30T14:01:03.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fade5ffb640 1 -- 172.21.15.46:0/2323803176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec0b3980 con 0x7fadf00c1ee0 2023-10-30T14:01:03.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fade5ffb640 1 -- 172.21.15.46:0/2323803176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec0b3d30 con 0x7fadf00c1ee0 2023-10-30T14:01:03.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf746b640 1 -- 172.21.15.46:0/2323803176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf01f5e30 con 0x7fadf00c1ee0 2023-10-30T14:01:03.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf746b640 1 -- 172.21.15.46:0/2323803176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf01f63f0 con 0x7fadf00c1ee0 2023-10-30T14:01:03.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fadf746b640 1 -- 172.21.15.46:0/2323803176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf004e170 con 0x7fadf00c1ee0 2023-10-30T14:01:03.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.386+0000 7fade5ffb640 1 -- 172.21.15.46:0/2323803176 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fadec0b2380 con 0x7fadf00c1ee0 2023-10-30T14:01:03.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.390+0000 7fade5ffb640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fadd40664a0 0x7fadd4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:03.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.390+0000 7fade5ffb640 1 -- 172.21.15.46:0/2323803176 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fadec0a7070 con 0x7fadf00c1ee0 2023-10-30T14:01:03.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.390+0000 7fadf4a06640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fadd40664a0 0x7fadd4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:03.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.390+0000 7fadf4a06640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fadd40664a0 0x7fadd4068960 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fadf006eea0 tx=0x7fade0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:03.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.394+0000 7fade5ffb640 1 -- 172.21.15.46:0/2323803176 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fadec0eb8f0 con 0x7fadf00c1ee0 2023-10-30T14:01:03.411 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T14:01:03.410+0000 7fdd5c91c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:01:03.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 -- 172.21.15.46:0/2323803176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadf00c6de0 con 0x7fadf00c1ee0 2023-10-30T14:01:03.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fade5ffb640 1 -- 172.21.15.46:0/2323803176 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fadec0eef10 con 0x7fadf00c1ee0 2023-10-30T14:01:03.589 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:03.589 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:01.822914+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:03.590 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 164 2023-10-30T14:01:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 -- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fadd40664a0 msgr2=0x7fadd4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fadd40664a0 0x7fadd4068960 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fadf006eea0 tx=0x7fade0008040 comp rx=0 tx=0).stop 2023-10-30T14:01:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 -- 172.21.15.46:0/2323803176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf00c1ee0 msgr2=0x7fadf0152960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf00c1ee0 0x7fadf0152960 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fadec01ab80 tx=0x7fadec01ad10 comp rx=0 tx=0).stop 2023-10-30T14:01:03.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 -- 172.21.15.46:0/2323803176 shutdown_connections 2023-10-30T14:01:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fadd40664a0 0x7fadd4068960 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadf0151800 0x7fadf00cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf00c9230 0x7fadf00cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 --2- 172.21.15.46:0/2323803176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf00c1ee0 0x7fadf0152960 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 -- 172.21.15.46:0/2323803176 >> 172.21.15.46:0/2323803176 conn(0x7fadf0071ea0 msgr2=0x7fadf00c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 -- 172.21.15.46:0/2323803176 shutdown_connections 2023-10-30T14:01:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:03.586+0000 7fadf746b640 1 -- 172.21.15.46:0/2323803176 wait complete. 2023-10-30T14:01:03.610 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=1, replacing gid=7518 2023-10-30T14:01:03.610 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2023-10-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 70 -eq 48 ] 2023-10-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.70 2023-10-30T14:01:07.614 INFO:tasks.workunit.client.0.smithi046.stdout:v0.70 2023-10-30T14:01:07.614 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.70 2023-10-30T14:01:07.614 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.70 2023-10-30T14:01:11.611 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T14:01:11.611 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T14:01:11.611 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T14:01:11.615 INFO:tasks.ceph.mds.a:Started 2023-10-30T14:01:11.616 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:11.712 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-10-30T14:01:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.766+0000 7f90804ea640 1 Processor -- start 2023-10-30T14:01:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.766+0000 7f90804ea640 1 -- start start 2023-10-30T14:01:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.766+0000 7f90804ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9078069e90 0x7f907815a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.766+0000 7f90804ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f907806a270 0x7f90780c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.766+0000 7f90804ea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90780c6c10 con 0x7f907806cb80 2023-10-30T14:01:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.766+0000 7f90804ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90780c8570 con 0x7f907806a270 2023-10-30T14:01:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.766+0000 7f90804ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90780c6ef0 con 0x7f9078069e90 2023-10-30T14:01:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.766+0000 7f907e286640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9078069e90 0x7f907815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:11.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.766+0000 7f907da85640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f907806a270 0x7f90780c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.770+0000 7f907e286640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9078069e90 0x7f907815a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:36646/0 (socket says 172.21.15.46:36646) 2023-10-30T14:01:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.770+0000 7f907e286640 1 -- 172.21.15.46:0/1440245460 learned_addr learned my addr 172.21.15.46:0/1440245460 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.770+0000 7f907ea87640 1 --1- 172.21.15.46:0/1440245460 >> v1:172.21.15.154:6789/0 conn(0x7f907806cb80 0x7f907806d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:59544/0 (socket says 172.21.15.46:59544) 2023-10-30T14:01:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.770+0000 7f907e286640 1 -- 172.21.15.46:0/1440245460 >> v2:172.21.15.154:3300/0 conn(0x7f907806a270 msgr2=0x7f90780c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.770+0000 7f907e286640 1 --2- 172.21.15.46:0/1440245460 >> v2:172.21.15.154:3300/0 conn(0x7f907806a270 0x7f90780c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.770+0000 7f907e286640 1 -- 172.21.15.46:0/1440245460 >> v1:172.21.15.154:6789/0 conn(0x7f907806cb80 legacy=0x7f907806d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:11.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.770+0000 7f907e286640 1 -- 172.21.15.46:0/1440245460 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90780c71d0 con 0x7f9078069e90 2023-10-30T14:01:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f907e286640 1 --2- 172.21.15.46:0/1440245460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9078069e90 0x7f907815a410 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9074009870 tx=0x7f907401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81ab29049179795a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:11.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f907d284640 1 -- 172.21.15.46:0/1440245460 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90740a7070 con 0x7f9078069e90 2023-10-30T14:01:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f907d284640 1 -- 172.21.15.46:0/1440245460 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90740ab430 con 0x7f9078069e90 2023-10-30T14:01:11.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f907d284640 1 -- 172.21.15.46:0/1440245460 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90740ab7c0 con 0x7f9078069e90 2023-10-30T14:01:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 -- 172.21.15.46:0/1440245460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9078069e90 msgr2=0x7f907815a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 --2- 172.21.15.46:0/1440245460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9078069e90 0x7f907815a410 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9074009870 tx=0x7f907401c540 comp rx=0 tx=0).stop 2023-10-30T14:01:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 -- 172.21.15.46:0/1440245460 shutdown_connections 2023-10-30T14:01:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 --2- 172.21.15.46:0/1440245460 >> v2:172.21.15.154:3300/0 conn(0x7f907806a270 0x7f90780c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 --2- 172.21.15.46:0/1440245460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9078069e90 0x7f907815a410 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 -- 172.21.15.46:0/1440245460 >> 172.21.15.46:0/1440245460 conn(0x7f907806dfd0 msgr2=0x7f907806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 -- 172.21.15.46:0/1440245460 shutdown_connections 2023-10-30T14:01:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 -- 172.21.15.46:0/1440245460 wait complete. 2023-10-30T14:01:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 Processor -- start 2023-10-30T14:01:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 -- start start 2023-10-30T14:01:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9078069e90 0x7f907814fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f907806a270 0x7f90781f5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f907806cb80 0x7f90781f5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9078151a20 con 0x7f907806cb80 2023-10-30T14:01:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907815da70 con 0x7f907806a270 2023-10-30T14:01:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.774+0000 7f90804ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9078151360 con 0x7f9078069e90 2023-10-30T14:01:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f907ea87640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f907806cb80 0x7f90781f5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f907ea87640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f907806cb80 0x7f90781f5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45666/0 (socket says 172.21.15.46:45666) 2023-10-30T14:01:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f907ea87640 1 -- 172.21.15.46:0/989726997 learned_addr learned my addr 172.21.15.46:0/989726997 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f907da85640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f907806a270 0x7f90781f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f907e286640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9078069e90 0x7f907814fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f907ea87640 1 -- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9078069e90 msgr2=0x7f907814fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f907ea87640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9078069e90 0x7f907814fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f907ea87640 1 -- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f907806a270 msgr2=0x7f90781f5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f907ea87640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f907806a270 0x7f90781f5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f907ea87640 1 -- 172.21.15.46:0/989726997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90740094d0 con 0x7f907806cb80 2023-10-30T14:01:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f907ea87640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f907806cb80 0x7f90781f5770 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f907000c730 tx=0x7f907000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f906effd640 1 -- 172.21.15.46:0/989726997 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907000ce40 con 0x7f907806cb80 2023-10-30T14:01:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f906effd640 1 -- 172.21.15.46:0/989726997 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907000a640 con 0x7f907806cb80 2023-10-30T14:01:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f906effd640 1 -- 172.21.15.46:0/989726997 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907000a9b0 con 0x7f907806cb80 2023-10-30T14:01:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f90804ea640 1 -- 172.21.15.46:0/989726997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907815d0d0 con 0x7f907806cb80 2023-10-30T14:01:11.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f90804ea640 1 -- 172.21.15.46:0/989726997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907815d690 con 0x7f907806cb80 2023-10-30T14:01:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f90804ea640 1 -- 172.21.15.46:0/989726997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907804e170 con 0x7f907806cb80 2023-10-30T14:01:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.778+0000 7f906effd640 1 -- 172.21.15.46:0/989726997 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f907000ab10 con 0x7f907806cb80 2023-10-30T14:01:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.782+0000 7f906effd640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f905c0664a0 0x7f905c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:11.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.782+0000 7f906effd640 1 -- 172.21.15.46:0/989726997 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f9070083cc0 con 0x7f907806cb80 2023-10-30T14:01:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.782+0000 7f907e286640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f905c0664a0 0x7f905c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.782+0000 7f907e286640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f905c0664a0 0x7f905c068960 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f90740af000 tx=0x7f9074008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.786+0000 7f906effd640 1 -- 172.21.15.46:0/989726997 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f907008a080 con 0x7f907806cb80 2023-10-30T14:01:11.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.978+0000 7f90804ea640 1 -- 172.21.15.46:0/989726997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f907806ef60 con 0x7f907806cb80 2023-10-30T14:01:11.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.978+0000 7f906effd640 1 -- 172.21.15.46:0/989726997 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f9070017090 con 0x7f907806cb80 2023-10-30T14:01:11.981 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:11.982 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:01.822914+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:11.982 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 164 2023-10-30T14:01:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.978+0000 7f90804ea640 1 -- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f905c0664a0 msgr2=0x7f905c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.978+0000 7f90804ea640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f905c0664a0 0x7f905c068960 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f90740af000 tx=0x7f9074008040 comp rx=0 tx=0).stop 2023-10-30T14:01:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.978+0000 7f90804ea640 1 -- 172.21.15.46:0/989726997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f907806cb80 msgr2=0x7f90781f5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.978+0000 7f90804ea640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f907806cb80 0x7f90781f5770 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f907000c730 tx=0x7f907000cbf0 comp rx=0 tx=0).stop 2023-10-30T14:01:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.982+0000 7f90804ea640 1 -- 172.21.15.46:0/989726997 shutdown_connections 2023-10-30T14:01:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.982+0000 7f90804ea640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f905c0664a0 0x7f905c068960 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.982+0000 7f90804ea640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f907806cb80 0x7f90781f5770 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.982+0000 7f90804ea640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f907806a270 0x7f90781f5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.982+0000 7f90804ea640 1 --2- 172.21.15.46:0/989726997 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9078069e90 0x7f907814fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.982+0000 7f90804ea640 1 -- 172.21.15.46:0/989726997 >> 172.21.15.46:0/989726997 conn(0x7f907806dfd0 msgr2=0x7f90780715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.982+0000 7f90804ea640 1 -- 172.21.15.46:0/989726997 shutdown_connections 2023-10-30T14:01:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:11.982+0000 7f90804ea640 1 -- 172.21.15.46:0/989726997 wait complete. 2023-10-30T14:01:12.002 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T14:01:14.003 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:14.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.142+0000 7f35bf70f640 1 Processor -- start 2023-10-30T14:01:14.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.142+0000 7f35bf70f640 1 -- start start 2023-10-30T14:01:14.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bf70f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f35b806f7a0 0x7f35b806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:14.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bf70f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35b80c6bd0 0x7f35b8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:14.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bf70f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b806a3d0 con 0x7f35b8071110 2023-10-30T14:01:14.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bf70f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b80700c0 con 0x7f35b806f7a0 2023-10-30T14:01:14.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bf70f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b8066cf0 con 0x7f35b80c6bd0 2023-10-30T14:01:14.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bd4ab640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f35b806f7a0 0x7f35b806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:14.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bd4ab640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f35b806f7a0 0x7f35b806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45674/0 (socket says 172.21.15.46:45674) 2023-10-30T14:01:14.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bd4ab640 1 -- 172.21.15.46:0/3990390019 learned_addr learned my addr 172.21.15.46:0/3990390019 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:14.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bccaa640 1 --2- 172.21.15.46:0/3990390019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35b80c6bd0 0x7f35b8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:14.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bd4ab640 1 -- 172.21.15.46:0/3990390019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35b80c6bd0 msgr2=0x7f35b8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:14.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bd4ab640 1 --2- 172.21.15.46:0/3990390019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35b80c6bd0 0x7f35b8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:14.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bd4ab640 1 -- 172.21.15.46:0/3990390019 >> v1:172.21.15.154:6789/0 conn(0x7f35b8071110 legacy=0x7f35b8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:14.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bd4ab640 1 -- 172.21.15.46:0/3990390019 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80c70d0 con 0x7f35b806f7a0 2023-10-30T14:01:14.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.146+0000 7f35bd4ab640 1 --2- 172.21.15.46:0/3990390019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b806f7a0 0x7f35b806fb80 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009870 tx=0x7f35b401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da220a66793f9dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:14.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35affff640 1 -- 172.21.15.46:0/3990390019 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b40a8a30 con 0x7f35b806f7a0 2023-10-30T14:01:14.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35affff640 1 -- 172.21.15.46:0/3990390019 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b40a8b90 con 0x7f35b806f7a0 2023-10-30T14:01:14.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35affff640 1 -- 172.21.15.46:0/3990390019 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b40b0440 con 0x7f35b806f7a0 2023-10-30T14:01:14.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 -- 172.21.15.46:0/3990390019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b806f7a0 msgr2=0x7f35b806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:14.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 --2- 172.21.15.46:0/3990390019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b806f7a0 0x7f35b806fb80 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009870 tx=0x7f35b401ed30 comp rx=0 tx=0).stop 2023-10-30T14:01:14.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 -- 172.21.15.46:0/3990390019 shutdown_connections 2023-10-30T14:01:14.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 --2- 172.21.15.46:0/3990390019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35b80c6bd0 0x7f35b8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:14.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 --2- 172.21.15.46:0/3990390019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b806f7a0 0x7f35b806fb80 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:14.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 -- 172.21.15.46:0/3990390019 >> 172.21.15.46:0/3990390019 conn(0x7f35b80c7910 msgr2=0x7f35b80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:14.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 -- 172.21.15.46:0/3990390019 shutdown_connections 2023-10-30T14:01:14.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 -- 172.21.15.46:0/3990390019 wait complete. 2023-10-30T14:01:14.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 Processor -- start 2023-10-30T14:01:14.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 -- start start 2023-10-30T14:01:14.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35b806f7a0 0x7f35b80c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:14.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b8071110 0x7f35b80c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:14.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b80c6bd0 0x7f35b80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:14.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b80af6b0 con 0x7f35b80c6bd0 2023-10-30T14:01:14.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b80afad0 con 0x7f35b8071110 2023-10-30T14:01:14.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.150+0000 7f35bf70f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b814b400 con 0x7f35b806f7a0 2023-10-30T14:01:14.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bdcac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b80c6bd0 0x7f35b80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:14.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bdcac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b80c6bd0 0x7f35b80bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45690/0 (socket says 172.21.15.46:45690) 2023-10-30T14:01:14.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bccaa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b8071110 0x7f35b80c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:14.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bdcac640 1 -- 172.21.15.46:0/3031957539 learned_addr learned my addr 172.21.15.46:0/3031957539 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:14.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bd4ab640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35b806f7a0 0x7f35b80c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:14.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bdcac640 1 -- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35b806f7a0 msgr2=0x7f35b80c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:14.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bdcac640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35b806f7a0 0x7f35b80c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:14.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bdcac640 1 -- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b8071110 msgr2=0x7f35b80c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:14.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bdcac640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b8071110 0x7f35b80c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:14.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bdcac640 1 -- 172.21.15.46:0/3031957539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b40094d0 con 0x7f35b80c6bd0 2023-10-30T14:01:14.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bdcac640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b80c6bd0 0x7f35b80bfec0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f35b000c760 tx=0x7f35b000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:14.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35adffb640 1 -- 172.21.15.46:0/3031957539 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0003b80 con 0x7f35b80c6bd0 2023-10-30T14:01:14.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35adffb640 1 -- 172.21.15.46:0/3031957539 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b0003ce0 con 0x7f35b80c6bd0 2023-10-30T14:01:14.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35adffb640 1 -- 172.21.15.46:0/3031957539 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b000aa30 con 0x7f35b80c6bd0 2023-10-30T14:01:14.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bf70f640 1 -- 172.21.15.46:0/3031957539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b8155090 con 0x7f35b80c6bd0 2023-10-30T14:01:14.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bf70f640 1 -- 172.21.15.46:0/3031957539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b80aed10 con 0x7f35b80c6bd0 2023-10-30T14:01:14.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35bf70f640 1 -- 172.21.15.46:0/3031957539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b80af7f0 con 0x7f35b80c6bd0 2023-10-30T14:01:14.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.154+0000 7f35adffb640 1 -- 172.21.15.46:0/3031957539 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f35b000ab90 con 0x7f35b80c6bd0 2023-10-30T14:01:14.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.158+0000 7f35adffb640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f359c0664a0 0x7f359c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:14.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.158+0000 7f35adffb640 1 -- 172.21.15.46:0/3031957539 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f35b00841b0 con 0x7f35b80c6bd0 2023-10-30T14:01:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.158+0000 7f35bd4ab640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f359c0664a0 0x7f359c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.158+0000 7f35bd4ab640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f359c0664a0 0x7f359c068960 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f35b40b2000 tx=0x7f35b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.158+0000 7f35adffb640 1 -- 172.21.15.46:0/3031957539 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f35b0051000 con 0x7f35b80c6bd0 2023-10-30T14:01:14.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.354+0000 7f35bf70f640 1 -- 172.21.15.46:0/3031957539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35b804e170 con 0x7f35b80c6bd0 2023-10-30T14:01:14.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.354+0000 7f35adffb640 1 -- 172.21.15.46:0/3031957539 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6526 (secure 0 0 0) 0x7f35b0054620 con 0x7f35b80c6bd0 2023-10-30T14:01:14.358 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:14.358 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8034,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:12.419821+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 165 2023-10-30T14:01:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.358+0000 7f35bf70f640 1 -- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f359c0664a0 msgr2=0x7f359c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.358+0000 7f35bf70f640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f359c0664a0 0x7f359c068960 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f35b40b2000 tx=0x7f35b4009040 comp rx=0 tx=0).stop 2023-10-30T14:01:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.358+0000 7f35bf70f640 1 -- 172.21.15.46:0/3031957539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b80c6bd0 msgr2=0x7f35b80bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.358+0000 7f35bf70f640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b80c6bd0 0x7f35b80bfec0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f35b000c760 tx=0x7f35b000cc20 comp rx=0 tx=0).stop 2023-10-30T14:01:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.358+0000 7f35bf70f640 1 -- 172.21.15.46:0/3031957539 shutdown_connections 2023-10-30T14:01:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.358+0000 7f35bf70f640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f359c0664a0 0x7f359c068960 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.358+0000 7f35bf70f640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b80c6bd0 0x7f35b80bfec0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.358+0000 7f35bf70f640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b8071110 0x7f35b80c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.358+0000 7f35bf70f640 1 --2- 172.21.15.46:0/3031957539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35b806f7a0 0x7f35b80c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.358+0000 7f35bf70f640 1 -- 172.21.15.46:0/3031957539 >> 172.21.15.46:0/3031957539 conn(0x7f35b80c7910 msgr2=0x7f35b80c3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.358+0000 7f35bf70f640 1 -- 172.21.15.46:0/3031957539 shutdown_connections 2023-10-30T14:01:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:14.358+0000 7f35bf70f640 1 -- 172.21.15.46:0/3031957539 wait complete. 2023-10-30T14:01:14.378 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T14:01:14.378 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2023-10-30T14:01:17.310 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 52% (23/44) Updating files: 54% (24/44) Updating files: 56% (25/44) Updating files: 59% (26/44) Updating files: 61% (27/44) Updating files: 63% (28/44) Updating files: 65% (29/44) Updating files: 68% (30/44) Updating files: 70% (31/44) Updating files: 72% (32/44) Updating files: 75% (33/44) Updating files: 77% (34/44) Updating files: 79% (35/44) Updating files: 81% (36/44) Updating files: 84% (37/44) Updating files: 86% (38/44) Updating files: 88% (39/44) Updating files: 90% (40/44) Updating files: 93% (41/44) Updating files: 95% (42/44) Updating files: 97% (43/44) Updating files: 100% (44/44) Updating files: 100% (44/44), done. 2023-10-30T14:01:17.532 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at e3bb0656d92 v0.70 2023-10-30T14:01:17.534 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.70 2023-10-30T14:01:17.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 71 -eq 48 ] 2023-10-30T14:01:17.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.71 2023-10-30T14:01:17.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.71 2023-10-30T14:01:17.561 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.71 2023-10-30T14:01:17.562 INFO:tasks.workunit.client.0.smithi046.stdout:v0.71 2023-10-30T14:01:24.297 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:01:24.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:01:30.806 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 14% (59/405) Updating files: 15% (61/405) Updating files: 16% (65/405) Updating files: 17% (69/405) Updating files: 18% (73/405) Updating files: 19% (77/405) Updating files: 20% (81/405) Updating files: 21% (86/405) Updating files: 22% (90/405) Updating files: 23% (94/405) Updating files: 24% (98/405) Updating files: 25% (102/405) Updating files: 26% (106/405) Updating files: 27% (110/405) Updating files: 27% (113/405) Updating files: 28% (114/405) Updating files: 29% (118/405) Updating files: 30% (122/405) Updating files: 31% (126/405) Updating files: 32% (130/405) Updating files: 32% (132/405) Updating files: 33% (134/405) Updating files: 34% (138/405) Updating files: 35% (142/405) Updating files: 36% (146/405) Updating files: 37% (150/405) Updating files: 38% (154/405) Updating files: 39% (158/405) Updating files: 40% (162/405) Updating files: 41% (167/405) Updating files: 42% (171/405) Updating files: 42% (173/405) Updating files: 43% (175/405) Updating files: 44% (179/405) Updating files: 44% (180/405) Updating files: 45% (183/405) Updating files: 46% (187/405) Updating files: 47% (191/405) Updating files: 48% (195/405) Updating files: 49% (199/405) Updating files: 50% (203/405) Updating files: 51% (207/405) Updating files: 52% (211/405) Updating files: 52% (214/405) Updating files: 53% (215/405) Updating files: 54% (219/405) Updating files: 55% (223/405) Updating files: 56% (227/405) Updating files: 57% (231/405) Updating files: 58% (235/405) Updating files: 59% (239/405) Updating files: 60% (243/405) Updating files: 61% (248/405) Updating files: 62% (252/405) Updating files: 63% (256/405) Updating files: 64% (260/405) Updating files: 65% (264/405) Updating files: 66% (268/405) Updating files: 67% (272/405) Updating files: 68% (276/405) Updating files: 68% (279/405) Updating files: 69% (280/405) Updating files: 70% (284/405) Updating files: 71% (288/405) Updating files: 72% (292/405) Updating files: 73% (296/405) Updating files: 74% (300/405) Updating files: 75% (304/405) Updating files: 76% (308/405) Updating files: 77% (312/405) Updating files: 78% (316/405) Updating files: 79% (320/405) Updating files: 80% (324/405) Updating files: 81% (329/405) Updating files: 82% (333/405) Updating files: 83% (337/405) Updating files: 84% (341/405) Updating files: 85% (345/405) Updating files: 86% (349/405) Updating files: 87% (353/405) Updating files: 87% (356/405) Updating files: 88% (357/405) Updating files: 89% (361/405) Updating files: 90% (365/405) Updating files: 91% (369/405) Updating files: 92% (373/405) Updating files: 93% (377/405) Updating files: 94% (381/405) Updating files: 95% (385/405) Updating files: 96% (389/405) Updating files: 97% (393/405) Updating files: 98% (397/405) Updating files: 99% (401/405) Updating files: 100% (405/405) Updating files: 100% (405/405), done. 2023-10-30T14:01:30.937 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at e509cb1e69c v0.71 2023-10-30T14:01:30.947 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.71 2023-10-30T14:01:31.063 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 72 -eq 48 ] 2023-10-30T14:01:31.063 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.72 2023-10-30T14:01:31.063 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.72 2023-10-30T14:01:31.064 INFO:tasks.workunit.client.0.smithi046.stdout:v0.72 2023-10-30T14:01:31.064 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.72 2023-10-30T14:01:31.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:31.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.522+0000 7ff5dcf3e640 1 Processor -- start 2023-10-30T14:01:31.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.522+0000 7ff5dcf3e640 1 -- start start 2023-10-30T14:01:31.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5dcf3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d81538a0 0x7ff5d815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5dcf3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d80cb800 con 0x7ff5d80cf160 2023-10-30T14:01:31.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5dcf3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d815dc30 con 0x7ff5d80ca0e0 2023-10-30T14:01:31.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5dcf3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d80cb330 con 0x7ff5d81538a0 2023-10-30T14:01:31.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d6d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff5d80cf160 0x7ff5d8164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47988/0 (socket says 172.21.15.46:47988) 2023-10-30T14:01:31.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d6d9d640 1 -- 172.21.15.46:0/1941054049 learned_addr learned my addr 172.21.15.46:0/1941054049 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:01:31.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d659c640 1 --2- 172.21.15.46:0/1941054049 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d81538a0 0x7ff5d815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106583496 0 0) 0x7ff5d80cb800 con 0x7ff5d80cf160 2023-10-30T14:01:31.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c0003530 con 0x7ff5d80cf160 2023-10-30T14:01:31.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689676632 0 0) 0x7ff5cc001120 con 0x7ff5d80ca0e0 2023-10-30T14:01:31.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5d80cb800 con 0x7ff5d80ca0e0 2023-10-30T14:01:31.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1179010540 0 0) 0x7ff5c0003530 con 0x7ff5d80cf160 2023-10-30T14:01:31.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d81538a0 msgr2=0x7ff5d815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:31.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 --2- 172.21.15.46:0/1941054049 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d81538a0 0x7ff5d815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 >> v1:172.21.15.154:6789/0 conn(0x7ff5d80ca0e0 legacy=0x7ff5d815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d80cad70 con 0x7ff5d80cf160 2023-10-30T14:01:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7ff5d0001a00 con 0x7ff5d80cf160 2023-10-30T14:01:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 >> v1:172.21.15.46:6789/0 conn(0x7ff5d80cf160 legacy=0x7ff5d8164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 --2- 172.21.15.46:0/1941054049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c0003b40 0x7ff5c0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 --2- 172.21.15.46:0/1941054049 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d81538a0 0x7ff5c00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 --2- 172.21.15.46:0/1941054049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c0004d10 0x7ff5c0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d80cb330 con 0x7ff5c0004d10 2023-10-30T14:01:31.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c000ba40 con 0x7ff5c0003b40 2023-10-30T14:01:31.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c000bbb0 con 0x7ff5d81538a0 2023-10-30T14:01:31.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d6d9d640 1 --2- 172.21.15.46:0/1941054049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c0004d10 0x7ff5c0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d659c640 1 --2- 172.21.15.46:0/1941054049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c0003b40 0x7ff5c0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d5d9b640 1 --2- 172.21.15.46:0/1941054049 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d81538a0 0x7ff5c00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d6d9d640 1 -- 172.21.15.46:0/1941054049 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d81538a0 msgr2=0x7ff5c00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d6d9d640 1 --2- 172.21.15.46:0/1941054049 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d81538a0 0x7ff5c00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d6d9d640 1 -- 172.21.15.46:0/1941054049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c0003b40 msgr2=0x7ff5c0003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d6d9d640 1 --2- 172.21.15.46:0/1941054049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c0003b40 0x7ff5c0003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.526+0000 7ff5d6d9d640 1 -- 172.21.15.46:0/1941054049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c0003530 con 0x7ff5c0004d10 2023-10-30T14:01:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.530+0000 7ff5d6d9d640 1 --2- 172.21.15.46:0/1941054049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c0004d10 0x7ff5c0005190 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000a7b0 tx=0x7ff5d001d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3954fa300e979ffb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.534+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d00ad630 con 0x7ff5c0004d10 2023-10-30T14:01:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.534+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d00a8070 con 0x7ff5c0004d10 2023-10-30T14:01:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.534+0000 7ff5d559a640 1 -- 172.21.15.46:0/1941054049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d00ad9d0 con 0x7ff5c0004d10 2023-10-30T14:01:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.534+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/1941054049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c0004d10 msgr2=0x7ff5c0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.534+0000 7ff5dcf3e640 1 --2- 172.21.15.46:0/1941054049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c0004d10 0x7ff5c0005190 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000a7b0 tx=0x7ff5d001d2a0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.534+0000 7ff5d6d9d640 1 -- 172.21.15.46:0/1941054049 reap_dead start 2023-10-30T14:01:31.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.534+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/1941054049 shutdown_connections 2023-10-30T14:01:31.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.534+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/1941054049 >> 172.21.15.46:0/1941054049 conn(0x7ff5d806bfe0 msgr2=0x7ff5d806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:31.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.534+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/1941054049 shutdown_connections 2023-10-30T14:01:31.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.534+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/1941054049 wait complete. 2023-10-30T14:01:31.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5dcf3e640 1 Processor -- start 2023-10-30T14:01:31.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5dcf3e640 1 -- start start 2023-10-30T14:01:31.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5dcf3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c0004d10 0x7ff5d80cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5dcf3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d8141c30 0x7ff5d81efc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5dcf3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5d80ce480 0x7ff5d80ce900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5dcf3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d81ef450 con 0x7ff5d80ce480 2023-10-30T14:01:31.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5dcf3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d80b4860 con 0x7ff5c0004d10 2023-10-30T14:01:31.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5dcf3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d80b4b40 con 0x7ff5d8141c30 2023-10-30T14:01:31.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5d6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5d80ce480 0x7ff5d80ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5d6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5d80ce480 0x7ff5d80ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55958/0 (socket says 172.21.15.46:55958) 2023-10-30T14:01:31.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5d6d9d640 1 -- 172.21.15.46:0/3920881768 learned_addr learned my addr 172.21.15.46:0/3920881768 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:31.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5d659c640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c0004d10 0x7ff5d80cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.538+0000 7ff5d5d9b640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d8141c30 0x7ff5d81efc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.542+0000 7ff5d6d9d640 1 -- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d8141c30 msgr2=0x7ff5d81efc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:31.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.542+0000 7ff5d6d9d640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d8141c30 0x7ff5d81efc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.542+0000 7ff5d6d9d640 1 -- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c0004d10 msgr2=0x7ff5d80cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:31.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.542+0000 7ff5d6d9d640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c0004d10 0x7ff5d80cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.542+0000 7ff5d6d9d640 1 -- 172.21.15.46:0/3920881768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d80b4e20 con 0x7ff5d80ce480 2023-10-30T14:01:31.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.542+0000 7ff5d6d9d640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5d80ce480 0x7ff5d80ce900 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7ff5d001d0d0 tx=0x7ff5c0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:31.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.542+0000 7ff5c6ffd640 1 -- 172.21.15.46:0/3920881768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0001800 con 0x7ff5d80ce480 2023-10-30T14:01:31.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.542+0000 7ff5c6ffd640 1 -- 172.21.15.46:0/3920881768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0001960 con 0x7ff5d80ce480 2023-10-30T14:01:31.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.542+0000 7ff5c6ffd640 1 -- 172.21.15.46:0/3920881768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0004d70 con 0x7ff5d80ce480 2023-10-30T14:01:31.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.542+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/3920881768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d81ef6f0 con 0x7ff5d80ce480 2023-10-30T14:01:31.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.542+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/3920881768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d8159640 con 0x7ff5d80ce480 2023-10-30T14:01:31.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.542+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/3920881768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d8159860 con 0x7ff5d80ce480 2023-10-30T14:01:31.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.546+0000 7ff5c6ffd640 1 -- 172.21.15.46:0/3920881768 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5d00aa030 con 0x7ff5d80ce480 2023-10-30T14:01:31.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.546+0000 7ff5c6ffd640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5b80664a0 0x7ff5b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.546+0000 7ff5c6ffd640 1 -- 172.21.15.46:0/3920881768 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7ff5d011e0b0 con 0x7ff5d80ce480 2023-10-30T14:01:31.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.546+0000 7ff5d659c640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5b80664a0 0x7ff5b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.546+0000 7ff5d659c640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5b80664a0 0x7ff5b8068960 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009be0 tx=0x7ff5c8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:31.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.550+0000 7ff5c6ffd640 1 -- 172.21.15.46:0/3920881768 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5d00eaf00 con 0x7ff5d80ce480 2023-10-30T14:01:31.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.742+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/3920881768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5d804e170 con 0x7ff5d80ce480 2023-10-30T14:01:31.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.742+0000 7ff5c6ffd640 1 -- 172.21.15.46:0/3920881768 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6526 (secure 0 0 0) 0x7ff5d00bc020 con 0x7ff5d80ce480 2023-10-30T14:01:31.747 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:31.747 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8034,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:12.419821+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:31.747 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 165 2023-10-30T14:01:31.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.746+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5b80664a0 msgr2=0x7ff5b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:31.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.746+0000 7ff5dcf3e640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5b80664a0 0x7ff5b8068960 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009be0 tx=0x7ff5c8009190 comp rx=0 tx=0).stop 2023-10-30T14:01:31.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.746+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/3920881768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5d80ce480 msgr2=0x7ff5d80ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:31.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.746+0000 7ff5dcf3e640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5d80ce480 0x7ff5d80ce900 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7ff5d001d0d0 tx=0x7ff5c0005800 comp rx=0 tx=0).stop 2023-10-30T14:01:31.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.746+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/3920881768 shutdown_connections 2023-10-30T14:01:31.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.746+0000 7ff5dcf3e640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5b80664a0 0x7ff5b8068960 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.746+0000 7ff5dcf3e640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5d80ce480 0x7ff5d80ce900 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.746+0000 7ff5dcf3e640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5d8141c30 0x7ff5d81efc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.746+0000 7ff5dcf3e640 1 --2- 172.21.15.46:0/3920881768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5c0004d10 0x7ff5d80cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.746+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/3920881768 >> 172.21.15.46:0/3920881768 conn(0x7ff5d806bfe0 msgr2=0x7ff5d8151750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:31.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.746+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/3920881768 shutdown_connections 2023-10-30T14:01:31.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.746+0000 7ff5dcf3e640 1 -- 172.21.15.46:0/3920881768 wait complete. 2023-10-30T14:01:31.771 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=1) 2023-10-30T14:01:31.771 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:01:31.771 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:01:31.795 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:01:31.796 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:31.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbeaf0bb640 1 Processor -- start 2023-10-30T14:01:31.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbeaf0bb640 1 -- start start 2023-10-30T14:01:31.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbeaf0bb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbea80c6b80 0x7fbea80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbeaf0bb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea80c6f60 0x7fbea80c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbeaf0bb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbea80c8430 0x7fbea813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbeaf0bb640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbea80c0a50 con 0x7fbea80c8430 2023-10-30T14:01:31.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbeaf0bb640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbea806f6e0 con 0x7fbea80c6b80 2023-10-30T14:01:31.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbeaf0bb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea806f9c0 con 0x7fbea80c6f60 2023-10-30T14:01:31.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbead658640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbea80c8430 0x7fbea813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbead658640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbea80c8430 0x7fbea813ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55972/0 (socket says 172.21.15.46:55972) 2023-10-30T14:01:31.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbead658640 1 -- 172.21.15.46:0/2841381131 learned_addr learned my addr 172.21.15.46:0/2841381131 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:31.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbea7fff640 1 --2- 172.21.15.46:0/2841381131 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea80c6f60 0x7fbea80c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbeace57640 1 --2- 172.21.15.46:0/2841381131 >> v2:172.21.15.154:3300/0 conn(0x7fbea80c6b80 0x7fbea80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbead658640 1 -- 172.21.15.46:0/2841381131 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea80c6f60 msgr2=0x7fbea80c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:31.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbead658640 1 --2- 172.21.15.46:0/2841381131 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea80c6f60 0x7fbea80c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbead658640 1 -- 172.21.15.46:0/2841381131 >> v2:172.21.15.154:3300/0 conn(0x7fbea80c6b80 msgr2=0x7fbea80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:31.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbead658640 1 --2- 172.21.15.46:0/2841381131 >> v2:172.21.15.154:3300/0 conn(0x7fbea80c6b80 0x7fbea80bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.982+0000 7fbead658640 1 -- 172.21.15.46:0/2841381131 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea806fca0 con 0x7fbea80c8430 2023-10-30T14:01:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbead658640 1 --2- 172.21.15.46:0/2841381131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80c8430 0x7fbea813ed90 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fbea806f1b0 tx=0x7fbea001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9847d5c347293cc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbea77fe640 1 -- 172.21.15.46:0/2841381131 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea00a56d0 con 0x7fbea80c8430 2023-10-30T14:01:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbea77fe640 1 -- 172.21.15.46:0/2841381131 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea00a5830 con 0x7fbea80c8430 2023-10-30T14:01:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbea77fe640 1 -- 172.21.15.46:0/2841381131 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea00a5bc0 con 0x7fbea80c8430 2023-10-30T14:01:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/2841381131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80c8430 msgr2=0x7fbea813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:31.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbeaf0bb640 1 --2- 172.21.15.46:0/2841381131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80c8430 0x7fbea813ed90 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fbea806f1b0 tx=0x7fbea001c940 comp rx=0 tx=0).stop 2023-10-30T14:01:31.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/2841381131 shutdown_connections 2023-10-30T14:01:31.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbeaf0bb640 1 --2- 172.21.15.46:0/2841381131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80c8430 0x7fbea813ed90 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbeaf0bb640 1 --2- 172.21.15.46:0/2841381131 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea80c6f60 0x7fbea80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbeaf0bb640 1 --2- 172.21.15.46:0/2841381131 >> v2:172.21.15.154:3300/0 conn(0x7fbea80c6b80 0x7fbea80bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/2841381131 >> 172.21.15.46:0/2841381131 conn(0x7fbea80c7910 msgr2=0x7fbea80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/2841381131 shutdown_connections 2023-10-30T14:01:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.986+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/2841381131 wait complete. 2023-10-30T14:01:31.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbeaf0bb640 1 Processor -- start 2023-10-30T14:01:31.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbeaf0bb640 1 -- start start 2023-10-30T14:01:31.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbeaf0bb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea80c6b80 0x7fbea813c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbeaf0bb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbea80c6f60 0x7fbea80c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbeaf0bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80c8430 0x7fbea8132810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:31.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbeaf0bb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea813e0e0 con 0x7fbea80c8430 2023-10-30T14:01:31.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbeaf0bb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea813e350 con 0x7fbea80c6f60 2023-10-30T14:01:31.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbeaf0bb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea813e630 con 0x7fbea80c6b80 2023-10-30T14:01:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbead658640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80c8430 0x7fbea8132810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbead658640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80c8430 0x7fbea8132810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55988/0 (socket says 172.21.15.46:55988) 2023-10-30T14:01:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbead658640 1 -- 172.21.15.46:0/3605422450 learned_addr learned my addr 172.21.15.46:0/3605422450 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbeace57640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea80c6b80 0x7fbea813c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbea7fff640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbea80c6f60 0x7fbea80c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbead658640 1 -- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea80c6b80 msgr2=0x7fbea813c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbead658640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea80c6b80 0x7fbea813c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbead658640 1 -- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbea80c6f60 msgr2=0x7fbea80c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbead658640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbea80c6f60 0x7fbea80c91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbead658640 1 -- 172.21.15.46:0/3605422450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea0009b30 con 0x7fbea80c8430 2023-10-30T14:01:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbea7fff640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbea80c6f60 0x7fbea80c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:01:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbead658640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80c8430 0x7fbea8132810 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fbea00af040 tx=0x7fbea00196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbea57fa640 1 -- 172.21.15.46:0/3605422450 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea00b8880 con 0x7fbea80c8430 2023-10-30T14:01:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbea57fa640 1 -- 172.21.15.46:0/3605422450 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea00b8a00 con 0x7fbea80c8430 2023-10-30T14:01:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/3605422450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea80ca220 con 0x7fbea80c8430 2023-10-30T14:01:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.990+0000 7fbea57fa640 1 -- 172.21.15.46:0/3605422450 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea00b8db0 con 0x7fbea80c8430 2023-10-30T14:01:32.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.994+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/3605422450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea80ca7e0 con 0x7fbea80c8430 2023-10-30T14:01:32.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.994+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/3605422450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea80ca080 con 0x7fbea80c8430 2023-10-30T14:01:32.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.994+0000 7fbea57fa640 1 -- 172.21.15.46:0/3605422450 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbea00b6440 con 0x7fbea80c8430 2023-10-30T14:01:32.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.994+0000 7fbea57fa640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe8c0664d0 0x7fbe8c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:32.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.994+0000 7fbea57fa640 1 -- 172.21.15.46:0/3605422450 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fbea011fc90 con 0x7fbea80c8430 2023-10-30T14:01:32.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.994+0000 7fbeace57640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe8c0664d0 0x7fbe8c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:32.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.994+0000 7fbeace57640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe8c0664d0 0x7fbe8c068990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c00ba30 tx=0x7fbe9c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:32.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:31.998+0000 7fbea57fa640 1 -- 172.21.15.46:0/3605422450 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbea80ca080 con 0x7fbea80c8430 2023-10-30T14:01:32.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/3605422450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbea80ca080 con 0x7fbea80c8430 2023-10-30T14:01:32.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbea57fa640 1 -- 172.21.15.46:0/3605422450 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6526 (secure 0 0 0) 0x7fbea80ca080 con 0x7fbea80c8430 2023-10-30T14:01:32.192 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:32.193 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8034,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:12.419821+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:32.193 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 165 2023-10-30T14:01:32.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe8c0664d0 msgr2=0x7fbe8c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:32.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe8c0664d0 0x7fbe8c068990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c00ba30 tx=0x7fbe9c008040 comp rx=0 tx=0).stop 2023-10-30T14:01:32.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/3605422450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80c8430 msgr2=0x7fbea8132810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:32.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80c8430 0x7fbea8132810 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fbea00af040 tx=0x7fbea00196e0 comp rx=0 tx=0).stop 2023-10-30T14:01:32.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/3605422450 shutdown_connections 2023-10-30T14:01:32.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe8c0664d0 0x7fbe8c068990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:32.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80c8430 0x7fbea8132810 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbea80c6f60 0x7fbea80c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 --2- 172.21.15.46:0/3605422450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea80c6b80 0x7fbea813c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/3605422450 >> 172.21.15.46:0/3605422450 conn(0x7fbea80c7910 msgr2=0x7fbea81314b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/3605422450 shutdown_connections 2023-10-30T14:01:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:32.190+0000 7fbeaf0bb640 1 -- 172.21.15.46:0/3605422450 wait complete. 2023-10-30T14:01:32.214 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:01:34.215 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:34.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.366+0000 7fe46fd1d640 1 Processor -- start 2023-10-30T14:01:34.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.366+0000 7fe46fd1d640 1 -- start start 2023-10-30T14:01:34.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.366+0000 7fe46fd1d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe4681580e0 0x7fe468070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.366+0000 7fe46fd1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4680c6540 0x7fe46815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.366+0000 7fe46fd1d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4680c5810 con 0x7fe468069e90 2023-10-30T14:01:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.366+0000 7fe46fd1d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4680c6960 con 0x7fe4681580e0 2023-10-30T14:01:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.366+0000 7fe46fd1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4680c6bb0 con 0x7fe4680c6540 2023-10-30T14:01:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.366+0000 7fe46d2b8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4680c6540 0x7fe46815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.366+0000 7fe46dab9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe4681580e0 0x7fe468070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.370+0000 7fe46dab9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe4681580e0 0x7fe468070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:36462/0 (socket says 172.21.15.46:36462) 2023-10-30T14:01:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.370+0000 7fe46d2b8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4680c6540 0x7fe46815e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:54986/0 (socket says 172.21.15.46:54986) 2023-10-30T14:01:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.370+0000 7fe46e2ba640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fe468069e90 0x7fe46806a320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:58410/0 (socket says 172.21.15.46:58410) 2023-10-30T14:01:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.370+0000 7fe46dab9640 1 -- 172.21.15.46:0/1370784806 learned_addr learned my addr 172.21.15.46:0/1370784806 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.370+0000 7fe46dab9640 1 -- 172.21.15.46:0/1370784806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4680c6540 msgr2=0x7fe46815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.370+0000 7fe46dab9640 1 --2- 172.21.15.46:0/1370784806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4680c6540 0x7fe46815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.370+0000 7fe46dab9640 1 -- 172.21.15.46:0/1370784806 >> v1:172.21.15.154:6789/0 conn(0x7fe468069e90 legacy=0x7fe46806a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.370+0000 7fe46dab9640 1 -- 172.21.15.46:0/1370784806 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4680c6e70 con 0x7fe4681580e0 2023-10-30T14:01:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.370+0000 7fe46dab9640 1 --2- 172.21.15.46:0/1370784806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4681580e0 0x7fe468070740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe464009870 tx=0x7fe46401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10cd97bf90152ebe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.370+0000 7fe46cab7640 1 -- 172.21.15.46:0/1370784806 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4640a9400 con 0x7fe4681580e0 2023-10-30T14:01:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.370+0000 7fe46cab7640 1 -- 172.21.15.46:0/1370784806 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4640a9560 con 0x7fe4681580e0 2023-10-30T14:01:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.370+0000 7fe46cab7640 1 -- 172.21.15.46:0/1370784806 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4640a98f0 con 0x7fe4681580e0 2023-10-30T14:01:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 -- 172.21.15.46:0/1370784806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4681580e0 msgr2=0x7fe468070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 --2- 172.21.15.46:0/1370784806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4681580e0 0x7fe468070740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe464009870 tx=0x7fe46401c540 comp rx=0 tx=0).stop 2023-10-30T14:01:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 -- 172.21.15.46:0/1370784806 shutdown_connections 2023-10-30T14:01:34.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 --2- 172.21.15.46:0/1370784806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4680c6540 0x7fe46815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:34.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 --2- 172.21.15.46:0/1370784806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4681580e0 0x7fe468070740 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:34.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 -- 172.21.15.46:0/1370784806 >> 172.21.15.46:0/1370784806 conn(0x7fe4680c7910 msgr2=0x7fe4680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:34.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 -- 172.21.15.46:0/1370784806 shutdown_connections 2023-10-30T14:01:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 -- 172.21.15.46:0/1370784806 wait complete. 2023-10-30T14:01:34.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 Processor -- start 2023-10-30T14:01:34.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 -- start start 2023-10-30T14:01:34.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe468069e90 0x7fe4681fa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:34.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4680c6540 0x7fe468071f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:34.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4681580e0 0x7fe4681613a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:34.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46815faf0 con 0x7fe4680c6540 2023-10-30T14:01:34.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4681faa30 con 0x7fe468069e90 2023-10-30T14:01:34.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46fd1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe468070760 con 0x7fe4681580e0 2023-10-30T14:01:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46d2b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4680c6540 0x7fe468071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46d2b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4680c6540 0x7fe468071f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56000/0 (socket says 172.21.15.46:56000) 2023-10-30T14:01:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46d2b8640 1 -- 172.21.15.46:0/948399425 learned_addr learned my addr 172.21.15.46:0/948399425 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46e2ba640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4681580e0 0x7fe4681613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.374+0000 7fe46dab9640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe468069e90 0x7fe4681fa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe46d2b8640 1 -- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4681580e0 msgr2=0x7fe4681613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:34.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe46d2b8640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4681580e0 0x7fe4681613a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:34.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe46d2b8640 1 -- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe468069e90 msgr2=0x7fe4681fa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:34.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe46d2b8640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe468069e90 0x7fe4681fa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:34.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe46d2b8640 1 -- 172.21.15.46:0/948399425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640094d0 con 0x7fe4680c6540 2023-10-30T14:01:34.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe46d2b8640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4680c6540 0x7fe468071f00 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fe45800e930 tx=0x7fe45800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:34.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe45e7fc640 1 -- 172.21.15.46:0/948399425 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe458009690 con 0x7fe4680c6540 2023-10-30T14:01:34.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe45e7fc640 1 -- 172.21.15.46:0/948399425 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45800ca00 con 0x7fe4680c6540 2023-10-30T14:01:34.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe45e7fc640 1 -- 172.21.15.46:0/948399425 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45800cd70 con 0x7fe4680c6540 2023-10-30T14:01:34.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe46fd1d640 1 -- 172.21.15.46:0/948399425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe468072440 con 0x7fe4680c6540 2023-10-30T14:01:34.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe46fd1d640 1 -- 172.21.15.46:0/948399425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe46815f1c0 con 0x7fe4680c6540 2023-10-30T14:01:34.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe46fd1d640 1 -- 172.21.15.46:0/948399425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe46804e170 con 0x7fe4680c6540 2023-10-30T14:01:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.378+0000 7fe45e7fc640 1 -- 172.21.15.46:0/948399425 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe458013470 con 0x7fe4680c6540 2023-10-30T14:01:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.382+0000 7fe45e7fc640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe44c0664a0 0x7fe44c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.382+0000 7fe45e7fc640 1 -- 172.21.15.46:0/948399425 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fe458084820 con 0x7fe4680c6540 2023-10-30T14:01:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.382+0000 7fe46dab9640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe44c0664a0 0x7fe44c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:34.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.382+0000 7fe46dab9640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe44c0664a0 0x7fe44c068960 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe4640ae000 tx=0x7fe464008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.386+0000 7fe45e7fc640 1 -- 172.21.15.46:0/948399425 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe458051610 con 0x7fe4680c6540 2023-10-30T14:01:34.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.574+0000 7fe46fd1d640 1 -- 172.21.15.46:0/948399425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe468153e90 con 0x7fe4680c6540 2023-10-30T14:01:34.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.574+0000 7fe45e7fc640 1 -- 172.21.15.46:0/948399425 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6526 (secure 0 0 0) 0x7fe458054c30 con 0x7fe4680c6540 2023-10-30T14:01:34.578 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:34.578 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8034,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:12.419821+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:34.579 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 165 2023-10-30T14:01:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.578+0000 7fe46fd1d640 1 -- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe44c0664a0 msgr2=0x7fe44c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.578+0000 7fe46fd1d640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe44c0664a0 0x7fe44c068960 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe4640ae000 tx=0x7fe464008040 comp rx=0 tx=0).stop 2023-10-30T14:01:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.578+0000 7fe46fd1d640 1 -- 172.21.15.46:0/948399425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4680c6540 msgr2=0x7fe468071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.578+0000 7fe46fd1d640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4680c6540 0x7fe468071f00 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fe45800e930 tx=0x7fe45800edf0 comp rx=0 tx=0).stop 2023-10-30T14:01:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.578+0000 7fe46fd1d640 1 -- 172.21.15.46:0/948399425 shutdown_connections 2023-10-30T14:01:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.578+0000 7fe46fd1d640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe44c0664a0 0x7fe44c068960 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.578+0000 7fe46fd1d640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4681580e0 0x7fe4681613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.578+0000 7fe46fd1d640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4680c6540 0x7fe468071f00 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.578+0000 7fe46fd1d640 1 --2- 172.21.15.46:0/948399425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe468069e90 0x7fe4681fa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.578+0000 7fe46fd1d640 1 -- 172.21.15.46:0/948399425 >> 172.21.15.46:0/948399425 conn(0x7fe4680c7910 msgr2=0x7fe46806a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:34.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.578+0000 7fe46fd1d640 1 -- 172.21.15.46:0/948399425 shutdown_connections 2023-10-30T14:01:34.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:34.578+0000 7fe46fd1d640 1 -- 172.21.15.46:0/948399425 wait complete. 2023-10-30T14:01:34.599 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:01:36.600 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:36.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f000871d640 1 Processor -- start 2023-10-30T14:01:36.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f000871d640 1 -- start start 2023-10-30T14:01:36.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f000871d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f00001538a0 0x7f000015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:36.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f000871d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00000cf160 0x7f00000cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f000871d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00000cb800 con 0x7f00000ca0e0 2023-10-30T14:01:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f000871d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f000015dc30 con 0x7f00000cf160 2023-10-30T14:01:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f000871d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00000cb330 con 0x7f00001538a0 2023-10-30T14:01:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f0006cba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00000cf160 0x7f00000cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f0006cba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00000cf160 0x7f00000cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58830/0 (socket says 172.21.15.46:58830) 2023-10-30T14:01:36.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f0006cba640 1 -- 172.21.15.46:0/3025789921 learned_addr learned my addr 172.21.15.46:0/3025789921 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:36.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f00064b9640 1 --2- 172.21.15.46:0/3025789921 >> v2:172.21.15.154:3300/0 conn(0x7f00001538a0 0x7f000015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f0006cba640 1 -- 172.21.15.46:0/3025789921 >> v2:172.21.15.154:3300/0 conn(0x7f00001538a0 msgr2=0x7f000015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f0006cba640 1 --2- 172.21.15.46:0/3025789921 >> v2:172.21.15.154:3300/0 conn(0x7f00001538a0 0x7f000015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f0006cba640 1 -- 172.21.15.46:0/3025789921 >> v1:172.21.15.154:6789/0 conn(0x7f00000ca0e0 legacy=0x7f000015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.754+0000 7f0006cba640 1 -- 172.21.15.46:0/3025789921 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00000cad70 con 0x7f00000cf160 2023-10-30T14:01:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.758+0000 7f0006cba640 1 --2- 172.21.15.46:0/3025789921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00000cf160 0x7f00000cf590 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f00000cfd10 tx=0x7efff801f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ac9613caaff4a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.758+0000 7f00054b7640 1 -- 172.21.15.46:0/3025789921 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff80af400 con 0x7f00000cf160 2023-10-30T14:01:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.758+0000 7f00054b7640 1 -- 172.21.15.46:0/3025789921 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff80ab050 con 0x7f00000cf160 2023-10-30T14:01:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.758+0000 7f00054b7640 1 -- 172.21.15.46:0/3025789921 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff80af780 con 0x7f00000cf160 2023-10-30T14:01:36.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.758+0000 7f000871d640 1 -- 172.21.15.46:0/3025789921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00000cf160 msgr2=0x7f00000cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:36.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.758+0000 7f000871d640 1 --2- 172.21.15.46:0/3025789921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00000cf160 0x7f00000cf590 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f00000cfd10 tx=0x7efff801f180 comp rx=0 tx=0).stop 2023-10-30T14:01:36.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.758+0000 7f000871d640 1 -- 172.21.15.46:0/3025789921 shutdown_connections 2023-10-30T14:01:36.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.758+0000 7f000871d640 1 --2- 172.21.15.46:0/3025789921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00000cf160 0x7f00000cf590 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:36.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.758+0000 7f000871d640 1 --2- 172.21.15.46:0/3025789921 >> v2:172.21.15.154:3300/0 conn(0x7f00001538a0 0x7f000015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:36.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.758+0000 7f000871d640 1 -- 172.21.15.46:0/3025789921 >> 172.21.15.46:0/3025789921 conn(0x7f000006bfe0 msgr2=0x7f000006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:36.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.758+0000 7f000871d640 1 -- 172.21.15.46:0/3025789921 shutdown_connections 2023-10-30T14:01:36.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f000871d640 1 -- 172.21.15.46:0/3025789921 wait complete. 2023-10-30T14:01:36.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f000871d640 1 Processor -- start 2023-10-30T14:01:36.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f000871d640 1 -- start start 2023-10-30T14:01:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f000871d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00000ca0e0 0x7f00000b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f000871d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00000cf160 0x7f00000b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f000871d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00001538a0 0x7f00001438b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f000871d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00000c9b80 con 0x7f00000ca0e0 2023-10-30T14:01:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f000871d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00000b31a0 con 0x7f00000cf160 2023-10-30T14:01:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f000871d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00000c98c0 con 0x7f00001538a0 2023-10-30T14:01:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f00064b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00000ca0e0 0x7f00000b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f00064b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00000ca0e0 0x7f00000b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58836/0 (socket says 172.21.15.46:58836) 2023-10-30T14:01:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f00064b9640 1 -- 172.21.15.46:0/3951435619 learned_addr learned my addr 172.21.15.46:0/3951435619 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f0006cba640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00001538a0 0x7f00001438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f0005cb8640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00000cf160 0x7f00000b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:36.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f00064b9640 1 -- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00001538a0 msgr2=0x7f00001438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:36.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f00064b9640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00001538a0 0x7f00001438b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:36.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f00064b9640 1 -- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00000cf160 msgr2=0x7f00000b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:36.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f00064b9640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00000cf160 0x7f00000b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:36.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.762+0000 7f00064b9640 1 -- 172.21.15.46:0/3951435619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff8009b30 con 0x7f00000ca0e0 2023-10-30T14:01:36.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.766+0000 7f00064b9640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00000ca0e0 0x7f00000b3630 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7efffc00d830 tx=0x7efffc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:36.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.766+0000 7efff6ffd640 1 -- 172.21.15.46:0/3951435619 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc00b6c0 con 0x7f00000ca0e0 2023-10-30T14:01:36.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.766+0000 7efff6ffd640 1 -- 172.21.15.46:0/3951435619 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efffc00b820 con 0x7f00000ca0e0 2023-10-30T14:01:36.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.766+0000 7efff6ffd640 1 -- 172.21.15.46:0/3951435619 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc00bb90 con 0x7f00000ca0e0 2023-10-30T14:01:36.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.766+0000 7f000871d640 1 -- 172.21.15.46:0/3951435619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00001e3a70 con 0x7f00000ca0e0 2023-10-30T14:01:36.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.766+0000 7f000871d640 1 -- 172.21.15.46:0/3951435619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00001e4030 con 0x7f00000ca0e0 2023-10-30T14:01:36.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.766+0000 7f000871d640 1 -- 172.21.15.46:0/3951435619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00000b2a50 con 0x7f00000ca0e0 2023-10-30T14:01:36.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.766+0000 7efff6ffd640 1 -- 172.21.15.46:0/3951435619 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efffc01e460 con 0x7f00000ca0e0 2023-10-30T14:01:36.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.766+0000 7efff6ffd640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7effe40664a0 0x7effe4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:36.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.766+0000 7efff6ffd640 1 -- 172.21.15.46:0/3951435619 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7efffc084c20 con 0x7f00000ca0e0 2023-10-30T14:01:36.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.766+0000 7f0005cb8640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7effe40664a0 0x7effe4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:36.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.770+0000 7f0005cb8640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7effe40664a0 0x7effe4068960 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0000146ce0 tx=0x7efff0009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:36.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.770+0000 7efff6ffd640 1 -- 172.21.15.46:0/3951435619 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efffc0519e0 con 0x7f00000ca0e0 2023-10-30T14:01:36.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 -- 172.21.15.46:0/3951435619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00001e4250 con 0x7f00000ca0e0 2023-10-30T14:01:36.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7efff6ffd640 1 -- 172.21.15.46:0/3951435619 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6526 (secure 0 0 0) 0x7efffc055000 con 0x7f00000ca0e0 2023-10-30T14:01:36.964 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:36.964 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8034,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:12.419821+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:36.965 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 165 2023-10-30T14:01:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 -- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7effe40664a0 msgr2=0x7effe4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7effe40664a0 0x7effe4068960 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0000146ce0 tx=0x7efff0009060 comp rx=0 tx=0).stop 2023-10-30T14:01:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 -- 172.21.15.46:0/3951435619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00000ca0e0 msgr2=0x7f00000b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:36.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00000ca0e0 0x7f00000b3630 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7efffc00d830 tx=0x7efffc00dcf0 comp rx=0 tx=0).stop 2023-10-30T14:01:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 -- 172.21.15.46:0/3951435619 shutdown_connections 2023-10-30T14:01:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7effe40664a0 0x7effe4068960 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00001538a0 0x7f00001438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00000cf160 0x7f00000b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:36.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 --2- 172.21.15.46:0/3951435619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00000ca0e0 0x7f00000b3630 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:36.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 -- 172.21.15.46:0/3951435619 >> 172.21.15.46:0/3951435619 conn(0x7f000006bfe0 msgr2=0x7f0000145d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:36.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 -- 172.21.15.46:0/3951435619 shutdown_connections 2023-10-30T14:01:36.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:36.962+0000 7f000871d640 1 -- 172.21.15.46:0/3951435619 wait complete. 2023-10-30T14:01:36.985 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:01:38.986 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:39.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.138+0000 7f6e2fa9e640 1 Processor -- start 2023-10-30T14:01:39.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.138+0000 7f6e2fa9e640 1 -- start start 2023-10-30T14:01:39.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2fa9e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e28070740 0x7f6e280c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:39.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2fa9e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6e2815e700 0x7f6e2806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2fa9e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e280c5810 con 0x7f6e281580e0 2023-10-30T14:01:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2fa9e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e28070b60 con 0x7f6e2815e700 2023-10-30T14:01:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2fa9e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2815eb90 con 0x7f6e28070740 2023-10-30T14:01:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2e03b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6e2815e700 0x7f6e2806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:39.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2e03b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6e2815e700 0x7f6e2806a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58838/0 (socket says 172.21.15.46:58838) 2023-10-30T14:01:39.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2e03b640 1 -- 172.21.15.46:0/1700023574 learned_addr learned my addr 172.21.15.46:0/1700023574 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:39.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2d039640 1 --2- 172.21.15.46:0/1700023574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e28070740 0x7f6e280c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2e03b640 1 -- 172.21.15.46:0/1700023574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e28070740 msgr2=0x7f6e280c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2e03b640 1 --2- 172.21.15.46:0/1700023574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e28070740 0x7f6e280c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2e03b640 1 -- 172.21.15.46:0/1700023574 >> v1:172.21.15.154:6789/0 conn(0x7f6e281580e0 legacy=0x7f6e28069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.142+0000 7f6e2e03b640 1 -- 172.21.15.46:0/1700023574 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e280c6bb0 con 0x7f6e2815e700 2023-10-30T14:01:39.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2e03b640 1 --2- 172.21.15.46:0/1700023574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e2815e700 0x7f6e2806a5a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009e50 tx=0x7f6e2001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4c9124926a0ce4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:39.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2c838640 1 -- 172.21.15.46:0/1700023574 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e200a58a0 con 0x7f6e2815e700 2023-10-30T14:01:39.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2c838640 1 -- 172.21.15.46:0/1700023574 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e200a5a00 con 0x7f6e2815e700 2023-10-30T14:01:39.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2c838640 1 -- 172.21.15.46:0/1700023574 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e200a5d90 con 0x7f6e2815e700 2023-10-30T14:01:39.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1700023574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e2815e700 msgr2=0x7f6e2806a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:39.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2fa9e640 1 --2- 172.21.15.46:0/1700023574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e2815e700 0x7f6e2806a5a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009e50 tx=0x7f6e2001c9e0 comp rx=0 tx=0).stop 2023-10-30T14:01:39.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1700023574 shutdown_connections 2023-10-30T14:01:39.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2fa9e640 1 --2- 172.21.15.46:0/1700023574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e2815e700 0x7f6e2806a5a0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:39.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2fa9e640 1 --2- 172.21.15.46:0/1700023574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e28070740 0x7f6e280c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:39.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1700023574 >> 172.21.15.46:0/1700023574 conn(0x7f6e280c7910 msgr2=0x7f6e280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:39.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1700023574 shutdown_connections 2023-10-30T14:01:39.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1700023574 wait complete. 2023-10-30T14:01:39.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2fa9e640 1 Processor -- start 2023-10-30T14:01:39.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.146+0000 7f6e2fa9e640 1 -- start start 2023-10-30T14:01:39.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2fa9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e28070740 0x7f6e2814e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:39.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2fa9e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e281580e0 0x7f6e28160570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2fa9e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e2815e700 0x7f6e28148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2fa9e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e28162050 con 0x7f6e28070740 2023-10-30T14:01:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2fa9e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e28153300 con 0x7f6e281580e0 2023-10-30T14:01:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2fa9e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2815fdb0 con 0x7f6e2815e700 2023-10-30T14:01:39.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2d83a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e28070740 0x7f6e2814e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:39.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2d83a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e28070740 0x7f6e2814e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58854/0 (socket says 172.21.15.46:58854) 2023-10-30T14:01:39.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2d83a640 1 -- 172.21.15.46:0/1702841591 learned_addr learned my addr 172.21.15.46:0/1702841591 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:39.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2e03b640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e2815e700 0x7f6e28148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:39.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2d039640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e281580e0 0x7f6e28160570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:39.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2d83a640 1 -- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e2815e700 msgr2=0x7f6e28148960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:39.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2d83a640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e2815e700 0x7f6e28148960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:39.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2d83a640 1 -- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e281580e0 msgr2=0x7f6e28160570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:39.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2d83a640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e281580e0 0x7f6e28160570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:39.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2d83a640 1 -- 172.21.15.46:0/1702841591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e20009b30 con 0x7f6e28070740 2023-10-30T14:01:39.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2d039640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e281580e0 0x7f6e28160570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:01:39.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2d83a640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e28070740 0x7f6e2814e220 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400c500 tx=0x7f6e2400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:39.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e1e7fc640 1 -- 172.21.15.46:0/1702841591 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e24005220 con 0x7f6e28070740 2023-10-30T14:01:39.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e1e7fc640 1 -- 172.21.15.46:0/1702841591 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e24005380 con 0x7f6e28070740 2023-10-30T14:01:39.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e1e7fc640 1 -- 172.21.15.46:0/1702841591 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e24007270 con 0x7f6e28070740 2023-10-30T14:01:39.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1702841591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e28152c40 con 0x7f6e28070740 2023-10-30T14:01:39.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1702841591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e281fa100 con 0x7f6e28070740 2023-10-30T14:01:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1702841591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2804e170 con 0x7f6e28070740 2023-10-30T14:01:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.150+0000 7f6e1e7fc640 1 -- 172.21.15.46:0/1702841591 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6e240073d0 con 0x7f6e28070740 2023-10-30T14:01:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.154+0000 7f6e1e7fc640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e0c0664a0 0x7f6e0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.154+0000 7f6e1e7fc640 1 -- 172.21.15.46:0/1702841591 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f6e24083e50 con 0x7f6e28070740 2023-10-30T14:01:39.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.154+0000 7f6e2d039640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e0c0664a0 0x7f6e0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:39.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.154+0000 7f6e2d039640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e0c0664a0 0x7f6e0c068960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6e28157700 tx=0x7f6e18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:39.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.158+0000 7f6e1e7fc640 1 -- 172.21.15.46:0/1702841591 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6e24050b70 con 0x7f6e28070740 2023-10-30T14:01:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.346+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1702841591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e28073e10 con 0x7f6e28070740 2023-10-30T14:01:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.346+0000 7f6e1e7fc640 1 -- 172.21.15.46:0/1702841591 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6526 (secure 0 0 0) 0x7f6e24054190 con 0x7f6e28070740 2023-10-30T14:01:39.350 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:39.350 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8034,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:12.419821+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 165 2023-10-30T14:01:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.350+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e0c0664a0 msgr2=0x7f6e0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.350+0000 7f6e2fa9e640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e0c0664a0 0x7f6e0c068960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6e28157700 tx=0x7f6e18008040 comp rx=0 tx=0).stop 2023-10-30T14:01:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.350+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1702841591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e28070740 msgr2=0x7f6e2814e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.350+0000 7f6e2fa9e640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e28070740 0x7f6e2814e220 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400c500 tx=0x7f6e2400c9c0 comp rx=0 tx=0).stop 2023-10-30T14:01:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.350+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1702841591 shutdown_connections 2023-10-30T14:01:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.350+0000 7f6e2fa9e640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6e0c0664a0 0x7f6e0c068960 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.350+0000 7f6e2fa9e640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e2815e700 0x7f6e28148960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.350+0000 7f6e2fa9e640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e281580e0 0x7f6e28160570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.350+0000 7f6e2fa9e640 1 --2- 172.21.15.46:0/1702841591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e28070740 0x7f6e2814e220 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.350+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1702841591 >> 172.21.15.46:0/1702841591 conn(0x7f6e280c7910 msgr2=0x7f6e28156e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.350+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1702841591 shutdown_connections 2023-10-30T14:01:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:39.350+0000 7f6e2fa9e640 1 -- 172.21.15.46:0/1702841591 wait complete. 2023-10-30T14:01:39.371 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:01:41.372 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.522+0000 7fbe6d30f640 1 Processor -- start 2023-10-30T14:01:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe6d30f640 1 -- start start 2023-10-30T14:01:41.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe6d30f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbe6806f7a0 0x7fbe6806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:41.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe6d30f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbe68071110 0x7fbe68158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:41.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe6d30f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe68066cf0 con 0x7fbe680c6bd0 2023-10-30T14:01:41.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe6d30f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe680700c0 con 0x7fbe68071110 2023-10-30T14:01:41.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe6d30f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe680c70d0 con 0x7fbe6806f7a0 2023-10-30T14:01:41.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe667fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbe680c6bd0 0x7fbe68069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48064/0 (socket says 172.21.15.46:48064) 2023-10-30T14:01:41.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe667fc640 1 -- 172.21.15.46:0/2455538175 learned_addr learned my addr 172.21.15.46:0/2455538175 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:01:41.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe677fe640 1 --2- 172.21.15.46:0/2455538175 >> v2:172.21.15.46:3300/0 conn(0x7fbe68071110 0x7fbe68158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe66ffd640 1 --2- 172.21.15.46:0/2455538175 >> v2:172.21.15.154:3300/0 conn(0x7fbe6806f7a0 0x7fbe6806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe65ffb640 1 -- 172.21.15.46:0/2455538175 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976921235 0 0) 0x7fbe68066cf0 con 0x7fbe680c6bd0 2023-10-30T14:01:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe677fe640 1 -- 172.21.15.46:0/2455538175 >> v2:172.21.15.154:3300/0 conn(0x7fbe6806f7a0 msgr2=0x7fbe6806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe677fe640 1 --2- 172.21.15.46:0/2455538175 >> v2:172.21.15.154:3300/0 conn(0x7fbe6806f7a0 0x7fbe6806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe677fe640 1 -- 172.21.15.46:0/2455538175 >> v1:172.21.15.46:6789/0 conn(0x7fbe680c6bd0 legacy=0x7fbe68069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.526+0000 7fbe677fe640 1 -- 172.21.15.46:0/2455538175 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe680c7390 con 0x7fbe68071110 2023-10-30T14:01:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe677fe640 1 --2- 172.21.15.46:0/2455538175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe68071110 0x7fbe68158110 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fbe6806c830 tx=0x7fbe6001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab053d8c1e052b15 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe65ffb640 1 -- 172.21.15.46:0/2455538175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe600b0440 con 0x7fbe68071110 2023-10-30T14:01:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe65ffb640 1 -- 172.21.15.46:0/2455538175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe600b05a0 con 0x7fbe68071110 2023-10-30T14:01:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe65ffb640 1 -- 172.21.15.46:0/2455538175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe600b0930 con 0x7fbe68071110 2023-10-30T14:01:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe6d30f640 1 -- 172.21.15.46:0/2455538175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe68071110 msgr2=0x7fbe68158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe6d30f640 1 --2- 172.21.15.46:0/2455538175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe68071110 0x7fbe68158110 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fbe6806c830 tx=0x7fbe6001f0f0 comp rx=0 tx=0).stop 2023-10-30T14:01:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe6d30f640 1 -- 172.21.15.46:0/2455538175 shutdown_connections 2023-10-30T14:01:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe6d30f640 1 --2- 172.21.15.46:0/2455538175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe68071110 0x7fbe68158110 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe6d30f640 1 --2- 172.21.15.46:0/2455538175 >> v2:172.21.15.154:3300/0 conn(0x7fbe6806f7a0 0x7fbe6806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe6d30f640 1 -- 172.21.15.46:0/2455538175 >> 172.21.15.46:0/2455538175 conn(0x7fbe680c7910 msgr2=0x7fbe680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe6d30f640 1 -- 172.21.15.46:0/2455538175 shutdown_connections 2023-10-30T14:01:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe6d30f640 1 -- 172.21.15.46:0/2455538175 wait complete. 2023-10-30T14:01:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe6d30f640 1 Processor -- start 2023-10-30T14:01:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.530+0000 7fbe6d30f640 1 -- start start 2023-10-30T14:01:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe6d30f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe6806f7a0 0x7fbe68154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe6d30f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe68071110 0x7fbe680c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:41.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe6d30f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe680c6bd0 0x7fbe681f24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:41.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe6d30f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe681570a0 con 0x7fbe6806f7a0 2023-10-30T14:01:41.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe6d30f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe681575d0 con 0x7fbe680c6bd0 2023-10-30T14:01:41.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe6d30f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe681f2050 con 0x7fbe68071110 2023-10-30T14:01:41.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe66ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe6806f7a0 0x7fbe68154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe66ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe6806f7a0 0x7fbe68154470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58860/0 (socket says 172.21.15.46:58860) 2023-10-30T14:01:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe66ffd640 1 -- 172.21.15.46:0/3374599488 learned_addr learned my addr 172.21.15.46:0/3374599488 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe677fe640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe680c6bd0 0x7fbe681f24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe667fc640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe68071110 0x7fbe680c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe66ffd640 1 -- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe68071110 msgr2=0x7fbe680c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe66ffd640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe68071110 0x7fbe680c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe66ffd640 1 -- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe680c6bd0 msgr2=0x7fbe681f24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe66ffd640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe680c6bd0 0x7fbe681f24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe66ffd640 1 -- 172.21.15.46:0/3374599488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe60009a70 con 0x7fbe6806f7a0 2023-10-30T14:01:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe677fe640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe680c6bd0 0x7fbe681f24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:01:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe66ffd640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe6806f7a0 0x7fbe68154470 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800d620 tx=0x7fbe5800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe4f7fe640 1 -- 172.21.15.46:0/3374599488 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800b620 con 0x7fbe6806f7a0 2023-10-30T14:01:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe4f7fe640 1 -- 172.21.15.46:0/3374599488 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5800b780 con 0x7fbe6806f7a0 2023-10-30T14:01:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe6d30f640 1 -- 172.21.15.46:0/3374599488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe68158400 con 0x7fbe6806f7a0 2023-10-30T14:01:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe4f7fe640 1 -- 172.21.15.46:0/3374599488 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800bb10 con 0x7fbe6806f7a0 2023-10-30T14:01:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe6d30f640 1 -- 172.21.15.46:0/3374599488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe68156a20 con 0x7fbe6806f7a0 2023-10-30T14:01:41.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.534+0000 7fbe6d30f640 1 -- 172.21.15.46:0/3374599488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6804e170 con 0x7fbe6806f7a0 2023-10-30T14:01:41.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.538+0000 7fbe4f7fe640 1 -- 172.21.15.46:0/3374599488 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbe5800bc70 con 0x7fbe6806f7a0 2023-10-30T14:01:41.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.538+0000 7fbe4f7fe640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe540664d0 0x7fbe54068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:41.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.538+0000 7fbe4f7fe640 1 -- 172.21.15.46:0/3374599488 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fbe58084970 con 0x7fbe6806f7a0 2023-10-30T14:01:41.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.538+0000 7fbe667fc640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe540664d0 0x7fbe54068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:41.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.538+0000 7fbe667fc640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe540664d0 0x7fbe54068990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fbe680cabc0 tx=0x7fbe5c007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:41.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.542+0000 7fbe4f7fe640 1 -- 172.21.15.46:0/3374599488 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbe58051790 con 0x7fbe6806f7a0 2023-10-30T14:01:41.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.778+0000 7fbe6d30f640 1 -- 172.21.15.46:0/3374599488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe680c8400 con 0x7fbe6806f7a0 2023-10-30T14:01:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe4f7fe640 1 -- 172.21.15.46:0/3374599488 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+6525 (secure 0 0 0) 0x7fbe58054db0 con 0x7fbe6806f7a0 2023-10-30T14:01:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:41.786 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8034,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:38.672064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 166 2023-10-30T14:01:41.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe6d30f640 1 -- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe540664d0 msgr2=0x7fbe54068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:41.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe6d30f640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe540664d0 0x7fbe54068990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fbe680cabc0 tx=0x7fbe5c007b80 comp rx=0 tx=0).stop 2023-10-30T14:01:41.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe6d30f640 1 -- 172.21.15.46:0/3374599488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe6806f7a0 msgr2=0x7fbe68154470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:41.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe6d30f640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe6806f7a0 0x7fbe68154470 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800d620 tx=0x7fbe5800dae0 comp rx=0 tx=0).stop 2023-10-30T14:01:41.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe6d30f640 1 -- 172.21.15.46:0/3374599488 shutdown_connections 2023-10-30T14:01:41.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe6d30f640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe540664d0 0x7fbe54068990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:41.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe6d30f640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe680c6bd0 0x7fbe681f24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:41.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe6d30f640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe68071110 0x7fbe680c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:41.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe6d30f640 1 --2- 172.21.15.46:0/3374599488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe6806f7a0 0x7fbe68154470 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:41.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe6d30f640 1 -- 172.21.15.46:0/3374599488 >> 172.21.15.46:0/3374599488 conn(0x7fbe680c7910 msgr2=0x7fbe680c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:41.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe6d30f640 1 -- 172.21.15.46:0/3374599488 shutdown_connections 2023-10-30T14:01:41.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:41.782+0000 7fbe6d30f640 1 -- 172.21.15.46:0/3374599488 wait complete. 2023-10-30T14:01:41.806 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:01:43.807 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:43.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.958+0000 7f0e4b59e640 1 Processor -- start 2023-10-30T14:01:43.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.958+0000 7f0e4b59e640 1 -- start start 2023-10-30T14:01:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e4b59e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0e4c1538f0 0x7f0e4c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e4b59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e4c0cefa0 0x7f0e4c0cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e4b59e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4c0ca6a0 con 0x7f0e4c15cd50 2023-10-30T14:01:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e4b59e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e4c0cf8f0 con 0x7f0e4c0cefa0 2023-10-30T14:01:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e4b59e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e4c15d300 con 0x7f0e4c1538f0 2023-10-30T14:01:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e4ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e4c0cefa0 0x7f0e4c0cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e43fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0e4c15cd50 0x7f0e4c0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48080/0 (socket says 172.21.15.46:48080) 2023-10-30T14:01:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e43fff640 1 -- 172.21.15.46:0/951670113 learned_addr learned my addr 172.21.15.46:0/951670113 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:01:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e4a59c640 1 --2- 172.21.15.46:0/951670113 >> v2:172.21.15.154:3300/0 conn(0x7f0e4c1538f0 0x7f0e4c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e4ad9d640 1 -- 172.21.15.46:0/951670113 >> v2:172.21.15.154:3300/0 conn(0x7f0e4c1538f0 msgr2=0x7f0e4c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e4ad9d640 1 --2- 172.21.15.46:0/951670113 >> v2:172.21.15.154:3300/0 conn(0x7f0e4c1538f0 0x7f0e4c159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e4ad9d640 1 -- 172.21.15.46:0/951670113 >> v1:172.21.15.46:6789/0 conn(0x7f0e4c15cd50 legacy=0x7f0e4c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.962+0000 7f0e4ad9d640 1 -- 172.21.15.46:0/951670113 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c15d5e0 con 0x7f0e4c0cefa0 2023-10-30T14:01:43.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e4ad9d640 1 --2- 172.21.15.46:0/951670113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c0cefa0 0x7f0e4c0cf3b0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0c9cb0 tx=0x7f0e4401c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c2623aaa75549a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:43.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e49d9b640 1 -- 172.21.15.46:0/951670113 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e440a5890 con 0x7f0e4c0cefa0 2023-10-30T14:01:43.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e49d9b640 1 -- 172.21.15.46:0/951670113 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e440a59f0 con 0x7f0e4c0cefa0 2023-10-30T14:01:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e49d9b640 1 -- 172.21.15.46:0/951670113 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e440a5d80 con 0x7f0e4c0cefa0 2023-10-30T14:01:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e4b59e640 1 -- 172.21.15.46:0/951670113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c0cefa0 msgr2=0x7f0e4c0cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e4b59e640 1 --2- 172.21.15.46:0/951670113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c0cefa0 0x7f0e4c0cf3b0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0c9cb0 tx=0x7f0e4401c830 comp rx=0 tx=0).stop 2023-10-30T14:01:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e4b59e640 1 -- 172.21.15.46:0/951670113 shutdown_connections 2023-10-30T14:01:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e4b59e640 1 --2- 172.21.15.46:0/951670113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c0cefa0 0x7f0e4c0cf3b0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:43.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e4b59e640 1 --2- 172.21.15.46:0/951670113 >> v2:172.21.15.154:3300/0 conn(0x7f0e4c1538f0 0x7f0e4c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e4b59e640 1 -- 172.21.15.46:0/951670113 >> 172.21.15.46:0/951670113 conn(0x7f0e4c06a640 msgr2=0x7f0e4c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e4b59e640 1 -- 172.21.15.46:0/951670113 shutdown_connections 2023-10-30T14:01:43.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e4b59e640 1 -- 172.21.15.46:0/951670113 wait complete. 2023-10-30T14:01:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.966+0000 7f0e4b59e640 1 Processor -- start 2023-10-30T14:01:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4b59e640 1 -- start start 2023-10-30T14:01:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4b59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4c0cefa0 0x7f0e4c0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4b59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4c1538f0 0x7f0e4c148050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c15cd50 0x7f0e4c1f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4b59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c158fb0 con 0x7f0e4c15cd50 2023-10-30T14:01:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4b59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c152590 con 0x7f0e4c0cefa0 2023-10-30T14:01:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4b59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c0b51f0 con 0x7f0e4c1538f0 2023-10-30T14:01:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c15cd50 0x7f0e4c1f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c15cd50 0x7f0e4c1f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58884/0 (socket says 172.21.15.46:58884) 2023-10-30T14:01:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4ad9d640 1 -- 172.21.15.46:0/1767629754 learned_addr learned my addr 172.21.15.46:0/1767629754 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:43.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4a59c640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4c0cefa0 0x7f0e4c0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e43fff640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4c1538f0 0x7f0e4c148050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4ad9d640 1 -- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4c1538f0 msgr2=0x7f0e4c148050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4ad9d640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4c1538f0 0x7f0e4c148050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4ad9d640 1 -- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4c0cefa0 msgr2=0x7f0e4c0b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4ad9d640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4c0cefa0 0x7f0e4c0b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:43.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4ad9d640 1 -- 172.21.15.46:0/1767629754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e44009a70 con 0x7f0e4c15cd50 2023-10-30T14:01:43.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e43fff640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4c1538f0 0x7f0e4c148050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:01:43.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4ad9d640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c15cd50 0x7f0e4c1f2660 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f0e4401a460 tx=0x7f0e440195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:43.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e42ffd640 1 -- 172.21.15.46:0/1767629754 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e440adc00 con 0x7f0e4c15cd50 2023-10-30T14:01:43.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e42ffd640 1 -- 172.21.15.46:0/1767629754 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e440add60 con 0x7f0e4c15cd50 2023-10-30T14:01:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e42ffd640 1 -- 172.21.15.46:0/1767629754 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e440b9690 con 0x7f0e4c15cd50 2023-10-30T14:01:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4b59e640 1 -- 172.21.15.46:0/1767629754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4c157d70 con 0x7f0e4c15cd50 2023-10-30T14:01:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4b59e640 1 -- 172.21.15.46:0/1767629754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e4c0b3c60 con 0x7f0e4c15cd50 2023-10-30T14:01:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.970+0000 7f0e4b59e640 1 -- 172.21.15.46:0/1767629754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e4c1f2380 con 0x7f0e4c15cd50 2023-10-30T14:01:43.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.974+0000 7f0e42ffd640 1 -- 172.21.15.46:0/1767629754 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e440b97f0 con 0x7f0e4c15cd50 2023-10-30T14:01:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.974+0000 7f0e42ffd640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e2c0664a0 0x7f0e2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.974+0000 7f0e42ffd640 1 -- 172.21.15.46:0/1767629754 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f0e4411fdc0 con 0x7f0e4c15cd50 2023-10-30T14:01:43.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.974+0000 7f0e4a59c640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e2c0664a0 0x7f0e2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:43.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.974+0000 7f0e4a59c640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e2c0664a0 0x7f0e2c068960 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c007a60 tx=0x7f0e3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:43.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:43.978+0000 7f0e42ffd640 1 -- 172.21.15.46:0/1767629754 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e4c1f2380 con 0x7f0e4c15cd50 2023-10-30T14:01:44.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.166+0000 7f0e4b59e640 1 -- 172.21.15.46:0/1767629754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e4c1f2380 con 0x7f0e4c15cd50 2023-10-30T14:01:44.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.166+0000 7f0e42ffd640 1 -- 172.21.15.46:0/1767629754 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+6525 (secure 0 0 0) 0x7f0e4c1f2380 con 0x7f0e4c15cd50 2023-10-30T14:01:44.171 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:44.171 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8034,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:38.672064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:44.172 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 166 2023-10-30T14:01:44.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.170+0000 7f0e4b59e640 1 -- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e2c0664a0 msgr2=0x7f0e2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:44.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.170+0000 7f0e4b59e640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e2c0664a0 0x7f0e2c068960 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c007a60 tx=0x7f0e3c008040 comp rx=0 tx=0).stop 2023-10-30T14:01:44.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.170+0000 7f0e4b59e640 1 -- 172.21.15.46:0/1767629754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c15cd50 msgr2=0x7f0e4c1f2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:44.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.170+0000 7f0e4b59e640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c15cd50 0x7f0e4c1f2660 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f0e4401a460 tx=0x7f0e440195d0 comp rx=0 tx=0).stop 2023-10-30T14:01:44.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.170+0000 7f0e4b59e640 1 -- 172.21.15.46:0/1767629754 shutdown_connections 2023-10-30T14:01:44.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.170+0000 7f0e4b59e640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e2c0664a0 0x7f0e2c068960 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:44.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.170+0000 7f0e4b59e640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c15cd50 0x7f0e4c1f2660 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:44.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.170+0000 7f0e4b59e640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4c1538f0 0x7f0e4c148050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:44.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.170+0000 7f0e4b59e640 1 --2- 172.21.15.46:0/1767629754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4c0cefa0 0x7f0e4c0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:44.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.170+0000 7f0e4b59e640 1 -- 172.21.15.46:0/1767629754 >> 172.21.15.46:0/1767629754 conn(0x7f0e4c06a640 msgr2=0x7f0e4c0c52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:44.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.170+0000 7f0e4b59e640 1 -- 172.21.15.46:0/1767629754 shutdown_connections 2023-10-30T14:01:44.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:44.170+0000 7f0e4b59e640 1 -- 172.21.15.46:0/1767629754 wait complete. 2023-10-30T14:01:44.192 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:01:46.193 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:46.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf3d80640 1 Processor -- start 2023-10-30T14:01:46.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf3d80640 1 -- start start 2023-10-30T14:01:46.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf3d80640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cec0cf160 0x7f4cec0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:46.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf3d80640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec0cb800 con 0x7f4cec15d6f0 2023-10-30T14:01:46.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf3d80640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec0cb330 con 0x7f4cec1538a0 2023-10-30T14:01:46.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf3d80640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cec0cad70 con 0x7f4cec0cf160 2023-10-30T14:01:46.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf231d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cec0cf160 0x7f4cec0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:46.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf131b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4cec15d6f0 0x7f4cec15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48094/0 (socket says 172.21.15.46:48094) 2023-10-30T14:01:46.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf131b640 1 -- 172.21.15.46:0/2865260750 learned_addr learned my addr 172.21.15.46:0/2865260750 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:01:46.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf231d640 1 -- 172.21.15.46:0/2865260750 >> v1:172.21.15.154:6789/0 conn(0x7f4cec1538a0 legacy=0x7f4cec0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:46.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf231d640 1 -- 172.21.15.46:0/2865260750 >> v1:172.21.15.46:6789/0 conn(0x7f4cec15d6f0 legacy=0x7f4cec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:46.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf231d640 1 -- 172.21.15.46:0/2865260750 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec0cb050 con 0x7f4cec0cf160 2023-10-30T14:01:46.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.346+0000 7f4cf0b1a640 1 -- 172.21.15.46:0/2865260750 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845475741 0 0) 0x7f4cec0cb800 con 0x7f4cec15d6f0 2023-10-30T14:01:46.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.350+0000 7f4cf231d640 1 --2- 172.21.15.46:0/2865260750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cec0cf160 0x7f4cec0cf590 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009d90 tx=0x7f4ce401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af9ea8def4536d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:46.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.350+0000 7f4cf0b1a640 1 -- 172.21.15.46:0/2865260750 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce40af600 con 0x7f4cec0cf160 2023-10-30T14:01:46.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.350+0000 7f4cf0b1a640 1 -- 172.21.15.46:0/2865260750 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce40ab050 con 0x7f4cec0cf160 2023-10-30T14:01:46.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.350+0000 7f4cf0b1a640 1 -- 172.21.15.46:0/2865260750 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce40af980 con 0x7f4cec0cf160 2023-10-30T14:01:46.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.350+0000 7f4cf3d80640 1 -- 172.21.15.46:0/2865260750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cec0cf160 msgr2=0x7f4cec0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:46.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.350+0000 7f4cf3d80640 1 --2- 172.21.15.46:0/2865260750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cec0cf160 0x7f4cec0cf590 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009d90 tx=0x7f4ce401f130 comp rx=0 tx=0).stop 2023-10-30T14:01:46.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.350+0000 7f4cf3d80640 1 -- 172.21.15.46:0/2865260750 shutdown_connections 2023-10-30T14:01:46.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.350+0000 7f4cf3d80640 1 --2- 172.21.15.46:0/2865260750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cec0cf160 0x7f4cec0cf590 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:46.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.350+0000 7f4cf3d80640 1 -- 172.21.15.46:0/2865260750 >> 172.21.15.46:0/2865260750 conn(0x7f4cec06bfe0 msgr2=0x7f4cec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:46.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.350+0000 7f4cf3d80640 1 -- 172.21.15.46:0/2865260750 shutdown_connections 2023-10-30T14:01:46.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.350+0000 7f4cf3d80640 1 -- 172.21.15.46:0/2865260750 wait complete. 2023-10-30T14:01:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf3d80640 1 Processor -- start 2023-10-30T14:01:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf3d80640 1 -- start start 2023-10-30T14:01:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf3d80640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cec0cf160 0x7f4cec14fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf3d80640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cec1538a0 0x7f4cec1500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf3d80640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cec15d6f0 0x7f4cec0b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf3d80640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec14f630 con 0x7f4cec0cf160 2023-10-30T14:01:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf3d80640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec1519c0 con 0x7f4cec15d6f0 2023-10-30T14:01:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf3d80640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec14dc80 con 0x7f4cec1538a0 2023-10-30T14:01:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf1b1c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cec0cf160 0x7f4cec14fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf1b1c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cec0cf160 0x7f4cec14fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58908/0 (socket says 172.21.15.46:58908) 2023-10-30T14:01:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf1b1c640 1 -- 172.21.15.46:0/3956637093 learned_addr learned my addr 172.21.15.46:0/3956637093 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf231d640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cec15d6f0 0x7f4cec0b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf131b640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cec1538a0 0x7f4cec1500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf1b1c640 1 -- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cec1538a0 msgr2=0x7f4cec1500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf1b1c640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cec1538a0 0x7f4cec1500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf1b1c640 1 -- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cec15d6f0 msgr2=0x7f4cec0b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf1b1c640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cec15d6f0 0x7f4cec0b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf1b1c640 1 -- 172.21.15.46:0/3956637093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce4009a70 con 0x7f4cec0cf160 2023-10-30T14:01:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf1b1c640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cec0cf160 0x7f4cec14fb70 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800c500 tx=0x7f4ce800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4ce27fc640 1 -- 172.21.15.46:0/3956637093 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8005220 con 0x7f4cec0cf160 2023-10-30T14:01:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4ce27fc640 1 -- 172.21.15.46:0/3956637093 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce8005380 con 0x7f4cec0cf160 2023-10-30T14:01:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf3d80640 1 -- 172.21.15.46:0/3956637093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cec151700 con 0x7f4cec0cf160 2023-10-30T14:01:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4ce27fc640 1 -- 172.21.15.46:0/3956637093 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8007270 con 0x7f4cec0cf160 2023-10-30T14:01:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf3d80640 1 -- 172.21.15.46:0/3956637093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cec14e200 con 0x7f4cec0cf160 2023-10-30T14:01:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.354+0000 7f4cf3d80640 1 -- 172.21.15.46:0/3956637093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cec04e170 con 0x7f4cec0cf160 2023-10-30T14:01:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.358+0000 7f4ce27fc640 1 -- 172.21.15.46:0/3956637093 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4ce80073d0 con 0x7f4cec0cf160 2023-10-30T14:01:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.358+0000 7f4ce27fc640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4cd00664d0 0x7f4cd0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.358+0000 7f4ce27fc640 1 -- 172.21.15.46:0/3956637093 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f4ce8083f10 con 0x7f4cec0cf160 2023-10-30T14:01:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.358+0000 7f4cf131b640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4cd00664d0 0x7f4cd0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.358+0000 7f4cf131b640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4cd00664d0 0x7f4cd0068990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4cec15ac90 tx=0x7f4cdc007d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:46.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.362+0000 7f4ce27fc640 1 -- 172.21.15.46:0/3956637093 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4ce8050c30 con 0x7f4cec0cf160 2023-10-30T14:01:46.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 -- 172.21.15.46:0/3956637093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cec0b1750 con 0x7f4cec0cf160 2023-10-30T14:01:46.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4ce27fc640 1 -- 172.21.15.46:0/3956637093 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+6525 (secure 0 0 0) 0x7f4ce8054250 con 0x7f4cec0cf160 2023-10-30T14:01:46.556 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:46.556 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8034,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:38.672064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:46.557 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 166 2023-10-30T14:01:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 -- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4cd00664d0 msgr2=0x7f4cd0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4cd00664d0 0x7f4cd0068990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4cec15ac90 tx=0x7f4cdc007d10 comp rx=0 tx=0).stop 2023-10-30T14:01:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 -- 172.21.15.46:0/3956637093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cec0cf160 msgr2=0x7f4cec14fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cec0cf160 0x7f4cec14fb70 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800c500 tx=0x7f4ce800c9c0 comp rx=0 tx=0).stop 2023-10-30T14:01:46.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 -- 172.21.15.46:0/3956637093 shutdown_connections 2023-10-30T14:01:46.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4cd00664d0 0x7f4cd0068990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:46.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cec15d6f0 0x7f4cec0b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:46.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cec1538a0 0x7f4cec1500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:46.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 --2- 172.21.15.46:0/3956637093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cec0cf160 0x7f4cec14fb70 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:46.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 -- 172.21.15.46:0/3956637093 >> 172.21.15.46:0/3956637093 conn(0x7f4cec06bfe0 msgr2=0x7f4cec0cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:46.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 -- 172.21.15.46:0/3956637093 shutdown_connections 2023-10-30T14:01:46.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:46.554+0000 7f4cf3d80640 1 -- 172.21.15.46:0/3956637093 wait complete. 2023-10-30T14:01:46.577 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:01:48.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:48.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6e0ed6640 1 Processor -- start 2023-10-30T14:01:48.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6e0ed6640 1 -- start start 2023-10-30T14:01:48.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6e0ed6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff6dc15d6f0 0x7ff6dc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:48.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6e0ed6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6dc0cf160 0x7ff6dc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:48.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6e0ed6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6dc0cb800 con 0x7ff6dc1538a0 2023-10-30T14:01:48.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6e0ed6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6dc0cb330 con 0x7ff6dc0cf160 2023-10-30T14:01:48.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6e0ed6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6dc0cad70 con 0x7ff6dc15d6f0 2023-10-30T14:01:48.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6dad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6dc0cf160 0x7ff6dc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:48.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6da59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff6dc1538a0 0x7ff6dc0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35240/0 (socket says 172.21.15.46:35240) 2023-10-30T14:01:48.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6da59c640 1 -- 172.21.15.46:0/4011456190 learned_addr learned my addr 172.21.15.46:0/4011456190 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:01:48.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6d3fff640 1 --2- 172.21.15.46:0/4011456190 >> v2:172.21.15.154:3300/0 conn(0x7ff6dc15d6f0 0x7ff6dc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:48.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6dad9d640 1 -- 172.21.15.46:0/4011456190 >> v2:172.21.15.154:3300/0 conn(0x7ff6dc15d6f0 msgr2=0x7ff6dc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:48.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6dad9d640 1 --2- 172.21.15.46:0/4011456190 >> v2:172.21.15.154:3300/0 conn(0x7ff6dc15d6f0 0x7ff6dc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:48.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6dad9d640 1 -- 172.21.15.46:0/4011456190 >> v1:172.21.15.46:6789/0 conn(0x7ff6dc1538a0 legacy=0x7ff6dc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:48.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6dad9d640 1 -- 172.21.15.46:0/4011456190 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc0cb050 con 0x7ff6dc0cf160 2023-10-30T14:01:48.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.734+0000 7ff6d9d9b640 1 -- 172.21.15.46:0/4011456190 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668680265 0 0) 0x7ff6dc0cb800 con 0x7ff6dc1538a0 2023-10-30T14:01:48.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6dad9d640 1 --2- 172.21.15.46:0/4011456190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc0cf590 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0b30a0 tx=0x7ff6d401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592ff7cd53ad2bfc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:48.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6d9d9b640 1 -- 172.21.15.46:0/4011456190 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40b0440 con 0x7ff6dc0cf160 2023-10-30T14:01:48.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6d9d9b640 1 -- 172.21.15.46:0/4011456190 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d40b05a0 con 0x7ff6dc0cf160 2023-10-30T14:01:48.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6d9d9b640 1 -- 172.21.15.46:0/4011456190 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40b0930 con 0x7ff6dc0cf160 2023-10-30T14:01:48.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/4011456190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc0cf160 msgr2=0x7ff6dc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:48.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6e0ed6640 1 --2- 172.21.15.46:0/4011456190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc0cf590 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0b30a0 tx=0x7ff6d401f0f0 comp rx=0 tx=0).stop 2023-10-30T14:01:48.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/4011456190 shutdown_connections 2023-10-30T14:01:48.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6e0ed6640 1 --2- 172.21.15.46:0/4011456190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc0cf590 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:48.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6e0ed6640 1 --2- 172.21.15.46:0/4011456190 >> v2:172.21.15.154:3300/0 conn(0x7ff6dc15d6f0 0x7ff6dc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:48.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/4011456190 >> 172.21.15.46:0/4011456190 conn(0x7ff6dc06bfe0 msgr2=0x7ff6dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:48.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/4011456190 shutdown_connections 2023-10-30T14:01:48.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/4011456190 wait complete. 2023-10-30T14:01:48.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.738+0000 7ff6e0ed6640 1 Processor -- start 2023-10-30T14:01:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6e0ed6640 1 -- start start 2023-10-30T14:01:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6e0ed6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc1542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6e0ed6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc1538a0 0x7ff6dc0b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6e0ed6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc15d6f0 0x7ff6dc1ef880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6e0ed6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc156c20 con 0x7ff6dc1538a0 2023-10-30T14:01:48.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6e0ed6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc1f3690 con 0x7ff6dc0cf160 2023-10-30T14:01:48.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6e0ed6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc1f3110 con 0x7ff6dc15d6f0 2023-10-30T14:01:48.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc1538a0 0x7ff6dc0b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:48.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc1538a0 0x7ff6dc0b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56938/0 (socket says 172.21.15.46:56938) 2023-10-30T14:01:48.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6d3fff640 1 -- 172.21.15.46:0/3493898293 learned_addr learned my addr 172.21.15.46:0/3493898293 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:48.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6dad9d640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc15d6f0 0x7ff6dc1ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:48.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6da59c640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc1542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:48.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6d3fff640 1 -- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc15d6f0 msgr2=0x7ff6dc1ef880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:48.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6d3fff640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc15d6f0 0x7ff6dc1ef880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:48.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6d3fff640 1 -- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc0cf160 msgr2=0x7ff6dc1542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:48.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6d3fff640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc1542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:48.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6d3fff640 1 -- 172.21.15.46:0/3493898293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d4009a70 con 0x7ff6dc1538a0 2023-10-30T14:01:48.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6d3fff640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc1538a0 0x7ff6dc0b4580 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400d5f0 tx=0x7ff6c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:48.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6d2ffd640 1 -- 172.21.15.46:0/3493898293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c40125f0 con 0x7ff6dc1538a0 2023-10-30T14:01:48.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6d2ffd640 1 -- 172.21.15.46:0/3493898293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c4012750 con 0x7ff6dc1538a0 2023-10-30T14:01:48.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6d2ffd640 1 -- 172.21.15.46:0/3493898293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4012ae0 con 0x7ff6dc1538a0 2023-10-30T14:01:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/3493898293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6dc154800 con 0x7ff6dc1538a0 2023-10-30T14:01:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/3493898293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6dc1f33b0 con 0x7ff6dc1538a0 2023-10-30T14:01:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.742+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/3493898293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6dc0b2a50 con 0x7ff6dc1538a0 2023-10-30T14:01:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.746+0000 7ff6d2ffd640 1 -- 172.21.15.46:0/3493898293 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff6c4012c40 con 0x7ff6dc1538a0 2023-10-30T14:01:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.746+0000 7ff6d2ffd640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6bc0664a0 0x7ff6bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.746+0000 7ff6d2ffd640 1 -- 172.21.15.46:0/3493898293 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7488+0+0 (secure 0 0 0) 0x7ff6c4084a30 con 0x7ff6dc1538a0 2023-10-30T14:01:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.746+0000 7ff6da59c640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6bc0664a0 0x7ff6bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.746+0000 7ff6da59c640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6bc0664a0 0x7ff6bc068960 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc000f70 tx=0x7ff6cc007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:48.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.750+0000 7ff6d2ffd640 1 -- 172.21.15.46:0/3493898293 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6c4051740 con 0x7ff6dc1538a0 2023-10-30T14:01:48.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/3493898293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6dc04e170 con 0x7ff6dc1538a0 2023-10-30T14:01:48.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6d2ffd640 1 -- 172.21.15.46:0/3493898293 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+6525 (secure 0 0 0) 0x7ff6c4007380 con 0x7ff6dc1538a0 2023-10-30T14:01:48.942 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:48.943 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8034,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:38.672064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":6809},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6809":{"gid":6809,"name":"e","rank":1,"incarnation":159,"state":"up:active","state_seq":45,"addr":"172.21.15.46:6837/1135482564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":1135482564},{"type":"v1","addr":"172.21.15.46:6837","nonce":1135482564}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:48.944 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 166 2023-10-30T14:01:48.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6bc0664a0 msgr2=0x7ff6bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:48.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6bc0664a0 0x7ff6bc068960 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc000f70 tx=0x7ff6cc007b60 comp rx=0 tx=0).stop 2023-10-30T14:01:48.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/3493898293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc1538a0 msgr2=0x7ff6dc0b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:48.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc1538a0 0x7ff6dc0b4580 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400d5f0 tx=0x7ff6c400dab0 comp rx=0 tx=0).stop 2023-10-30T14:01:48.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/3493898293 shutdown_connections 2023-10-30T14:01:48.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff6bc0664a0 0x7ff6bc068960 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:48.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc15d6f0 0x7ff6dc1ef880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:48.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6dc1538a0 0x7ff6dc0b4580 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:48.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 --2- 172.21.15.46:0/3493898293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc0cf160 0x7ff6dc1542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:48.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/3493898293 >> 172.21.15.46:0/3493898293 conn(0x7ff6dc06bfe0 msgr2=0x7ff6dc15aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:48.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/3493898293 shutdown_connections 2023-10-30T14:01:48.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:48.938+0000 7ff6e0ed6640 1 -- 172.21.15.46:0/3493898293 wait complete. 2023-10-30T14:01:48.961 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:01:50.963 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.114+0000 7fd673980640 1 Processor -- start 2023-10-30T14:01:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd673980640 1 -- start start 2023-10-30T14:01:51.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd673980640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd66c06f7a0 0x7fd66c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd673980640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd66c0c6bd0 0x7fd66c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd673980640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd66c071110 0x7fd66c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd673980640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c0700c0 con 0x7fd66c06f7a0 2023-10-30T14:01:51.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd673980640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c06a3d0 con 0x7fd66c071110 2023-10-30T14:01:51.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd673980640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c066cf0 con 0x7fd66c0c6bd0 2023-10-30T14:01:51.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd67171c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd66c06f7a0 0x7fd66c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd67171c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd66c06f7a0 0x7fd66c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56940/0 (socket says 172.21.15.46:56940) 2023-10-30T14:01:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd67171c640 1 -- 172.21.15.46:0/306666094 learned_addr learned my addr 172.21.15.46:0/306666094 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd670f1b640 1 --2- 172.21.15.46:0/306666094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd66c0c6bd0 0x7fd66c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd671f1d640 1 --2- 172.21.15.46:0/306666094 >> v2:172.21.15.154:3300/0 conn(0x7fd66c071110 0x7fd66c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd67171c640 1 -- 172.21.15.46:0/306666094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd66c0c6bd0 msgr2=0x7fd66c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd67171c640 1 --2- 172.21.15.46:0/306666094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd66c0c6bd0 0x7fd66c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd67171c640 1 -- 172.21.15.46:0/306666094 >> v2:172.21.15.154:3300/0 conn(0x7fd66c071110 msgr2=0x7fd66c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd67171c640 1 --2- 172.21.15.46:0/306666094 >> v2:172.21.15.154:3300/0 conn(0x7fd66c071110 0x7fd66c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd67171c640 1 -- 172.21.15.46:0/306666094 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0c70d0 con 0x7fd66c06f7a0 2023-10-30T14:01:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.118+0000 7fd671f1d640 1 --2- 172.21.15.46:0/306666094 >> v2:172.21.15.154:3300/0 conn(0x7fd66c071110 0x7fd66c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:01:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd67171c640 1 --2- 172.21.15.46:0/306666094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c06f7a0 0x7fd66c06fb80 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fd66c06c7d0 tx=0x7fd66801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123956a09beb176 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd663fff640 1 -- 172.21.15.46:0/306666094 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6680a89b0 con 0x7fd66c06f7a0 2023-10-30T14:01:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd663fff640 1 -- 172.21.15.46:0/306666094 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6680a8b10 con 0x7fd66c06f7a0 2023-10-30T14:01:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd663fff640 1 -- 172.21.15.46:0/306666094 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6680a8ea0 con 0x7fd66c06f7a0 2023-10-30T14:01:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd673980640 1 -- 172.21.15.46:0/306666094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c06f7a0 msgr2=0x7fd66c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd673980640 1 --2- 172.21.15.46:0/306666094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c06f7a0 0x7fd66c06fb80 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fd66c06c7d0 tx=0x7fd66801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd673980640 1 -- 172.21.15.46:0/306666094 shutdown_connections 2023-10-30T14:01:51.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd673980640 1 --2- 172.21.15.46:0/306666094 >> v2:172.21.15.154:3300/0 conn(0x7fd66c071110 0x7fd66c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd673980640 1 --2- 172.21.15.46:0/306666094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd66c0c6bd0 0x7fd66c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd673980640 1 --2- 172.21.15.46:0/306666094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c06f7a0 0x7fd66c06fb80 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd673980640 1 -- 172.21.15.46:0/306666094 >> 172.21.15.46:0/306666094 conn(0x7fd66c0c7910 msgr2=0x7fd66c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:51.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd673980640 1 -- 172.21.15.46:0/306666094 shutdown_connections 2023-10-30T14:01:51.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd673980640 1 -- 172.21.15.46:0/306666094 wait complete. 2023-10-30T14:01:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd673980640 1 Processor -- start 2023-10-30T14:01:51.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.122+0000 7fd673980640 1 -- start start 2023-10-30T14:01:51.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd673980640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd66c06f7a0 0x7fd66c1efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd673980640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c071110 0x7fd66c150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd673980640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66c0c6bd0 0x7fd66c149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd673980640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c14b4e0 con 0x7fd66c071110 2023-10-30T14:01:51.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd673980640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c1f0340 con 0x7fd66c0c6bd0 2023-10-30T14:01:51.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd673980640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c14af20 con 0x7fd66c06f7a0 2023-10-30T14:01:51.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd670f1b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c071110 0x7fd66c150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd670f1b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c071110 0x7fd66c150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56948/0 (socket says 172.21.15.46:56948) 2023-10-30T14:01:51.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd670f1b640 1 -- 172.21.15.46:0/921447685 learned_addr learned my addr 172.21.15.46:0/921447685 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:51.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd67171c640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd66c06f7a0 0x7fd66c1efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd671f1d640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66c0c6bd0 0x7fd66c149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd670f1b640 1 -- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd66c06f7a0 msgr2=0x7fd66c1efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:51.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd670f1b640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd66c06f7a0 0x7fd66c1efe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd670f1b640 1 -- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66c0c6bd0 msgr2=0x7fd66c149630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:51.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd670f1b640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66c0c6bd0 0x7fd66c149630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd670f1b640 1 -- 172.21.15.46:0/921447685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6680094d0 con 0x7fd66c071110 2023-10-30T14:01:51.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd671f1d640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66c0c6bd0 0x7fd66c149630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:01:51.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd670f1b640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c071110 0x7fd66c150350 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00e870 tx=0x7fd65c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:51.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd661ffb640 1 -- 172.21.15.46:0/921447685 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c009510 con 0x7fd66c071110 2023-10-30T14:01:51.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd661ffb640 1 -- 172.21.15.46:0/921447685 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65c00eec0 con 0x7fd66c071110 2023-10-30T14:01:51.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd673980640 1 -- 172.21.15.46:0/921447685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c14b200 con 0x7fd66c071110 2023-10-30T14:01:51.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd661ffb640 1 -- 172.21.15.46:0/921447685 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c00cb40 con 0x7fd66c071110 2023-10-30T14:01:51.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd673980640 1 -- 172.21.15.46:0/921447685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c14cea0 con 0x7fd66c071110 2023-10-30T14:01:51.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.126+0000 7fd673980640 1 -- 172.21.15.46:0/921447685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66c14ca20 con 0x7fd66c071110 2023-10-30T14:01:51.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.130+0000 7fd661ffb640 1 -- 172.21.15.46:0/921447685 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd65c00cca0 con 0x7fd66c071110 2023-10-30T14:01:51.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.130+0000 7fd661ffb640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6500664d0 0x7fd650068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.130+0000 7fd661ffb640 1 -- 172.21.15.46:0/921447685 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7fd65c084720 con 0x7fd66c071110 2023-10-30T14:01:51.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.130+0000 7fd67171c640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6500664d0 0x7fd650068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.130+0000 7fd67171c640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6500664d0 0x7fd650068990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd6680b2000 tx=0x7fd668009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:51.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.134+0000 7fd661ffb640 1 -- 172.21.15.46:0/921447685 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd66c14ca20 con 0x7fd66c071110 2023-10-30T14:01:51.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.322+0000 7fd673980640 1 -- 172.21.15.46:0/921447685 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd66c14ca20 con 0x7fd66c071110 2023-10-30T14:01:51.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.322+0000 7fd661ffb640 1 -- 172.21.15.46:0/921447685 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5731 (secure 0 0 0) 0x7fd66c14ca20 con 0x7fd66c071110 2023-10-30T14:01:51.326 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:51.326 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:49.404315+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:51.327 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 167 2023-10-30T14:01:51.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.326+0000 7fd673980640 1 -- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6500664d0 msgr2=0x7fd650068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:51.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.326+0000 7fd673980640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6500664d0 0x7fd650068990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd6680b2000 tx=0x7fd668009040 comp rx=0 tx=0).stop 2023-10-30T14:01:51.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.326+0000 7fd673980640 1 -- 172.21.15.46:0/921447685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c071110 msgr2=0x7fd66c150350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:51.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.326+0000 7fd673980640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c071110 0x7fd66c150350 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00e870 tx=0x7fd65c00ec70 comp rx=0 tx=0).stop 2023-10-30T14:01:51.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.326+0000 7fd673980640 1 -- 172.21.15.46:0/921447685 shutdown_connections 2023-10-30T14:01:51.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.326+0000 7fd673980640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6500664d0 0x7fd650068990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.326+0000 7fd673980640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66c0c6bd0 0x7fd66c149630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.326+0000 7fd673980640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c071110 0x7fd66c150350 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.326+0000 7fd673980640 1 --2- 172.21.15.46:0/921447685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd66c06f7a0 0x7fd66c1efe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.326+0000 7fd673980640 1 -- 172.21.15.46:0/921447685 >> 172.21.15.46:0/921447685 conn(0x7fd66c0c7910 msgr2=0x7fd66c06c050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:51.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.326+0000 7fd673980640 1 -- 172.21.15.46:0/921447685 shutdown_connections 2023-10-30T14:01:51.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.326+0000 7fd673980640 1 -- 172.21.15.46:0/921447685 wait complete. 2023-10-30T14:01:51.347 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T14:01:51.347 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:01:51.347 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c045a1640 1 Processor -- start 2023-10-30T14:01:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c045a1640 1 -- start start 2023-10-30T14:01:51.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c045a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bfc1538a0 0x7f1bfc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c045a1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1bfc0ca0e0 0x7f1bfc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c045a1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bfc0cb800 con 0x7f1bfc0cf160 2023-10-30T14:01:51.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c045a1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bfc15dc30 con 0x7f1bfc0ca0e0 2023-10-30T14:01:51.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c045a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc0cb330 con 0x7f1bfc1538a0 2023-10-30T14:01:51.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c0233d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bfc1538a0 0x7f1bfc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c01b3c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1bfc0ca0e0 0x7f1bfc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c0233d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bfc1538a0 0x7f1bfc15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:39080/0 (socket says 172.21.15.46:39080) 2023-10-30T14:01:51.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c0233d640 1 -- 172.21.15.46:0/1684573319 learned_addr learned my addr 172.21.15.46:0/1684573319 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:51.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c0233d640 1 -- 172.21.15.46:0/1684573319 >> v2:172.21.15.154:3300/0 conn(0x7f1bfc0ca0e0 msgr2=0x7f1bfc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:51.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c0233d640 1 --2- 172.21.15.46:0/1684573319 >> v2:172.21.15.154:3300/0 conn(0x7f1bfc0ca0e0 0x7f1bfc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c0233d640 1 -- 172.21.15.46:0/1684573319 >> v1:172.21.15.154:6789/0 conn(0x7f1bfc0cf160 legacy=0x7f1bfc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:51.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.490+0000 7f1c0233d640 1 -- 172.21.15.46:0/1684573319 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc0cad70 con 0x7f1bfc1538a0 2023-10-30T14:01:51.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.494+0000 7f1c0233d640 1 --2- 172.21.15.46:0/1684573319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bfc1538a0 0x7f1bfc15d6f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009870 tx=0x7f1bf801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18a4ddb0fd1a0318 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:51.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.494+0000 7f1c0133b640 1 -- 172.21.15.46:0/1684573319 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf80a8a30 con 0x7f1bfc1538a0 2023-10-30T14:01:51.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.494+0000 7f1c0133b640 1 -- 172.21.15.46:0/1684573319 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf80a8b90 con 0x7f1bfc1538a0 2023-10-30T14:01:51.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.494+0000 7f1c0133b640 1 -- 172.21.15.46:0/1684573319 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf80b0440 con 0x7f1bfc1538a0 2023-10-30T14:01:51.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.494+0000 7f1c045a1640 1 -- 172.21.15.46:0/1684573319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bfc1538a0 msgr2=0x7f1bfc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:51.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.494+0000 7f1c045a1640 1 --2- 172.21.15.46:0/1684573319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bfc1538a0 0x7f1bfc15d6f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009870 tx=0x7f1bf801ed30 comp rx=0 tx=0).stop 2023-10-30T14:01:51.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.494+0000 7f1c045a1640 1 -- 172.21.15.46:0/1684573319 shutdown_connections 2023-10-30T14:01:51.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.494+0000 7f1c045a1640 1 --2- 172.21.15.46:0/1684573319 >> v2:172.21.15.154:3300/0 conn(0x7f1bfc0ca0e0 0x7f1bfc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.494+0000 7f1c045a1640 1 --2- 172.21.15.46:0/1684573319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bfc1538a0 0x7f1bfc15d6f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.494+0000 7f1c045a1640 1 -- 172.21.15.46:0/1684573319 >> 172.21.15.46:0/1684573319 conn(0x7f1bfc06bfe0 msgr2=0x7f1bfc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:51.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.494+0000 7f1c045a1640 1 -- 172.21.15.46:0/1684573319 shutdown_connections 2023-10-30T14:01:51.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.494+0000 7f1c045a1640 1 -- 172.21.15.46:0/1684573319 wait complete. 2023-10-30T14:01:51.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c045a1640 1 Processor -- start 2023-10-30T14:01:51.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c045a1640 1 -- start start 2023-10-30T14:01:51.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c045a1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bfc0ca0e0 0x7f1bfc0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c045a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bfc0cf160 0x7f1bfc0b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c045a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bfc1538a0 0x7f1bfc0b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c045a1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc1efcc0 con 0x7f1bfc1538a0 2023-10-30T14:01:51.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c045a1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc1f01f0 con 0x7f1bfc0ca0e0 2023-10-30T14:01:51.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c045a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc1eff10 con 0x7f1bfc0cf160 2023-10-30T14:01:51.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c02b3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bfc1538a0 0x7f1bfc0b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c02b3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bfc1538a0 0x7f1bfc0b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56960/0 (socket says 172.21.15.46:56960) 2023-10-30T14:01:51.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c02b3e640 1 -- 172.21.15.46:0/2046424891 learned_addr learned my addr 172.21.15.46:0/2046424891 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:51.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c01b3c640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bfc0cf160 0x7f1bfc0b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c0233d640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bfc0ca0e0 0x7f1bfc0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c02b3e640 1 -- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bfc0cf160 msgr2=0x7f1bfc0b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:51.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c02b3e640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bfc0cf160 0x7f1bfc0b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c02b3e640 1 -- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bfc0ca0e0 msgr2=0x7f1bfc0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:51.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c02b3e640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bfc0ca0e0 0x7f1bfc0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c02b3e640 1 -- 172.21.15.46:0/2046424891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf80094d0 con 0x7f1bfc1538a0 2023-10-30T14:01:51.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c02b3e640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bfc1538a0 0x7f1bfc0b42f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400c6c0 tx=0x7f1bf400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:51.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1bf2ffd640 1 -- 172.21.15.46:0/2046424891 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4003b80 con 0x7f1bfc1538a0 2023-10-30T14:01:51.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1bf2ffd640 1 -- 172.21.15.46:0/2046424891 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf4003ce0 con 0x7f1bfc1538a0 2023-10-30T14:01:51.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c045a1640 1 -- 172.21.15.46:0/2046424891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc0b32d0 con 0x7f1bfc1538a0 2023-10-30T14:01:51.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1bf2ffd640 1 -- 172.21.15.46:0/2046424891 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40052f0 con 0x7f1bfc1538a0 2023-10-30T14:01:51.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.498+0000 7f1c045a1640 1 -- 172.21.15.46:0/2046424891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bfc0b3890 con 0x7f1bfc1538a0 2023-10-30T14:01:51.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.502+0000 7f1c045a1640 1 -- 172.21.15.46:0/2046424891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc0b2a50 con 0x7f1bfc1538a0 2023-10-30T14:01:51.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.502+0000 7f1bf2ffd640 1 -- 172.21.15.46:0/2046424891 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1bf4005450 con 0x7f1bfc1538a0 2023-10-30T14:01:51.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.502+0000 7f1bf2ffd640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1be00664d0 0x7f1be0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:51.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.502+0000 7f1bf2ffd640 1 -- 172.21.15.46:0/2046424891 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f1bf4084120 con 0x7f1bfc1538a0 2023-10-30T14:01:51.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.502+0000 7f1c0233d640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1be00664d0 0x7f1be0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:51.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.502+0000 7f1c0233d640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1be00664d0 0x7f1be0068990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80b2000 tx=0x7f1bf80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:51.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.506+0000 7f1bf2ffd640 1 -- 172.21.15.46:0/2046424891 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1bf4050f90 con 0x7f1bfc1538a0 2023-10-30T14:01:51.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 -- 172.21.15.46:0/2046424891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bfc04e170 con 0x7f1bfc1538a0 2023-10-30T14:01:51.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1bf2ffd640 1 -- 172.21.15.46:0/2046424891 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f1bf40545b0 con 0x7f1bfc1538a0 2023-10-30T14:01:51.701 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:51.701 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:49.404315+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:51.702 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 167 2023-10-30T14:01:51.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 -- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1be00664d0 msgr2=0x7f1be0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:51.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1be00664d0 0x7f1be0068990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80b2000 tx=0x7f1bf80aa070 comp rx=0 tx=0).stop 2023-10-30T14:01:51.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 -- 172.21.15.46:0/2046424891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bfc1538a0 msgr2=0x7f1bfc0b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:51.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bfc1538a0 0x7f1bfc0b42f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400c6c0 tx=0x7f1bf400cb80 comp rx=0 tx=0).stop 2023-10-30T14:01:51.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 -- 172.21.15.46:0/2046424891 shutdown_connections 2023-10-30T14:01:51.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1be00664d0 0x7f1be0068990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bfc1538a0 0x7f1bfc0b42f0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bfc0cf160 0x7f1bfc0b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 --2- 172.21.15.46:0/2046424891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bfc0ca0e0 0x7f1bfc0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:51.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 -- 172.21.15.46:0/2046424891 >> 172.21.15.46:0/2046424891 conn(0x7f1bfc06bfe0 msgr2=0x7f1bfc152870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:51.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 -- 172.21.15.46:0/2046424891 shutdown_connections 2023-10-30T14:01:51.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:51.698+0000 7f1c045a1640 1 -- 172.21.15.46:0/2046424891 wait complete. 2023-10-30T14:01:51.723 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 167, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 167, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7704": { "addr": "172.21.15.46:6836/3734066164", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3734066164, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3734066164, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7704, "incarnation": 152, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_8034": { "addr": "172.21.15.46:6833/2075693104", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2075693104, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2075693104, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8034, "incarnation": 167, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:01:49.404315+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7704, "mds_1": 8034 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:01:53.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:53.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.874+0000 7feaf4b1d640 1 Processor -- start 2023-10-30T14:01:53.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.874+0000 7feaf4b1d640 1 -- start start 2023-10-30T14:01:53.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feaf4b1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaf00c05d0 0x7feaf013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:53.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feaf4b1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf00c0a50 con 0x7feaf00c6f60 2023-10-30T14:01:53.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feaf4b1d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf006f6e0 con 0x7feaf00c6b80 2023-10-30T14:01:53.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feaf4b1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf006f9c0 con 0x7feaf00c05d0 2023-10-30T14:01:53.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feaeed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7feaf00c6f60 0x7feaf01566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35256/0 (socket says 172.21.15.46:35256) 2023-10-30T14:01:53.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feaeed9d640 1 -- 172.21.15.46:0/1093010923 learned_addr learned my addr 172.21.15.46:0/1093010923 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:01:53.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feae7fff640 1 --2- 172.21.15.46:0/1093010923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaf00c05d0 0x7feaf013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:53.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feaedd9b640 1 -- 172.21.15.46:0/1093010923 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688589051 0 0) 0x7feaf00c0a50 con 0x7feaf00c6f60 2023-10-30T14:01:53.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feaedd9b640 1 -- 172.21.15.46:0/1093010923 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feadc003530 con 0x7feaf00c6f60 2023-10-30T14:01:53.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feae7fff640 1 -- 172.21.15.46:0/1093010923 >> v1:172.21.15.154:6789/0 conn(0x7feaf00c6b80 legacy=0x7feaf00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:53.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feae7fff640 1 -- 172.21.15.46:0/1093010923 >> v1:172.21.15.46:6789/0 conn(0x7feaf00c6f60 legacy=0x7feaf01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:53.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feaedd9b640 1 -- 172.21.15.46:0/1093010923 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3746894312 0 0) 0x7feadc003530 con 0x7feaf00c6f60 2023-10-30T14:01:53.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.878+0000 7feae7fff640 1 -- 172.21.15.46:0/1093010923 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf006fca0 con 0x7feaf00c05d0 2023-10-30T14:01:53.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.882+0000 7feae7fff640 1 --2- 172.21.15.46:0/1093010923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaf00c05d0 0x7feaf013ed90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fead80097f0 tx=0x7fead801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d45c02d0f80d2dbb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:53.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.882+0000 7feaedd9b640 1 -- 172.21.15.46:0/1093010923 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae8001a00 con 0x7feaf00c05d0 2023-10-30T14:01:53.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.882+0000 7feaedd9b640 1 -- 172.21.15.46:0/1093010923 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead80a7050 con 0x7feaf00c05d0 2023-10-30T14:01:53.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.882+0000 7feaedd9b640 1 -- 172.21.15.46:0/1093010923 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead80ab600 con 0x7feaf00c05d0 2023-10-30T14:01:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.882+0000 7feaf4b1d640 1 -- 172.21.15.46:0/1093010923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaf00c05d0 msgr2=0x7feaf013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.882+0000 7feaf4b1d640 1 --2- 172.21.15.46:0/1093010923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaf00c05d0 0x7feaf013ed90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fead80097f0 tx=0x7fead801c3e0 comp rx=0 tx=0).stop 2023-10-30T14:01:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.882+0000 7feaf4b1d640 1 -- 172.21.15.46:0/1093010923 shutdown_connections 2023-10-30T14:01:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.882+0000 7feaf4b1d640 1 --2- 172.21.15.46:0/1093010923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaf00c05d0 0x7feaf013ed90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.882+0000 7feaf4b1d640 1 -- 172.21.15.46:0/1093010923 >> 172.21.15.46:0/1093010923 conn(0x7feaf00c7910 msgr2=0x7feaf00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.882+0000 7feaf4b1d640 1 -- 172.21.15.46:0/1093010923 shutdown_connections 2023-10-30T14:01:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.882+0000 7feaf4b1d640 1 -- 172.21.15.46:0/1093010923 wait complete. 2023-10-30T14:01:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaf4b1d640 1 Processor -- start 2023-10-30T14:01:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaf4b1d640 1 -- start start 2023-10-30T14:01:53.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaf4b1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaf00c05d0 0x7feaf01591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:53.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaf4b1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaf00c6b80 0x7feaf00c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:53.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaf4b1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaf00c6f60 0x7feaf0137e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:53.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaf4b1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf0155530 con 0x7feaf00c6f60 2023-10-30T14:01:53.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaf4b1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf01557f0 con 0x7feaf00c6b80 2023-10-30T14:01:53.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaf4b1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf0155ad0 con 0x7feaf00c05d0 2023-10-30T14:01:53.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaeed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaf00c6f60 0x7feaf0137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:53.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaeed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaf00c6f60 0x7feaf0137e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56970/0 (socket says 172.21.15.46:56970) 2023-10-30T14:01:53.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaeed9d640 1 -- 172.21.15.46:0/123761942 learned_addr learned my addr 172.21.15.46:0/123761942 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:53.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feae7fff640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaf00c6b80 0x7feaf00c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:53.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaee59c640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaf00c05d0 0x7feaf01591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:53.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaeed9d640 1 -- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaf00c05d0 msgr2=0x7feaf01591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:53.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaeed9d640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaf00c05d0 0x7feaf01591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:53.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaeed9d640 1 -- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaf00c6b80 msgr2=0x7feaf00c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:53.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaeed9d640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaf00c6b80 0x7feaf00c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:53.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaeed9d640 1 -- 172.21.15.46:0/123761942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead8009420 con 0x7feaf00c6f60 2023-10-30T14:01:53.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feae7fff640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaf00c6b80 0x7feaf00c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:01:53.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaeed9d640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaf00c6f60 0x7feaf0137e90 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7feae800cc70 tx=0x7feae800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:53.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feae6ffd640 1 -- 172.21.15.46:0/123761942 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae800ce80 con 0x7feaf00c6f60 2023-10-30T14:01:53.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feae6ffd640 1 -- 172.21.15.46:0/123761942 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae8012400 con 0x7feaf00c6f60 2023-10-30T14:01:53.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feae6ffd640 1 -- 172.21.15.46:0/123761942 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae80127b0 con 0x7feaf00c6f60 2023-10-30T14:01:53.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaf4b1d640 1 -- 172.21.15.46:0/123761942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf0155db0 con 0x7feaf00c6f60 2023-10-30T14:01:53.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaf4b1d640 1 -- 172.21.15.46:0/123761942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf006b910 con 0x7feaf00c6f60 2023-10-30T14:01:53.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.886+0000 7feaf4b1d640 1 -- 172.21.15.46:0/123761942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf00c3380 con 0x7feaf00c6f60 2023-10-30T14:01:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.890+0000 7feae6ffd640 1 -- 172.21.15.46:0/123761942 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feae8012910 con 0x7feaf00c6f60 2023-10-30T14:01:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.890+0000 7feae6ffd640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fead00664a0 0x7fead0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.890+0000 7feae6ffd640 1 -- 172.21.15.46:0/123761942 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7feae8084d30 con 0x7feaf00c6f60 2023-10-30T14:01:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.890+0000 7feaee59c640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fead00664a0 0x7fead0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.890+0000 7feaee59c640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fead00664a0 0x7fead0068960 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7feae0007fd0 tx=0x7feae0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:53.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:53.894+0000 7feae6ffd640 1 -- 172.21.15.46:0/123761942 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feaf00c3380 con 0x7feaf00c6f60 2023-10-30T14:01:54.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.086+0000 7feaf4b1d640 1 -- 172.21.15.46:0/123761942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feaf00c3380 con 0x7feaf00c6f60 2023-10-30T14:01:54.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.086+0000 7feae6ffd640 1 -- 172.21.15.46:0/123761942 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5731 (secure 0 0 0) 0x7feaf00c3380 con 0x7feaf00c6f60 2023-10-30T14:01:54.091 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:54.092 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:49.404315+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:54.092 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 167 2023-10-30T14:01:54.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.090+0000 7feaf4b1d640 1 -- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fead00664a0 msgr2=0x7fead0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:54.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.090+0000 7feaf4b1d640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fead00664a0 0x7fead0068960 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7feae0007fd0 tx=0x7feae0009040 comp rx=0 tx=0).stop 2023-10-30T14:01:54.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.090+0000 7feaf4b1d640 1 -- 172.21.15.46:0/123761942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaf00c6f60 msgr2=0x7feaf0137e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:54.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.090+0000 7feaf4b1d640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaf00c6f60 0x7feaf0137e90 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7feae800cc70 tx=0x7feae800a430 comp rx=0 tx=0).stop 2023-10-30T14:01:54.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.090+0000 7feaf4b1d640 1 -- 172.21.15.46:0/123761942 shutdown_connections 2023-10-30T14:01:54.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.090+0000 7feaf4b1d640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fead00664a0 0x7fead0068960 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:54.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.090+0000 7feaf4b1d640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feaf00c6f60 0x7feaf0137e90 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:54.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.090+0000 7feaf4b1d640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaf00c6b80 0x7feaf00c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:54.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.090+0000 7feaf4b1d640 1 --2- 172.21.15.46:0/123761942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaf00c05d0 0x7feaf01591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:54.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.090+0000 7feaf4b1d640 1 -- 172.21.15.46:0/123761942 >> 172.21.15.46:0/123761942 conn(0x7feaf00c7910 msgr2=0x7feaf00c1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:54.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.090+0000 7feaf4b1d640 1 -- 172.21.15.46:0/123761942 shutdown_connections 2023-10-30T14:01:54.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:54.090+0000 7feaf4b1d640 1 -- 172.21.15.46:0/123761942 wait complete. 2023-10-30T14:01:54.113 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:01:54.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:01:54.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:01:56.114 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:56.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.262+0000 7f7790a74640 1 Processor -- start 2023-10-30T14:01:56.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.262+0000 7f7790a74640 1 -- start start 2023-10-30T14:01:56.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f7790a74640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f778c1538f0 0x7f778c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:56.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f7790a74640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f778c0cefa0 0x7f778c0cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:56.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f7790a74640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c0ca6a0 con 0x7f778c15cd50 2023-10-30T14:01:56.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f7790a74640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f778c0cf8f0 con 0x7f778c1538f0 2023-10-30T14:01:56.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f7790a74640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f778c15d300 con 0x7f778c0cefa0 2023-10-30T14:01:56.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f778affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f778c15cd50 0x7f778c0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35270/0 (socket says 172.21.15.46:35270) 2023-10-30T14:01:56.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f778affd640 1 -- 172.21.15.46:0/2670115425 learned_addr learned my addr 172.21.15.46:0/2670115425 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:01:56.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f778b7fe640 1 --2- 172.21.15.46:0/2670115425 >> v2:172.21.15.46:3300/0 conn(0x7f778c1538f0 0x7f778c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:56.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f778bfff640 1 --2- 172.21.15.46:0/2670115425 >> v2:172.21.15.154:3300/0 conn(0x7f778c0cefa0 0x7f778c0cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:56.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f778a7fc640 1 -- 172.21.15.46:0/2670115425 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735034399 0 0) 0x7f778c0ca6a0 con 0x7f778c15cd50 2023-10-30T14:01:56.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f778a7fc640 1 -- 172.21.15.46:0/2670115425 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7774003530 con 0x7f778c15cd50 2023-10-30T14:01:56.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f778b7fe640 1 -- 172.21.15.46:0/2670115425 >> v2:172.21.15.154:3300/0 conn(0x7f778c0cefa0 msgr2=0x7f778c0cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:56.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f778b7fe640 1 --2- 172.21.15.46:0/2670115425 >> v2:172.21.15.154:3300/0 conn(0x7f778c0cefa0 0x7f778c0cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:56.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f778b7fe640 1 -- 172.21.15.46:0/2670115425 >> v1:172.21.15.46:6789/0 conn(0x7f778c15cd50 legacy=0x7f778c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:56.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.266+0000 7f778b7fe640 1 -- 172.21.15.46:0/2670115425 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c15d5e0 con 0x7f778c1538f0 2023-10-30T14:01:56.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.270+0000 7f778b7fe640 1 --2- 172.21.15.46:0/2670115425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c1538f0 0x7f778c159fb0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f778c066a40 tx=0x7f777c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3fccf67a56c5c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:56.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.270+0000 7f778a7fc640 1 -- 172.21.15.46:0/2670115425 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c0a8950 con 0x7f778c1538f0 2023-10-30T14:01:56.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.274+0000 7f778a7fc640 1 -- 172.21.15.46:0/2670115425 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777c0a8ab0 con 0x7f778c1538f0 2023-10-30T14:01:56.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.274+0000 7f778a7fc640 1 -- 172.21.15.46:0/2670115425 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c0a8e20 con 0x7f778c1538f0 2023-10-30T14:01:56.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.274+0000 7f7790a74640 1 -- 172.21.15.46:0/2670115425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c1538f0 msgr2=0x7f778c159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:56.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.274+0000 7f7790a74640 1 --2- 172.21.15.46:0/2670115425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c1538f0 0x7f778c159fb0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f778c066a40 tx=0x7f777c01ec50 comp rx=0 tx=0).stop 2023-10-30T14:01:56.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.274+0000 7f7790a74640 1 -- 172.21.15.46:0/2670115425 shutdown_connections 2023-10-30T14:01:56.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.274+0000 7f7790a74640 1 --2- 172.21.15.46:0/2670115425 >> v2:172.21.15.154:3300/0 conn(0x7f778c0cefa0 0x7f778c0cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:56.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.274+0000 7f7790a74640 1 --2- 172.21.15.46:0/2670115425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c1538f0 0x7f778c159fb0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:56.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.274+0000 7f7790a74640 1 -- 172.21.15.46:0/2670115425 >> 172.21.15.46:0/2670115425 conn(0x7f778c06a640 msgr2=0x7f778c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.274+0000 7f7790a74640 1 -- 172.21.15.46:0/2670115425 shutdown_connections 2023-10-30T14:01:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.274+0000 7f7790a74640 1 -- 172.21.15.46:0/2670115425 wait complete. 2023-10-30T14:01:56.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f7790a74640 1 Processor -- start 2023-10-30T14:01:56.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f7790a74640 1 -- start start 2023-10-30T14:01:56.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f7790a74640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c0cefa0 0x7f778c1467d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:56.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f7790a74640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c1538f0 0x7f778c146d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:56.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f7790a74640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f778c15cd50 0x7f778c1e4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:56.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f7790a74640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c15be20 con 0x7f778c1538f0 2023-10-30T14:01:56.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f7790a74640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c1e3cf0 con 0x7f778c0cefa0 2023-10-30T14:01:56.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f7790a74640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778c15aaa0 con 0x7f778c15cd50 2023-10-30T14:01:56.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f778affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c1538f0 0x7f778c146d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:56.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f778affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c1538f0 0x7f778c146d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56980/0 (socket says 172.21.15.46:56980) 2023-10-30T14:01:56.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f778affd640 1 -- 172.21.15.46:0/3588387729 learned_addr learned my addr 172.21.15.46:0/3588387729 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:56.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f778bfff640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f778c15cd50 0x7f778c1e4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:56.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f778b7fe640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c0cefa0 0x7f778c1467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:56.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f778affd640 1 -- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f778c15cd50 msgr2=0x7f778c1e4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:56.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f778affd640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f778c15cd50 0x7f778c1e4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:56.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f778affd640 1 -- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c0cefa0 msgr2=0x7f778c1467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:56.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f778affd640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c0cefa0 0x7f778c1467d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:56.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f778affd640 1 -- 172.21.15.46:0/3588387729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c009420 con 0x7f778c1538f0 2023-10-30T14:01:56.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.278+0000 7f778affd640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c1538f0 0x7f778c146d10 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f778000c450 tx=0x7f778000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:56.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.282+0000 7f7773fff640 1 -- 172.21.15.46:0/3588387729 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7780005cf0 con 0x7f778c1538f0 2023-10-30T14:01:56.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.282+0000 7f7773fff640 1 -- 172.21.15.46:0/3588387729 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7780005e50 con 0x7f778c1538f0 2023-10-30T14:01:56.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.282+0000 7f7773fff640 1 -- 172.21.15.46:0/3588387729 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7780007270 con 0x7f778c1538f0 2023-10-30T14:01:56.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.282+0000 7f7790a74640 1 -- 172.21.15.46:0/3588387729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778c15ad40 con 0x7f778c1538f0 2023-10-30T14:01:56.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.282+0000 7f7790a74640 1 -- 172.21.15.46:0/3588387729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778c15b2e0 con 0x7f778c1538f0 2023-10-30T14:01:56.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.282+0000 7f7790a74640 1 -- 172.21.15.46:0/3588387729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778c04e170 con 0x7f778c1538f0 2023-10-30T14:01:56.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.282+0000 7f7773fff640 1 -- 172.21.15.46:0/3588387729 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f77800073d0 con 0x7f778c1538f0 2023-10-30T14:01:56.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.282+0000 7f7773fff640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f77780664a0 0x7f7778068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:56.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.286+0000 7f7773fff640 1 -- 172.21.15.46:0/3588387729 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f7780083ae0 con 0x7f778c1538f0 2023-10-30T14:01:56.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.286+0000 7f778b7fe640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f77780664a0 0x7f7778068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:56.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.286+0000 7f778b7fe640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f77780664a0 0x7f7778068960 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f777c0b2000 tx=0x7f777c0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:56.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.290+0000 7f7773fff640 1 -- 172.21.15.46:0/3588387729 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7780050830 con 0x7f778c1538f0 2023-10-30T14:01:56.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.494+0000 7f7790a74640 1 -- 172.21.15.46:0/3588387729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f778c14d980 con 0x7f778c1538f0 2023-10-30T14:01:56.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.494+0000 7f7773fff640 1 -- 172.21.15.46:0/3588387729 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f7780017020 con 0x7f778c1538f0 2023-10-30T14:01:56.499 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:56.499 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:49.404315+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:56.499 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 167 2023-10-30T14:01:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.498+0000 7f7790a74640 1 -- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f77780664a0 msgr2=0x7f7778068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.498+0000 7f7790a74640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f77780664a0 0x7f7778068960 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f777c0b2000 tx=0x7f777c0aa070 comp rx=0 tx=0).stop 2023-10-30T14:01:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.498+0000 7f7790a74640 1 -- 172.21.15.46:0/3588387729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c1538f0 msgr2=0x7f778c146d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.498+0000 7f7790a74640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c1538f0 0x7f778c146d10 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f778000c450 tx=0x7f778000c910 comp rx=0 tx=0).stop 2023-10-30T14:01:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.498+0000 7f7790a74640 1 -- 172.21.15.46:0/3588387729 shutdown_connections 2023-10-30T14:01:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.498+0000 7f7790a74640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f77780664a0 0x7f7778068960 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.498+0000 7f7790a74640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f778c15cd50 0x7f778c1e4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.498+0000 7f7790a74640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c1538f0 0x7f778c146d10 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.498+0000 7f7790a74640 1 --2- 172.21.15.46:0/3588387729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c0cefa0 0x7f778c1467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.498+0000 7f7790a74640 1 -- 172.21.15.46:0/3588387729 >> 172.21.15.46:0/3588387729 conn(0x7f778c06a640 msgr2=0x7f778c0c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.498+0000 7f7790a74640 1 -- 172.21.15.46:0/3588387729 shutdown_connections 2023-10-30T14:01:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:56.498+0000 7f7790a74640 1 -- 172.21.15.46:0/3588387729 wait complete. 2023-10-30T14:01:56.521 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:01:58.522 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:01:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.670+0000 7fa2a22ea640 1 Processor -- start 2023-10-30T14:01:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.670+0000 7fa2a22ea640 1 -- start start 2023-10-30T14:01:58.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa2a22ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa29c15d6f0 0x7fa29c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:58.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa2a22ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa29c0cf160 0x7fa29c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:58.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa2a22ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29c0cb800 con 0x7fa29c1538a0 2023-10-30T14:01:58.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa2a22ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa29c0cb330 con 0x7fa29c0cf160 2023-10-30T14:01:58.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa2a22ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0cad70 con 0x7fa29c15d6f0 2023-10-30T14:01:58.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa29bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa29c1538a0 0x7fa29c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59472/0 (socket says 172.21.15.46:59472) 2023-10-30T14:01:58.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa29bfff640 1 -- 172.21.15.46:0/2348167203 learned_addr learned my addr 172.21.15.46:0/2348167203 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:01:58.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa29b7fe640 1 --2- 172.21.15.46:0/2348167203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa29c15d6f0 0x7fa29c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:58.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa2a0887640 1 --2- 172.21.15.46:0/2348167203 >> v2:172.21.15.154:3300/0 conn(0x7fa29c0cf160 0x7fa29c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:58.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa29affd640 1 -- 172.21.15.46:0/2348167203 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407209848 0 0) 0x7fa29c0cb800 con 0x7fa29c1538a0 2023-10-30T14:01:58.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa29affd640 1 -- 172.21.15.46:0/2348167203 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa284003530 con 0x7fa29c1538a0 2023-10-30T14:01:58.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa29b7fe640 1 -- 172.21.15.46:0/2348167203 >> v2:172.21.15.154:3300/0 conn(0x7fa29c0cf160 msgr2=0x7fa29c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:01:58.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa29b7fe640 1 --2- 172.21.15.46:0/2348167203 >> v2:172.21.15.154:3300/0 conn(0x7fa29c0cf160 0x7fa29c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:58.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa29b7fe640 1 -- 172.21.15.46:0/2348167203 >> v1:172.21.15.46:6789/0 conn(0x7fa29c1538a0 legacy=0x7fa29c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:58.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.674+0000 7fa29b7fe640 1 -- 172.21.15.46:0/2348167203 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c0cb050 con 0x7fa29c15d6f0 2023-10-30T14:01:58.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.678+0000 7fa29b7fe640 1 --2- 172.21.15.46:0/2348167203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa29c15d6f0 0x7fa29c15a090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa2900097c0 tx=0x7fa29001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=def583b695622d17 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:58.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.678+0000 7fa29affd640 1 -- 172.21.15.46:0/2348167203 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900a8970 con 0x7fa29c15d6f0 2023-10-30T14:01:58.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.678+0000 7fa29affd640 1 -- 172.21.15.46:0/2348167203 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2900a8ad0 con 0x7fa29c15d6f0 2023-10-30T14:01:58.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.678+0000 7fa29affd640 1 -- 172.21.15.46:0/2348167203 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900a8e60 con 0x7fa29c15d6f0 2023-10-30T14:01:58.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.678+0000 7fa2a22ea640 1 -- 172.21.15.46:0/2348167203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa29c15d6f0 msgr2=0x7fa29c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:58.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.678+0000 7fa2a22ea640 1 --2- 172.21.15.46:0/2348167203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa29c15d6f0 0x7fa29c15a090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa2900097c0 tx=0x7fa29001ec70 comp rx=0 tx=0).stop 2023-10-30T14:01:58.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.678+0000 7fa2a22ea640 1 -- 172.21.15.46:0/2348167203 shutdown_connections 2023-10-30T14:01:58.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.678+0000 7fa2a22ea640 1 --2- 172.21.15.46:0/2348167203 >> v2:172.21.15.154:3300/0 conn(0x7fa29c0cf160 0x7fa29c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:58.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.678+0000 7fa2a22ea640 1 --2- 172.21.15.46:0/2348167203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa29c15d6f0 0x7fa29c15a090 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:58.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.678+0000 7fa2a22ea640 1 -- 172.21.15.46:0/2348167203 >> 172.21.15.46:0/2348167203 conn(0x7fa29c06bfe0 msgr2=0x7fa29c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:58.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.678+0000 7fa2a22ea640 1 -- 172.21.15.46:0/2348167203 shutdown_connections 2023-10-30T14:01:58.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.678+0000 7fa2a22ea640 1 -- 172.21.15.46:0/2348167203 wait complete. 2023-10-30T14:01:58.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a22ea640 1 Processor -- start 2023-10-30T14:01:58.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a22ea640 1 -- start start 2023-10-30T14:01:58.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a22ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa29c0cf160 0x7fa29c14f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:58.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a22ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa29c1538a0 0x7fa29c14e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:58.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a22ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa29c15d6f0 0x7fa29c0b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:58.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a22ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0b41f0 con 0x7fa29c15d6f0 2023-10-30T14:01:58.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a22ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c14eb50 con 0x7fa29c1538a0 2023-10-30T14:01:58.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a22ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c150500 con 0x7fa29c0cf160 2023-10-30T14:01:58.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a0887640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa29c15d6f0 0x7fa29c0b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:58.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a0887640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa29c15d6f0 0x7fa29c0b3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41722/0 (socket says 172.21.15.46:41722) 2023-10-30T14:01:58.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a0887640 1 -- 172.21.15.46:0/3605553005 learned_addr learned my addr 172.21.15.46:0/3605553005 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:01:58.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa29bfff640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa29c0cf160 0x7fa29c14f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa29b7fe640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa29c1538a0 0x7fa29c14e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a0887640 1 -- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa29c0cf160 msgr2=0x7fa29c14f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a0887640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa29c0cf160 0x7fa29c14f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a0887640 1 -- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa29c1538a0 msgr2=0x7fa29c14e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:58.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a0887640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa29c1538a0 0x7fa29c14e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:58.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.682+0000 7fa2a0887640 1 -- 172.21.15.46:0/3605553005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa290009420 con 0x7fa29c15d6f0 2023-10-30T14:01:58.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.686+0000 7fa2a0887640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa29c15d6f0 0x7fa29c0b3370 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fa29400efc0 tx=0x7fa29400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:58.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.686+0000 7fa298ff9640 1 -- 172.21.15.46:0/3605553005 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa294009b10 con 0x7fa29c15d6f0 2023-10-30T14:01:58.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.686+0000 7fa298ff9640 1 -- 172.21.15.46:0/3605553005 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29400f040 con 0x7fa29c15d6f0 2023-10-30T14:01:58.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.686+0000 7fa298ff9640 1 -- 172.21.15.46:0/3605553005 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2940146c0 con 0x7fa29c15d6f0 2023-10-30T14:01:58.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.686+0000 7fa2a22ea640 1 -- 172.21.15.46:0/3605553005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29c1507e0 con 0x7fa29c15d6f0 2023-10-30T14:01:58.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.686+0000 7fa2a22ea640 1 -- 172.21.15.46:0/3605553005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29c0b3e80 con 0x7fa29c15d6f0 2023-10-30T14:01:58.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.686+0000 7fa2a22ea640 1 -- 172.21.15.46:0/3605553005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29c0b2a50 con 0x7fa29c15d6f0 2023-10-30T14:01:58.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.686+0000 7fa298ff9640 1 -- 172.21.15.46:0/3605553005 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa294014820 con 0x7fa29c15d6f0 2023-10-30T14:01:58.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.686+0000 7fa298ff9640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa2880664a0 0x7fa288068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:01:58.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.686+0000 7fa298ff9640 1 -- 172.21.15.46:0/3605553005 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7fa2940853b0 con 0x7fa29c15d6f0 2023-10-30T14:01:58.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.690+0000 7fa29bfff640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa2880664a0 0x7fa288068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:01:58.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.690+0000 7fa29bfff640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa2880664a0 0x7fa288068960 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fa28c0011f0 tx=0x7fa28c007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:01:58.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.690+0000 7fa298ff9640 1 -- 172.21.15.46:0/3605553005 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa2940520f0 con 0x7fa29c15d6f0 2023-10-30T14:01:58.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.886+0000 7fa2a22ea640 1 -- 172.21.15.46:0/3605553005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa29c0b43a0 con 0x7fa29c15d6f0 2023-10-30T14:01:58.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.886+0000 7fa298ff9640 1 -- 172.21.15.46:0/3605553005 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5731 (secure 0 0 0) 0x7fa29c0b43a0 con 0x7fa29c15d6f0 2023-10-30T14:01:58.889 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:01:58.890 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:49.404315+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:01:58.890 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 167 2023-10-30T14:01:58.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.890+0000 7fa2a22ea640 1 -- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa2880664a0 msgr2=0x7fa288068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:58.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.890+0000 7fa2a22ea640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa2880664a0 0x7fa288068960 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fa28c0011f0 tx=0x7fa28c007d40 comp rx=0 tx=0).stop 2023-10-30T14:01:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.890+0000 7fa2a22ea640 1 -- 172.21.15.46:0/3605553005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa29c15d6f0 msgr2=0x7fa29c0b3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:01:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.890+0000 7fa2a22ea640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa29c15d6f0 0x7fa29c0b3370 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fa29400efc0 tx=0x7fa29400c490 comp rx=0 tx=0).stop 2023-10-30T14:01:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.890+0000 7fa2a22ea640 1 -- 172.21.15.46:0/3605553005 shutdown_connections 2023-10-30T14:01:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.890+0000 7fa2a22ea640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa2880664a0 0x7fa288068960 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.890+0000 7fa2a22ea640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa29c15d6f0 0x7fa29c0b3370 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.890+0000 7fa2a22ea640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa29c1538a0 0x7fa29c14e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.890+0000 7fa2a22ea640 1 --2- 172.21.15.46:0/3605553005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa29c0cf160 0x7fa29c14f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:01:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.890+0000 7fa2a22ea640 1 -- 172.21.15.46:0/3605553005 >> 172.21.15.46:0/3605553005 conn(0x7fa29c06bfe0 msgr2=0x7fa29c0ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:01:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.890+0000 7fa2a22ea640 1 -- 172.21.15.46:0/3605553005 shutdown_connections 2023-10-30T14:01:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:01:58.890+0000 7fa2a22ea640 1 -- 172.21.15.46:0/3605553005 wait complete. 2023-10-30T14:01:58.913 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:02:00.914 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:02:01.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.066+0000 7f8d6b594640 1 Processor -- start 2023-10-30T14:02:01.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.066+0000 7f8d6b594640 1 -- start start 2023-10-30T14:02:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.066+0000 7f8d6b594640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8d640c6b80 0x7f8d640bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.066+0000 7f8d6b594640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d640c6f60 0x7f8d640c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.066+0000 7f8d6b594640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8d640c8430 0x7f8d6413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.066+0000 7f8d6b594640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d640c0a50 con 0x7f8d640c6b80 2023-10-30T14:02:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.066+0000 7f8d6b594640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d6406f6e0 con 0x7f8d640c8430 2023-10-30T14:02:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.066+0000 7f8d6b594640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6406f9c0 con 0x7f8d640c6f60 2023-10-30T14:02:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d69330640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8d640c6b80 0x7f8d640bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d69330640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8d640c6b80 0x7f8d640bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41736/0 (socket says 172.21.15.46:41736) 2023-10-30T14:02:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d69330640 1 -- 172.21.15.46:0/2365413900 learned_addr learned my addr 172.21.15.46:0/2365413900 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d68b2f640 1 --2- 172.21.15.46:0/2365413900 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d640c6f60 0x7f8d640c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d69b31640 1 --2- 172.21.15.46:0/2365413900 >> v2:172.21.15.154:3300/0 conn(0x7f8d640c8430 0x7f8d6413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d69330640 1 -- 172.21.15.46:0/2365413900 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d640c6f60 msgr2=0x7f8d640c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d69330640 1 --2- 172.21.15.46:0/2365413900 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d640c6f60 0x7f8d640c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d69330640 1 -- 172.21.15.46:0/2365413900 >> v2:172.21.15.154:3300/0 conn(0x7f8d640c8430 msgr2=0x7f8d6413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d69330640 1 --2- 172.21.15.46:0/2365413900 >> v2:172.21.15.154:3300/0 conn(0x7f8d640c8430 0x7f8d6413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d69330640 1 -- 172.21.15.46:0/2365413900 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6406fca0 con 0x7f8d640c6b80 2023-10-30T14:02:01.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d69b31640 1 --2- 172.21.15.46:0/2365413900 >> v2:172.21.15.154:3300/0 conn(0x7f8d640c8430 0x7f8d6413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:02:01.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d69330640 1 --2- 172.21.15.46:0/2365413900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d640c6b80 0x7f8d640bfec0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f8d640b0860 tx=0x7f8d6001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f261ed0b2c9ef0f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:01.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.070+0000 7f8d5bfff640 1 -- 172.21.15.46:0/2365413900 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d600ab600 con 0x7f8d640c6b80 2023-10-30T14:02:01.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d5bfff640 1 -- 172.21.15.46:0/2365413900 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d600a7050 con 0x7f8d640c6b80 2023-10-30T14:02:01.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d5bfff640 1 -- 172.21.15.46:0/2365413900 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d600ab980 con 0x7f8d640c6b80 2023-10-30T14:02:01.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 -- 172.21.15.46:0/2365413900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d640c6b80 msgr2=0x7f8d640bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:01.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 --2- 172.21.15.46:0/2365413900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d640c6b80 0x7f8d640bfec0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f8d640b0860 tx=0x7f8d6001c470 comp rx=0 tx=0).stop 2023-10-30T14:02:01.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 -- 172.21.15.46:0/2365413900 shutdown_connections 2023-10-30T14:02:01.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 --2- 172.21.15.46:0/2365413900 >> v2:172.21.15.154:3300/0 conn(0x7f8d640c8430 0x7f8d6413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:01.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 --2- 172.21.15.46:0/2365413900 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d640c6f60 0x7f8d640c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:01.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 --2- 172.21.15.46:0/2365413900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d640c6b80 0x7f8d640bfec0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:01.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 -- 172.21.15.46:0/2365413900 >> 172.21.15.46:0/2365413900 conn(0x7f8d640c7910 msgr2=0x7f8d640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:01.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 -- 172.21.15.46:0/2365413900 shutdown_connections 2023-10-30T14:02:01.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 -- 172.21.15.46:0/2365413900 wait complete. 2023-10-30T14:02:01.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 Processor -- start 2023-10-30T14:02:01.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 -- start start 2023-10-30T14:02:01.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d640c6b80 0x7f8d641579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:01.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d640c6f60 0x7f8d641f52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:01.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d640c8430 0x7f8d641f5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d64160dd0 con 0x7f8d640c6f60 2023-10-30T14:02:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d64157390 con 0x7f8d640c8430 2023-10-30T14:02:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d6b594640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d641593a0 con 0x7f8d640c6b80 2023-10-30T14:02:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d68b2f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d640c6f60 0x7f8d641f52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d68b2f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d640c6f60 0x7f8d641f52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41738/0 (socket says 172.21.15.46:41738) 2023-10-30T14:02:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d69330640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d640c6b80 0x7f8d641579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d68b2f640 1 -- 172.21.15.46:0/3125953642 learned_addr learned my addr 172.21.15.46:0/3125953642 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.074+0000 7f8d69b31640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d640c8430 0x7f8d641f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d68b2f640 1 -- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d640c6b80 msgr2=0x7f8d641579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d68b2f640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d640c6b80 0x7f8d641579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d68b2f640 1 -- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d640c8430 msgr2=0x7f8d641f5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d68b2f640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d640c8430 0x7f8d641f5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d68b2f640 1 -- 172.21.15.46:0/3125953642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d600094d0 con 0x7f8d640c6f60 2023-10-30T14:02:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d68b2f640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d640c6f60 0x7f8d641f52f0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400d800 tx=0x7f8d5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d59ffb640 1 -- 172.21.15.46:0/3125953642 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54009640 con 0x7f8d640c6f60 2023-10-30T14:02:01.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d59ffb640 1 -- 172.21.15.46:0/3125953642 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d540097a0 con 0x7f8d640c6f60 2023-10-30T14:02:01.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d59ffb640 1 -- 172.21.15.46:0/3125953642 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54009b10 con 0x7f8d640c6f60 2023-10-30T14:02:01.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d6b594640 1 -- 172.21.15.46:0/3125953642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6412fa60 con 0x7f8d640c6f60 2023-10-30T14:02:01.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d6b594640 1 -- 172.21.15.46:0/3125953642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d64160af0 con 0x7f8d640c6f60 2023-10-30T14:02:01.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d6b594640 1 -- 172.21.15.46:0/3125953642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6412f270 con 0x7f8d640c6f60 2023-10-30T14:02:01.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d59ffb640 1 -- 172.21.15.46:0/3125953642 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8d54009c70 con 0x7f8d640c6f60 2023-10-30T14:02:01.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.078+0000 7f8d59ffb640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d480664a0 0x7f8d48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:01.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.082+0000 7f8d59ffb640 1 -- 172.21.15.46:0/3125953642 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f8d54084c00 con 0x7f8d640c6f60 2023-10-30T14:02:01.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.082+0000 7f8d69330640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d480664a0 0x7f8d48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:01.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.082+0000 7f8d69330640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d480664a0 0x7f8d48068960 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8d6406f1b0 tx=0x7f8d600a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:01.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.086+0000 7f8d59ffb640 1 -- 172.21.15.46:0/3125953642 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8d6412f270 con 0x7f8d640c6f60 2023-10-30T14:02:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.274+0000 7f8d6b594640 1 -- 172.21.15.46:0/3125953642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d6412f270 con 0x7f8d640c6f60 2023-10-30T14:02:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.274+0000 7f8d59ffb640 1 -- 172.21.15.46:0/3125953642 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f8d6412f270 con 0x7f8d640c6f60 2023-10-30T14:02:01.278 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:02:01.278 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:59.837709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:resolve","state_seq":14,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:02:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 168 2023-10-30T14:02:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.278+0000 7f8d6b594640 1 -- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d480664a0 msgr2=0x7f8d48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.278+0000 7f8d6b594640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d480664a0 0x7f8d48068960 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8d6406f1b0 tx=0x7f8d600a6000 comp rx=0 tx=0).stop 2023-10-30T14:02:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.278+0000 7f8d6b594640 1 -- 172.21.15.46:0/3125953642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d640c6f60 msgr2=0x7f8d641f52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.278+0000 7f8d6b594640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d640c6f60 0x7f8d641f52f0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400d800 tx=0x7f8d5400dcc0 comp rx=0 tx=0).stop 2023-10-30T14:02:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.278+0000 7f8d6b594640 1 -- 172.21.15.46:0/3125953642 shutdown_connections 2023-10-30T14:02:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.278+0000 7f8d6b594640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8d480664a0 0x7f8d48068960 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.278+0000 7f8d6b594640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d640c8430 0x7f8d641f5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.278+0000 7f8d6b594640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d640c6f60 0x7f8d641f52f0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.278+0000 7f8d6b594640 1 --2- 172.21.15.46:0/3125953642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d640c6b80 0x7f8d641579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.278+0000 7f8d6b594640 1 -- 172.21.15.46:0/3125953642 >> 172.21.15.46:0/3125953642 conn(0x7f8d640c7910 msgr2=0x7f8d640c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.278+0000 7f8d6b594640 1 -- 172.21.15.46:0/3125953642 shutdown_connections 2023-10-30T14:02:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:01.278+0000 7f8d6b594640 1 -- 172.21.15.46:0/3125953642 wait complete. 2023-10-30T14:02:01.299 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:02:03.300 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:02:03.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f986d3f5640 1 Processor -- start 2023-10-30T14:02:03.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f986d3f5640 1 -- start start 2023-10-30T14:02:03.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f986d3f5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98681538a0 0x7f986815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f986d3f5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98680ca0e0 0x7f986815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f986d3f5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98680cb800 con 0x7f98680cf160 2023-10-30T14:02:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f986d3f5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f986815dc30 con 0x7f98680ca0e0 2023-10-30T14:02:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f986d3f5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98680cb330 con 0x7f98681538a0 2023-10-30T14:02:03.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f985ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98680ca0e0 0x7f986815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:03.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f985ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98680ca0e0 0x7f986815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41752/0 (socket says 172.21.15.46:41752) 2023-10-30T14:02:03.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f985ffff640 1 -- 172.21.15.46:0/4268974805 learned_addr learned my addr 172.21.15.46:0/4268974805 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:03.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f9866ffd640 1 --2- 172.21.15.46:0/4268974805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98681538a0 0x7f986815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:03.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f985ffff640 1 -- 172.21.15.46:0/4268974805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98681538a0 msgr2=0x7f986815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:03.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f985ffff640 1 --2- 172.21.15.46:0/4268974805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98681538a0 0x7f986815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:03.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f985ffff640 1 -- 172.21.15.46:0/4268974805 >> v1:172.21.15.154:6789/0 conn(0x7f98680cf160 legacy=0x7f98680cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:03.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.454+0000 7f985ffff640 1 -- 172.21.15.46:0/4268974805 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98680cad70 con 0x7f98680ca0e0 2023-10-30T14:02:03.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.458+0000 7f985ffff640 1 --2- 172.21.15.46:0/4268974805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98680ca0e0 0x7f986815a090 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f9850009870 tx=0x7f985001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e65229ee9f5bc91 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:03.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.458+0000 7f98667fc640 1 -- 172.21.15.46:0/4268974805 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98500a89b0 con 0x7f98680ca0e0 2023-10-30T14:02:03.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.458+0000 7f98667fc640 1 -- 172.21.15.46:0/4268974805 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98500a8b10 con 0x7f98680ca0e0 2023-10-30T14:02:03.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.458+0000 7f98667fc640 1 -- 172.21.15.46:0/4268974805 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98500a8ea0 con 0x7f98680ca0e0 2023-10-30T14:02:03.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.458+0000 7f986d3f5640 1 -- 172.21.15.46:0/4268974805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98680ca0e0 msgr2=0x7f986815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:03.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.458+0000 7f986d3f5640 1 --2- 172.21.15.46:0/4268974805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98680ca0e0 0x7f986815a090 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f9850009870 tx=0x7f985001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:02:03.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.458+0000 7f986d3f5640 1 -- 172.21.15.46:0/4268974805 shutdown_connections 2023-10-30T14:02:03.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.458+0000 7f986d3f5640 1 --2- 172.21.15.46:0/4268974805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98680ca0e0 0x7f986815a090 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:03.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.458+0000 7f986d3f5640 1 --2- 172.21.15.46:0/4268974805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98681538a0 0x7f986815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:03.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.458+0000 7f986d3f5640 1 -- 172.21.15.46:0/4268974805 >> 172.21.15.46:0/4268974805 conn(0x7f986806bfe0 msgr2=0x7f986806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:03.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.458+0000 7f986d3f5640 1 -- 172.21.15.46:0/4268974805 shutdown_connections 2023-10-30T14:02:03.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.458+0000 7f986d3f5640 1 -- 172.21.15.46:0/4268974805 wait complete. 2023-10-30T14:02:03.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f986d3f5640 1 Processor -- start 2023-10-30T14:02:03.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f986d3f5640 1 -- start start 2023-10-30T14:02:03.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f986d3f5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98680ca0e0 0x7f98681e3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:03.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f986d3f5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98680cf160 0x7f98681ef1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:03.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f986d3f5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98681538a0 0x7f98681ef6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:03.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f986d3f5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9868159700 con 0x7f98680cf160 2023-10-30T14:02:03.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f986d3f5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98681f3670 con 0x7f98681538a0 2023-10-30T14:02:03.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f986d3f5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9868159160 con 0x7f98680ca0e0 2023-10-30T14:02:03.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f985ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98680cf160 0x7f98681ef1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:03.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f985ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98680cf160 0x7f98681ef1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41764/0 (socket says 172.21.15.46:41764) 2023-10-30T14:02:03.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f985ffff640 1 -- 172.21.15.46:0/1471599446 learned_addr learned my addr 172.21.15.46:0/1471599446 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:03.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f9866ffd640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98680ca0e0 0x7f98681e3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:03.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f98677fe640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98681538a0 0x7f98681ef6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:03.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f985ffff640 1 -- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98680ca0e0 msgr2=0x7f98681e3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:03.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f985ffff640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98680ca0e0 0x7f98681e3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:03.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f985ffff640 1 -- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98681538a0 msgr2=0x7f98681ef6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:03.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f985ffff640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98681538a0 0x7f98681ef6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:03.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f985ffff640 1 -- 172.21.15.46:0/1471599446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98500094d0 con 0x7f98680cf160 2023-10-30T14:02:03.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f98677fe640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98681538a0 0x7f98681ef6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:02:03.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f985ffff640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98680cf160 0x7f98681ef1a0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f98500b3040 tx=0x7f985001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:03.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f985f7fe640 1 -- 172.21.15.46:0/1471599446 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98500a89b0 con 0x7f98680cf160 2023-10-30T14:02:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f985f7fe640 1 -- 172.21.15.46:0/1471599446 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98500bb480 con 0x7f98680cf160 2023-10-30T14:02:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f985f7fe640 1 -- 172.21.15.46:0/1471599446 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98500bb830 con 0x7f98680cf160 2023-10-30T14:02:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f986d3f5640 1 -- 172.21.15.46:0/1471599446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9868152f00 con 0x7f98680cf160 2023-10-30T14:02:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.462+0000 7f986d3f5640 1 -- 172.21.15.46:0/1471599446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98681f28b0 con 0x7f98680cf160 2023-10-30T14:02:03.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.466+0000 7f986d3f5640 1 -- 172.21.15.46:0/1471599446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98680b2a50 con 0x7f98680cf160 2023-10-30T14:02:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.466+0000 7f985f7fe640 1 -- 172.21.15.46:0/1471599446 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f98500bb990 con 0x7f98680cf160 2023-10-30T14:02:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.466+0000 7f985f7fe640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98480664a0 0x7f9848068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.466+0000 7f985f7fe640 1 -- 172.21.15.46:0/1471599446 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f9850122d70 con 0x7f98680cf160 2023-10-30T14:02:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.466+0000 7f9866ffd640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98480664a0 0x7f9848068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:03.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.466+0000 7f9866ffd640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98480664a0 0x7f9848068960 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f985800b930 tx=0x7f9858008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:03.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.470+0000 7f985f7fe640 1 -- 172.21.15.46:0/1471599446 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f98500ef970 con 0x7f98680cf160 2023-10-30T14:02:03.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.662+0000 7f986d3f5640 1 -- 172.21.15.46:0/1471599446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98680b2a50 con 0x7f98680cf160 2023-10-30T14:02:03.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.662+0000 7f985f7fe640 1 -- 172.21.15.46:0/1471599446 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f98500b5020 con 0x7f98680cf160 2023-10-30T14:02:03.666 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:02:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:01:59.837709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:resolve","state_seq":14,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:02:03.666 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 168 2023-10-30T14:02:03.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.662+0000 7f986d3f5640 1 -- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98480664a0 msgr2=0x7f9848068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:03.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.662+0000 7f986d3f5640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98480664a0 0x7f9848068960 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f985800b930 tx=0x7f9858008040 comp rx=0 tx=0).stop 2023-10-30T14:02:03.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.662+0000 7f986d3f5640 1 -- 172.21.15.46:0/1471599446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98680cf160 msgr2=0x7f98681ef1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:03.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.662+0000 7f986d3f5640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98680cf160 0x7f98681ef1a0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f98500b3040 tx=0x7f985001ba90 comp rx=0 tx=0).stop 2023-10-30T14:02:03.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.662+0000 7f986d3f5640 1 -- 172.21.15.46:0/1471599446 shutdown_connections 2023-10-30T14:02:03.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.662+0000 7f986d3f5640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98480664a0 0x7f9848068960 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:03.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.662+0000 7f986d3f5640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98681538a0 0x7f98681ef6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:03.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.662+0000 7f986d3f5640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98680cf160 0x7f98681ef1a0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:03.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.662+0000 7f986d3f5640 1 --2- 172.21.15.46:0/1471599446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98680ca0e0 0x7f98681e3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:03.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.662+0000 7f986d3f5640 1 -- 172.21.15.46:0/1471599446 >> 172.21.15.46:0/1471599446 conn(0x7f986806bfe0 msgr2=0x7f98680c5590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:03.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.666+0000 7f986d3f5640 1 -- 172.21.15.46:0/1471599446 shutdown_connections 2023-10-30T14:02:03.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:03.666+0000 7f986d3f5640 1 -- 172.21.15.46:0/1471599446 wait complete. 2023-10-30T14:02:03.687 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 168, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 168, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7704": { "addr": "172.21.15.46:6836/3734066164", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3734066164, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3734066164, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7704, "incarnation": 152, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_8034": { "addr": "172.21.15.46:6833/2075693104", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2075693104, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2075693104, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8034, "incarnation": 167, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:resolve", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:01:59.837709+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7704, "mds_1": 8034 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:02:05.687 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:02:05.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.830+0000 7fb0ce025640 1 Processor -- start 2023-10-30T14:02:05.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.830+0000 7fb0ce025640 1 -- start start 2023-10-30T14:02:05.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0ce025640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb0c80c05d0 0x7fb0c813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0ce025640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c80c6f60 0x7fb0c813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0ce025640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c80c0a50 con 0x7fb0c80c6b80 2023-10-30T14:02:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0ce025640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c806f6e0 con 0x7fb0c80c05d0 2023-10-30T14:02:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0ce025640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c806f9c0 con 0x7fb0c80c6f60 2023-10-30T14:02:05.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0c6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb0c80c05d0 0x7fb0c813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:05.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0c6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb0c80c05d0 0x7fb0c813ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41766/0 (socket says 172.21.15.46:41766) 2023-10-30T14:02:05.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0c6ffd640 1 -- 172.21.15.46:0/1867231597 learned_addr learned my addr 172.21.15.46:0/1867231597 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:05.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0c7fff640 1 --2- 172.21.15.46:0/1867231597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c80c6f60 0x7fb0c813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:05.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0c6ffd640 1 -- 172.21.15.46:0/1867231597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c80c6f60 msgr2=0x7fb0c813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0c6ffd640 1 --2- 172.21.15.46:0/1867231597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c80c6f60 0x7fb0c813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0c6ffd640 1 -- 172.21.15.46:0/1867231597 >> v1:172.21.15.154:6789/0 conn(0x7fb0c80c6b80 legacy=0x7fb0c80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0c6ffd640 1 -- 172.21.15.46:0/1867231597 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c806fca0 con 0x7fb0c80c05d0 2023-10-30T14:02:05.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.834+0000 7fb0c6ffd640 1 --2- 172.21.15.46:0/1867231597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c80c05d0 0x7fb0c813ed90 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009870 tx=0x7fb0bc01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c1a3c376c7ae23 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0c67fc640 1 -- 172.21.15.46:0/1867231597 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0ab650 con 0x7fb0c80c05d0 2023-10-30T14:02:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0c67fc640 1 -- 172.21.15.46:0/1867231597 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc0ab7b0 con 0x7fb0c80c05d0 2023-10-30T14:02:05.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0c67fc640 1 -- 172.21.15.46:0/1867231597 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0abb40 con 0x7fb0c80c05d0 2023-10-30T14:02:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 -- 172.21.15.46:0/1867231597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c80c05d0 msgr2=0x7fb0c813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:05.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 --2- 172.21.15.46:0/1867231597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c80c05d0 0x7fb0c813ed90 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009870 tx=0x7fb0bc01c540 comp rx=0 tx=0).stop 2023-10-30T14:02:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 -- 172.21.15.46:0/1867231597 shutdown_connections 2023-10-30T14:02:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 --2- 172.21.15.46:0/1867231597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c80c6f60 0x7fb0c813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 --2- 172.21.15.46:0/1867231597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c80c05d0 0x7fb0c813ed90 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 -- 172.21.15.46:0/1867231597 >> 172.21.15.46:0/1867231597 conn(0x7fb0c80c7910 msgr2=0x7fb0c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:05.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 -- 172.21.15.46:0/1867231597 shutdown_connections 2023-10-30T14:02:05.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 -- 172.21.15.46:0/1867231597 wait complete. 2023-10-30T14:02:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 Processor -- start 2023-10-30T14:02:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 -- start start 2023-10-30T14:02:05.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c80c05d0 0x7fb0c8155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c80c6b80 0x7fb0c81545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0c80c6f60 0x7fb0c81501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c813e800 con 0x7fb0c80c6b80 2023-10-30T14:02:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c81f2ff0 con 0x7fb0c80c6f60 2023-10-30T14:02:05.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0ce025640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c8154f90 con 0x7fb0c80c05d0 2023-10-30T14:02:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0c6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c80c6b80 0x7fb0c81545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0c6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c80c6b80 0x7fb0c81545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41782/0 (socket says 172.21.15.46:41782) 2023-10-30T14:02:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0c6ffd640 1 -- 172.21.15.46:0/423302255 learned_addr learned my addr 172.21.15.46:0/423302255 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.838+0000 7fb0c77fe640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c80c05d0 0x7fb0c8155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:05.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0c7fff640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0c80c6f60 0x7fb0c81501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0c6ffd640 1 -- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c80c05d0 msgr2=0x7fb0c8155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0c6ffd640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c80c05d0 0x7fb0c8155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0c6ffd640 1 -- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0c80c6f60 msgr2=0x7fb0c81501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0c6ffd640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0c80c6f60 0x7fb0c81501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:05.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0c6ffd640 1 -- 172.21.15.46:0/423302255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc0094d0 con 0x7fb0c80c6b80 2023-10-30T14:02:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0c6ffd640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c80c6b80 0x7fb0c81545b0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc01a170 tx=0x7fb0bc01ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0affff640 1 -- 172.21.15.46:0/423302255 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0aa980 con 0x7fb0c80c6b80 2023-10-30T14:02:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0affff640 1 -- 172.21.15.46:0/423302255 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc0aab00 con 0x7fb0c80c6b80 2023-10-30T14:02:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0affff640 1 -- 172.21.15.46:0/423302255 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0aaeb0 con 0x7fb0c80c6b80 2023-10-30T14:02:05.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0ce025640 1 -- 172.21.15.46:0/423302255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c8155270 con 0x7fb0c80c6b80 2023-10-30T14:02:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0ce025640 1 -- 172.21.15.46:0/423302255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c8155830 con 0x7fb0c80c6b80 2023-10-30T14:02:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0ce025640 1 -- 172.21.15.46:0/423302255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c804e170 con 0x7fb0c80c6b80 2023-10-30T14:02:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0affff640 1 -- 172.21.15.46:0/423302255 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb0bc0b43f0 con 0x7fb0c80c6b80 2023-10-30T14:02:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0affff640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0b40664a0 0x7fb0b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.842+0000 7fb0affff640 1 -- 172.21.15.46:0/423302255 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7fb0bc11f3d0 con 0x7fb0c80c6b80 2023-10-30T14:02:05.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.846+0000 7fb0c77fe640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0b40664a0 0x7fb0b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:05.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.846+0000 7fb0c77fe640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0b40664a0 0x7fb0b4068960 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8007fd0 tx=0x7fb0b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:05.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:05.850+0000 7fb0affff640 1 -- 172.21.15.46:0/423302255 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb0bc0ec110 con 0x7fb0c80c6b80 2023-10-30T14:02:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.038+0000 7fb0ce025640 1 -- 172.21.15.46:0/423302255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0c8071cc0 con 0x7fb0c80c6b80 2023-10-30T14:02:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.038+0000 7fb0affff640 1 -- 172.21.15.46:0/423302255 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5735 (secure 0 0 0) 0x7fb0bc0ef730 con 0x7fb0c80c6b80 2023-10-30T14:02:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:02:06.042 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:05.559049+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:02:06.043 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 169 2023-10-30T14:02:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.042+0000 7fb0ce025640 1 -- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0b40664a0 msgr2=0x7fb0b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.042+0000 7fb0ce025640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0b40664a0 0x7fb0b4068960 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8007fd0 tx=0x7fb0b8009040 comp rx=0 tx=0).stop 2023-10-30T14:02:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.042+0000 7fb0ce025640 1 -- 172.21.15.46:0/423302255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c80c6b80 msgr2=0x7fb0c81545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.042+0000 7fb0ce025640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c80c6b80 0x7fb0c81545b0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc01a170 tx=0x7fb0bc01ad80 comp rx=0 tx=0).stop 2023-10-30T14:02:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.042+0000 7fb0ce025640 1 -- 172.21.15.46:0/423302255 shutdown_connections 2023-10-30T14:02:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.042+0000 7fb0ce025640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0b40664a0 0x7fb0b4068960 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.042+0000 7fb0ce025640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0c80c6f60 0x7fb0c81501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.042+0000 7fb0ce025640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c80c6b80 0x7fb0c81545b0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.042+0000 7fb0ce025640 1 --2- 172.21.15.46:0/423302255 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c80c05d0 0x7fb0c8155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.042+0000 7fb0ce025640 1 -- 172.21.15.46:0/423302255 >> 172.21.15.46:0/423302255 conn(0x7fb0c80c7910 msgr2=0x7fb0c80bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.042+0000 7fb0ce025640 1 -- 172.21.15.46:0/423302255 shutdown_connections 2023-10-30T14:02:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:06.042+0000 7fb0ce025640 1 -- 172.21.15.46:0/423302255 wait complete. 2023-10-30T14:02:06.063 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:02:08.064 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:02:08.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4549268640 1 Processor -- start 2023-10-30T14:02:08.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4549268640 1 -- start start 2023-10-30T14:02:08.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4549268640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45441538a0 0x7f454415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:08.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4549268640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45440ca0e0 0x7f454415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:08.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4549268640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45440cb800 con 0x7f45440cf160 2023-10-30T14:02:08.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4549268640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f454415dc30 con 0x7f45441538a0 2023-10-30T14:02:08.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4549268640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45440cb330 con 0x7f45440ca0e0 2023-10-30T14:02:08.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4542ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45441538a0 0x7f454415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:08.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4542ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45441538a0 0x7f454415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59664/0 (socket says 172.21.15.46:59664) 2023-10-30T14:02:08.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4542ffd640 1 -- 172.21.15.46:0/4293915279 learned_addr learned my addr 172.21.15.46:0/4293915279 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:08.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f45427fc640 1 --2- 172.21.15.46:0/4293915279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45440ca0e0 0x7f454415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:08.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4542ffd640 1 -- 172.21.15.46:0/4293915279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45440ca0e0 msgr2=0x7f454415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:08.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4542ffd640 1 --2- 172.21.15.46:0/4293915279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45440ca0e0 0x7f454415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:08.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4542ffd640 1 -- 172.21.15.46:0/4293915279 >> v1:172.21.15.154:6789/0 conn(0x7f45440cf160 legacy=0x7f45440cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:08.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.218+0000 7f4542ffd640 1 -- 172.21.15.46:0/4293915279 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45440cad70 con 0x7f45441538a0 2023-10-30T14:02:08.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.222+0000 7f4542ffd640 1 --2- 172.21.15.46:0/4293915279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45441538a0 0x7f454415d6f0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f4534009870 tx=0x7f453401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6187b8252477d5fb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:08.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.222+0000 7f4541ffb640 1 -- 172.21.15.46:0/4293915279 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45340a8a30 con 0x7f45441538a0 2023-10-30T14:02:08.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.222+0000 7f4541ffb640 1 -- 172.21.15.46:0/4293915279 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45340a8b90 con 0x7f45441538a0 2023-10-30T14:02:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.222+0000 7f4541ffb640 1 -- 172.21.15.46:0/4293915279 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45340b0440 con 0x7f45441538a0 2023-10-30T14:02:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.222+0000 7f4549268640 1 -- 172.21.15.46:0/4293915279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45441538a0 msgr2=0x7f454415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.222+0000 7f4549268640 1 --2- 172.21.15.46:0/4293915279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45441538a0 0x7f454415d6f0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f4534009870 tx=0x7f453401ed30 comp rx=0 tx=0).stop 2023-10-30T14:02:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.222+0000 7f4549268640 1 -- 172.21.15.46:0/4293915279 shutdown_connections 2023-10-30T14:02:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.222+0000 7f4549268640 1 --2- 172.21.15.46:0/4293915279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45440ca0e0 0x7f454415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:08.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.222+0000 7f4549268640 1 --2- 172.21.15.46:0/4293915279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45441538a0 0x7f454415d6f0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:08.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.222+0000 7f4549268640 1 -- 172.21.15.46:0/4293915279 >> 172.21.15.46:0/4293915279 conn(0x7f454406bfe0 msgr2=0x7f454406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:08.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.222+0000 7f4549268640 1 -- 172.21.15.46:0/4293915279 shutdown_connections 2023-10-30T14:02:08.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.222+0000 7f4549268640 1 -- 172.21.15.46:0/4293915279 wait complete. 2023-10-30T14:02:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4549268640 1 Processor -- start 2023-10-30T14:02:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4549268640 1 -- start start 2023-10-30T14:02:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4549268640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 0x7f45440ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4549268640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45440cf160 0x7f45441e42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4549268640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45441538a0 0x7f4544147a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4549268640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441590a0 con 0x7f45440ca0e0 2023-10-30T14:02:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4549268640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441e3560 con 0x7f45441538a0 2023-10-30T14:02:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4549268640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45441f2e60 con 0x7f45440cf160 2023-10-30T14:02:08.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4542ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 0x7f45440ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:08.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4542ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 0x7f45440ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59680/0 (socket says 172.21.15.46:59680) 2023-10-30T14:02:08.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4542ffd640 1 -- 172.21.15.46:0/1115302799 learned_addr learned my addr 172.21.15.46:0/1115302799 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:08.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f45427fc640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45440cf160 0x7f45441e42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:08.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f45437fe640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45441538a0 0x7f4544147a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:08.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4542ffd640 1 -- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45440cf160 msgr2=0x7f45441e42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:08.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4542ffd640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45440cf160 0x7f45441e42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4542ffd640 1 -- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45441538a0 msgr2=0x7f4544147a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4542ffd640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45441538a0 0x7f4544147a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4542ffd640 1 -- 172.21.15.46:0/1115302799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45340094d0 con 0x7f45440ca0e0 2023-10-30T14:02:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f45437fe640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45441538a0 0x7f4544147a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:02:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4542ffd640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 0x7f45440ce770 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f45340b3040 tx=0x7f453401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f452b7fe640 1 -- 172.21.15.46:0/1115302799 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45340be070 con 0x7f45440ca0e0 2023-10-30T14:02:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f452b7fe640 1 -- 172.21.15.46:0/1115302799 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45340ba400 con 0x7f45440ca0e0 2023-10-30T14:02:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4549268640 1 -- 172.21.15.46:0/1115302799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45441592b0 con 0x7f45440ca0e0 2023-10-30T14:02:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f452b7fe640 1 -- 172.21.15.46:0/1115302799 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45340ba7d0 con 0x7f45440ca0e0 2023-10-30T14:02:08.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.226+0000 7f4549268640 1 -- 172.21.15.46:0/1115302799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45441f3120 con 0x7f45440ca0e0 2023-10-30T14:02:08.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.230+0000 7f4549268640 1 -- 172.21.15.46:0/1115302799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45440b2a50 con 0x7f45440ca0e0 2023-10-30T14:02:08.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.230+0000 7f452b7fe640 1 -- 172.21.15.46:0/1115302799 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f45340c3410 con 0x7f45440ca0e0 2023-10-30T14:02:08.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.230+0000 7f452b7fe640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45300664a0 0x7f4530068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:08.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.230+0000 7f452b7fe640 1 -- 172.21.15.46:0/1115302799 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f4534122210 con 0x7f45440ca0e0 2023-10-30T14:02:08.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.230+0000 7f45427fc640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45300664a0 0x7f4530068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:08.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.234+0000 7f45427fc640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45300664a0 0x7f4530068960 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4544151ed0 tx=0x7f4538008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:08.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.234+0000 7f452b7fe640 1 -- 172.21.15.46:0/1115302799 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f45340eeea0 con 0x7f45440ca0e0 2023-10-30T14:02:08.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 -- 172.21.15.46:0/1115302799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4544159870 con 0x7f45440ca0e0 2023-10-30T14:02:08.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f452b7fe640 1 -- 172.21.15.46:0/1115302799 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5735 (secure 0 0 0) 0x7f4544159870 con 0x7f45440ca0e0 2023-10-30T14:02:08.429 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:02:08.429 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:05.559049+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:02:08.429 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 169 2023-10-30T14:02:08.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 -- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45300664a0 msgr2=0x7f4530068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45300664a0 0x7f4530068960 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4544151ed0 tx=0x7f4538008040 comp rx=0 tx=0).stop 2023-10-30T14:02:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 -- 172.21.15.46:0/1115302799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 msgr2=0x7f45440ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 0x7f45440ce770 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f45340b3040 tx=0x7f453401bad0 comp rx=0 tx=0).stop 2023-10-30T14:02:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 -- 172.21.15.46:0/1115302799 shutdown_connections 2023-10-30T14:02:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45300664a0 0x7f4530068960 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45441538a0 0x7f4544147a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45440cf160 0x7f45441e42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 --2- 172.21.15.46:0/1115302799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 0x7f45440ce770 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 -- 172.21.15.46:0/1115302799 >> 172.21.15.46:0/1115302799 conn(0x7f454406bfe0 msgr2=0x7f4544065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 -- 172.21.15.46:0/1115302799 shutdown_connections 2023-10-30T14:02:08.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:08.426+0000 7f4549268640 1 -- 172.21.15.46:0/1115302799 wait complete. 2023-10-30T14:02:08.450 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:02:10.451 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:02:10.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.602+0000 7f731acc9640 1 Processor -- start 2023-10-30T14:02:10.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.602+0000 7f731acc9640 1 -- start start 2023-10-30T14:02:10.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f731acc9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f73140c05d0 0x7f731413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:10.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f731acc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73140c6f60 0x7f731413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:10.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f731acc9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73140c0a50 con 0x7f73140c6b80 2023-10-30T14:02:10.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f731acc9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731406f6e0 con 0x7f73140c05d0 2023-10-30T14:02:10.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f731acc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731406f9c0 con 0x7f73140c6f60 2023-10-30T14:02:10.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f7319266640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73140c6f60 0x7f731413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:10.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f7318a65640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f73140c6b80 0x7f73140bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:44170/0 (socket says 172.21.15.46:44170) 2023-10-30T14:02:10.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f7313fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f73140c05d0 0x7f731413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:10.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f7318a65640 1 -- 172.21.15.46:0/403267427 learned_addr learned my addr 172.21.15.46:0/403267427 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:02:10.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f7313fff640 1 -- 172.21.15.46:0/403267427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73140c6f60 msgr2=0x7f731413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f7313fff640 1 --2- 172.21.15.46:0/403267427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73140c6f60 0x7f731413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f7313fff640 1 -- 172.21.15.46:0/403267427 >> v1:172.21.15.154:6789/0 conn(0x7f73140c6b80 legacy=0x7f73140bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f7313fff640 1 -- 172.21.15.46:0/403267427 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731406fca0 con 0x7f73140c05d0 2023-10-30T14:02:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.606+0000 7f7319266640 1 --2- 172.21.15.46:0/403267427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73140c6f60 0x7f731413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:02:10.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f7313fff640 1 --2- 172.21.15.46:0/403267427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73140c05d0 0x7f731413ed90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f73000097c0 tx=0x7f730001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d979623afa5e059e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:10.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f73137fe640 1 -- 172.21.15.46:0/403267427 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73000ad640 con 0x7f73140c05d0 2023-10-30T14:02:10.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f73137fe640 1 -- 172.21.15.46:0/403267427 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73000ad7a0 con 0x7f73140c05d0 2023-10-30T14:02:10.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f73137fe640 1 -- 172.21.15.46:0/403267427 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73000adb30 con 0x7f73140c05d0 2023-10-30T14:02:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f731acc9640 1 -- 172.21.15.46:0/403267427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73140c05d0 msgr2=0x7f731413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f731acc9640 1 --2- 172.21.15.46:0/403267427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73140c05d0 0x7f731413ed90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f73000097c0 tx=0x7f730001c480 comp rx=0 tx=0).stop 2023-10-30T14:02:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f731acc9640 1 -- 172.21.15.46:0/403267427 shutdown_connections 2023-10-30T14:02:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f731acc9640 1 --2- 172.21.15.46:0/403267427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73140c6f60 0x7f731413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f731acc9640 1 --2- 172.21.15.46:0/403267427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73140c05d0 0x7f731413ed90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f731acc9640 1 -- 172.21.15.46:0/403267427 >> 172.21.15.46:0/403267427 conn(0x7f73140c7910 msgr2=0x7f73140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f731acc9640 1 -- 172.21.15.46:0/403267427 shutdown_connections 2023-10-30T14:02:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f731acc9640 1 -- 172.21.15.46:0/403267427 wait complete. 2023-10-30T14:02:10.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f731acc9640 1 Processor -- start 2023-10-30T14:02:10.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.610+0000 7f731acc9640 1 -- start start 2023-10-30T14:02:10.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f731acc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73140c05d0 0x7f731414e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:10.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f731acc9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73140c6b80 0x7f73141f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:10.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f731acc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73140c6f60 0x7f7314130a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:10.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f731acc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7314132c70 con 0x7f73140c6f60 2023-10-30T14:02:10.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f731acc9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7314130f40 con 0x7f73140c6b80 2023-10-30T14:02:10.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f731acc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7314133550 con 0x7f73140c05d0 2023-10-30T14:02:10.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f7319266640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73140c6f60 0x7f7314130a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:10.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f7319266640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73140c6f60 0x7f7314130a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59682/0 (socket says 172.21.15.46:59682) 2023-10-30T14:02:10.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f7319266640 1 -- 172.21.15.46:0/2810302634 learned_addr learned my addr 172.21.15.46:0/2810302634 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:10.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f7318a65640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73140c05d0 0x7f731414e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:10.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f7313fff640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73140c6b80 0x7f73141f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:10.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f7319266640 1 -- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73140c05d0 msgr2=0x7f731414e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:10.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f7319266640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73140c05d0 0x7f731414e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f7319266640 1 -- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73140c6b80 msgr2=0x7f73141f1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f7319266640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73140c6b80 0x7f73141f1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f7319266640 1 -- 172.21.15.46:0/2810302634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c009a70 con 0x7f73140c6f60 2023-10-30T14:02:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f7319266640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73140c6f60 0x7f7314130a00 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f730c00efc0 tx=0x7f730c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f73117fa640 1 -- 172.21.15.46:0/2810302634 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c009cd0 con 0x7f73140c6f60 2023-10-30T14:02:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f73117fa640 1 -- 172.21.15.46:0/2810302634 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c00f040 con 0x7f73140c6f60 2023-10-30T14:02:10.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f73117fa640 1 -- 172.21.15.46:0/2810302634 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c0039a0 con 0x7f73140c6f60 2023-10-30T14:02:10.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f731acc9640 1 -- 172.21.15.46:0/2810302634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7300009420 con 0x7f73140c6f60 2023-10-30T14:02:10.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f731acc9640 1 -- 172.21.15.46:0/2810302634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7314130570 con 0x7f73140c6f60 2023-10-30T14:02:10.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f731acc9640 1 -- 172.21.15.46:0/2810302634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7314130700 con 0x7f73140c6f60 2023-10-30T14:02:10.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.614+0000 7f73117fa640 1 -- 172.21.15.46:0/2810302634 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f730c014480 con 0x7f73140c6f60 2023-10-30T14:02:10.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.618+0000 7f73117fa640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72f80664a0 0x7f72f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:10.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.618+0000 7f73117fa640 1 -- 172.21.15.46:0/2810302634 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f730c0851d0 con 0x7f73140c6f60 2023-10-30T14:02:10.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.618+0000 7f7318a65640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72f80664a0 0x7f72f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:10.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.618+0000 7f7318a65640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72f80664a0 0x7f72f8068960 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f7308005890 tx=0x7f7308009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:10.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.622+0000 7f73117fa640 1 -- 172.21.15.46:0/2810302634 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f730c051e80 con 0x7f73140c6f60 2023-10-30T14:02:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 -- 172.21.15.46:0/2810302634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f731404e170 con 0x7f73140c6f60 2023-10-30T14:02:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f73117fa640 1 -- 172.21.15.46:0/2810302634 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5735 (secure 0 0 0) 0x7f730c0554a0 con 0x7f73140c6f60 2023-10-30T14:02:10.820 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:02:10.821 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:05.559049+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:02:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 169 2023-10-30T14:02:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 -- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72f80664a0 msgr2=0x7f72f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72f80664a0 0x7f72f8068960 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f7308005890 tx=0x7f7308009040 comp rx=0 tx=0).stop 2023-10-30T14:02:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 -- 172.21.15.46:0/2810302634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73140c6f60 msgr2=0x7f7314130a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73140c6f60 0x7f7314130a00 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f730c00efc0 tx=0x7f730c00c490 comp rx=0 tx=0).stop 2023-10-30T14:02:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 -- 172.21.15.46:0/2810302634 shutdown_connections 2023-10-30T14:02:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72f80664a0 0x7f72f8068960 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73140c6f60 0x7f7314130a00 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73140c6b80 0x7f73141f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 --2- 172.21.15.46:0/2810302634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73140c05d0 0x7f731414e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 -- 172.21.15.46:0/2810302634 >> 172.21.15.46:0/2810302634 conn(0x7f73140c7910 msgr2=0x7f73140bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 -- 172.21.15.46:0/2810302634 shutdown_connections 2023-10-30T14:02:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:10.818+0000 7f731acc9640 1 -- 172.21.15.46:0/2810302634 wait complete. 2023-10-30T14:02:10.841 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:02:12.842 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:02:12.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.994+0000 7f5ffc93e640 1 Processor -- start 2023-10-30T14:02:12.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.994+0000 7f5ffc93e640 1 -- start start 2023-10-30T14:02:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ffc93e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ff813f9c0 0x7f5ff80c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ffc93e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ff80c5f70 0x7f5ff8159b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ffc93e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5ff815a0c0 0x7f5ff806b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ffc93e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff8157360 con 0x7f5ff813f9c0 2023-10-30T14:02:13.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ffc93e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff813fda0 con 0x7f5ff815a0c0 2023-10-30T14:02:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ffc93e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff81564d0 con 0x7f5ff80c5f70 2023-10-30T14:02:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ff659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ff813f9c0 0x7f5ff80c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ff659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5ff813f9c0 0x7f5ff80c5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59696/0 (socket says 172.21.15.46:59696) 2023-10-30T14:02:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ff659c640 1 -- 172.21.15.46:0/2227018529 learned_addr learned my addr 172.21.15.46:0/2227018529 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:13.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5feffff640 1 --2- 172.21.15.46:0/2227018529 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ff80c5f70 0x7f5ff8159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ff6d9d640 1 --2- 172.21.15.46:0/2227018529 >> v2:172.21.15.154:3300/0 conn(0x7f5ff815a0c0 0x7f5ff806b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ff659c640 1 -- 172.21.15.46:0/2227018529 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ff80c5f70 msgr2=0x7f5ff8159b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ff659c640 1 --2- 172.21.15.46:0/2227018529 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ff80c5f70 0x7f5ff8159b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ff659c640 1 -- 172.21.15.46:0/2227018529 >> v2:172.21.15.154:3300/0 conn(0x7f5ff815a0c0 msgr2=0x7f5ff806b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:13.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ff659c640 1 --2- 172.21.15.46:0/2227018529 >> v2:172.21.15.154:3300/0 conn(0x7f5ff815a0c0 0x7f5ff806b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:12.998+0000 7f5ff659c640 1 -- 172.21.15.46:0/2227018529 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff815a530 con 0x7f5ff813f9c0 2023-10-30T14:02:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ff659c640 1 --2- 172.21.15.46:0/2227018529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff813f9c0 0x7f5ff80c5a30 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f5ff806f290 tx=0x7f5fe801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315b0c595ba4144c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:13.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ff5d9b640 1 -- 172.21.15.46:0/2227018529 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe80ac410 con 0x7f5ff813f9c0 2023-10-30T14:02:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ff5d9b640 1 -- 172.21.15.46:0/2227018529 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe80ac570 con 0x7f5ff813f9c0 2023-10-30T14:02:13.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ff5d9b640 1 -- 172.21.15.46:0/2227018529 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe80ac900 con 0x7f5ff813f9c0 2023-10-30T14:02:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 -- 172.21.15.46:0/2227018529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff813f9c0 msgr2=0x7f5ff80c5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 --2- 172.21.15.46:0/2227018529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff813f9c0 0x7f5ff80c5a30 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f5ff806f290 tx=0x7f5fe801c500 comp rx=0 tx=0).stop 2023-10-30T14:02:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 -- 172.21.15.46:0/2227018529 shutdown_connections 2023-10-30T14:02:13.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 --2- 172.21.15.46:0/2227018529 >> v2:172.21.15.154:3300/0 conn(0x7f5ff815a0c0 0x7f5ff806b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 --2- 172.21.15.46:0/2227018529 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ff80c5f70 0x7f5ff8159b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 --2- 172.21.15.46:0/2227018529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff813f9c0 0x7f5ff80c5a30 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 -- 172.21.15.46:0/2227018529 >> 172.21.15.46:0/2227018529 conn(0x7f5ff80cace0 msgr2=0x7f5ff80cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 -- 172.21.15.46:0/2227018529 shutdown_connections 2023-10-30T14:02:13.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 -- 172.21.15.46:0/2227018529 wait complete. 2023-10-30T14:02:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 Processor -- start 2023-10-30T14:02:13.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 -- start start 2023-10-30T14:02:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ff80c5f70 0x7f5ff80c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ff813f9c0 0x7f5ff813b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff815a0c0 0x7f5ff8136530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff81f5820 con 0x7f5ff815a0c0 2023-10-30T14:02:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff80c6f10 con 0x7f5ff813f9c0 2023-10-30T14:02:13.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.002+0000 7f5ffc93e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff80c77f0 con 0x7f5ff80c5f70 2023-10-30T14:02:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ff6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff815a0c0 0x7f5ff8136530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ff6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff815a0c0 0x7f5ff8136530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59708/0 (socket says 172.21.15.46:59708) 2023-10-30T14:02:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ff6d9d640 1 -- 172.21.15.46:0/1825611712 learned_addr learned my addr 172.21.15.46:0/1825611712 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ff659c640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ff80c5f70 0x7f5ff80c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:13.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5feffff640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ff813f9c0 0x7f5ff813b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ff6d9d640 1 -- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ff80c5f70 msgr2=0x7f5ff80c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ff6d9d640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ff80c5f70 0x7f5ff80c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ff6d9d640 1 -- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ff813f9c0 msgr2=0x7f5ff813b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ff6d9d640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ff813f9c0 0x7f5ff813b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:13.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ff6d9d640 1 -- 172.21.15.46:0/1825611712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe80094d0 con 0x7f5ff815a0c0 2023-10-30T14:02:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ff6d9d640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff815a0c0 0x7f5ff8136530 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0007d90 tx=0x7f5ff000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5feeffd640 1 -- 172.21.15.46:0/1825611712 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0014420 con 0x7f5ff815a0c0 2023-10-30T14:02:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5feeffd640 1 -- 172.21.15.46:0/1825611712 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff00145a0 con 0x7f5ff815a0c0 2023-10-30T14:02:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ffc93e640 1 -- 172.21.15.46:0/1825611712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff81594b0 con 0x7f5ff815a0c0 2023-10-30T14:02:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5feeffd640 1 -- 172.21.15.46:0/1825611712 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0014950 con 0x7f5ff815a0c0 2023-10-30T14:02:13.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ffc93e640 1 -- 172.21.15.46:0/1825611712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff8159750 con 0x7f5ff815a0c0 2023-10-30T14:02:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5ffc93e640 1 -- 172.21.15.46:0/1825611712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff81f5960 con 0x7f5ff815a0c0 2023-10-30T14:02:13.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.006+0000 7f5feeffd640 1 -- 172.21.15.46:0/1825611712 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5ff0014ab0 con 0x7f5ff815a0c0 2023-10-30T14:02:13.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.010+0000 7f5feeffd640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5fd80664a0 0x7f5fd8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:13.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.010+0000 7f5feeffd640 1 -- 172.21.15.46:0/1825611712 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f5ff0084f10 con 0x7f5ff815a0c0 2023-10-30T14:02:13.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.010+0000 7f5ff659c640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5fd80664a0 0x7f5fd8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:13.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.010+0000 7f5ff659c640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5fd80664a0 0x7f5fd8068960 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5fe80ae000 tx=0x7f5fe8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:13.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.010+0000 7f5feeffd640 1 -- 172.21.15.46:0/1825611712 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5ff81f5960 con 0x7f5ff815a0c0 2023-10-30T14:02:13.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.202+0000 7f5ffc93e640 1 -- 172.21.15.46:0/1825611712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ff81f5960 con 0x7f5ff815a0c0 2023-10-30T14:02:13.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.202+0000 7f5feeffd640 1 -- 172.21.15.46:0/1825611712 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f5ff81f5960 con 0x7f5ff815a0c0 2023-10-30T14:02:13.205 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:02:13.205 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:10.662541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:02:13.206 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 170 2023-10-30T14:02:13.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.202+0000 7f5ffc93e640 1 -- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5fd80664a0 msgr2=0x7f5fd8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:13.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.202+0000 7f5ffc93e640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5fd80664a0 0x7f5fd8068960 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5fe80ae000 tx=0x7f5fe8009040 comp rx=0 tx=0).stop 2023-10-30T14:02:13.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.202+0000 7f5ffc93e640 1 -- 172.21.15.46:0/1825611712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff815a0c0 msgr2=0x7f5ff8136530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:13.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.202+0000 7f5ffc93e640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff815a0c0 0x7f5ff8136530 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0007d90 tx=0x7f5ff000e4b0 comp rx=0 tx=0).stop 2023-10-30T14:02:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.202+0000 7f5ffc93e640 1 -- 172.21.15.46:0/1825611712 shutdown_connections 2023-10-30T14:02:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.202+0000 7f5ffc93e640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5fd80664a0 0x7f5fd8068960 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.202+0000 7f5ffc93e640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff815a0c0 0x7f5ff8136530 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.202+0000 7f5ffc93e640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ff813f9c0 0x7f5ff813b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.202+0000 7f5ffc93e640 1 --2- 172.21.15.46:0/1825611712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ff80c5f70 0x7f5ff80c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.202+0000 7f5ffc93e640 1 -- 172.21.15.46:0/1825611712 >> 172.21.15.46:0/1825611712 conn(0x7f5ff80cace0 msgr2=0x7f5ff80bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.206+0000 7f5ffc93e640 1 -- 172.21.15.46:0/1825611712 shutdown_connections 2023-10-30T14:02:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:13.206+0000 7f5ffc93e640 1 -- 172.21.15.46:0/1825611712 wait complete. 2023-10-30T14:02:13.227 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:02:15.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:02:15.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.378+0000 7f974f1c5640 1 Processor -- start 2023-10-30T14:02:15.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f974f1c5640 1 -- start start 2023-10-30T14:02:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f974f1c5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9748153bf0 0x7f9748071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f974f1c5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97480c6fe0 0x7f97480c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:15.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f974f1c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97480c79e0 0x7f97480cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:15.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f974f1c5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97480c6810 con 0x7f97480c6fe0 2023-10-30T14:02:15.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f974f1c5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9748159900 con 0x7f9748153bf0 2023-10-30T14:02:15.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f974f1c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97480700b0 con 0x7f97480c79e0 2023-10-30T14:02:15.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f9747fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97480c6fe0 0x7f97480c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:15.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f9747fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97480c6fe0 0x7f97480c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59720/0 (socket says 172.21.15.46:59720) 2023-10-30T14:02:15.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f9747fff640 1 -- 172.21.15.46:0/1919550246 learned_addr learned my addr 172.21.15.46:0/1919550246 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:15.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f974cf61640 1 --2- 172.21.15.46:0/1919550246 >> v2:172.21.15.154:3300/0 conn(0x7f9748153bf0 0x7f9748071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:15.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f974d762640 1 --2- 172.21.15.46:0/1919550246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97480c79e0 0x7f97480cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:15.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f9747fff640 1 -- 172.21.15.46:0/1919550246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97480c79e0 msgr2=0x7f97480cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:15.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f9747fff640 1 --2- 172.21.15.46:0/1919550246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97480c79e0 0x7f97480cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:15.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f9747fff640 1 -- 172.21.15.46:0/1919550246 >> v2:172.21.15.154:3300/0 conn(0x7f9748153bf0 msgr2=0x7f9748071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:15.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f9747fff640 1 --2- 172.21.15.46:0/1919550246 >> v2:172.21.15.154:3300/0 conn(0x7f9748153bf0 0x7f9748071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:15.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.382+0000 7f9747fff640 1 -- 172.21.15.46:0/1919550246 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97480c6530 con 0x7f97480c6fe0 2023-10-30T14:02:15.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f9747fff640 1 --2- 172.21.15.46:0/1919550246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97480c6fe0 0x7f97480c74a0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f97480c9150 tx=0x7f973401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb20a56e6c1abe4a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:15.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f97477fe640 1 -- 172.21.15.46:0/1919550246 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97340a89b0 con 0x7f97480c6fe0 2023-10-30T14:02:15.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f97477fe640 1 -- 172.21.15.46:0/1919550246 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97340a8b10 con 0x7f97480c6fe0 2023-10-30T14:02:15.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f97477fe640 1 -- 172.21.15.46:0/1919550246 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97340a8ea0 con 0x7f97480c6fe0 2023-10-30T14:02:15.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f974f1c5640 1 -- 172.21.15.46:0/1919550246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97480c6fe0 msgr2=0x7f97480c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:15.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f974f1c5640 1 --2- 172.21.15.46:0/1919550246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97480c6fe0 0x7f97480c74a0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f97480c9150 tx=0x7f973401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:02:15.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f974f1c5640 1 -- 172.21.15.46:0/1919550246 shutdown_connections 2023-10-30T14:02:15.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f974f1c5640 1 --2- 172.21.15.46:0/1919550246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97480c79e0 0x7f97480cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:15.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f974f1c5640 1 --2- 172.21.15.46:0/1919550246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97480c6fe0 0x7f97480c74a0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:15.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f974f1c5640 1 --2- 172.21.15.46:0/1919550246 >> v2:172.21.15.154:3300/0 conn(0x7f9748153bf0 0x7f9748071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:15.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f974f1c5640 1 -- 172.21.15.46:0/1919550246 >> 172.21.15.46:0/1919550246 conn(0x7f974806c6e0 msgr2=0x7f9748069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:15.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f974f1c5640 1 -- 172.21.15.46:0/1919550246 shutdown_connections 2023-10-30T14:02:15.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f974f1c5640 1 -- 172.21.15.46:0/1919550246 wait complete. 2023-10-30T14:02:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f974f1c5640 1 Processor -- start 2023-10-30T14:02:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.386+0000 7f974f1c5640 1 -- start start 2023-10-30T14:02:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f974f1c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97480c6fe0 0x7f974814bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f974f1c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97480c79e0 0x7f974814c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:15.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f974f1c5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9748153bf0 0x7f97481550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:15.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f974f1c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9748154c30 con 0x7f97480c79e0 2023-10-30T14:02:15.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f974f1c5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9748070cf0 con 0x7f9748153bf0 2023-10-30T14:02:15.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f974f1c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974814d210 con 0x7f97480c6fe0 2023-10-30T14:02:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f9747fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97480c79e0 0x7f974814c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f9747fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97480c79e0 0x7f974814c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59722/0 (socket says 172.21.15.46:59722) 2023-10-30T14:02:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f9747fff640 1 -- 172.21.15.46:0/3360282252 learned_addr learned my addr 172.21.15.46:0/3360282252 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f974cf61640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97480c6fe0 0x7f974814bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f974d762640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9748153bf0 0x7f97481550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f9747fff640 1 -- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97480c6fe0 msgr2=0x7f974814bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f9747fff640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97480c6fe0 0x7f974814bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f9747fff640 1 -- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9748153bf0 msgr2=0x7f97481550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f9747fff640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9748153bf0 0x7f97481550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f9747fff640 1 -- 172.21.15.46:0/3360282252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97340094d0 con 0x7f97480c79e0 2023-10-30T14:02:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f9747fff640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97480c79e0 0x7f974814c1a0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f97340b3040 tx=0x7f973401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f97457fa640 1 -- 172.21.15.46:0/3360282252 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97340b0e00 con 0x7f97480c79e0 2023-10-30T14:02:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f97457fa640 1 -- 172.21.15.46:0/3360282252 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97340bb470 con 0x7f97480c79e0 2023-10-30T14:02:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f97457fa640 1 -- 172.21.15.46:0/3360282252 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97340bb820 con 0x7f97480c79e0 2023-10-30T14:02:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f974f1c5640 1 -- 172.21.15.46:0/3360282252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974814d4d0 con 0x7f97480c79e0 2023-10-30T14:02:15.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f974f1c5640 1 -- 172.21.15.46:0/3360282252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9748070f90 con 0x7f97480c79e0 2023-10-30T14:02:15.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.390+0000 7f974f1c5640 1 -- 172.21.15.46:0/3360282252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974804e170 con 0x7f97480c79e0 2023-10-30T14:02:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.394+0000 7f97457fa640 1 -- 172.21.15.46:0/3360282252 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f97340b7070 con 0x7f97480c79e0 2023-10-30T14:02:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.394+0000 7f97457fa640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f972c0664a0 0x7f972c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.394+0000 7f97457fa640 1 -- 172.21.15.46:0/3360282252 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f9734122970 con 0x7f97480c79e0 2023-10-30T14:02:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.394+0000 7f974cf61640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f972c0664a0 0x7f972c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.394+0000 7f974cf61640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f972c0664a0 0x7f972c068960 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f973c0079c0 tx=0x7f973c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.398+0000 7f97457fa640 1 -- 172.21.15.46:0/3360282252 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97340ef620 con 0x7f97480c79e0 2023-10-30T14:02:15.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.589+0000 7f974f1c5640 1 -- 172.21.15.46:0/3360282252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f974806fe90 con 0x7f97480c79e0 2023-10-30T14:02:15.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.589+0000 7f97457fa640 1 -- 172.21.15.46:0/3360282252 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f97340b5020 con 0x7f97480c79e0 2023-10-30T14:02:15.593 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:02:15.594 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:10.662541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:02:15.594 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 170 2023-10-30T14:02:15.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.589+0000 7f974f1c5640 1 -- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f972c0664a0 msgr2=0x7f972c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:15.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.589+0000 7f974f1c5640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f972c0664a0 0x7f972c068960 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f973c0079c0 tx=0x7f973c008040 comp rx=0 tx=0).stop 2023-10-30T14:02:15.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.589+0000 7f974f1c5640 1 -- 172.21.15.46:0/3360282252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97480c79e0 msgr2=0x7f974814c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:15.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.589+0000 7f974f1c5640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97480c79e0 0x7f974814c1a0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f97340b3040 tx=0x7f973401ba90 comp rx=0 tx=0).stop 2023-10-30T14:02:15.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.593+0000 7f974f1c5640 1 -- 172.21.15.46:0/3360282252 shutdown_connections 2023-10-30T14:02:15.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.593+0000 7f974f1c5640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f972c0664a0 0x7f972c068960 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:15.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.593+0000 7f974f1c5640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9748153bf0 0x7f97481550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:15.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.593+0000 7f974f1c5640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97480c79e0 0x7f974814c1a0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:15.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.593+0000 7f974f1c5640 1 --2- 172.21.15.46:0/3360282252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97480c6fe0 0x7f974814bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:15.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.593+0000 7f974f1c5640 1 -- 172.21.15.46:0/3360282252 >> 172.21.15.46:0/3360282252 conn(0x7f974806c6e0 msgr2=0x7f9748151870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:15.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.593+0000 7f974f1c5640 1 -- 172.21.15.46:0/3360282252 shutdown_connections 2023-10-30T14:02:15.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:15.593+0000 7f974f1c5640 1 -- 172.21.15.46:0/3360282252 wait complete. 2023-10-30T14:02:15.615 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 170, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 170, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7704": { "addr": "172.21.15.46:6836/3734066164", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3734066164, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3734066164, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7704, "incarnation": 152, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_8034": { "addr": "172.21.15.46:6833/2075693104", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2075693104, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2075693104, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8034, "incarnation": 167, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:rejoin", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:02:10.662541+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7704, "mds_1": 8034 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:02:17.616 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:02:17.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.769+0000 7f3e8d2d6640 1 Processor -- start 2023-10-30T14:02:17.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.769+0000 7f3e8d2d6640 1 -- start start 2023-10-30T14:02:17.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.769+0000 7f3e8d2d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8806f7a0 0x7f3e8806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:17.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.769+0000 7f3e8d2d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3e88071110 0x7f3e88158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:17.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.769+0000 7f3e8d2d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e88066cf0 con 0x7f3e880c6bd0 2023-10-30T14:02:17.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.769+0000 7f3e8d2d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e880700c0 con 0x7f3e88071110 2023-10-30T14:02:17.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.769+0000 7f3e8d2d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e880c70d0 con 0x7f3e8806f7a0 2023-10-30T14:02:17.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e7ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3e880c6bd0 0x7f3e88069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54796/0 (socket says 172.21.15.46:54796) 2023-10-30T14:02:17.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e7ffff640 1 -- 172.21.15.46:0/2352999887 learned_addr learned my addr 172.21.15.46:0/2352999887 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:02:17.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e86ffd640 1 --2- 172.21.15.46:0/2352999887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8806f7a0 0x7f3e8806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:17.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e877fe640 1 --2- 172.21.15.46:0/2352999887 >> v2:172.21.15.154:3300/0 conn(0x7f3e88071110 0x7f3e88158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:17.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524213024 0 0) 0x7f3e88066cf0 con 0x7f3e880c6bd0 2023-10-30T14:02:17.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e74003530 con 0x7f3e880c6bd0 2023-10-30T14:02:17.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3232244149 0 0) 0x7f3e74003530 con 0x7f3e880c6bd0 2023-10-30T14:02:17.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8806f7a0 msgr2=0x7f3e8806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:17.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 --2- 172.21.15.46:0/2352999887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8806f7a0 0x7f3e8806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:17.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 >> v2:172.21.15.154:3300/0 conn(0x7f3e88071110 msgr2=0x7f3e88158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:17.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 --2- 172.21.15.46:0/2352999887 >> v2:172.21.15.154:3300/0 conn(0x7f3e88071110 0x7f3e88158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:17.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e880c7390 con 0x7f3e880c6bd0 2023-10-30T14:02:17.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f3e70001680 con 0x7f3e880c6bd0 2023-10-30T14:02:17.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 >> v1:172.21.15.46:6789/0 conn(0x7f3e880c6bd0 legacy=0x7f3e88069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:17.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 --2- 172.21.15.46:0/2352999887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8806f7a0 0x7f3e74003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:17.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 --2- 172.21.15.46:0/2352999887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74003f00 0x7f3e740047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:17.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 --2- 172.21.15.46:0/2352999887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e74004d10 0x7f3e74005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:17.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e880700c0 con 0x7f3e74003f00 2023-10-30T14:02:17.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e880c70d0 con 0x7f3e74004d10 2023-10-30T14:02:17.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7400b9c0 con 0x7f3e8806f7a0 2023-10-30T14:02:17.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e7ffff640 1 --2- 172.21.15.46:0/2352999887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74003f00 0x7f3e740047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:17.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e86ffd640 1 --2- 172.21.15.46:0/2352999887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8806f7a0 0x7f3e74003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:17.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e877fe640 1 --2- 172.21.15.46:0/2352999887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e74004d10 0x7f3e74005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:17.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e7ffff640 1 -- 172.21.15.46:0/2352999887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8806f7a0 msgr2=0x7f3e74003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:17.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e7ffff640 1 --2- 172.21.15.46:0/2352999887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8806f7a0 0x7f3e74003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:17.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e7ffff640 1 -- 172.21.15.46:0/2352999887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e74004d10 msgr2=0x7f3e74005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:17.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e7ffff640 1 --2- 172.21.15.46:0/2352999887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e74004d10 0x7f3e74005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:17.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.773+0000 7f3e7ffff640 1 -- 172.21.15.46:0/2352999887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e74003530 con 0x7f3e74003f00 2023-10-30T14:02:17.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e7ffff640 1 --2- 172.21.15.46:0/2352999887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74003f00 0x7f3e740047d0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f3e7000c240 tx=0x7f3e7001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96b11d1e3156eae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:17.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e700a5960 con 0x7f3e74003f00 2023-10-30T14:02:17.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e700a5ac0 con 0x7f3e74003f00 2023-10-30T14:02:17.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e867fc640 1 -- 172.21.15.46:0/2352999887 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e700058e0 con 0x7f3e74003f00 2023-10-30T14:02:17.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/2352999887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74003f00 msgr2=0x7f3e740047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:17.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e8d2d6640 1 --2- 172.21.15.46:0/2352999887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74003f00 0x7f3e740047d0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f3e7000c240 tx=0x7f3e7001cc60 comp rx=0 tx=0).stop 2023-10-30T14:02:17.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e877fe640 1 -- 172.21.15.46:0/2352999887 reap_dead start 2023-10-30T14:02:17.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/2352999887 shutdown_connections 2023-10-30T14:02:17.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/2352999887 >> 172.21.15.46:0/2352999887 conn(0x7f3e880c7910 msgr2=0x7f3e880c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:17.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/2352999887 shutdown_connections 2023-10-30T14:02:17.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/2352999887 wait complete. 2023-10-30T14:02:17.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e8d2d6640 1 Processor -- start 2023-10-30T14:02:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e8d2d6640 1 -- start start 2023-10-30T14:02:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e8d2d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e74003f00 0x7f3e88159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.777+0000 7f3e8d2d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e88158210 0x7f3e8815a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:17.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e8d2d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8815b290 0x7f3e8814c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:17.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e8d2d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8815ae70 con 0x7f3e88158210 2023-10-30T14:02:17.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e8d2d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e88156d60 con 0x7f3e74003f00 2023-10-30T14:02:17.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e8d2d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e880c8cf0 con 0x7f3e8815b290 2023-10-30T14:02:17.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e88158210 0x7f3e8815a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:17.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e88158210 0x7f3e8815a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58220/0 (socket says 172.21.15.46:58220) 2023-10-30T14:02:17.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7ffff640 1 -- 172.21.15.46:0/398269220 learned_addr learned my addr 172.21.15.46:0/398269220 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:17.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e86ffd640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e74003f00 0x7f3e88159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:17.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e877fe640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8815b290 0x7f3e8814c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:17.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7ffff640 1 -- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8815b290 msgr2=0x7f3e8814c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:17.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7ffff640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8815b290 0x7f3e8814c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:17.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7ffff640 1 -- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e74003f00 msgr2=0x7f3e88159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:17.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7ffff640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e74003f00 0x7f3e88159b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:17.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7ffff640 1 -- 172.21.15.46:0/398269220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e881561e0 con 0x7f3e88158210 2023-10-30T14:02:17.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e86ffd640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e74003f00 0x7f3e88159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:02:17.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7ffff640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e88158210 0x7f3e8815a2d0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f3e700ab040 tx=0x7f3e74005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:17.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7f7fe640 1 -- 172.21.15.46:0/398269220 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e70001c60 con 0x7f3e88158210 2023-10-30T14:02:17.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7f7fe640 1 -- 172.21.15.46:0/398269220 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e700b3070 con 0x7f3e88158210 2023-10-30T14:02:17.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7f7fe640 1 -- 172.21.15.46:0/398269220 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e700072f0 con 0x7f3e88158210 2023-10-30T14:02:17.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/398269220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e881564c0 con 0x7f3e88158210 2023-10-30T14:02:17.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/398269220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e88156a80 con 0x7f3e88158210 2023-10-30T14:02:17.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/398269220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8815abd0 con 0x7f3e88158210 2023-10-30T14:02:17.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.781+0000 7f3e7f7fe640 1 -- 172.21.15.46:0/398269220 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3e70007450 con 0x7f3e88158210 2023-10-30T14:02:17.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.785+0000 7f3e7f7fe640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3e680664d0 0x7f3e68068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:17.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.785+0000 7f3e7f7fe640 1 -- 172.21.15.46:0/398269220 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f3e700b5070 con 0x7f3e88158210 2023-10-30T14:02:17.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.785+0000 7f3e86ffd640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3e680664d0 0x7f3e68068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:17.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.785+0000 7f3e86ffd640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3e680664d0 0x7f3e68068990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009990 tx=0x7f3e78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:17.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.789+0000 7f3e7f7fe640 1 -- 172.21.15.46:0/398269220 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3e8815abd0 con 0x7f3e88158210 2023-10-30T14:02:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.977+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/398269220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e8815abd0 con 0x7f3e88158210 2023-10-30T14:02:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e7f7fe640 1 -- 172.21.15.46:0/398269220 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f3e8815abd0 con 0x7f3e88158210 2023-10-30T14:02:17.983 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:02:17.984 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:16.527331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:02:17.984 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 171 2023-10-30T14:02:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3e680664d0 msgr2=0x7f3e68068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e8d2d6640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3e680664d0 0x7f3e68068990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009990 tx=0x7f3e78008040 comp rx=0 tx=0).stop 2023-10-30T14:02:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/398269220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e88158210 msgr2=0x7f3e8815a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e8d2d6640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e88158210 0x7f3e8815a2d0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f3e700ab040 tx=0x7f3e74005800 comp rx=0 tx=0).stop 2023-10-30T14:02:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/398269220 shutdown_connections 2023-10-30T14:02:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e8d2d6640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3e680664d0 0x7f3e68068990 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:17.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e8d2d6640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8815b290 0x7f3e8814c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:17.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e8d2d6640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e88158210 0x7f3e8815a2d0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:17.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e8d2d6640 1 --2- 172.21.15.46:0/398269220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e74003f00 0x7f3e88159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:17.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/398269220 >> 172.21.15.46:0/398269220 conn(0x7f3e880c7910 msgr2=0x7f3e880c7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:17.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/398269220 shutdown_connections 2023-10-30T14:02:17.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:17.981+0000 7f3e8d2d6640 1 -- 172.21.15.46:0/398269220 wait complete. 2023-10-30T14:02:18.005 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=6809 2023-10-30T14:02:18.006 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.e 2023-10-30T14:02:18.416 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T14:02:18.413+0000 7fdd5c91c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:02:24.533 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:02:24.537 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:02:25.007 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T14:02:25.007 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T14:02:25.008 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T14:02:25.011 INFO:tasks.ceph.mds.e:Started 2023-10-30T14:02:25.012 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:02:25.111 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T14:02:25.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.165+0000 7f5f65a79640 1 Processor -- start 2023-10-30T14:02:25.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.165+0000 7f5f65a79640 1 -- start start 2023-10-30T14:02:25.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.165+0000 7f5f65a79640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f6015d6f0 0x7f5f6015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:25.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.165+0000 7f5f65a79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f600cf160 0x7f5f600cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:25.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.165+0000 7f5f65a79640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600cb800 con 0x7f5f601538a0 2023-10-30T14:02:25.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f65a79640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f600cb330 con 0x7f5f6015d6f0 2023-10-30T14:02:25.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f65a79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600cad70 con 0x7f5f600cf160 2023-10-30T14:02:25.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f5effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f6015d6f0 0x7f5f6015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:25.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f5effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f6015d6f0 0x7f5f6015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58248/0 (socket says 172.21.15.46:58248) 2023-10-30T14:02:25.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f5effd640 1 -- 172.21.15.46:0/3834838089 learned_addr learned my addr 172.21.15.46:0/3834838089 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:25.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f5ffff640 1 --2- 172.21.15.46:0/3834838089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f600cf160 0x7f5f600cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:25.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f5effd640 1 -- 172.21.15.46:0/3834838089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f600cf160 msgr2=0x7f5f600cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:02:25.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f5effd640 1 --2- 172.21.15.46:0/3834838089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f600cf160 0x7f5f600cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:25.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f5effd640 1 -- 172.21.15.46:0/3834838089 >> v1:172.21.15.154:6789/0 conn(0x7f5f601538a0 legacy=0x7f5f600ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:25.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f5effd640 1 -- 172.21.15.46:0/3834838089 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600cb050 con 0x7f5f6015d6f0 2023-10-30T14:02:25.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f5effd640 1 --2- 172.21.15.46:0/3834838089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f6015d6f0 0x7f5f6015a090 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009870 tx=0x7f5f5401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be1274b8631bf0b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:25.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f5e7fc640 1 -- 172.21.15.46:0/3834838089 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f540a8a30 con 0x7f5f6015d6f0 2023-10-30T14:02:25.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f5e7fc640 1 -- 172.21.15.46:0/3834838089 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f540a8b90 con 0x7f5f6015d6f0 2023-10-30T14:02:25.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.169+0000 7f5f5e7fc640 1 -- 172.21.15.46:0/3834838089 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f540b0440 con 0x7f5f6015d6f0 2023-10-30T14:02:25.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 -- 172.21.15.46:0/3834838089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f6015d6f0 msgr2=0x7f5f6015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:25.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 --2- 172.21.15.46:0/3834838089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f6015d6f0 0x7f5f6015a090 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009870 tx=0x7f5f5401ed30 comp rx=0 tx=0).stop 2023-10-30T14:02:25.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 -- 172.21.15.46:0/3834838089 shutdown_connections 2023-10-30T14:02:25.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 --2- 172.21.15.46:0/3834838089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f600cf160 0x7f5f600cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:25.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 --2- 172.21.15.46:0/3834838089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f6015d6f0 0x7f5f6015a090 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:25.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 -- 172.21.15.46:0/3834838089 >> 172.21.15.46:0/3834838089 conn(0x7f5f6006bfe0 msgr2=0x7f5f6006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:25.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 -- 172.21.15.46:0/3834838089 shutdown_connections 2023-10-30T14:02:25.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 -- 172.21.15.46:0/3834838089 wait complete. 2023-10-30T14:02:25.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 Processor -- start 2023-10-30T14:02:25.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 -- start start 2023-10-30T14:02:25.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f600cf160 0x7f5f6015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:25.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f601538a0 0x7f5f6015c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:25.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f6015d6f0 0x7f5f600c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:25.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600c7b80 con 0x7f5f601538a0 2023-10-30T14:02:25.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600c6a50 con 0x7f5f600cf160 2023-10-30T14:02:25.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f65a79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600c9660 con 0x7f5f6015d6f0 2023-10-30T14:02:25.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f5effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f601538a0 0x7f5f6015c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:25.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f5effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f601538a0 0x7f5f6015c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58256/0 (socket says 172.21.15.46:58256) 2023-10-30T14:02:25.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f5effd640 1 -- 172.21.15.46:0/1154947939 learned_addr learned my addr 172.21.15.46:0/1154947939 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f5f7fe640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f600cf160 0x7f5f6015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f5ffff640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f6015d6f0 0x7f5f600c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f5effd640 1 -- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f6015d6f0 msgr2=0x7f5f600c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f5effd640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f6015d6f0 0x7f5f600c6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f5effd640 1 -- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f600cf160 msgr2=0x7f5f6015b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:25.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f5effd640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f600cf160 0x7f5f6015b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:25.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.173+0000 7f5f5effd640 1 -- 172.21.15.46:0/1154947939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f540094d0 con 0x7f5f601538a0 2023-10-30T14:02:25.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.177+0000 7f5f5f7fe640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f600cf160 0x7f5f6015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:02:25.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.177+0000 7f5f5ffff640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f6015d6f0 0x7f5f600c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:02:25.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.177+0000 7f5f5effd640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f601538a0 0x7f5f6015c740 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f5f5401c960 tx=0x7f5f5401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.177+0000 7f5f47fff640 1 -- 172.21.15.46:0/1154947939 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f540b1d90 con 0x7f5f601538a0 2023-10-30T14:02:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.177+0000 7f5f47fff640 1 -- 172.21.15.46:0/1154947939 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f540b9440 con 0x7f5f601538a0 2023-10-30T14:02:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.177+0000 7f5f65a79640 1 -- 172.21.15.46:0/1154947939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f600ccfb0 con 0x7f5f601538a0 2023-10-30T14:02:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.177+0000 7f5f47fff640 1 -- 172.21.15.46:0/1154947939 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f540b97f0 con 0x7f5f601538a0 2023-10-30T14:02:25.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.177+0000 7f5f65a79640 1 -- 172.21.15.46:0/1154947939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f600c8e00 con 0x7f5f601538a0 2023-10-30T14:02:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.177+0000 7f5f65a79640 1 -- 172.21.15.46:0/1154947939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f600b2a50 con 0x7f5f601538a0 2023-10-30T14:02:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.177+0000 7f5f47fff640 1 -- 172.21.15.46:0/1154947939 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5f540b7070 con 0x7f5f601538a0 2023-10-30T14:02:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.181+0000 7f5f47fff640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f4c0664d0 0x7f5f4c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.181+0000 7f5f47fff640 1 -- 172.21.15.46:0/1154947939 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f5f54121e20 con 0x7f5f601538a0 2023-10-30T14:02:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.181+0000 7f5f5f7fe640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f4c0664d0 0x7f5f4c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.181+0000 7f5f5f7fe640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f4c0664d0 0x7f5f4c068990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5f6015d100 tx=0x7f5f50009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.185+0000 7f5f47fff640 1 -- 172.21.15.46:0/1154947939 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5f540eec60 con 0x7f5f601538a0 2023-10-30T14:02:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 -- 172.21.15.46:0/1154947939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f600c6b90 con 0x7f5f601538a0 2023-10-30T14:02:25.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f47fff640 1 -- 172.21.15.46:0/1154947939 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f5f600c6b90 con 0x7f5f601538a0 2023-10-30T14:02:25.376 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:02:25.376 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:16.527331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:02:25.377 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 171 2023-10-30T14:02:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 -- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f4c0664d0 msgr2=0x7f5f4c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f4c0664d0 0x7f5f4c068990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5f6015d100 tx=0x7f5f50009040 comp rx=0 tx=0).stop 2023-10-30T14:02:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 -- 172.21.15.46:0/1154947939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f601538a0 msgr2=0x7f5f6015c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:25.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f601538a0 0x7f5f6015c740 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f5f5401c960 tx=0x7f5f5401bad0 comp rx=0 tx=0).stop 2023-10-30T14:02:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 -- 172.21.15.46:0/1154947939 shutdown_connections 2023-10-30T14:02:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f4c0664d0 0x7f5f4c068990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f6015d6f0 0x7f5f600c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f601538a0 0x7f5f6015c740 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:25.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 --2- 172.21.15.46:0/1154947939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f600cf160 0x7f5f6015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 -- 172.21.15.46:0/1154947939 >> 172.21.15.46:0/1154947939 conn(0x7f5f6006bfe0 msgr2=0x7f5f600cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 -- 172.21.15.46:0/1154947939 shutdown_connections 2023-10-30T14:02:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:25.373+0000 7f5f65a79640 1 -- 172.21.15.46:0/1154947939 wait complete. 2023-10-30T14:02:25.397 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T14:02:27.397 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:02:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90aab3f640 1 Processor -- start 2023-10-30T14:02:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90aab3f640 1 -- start start 2023-10-30T14:02:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90aab3f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90a415cd50 0x7f90a40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90aab3f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a415a6c0 con 0x7f90a41538f0 2023-10-30T14:02:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90aab3f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a415d300 con 0x7f90a40cefa0 2023-10-30T14:02:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90aab3f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a415d5e0 con 0x7f90a415cd50 2023-10-30T14:02:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90a933c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90a415cd50 0x7f90a40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90a933c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90a415cd50 0x7f90a40c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40694/0 (socket says 172.21.15.46:40694) 2023-10-30T14:02:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90a933c640 1 -- 172.21.15.46:0/3229225529 learned_addr learned my addr 172.21.15.46:0/3229225529 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90a9b3d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f90a41538f0 0x7f90a4159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35056/0 (socket says 172.21.15.46:35056) 2023-10-30T14:02:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90a933c640 1 -- 172.21.15.46:0/3229225529 >> v1:172.21.15.154:6789/0 conn(0x7f90a40cefa0 legacy=0x7f90a41640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90a933c640 1 -- 172.21.15.46:0/3229225529 >> v1:172.21.15.46:6789/0 conn(0x7f90a41538f0 legacy=0x7f90a4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90a933c640 1 -- 172.21.15.46:0/3229225529 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a415d8c0 con 0x7f90a415cd50 2023-10-30T14:02:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.545+0000 7f90a8b3b640 1 -- 172.21.15.46:0/3229225529 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440816500 0 0) 0x7f90a415a6c0 con 0x7f90a41538f0 2023-10-30T14:02:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.549+0000 7f90a933c640 1 --2- 172.21.15.46:0/3229225529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90a415cd50 0x7f90a40c9f90 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f9094009870 tx=0x7f909401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5137f3351c8708 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.549+0000 7f90a8b3b640 1 -- 172.21.15.46:0/3229225529 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90940a8a30 con 0x7f90a415cd50 2023-10-30T14:02:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.549+0000 7f90a8b3b640 1 -- 172.21.15.46:0/3229225529 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90940a8b90 con 0x7f90a415cd50 2023-10-30T14:02:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.549+0000 7f90a8b3b640 1 -- 172.21.15.46:0/3229225529 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90940b0440 con 0x7f90a415cd50 2023-10-30T14:02:27.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.549+0000 7f90aab3f640 1 -- 172.21.15.46:0/3229225529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90a415cd50 msgr2=0x7f90a40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:27.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.549+0000 7f90aab3f640 1 --2- 172.21.15.46:0/3229225529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90a415cd50 0x7f90a40c9f90 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f9094009870 tx=0x7f909401ed30 comp rx=0 tx=0).stop 2023-10-30T14:02:27.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.549+0000 7f90aab3f640 1 -- 172.21.15.46:0/3229225529 shutdown_connections 2023-10-30T14:02:27.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.549+0000 7f90aab3f640 1 --2- 172.21.15.46:0/3229225529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90a415cd50 0x7f90a40c9f90 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.549+0000 7f90aab3f640 1 -- 172.21.15.46:0/3229225529 >> 172.21.15.46:0/3229225529 conn(0x7f90a406a640 msgr2=0x7f90a4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.549+0000 7f90aab3f640 1 -- 172.21.15.46:0/3229225529 shutdown_connections 2023-10-30T14:02:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.549+0000 7f90aab3f640 1 -- 172.21.15.46:0/3229225529 wait complete. 2023-10-30T14:02:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.549+0000 7f90aab3f640 1 Processor -- start 2023-10-30T14:02:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aab3f640 1 -- start start 2023-10-30T14:02:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aab3f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90a40cefa0 0x7f90a40b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aab3f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90a41538f0 0x7f90a40b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aab3f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90a415cd50 0x7f90a415b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aab3f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a4148450 con 0x7f90a415cd50 2023-10-30T14:02:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aab3f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a41f1fb0 con 0x7f90a40cefa0 2023-10-30T14:02:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aab3f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a4146730 con 0x7f90a41538f0 2023-10-30T14:02:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aa33e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90a415cd50 0x7f90a415b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aa33e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90a415cd50 0x7f90a415b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40696/0 (socket says 172.21.15.46:40696) 2023-10-30T14:02:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aa33e640 1 -- 172.21.15.46:0/409710810 learned_addr learned my addr 172.21.15.46:0/409710810 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:02:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90a933c640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90a41538f0 0x7f90a40b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90a9b3d640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90a40cefa0 0x7f90a40b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aa33e640 1 -- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90a41538f0 msgr2=0x7f90a40b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aa33e640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90a41538f0 0x7f90a40b2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aa33e640 1 -- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90a40cefa0 msgr2=0x7f90a40b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aa33e640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90a40cefa0 0x7f90a40b4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aa33e640 1 -- 172.21.15.46:0/409710810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90940094d0 con 0x7f90a415cd50 2023-10-30T14:02:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aa33e640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90a415cd50 0x7f90a415b970 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f909c00c770 tx=0x7f909c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f909a7fc640 1 -- 172.21.15.46:0/409710810 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c003b80 con 0x7f90a415cd50 2023-10-30T14:02:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f909a7fc640 1 -- 172.21.15.46:0/409710810 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909c003ce0 con 0x7f90a415cd50 2023-10-30T14:02:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f909a7fc640 1 -- 172.21.15.46:0/409710810 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c00aa90 con 0x7f90a415cd50 2023-10-30T14:02:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aab3f640 1 -- 172.21.15.46:0/409710810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a4146a10 con 0x7f90a415cd50 2023-10-30T14:02:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aab3f640 1 -- 172.21.15.46:0/409710810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a4146fd0 con 0x7f90a415cd50 2023-10-30T14:02:27.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.553+0000 7f90aab3f640 1 -- 172.21.15.46:0/409710810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a41f2120 con 0x7f90a415cd50 2023-10-30T14:02:27.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.557+0000 7f909a7fc640 1 -- 172.21.15.46:0/409710810 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f909c00abf0 con 0x7f90a415cd50 2023-10-30T14:02:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.557+0000 7f909a7fc640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90880664d0 0x7f9088068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:02:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.557+0000 7f909a7fc640 1 -- 172.21.15.46:0/409710810 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f909c084340 con 0x7f90a415cd50 2023-10-30T14:02:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.557+0000 7f90a9b3d640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90880664d0 0x7f9088068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:02:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.557+0000 7f90a9b3d640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90880664d0 0x7f9088068990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f90a0000f70 tx=0x7f90a0007240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:02:27.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.561+0000 7f909a7fc640 1 -- 172.21.15.46:0/409710810 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f909c051090 con 0x7f90a415cd50 2023-10-30T14:02:27.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.765+0000 7f90aab3f640 1 -- 172.21.15.46:0/409710810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90a404e170 con 0x7f90a415cd50 2023-10-30T14:02:27.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f909a7fc640 1 -- 172.21.15.46:0/409710810 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+6524 (secure 0 0 0) 0x7f909c0546b0 con 0x7f90a415cd50 2023-10-30T14:02:27.772 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:02:27.772 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:16.527331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:02:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 172 2023-10-30T14:02:27.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f90aab3f640 1 -- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90880664d0 msgr2=0x7f9088068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:27.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f90aab3f640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90880664d0 0x7f9088068990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f90a0000f70 tx=0x7f90a0007240 comp rx=0 tx=0).stop 2023-10-30T14:02:27.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f90aab3f640 1 -- 172.21.15.46:0/409710810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90a415cd50 msgr2=0x7f90a415b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:02:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f90aab3f640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90a415cd50 0x7f90a415b970 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f909c00c770 tx=0x7f909c00cc30 comp rx=0 tx=0).stop 2023-10-30T14:02:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f90aab3f640 1 -- 172.21.15.46:0/409710810 shutdown_connections 2023-10-30T14:02:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f90aab3f640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90880664d0 0x7f9088068990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f90aab3f640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90a415cd50 0x7f90a415b970 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f90aab3f640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90a41538f0 0x7f90a40b2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:27.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f90aab3f640 1 --2- 172.21.15.46:0/409710810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90a40cefa0 0x7f90a40b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:02:27.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f90aab3f640 1 -- 172.21.15.46:0/409710810 >> 172.21.15.46:0/409710810 conn(0x7f90a406a640 msgr2=0x7f90a40cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:02:27.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f90aab3f640 1 -- 172.21.15.46:0/409710810 shutdown_connections 2023-10-30T14:02:27.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:02:27.769+0000 7f90aab3f640 1 -- 172.21.15.46:0/409710810 wait complete. 2023-10-30T14:02:27.793 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T14:02:27.793 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 104 secs before thrashing 2023-10-30T14:02:37.572 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 13% (25/191) Updating files: 14% (27/191) Updating files: 15% (29/191) Updating files: 16% (31/191) Updating files: 17% (33/191) Updating files: 18% (35/191) Updating files: 19% (37/191) Updating files: 20% (39/191) Updating files: 20% (40/191) Updating files: 21% (41/191) Updating files: 22% (43/191) Updating files: 23% (44/191) Updating files: 24% (46/191) Updating files: 25% (48/191) Updating files: 26% (50/191) Updating files: 27% (52/191) Updating files: 28% (54/191) Updating files: 29% (56/191) Updating files: 30% (58/191) Updating files: 31% (60/191) Updating files: 32% (62/191) Updating files: 33% (64/191) Updating files: 34% (65/191) Updating files: 35% (67/191) Updating files: 36% (69/191) Updating files: 37% (71/191) Updating files: 38% (73/191) Updating files: 39% (75/191) Updating files: 40% (77/191) Updating files: 41% (79/191) Updating files: 42% (81/191) Updating files: 43% (83/191) Updating files: 44% (85/191) Updating files: 45% (86/191) Updating files: 46% (88/191) Updating files: 47% (90/191) Updating files: 48% (92/191) Updating files: 49% (94/191) Updating files: 50% (96/191) Updating files: 51% (98/191) Updating files: 52% (100/191) Updating files: 53% (102/191) Updating files: 54% (104/191) Updating files: 54% (105/191) Updating files: 55% (106/191) Updating files: 56% (107/191) Updating files: 57% (109/191) Updating files: 58% (111/191) Updating files: 59% (113/191) Updating files: 60% (115/191) Updating files: 61% (117/191) Updating files: 62% (119/191) Updating files: 63% (121/191) Updating files: 64% (123/191) Updating files: 65% (125/191) Updating files: 66% (127/191) Updating files: 67% (128/191) Updating files: 68% (130/191) Updating files: 69% (132/191) Updating files: 70% (134/191) Updating files: 71% (136/191) Updating files: 72% (138/191) Updating files: 73% (140/191) Updating files: 74% (142/191) Updating files: 75% (144/191) Updating files: 76% (146/191) Updating files: 77% (148/191) Updating files: 78% (149/191) Updating files: 79% (151/191) Updating files: 80% (153/191) Updating files: 81% (155/191) Updating files: 82% (157/191) Updating files: 83% (159/191) Updating files: 83% (160/191) Updating files: 84% (161/191) Updating files: 85% (163/191) Updating files: 86% (165/191) Updating files: 87% (167/191) Updating files: 87% (168/191) Updating files: 88% (169/191) Updating files: 89% (170/191) Updating files: 90% (172/191) Updating files: 91% (174/191) Updating files: 92% (176/191) Updating files: 93% (178/191) Updating files: 94% (180/191) Updating files: 95% (182/191) Updating files: 96% (184/191) Updating files: 97% (186/191) Updating files: 98% (188/191) Updating files: 99% (190/191) Updating files: 100% (191/191) Updating files: 100% (191/191), done. 2023-10-30T14:02:38.620 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 5832e2603c7 v0.72 2023-10-30T14:02:38.627 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.72 2023-10-30T14:02:38.708 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 73 -eq 48 ] 2023-10-30T14:02:38.708 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.73 2023-10-30T14:02:38.708 INFO:tasks.workunit.client.0.smithi046.stdout:v0.73 2023-10-30T14:02:38.708 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.73 2023-10-30T14:02:38.709 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.73 2023-10-30T14:02:45.899 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 54% (80/148) Updating files: 55% (82/148) Updating files: 56% (83/148) Updating files: 57% (85/148) Updating files: 58% (86/148) Updating files: 59% (88/148) Updating files: 60% (89/148) Updating files: 61% (91/148) Updating files: 62% (92/148) Updating files: 63% (94/148) Updating files: 64% (95/148) Updating files: 65% (97/148) Updating files: 66% (98/148) Updating files: 67% (100/148) Updating files: 68% (101/148) Updating files: 69% (103/148) Updating files: 70% (104/148) Updating files: 71% (106/148) Updating files: 72% (107/148) Updating files: 73% (109/148) Updating files: 74% (110/148) Updating files: 75% (111/148) Updating files: 76% (113/148) Updating files: 77% (114/148) Updating files: 78% (116/148) Updating files: 79% (117/148) Updating files: 80% (119/148) Updating files: 81% (120/148) Updating files: 82% (122/148) Updating files: 83% (123/148) Updating files: 84% (125/148) Updating files: 85% (126/148) Updating files: 86% (128/148) Updating files: 87% (129/148) Updating files: 88% (131/148) Updating files: 89% (132/148) Updating files: 90% (134/148) Updating files: 91% (135/148) Updating files: 92% (137/148) Updating files: 93% (138/148) Updating files: 94% (140/148) Updating files: 95% (141/148) Updating files: 96% (143/148) Updating files: 97% (144/148) Updating files: 98% (146/148) Updating files: 99% (147/148) Updating files: 100% (148/148) Updating files: 100% (148/148), done. 2023-10-30T14:02:46.053 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at d8ad51ee8ab v0.73 2023-10-30T14:02:46.058 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.73 2023-10-30T14:02:46.662 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 74 -eq 48 ] 2023-10-30T14:02:46.663 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.74 2023-10-30T14:02:46.663 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.74 2023-10-30T14:02:46.663 INFO:tasks.workunit.client.0.smithi046.stdout:v0.74 2023-10-30T14:02:46.663 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.74 2023-10-30T14:02:52.119 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 28% (54/192) Updating files: 29% (56/192) Updating files: 30% (58/192) Updating files: 31% (60/192) Updating files: 32% (62/192) Updating files: 33% (64/192) Updating files: 34% (66/192) Updating files: 35% (68/192) Updating files: 36% (70/192) Updating files: 37% (72/192) Updating files: 38% (73/192) Updating files: 39% (75/192) Updating files: 40% (77/192) Updating files: 41% (79/192) Updating files: 42% (81/192) Updating files: 43% (83/192) Updating files: 44% (85/192) Updating files: 45% (87/192) Updating files: 46% (89/192) Updating files: 47% (91/192) Updating files: 48% (93/192) Updating files: 49% (95/192) Updating files: 50% (96/192) Updating files: 51% (98/192) Updating files: 52% (100/192) Updating files: 53% (102/192) Updating files: 54% (104/192) Updating files: 55% (106/192) Updating files: 56% (108/192) Updating files: 57% (110/192) Updating files: 58% (112/192) Updating files: 59% (114/192) Updating files: 60% (116/192) Updating files: 61% (118/192) Updating files: 62% (120/192) Updating files: 63% (121/192) Updating files: 64% (123/192) Updating files: 65% (125/192) Updating files: 66% (127/192) Updating files: 67% (129/192) Updating files: 68% (131/192) Updating files: 69% (133/192) Updating files: 70% (135/192) Updating files: 71% (137/192) Updating files: 72% (139/192) Updating files: 73% (141/192) Updating files: 74% (143/192) Updating files: 75% (144/192) Updating files: 76% (146/192) Updating files: 77% (148/192) Updating files: 78% (150/192) Updating files: 79% (152/192) Updating files: 80% (154/192) Updating files: 81% (156/192) Updating files: 82% (158/192) Updating files: 83% (160/192) Updating files: 84% (162/192) Updating files: 85% (164/192) Updating files: 86% (166/192) Updating files: 87% (168/192) Updating files: 88% (169/192) Updating files: 89% (171/192) Updating files: 90% (173/192) Updating files: 91% (175/192) Updating files: 92% (177/192) Updating files: 93% (179/192) Updating files: 94% (181/192) Updating files: 95% (183/192) Updating files: 96% (185/192) Updating files: 97% (187/192) Updating files: 98% (189/192) Updating files: 99% (191/192) Updating files: 100% (192/192) Updating files: 100% (192/192), done. 2023-10-30T14:02:52.254 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at c165483bc72 v0.74 2023-10-30T14:02:52.260 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.74 2023-10-30T14:02:52.481 INFO:tasks.workunit.client.0.smithi046.stdout:v0.75 2023-10-30T14:02:52.481 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 75 -eq 48 ] 2023-10-30T14:02:52.481 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.75 2023-10-30T14:02:52.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.75 2023-10-30T14:02:52.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.75 2023-10-30T14:02:54.612 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:02:54.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:03:10.605 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 27% (79/291) Updating files: 28% (82/291) Updating files: 29% (85/291) Updating files: 30% (88/291) Updating files: 31% (91/291) Updating files: 32% (94/291) Updating files: 33% (97/291) Updating files: 34% (99/291) Updating files: 35% (102/291) Updating files: 36% (105/291) Updating files: 37% (108/291) Updating files: 38% (111/291) Updating files: 39% (114/291) Updating files: 40% (117/291) Updating files: 41% (120/291) Updating files: 42% (123/291) Updating files: 43% (126/291) Updating files: 44% (129/291) Updating files: 45% (131/291) Updating files: 46% (134/291) Updating files: 47% (137/291) Updating files: 48% (140/291) Updating files: 49% (143/291) Updating files: 50% (146/291) Updating files: 51% (149/291) Updating files: 52% (152/291) Updating files: 53% (155/291) Updating files: 54% (158/291) Updating files: 54% (159/291) Updating files: 55% (161/291) Updating files: 56% (163/291) Updating files: 57% (166/291) Updating files: 58% (169/291) Updating files: 59% (172/291) Updating files: 60% (175/291) Updating files: 61% (178/291) Updating files: 62% (181/291) Updating files: 62% (183/291) Updating files: 63% (184/291) Updating files: 64% (187/291) Updating files: 65% (190/291) Updating files: 66% (193/291) Updating files: 67% (195/291) Updating files: 68% (198/291) Updating files: 69% (201/291) Updating files: 70% (204/291) Updating files: 71% (207/291) Updating files: 72% (210/291) Updating files: 73% (213/291) Updating files: 74% (216/291) Updating files: 75% (219/291) Updating files: 76% (222/291) Updating files: 77% (225/291) Updating files: 78% (227/291) Updating files: 79% (230/291) Updating files: 80% (233/291) Updating files: 81% (236/291) Updating files: 82% (239/291) Updating files: 83% (242/291) Updating files: 84% (245/291) Updating files: 85% (248/291) Updating files: 86% (251/291) Updating files: 87% (254/291) Updating files: 88% (257/291) Updating files: 89% (259/291) Updating files: 90% (262/291) Updating files: 91% (265/291) Updating files: 92% (268/291) Updating files: 93% (271/291) Updating files: 94% (274/291) Updating files: 95% (277/291) Updating files: 96% (280/291) Updating files: 97% (283/291) Updating files: 98% (286/291) Updating files: 99% (289/291) Updating files: 100% (291/291) Updating files: 100% (291/291), done. 2023-10-30T14:03:10.698 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 946d6036958 v0.75 2023-10-30T14:03:10.704 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.75 2023-10-30T14:03:11.195 INFO:tasks.workunit.client.0.smithi046.stdout:v0.76 2023-10-30T14:03:11.196 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 76 -eq 48 ] 2023-10-30T14:03:11.196 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.76 2023-10-30T14:03:11.196 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.76 2023-10-30T14:03:11.196 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.76 2023-10-30T14:03:14.224 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 58% (99/168) Updating files: 59% (100/168) Updating files: 60% (101/168) Updating files: 61% (103/168) Updating files: 62% (105/168) Updating files: 63% (106/168) Updating files: 64% (108/168) Updating files: 65% (110/168) Updating files: 66% (111/168) Updating files: 67% (113/168) Updating files: 68% (115/168) Updating files: 69% (116/168) Updating files: 70% (118/168) Updating files: 71% (120/168) Updating files: 72% (121/168) Updating files: 73% (123/168) Updating files: 74% (125/168) Updating files: 75% (126/168) Updating files: 76% (128/168) Updating files: 77% (130/168) Updating files: 78% (132/168) Updating files: 79% (133/168) Updating files: 80% (135/168) Updating files: 81% (137/168) Updating files: 82% (138/168) Updating files: 83% (140/168) Updating files: 84% (142/168) Updating files: 85% (143/168) Updating files: 86% (145/168) Updating files: 87% (147/168) Updating files: 88% (148/168) Updating files: 89% (150/168) Updating files: 90% (152/168) Updating files: 91% (153/168) Updating files: 92% (155/168) Updating files: 93% (157/168) Updating files: 94% (158/168) Updating files: 95% (160/168) Updating files: 96% (162/168) Updating files: 97% (163/168) Updating files: 98% (165/168) Updating files: 99% (167/168) Updating files: 100% (168/168) Updating files: 100% (168/168), done. 2023-10-30T14:03:14.601 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 3b990136bfa v0.76 2023-10-30T14:03:14.606 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.76 2023-10-30T14:03:15.549 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 77 -eq 48 ] 2023-10-30T14:03:15.550 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.77 2023-10-30T14:03:15.550 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.77 2023-10-30T14:03:15.550 INFO:tasks.workunit.client.0.smithi046.stdout:v0.77 2023-10-30T14:03:15.550 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.77 2023-10-30T14:03:24.670 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:03:24.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:03:25.204 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 19% (57/298) Updating files: 20% (60/298) Updating files: 21% (63/298) Updating files: 22% (66/298) Updating files: 23% (69/298) Updating files: 24% (72/298) Updating files: 25% (75/298) Updating files: 26% (78/298) Updating files: 27% (81/298) Updating files: 28% (84/298) Updating files: 28% (86/298) Updating files: 29% (87/298) Updating files: 30% (90/298) Updating files: 31% (93/298) Updating files: 32% (96/298) Updating files: 33% (99/298) Updating files: 33% (100/298) Updating files: 34% (102/298) Updating files: 35% (105/298) Updating files: 36% (108/298) Updating files: 37% (111/298) Updating files: 38% (114/298) Updating files: 39% (117/298) Updating files: 40% (120/298) Updating files: 41% (123/298) Updating files: 42% (126/298) Updating files: 43% (129/298) Updating files: 44% (132/298) Updating files: 45% (135/298) Updating files: 46% (138/298) Updating files: 47% (141/298) Updating files: 48% (144/298) Updating files: 49% (147/298) Updating files: 50% (149/298) Updating files: 51% (152/298) Updating files: 52% (155/298) Updating files: 53% (158/298) Updating files: 54% (161/298) Updating files: 55% (164/298) Updating files: 56% (167/298) Updating files: 56% (169/298) Updating files: 57% (170/298) Updating files: 58% (173/298) Updating files: 59% (176/298) Updating files: 60% (179/298) Updating files: 61% (182/298) Updating files: 62% (185/298) Updating files: 63% (188/298) Updating files: 64% (191/298) Updating files: 65% (194/298) Updating files: 66% (197/298) Updating files: 67% (200/298) Updating files: 68% (203/298) Updating files: 69% (206/298) Updating files: 70% (209/298) Updating files: 71% (212/298) Updating files: 72% (215/298) Updating files: 73% (218/298) Updating files: 74% (221/298) Updating files: 75% (224/298) Updating files: 76% (227/298) Updating files: 77% (230/298) Updating files: 78% (233/298) Updating files: 79% (236/298) Updating files: 80% (239/298) Updating files: 81% (242/298) Updating files: 82% (245/298) Updating files: 83% (248/298) Updating files: 84% (251/298) Updating files: 85% (254/298) Updating files: 86% (257/298) Updating files: 87% (260/298) Updating files: 88% (263/298) Updating files: 89% (266/298) Updating files: 90% (269/298) Updating files: 91% (272/298) Updating files: 92% (275/298) Updating files: 93% (278/298) Updating files: 94% (281/298) Updating files: 95% (284/298) Updating files: 96% (287/298) Updating files: 97% (290/298) Updating files: 98% (293/298) Updating files: 99% (296/298) Updating files: 100% (298/298) Updating files: 100% (298/298), done. 2023-10-30T14:03:25.553 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 1bca9c5c412 v0.77 2023-10-30T14:03:25.560 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.77 2023-10-30T14:03:25.731 INFO:tasks.workunit.client.0.smithi046.stdout:v0.78 2023-10-30T14:03:25.731 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 78 -eq 48 ] 2023-10-30T14:03:25.731 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.78 2023-10-30T14:03:25.732 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.78 2023-10-30T14:03:25.732 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.78 2023-10-30T14:03:48.050 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 2% (12/496) Updating files: 3% (15/496) Updating files: 4% (20/496) Updating files: 5% (25/496) Updating files: 6% (30/496) Updating files: 7% (35/496) Updating files: 8% (40/496) Updating files: 9% (45/496) Updating files: 10% (50/496) Updating files: 11% (55/496) Updating files: 12% (60/496) Updating files: 13% (65/496) Updating files: 14% (70/496) Updating files: 15% (75/496) Updating files: 16% (80/496) Updating files: 16% (81/496) Updating files: 17% (85/496) Updating files: 18% (90/496) Updating files: 19% (95/496) Updating files: 20% (100/496) Updating files: 21% (105/496) Updating files: 22% (110/496) Updating files: 23% (115/496) Updating files: 24% (120/496) Updating files: 25% (124/496) Updating files: 26% (129/496) Updating files: 27% (134/496) Updating files: 28% (139/496) Updating files: 29% (144/496) Updating files: 30% (149/496) Updating files: 31% (154/496) Updating files: 32% (159/496) Updating files: 33% (164/496) Updating files: 34% (169/496) Updating files: 35% (174/496) Updating files: 36% (179/496) Updating files: 37% (184/496) Updating files: 38% (189/496) Updating files: 39% (194/496) Updating files: 40% (199/496) Updating files: 41% (204/496) Updating files: 41% (206/496) Updating files: 42% (209/496) Updating files: 43% (214/496) Updating files: 44% (219/496) Updating files: 45% (224/496) Updating files: 46% (229/496) Updating files: 47% (234/496) Updating files: 48% (239/496) Updating files: 49% (244/496) Updating files: 50% (248/496) Updating files: 51% (253/496) Updating files: 52% (258/496) Updating files: 53% (263/496) Updating files: 54% (268/496) Updating files: 55% (273/496) Updating files: 56% (278/496) Updating files: 57% (283/496) Updating files: 58% (288/496) Updating files: 59% (293/496) Updating files: 60% (298/496) Updating files: 61% (303/496) Updating files: 62% (308/496) Updating files: 63% (313/496) Updating files: 64% (318/496) Updating files: 65% (323/496) Updating files: 66% (328/496) Updating files: 66% (330/496) Updating files: 67% (333/496) Updating files: 68% (338/496) Updating files: 69% (343/496) Updating files: 70% (348/496) Updating files: 70% (350/496) Updating files: 71% (353/496) Updating files: 72% (358/496) Updating files: 73% (363/496) Updating files: 74% (368/496) Updating files: 75% (372/496) Updating files: 76% (377/496) Updating files: 77% (382/496) Updating files: 78% (387/496) Updating files: 78% (388/496) Updating files: 79% (392/496) Updating files: 80% (397/496) Updating files: 81% (402/496) Updating files: 82% (407/496) Updating files: 82% (409/496) Updating files: 83% (412/496) Updating files: 84% (417/496) Updating files: 85% (422/496) Updating files: 86% (427/496) Updating files: 87% (432/496) Updating files: 88% (437/496) Updating files: 89% (442/496) Updating files: 90% (447/496) Updating files: 91% (452/496) Updating files: 92% (457/496) Updating files: 93% (462/496) Updating files: 94% (467/496) Updating files: 95% (472/496) Updating files: 96% (477/496) Updating files: 96% (480/496) Updating files: 97% (482/496) Updating files: 98% (487/496) Updating files: 99% (492/496) Updating files: 100% (496/496) Updating files: 100% (496/496), done. 2023-10-30T14:03:48.149 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at f6c746c314d 0.78 2023-10-30T14:03:48.156 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.78 2023-10-30T14:03:48.209 INFO:tasks.workunit.client.0.smithi046.stdout:v0.79 2023-10-30T14:03:48.210 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 79 -eq 48 ] 2023-10-30T14:03:48.210 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.79 2023-10-30T14:03:48.210 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.79 2023-10-30T14:03:48.210 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.79 2023-10-30T14:03:55.144 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:03:55.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:03:59.075 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 37% (124/329) Updating files: 38% (126/329) Updating files: 39% (129/329) Updating files: 40% (132/329) Updating files: 41% (135/329) Updating files: 42% (139/329) Updating files: 43% (142/329) Updating files: 43% (144/329) Updating files: 44% (145/329) Updating files: 45% (149/329) Updating files: 46% (152/329) Updating files: 47% (155/329) Updating files: 48% (158/329) Updating files: 49% (162/329) Updating files: 50% (165/329) Updating files: 51% (168/329) Updating files: 52% (172/329) Updating files: 53% (175/329) Updating files: 54% (178/329) Updating files: 55% (181/329) Updating files: 55% (184/329) Updating files: 56% (185/329) Updating files: 57% (188/329) Updating files: 58% (191/329) Updating files: 59% (195/329) Updating files: 60% (198/329) Updating files: 61% (201/329) Updating files: 62% (204/329) Updating files: 63% (208/329) Updating files: 64% (211/329) Updating files: 65% (214/329) Updating files: 66% (218/329) Updating files: 67% (221/329) Updating files: 68% (224/329) Updating files: 69% (228/329) Updating files: 70% (231/329) Updating files: 71% (234/329) Updating files: 72% (237/329) Updating files: 73% (241/329) Updating files: 74% (244/329) Updating files: 75% (247/329) Updating files: 76% (251/329) Updating files: 77% (254/329) Updating files: 78% (257/329) Updating files: 79% (260/329) Updating files: 80% (264/329) Updating files: 81% (267/329) Updating files: 82% (270/329) Updating files: 83% (274/329) Updating files: 84% (277/329) Updating files: 85% (280/329) Updating files: 86% (283/329) Updating files: 87% (287/329) Updating files: 88% (290/329) Updating files: 89% (293/329) Updating files: 90% (297/329) Updating files: 91% (300/329) Updating files: 92% (303/329) Updating files: 93% (306/329) Updating files: 94% (310/329) Updating files: 95% (313/329) Updating files: 96% (316/329) Updating files: 97% (320/329) Updating files: 98% (323/329) Updating files: 99% (326/329) Updating files: 100% (329/329) Updating files: 100% (329/329), done. 2023-10-30T14:03:59.192 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 4c2d73a5095 0.79 2023-10-30T14:03:59.198 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.79 2023-10-30T14:03:59.255 INFO:tasks.workunit.client.0.smithi046.stdout:v0.80 2023-10-30T14:03:59.256 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 80 -eq 48 ] 2023-10-30T14:03:59.256 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.80 2023-10-30T14:03:59.256 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.80 2023-10-30T14:03:59.256 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.80 2023-10-30T14:04:02.215 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 71% (126/176) Updating files: 72% (127/176) Updating files: 73% (129/176) Updating files: 74% (131/176) Updating files: 75% (132/176) Updating files: 76% (134/176) Updating files: 77% (136/176) Updating files: 78% (138/176) Updating files: 79% (140/176) Updating files: 80% (141/176) Updating files: 81% (143/176) Updating files: 82% (145/176) Updating files: 83% (147/176) Updating files: 84% (148/176) Updating files: 85% (150/176) Updating files: 86% (152/176) Updating files: 87% (154/176) Updating files: 88% (155/176) Updating files: 89% (157/176) Updating files: 90% (159/176) Updating files: 91% (161/176) Updating files: 92% (162/176) Updating files: 93% (164/176) Updating files: 94% (166/176) Updating files: 95% (168/176) Updating files: 96% (169/176) Updating files: 97% (171/176) Updating files: 98% (173/176) Updating files: 99% (175/176) Updating files: 100% (176/176) Updating files: 100% (176/176), done. 2023-10-30T14:04:02.336 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at b78644e7dee 0.80 2023-10-30T14:04:02.344 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.80 2023-10-30T14:04:02.413 INFO:tasks.workunit.client.0.smithi046.stdout:v0.81 2023-10-30T14:04:02.414 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 81 -eq 48 ] 2023-10-30T14:04:02.414 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.81 2023-10-30T14:04:02.414 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.81 2023-10-30T14:04:02.414 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.81 2023-10-30T14:04:11.794 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:11.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f6736380640 1 Processor -- start 2023-10-30T14:04:11.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f6736380640 1 -- start start 2023-10-30T14:04:11.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f6736380640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6730159e60 0x7f673006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:11.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f6736380640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f67300bfec0 0x7f67300c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:11.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f6736380640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673006fa80 con 0x7f673006a720 2023-10-30T14:04:11.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f6736380640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6730071290 con 0x7f6730159e60 2023-10-30T14:04:11.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f6736380640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6730071570 con 0x7f67300bfec0 2023-10-30T14:04:11.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f672ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6730159e60 0x7f673006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:11.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f672ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6730159e60 0x7f673006f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50620/0 (socket says 172.21.15.46:50620) 2023-10-30T14:04:11.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f672ffff640 1 -- 172.21.15.46:0/1611594347 learned_addr learned my addr 172.21.15.46:0/1611594347 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:11.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f672f7fe640 1 --2- 172.21.15.46:0/1611594347 >> v2:172.21.15.154:3300/0 conn(0x7f67300bfec0 0x7f67300c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:11.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f672ffff640 1 -- 172.21.15.46:0/1611594347 >> v2:172.21.15.154:3300/0 conn(0x7f67300bfec0 msgr2=0x7f67300c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:11.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f672ffff640 1 --2- 172.21.15.46:0/1611594347 >> v2:172.21.15.154:3300/0 conn(0x7f67300bfec0 0x7f67300c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:11.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f672ffff640 1 -- 172.21.15.46:0/1611594347 >> v1:172.21.15.46:6789/0 conn(0x7f673006a720 legacy=0x7f6730155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:11.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.944+0000 7f672ffff640 1 -- 172.21.15.46:0/1611594347 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6730071850 con 0x7f6730159e60 2023-10-30T14:04:11.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.948+0000 7f672ffff640 1 --2- 172.21.15.46:0/1611594347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6730159e60 0x7f673006f540 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f673006f200 tx=0x7f672001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f717cf669c0736b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:11.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.948+0000 7f672effd640 1 -- 172.21.15.46:0/1611594347 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67200a89f0 con 0x7f6730159e60 2023-10-30T14:04:11.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.948+0000 7f672effd640 1 -- 172.21.15.46:0/1611594347 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67200a8b50 con 0x7f6730159e60 2023-10-30T14:04:11.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.948+0000 7f672effd640 1 -- 172.21.15.46:0/1611594347 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67200b0440 con 0x7f6730159e60 2023-10-30T14:04:11.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.948+0000 7f6736380640 1 -- 172.21.15.46:0/1611594347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6730159e60 msgr2=0x7f673006f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:11.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.948+0000 7f6736380640 1 --2- 172.21.15.46:0/1611594347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6730159e60 0x7f673006f540 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f673006f200 tx=0x7f672001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:04:11.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.948+0000 7f6736380640 1 -- 172.21.15.46:0/1611594347 shutdown_connections 2023-10-30T14:04:11.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.948+0000 7f6736380640 1 --2- 172.21.15.46:0/1611594347 >> v2:172.21.15.154:3300/0 conn(0x7f67300bfec0 0x7f67300c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:11.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.948+0000 7f6736380640 1 --2- 172.21.15.46:0/1611594347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6730159e60 0x7f673006f540 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:11.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.948+0000 7f6736380640 1 -- 172.21.15.46:0/1611594347 >> 172.21.15.46:0/1611594347 conn(0x7f67300c7910 msgr2=0x7f67300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:11.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.948+0000 7f6736380640 1 -- 172.21.15.46:0/1611594347 shutdown_connections 2023-10-30T14:04:11.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.948+0000 7f6736380640 1 -- 172.21.15.46:0/1611594347 wait complete. 2023-10-30T14:04:11.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f6736380640 1 Processor -- start 2023-10-30T14:04:11.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f6736380640 1 -- start start 2023-10-30T14:04:11.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f6736380640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f673006a720 0x7f6730156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:11.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f6736380640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67300bfec0 0x7f6730150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:11.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f6736380640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6730159e60 0x7f67300c1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:11.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f6736380640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67300c3bb0 con 0x7f6730159e60 2023-10-30T14:04:11.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f6736380640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67300c3e70 con 0x7f673006a720 2023-10-30T14:04:11.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f6736380640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67300c4150 con 0x7f67300bfec0 2023-10-30T14:04:11.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f673491d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6730159e60 0x7f67300c1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:11.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f673491d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6730159e60 0x7f67300c1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50632/0 (socket says 172.21.15.46:50632) 2023-10-30T14:04:11.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f673491d640 1 -- 172.21.15.46:0/647291382 learned_addr learned my addr 172.21.15.46:0/647291382 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:11.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f672ffff640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f673006a720 0x7f6730156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:11.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f672f7fe640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67300bfec0 0x7f6730150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:11.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f673491d640 1 -- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67300bfec0 msgr2=0x7f6730150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f673491d640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67300bfec0 0x7f6730150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f673491d640 1 -- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f673006a720 msgr2=0x7f6730156810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f673491d640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f673006a720 0x7f6730156810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f673491d640 1 -- 172.21.15.46:0/647291382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67200094d0 con 0x7f6730159e60 2023-10-30T14:04:11.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f673491d640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6730159e60 0x7f67300c1ba0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f672800c770 tx=0x7f672800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:11.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f672cff9640 1 -- 172.21.15.46:0/647291382 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728003b80 con 0x7f6730159e60 2023-10-30T14:04:11.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f672cff9640 1 -- 172.21.15.46:0/647291382 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6728003ce0 con 0x7f6730159e60 2023-10-30T14:04:11.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f672cff9640 1 -- 172.21.15.46:0/647291382 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672800aa30 con 0x7f6730159e60 2023-10-30T14:04:11.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.952+0000 7f6736380640 1 -- 172.21.15.46:0/647291382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67300c4430 con 0x7f6730159e60 2023-10-30T14:04:11.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.956+0000 7f6736380640 1 -- 172.21.15.46:0/647291382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67301eee40 con 0x7f6730159e60 2023-10-30T14:04:11.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.956+0000 7f6736380640 1 -- 172.21.15.46:0/647291382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673004e170 con 0x7f6730159e60 2023-10-30T14:04:11.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.956+0000 7f672cff9640 1 -- 172.21.15.46:0/647291382 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f672800ab90 con 0x7f6730159e60 2023-10-30T14:04:11.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.956+0000 7f672cff9640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f671c0664a0 0x7f671c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:11.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.956+0000 7f672cff9640 1 -- 172.21.15.46:0/647291382 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f6728084390 con 0x7f6730159e60 2023-10-30T14:04:11.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.956+0000 7f672ffff640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f671c0664a0 0x7f671c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:11.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.960+0000 7f672ffff640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f671c0664a0 0x7f671c068960 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f67200b2000 tx=0x7f6720009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:11.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:11.960+0000 7f672cff9640 1 -- 172.21.15.46:0/647291382 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6728051040 con 0x7f6730159e60 2023-10-30T14:04:12.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.176+0000 7f6736380640 1 -- 172.21.15.46:0/647291382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6730070fd0 con 0x7f6730159e60 2023-10-30T14:04:12.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.176+0000 7f672cff9640 1 -- 172.21.15.46:0/647291382 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6526 (secure 0 0 0) 0x7f6728054660 con 0x7f6730159e60 2023-10-30T14:04:12.182 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:12.182 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:53.704283+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:12.183 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 174 2023-10-30T14:04:12.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.176+0000 7f6736380640 1 -- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f671c0664a0 msgr2=0x7f671c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:12.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.176+0000 7f6736380640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f671c0664a0 0x7f671c068960 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f67200b2000 tx=0x7f6720009040 comp rx=0 tx=0).stop 2023-10-30T14:04:12.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.176+0000 7f6736380640 1 -- 172.21.15.46:0/647291382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6730159e60 msgr2=0x7f67300c1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:12.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.176+0000 7f6736380640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6730159e60 0x7f67300c1ba0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f672800c770 tx=0x7f672800cc30 comp rx=0 tx=0).stop 2023-10-30T14:04:12.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.176+0000 7f6736380640 1 -- 172.21.15.46:0/647291382 shutdown_connections 2023-10-30T14:04:12.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.176+0000 7f6736380640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f671c0664a0 0x7f671c068960 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.176+0000 7f6736380640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6730159e60 0x7f67300c1ba0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.176+0000 7f6736380640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67300bfec0 0x7f6730150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.176+0000 7f6736380640 1 --2- 172.21.15.46:0/647291382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f673006a720 0x7f6730156810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.176+0000 7f6736380640 1 -- 172.21.15.46:0/647291382 >> 172.21.15.46:0/647291382 conn(0x7f67300c7910 msgr2=0x7f67300c9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:12.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.180+0000 7f6736380640 1 -- 172.21.15.46:0/647291382 shutdown_connections 2023-10-30T14:04:12.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.180+0000 7f6736380640 1 -- 172.21.15.46:0/647291382 wait complete. 2023-10-30T14:04:12.202 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T14:04:12.203 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:04:12.203 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:04:12.235 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:04:12.235 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:12.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed60809640 1 Processor -- start 2023-10-30T14:04:12.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed60809640 1 -- start start 2023-10-30T14:04:12.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed60809640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fed5c15d6f0 0x7fed5c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:12.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed60809640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fed5c0cf160 0x7fed5c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed60809640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5c0cb800 con 0x7fed5c1538a0 2023-10-30T14:04:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed60809640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5c0cb330 con 0x7fed5c15d6f0 2023-10-30T14:04:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed60809640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5c0cad70 con 0x7fed5c0cf160 2023-10-30T14:04:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed5a59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fed5c1538a0 0x7fed5c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39184/0 (socket says 172.21.15.46:39184) 2023-10-30T14:04:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed5a59c640 1 -- 172.21.15.46:0/1497379133 learned_addr learned my addr 172.21.15.46:0/1497379133 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:04:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed51d9b640 1 --2- 172.21.15.46:0/1497379133 >> v2:172.21.15.46:3300/0 conn(0x7fed5c15d6f0 0x7fed5c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed5ad9d640 1 --2- 172.21.15.46:0/1497379133 >> v2:172.21.15.154:3300/0 conn(0x7fed5c0cf160 0x7fed5c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed59d9b640 1 -- 172.21.15.46:0/1497379133 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339604952 0 0) 0x7fed5c0cb800 con 0x7fed5c1538a0 2023-10-30T14:04:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed59d9b640 1 -- 172.21.15.46:0/1497379133 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed3c003530 con 0x7fed5c1538a0 2023-10-30T14:04:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed51d9b640 1 -- 172.21.15.46:0/1497379133 >> v2:172.21.15.154:3300/0 conn(0x7fed5c0cf160 msgr2=0x7fed5c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed51d9b640 1 --2- 172.21.15.46:0/1497379133 >> v2:172.21.15.154:3300/0 conn(0x7fed5c0cf160 0x7fed5c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed51d9b640 1 -- 172.21.15.46:0/1497379133 >> v1:172.21.15.46:6789/0 conn(0x7fed5c1538a0 legacy=0x7fed5c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.424+0000 7fed51d9b640 1 -- 172.21.15.46:0/1497379133 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c0cb050 con 0x7fed5c15d6f0 2023-10-30T14:04:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.428+0000 7fed51d9b640 1 --2- 172.21.15.46:0/1497379133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed5c15d6f0 0x7fed5c15a090 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0b5330 tx=0x7fed4c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c846dd6029917c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.428+0000 7fed59d9b640 1 -- 172.21.15.46:0/1497379133 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c0a8950 con 0x7fed5c15d6f0 2023-10-30T14:04:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.428+0000 7fed59d9b640 1 -- 172.21.15.46:0/1497379133 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c0a8ab0 con 0x7fed5c15d6f0 2023-10-30T14:04:12.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.428+0000 7fed59d9b640 1 -- 172.21.15.46:0/1497379133 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c0a8e40 con 0x7fed5c15d6f0 2023-10-30T14:04:12.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.428+0000 7fed60809640 1 -- 172.21.15.46:0/1497379133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed5c15d6f0 msgr2=0x7fed5c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:12.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.428+0000 7fed60809640 1 --2- 172.21.15.46:0/1497379133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed5c15d6f0 0x7fed5c15a090 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0b5330 tx=0x7fed4c01ec50 comp rx=0 tx=0).stop 2023-10-30T14:04:12.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.428+0000 7fed60809640 1 -- 172.21.15.46:0/1497379133 shutdown_connections 2023-10-30T14:04:12.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.428+0000 7fed60809640 1 --2- 172.21.15.46:0/1497379133 >> v2:172.21.15.154:3300/0 conn(0x7fed5c0cf160 0x7fed5c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.428+0000 7fed60809640 1 --2- 172.21.15.46:0/1497379133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed5c15d6f0 0x7fed5c15a090 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.428+0000 7fed60809640 1 -- 172.21.15.46:0/1497379133 >> 172.21.15.46:0/1497379133 conn(0x7fed5c06bfe0 msgr2=0x7fed5c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.428+0000 7fed60809640 1 -- 172.21.15.46:0/1497379133 shutdown_connections 2023-10-30T14:04:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.428+0000 7fed60809640 1 -- 172.21.15.46:0/1497379133 wait complete. 2023-10-30T14:04:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed60809640 1 Processor -- start 2023-10-30T14:04:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed60809640 1 -- start start 2023-10-30T14:04:12.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed60809640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed5c0cf160 0x7fed5c1ef9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed5a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed5c0cf160 0x7fed5c1ef9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed5a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed5c0cf160 0x7fed5c1ef9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50658/0 (socket says 172.21.15.46:50658) 2023-10-30T14:04:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed60809640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed5c1538a0 0x7fed5c158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed60809640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed5c15d6f0 0x7fed5c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed5a59c640 1 -- 172.21.15.46:0/1417575445 learned_addr learned my addr 172.21.15.46:0/1417575445 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c1eff20 con 0x7fed5c0cf160 2023-10-30T14:04:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c15a8b0 con 0x7fed5c1538a0 2023-10-30T14:04:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c15bf40 con 0x7fed5c15d6f0 2023-10-30T14:04:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed5a59c640 1 -- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed5c15d6f0 msgr2=0x7fed5c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed5a59c640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed5c15d6f0 0x7fed5c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed51d9b640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed5c1538a0 0x7fed5c158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed5a59c640 1 -- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed5c1538a0 msgr2=0x7fed5c158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed5a59c640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed5c1538a0 0x7fed5c158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed5a59c640 1 -- 172.21.15.46:0/1417575445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c009420 con 0x7fed5c0cf160 2023-10-30T14:04:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed5a59c640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed5c0cf160 0x7fed5c1ef9e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fed4400c4d0 tx=0x7fed4400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed537fe640 1 -- 172.21.15.46:0/1417575445 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44005cf0 con 0x7fed5c0cf160 2023-10-30T14:04:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed537fe640 1 -- 172.21.15.46:0/1417575445 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed44005e50 con 0x7fed5c0cf160 2023-10-30T14:04:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed537fe640 1 -- 172.21.15.46:0/1417575445 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44007270 con 0x7fed5c0cf160 2023-10-30T14:04:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed5c15c210 con 0x7fed5c0cf160 2023-10-30T14:04:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed5c15c7d0 con 0x7fed5c0cf160 2023-10-30T14:04:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.432+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5c0b2a50 con 0x7fed5c0cf160 2023-10-30T14:04:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.436+0000 7fed537fe640 1 -- 172.21.15.46:0/1417575445 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed440073d0 con 0x7fed5c0cf160 2023-10-30T14:04:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.436+0000 7fed537fe640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed400664a0 0x7fed40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.436+0000 7fed537fe640 1 -- 172.21.15.46:0/1417575445 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7fed44083c50 con 0x7fed5c0cf160 2023-10-30T14:04:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.436+0000 7fed51d9b640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed400664a0 0x7fed40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.436+0000 7fed51d9b640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed400664a0 0x7fed40068960 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fed4c0b2000 tx=0x7fed4c0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.440+0000 7fed537fe640 1 -- 172.21.15.46:0/1417575445 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fed44050900 con 0x7fed5c0cf160 2023-10-30T14:04:12.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.656+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed5c159860 con 0x7fed5c0cf160 2023-10-30T14:04:12.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed537fe640 1 -- 172.21.15.46:0/1417575445 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6526 (secure 0 0 0) 0x7fed44017020 con 0x7fed5c0cf160 2023-10-30T14:04:12.663 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:12.664 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:53.704283+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:12.664 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 174 2023-10-30T14:04:12.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed400664a0 msgr2=0x7fed40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:12.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed60809640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed400664a0 0x7fed40068960 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fed4c0b2000 tx=0x7fed4c0aa070 comp rx=0 tx=0).stop 2023-10-30T14:04:12.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed5c0cf160 msgr2=0x7fed5c1ef9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:12.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed60809640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed5c0cf160 0x7fed5c1ef9e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fed4400c4d0 tx=0x7fed4400c990 comp rx=0 tx=0).stop 2023-10-30T14:04:12.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 shutdown_connections 2023-10-30T14:04:12.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed60809640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed400664a0 0x7fed40068960 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed60809640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed5c15d6f0 0x7fed5c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed60809640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed5c1538a0 0x7fed5c158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed60809640 1 --2- 172.21.15.46:0/1417575445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed5c0cf160 0x7fed5c1ef9e0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:12.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 >> 172.21.15.46:0/1417575445 conn(0x7fed5c06bfe0 msgr2=0x7fed5c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:12.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 shutdown_connections 2023-10-30T14:04:12.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:12.660+0000 7fed60809640 1 -- 172.21.15.46:0/1417575445 wait complete. 2023-10-30T14:04:12.687 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:04:14.687 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.836+0000 7f329889e640 1 Processor -- start 2023-10-30T14:04:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.836+0000 7f329889e640 1 -- start start 2023-10-30T14:04:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.836+0000 7f329889e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32940c6b80 0x7f32940bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.836+0000 7f329889e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32940c6f60 0x7f32940c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f329889e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32940c0a50 con 0x7f32940c8430 2023-10-30T14:04:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f329889e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f329406f6e0 con 0x7f32940c6b80 2023-10-30T14:04:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f329889e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329406f9c0 con 0x7f32940c6f60 2023-10-30T14:04:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f329259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32940c6b80 0x7f32940bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f329259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32940c6b80 0x7f32940bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50672/0 (socket says 172.21.15.46:50672) 2023-10-30T14:04:14.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f3292d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f32940c8430 0x7f329413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39194/0 (socket says 172.21.15.46:39194) 2023-10-30T14:04:14.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f329259c640 1 -- 172.21.15.46:0/3190607317 learned_addr learned my addr 172.21.15.46:0/3190607317 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:14.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f328bfff640 1 --2- 172.21.15.46:0/3190607317 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32940c6f60 0x7f32940c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f329259c640 1 -- 172.21.15.46:0/3190607317 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32940c6f60 msgr2=0x7f32940c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f329259c640 1 --2- 172.21.15.46:0/3190607317 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32940c6f60 0x7f32940c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f329259c640 1 -- 172.21.15.46:0/3190607317 >> v1:172.21.15.46:6789/0 conn(0x7f32940c8430 legacy=0x7f329413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f329259c640 1 -- 172.21.15.46:0/3190607317 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329406fca0 con 0x7f32940c6b80 2023-10-30T14:04:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f3291d9b640 1 -- 172.21.15.46:0/3190607317 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198042549 0 0) 0x7f32940c0a50 con 0x7f32940c8430 2023-10-30T14:04:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f329259c640 1 --2- 172.21.15.46:0/3190607317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32940c6b80 0x7f32940bfec0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f3284009870 tx=0x7f328401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b098555b165d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.840+0000 7f3291d9b640 1 -- 172.21.15.46:0/3190607317 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32840ac620 con 0x7f32940c6b80 2023-10-30T14:04:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f3291d9b640 1 -- 172.21.15.46:0/3190607317 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32840ac780 con 0x7f32940c6b80 2023-10-30T14:04:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f3291d9b640 1 -- 172.21.15.46:0/3190607317 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32840acb10 con 0x7f32940c6b80 2023-10-30T14:04:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 -- 172.21.15.46:0/3190607317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32940c6b80 msgr2=0x7f32940bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 --2- 172.21.15.46:0/3190607317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32940c6b80 0x7f32940bfec0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f3284009870 tx=0x7f328401c4b0 comp rx=0 tx=0).stop 2023-10-30T14:04:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 -- 172.21.15.46:0/3190607317 shutdown_connections 2023-10-30T14:04:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 --2- 172.21.15.46:0/3190607317 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32940c6f60 0x7f32940c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 --2- 172.21.15.46:0/3190607317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32940c6b80 0x7f32940bfec0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:14.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 -- 172.21.15.46:0/3190607317 >> 172.21.15.46:0/3190607317 conn(0x7f32940c7910 msgr2=0x7f32940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:14.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 -- 172.21.15.46:0/3190607317 shutdown_connections 2023-10-30T14:04:14.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 -- 172.21.15.46:0/3190607317 wait complete. 2023-10-30T14:04:14.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 Processor -- start 2023-10-30T14:04:14.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 -- start start 2023-10-30T14:04:14.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32940c6b80 0x7f3294160670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:14.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32940c6f60 0x7f3294139240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:14.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32940c8430 0x7f3294134980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:14.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3294159300 con 0x7f32940c6f60 2023-10-30T14:04:14.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32941598c0 con 0x7f32940c6b80 2023-10-30T14:04:14.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f329889e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32941f5da0 con 0x7f32940c8430 2023-10-30T14:04:14.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f328bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32940c6f60 0x7f3294139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:14.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f328bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32940c6f60 0x7f3294139240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50680/0 (socket says 172.21.15.46:50680) 2023-10-30T14:04:14.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.844+0000 7f328bfff640 1 -- 172.21.15.46:0/744836562 learned_addr learned my addr 172.21.15.46:0/744836562 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:14.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f3292d9d640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32940c8430 0x7f3294134980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:14.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f329259c640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32940c6b80 0x7f3294160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:14.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f328bfff640 1 -- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32940c8430 msgr2=0x7f3294134980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:14.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f328bfff640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32940c8430 0x7f3294134980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:14.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f328bfff640 1 -- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32940c6b80 msgr2=0x7f3294160670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:14.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f328bfff640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32940c6b80 0x7f3294160670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:14.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f328bfff640 1 -- 172.21.15.46:0/744836562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32840094d0 con 0x7f32940c6f60 2023-10-30T14:04:14.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f329259c640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32940c6b80 0x7f3294160670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:04:14.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f328bfff640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32940c6f60 0x7f3294139240 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f327c00d800 tx=0x7f327c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:14.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f328affd640 1 -- 172.21.15.46:0/744836562 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c00b6c0 con 0x7f32940c6f60 2023-10-30T14:04:14.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f328affd640 1 -- 172.21.15.46:0/744836562 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c00b820 con 0x7f32940c6f60 2023-10-30T14:04:14.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f328affd640 1 -- 172.21.15.46:0/744836562 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c00bb90 con 0x7f32940c6f60 2023-10-30T14:04:14.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f329889e640 1 -- 172.21.15.46:0/744836562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3294139780 con 0x7f32940c6f60 2023-10-30T14:04:14.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f329889e640 1 -- 172.21.15.46:0/744836562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32941f6040 con 0x7f32940c6f60 2023-10-30T14:04:14.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f329889e640 1 -- 172.21.15.46:0/744836562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f329404e170 con 0x7f32940c6f60 2023-10-30T14:04:14.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f328affd640 1 -- 172.21.15.46:0/744836562 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f327c00bcf0 con 0x7f32940c6f60 2023-10-30T14:04:14.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f328affd640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32740664a0 0x7f3274068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:14.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.848+0000 7f328affd640 1 -- 172.21.15.46:0/744836562 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f327c0849c0 con 0x7f32940c6f60 2023-10-30T14:04:14.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.852+0000 7f329259c640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32740664a0 0x7f3274068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:14.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.852+0000 7f329259c640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32740664a0 0x7f3274068960 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f32840ae000 tx=0x7f32840a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:14.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:14.856+0000 7f328affd640 1 -- 172.21.15.46:0/744836562 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f327c051780 con 0x7f32940c6f60 2023-10-30T14:04:15.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.044+0000 7f329889e640 1 -- 172.21.15.46:0/744836562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32940b0cf0 con 0x7f32940c6f60 2023-10-30T14:04:15.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.044+0000 7f328affd640 1 -- 172.21.15.46:0/744836562 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6526 (secure 0 0 0) 0x7f327c054da0 con 0x7f32940c6f60 2023-10-30T14:04:15.051 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:15.051 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:53.704283+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:15.052 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 174 2023-10-30T14:04:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.048+0000 7f329889e640 1 -- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32740664a0 msgr2=0x7f3274068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.048+0000 7f329889e640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32740664a0 0x7f3274068960 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f32840ae000 tx=0x7f32840a7000 comp rx=0 tx=0).stop 2023-10-30T14:04:15.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.048+0000 7f329889e640 1 -- 172.21.15.46:0/744836562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32940c6f60 msgr2=0x7f3294139240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.048+0000 7f329889e640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32940c6f60 0x7f3294139240 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f327c00d800 tx=0x7f327c00dcc0 comp rx=0 tx=0).stop 2023-10-30T14:04:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.048+0000 7f329889e640 1 -- 172.21.15.46:0/744836562 shutdown_connections 2023-10-30T14:04:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.048+0000 7f329889e640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f32740664a0 0x7f3274068960 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.048+0000 7f329889e640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32940c8430 0x7f3294134980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:15.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.048+0000 7f329889e640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32940c6f60 0x7f3294139240 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:15.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.048+0000 7f329889e640 1 --2- 172.21.15.46:0/744836562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32940c6b80 0x7f3294160670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:15.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.048+0000 7f329889e640 1 -- 172.21.15.46:0/744836562 >> 172.21.15.46:0/744836562 conn(0x7f32940c7910 msgr2=0x7f32940bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:15.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.048+0000 7f329889e640 1 -- 172.21.15.46:0/744836562 shutdown_connections 2023-10-30T14:04:15.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:15.048+0000 7f329889e640 1 -- 172.21.15.46:0/744836562 wait complete. 2023-10-30T14:04:15.073 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:04:17.074 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:17.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.224+0000 7fbed5f0f640 1 Processor -- start 2023-10-30T14:04:17.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.224+0000 7fbed5f0f640 1 -- start start 2023-10-30T14:04:17.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbed5f0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbed006f7a0 0x7fbed006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:17.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbed5f0f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbed0071110 0x7fbed0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:17.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbed5f0f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed0066cf0 con 0x7fbed00c6bd0 2023-10-30T14:04:17.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbed5f0f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbed00700c0 con 0x7fbed0071110 2023-10-30T14:04:17.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbed5f0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed00c70d0 con 0x7fbed006f7a0 2023-10-30T14:04:17.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbeceffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbed00c6bd0 0x7fbed0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56056/0 (socket says 172.21.15.46:56056) 2023-10-30T14:04:17.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbeceffd640 1 -- 172.21.15.46:0/359922855 learned_addr learned my addr 172.21.15.46:0/359922855 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:04:17.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbecffff640 1 --2- 172.21.15.46:0/359922855 >> v2:172.21.15.154:3300/0 conn(0x7fbed0071110 0x7fbed0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:17.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbecf7fe640 1 --2- 172.21.15.46:0/359922855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbed006f7a0 0x7fbed006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbece7fc640 1 -- 172.21.15.46:0/359922855 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017582287 0 0) 0x7fbed0066cf0 con 0x7fbed00c6bd0 2023-10-30T14:04:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbece7fc640 1 -- 172.21.15.46:0/359922855 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeb8003530 con 0x7fbed00c6bd0 2023-10-30T14:04:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbecffff640 1 -- 172.21.15.46:0/359922855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbed006f7a0 msgr2=0x7fbed006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbecffff640 1 --2- 172.21.15.46:0/359922855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbed006f7a0 0x7fbed006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:17.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbecffff640 1 -- 172.21.15.46:0/359922855 >> v1:172.21.15.46:6789/0 conn(0x7fbed00c6bd0 legacy=0x7fbed0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.228+0000 7fbecffff640 1 -- 172.21.15.46:0/359922855 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed00c7390 con 0x7fbed0071110 2023-10-30T14:04:17.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbecffff640 1 --2- 172.21.15.46:0/359922855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbed0071110 0x7fbed0158110 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009d90 tx=0x7fbec801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd535d5de2e2e690 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:17.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbece7fc640 1 -- 172.21.15.46:0/359922855 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec80af600 con 0x7fbed0071110 2023-10-30T14:04:17.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbece7fc640 1 -- 172.21.15.46:0/359922855 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec80ab050 con 0x7fbed0071110 2023-10-30T14:04:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbece7fc640 1 -- 172.21.15.46:0/359922855 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec80af980 con 0x7fbed0071110 2023-10-30T14:04:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbed5f0f640 1 -- 172.21.15.46:0/359922855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbed0071110 msgr2=0x7fbed0158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbed5f0f640 1 --2- 172.21.15.46:0/359922855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbed0071110 0x7fbed0158110 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009d90 tx=0x7fbec801f130 comp rx=0 tx=0).stop 2023-10-30T14:04:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbed5f0f640 1 -- 172.21.15.46:0/359922855 shutdown_connections 2023-10-30T14:04:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbed5f0f640 1 --2- 172.21.15.46:0/359922855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbed0071110 0x7fbed0158110 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbed5f0f640 1 --2- 172.21.15.46:0/359922855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbed006f7a0 0x7fbed006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:17.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbed5f0f640 1 -- 172.21.15.46:0/359922855 >> 172.21.15.46:0/359922855 conn(0x7fbed00c7910 msgr2=0x7fbed00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:17.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbed5f0f640 1 -- 172.21.15.46:0/359922855 shutdown_connections 2023-10-30T14:04:17.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbed5f0f640 1 -- 172.21.15.46:0/359922855 wait complete. 2023-10-30T14:04:17.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbed5f0f640 1 Processor -- start 2023-10-30T14:04:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.232+0000 7fbed5f0f640 1 -- start start 2023-10-30T14:04:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbed5f0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed006f7a0 0x7fbed01594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbed5f0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbed0071110 0x7fbed015a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbed5f0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbed00c6bd0 0x7fbed015a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbed5f0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed015b8d0 con 0x7fbed006f7a0 2023-10-30T14:04:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbed5f0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed01f28d0 con 0x7fbed00c6bd0 2023-10-30T14:04:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbed5f0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed0159d90 con 0x7fbed0071110 2023-10-30T14:04:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbecf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed006f7a0 0x7fbed01594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbecf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed006f7a0 0x7fbed01594c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38086/0 (socket says 172.21.15.46:38086) 2023-10-30T14:04:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbecf7fe640 1 -- 172.21.15.46:0/2232533913 learned_addr learned my addr 172.21.15.46:0/2232533913 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbeceffd640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbed0071110 0x7fbed015a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:17.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbecffff640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbed00c6bd0 0x7fbed015a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:17.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbecf7fe640 1 -- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbed0071110 msgr2=0x7fbed015a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:17.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbecf7fe640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbed0071110 0x7fbed015a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:17.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbecf7fe640 1 -- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbed00c6bd0 msgr2=0x7fbed015a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:17.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbecf7fe640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbed00c6bd0 0x7fbed015a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:17.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbecf7fe640 1 -- 172.21.15.46:0/2232533913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec8009a70 con 0x7fbed006f7a0 2023-10-30T14:04:17.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbecffff640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbed00c6bd0 0x7fbed015a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:04:17.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbeceffd640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbed0071110 0x7fbed015a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:04:17.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbecf7fe640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed006f7a0 0x7fbed01594c0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fbec000e870 tx=0x7fbec000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:17.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbeb7fff640 1 -- 172.21.15.46:0/2232533913 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec0009510 con 0x7fbed006f7a0 2023-10-30T14:04:17.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbeb7fff640 1 -- 172.21.15.46:0/2232533913 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec000eec0 con 0x7fbed006f7a0 2023-10-30T14:04:17.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbeb7fff640 1 -- 172.21.15.46:0/2232533913 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec000cb40 con 0x7fbed006f7a0 2023-10-30T14:04:17.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbed5f0f640 1 -- 172.21.15.46:0/2232533913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed01f2ae0 con 0x7fbed006f7a0 2023-10-30T14:04:17.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbed5f0f640 1 -- 172.21.15.46:0/2232533913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed01f30a0 con 0x7fbed006f7a0 2023-10-30T14:04:17.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbed5f0f640 1 -- 172.21.15.46:0/2232533913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed004e170 con 0x7fbed006f7a0 2023-10-30T14:04:17.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.236+0000 7fbeb7fff640 1 -- 172.21.15.46:0/2232533913 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbec000ccc0 con 0x7fbed006f7a0 2023-10-30T14:04:17.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.240+0000 7fbeb7fff640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbebc0664a0 0x7fbebc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:17.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.240+0000 7fbeb7fff640 1 -- 172.21.15.46:0/2232533913 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7fbec00846d0 con 0x7fbed006f7a0 2023-10-30T14:04:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.240+0000 7fbeceffd640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbebc0664a0 0x7fbebc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.240+0000 7fbeceffd640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbebc0664a0 0x7fbebc068960 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fbed006c7a0 tx=0x7fbec4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.244+0000 7fbeb7fff640 1 -- 172.21.15.46:0/2232533913 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbec00513e0 con 0x7fbed006f7a0 2023-10-30T14:04:17.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.432+0000 7fbed5f0f640 1 -- 172.21.15.46:0/2232533913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbed00b0a40 con 0x7fbed006f7a0 2023-10-30T14:04:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.432+0000 7fbeb7fff640 1 -- 172.21.15.46:0/2232533913 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6526 (secure 0 0 0) 0x7fbec0054a00 con 0x7fbed006f7a0 2023-10-30T14:04:17.438 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:17.438 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:53.704283+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 174 2023-10-30T14:04:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.436+0000 7fbed5f0f640 1 -- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbebc0664a0 msgr2=0x7fbebc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.436+0000 7fbed5f0f640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbebc0664a0 0x7fbebc068960 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fbed006c7a0 tx=0x7fbec4009040 comp rx=0 tx=0).stop 2023-10-30T14:04:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.436+0000 7fbed5f0f640 1 -- 172.21.15.46:0/2232533913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed006f7a0 msgr2=0x7fbed01594c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.436+0000 7fbed5f0f640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed006f7a0 0x7fbed01594c0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fbec000e870 tx=0x7fbec000ec70 comp rx=0 tx=0).stop 2023-10-30T14:04:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.436+0000 7fbed5f0f640 1 -- 172.21.15.46:0/2232533913 shutdown_connections 2023-10-30T14:04:17.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.436+0000 7fbed5f0f640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbebc0664a0 0x7fbebc068960 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.436+0000 7fbed5f0f640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbed00c6bd0 0x7fbed015a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.436+0000 7fbed5f0f640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbed0071110 0x7fbed015a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.436+0000 7fbed5f0f640 1 --2- 172.21.15.46:0/2232533913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbed006f7a0 0x7fbed01594c0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.436+0000 7fbed5f0f640 1 -- 172.21.15.46:0/2232533913 >> 172.21.15.46:0/2232533913 conn(0x7fbed00c7910 msgr2=0x7fbed00c93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.436+0000 7fbed5f0f640 1 -- 172.21.15.46:0/2232533913 shutdown_connections 2023-10-30T14:04:17.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:17.436+0000 7fbed5f0f640 1 -- 172.21.15.46:0/2232533913 wait complete. 2023-10-30T14:04:17.459 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:04:19.461 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:19.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f737f59e640 1 Processor -- start 2023-10-30T14:04:19.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f737f59e640 1 -- start start 2023-10-30T14:04:19.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f737f59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f738015cd50 0x7f73800c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:19.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f737f59e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f73800cefa0 0x7f73800cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:19.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f737f59e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738015a6c0 con 0x7f73801538f0 2023-10-30T14:04:19.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f737f59e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738015d300 con 0x7f73800cefa0 2023-10-30T14:04:19.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f737f59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738015d5e0 con 0x7f738015cd50 2023-10-30T14:04:19.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f7377fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f738015cd50 0x7f73800c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:19.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f737ed9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f73800cefa0 0x7f73800cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f7377fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f738015cd50 0x7f73800c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:40176/0 (socket says 172.21.15.46:40176) 2023-10-30T14:04:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f7377fff640 1 -- 172.21.15.46:0/2711774154 learned_addr learned my addr 172.21.15.46:0/2711774154 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f737ed9d640 1 -- 172.21.15.46:0/2711774154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f738015cd50 msgr2=0x7f73800c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f737ed9d640 1 --2- 172.21.15.46:0/2711774154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f738015cd50 0x7f73800c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f737ed9d640 1 -- 172.21.15.46:0/2711774154 >> v1:172.21.15.154:6789/0 conn(0x7f73801538f0 legacy=0x7f7380159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:19.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f737ed9d640 1 -- 172.21.15.46:0/2711774154 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738015d8c0 con 0x7f73800cefa0 2023-10-30T14:04:19.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.608+0000 7f7377fff640 1 --2- 172.21.15.46:0/2711774154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f738015cd50 0x7f73800c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:04:19.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.612+0000 7f737ed9d640 1 --2- 172.21.15.46:0/2711774154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73800cefa0 0x7f73800cf3f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7378009d90 tx=0x7f737801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=359a4063ed8a8330 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.612+0000 7f737dd9b640 1 -- 172.21.15.46:0/2711774154 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73780af600 con 0x7f73800cefa0 2023-10-30T14:04:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.612+0000 7f737dd9b640 1 -- 172.21.15.46:0/2711774154 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73780ab050 con 0x7f73800cefa0 2023-10-30T14:04:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.612+0000 7f737dd9b640 1 -- 172.21.15.46:0/2711774154 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73780af980 con 0x7f73800cefa0 2023-10-30T14:04:19.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.612+0000 7f737f59e640 1 -- 172.21.15.46:0/2711774154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73800cefa0 msgr2=0x7f73800cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:19.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.612+0000 7f737f59e640 1 --2- 172.21.15.46:0/2711774154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73800cefa0 0x7f73800cf3f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7378009d90 tx=0x7f737801f130 comp rx=0 tx=0).stop 2023-10-30T14:04:19.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.612+0000 7f737f59e640 1 -- 172.21.15.46:0/2711774154 shutdown_connections 2023-10-30T14:04:19.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.612+0000 7f737f59e640 1 --2- 172.21.15.46:0/2711774154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73800cefa0 0x7f73800cf3f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:19.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.612+0000 7f737f59e640 1 --2- 172.21.15.46:0/2711774154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f738015cd50 0x7f73800c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:19.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.612+0000 7f737f59e640 1 -- 172.21.15.46:0/2711774154 >> 172.21.15.46:0/2711774154 conn(0x7f738006a640 msgr2=0x7f7380065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:19.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.612+0000 7f737f59e640 1 -- 172.21.15.46:0/2711774154 shutdown_connections 2023-10-30T14:04:19.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.612+0000 7f737f59e640 1 -- 172.21.15.46:0/2711774154 wait complete. 2023-10-30T14:04:19.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737f59e640 1 Processor -- start 2023-10-30T14:04:19.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737f59e640 1 -- start start 2023-10-30T14:04:19.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737f59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73800cefa0 0x7f73801eeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:19.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737f59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73801538f0 0x7f73800b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:19.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f738015cd50 0x7f7380147c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737f59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73800b5270 con 0x7f738015cd50 2023-10-30T14:04:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737f59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73800b3460 con 0x7f73800cefa0 2023-10-30T14:04:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737f59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7380153050 con 0x7f73801538f0 2023-10-30T14:04:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f738015cd50 0x7f7380147c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f738015cd50 0x7f7380147c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38102/0 (socket says 172.21.15.46:38102) 2023-10-30T14:04:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737ed9d640 1 -- 172.21.15.46:0/137645971 learned_addr learned my addr 172.21.15.46:0/137645971 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f7377fff640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73801538f0 0x7f73800b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737e59c640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73800cefa0 0x7f73801eeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737ed9d640 1 -- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73801538f0 msgr2=0x7f73800b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737ed9d640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73801538f0 0x7f73800b38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737ed9d640 1 -- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73800cefa0 msgr2=0x7f73801eeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737ed9d640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73800cefa0 0x7f73801eeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737ed9d640 1 -- 172.21.15.46:0/137645971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73680094d0 con 0x7f738015cd50 2023-10-30T14:04:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737ed9d640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f738015cd50 0x7f7380147c40 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f7378009d90 tx=0x7f737801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f7376ffd640 1 -- 172.21.15.46:0/137645971 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7378009450 con 0x7f738015cd50 2023-10-30T14:04:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f7376ffd640 1 -- 172.21.15.46:0/137645971 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73780b9410 con 0x7f738015cd50 2023-10-30T14:04:19.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f7376ffd640 1 -- 172.21.15.46:0/137645971 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73780b97c0 con 0x7f738015cd50 2023-10-30T14:04:19.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737f59e640 1 -- 172.21.15.46:0/137645971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7378009a70 con 0x7f738015cd50 2023-10-30T14:04:19.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737f59e640 1 -- 172.21.15.46:0/137645971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7380152940 con 0x7f738015cd50 2023-10-30T14:04:19.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.616+0000 7f737f59e640 1 -- 172.21.15.46:0/137645971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73800b35f0 con 0x7f738015cd50 2023-10-30T14:04:19.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.620+0000 7f7376ffd640 1 -- 172.21.15.46:0/137645971 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f73780b6070 con 0x7f738015cd50 2023-10-30T14:04:19.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.620+0000 7f7376ffd640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f73600664a0 0x7f7360068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:19.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.620+0000 7f7376ffd640 1 -- 172.21.15.46:0/137645971 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f7378122b60 con 0x7f738015cd50 2023-10-30T14:04:19.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.620+0000 7f737e59c640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f73600664a0 0x7f7360068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:19.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.620+0000 7f737e59c640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f73600664a0 0x7f7360068960 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f7370007fd0 tx=0x7f7370009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:19.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.624+0000 7f7376ffd640 1 -- 172.21.15.46:0/137645971 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f73780efa50 con 0x7f738015cd50 2023-10-30T14:04:19.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.816+0000 7f737f59e640 1 -- 172.21.15.46:0/137645971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f738004e170 con 0x7f738015cd50 2023-10-30T14:04:19.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.816+0000 7f7376ffd640 1 -- 172.21.15.46:0/137645971 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6526 (secure 0 0 0) 0x7f73780c7090 con 0x7f738015cd50 2023-10-30T14:04:19.821 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:19.822 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:53.704283+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 174 2023-10-30T14:04:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.820+0000 7f737f59e640 1 -- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f73600664a0 msgr2=0x7f7360068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.820+0000 7f737f59e640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f73600664a0 0x7f7360068960 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f7370007fd0 tx=0x7f7370009040 comp rx=0 tx=0).stop 2023-10-30T14:04:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.820+0000 7f737f59e640 1 -- 172.21.15.46:0/137645971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f738015cd50 msgr2=0x7f7380147c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.820+0000 7f737f59e640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f738015cd50 0x7f7380147c40 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f7378009d90 tx=0x7f737801bed0 comp rx=0 tx=0).stop 2023-10-30T14:04:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.820+0000 7f737f59e640 1 -- 172.21.15.46:0/137645971 shutdown_connections 2023-10-30T14:04:19.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.820+0000 7f737f59e640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f73600664a0 0x7f7360068960 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.820+0000 7f737f59e640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f738015cd50 0x7f7380147c40 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.820+0000 7f737f59e640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73801538f0 0x7f73800b38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.820+0000 7f737f59e640 1 --2- 172.21.15.46:0/137645971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73800cefa0 0x7f73801eeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.820+0000 7f737f59e640 1 -- 172.21.15.46:0/137645971 >> 172.21.15.46:0/137645971 conn(0x7f738006a640 msgr2=0x7f7380065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.820+0000 7f737f59e640 1 -- 172.21.15.46:0/137645971 shutdown_connections 2023-10-30T14:04:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:19.820+0000 7f737f59e640 1 -- 172.21.15.46:0/137645971 wait complete. 2023-10-30T14:04:19.843 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:04:21.844 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:21.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.992+0000 7fa783209640 1 Processor -- start 2023-10-30T14:04:21.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.992+0000 7fa783209640 1 -- start start 2023-10-30T14:04:21.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa783209640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c153bf0 0x7fa77c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:22.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa783209640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0700b0 con 0x7fa77c0c6fe0 2023-10-30T14:04:22.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa783209640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0c6530 con 0x7fa77c0c5f60 2023-10-30T14:04:22.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa783209640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0c6810 con 0x7fa77c153bf0 2023-10-30T14:04:22.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa77c0c6fe0 0x7fa77c0c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56070/0 (socket says 172.21.15.46:56070) 2023-10-30T14:04:22.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77bfff640 1 -- 172.21.15.46:0/841686075 learned_addr learned my addr 172.21.15.46:0/841686075 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:04:22.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa780fa5640 1 --2- 172.21.15.46:0/841686075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c153bf0 0x7fa77c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:22.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389521521 0 0) 0x7fa77c0700b0 con 0x7fa77c0c6fe0 2023-10-30T14:04:22.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c003530 con 0x7fa77c0c6fe0 2023-10-30T14:04:22.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2449273986 0 0) 0x7fa76c003530 con 0x7fa77c0c6fe0 2023-10-30T14:04:22.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c153bf0 msgr2=0x7fa77c071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:22.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 --2- 172.21.15.46:0/841686075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c153bf0 0x7fa77c071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:22.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 >> v1:172.21.15.154:6789/0 conn(0x7fa77c0c5f60 legacy=0x7fa77c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:22.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c159b10 con 0x7fa77c0c6fe0 2023-10-30T14:04:22.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fa768001680 con 0x7fa77c0c6fe0 2023-10-30T14:04:22.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 >> v1:172.21.15.46:6789/0 conn(0x7fa77c0c6fe0 legacy=0x7fa77c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:22.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 --2- 172.21.15.46:0/841686075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa76c0039b0 0x7fa76c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:22.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 --2- 172.21.15.46:0/841686075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa76c0047d0 0x7fa76c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:22.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 --2- 172.21.15.46:0/841686075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c153bf0 0x7fa76c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:22.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0c6810 con 0x7fa76c0047d0 2023-10-30T14:04:22.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c00bad0 con 0x7fa76c0039b0 2023-10-30T14:04:22.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c00bc90 con 0x7fa77c153bf0 2023-10-30T14:04:22.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77bfff640 1 --2- 172.21.15.46:0/841686075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa76c0047d0 0x7fa76c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:22.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa7817a6640 1 --2- 172.21.15.46:0/841686075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c153bf0 0x7fa76c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:22.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa780fa5640 1 --2- 172.21.15.46:0/841686075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa76c0039b0 0x7fa76c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:22.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77bfff640 1 -- 172.21.15.46:0/841686075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c153bf0 msgr2=0x7fa76c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:22.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77bfff640 1 --2- 172.21.15.46:0/841686075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c153bf0 0x7fa76c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:22.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77bfff640 1 -- 172.21.15.46:0/841686075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa76c0039b0 msgr2=0x7fa76c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:22.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77bfff640 1 --2- 172.21.15.46:0/841686075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa76c0039b0 0x7fa76c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:22.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:21.996+0000 7fa77bfff640 1 -- 172.21.15.46:0/841686075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c003530 con 0x7fa76c0047d0 2023-10-30T14:04:22.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.000+0000 7fa77bfff640 1 --2- 172.21.15.46:0/841686075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa76c0047d0 0x7fa76c004c50 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fa76800c210 tx=0x7fa76801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905f6d6f1097ceb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:22.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.000+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7680a5b00 con 0x7fa76c0047d0 2023-10-30T14:04:22.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.000+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7680a5c60 con 0x7fa76c0047d0 2023-10-30T14:04:22.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.000+0000 7fa77b7fe640 1 -- 172.21.15.46:0/841686075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa768005b20 con 0x7fa76c0047d0 2023-10-30T14:04:22.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.000+0000 7fa783209640 1 -- 172.21.15.46:0/841686075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa76c0047d0 msgr2=0x7fa76c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:22.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.000+0000 7fa783209640 1 --2- 172.21.15.46:0/841686075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa76c0047d0 0x7fa76c004c50 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fa76800c210 tx=0x7fa76801cc00 comp rx=0 tx=0).stop 2023-10-30T14:04:22.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.000+0000 7fa7817a6640 1 -- 172.21.15.46:0/841686075 reap_dead start 2023-10-30T14:04:22.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.000+0000 7fa783209640 1 -- 172.21.15.46:0/841686075 shutdown_connections 2023-10-30T14:04:22.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.000+0000 7fa783209640 1 -- 172.21.15.46:0/841686075 >> 172.21.15.46:0/841686075 conn(0x7fa77c06c6e0 msgr2=0x7fa77c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:22.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.000+0000 7fa783209640 1 -- 172.21.15.46:0/841686075 shutdown_connections 2023-10-30T14:04:22.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.000+0000 7fa783209640 1 -- 172.21.15.46:0/841686075 wait complete. 2023-10-30T14:04:22.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.000+0000 7fa783209640 1 Processor -- start 2023-10-30T14:04:22.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa783209640 1 -- start start 2023-10-30T14:04:22.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa783209640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa76c0047d0 0x7fa77c159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:22.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa783209640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c159570 0x7fa77c1f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:22.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa783209640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa77c1f2a70 0x7fa77c154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:22.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa783209640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c1e5230 con 0x7fa77c1f2a70 2023-10-30T14:04:22.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa783209640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0c4fa0 con 0x7fa76c0047d0 2023-10-30T14:04:22.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa783209640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0c5280 con 0x7fa77c159570 2023-10-30T14:04:22.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa7817a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa77c1f2a70 0x7fa77c154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:22.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa7817a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa77c1f2a70 0x7fa77c154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38126/0 (socket says 172.21.15.46:38126) 2023-10-30T14:04:22.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa7817a6640 1 -- 172.21.15.46:0/2286165388 learned_addr learned my addr 172.21.15.46:0/2286165388 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:22.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa77bfff640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c159570 0x7fa77c1f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:22.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa780fa5640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa76c0047d0 0x7fa77c159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:22.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa7817a6640 1 -- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c159570 msgr2=0x7fa77c1f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:22.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa7817a6640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c159570 0x7fa77c1f2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:22.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa7817a6640 1 -- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa76c0047d0 msgr2=0x7fa77c159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:22.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa7817a6640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa76c0047d0 0x7fa77c159030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:22.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa7817a6640 1 -- 172.21.15.46:0/2286165388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c0c5710 con 0x7fa77c1f2a70 2023-10-30T14:04:22.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa780fa5640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa76c0047d0 0x7fa77c159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:04:22.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa7817a6640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa77c1f2a70 0x7fa77c154270 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fa77c06b990 tx=0x7fa77400fcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:22.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa7797fa640 1 -- 172.21.15.46:0/2286165388 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774006d70 con 0x7fa77c1f2a70 2023-10-30T14:04:22.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa7797fa640 1 -- 172.21.15.46:0/2286165388 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa774003980 con 0x7fa77c1f2a70 2023-10-30T14:04:22.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa7797fa640 1 -- 172.21.15.46:0/2286165388 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774003d30 con 0x7fa77c1f2a70 2023-10-30T14:04:22.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa783209640 1 -- 172.21.15.46:0/2286165388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c1552f0 con 0x7fa77c1f2a70 2023-10-30T14:04:22.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa783209640 1 -- 172.21.15.46:0/2286165388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c1e5480 con 0x7fa77c1f2a70 2023-10-30T14:04:22.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.004+0000 7fa783209640 1 -- 172.21.15.46:0/2286165388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c04e170 con 0x7fa77c1f2a70 2023-10-30T14:04:22.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.008+0000 7fa7797fa640 1 -- 172.21.15.46:0/2286165388 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa774005110 con 0x7fa77c1f2a70 2023-10-30T14:04:22.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.008+0000 7fa7797fa640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7600664a0 0x7fa760068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:22.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.008+0000 7fa7797fa640 1 -- 172.21.15.46:0/2286165388 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7fa774083a60 con 0x7fa77c1f2a70 2023-10-30T14:04:22.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.008+0000 7fa780fa5640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7600664a0 0x7fa760068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:22.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.008+0000 7fa780fa5640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7600664a0 0x7fa760068960 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fa770009be0 tx=0x7fa770008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:22.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.012+0000 7fa7797fa640 1 -- 172.21.15.46:0/2286165388 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa774050830 con 0x7fa77c1f2a70 2023-10-30T14:04:22.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 -- 172.21.15.46:0/2286165388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa77c159dc0 con 0x7fa77c1f2a70 2023-10-30T14:04:22.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa7797fa640 1 -- 172.21.15.46:0/2286165388 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+6526 (secure 0 0 0) 0x7fa774053e50 con 0x7fa77c1f2a70 2023-10-30T14:04:22.210 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:22.210 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8190,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:02:53.704283+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7704,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7704":{"gid":7704,"name":"c","rank":0,"incarnation":152,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6836/3734066164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3734066164},{"type":"v1","addr":"172.21.15.46:6836","nonce":3734066164}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:22.211 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 174 2023-10-30T14:04:22.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 -- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7600664a0 msgr2=0x7fa760068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:22.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7600664a0 0x7fa760068960 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fa770009be0 tx=0x7fa770008040 comp rx=0 tx=0).stop 2023-10-30T14:04:22.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 -- 172.21.15.46:0/2286165388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa77c1f2a70 msgr2=0x7fa77c154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:22.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa77c1f2a70 0x7fa77c154270 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fa77c06b990 tx=0x7fa77400fcd0 comp rx=0 tx=0).stop 2023-10-30T14:04:22.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 -- 172.21.15.46:0/2286165388 shutdown_connections 2023-10-30T14:04:22.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7600664a0 0x7fa760068960 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:22.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa77c1f2a70 0x7fa77c154270 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:22.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa77c159570 0x7fa77c1f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:22.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 --2- 172.21.15.46:0/2286165388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa76c0047d0 0x7fa77c159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:22.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 -- 172.21.15.46:0/2286165388 >> 172.21.15.46:0/2286165388 conn(0x7fa77c06c6e0 msgr2=0x7fa77c0c7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:22.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 -- 172.21.15.46:0/2286165388 shutdown_connections 2023-10-30T14:04:22.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:22.204+0000 7fa783209640 1 -- 172.21.15.46:0/2286165388 wait complete. 2023-10-30T14:04:22.230 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:04:24.232 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:24.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.380+0000 7f6503046640 1 Processor -- start 2023-10-30T14:04:24.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.380+0000 7f6503046640 1 -- start start 2023-10-30T14:04:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f6503046640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f64fc0c6540 0x7f64fc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f6503046640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc0c5810 con 0x7f64fc070740 2023-10-30T14:04:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f6503046640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc0c69b0 con 0x7f64fc1580e0 2023-10-30T14:04:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f6503046640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64fc06acb0 con 0x7f64fc0c6540 2023-10-30T14:04:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fbfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f64fc070740 0x7f64fc06a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56084/0 (socket says 172.21.15.46:56084) 2023-10-30T14:04:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fbfff640 1 -- 172.21.15.46:0/2841912311 learned_addr learned my addr 172.21.15.46:0/2841912311 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:04:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f65015e3640 1 --2- 172.21.15.46:0/2841912311 >> v2:172.21.15.154:3300/0 conn(0x7f64fc0c6540 0x7f64fc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243689197 0 0) 0x7f64fc0c5810 con 0x7f64fc070740 2023-10-30T14:04:24.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ec003530 con 0x7f64fc070740 2023-10-30T14:04:24.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1459577849 0 0) 0x7f64ec003530 con 0x7f64fc070740 2023-10-30T14:04:24.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 >> v2:172.21.15.154:3300/0 conn(0x7f64fc0c6540 msgr2=0x7f64fc15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 --2- 172.21.15.46:0/2841912311 >> v2:172.21.15.154:3300/0 conn(0x7f64fc0c6540 0x7f64fc15e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 >> v1:172.21.15.154:6789/0 conn(0x7f64fc1580e0 legacy=0x7f64fc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc0c6bb0 con 0x7f64fc070740 2023-10-30T14:04:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f64e8001680 con 0x7f64fc070740 2023-10-30T14:04:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 >> v1:172.21.15.46:6789/0 conn(0x7f64fc070740 legacy=0x7f64fc06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 --2- 172.21.15.46:0/2841912311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64ec003980 0x7f64ec003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 --2- 172.21.15.46:0/2841912311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec0047d0 0x7f64ec004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 --2- 172.21.15.46:0/2841912311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64ec005190 0x7f64ec00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec00c000 con 0x7f64ec0047d0 2023-10-30T14:04:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec00c170 con 0x7f64ec003980 2023-10-30T14:04:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec00c330 con 0x7f64ec005190 2023-10-30T14:04:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fbfff640 1 --2- 172.21.15.46:0/2841912311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec0047d0 0x7f64ec004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f6500de2640 1 --2- 172.21.15.46:0/2841912311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64ec003980 0x7f64ec003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f65015e3640 1 --2- 172.21.15.46:0/2841912311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64ec005190 0x7f64ec00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fbfff640 1 -- 172.21.15.46:0/2841912311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64ec005190 msgr2=0x7f64ec00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:24.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fbfff640 1 --2- 172.21.15.46:0/2841912311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64ec005190 0x7f64ec00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fbfff640 1 -- 172.21.15.46:0/2841912311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64ec003980 msgr2=0x7f64ec003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:24.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fbfff640 1 --2- 172.21.15.46:0/2841912311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64ec003980 0x7f64ec003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f64fbfff640 1 -- 172.21.15.46:0/2841912311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec003530 con 0x7f64ec0047d0 2023-10-30T14:04:24.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.384+0000 7f65015e3640 1 -- 172.21.15.46:0/2841912311 reap_dead start 2023-10-30T14:04:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.388+0000 7f64fbfff640 1 --2- 172.21.15.46:0/2841912311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec0047d0 0x7f64ec004c50 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f64e800c240 tx=0x7f64e801f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15156e7c0ff4eac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.388+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8005ae0 con 0x7f64ec0047d0 2023-10-30T14:04:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.388+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e80ab070 con 0x7f64ec0047d0 2023-10-30T14:04:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.388+0000 7f64fb7fe640 1 -- 172.21.15.46:0/2841912311 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8007070 con 0x7f64ec0047d0 2023-10-30T14:04:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.388+0000 7f6503046640 1 -- 172.21.15.46:0/2841912311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec0047d0 msgr2=0x7f64ec004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.388+0000 7f6503046640 1 --2- 172.21.15.46:0/2841912311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec0047d0 0x7f64ec004c50 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f64e800c240 tx=0x7f64e801f330 comp rx=0 tx=0).stop 2023-10-30T14:04:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.388+0000 7f6503046640 1 -- 172.21.15.46:0/2841912311 shutdown_connections 2023-10-30T14:04:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.388+0000 7f6503046640 1 --2- 172.21.15.46:0/2841912311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec0047d0 0x7f64ec004c50 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.388+0000 7f6503046640 1 -- 172.21.15.46:0/2841912311 >> 172.21.15.46:0/2841912311 conn(0x7f64fc0c7910 msgr2=0x7f64fc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.388+0000 7f6503046640 1 -- 172.21.15.46:0/2841912311 shutdown_connections 2023-10-30T14:04:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.388+0000 7f6503046640 1 -- 172.21.15.46:0/2841912311 wait complete. 2023-10-30T14:04:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f6503046640 1 Processor -- start 2023-10-30T14:04:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f6503046640 1 -- start start 2023-10-30T14:04:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f6503046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64ec0047d0 0x7f64fc0c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f6503046640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64fc1f4900 0x7f64fc1f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f6503046640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc1f6500 0x7f64fc1f6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f6503046640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc1600a0 con 0x7f64fc1f6500 2023-10-30T14:04:24.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f6503046640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc0c2730 con 0x7f64fc1f4900 2023-10-30T14:04:24.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f6503046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc1f4470 con 0x7f64ec0047d0 2023-10-30T14:04:24.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f65015e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc1f6500 0x7f64fc1f6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f65015e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc1f6500 0x7f64fc1f6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38134/0 (socket says 172.21.15.46:38134) 2023-10-30T14:04:24.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f65015e3640 1 -- 172.21.15.46:0/1949045276 learned_addr learned my addr 172.21.15.46:0/1949045276 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:24.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f6500de2640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64ec0047d0 0x7f64fc0c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f64fbfff640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64fc1f4900 0x7f64fc1f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f65015e3640 1 -- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64ec0047d0 msgr2=0x7f64fc0c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f65015e3640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64ec0047d0 0x7f64fc0c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f65015e3640 1 -- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64fc1f4900 msgr2=0x7f64fc1f4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:24.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f65015e3640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64fc1f4900 0x7f64fc1f4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f65015e3640 1 -- 172.21.15.46:0/1949045276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc0c5d80 con 0x7f64fc1f6500 2023-10-30T14:04:24.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f64fbfff640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64fc1f4900 0x7f64fc1f4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:04:24.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f65015e3640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc1f6500 0x7f64fc1f6980 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f64fc06f4f0 tx=0x7f64f4011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:24.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f64f97fa640 1 -- 172.21.15.46:0/1949045276 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f400e040 con 0x7f64fc1f6500 2023-10-30T14:04:24.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f64f97fa640 1 -- 172.21.15.46:0/1949045276 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f400f040 con 0x7f64fc1f6500 2023-10-30T14:04:24.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f64f97fa640 1 -- 172.21.15.46:0/1949045276 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f400fe30 con 0x7f64fc1f6500 2023-10-30T14:04:24.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f6503046640 1 -- 172.21.15.46:0/1949045276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64fc1f52a0 con 0x7f64fc1f6500 2023-10-30T14:04:24.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.392+0000 7f6503046640 1 -- 172.21.15.46:0/1949045276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64fc160fc0 con 0x7f64fc1f6500 2023-10-30T14:04:24.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.396+0000 7f6503046640 1 -- 172.21.15.46:0/1949045276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64fc1f4620 con 0x7f64fc1f6500 2023-10-30T14:04:24.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.396+0000 7f64f97fa640 1 -- 172.21.15.46:0/1949045276 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64f4022460 con 0x7f64fc1f6500 2023-10-30T14:04:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.396+0000 7f64f97fa640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64e00664d0 0x7f64e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.396+0000 7f64f97fa640 1 -- 172.21.15.46:0/1949045276 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f64f4084d70 con 0x7f64fc1f6500 2023-10-30T14:04:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.396+0000 7f6500de2640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64e00664d0 0x7f64e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.396+0000 7f6500de2640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64e00664d0 0x7f64e0068990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f64f0007fd0 tx=0x7f64f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:24.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.400+0000 7f64f97fa640 1 -- 172.21.15.46:0/1949045276 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64fc1f4620 con 0x7f64fc1f6500 2023-10-30T14:04:24.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 -- 172.21.15.46:0/1949045276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64fc1f4620 con 0x7f64fc1f6500 2023-10-30T14:04:24.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f64f97fa640 1 -- 172.21.15.46:0/1949045276 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f64fc1f4620 con 0x7f64fc1f6500 2023-10-30T14:04:24.601 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:24.601 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:24.459809+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:24.602 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 175 2023-10-30T14:04:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 -- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64e00664d0 msgr2=0x7f64e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64e00664d0 0x7f64e0068990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f64f0007fd0 tx=0x7f64f0009040 comp rx=0 tx=0).stop 2023-10-30T14:04:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 -- 172.21.15.46:0/1949045276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc1f6500 msgr2=0x7f64fc1f6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:24.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc1f6500 0x7f64fc1f6980 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f64fc06f4f0 tx=0x7f64f4011f10 comp rx=0 tx=0).stop 2023-10-30T14:04:24.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 -- 172.21.15.46:0/1949045276 shutdown_connections 2023-10-30T14:04:24.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64e00664d0 0x7f64e0068990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc1f6500 0x7f64fc1f6980 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64fc1f4900 0x7f64fc1f4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 --2- 172.21.15.46:0/1949045276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64ec0047d0 0x7f64fc0c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 -- 172.21.15.46:0/1949045276 >> 172.21.15.46:0/1949045276 conn(0x7f64fc0c7910 msgr2=0x7f64fc070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:24.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 -- 172.21.15.46:0/1949045276 shutdown_connections 2023-10-30T14:04:24.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.596+0000 7f6503046640 1 -- 172.21.15.46:0/1949045276 wait complete. 2023-10-30T14:04:24.623 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T14:04:24.623 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:04:24.624 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:24.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85cda79640 1 Processor -- start 2023-10-30T14:04:24.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85cda79640 1 -- start start 2023-10-30T14:04:24.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85cda79640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f85c806f7a0 0x7f85c806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85cda79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c8071110 0x7f85c8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85cda79640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c8066cf0 con 0x7f85c80c6bd0 2023-10-30T14:04:24.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85cda79640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c80700c0 con 0x7f85c806f7a0 2023-10-30T14:04:24.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85cda79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c80c70d0 con 0x7f85c8071110 2023-10-30T14:04:24.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85c77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f85c806f7a0 0x7f85c806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85c77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f85c806f7a0 0x7f85c806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38150/0 (socket says 172.21.15.46:38150) 2023-10-30T14:04:24.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85c77fe640 1 -- 172.21.15.46:0/2687097890 learned_addr learned my addr 172.21.15.46:0/2687097890 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:24.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85c7fff640 1 --2- 172.21.15.46:0/2687097890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c8071110 0x7f85c8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85c77fe640 1 -- 172.21.15.46:0/2687097890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c8071110 msgr2=0x7f85c8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:24.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85c77fe640 1 --2- 172.21.15.46:0/2687097890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c8071110 0x7f85c8158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85c77fe640 1 -- 172.21.15.46:0/2687097890 >> v1:172.21.15.154:6789/0 conn(0x7f85c80c6bd0 legacy=0x7f85c8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:24.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.764+0000 7f85c77fe640 1 -- 172.21.15.46:0/2687097890 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c80c7390 con 0x7f85c806f7a0 2023-10-30T14:04:24.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.768+0000 7f85c77fe640 1 --2- 172.21.15.46:0/2687097890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c806f7a0 0x7f85c806fb80 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009870 tx=0x7f85b801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29282d7f74e612df server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:24.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.768+0000 7f85c67fc640 1 -- 172.21.15.46:0/2687097890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b80a89b0 con 0x7f85c806f7a0 2023-10-30T14:04:24.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.768+0000 7f85c67fc640 1 -- 172.21.15.46:0/2687097890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b80a8b10 con 0x7f85c806f7a0 2023-10-30T14:04:24.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.768+0000 7f85c67fc640 1 -- 172.21.15.46:0/2687097890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b80a8ea0 con 0x7f85c806f7a0 2023-10-30T14:04:24.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.768+0000 7f85cda79640 1 -- 172.21.15.46:0/2687097890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c806f7a0 msgr2=0x7f85c806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:24.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.768+0000 7f85cda79640 1 --2- 172.21.15.46:0/2687097890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c806f7a0 0x7f85c806fb80 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009870 tx=0x7f85b801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.768+0000 7f85cda79640 1 -- 172.21.15.46:0/2687097890 shutdown_connections 2023-10-30T14:04:24.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.768+0000 7f85cda79640 1 --2- 172.21.15.46:0/2687097890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c8071110 0x7f85c8158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.768+0000 7f85cda79640 1 --2- 172.21.15.46:0/2687097890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c806f7a0 0x7f85c806fb80 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.768+0000 7f85cda79640 1 -- 172.21.15.46:0/2687097890 >> 172.21.15.46:0/2687097890 conn(0x7f85c80c7910 msgr2=0x7f85c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:24.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.768+0000 7f85cda79640 1 -- 172.21.15.46:0/2687097890 shutdown_connections 2023-10-30T14:04:24.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.768+0000 7f85cda79640 1 -- 172.21.15.46:0/2687097890 wait complete. 2023-10-30T14:04:24.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85cda79640 1 Processor -- start 2023-10-30T14:04:24.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85cda79640 1 -- start start 2023-10-30T14:04:24.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85cda79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c806f7a0 0x7f85c80c1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85cda79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c8071110 0x7f85c80c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85cda79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85c80c6bd0 0x7f85c81f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85cda79640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c81583d0 con 0x7f85c806f7a0 2023-10-30T14:04:24.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85cda79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c81eeb90 con 0x7f85c80c6bd0 2023-10-30T14:04:24.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85cda79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c81eee70 con 0x7f85c8071110 2023-10-30T14:04:24.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c806f7a0 0x7f85c80c1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c806f7a0 0x7f85c80c1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38160/0 (socket says 172.21.15.46:38160) 2023-10-30T14:04:24.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85c77fe640 1 -- 172.21.15.46:0/3551522846 learned_addr learned my addr 172.21.15.46:0/3551522846 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:24.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85c6ffd640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c8071110 0x7f85c80c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85c7fff640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85c80c6bd0 0x7f85c81f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85c77fe640 1 -- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c8071110 msgr2=0x7f85c80c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:24.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85c77fe640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c8071110 0x7f85c80c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85c77fe640 1 -- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85c80c6bd0 msgr2=0x7f85c81f2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:24.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85c77fe640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85c80c6bd0 0x7f85c81f2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85c77fe640 1 -- 172.21.15.46:0/3551522846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b80094d0 con 0x7f85c806f7a0 2023-10-30T14:04:24.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85c77fe640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c806f7a0 0x7f85c80c1910 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f85b801c920 tx=0x7f85b801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:24.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85affff640 1 -- 172.21.15.46:0/3551522846 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b80b1df0 con 0x7f85c806f7a0 2023-10-30T14:04:24.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85affff640 1 -- 172.21.15.46:0/3551522846 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b80bb470 con 0x7f85c806f7a0 2023-10-30T14:04:24.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85cda79640 1 -- 172.21.15.46:0/3551522846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c81ef150 con 0x7f85c806f7a0 2023-10-30T14:04:24.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85affff640 1 -- 172.21.15.46:0/3551522846 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b80bb7e0 con 0x7f85c806f7a0 2023-10-30T14:04:24.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.772+0000 7f85cda79640 1 -- 172.21.15.46:0/3551522846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c81f1b50 con 0x7f85c806f7a0 2023-10-30T14:04:24.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.776+0000 7f85cda79640 1 -- 172.21.15.46:0/3551522846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c804e170 con 0x7f85c806f7a0 2023-10-30T14:04:24.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.776+0000 7f85affff640 1 -- 172.21.15.46:0/3551522846 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f85b80b7070 con 0x7f85c806f7a0 2023-10-30T14:04:24.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.776+0000 7f85affff640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f85b40664d0 0x7f85b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:24.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.776+0000 7f85affff640 1 -- 172.21.15.46:0/3551522846 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f85b8122650 con 0x7f85c806f7a0 2023-10-30T14:04:24.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.776+0000 7f85c6ffd640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f85b40664d0 0x7f85b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:24.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.776+0000 7f85c6ffd640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f85b40664d0 0x7f85b4068990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f85c81580d0 tx=0x7f85bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:24.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.780+0000 7f85affff640 1 -- 172.21.15.46:0/3551522846 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85b80ef320 con 0x7f85c806f7a0 2023-10-30T14:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 -- 172.21.15.46:0/3551522846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85c804e170 con 0x7f85c806f7a0 2023-10-30T14:04:24.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85affff640 1 -- 172.21.15.46:0/3551522846 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f85b80f2940 con 0x7f85c806f7a0 2023-10-30T14:04:24.977 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:24.977 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:24.459809+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:24.978 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 175 2023-10-30T14:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 -- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f85b40664d0 msgr2=0x7f85b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f85b40664d0 0x7f85b4068990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f85c81580d0 tx=0x7f85bc009040 comp rx=0 tx=0).stop 2023-10-30T14:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 -- 172.21.15.46:0/3551522846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c806f7a0 msgr2=0x7f85c80c1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c806f7a0 0x7f85c80c1910 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f85b801c920 tx=0x7f85b801ba90 comp rx=0 tx=0).stop 2023-10-30T14:04:24.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 -- 172.21.15.46:0/3551522846 shutdown_connections 2023-10-30T14:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f85b40664d0 0x7f85b4068990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85c80c6bd0 0x7f85c81f2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c8071110 0x7f85c80c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 --2- 172.21.15.46:0/3551522846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c806f7a0 0x7f85c80c1910 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:24.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 -- 172.21.15.46:0/3551522846 >> 172.21.15.46:0/3551522846 conn(0x7f85c80c7910 msgr2=0x7f85c806a790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 -- 172.21.15.46:0/3551522846 shutdown_connections 2023-10-30T14:04:24.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:24.972+0000 7f85cda79640 1 -- 172.21.15.46:0/3551522846 wait complete. 2023-10-30T14:04:24.999 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 175, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 175, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8034": { "addr": "172.21.15.46:6833/2075693104", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2075693104, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2075693104, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8034, "incarnation": 167, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 20 }, "gid_8190": { "addr": "172.21.15.46:6837/3154171789", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3154171789, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3154171789, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8190, "incarnation": 175, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 78, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:04:24.459809+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8190, "mds_1": 8034 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:04:25.202 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:04:25.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:04:27.001 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:27.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.152+0000 7fbf203d3640 1 Processor -- start 2023-10-30T14:04:27.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.152+0000 7fbf203d3640 1 -- start start 2023-10-30T14:04:27.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.152+0000 7fbf203d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbf1806f7a0 0x7fbf1806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:27.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.152+0000 7fbf203d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf180c6bd0 0x7fbf18069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:27.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.152+0000 7fbf203d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf1806a3d0 con 0x7fbf18071110 2023-10-30T14:04:27.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.152+0000 7fbf203d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf180700c0 con 0x7fbf180c6bd0 2023-10-30T14:04:27.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.152+0000 7fbf203d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf18066cf0 con 0x7fbf1806f7a0 2023-10-30T14:04:27.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.152+0000 7fbf1d96e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf180c6bd0 0x7fbf18069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:27.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.156+0000 7fbf1d96e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf180c6bd0 0x7fbf18069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53258/0 (socket says 172.21.15.46:53258) 2023-10-30T14:04:27.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.156+0000 7fbf1d96e640 1 -- 172.21.15.46:0/1542129716 learned_addr learned my addr 172.21.15.46:0/1542129716 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:27.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.156+0000 7fbf1e16f640 1 --2- 172.21.15.46:0/1542129716 >> v2:172.21.15.154:3300/0 conn(0x7fbf1806f7a0 0x7fbf1806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:27.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.156+0000 7fbf1d96e640 1 -- 172.21.15.46:0/1542129716 >> v2:172.21.15.154:3300/0 conn(0x7fbf1806f7a0 msgr2=0x7fbf1806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:27.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.156+0000 7fbf1d96e640 1 --2- 172.21.15.46:0/1542129716 >> v2:172.21.15.154:3300/0 conn(0x7fbf1806f7a0 0x7fbf1806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:27.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.156+0000 7fbf1d96e640 1 -- 172.21.15.46:0/1542129716 >> v1:172.21.15.154:6789/0 conn(0x7fbf18071110 legacy=0x7fbf18158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:27.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.156+0000 7fbf1d96e640 1 -- 172.21.15.46:0/1542129716 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf180c70d0 con 0x7fbf180c6bd0 2023-10-30T14:04:27.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.156+0000 7fbf1d96e640 1 --2- 172.21.15.46:0/1542129716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf180c6bd0 0x7fbf18069e90 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fbf1806c830 tx=0x7fbf0801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6074323bd74ca2e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:27.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.156+0000 7fbf1d16d640 1 -- 172.21.15.46:0/1542129716 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf080a89b0 con 0x7fbf180c6bd0 2023-10-30T14:04:27.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.156+0000 7fbf1d16d640 1 -- 172.21.15.46:0/1542129716 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf080a8b10 con 0x7fbf180c6bd0 2023-10-30T14:04:27.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.156+0000 7fbf1d16d640 1 -- 172.21.15.46:0/1542129716 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf080a8ea0 con 0x7fbf180c6bd0 2023-10-30T14:04:27.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 -- 172.21.15.46:0/1542129716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf180c6bd0 msgr2=0x7fbf18069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:27.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 --2- 172.21.15.46:0/1542129716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf180c6bd0 0x7fbf18069e90 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fbf1806c830 tx=0x7fbf0801ecb0 comp rx=0 tx=0).stop 2023-10-30T14:04:27.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 -- 172.21.15.46:0/1542129716 shutdown_connections 2023-10-30T14:04:27.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 --2- 172.21.15.46:0/1542129716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf180c6bd0 0x7fbf18069e90 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:27.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 --2- 172.21.15.46:0/1542129716 >> v2:172.21.15.154:3300/0 conn(0x7fbf1806f7a0 0x7fbf1806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:27.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 -- 172.21.15.46:0/1542129716 >> 172.21.15.46:0/1542129716 conn(0x7fbf180c7910 msgr2=0x7fbf180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:27.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 -- 172.21.15.46:0/1542129716 shutdown_connections 2023-10-30T14:04:27.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 -- 172.21.15.46:0/1542129716 wait complete. 2023-10-30T14:04:27.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 Processor -- start 2023-10-30T14:04:27.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 -- start start 2023-10-30T14:04:27.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806f7a0 0x7fbf18150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:27.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf18071110 0x7fbf1806bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:27.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf180c6bd0 0x7fbf181ed000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:27.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf18157030 con 0x7fbf1806f7a0 2023-10-30T14:04:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf18158690 con 0x7fbf180c6bd0 2023-10-30T14:04:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf203d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf181f1040 con 0x7fbf18071110 2023-10-30T14:04:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf1e16f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806f7a0 0x7fbf18150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf1e16f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806f7a0 0x7fbf18150bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53262/0 (socket says 172.21.15.46:53262) 2023-10-30T14:04:27.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf1e16f640 1 -- 172.21.15.46:0/2414345406 learned_addr learned my addr 172.21.15.46:0/2414345406 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:27.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf1d96e640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf18071110 0x7fbf1806bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:27.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf1e970640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf180c6bd0 0x7fbf181ed000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:27.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf1e16f640 1 -- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf18071110 msgr2=0x7fbf1806bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:27.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf1e16f640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf18071110 0x7fbf1806bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf1e16f640 1 -- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf180c6bd0 msgr2=0x7fbf181ed000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf1e16f640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf180c6bd0 0x7fbf181ed000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf1e16f640 1 -- 172.21.15.46:0/2414345406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf080094d0 con 0x7fbf1806f7a0 2023-10-30T14:04:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.160+0000 7fbf1e970640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf180c6bd0 0x7fbf181ed000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:04:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.164+0000 7fbf1e16f640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806f7a0 0x7fbf18150bf0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fbf1400d5f0 tx=0x7fbf1400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:27.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.164+0000 7fbf0effd640 1 -- 172.21.15.46:0/2414345406 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf140125f0 con 0x7fbf1806f7a0 2023-10-30T14:04:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.164+0000 7fbf0effd640 1 -- 172.21.15.46:0/2414345406 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf14012750 con 0x7fbf1806f7a0 2023-10-30T14:04:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.164+0000 7fbf0effd640 1 -- 172.21.15.46:0/2414345406 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf14012ae0 con 0x7fbf1806f7a0 2023-10-30T14:04:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.164+0000 7fbf203d3640 1 -- 172.21.15.46:0/2414345406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf181f14d0 con 0x7fbf1806f7a0 2023-10-30T14:04:27.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.164+0000 7fbf203d3640 1 -- 172.21.15.46:0/2414345406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf181f0710 con 0x7fbf1806f7a0 2023-10-30T14:04:27.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.164+0000 7fbf203d3640 1 -- 172.21.15.46:0/2414345406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1804e170 con 0x7fbf1806f7a0 2023-10-30T14:04:27.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.164+0000 7fbf0effd640 1 -- 172.21.15.46:0/2414345406 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbf14012c40 con 0x7fbf1806f7a0 2023-10-30T14:04:27.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.164+0000 7fbf0effd640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbefc0664a0 0x7fbefc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:27.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.164+0000 7fbf0effd640 1 -- 172.21.15.46:0/2414345406 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fbf14084b30 con 0x7fbf1806f7a0 2023-10-30T14:04:27.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.164+0000 7fbf1d96e640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbefc0664a0 0x7fbefc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:27.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.168+0000 7fbf1d96e640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbefc0664a0 0x7fbefc068960 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fbf080b2000 tx=0x7fbf080aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:27.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.168+0000 7fbf0effd640 1 -- 172.21.15.46:0/2414345406 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbf14051800 con 0x7fbf1806f7a0 2023-10-30T14:04:27.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.360+0000 7fbf203d3640 1 -- 172.21.15.46:0/2414345406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf180c8710 con 0x7fbf1806f7a0 2023-10-30T14:04:27.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf0effd640 1 -- 172.21.15.46:0/2414345406 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fbf14054e20 con 0x7fbf1806f7a0 2023-10-30T14:04:27.368 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:27.368 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:24.459809+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:27.368 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 175 2023-10-30T14:04:27.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf203d3640 1 -- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbefc0664a0 msgr2=0x7fbefc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf203d3640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbefc0664a0 0x7fbefc068960 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fbf080b2000 tx=0x7fbf080aa070 comp rx=0 tx=0).stop 2023-10-30T14:04:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf203d3640 1 -- 172.21.15.46:0/2414345406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806f7a0 msgr2=0x7fbf18150bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf203d3640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806f7a0 0x7fbf18150bf0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fbf1400d5f0 tx=0x7fbf1400dab0 comp rx=0 tx=0).stop 2023-10-30T14:04:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf203d3640 1 -- 172.21.15.46:0/2414345406 shutdown_connections 2023-10-30T14:04:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf203d3640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbefc0664a0 0x7fbefc068960 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:27.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf203d3640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf180c6bd0 0x7fbf181ed000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:27.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf203d3640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf18071110 0x7fbf1806bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:27.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf203d3640 1 --2- 172.21.15.46:0/2414345406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1806f7a0 0x7fbf18150bf0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:27.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf203d3640 1 -- 172.21.15.46:0/2414345406 >> 172.21.15.46:0/2414345406 conn(0x7fbf180c7910 msgr2=0x7fbf180c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:27.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf203d3640 1 -- 172.21.15.46:0/2414345406 shutdown_connections 2023-10-30T14:04:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:27.364+0000 7fbf203d3640 1 -- 172.21.15.46:0/2414345406 wait complete. 2023-10-30T14:04:27.389 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:04:29.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f072510f640 1 Processor -- start 2023-10-30T14:04:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f072510f640 1 -- start start 2023-10-30T14:04:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f072510f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f07200cf160 0x7f07200cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f072510f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07200cb800 con 0x7f072015d6f0 2023-10-30T14:04:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f072510f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07200cb330 con 0x7f07201538a0 2023-10-30T14:04:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f072510f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07200cad70 con 0x7f07200cf160 2023-10-30T14:04:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071e59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f072015d6f0 0x7f072015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36092/0 (socket says 172.21.15.46:36092) 2023-10-30T14:04:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071e59c640 1 -- 172.21.15.46:0/198033407 learned_addr learned my addr 172.21.15.46:0/198033407 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:04:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071f59e640 1 --2- 172.21.15.46:0/198033407 >> v2:172.21.15.154:3300/0 conn(0x7f07200cf160 0x7f07200cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:29.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845606555 0 0) 0x7f07200cb800 con 0x7f072015d6f0 2023-10-30T14:04:29.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0708003530 con 0x7f072015d6f0 2023-10-30T14:04:29.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478027551 0 0) 0x7f0708003530 con 0x7f072015d6f0 2023-10-30T14:04:29.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 >> v2:172.21.15.154:3300/0 conn(0x7f07200cf160 msgr2=0x7f07200cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:29.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 --2- 172.21.15.46:0/198033407 >> v2:172.21.15.154:3300/0 conn(0x7f07200cf160 0x7f07200cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:29.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 >> v1:172.21.15.154:6789/0 conn(0x7f07201538a0 legacy=0x7f07200ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:29.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07200cb050 con 0x7f072015d6f0 2023-10-30T14:04:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f0714001680 con 0x7f072015d6f0 2023-10-30T14:04:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 >> v1:172.21.15.46:6789/0 conn(0x7f072015d6f0 legacy=0x7f072015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 --2- 172.21.15.46:0/198033407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0708003980 0x7f0708003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 --2- 172.21.15.46:0/198033407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07080047d0 0x7f0708004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:29.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 --2- 172.21.15.46:0/198033407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0708005190 0x7f070800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07200cad70 con 0x7f0708003980 2023-10-30T14:04:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070800c040 con 0x7f07080047d0 2023-10-30T14:04:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070800c200 con 0x7f0708005190 2023-10-30T14:04:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071ed9d640 1 --2- 172.21.15.46:0/198033407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0708003980 0x7f0708003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071e59c640 1 --2- 172.21.15.46:0/198033407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07080047d0 0x7f0708004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.552+0000 7f071f59e640 1 --2- 172.21.15.46:0/198033407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0708005190 0x7f070800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:29.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.556+0000 7f071ed9d640 1 -- 172.21.15.46:0/198033407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0708005190 msgr2=0x7f070800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:29.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.556+0000 7f071ed9d640 1 --2- 172.21.15.46:0/198033407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0708005190 0x7f070800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:29.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.556+0000 7f071ed9d640 1 -- 172.21.15.46:0/198033407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07080047d0 msgr2=0x7f0708004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:29.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.556+0000 7f071ed9d640 1 --2- 172.21.15.46:0/198033407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07080047d0 0x7f0708004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:29.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.556+0000 7f071ed9d640 1 -- 172.21.15.46:0/198033407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0708003530 con 0x7f0708003980 2023-10-30T14:04:29.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.556+0000 7f071f59e640 1 -- 172.21.15.46:0/198033407 reap_dead start 2023-10-30T14:04:29.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.560+0000 7f071ed9d640 1 --2- 172.21.15.46:0/198033407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0708003980 0x7f0708003da0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f0710001780 tx=0x7f071001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72f98022e8eb5da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:29.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.560+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07100052d0 con 0x7f0708003980 2023-10-30T14:04:29.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.560+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07100a7050 con 0x7f0708003980 2023-10-30T14:04:29.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.560+0000 7f071dd9b640 1 -- 172.21.15.46:0/198033407 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710003600 con 0x7f0708003980 2023-10-30T14:04:29.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.560+0000 7f072510f640 1 -- 172.21.15.46:0/198033407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0708003980 msgr2=0x7f0708003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:29.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.560+0000 7f072510f640 1 --2- 172.21.15.46:0/198033407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0708003980 0x7f0708003da0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f0710001780 tx=0x7f071001c3e0 comp rx=0 tx=0).stop 2023-10-30T14:04:29.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.560+0000 7f072510f640 1 -- 172.21.15.46:0/198033407 shutdown_connections 2023-10-30T14:04:29.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.560+0000 7f072510f640 1 --2- 172.21.15.46:0/198033407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0708003980 0x7f0708003da0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:29.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.560+0000 7f072510f640 1 -- 172.21.15.46:0/198033407 >> 172.21.15.46:0/198033407 conn(0x7f072006bfe0 msgr2=0x7f072006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.560+0000 7f072510f640 1 -- 172.21.15.46:0/198033407 shutdown_connections 2023-10-30T14:04:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.560+0000 7f072510f640 1 -- 172.21.15.46:0/198033407 wait complete. 2023-10-30T14:04:29.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f072510f640 1 Processor -- start 2023-10-30T14:04:29.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f072510f640 1 -- start start 2023-10-30T14:04:29.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f072510f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0708003980 0x7f07200c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:29.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f072510f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07201f2d20 0x7f07201e64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:29.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f072510f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07200c71c0 0x7f07200c7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:29.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f072510f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07200c6da0 con 0x7f07201f2d20 2023-10-30T14:04:29.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f072510f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07200c7b80 con 0x7f07200c71c0 2023-10-30T14:04:29.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f072510f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07201e3810 con 0x7f0708003980 2023-10-30T14:04:29.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07201f2d20 0x7f07201e64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:29.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07201f2d20 0x7f07201e64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53276/0 (socket says 172.21.15.46:53276) 2023-10-30T14:04:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071e59c640 1 -- 172.21.15.46:0/3078072228 learned_addr learned my addr 172.21.15.46:0/3078072228 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071ed9d640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0708003980 0x7f07200c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071f59e640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07200c71c0 0x7f07200c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071e59c640 1 -- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0708003980 msgr2=0x7f07200c8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071e59c640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0708003980 0x7f07200c8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071e59c640 1 -- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07200c71c0 msgr2=0x7f07200c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071e59c640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07200c71c0 0x7f07200c7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071e59c640 1 -- 172.21.15.46:0/3078072228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07200c8fb0 con 0x7f07201f2d20 2023-10-30T14:04:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071ed9d640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0708003980 0x7f07200c8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:04:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071f59e640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07200c71c0 0x7f07200c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:04:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f071e59c640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07201f2d20 0x7f07201e64e0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f0714004880 tx=0x7f0714012920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f07077fe640 1 -- 172.21.15.46:0/3078072228 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0714014750 con 0x7f07201f2d20 2023-10-30T14:04:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f07077fe640 1 -- 172.21.15.46:0/3078072228 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07140148b0 con 0x7f07201f2d20 2023-10-30T14:04:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f07077fe640 1 -- 172.21.15.46:0/3078072228 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0714014c40 con 0x7f07201f2d20 2023-10-30T14:04:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f072510f640 1 -- 172.21.15.46:0/3078072228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07200c9220 con 0x7f07201f2d20 2023-10-30T14:04:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.564+0000 7f072510f640 1 -- 172.21.15.46:0/3078072228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07200c66e0 con 0x7f07201f2d20 2023-10-30T14:04:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.568+0000 7f072510f640 1 -- 172.21.15.46:0/3078072228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07200c6ee0 con 0x7f07201f2d20 2023-10-30T14:04:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.568+0000 7f07077fe640 1 -- 172.21.15.46:0/3078072228 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f07140153e0 con 0x7f07201f2d20 2023-10-30T14:04:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.568+0000 7f07077fe640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f070c0664d0 0x7f070c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.568+0000 7f07077fe640 1 -- 172.21.15.46:0/3078072228 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f0714086eb0 con 0x7f07201f2d20 2023-10-30T14:04:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.568+0000 7f071ed9d640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f070c0664d0 0x7f070c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.568+0000 7f071ed9d640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f070c0664d0 0x7f070c068990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0710001750 tx=0x7f07100011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:29.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.576+0000 7f07077fe640 1 -- 172.21.15.46:0/3078072228 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07200c6ee0 con 0x7f07201f2d20 2023-10-30T14:04:29.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.800+0000 7f072510f640 1 -- 172.21.15.46:0/3078072228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07200c6ee0 con 0x7f07201f2d20 2023-10-30T14:04:29.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.800+0000 7f07077fe640 1 -- 172.21.15.46:0/3078072228 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f07200c6ee0 con 0x7f07201f2d20 2023-10-30T14:04:29.806 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:29.807 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:24.459809+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:29.807 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 175 2023-10-30T14:04:29.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.804+0000 7f072510f640 1 -- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f070c0664d0 msgr2=0x7f070c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.804+0000 7f072510f640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f070c0664d0 0x7f070c068990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0710001750 tx=0x7f07100011a0 comp rx=0 tx=0).stop 2023-10-30T14:04:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.804+0000 7f072510f640 1 -- 172.21.15.46:0/3078072228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07201f2d20 msgr2=0x7f07201e64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.804+0000 7f072510f640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07201f2d20 0x7f07201e64e0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f0714004880 tx=0x7f0714012920 comp rx=0 tx=0).stop 2023-10-30T14:04:29.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.804+0000 7f072510f640 1 -- 172.21.15.46:0/3078072228 shutdown_connections 2023-10-30T14:04:29.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.804+0000 7f072510f640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f070c0664d0 0x7f070c068990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:29.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.804+0000 7f072510f640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07200c71c0 0x7f07200c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:29.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.804+0000 7f072510f640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07201f2d20 0x7f07201e64e0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:29.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.804+0000 7f072510f640 1 --2- 172.21.15.46:0/3078072228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0708003980 0x7f07200c8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:29.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.804+0000 7f072510f640 1 -- 172.21.15.46:0/3078072228 >> 172.21.15.46:0/3078072228 conn(0x7f072006bfe0 msgr2=0x7f072015b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:29.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.804+0000 7f072510f640 1 -- 172.21.15.46:0/3078072228 shutdown_connections 2023-10-30T14:04:29.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:29.804+0000 7f072510f640 1 -- 172.21.15.46:0/3078072228 wait complete. 2023-10-30T14:04:29.833 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:04:31.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdaed30f640 1 Processor -- start 2023-10-30T14:04:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdaed30f640 1 -- start start 2023-10-30T14:04:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdaed30f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdae80c6bd0 0x7fdae806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdaed30f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae8066cf0 con 0x7fdae8071110 2023-10-30T14:04:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdaed30f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae80c70d0 con 0x7fdae806f7a0 2023-10-30T14:04:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdaed30f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae80700c0 con 0x7fdae80c6bd0 2023-10-30T14:04:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdae67fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdae80c6bd0 0x7fdae806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdae67fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdae80c6bd0 0x7fdae806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53284/0 (socket says 172.21.15.46:53284) 2023-10-30T14:04:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdae67fc640 1 -- 172.21.15.46:0/138641607 learned_addr learned my addr 172.21.15.46:0/138641607 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdae67fc640 1 -- 172.21.15.46:0/138641607 >> v1:172.21.15.154:6789/0 conn(0x7fdae806f7a0 legacy=0x7fdae8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdae67fc640 1 -- 172.21.15.46:0/138641607 >> v1:172.21.15.46:6789/0 conn(0x7fdae8071110 legacy=0x7fdae8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdae5ffb640 1 -- 172.21.15.46:0/138641607 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364952565 0 0) 0x7fdae8066cf0 con 0x7fdae8071110 2023-10-30T14:04:32.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.084+0000 7fdae67fc640 1 -- 172.21.15.46:0/138641607 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae80c7390 con 0x7fdae80c6bd0 2023-10-30T14:04:32.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.088+0000 7fdae67fc640 1 --2- 172.21.15.46:0/138641607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdae80c6bd0 0x7fdae806fb80 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009870 tx=0x7fdadc01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc6520c63517170 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:32.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdae5ffb640 1 -- 172.21.15.46:0/138641607 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0a8a30 con 0x7fdae80c6bd0 2023-10-30T14:04:32.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdae5ffb640 1 -- 172.21.15.46:0/138641607 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc0a8b90 con 0x7fdae80c6bd0 2023-10-30T14:04:32.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdae5ffb640 1 -- 172.21.15.46:0/138641607 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0b0440 con 0x7fdae80c6bd0 2023-10-30T14:04:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdaed30f640 1 -- 172.21.15.46:0/138641607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdae80c6bd0 msgr2=0x7fdae806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdaed30f640 1 --2- 172.21.15.46:0/138641607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdae80c6bd0 0x7fdae806fb80 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009870 tx=0x7fdadc01ed30 comp rx=0 tx=0).stop 2023-10-30T14:04:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdaed30f640 1 -- 172.21.15.46:0/138641607 shutdown_connections 2023-10-30T14:04:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdaed30f640 1 --2- 172.21.15.46:0/138641607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdae80c6bd0 0x7fdae806fb80 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:32.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdaed30f640 1 -- 172.21.15.46:0/138641607 >> 172.21.15.46:0/138641607 conn(0x7fdae80c7910 msgr2=0x7fdae80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdaed30f640 1 -- 172.21.15.46:0/138641607 shutdown_connections 2023-10-30T14:04:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdaed30f640 1 -- 172.21.15.46:0/138641607 wait complete. 2023-10-30T14:04:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdaed30f640 1 Processor -- start 2023-10-30T14:04:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdaed30f640 1 -- start start 2023-10-30T14:04:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.092+0000 7fdaed30f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdae806f7a0 0x7fdae80c16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdaed30f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdae8071110 0x7fdae8156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdaed30f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdae80c6bd0 0x7fdae8157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdaed30f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae81efb20 con 0x7fdae806f7a0 2023-10-30T14:04:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdaed30f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae8156b50 con 0x7fdae8071110 2023-10-30T14:04:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdaed30f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae81561e0 con 0x7fdae80c6bd0 2023-10-30T14:04:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdae6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdae806f7a0 0x7fdae80c16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdae6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdae806f7a0 0x7fdae80c16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53296/0 (socket says 172.21.15.46:53296) 2023-10-30T14:04:32.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdae6ffd640 1 -- 172.21.15.46:0/2896623792 learned_addr learned my addr 172.21.15.46:0/2896623792 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdae77fe640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdae80c6bd0 0x7fdae8157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdae67fc640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdae8071110 0x7fdae8156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdae6ffd640 1 -- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdae80c6bd0 msgr2=0x7fdae8157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:32.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdae6ffd640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdae80c6bd0 0x7fdae8157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdae6ffd640 1 -- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdae8071110 msgr2=0x7fdae8156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdae6ffd640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdae8071110 0x7fdae8156fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdae6ffd640 1 -- 172.21.15.46:0/2896623792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc0094d0 con 0x7fdae806f7a0 2023-10-30T14:04:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdae6ffd640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdae806f7a0 0x7fdae80c16e0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fdad800c3c0 tx=0x7fdad800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:32.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdacf7fe640 1 -- 172.21.15.46:0/2896623792 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad8011790 con 0x7fdae806f7a0 2023-10-30T14:04:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdacf7fe640 1 -- 172.21.15.46:0/2896623792 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad80118f0 con 0x7fdae806f7a0 2023-10-30T14:04:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdacf7fe640 1 -- 172.21.15.46:0/2896623792 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad8011ca0 con 0x7fdae806f7a0 2023-10-30T14:04:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdaed30f640 1 -- 172.21.15.46:0/2896623792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae81f0010 con 0x7fdae806f7a0 2023-10-30T14:04:32.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdaed30f640 1 -- 172.21.15.46:0/2896623792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae81567e0 con 0x7fdae806f7a0 2023-10-30T14:04:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.096+0000 7fdaed30f640 1 -- 172.21.15.46:0/2896623792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae804e170 con 0x7fdae806f7a0 2023-10-30T14:04:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.100+0000 7fdacf7fe640 1 -- 172.21.15.46:0/2896623792 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdad8005080 con 0x7fdae806f7a0 2023-10-30T14:04:32.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.100+0000 7fdacf7fe640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdad40664a0 0x7fdad4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.100+0000 7fdacf7fe640 1 -- 172.21.15.46:0/2896623792 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fdad8084980 con 0x7fdae806f7a0 2023-10-30T14:04:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.100+0000 7fdae67fc640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdad40664a0 0x7fdad4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:32.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.100+0000 7fdae67fc640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdad40664a0 0x7fdad4068960 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0b2000 tx=0x7fdadc0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:32.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.100+0000 7fdacf7fe640 1 -- 172.21.15.46:0/2896623792 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdad8051640 con 0x7fdae806f7a0 2023-10-30T14:04:32.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 -- 172.21.15.46:0/2896623792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdae8150cf0 con 0x7fdae806f7a0 2023-10-30T14:04:32.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdacf7fe640 1 -- 172.21.15.46:0/2896623792 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fdad8054c60 con 0x7fdae806f7a0 2023-10-30T14:04:32.305 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:32.305 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:24.459809+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 175 2023-10-30T14:04:32.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 -- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdad40664a0 msgr2=0x7fdad4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:32.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdad40664a0 0x7fdad4068960 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0b2000 tx=0x7fdadc0aa070 comp rx=0 tx=0).stop 2023-10-30T14:04:32.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 -- 172.21.15.46:0/2896623792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdae806f7a0 msgr2=0x7fdae80c16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:32.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdae806f7a0 0x7fdae80c16e0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fdad800c3c0 tx=0x7fdad800c880 comp rx=0 tx=0).stop 2023-10-30T14:04:32.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 -- 172.21.15.46:0/2896623792 shutdown_connections 2023-10-30T14:04:32.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdad40664a0 0x7fdad4068960 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:32.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdae80c6bd0 0x7fdae8157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:32.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdae8071110 0x7fdae8156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:32.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 --2- 172.21.15.46:0/2896623792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdae806f7a0 0x7fdae80c16e0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:32.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 -- 172.21.15.46:0/2896623792 >> 172.21.15.46:0/2896623792 conn(0x7fdae80c7910 msgr2=0x7fdae80c8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:32.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 -- 172.21.15.46:0/2896623792 shutdown_connections 2023-10-30T14:04:32.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:32.300+0000 7fdaed30f640 1 -- 172.21.15.46:0/2896623792 wait complete. 2023-10-30T14:04:32.326 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:04:34.327 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:34.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cbe5aa640 1 Processor -- start 2023-10-30T14:04:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cbe5aa640 1 -- start start 2023-10-30T14:04:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cbe5aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8cb81538a0 0x7f8cb815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cbe5aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80ca0e0 0x7f8cb815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cbe5aa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80cb800 con 0x7f8cb80cf160 2023-10-30T14:04:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cbe5aa640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb815dc30 con 0x7f8cb81538a0 2023-10-30T14:04:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cbe5aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80cb330 con 0x7f8cb80ca0e0 2023-10-30T14:04:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cb7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8cb81538a0 0x7f8cb815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:34.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cb7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8cb81538a0 0x7f8cb815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53298/0 (socket says 172.21.15.46:53298) 2023-10-30T14:04:34.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cb7fff640 1 -- 172.21.15.46:0/287283819 learned_addr learned my addr 172.21.15.46:0/287283819 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:34.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cb77fe640 1 --2- 172.21.15.46:0/287283819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80ca0e0 0x7f8cb815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:34.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cb7fff640 1 -- 172.21.15.46:0/287283819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80ca0e0 msgr2=0x7f8cb815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:34.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cb7fff640 1 --2- 172.21.15.46:0/287283819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80ca0e0 0x7f8cb815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:34.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cb7fff640 1 -- 172.21.15.46:0/287283819 >> v1:172.21.15.154:6789/0 conn(0x7f8cb80cf160 legacy=0x7f8cb80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:34.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.468+0000 7f8cb7fff640 1 -- 172.21.15.46:0/287283819 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80cad70 con 0x7f8cb81538a0 2023-10-30T14:04:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.472+0000 7f8cb7fff640 1 --2- 172.21.15.46:0/287283819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb81538a0 0x7f8cb815d6f0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009870 tx=0x7f8ca801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175cc118379fb433 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.472+0000 7f8cb6ffd640 1 -- 172.21.15.46:0/287283819 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca80a89b0 con 0x7f8cb81538a0 2023-10-30T14:04:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.472+0000 7f8cb6ffd640 1 -- 172.21.15.46:0/287283819 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca80a8b10 con 0x7f8cb81538a0 2023-10-30T14:04:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.472+0000 7f8cb6ffd640 1 -- 172.21.15.46:0/287283819 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca80a8ea0 con 0x7f8cb81538a0 2023-10-30T14:04:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.472+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/287283819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb81538a0 msgr2=0x7f8cb815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.472+0000 7f8cbe5aa640 1 --2- 172.21.15.46:0/287283819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb81538a0 0x7f8cb815d6f0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009870 tx=0x7f8ca801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:04:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.472+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/287283819 shutdown_connections 2023-10-30T14:04:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.472+0000 7f8cbe5aa640 1 --2- 172.21.15.46:0/287283819 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80ca0e0 0x7f8cb815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.472+0000 7f8cbe5aa640 1 --2- 172.21.15.46:0/287283819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb81538a0 0x7f8cb815d6f0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.472+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/287283819 >> 172.21.15.46:0/287283819 conn(0x7f8cb806bfe0 msgr2=0x7f8cb806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.472+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/287283819 shutdown_connections 2023-10-30T14:04:34.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.472+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/287283819 wait complete. 2023-10-30T14:04:34.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbe5aa640 1 Processor -- start 2023-10-30T14:04:34.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbe5aa640 1 -- start start 2023-10-30T14:04:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbe5aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80ca0e0 0x7f8cb81e2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbe5aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cb80cf160 0x7f8cb8158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbe5aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb81538a0 0x7f8cb81ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbe5aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb81e3810 con 0x7f8cb81538a0 2023-10-30T14:04:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbe5aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb81f3100 con 0x7f8cb80cf160 2023-10-30T14:04:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbe5aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb81f3590 con 0x7f8cb80ca0e0 2023-10-30T14:04:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbcb47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb81538a0 0x7f8cb81ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbcb47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb81538a0 0x7f8cb81ef130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53310/0 (socket says 172.21.15.46:53310) 2023-10-30T14:04:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbcb47640 1 -- 172.21.15.46:0/3997131256 learned_addr learned my addr 172.21.15.46:0/3997131256 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cb7fff640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80ca0e0 0x7f8cb81e2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cb77fe640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cb80cf160 0x7f8cb8158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbcb47640 1 -- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80ca0e0 msgr2=0x7f8cb81e2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbcb47640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80ca0e0 0x7f8cb81e2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbcb47640 1 -- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cb80cf160 msgr2=0x7f8cb8158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbcb47640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cb80cf160 0x7f8cb8158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbcb47640 1 -- 172.21.15.46:0/3997131256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca80094d0 con 0x7f8cb81538a0 2023-10-30T14:04:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbcb47640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb81538a0 0x7f8cb81ef130 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000c7b0 tx=0x7f8cb000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:34.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cb4ff9640 1 -- 172.21.15.46:0/3997131256 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0004b20 con 0x7f8cb81538a0 2023-10-30T14:04:34.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cb4ff9640 1 -- 172.21.15.46:0/3997131256 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb0004c80 con 0x7f8cb81538a0 2023-10-30T14:04:34.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/3997131256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb81585e0 con 0x7f8cb81538a0 2023-10-30T14:04:34.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.476+0000 7f8cb4ff9640 1 -- 172.21.15.46:0/3997131256 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb000ce60 con 0x7f8cb81538a0 2023-10-30T14:04:34.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.480+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/3997131256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb81f27d0 con 0x7f8cb81538a0 2023-10-30T14:04:34.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.480+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/3997131256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb80b2a50 con 0x7f8cb81538a0 2023-10-30T14:04:34.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.480+0000 7f8cb4ff9640 1 -- 172.21.15.46:0/3997131256 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8cb000a850 con 0x7f8cb81538a0 2023-10-30T14:04:34.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.480+0000 7f8cb4ff9640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ca40664a0 0x7f8ca4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:34.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.480+0000 7f8cb4ff9640 1 -- 172.21.15.46:0/3997131256 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f8cb0084660 con 0x7f8cb81538a0 2023-10-30T14:04:34.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.480+0000 7f8cb7fff640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ca40664a0 0x7f8ca4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:34.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.480+0000 7f8cb7fff640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ca40664a0 0x7f8ca4068960 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80b2000 tx=0x7f8ca8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:34.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.484+0000 7f8cb4ff9640 1 -- 172.21.15.46:0/3997131256 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8cb0051350 con 0x7f8cb81538a0 2023-10-30T14:04:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/3997131256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cb804e170 con 0x7f8cb81538a0 2023-10-30T14:04:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cb4ff9640 1 -- 172.21.15.46:0/3997131256 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f8cb0054970 con 0x7f8cb81538a0 2023-10-30T14:04:34.680 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:34.681 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:24.459809+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:34.681 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 175 2023-10-30T14:04:34.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ca40664a0 msgr2=0x7f8ca4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:34.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ca40664a0 0x7f8ca4068960 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80b2000 tx=0x7f8ca8009040 comp rx=0 tx=0).stop 2023-10-30T14:04:34.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/3997131256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb81538a0 msgr2=0x7f8cb81ef130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:34.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb81538a0 0x7f8cb81ef130 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000c7b0 tx=0x7f8cb000cc70 comp rx=0 tx=0).stop 2023-10-30T14:04:34.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/3997131256 shutdown_connections 2023-10-30T14:04:34.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ca40664a0 0x7f8ca4068960 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:34.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb81538a0 0x7f8cb81ef130 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cb80cf160 0x7f8cb8158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 --2- 172.21.15.46:0/3997131256 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80ca0e0 0x7f8cb81e2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/3997131256 >> 172.21.15.46:0/3997131256 conn(0x7f8cb806bfe0 msgr2=0x7f8cb80c4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/3997131256 shutdown_connections 2023-10-30T14:04:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:34.676+0000 7f8cbe5aa640 1 -- 172.21.15.46:0/3997131256 wait complete. 2023-10-30T14:04:34.702 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:04:36.703 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.852+0000 7fb4c8217640 1 Processor -- start 2023-10-30T14:04:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.852+0000 7fb4c8217640 1 -- start start 2023-10-30T14:04:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c8217640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb4c00c6b80 0x7fb4c00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c8217640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4c013f4a0 0x7fb4c00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c8217640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c00c0a50 con 0x7fb4c00c6f60 2023-10-30T14:04:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c8217640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c006f6e0 con 0x7fb4c013f4a0 2023-10-30T14:04:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c8217640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c006f9c0 con 0x7fb4c00c6b80 2023-10-30T14:04:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c67b4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4c013f4a0 0x7fb4c00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c57b2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb4c00c6f60 0x7fb4c013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54994/0 (socket says 172.21.15.46:54994) 2023-10-30T14:04:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c67b4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4c013f4a0 0x7fb4c00c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59658/0 (socket says 172.21.15.46:59658) 2023-10-30T14:04:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c57b2640 1 -- 172.21.15.46:0/3406894253 learned_addr learned my addr 172.21.15.46:0/3406894253 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:04:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c5fb3640 1 --2- 172.21.15.46:0/3406894253 >> v2:172.21.15.154:3300/0 conn(0x7fb4c00c6b80 0x7fb4c00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c67b4640 1 -- 172.21.15.46:0/3406894253 >> v2:172.21.15.154:3300/0 conn(0x7fb4c00c6b80 msgr2=0x7fb4c00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c67b4640 1 --2- 172.21.15.46:0/3406894253 >> v2:172.21.15.154:3300/0 conn(0x7fb4c00c6b80 0x7fb4c00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c67b4640 1 -- 172.21.15.46:0/3406894253 >> v1:172.21.15.46:6789/0 conn(0x7fb4c00c6f60 legacy=0x7fb4c013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c67b4640 1 -- 172.21.15.46:0/3406894253 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c006fca0 con 0x7fb4c013f4a0 2023-10-30T14:04:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c4fb1640 1 -- 172.21.15.46:0/3406894253 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037347801 0 0) 0x7fb4c00c0a50 con 0x7fb4c00c6f60 2023-10-30T14:04:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.856+0000 7fb4c67b4640 1 --2- 172.21.15.46:0/3406894253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4c013f4a0 0x7fb4c00c0400 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fb4c006f1b0 tx=0x7fb4b801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e82f0c4930f2b5e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c4fb1640 1 -- 172.21.15.46:0/3406894253 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b80a58e0 con 0x7fb4c013f4a0 2023-10-30T14:04:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c4fb1640 1 -- 172.21.15.46:0/3406894253 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b80a5a40 con 0x7fb4c013f4a0 2023-10-30T14:04:36.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c4fb1640 1 -- 172.21.15.46:0/3406894253 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b80a5dd0 con 0x7fb4c013f4a0 2023-10-30T14:04:36.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 -- 172.21.15.46:0/3406894253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4c013f4a0 msgr2=0x7fb4c00c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 --2- 172.21.15.46:0/3406894253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4c013f4a0 0x7fb4c00c0400 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fb4c006f1b0 tx=0x7fb4b801c910 comp rx=0 tx=0).stop 2023-10-30T14:04:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 -- 172.21.15.46:0/3406894253 shutdown_connections 2023-10-30T14:04:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 --2- 172.21.15.46:0/3406894253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4c013f4a0 0x7fb4c00c0400 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 --2- 172.21.15.46:0/3406894253 >> v2:172.21.15.154:3300/0 conn(0x7fb4c00c6b80 0x7fb4c00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 -- 172.21.15.46:0/3406894253 >> 172.21.15.46:0/3406894253 conn(0x7fb4c00c7910 msgr2=0x7fb4c00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:36.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 -- 172.21.15.46:0/3406894253 shutdown_connections 2023-10-30T14:04:36.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 -- 172.21.15.46:0/3406894253 wait complete. 2023-10-30T14:04:36.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 Processor -- start 2023-10-30T14:04:36.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 -- start start 2023-10-30T14:04:36.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4c00c6b80 0x7fb4c01351f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:36.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4c00c6f60 0x7fb4c00c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:36.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4c013f4a0 0x7fb4c00ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:36.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c0139930 con 0x7fb4c013f4a0 2023-10-30T14:04:36.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c0134d60 con 0x7fb4c00c6f60 2023-10-30T14:04:36.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c8217640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c0135730 con 0x7fb4c00c6b80 2023-10-30T14:04:36.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c67b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4c013f4a0 0x7fb4c00ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:36.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c67b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4c013f4a0 0x7fb4c00ca530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59664/0 (socket says 172.21.15.46:59664) 2023-10-30T14:04:36.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c67b4640 1 -- 172.21.15.46:0/1358899516 learned_addr learned my addr 172.21.15.46:0/1358899516 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:36.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c5fb3640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4c00c6b80 0x7fb4c01351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:36.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.860+0000 7fb4c57b2640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4c00c6f60 0x7fb4c00c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:36.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4c67b4640 1 -- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4c00c6b80 msgr2=0x7fb4c01351f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:36.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4c67b4640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4c00c6b80 0x7fb4c01351f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:36.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4c67b4640 1 -- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4c00c6f60 msgr2=0x7fb4c00c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:36.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4c67b4640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4c00c6f60 0x7fb4c00c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:36.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4c67b4640 1 -- 172.21.15.46:0/1358899516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b8009a70 con 0x7fb4c013f4a0 2023-10-30T14:04:36.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4c67b4640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4c013f4a0 0x7fb4c00ca530 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8019520 tx=0x7fb4b80196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:36.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4b67fc640 1 -- 172.21.15.46:0/1358899516 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b80ad9d0 con 0x7fb4c013f4a0 2023-10-30T14:04:36.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4b67fc640 1 -- 172.21.15.46:0/1358899516 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b80adb30 con 0x7fb4c013f4a0 2023-10-30T14:04:36.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4b67fc640 1 -- 172.21.15.46:0/1358899516 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b80b9450 con 0x7fb4c013f4a0 2023-10-30T14:04:36.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4c8217640 1 -- 172.21.15.46:0/1358899516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c0135a70 con 0x7fb4c013f4a0 2023-10-30T14:04:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4c8217640 1 -- 172.21.15.46:0/1358899516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c013abd0 con 0x7fb4c013f4a0 2023-10-30T14:04:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4c8217640 1 -- 172.21.15.46:0/1358899516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c0134f10 con 0x7fb4c013f4a0 2023-10-30T14:04:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.864+0000 7fb4b67fc640 1 -- 172.21.15.46:0/1358899516 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb4b80b95b0 con 0x7fb4c013f4a0 2023-10-30T14:04:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.868+0000 7fb4b67fc640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4a40664a0 0x7fb4a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.868+0000 7fb4b67fc640 1 -- 172.21.15.46:0/1358899516 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fb4b811fcd0 con 0x7fb4c013f4a0 2023-10-30T14:04:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.868+0000 7fb4c5fb3640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4a40664a0 0x7fb4a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:36.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.868+0000 7fb4c5fb3640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4a40664a0 0x7fb4a4068960 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc007a60 tx=0x7fb4bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:36.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:36.872+0000 7fb4b67fc640 1 -- 172.21.15.46:0/1358899516 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4c0134f10 con 0x7fb4c013f4a0 2023-10-30T14:04:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.064+0000 7fb4c8217640 1 -- 172.21.15.46:0/1358899516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4c0134f10 con 0x7fb4c013f4a0 2023-10-30T14:04:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.064+0000 7fb4b67fc640 1 -- 172.21.15.46:0/1358899516 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+5734 (secure 0 0 0) 0x7fb4c0134f10 con 0x7fb4c013f4a0 2023-10-30T14:04:37.070 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:37.070 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:34.161880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:resolve","state_seq":34,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:37.071 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 176 2023-10-30T14:04:37.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.068+0000 7fb4c8217640 1 -- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4a40664a0 msgr2=0x7fb4a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:37.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.068+0000 7fb4c8217640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4a40664a0 0x7fb4a4068960 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc007a60 tx=0x7fb4bc008040 comp rx=0 tx=0).stop 2023-10-30T14:04:37.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.068+0000 7fb4c8217640 1 -- 172.21.15.46:0/1358899516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4c013f4a0 msgr2=0x7fb4c00ca530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:37.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.068+0000 7fb4c8217640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4c013f4a0 0x7fb4c00ca530 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8019520 tx=0x7fb4b80196b0 comp rx=0 tx=0).stop 2023-10-30T14:04:37.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.068+0000 7fb4c8217640 1 -- 172.21.15.46:0/1358899516 shutdown_connections 2023-10-30T14:04:37.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.068+0000 7fb4c8217640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb4a40664a0 0x7fb4a4068960 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:37.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.068+0000 7fb4c8217640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4c013f4a0 0x7fb4c00ca530 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:37.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.068+0000 7fb4c8217640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4c00c6f60 0x7fb4c00c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:37.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.068+0000 7fb4c8217640 1 --2- 172.21.15.46:0/1358899516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4c00c6b80 0x7fb4c01351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:37.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.068+0000 7fb4c8217640 1 -- 172.21.15.46:0/1358899516 >> 172.21.15.46:0/1358899516 conn(0x7fb4c00c7910 msgr2=0x7fb4c00c0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:37.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.068+0000 7fb4c8217640 1 -- 172.21.15.46:0/1358899516 shutdown_connections 2023-10-30T14:04:37.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:37.068+0000 7fb4c8217640 1 -- 172.21.15.46:0/1358899516 wait complete. 2023-10-30T14:04:37.091 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 176, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 176, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8034": { "addr": "172.21.15.46:6833/2075693104", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2075693104, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2075693104, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8034, "incarnation": 167, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 20 }, "gid_8190": { "addr": "172.21.15.46:6837/3154171789", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3154171789, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3154171789, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8190, "incarnation": 175, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:resolve", "state_seq": 34 } }, "last_failure": 0, "last_failure_osd_epoch": 78, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:04:34.161880+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8190, "mds_1": 8034 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:04:39.092 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.240+0000 7fccf75f8640 1 Processor -- start 2023-10-30T14:04:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.240+0000 7fccf75f8640 1 -- start start 2023-10-30T14:04:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf75f8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fccf0065a80 0x7fccf00cb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf75f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf0065f10 0x7fccf00cba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:39.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf75f8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf01560a0 con 0x7fccf014fd50 2023-10-30T14:04:39.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf75f8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf0156380 con 0x7fccf0065a80 2023-10-30T14:04:39.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf75f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf0156660 con 0x7fccf0065f10 2023-10-30T14:04:39.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf5394640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fccf014fd50 0x7fccf00c65d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55002/0 (socket says 172.21.15.46:55002) 2023-10-30T14:04:39.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf5394640 1 -- 172.21.15.46:0/2691133447 learned_addr learned my addr 172.21.15.46:0/2691133447 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:04:39.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf4b93640 1 --2- 172.21.15.46:0/2691133447 >> v2:172.21.15.46:3300/0 conn(0x7fccf0065a80 0x7fccf00cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:39.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf5b95640 1 --2- 172.21.15.46:0/2691133447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf0065f10 0x7fccf00cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:39.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fcce7fff640 1 -- 172.21.15.46:0/2691133447 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355942570 0 0) 0x7fccf01560a0 con 0x7fccf014fd50 2023-10-30T14:04:39.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf4b93640 1 -- 172.21.15.46:0/2691133447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf0065f10 msgr2=0x7fccf00cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:39.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf4b93640 1 --2- 172.21.15.46:0/2691133447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf0065f10 0x7fccf00cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:39.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf4b93640 1 -- 172.21.15.46:0/2691133447 >> v1:172.21.15.46:6789/0 conn(0x7fccf014fd50 legacy=0x7fccf00c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:39.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.244+0000 7fccf4b93640 1 -- 172.21.15.46:0/2691133447 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf0156940 con 0x7fccf0065a80 2023-10-30T14:04:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fccf4b93640 1 --2- 172.21.15.46:0/2691133447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccf0065a80 0x7fccf00cb550 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fcce00097c0 tx=0x7fcce001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=264a6da5279fcb45 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fcce7fff640 1 -- 172.21.15.46:0/2691133447 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce00ac620 con 0x7fccf0065a80 2023-10-30T14:04:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fcce7fff640 1 -- 172.21.15.46:0/2691133447 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce00ac780 con 0x7fccf0065a80 2023-10-30T14:04:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fcce7fff640 1 -- 172.21.15.46:0/2691133447 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce00acb10 con 0x7fccf0065a80 2023-10-30T14:04:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fccf75f8640 1 -- 172.21.15.46:0/2691133447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccf0065a80 msgr2=0x7fccf00cb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fccf75f8640 1 --2- 172.21.15.46:0/2691133447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccf0065a80 0x7fccf00cb550 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fcce00097c0 tx=0x7fcce001c420 comp rx=0 tx=0).stop 2023-10-30T14:04:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fccf75f8640 1 -- 172.21.15.46:0/2691133447 shutdown_connections 2023-10-30T14:04:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fccf75f8640 1 --2- 172.21.15.46:0/2691133447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf0065f10 0x7fccf00cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fccf75f8640 1 --2- 172.21.15.46:0/2691133447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccf0065a80 0x7fccf00cb550 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:39.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fccf75f8640 1 -- 172.21.15.46:0/2691133447 >> 172.21.15.46:0/2691133447 conn(0x7fccf006c6e0 msgr2=0x7fccf0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:39.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fccf75f8640 1 -- 172.21.15.46:0/2691133447 shutdown_connections 2023-10-30T14:04:39.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fccf75f8640 1 -- 172.21.15.46:0/2691133447 wait complete. 2023-10-30T14:04:39.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fccf75f8640 1 Processor -- start 2023-10-30T14:04:39.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.248+0000 7fccf75f8640 1 -- start start 2023-10-30T14:04:39.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf75f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccf0065a80 0x7fccf0145190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:39.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf75f8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccf0065f10 0x7fccf01f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:39.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf75f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf014fd50 0x7fccf01f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:39.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf75f8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf01f25c0 con 0x7fccf0065f10 2023-10-30T14:04:39.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf75f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf014f2b0 con 0x7fccf0065a80 2023-10-30T14:04:39.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf75f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf01552b0 con 0x7fccf014fd50 2023-10-30T14:04:39.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf4b93640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccf0065f10 0x7fccf01f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:39.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf4b93640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccf0065f10 0x7fccf01f1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59682/0 (socket says 172.21.15.46:59682) 2023-10-30T14:04:39.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf4b93640 1 -- 172.21.15.46:0/4249025828 learned_addr learned my addr 172.21.15.46:0/4249025828 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:39.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf5394640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccf0065a80 0x7fccf0145190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:39.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf5b95640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf014fd50 0x7fccf01f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:39.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf4b93640 1 -- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf014fd50 msgr2=0x7fccf01f2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:39.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf4b93640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf014fd50 0x7fccf01f2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:39.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf4b93640 1 -- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccf0065a80 msgr2=0x7fccf0145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:39.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf4b93640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccf0065a80 0x7fccf0145190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf4b93640 1 -- 172.21.15.46:0/4249025828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce0009420 con 0x7fccf0065f10 2023-10-30T14:04:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf4b93640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccf0065f10 0x7fccf01f1b40 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fcce001a0b0 tx=0x7fcce0019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fcce5ffb640 1 -- 172.21.15.46:0/4249025828 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce00ac620 con 0x7fccf0065f10 2023-10-30T14:04:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fcce5ffb640 1 -- 172.21.15.46:0/4249025828 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce0004030 con 0x7fccf0065f10 2023-10-30T14:04:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fcce5ffb640 1 -- 172.21.15.46:0/4249025828 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce00043a0 con 0x7fccf0065f10 2023-10-30T14:04:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf75f8640 1 -- 172.21.15.46:0/4249025828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf0155850 con 0x7fccf0065f10 2023-10-30T14:04:39.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf75f8640 1 -- 172.21.15.46:0/4249025828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf014e980 con 0x7fccf0065f10 2023-10-30T14:04:39.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fccf75f8640 1 -- 172.21.15.46:0/4249025828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf004e170 con 0x7fccf0065f10 2023-10-30T14:04:39.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.252+0000 7fcce5ffb640 1 -- 172.21.15.46:0/4249025828 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcce00b3040 con 0x7fccf0065f10 2023-10-30T14:04:39.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.256+0000 7fcce5ffb640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fccd40664a0 0x7fccd4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:39.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.256+0000 7fcce5ffb640 1 -- 172.21.15.46:0/4249025828 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fcce011da00 con 0x7fccf0065f10 2023-10-30T14:04:39.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.256+0000 7fccf5394640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fccd40664a0 0x7fccd4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:39.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.256+0000 7fccf5394640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fccd40664a0 0x7fccd4068960 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fccec000f70 tx=0x7fccec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:39.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.260+0000 7fcce5ffb640 1 -- 172.21.15.46:0/4249025828 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcce00ea5b0 con 0x7fccf0065f10 2023-10-30T14:04:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.448+0000 7fccf75f8640 1 -- 172.21.15.46:0/4249025828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccf00c7940 con 0x7fccf0065f10 2023-10-30T14:04:39.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.448+0000 7fcce5ffb640 1 -- 172.21.15.46:0/4249025828 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+5734 (secure 0 0 0) 0x7fcce00bd090 con 0x7fccf0065f10 2023-10-30T14:04:39.454 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:39.455 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:34.161880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:resolve","state_seq":34,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 176 2023-10-30T14:04:39.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.452+0000 7fccf75f8640 1 -- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fccd40664a0 msgr2=0x7fccd4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:39.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.452+0000 7fccf75f8640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fccd40664a0 0x7fccd4068960 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fccec000f70 tx=0x7fccec009040 comp rx=0 tx=0).stop 2023-10-30T14:04:39.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.452+0000 7fccf75f8640 1 -- 172.21.15.46:0/4249025828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccf0065f10 msgr2=0x7fccf01f1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:39.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.452+0000 7fccf75f8640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccf0065f10 0x7fccf01f1b40 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fcce001a0b0 tx=0x7fcce0019220 comp rx=0 tx=0).stop 2023-10-30T14:04:39.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.452+0000 7fccf75f8640 1 -- 172.21.15.46:0/4249025828 shutdown_connections 2023-10-30T14:04:39.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.452+0000 7fccf75f8640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fccd40664a0 0x7fccd4068960 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:39.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.452+0000 7fccf75f8640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf014fd50 0x7fccf01f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:39.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.452+0000 7fccf75f8640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccf0065f10 0x7fccf01f1b40 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:39.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.452+0000 7fccf75f8640 1 --2- 172.21.15.46:0/4249025828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccf0065a80 0x7fccf0145190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:39.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.452+0000 7fccf75f8640 1 -- 172.21.15.46:0/4249025828 >> 172.21.15.46:0/4249025828 conn(0x7fccf006c6e0 msgr2=0x7fccf00cab80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:39.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.452+0000 7fccf75f8640 1 -- 172.21.15.46:0/4249025828 shutdown_connections 2023-10-30T14:04:39.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:39.452+0000 7fccf75f8640 1 -- 172.21.15.46:0/4249025828 wait complete. 2023-10-30T14:04:39.476 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:04:41.477 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:41.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.628+0000 7fe43f63f640 1 Processor -- start 2023-10-30T14:04:41.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.628+0000 7fe43f63f640 1 -- start start 2023-10-30T14:04:41.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43f63f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe43815d6f0 0x7fe43815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:41.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43f63f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe4380cf160 0x7fe4380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:41.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43f63f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4380cb800 con 0x7fe4381538a0 2023-10-30T14:04:41.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43f63f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4380cb330 con 0x7fe4380cf160 2023-10-30T14:04:41.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43f63f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4380cad70 con 0x7fe43815d6f0 2023-10-30T14:04:41.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43d3db640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fe4381538a0 0x7fe4380ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:53046/0 (socket says 172.21.15.46:53046) 2023-10-30T14:04:41.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43d3db640 1 -- 172.21.15.46:0/2438994903 learned_addr learned my addr 172.21.15.46:0/2438994903 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:04:41.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43dbdc640 1 --2- 172.21.15.46:0/2438994903 >> v2:172.21.15.154:3300/0 conn(0x7fe4380cf160 0x7fe4380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:41.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43cbda640 1 --2- 172.21.15.46:0/2438994903 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe43815d6f0 0x7fe43815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:41.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe42ffff640 1 -- 172.21.15.46:0/2438994903 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35789447 0 0) 0x7fe4380cb800 con 0x7fe4381538a0 2023-10-30T14:04:41.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe42ffff640 1 -- 172.21.15.46:0/2438994903 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe424003530 con 0x7fe4381538a0 2023-10-30T14:04:41.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43dbdc640 1 -- 172.21.15.46:0/2438994903 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe43815d6f0 msgr2=0x7fe43815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:41.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43dbdc640 1 --2- 172.21.15.46:0/2438994903 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe43815d6f0 0x7fe43815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:41.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43dbdc640 1 -- 172.21.15.46:0/2438994903 >> v1:172.21.15.154:6789/0 conn(0x7fe4381538a0 legacy=0x7fe4380ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:41.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.632+0000 7fe43dbdc640 1 -- 172.21.15.46:0/2438994903 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4380cb050 con 0x7fe4380cf160 2023-10-30T14:04:41.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe43dbdc640 1 --2- 172.21.15.46:0/2438994903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4380cf160 0x7fe4380cf590 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe430009d90 tx=0x7fe43001f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3840bedce462ffe2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:41.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe42ffff640 1 -- 172.21.15.46:0/2438994903 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4300af600 con 0x7fe4380cf160 2023-10-30T14:04:41.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe42ffff640 1 -- 172.21.15.46:0/2438994903 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4300ab050 con 0x7fe4380cf160 2023-10-30T14:04:41.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe42ffff640 1 -- 172.21.15.46:0/2438994903 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4300af980 con 0x7fe4380cf160 2023-10-30T14:04:41.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe43f63f640 1 -- 172.21.15.46:0/2438994903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4380cf160 msgr2=0x7fe4380cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:41.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe43f63f640 1 --2- 172.21.15.46:0/2438994903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4380cf160 0x7fe4380cf590 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe430009d90 tx=0x7fe43001f130 comp rx=0 tx=0).stop 2023-10-30T14:04:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe43f63f640 1 -- 172.21.15.46:0/2438994903 shutdown_connections 2023-10-30T14:04:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe43f63f640 1 --2- 172.21.15.46:0/2438994903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4380cf160 0x7fe4380cf590 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe43f63f640 1 --2- 172.21.15.46:0/2438994903 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe43815d6f0 0x7fe43815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe43f63f640 1 -- 172.21.15.46:0/2438994903 >> 172.21.15.46:0/2438994903 conn(0x7fe43806bfe0 msgr2=0x7fe43806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe43f63f640 1 -- 172.21.15.46:0/2438994903 shutdown_connections 2023-10-30T14:04:41.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe43f63f640 1 -- 172.21.15.46:0/2438994903 wait complete. 2023-10-30T14:04:41.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe43f63f640 1 Processor -- start 2023-10-30T14:04:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.636+0000 7fe43f63f640 1 -- start start 2023-10-30T14:04:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43f63f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380cf160 0x7fe43815c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43f63f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4381538a0 0x7fe4380cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43f63f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe43815d6f0 0x7fe4380cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43f63f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4380ccfb0 con 0x7fe4380cf160 2023-10-30T14:04:41.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43f63f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4380b36b0 con 0x7fe4381538a0 2023-10-30T14:04:41.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43f63f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4380b4760 con 0x7fe43815d6f0 2023-10-30T14:04:41.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43d3db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380cf160 0x7fe43815c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:41.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43d3db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380cf160 0x7fe43815c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59690/0 (socket says 172.21.15.46:59690) 2023-10-30T14:04:41.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43d3db640 1 -- 172.21.15.46:0/2007648766 learned_addr learned my addr 172.21.15.46:0/2007648766 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43dbdc640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe43815d6f0 0x7fe4380cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43cbda640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4381538a0 0x7fe4380cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43d3db640 1 -- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe43815d6f0 msgr2=0x7fe4380cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43d3db640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe43815d6f0 0x7fe4380cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:41.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43d3db640 1 -- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4381538a0 msgr2=0x7fe4380cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:41.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43d3db640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4381538a0 0x7fe4380cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:41.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43d3db640 1 -- 172.21.15.46:0/2007648766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe430009a70 con 0x7fe4380cf160 2023-10-30T14:04:41.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43d3db640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380cf160 0x7fe43815c150 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fe43400c500 tx=0x7fe43400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:41.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe42dffb640 1 -- 172.21.15.46:0/2007648766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe434005cf0 con 0x7fe4380cf160 2023-10-30T14:04:41.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe42dffb640 1 -- 172.21.15.46:0/2007648766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe434005e50 con 0x7fe4380cf160 2023-10-30T14:04:41.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe42dffb640 1 -- 172.21.15.46:0/2007648766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4340072f0 con 0x7fe4380cf160 2023-10-30T14:04:41.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43f63f640 1 -- 172.21.15.46:0/2007648766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4380cea40 con 0x7fe4380cf160 2023-10-30T14:04:41.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43f63f640 1 -- 172.21.15.46:0/2007648766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4380b3340 con 0x7fe4380cf160 2023-10-30T14:04:41.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe43f63f640 1 -- 172.21.15.46:0/2007648766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4380b2a50 con 0x7fe4380cf160 2023-10-30T14:04:41.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.640+0000 7fe42dffb640 1 -- 172.21.15.46:0/2007648766 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe434007450 con 0x7fe4380cf160 2023-10-30T14:04:41.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.644+0000 7fe42dffb640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe41c0664d0 0x7fe41c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:41.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.644+0000 7fe42dffb640 1 -- 172.21.15.46:0/2007648766 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fe434083b60 con 0x7fe4380cf160 2023-10-30T14:04:41.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.644+0000 7fe43cbda640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe41c0664d0 0x7fe41c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:41.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.644+0000 7fe43cbda640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe41c0664d0 0x7fe41c068990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe438152f00 tx=0x7fe428009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:41.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.648+0000 7fe42dffb640 1 -- 172.21.15.46:0/2007648766 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe4340508e0 con 0x7fe4380cf160 2023-10-30T14:04:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 -- 172.21.15.46:0/2007648766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe43804e170 con 0x7fe4380cf160 2023-10-30T14:04:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe42dffb640 1 -- 172.21.15.46:0/2007648766 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+5736 (secure 0 0 0) 0x7fe434017090 con 0x7fe4380cf160 2023-10-30T14:04:41.841 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:41.841 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:39.766053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:41.842 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 177 2023-10-30T14:04:41.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 -- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe41c0664d0 msgr2=0x7fe41c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:41.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe41c0664d0 0x7fe41c068990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe438152f00 tx=0x7fe428009190 comp rx=0 tx=0).stop 2023-10-30T14:04:41.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 -- 172.21.15.46:0/2007648766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380cf160 msgr2=0x7fe43815c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:41.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380cf160 0x7fe43815c150 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fe43400c500 tx=0x7fe43400c9c0 comp rx=0 tx=0).stop 2023-10-30T14:04:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 -- 172.21.15.46:0/2007648766 shutdown_connections 2023-10-30T14:04:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe41c0664d0 0x7fe41c068990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe43815d6f0 0x7fe4380cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4381538a0 0x7fe4380cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 --2- 172.21.15.46:0/2007648766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380cf160 0x7fe43815c150 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 -- 172.21.15.46:0/2007648766 >> 172.21.15.46:0/2007648766 conn(0x7fe43806bfe0 msgr2=0x7fe438151d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 -- 172.21.15.46:0/2007648766 shutdown_connections 2023-10-30T14:04:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:41.836+0000 7fe43f63f640 1 -- 172.21.15.46:0/2007648766 wait complete. 2023-10-30T14:04:41.862 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:04:43.863 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.012+0000 7f0d5b90e640 1 Processor -- start 2023-10-30T14:04:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d5b90e640 1 -- start start 2023-10-30T14:04:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d5b90e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0d540c6b80 0x7f0d540bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d5b90e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0d5413f4a0 0x7f0d540c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d5b90e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d540c0a50 con 0x7f0d540c6f60 2023-10-30T14:04:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d5b90e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5406f6e0 con 0x7f0d540c6b80 2023-10-30T14:04:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d5b90e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5406f9c0 con 0x7f0d5413f4a0 2023-10-30T14:04:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d596aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0d540c6b80 0x7f0d540bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d596aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0d540c6b80 0x7f0d540bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59700/0 (socket says 172.21.15.46:59700) 2023-10-30T14:04:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d596aa640 1 -- 172.21.15.46:0/2396868591 learned_addr learned my addr 172.21.15.46:0/2396868591 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d59eab640 1 --2- 172.21.15.46:0/2396868591 >> v2:172.21.15.154:3300/0 conn(0x7f0d5413f4a0 0x7f0d540c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d596aa640 1 -- 172.21.15.46:0/2396868591 >> v2:172.21.15.154:3300/0 conn(0x7f0d5413f4a0 msgr2=0x7f0d540c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d596aa640 1 --2- 172.21.15.46:0/2396868591 >> v2:172.21.15.154:3300/0 conn(0x7f0d5413f4a0 0x7f0d540c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d596aa640 1 -- 172.21.15.46:0/2396868591 >> v1:172.21.15.154:6789/0 conn(0x7f0d540c6f60 legacy=0x7f0d5413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.016+0000 7f0d596aa640 1 -- 172.21.15.46:0/2396868591 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5406fca0 con 0x7f0d540c6b80 2023-10-30T14:04:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d596aa640 1 --2- 172.21.15.46:0/2396868591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d540c6b80 0x7f0d540bfec0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f0d540c8980 tx=0x7f0d5001c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3880744590a1e9d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d4bfff640 1 -- 172.21.15.46:0/2396868591 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d500ac630 con 0x7f0d540c6b80 2023-10-30T14:04:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d4bfff640 1 -- 172.21.15.46:0/2396868591 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d500ac790 con 0x7f0d540c6b80 2023-10-30T14:04:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d4bfff640 1 -- 172.21.15.46:0/2396868591 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d500acb20 con 0x7f0d540c6b80 2023-10-30T14:04:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d5b90e640 1 -- 172.21.15.46:0/2396868591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d540c6b80 msgr2=0x7f0d540bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d5b90e640 1 --2- 172.21.15.46:0/2396868591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d540c6b80 0x7f0d540bfec0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f0d540c8980 tx=0x7f0d5001c4c0 comp rx=0 tx=0).stop 2023-10-30T14:04:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d5b90e640 1 -- 172.21.15.46:0/2396868591 shutdown_connections 2023-10-30T14:04:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d5b90e640 1 --2- 172.21.15.46:0/2396868591 >> v2:172.21.15.154:3300/0 conn(0x7f0d5413f4a0 0x7f0d540c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d5b90e640 1 --2- 172.21.15.46:0/2396868591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d540c6b80 0x7f0d540bfec0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d5b90e640 1 -- 172.21.15.46:0/2396868591 >> 172.21.15.46:0/2396868591 conn(0x7f0d540c7910 msgr2=0x7f0d540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d5b90e640 1 -- 172.21.15.46:0/2396868591 shutdown_connections 2023-10-30T14:04:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d5b90e640 1 -- 172.21.15.46:0/2396868591 wait complete. 2023-10-30T14:04:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d5b90e640 1 Processor -- start 2023-10-30T14:04:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.020+0000 7f0d5b90e640 1 -- start start 2023-10-30T14:04:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d5b90e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d540c6b80 0x7f0d5413b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:44.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d5b90e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d540c6f60 0x7f0d54161f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:44.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d5b90e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d5413f4a0 0x7f0d54162480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:44.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d5b90e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d54163fd0 con 0x7f0d540c6f60 2023-10-30T14:04:44.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d5b90e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d54163a10 con 0x7f0d5413f4a0 2023-10-30T14:04:44.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d5b90e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d54163cf0 con 0x7f0d540c6b80 2023-10-30T14:04:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d58ea9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d540c6f60 0x7f0d54161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d58ea9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d540c6f60 0x7f0d54161f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59708/0 (socket says 172.21.15.46:59708) 2023-10-30T14:04:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d58ea9640 1 -- 172.21.15.46:0/3066774986 learned_addr learned my addr 172.21.15.46:0/3066774986 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d596aa640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d540c6b80 0x7f0d5413b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d59eab640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d5413f4a0 0x7f0d54162480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d58ea9640 1 -- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d540c6b80 msgr2=0x7f0d5413b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d58ea9640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d540c6b80 0x7f0d5413b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d58ea9640 1 -- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d5413f4a0 msgr2=0x7f0d54162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d58ea9640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d5413f4a0 0x7f0d54162480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d58ea9640 1 -- 172.21.15.46:0/3066774986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d500094d0 con 0x7f0d540c6f60 2023-10-30T14:04:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d59eab640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d5413f4a0 0x7f0d54162480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:04:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d58ea9640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d540c6f60 0x7f0d54161f40 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400c450 tx=0x7f0d4400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d49ffb640 1 -- 172.21.15.46:0/3066774986 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44012830 con 0x7f0d540c6f60 2023-10-30T14:04:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d49ffb640 1 -- 172.21.15.46:0/3066774986 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d440129b0 con 0x7f0d540c6f60 2023-10-30T14:04:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d49ffb640 1 -- 172.21.15.46:0/3066774986 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44012d60 con 0x7f0d540c6f60 2023-10-30T14:04:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d5b90e640 1 -- 172.21.15.46:0/3066774986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d540ca0c0 con 0x7f0d540c6f60 2023-10-30T14:04:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d5b90e640 1 -- 172.21.15.46:0/3066774986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d540ca680 con 0x7f0d540c6f60 2023-10-30T14:04:44.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d5b90e640 1 -- 172.21.15.46:0/3066774986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d540c9f20 con 0x7f0d540c6f60 2023-10-30T14:04:44.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.024+0000 7f0d49ffb640 1 -- 172.21.15.46:0/3066774986 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0d4401c460 con 0x7f0d540c6f60 2023-10-30T14:04:44.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.028+0000 7f0d49ffb640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0d380664d0 0x7f0d38068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.028+0000 7f0d49ffb640 1 -- 172.21.15.46:0/3066774986 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f0d44084420 con 0x7f0d540c6f60 2023-10-30T14:04:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.028+0000 7f0d596aa640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0d380664d0 0x7f0d38068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.028+0000 7f0d596aa640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0d380664d0 0x7f0d38068990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f0d500ae000 tx=0x7f0d500a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.032+0000 7f0d49ffb640 1 -- 172.21.15.46:0/3066774986 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0d540c9f20 con 0x7f0d540c6f60 2023-10-30T14:04:44.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 -- 172.21.15.46:0/3066774986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d540c9f20 con 0x7f0d540c6f60 2023-10-30T14:04:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d49ffb640 1 -- 172.21.15.46:0/3066774986 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f0d540c9f20 con 0x7f0d540c6f60 2023-10-30T14:04:44.225 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:44.225 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:39.766053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 177 2023-10-30T14:04:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 -- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0d380664d0 msgr2=0x7f0d38068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0d380664d0 0x7f0d38068990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f0d500ae000 tx=0x7f0d500a7000 comp rx=0 tx=0).stop 2023-10-30T14:04:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 -- 172.21.15.46:0/3066774986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d540c6f60 msgr2=0x7f0d54161f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d540c6f60 0x7f0d54161f40 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400c450 tx=0x7f0d4400c910 comp rx=0 tx=0).stop 2023-10-30T14:04:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 -- 172.21.15.46:0/3066774986 shutdown_connections 2023-10-30T14:04:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0d380664d0 0x7f0d38068990 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d5413f4a0 0x7f0d54162480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d540c6f60 0x7f0d54161f40 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 --2- 172.21.15.46:0/3066774986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d540c6b80 0x7f0d5413b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:44.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 -- 172.21.15.46:0/3066774986 >> 172.21.15.46:0/3066774986 conn(0x7f0d540c7910 msgr2=0x7f0d54160220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:44.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 -- 172.21.15.46:0/3066774986 shutdown_connections 2023-10-30T14:04:44.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:44.220+0000 7f0d5b90e640 1 -- 172.21.15.46:0/3066774986 wait complete. 2023-10-30T14:04:44.246 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:04:46.247 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.396+0000 7f20eb654640 1 Processor -- start 2023-10-30T14:04:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.396+0000 7f20eb654640 1 -- start start 2023-10-30T14:04:46.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.396+0000 7f20eb654640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f20e40c6540 0x7f20e415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:46.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.396+0000 7f20eb654640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e40c5810 con 0x7f20e4070740 2023-10-30T14:04:46.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.396+0000 7f20eb654640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e40c69b0 con 0x7f20e41580e0 2023-10-30T14:04:46.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.396+0000 7f20eb654640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e406acb0 con 0x7f20e40c6540 2023-10-30T14:04:46.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.400+0000 7f20e8bef640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f20e4070740 0x7f20e406a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55006/0 (socket says 172.21.15.46:55006) 2023-10-30T14:04:46.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.400+0000 7f20e8bef640 1 -- 172.21.15.46:0/1829870651 learned_addr learned my addr 172.21.15.46:0/1829870651 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:04:46.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.400+0000 7f20e9bf1640 1 --2- 172.21.15.46:0/1829870651 >> v2:172.21.15.154:3300/0 conn(0x7f20e40c6540 0x7f20e415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:46.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.400+0000 7f20dbfff640 1 -- 172.21.15.46:0/1829870651 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740496752 0 0) 0x7f20e40c5810 con 0x7f20e4070740 2023-10-30T14:04:46.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.400+0000 7f20dbfff640 1 -- 172.21.15.46:0/1829870651 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20d0003530 con 0x7f20e4070740 2023-10-30T14:04:46.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.400+0000 7f20e9bf1640 1 -- 172.21.15.46:0/1829870651 >> v1:172.21.15.154:6789/0 conn(0x7f20e41580e0 legacy=0x7f20e4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:46.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.400+0000 7f20e9bf1640 1 -- 172.21.15.46:0/1829870651 >> v1:172.21.15.46:6789/0 conn(0x7f20e4070740 legacy=0x7f20e406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:46.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.400+0000 7f20dbfff640 1 -- 172.21.15.46:0/1829870651 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1523702348 0 0) 0x7f20d0003530 con 0x7f20e4070740 2023-10-30T14:04:46.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.400+0000 7f20e9bf1640 1 -- 172.21.15.46:0/1829870651 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e40c6bb0 con 0x7f20e40c6540 2023-10-30T14:04:46.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.400+0000 7f20e9bf1640 1 --2- 172.21.15.46:0/1829870651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20e40c6540 0x7f20e415e700 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f20dc009d90 tx=0x7f20dc01c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ff7d684906a73e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:46.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20dbfff640 1 -- 172.21.15.46:0/1829870651 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4001680 con 0x7f20e40c6540 2023-10-30T14:04:46.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20dbfff640 1 -- 172.21.15.46:0/1829870651 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20dc0a55c0 con 0x7f20e40c6540 2023-10-30T14:04:46.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20dbfff640 1 -- 172.21.15.46:0/1829870651 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc0a5950 con 0x7f20e40c6540 2023-10-30T14:04:46.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 -- 172.21.15.46:0/1829870651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20e40c6540 msgr2=0x7f20e415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:46.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 --2- 172.21.15.46:0/1829870651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20e40c6540 0x7f20e415e700 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f20dc009d90 tx=0x7f20dc01c870 comp rx=0 tx=0).stop 2023-10-30T14:04:46.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 -- 172.21.15.46:0/1829870651 shutdown_connections 2023-10-30T14:04:46.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 --2- 172.21.15.46:0/1829870651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20e40c6540 0x7f20e415e700 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:46.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 -- 172.21.15.46:0/1829870651 >> 172.21.15.46:0/1829870651 conn(0x7f20e40c7910 msgr2=0x7f20e40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:46.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 -- 172.21.15.46:0/1829870651 shutdown_connections 2023-10-30T14:04:46.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 -- 172.21.15.46:0/1829870651 wait complete. 2023-10-30T14:04:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 Processor -- start 2023-10-30T14:04:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 -- start start 2023-10-30T14:04:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20e4070740 0x7f20e4148180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e40c6540 0x7f20e40c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20e41580e0 0x7f20e4152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e41486c0 con 0x7f20e40c6540 2023-10-30T14:04:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e41541d0 con 0x7f20e41580e0 2023-10-30T14:04:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20eb654640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e4147730 con 0x7f20e4070740 2023-10-30T14:04:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20e8bef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e40c6540 0x7f20e40c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20e8bef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e40c6540 0x7f20e40c10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59722/0 (socket says 172.21.15.46:59722) 2023-10-30T14:04:46.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.404+0000 7f20e8bef640 1 -- 172.21.15.46:0/2723088681 learned_addr learned my addr 172.21.15.46:0/2723088681 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20e9bf1640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20e41580e0 0x7f20e4152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20e93f0640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20e4070740 0x7f20e4148180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20e8bef640 1 -- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20e4070740 msgr2=0x7f20e4148180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20e8bef640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20e4070740 0x7f20e4148180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20e8bef640 1 -- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20e41580e0 msgr2=0x7f20e4152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20e8bef640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20e41580e0 0x7f20e4152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20e8bef640 1 -- 172.21.15.46:0/2723088681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20dc009a70 con 0x7f20e40c6540 2023-10-30T14:04:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20e8bef640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e40c6540 0x7f20e40c10e0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f20d400c730 tx=0x7f20d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20d9ffb640 1 -- 172.21.15.46:0/2723088681 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4005ae0 con 0x7f20e40c6540 2023-10-30T14:04:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20d9ffb640 1 -- 172.21.15.46:0/2723088681 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d4005c40 con 0x7f20e40c6540 2023-10-30T14:04:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20d9ffb640 1 -- 172.21.15.46:0/2723088681 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4007070 con 0x7f20e40c6540 2023-10-30T14:04:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20eb654640 1 -- 172.21.15.46:0/2723088681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e4071e00 con 0x7f20e40c6540 2023-10-30T14:04:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20eb654640 1 -- 172.21.15.46:0/2723088681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e4149240 con 0x7f20e40c6540 2023-10-30T14:04:46.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20eb654640 1 -- 172.21.15.46:0/2723088681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e404e170 con 0x7f20e40c6540 2023-10-30T14:04:46.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20d9ffb640 1 -- 172.21.15.46:0/2723088681 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f20d40071d0 con 0x7f20e40c6540 2023-10-30T14:04:46.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.408+0000 7f20d9ffb640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20c80664f0 0x7f20c80689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:46.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.412+0000 7f20d9ffb640 1 -- 172.21.15.46:0/2723088681 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f20d4083e80 con 0x7f20e40c6540 2023-10-30T14:04:46.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.412+0000 7f20e93f0640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20c80664f0 0x7f20c80689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:46.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.412+0000 7f20e93f0640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20c80664f0 0x7f20c80689b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f20e0000f70 tx=0x7f20e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:46.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.412+0000 7f20d9ffb640 1 -- 172.21.15.46:0/2723088681 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f20d4050b40 con 0x7f20e40c6540 2023-10-30T14:04:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.604+0000 7f20eb654640 1 -- 172.21.15.46:0/2723088681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20e4143d60 con 0x7f20e40c6540 2023-10-30T14:04:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.604+0000 7f20d9ffb640 1 -- 172.21.15.46:0/2723088681 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f20d4017070 con 0x7f20e40c6540 2023-10-30T14:04:46.609 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:46.610 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:45.559611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:46.610 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 178 2023-10-30T14:04:46.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.604+0000 7f20eb654640 1 -- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20c80664f0 msgr2=0x7f20c80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:46.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.604+0000 7f20eb654640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20c80664f0 0x7f20c80689b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f20e0000f70 tx=0x7f20e0009040 comp rx=0 tx=0).stop 2023-10-30T14:04:46.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.604+0000 7f20eb654640 1 -- 172.21.15.46:0/2723088681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e40c6540 msgr2=0x7f20e40c10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:46.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.604+0000 7f20eb654640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e40c6540 0x7f20e40c10e0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f20d400c730 tx=0x7f20d400cbf0 comp rx=0 tx=0).stop 2023-10-30T14:04:46.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.604+0000 7f20eb654640 1 -- 172.21.15.46:0/2723088681 shutdown_connections 2023-10-30T14:04:46.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.604+0000 7f20eb654640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20c80664f0 0x7f20c80689b0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:46.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.604+0000 7f20eb654640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20e41580e0 0x7f20e4152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:46.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.604+0000 7f20eb654640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e40c6540 0x7f20e40c10e0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:46.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.604+0000 7f20eb654640 1 --2- 172.21.15.46:0/2723088681 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20e4070740 0x7f20e4148180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:46.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.604+0000 7f20eb654640 1 -- 172.21.15.46:0/2723088681 >> 172.21.15.46:0/2723088681 conn(0x7f20e40c7910 msgr2=0x7f20e414e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:46.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.608+0000 7f20eb654640 1 -- 172.21.15.46:0/2723088681 shutdown_connections 2023-10-30T14:04:46.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:46.608+0000 7f20eb654640 1 -- 172.21.15.46:0/2723088681 wait complete. 2023-10-30T14:04:46.631 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:04:48.632 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:48.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b44291640 1 Processor -- start 2023-10-30T14:04:48.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b44291640 1 -- start start 2023-10-30T14:04:48.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b44291640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b3c0c6b80 0x7f2b3c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:48.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b44291640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3c0c0a50 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b44291640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3c06f6e0 con 0x7f2b3c0c6f60 2023-10-30T14:04:48.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b44291640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b3c06f9c0 con 0x7f2b3c0c6b80 2023-10-30T14:04:48.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b4202d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b3c0c6b80 0x7f2b3c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:48.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b4202d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b3c0c6b80 0x7f2b3c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55698/0 (socket says 172.21.15.46:55698) 2023-10-30T14:04:48.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b4202d640 1 -- 172.21.15.46:0/3768098474 learned_addr learned my addr 172.21.15.46:0/3768098474 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:48.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b4202d640 1 -- 172.21.15.46:0/3768098474 >> v1:172.21.15.154:6789/0 conn(0x7f2b3c0c6f60 legacy=0x7f2b3c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:48.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b4202d640 1 -- 172.21.15.46:0/3768098474 >> v1:172.21.15.46:6789/0 conn(0x7f2b3c13f4a0 legacy=0x7f2b3c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b4102b640 1 -- 172.21.15.46:0/3768098474 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117629138 0 0) 0x7f2b3c0c0a50 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.784+0000 7f2b4202d640 1 -- 172.21.15.46:0/3768098474 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c06fca0 con 0x7f2b3c0c6b80 2023-10-30T14:04:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.788+0000 7f2b4202d640 1 --2- 172.21.15.46:0/3768098474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b3c0c6b80 0x7f2b3c0bfec0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009870 tx=0x7f2b3801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f87083ab63ef40c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.788+0000 7f2b4102b640 1 -- 172.21.15.46:0/3768098474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b380ab650 con 0x7f2b3c0c6b80 2023-10-30T14:04:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.788+0000 7f2b4102b640 1 -- 172.21.15.46:0/3768098474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b380ab7b0 con 0x7f2b3c0c6b80 2023-10-30T14:04:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.788+0000 7f2b4102b640 1 -- 172.21.15.46:0/3768098474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b380abb40 con 0x7f2b3c0c6b80 2023-10-30T14:04:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.788+0000 7f2b44291640 1 -- 172.21.15.46:0/3768098474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b3c0c6b80 msgr2=0x7f2b3c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.788+0000 7f2b44291640 1 --2- 172.21.15.46:0/3768098474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b3c0c6b80 0x7f2b3c0bfec0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009870 tx=0x7f2b3801c540 comp rx=0 tx=0).stop 2023-10-30T14:04:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.788+0000 7f2b44291640 1 -- 172.21.15.46:0/3768098474 shutdown_connections 2023-10-30T14:04:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.788+0000 7f2b44291640 1 --2- 172.21.15.46:0/3768098474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b3c0c6b80 0x7f2b3c0bfec0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.788+0000 7f2b44291640 1 -- 172.21.15.46:0/3768098474 >> 172.21.15.46:0/3768098474 conn(0x7f2b3c0c7910 msgr2=0x7f2b3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.788+0000 7f2b44291640 1 -- 172.21.15.46:0/3768098474 shutdown_connections 2023-10-30T14:04:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.788+0000 7f2b44291640 1 -- 172.21.15.46:0/3768098474 wait complete. 2023-10-30T14:04:48.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b44291640 1 Processor -- start 2023-10-30T14:04:48.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b44291640 1 -- start start 2023-10-30T14:04:48.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b44291640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3c0c6b80 0x7f2b3c13afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:48.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b44291640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b3c0c6f60 0x7f2b3c13ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:48.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b44291640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b3c13f4a0 0x7f2b3c1396f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:48.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b44291640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c156490 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b44291640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c155240 con 0x7f2b3c0c6f60 2023-10-30T14:04:48.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b44291640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c154db0 con 0x7f2b3c0c6b80 2023-10-30T14:04:48.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b4282e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b3c13f4a0 0x7f2b3c1396f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:48.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b4282e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b3c13f4a0 0x7f2b3c1396f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55702/0 (socket says 172.21.15.46:55702) 2023-10-30T14:04:48.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b4282e640 1 -- 172.21.15.46:0/4146594480 learned_addr learned my addr 172.21.15.46:0/4146594480 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:48.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b4202d640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3c0c6b80 0x7f2b3c13afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:48.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b4182c640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b3c0c6f60 0x7f2b3c13ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:48.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b4282e640 1 -- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3c0c6b80 msgr2=0x7f2b3c13afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:48.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b4282e640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3c0c6b80 0x7f2b3c13afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:48.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b4282e640 1 -- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b3c0c6f60 msgr2=0x7f2b3c13ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:48.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b4282e640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b3c0c6f60 0x7f2b3c13ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:48.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b4282e640 1 -- 172.21.15.46:0/4146594480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b380094d0 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b4182c640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b3c0c6f60 0x7f2b3c13ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:04:48.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b4282e640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b3c13f4a0 0x7f2b3c1396f0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f2b34002240 tx=0x7f2b3400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:48.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b32ffd640 1 -- 172.21.15.46:0/4146594480 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34003db0 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b32ffd640 1 -- 172.21.15.46:0/4146594480 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3400ce30 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b32ffd640 1 -- 172.21.15.46:0/4146594480 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34005270 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b44291640 1 -- 172.21.15.46:0/4146594480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3c0c9050 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.792+0000 7f2b44291640 1 -- 172.21.15.46:0/4146594480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c154480 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.796+0000 7f2b44291640 1 -- 172.21.15.46:0/4146594480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c154f20 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.796+0000 7f2b32ffd640 1 -- 172.21.15.46:0/4146594480 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2b340053d0 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.796+0000 7f2b32ffd640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b200664a0 0x7f2b20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:48.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.796+0000 7f2b32ffd640 1 -- 172.21.15.46:0/4146594480 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f2b34083ea0 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.796+0000 7f2b4202d640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b200664a0 0x7f2b20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:48.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.796+0000 7f2b4202d640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b200664a0 0x7f2b20068960 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2b380af000 tx=0x7f2b380b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:48.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.800+0000 7f2b32ffd640 1 -- 172.21.15.46:0/4146594480 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2b34050b00 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.992+0000 7f2b44291640 1 -- 172.21.15.46:0/4146594480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b3c04e170 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.992+0000 7f2b32ffd640 1 -- 172.21.15.46:0/4146594480 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f2b34017090 con 0x7f2b3c13f4a0 2023-10-30T14:04:48.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:48.999 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:45.559611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:48.999 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 178 2023-10-30T14:04:49.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.996+0000 7f2b44291640 1 -- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b200664a0 msgr2=0x7f2b20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:49.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.996+0000 7f2b44291640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b200664a0 0x7f2b20068960 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2b380af000 tx=0x7f2b380b6040 comp rx=0 tx=0).stop 2023-10-30T14:04:49.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.996+0000 7f2b44291640 1 -- 172.21.15.46:0/4146594480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b3c13f4a0 msgr2=0x7f2b3c1396f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:49.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.996+0000 7f2b44291640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b3c13f4a0 0x7f2b3c1396f0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f2b34002240 tx=0x7f2b3400cbe0 comp rx=0 tx=0).stop 2023-10-30T14:04:49.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.996+0000 7f2b44291640 1 -- 172.21.15.46:0/4146594480 shutdown_connections 2023-10-30T14:04:49.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.996+0000 7f2b44291640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b200664a0 0x7f2b20068960 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.996+0000 7f2b44291640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b3c13f4a0 0x7f2b3c1396f0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.996+0000 7f2b44291640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b3c0c6f60 0x7f2b3c13ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.996+0000 7f2b44291640 1 --2- 172.21.15.46:0/4146594480 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b3c0c6b80 0x7f2b3c13afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.996+0000 7f2b44291640 1 -- 172.21.15.46:0/4146594480 >> 172.21.15.46:0/4146594480 conn(0x7f2b3c0c7910 msgr2=0x7f2b3c0c1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:49.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.996+0000 7f2b44291640 1 -- 172.21.15.46:0/4146594480 shutdown_connections 2023-10-30T14:04:49.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:48.996+0000 7f2b44291640 1 -- 172.21.15.46:0/4146594480 wait complete. 2023-10-30T14:04:49.021 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 178, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 178, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8034": { "addr": "172.21.15.46:6833/2075693104", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2075693104, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2075693104, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8034, "incarnation": 167, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 20 }, "gid_8190": { "addr": "172.21.15.46:6837/3154171789", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3154171789, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3154171789, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8190, "incarnation": 175, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 38 } }, "last_failure": 0, "last_failure_osd_epoch": 78, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:04:45.559611+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8190, "mds_1": 8034 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:04:51.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:51.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe173c5640 1 Processor -- start 2023-10-30T14:04:51.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe173c5640 1 -- start start 2023-10-30T14:04:51.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe173c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe101538a0 0x7fbe1015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:51.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe173c5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbe100cf160 0x7fbe100cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:51.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe173c5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe100cb800 con 0x7fbe100ca0e0 2023-10-30T14:04:51.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe173c5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe1015dc30 con 0x7fbe100cf160 2023-10-30T14:04:51.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe173c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe100cb330 con 0x7fbe101538a0 2023-10-30T14:04:51.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe15962640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbe100cf160 0x7fbe100cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:51.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe15962640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbe100cf160 0x7fbe100cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55704/0 (socket says 172.21.15.46:55704) 2023-10-30T14:04:51.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe15962640 1 -- 172.21.15.46:0/2657941694 learned_addr learned my addr 172.21.15.46:0/2657941694 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:51.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe15161640 1 --2- 172.21.15.46:0/2657941694 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe101538a0 0x7fbe1015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:51.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe15962640 1 -- 172.21.15.46:0/2657941694 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe101538a0 msgr2=0x7fbe1015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:51.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe15962640 1 --2- 172.21.15.46:0/2657941694 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe101538a0 0x7fbe1015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:51.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe15962640 1 -- 172.21.15.46:0/2657941694 >> v1:172.21.15.154:6789/0 conn(0x7fbe100ca0e0 legacy=0x7fbe1015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:51.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.172+0000 7fbe15962640 1 -- 172.21.15.46:0/2657941694 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe100cad70 con 0x7fbe100cf160 2023-10-30T14:04:51.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.176+0000 7fbe15962640 1 --2- 172.21.15.46:0/2657941694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe100cf160 0x7fbe100cf590 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009e50 tx=0x7fbe0801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e598f4d9a4e838 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:51.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.176+0000 7fbe07fff640 1 -- 172.21.15.46:0/2657941694 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe080af400 con 0x7fbe100cf160 2023-10-30T14:04:51.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.176+0000 7fbe07fff640 1 -- 172.21.15.46:0/2657941694 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe080ab050 con 0x7fbe100cf160 2023-10-30T14:04:51.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.176+0000 7fbe07fff640 1 -- 172.21.15.46:0/2657941694 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe080af780 con 0x7fbe100cf160 2023-10-30T14:04:51.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.176+0000 7fbe173c5640 1 -- 172.21.15.46:0/2657941694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe100cf160 msgr2=0x7fbe100cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:51.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.176+0000 7fbe173c5640 1 --2- 172.21.15.46:0/2657941694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe100cf160 0x7fbe100cf590 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009e50 tx=0x7fbe0801f1f0 comp rx=0 tx=0).stop 2023-10-30T14:04:51.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.176+0000 7fbe173c5640 1 -- 172.21.15.46:0/2657941694 shutdown_connections 2023-10-30T14:04:51.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.176+0000 7fbe173c5640 1 --2- 172.21.15.46:0/2657941694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe100cf160 0x7fbe100cf590 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:51.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.176+0000 7fbe173c5640 1 --2- 172.21.15.46:0/2657941694 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe101538a0 0x7fbe1015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:51.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.176+0000 7fbe173c5640 1 -- 172.21.15.46:0/2657941694 >> 172.21.15.46:0/2657941694 conn(0x7fbe1006bfe0 msgr2=0x7fbe1006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:51.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.176+0000 7fbe173c5640 1 -- 172.21.15.46:0/2657941694 shutdown_connections 2023-10-30T14:04:51.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.176+0000 7fbe173c5640 1 -- 172.21.15.46:0/2657941694 wait complete. 2023-10-30T14:04:51.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe173c5640 1 Processor -- start 2023-10-30T14:04:51.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe173c5640 1 -- start start 2023-10-30T14:04:51.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe173c5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe100ca0e0 0x7fbe1015a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:51.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe173c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe100cf160 0x7fbe1015b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:51.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe173c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe101538a0 0x7fbe1015b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:51.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe173c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe101e3960 con 0x7fbe100cf160 2023-10-30T14:04:51.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe173c5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe101e4750 con 0x7fbe100ca0e0 2023-10-30T14:04:51.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe173c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe101e3ba0 con 0x7fbe101538a0 2023-10-30T14:04:51.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe14960640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe100cf160 0x7fbe1015b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:51.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe14960640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe100cf160 0x7fbe1015b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55720/0 (socket says 172.21.15.46:55720) 2023-10-30T14:04:51.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe14960640 1 -- 172.21.15.46:0/3348510630 learned_addr learned my addr 172.21.15.46:0/3348510630 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:51.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe15962640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe101538a0 0x7fbe1015b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:51.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe15161640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe100ca0e0 0x7fbe1015a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:51.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe14960640 1 -- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe101538a0 msgr2=0x7fbe1015b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:51.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe14960640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe101538a0 0x7fbe1015b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:51.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe14960640 1 -- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe100ca0e0 msgr2=0x7fbe1015a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:51.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe14960640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe100ca0e0 0x7fbe1015a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:51.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe14960640 1 -- 172.21.15.46:0/3348510630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe08009b30 con 0x7fbe100cf160 2023-10-30T14:04:51.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe14960640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe100cf160 0x7fbe1015b300 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fbe0000c500 tx=0x7fbe0000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:51.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3348510630 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00005220 con 0x7fbe100cf160 2023-10-30T14:04:51.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3348510630 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe00005380 con 0x7fbe100cf160 2023-10-30T14:04:51.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3348510630 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00007270 con 0x7fbe100cf160 2023-10-30T14:04:51.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe173c5640 1 -- 172.21.15.46:0/3348510630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe101e3430 con 0x7fbe100cf160 2023-10-30T14:04:51.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.180+0000 7fbe173c5640 1 -- 172.21.15.46:0/3348510630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe101e3db0 con 0x7fbe100cf160 2023-10-30T14:04:51.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.184+0000 7fbe173c5640 1 -- 172.21.15.46:0/3348510630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe100b2a50 con 0x7fbe100cf160 2023-10-30T14:04:51.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.184+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3348510630 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbe000073d0 con 0x7fbe100cf160 2023-10-30T14:04:51.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.184+0000 7fbe05ffb640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbdf40664a0 0x7fbdf4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:51.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.184+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3348510630 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fbe00083ff0 con 0x7fbe100cf160 2023-10-30T14:04:51.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.184+0000 7fbe15161640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbdf40664a0 0x7fbdf4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:51.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.184+0000 7fbe15161640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbdf40664a0 0x7fbdf4068960 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00b930 tx=0x7fbe0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.188+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3348510630 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbe00050c40 con 0x7fbe100cf160 2023-10-30T14:04:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.392+0000 7fbe173c5640 1 -- 172.21.15.46:0/3348510630 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe1004e170 con 0x7fbe100cf160 2023-10-30T14:04:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.392+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3348510630 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fbe00054260 con 0x7fbe100cf160 2023-10-30T14:04:51.397 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:51.398 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:45.559611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:51.398 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 178 2023-10-30T14:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.392+0000 7fbe173c5640 1 -- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbdf40664a0 msgr2=0x7fbdf4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:51.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.392+0000 7fbe173c5640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbdf40664a0 0x7fbdf4068960 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00b930 tx=0x7fbe0c008040 comp rx=0 tx=0).stop 2023-10-30T14:04:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.392+0000 7fbe173c5640 1 -- 172.21.15.46:0/3348510630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe100cf160 msgr2=0x7fbe1015b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.392+0000 7fbe173c5640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe100cf160 0x7fbe1015b300 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fbe0000c500 tx=0x7fbe0000c9c0 comp rx=0 tx=0).stop 2023-10-30T14:04:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.396+0000 7fbe173c5640 1 -- 172.21.15.46:0/3348510630 shutdown_connections 2023-10-30T14:04:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.396+0000 7fbe173c5640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbdf40664a0 0x7fbdf4068960 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:51.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.396+0000 7fbe173c5640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe101538a0 0x7fbe1015b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:51.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.396+0000 7fbe173c5640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe100cf160 0x7fbe1015b300 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:51.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.396+0000 7fbe173c5640 1 --2- 172.21.15.46:0/3348510630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe100ca0e0 0x7fbe1015a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:51.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.396+0000 7fbe173c5640 1 -- 172.21.15.46:0/3348510630 >> 172.21.15.46:0/3348510630 conn(0x7fbe1006bfe0 msgr2=0x7fbe100c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:51.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.396+0000 7fbe173c5640 1 -- 172.21.15.46:0/3348510630 shutdown_connections 2023-10-30T14:04:51.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:51.396+0000 7fbe173c5640 1 -- 172.21.15.46:0/3348510630 wait complete. 2023-10-30T14:04:51.419 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:04:53.421 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.564+0000 7f83bd217640 1 Processor -- start 2023-10-30T14:04:53.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.564+0000 7f83bd217640 1 -- start start 2023-10-30T14:04:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.568+0000 7f83bd217640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83b806f7a0 0x7f83b806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.568+0000 7f83bd217640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b8066cf0 con 0x7f83b80c6bd0 2023-10-30T14:04:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.568+0000 7f83bd217640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b80c70d0 con 0x7f83b8071110 2023-10-30T14:04:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.568+0000 7f83bd217640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b80700c0 con 0x7f83b806f7a0 2023-10-30T14:04:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.568+0000 7f83b6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83b806f7a0 0x7f83b806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.568+0000 7f83b659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f83b80c6bd0 0x7f83b8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53250/0 (socket says 172.21.15.46:53250) 2023-10-30T14:04:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.568+0000 7f83b6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83b806f7a0 0x7f83b806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55730/0 (socket says 172.21.15.46:55730) 2023-10-30T14:04:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.568+0000 7f83b659c640 1 -- 172.21.15.46:0/1606541072 learned_addr learned my addr 172.21.15.46:0/1606541072 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:04:53.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.568+0000 7f83b6d9d640 1 -- 172.21.15.46:0/1606541072 >> v1:172.21.15.154:6789/0 conn(0x7f83b8071110 legacy=0x7f83b8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:53.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.568+0000 7f83b6d9d640 1 -- 172.21.15.46:0/1606541072 >> v1:172.21.15.46:6789/0 conn(0x7f83b80c6bd0 legacy=0x7f83b8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:53.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.568+0000 7f83b6d9d640 1 -- 172.21.15.46:0/1606541072 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b80c7390 con 0x7f83b806f7a0 2023-10-30T14:04:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83b6d9d640 1 --2- 172.21.15.46:0/1606541072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83b806f7a0 0x7f83b806fb80 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f83a80097c0 tx=0x7f83a801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9240c4ef6bac6ef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:53.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83b5d9b640 1 -- 172.21.15.46:0/1606541072 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a80a8970 con 0x7f83b806f7a0 2023-10-30T14:04:53.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83b5d9b640 1 -- 172.21.15.46:0/1606541072 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a80a8ad0 con 0x7f83b806f7a0 2023-10-30T14:04:53.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83b5d9b640 1 -- 172.21.15.46:0/1606541072 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a80a8e60 con 0x7f83b806f7a0 2023-10-30T14:04:53.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83bd217640 1 -- 172.21.15.46:0/1606541072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83b806f7a0 msgr2=0x7f83b806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:53.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83bd217640 1 --2- 172.21.15.46:0/1606541072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83b806f7a0 0x7f83b806fb80 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f83a80097c0 tx=0x7f83a801ec70 comp rx=0 tx=0).stop 2023-10-30T14:04:53.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83bd217640 1 -- 172.21.15.46:0/1606541072 shutdown_connections 2023-10-30T14:04:53.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83bd217640 1 --2- 172.21.15.46:0/1606541072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83b806f7a0 0x7f83b806fb80 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:53.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83bd217640 1 -- 172.21.15.46:0/1606541072 >> 172.21.15.46:0/1606541072 conn(0x7f83b80c7910 msgr2=0x7f83b80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:53.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83bd217640 1 -- 172.21.15.46:0/1606541072 shutdown_connections 2023-10-30T14:04:53.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83bd217640 1 -- 172.21.15.46:0/1606541072 wait complete. 2023-10-30T14:04:53.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83bd217640 1 Processor -- start 2023-10-30T14:04:53.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.572+0000 7f83bd217640 1 -- start start 2023-10-30T14:04:53.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83bd217640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83b806f7a0 0x7f83b8155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:53.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83bd217640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83b8071110 0x7f83b8150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:53.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83bd217640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83b80c6bd0 0x7f83b8154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:53.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83bd217640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b8156780 con 0x7f83b80c6bd0 2023-10-30T14:04:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83bd217640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80c96d0 con 0x7f83b806f7a0 2023-10-30T14:04:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83bd217640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b8146910 con 0x7f83b8071110 2023-10-30T14:04:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83b759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83b80c6bd0 0x7f83b8154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83b759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83b80c6bd0 0x7f83b8154290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55740/0 (socket says 172.21.15.46:55740) 2023-10-30T14:04:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83b759e640 1 -- 172.21.15.46:0/1609838713 learned_addr learned my addr 172.21.15.46:0/1609838713 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83b6d9d640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83b806f7a0 0x7f83b8155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83b659c640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83b8071110 0x7f83b8150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83b759e640 1 -- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83b8071110 msgr2=0x7f83b8150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83b759e640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83b8071110 0x7f83b8150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83b759e640 1 -- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83b806f7a0 msgr2=0x7f83b8155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83b759e640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83b806f7a0 0x7f83b8155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83b759e640 1 -- 172.21.15.46:0/1609838713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a8009420 con 0x7f83b80c6bd0 2023-10-30T14:04:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83b759e640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83b80c6bd0 0x7f83b8154290 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f83b000c760 tx=0x7f83b000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f839f7fe640 1 -- 172.21.15.46:0/1609838713 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0003b80 con 0x7f83b80c6bd0 2023-10-30T14:04:53.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f839f7fe640 1 -- 172.21.15.46:0/1609838713 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b0003ce0 con 0x7f83b80c6bd0 2023-10-30T14:04:53.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f839f7fe640 1 -- 172.21.15.46:0/1609838713 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b000aa30 con 0x7f83b80c6bd0 2023-10-30T14:04:53.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83bd217640 1 -- 172.21.15.46:0/1609838713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b80c93f0 con 0x7f83b80c6bd0 2023-10-30T14:04:53.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83bd217640 1 -- 172.21.15.46:0/1609838713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b8150640 con 0x7f83b80c6bd0 2023-10-30T14:04:53.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f83bd217640 1 -- 172.21.15.46:0/1609838713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b8150860 con 0x7f83b80c6bd0 2023-10-30T14:04:53.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.576+0000 7f839f7fe640 1 -- 172.21.15.46:0/1609838713 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83b000ab90 con 0x7f83b80c6bd0 2023-10-30T14:04:53.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.580+0000 7f839f7fe640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83a40664d0 0x7f83a4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:53.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.580+0000 7f839f7fe640 1 -- 172.21.15.46:0/1609838713 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f83b00843c0 con 0x7f83b80c6bd0 2023-10-30T14:04:53.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.580+0000 7f83b6d9d640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83a40664d0 0x7f83a4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:53.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.580+0000 7f83b6d9d640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83a40664d0 0x7f83a4068990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f83a80b2000 tx=0x7f83a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:53.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.584+0000 7f839f7fe640 1 -- 172.21.15.46:0/1609838713 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f83b0051090 con 0x7f83b80c6bd0 2023-10-30T14:04:53.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.772+0000 7f83bd217640 1 -- 172.21.15.46:0/1609838713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83b804e170 con 0x7f83b80c6bd0 2023-10-30T14:04:53.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.772+0000 7f839f7fe640 1 -- 172.21.15.46:0/1609838713 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f83b00546b0 con 0x7f83b80c6bd0 2023-10-30T14:04:53.778 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:53.779 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:50.820260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:53.779 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 179 2023-10-30T14:04:53.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.776+0000 7f83bd217640 1 -- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83a40664d0 msgr2=0x7f83a4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:53.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.776+0000 7f83bd217640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83a40664d0 0x7f83a4068990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f83a80b2000 tx=0x7f83a8008040 comp rx=0 tx=0).stop 2023-10-30T14:04:53.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.776+0000 7f83bd217640 1 -- 172.21.15.46:0/1609838713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83b80c6bd0 msgr2=0x7f83b8154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:53.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.776+0000 7f83bd217640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83b80c6bd0 0x7f83b8154290 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f83b000c760 tx=0x7f83b000cc20 comp rx=0 tx=0).stop 2023-10-30T14:04:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.776+0000 7f83bd217640 1 -- 172.21.15.46:0/1609838713 shutdown_connections 2023-10-30T14:04:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.776+0000 7f83bd217640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83a40664d0 0x7f83a4068990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.776+0000 7f83bd217640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83b80c6bd0 0x7f83b8154290 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.776+0000 7f83bd217640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83b8071110 0x7f83b8150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:53.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.776+0000 7f83bd217640 1 --2- 172.21.15.46:0/1609838713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83b806f7a0 0x7f83b8155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.776+0000 7f83bd217640 1 -- 172.21.15.46:0/1609838713 >> 172.21.15.46:0/1609838713 conn(0x7f83b80c7910 msgr2=0x7f83b80c3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.776+0000 7f83bd217640 1 -- 172.21.15.46:0/1609838713 shutdown_connections 2023-10-30T14:04:53.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:53.776+0000 7f83bd217640 1 -- 172.21.15.46:0/1609838713 wait complete. 2023-10-30T14:04:53.799 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=7704 2023-10-30T14:04:53.799 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.c 2023-10-30T14:04:55.308 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:04:55.313 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:04:56.801 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T14:04:56.801 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T14:04:56.801 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T14:04:56.805 INFO:tasks.ceph.mds.c:Started 2023-10-30T14:04:56.805 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:56.918 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-10-30T14:04:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd84009e640 1 Processor -- start 2023-10-30T14:04:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd84009e640 1 -- start start 2023-10-30T14:04:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd84009e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd83815d6f0 0x7fd83815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:56.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd84009e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd8380cf160 0x7fd8380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:56.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd84009e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8380cb800 con 0x7fd8381538a0 2023-10-30T14:04:56.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd84009e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8380cb330 con 0x7fd83815d6f0 2023-10-30T14:04:56.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd84009e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8380cad70 con 0x7fd8380cf160 2023-10-30T14:04:56.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd83d639640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd83815d6f0 0x7fd83815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:56.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd83d639640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd83815d6f0 0x7fd83815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50234/0 (socket says 172.21.15.46:50234) 2023-10-30T14:04:56.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd83d639640 1 -- 172.21.15.46:0/773750018 learned_addr learned my addr 172.21.15.46:0/773750018 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:56.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd83e63b640 1 --2- 172.21.15.46:0/773750018 >> v2:172.21.15.154:3300/0 conn(0x7fd8380cf160 0x7fd8380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:56.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd83d639640 1 -- 172.21.15.46:0/773750018 >> v2:172.21.15.154:3300/0 conn(0x7fd8380cf160 msgr2=0x7fd8380cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:56.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd83d639640 1 --2- 172.21.15.46:0/773750018 >> v2:172.21.15.154:3300/0 conn(0x7fd8380cf160 0x7fd8380cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:56.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd83d639640 1 -- 172.21.15.46:0/773750018 >> v1:172.21.15.154:6789/0 conn(0x7fd8381538a0 legacy=0x7fd8380ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:56.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.968+0000 7fd83d639640 1 -- 172.21.15.46:0/773750018 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8380cb050 con 0x7fd83815d6f0 2023-10-30T14:04:56.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.972+0000 7fd83d639640 1 --2- 172.21.15.46:0/773750018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd83815d6f0 0x7fd83815a090 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fd8380b5330 tx=0x7fd82801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31ebc61014be28e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:56.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.972+0000 7fd83ce38640 1 -- 172.21.15.46:0/773750018 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8280a89f0 con 0x7fd83815d6f0 2023-10-30T14:04:56.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.972+0000 7fd83ce38640 1 -- 172.21.15.46:0/773750018 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8280a8b50 con 0x7fd83815d6f0 2023-10-30T14:04:56.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.972+0000 7fd83ce38640 1 -- 172.21.15.46:0/773750018 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8280b0440 con 0x7fd83815d6f0 2023-10-30T14:04:56.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.972+0000 7fd84009e640 1 -- 172.21.15.46:0/773750018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd83815d6f0 msgr2=0x7fd83815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:56.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.972+0000 7fd84009e640 1 --2- 172.21.15.46:0/773750018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd83815d6f0 0x7fd83815a090 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fd8380b5330 tx=0x7fd82801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:04:56.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.972+0000 7fd84009e640 1 -- 172.21.15.46:0/773750018 shutdown_connections 2023-10-30T14:04:56.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.972+0000 7fd84009e640 1 --2- 172.21.15.46:0/773750018 >> v2:172.21.15.154:3300/0 conn(0x7fd8380cf160 0x7fd8380cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:56.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.972+0000 7fd84009e640 1 --2- 172.21.15.46:0/773750018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd83815d6f0 0x7fd83815a090 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:56.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.972+0000 7fd84009e640 1 -- 172.21.15.46:0/773750018 >> 172.21.15.46:0/773750018 conn(0x7fd83806bfe0 msgr2=0x7fd83806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:56.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.972+0000 7fd84009e640 1 -- 172.21.15.46:0/773750018 shutdown_connections 2023-10-30T14:04:56.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.972+0000 7fd84009e640 1 -- 172.21.15.46:0/773750018 wait complete. 2023-10-30T14:04:56.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd84009e640 1 Processor -- start 2023-10-30T14:04:56.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd84009e640 1 -- start start 2023-10-30T14:04:56.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd84009e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8380cf160 0x7fd8380c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:56.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd84009e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8381538a0 0x7fd8380c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:56.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd84009e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd83815d6f0 0x7fd838157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:56.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd84009e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8381e40a0 con 0x7fd8381538a0 2023-10-30T14:04:56.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd84009e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd838150f30 con 0x7fd8380cf160 2023-10-30T14:04:56.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd84009e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8381e3dc0 con 0x7fd83815d6f0 2023-10-30T14:04:56.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd83d639640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8381538a0 0x7fd8380c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:56.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd83d639640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8381538a0 0x7fd8380c6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50248/0 (socket says 172.21.15.46:50248) 2023-10-30T14:04:56.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd83d639640 1 -- 172.21.15.46:0/2347735518 learned_addr learned my addr 172.21.15.46:0/2347735518 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:56.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd83de3a640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8380cf160 0x7fd8380c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:56.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd83e63b640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd83815d6f0 0x7fd838157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:56.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd83d639640 1 -- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd83815d6f0 msgr2=0x7fd838157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:56.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd83d639640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd83815d6f0 0x7fd838157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:56.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd83d639640 1 -- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8380cf160 msgr2=0x7fd8380c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:56.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd83d639640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8380cf160 0x7fd8380c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:56.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd83d639640 1 -- 172.21.15.46:0/2347735518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8280094d0 con 0x7fd8381538a0 2023-10-30T14:04:56.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd83de3a640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8380cf160 0x7fd8380c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:04:56.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd83d639640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8381538a0 0x7fd8380c6730 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fd82801b900 tx=0x7fd82801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:56.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd82e7fc640 1 -- 172.21.15.46:0/2347735518 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8280ba600 con 0x7fd8381538a0 2023-10-30T14:04:56.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd82e7fc640 1 -- 172.21.15.46:0/2347735518 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8280ba760 con 0x7fd8381538a0 2023-10-30T14:04:56.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd82e7fc640 1 -- 172.21.15.46:0/2347735518 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8280bab10 con 0x7fd8381538a0 2023-10-30T14:04:56.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd84009e640 1 -- 172.21.15.46:0/2347735518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8381e5250 con 0x7fd8381538a0 2023-10-30T14:04:56.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd84009e640 1 -- 172.21.15.46:0/2347735518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8380c8e10 con 0x7fd8381538a0 2023-10-30T14:04:56.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.976+0000 7fd84009e640 1 -- 172.21.15.46:0/2347735518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8380b2a50 con 0x7fd8381538a0 2023-10-30T14:04:56.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.980+0000 7fd82e7fc640 1 -- 172.21.15.46:0/2347735518 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd8280bac70 con 0x7fd8381538a0 2023-10-30T14:04:56.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.980+0000 7fd82e7fc640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd81c0664a0 0x7fd81c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:56.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.980+0000 7fd82e7fc640 1 -- 172.21.15.46:0/2347735518 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fd828122c30 con 0x7fd8381538a0 2023-10-30T14:04:56.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.980+0000 7fd83de3a640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd81c0664a0 0x7fd81c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:56.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.980+0000 7fd83de3a640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd81c0664a0 0x7fd81c068960 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd834000f70 tx=0x7fd834009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:56.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:56.984+0000 7fd82e7fc640 1 -- 172.21.15.46:0/2347735518 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd8280ef870 con 0x7fd8381538a0 2023-10-30T14:04:57.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.172+0000 7fd84009e640 1 -- 172.21.15.46:0/2347735518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd83804e170 con 0x7fd8381538a0 2023-10-30T14:04:57.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.172+0000 7fd82e7fc640 1 -- 172.21.15.46:0/2347735518 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fd8280b5020 con 0x7fd8381538a0 2023-10-30T14:04:57.179 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:57.179 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:50.820260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:57.179 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 179 2023-10-30T14:04:57.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.176+0000 7fd84009e640 1 -- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd81c0664a0 msgr2=0x7fd81c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:57.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.176+0000 7fd84009e640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd81c0664a0 0x7fd81c068960 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd834000f70 tx=0x7fd834009040 comp rx=0 tx=0).stop 2023-10-30T14:04:57.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.176+0000 7fd84009e640 1 -- 172.21.15.46:0/2347735518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8381538a0 msgr2=0x7fd8380c6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:57.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.176+0000 7fd84009e640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8381538a0 0x7fd8380c6730 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fd82801b900 tx=0x7fd82801ba90 comp rx=0 tx=0).stop 2023-10-30T14:04:57.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.176+0000 7fd84009e640 1 -- 172.21.15.46:0/2347735518 shutdown_connections 2023-10-30T14:04:57.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.176+0000 7fd84009e640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd81c0664a0 0x7fd81c068960 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:57.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.176+0000 7fd84009e640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd83815d6f0 0x7fd838157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:57.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.176+0000 7fd84009e640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8381538a0 0x7fd8380c6730 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:57.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.176+0000 7fd84009e640 1 --2- 172.21.15.46:0/2347735518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8380cf160 0x7fd8380c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:57.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.176+0000 7fd84009e640 1 -- 172.21.15.46:0/2347735518 >> 172.21.15.46:0/2347735518 conn(0x7fd83806bfe0 msgr2=0x7fd83806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:57.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.176+0000 7fd84009e640 1 -- 172.21.15.46:0/2347735518 shutdown_connections 2023-10-30T14:04:57.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:57.176+0000 7fd84009e640 1 -- 172.21.15.46:0/2347735518 wait complete. 2023-10-30T14:04:57.200 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T14:04:59.201 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:04:59.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa85913b640 1 Processor -- start 2023-10-30T14:04:59.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa85913b640 1 -- start start 2023-10-30T14:04:59.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa85913b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8541538f0 0x7fa854159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:59.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa85913b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8540cefa0 0x7fa8540cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:59.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa85913b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8540ca6a0 con 0x7fa85415cd50 2023-10-30T14:04:59.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa85913b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa85415d300 con 0x7fa8541538f0 2023-10-30T14:04:59.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa85913b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa85415d5e0 con 0x7fa8540cefa0 2023-10-30T14:04:59.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa853fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8541538f0 0x7fa854159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:59.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa853fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8541538f0 0x7fa854159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50250/0 (socket says 172.21.15.46:50250) 2023-10-30T14:04:59.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa853fff640 1 -- 172.21.15.46:0/3403561875 learned_addr learned my addr 172.21.15.46:0/3403561875 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:59.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa85893a640 1 --2- 172.21.15.46:0/3403561875 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8540cefa0 0x7fa8540cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:59.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa853fff640 1 -- 172.21.15.46:0/3403561875 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8540cefa0 msgr2=0x7fa8540cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:04:59.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa853fff640 1 --2- 172.21.15.46:0/3403561875 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8540cefa0 0x7fa8540cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:59.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa853fff640 1 -- 172.21.15.46:0/3403561875 >> v1:172.21.15.154:6789/0 conn(0x7fa85415cd50 legacy=0x7fa8540c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:59.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.352+0000 7fa853fff640 1 -- 172.21.15.46:0/3403561875 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85415d8c0 con 0x7fa8541538f0 2023-10-30T14:04:59.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa853fff640 1 --2- 172.21.15.46:0/3403561875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8541538f0 0x7fa854159fb0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fa844009870 tx=0x7fa84401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff61de4e67104649 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:59.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa852ffd640 1 -- 172.21.15.46:0/3403561875 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8440a8a30 con 0x7fa8541538f0 2023-10-30T14:04:59.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa852ffd640 1 -- 172.21.15.46:0/3403561875 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8440a8b90 con 0x7fa8541538f0 2023-10-30T14:04:59.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa852ffd640 1 -- 172.21.15.46:0/3403561875 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8440b0440 con 0x7fa8541538f0 2023-10-30T14:04:59.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa85913b640 1 -- 172.21.15.46:0/3403561875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8541538f0 msgr2=0x7fa854159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:59.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa85913b640 1 --2- 172.21.15.46:0/3403561875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8541538f0 0x7fa854159fb0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fa844009870 tx=0x7fa84401ed30 comp rx=0 tx=0).stop 2023-10-30T14:04:59.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa85913b640 1 -- 172.21.15.46:0/3403561875 shutdown_connections 2023-10-30T14:04:59.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa85913b640 1 --2- 172.21.15.46:0/3403561875 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8540cefa0 0x7fa8540cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:59.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa85913b640 1 --2- 172.21.15.46:0/3403561875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8541538f0 0x7fa854159fb0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:59.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa85913b640 1 -- 172.21.15.46:0/3403561875 >> 172.21.15.46:0/3403561875 conn(0x7fa85406a640 msgr2=0x7fa854065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:59.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa85913b640 1 -- 172.21.15.46:0/3403561875 shutdown_connections 2023-10-30T14:04:59.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa85913b640 1 -- 172.21.15.46:0/3403561875 wait complete. 2023-10-30T14:04:59.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.356+0000 7fa85913b640 1 Processor -- start 2023-10-30T14:04:59.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa85913b640 1 -- start start 2023-10-30T14:04:59.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa85913b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8540cefa0 0x7fa8540ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:59.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa85913b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8541538f0 0x7fa8541eeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:59.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa85913b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa85415cd50 0x7fa854152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:59.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa85913b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa854148270 con 0x7fa8540cefa0 2023-10-30T14:04:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa85913b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85415a9c0 con 0x7fa85415cd50 2023-10-30T14:04:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa85913b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa854147f40 con 0x7fa8541538f0 2023-10-30T14:04:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa853fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8540cefa0 0x7fa8540ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa853fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8540cefa0 0x7fa8540ce1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50254/0 (socket says 172.21.15.46:50254) 2023-10-30T14:04:59.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa853fff640 1 -- 172.21.15.46:0/220852675 learned_addr learned my addr 172.21.15.46:0/220852675 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:04:59.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa8537fe640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8541538f0 0x7fa8541eeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:59.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa85893a640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa85415cd50 0x7fa854152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:59.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa853fff640 1 -- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8541538f0 msgr2=0x7fa8541eeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:59.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa853fff640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8541538f0 0x7fa8541eeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:59.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa853fff640 1 -- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa85415cd50 msgr2=0x7fa854152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:59.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa853fff640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa85415cd50 0x7fa854152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:59.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa853fff640 1 -- 172.21.15.46:0/220852675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8440094d0 con 0x7fa8540cefa0 2023-10-30T14:04:59.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa853fff640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8540cefa0 0x7fa8540ce1c0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fa84401d3e0 tx=0x7fa84401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:59.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa850ff9640 1 -- 172.21.15.46:0/220852675 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8440b0dd0 con 0x7fa8540cefa0 2023-10-30T14:04:59.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa850ff9640 1 -- 172.21.15.46:0/220852675 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8440bb460 con 0x7fa8540cefa0 2023-10-30T14:04:59.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa850ff9640 1 -- 172.21.15.46:0/220852675 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8440bb810 con 0x7fa8540cefa0 2023-10-30T14:04:59.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa85913b640 1 -- 172.21.15.46:0/220852675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85415ac30 con 0x7fa8540cefa0 2023-10-30T14:04:59.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa85913b640 1 -- 172.21.15.46:0/220852675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8540ce700 con 0x7fa8540cefa0 2023-10-30T14:04:59.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.360+0000 7fa85913b640 1 -- 172.21.15.46:0/220852675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85404e170 con 0x7fa8540cefa0 2023-10-30T14:04:59.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.364+0000 7fa850ff9640 1 -- 172.21.15.46:0/220852675 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa8440b83f0 con 0x7fa8540cefa0 2023-10-30T14:04:59.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.364+0000 7fa850ff9640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8400664a0 0x7fa840068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:04:59.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.364+0000 7fa850ff9640 1 -- 172.21.15.46:0/220852675 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fa844122060 con 0x7fa8540cefa0 2023-10-30T14:04:59.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.364+0000 7fa8537fe640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8400664a0 0x7fa840068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:04:59.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.364+0000 7fa8537fe640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8400664a0 0x7fa840068960 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa854147e30 tx=0x7fa848009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:04:59.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.368+0000 7fa850ff9640 1 -- 172.21.15.46:0/220852675 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa8440eec10 con 0x7fa8540cefa0 2023-10-30T14:04:59.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 -- 172.21.15.46:0/220852675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8540b2290 con 0x7fa8540cefa0 2023-10-30T14:04:59.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa850ff9640 1 -- 172.21.15.46:0/220852675 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+6522 (secure 0 0 0) 0x7fa8440c0090 con 0x7fa8540cefa0 2023-10-30T14:04:59.565 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:04:59.565 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:04:50.820260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:04:59.565 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 180 2023-10-30T14:04:59.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 -- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8400664a0 msgr2=0x7fa840068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:59.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8400664a0 0x7fa840068960 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa854147e30 tx=0x7fa848009040 comp rx=0 tx=0).stop 2023-10-30T14:04:59.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 -- 172.21.15.46:0/220852675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8540cefa0 msgr2=0x7fa8540ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:04:59.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8540cefa0 0x7fa8540ce1c0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fa84401d3e0 tx=0x7fa84401bad0 comp rx=0 tx=0).stop 2023-10-30T14:04:59.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 -- 172.21.15.46:0/220852675 shutdown_connections 2023-10-30T14:04:59.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8400664a0 0x7fa840068960 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:59.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa85415cd50 0x7fa854152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:59.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8541538f0 0x7fa8541eeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:59.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 --2- 172.21.15.46:0/220852675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8540cefa0 0x7fa8540ce1c0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:04:59.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 -- 172.21.15.46:0/220852675 >> 172.21.15.46:0/220852675 conn(0x7fa85406a640 msgr2=0x7fa854065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:04:59.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 -- 172.21.15.46:0/220852675 shutdown_connections 2023-10-30T14:04:59.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:04:59.560+0000 7fa85913b640 1 -- 172.21.15.46:0/220852675 wait complete. 2023-10-30T14:04:59.585 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T14:04:59.585 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 21 secs before thrashing 2023-10-30T14:05:00.458 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T14:05:00.456+0000 7fa87ff1c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:05:00.458 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T14:05:00.456+0000 7fa87ff1c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:05:03.319 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 32% (99/309) Updating files: 33% (102/309) Updating files: 34% (106/309) Updating files: 35% (109/309) Updating files: 36% (112/309) Updating files: 37% (115/309) Updating files: 38% (118/309) Updating files: 39% (121/309) Updating files: 40% (124/309) Updating files: 41% (127/309) Updating files: 42% (130/309) Updating files: 43% (133/309) Updating files: 44% (136/309) Updating files: 45% (140/309) Updating files: 46% (143/309) Updating files: 47% (146/309) Updating files: 48% (149/309) Updating files: 48% (150/309) Updating files: 49% (152/309) Updating files: 50% (155/309) Updating files: 50% (156/309) Updating files: 51% (158/309) Updating files: 52% (161/309) Updating files: 53% (164/309) Updating files: 54% (167/309) Updating files: 55% (170/309) Updating files: 56% (174/309) Updating files: 57% (177/309) Updating files: 58% (180/309) Updating files: 59% (183/309) Updating files: 60% (186/309) Updating files: 61% (189/309) Updating files: 62% (192/309) Updating files: 63% (195/309) Updating files: 64% (198/309) Updating files: 65% (201/309) Updating files: 66% (204/309) Updating files: 67% (208/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 71% (220/309) Updating files: 71% (221/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 74% (231/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 78% (242/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 84% (261/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 89% (276/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 99% (308/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2023-10-30T14:05:03.724 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 8de9501df27 0.81 2023-10-30T14:05:03.810 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.81 2023-10-30T14:05:03.866 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 82 -eq 48 ] 2023-10-30T14:05:03.867 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.82 2023-10-30T14:05:03.867 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.82 2023-10-30T14:05:03.867 INFO:tasks.workunit.client.0.smithi046.stdout:v0.82 2023-10-30T14:05:03.868 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.82 2023-10-30T14:05:18.248 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 14% (30/204) Updating files: 15% (31/204) Updating files: 16% (33/204) Updating files: 17% (35/204) Updating files: 18% (37/204) Updating files: 19% (39/204) Updating files: 20% (41/204) Updating files: 21% (43/204) Updating files: 22% (45/204) Updating files: 23% (47/204) Updating files: 24% (49/204) Updating files: 25% (51/204) Updating files: 26% (54/204) Updating files: 27% (56/204) Updating files: 28% (58/204) Updating files: 29% (60/204) Updating files: 30% (62/204) Updating files: 31% (64/204) Updating files: 32% (66/204) Updating files: 33% (68/204) Updating files: 34% (70/204) Updating files: 35% (72/204) Updating files: 36% (74/204) Updating files: 37% (76/204) Updating files: 38% (78/204) Updating files: 39% (80/204) Updating files: 40% (82/204) Updating files: 41% (84/204) Updating files: 42% (86/204) Updating files: 43% (88/204) Updating files: 44% (90/204) Updating files: 45% (92/204) Updating files: 46% (94/204) Updating files: 47% (96/204) Updating files: 48% (98/204) Updating files: 49% (100/204) Updating files: 50% (102/204) Updating files: 51% (105/204) Updating files: 52% (107/204) Updating files: 53% (109/204) Updating files: 54% (111/204) Updating files: 55% (113/204) Updating files: 56% (115/204) Updating files: 57% (117/204) Updating files: 58% (119/204) Updating files: 59% (121/204) Updating files: 60% (123/204) Updating files: 61% (125/204) Updating files: 62% (127/204) Updating files: 63% (129/204) Updating files: 64% (131/204) Updating files: 65% (133/204) Updating files: 66% (135/204) Updating files: 67% (137/204) Updating files: 68% (139/204) Updating files: 69% (141/204) Updating files: 70% (143/204) Updating files: 71% (145/204) Updating files: 72% (147/204) Updating files: 73% (149/204) Updating files: 74% (151/204) Updating files: 75% (153/204) Updating files: 76% (156/204) Updating files: 77% (158/204) Updating files: 78% (160/204) Updating files: 79% (162/204) Updating files: 80% (164/204) Updating files: 81% (166/204) Updating files: 82% (168/204) Updating files: 83% (170/204) Updating files: 84% (172/204) Updating files: 85% (174/204) Updating files: 86% (176/204) Updating files: 87% (178/204) Updating files: 88% (180/204) Updating files: 89% (182/204) Updating files: 89% (183/204) Updating files: 90% (184/204) Updating files: 91% (186/204) Updating files: 92% (188/204) Updating files: 93% (190/204) Updating files: 94% (192/204) Updating files: 95% (194/204) Updating files: 96% (196/204) Updating files: 97% (198/204) Updating files: 98% (200/204) Updating files: 99% (202/204) Updating files: 100% (204/204) Updating files: 100% (204/204), done. 2023-10-30T14:05:18.871 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 14085f42ddd 0.82 2023-10-30T14:05:18.878 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.82 2023-10-30T14:05:18.947 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 83 -eq 48 ] 2023-10-30T14:05:18.948 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.83 2023-10-30T14:05:18.948 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.83 2023-10-30T14:05:18.948 INFO:tasks.workunit.client.0.smithi046.stdout:v0.83 2023-10-30T14:05:18.949 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.83 2023-10-30T14:05:20.587 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:20.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.728+0000 7f48e413e640 1 Processor -- start 2023-10-30T14:05:20.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.728+0000 7f48e413e640 1 -- start start 2023-10-30T14:05:20.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.728+0000 7f48e413e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48dc15d6f0 0x7f48dc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:20.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.728+0000 7f48e413e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f48dc0cf160 0x7f48dc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:20.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.728+0000 7f48e413e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48dc0cb800 con 0x7f48dc1538a0 2023-10-30T14:05:20.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.728+0000 7f48e413e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48dc0cb330 con 0x7f48dc0cf160 2023-10-30T14:05:20.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.728+0000 7f48e413e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc0cad70 con 0x7f48dc15d6f0 2023-10-30T14:05:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.728+0000 7f48e26db640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f48dc0cf160 0x7f48dc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.728+0000 7f48e26db640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f48dc0cf160 0x7f48dc0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:56942/0 (socket says 172.21.15.46:56942) 2023-10-30T14:05:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.728+0000 7f48e26db640 1 -- 172.21.15.46:0/3334391771 learned_addr learned my addr 172.21.15.46:0/3334391771 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:20.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.728+0000 7f48e16d9640 1 --2- 172.21.15.46:0/3334391771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48dc15d6f0 0x7f48dc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:20.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e26db640 1 -- 172.21.15.46:0/3334391771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48dc15d6f0 msgr2=0x7f48dc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:20.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e26db640 1 --2- 172.21.15.46:0/3334391771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48dc15d6f0 0x7f48dc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:20.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e26db640 1 -- 172.21.15.46:0/3334391771 >> v1:172.21.15.154:6789/0 conn(0x7f48dc1538a0 legacy=0x7f48dc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:20.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e26db640 1 -- 172.21.15.46:0/3334391771 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc0cb050 con 0x7f48dc0cf160 2023-10-30T14:05:20.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e0ed8640 1 -- 172.21.15.46:0/3334391771 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152233756 0 0) 0x7f48dc0cb800 con 0x7f48dc1538a0 2023-10-30T14:05:20.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e26db640 1 --2- 172.21.15.46:0/3334391771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48dc0cf160 0x7f48dc0cf590 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009e50 tx=0x7f48d401f1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d04820f01db7bb82 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:20.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e0ed8640 1 -- 172.21.15.46:0/3334391771 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d40af400 con 0x7f48dc0cf160 2023-10-30T14:05:20.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e0ed8640 1 -- 172.21.15.46:0/3334391771 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d40ab050 con 0x7f48dc0cf160 2023-10-30T14:05:20.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e0ed8640 1 -- 172.21.15.46:0/3334391771 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d40af780 con 0x7f48dc0cf160 2023-10-30T14:05:20.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e413e640 1 -- 172.21.15.46:0/3334391771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48dc0cf160 msgr2=0x7f48dc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:20.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e413e640 1 --2- 172.21.15.46:0/3334391771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48dc0cf160 0x7f48dc0cf590 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009e50 tx=0x7f48d401f1f0 comp rx=0 tx=0).stop 2023-10-30T14:05:20.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e413e640 1 -- 172.21.15.46:0/3334391771 shutdown_connections 2023-10-30T14:05:20.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e413e640 1 --2- 172.21.15.46:0/3334391771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48dc0cf160 0x7f48dc0cf590 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:20.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e413e640 1 --2- 172.21.15.46:0/3334391771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48dc15d6f0 0x7f48dc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:20.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.732+0000 7f48e413e640 1 -- 172.21.15.46:0/3334391771 >> 172.21.15.46:0/3334391771 conn(0x7f48dc06bfe0 msgr2=0x7f48dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:20.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e413e640 1 -- 172.21.15.46:0/3334391771 shutdown_connections 2023-10-30T14:05:20.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e413e640 1 -- 172.21.15.46:0/3334391771 wait complete. 2023-10-30T14:05:20.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e413e640 1 Processor -- start 2023-10-30T14:05:20.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e413e640 1 -- start start 2023-10-30T14:05:20.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e413e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48dc0cf160 0x7f48dc1e5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:20.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e413e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48dc1538a0 0x7f48dc1e3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:20.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e413e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc15d6f0 0x7f48dc0c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:20.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e413e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc1e55c0 con 0x7f48dc15d6f0 2023-10-30T14:05:20.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e413e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc0b3ff0 con 0x7f48dc0cf160 2023-10-30T14:05:20.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e413e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc0b3700 con 0x7f48dc1538a0 2023-10-30T14:05:20.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e26db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc15d6f0 0x7f48dc0c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:20.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e26db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc15d6f0 0x7f48dc0c4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33926/0 (socket says 172.21.15.46:33926) 2023-10-30T14:05:20.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e26db640 1 -- 172.21.15.46:0/2971384183 learned_addr learned my addr 172.21.15.46:0/2971384183 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:20.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e16d9640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48dc1538a0 0x7f48dc1e3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:20.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e1eda640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48dc0cf160 0x7f48dc1e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:20.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e26db640 1 -- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48dc1538a0 msgr2=0x7f48dc1e3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:20.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e26db640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48dc1538a0 0x7f48dc1e3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:20.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e26db640 1 -- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48dc0cf160 msgr2=0x7f48dc1e5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:20.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e26db640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48dc0cf160 0x7f48dc1e5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:20.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e26db640 1 -- 172.21.15.46:0/2971384183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d4009b30 con 0x7f48dc15d6f0 2023-10-30T14:05:20.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e16d9640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48dc1538a0 0x7f48dc1e3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:20.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e1eda640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48dc0cf160 0x7f48dc1e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:20.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e26db640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc15d6f0 0x7f48dc0c4f50 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f48d401ce20 tx=0x7f48d401bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:20.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48d27fc640 1 -- 172.21.15.46:0/2971384183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d40afd00 con 0x7f48dc15d6f0 2023-10-30T14:05:20.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48d27fc640 1 -- 172.21.15.46:0/2971384183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d40afe60 con 0x7f48dc15d6f0 2023-10-30T14:05:20.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48e413e640 1 -- 172.21.15.46:0/2971384183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48dc0b5100 con 0x7f48dc15d6f0 2023-10-30T14:05:20.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.736+0000 7f48d27fc640 1 -- 172.21.15.46:0/2971384183 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4003ba0 con 0x7f48dc15d6f0 2023-10-30T14:05:20.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.740+0000 7f48e413e640 1 -- 172.21.15.46:0/2971384183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48dc0b39a0 con 0x7f48dc15d6f0 2023-10-30T14:05:20.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.740+0000 7f48e413e640 1 -- 172.21.15.46:0/2971384183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48dc0b2a50 con 0x7f48dc15d6f0 2023-10-30T14:05:20.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.740+0000 7f48d27fc640 1 -- 172.21.15.46:0/2971384183 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f48d40b6070 con 0x7f48dc15d6f0 2023-10-30T14:05:20.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.740+0000 7f48d27fc640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48c00664a0 0x7f48c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:20.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.740+0000 7f48d27fc640 1 -- 172.21.15.46:0/2971384183 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f48d4121c10 con 0x7f48dc15d6f0 2023-10-30T14:05:20.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.740+0000 7f48e1eda640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48c00664a0 0x7f48c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:20.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.744+0000 7f48e1eda640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48c00664a0 0x7f48c0068960 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f48d8000f70 tx=0x7f48d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:20.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.744+0000 7f48d27fc640 1 -- 172.21.15.46:0/2971384183 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f48d40ee7c0 con 0x7f48dc15d6f0 2023-10-30T14:05:20.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.936+0000 7f48e413e640 1 -- 172.21.15.46:0/2971384183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48dc0b41a0 con 0x7f48dc15d6f0 2023-10-30T14:05:20.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.936+0000 7f48d27fc640 1 -- 172.21.15.46:0/2971384183 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f48dc0b41a0 con 0x7f48dc15d6f0 2023-10-30T14:05:20.942 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:20.942 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:01.640919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:20.942 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 181 2023-10-30T14:05:20.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.940+0000 7f48e413e640 1 -- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48c00664a0 msgr2=0x7f48c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:20.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.940+0000 7f48e413e640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48c00664a0 0x7f48c0068960 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f48d8000f70 tx=0x7f48d8009040 comp rx=0 tx=0).stop 2023-10-30T14:05:20.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.940+0000 7f48e413e640 1 -- 172.21.15.46:0/2971384183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc15d6f0 msgr2=0x7f48dc0c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:20.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.940+0000 7f48e413e640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc15d6f0 0x7f48dc0c4f50 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f48d401ce20 tx=0x7f48d401bf90 comp rx=0 tx=0).stop 2023-10-30T14:05:20.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.940+0000 7f48e413e640 1 -- 172.21.15.46:0/2971384183 shutdown_connections 2023-10-30T14:05:20.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.940+0000 7f48e413e640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48c00664a0 0x7f48c0068960 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:20.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.940+0000 7f48e413e640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc15d6f0 0x7f48dc0c4f50 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:20.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.940+0000 7f48e413e640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48dc1538a0 0x7f48dc1e3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:20.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.940+0000 7f48e413e640 1 --2- 172.21.15.46:0/2971384183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48dc0cf160 0x7f48dc1e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:20.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.940+0000 7f48e413e640 1 -- 172.21.15.46:0/2971384183 >> 172.21.15.46:0/2971384183 conn(0x7f48dc06bfe0 msgr2=0x7f48dc0ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:20.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.940+0000 7f48e413e640 1 -- 172.21.15.46:0/2971384183 shutdown_connections 2023-10-30T14:05:20.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:20.940+0000 7f48e413e640 1 -- 172.21.15.46:0/2971384183 wait complete. 2023-10-30T14:05:20.962 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=1) 2023-10-30T14:05:20.963 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T14:05:20.963 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:05:20.997 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T14:05:20.997 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:21.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.180+0000 7f23bf59e640 1 Processor -- start 2023-10-30T14:05:21.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.180+0000 7f23bf59e640 1 -- start start 2023-10-30T14:05:21.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.180+0000 7f23bf59e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f23c0071b90 0x7f23c015ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:21.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.180+0000 7f23bf59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 0x7f23c0161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:21.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.180+0000 7f23bf59e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c015b060 con 0x7f23c00717b0 2023-10-30T14:05:21.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.180+0000 7f23bf59e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c0157530 con 0x7f23c0071b90 2023-10-30T14:05:21.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.180+0000 7f23bf59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c00c4ab0 con 0x7f23c00c6290 2023-10-30T14:05:21.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.180+0000 7f23bdd9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f23c0071b90 0x7f23c015ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:21.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.180+0000 7f23bed9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 0x7f23c0161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:21.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.180+0000 7f23bed9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 0x7f23c0161eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:34518/0 (socket says 172.21.15.46:34518) 2023-10-30T14:05:21.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.180+0000 7f23bed9d640 1 -- 172.21.15.46:0/1935032512 learned_addr learned my addr 172.21.15.46:0/1935032512 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:21.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.184+0000 7f23bed9d640 1 -- 172.21.15.46:0/1935032512 >> v2:172.21.15.154:3300/0 conn(0x7f23c0071b90 msgr2=0x7f23c015ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:21.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.184+0000 7f23bed9d640 1 --2- 172.21.15.46:0/1935032512 >> v2:172.21.15.154:3300/0 conn(0x7f23c0071b90 0x7f23c015ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:21.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.184+0000 7f23bed9d640 1 -- 172.21.15.46:0/1935032512 >> v1:172.21.15.154:6789/0 conn(0x7f23c00717b0 legacy=0x7f23c01513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:21.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.184+0000 7f23bed9d640 1 -- 172.21.15.46:0/1935032512 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c00c4d90 con 0x7f23c00c6290 2023-10-30T14:05:21.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.184+0000 7f23bdd9b640 1 --2- 172.21.15.46:0/1935032512 >> v2:172.21.15.154:3300/0 conn(0x7f23c0071b90 0x7f23c015ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:05:21.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.184+0000 7f23bed9d640 1 --2- 172.21.15.46:0/1935032512 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 0x7f23c0161eb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009e50 tx=0x7f23b801f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4553f89c02bc8a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:21.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.184+0000 7f23bd59a640 1 -- 172.21.15.46:0/1935032512 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b80af400 con 0x7f23c00c6290 2023-10-30T14:05:21.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.184+0000 7f23bd59a640 1 -- 172.21.15.46:0/1935032512 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b80ab050 con 0x7f23c00c6290 2023-10-30T14:05:21.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.184+0000 7f23bd59a640 1 -- 172.21.15.46:0/1935032512 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b80af780 con 0x7f23c00c6290 2023-10-30T14:05:21.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.184+0000 7f23bf59e640 1 -- 172.21.15.46:0/1935032512 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 msgr2=0x7f23c0161eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:21.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.184+0000 7f23bf59e640 1 --2- 172.21.15.46:0/1935032512 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 0x7f23c0161eb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f23b8009e50 tx=0x7f23b801f1f0 comp rx=0 tx=0).stop 2023-10-30T14:05:21.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 -- 172.21.15.46:0/1935032512 shutdown_connections 2023-10-30T14:05:21.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 --2- 172.21.15.46:0/1935032512 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 0x7f23c0161eb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:21.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 --2- 172.21.15.46:0/1935032512 >> v2:172.21.15.154:3300/0 conn(0x7f23c0071b90 0x7f23c015ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 -- 172.21.15.46:0/1935032512 >> 172.21.15.46:0/1935032512 conn(0x7f23c00c75d0 msgr2=0x7f23c00c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 -- 172.21.15.46:0/1935032512 shutdown_connections 2023-10-30T14:05:21.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 -- 172.21.15.46:0/1935032512 wait complete. 2023-10-30T14:05:21.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 Processor -- start 2023-10-30T14:05:21.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 -- start start 2023-10-30T14:05:21.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23c00717b0 0x7f23c0155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:21.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23c0071b90 0x7f23c01f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:21.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 0x7f23c014fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:21.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c0157370 con 0x7f23c0071b90 2023-10-30T14:05:21.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c0156520 con 0x7f23c00717b0 2023-10-30T14:05:21.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bf59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c0152c90 con 0x7f23c00c6290 2023-10-30T14:05:21.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23c0071b90 0x7f23c01f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:21.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23c0071b90 0x7f23c01f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33934/0 (socket says 172.21.15.46:33934) 2023-10-30T14:05:21.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bdd9b640 1 -- 172.21.15.46:0/3003013750 learned_addr learned my addr 172.21.15.46:0/3003013750 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:21.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23be59c640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23c00717b0 0x7f23c0155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:21.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bed9d640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 0x7f23c014fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bdd9b640 1 -- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 msgr2=0x7f23c014fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bdd9b640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 0x7f23c014fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:21.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bdd9b640 1 -- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23c00717b0 msgr2=0x7f23c0155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bdd9b640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23c00717b0 0x7f23c0155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bdd9b640 1 -- 172.21.15.46:0/3003013750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b4009420 con 0x7f23c0071b90 2023-10-30T14:05:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23be59c640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23c00717b0 0x7f23c0155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.188+0000 7f23bed9d640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 0x7f23c014fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.192+0000 7f23bdd9b640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23c0071b90 0x7f23c01f6b50 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f23b400e980 tx=0x7f23b400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:21.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.192+0000 7f23aeffd640 1 -- 172.21.15.46:0/3003013750 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b400ca00 con 0x7f23c0071b90 2023-10-30T14:05:21.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.192+0000 7f23aeffd640 1 -- 172.21.15.46:0/3003013750 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b400cb60 con 0x7f23c0071b90 2023-10-30T14:05:21.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.192+0000 7f23aeffd640 1 -- 172.21.15.46:0/3003013750 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4013470 con 0x7f23c0071b90 2023-10-30T14:05:21.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.192+0000 7f23bf59e640 1 -- 172.21.15.46:0/3003013750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b8009b30 con 0x7f23c0071b90 2023-10-30T14:05:21.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.192+0000 7f23bf59e640 1 -- 172.21.15.46:0/3003013750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c0157090 con 0x7f23c0071b90 2023-10-30T14:05:21.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.192+0000 7f23bf59e640 1 -- 172.21.15.46:0/3003013750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c004e170 con 0x7f23c0071b90 2023-10-30T14:05:21.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.192+0000 7f23aeffd640 1 -- 172.21.15.46:0/3003013750 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f23b40135d0 con 0x7f23c0071b90 2023-10-30T14:05:21.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.192+0000 7f23aeffd640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23a00664a0 0x7f23a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:21.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.192+0000 7f23aeffd640 1 -- 172.21.15.46:0/3003013750 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f23b4084970 con 0x7f23c0071b90 2023-10-30T14:05:21.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.192+0000 7f23be59c640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23a00664a0 0x7f23a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:21.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.196+0000 7f23be59c640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23a00664a0 0x7f23a0068960 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f23b0007fd0 tx=0x7f23b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:21.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.196+0000 7f23aeffd640 1 -- 172.21.15.46:0/3003013750 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23b40515b0 con 0x7f23c0071b90 2023-10-30T14:05:21.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.388+0000 7f23bf59e640 1 -- 172.21.15.46:0/3003013750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23c00c3410 con 0x7f23c0071b90 2023-10-30T14:05:21.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.388+0000 7f23aeffd640 1 -- 172.21.15.46:0/3003013750 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f23b4054bd0 con 0x7f23c0071b90 2023-10-30T14:05:21.394 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:21.394 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:01.640919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:21.394 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 181 2023-10-30T14:05:21.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.388+0000 7f23bf59e640 1 -- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23a00664a0 msgr2=0x7f23a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:21.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.388+0000 7f23bf59e640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23a00664a0 0x7f23a0068960 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f23b0007fd0 tx=0x7f23b0009040 comp rx=0 tx=0).stop 2023-10-30T14:05:21.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.388+0000 7f23bf59e640 1 -- 172.21.15.46:0/3003013750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23c0071b90 msgr2=0x7f23c01f6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:21.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.388+0000 7f23bf59e640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23c0071b90 0x7f23c01f6b50 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f23b400e980 tx=0x7f23b400ee40 comp rx=0 tx=0).stop 2023-10-30T14:05:21.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.392+0000 7f23bf59e640 1 -- 172.21.15.46:0/3003013750 shutdown_connections 2023-10-30T14:05:21.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.392+0000 7f23bf59e640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23a00664a0 0x7f23a0068960 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:21.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.392+0000 7f23bf59e640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23c00c6290 0x7f23c014fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:21.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.392+0000 7f23bf59e640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23c0071b90 0x7f23c01f6b50 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:21.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.392+0000 7f23bf59e640 1 --2- 172.21.15.46:0/3003013750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23c00717b0 0x7f23c0155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:21.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.392+0000 7f23bf59e640 1 -- 172.21.15.46:0/3003013750 >> 172.21.15.46:0/3003013750 conn(0x7f23c00c75d0 msgr2=0x7f23c00c9640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:21.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.392+0000 7f23bf59e640 1 -- 172.21.15.46:0/3003013750 shutdown_connections 2023-10-30T14:05:21.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:21.392+0000 7f23bf59e640 1 -- 172.21.15.46:0/3003013750 wait complete. 2023-10-30T14:05:21.415 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:05:23.416 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febba09a640 1 Processor -- start 2023-10-30T14:05:23.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febba09a640 1 -- start start 2023-10-30T14:05:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febba09a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febb41538a0 0x7febb415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febba09a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7febb40ca0e0 0x7febb415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:23.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febba09a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb40cb800 con 0x7febb40cf160 2023-10-30T14:05:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febba09a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb415dc30 con 0x7febb40ca0e0 2023-10-30T14:05:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febba09a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb40cb330 con 0x7febb41538a0 2023-10-30T14:05:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febb2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7febb40ca0e0 0x7febb415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febb2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7febb40ca0e0 0x7febb415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33942/0 (socket says 172.21.15.46:33942) 2023-10-30T14:05:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febb2ffd640 1 -- 172.21.15.46:0/2752215299 learned_addr learned my addr 172.21.15.46:0/2752215299 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febb37fe640 1 --2- 172.21.15.46:0/2752215299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febb41538a0 0x7febb415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febb2ffd640 1 -- 172.21.15.46:0/2752215299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febb41538a0 msgr2=0x7febb415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febb2ffd640 1 --2- 172.21.15.46:0/2752215299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febb41538a0 0x7febb415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febb2ffd640 1 -- 172.21.15.46:0/2752215299 >> v1:172.21.15.46:6789/0 conn(0x7febb40cf160 legacy=0x7febb40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:23.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febb2ffd640 1 -- 172.21.15.46:0/2752215299 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb40cad70 con 0x7febb40ca0e0 2023-10-30T14:05:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.567+0000 7febb37fe640 1 --2- 172.21.15.46:0/2752215299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febb41538a0 0x7febb415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:23.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febb2ffd640 1 --2- 172.21.15.46:0/2752215299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febb40ca0e0 0x7febb415a090 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7feba8009870 tx=0x7feba801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2244a6f4894806a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febb27fc640 1 -- 172.21.15.46:0/2752215299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba80a8a30 con 0x7febb40ca0e0 2023-10-30T14:05:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febb27fc640 1 -- 172.21.15.46:0/2752215299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba80a8b90 con 0x7febb40ca0e0 2023-10-30T14:05:23.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febb27fc640 1 -- 172.21.15.46:0/2752215299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba80b0440 con 0x7febb40ca0e0 2023-10-30T14:05:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febba09a640 1 -- 172.21.15.46:0/2752215299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febb40ca0e0 msgr2=0x7febb415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febba09a640 1 --2- 172.21.15.46:0/2752215299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febb40ca0e0 0x7febb415a090 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7feba8009870 tx=0x7feba801ed30 comp rx=0 tx=0).stop 2023-10-30T14:05:23.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febba09a640 1 -- 172.21.15.46:0/2752215299 shutdown_connections 2023-10-30T14:05:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febba09a640 1 --2- 172.21.15.46:0/2752215299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febb40ca0e0 0x7febb415a090 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febba09a640 1 --2- 172.21.15.46:0/2752215299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febb41538a0 0x7febb415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febba09a640 1 -- 172.21.15.46:0/2752215299 >> 172.21.15.46:0/2752215299 conn(0x7febb406bfe0 msgr2=0x7febb406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febba09a640 1 -- 172.21.15.46:0/2752215299 shutdown_connections 2023-10-30T14:05:23.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febba09a640 1 -- 172.21.15.46:0/2752215299 wait complete. 2023-10-30T14:05:23.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.571+0000 7febba09a640 1 Processor -- start 2023-10-30T14:05:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febba09a640 1 -- start start 2023-10-30T14:05:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febba09a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febb40ca0e0 0x7febb4147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febba09a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febb40cf160 0x7febb415c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febba09a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febb41538a0 0x7febb415ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:23.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febba09a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb41f0c50 con 0x7febb40cf160 2023-10-30T14:05:23.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febba09a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb41436f0 con 0x7febb41538a0 2023-10-30T14:05:23.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febba09a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb415acf0 con 0x7febb40ca0e0 2023-10-30T14:05:23.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febb2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febb40cf160 0x7febb415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:23.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febb2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febb40cf160 0x7febb415c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33952/0 (socket says 172.21.15.46:33952) 2023-10-30T14:05:23.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febb2ffd640 1 -- 172.21.15.46:0/1468300810 learned_addr learned my addr 172.21.15.46:0/1468300810 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febb37fe640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febb40ca0e0 0x7febb4147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febb3fff640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febb41538a0 0x7febb415ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febb2ffd640 1 -- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febb40ca0e0 msgr2=0x7febb4147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febb2ffd640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febb40ca0e0 0x7febb4147e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febb2ffd640 1 -- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febb41538a0 msgr2=0x7febb415ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:23.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febb2ffd640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febb41538a0 0x7febb415ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:23.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febb2ffd640 1 -- 172.21.15.46:0/1468300810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba80094d0 con 0x7febb40cf160 2023-10-30T14:05:23.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febb2ffd640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febb40cf160 0x7febb415c500 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7feba80b3040 tx=0x7feba801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:23.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7feb9bfff640 1 -- 172.21.15.46:0/1468300810 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba80be070 con 0x7febb40cf160 2023-10-30T14:05:23.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7feb9bfff640 1 -- 172.21.15.46:0/1468300810 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba80ba400 con 0x7febb40cf160 2023-10-30T14:05:23.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7feb9bfff640 1 -- 172.21.15.46:0/1468300810 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba80ba7d0 con 0x7febb40cf160 2023-10-30T14:05:23.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febba09a640 1 -- 172.21.15.46:0/1468300810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb415afd0 con 0x7febb40cf160 2023-10-30T14:05:23.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febba09a640 1 -- 172.21.15.46:0/1468300810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb415b590 con 0x7febb40cf160 2023-10-30T14:05:23.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.575+0000 7febba09a640 1 -- 172.21.15.46:0/1468300810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb40b2a50 con 0x7febb40cf160 2023-10-30T14:05:23.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.579+0000 7feb9bfff640 1 -- 172.21.15.46:0/1468300810 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feba80c3410 con 0x7febb40cf160 2023-10-30T14:05:23.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.579+0000 7feb9bfff640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feba00664a0 0x7feba0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:23.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.579+0000 7feb9bfff640 1 -- 172.21.15.46:0/1468300810 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7feba8122320 con 0x7febb40cf160 2023-10-30T14:05:23.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.579+0000 7febb37fe640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feba00664a0 0x7feba0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:23.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.579+0000 7febb37fe640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feba00664a0 0x7feba0068960 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7feba400ba10 tx=0x7feba4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:23.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.583+0000 7feb9bfff640 1 -- 172.21.15.46:0/1468300810 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feba80eef60 con 0x7febb40cf160 2023-10-30T14:05:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.771+0000 7febba09a640 1 -- 172.21.15.46:0/1468300810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febb41f0d90 con 0x7febb40cf160 2023-10-30T14:05:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7feb9bfff640 1 -- 172.21.15.46:0/1468300810 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+6523 (secure 0 0 0) 0x7febb41f0d90 con 0x7febb40cf160 2023-10-30T14:05:23.780 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:23.780 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:01.640919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:23.780 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 181 2023-10-30T14:05:23.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7febba09a640 1 -- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feba00664a0 msgr2=0x7feba0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:23.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7febba09a640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feba00664a0 0x7feba0068960 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7feba400ba10 tx=0x7feba4008040 comp rx=0 tx=0).stop 2023-10-30T14:05:23.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7febba09a640 1 -- 172.21.15.46:0/1468300810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febb40cf160 msgr2=0x7febb415c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:23.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7febba09a640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febb40cf160 0x7febb415c500 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7feba80b3040 tx=0x7feba801bad0 comp rx=0 tx=0).stop 2023-10-30T14:05:23.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7febba09a640 1 -- 172.21.15.46:0/1468300810 shutdown_connections 2023-10-30T14:05:23.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7febba09a640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feba00664a0 0x7feba0068960 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:23.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7febba09a640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febb41538a0 0x7febb415ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:23.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7febba09a640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febb40cf160 0x7febb415c500 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:23.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7febba09a640 1 --2- 172.21.15.46:0/1468300810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febb40ca0e0 0x7febb4147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:23.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7febba09a640 1 -- 172.21.15.46:0/1468300810 >> 172.21.15.46:0/1468300810 conn(0x7febb406bfe0 msgr2=0x7febb40cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:23.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7febba09a640 1 -- 172.21.15.46:0/1468300810 shutdown_connections 2023-10-30T14:05:23.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:23.775+0000 7febba09a640 1 -- 172.21.15.46:0/1468300810 wait complete. 2023-10-30T14:05:23.802 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:05:25.421 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:05:25.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:05:25.803 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a7e54640 1 Processor -- start 2023-10-30T14:05:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a7e54640 1 -- start start 2023-10-30T14:05:25.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a7e54640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f95a01580e0 0x7f95a0070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:25.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a7e54640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95a00c6540 0x7f95a015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:25.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a7e54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95a0069e90 0x7f95a006a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:25.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a7e54640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a00c5810 con 0x7f95a00c6540 2023-10-30T14:05:25.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a7e54640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a00c6990 con 0x7f95a01580e0 2023-10-30T14:05:25.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a7e54640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00c6bb0 con 0x7f95a0069e90 2023-10-30T14:05:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a53ef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95a00c6540 0x7f95a015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a53ef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95a00c6540 0x7f95a015e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33958/0 (socket says 172.21.15.46:33958) 2023-10-30T14:05:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a53ef640 1 -- 172.21.15.46:0/629772194 learned_addr learned my addr 172.21.15.46:0/629772194 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a63f1640 1 --2- 172.21.15.46:0/629772194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95a0069e90 0x7f95a006a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a5bf0640 1 --2- 172.21.15.46:0/629772194 >> v2:172.21.15.154:3300/0 conn(0x7f95a01580e0 0x7f95a0070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a53ef640 1 -- 172.21.15.46:0/629772194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95a0069e90 msgr2=0x7f95a006a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a53ef640 1 --2- 172.21.15.46:0/629772194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95a0069e90 0x7f95a006a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a53ef640 1 -- 172.21.15.46:0/629772194 >> v2:172.21.15.154:3300/0 conn(0x7f95a01580e0 msgr2=0x7f95a0070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a53ef640 1 --2- 172.21.15.46:0/629772194 >> v2:172.21.15.154:3300/0 conn(0x7f95a01580e0 0x7f95a0070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a53ef640 1 -- 172.21.15.46:0/629772194 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a00c6e70 con 0x7f95a00c6540 2023-10-30T14:05:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a5bf0640 1 --2- 172.21.15.46:0/629772194 >> v2:172.21.15.154:3300/0 conn(0x7f95a01580e0 0x7f95a0070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a53ef640 1 --2- 172.21.15.46:0/629772194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a00c6540 0x7f95a015e700 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f95a006eea0 tx=0x7f959001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd40177c808e4ecd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a4bee640 1 -- 172.21.15.46:0/629772194 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95900ac630 con 0x7f95a00c6540 2023-10-30T14:05:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a4bee640 1 -- 172.21.15.46:0/629772194 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95900ac790 con 0x7f95a00c6540 2023-10-30T14:05:25.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.955+0000 7f95a4bee640 1 -- 172.21.15.46:0/629772194 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95900acb20 con 0x7f95a00c6540 2023-10-30T14:05:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 -- 172.21.15.46:0/629772194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a00c6540 msgr2=0x7f95a015e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 --2- 172.21.15.46:0/629772194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a00c6540 0x7f95a015e700 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f95a006eea0 tx=0x7f959001c500 comp rx=0 tx=0).stop 2023-10-30T14:05:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 -- 172.21.15.46:0/629772194 shutdown_connections 2023-10-30T14:05:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 --2- 172.21.15.46:0/629772194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95a0069e90 0x7f95a006a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 --2- 172.21.15.46:0/629772194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a00c6540 0x7f95a015e700 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:25.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 --2- 172.21.15.46:0/629772194 >> v2:172.21.15.154:3300/0 conn(0x7f95a01580e0 0x7f95a0070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:25.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 -- 172.21.15.46:0/629772194 >> 172.21.15.46:0/629772194 conn(0x7f95a00c7910 msgr2=0x7f95a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:25.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 -- 172.21.15.46:0/629772194 shutdown_connections 2023-10-30T14:05:25.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 -- 172.21.15.46:0/629772194 wait complete. 2023-10-30T14:05:25.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 Processor -- start 2023-10-30T14:05:25.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 -- start start 2023-10-30T14:05:25.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95a0069e90 0x7f95a0147a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:25.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a00c6540 0x7f95a0143370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:25.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95a01580e0 0x7f95a01f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:25.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a015f2f0 con 0x7f95a00c6540 2023-10-30T14:05:25.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a0147fc0 con 0x7f95a0069e90 2023-10-30T14:05:25.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a7e54640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00c0050 con 0x7f95a01580e0 2023-10-30T14:05:25.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a53ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a00c6540 0x7f95a0143370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:25.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a53ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a00c6540 0x7f95a0143370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33962/0 (socket says 172.21.15.46:33962) 2023-10-30T14:05:25.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a53ef640 1 -- 172.21.15.46:0/2101030273 learned_addr learned my addr 172.21.15.46:0/2101030273 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:25.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a5bf0640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95a0069e90 0x7f95a0147a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:25.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.959+0000 7f95a63f1640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95a01580e0 0x7f95a01f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:25.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95a53ef640 1 -- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95a01580e0 msgr2=0x7f95a01f7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:25.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95a53ef640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95a01580e0 0x7f95a01f7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:25.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95a53ef640 1 -- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95a0069e90 msgr2=0x7f95a0147a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:25.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95a53ef640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95a0069e90 0x7f95a0147a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:25.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95a53ef640 1 -- 172.21.15.46:0/2101030273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95900094d0 con 0x7f95a00c6540 2023-10-30T14:05:25.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95a53ef640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a00c6540 0x7f95a0143370 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f959001abb0 tx=0x7f95900192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:25.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95967fc640 1 -- 172.21.15.46:0/2101030273 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95900b5640 con 0x7f95a00c6540 2023-10-30T14:05:25.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95967fc640 1 -- 172.21.15.46:0/2101030273 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95900b57a0 con 0x7f95a00c6540 2023-10-30T14:05:25.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95967fc640 1 -- 172.21.15.46:0/2101030273 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95900b5b30 con 0x7f95a00c6540 2023-10-30T14:05:25.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95a7e54640 1 -- 172.21.15.46:0/2101030273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a01438b0 con 0x7f95a00c6540 2023-10-30T14:05:25.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95a7e54640 1 -- 172.21.15.46:0/2101030273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a00c0c90 con 0x7f95a00c6540 2023-10-30T14:05:25.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95a7e54640 1 -- 172.21.15.46:0/2101030273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a004e170 con 0x7f95a00c6540 2023-10-30T14:05:25.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95967fc640 1 -- 172.21.15.46:0/2101030273 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f95900b3070 con 0x7f95a00c6540 2023-10-30T14:05:25.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95967fc640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f95840664f0 0x7f95840689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:25.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95967fc640 1 -- 172.21.15.46:0/2101030273 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f959011e510 con 0x7f95a00c6540 2023-10-30T14:05:25.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.963+0000 7f95a5bf0640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f95840664f0 0x7f95840689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:25.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.967+0000 7f95a5bf0640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f95840664f0 0x7f95840689b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f959c00ba10 tx=0x7f959c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:25.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:25.967+0000 7f95967fc640 1 -- 172.21.15.46:0/2101030273 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f95900eb380 con 0x7f95a00c6540 2023-10-30T14:05:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.159+0000 7f95a7e54640 1 -- 172.21.15.46:0/2101030273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95a00b0cd0 con 0x7f95a00c6540 2023-10-30T14:05:26.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.159+0000 7f95967fc640 1 -- 172.21.15.46:0/2101030273 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f95900ee9a0 con 0x7f95a00c6540 2023-10-30T14:05:26.166 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:26.166 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:01.640919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:26.166 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 181 2023-10-30T14:05:26.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.163+0000 7f95a7e54640 1 -- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f95840664f0 msgr2=0x7f95840689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:26.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.163+0000 7f95a7e54640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f95840664f0 0x7f95840689b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f959c00ba10 tx=0x7f959c008040 comp rx=0 tx=0).stop 2023-10-30T14:05:26.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.163+0000 7f95a7e54640 1 -- 172.21.15.46:0/2101030273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a00c6540 msgr2=0x7f95a0143370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:26.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.163+0000 7f95a7e54640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a00c6540 0x7f95a0143370 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f959001abb0 tx=0x7f95900192a0 comp rx=0 tx=0).stop 2023-10-30T14:05:26.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.163+0000 7f95a7e54640 1 -- 172.21.15.46:0/2101030273 shutdown_connections 2023-10-30T14:05:26.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.163+0000 7f95a7e54640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f95840664f0 0x7f95840689b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:26.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.163+0000 7f95a7e54640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95a01580e0 0x7f95a01f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:26.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.163+0000 7f95a7e54640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a00c6540 0x7f95a0143370 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:26.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.163+0000 7f95a7e54640 1 --2- 172.21.15.46:0/2101030273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95a0069e90 0x7f95a0147a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:26.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.163+0000 7f95a7e54640 1 -- 172.21.15.46:0/2101030273 >> 172.21.15.46:0/2101030273 conn(0x7f95a00c7910 msgr2=0x7f95a0152350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:26.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.163+0000 7f95a7e54640 1 -- 172.21.15.46:0/2101030273 shutdown_connections 2023-10-30T14:05:26.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:26.163+0000 7f95a7e54640 1 -- 172.21.15.46:0/2101030273 wait complete. 2023-10-30T14:05:26.188 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:05:28.189 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:28.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.339+0000 7f5f27765640 1 Processor -- start 2023-10-30T14:05:28.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.339+0000 7f5f27765640 1 -- start start 2023-10-30T14:05:28.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f27765640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5f2006d290 0x7f5f2006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:28.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f27765640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f200c6c10 con 0x7f5f2015a410 2023-10-30T14:05:28.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f27765640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f200c6ef0 con 0x7f5f20069e90 2023-10-30T14:05:28.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f27765640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f200c8570 con 0x7f5f2006d290 2023-10-30T14:05:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f24d00640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5f2015a410 0x7f5f2006cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50872/0 (socket says 172.21.15.46:50872) 2023-10-30T14:05:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f24d00640 1 -- 172.21.15.46:0/498764554 learned_addr learned my addr 172.21.15.46:0/498764554 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:05:28.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f25d02640 1 --2- 172.21.15.46:0/498764554 >> v2:172.21.15.154:3300/0 conn(0x7f5f2006d290 0x7f5f2006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:28.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350937933 0 0) 0x7f5f200c6c10 con 0x7f5f2015a410 2023-10-30T14:05:28.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f0c003530 con 0x7f5f2015a410 2023-10-30T14:05:28.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839968649 0 0) 0x7f5f200c6ef0 con 0x7f5f20069e90 2023-10-30T14:05:28.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f200c6c10 con 0x7f5f20069e90 2023-10-30T14:05:28.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2552382348 0 0) 0x7f5f0c003530 con 0x7f5f2015a410 2023-10-30T14:05:28.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 >> v2:172.21.15.154:3300/0 conn(0x7f5f2006d290 msgr2=0x7f5f2006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:28.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 --2- 172.21.15.46:0/498764554 >> v2:172.21.15.154:3300/0 conn(0x7f5f2006d290 0x7f5f2006a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:28.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 >> v1:172.21.15.154:6789/0 conn(0x7f5f20069e90 legacy=0x7f5f200c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:28.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f200c71d0 con 0x7f5f2015a410 2023-10-30T14:05:28.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f5f10001680 con 0x7f5f2015a410 2023-10-30T14:05:28.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 >> v1:172.21.15.46:6789/0 conn(0x7f5f2015a410 legacy=0x7f5f2006cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:28.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 --2- 172.21.15.46:0/498764554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f0c003b60 0x7f5f0c003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:28.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 --2- 172.21.15.46:0/498764554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0c0047d0 0x7f5f0c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:28.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 --2- 172.21.15.46:0/498764554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f0c005190 0x7f5f0c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:28.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f200c8570 con 0x7f5f0c005190 2023-10-30T14:05:28.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c00c060 con 0x7f5f0c0047d0 2023-10-30T14:05:28.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c00c220 con 0x7f5f0c003b60 2023-10-30T14:05:28.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f25d02640 1 --2- 172.21.15.46:0/498764554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f0c005190 0x7f5f0c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:28.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f24d00640 1 --2- 172.21.15.46:0/498764554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0c0047d0 0x7f5f0c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:28.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f25501640 1 --2- 172.21.15.46:0/498764554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f0c003b60 0x7f5f0c003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:28.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f25d02640 1 -- 172.21.15.46:0/498764554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f0c003b60 msgr2=0x7f5f0c003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:28.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f25d02640 1 --2- 172.21.15.46:0/498764554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f0c003b60 0x7f5f0c003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:28.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f25d02640 1 -- 172.21.15.46:0/498764554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0c0047d0 msgr2=0x7f5f0c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:28.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f25d02640 1 --2- 172.21.15.46:0/498764554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0c0047d0 0x7f5f0c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:28.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.343+0000 7f5f25d02640 1 -- 172.21.15.46:0/498764554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c003530 con 0x7f5f0c005190 2023-10-30T14:05:28.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.347+0000 7f5f25d02640 1 -- 172.21.15.46:0/498764554 reap_dead start 2023-10-30T14:05:28.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.347+0000 7f5f25d02640 1 --2- 172.21.15.46:0/498764554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f0c005190 0x7f5f0c00b990 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f5f18009fc0 tx=0x7f5f1801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b386adbf4fd9fd32 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:28.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.347+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f18003ba0 con 0x7f5f0c005190 2023-10-30T14:05:28.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.347+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1800f040 con 0x7f5f0c005190 2023-10-30T14:05:28.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.347+0000 7f5f17fff640 1 -- 172.21.15.46:0/498764554 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f180a59a0 con 0x7f5f0c005190 2023-10-30T14:05:28.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.347+0000 7f5f27765640 1 -- 172.21.15.46:0/498764554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f0c005190 msgr2=0x7f5f0c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:28.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.347+0000 7f5f27765640 1 --2- 172.21.15.46:0/498764554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f0c005190 0x7f5f0c00b990 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f5f18009fc0 tx=0x7f5f1801cca0 comp rx=0 tx=0).stop 2023-10-30T14:05:28.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.347+0000 7f5f27765640 1 -- 172.21.15.46:0/498764554 shutdown_connections 2023-10-30T14:05:28.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.347+0000 7f5f27765640 1 --2- 172.21.15.46:0/498764554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f0c005190 0x7f5f0c00b990 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:28.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.347+0000 7f5f27765640 1 -- 172.21.15.46:0/498764554 >> 172.21.15.46:0/498764554 conn(0x7f5f2006dfd0 msgr2=0x7f5f2006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:28.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.347+0000 7f5f27765640 1 -- 172.21.15.46:0/498764554 shutdown_connections 2023-10-30T14:05:28.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.347+0000 7f5f27765640 1 -- 172.21.15.46:0/498764554 wait complete. 2023-10-30T14:05:28.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f27765640 1 Processor -- start 2023-10-30T14:05:28.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f27765640 1 -- start start 2023-10-30T14:05:28.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f27765640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0c005190 0x7f5f201f2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:28.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f27765640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f20156df0 0x7f5f20158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:28.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f27765640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f2015a9c0 0x7f5f20155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:28.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f27765640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f201f2ec0 con 0x7f5f2015a9c0 2023-10-30T14:05:28.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f27765640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f201ef440 con 0x7f5f0c005190 2023-10-30T14:05:28.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f27765640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f200c3170 con 0x7f5f20156df0 2023-10-30T14:05:28.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f25d02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f2015a9c0 0x7f5f20155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:28.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f25501640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0c005190 0x7f5f201f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:28.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f25d02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f2015a9c0 0x7f5f20155140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52194/0 (socket says 172.21.15.46:52194) 2023-10-30T14:05:28.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f25d02640 1 -- 172.21.15.46:0/4179341495 learned_addr learned my addr 172.21.15.46:0/4179341495 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:28.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f24d00640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f20156df0 0x7f5f20158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:28.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f25d02640 1 -- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f20156df0 msgr2=0x7f5f20158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:28.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f25d02640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f20156df0 0x7f5f20158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:28.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f25d02640 1 -- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0c005190 msgr2=0x7f5f201f2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:28.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f25d02640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0c005190 0x7f5f201f2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:28.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f25d02640 1 -- 172.21.15.46:0/4179341495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2015af20 con 0x7f5f2015a9c0 2023-10-30T14:05:28.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f25d02640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f2015a9c0 0x7f5f20155140 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f5f180986c0 tx=0x7f5f18009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:28.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f15ffb640 1 -- 172.21.15.46:0/4179341495 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f180b8670 con 0x7f5f2015a9c0 2023-10-30T14:05:28.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f15ffb640 1 -- 172.21.15.46:0/4179341495 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f180b87d0 con 0x7f5f2015a9c0 2023-10-30T14:05:28.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f27765640 1 -- 172.21.15.46:0/4179341495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2015b200 con 0x7f5f2015a9c0 2023-10-30T14:05:28.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f15ffb640 1 -- 172.21.15.46:0/4179341495 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f180b8b60 con 0x7f5f2015a9c0 2023-10-30T14:05:28.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.351+0000 7f5f27765640 1 -- 172.21.15.46:0/4179341495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f201eedf0 con 0x7f5f2015a9c0 2023-10-30T14:05:28.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.355+0000 7f5f27765640 1 -- 172.21.15.46:0/4179341495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2004e170 con 0x7f5f2015a9c0 2023-10-30T14:05:28.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.355+0000 7f5f15ffb640 1 -- 172.21.15.46:0/4179341495 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5f180b8cc0 con 0x7f5f2015a9c0 2023-10-30T14:05:28.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.355+0000 7f5f15ffb640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f040664d0 0x7f5f04068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:28.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.355+0000 7f5f15ffb640 1 -- 172.21.15.46:0/4179341495 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f5f1811fa70 con 0x7f5f2015a9c0 2023-10-30T14:05:28.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.355+0000 7f5f25501640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f040664d0 0x7f5f04068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:28.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.355+0000 7f5f25501640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f040664d0 0x7f5f04068990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c000c10 tx=0x7f5f1c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:28.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.359+0000 7f5f15ffb640 1 -- 172.21.15.46:0/4179341495 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5f180ec920 con 0x7f5f2015a9c0 2023-10-30T14:05:28.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.551+0000 7f5f27765640 1 -- 172.21.15.46:0/4179341495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f201ef580 con 0x7f5f2015a9c0 2023-10-30T14:05:28.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.551+0000 7f5f15ffb640 1 -- 172.21.15.46:0/4179341495 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f5f180eff40 con 0x7f5f2015a9c0 2023-10-30T14:05:28.554 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:28.554 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:01.640919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:28.555 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 181 2023-10-30T14:05:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.555+0000 7f5f27765640 1 -- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f040664d0 msgr2=0x7f5f04068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.555+0000 7f5f27765640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f040664d0 0x7f5f04068990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c000c10 tx=0x7f5f1c009040 comp rx=0 tx=0).stop 2023-10-30T14:05:28.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.555+0000 7f5f27765640 1 -- 172.21.15.46:0/4179341495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f2015a9c0 msgr2=0x7f5f20155140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:28.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.555+0000 7f5f27765640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f2015a9c0 0x7f5f20155140 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f5f180986c0 tx=0x7f5f18009750 comp rx=0 tx=0).stop 2023-10-30T14:05:28.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.555+0000 7f5f27765640 1 -- 172.21.15.46:0/4179341495 shutdown_connections 2023-10-30T14:05:28.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.555+0000 7f5f27765640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5f040664d0 0x7f5f04068990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:28.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.555+0000 7f5f27765640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f2015a9c0 0x7f5f20155140 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:28.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.555+0000 7f5f27765640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f20156df0 0x7f5f20158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:28.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.555+0000 7f5f27765640 1 --2- 172.21.15.46:0/4179341495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0c005190 0x7f5f201f2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:28.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.555+0000 7f5f27765640 1 -- 172.21.15.46:0/4179341495 >> 172.21.15.46:0/4179341495 conn(0x7f5f2006dfd0 msgr2=0x7f5f200c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:28.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.555+0000 7f5f27765640 1 -- 172.21.15.46:0/4179341495 shutdown_connections 2023-10-30T14:05:28.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:28.555+0000 7f5f27765640 1 -- 172.21.15.46:0/4179341495 wait complete. 2023-10-30T14:05:28.575 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:05:30.576 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:30.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3e332640 1 Processor -- start 2023-10-30T14:05:30.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3e332640 1 -- start start 2023-10-30T14:05:30.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3e332640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6c381538f0 0x7f6c38159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:30.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3e332640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c380cefa0 0x7f6c380cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:30.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3e332640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c380ca6a0 con 0x7f6c3815cd50 2023-10-30T14:05:30.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3e332640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3815d300 con 0x7f6c381538f0 2023-10-30T14:05:30.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3e332640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3815d5e0 con 0x7f6c380cefa0 2023-10-30T14:05:30.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3d330640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6c381538f0 0x7f6c38159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:30.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3d330640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6c381538f0 0x7f6c38159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52210/0 (socket says 172.21.15.46:52210) 2023-10-30T14:05:30.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3d330640 1 -- 172.21.15.46:0/1598441022 learned_addr learned my addr 172.21.15.46:0/1598441022 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:30.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3db31640 1 --2- 172.21.15.46:0/1598441022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c380cefa0 0x7f6c380cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:30.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3d330640 1 -- 172.21.15.46:0/1598441022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c380cefa0 msgr2=0x7f6c380cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:30.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3d330640 1 --2- 172.21.15.46:0/1598441022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c380cefa0 0x7f6c380cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:30.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3d330640 1 -- 172.21.15.46:0/1598441022 >> v1:172.21.15.46:6789/0 conn(0x7f6c3815cd50 legacy=0x7f6c380c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:30.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3d330640 1 -- 172.21.15.46:0/1598441022 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3815d8c0 con 0x7f6c381538f0 2023-10-30T14:05:30.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.727+0000 7f6c3db31640 1 --2- 172.21.15.46:0/1598441022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c380cefa0 0x7f6c380cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:30.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c3d330640 1 --2- 172.21.15.46:0/1598441022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c381538f0 0x7f6c38159fb0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009870 tx=0x7f6c3401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1285ede3e39b8ab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:30.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c2ffff640 1 -- 172.21.15.46:0/1598441022 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c340a8a30 con 0x7f6c381538f0 2023-10-30T14:05:30.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c2ffff640 1 -- 172.21.15.46:0/1598441022 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c340a8b90 con 0x7f6c381538f0 2023-10-30T14:05:30.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c2ffff640 1 -- 172.21.15.46:0/1598441022 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c340b0440 con 0x7f6c381538f0 2023-10-30T14:05:30.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c3e332640 1 -- 172.21.15.46:0/1598441022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c381538f0 msgr2=0x7f6c38159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:30.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c3e332640 1 --2- 172.21.15.46:0/1598441022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c381538f0 0x7f6c38159fb0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009870 tx=0x7f6c3401ed30 comp rx=0 tx=0).stop 2023-10-30T14:05:30.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c3e332640 1 -- 172.21.15.46:0/1598441022 shutdown_connections 2023-10-30T14:05:30.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c3e332640 1 --2- 172.21.15.46:0/1598441022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c380cefa0 0x7f6c380cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:30.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c3e332640 1 --2- 172.21.15.46:0/1598441022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c381538f0 0x7f6c38159fb0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:30.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c3e332640 1 -- 172.21.15.46:0/1598441022 >> 172.21.15.46:0/1598441022 conn(0x7f6c3806a640 msgr2=0x7f6c38065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:30.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c3e332640 1 -- 172.21.15.46:0/1598441022 shutdown_connections 2023-10-30T14:05:30.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c3e332640 1 -- 172.21.15.46:0/1598441022 wait complete. 2023-10-30T14:05:30.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.731+0000 7f6c3e332640 1 Processor -- start 2023-10-30T14:05:30.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3e332640 1 -- start start 2023-10-30T14:05:30.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3e332640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c380cefa0 0x7f6c380b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:30.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3e332640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c381538f0 0x7f6c3815bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:30.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3e332640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3815cd50 0x7f6c3815c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:30.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3e332640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c38142a80 con 0x7f6c380cefa0 2023-10-30T14:05:30.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3e332640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c380b4740 con 0x7f6c3815cd50 2023-10-30T14:05:30.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3e332640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c38141e50 con 0x7f6c381538f0 2023-10-30T14:05:30.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3d330640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c380cefa0 0x7f6c380b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:30.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3d330640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c380cefa0 0x7f6c380b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52218/0 (socket says 172.21.15.46:52218) 2023-10-30T14:05:30.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3d330640 1 -- 172.21.15.46:0/1844522440 learned_addr learned my addr 172.21.15.46:0/1844522440 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:30.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3cb2f640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c381538f0 0x7f6c3815bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:30.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3db31640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3815cd50 0x7f6c3815c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:30.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3d330640 1 -- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c381538f0 msgr2=0x7f6c3815bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:30.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3d330640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c381538f0 0x7f6c3815bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:30.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3d330640 1 -- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3815cd50 msgr2=0x7f6c3815c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:30.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3d330640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3815cd50 0x7f6c3815c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:30.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3d330640 1 -- 172.21.15.46:0/1844522440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c340094d0 con 0x7f6c380cefa0 2023-10-30T14:05:30.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3d330640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c380cefa0 0x7f6c380b4b00 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f6c340b3040 tx=0x7f6c3401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:30.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c2dffb640 1 -- 172.21.15.46:0/1844522440 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c340be070 con 0x7f6c380cefa0 2023-10-30T14:05:30.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c2dffb640 1 -- 172.21.15.46:0/1844522440 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c340ba400 con 0x7f6c380cefa0 2023-10-30T14:05:30.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c2dffb640 1 -- 172.21.15.46:0/1844522440 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c340ba7d0 con 0x7f6c380cefa0 2023-10-30T14:05:30.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3e332640 1 -- 172.21.15.46:0/1844522440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c38142110 con 0x7f6c380cefa0 2023-10-30T14:05:30.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3e332640 1 -- 172.21.15.46:0/1844522440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c381426a0 con 0x7f6c380cefa0 2023-10-30T14:05:30.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.735+0000 7f6c3e332640 1 -- 172.21.15.46:0/1844522440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3804e170 con 0x7f6c380cefa0 2023-10-30T14:05:30.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.739+0000 7f6c2dffb640 1 -- 172.21.15.46:0/1844522440 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6c340c3410 con 0x7f6c380cefa0 2023-10-30T14:05:30.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.739+0000 7f6c2dffb640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6c1c0664a0 0x7f6c1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:30.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.739+0000 7f6c2dffb640 1 -- 172.21.15.46:0/1844522440 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f6c34122230 con 0x7f6c380cefa0 2023-10-30T14:05:30.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.739+0000 7f6c3cb2f640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6c1c0664a0 0x7f6c1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:30.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.739+0000 7f6c3cb2f640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6c1c0664a0 0x7f6c1c068960 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6c38146d40 tx=0x7f6c28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:30.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.743+0000 7f6c2dffb640 1 -- 172.21.15.46:0/1844522440 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6c340eef00 con 0x7f6c380cefa0 2023-10-30T14:05:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.947+0000 7f6c3e332640 1 -- 172.21.15.46:0/1844522440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c380b2290 con 0x7f6c380cefa0 2023-10-30T14:05:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.947+0000 7f6c2dffb640 1 -- 172.21.15.46:0/1844522440 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f6c340f2520 con 0x7f6c380cefa0 2023-10-30T14:05:30.950 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:30.950 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:01.640919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 181 2023-10-30T14:05:30.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.947+0000 7f6c3e332640 1 -- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6c1c0664a0 msgr2=0x7f6c1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.947+0000 7f6c3e332640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6c1c0664a0 0x7f6c1c068960 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6c38146d40 tx=0x7f6c28009040 comp rx=0 tx=0).stop 2023-10-30T14:05:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.947+0000 7f6c3e332640 1 -- 172.21.15.46:0/1844522440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c380cefa0 msgr2=0x7f6c380b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.951+0000 7f6c3e332640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c380cefa0 0x7f6c380b4b00 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f6c340b3040 tx=0x7f6c3401bad0 comp rx=0 tx=0).stop 2023-10-30T14:05:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.951+0000 7f6c3e332640 1 -- 172.21.15.46:0/1844522440 shutdown_connections 2023-10-30T14:05:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.951+0000 7f6c3e332640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6c1c0664a0 0x7f6c1c068960 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:30.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.951+0000 7f6c3e332640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3815cd50 0x7f6c3815c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:30.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.951+0000 7f6c3e332640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c381538f0 0x7f6c3815bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:30.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.951+0000 7f6c3e332640 1 --2- 172.21.15.46:0/1844522440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c380cefa0 0x7f6c380b4b00 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:30.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.951+0000 7f6c3e332640 1 -- 172.21.15.46:0/1844522440 >> 172.21.15.46:0/1844522440 conn(0x7f6c3806a640 msgr2=0x7f6c380c9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:30.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.951+0000 7f6c3e332640 1 -- 172.21.15.46:0/1844522440 shutdown_connections 2023-10-30T14:05:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:30.951+0000 7f6c3e332640 1 -- 172.21.15.46:0/1844522440 wait complete. 2023-10-30T14:05:30.973 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:05:32.974 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:33.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.127+0000 7f289baa9640 1 Processor -- start 2023-10-30T14:05:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.127+0000 7f289baa9640 1 -- start start 2023-10-30T14:05:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.127+0000 7f289baa9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f28941580e0 0x7f2894070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.127+0000 7f289baa9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28940c6540 0x7f289415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.127+0000 7f289baa9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28940c5810 con 0x7f2894069e90 2023-10-30T14:05:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.127+0000 7f289baa9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28940c6990 con 0x7f28940c6540 2023-10-30T14:05:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.127+0000 7f289baa9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28940c6bb0 con 0x7f28941580e0 2023-10-30T14:05:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.127+0000 7f2899044640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28940c6540 0x7f289415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.131+0000 7f2899044640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28940c6540 0x7f289415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52234/0 (socket says 172.21.15.46:52234) 2023-10-30T14:05:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.131+0000 7f2899044640 1 -- 172.21.15.46:0/283113562 learned_addr learned my addr 172.21.15.46:0/283113562 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.131+0000 7f2899845640 1 --2- 172.21.15.46:0/283113562 >> v2:172.21.15.154:3300/0 conn(0x7f28941580e0 0x7f2894070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:33.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.131+0000 7f2899044640 1 -- 172.21.15.46:0/283113562 >> v2:172.21.15.154:3300/0 conn(0x7f28941580e0 msgr2=0x7f2894070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:33.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.131+0000 7f2899044640 1 --2- 172.21.15.46:0/283113562 >> v2:172.21.15.154:3300/0 conn(0x7f28941580e0 0x7f2894070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:33.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.131+0000 7f2899044640 1 -- 172.21.15.46:0/283113562 >> v1:172.21.15.46:6789/0 conn(0x7f2894069e90 legacy=0x7f289406a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.131+0000 7f2899044640 1 -- 172.21.15.46:0/283113562 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28940c6e70 con 0x7f28940c6540 2023-10-30T14:05:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.131+0000 7f2899044640 1 --2- 172.21.15.46:0/283113562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28940c6540 0x7f289415e700 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f289406eea0 tx=0x7f288401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f82e2f0f61caef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.131+0000 7f2898843640 1 -- 172.21.15.46:0/283113562 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28840aa660 con 0x7f28940c6540 2023-10-30T14:05:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.131+0000 7f2898843640 1 -- 172.21.15.46:0/283113562 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28840aa7c0 con 0x7f28940c6540 2023-10-30T14:05:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.131+0000 7f2898843640 1 -- 172.21.15.46:0/283113562 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28840aab50 con 0x7f28940c6540 2023-10-30T14:05:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 -- 172.21.15.46:0/283113562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28940c6540 msgr2=0x7f289415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 --2- 172.21.15.46:0/283113562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28940c6540 0x7f289415e700 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f289406eea0 tx=0x7f288401c500 comp rx=0 tx=0).stop 2023-10-30T14:05:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 -- 172.21.15.46:0/283113562 shutdown_connections 2023-10-30T14:05:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 --2- 172.21.15.46:0/283113562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28940c6540 0x7f289415e700 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 --2- 172.21.15.46:0/283113562 >> v2:172.21.15.154:3300/0 conn(0x7f28941580e0 0x7f2894070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 -- 172.21.15.46:0/283113562 >> 172.21.15.46:0/283113562 conn(0x7f28940c7910 msgr2=0x7f28940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 -- 172.21.15.46:0/283113562 shutdown_connections 2023-10-30T14:05:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 -- 172.21.15.46:0/283113562 wait complete. 2023-10-30T14:05:33.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 Processor -- start 2023-10-30T14:05:33.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 -- start start 2023-10-30T14:05:33.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2894069e90 0x7f2894148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:33.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28940c6540 0x7f2894146220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:33.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28941580e0 0x7f28940c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:33.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28941f0bb0 con 0x7f2894069e90 2023-10-30T14:05:33.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2894145e00 con 0x7f28940c6540 2023-10-30T14:05:33.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289baa9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28940ca4e0 con 0x7f28941580e0 2023-10-30T14:05:33.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f2899845640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2894069e90 0x7f2894148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:33.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f2899845640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2894069e90 0x7f2894148b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52238/0 (socket says 172.21.15.46:52238) 2023-10-30T14:05:33.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f2899845640 1 -- 172.21.15.46:0/976604497 learned_addr learned my addr 172.21.15.46:0/976604497 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:33.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f2899044640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28940c6540 0x7f2894146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:33.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289a046640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28941580e0 0x7f28940c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:33.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f2899845640 1 -- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28941580e0 msgr2=0x7f28940c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:33.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f2899845640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28941580e0 0x7f28940c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:33.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f2899845640 1 -- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28940c6540 msgr2=0x7f2894146220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:33.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f2899845640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28940c6540 0x7f2894146220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:33.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f2899845640 1 -- 172.21.15.46:0/976604497 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28840094d0 con 0x7f2894069e90 2023-10-30T14:05:33.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f289a046640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28941580e0 0x7f28940c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:33.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.135+0000 7f2899044640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28940c6540 0x7f2894146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:33.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.139+0000 7f2899845640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2894069e90 0x7f2894148b70 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f289000d5f0 tx=0x7f289000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:33.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.139+0000 7f288a7fc640 1 -- 172.21.15.46:0/976604497 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28900125f0 con 0x7f2894069e90 2023-10-30T14:05:33.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.139+0000 7f288a7fc640 1 -- 172.21.15.46:0/976604497 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2890012750 con 0x7f2894069e90 2023-10-30T14:05:33.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.139+0000 7f288a7fc640 1 -- 172.21.15.46:0/976604497 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2890012ae0 con 0x7f2894069e90 2023-10-30T14:05:33.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.139+0000 7f289baa9640 1 -- 172.21.15.46:0/976604497 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28941f08d0 con 0x7f2894069e90 2023-10-30T14:05:33.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.139+0000 7f289baa9640 1 -- 172.21.15.46:0/976604497 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2894145740 con 0x7f2894069e90 2023-10-30T14:05:33.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.139+0000 7f289baa9640 1 -- 172.21.15.46:0/976604497 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2894146070 con 0x7f2894069e90 2023-10-30T14:05:33.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.139+0000 7f288a7fc640 1 -- 172.21.15.46:0/976604497 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2890012c40 con 0x7f2894069e90 2023-10-30T14:05:33.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.139+0000 7f288a7fc640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28780664a0 0x7f2878068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:33.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.139+0000 7f288a7fc640 1 -- 172.21.15.46:0/976604497 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f2890084bb0 con 0x7f2894069e90 2023-10-30T14:05:33.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.139+0000 7f2899044640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28780664a0 0x7f2878068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:33.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.143+0000 7f2899044640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28780664a0 0x7f2878068960 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f28840ae000 tx=0x7f28840a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:33.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.143+0000 7f288a7fc640 1 -- 172.21.15.46:0/976604497 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2890051760 con 0x7f2894069e90 2023-10-30T14:05:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 -- 172.21.15.46:0/976604497 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f289404e170 con 0x7f2894069e90 2023-10-30T14:05:33.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f288a7fc640 1 -- 172.21.15.46:0/976604497 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f2890007380 con 0x7f2894069e90 2023-10-30T14:05:33.336 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:33.337 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:01.640919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:33.337 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 181 2023-10-30T14:05:33.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 -- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28780664a0 msgr2=0x7f2878068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:33.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28780664a0 0x7f2878068960 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f28840ae000 tx=0x7f28840a7000 comp rx=0 tx=0).stop 2023-10-30T14:05:33.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 -- 172.21.15.46:0/976604497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2894069e90 msgr2=0x7f2894148b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:33.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2894069e90 0x7f2894148b70 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f289000d5f0 tx=0x7f289000dab0 comp rx=0 tx=0).stop 2023-10-30T14:05:33.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 -- 172.21.15.46:0/976604497 shutdown_connections 2023-10-30T14:05:33.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28780664a0 0x7f2878068960 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:33.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28941580e0 0x7f28940c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:33.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28940c6540 0x7f2894146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:33.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 --2- 172.21.15.46:0/976604497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2894069e90 0x7f2894148b70 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:33.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 -- 172.21.15.46:0/976604497 >> 172.21.15.46:0/976604497 conn(0x7f28940c7910 msgr2=0x7f289415da80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:33.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 -- 172.21.15.46:0/976604497 shutdown_connections 2023-10-30T14:05:33.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:33.335+0000 7f289baa9640 1 -- 172.21.15.46:0/976604497 wait complete. 2023-10-30T14:05:33.357 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:05:35.359 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:35.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.511+0000 7f79e811d640 1 Processor -- start 2023-10-30T14:05:35.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.511+0000 7f79e811d640 1 -- start start 2023-10-30T14:05:35.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e811d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79e0070740 0x7f79e00c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:35.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e811d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f79e015e700 0x7f79e006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:35.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e811d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e00c5810 con 0x7f79e01580e0 2023-10-30T14:05:35.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e811d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e0070b60 con 0x7f79e015e700 2023-10-30T14:05:35.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e811d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e015eb90 con 0x7f79e0070740 2023-10-30T14:05:35.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e56b8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79e0070740 0x7f79e00c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:35.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e66ba640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f79e015e700 0x7f79e006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:35.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e56b8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79e0070740 0x7f79e00c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:38274/0 (socket says 172.21.15.46:38274) 2023-10-30T14:05:35.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e56b8640 1 -- 172.21.15.46:0/1089501301 learned_addr learned my addr 172.21.15.46:0/1089501301 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:35.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e66ba640 1 -- 172.21.15.46:0/1089501301 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79e0070740 msgr2=0x7f79e00c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:35.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e66ba640 1 --2- 172.21.15.46:0/1089501301 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79e0070740 0x7f79e00c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:35.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e66ba640 1 -- 172.21.15.46:0/1089501301 >> v1:172.21.15.154:6789/0 conn(0x7f79e01580e0 legacy=0x7f79e0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:35.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.515+0000 7f79e66ba640 1 -- 172.21.15.46:0/1089501301 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e00c6bb0 con 0x7f79e015e700 2023-10-30T14:05:35.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e66ba640 1 --2- 172.21.15.46:0/1089501301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79e015e700 0x7f79e006a5a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f79d8009d90 tx=0x7f79d801c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3d5a68c3036237 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:35.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e4eb7640 1 -- 172.21.15.46:0/1089501301 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d80a59e0 con 0x7f79e015e700 2023-10-30T14:05:35.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e4eb7640 1 -- 172.21.15.46:0/1089501301 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d80a5b40 con 0x7f79e015e700 2023-10-30T14:05:35.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e4eb7640 1 -- 172.21.15.46:0/1089501301 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d80ad440 con 0x7f79e015e700 2023-10-30T14:05:35.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e811d640 1 -- 172.21.15.46:0/1089501301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79e015e700 msgr2=0x7f79e006a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:35.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e811d640 1 --2- 172.21.15.46:0/1089501301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79e015e700 0x7f79e006a5a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f79d8009d90 tx=0x7f79d801c920 comp rx=0 tx=0).stop 2023-10-30T14:05:35.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e811d640 1 -- 172.21.15.46:0/1089501301 shutdown_connections 2023-10-30T14:05:35.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e811d640 1 --2- 172.21.15.46:0/1089501301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79e015e700 0x7f79e006a5a0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:35.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e811d640 1 --2- 172.21.15.46:0/1089501301 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79e0070740 0x7f79e00c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:35.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e811d640 1 -- 172.21.15.46:0/1089501301 >> 172.21.15.46:0/1089501301 conn(0x7f79e00c7910 msgr2=0x7f79e00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:35.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e811d640 1 -- 172.21.15.46:0/1089501301 shutdown_connections 2023-10-30T14:05:35.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e811d640 1 -- 172.21.15.46:0/1089501301 wait complete. 2023-10-30T14:05:35.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e811d640 1 Processor -- start 2023-10-30T14:05:35.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.519+0000 7f79e811d640 1 -- start start 2023-10-30T14:05:35.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e811d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79e0070740 0x7f79e01f7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e811d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e01580e0 0x7f79e015f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e811d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79e015e700 0x7f79e015f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e811d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e0160900 con 0x7f79e01580e0 2023-10-30T14:05:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e811d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e01f6900 con 0x7f79e015e700 2023-10-30T14:05:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e811d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e015fca0 con 0x7f79e0070740 2023-10-30T14:05:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e56b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e01580e0 0x7f79e015f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e56b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e01580e0 0x7f79e015f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52250/0 (socket says 172.21.15.46:52250) 2023-10-30T14:05:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e56b8640 1 -- 172.21.15.46:0/4285386019 learned_addr learned my addr 172.21.15.46:0/4285386019 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e5eb9640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79e0070740 0x7f79e01f7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e66ba640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79e015e700 0x7f79e015f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e56b8640 1 -- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79e0070740 msgr2=0x7f79e01f7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e56b8640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79e0070740 0x7f79e01f7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e56b8640 1 -- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79e015e700 msgr2=0x7f79e015f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e56b8640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79e015e700 0x7f79e015f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:35.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e56b8640 1 -- 172.21.15.46:0/4285386019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d8009a70 con 0x7f79e01580e0 2023-10-30T14:05:35.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e66ba640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79e015e700 0x7f79e015f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:35.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e56b8640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e01580e0 0x7f79e015f0a0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f79d000e890 tx=0x7f79d000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:35.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79d67fc640 1 -- 172.21.15.46:0/4285386019 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0009620 con 0x7f79e01580e0 2023-10-30T14:05:35.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79d67fc640 1 -- 172.21.15.46:0/4285386019 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d000c940 con 0x7f79e01580e0 2023-10-30T14:05:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79d67fc640 1 -- 172.21.15.46:0/4285386019 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d000ccf0 con 0x7f79e01580e0 2023-10-30T14:05:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e811d640 1 -- 172.21.15.46:0/4285386019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e01f5ff0 con 0x7f79e01580e0 2023-10-30T14:05:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e811d640 1 -- 172.21.15.46:0/4285386019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e01f62b0 con 0x7f79e01580e0 2023-10-30T14:05:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79e811d640 1 -- 172.21.15.46:0/4285386019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e01f6a70 con 0x7f79e01580e0 2023-10-30T14:05:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.523+0000 7f79d67fc640 1 -- 172.21.15.46:0/4285386019 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f79d0013470 con 0x7f79e01580e0 2023-10-30T14:05:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.527+0000 7f79d67fc640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f79c40664a0 0x7f79c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.527+0000 7f79d67fc640 1 -- 172.21.15.46:0/4285386019 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f79d0084710 con 0x7f79e01580e0 2023-10-30T14:05:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.527+0000 7f79e5eb9640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f79c40664a0 0x7f79c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.527+0000 7f79e5eb9640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f79c40664a0 0x7f79c4068960 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f79dc007fd0 tx=0x7f79dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.531+0000 7f79d67fc640 1 -- 172.21.15.46:0/4285386019 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f79d0051490 con 0x7f79e01580e0 2023-10-30T14:05:35.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.719+0000 7f79e811d640 1 -- 172.21.15.46:0/4285386019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79e00c5d10 con 0x7f79e01580e0 2023-10-30T14:05:35.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79d67fc640 1 -- 172.21.15.46:0/4285386019 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f79d0054ab0 con 0x7f79e01580e0 2023-10-30T14:05:35.724 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:35.724 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:01.640919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:35.724 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 181 2023-10-30T14:05:35.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79e811d640 1 -- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f79c40664a0 msgr2=0x7f79c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:35.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79e811d640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f79c40664a0 0x7f79c4068960 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f79dc007fd0 tx=0x7f79dc009040 comp rx=0 tx=0).stop 2023-10-30T14:05:35.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79e811d640 1 -- 172.21.15.46:0/4285386019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e01580e0 msgr2=0x7f79e015f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:35.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79e811d640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e01580e0 0x7f79e015f0a0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f79d000e890 tx=0x7f79d000ed50 comp rx=0 tx=0).stop 2023-10-30T14:05:35.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79e811d640 1 -- 172.21.15.46:0/4285386019 shutdown_connections 2023-10-30T14:05:35.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79e811d640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f79c40664a0 0x7f79c4068960 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:35.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79e811d640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79e015e700 0x7f79e015f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:35.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79e811d640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e01580e0 0x7f79e015f0a0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:35.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79e811d640 1 --2- 172.21.15.46:0/4285386019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79e0070740 0x7f79e01f7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:35.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79e811d640 1 -- 172.21.15.46:0/4285386019 >> 172.21.15.46:0/4285386019 conn(0x7f79e00c7910 msgr2=0x7f79e006b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:35.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79e811d640 1 -- 172.21.15.46:0/4285386019 shutdown_connections 2023-10-30T14:05:35.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:35.723+0000 7f79e811d640 1 -- 172.21.15.46:0/4285386019 wait complete. 2023-10-30T14:05:35.745 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:05:37.746 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:37.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979d217640 1 Processor -- start 2023-10-30T14:05:37.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979d217640 1 -- start start 2023-10-30T14:05:37.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979d217640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f97981538a0 0x7f979815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:37.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979d217640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97980ca0e0 0x7f979815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:37.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979d217640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97980cb800 con 0x7f97980cf160 2023-10-30T14:05:37.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979d217640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f979815dc30 con 0x7f97981538a0 2023-10-30T14:05:37.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979d217640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97980cb330 con 0x7f97980ca0e0 2023-10-30T14:05:37.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979759e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f97980cf160 0x7f97980cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45394/0 (socket says 172.21.15.46:45394) 2023-10-30T14:05:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979759e640 1 -- 172.21.15.46:0/1041900164 learned_addr learned my addr 172.21.15.46:0/1041900164 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:05:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f9796d9d640 1 --2- 172.21.15.46:0/1041900164 >> v2:172.21.15.154:3300/0 conn(0x7f97981538a0 0x7f979815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f978ffff640 1 --2- 172.21.15.46:0/1041900164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97980ca0e0 0x7f979815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:37.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328727038 0 0) 0x7f97980cb800 con 0x7f97980cf160 2023-10-30T14:05:37.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9784003530 con 0x7f97980cf160 2023-10-30T14:05:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 766334017 0 0) 0x7f9784003530 con 0x7f97980cf160 2023-10-30T14:05:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97980ca0e0 msgr2=0x7f979815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979659c640 1 --2- 172.21.15.46:0/1041900164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97980ca0e0 0x7f979815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 >> v2:172.21.15.154:3300/0 conn(0x7f97981538a0 msgr2=0x7f979815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:37.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979659c640 1 --2- 172.21.15.46:0/1041900164 >> v2:172.21.15.154:3300/0 conn(0x7f97981538a0 0x7f979815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97980cad70 con 0x7f97980cf160 2023-10-30T14:05:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f9790001a00 con 0x7f97980cf160 2023-10-30T14:05:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 >> v1:172.21.15.46:6789/0 conn(0x7f97980cf160 legacy=0x7f97980cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.899+0000 7f979659c640 1 --2- 172.21.15.46:0/1041900164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9784003980 0x7f9784003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:37.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f979659c640 1 --2- 172.21.15.46:0/1041900164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97980ca0e0 0x7f97840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f979659c640 1 --2- 172.21.15.46:0/1041900164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784004d10 0x7f9784005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979815dc30 con 0x7f9784003980 2023-10-30T14:05:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97980cb330 con 0x7f9784004d10 2023-10-30T14:05:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978400b990 con 0x7f97980ca0e0 2023-10-30T14:05:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f9796d9d640 1 --2- 172.21.15.46:0/1041900164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9784003980 0x7f9784003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:37.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f978ffff640 1 --2- 172.21.15.46:0/1041900164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97980ca0e0 0x7f97840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:37.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f979759e640 1 --2- 172.21.15.46:0/1041900164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784004d10 0x7f9784005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:37.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f9796d9d640 1 -- 172.21.15.46:0/1041900164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97980ca0e0 msgr2=0x7f97840047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:37.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f9796d9d640 1 --2- 172.21.15.46:0/1041900164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97980ca0e0 0x7f97840047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:37.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f9796d9d640 1 -- 172.21.15.46:0/1041900164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784004d10 msgr2=0x7f9784005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:37.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f9796d9d640 1 --2- 172.21.15.46:0/1041900164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784004d10 0x7f9784005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:37.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f9796d9d640 1 -- 172.21.15.46:0/1041900164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9784003530 con 0x7f9784003980 2023-10-30T14:05:37.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f9796d9d640 1 --2- 172.21.15.46:0/1041900164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9784003980 0x7f9784003da0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f9788009ae0 tx=0x7f978801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a87ee76da5388eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:37.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97880a5570 con 0x7f9784003980 2023-10-30T14:05:37.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97880a56d0 con 0x7f9784003980 2023-10-30T14:05:37.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.903+0000 7f979659c640 1 -- 172.21.15.46:0/1041900164 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97880a5a80 con 0x7f9784003980 2023-10-30T14:05:37.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 -- 172.21.15.46:0/1041900164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9784003980 msgr2=0x7f9784003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:37.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 --2- 172.21.15.46:0/1041900164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9784003980 0x7f9784003da0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f9788009ae0 tx=0x7f978801ca90 comp rx=0 tx=0).stop 2023-10-30T14:05:37.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979759e640 1 -- 172.21.15.46:0/1041900164 reap_dead start 2023-10-30T14:05:37.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 -- 172.21.15.46:0/1041900164 shutdown_connections 2023-10-30T14:05:37.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 -- 172.21.15.46:0/1041900164 >> 172.21.15.46:0/1041900164 conn(0x7f979806bfe0 msgr2=0x7f979806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:37.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 -- 172.21.15.46:0/1041900164 shutdown_connections 2023-10-30T14:05:37.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 -- 172.21.15.46:0/1041900164 wait complete. 2023-10-30T14:05:37.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 Processor -- start 2023-10-30T14:05:37.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 -- start start 2023-10-30T14:05:37.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9784003980 0x7f97980ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:37.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97981f0000 0x7f9798157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:37.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97980cd550 0x7f97980b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:37.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97980b4f00 con 0x7f97980cd550 2023-10-30T14:05:37.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97981efb70 con 0x7f97981f0000 2023-10-30T14:05:37.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979d217640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97980b3650 con 0x7f9784003980 2023-10-30T14:05:37.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97980cd550 0x7f97980b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:37.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97980cd550 0x7f97980b3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34216/0 (socket says 172.21.15.46:34216) 2023-10-30T14:05:37.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f979759e640 1 -- 172.21.15.46:0/1198778501 learned_addr learned my addr 172.21.15.46:0/1198778501 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:37.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f9796d9d640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9784003980 0x7f97980ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:37.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.907+0000 7f978ffff640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97981f0000 0x7f9798157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:37.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f979759e640 1 -- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9784003980 msgr2=0x7f97980ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:37.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f979759e640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9784003980 0x7f97980ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:37.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f979759e640 1 -- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97981f0000 msgr2=0x7f9798157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:37.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f979759e640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97981f0000 0x7f9798157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:37.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f979759e640 1 -- 172.21.15.46:0/1198778501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97980b4980 con 0x7f97980cd550 2023-10-30T14:05:37.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f978ffff640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97981f0000 0x7f9798157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:37.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f979759e640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97980cd550 0x7f97980b3110 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f97980ccd10 tx=0x7f9790011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:37.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f978f7fe640 1 -- 172.21.15.46:0/1198778501 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979000f8c0 con 0x7f97980cd550 2023-10-30T14:05:37.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f978f7fe640 1 -- 172.21.15.46:0/1198778501 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979000fa40 con 0x7f97980cd550 2023-10-30T14:05:37.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f979d217640 1 -- 172.21.15.46:0/1198778501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97980b4c20 con 0x7f97980cd550 2023-10-30T14:05:37.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f978f7fe640 1 -- 172.21.15.46:0/1198778501 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979000fdf0 con 0x7f97980cd550 2023-10-30T14:05:37.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f979d217640 1 -- 172.21.15.46:0/1198778501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97981ef800 con 0x7f97980cd550 2023-10-30T14:05:37.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f979d217640 1 -- 172.21.15.46:0/1198778501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97981ef9c0 con 0x7f97980cd550 2023-10-30T14:05:37.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f978f7fe640 1 -- 172.21.15.46:0/1198778501 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f979001f460 con 0x7f97980cd550 2023-10-30T14:05:37.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f978f7fe640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9778066470 0x7f9778068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:37.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.911+0000 7f978f7fe640 1 -- 172.21.15.46:0/1198778501 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f9790087610 con 0x7f97980cd550 2023-10-30T14:05:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.915+0000 7f9796d9d640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9778066470 0x7f9778068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.915+0000 7f9796d9d640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9778066470 0x7f9778068930 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f97880af000 tx=0x7f97880046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:37.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:37.915+0000 7f978f7fe640 1 -- 172.21.15.46:0/1198778501 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97900541c0 con 0x7f97980cd550 2023-10-30T14:05:38.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.103+0000 7f979d217640 1 -- 172.21.15.46:0/1198778501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97981ef380 con 0x7f97980cd550 2023-10-30T14:05:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.103+0000 7f978f7fe640 1 -- 172.21.15.46:0/1198778501 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f97981ef380 con 0x7f97980cd550 2023-10-30T14:05:38.107 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:38.107 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:01.640919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8034},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8034":{"gid":8034,"name":"a","rank":1,"incarnation":167,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6833/2075693104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2075693104},{"type":"v1","addr":"172.21.15.46:6833","nonce":2075693104}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:38.108 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 181 2023-10-30T14:05:38.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.107+0000 7f979d217640 1 -- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9778066470 msgr2=0x7f9778068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:38.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.107+0000 7f979d217640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9778066470 0x7f9778068930 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f97880af000 tx=0x7f97880046d0 comp rx=0 tx=0).stop 2023-10-30T14:05:38.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.107+0000 7f979d217640 1 -- 172.21.15.46:0/1198778501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97980cd550 msgr2=0x7f97980b3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:38.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.107+0000 7f979d217640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97980cd550 0x7f97980b3110 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f97980ccd10 tx=0x7f9790011c20 comp rx=0 tx=0).stop 2023-10-30T14:05:38.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.107+0000 7f979d217640 1 -- 172.21.15.46:0/1198778501 shutdown_connections 2023-10-30T14:05:38.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.107+0000 7f979d217640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9778066470 0x7f9778068930 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:38.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.107+0000 7f979d217640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97980cd550 0x7f97980b3110 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:38.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.107+0000 7f979d217640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97981f0000 0x7f9798157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.107+0000 7f979d217640 1 --2- 172.21.15.46:0/1198778501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9784003980 0x7f97980ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.107+0000 7f979d217640 1 -- 172.21.15.46:0/1198778501 >> 172.21.15.46:0/1198778501 conn(0x7f979806bfe0 msgr2=0x7f97980c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.107+0000 7f979d217640 1 -- 172.21.15.46:0/1198778501 shutdown_connections 2023-10-30T14:05:38.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:38.107+0000 7f979d217640 1 -- 172.21.15.46:0/1198778501 wait complete. 2023-10-30T14:05:38.128 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:05:40.130 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:40.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.283+0000 7f4363868640 1 Processor -- start 2023-10-30T14:05:40.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.283+0000 7f4363868640 1 -- start start 2023-10-30T14:05:40.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.283+0000 7f4363868640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f435c0c6540 0x7f435c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.283+0000 7f4363868640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c0c69d0 con 0x7f435c1580e0 2023-10-30T14:05:40.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.283+0000 7f4363868640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c0c5810 con 0x7f435c070740 2023-10-30T14:05:40.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.283+0000 7f4363868640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c06acb0 con 0x7f435c0c6540 2023-10-30T14:05:40.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4361604640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f435c1580e0 0x7f435c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45396/0 (socket says 172.21.15.46:45396) 2023-10-30T14:05:40.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4361e05640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f435c0c6540 0x7f435c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4361604640 1 -- 172.21.15.46:0/251109374 learned_addr learned my addr 172.21.15.46:0/251109374 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:05:40.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4360e03640 1 --1- 172.21.15.46:0/251109374 >> v1:172.21.15.154:6789/0 conn(0x7f435c070740 0x7f435c06a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:54316/0 (socket says 172.21.15.46:54316) 2023-10-30T14:05:40.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215874285 0 0) 0x7f435c0c69d0 con 0x7f435c1580e0 2023-10-30T14:05:40.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4348003530 con 0x7f435c1580e0 2023-10-30T14:05:40.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852866905 0 0) 0x7f435c0c5810 con 0x7f435c070740 2023-10-30T14:05:40.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c0c69d0 con 0x7f435c070740 2023-10-30T14:05:40.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 666103301 0 0) 0x7f4348003530 con 0x7f435c1580e0 2023-10-30T14:05:40.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f435c0c6540 msgr2=0x7f435c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:40.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 --2- 172.21.15.46:0/251109374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f435c0c6540 0x7f435c15e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 >> v1:172.21.15.154:6789/0 conn(0x7f435c070740 legacy=0x7f435c06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c0c6bb0 con 0x7f435c1580e0 2023-10-30T14:05:40.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f4358001680 con 0x7f435c1580e0 2023-10-30T14:05:40.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 >> v1:172.21.15.46:6789/0 conn(0x7f435c1580e0 legacy=0x7f435c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 --2- 172.21.15.46:0/251109374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f435c0c6540 0x7f4348004ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 --2- 172.21.15.46:0/251109374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4348005100 0x7f4348003770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 --2- 172.21.15.46:0/251109374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4348003cb0 0x7f4348004130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434800ba40 con 0x7f4348003cb0 2023-10-30T14:05:40.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434800bbb0 con 0x7f4348005100 2023-10-30T14:05:40.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434800bd70 con 0x7f435c0c6540 2023-10-30T14:05:40.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4361e05640 1 --2- 172.21.15.46:0/251109374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4348003cb0 0x7f4348004130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4361604640 1 --2- 172.21.15.46:0/251109374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f435c0c6540 0x7f4348004ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4360e03640 1 --2- 172.21.15.46:0/251109374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4348005100 0x7f4348003770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4361e05640 1 -- 172.21.15.46:0/251109374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f435c0c6540 msgr2=0x7f4348004ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:40.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4361e05640 1 --2- 172.21.15.46:0/251109374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f435c0c6540 0x7f4348004ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4361e05640 1 -- 172.21.15.46:0/251109374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4348005100 msgr2=0x7f4348003770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:40.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4361e05640 1 --2- 172.21.15.46:0/251109374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4348005100 0x7f4348003770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.287+0000 7f4361e05640 1 -- 172.21.15.46:0/251109374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4348003530 con 0x7f4348003cb0 2023-10-30T14:05:40.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.291+0000 7f4361e05640 1 --2- 172.21.15.46:0/251109374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4348003cb0 0x7f4348004130 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f4354009fc0 tx=0x7f435401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a091fdbb9f0cfc98 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:40.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.291+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4358007ea0 con 0x7f4348003cb0 2023-10-30T14:05:40.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.291+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435400f040 con 0x7f4348003cb0 2023-10-30T14:05:40.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.291+0000 7f4353fff640 1 -- 172.21.15.46:0/251109374 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43540a5da0 con 0x7f4348003cb0 2023-10-30T14:05:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.291+0000 7f4363868640 1 -- 172.21.15.46:0/251109374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4348003cb0 msgr2=0x7f4348004130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.291+0000 7f4363868640 1 --2- 172.21.15.46:0/251109374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4348003cb0 0x7f4348004130 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f4354009fc0 tx=0x7f435401cc60 comp rx=0 tx=0).stop 2023-10-30T14:05:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.291+0000 7f4361e05640 1 -- 172.21.15.46:0/251109374 reap_dead start 2023-10-30T14:05:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.291+0000 7f4363868640 1 -- 172.21.15.46:0/251109374 shutdown_connections 2023-10-30T14:05:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.291+0000 7f4363868640 1 -- 172.21.15.46:0/251109374 >> 172.21.15.46:0/251109374 conn(0x7f435c0c7910 msgr2=0x7f435c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:40.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.291+0000 7f4363868640 1 -- 172.21.15.46:0/251109374 shutdown_connections 2023-10-30T14:05:40.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.291+0000 7f4363868640 1 -- 172.21.15.46:0/251109374 wait complete. 2023-10-30T14:05:40.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.291+0000 7f4363868640 1 Processor -- start 2023-10-30T14:05:40.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4363868640 1 -- start start 2023-10-30T14:05:40.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4363868640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4348003cb0 0x7f435c1f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4363868640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f435c0c48a0 0x7f435c14ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4363868640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f435c1f6f80 0x7f435c14d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4363868640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1f4d10 con 0x7f435c0c48a0 2023-10-30T14:05:40.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4363868640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c14ee10 con 0x7f435c1f6f80 2023-10-30T14:05:40.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4363868640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1f46e0 con 0x7f4348003cb0 2023-10-30T14:05:40.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4360e03640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f435c0c48a0 0x7f435c14ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4360e03640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f435c0c48a0 0x7f435c14ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34220/0 (socket says 172.21.15.46:34220) 2023-10-30T14:05:40.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4360e03640 1 -- 172.21.15.46:0/1341305224 learned_addr learned my addr 172.21.15.46:0/1341305224 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:40.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4361604640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4348003cb0 0x7f435c1f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4361e05640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f435c1f6f80 0x7f435c14d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4360e03640 1 -- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4348003cb0 msgr2=0x7f435c1f6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4360e03640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4348003cb0 0x7f435c1f6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4360e03640 1 -- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f435c1f6f80 msgr2=0x7f435c14d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4360e03640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f435c1f6f80 0x7f435c14d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4360e03640 1 -- 172.21.15.46:0/1341305224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c158ac0 con 0x7f435c0c48a0 2023-10-30T14:05:40.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4360e03640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f435c0c48a0 0x7f435c14ce60 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f434c001670 tx=0x7f434c001b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:40.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4351ffb640 1 -- 172.21.15.46:0/1341305224 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c0049d0 con 0x7f435c0c48a0 2023-10-30T14:05:40.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4351ffb640 1 -- 172.21.15.46:0/1341305224 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434c001d20 con 0x7f435c0c48a0 2023-10-30T14:05:40.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4351ffb640 1 -- 172.21.15.46:0/1341305224 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c00fba0 con 0x7f435c0c48a0 2023-10-30T14:05:40.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4363868640 1 -- 172.21.15.46:0/1341305224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435c14f040 con 0x7f435c0c48a0 2023-10-30T14:05:40.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4363868640 1 -- 172.21.15.46:0/1341305224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435c1f4930 con 0x7f435c0c48a0 2023-10-30T14:05:40.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.295+0000 7f4363868640 1 -- 172.21.15.46:0/1341305224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435c1f4e50 con 0x7f435c0c48a0 2023-10-30T14:05:40.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.299+0000 7f4351ffb640 1 -- 172.21.15.46:0/1341305224 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f434c00fd00 con 0x7f435c0c48a0 2023-10-30T14:05:40.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.299+0000 7f4351ffb640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43400664d0 0x7f4340068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.299+0000 7f4351ffb640 1 -- 172.21.15.46:0/1341305224 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7f434c084580 con 0x7f435c0c48a0 2023-10-30T14:05:40.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.299+0000 7f4361604640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43400664d0 0x7f4340068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.299+0000 7f4361604640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43400664d0 0x7f4340068990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4358001060 tx=0x7f4358001e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:40.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.303+0000 7f4351ffb640 1 -- 172.21.15.46:0/1341305224 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f435c1f4e50 con 0x7f435c0c48a0 2023-10-30T14:05:40.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.491+0000 7f4363868640 1 -- 172.21.15.46:0/1341305224 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f435c1f4e50 con 0x7f435c0c48a0 2023-10-30T14:05:40.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.491+0000 7f4351ffb640 1 -- 172.21.15.46:0/1341305224 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f435c1f4e50 con 0x7f435c0c48a0 2023-10-30T14:05:40.495 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:40.495 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:39.493092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 182 2023-10-30T14:05:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.495+0000 7f4363868640 1 -- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43400664d0 msgr2=0x7f4340068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.495+0000 7f4363868640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43400664d0 0x7f4340068990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4358001060 tx=0x7f4358001e30 comp rx=0 tx=0).stop 2023-10-30T14:05:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.495+0000 7f4363868640 1 -- 172.21.15.46:0/1341305224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f435c0c48a0 msgr2=0x7f435c14ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.495+0000 7f4363868640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f435c0c48a0 0x7f435c14ce60 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f434c001670 tx=0x7f434c001b30 comp rx=0 tx=0).stop 2023-10-30T14:05:40.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.495+0000 7f4363868640 1 -- 172.21.15.46:0/1341305224 shutdown_connections 2023-10-30T14:05:40.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.495+0000 7f4363868640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f43400664d0 0x7f4340068990 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.495+0000 7f4363868640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f435c1f6f80 0x7f435c14d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.495+0000 7f4363868640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f435c0c48a0 0x7f435c14ce60 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.495+0000 7f4363868640 1 --2- 172.21.15.46:0/1341305224 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4348003cb0 0x7f435c1f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.495+0000 7f4363868640 1 -- 172.21.15.46:0/1341305224 >> 172.21.15.46:0/1341305224 conn(0x7f435c0c7910 msgr2=0x7f435c06b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:40.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.495+0000 7f4363868640 1 -- 172.21.15.46:0/1341305224 shutdown_connections 2023-10-30T14:05:40.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.495+0000 7f4363868640 1 -- 172.21.15.46:0/1341305224 wait complete. 2023-10-30T14:05:40.517 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T14:05:40.517 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:05:40.518 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:40.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f184d6640 1 Processor -- start 2023-10-30T14:05:40.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f184d6640 1 -- start start 2023-10-30T14:05:40.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f184d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7f101538a0 0x7f7f1015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f184d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f100cb800 con 0x7f7f100cf160 2023-10-30T14:05:40.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f184d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1015dc30 con 0x7f7f100ca0e0 2023-10-30T14:05:40.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f184d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f100cb330 con 0x7f7f101538a0 2023-10-30T14:05:40.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f16a73640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7f100cf160 0x7f7f10164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45404/0 (socket says 172.21.15.46:45404) 2023-10-30T14:05:40.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f16a73640 1 -- 172.21.15.46:0/742833912 learned_addr learned my addr 172.21.15.46:0/742833912 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:05:40.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f16272640 1 --2- 172.21.15.46:0/742833912 >> v2:172.21.15.154:3300/0 conn(0x7f7f101538a0 0x7f7f1015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702780126 0 0) 0x7f7f100cb800 con 0x7f7f100cf160 2023-10-30T14:05:40.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7efc003530 con 0x7f7f100cf160 2023-10-30T14:05:40.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973475417 0 0) 0x7f7f00001120 con 0x7f7f100ca0e0 2023-10-30T14:05:40.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f100cb800 con 0x7f7f100ca0e0 2023-10-30T14:05:40.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1195559946 0 0) 0x7f7efc003530 con 0x7f7f100cf160 2023-10-30T14:05:40.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 >> v2:172.21.15.154:3300/0 conn(0x7f7f101538a0 msgr2=0x7f7f1015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:40.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 --2- 172.21.15.46:0/742833912 >> v2:172.21.15.154:3300/0 conn(0x7f7f101538a0 0x7f7f1015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 >> v1:172.21.15.154:6789/0 conn(0x7f7f100ca0e0 legacy=0x7f7f1015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f100cad70 con 0x7f7f100cf160 2023-10-30T14:05:40.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f7f08001a00 con 0x7f7f100cf160 2023-10-30T14:05:40.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 >> v1:172.21.15.46:6789/0 conn(0x7f7f100cf160 legacy=0x7f7f10164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 --2- 172.21.15.46:0/742833912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7efc003b60 0x7f7efc003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 --2- 172.21.15.46:0/742833912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc0047d0 0x7f7efc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 --2- 172.21.15.46:0/742833912 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc005190 0x7f7efc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100cb330 con 0x7f7efc003b60 2023-10-30T14:05:40.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc00bfb0 con 0x7f7efc0047d0 2023-10-30T14:05:40.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.659+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc00c170 con 0x7f7efc005190 2023-10-30T14:05:40.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f16272640 1 --2- 172.21.15.46:0/742833912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7efc003b60 0x7f7efc003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f15a71640 1 --2- 172.21.15.46:0/742833912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc0047d0 0x7f7efc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f16a73640 1 --2- 172.21.15.46:0/742833912 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc005190 0x7f7efc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f16272640 1 -- 172.21.15.46:0/742833912 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc005190 msgr2=0x7f7efc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:40.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f16272640 1 --2- 172.21.15.46:0/742833912 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7efc005190 0x7f7efc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f16272640 1 -- 172.21.15.46:0/742833912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc0047d0 msgr2=0x7f7efc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:40.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f16272640 1 --2- 172.21.15.46:0/742833912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc0047d0 0x7f7efc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f16272640 1 -- 172.21.15.46:0/742833912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc003530 con 0x7f7efc003b60 2023-10-30T14:05:40.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f15a71640 1 --2- 172.21.15.46:0/742833912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc0047d0 0x7f7efc004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:40.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f16a73640 1 -- 172.21.15.46:0/742833912 reap_dead start 2023-10-30T14:05:40.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f16272640 1 --2- 172.21.15.46:0/742833912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7efc003b60 0x7f7efc003f60 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009ae0 tx=0x7f7f0c01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5dfe45ca4a061f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:40.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c0a5660 con 0x7f7efc003b60 2023-10-30T14:05:40.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f00001d80 con 0x7f7efc003b60 2023-10-30T14:05:40.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.663+0000 7f7f15270640 1 -- 172.21.15.46:0/742833912 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c0a5a00 con 0x7f7efc003b60 2023-10-30T14:05:40.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 -- 172.21.15.46:0/742833912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7efc003b60 msgr2=0x7f7efc003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 --2- 172.21.15.46:0/742833912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7efc003b60 0x7f7efc003f60 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009ae0 tx=0x7f7f0c01ca70 comp rx=0 tx=0).stop 2023-10-30T14:05:40.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 -- 172.21.15.46:0/742833912 shutdown_connections 2023-10-30T14:05:40.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 --2- 172.21.15.46:0/742833912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7efc003b60 0x7f7efc003f60 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 -- 172.21.15.46:0/742833912 >> 172.21.15.46:0/742833912 conn(0x7f7f1006bfe0 msgr2=0x7f7f1006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 -- 172.21.15.46:0/742833912 shutdown_connections 2023-10-30T14:05:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 -- 172.21.15.46:0/742833912 wait complete. 2023-10-30T14:05:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 Processor -- start 2023-10-30T14:05:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 -- start start 2023-10-30T14:05:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc003b60 0x7f7f10157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f101efab0 0x7f7f1014dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f101f2880 0x7f7f101f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f10159700 con 0x7f7f101f2880 2023-10-30T14:05:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f10158310 con 0x7f7efc003b60 2023-10-30T14:05:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f184d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f10158b20 con 0x7f7f101efab0 2023-10-30T14:05:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f16a73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f101f2880 0x7f7f101f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f16a73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f101f2880 0x7f7f101f2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34246/0 (socket says 172.21.15.46:34246) 2023-10-30T14:05:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f16a73640 1 -- 172.21.15.46:0/3561712522 learned_addr learned my addr 172.21.15.46:0/3561712522 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f16272640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc003b60 0x7f7f10157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f15a71640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f101efab0 0x7f7f1014dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f16a73640 1 -- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f101efab0 msgr2=0x7f7f1014dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f16a73640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f101efab0 0x7f7f1014dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f16a73640 1 -- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc003b60 msgr2=0x7f7f10157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f16a73640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc003b60 0x7f7f10157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f16a73640 1 -- 172.21.15.46:0/3561712522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f101f3240 con 0x7f7f101f2880 2023-10-30T14:05:40.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.667+0000 7f7f15a71640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f101efab0 0x7f7f1014dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:40.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.671+0000 7f7f16a73640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f101f2880 0x7f7f101f2d00 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7f10072ce0 tx=0x7f7f08007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:40.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.671+0000 7f7f06ffd640 1 -- 172.21.15.46:0/3561712522 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08017070 con 0x7f7f101f2880 2023-10-30T14:05:40.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.671+0000 7f7f06ffd640 1 -- 172.21.15.46:0/3561712522 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0800e880 con 0x7f7f101f2880 2023-10-30T14:05:40.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.671+0000 7f7f06ffd640 1 -- 172.21.15.46:0/3561712522 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0800ea10 con 0x7f7f101f2880 2023-10-30T14:05:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.671+0000 7f7f184d6640 1 -- 172.21.15.46:0/3561712522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f10158840 con 0x7f7f101f2880 2023-10-30T14:05:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.671+0000 7f7f184d6640 1 -- 172.21.15.46:0/3561712522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f10159320 con 0x7f7f101f2880 2023-10-30T14:05:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.671+0000 7f7f184d6640 1 -- 172.21.15.46:0/3561712522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f101f25a0 con 0x7f7f101f2880 2023-10-30T14:05:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.671+0000 7f7f06ffd640 1 -- 172.21.15.46:0/3561712522 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7f0800eb70 con 0x7f7f101f2880 2023-10-30T14:05:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.671+0000 7f7f06ffd640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7ef40664d0 0x7f7ef4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.671+0000 7f7f06ffd640 1 -- 172.21.15.46:0/3561712522 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7f7f08083f40 con 0x7f7f101f2880 2023-10-30T14:05:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.671+0000 7f7f16272640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7ef40664d0 0x7f7ef4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:40.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.675+0000 7f7f16272640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7ef40664d0 0x7f7ef4068990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0af000 tx=0x7f7f0c0046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:40.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.675+0000 7f7f06ffd640 1 -- 172.21.15.46:0/3561712522 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7f101f25a0 con 0x7f7f101f2880 2023-10-30T14:05:40.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.867+0000 7f7f184d6640 1 -- 172.21.15.46:0/3561712522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f101f25a0 con 0x7f7f101f2880 2023-10-30T14:05:40.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.867+0000 7f7f06ffd640 1 -- 172.21.15.46:0/3561712522 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f7f101f25a0 con 0x7f7f101f2880 2023-10-30T14:05:40.871 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:40.871 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:39.493092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:40.872 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 182 2023-10-30T14:05:40.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.871+0000 7f7f184d6640 1 -- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7ef40664d0 msgr2=0x7f7ef4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.871+0000 7f7f184d6640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7ef40664d0 0x7f7ef4068990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0af000 tx=0x7f7f0c0046d0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.871+0000 7f7f184d6640 1 -- 172.21.15.46:0/3561712522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f101f2880 msgr2=0x7f7f101f2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:40.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.871+0000 7f7f184d6640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f101f2880 0x7f7f101f2d00 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7f10072ce0 tx=0x7f7f08007ad0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.871+0000 7f7f184d6640 1 -- 172.21.15.46:0/3561712522 shutdown_connections 2023-10-30T14:05:40.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.871+0000 7f7f184d6640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7ef40664d0 0x7f7ef4068990 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.871+0000 7f7f184d6640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f101f2880 0x7f7f101f2d00 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.871+0000 7f7f184d6640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f101efab0 0x7f7f1014dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.871+0000 7f7f184d6640 1 --2- 172.21.15.46:0/3561712522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc003b60 0x7f7f10157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:40.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.871+0000 7f7f184d6640 1 -- 172.21.15.46:0/3561712522 >> 172.21.15.46:0/3561712522 conn(0x7f7f1006bfe0 msgr2=0x7f7f1014fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:40.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.871+0000 7f7f184d6640 1 -- 172.21.15.46:0/3561712522 shutdown_connections 2023-10-30T14:05:40.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:40.871+0000 7f7f184d6640 1 -- 172.21.15.46:0/3561712522 wait complete. 2023-10-30T14:05:40.894 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 182, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 182, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8190": { "addr": "172.21.15.46:6837/3154171789", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3154171789, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3154171789, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 8190, "incarnation": 175, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 40 }, "gid_8352": { "addr": "172.21.15.46:6836/128865315", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 128865315, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 128865315, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8352, "incarnation": 182, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:05:39.493092+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8190, "mds_1": 8352 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:05:42.895 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:43.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.047+0000 7ff8b16d6640 1 Processor -- start 2023-10-30T14:05:43.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.047+0000 7ff8b16d6640 1 -- start start 2023-10-30T14:05:43.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.051+0000 7ff8b16d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8ac0cf160 0x7ff8ac0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:43.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.051+0000 7ff8b16d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac0cb800 con 0x7ff8ac15d6f0 2023-10-30T14:05:43.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.051+0000 7ff8b16d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac0cb330 con 0x7ff8ac1538a0 2023-10-30T14:05:43.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.051+0000 7ff8b16d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ac0cad70 con 0x7ff8ac0cf160 2023-10-30T14:05:43.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.051+0000 7ff8ab7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8ac0cf160 0x7ff8ac0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:43.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.051+0000 7ff8ab7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8ac0cf160 0x7ff8ac0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34250/0 (socket says 172.21.15.46:34250) 2023-10-30T14:05:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.051+0000 7ff8ab7fe640 1 -- 172.21.15.46:0/834963653 learned_addr learned my addr 172.21.15.46:0/834963653 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.051+0000 7ff8ab7fe640 1 -- 172.21.15.46:0/834963653 >> v1:172.21.15.154:6789/0 conn(0x7ff8ac1538a0 legacy=0x7ff8ac0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.051+0000 7ff8ab7fe640 1 -- 172.21.15.46:0/834963653 >> v1:172.21.15.46:6789/0 conn(0x7ff8ac15d6f0 legacy=0x7ff8ac15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.051+0000 7ff8ab7fe640 1 -- 172.21.15.46:0/834963653 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac0cb050 con 0x7ff8ac0cf160 2023-10-30T14:05:43.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8ab7fe640 1 --2- 172.21.15.46:0/834963653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8ac0cf160 0x7ff8ac0cf590 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009e50 tx=0x7ff8a401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1058cab329648329 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:43.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8a9ffb640 1 -- 172.21.15.46:0/834963653 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a40af400 con 0x7ff8ac0cf160 2023-10-30T14:05:43.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8a9ffb640 1 -- 172.21.15.46:0/834963653 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a40ab050 con 0x7ff8ac0cf160 2023-10-30T14:05:43.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8a9ffb640 1 -- 172.21.15.46:0/834963653 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a40af780 con 0x7ff8ac0cf160 2023-10-30T14:05:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8b16d6640 1 -- 172.21.15.46:0/834963653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8ac0cf160 msgr2=0x7ff8ac0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8b16d6640 1 --2- 172.21.15.46:0/834963653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8ac0cf160 0x7ff8ac0cf590 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009e50 tx=0x7ff8a401f1f0 comp rx=0 tx=0).stop 2023-10-30T14:05:43.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8b16d6640 1 -- 172.21.15.46:0/834963653 shutdown_connections 2023-10-30T14:05:43.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8b16d6640 1 --2- 172.21.15.46:0/834963653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8ac0cf160 0x7ff8ac0cf590 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:43.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8b16d6640 1 -- 172.21.15.46:0/834963653 >> 172.21.15.46:0/834963653 conn(0x7ff8ac06bfe0 msgr2=0x7ff8ac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:43.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8b16d6640 1 -- 172.21.15.46:0/834963653 shutdown_connections 2023-10-30T14:05:43.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8b16d6640 1 -- 172.21.15.46:0/834963653 wait complete. 2023-10-30T14:05:43.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8b16d6640 1 Processor -- start 2023-10-30T14:05:43.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.055+0000 7ff8b16d6640 1 -- start start 2023-10-30T14:05:43.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8b16d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8ac0cf160 0x7ff8ac154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:43.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8b16d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8ac1538a0 0x7ff8ac0b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:43.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8b16d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8ac15d6f0 0x7ff8ac1e2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8b16d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1559b0 con 0x7ff8ac0cf160 2023-10-30T14:05:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8b16d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac159410 con 0x7ff8ac1538a0 2023-10-30T14:05:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8b16d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1e3de0 con 0x7ff8ac15d6f0 2023-10-30T14:05:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8aaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8ac0cf160 0x7ff8ac154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8aaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8ac0cf160 0x7ff8ac154dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34258/0 (socket says 172.21.15.46:34258) 2023-10-30T14:05:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8aaffd640 1 -- 172.21.15.46:0/1673729337 learned_addr learned my addr 172.21.15.46:0/1673729337 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8ab7fe640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8ac15d6f0 0x7ff8ac1e2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8aa7fc640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8ac1538a0 0x7ff8ac0b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:43.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8aaffd640 1 -- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8ac15d6f0 msgr2=0x7ff8ac1e2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8aaffd640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8ac15d6f0 0x7ff8ac1e2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8aaffd640 1 -- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8ac1538a0 msgr2=0x7ff8ac0b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8aaffd640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8ac1538a0 0x7ff8ac0b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8aaffd640 1 -- 172.21.15.46:0/1673729337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a4009b30 con 0x7ff8ac0cf160 2023-10-30T14:05:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8aaffd640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8ac0cf160 0x7ff8ac154dc0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00c530 tx=0x7ff89c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:43.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8937fe640 1 -- 172.21.15.46:0/1673729337 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c005220 con 0x7ff8ac0cf160 2023-10-30T14:05:43.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8937fe640 1 -- 172.21.15.46:0/1673729337 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89c005380 con 0x7ff8ac0cf160 2023-10-30T14:05:43.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8937fe640 1 -- 172.21.15.46:0/1673729337 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c007270 con 0x7ff8ac0cf160 2023-10-30T14:05:43.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8b16d6640 1 -- 172.21.15.46:0/1673729337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ac156ce0 con 0x7ff8ac0cf160 2023-10-30T14:05:43.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8b16d6640 1 -- 172.21.15.46:0/1673729337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac155300 con 0x7ff8ac0cf160 2023-10-30T14:05:43.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8b16d6640 1 -- 172.21.15.46:0/1673729337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ac04e170 con 0x7ff8ac0cf160 2023-10-30T14:05:43.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.059+0000 7ff8937fe640 1 -- 172.21.15.46:0/1673729337 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff89c0073d0 con 0x7ff8ac0cf160 2023-10-30T14:05:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.063+0000 7ff8937fe640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8980664a0 0x7ff898068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.063+0000 7ff8937fe640 1 -- 172.21.15.46:0/1673729337 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7ff89c083e60 con 0x7ff8ac0cf160 2023-10-30T14:05:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.063+0000 7ff8aa7fc640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8980664a0 0x7ff898068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.063+0000 7ff8aa7fc640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8980664a0 0x7ff898068960 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0c47a0 tx=0x7ff8a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.067+0000 7ff8937fe640 1 -- 172.21.15.46:0/1673729337 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff89c050b80 con 0x7ff8ac0cf160 2023-10-30T14:05:43.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.255+0000 7ff8b16d6640 1 -- 172.21.15.46:0/1673729337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8ac0b1dd0 con 0x7ff8ac0cf160 2023-10-30T14:05:43.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8937fe640 1 -- 172.21.15.46:0/1673729337 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5729 (secure 0 0 0) 0x7ff89c0541a0 con 0x7ff8ac0cf160 2023-10-30T14:05:43.260 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:43.260 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:39.493092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:43.261 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 182 2023-10-30T14:05:43.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8b16d6640 1 -- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8980664a0 msgr2=0x7ff898068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:43.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8b16d6640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8980664a0 0x7ff898068960 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0c47a0 tx=0x7ff8a0009040 comp rx=0 tx=0).stop 2023-10-30T14:05:43.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8b16d6640 1 -- 172.21.15.46:0/1673729337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8ac0cf160 msgr2=0x7ff8ac154dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:43.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8b16d6640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8ac0cf160 0x7ff8ac154dc0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00c530 tx=0x7ff89c00c9f0 comp rx=0 tx=0).stop 2023-10-30T14:05:43.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8b16d6640 1 -- 172.21.15.46:0/1673729337 shutdown_connections 2023-10-30T14:05:43.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8b16d6640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8980664a0 0x7ff898068960 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:43.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8b16d6640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8ac15d6f0 0x7ff8ac1e2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:43.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8b16d6640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8ac1538a0 0x7ff8ac0b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:43.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8b16d6640 1 --2- 172.21.15.46:0/1673729337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8ac0cf160 0x7ff8ac154dc0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:43.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8b16d6640 1 -- 172.21.15.46:0/1673729337 >> 172.21.15.46:0/1673729337 conn(0x7ff8ac06bfe0 msgr2=0x7ff8ac0735c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:43.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8b16d6640 1 -- 172.21.15.46:0/1673729337 shutdown_connections 2023-10-30T14:05:43.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:43.259+0000 7ff8b16d6640 1 -- 172.21.15.46:0/1673729337 wait complete. 2023-10-30T14:05:43.281 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:05:45.282 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:45.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.427+0000 7f4bc06d6640 1 Processor -- start 2023-10-30T14:05:45.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.427+0000 7f4bc06d6640 1 -- start start 2023-10-30T14:05:45.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bc06d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4bb81538a0 0x7f4bb815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:45.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bc06d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4bb80cf160 0x7f4bb80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:45.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bc06d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb80cb800 con 0x7f4bb80ca0e0 2023-10-30T14:05:45.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bc06d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb815dc30 con 0x7f4bb80cf160 2023-10-30T14:05:45.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bc06d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb80cb330 con 0x7f4bb81538a0 2023-10-30T14:05:45.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bbec73640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4bb80cf160 0x7f4bb80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:45.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bbec73640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4bb80cf160 0x7f4bb80cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34262/0 (socket says 172.21.15.46:34262) 2023-10-30T14:05:45.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bbec73640 1 -- 172.21.15.46:0/1213522999 learned_addr learned my addr 172.21.15.46:0/1213522999 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:45.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bbe472640 1 --2- 172.21.15.46:0/1213522999 >> v2:172.21.15.154:3300/0 conn(0x7f4bb81538a0 0x7f4bb815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:45.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bbec73640 1 -- 172.21.15.46:0/1213522999 >> v2:172.21.15.154:3300/0 conn(0x7f4bb81538a0 msgr2=0x7f4bb815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:45.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bbec73640 1 --2- 172.21.15.46:0/1213522999 >> v2:172.21.15.154:3300/0 conn(0x7f4bb81538a0 0x7f4bb815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:45.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bbec73640 1 -- 172.21.15.46:0/1213522999 >> v1:172.21.15.46:6789/0 conn(0x7f4bb80ca0e0 legacy=0x7f4bb815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:45.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.431+0000 7f4bbec73640 1 -- 172.21.15.46:0/1213522999 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb80cad70 con 0x7f4bb80cf160 2023-10-30T14:05:45.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.435+0000 7f4bbec73640 1 --2- 172.21.15.46:0/1213522999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb80cf160 0x7f4bb80cf590 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80cfd10 tx=0x7f4bb001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8254e85b1f740c89 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:45.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.435+0000 7f4bbd470640 1 -- 172.21.15.46:0/1213522999 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb00af400 con 0x7f4bb80cf160 2023-10-30T14:05:45.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.435+0000 7f4bbd470640 1 -- 172.21.15.46:0/1213522999 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb00ab050 con 0x7f4bb80cf160 2023-10-30T14:05:45.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.435+0000 7f4bbd470640 1 -- 172.21.15.46:0/1213522999 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb00af780 con 0x7f4bb80cf160 2023-10-30T14:05:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.435+0000 7f4bc06d6640 1 -- 172.21.15.46:0/1213522999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb80cf160 msgr2=0x7f4bb80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.435+0000 7f4bc06d6640 1 --2- 172.21.15.46:0/1213522999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb80cf160 0x7f4bb80cf590 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80cfd10 tx=0x7f4bb001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:05:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.435+0000 7f4bc06d6640 1 -- 172.21.15.46:0/1213522999 shutdown_connections 2023-10-30T14:05:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.435+0000 7f4bc06d6640 1 --2- 172.21.15.46:0/1213522999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb80cf160 0x7f4bb80cf590 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.435+0000 7f4bc06d6640 1 --2- 172.21.15.46:0/1213522999 >> v2:172.21.15.154:3300/0 conn(0x7f4bb81538a0 0x7f4bb815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.435+0000 7f4bc06d6640 1 -- 172.21.15.46:0/1213522999 >> 172.21.15.46:0/1213522999 conn(0x7f4bb806bfe0 msgr2=0x7f4bb806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.435+0000 7f4bc06d6640 1 -- 172.21.15.46:0/1213522999 shutdown_connections 2023-10-30T14:05:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.435+0000 7f4bc06d6640 1 -- 172.21.15.46:0/1213522999 wait complete. 2023-10-30T14:05:45.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bc06d6640 1 Processor -- start 2023-10-30T14:05:45.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bc06d6640 1 -- start start 2023-10-30T14:05:45.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bc06d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bb80ca0e0 0x7f4bb8159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:45.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bc06d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb80cf160 0x7f4bb81480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:45.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bc06d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bb81538a0 0x7f4bb81ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:45.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bc06d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb8146880 con 0x7f4bb80cf160 2023-10-30T14:05:45.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bc06d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb80b4b60 con 0x7f4bb81538a0 2023-10-30T14:05:45.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bc06d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb81e2ae0 con 0x7f4bb80ca0e0 2023-10-30T14:05:45.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bbdc71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb80cf160 0x7f4bb81480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:45.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bbdc71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb80cf160 0x7f4bb81480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34266/0 (socket says 172.21.15.46:34266) 2023-10-30T14:05:45.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bbdc71640 1 -- 172.21.15.46:0/828259141 learned_addr learned my addr 172.21.15.46:0/828259141 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:45.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bbe472640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bb80ca0e0 0x7f4bb8159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:45.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bbec73640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bb81538a0 0x7f4bb81ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:45.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bbdc71640 1 -- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bb80ca0e0 msgr2=0x7f4bb8159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:45.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bbdc71640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bb80ca0e0 0x7f4bb8159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:45.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bbdc71640 1 -- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bb81538a0 msgr2=0x7f4bb81ef130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:45.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bbdc71640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bb81538a0 0x7f4bb81ef130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:45.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bbdc71640 1 -- 172.21.15.46:0/828259141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb0009b30 con 0x7f4bb80cf160 2023-10-30T14:05:45.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bbdc71640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb80cf160 0x7f4bb81480f0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800c3c0 tx=0x7f4ba800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:45.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4baeffd640 1 -- 172.21.15.46:0/828259141 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba80052d0 con 0x7f4bb80cf160 2023-10-30T14:05:45.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4baeffd640 1 -- 172.21.15.46:0/828259141 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba8005430 con 0x7f4bb80cf160 2023-10-30T14:05:45.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4baeffd640 1 -- 172.21.15.46:0/828259141 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8003840 con 0x7f4bb80cf160 2023-10-30T14:05:45.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bc06d6640 1 -- 172.21.15.46:0/828259141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb8159540 con 0x7f4bb80cf160 2023-10-30T14:05:45.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.439+0000 7f4bc06d6640 1 -- 172.21.15.46:0/828259141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb81e2cf0 con 0x7f4bb80cf160 2023-10-30T14:05:45.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.443+0000 7f4bc06d6640 1 -- 172.21.15.46:0/828259141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb80b2a50 con 0x7f4bb80cf160 2023-10-30T14:05:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.443+0000 7f4baeffd640 1 -- 172.21.15.46:0/828259141 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4ba80039a0 con 0x7f4bb80cf160 2023-10-30T14:05:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.443+0000 7f4baeffd640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4b9c0664a0 0x7f4b9c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.443+0000 7f4baeffd640 1 -- 172.21.15.46:0/828259141 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7f4ba8083aa0 con 0x7f4bb80cf160 2023-10-30T14:05:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.443+0000 7f4bbe472640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4b9c0664a0 0x7f4b9c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:45.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.447+0000 7f4bbe472640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4b9c0664a0 0x7f4b9c068960 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009b60 tx=0x7f4bb4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:45.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.447+0000 7f4baeffd640 1 -- 172.21.15.46:0/828259141 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4ba8050860 con 0x7f4bb80cf160 2023-10-30T14:05:45.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.643+0000 7f4bc06d6640 1 -- 172.21.15.46:0/828259141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bb80b4ca0 con 0x7f4bb80cf160 2023-10-30T14:05:45.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4baeffd640 1 -- 172.21.15.46:0/828259141 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f4bb80b4ca0 con 0x7f4bb80cf160 2023-10-30T14:05:45.648 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:45.648 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:39.493092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:45.649 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 182 2023-10-30T14:05:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4bc06d6640 1 -- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4b9c0664a0 msgr2=0x7f4b9c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4bc06d6640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4b9c0664a0 0x7f4b9c068960 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009b60 tx=0x7f4bb4009190 comp rx=0 tx=0).stop 2023-10-30T14:05:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4bc06d6640 1 -- 172.21.15.46:0/828259141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb80cf160 msgr2=0x7f4bb81480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:45.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4bc06d6640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb80cf160 0x7f4bb81480f0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800c3c0 tx=0x7f4ba800c880 comp rx=0 tx=0).stop 2023-10-30T14:05:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4bc06d6640 1 -- 172.21.15.46:0/828259141 shutdown_connections 2023-10-30T14:05:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4bc06d6640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4b9c0664a0 0x7f4b9c068960 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4bc06d6640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bb81538a0 0x7f4bb81ef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4bc06d6640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb80cf160 0x7f4bb81480f0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:45.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4bc06d6640 1 --2- 172.21.15.46:0/828259141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bb80ca0e0 0x7f4bb8159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4bc06d6640 1 -- 172.21.15.46:0/828259141 >> 172.21.15.46:0/828259141 conn(0x7f4bb806bfe0 msgr2=0x7f4bb806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4bc06d6640 1 -- 172.21.15.46:0/828259141 shutdown_connections 2023-10-30T14:05:45.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:45.647+0000 7f4bc06d6640 1 -- 172.21.15.46:0/828259141 wait complete. 2023-10-30T14:05:45.670 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:05:47.671 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454c3b8640 1 Processor -- start 2023-10-30T14:05:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454c3b8640 1 -- start start 2023-10-30T14:05:47.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454c3b8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f45441538a0 0x7f454415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:47.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454c3b8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45440cf160 0x7f45440cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:47.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454c3b8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45440cb800 con 0x7f45440ca0e0 2023-10-30T14:05:47.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454c3b8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f454415dc30 con 0x7f45440cf160 2023-10-30T14:05:47.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454c3b8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45440cb330 con 0x7f45441538a0 2023-10-30T14:05:47.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454a955640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45440cf160 0x7f45440cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:47.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f4549953640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f45440ca0e0 0x7f454415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54282/0 (socket says 172.21.15.46:54282) 2023-10-30T14:05:47.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f4549953640 1 -- 172.21.15.46:0/3782664945 learned_addr learned my addr 172.21.15.46:0/3782664945 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:05:47.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454a154640 1 --2- 172.21.15.46:0/3782664945 >> v2:172.21.15.154:3300/0 conn(0x7f45441538a0 0x7f454415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:47.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454a955640 1 -- 172.21.15.46:0/3782664945 >> v2:172.21.15.154:3300/0 conn(0x7f45441538a0 msgr2=0x7f454415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454a955640 1 --2- 172.21.15.46:0/3782664945 >> v2:172.21.15.154:3300/0 conn(0x7f45441538a0 0x7f454415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454a955640 1 -- 172.21.15.46:0/3782664945 >> v1:172.21.15.46:6789/0 conn(0x7f45440ca0e0 legacy=0x7f454415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.823+0000 7f454a955640 1 -- 172.21.15.46:0/3782664945 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45440cad70 con 0x7f45440cf160 2023-10-30T14:05:47.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.827+0000 7f454a955640 1 --2- 172.21.15.46:0/3782664945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440cf160 0x7f45440cf590 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f45440cfd10 tx=0x7f453c01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc22ab689fbf1586 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.827+0000 7f4549152640 1 -- 172.21.15.46:0/3782664945 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c0b0440 con 0x7f45440cf160 2023-10-30T14:05:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.827+0000 7f4549152640 1 -- 172.21.15.46:0/3782664945 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453c0b05a0 con 0x7f45440cf160 2023-10-30T14:05:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.827+0000 7f4549152640 1 -- 172.21.15.46:0/3782664945 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c0b0930 con 0x7f45440cf160 2023-10-30T14:05:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 -- 172.21.15.46:0/3782664945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440cf160 msgr2=0x7f45440cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 --2- 172.21.15.46:0/3782664945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440cf160 0x7f45440cf590 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f45440cfd10 tx=0x7f453c01f0f0 comp rx=0 tx=0).stop 2023-10-30T14:05:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 -- 172.21.15.46:0/3782664945 shutdown_connections 2023-10-30T14:05:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 --2- 172.21.15.46:0/3782664945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440cf160 0x7f45440cf590 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 --2- 172.21.15.46:0/3782664945 >> v2:172.21.15.154:3300/0 conn(0x7f45441538a0 0x7f454415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 -- 172.21.15.46:0/3782664945 >> 172.21.15.46:0/3782664945 conn(0x7f454406bfe0 msgr2=0x7f454406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 -- 172.21.15.46:0/3782664945 shutdown_connections 2023-10-30T14:05:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 -- 172.21.15.46:0/3782664945 wait complete. 2023-10-30T14:05:47.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 Processor -- start 2023-10-30T14:05:47.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 -- start start 2023-10-30T14:05:47.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 0x7f4544159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45440cf160 0x7f45441480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45441538a0 0x7f45441ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4544146880 con 0x7f45440ca0e0 2023-10-30T14:05:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45440b4b60 con 0x7f45440cf160 2023-10-30T14:05:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454c3b8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45441e2ae0 con 0x7f45441538a0 2023-10-30T14:05:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454a154640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 0x7f4544159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454a154640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 0x7f4544159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52282/0 (socket says 172.21.15.46:52282) 2023-10-30T14:05:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454a154640 1 -- 172.21.15.46:0/2747123860 learned_addr learned my addr 172.21.15.46:0/2747123860 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f4549953640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45440cf160 0x7f45441480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.831+0000 7f454a955640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45441538a0 0x7f45441ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f454a154640 1 -- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45441538a0 msgr2=0x7f45441ef130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:47.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f454a154640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45441538a0 0x7f45441ef130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:47.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f454a154640 1 -- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45440cf160 msgr2=0x7f45441480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:47.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f454a154640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45440cf160 0x7f45441480f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:47.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f454a154640 1 -- 172.21.15.46:0/2747123860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c009a70 con 0x7f45440ca0e0 2023-10-30T14:05:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f454a154640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 0x7f4544159000 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f454000d800 tx=0x7f454000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f453affd640 1 -- 172.21.15.46:0/2747123860 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454000b660 con 0x7f45440ca0e0 2023-10-30T14:05:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f453affd640 1 -- 172.21.15.46:0/2747123860 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454000b7c0 con 0x7f45440ca0e0 2023-10-30T14:05:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f454c3b8640 1 -- 172.21.15.46:0/2747123860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4544159540 con 0x7f45440ca0e0 2023-10-30T14:05:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f453affd640 1 -- 172.21.15.46:0/2747123860 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454000bb30 con 0x7f45440ca0e0 2023-10-30T14:05:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f454c3b8640 1 -- 172.21.15.46:0/2747123860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45441e2cf0 con 0x7f45440ca0e0 2023-10-30T14:05:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f454c3b8640 1 -- 172.21.15.46:0/2747123860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45440b2a50 con 0x7f45440ca0e0 2023-10-30T14:05:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.835+0000 7f453affd640 1 -- 172.21.15.46:0/2747123860 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f454000bc90 con 0x7f45440ca0e0 2023-10-30T14:05:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.839+0000 7f453affd640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45280664a0 0x7f4528068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.839+0000 7f453affd640 1 -- 172.21.15.46:0/2747123860 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7f4540084bf0 con 0x7f45440ca0e0 2023-10-30T14:05:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.839+0000 7f4549953640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45280664a0 0x7f4528068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.839+0000 7f4549953640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45280664a0 0x7f4528068960 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f45441e44e0 tx=0x7f4534009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:47.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:47.839+0000 7f453affd640 1 -- 172.21.15.46:0/2747123860 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4540051860 con 0x7f45440ca0e0 2023-10-30T14:05:48.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.051+0000 7f454c3b8640 1 -- 172.21.15.46:0/2747123860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4544159b40 con 0x7f45440ca0e0 2023-10-30T14:05:48.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f453affd640 1 -- 172.21.15.46:0/2747123860 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f4540011340 con 0x7f45440ca0e0 2023-10-30T14:05:48.056 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:48.056 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:39.493092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 182 2023-10-30T14:05:48.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f454c3b8640 1 -- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45280664a0 msgr2=0x7f4528068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:48.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f454c3b8640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45280664a0 0x7f4528068960 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f45441e44e0 tx=0x7f4534009040 comp rx=0 tx=0).stop 2023-10-30T14:05:48.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f454c3b8640 1 -- 172.21.15.46:0/2747123860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 msgr2=0x7f4544159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:48.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f454c3b8640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 0x7f4544159000 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f454000d800 tx=0x7f454000dcc0 comp rx=0 tx=0).stop 2023-10-30T14:05:48.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f454c3b8640 1 -- 172.21.15.46:0/2747123860 shutdown_connections 2023-10-30T14:05:48.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f454c3b8640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45280664a0 0x7f4528068960 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:48.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f454c3b8640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45441538a0 0x7f45441ef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:48.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f454c3b8640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45440cf160 0x7f45441480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:48.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f454c3b8640 1 --2- 172.21.15.46:0/2747123860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45440ca0e0 0x7f4544159000 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:48.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f454c3b8640 1 -- 172.21.15.46:0/2747123860 >> 172.21.15.46:0/2747123860 conn(0x7f454406bfe0 msgr2=0x7f454406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:48.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f454c3b8640 1 -- 172.21.15.46:0/2747123860 shutdown_connections 2023-10-30T14:05:48.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:48.055+0000 7f454c3b8640 1 -- 172.21.15.46:0/2747123860 wait complete. 2023-10-30T14:05:48.078 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:05:50.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:50.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.223+0000 7f3336165640 1 Processor -- start 2023-10-30T14:05:50.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.223+0000 7f3336165640 1 -- start start 2023-10-30T14:05:50.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.223+0000 7f3336165640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3330153bf0 0x7f3330071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:50.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.223+0000 7f3336165640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33300c6fe0 0x7f33300c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:50.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.223+0000 7f3336165640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33300700b0 con 0x7f33300c79e0 2023-10-30T14:05:50.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.223+0000 7f3336165640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33300c6810 con 0x7f33300c6fe0 2023-10-30T14:05:50.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.223+0000 7f3336165640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3330159900 con 0x7f3330153bf0 2023-10-30T14:05:50.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.223+0000 7f332effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33300c6fe0 0x7f33300c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:50.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.223+0000 7f332effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33300c6fe0 0x7f33300c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52292/0 (socket says 172.21.15.46:52292) 2023-10-30T14:05:50.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.223+0000 7f332effd640 1 -- 172.21.15.46:0/2988823296 learned_addr learned my addr 172.21.15.46:0/2988823296 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:50.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.223+0000 7f332f7fe640 1 --2- 172.21.15.46:0/2988823296 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3330153bf0 0x7f3330071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:50.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f332effd640 1 -- 172.21.15.46:0/2988823296 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3330153bf0 msgr2=0x7f3330071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:50.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f332effd640 1 --2- 172.21.15.46:0/2988823296 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3330153bf0 0x7f3330071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:50.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f332effd640 1 -- 172.21.15.46:0/2988823296 >> v1:172.21.15.154:6789/0 conn(0x7f33300c79e0 legacy=0x7f33300cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:50.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f332effd640 1 -- 172.21.15.46:0/2988823296 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33300c6530 con 0x7f33300c6fe0 2023-10-30T14:05:50.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f332f7fe640 1 --2- 172.21.15.46:0/2988823296 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3330153bf0 0x7f3330071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:50.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f332effd640 1 --2- 172.21.15.46:0/2988823296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33300c6fe0 0x7f33300c74a0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f3324009870 tx=0x7f332401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b564ec651642dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:50.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f332e7fc640 1 -- 172.21.15.46:0/2988823296 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33240a8a30 con 0x7f33300c6fe0 2023-10-30T14:05:50.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f332e7fc640 1 -- 172.21.15.46:0/2988823296 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33240a8b90 con 0x7f33300c6fe0 2023-10-30T14:05:50.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f332e7fc640 1 -- 172.21.15.46:0/2988823296 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33240b0440 con 0x7f33300c6fe0 2023-10-30T14:05:50.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f3336165640 1 -- 172.21.15.46:0/2988823296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33300c6fe0 msgr2=0x7f33300c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:50.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f3336165640 1 --2- 172.21.15.46:0/2988823296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33300c6fe0 0x7f33300c74a0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f3324009870 tx=0x7f332401ed30 comp rx=0 tx=0).stop 2023-10-30T14:05:50.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f3336165640 1 -- 172.21.15.46:0/2988823296 shutdown_connections 2023-10-30T14:05:50.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f3336165640 1 --2- 172.21.15.46:0/2988823296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33300c6fe0 0x7f33300c74a0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:50.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f3336165640 1 --2- 172.21.15.46:0/2988823296 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3330153bf0 0x7f3330071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:50.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f3336165640 1 -- 172.21.15.46:0/2988823296 >> 172.21.15.46:0/2988823296 conn(0x7f333006c6e0 msgr2=0x7f3330069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:50.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f3336165640 1 -- 172.21.15.46:0/2988823296 shutdown_connections 2023-10-30T14:05:50.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.227+0000 7f3336165640 1 -- 172.21.15.46:0/2988823296 wait complete. 2023-10-30T14:05:50.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f3336165640 1 Processor -- start 2023-10-30T14:05:50.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f3336165640 1 -- start start 2023-10-30T14:05:50.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f3336165640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33300c6fe0 0x7f333014a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:50.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f3336165640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33300c79e0 0x7f3330146ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:50.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f3336165640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3330153bf0 0x7f3330147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:50.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f3336165640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3330145300 con 0x7f33300c6fe0 2023-10-30T14:05:50.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f3336165640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333014a070 con 0x7f3330153bf0 2023-10-30T14:05:50.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f3336165640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3330145020 con 0x7f33300c79e0 2023-10-30T14:05:50.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f332f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33300c6fe0 0x7f333014a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:50.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f332f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33300c6fe0 0x7f333014a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52302/0 (socket says 172.21.15.46:52302) 2023-10-30T14:05:50.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f332f7fe640 1 -- 172.21.15.46:0/4251215879 learned_addr learned my addr 172.21.15.46:0/4251215879 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:50.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f332effd640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33300c79e0 0x7f3330146ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:50.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f332ffff640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3330153bf0 0x7f3330147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:50.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f332f7fe640 1 -- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33300c79e0 msgr2=0x7f3330146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:50.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f332f7fe640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33300c79e0 0x7f3330146ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:50.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f332f7fe640 1 -- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3330153bf0 msgr2=0x7f3330147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:50.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f332f7fe640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3330153bf0 0x7f3330147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:50.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f332f7fe640 1 -- 172.21.15.46:0/4251215879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33240094d0 con 0x7f33300c6fe0 2023-10-30T14:05:50.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f332f7fe640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33300c6fe0 0x7f333014a790 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f332000d800 tx=0x7f332000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:50.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f3317fff640 1 -- 172.21.15.46:0/4251215879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3320009660 con 0x7f33300c6fe0 2023-10-30T14:05:50.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f3317fff640 1 -- 172.21.15.46:0/4251215879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33200097c0 con 0x7f33300c6fe0 2023-10-30T14:05:50.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f3317fff640 1 -- 172.21.15.46:0/4251215879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3320009b70 con 0x7f33300c6fe0 2023-10-30T14:05:50.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.231+0000 7f3336165640 1 -- 172.21.15.46:0/4251215879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3330149740 con 0x7f33300c6fe0 2023-10-30T14:05:50.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.235+0000 7f3336165640 1 -- 172.21.15.46:0/4251215879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3330149d00 con 0x7f33300c6fe0 2023-10-30T14:05:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.235+0000 7f3336165640 1 -- 172.21.15.46:0/4251215879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f333004e170 con 0x7f33300c6fe0 2023-10-30T14:05:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.235+0000 7f3317fff640 1 -- 172.21.15.46:0/4251215879 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3320009cd0 con 0x7f33300c6fe0 2023-10-30T14:05:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.235+0000 7f3317fff640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f331c0664a0 0x7f331c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:50.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.235+0000 7f3317fff640 1 -- 172.21.15.46:0/4251215879 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7f3320084c70 con 0x7f33300c6fe0 2023-10-30T14:05:50.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.235+0000 7f332effd640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f331c0664a0 0x7f331c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:50.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.239+0000 7f332effd640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f331c0664a0 0x7f331c068960 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f33240b2000 tx=0x7f3324009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:50.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.239+0000 7f3317fff640 1 -- 172.21.15.46:0/4251215879 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3320051990 con 0x7f33300c6fe0 2023-10-30T14:05:50.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 -- 172.21.15.46:0/4251215879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f333015a0a0 con 0x7f33300c6fe0 2023-10-30T14:05:50.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3317fff640 1 -- 172.21.15.46:0/4251215879 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f3320054fb0 con 0x7f33300c6fe0 2023-10-30T14:05:50.433 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:50.433 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:49.142172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:resolve","state_seq":15,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:50.433 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 183 2023-10-30T14:05:50.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 -- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f331c0664a0 msgr2=0x7f331c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:50.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f331c0664a0 0x7f331c068960 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f33240b2000 tx=0x7f3324009040 comp rx=0 tx=0).stop 2023-10-30T14:05:50.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 -- 172.21.15.46:0/4251215879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33300c6fe0 msgr2=0x7f333014a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:50.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33300c6fe0 0x7f333014a790 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f332000d800 tx=0x7f332000dcc0 comp rx=0 tx=0).stop 2023-10-30T14:05:50.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 -- 172.21.15.46:0/4251215879 shutdown_connections 2023-10-30T14:05:50.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f331c0664a0 0x7f331c068960 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:50.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3330153bf0 0x7f3330147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:50.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33300c79e0 0x7f3330146ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:50.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 --2- 172.21.15.46:0/4251215879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33300c6fe0 0x7f333014a790 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:50.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 -- 172.21.15.46:0/4251215879 >> 172.21.15.46:0/4251215879 conn(0x7f333006c6e0 msgr2=0x7f3330159020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:50.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 -- 172.21.15.46:0/4251215879 shutdown_connections 2023-10-30T14:05:50.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:50.431+0000 7f3336165640 1 -- 172.21.15.46:0/4251215879 wait complete. 2023-10-30T14:05:50.454 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:05:52.456 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.607+0000 7fba8c609640 1 Processor -- start 2023-10-30T14:05:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.607+0000 7fba8c609640 1 -- start start 2023-10-30T14:05:52.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba8c609640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba840c6b80 0x7fba840bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:52.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba8c609640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840c0a50 con 0x7fba840c6f60 2023-10-30T14:05:52.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba8c609640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8406f6e0 con 0x7fba8413f4a0 2023-10-30T14:05:52.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba8c609640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8406f9c0 con 0x7fba840c6b80 2023-10-30T14:05:52.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba89ba4640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fba840c6f60 0x7fba8413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54296/0 (socket says 172.21.15.46:54296) 2023-10-30T14:05:52.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba89ba4640 1 -- 172.21.15.46:0/3204647516 learned_addr learned my addr 172.21.15.46:0/3204647516 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:05:52.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba8a3a5640 1 --2- 172.21.15.46:0/3204647516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba840c6b80 0x7fba840bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:52.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503079369 0 0) 0x7fba840c0a50 con 0x7fba840c6f60 2023-10-30T14:05:52.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba70005740 con 0x7fba840c6f60 2023-10-30T14:05:52.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3444138244 0 0) 0x7fba70005740 con 0x7fba840c6f60 2023-10-30T14:05:52.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba840c6b80 msgr2=0x7fba840bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:52.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 --2- 172.21.15.46:0/3204647516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba840c6b80 0x7fba840bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:52.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 >> v1:172.21.15.154:6789/0 conn(0x7fba8413f4a0 legacy=0x7fba841566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:52.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8406fca0 con 0x7fba840c6f60 2023-10-30T14:05:52.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fba74001680 con 0x7fba840c6f60 2023-10-30T14:05:52.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 >> v1:172.21.15.46:6789/0 conn(0x7fba840c6f60 legacy=0x7fba8413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:52.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 --2- 172.21.15.46:0/3204647516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba840c6b80 0x7fba70005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:52.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 --2- 172.21.15.46:0/3204647516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba70006140 0x7fba700069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:52.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 --2- 172.21.15.46:0/3204647516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba70006f20 0x7fba700073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:52.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8406f9c0 con 0x7fba70006140 2023-10-30T14:05:52.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7000dc80 con 0x7fba70006f20 2023-10-30T14:05:52.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7000de40 con 0x7fba840c6b80 2023-10-30T14:05:52.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba89ba4640 1 --2- 172.21.15.46:0/3204647516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba70006140 0x7fba700069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:52.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba8a3a5640 1 --2- 172.21.15.46:0/3204647516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba840c6b80 0x7fba70005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:52.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba8aba6640 1 --2- 172.21.15.46:0/3204647516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba70006f20 0x7fba700073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:52.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba89ba4640 1 -- 172.21.15.46:0/3204647516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba840c6b80 msgr2=0x7fba70005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:52.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba89ba4640 1 --2- 172.21.15.46:0/3204647516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba840c6b80 0x7fba70005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:52.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba89ba4640 1 -- 172.21.15.46:0/3204647516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba70006f20 msgr2=0x7fba700073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:52.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba89ba4640 1 --2- 172.21.15.46:0/3204647516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba70006f20 0x7fba700073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:52.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.611+0000 7fba89ba4640 1 -- 172.21.15.46:0/3204647516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba70005740 con 0x7fba70006140 2023-10-30T14:05:52.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.615+0000 7fba89ba4640 1 --2- 172.21.15.46:0/3204647516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba70006140 0x7fba700069e0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fba7400c210 tx=0x7fba7401f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1e59628683e131 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:52.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.615+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74005ae0 con 0x7fba70006140 2023-10-30T14:05:52.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.615+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba740ab070 con 0x7fba70006140 2023-10-30T14:05:52.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.615+0000 7fba893a3640 1 -- 172.21.15.46:0/3204647516 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74007070 con 0x7fba70006140 2023-10-30T14:05:52.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.615+0000 7fba8c609640 1 -- 172.21.15.46:0/3204647516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba70006140 msgr2=0x7fba700069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:52.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.615+0000 7fba8c609640 1 --2- 172.21.15.46:0/3204647516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba70006140 0x7fba700069e0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fba7400c210 tx=0x7fba7401f370 comp rx=0 tx=0).stop 2023-10-30T14:05:52.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.615+0000 7fba8aba6640 1 -- 172.21.15.46:0/3204647516 reap_dead start 2023-10-30T14:05:52.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.615+0000 7fba8c609640 1 -- 172.21.15.46:0/3204647516 shutdown_connections 2023-10-30T14:05:52.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.615+0000 7fba8c609640 1 -- 172.21.15.46:0/3204647516 >> 172.21.15.46:0/3204647516 conn(0x7fba840c7910 msgr2=0x7fba840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:52.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.615+0000 7fba8c609640 1 -- 172.21.15.46:0/3204647516 shutdown_connections 2023-10-30T14:05:52.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.615+0000 7fba8c609640 1 -- 172.21.15.46:0/3204647516 wait complete. 2023-10-30T14:05:52.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8c609640 1 Processor -- start 2023-10-30T14:05:52.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8c609640 1 -- start start 2023-10-30T14:05:52.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8c609640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba70006140 0x7fba841556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:52.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8c609640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba84155c20 0x7fba840ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:52.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8c609640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba8413c0e0 0x7fba840ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:52.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8c609640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba841ee9c0 con 0x7fba70006140 2023-10-30T14:05:52.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8c609640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba840b0710 con 0x7fba8413c0e0 2023-10-30T14:05:52.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8c609640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba840b0470 con 0x7fba84155c20 2023-10-30T14:05:52.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8a3a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba70006140 0x7fba841556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:52.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8a3a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba70006140 0x7fba841556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52320/0 (socket says 172.21.15.46:52320) 2023-10-30T14:05:52.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8a3a5640 1 -- 172.21.15.46:0/724737636 learned_addr learned my addr 172.21.15.46:0/724737636 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:52.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba89ba4640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba84155c20 0x7fba840ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:52.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8aba6640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba8413c0e0 0x7fba840ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:52.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8a3a5640 1 -- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba84155c20 msgr2=0x7fba840ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:52.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8a3a5640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba84155c20 0x7fba840ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:52.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8a3a5640 1 -- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba8413c0e0 msgr2=0x7fba840ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:52.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8a3a5640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba8413c0e0 0x7fba840ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:52.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8a3a5640 1 -- 172.21.15.46:0/724737636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba84156700 con 0x7fba70006140 2023-10-30T14:05:52.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8a3a5640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba70006140 0x7fba841556e0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fba80011a00 tx=0x7fba80011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:52.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba7affd640 1 -- 172.21.15.46:0/724737636 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8000faf0 con 0x7fba70006140 2023-10-30T14:05:52.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba7affd640 1 -- 172.21.15.46:0/724737636 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8000fc50 con 0x7fba70006140 2023-10-30T14:05:52.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba7affd640 1 -- 172.21.15.46:0/724737636 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba80014440 con 0x7fba70006140 2023-10-30T14:05:52.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8c609640 1 -- 172.21.15.46:0/724737636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba841eebd0 con 0x7fba70006140 2023-10-30T14:05:52.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.619+0000 7fba8c609640 1 -- 172.21.15.46:0/724737636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba841ef190 con 0x7fba70006140 2023-10-30T14:05:52.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.623+0000 7fba8c609640 1 -- 172.21.15.46:0/724737636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8404e170 con 0x7fba70006140 2023-10-30T14:05:52.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.623+0000 7fba7affd640 1 -- 172.21.15.46:0/724737636 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fba800145a0 con 0x7fba70006140 2023-10-30T14:05:52.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.623+0000 7fba7affd640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fba680664a0 0x7fba68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:52.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.623+0000 7fba7affd640 1 -- 172.21.15.46:0/724737636 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7fba80084fe0 con 0x7fba70006140 2023-10-30T14:05:52.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.623+0000 7fba89ba4640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fba680664a0 0x7fba68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:52.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.623+0000 7fba89ba4640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fba680664a0 0x7fba68068960 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fba740af000 tx=0x7fba740aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:52.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.627+0000 7fba7affd640 1 -- 172.21.15.46:0/724737636 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fba80051c30 con 0x7fba70006140 2023-10-30T14:05:52.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.815+0000 7fba8c609640 1 -- 172.21.15.46:0/724737636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba8413f280 con 0x7fba70006140 2023-10-30T14:05:52.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba7affd640 1 -- 172.21.15.46:0/724737636 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+5731 (secure 0 0 0) 0x7fba80022070 con 0x7fba70006140 2023-10-30T14:05:52.820 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:52.820 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:49.142172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:resolve","state_seq":15,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:52.820 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 183 2023-10-30T14:05:52.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba8c609640 1 -- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fba680664a0 msgr2=0x7fba68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:52.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba8c609640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fba680664a0 0x7fba68068960 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fba740af000 tx=0x7fba740aa000 comp rx=0 tx=0).stop 2023-10-30T14:05:52.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba8c609640 1 -- 172.21.15.46:0/724737636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba70006140 msgr2=0x7fba841556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:52.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba8c609640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba70006140 0x7fba841556e0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fba80011a00 tx=0x7fba80011ec0 comp rx=0 tx=0).stop 2023-10-30T14:05:52.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba8c609640 1 -- 172.21.15.46:0/724737636 shutdown_connections 2023-10-30T14:05:52.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba8c609640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fba680664a0 0x7fba68068960 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:52.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba8c609640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba8413c0e0 0x7fba840ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:52.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba8c609640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba84155c20 0x7fba840ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:52.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba8c609640 1 --2- 172.21.15.46:0/724737636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba70006140 0x7fba841556e0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba8c609640 1 -- 172.21.15.46:0/724737636 >> 172.21.15.46:0/724737636 conn(0x7fba840c7910 msgr2=0x7fba8413af10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba8c609640 1 -- 172.21.15.46:0/724737636 shutdown_connections 2023-10-30T14:05:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:52.819+0000 7fba8c609640 1 -- 172.21.15.46:0/724737636 wait complete. 2023-10-30T14:05:52.841 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 183, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 183, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8190": { "addr": "172.21.15.46:6837/3154171789", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3154171789, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3154171789, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 8190, "incarnation": 175, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 40 }, "gid_8352": { "addr": "172.21.15.46:6836/128865315", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 128865315, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 128865315, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8352, "incarnation": 182, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:resolve", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:05:49.142172+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8190, "mds_1": 8352 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:05:54.843 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:54.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7ffc6b640 1 Processor -- start 2023-10-30T14:05:54.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7ffc6b640 1 -- start start 2023-10-30T14:05:54.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7ffc6b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc7f806f7a0 0x7fc7f806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:54.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7ffc6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f80c6bd0 0x7fc7f8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7ffc6b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7f8071110 0x7fc7f8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7ffc6b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f80700c0 con 0x7fc7f8071110 2023-10-30T14:05:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7ffc6b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f806a3d0 con 0x7fc7f806f7a0 2023-10-30T14:05:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7ffc6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f8066cf0 con 0x7fc7f80c6bd0 2023-10-30T14:05:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7fe208640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7f8071110 0x7fc7f8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7fe208640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7f8071110 0x7fc7f8158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52336/0 (socket says 172.21.15.46:52336) 2023-10-30T14:05:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7fe208640 1 -- 172.21.15.46:0/80951538 learned_addr learned my addr 172.21.15.46:0/80951538 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7fd206640 1 --2- 172.21.15.46:0/80951538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f80c6bd0 0x7fc7f8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7fda07640 1 --2- 172.21.15.46:0/80951538 >> v2:172.21.15.154:3300/0 conn(0x7fc7f806f7a0 0x7fc7f806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7fe208640 1 -- 172.21.15.46:0/80951538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f80c6bd0 msgr2=0x7fc7f8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:54.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7fe208640 1 --2- 172.21.15.46:0/80951538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f80c6bd0 0x7fc7f8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7fe208640 1 -- 172.21.15.46:0/80951538 >> v2:172.21.15.154:3300/0 conn(0x7fc7f806f7a0 msgr2=0x7fc7f806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:54.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7fe208640 1 --2- 172.21.15.46:0/80951538 >> v2:172.21.15.154:3300/0 conn(0x7fc7f806f7a0 0x7fc7f806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.991+0000 7fc7fe208640 1 -- 172.21.15.46:0/80951538 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f80c70d0 con 0x7fc7f8071110 2023-10-30T14:05:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7fe208640 1 --2- 172.21.15.46:0/80951538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8071110 0x7fc7f8158110 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fc7f806c830 tx=0x7fc7f001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=745bf358301c60af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7fca05640 1 -- 172.21.15.46:0/80951538 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00af400 con 0x7fc7f8071110 2023-10-30T14:05:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7fca05640 1 -- 172.21.15.46:0/80951538 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f00ab050 con 0x7fc7f8071110 2023-10-30T14:05:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7fca05640 1 -- 172.21.15.46:0/80951538 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00af780 con 0x7fc7f8071110 2023-10-30T14:05:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/80951538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8071110 msgr2=0x7fc7f8158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:55.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7ffc6b640 1 --2- 172.21.15.46:0/80951538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8071110 0x7fc7f8158110 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fc7f806c830 tx=0x7fc7f001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:05:55.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/80951538 shutdown_connections 2023-10-30T14:05:55.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7ffc6b640 1 --2- 172.21.15.46:0/80951538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8071110 0x7fc7f8158110 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:55.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7ffc6b640 1 --2- 172.21.15.46:0/80951538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f80c6bd0 0x7fc7f8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:55.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7ffc6b640 1 --2- 172.21.15.46:0/80951538 >> v2:172.21.15.154:3300/0 conn(0x7fc7f806f7a0 0x7fc7f806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:55.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/80951538 >> 172.21.15.46:0/80951538 conn(0x7fc7f80c7910 msgr2=0x7fc7f80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:55.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/80951538 shutdown_connections 2023-10-30T14:05:55.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.995+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/80951538 wait complete. 2023-10-30T14:05:55.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ffc6b640 1 Processor -- start 2023-10-30T14:05:55.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ffc6b640 1 -- start start 2023-10-30T14:05:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ffc6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f806f7a0 0x7fc7f8149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ffc6b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f8071110 0x7fc7f80c14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ffc6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f80c6bd0 0x7fc7f80c19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ffc6b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f81eed70 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ffc6b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f814a000 con 0x7fc7f8071110 2023-10-30T14:05:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ffc6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f80afda0 con 0x7fc7f806f7a0 2023-10-30T14:05:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7fe208640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f80c6bd0 0x7fc7f80c19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7fe208640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f80c6bd0 0x7fc7f80c19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52346/0 (socket says 172.21.15.46:52346) 2023-10-30T14:05:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7fe208640 1 -- 172.21.15.46:0/851735510 learned_addr learned my addr 172.21.15.46:0/851735510 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7fd206640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f8071110 0x7fc7f80c14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:55.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7fda07640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f806f7a0 0x7fc7f8149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:55.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7fe208640 1 -- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f806f7a0 msgr2=0x7fc7f8149ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:55.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7fe208640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f806f7a0 0x7fc7f8149ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:55.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7fe208640 1 -- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f8071110 msgr2=0x7fc7f80c14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:55.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7fe208640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f8071110 0x7fc7f80c14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:55.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7fe208640 1 -- 172.21.15.46:0/851735510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f0009b30 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7fe208640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f80c6bd0 0x7fc7f80c19e0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00b2040 tx=0x7fc7f001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:55.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ee7fc640 1 -- 172.21.15.46:0/851735510 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00afc00 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ee7fc640 1 -- 172.21.15.46:0/851735510 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f00afd80 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/851735510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f81eefe0 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ee7fc640 1 -- 172.21.15.46:0/851735510 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00ba660 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:54.999+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/851735510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f81ef5a0 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.003+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/851735510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f804e170 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.003+0000 7fc7ee7fc640 1 -- 172.21.15.46:0/851735510 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7f00b6040 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.003+0000 7fc7ee7fc640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7dc0664d0 0x7fc7dc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:55.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.003+0000 7fc7ee7fc640 1 -- 172.21.15.46:0/851735510 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7fc7f01225e0 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.003+0000 7fc7fda07640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7dc0664d0 0x7fc7dc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.003+0000 7fc7fda07640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7dc0664d0 0x7fc7dc068990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4009a40 tx=0x7fc7f4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.007+0000 7fc7ee7fc640 1 -- 172.21.15.46:0/851735510 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7f00ef2f0 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/851735510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7f80c9c60 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ee7fc640 1 -- 172.21.15.46:0/851735510 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+5731 (secure 0 0 0) 0x7fc7f0009510 con 0x7fc7f80c6bd0 2023-10-30T14:05:55.201 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:55.202 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:49.142172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:resolve","state_seq":15,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:55.203 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 183 2023-10-30T14:05:55.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7dc0664d0 msgr2=0x7fc7dc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:55.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ffc6b640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7dc0664d0 0x7fc7dc068990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4009a40 tx=0x7fc7f4009190 comp rx=0 tx=0).stop 2023-10-30T14:05:55.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/851735510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f80c6bd0 msgr2=0x7fc7f80c19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:55.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ffc6b640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f80c6bd0 0x7fc7f80c19e0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00b2040 tx=0x7fc7f001bf60 comp rx=0 tx=0).stop 2023-10-30T14:05:55.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/851735510 shutdown_connections 2023-10-30T14:05:55.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ffc6b640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7dc0664d0 0x7fc7dc068990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:55.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ffc6b640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f80c6bd0 0x7fc7f80c19e0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:55.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ffc6b640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f8071110 0x7fc7f80c14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:55.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ffc6b640 1 --2- 172.21.15.46:0/851735510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7f806f7a0 0x7fc7f8149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:55.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/851735510 >> 172.21.15.46:0/851735510 conn(0x7fc7f80c7910 msgr2=0x7fc7f81501a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:55.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.199+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/851735510 shutdown_connections 2023-10-30T14:05:55.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:55.203+0000 7fc7ffc6b640 1 -- 172.21.15.46:0/851735510 wait complete. 2023-10-30T14:05:55.222 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:05:55.493 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:05:55.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:05:57.223 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:57.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.375+0000 7f8cbfcd6640 1 Processor -- start 2023-10-30T14:05:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.375+0000 7f8cbfcd6640 1 -- start start 2023-10-30T14:05:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.375+0000 7f8cbfcd6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8cb8153bf0 0x7f8cb8071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.375+0000 7f8cbfcd6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8cb80c6fe0 0x7f8cb80c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.375+0000 7f8cbfcd6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80c79e0 0x7f8cb80cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbfcd6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb80c6810 con 0x7f8cb8153bf0 2023-10-30T14:05:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbfcd6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb8159900 con 0x7f8cb80c6fe0 2023-10-30T14:05:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbfcd6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80700b0 con 0x7f8cb80c79e0 2023-10-30T14:05:57.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbda72640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8cb8153bf0 0x7f8cb8071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:57.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbda72640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8cb8153bf0 0x7f8cb8071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52322/0 (socket says 172.21.15.46:52322) 2023-10-30T14:05:57.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbda72640 1 -- 172.21.15.46:0/178153993 learned_addr learned my addr 172.21.15.46:0/178153993 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:57.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbe273640 1 --2- 172.21.15.46:0/178153993 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80c79e0 0x7f8cb80cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:57.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbd271640 1 --2- 172.21.15.46:0/178153993 >> v2:172.21.15.154:3300/0 conn(0x7f8cb80c6fe0 0x7f8cb80c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:57.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbda72640 1 -- 172.21.15.46:0/178153993 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80c79e0 msgr2=0x7f8cb80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:57.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbda72640 1 --2- 172.21.15.46:0/178153993 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80c79e0 0x7f8cb80cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:57.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbda72640 1 -- 172.21.15.46:0/178153993 >> v2:172.21.15.154:3300/0 conn(0x7f8cb80c6fe0 msgr2=0x7f8cb80c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:57.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbda72640 1 --2- 172.21.15.46:0/178153993 >> v2:172.21.15.154:3300/0 conn(0x7f8cb80c6fe0 0x7f8cb80c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:57.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbda72640 1 -- 172.21.15.46:0/178153993 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80c6530 con 0x7f8cb8153bf0 2023-10-30T14:05:57.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbd271640 1 --2- 172.21.15.46:0/178153993 >> v2:172.21.15.154:3300/0 conn(0x7f8cb80c6fe0 0x7f8cb80c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:05:57.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbda72640 1 --2- 172.21.15.46:0/178153993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb8153bf0 0x7f8cb8071260 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80723d0 tx=0x7f8cb401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19936613d20600c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:57.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.379+0000 7f8cbca70640 1 -- 172.21.15.46:0/178153993 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb40a89b0 con 0x7f8cb8153bf0 2023-10-30T14:05:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbca70640 1 -- 172.21.15.46:0/178153993 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb40a8b10 con 0x7f8cb8153bf0 2023-10-30T14:05:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbca70640 1 -- 172.21.15.46:0/178153993 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb40a8ea0 con 0x7f8cb8153bf0 2023-10-30T14:05:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/178153993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb8153bf0 msgr2=0x7f8cb8071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 --2- 172.21.15.46:0/178153993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb8153bf0 0x7f8cb8071260 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80723d0 tx=0x7f8cb401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:05:57.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/178153993 shutdown_connections 2023-10-30T14:05:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 --2- 172.21.15.46:0/178153993 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80c79e0 0x7f8cb80cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 --2- 172.21.15.46:0/178153993 >> v2:172.21.15.154:3300/0 conn(0x7f8cb80c6fe0 0x7f8cb80c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 --2- 172.21.15.46:0/178153993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb8153bf0 0x7f8cb8071260 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/178153993 >> 172.21.15.46:0/178153993 conn(0x7f8cb806c6e0 msgr2=0x7f8cb8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:57.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/178153993 shutdown_connections 2023-10-30T14:05:57.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/178153993 wait complete. 2023-10-30T14:05:57.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 Processor -- start 2023-10-30T14:05:57.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 -- start start 2023-10-30T14:05:57.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80c6fe0 0x7f8cb8152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:57.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cb80c79e0 0x7f8cb8155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:57.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb8153bf0 0x7f8cb8155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:57.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb814ca20 con 0x7f8cb8153bf0 2023-10-30T14:05:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80c7640 con 0x7f8cb80c79e0 2023-10-30T14:05:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbfcd6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb814bea0 con 0x7f8cb80c6fe0 2023-10-30T14:05:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbe273640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb8153bf0 0x7f8cb8155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbe273640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb8153bf0 0x7f8cb8155cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52326/0 (socket says 172.21.15.46:52326) 2023-10-30T14:05:57.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.383+0000 7f8cbe273640 1 -- 172.21.15.46:0/1508869656 learned_addr learned my addr 172.21.15.46:0/1508869656 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cbda72640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80c6fe0 0x7f8cb8152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cbd271640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cb80c79e0 0x7f8cb8155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cbe273640 1 -- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80c6fe0 msgr2=0x7f8cb8152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cbe273640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80c6fe0 0x7f8cb8152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:57.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cbe273640 1 -- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cb80c79e0 msgr2=0x7f8cb8155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cbe273640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cb80c79e0 0x7f8cb8155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cbe273640 1 -- 172.21.15.46:0/1508869656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb40094d0 con 0x7f8cb8153bf0 2023-10-30T14:05:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cbe273640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb8153bf0 0x7f8cb8155cd0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000efc0 tx=0x7f8cb000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cae7fc640 1 -- 172.21.15.46:0/1508869656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0014480 con 0x7f8cb8153bf0 2023-10-30T14:05:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cae7fc640 1 -- 172.21.15.46:0/1508869656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb000f040 con 0x7f8cb8153bf0 2023-10-30T14:05:57.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/1508869656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb814c180 con 0x7f8cb8153bf0 2023-10-30T14:05:57.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cae7fc640 1 -- 172.21.15.46:0/1508869656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0014820 con 0x7f8cb8153bf0 2023-10-30T14:05:57.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/1508869656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb814c740 con 0x7f8cb8153bf0 2023-10-30T14:05:57.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/1508869656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb804e170 con 0x7f8cb8153bf0 2023-10-30T14:05:57.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.387+0000 7f8cae7fc640 1 -- 172.21.15.46:0/1508869656 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8cb0014980 con 0x7f8cb8153bf0 2023-10-30T14:05:57.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.391+0000 7f8cae7fc640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8c9c0664d0 0x7f8c9c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:57.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.391+0000 7f8cae7fc640 1 -- 172.21.15.46:0/1508869656 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7f8cb0085060 con 0x7f8cb8153bf0 2023-10-30T14:05:57.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.391+0000 7f8cbda72640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8c9c0664d0 0x7f8c9c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:57.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.391+0000 7f8cbda72640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8c9c0664d0 0x7f8c9c068990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8cb40b2000 tx=0x7f8cb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:57.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.391+0000 7f8cae7fc640 1 -- 172.21.15.46:0/1508869656 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8cb0051d90 con 0x7f8cb8153bf0 2023-10-30T14:05:57.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.583+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/1508869656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cb806fe90 con 0x7f8cb8153bf0 2023-10-30T14:05:57.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.583+0000 7f8cae7fc640 1 -- 172.21.15.46:0/1508869656 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f8cb0018090 con 0x7f8cb8153bf0 2023-10-30T14:05:57.587 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:57.588 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:54.890950+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 184 2023-10-30T14:05:57.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.587+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8c9c0664d0 msgr2=0x7f8c9c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:57.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.587+0000 7f8cbfcd6640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8c9c0664d0 0x7f8c9c068990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8cb40b2000 tx=0x7f8cb4008040 comp rx=0 tx=0).stop 2023-10-30T14:05:57.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.587+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/1508869656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb8153bf0 msgr2=0x7f8cb8155cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.587+0000 7f8cbfcd6640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb8153bf0 0x7f8cb8155cd0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000efc0 tx=0x7f8cb000c490 comp rx=0 tx=0).stop 2023-10-30T14:05:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.587+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/1508869656 shutdown_connections 2023-10-30T14:05:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.587+0000 7f8cbfcd6640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8c9c0664d0 0x7f8c9c068990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.587+0000 7f8cbfcd6640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cb8153bf0 0x7f8cb8155cd0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.587+0000 7f8cbfcd6640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cb80c79e0 0x7f8cb8155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:57.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.587+0000 7f8cbfcd6640 1 --2- 172.21.15.46:0/1508869656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cb80c6fe0 0x7f8cb8152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:57.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.587+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/1508869656 >> 172.21.15.46:0/1508869656 conn(0x7f8cb806c6e0 msgr2=0x7f8cb8151be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:57.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.587+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/1508869656 shutdown_connections 2023-10-30T14:05:57.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:57.587+0000 7f8cbfcd6640 1 -- 172.21.15.46:0/1508869656 wait complete. 2023-10-30T14:05:57.608 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:05:59.610 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:05:59.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.763+0000 7fb3269a1640 1 Processor -- start 2023-10-30T14:05:59.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.763+0000 7fb3269a1640 1 -- start start 2023-10-30T14:05:59.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.763+0000 7fb3269a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3200c6b80 0x7fb3200bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:59.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.763+0000 7fb3269a1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb32013f4a0 0x7fb3200c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:59.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.763+0000 7fb3269a1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3200c0a50 con 0x7fb3200c6f60 2023-10-30T14:05:59.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.763+0000 7fb3269a1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32006f6e0 con 0x7fb32013f4a0 2023-10-30T14:05:59.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.763+0000 7fb3269a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32006f9c0 con 0x7fb3200c6b80 2023-10-30T14:05:59.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.763+0000 7fb31f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb3200c6f60 0x7fb32013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43252/0 (socket says 172.21.15.46:43252) 2023-10-30T14:05:59.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.763+0000 7fb31f7fe640 1 -- 172.21.15.46:0/2733934949 learned_addr learned my addr 172.21.15.46:0/2733934949 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:05:59.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.763+0000 7fb324f3e640 1 --2- 172.21.15.46:0/2733934949 >> v2:172.21.15.46:3300/0 conn(0x7fb32013f4a0 0x7fb3200c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:59.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.763+0000 7fb31ffff640 1 --2- 172.21.15.46:0/2733934949 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3200c6b80 0x7fb3200bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:59.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.767+0000 7fb324f3e640 1 -- 172.21.15.46:0/2733934949 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3200c6b80 msgr2=0x7fb3200bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:05:59.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.767+0000 7fb324f3e640 1 --2- 172.21.15.46:0/2733934949 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3200c6b80 0x7fb3200bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:59.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.767+0000 7fb324f3e640 1 -- 172.21.15.46:0/2733934949 >> v1:172.21.15.46:6789/0 conn(0x7fb3200c6f60 legacy=0x7fb32013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:59.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.767+0000 7fb324f3e640 1 -- 172.21.15.46:0/2733934949 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32006fca0 con 0x7fb32013f4a0 2023-10-30T14:05:59.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.767+0000 7fb324f3e640 1 --2- 172.21.15.46:0/2733934949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb32013f4a0 0x7fb3200c0400 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb318009d90 tx=0x7fb31801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b011e816102f0a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:59.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.767+0000 7fb31effd640 1 -- 172.21.15.46:0/2733934949 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3180a5920 con 0x7fb32013f4a0 2023-10-30T14:05:59.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.767+0000 7fb31effd640 1 -- 172.21.15.46:0/2733934949 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3180a5a80 con 0x7fb32013f4a0 2023-10-30T14:05:59.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.767+0000 7fb31effd640 1 -- 172.21.15.46:0/2733934949 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3180a5e10 con 0x7fb32013f4a0 2023-10-30T14:05:59.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.767+0000 7fb3269a1640 1 -- 172.21.15.46:0/2733934949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb32013f4a0 msgr2=0x7fb3200c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:59.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.767+0000 7fb3269a1640 1 --2- 172.21.15.46:0/2733934949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb32013f4a0 0x7fb3200c0400 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb318009d90 tx=0x7fb31801c950 comp rx=0 tx=0).stop 2023-10-30T14:05:59.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 -- 172.21.15.46:0/2733934949 shutdown_connections 2023-10-30T14:05:59.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 --2- 172.21.15.46:0/2733934949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb32013f4a0 0x7fb3200c0400 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:59.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 --2- 172.21.15.46:0/2733934949 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3200c6b80 0x7fb3200bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:59.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 -- 172.21.15.46:0/2733934949 >> 172.21.15.46:0/2733934949 conn(0x7fb3200c7910 msgr2=0x7fb3200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:59.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 -- 172.21.15.46:0/2733934949 shutdown_connections 2023-10-30T14:05:59.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 -- 172.21.15.46:0/2733934949 wait complete. 2023-10-30T14:05:59.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 Processor -- start 2023-10-30T14:05:59.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 -- start start 2023-10-30T14:05:59.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200c6b80 0x7fb320153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:59.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3200c6f60 0x7fb320134650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:59.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb32013f4a0 0x7fb320136700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:59.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb320154000 con 0x7fb3200c6b80 2023-10-30T14:05:59.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3201364d0 con 0x7fb3200c6f60 2023-10-30T14:05:59.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb3269a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3201f59a0 con 0x7fb32013f4a0 2023-10-30T14:05:59.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb31ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200c6b80 0x7fb320153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:59.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb31ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200c6b80 0x7fb320153ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52354/0 (socket says 172.21.15.46:52354) 2023-10-30T14:05:59.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb31ffff640 1 -- 172.21.15.46:0/3720710352 learned_addr learned my addr 172.21.15.46:0/3720710352 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:05:59.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb324f3e640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb32013f4a0 0x7fb320136700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:59.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb31f7fe640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3200c6f60 0x7fb320134650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:59.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb31ffff640 1 -- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb32013f4a0 msgr2=0x7fb320136700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:59.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb31ffff640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb32013f4a0 0x7fb320136700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:59.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb31ffff640 1 -- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3200c6f60 msgr2=0x7fb320134650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb31ffff640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3200c6f60 0x7fb320134650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb31ffff640 1 -- 172.21.15.46:0/3720710352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb318009a70 con 0x7fb3200c6b80 2023-10-30T14:05:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb31f7fe640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3200c6f60 0x7fb320134650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:05:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.771+0000 7fb31ffff640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200c6b80 0x7fb320153ac0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fb31000e930 tx=0x7fb31000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:59.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.775+0000 7fb31cff9640 1 -- 172.21.15.46:0/3720710352 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb310009690 con 0x7fb3200c6b80 2023-10-30T14:05:59.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.775+0000 7fb31cff9640 1 -- 172.21.15.46:0/3720710352 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31000ca00 con 0x7fb3200c6b80 2023-10-30T14:05:59.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.775+0000 7fb3269a1640 1 -- 172.21.15.46:0/3720710352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3201f5e30 con 0x7fb3200c6b80 2023-10-30T14:05:59.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.775+0000 7fb31cff9640 1 -- 172.21.15.46:0/3720710352 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31000cd70 con 0x7fb3200c6b80 2023-10-30T14:05:59.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.775+0000 7fb3269a1640 1 -- 172.21.15.46:0/3720710352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3201361f0 con 0x7fb3200c6b80 2023-10-30T14:05:59.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.775+0000 7fb3269a1640 1 -- 172.21.15.46:0/3720710352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32004e170 con 0x7fb3200c6b80 2023-10-30T14:05:59.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.775+0000 7fb31cff9640 1 -- 172.21.15.46:0/3720710352 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb310013470 con 0x7fb3200c6b80 2023-10-30T14:05:59.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.775+0000 7fb31cff9640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb30c0664a0 0x7fb30c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:05:59.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.775+0000 7fb31cff9640 1 -- 172.21.15.46:0/3720710352 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7fb310084d60 con 0x7fb3200c6b80 2023-10-30T14:05:59.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.775+0000 7fb31f7fe640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb30c0664a0 0x7fb30c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:05:59.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.779+0000 7fb31f7fe640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb30c0664a0 0x7fb30c068960 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb314007fd0 tx=0x7fb314009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:05:59.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.779+0000 7fb31cff9640 1 -- 172.21.15.46:0/3720710352 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb310051980 con 0x7fb3200c6b80 2023-10-30T14:05:59.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 -- 172.21.15.46:0/3720710352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb32013e820 con 0x7fb3200c6b80 2023-10-30T14:05:59.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb31cff9640 1 -- 172.21.15.46:0/3720710352 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fb310017090 con 0x7fb3200c6b80 2023-10-30T14:05:59.976 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:05:59.977 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:05:54.890950+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:05:59.977 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 184 2023-10-30T14:05:59.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 -- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb30c0664a0 msgr2=0x7fb30c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:59.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb30c0664a0 0x7fb30c068960 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb314007fd0 tx=0x7fb314009040 comp rx=0 tx=0).stop 2023-10-30T14:05:59.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 -- 172.21.15.46:0/3720710352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200c6b80 msgr2=0x7fb320153ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:05:59.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200c6b80 0x7fb320153ac0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fb31000e930 tx=0x7fb31000edf0 comp rx=0 tx=0).stop 2023-10-30T14:05:59.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 -- 172.21.15.46:0/3720710352 shutdown_connections 2023-10-30T14:05:59.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb30c0664a0 0x7fb30c068960 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:59.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb32013f4a0 0x7fb320136700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:59.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3200c6f60 0x7fb320134650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:59.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 --2- 172.21.15.46:0/3720710352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200c6b80 0x7fb320153ac0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:05:59.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 -- 172.21.15.46:0/3720710352 >> 172.21.15.46:0/3720710352 conn(0x7fb3200c7910 msgr2=0x7fb3200bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:05:59.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 -- 172.21.15.46:0/3720710352 shutdown_connections 2023-10-30T14:05:59.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:05:59.975+0000 7fb3269a1640 1 -- 172.21.15.46:0/3720710352 wait complete. 2023-10-30T14:05:59.998 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:06:01.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:06:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.151+0000 7fe0c41b8640 1 Processor -- start 2023-10-30T14:06:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c41b8640 1 -- start start 2023-10-30T14:06:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c41b8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe0bc0c6b80 0x7fe0bc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c41b8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe0bc0c6f60 0x7fe0bc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c41b8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0bc0c0a50 con 0x7fe0bc0c8430 2023-10-30T14:06:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c41b8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0bc06f6e0 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c41b8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0bc06f9c0 con 0x7fe0bc0c6b80 2023-10-30T14:06:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c1753640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe0bc0c6f60 0x7fe0bc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c1753640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe0bc0c6f60 0x7fe0bc0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52362/0 (socket says 172.21.15.46:52362) 2023-10-30T14:06:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c1753640 1 -- 172.21.15.46:0/395983544 learned_addr learned my addr 172.21.15.46:0/395983544 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:06:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c1f54640 1 --2- 172.21.15.46:0/395983544 >> v2:172.21.15.154:3300/0 conn(0x7fe0bc0c6b80 0x7fe0bc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:02.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c1753640 1 -- 172.21.15.46:0/395983544 >> v2:172.21.15.154:3300/0 conn(0x7fe0bc0c6b80 msgr2=0x7fe0bc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:06:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c1753640 1 --2- 172.21.15.46:0/395983544 >> v2:172.21.15.154:3300/0 conn(0x7fe0bc0c6b80 0x7fe0bc0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c1753640 1 -- 172.21.15.46:0/395983544 >> v1:172.21.15.154:6789/0 conn(0x7fe0bc0c8430 legacy=0x7fe0bc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:02.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.155+0000 7fe0c1753640 1 -- 172.21.15.46:0/395983544 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0bc06fca0 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c1753640 1 --2- 172.21.15.46:0/395983544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0bc0c6f60 0x7fe0bc0c0400 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc06f1b0 tx=0x7fe0ac01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04b86fd3bebd1de server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c0f52640 1 -- 172.21.15.46:0/395983544 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0ab610 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c0f52640 1 -- 172.21.15.46:0/395983544 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ac0ab770 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c0f52640 1 -- 172.21.15.46:0/395983544 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0abb00 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c41b8640 1 -- 172.21.15.46:0/395983544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0bc0c6f60 msgr2=0x7fe0bc0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c41b8640 1 --2- 172.21.15.46:0/395983544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0bc0c6f60 0x7fe0bc0c0400 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc06f1b0 tx=0x7fe0ac01c500 comp rx=0 tx=0).stop 2023-10-30T14:06:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c41b8640 1 -- 172.21.15.46:0/395983544 shutdown_connections 2023-10-30T14:06:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c41b8640 1 --2- 172.21.15.46:0/395983544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0bc0c6f60 0x7fe0bc0c0400 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c41b8640 1 --2- 172.21.15.46:0/395983544 >> v2:172.21.15.154:3300/0 conn(0x7fe0bc0c6b80 0x7fe0bc0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c41b8640 1 -- 172.21.15.46:0/395983544 >> 172.21.15.46:0/395983544 conn(0x7fe0bc0c7910 msgr2=0x7fe0bc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c41b8640 1 -- 172.21.15.46:0/395983544 shutdown_connections 2023-10-30T14:06:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c41b8640 1 -- 172.21.15.46:0/395983544 wait complete. 2023-10-30T14:06:02.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.159+0000 7fe0c41b8640 1 Processor -- start 2023-10-30T14:06:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c41b8640 1 -- start start 2023-10-30T14:06:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c41b8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0bc0c6b80 0x7fe0bc1389b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c41b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0bc0c6f60 0x7fe0bc1f5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c41b8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0bc0c8430 0x7fe0bc1543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:02.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c41b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc155aa0 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c41b8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc160610 con 0x7fe0bc0c8430 2023-10-30T14:06:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c41b8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc138ef0 con 0x7fe0bc0c6b80 2023-10-30T14:06:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c1753640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0bc0c6f60 0x7fe0bc1f5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c1753640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0bc0c6f60 0x7fe0bc1f5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52366/0 (socket says 172.21.15.46:52366) 2023-10-30T14:06:02.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c1753640 1 -- 172.21.15.46:0/1127819613 learned_addr learned my addr 172.21.15.46:0/1127819613 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:06:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c1f54640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0bc0c6b80 0x7fe0bc1389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c2755640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0bc0c8430 0x7fe0bc1543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c1753640 1 -- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0bc0c6b80 msgr2=0x7fe0bc1389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c1753640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0bc0c6b80 0x7fe0bc1389b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:02.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c1753640 1 -- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0bc0c8430 msgr2=0x7fe0bc1543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c1753640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0bc0c8430 0x7fe0bc1543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c1753640 1 -- 172.21.15.46:0/1127819613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac0094d0 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c1753640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0bc0c6f60 0x7fe0bc1f5080 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac01c9e0 tx=0x7fe0ac01ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0b27fc640 1 -- 172.21.15.46:0/1127819613 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0b4800 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0b27fc640 1 -- 172.21.15.46:0/1127819613 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ac0b4960 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0b27fc640 1 -- 172.21.15.46:0/1127819613 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0b4d10 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c41b8640 1 -- 172.21.15.46:0/1127819613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0bc15f450 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c41b8640 1 -- 172.21.15.46:0/1127819613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0bc1f1bc0 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.163+0000 7fe0c41b8640 1 -- 172.21.15.46:0/1127819613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0bc04e170 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.167+0000 7fe0b27fc640 1 -- 172.21.15.46:0/1127819613 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe0ac0b2410 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.167+0000 7fe0b27fc640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0a00664d0 0x7fe0a0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.167+0000 7fe0b27fc640 1 -- 172.21.15.46:0/1127819613 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7fe0ac11ef00 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.167+0000 7fe0c1f54640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0a00664d0 0x7fe0a0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.167+0000 7fe0c1f54640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0a00664d0 0x7fe0a0068990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800b9d0 tx=0x7fe0b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.171+0000 7fe0b27fc640 1 -- 172.21.15.46:0/1127819613 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe0ac0ebc20 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.359+0000 7fe0c41b8640 1 -- 172.21.15.46:0/1127819613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0bc0723c0 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.359+0000 7fe0b27fc640 1 -- 172.21.15.46:0/1127819613 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fe0ac0c4020 con 0x7fe0bc0c6f60 2023-10-30T14:06:02.362 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:06:02.362 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:00.843055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:06:02.363 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 185 2023-10-30T14:06:02.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.363+0000 7fe0c41b8640 1 -- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0a00664d0 msgr2=0x7fe0a0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:02.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.363+0000 7fe0c41b8640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0a00664d0 0x7fe0a0068990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800b9d0 tx=0x7fe0b8008040 comp rx=0 tx=0).stop 2023-10-30T14:06:02.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.363+0000 7fe0c41b8640 1 -- 172.21.15.46:0/1127819613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0bc0c6f60 msgr2=0x7fe0bc1f5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:02.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.363+0000 7fe0c41b8640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0bc0c6f60 0x7fe0bc1f5080 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac01c9e0 tx=0x7fe0ac01ad40 comp rx=0 tx=0).stop 2023-10-30T14:06:02.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.363+0000 7fe0c41b8640 1 -- 172.21.15.46:0/1127819613 shutdown_connections 2023-10-30T14:06:02.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.363+0000 7fe0c41b8640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0a00664d0 0x7fe0a0068990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:02.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.363+0000 7fe0c41b8640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0bc0c8430 0x7fe0bc1543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:02.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.363+0000 7fe0c41b8640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0bc0c6f60 0x7fe0bc1f5080 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:02.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.363+0000 7fe0c41b8640 1 --2- 172.21.15.46:0/1127819613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0bc0c6b80 0x7fe0bc1389b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:02.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.363+0000 7fe0c41b8640 1 -- 172.21.15.46:0/1127819613 >> 172.21.15.46:0/1127819613 conn(0x7fe0bc0c7910 msgr2=0x7fe0bc0c5810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:02.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.363+0000 7fe0c41b8640 1 -- 172.21.15.46:0/1127819613 shutdown_connections 2023-10-30T14:06:02.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:02.363+0000 7fe0c41b8640 1 -- 172.21.15.46:0/1127819613 wait complete. 2023-10-30T14:06:02.382 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:06:04.383 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:06:04.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.535+0000 7f0e3d49c640 1 Processor -- start 2023-10-30T14:06:04.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e3d49c640 1 -- start start 2023-10-30T14:06:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e3d49c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0e381538a0 0x7f0e3815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e3d49c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e380cf160 0x7f0e380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:04.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e3d49c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e380cb800 con 0x7f0e380ca0e0 2023-10-30T14:06:04.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e3d49c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e3815dc30 con 0x7f0e381538a0 2023-10-30T14:06:04.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e3d49c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e380cb330 con 0x7f0e380cf160 2023-10-30T14:06:04.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e36ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0e381538a0 0x7f0e3815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:04.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e377fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e380cf160 0x7f0e380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:04.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e36ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0e381538a0 0x7f0e3815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:41750/0 (socket says 172.21.15.46:41750) 2023-10-30T14:06:04.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e377fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e380cf160 0x7f0e380cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:42894/0 (socket says 172.21.15.46:42894) 2023-10-30T14:06:04.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e36ffd640 1 -- 172.21.15.46:0/3440644682 learned_addr learned my addr 172.21.15.46:0/3440644682 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:06:04.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e367fc640 1 --1- 172.21.15.46:0/3440644682 >> v1:172.21.15.154:6789/0 conn(0x7f0e380ca0e0 0x7f0e3815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:44166/0 (socket says 172.21.15.46:44166) 2023-10-30T14:06:04.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e36ffd640 1 -- 172.21.15.46:0/3440644682 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e380cf160 msgr2=0x7f0e380cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:06:04.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e36ffd640 1 --2- 172.21.15.46:0/3440644682 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e380cf160 0x7f0e380cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:04.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e36ffd640 1 -- 172.21.15.46:0/3440644682 >> v1:172.21.15.154:6789/0 conn(0x7f0e380ca0e0 legacy=0x7f0e3815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:04.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e36ffd640 1 -- 172.21.15.46:0/3440644682 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e380cad70 con 0x7f0e381538a0 2023-10-30T14:06:04.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.539+0000 7f0e377fe640 1 --2- 172.21.15.46:0/3440644682 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e380cf160 0x7f0e380cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:06:04.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.543+0000 7f0e36ffd640 1 --2- 172.21.15.46:0/3440644682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e381538a0 0x7f0e3815d6f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009870 tx=0x7f0e2801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99d7e9bff5093ad9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:04.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.543+0000 7f0e35ffb640 1 -- 172.21.15.46:0/3440644682 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e280a8a30 con 0x7f0e381538a0 2023-10-30T14:06:04.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.543+0000 7f0e35ffb640 1 -- 172.21.15.46:0/3440644682 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e280a8b90 con 0x7f0e381538a0 2023-10-30T14:06:04.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.543+0000 7f0e35ffb640 1 -- 172.21.15.46:0/3440644682 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e280b0440 con 0x7f0e381538a0 2023-10-30T14:06:04.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.543+0000 7f0e3d49c640 1 -- 172.21.15.46:0/3440644682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e381538a0 msgr2=0x7f0e3815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:04.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.543+0000 7f0e3d49c640 1 --2- 172.21.15.46:0/3440644682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e381538a0 0x7f0e3815d6f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009870 tx=0x7f0e2801ed30 comp rx=0 tx=0).stop 2023-10-30T14:06:04.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.543+0000 7f0e3d49c640 1 -- 172.21.15.46:0/3440644682 shutdown_connections 2023-10-30T14:06:04.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.543+0000 7f0e3d49c640 1 --2- 172.21.15.46:0/3440644682 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e380cf160 0x7f0e380cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:04.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.543+0000 7f0e3d49c640 1 --2- 172.21.15.46:0/3440644682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e381538a0 0x7f0e3815d6f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:04.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.543+0000 7f0e3d49c640 1 -- 172.21.15.46:0/3440644682 >> 172.21.15.46:0/3440644682 conn(0x7f0e3806bfe0 msgr2=0x7f0e3806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:04.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.543+0000 7f0e3d49c640 1 -- 172.21.15.46:0/3440644682 shutdown_connections 2023-10-30T14:06:04.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.543+0000 7f0e3d49c640 1 -- 172.21.15.46:0/3440644682 wait complete. 2023-10-30T14:06:04.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e3d49c640 1 Processor -- start 2023-10-30T14:06:04.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e3d49c640 1 -- start start 2023-10-30T14:06:04.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e3d49c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e380ca0e0 0x7f0e381f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:04.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e3d49c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e380cf160 0x7f0e38066180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:04.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e3d49c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e381538a0 0x7f0e3814cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:04.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e3d49c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e381e2f00 con 0x7f0e380cf160 2023-10-30T14:06:04.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e3d49c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e381e31e0 con 0x7f0e381538a0 2023-10-30T14:06:04.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e3d49c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e381e34c0 con 0x7f0e380ca0e0 2023-10-30T14:06:04.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e367fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e380cf160 0x7f0e38066180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:04.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e36ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e380ca0e0 0x7f0e381f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:04.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e367fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e380cf160 0x7f0e38066180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52372/0 (socket says 172.21.15.46:52372) 2023-10-30T14:06:04.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e367fc640 1 -- 172.21.15.46:0/2126066797 learned_addr learned my addr 172.21.15.46:0/2126066797 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:06:04.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e377fe640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e381538a0 0x7f0e3814cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:04.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e367fc640 1 -- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e380ca0e0 msgr2=0x7f0e381f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:04.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e367fc640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e380ca0e0 0x7f0e381f2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:04.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e367fc640 1 -- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e381538a0 msgr2=0x7f0e3814cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:04.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e367fc640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e381538a0 0x7f0e3814cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:04.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e367fc640 1 -- 172.21.15.46:0/2126066797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e30009a70 con 0x7f0e380cf160 2023-10-30T14:06:04.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e367fc640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e380cf160 0x7f0e38066180 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00c490 tx=0x7f0e2c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:04.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e1f7fe640 1 -- 172.21.15.46:0/2126066797 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c007480 con 0x7f0e380cf160 2023-10-30T14:06:04.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e1f7fe640 1 -- 172.21.15.46:0/2126066797 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c0075e0 con 0x7f0e380cf160 2023-10-30T14:06:04.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e1f7fe640 1 -- 172.21.15.46:0/2126066797 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c004a10 con 0x7f0e380cf160 2023-10-30T14:06:04.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e3d49c640 1 -- 172.21.15.46:0/2126066797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e280094d0 con 0x7f0e380cf160 2023-10-30T14:06:04.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.547+0000 7f0e3d49c640 1 -- 172.21.15.46:0/2126066797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e381f2a20 con 0x7f0e380cf160 2023-10-30T14:06:04.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.551+0000 7f0e3d49c640 1 -- 172.21.15.46:0/2126066797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e380b2a50 con 0x7f0e380cf160 2023-10-30T14:06:04.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.551+0000 7f0e1f7fe640 1 -- 172.21.15.46:0/2126066797 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e2c004b70 con 0x7f0e380cf160 2023-10-30T14:06:04.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.551+0000 7f0e1f7fe640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e240664a0 0x7f0e24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:04.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.551+0000 7f0e1f7fe640 1 -- 172.21.15.46:0/2126066797 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7f0e2c083870 con 0x7f0e380cf160 2023-10-30T14:06:04.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.551+0000 7f0e36ffd640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e240664a0 0x7f0e24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:04.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.551+0000 7f0e36ffd640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e240664a0 0x7f0e24068960 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0e280b2000 tx=0x7f0e28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:04.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.555+0000 7f0e1f7fe640 1 -- 172.21.15.46:0/2126066797 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e2c050500 con 0x7f0e380cf160 2023-10-30T14:06:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.743+0000 7f0e3d49c640 1 -- 172.21.15.46:0/2126066797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e3804e170 con 0x7f0e380cf160 2023-10-30T14:06:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.743+0000 7f0e1f7fe640 1 -- 172.21.15.46:0/2126066797 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f0e2c018090 con 0x7f0e380cf160 2023-10-30T14:06:04.747 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:06:04.748 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:00.843055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:06:04.749 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 185 2023-10-30T14:06:04.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.747+0000 7f0e3d49c640 1 -- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e240664a0 msgr2=0x7f0e24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:04.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.747+0000 7f0e3d49c640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e240664a0 0x7f0e24068960 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0e280b2000 tx=0x7f0e28009040 comp rx=0 tx=0).stop 2023-10-30T14:06:04.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.747+0000 7f0e3d49c640 1 -- 172.21.15.46:0/2126066797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e380cf160 msgr2=0x7f0e38066180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:04.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.747+0000 7f0e3d49c640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e380cf160 0x7f0e38066180 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00c490 tx=0x7f0e2c00c950 comp rx=0 tx=0).stop 2023-10-30T14:06:04.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.747+0000 7f0e3d49c640 1 -- 172.21.15.46:0/2126066797 shutdown_connections 2023-10-30T14:06:04.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.747+0000 7f0e3d49c640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0e240664a0 0x7f0e24068960 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:04.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.747+0000 7f0e3d49c640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e381538a0 0x7f0e3814cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:04.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.747+0000 7f0e3d49c640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e380cf160 0x7f0e38066180 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:04.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.747+0000 7f0e3d49c640 1 --2- 172.21.15.46:0/2126066797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e380ca0e0 0x7f0e381f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:04.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.747+0000 7f0e3d49c640 1 -- 172.21.15.46:0/2126066797 >> 172.21.15.46:0/2126066797 conn(0x7f0e3806bfe0 msgr2=0x7f0e38151940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:04.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.747+0000 7f0e3d49c640 1 -- 172.21.15.46:0/2126066797 shutdown_connections 2023-10-30T14:06:04.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:04.747+0000 7f0e3d49c640 1 -- 172.21.15.46:0/2126066797 wait complete. 2023-10-30T14:06:04.768 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 185, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 185, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8190": { "addr": "172.21.15.46:6837/3154171789", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3154171789, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3154171789, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 8190, "incarnation": 175, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 40 }, "gid_8352": { "addr": "172.21.15.46:6836/128865315", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 128865315, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 128865315, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8352, "incarnation": 182, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:rejoin", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:06:00.843055+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8190, "mds_1": 8352 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:06:06.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:06:06.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.911+0000 7febf1780640 1 Processor -- start 2023-10-30T14:06:06.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.911+0000 7febf1780640 1 -- start start 2023-10-30T14:06:06.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.911+0000 7febf1780640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7febec159b80 0x7febec159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:06.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.911+0000 7febf1780640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7febec06b440 0x7febec06b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:06.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.911+0000 7febf1780640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febec1566e0 con 0x7febec13f9c0 2023-10-30T14:06:06.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.911+0000 7febf1780640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febec157360 con 0x7febec159b80 2023-10-30T14:06:06.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.911+0000 7febf1780640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febec13fda0 con 0x7febec06b440 2023-10-30T14:06:06.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.911+0000 7febeaffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7febec13f9c0 0x7febec0c5a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59786/0 (socket says 172.21.15.46:59786) 2023-10-30T14:06:06.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.911+0000 7febeaffd640 1 -- 172.21.15.46:0/3078081558 learned_addr learned my addr 172.21.15.46:0/3078081558 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:06:06.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.911+0000 7febe3fff640 1 --2- 172.21.15.46:0/3078081558 >> v2:172.21.15.46:3300/0 conn(0x7febec159b80 0x7febec159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:06.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.915+0000 7febeb7fe640 1 --2- 172.21.15.46:0/3078081558 >> v2:172.21.15.154:3300/0 conn(0x7febec06b440 0x7febec06b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.915+0000 7febe3fff640 1 -- 172.21.15.46:0/3078081558 >> v2:172.21.15.154:3300/0 conn(0x7febec06b440 msgr2=0x7febec06b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:06:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.915+0000 7febe3fff640 1 --2- 172.21.15.46:0/3078081558 >> v2:172.21.15.154:3300/0 conn(0x7febec06b440 0x7febec06b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.915+0000 7febe3fff640 1 -- 172.21.15.46:0/3078081558 >> v1:172.21.15.46:6789/0 conn(0x7febec13f9c0 legacy=0x7febec0c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.915+0000 7febe3fff640 1 -- 172.21.15.46:0/3078081558 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec1569c0 con 0x7febec159b80 2023-10-30T14:06:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.915+0000 7febe3fff640 1 --2- 172.21.15.46:0/3078081558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febec159b80 0x7febec159fa0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7febec0cba00 tx=0x7febd401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa37a9c6d003ae7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.915+0000 7febea7fc640 1 -- 172.21.15.46:0/3078081558 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4004230 con 0x7febec159b80 2023-10-30T14:06:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.915+0000 7febea7fc640 1 -- 172.21.15.46:0/3078081558 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd40a7050 con 0x7febec159b80 2023-10-30T14:06:06.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.915+0000 7febea7fc640 1 -- 172.21.15.46:0/3078081558 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4003620 con 0x7febec159b80 2023-10-30T14:06:06.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 -- 172.21.15.46:0/3078081558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febec159b80 msgr2=0x7febec159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:06.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 --2- 172.21.15.46:0/3078081558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febec159b80 0x7febec159fa0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7febec0cba00 tx=0x7febd401c460 comp rx=0 tx=0).stop 2023-10-30T14:06:06.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 -- 172.21.15.46:0/3078081558 shutdown_connections 2023-10-30T14:06:06.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 --2- 172.21.15.46:0/3078081558 >> v2:172.21.15.154:3300/0 conn(0x7febec06b440 0x7febec06b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:06.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 --2- 172.21.15.46:0/3078081558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febec159b80 0x7febec159fa0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:06.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 -- 172.21.15.46:0/3078081558 >> 172.21.15.46:0/3078081558 conn(0x7febec0cace0 msgr2=0x7febec0cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:06.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 -- 172.21.15.46:0/3078081558 shutdown_connections 2023-10-30T14:06:06.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 -- 172.21.15.46:0/3078081558 wait complete. 2023-10-30T14:06:06.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 Processor -- start 2023-10-30T14:06:06.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 -- start start 2023-10-30T14:06:06.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febec06b440 0x7febec0c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:06.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febec13f9c0 0x7febec0c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:06.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febec159b80 0x7febec1f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:06.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec0c6e90 con 0x7febec13f9c0 2023-10-30T14:06:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec0ca5c0 con 0x7febec159b80 2023-10-30T14:06:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febf1780640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febec0caa50 con 0x7febec06b440 2023-10-30T14:06:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febe3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febec13f9c0 0x7febec0c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:06.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febe3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febec13f9c0 0x7febec0c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46122/0 (socket says 172.21.15.46:46122) 2023-10-30T14:06:06.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febe3fff640 1 -- 172.21.15.46:0/232709682 learned_addr learned my addr 172.21.15.46:0/232709682 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:06:06.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febeaffd640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febec06b440 0x7febec0c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:06.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febeb7fe640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febec159b80 0x7febec1f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:06.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febe3fff640 1 -- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febec06b440 msgr2=0x7febec0c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:06.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febe3fff640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febec06b440 0x7febec0c9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:06.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febe3fff640 1 -- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febec159b80 msgr2=0x7febec1f1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:06.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febe3fff640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febec159b80 0x7febec1f1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:06.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febe3fff640 1 -- 172.21.15.46:0/232709682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd4009420 con 0x7febec13f9c0 2023-10-30T14:06:06.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febeb7fe640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febec159b80 0x7febec1f1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:06:06.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.919+0000 7febe3fff640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febec13f9c0 0x7febec0c7f70 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7febd40ae040 tx=0x7febd4019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.923+0000 7febe37fe640 1 -- 172.21.15.46:0/232709682 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4003b70 con 0x7febec13f9c0 2023-10-30T14:06:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.923+0000 7febe37fe640 1 -- 172.21.15.46:0/232709682 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd4003cd0 con 0x7febec13f9c0 2023-10-30T14:06:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.923+0000 7febe37fe640 1 -- 172.21.15.46:0/232709682 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd40ab680 con 0x7febec13f9c0 2023-10-30T14:06:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.923+0000 7febf1780640 1 -- 172.21.15.46:0/232709682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febec0c9090 con 0x7febec13f9c0 2023-10-30T14:06:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.923+0000 7febf1780640 1 -- 172.21.15.46:0/232709682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febec0c9f70 con 0x7febec13f9c0 2023-10-30T14:06:06.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.923+0000 7febf1780640 1 -- 172.21.15.46:0/232709682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febec04e170 con 0x7febec13f9c0 2023-10-30T14:06:06.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.923+0000 7febe37fe640 1 -- 172.21.15.46:0/232709682 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7febd40b2040 con 0x7febec13f9c0 2023-10-30T14:06:06.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.923+0000 7febe37fe640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7febcc0664a0 0x7febcc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:06.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.923+0000 7febe37fe640 1 -- 172.21.15.46:0/232709682 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7febd411e5f0 con 0x7febec13f9c0 2023-10-30T14:06:06.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.923+0000 7febeaffd640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7febcc0664a0 0x7febcc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:06.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.927+0000 7febeaffd640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7febcc0664a0 0x7febcc068960 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7febdc005890 tx=0x7febdc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:06.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:06.927+0000 7febe37fe640 1 -- 172.21.15.46:0/232709682 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7febd40eb1d0 con 0x7febec13f9c0 2023-10-30T14:06:07.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 -- 172.21.15.46:0/232709682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febec0c5fd0 con 0x7febec13f9c0 2023-10-30T14:06:07.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febe37fe640 1 -- 172.21.15.46:0/232709682 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+5730 (secure 0 0 0) 0x7febd40ee7f0 con 0x7febec13f9c0 2023-10-30T14:06:07.120 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:06:07.121 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:06.729509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:06:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 186 2023-10-30T14:06:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 -- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7febcc0664a0 msgr2=0x7febcc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7febcc0664a0 0x7febcc068960 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7febdc005890 tx=0x7febdc009040 comp rx=0 tx=0).stop 2023-10-30T14:06:07.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 -- 172.21.15.46:0/232709682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febec13f9c0 msgr2=0x7febec0c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:07.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febec13f9c0 0x7febec0c7f70 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7febd40ae040 tx=0x7febd4019200 comp rx=0 tx=0).stop 2023-10-30T14:06:07.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 -- 172.21.15.46:0/232709682 shutdown_connections 2023-10-30T14:06:07.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7febcc0664a0 0x7febcc068960 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:07.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febec159b80 0x7febec1f1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:07.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febec13f9c0 0x7febec0c7f70 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:07.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 --2- 172.21.15.46:0/232709682 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febec06b440 0x7febec0c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:07.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 -- 172.21.15.46:0/232709682 >> 172.21.15.46:0/232709682 conn(0x7febec0cace0 msgr2=0x7febec1377e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:07.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 -- 172.21.15.46:0/232709682 shutdown_connections 2023-10-30T14:06:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:07.119+0000 7febf1780640 1 -- 172.21.15.46:0/232709682 wait complete. 2023-10-30T14:06:07.141 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=8034 2023-10-30T14:06:07.141 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.a 2023-10-30T14:06:09.143 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T14:06:09.143 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T14:06:09.143 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T14:06:09.147 INFO:tasks.ceph.mds.a:Started 2023-10-30T14:06:09.148 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:06:09.245 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-10-30T14:06:09.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.299+0000 7fa4c2246640 1 Processor -- start 2023-10-30T14:06:09.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.299+0000 7fa4c2246640 1 -- start start 2023-10-30T14:06:09.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4c2246640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa4bc06f7a0 0x7fa4bc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:09.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4c2246640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4bc0c6bd0 0x7fa4bc069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:09.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4c2246640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4bc06a3d0 con 0x7fa4bc071110 2023-10-30T14:06:09.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4c2246640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4bc0700c0 con 0x7fa4bc06f7a0 2023-10-30T14:06:09.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4c2246640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc066cf0 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4bbfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa4bc071110 0x7fa4bc158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59792/0 (socket says 172.21.15.46:59792) 2023-10-30T14:06:09.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4bbfff640 1 -- 172.21.15.46:0/2628683232 learned_addr learned my addr 172.21.15.46:0/2628683232 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:06:09.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4baffd640 1 --2- 172.21.15.46:0/2628683232 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4bc0c6bd0 0x7fa4bc069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:09.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4bb7fe640 1 --2- 172.21.15.46:0/2628683232 >> v2:172.21.15.154:3300/0 conn(0x7fa4bc06f7a0 0x7fa4bc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:09.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4ba7fc640 1 -- 172.21.15.46:0/2628683232 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986779507 0 0) 0x7fa4bc06a3d0 con 0x7fa4bc071110 2023-10-30T14:06:09.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4ba7fc640 1 -- 172.21.15.46:0/2628683232 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a4003530 con 0x7fa4bc071110 2023-10-30T14:06:09.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4baffd640 1 -- 172.21.15.46:0/2628683232 >> v2:172.21.15.154:3300/0 conn(0x7fa4bc06f7a0 msgr2=0x7fa4bc06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:06:09.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4baffd640 1 --2- 172.21.15.46:0/2628683232 >> v2:172.21.15.154:3300/0 conn(0x7fa4bc06f7a0 0x7fa4bc06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:09.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4baffd640 1 -- 172.21.15.46:0/2628683232 >> v1:172.21.15.46:6789/0 conn(0x7fa4bc071110 legacy=0x7fa4bc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:09.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4baffd640 1 -- 172.21.15.46:0/2628683232 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4bc0c70d0 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.303+0000 7fa4bb7fe640 1 --2- 172.21.15.46:0/2628683232 >> v2:172.21.15.154:3300/0 conn(0x7fa4bc06f7a0 0x7fa4bc06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:06:09.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4baffd640 1 --2- 172.21.15.46:0/2628683232 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4bc0c6bd0 0x7fa4bc069e90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa4b00097c0 tx=0x7fa4b001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9da4ad86d6b038 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:09.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4ba7fc640 1 -- 172.21.15.46:0/2628683232 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00a8970 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4ba7fc640 1 -- 172.21.15.46:0/2628683232 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b00a8ad0 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4ba7fc640 1 -- 172.21.15.46:0/2628683232 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00a8e60 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4c2246640 1 -- 172.21.15.46:0/2628683232 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4bc0c6bd0 msgr2=0x7fa4bc069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:09.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4c2246640 1 --2- 172.21.15.46:0/2628683232 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4bc0c6bd0 0x7fa4bc069e90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa4b00097c0 tx=0x7fa4b001ec70 comp rx=0 tx=0).stop 2023-10-30T14:06:09.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4c2246640 1 -- 172.21.15.46:0/2628683232 shutdown_connections 2023-10-30T14:06:09.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4c2246640 1 --2- 172.21.15.46:0/2628683232 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4bc0c6bd0 0x7fa4bc069e90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:09.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4c2246640 1 --2- 172.21.15.46:0/2628683232 >> v2:172.21.15.154:3300/0 conn(0x7fa4bc06f7a0 0x7fa4bc06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:09.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4c2246640 1 -- 172.21.15.46:0/2628683232 >> 172.21.15.46:0/2628683232 conn(0x7fa4bc0c7910 msgr2=0x7fa4bc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:09.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4c2246640 1 -- 172.21.15.46:0/2628683232 shutdown_connections 2023-10-30T14:06:09.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4c2246640 1 -- 172.21.15.46:0/2628683232 wait complete. 2023-10-30T14:06:09.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4c2246640 1 Processor -- start 2023-10-30T14:06:09.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.307+0000 7fa4c2246640 1 -- start start 2023-10-30T14:06:09.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4c2246640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4bc06f7a0 0x7fa4bc1f0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:09.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4c2246640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4bc071110 0x7fa4bc145780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:09.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4c2246640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4bc0c6bd0 0x7fa4bc158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:09.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4c2246640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc1edc80 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4c2246640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc1550a0 con 0x7fa4bc06f7a0 2023-10-30T14:06:09.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4c2246640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc1ecd40 con 0x7fa4bc071110 2023-10-30T14:06:09.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4bc0c6bd0 0x7fa4bc158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:09.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4bc0c6bd0 0x7fa4bc158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46144/0 (socket says 172.21.15.46:46144) 2023-10-30T14:06:09.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4bbfff640 1 -- 172.21.15.46:0/4145522516 learned_addr learned my addr 172.21.15.46:0/4145522516 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:06:09.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4baffd640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4bc071110 0x7fa4bc145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:09.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4bb7fe640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4bc06f7a0 0x7fa4bc1f0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4bbfff640 1 -- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4bc071110 msgr2=0x7fa4bc145780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4bbfff640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4bc071110 0x7fa4bc145780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4bbfff640 1 -- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4bc06f7a0 msgr2=0x7fa4bc1f0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4bbfff640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4bc06f7a0 0x7fa4bc1f0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4bbfff640 1 -- 172.21.15.46:0/4145522516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac009420 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4bb7fe640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4bc06f7a0 0x7fa4bc1f0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:06:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4bbfff640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4bc0c6bd0 0x7fa4bc158aa0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400caa0 tx=0x7fa4b400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4a3fff640 1 -- 172.21.15.46:0/4145522516 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b400ac00 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4a3fff640 1 -- 172.21.15.46:0/4145522516 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b400ad60 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4c2246640 1 -- 172.21.15.46:0/4145522516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b0009420 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4a3fff640 1 -- 172.21.15.46:0/4145522516 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4012620 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4c2246640 1 -- 172.21.15.46:0/4145522516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4bc1ed2e0 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.311+0000 7fa4c2246640 1 -- 172.21.15.46:0/4145522516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4bc1588a0 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.315+0000 7fa4a3fff640 1 -- 172.21.15.46:0/4145522516 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4b4007070 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.315+0000 7fa4a3fff640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4a80664a0 0x7fa4a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:09.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.315+0000 7fa4a3fff640 1 -- 172.21.15.46:0/4145522516 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7fa4b4084700 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.315+0000 7fa4bb7fe640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4a80664a0 0x7fa4a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:09.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.315+0000 7fa4bb7fe640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4a80664a0 0x7fa4a8068960 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009ca0 tx=0x7fa4ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:09.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.319+0000 7fa4a3fff640 1 -- 172.21.15.46:0/4145522516 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4b4051310 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.507+0000 7fa4c2246640 1 -- 172.21.15.46:0/4145522516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4bc04e170 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.507+0000 7fa4a3fff640 1 -- 172.21.15.46:0/4145522516 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fa4b4054930 con 0x7fa4bc0c6bd0 2023-10-30T14:06:09.511 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:06:09.511 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:06.729509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:06:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 186 2023-10-30T14:06:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.511+0000 7fa4c2246640 1 -- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4a80664a0 msgr2=0x7fa4a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.511+0000 7fa4c2246640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4a80664a0 0x7fa4a8068960 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009ca0 tx=0x7fa4ac008040 comp rx=0 tx=0).stop 2023-10-30T14:06:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.511+0000 7fa4c2246640 1 -- 172.21.15.46:0/4145522516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4bc0c6bd0 msgr2=0x7fa4bc158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.511+0000 7fa4c2246640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4bc0c6bd0 0x7fa4bc158aa0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400caa0 tx=0x7fa4b400cf60 comp rx=0 tx=0).stop 2023-10-30T14:06:09.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.511+0000 7fa4c2246640 1 -- 172.21.15.46:0/4145522516 shutdown_connections 2023-10-30T14:06:09.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.511+0000 7fa4c2246640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4a80664a0 0x7fa4a8068960 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:09.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.511+0000 7fa4c2246640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4bc0c6bd0 0x7fa4bc158aa0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:09.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.511+0000 7fa4c2246640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4bc071110 0x7fa4bc145780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:09.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.511+0000 7fa4c2246640 1 --2- 172.21.15.46:0/4145522516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4bc06f7a0 0x7fa4bc1f0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.511+0000 7fa4c2246640 1 -- 172.21.15.46:0/4145522516 >> 172.21.15.46:0/4145522516 conn(0x7fa4bc0c7910 msgr2=0x7fa4bc0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.511+0000 7fa4c2246640 1 -- 172.21.15.46:0/4145522516 shutdown_connections 2023-10-30T14:06:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:09.511+0000 7fa4c2246640 1 -- 172.21.15.46:0/4145522516 wait complete. 2023-10-30T14:06:09.532 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T14:06:10.462 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T14:06:10.459+0000 7fa87ff1c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:06:11.533 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:06:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f500d965640 1 Processor -- start 2023-10-30T14:06:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f500d965640 1 -- start start 2023-10-30T14:06:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f500d965640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500806e470 0x7f500806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f500d965640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f50080c8490 0x7f50080c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f500d965640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500806afa0 con 0x7f5008153d70 2023-10-30T14:06:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f500d965640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f500806d780 con 0x7f50080c8490 2023-10-30T14:06:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f500d965640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5008154150 con 0x7f500806e470 2023-10-30T14:06:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f4ffffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500806e470 0x7f500806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f50077fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f50080c8490 0x7f50080c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f4ffffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500806e470 0x7f500806ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:37414/0 (socket says 172.21.15.46:37414) 2023-10-30T14:06:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f5006ffd640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f5008153d70 0x7f50080cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:41094/0 (socket says 172.21.15.46:41094) 2023-10-30T14:06:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f4ffffff640 1 -- 172.21.15.46:0/1711254665 learned_addr learned my addr 172.21.15.46:0/1711254665 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:06:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f4ffffff640 1 -- 172.21.15.46:0/1711254665 >> v2:172.21.15.154:3300/0 conn(0x7f50080c8490 msgr2=0x7f50080c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:06:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f4ffffff640 1 --2- 172.21.15.46:0/1711254665 >> v2:172.21.15.154:3300/0 conn(0x7f50080c8490 0x7f50080c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f4ffffff640 1 -- 172.21.15.46:0/1711254665 >> v1:172.21.15.154:6789/0 conn(0x7f5008153d70 legacy=0x7f50080cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.687+0000 7f4ffffff640 1 -- 172.21.15.46:0/1711254665 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500806e030 con 0x7f500806e470 2023-10-30T14:06:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.691+0000 7f4ffffff640 1 --2- 172.21.15.46:0/1711254665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500806e470 0x7f500806ca80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009870 tx=0x7f4ff001c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b572e3583d919a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.691+0000 7f50067fc640 1 -- 172.21.15.46:0/1711254665 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff00aa670 con 0x7f500806e470 2023-10-30T14:06:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.691+0000 7f50067fc640 1 -- 172.21.15.46:0/1711254665 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff00aa7d0 con 0x7f500806e470 2023-10-30T14:06:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.691+0000 7f50067fc640 1 -- 172.21.15.46:0/1711254665 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff00aab60 con 0x7f500806e470 2023-10-30T14:06:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.691+0000 7f500d965640 1 -- 172.21.15.46:0/1711254665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500806e470 msgr2=0x7f500806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.691+0000 7f500d965640 1 --2- 172.21.15.46:0/1711254665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500806e470 0x7f500806ca80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009870 tx=0x7f4ff001c510 comp rx=0 tx=0).stop 2023-10-30T14:06:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.691+0000 7f500d965640 1 -- 172.21.15.46:0/1711254665 shutdown_connections 2023-10-30T14:06:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.691+0000 7f500d965640 1 --2- 172.21.15.46:0/1711254665 >> v2:172.21.15.154:3300/0 conn(0x7f50080c8490 0x7f50080c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.691+0000 7f500d965640 1 --2- 172.21.15.46:0/1711254665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500806e470 0x7f500806ca80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.691+0000 7f500d965640 1 -- 172.21.15.46:0/1711254665 >> 172.21.15.46:0/1711254665 conn(0x7f5008071ea0 msgr2=0x7f5008065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.691+0000 7f500d965640 1 -- 172.21.15.46:0/1711254665 shutdown_connections 2023-10-30T14:06:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.691+0000 7f500d965640 1 -- 172.21.15.46:0/1711254665 wait complete. 2023-10-30T14:06:11.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f500d965640 1 Processor -- start 2023-10-30T14:06:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f500d965640 1 -- start start 2023-10-30T14:06:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f500d965640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f500806e470 0x7f50081557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f500d965640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50080c8490 0x7f500813e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f500d965640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5008153d70 0x7f5008142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:11.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f500d965640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50081f2a10 con 0x7f500806e470 2023-10-30T14:06:11.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f500d965640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50080c7f60 con 0x7f5008153d70 2023-10-30T14:06:11.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f500d965640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5008143260 con 0x7f50080c8490 2023-10-30T14:06:11.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f5006ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f500806e470 0x7f50081557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:11.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f5006ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f500806e470 0x7f50081557e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46154/0 (socket says 172.21.15.46:46154) 2023-10-30T14:06:11.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f5006ffd640 1 -- 172.21.15.46:0/463515870 learned_addr learned my addr 172.21.15.46:0/463515870 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:06:11.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f4ffffff640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50080c8490 0x7f500813e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:11.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f50077fe640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5008153d70 0x7f5008142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:11.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f5006ffd640 1 -- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50080c8490 msgr2=0x7f500813e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:11.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f5006ffd640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50080c8490 0x7f500813e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:11.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f5006ffd640 1 -- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5008153d70 msgr2=0x7f5008142bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:11.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f5006ffd640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5008153d70 0x7f5008142bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:11.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f5006ffd640 1 -- 172.21.15.46:0/463515870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff00094d0 con 0x7f500806e470 2023-10-30T14:06:11.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f5006ffd640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f500806e470 0x7f50081557e0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800c430 tx=0x7f4ff800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:11.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f5004ff9640 1 -- 172.21.15.46:0/463515870 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff80074e0 con 0x7f500806e470 2023-10-30T14:06:11.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f5004ff9640 1 -- 172.21.15.46:0/463515870 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff8007640 con 0x7f500806e470 2023-10-30T14:06:11.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f5004ff9640 1 -- 172.21.15.46:0/463515870 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8004a10 con 0x7f500806e470 2023-10-30T14:06:11.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f500d965640 1 -- 172.21.15.46:0/463515870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5008143540 con 0x7f500806e470 2023-10-30T14:06:11.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.695+0000 7f500d965640 1 -- 172.21.15.46:0/463515870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50081f2cd0 con 0x7f500806e470 2023-10-30T14:06:11.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.699+0000 7f500d965640 1 -- 172.21.15.46:0/463515870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5008143a60 con 0x7f500806e470 2023-10-30T14:06:11.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.699+0000 7f5004ff9640 1 -- 172.21.15.46:0/463515870 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4ff8004b70 con 0x7f500806e470 2023-10-30T14:06:11.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.699+0000 7f5004ff9640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4fe80664a0 0x7f4fe8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:11.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.699+0000 7f5004ff9640 1 -- 172.21.15.46:0/463515870 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7f4ff8084090 con 0x7f500806e470 2023-10-30T14:06:11.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.699+0000 7f4ffffff640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4fe80664a0 0x7f4fe8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:11.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.703+0000 7f4ffffff640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4fe80664a0 0x7f4fe8068960 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00ae000 tx=0x7f4ff00b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:11.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:11.703+0000 7f5004ff9640 1 -- 172.21.15.46:0/463515870 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4ff8050c70 con 0x7f500806e470 2023-10-30T14:06:12.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.011+0000 7f500d965640 1 -- 172.21.15.46:0/463515870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f500804e170 con 0x7f500806e470 2023-10-30T14:06:12.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f5004ff9640 1 -- 172.21.15.46:0/463515870 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f4ff8054290 con 0x7f500806e470 2023-10-30T14:06:12.016 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:06:12.016 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:06.729509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:06:12.016 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 187 2023-10-30T14:06:12.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f500d965640 1 -- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4fe80664a0 msgr2=0x7f4fe8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:12.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f500d965640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4fe80664a0 0x7f4fe8068960 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00ae000 tx=0x7f4ff00b5040 comp rx=0 tx=0).stop 2023-10-30T14:06:12.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f500d965640 1 -- 172.21.15.46:0/463515870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f500806e470 msgr2=0x7f50081557e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:12.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f500d965640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f500806e470 0x7f50081557e0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800c430 tx=0x7f4ff800c8f0 comp rx=0 tx=0).stop 2023-10-30T14:06:12.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f500d965640 1 -- 172.21.15.46:0/463515870 shutdown_connections 2023-10-30T14:06:12.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f500d965640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4fe80664a0 0x7f4fe8068960 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:12.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f500d965640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5008153d70 0x7f5008142bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:12.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f500d965640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50080c8490 0x7f500813e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:12.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f500d965640 1 --2- 172.21.15.46:0/463515870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f500806e470 0x7f50081557e0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:12.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f500d965640 1 -- 172.21.15.46:0/463515870 >> 172.21.15.46:0/463515870 conn(0x7f5008071ea0 msgr2=0x7f500814e430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:12.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f500d965640 1 -- 172.21.15.46:0/463515870 shutdown_connections 2023-10-30T14:06:12.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:12.015+0000 7f500d965640 1 -- 172.21.15.46:0/463515870 wait complete. 2023-10-30T14:06:12.039 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T14:06:12.039 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 47 secs before thrashing 2023-10-30T14:06:25.174 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 66% (144/218) Updating files: 67% (147/218) Updating files: 68% (149/218) Updating files: 69% (151/218) Updating files: 70% (153/218) Updating files: 71% (155/218) Updating files: 72% (157/218) Updating files: 73% (160/218) Updating files: 74% (162/218) Updating files: 75% (164/218) Updating files: 76% (166/218) Updating files: 77% (168/218) Updating files: 78% (171/218) Updating files: 79% (173/218) Updating files: 80% (175/218) Updating files: 81% (177/218) Updating files: 82% (179/218) Updating files: 83% (181/218) Updating files: 84% (184/218) Updating files: 85% (186/218) Updating files: 86% (188/218) Updating files: 87% (190/218) Updating files: 88% (192/218) Updating files: 89% (195/218) Updating files: 90% (197/218) Updating files: 91% (199/218) Updating files: 92% (201/218) Updating files: 93% (203/218) Updating files: 94% (205/218) Updating files: 95% (208/218) Updating files: 96% (210/218) Updating files: 96% (211/218) Updating files: 97% (212/218) Updating files: 98% (214/218) Updating files: 99% (216/218) Updating files: 100% (218/218) Updating files: 100% (218/218), done. 2023-10-30T14:06:25.285 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 78ff1f0a5df 0.83 2023-10-30T14:06:25.293 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.83 2023-10-30T14:06:25.624 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:06:25.628 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:06:26.090 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 84 -eq 48 ] 2023-10-30T14:06:26.091 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.84 2023-10-30T14:06:26.091 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.84 2023-10-30T14:06:26.091 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.84 2023-10-30T14:06:26.091 INFO:tasks.workunit.client.0.smithi046.stdout:v0.84 2023-10-30T14:06:37.854 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 37% (134/356) Updating files: 38% (136/356) Updating files: 39% (139/356) Updating files: 40% (143/356) Updating files: 41% (146/356) Updating files: 42% (150/356) Updating files: 43% (154/356) Updating files: 44% (157/356) Updating files: 45% (161/356) Updating files: 46% (164/356) Updating files: 47% (168/356) Updating files: 48% (171/356) Updating files: 48% (172/356) Updating files: 49% (175/356) Updating files: 50% (178/356) Updating files: 51% (182/356) Updating files: 52% (186/356) Updating files: 53% (189/356) Updating files: 54% (193/356) Updating files: 55% (196/356) Updating files: 56% (200/356) Updating files: 57% (203/356) Updating files: 58% (207/356) Updating files: 59% (211/356) Updating files: 60% (214/356) Updating files: 61% (218/356) Updating files: 62% (221/356) Updating files: 63% (225/356) Updating files: 64% (228/356) Updating files: 65% (232/356) Updating files: 66% (235/356) Updating files: 67% (239/356) Updating files: 68% (243/356) Updating files: 69% (246/356) Updating files: 70% (250/356) Updating files: 71% (253/356) Updating files: 72% (257/356) Updating files: 73% (260/356) Updating files: 74% (264/356) Updating files: 75% (267/356) Updating files: 76% (271/356) Updating files: 77% (275/356) Updating files: 78% (278/356) Updating files: 79% (282/356) Updating files: 80% (285/356) Updating files: 81% (289/356) Updating files: 82% (292/356) Updating files: 83% (296/356) Updating files: 84% (300/356) Updating files: 85% (303/356) Updating files: 86% (307/356) Updating files: 87% (310/356) Updating files: 88% (314/356) Updating files: 89% (317/356) Updating files: 90% (321/356) Updating files: 91% (324/356) Updating files: 92% (328/356) Updating files: 93% (332/356) Updating files: 94% (335/356) Updating files: 95% (339/356) Updating files: 96% (342/356) Updating files: 97% (346/356) Updating files: 98% (349/356) Updating files: 99% (353/356) Updating files: 100% (356/356) Updating files: 100% (356/356), done. 2023-10-30T14:06:37.887 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 8336f81c5c7 0.84 2023-10-30T14:06:37.896 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.84 2023-10-30T14:06:38.858 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 85 -eq 48 ] 2023-10-30T14:06:38.858 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.85 2023-10-30T14:06:38.859 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.85 2023-10-30T14:06:38.859 INFO:tasks.workunit.client.0.smithi046.stdout:v0.85 2023-10-30T14:06:38.859 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.85 2023-10-30T14:06:46.154 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 13% (34/243) Updating files: 14% (35/243) Updating files: 15% (37/243) Updating files: 16% (39/243) Updating files: 17% (42/243) Updating files: 18% (44/243) Updating files: 19% (47/243) Updating files: 20% (49/243) Updating files: 21% (52/243) Updating files: 22% (54/243) Updating files: 23% (56/243) Updating files: 24% (59/243) Updating files: 25% (61/243) Updating files: 26% (64/243) Updating files: 27% (66/243) Updating files: 28% (69/243) Updating files: 29% (71/243) Updating files: 30% (73/243) Updating files: 31% (76/243) Updating files: 32% (78/243) Updating files: 33% (81/243) Updating files: 34% (83/243) Updating files: 35% (86/243) Updating files: 36% (88/243) Updating files: 36% (89/243) Updating files: 37% (90/243) Updating files: 38% (93/243) Updating files: 39% (95/243) Updating files: 40% (98/243) Updating files: 41% (100/243) Updating files: 42% (103/243) Updating files: 43% (105/243) Updating files: 44% (107/243) Updating files: 45% (110/243) Updating files: 46% (112/243) Updating files: 47% (115/243) Updating files: 48% (117/243) Updating files: 49% (120/243) Updating files: 50% (122/243) Updating files: 51% (124/243) Updating files: 52% (127/243) Updating files: 53% (129/243) Updating files: 54% (132/243) Updating files: 55% (134/243) Updating files: 56% (137/243) Updating files: 57% (139/243) Updating files: 58% (141/243) Updating files: 59% (144/243) Updating files: 60% (146/243) Updating files: 61% (149/243) Updating files: 62% (151/243) Updating files: 63% (154/243) Updating files: 64% (156/243) Updating files: 65% (158/243) Updating files: 66% (161/243) Updating files: 67% (163/243) Updating files: 68% (166/243) Updating files: 69% (168/243) Updating files: 70% (171/243) Updating files: 71% (173/243) Updating files: 72% (175/243) Updating files: 72% (177/243) Updating files: 73% (178/243) Updating files: 74% (180/243) Updating files: 75% (183/243) Updating files: 76% (185/243) Updating files: 77% (188/243) Updating files: 78% (190/243) Updating files: 79% (192/243) Updating files: 80% (195/243) Updating files: 81% (197/243) Updating files: 82% (200/243) Updating files: 83% (202/243) Updating files: 84% (205/243) Updating files: 85% (207/243) Updating files: 86% (209/243) Updating files: 87% (212/243) Updating files: 88% (214/243) Updating files: 89% (217/243) Updating files: 90% (219/243) Updating files: 91% (222/243) Updating files: 92% (224/243) Updating files: 93% (226/243) Updating files: 94% (229/243) Updating files: 95% (231/243) Updating files: 96% (234/243) Updating files: 97% (236/243) Updating files: 98% (239/243) Updating files: 99% (241/243) Updating files: 100% (243/243) Updating files: 100% (243/243), done. 2023-10-30T14:06:46.203 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at a0c22842db9 0.85 2023-10-30T14:06:46.214 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.85 2023-10-30T14:06:46.305 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 86 -eq 48 ] 2023-10-30T14:06:46.305 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.86 2023-10-30T14:06:46.306 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.86 2023-10-30T14:06:46.306 INFO:tasks.workunit.client.0.smithi046.stdout:v0.86 2023-10-30T14:06:46.306 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.86 2023-10-30T14:06:55.737 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:06:55.742 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:06:57.412 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 9% (41/447) Updating files: 10% (45/447) Updating files: 11% (50/447) Updating files: 11% (52/447) Updating files: 12% (54/447) Updating files: 13% (59/447) Updating files: 14% (63/447) Updating files: 15% (68/447) Updating files: 16% (72/447) Updating files: 17% (76/447) Updating files: 18% (81/447) Updating files: 19% (85/447) Updating files: 19% (87/447) Updating files: 20% (90/447) Updating files: 21% (94/447) Updating files: 22% (99/447) Updating files: 23% (103/447) Updating files: 24% (108/447) Updating files: 25% (112/447) Updating files: 26% (117/447) Updating files: 27% (121/447) Updating files: 28% (126/447) Updating files: 29% (130/447) Updating files: 30% (135/447) Updating files: 31% (139/447) Updating files: 31% (140/447) Updating files: 32% (144/447) Updating files: 33% (148/447) Updating files: 34% (152/447) Updating files: 35% (157/447) Updating files: 36% (161/447) Updating files: 37% (166/447) Updating files: 38% (170/447) Updating files: 38% (172/447) Updating files: 39% (175/447) Updating files: 40% (179/447) Updating files: 41% (184/447) Updating files: 42% (188/447) Updating files: 43% (193/447) Updating files: 44% (197/447) Updating files: 45% (202/447) Updating files: 46% (206/447) Updating files: 47% (211/447) Updating files: 48% (215/447) Updating files: 49% (220/447) Updating files: 50% (224/447) Updating files: 51% (228/447) Updating files: 52% (233/447) Updating files: 53% (237/447) Updating files: 54% (242/447) Updating files: 54% (244/447) Updating files: 55% (246/447) Updating files: 56% (251/447) Updating files: 57% (255/447) Updating files: 58% (260/447) Updating files: 59% (264/447) Updating files: 60% (269/447) Updating files: 61% (273/447) Updating files: 62% (278/447) Updating files: 63% (282/447) Updating files: 64% (287/447) Updating files: 65% (291/447) Updating files: 66% (296/447) Updating files: 67% (300/447) Updating files: 68% (304/447) Updating files: 69% (309/447) Updating files: 70% (313/447) Updating files: 71% (318/447) Updating files: 72% (322/447) Updating files: 73% (327/447) Updating files: 74% (331/447) Updating files: 75% (336/447) Updating files: 76% (340/447) Updating files: 77% (345/447) Updating files: 78% (349/447) Updating files: 79% (354/447) Updating files: 80% (358/447) Updating files: 81% (363/447) Updating files: 82% (367/447) Updating files: 83% (372/447) Updating files: 84% (376/447) Updating files: 85% (380/447) Updating files: 86% (385/447) Updating files: 87% (389/447) Updating files: 88% (394/447) Updating files: 89% (398/447) Updating files: 90% (403/447) Updating files: 91% (407/447) Updating files: 92% (412/447) Updating files: 93% (416/447) Updating files: 94% (421/447) Updating files: 95% (425/447) Updating files: 96% (430/447) Updating files: 97% (434/447) Updating files: 98% (439/447) Updating files: 99% (443/447) Updating files: 99% (444/447) Updating files: 100% (447/447) Updating files: 100% (447/447), done. 2023-10-30T14:06:57.460 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 97dcc0539df 0.86 2023-10-30T14:06:57.469 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.86 2023-10-30T14:06:58.069 INFO:tasks.workunit.client.0.smithi046.stdout:v0.87 2023-10-30T14:06:58.069 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 87 -eq 48 ] 2023-10-30T14:06:58.069 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.87 2023-10-30T14:06:58.070 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.87 2023-10-30T14:06:58.070 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.87 2023-10-30T14:06:59.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:06:59.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.214+0000 7f1c4993e640 1 Processor -- start 2023-10-30T14:06:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.214+0000 7f1c4993e640 1 -- start start 2023-10-30T14:06:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.214+0000 7f1c4993e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1c441538a0 0x7f1c4415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.214+0000 7f1c4993e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c440ca0e0 0x7f1c4415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.214+0000 7f1c4993e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440cb800 con 0x7f1c440cf160 2023-10-30T14:06:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.214+0000 7f1c4993e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c4415dc30 con 0x7f1c440ca0e0 2023-10-30T14:06:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.214+0000 7f1c4993e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c440cb330 con 0x7f1c441538a0 2023-10-30T14:06:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.214+0000 7f1c427fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c440ca0e0 0x7f1c4415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c427fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c440ca0e0 0x7f1c4415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39670/0 (socket says 172.21.15.46:39670) 2023-10-30T14:06:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c427fc640 1 -- 172.21.15.46:0/717080523 learned_addr learned my addr 172.21.15.46:0/717080523 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:06:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c437fe640 1 --1- 172.21.15.46:0/717080523 >> v1:172.21.15.46:6789/0 conn(0x7f1c440cf160 0x7f1c440cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54272/0 (socket says 172.21.15.46:54272) 2023-10-30T14:06:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c42ffd640 1 --2- 172.21.15.46:0/717080523 >> v2:172.21.15.154:3300/0 conn(0x7f1c441538a0 0x7f1c4415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c427fc640 1 -- 172.21.15.46:0/717080523 >> v2:172.21.15.154:3300/0 conn(0x7f1c441538a0 msgr2=0x7f1c4415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:06:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c427fc640 1 --2- 172.21.15.46:0/717080523 >> v2:172.21.15.154:3300/0 conn(0x7f1c441538a0 0x7f1c4415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c427fc640 1 -- 172.21.15.46:0/717080523 >> v1:172.21.15.46:6789/0 conn(0x7f1c440cf160 legacy=0x7f1c440cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c427fc640 1 -- 172.21.15.46:0/717080523 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c440cad70 con 0x7f1c440ca0e0 2023-10-30T14:06:59.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c41ffb640 1 -- 172.21.15.46:0/717080523 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891279663 0 0) 0x7f1c440cb800 con 0x7f1c440cf160 2023-10-30T14:06:59.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c427fc640 1 --2- 172.21.15.46:0/717080523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c440ca0e0 0x7f1c4415a090 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f1c440cfd10 tx=0x7f1c3801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ae2ac1bc3d09a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:59.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c41ffb640 1 -- 172.21.15.46:0/717080523 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c380a89f0 con 0x7f1c440ca0e0 2023-10-30T14:06:59.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c41ffb640 1 -- 172.21.15.46:0/717080523 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c380a8b50 con 0x7f1c440ca0e0 2023-10-30T14:06:59.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.218+0000 7f1c41ffb640 1 -- 172.21.15.46:0/717080523 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c380b0440 con 0x7f1c440ca0e0 2023-10-30T14:06:59.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 -- 172.21.15.46:0/717080523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c440ca0e0 msgr2=0x7f1c4415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 --2- 172.21.15.46:0/717080523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c440ca0e0 0x7f1c4415a090 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f1c440cfd10 tx=0x7f1c3801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 -- 172.21.15.46:0/717080523 shutdown_connections 2023-10-30T14:06:59.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 --2- 172.21.15.46:0/717080523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c440ca0e0 0x7f1c4415a090 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 --2- 172.21.15.46:0/717080523 >> v2:172.21.15.154:3300/0 conn(0x7f1c441538a0 0x7f1c4415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 -- 172.21.15.46:0/717080523 >> 172.21.15.46:0/717080523 conn(0x7f1c4406bfe0 msgr2=0x7f1c4406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:59.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 -- 172.21.15.46:0/717080523 shutdown_connections 2023-10-30T14:06:59.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 -- 172.21.15.46:0/717080523 wait complete. 2023-10-30T14:06:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 Processor -- start 2023-10-30T14:06:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 -- start start 2023-10-30T14:06:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c440ca0e0 0x7f1c440b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c440cf160 0x7f1c44158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c441538a0 0x7f1c44158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c440b3b10 con 0x7f1c440cf160 2023-10-30T14:06:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c441e35b0 con 0x7f1c441538a0 2023-10-30T14:06:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c4993e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c440ce8f0 con 0x7f1c440ca0e0 2023-10-30T14:06:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c427fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c440cf160 0x7f1c44158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c427fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c440cf160 0x7f1c44158330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39686/0 (socket says 172.21.15.46:39686) 2023-10-30T14:06:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c427fc640 1 -- 172.21.15.46:0/1858845982 learned_addr learned my addr 172.21.15.46:0/1858845982 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:06:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c42ffd640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c440ca0e0 0x7f1c440b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.222+0000 7f1c437fe640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c441538a0 0x7f1c44158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c427fc640 1 -- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c440ca0e0 msgr2=0x7f1c440b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c427fc640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c440ca0e0 0x7f1c440b4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c427fc640 1 -- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c441538a0 msgr2=0x7f1c44158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c427fc640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c441538a0 0x7f1c44158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c427fc640 1 -- 172.21.15.46:0/1858845982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c380094d0 con 0x7f1c440cf160 2023-10-30T14:06:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c42ffd640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c440ca0e0 0x7f1c440b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:06:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c427fc640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c440cf160 0x7f1c44158330 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f1c3801b900 tx=0x7f1c3801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c2bfff640 1 -- 172.21.15.46:0/1858845982 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c380ba600 con 0x7f1c440cf160 2023-10-30T14:06:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c2bfff640 1 -- 172.21.15.46:0/1858845982 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c380ba760 con 0x7f1c440cf160 2023-10-30T14:06:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c2bfff640 1 -- 172.21.15.46:0/1858845982 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c380bab10 con 0x7f1c440cf160 2023-10-30T14:06:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c4993e640 1 -- 172.21.15.46:0/1858845982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c440b4a40 con 0x7f1c440cf160 2023-10-30T14:06:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c4993e640 1 -- 172.21.15.46:0/1858845982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c440b4ce0 con 0x7f1c440cf160 2023-10-30T14:06:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c4993e640 1 -- 172.21.15.46:0/1858845982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c440b2a50 con 0x7f1c440cf160 2023-10-30T14:06:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c2bfff640 1 -- 172.21.15.46:0/1858845982 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1c380bac70 con 0x7f1c440cf160 2023-10-30T14:06:59.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c2bfff640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c300664a0 0x7f1c30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:59.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c2bfff640 1 -- 172.21.15.46:0/1858845982 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7f1c38122ca0 con 0x7f1c440cf160 2023-10-30T14:06:59.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.226+0000 7f1c42ffd640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c300664a0 0x7f1c30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:59.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.230+0000 7f1c42ffd640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c300664a0 0x7f1c30068960 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1c3400b9f0 tx=0x7f1c34008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:59.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.230+0000 7f1c2bfff640 1 -- 172.21.15.46:0/1858845982 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1c380ef880 con 0x7f1c440cf160 2023-10-30T14:06:59.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 -- 172.21.15.46:0/1858845982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c4404e170 con 0x7f1c440cf160 2023-10-30T14:06:59.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c2bfff640 1 -- 172.21.15.46:0/1858845982 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f1c380b5020 con 0x7f1c440cf160 2023-10-30T14:06:59.429 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:06:59.429 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:17.041488+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:06:59.430 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 188 2023-10-30T14:06:59.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 -- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c300664a0 msgr2=0x7f1c30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c300664a0 0x7f1c30068960 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1c3400b9f0 tx=0x7f1c34008040 comp rx=0 tx=0).stop 2023-10-30T14:06:59.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 -- 172.21.15.46:0/1858845982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c440cf160 msgr2=0x7f1c44158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c440cf160 0x7f1c44158330 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f1c3801b900 tx=0x7f1c3801ba90 comp rx=0 tx=0).stop 2023-10-30T14:06:59.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 -- 172.21.15.46:0/1858845982 shutdown_connections 2023-10-30T14:06:59.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1c300664a0 0x7f1c30068960 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c441538a0 0x7f1c44158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c440cf160 0x7f1c44158330 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 --2- 172.21.15.46:0/1858845982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c440ca0e0 0x7f1c440b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 -- 172.21.15.46:0/1858845982 >> 172.21.15.46:0/1858845982 conn(0x7f1c4406bfe0 msgr2=0x7f1c441e5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:59.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 -- 172.21.15.46:0/1858845982 shutdown_connections 2023-10-30T14:06:59.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.426+0000 7f1c4993e640 1 -- 172.21.15.46:0/1858845982 wait complete. 2023-10-30T14:06:59.450 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T14:06:59.451 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:06:59.451 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:06:59.478 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:06:59.479 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:06:59.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.658+0000 7fd5512c9640 1 Processor -- start 2023-10-30T14:06:59.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.658+0000 7fd5512c9640 1 -- start start 2023-10-30T14:06:59.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.658+0000 7fd5512c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd54c071110 0x7fd54c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:59.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.658+0000 7fd5512c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c066cf0 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.658+0000 7fd5512c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0c70d0 con 0x7fd54c06f7a0 2023-10-30T14:06:59.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.658+0000 7fd5512c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0700c0 con 0x7fd54c071110 2023-10-30T14:06:59.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.658+0000 7fd543fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd54c0c6bd0 0x7fd54c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54282/0 (socket says 172.21.15.46:54282) 2023-10-30T14:06:59.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.658+0000 7fd543fff640 1 -- 172.21.15.46:0/2432279562 learned_addr learned my addr 172.21.15.46:0/2432279562 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:06:59.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.658+0000 7fd54b7fe640 1 --2- 172.21.15.46:0/2432279562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd54c071110 0x7fd54c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:59.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.658+0000 7fd54a7fc640 1 -- 172.21.15.46:0/2432279562 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284926434 0 0) 0x7fd54c066cf0 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.658+0000 7fd54a7fc640 1 -- 172.21.15.46:0/2432279562 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd538003530 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.662+0000 7fd54a7fc640 1 -- 172.21.15.46:0/2432279562 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1639521927 0 0) 0x7fd538003530 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.662+0000 7fd54b7fe640 1 -- 172.21.15.46:0/2432279562 >> v1:172.21.15.154:6789/0 conn(0x7fd54c06f7a0 legacy=0x7fd54c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.662+0000 7fd54b7fe640 1 -- 172.21.15.46:0/2432279562 >> v1:172.21.15.46:6789/0 conn(0x7fd54c0c6bd0 legacy=0x7fd54c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.662+0000 7fd54b7fe640 1 -- 172.21.15.46:0/2432279562 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0c7390 con 0x7fd54c071110 2023-10-30T14:06:59.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.662+0000 7fd54b7fe640 1 --2- 172.21.15.46:0/2432279562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd54c071110 0x7fd54c06fb80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd544009dc0 tx=0x7fd54401f080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f22db268e38517e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:59.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.662+0000 7fd54a7fc640 1 -- 172.21.15.46:0/2432279562 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534001680 con 0x7fd54c071110 2023-10-30T14:06:59.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.662+0000 7fd54a7fc640 1 -- 172.21.15.46:0/2432279562 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5440a8c10 con 0x7fd54c071110 2023-10-30T14:06:59.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.662+0000 7fd54a7fc640 1 -- 172.21.15.46:0/2432279562 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5440b0440 con 0x7fd54c071110 2023-10-30T14:06:59.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 -- 172.21.15.46:0/2432279562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd54c071110 msgr2=0x7fd54c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 --2- 172.21.15.46:0/2432279562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd54c071110 0x7fd54c06fb80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd544009dc0 tx=0x7fd54401f080 comp rx=0 tx=0).stop 2023-10-30T14:06:59.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 -- 172.21.15.46:0/2432279562 shutdown_connections 2023-10-30T14:06:59.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 --2- 172.21.15.46:0/2432279562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd54c071110 0x7fd54c06fb80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 -- 172.21.15.46:0/2432279562 >> 172.21.15.46:0/2432279562 conn(0x7fd54c0c7910 msgr2=0x7fd54c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:59.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 -- 172.21.15.46:0/2432279562 shutdown_connections 2023-10-30T14:06:59.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 -- 172.21.15.46:0/2432279562 wait complete. 2023-10-30T14:06:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 Processor -- start 2023-10-30T14:06:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 -- start start 2023-10-30T14:06:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd54c06f7a0 0x7fd54c14abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:59.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd54c071110 0x7fd54c0ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:59.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd54c0c6bd0 0x7fd54c0ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:59.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c1ec710 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0c9c60 con 0x7fd54c06f7a0 2023-10-30T14:06:59.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd5512c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c06fd30 con 0x7fd54c071110 2023-10-30T14:06:59.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd54b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd54c0c6bd0 0x7fd54c0ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:59.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd54b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd54c0c6bd0 0x7fd54c0ca630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39700/0 (socket says 172.21.15.46:39700) 2023-10-30T14:06:59.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd54b7fe640 1 -- 172.21.15.46:0/3855348390 learned_addr learned my addr 172.21.15.46:0/3855348390 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:06:59.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd54affd640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd54c06f7a0 0x7fd54c14abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:59.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd543fff640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd54c071110 0x7fd54c0ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:59.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd54b7fe640 1 -- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd54c071110 msgr2=0x7fd54c0ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd54b7fe640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd54c071110 0x7fd54c0ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd54b7fe640 1 -- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd54c06f7a0 msgr2=0x7fd54c14abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd54b7fe640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd54c06f7a0 0x7fd54c14abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd54b7fe640 1 -- 172.21.15.46:0/3855348390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd544009a70 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.666+0000 7fd54b7fe640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd54c0c6bd0 0x7fd54c0ca630 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fd544009eb0 tx=0x7fd54401be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:59.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.670+0000 7fd5437fe640 1 -- 172.21.15.46:0/3855348390 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544009450 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.670+0000 7fd5437fe640 1 -- 172.21.15.46:0/3855348390 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5440ba410 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.670+0000 7fd5437fe640 1 -- 172.21.15.46:0/3855348390 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5440ba780 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.670+0000 7fd5512c9640 1 -- 172.21.15.46:0/3855348390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c1ec920 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.670+0000 7fd5512c9640 1 -- 172.21.15.46:0/3855348390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c1ecee0 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.670+0000 7fd5512c9640 1 -- 172.21.15.46:0/3855348390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c04e170 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.670+0000 7fd5437fe640 1 -- 172.21.15.46:0/3855348390 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd5440b7050 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.670+0000 7fd5437fe640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd52c0664d0 0x7fd52c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:06:59.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.670+0000 7fd5437fe640 1 -- 172.21.15.46:0/3855348390 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7fd544122b70 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.670+0000 7fd54affd640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd52c0664d0 0x7fd52c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:06:59.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.674+0000 7fd54affd640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd52c0664d0 0x7fd52c068990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd53c000f70 tx=0x7fd53c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:06:59.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.674+0000 7fd5437fe640 1 -- 172.21.15.46:0/3855348390 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd5440ef750 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 -- 172.21.15.46:0/3855348390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd54c0c30d0 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5437fe640 1 -- 172.21.15.46:0/3855348390 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fd5440b5070 con 0x7fd54c0c6bd0 2023-10-30T14:06:59.874 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:06:59.874 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:17.041488+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:06:59.875 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 188 2023-10-30T14:06:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 -- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd52c0664d0 msgr2=0x7fd52c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd52c0664d0 0x7fd52c068990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd53c000f70 tx=0x7fd53c009040 comp rx=0 tx=0).stop 2023-10-30T14:06:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 -- 172.21.15.46:0/3855348390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd54c0c6bd0 msgr2=0x7fd54c0ca630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:06:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd54c0c6bd0 0x7fd54c0ca630 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fd544009eb0 tx=0x7fd54401be20 comp rx=0 tx=0).stop 2023-10-30T14:06:59.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 -- 172.21.15.46:0/3855348390 shutdown_connections 2023-10-30T14:06:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd52c0664d0 0x7fd52c068990 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd54c0c6bd0 0x7fd54c0ca630 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd54c071110 0x7fd54c0ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 --2- 172.21.15.46:0/3855348390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd54c06f7a0 0x7fd54c14abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:06:59.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 -- 172.21.15.46:0/3855348390 >> 172.21.15.46:0/3855348390 conn(0x7fd54c0c7910 msgr2=0x7fd54c0c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:06:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 -- 172.21.15.46:0/3855348390 shutdown_connections 2023-10-30T14:06:59.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:06:59.870+0000 7fd5512c9640 1 -- 172.21.15.46:0/3855348390 wait complete. 2023-10-30T14:06:59.895 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:07:01.896 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc83d009640 1 Processor -- start 2023-10-30T14:07:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc83d009640 1 -- start start 2023-10-30T14:07:02.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc83d009640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8380c6b80 0x7fc8380bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:02.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc83d009640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8380c0a50 con 0x7fc83813f4a0 2023-10-30T14:07:02.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc83d009640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83806f6e0 con 0x7fc8380c6f60 2023-10-30T14:07:02.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc83d009640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc83806f9c0 con 0x7fc8380c6b80 2023-10-30T14:07:02.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc836d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8380c6b80 0x7fc8380bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:02.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc83759e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc83813f4a0 0x7fc8381566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54288/0 (socket says 172.21.15.46:54288) 2023-10-30T14:07:02.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc83759e640 1 -- 172.21.15.46:0/1156430030 learned_addr learned my addr 172.21.15.46:0/1156430030 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc83659c640 1 -- 172.21.15.46:0/1156430030 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952632988 0 0) 0x7fc8380c0a50 con 0x7fc83813f4a0 2023-10-30T14:07:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc83659c640 1 -- 172.21.15.46:0/1156430030 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc824005740 con 0x7fc83813f4a0 2023-10-30T14:07:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc836d9d640 1 -- 172.21.15.46:0/1156430030 >> v1:172.21.15.154:6789/0 conn(0x7fc8380c6f60 legacy=0x7fc83813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc836d9d640 1 -- 172.21.15.46:0/1156430030 >> v1:172.21.15.46:6789/0 conn(0x7fc83813f4a0 legacy=0x7fc8381566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.046+0000 7fc836d9d640 1 -- 172.21.15.46:0/1156430030 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83806fca0 con 0x7fc8380c6b80 2023-10-30T14:07:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.050+0000 7fc836d9d640 1 --2- 172.21.15.46:0/1156430030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8380c6b80 0x7fc8380bfec0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fc8280097c0 tx=0x7fc82801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5400124c54ffc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.050+0000 7fc83659c640 1 -- 172.21.15.46:0/1156430030 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8280ab600 con 0x7fc8380c6b80 2023-10-30T14:07:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.050+0000 7fc83659c640 1 -- 172.21.15.46:0/1156430030 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8280a7050 con 0x7fc8380c6b80 2023-10-30T14:07:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.050+0000 7fc83659c640 1 -- 172.21.15.46:0/1156430030 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8280ab980 con 0x7fc8380c6b80 2023-10-30T14:07:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.050+0000 7fc83d009640 1 -- 172.21.15.46:0/1156430030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8380c6b80 msgr2=0x7fc8380bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.050+0000 7fc83d009640 1 --2- 172.21.15.46:0/1156430030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8380c6b80 0x7fc8380bfec0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fc8280097c0 tx=0x7fc82801c480 comp rx=0 tx=0).stop 2023-10-30T14:07:02.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.050+0000 7fc83d009640 1 -- 172.21.15.46:0/1156430030 shutdown_connections 2023-10-30T14:07:02.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.050+0000 7fc83d009640 1 --2- 172.21.15.46:0/1156430030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8380c6b80 0x7fc8380bfec0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:02.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.050+0000 7fc83d009640 1 -- 172.21.15.46:0/1156430030 >> 172.21.15.46:0/1156430030 conn(0x7fc8380c7910 msgr2=0x7fc8380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:02.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.050+0000 7fc83d009640 1 -- 172.21.15.46:0/1156430030 shutdown_connections 2023-10-30T14:07:02.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.050+0000 7fc83d009640 1 -- 172.21.15.46:0/1156430030 wait complete. 2023-10-30T14:07:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc83d009640 1 Processor -- start 2023-10-30T14:07:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc83d009640 1 -- start start 2023-10-30T14:07:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc83d009640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8380c6b80 0x7fc8380c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc83d009640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8380c6f60 0x7fc838150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:02.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc83d009640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83813f4a0 0x7fc8381508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:02.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc83d009640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83814f910 con 0x7fc8380c6b80 2023-10-30T14:07:02.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc83d009640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8380c52d0 con 0x7fc83813f4a0 2023-10-30T14:07:02.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc83d009640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8381f6670 con 0x7fc8380c6f60 2023-10-30T14:07:02.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc836d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8380c6b80 0x7fc8380c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:02.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc836d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8380c6b80 0x7fc8380c4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39718/0 (socket says 172.21.15.46:39718) 2023-10-30T14:07:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc836d9d640 1 -- 172.21.15.46:0/1037518859 learned_addr learned my addr 172.21.15.46:0/1037518859 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc82ffff640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8380c6f60 0x7fc838150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc83759e640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83813f4a0 0x7fc8381508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:02.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc836d9d640 1 -- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8380c6f60 msgr2=0x7fc838150390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc836d9d640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8380c6f60 0x7fc838150390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc836d9d640 1 -- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83813f4a0 msgr2=0x7fc8381508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc836d9d640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83813f4a0 0x7fc8381508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc836d9d640 1 -- 172.21.15.46:0/1037518859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc828009420 con 0x7fc8380c6b80 2023-10-30T14:07:02.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc82ffff640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8380c6f60 0x7fc838150390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc83759e640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83813f4a0 0x7fc8381508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc836d9d640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8380c6b80 0x7fc8380c4d90 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fc8280ae040 tx=0x7fc828019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc82f7fe640 1 -- 172.21.15.46:0/1037518859 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8280b4430 con 0x7fc8380c6b80 2023-10-30T14:07:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc82f7fe640 1 -- 172.21.15.46:0/1037518859 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8280b4590 con 0x7fc8380c6b80 2023-10-30T14:07:02.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc83d009640 1 -- 172.21.15.46:0/1037518859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83814f350 con 0x7fc8380c6b80 2023-10-30T14:07:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc82f7fe640 1 -- 172.21.15.46:0/1037518859 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8280b4940 con 0x7fc8380c6b80 2023-10-30T14:07:02.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.054+0000 7fc83d009640 1 -- 172.21.15.46:0/1037518859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc838155c50 con 0x7fc8380c6b80 2023-10-30T14:07:02.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.058+0000 7fc83d009640 1 -- 172.21.15.46:0/1037518859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc838155ab0 con 0x7fc8380c6b80 2023-10-30T14:07:02.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.058+0000 7fc82f7fe640 1 -- 172.21.15.46:0/1037518859 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc8280b2040 con 0x7fc8380c6b80 2023-10-30T14:07:02.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.058+0000 7fc82f7fe640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8180664a0 0x7fc818068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:02.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.058+0000 7fc82f7fe640 1 -- 172.21.15.46:0/1037518859 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7fc82811e5a0 con 0x7fc8380c6b80 2023-10-30T14:07:02.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.058+0000 7fc82ffff640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8180664a0 0x7fc818068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:02.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.058+0000 7fc82ffff640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8180664a0 0x7fc818068960 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc838156a90 tx=0x7fc820009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:02.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.062+0000 7fc82f7fe640 1 -- 172.21.15.46:0/1037518859 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc838155ab0 con 0x7fc8380c6b80 2023-10-30T14:07:02.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.250+0000 7fc83d009640 1 -- 172.21.15.46:0/1037518859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc838155ab0 con 0x7fc8380c6b80 2023-10-30T14:07:02.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc82f7fe640 1 -- 172.21.15.46:0/1037518859 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fc838155ab0 con 0x7fc8380c6b80 2023-10-30T14:07:02.256 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:02.256 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:17.041488+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:02.256 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 188 2023-10-30T14:07:02.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc83d009640 1 -- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8180664a0 msgr2=0x7fc818068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:02.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc83d009640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8180664a0 0x7fc818068960 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc838156a90 tx=0x7fc820009040 comp rx=0 tx=0).stop 2023-10-30T14:07:02.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc83d009640 1 -- 172.21.15.46:0/1037518859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8380c6b80 msgr2=0x7fc8380c4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:02.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc83d009640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8380c6b80 0x7fc8380c4d90 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fc8280ae040 tx=0x7fc828019220 comp rx=0 tx=0).stop 2023-10-30T14:07:02.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc83d009640 1 -- 172.21.15.46:0/1037518859 shutdown_connections 2023-10-30T14:07:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc83d009640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc8180664a0 0x7fc818068960 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc83d009640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83813f4a0 0x7fc8381508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc83d009640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8380c6f60 0x7fc838150390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc83d009640 1 --2- 172.21.15.46:0/1037518859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8380c6b80 0x7fc8380c4d90 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:02.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc83d009640 1 -- 172.21.15.46:0/1037518859 >> 172.21.15.46:0/1037518859 conn(0x7fc8380c7910 msgr2=0x7fc8381536e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:02.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc83d009640 1 -- 172.21.15.46:0/1037518859 shutdown_connections 2023-10-30T14:07:02.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:02.254+0000 7fc83d009640 1 -- 172.21.15.46:0/1037518859 wait complete. 2023-10-30T14:07:02.277 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:07:04.278 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:04.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.430+0000 7fcd645f5640 1 Processor -- start 2023-10-30T14:07:04.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.430+0000 7fcd645f5640 1 -- start start 2023-10-30T14:07:04.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.430+0000 7fcd645f5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd5c0c6b80 0x7fcd5c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:04.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.430+0000 7fcd645f5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd5c0c0a50 con 0x7fcd5c13f4a0 2023-10-30T14:07:04.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.430+0000 7fcd645f5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd5c06f6e0 con 0x7fcd5c0c6f60 2023-10-30T14:07:04.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.430+0000 7fcd645f5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c06f9c0 con 0x7fcd5c0c6b80 2023-10-30T14:07:04.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.430+0000 7fcd62b92640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcd5c13f4a0 0x7fcd5c1566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54302/0 (socket says 172.21.15.46:54302) 2023-10-30T14:07:04.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd62b92640 1 -- 172.21.15.46:0/228126719 learned_addr learned my addr 172.21.15.46:0/228126719 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:04.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd62391640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd5c0c6b80 0x7fcd5c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:04.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054591466 0 0) 0x7fcd5c0c0a50 con 0x7fcd5c13f4a0 2023-10-30T14:07:04.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd48003530 con 0x7fcd5c13f4a0 2023-10-30T14:07:04.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183194690 0 0) 0x7fcd5c06f6e0 con 0x7fcd5c0c6f60 2023-10-30T14:07:04.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd5c0c0a50 con 0x7fcd5c0c6f60 2023-10-30T14:07:04.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2323390855 0 0) 0x7fcd48003530 con 0x7fcd5c13f4a0 2023-10-30T14:07:04.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd5c0c6b80 msgr2=0x7fcd5c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:04.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd5c0c6b80 0x7fcd5c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:04.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 >> v1:172.21.15.154:6789/0 conn(0x7fcd5c0c6f60 legacy=0x7fcd5c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:04.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5c06fca0 con 0x7fcd5c13f4a0 2023-10-30T14:07:04.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fcd54001980 con 0x7fcd5c13f4a0 2023-10-30T14:07:04.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 >> v1:172.21.15.46:6789/0 conn(0x7fcd5c13f4a0 legacy=0x7fcd5c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:04.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd48003b40 0x7fcd48003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:04.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd480047d0 0x7fcd48004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:04.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd48005190 0x7fcd4800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:04.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c06f9c0 con 0x7fcd48005190 2023-10-30T14:07:04.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4800e1c0 con 0x7fcd48003b40 2023-10-30T14:07:04.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4800e380 con 0x7fcd480047d0 2023-10-30T14:07:04.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd62391640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd5c0c6b80 0x7fcd5c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:07:04.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd62b92640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd48005190 0x7fcd4800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:04.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd61b90640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd480047d0 0x7fcd48004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:04.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd62391640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd48003b40 0x7fcd48003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:04.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd62b92640 1 -- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd480047d0 msgr2=0x7fcd48004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:04.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd62b92640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd480047d0 0x7fcd48004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:04.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd62b92640 1 -- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd48003b40 msgr2=0x7fcd48003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:04.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd62b92640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd48003b40 0x7fcd48003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:04.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd62b92640 1 -- 172.21.15.46:0/228126719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd48003530 con 0x7fcd48005190 2023-10-30T14:07:04.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd61b90640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd480047d0 0x7fcd48004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:04.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.434+0000 7fcd62391640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd48003b40 0x7fcd48003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd62b92640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd48005190 0x7fcd4800dba0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c06f4f0 tx=0x7fcd5401f880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2abc368e2a531e2e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd54007330 con 0x7fcd48005190 2023-10-30T14:07:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd540074b0 con 0x7fcd48005190 2023-10-30T14:07:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd6138f640 1 -- 172.21.15.46:0/228126719 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd540078a0 con 0x7fcd48005190 2023-10-30T14:07:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd645f5640 1 -- 172.21.15.46:0/228126719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd48005190 msgr2=0x7fcd4800dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd645f5640 1 --2- 172.21.15.46:0/228126719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd48005190 0x7fcd4800dba0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c06f4f0 tx=0x7fcd5401f880 comp rx=0 tx=0).stop 2023-10-30T14:07:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd62b92640 1 -- 172.21.15.46:0/228126719 reap_dead start 2023-10-30T14:07:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd645f5640 1 -- 172.21.15.46:0/228126719 shutdown_connections 2023-10-30T14:07:04.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd645f5640 1 -- 172.21.15.46:0/228126719 >> 172.21.15.46:0/228126719 conn(0x7fcd5c0c7910 msgr2=0x7fcd5c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:04.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd645f5640 1 -- 172.21.15.46:0/228126719 shutdown_connections 2023-10-30T14:07:04.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd645f5640 1 -- 172.21.15.46:0/228126719 wait complete. 2023-10-30T14:07:04.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd645f5640 1 Processor -- start 2023-10-30T14:07:04.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.438+0000 7fcd645f5640 1 -- start start 2023-10-30T14:07:04.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd645f5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd48005190 0x7fcd5c1544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd645f5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5c13e640 0x7fcd5c139d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd645f5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd5c1ee2b0 0x7fcd5c1ee760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd645f5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c154d50 con 0x7fcd48005190 2023-10-30T14:07:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd645f5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c0c4600 con 0x7fcd5c13e640 2023-10-30T14:07:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd645f5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c1eab90 con 0x7fcd5c1ee2b0 2023-10-30T14:07:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd62391640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd48005190 0x7fcd5c1544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd62391640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd48005190 0x7fcd5c1544a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39726/0 (socket says 172.21.15.46:39726) 2023-10-30T14:07:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd62391640 1 -- 172.21.15.46:0/929465928 learned_addr learned my addr 172.21.15.46:0/929465928 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd61b90640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5c13e640 0x7fcd5c139d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd62b92640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd5c1ee2b0 0x7fcd5c1ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd62391640 1 -- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd5c1ee2b0 msgr2=0x7fcd5c1ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd62391640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd5c1ee2b0 0x7fcd5c1ee760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd62391640 1 -- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5c13e640 msgr2=0x7fcd5c139d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd62391640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5c13e640 0x7fcd5c139d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd62391640 1 -- 172.21.15.46:0/929465928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5c1eb9b0 con 0x7fcd48005190 2023-10-30T14:07:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd62391640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd48005190 0x7fcd5c1544a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fcd58011640 tx=0x7fcd58011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd52ffd640 1 -- 172.21.15.46:0/929465928 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd58015870 con 0x7fcd48005190 2023-10-30T14:07:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd52ffd640 1 -- 172.21.15.46:0/929465928 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd580159d0 con 0x7fcd48005190 2023-10-30T14:07:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd52ffd640 1 -- 172.21.15.46:0/929465928 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd58015d40 con 0x7fcd48005190 2023-10-30T14:07:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd645f5640 1 -- 172.21.15.46:0/929465928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd5c1eeca0 con 0x7fcd48005190 2023-10-30T14:07:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd645f5640 1 -- 172.21.15.46:0/929465928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5c1eb080 con 0x7fcd48005190 2023-10-30T14:07:04.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd645f5640 1 -- 172.21.15.46:0/929465928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5c04e170 con 0x7fcd48005190 2023-10-30T14:07:04.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.442+0000 7fcd52ffd640 1 -- 172.21.15.46:0/929465928 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd5800f450 con 0x7fcd48005190 2023-10-30T14:07:04.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.446+0000 7fcd52ffd640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd400664d0 0x7fcd40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:04.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.446+0000 7fcd52ffd640 1 -- 172.21.15.46:0/929465928 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7fcd58084b40 con 0x7fcd48005190 2023-10-30T14:07:04.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.446+0000 7fcd61b90640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd400664d0 0x7fcd40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:04.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.446+0000 7fcd61b90640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd400664d0 0x7fcd40068990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c13c5c0 tx=0x7fcd4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:04.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.450+0000 7fcd52ffd640 1 -- 172.21.15.46:0/929465928 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd580517c0 con 0x7fcd48005190 2023-10-30T14:07:04.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.638+0000 7fcd645f5640 1 -- 172.21.15.46:0/929465928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd5c0c7d20 con 0x7fcd48005190 2023-10-30T14:07:04.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.638+0000 7fcd52ffd640 1 -- 172.21.15.46:0/929465928 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fcd58054de0 con 0x7fcd48005190 2023-10-30T14:07:04.643 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:04.643 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:17.041488+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 188 2023-10-30T14:07:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.642+0000 7fcd645f5640 1 -- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd400664d0 msgr2=0x7fcd40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.642+0000 7fcd645f5640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd400664d0 0x7fcd40068990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c13c5c0 tx=0x7fcd4c009040 comp rx=0 tx=0).stop 2023-10-30T14:07:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.642+0000 7fcd645f5640 1 -- 172.21.15.46:0/929465928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd48005190 msgr2=0x7fcd5c1544a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.642+0000 7fcd645f5640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd48005190 0x7fcd5c1544a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fcd58011640 tx=0x7fcd58011b00 comp rx=0 tx=0).stop 2023-10-30T14:07:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.642+0000 7fcd645f5640 1 -- 172.21.15.46:0/929465928 shutdown_connections 2023-10-30T14:07:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.642+0000 7fcd645f5640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd400664d0 0x7fcd40068990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.642+0000 7fcd645f5640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd5c1ee2b0 0x7fcd5c1ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.642+0000 7fcd645f5640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5c13e640 0x7fcd5c139d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.642+0000 7fcd645f5640 1 --2- 172.21.15.46:0/929465928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd48005190 0x7fcd5c1544a0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.642+0000 7fcd645f5640 1 -- 172.21.15.46:0/929465928 >> 172.21.15.46:0/929465928 conn(0x7fcd5c0c7910 msgr2=0x7fcd5c0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.642+0000 7fcd645f5640 1 -- 172.21.15.46:0/929465928 shutdown_connections 2023-10-30T14:07:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:04.642+0000 7fcd645f5640 1 -- 172.21.15.46:0/929465928 wait complete. 2023-10-30T14:07:04.664 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:07:06.665 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20b5679640 1 Processor -- start 2023-10-30T14:07:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20b5679640 1 -- start start 2023-10-30T14:07:06.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20b5679640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20b015d6f0 0x7f20b015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:06.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20b5679640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f20b00cf160 0x7f20b00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:06.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20b5679640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b00cb800 con 0x7f20b01538a0 2023-10-30T14:07:06.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20b5679640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b00cb330 con 0x7f20b00cf160 2023-10-30T14:07:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20b5679640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b00cad70 con 0x7f20b015d6f0 2023-10-30T14:07:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20aeffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f20b01538a0 0x7f20b00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38898/0 (socket says 172.21.15.46:38898) 2023-10-30T14:07:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20aeffd640 1 -- 172.21.15.46:0/4276580759 learned_addr learned my addr 172.21.15.46:0/4276580759 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20af7fe640 1 --2- 172.21.15.46:0/4276580759 >> v2:172.21.15.46:3300/0 conn(0x7f20b00cf160 0x7f20b00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:06.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20a7fff640 1 --2- 172.21.15.46:0/4276580759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20b015d6f0 0x7f20b015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20ae7fc640 1 -- 172.21.15.46:0/4276580759 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414329635 0 0) 0x7f20b00cb800 con 0x7f20b01538a0 2023-10-30T14:07:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20ae7fc640 1 -- 172.21.15.46:0/4276580759 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f209c003530 con 0x7f20b01538a0 2023-10-30T14:07:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20af7fe640 1 -- 172.21.15.46:0/4276580759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20b015d6f0 msgr2=0x7f20b015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20af7fe640 1 --2- 172.21.15.46:0/4276580759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20b015d6f0 0x7f20b015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:06.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20af7fe640 1 -- 172.21.15.46:0/4276580759 >> v1:172.21.15.46:6789/0 conn(0x7f20b01538a0 legacy=0x7f20b00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:06.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20af7fe640 1 -- 172.21.15.46:0/4276580759 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b00cb050 con 0x7f20b00cf160 2023-10-30T14:07:06.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.818+0000 7f20a7fff640 1 --2- 172.21.15.46:0/4276580759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20b015d6f0 0x7f20b015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:06.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.822+0000 7f20af7fe640 1 --2- 172.21.15.46:0/4276580759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b00cf160 0x7f20b00cf590 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009d90 tx=0x7f20a801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c7b23179e8f86ef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:06.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.822+0000 7f20ae7fc640 1 -- 172.21.15.46:0/4276580759 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a80af600 con 0x7f20b00cf160 2023-10-30T14:07:06.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.822+0000 7f20ae7fc640 1 -- 172.21.15.46:0/4276580759 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a80ab050 con 0x7f20b00cf160 2023-10-30T14:07:06.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.822+0000 7f20ae7fc640 1 -- 172.21.15.46:0/4276580759 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a80af980 con 0x7f20b00cf160 2023-10-30T14:07:06.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.822+0000 7f20b5679640 1 -- 172.21.15.46:0/4276580759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b00cf160 msgr2=0x7f20b00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:06.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.822+0000 7f20b5679640 1 --2- 172.21.15.46:0/4276580759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b00cf160 0x7f20b00cf590 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009d90 tx=0x7f20a801f130 comp rx=0 tx=0).stop 2023-10-30T14:07:06.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.822+0000 7f20b5679640 1 -- 172.21.15.46:0/4276580759 shutdown_connections 2023-10-30T14:07:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.822+0000 7f20b5679640 1 --2- 172.21.15.46:0/4276580759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b00cf160 0x7f20b00cf590 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.822+0000 7f20b5679640 1 --2- 172.21.15.46:0/4276580759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20b015d6f0 0x7f20b015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.822+0000 7f20b5679640 1 -- 172.21.15.46:0/4276580759 >> 172.21.15.46:0/4276580759 conn(0x7f20b006bfe0 msgr2=0x7f20b006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:06.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.822+0000 7f20b5679640 1 -- 172.21.15.46:0/4276580759 shutdown_connections 2023-10-30T14:07:06.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.822+0000 7f20b5679640 1 -- 172.21.15.46:0/4276580759 wait complete. 2023-10-30T14:07:06.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20b5679640 1 Processor -- start 2023-10-30T14:07:06.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20b5679640 1 -- start start 2023-10-30T14:07:06.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20b5679640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b00cf160 0x7f20b015c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:06.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20b5679640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b01538a0 0x7f20b00cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20b5679640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20b015d6f0 0x7f20b00cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20b5679640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b00ccfb0 con 0x7f20b01538a0 2023-10-30T14:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20b5679640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b00b36b0 con 0x7f20b00cf160 2023-10-30T14:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20b5679640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b00b4760 con 0x7f20b015d6f0 2023-10-30T14:07:06.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b01538a0 0x7f20b00cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b01538a0 0x7f20b00cd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58716/0 (socket says 172.21.15.46:58716) 2023-10-30T14:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20a7fff640 1 -- 172.21.15.46:0/4215928984 learned_addr learned my addr 172.21.15.46:0/4215928984 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20af7fe640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20b015d6f0 0x7f20b00cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:06.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20aeffd640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b00cf160 0x7f20b015c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20a7fff640 1 -- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20b015d6f0 msgr2=0x7f20b00cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20a7fff640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20b015d6f0 0x7f20b00cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20a7fff640 1 -- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b00cf160 msgr2=0x7f20b015c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20a7fff640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b00cf160 0x7f20b015c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20a7fff640 1 -- 172.21.15.46:0/4215928984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a8009a70 con 0x7f20b01538a0 2023-10-30T14:07:06.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20a7fff640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b01538a0 0x7f20b00cd5f0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f209800d830 tx=0x7f209800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20a77fe640 1 -- 172.21.15.46:0/4215928984 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20980096c0 con 0x7f20b01538a0 2023-10-30T14:07:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20a77fe640 1 -- 172.21.15.46:0/4215928984 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2098009820 con 0x7f20b01538a0 2023-10-30T14:07:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20a77fe640 1 -- 172.21.15.46:0/4215928984 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098009bd0 con 0x7f20b01538a0 2023-10-30T14:07:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20b5679640 1 -- 172.21.15.46:0/4215928984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b00cea40 con 0x7f20b01538a0 2023-10-30T14:07:06.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.826+0000 7f20b5679640 1 -- 172.21.15.46:0/4215928984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b00b3340 con 0x7f20b01538a0 2023-10-30T14:07:06.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.830+0000 7f20b5679640 1 -- 172.21.15.46:0/4215928984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b00b2a50 con 0x7f20b01538a0 2023-10-30T14:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.830+0000 7f20a77fe640 1 -- 172.21.15.46:0/4215928984 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2098014410 con 0x7f20b01538a0 2023-10-30T14:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.830+0000 7f20a77fe640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20900664d0 0x7f2090068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.830+0000 7f20a77fe640 1 -- 172.21.15.46:0/4215928984 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7f2098084eb0 con 0x7f20b01538a0 2023-10-30T14:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.830+0000 7f20aeffd640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20900664d0 0x7f2090068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:06.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.830+0000 7f20aeffd640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20900664d0 0x7f2090068990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f20a00011f0 tx=0x7f20a0007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:06.834+0000 7f20a77fe640 1 -- 172.21.15.46:0/4215928984 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2098051bd0 con 0x7f20b01538a0 2023-10-30T14:07:07.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.034+0000 7f20b5679640 1 -- 172.21.15.46:0/4215928984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20b004e170 con 0x7f20b01538a0 2023-10-30T14:07:07.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20a77fe640 1 -- 172.21.15.46:0/4215928984 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f20980551f0 con 0x7f20b01538a0 2023-10-30T14:07:07.040 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:07.040 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:17.041488+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:07.040 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 188 2023-10-30T14:07:07.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20b5679640 1 -- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20900664d0 msgr2=0x7f2090068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20b5679640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20900664d0 0x7f2090068990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f20a00011f0 tx=0x7f20a0007d40 comp rx=0 tx=0).stop 2023-10-30T14:07:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20b5679640 1 -- 172.21.15.46:0/4215928984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b01538a0 msgr2=0x7f20b00cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20b5679640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b01538a0 0x7f20b00cd5f0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f209800d830 tx=0x7f209800dcf0 comp rx=0 tx=0).stop 2023-10-30T14:07:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20b5679640 1 -- 172.21.15.46:0/4215928984 shutdown_connections 2023-10-30T14:07:07.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20b5679640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20900664d0 0x7f2090068990 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20b5679640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20b015d6f0 0x7f20b00cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20b5679640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b01538a0 0x7f20b00cd5f0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20b5679640 1 --2- 172.21.15.46:0/4215928984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b00cf160 0x7f20b015c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20b5679640 1 -- 172.21.15.46:0/4215928984 >> 172.21.15.46:0/4215928984 conn(0x7f20b006bfe0 msgr2=0x7f20b0151d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:07.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20b5679640 1 -- 172.21.15.46:0/4215928984 shutdown_connections 2023-10-30T14:07:07.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:07.038+0000 7f20b5679640 1 -- 172.21.15.46:0/4215928984 wait complete. 2023-10-30T14:07:07.061 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:07:09.062 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:09.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2f06dd3640 1 Processor -- start 2023-10-30T14:07:09.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2f06dd3640 1 -- start start 2023-10-30T14:07:09.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2f06dd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f00069e90 0x7f2f0015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:09.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2f06dd3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f0006a270 0x7f2f000c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:09.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2f06dd3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2f0006cb80 0x7f2f0006d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:09.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2f06dd3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f000c8570 con 0x7f2f0006a270 2023-10-30T14:07:09.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2f06dd3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0006d550 con 0x7f2f0006cb80 2023-10-30T14:07:09.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2f06dd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000c6c10 con 0x7f2f00069e90 2023-10-30T14:07:09.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2efffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f0006a270 0x7f2f000c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:09.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2efffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f0006a270 0x7f2f000c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58726/0 (socket says 172.21.15.46:58726) 2023-10-30T14:07:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2efffff640 1 -- 172.21.15.46:0/3588921556 learned_addr learned my addr 172.21.15.46:0/3588921556 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2f05370640 1 --2- 172.21.15.46:0/3588921556 >> v2:172.21.15.154:3300/0 conn(0x7f2f0006cb80 0x7f2f0006d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.214+0000 7f2f04b6f640 1 --2- 172.21.15.46:0/3588921556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f00069e90 0x7f2f0015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2efffff640 1 -- 172.21.15.46:0/3588921556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f00069e90 msgr2=0x7f2f0015a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2efffff640 1 --2- 172.21.15.46:0/3588921556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f00069e90 0x7f2f0015a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:09.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2efffff640 1 -- 172.21.15.46:0/3588921556 >> v2:172.21.15.154:3300/0 conn(0x7f2f0006cb80 msgr2=0x7f2f0006d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:09.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2efffff640 1 --2- 172.21.15.46:0/3588921556 >> v2:172.21.15.154:3300/0 conn(0x7f2f0006cb80 0x7f2f0006d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:09.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2efffff640 1 -- 172.21.15.46:0/3588921556 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f000c6ef0 con 0x7f2f0006a270 2023-10-30T14:07:09.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2f04b6f640 1 --2- 172.21.15.46:0/3588921556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f00069e90 0x7f2f0015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:09.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2f05370640 1 --2- 172.21.15.46:0/3588921556 >> v2:172.21.15.154:3300/0 conn(0x7f2f0006cb80 0x7f2f0006d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:09.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2efffff640 1 --2- 172.21.15.46:0/3588921556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f0006a270 0x7f2f000c9cb0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f2f0006ff00 tx=0x7f2eec01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57247189d6a3cfe0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:09.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2eff7fe640 1 -- 172.21.15.46:0/3588921556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0ac410 con 0x7f2f0006a270 2023-10-30T14:07:09.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2eff7fe640 1 -- 172.21.15.46:0/3588921556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec0ac570 con 0x7f2f0006a270 2023-10-30T14:07:09.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2eff7fe640 1 -- 172.21.15.46:0/3588921556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0ac900 con 0x7f2f0006a270 2023-10-30T14:07:09.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3588921556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f0006a270 msgr2=0x7f2f000c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:09.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2f06dd3640 1 --2- 172.21.15.46:0/3588921556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f0006a270 0x7f2f000c9cb0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f2f0006ff00 tx=0x7f2eec01c500 comp rx=0 tx=0).stop 2023-10-30T14:07:09.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3588921556 shutdown_connections 2023-10-30T14:07:09.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2f06dd3640 1 --2- 172.21.15.46:0/3588921556 >> v2:172.21.15.154:3300/0 conn(0x7f2f0006cb80 0x7f2f0006d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:09.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2f06dd3640 1 --2- 172.21.15.46:0/3588921556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f0006a270 0x7f2f000c9cb0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:09.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2f06dd3640 1 --2- 172.21.15.46:0/3588921556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f00069e90 0x7f2f0015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:09.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3588921556 >> 172.21.15.46:0/3588921556 conn(0x7f2f0006dfd0 msgr2=0x7f2f0006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:09.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.218+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3588921556 shutdown_connections 2023-10-30T14:07:09.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3588921556 wait complete. 2023-10-30T14:07:09.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f06dd3640 1 Processor -- start 2023-10-30T14:07:09.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f06dd3640 1 -- start start 2023-10-30T14:07:09.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f06dd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f00069e90 0x7f2f0014afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:09.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f06dd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f0006a270 0x7f2f000c4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:09.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f06dd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f0006cb80 0x7f2f001f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:09.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f06dd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000c4b80 con 0x7f2f0006cb80 2023-10-30T14:07:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f06dd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000c6430 con 0x7f2f0006a270 2023-10-30T14:07:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f06dd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000c5e70 con 0x7f2f00069e90 2023-10-30T14:07:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f05370640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f0006cb80 0x7f2f001f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f05370640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f0006cb80 0x7f2f001f2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58732/0 (socket says 172.21.15.46:58732) 2023-10-30T14:07:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f05370640 1 -- 172.21.15.46:0/3407661678 learned_addr learned my addr 172.21.15.46:0/3407661678 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2efffff640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f0006a270 0x7f2f000c4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f04b6f640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f00069e90 0x7f2f0014afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f05370640 1 -- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f00069e90 msgr2=0x7f2f0014afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f05370640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f00069e90 0x7f2f0014afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f05370640 1 -- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f0006a270 msgr2=0x7f2f000c4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f05370640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f0006a270 0x7f2f000c4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f05370640 1 -- 172.21.15.46:0/3407661678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec0094d0 con 0x7f2f0006cb80 2023-10-30T14:07:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2efffff640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f0006a270 0x7f2f000c4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.222+0000 7f2f05370640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f0006cb80 0x7f2f001f2ea0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8007df0 tx=0x7f2ef800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.226+0000 7f2efd7fa640 1 -- 172.21.15.46:0/3407661678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef80039a0 con 0x7f2f0006cb80 2023-10-30T14:07:09.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.226+0000 7f2efd7fa640 1 -- 172.21.15.46:0/3407661678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef800f040 con 0x7f2f0006cb80 2023-10-30T14:07:09.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.226+0000 7f2efd7fa640 1 -- 172.21.15.46:0/3407661678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8003d40 con 0x7f2f0006cb80 2023-10-30T14:07:09.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.226+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3407661678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f000c6150 con 0x7f2f0006cb80 2023-10-30T14:07:09.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.226+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3407661678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f000718f0 con 0x7f2f0006cb80 2023-10-30T14:07:09.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.226+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3407661678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f00071470 con 0x7f2f0006cb80 2023-10-30T14:07:09.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.226+0000 7f2efd7fa640 1 -- 172.21.15.46:0/3407661678 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ef8004030 con 0x7f2f0006cb80 2023-10-30T14:07:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.226+0000 7f2efd7fa640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2ee40664f0 0x7f2ee40689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.226+0000 7f2efd7fa640 1 -- 172.21.15.46:0/3407661678 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7f2ef8084950 con 0x7f2f0006cb80 2023-10-30T14:07:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.226+0000 7f2f04b6f640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2ee40664f0 0x7f2ee40689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.230+0000 7f2f04b6f640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2ee40664f0 0x7f2ee40689b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c880 tx=0x7f2ef4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.230+0000 7f2efd7fa640 1 -- 172.21.15.46:0/3407661678 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2f00071470 con 0x7f2f0006cb80 2023-10-30T14:07:09.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.422+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3407661678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f00071470 con 0x7f2f0006cb80 2023-10-30T14:07:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.422+0000 7f2efd7fa640 1 -- 172.21.15.46:0/3407661678 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f2f00071470 con 0x7f2f0006cb80 2023-10-30T14:07:09.426 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:09.426 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:17.041488+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 188 2023-10-30T14:07:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.426+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2ee40664f0 msgr2=0x7f2ee40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.426+0000 7f2f06dd3640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2ee40664f0 0x7f2ee40689b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c880 tx=0x7f2ef4008040 comp rx=0 tx=0).stop 2023-10-30T14:07:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.426+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3407661678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f0006cb80 msgr2=0x7f2f001f2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.426+0000 7f2f06dd3640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f0006cb80 0x7f2f001f2ea0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8007df0 tx=0x7f2ef800d4a0 comp rx=0 tx=0).stop 2023-10-30T14:07:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.426+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3407661678 shutdown_connections 2023-10-30T14:07:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.426+0000 7f2f06dd3640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2ee40664f0 0x7f2ee40689b0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.426+0000 7f2f06dd3640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f0006cb80 0x7f2f001f2ea0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.426+0000 7f2f06dd3640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f0006a270 0x7f2f000c4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.426+0000 7f2f06dd3640 1 --2- 172.21.15.46:0/3407661678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f00069e90 0x7f2f0014afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.426+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3407661678 >> 172.21.15.46:0/3407661678 conn(0x7f2f0006dfd0 msgr2=0x7f2f000ca5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.426+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3407661678 shutdown_connections 2023-10-30T14:07:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:09.426+0000 7f2f06dd3640 1 -- 172.21.15.46:0/3407661678 wait complete. 2023-10-30T14:07:09.447 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:07:11.448 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:11.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.598+0000 7eff06c9c640 1 Processor -- start 2023-10-30T14:07:11.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.598+0000 7eff06c9c640 1 -- start start 2023-10-30T14:07:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7eff06c9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff000c6bd0 0x7eff0006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7eff06c9c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff00071110 0x7eff00158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7eff06c9c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff00066cf0 con 0x7eff0006f7a0 2023-10-30T14:07:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7eff06c9c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff000700c0 con 0x7eff000c6bd0 2023-10-30T14:07:11.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7eff06c9c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff000c70d0 con 0x7eff00071110 2023-10-30T14:07:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7efefffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff000c6bd0 0x7eff0006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7efefffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff000c6bd0 0x7eff0006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58746/0 (socket says 172.21.15.46:58746) 2023-10-30T14:07:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7efefffff640 1 -- 172.21.15.46:0/2627980713 learned_addr learned my addr 172.21.15.46:0/2627980713 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:11.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7eff05239640 1 --2- 172.21.15.46:0/2627980713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff00071110 0x7eff00158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7efefffff640 1 -- 172.21.15.46:0/2627980713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff00071110 msgr2=0x7eff00158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7efefffff640 1 --2- 172.21.15.46:0/2627980713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff00071110 0x7eff00158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:11.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7efefffff640 1 -- 172.21.15.46:0/2627980713 >> v1:172.21.15.154:6789/0 conn(0x7eff0006f7a0 legacy=0x7eff00069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:11.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.602+0000 7efefffff640 1 -- 172.21.15.46:0/2627980713 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff000c7390 con 0x7eff000c6bd0 2023-10-30T14:07:11.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7efefffff640 1 --2- 172.21.15.46:0/2627980713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000c6bd0 0x7eff0006fb80 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7efeec009870 tx=0x7efeec01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85123706496e4dc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:11.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7efeff7fe640 1 -- 172.21.15.46:0/2627980713 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec0a8a30 con 0x7eff000c6bd0 2023-10-30T14:07:11.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7efeff7fe640 1 -- 172.21.15.46:0/2627980713 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec0a8b90 con 0x7eff000c6bd0 2023-10-30T14:07:11.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7efeff7fe640 1 -- 172.21.15.46:0/2627980713 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec0b0440 con 0x7eff000c6bd0 2023-10-30T14:07:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7eff06c9c640 1 -- 172.21.15.46:0/2627980713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000c6bd0 msgr2=0x7eff0006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7eff06c9c640 1 --2- 172.21.15.46:0/2627980713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000c6bd0 0x7eff0006fb80 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7efeec009870 tx=0x7efeec01ed30 comp rx=0 tx=0).stop 2023-10-30T14:07:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7eff06c9c640 1 -- 172.21.15.46:0/2627980713 shutdown_connections 2023-10-30T14:07:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7eff06c9c640 1 --2- 172.21.15.46:0/2627980713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff00071110 0x7eff00158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:11.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7eff06c9c640 1 --2- 172.21.15.46:0/2627980713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000c6bd0 0x7eff0006fb80 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:11.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7eff06c9c640 1 -- 172.21.15.46:0/2627980713 >> 172.21.15.46:0/2627980713 conn(0x7eff000c7910 msgr2=0x7eff000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:11.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7eff06c9c640 1 -- 172.21.15.46:0/2627980713 shutdown_connections 2023-10-30T14:07:11.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7eff06c9c640 1 -- 172.21.15.46:0/2627980713 wait complete. 2023-10-30T14:07:11.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7eff06c9c640 1 Processor -- start 2023-10-30T14:07:11.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.606+0000 7eff06c9c640 1 -- start start 2023-10-30T14:07:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff06c9c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff0006f7a0 0x7eff00157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff06c9c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff00071110 0x7eff00152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff06c9c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000c6bd0 0x7eff00153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff06c9c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff001538b0 con 0x7eff000c6bd0 2023-10-30T14:07:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff06c9c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff00154ab0 con 0x7eff00071110 2023-10-30T14:07:11.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff06c9c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff00157520 con 0x7eff0006f7a0 2023-10-30T14:07:11.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff05239640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000c6bd0 0x7eff00153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:11.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff04a38640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff0006f7a0 0x7eff00157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:11.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff05239640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000c6bd0 0x7eff00153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58756/0 (socket says 172.21.15.46:58756) 2023-10-30T14:07:11.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff05239640 1 -- 172.21.15.46:0/2269975269 learned_addr learned my addr 172.21.15.46:0/2269975269 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:11.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7efefffff640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff00071110 0x7eff00152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:11.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff05239640 1 -- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff0006f7a0 msgr2=0x7eff00157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:11.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff05239640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff0006f7a0 0x7eff00157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:11.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff05239640 1 -- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff00071110 msgr2=0x7eff00152b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:11.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff05239640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff00071110 0x7eff00152b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:11.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff05239640 1 -- 172.21.15.46:0/2269975269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec0094d0 con 0x7eff000c6bd0 2023-10-30T14:07:11.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff05239640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000c6bd0 0x7eff00153090 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7efef800efc0 tx=0x7efef800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:11.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7efefd7fa640 1 -- 172.21.15.46:0/2269975269 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8014480 con 0x7eff000c6bd0 2023-10-30T14:07:11.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7efefd7fa640 1 -- 172.21.15.46:0/2269975269 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef800f040 con 0x7eff000c6bd0 2023-10-30T14:07:11.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff06c9c640 1 -- 172.21.15.46:0/2269975269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff00154d50 con 0x7eff000c6bd0 2023-10-30T14:07:11.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7efefd7fa640 1 -- 172.21.15.46:0/2269975269 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8014820 con 0x7eff000c6bd0 2023-10-30T14:07:11.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff06c9c640 1 -- 172.21.15.46:0/2269975269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff00155030 con 0x7eff000c6bd0 2023-10-30T14:07:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7eff06c9c640 1 -- 172.21.15.46:0/2269975269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff001579d0 con 0x7eff000c6bd0 2023-10-30T14:07:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.610+0000 7efefd7fa640 1 -- 172.21.15.46:0/2269975269 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efef8014980 con 0x7eff000c6bd0 2023-10-30T14:07:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.614+0000 7efefd7fa640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efee40664a0 0x7efee4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.614+0000 7efefd7fa640 1 -- 172.21.15.46:0/2269975269 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7efef8084ff0 con 0x7eff000c6bd0 2023-10-30T14:07:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.614+0000 7eff04a38640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efee40664a0 0x7efee4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.614+0000 7eff04a38640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efee40664a0 0x7efee4068960 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efef4007fd0 tx=0x7efef4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.618+0000 7efefd7fa640 1 -- 172.21.15.46:0/2269975269 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7eff001579d0 con 0x7eff000c6bd0 2023-10-30T14:07:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.810+0000 7eff06c9c640 1 -- 172.21.15.46:0/2269975269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff001579d0 con 0x7eff000c6bd0 2023-10-30T14:07:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.810+0000 7efefd7fa640 1 -- 172.21.15.46:0/2269975269 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+6523 (secure 0 0 0) 0x7eff001579d0 con 0x7eff000c6bd0 2023-10-30T14:07:11.816 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:11.816 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:17.041488+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 188 2023-10-30T14:07:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.814+0000 7eff06c9c640 1 -- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efee40664a0 msgr2=0x7efee4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.814+0000 7eff06c9c640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efee40664a0 0x7efee4068960 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efef4007fd0 tx=0x7efef4009040 comp rx=0 tx=0).stop 2023-10-30T14:07:11.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.814+0000 7eff06c9c640 1 -- 172.21.15.46:0/2269975269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000c6bd0 msgr2=0x7eff00153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.814+0000 7eff06c9c640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000c6bd0 0x7eff00153090 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7efef800efc0 tx=0x7efef800c490 comp rx=0 tx=0).stop 2023-10-30T14:07:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.814+0000 7eff06c9c640 1 -- 172.21.15.46:0/2269975269 shutdown_connections 2023-10-30T14:07:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.814+0000 7eff06c9c640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efee40664a0 0x7efee4068960 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.814+0000 7eff06c9c640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000c6bd0 0x7eff00153090 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.814+0000 7eff06c9c640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff00071110 0x7eff00152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.814+0000 7eff06c9c640 1 --2- 172.21.15.46:0/2269975269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff0006f7a0 0x7eff00157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:11.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.814+0000 7eff06c9c640 1 -- 172.21.15.46:0/2269975269 >> 172.21.15.46:0/2269975269 conn(0x7eff000c7910 msgr2=0x7eff000c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:11.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.814+0000 7eff06c9c640 1 -- 172.21.15.46:0/2269975269 shutdown_connections 2023-10-30T14:07:11.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:11.814+0000 7eff06c9c640 1 -- 172.21.15.46:0/2269975269 wait complete. 2023-10-30T14:07:11.837 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:07:13.838 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.990+0000 7f5d22df8640 1 Processor -- start 2023-10-30T14:07:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.990+0000 7f5d22df8640 1 -- start start 2023-10-30T14:07:13.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d22df8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c070a00 0x7f5d1c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:13.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d22df8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5d1c069e90 0x7f5d1c06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:13.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d22df8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1c071380 con 0x7f5d1c1539a0 2023-10-30T14:07:13.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d22df8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1c06fbb0 con 0x7f5d1c069e90 2023-10-30T14:07:13.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d22df8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c153d80 con 0x7f5d1c070a00 2023-10-30T14:07:13.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d1bfff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c070a00 0x7f5d1c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:13.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d1bfff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c070a00 0x7f5d1c070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:49098/0 (socket says 172.21.15.46:49098) 2023-10-30T14:07:13.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d1bfff640 1 -- 172.21.15.46:0/2266986589 learned_addr learned my addr 172.21.15.46:0/2266986589 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:13.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d21395640 1 --2- 172.21.15.46:0/2266986589 >> v2:172.21.15.154:3300/0 conn(0x7f5d1c069e90 0x7f5d1c06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:13.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d1bfff640 1 -- 172.21.15.46:0/2266986589 >> v2:172.21.15.154:3300/0 conn(0x7f5d1c069e90 msgr2=0x7f5d1c06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:13.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d1bfff640 1 --2- 172.21.15.46:0/2266986589 >> v2:172.21.15.154:3300/0 conn(0x7f5d1c069e90 0x7f5d1c06a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:13.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d1bfff640 1 -- 172.21.15.46:0/2266986589 >> v1:172.21.15.154:6789/0 conn(0x7f5d1c1539a0 legacy=0x7f5d1c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:13.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d1bfff640 1 -- 172.21.15.46:0/2266986589 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1c0c5c50 con 0x7f5d1c070a00 2023-10-30T14:07:13.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.994+0000 7f5d21395640 1 --2- 172.21.15.46:0/2266986589 >> v2:172.21.15.154:3300/0 conn(0x7f5d1c069e90 0x7f5d1c06a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:07:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d1bfff640 1 --2- 172.21.15.46:0/2266986589 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c070a00 0x7f5d1c070e40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009870 tx=0x7f5d0801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28bc14e3850f2158 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d1b7fe640 1 -- 172.21.15.46:0/2266986589 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d080ad640 con 0x7f5d1c070a00 2023-10-30T14:07:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d1b7fe640 1 -- 172.21.15.46:0/2266986589 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d080ad7a0 con 0x7f5d1c070a00 2023-10-30T14:07:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d1b7fe640 1 -- 172.21.15.46:0/2266986589 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d080adb30 con 0x7f5d1c070a00 2023-10-30T14:07:14.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d22df8640 1 -- 172.21.15.46:0/2266986589 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c070a00 msgr2=0x7f5d1c070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:14.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d22df8640 1 --2- 172.21.15.46:0/2266986589 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c070a00 0x7f5d1c070e40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009870 tx=0x7f5d0801c480 comp rx=0 tx=0).stop 2023-10-30T14:07:14.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d22df8640 1 -- 172.21.15.46:0/2266986589 shutdown_connections 2023-10-30T14:07:14.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d22df8640 1 --2- 172.21.15.46:0/2266986589 >> v2:172.21.15.154:3300/0 conn(0x7f5d1c069e90 0x7f5d1c06a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:14.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d22df8640 1 --2- 172.21.15.46:0/2266986589 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c070a00 0x7f5d1c070e40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:14.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d22df8640 1 -- 172.21.15.46:0/2266986589 >> 172.21.15.46:0/2266986589 conn(0x7f5d1c0c7910 msgr2=0x7f5d1c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:14.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d22df8640 1 -- 172.21.15.46:0/2266986589 shutdown_connections 2023-10-30T14:07:14.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d22df8640 1 -- 172.21.15.46:0/2266986589 wait complete. 2023-10-30T14:07:14.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d22df8640 1 Processor -- start 2023-10-30T14:07:14.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:13.998+0000 7f5d22df8640 1 -- start start 2023-10-30T14:07:14.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d22df8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d1c069e90 0x7f5d1c156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:14.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d22df8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d1c070a00 0x7f5d1c156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:14.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d22df8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c1539a0 0x7f5d1c1f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:14.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d22df8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c156b00 con 0x7f5d1c069e90 2023-10-30T14:07:14.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d22df8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c155530 con 0x7f5d1c070a00 2023-10-30T14:07:14.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d22df8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c155250 con 0x7f5d1c1539a0 2023-10-30T14:07:14.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d20b94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d1c069e90 0x7f5d1c156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:14.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d20b94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d1c069e90 0x7f5d1c156f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58768/0 (socket says 172.21.15.46:58768) 2023-10-30T14:07:14.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d20b94640 1 -- 172.21.15.46:0/1746388178 learned_addr learned my addr 172.21.15.46:0/1746388178 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:14.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d1bfff640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d1c070a00 0x7f5d1c156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:14.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d21395640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c1539a0 0x7f5d1c1f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:14.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d20b94640 1 -- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c1539a0 msgr2=0x7f5d1c1f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:14.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d20b94640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c1539a0 0x7f5d1c1f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:14.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d20b94640 1 -- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d1c070a00 msgr2=0x7f5d1c156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:14.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d20b94640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d1c070a00 0x7f5d1c156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:14.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d20b94640 1 -- 172.21.15.46:0/1746388178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d14009a70 con 0x7f5d1c069e90 2023-10-30T14:07:14.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d21395640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c1539a0 0x7f5d1c1f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d1bfff640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d1c070a00 0x7f5d1c156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d20b94640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d1c069e90 0x7f5d1c156f50 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f5d1000c500 tx=0x7f5d1000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d197fa640 1 -- 172.21.15.46:0/1746388178 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d10005220 con 0x7f5d1c069e90 2023-10-30T14:07:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d197fa640 1 -- 172.21.15.46:0/1746388178 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d10005380 con 0x7f5d1c069e90 2023-10-30T14:07:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d22df8640 1 -- 172.21.15.46:0/1746388178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d080094d0 con 0x7f5d1c069e90 2023-10-30T14:07:14.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d197fa640 1 -- 172.21.15.46:0/1746388178 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d10007270 con 0x7f5d1c069e90 2023-10-30T14:07:14.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d22df8640 1 -- 172.21.15.46:0/1746388178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1c14d550 con 0x7f5d1c069e90 2023-10-30T14:07:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d22df8640 1 -- 172.21.15.46:0/1746388178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d1c04e170 con 0x7f5d1c069e90 2023-10-30T14:07:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.002+0000 7f5d197fa640 1 -- 172.21.15.46:0/1746388178 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5d100073d0 con 0x7f5d1c069e90 2023-10-30T14:07:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.006+0000 7f5d197fa640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d000664d0 0x7f5d00068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.006+0000 7f5d197fa640 1 -- 172.21.15.46:0/1746388178 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7746+0+0 (secure 0 0 0) 0x7f5d10083fe0 con 0x7f5d1c069e90 2023-10-30T14:07:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.006+0000 7f5d1bfff640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d000664d0 0x7f5d00068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:14.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.006+0000 7f5d1bfff640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d000664d0 0x7f5d00068990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f5d080af000 tx=0x7f5d08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:14.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.010+0000 7f5d197fa640 1 -- 172.21.15.46:0/1746388178 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5d10050c30 con 0x7f5d1c069e90 2023-10-30T14:07:14.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.198+0000 7f5d22df8640 1 -- 172.21.15.46:0/1746388178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d1c15db50 con 0x7f5d1c069e90 2023-10-30T14:07:14.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.198+0000 7f5d197fa640 1 -- 172.21.15.46:0/1746388178 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f5d10054250 con 0x7f5d1c069e90 2023-10-30T14:07:14.202 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:14.202 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:06:17.041488+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8190,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8190":{"gid":8190,"name":"e","rank":0,"incarnation":175,"state":"up:active","state_seq":40,"addr":"172.21.15.46:6837/3154171789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3154171789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3154171789}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:14.202 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 188 2023-10-30T14:07:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.202+0000 7f5d22df8640 1 -- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d000664d0 msgr2=0x7f5d00068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.202+0000 7f5d22df8640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d000664d0 0x7f5d00068990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f5d080af000 tx=0x7f5d08008040 comp rx=0 tx=0).stop 2023-10-30T14:07:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.202+0000 7f5d22df8640 1 -- 172.21.15.46:0/1746388178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d1c069e90 msgr2=0x7f5d1c156f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.202+0000 7f5d22df8640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d1c069e90 0x7f5d1c156f50 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f5d1000c500 tx=0x7f5d1000c9c0 comp rx=0 tx=0).stop 2023-10-30T14:07:14.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.202+0000 7f5d22df8640 1 -- 172.21.15.46:0/1746388178 shutdown_connections 2023-10-30T14:07:14.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.202+0000 7f5d22df8640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d000664d0 0x7f5d00068990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:14.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.202+0000 7f5d22df8640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d1c1539a0 0x7f5d1c1f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:14.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.202+0000 7f5d22df8640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d1c070a00 0x7f5d1c156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:14.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.202+0000 7f5d22df8640 1 --2- 172.21.15.46:0/1746388178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d1c069e90 0x7f5d1c156f50 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:14.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.202+0000 7f5d22df8640 1 -- 172.21.15.46:0/1746388178 >> 172.21.15.46:0/1746388178 conn(0x7f5d1c0c7910 msgr2=0x7f5d1c14e830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:14.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.202+0000 7f5d22df8640 1 -- 172.21.15.46:0/1746388178 shutdown_connections 2023-10-30T14:07:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:14.202+0000 7f5d22df8640 1 -- 172.21.15.46:0/1746388178 wait complete. 2023-10-30T14:07:14.223 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:07:16.224 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:16.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f8877b1d640 1 Processor -- start 2023-10-30T14:07:16.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f8877b1d640 1 -- start start 2023-10-30T14:07:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f8877b1d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88701538a0 0x7f887015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f8877b1d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f88700ca0e0 0x7f887015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f8877b1d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88700cb800 con 0x7f88700cf160 2023-10-30T14:07:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f8877b1d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f887015dc30 con 0x7f88701538a0 2023-10-30T14:07:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f8877b1d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88700cb330 con 0x7f88700ca0e0 2023-10-30T14:07:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f88758b9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88701538a0 0x7f887015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f88758b9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88701538a0 0x7f887015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58784/0 (socket says 172.21.15.46:58784) 2023-10-30T14:07:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f88758b9640 1 -- 172.21.15.46:0/2788704817 learned_addr learned my addr 172.21.15.46:0/2788704817 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f88750b8640 1 --2- 172.21.15.46:0/2788704817 >> v2:172.21.15.154:3300/0 conn(0x7f88700ca0e0 0x7f887015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f88758b9640 1 -- 172.21.15.46:0/2788704817 >> v2:172.21.15.154:3300/0 conn(0x7f88700ca0e0 msgr2=0x7f887015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f88758b9640 1 --2- 172.21.15.46:0/2788704817 >> v2:172.21.15.154:3300/0 conn(0x7f88700ca0e0 0x7f887015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f88758b9640 1 -- 172.21.15.46:0/2788704817 >> v1:172.21.15.154:6789/0 conn(0x7f88700cf160 legacy=0x7f88700cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.378+0000 7f88758b9640 1 -- 172.21.15.46:0/2788704817 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88700cad70 con 0x7f88701538a0 2023-10-30T14:07:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.382+0000 7f88758b9640 1 --2- 172.21.15.46:0/2788704817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88701538a0 0x7f887015d6f0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f88700b30a0 tx=0x7f886c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d722327eb07085 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.382+0000 7f88748b7640 1 -- 172.21.15.46:0/2788704817 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c0a89f0 con 0x7f88701538a0 2023-10-30T14:07:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.382+0000 7f88748b7640 1 -- 172.21.15.46:0/2788704817 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886c0a8b50 con 0x7f88701538a0 2023-10-30T14:07:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.382+0000 7f88748b7640 1 -- 172.21.15.46:0/2788704817 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c0b0440 con 0x7f88701538a0 2023-10-30T14:07:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.382+0000 7f8877b1d640 1 -- 172.21.15.46:0/2788704817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88701538a0 msgr2=0x7f887015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.382+0000 7f8877b1d640 1 --2- 172.21.15.46:0/2788704817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88701538a0 0x7f887015d6f0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f88700b30a0 tx=0x7f886c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.382+0000 7f8877b1d640 1 -- 172.21.15.46:0/2788704817 shutdown_connections 2023-10-30T14:07:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.382+0000 7f8877b1d640 1 --2- 172.21.15.46:0/2788704817 >> v2:172.21.15.154:3300/0 conn(0x7f88700ca0e0 0x7f887015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.382+0000 7f8877b1d640 1 --2- 172.21.15.46:0/2788704817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88701538a0 0x7f887015d6f0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.382+0000 7f8877b1d640 1 -- 172.21.15.46:0/2788704817 >> 172.21.15.46:0/2788704817 conn(0x7f887006bfe0 msgr2=0x7f887006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.382+0000 7f8877b1d640 1 -- 172.21.15.46:0/2788704817 shutdown_connections 2023-10-30T14:07:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.382+0000 7f8877b1d640 1 -- 172.21.15.46:0/2788704817 wait complete. 2023-10-30T14:07:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f8877b1d640 1 Processor -- start 2023-10-30T14:07:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f8877b1d640 1 -- start start 2023-10-30T14:07:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f8877b1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88700ca0e0 0x7f887014ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f8877b1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88700cf160 0x7f88701ef1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f8877b1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88701538a0 0x7f88701ef6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f8877b1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88700c88f0 con 0x7f88700cf160 2023-10-30T14:07:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f8877b1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88701efc20 con 0x7f88700ca0e0 2023-10-30T14:07:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f8877b1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88701f3320 con 0x7f88701538a0 2023-10-30T14:07:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88750b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88700cf160 0x7f88701ef1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88750b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88700cf160 0x7f88701ef1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58796/0 (socket says 172.21.15.46:58796) 2023-10-30T14:07:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88750b8640 1 -- 172.21.15.46:0/477802747 learned_addr learned my addr 172.21.15.46:0/477802747 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88758b9640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88700ca0e0 0x7f887014ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88760ba640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88701538a0 0x7f88701ef6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88750b8640 1 -- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88701538a0 msgr2=0x7f88701ef6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88750b8640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88701538a0 0x7f88701ef6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88750b8640 1 -- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88700ca0e0 msgr2=0x7f887014ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88750b8640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88700ca0e0 0x7f887014ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88750b8640 1 -- 172.21.15.46:0/477802747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c0094d0 con 0x7f88700cf160 2023-10-30T14:07:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88750b8640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88700cf160 0x7f88701ef1a0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f886000d830 tx=0x7f886000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:16.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88667fc640 1 -- 172.21.15.46:0/477802747 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886000b6c0 con 0x7f88700cf160 2023-10-30T14:07:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88667fc640 1 -- 172.21.15.46:0/477802747 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886000b820 con 0x7f88700cf160 2023-10-30T14:07:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f88667fc640 1 -- 172.21.15.46:0/477802747 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886000bb90 con 0x7f88700cf160 2023-10-30T14:07:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f8877b1d640 1 -- 172.21.15.46:0/477802747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88700c9070 con 0x7f88700cf160 2023-10-30T14:07:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.386+0000 7f8877b1d640 1 -- 172.21.15.46:0/477802747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88701f2710 con 0x7f88700cf160 2023-10-30T14:07:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.390+0000 7f8877b1d640 1 -- 172.21.15.46:0/477802747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88700b2a50 con 0x7f88700cf160 2023-10-30T14:07:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.390+0000 7f88667fc640 1 -- 172.21.15.46:0/477802747 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f886001e460 con 0x7f88700cf160 2023-10-30T14:07:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.390+0000 7f88667fc640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8854066520 0x7f88540689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.390+0000 7f88667fc640 1 -- 172.21.15.46:0/477802747 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f8860084dd0 con 0x7f88700cf160 2023-10-30T14:07:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.390+0000 7f88758b9640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8854066520 0x7f88540689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.390+0000 7f88758b9640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8854066520 0x7f88540689e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f886c0b2000 tx=0x7f886c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:16.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.394+0000 7f88667fc640 1 -- 172.21.15.46:0/477802747 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8860051a00 con 0x7f88700cf160 2023-10-30T14:07:16.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.586+0000 7f8877b1d640 1 -- 172.21.15.46:0/477802747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f887004e170 con 0x7f88700cf160 2023-10-30T14:07:16.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.586+0000 7f88667fc640 1 -- 172.21.15.46:0/477802747 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f8860055020 con 0x7f88700cf160 2023-10-30T14:07:16.590 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:16.591 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:14.520426+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:16.591 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 189 2023-10-30T14:07:16.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.590+0000 7f8877b1d640 1 -- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8854066520 msgr2=0x7f88540689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.590+0000 7f8877b1d640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8854066520 0x7f88540689e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f886c0b2000 tx=0x7f886c009040 comp rx=0 tx=0).stop 2023-10-30T14:07:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.590+0000 7f8877b1d640 1 -- 172.21.15.46:0/477802747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88700cf160 msgr2=0x7f88701ef1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.590+0000 7f8877b1d640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88700cf160 0x7f88701ef1a0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f886000d830 tx=0x7f886000dcf0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.590+0000 7f8877b1d640 1 -- 172.21.15.46:0/477802747 shutdown_connections 2023-10-30T14:07:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.590+0000 7f8877b1d640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8854066520 0x7f88540689e0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.590+0000 7f8877b1d640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88701538a0 0x7f88701ef6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.590+0000 7f8877b1d640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88700cf160 0x7f88701ef1a0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.590+0000 7f8877b1d640 1 --2- 172.21.15.46:0/477802747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88700ca0e0 0x7f887014ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.590+0000 7f8877b1d640 1 -- 172.21.15.46:0/477802747 >> 172.21.15.46:0/477802747 conn(0x7f887006bfe0 msgr2=0x7f88700c4500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.590+0000 7f8877b1d640 1 -- 172.21.15.46:0/477802747 shutdown_connections 2023-10-30T14:07:16.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.590+0000 7f8877b1d640 1 -- 172.21.15.46:0/477802747 wait complete. 2023-10-30T14:07:16.611 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T14:07:16.611 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:07:16.612 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:16.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.750+0000 7f9ac346b640 1 Processor -- start 2023-10-30T14:07:16.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.750+0000 7f9ac346b640 1 -- start start 2023-10-30T14:07:16.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac346b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9abc0c6540 0x7f9abc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac346b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9abc0c5810 con 0x7f9abc070740 2023-10-30T14:07:16.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac346b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9abc0c69b0 con 0x7f9abc1580e0 2023-10-30T14:07:16.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac346b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc06acb0 con 0x7f9abc0c6540 2023-10-30T14:07:16.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac0a06640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9abc070740 0x7f9abc06a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59088/0 (socket says 172.21.15.46:59088) 2023-10-30T14:07:16.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac0a06640 1 -- 172.21.15.46:0/1131693113 learned_addr learned my addr 172.21.15.46:0/1131693113 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:16.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac1a08640 1 --2- 172.21.15.46:0/1131693113 >> v2:172.21.15.154:3300/0 conn(0x7f9abc0c6540 0x7f9abc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443804872 0 0) 0x7f9abc0c5810 con 0x7f9abc070740 2023-10-30T14:07:16.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa8003530 con 0x7f9abc070740 2023-10-30T14:07:16.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 803875886 0 0) 0x7f9aa8003530 con 0x7f9abc070740 2023-10-30T14:07:16.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 >> v2:172.21.15.154:3300/0 conn(0x7f9abc0c6540 msgr2=0x7f9abc15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 --2- 172.21.15.46:0/1131693113 >> v2:172.21.15.154:3300/0 conn(0x7f9abc0c6540 0x7f9abc15e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 >> v1:172.21.15.154:6789/0 conn(0x7f9abc1580e0 legacy=0x7f9abc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc0c6bb0 con 0x7f9abc070740 2023-10-30T14:07:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f9aac001680 con 0x7f9abc070740 2023-10-30T14:07:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 >> v1:172.21.15.46:6789/0 conn(0x7f9abc070740 legacy=0x7f9abc06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 --2- 172.21.15.46:0/1131693113 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9aa8003980 0x7f9aa8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 --2- 172.21.15.46:0/1131693113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa80047d0 0x7f9aa8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 --2- 172.21.15.46:0/1131693113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 0x7f9aa800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa800c000 con 0x7f9aa8005190 2023-10-30T14:07:16.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa800c170 con 0x7f9aa80047d0 2023-10-30T14:07:16.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa800c330 con 0x7f9aa8003980 2023-10-30T14:07:16.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac0a06640 1 --2- 172.21.15.46:0/1131693113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa80047d0 0x7f9aa8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac1a08640 1 --2- 172.21.15.46:0/1131693113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 0x7f9aa800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac1207640 1 --2- 172.21.15.46:0/1131693113 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9aa8003980 0x7f9aa8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac1a08640 1 -- 172.21.15.46:0/1131693113 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9aa8003980 msgr2=0x7f9aa8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:16.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac1a08640 1 --2- 172.21.15.46:0/1131693113 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9aa8003980 0x7f9aa8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac1a08640 1 -- 172.21.15.46:0/1131693113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa80047d0 msgr2=0x7f9aa8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:16.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac1a08640 1 --2- 172.21.15.46:0/1131693113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa80047d0 0x7f9aa8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.754+0000 7f9ac1a08640 1 -- 172.21.15.46:0/1131693113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa8003530 con 0x7f9aa8005190 2023-10-30T14:07:16.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.758+0000 7f9ac1a08640 1 -- 172.21.15.46:0/1131693113 reap_dead start 2023-10-30T14:07:16.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.758+0000 7f9ac1a08640 1 --2- 172.21.15.46:0/1131693113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 0x7f9aa800b990 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009fc0 tx=0x7f9ab401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e3fa0e26ddc7f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:16.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.758+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab40a5cb0 con 0x7f9aa8005190 2023-10-30T14:07:16.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.758+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab400f040 con 0x7f9aa8005190 2023-10-30T14:07:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.758+0000 7f9ab3fff640 1 -- 172.21.15.46:0/1131693113 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4003be0 con 0x7f9aa8005190 2023-10-30T14:07:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.758+0000 7f9ac346b640 1 -- 172.21.15.46:0/1131693113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 msgr2=0x7f9aa800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.758+0000 7f9ac346b640 1 --2- 172.21.15.46:0/1131693113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 0x7f9aa800b990 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009fc0 tx=0x7f9ab401cca0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.758+0000 7f9ac346b640 1 -- 172.21.15.46:0/1131693113 shutdown_connections 2023-10-30T14:07:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.758+0000 7f9ac346b640 1 --2- 172.21.15.46:0/1131693113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 0x7f9aa800b990 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.758+0000 7f9ac346b640 1 -- 172.21.15.46:0/1131693113 >> 172.21.15.46:0/1131693113 conn(0x7f9abc0c7910 msgr2=0x7f9abc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:16.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.758+0000 7f9ac346b640 1 -- 172.21.15.46:0/1131693113 shutdown_connections 2023-10-30T14:07:16.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.758+0000 7f9ac346b640 1 -- 172.21.15.46:0/1131693113 wait complete. 2023-10-30T14:07:16.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac346b640 1 Processor -- start 2023-10-30T14:07:16.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac346b640 1 -- start start 2023-10-30T14:07:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac346b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 0x7f9abc1f3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac346b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9abc0c9d80 0x7f9abc0c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac346b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9abc0ca1c0 0x7f9abc149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac346b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc0c2b60 con 0x7f9aa8005190 2023-10-30T14:07:16.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac346b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc1f4210 con 0x7f9abc0c9d80 2023-10-30T14:07:16.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac346b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc0c2320 con 0x7f9abc0ca1c0 2023-10-30T14:07:16.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac1207640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 0x7f9abc1f3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac1207640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 0x7f9abc1f3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36058/0 (socket says 172.21.15.46:36058) 2023-10-30T14:07:16.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac1207640 1 -- 172.21.15.46:0/3737981363 learned_addr learned my addr 172.21.15.46:0/3737981363 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:16.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac1a08640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9abc0ca1c0 0x7f9abc149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac0a06640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9abc0c9d80 0x7f9abc0c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac1207640 1 -- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9abc0ca1c0 msgr2=0x7f9abc149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac1207640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9abc0ca1c0 0x7f9abc149cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac1207640 1 -- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9abc0c9d80 msgr2=0x7f9abc0c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac1207640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9abc0c9d80 0x7f9abc0c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac1207640 1 -- 172.21.15.46:0/3737981363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc0c25a0 con 0x7f9aa8005190 2023-10-30T14:07:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac0a06640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9abc0c9d80 0x7f9abc0c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:07:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac1207640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 0x7f9abc1f3cd0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800f5f0 tx=0x7f9ab800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:16.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ab1ffb640 1 -- 172.21.15.46:0/3737981363 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80052e0 con 0x7f9aa8005190 2023-10-30T14:07:16.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ab1ffb640 1 -- 172.21.15.46:0/3737981363 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab8005440 con 0x7f9aa8005190 2023-10-30T14:07:16.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ab1ffb640 1 -- 172.21.15.46:0/3737981363 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8003820 con 0x7f9aa8005190 2023-10-30T14:07:16.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac346b640 1 -- 172.21.15.46:0/3737981363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc0c2880 con 0x7f9aa8005190 2023-10-30T14:07:16.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac346b640 1 -- 172.21.15.46:0/3737981363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc0c38f0 con 0x7f9aa8005190 2023-10-30T14:07:16.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.762+0000 7f9ac346b640 1 -- 172.21.15.46:0/3737981363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9abc0c3ab0 con 0x7f9aa8005190 2023-10-30T14:07:16.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.766+0000 7f9ab1ffb640 1 -- 172.21.15.46:0/3737981363 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ab8003980 con 0x7f9aa8005190 2023-10-30T14:07:16.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.766+0000 7f9ab1ffb640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9aa00664a0 0x7f9aa0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:16.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.766+0000 7f9ab1ffb640 1 -- 172.21.15.46:0/3737981363 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f9ab8084680 con 0x7f9aa8005190 2023-10-30T14:07:16.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.766+0000 7f9ac0a06640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9aa00664a0 0x7f9aa0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:16.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.766+0000 7f9ac0a06640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9aa00664a0 0x7f9aa0068960 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0c1990 tx=0x7f9aac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:16.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.770+0000 7f9ab1ffb640 1 -- 172.21.15.46:0/3737981363 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ab8051180 con 0x7f9aa8005190 2023-10-30T14:07:16.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.962+0000 7f9ac346b640 1 -- 172.21.15.46:0/3737981363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9abc0c3470 con 0x7f9aa8005190 2023-10-30T14:07:16.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.962+0000 7f9ab1ffb640 1 -- 172.21.15.46:0/3737981363 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f9abc0c3470 con 0x7f9aa8005190 2023-10-30T14:07:16.967 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:16.967 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:14.520426+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:16.968 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 189 2023-10-30T14:07:16.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.966+0000 7f9ac346b640 1 -- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9aa00664a0 msgr2=0x7f9aa0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.966+0000 7f9ac346b640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9aa00664a0 0x7f9aa0068960 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0c1990 tx=0x7f9aac009040 comp rx=0 tx=0).stop 2023-10-30T14:07:16.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.966+0000 7f9ac346b640 1 -- 172.21.15.46:0/3737981363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 msgr2=0x7f9abc1f3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:16.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.966+0000 7f9ac346b640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 0x7f9abc1f3cd0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800f5f0 tx=0x7f9ab800fab0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.966+0000 7f9ac346b640 1 -- 172.21.15.46:0/3737981363 shutdown_connections 2023-10-30T14:07:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.966+0000 7f9ac346b640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9aa00664a0 0x7f9aa0068960 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.966+0000 7f9ac346b640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9abc0ca1c0 0x7f9abc149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.966+0000 7f9ac346b640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9abc0c9d80 0x7f9abc0c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.966+0000 7f9ac346b640 1 --2- 172.21.15.46:0/3737981363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9aa8005190 0x7f9abc1f3cd0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:16.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.966+0000 7f9ac346b640 1 -- 172.21.15.46:0/3737981363 >> 172.21.15.46:0/3737981363 conn(0x7f9abc0c7910 msgr2=0x7f9abc070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:16.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.966+0000 7f9ac346b640 1 -- 172.21.15.46:0/3737981363 shutdown_connections 2023-10-30T14:07:16.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:16.966+0000 7f9ac346b640 1 -- 172.21.15.46:0/3737981363 wait complete. 2023-10-30T14:07:16.988 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 189, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 189, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8352": { "addr": "172.21.15.46:6836/128865315", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 128865315, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 128865315, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8352, "incarnation": 182, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 21 }, "gid_8505": { "addr": "172.21.15.46:6833/4078242540", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4078242540, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4078242540, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8505, "incarnation": 189, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 80, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:07:14.520426+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8505, "mds_1": 8352 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:07:18.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:19.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.142+0000 7f3f6d846640 1 Processor -- start 2023-10-30T14:07:19.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.142+0000 7f3f6d846640 1 -- start start 2023-10-30T14:07:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.146+0000 7f3f6d846640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3f681538a0 0x7f3f6815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.146+0000 7f3f6d846640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f680cb800 con 0x7f3f680cf160 2023-10-30T14:07:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.146+0000 7f3f6d846640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6815dc30 con 0x7f3f680ca0e0 2023-10-30T14:07:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.146+0000 7f3f6d846640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f680cb330 con 0x7f3f681538a0 2023-10-30T14:07:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.146+0000 7f3f677fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3f680cf160 0x7f3f68164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59090/0 (socket says 172.21.15.46:59090) 2023-10-30T14:07:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.146+0000 7f3f677fe640 1 -- 172.21.15.46:0/4239862612 learned_addr learned my addr 172.21.15.46:0/4239862612 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:19.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.146+0000 7f3f66ffd640 1 --2- 172.21.15.46:0/4239862612 >> v2:172.21.15.46:3300/0 conn(0x7f3f681538a0 0x7f3f6815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:19.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.146+0000 7f3f667fc640 1 -- 172.21.15.46:0/4239862612 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450059478 0 0) 0x7f3f680cb800 con 0x7f3f680cf160 2023-10-30T14:07:19.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.146+0000 7f3f667fc640 1 -- 172.21.15.46:0/4239862612 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f54003530 con 0x7f3f680cf160 2023-10-30T14:07:19.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.146+0000 7f3f66ffd640 1 -- 172.21.15.46:0/4239862612 >> v1:172.21.15.154:6789/0 conn(0x7f3f680ca0e0 legacy=0x7f3f6815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:19.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.146+0000 7f3f66ffd640 1 -- 172.21.15.46:0/4239862612 >> v1:172.21.15.46:6789/0 conn(0x7f3f680cf160 legacy=0x7f3f68164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:19.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.146+0000 7f3f66ffd640 1 -- 172.21.15.46:0/4239862612 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f680cad70 con 0x7f3f681538a0 2023-10-30T14:07:19.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f66ffd640 1 --2- 172.21.15.46:0/4239862612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f681538a0 0x7f3f6815d6f0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f3f580097c0 tx=0x7f3f5801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b59e9c42aa5505 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:19.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f667fc640 1 -- 172.21.15.46:0/4239862612 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f580a8970 con 0x7f3f681538a0 2023-10-30T14:07:19.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f667fc640 1 -- 172.21.15.46:0/4239862612 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f580a8ad0 con 0x7f3f681538a0 2023-10-30T14:07:19.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f667fc640 1 -- 172.21.15.46:0/4239862612 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f580a8e60 con 0x7f3f681538a0 2023-10-30T14:07:19.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f6d846640 1 -- 172.21.15.46:0/4239862612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f681538a0 msgr2=0x7f3f6815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:19.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f6d846640 1 --2- 172.21.15.46:0/4239862612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f681538a0 0x7f3f6815d6f0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f3f580097c0 tx=0x7f3f5801ec70 comp rx=0 tx=0).stop 2023-10-30T14:07:19.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f6d846640 1 -- 172.21.15.46:0/4239862612 shutdown_connections 2023-10-30T14:07:19.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f6d846640 1 --2- 172.21.15.46:0/4239862612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f681538a0 0x7f3f6815d6f0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:19.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f6d846640 1 -- 172.21.15.46:0/4239862612 >> 172.21.15.46:0/4239862612 conn(0x7f3f6806bfe0 msgr2=0x7f3f6806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:19.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f6d846640 1 -- 172.21.15.46:0/4239862612 shutdown_connections 2023-10-30T14:07:19.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f6d846640 1 -- 172.21.15.46:0/4239862612 wait complete. 2023-10-30T14:07:19.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f6d846640 1 Processor -- start 2023-10-30T14:07:19.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.150+0000 7f3f6d846640 1 -- start start 2023-10-30T14:07:19.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f6d846640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f680ca0e0 0x7f3f680ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:19.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f6d846640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680cf160 0x7f3f68148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:19.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f6d846640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f681538a0 0x7f3f681ef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:19.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f6d846640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f68146720 con 0x7f3f681538a0 2023-10-30T14:07:19.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f6d846640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f680b31a0 con 0x7f3f680ca0e0 2023-10-30T14:07:19.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f6d846640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f680b42d0 con 0x7f3f680cf160 2023-10-30T14:07:19.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f681538a0 0x7f3f681ef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:19.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f681538a0 0x7f3f681ef3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36070/0 (socket says 172.21.15.46:36070) 2023-10-30T14:07:19.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f677fe640 1 -- 172.21.15.46:0/2022809992 learned_addr learned my addr 172.21.15.46:0/2022809992 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:19.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f66ffd640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f680ca0e0 0x7f3f680ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:19.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f5ffff640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680cf160 0x7f3f68148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:19.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f677fe640 1 -- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680cf160 msgr2=0x7f3f68148f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:19.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f677fe640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680cf160 0x7f3f68148f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:19.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f677fe640 1 -- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f680ca0e0 msgr2=0x7f3f680ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:19.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f677fe640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f680ca0e0 0x7f3f680ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:19.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f677fe640 1 -- 172.21.15.46:0/2022809992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f58009420 con 0x7f3f681538a0 2023-10-30T14:07:19.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f677fe640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f681538a0 0x7f3f681ef3f0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f3f6000cad0 tx=0x7f3f6000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f64ff9640 1 -- 172.21.15.46:0/2022809992 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6000abd0 con 0x7f3f681538a0 2023-10-30T14:07:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f64ff9640 1 -- 172.21.15.46:0/2022809992 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6000ad30 con 0x7f3f681538a0 2023-10-30T14:07:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f6d846640 1 -- 172.21.15.46:0/2022809992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f681f36b0 con 0x7f3f681538a0 2023-10-30T14:07:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f64ff9640 1 -- 172.21.15.46:0/2022809992 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6000aea0 con 0x7f3f681538a0 2023-10-30T14:07:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f6d846640 1 -- 172.21.15.46:0/2022809992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f680b33f0 con 0x7f3f681538a0 2023-10-30T14:07:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f6d846640 1 -- 172.21.15.46:0/2022809992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6804e170 con 0x7f3f681538a0 2023-10-30T14:07:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.154+0000 7f3f64ff9640 1 -- 172.21.15.46:0/2022809992 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3f60007070 con 0x7f3f681538a0 2023-10-30T14:07:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.158+0000 7f3f64ff9640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f480664d0 0x7f3f48068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.158+0000 7f3f64ff9640 1 -- 172.21.15.46:0/2022809992 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f3f60084680 con 0x7f3f681538a0 2023-10-30T14:07:19.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.158+0000 7f3f66ffd640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f480664d0 0x7f3f48068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:19.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.158+0000 7f3f66ffd640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f480664d0 0x7f3f48068990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f3f580b2000 tx=0x7f3f58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:19.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.162+0000 7f3f64ff9640 1 -- 172.21.15.46:0/2022809992 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3f60051210 con 0x7f3f681538a0 2023-10-30T14:07:19.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.354+0000 7f3f6d846640 1 -- 172.21.15.46:0/2022809992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f680b4410 con 0x7f3f681538a0 2023-10-30T14:07:19.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.354+0000 7f3f64ff9640 1 -- 172.21.15.46:0/2022809992 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f3f60054830 con 0x7f3f681538a0 2023-10-30T14:07:19.359 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:19.359 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:14.520426+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:19.359 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 189 2023-10-30T14:07:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.358+0000 7f3f6d846640 1 -- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f480664d0 msgr2=0x7f3f48068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.358+0000 7f3f6d846640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f480664d0 0x7f3f48068990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f3f580b2000 tx=0x7f3f58008040 comp rx=0 tx=0).stop 2023-10-30T14:07:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.358+0000 7f3f6d846640 1 -- 172.21.15.46:0/2022809992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f681538a0 msgr2=0x7f3f681ef3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.358+0000 7f3f6d846640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f681538a0 0x7f3f681ef3f0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f3f6000cad0 tx=0x7f3f6000cf90 comp rx=0 tx=0).stop 2023-10-30T14:07:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.358+0000 7f3f6d846640 1 -- 172.21.15.46:0/2022809992 shutdown_connections 2023-10-30T14:07:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.358+0000 7f3f6d846640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f480664d0 0x7f3f48068990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.358+0000 7f3f6d846640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f681538a0 0x7f3f681ef3f0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.358+0000 7f3f6d846640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680cf160 0x7f3f68148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.358+0000 7f3f6d846640 1 --2- 172.21.15.46:0/2022809992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f680ca0e0 0x7f3f680ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.358+0000 7f3f6d846640 1 -- 172.21.15.46:0/2022809992 >> 172.21.15.46:0/2022809992 conn(0x7f3f6806bfe0 msgr2=0x7f3f680c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.358+0000 7f3f6d846640 1 -- 172.21.15.46:0/2022809992 shutdown_connections 2023-10-30T14:07:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:19.358+0000 7f3f6d846640 1 -- 172.21.15.46:0/2022809992 wait complete. 2023-10-30T14:07:19.380 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:07:21.381 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:21.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.526+0000 7fe0d8e74640 1 Processor -- start 2023-10-30T14:07:21.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.526+0000 7fe0d8e74640 1 -- start start 2023-10-30T14:07:21.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.526+0000 7fe0d8e74640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe0d41538f0 0x7fe0d4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:21.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.526+0000 7fe0d8e74640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d415cd50 0x7fe0d40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:21.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.526+0000 7fe0d8e74640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe0d40cefa0 0x7fe0d40cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:21.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.526+0000 7fe0d8e74640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d415d300 con 0x7fe0d41538f0 2023-10-30T14:07:21.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.526+0000 7fe0d8e74640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d415d5e0 con 0x7fe0d40cefa0 2023-10-30T14:07:21.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.526+0000 7fe0d8e74640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d415d8c0 con 0x7fe0d415cd50 2023-10-30T14:07:21.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.526+0000 7fe0d37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe0d41538f0 0x7fe0d4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:21.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.526+0000 7fe0d37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe0d41538f0 0x7fe0d4159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36078/0 (socket says 172.21.15.46:36078) 2023-10-30T14:07:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.526+0000 7fe0d37fe640 1 -- 172.21.15.46:0/561809625 learned_addr learned my addr 172.21.15.46:0/561809625 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.530+0000 7fe0d3fff640 1 --2- 172.21.15.46:0/561809625 >> v2:172.21.15.154:3300/0 conn(0x7fe0d40cefa0 0x7fe0d40cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.530+0000 7fe0d2ffd640 1 --2- 172.21.15.46:0/561809625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d415cd50 0x7fe0d40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.530+0000 7fe0d37fe640 1 -- 172.21.15.46:0/561809625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d415cd50 msgr2=0x7fe0d40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.530+0000 7fe0d37fe640 1 --2- 172.21.15.46:0/561809625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d415cd50 0x7fe0d40c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.530+0000 7fe0d37fe640 1 -- 172.21.15.46:0/561809625 >> v2:172.21.15.154:3300/0 conn(0x7fe0d40cefa0 msgr2=0x7fe0d40cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:21.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.530+0000 7fe0d37fe640 1 --2- 172.21.15.46:0/561809625 >> v2:172.21.15.154:3300/0 conn(0x7fe0d40cefa0 0x7fe0d40cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.530+0000 7fe0d37fe640 1 -- 172.21.15.46:0/561809625 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d415dba0 con 0x7fe0d41538f0 2023-10-30T14:07:21.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.530+0000 7fe0d37fe640 1 --2- 172.21.15.46:0/561809625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d41538f0 0x7fe0d4159fb0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fe0d40b1ae0 tx=0x7fe0c401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24b6cdd228552f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:21.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.530+0000 7fe0d27fc640 1 -- 172.21.15.46:0/561809625 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c40a89b0 con 0x7fe0d41538f0 2023-10-30T14:07:21.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.530+0000 7fe0d27fc640 1 -- 172.21.15.46:0/561809625 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c40a8b10 con 0x7fe0d41538f0 2023-10-30T14:07:21.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.530+0000 7fe0d27fc640 1 -- 172.21.15.46:0/561809625 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c40a8ea0 con 0x7fe0d41538f0 2023-10-30T14:07:21.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 -- 172.21.15.46:0/561809625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d41538f0 msgr2=0x7fe0d4159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:21.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 --2- 172.21.15.46:0/561809625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d41538f0 0x7fe0d4159fb0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fe0d40b1ae0 tx=0x7fe0c401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 -- 172.21.15.46:0/561809625 shutdown_connections 2023-10-30T14:07:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 --2- 172.21.15.46:0/561809625 >> v2:172.21.15.154:3300/0 conn(0x7fe0d40cefa0 0x7fe0d40cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 --2- 172.21.15.46:0/561809625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d415cd50 0x7fe0d40c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 --2- 172.21.15.46:0/561809625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d41538f0 0x7fe0d4159fb0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 -- 172.21.15.46:0/561809625 >> 172.21.15.46:0/561809625 conn(0x7fe0d406a640 msgr2=0x7fe0d4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:21.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 -- 172.21.15.46:0/561809625 shutdown_connections 2023-10-30T14:07:21.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 -- 172.21.15.46:0/561809625 wait complete. 2023-10-30T14:07:21.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 Processor -- start 2023-10-30T14:07:21.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 -- start start 2023-10-30T14:07:21.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d40cefa0 0x7fe0d41f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:21.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0d41538f0 0x7fe0d41505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:21.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d415cd50 0x7fe0d41e6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:21.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d40ce720 con 0x7fe0d415cd50 2023-10-30T14:07:21.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d40cea00 con 0x7fe0d41538f0 2023-10-30T14:07:21.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d8e74640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d41f26f0 con 0x7fe0d40cefa0 2023-10-30T14:07:21.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d415cd50 0x7fe0d41e6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:21.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d415cd50 0x7fe0d41e6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36084/0 (socket says 172.21.15.46:36084) 2023-10-30T14:07:21.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d3fff640 1 -- 172.21.15.46:0/2050396935 learned_addr learned my addr 172.21.15.46:0/2050396935 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:21.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d37fe640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d40cefa0 0x7fe0d41f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:21.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.534+0000 7fe0d2ffd640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0d41538f0 0x7fe0d41505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:21.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0d3fff640 1 -- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d40cefa0 msgr2=0x7fe0d41f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:21.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0d3fff640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d40cefa0 0x7fe0d41f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0d3fff640 1 -- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0d41538f0 msgr2=0x7fe0d41505a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:21.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0d3fff640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0d41538f0 0x7fe0d41505a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0d3fff640 1 -- 172.21.15.46:0/2050396935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c40094d0 con 0x7fe0d415cd50 2023-10-30T14:07:21.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0d3fff640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d415cd50 0x7fe0d41e6030 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc007e60 tx=0x7fe0cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0bbfff640 1 -- 172.21.15.46:0/2050396935 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc014620 con 0x7fe0d415cd50 2023-10-30T14:07:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0bbfff640 1 -- 172.21.15.46:0/2050396935 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0cc00ee90 con 0x7fe0d415cd50 2023-10-30T14:07:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0bbfff640 1 -- 172.21.15.46:0/2050396935 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc0149e0 con 0x7fe0d415cd50 2023-10-30T14:07:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0d8e74640 1 -- 172.21.15.46:0/2050396935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d41f29d0 con 0x7fe0d415cd50 2023-10-30T14:07:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0d8e74640 1 -- 172.21.15.46:0/2050396935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d41523a0 con 0x7fe0d415cd50 2023-10-30T14:07:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0d8e74640 1 -- 172.21.15.46:0/2050396935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d404e170 con 0x7fe0d415cd50 2023-10-30T14:07:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.538+0000 7fe0bbfff640 1 -- 172.21.15.46:0/2050396935 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe0cc014b40 con 0x7fe0d415cd50 2023-10-30T14:07:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.542+0000 7fe0bbfff640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0c00664a0 0x7fe0c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.542+0000 7fe0bbfff640 1 -- 172.21.15.46:0/2050396935 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7fe0cc084f70 con 0x7fe0d415cd50 2023-10-30T14:07:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.542+0000 7fe0d37fe640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0c00664a0 0x7fe0c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.542+0000 7fe0d37fe640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0c00664a0 0x7fe0c0068960 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe0c40b2000 tx=0x7fe0c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.550+0000 7fe0bbfff640 1 -- 172.21.15.46:0/2050396935 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe0cc0b6090 con 0x7fe0d415cd50 2023-10-30T14:07:21.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.770+0000 7fe0d8e74640 1 -- 172.21.15.46:0/2050396935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0d41f2e80 con 0x7fe0d415cd50 2023-10-30T14:07:21.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0bbfff640 1 -- 172.21.15.46:0/2050396935 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fe0cc051ba0 con 0x7fe0d415cd50 2023-10-30T14:07:21.775 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:21.776 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:14.520426+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:21.776 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 189 2023-10-30T14:07:21.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0d8e74640 1 -- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0c00664a0 msgr2=0x7fe0c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:21.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0d8e74640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0c00664a0 0x7fe0c0068960 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe0c40b2000 tx=0x7fe0c4009040 comp rx=0 tx=0).stop 2023-10-30T14:07:21.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0d8e74640 1 -- 172.21.15.46:0/2050396935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d415cd50 msgr2=0x7fe0d41e6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:21.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0d8e74640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d415cd50 0x7fe0d41e6030 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc007e60 tx=0x7fe0cc00e4b0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0d8e74640 1 -- 172.21.15.46:0/2050396935 shutdown_connections 2023-10-30T14:07:21.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0d8e74640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0c00664a0 0x7fe0c0068960 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0d8e74640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d415cd50 0x7fe0d41e6030 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0d8e74640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0d41538f0 0x7fe0d41505a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0d8e74640 1 --2- 172.21.15.46:0/2050396935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d40cefa0 0x7fe0d41f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:21.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0d8e74640 1 -- 172.21.15.46:0/2050396935 >> 172.21.15.46:0/2050396935 conn(0x7fe0d406a640 msgr2=0x7fe0d40c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:21.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0d8e74640 1 -- 172.21.15.46:0/2050396935 shutdown_connections 2023-10-30T14:07:21.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:21.774+0000 7fe0d8e74640 1 -- 172.21.15.46:0/2050396935 wait complete. 2023-10-30T14:07:21.798 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:07:23.798 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.954+0000 7f2779091640 1 Processor -- start 2023-10-30T14:07:23.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.954+0000 7f2779091640 1 -- start start 2023-10-30T14:07:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f2779091640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f277406f7a0 0x7f277406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f2779091640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2774071110 0x7f2774158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:23.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f2779091640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2774066cf0 con 0x7f27740c6bd0 2023-10-30T14:07:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f2779091640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27740700c0 con 0x7f2774071110 2023-10-30T14:07:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f2779091640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27740c70d0 con 0x7f277406f7a0 2023-10-30T14:07:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f2772d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f277406f7a0 0x7f277406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f277259c640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f27740c6bd0 0x7f2774069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:48626/0 (socket says 172.21.15.46:48626) 2023-10-30T14:07:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f277259c640 1 -- 172.21.15.46:0/2401143314 learned_addr learned my addr 172.21.15.46:0/2401143314 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f277359e640 1 --2- 172.21.15.46:0/2401143314 >> v2:172.21.15.154:3300/0 conn(0x7f2774071110 0x7f2774158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:23.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f2771d9b640 1 -- 172.21.15.46:0/2401143314 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442081608 0 0) 0x7f2774066cf0 con 0x7f27740c6bd0 2023-10-30T14:07:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f2771d9b640 1 -- 172.21.15.46:0/2401143314 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f275c003530 con 0x7f27740c6bd0 2023-10-30T14:07:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f2772d9d640 1 -- 172.21.15.46:0/2401143314 >> v2:172.21.15.154:3300/0 conn(0x7f2774071110 msgr2=0x7f2774158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f2772d9d640 1 --2- 172.21.15.46:0/2401143314 >> v2:172.21.15.154:3300/0 conn(0x7f2774071110 0x7f2774158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f2772d9d640 1 -- 172.21.15.46:0/2401143314 >> v1:172.21.15.154:6789/0 conn(0x7f27740c6bd0 legacy=0x7f2774069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.958+0000 7f2772d9d640 1 -- 172.21.15.46:0/2401143314 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27740c7390 con 0x7f277406f7a0 2023-10-30T14:07:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.962+0000 7f2772d9d640 1 --2- 172.21.15.46:0/2401143314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f277406f7a0 0x7f277406fb80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f27640097c0 tx=0x7f276401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a52365d3d9596037 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.962+0000 7f2771d9b640 1 -- 172.21.15.46:0/2401143314 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27640a8970 con 0x7f277406f7a0 2023-10-30T14:07:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.962+0000 7f2771d9b640 1 -- 172.21.15.46:0/2401143314 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27640a8ad0 con 0x7f277406f7a0 2023-10-30T14:07:23.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.962+0000 7f2771d9b640 1 -- 172.21.15.46:0/2401143314 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27640a8e40 con 0x7f277406f7a0 2023-10-30T14:07:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.962+0000 7f2779091640 1 -- 172.21.15.46:0/2401143314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f277406f7a0 msgr2=0x7f277406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.962+0000 7f2779091640 1 --2- 172.21.15.46:0/2401143314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f277406f7a0 0x7f277406fb80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f27640097c0 tx=0x7f276401ec70 comp rx=0 tx=0).stop 2023-10-30T14:07:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.962+0000 7f2779091640 1 -- 172.21.15.46:0/2401143314 shutdown_connections 2023-10-30T14:07:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.962+0000 7f2779091640 1 --2- 172.21.15.46:0/2401143314 >> v2:172.21.15.154:3300/0 conn(0x7f2774071110 0x7f2774158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.962+0000 7f2779091640 1 --2- 172.21.15.46:0/2401143314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f277406f7a0 0x7f277406fb80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.962+0000 7f2779091640 1 -- 172.21.15.46:0/2401143314 >> 172.21.15.46:0/2401143314 conn(0x7f27740c7910 msgr2=0x7f27740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.962+0000 7f2779091640 1 -- 172.21.15.46:0/2401143314 shutdown_connections 2023-10-30T14:07:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.962+0000 7f2779091640 1 -- 172.21.15.46:0/2401143314 wait complete. 2023-10-30T14:07:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2779091640 1 Processor -- start 2023-10-30T14:07:23.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2779091640 1 -- start start 2023-10-30T14:07:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2779091640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f277406f7a0 0x7f27740c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2779091640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2774071110 0x7f27741588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2779091640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27740c6bd0 0x7f277414be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2779091640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27741594b0 con 0x7f277406f7a0 2023-10-30T14:07:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2779091640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277414a260 con 0x7f27740c6bd0 2023-10-30T14:07:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2779091640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2774149250 con 0x7f2774071110 2023-10-30T14:07:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2772d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f277406f7a0 0x7f27740c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2772d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f277406f7a0 0x7f27740c9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36100/0 (socket says 172.21.15.46:36100) 2023-10-30T14:07:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2772d9d640 1 -- 172.21.15.46:0/2458018102 learned_addr learned my addr 172.21.15.46:0/2458018102 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f277259c640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2774071110 0x7f27741588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f277359e640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27740c6bd0 0x7f277414be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2772d9d640 1 -- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2774071110 msgr2=0x7f27741588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2772d9d640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2774071110 0x7f27741588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2772d9d640 1 -- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27740c6bd0 msgr2=0x7f277414be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2772d9d640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27740c6bd0 0x7f277414be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2772d9d640 1 -- 172.21.15.46:0/2458018102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2764009420 con 0x7f277406f7a0 2023-10-30T14:07:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f277359e640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27740c6bd0 0x7f277414be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2772d9d640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f277406f7a0 0x7f27740c9ff0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f276401c8a0 tx=0x7f276401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f275b7fe640 1 -- 172.21.15.46:0/2458018102 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2764004450 con 0x7f277406f7a0 2023-10-30T14:07:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f275b7fe640 1 -- 172.21.15.46:0/2458018102 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2764003620 con 0x7f277406f7a0 2023-10-30T14:07:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f275b7fe640 1 -- 172.21.15.46:0/2458018102 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2764003990 con 0x7f277406f7a0 2023-10-30T14:07:23.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.966+0000 7f2779091640 1 -- 172.21.15.46:0/2458018102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277414b990 con 0x7f277406f7a0 2023-10-30T14:07:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.970+0000 7f2779091640 1 -- 172.21.15.46:0/2458018102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277414a790 con 0x7f277406f7a0 2023-10-30T14:07:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.970+0000 7f2779091640 1 -- 172.21.15.46:0/2458018102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f277414bb20 con 0x7f277406f7a0 2023-10-30T14:07:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.970+0000 7f275b7fe640 1 -- 172.21.15.46:0/2458018102 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f27640b7040 con 0x7f277406f7a0 2023-10-30T14:07:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.970+0000 7f275b7fe640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27600664d0 0x7f2760068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.970+0000 7f275b7fe640 1 -- 172.21.15.46:0/2458018102 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f2764121ce0 con 0x7f277406f7a0 2023-10-30T14:07:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.970+0000 7f277259c640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27600664d0 0x7f2760068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.974+0000 7f277259c640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27600664d0 0x7f2760068990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f27740c3310 tx=0x7f2768007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:23.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:23.974+0000 7f275b7fe640 1 -- 172.21.15.46:0/2458018102 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f27640ee990 con 0x7f277406f7a0 2023-10-30T14:07:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.174+0000 7f2779091640 1 -- 172.21.15.46:0/2458018102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f277404e170 con 0x7f277406f7a0 2023-10-30T14:07:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.174+0000 7f275b7fe640 1 -- 172.21.15.46:0/2458018102 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f27640f1fb0 con 0x7f277406f7a0 2023-10-30T14:07:24.177 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:24.177 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:14.520426+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 189 2023-10-30T14:07:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.174+0000 7f2779091640 1 -- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27600664d0 msgr2=0x7f2760068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.174+0000 7f2779091640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27600664d0 0x7f2760068990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f27740c3310 tx=0x7f2768007d40 comp rx=0 tx=0).stop 2023-10-30T14:07:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.174+0000 7f2779091640 1 -- 172.21.15.46:0/2458018102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f277406f7a0 msgr2=0x7f27740c9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.174+0000 7f2779091640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f277406f7a0 0x7f27740c9ff0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f276401c8a0 tx=0x7f276401ba10 comp rx=0 tx=0).stop 2023-10-30T14:07:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.174+0000 7f2779091640 1 -- 172.21.15.46:0/2458018102 shutdown_connections 2023-10-30T14:07:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.174+0000 7f2779091640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f27600664d0 0x7f2760068990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.174+0000 7f2779091640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27740c6bd0 0x7f277414be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.174+0000 7f2779091640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2774071110 0x7f27741588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.174+0000 7f2779091640 1 --2- 172.21.15.46:0/2458018102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f277406f7a0 0x7f27740c9ff0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.174+0000 7f2779091640 1 -- 172.21.15.46:0/2458018102 >> 172.21.15.46:0/2458018102 conn(0x7f27740c7910 msgr2=0x7f277406a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.178+0000 7f2779091640 1 -- 172.21.15.46:0/2458018102 shutdown_connections 2023-10-30T14:07:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:24.178+0000 7f2779091640 1 -- 172.21.15.46:0/2458018102 wait complete. 2023-10-30T14:07:24.199 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:07:26.201 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:26.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.354+0000 7f9bdcda0640 1 Processor -- start 2023-10-30T14:07:26.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.354+0000 7f9bdcda0640 1 -- start start 2023-10-30T14:07:26.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.354+0000 7f9bdcda0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9bd80cefa0 0x7f9bd80cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:26.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.354+0000 7f9bdcda0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd815a6c0 con 0x7f9bd81538f0 2023-10-30T14:07:26.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.354+0000 7f9bdcda0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd80ca6a0 con 0x7f9bd815cd50 2023-10-30T14:07:26.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.354+0000 7f9bdcda0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd80cf8f0 con 0x7f9bd80cefa0 2023-10-30T14:07:26.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.354+0000 7f9bd77fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9bd81538f0 0x7f9bd8159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59092/0 (socket says 172.21.15.46:59092) 2023-10-30T14:07:26.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.354+0000 7f9bd77fe640 1 -- 172.21.15.46:0/1114277293 learned_addr learned my addr 172.21.15.46:0/1114277293 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:26.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.354+0000 7f9bd7fff640 1 --2- 172.21.15.46:0/1114277293 >> v2:172.21.15.154:3300/0 conn(0x7f9bd80cefa0 0x7f9bd80cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:26.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615540684 0 0) 0x7f9bd815a6c0 con 0x7f9bd81538f0 2023-10-30T14:07:26.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc0003530 con 0x7f9bd81538f0 2023-10-30T14:07:26.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1903273896 0 0) 0x7f9bc0003530 con 0x7f9bd81538f0 2023-10-30T14:07:26.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 >> v2:172.21.15.154:3300/0 conn(0x7f9bd80cefa0 msgr2=0x7f9bd80cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:26.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 --2- 172.21.15.46:0/1114277293 >> v2:172.21.15.154:3300/0 conn(0x7f9bd80cefa0 0x7f9bd80cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:26.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 >> v1:172.21.15.154:6789/0 conn(0x7f9bd815cd50 legacy=0x7f9bd80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:26.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd815d300 con 0x7f9bd81538f0 2023-10-30T14:07:26.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f9bc8001680 con 0x7f9bd81538f0 2023-10-30T14:07:26.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 >> v1:172.21.15.46:6789/0 conn(0x7f9bd81538f0 legacy=0x7f9bd8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:26.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 --2- 172.21.15.46:0/1114277293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bc0003980 0x7f9bc0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:26.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 --2- 172.21.15.46:0/1114277293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bc00047d0 0x7f9bc0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:26.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 --2- 172.21.15.46:0/1114277293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bc0005190 0x7f9bc000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:26.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80cf8f0 con 0x7f9bc0005190 2023-10-30T14:07:26.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc000c040 con 0x7f9bc0003980 2023-10-30T14:07:26.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc000c200 con 0x7f9bc00047d0 2023-10-30T14:07:26.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd7fff640 1 --2- 172.21.15.46:0/1114277293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bc0005190 0x7f9bc000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:26.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd77fe640 1 --2- 172.21.15.46:0/1114277293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bc0003980 0x7f9bc0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:26.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd6ffd640 1 --2- 172.21.15.46:0/1114277293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bc00047d0 0x7f9bc0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:26.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd7fff640 1 -- 172.21.15.46:0/1114277293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bc00047d0 msgr2=0x7f9bc0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:26.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd7fff640 1 --2- 172.21.15.46:0/1114277293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bc00047d0 0x7f9bc0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:26.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd7fff640 1 -- 172.21.15.46:0/1114277293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bc0003980 msgr2=0x7f9bc0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:26.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd7fff640 1 --2- 172.21.15.46:0/1114277293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bc0003980 0x7f9bc0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:26.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.358+0000 7f9bd7fff640 1 -- 172.21.15.46:0/1114277293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc0003530 con 0x7f9bc0005190 2023-10-30T14:07:26.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bd7fff640 1 -- 172.21.15.46:0/1114277293 reap_dead start 2023-10-30T14:07:26.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bd7fff640 1 --2- 172.21.15.46:0/1114277293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bc0005190 0x7f9bc000b990 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009fc0 tx=0x7f9bd001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b18840d80a9c3e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00a5cb0 con 0x7f9bc0005190 2023-10-30T14:07:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd000f040 con 0x7f9bc0005190 2023-10-30T14:07:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bd67fc640 1 -- 172.21.15.46:0/1114277293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0003be0 con 0x7f9bc0005190 2023-10-30T14:07:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bdcda0640 1 -- 172.21.15.46:0/1114277293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bc0005190 msgr2=0x7f9bc000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bdcda0640 1 --2- 172.21.15.46:0/1114277293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bc0005190 0x7f9bc000b990 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009fc0 tx=0x7f9bd001cca0 comp rx=0 tx=0).stop 2023-10-30T14:07:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bdcda0640 1 -- 172.21.15.46:0/1114277293 shutdown_connections 2023-10-30T14:07:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bdcda0640 1 --2- 172.21.15.46:0/1114277293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bc0005190 0x7f9bc000b990 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bdcda0640 1 -- 172.21.15.46:0/1114277293 >> 172.21.15.46:0/1114277293 conn(0x7f9bd806a640 msgr2=0x7f9bd8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bdcda0640 1 -- 172.21.15.46:0/1114277293 shutdown_connections 2023-10-30T14:07:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bdcda0640 1 -- 172.21.15.46:0/1114277293 wait complete. 2023-10-30T14:07:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bdcda0640 1 Processor -- start 2023-10-30T14:07:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bdcda0640 1 -- start start 2023-10-30T14:07:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bdcda0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bc0005190 0x7f9bd815b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.362+0000 7f9bdcda0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd80c4a30 0x7f9bd80b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bdcda0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81f1aa0 0x7f9bd8145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bdcda0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80b3ce0 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bdcda0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd815b700 con 0x7f9bc0005190 2023-10-30T14:07:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bdcda0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd814fad0 con 0x7f9bd80c4a30 2023-10-30T14:07:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bd7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81f1aa0 0x7f9bd8145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bd7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81f1aa0 0x7f9bd8145710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36106/0 (socket says 172.21.15.46:36106) 2023-10-30T14:07:26.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bd7fff640 1 -- 172.21.15.46:0/2808511564 learned_addr learned my addr 172.21.15.46:0/2808511564 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:26.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bd6ffd640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd80c4a30 0x7f9bd80b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:26.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bd77fe640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bc0005190 0x7f9bd815b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:26.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bd7fff640 1 -- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd80c4a30 msgr2=0x7f9bd80b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:26.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bd7fff640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd80c4a30 0x7f9bd80b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:26.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bd7fff640 1 -- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bc0005190 msgr2=0x7f9bd815b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:26.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bd7fff640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bc0005190 0x7f9bd815b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:26.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bd7fff640 1 -- 172.21.15.46:0/2808511564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd814f510 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bd77fe640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bc0005190 0x7f9bd815b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:26.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bd7fff640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81f1aa0 0x7f9bd8145710 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00987d0 tx=0x7f9bd0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:26.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bbffff640 1 -- 172.21.15.46:0/2808511564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00aecb0 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bbffff640 1 -- 172.21.15.46:0/2808511564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd00aee30 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bbffff640 1 -- 172.21.15.46:0/2808511564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0003be0 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bdcda0640 1 -- 172.21.15.46:0/2808511564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd814f7f0 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bdcda0640 1 -- 172.21.15.46:0/2808511564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd815aca0 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bdcda0640 1 -- 172.21.15.46:0/2808511564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd804e170 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.366+0000 7f9bbffff640 1 -- 172.21.15.46:0/2808511564 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9bd0003d40 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.370+0000 7f9bbffff640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9bc40664a0 0x7f9bc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:26.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.370+0000 7f9bbffff640 1 -- 172.21.15.46:0/2808511564 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f9bd00a6e40 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.370+0000 7f9bd77fe640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9bc40664a0 0x7f9bc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:26.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.370+0000 7f9bd77fe640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9bc40664a0 0x7f9bc4068960 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8007d50 tx=0x7f9bc8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:26.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.374+0000 7f9bbffff640 1 -- 172.21.15.46:0/2808511564 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9bd00ec510 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.566+0000 7f9bdcda0640 1 -- 172.21.15.46:0/2808511564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bd80b2290 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.566+0000 7f9bbffff640 1 -- 172.21.15.46:0/2808511564 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f9bd00be020 con 0x7f9bd81f1aa0 2023-10-30T14:07:26.570 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:26.570 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:24.290316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:resolve","state_seq":20,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:26.571 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 190 2023-10-30T14:07:26.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.570+0000 7f9bdcda0640 1 -- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9bc40664a0 msgr2=0x7f9bc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:26.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.570+0000 7f9bdcda0640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9bc40664a0 0x7f9bc4068960 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8007d50 tx=0x7f9bc8009040 comp rx=0 tx=0).stop 2023-10-30T14:07:26.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.570+0000 7f9bdcda0640 1 -- 172.21.15.46:0/2808511564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81f1aa0 msgr2=0x7f9bd8145710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:26.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.570+0000 7f9bdcda0640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81f1aa0 0x7f9bd8145710 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00987d0 tx=0x7f9bd0009750 comp rx=0 tx=0).stop 2023-10-30T14:07:26.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.570+0000 7f9bdcda0640 1 -- 172.21.15.46:0/2808511564 shutdown_connections 2023-10-30T14:07:26.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.570+0000 7f9bdcda0640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9bc40664a0 0x7f9bc4068960 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:26.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.570+0000 7f9bdcda0640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd81f1aa0 0x7f9bd8145710 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:26.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.570+0000 7f9bdcda0640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd80c4a30 0x7f9bd80b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:26.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.570+0000 7f9bdcda0640 1 --2- 172.21.15.46:0/2808511564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bc0005190 0x7f9bd815b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:26.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.570+0000 7f9bdcda0640 1 -- 172.21.15.46:0/2808511564 >> 172.21.15.46:0/2808511564 conn(0x7f9bd806a640 msgr2=0x7f9bd80c3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:26.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.570+0000 7f9bdcda0640 1 -- 172.21.15.46:0/2808511564 shutdown_connections 2023-10-30T14:07:26.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:26.570+0000 7f9bdcda0640 1 -- 172.21.15.46:0/2808511564 wait complete. 2023-10-30T14:07:26.590 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:07:26.592 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:07:26.641 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:07:28.593 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:28.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f21a56d6640 1 Processor -- start 2023-10-30T14:07:28.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f21a56d6640 1 -- start start 2023-10-30T14:07:28.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f21a56d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f21a00c05d0 0x7f21a013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:28.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f21a56d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a00c0a50 con 0x7f21a00c6b80 2023-10-30T14:07:28.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f21a56d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a006f6e0 con 0x7f21a00c6f60 2023-10-30T14:07:28.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f21a56d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a006f9c0 con 0x7f21a00c05d0 2023-10-30T14:07:28.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f2197fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f21a00c05d0 0x7f21a013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:28.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f219effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f21a00c6b80 0x7f21a00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45078/0 (socket says 172.21.15.46:45078) 2023-10-30T14:07:28.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f219effd640 1 -- 172.21.15.46:0/3249946056 learned_addr learned my addr 172.21.15.46:0/3249946056 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:28.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f2197fff640 1 -- 172.21.15.46:0/3249946056 >> v1:172.21.15.154:6789/0 conn(0x7f21a00c6f60 legacy=0x7f21a01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:28.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f2197fff640 1 -- 172.21.15.46:0/3249946056 >> v1:172.21.15.46:6789/0 conn(0x7f21a00c6b80 legacy=0x7f21a00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:28.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f2197fff640 1 -- 172.21.15.46:0/3249946056 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a006fca0 con 0x7f21a00c05d0 2023-10-30T14:07:28.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.746+0000 7f219e7fc640 1 -- 172.21.15.46:0/3249946056 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093114475 0 0) 0x7f21a00c0a50 con 0x7f21a00c6b80 2023-10-30T14:07:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.750+0000 7f2197fff640 1 --2- 172.21.15.46:0/3249946056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00c05d0 0x7f21a013ed90 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f21880097c0 tx=0x7f218801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff634adfa2081cda server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:28.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.750+0000 7f219e7fc640 1 -- 172.21.15.46:0/3249946056 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21880a8050 con 0x7f21a00c05d0 2023-10-30T14:07:28.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.750+0000 7f219e7fc640 1 -- 172.21.15.46:0/3249946056 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21880ac5f0 con 0x7f21a00c05d0 2023-10-30T14:07:28.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.750+0000 7f219e7fc640 1 -- 172.21.15.46:0/3249946056 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21880ac980 con 0x7f21a00c05d0 2023-10-30T14:07:28.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.750+0000 7f21a56d6640 1 -- 172.21.15.46:0/3249946056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00c05d0 msgr2=0x7f21a013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:28.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.750+0000 7f21a56d6640 1 --2- 172.21.15.46:0/3249946056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00c05d0 0x7f21a013ed90 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f21880097c0 tx=0x7f218801c480 comp rx=0 tx=0).stop 2023-10-30T14:07:28.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.750+0000 7f21a56d6640 1 -- 172.21.15.46:0/3249946056 shutdown_connections 2023-10-30T14:07:28.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.750+0000 7f21a56d6640 1 --2- 172.21.15.46:0/3249946056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00c05d0 0x7f21a013ed90 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:28.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.750+0000 7f21a56d6640 1 -- 172.21.15.46:0/3249946056 >> 172.21.15.46:0/3249946056 conn(0x7f21a00c7910 msgr2=0x7f21a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:28.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.750+0000 7f21a56d6640 1 -- 172.21.15.46:0/3249946056 shutdown_connections 2023-10-30T14:07:28.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.750+0000 7f21a56d6640 1 -- 172.21.15.46:0/3249946056 wait complete. 2023-10-30T14:07:28.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21a56d6640 1 Processor -- start 2023-10-30T14:07:28.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21a56d6640 1 -- start start 2023-10-30T14:07:28.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21a56d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00c05d0 0x7f21a01f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:28.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21a56d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21a00c6b80 0x7f21a01ec1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:28.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21a56d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21a00c6f60 0x7f21a0138970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:28.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21a56d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a013df90 con 0x7f21a00c05d0 2023-10-30T14:07:28.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21a56d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00b0b30 con 0x7f21a00c6f60 2023-10-30T14:07:28.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21a56d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a0164780 con 0x7f21a00c6b80 2023-10-30T14:07:28.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f219effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00c05d0 0x7f21a01f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:28.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f219effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00c05d0 0x7f21a01f2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40630/0 (socket says 172.21.15.46:40630) 2023-10-30T14:07:28.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f219effd640 1 -- 172.21.15.46:0/3635890898 learned_addr learned my addr 172.21.15.46:0/3635890898 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:28.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f219f7fe640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21a00c6f60 0x7f21a0138970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:28.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f2197fff640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21a00c6b80 0x7f21a01ec1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:28.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f219effd640 1 -- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21a00c6b80 msgr2=0x7f21a01ec1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:28.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f219effd640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21a00c6b80 0x7f21a01ec1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:28.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f219effd640 1 -- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21a00c6f60 msgr2=0x7f21a0138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:28.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f219effd640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21a00c6f60 0x7f21a0138970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:28.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f219effd640 1 -- 172.21.15.46:0/3635890898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2188009420 con 0x7f21a00c05d0 2023-10-30T14:07:28.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f219effd640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00c05d0 0x7f21a01f2a40 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f219000c4d0 tx=0x7f219000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:28.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21977fe640 1 -- 172.21.15.46:0/3635890898 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21900074e0 con 0x7f21a00c05d0 2023-10-30T14:07:28.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21977fe640 1 -- 172.21.15.46:0/3635890898 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2190007640 con 0x7f21a00c05d0 2023-10-30T14:07:28.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21977fe640 1 -- 172.21.15.46:0/3635890898 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2190004a30 con 0x7f21a00c05d0 2023-10-30T14:07:28.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21a56d6640 1 -- 172.21.15.46:0/3635890898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a01eb740 con 0x7f21a00c05d0 2023-10-30T14:07:28.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.754+0000 7f21a56d6640 1 -- 172.21.15.46:0/3635890898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a01649d0 con 0x7f21a00c05d0 2023-10-30T14:07:28.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.758+0000 7f21a56d6640 1 -- 172.21.15.46:0/3635890898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a004e170 con 0x7f21a00c05d0 2023-10-30T14:07:28.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.758+0000 7f21977fe640 1 -- 172.21.15.46:0/3635890898 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2190004b90 con 0x7f21a00c05d0 2023-10-30T14:07:28.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.758+0000 7f21977fe640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21800664a0 0x7f2180068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:28.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.758+0000 7f21977fe640 1 -- 172.21.15.46:0/3635890898 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f2190083970 con 0x7f21a00c05d0 2023-10-30T14:07:28.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.758+0000 7f2197fff640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21800664a0 0x7f2180068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:28.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.762+0000 7f2197fff640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21800664a0 0x7f2180068960 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f21880ae000 tx=0x7f21880a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:28.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.762+0000 7f21977fe640 1 -- 172.21.15.46:0/3635890898 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2190050470 con 0x7f21a00c05d0 2023-10-30T14:07:28.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.954+0000 7f21a56d6640 1 -- 172.21.15.46:0/3635890898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21a00c1210 con 0x7f21a00c05d0 2023-10-30T14:07:28.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.954+0000 7f21977fe640 1 -- 172.21.15.46:0/3635890898 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f2190018070 con 0x7f21a00c05d0 2023-10-30T14:07:28.957 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:28.958 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:24.290316+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:resolve","state_seq":20,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:28.958 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 190 2023-10-30T14:07:28.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.954+0000 7f21a56d6640 1 -- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21800664a0 msgr2=0x7f2180068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:28.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.954+0000 7f21a56d6640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21800664a0 0x7f2180068960 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f21880ae000 tx=0x7f21880a7000 comp rx=0 tx=0).stop 2023-10-30T14:07:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.954+0000 7f21a56d6640 1 -- 172.21.15.46:0/3635890898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00c05d0 msgr2=0x7f21a01f2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.954+0000 7f21a56d6640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00c05d0 0x7f21a01f2a40 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f219000c4d0 tx=0x7f219000c990 comp rx=0 tx=0).stop 2023-10-30T14:07:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.954+0000 7f21a56d6640 1 -- 172.21.15.46:0/3635890898 shutdown_connections 2023-10-30T14:07:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.958+0000 7f21a56d6640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21800664a0 0x7f2180068960 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.958+0000 7f21a56d6640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21a00c6f60 0x7f21a0138970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:28.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.958+0000 7f21a56d6640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21a00c6b80 0x7f21a01ec1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:28.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.958+0000 7f21a56d6640 1 --2- 172.21.15.46:0/3635890898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00c05d0 0x7f21a01f2a40 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:28.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.958+0000 7f21a56d6640 1 -- 172.21.15.46:0/3635890898 >> 172.21.15.46:0/3635890898 conn(0x7f21a00c7910 msgr2=0x7f21a00c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:28.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.958+0000 7f21a56d6640 1 -- 172.21.15.46:0/3635890898 shutdown_connections 2023-10-30T14:07:28.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:28.958+0000 7f21a56d6640 1 -- 172.21.15.46:0/3635890898 wait complete. 2023-10-30T14:07:28.979 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 190, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 190, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8352": { "addr": "172.21.15.46:6836/128865315", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 128865315, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 128865315, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8352, "incarnation": 182, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 21 }, "gid_8505": { "addr": "172.21.15.46:6833/4078242540", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4078242540, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4078242540, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8505, "incarnation": 189, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:resolve", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 80, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:07:24.290316+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8505, "mds_1": 8352 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:07:30.981 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:31.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.122+0000 7f915bdf5640 1 Processor -- start 2023-10-30T14:07:31.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.122+0000 7f915bdf5640 1 -- start start 2023-10-30T14:07:31.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.122+0000 7f915bdf5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f91540c93d0 0x7f915406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:31.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.122+0000 7f915bdf5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915406afa0 con 0x7f915406e470 2023-10-30T14:07:31.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.122+0000 7f915bdf5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9154154150 con 0x7f9154153d70 2023-10-30T14:07:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.122+0000 7f915bdf5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915406d780 con 0x7f91540c93d0 2023-10-30T14:07:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.122+0000 7f9159390640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f915406e470 0x7f91540c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45080/0 (socket says 172.21.15.46:45080) 2023-10-30T14:07:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9159390640 1 -- 172.21.15.46:0/2746122173 learned_addr learned my addr 172.21.15.46:0/2746122173 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f915a392640 1 --2- 172.21.15.46:0/2746122173 >> v2:172.21.15.154:3300/0 conn(0x7f91540c93d0 0x7f915406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564468518 0 0) 0x7f915406afa0 con 0x7f915406e470 2023-10-30T14:07:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9140003530 con 0x7f915406e470 2023-10-30T14:07:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 744706998 0 0) 0x7f9140003530 con 0x7f915406e470 2023-10-30T14:07:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 >> v2:172.21.15.154:3300/0 conn(0x7f91540c93d0 msgr2=0x7f915406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 --2- 172.21.15.46:0/2746122173 >> v2:172.21.15.154:3300/0 conn(0x7f91540c93d0 0x7f915406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:31.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 >> v1:172.21.15.154:6789/0 conn(0x7f9154153d70 legacy=0x7f91540cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:31.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915406e030 con 0x7f915406e470 2023-10-30T14:07:31.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f9144001680 con 0x7f915406e470 2023-10-30T14:07:31.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f915a392640 1 --2- 172.21.15.46:0/2746122173 >> v2:172.21.15.154:3300/0 conn(0x7f91540c93d0 0x7f915406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:07:31.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 >> v1:172.21.15.46:6789/0 conn(0x7f915406e470 legacy=0x7f91540c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:31.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 --2- 172.21.15.46:0/2746122173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9140003980 0x7f9140003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:31.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 --2- 172.21.15.46:0/2746122173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91400047d0 0x7f9140004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:31.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 --2- 172.21.15.46:0/2746122173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9140005190 0x7f914000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:31.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915406d780 con 0x7f9140005190 2023-10-30T14:07:31.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914000c030 con 0x7f91400047d0 2023-10-30T14:07:31.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914000c1f0 con 0x7f9140003980 2023-10-30T14:07:31.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f915a392640 1 --2- 172.21.15.46:0/2746122173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9140005190 0x7f914000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:31.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9159b91640 1 --2- 172.21.15.46:0/2746122173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9140003980 0x7f9140003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:31.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f9159390640 1 --2- 172.21.15.46:0/2746122173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91400047d0 0x7f9140004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:31.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f915a392640 1 -- 172.21.15.46:0/2746122173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9140003980 msgr2=0x7f9140003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:31.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f915a392640 1 --2- 172.21.15.46:0/2746122173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9140003980 0x7f9140003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:31.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f915a392640 1 -- 172.21.15.46:0/2746122173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91400047d0 msgr2=0x7f9140004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:31.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f915a392640 1 --2- 172.21.15.46:0/2746122173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91400047d0 0x7f9140004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:31.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.126+0000 7f915a392640 1 -- 172.21.15.46:0/2746122173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9140003530 con 0x7f9140005190 2023-10-30T14:07:31.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f915a392640 1 -- 172.21.15.46:0/2746122173 reap_dead start 2023-10-30T14:07:31.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f915a392640 1 --2- 172.21.15.46:0/2746122173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9140005190 0x7f914000b990 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f914c00e350 tx=0x7f914c01ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ceebc7c478e07b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:31.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c0a5e70 con 0x7f9140005190 2023-10-30T14:07:31.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914c00f040 con 0x7f9140005190 2023-10-30T14:07:31.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f9158b8f640 1 -- 172.21.15.46:0/2746122173 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c003be0 con 0x7f9140005190 2023-10-30T14:07:31.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f915bdf5640 1 -- 172.21.15.46:0/2746122173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9140005190 msgr2=0x7f914000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:31.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f915bdf5640 1 --2- 172.21.15.46:0/2746122173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9140005190 0x7f914000b990 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f914c00e350 tx=0x7f914c01ce60 comp rx=0 tx=0).stop 2023-10-30T14:07:31.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f915bdf5640 1 -- 172.21.15.46:0/2746122173 shutdown_connections 2023-10-30T14:07:31.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f915bdf5640 1 --2- 172.21.15.46:0/2746122173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9140005190 0x7f914000b990 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:31.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f915bdf5640 1 -- 172.21.15.46:0/2746122173 >> 172.21.15.46:0/2746122173 conn(0x7f9154071ea0 msgr2=0x7f9154065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:31.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f915bdf5640 1 -- 172.21.15.46:0/2746122173 shutdown_connections 2023-10-30T14:07:31.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f915bdf5640 1 -- 172.21.15.46:0/2746122173 wait complete. 2023-10-30T14:07:31.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f915bdf5640 1 Processor -- start 2023-10-30T14:07:31.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.130+0000 7f915bdf5640 1 -- start start 2023-10-30T14:07:31.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915bdf5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9140005190 0x7f915415c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:31.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915bdf5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f915415b4c0 0x7f9154154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:31.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915bdf5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91541558d0 0x7f915415cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:31.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915bdf5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915414e0e0 con 0x7f91541558d0 2023-10-30T14:07:31.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915bdf5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915414de20 con 0x7f915415b4c0 2023-10-30T14:07:31.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915bdf5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9154156470 con 0x7f9140005190 2023-10-30T14:07:31.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915a392640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91541558d0 0x7f915415cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:31.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915a392640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91541558d0 0x7f915415cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40640/0 (socket says 172.21.15.46:40640) 2023-10-30T14:07:31.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915a392640 1 -- 172.21.15.46:0/3877022177 learned_addr learned my addr 172.21.15.46:0/3877022177 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:31.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f9159390640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f915415b4c0 0x7f9154154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:31.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f9159b91640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9140005190 0x7f915415c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:31.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915a392640 1 -- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9140005190 msgr2=0x7f915415c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:31.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915a392640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9140005190 0x7f915415c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:31.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915a392640 1 -- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f915415b4c0 msgr2=0x7f9154154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:31.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915a392640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f915415b4c0 0x7f9154154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:31.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915a392640 1 -- 172.21.15.46:0/3877022177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9154156750 con 0x7f91541558d0 2023-10-30T14:07:31.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f9159390640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f915415b4c0 0x7f9154154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:31.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915a392640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91541558d0 0x7f915415cca0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f914c01d700 tx=0x7f914c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:31.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f914a7fc640 1 -- 172.21.15.46:0/3877022177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c0bd070 con 0x7f91541558d0 2023-10-30T14:07:31.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f914a7fc640 1 -- 172.21.15.46:0/3877022177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914c0039a0 con 0x7f91541558d0 2023-10-30T14:07:31.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f914a7fc640 1 -- 172.21.15.46:0/3877022177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c003d50 con 0x7f91541558d0 2023-10-30T14:07:31.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915bdf5640 1 -- 172.21.15.46:0/3877022177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9154156a30 con 0x7f91541558d0 2023-10-30T14:07:31.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915bdf5640 1 -- 172.21.15.46:0/3877022177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9154154f50 con 0x7f91541558d0 2023-10-30T14:07:31.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f915bdf5640 1 -- 172.21.15.46:0/3877022177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915406da50 con 0x7f91541558d0 2023-10-30T14:07:31.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.134+0000 7f914a7fc640 1 -- 172.21.15.46:0/3877022177 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f914c004020 con 0x7f91541558d0 2023-10-30T14:07:31.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.138+0000 7f914a7fc640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f91380664d0 0x7f9138068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:31.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.138+0000 7f914a7fc640 1 -- 172.21.15.46:0/3877022177 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f914c11fb00 con 0x7f91541558d0 2023-10-30T14:07:31.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.138+0000 7f9159b91640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f91380664d0 0x7f9138068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:31.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.138+0000 7f9159b91640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f91380664d0 0x7f9138068990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f9150005490 tx=0x7f9150009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:31.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.142+0000 7f914a7fc640 1 -- 172.21.15.46:0/3877022177 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f915406da50 con 0x7f91541558d0 2023-10-30T14:07:31.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.330+0000 7f915bdf5640 1 -- 172.21.15.46:0/3877022177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f915406da50 con 0x7f91541558d0 2023-10-30T14:07:31.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f914a7fc640 1 -- 172.21.15.46:0/3877022177 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f915406da50 con 0x7f91541558d0 2023-10-30T14:07:31.336 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:31.337 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:29.943762+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:31.338 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 191 2023-10-30T14:07:31.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f915bdf5640 1 -- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f91380664d0 msgr2=0x7f9138068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:31.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f915bdf5640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f91380664d0 0x7f9138068990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f9150005490 tx=0x7f9150009040 comp rx=0 tx=0).stop 2023-10-30T14:07:31.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f915bdf5640 1 -- 172.21.15.46:0/3877022177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91541558d0 msgr2=0x7f915415cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:31.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f915bdf5640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91541558d0 0x7f915415cca0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f914c01d700 tx=0x7f914c009750 comp rx=0 tx=0).stop 2023-10-30T14:07:31.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f915bdf5640 1 -- 172.21.15.46:0/3877022177 shutdown_connections 2023-10-30T14:07:31.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f915bdf5640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f91380664d0 0x7f9138068990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:31.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f915bdf5640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91541558d0 0x7f915415cca0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:31.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f915bdf5640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f915415b4c0 0x7f9154154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:31.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f915bdf5640 1 --2- 172.21.15.46:0/3877022177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9140005190 0x7f915415c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:31.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f915bdf5640 1 -- 172.21.15.46:0/3877022177 >> 172.21.15.46:0/3877022177 conn(0x7f9154071ea0 msgr2=0x7f91541515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:31.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f915bdf5640 1 -- 172.21.15.46:0/3877022177 shutdown_connections 2023-10-30T14:07:31.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:31.334+0000 7f915bdf5640 1 -- 172.21.15.46:0/3877022177 wait complete. 2023-10-30T14:07:31.357 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:07:33.358 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:33.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11f53e640 1 Processor -- start 2023-10-30T14:07:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11f53e640 1 -- start start 2023-10-30T14:07:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11f53e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc11815d6f0 0x7fc11815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11f53e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1180cf160 0x7fc1180cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11f53e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1180cb800 con 0x7fc1181538a0 2023-10-30T14:07:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11f53e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1180cb330 con 0x7fc1180cf160 2023-10-30T14:07:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11f53e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1180cad70 con 0x7fc11815d6f0 2023-10-30T14:07:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11dadb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1180cf160 0x7fc1180cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11dadb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1180cf160 0x7fc1180cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40654/0 (socket says 172.21.15.46:40654) 2023-10-30T14:07:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11dadb640 1 -- 172.21.15.46:0/3164987674 learned_addr learned my addr 172.21.15.46:0/3164987674 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11cad9640 1 --2- 172.21.15.46:0/3164987674 >> v2:172.21.15.154:3300/0 conn(0x7fc11815d6f0 0x7fc11815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11dadb640 1 -- 172.21.15.46:0/3164987674 >> v2:172.21.15.154:3300/0 conn(0x7fc11815d6f0 msgr2=0x7fc11815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11dadb640 1 --2- 172.21.15.46:0/3164987674 >> v2:172.21.15.154:3300/0 conn(0x7fc11815d6f0 0x7fc11815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11dadb640 1 -- 172.21.15.46:0/3164987674 >> v1:172.21.15.154:6789/0 conn(0x7fc1181538a0 legacy=0x7fc1180ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.510+0000 7fc11dadb640 1 -- 172.21.15.46:0/3164987674 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1180cb050 con 0x7fc1180cf160 2023-10-30T14:07:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.514+0000 7fc11dadb640 1 --2- 172.21.15.46:0/3164987674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1180cf160 0x7fc1180cf590 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fc1180b30a0 tx=0x7fc11001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59aa7e9d2a14949a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.514+0000 7fc10ffff640 1 -- 172.21.15.46:0/3164987674 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1100af400 con 0x7fc1180cf160 2023-10-30T14:07:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.514+0000 7fc10ffff640 1 -- 172.21.15.46:0/3164987674 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1100ab050 con 0x7fc1180cf160 2023-10-30T14:07:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.514+0000 7fc10ffff640 1 -- 172.21.15.46:0/3164987674 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1100af780 con 0x7fc1180cf160 2023-10-30T14:07:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.514+0000 7fc11f53e640 1 -- 172.21.15.46:0/3164987674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1180cf160 msgr2=0x7fc1180cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.514+0000 7fc11f53e640 1 --2- 172.21.15.46:0/3164987674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1180cf160 0x7fc1180cf590 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fc1180b30a0 tx=0x7fc11001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:07:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.514+0000 7fc11f53e640 1 -- 172.21.15.46:0/3164987674 shutdown_connections 2023-10-30T14:07:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.514+0000 7fc11f53e640 1 --2- 172.21.15.46:0/3164987674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1180cf160 0x7fc1180cf590 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.514+0000 7fc11f53e640 1 --2- 172.21.15.46:0/3164987674 >> v2:172.21.15.154:3300/0 conn(0x7fc11815d6f0 0x7fc11815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.514+0000 7fc11f53e640 1 -- 172.21.15.46:0/3164987674 >> 172.21.15.46:0/3164987674 conn(0x7fc11806bfe0 msgr2=0x7fc11806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.514+0000 7fc11f53e640 1 -- 172.21.15.46:0/3164987674 shutdown_connections 2023-10-30T14:07:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.514+0000 7fc11f53e640 1 -- 172.21.15.46:0/3164987674 wait complete. 2023-10-30T14:07:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11f53e640 1 Processor -- start 2023-10-30T14:07:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11f53e640 1 -- start start 2023-10-30T14:07:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11f53e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1180cf160 0x7fc11814db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11f53e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1181538a0 0x7fc1180b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11f53e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc11815d6f0 0x7fc11814cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11f53e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1181e4ae0 con 0x7fc1180cf160 2023-10-30T14:07:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11f53e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1181e4dc0 con 0x7fc1181538a0 2023-10-30T14:07:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11f53e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1181f0320 con 0x7fc11815d6f0 2023-10-30T14:07:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11d2da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1180cf160 0x7fc11814db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11d2da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1180cf160 0x7fc11814db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40664/0 (socket says 172.21.15.46:40664) 2023-10-30T14:07:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11d2da640 1 -- 172.21.15.46:0/142579250 learned_addr learned my addr 172.21.15.46:0/142579250 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11cad9640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1181538a0 0x7fc1180b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11dadb640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc11815d6f0 0x7fc11814cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11d2da640 1 -- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc11815d6f0 msgr2=0x7fc11814cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11d2da640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc11815d6f0 0x7fc11814cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11d2da640 1 -- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1181538a0 msgr2=0x7fc1180b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11d2da640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1181538a0 0x7fc1180b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11d2da640 1 -- 172.21.15.46:0/142579250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc110009b30 con 0x7fc1180cf160 2023-10-30T14:07:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11dadb640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc11815d6f0 0x7fc11814cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11cad9640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1181538a0 0x7fc1180b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11d2da640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1180cf160 0x7fc11814db50 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fc11400c500 tx=0x7fc11400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc10dffb640 1 -- 172.21.15.46:0/142579250 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc114005220 con 0x7fc1180cf160 2023-10-30T14:07:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc10dffb640 1 -- 172.21.15.46:0/142579250 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc114005380 con 0x7fc1180cf160 2023-10-30T14:07:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc10dffb640 1 -- 172.21.15.46:0/142579250 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc114007270 con 0x7fc1180cf160 2023-10-30T14:07:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11f53e640 1 -- 172.21.15.46:0/142579250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1181f0600 con 0x7fc1180cf160 2023-10-30T14:07:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.518+0000 7fc11f53e640 1 -- 172.21.15.46:0/142579250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1181f0bc0 con 0x7fc1180cf160 2023-10-30T14:07:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.522+0000 7fc11f53e640 1 -- 172.21.15.46:0/142579250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1180b2a50 con 0x7fc1180cf160 2023-10-30T14:07:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.522+0000 7fc10dffb640 1 -- 172.21.15.46:0/142579250 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc1140073d0 con 0x7fc1180cf160 2023-10-30T14:07:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.522+0000 7fc10dffb640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0fc0664d0 0x7fc0fc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.522+0000 7fc10dffb640 1 -- 172.21.15.46:0/142579250 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7fc114083fe0 con 0x7fc1180cf160 2023-10-30T14:07:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.522+0000 7fc11cad9640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0fc0664d0 0x7fc0fc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.526+0000 7fc11cad9640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0fc0664d0 0x7fc0fc068990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc118151280 tx=0x7fc108008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.526+0000 7fc10dffb640 1 -- 172.21.15.46:0/142579250 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc114050ba0 con 0x7fc1180cf160 2023-10-30T14:07:33.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.714+0000 7fc11f53e640 1 -- 172.21.15.46:0/142579250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc11804e170 con 0x7fc1180cf160 2023-10-30T14:07:33.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.714+0000 7fc10dffb640 1 -- 172.21.15.46:0/142579250 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fc1140541c0 con 0x7fc1180cf160 2023-10-30T14:07:33.720 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:33.720 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:29.943762+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:33.722 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 191 2023-10-30T14:07:33.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.718+0000 7fc11f53e640 1 -- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0fc0664d0 msgr2=0x7fc0fc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:33.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.718+0000 7fc11f53e640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0fc0664d0 0x7fc0fc068990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc118151280 tx=0x7fc108008040 comp rx=0 tx=0).stop 2023-10-30T14:07:33.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.718+0000 7fc11f53e640 1 -- 172.21.15.46:0/142579250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1180cf160 msgr2=0x7fc11814db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:33.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.718+0000 7fc11f53e640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1180cf160 0x7fc11814db50 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fc11400c500 tx=0x7fc11400c9c0 comp rx=0 tx=0).stop 2023-10-30T14:07:33.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.718+0000 7fc11f53e640 1 -- 172.21.15.46:0/142579250 shutdown_connections 2023-10-30T14:07:33.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.718+0000 7fc11f53e640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0fc0664d0 0x7fc0fc068990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:33.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.718+0000 7fc11f53e640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc11815d6f0 0x7fc11814cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:33.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.718+0000 7fc11f53e640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1181538a0 0x7fc1180b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:33.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.718+0000 7fc11f53e640 1 --2- 172.21.15.46:0/142579250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1180cf160 0x7fc11814db50 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:33.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.718+0000 7fc11f53e640 1 -- 172.21.15.46:0/142579250 >> 172.21.15.46:0/142579250 conn(0x7fc11806bfe0 msgr2=0x7fc1180c48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:33.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.718+0000 7fc11f53e640 1 -- 172.21.15.46:0/142579250 shutdown_connections 2023-10-30T14:07:33.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:33.718+0000 7fc11f53e640 1 -- 172.21.15.46:0/142579250 wait complete. 2023-10-30T14:07:33.741 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:07:35.742 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:35.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f7e90f640 1 Processor -- start 2023-10-30T14:07:35.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f7e90f640 1 -- start start 2023-10-30T14:07:35.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f7e90f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4f781538a0 0x7f4f7815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:35.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f7e90f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f780ca0e0 0x7f4f7815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:35.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f7e90f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f780cb800 con 0x7f4f780cf160 2023-10-30T14:07:35.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f7e90f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7815dc30 con 0x7f4f781538a0 2023-10-30T14:07:35.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f7e90f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f780cb330 con 0x7f4f780ca0e0 2023-10-30T14:07:35.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f77fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4f781538a0 0x7f4f7815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:35.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f777fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f780ca0e0 0x7f4f7815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:35.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f7ceac640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f4f780cf160 0x7f4f780cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:43806/0 (socket says 172.21.15.46:43806) 2023-10-30T14:07:35.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f7ceac640 1 -- 172.21.15.46:0/2594627777 learned_addr learned my addr 172.21.15.46:0/2594627777 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:35.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f77fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4f781538a0 0x7f4f7815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:56214/0 (socket says 172.21.15.46:56214) 2023-10-30T14:07:35.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f77fff640 1 -- 172.21.15.46:0/2594627777 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f780ca0e0 msgr2=0x7f4f7815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:35.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f77fff640 1 --2- 172.21.15.46:0/2594627777 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f780ca0e0 0x7f4f7815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:35.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f77fff640 1 -- 172.21.15.46:0/2594627777 >> v1:172.21.15.154:6789/0 conn(0x7f4f780cf160 legacy=0x7f4f780cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:35.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.898+0000 7f4f77fff640 1 -- 172.21.15.46:0/2594627777 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f780cad70 con 0x7f4f781538a0 2023-10-30T14:07:35.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.902+0000 7f4f77fff640 1 --2- 172.21.15.46:0/2594627777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f781538a0 0x7f4f7815d6f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4f680097c0 tx=0x7f4f6801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfaa2dffbcbec1ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:35.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.902+0000 7f4f76ffd640 1 -- 172.21.15.46:0/2594627777 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f680a8970 con 0x7f4f781538a0 2023-10-30T14:07:35.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.902+0000 7f4f76ffd640 1 -- 172.21.15.46:0/2594627777 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f680a8ad0 con 0x7f4f781538a0 2023-10-30T14:07:35.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.902+0000 7f4f76ffd640 1 -- 172.21.15.46:0/2594627777 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f680a8e60 con 0x7f4f781538a0 2023-10-30T14:07:35.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.902+0000 7f4f7e90f640 1 -- 172.21.15.46:0/2594627777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f781538a0 msgr2=0x7f4f7815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:35.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.902+0000 7f4f7e90f640 1 --2- 172.21.15.46:0/2594627777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f781538a0 0x7f4f7815d6f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4f680097c0 tx=0x7f4f6801ec70 comp rx=0 tx=0).stop 2023-10-30T14:07:35.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.902+0000 7f4f7e90f640 1 -- 172.21.15.46:0/2594627777 shutdown_connections 2023-10-30T14:07:35.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.902+0000 7f4f7e90f640 1 --2- 172.21.15.46:0/2594627777 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f780ca0e0 0x7f4f7815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:35.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.902+0000 7f4f7e90f640 1 --2- 172.21.15.46:0/2594627777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f781538a0 0x7f4f7815d6f0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:35.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.902+0000 7f4f7e90f640 1 -- 172.21.15.46:0/2594627777 >> 172.21.15.46:0/2594627777 conn(0x7f4f7806bfe0 msgr2=0x7f4f7806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:35.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.902+0000 7f4f7e90f640 1 -- 172.21.15.46:0/2594627777 shutdown_connections 2023-10-30T14:07:35.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.902+0000 7f4f7e90f640 1 -- 172.21.15.46:0/2594627777 wait complete. 2023-10-30T14:07:35.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f7e90f640 1 Processor -- start 2023-10-30T14:07:35.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f7e90f640 1 -- start start 2023-10-30T14:07:35.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f7e90f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f780ca0e0 0x7f4f780ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:35.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f7e90f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f780cf160 0x7f4f78158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:35.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f7e90f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f781538a0 0x7f4f7814cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:35.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f7e90f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f781ef540 con 0x7f4f780cf160 2023-10-30T14:07:35.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f7e90f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f781f0510 con 0x7f4f781538a0 2023-10-30T14:07:35.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f7e90f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f780666b0 con 0x7f4f780ca0e0 2023-10-30T14:07:35.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f780cf160 0x7f4f78158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:35.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f780cf160 0x7f4f78158250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40670/0 (socket says 172.21.15.46:40670) 2023-10-30T14:07:35.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f777fe640 1 -- 172.21.15.46:0/320945419 learned_addr learned my addr 172.21.15.46:0/320945419 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:35.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f77fff640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f780ca0e0 0x7f4f780ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:35.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f7ceac640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f781538a0 0x7f4f7814cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:35.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f777fe640 1 -- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f780ca0e0 msgr2=0x7f4f780ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:35.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f777fe640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f780ca0e0 0x7f4f780ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:35.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f777fe640 1 -- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f781538a0 msgr2=0x7f4f7814cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:35.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f777fe640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f781538a0 0x7f4f7814cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:35.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f777fe640 1 -- 172.21.15.46:0/320945419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f68009420 con 0x7f4f780cf160 2023-10-30T14:07:35.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f77fff640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f780ca0e0 0x7f4f780ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:35.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f777fe640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f780cf160 0x7f4f78158250 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c00e930 tx=0x7f4f6c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:35.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f74ff9640 1 -- 172.21.15.46:0/320945419 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c009690 con 0x7f4f780cf160 2023-10-30T14:07:35.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f74ff9640 1 -- 172.21.15.46:0/320945419 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6c00ca00 con 0x7f4f780cf160 2023-10-30T14:07:35.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f7e90f640 1 -- 172.21.15.46:0/320945419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f781ef750 con 0x7f4f780cf160 2023-10-30T14:07:35.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f74ff9640 1 -- 172.21.15.46:0/320945419 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c00cd70 con 0x7f4f780cf160 2023-10-30T14:07:35.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.906+0000 7f4f7e90f640 1 -- 172.21.15.46:0/320945419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f781efd10 con 0x7f4f780cf160 2023-10-30T14:07:35.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.910+0000 7f4f7e90f640 1 -- 172.21.15.46:0/320945419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f780b2a50 con 0x7f4f780cf160 2023-10-30T14:07:35.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.910+0000 7f4f74ff9640 1 -- 172.21.15.46:0/320945419 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4f6c013470 con 0x7f4f780cf160 2023-10-30T14:07:35.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.910+0000 7f4f74ff9640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f640664a0 0x7f4f64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:35.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.910+0000 7f4f74ff9640 1 -- 172.21.15.46:0/320945419 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f4f6c0849c0 con 0x7f4f780cf160 2023-10-30T14:07:35.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.910+0000 7f4f77fff640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f640664a0 0x7f4f64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:35.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.910+0000 7f4f77fff640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f640664a0 0x7f4f64068960 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f4f680b2000 tx=0x7f4f68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:35.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:35.914+0000 7f4f74ff9640 1 -- 172.21.15.46:0/320945419 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4f6c051610 con 0x7f4f780cf160 2023-10-30T14:07:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f7e90f640 1 -- 172.21.15.46:0/320945419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f7804e170 con 0x7f4f780cf160 2023-10-30T14:07:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f74ff9640 1 -- 172.21.15.46:0/320945419 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f4f6c021090 con 0x7f4f780cf160 2023-10-30T14:07:36.109 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:36.110 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:35.889864+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 192 2023-10-30T14:07:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f7e90f640 1 -- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f640664a0 msgr2=0x7f4f64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f7e90f640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f640664a0 0x7f4f64068960 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f4f680b2000 tx=0x7f4f68008040 comp rx=0 tx=0).stop 2023-10-30T14:07:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f7e90f640 1 -- 172.21.15.46:0/320945419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f780cf160 msgr2=0x7f4f78158250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f7e90f640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f780cf160 0x7f4f78158250 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c00e930 tx=0x7f4f6c00edf0 comp rx=0 tx=0).stop 2023-10-30T14:07:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f7e90f640 1 -- 172.21.15.46:0/320945419 shutdown_connections 2023-10-30T14:07:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f7e90f640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f640664a0 0x7f4f64068960 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f7e90f640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f781538a0 0x7f4f7814cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:36.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f7e90f640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f780cf160 0x7f4f78158250 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:36.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f7e90f640 1 --2- 172.21.15.46:0/320945419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f780ca0e0 0x7f4f780ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:36.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f7e90f640 1 -- 172.21.15.46:0/320945419 >> 172.21.15.46:0/320945419 conn(0x7f4f7806bfe0 msgr2=0x7f4f7814db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:36.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.106+0000 7f4f7e90f640 1 -- 172.21.15.46:0/320945419 shutdown_connections 2023-10-30T14:07:36.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:36.110+0000 7f4f7e90f640 1 -- 172.21.15.46:0/320945419 wait complete. 2023-10-30T14:07:36.131 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:07:38.132 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:38.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.282+0000 7ff0dc572640 1 Processor -- start 2023-10-30T14:07:38.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.282+0000 7ff0dc572640 1 -- start start 2023-10-30T14:07:38.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.286+0000 7ff0dc572640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0d40c6f60 0x7ff0d41566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:38.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.286+0000 7ff0dc572640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d40c0a50 con 0x7ff0d40c05d0 2023-10-30T14:07:38.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.286+0000 7ff0dc572640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d406f6e0 con 0x7ff0d40c6b80 2023-10-30T14:07:38.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.286+0000 7ff0dc572640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d406f9c0 con 0x7ff0d40c6f60 2023-10-30T14:07:38.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.286+0000 7ff0d9b0d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff0d40c05d0 0x7ff0d413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55666/0 (socket says 172.21.15.46:55666) 2023-10-30T14:07:38.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.286+0000 7ff0d9b0d640 1 -- 172.21.15.46:0/158841784 learned_addr learned my addr 172.21.15.46:0/158841784 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:38.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.286+0000 7ff0dab0f640 1 --2- 172.21.15.46:0/158841784 >> v2:172.21.15.46:3300/0 conn(0x7ff0d40c6f60 0x7ff0d41566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:38.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.286+0000 7ff0dab0f640 1 -- 172.21.15.46:0/158841784 >> v1:172.21.15.154:6789/0 conn(0x7ff0d40c6b80 legacy=0x7ff0d40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:38.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.286+0000 7ff0dab0f640 1 -- 172.21.15.46:0/158841784 >> v1:172.21.15.46:6789/0 conn(0x7ff0d40c05d0 legacy=0x7ff0d413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:38.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.286+0000 7ff0dab0f640 1 -- 172.21.15.46:0/158841784 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d406fca0 con 0x7ff0d40c6f60 2023-10-30T14:07:38.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dab0f640 1 --2- 172.21.15.46:0/158841784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0d40c6f60 0x7ff0d41566a0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009d90 tx=0x7ff0cc01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24dde87cb61d929a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:38.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0d930c640 1 -- 172.21.15.46:0/158841784 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc0a5820 con 0x7ff0d40c6f60 2023-10-30T14:07:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0d930c640 1 -- 172.21.15.46:0/158841784 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0cc0a5980 con 0x7ff0d40c6f60 2023-10-30T14:07:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0d930c640 1 -- 172.21.15.46:0/158841784 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc0a5d10 con 0x7ff0d40c6f60 2023-10-30T14:07:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 -- 172.21.15.46:0/158841784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0d40c6f60 msgr2=0x7ff0d41566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 --2- 172.21.15.46:0/158841784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0d40c6f60 0x7ff0d41566a0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009d90 tx=0x7ff0cc01c950 comp rx=0 tx=0).stop 2023-10-30T14:07:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 -- 172.21.15.46:0/158841784 shutdown_connections 2023-10-30T14:07:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 --2- 172.21.15.46:0/158841784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0d40c6f60 0x7ff0d41566a0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 -- 172.21.15.46:0/158841784 >> 172.21.15.46:0/158841784 conn(0x7ff0d40c7910 msgr2=0x7ff0d40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:38.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 -- 172.21.15.46:0/158841784 shutdown_connections 2023-10-30T14:07:38.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 -- 172.21.15.46:0/158841784 wait complete. 2023-10-30T14:07:38.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 Processor -- start 2023-10-30T14:07:38.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 -- start start 2023-10-30T14:07:38.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0d40c05d0 0x7ff0d413e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:38.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0d40c6b80 0x7ff0d413a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:38.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0d40c6f60 0x7ff0d414fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:38.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d413dd60 con 0x7ff0d40c05d0 2023-10-30T14:07:38.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d4155ee0 con 0x7ff0d40c6b80 2023-10-30T14:07:38.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0dc572640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d41561c0 con 0x7ff0d40c6f60 2023-10-30T14:07:38.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0da30e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0d40c05d0 0x7ff0d413e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:38.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0da30e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0d40c05d0 0x7ff0d413e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41940/0 (socket says 172.21.15.46:41940) 2023-10-30T14:07:38.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.290+0000 7ff0da30e640 1 -- 172.21.15.46:0/1763729722 learned_addr learned my addr 172.21.15.46:0/1763729722 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:38.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0dab0f640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0d40c6f60 0x7ff0d414fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:38.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0d9b0d640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0d40c6b80 0x7ff0d413a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:38.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0da30e640 1 -- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0d40c6f60 msgr2=0x7ff0d414fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:38.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0da30e640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0d40c6f60 0x7ff0d414fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:38.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0da30e640 1 -- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0d40c6b80 msgr2=0x7ff0d413a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:38.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0da30e640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0d40c6b80 0x7ff0d413a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:38.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0da30e640 1 -- 172.21.15.46:0/1763729722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0cc009a70 con 0x7ff0d40c05d0 2023-10-30T14:07:38.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0d9b0d640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0d40c6b80 0x7ff0d413a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:07:38.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0da30e640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0d40c05d0 0x7ff0d413e1f0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000c500 tx=0x7ff0d000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:38.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0caffd640 1 -- 172.21.15.46:0/1763729722 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0005220 con 0x7ff0d40c05d0 2023-10-30T14:07:38.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0caffd640 1 -- 172.21.15.46:0/1763729722 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d0005380 con 0x7ff0d40c05d0 2023-10-30T14:07:38.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0caffd640 1 -- 172.21.15.46:0/1763729722 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0007270 con 0x7ff0d40c05d0 2023-10-30T14:07:38.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0dc572640 1 -- 172.21.15.46:0/1763729722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d41564a0 con 0x7ff0d40c05d0 2023-10-30T14:07:38.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0dc572640 1 -- 172.21.15.46:0/1763729722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d41358f0 con 0x7ff0d40c05d0 2023-10-30T14:07:38.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0dc572640 1 -- 172.21.15.46:0/1763729722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d404e170 con 0x7ff0d40c05d0 2023-10-30T14:07:38.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0caffd640 1 -- 172.21.15.46:0/1763729722 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff0d00073d0 con 0x7ff0d40c05d0 2023-10-30T14:07:38.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0caffd640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0b80664a0 0x7ff0b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:38.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.294+0000 7ff0caffd640 1 -- 172.21.15.46:0/1763729722 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7ff0d0084030 con 0x7ff0d40c05d0 2023-10-30T14:07:38.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.298+0000 7ff0d9b0d640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0b80664a0 0x7ff0b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:38.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.298+0000 7ff0d9b0d640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0b80664a0 0x7ff0b8068960 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff0d413db90 tx=0x7ff0c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:38.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.298+0000 7ff0caffd640 1 -- 172.21.15.46:0/1763729722 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff0d0050c70 con 0x7ff0d40c05d0 2023-10-30T14:07:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.486+0000 7ff0dc572640 1 -- 172.21.15.46:0/1763729722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0d4161d40 con 0x7ff0d40c05d0 2023-10-30T14:07:38.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.486+0000 7ff0caffd640 1 -- 172.21.15.46:0/1763729722 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+5730 (secure 0 0 0) 0x7ff0d0054290 con 0x7ff0d40c05d0 2023-10-30T14:07:38.491 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:38.492 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:35.889864+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 192 2023-10-30T14:07:38.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.490+0000 7ff0dc572640 1 -- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0b80664a0 msgr2=0x7ff0b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:38.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.490+0000 7ff0dc572640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0b80664a0 0x7ff0b8068960 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff0d413db90 tx=0x7ff0c4009040 comp rx=0 tx=0).stop 2023-10-30T14:07:38.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.490+0000 7ff0dc572640 1 -- 172.21.15.46:0/1763729722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0d40c05d0 msgr2=0x7ff0d413e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.490+0000 7ff0dc572640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0d40c05d0 0x7ff0d413e1f0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000c500 tx=0x7ff0d000c9c0 comp rx=0 tx=0).stop 2023-10-30T14:07:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.490+0000 7ff0dc572640 1 -- 172.21.15.46:0/1763729722 shutdown_connections 2023-10-30T14:07:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.490+0000 7ff0dc572640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff0b80664a0 0x7ff0b8068960 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.490+0000 7ff0dc572640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0d40c6f60 0x7ff0d414fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.490+0000 7ff0dc572640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0d40c6b80 0x7ff0d413a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.490+0000 7ff0dc572640 1 --2- 172.21.15.46:0/1763729722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0d40c05d0 0x7ff0d413e1f0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.490+0000 7ff0dc572640 1 -- 172.21.15.46:0/1763729722 >> 172.21.15.46:0/1763729722 conn(0x7ff0d40c7910 msgr2=0x7ff0d4138bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.490+0000 7ff0dc572640 1 -- 172.21.15.46:0/1763729722 shutdown_connections 2023-10-30T14:07:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:38.490+0000 7ff0dc572640 1 -- 172.21.15.46:0/1763729722 wait complete. 2023-10-30T14:07:38.512 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:07:40.513 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:40.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.666+0000 7f672153e640 1 Processor -- start 2023-10-30T14:07:40.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.666+0000 7f672153e640 1 -- start start 2023-10-30T14:07:40.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f672153e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f671c159e60 0x7f671c06f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:40.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f672153e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f671c06a720 0x7f671c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:40.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f672153e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c06fa80 con 0x7f671c0bfec0 2023-10-30T14:07:40.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f672153e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671c071290 con 0x7f671c159e60 2023-10-30T14:07:40.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f672153e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671c071570 con 0x7f671c06a720 2023-10-30T14:07:40.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f6713fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f671c0bfec0 0x7f671c0c1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55674/0 (socket says 172.21.15.46:55674) 2023-10-30T14:07:40.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f6713fff640 1 -- 172.21.15.46:0/4076060510 learned_addr learned my addr 172.21.15.46:0/4076060510 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:40.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f671b7fe640 1 --2- 172.21.15.46:0/4076060510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f671c06a720 0x7f671c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:40.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f671affd640 1 --2- 172.21.15.46:0/4076060510 >> v2:172.21.15.154:3300/0 conn(0x7f671c159e60 0x7f671c06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:40.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f671a7fc640 1 -- 172.21.15.46:0/4076060510 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694498063 0 0) 0x7f671c06fa80 con 0x7f671c0bfec0 2023-10-30T14:07:40.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f671a7fc640 1 -- 172.21.15.46:0/4076060510 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6708003530 con 0x7f671c0bfec0 2023-10-30T14:07:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f671b7fe640 1 -- 172.21.15.46:0/4076060510 >> v2:172.21.15.154:3300/0 conn(0x7f671c159e60 msgr2=0x7f671c06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f671b7fe640 1 --2- 172.21.15.46:0/4076060510 >> v2:172.21.15.154:3300/0 conn(0x7f671c159e60 0x7f671c06f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f671b7fe640 1 -- 172.21.15.46:0/4076060510 >> v1:172.21.15.46:6789/0 conn(0x7f671c0bfec0 legacy=0x7f671c0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.670+0000 7f671b7fe640 1 -- 172.21.15.46:0/4076060510 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c071850 con 0x7f671c06a720 2023-10-30T14:07:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f671b7fe640 1 --2- 172.21.15.46:0/4076060510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f671c06a720 0x7f671c06abb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6714007c80 tx=0x7f671401f410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab72f6f2639c36a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f671a7fc640 1 -- 172.21.15.46:0/4076060510 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671400a2d0 con 0x7f671c06a720 2023-10-30T14:07:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f671a7fc640 1 -- 172.21.15.46:0/4076060510 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67140ab050 con 0x7f671c06a720 2023-10-30T14:07:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f671a7fc640 1 -- 172.21.15.46:0/4076060510 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6714008680 con 0x7f671c06a720 2023-10-30T14:07:40.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f672153e640 1 -- 172.21.15.46:0/4076060510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f671c06a720 msgr2=0x7f671c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:40.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f672153e640 1 --2- 172.21.15.46:0/4076060510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f671c06a720 0x7f671c06abb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6714007c80 tx=0x7f671401f410 comp rx=0 tx=0).stop 2023-10-30T14:07:40.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f672153e640 1 -- 172.21.15.46:0/4076060510 shutdown_connections 2023-10-30T14:07:40.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f672153e640 1 --2- 172.21.15.46:0/4076060510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f671c06a720 0x7f671c06abb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:40.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f672153e640 1 --2- 172.21.15.46:0/4076060510 >> v2:172.21.15.154:3300/0 conn(0x7f671c159e60 0x7f671c06f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:40.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f672153e640 1 -- 172.21.15.46:0/4076060510 >> 172.21.15.46:0/4076060510 conn(0x7f671c0c7910 msgr2=0x7f671c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:40.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f672153e640 1 -- 172.21.15.46:0/4076060510 shutdown_connections 2023-10-30T14:07:40.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f672153e640 1 -- 172.21.15.46:0/4076060510 wait complete. 2023-10-30T14:07:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f672153e640 1 Processor -- start 2023-10-30T14:07:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.674+0000 7f672153e640 1 -- start start 2023-10-30T14:07:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f672153e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f671c06a720 0x7f671c0c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:40.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f672153e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f671c0bfec0 0x7f671c0c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:40.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f672153e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f671c159e60 0x7f671c1ef370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:40.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f672153e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c0c4080 con 0x7f671c0bfec0 2023-10-30T14:07:40.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f672153e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c0c21e0 con 0x7f671c159e60 2023-10-30T14:07:40.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f672153e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671c1eebd0 con 0x7f671c06a720 2023-10-30T14:07:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f6713fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f671c0bfec0 0x7f671c0c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f6713fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f671c0bfec0 0x7f671c0c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41942/0 (socket says 172.21.15.46:41942) 2023-10-30T14:07:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f6713fff640 1 -- 172.21.15.46:0/3603875898 learned_addr learned my addr 172.21.15.46:0/3603875898 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f671affd640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f671c06a720 0x7f671c0c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f671b7fe640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f671c159e60 0x7f671c1ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f6713fff640 1 -- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f671c06a720 msgr2=0x7f671c0c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:40.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f6713fff640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f671c06a720 0x7f671c0c4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:40.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f6713fff640 1 -- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f671c159e60 msgr2=0x7f671c1ef370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:40.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f6713fff640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f671c159e60 0x7f671c1ef370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:40.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f6713fff640 1 -- 172.21.15.46:0/3603875898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67140078e0 con 0x7f671c0bfec0 2023-10-30T14:07:40.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f671b7fe640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f671c159e60 0x7f671c1ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f6713fff640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f671c0bfec0 0x7f671c0c8e70 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f670400c500 tx=0x7f670400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f67137fe640 1 -- 172.21.15.46:0/3603875898 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704005cf0 con 0x7f671c0bfec0 2023-10-30T14:07:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f67137fe640 1 -- 172.21.15.46:0/3603875898 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6704005e50 con 0x7f671c0bfec0 2023-10-30T14:07:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f67137fe640 1 -- 172.21.15.46:0/3603875898 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67040072f0 con 0x7f671c0bfec0 2023-10-30T14:07:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f672153e640 1 -- 172.21.15.46:0/3603875898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c0c9c60 con 0x7f671c0bfec0 2023-10-30T14:07:40.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f672153e640 1 -- 172.21.15.46:0/3603875898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c14bd80 con 0x7f671c0bfec0 2023-10-30T14:07:40.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f672153e640 1 -- 172.21.15.46:0/3603875898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671c04e170 con 0x7f671c0bfec0 2023-10-30T14:07:40.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.678+0000 7f67137fe640 1 -- 172.21.15.46:0/3603875898 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6704007450 con 0x7f671c0bfec0 2023-10-30T14:07:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.682+0000 7f67137fe640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66fc0664a0 0x7f66fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.682+0000 7f67137fe640 1 -- 172.21.15.46:0/3603875898 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f6704083c60 con 0x7f671c0bfec0 2023-10-30T14:07:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.682+0000 7f671affd640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66fc0664a0 0x7f66fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.682+0000 7f671affd640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66fc0664a0 0x7f66fc068960 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f670c005c10 tx=0x7f670c00b750 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.686+0000 7f67137fe640 1 -- 172.21.15.46:0/3603875898 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6704050930 con 0x7f671c0bfec0 2023-10-30T14:07:40.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 -- 172.21.15.46:0/3603875898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f671c0c8100 con 0x7f671c0bfec0 2023-10-30T14:07:40.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f67137fe640 1 -- 172.21.15.46:0/3603875898 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f6704017070 con 0x7f671c0bfec0 2023-10-30T14:07:40.877 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:40.877 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:35.889864+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 192 2023-10-30T14:07:40.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 -- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66fc0664a0 msgr2=0x7f66fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:40.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66fc0664a0 0x7f66fc068960 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f670c005c10 tx=0x7f670c00b750 comp rx=0 tx=0).stop 2023-10-30T14:07:40.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 -- 172.21.15.46:0/3603875898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f671c0bfec0 msgr2=0x7f671c0c8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:40.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f671c0bfec0 0x7f671c0c8e70 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f670400c500 tx=0x7f670400c9c0 comp rx=0 tx=0).stop 2023-10-30T14:07:40.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 -- 172.21.15.46:0/3603875898 shutdown_connections 2023-10-30T14:07:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f66fc0664a0 0x7f66fc068960 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f671c159e60 0x7f671c1ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f671c0bfec0 0x7f671c0c8e70 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 --2- 172.21.15.46:0/3603875898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f671c06a720 0x7f671c0c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 -- 172.21.15.46:0/3603875898 >> 172.21.15.46:0/3603875898 conn(0x7f671c0c7910 msgr2=0x7f671c0c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 -- 172.21.15.46:0/3603875898 shutdown_connections 2023-10-30T14:07:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:40.874+0000 7f672153e640 1 -- 172.21.15.46:0/3603875898 wait complete. 2023-10-30T14:07:40.899 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 192, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 192, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8352": { "addr": "172.21.15.46:6836/128865315", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 128865315, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 128865315, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8352, "incarnation": 182, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 21 }, "gid_8505": { "addr": "172.21.15.46:6833/4078242540", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4078242540, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4078242540, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8505, "incarnation": 189, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 80, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:07:35.889864+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8505, "mds_1": 8352 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:07:42.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.050+0000 7f5e96b72640 1 Processor -- start 2023-10-30T14:07:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.050+0000 7f5e96b72640 1 -- start start 2023-10-30T14:07:43.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e96b72640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5e901538a0 0x7f5e9015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:43.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e96b72640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5e900ca0e0 0x7f5e9015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e96b72640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e900cb800 con 0x7f5e900cf160 2023-10-30T14:07:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e96b72640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9015dc30 con 0x7f5e900ca0e0 2023-10-30T14:07:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e96b72640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e900cb330 con 0x7f5e901538a0 2023-10-30T14:07:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e8ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5e900ca0e0 0x7f5e9015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e8ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5e900ca0e0 0x7f5e9015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41954/0 (socket says 172.21.15.46:41954) 2023-10-30T14:07:43.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e8ffff640 1 -- 172.21.15.46:0/836386587 learned_addr learned my addr 172.21.15.46:0/836386587 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:43.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e9490e640 1 --2- 172.21.15.46:0/836386587 >> v2:172.21.15.154:3300/0 conn(0x7f5e901538a0 0x7f5e9015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:43.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e8ffff640 1 -- 172.21.15.46:0/836386587 >> v2:172.21.15.154:3300/0 conn(0x7f5e901538a0 msgr2=0x7f5e9015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:43.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e8ffff640 1 --2- 172.21.15.46:0/836386587 >> v2:172.21.15.154:3300/0 conn(0x7f5e901538a0 0x7f5e9015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:43.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e8ffff640 1 -- 172.21.15.46:0/836386587 >> v1:172.21.15.154:6789/0 conn(0x7f5e900cf160 legacy=0x7f5e900cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:43.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.054+0000 7f5e8ffff640 1 -- 172.21.15.46:0/836386587 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e900cad70 con 0x7f5e900ca0e0 2023-10-30T14:07:43.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e8ffff640 1 --2- 172.21.15.46:0/836386587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e900ca0e0 0x7f5e9015a090 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f5e900cfd10 tx=0x7f5e7c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b26ee6d25440631 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:43.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e8f7fe640 1 -- 172.21.15.46:0/836386587 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c0a89f0 con 0x7f5e900ca0e0 2023-10-30T14:07:43.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e8f7fe640 1 -- 172.21.15.46:0/836386587 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7c0a8b50 con 0x7f5e900ca0e0 2023-10-30T14:07:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e8f7fe640 1 -- 172.21.15.46:0/836386587 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c0b0440 con 0x7f5e900ca0e0 2023-10-30T14:07:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e96b72640 1 -- 172.21.15.46:0/836386587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e900ca0e0 msgr2=0x7f5e9015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e96b72640 1 --2- 172.21.15.46:0/836386587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e900ca0e0 0x7f5e9015a090 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f5e900cfd10 tx=0x7f5e7c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:07:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e96b72640 1 -- 172.21.15.46:0/836386587 shutdown_connections 2023-10-30T14:07:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e96b72640 1 --2- 172.21.15.46:0/836386587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e900ca0e0 0x7f5e9015a090 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:43.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e96b72640 1 --2- 172.21.15.46:0/836386587 >> v2:172.21.15.154:3300/0 conn(0x7f5e901538a0 0x7f5e9015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e96b72640 1 -- 172.21.15.46:0/836386587 >> 172.21.15.46:0/836386587 conn(0x7f5e9006bfe0 msgr2=0x7f5e9006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e96b72640 1 -- 172.21.15.46:0/836386587 shutdown_connections 2023-10-30T14:07:43.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e96b72640 1 -- 172.21.15.46:0/836386587 wait complete. 2023-10-30T14:07:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e96b72640 1 Processor -- start 2023-10-30T14:07:43.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e96b72640 1 -- start start 2023-10-30T14:07:43.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.058+0000 7f5e96b72640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e900ca0e0 0x7f5e900ce630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:43.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e96b72640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e900cf160 0x7f5e90142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:43.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e96b72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e901538a0 0x7f5e901ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:43.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e96b72640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e901584d0 con 0x7f5e901538a0 2023-10-30T14:07:43.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e96b72640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e901ef390 con 0x7f5e900ca0e0 2023-10-30T14:07:43.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e96b72640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900cec20 con 0x7f5e900cf160 2023-10-30T14:07:43.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e9510f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e901538a0 0x7f5e901ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:43.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e9510f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e901538a0 0x7f5e901ef7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41970/0 (socket says 172.21.15.46:41970) 2023-10-30T14:07:43.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e9510f640 1 -- 172.21.15.46:0/3708425089 learned_addr learned my addr 172.21.15.46:0/3708425089 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:43.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e8ffff640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e900cf160 0x7f5e90142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e9490e640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e900ca0e0 0x7f5e900ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e9510f640 1 -- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e900cf160 msgr2=0x7f5e90142550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e9510f640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e900cf160 0x7f5e90142550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e9510f640 1 -- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e900ca0e0 msgr2=0x7f5e900ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e9510f640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e900ca0e0 0x7f5e900ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e9510f640 1 -- 172.21.15.46:0/3708425089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c0094d0 con 0x7f5e901538a0 2023-10-30T14:07:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e9490e640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e900ca0e0 0x7f5e900ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e9510f640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e901538a0 0x7f5e901ef7c0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800c6c0 tx=0x7f5e8800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e8d7fa640 1 -- 172.21.15.46:0/3708425089 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88003b80 con 0x7f5e901538a0 2023-10-30T14:07:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e8d7fa640 1 -- 172.21.15.46:0/3708425089 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e88003ce0 con 0x7f5e901538a0 2023-10-30T14:07:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e96b72640 1 -- 172.21.15.46:0/3708425089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e900ceec0 con 0x7f5e901538a0 2023-10-30T14:07:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e8d7fa640 1 -- 172.21.15.46:0/3708425089 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880052f0 con 0x7f5e901538a0 2023-10-30T14:07:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e96b72640 1 -- 172.21.15.46:0/3708425089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e90149080 con 0x7f5e901538a0 2023-10-30T14:07:43.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e96b72640 1 -- 172.21.15.46:0/3708425089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e900b2a50 con 0x7f5e901538a0 2023-10-30T14:07:43.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.062+0000 7f5e8d7fa640 1 -- 172.21.15.46:0/3708425089 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5e88005450 con 0x7f5e901538a0 2023-10-30T14:07:43.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.066+0000 7f5e8d7fa640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e740664a0 0x7f5e74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:43.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.066+0000 7f5e8d7fa640 1 -- 172.21.15.46:0/3708425089 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f5e88084470 con 0x7f5e901538a0 2023-10-30T14:07:43.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.066+0000 7f5e9490e640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e740664a0 0x7f5e74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:43.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.066+0000 7f5e9490e640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e740664a0 0x7f5e74068960 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400b9f0 tx=0x7f5e84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:43.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.066+0000 7f5e8d7fa640 1 -- 172.21.15.46:0/3708425089 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5e88051000 con 0x7f5e901538a0 2023-10-30T14:07:43.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 -- 172.21.15.46:0/3708425089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e901ef4e0 con 0x7f5e901538a0 2023-10-30T14:07:43.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e8d7fa640 1 -- 172.21.15.46:0/3708425089 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f5e901ef4e0 con 0x7f5e901538a0 2023-10-30T14:07:43.265 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:43.265 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:41.902835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:43.265 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 193 2023-10-30T14:07:43.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 -- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e740664a0 msgr2=0x7f5e74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:43.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e740664a0 0x7f5e74068960 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400b9f0 tx=0x7f5e84008040 comp rx=0 tx=0).stop 2023-10-30T14:07:43.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 -- 172.21.15.46:0/3708425089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e901538a0 msgr2=0x7f5e901ef7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:43.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e901538a0 0x7f5e901ef7c0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800c6c0 tx=0x7f5e8800cb80 comp rx=0 tx=0).stop 2023-10-30T14:07:43.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 -- 172.21.15.46:0/3708425089 shutdown_connections 2023-10-30T14:07:43.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5e740664a0 0x7f5e74068960 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:43.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e901538a0 0x7f5e901ef7c0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:43.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e900cf160 0x7f5e90142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:43.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 --2- 172.21.15.46:0/3708425089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e900ca0e0 0x7f5e900ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:43.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 -- 172.21.15.46:0/3708425089 >> 172.21.15.46:0/3708425089 conn(0x7f5e9006bfe0 msgr2=0x7f5e90152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 -- 172.21.15.46:0/3708425089 shutdown_connections 2023-10-30T14:07:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:43.262+0000 7f5e96b72640 1 -- 172.21.15.46:0/3708425089 wait complete. 2023-10-30T14:07:43.286 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=8190 2023-10-30T14:07:43.286 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.e 2023-10-30T14:07:48.287 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T14:07:48.287 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T14:07:48.288 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T14:07:48.292 INFO:tasks.ceph.mds.e:Started 2023-10-30T14:07:48.292 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:48.389 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T14:07:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4910c46640 1 Processor -- start 2023-10-30T14:07:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4910c46640 1 -- start start 2023-10-30T14:07:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4910c46640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f490c1539a0 0x7f490c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4910c46640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f490c070a00 0x7f490c070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4910c46640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490c15a8c0 con 0x7f490c069e90 2023-10-30T14:07:48.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4910c46640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f490c06fbb0 con 0x7f490c070a00 2023-10-30T14:07:48.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4910c46640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f490c0713b0 con 0x7f490c1539a0 2023-10-30T14:07:48.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4903fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f490c070a00 0x7f490c070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:48.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4903fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f490c070a00 0x7f490c070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51034/0 (socket says 172.21.15.46:51034) 2023-10-30T14:07:48.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4903fff640 1 -- 172.21.15.46:0/3370587310 learned_addr learned my addr 172.21.15.46:0/3370587310 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:48.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f490a59c640 1 --2- 172.21.15.46:0/3370587310 >> v2:172.21.15.154:3300/0 conn(0x7f490c1539a0 0x7f490c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:48.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4903fff640 1 -- 172.21.15.46:0/3370587310 >> v2:172.21.15.154:3300/0 conn(0x7f490c1539a0 msgr2=0x7f490c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:07:48.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4903fff640 1 --2- 172.21.15.46:0/3370587310 >> v2:172.21.15.154:3300/0 conn(0x7f490c1539a0 0x7f490c15a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:48.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4903fff640 1 -- 172.21.15.46:0/3370587310 >> v1:172.21.15.154:6789/0 conn(0x7f490c069e90 legacy=0x7f490c06a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:48.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.442+0000 7f4903fff640 1 -- 172.21.15.46:0/3370587310 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490c153d80 con 0x7f490c070a00 2023-10-30T14:07:48.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.446+0000 7f4903fff640 1 --2- 172.21.15.46:0/3370587310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f490c070a00 0x7f490c070e70 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f490c0c8950 tx=0x7f48f401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7ad73041a119f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:48.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.446+0000 7f4909d9b640 1 -- 172.21.15.46:0/3370587310 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f40ab640 con 0x7f490c070a00 2023-10-30T14:07:48.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.446+0000 7f4909d9b640 1 -- 172.21.15.46:0/3370587310 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f40a7070 con 0x7f490c070a00 2023-10-30T14:07:48.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.446+0000 7f4909d9b640 1 -- 172.21.15.46:0/3370587310 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f40ab9c0 con 0x7f490c070a00 2023-10-30T14:07:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.446+0000 7f4910c46640 1 -- 172.21.15.46:0/3370587310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f490c070a00 msgr2=0x7f490c070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.446+0000 7f4910c46640 1 --2- 172.21.15.46:0/3370587310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f490c070a00 0x7f490c070e70 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f490c0c8950 tx=0x7f48f401c440 comp rx=0 tx=0).stop 2023-10-30T14:07:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.446+0000 7f4910c46640 1 -- 172.21.15.46:0/3370587310 shutdown_connections 2023-10-30T14:07:48.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.446+0000 7f4910c46640 1 --2- 172.21.15.46:0/3370587310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f490c070a00 0x7f490c070e70 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:48.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.446+0000 7f4910c46640 1 --2- 172.21.15.46:0/3370587310 >> v2:172.21.15.154:3300/0 conn(0x7f490c1539a0 0x7f490c15a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:48.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.446+0000 7f4910c46640 1 -- 172.21.15.46:0/3370587310 >> 172.21.15.46:0/3370587310 conn(0x7f490c0c7910 msgr2=0x7f490c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:48.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.446+0000 7f4910c46640 1 -- 172.21.15.46:0/3370587310 shutdown_connections 2023-10-30T14:07:48.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.446+0000 7f4910c46640 1 -- 172.21.15.46:0/3370587310 wait complete. 2023-10-30T14:07:48.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4910c46640 1 Processor -- start 2023-10-30T14:07:48.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4910c46640 1 -- start start 2023-10-30T14:07:48.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4910c46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f490c069e90 0x7f490c158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:48.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4910c46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f490c070a00 0x7f490c158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:48.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4910c46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f490c1539a0 0x7f490c1429a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:48.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4910c46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490c0c9620 con 0x7f490c069e90 2023-10-30T14:07:48.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4910c46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490c0c98c0 con 0x7f490c1539a0 2023-10-30T14:07:48.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4910c46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490c0c9ba0 con 0x7f490c070a00 2023-10-30T14:07:48.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f490a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f490c069e90 0x7f490c158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:48.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f490a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f490c069e90 0x7f490c158550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51046/0 (socket says 172.21.15.46:51046) 2023-10-30T14:07:48.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f490a59c640 1 -- 172.21.15.46:0/4077806012 learned_addr learned my addr 172.21.15.46:0/4077806012 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:48.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4903fff640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f490c070a00 0x7f490c158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f490ad9d640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f490c1539a0 0x7f490c1429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f490a59c640 1 -- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f490c070a00 msgr2=0x7f490c158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f490a59c640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f490c070a00 0x7f490c158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f490a59c640 1 -- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f490c1539a0 msgr2=0x7f490c1429a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f490a59c640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f490c1539a0 0x7f490c1429a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:48.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f490a59c640 1 -- 172.21.15.46:0/4077806012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f40094d0 con 0x7f490c069e90 2023-10-30T14:07:48.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f490ad9d640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f490c1539a0 0x7f490c1429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:07:48.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f490a59c640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f490c069e90 0x7f490c158550 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00d800 tx=0x7f48fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:48.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4902ffd640 1 -- 172.21.15.46:0/4077806012 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc00b6c0 con 0x7f490c069e90 2023-10-30T14:07:48.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4902ffd640 1 -- 172.21.15.46:0/4077806012 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48fc00b820 con 0x7f490c069e90 2023-10-30T14:07:48.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4902ffd640 1 -- 172.21.15.46:0/4077806012 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc00bb90 con 0x7f490c069e90 2023-10-30T14:07:48.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4910c46640 1 -- 172.21.15.46:0/4077806012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490c0c9e80 con 0x7f490c069e90 2023-10-30T14:07:48.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.450+0000 7f4910c46640 1 -- 172.21.15.46:0/4077806012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490c144420 con 0x7f490c069e90 2023-10-30T14:07:48.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.454+0000 7f4910c46640 1 -- 172.21.15.46:0/4077806012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490c04e170 con 0x7f490c069e90 2023-10-30T14:07:48.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.454+0000 7f4902ffd640 1 -- 172.21.15.46:0/4077806012 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f48fc00bcf0 con 0x7f490c069e90 2023-10-30T14:07:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.454+0000 7f4902ffd640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48ec0664a0 0x7f48ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.454+0000 7f4902ffd640 1 -- 172.21.15.46:0/4077806012 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f48fc084cd0 con 0x7f490c069e90 2023-10-30T14:07:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.454+0000 7f4903fff640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48ec0664a0 0x7f48ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:48.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.458+0000 7f4903fff640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48ec0664a0 0x7f48ec068960 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f48f40af000 tx=0x7f48f40b6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:48.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.458+0000 7f4902ffd640 1 -- 172.21.15.46:0/4077806012 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f48fc051860 con 0x7f490c069e90 2023-10-30T14:07:48.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.658+0000 7f4910c46640 1 -- 172.21.15.46:0/4077806012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f490c1425c0 con 0x7f490c069e90 2023-10-30T14:07:48.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.658+0000 7f4902ffd640 1 -- 172.21.15.46:0/4077806012 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f48fc054e80 con 0x7f490c069e90 2023-10-30T14:07:48.662 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:48.662 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:41.902835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:48.662 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 193 2023-10-30T14:07:48.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.662+0000 7f4910c46640 1 -- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48ec0664a0 msgr2=0x7f48ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:48.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.662+0000 7f4910c46640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48ec0664a0 0x7f48ec068960 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f48f40af000 tx=0x7f48f40b6040 comp rx=0 tx=0).stop 2023-10-30T14:07:48.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.662+0000 7f4910c46640 1 -- 172.21.15.46:0/4077806012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f490c069e90 msgr2=0x7f490c158550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:48.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.662+0000 7f4910c46640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f490c069e90 0x7f490c158550 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00d800 tx=0x7f48fc00dcc0 comp rx=0 tx=0).stop 2023-10-30T14:07:48.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.662+0000 7f4910c46640 1 -- 172.21.15.46:0/4077806012 shutdown_connections 2023-10-30T14:07:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.662+0000 7f4910c46640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48ec0664a0 0x7f48ec068960 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.662+0000 7f4910c46640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f490c1539a0 0x7f490c1429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.662+0000 7f4910c46640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f490c070a00 0x7f490c158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.662+0000 7f4910c46640 1 --2- 172.21.15.46:0/4077806012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f490c069e90 0x7f490c158550 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.662+0000 7f4910c46640 1 -- 172.21.15.46:0/4077806012 >> 172.21.15.46:0/4077806012 conn(0x7f490c0c7910 msgr2=0x7f490c14fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.662+0000 7f4910c46640 1 -- 172.21.15.46:0/4077806012 shutdown_connections 2023-10-30T14:07:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:48.662+0000 7f4910c46640 1 -- 172.21.15.46:0/4077806012 wait complete. 2023-10-30T14:07:48.690 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T14:07:49.695 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 3% (2/59) Updating files: 5% (3/59) Updating files: 6% (4/59) Updating files: 8% (5/59) Updating files: 10% (6/59) Updating files: 11% (7/59) Updating files: 13% (8/59) Updating files: 15% (9/59) Updating files: 16% (10/59) Updating files: 18% (11/59) Updating files: 20% (12/59) Updating files: 22% (13/59) Updating files: 23% (14/59) Updating files: 25% (15/59) Updating files: 27% (16/59) Updating files: 28% (17/59) Updating files: 30% (18/59) Updating files: 32% (19/59) Updating files: 33% (20/59) Updating files: 35% (21/59) Updating files: 37% (22/59) Updating files: 38% (23/59) Updating files: 40% (24/59) Updating files: 42% (25/59) Updating files: 44% (26/59) Updating files: 45% (27/59) Updating files: 47% (28/59) Updating files: 49% (29/59) Updating files: 50% (30/59) Updating files: 52% (31/59) Updating files: 54% (32/59) Updating files: 55% (33/59) Updating files: 57% (34/59) Updating files: 59% (35/59) Updating files: 61% (36/59) Updating files: 62% (37/59) Updating files: 64% (38/59) Updating files: 66% (39/59) Updating files: 67% (40/59) Updating files: 69% (41/59) Updating files: 71% (42/59) Updating files: 72% (43/59) Updating files: 74% (44/59) Updating files: 76% (45/59) Updating files: 77% (46/59) Updating files: 79% (47/59) Updating files: 81% (48/59) Updating files: 83% (49/59) Updating files: 84% (50/59) Updating files: 86% (51/59) Updating files: 88% (52/59) Updating files: 89% (53/59) Updating files: 91% (54/59) Updating files: 93% (55/59) Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2023-10-30T14:07:49.777 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-10-30T14:07:49.774+0000 7f0797c92640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:07:49.777 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-10-30T14:07:49.774+0000 7f0797c92640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:07:49.927 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at c51c8f9d80f 0.87 2023-10-30T14:07:49.931 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.87 2023-10-30T14:07:49.986 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 88 -eq 48 ] 2023-10-30T14:07:49.986 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.88 2023-10-30T14:07:49.986 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.88 2023-10-30T14:07:49.987 INFO:tasks.workunit.client.0.smithi046.stdout:v0.88 2023-10-30T14:07:49.987 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.88 2023-10-30T14:07:50.692 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:07:50.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45bd596640 1 Processor -- start 2023-10-30T14:07:50.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45bd596640 1 -- start start 2023-10-30T14:07:50.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45bd596640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f45b815cd50 0x7f45b80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:50.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45bd596640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b815a6c0 con 0x7f45b80cefa0 2023-10-30T14:07:50.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45bd596640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b815d300 con 0x7f45b81538f0 2023-10-30T14:07:50.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45bd596640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b815d5e0 con 0x7f45b815cd50 2023-10-30T14:07:50.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45bcd95640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f45b80cefa0 0x7f45b81640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50292/0 (socket says 172.21.15.46:50292) 2023-10-30T14:07:50.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45bcd95640 1 -- 172.21.15.46:0/1576527984 learned_addr learned my addr 172.21.15.46:0/1576527984 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:07:50.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45b77fe640 1 --2- 172.21.15.46:0/1576527984 >> v2:172.21.15.154:3300/0 conn(0x7f45b815cd50 0x7f45b80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:50.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45b6ffd640 1 -- 172.21.15.46:0/1576527984 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253336403 0 0) 0x7f45b815a6c0 con 0x7f45b80cefa0 2023-10-30T14:07:50.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45b6ffd640 1 -- 172.21.15.46:0/1576527984 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a0003530 con 0x7f45b80cefa0 2023-10-30T14:07:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45b77fe640 1 -- 172.21.15.46:0/1576527984 >> v1:172.21.15.154:6789/0 conn(0x7f45b81538f0 legacy=0x7f45b8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45b6ffd640 1 -- 172.21.15.46:0/1576527984 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1143571076 0 0) 0x7f45a0003530 con 0x7f45b80cefa0 2023-10-30T14:07:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45b77fe640 1 -- 172.21.15.46:0/1576527984 >> v1:172.21.15.46:6789/0 conn(0x7f45b80cefa0 legacy=0x7f45b81640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:50.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.842+0000 7f45b77fe640 1 -- 172.21.15.46:0/1576527984 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b815d8c0 con 0x7f45b815cd50 2023-10-30T14:07:50.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.846+0000 7f45b77fe640 1 --2- 172.21.15.46:0/1576527984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b815cd50 0x7f45b80c9f90 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0097c0 tx=0x7f45ac01eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc8289994a28216d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:50.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.846+0000 7f45b6ffd640 1 -- 172.21.15.46:0/1576527984 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b0001a00 con 0x7f45b815cd50 2023-10-30T14:07:50.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.846+0000 7f45b6ffd640 1 -- 172.21.15.46:0/1576527984 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac0a8610 con 0x7f45b815cd50 2023-10-30T14:07:50.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.846+0000 7f45b6ffd640 1 -- 172.21.15.46:0/1576527984 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac0a8980 con 0x7f45b815cd50 2023-10-30T14:07:50.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.850+0000 7f45bd596640 1 -- 172.21.15.46:0/1576527984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b815cd50 msgr2=0x7f45b80c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:50.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.850+0000 7f45bd596640 1 --2- 172.21.15.46:0/1576527984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b815cd50 0x7f45b80c9f90 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0097c0 tx=0x7f45ac01eb90 comp rx=0 tx=0).stop 2023-10-30T14:07:50.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.850+0000 7f45bd596640 1 -- 172.21.15.46:0/1576527984 shutdown_connections 2023-10-30T14:07:50.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.850+0000 7f45bd596640 1 --2- 172.21.15.46:0/1576527984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b815cd50 0x7f45b80c9f90 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:50.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.850+0000 7f45bd596640 1 -- 172.21.15.46:0/1576527984 >> 172.21.15.46:0/1576527984 conn(0x7f45b806a640 msgr2=0x7f45b8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:50.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.850+0000 7f45bd596640 1 -- 172.21.15.46:0/1576527984 shutdown_connections 2023-10-30T14:07:50.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.850+0000 7f45bd596640 1 -- 172.21.15.46:0/1576527984 wait complete. 2023-10-30T14:07:50.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.850+0000 7f45bd596640 1 Processor -- start 2023-10-30T14:07:50.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.850+0000 7f45bd596640 1 -- start start 2023-10-30T14:07:50.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45bd596640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80cefa0 0x7f45b81f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:50.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45bd596640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b81538f0 0x7f45b8145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:50.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45bd596640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b815cd50 0x7f45b8152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:50.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45bd596640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b81531f0 con 0x7f45b80cefa0 2023-10-30T14:07:50.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45bd596640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b81597f0 con 0x7f45b81538f0 2023-10-30T14:07:50.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45bd596640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b815c4c0 con 0x7f45b815cd50 2023-10-30T14:07:50.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80cefa0 0x7f45b81f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:50.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80cefa0 0x7f45b81f2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51062/0 (socket says 172.21.15.46:51062) 2023-10-30T14:07:50.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b7fff640 1 -- 172.21.15.46:0/1549047984 learned_addr learned my addr 172.21.15.46:0/1549047984 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:07:50.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b77fe640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b81538f0 0x7f45b8145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:50.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45bcd95640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b815cd50 0x7f45b8152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:50.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b7fff640 1 -- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b815cd50 msgr2=0x7f45b8152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:50.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b7fff640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b815cd50 0x7f45b8152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:50.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b7fff640 1 -- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b81538f0 msgr2=0x7f45b8145ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:50.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b7fff640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b81538f0 0x7f45b8145ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:50.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b7fff640 1 -- 172.21.15.46:0/1549047984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac009420 con 0x7f45b80cefa0 2023-10-30T14:07:50.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b7fff640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80cefa0 0x7f45b81f2750 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f45a800c560 tx=0x7f45a800ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:50.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b4ff9640 1 -- 172.21.15.46:0/1549047984 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8005220 con 0x7f45b80cefa0 2023-10-30T14:07:50.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b4ff9640 1 -- 172.21.15.46:0/1549047984 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a8005380 con 0x7f45b80cefa0 2023-10-30T14:07:50.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45bd596640 1 -- 172.21.15.46:0/1549047984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b815c710 con 0x7f45b80cefa0 2023-10-30T14:07:50.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45b4ff9640 1 -- 172.21.15.46:0/1549047984 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8007270 con 0x7f45b80cefa0 2023-10-30T14:07:50.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.854+0000 7f45bd596640 1 -- 172.21.15.46:0/1549047984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b8150ef0 con 0x7f45b80cefa0 2023-10-30T14:07:50.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.858+0000 7f45bd596640 1 -- 172.21.15.46:0/1549047984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b8145b30 con 0x7f45b80cefa0 2023-10-30T14:07:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.858+0000 7f45b4ff9640 1 -- 172.21.15.46:0/1549047984 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f45a80073d0 con 0x7f45b80cefa0 2023-10-30T14:07:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.858+0000 7f45b4ff9640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45a40664d0 0x7f45a4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:07:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.858+0000 7f45b4ff9640 1 -- 172.21.15.46:0/1549047984 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f45a8083e10 con 0x7f45b80cefa0 2023-10-30T14:07:50.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.858+0000 7f45b77fe640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45a40664d0 0x7f45a4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:07:50.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.862+0000 7f45b77fe640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45a40664d0 0x7f45a4068990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0b2000 tx=0x7f45ac0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:07:50.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:50.866+0000 7f45b4ff9640 1 -- 172.21.15.46:0/1549047984 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f45a8050af0 con 0x7f45b80cefa0 2023-10-30T14:07:51.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.058+0000 7f45bd596640 1 -- 172.21.15.46:0/1549047984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45b804e170 con 0x7f45b80cefa0 2023-10-30T14:07:51.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.058+0000 7f45b4ff9640 1 -- 172.21.15.46:0/1549047984 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f45a8054110 con 0x7f45b80cefa0 2023-10-30T14:07:51.062 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:07:51.062 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8655,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":194}],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:41.902835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:07:51.062 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 194 2023-10-30T14:07:51.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.062+0000 7f45bd596640 1 -- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45a40664d0 msgr2=0x7f45a4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:51.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.062+0000 7f45bd596640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45a40664d0 0x7f45a4068990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0b2000 tx=0x7f45ac0aa070 comp rx=0 tx=0).stop 2023-10-30T14:07:51.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.062+0000 7f45bd596640 1 -- 172.21.15.46:0/1549047984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80cefa0 msgr2=0x7f45b81f2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:07:51.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.062+0000 7f45bd596640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80cefa0 0x7f45b81f2750 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f45a800c560 tx=0x7f45a800ca20 comp rx=0 tx=0).stop 2023-10-30T14:07:51.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.062+0000 7f45bd596640 1 -- 172.21.15.46:0/1549047984 shutdown_connections 2023-10-30T14:07:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.062+0000 7f45bd596640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45a40664d0 0x7f45a4068990 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.062+0000 7f45bd596640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45b815cd50 0x7f45b8152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.062+0000 7f45bd596640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b81538f0 0x7f45b8145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.062+0000 7f45bd596640 1 --2- 172.21.15.46:0/1549047984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b80cefa0 0x7f45b81f2750 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:07:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.062+0000 7f45bd596640 1 -- 172.21.15.46:0/1549047984 >> 172.21.15.46:0/1549047984 conn(0x7f45b806a640 msgr2=0x7f45b80c4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:07:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.062+0000 7f45bd596640 1 -- 172.21.15.46:0/1549047984 shutdown_connections 2023-10-30T14:07:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:07:51.062+0000 7f45bd596640 1 -- 172.21.15.46:0/1549047984 wait complete. 2023-10-30T14:07:51.083 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T14:07:51.083 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 36 secs before thrashing 2023-10-30T14:07:56.705 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:07:56.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:08:06.082 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 36% (121/335) Updating files: 37% (124/335) Updating files: 37% (126/335) Updating files: 38% (128/335) Updating files: 39% (131/335) Updating files: 40% (134/335) Updating files: 41% (138/335) Updating files: 42% (141/335) Updating files: 43% (145/335) Updating files: 44% (148/335) Updating files: 45% (151/335) Updating files: 46% (155/335) Updating files: 47% (158/335) Updating files: 48% (161/335) Updating files: 49% (165/335) Updating files: 50% (168/335) Updating files: 51% (171/335) Updating files: 52% (175/335) Updating files: 53% (178/335) Updating files: 54% (181/335) Updating files: 55% (185/335) Updating files: 56% (188/335) Updating files: 57% (191/335) Updating files: 58% (195/335) Updating files: 59% (198/335) Updating files: 60% (201/335) Updating files: 61% (205/335) Updating files: 62% (208/335) Updating files: 63% (212/335) Updating files: 64% (215/335) Updating files: 65% (218/335) Updating files: 66% (222/335) Updating files: 67% (225/335) Updating files: 68% (228/335) Updating files: 69% (232/335) Updating files: 70% (235/335) Updating files: 70% (237/335) Updating files: 71% (238/335) Updating files: 72% (242/335) Updating files: 73% (245/335) Updating files: 73% (246/335) Updating files: 74% (248/335) Updating files: 75% (252/335) Updating files: 76% (255/335) Updating files: 77% (258/335) Updating files: 78% (262/335) Updating files: 79% (265/335) Updating files: 80% (268/335) Updating files: 81% (272/335) Updating files: 82% (275/335) Updating files: 83% (279/335) Updating files: 84% (282/335) Updating files: 85% (285/335) Updating files: 86% (289/335) Updating files: 87% (292/335) Updating files: 88% (295/335) Updating files: 89% (299/335) Updating files: 90% (302/335) Updating files: 91% (305/335) Updating files: 92% (309/335) Updating files: 93% (312/335) Updating files: 94% (315/335) Updating files: 95% (319/335) Updating files: 96% (322/335) Updating files: 97% (325/335) Updating files: 98% (329/335) Updating files: 99% (332/335) Updating files: 100% (335/335) Updating files: 100% (335/335), done. 2023-10-30T14:08:06.777 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 4be687bf448 0.88 2023-10-30T14:08:06.785 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.88 2023-10-30T14:08:07.043 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 89 -eq 48 ] 2023-10-30T14:08:07.043 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.89 2023-10-30T14:08:07.043 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.89 2023-10-30T14:08:07.043 INFO:tasks.workunit.client.0.smithi046.stdout:v0.89 2023-10-30T14:08:07.044 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.89 2023-10-30T14:08:14.258 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 62% (124/197) Updating files: 63% (125/197) Updating files: 64% (127/197) Updating files: 65% (129/197) Updating files: 66% (131/197) Updating files: 67% (132/197) Updating files: 68% (134/197) Updating files: 69% (136/197) Updating files: 70% (138/197) Updating files: 71% (140/197) Updating files: 72% (142/197) Updating files: 73% (144/197) Updating files: 74% (146/197) Updating files: 75% (148/197) Updating files: 76% (150/197) Updating files: 77% (152/197) Updating files: 78% (154/197) Updating files: 79% (156/197) Updating files: 80% (158/197) Updating files: 81% (160/197) Updating files: 82% (162/197) Updating files: 83% (164/197) Updating files: 84% (166/197) Updating files: 85% (168/197) Updating files: 86% (170/197) Updating files: 87% (172/197) Updating files: 88% (174/197) Updating files: 89% (176/197) Updating files: 90% (178/197) Updating files: 91% (180/197) Updating files: 92% (182/197) Updating files: 93% (184/197) Updating files: 94% (186/197) Updating files: 95% (188/197) Updating files: 96% (190/197) Updating files: 97% (192/197) Updating files: 98% (194/197) Updating files: 99% (196/197) Updating files: 100% (197/197) Updating files: 100% (197/197), done. 2023-10-30T14:08:14.314 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 68fdc0f68e6 0.89 2023-10-30T14:08:14.320 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.89 2023-10-30T14:08:14.416 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 90 -eq 48 ] 2023-10-30T14:08:14.416 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.90 2023-10-30T14:08:14.416 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo v0.90 2023-10-30T14:08:14.416 INFO:tasks.workunit.client.0.smithi046.stderr:+ git reset --hard v0.90 2023-10-30T14:08:14.417 INFO:tasks.workunit.client.0.smithi046.stdout:v0.90 2023-10-30T14:08:20.460 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 39% (105/265) Updating files: 40% (106/265) Updating files: 41% (109/265) Updating files: 42% (112/265) Updating files: 43% (114/265) Updating files: 44% (117/265) Updating files: 45% (120/265) Updating files: 46% (122/265) Updating files: 47% (125/265) Updating files: 48% (128/265) Updating files: 49% (130/265) Updating files: 49% (132/265) Updating files: 50% (133/265) Updating files: 51% (136/265) Updating files: 52% (138/265) Updating files: 53% (141/265) Updating files: 54% (144/265) Updating files: 55% (146/265) Updating files: 56% (149/265) Updating files: 57% (152/265) Updating files: 58% (154/265) Updating files: 59% (157/265) Updating files: 60% (159/265) Updating files: 61% (162/265) Updating files: 62% (165/265) Updating files: 63% (167/265) Updating files: 64% (170/265) Updating files: 65% (173/265) Updating files: 66% (175/265) Updating files: 67% (178/265) Updating files: 68% (181/265) Updating files: 69% (183/265) Updating files: 70% (186/265) Updating files: 71% (189/265) Updating files: 72% (191/265) Updating files: 73% (194/265) Updating files: 74% (197/265) Updating files: 75% (199/265) Updating files: 76% (202/265) Updating files: 77% (205/265) Updating files: 78% (207/265) Updating files: 79% (210/265) Updating files: 80% (212/265) Updating files: 81% (215/265) Updating files: 82% (218/265) Updating files: 83% (220/265) Updating files: 84% (223/265) Updating files: 85% (226/265) Updating files: 86% (228/265) Updating files: 87% (231/265) Updating files: 88% (234/265) Updating files: 89% (236/265) Updating files: 90% (239/265) Updating files: 91% (242/265) Updating files: 92% (244/265) Updating files: 93% (247/265) Updating files: 94% (250/265) Updating files: 95% (252/265) Updating files: 96% (255/265) Updating files: 97% (258/265) Updating files: 98% (260/265) Updating files: 99% (263/265) Updating files: 100% (265/265) Updating files: 100% (265/265), done. 2023-10-30T14:08:20.599 INFO:tasks.workunit.client.0.smithi046.stdout:HEAD is now at 08bd1e1eee3 0.90 2023-10-30T14:08:20.604 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/v0.90 2023-10-30T14:08:20.656 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 1 -eq 48 ] 2023-10-30T14:08:20.656 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.1 2023-10-30T14:08:20.656 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.1 2023-10-30T14:08:20.657 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.1 2023-10-30T14:08:20.657 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.1 2023-10-30T14:08:20.657 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:08:26.902 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:08:26.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:08:27.085 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:27.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1419daa640 1 Processor -- start 2023-10-30T14:08:27.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1419daa640 1 -- start start 2023-10-30T14:08:27.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1419daa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f14140c6fe0 0x7f1414071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:27.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1419daa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14140c7b90 con 0x7f14140cb550 2023-10-30T14:08:27.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1419daa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14140c60e0 con 0x7f1414153bf0 2023-10-30T14:08:27.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1419daa640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f141415d870 con 0x7f14140c6fe0 2023-10-30T14:08:27.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1412ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f14140c6fe0 0x7f1414071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:27.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1412ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f14140c6fe0 0x7f1414071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40740/0 (socket says 172.21.15.46:40740) 2023-10-30T14:08:27.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1412ffd640 1 -- 172.21.15.46:0/734238373 learned_addr learned my addr 172.21.15.46:0/734238373 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:27.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1413fff640 1 --1- 172.21.15.46:0/734238373 >> v1:172.21.15.46:6789/0 conn(0x7f14140cb550 0x7f1414164260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44004/0 (socket says 172.21.15.46:44004) 2023-10-30T14:08:27.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1412ffd640 1 -- 172.21.15.46:0/734238373 >> v1:172.21.15.154:6789/0 conn(0x7f1414153bf0 legacy=0x7f14140c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:27.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1412ffd640 1 -- 172.21.15.46:0/734238373 >> v1:172.21.15.46:6789/0 conn(0x7f14140cb550 legacy=0x7f1414164260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:27.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.330+0000 7f1412ffd640 1 -- 172.21.15.46:0/734238373 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14140c63a0 con 0x7f14140c6fe0 2023-10-30T14:08:27.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.334+0000 7f1412ffd640 1 --2- 172.21.15.46:0/734238373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14140c6fe0 0x7f1414071260 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f1408009870 tx=0x7f140801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd92113ccb0ea666 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:27.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.334+0000 7f14127fc640 1 -- 172.21.15.46:0/734238373 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14080aa430 con 0x7f14140c6fe0 2023-10-30T14:08:27.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.334+0000 7f14127fc640 1 -- 172.21.15.46:0/734238373 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14080aa590 con 0x7f14140c6fe0 2023-10-30T14:08:27.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.334+0000 7f14127fc640 1 -- 172.21.15.46:0/734238373 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14080aa920 con 0x7f14140c6fe0 2023-10-30T14:08:27.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.334+0000 7f1419daa640 1 -- 172.21.15.46:0/734238373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14140c6fe0 msgr2=0x7f1414071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:27.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.334+0000 7f1419daa640 1 --2- 172.21.15.46:0/734238373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14140c6fe0 0x7f1414071260 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f1408009870 tx=0x7f140801c4d0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.334+0000 7f1419daa640 1 -- 172.21.15.46:0/734238373 shutdown_connections 2023-10-30T14:08:27.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.334+0000 7f1419daa640 1 --2- 172.21.15.46:0/734238373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14140c6fe0 0x7f1414071260 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.334+0000 7f1419daa640 1 -- 172.21.15.46:0/734238373 >> 172.21.15.46:0/734238373 conn(0x7f141406c6e0 msgr2=0x7f1414069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:27.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.334+0000 7f1419daa640 1 -- 172.21.15.46:0/734238373 shutdown_connections 2023-10-30T14:08:27.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.334+0000 7f1419daa640 1 -- 172.21.15.46:0/734238373 wait complete. 2023-10-30T14:08:27.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.334+0000 7f1419daa640 1 Processor -- start 2023-10-30T14:08:27.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1419daa640 1 -- start start 2023-10-30T14:08:27.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1419daa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14140c6fe0 0x7f141415c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:27.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1419daa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14140cb550 0x7f14141576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:27.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1419daa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1414153bf0 0x7f1414157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:27.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1419daa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14140c4250 con 0x7f14140cb550 2023-10-30T14:08:27.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1419daa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14141f3550 con 0x7f14140c6fe0 2023-10-30T14:08:27.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1419daa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14140c4ca0 con 0x7f1414153bf0 2023-10-30T14:08:27.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1412ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14140cb550 0x7f14141576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:27.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1412ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14140cb550 0x7f14141576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40756/0 (socket says 172.21.15.46:40756) 2023-10-30T14:08:27.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1412ffd640 1 -- 172.21.15.46:0/2382385577 learned_addr learned my addr 172.21.15.46:0/2382385577 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:27.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1413fff640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1414153bf0 0x7f1414157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:27.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f14137fe640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14140c6fe0 0x7f141415c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:27.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1412ffd640 1 -- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1414153bf0 msgr2=0x7f1414157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:27.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1412ffd640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1414153bf0 0x7f1414157c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1412ffd640 1 -- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14140c6fe0 msgr2=0x7f141415c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:27.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1412ffd640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14140c6fe0 0x7f141415c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1412ffd640 1 -- 172.21.15.46:0/2382385577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14080094d0 con 0x7f14140cb550 2023-10-30T14:08:27.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1412ffd640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14140cb550 0x7f14141576d0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f140801a100 tx=0x7f140801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:27.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f13fbfff640 1 -- 172.21.15.46:0/2382385577 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14080b38c0 con 0x7f14140cb550 2023-10-30T14:08:27.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f13fbfff640 1 -- 172.21.15.46:0/2382385577 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14080b3a40 con 0x7f14140cb550 2023-10-30T14:08:27.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1419daa640 1 -- 172.21.15.46:0/2382385577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14140c5260 con 0x7f14140cb550 2023-10-30T14:08:27.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f13fbfff640 1 -- 172.21.15.46:0/2382385577 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14080b3df0 con 0x7f14140cb550 2023-10-30T14:08:27.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1419daa640 1 -- 172.21.15.46:0/2382385577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14140c4740 con 0x7f14140cb550 2023-10-30T14:08:27.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.338+0000 7f1419daa640 1 -- 172.21.15.46:0/2382385577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14140c53f0 con 0x7f14140cb550 2023-10-30T14:08:27.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.342+0000 7f13fbfff640 1 -- 172.21.15.46:0/2382385577 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f14080b2380 con 0x7f14140cb550 2023-10-30T14:08:27.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.342+0000 7f13fbfff640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f14000664a0 0x7f1400068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:27.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.342+0000 7f13fbfff640 1 -- 172.21.15.46:0/2382385577 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f140811ed20 con 0x7f14140cb550 2023-10-30T14:08:27.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.342+0000 7f14137fe640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f14000664a0 0x7f1400068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:27.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.342+0000 7f14137fe640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f14000664a0 0x7f1400068960 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f140400b810 tx=0x7f1404008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:27.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.346+0000 7f13fbfff640 1 -- 172.21.15.46:0/2382385577 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f14080eb940 con 0x7f14140cb550 2023-10-30T14:08:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.542+0000 7f1419daa640 1 -- 172.21.15.46:0/2382385577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14140c53f0 con 0x7f14140cb550 2023-10-30T14:08:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.542+0000 7f13fbfff640 1 -- 172.21.15.46:0/2382385577 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f14080eef60 con 0x7f14140cb550 2023-10-30T14:08:27.546 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:27.546 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8655,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":194}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:52.610131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 195 2023-10-30T14:08:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.546+0000 7f1419daa640 1 -- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f14000664a0 msgr2=0x7f1400068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.546+0000 7f1419daa640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f14000664a0 0x7f1400068960 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f140400b810 tx=0x7f1404008040 comp rx=0 tx=0).stop 2023-10-30T14:08:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.546+0000 7f1419daa640 1 -- 172.21.15.46:0/2382385577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14140cb550 msgr2=0x7f14141576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.546+0000 7f1419daa640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14140cb550 0x7f14141576d0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f140801a100 tx=0x7f140801ad10 comp rx=0 tx=0).stop 2023-10-30T14:08:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.546+0000 7f1419daa640 1 -- 172.21.15.46:0/2382385577 shutdown_connections 2023-10-30T14:08:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.546+0000 7f1419daa640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f14000664a0 0x7f1400068960 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.546+0000 7f1419daa640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1414153bf0 0x7f1414157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.546+0000 7f1419daa640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14140cb550 0x7f14141576d0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.546+0000 7f1419daa640 1 --2- 172.21.15.46:0/2382385577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14140c6fe0 0x7f141415c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.546+0000 7f1419daa640 1 -- 172.21.15.46:0/2382385577 >> 172.21.15.46:0/2382385577 conn(0x7f141406c6e0 msgr2=0x7f1414069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.546+0000 7f1419daa640 1 -- 172.21.15.46:0/2382385577 shutdown_connections 2023-10-30T14:08:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.546+0000 7f1419daa640 1 -- 172.21.15.46:0/2382385577 wait complete. 2023-10-30T14:08:27.574 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=1) 2023-10-30T14:08:27.574 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:08:27.575 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:08:27.610 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:08:27.610 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:27.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.814+0000 7fa36331d640 1 Processor -- start 2023-10-30T14:08:27.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.814+0000 7fa36331d640 1 -- start start 2023-10-30T14:08:27.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa36331d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa35c1538a0 0x7fa35c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:27.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa36331d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa35c0cf160 0x7fa35c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:27.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa36331d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c0cb800 con 0x7fa35c0ca0e0 2023-10-30T14:08:27.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa36331d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa35c15dc30 con 0x7fa35c0cf160 2023-10-30T14:08:27.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa36331d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa35c0cb330 con 0x7fa35c1538a0 2023-10-30T14:08:27.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa3618ba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa35c0cf160 0x7fa35c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:27.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa3618ba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa35c0cf160 0x7fa35c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40772/0 (socket says 172.21.15.46:40772) 2023-10-30T14:08:27.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa3618ba640 1 -- 172.21.15.46:0/1604673608 learned_addr learned my addr 172.21.15.46:0/1604673608 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:27.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa3608b8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa35c0ca0e0 0x7fa35c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44016/0 (socket says 172.21.15.46:44016) 2023-10-30T14:08:27.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa3610b9640 1 --2- 172.21.15.46:0/1604673608 >> v2:172.21.15.154:3300/0 conn(0x7fa35c1538a0 0x7fa35c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:27.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa3618ba640 1 -- 172.21.15.46:0/1604673608 >> v2:172.21.15.154:3300/0 conn(0x7fa35c1538a0 msgr2=0x7fa35c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:27.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa3618ba640 1 --2- 172.21.15.46:0/1604673608 >> v2:172.21.15.154:3300/0 conn(0x7fa35c1538a0 0x7fa35c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa3618ba640 1 -- 172.21.15.46:0/1604673608 >> v1:172.21.15.46:6789/0 conn(0x7fa35c0ca0e0 legacy=0x7fa35c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:27.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa3618ba640 1 -- 172.21.15.46:0/1604673608 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c0cad70 con 0x7fa35c0cf160 2023-10-30T14:08:27.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.818+0000 7fa353fff640 1 -- 172.21.15.46:0/1604673608 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609571485 0 0) 0x7fa35c0cb800 con 0x7fa35c0ca0e0 2023-10-30T14:08:27.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa3618ba640 1 --2- 172.21.15.46:0/1604673608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa35c0cf160 0x7fa35c0cf590 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0cfd10 tx=0x7fa35401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0868477bce2f68 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:27.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa353fff640 1 -- 172.21.15.46:0/1604673608 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3540af400 con 0x7fa35c0cf160 2023-10-30T14:08:27.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa353fff640 1 -- 172.21.15.46:0/1604673608 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3540ab050 con 0x7fa35c0cf160 2023-10-30T14:08:27.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa353fff640 1 -- 172.21.15.46:0/1604673608 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3540af780 con 0x7fa35c0cf160 2023-10-30T14:08:27.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa36331d640 1 -- 172.21.15.46:0/1604673608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa35c0cf160 msgr2=0x7fa35c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:27.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa36331d640 1 --2- 172.21.15.46:0/1604673608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa35c0cf160 0x7fa35c0cf590 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0cfd10 tx=0x7fa35401f1c0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa36331d640 1 -- 172.21.15.46:0/1604673608 shutdown_connections 2023-10-30T14:08:27.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa36331d640 1 --2- 172.21.15.46:0/1604673608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa35c0cf160 0x7fa35c0cf590 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa36331d640 1 --2- 172.21.15.46:0/1604673608 >> v2:172.21.15.154:3300/0 conn(0x7fa35c1538a0 0x7fa35c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa36331d640 1 -- 172.21.15.46:0/1604673608 >> 172.21.15.46:0/1604673608 conn(0x7fa35c06bfe0 msgr2=0x7fa35c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:27.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa36331d640 1 -- 172.21.15.46:0/1604673608 shutdown_connections 2023-10-30T14:08:27.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa36331d640 1 -- 172.21.15.46:0/1604673608 wait complete. 2023-10-30T14:08:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa36331d640 1 Processor -- start 2023-10-30T14:08:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.822+0000 7fa36331d640 1 -- start start 2023-10-30T14:08:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa36331d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa35c0ca0e0 0x7fa35c0b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:27.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa36331d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa35c0cf160 0x7fa35c0ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa36331d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa35c1538a0 0x7fa35c0b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa36331d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c0c5a00 con 0x7fa35c0cf160 2023-10-30T14:08:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa36331d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c158200 con 0x7fa35c0ca0e0 2023-10-30T14:08:27.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa36331d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c157470 con 0x7fa35c1538a0 2023-10-30T14:08:27.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa3608b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa35c0cf160 0x7fa35c0ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:27.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa3608b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa35c0cf160 0x7fa35c0ce650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40780/0 (socket says 172.21.15.46:40780) 2023-10-30T14:08:27.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa3608b8640 1 -- 172.21.15.46:0/74417247 learned_addr learned my addr 172.21.15.46:0/74417247 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:27.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa3610b9640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa35c0ca0e0 0x7fa35c0b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:27.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa3618ba640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa35c1538a0 0x7fa35c0b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa3608b8640 1 -- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa35c1538a0 msgr2=0x7fa35c0b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa3608b8640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa35c1538a0 0x7fa35c0b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa3608b8640 1 -- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa35c0ca0e0 msgr2=0x7fa35c0b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa3608b8640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa35c0ca0e0 0x7fa35c0b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa3608b8640 1 -- 172.21.15.46:0/74417247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa354009b30 con 0x7fa35c0cf160 2023-10-30T14:08:27.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa3608b8640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa35c0cf160 0x7fa35c0ce650 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fa34c00c480 tx=0x7fa34c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa351ffb640 1 -- 172.21.15.46:0/74417247 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c0074e0 con 0x7fa35c0cf160 2023-10-30T14:08:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa351ffb640 1 -- 172.21.15.46:0/74417247 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34c007660 con 0x7fa35c0cf160 2023-10-30T14:08:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa351ffb640 1 -- 172.21.15.46:0/74417247 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c004a30 con 0x7fa35c0cf160 2023-10-30T14:08:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa36331d640 1 -- 172.21.15.46:0/74417247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa35c1efb50 con 0x7fa35c0cf160 2023-10-30T14:08:27.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa36331d640 1 -- 172.21.15.46:0/74417247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35c1eed90 con 0x7fa35c0cf160 2023-10-30T14:08:27.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa36331d640 1 -- 172.21.15.46:0/74417247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa35c0b2a50 con 0x7fa35c0cf160 2023-10-30T14:08:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.826+0000 7fa351ffb640 1 -- 172.21.15.46:0/74417247 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa34c004b90 con 0x7fa35c0cf160 2023-10-30T14:08:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.830+0000 7fa351ffb640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3400664a0 0x7fa340068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.830+0000 7fa351ffb640 1 -- 172.21.15.46:0/74417247 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7fa34c083880 con 0x7fa35c0cf160 2023-10-30T14:08:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.830+0000 7fa3610b9640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3400664a0 0x7fa340068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:27.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.830+0000 7fa3610b9640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3400664a0 0x7fa340068960 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa358007a60 tx=0x7fa358008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:27.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:27.834+0000 7fa351ffb640 1 -- 172.21.15.46:0/74417247 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa34c053f50 con 0x7fa35c0cf160 2023-10-30T14:08:28.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.022+0000 7fa36331d640 1 -- 172.21.15.46:0/74417247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa35c04e170 con 0x7fa35c0cf160 2023-10-30T14:08:28.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa351ffb640 1 -- 172.21.15.46:0/74417247 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fa34c053ae0 con 0x7fa35c0cf160 2023-10-30T14:08:28.028 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:28.028 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8655,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":194}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:52.610131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:28.029 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 195 2023-10-30T14:08:28.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa36331d640 1 -- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3400664a0 msgr2=0x7fa340068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:28.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa36331d640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3400664a0 0x7fa340068960 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa358007a60 tx=0x7fa358008040 comp rx=0 tx=0).stop 2023-10-30T14:08:28.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa36331d640 1 -- 172.21.15.46:0/74417247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa35c0cf160 msgr2=0x7fa35c0ce650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:28.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa36331d640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa35c0cf160 0x7fa35c0ce650 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fa34c00c480 tx=0x7fa34c00c940 comp rx=0 tx=0).stop 2023-10-30T14:08:28.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa36331d640 1 -- 172.21.15.46:0/74417247 shutdown_connections 2023-10-30T14:08:28.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa36331d640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa3400664a0 0x7fa340068960 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:28.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa36331d640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa35c1538a0 0x7fa35c0b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:28.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa36331d640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa35c0cf160 0x7fa35c0ce650 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:28.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa36331d640 1 --2- 172.21.15.46:0/74417247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa35c0ca0e0 0x7fa35c0b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa36331d640 1 -- 172.21.15.46:0/74417247 >> 172.21.15.46:0/74417247 conn(0x7fa35c06bfe0 msgr2=0x7fa35c066310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa36331d640 1 -- 172.21.15.46:0/74417247 shutdown_connections 2023-10-30T14:08:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:28.026+0000 7fa36331d640 1 -- 172.21.15.46:0/74417247 wait complete. 2023-10-30T14:08:28.049 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:08:30.051 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.202+0000 7f4f5bcea640 1 Processor -- start 2023-10-30T14:08:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.202+0000 7f4f5bcea640 1 -- start start 2023-10-30T14:08:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f5bcea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4f540bfec0 0x7f4f540c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f5bcea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f5406a720 0x7f4f5406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f5bcea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f54071290 con 0x7f4f54159e60 2023-10-30T14:08:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f5bcea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f54071570 con 0x7f4f5406a720 2023-10-30T14:08:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f5bcea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f54071850 con 0x7f4f540bfec0 2023-10-30T14:08:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f5a287640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f5406a720 0x7f4f5406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f5a287640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f5406a720 0x7f4f5406abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40792/0 (socket says 172.21.15.46:40792) 2023-10-30T14:08:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f5a287640 1 -- 172.21.15.46:0/1063973709 learned_addr learned my addr 172.21.15.46:0/1063973709 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f59285640 1 --2- 172.21.15.46:0/1063973709 >> v2:172.21.15.154:3300/0 conn(0x7f4f540bfec0 0x7f4f540c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f5a287640 1 -- 172.21.15.46:0/1063973709 >> v2:172.21.15.154:3300/0 conn(0x7f4f540bfec0 msgr2=0x7f4f540c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f5a287640 1 --2- 172.21.15.46:0/1063973709 >> v2:172.21.15.154:3300/0 conn(0x7f4f540bfec0 0x7f4f540c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:30.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f5a287640 1 -- 172.21.15.46:0/1063973709 >> v1:172.21.15.154:6789/0 conn(0x7f4f54159e60 legacy=0x7f4f5406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:30.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.206+0000 7f4f5a287640 1 -- 172.21.15.46:0/1063973709 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f54071b30 con 0x7f4f5406a720 2023-10-30T14:08:30.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5a287640 1 --2- 172.21.15.46:0/1063973709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5406a720 0x7f4f5406abb0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f4f5406f200 tx=0x7f4f4c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b303ad78a4a2b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:30.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f58a84640 1 -- 172.21.15.46:0/1063973709 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c0af400 con 0x7f4f5406a720 2023-10-30T14:08:30.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f58a84640 1 -- 172.21.15.46:0/1063973709 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c0ab050 con 0x7f4f5406a720 2023-10-30T14:08:30.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f58a84640 1 -- 172.21.15.46:0/1063973709 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c0af780 con 0x7f4f5406a720 2023-10-30T14:08:30.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 -- 172.21.15.46:0/1063973709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5406a720 msgr2=0x7f4f5406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:30.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 --2- 172.21.15.46:0/1063973709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5406a720 0x7f4f5406abb0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f4f5406f200 tx=0x7f4f4c01f1c0 comp rx=0 tx=0).stop 2023-10-30T14:08:30.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 -- 172.21.15.46:0/1063973709 shutdown_connections 2023-10-30T14:08:30.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 --2- 172.21.15.46:0/1063973709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f5406a720 0x7f4f5406abb0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:30.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 --2- 172.21.15.46:0/1063973709 >> v2:172.21.15.154:3300/0 conn(0x7f4f540bfec0 0x7f4f540c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:30.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 -- 172.21.15.46:0/1063973709 >> 172.21.15.46:0/1063973709 conn(0x7f4f540c7910 msgr2=0x7f4f540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:30.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 -- 172.21.15.46:0/1063973709 shutdown_connections 2023-10-30T14:08:30.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 -- 172.21.15.46:0/1063973709 wait complete. 2023-10-30T14:08:30.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 Processor -- start 2023-10-30T14:08:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 -- start start 2023-10-30T14:08:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f5406a720 0x7f4f540c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f540bfec0 0x7f4f540ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54159e60 0x7f4f541f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:30.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f54159570 con 0x7f4f54159e60 2023-10-30T14:08:30.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f541592b0 con 0x7f4f5406a720 2023-10-30T14:08:30.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.210+0000 7f4f5bcea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f541e25d0 con 0x7f4f540bfec0 2023-10-30T14:08:30.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f5a287640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54159e60 0x7f4f541f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:30.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f5a287640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54159e60 0x7f4f541f1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40800/0 (socket says 172.21.15.46:40800) 2023-10-30T14:08:30.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f5a287640 1 -- 172.21.15.46:0/3473587046 learned_addr learned my addr 172.21.15.46:0/3473587046 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:30.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f59285640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f540bfec0 0x7f4f540ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:30.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f59a86640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f5406a720 0x7f4f540c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:30.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f5a287640 1 -- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f540bfec0 msgr2=0x7f4f540ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f5a287640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f540bfec0 0x7f4f540ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f5a287640 1 -- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f5406a720 msgr2=0x7f4f540c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f5a287640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f5406a720 0x7f4f540c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f5a287640 1 -- 172.21.15.46:0/3473587046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c009b30 con 0x7f4f54159e60 2023-10-30T14:08:30.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f59a86640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f5406a720 0x7f4f540c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:08:30.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f5a287640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54159e60 0x7f4f541f1ff0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c0b2040 tx=0x7f4f4c01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:30.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f4a7fc640 1 -- 172.21.15.46:0/3473587046 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c0afd30 con 0x7f4f54159e60 2023-10-30T14:08:30.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f4a7fc640 1 -- 172.21.15.46:0/3473587046 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c009510 con 0x7f4f54159e60 2023-10-30T14:08:30.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f4a7fc640 1 -- 172.21.15.46:0/3473587046 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c0ba660 con 0x7f4f54159e60 2023-10-30T14:08:30.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f5bcea640 1 -- 172.21.15.46:0/3473587046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f541e28b0 con 0x7f4f54159e60 2023-10-30T14:08:30.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f5bcea640 1 -- 172.21.15.46:0/3473587046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f541e2e70 con 0x7f4f54159e60 2023-10-30T14:08:30.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f5bcea640 1 -- 172.21.15.46:0/3473587046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f541e3060 con 0x7f4f54159e60 2023-10-30T14:08:30.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f4a7fc640 1 -- 172.21.15.46:0/3473587046 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4f4c0b6040 con 0x7f4f54159e60 2023-10-30T14:08:30.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f4a7fc640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f380664a0 0x7f4f38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:30.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.214+0000 7f4f4a7fc640 1 -- 172.21.15.46:0/3473587046 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f4f4c122870 con 0x7f4f54159e60 2023-10-30T14:08:30.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.218+0000 7f4f59a86640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f380664a0 0x7f4f38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:30.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.218+0000 7f4f59a86640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f380664a0 0x7f4f38068960 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4f50000f70 tx=0x7f4f50009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:30.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.222+0000 7f4f4a7fc640 1 -- 172.21.15.46:0/3473587046 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4f4c0ef4c0 con 0x7f4f54159e60 2023-10-30T14:08:30.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 -- 172.21.15.46:0/3473587046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f5404e170 con 0x7f4f54159e60 2023-10-30T14:08:30.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f4a7fc640 1 -- 172.21.15.46:0/3473587046 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f4f4c0f2ae0 con 0x7f4f54159e60 2023-10-30T14:08:30.413 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:30.413 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8655,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":194}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:52.610131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 195 2023-10-30T14:08:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 -- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f380664a0 msgr2=0x7f4f38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f380664a0 0x7f4f38068960 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4f50000f70 tx=0x7f4f50009040 comp rx=0 tx=0).stop 2023-10-30T14:08:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 -- 172.21.15.46:0/3473587046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54159e60 msgr2=0x7f4f541f1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54159e60 0x7f4f541f1ff0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c0b2040 tx=0x7f4f4c01bf60 comp rx=0 tx=0).stop 2023-10-30T14:08:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 -- 172.21.15.46:0/3473587046 shutdown_connections 2023-10-30T14:08:30.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4f380664a0 0x7f4f38068960 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:30.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54159e60 0x7f4f541f1ff0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:30.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f540bfec0 0x7f4f540ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:30.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 --2- 172.21.15.46:0/3473587046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f5406a720 0x7f4f540c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:30.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 -- 172.21.15.46:0/3473587046 >> 172.21.15.46:0/3473587046 conn(0x7f4f540c7910 msgr2=0x7f4f541e5930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:30.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 -- 172.21.15.46:0/3473587046 shutdown_connections 2023-10-30T14:08:30.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:30.410+0000 7f4f5bcea640 1 -- 172.21.15.46:0/3473587046 wait complete. 2023-10-30T14:08:30.434 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:08:32.435 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:32.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.586+0000 7feb76446640 1 Processor -- start 2023-10-30T14:08:32.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.586+0000 7feb76446640 1 -- start start 2023-10-30T14:08:32.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb76446640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feb700c6b80 0x7feb700bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:32.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb76446640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb7013f4a0 0x7feb700c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:32.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb76446640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb700c0a50 con 0x7feb700c6f60 2023-10-30T14:08:32.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb76446640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb7006f6e0 con 0x7feb7013f4a0 2023-10-30T14:08:32.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb76446640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb7006f9c0 con 0x7feb700c6b80 2023-10-30T14:08:32.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb749e3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb7013f4a0 0x7feb700c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:32.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb749e3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb7013f4a0 0x7feb700c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40806/0 (socket says 172.21.15.46:40806) 2023-10-30T14:08:32.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb749e3640 1 -- 172.21.15.46:0/1658615565 learned_addr learned my addr 172.21.15.46:0/1658615565 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:32.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb6ffff640 1 --2- 172.21.15.46:0/1658615565 >> v2:172.21.15.154:3300/0 conn(0x7feb700c6b80 0x7feb700bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:32.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb749e3640 1 -- 172.21.15.46:0/1658615565 >> v2:172.21.15.154:3300/0 conn(0x7feb700c6b80 msgr2=0x7feb700bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:32.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb749e3640 1 --2- 172.21.15.46:0/1658615565 >> v2:172.21.15.154:3300/0 conn(0x7feb700c6b80 0x7feb700bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:32.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb749e3640 1 -- 172.21.15.46:0/1658615565 >> v1:172.21.15.154:6789/0 conn(0x7feb700c6f60 legacy=0x7feb7013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:32.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.590+0000 7feb749e3640 1 -- 172.21.15.46:0/1658615565 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7006fca0 con 0x7feb7013f4a0 2023-10-30T14:08:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb749e3640 1 --2- 172.21.15.46:0/1658615565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb7013f4a0 0x7feb700c0400 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7feb7006f1b0 tx=0x7feb6801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112169c90ccac253 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb6effd640 1 -- 172.21.15.46:0/1658615565 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb680a57a0 con 0x7feb7013f4a0 2023-10-30T14:08:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb6effd640 1 -- 172.21.15.46:0/1658615565 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb680a5900 con 0x7feb7013f4a0 2023-10-30T14:08:32.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb6effd640 1 -- 172.21.15.46:0/1658615565 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb680a5c90 con 0x7feb7013f4a0 2023-10-30T14:08:32.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb76446640 1 -- 172.21.15.46:0/1658615565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb7013f4a0 msgr2=0x7feb700c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:32.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb76446640 1 --2- 172.21.15.46:0/1658615565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb7013f4a0 0x7feb700c0400 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7feb7006f1b0 tx=0x7feb6801c9d0 comp rx=0 tx=0).stop 2023-10-30T14:08:32.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb76446640 1 -- 172.21.15.46:0/1658615565 shutdown_connections 2023-10-30T14:08:32.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb76446640 1 --2- 172.21.15.46:0/1658615565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb7013f4a0 0x7feb700c0400 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:32.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb76446640 1 --2- 172.21.15.46:0/1658615565 >> v2:172.21.15.154:3300/0 conn(0x7feb700c6b80 0x7feb700bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:32.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb76446640 1 -- 172.21.15.46:0/1658615565 >> 172.21.15.46:0/1658615565 conn(0x7feb700c7910 msgr2=0x7feb700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:32.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb76446640 1 -- 172.21.15.46:0/1658615565 shutdown_connections 2023-10-30T14:08:32.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb76446640 1 -- 172.21.15.46:0/1658615565 wait complete. 2023-10-30T14:08:32.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb76446640 1 Processor -- start 2023-10-30T14:08:32.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.594+0000 7feb76446640 1 -- start start 2023-10-30T14:08:32.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb76446640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb700c6b80 0x7feb701efa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:32.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb76446640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb700c6f60 0x7feb7014e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:32.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb76446640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb7013f4a0 0x7feb7014ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:32.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb76446640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb70150720 con 0x7feb700c6b80 2023-10-30T14:08:32.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb76446640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7014f500 con 0x7feb700c6f60 2023-10-30T14:08:32.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb76446640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7014f7c0 con 0x7feb7013f4a0 2023-10-30T14:08:32.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb700c6b80 0x7feb701efa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:32.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb700c6b80 0x7feb701efa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40820/0 (socket says 172.21.15.46:40820) 2023-10-30T14:08:32.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6ffff640 1 -- 172.21.15.46:0/3460660170 learned_addr learned my addr 172.21.15.46:0/3460660170 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:32.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6f7fe640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb700c6f60 0x7feb7014e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:32.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb749e3640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb7013f4a0 0x7feb7014ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:32.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6ffff640 1 -- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb7013f4a0 msgr2=0x7feb7014ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:32.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6ffff640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb7013f4a0 0x7feb7014ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:32.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6ffff640 1 -- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb700c6f60 msgr2=0x7feb7014e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:32.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6ffff640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb700c6f60 0x7feb7014e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:32.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6ffff640 1 -- 172.21.15.46:0/3460660170 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb68009b30 con 0x7feb700c6b80 2023-10-30T14:08:32.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6f7fe640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb700c6f60 0x7feb7014e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:08:32.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6ffff640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb700c6b80 0x7feb701efa20 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7feb6000e930 tx=0x7feb6000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:32.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6cff9640 1 -- 172.21.15.46:0/3460660170 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60009690 con 0x7feb700c6b80 2023-10-30T14:08:32.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6cff9640 1 -- 172.21.15.46:0/3460660170 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6000ca00 con 0x7feb700c6b80 2023-10-30T14:08:32.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb6cff9640 1 -- 172.21.15.46:0/3460660170 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6000cd70 con 0x7feb700c6b80 2023-10-30T14:08:32.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb76446640 1 -- 172.21.15.46:0/3460660170 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7014faa0 con 0x7feb700c6b80 2023-10-30T14:08:32.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb76446640 1 -- 172.21.15.46:0/3460660170 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb70150060 con 0x7feb700c6b80 2023-10-30T14:08:32.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.598+0000 7feb76446640 1 -- 172.21.15.46:0/3460660170 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7013c5f0 con 0x7feb700c6b80 2023-10-30T14:08:32.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.602+0000 7feb6cff9640 1 -- 172.21.15.46:0/3460660170 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feb60013470 con 0x7feb700c6b80 2023-10-30T14:08:32.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.602+0000 7feb6cff9640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb5c0664a0 0x7feb5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:32.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.602+0000 7feb6cff9640 1 -- 172.21.15.46:0/3460660170 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7feb600849a0 con 0x7feb700c6b80 2023-10-30T14:08:32.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.602+0000 7feb6f7fe640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb5c0664a0 0x7feb5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:32.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.602+0000 7feb6f7fe640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb5c0664a0 0x7feb5c068960 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7feb7013b270 tx=0x7feb64009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:32.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.606+0000 7feb6cff9640 1 -- 172.21.15.46:0/3460660170 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feb600515f0 con 0x7feb700c6b80 2023-10-30T14:08:32.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.794+0000 7feb76446640 1 -- 172.21.15.46:0/3460660170 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb7004e170 con 0x7feb700c6b80 2023-10-30T14:08:32.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.794+0000 7feb6cff9640 1 -- 172.21.15.46:0/3460660170 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+6523 (secure 0 0 0) 0x7feb60021090 con 0x7feb700c6b80 2023-10-30T14:08:32.798 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:32.798 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8655,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":194}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:52.610131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:32.799 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 195 2023-10-30T14:08:32.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.798+0000 7feb76446640 1 -- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb5c0664a0 msgr2=0x7feb5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:32.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.798+0000 7feb76446640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb5c0664a0 0x7feb5c068960 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7feb7013b270 tx=0x7feb64009040 comp rx=0 tx=0).stop 2023-10-30T14:08:32.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.798+0000 7feb76446640 1 -- 172.21.15.46:0/3460660170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb700c6b80 msgr2=0x7feb701efa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:32.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.798+0000 7feb76446640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb700c6b80 0x7feb701efa20 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7feb6000e930 tx=0x7feb6000edf0 comp rx=0 tx=0).stop 2023-10-30T14:08:32.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.798+0000 7feb76446640 1 -- 172.21.15.46:0/3460660170 shutdown_connections 2023-10-30T14:08:32.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.798+0000 7feb76446640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb5c0664a0 0x7feb5c068960 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:32.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.798+0000 7feb76446640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb7013f4a0 0x7feb7014ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:32.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.798+0000 7feb76446640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb700c6f60 0x7feb7014e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:32.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.798+0000 7feb76446640 1 --2- 172.21.15.46:0/3460660170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb700c6b80 0x7feb701efa20 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:32.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.798+0000 7feb76446640 1 -- 172.21.15.46:0/3460660170 >> 172.21.15.46:0/3460660170 conn(0x7feb700c7910 msgr2=0x7feb70136fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:32.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.798+0000 7feb76446640 1 -- 172.21.15.46:0/3460660170 shutdown_connections 2023-10-30T14:08:32.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:32.798+0000 7feb76446640 1 -- 172.21.15.46:0/3460660170 wait complete. 2023-10-30T14:08:32.820 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:08:34.821 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:34.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a9dd9d640 1 Processor -- start 2023-10-30T14:08:34.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a9dd9d640 1 -- start start 2023-10-30T14:08:34.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a9dd9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a9814fd50 0x7f1a980c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:34.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a9dd9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a980cb550 0x7f1a980cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:34.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a9dd9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a98071250 0x7f1a980716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:34.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a9dd9d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a98071be0 con 0x7f1a9814fd50 2023-10-30T14:08:34.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a9dd9d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a981560a0 con 0x7f1a98071250 2023-10-30T14:08:34.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a9dd9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a98156380 con 0x7f1a980cb550 2023-10-30T14:08:34.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a977fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a9814fd50 0x7f1a980c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:34.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a977fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a9814fd50 0x7f1a980c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40830/0 (socket says 172.21.15.46:40830) 2023-10-30T14:08:34.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a977fe640 1 -- 172.21.15.46:0/1637084983 learned_addr learned my addr 172.21.15.46:0/1637084983 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:34.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a97fff640 1 --2- 172.21.15.46:0/1637084983 >> v2:172.21.15.154:3300/0 conn(0x7f1a98071250 0x7f1a980716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:34.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a96ffd640 1 --2- 172.21.15.46:0/1637084983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a980cb550 0x7f1a980cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:34.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a977fe640 1 -- 172.21.15.46:0/1637084983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a980cb550 msgr2=0x7f1a980cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:34.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a977fe640 1 --2- 172.21.15.46:0/1637084983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a980cb550 0x7f1a980cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:34.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a977fe640 1 -- 172.21.15.46:0/1637084983 >> v2:172.21.15.154:3300/0 conn(0x7f1a98071250 msgr2=0x7f1a980716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:34.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a977fe640 1 --2- 172.21.15.46:0/1637084983 >> v2:172.21.15.154:3300/0 conn(0x7f1a98071250 0x7f1a980716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:34.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.970+0000 7f1a977fe640 1 -- 172.21.15.46:0/1637084983 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a98156660 con 0x7f1a9814fd50 2023-10-30T14:08:34.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a977fe640 1 --2- 172.21.15.46:0/1637084983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a9814fd50 0x7f1a980c65d0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f1a981538d0 tx=0x7f1a8801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3913100d40ec4bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:34.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a967fc640 1 -- 172.21.15.46:0/1637084983 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a880ab610 con 0x7f1a9814fd50 2023-10-30T14:08:34.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a967fc640 1 -- 172.21.15.46:0/1637084983 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a880ab770 con 0x7f1a9814fd50 2023-10-30T14:08:34.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a967fc640 1 -- 172.21.15.46:0/1637084983 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a880abb00 con 0x7f1a9814fd50 2023-10-30T14:08:34.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/1637084983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a9814fd50 msgr2=0x7f1a980c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:34.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a9dd9d640 1 --2- 172.21.15.46:0/1637084983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a9814fd50 0x7f1a980c65d0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f1a981538d0 tx=0x7f1a8801c4d0 comp rx=0 tx=0).stop 2023-10-30T14:08:34.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/1637084983 shutdown_connections 2023-10-30T14:08:34.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a9dd9d640 1 --2- 172.21.15.46:0/1637084983 >> v2:172.21.15.154:3300/0 conn(0x7f1a98071250 0x7f1a980716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:34.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a9dd9d640 1 --2- 172.21.15.46:0/1637084983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a980cb550 0x7f1a980cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:34.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a9dd9d640 1 --2- 172.21.15.46:0/1637084983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a9814fd50 0x7f1a980c65d0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:34.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/1637084983 >> 172.21.15.46:0/1637084983 conn(0x7f1a9806c6e0 msgr2=0x7f1a98069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:34.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/1637084983 shutdown_connections 2023-10-30T14:08:34.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.974+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/1637084983 wait complete. 2023-10-30T14:08:34.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a9dd9d640 1 Processor -- start 2023-10-30T14:08:34.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a9dd9d640 1 -- start start 2023-10-30T14:08:34.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a9dd9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a98071250 0x7f1a980c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:34.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a9dd9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a980cb550 0x7f1a980c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:34.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a9dd9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a9814fd50 0x7f1a98161820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:34.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a9dd9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a980c38f0 con 0x7f1a98071250 2023-10-30T14:08:34.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a9dd9d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a98142630 con 0x7f1a980cb550 2023-10-30T14:08:34.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a9dd9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a98162500 con 0x7f1a9814fd50 2023-10-30T14:08:34.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a977fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a98071250 0x7f1a980c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:34.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a977fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a98071250 0x7f1a980c42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40840/0 (socket says 172.21.15.46:40840) 2023-10-30T14:08:34.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a977fe640 1 -- 172.21.15.46:0/3139386986 learned_addr learned my addr 172.21.15.46:0/3139386986 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:34.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a97fff640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a9814fd50 0x7f1a98161820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:34.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a96ffd640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a980cb550 0x7f1a980c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:34.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a977fe640 1 -- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a9814fd50 msgr2=0x7f1a98161820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:34.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a977fe640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a9814fd50 0x7f1a98161820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:34.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a977fe640 1 -- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a980cb550 msgr2=0x7f1a980c4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:34.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a977fe640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a980cb550 0x7f1a980c4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:34.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a977fe640 1 -- 172.21.15.46:0/3139386986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a880094d0 con 0x7f1a98071250 2023-10-30T14:08:34.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a96ffd640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a980cb550 0x7f1a980c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:08:34.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a977fe640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a98071250 0x7f1a980c42d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f1a8801ab80 tx=0x7f1a8801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:34.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a7ffff640 1 -- 172.21.15.46:0/3139386986 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a880b4800 con 0x7f1a98071250 2023-10-30T14:08:34.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a7ffff640 1 -- 172.21.15.46:0/3139386986 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a880b4980 con 0x7f1a98071250 2023-10-30T14:08:34.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/3139386986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a981627e0 con 0x7f1a98071250 2023-10-30T14:08:34.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a7ffff640 1 -- 172.21.15.46:0/3139386986 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a880b4d30 con 0x7f1a98071250 2023-10-30T14:08:34.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.978+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/3139386986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a980c3db0 con 0x7f1a98071250 2023-10-30T14:08:34.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.982+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/3139386986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a98162a20 con 0x7f1a98071250 2023-10-30T14:08:34.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.982+0000 7f1a7ffff640 1 -- 172.21.15.46:0/3139386986 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1a880b3380 con 0x7f1a98071250 2023-10-30T14:08:34.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.982+0000 7f1a7ffff640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a840664a0 0x7f1a84068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:34.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.982+0000 7f1a7ffff640 1 -- 172.21.15.46:0/3139386986 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f1a8811ed80 con 0x7f1a98071250 2023-10-30T14:08:34.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.982+0000 7f1a96ffd640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a840664a0 0x7f1a84068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:34.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.982+0000 7f1a96ffd640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a840664a0 0x7f1a84068960 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1a980c9bd0 tx=0x7f1a8c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:34.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:34.986+0000 7f1a7ffff640 1 -- 172.21.15.46:0/3139386986 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a880eb930 con 0x7f1a98071250 2023-10-30T14:08:35.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.174+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/3139386986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a980c3fa0 con 0x7f1a98071250 2023-10-30T14:08:35.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a7ffff640 1 -- 172.21.15.46:0/3139386986 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f1a880c4070 con 0x7f1a98071250 2023-10-30T14:08:35.180 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:35.180 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8655,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":194}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:52.610131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:35.180 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 195 2023-10-30T14:08:35.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a840664a0 msgr2=0x7f1a84068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:35.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a9dd9d640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a840664a0 0x7f1a84068960 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1a980c9bd0 tx=0x7f1a8c008040 comp rx=0 tx=0).stop 2023-10-30T14:08:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/3139386986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a98071250 msgr2=0x7f1a980c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a9dd9d640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a98071250 0x7f1a980c42d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f1a8801ab80 tx=0x7f1a8801ad10 comp rx=0 tx=0).stop 2023-10-30T14:08:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/3139386986 shutdown_connections 2023-10-30T14:08:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a9dd9d640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a840664a0 0x7f1a84068960 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a9dd9d640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a9814fd50 0x7f1a98161820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a9dd9d640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a980cb550 0x7f1a980c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a9dd9d640 1 --2- 172.21.15.46:0/3139386986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a98071250 0x7f1a980c42d0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/3139386986 >> 172.21.15.46:0/3139386986 conn(0x7f1a9806c6e0 msgr2=0x7f1a980c9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/3139386986 shutdown_connections 2023-10-30T14:08:35.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:35.178+0000 7f1a9dd9d640 1 -- 172.21.15.46:0/3139386986 wait complete. 2023-10-30T14:08:35.201 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:08:37.203 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:37.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.354+0000 7f2560e09640 1 Processor -- start 2023-10-30T14:08:37.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.354+0000 7f2560e09640 1 -- start start 2023-10-30T14:08:37.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.354+0000 7f2560e09640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f255c070740 0x7f255c0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:37.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.358+0000 7f2560e09640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f255c0c5810 con 0x7f255c15e700 2023-10-30T14:08:37.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.358+0000 7f2560e09640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f255c15eb70 con 0x7f255c1580e0 2023-10-30T14:08:37.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.358+0000 7f2560e09640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f255c06acb0 con 0x7f255c070740 2023-10-30T14:08:37.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.358+0000 7f2553fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f255c070740 0x7f255c0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:37.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.358+0000 7f255ad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f255c15e700 0x7f255c06a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52590/0 (socket says 172.21.15.46:52590) 2023-10-30T14:08:37.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.358+0000 7f255ad9d640 1 -- 172.21.15.46:0/993065511 learned_addr learned my addr 172.21.15.46:0/993065511 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:08:37.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.358+0000 7f2559d9b640 1 -- 172.21.15.46:0/993065511 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959616964 0 0) 0x7f255c0c5810 con 0x7f255c15e700 2023-10-30T14:08:37.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.358+0000 7f2553fff640 1 -- 172.21.15.46:0/993065511 >> v1:172.21.15.154:6789/0 conn(0x7f255c1580e0 legacy=0x7f255c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:37.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.358+0000 7f2553fff640 1 -- 172.21.15.46:0/993065511 >> v1:172.21.15.46:6789/0 conn(0x7f255c15e700 legacy=0x7f255c06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:37.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.358+0000 7f2553fff640 1 -- 172.21.15.46:0/993065511 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c0c6bb0 con 0x7f255c070740 2023-10-30T14:08:37.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.358+0000 7f2553fff640 1 --2- 172.21.15.46:0/993065511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f255c070740 0x7f255c0c6540 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f25440097c0 tx=0x7f254401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57bcfbeb11b3f80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:37.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.358+0000 7f2559d9b640 1 -- 172.21.15.46:0/993065511 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25440ab600 con 0x7f255c070740 2023-10-30T14:08:37.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2559d9b640 1 -- 172.21.15.46:0/993065511 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25440a7050 con 0x7f255c070740 2023-10-30T14:08:37.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2559d9b640 1 -- 172.21.15.46:0/993065511 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25440ab980 con 0x7f255c070740 2023-10-30T14:08:37.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 -- 172.21.15.46:0/993065511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f255c070740 msgr2=0x7f255c0c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:37.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 --2- 172.21.15.46:0/993065511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f255c070740 0x7f255c0c6540 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f25440097c0 tx=0x7f254401c480 comp rx=0 tx=0).stop 2023-10-30T14:08:37.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 -- 172.21.15.46:0/993065511 shutdown_connections 2023-10-30T14:08:37.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 --2- 172.21.15.46:0/993065511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f255c070740 0x7f255c0c6540 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:37.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 -- 172.21.15.46:0/993065511 >> 172.21.15.46:0/993065511 conn(0x7f255c0c7910 msgr2=0x7f255c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:37.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 -- 172.21.15.46:0/993065511 shutdown_connections 2023-10-30T14:08:37.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 -- 172.21.15.46:0/993065511 wait complete. 2023-10-30T14:08:37.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 Processor -- start 2023-10-30T14:08:37.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 -- start start 2023-10-30T14:08:37.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f255c070740 0x7f255c0c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:37.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f255c1580e0 0x7f255c1537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:37.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f255c15e700 0x7f255c153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:37.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255c153310 con 0x7f255c1580e0 2023-10-30T14:08:37.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255c1490d0 con 0x7f255c15e700 2023-10-30T14:08:37.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2560e09640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f255c143ad0 con 0x7f255c070740 2023-10-30T14:08:37.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2553fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f255c1580e0 0x7f255c1537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:37.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2553fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f255c1580e0 0x7f255c1537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34956/0 (socket says 172.21.15.46:34956) 2023-10-30T14:08:37.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f2553fff640 1 -- 172.21.15.46:0/3959695905 learned_addr learned my addr 172.21.15.46:0/3959695905 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:37.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f255a59c640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f255c070740 0x7f255c0c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:37.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.362+0000 7f255ad9d640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f255c15e700 0x7f255c153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:37.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2553fff640 1 -- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f255c070740 msgr2=0x7f255c0c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:37.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2553fff640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f255c070740 0x7f255c0c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:37.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2553fff640 1 -- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f255c15e700 msgr2=0x7f255c153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:37.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2553fff640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f255c15e700 0x7f255c153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:37.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2553fff640 1 -- 172.21.15.46:0/3959695905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2544009420 con 0x7f255c1580e0 2023-10-30T14:08:37.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2553fff640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f255c1580e0 0x7f255c1537a0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f2544019090 tx=0x7f2544019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:37.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2552ffd640 1 -- 172.21.15.46:0/3959695905 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25440b4610 con 0x7f255c1580e0 2023-10-30T14:08:37.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2552ffd640 1 -- 172.21.15.46:0/3959695905 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25440abe90 con 0x7f255c1580e0 2023-10-30T14:08:37.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2560e09640 1 -- 172.21.15.46:0/3959695905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f255c149390 con 0x7f255c1580e0 2023-10-30T14:08:37.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2552ffd640 1 -- 172.21.15.46:0/3959695905 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25440b49d0 con 0x7f255c1580e0 2023-10-30T14:08:37.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2560e09640 1 -- 172.21.15.46:0/3959695905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f255c152fa0 con 0x7f255c1580e0 2023-10-30T14:08:37.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2560e09640 1 -- 172.21.15.46:0/3959695905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f255c04e170 con 0x7f255c1580e0 2023-10-30T14:08:37.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2552ffd640 1 -- 172.21.15.46:0/3959695905 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25440b2040 con 0x7f255c1580e0 2023-10-30T14:08:37.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2552ffd640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f253c0664a0 0x7f253c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:37.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.366+0000 7f2552ffd640 1 -- 172.21.15.46:0/3959695905 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f254411e7f0 con 0x7f255c1580e0 2023-10-30T14:08:37.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.370+0000 7f255a59c640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f253c0664a0 0x7f253c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:37.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.370+0000 7f255a59c640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f253c0664a0 0x7f253c068960 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f254c000f70 tx=0x7f254c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:37.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.370+0000 7f2552ffd640 1 -- 172.21.15.46:0/3959695905 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25440eb380 con 0x7f255c1580e0 2023-10-30T14:08:37.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.561+0000 7f2560e09640 1 -- 172.21.15.46:0/3959695905 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f255c0c5a30 con 0x7f255c1580e0 2023-10-30T14:08:37.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.561+0000 7f2552ffd640 1 -- 172.21.15.46:0/3959695905 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f25440ee9a0 con 0x7f255c1580e0 2023-10-30T14:08:37.566 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:37.566 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8655,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":194}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:52.610131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:37.568 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 195 2023-10-30T14:08:37.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.565+0000 7f2560e09640 1 -- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f253c0664a0 msgr2=0x7f253c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:37.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.565+0000 7f2560e09640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f253c0664a0 0x7f253c068960 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f254c000f70 tx=0x7f254c009040 comp rx=0 tx=0).stop 2023-10-30T14:08:37.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.565+0000 7f2560e09640 1 -- 172.21.15.46:0/3959695905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f255c1580e0 msgr2=0x7f255c1537a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:37.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.565+0000 7f2560e09640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f255c1580e0 0x7f255c1537a0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f2544019090 tx=0x7f2544019220 comp rx=0 tx=0).stop 2023-10-30T14:08:37.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.565+0000 7f2560e09640 1 -- 172.21.15.46:0/3959695905 shutdown_connections 2023-10-30T14:08:37.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.565+0000 7f2560e09640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f253c0664a0 0x7f253c068960 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:37.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.565+0000 7f2560e09640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f255c15e700 0x7f255c153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:37.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.565+0000 7f2560e09640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f255c1580e0 0x7f255c1537a0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:37.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.565+0000 7f2560e09640 1 --2- 172.21.15.46:0/3959695905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f255c070740 0x7f255c0c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:37.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.565+0000 7f2560e09640 1 -- 172.21.15.46:0/3959695905 >> 172.21.15.46:0/3959695905 conn(0x7f255c0c7910 msgr2=0x7f255c146440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:37.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.565+0000 7f2560e09640 1 -- 172.21.15.46:0/3959695905 shutdown_connections 2023-10-30T14:08:37.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:37.565+0000 7f2560e09640 1 -- 172.21.15.46:0/3959695905 wait complete. 2023-10-30T14:08:37.587 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:08:39.588 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.737+0000 7f48de6d6640 1 Processor -- start 2023-10-30T14:08:39.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.737+0000 7f48de6d6640 1 -- start start 2023-10-30T14:08:39.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.737+0000 7f48de6d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f48d806e470 0x7f48d806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:39.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.737+0000 7f48de6d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48d80c8490 0x7f48d80c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:39.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.737+0000 7f48de6d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d806afa0 con 0x7f48d8153d70 2023-10-30T14:08:39.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48de6d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d806d780 con 0x7f48d80c8490 2023-10-30T14:08:39.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48de6d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d8154150 con 0x7f48d806e470 2023-10-30T14:08:39.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48dcc73640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48d80c8490 0x7f48d80c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48d7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f48d8153d70 0x7f48d80cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52604/0 (socket says 172.21.15.46:52604) 2023-10-30T14:08:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48d7fff640 1 -- 172.21.15.46:0/4050028983 learned_addr learned my addr 172.21.15.46:0/4050028983 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:08:39.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48d77fe640 1 --2- 172.21.15.46:0/4050028983 >> v2:172.21.15.154:3300/0 conn(0x7f48d806e470 0x7f48d806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48d6ffd640 1 -- 172.21.15.46:0/4050028983 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605447513 0 0) 0x7f48d806afa0 con 0x7f48d8153d70 2023-10-30T14:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48dcc73640 1 -- 172.21.15.46:0/4050028983 >> v2:172.21.15.154:3300/0 conn(0x7f48d806e470 msgr2=0x7f48d806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48dcc73640 1 --2- 172.21.15.46:0/4050028983 >> v2:172.21.15.154:3300/0 conn(0x7f48d806e470 0x7f48d806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:39.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48dcc73640 1 -- 172.21.15.46:0/4050028983 >> v1:172.21.15.46:6789/0 conn(0x7f48d8153d70 legacy=0x7f48d80cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48dcc73640 1 -- 172.21.15.46:0/4050028983 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d806e030 con 0x7f48d80c8490 2023-10-30T14:08:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48dcc73640 1 --2- 172.21.15.46:0/4050028983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d80c8490 0x7f48d80c8cc0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f48d8070230 tx=0x7f48d001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88953368d471ad59 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48d6ffd640 1 -- 172.21.15.46:0/4050028983 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d00a59a0 con 0x7f48d80c8490 2023-10-30T14:08:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48d6ffd640 1 -- 172.21.15.46:0/4050028983 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d00a5b00 con 0x7f48d80c8490 2023-10-30T14:08:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.741+0000 7f48d6ffd640 1 -- 172.21.15.46:0/4050028983 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d00a5e90 con 0x7f48d80c8490 2023-10-30T14:08:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 -- 172.21.15.46:0/4050028983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d80c8490 msgr2=0x7f48d80c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 --2- 172.21.15.46:0/4050028983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d80c8490 0x7f48d80c8cc0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f48d8070230 tx=0x7f48d001c8e0 comp rx=0 tx=0).stop 2023-10-30T14:08:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 -- 172.21.15.46:0/4050028983 shutdown_connections 2023-10-30T14:08:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 --2- 172.21.15.46:0/4050028983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d80c8490 0x7f48d80c8cc0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 --2- 172.21.15.46:0/4050028983 >> v2:172.21.15.154:3300/0 conn(0x7f48d806e470 0x7f48d806ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 -- 172.21.15.46:0/4050028983 >> 172.21.15.46:0/4050028983 conn(0x7f48d8071ea0 msgr2=0x7f48d8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 -- 172.21.15.46:0/4050028983 shutdown_connections 2023-10-30T14:08:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 -- 172.21.15.46:0/4050028983 wait complete. 2023-10-30T14:08:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 Processor -- start 2023-10-30T14:08:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 -- start start 2023-10-30T14:08:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d806e470 0x7f48d8145c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48d80c8490 0x7f48d8139ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d8153d70 0x7f48d80cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8141ef0 con 0x7f48d8153d70 2023-10-30T14:08:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d813a010 con 0x7f48d806e470 2023-10-30T14:08:39.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48de6d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d80cbe10 con 0x7f48d80c8490 2023-10-30T14:08:39.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48dcc73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d8153d70 0x7f48d80cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:39.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48dcc73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d8153d70 0x7f48d80cac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34968/0 (socket says 172.21.15.46:34968) 2023-10-30T14:08:39.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48dcc73640 1 -- 172.21.15.46:0/2716261701 learned_addr learned my addr 172.21.15.46:0/2716261701 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:39.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48d77fe640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48d80c8490 0x7f48d8139ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:39.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48d7fff640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d806e470 0x7f48d8145c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:39.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48dcc73640 1 -- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48d80c8490 msgr2=0x7f48d8139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:39.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.745+0000 7f48dcc73640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48d80c8490 0x7f48d8139ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:39.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48dcc73640 1 -- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d806e470 msgr2=0x7f48d8145c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:39.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48dcc73640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d806e470 0x7f48d8145c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:39.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48dcc73640 1 -- 172.21.15.46:0/2716261701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d0009a70 con 0x7f48d8153d70 2023-10-30T14:08:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48dcc73640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d8153d70 0x7f48d80cac70 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f48d001cdc0 tx=0x7f48d0019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48d4ff9640 1 -- 172.21.15.46:0/2716261701 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d00add70 con 0x7f48d8153d70 2023-10-30T14:08:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48d4ff9640 1 -- 172.21.15.46:0/2716261701 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d0009450 con 0x7f48d8153d70 2023-10-30T14:08:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48d4ff9640 1 -- 172.21.15.46:0/2716261701 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d00b9690 con 0x7f48d8153d70 2023-10-30T14:08:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48de6d6640 1 -- 172.21.15.46:0/2716261701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d8144bc0 con 0x7f48d8153d70 2023-10-30T14:08:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48de6d6640 1 -- 172.21.15.46:0/2716261701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d8145050 con 0x7f48d8153d70 2023-10-30T14:08:39.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48de6d6640 1 -- 172.21.15.46:0/2716261701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d8144d50 con 0x7f48d8153d70 2023-10-30T14:08:39.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48d4ff9640 1 -- 172.21.15.46:0/2716261701 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f48d00b97f0 con 0x7f48d8153d70 2023-10-30T14:08:39.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48d4ff9640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48c40664d0 0x7f48c4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:39.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.749+0000 7f48d4ff9640 1 -- 172.21.15.46:0/2716261701 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f48d011f790 con 0x7f48d8153d70 2023-10-30T14:08:39.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.753+0000 7f48d7fff640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48c40664d0 0x7f48c4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:39.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.753+0000 7f48d7fff640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48c40664d0 0x7f48c4068990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f48c8000f70 tx=0x7f48c8007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:39.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.757+0000 7f48d4ff9640 1 -- 172.21.15.46:0/2716261701 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f48d00ec3c0 con 0x7f48d8153d70 2023-10-30T14:08:39.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.953+0000 7f48de6d6640 1 -- 172.21.15.46:0/2716261701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48d804e170 con 0x7f48d8153d70 2023-10-30T14:08:39.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.953+0000 7f48d4ff9640 1 -- 172.21.15.46:0/2716261701 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f48d00ef9e0 con 0x7f48d8153d70 2023-10-30T14:08:39.958 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:39.959 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8655,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":194}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:52.610131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:39.959 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 195 2023-10-30T14:08:39.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.957+0000 7f48de6d6640 1 -- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48c40664d0 msgr2=0x7f48c4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:39.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.957+0000 7f48de6d6640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48c40664d0 0x7f48c4068990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f48c8000f70 tx=0x7f48c8007b60 comp rx=0 tx=0).stop 2023-10-30T14:08:39.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.957+0000 7f48de6d6640 1 -- 172.21.15.46:0/2716261701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d8153d70 msgr2=0x7f48d80cac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.957+0000 7f48de6d6640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d8153d70 0x7f48d80cac70 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f48d001cdc0 tx=0x7f48d0019680 comp rx=0 tx=0).stop 2023-10-30T14:08:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.957+0000 7f48de6d6640 1 -- 172.21.15.46:0/2716261701 shutdown_connections 2023-10-30T14:08:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.957+0000 7f48de6d6640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48c40664d0 0x7f48c4068990 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.957+0000 7f48de6d6640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d8153d70 0x7f48d80cac70 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.957+0000 7f48de6d6640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48d80c8490 0x7f48d8139ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.957+0000 7f48de6d6640 1 --2- 172.21.15.46:0/2716261701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d806e470 0x7f48d8145c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.957+0000 7f48de6d6640 1 -- 172.21.15.46:0/2716261701 >> 172.21.15.46:0/2716261701 conn(0x7f48d8071ea0 msgr2=0x7f48d80c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.957+0000 7f48de6d6640 1 -- 172.21.15.46:0/2716261701 shutdown_connections 2023-10-30T14:08:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:39.957+0000 7f48de6d6640 1 -- 172.21.15.46:0/2716261701 wait complete. 2023-10-30T14:08:39.980 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:08:41.982 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:42.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7860c9e640 1 Processor -- start 2023-10-30T14:08:42.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7860c9e640 1 -- start start 2023-10-30T14:08:42.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7860c9e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f785c0c6bd0 0x7f785c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7860c9e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f785c071110 0x7f785c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7860c9e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785c066cf0 con 0x7f785c06f7a0 2023-10-30T14:08:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7860c9e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f785c0700c0 con 0x7f785c0c6bd0 2023-10-30T14:08:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7860c9e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785c0c70d0 con 0x7f785c071110 2023-10-30T14:08:42.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7859d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f785c0c6bd0 0x7f785c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7859d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f785c0c6bd0 0x7f785c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34978/0 (socket says 172.21.15.46:34978) 2023-10-30T14:08:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7859d9b640 1 -- 172.21.15.46:0/4086698729 learned_addr learned my addr 172.21.15.46:0/4086698729 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f785a59c640 1 --1- 172.21.15.46:0/4086698729 >> v1:172.21.15.46:6789/0 conn(0x7f785c06f7a0 0x7f785c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52618/0 (socket says 172.21.15.46:52618) 2023-10-30T14:08:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f785ad9d640 1 --2- 172.21.15.46:0/4086698729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f785c071110 0x7f785c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:42.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7859d9b640 1 -- 172.21.15.46:0/4086698729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f785c071110 msgr2=0x7f785c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7859d9b640 1 --2- 172.21.15.46:0/4086698729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f785c071110 0x7f785c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7859d9b640 1 -- 172.21.15.46:0/4086698729 >> v1:172.21.15.46:6789/0 conn(0x7f785c06f7a0 legacy=0x7f785c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f7859d9b640 1 -- 172.21.15.46:0/4086698729 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c0c7390 con 0x7f785c0c6bd0 2023-10-30T14:08:42.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.133+0000 7f785959a640 1 -- 172.21.15.46:0/4086698729 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405563997 0 0) 0x7f785c066cf0 con 0x7f785c06f7a0 2023-10-30T14:08:42.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.137+0000 7f7859d9b640 1 --2- 172.21.15.46:0/4086698729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c0c6bd0 0x7f785c06fb80 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f7850009870 tx=0x7f785001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116bc814b08aa6f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:42.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.137+0000 7f785959a640 1 -- 172.21.15.46:0/4086698729 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78500a8a30 con 0x7f785c0c6bd0 2023-10-30T14:08:42.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.137+0000 7f785959a640 1 -- 172.21.15.46:0/4086698729 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78500a8b90 con 0x7f785c0c6bd0 2023-10-30T14:08:42.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.137+0000 7f785959a640 1 -- 172.21.15.46:0/4086698729 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78500b0440 con 0x7f785c0c6bd0 2023-10-30T14:08:42.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.137+0000 7f7860c9e640 1 -- 172.21.15.46:0/4086698729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c0c6bd0 msgr2=0x7f785c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:42.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.137+0000 7f7860c9e640 1 --2- 172.21.15.46:0/4086698729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c0c6bd0 0x7f785c06fb80 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f7850009870 tx=0x7f785001ed30 comp rx=0 tx=0).stop 2023-10-30T14:08:42.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.137+0000 7f7860c9e640 1 -- 172.21.15.46:0/4086698729 shutdown_connections 2023-10-30T14:08:42.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.137+0000 7f7860c9e640 1 --2- 172.21.15.46:0/4086698729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f785c071110 0x7f785c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:42.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.137+0000 7f7860c9e640 1 --2- 172.21.15.46:0/4086698729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c0c6bd0 0x7f785c06fb80 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:42.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.137+0000 7f7860c9e640 1 -- 172.21.15.46:0/4086698729 >> 172.21.15.46:0/4086698729 conn(0x7f785c0c7910 msgr2=0x7f785c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:42.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.137+0000 7f7860c9e640 1 -- 172.21.15.46:0/4086698729 shutdown_connections 2023-10-30T14:08:42.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.137+0000 7f7860c9e640 1 -- 172.21.15.46:0/4086698729 wait complete. 2023-10-30T14:08:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7860c9e640 1 Processor -- start 2023-10-30T14:08:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7860c9e640 1 -- start start 2023-10-30T14:08:42.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7860c9e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f785c06f7a0 0x7f785c1de910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:42.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7860c9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c071110 0x7f785c1dda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:42.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7860c9e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f785c0c6bd0 0x7f785c14f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:42.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7860c9e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c1dee50 con 0x7f785c071110 2023-10-30T14:08:42.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7860c9e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c1ed130 con 0x7f785c06f7a0 2023-10-30T14:08:42.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7860c9e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785c1df0c0 con 0x7f785c0c6bd0 2023-10-30T14:08:42.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7859d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c071110 0x7f785c1dda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:42.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7859d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c071110 0x7f785c1dda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34990/0 (socket says 172.21.15.46:34990) 2023-10-30T14:08:42.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7859d9b640 1 -- 172.21.15.46:0/830757259 learned_addr learned my addr 172.21.15.46:0/830757259 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:42.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f785ad9d640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f785c0c6bd0 0x7f785c14f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:42.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f785a59c640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f785c06f7a0 0x7f785c1de910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:42.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7859d9b640 1 -- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f785c0c6bd0 msgr2=0x7f785c14f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:42.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7859d9b640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f785c0c6bd0 0x7f785c14f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:42.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7859d9b640 1 -- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f785c06f7a0 msgr2=0x7f785c1de910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:42.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7859d9b640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f785c06f7a0 0x7f785c1de910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:42.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7859d9b640 1 -- 172.21.15.46:0/830757259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78500094d0 con 0x7f785c071110 2023-10-30T14:08:42.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7859d9b640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c071110 0x7f785c1dda20 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f78500b3040 tx=0x7f785001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:42.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f784affd640 1 -- 172.21.15.46:0/830757259 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78500b9620 con 0x7f785c071110 2023-10-30T14:08:42.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f784affd640 1 -- 172.21.15.46:0/830757259 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78500b97c0 con 0x7f785c071110 2023-10-30T14:08:42.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f784affd640 1 -- 172.21.15.46:0/830757259 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78500b9b70 con 0x7f785c071110 2023-10-30T14:08:42.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7860c9e640 1 -- 172.21.15.46:0/830757259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785c1ec370 con 0x7f785c071110 2023-10-30T14:08:42.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.141+0000 7f7860c9e640 1 -- 172.21.15.46:0/830757259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785c1ec930 con 0x7f785c071110 2023-10-30T14:08:42.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.145+0000 7f7860c9e640 1 -- 172.21.15.46:0/830757259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785c04e170 con 0x7f785c071110 2023-10-30T14:08:42.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.145+0000 7f784affd640 1 -- 172.21.15.46:0/830757259 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f78500b7070 con 0x7f785c071110 2023-10-30T14:08:42.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.145+0000 7f784affd640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f783c0664a0 0x7f783c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:42.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.145+0000 7f784affd640 1 -- 172.21.15.46:0/830757259 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f7850122710 con 0x7f785c071110 2023-10-30T14:08:42.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.145+0000 7f785a59c640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f783c0664a0 0x7f783c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:42.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.145+0000 7f785a59c640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f783c0664a0 0x7f783c068960 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f784c000f70 tx=0x7f784c007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:42.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.149+0000 7f784affd640 1 -- 172.21.15.46:0/830757259 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f78500ef210 con 0x7f785c071110 2023-10-30T14:08:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.341+0000 7f7860c9e640 1 -- 172.21.15.46:0/830757259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f785c0c97f0 con 0x7f785c071110 2023-10-30T14:08:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.341+0000 7f784affd640 1 -- 172.21.15.46:0/830757259 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f78500b5070 con 0x7f785c071110 2023-10-30T14:08:42.346 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:42.347 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8655,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":194}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:07:52.610131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8352},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8352":{"gid":8352,"name":"c","rank":1,"incarnation":182,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6836/128865315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":128865315},{"type":"v1","addr":"172.21.15.46:6836","nonce":128865315}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:42.347 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 195 2023-10-30T14:08:42.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.345+0000 7f7860c9e640 1 -- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f783c0664a0 msgr2=0x7f783c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:42.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.345+0000 7f7860c9e640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f783c0664a0 0x7f783c068960 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f784c000f70 tx=0x7f784c007c10 comp rx=0 tx=0).stop 2023-10-30T14:08:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.345+0000 7f7860c9e640 1 -- 172.21.15.46:0/830757259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c071110 msgr2=0x7f785c1dda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.345+0000 7f7860c9e640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c071110 0x7f785c1dda20 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f78500b3040 tx=0x7f785001bad0 comp rx=0 tx=0).stop 2023-10-30T14:08:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.345+0000 7f7860c9e640 1 -- 172.21.15.46:0/830757259 shutdown_connections 2023-10-30T14:08:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.345+0000 7f7860c9e640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f783c0664a0 0x7f783c068960 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.345+0000 7f7860c9e640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f785c0c6bd0 0x7f785c14f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.345+0000 7f7860c9e640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c071110 0x7f785c1dda20 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:42.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.345+0000 7f7860c9e640 1 --2- 172.21.15.46:0/830757259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f785c06f7a0 0x7f785c1de910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:42.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.345+0000 7f7860c9e640 1 -- 172.21.15.46:0/830757259 >> 172.21.15.46:0/830757259 conn(0x7f785c0c7910 msgr2=0x7f785c06a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:42.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.345+0000 7f7860c9e640 1 -- 172.21.15.46:0/830757259 shutdown_connections 2023-10-30T14:08:42.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:42.345+0000 7f7860c9e640 1 -- 172.21.15.46:0/830757259 wait complete. 2023-10-30T14:08:42.368 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:08:44.369 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:44.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4ebf2640 1 Processor -- start 2023-10-30T14:08:44.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4ebf2640 1 -- start start 2023-10-30T14:08:44.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4ebf2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a4815cd50 0x7f5a480c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:44.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4ebf2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5a480cefa0 0x7f5a480cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:44.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4ebf2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4815a6c0 con 0x7f5a481538f0 2023-10-30T14:08:44.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4ebf2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a480cf8f0 con 0x7f5a4815cd50 2023-10-30T14:08:44.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4ebf2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4815d300 con 0x7f5a480cefa0 2023-10-30T14:08:44.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4d3ef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a4815cd50 0x7f5a480c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:44.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4d3ef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a4815cd50 0x7f5a480c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35002/0 (socket says 172.21.15.46:35002) 2023-10-30T14:08:44.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4dbf0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5a481538f0 0x7f5a48159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52620/0 (socket says 172.21.15.46:52620) 2023-10-30T14:08:44.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4d3ef640 1 -- 172.21.15.46:0/99040111 learned_addr learned my addr 172.21.15.46:0/99040111 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:44.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4e3f1640 1 --2- 172.21.15.46:0/99040111 >> v2:172.21.15.154:3300/0 conn(0x7f5a480cefa0 0x7f5a480cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:44.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4d3ef640 1 -- 172.21.15.46:0/99040111 >> v2:172.21.15.154:3300/0 conn(0x7f5a480cefa0 msgr2=0x7f5a480cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:44.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4d3ef640 1 --2- 172.21.15.46:0/99040111 >> v2:172.21.15.154:3300/0 conn(0x7f5a480cefa0 0x7f5a480cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4d3ef640 1 -- 172.21.15.46:0/99040111 >> v1:172.21.15.46:6789/0 conn(0x7f5a481538f0 legacy=0x7f5a48159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:44.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.521+0000 7f5a4d3ef640 1 -- 172.21.15.46:0/99040111 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4815d5e0 con 0x7f5a4815cd50 2023-10-30T14:08:44.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.525+0000 7f5a4d3ef640 1 --2- 172.21.15.46:0/99040111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a4815cd50 0x7f5a480c9f90 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f5a48066a40 tx=0x7f5a3801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02b972b1af86b82 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:44.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.525+0000 7f5a4cbee640 1 -- 172.21.15.46:0/99040111 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a380a89f0 con 0x7f5a4815cd50 2023-10-30T14:08:44.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.525+0000 7f5a4cbee640 1 -- 172.21.15.46:0/99040111 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a380a8b50 con 0x7f5a4815cd50 2023-10-30T14:08:44.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.525+0000 7f5a4cbee640 1 -- 172.21.15.46:0/99040111 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a380b0440 con 0x7f5a4815cd50 2023-10-30T14:08:44.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.525+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/99040111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a4815cd50 msgr2=0x7f5a480c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:44.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.525+0000 7f5a4ebf2640 1 --2- 172.21.15.46:0/99040111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a4815cd50 0x7f5a480c9f90 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f5a48066a40 tx=0x7f5a3801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.525+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/99040111 shutdown_connections 2023-10-30T14:08:44.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.525+0000 7f5a4ebf2640 1 --2- 172.21.15.46:0/99040111 >> v2:172.21.15.154:3300/0 conn(0x7f5a480cefa0 0x7f5a480cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.525+0000 7f5a4ebf2640 1 --2- 172.21.15.46:0/99040111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a4815cd50 0x7f5a480c9f90 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.525+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/99040111 >> 172.21.15.46:0/99040111 conn(0x7f5a4806a640 msgr2=0x7f5a48065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:44.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.525+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/99040111 shutdown_connections 2023-10-30T14:08:44.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.525+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/99040111 wait complete. 2023-10-30T14:08:44.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4ebf2640 1 Processor -- start 2023-10-30T14:08:44.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4ebf2640 1 -- start start 2023-10-30T14:08:44.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4ebf2640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a480cefa0 0x7f5a481453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4ebf2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a481538f0 0x7f5a4815a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4ebf2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a4815cd50 0x7f5a48158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4ebf2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a481f02f0 con 0x7f5a481538f0 2023-10-30T14:08:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4ebf2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a48157a20 con 0x7f5a4815cd50 2023-10-30T14:08:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4ebf2640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a481ef2e0 con 0x7f5a480cefa0 2023-10-30T14:08:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4d3ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a481538f0 0x7f5a4815a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4d3ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a481538f0 0x7f5a4815a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35012/0 (socket says 172.21.15.46:35012) 2023-10-30T14:08:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4d3ef640 1 -- 172.21.15.46:0/3014878815 learned_addr learned my addr 172.21.15.46:0/3014878815 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4dbf0640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a480cefa0 0x7f5a481453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:44.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4e3f1640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a4815cd50 0x7f5a48158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:44.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4d3ef640 1 -- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a480cefa0 msgr2=0x7f5a481453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:44.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4d3ef640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a480cefa0 0x7f5a481453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4d3ef640 1 -- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a4815cd50 msgr2=0x7f5a48158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:44.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4d3ef640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a4815cd50 0x7f5a48158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4d3ef640 1 -- 172.21.15.46:0/3014878815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a380094d0 con 0x7f5a481538f0 2023-10-30T14:08:44.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4e3f1640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a4815cd50 0x7f5a48158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:08:44.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4d3ef640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a481538f0 0x7f5a4815a920 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f5a3801b900 tx=0x7f5a3801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:44.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a3e7fc640 1 -- 172.21.15.46:0/3014878815 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a380ba600 con 0x7f5a481538f0 2023-10-30T14:08:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a3e7fc640 1 -- 172.21.15.46:0/3014878815 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a380ba760 con 0x7f5a481538f0 2023-10-30T14:08:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/3014878815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a481efe60 con 0x7f5a481538f0 2023-10-30T14:08:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a3e7fc640 1 -- 172.21.15.46:0/3014878815 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a380bab10 con 0x7f5a481538f0 2023-10-30T14:08:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/3014878815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a481ef530 con 0x7f5a481538f0 2023-10-30T14:08:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.529+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/3014878815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4804e170 con 0x7f5a481538f0 2023-10-30T14:08:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.533+0000 7f5a3e7fc640 1 -- 172.21.15.46:0/3014878815 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5a380bac70 con 0x7f5a481538f0 2023-10-30T14:08:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.533+0000 7f5a3e7fc640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5a2c0664d0 0x7f5a2c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.533+0000 7f5a3e7fc640 1 -- 172.21.15.46:0/3014878815 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f5a38122c50 con 0x7f5a481538f0 2023-10-30T14:08:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.533+0000 7f5a4dbf0640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5a2c0664d0 0x7f5a2c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.533+0000 7f5a4dbf0640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5a2c0664d0 0x7f5a2c068990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f5a44000f70 tx=0x7f5a44007c70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.537+0000 7f5a3e7fc640 1 -- 172.21.15.46:0/3014878815 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5a380ef900 con 0x7f5a481538f0 2023-10-30T14:08:44.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.729+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/3014878815 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a4815c470 con 0x7f5a481538f0 2023-10-30T14:08:44.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.729+0000 7f5a3e7fc640 1 -- 172.21.15.46:0/3014878815 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f5a380b5020 con 0x7f5a481538f0 2023-10-30T14:08:44.735 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:44.736 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:08:44.543694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:44.736 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 196 2023-10-30T14:08:44.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.733+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5a2c0664d0 msgr2=0x7f5a2c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:44.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.733+0000 7f5a4ebf2640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5a2c0664d0 0x7f5a2c068990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f5a44000f70 tx=0x7f5a44007c70 comp rx=0 tx=0).stop 2023-10-30T14:08:44.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.733+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/3014878815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a481538f0 msgr2=0x7f5a4815a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.733+0000 7f5a4ebf2640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a481538f0 0x7f5a4815a920 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f5a3801b900 tx=0x7f5a3801ba90 comp rx=0 tx=0).stop 2023-10-30T14:08:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.733+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/3014878815 shutdown_connections 2023-10-30T14:08:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.733+0000 7f5a4ebf2640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5a2c0664d0 0x7f5a2c068990 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.733+0000 7f5a4ebf2640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a4815cd50 0x7f5a48158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.733+0000 7f5a4ebf2640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a481538f0 0x7f5a4815a920 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.733+0000 7f5a4ebf2640 1 --2- 172.21.15.46:0/3014878815 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a480cefa0 0x7f5a481453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.733+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/3014878815 >> 172.21.15.46:0/3014878815 conn(0x7f5a4806a640 msgr2=0x7f5a480cd710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.733+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/3014878815 shutdown_connections 2023-10-30T14:08:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.733+0000 7f5a4ebf2640 1 -- 172.21.15.46:0/3014878815 wait complete. 2023-10-30T14:08:44.757 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T14:08:44.757 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:08:44.757 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fd5f3e640 1 Processor -- start 2023-10-30T14:08:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fd5f3e640 1 -- start start 2023-10-30T14:08:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fd5f3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6fd01580e0 0x7f6fd0070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fd5f3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6fd00c6540 0x7f6fd015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fd5f3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00c5810 con 0x7f6fd0069e90 2023-10-30T14:08:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fd5f3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd00c6990 con 0x7f6fd01580e0 2023-10-30T14:08:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fd5f3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd00c6bb0 con 0x7f6fd00c6540 2023-10-30T14:08:44.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fcf7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6fd01580e0 0x7f6fd0070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fcf7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6fd01580e0 0x7f6fd0070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35022/0 (socket says 172.21.15.46:35022) 2023-10-30T14:08:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fcf7fe640 1 -- 172.21.15.46:0/19674329 learned_addr learned my addr 172.21.15.46:0/19674329 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fceffd640 1 --2- 172.21.15.46:0/19674329 >> v2:172.21.15.154:3300/0 conn(0x7f6fd00c6540 0x7f6fd015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fcf7fe640 1 -- 172.21.15.46:0/19674329 >> v2:172.21.15.154:3300/0 conn(0x7f6fd00c6540 msgr2=0x7f6fd015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:44.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fcf7fe640 1 --2- 172.21.15.46:0/19674329 >> v2:172.21.15.154:3300/0 conn(0x7f6fd00c6540 0x7f6fd015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fcf7fe640 1 -- 172.21.15.46:0/19674329 >> v1:172.21.15.154:6789/0 conn(0x7f6fd0069e90 legacy=0x7f6fd006a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.901+0000 7f6fcf7fe640 1 -- 172.21.15.46:0/19674329 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd00c6e70 con 0x7f6fd01580e0 2023-10-30T14:08:44.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.905+0000 7f6fcf7fe640 1 --2- 172.21.15.46:0/19674329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fd01580e0 0x7f6fd0070740 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f6fd006b1a0 tx=0x7f6fc001c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359413a5c2dc5a3e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.905+0000 7f6fce7fc640 1 -- 172.21.15.46:0/19674329 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc00aa620 con 0x7f6fd01580e0 2023-10-30T14:08:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.905+0000 7f6fce7fc640 1 -- 172.21.15.46:0/19674329 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc00aa780 con 0x7f6fd01580e0 2023-10-30T14:08:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.905+0000 7f6fce7fc640 1 -- 172.21.15.46:0/19674329 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc00aab10 con 0x7f6fd01580e0 2023-10-30T14:08:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.905+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/19674329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fd01580e0 msgr2=0x7f6fd0070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.905+0000 7f6fd5f3e640 1 --2- 172.21.15.46:0/19674329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fd01580e0 0x7f6fd0070740 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f6fd006b1a0 tx=0x7f6fc001c4c0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.905+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/19674329 shutdown_connections 2023-10-30T14:08:44.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.905+0000 7f6fd5f3e640 1 --2- 172.21.15.46:0/19674329 >> v2:172.21.15.154:3300/0 conn(0x7f6fd00c6540 0x7f6fd015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.905+0000 7f6fd5f3e640 1 --2- 172.21.15.46:0/19674329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fd01580e0 0x7f6fd0070740 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.905+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/19674329 >> 172.21.15.46:0/19674329 conn(0x7f6fd00c7910 msgr2=0x7f6fd00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:44.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.905+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/19674329 shutdown_connections 2023-10-30T14:08:44.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.905+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/19674329 wait complete. 2023-10-30T14:08:44.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fd5f3e640 1 Processor -- start 2023-10-30T14:08:44.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fd5f3e640 1 -- start start 2023-10-30T14:08:44.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fd5f3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fd0069e90 0x7f6fd014c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:44.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fd5f3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fd00c6540 0x7f6fd01f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:44.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fd5f3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fd01580e0 0x7f6fd01f67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:44.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fd5f3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0153280 con 0x7f6fd00c6540 2023-10-30T14:08:44.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fd5f3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd01f6cf0 con 0x7f6fd0069e90 2023-10-30T14:08:44.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fd5f3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd014b470 con 0x7f6fd01580e0 2023-10-30T14:08:44.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fd00c6540 0x7f6fd01f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:44.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fd00c6540 0x7f6fd01f6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35030/0 (socket says 172.21.15.46:35030) 2023-10-30T14:08:44.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fceffd640 1 -- 172.21.15.46:0/315600361 learned_addr learned my addr 172.21.15.46:0/315600361 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:44.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fcf7fe640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fd0069e90 0x7f6fd014c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:44.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fcffff640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fd01580e0 0x7f6fd01f67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fceffd640 1 -- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fd01580e0 msgr2=0x7f6fd01f67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fceffd640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fd01580e0 0x7f6fd01f67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fceffd640 1 -- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fd0069e90 msgr2=0x7f6fd014c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:44.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fceffd640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fd0069e90 0x7f6fd014c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:44.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fceffd640 1 -- 172.21.15.46:0/315600361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc00094d0 con 0x7f6fd00c6540 2023-10-30T14:08:44.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fcf7fe640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fd0069e90 0x7f6fd014c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:08:44.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fceffd640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fd00c6540 0x7f6fd01f6270 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400d800 tx=0x7f6fc400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:44.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fb7fff640 1 -- 172.21.15.46:0/315600361 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc400b6c0 con 0x7f6fd00c6540 2023-10-30T14:08:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fb7fff640 1 -- 172.21.15.46:0/315600361 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc400b820 con 0x7f6fd00c6540 2023-10-30T14:08:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/315600361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd014b6d0 con 0x7f6fd00c6540 2023-10-30T14:08:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fb7fff640 1 -- 172.21.15.46:0/315600361 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc400bb90 con 0x7f6fd00c6540 2023-10-30T14:08:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/315600361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd0152c30 con 0x7f6fd00c6540 2023-10-30T14:08:44.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.909+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/315600361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd0153430 con 0x7f6fd00c6540 2023-10-30T14:08:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.913+0000 7f6fb7fff640 1 -- 172.21.15.46:0/315600361 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6fc400bcf0 con 0x7f6fd00c6540 2023-10-30T14:08:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.913+0000 7f6fb7fff640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6fbc0664a0 0x7f6fbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.913+0000 7f6fb7fff640 1 -- 172.21.15.46:0/315600361 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7f6fc4084e20 con 0x7f6fd00c6540 2023-10-30T14:08:44.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.913+0000 7f6fcf7fe640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6fbc0664a0 0x7f6fbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:44.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.913+0000 7f6fcf7fe640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6fbc0664a0 0x7f6fbc068960 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00ae000 tx=0x7f6fc00a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:44.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:44.917+0000 7f6fb7fff640 1 -- 172.21.15.46:0/315600361 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6fd0153430 con 0x7f6fd00c6540 2023-10-30T14:08:45.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.105+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/315600361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fd0153430 con 0x7f6fd00c6540 2023-10-30T14:08:45.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.105+0000 7f6fb7fff640 1 -- 172.21.15.46:0/315600361 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f6fd0153430 con 0x7f6fd00c6540 2023-10-30T14:08:45.110 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:45.110 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:08:44.543694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:45.111 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 196 2023-10-30T14:08:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.109+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6fbc0664a0 msgr2=0x7f6fbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.109+0000 7f6fd5f3e640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6fbc0664a0 0x7f6fbc068960 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00ae000 tx=0x7f6fc00a7000 comp rx=0 tx=0).stop 2023-10-30T14:08:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.109+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/315600361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fd00c6540 msgr2=0x7f6fd01f6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.109+0000 7f6fd5f3e640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fd00c6540 0x7f6fd01f6270 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400d800 tx=0x7f6fc400dcc0 comp rx=0 tx=0).stop 2023-10-30T14:08:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.109+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/315600361 shutdown_connections 2023-10-30T14:08:45.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.109+0000 7f6fd5f3e640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6fbc0664a0 0x7f6fbc068960 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:45.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.109+0000 7f6fd5f3e640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fd01580e0 0x7f6fd01f67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:45.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.109+0000 7f6fd5f3e640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fd00c6540 0x7f6fd01f6270 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:45.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.109+0000 7f6fd5f3e640 1 --2- 172.21.15.46:0/315600361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fd0069e90 0x7f6fd014c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:45.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.109+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/315600361 >> 172.21.15.46:0/315600361 conn(0x7f6fd00c7910 msgr2=0x7f6fd00c1590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:45.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.109+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/315600361 shutdown_connections 2023-10-30T14:08:45.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:45.109+0000 7f6fd5f3e640 1 -- 172.21.15.46:0/315600361 wait complete. 2023-10-30T14:08:45.132 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 196, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 196, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8505": { "addr": "172.21.15.46:6833/4078242540", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4078242540, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4078242540, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 8505, "incarnation": 189, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_8655": { "addr": "172.21.15.46:6837/3044281789", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3044281789, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3044281789, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8655, "incarnation": 196, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 81, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:08:44.543694+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8505, "mds_1": 8655 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:08:47.132 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:47.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.281+0000 7fadaa980640 1 Processor -- start 2023-10-30T14:08:47.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fadaa980640 1 -- start start 2023-10-30T14:08:47.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fadaa980640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fada40c6bd0 0x7fada406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:47.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fadaa980640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada4066cf0 con 0x7fada4071110 2023-10-30T14:08:47.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fadaa980640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada40c70d0 con 0x7fada406f7a0 2023-10-30T14:08:47.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fadaa980640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada40700c0 con 0x7fada40c6bd0 2023-10-30T14:08:47.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fada8f1d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fada4071110 0x7fada4158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57282/0 (socket says 172.21.15.46:57282) 2023-10-30T14:08:47.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fada8f1d640 1 -- 172.21.15.46:0/3406686055 learned_addr learned my addr 172.21.15.46:0/3406686055 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:08:47.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fada37fe640 1 --2- 172.21.15.46:0/3406686055 >> v2:172.21.15.154:3300/0 conn(0x7fada40c6bd0 0x7fada406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:47.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fada2ffd640 1 -- 172.21.15.46:0/3406686055 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207493683 0 0) 0x7fada4066cf0 con 0x7fada4071110 2023-10-30T14:08:47.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fada2ffd640 1 -- 172.21.15.46:0/3406686055 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad8c003530 con 0x7fada4071110 2023-10-30T14:08:47.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fada37fe640 1 -- 172.21.15.46:0/3406686055 >> v1:172.21.15.154:6789/0 conn(0x7fada406f7a0 legacy=0x7fada4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:47.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fada37fe640 1 -- 172.21.15.46:0/3406686055 >> v1:172.21.15.46:6789/0 conn(0x7fada4071110 legacy=0x7fada4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:47.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.285+0000 7fada37fe640 1 -- 172.21.15.46:0/3406686055 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada40c7390 con 0x7fada40c6bd0 2023-10-30T14:08:47.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.289+0000 7fada37fe640 1 --2- 172.21.15.46:0/3406686055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada40c6bd0 0x7fada406fb80 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fad980097c0 tx=0x7fad9801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8280535f63c407dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:47.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.289+0000 7fada2ffd640 1 -- 172.21.15.46:0/3406686055 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad980a8970 con 0x7fada40c6bd0 2023-10-30T14:08:47.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.289+0000 7fada2ffd640 1 -- 172.21.15.46:0/3406686055 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad980a8ad0 con 0x7fada40c6bd0 2023-10-30T14:08:47.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.289+0000 7fada2ffd640 1 -- 172.21.15.46:0/3406686055 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad980a8e60 con 0x7fada40c6bd0 2023-10-30T14:08:47.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.289+0000 7fadaa980640 1 -- 172.21.15.46:0/3406686055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada40c6bd0 msgr2=0x7fada406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:47.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.289+0000 7fadaa980640 1 --2- 172.21.15.46:0/3406686055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada40c6bd0 0x7fada406fb80 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fad980097c0 tx=0x7fad9801ec70 comp rx=0 tx=0).stop 2023-10-30T14:08:47.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.289+0000 7fadaa980640 1 -- 172.21.15.46:0/3406686055 shutdown_connections 2023-10-30T14:08:47.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.289+0000 7fadaa980640 1 --2- 172.21.15.46:0/3406686055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada40c6bd0 0x7fada406fb80 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:47.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.289+0000 7fadaa980640 1 -- 172.21.15.46:0/3406686055 >> 172.21.15.46:0/3406686055 conn(0x7fada40c7910 msgr2=0x7fada40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:47.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.289+0000 7fadaa980640 1 -- 172.21.15.46:0/3406686055 shutdown_connections 2023-10-30T14:08:47.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.289+0000 7fadaa980640 1 -- 172.21.15.46:0/3406686055 wait complete. 2023-10-30T14:08:47.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fadaa980640 1 Processor -- start 2023-10-30T14:08:47.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fadaa980640 1 -- start start 2023-10-30T14:08:47.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fadaa980640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fada406f7a0 0x7fada4151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:47.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fadaa980640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada4071110 0x7fada41f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:47.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fadaa980640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fada40c6bd0 0x7fada41f3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:47.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fadaa980640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada41f74c0 con 0x7fada40c6bd0 2023-10-30T14:08:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fadaa980640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada41f7030 con 0x7fada4071110 2023-10-30T14:08:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fadaa980640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada4158380 con 0x7fada406f7a0 2023-10-30T14:08:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada8f1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fada40c6bd0 0x7fada41f3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada8f1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fada40c6bd0 0x7fada41f3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50064/0 (socket says 172.21.15.46:50064) 2023-10-30T14:08:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada8f1d640 1 -- 172.21.15.46:0/561357843 learned_addr learned my addr 172.21.15.46:0/561357843 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada37fe640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada4071110 0x7fada41f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada3fff640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fada406f7a0 0x7fada4151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada8f1d640 1 -- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fada406f7a0 msgr2=0x7fada4151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada8f1d640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fada406f7a0 0x7fada4151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:47.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada8f1d640 1 -- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada4071110 msgr2=0x7fada41f2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:47.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada8f1d640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada4071110 0x7fada41f2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:47.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada8f1d640 1 -- 172.21.15.46:0/561357843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad98009420 con 0x7fada40c6bd0 2023-10-30T14:08:47.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada37fe640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada4071110 0x7fada41f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:08:47.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada8f1d640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fada40c6bd0 0x7fada41f3530 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00caa0 tx=0x7fad9c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada0ff9640 1 -- 172.21.15.46:0/561357843 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c00aba0 con 0x7fada40c6bd0 2023-10-30T14:08:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada0ff9640 1 -- 172.21.15.46:0/561357843 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c00ad00 con 0x7fada40c6bd0 2023-10-30T14:08:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fadaa980640 1 -- 172.21.15.46:0/561357843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada41f6700 con 0x7fada40c6bd0 2023-10-30T14:08:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fada0ff9640 1 -- 172.21.15.46:0/561357843 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c012620 con 0x7fada40c6bd0 2023-10-30T14:08:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.293+0000 7fadaa980640 1 -- 172.21.15.46:0/561357843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada41f6cc0 con 0x7fada40c6bd0 2023-10-30T14:08:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.297+0000 7fadaa980640 1 -- 172.21.15.46:0/561357843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada41f71e0 con 0x7fada40c6bd0 2023-10-30T14:08:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.297+0000 7fada0ff9640 1 -- 172.21.15.46:0/561357843 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fad9c007070 con 0x7fada40c6bd0 2023-10-30T14:08:47.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.297+0000 7fada0ff9640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad900664a0 0x7fad90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:47.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.297+0000 7fada0ff9640 1 -- 172.21.15.46:0/561357843 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7fad9c084890 con 0x7fada40c6bd0 2023-10-30T14:08:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.297+0000 7fada3fff640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad900664a0 0x7fad90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.297+0000 7fada3fff640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad900664a0 0x7fad90068960 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fad94000f70 tx=0x7fad94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:47.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.301+0000 7fada0ff9640 1 -- 172.21.15.46:0/561357843 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fada41f71e0 con 0x7fada40c6bd0 2023-10-30T14:08:47.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 -- 172.21.15.46:0/561357843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fada41f71e0 con 0x7fada40c6bd0 2023-10-30T14:08:47.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fada0ff9640 1 -- 172.21.15.46:0/561357843 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fada41f71e0 con 0x7fada40c6bd0 2023-10-30T14:08:47.497 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:47.497 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:08:44.543694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:47.497 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 196 2023-10-30T14:08:47.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 -- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad900664a0 msgr2=0x7fad90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:47.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad900664a0 0x7fad90068960 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fad94000f70 tx=0x7fad94009040 comp rx=0 tx=0).stop 2023-10-30T14:08:47.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 -- 172.21.15.46:0/561357843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fada40c6bd0 msgr2=0x7fada41f3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:47.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fada40c6bd0 0x7fada41f3530 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00caa0 tx=0x7fad9c00cf60 comp rx=0 tx=0).stop 2023-10-30T14:08:47.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 -- 172.21.15.46:0/561357843 shutdown_connections 2023-10-30T14:08:47.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad900664a0 0x7fad90068960 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:47.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fada40c6bd0 0x7fada41f3530 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:47.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fada4071110 0x7fada41f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:47.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 --2- 172.21.15.46:0/561357843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fada406f7a0 0x7fada4151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:47.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 -- 172.21.15.46:0/561357843 >> 172.21.15.46:0/561357843 conn(0x7fada40c7910 msgr2=0x7fada40c3910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:47.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 -- 172.21.15.46:0/561357843 shutdown_connections 2023-10-30T14:08:47.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:47.493+0000 7fadaa980640 1 -- 172.21.15.46:0/561357843 wait complete. 2023-10-30T14:08:47.518 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:08:49.519 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:49.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e9a32640 1 Processor -- start 2023-10-30T14:08:49.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e9a32640 1 -- start start 2023-10-30T14:08:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e9a32640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e41538a0 0x7fa1e415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e9a32640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa1e40cf160 0x7fa1e40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e9a32640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e40cb800 con 0x7fa1e40ca0e0 2023-10-30T14:08:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e9a32640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e415dc30 con 0x7fa1e40cf160 2023-10-30T14:08:49.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e9a32640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e40cb330 con 0x7fa1e41538a0 2023-10-30T14:08:49.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e27fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa1e40ca0e0 0x7fa1e415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57284/0 (socket says 172.21.15.46:57284) 2023-10-30T14:08:49.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e27fc640 1 -- 172.21.15.46:0/353160171 learned_addr learned my addr 172.21.15.46:0/353160171 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:08:49.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e37fe640 1 --2- 172.21.15.46:0/353160171 >> v2:172.21.15.46:3300/0 conn(0x7fa1e40cf160 0x7fa1e40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:49.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e2ffd640 1 --2- 172.21.15.46:0/353160171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e41538a0 0x7fa1e415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:49.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e1ffb640 1 -- 172.21.15.46:0/353160171 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460366549 0 0) 0x7fa1e40cb800 con 0x7fa1e40ca0e0 2023-10-30T14:08:49.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e1ffb640 1 -- 172.21.15.46:0/353160171 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1cc003530 con 0x7fa1e40ca0e0 2023-10-30T14:08:49.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e37fe640 1 -- 172.21.15.46:0/353160171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e41538a0 msgr2=0x7fa1e415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:49.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e37fe640 1 --2- 172.21.15.46:0/353160171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e41538a0 0x7fa1e415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:49.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e37fe640 1 -- 172.21.15.46:0/353160171 >> v1:172.21.15.46:6789/0 conn(0x7fa1e40ca0e0 legacy=0x7fa1e415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:49.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.697+0000 7fa1e37fe640 1 -- 172.21.15.46:0/353160171 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e40cad70 con 0x7fa1e40cf160 2023-10-30T14:08:49.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.701+0000 7fa1e37fe640 1 --2- 172.21.15.46:0/353160171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1e40cf160 0x7fa1e40cf590 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009d90 tx=0x7fa1dc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45323b87b4fcd93e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:49.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.701+0000 7fa1e1ffb640 1 -- 172.21.15.46:0/353160171 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0af600 con 0x7fa1e40cf160 2023-10-30T14:08:49.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.701+0000 7fa1e1ffb640 1 -- 172.21.15.46:0/353160171 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc0ab050 con 0x7fa1e40cf160 2023-10-30T14:08:49.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.701+0000 7fa1e1ffb640 1 -- 172.21.15.46:0/353160171 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0af980 con 0x7fa1e40cf160 2023-10-30T14:08:49.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.705+0000 7fa1e9a32640 1 -- 172.21.15.46:0/353160171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1e40cf160 msgr2=0x7fa1e40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:49.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.705+0000 7fa1e9a32640 1 --2- 172.21.15.46:0/353160171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1e40cf160 0x7fa1e40cf590 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009d90 tx=0x7fa1dc01f130 comp rx=0 tx=0).stop 2023-10-30T14:08:49.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.705+0000 7fa1e9a32640 1 -- 172.21.15.46:0/353160171 shutdown_connections 2023-10-30T14:08:49.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.705+0000 7fa1e9a32640 1 --2- 172.21.15.46:0/353160171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1e40cf160 0x7fa1e40cf590 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:49.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.705+0000 7fa1e9a32640 1 --2- 172.21.15.46:0/353160171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e41538a0 0x7fa1e415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:49.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.705+0000 7fa1e9a32640 1 -- 172.21.15.46:0/353160171 >> 172.21.15.46:0/353160171 conn(0x7fa1e406bfe0 msgr2=0x7fa1e406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:49.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.705+0000 7fa1e9a32640 1 -- 172.21.15.46:0/353160171 shutdown_connections 2023-10-30T14:08:49.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.705+0000 7fa1e9a32640 1 -- 172.21.15.46:0/353160171 wait complete. 2023-10-30T14:08:49.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.705+0000 7fa1e9a32640 1 Processor -- start 2023-10-30T14:08:49.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.705+0000 7fa1e9a32640 1 -- start start 2023-10-30T14:08:49.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.705+0000 7fa1e9a32640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1e40ca0e0 0x7fa1e41ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:49.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.705+0000 7fa1e9a32640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1e40cf160 0x7fa1e40b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:49.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e9a32640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e41538a0 0x7fa1e40b4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:49.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e9a32640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e415b2f0 con 0x7fa1e40cf160 2023-10-30T14:08:49.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e9a32640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e4143ba0 con 0x7fa1e40ca0e0 2023-10-30T14:08:49.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e9a32640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e41ef310 con 0x7fa1e41538a0 2023-10-30T14:08:49.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1e40cf160 0x7fa1e40b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:49.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1e40cf160 0x7fa1e40b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50082/0 (socket says 172.21.15.46:50082) 2023-10-30T14:08:49.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e27fc640 1 -- 172.21.15.46:0/2231885572 learned_addr learned my addr 172.21.15.46:0/2231885572 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:49.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e37fe640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e41538a0 0x7fa1e40b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:49.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e2ffd640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1e40ca0e0 0x7fa1e41ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:49.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e27fc640 1 -- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e41538a0 msgr2=0x7fa1e40b4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:49.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e27fc640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e41538a0 0x7fa1e40b4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:49.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e27fc640 1 -- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1e40ca0e0 msgr2=0x7fa1e41ef7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:49.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e27fc640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1e40ca0e0 0x7fa1e41ef7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:49.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e27fc640 1 -- 172.21.15.46:0/2231885572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc009a70 con 0x7fa1e40cf160 2023-10-30T14:08:49.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e27fc640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1e40cf160 0x7fa1e40b4450 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800c4d0 tx=0x7fa1d800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:49.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1cbfff640 1 -- 172.21.15.46:0/2231885572 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8005cf0 con 0x7fa1e40cf160 2023-10-30T14:08:49.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1cbfff640 1 -- 172.21.15.46:0/2231885572 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d8005e50 con 0x7fa1e40cf160 2023-10-30T14:08:49.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e9a32640 1 -- 172.21.15.46:0/2231885572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e4158fb0 con 0x7fa1e40cf160 2023-10-30T14:08:49.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1cbfff640 1 -- 172.21.15.46:0/2231885572 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8007270 con 0x7fa1e40cf160 2023-10-30T14:08:49.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.709+0000 7fa1e9a32640 1 -- 172.21.15.46:0/2231885572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e41eefa0 con 0x7fa1e40cf160 2023-10-30T14:08:49.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.713+0000 7fa1e9a32640 1 -- 172.21.15.46:0/2231885572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e40b2a50 con 0x7fa1e40cf160 2023-10-30T14:08:49.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.713+0000 7fa1cbfff640 1 -- 172.21.15.46:0/2231885572 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1d80073d0 con 0x7fa1e40cf160 2023-10-30T14:08:49.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.713+0000 7fa1cbfff640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1d00664d0 0x7fa1d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:49.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.713+0000 7fa1cbfff640 1 -- 172.21.15.46:0/2231885572 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7fa1d8083be0 con 0x7fa1e40cf160 2023-10-30T14:08:49.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.713+0000 7fa1e2ffd640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1d00664d0 0x7fa1d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:49.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.713+0000 7fa1e2ffd640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1d00664d0 0x7fa1d0068990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4009ac0 tx=0x7fa1d4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.717+0000 7fa1cbfff640 1 -- 172.21.15.46:0/2231885572 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa1d8050870 con 0x7fa1e40cf160 2023-10-30T14:08:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.973+0000 7fa1e9a32640 1 -- 172.21.15.46:0/2231885572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1e404e170 con 0x7fa1e40cf160 2023-10-30T14:08:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.973+0000 7fa1cbfff640 1 -- 172.21.15.46:0/2231885572 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fa1d8017070 con 0x7fa1e40cf160 2023-10-30T14:08:49.977 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:49.977 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:08:44.543694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:49.977 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 196 2023-10-30T14:08:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.977+0000 7fa1e9a32640 1 -- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1d00664d0 msgr2=0x7fa1d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:49.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.977+0000 7fa1e9a32640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1d00664d0 0x7fa1d0068990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4009ac0 tx=0x7fa1d4009190 comp rx=0 tx=0).stop 2023-10-30T14:08:49.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.977+0000 7fa1e9a32640 1 -- 172.21.15.46:0/2231885572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1e40cf160 msgr2=0x7fa1e40b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:49.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.977+0000 7fa1e9a32640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1e40cf160 0x7fa1e40b4450 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800c4d0 tx=0x7fa1d800c990 comp rx=0 tx=0).stop 2023-10-30T14:08:49.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.977+0000 7fa1e9a32640 1 -- 172.21.15.46:0/2231885572 shutdown_connections 2023-10-30T14:08:49.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.977+0000 7fa1e9a32640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa1d00664d0 0x7fa1d0068990 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:49.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.977+0000 7fa1e9a32640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e41538a0 0x7fa1e40b4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:49.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.977+0000 7fa1e9a32640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1e40cf160 0x7fa1e40b4450 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:49.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.977+0000 7fa1e9a32640 1 --2- 172.21.15.46:0/2231885572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1e40ca0e0 0x7fa1e41ef7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:49.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.977+0000 7fa1e9a32640 1 -- 172.21.15.46:0/2231885572 >> 172.21.15.46:0/2231885572 conn(0x7fa1e406bfe0 msgr2=0x7fa1e4152410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:49.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.977+0000 7fa1e9a32640 1 -- 172.21.15.46:0/2231885572 shutdown_connections 2023-10-30T14:08:49.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:49.977+0000 7fa1e9a32640 1 -- 172.21.15.46:0/2231885572 wait complete. 2023-10-30T14:08:50.015 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:08:52.016 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:52.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e3b65640 1 Processor -- start 2023-10-30T14:08:52.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e3b65640 1 -- start start 2023-10-30T14:08:52.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e3b65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc1538a0 0x7f17dc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:52.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e3b65640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f17dc0ca0e0 0x7f17dc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:52.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e3b65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc0cb800 con 0x7f17dc0cf160 2023-10-30T14:08:52.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e3b65640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17dc15dc30 con 0x7f17dc0ca0e0 2023-10-30T14:08:52.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e3b65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0cb330 con 0x7f17dc1538a0 2023-10-30T14:08:52.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e1901640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc1538a0 0x7f17dc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:52.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e1100640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f17dc0ca0e0 0x7f17dc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:52.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e1901640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc1538a0 0x7f17dc15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:43742/0 (socket says 172.21.15.46:43742) 2023-10-30T14:08:52.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e1901640 1 -- 172.21.15.46:0/2583680260 learned_addr learned my addr 172.21.15.46:0/2583680260 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:52.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e1901640 1 -- 172.21.15.46:0/2583680260 >> v2:172.21.15.154:3300/0 conn(0x7f17dc0ca0e0 msgr2=0x7f17dc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:52.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e1901640 1 --2- 172.21.15.46:0/2583680260 >> v2:172.21.15.154:3300/0 conn(0x7f17dc0ca0e0 0x7f17dc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:52.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e1901640 1 -- 172.21.15.46:0/2583680260 >> v1:172.21.15.154:6789/0 conn(0x7f17dc0cf160 legacy=0x7f17dc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:52.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.193+0000 7f17e1901640 1 -- 172.21.15.46:0/2583680260 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc0cad70 con 0x7f17dc1538a0 2023-10-30T14:08:52.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.197+0000 7f17e1901640 1 --2- 172.21.15.46:0/2583680260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc1538a0 0x7f17dc15d6f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009870 tx=0x7f17d801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b186cf054ceec77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:52.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.197+0000 7f17e08ff640 1 -- 172.21.15.46:0/2583680260 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d80a8a30 con 0x7f17dc1538a0 2023-10-30T14:08:52.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.197+0000 7f17e08ff640 1 -- 172.21.15.46:0/2583680260 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d80a8b90 con 0x7f17dc1538a0 2023-10-30T14:08:52.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.197+0000 7f17e08ff640 1 -- 172.21.15.46:0/2583680260 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d80b0440 con 0x7f17dc1538a0 2023-10-30T14:08:52.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.197+0000 7f17e3b65640 1 -- 172.21.15.46:0/2583680260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc1538a0 msgr2=0x7f17dc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:52.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.197+0000 7f17e3b65640 1 --2- 172.21.15.46:0/2583680260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc1538a0 0x7f17dc15d6f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009870 tx=0x7f17d801ed30 comp rx=0 tx=0).stop 2023-10-30T14:08:52.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.197+0000 7f17e3b65640 1 -- 172.21.15.46:0/2583680260 shutdown_connections 2023-10-30T14:08:52.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.197+0000 7f17e3b65640 1 --2- 172.21.15.46:0/2583680260 >> v2:172.21.15.154:3300/0 conn(0x7f17dc0ca0e0 0x7f17dc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:52.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.197+0000 7f17e3b65640 1 --2- 172.21.15.46:0/2583680260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc1538a0 0x7f17dc15d6f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:52.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.197+0000 7f17e3b65640 1 -- 172.21.15.46:0/2583680260 >> 172.21.15.46:0/2583680260 conn(0x7f17dc06bfe0 msgr2=0x7f17dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:52.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e3b65640 1 -- 172.21.15.46:0/2583680260 shutdown_connections 2023-10-30T14:08:52.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e3b65640 1 -- 172.21.15.46:0/2583680260 wait complete. 2023-10-30T14:08:52.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e3b65640 1 Processor -- start 2023-10-30T14:08:52.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e3b65640 1 -- start start 2023-10-30T14:08:52.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e3b65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc0ca0e0 0x7f17dc0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:52.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e3b65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17dc0cf160 0x7f17dc0b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:52.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e3b65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17dc1538a0 0x7f17dc0b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:52.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e3b65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc1efcc0 con 0x7f17dc0cf160 2023-10-30T14:08:52.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e3b65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc1f01f0 con 0x7f17dc1538a0 2023-10-30T14:08:52.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e3b65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc1eff10 con 0x7f17dc0ca0e0 2023-10-30T14:08:52.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e1100640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17dc0cf160 0x7f17dc0b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:52.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e1901640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc0ca0e0 0x7f17dc0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:52.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e1901640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc0ca0e0 0x7f17dc0ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:43758/0 (socket says 172.21.15.46:43758) 2023-10-30T14:08:52.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e1901640 1 -- 172.21.15.46:0/4235197226 learned_addr learned my addr 172.21.15.46:0/4235197226 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:52.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e2102640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17dc1538a0 0x7f17dc0b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:52.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e1100640 1 -- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc0ca0e0 msgr2=0x7f17dc0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:52.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e1100640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc0ca0e0 0x7f17dc0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:52.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e1100640 1 -- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17dc1538a0 msgr2=0x7f17dc0b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:52.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e1100640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17dc1538a0 0x7f17dc0b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:52.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e1100640 1 -- 172.21.15.46:0/4235197226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d80094d0 con 0x7f17dc0cf160 2023-10-30T14:08:52.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.201+0000 7f17e2102640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17dc1538a0 0x7f17dc0b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:08:52.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.205+0000 7f17e1100640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17dc0cf160 0x7f17dc0b3db0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00e870 tx=0x7f17cc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:52.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.205+0000 7f17d27fc640 1 -- 172.21.15.46:0/4235197226 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0095d0 con 0x7f17dc0cf160 2023-10-30T14:08:52.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.205+0000 7f17d27fc640 1 -- 172.21.15.46:0/4235197226 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17cc00c940 con 0x7f17dc0cf160 2023-10-30T14:08:52.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.205+0000 7f17d27fc640 1 -- 172.21.15.46:0/4235197226 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc00ccb0 con 0x7f17dc0cf160 2023-10-30T14:08:52.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.205+0000 7f17e3b65640 1 -- 172.21.15.46:0/4235197226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17dc0b32d0 con 0x7f17dc0cf160 2023-10-30T14:08:52.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.205+0000 7f17e3b65640 1 -- 172.21.15.46:0/4235197226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17dc0b3890 con 0x7f17dc0cf160 2023-10-30T14:08:52.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.205+0000 7f17e3b65640 1 -- 172.21.15.46:0/4235197226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17dc0b2a50 con 0x7f17dc0cf160 2023-10-30T14:08:52.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.209+0000 7f17d27fc640 1 -- 172.21.15.46:0/4235197226 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f17cc013470 con 0x7f17dc0cf160 2023-10-30T14:08:52.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.209+0000 7f17d27fc640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17c0066520 0x7f17c00689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:52.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.209+0000 7f17d27fc640 1 -- 172.21.15.46:0/4235197226 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7f17cc084660 con 0x7f17dc0cf160 2023-10-30T14:08:52.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.213+0000 7f17e1901640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17c0066520 0x7f17c00689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:52.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.213+0000 7f17e1901640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17c0066520 0x7f17c00689e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f17d801ed70 tx=0x7f17d80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:52.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.217+0000 7f17d27fc640 1 -- 172.21.15.46:0/4235197226 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f17cc051460 con 0x7f17dc0cf160 2023-10-30T14:08:52.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 -- 172.21.15.46:0/4235197226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17dc04e170 con 0x7f17dc0cf160 2023-10-30T14:08:52.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17d27fc640 1 -- 172.21.15.46:0/4235197226 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f17cc054a80 con 0x7f17dc0cf160 2023-10-30T14:08:52.429 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:52.429 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:08:44.543694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:52.429 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 196 2023-10-30T14:08:52.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 -- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17c0066520 msgr2=0x7f17c00689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17c0066520 0x7f17c00689e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f17d801ed70 tx=0x7f17d80aa070 comp rx=0 tx=0).stop 2023-10-30T14:08:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 -- 172.21.15.46:0/4235197226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17dc0cf160 msgr2=0x7f17dc0b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17dc0cf160 0x7f17dc0b3db0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00e870 tx=0x7f17cc00ed30 comp rx=0 tx=0).stop 2023-10-30T14:08:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 -- 172.21.15.46:0/4235197226 shutdown_connections 2023-10-30T14:08:52.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17c0066520 0x7f17c00689e0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:52.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17dc1538a0 0x7f17dc0b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:52.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17dc0cf160 0x7f17dc0b3db0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:52.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 --2- 172.21.15.46:0/4235197226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc0ca0e0 0x7f17dc0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:52.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 -- 172.21.15.46:0/4235197226 >> 172.21.15.46:0/4235197226 conn(0x7f17dc06bfe0 msgr2=0x7f17dc152870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:52.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 -- 172.21.15.46:0/4235197226 shutdown_connections 2023-10-30T14:08:52.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:52.425+0000 7f17e3b65640 1 -- 172.21.15.46:0/4235197226 wait complete. 2023-10-30T14:08:52.456 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:08:54.457 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8ef59e640 1 Processor -- start 2023-10-30T14:08:54.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8ef59e640 1 -- start start 2023-10-30T14:08:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8ef59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f015cd50 0x7fd8f00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8ef59e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f015a6c0 con 0x7fd8f01538f0 2023-10-30T14:08:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8ef59e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f015d300 con 0x7fd8f00cefa0 2023-10-30T14:08:54.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8ef59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f015d5e0 con 0x7fd8f015cd50 2023-10-30T14:08:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8ee59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd8f01538f0 0x7fd8f0159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57300/0 (socket says 172.21.15.46:57300) 2023-10-30T14:08:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8ee59c640 1 -- 172.21.15.46:0/2859661102 learned_addr learned my addr 172.21.15.46:0/2859661102 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:08:54.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8e7fff640 1 --2- 172.21.15.46:0/2859661102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f015cd50 0x7fd8f00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107126625 0 0) 0x7fd8f015a6c0 con 0x7fd8f01538f0 2023-10-30T14:08:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8dc003530 con 0x7fd8f01538f0 2023-10-30T14:08:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1483863429 0 0) 0x7fd8dc003530 con 0x7fd8f01538f0 2023-10-30T14:08:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f015cd50 msgr2=0x7fd8f00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 --2- 172.21.15.46:0/2859661102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f015cd50 0x7fd8f00c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 >> v1:172.21.15.154:6789/0 conn(0x7fd8f00cefa0 legacy=0x7fd8f01640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f015d8c0 con 0x7fd8f01538f0 2023-10-30T14:08:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fd8e0001680 con 0x7fd8f01538f0 2023-10-30T14:08:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 >> v1:172.21.15.46:6789/0 conn(0x7fd8f01538f0 legacy=0x7fd8f0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 --2- 172.21.15.46:0/2859661102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8dc0039b0 0x7fd8dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 --2- 172.21.15.46:0/2859661102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8dc0047d0 0x7fd8dc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 --2- 172.21.15.46:0/2859661102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f015cd50 0x7fd8dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f015d5e0 con 0x7fd8dc0047d0 2023-10-30T14:08:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc00bb00 con 0x7fd8dc0039b0 2023-10-30T14:08:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.617+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc00bcc0 con 0x7fd8f015cd50 2023-10-30T14:08:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8e7fff640 1 --2- 172.21.15.46:0/2859661102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8dc0047d0 0x7fd8dc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8ee59c640 1 --2- 172.21.15.46:0/2859661102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8dc0039b0 0x7fd8dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8eed9d640 1 --2- 172.21.15.46:0/2859661102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f015cd50 0x7fd8dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8e7fff640 1 -- 172.21.15.46:0/2859661102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f015cd50 msgr2=0x7fd8dc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8e7fff640 1 --2- 172.21.15.46:0/2859661102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f015cd50 0x7fd8dc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8e7fff640 1 -- 172.21.15.46:0/2859661102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8dc0039b0 msgr2=0x7fd8dc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8e7fff640 1 --2- 172.21.15.46:0/2859661102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8dc0039b0 0x7fd8dc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8e7fff640 1 -- 172.21.15.46:0/2859661102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc003530 con 0x7fd8dc0047d0 2023-10-30T14:08:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8e7fff640 1 --2- 172.21.15.46:0/2859661102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8dc0047d0 0x7fd8dc004c50 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009ab0 tx=0x7fd8d801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3f111763fc54bf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d80a5600 con 0x7fd8dc0047d0 2023-10-30T14:08:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d80a5760 con 0x7fd8dc0047d0 2023-10-30T14:08:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8edd9b640 1 -- 172.21.15.46:0/2859661102 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d80a5b10 con 0x7fd8dc0047d0 2023-10-30T14:08:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2859661102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8dc0047d0 msgr2=0x7fd8dc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:54.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8ef59e640 1 --2- 172.21.15.46:0/2859661102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8dc0047d0 0x7fd8dc004c50 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009ab0 tx=0x7fd8d801ca10 comp rx=0 tx=0).stop 2023-10-30T14:08:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.621+0000 7fd8eed9d640 1 -- 172.21.15.46:0/2859661102 reap_dead start 2023-10-30T14:08:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2859661102 shutdown_connections 2023-10-30T14:08:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2859661102 >> 172.21.15.46:0/2859661102 conn(0x7fd8f006a640 msgr2=0x7fd8f0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:54.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2859661102 shutdown_connections 2023-10-30T14:08:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2859661102 wait complete. 2023-10-30T14:08:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ef59e640 1 Processor -- start 2023-10-30T14:08:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ef59e640 1 -- start start 2023-10-30T14:08:54.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ef59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8dc0047d0 0x7fd8f015c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ef59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f0152db0 0x7fd8f01f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ef59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00b4940 0x7fd8f0159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ef59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f015ba40 con 0x7fd8f0152db0 2023-10-30T14:08:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ef59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f0151870 con 0x7fd8f00b4940 2023-10-30T14:08:54.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ef59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f015b110 con 0x7fd8dc0047d0 2023-10-30T14:08:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f0152db0 0x7fd8f01f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f0152db0 0x7fd8f01f2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50108/0 (socket says 172.21.15.46:50108) 2023-10-30T14:08:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8e7fff640 1 -- 172.21.15.46:0/2316056426 learned_addr learned my addr 172.21.15.46:0/2316056426 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8ee59c640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8dc0047d0 0x7fd8f015c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:54.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8eed9d640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00b4940 0x7fd8f0159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8e7fff640 1 -- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8dc0047d0 msgr2=0x7fd8f015c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8e7fff640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8dc0047d0 0x7fd8f015c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8e7fff640 1 -- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00b4940 msgr2=0x7fd8f0159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:54.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8e7fff640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00b4940 0x7fd8f0159030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:54.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8e7fff640 1 -- 172.21.15.46:0/2316056426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f01f3490 con 0x7fd8f0152db0 2023-10-30T14:08:54.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.625+0000 7fd8e7fff640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f0152db0 0x7fd8f01f2f50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009ab0 tx=0x7fd8dc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:54.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.629+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/2316056426 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8004230 con 0x7fd8f0152db0 2023-10-30T14:08:54.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.629+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/2316056426 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d8004390 con 0x7fd8f0152db0 2023-10-30T14:08:54.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.629+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/2316056426 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d80ad850 con 0x7fd8f0152db0 2023-10-30T14:08:54.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.629+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2316056426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f015a900 con 0x7fd8f0152db0 2023-10-30T14:08:54.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.629+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2316056426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f01f2ac0 con 0x7fd8f0152db0 2023-10-30T14:08:54.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.629+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2316056426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f01f2c50 con 0x7fd8f0152db0 2023-10-30T14:08:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.629+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/2316056426 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd8d80b4040 con 0x7fd8f0152db0 2023-10-30T14:08:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.629+0000 7fd8e6ffd640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8d00664d0 0x7fd8d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.629+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/2316056426 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7fd8d811ed10 con 0x7fd8f0152db0 2023-10-30T14:08:54.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.629+0000 7fd8ee59c640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8d00664d0 0x7fd8d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.633+0000 7fd8ee59c640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8d00664d0 0x7fd8d0068990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0007d30 tx=0x7fd8e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:54.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.633+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/2316056426 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd8d80eb990 con 0x7fd8f0152db0 2023-10-30T14:08:54.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2316056426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8f01f3490 con 0x7fd8f0152db0 2023-10-30T14:08:54.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8e6ffd640 1 -- 172.21.15.46:0/2316056426 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+5734 (secure 0 0 0) 0x7fd8f01f3490 con 0x7fd8f0152db0 2023-10-30T14:08:54.829 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:54.829 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:08:54.204176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:resolve","state_seq":18,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 197 2023-10-30T14:08:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8d00664d0 msgr2=0x7fd8d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:54.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8d00664d0 0x7fd8d0068990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0007d30 tx=0x7fd8e0009040 comp rx=0 tx=0).stop 2023-10-30T14:08:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2316056426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f0152db0 msgr2=0x7fd8f01f2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f0152db0 0x7fd8f01f2f50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009ab0 tx=0x7fd8dc005800 comp rx=0 tx=0).stop 2023-10-30T14:08:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2316056426 shutdown_connections 2023-10-30T14:08:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8d00664d0 0x7fd8d0068990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:54.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00b4940 0x7fd8f0159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8f0152db0 0x7fd8f01f2f50 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 --2- 172.21.15.46:0/2316056426 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8dc0047d0 0x7fd8f015c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2316056426 >> 172.21.15.46:0/2316056426 conn(0x7fd8f006a640 msgr2=0x7fd8f006bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2316056426 shutdown_connections 2023-10-30T14:08:54.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:54.825+0000 7fd8ef59e640 1 -- 172.21.15.46:0/2316056426 wait complete. 2023-10-30T14:08:54.850 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:08:56.851 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:57.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:56.997+0000 7ff33b032640 1 Processor -- start 2023-10-30T14:08:57.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:56.997+0000 7ff33b032640 1 -- start start 2023-10-30T14:08:57.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff33b032640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff3340c6b80 0x7ff3340bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:57.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff33b032640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3340c6f60 0x7ff3340c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:57.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff33b032640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3340c8430 0x7ff33413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:57.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff33b032640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3340c0a50 con 0x7ff3340c8430 2023-10-30T14:08:57.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff33b032640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33406f6e0 con 0x7ff3340c6b80 2023-10-30T14:08:57.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff33b032640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33406f9c0 con 0x7ff3340c6f60 2023-10-30T14:08:57.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff3395cf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3340c8430 0x7ff33413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:57.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff3395cf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3340c8430 0x7ff33413ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55984/0 (socket says 172.21.15.46:55984) 2023-10-30T14:08:57.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff3395cf640 1 -- 172.21.15.46:0/86523044 learned_addr learned my addr 172.21.15.46:0/86523044 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:57.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff338dce640 1 --2- 172.21.15.46:0/86523044 >> v2:172.21.15.154:3300/0 conn(0x7ff3340c6b80 0x7ff3340bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:57.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff333fff640 1 --2- 172.21.15.46:0/86523044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3340c6f60 0x7ff3340c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:57.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff3395cf640 1 -- 172.21.15.46:0/86523044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3340c6f60 msgr2=0x7ff3340c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:57.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff3395cf640 1 --2- 172.21.15.46:0/86523044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3340c6f60 0x7ff3340c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:57.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff3395cf640 1 -- 172.21.15.46:0/86523044 >> v2:172.21.15.154:3300/0 conn(0x7ff3340c6b80 msgr2=0x7ff3340bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:57.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff3395cf640 1 --2- 172.21.15.46:0/86523044 >> v2:172.21.15.154:3300/0 conn(0x7ff3340c6b80 0x7ff3340bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:57.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.001+0000 7ff3395cf640 1 -- 172.21.15.46:0/86523044 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33406fca0 con 0x7ff3340c8430 2023-10-30T14:08:57.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff3395cf640 1 --2- 172.21.15.46:0/86523044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3340c8430 0x7ff33413ed90 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7ff33406f1b0 tx=0x7ff32c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfa7087f1fad1c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:57.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff3337fe640 1 -- 172.21.15.46:0/86523044 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c0a56d0 con 0x7ff3340c8430 2023-10-30T14:08:57.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff3337fe640 1 -- 172.21.15.46:0/86523044 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c0a5830 con 0x7ff3340c8430 2023-10-30T14:08:57.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff3337fe640 1 -- 172.21.15.46:0/86523044 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c0a5bc0 con 0x7ff3340c8430 2023-10-30T14:08:57.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff33b032640 1 -- 172.21.15.46:0/86523044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3340c8430 msgr2=0x7ff33413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:57.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff33b032640 1 --2- 172.21.15.46:0/86523044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3340c8430 0x7ff33413ed90 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7ff33406f1b0 tx=0x7ff32c01c940 comp rx=0 tx=0).stop 2023-10-30T14:08:57.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff33b032640 1 -- 172.21.15.46:0/86523044 shutdown_connections 2023-10-30T14:08:57.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff33b032640 1 --2- 172.21.15.46:0/86523044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3340c8430 0x7ff33413ed90 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:57.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff33b032640 1 --2- 172.21.15.46:0/86523044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3340c6f60 0x7ff3340c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:57.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff33b032640 1 --2- 172.21.15.46:0/86523044 >> v2:172.21.15.154:3300/0 conn(0x7ff3340c6b80 0x7ff3340bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:57.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff33b032640 1 -- 172.21.15.46:0/86523044 >> 172.21.15.46:0/86523044 conn(0x7ff3340c7910 msgr2=0x7ff3340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:57.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff33b032640 1 -- 172.21.15.46:0/86523044 shutdown_connections 2023-10-30T14:08:57.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff33b032640 1 -- 172.21.15.46:0/86523044 wait complete. 2023-10-30T14:08:57.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff33b032640 1 Processor -- start 2023-10-30T14:08:57.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.005+0000 7ff33b032640 1 -- start start 2023-10-30T14:08:57.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff33b032640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3340c6b80 0x7ff33413d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:57.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff33b032640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3340c6f60 0x7ff334155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:57.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff33b032640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3340c8430 0x7ff3340c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:57.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff33b032640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33413cdc0 con 0x7ff3340c6f60 2023-10-30T14:08:57.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff33b032640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff334134560 con 0x7ff3340c8430 2023-10-30T14:08:57.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff33b032640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33413cb00 con 0x7ff3340c6b80 2023-10-30T14:08:57.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff333fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3340c6f60 0x7ff334155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:57.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff333fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3340c6f60 0x7ff334155cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55992/0 (socket says 172.21.15.46:55992) 2023-10-30T14:08:57.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff333fff640 1 -- 172.21.15.46:0/3685180399 learned_addr learned my addr 172.21.15.46:0/3685180399 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:57.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff338dce640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3340c6b80 0x7ff33413d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:57.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff3395cf640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3340c8430 0x7ff3340c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:57.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff333fff640 1 -- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3340c6b80 msgr2=0x7ff33413d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:57.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff333fff640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3340c6b80 0x7ff33413d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:57.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff333fff640 1 -- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3340c8430 msgr2=0x7ff3340c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:57.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff333fff640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3340c8430 0x7ff3340c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:57.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff333fff640 1 -- 172.21.15.46:0/3685180399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c009b30 con 0x7ff3340c6f60 2023-10-30T14:08:57.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff333fff640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3340c6f60 0x7ff334155cd0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7ff32000e960 tx=0x7ff32000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:57.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff3317fa640 1 -- 172.21.15.46:0/3685180399 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff320009690 con 0x7ff3340c6f60 2023-10-30T14:08:57.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff3317fa640 1 -- 172.21.15.46:0/3685180399 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32000ca00 con 0x7ff3340c6f60 2023-10-30T14:08:57.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff3317fa640 1 -- 172.21.15.46:0/3685180399 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32000cd70 con 0x7ff3340c6f60 2023-10-30T14:08:57.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff33b032640 1 -- 172.21.15.46:0/3685180399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33413c820 con 0x7ff3340c6f60 2023-10-30T14:08:57.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff33b032640 1 -- 172.21.15.46:0/3685180399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff334134280 con 0x7ff3340c6f60 2023-10-30T14:08:57.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff33b032640 1 -- 172.21.15.46:0/3685180399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33404e170 con 0x7ff3340c6f60 2023-10-30T14:08:57.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.009+0000 7ff3317fa640 1 -- 172.21.15.46:0/3685180399 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff320013470 con 0x7ff3340c6f60 2023-10-30T14:08:57.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.013+0000 7ff3317fa640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff3180664a0 0x7ff318068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:57.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.013+0000 7ff3317fa640 1 -- 172.21.15.46:0/3685180399 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7ff3200848b0 con 0x7ff3340c6f60 2023-10-30T14:08:57.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.013+0000 7ff338dce640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff3180664a0 0x7ff318068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:57.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.013+0000 7ff338dce640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff3180664a0 0x7ff318068960 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff328009ae0 tx=0x7ff328009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:57.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.017+0000 7ff3317fa640 1 -- 172.21.15.46:0/3685180399 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff320051530 con 0x7ff3340c6f60 2023-10-30T14:08:57.111 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:08:57.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:08:57.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 -- 172.21.15.46:0/3685180399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3341391f0 con 0x7ff3340c6f60 2023-10-30T14:08:57.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff3317fa640 1 -- 172.21.15.46:0/3685180399 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+5734 (secure 0 0 0) 0x7ff320017090 con 0x7ff3340c6f60 2023-10-30T14:08:57.213 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:57.213 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:08:54.204176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:resolve","state_seq":18,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:57.213 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 197 2023-10-30T14:08:57.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 -- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff3180664a0 msgr2=0x7ff318068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:57.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff3180664a0 0x7ff318068960 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff328009ae0 tx=0x7ff328009190 comp rx=0 tx=0).stop 2023-10-30T14:08:57.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 -- 172.21.15.46:0/3685180399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3340c6f60 msgr2=0x7ff334155cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:57.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3340c6f60 0x7ff334155cd0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7ff32000e960 tx=0x7ff32000ee20 comp rx=0 tx=0).stop 2023-10-30T14:08:57.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 -- 172.21.15.46:0/3685180399 shutdown_connections 2023-10-30T14:08:57.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff3180664a0 0x7ff318068960 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:57.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3340c8430 0x7ff3340c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:57.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3340c6f60 0x7ff334155cd0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:57.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 --2- 172.21.15.46:0/3685180399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3340c6b80 0x7ff33413d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:57.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 -- 172.21.15.46:0/3685180399 >> 172.21.15.46:0/3685180399 conn(0x7ff3340c7910 msgr2=0x7ff3341314b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:57.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 -- 172.21.15.46:0/3685180399 shutdown_connections 2023-10-30T14:08:57.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:57.209+0000 7ff33b032640 1 -- 172.21.15.46:0/3685180399 wait complete. 2023-10-30T14:08:57.234 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 197, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 197, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8505": { "addr": "172.21.15.46:6833/4078242540", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4078242540, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4078242540, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 8505, "incarnation": 189, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_8655": { "addr": "172.21.15.46:6837/3044281789", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3044281789, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3044281789, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8655, "incarnation": 196, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:resolve", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 81, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:08:54.204176+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8505, "mds_1": 8655 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:08:59.235 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:08:59.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.385+0000 7fb12493e640 1 Processor -- start 2023-10-30T14:08:59.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.385+0000 7fb12493e640 1 -- start start 2023-10-30T14:08:59.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.385+0000 7fb12493e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb12006f7a0 0x7fb12006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:59.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.385+0000 7fb12493e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb1200c6bd0 0x7fb120069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:59.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.385+0000 7fb12493e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb120071110 0x7fb120158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:59.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.385+0000 7fb12493e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1200700c0 con 0x7fb1200c6bd0 2023-10-30T14:08:59.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.385+0000 7fb12493e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb12006a3d0 con 0x7fb120071110 2023-10-30T14:08:59.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.385+0000 7fb12493e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb120066cf0 con 0x7fb12006f7a0 2023-10-30T14:08:59.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.389+0000 7fb11dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb1200c6bd0 0x7fb120069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:59.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.389+0000 7fb11dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb1200c6bd0 0x7fb120069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56002/0 (socket says 172.21.15.46:56002) 2023-10-30T14:08:59.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.389+0000 7fb11dd9b640 1 -- 172.21.15.46:0/2835337803 learned_addr learned my addr 172.21.15.46:0/2835337803 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:59.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.389+0000 7fb11ed9d640 1 --2- 172.21.15.46:0/2835337803 >> v2:172.21.15.154:3300/0 conn(0x7fb120071110 0x7fb120158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:59.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.389+0000 7fb11e59c640 1 --2- 172.21.15.46:0/2835337803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb12006f7a0 0x7fb12006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.389+0000 7fb11dd9b640 1 -- 172.21.15.46:0/2835337803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb12006f7a0 msgr2=0x7fb12006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.389+0000 7fb11dd9b640 1 --2- 172.21.15.46:0/2835337803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb12006f7a0 0x7fb12006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.389+0000 7fb11dd9b640 1 -- 172.21.15.46:0/2835337803 >> v2:172.21.15.154:3300/0 conn(0x7fb120071110 msgr2=0x7fb120158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:08:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.389+0000 7fb11dd9b640 1 --2- 172.21.15.46:0/2835337803 >> v2:172.21.15.154:3300/0 conn(0x7fb120071110 0x7fb120158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:59.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.389+0000 7fb11dd9b640 1 -- 172.21.15.46:0/2835337803 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1200c70d0 con 0x7fb1200c6bd0 2023-10-30T14:08:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.389+0000 7fb11dd9b640 1 --2- 172.21.15.46:0/2835337803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1200c6bd0 0x7fb120069e90 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fb12006c7d0 tx=0x7fb11401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc921b2b4626f249 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:59.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb11d59a640 1 -- 172.21.15.46:0/2835337803 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1140a89b0 con 0x7fb1200c6bd0 2023-10-30T14:08:59.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb11d59a640 1 -- 172.21.15.46:0/2835337803 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1140a8b10 con 0x7fb1200c6bd0 2023-10-30T14:08:59.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb11d59a640 1 -- 172.21.15.46:0/2835337803 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1140a8ea0 con 0x7fb1200c6bd0 2023-10-30T14:08:59.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 -- 172.21.15.46:0/2835337803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1200c6bd0 msgr2=0x7fb120069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:59.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 --2- 172.21.15.46:0/2835337803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1200c6bd0 0x7fb120069e90 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fb12006c7d0 tx=0x7fb11401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:08:59.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 -- 172.21.15.46:0/2835337803 shutdown_connections 2023-10-30T14:08:59.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 --2- 172.21.15.46:0/2835337803 >> v2:172.21.15.154:3300/0 conn(0x7fb120071110 0x7fb120158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:59.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 --2- 172.21.15.46:0/2835337803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1200c6bd0 0x7fb120069e90 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:59.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 --2- 172.21.15.46:0/2835337803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb12006f7a0 0x7fb12006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:59.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 -- 172.21.15.46:0/2835337803 >> 172.21.15.46:0/2835337803 conn(0x7fb1200c7910 msgr2=0x7fb1200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:59.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 -- 172.21.15.46:0/2835337803 shutdown_connections 2023-10-30T14:08:59.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 -- 172.21.15.46:0/2835337803 wait complete. 2023-10-30T14:08:59.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 Processor -- start 2023-10-30T14:08:59.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 -- start start 2023-10-30T14:08:59.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb12006f7a0 0x7fb12014ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:59.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb120071110 0x7fb12014b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:59.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1200c6bd0 0x7fb1200c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:59.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12014b4c0 con 0x7fb12006f7a0 2023-10-30T14:08:59.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12014d3a0 con 0x7fb1200c6bd0 2023-10-30T14:08:59.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb12493e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1200c8e40 con 0x7fb120071110 2023-10-30T14:08:59.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb11e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb12006f7a0 0x7fb12014ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:59.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb11e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb12006f7a0 0x7fb12014ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56010/0 (socket says 172.21.15.46:56010) 2023-10-30T14:08:59.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.393+0000 7fb11e59c640 1 -- 172.21.15.46:0/1954904799 learned_addr learned my addr 172.21.15.46:0/1954904799 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:08:59.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb11dd9b640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb120071110 0x7fb12014b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:59.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb11ed9d640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1200c6bd0 0x7fb1200c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:59.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb11e59c640 1 -- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb120071110 msgr2=0x7fb12014b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:59.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb11e59c640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb120071110 0x7fb12014b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:59.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb11e59c640 1 -- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1200c6bd0 msgr2=0x7fb1200c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:59.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb11e59c640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1200c6bd0 0x7fb1200c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:59.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb11e59c640 1 -- 172.21.15.46:0/1954904799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1140094d0 con 0x7fb12006f7a0 2023-10-30T14:08:59.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb11e59c640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb12006f7a0 0x7fb12014ce60 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fb11000e870 tx=0x7fb11000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:59.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb10effd640 1 -- 172.21.15.46:0/1954904799 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11000ee60 con 0x7fb12006f7a0 2023-10-30T14:08:59.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb10effd640 1 -- 172.21.15.46:0/1954904799 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11000c940 con 0x7fb12006f7a0 2023-10-30T14:08:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb12493e640 1 -- 172.21.15.46:0/1954904799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12014af40 con 0x7fb12006f7a0 2023-10-30T14:08:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb10effd640 1 -- 172.21.15.46:0/1954904799 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11000ccf0 con 0x7fb12006f7a0 2023-10-30T14:08:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb12493e640 1 -- 172.21.15.46:0/1954904799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12014b1e0 con 0x7fb12006f7a0 2023-10-30T14:08:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb12493e640 1 -- 172.21.15.46:0/1954904799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1200c8c10 con 0x7fb12006f7a0 2023-10-30T14:08:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb10effd640 1 -- 172.21.15.46:0/1954904799 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb110013470 con 0x7fb12006f7a0 2023-10-30T14:08:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb10effd640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb1000664a0 0x7fb100068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:08:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.397+0000 7fb10effd640 1 -- 172.21.15.46:0/1954904799 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7fb1100847b0 con 0x7fb12006f7a0 2023-10-30T14:08:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.401+0000 7fb11dd9b640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb1000664a0 0x7fb100068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:08:59.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.401+0000 7fb11dd9b640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb1000664a0 0x7fb100068960 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb1140b2000 tx=0x7fb114008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:08:59.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.401+0000 7fb10effd640 1 -- 172.21.15.46:0/1954904799 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb1200c8c10 con 0x7fb12006f7a0 2023-10-30T14:08:59.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.593+0000 7fb12493e640 1 -- 172.21.15.46:0/1954904799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1200c8c10 con 0x7fb12006f7a0 2023-10-30T14:08:59.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.593+0000 7fb10effd640 1 -- 172.21.15.46:0/1954904799 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+5734 (secure 0 0 0) 0x7fb1200c8c10 con 0x7fb12006f7a0 2023-10-30T14:08:59.599 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:08:59.599 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:08:54.204176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:resolve","state_seq":18,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:08:59.600 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 197 2023-10-30T14:08:59.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.597+0000 7fb12493e640 1 -- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb1000664a0 msgr2=0x7fb100068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:59.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.597+0000 7fb12493e640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb1000664a0 0x7fb100068960 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb1140b2000 tx=0x7fb114008040 comp rx=0 tx=0).stop 2023-10-30T14:08:59.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.597+0000 7fb12493e640 1 -- 172.21.15.46:0/1954904799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb12006f7a0 msgr2=0x7fb12014ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:08:59.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.597+0000 7fb12493e640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb12006f7a0 0x7fb12014ce60 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fb11000e870 tx=0x7fb11000ed30 comp rx=0 tx=0).stop 2023-10-30T14:08:59.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.597+0000 7fb12493e640 1 -- 172.21.15.46:0/1954904799 shutdown_connections 2023-10-30T14:08:59.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.597+0000 7fb12493e640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb1000664a0 0x7fb100068960 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:59.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.597+0000 7fb12493e640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1200c6bd0 0x7fb1200c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:59.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.597+0000 7fb12493e640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb120071110 0x7fb12014b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:59.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.597+0000 7fb12493e640 1 --2- 172.21.15.46:0/1954904799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb12006f7a0 0x7fb12014ce60 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:08:59.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.597+0000 7fb12493e640 1 -- 172.21.15.46:0/1954904799 >> 172.21.15.46:0/1954904799 conn(0x7fb1200c7910 msgr2=0x7fb12006be40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:08:59.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.597+0000 7fb12493e640 1 -- 172.21.15.46:0/1954904799 shutdown_connections 2023-10-30T14:08:59.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:08:59.597+0000 7fb12493e640 1 -- 172.21.15.46:0/1954904799 wait complete. 2023-10-30T14:08:59.619 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:09:01.621 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:01.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345f69c640 1 Processor -- start 2023-10-30T14:09:01.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345f69c640 1 -- start start 2023-10-30T14:09:01.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345f69c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3458153200 0x7f34580c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:01.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345f69c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3458071470 0x7f34580c8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:01.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345f69c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34580c6050 con 0x7f3458071a70 2023-10-30T14:09:01.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345f69c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34580c6e50 con 0x7f3458071470 2023-10-30T14:09:01.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345f69c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34580c62a0 con 0x7f3458153200 2023-10-30T14:09:01.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345cc37640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3458071470 0x7f34580c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:01.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345cc37640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3458071470 0x7f34580c8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56026/0 (socket says 172.21.15.46:56026) 2023-10-30T14:09:01.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345cc37640 1 -- 172.21.15.46:0/2925968356 learned_addr learned my addr 172.21.15.46:0/2925968356 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:01.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345d438640 1 --2- 172.21.15.46:0/2925968356 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3458153200 0x7f34580c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:01.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345cc37640 1 -- 172.21.15.46:0/2925968356 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3458153200 msgr2=0x7f34580c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:01.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345cc37640 1 --2- 172.21.15.46:0/2925968356 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3458153200 0x7f34580c7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:01.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345cc37640 1 -- 172.21.15.46:0/2925968356 >> v1:172.21.15.154:6789/0 conn(0x7f3458071a70 legacy=0x7f3458158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:01.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.773+0000 7f345cc37640 1 -- 172.21.15.46:0/2925968356 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34580c70f0 con 0x7f3458071470 2023-10-30T14:09:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.777+0000 7f345cc37640 1 --2- 172.21.15.46:0/2925968356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3458071470 0x7f34580c8440 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f3448008870 tx=0x7f344801f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69574b329eb84865 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.777+0000 7f344ffff640 1 -- 172.21.15.46:0/2925968356 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34480a7b10 con 0x7f3458071470 2023-10-30T14:09:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.777+0000 7f344ffff640 1 -- 172.21.15.46:0/2925968356 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34480ab070 con 0x7f3458071470 2023-10-30T14:09:01.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.777+0000 7f344ffff640 1 -- 172.21.15.46:0/2925968356 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34480a7e90 con 0x7f3458071470 2023-10-30T14:09:01.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.777+0000 7f345f69c640 1 -- 172.21.15.46:0/2925968356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3458071470 msgr2=0x7f34580c8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:01.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.777+0000 7f345f69c640 1 --2- 172.21.15.46:0/2925968356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3458071470 0x7f34580c8440 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f3448008870 tx=0x7f344801f480 comp rx=0 tx=0).stop 2023-10-30T14:09:01.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.777+0000 7f345f69c640 1 -- 172.21.15.46:0/2925968356 shutdown_connections 2023-10-30T14:09:01.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.777+0000 7f345f69c640 1 --2- 172.21.15.46:0/2925968356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3458071470 0x7f34580c8440 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:01.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.777+0000 7f345f69c640 1 --2- 172.21.15.46:0/2925968356 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3458153200 0x7f34580c7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:01.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.777+0000 7f345f69c640 1 -- 172.21.15.46:0/2925968356 >> 172.21.15.46:0/2925968356 conn(0x7f345806d770 msgr2=0x7f345806db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.777+0000 7f345f69c640 1 -- 172.21.15.46:0/2925968356 shutdown_connections 2023-10-30T14:09:01.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.777+0000 7f345f69c640 1 -- 172.21.15.46:0/2925968356 wait complete. 2023-10-30T14:09:01.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345f69c640 1 Processor -- start 2023-10-30T14:09:01.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345f69c640 1 -- start start 2023-10-30T14:09:01.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345f69c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3458071470 0x7f34581f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:01.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345f69c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3458071a70 0x7f345815eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:01.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345f69c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3458153200 0x7f34580c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:01.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345f69c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345815df70 con 0x7f3458153200 2023-10-30T14:09:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345f69c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34581f6210 con 0x7f3458071a70 2023-10-30T14:09:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345f69c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345815e230 con 0x7f3458071470 2023-10-30T14:09:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345dc39640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3458153200 0x7f34580c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345dc39640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3458153200 0x7f34580c43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56038/0 (socket says 172.21.15.46:56038) 2023-10-30T14:09:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345dc39640 1 -- 172.21.15.46:0/2316306851 learned_addr learned my addr 172.21.15.46:0/2316306851 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345d438640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3458071470 0x7f34581f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345cc37640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3458071a70 0x7f345815eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345dc39640 1 -- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3458071470 msgr2=0x7f34581f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345dc39640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3458071470 0x7f34581f2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345dc39640 1 -- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3458071a70 msgr2=0x7f345815eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:01.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345dc39640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3458071a70 0x7f345815eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:01.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345dc39640 1 -- 172.21.15.46:0/2316306851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3448008550 con 0x7f3458153200 2023-10-30T14:09:01.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345dc39640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3458153200 0x7f34580c43a0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f345000c730 tx=0x7f345000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:01.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f344dffb640 1 -- 172.21.15.46:0/2316306851 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450003b80 con 0x7f3458153200 2023-10-30T14:09:01.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f344dffb640 1 -- 172.21.15.46:0/2316306851 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3450003ce0 con 0x7f3458153200 2023-10-30T14:09:01.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345f69c640 1 -- 172.21.15.46:0/2316306851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345815e510 con 0x7f3458153200 2023-10-30T14:09:01.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f344dffb640 1 -- 172.21.15.46:0/2316306851 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345000aa30 con 0x7f3458153200 2023-10-30T14:09:01.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.781+0000 7f345f69c640 1 -- 172.21.15.46:0/2316306851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34581f5730 con 0x7f3458153200 2023-10-30T14:09:01.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.785+0000 7f345f69c640 1 -- 172.21.15.46:0/2316306851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3458074250 con 0x7f3458153200 2023-10-30T14:09:01.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.785+0000 7f344dffb640 1 -- 172.21.15.46:0/2316306851 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f345000ab90 con 0x7f3458153200 2023-10-30T14:09:01.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.785+0000 7f344dffb640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f343c0664d0 0x7f343c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:01.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.785+0000 7f344dffb640 1 -- 172.21.15.46:0/2316306851 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7f34500844e0 con 0x7f3458153200 2023-10-30T14:09:01.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.785+0000 7f345d438640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f343c0664d0 0x7f343c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:01.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.785+0000 7f345d438640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f343c0664d0 0x7f343c068990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3454006aa0 tx=0x7f3454008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:01.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.789+0000 7f344dffb640 1 -- 172.21.15.46:0/2316306851 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3450051040 con 0x7f3458153200 2023-10-30T14:09:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.981+0000 7f345f69c640 1 -- 172.21.15.46:0/2316306851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34580c6800 con 0x7f3458153200 2023-10-30T14:09:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.981+0000 7f344dffb640 1 -- 172.21.15.46:0/2316306851 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f3450054660 con 0x7f3458153200 2023-10-30T14:09:01.986 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:01.986 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:08:59.931089+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 198 2023-10-30T14:09:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.985+0000 7f345f69c640 1 -- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f343c0664d0 msgr2=0x7f343c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.985+0000 7f345f69c640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f343c0664d0 0x7f343c068990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3454006aa0 tx=0x7f3454008040 comp rx=0 tx=0).stop 2023-10-30T14:09:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.985+0000 7f345f69c640 1 -- 172.21.15.46:0/2316306851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3458153200 msgr2=0x7f34580c43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.985+0000 7f345f69c640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3458153200 0x7f34580c43a0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f345000c730 tx=0x7f345000cbf0 comp rx=0 tx=0).stop 2023-10-30T14:09:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.985+0000 7f345f69c640 1 -- 172.21.15.46:0/2316306851 shutdown_connections 2023-10-30T14:09:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.985+0000 7f345f69c640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f343c0664d0 0x7f343c068990 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.985+0000 7f345f69c640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3458153200 0x7f34580c43a0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.985+0000 7f345f69c640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3458071a70 0x7f345815eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.985+0000 7f345f69c640 1 --2- 172.21.15.46:0/2316306851 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3458071470 0x7f34581f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.985+0000 7f345f69c640 1 -- 172.21.15.46:0/2316306851 >> 172.21.15.46:0/2316306851 conn(0x7f345806d770 msgr2=0x7f34580b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:01.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.985+0000 7f345f69c640 1 -- 172.21.15.46:0/2316306851 shutdown_connections 2023-10-30T14:09:01.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:01.985+0000 7f345f69c640 1 -- 172.21.15.46:0/2316306851 wait complete. 2023-10-30T14:09:02.006 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:09:04.008 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:04.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.157+0000 7fb7f170f640 1 Processor -- start 2023-10-30T14:09:04.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.157+0000 7fb7f170f640 1 -- start start 2023-10-30T14:09:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7f170f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7ec0c6b80 0x7fb7ec0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7f170f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb7ec13f4a0 0x7fb7ec0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7f170f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ec0c0a50 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7f170f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ec06f6e0 con 0x7fb7ec13f4a0 2023-10-30T14:09:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7f170f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec06f9c0 con 0x7fb7ec0c6b80 2023-10-30T14:09:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7eb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb7ec13f4a0 0x7fb7ec0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7eb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb7ec13f4a0 0x7fb7ec0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56046/0 (socket says 172.21.15.46:56046) 2023-10-30T14:09:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7eb7fe640 1 -- 172.21.15.46:0/3579489554 learned_addr learned my addr 172.21.15.46:0/3579489554 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7e3fff640 1 --1- 172.21.15.46:0/3579489554 >> v1:172.21.15.46:6789/0 conn(0x7fb7ec0c6f60 0x7fb7ec13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38216/0 (socket says 172.21.15.46:38216) 2023-10-30T14:09:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7eaffd640 1 --2- 172.21.15.46:0/3579489554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7ec0c6b80 0x7fb7ec0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7eb7fe640 1 -- 172.21.15.46:0/3579489554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7ec0c6b80 msgr2=0x7fb7ec0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:04.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7eb7fe640 1 --2- 172.21.15.46:0/3579489554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7ec0c6b80 0x7fb7ec0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:04.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7eb7fe640 1 -- 172.21.15.46:0/3579489554 >> v1:172.21.15.46:6789/0 conn(0x7fb7ec0c6f60 legacy=0x7fb7ec13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:04.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.161+0000 7fb7eb7fe640 1 -- 172.21.15.46:0/3579489554 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec06fca0 con 0x7fb7ec13f4a0 2023-10-30T14:09:04.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7eb7fe640 1 --2- 172.21.15.46:0/3579489554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec13f4a0 0x7fb7ec0c0400 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4009e50 tx=0x7fb7e401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ec9d173eca328d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:04.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7ea7fc640 1 -- 172.21.15.46:0/3579489554 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e40a57f0 con 0x7fb7ec13f4a0 2023-10-30T14:09:04.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7ea7fc640 1 -- 172.21.15.46:0/3579489554 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e40a5950 con 0x7fb7ec13f4a0 2023-10-30T14:09:04.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7ea7fc640 1 -- 172.21.15.46:0/3579489554 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e40a5ce0 con 0x7fb7ec13f4a0 2023-10-30T14:09:04.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 -- 172.21.15.46:0/3579489554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec13f4a0 msgr2=0x7fb7ec0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:04.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 --2- 172.21.15.46:0/3579489554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec13f4a0 0x7fb7ec0c0400 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4009e50 tx=0x7fb7e401ca10 comp rx=0 tx=0).stop 2023-10-30T14:09:04.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 -- 172.21.15.46:0/3579489554 shutdown_connections 2023-10-30T14:09:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 --2- 172.21.15.46:0/3579489554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec13f4a0 0x7fb7ec0c0400 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 --2- 172.21.15.46:0/3579489554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7ec0c6b80 0x7fb7ec0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 -- 172.21.15.46:0/3579489554 >> 172.21.15.46:0/3579489554 conn(0x7fb7ec0c7910 msgr2=0x7fb7ec0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 -- 172.21.15.46:0/3579489554 shutdown_connections 2023-10-30T14:09:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 -- 172.21.15.46:0/3579489554 wait complete. 2023-10-30T14:09:04.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 Processor -- start 2023-10-30T14:09:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 -- start start 2023-10-30T14:09:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7ec0c6b80 0x7fb7ec135d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0c6f60 0x7fb7ec13c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7ec13f4a0 0x7fb7ec0c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec156210 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec1576e0 con 0x7fb7ec13f4a0 2023-10-30T14:09:04.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.165+0000 7fb7f170f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec1f4100 con 0x7fb7ec0c6b80 2023-10-30T14:09:04.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0c6f60 0x7fb7ec13c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:04.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0c6f60 0x7fb7ec13c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56052/0 (socket says 172.21.15.46:56052) 2023-10-30T14:09:04.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e3fff640 1 -- 172.21.15.46:0/3552454717 learned_addr learned my addr 172.21.15.46:0/3552454717 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:04.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7eaffd640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7ec0c6b80 0x7fb7ec135d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:04.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7eb7fe640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7ec13f4a0 0x7fb7ec0c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:04.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e3fff640 1 -- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7ec0c6b80 msgr2=0x7fb7ec135d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:04.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e3fff640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7ec0c6b80 0x7fb7ec135d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:04.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e3fff640 1 -- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7ec13f4a0 msgr2=0x7fb7ec0c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:04.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e3fff640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7ec13f4a0 0x7fb7ec0c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:04.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e3fff640 1 -- 172.21.15.46:0/3552454717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e4009b30 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7eaffd640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7ec0c6b80 0x7fb7ec135d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:04.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e3fff640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0c6f60 0x7fb7ec13c820 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400c530 tx=0x7fb7d400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:04.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e37fe640 1 -- 172.21.15.46:0/3552454717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4007500 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e37fe640 1 -- 172.21.15.46:0/3552454717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d4007660 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e37fe640 1 -- 172.21.15.46:0/3552454717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4004a10 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7f170f640 1 -- 172.21.15.46:0/3552454717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ec157400 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7f170f640 1 -- 172.21.15.46:0/3552454717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ec1f3620 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7f170f640 1 -- 172.21.15.46:0/3552454717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ec04e170 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.169+0000 7fb7e37fe640 1 -- 172.21.15.46:0/3552454717 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb7d4004b70 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.173+0000 7fb7e37fe640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7cc0664a0 0x7fb7cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:04.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.173+0000 7fb7e37fe640 1 -- 172.21.15.46:0/3552454717 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7fb7d4083940 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.173+0000 7fb7eaffd640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7cc0664a0 0x7fb7cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:04.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.173+0000 7fb7eaffd640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7cc0664a0 0x7fb7cc068960 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc007a70 tx=0x7fb7dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:04.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.177+0000 7fb7e37fe640 1 -- 172.21.15.46:0/3552454717 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb7d4050510 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 -- 172.21.15.46:0/3552454717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7ec0c7fb0 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7e37fe640 1 -- 172.21.15.46:0/3552454717 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+5736 (secure 0 0 0) 0x7fb7d4018070 con 0x7fb7ec0c6f60 2023-10-30T14:09:04.369 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:04.369 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:08:59.931089+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:04.370 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 198 2023-10-30T14:09:04.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 -- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7cc0664a0 msgr2=0x7fb7cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:04.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7cc0664a0 0x7fb7cc068960 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc007a70 tx=0x7fb7dc008040 comp rx=0 tx=0).stop 2023-10-30T14:09:04.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 -- 172.21.15.46:0/3552454717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0c6f60 msgr2=0x7fb7ec13c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:04.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0c6f60 0x7fb7ec13c820 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400c530 tx=0x7fb7d400c9f0 comp rx=0 tx=0).stop 2023-10-30T14:09:04.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 -- 172.21.15.46:0/3552454717 shutdown_connections 2023-10-30T14:09:04.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb7cc0664a0 0x7fb7cc068960 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:04.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7ec13f4a0 0x7fb7ec0c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:04.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0c6f60 0x7fb7ec13c820 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:04.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 --2- 172.21.15.46:0/3552454717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7ec0c6b80 0x7fb7ec135d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:04.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 -- 172.21.15.46:0/3552454717 >> 172.21.15.46:0/3552454717 conn(0x7fb7ec0c7910 msgr2=0x7fb7ec1532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:04.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 -- 172.21.15.46:0/3552454717 shutdown_connections 2023-10-30T14:09:04.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:04.365+0000 7fb7f170f640 1 -- 172.21.15.46:0/3552454717 wait complete. 2023-10-30T14:09:04.390 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:09:06.391 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:06.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.541+0000 7fb354cd6640 1 Processor -- start 2023-10-30T14:09:06.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb354cd6640 1 -- start start 2023-10-30T14:09:06.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb354cd6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb350153c30 0x7fb35006a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:06.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb354cd6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb3500cc0b0 0x7fb3500cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:06.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb354cd6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35006ade0 con 0x7fb3500b1ea0 2023-10-30T14:09:06.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb354cd6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3501596e0 con 0x7fb3500cc0b0 2023-10-30T14:09:06.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb354cd6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3501598f0 con 0x7fb350153c30 2023-10-30T14:09:06.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb347fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb3500b1ea0 0x7fb3500cb9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48280/0 (socket says 172.21.15.46:48280) 2023-10-30T14:09:06.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb347fff640 1 -- 172.21.15.46:0/688253134 learned_addr learned my addr 172.21.15.46:0/688253134 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:09:06.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34e59c640 1 --2- 172.21.15.46:0/688253134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb350153c30 0x7fb35006a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:06.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34ed9d640 1 --2- 172.21.15.46:0/688253134 >> v2:172.21.15.154:3300/0 conn(0x7fb3500cc0b0 0x7fb3500cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:06.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266574051 0 0) 0x7fb3380012a0 con 0x7fb3500b1ea0 2023-10-30T14:09:06.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb33c003530 con 0x7fb3500b1ea0 2023-10-30T14:09:06.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3784760062 0 0) 0x7fb33c003530 con 0x7fb3500b1ea0 2023-10-30T14:09:06.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb350153c30 msgr2=0x7fb35006a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:06.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 --2- 172.21.15.46:0/688253134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb350153c30 0x7fb35006a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:06.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 >> v2:172.21.15.154:3300/0 conn(0x7fb3500cc0b0 msgr2=0x7fb3500cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:06.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 --2- 172.21.15.46:0/688253134 >> v2:172.21.15.154:3300/0 conn(0x7fb3500cc0b0 0x7fb3500cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:06.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb350159b90 con 0x7fb3500b1ea0 2023-10-30T14:09:06.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fb338001990 con 0x7fb3500b1ea0 2023-10-30T14:09:06.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 >> v1:172.21.15.46:6789/0 conn(0x7fb3500b1ea0 legacy=0x7fb3500cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:06.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 --2- 172.21.15.46:0/688253134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33c003980 0x7fb33c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:06.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 --2- 172.21.15.46:0/688253134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb350153c30 0x7fb33c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:06.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 --2- 172.21.15.46:0/688253134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb33c004d10 0x7fb33c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:06.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3501596e0 con 0x7fb33c003980 2023-10-30T14:09:06.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3501598f0 con 0x7fb33c004d10 2023-10-30T14:09:06.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c00b990 con 0x7fb350153c30 2023-10-30T14:09:06.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34e59c640 1 --2- 172.21.15.46:0/688253134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33c003980 0x7fb33c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:06.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb347fff640 1 --2- 172.21.15.46:0/688253134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb350153c30 0x7fb33c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:06.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34ed9d640 1 --2- 172.21.15.46:0/688253134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb33c004d10 0x7fb33c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:06.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34e59c640 1 -- 172.21.15.46:0/688253134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb350153c30 msgr2=0x7fb33c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:06.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34e59c640 1 --2- 172.21.15.46:0/688253134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb350153c30 0x7fb33c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:06.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34e59c640 1 -- 172.21.15.46:0/688253134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb33c004d10 msgr2=0x7fb33c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:06.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34e59c640 1 --2- 172.21.15.46:0/688253134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb33c004d10 0x7fb33c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:06.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.545+0000 7fb34e59c640 1 -- 172.21.15.46:0/688253134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c003530 con 0x7fb33c003980 2023-10-30T14:09:06.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.549+0000 7fb34e59c640 1 --2- 172.21.15.46:0/688253134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33c003980 0x7fb33c003da0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fb340009ae0 tx=0x7fb34001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455a7df05a9b851c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:06.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.549+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3400a5640 con 0x7fb33c003980 2023-10-30T14:09:06.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.549+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3400a57a0 con 0x7fb33c003980 2023-10-30T14:09:06.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.549+0000 7fb34dd9b640 1 -- 172.21.15.46:0/688253134 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3400a5b50 con 0x7fb33c003980 2023-10-30T14:09:06.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.549+0000 7fb354cd6640 1 -- 172.21.15.46:0/688253134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33c003980 msgr2=0x7fb33c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:06.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.549+0000 7fb354cd6640 1 --2- 172.21.15.46:0/688253134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33c003980 0x7fb33c003da0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fb340009ae0 tx=0x7fb34001ca50 comp rx=0 tx=0).stop 2023-10-30T14:09:06.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.549+0000 7fb34ed9d640 1 -- 172.21.15.46:0/688253134 reap_dead start 2023-10-30T14:09:06.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.549+0000 7fb354cd6640 1 -- 172.21.15.46:0/688253134 shutdown_connections 2023-10-30T14:09:06.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.549+0000 7fb354cd6640 1 -- 172.21.15.46:0/688253134 >> 172.21.15.46:0/688253134 conn(0x7fb35006bf40 msgr2=0x7fb35006c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:06.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.549+0000 7fb354cd6640 1 -- 172.21.15.46:0/688253134 shutdown_connections 2023-10-30T14:09:06.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.549+0000 7fb354cd6640 1 -- 172.21.15.46:0/688253134 wait complete. 2023-10-30T14:09:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb354cd6640 1 Processor -- start 2023-10-30T14:09:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb354cd6640 1 -- start start 2023-10-30T14:09:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb354cd6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb33c003980 0x7fb35015aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb354cd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb350156e10 0x7fb35015b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:06.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb354cd6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35015cec0 0x7fb35015d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:06.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb354cd6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb350149c40 con 0x7fb350156e10 2023-10-30T14:09:06.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb354cd6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35014c880 con 0x7fb35015cec0 2023-10-30T14:09:06.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb354cd6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35015a840 con 0x7fb33c003980 2023-10-30T14:09:06.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb347fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb350156e10 0x7fb35015b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:06.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb347fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb350156e10 0x7fb35015b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37540/0 (socket says 172.21.15.46:37540) 2023-10-30T14:09:06.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb347fff640 1 -- 172.21.15.46:0/1098992284 learned_addr learned my addr 172.21.15.46:0/1098992284 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:06.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb34e59c640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb33c003980 0x7fb35015aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:06.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb34ed9d640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35015cec0 0x7fb35015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:06.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb347fff640 1 -- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb33c003980 msgr2=0x7fb35015aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:06.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb347fff640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb33c003980 0x7fb35015aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:06.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb347fff640 1 -- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35015cec0 msgr2=0x7fb35015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:06.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb347fff640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35015cec0 0x7fb35015d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:06.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb347fff640 1 -- 172.21.15.46:0/1098992284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb350149960 con 0x7fb350156e10 2023-10-30T14:09:06.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb34ed9d640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35015cec0 0x7fb35015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:06.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb347fff640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb350156e10 0x7fb35015b000 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fb338011760 tx=0x7fb338011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:06.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb346ffd640 1 -- 172.21.15.46:0/1098992284 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33800fb60 con 0x7fb350156e10 2023-10-30T14:09:06.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb346ffd640 1 -- 172.21.15.46:0/1098992284 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33800fcc0 con 0x7fb350156e10 2023-10-30T14:09:06.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb346ffd640 1 -- 172.21.15.46:0/1098992284 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338015650 con 0x7fb350156e10 2023-10-30T14:09:06.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb354cd6640 1 -- 172.21.15.46:0/1098992284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35014c230 con 0x7fb350156e10 2023-10-30T14:09:06.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.553+0000 7fb354cd6640 1 -- 172.21.15.46:0/1098992284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35015d880 con 0x7fb350156e10 2023-10-30T14:09:06.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.557+0000 7fb354cd6640 1 -- 172.21.15.46:0/1098992284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35014ca30 con 0x7fb350156e10 2023-10-30T14:09:06.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.557+0000 7fb346ffd640 1 -- 172.21.15.46:0/1098992284 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb3380157b0 con 0x7fb350156e10 2023-10-30T14:09:06.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.557+0000 7fb346ffd640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3300664d0 0x7fb330068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:06.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.557+0000 7fb346ffd640 1 -- 172.21.15.46:0/1098992284 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7fb338087870 con 0x7fb350156e10 2023-10-30T14:09:06.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.557+0000 7fb34e59c640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3300664d0 0x7fb330068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:06.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.561+0000 7fb34e59c640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3300664d0 0x7fb330068990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb3400af000 tx=0x7fb3400046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:06.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.561+0000 7fb346ffd640 1 -- 172.21.15.46:0/1098992284 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb35014ca30 con 0x7fb350156e10 2023-10-30T14:09:06.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.757+0000 7fb354cd6640 1 -- 172.21.15.46:0/1098992284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb35006ade0 con 0x7fb350156e10 2023-10-30T14:09:06.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.757+0000 7fb346ffd640 1 -- 172.21.15.46:0/1098992284 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fb35006ade0 con 0x7fb350156e10 2023-10-30T14:09:06.761 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:06.762 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:05.580920+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:06.762 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 199 2023-10-30T14:09:06.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.757+0000 7fb354cd6640 1 -- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3300664d0 msgr2=0x7fb330068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:06.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.757+0000 7fb354cd6640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3300664d0 0x7fb330068990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb3400af000 tx=0x7fb3400046d0 comp rx=0 tx=0).stop 2023-10-30T14:09:06.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.757+0000 7fb354cd6640 1 -- 172.21.15.46:0/1098992284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb350156e10 msgr2=0x7fb35015b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:06.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.757+0000 7fb354cd6640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb350156e10 0x7fb35015b000 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fb338011760 tx=0x7fb338011c20 comp rx=0 tx=0).stop 2023-10-30T14:09:06.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.761+0000 7fb354cd6640 1 -- 172.21.15.46:0/1098992284 shutdown_connections 2023-10-30T14:09:06.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.761+0000 7fb354cd6640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3300664d0 0x7fb330068990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:06.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.761+0000 7fb354cd6640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35015cec0 0x7fb35015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:06.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.761+0000 7fb354cd6640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb350156e10 0x7fb35015b000 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:06.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.761+0000 7fb354cd6640 1 --2- 172.21.15.46:0/1098992284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb33c003980 0x7fb35015aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:06.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.761+0000 7fb354cd6640 1 -- 172.21.15.46:0/1098992284 >> 172.21.15.46:0/1098992284 conn(0x7fb35006bf40 msgr2=0x7fb3500c4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:06.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.761+0000 7fb354cd6640 1 -- 172.21.15.46:0/1098992284 shutdown_connections 2023-10-30T14:09:06.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:06.761+0000 7fb354cd6640 1 -- 172.21.15.46:0/1098992284 wait complete. 2023-10-30T14:09:06.783 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:09:08.784 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:08.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7e2b2a640 1 Processor -- start 2023-10-30T14:09:08.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7e2b2a640 1 -- start start 2023-10-30T14:09:08.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7e2b2a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa7dc1538a0 0x7fa7dc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:08.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7e2b2a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa7dc0ca0e0 0x7fa7dc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:08.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7e2b2a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7dc0cf160 0x7fa7dc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:08.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7e2b2a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7dc0cb800 con 0x7fa7dc0ca0e0 2023-10-30T14:09:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7e2b2a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7dc15dc30 con 0x7fa7dc1538a0 2023-10-30T14:09:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7e2b2a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0cb330 con 0x7fa7dc0cf160 2023-10-30T14:09:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7dbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa7dc0ca0e0 0x7fa7dc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7dbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa7dc0ca0e0 0x7fa7dc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37550/0 (socket says 172.21.15.46:37550) 2023-10-30T14:09:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7dbfff640 1 -- 172.21.15.46:0/401395139 learned_addr learned my addr 172.21.15.46:0/401395139 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7e10c7640 1 --2- 172.21.15.46:0/401395139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7dc0cf160 0x7fa7dc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7e08c6640 1 --2- 172.21.15.46:0/401395139 >> v2:172.21.15.154:3300/0 conn(0x7fa7dc1538a0 0x7fa7dc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7dbfff640 1 -- 172.21.15.46:0/401395139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7dc0cf160 msgr2=0x7fa7dc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7dbfff640 1 --2- 172.21.15.46:0/401395139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7dc0cf160 0x7fa7dc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7dbfff640 1 -- 172.21.15.46:0/401395139 >> v2:172.21.15.154:3300/0 conn(0x7fa7dc1538a0 msgr2=0x7fa7dc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7dbfff640 1 --2- 172.21.15.46:0/401395139 >> v2:172.21.15.154:3300/0 conn(0x7fa7dc1538a0 0x7fa7dc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7dbfff640 1 -- 172.21.15.46:0/401395139 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc0cad70 con 0x7fa7dc0ca0e0 2023-10-30T14:09:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7e10c7640 1 --2- 172.21.15.46:0/401395139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7dc0cf160 0x7fa7dc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.937+0000 7fa7e08c6640 1 --2- 172.21.15.46:0/401395139 >> v2:172.21.15.154:3300/0 conn(0x7fa7dc1538a0 0x7fa7dc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:08.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7dbfff640 1 --2- 172.21.15.46:0/401395139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7dc0ca0e0 0x7fa7dc15a090 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0cfd10 tx=0x7fa7c801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15992bfb9ffc0ac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:08.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7db7fe640 1 -- 172.21.15.46:0/401395139 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80a89b0 con 0x7fa7dc0ca0e0 2023-10-30T14:09:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7db7fe640 1 -- 172.21.15.46:0/401395139 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c80a8b10 con 0x7fa7dc0ca0e0 2023-10-30T14:09:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7db7fe640 1 -- 172.21.15.46:0/401395139 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80a8ea0 con 0x7fa7dc0ca0e0 2023-10-30T14:09:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/401395139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7dc0ca0e0 msgr2=0x7fa7dc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7e2b2a640 1 --2- 172.21.15.46:0/401395139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7dc0ca0e0 0x7fa7dc15a090 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0cfd10 tx=0x7fa7c801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:09:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/401395139 shutdown_connections 2023-10-30T14:09:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7e2b2a640 1 --2- 172.21.15.46:0/401395139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7dc0cf160 0x7fa7dc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7e2b2a640 1 --2- 172.21.15.46:0/401395139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7dc0ca0e0 0x7fa7dc15a090 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7e2b2a640 1 --2- 172.21.15.46:0/401395139 >> v2:172.21.15.154:3300/0 conn(0x7fa7dc1538a0 0x7fa7dc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/401395139 >> 172.21.15.46:0/401395139 conn(0x7fa7dc06bfe0 msgr2=0x7fa7dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:08.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/401395139 shutdown_connections 2023-10-30T14:09:08.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.941+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/401395139 wait complete. 2023-10-30T14:09:08.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e2b2a640 1 Processor -- start 2023-10-30T14:09:08.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e2b2a640 1 -- start start 2023-10-30T14:09:08.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e2b2a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7dc0ca0e0 0x7fa7dc148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e2b2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7dc0cf160 0x7fa7dc146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e2b2a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7dc1538a0 0x7fa7dc147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e2b2a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0cef70 con 0x7fa7dc0cf160 2023-10-30T14:09:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e2b2a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0ce3f0 con 0x7fa7dc0ca0e0 2023-10-30T14:09:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e2b2a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc15ac60 con 0x7fa7dc1538a0 2023-10-30T14:09:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7dbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7dc0cf160 0x7fa7dc146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7dbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7dc0cf160 0x7fa7dc146e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37562/0 (socket says 172.21.15.46:37562) 2023-10-30T14:09:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7dbfff640 1 -- 172.21.15.46:0/3764840860 learned_addr learned my addr 172.21.15.46:0/3764840860 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e08c6640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7dc0ca0e0 0x7fa7dc148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e10c7640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7dc1538a0 0x7fa7dc147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7dbfff640 1 -- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7dc1538a0 msgr2=0x7fa7dc147390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7dbfff640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7dc1538a0 0x7fa7dc147390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7dbfff640 1 -- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7dc0ca0e0 msgr2=0x7fa7dc148f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7dbfff640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7dc0ca0e0 0x7fa7dc148f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7dbfff640 1 -- 172.21.15.46:0/3764840860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c80094d0 con 0x7fa7dc0cf160 2023-10-30T14:09:08.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e10c7640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7dc1538a0 0x7fa7dc147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:08.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7dbfff640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7dc0cf160 0x7fa7dc146e50 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fa7c801d3a0 tx=0x7fa7c801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:08.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7d97fa640 1 -- 172.21.15.46:0/3764840860 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80ba600 con 0x7fa7dc0cf160 2023-10-30T14:09:08.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7d97fa640 1 -- 172.21.15.46:0/3764840860 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c80ba760 con 0x7fa7dc0cf160 2023-10-30T14:09:08.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/3764840860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7dc0ce640 con 0x7fa7dc0cf160 2023-10-30T14:09:08.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7d97fa640 1 -- 172.21.15.46:0/3764840860 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80bab10 con 0x7fa7dc0cf160 2023-10-30T14:09:08.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.945+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/3764840860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7dc0cec00 con 0x7fa7dc0cf160 2023-10-30T14:09:08.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.949+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/3764840860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7dc0b2a50 con 0x7fa7dc0cf160 2023-10-30T14:09:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.949+0000 7fa7d97fa640 1 -- 172.21.15.46:0/3764840860 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7c80bac70 con 0x7fa7dc0cf160 2023-10-30T14:09:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.949+0000 7fa7d97fa640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7c00664a0 0x7fa7c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.949+0000 7fa7d97fa640 1 -- 172.21.15.46:0/3764840860 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7fa7c8122bc0 con 0x7fa7dc0cf160 2023-10-30T14:09:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.949+0000 7fa7e08c6640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7c00664a0 0x7fa7c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.949+0000 7fa7e08c6640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7c00664a0 0x7fa7c0068960 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00045f0 tx=0x7fa7d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:08.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:08.953+0000 7fa7d97fa640 1 -- 172.21.15.46:0/3764840860 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7c80ef830 con 0x7fa7dc0cf160 2023-10-30T14:09:09.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.157+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/3764840860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7dc04e170 con 0x7fa7dc0cf160 2023-10-30T14:09:09.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.157+0000 7fa7d97fa640 1 -- 172.21.15.46:0/3764840860 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fa7c80b5070 con 0x7fa7dc0cf160 2023-10-30T14:09:09.162 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:09.162 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:05.580920+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:09.162 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 199 2023-10-30T14:09:09.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.161+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7c00664a0 msgr2=0x7fa7c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:09.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.161+0000 7fa7e2b2a640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7c00664a0 0x7fa7c0068960 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00045f0 tx=0x7fa7d0008040 comp rx=0 tx=0).stop 2023-10-30T14:09:09.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.161+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/3764840860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7dc0cf160 msgr2=0x7fa7dc146e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:09.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.161+0000 7fa7e2b2a640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7dc0cf160 0x7fa7dc146e50 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fa7c801d3a0 tx=0x7fa7c801ba90 comp rx=0 tx=0).stop 2023-10-30T14:09:09.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.161+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/3764840860 shutdown_connections 2023-10-30T14:09:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.161+0000 7fa7e2b2a640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7c00664a0 0x7fa7c0068960 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.161+0000 7fa7e2b2a640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7dc1538a0 0x7fa7dc147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.161+0000 7fa7e2b2a640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7dc0cf160 0x7fa7dc146e50 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.161+0000 7fa7e2b2a640 1 --2- 172.21.15.46:0/3764840860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7dc0ca0e0 0x7fa7dc148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.161+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/3764840860 >> 172.21.15.46:0/3764840860 conn(0x7fa7dc06bfe0 msgr2=0x7fa7dc06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.161+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/3764840860 shutdown_connections 2023-10-30T14:09:09.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:09.161+0000 7fa7e2b2a640 1 -- 172.21.15.46:0/3764840860 wait complete. 2023-10-30T14:09:09.183 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 199, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 199, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8505": { "addr": "172.21.15.46:6833/4078242540", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 4078242540, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 4078242540, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 8505, "incarnation": 189, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_8655": { "addr": "172.21.15.46:6837/3044281789", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3044281789, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3044281789, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8655, "incarnation": 196, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 81, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:09:05.580920+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8505, "mds_1": 8655 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:09:11.185 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.349+0000 7f898d5c4640 1 Processor -- start 2023-10-30T14:09:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.349+0000 7f898d5c4640 1 -- start start 2023-10-30T14:09:11.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f898d5c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89880c3c20 0x7f898815a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f898d5c4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f898806fa00 0x7f89880ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f898d5c4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898815a640 con 0x7f89880cd2c0 2023-10-30T14:09:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f898d5c4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89880c86a0 con 0x7f898806fa00 2023-10-30T14:09:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f898d5c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898815cc50 con 0x7f89880c3c20 2023-10-30T14:09:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f897ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f898806fa00 0x7f89880ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f897ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f898806fa00 0x7f89880ccd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37574/0 (socket says 172.21.15.46:37574) 2023-10-30T14:09:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f897ffff640 1 -- 172.21.15.46:0/2705062747 learned_addr learned my addr 172.21.15.46:0/2705062747 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f8986ffd640 1 --2- 172.21.15.46:0/2705062747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89880c3c20 0x7f898815a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f897ffff640 1 -- 172.21.15.46:0/2705062747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89880c3c20 msgr2=0x7f898815a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f897ffff640 1 --2- 172.21.15.46:0/2705062747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89880c3c20 0x7f898815a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f897ffff640 1 -- 172.21.15.46:0/2705062747 >> v1:172.21.15.154:6789/0 conn(0x7f89880cd2c0 legacy=0x7f898815d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.353+0000 7f897ffff640 1 -- 172.21.15.46:0/2705062747 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89880c88f0 con 0x7f898806fa00 2023-10-30T14:09:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f897ffff640 1 --2- 172.21.15.46:0/2705062747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f898806fa00 0x7f89880ccd80 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f8970009870 tx=0x7f897001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dab5fc6858e0c07 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f89867fc640 1 -- 172.21.15.46:0/2705062747 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89700a8a30 con 0x7f898806fa00 2023-10-30T14:09:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f89867fc640 1 -- 172.21.15.46:0/2705062747 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89700a8b90 con 0x7f898806fa00 2023-10-30T14:09:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f89867fc640 1 -- 172.21.15.46:0/2705062747 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89700b0440 con 0x7f898806fa00 2023-10-30T14:09:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 -- 172.21.15.46:0/2705062747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f898806fa00 msgr2=0x7f89880ccd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 --2- 172.21.15.46:0/2705062747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f898806fa00 0x7f89880ccd80 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f8970009870 tx=0x7f897001ed30 comp rx=0 tx=0).stop 2023-10-30T14:09:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 -- 172.21.15.46:0/2705062747 shutdown_connections 2023-10-30T14:09:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 --2- 172.21.15.46:0/2705062747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f898806fa00 0x7f89880ccd80 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 --2- 172.21.15.46:0/2705062747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89880c3c20 0x7f898815a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 -- 172.21.15.46:0/2705062747 >> 172.21.15.46:0/2705062747 conn(0x7f8988071790 msgr2=0x7f8988071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 -- 172.21.15.46:0/2705062747 shutdown_connections 2023-10-30T14:09:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 -- 172.21.15.46:0/2705062747 wait complete. 2023-10-30T14:09:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 Processor -- start 2023-10-30T14:09:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 -- start start 2023-10-30T14:09:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f898806fa00 0x7f8988156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89880c3c20 0x7f8988155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.357+0000 7f898d5c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89880cd2c0 0x7f8988157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:11.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f898d5c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898815c930 con 0x7f89880cd2c0 2023-10-30T14:09:11.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f898d5c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8988154b30 con 0x7f898806fa00 2023-10-30T14:09:11.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f898d5c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8988157620 con 0x7f89880c3c20 2023-10-30T14:09:11.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f89877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89880cd2c0 0x7f8988157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:11.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f897ffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89880c3c20 0x7f8988155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:11.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f89877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89880cd2c0 0x7f8988157de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37580/0 (socket says 172.21.15.46:37580) 2023-10-30T14:09:11.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f89877fe640 1 -- 172.21.15.46:0/4232872206 learned_addr learned my addr 172.21.15.46:0/4232872206 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:11.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f8986ffd640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f898806fa00 0x7f8988156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:11.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f89877fe640 1 -- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89880c3c20 msgr2=0x7f8988155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:11.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f89877fe640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89880c3c20 0x7f8988155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f89877fe640 1 -- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f898806fa00 msgr2=0x7f8988156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f89877fe640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f898806fa00 0x7f8988156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f89877fe640 1 -- 172.21.15.46:0/4232872206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89700094d0 con 0x7f89880cd2c0 2023-10-30T14:09:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f89877fe640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89880cd2c0 0x7f8988157de0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f898000c770 tx=0x7f898000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f897f7fe640 1 -- 172.21.15.46:0/4232872206 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980003b80 con 0x7f89880cd2c0 2023-10-30T14:09:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f897f7fe640 1 -- 172.21.15.46:0/4232872206 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8980003ce0 con 0x7f89880cd2c0 2023-10-30T14:09:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f897f7fe640 1 -- 172.21.15.46:0/4232872206 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898000aa90 con 0x7f89880cd2c0 2023-10-30T14:09:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f898d5c4640 1 -- 172.21.15.46:0/4232872206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8988159240 con 0x7f89880cd2c0 2023-10-30T14:09:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f898d5c4640 1 -- 172.21.15.46:0/4232872206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89881578c0 con 0x7f89880cd2c0 2023-10-30T14:09:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f898d5c4640 1 -- 172.21.15.46:0/4232872206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8988157ab0 con 0x7f89880cd2c0 2023-10-30T14:09:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.361+0000 7f897f7fe640 1 -- 172.21.15.46:0/4232872206 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f898000abf0 con 0x7f89880cd2c0 2023-10-30T14:09:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.365+0000 7f897f7fe640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89680664a0 0x7f8968068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.365+0000 7f897f7fe640 1 -- 172.21.15.46:0/4232872206 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7f89800843a0 con 0x7f89880cd2c0 2023-10-30T14:09:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.365+0000 7f8986ffd640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89680664a0 0x7f8968068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.365+0000 7f8986ffd640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89680664a0 0x7f8968068960 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f8978007a60 tx=0x7f8978008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.369+0000 7f897f7fe640 1 -- 172.21.15.46:0/4232872206 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f89800510a0 con 0x7f89880cd2c0 2023-10-30T14:09:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 -- 172.21.15.46:0/4232872206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f898804e170 con 0x7f89880cd2c0 2023-10-30T14:09:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f897f7fe640 1 -- 172.21.15.46:0/4232872206 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f89800546c0 con 0x7f89880cd2c0 2023-10-30T14:09:11.565 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:11.565 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:05.580920+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:11.565 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 199 2023-10-30T14:09:11.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 -- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89680664a0 msgr2=0x7f8968068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:11.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89680664a0 0x7f8968068960 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f8978007a60 tx=0x7f8978008040 comp rx=0 tx=0).stop 2023-10-30T14:09:11.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 -- 172.21.15.46:0/4232872206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89880cd2c0 msgr2=0x7f8988157de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:11.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89880cd2c0 0x7f8988157de0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f898000c770 tx=0x7f898000cc30 comp rx=0 tx=0).stop 2023-10-30T14:09:11.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 -- 172.21.15.46:0/4232872206 shutdown_connections 2023-10-30T14:09:11.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89680664a0 0x7f8968068960 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:11.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89880cd2c0 0x7f8988157de0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:11.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89880c3c20 0x7f8988155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:11.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 --2- 172.21.15.46:0/4232872206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f898806fa00 0x7f8988156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:11.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 -- 172.21.15.46:0/4232872206 >> 172.21.15.46:0/4232872206 conn(0x7f8988071790 msgr2=0x7f89880cb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:11.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 -- 172.21.15.46:0/4232872206 shutdown_connections 2023-10-30T14:09:11.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:11.561+0000 7f898d5c4640 1 -- 172.21.15.46:0/4232872206 wait complete. 2023-10-30T14:09:11.586 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:09:13.588 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:13.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f845172a640 1 Processor -- start 2023-10-30T14:09:13.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f845172a640 1 -- start start 2023-10-30T14:09:13.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f845172a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f844c0c8cf0 0x7f844c0c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f845172a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f844c0c90d0 0x7f844c0ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f845172a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f844c0cef90 0x7f844c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f845172a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844c15ce30 con 0x7f844c0c8cf0 2023-10-30T14:09:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f845172a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844c159880 con 0x7f844c0cef90 2023-10-30T14:09:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f845172a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c159ae0 con 0x7f844c0c90d0 2023-10-30T14:09:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f844affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f844c0c8cf0 0x7f844c0c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f844affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f844c0c8cf0 0x7f844c0c9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37594/0 (socket says 172.21.15.46:37594) 2023-10-30T14:09:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f844affd640 1 -- 172.21.15.46:0/2724237845 learned_addr learned my addr 172.21.15.46:0/2724237845 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f844b7fe640 1 --2- 172.21.15.46:0/2724237845 >> v2:172.21.15.154:3300/0 conn(0x7f844c0cef90 0x7f844c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f844a7fc640 1 --2- 172.21.15.46:0/2724237845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f844c0c90d0 0x7f844c0ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:13.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f844affd640 1 -- 172.21.15.46:0/2724237845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f844c0c90d0 msgr2=0x7f844c0ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:13.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f844affd640 1 --2- 172.21.15.46:0/2724237845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f844c0c90d0 0x7f844c0ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f844affd640 1 -- 172.21.15.46:0/2724237845 >> v2:172.21.15.154:3300/0 conn(0x7f844c0cef90 msgr2=0x7f844c0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:13.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f844affd640 1 --2- 172.21.15.46:0/2724237845 >> v2:172.21.15.154:3300/0 conn(0x7f844c0cef90 0x7f844c0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.741+0000 7f844affd640 1 -- 172.21.15.46:0/2724237845 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c159dc0 con 0x7f844c0c8cf0 2023-10-30T14:09:13.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f844affd640 1 --2- 172.21.15.46:0/2724237845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f844c0c8cf0 0x7f844c0c9e00 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f844c06c9f0 tx=0x7f843c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4709b05c7e4ae9b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:13.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f8449ffb640 1 -- 172.21.15.46:0/2724237845 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c0aa5f0 con 0x7f844c0c8cf0 2023-10-30T14:09:13.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f8449ffb640 1 -- 172.21.15.46:0/2724237845 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c0aa750 con 0x7f844c0c8cf0 2023-10-30T14:09:13.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f8449ffb640 1 -- 172.21.15.46:0/2724237845 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c0aaae0 con 0x7f844c0c8cf0 2023-10-30T14:09:13.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f845172a640 1 -- 172.21.15.46:0/2724237845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f844c0c8cf0 msgr2=0x7f844c0c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:13.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f845172a640 1 --2- 172.21.15.46:0/2724237845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f844c0c8cf0 0x7f844c0c9e00 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f844c06c9f0 tx=0x7f843c01c4d0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f845172a640 1 -- 172.21.15.46:0/2724237845 shutdown_connections 2023-10-30T14:09:13.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f845172a640 1 --2- 172.21.15.46:0/2724237845 >> v2:172.21.15.154:3300/0 conn(0x7f844c0cef90 0x7f844c0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f845172a640 1 --2- 172.21.15.46:0/2724237845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f844c0c90d0 0x7f844c0ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f845172a640 1 --2- 172.21.15.46:0/2724237845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f844c0c8cf0 0x7f844c0c9e00 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f845172a640 1 -- 172.21.15.46:0/2724237845 >> 172.21.15.46:0/2724237845 conn(0x7f844c06be40 msgr2=0x7f844c06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:13.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f845172a640 1 -- 172.21.15.46:0/2724237845 shutdown_connections 2023-10-30T14:09:13.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f845172a640 1 -- 172.21.15.46:0/2724237845 wait complete. 2023-10-30T14:09:13.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.745+0000 7f845172a640 1 Processor -- start 2023-10-30T14:09:13.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f845172a640 1 -- start start 2023-10-30T14:09:13.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f845172a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f844c0c8cf0 0x7f844c14e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:13.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f845172a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f844c0c90d0 0x7f844c14c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:13.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f845172a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f844c0cef90 0x7f844c14d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:13.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f845172a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c1f5270 con 0x7f844c0cef90 2023-10-30T14:09:13.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f845172a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c1f5b10 con 0x7f844c0c8cf0 2023-10-30T14:09:13.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f845172a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c14e8c0 con 0x7f844c0c90d0 2023-10-30T14:09:13.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f844b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f844c0cef90 0x7f844c14d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f844b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f844c0cef90 0x7f844c14d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37610/0 (socket says 172.21.15.46:37610) 2023-10-30T14:09:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f844b7fe640 1 -- 172.21.15.46:0/1436230149 learned_addr learned my addr 172.21.15.46:0/1436230149 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f844a7fc640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f844c0c90d0 0x7f844c14c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f844affd640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f844c0c8cf0 0x7f844c14e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:13.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f844b7fe640 1 -- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f844c0c90d0 msgr2=0x7f844c14c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:13.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f844b7fe640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f844c0c90d0 0x7f844c14c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f844b7fe640 1 -- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f844c0c8cf0 msgr2=0x7f844c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:13.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f844b7fe640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f844c0c8cf0 0x7f844c14e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f844b7fe640 1 -- 172.21.15.46:0/1436230149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c0094d0 con 0x7f844c0cef90 2023-10-30T14:09:13.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f844b7fe640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f844c0cef90 0x7f844c14d760 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f8444007e40 tx=0x7f844400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:13.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f84337fe640 1 -- 172.21.15.46:0/1436230149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444014620 con 0x7f844c0cef90 2023-10-30T14:09:13.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f84337fe640 1 -- 172.21.15.46:0/1436230149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844400ee90 con 0x7f844c0cef90 2023-10-30T14:09:13.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f84337fe640 1 -- 172.21.15.46:0/1436230149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84440149e0 con 0x7f844c0cef90 2023-10-30T14:09:13.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f845172a640 1 -- 172.21.15.46:0/1436230149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c14eba0 con 0x7f844c0cef90 2023-10-30T14:09:13.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f845172a640 1 -- 172.21.15.46:0/1436230149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c1f57a0 con 0x7f844c0cef90 2023-10-30T14:09:13.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.749+0000 7f845172a640 1 -- 172.21.15.46:0/1436230149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c1f5960 con 0x7f844c0cef90 2023-10-30T14:09:13.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.753+0000 7f84337fe640 1 -- 172.21.15.46:0/1436230149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8444014b40 con 0x7f844c0cef90 2023-10-30T14:09:13.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.753+0000 7f84337fe640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84380664a0 0x7f8438068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:13.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.753+0000 7f84337fe640 1 -- 172.21.15.46:0/1436230149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7f8444084fe0 con 0x7f844c0cef90 2023-10-30T14:09:13.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.753+0000 7f844affd640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84380664a0 0x7f8438068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:13.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.753+0000 7f844affd640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84380664a0 0x7f8438068960 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f843c0ae000 tx=0x7f843c0b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:13.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.757+0000 7f84337fe640 1 -- 172.21.15.46:0/1436230149 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8444051bf0 con 0x7f844c0cef90 2023-10-30T14:09:13.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.949+0000 7f845172a640 1 -- 172.21.15.46:0/1436230149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f844c1f5c80 con 0x7f844c0cef90 2023-10-30T14:09:13.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.949+0000 7f84337fe640 1 -- 172.21.15.46:0/1436230149 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f8444055210 con 0x7f844c0cef90 2023-10-30T14:09:13.953 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:13.954 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:11.299927+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:13.954 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 200 2023-10-30T14:09:13.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.949+0000 7f845172a640 1 -- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84380664a0 msgr2=0x7f8438068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:13.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.949+0000 7f845172a640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84380664a0 0x7f8438068960 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f843c0ae000 tx=0x7f843c0b5040 comp rx=0 tx=0).stop 2023-10-30T14:09:13.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.949+0000 7f845172a640 1 -- 172.21.15.46:0/1436230149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f844c0cef90 msgr2=0x7f844c14d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:13.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.949+0000 7f845172a640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f844c0cef90 0x7f844c14d760 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f8444007e40 tx=0x7f844400e4b0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.953+0000 7f845172a640 1 -- 172.21.15.46:0/1436230149 shutdown_connections 2023-10-30T14:09:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.953+0000 7f845172a640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84380664a0 0x7f8438068960 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.953+0000 7f845172a640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f844c0cef90 0x7f844c14d760 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.953+0000 7f845172a640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f844c0c90d0 0x7f844c14c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.953+0000 7f845172a640 1 --2- 172.21.15.46:0/1436230149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f844c0c8cf0 0x7f844c14e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.953+0000 7f845172a640 1 -- 172.21.15.46:0/1436230149 >> 172.21.15.46:0/1436230149 conn(0x7f844c06be40 msgr2=0x7f844c0c20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:13.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.953+0000 7f845172a640 1 -- 172.21.15.46:0/1436230149 shutdown_connections 2023-10-30T14:09:13.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:13.953+0000 7f845172a640 1 -- 172.21.15.46:0/1436230149 wait complete. 2023-10-30T14:09:13.975 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=1, replacing gid=8352 2023-10-30T14:09:13.975 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.c 2023-10-30T14:09:18.528 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:09:18.529 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 2 -eq 48 ] 2023-10-30T14:09:18.529 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.2 2023-10-30T14:09:18.529 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.2 2023-10-30T14:09:18.530 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.2 2023-10-30T14:09:18.530 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.2 2023-10-30T14:09:18.530 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:09:19.976 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T14:09:19.976 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T14:09:19.977 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T14:09:19.981 INFO:tasks.ceph.mds.c:Started 2023-10-30T14:09:19.981 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:20.099 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-10-30T14:09:20.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.133+0000 7f42ad9a7640 1 Processor -- start 2023-10-30T14:09:20.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ad9a7640 1 -- start start 2023-10-30T14:09:20.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ad9a7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f42a81538f0 0x7f42a8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:20.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ad9a7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f42a815cd50 0x7f42a80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:20.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ad9a7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a815d300 con 0x7f42a80cefa0 2023-10-30T14:09:20.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ad9a7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a815d5e0 con 0x7f42a81538f0 2023-10-30T14:09:20.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ad9a7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a815d8c0 con 0x7f42a815cd50 2023-10-30T14:09:20.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ac9a5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f42a81538f0 0x7f42a8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:20.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ac9a5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f42a81538f0 0x7f42a8159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44338/0 (socket says 172.21.15.46:44338) 2023-10-30T14:09:20.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ac9a5640 1 -- 172.21.15.46:0/2653501642 learned_addr learned my addr 172.21.15.46:0/2653501642 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:20.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42a7fff640 1 --2- 172.21.15.46:0/2653501642 >> v2:172.21.15.154:3300/0 conn(0x7f42a815cd50 0x7f42a80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:20.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ac9a5640 1 -- 172.21.15.46:0/2653501642 >> v2:172.21.15.154:3300/0 conn(0x7f42a815cd50 msgr2=0x7f42a80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:20.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ac9a5640 1 --2- 172.21.15.46:0/2653501642 >> v2:172.21.15.154:3300/0 conn(0x7f42a815cd50 0x7f42a80c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:20.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ac9a5640 1 -- 172.21.15.46:0/2653501642 >> v1:172.21.15.154:6789/0 conn(0x7f42a80cefa0 legacy=0x7f42a80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:20.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.137+0000 7f42ac9a5640 1 -- 172.21.15.46:0/2653501642 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a815dba0 con 0x7f42a81538f0 2023-10-30T14:09:20.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42ac9a5640 1 --2- 172.21.15.46:0/2653501642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42a81538f0 0x7f42a8159fb0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f42a8072e90 tx=0x7f429c01ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74629d8c1b21ee18 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:20.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42a77fe640 1 -- 172.21.15.46:0/2653501642 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c0a89b0 con 0x7f42a81538f0 2023-10-30T14:09:20.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42a77fe640 1 -- 172.21.15.46:0/2653501642 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429c0a8b10 con 0x7f42a81538f0 2023-10-30T14:09:20.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42a77fe640 1 -- 172.21.15.46:0/2653501642 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c0a8ea0 con 0x7f42a81538f0 2023-10-30T14:09:20.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42ad9a7640 1 -- 172.21.15.46:0/2653501642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42a81538f0 msgr2=0x7f42a8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:20.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42ad9a7640 1 --2- 172.21.15.46:0/2653501642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42a81538f0 0x7f42a8159fb0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f42a8072e90 tx=0x7f429c01ecb0 comp rx=0 tx=0).stop 2023-10-30T14:09:20.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42ad9a7640 1 -- 172.21.15.46:0/2653501642 shutdown_connections 2023-10-30T14:09:20.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42ad9a7640 1 --2- 172.21.15.46:0/2653501642 >> v2:172.21.15.154:3300/0 conn(0x7f42a815cd50 0x7f42a80c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:20.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42ad9a7640 1 --2- 172.21.15.46:0/2653501642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42a81538f0 0x7f42a8159fb0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:20.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42ad9a7640 1 -- 172.21.15.46:0/2653501642 >> 172.21.15.46:0/2653501642 conn(0x7f42a806a640 msgr2=0x7f42a8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:20.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42ad9a7640 1 -- 172.21.15.46:0/2653501642 shutdown_connections 2023-10-30T14:09:20.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42ad9a7640 1 -- 172.21.15.46:0/2653501642 wait complete. 2023-10-30T14:09:20.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.141+0000 7f42ad9a7640 1 Processor -- start 2023-10-30T14:09:20.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ad9a7640 1 -- start start 2023-10-30T14:09:20.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ad9a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42a80cefa0 0x7f42a81f4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:20.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ad9a7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42a81538f0 0x7f42a81573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:20.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ad9a7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42a815cd50 0x7f42a81523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:20.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ad9a7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a815a6d0 con 0x7f42a80cefa0 2023-10-30T14:09:20.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ad9a7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a81f4660 con 0x7f42a815cd50 2023-10-30T14:09:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ad9a7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a814f880 con 0x7f42a81538f0 2023-10-30T14:09:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ac9a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42a80cefa0 0x7f42a81f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ac9a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42a80cefa0 0x7f42a81f4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44340/0 (socket says 172.21.15.46:44340) 2023-10-30T14:09:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ac9a5640 1 -- 172.21.15.46:0/613077154 learned_addr learned my addr 172.21.15.46:0/613077154 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42a7fff640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42a81538f0 0x7f42a81573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:20.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ad1a6640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42a815cd50 0x7f42a81523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:20.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ac9a5640 1 -- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42a81538f0 msgr2=0x7f42a81573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:20.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ac9a5640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42a81538f0 0x7f42a81573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:20.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ac9a5640 1 -- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42a815cd50 msgr2=0x7f42a81523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:20.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ac9a5640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42a815cd50 0x7f42a81523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:20.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ac9a5640 1 -- 172.21.15.46:0/613077154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c0094d0 con 0x7f42a80cefa0 2023-10-30T14:09:20.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ac9a5640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42a80cefa0 0x7f42a81f4120 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f429c0b3040 tx=0x7f429c01ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:20.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42a57fa640 1 -- 172.21.15.46:0/613077154 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c0ba600 con 0x7f42a80cefa0 2023-10-30T14:09:20.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42a57fa640 1 -- 172.21.15.46:0/613077154 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429c0ba760 con 0x7f42a80cefa0 2023-10-30T14:09:20.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42a57fa640 1 -- 172.21.15.46:0/613077154 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c0bab10 con 0x7f42a80cefa0 2023-10-30T14:09:20.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ad9a7640 1 -- 172.21.15.46:0/613077154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a815af30 con 0x7f42a80cefa0 2023-10-30T14:09:20.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ad9a7640 1 -- 172.21.15.46:0/613077154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a814fad0 con 0x7f42a80cefa0 2023-10-30T14:09:20.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.145+0000 7f42ad9a7640 1 -- 172.21.15.46:0/613077154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a804e170 con 0x7f42a80cefa0 2023-10-30T14:09:20.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.149+0000 7f42a57fa640 1 -- 172.21.15.46:0/613077154 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f429c0bac70 con 0x7f42a80cefa0 2023-10-30T14:09:20.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.149+0000 7f42a57fa640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f428c0664d0 0x7f428c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:20.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.149+0000 7f42a57fa640 1 -- 172.21.15.46:0/613077154 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7f429c122cd0 con 0x7f42a80cefa0 2023-10-30T14:09:20.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.149+0000 7f42a7fff640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f428c0664d0 0x7f428c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:20.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.149+0000 7f42a7fff640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f428c0664d0 0x7f428c068990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f42a80c4bf0 tx=0x7f4294008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.153+0000 7f42a57fa640 1 -- 172.21.15.46:0/613077154 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f429c0ef8a0 con 0x7f42a80cefa0 2023-10-30T14:09:20.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42ad9a7640 1 -- 172.21.15.46:0/613077154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42a804e170 con 0x7f42a80cefa0 2023-10-30T14:09:20.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42a57fa640 1 -- 172.21.15.46:0/613077154 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f429c0b5070 con 0x7f42a80cefa0 2023-10-30T14:09:20.349 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:20.349 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:11.299927+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:20.349 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 200 2023-10-30T14:09:20.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42ad9a7640 1 -- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f428c0664d0 msgr2=0x7f428c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:20.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42ad9a7640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f428c0664d0 0x7f428c068990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f42a80c4bf0 tx=0x7f4294008040 comp rx=0 tx=0).stop 2023-10-30T14:09:20.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42ad9a7640 1 -- 172.21.15.46:0/613077154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42a80cefa0 msgr2=0x7f42a81f4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:20.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42ad9a7640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42a80cefa0 0x7f42a81f4120 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f429c0b3040 tx=0x7f429c01ba50 comp rx=0 tx=0).stop 2023-10-30T14:09:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42ad9a7640 1 -- 172.21.15.46:0/613077154 shutdown_connections 2023-10-30T14:09:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42ad9a7640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f428c0664d0 0x7f428c068990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42ad9a7640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42a815cd50 0x7f42a81523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42ad9a7640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42a81538f0 0x7f42a81573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42ad9a7640 1 --2- 172.21.15.46:0/613077154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42a80cefa0 0x7f42a81f4120 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:20.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42ad9a7640 1 -- 172.21.15.46:0/613077154 >> 172.21.15.46:0/613077154 conn(0x7f42a806a640 msgr2=0x7f42a8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:20.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.345+0000 7f42ad9a7640 1 -- 172.21.15.46:0/613077154 shutdown_connections 2023-10-30T14:09:20.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:20.349+0000 7f42ad9a7640 1 -- 172.21.15.46:0/613077154 wait complete. 2023-10-30T14:09:20.374 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T14:09:22.375 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:22.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.517+0000 7f9831b01640 1 Processor -- start 2023-10-30T14:09:22.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.517+0000 7f9831b01640 1 -- start start 2023-10-30T14:09:22.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.517+0000 7f9831b01640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f982c0c6bd0 0x7f982c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:22.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.517+0000 7f9831b01640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f982c071110 0x7f982c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:22.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.517+0000 7f9831b01640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c066cf0 con 0x7f982c06f7a0 2023-10-30T14:09:22.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.517+0000 7f9831b01640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982c0700c0 con 0x7f982c0c6bd0 2023-10-30T14:09:22.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.517+0000 7f9831b01640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982c0c70d0 con 0x7f982c071110 2023-10-30T14:09:22.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f982c0c6bd0 0x7f982c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:22.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f982c06f7a0 0x7f982c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56748/0 (socket says 172.21.15.46:56748) 2023-10-30T14:09:22.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982b7fe640 1 -- 172.21.15.46:0/2507493284 learned_addr learned my addr 172.21.15.46:0/2507493284 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:09:22.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982bfff640 1 --2- 172.21.15.46:0/2507493284 >> v2:172.21.15.154:3300/0 conn(0x7f982c071110 0x7f982c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:22.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982affd640 1 -- 172.21.15.46:0/2507493284 >> v2:172.21.15.154:3300/0 conn(0x7f982c071110 msgr2=0x7f982c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:22.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982affd640 1 --2- 172.21.15.46:0/2507493284 >> v2:172.21.15.154:3300/0 conn(0x7f982c071110 0x7f982c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:22.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982affd640 1 -- 172.21.15.46:0/2507493284 >> v1:172.21.15.46:6789/0 conn(0x7f982c06f7a0 legacy=0x7f982c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:22.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982affd640 1 -- 172.21.15.46:0/2507493284 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c0c7390 con 0x7f982c0c6bd0 2023-10-30T14:09:22.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982a7fc640 1 -- 172.21.15.46:0/2507493284 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112338211 0 0) 0x7f982c066cf0 con 0x7f982c06f7a0 2023-10-30T14:09:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982affd640 1 --2- 172.21.15.46:0/2507493284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f982c0c6bd0 0x7f982c06fb80 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f982c06cdd0 tx=0x7f982001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecfcafb1cf2822e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982a7fc640 1 -- 172.21.15.46:0/2507493284 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98200a8950 con 0x7f982c0c6bd0 2023-10-30T14:09:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982a7fc640 1 -- 172.21.15.46:0/2507493284 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98200a8ab0 con 0x7f982c0c6bd0 2023-10-30T14:09:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.521+0000 7f982a7fc640 1 -- 172.21.15.46:0/2507493284 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98200a8e40 con 0x7f982c0c6bd0 2023-10-30T14:09:22.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 -- 172.21.15.46:0/2507493284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f982c0c6bd0 msgr2=0x7f982c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:22.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 --2- 172.21.15.46:0/2507493284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f982c0c6bd0 0x7f982c06fb80 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f982c06cdd0 tx=0x7f982001ec50 comp rx=0 tx=0).stop 2023-10-30T14:09:22.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 -- 172.21.15.46:0/2507493284 shutdown_connections 2023-10-30T14:09:22.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 --2- 172.21.15.46:0/2507493284 >> v2:172.21.15.154:3300/0 conn(0x7f982c071110 0x7f982c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:22.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 --2- 172.21.15.46:0/2507493284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f982c0c6bd0 0x7f982c06fb80 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:22.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 -- 172.21.15.46:0/2507493284 >> 172.21.15.46:0/2507493284 conn(0x7f982c0c7910 msgr2=0x7f982c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:22.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 -- 172.21.15.46:0/2507493284 shutdown_connections 2023-10-30T14:09:22.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 -- 172.21.15.46:0/2507493284 wait complete. 2023-10-30T14:09:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 Processor -- start 2023-10-30T14:09:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 -- start start 2023-10-30T14:09:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f982c06f7a0 0x7f982c153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c071110 0x7f982c157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f982c0c6bd0 0x7f982c151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c1efd00 con 0x7f982c06f7a0 2023-10-30T14:09:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c14dc40 con 0x7f982c071110 2023-10-30T14:09:22.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f9831b01640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982c1effa0 con 0x7f982c0c6bd0 2023-10-30T14:09:22.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f982b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f982c06f7a0 0x7f982c153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:22.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f982b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f982c06f7a0 0x7f982c153840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44362/0 (socket says 172.21.15.46:44362) 2023-10-30T14:09:22.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f982b7fe640 1 -- 172.21.15.46:0/483334205 learned_addr learned my addr 172.21.15.46:0/483334205 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:22.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f982affd640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c071110 0x7f982c157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f982bfff640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f982c0c6bd0 0x7f982c151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f982b7fe640 1 -- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f982c0c6bd0 msgr2=0x7f982c151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f982b7fe640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f982c0c6bd0 0x7f982c151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f982b7fe640 1 -- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c071110 msgr2=0x7f982c157730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.525+0000 7f982b7fe640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c071110 0x7f982c157730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.529+0000 7f982b7fe640 1 -- 172.21.15.46:0/483334205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9820009420 con 0x7f982c06f7a0 2023-10-30T14:09:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.529+0000 7f982affd640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c071110 0x7f982c157730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.529+0000 7f982b7fe640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f982c06f7a0 0x7f982c153840 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f981c00c480 tx=0x7f981c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.529+0000 7f9813fff640 1 -- 172.21.15.46:0/483334205 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c0074e0 con 0x7f982c06f7a0 2023-10-30T14:09:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.529+0000 7f9813fff640 1 -- 172.21.15.46:0/483334205 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981c007660 con 0x7f982c06f7a0 2023-10-30T14:09:22.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.529+0000 7f9813fff640 1 -- 172.21.15.46:0/483334205 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c004a30 con 0x7f982c06f7a0 2023-10-30T14:09:22.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.529+0000 7f9831b01640 1 -- 172.21.15.46:0/483334205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982c1f0280 con 0x7f982c06f7a0 2023-10-30T14:09:22.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.529+0000 7f9831b01640 1 -- 172.21.15.46:0/483334205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982c14df20 con 0x7f982c06f7a0 2023-10-30T14:09:22.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.529+0000 7f9831b01640 1 -- 172.21.15.46:0/483334205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982c04e170 con 0x7f982c06f7a0 2023-10-30T14:09:22.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.529+0000 7f9813fff640 1 -- 172.21.15.46:0/483334205 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f981c004b90 con 0x7f982c06f7a0 2023-10-30T14:09:22.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.533+0000 7f9813fff640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98180664a0 0x7f9818068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.533+0000 7f9813fff640 1 -- 172.21.15.46:0/483334205 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7f981c0838e0 con 0x7f982c06f7a0 2023-10-30T14:09:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.533+0000 7f982affd640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98180664a0 0x7f9818068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.533+0000 7f982affd640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98180664a0 0x7f9818068960 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f98200b2000 tx=0x7f98200aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:22.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.537+0000 7f9813fff640 1 -- 172.21.15.46:0/483334205 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f981c053f40 con 0x7f982c06f7a0 2023-10-30T14:09:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.733+0000 7f9831b01640 1 -- 172.21.15.46:0/483334205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f982c0c6620 con 0x7f982c06f7a0 2023-10-30T14:09:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9813fff640 1 -- 172.21.15.46:0/483334205 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f981c018090 con 0x7f982c06f7a0 2023-10-30T14:09:22.740 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:22.740 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":201,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:11.299927+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 201 2023-10-30T14:09:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9831b01640 1 -- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98180664a0 msgr2=0x7f9818068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9831b01640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98180664a0 0x7f9818068960 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f98200b2000 tx=0x7f98200aa070 comp rx=0 tx=0).stop 2023-10-30T14:09:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9831b01640 1 -- 172.21.15.46:0/483334205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f982c06f7a0 msgr2=0x7f982c153840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9831b01640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f982c06f7a0 0x7f982c153840 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f981c00c480 tx=0x7f981c00c940 comp rx=0 tx=0).stop 2023-10-30T14:09:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9831b01640 1 -- 172.21.15.46:0/483334205 shutdown_connections 2023-10-30T14:09:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9831b01640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98180664a0 0x7f9818068960 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9831b01640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f982c0c6bd0 0x7f982c151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9831b01640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c071110 0x7f982c157730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9831b01640 1 --2- 172.21.15.46:0/483334205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f982c06f7a0 0x7f982c153840 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9831b01640 1 -- 172.21.15.46:0/483334205 >> 172.21.15.46:0/483334205 conn(0x7f982c0c7910 msgr2=0x7f982c0b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9831b01640 1 -- 172.21.15.46:0/483334205 shutdown_connections 2023-10-30T14:09:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:22.737+0000 7f9831b01640 1 -- 172.21.15.46:0/483334205 wait complete. 2023-10-30T14:09:22.760 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T14:09:22.761 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 13 secs before thrashing 2023-10-30T14:09:23.382 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:09:23.382 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.3 2023-10-30T14:09:23.383 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 3 -eq 48 ] 2023-10-30T14:09:23.383 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.3 2023-10-30T14:09:23.383 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.3 2023-10-30T14:09:23.383 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.3 2023-10-30T14:09:23.384 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:09:27.254 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:09:27.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:09:30.416 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:09:30.416 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 4 -eq 48 ] 2023-10-30T14:09:30.417 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.4 2023-10-30T14:09:30.417 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.4 2023-10-30T14:09:30.418 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.4 2023-10-30T14:09:30.418 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.4 2023-10-30T14:09:30.418 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:09:32.373 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:09:32.374 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 5 -eq 48 ] 2023-10-30T14:09:32.374 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.5 2023-10-30T14:09:32.374 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.5 2023-10-30T14:09:32.375 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.5 2023-10-30T14:09:32.375 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.5 2023-10-30T14:09:32.375 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:09:34.752 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:09:34.752 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 6 -eq 48 ] 2023-10-30T14:09:34.753 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.6 2023-10-30T14:09:34.753 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.6 2023-10-30T14:09:34.754 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.6 2023-10-30T14:09:34.754 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.6 2023-10-30T14:09:34.754 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:09:35.762 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:35.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f685570f640 1 Processor -- start 2023-10-30T14:09:35.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f685570f640 1 -- start start 2023-10-30T14:09:35.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f685570f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6850159e60 0x7f685006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:35.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f685570f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f68500bfec0 0x7f68500c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:35.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f685570f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f685006a720 0x7f685006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:35.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f685570f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f685006fa80 con 0x7f685006a720 2023-10-30T14:09:35.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f685570f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6850071290 con 0x7f68500bfec0 2023-10-30T14:09:35.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f685570f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6850071570 con 0x7f6850159e60 2023-10-30T14:09:35.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f684f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f685006a720 0x7f685006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:35.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f684f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f685006a720 0x7f685006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38676/0 (socket says 172.21.15.46:38676) 2023-10-30T14:09:35.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f684f7fe640 1 -- 172.21.15.46:0/329931347 learned_addr learned my addr 172.21.15.46:0/329931347 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:35.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f684effd640 1 --2- 172.21.15.46:0/329931347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6850159e60 0x7f685006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:35.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f6847fff640 1 --2- 172.21.15.46:0/329931347 >> v2:172.21.15.154:3300/0 conn(0x7f68500bfec0 0x7f68500c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:35.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f684f7fe640 1 -- 172.21.15.46:0/329931347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6850159e60 msgr2=0x7f685006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:35.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f684f7fe640 1 --2- 172.21.15.46:0/329931347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6850159e60 0x7f685006f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:35.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f684f7fe640 1 -- 172.21.15.46:0/329931347 >> v2:172.21.15.154:3300/0 conn(0x7f68500bfec0 msgr2=0x7f68500c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:35.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f684f7fe640 1 --2- 172.21.15.46:0/329931347 >> v2:172.21.15.154:3300/0 conn(0x7f68500bfec0 0x7f68500c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:35.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f684f7fe640 1 -- 172.21.15.46:0/329931347 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6850071850 con 0x7f685006a720 2023-10-30T14:09:35.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.917+0000 7f684effd640 1 --2- 172.21.15.46:0/329931347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6850159e60 0x7f685006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:35.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f684f7fe640 1 --2- 172.21.15.46:0/329931347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f685006a720 0x7f685006abb0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f685006f230 tx=0x7f68480202d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7473dfad74f63059 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:35.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f684e7fc640 1 -- 172.21.15.46:0/329931347 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68480b0400 con 0x7f685006a720 2023-10-30T14:09:35.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f684e7fc640 1 -- 172.21.15.46:0/329931347 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68480ac050 con 0x7f685006a720 2023-10-30T14:09:35.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f684e7fc640 1 -- 172.21.15.46:0/329931347 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68480b0780 con 0x7f685006a720 2023-10-30T14:09:35.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f685570f640 1 -- 172.21.15.46:0/329931347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f685006a720 msgr2=0x7f685006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:35.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f685570f640 1 --2- 172.21.15.46:0/329931347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f685006a720 0x7f685006abb0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f685006f230 tx=0x7f68480202d0 comp rx=0 tx=0).stop 2023-10-30T14:09:35.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f685570f640 1 -- 172.21.15.46:0/329931347 shutdown_connections 2023-10-30T14:09:35.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f685570f640 1 --2- 172.21.15.46:0/329931347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f685006a720 0x7f685006abb0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:35.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f685570f640 1 --2- 172.21.15.46:0/329931347 >> v2:172.21.15.154:3300/0 conn(0x7f68500bfec0 0x7f68500c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:35.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f685570f640 1 --2- 172.21.15.46:0/329931347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6850159e60 0x7f685006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:35.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f685570f640 1 -- 172.21.15.46:0/329931347 >> 172.21.15.46:0/329931347 conn(0x7f68500c7910 msgr2=0x7f68500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:35.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f685570f640 1 -- 172.21.15.46:0/329931347 shutdown_connections 2023-10-30T14:09:35.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.921+0000 7f685570f640 1 -- 172.21.15.46:0/329931347 wait complete. 2023-10-30T14:09:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f685570f640 1 Processor -- start 2023-10-30T14:09:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f685570f640 1 -- start start 2023-10-30T14:09:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f685570f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f685006a720 0x7f685014f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f685570f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68500bfec0 0x7f6850150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f685570f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6850159e60 0x7f685014d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:35.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f685570f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6850153420 con 0x7f68500bfec0 2023-10-30T14:09:35.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f685570f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6850152100 con 0x7f685006a720 2023-10-30T14:09:35.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f685570f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6850151580 con 0x7f6850159e60 2023-10-30T14:09:35.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f6847fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68500bfec0 0x7f6850150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:35.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f6847fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68500bfec0 0x7f6850150c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38690/0 (socket says 172.21.15.46:38690) 2023-10-30T14:09:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f6847fff640 1 -- 172.21.15.46:0/3226925363 learned_addr learned my addr 172.21.15.46:0/3226925363 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f684f7fe640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6850159e60 0x7f685014d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f684effd640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f685006a720 0x7f685014f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f6847fff640 1 -- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6850159e60 msgr2=0x7f685014d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f6847fff640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6850159e60 0x7f685014d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f6847fff640 1 -- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f685006a720 msgr2=0x7f685014f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f6847fff640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f685006a720 0x7f685014f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f6847fff640 1 -- 172.21.15.46:0/3226925363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6848009b30 con 0x7f68500bfec0 2023-10-30T14:09:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f684f7fe640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6850159e60 0x7f685014d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:35.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f6847fff640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68500bfec0 0x7f6850150c60 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f683800d800 tx=0x7f683800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:35.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f68477fe640 1 -- 172.21.15.46:0/3226925363 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6838009640 con 0x7f68500bfec0 2023-10-30T14:09:35.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f68477fe640 1 -- 172.21.15.46:0/3226925363 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68380097a0 con 0x7f68500bfec0 2023-10-30T14:09:35.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f68477fe640 1 -- 172.21.15.46:0/3226925363 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6838009b10 con 0x7f68500bfec0 2023-10-30T14:09:35.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f685570f640 1 -- 172.21.15.46:0/3226925363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6850151860 con 0x7f68500bfec0 2023-10-30T14:09:35.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.925+0000 7f685570f640 1 -- 172.21.15.46:0/3226925363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6850151e20 con 0x7f68500bfec0 2023-10-30T14:09:35.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.929+0000 7f685570f640 1 -- 172.21.15.46:0/3226925363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685004e170 con 0x7f68500bfec0 2023-10-30T14:09:35.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.929+0000 7f68477fe640 1 -- 172.21.15.46:0/3226925363 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6838009c70 con 0x7f68500bfec0 2023-10-30T14:09:35.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.929+0000 7f68477fe640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68300664a0 0x7f6830068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:35.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.929+0000 7f68477fe640 1 -- 172.21.15.46:0/3226925363 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7f6838084f00 con 0x7f68500bfec0 2023-10-30T14:09:35.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.929+0000 7f684effd640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68300664a0 0x7f6830068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:35.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.929+0000 7f684effd640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68300664a0 0x7f6830068960 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f684000ba10 tx=0x7f6840008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:35.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:35.933+0000 7f68477fe640 1 -- 172.21.15.46:0/3226925363 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6838051a40 con 0x7f68500bfec0 2023-10-30T14:09:36.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.133+0000 7f685570f640 1 -- 172.21.15.46:0/3226925363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68500ca310 con 0x7f68500bfec0 2023-10-30T14:09:36.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f68477fe640 1 -- 172.21.15.46:0/3226925363 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f6838055060 con 0x7f68500bfec0 2023-10-30T14:09:36.140 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:36.140 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:26.948035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:36.140 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 202 2023-10-30T14:09:36.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f685570f640 1 -- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68300664a0 msgr2=0x7f6830068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:36.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f685570f640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68300664a0 0x7f6830068960 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f684000ba10 tx=0x7f6840008040 comp rx=0 tx=0).stop 2023-10-30T14:09:36.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f685570f640 1 -- 172.21.15.46:0/3226925363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68500bfec0 msgr2=0x7f6850150c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:36.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f685570f640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68500bfec0 0x7f6850150c60 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f683800d800 tx=0x7f683800dcc0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f685570f640 1 -- 172.21.15.46:0/3226925363 shutdown_connections 2023-10-30T14:09:36.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f685570f640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68300664a0 0x7f6830068960 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f685570f640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6850159e60 0x7f685014d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f685570f640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68500bfec0 0x7f6850150c60 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f685570f640 1 --2- 172.21.15.46:0/3226925363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f685006a720 0x7f685014f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f685570f640 1 -- 172.21.15.46:0/3226925363 >> 172.21.15.46:0/3226925363 conn(0x7f68500c7910 msgr2=0x7f68500c9850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:36.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f685570f640 1 -- 172.21.15.46:0/3226925363 shutdown_connections 2023-10-30T14:09:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.137+0000 7f685570f640 1 -- 172.21.15.46:0/3226925363 wait complete. 2023-10-30T14:09:36.161 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T14:09:36.161 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T14:09:36.161 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:09:36.191 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T14:09:36.192 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:36.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.373+0000 7f0d3c446640 1 Processor -- start 2023-10-30T14:09:36.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.373+0000 7f0d3c446640 1 -- start start 2023-10-30T14:09:36.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.373+0000 7f0d3c446640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3406d290 0x7f0d3406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:36.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.373+0000 7f0d3c446640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d340c6c10 con 0x7f0d3415a410 2023-10-30T14:09:36.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.373+0000 7f0d3c446640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d340c6ef0 con 0x7f0d34069e90 2023-10-30T14:09:36.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.373+0000 7f0d3c446640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d340c8570 con 0x7f0d3406d290 2023-10-30T14:09:36.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.373+0000 7f0d399e1640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0d3415a410 0x7f0d3406cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52112/0 (socket says 172.21.15.46:52112) 2023-10-30T14:09:36.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.373+0000 7f0d399e1640 1 -- 172.21.15.46:0/3099726109 learned_addr learned my addr 172.21.15.46:0/3099726109 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:09:36.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.373+0000 7f0d3a9e3640 1 --2- 172.21.15.46:0/3099726109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3406d290 0x7f0d3406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:36.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.373+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669720044 0 0) 0x7f0d340c6c10 con 0x7f0d3415a410 2023-10-30T14:09:36.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d20003530 con 0x7f0d3415a410 2023-10-30T14:09:36.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2718901609 0 0) 0x7f0d20003530 con 0x7f0d3415a410 2023-10-30T14:09:36.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3406d290 msgr2=0x7f0d3406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:36.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 --2- 172.21.15.46:0/3099726109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3406d290 0x7f0d3406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 >> v1:172.21.15.154:6789/0 conn(0x7f0d34069e90 legacy=0x7f0d340c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:36.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d340c71d0 con 0x7f0d3415a410 2023-10-30T14:09:36.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f0d24001680 con 0x7f0d3415a410 2023-10-30T14:09:36.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 >> v1:172.21.15.46:6789/0 conn(0x7f0d3415a410 legacy=0x7f0d3406cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:36.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 --2- 172.21.15.46:0/3099726109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d200039b0 0x7f0d20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:36.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 --2- 172.21.15.46:0/3099726109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d200047d0 0x7f0d20004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:36.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 --2- 172.21.15.46:0/3099726109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3406d290 0x7f0d20005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:36.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d340c8570 con 0x7f0d200039b0 2023-10-30T14:09:36.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2000ba40 con 0x7f0d200047d0 2023-10-30T14:09:36.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2000bc00 con 0x7f0d3406d290 2023-10-30T14:09:36.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d3a1e2640 1 --2- 172.21.15.46:0/3099726109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d200039b0 0x7f0d20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:36.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d399e1640 1 --2- 172.21.15.46:0/3099726109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d200047d0 0x7f0d20004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:36.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d3a9e3640 1 --2- 172.21.15.46:0/3099726109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3406d290 0x7f0d20005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:36.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d3a1e2640 1 -- 172.21.15.46:0/3099726109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3406d290 msgr2=0x7f0d20005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:36.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d3a1e2640 1 --2- 172.21.15.46:0/3099726109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3406d290 0x7f0d20005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d3a1e2640 1 -- 172.21.15.46:0/3099726109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d200047d0 msgr2=0x7f0d20004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:36.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d3a1e2640 1 --2- 172.21.15.46:0/3099726109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d200047d0 0x7f0d20004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d3a1e2640 1 -- 172.21.15.46:0/3099726109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d20003530 con 0x7f0d200039b0 2023-10-30T14:09:36.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.377+0000 7f0d3a1e2640 1 --2- 172.21.15.46:0/3099726109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d200039b0 0x7f0d20003dd0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f0d30001750 tx=0x7f0d3001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd611b836f2a148d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:36.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d300052d0 con 0x7f0d200039b0 2023-10-30T14:09:36.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d300a7050 con 0x7f0d200039b0 2023-10-30T14:09:36.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d391e0640 1 -- 172.21.15.46:0/3099726109 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30003600 con 0x7f0d200039b0 2023-10-30T14:09:36.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 -- 172.21.15.46:0/3099726109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d200039b0 msgr2=0x7f0d20003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 --2- 172.21.15.46:0/3099726109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d200039b0 0x7f0d20003dd0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f0d30001750 tx=0x7f0d3001c3a0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3a9e3640 1 -- 172.21.15.46:0/3099726109 reap_dead start 2023-10-30T14:09:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 -- 172.21.15.46:0/3099726109 shutdown_connections 2023-10-30T14:09:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 -- 172.21.15.46:0/3099726109 >> 172.21.15.46:0/3099726109 conn(0x7f0d3406dfd0 msgr2=0x7f0d3406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 -- 172.21.15.46:0/3099726109 shutdown_connections 2023-10-30T14:09:36.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 -- 172.21.15.46:0/3099726109 wait complete. 2023-10-30T14:09:36.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 Processor -- start 2023-10-30T14:09:36.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 -- start start 2023-10-30T14:09:36.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d200039b0 0x7f0d340cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:36.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d340c5fb0 0x7f0d340ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:36.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3415b9a0 0x7f0d3415be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:36.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d340cb970 con 0x7f0d340c5fb0 2023-10-30T14:09:36.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3415c360 con 0x7f0d200039b0 2023-10-30T14:09:36.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.381+0000 7f0d3c446640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341f2c60 con 0x7f0d3415b9a0 2023-10-30T14:09:36.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d399e1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d340c5fb0 0x7f0d340ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:36.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d399e1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d340c5fb0 0x7f0d340ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38710/0 (socket says 172.21.15.46:38710) 2023-10-30T14:09:36.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d399e1640 1 -- 172.21.15.46:0/3857076144 learned_addr learned my addr 172.21.15.46:0/3857076144 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:36.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d3a9e3640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3415b9a0 0x7f0d3415be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:36.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d3a1e2640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d200039b0 0x7f0d340cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:36.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d399e1640 1 -- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3415b9a0 msgr2=0x7f0d3415be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:36.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d399e1640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3415b9a0 0x7f0d3415be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d399e1640 1 -- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d200039b0 msgr2=0x7f0d340cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:36.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d399e1640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d200039b0 0x7f0d340cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d399e1640 1 -- 172.21.15.46:0/3857076144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d341f1e00 con 0x7f0d340c5fb0 2023-10-30T14:09:36.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d399e1640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d340c5fb0 0x7f0d340ca870 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400e830 tx=0x7f0d2400ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:36.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d2affd640 1 -- 172.21.15.46:0/3857076144 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2400c6c0 con 0x7f0d340c5fb0 2023-10-30T14:09:36.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d2affd640 1 -- 172.21.15.46:0/3857076144 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2400c820 con 0x7f0d340c5fb0 2023-10-30T14:09:36.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d3c446640 1 -- 172.21.15.46:0/3857076144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d341f20e0 con 0x7f0d340c5fb0 2023-10-30T14:09:36.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d2affd640 1 -- 172.21.15.46:0/3857076144 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2400cbd0 con 0x7f0d340c5fb0 2023-10-30T14:09:36.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d3c446640 1 -- 172.21.15.46:0/3857076144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d341f26a0 con 0x7f0d340c5fb0 2023-10-30T14:09:36.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d3c446640 1 -- 172.21.15.46:0/3857076144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d3404e170 con 0x7f0d340c5fb0 2023-10-30T14:09:36.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.385+0000 7f0d2affd640 1 -- 172.21.15.46:0/3857076144 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0d2401d460 con 0x7f0d340c5fb0 2023-10-30T14:09:36.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.389+0000 7f0d2affd640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0d180664d0 0x7f0d18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:36.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.389+0000 7f0d2affd640 1 -- 172.21.15.46:0/3857076144 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7f0d24084820 con 0x7f0d340c5fb0 2023-10-30T14:09:36.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.389+0000 7f0d3a1e2640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0d180664d0 0x7f0d18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:36.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.389+0000 7f0d3a1e2640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0d180664d0 0x7f0d18068990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0d200057b0 tx=0x7f0d300011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:36.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.389+0000 7f0d2affd640 1 -- 172.21.15.46:0/3857076144 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0d24051380 con 0x7f0d340c5fb0 2023-10-30T14:09:36.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.577+0000 7f0d3c446640 1 -- 172.21.15.46:0/3857076144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d3415b700 con 0x7f0d340c5fb0 2023-10-30T14:09:36.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d2affd640 1 -- 172.21.15.46:0/3857076144 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f0d240549a0 con 0x7f0d340c5fb0 2023-10-30T14:09:36.584 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:26.948035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:36.584 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 202 2023-10-30T14:09:36.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d3c446640 1 -- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0d180664d0 msgr2=0x7f0d18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:36.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d3c446640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0d180664d0 0x7f0d18068990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0d200057b0 tx=0x7f0d300011a0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d3c446640 1 -- 172.21.15.46:0/3857076144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d340c5fb0 msgr2=0x7f0d340ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:36.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d3c446640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d340c5fb0 0x7f0d340ca870 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400e830 tx=0x7f0d2400ecf0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d3c446640 1 -- 172.21.15.46:0/3857076144 shutdown_connections 2023-10-30T14:09:36.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d3c446640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0d180664d0 0x7f0d18068990 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d3c446640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d3415b9a0 0x7f0d3415be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d3c446640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d340c5fb0 0x7f0d340ca870 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d3c446640 1 --2- 172.21.15.46:0/3857076144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d200039b0 0x7f0d340cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:36.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d3c446640 1 -- 172.21.15.46:0/3857076144 >> 172.21.15.46:0/3857076144 conn(0x7f0d3406dfd0 msgr2=0x7f0d34159410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:36.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d3c446640 1 -- 172.21.15.46:0/3857076144 shutdown_connections 2023-10-30T14:09:36.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:36.581+0000 7f0d3c446640 1 -- 172.21.15.46:0/3857076144 wait complete. 2023-10-30T14:09:36.605 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:09:38.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:38.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49fe6b640 1 Processor -- start 2023-10-30T14:09:38.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49fe6b640 1 -- start start 2023-10-30T14:09:38.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49fe6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc498070740 0x7fc4980c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:38.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49fe6b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc49815e700 0x7fc49806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:38.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49fe6b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4980c5810 con 0x7fc4981580e0 2023-10-30T14:09:38.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49fe6b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc498070b60 con 0x7fc49815e700 2023-10-30T14:09:38.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49fe6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49815eb90 con 0x7fc498070740 2023-10-30T14:09:38.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49e408640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc49815e700 0x7fc49806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:38.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49e408640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc49815e700 0x7fc49806a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59864/0 (socket says 172.21.15.46:59864) 2023-10-30T14:09:38.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49e408640 1 -- 172.21.15.46:0/238348507 learned_addr learned my addr 172.21.15.46:0/238348507 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:38.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49d406640 1 --2- 172.21.15.46:0/238348507 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc498070740 0x7fc4980c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:38.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49e408640 1 -- 172.21.15.46:0/238348507 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc498070740 msgr2=0x7fc4980c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:38.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49e408640 1 --2- 172.21.15.46:0/238348507 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc498070740 0x7fc4980c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:38.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49e408640 1 -- 172.21.15.46:0/238348507 >> v1:172.21.15.154:6789/0 conn(0x7fc4981580e0 legacy=0x7fc498069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:38.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.757+0000 7fc49e408640 1 -- 172.21.15.46:0/238348507 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4980c6bb0 con 0x7fc49815e700 2023-10-30T14:09:38.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.761+0000 7fc49e408640 1 --2- 172.21.15.46:0/238348507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49815e700 0x7fc49806a5a0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fc490009e50 tx=0x7fc49001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109ff586a91f2f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:38.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.761+0000 7fc49cc05640 1 -- 172.21.15.46:0/238348507 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4900a58a0 con 0x7fc49815e700 2023-10-30T14:09:38.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.761+0000 7fc49cc05640 1 -- 172.21.15.46:0/238348507 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4900a5a00 con 0x7fc49815e700 2023-10-30T14:09:38.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.761+0000 7fc49cc05640 1 -- 172.21.15.46:0/238348507 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4900a5d90 con 0x7fc49815e700 2023-10-30T14:09:38.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.761+0000 7fc49fe6b640 1 -- 172.21.15.46:0/238348507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49815e700 msgr2=0x7fc49806a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:38.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.761+0000 7fc49fe6b640 1 --2- 172.21.15.46:0/238348507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49815e700 0x7fc49806a5a0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fc490009e50 tx=0x7fc49001c9e0 comp rx=0 tx=0).stop 2023-10-30T14:09:38.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.761+0000 7fc49fe6b640 1 -- 172.21.15.46:0/238348507 shutdown_connections 2023-10-30T14:09:38.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.761+0000 7fc49fe6b640 1 --2- 172.21.15.46:0/238348507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49815e700 0x7fc49806a5a0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:38.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.761+0000 7fc49fe6b640 1 --2- 172.21.15.46:0/238348507 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc498070740 0x7fc4980c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:38.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.761+0000 7fc49fe6b640 1 -- 172.21.15.46:0/238348507 >> 172.21.15.46:0/238348507 conn(0x7fc4980c7910 msgr2=0x7fc4980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:38.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.761+0000 7fc49fe6b640 1 -- 172.21.15.46:0/238348507 shutdown_connections 2023-10-30T14:09:38.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.761+0000 7fc49fe6b640 1 -- 172.21.15.46:0/238348507 wait complete. 2023-10-30T14:09:38.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49fe6b640 1 Processor -- start 2023-10-30T14:09:38.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49fe6b640 1 -- start start 2023-10-30T14:09:38.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49fe6b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc498070740 0x7fc49814e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49fe6b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4981580e0 0x7fc498160570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49fe6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49815e700 0x7fc498148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49fe6b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc498162050 con 0x7fc49815e700 2023-10-30T14:09:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49fe6b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc498153300 con 0x7fc4981580e0 2023-10-30T14:09:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49fe6b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49815fdb0 con 0x7fc498070740 2023-10-30T14:09:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49e408640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49815e700 0x7fc498148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49e408640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49815e700 0x7fc498148960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59870/0 (socket says 172.21.15.46:59870) 2023-10-30T14:09:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49e408640 1 -- 172.21.15.46:0/3588090448 learned_addr learned my addr 172.21.15.46:0/3588090448 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49dc07640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc498070740 0x7fc49814e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49d406640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4981580e0 0x7fc498160570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49e408640 1 -- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc498070740 msgr2=0x7fc49814e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49e408640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc498070740 0x7fc49814e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49e408640 1 -- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4981580e0 msgr2=0x7fc498160570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49e408640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4981580e0 0x7fc498160570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49e408640 1 -- 172.21.15.46:0/3588090448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc490009b30 con 0x7fc49815e700 2023-10-30T14:09:38.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49d406640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4981580e0 0x7fc498160570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:38.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49e408640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49815e700 0x7fc498148960 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fc4900af040 tx=0x7fc490019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:38.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc48e7fc640 1 -- 172.21.15.46:0/3588090448 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4900b8670 con 0x7fc49815e700 2023-10-30T14:09:38.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc48e7fc640 1 -- 172.21.15.46:0/3588090448 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4900b87d0 con 0x7fc49815e700 2023-10-30T14:09:38.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49fe6b640 1 -- 172.21.15.46:0/3588090448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc498152c40 con 0x7fc49815e700 2023-10-30T14:09:38.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc48e7fc640 1 -- 172.21.15.46:0/3588090448 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4900b8b80 con 0x7fc49815e700 2023-10-30T14:09:38.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.765+0000 7fc49fe6b640 1 -- 172.21.15.46:0/3588090448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4981fa100 con 0x7fc49815e700 2023-10-30T14:09:38.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.769+0000 7fc49fe6b640 1 -- 172.21.15.46:0/3588090448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc49804e170 con 0x7fc49815e700 2023-10-30T14:09:38.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.769+0000 7fc48e7fc640 1 -- 172.21.15.46:0/3588090448 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc4900b8ce0 con 0x7fc49815e700 2023-10-30T14:09:38.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.769+0000 7fc48e7fc640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc47c0664a0 0x7fc47c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:38.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.769+0000 7fc48e7fc640 1 -- 172.21.15.46:0/3588090448 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7fc49011fd90 con 0x7fc49815e700 2023-10-30T14:09:38.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.769+0000 7fc49dc07640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc47c0664a0 0x7fc47c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:38.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.769+0000 7fc49dc07640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc47c0664a0 0x7fc47c068960 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc494007fd0 tx=0x7fc494009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:38.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.773+0000 7fc48e7fc640 1 -- 172.21.15.46:0/3588090448 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc4900ec9f0 con 0x7fc49815e700 2023-10-30T14:09:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.965+0000 7fc49fe6b640 1 -- 172.21.15.46:0/3588090448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc498073e10 con 0x7fc49815e700 2023-10-30T14:09:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.965+0000 7fc48e7fc640 1 -- 172.21.15.46:0/3588090448 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+6520 (secure 0 0 0) 0x7fc4900f0010 con 0x7fc49815e700 2023-10-30T14:09:38.970 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:38.971 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:26.948035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:38.971 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 202 2023-10-30T14:09:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.969+0000 7fc49fe6b640 1 -- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc47c0664a0 msgr2=0x7fc47c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.969+0000 7fc49fe6b640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc47c0664a0 0x7fc47c068960 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc494007fd0 tx=0x7fc494009040 comp rx=0 tx=0).stop 2023-10-30T14:09:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.969+0000 7fc49fe6b640 1 -- 172.21.15.46:0/3588090448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49815e700 msgr2=0x7fc498148960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.969+0000 7fc49fe6b640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49815e700 0x7fc498148960 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fc4900af040 tx=0x7fc490019780 comp rx=0 tx=0).stop 2023-10-30T14:09:38.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.969+0000 7fc49fe6b640 1 -- 172.21.15.46:0/3588090448 shutdown_connections 2023-10-30T14:09:38.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.969+0000 7fc49fe6b640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc47c0664a0 0x7fc47c068960 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:38.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.969+0000 7fc49fe6b640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc49815e700 0x7fc498148960 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:38.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.969+0000 7fc49fe6b640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4981580e0 0x7fc498160570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:38.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.969+0000 7fc49fe6b640 1 --2- 172.21.15.46:0/3588090448 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc498070740 0x7fc49814e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:38.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.969+0000 7fc49fe6b640 1 -- 172.21.15.46:0/3588090448 >> 172.21.15.46:0/3588090448 conn(0x7fc4980c7910 msgr2=0x7fc498156e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:38.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.969+0000 7fc49fe6b640 1 -- 172.21.15.46:0/3588090448 shutdown_connections 2023-10-30T14:09:38.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:38.969+0000 7fc49fe6b640 1 -- 172.21.15.46:0/3588090448 wait complete. 2023-10-30T14:09:38.992 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:09:40.993 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:41.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.145+0000 7f648cd1d640 1 Processor -- start 2023-10-30T14:09:41.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.145+0000 7f648cd1d640 1 -- start start 2023-10-30T14:09:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.145+0000 7f648cd1d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f648815d6f0 0x7f648815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.145+0000 7f648cd1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64880cb800 con 0x7f64880cf160 2023-10-30T14:09:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.145+0000 7f648cd1d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64880cb330 con 0x7f64881538a0 2023-10-30T14:09:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.145+0000 7f648cd1d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64880cad70 con 0x7f648815d6f0 2023-10-30T14:09:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.145+0000 7f647ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f648815d6f0 0x7f648815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.145+0000 7f647ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f648815d6f0 0x7f648815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59882/0 (socket says 172.21.15.46:59882) 2023-10-30T14:09:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.145+0000 7f647ffff640 1 -- 172.21.15.46:0/50164018 learned_addr learned my addr 172.21.15.46:0/50164018 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.145+0000 7f647ffff640 1 -- 172.21.15.46:0/50164018 >> v1:172.21.15.154:6789/0 conn(0x7f64881538a0 legacy=0x7f64880ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.145+0000 7f647ffff640 1 -- 172.21.15.46:0/50164018 >> v1:172.21.15.46:6789/0 conn(0x7f64880cf160 legacy=0x7f6488164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.145+0000 7f647ffff640 1 -- 172.21.15.46:0/50164018 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64880cb050 con 0x7f648815d6f0 2023-10-30T14:09:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.149+0000 7f647ffff640 1 --2- 172.21.15.46:0/50164018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f648815d6f0 0x7f648815a090 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f6470009870 tx=0x7f647001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65746eb9e886eec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:41.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.149+0000 7f6485d9b640 1 -- 172.21.15.46:0/50164018 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700a8a30 con 0x7f648815d6f0 2023-10-30T14:09:41.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.149+0000 7f6485d9b640 1 -- 172.21.15.46:0/50164018 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64700a8b90 con 0x7f648815d6f0 2023-10-30T14:09:41.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.149+0000 7f6485d9b640 1 -- 172.21.15.46:0/50164018 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700b0440 con 0x7f648815d6f0 2023-10-30T14:09:41.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.149+0000 7f648cd1d640 1 -- 172.21.15.46:0/50164018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f648815d6f0 msgr2=0x7f648815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:41.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.149+0000 7f648cd1d640 1 --2- 172.21.15.46:0/50164018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f648815d6f0 0x7f648815a090 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f6470009870 tx=0x7f647001ed30 comp rx=0 tx=0).stop 2023-10-30T14:09:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.149+0000 7f648cd1d640 1 -- 172.21.15.46:0/50164018 shutdown_connections 2023-10-30T14:09:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.149+0000 7f648cd1d640 1 --2- 172.21.15.46:0/50164018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f648815d6f0 0x7f648815a090 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.149+0000 7f648cd1d640 1 -- 172.21.15.46:0/50164018 >> 172.21.15.46:0/50164018 conn(0x7f648806bfe0 msgr2=0x7f648806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.149+0000 7f648cd1d640 1 -- 172.21.15.46:0/50164018 shutdown_connections 2023-10-30T14:09:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.149+0000 7f648cd1d640 1 -- 172.21.15.46:0/50164018 wait complete. 2023-10-30T14:09:41.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f648cd1d640 1 Processor -- start 2023-10-30T14:09:41.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f648cd1d640 1 -- start start 2023-10-30T14:09:41.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f648cd1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64880cf160 0x7f64881e3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:41.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f648cd1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64881538a0 0x7f64881592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:41.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f648cd1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f648815d6f0 0x7f64880c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:41.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f648cd1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64880c8ce0 con 0x7f64881538a0 2023-10-30T14:09:41.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f648cd1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64880c8310 con 0x7f648815d6f0 2023-10-30T14:09:41.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f648cd1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64880c7060 con 0x7f64880cf160 2023-10-30T14:09:41.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f647ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64881538a0 0x7f64881592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:41.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f647ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64881538a0 0x7f64881592a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59886/0 (socket says 172.21.15.46:59886) 2023-10-30T14:09:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f647ffff640 1 -- 172.21.15.46:0/3205010253 learned_addr learned my addr 172.21.15.46:0/3205010253 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f648659c640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64880cf160 0x7f64881e3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f6486d9d640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f648815d6f0 0x7f64880c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f647ffff640 1 -- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64880cf160 msgr2=0x7f64881e3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f647ffff640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64880cf160 0x7f64881e3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f647ffff640 1 -- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f648815d6f0 msgr2=0x7f64880c87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f647ffff640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f648815d6f0 0x7f64880c87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f647ffff640 1 -- 172.21.15.46:0/3205010253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64700094d0 con 0x7f64881538a0 2023-10-30T14:09:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f647ffff640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64881538a0 0x7f64881592a0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f64700b3040 tx=0x7f647001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f647effd640 1 -- 172.21.15.46:0/3205010253 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700be070 con 0x7f64881538a0 2023-10-30T14:09:41.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f647effd640 1 -- 172.21.15.46:0/3205010253 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64700ba400 con 0x7f64881538a0 2023-10-30T14:09:41.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f647effd640 1 -- 172.21.15.46:0/3205010253 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700ba7b0 con 0x7f64881538a0 2023-10-30T14:09:41.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f648cd1d640 1 -- 172.21.15.46:0/3205010253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64880c7d00 con 0x7f64881538a0 2023-10-30T14:09:41.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.153+0000 7f648cd1d640 1 -- 172.21.15.46:0/3205010253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64880c6aa0 con 0x7f64881538a0 2023-10-30T14:09:41.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.157+0000 7f648cd1d640 1 -- 172.21.15.46:0/3205010253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648804e170 con 0x7f64881538a0 2023-10-30T14:09:41.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.157+0000 7f647effd640 1 -- 172.21.15.46:0/3205010253 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64700ba910 con 0x7f64881538a0 2023-10-30T14:09:41.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.157+0000 7f647effd640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64680664a0 0x7f6468068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:41.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.157+0000 7f647effd640 1 -- 172.21.15.46:0/3205010253 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7f6470122480 con 0x7f64881538a0 2023-10-30T14:09:41.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.157+0000 7f648659c640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64680664a0 0x7f6468068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:41.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.157+0000 7f648659c640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64680664a0 0x7f6468068960 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6478007fd0 tx=0x7f6478009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:41.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.161+0000 7f647effd640 1 -- 172.21.15.46:0/3205010253 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64700ef100 con 0x7f64881538a0 2023-10-30T14:09:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 -- 172.21.15.46:0/3205010253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64880c8450 con 0x7f64881538a0 2023-10-30T14:09:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f647effd640 1 -- 172.21.15.46:0/3205010253 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f64700f2720 con 0x7f64881538a0 2023-10-30T14:09:41.357 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:41.357 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:26.948035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:41.357 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 202 2023-10-30T14:09:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 -- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64680664a0 msgr2=0x7f6468068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64680664a0 0x7f6468068960 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6478007fd0 tx=0x7f6478009040 comp rx=0 tx=0).stop 2023-10-30T14:09:41.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 -- 172.21.15.46:0/3205010253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64881538a0 msgr2=0x7f64881592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:41.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64881538a0 0x7f64881592a0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f64700b3040 tx=0x7f647001bad0 comp rx=0 tx=0).stop 2023-10-30T14:09:41.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 -- 172.21.15.46:0/3205010253 shutdown_connections 2023-10-30T14:09:41.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64680664a0 0x7f6468068960 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:41.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f648815d6f0 0x7f64880c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:41.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64881538a0 0x7f64881592a0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:41.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 --2- 172.21.15.46:0/3205010253 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64880cf160 0x7f64881e3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:41.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 -- 172.21.15.46:0/3205010253 >> 172.21.15.46:0/3205010253 conn(0x7f648806bfe0 msgr2=0x7f64880ce1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:41.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 -- 172.21.15.46:0/3205010253 shutdown_connections 2023-10-30T14:09:41.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:41.353+0000 7f648cd1d640 1 -- 172.21.15.46:0/3205010253 wait complete. 2023-10-30T14:09:41.378 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:09:43.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:43.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.529+0000 7fa06b701640 1 Processor -- start 2023-10-30T14:09:43.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.529+0000 7fa06b701640 1 -- start start 2023-10-30T14:09:43.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa06b701640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa0641580e0 0x7fa064070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:43.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa06b701640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0640c6540 0x7fa06415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:43.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa06b701640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa064069e90 0x7fa06406a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:43.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa06b701640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0640c5810 con 0x7fa064069e90 2023-10-30T14:09:43.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa06b701640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0640c6960 con 0x7fa0641580e0 2023-10-30T14:09:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa06b701640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640c6bb0 con 0x7fa0640c6540 2023-10-30T14:09:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa069c9e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa064069e90 0x7fa06406a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa069c9e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa064069e90 0x7fa06406a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59894/0 (socket says 172.21.15.46:59894) 2023-10-30T14:09:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa069c9e640 1 -- 172.21.15.46:0/1758038769 learned_addr learned my addr 172.21.15.46:0/1758038769 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa068c9c640 1 --2- 172.21.15.46:0/1758038769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0640c6540 0x7fa06415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa06949d640 1 --2- 172.21.15.46:0/1758038769 >> v2:172.21.15.154:3300/0 conn(0x7fa0641580e0 0x7fa064070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa069c9e640 1 -- 172.21.15.46:0/1758038769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0640c6540 msgr2=0x7fa06415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa069c9e640 1 --2- 172.21.15.46:0/1758038769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0640c6540 0x7fa06415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa069c9e640 1 -- 172.21.15.46:0/1758038769 >> v2:172.21.15.154:3300/0 conn(0x7fa0641580e0 msgr2=0x7fa064070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa069c9e640 1 --2- 172.21.15.46:0/1758038769 >> v2:172.21.15.154:3300/0 conn(0x7fa0641580e0 0x7fa064070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:43.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa069c9e640 1 -- 172.21.15.46:0/1758038769 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0640c6e70 con 0x7fa064069e90 2023-10-30T14:09:43.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.533+0000 7fa068c9c640 1 --2- 172.21.15.46:0/1758038769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0640c6540 0x7fa06415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa069c9e640 1 --2- 172.21.15.46:0/1758038769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064069e90 0x7fa06406a320 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fa06406eea0 tx=0x7fa05c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d406d382c8dfca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa05bfff640 1 -- 172.21.15.46:0/1758038769 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c0a5760 con 0x7fa064069e90 2023-10-30T14:09:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa05bfff640 1 -- 172.21.15.46:0/1758038769 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05c0a58c0 con 0x7fa064069e90 2023-10-30T14:09:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa05bfff640 1 -- 172.21.15.46:0/1758038769 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c0a5c50 con 0x7fa064069e90 2023-10-30T14:09:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa06b701640 1 -- 172.21.15.46:0/1758038769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064069e90 msgr2=0x7fa06406a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa06b701640 1 --2- 172.21.15.46:0/1758038769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064069e90 0x7fa06406a320 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fa06406eea0 tx=0x7fa05c01c9d0 comp rx=0 tx=0).stop 2023-10-30T14:09:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa06b701640 1 -- 172.21.15.46:0/1758038769 shutdown_connections 2023-10-30T14:09:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa06b701640 1 --2- 172.21.15.46:0/1758038769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064069e90 0x7fa06406a320 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa06b701640 1 --2- 172.21.15.46:0/1758038769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0640c6540 0x7fa06415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa06b701640 1 --2- 172.21.15.46:0/1758038769 >> v2:172.21.15.154:3300/0 conn(0x7fa0641580e0 0x7fa064070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa06b701640 1 -- 172.21.15.46:0/1758038769 >> 172.21.15.46:0/1758038769 conn(0x7fa0640c7910 msgr2=0x7fa0640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa06b701640 1 -- 172.21.15.46:0/1758038769 shutdown_connections 2023-10-30T14:09:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa06b701640 1 -- 172.21.15.46:0/1758038769 wait complete. 2023-10-30T14:09:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa06b701640 1 Processor -- start 2023-10-30T14:09:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.537+0000 7fa06b701640 1 -- start start 2023-10-30T14:09:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa06b701640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa064069e90 0x7fa0641f6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa06b701640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0640c6540 0x7fa0640c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa06b701640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0641580e0 0x7fa064160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa06b701640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640c3ec0 con 0x7fa0641580e0 2023-10-30T14:09:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa06b701640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0641f74c0 con 0x7fa0640c6540 2023-10-30T14:09:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa06b701640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640c9c60 con 0x7fa064069e90 2023-10-30T14:09:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa069c9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0641580e0 0x7fa064160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa06949d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa064069e90 0x7fa0641f6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa069c9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0641580e0 0x7fa064160930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59904/0 (socket says 172.21.15.46:59904) 2023-10-30T14:09:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa069c9e640 1 -- 172.21.15.46:0/2560589373 learned_addr learned my addr 172.21.15.46:0/2560589373 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa06949d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa064069e90 0x7fa0641f6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:59592/0 (socket says 172.21.15.46:59592) 2023-10-30T14:09:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa068c9c640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0640c6540 0x7fa0640c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa069c9e640 1 -- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa064069e90 msgr2=0x7fa0641f6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa069c9e640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa064069e90 0x7fa0641f6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa069c9e640 1 -- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0640c6540 msgr2=0x7fa0640c42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa069c9e640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0640c6540 0x7fa0640c42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa069c9e640 1 -- 172.21.15.46:0/2560589373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c009b30 con 0x7fa0641580e0 2023-10-30T14:09:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa069c9e640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0641580e0 0x7fa064160930 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fa05c01b080 tx=0x7fa05c019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa059ffb640 1 -- 172.21.15.46:0/2560589373 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c0bd070 con 0x7fa0641580e0 2023-10-30T14:09:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa059ffb640 1 -- 172.21.15.46:0/2560589373 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05c0b8470 con 0x7fa0641580e0 2023-10-30T14:09:43.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa059ffb640 1 -- 172.21.15.46:0/2560589373 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c0b8820 con 0x7fa0641580e0 2023-10-30T14:09:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa06b701640 1 -- 172.21.15.46:0/2560589373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0640c9f00 con 0x7fa0641580e0 2023-10-30T14:09:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa06b701640 1 -- 172.21.15.46:0/2560589373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0640ca4a0 con 0x7fa0641580e0 2023-10-30T14:09:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.541+0000 7fa06b701640 1 -- 172.21.15.46:0/2560589373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0640c4000 con 0x7fa0641580e0 2023-10-30T14:09:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.545+0000 7fa059ffb640 1 -- 172.21.15.46:0/2560589373 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa05c0b8980 con 0x7fa0641580e0 2023-10-30T14:09:43.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.545+0000 7fa059ffb640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0480664a0 0x7fa048068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:43.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.545+0000 7fa059ffb640 1 -- 172.21.15.46:0/2560589373 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7fa05c11f7c0 con 0x7fa0641580e0 2023-10-30T14:09:43.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.545+0000 7fa06949d640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0480664a0 0x7fa048068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:43.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.545+0000 7fa06949d640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0480664a0 0x7fa048068960 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa0600079c0 tx=0x7fa060008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:43.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.549+0000 7fa059ffb640 1 -- 172.21.15.46:0/2560589373 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa0640c4000 con 0x7fa0641580e0 2023-10-30T14:09:43.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.737+0000 7fa06b701640 1 -- 172.21.15.46:0/2560589373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0640c4000 con 0x7fa0641580e0 2023-10-30T14:09:43.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.737+0000 7fa059ffb640 1 -- 172.21.15.46:0/2560589373 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+6520 (secure 0 0 0) 0x7fa0640c4000 con 0x7fa0641580e0 2023-10-30T14:09:43.742 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:43.742 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:26.948035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:43.743 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 202 2023-10-30T14:09:43.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.741+0000 7fa06b701640 1 -- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0480664a0 msgr2=0x7fa048068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:43.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.741+0000 7fa06b701640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0480664a0 0x7fa048068960 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa0600079c0 tx=0x7fa060008040 comp rx=0 tx=0).stop 2023-10-30T14:09:43.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.741+0000 7fa06b701640 1 -- 172.21.15.46:0/2560589373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0641580e0 msgr2=0x7fa064160930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:43.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.741+0000 7fa06b701640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0641580e0 0x7fa064160930 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fa05c01b080 tx=0x7fa05c019770 comp rx=0 tx=0).stop 2023-10-30T14:09:43.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.741+0000 7fa06b701640 1 -- 172.21.15.46:0/2560589373 shutdown_connections 2023-10-30T14:09:43.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.741+0000 7fa06b701640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa0480664a0 0x7fa048068960 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:43.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.741+0000 7fa06b701640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0641580e0 0x7fa064160930 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:43.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.741+0000 7fa06b701640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0640c6540 0x7fa0640c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:43.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.741+0000 7fa06b701640 1 --2- 172.21.15.46:0/2560589373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa064069e90 0x7fa0641f6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:43.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.741+0000 7fa06b701640 1 -- 172.21.15.46:0/2560589373 >> 172.21.15.46:0/2560589373 conn(0x7fa0640c7910 msgr2=0x7fa0640c0720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:43.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.741+0000 7fa06b701640 1 -- 172.21.15.46:0/2560589373 shutdown_connections 2023-10-30T14:09:43.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:43.741+0000 7fa06b701640 1 -- 172.21.15.46:0/2560589373 wait complete. 2023-10-30T14:09:43.763 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:09:45.764 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f57a53c4640 1 Processor -- start 2023-10-30T14:09:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f57a53c4640 1 -- start start 2023-10-30T14:09:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f57a53c4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f57a01538a0 0x7f57a015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f57a53c4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f57a00cf160 0x7f57a00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f57a53c4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a00cb800 con 0x7f57a00ca0e0 2023-10-30T14:09:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f57a53c4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a015dc30 con 0x7f57a00cf160 2023-10-30T14:09:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f57a53c4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a00cb330 con 0x7f57a01538a0 2023-10-30T14:09:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f579f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f57a00cf160 0x7f57a00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f579effd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f57a01538a0 0x7f57a015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:45.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f579f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f57a00cf160 0x7f57a00cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59906/0 (socket says 172.21.15.46:59906) 2023-10-30T14:09:45.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f579f7fe640 1 -- 172.21.15.46:0/506966873 learned_addr learned my addr 172.21.15.46:0/506966873 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f579f7fe640 1 -- 172.21.15.46:0/506966873 >> v2:172.21.15.154:3300/0 conn(0x7f57a01538a0 msgr2=0x7f57a015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f579f7fe640 1 --2- 172.21.15.46:0/506966873 >> v2:172.21.15.154:3300/0 conn(0x7f57a01538a0 0x7f57a015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f579f7fe640 1 -- 172.21.15.46:0/506966873 >> v1:172.21.15.46:6789/0 conn(0x7f57a00ca0e0 legacy=0x7f57a015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.917+0000 7f579f7fe640 1 -- 172.21.15.46:0/506966873 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a00cad70 con 0x7f57a00cf160 2023-10-30T14:09:45.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.921+0000 7f579f7fe640 1 --2- 172.21.15.46:0/506966873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57a00cf160 0x7f57a00cf590 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f57a00cfd10 tx=0x7f579801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95653033f7194d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:45.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.921+0000 7f579dffb640 1 -- 172.21.15.46:0/506966873 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57980af400 con 0x7f57a00cf160 2023-10-30T14:09:45.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.921+0000 7f579dffb640 1 -- 172.21.15.46:0/506966873 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57980ab050 con 0x7f57a00cf160 2023-10-30T14:09:45.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.921+0000 7f579dffb640 1 -- 172.21.15.46:0/506966873 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57980af780 con 0x7f57a00cf160 2023-10-30T14:09:45.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.921+0000 7f57a53c4640 1 -- 172.21.15.46:0/506966873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57a00cf160 msgr2=0x7f57a00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.921+0000 7f57a53c4640 1 --2- 172.21.15.46:0/506966873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57a00cf160 0x7f57a00cf590 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f57a00cfd10 tx=0x7f579801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:09:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.921+0000 7f57a53c4640 1 -- 172.21.15.46:0/506966873 shutdown_connections 2023-10-30T14:09:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.921+0000 7f57a53c4640 1 --2- 172.21.15.46:0/506966873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57a00cf160 0x7f57a00cf590 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.921+0000 7f57a53c4640 1 --2- 172.21.15.46:0/506966873 >> v2:172.21.15.154:3300/0 conn(0x7f57a01538a0 0x7f57a015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.921+0000 7f57a53c4640 1 -- 172.21.15.46:0/506966873 >> 172.21.15.46:0/506966873 conn(0x7f57a006bfe0 msgr2=0x7f57a006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:45.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.921+0000 7f57a53c4640 1 -- 172.21.15.46:0/506966873 shutdown_connections 2023-10-30T14:09:45.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.921+0000 7f57a53c4640 1 -- 172.21.15.46:0/506966873 wait complete. 2023-10-30T14:09:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57a53c4640 1 Processor -- start 2023-10-30T14:09:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57a53c4640 1 -- start start 2023-10-30T14:09:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57a53c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57a00ca0e0 0x7f57a01480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57a53c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57a00cf160 0x7f57a0158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:45.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57a53c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a01538a0 0x7f57a01f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:45.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57a53c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a01e5330 con 0x7f57a00cf160 2023-10-30T14:09:45.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57a53c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a00b4f30 con 0x7f57a00ca0e0 2023-10-30T14:09:45.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57a53c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a01e4d70 con 0x7f57a01538a0 2023-10-30T14:09:45.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f579e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57a00cf160 0x7f57a0158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f579e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57a00cf160 0x7f57a0158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59912/0 (socket says 172.21.15.46:59912) 2023-10-30T14:09:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f579e7fc640 1 -- 172.21.15.46:0/1845376227 learned_addr learned my addr 172.21.15.46:0/1845376227 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f579f7fe640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a01538a0 0x7f57a01f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f579effd640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57a00ca0e0 0x7f57a01480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f579e7fc640 1 -- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a01538a0 msgr2=0x7f57a01f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:45.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f579e7fc640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a01538a0 0x7f57a01f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:45.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f579e7fc640 1 -- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57a00ca0e0 msgr2=0x7f57a01480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:45.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f579e7fc640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57a00ca0e0 0x7f57a01480f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:45.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f579e7fc640 1 -- 172.21.15.46:0/1845376227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5798009b30 con 0x7f57a00cf160 2023-10-30T14:09:45.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f579effd640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57a00ca0e0 0x7f57a01480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:45.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f579e7fc640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57a00cf160 0x7f57a0158f80 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f579000c3c0 tx=0x7f579000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:45.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57977fe640 1 -- 172.21.15.46:0/1845376227 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57900052d0 con 0x7f57a00cf160 2023-10-30T14:09:45.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57977fe640 1 -- 172.21.15.46:0/1845376227 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5790005430 con 0x7f57a00cf160 2023-10-30T14:09:45.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57977fe640 1 -- 172.21.15.46:0/1845376227 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5790003840 con 0x7f57a00cf160 2023-10-30T14:09:45.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57a53c4640 1 -- 172.21.15.46:0/1845376227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a01e5050 con 0x7f57a00cf160 2023-10-30T14:09:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.925+0000 7f57a53c4640 1 -- 172.21.15.46:0/1845376227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a00b4bc0 con 0x7f57a00cf160 2023-10-30T14:09:45.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.929+0000 7f57a53c4640 1 -- 172.21.15.46:0/1845376227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a00b4d80 con 0x7f57a00cf160 2023-10-30T14:09:45.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.929+0000 7f57977fe640 1 -- 172.21.15.46:0/1845376227 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f57900039a0 con 0x7f57a00cf160 2023-10-30T14:09:45.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.929+0000 7f57977fe640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f57800664a0 0x7f5780068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:45.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.929+0000 7f57977fe640 1 -- 172.21.15.46:0/1845376227 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7f5790083c90 con 0x7f57a00cf160 2023-10-30T14:09:45.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.929+0000 7f579effd640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f57800664a0 0x7f5780068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:45.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.933+0000 7f579effd640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f57800664a0 0x7f5780068960 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f5788009bb0 tx=0x7f5788008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:45.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:45.933+0000 7f57977fe640 1 -- 172.21.15.46:0/1845376227 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f57900507f0 con 0x7f57a00cf160 2023-10-30T14:09:46.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.125+0000 7f57a53c4640 1 -- 172.21.15.46:0/1845376227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57a00b2a50 con 0x7f57a00cf160 2023-10-30T14:09:46.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.125+0000 7f57977fe640 1 -- 172.21.15.46:0/1845376227 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f5790017070 con 0x7f57a00cf160 2023-10-30T14:09:46.130 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:46.130 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:26.948035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:46.131 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 202 2023-10-30T14:09:46.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.129+0000 7f57a53c4640 1 -- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f57800664a0 msgr2=0x7f5780068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:46.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.129+0000 7f57a53c4640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f57800664a0 0x7f5780068960 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f5788009bb0 tx=0x7f5788008040 comp rx=0 tx=0).stop 2023-10-30T14:09:46.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.129+0000 7f57a53c4640 1 -- 172.21.15.46:0/1845376227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57a00cf160 msgr2=0x7f57a0158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.129+0000 7f57a53c4640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57a00cf160 0x7f57a0158f80 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f579000c3c0 tx=0x7f579000c880 comp rx=0 tx=0).stop 2023-10-30T14:09:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.129+0000 7f57a53c4640 1 -- 172.21.15.46:0/1845376227 shutdown_connections 2023-10-30T14:09:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.129+0000 7f57a53c4640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f57800664a0 0x7f5780068960 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.129+0000 7f57a53c4640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a01538a0 0x7f57a01f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:46.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.129+0000 7f57a53c4640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57a00cf160 0x7f57a0158f80 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:46.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.129+0000 7f57a53c4640 1 --2- 172.21.15.46:0/1845376227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57a00ca0e0 0x7f57a01480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:46.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.129+0000 7f57a53c4640 1 -- 172.21.15.46:0/1845376227 >> 172.21.15.46:0/1845376227 conn(0x7f57a006bfe0 msgr2=0x7f57a006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:46.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.129+0000 7f57a53c4640 1 -- 172.21.15.46:0/1845376227 shutdown_connections 2023-10-30T14:09:46.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:46.129+0000 7f57a53c4640 1 -- 172.21.15.46:0/1845376227 wait complete. 2023-10-30T14:09:46.152 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:09:48.152 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3c5c5640 1 Processor -- start 2023-10-30T14:09:48.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3c5c5640 1 -- start start 2023-10-30T14:09:48.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3c5c5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e341538a0 0x7f8e3415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:48.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3c5c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e340cf160 0x7f8e340cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:48.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3c5c5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e340cb800 con 0x7f8e340ca0e0 2023-10-30T14:09:48.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3c5c5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e3415dc30 con 0x7f8e341538a0 2023-10-30T14:09:48.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3c5c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e340cb330 con 0x7f8e340cf160 2023-10-30T14:09:48.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3a361640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e341538a0 0x7f8e3415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:48.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3a361640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e341538a0 0x7f8e3415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48156/0 (socket says 172.21.15.46:48156) 2023-10-30T14:09:48.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3a361640 1 -- 172.21.15.46:0/3025779258 learned_addr learned my addr 172.21.15.46:0/3025779258 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:48.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3ab62640 1 --2- 172.21.15.46:0/3025779258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e340cf160 0x7f8e340cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:48.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3a361640 1 -- 172.21.15.46:0/3025779258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e340cf160 msgr2=0x7f8e340cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:48.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3a361640 1 --2- 172.21.15.46:0/3025779258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e340cf160 0x7f8e340cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:48.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3a361640 1 -- 172.21.15.46:0/3025779258 >> v1:172.21.15.154:6789/0 conn(0x7f8e340ca0e0 legacy=0x7f8e3415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:48.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.305+0000 7f8e3a361640 1 -- 172.21.15.46:0/3025779258 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e340cad70 con 0x7f8e341538a0 2023-10-30T14:09:48.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.309+0000 7f8e3a361640 1 --2- 172.21.15.46:0/3025779258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e341538a0 0x7f8e3415d6f0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f8e30009870 tx=0x7f8e3001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad28cccbeea59e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:48.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.309+0000 7f8e3935f640 1 -- 172.21.15.46:0/3025779258 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e300a8a30 con 0x7f8e341538a0 2023-10-30T14:09:48.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.309+0000 7f8e3935f640 1 -- 172.21.15.46:0/3025779258 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e300a8b90 con 0x7f8e341538a0 2023-10-30T14:09:48.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.309+0000 7f8e3935f640 1 -- 172.21.15.46:0/3025779258 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e300b0440 con 0x7f8e341538a0 2023-10-30T14:09:48.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.309+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3025779258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e341538a0 msgr2=0x7f8e3415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:48.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.309+0000 7f8e3c5c5640 1 --2- 172.21.15.46:0/3025779258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e341538a0 0x7f8e3415d6f0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f8e30009870 tx=0x7f8e3001ed30 comp rx=0 tx=0).stop 2023-10-30T14:09:48.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.309+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3025779258 shutdown_connections 2023-10-30T14:09:48.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.309+0000 7f8e3c5c5640 1 --2- 172.21.15.46:0/3025779258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e340cf160 0x7f8e340cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:48.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.309+0000 7f8e3c5c5640 1 --2- 172.21.15.46:0/3025779258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e341538a0 0x7f8e3415d6f0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:48.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.309+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3025779258 >> 172.21.15.46:0/3025779258 conn(0x7f8e3406bfe0 msgr2=0x7f8e3406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:48.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.309+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3025779258 shutdown_connections 2023-10-30T14:09:48.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.309+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3025779258 wait complete. 2023-10-30T14:09:48.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3c5c5640 1 Processor -- start 2023-10-30T14:09:48.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3c5c5640 1 -- start start 2023-10-30T14:09:48.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3c5c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e340ca0e0 0x7f8e341e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:48.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3c5c5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e340cf160 0x7f8e3415b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:48.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3c5c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e341538a0 0x7f8e3415bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:48.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3c5c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3414d4f0 con 0x7f8e340ca0e0 2023-10-30T14:09:48.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3c5c5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e341e4d40 con 0x7f8e340cf160 2023-10-30T14:09:48.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3c5c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3415b2a0 con 0x7f8e341538a0 2023-10-30T14:09:48.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3a361640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e340ca0e0 0x7f8e341e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3a361640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e340ca0e0 0x7f8e341e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48158/0 (socket says 172.21.15.46:48158) 2023-10-30T14:09:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3a361640 1 -- 172.21.15.46:0/3861301741 learned_addr learned my addr 172.21.15.46:0/3861301741 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3ab62640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e341538a0 0x7f8e3415bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e39b60640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e340cf160 0x7f8e3415b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3a361640 1 -- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e341538a0 msgr2=0x7f8e3415bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3a361640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e341538a0 0x7f8e3415bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3a361640 1 -- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e340cf160 msgr2=0x7f8e3415b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3a361640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e340cf160 0x7f8e3415b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3a361640 1 -- 172.21.15.46:0/3861301741 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e300094d0 con 0x7f8e340ca0e0 2023-10-30T14:09:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e39b60640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e340cf160 0x7f8e3415b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3a361640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e340ca0e0 0x7f8e341e4090 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f8e300b3040 tx=0x7f8e3001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e2affd640 1 -- 172.21.15.46:0/3861301741 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e300b0dd0 con 0x7f8e340ca0e0 2023-10-30T14:09:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e2affd640 1 -- 172.21.15.46:0/3861301741 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e300bb460 con 0x7f8e340ca0e0 2023-10-30T14:09:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e2affd640 1 -- 172.21.15.46:0/3861301741 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e300bb810 con 0x7f8e340ca0e0 2023-10-30T14:09:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3861301741 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e340cef10 con 0x7f8e340ca0e0 2023-10-30T14:09:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3861301741 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e340c50b0 con 0x7f8e340ca0e0 2023-10-30T14:09:48.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.313+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3861301741 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e340b2a50 con 0x7f8e340ca0e0 2023-10-30T14:09:48.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.317+0000 7f8e2affd640 1 -- 172.21.15.46:0/3861301741 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e300b83f0 con 0x7f8e340ca0e0 2023-10-30T14:09:48.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.317+0000 7f8e2affd640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e180664a0 0x7f8e18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:48.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.317+0000 7f8e2affd640 1 -- 172.21.15.46:0/3861301741 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7918+0+0 (secure 0 0 0) 0x7f8e30121f50 con 0x7f8e340ca0e0 2023-10-30T14:09:48.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.317+0000 7f8e39b60640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e180664a0 0x7f8e18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:48.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.317+0000 7f8e39b60640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e180664a0 0x7f8e18068960 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8e340c4660 tx=0x7f8e24009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:48.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.321+0000 7f8e2affd640 1 -- 172.21.15.46:0/3861301741 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e300eeb40 con 0x7f8e340ca0e0 2023-10-30T14:09:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.509+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3861301741 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e34151940 con 0x7f8e340ca0e0 2023-10-30T14:09:48.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.509+0000 7f8e2affd640 1 -- 172.21.15.46:0/3861301741 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f8e300b5020 con 0x7f8e340ca0e0 2023-10-30T14:09:48.515 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:48.516 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8826,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:26.948035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8505,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8505":{"gid":8505,"name":"a","rank":0,"incarnation":189,"state":"up:active","state_seq":26,"addr":"172.21.15.46:6833/4078242540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4078242540},{"type":"v1","addr":"172.21.15.46:6833","nonce":4078242540}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:48.517 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 202 2023-10-30T14:09:48.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.513+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e180664a0 msgr2=0x7f8e18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:48.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.513+0000 7f8e3c5c5640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e180664a0 0x7f8e18068960 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8e340c4660 tx=0x7f8e24009040 comp rx=0 tx=0).stop 2023-10-30T14:09:48.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.513+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3861301741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e340ca0e0 msgr2=0x7f8e341e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:48.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.513+0000 7f8e3c5c5640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e340ca0e0 0x7f8e341e4090 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f8e300b3040 tx=0x7f8e3001bad0 comp rx=0 tx=0).stop 2023-10-30T14:09:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.513+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3861301741 shutdown_connections 2023-10-30T14:09:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.513+0000 7f8e3c5c5640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e180664a0 0x7f8e18068960 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.513+0000 7f8e3c5c5640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e341538a0 0x7f8e3415bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.513+0000 7f8e3c5c5640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e340cf160 0x7f8e3415b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:48.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.513+0000 7f8e3c5c5640 1 --2- 172.21.15.46:0/3861301741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e340ca0e0 0x7f8e341e4090 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.513+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3861301741 >> 172.21.15.46:0/3861301741 conn(0x7f8e3406bfe0 msgr2=0x7f8e34065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.513+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3861301741 shutdown_connections 2023-10-30T14:09:48.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:48.513+0000 7f8e3c5c5640 1 -- 172.21.15.46:0/3861301741 wait complete. 2023-10-30T14:09:48.536 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:09:50.537 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:50.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.685+0000 7f609aef7640 1 Processor -- start 2023-10-30T14:09:50.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.685+0000 7f609aef7640 1 -- start start 2023-10-30T14:09:50.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.685+0000 7f609aef7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f609406f7a0 0x7f609406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.685+0000 7f609aef7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6094071110 0x7f6094158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.685+0000 7f609aef7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6094066cf0 con 0x7f60940c6bd0 2023-10-30T14:09:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.685+0000 7f609aef7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60940700c0 con 0x7f6094071110 2023-10-30T14:09:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.685+0000 7f609aef7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60940c70d0 con 0x7f609406f7a0 2023-10-30T14:09:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.685+0000 7f6099494640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6094071110 0x7f6094158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:50.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.685+0000 7f6099494640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6094071110 0x7f6094158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48170/0 (socket says 172.21.15.46:48170) 2023-10-30T14:09:50.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.685+0000 7f6099494640 1 -- 172.21.15.46:0/1293898420 learned_addr learned my addr 172.21.15.46:0/1293898420 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:50.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.689+0000 7f6098c93640 1 --2- 172.21.15.46:0/1293898420 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f609406f7a0 0x7f609406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:50.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.689+0000 7f6099494640 1 -- 172.21.15.46:0/1293898420 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f609406f7a0 msgr2=0x7f609406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:50.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.689+0000 7f6099494640 1 --2- 172.21.15.46:0/1293898420 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f609406f7a0 0x7f609406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:50.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.689+0000 7f6099494640 1 -- 172.21.15.46:0/1293898420 >> v1:172.21.15.46:6789/0 conn(0x7f60940c6bd0 legacy=0x7f6094069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:50.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.689+0000 7f6099494640 1 -- 172.21.15.46:0/1293898420 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60940c7390 con 0x7f6094071110 2023-10-30T14:09:50.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.689+0000 7f6099494640 1 --2- 172.21.15.46:0/1293898420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6094071110 0x7f6094158110 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f608c009e50 tx=0x7f608c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd8012a9d0a6e48 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:50.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.689+0000 7f60937fe640 1 -- 172.21.15.46:0/1293898420 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c0af400 con 0x7f6094071110 2023-10-30T14:09:50.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.689+0000 7f60937fe640 1 -- 172.21.15.46:0/1293898420 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608c0ab050 con 0x7f6094071110 2023-10-30T14:09:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.689+0000 7f60937fe640 1 -- 172.21.15.46:0/1293898420 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c0af780 con 0x7f6094071110 2023-10-30T14:09:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 -- 172.21.15.46:0/1293898420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6094071110 msgr2=0x7f6094158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 --2- 172.21.15.46:0/1293898420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6094071110 0x7f6094158110 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f608c009e50 tx=0x7f608c01f1f0 comp rx=0 tx=0).stop 2023-10-30T14:09:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 -- 172.21.15.46:0/1293898420 shutdown_connections 2023-10-30T14:09:50.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 --2- 172.21.15.46:0/1293898420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6094071110 0x7f6094158110 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:50.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 --2- 172.21.15.46:0/1293898420 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f609406f7a0 0x7f609406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:50.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 -- 172.21.15.46:0/1293898420 >> 172.21.15.46:0/1293898420 conn(0x7f60940c7910 msgr2=0x7f60940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:50.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 -- 172.21.15.46:0/1293898420 shutdown_connections 2023-10-30T14:09:50.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 -- 172.21.15.46:0/1293898420 wait complete. 2023-10-30T14:09:50.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 Processor -- start 2023-10-30T14:09:50.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 -- start start 2023-10-30T14:09:50.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f609406f7a0 0x7f60941f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:50.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6094071110 0x7f6094157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:50.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60940c6bd0 0x7f6094157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:50.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60941588a0 con 0x7f609406f7a0 2023-10-30T14:09:50.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609406c680 con 0x7f60940c6bd0 2023-10-30T14:09:50.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f609aef7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609414fd40 con 0x7f6094071110 2023-10-30T14:09:50.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f6098c93640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f609406f7a0 0x7f60941f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:50.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f6098c93640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f609406f7a0 0x7f60941f1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48180/0 (socket says 172.21.15.46:48180) 2023-10-30T14:09:50.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f6098c93640 1 -- 172.21.15.46:0/2143430260 learned_addr learned my addr 172.21.15.46:0/2143430260 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:50.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f6093fff640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6094071110 0x7f6094157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:50.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f6099494640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60940c6bd0 0x7f6094157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:50.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f6098c93640 1 -- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6094071110 msgr2=0x7f6094157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:50.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f6098c93640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6094071110 0x7f6094157700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:50.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f6098c93640 1 -- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60940c6bd0 msgr2=0x7f6094157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:50.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f6098c93640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60940c6bd0 0x7f6094157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:50.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f6098c93640 1 -- 172.21.15.46:0/2143430260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608c009b30 con 0x7f609406f7a0 2023-10-30T14:09:50.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f6099494640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60940c6bd0 0x7f6094157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:09:50.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.693+0000 7f6098c93640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f609406f7a0 0x7f60941f1a20 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f608800d5f0 tx=0x7f608800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:50.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.697+0000 7f60917fa640 1 -- 172.21.15.46:0/2143430260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880125f0 con 0x7f609406f7a0 2023-10-30T14:09:50.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.697+0000 7f60917fa640 1 -- 172.21.15.46:0/2143430260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6088012750 con 0x7f609406f7a0 2023-10-30T14:09:50.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.697+0000 7f60917fa640 1 -- 172.21.15.46:0/2143430260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6088012ae0 con 0x7f609406f7a0 2023-10-30T14:09:50.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.697+0000 7f609aef7640 1 -- 172.21.15.46:0/2143430260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6094150020 con 0x7f609406f7a0 2023-10-30T14:09:50.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.697+0000 7f609aef7640 1 -- 172.21.15.46:0/2143430260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6094158af0 con 0x7f609406f7a0 2023-10-30T14:09:50.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.697+0000 7f609aef7640 1 -- 172.21.15.46:0/2143430260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609404e170 con 0x7f609406f7a0 2023-10-30T14:09:50.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.697+0000 7f60917fa640 1 -- 172.21.15.46:0/2143430260 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6088012c40 con 0x7f609406f7a0 2023-10-30T14:09:50.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.697+0000 7f60917fa640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60780664a0 0x7f6078068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:50.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.697+0000 7f60917fa640 1 -- 172.21.15.46:0/2143430260 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f6088084cd0 con 0x7f609406f7a0 2023-10-30T14:09:50.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.697+0000 7f6093fff640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60780664a0 0x7f6078068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:50.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.701+0000 7f6093fff640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60780664a0 0x7f6078068960 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f60940c4560 tx=0x7f6080009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:50.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.701+0000 7f60917fa640 1 -- 172.21.15.46:0/2143430260 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6088051840 con 0x7f609406f7a0 2023-10-30T14:09:50.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 -- 172.21.15.46:0/2143430260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60941510d0 con 0x7f609406f7a0 2023-10-30T14:09:50.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f60917fa640 1 -- 172.21.15.46:0/2143430260 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+5726 (secure 0 0 0) 0x7f6088007380 con 0x7f609406f7a0 2023-10-30T14:09:50.896 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:50.897 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:49.576232+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:50.897 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 203 2023-10-30T14:09:50.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 -- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60780664a0 msgr2=0x7f6078068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:50.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60780664a0 0x7f6078068960 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f60940c4560 tx=0x7f6080009040 comp rx=0 tx=0).stop 2023-10-30T14:09:50.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 -- 172.21.15.46:0/2143430260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f609406f7a0 msgr2=0x7f60941f1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:50.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f609406f7a0 0x7f60941f1a20 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f608800d5f0 tx=0x7f608800dab0 comp rx=0 tx=0).stop 2023-10-30T14:09:50.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 -- 172.21.15.46:0/2143430260 shutdown_connections 2023-10-30T14:09:50.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60780664a0 0x7f6078068960 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:50.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60940c6bd0 0x7f6094157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:50.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6094071110 0x7f6094157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:50.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 --2- 172.21.15.46:0/2143430260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f609406f7a0 0x7f60941f1a20 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:50.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 -- 172.21.15.46:0/2143430260 >> 172.21.15.46:0/2143430260 conn(0x7f60940c7910 msgr2=0x7f60940ca670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:50.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 -- 172.21.15.46:0/2143430260 shutdown_connections 2023-10-30T14:09:50.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:50.893+0000 7f609aef7640 1 -- 172.21.15.46:0/2143430260 wait complete. 2023-10-30T14:09:50.918 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T14:09:50.918 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:09:50.919 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:51.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.057+0000 7fabb7f3e640 1 Processor -- start 2023-10-30T14:09:51.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb7f3e640 1 -- start start 2023-10-30T14:09:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb7f3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb01538a0 0x7fabb015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb7f3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fabb00ca0e0 0x7fabb015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb7f3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fabb00cf160 0x7fabb00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb7f3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb00cb800 con 0x7fabb00ca0e0 2023-10-30T14:09:51.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb7f3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb015dc30 con 0x7fabb00cf160 2023-10-30T14:09:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb7f3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb00cb330 con 0x7fabb01538a0 2023-10-30T14:09:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb54d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fabb00ca0e0 0x7fabb015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb54d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fabb00ca0e0 0x7fabb015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48186/0 (socket says 172.21.15.46:48186) 2023-10-30T14:09:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb54d9640 1 -- 172.21.15.46:0/2747082564 learned_addr learned my addr 172.21.15.46:0/2747082564 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb64db640 1 --2- 172.21.15.46:0/2747082564 >> v2:172.21.15.154:3300/0 conn(0x7fabb00cf160 0x7fabb00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb5cda640 1 --2- 172.21.15.46:0/2747082564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb01538a0 0x7fabb015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb54d9640 1 -- 172.21.15.46:0/2747082564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb01538a0 msgr2=0x7fabb015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb54d9640 1 --2- 172.21.15.46:0/2747082564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb01538a0 0x7fabb015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb54d9640 1 -- 172.21.15.46:0/2747082564 >> v2:172.21.15.154:3300/0 conn(0x7fabb00cf160 msgr2=0x7fabb00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb54d9640 1 --2- 172.21.15.46:0/2747082564 >> v2:172.21.15.154:3300/0 conn(0x7fabb00cf160 0x7fabb00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.061+0000 7fabb54d9640 1 -- 172.21.15.46:0/2747082564 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb00cad70 con 0x7fabb00ca0e0 2023-10-30T14:09:51.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb54d9640 1 --2- 172.21.15.46:0/2747082564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabb00ca0e0 0x7fabb015a090 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fabb006cb00 tx=0x7faba001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b461a9069dbbf598 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:51.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb4cd8640 1 -- 172.21.15.46:0/2747082564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba00a89b0 con 0x7fabb00ca0e0 2023-10-30T14:09:51.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb4cd8640 1 -- 172.21.15.46:0/2747082564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba00a8b10 con 0x7fabb00ca0e0 2023-10-30T14:09:51.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb4cd8640 1 -- 172.21.15.46:0/2747082564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba00a8ea0 con 0x7fabb00ca0e0 2023-10-30T14:09:51.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb7f3e640 1 -- 172.21.15.46:0/2747082564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabb00ca0e0 msgr2=0x7fabb015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb7f3e640 1 --2- 172.21.15.46:0/2747082564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabb00ca0e0 0x7fabb015a090 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fabb006cb00 tx=0x7faba001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:09:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb7f3e640 1 -- 172.21.15.46:0/2747082564 shutdown_connections 2023-10-30T14:09:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb7f3e640 1 --2- 172.21.15.46:0/2747082564 >> v2:172.21.15.154:3300/0 conn(0x7fabb00cf160 0x7fabb00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb7f3e640 1 --2- 172.21.15.46:0/2747082564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabb00ca0e0 0x7fabb015a090 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb7f3e640 1 --2- 172.21.15.46:0/2747082564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb01538a0 0x7fabb015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:51.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb7f3e640 1 -- 172.21.15.46:0/2747082564 >> 172.21.15.46:0/2747082564 conn(0x7fabb006bfe0 msgr2=0x7fabb006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:51.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb7f3e640 1 -- 172.21.15.46:0/2747082564 shutdown_connections 2023-10-30T14:09:51.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb7f3e640 1 -- 172.21.15.46:0/2747082564 wait complete. 2023-10-30T14:09:51.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb7f3e640 1 Processor -- start 2023-10-30T14:09:51.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.065+0000 7fabb7f3e640 1 -- start start 2023-10-30T14:09:51.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb7f3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb00ca0e0 0x7fabb015d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb7f3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabb00cf160 0x7fabb014a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb7f3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb01538a0 0x7fabb01f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb7f3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb01f2fd0 con 0x7fabb00cf160 2023-10-30T14:09:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb7f3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb015ad70 con 0x7fabb01538a0 2023-10-30T14:09:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb7f3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb014bab0 con 0x7fabb00ca0e0 2023-10-30T14:09:51.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb54d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabb00cf160 0x7fabb014a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:51.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb54d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabb00cf160 0x7fabb014a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48192/0 (socket says 172.21.15.46:48192) 2023-10-30T14:09:51.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb54d9640 1 -- 172.21.15.46:0/1145678566 learned_addr learned my addr 172.21.15.46:0/1145678566 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:51.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb5cda640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb00ca0e0 0x7fabb015d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:51.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb64db640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb01538a0 0x7fabb01f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:51.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb54d9640 1 -- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb00ca0e0 msgr2=0x7fabb015d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:51.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb54d9640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb00ca0e0 0x7fabb015d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:51.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb54d9640 1 -- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb01538a0 msgr2=0x7fabb01f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:51.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb54d9640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb01538a0 0x7fabb01f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:51.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb54d9640 1 -- 172.21.15.46:0/1145678566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba00094d0 con 0x7fabb00cf160 2023-10-30T14:09:51.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb54d9640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabb00cf160 0x7fabb014a6f0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7faba00b3040 tx=0x7faba001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:51.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7faba67fc640 1 -- 172.21.15.46:0/1145678566 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba00ba600 con 0x7fabb00cf160 2023-10-30T14:09:51.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7faba67fc640 1 -- 172.21.15.46:0/1145678566 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba00ba760 con 0x7fabb00cf160 2023-10-30T14:09:51.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7faba67fc640 1 -- 172.21.15.46:0/1145678566 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba00bab10 con 0x7fabb00cf160 2023-10-30T14:09:51.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb7f3e640 1 -- 172.21.15.46:0/1145678566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb00ce130 con 0x7fabb00cf160 2023-10-30T14:09:51.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb7f3e640 1 -- 172.21.15.46:0/1145678566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb00ce380 con 0x7fabb00cf160 2023-10-30T14:09:51.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.069+0000 7fabb7f3e640 1 -- 172.21.15.46:0/1145678566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb00b2a50 con 0x7fabb00cf160 2023-10-30T14:09:51.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.073+0000 7faba67fc640 1 -- 172.21.15.46:0/1145678566 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faba00bac70 con 0x7fabb00cf160 2023-10-30T14:09:51.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.073+0000 7faba67fc640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fab94066520 0x7fab940689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:51.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.073+0000 7faba67fc640 1 -- 172.21.15.46:0/1145678566 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7faba0122cc0 con 0x7fabb00cf160 2023-10-30T14:09:51.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.073+0000 7fabb5cda640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fab94066520 0x7fab940689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:51.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.073+0000 7fabb5cda640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fab94066520 0x7fab940689e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fabac0079c0 tx=0x7fabac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:51.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.077+0000 7faba67fc640 1 -- 172.21.15.46:0/1145678566 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faba0129080 con 0x7fabb00cf160 2023-10-30T14:09:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.269+0000 7fabb7f3e640 1 -- 172.21.15.46:0/1145678566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabb01f3110 con 0x7fabb00cf160 2023-10-30T14:09:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.269+0000 7faba67fc640 1 -- 172.21.15.46:0/1145678566 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+5726 (secure 0 0 0) 0x7fabb01f3110 con 0x7fabb00cf160 2023-10-30T14:09:51.274 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:51.275 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:49.576232+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 203 2023-10-30T14:09:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.273+0000 7fabb7f3e640 1 -- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fab94066520 msgr2=0x7fab940689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.273+0000 7fabb7f3e640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fab94066520 0x7fab940689e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fabac0079c0 tx=0x7fabac008040 comp rx=0 tx=0).stop 2023-10-30T14:09:51.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.273+0000 7fabb7f3e640 1 -- 172.21.15.46:0/1145678566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabb00cf160 msgr2=0x7fabb014a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.273+0000 7fabb7f3e640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabb00cf160 0x7fabb014a6f0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7faba00b3040 tx=0x7faba001ba90 comp rx=0 tx=0).stop 2023-10-30T14:09:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.273+0000 7fabb7f3e640 1 -- 172.21.15.46:0/1145678566 shutdown_connections 2023-10-30T14:09:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.273+0000 7fabb7f3e640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fab94066520 0x7fab940689e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.273+0000 7fabb7f3e640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb01538a0 0x7fabb01f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.273+0000 7fabb7f3e640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabb00cf160 0x7fabb014a6f0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.273+0000 7fabb7f3e640 1 --2- 172.21.15.46:0/1145678566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb00ca0e0 0x7fabb015d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.273+0000 7fabb7f3e640 1 -- 172.21.15.46:0/1145678566 >> 172.21.15.46:0/1145678566 conn(0x7fabb006bfe0 msgr2=0x7fabb01483a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.273+0000 7fabb7f3e640 1 -- 172.21.15.46:0/1145678566 shutdown_connections 2023-10-30T14:09:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:51.273+0000 7fabb7f3e640 1 -- 172.21.15.46:0/1145678566 wait complete. 2023-10-30T14:09:51.296 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 203, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 203, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8655": { "addr": "172.21.15.46:6837/3044281789", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3044281789, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3044281789, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8655, "incarnation": 196, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_8826": { "addr": "172.21.15.46:6836/66265765", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 66265765, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 66265765, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8826, "incarnation": 203, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:09:49.576232+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8826, "mds_1": 8655 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:09:53.297 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:53.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c1cdc5640 1 Processor -- start 2023-10-30T14:09:53.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c1cdc5640 1 -- start start 2023-10-30T14:09:53.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c1cdc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c181538a0 0x7f2c1815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:53.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c1cdc5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c180cf160 0x7f2c180cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:53.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c1cdc5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c180cb800 con 0x7f2c180ca0e0 2023-10-30T14:09:53.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c1cdc5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1815dc30 con 0x7f2c180cf160 2023-10-30T14:09:53.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c1cdc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180cb330 con 0x7f2c181538a0 2023-10-30T14:09:53.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c16d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c180cf160 0x7f2c180cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:53.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c16d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c180cf160 0x7f2c180cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48202/0 (socket says 172.21.15.46:48202) 2023-10-30T14:09:53.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c16d9d640 1 -- 172.21.15.46:0/1405168177 learned_addr learned my addr 172.21.15.46:0/1405168177 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:53.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c1659c640 1 --2- 172.21.15.46:0/1405168177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c181538a0 0x7f2c1815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:53.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c16d9d640 1 -- 172.21.15.46:0/1405168177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c181538a0 msgr2=0x7f2c1815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:53.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c16d9d640 1 --2- 172.21.15.46:0/1405168177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c181538a0 0x7f2c1815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:53.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c16d9d640 1 -- 172.21.15.46:0/1405168177 >> v1:172.21.15.46:6789/0 conn(0x7f2c180ca0e0 legacy=0x7f2c1815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:53.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c16d9d640 1 -- 172.21.15.46:0/1405168177 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c180cad70 con 0x7f2c180cf160 2023-10-30T14:09:53.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.449+0000 7f2c1659c640 1 --2- 172.21.15.46:0/1405168177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c181538a0 0x7f2c1815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:09:53.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.453+0000 7f2c16d9d640 1 --2- 172.21.15.46:0/1405168177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c180cf160 0x7f2c180cf590 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009e50 tx=0x7f2c1001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721bf87bcc3f0523 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:53.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.453+0000 7f2c15d9b640 1 -- 172.21.15.46:0/1405168177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100af400 con 0x7f2c180cf160 2023-10-30T14:09:53.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.453+0000 7f2c15d9b640 1 -- 172.21.15.46:0/1405168177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c100ab050 con 0x7f2c180cf160 2023-10-30T14:09:53.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.453+0000 7f2c15d9b640 1 -- 172.21.15.46:0/1405168177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100af780 con 0x7f2c180cf160 2023-10-30T14:09:53.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.453+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/1405168177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c180cf160 msgr2=0x7f2c180cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:53.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.453+0000 7f2c1cdc5640 1 --2- 172.21.15.46:0/1405168177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c180cf160 0x7f2c180cf590 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009e50 tx=0x7f2c1001f1f0 comp rx=0 tx=0).stop 2023-10-30T14:09:53.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.453+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/1405168177 shutdown_connections 2023-10-30T14:09:53.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.453+0000 7f2c1cdc5640 1 --2- 172.21.15.46:0/1405168177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c180cf160 0x7f2c180cf590 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:53.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.453+0000 7f2c1cdc5640 1 --2- 172.21.15.46:0/1405168177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c181538a0 0x7f2c1815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:53.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.453+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/1405168177 >> 172.21.15.46:0/1405168177 conn(0x7f2c1806bfe0 msgr2=0x7f2c1806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:53.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.453+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/1405168177 shutdown_connections 2023-10-30T14:09:53.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.453+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/1405168177 wait complete. 2023-10-30T14:09:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c1cdc5640 1 Processor -- start 2023-10-30T14:09:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c1cdc5640 1 -- start start 2023-10-30T14:09:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c1cdc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c180ca0e0 0x7f2c180ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c1cdc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c180cf160 0x7f2c1815c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c1cdc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c181538a0 0x7f2c18144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c1cdc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c181442e0 con 0x7f2c181538a0 2023-10-30T14:09:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c1cdc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18143fe0 con 0x7f2c180cf160 2023-10-30T14:09:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c1cdc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1814c570 con 0x7f2c180ca0e0 2023-10-30T14:09:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c16d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c181538a0 0x7f2c18144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c16d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c181538a0 0x7f2c18144770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48216/0 (socket says 172.21.15.46:48216) 2023-10-30T14:09:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c16d9d640 1 -- 172.21.15.46:0/3980310505 learned_addr learned my addr 172.21.15.46:0/3980310505 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c1659c640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c180ca0e0 0x7f2c180ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c0ffff640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c180cf160 0x7f2c1815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c16d9d640 1 -- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c180ca0e0 msgr2=0x7f2c180ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c16d9d640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c180ca0e0 0x7f2c180ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c16d9d640 1 -- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c180cf160 msgr2=0x7f2c1815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c16d9d640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c180cf160 0x7f2c1815c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c16d9d640 1 -- 172.21.15.46:0/3980310505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c10009b30 con 0x7f2c181538a0 2023-10-30T14:09:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c16d9d640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c181538a0 0x7f2c18144770 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f2c100b2040 tx=0x7f2c1001bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c0effd640 1 -- 172.21.15.46:0/3980310505 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100afd30 con 0x7f2c181538a0 2023-10-30T14:09:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c0effd640 1 -- 172.21.15.46:0/3980310505 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c100afe90 con 0x7f2c181538a0 2023-10-30T14:09:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/3980310505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1814c850 con 0x7f2c181538a0 2023-10-30T14:09:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c0effd640 1 -- 172.21.15.46:0/3980310505 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100ba680 con 0x7f2c181538a0 2023-10-30T14:09:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.457+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/3980310505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1814ce10 con 0x7f2c181538a0 2023-10-30T14:09:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.461+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/3980310505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c180b2a50 con 0x7f2c181538a0 2023-10-30T14:09:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.461+0000 7f2c0effd640 1 -- 172.21.15.46:0/3980310505 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c100b6040 con 0x7f2c181538a0 2023-10-30T14:09:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.461+0000 7f2c0effd640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2bf80664a0 0x7f2bf8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.461+0000 7f2c0effd640 1 -- 172.21.15.46:0/3980310505 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f2c101228c0 con 0x7f2c181538a0 2023-10-30T14:09:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.461+0000 7f2c1659c640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2bf80664a0 0x7f2bf8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.461+0000 7f2c1659c640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2bf80664a0 0x7f2bf8068960 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2c080045f0 tx=0x7f2c08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.465+0000 7f2c0effd640 1 -- 172.21.15.46:0/3980310505 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c100ef6c0 con 0x7f2c181538a0 2023-10-30T14:09:53.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.657+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/3980310505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c18144490 con 0x7f2c181538a0 2023-10-30T14:09:53.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.657+0000 7f2c0effd640 1 -- 172.21.15.46:0/3980310505 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+5726 (secure 0 0 0) 0x7f2c18144490 con 0x7f2c181538a0 2023-10-30T14:09:53.663 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:53.663 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:49.576232+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:53.663 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 203 2023-10-30T14:09:53.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.661+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2bf80664a0 msgr2=0x7f2bf8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:53.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.661+0000 7f2c1cdc5640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2bf80664a0 0x7f2bf8068960 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2c080045f0 tx=0x7f2c08008040 comp rx=0 tx=0).stop 2023-10-30T14:09:53.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.661+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/3980310505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c181538a0 msgr2=0x7f2c18144770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:53.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.661+0000 7f2c1cdc5640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c181538a0 0x7f2c18144770 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f2c100b2040 tx=0x7f2c1001bf90 comp rx=0 tx=0).stop 2023-10-30T14:09:53.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.661+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/3980310505 shutdown_connections 2023-10-30T14:09:53.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.661+0000 7f2c1cdc5640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2bf80664a0 0x7f2bf8068960 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:53.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.661+0000 7f2c1cdc5640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c181538a0 0x7f2c18144770 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:53.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.661+0000 7f2c1cdc5640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c180cf160 0x7f2c1815c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:53.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.661+0000 7f2c1cdc5640 1 --2- 172.21.15.46:0/3980310505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c180ca0e0 0x7f2c180ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:53.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.661+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/3980310505 >> 172.21.15.46:0/3980310505 conn(0x7f2c1806bfe0 msgr2=0x7f2c18151320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:53.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.661+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/3980310505 shutdown_connections 2023-10-30T14:09:53.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:53.661+0000 7f2c1cdc5640 1 -- 172.21.15.46:0/3980310505 wait complete. 2023-10-30T14:09:53.684 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:09:55.685 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.845+0000 7f152b59e640 1 Processor -- start 2023-10-30T14:09:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.845+0000 7f152b59e640 1 -- start start 2023-10-30T14:09:55.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152b59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f152c1538f0 0x7f152c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:55.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152b59e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f152c15cd50 0x7f152c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:55.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152b59e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c15d300 con 0x7f152c0cefa0 2023-10-30T14:09:55.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152b59e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f152c15d5e0 con 0x7f152c15cd50 2023-10-30T14:09:55.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152b59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152c15d8c0 con 0x7f152c1538f0 2023-10-30T14:09:55.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152a59c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f152c1538f0 0x7f152c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:55.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f1529d9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f152c15cd50 0x7f152c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:55.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152a59c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f152c1538f0 0x7f152c159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:36750/0 (socket says 172.21.15.46:36750) 2023-10-30T14:09:55.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f1529d9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f152c15cd50 0x7f152c0c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:42374/0 (socket says 172.21.15.46:42374) 2023-10-30T14:09:55.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152ad9d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f152c0cefa0 0x7f152c0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:59918/0 (socket says 172.21.15.46:59918) 2023-10-30T14:09:55.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152a59c640 1 -- 172.21.15.46:0/3591936766 learned_addr learned my addr 172.21.15.46:0/3591936766 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:55.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152a59c640 1 -- 172.21.15.46:0/3591936766 >> v2:172.21.15.154:3300/0 conn(0x7f152c15cd50 msgr2=0x7f152c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:55.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152a59c640 1 --2- 172.21.15.46:0/3591936766 >> v2:172.21.15.154:3300/0 conn(0x7f152c15cd50 0x7f152c0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:55.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152a59c640 1 -- 172.21.15.46:0/3591936766 >> v1:172.21.15.154:6789/0 conn(0x7f152c0cefa0 legacy=0x7f152c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:55.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.849+0000 7f152a59c640 1 -- 172.21.15.46:0/3591936766 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c15dba0 con 0x7f152c1538f0 2023-10-30T14:09:55.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152a59c640 1 --2- 172.21.15.46:0/3591936766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f152c1538f0 0x7f152c159fb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f151c009870 tx=0x7f151c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3019485957556b02 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:55.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152959a640 1 -- 172.21.15.46:0/3591936766 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151c0a8a30 con 0x7f152c1538f0 2023-10-30T14:09:55.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152959a640 1 -- 172.21.15.46:0/3591936766 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151c0a8b90 con 0x7f152c1538f0 2023-10-30T14:09:55.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152959a640 1 -- 172.21.15.46:0/3591936766 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151c0b0440 con 0x7f152c1538f0 2023-10-30T14:09:55.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152b59e640 1 -- 172.21.15.46:0/3591936766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f152c1538f0 msgr2=0x7f152c159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:55.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152b59e640 1 --2- 172.21.15.46:0/3591936766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f152c1538f0 0x7f152c159fb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f151c009870 tx=0x7f151c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:09:55.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152b59e640 1 -- 172.21.15.46:0/3591936766 shutdown_connections 2023-10-30T14:09:55.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152b59e640 1 --2- 172.21.15.46:0/3591936766 >> v2:172.21.15.154:3300/0 conn(0x7f152c15cd50 0x7f152c0c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:55.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152b59e640 1 --2- 172.21.15.46:0/3591936766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f152c1538f0 0x7f152c159fb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:55.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152b59e640 1 -- 172.21.15.46:0/3591936766 >> 172.21.15.46:0/3591936766 conn(0x7f152c06a640 msgr2=0x7f152c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:55.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152b59e640 1 -- 172.21.15.46:0/3591936766 shutdown_connections 2023-10-30T14:09:55.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152b59e640 1 -- 172.21.15.46:0/3591936766 wait complete. 2023-10-30T14:09:55.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152b59e640 1 Processor -- start 2023-10-30T14:09:55.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.853+0000 7f152b59e640 1 -- start start 2023-10-30T14:09:55.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152b59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f152c0cefa0 0x7f152c14e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:55.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152b59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f152c1538f0 0x7f152c1429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:55.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f152c15cd50 0x7f152c15a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:55.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152b59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152c14ee30 con 0x7f152c15cd50 2023-10-30T14:09:55.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152b59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152c0cde70 con 0x7f152c1538f0 2023-10-30T14:09:55.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152b59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152c0cd9e0 con 0x7f152c0cefa0 2023-10-30T14:09:55.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f152c15cd50 0x7f152c15a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:55.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f152c15cd50 0x7f152c15a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48220/0 (socket says 172.21.15.46:48220) 2023-10-30T14:09:55.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152ad9d640 1 -- 172.21.15.46:0/93492510 learned_addr learned my addr 172.21.15.46:0/93492510 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:55.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152a59c640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f152c0cefa0 0x7f152c14e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:55.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f1529d9b640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f152c1538f0 0x7f152c1429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:55.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152ad9d640 1 -- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f152c0cefa0 msgr2=0x7f152c14e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:55.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152ad9d640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f152c0cefa0 0x7f152c14e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:55.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152ad9d640 1 -- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f152c1538f0 msgr2=0x7f152c1429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:55.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152ad9d640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f152c1538f0 0x7f152c1429d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:55.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152ad9d640 1 -- 172.21.15.46:0/93492510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c0094d0 con 0x7f152c15cd50 2023-10-30T14:09:55.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152ad9d640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f152c15cd50 0x7f152c15a900 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f152400c7d0 tx=0x7f152400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:55.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f151affd640 1 -- 172.21.15.46:0/93492510 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152400a640 con 0x7f152c15cd50 2023-10-30T14:09:55.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f151affd640 1 -- 172.21.15.46:0/93492510 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152400a7a0 con 0x7f152c15cd50 2023-10-30T14:09:55.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152b59e640 1 -- 172.21.15.46:0/93492510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152c14fca0 con 0x7f152c15cd50 2023-10-30T14:09:55.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f151affd640 1 -- 172.21.15.46:0/93492510 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152400ab10 con 0x7f152c15cd50 2023-10-30T14:09:55.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152b59e640 1 -- 172.21.15.46:0/93492510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152c0cea50 con 0x7f152c15cd50 2023-10-30T14:09:55.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.857+0000 7f152b59e640 1 -- 172.21.15.46:0/93492510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152c0cdb50 con 0x7f152c15cd50 2023-10-30T14:09:55.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.861+0000 7f151affd640 1 -- 172.21.15.46:0/93492510 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f152400ac70 con 0x7f152c15cd50 2023-10-30T14:09:55.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.861+0000 7f151affd640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f150c0664d0 0x7f150c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:55.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.861+0000 7f151affd640 1 -- 172.21.15.46:0/93492510 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f1524083f40 con 0x7f152c15cd50 2023-10-30T14:09:55.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.861+0000 7f152a59c640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f150c0664d0 0x7f150c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:55.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.865+0000 7f152a59c640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f150c0664d0 0x7f150c068990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f151c0b2000 tx=0x7f151c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:55.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:55.865+0000 7f151affd640 1 -- 172.21.15.46:0/93492510 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1524050ad0 con 0x7f152c15cd50 2023-10-30T14:09:56.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.065+0000 7f152b59e640 1 -- 172.21.15.46:0/93492510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f152c04e170 con 0x7f152c15cd50 2023-10-30T14:09:56.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.065+0000 7f151affd640 1 -- 172.21.15.46:0/93492510 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+5726 (secure 0 0 0) 0x7f1524017070 con 0x7f152c15cd50 2023-10-30T14:09:56.070 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:56.070 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:49.576232+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:56.070 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 203 2023-10-30T14:09:56.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.069+0000 7f152b59e640 1 -- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f150c0664d0 msgr2=0x7f150c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:56.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.069+0000 7f152b59e640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f150c0664d0 0x7f150c068990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f151c0b2000 tx=0x7f151c009040 comp rx=0 tx=0).stop 2023-10-30T14:09:56.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.069+0000 7f152b59e640 1 -- 172.21.15.46:0/93492510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f152c15cd50 msgr2=0x7f152c15a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:56.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.069+0000 7f152b59e640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f152c15cd50 0x7f152c15a900 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f152400c7d0 tx=0x7f152400cc90 comp rx=0 tx=0).stop 2023-10-30T14:09:56.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.069+0000 7f152b59e640 1 -- 172.21.15.46:0/93492510 shutdown_connections 2023-10-30T14:09:56.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.069+0000 7f152b59e640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f150c0664d0 0x7f150c068990 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:56.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.069+0000 7f152b59e640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f152c15cd50 0x7f152c15a900 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:56.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.069+0000 7f152b59e640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f152c1538f0 0x7f152c1429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:56.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.069+0000 7f152b59e640 1 --2- 172.21.15.46:0/93492510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f152c0cefa0 0x7f152c14e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:56.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.069+0000 7f152b59e640 1 -- 172.21.15.46:0/93492510 >> 172.21.15.46:0/93492510 conn(0x7f152c06a640 msgr2=0x7f152c0caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:56.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.069+0000 7f152b59e640 1 -- 172.21.15.46:0/93492510 shutdown_connections 2023-10-30T14:09:56.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:56.069+0000 7f152b59e640 1 -- 172.21.15.46:0/93492510 wait complete. 2023-10-30T14:09:56.093 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:09:57.350 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:09:57.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:09:58.094 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:09:58.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.261+0000 7fb0c9f3e640 1 Processor -- start 2023-10-30T14:09:58.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.261+0000 7fb0c9f3e640 1 -- start start 2023-10-30T14:09:58.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c9f3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb0c41538a0 0x7fb0c415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:58.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c9f3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb0c40ca0e0 0x7fb0c415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:58.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c9f3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c40cb800 con 0x7fb0c40cf160 2023-10-30T14:09:58.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c9f3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c415dc30 con 0x7fb0c41538a0 2023-10-30T14:09:58.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c9f3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c40cb330 con 0x7fb0c40ca0e0 2023-10-30T14:09:58.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb0c41538a0 0x7fb0c415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:58.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb0c41538a0 0x7fb0c415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35988/0 (socket says 172.21.15.46:35988) 2023-10-30T14:09:58.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c37fe640 1 -- 172.21.15.46:0/1882564058 learned_addr learned my addr 172.21.15.46:0/1882564058 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:58.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c2ffd640 1 --2- 172.21.15.46:0/1882564058 >> v2:172.21.15.154:3300/0 conn(0x7fb0c40ca0e0 0x7fb0c415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:58.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c37fe640 1 -- 172.21.15.46:0/1882564058 >> v2:172.21.15.154:3300/0 conn(0x7fb0c40ca0e0 msgr2=0x7fb0c415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:58.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c37fe640 1 --2- 172.21.15.46:0/1882564058 >> v2:172.21.15.154:3300/0 conn(0x7fb0c40ca0e0 0x7fb0c415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:58.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c37fe640 1 -- 172.21.15.46:0/1882564058 >> v1:172.21.15.154:6789/0 conn(0x7fb0c40cf160 legacy=0x7fb0c40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:09:58.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.265+0000 7fb0c37fe640 1 -- 172.21.15.46:0/1882564058 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c40cad70 con 0x7fb0c41538a0 2023-10-30T14:09:58.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.269+0000 7fb0c37fe640 1 --2- 172.21.15.46:0/1882564058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c41538a0 0x7fb0c415d6f0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40b30a0 tx=0x7fb0b401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68d31352501b479 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:58.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.269+0000 7fb0c27fc640 1 -- 172.21.15.46:0/1882564058 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40a89f0 con 0x7fb0c41538a0 2023-10-30T14:09:58.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.269+0000 7fb0c27fc640 1 -- 172.21.15.46:0/1882564058 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b40a8b50 con 0x7fb0c41538a0 2023-10-30T14:09:58.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.269+0000 7fb0c27fc640 1 -- 172.21.15.46:0/1882564058 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40b0440 con 0x7fb0c41538a0 2023-10-30T14:09:58.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.269+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/1882564058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c41538a0 msgr2=0x7fb0c415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:58.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.269+0000 7fb0c9f3e640 1 --2- 172.21.15.46:0/1882564058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c41538a0 0x7fb0c415d6f0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40b30a0 tx=0x7fb0b401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:09:58.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.269+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/1882564058 shutdown_connections 2023-10-30T14:09:58.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.269+0000 7fb0c9f3e640 1 --2- 172.21.15.46:0/1882564058 >> v2:172.21.15.154:3300/0 conn(0x7fb0c40ca0e0 0x7fb0c415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:58.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.269+0000 7fb0c9f3e640 1 --2- 172.21.15.46:0/1882564058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c41538a0 0x7fb0c415d6f0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:58.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.269+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/1882564058 >> 172.21.15.46:0/1882564058 conn(0x7fb0c406bfe0 msgr2=0x7fb0c406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:58.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.269+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/1882564058 shutdown_connections 2023-10-30T14:09:58.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.269+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/1882564058 wait complete. 2023-10-30T14:09:58.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c9f3e640 1 Processor -- start 2023-10-30T14:09:58.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c9f3e640 1 -- start start 2023-10-30T14:09:58.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c9f3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0c40ca0e0 0x7fb0c4158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:58.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c9f3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c40cf160 0x7fb0c40c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:58.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c9f3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c41538a0 0x7fb0c41ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:58.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c9f3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c41e4db0 con 0x7fb0c41538a0 2023-10-30T14:09:58.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c9f3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c41ef520 con 0x7fb0c40ca0e0 2023-10-30T14:09:58.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c9f3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c41f2960 con 0x7fb0c40cf160 2023-10-30T14:09:58.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c41538a0 0x7fb0c41ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:58.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0c40ca0e0 0x7fb0c4158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:58.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0c40ca0e0 0x7fb0c4158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:50226/0 (socket says 172.21.15.46:50226) 2023-10-30T14:09:58.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c2ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c40cf160 0x7fb0c40c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:58.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c37fe640 1 -- 172.21.15.46:0/566504883 learned_addr learned my addr 172.21.15.46:0/566504883 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:09:58.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c3fff640 1 -- 172.21.15.46:0/566504883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c40cf160 msgr2=0x7fb0c40c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:58.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c3fff640 1 --2- 172.21.15.46:0/566504883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c40cf160 0x7fb0c40c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:58.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c3fff640 1 -- 172.21.15.46:0/566504883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0c40ca0e0 msgr2=0x7fb0c4158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:58.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c3fff640 1 --2- 172.21.15.46:0/566504883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0c40ca0e0 0x7fb0c4158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:58.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c3fff640 1 -- 172.21.15.46:0/566504883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b40094d0 con 0x7fb0c41538a0 2023-10-30T14:09:58.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0c3fff640 1 --2- 172.21.15.46:0/566504883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c41538a0 0x7fb0c41ef9b0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00c700 tx=0x7fb0bc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:58.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.273+0000 7fb0abfff640 1 -- 172.21.15.46:0/566504883 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc003b80 con 0x7fb0c41538a0 2023-10-30T14:09:58.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.277+0000 7fb0abfff640 1 -- 172.21.15.46:0/566504883 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc003ce0 con 0x7fb0c41538a0 2023-10-30T14:09:58.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.277+0000 7fb0abfff640 1 -- 172.21.15.46:0/566504883 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc005310 con 0x7fb0c41538a0 2023-10-30T14:09:58.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.277+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/566504883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c41f2c20 con 0x7fb0c41538a0 2023-10-30T14:09:58.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.277+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/566504883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c41f31e0 con 0x7fb0c41538a0 2023-10-30T14:09:58.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.277+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/566504883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c40b2a50 con 0x7fb0c41538a0 2023-10-30T14:09:58.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.277+0000 7fb0abfff640 1 -- 172.21.15.46:0/566504883 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb0bc005470 con 0x7fb0c41538a0 2023-10-30T14:09:58.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.281+0000 7fb0abfff640 1 --2- 172.21.15.46:0/566504883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0b00664d0 0x7fb0b0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:09:58.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.281+0000 7fb0abfff640 1 -- 172.21.15.46:0/566504883 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7fb0bc084390 con 0x7fb0c41538a0 2023-10-30T14:09:58.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.281+0000 7fb0c37fe640 1 --2- 172.21.15.46:0/566504883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0b00664d0 0x7fb0b0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:09:58.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.281+0000 7fb0c37fe640 1 --2- 172.21.15.46:0/566504883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0b00664d0 0x7fb0b0068990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fb0b401ed30 tx=0x7fb0b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:09:58.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.285+0000 7fb0abfff640 1 -- 172.21.15.46:0/566504883 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb0bc050f20 con 0x7fb0c41538a0 2023-10-30T14:09:58.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.521+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/566504883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0c4159850 con 0x7fb0c41538a0 2023-10-30T14:09:58.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.521+0000 7fb0abfff640 1 -- 172.21.15.46:0/566504883 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+5726 (secure 0 0 0) 0x7fb0bc054540 con 0x7fb0c41538a0 2023-10-30T14:09:58.525 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:09:58.526 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:49.576232+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:09:58.526 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 203 2023-10-30T14:09:58.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.525+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/566504883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0b00664d0 msgr2=0x7fb0b0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:58.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.525+0000 7fb0c9f3e640 1 --2- 172.21.15.46:0/566504883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0b00664d0 0x7fb0b0068990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fb0b401ed30 tx=0x7fb0b4009040 comp rx=0 tx=0).stop 2023-10-30T14:09:58.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.525+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/566504883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c41538a0 msgr2=0x7fb0c41ef9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:09:58.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.525+0000 7fb0c9f3e640 1 --2- 172.21.15.46:0/566504883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c41538a0 0x7fb0c41ef9b0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00c700 tx=0x7fb0bc00cbc0 comp rx=0 tx=0).stop 2023-10-30T14:09:58.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.525+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/566504883 shutdown_connections 2023-10-30T14:09:58.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.525+0000 7fb0c9f3e640 1 --2- 172.21.15.46:0/566504883 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0b00664d0 0x7fb0b0068990 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:58.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.525+0000 7fb0c9f3e640 1 --2- 172.21.15.46:0/566504883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0c41538a0 0x7fb0c41ef9b0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.525+0000 7fb0c9f3e640 1 --2- 172.21.15.46:0/566504883 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0c40cf160 0x7fb0c40c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.525+0000 7fb0c9f3e640 1 --2- 172.21.15.46:0/566504883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0c40ca0e0 0x7fb0c4158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:09:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.525+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/566504883 >> 172.21.15.46:0/566504883 conn(0x7fb0c406bfe0 msgr2=0x7fb0c4151f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:09:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.525+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/566504883 shutdown_connections 2023-10-30T14:09:58.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:09:58.525+0000 7fb0c9f3e640 1 -- 172.21.15.46:0/566504883 wait complete. 2023-10-30T14:09:58.551 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:10:00.552 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:10:00.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f203359e640 1 Processor -- start 2023-10-30T14:10:00.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f203359e640 1 -- start start 2023-10-30T14:10:00.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f203359e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20341538f0 0x7f2034159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:00.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f203359e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f20340cefa0 0x7f20340cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:00.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f203359e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20340ca6a0 con 0x7f203415cd50 2023-10-30T14:10:00.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f203359e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f203415d300 con 0x7f20340cefa0 2023-10-30T14:10:00.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f203359e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203415d5e0 con 0x7f20341538f0 2023-10-30T14:10:00.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f202bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f203415cd50 0x7f20340c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53792/0 (socket says 172.21.15.46:53792) 2023-10-30T14:10:00.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f202bfff640 1 -- 172.21.15.46:0/703800580 learned_addr learned my addr 172.21.15.46:0/703800580 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:10:00.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f2032d9d640 1 --2- 172.21.15.46:0/703800580 >> v2:172.21.15.46:3300/0 conn(0x7f20340cefa0 0x7f20340cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:00.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f203259c640 1 --2- 172.21.15.46:0/703800580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20341538f0 0x7f2034159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:00.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f2032d9d640 1 -- 172.21.15.46:0/703800580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20341538f0 msgr2=0x7f2034159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:00.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f2032d9d640 1 --2- 172.21.15.46:0/703800580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20341538f0 0x7f2034159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:00.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f2032d9d640 1 -- 172.21.15.46:0/703800580 >> v1:172.21.15.46:6789/0 conn(0x7f203415cd50 legacy=0x7f20340c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:00.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f2032d9d640 1 -- 172.21.15.46:0/703800580 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203415d8c0 con 0x7f20340cefa0 2023-10-30T14:10:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.705+0000 7f203259c640 1 --2- 172.21.15.46:0/703800580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20341538f0 0x7f2034159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:10:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.709+0000 7f2032d9d640 1 --2- 172.21.15.46:0/703800580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20340cefa0 0x7f20340cf3f0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f202c009d90 tx=0x7f202c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1308c1237b5d53f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.709+0000 7f2031d9b640 1 -- 172.21.15.46:0/703800580 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0af600 con 0x7f20340cefa0 2023-10-30T14:10:00.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.709+0000 7f2031d9b640 1 -- 172.21.15.46:0/703800580 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c0ab050 con 0x7f20340cefa0 2023-10-30T14:10:00.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.709+0000 7f2031d9b640 1 -- 172.21.15.46:0/703800580 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0af980 con 0x7f20340cefa0 2023-10-30T14:10:00.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.709+0000 7f203359e640 1 -- 172.21.15.46:0/703800580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20340cefa0 msgr2=0x7f20340cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:00.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.709+0000 7f203359e640 1 --2- 172.21.15.46:0/703800580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20340cefa0 0x7f20340cf3f0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f202c009d90 tx=0x7f202c01f130 comp rx=0 tx=0).stop 2023-10-30T14:10:00.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.709+0000 7f203359e640 1 -- 172.21.15.46:0/703800580 shutdown_connections 2023-10-30T14:10:00.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.709+0000 7f203359e640 1 --2- 172.21.15.46:0/703800580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20340cefa0 0x7f20340cf3f0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:00.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.709+0000 7f203359e640 1 --2- 172.21.15.46:0/703800580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20341538f0 0x7f2034159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:00.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.709+0000 7f203359e640 1 -- 172.21.15.46:0/703800580 >> 172.21.15.46:0/703800580 conn(0x7f203406a640 msgr2=0x7f2034065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:00.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.709+0000 7f203359e640 1 -- 172.21.15.46:0/703800580 shutdown_connections 2023-10-30T14:10:00.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.709+0000 7f203359e640 1 -- 172.21.15.46:0/703800580 wait complete. 2023-10-30T14:10:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f203359e640 1 Processor -- start 2023-10-30T14:10:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f203359e640 1 -- start start 2023-10-30T14:10:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f203359e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20340cefa0 0x7f20341ef040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:00.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f203359e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20341538f0 0x7f20341f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f203359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f203415cd50 0x7f2034148ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f203359e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20341e3cf0 con 0x7f203415cd50 2023-10-30T14:10:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f203359e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2034153130 con 0x7f20340cefa0 2023-10-30T14:10:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f203359e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20341f2580 con 0x7f20341538f0 2023-10-30T14:10:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f2032d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f203415cd50 0x7f2034148ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:00.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f2032d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f203415cd50 0x7f2034148ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36032/0 (socket says 172.21.15.46:36032) 2023-10-30T14:10:00.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f2032d9d640 1 -- 172.21.15.46:0/2697982364 learned_addr learned my addr 172.21.15.46:0/2697982364 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:00.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f202bfff640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20341538f0 0x7f20341f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:00.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f203259c640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20340cefa0 0x7f20341ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:00.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f2032d9d640 1 -- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20341538f0 msgr2=0x7f20341f3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:00.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f2032d9d640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20341538f0 0x7f20341f3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:00.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f2032d9d640 1 -- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20340cefa0 msgr2=0x7f20341ef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:00.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f2032d9d640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20340cefa0 0x7f20341ef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:00.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f2032d9d640 1 -- 172.21.15.46:0/2697982364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202c009a70 con 0x7f203415cd50 2023-10-30T14:10:00.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f2032d9d640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f203415cd50 0x7f2034148ca0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f202c01cd60 tx=0x7f202c01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:00.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f202affd640 1 -- 172.21.15.46:0/2697982364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0afd50 con 0x7f203415cd50 2023-10-30T14:10:00.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f202affd640 1 -- 172.21.15.46:0/2697982364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c0ba420 con 0x7f203415cd50 2023-10-30T14:10:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f202affd640 1 -- 172.21.15.46:0/2697982364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0ba7d0 con 0x7f203415cd50 2023-10-30T14:10:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.713+0000 7f203359e640 1 -- 172.21.15.46:0/2697982364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203415c370 con 0x7f203415cd50 2023-10-30T14:10:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.717+0000 7f203359e640 1 -- 172.21.15.46:0/2697982364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203415c930 con 0x7f203415cd50 2023-10-30T14:10:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.717+0000 7f203359e640 1 -- 172.21.15.46:0/2697982364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203404e170 con 0x7f203415cd50 2023-10-30T14:10:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.717+0000 7f202affd640 1 -- 172.21.15.46:0/2697982364 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f202c0ba930 con 0x7f203415cd50 2023-10-30T14:10:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.717+0000 7f202affd640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20140664a0 0x7f2014068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.717+0000 7f202affd640 1 -- 172.21.15.46:0/2697982364 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f202c123270 con 0x7f203415cd50 2023-10-30T14:10:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.717+0000 7f203259c640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20140664a0 0x7f2014068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.717+0000 7f203259c640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20140664a0 0x7f2014068960 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f2024009950 tx=0x7f2024008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:00.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.721+0000 7f202affd640 1 -- 172.21.15.46:0/2697982364 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f202c0efd70 con 0x7f203415cd50 2023-10-30T14:10:00.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.913+0000 7f203359e640 1 -- 172.21.15.46:0/2697982364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20340b2290 con 0x7f203415cd50 2023-10-30T14:10:00.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.913+0000 7f202affd640 1 -- 172.21.15.46:0/2697982364 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+5728 (secure 0 0 0) 0x7f202c0b4090 con 0x7f203415cd50 2023-10-30T14:10:00.918 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:10:00.919 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:59.299411+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:resolve","state_seq":11,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:10:00.919 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 204 2023-10-30T14:10:00.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.917+0000 7f203359e640 1 -- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20140664a0 msgr2=0x7f2014068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:00.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.917+0000 7f203359e640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20140664a0 0x7f2014068960 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f2024009950 tx=0x7f2024008040 comp rx=0 tx=0).stop 2023-10-30T14:10:00.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.917+0000 7f203359e640 1 -- 172.21.15.46:0/2697982364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f203415cd50 msgr2=0x7f2034148ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:00.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.917+0000 7f203359e640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f203415cd50 0x7f2034148ca0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f202c01cd60 tx=0x7f202c01bed0 comp rx=0 tx=0).stop 2023-10-30T14:10:00.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.917+0000 7f203359e640 1 -- 172.21.15.46:0/2697982364 shutdown_connections 2023-10-30T14:10:00.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.917+0000 7f203359e640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f20140664a0 0x7f2014068960 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:00.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.917+0000 7f203359e640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f203415cd50 0x7f2034148ca0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:00.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.917+0000 7f203359e640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20341538f0 0x7f20341f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:00.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.917+0000 7f203359e640 1 --2- 172.21.15.46:0/2697982364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20340cefa0 0x7f20341ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:00.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.917+0000 7f203359e640 1 -- 172.21.15.46:0/2697982364 >> 172.21.15.46:0/2697982364 conn(0x7f203406a640 msgr2=0x7f20340cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:00.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.917+0000 7f203359e640 1 -- 172.21.15.46:0/2697982364 shutdown_connections 2023-10-30T14:10:00.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:00.917+0000 7f203359e640 1 -- 172.21.15.46:0/2697982364 wait complete. 2023-10-30T14:10:00.940 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:10:02.941 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:10:03.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.089+0000 7f77937d3640 1 Processor -- start 2023-10-30T14:10:03.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.089+0000 7f77937d3640 1 -- start start 2023-10-30T14:10:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f77937d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f778c070740 0x7f778c0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f77937d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c0c5810 con 0x7f778c1580e0 2023-10-30T14:10:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f77937d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c15eb70 con 0x7f778c15e700 2023-10-30T14:10:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f77937d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f778c06acb0 con 0x7f778c070740 2023-10-30T14:10:03.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f779156f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f778c1580e0 0x7f778c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53798/0 (socket says 172.21.15.46:53798) 2023-10-30T14:10:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f779156f640 1 -- 172.21.15.46:0/1936861748 learned_addr learned my addr 172.21.15.46:0/1936861748 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:10:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7790d6e640 1 --2- 172.21.15.46:0/1936861748 >> v2:172.21.15.154:3300/0 conn(0x7f778c070740 0x7f778c0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498626569 0 0) 0x7f778c0c5810 con 0x7f778c1580e0 2023-10-30T14:10:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7778003530 con 0x7f778c1580e0 2023-10-30T14:10:03.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 660739971 0 0) 0x7f7778003530 con 0x7f778c1580e0 2023-10-30T14:10:03.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 >> v2:172.21.15.154:3300/0 conn(0x7f778c070740 msgr2=0x7f778c0c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:03.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 --2- 172.21.15.46:0/1936861748 >> v2:172.21.15.154:3300/0 conn(0x7f778c070740 0x7f778c0c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:03.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 >> v1:172.21.15.154:6789/0 conn(0x7f778c15e700 legacy=0x7f778c06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:03.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c0c6bb0 con 0x7f778c1580e0 2023-10-30T14:10:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f7788001680 con 0x7f778c1580e0 2023-10-30T14:10:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 >> v1:172.21.15.46:6789/0 conn(0x7f778c1580e0 legacy=0x7f778c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 --2- 172.21.15.46:0/1936861748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778003980 0x7f7778003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 --2- 172.21.15.46:0/1936861748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77780047d0 0x7f7778004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 --2- 172.21.15.46:0/1936861748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7778005190 0x7f777800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777800c000 con 0x7f7778005190 2023-10-30T14:10:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777800c170 con 0x7f77780047d0 2023-10-30T14:10:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777800c330 con 0x7f7778003980 2023-10-30T14:10:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7791d70640 1 --2- 172.21.15.46:0/1936861748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7778005190 0x7f777800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f779156f640 1 --2- 172.21.15.46:0/1936861748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778003980 0x7f7778003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7790d6e640 1 --2- 172.21.15.46:0/1936861748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77780047d0 0x7f7778004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7791d70640 1 -- 172.21.15.46:0/1936861748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778003980 msgr2=0x7f7778003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7791d70640 1 --2- 172.21.15.46:0/1936861748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778003980 0x7f7778003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7791d70640 1 -- 172.21.15.46:0/1936861748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77780047d0 msgr2=0x7f7778004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7791d70640 1 --2- 172.21.15.46:0/1936861748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77780047d0 0x7f7778004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.093+0000 7f7791d70640 1 -- 172.21.15.46:0/1936861748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7778003530 con 0x7f7778005190 2023-10-30T14:10:03.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.097+0000 7f7791d70640 1 -- 172.21.15.46:0/1936861748 reap_dead start 2023-10-30T14:10:03.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.097+0000 7f7791d70640 1 --2- 172.21.15.46:0/1936861748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7778005190 0x7f777800b990 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f7784001e70 tx=0x7f778401c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a14bc69ae68b2acc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.097+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784003b80 con 0x7f7778005190 2023-10-30T14:10:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.097+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7784003d00 con 0x7f7778005190 2023-10-30T14:10:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.097+0000 7f7783fff640 1 -- 172.21.15.46:0/1936861748 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784005260 con 0x7f7778005190 2023-10-30T14:10:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.097+0000 7f77937d3640 1 -- 172.21.15.46:0/1936861748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7778005190 msgr2=0x7f777800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.097+0000 7f77937d3640 1 --2- 172.21.15.46:0/1936861748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7778005190 0x7f777800b990 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f7784001e70 tx=0x7f778401c760 comp rx=0 tx=0).stop 2023-10-30T14:10:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.097+0000 7f77937d3640 1 -- 172.21.15.46:0/1936861748 shutdown_connections 2023-10-30T14:10:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.097+0000 7f77937d3640 1 --2- 172.21.15.46:0/1936861748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7778005190 0x7f777800b990 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.097+0000 7f77937d3640 1 -- 172.21.15.46:0/1936861748 >> 172.21.15.46:0/1936861748 conn(0x7f778c0c7910 msgr2=0x7f778c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.097+0000 7f77937d3640 1 -- 172.21.15.46:0/1936861748 shutdown_connections 2023-10-30T14:10:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.097+0000 7f77937d3640 1 -- 172.21.15.46:0/1936861748 wait complete. 2023-10-30T14:10:03.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f77937d3640 1 Processor -- start 2023-10-30T14:10:03.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f77937d3640 1 -- start start 2023-10-30T14:10:03.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f77937d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778005190 0x7f778c158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:03.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f77937d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c1f7180 0x7f778c0c11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:03.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f77937d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c15c7a0 0x7f778c15cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:03.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f77937d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c149450 con 0x7f778c15c7a0 2023-10-30T14:10:03.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f77937d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c1f6cf0 con 0x7f778c1f7180 2023-10-30T14:10:03.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f77937d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778c159510 con 0x7f7778005190 2023-10-30T14:10:03.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7791d70640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c15c7a0 0x7f778c15cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:03.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7791d70640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c15c7a0 0x7f778c15cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36060/0 (socket says 172.21.15.46:36060) 2023-10-30T14:10:03.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7791d70640 1 -- 172.21.15.46:0/1360569639 learned_addr learned my addr 172.21.15.46:0/1360569639 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:03.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f779156f640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778005190 0x7f778c158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:03.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7790d6e640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c1f7180 0x7f778c0c11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:03.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7791d70640 1 -- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778005190 msgr2=0x7f778c158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:03.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7791d70640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778005190 0x7f778c158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:03.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7791d70640 1 -- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c1f7180 msgr2=0x7f778c0c11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:03.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7791d70640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c1f7180 0x7f778c0c11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:03.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7791d70640 1 -- 172.21.15.46:0/1360569639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c148d10 con 0x7f778c15c7a0 2023-10-30T14:10:03.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f779156f640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778005190 0x7f778c158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:10:03.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7790d6e640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c1f7180 0x7f778c0c11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:10:03.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7791d70640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c15c7a0 0x7f778c15cc20 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f778401c590 tx=0x7f7778005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:03.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7781ffb640 1 -- 172.21.15.46:0/1360569639 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77840ac070 con 0x7f778c15c7a0 2023-10-30T14:10:03.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7781ffb640 1 -- 172.21.15.46:0/1360569639 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7784003b90 con 0x7f778c15c7a0 2023-10-30T14:10:03.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f7781ffb640 1 -- 172.21.15.46:0/1360569639 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784006d80 con 0x7f778c15c7a0 2023-10-30T14:10:03.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f77937d3640 1 -- 172.21.15.46:0/1360569639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778c1f66a0 con 0x7f778c15c7a0 2023-10-30T14:10:03.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.101+0000 7f77937d3640 1 -- 172.21.15.46:0/1360569639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778c148ff0 con 0x7f778c15c7a0 2023-10-30T14:10:03.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.105+0000 7f77937d3640 1 -- 172.21.15.46:0/1360569639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778c04e170 con 0x7f778c15c7a0 2023-10-30T14:10:03.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.105+0000 7f7781ffb640 1 -- 172.21.15.46:0/1360569639 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7784007030 con 0x7f778c15c7a0 2023-10-30T14:10:03.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.105+0000 7f7781ffb640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f77700664d0 0x7f7770068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:03.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.105+0000 7f7781ffb640 1 -- 172.21.15.46:0/1360569639 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f778411d570 con 0x7f778c15c7a0 2023-10-30T14:10:03.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.105+0000 7f779156f640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f77700664d0 0x7f7770068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:03.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.105+0000 7f779156f640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f77700664d0 0x7f7770068990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7788000c10 tx=0x7f7788009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:03.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.109+0000 7f7781ffb640 1 -- 172.21.15.46:0/1360569639 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f77840ea100 con 0x7f778c15c7a0 2023-10-30T14:10:03.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.301+0000 7f77937d3640 1 -- 172.21.15.46:0/1360569639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f778c1f6e30 con 0x7f778c15c7a0 2023-10-30T14:10:03.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.301+0000 7f7781ffb640 1 -- 172.21.15.46:0/1360569639 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+5728 (secure 0 0 0) 0x7f77840b5020 con 0x7f778c15c7a0 2023-10-30T14:10:03.306 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:10:03.306 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:59.299411+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:resolve","state_seq":11,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:10:03.307 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 204 2023-10-30T14:10:03.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.301+0000 7f77937d3640 1 -- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f77700664d0 msgr2=0x7f7770068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:03.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.301+0000 7f77937d3640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f77700664d0 0x7f7770068990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7788000c10 tx=0x7f7788009040 comp rx=0 tx=0).stop 2023-10-30T14:10:03.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.301+0000 7f77937d3640 1 -- 172.21.15.46:0/1360569639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c15c7a0 msgr2=0x7f778c15cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:03.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.301+0000 7f77937d3640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c15c7a0 0x7f778c15cc20 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f778401c590 tx=0x7f7778005800 comp rx=0 tx=0).stop 2023-10-30T14:10:03.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.301+0000 7f77937d3640 1 -- 172.21.15.46:0/1360569639 shutdown_connections 2023-10-30T14:10:03.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.301+0000 7f77937d3640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f77700664d0 0x7f7770068990 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:03.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.301+0000 7f77937d3640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f778c15c7a0 0x7f778c15cc20 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:03.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.301+0000 7f77937d3640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c1f7180 0x7f778c0c11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:03.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.301+0000 7f77937d3640 1 --2- 172.21.15.46:0/1360569639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778005190 0x7f778c158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:03.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.301+0000 7f77937d3640 1 -- 172.21.15.46:0/1360569639 >> 172.21.15.46:0/1360569639 conn(0x7f778c0c7910 msgr2=0x7f778c071830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:03.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.305+0000 7f77937d3640 1 -- 172.21.15.46:0/1360569639 shutdown_connections 2023-10-30T14:10:03.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:03.305+0000 7f77937d3640 1 -- 172.21.15.46:0/1360569639 wait complete. 2023-10-30T14:10:03.327 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 204, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 204, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8655": { "addr": "172.21.15.46:6837/3044281789", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3044281789, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3044281789, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8655, "incarnation": 196, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_8826": { "addr": "172.21.15.46:6836/66265765", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 66265765, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 66265765, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8826, "incarnation": 203, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:resolve", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:09:59.299411+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8826, "mds_1": 8655 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:10:05.328 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:10:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.477+0000 7f490ddd3640 1 Processor -- start 2023-10-30T14:10:05.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.477+0000 7f490ddd3640 1 -- start start 2023-10-30T14:10:05.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.477+0000 7f490ddd3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f49081538a0 0x7f490815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:05.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.477+0000 7f490ddd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49080ca0e0 0x7f490815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:05.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.477+0000 7f490ddd3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49080cb800 con 0x7f49080cf160 2023-10-30T14:10:05.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.477+0000 7f490ddd3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f490815dc30 con 0x7f49081538a0 2023-10-30T14:10:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.477+0000 7f490ddd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49080cb330 con 0x7f49080ca0e0 2023-10-30T14:10:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.477+0000 7f4906ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49080ca0e0 0x7f490815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.477+0000 7f49077fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f49081538a0 0x7f490815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.477+0000 7f49077fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f49081538a0 0x7f490815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:50278/0 (socket says 172.21.15.46:50278) 2023-10-30T14:10:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.477+0000 7f49077fe640 1 -- 172.21.15.46:0/763988664 learned_addr learned my addr 172.21.15.46:0/763988664 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.477+0000 7f4906ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49080ca0e0 0x7f490815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:40558/0 (socket says 172.21.15.46:40558) 2023-10-30T14:10:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.481+0000 7f49077fe640 1 -- 172.21.15.46:0/763988664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49080ca0e0 msgr2=0x7f490815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.481+0000 7f49077fe640 1 --2- 172.21.15.46:0/763988664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49080ca0e0 0x7f490815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.481+0000 7f49077fe640 1 -- 172.21.15.46:0/763988664 >> v1:172.21.15.154:6789/0 conn(0x7f49080cf160 legacy=0x7f49080cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.481+0000 7f49077fe640 1 -- 172.21.15.46:0/763988664 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49080cad70 con 0x7f49081538a0 2023-10-30T14:10:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.481+0000 7f49067fc640 1 -- 172.21.15.46:0/763988664 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800995973 0 0) 0x7f49080cb800 con 0x7f49080cf160 2023-10-30T14:10:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.481+0000 7f4906ffd640 1 --2- 172.21.15.46:0/763988664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49080ca0e0 0x7f490815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:10:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.481+0000 7f49077fe640 1 --2- 172.21.15.46:0/763988664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49081538a0 0x7f490815d6f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009870 tx=0x7f48f801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8f781423eb381b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.481+0000 7f49067fc640 1 -- 172.21.15.46:0/763988664 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f80a8a30 con 0x7f49081538a0 2023-10-30T14:10:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.481+0000 7f49067fc640 1 -- 172.21.15.46:0/763988664 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f80a8b90 con 0x7f49081538a0 2023-10-30T14:10:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.481+0000 7f49067fc640 1 -- 172.21.15.46:0/763988664 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f80b0440 con 0x7f49081538a0 2023-10-30T14:10:05.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 -- 172.21.15.46:0/763988664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49081538a0 msgr2=0x7f490815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:05.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 --2- 172.21.15.46:0/763988664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49081538a0 0x7f490815d6f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009870 tx=0x7f48f801ed30 comp rx=0 tx=0).stop 2023-10-30T14:10:05.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 -- 172.21.15.46:0/763988664 shutdown_connections 2023-10-30T14:10:05.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 --2- 172.21.15.46:0/763988664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49080ca0e0 0x7f490815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:05.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 --2- 172.21.15.46:0/763988664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49081538a0 0x7f490815d6f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:05.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 -- 172.21.15.46:0/763988664 >> 172.21.15.46:0/763988664 conn(0x7f490806bfe0 msgr2=0x7f490806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:05.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 -- 172.21.15.46:0/763988664 shutdown_connections 2023-10-30T14:10:05.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 -- 172.21.15.46:0/763988664 wait complete. 2023-10-30T14:10:05.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 Processor -- start 2023-10-30T14:10:05.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 -- start start 2023-10-30T14:10:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49080ca0e0 0x7f4908148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49080cf160 0x7f49081e33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49081538a0 0x7f49081e3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49081f02f0 con 0x7f49080cf160 2023-10-30T14:10:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4908149bd0 con 0x7f49080ca0e0 2023-10-30T14:10:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f490ddd3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49081ef400 con 0x7f49081538a0 2023-10-30T14:10:05.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f4906ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49080cf160 0x7f49081e33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:05.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f4906ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49080cf160 0x7f49081e33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36074/0 (socket says 172.21.15.46:36074) 2023-10-30T14:10:05.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f4906ffd640 1 -- 172.21.15.46:0/174893331 learned_addr learned my addr 172.21.15.46:0/174893331 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:05.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f49077fe640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49080ca0e0 0x7f4908148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:05.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.485+0000 7f4907fff640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49081538a0 0x7f49081e3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f4906ffd640 1 -- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49081538a0 msgr2=0x7f49081e3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f4906ffd640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49081538a0 0x7f49081e3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f4906ffd640 1 -- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49080ca0e0 msgr2=0x7f4908148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f4906ffd640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49080ca0e0 0x7f4908148f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f4906ffd640 1 -- 172.21.15.46:0/174893331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc009420 con 0x7f49080cf160 2023-10-30T14:10:05.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f4906ffd640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49080cf160 0x7f49081e33f0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00e980 tx=0x7f48fc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:05.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f48effff640 1 -- 172.21.15.46:0/174893331 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc00ca00 con 0x7f49080cf160 2023-10-30T14:10:05.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f48effff640 1 -- 172.21.15.46:0/174893331 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48fc00cb60 con 0x7f49080cf160 2023-10-30T14:10:05.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f48effff640 1 -- 172.21.15.46:0/174893331 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc013470 con 0x7f49080cf160 2023-10-30T14:10:05.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f490ddd3640 1 -- 172.21.15.46:0/174893331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f80094d0 con 0x7f49080cf160 2023-10-30T14:10:05.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f490ddd3640 1 -- 172.21.15.46:0/174893331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49081ef9c0 con 0x7f49080cf160 2023-10-30T14:10:05.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f490ddd3640 1 -- 172.21.15.46:0/174893331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49080b2a50 con 0x7f49080cf160 2023-10-30T14:10:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f48effff640 1 -- 172.21.15.46:0/174893331 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f48fc0135d0 con 0x7f49080cf160 2023-10-30T14:10:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f48effff640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48f40664d0 0x7f48f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.489+0000 7f48effff640 1 -- 172.21.15.46:0/174893331 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f48fc084990 con 0x7f49080cf160 2023-10-30T14:10:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.493+0000 7f49077fe640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48f40664d0 0x7f48f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.493+0000 7f49077fe640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48f40664d0 0x7f48f4068990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f48f80b2000 tx=0x7f48f80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:05.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.493+0000 7f48effff640 1 -- 172.21.15.46:0/174893331 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f48fc051520 con 0x7f49080cf160 2023-10-30T14:10:05.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.685+0000 7f490ddd3640 1 -- 172.21.15.46:0/174893331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f490804e170 con 0x7f49080cf160 2023-10-30T14:10:05.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.685+0000 7f48effff640 1 -- 172.21.15.46:0/174893331 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+5728 (secure 0 0 0) 0x7f48fc017090 con 0x7f49080cf160 2023-10-30T14:10:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:10:05.690 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:09:59.299411+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:resolve","state_seq":11,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:10:05.690 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 204 2023-10-30T14:10:05.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.689+0000 7f490ddd3640 1 -- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48f40664d0 msgr2=0x7f48f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:05.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.689+0000 7f490ddd3640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48f40664d0 0x7f48f4068990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f48f80b2000 tx=0x7f48f80aa070 comp rx=0 tx=0).stop 2023-10-30T14:10:05.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.689+0000 7f490ddd3640 1 -- 172.21.15.46:0/174893331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49080cf160 msgr2=0x7f49081e33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:05.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.689+0000 7f490ddd3640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49080cf160 0x7f49081e33f0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00e980 tx=0x7f48fc00ee40 comp rx=0 tx=0).stop 2023-10-30T14:10:05.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.689+0000 7f490ddd3640 1 -- 172.21.15.46:0/174893331 shutdown_connections 2023-10-30T14:10:05.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.689+0000 7f490ddd3640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f48f40664d0 0x7f48f4068990 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:05.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.689+0000 7f490ddd3640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49081538a0 0x7f49081e3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:05.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.689+0000 7f490ddd3640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49080cf160 0x7f49081e33f0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:05.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.689+0000 7f490ddd3640 1 --2- 172.21.15.46:0/174893331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49080ca0e0 0x7f4908148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:05.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.689+0000 7f490ddd3640 1 -- 172.21.15.46:0/174893331 >> 172.21.15.46:0/174893331 conn(0x7f490806bfe0 msgr2=0x7f4908065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:05.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.689+0000 7f490ddd3640 1 -- 172.21.15.46:0/174893331 shutdown_connections 2023-10-30T14:10:05.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:05.689+0000 7f490ddd3640 1 -- 172.21.15.46:0/174893331 wait complete. 2023-10-30T14:10:05.711 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:10:07.712 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:10:07.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03f2a54640 1 Processor -- start 2023-10-30T14:10:07.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03f2a54640 1 -- start start 2023-10-30T14:10:07.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03f2a54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03ec0cf160 0x7f03ec0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:07.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03f2a54640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ec0cb800 con 0x7f03ec1538a0 2023-10-30T14:10:07.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03f2a54640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03ec0cb330 con 0x7f03ec15d6f0 2023-10-30T14:10:07.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03f2a54640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec0cad70 con 0x7f03ec0cf160 2023-10-30T14:10:07.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03ebfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f03ec1538a0 0x7f03ec0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44984/0 (socket says 172.21.15.46:44984) 2023-10-30T14:10:07.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03ebfff640 1 -- 172.21.15.46:0/3318899204 learned_addr learned my addr 172.21.15.46:0/3318899204 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:10:07.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03f0ff1640 1 --2- 172.21.15.46:0/3318899204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03ec0cf160 0x7f03ec0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:07.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834261244 0 0) 0x7f03ec0cb800 con 0x7f03ec1538a0 2023-10-30T14:10:07.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d4003530 con 0x7f03ec1538a0 2023-10-30T14:10:07.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1967201691 0 0) 0x7f03d4003530 con 0x7f03ec1538a0 2023-10-30T14:10:07.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03ec0cf160 msgr2=0x7f03ec0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:07.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03eaffd640 1 --2- 172.21.15.46:0/3318899204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03ec0cf160 0x7f03ec0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:07.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 >> v1:172.21.15.154:6789/0 conn(0x7f03ec15d6f0 legacy=0x7f03ec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:07.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec0cb050 con 0x7f03ec1538a0 2023-10-30T14:10:07.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f03dc001680 con 0x7f03ec1538a0 2023-10-30T14:10:07.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 >> v1:172.21.15.46:6789/0 conn(0x7f03ec1538a0 legacy=0x7f03ec0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:07.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.865+0000 7f03eaffd640 1 --2- 172.21.15.46:0/3318899204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03ec0cf160 0x7f03d40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:07.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eaffd640 1 --2- 172.21.15.46:0/3318899204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d4003ef0 0x7f03d40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:07.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eaffd640 1 --2- 172.21.15.46:0/3318899204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03d4004d10 0x7f03d4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:07.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec0cad70 con 0x7f03d4003ef0 2023-10-30T14:10:07.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d400b9f0 con 0x7f03d4004d10 2023-10-30T14:10:07.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d400bbb0 con 0x7f03ec0cf160 2023-10-30T14:10:07.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eb7fe640 1 --2- 172.21.15.46:0/3318899204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d4003ef0 0x7f03d40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:07.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03ebfff640 1 --2- 172.21.15.46:0/3318899204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03ec0cf160 0x7f03d40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:07.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03f0ff1640 1 --2- 172.21.15.46:0/3318899204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03d4004d10 0x7f03d4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:07.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eb7fe640 1 -- 172.21.15.46:0/3318899204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03ec0cf160 msgr2=0x7f03d40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:07.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eb7fe640 1 --2- 172.21.15.46:0/3318899204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03ec0cf160 0x7f03d40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:07.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eb7fe640 1 -- 172.21.15.46:0/3318899204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03d4004d10 msgr2=0x7f03d4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:07.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eb7fe640 1 --2- 172.21.15.46:0/3318899204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03d4004d10 0x7f03d4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:07.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eb7fe640 1 -- 172.21.15.46:0/3318899204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d4003530 con 0x7f03d4003ef0 2023-10-30T14:10:07.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eb7fe640 1 --2- 172.21.15.46:0/3318899204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d4003ef0 0x7f03d40047d0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f03e0001860 tx=0x7f03e001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca31b3ff617d263f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:07.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e00ab600 con 0x7f03d4003ef0 2023-10-30T14:10:07.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e00a7050 con 0x7f03d4003ef0 2023-10-30T14:10:07.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.869+0000 7f03eaffd640 1 -- 172.21.15.46:0/3318899204 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e00ab9e0 con 0x7f03d4003ef0 2023-10-30T14:10:07.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 -- 172.21.15.46:0/3318899204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d4003ef0 msgr2=0x7f03d40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:07.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 --2- 172.21.15.46:0/3318899204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d4003ef0 0x7f03d40047d0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f03e0001860 tx=0x7f03e001c420 comp rx=0 tx=0).stop 2023-10-30T14:10:07.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f0ff1640 1 -- 172.21.15.46:0/3318899204 reap_dead start 2023-10-30T14:10:07.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 -- 172.21.15.46:0/3318899204 shutdown_connections 2023-10-30T14:10:07.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 -- 172.21.15.46:0/3318899204 >> 172.21.15.46:0/3318899204 conn(0x7f03ec06bfe0 msgr2=0x7f03ec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:07.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 -- 172.21.15.46:0/3318899204 shutdown_connections 2023-10-30T14:10:07.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 -- 172.21.15.46:0/3318899204 wait complete. 2023-10-30T14:10:07.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 Processor -- start 2023-10-30T14:10:07.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 -- start start 2023-10-30T14:10:07.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03d4003ef0 0x7f03ec15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:07.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03ec14a5d0 0x7f03ec0cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:07.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03ec0cda10 0x7f03ec1f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:07.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec14ad30 con 0x7f03ec0cda10 2023-10-30T14:10:07.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec0cd040 con 0x7f03ec14a5d0 2023-10-30T14:10:07.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f2a54640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec157600 con 0x7f03d4003ef0 2023-10-30T14:10:07.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f0ff1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03ec0cda10 0x7f03ec1f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:07.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f0ff1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03ec0cda10 0x7f03ec1f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52804/0 (socket says 172.21.15.46:52804) 2023-10-30T14:10:07.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f0ff1640 1 -- 172.21.15.46:0/1846940448 learned_addr learned my addr 172.21.15.46:0/1846940448 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:07.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03ebfff640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03d4003ef0 0x7f03ec15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:07.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03eb7fe640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03ec14a5d0 0x7f03ec0cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:07.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f0ff1640 1 -- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03d4003ef0 msgr2=0x7f03ec15bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:07.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f0ff1640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03d4003ef0 0x7f03ec15bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f0ff1640 1 -- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03ec14a5d0 msgr2=0x7f03ec0cd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.873+0000 7f03f0ff1640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03ec14a5d0 0x7f03ec0cd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.877+0000 7f03f0ff1640 1 -- 172.21.15.46:0/1846940448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec0cde90 con 0x7f03ec0cda10 2023-10-30T14:10:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.877+0000 7f03f0ff1640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03ec0cda10 0x7f03ec1f22b0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f03e4011b40 tx=0x7f03e400f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:07.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.877+0000 7f03e8ff9640 1 -- 172.21.15.46:0/1846940448 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e400e040 con 0x7f03ec0cda10 2023-10-30T14:10:07.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.877+0000 7f03e8ff9640 1 -- 172.21.15.46:0/1846940448 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e400f040 con 0x7f03ec0cda10 2023-10-30T14:10:07.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.877+0000 7f03e8ff9640 1 -- 172.21.15.46:0/1846940448 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4015430 con 0x7f03ec0cda10 2023-10-30T14:10:07.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.877+0000 7f03f2a54640 1 -- 172.21.15.46:0/1846940448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ec157870 con 0x7f03ec0cda10 2023-10-30T14:10:07.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.877+0000 7f03f2a54640 1 -- 172.21.15.46:0/1846940448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ec157e30 con 0x7f03ec0cda10 2023-10-30T14:10:07.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.877+0000 7f03f2a54640 1 -- 172.21.15.46:0/1846940448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03ec04e170 con 0x7f03ec0cda10 2023-10-30T14:10:07.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.877+0000 7f03e8ff9640 1 -- 172.21.15.46:0/1846940448 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f03e4015590 con 0x7f03ec0cda10 2023-10-30T14:10:07.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.881+0000 7f03e8ff9640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03d80664d0 0x7f03d8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:07.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.881+0000 7f03e8ff9640 1 -- 172.21.15.46:0/1846940448 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f03e4084e80 con 0x7f03ec0cda10 2023-10-30T14:10:07.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.881+0000 7f03ebfff640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03d80664d0 0x7f03d8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:07.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.881+0000 7f03ebfff640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03d80664d0 0x7f03d8068990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f03dc001cc0 tx=0x7f03dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:07.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:07.885+0000 7f03e8ff9640 1 -- 172.21.15.46:0/1846940448 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f03e4051970 con 0x7f03ec0cda10 2023-10-30T14:10:08.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.073+0000 7f03f2a54640 1 -- 172.21.15.46:0/1846940448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03ec0cd180 con 0x7f03ec0cda10 2023-10-30T14:10:08.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.073+0000 7f03e8ff9640 1 -- 172.21.15.46:0/1846940448 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f03e4054f90 con 0x7f03ec0cda10 2023-10-30T14:10:08.078 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:10:08.078 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:05.008334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:10:08.079 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 205 2023-10-30T14:10:08.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.077+0000 7f03f2a54640 1 -- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03d80664d0 msgr2=0x7f03d8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.077+0000 7f03f2a54640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03d80664d0 0x7f03d8068990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f03dc001cc0 tx=0x7f03dc009040 comp rx=0 tx=0).stop 2023-10-30T14:10:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.077+0000 7f03f2a54640 1 -- 172.21.15.46:0/1846940448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03ec0cda10 msgr2=0x7f03ec1f22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.077+0000 7f03f2a54640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03ec0cda10 0x7f03ec1f22b0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f03e4011b40 tx=0x7f03e400f1b0 comp rx=0 tx=0).stop 2023-10-30T14:10:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.077+0000 7f03f2a54640 1 -- 172.21.15.46:0/1846940448 shutdown_connections 2023-10-30T14:10:08.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.077+0000 7f03f2a54640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03d80664d0 0x7f03d8068990 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:08.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.077+0000 7f03f2a54640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03ec0cda10 0x7f03ec1f22b0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:08.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.077+0000 7f03f2a54640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03ec14a5d0 0x7f03ec0cd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:08.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.077+0000 7f03f2a54640 1 --2- 172.21.15.46:0/1846940448 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03d4003ef0 0x7f03ec15bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:08.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.077+0000 7f03f2a54640 1 -- 172.21.15.46:0/1846940448 >> 172.21.15.46:0/1846940448 conn(0x7f03ec06bfe0 msgr2=0x7f03ec0ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:08.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.077+0000 7f03f2a54640 1 -- 172.21.15.46:0/1846940448 shutdown_connections 2023-10-30T14:10:08.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:08.077+0000 7f03f2a54640 1 -- 172.21.15.46:0/1846940448 wait complete. 2023-10-30T14:10:08.099 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:10:10.100 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:10:10.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.249+0000 7ff571f3e640 1 Processor -- start 2023-10-30T14:10:10.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff571f3e640 1 -- start start 2023-10-30T14:10:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff571f3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff56c06f7a0 0x7ff56c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff571f3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff56c0c6bd0 0x7ff56c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff571f3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff56c071110 0x7ff56c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff571f3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56c0700c0 con 0x7ff56c071110 2023-10-30T14:10:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff571f3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56c06a3d0 con 0x7ff56c0c6bd0 2023-10-30T14:10:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff571f3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c066cf0 con 0x7ff56c06f7a0 2023-10-30T14:10:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff56bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff56c071110 0x7ff56c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff56bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff56c071110 0x7ff56c158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52818/0 (socket says 172.21.15.46:52818) 2023-10-30T14:10:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff56bfff640 1 -- 172.21.15.46:0/2039992891 learned_addr learned my addr 172.21.15.46:0/2039992891 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff56affd640 1 --2- 172.21.15.46:0/2039992891 >> v2:172.21.15.154:3300/0 conn(0x7ff56c0c6bd0 0x7ff56c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff56b7fe640 1 --2- 172.21.15.46:0/2039992891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff56c06f7a0 0x7ff56c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff56bfff640 1 -- 172.21.15.46:0/2039992891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff56c06f7a0 msgr2=0x7ff56c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff56bfff640 1 --2- 172.21.15.46:0/2039992891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff56c06f7a0 0x7ff56c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff56bfff640 1 -- 172.21.15.46:0/2039992891 >> v2:172.21.15.154:3300/0 conn(0x7ff56c0c6bd0 msgr2=0x7ff56c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:10.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff56bfff640 1 --2- 172.21.15.46:0/2039992891 >> v2:172.21.15.154:3300/0 conn(0x7ff56c0c6bd0 0x7ff56c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.253+0000 7ff56bfff640 1 -- 172.21.15.46:0/2039992891 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c0c70d0 con 0x7ff56c071110 2023-10-30T14:10:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff56bfff640 1 --2- 172.21.15.46:0/2039992891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c071110 0x7ff56c158110 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0b1210 tx=0x7ff56401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11d44090c48f80e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff56a7fc640 1 -- 172.21.15.46:0/2039992891 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5640af400 con 0x7ff56c071110 2023-10-30T14:10:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff56a7fc640 1 -- 172.21.15.46:0/2039992891 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5640ab050 con 0x7ff56c071110 2023-10-30T14:10:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff56a7fc640 1 -- 172.21.15.46:0/2039992891 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5640af780 con 0x7ff56c071110 2023-10-30T14:10:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff571f3e640 1 -- 172.21.15.46:0/2039992891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c071110 msgr2=0x7ff56c158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff571f3e640 1 --2- 172.21.15.46:0/2039992891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c071110 0x7ff56c158110 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0b1210 tx=0x7ff56401f1c0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff571f3e640 1 -- 172.21.15.46:0/2039992891 shutdown_connections 2023-10-30T14:10:10.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff571f3e640 1 --2- 172.21.15.46:0/2039992891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c071110 0x7ff56c158110 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff571f3e640 1 --2- 172.21.15.46:0/2039992891 >> v2:172.21.15.154:3300/0 conn(0x7ff56c0c6bd0 0x7ff56c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff571f3e640 1 --2- 172.21.15.46:0/2039992891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff56c06f7a0 0x7ff56c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff571f3e640 1 -- 172.21.15.46:0/2039992891 >> 172.21.15.46:0/2039992891 conn(0x7ff56c0c7910 msgr2=0x7ff56c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:10.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff571f3e640 1 -- 172.21.15.46:0/2039992891 shutdown_connections 2023-10-30T14:10:10.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff571f3e640 1 -- 172.21.15.46:0/2039992891 wait complete. 2023-10-30T14:10:10.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff571f3e640 1 Processor -- start 2023-10-30T14:10:10.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.257+0000 7ff571f3e640 1 -- start start 2023-10-30T14:10:10.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff571f3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c06f7a0 0x7ff56c15af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:10.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff571f3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff56c071110 0x7ff56c0c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:10.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff571f3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff56c0c6bd0 0x7ff56c159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:10.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff571f3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c15b490 con 0x7ff56c06f7a0 2023-10-30T14:10:10.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff571f3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0c0f30 con 0x7ff56c071110 2023-10-30T14:10:10.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff571f3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0c0330 con 0x7ff56c0c6bd0 2023-10-30T14:10:10.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff56b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c06f7a0 0x7ff56c15af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:10.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff56b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c06f7a0 0x7ff56c15af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52820/0 (socket says 172.21.15.46:52820) 2023-10-30T14:10:10.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff56b7fe640 1 -- 172.21.15.46:0/838366985 learned_addr learned my addr 172.21.15.46:0/838366985 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:10.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff56affd640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff56c071110 0x7ff56c0c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:10.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff56bfff640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff56c0c6bd0 0x7ff56c159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:10.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff56b7fe640 1 -- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff56c0c6bd0 msgr2=0x7ff56c159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:10.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff56b7fe640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff56c0c6bd0 0x7ff56c159570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff56b7fe640 1 -- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff56c071110 msgr2=0x7ff56c0c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:10.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff56b7fe640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff56c071110 0x7ff56c0c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff56b7fe640 1 -- 172.21.15.46:0/838366985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff564009b30 con 0x7ff56c06f7a0 2023-10-30T14:10:10.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff56bfff640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff56c0c6bd0 0x7ff56c159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:10:10.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff56b7fe640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c06f7a0 0x7ff56c15af50 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff55c00d830 tx=0x7ff55c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:10.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff553fff640 1 -- 172.21.15.46:0/838366985 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c00be10 con 0x7ff56c06f7a0 2023-10-30T14:10:10.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff553fff640 1 -- 172.21.15.46:0/838366985 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55c0098b0 con 0x7ff56c06f7a0 2023-10-30T14:10:10.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff553fff640 1 -- 172.21.15.46:0/838366985 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c009a40 con 0x7ff56c06f7a0 2023-10-30T14:10:10.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff571f3e640 1 -- 172.21.15.46:0/838366985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c0c05d0 con 0x7ff56c06f7a0 2023-10-30T14:10:10.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff571f3e640 1 -- 172.21.15.46:0/838366985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c0c2090 con 0x7ff56c06f7a0 2023-10-30T14:10:10.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.261+0000 7ff571f3e640 1 -- 172.21.15.46:0/838366985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56c04e170 con 0x7ff56c06f7a0 2023-10-30T14:10:10.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.265+0000 7ff553fff640 1 -- 172.21.15.46:0/838366985 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff55c009ba0 con 0x7ff56c06f7a0 2023-10-30T14:10:10.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.265+0000 7ff553fff640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5580664a0 0x7ff558068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:10.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.265+0000 7ff553fff640 1 -- 172.21.15.46:0/838366985 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7ff55c084b10 con 0x7ff56c06f7a0 2023-10-30T14:10:10.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.265+0000 7ff56affd640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5580664a0 0x7ff558068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:10.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.265+0000 7ff56affd640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5580664a0 0x7ff558068960 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0c02c0 tx=0x7ff560008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:10.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.269+0000 7ff553fff640 1 -- 172.21.15.46:0/838366985 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff55c0515f0 con 0x7ff56c06f7a0 2023-10-30T14:10:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 -- 172.21.15.46:0/838366985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff56c073e10 con 0x7ff56c06f7a0 2023-10-30T14:10:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff553fff640 1 -- 172.21.15.46:0/838366985 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5730 (secure 0 0 0) 0x7ff55c018090 con 0x7ff56c06f7a0 2023-10-30T14:10:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:10:10.465 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:05.008334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:10:10.466 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 205 2023-10-30T14:10:10.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 -- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5580664a0 msgr2=0x7ff558068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:10.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5580664a0 0x7ff558068960 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0c02c0 tx=0x7ff560008040 comp rx=0 tx=0).stop 2023-10-30T14:10:10.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 -- 172.21.15.46:0/838366985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c06f7a0 msgr2=0x7ff56c15af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c06f7a0 0x7ff56c15af50 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff55c00d830 tx=0x7ff55c00dcf0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 -- 172.21.15.46:0/838366985 shutdown_connections 2023-10-30T14:10:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5580664a0 0x7ff558068960 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff56c0c6bd0 0x7ff56c159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff56c071110 0x7ff56c0c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 --2- 172.21.15.46:0/838366985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff56c06f7a0 0x7ff56c15af50 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 -- 172.21.15.46:0/838366985 >> 172.21.15.46:0/838366985 conn(0x7ff56c0c7910 msgr2=0x7ff56c1562a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 -- 172.21.15.46:0/838366985 shutdown_connections 2023-10-30T14:10:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:10.461+0000 7ff571f3e640 1 -- 172.21.15.46:0/838366985 wait complete. 2023-10-30T14:10:10.487 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:10:12.488 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:10:12.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b603f8640 1 Processor -- start 2023-10-30T14:10:12.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b603f8640 1 -- start start 2023-10-30T14:10:12.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b603f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b581538a0 0x7f1b5815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:12.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b603f8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1b580ca0e0 0x7f1b5815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:12.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b603f8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1b580cf160 0x7f1b580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:12.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b603f8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b580cb800 con 0x7f1b580ca0e0 2023-10-30T14:10:12.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b603f8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b5815dc30 con 0x7f1b580cf160 2023-10-30T14:10:12.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b603f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580cb330 con 0x7f1b581538a0 2023-10-30T14:10:12.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b5d993640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1b580ca0e0 0x7f1b5815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:12.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b5d993640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1b580ca0e0 0x7f1b5815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52830/0 (socket says 172.21.15.46:52830) 2023-10-30T14:10:12.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b5d993640 1 -- 172.21.15.46:0/1738877528 learned_addr learned my addr 172.21.15.46:0/1738877528 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:12.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b5e194640 1 --2- 172.21.15.46:0/1738877528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b581538a0 0x7f1b5815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:12.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b5e995640 1 --2- 172.21.15.46:0/1738877528 >> v2:172.21.15.154:3300/0 conn(0x7f1b580cf160 0x7f1b580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:12.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b5d993640 1 -- 172.21.15.46:0/1738877528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b581538a0 msgr2=0x7f1b5815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:12.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b5d993640 1 --2- 172.21.15.46:0/1738877528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b581538a0 0x7f1b5815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b5d993640 1 -- 172.21.15.46:0/1738877528 >> v2:172.21.15.154:3300/0 conn(0x7f1b580cf160 msgr2=0x7f1b580cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:12.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b5d993640 1 --2- 172.21.15.46:0/1738877528 >> v2:172.21.15.154:3300/0 conn(0x7f1b580cf160 0x7f1b580cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.641+0000 7f1b5d993640 1 -- 172.21.15.46:0/1738877528 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b580cad70 con 0x7f1b580ca0e0 2023-10-30T14:10:12.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b5d993640 1 --2- 172.21.15.46:0/1738877528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b580ca0e0 0x7f1b5815a090 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f1b5806cb00 tx=0x7f1b4801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f470011478f19a4b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:12.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b5d192640 1 -- 172.21.15.46:0/1738877528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b480a89b0 con 0x7f1b580ca0e0 2023-10-30T14:10:12.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b5d192640 1 -- 172.21.15.46:0/1738877528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b480a8b10 con 0x7f1b580ca0e0 2023-10-30T14:10:12.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b5d192640 1 -- 172.21.15.46:0/1738877528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b480a8ea0 con 0x7f1b580ca0e0 2023-10-30T14:10:12.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b603f8640 1 -- 172.21.15.46:0/1738877528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b580ca0e0 msgr2=0x7f1b5815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:12.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b603f8640 1 --2- 172.21.15.46:0/1738877528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b580ca0e0 0x7f1b5815a090 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f1b5806cb00 tx=0x7f1b4801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b603f8640 1 -- 172.21.15.46:0/1738877528 shutdown_connections 2023-10-30T14:10:12.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b603f8640 1 --2- 172.21.15.46:0/1738877528 >> v2:172.21.15.154:3300/0 conn(0x7f1b580cf160 0x7f1b580cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b603f8640 1 --2- 172.21.15.46:0/1738877528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b580ca0e0 0x7f1b5815a090 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b603f8640 1 --2- 172.21.15.46:0/1738877528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b581538a0 0x7f1b5815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b603f8640 1 -- 172.21.15.46:0/1738877528 >> 172.21.15.46:0/1738877528 conn(0x7f1b5806bfe0 msgr2=0x7f1b5806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:12.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b603f8640 1 -- 172.21.15.46:0/1738877528 shutdown_connections 2023-10-30T14:10:12.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.645+0000 7f1b603f8640 1 -- 172.21.15.46:0/1738877528 wait complete. 2023-10-30T14:10:12.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b603f8640 1 Processor -- start 2023-10-30T14:10:12.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b603f8640 1 -- start start 2023-10-30T14:10:12.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b603f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b580ca0e0 0x7f1b5815d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:12.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b603f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b580cf160 0x7f1b5814a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:12.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b603f8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b581538a0 0x7f1b581f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:12.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b603f8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b581f2fd0 con 0x7f1b581538a0 2023-10-30T14:10:12.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b603f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5815ad70 con 0x7f1b580cf160 2023-10-30T14:10:12.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b603f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5814bab0 con 0x7f1b580ca0e0 2023-10-30T14:10:12.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b5e995640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b581538a0 0x7f1b581f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:12.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b5e995640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b581538a0 0x7f1b581f33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52844/0 (socket says 172.21.15.46:52844) 2023-10-30T14:10:12.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b5e995640 1 -- 172.21.15.46:0/576622947 learned_addr learned my addr 172.21.15.46:0/576622947 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:12.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b5e194640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b580ca0e0 0x7f1b5815d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b5d993640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b580cf160 0x7f1b5814a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b5e995640 1 -- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b580ca0e0 msgr2=0x7f1b5815d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b5e995640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b580ca0e0 0x7f1b5815d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b5e995640 1 -- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b580cf160 msgr2=0x7f1b5814a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b5e995640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b580cf160 0x7f1b5814a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b5e995640 1 -- 172.21.15.46:0/576622947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b480094d0 con 0x7f1b581538a0 2023-10-30T14:10:12.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b5d993640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b580cf160 0x7f1b5814a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:10:12.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b5e995640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b581538a0 0x7f1b581f33f0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f1b50007d40 tx=0x7f1b5000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:12.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b4effd640 1 -- 172.21.15.46:0/576622947 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50014420 con 0x7f1b581538a0 2023-10-30T14:10:12.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b4effd640 1 -- 172.21.15.46:0/576622947 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b500145a0 con 0x7f1b581538a0 2023-10-30T14:10:12.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b603f8640 1 -- 172.21.15.46:0/576622947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b580ce130 con 0x7f1b581538a0 2023-10-30T14:10:12.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b4effd640 1 -- 172.21.15.46:0/576622947 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50014950 con 0x7f1b581538a0 2023-10-30T14:10:12.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.649+0000 7f1b603f8640 1 -- 172.21.15.46:0/576622947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b580ce380 con 0x7f1b581538a0 2023-10-30T14:10:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.653+0000 7f1b603f8640 1 -- 172.21.15.46:0/576622947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b580b2a50 con 0x7f1b581538a0 2023-10-30T14:10:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.653+0000 7f1b4effd640 1 -- 172.21.15.46:0/576622947 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1b50014ab0 con 0x7f1b581538a0 2023-10-30T14:10:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.653+0000 7f1b4effd640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1b3c0664d0 0x7f1b3c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.653+0000 7f1b4effd640 1 -- 172.21.15.46:0/576622947 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f1b50085100 con 0x7f1b581538a0 2023-10-30T14:10:12.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.653+0000 7f1b5e194640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1b3c0664d0 0x7f1b3c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:12.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.653+0000 7f1b5e194640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1b3c0664d0 0x7f1b3c068990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f1b5400b930 tx=0x7f1b54008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:12.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.657+0000 7f1b4effd640 1 -- 172.21.15.46:0/576622947 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1b50051b50 con 0x7f1b581538a0 2023-10-30T14:10:12.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 -- 172.21.15.46:0/576622947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b581f3110 con 0x7f1b581538a0 2023-10-30T14:10:12.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b4effd640 1 -- 172.21.15.46:0/576622947 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f1b581f3110 con 0x7f1b581538a0 2023-10-30T14:10:12.852 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:10:12.853 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":206,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:10.944299+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:10:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 206 2023-10-30T14:10:12.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 -- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1b3c0664d0 msgr2=0x7f1b3c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:12.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1b3c0664d0 0x7f1b3c068990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f1b5400b930 tx=0x7f1b54008040 comp rx=0 tx=0).stop 2023-10-30T14:10:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 -- 172.21.15.46:0/576622947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b581538a0 msgr2=0x7f1b581f33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b581538a0 0x7f1b581f33f0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f1b50007d40 tx=0x7f1b5000e4b0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 -- 172.21.15.46:0/576622947 shutdown_connections 2023-10-30T14:10:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1b3c0664d0 0x7f1b3c068990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b581538a0 0x7f1b581f33f0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b580cf160 0x7f1b5814a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 --2- 172.21.15.46:0/576622947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b580ca0e0 0x7f1b5815d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 -- 172.21.15.46:0/576622947 >> 172.21.15.46:0/576622947 conn(0x7f1b5806bfe0 msgr2=0x7f1b581483a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 -- 172.21.15.46:0/576622947 shutdown_connections 2023-10-30T14:10:12.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:12.849+0000 7f1b603f8640 1 -- 172.21.15.46:0/576622947 wait complete. 2023-10-30T14:10:12.874 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:10:14.874 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:10:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f332ea640 1 Processor -- start 2023-10-30T14:10:15.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f332ea640 1 -- start start 2023-10-30T14:10:15.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f332ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c15d6f0 0x7f3f2c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:15.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f332ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2c0cb800 con 0x7f3f2c1538a0 2023-10-30T14:10:15.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f332ea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2c0cb330 con 0x7f3f2c0cf160 2023-10-30T14:10:15.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f332ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c0cad70 con 0x7f3f2c15d6f0 2023-10-30T14:10:15.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f31086640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3f2c1538a0 0x7f3f2c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44986/0 (socket says 172.21.15.46:44986) 2023-10-30T14:10:15.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f31086640 1 -- 172.21.15.46:0/1218357184 learned_addr learned my addr 172.21.15.46:0/1218357184 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:10:15.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f30885640 1 --2- 172.21.15.46:0/1218357184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c15d6f0 0x7f3f2c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:15.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485780143 0 0) 0x7f3f2c0cb800 con 0x7f3f2c1538a0 2023-10-30T14:10:15.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f18003530 con 0x7f3f2c1538a0 2023-10-30T14:10:15.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1942113979 0 0) 0x7f3f18003530 con 0x7f3f2c1538a0 2023-10-30T14:10:15.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c15d6f0 msgr2=0x7f3f2c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:15.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 --2- 172.21.15.46:0/1218357184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c15d6f0 0x7f3f2c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:15.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 >> v1:172.21.15.154:6789/0 conn(0x7f3f2c0cf160 legacy=0x7f3f2c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:15.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c0cb050 con 0x7f3f2c1538a0 2023-10-30T14:10:15.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f3f28001680 con 0x7f3f2c1538a0 2023-10-30T14:10:15.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 >> v1:172.21.15.46:6789/0 conn(0x7f3f2c1538a0 legacy=0x7f3f2c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:15.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 --2- 172.21.15.46:0/1218357184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c15d6f0 0x7f3f180039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:15.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 --2- 172.21.15.46:0/1218357184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f18003f30 0x7f3f180047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:15.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 --2- 172.21.15.46:0/1218357184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f18004d10 0x7f3f18005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:15.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c0cad70 con 0x7f3f18004d10 2023-10-30T14:10:15.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1800bb00 con 0x7f3f18003f30 2023-10-30T14:10:15.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1800bcc0 con 0x7f3f2c15d6f0 2023-10-30T14:10:15.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f31086640 1 --2- 172.21.15.46:0/1218357184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c15d6f0 0x7f3f180039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:15.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.025+0000 7f3f31887640 1 --2- 172.21.15.46:0/1218357184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f18004d10 0x7f3f18005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:15.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f30885640 1 --2- 172.21.15.46:0/1218357184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f18003f30 0x7f3f180047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:15.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f31887640 1 -- 172.21.15.46:0/1218357184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c15d6f0 msgr2=0x7f3f180039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:15.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f31887640 1 --2- 172.21.15.46:0/1218357184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c15d6f0 0x7f3f180039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:15.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f31887640 1 -- 172.21.15.46:0/1218357184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f18003f30 msgr2=0x7f3f180047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:15.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f31887640 1 --2- 172.21.15.46:0/1218357184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f18003f30 0x7f3f180047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:15.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f31887640 1 -- 172.21.15.46:0/1218357184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f18003530 con 0x7f3f18004d10 2023-10-30T14:10:15.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f31887640 1 --2- 172.21.15.46:0/1218357184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f18004d10 0x7f3f18005190 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f3f24001e70 tx=0x7f3f2401c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad352310c2505320 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:15.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f24003b80 con 0x7f3f18004d10 2023-10-30T14:10:15.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f24003ce0 con 0x7f3f18004d10 2023-10-30T14:10:15.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f23fff640 1 -- 172.21.15.46:0/1218357184 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f240052c0 con 0x7f3f18004d10 2023-10-30T14:10:15.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f332ea640 1 -- 172.21.15.46:0/1218357184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f18004d10 msgr2=0x7f3f18005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:15.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f332ea640 1 --2- 172.21.15.46:0/1218357184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f18004d10 0x7f3f18005190 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f3f24001e70 tx=0x7f3f2401c760 comp rx=0 tx=0).stop 2023-10-30T14:10:15.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.029+0000 7f3f31887640 1 -- 172.21.15.46:0/1218357184 reap_dead start 2023-10-30T14:10:15.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f332ea640 1 -- 172.21.15.46:0/1218357184 shutdown_connections 2023-10-30T14:10:15.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f332ea640 1 -- 172.21.15.46:0/1218357184 >> 172.21.15.46:0/1218357184 conn(0x7f3f2c06bfe0 msgr2=0x7f3f2c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:15.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f332ea640 1 -- 172.21.15.46:0/1218357184 shutdown_connections 2023-10-30T14:10:15.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f332ea640 1 -- 172.21.15.46:0/1218357184 wait complete. 2023-10-30T14:10:15.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f332ea640 1 Processor -- start 2023-10-30T14:10:15.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f332ea640 1 -- start start 2023-10-30T14:10:15.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f332ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f18004d10 0x7f3f2c1e3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:15.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f332ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c1e3ab0 0x7f3f2c146de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:15.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f332ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c0cd5b0 0x7f3f2c147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:15.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f332ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c1e4f60 con 0x7f3f18004d10 2023-10-30T14:10:15.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f332ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c1e2aa0 con 0x7f3f2c0cd5b0 2023-10-30T14:10:15.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f332ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c0ce810 con 0x7f3f2c1e3ab0 2023-10-30T14:10:15.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f31086640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f18004d10 0x7f3f2c1e3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:15.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f31086640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f18004d10 0x7f3f2c1e3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52866/0 (socket says 172.21.15.46:52866) 2023-10-30T14:10:15.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f31086640 1 -- 172.21.15.46:0/2691430179 learned_addr learned my addr 172.21.15.46:0/2691430179 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:15.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f30885640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c1e3ab0 0x7f3f2c146de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:15.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f31887640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c0cd5b0 0x7f3f2c147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:15.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f31086640 1 -- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c1e3ab0 msgr2=0x7f3f2c146de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:15.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f31086640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c1e3ab0 0x7f3f2c146de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:15.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f31086640 1 -- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c0cd5b0 msgr2=0x7f3f2c147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:15.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f31086640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c0cd5b0 0x7f3f2c147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:15.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f31086640 1 -- 172.21.15.46:0/2691430179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c1e3210 con 0x7f3f18004d10 2023-10-30T14:10:15.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.033+0000 7f3f31086640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f18004d10 0x7f3f2c1e3570 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f3f280072f0 tx=0x7f3f280077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:15.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.037+0000 7f3f21ffb640 1 -- 172.21.15.46:0/2691430179 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2800f9c0 con 0x7f3f18004d10 2023-10-30T14:10:15.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.037+0000 7f3f21ffb640 1 -- 172.21.15.46:0/2691430179 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2800fb20 con 0x7f3f18004d10 2023-10-30T14:10:15.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.037+0000 7f3f21ffb640 1 -- 172.21.15.46:0/2691430179 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2800fed0 con 0x7f3f18004d10 2023-10-30T14:10:15.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.037+0000 7f3f332ea640 1 -- 172.21.15.46:0/2691430179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2c14d800 con 0x7f3f18004d10 2023-10-30T14:10:15.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.037+0000 7f3f332ea640 1 -- 172.21.15.46:0/2691430179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2c0ced60 con 0x7f3f18004d10 2023-10-30T14:10:15.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.037+0000 7f3f332ea640 1 -- 172.21.15.46:0/2691430179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2c04e170 con 0x7f3f18004d10 2023-10-30T14:10:15.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.037+0000 7f3f21ffb640 1 -- 172.21.15.46:0/2691430179 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3f2801d460 con 0x7f3f18004d10 2023-10-30T14:10:15.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.037+0000 7f3f21ffb640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f100664d0 0x7f3f10068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:15.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.037+0000 7f3f21ffb640 1 -- 172.21.15.46:0/2691430179 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f3f280849e0 con 0x7f3f18004d10 2023-10-30T14:10:15.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.037+0000 7f3f30885640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f100664d0 0x7f3f10068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:15.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.041+0000 7f3f30885640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f100664d0 0x7f3f10068990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c0b5200 tx=0x7f3f1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:15.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.041+0000 7f3f21ffb640 1 -- 172.21.15.46:0/2691430179 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3f280515f0 con 0x7f3f18004d10 2023-10-30T14:10:15.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.233+0000 7f3f332ea640 1 -- 172.21.15.46:0/2691430179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f2c0b1dd0 con 0x7f3f18004d10 2023-10-30T14:10:15.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.233+0000 7f3f21ffb640 1 -- 172.21.15.46:0/2691430179 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f3f28054c10 con 0x7f3f18004d10 2023-10-30T14:10:15.238 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:10:15.239 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":206,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:10.944299+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:10:15.240 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 206 2023-10-30T14:10:15.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.237+0000 7f3f332ea640 1 -- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f100664d0 msgr2=0x7f3f10068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:15.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.237+0000 7f3f332ea640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f100664d0 0x7f3f10068990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c0b5200 tx=0x7f3f1c008040 comp rx=0 tx=0).stop 2023-10-30T14:10:15.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.237+0000 7f3f332ea640 1 -- 172.21.15.46:0/2691430179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f18004d10 msgr2=0x7f3f2c1e3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:15.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.237+0000 7f3f332ea640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f18004d10 0x7f3f2c1e3570 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f3f280072f0 tx=0x7f3f280077b0 comp rx=0 tx=0).stop 2023-10-30T14:10:15.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.237+0000 7f3f332ea640 1 -- 172.21.15.46:0/2691430179 shutdown_connections 2023-10-30T14:10:15.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.237+0000 7f3f332ea640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3f100664d0 0x7f3f10068990 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:15.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.237+0000 7f3f332ea640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c0cd5b0 0x7f3f2c147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:15.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.237+0000 7f3f332ea640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f2c1e3ab0 0x7f3f2c146de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:15.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.237+0000 7f3f332ea640 1 --2- 172.21.15.46:0/2691430179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3f18004d10 0x7f3f2c1e3570 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:15.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.237+0000 7f3f332ea640 1 -- 172.21.15.46:0/2691430179 >> 172.21.15.46:0/2691430179 conn(0x7f3f2c06bfe0 msgr2=0x7f3f2c151240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:15.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.237+0000 7f3f332ea640 1 -- 172.21.15.46:0/2691430179 shutdown_connections 2023-10-30T14:10:15.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:15.237+0000 7f3f332ea640 1 -- 172.21.15.46:0/2691430179 wait complete. 2023-10-30T14:10:15.259 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 206, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 206, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8655": { "addr": "172.21.15.46:6837/3044281789", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 3044281789, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 3044281789, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8655, "incarnation": 196, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 24 }, "gid_8826": { "addr": "172.21.15.46:6836/66265765", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 66265765, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 66265765, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8826, "incarnation": 203, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:10:10.944299+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8826, "mds_1": 8655 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:10:17.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:10:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff9a4b0f640 1 Processor -- start 2023-10-30T14:10:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff9a4b0f640 1 -- start start 2023-10-30T14:10:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff9a4b0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9a00c05d0 0x7ff9a013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff9a4b0f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9a00c6f60 0x7ff9a013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff9a4b0f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a00c0a50 con 0x7ff9a00c6b80 2023-10-30T14:10:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff9a4b0f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a006f6e0 con 0x7ff9a00c6f60 2023-10-30T14:10:17.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff9a4b0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a006f9c0 con 0x7ff9a00c05d0 2023-10-30T14:10:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff99ed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9a00c6f60 0x7ff9a013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:17.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff99ed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9a00c6f60 0x7ff9a013f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34166/0 (socket says 172.21.15.46:34166) 2023-10-30T14:10:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff99ed9d640 1 -- 172.21.15.46:0/1418220933 learned_addr learned my addr 172.21.15.46:0/1418220933 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff997fff640 1 --2- 172.21.15.46:0/1418220933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9a00c05d0 0x7ff9a013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff99ed9d640 1 -- 172.21.15.46:0/1418220933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9a00c05d0 msgr2=0x7ff9a013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff99ed9d640 1 --2- 172.21.15.46:0/1418220933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9a00c05d0 0x7ff9a013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:17.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff99ed9d640 1 -- 172.21.15.46:0/1418220933 >> v1:172.21.15.46:6789/0 conn(0x7ff9a00c6b80 legacy=0x7ff9a00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.413+0000 7ff99ed9d640 1 -- 172.21.15.46:0/1418220933 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a006fca0 con 0x7ff9a00c6f60 2023-10-30T14:10:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff99ed9d640 1 --2- 172.21.15.46:0/1418220933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9a00c6f60 0x7ff9a013f2d0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7ff998009e50 tx=0x7ff99801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9bcb748225a94b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:17.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff99dd9b640 1 -- 172.21.15.46:0/1418220933 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980a57f0 con 0x7ff9a00c6f60 2023-10-30T14:10:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff99dd9b640 1 -- 172.21.15.46:0/1418220933 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9980a5950 con 0x7ff9a00c6f60 2023-10-30T14:10:17.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff99dd9b640 1 -- 172.21.15.46:0/1418220933 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980a5ce0 con 0x7ff9a00c6f60 2023-10-30T14:10:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1418220933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9a00c6f60 msgr2=0x7ff9a013f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff9a4b0f640 1 --2- 172.21.15.46:0/1418220933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9a00c6f60 0x7ff9a013f2d0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7ff998009e50 tx=0x7ff99801ca10 comp rx=0 tx=0).stop 2023-10-30T14:10:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1418220933 shutdown_connections 2023-10-30T14:10:17.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff9a4b0f640 1 --2- 172.21.15.46:0/1418220933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9a00c6f60 0x7ff9a013f2d0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff9a4b0f640 1 --2- 172.21.15.46:0/1418220933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9a00c05d0 0x7ff9a013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1418220933 >> 172.21.15.46:0/1418220933 conn(0x7ff9a00c7910 msgr2=0x7ff9a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1418220933 shutdown_connections 2023-10-30T14:10:17.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1418220933 wait complete. 2023-10-30T14:10:17.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff9a4b0f640 1 Processor -- start 2023-10-30T14:10:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.417+0000 7ff9a4b0f640 1 -- start start 2023-10-30T14:10:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff9a4b0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9a00c05d0 0x7ff9a0136e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff9a4b0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9a00c6b80 0x7ff9a01f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff9a4b0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9a00c6f60 0x7ff9a0153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:17.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff9a4b0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a0154010 con 0x7ff9a00c6f60 2023-10-30T14:10:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff9a4b0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a01f3500 con 0x7ff9a00c6b80 2023-10-30T14:10:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff9a4b0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a00c3d60 con 0x7ff9a00c05d0 2023-10-30T14:10:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff99ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9a00c6f60 0x7ff9a0153ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff99ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9a00c6f60 0x7ff9a0153ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34174/0 (socket says 172.21.15.46:34174) 2023-10-30T14:10:17.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff99ed9d640 1 -- 172.21.15.46:0/1576185091 learned_addr learned my addr 172.21.15.46:0/1576185091 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff997fff640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9a00c6b80 0x7ff9a01f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff99e59c640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9a00c05d0 0x7ff9a0136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff99ed9d640 1 -- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9a00c05d0 msgr2=0x7ff9a0136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:17.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff99ed9d640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9a00c05d0 0x7ff9a0136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff99ed9d640 1 -- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9a00c6b80 msgr2=0x7ff9a01f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff99ed9d640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9a00c6b80 0x7ff9a01f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff99ed9d640 1 -- 172.21.15.46:0/1576185091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff998009b30 con 0x7ff9a00c6f60 2023-10-30T14:10:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff99e59c640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9a00c05d0 0x7ff9a0136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:10:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff99ed9d640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9a00c6f60 0x7ff9a0153ad0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7ff99801a640 tx=0x7ff9980197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:17.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff996ffd640 1 -- 172.21.15.46:0/1576185091 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980bd070 con 0x7ff9a00c6f60 2023-10-30T14:10:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff996ffd640 1 -- 172.21.15.46:0/1576185091 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9980b8680 con 0x7ff9a00c6f60 2023-10-30T14:10:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff996ffd640 1 -- 172.21.15.46:0/1576185091 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980b8a50 con 0x7ff9a00c6f60 2023-10-30T14:10:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1576185091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a00c4550 con 0x7ff9a00c6f60 2023-10-30T14:10:17.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1576185091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a00c4b10 con 0x7ff9a00c6f60 2023-10-30T14:10:17.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1576185091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a004e170 con 0x7ff9a00c6f60 2023-10-30T14:10:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.421+0000 7ff996ffd640 1 -- 172.21.15.46:0/1576185091 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff9980b8bb0 con 0x7ff9a00c6f60 2023-10-30T14:10:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.425+0000 7ff996ffd640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9800664a0 0x7ff980068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.425+0000 7ff996ffd640 1 -- 172.21.15.46:0/1576185091 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7ff99811fa50 con 0x7ff9a00c6f60 2023-10-30T14:10:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.425+0000 7ff99e59c640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9800664a0 0x7ff980068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:17.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.425+0000 7ff99e59c640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9800664a0 0x7ff980068960 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff9a00ca340 tx=0x7ff990009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:17.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.428+0000 7ff996ffd640 1 -- 172.21.15.46:0/1576185091 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9980ec5c0 con 0x7ff9a00c6f60 2023-10-30T14:10:17.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1576185091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9a01f3640 con 0x7ff9a00c6f60 2023-10-30T14:10:17.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff996ffd640 1 -- 172.21.15.46:0/1576185091 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+5727 (secure 0 0 0) 0x7ff9980efbe0 con 0x7ff9a00c6f60 2023-10-30T14:10:17.645 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:10:17.645 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":206,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:10.944299+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:10:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 206 2023-10-30T14:10:17.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9800664a0 msgr2=0x7ff980068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:17.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9800664a0 0x7ff980068960 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff9a00ca340 tx=0x7ff990009040 comp rx=0 tx=0).stop 2023-10-30T14:10:17.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1576185091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9a00c6f60 msgr2=0x7ff9a0153ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:17.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9a00c6f60 0x7ff9a0153ad0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7ff99801a640 tx=0x7ff9980197b0 comp rx=0 tx=0).stop 2023-10-30T14:10:17.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1576185091 shutdown_connections 2023-10-30T14:10:17.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9800664a0 0x7ff980068960 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:17.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9a00c6f60 0x7ff9a0153ad0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:17.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9a00c6b80 0x7ff9a01f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:17.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 --2- 172.21.15.46:0/1576185091 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9a00c05d0 0x7ff9a0136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:17.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1576185091 >> 172.21.15.46:0/1576185091 conn(0x7ff9a00c7910 msgr2=0x7ff9a00c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:17.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1576185091 shutdown_connections 2023-10-30T14:10:17.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:17.640+0000 7ff9a4b0f640 1 -- 172.21.15.46:0/1576185091 wait complete. 2023-10-30T14:10:17.666 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:10:19.668 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:10:19.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0aa91640 1 Processor -- start 2023-10-30T14:10:19.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0aa91640 1 -- start start 2023-10-30T14:10:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0aa91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc041538a0 0x7fdc0415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0aa91640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc040cb800 con 0x7fdc040cf160 2023-10-30T14:10:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0aa91640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0415dc30 con 0x7fdc040ca0e0 2023-10-30T14:10:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0aa91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc040cb330 con 0x7fdc041538a0 2023-10-30T14:10:19.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0902e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdc040cf160 0x7fdc04164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53848/0 (socket says 172.21.15.46:53848) 2023-10-30T14:10:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0902e640 1 -- 172.21.15.46:0/2265241973 learned_addr learned my addr 172.21.15.46:0/2265241973 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:10:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0882d640 1 --2- 172.21.15.46:0/2265241973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc041538a0 0x7fdc0415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260554415 0 0) 0x7fdc040cb800 con 0x7fdc040cf160 2023-10-30T14:10:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf4003530 con 0x7fdc040cf160 2023-10-30T14:10:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 394403833 0 0) 0x7fdbf4003530 con 0x7fdc040cf160 2023-10-30T14:10:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc041538a0 msgr2=0x7fdc0415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 --2- 172.21.15.46:0/2265241973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc041538a0 0x7fdc0415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 >> v1:172.21.15.154:6789/0 conn(0x7fdc040ca0e0 legacy=0x7fdc0415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc040cad70 con 0x7fdc040cf160 2023-10-30T14:10:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fdbfc001a00 con 0x7fdc040cf160 2023-10-30T14:10:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 >> v1:172.21.15.46:6789/0 conn(0x7fdc040cf160 legacy=0x7fdc04164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 --2- 172.21.15.46:0/2265241973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbf40039b0 0x7fdbf4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:19.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 --2- 172.21.15.46:0/2265241973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc041538a0 0x7fdbf40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:19.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 --2- 172.21.15.46:0/2265241973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbf4004d10 0x7fdbf4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:19.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc040cb330 con 0x7fdbf4004d10 2023-10-30T14:10:19.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf400b990 con 0x7fdbf40039b0 2023-10-30T14:10:19.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf400bb50 con 0x7fdc041538a0 2023-10-30T14:10:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0902e640 1 --2- 172.21.15.46:0/2265241973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbf4004d10 0x7fdbf4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0882d640 1 --2- 172.21.15.46:0/2265241973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbf40039b0 0x7fdbf4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc03fff640 1 --2- 172.21.15.46:0/2265241973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc041538a0 0x7fdbf40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0902e640 1 -- 172.21.15.46:0/2265241973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc041538a0 msgr2=0x7fdbf40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0902e640 1 --2- 172.21.15.46:0/2265241973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc041538a0 0x7fdbf40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0902e640 1 -- 172.21.15.46:0/2265241973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbf40039b0 msgr2=0x7fdbf4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0902e640 1 --2- 172.21.15.46:0/2265241973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbf40039b0 0x7fdbf4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.820+0000 7fdc0902e640 1 -- 172.21.15.46:0/2265241973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf4003530 con 0x7fdbf4004d10 2023-10-30T14:10:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.824+0000 7fdc0902e640 1 --2- 172.21.15.46:0/2265241973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbf4004d10 0x7fdbf4005190 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc00a840 tx=0x7fdbfc01d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab1e9f282267db2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.824+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc0ad630 con 0x7fdbf4004d10 2023-10-30T14:10:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.824+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc0a8070 con 0x7fdbf4004d10 2023-10-30T14:10:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.824+0000 7fdc037fe640 1 -- 172.21.15.46:0/2265241973 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc0ad9d0 con 0x7fdbf4004d10 2023-10-30T14:10:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.824+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2265241973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbf4004d10 msgr2=0x7fdbf4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:19.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.824+0000 7fdc0aa91640 1 --2- 172.21.15.46:0/2265241973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbf4004d10 0x7fdbf4005190 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc00a840 tx=0x7fdbfc01d320 comp rx=0 tx=0).stop 2023-10-30T14:10:19.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.824+0000 7fdc0902e640 1 -- 172.21.15.46:0/2265241973 reap_dead start 2023-10-30T14:10:19.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.824+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2265241973 shutdown_connections 2023-10-30T14:10:19.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.824+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2265241973 >> 172.21.15.46:0/2265241973 conn(0x7fdc0406bfe0 msgr2=0x7fdc0406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:19.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.824+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2265241973 shutdown_connections 2023-10-30T14:10:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.824+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2265241973 wait complete. 2023-10-30T14:10:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0aa91640 1 Processor -- start 2023-10-30T14:10:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0aa91640 1 -- start start 2023-10-30T14:10:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0aa91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbf4004d10 0x7fdc041591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0aa91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc040b3a60 0x7fdc04157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:19.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0aa91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc04148e60 0x7fdc041459d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:19.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0aa91640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc040cea50 con 0x7fdbf4004d10 2023-10-30T14:10:19.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0aa91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc040b31a0 con 0x7fdc04148e60 2023-10-30T14:10:19.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0aa91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc040ce0e0 con 0x7fdc040b3a60 2023-10-30T14:10:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0882d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbf4004d10 0x7fdc041591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0882d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbf4004d10 0x7fdc041591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34202/0 (socket says 172.21.15.46:34202) 2023-10-30T14:10:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0882d640 1 -- 172.21.15.46:0/2893019350 learned_addr learned my addr 172.21.15.46:0/2893019350 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc03fff640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc040b3a60 0x7fdc04157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:19.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0902e640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc04148e60 0x7fdc041459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0882d640 1 -- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc040b3a60 msgr2=0x7fdc04157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0882d640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc040b3a60 0x7fdc04157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0882d640 1 -- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc04148e60 msgr2=0x7fdc041459d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0882d640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc04148e60 0x7fdc041459d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:19.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0882d640 1 -- 172.21.15.46:0/2893019350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc040b1bd0 con 0x7fdbf4004d10 2023-10-30T14:10:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc03fff640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc040b3a60 0x7fdc04157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:10:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0882d640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbf4004d10 0x7fdc041591a0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8011a00 tx=0x7fdbf8011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc017fa640 1 -- 172.21.15.46:0/2893019350 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf800fb50 con 0x7fdbf4004d10 2023-10-30T14:10:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc017fa640 1 -- 172.21.15.46:0/2893019350 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf800fcb0 con 0x7fdbf4004d10 2023-10-30T14:10:19.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2893019350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc041f31c0 con 0x7fdbf4004d10 2023-10-30T14:10:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc017fa640 1 -- 172.21.15.46:0/2893019350 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8014440 con 0x7fdbf4004d10 2023-10-30T14:10:19.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.828+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2893019350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc041f3480 con 0x7fdbf4004d10 2023-10-30T14:10:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.832+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2893019350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc040cecc0 con 0x7fdbf4004d10 2023-10-30T14:10:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.832+0000 7fdc017fa640 1 -- 172.21.15.46:0/2893019350 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdbf80145a0 con 0x7fdbf4004d10 2023-10-30T14:10:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.832+0000 7fdc017fa640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbe80664a0 0x7fdbe8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.832+0000 7fdc017fa640 1 -- 172.21.15.46:0/2893019350 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7fdbf8085150 con 0x7fdbf4004d10 2023-10-30T14:10:19.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.832+0000 7fdc03fff640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbe80664a0 0x7fdbe8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.832+0000 7fdc03fff640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbe80664a0 0x7fdbe8068960 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fdc04153610 tx=0x7fdbf0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:19.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:19.836+0000 7fdc017fa640 1 -- 172.21.15.46:0/2893019350 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdbf8051cf0 con 0x7fdbf4004d10 2023-10-30T14:10:20.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2893019350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc0404e170 con 0x7fdbf4004d10 2023-10-30T14:10:20.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc017fa640 1 -- 172.21.15.46:0/2893019350 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+5727 (secure 0 0 0) 0x7fdbf8055310 con 0x7fdbf4004d10 2023-10-30T14:10:20.033 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:10:20.033 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:17.646019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:10:20.034 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 207 2023-10-30T14:10:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbe80664a0 msgr2=0x7fdbe8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbe80664a0 0x7fdbe8068960 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fdc04153610 tx=0x7fdbf0009040 comp rx=0 tx=0).stop 2023-10-30T14:10:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2893019350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbf4004d10 msgr2=0x7fdc041591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbf4004d10 0x7fdc041591a0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8011a00 tx=0x7fdbf8011ec0 comp rx=0 tx=0).stop 2023-10-30T14:10:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2893019350 shutdown_connections 2023-10-30T14:10:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbe80664a0 0x7fdbe8068960 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc04148e60 0x7fdc041459d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc040b3a60 0x7fdc04157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 --2- 172.21.15.46:0/2893019350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbf4004d10 0x7fdc041591a0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2893019350 >> 172.21.15.46:0/2893019350 conn(0x7fdc0406bfe0 msgr2=0x7fdc04151000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2893019350 shutdown_connections 2023-10-30T14:10:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:20.028+0000 7fdc0aa91640 1 -- 172.21.15.46:0/2893019350 wait complete. 2023-10-30T14:10:20.054 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=8505 2023-10-30T14:10:20.055 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2023-10-30T14:10:25.589 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T14:10:25.584+0000 7f155a099640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:10:25.590 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T14:10:25.584+0000 7f155a099640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:10:27.403 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:10:27.407 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:10:27.448 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:10:27.449 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.7 2023-10-30T14:10:27.449 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 7 -eq 48 ] 2023-10-30T14:10:27.449 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.7 2023-10-30T14:10:27.449 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.7 2023-10-30T14:10:27.450 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.7 2023-10-30T14:10:27.450 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:10:28.056 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T14:10:28.057 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T14:10:28.057 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T14:10:28.061 INFO:tasks.ceph.mds.a:Started 2023-10-30T14:10:28.062 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:10:28.187 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-10-30T14:10:28.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.224+0000 7fbbe7394640 1 Processor -- start 2023-10-30T14:10:28.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe7394640 1 -- start start 2023-10-30T14:10:28.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe7394640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbbe00c6bd0 0x7fbbe006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:28.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe7394640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbe0071110 0x7fbbe0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:28.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe7394640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe0066cf0 con 0x7fbbe006f7a0 2023-10-30T14:10:28.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe7394640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe00700c0 con 0x7fbbe0071110 2023-10-30T14:10:28.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe7394640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe00c70d0 con 0x7fbbe00c6bd0 2023-10-30T14:10:28.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe5931640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbe0071110 0x7fbbe0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:28.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe492f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbbe00c6bd0 0x7fbbe006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:28.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe5931640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbe0071110 0x7fbbe0158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55752/0 (socket says 172.21.15.46:55752) 2023-10-30T14:10:28.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe5130640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fbbe006f7a0 0x7fbbe0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:57212/0 (socket says 172.21.15.46:57212) 2023-10-30T14:10:28.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe5931640 1 -- 172.21.15.46:0/118362534 learned_addr learned my addr 172.21.15.46:0/118362534 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:28.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe5931640 1 -- 172.21.15.46:0/118362534 >> v2:172.21.15.154:3300/0 conn(0x7fbbe00c6bd0 msgr2=0x7fbbe006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:28.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe5931640 1 --2- 172.21.15.46:0/118362534 >> v2:172.21.15.154:3300/0 conn(0x7fbbe00c6bd0 0x7fbbe006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:28.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe5931640 1 -- 172.21.15.46:0/118362534 >> v1:172.21.15.154:6789/0 conn(0x7fbbe006f7a0 legacy=0x7fbbe0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:28.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.228+0000 7fbbe5931640 1 -- 172.21.15.46:0/118362534 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe00c7390 con 0x7fbbe0071110 2023-10-30T14:10:28.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.232+0000 7fbbe5931640 1 --2- 172.21.15.46:0/118362534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbe0071110 0x7fbbe0158110 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fbbe006f1c0 tx=0x7fbbd801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8e74daa4fc57ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:28.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.232+0000 7fbbd7fff640 1 -- 172.21.15.46:0/118362534 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd80af400 con 0x7fbbe0071110 2023-10-30T14:10:28.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.232+0000 7fbbd7fff640 1 -- 172.21.15.46:0/118362534 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd80ab050 con 0x7fbbe0071110 2023-10-30T14:10:28.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.232+0000 7fbbd7fff640 1 -- 172.21.15.46:0/118362534 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd80af780 con 0x7fbbe0071110 2023-10-30T14:10:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.232+0000 7fbbe7394640 1 -- 172.21.15.46:0/118362534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbe0071110 msgr2=0x7fbbe0158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.232+0000 7fbbe7394640 1 --2- 172.21.15.46:0/118362534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbe0071110 0x7fbbe0158110 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fbbe006f1c0 tx=0x7fbbd801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:10:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.232+0000 7fbbe7394640 1 -- 172.21.15.46:0/118362534 shutdown_connections 2023-10-30T14:10:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.232+0000 7fbbe7394640 1 --2- 172.21.15.46:0/118362534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbe0071110 0x7fbbe0158110 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.232+0000 7fbbe7394640 1 --2- 172.21.15.46:0/118362534 >> v2:172.21.15.154:3300/0 conn(0x7fbbe00c6bd0 0x7fbbe006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.232+0000 7fbbe7394640 1 -- 172.21.15.46:0/118362534 >> 172.21.15.46:0/118362534 conn(0x7fbbe00c7910 msgr2=0x7fbbe00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.232+0000 7fbbe7394640 1 -- 172.21.15.46:0/118362534 shutdown_connections 2023-10-30T14:10:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.232+0000 7fbbe7394640 1 -- 172.21.15.46:0/118362534 wait complete. 2023-10-30T14:10:28.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.236+0000 7fbbe7394640 1 Processor -- start 2023-10-30T14:10:28.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.236+0000 7fbbe7394640 1 -- start start 2023-10-30T14:10:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.236+0000 7fbbe7394640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbe006f7a0 0x7fbbe00c1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.236+0000 7fbbe7394640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbe0071110 0x7fbbe01f2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:28.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.236+0000 7fbbe5130640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbe006f7a0 0x7fbbe00c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:28.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.236+0000 7fbbe5130640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbe006f7a0 0x7fbbe00c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55754/0 (socket says 172.21.15.46:55754) 2023-10-30T14:10:28.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.236+0000 7fbbe7394640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbe00c6bd0 0x7fbbe01f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:28.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.236+0000 7fbbe7394640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe0155170 con 0x7fbbe006f7a0 2023-10-30T14:10:28.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.236+0000 7fbbe7394640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe0154e90 con 0x7fbbe00c6bd0 2023-10-30T14:10:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.236+0000 7fbbe7394640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe01529e0 con 0x7fbbe0071110 2023-10-30T14:10:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.236+0000 7fbbe5130640 1 -- 172.21.15.46:0/2848228758 learned_addr learned my addr 172.21.15.46:0/2848228758 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.236+0000 7fbbe492f640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbe0071110 0x7fbbe01f2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbe5130640 1 -- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbe0071110 msgr2=0x7fbbe01f2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbe5130640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbe0071110 0x7fbbe01f2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbe5931640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbe00c6bd0 0x7fbbe01f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbe5130640 1 -- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbe00c6bd0 msgr2=0x7fbbe01f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbe5130640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbe00c6bd0 0x7fbbe01f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbe5130640 1 -- 172.21.15.46:0/2848228758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd8009b30 con 0x7fbbe006f7a0 2023-10-30T14:10:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbe5931640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbe00c6bd0 0x7fbbe01f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:10:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbe5130640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbe006f7a0 0x7fbbe00c1710 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00c430 tx=0x7fbbdc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbd5ffb640 1 -- 172.21.15.46:0/2848228758 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc0074e0 con 0x7fbbe006f7a0 2023-10-30T14:10:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbd5ffb640 1 -- 172.21.15.46:0/2848228758 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbdc007640 con 0x7fbbe006f7a0 2023-10-30T14:10:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbe7394640 1 -- 172.21.15.46:0/2848228758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe0152cc0 con 0x7fbbe006f7a0 2023-10-30T14:10:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbd5ffb640 1 -- 172.21.15.46:0/2848228758 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc004a10 con 0x7fbbe006f7a0 2023-10-30T14:10:28.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbe7394640 1 -- 172.21.15.46:0/2848228758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe0153280 con 0x7fbbe006f7a0 2023-10-30T14:10:28.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbe7394640 1 -- 172.21.15.46:0/2848228758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbe004e170 con 0x7fbbe006f7a0 2023-10-30T14:10:28.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.240+0000 7fbbd5ffb640 1 -- 172.21.15.46:0/2848228758 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbbdc004b70 con 0x7fbbe006f7a0 2023-10-30T14:10:28.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.244+0000 7fbbd5ffb640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbbc40664d0 0x7fbbc4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:28.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.244+0000 7fbbd5ffb640 1 -- 172.21.15.46:0/2848228758 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7fbbdc084160 con 0x7fbbe006f7a0 2023-10-30T14:10:28.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.244+0000 7fbbe492f640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbbc40664d0 0x7fbbc4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:28.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.244+0000 7fbbe492f640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbbc40664d0 0x7fbbc4068990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00c7f10 tx=0x7fbbd0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:28.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.248+0000 7fbbd5ffb640 1 -- 172.21.15.46:0/2848228758 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbbdc050ce0 con 0x7fbbe006f7a0 2023-10-30T14:10:28.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.440+0000 7fbbe7394640 1 -- 172.21.15.46:0/2848228758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbe00c4300 con 0x7fbbe006f7a0 2023-10-30T14:10:28.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.440+0000 7fbbd5ffb640 1 -- 172.21.15.46:0/2848228758 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+5727 (secure 0 0 0) 0x7fbbdc054300 con 0x7fbbe006f7a0 2023-10-30T14:10:28.447 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:10:28.447 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:17.646019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:10:28.448 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 207 2023-10-30T14:10:28.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.444+0000 7fbbe7394640 1 -- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbbc40664d0 msgr2=0x7fbbc4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:28.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.444+0000 7fbbe7394640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbbc40664d0 0x7fbbc4068990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00c7f10 tx=0x7fbbd0008040 comp rx=0 tx=0).stop 2023-10-30T14:10:28.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.444+0000 7fbbe7394640 1 -- 172.21.15.46:0/2848228758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbe006f7a0 msgr2=0x7fbbe00c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:28.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.444+0000 7fbbe7394640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbe006f7a0 0x7fbbe00c1710 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00c430 tx=0x7fbbdc00c8f0 comp rx=0 tx=0).stop 2023-10-30T14:10:28.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.444+0000 7fbbe7394640 1 -- 172.21.15.46:0/2848228758 shutdown_connections 2023-10-30T14:10:28.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.444+0000 7fbbe7394640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbbc40664d0 0x7fbbc4068990 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:28.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.444+0000 7fbbe7394640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbe00c6bd0 0x7fbbe01f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:28.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.444+0000 7fbbe7394640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbe0071110 0x7fbbe01f2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:28.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.444+0000 7fbbe7394640 1 --2- 172.21.15.46:0/2848228758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbe006f7a0 0x7fbbe00c1710 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:28.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.444+0000 7fbbe7394640 1 -- 172.21.15.46:0/2848228758 >> 172.21.15.46:0/2848228758 conn(0x7fbbe00c7910 msgr2=0x7fbbe00b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:28.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.444+0000 7fbbe7394640 1 -- 172.21.15.46:0/2848228758 shutdown_connections 2023-10-30T14:10:28.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:28.444+0000 7fbbe7394640 1 -- 172.21.15.46:0/2848228758 wait complete. 2023-10-30T14:10:28.474 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T14:10:30.476 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:10:30.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.628+0000 7fbb5afc4640 1 Processor -- start 2023-10-30T14:10:30.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.628+0000 7fbb5afc4640 1 -- start start 2023-10-30T14:10:30.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.628+0000 7fbb5afc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb540c05d0 0x7fbb5413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:30.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.628+0000 7fbb5afc4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb540c6f60 0x7fbb5413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:30.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.628+0000 7fbb5afc4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb540c0a50 con 0x7fbb540c6b80 2023-10-30T14:10:30.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.628+0000 7fbb5afc4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb5406f6e0 con 0x7fbb540c6f60 2023-10-30T14:10:30.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.628+0000 7fbb5afc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5406f9c0 con 0x7fbb540c05d0 2023-10-30T14:10:30.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.628+0000 7fbb58d60640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbb540c6b80 0x7fbb540bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41920/0 (socket says 172.21.15.46:41920) 2023-10-30T14:10:30.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.628+0000 7fbb58d60640 1 -- 172.21.15.46:0/1122057828 learned_addr learned my addr 172.21.15.46:0/1122057828 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:10:30.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.628+0000 7fbb59561640 1 --2- 172.21.15.46:0/1122057828 >> v2:172.21.15.46:3300/0 conn(0x7fbb540c6f60 0x7fbb5413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:30.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.628+0000 7fbb53fff640 1 --2- 172.21.15.46:0/1122057828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb540c05d0 0x7fbb5413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:30.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.628+0000 7fbb537fe640 1 -- 172.21.15.46:0/1122057828 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699815447 0 0) 0x7fbb540c0a50 con 0x7fbb540c6b80 2023-10-30T14:10:30.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb537fe640 1 -- 172.21.15.46:0/1122057828 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb44003530 con 0x7fbb540c6b80 2023-10-30T14:10:30.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb59561640 1 -- 172.21.15.46:0/1122057828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb540c05d0 msgr2=0x7fbb5413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:10:30.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb59561640 1 --2- 172.21.15.46:0/1122057828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb540c05d0 0x7fbb5413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:30.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb59561640 1 -- 172.21.15.46:0/1122057828 >> v1:172.21.15.46:6789/0 conn(0x7fbb540c6b80 legacy=0x7fbb540bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:30.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb59561640 1 -- 172.21.15.46:0/1122057828 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5406fca0 con 0x7fbb540c6f60 2023-10-30T14:10:30.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb59561640 1 --2- 172.21.15.46:0/1122057828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb540c6f60 0x7fbb5413f2d0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c009d90 tx=0x7fbb4c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790080cfbc11707d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:30.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb537fe640 1 -- 172.21.15.46:0/1122057828 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c0a5920 con 0x7fbb540c6f60 2023-10-30T14:10:30.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb537fe640 1 -- 172.21.15.46:0/1122057828 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4c0a5a80 con 0x7fbb540c6f60 2023-10-30T14:10:30.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb537fe640 1 -- 172.21.15.46:0/1122057828 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c0a5e10 con 0x7fbb540c6f60 2023-10-30T14:10:30.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb5afc4640 1 -- 172.21.15.46:0/1122057828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb540c6f60 msgr2=0x7fbb5413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:30.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb5afc4640 1 --2- 172.21.15.46:0/1122057828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb540c6f60 0x7fbb5413f2d0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c009d90 tx=0x7fbb4c01c950 comp rx=0 tx=0).stop 2023-10-30T14:10:30.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb5afc4640 1 -- 172.21.15.46:0/1122057828 shutdown_connections 2023-10-30T14:10:30.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb5afc4640 1 --2- 172.21.15.46:0/1122057828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb540c6f60 0x7fbb5413f2d0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:30.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb5afc4640 1 --2- 172.21.15.46:0/1122057828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb540c05d0 0x7fbb5413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:30.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb5afc4640 1 -- 172.21.15.46:0/1122057828 >> 172.21.15.46:0/1122057828 conn(0x7fbb540c7910 msgr2=0x7fbb540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:30.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.632+0000 7fbb5afc4640 1 -- 172.21.15.46:0/1122057828 shutdown_connections 2023-10-30T14:10:30.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb5afc4640 1 -- 172.21.15.46:0/1122057828 wait complete. 2023-10-30T14:10:30.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb5afc4640 1 Processor -- start 2023-10-30T14:10:30.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb5afc4640 1 -- start start 2023-10-30T14:10:30.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb5afc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb540c05d0 0x7fbb5414f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:30.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb5afc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb540c6b80 0x7fbb540c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:30.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb5afc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb540c6f60 0x7fbb541f5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:30.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb5afc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb540ca840 con 0x7fbb540c6b80 2023-10-30T14:10:30.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb5afc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb541620e0 con 0x7fbb540c6f60 2023-10-30T14:10:30.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb5afc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb54163350 con 0x7fbb540c05d0 2023-10-30T14:10:30.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb53fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb540c6b80 0x7fbb540c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:30.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb53fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb540c6b80 0x7fbb540c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55782/0 (socket says 172.21.15.46:55782) 2023-10-30T14:10:30.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb53fff640 1 -- 172.21.15.46:0/196163955 learned_addr learned my addr 172.21.15.46:0/196163955 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:10:30.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb58d60640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb540c05d0 0x7fbb5414f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:30.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb59561640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb540c6f60 0x7fbb541f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:30.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb53fff640 1 -- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb540c05d0 msgr2=0x7fbb5414f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:30.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb53fff640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb540c05d0 0x7fbb5414f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:30.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb53fff640 1 -- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb540c6f60 msgr2=0x7fbb541f5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:30.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb53fff640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb540c6f60 0x7fbb541f5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:30.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb53fff640 1 -- 172.21.15.46:0/196163955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c009a70 con 0x7fbb540c6b80 2023-10-30T14:10:30.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb58d60640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb540c05d0 0x7fbb5414f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:10:30.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb59561640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb540c6f60 0x7fbb541f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:10:30.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.636+0000 7fbb53fff640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb540c6b80 0x7fbb540c8c10 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000e930 tx=0x7fbb4000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:30.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.640+0000 7fbb517fa640 1 -- 172.21.15.46:0/196163955 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb40009690 con 0x7fbb540c6b80 2023-10-30T14:10:30.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.640+0000 7fbb517fa640 1 -- 172.21.15.46:0/196163955 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4000ca00 con 0x7fbb540c6b80 2023-10-30T14:10:30.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.640+0000 7fbb517fa640 1 -- 172.21.15.46:0/196163955 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4000cd70 con 0x7fbb540c6b80 2023-10-30T14:10:30.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.640+0000 7fbb5afc4640 1 -- 172.21.15.46:0/196163955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb54163630 con 0x7fbb540c6b80 2023-10-30T14:10:30.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.640+0000 7fbb5afc4640 1 -- 172.21.15.46:0/196163955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5414fd10 con 0x7fbb540c6b80 2023-10-30T14:10:30.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.640+0000 7fbb5afc4640 1 -- 172.21.15.46:0/196163955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5404e170 con 0x7fbb540c6b80 2023-10-30T14:10:30.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.640+0000 7fbb517fa640 1 -- 172.21.15.46:0/196163955 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb40013470 con 0x7fbb540c6b80 2023-10-30T14:10:30.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.640+0000 7fbb517fa640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbb380664a0 0x7fbb38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:10:30.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.640+0000 7fbb517fa640 1 -- 172.21.15.46:0/196163955 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7fbb40084a10 con 0x7fbb540c6b80 2023-10-30T14:10:30.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.640+0000 7fbb58d60640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbb380664a0 0x7fbb38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:10:30.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.644+0000 7fbb58d60640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbb380664a0 0x7fbb38068960 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbb48005890 tx=0x7fbb48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:10:30.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.644+0000 7fbb517fa640 1 -- 172.21.15.46:0/196163955 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb40051640 con 0x7fbb540c6b80 2023-10-30T14:10:30.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.836+0000 7fbb5afc4640 1 -- 172.21.15.46:0/196163955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb54163f50 con 0x7fbb540c6b80 2023-10-30T14:10:30.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.836+0000 7fbb517fa640 1 -- 172.21.15.46:0/196163955 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+6520 (secure 0 0 0) 0x7fbb40021090 con 0x7fbb540c6b80 2023-10-30T14:10:30.843 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:10:30.844 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8982,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":208}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:27.888611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:10:30.844 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 208 2023-10-30T14:10:30.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.840+0000 7fbb5afc4640 1 -- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbb380664a0 msgr2=0x7fbb38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:30.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.840+0000 7fbb5afc4640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbb380664a0 0x7fbb38068960 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbb48005890 tx=0x7fbb48009040 comp rx=0 tx=0).stop 2023-10-30T14:10:30.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.840+0000 7fbb5afc4640 1 -- 172.21.15.46:0/196163955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb540c6b80 msgr2=0x7fbb540c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:10:30.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.840+0000 7fbb5afc4640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb540c6b80 0x7fbb540c8c10 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000e930 tx=0x7fbb4000edf0 comp rx=0 tx=0).stop 2023-10-30T14:10:30.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.840+0000 7fbb5afc4640 1 -- 172.21.15.46:0/196163955 shutdown_connections 2023-10-30T14:10:30.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.840+0000 7fbb5afc4640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbb380664a0 0x7fbb38068960 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:30.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.840+0000 7fbb5afc4640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb540c6f60 0x7fbb541f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.840+0000 7fbb5afc4640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb540c6b80 0x7fbb540c8c10 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.840+0000 7fbb5afc4640 1 --2- 172.21.15.46:0/196163955 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb540c05d0 0x7fbb5414f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:10:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.840+0000 7fbb5afc4640 1 -- 172.21.15.46:0/196163955 >> 172.21.15.46:0/196163955 conn(0x7fbb540c7910 msgr2=0x7fbb540bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:10:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.840+0000 7fbb5afc4640 1 -- 172.21.15.46:0/196163955 shutdown_connections 2023-10-30T14:10:30.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:10:30.840+0000 7fbb5afc4640 1 -- 172.21.15.46:0/196163955 wait complete. 2023-10-30T14:10:30.865 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T14:10:30.865 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 88 secs before thrashing 2023-10-30T14:10:35.537 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:10:35.537 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.8 2023-10-30T14:10:35.538 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 8 -eq 48 ] 2023-10-30T14:10:35.538 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.8 2023-10-30T14:10:35.538 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.8 2023-10-30T14:10:35.538 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.8 2023-10-30T14:10:35.539 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:10:38.567 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:10:38.567 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 9 -eq 48 ] 2023-10-30T14:10:38.568 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.9 2023-10-30T14:10:38.568 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.9 2023-10-30T14:10:38.568 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.9 2023-10-30T14:10:38.569 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.9 2023-10-30T14:10:38.569 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:10:40.673 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:10:40.673 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.10 2023-10-30T14:10:40.674 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 10 -eq 48 ] 2023-10-30T14:10:40.674 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.10 2023-10-30T14:10:40.674 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.10 2023-10-30T14:10:40.675 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.10 2023-10-30T14:10:40.675 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:10:46.781 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:10:46.781 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 11 -eq 48 ] 2023-10-30T14:10:46.782 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.11 2023-10-30T14:10:46.782 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.11 2023-10-30T14:10:46.783 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.11 2023-10-30T14:10:46.783 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.11 2023-10-30T14:10:46.783 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:10:54.590 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:10:54.591 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.12 2023-10-30T14:10:54.592 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 12 -eq 48 ] 2023-10-30T14:10:54.592 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.12 2023-10-30T14:10:54.592 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.12 2023-10-30T14:10:54.592 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.12 2023-10-30T14:10:54.593 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:10:56.903 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:10:56.903 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 13 -eq 48 ] 2023-10-30T14:10:56.904 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.13 2023-10-30T14:10:56.904 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.13 2023-10-30T14:10:56.904 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.13 2023-10-30T14:10:56.904 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.13 2023-10-30T14:10:56.905 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:10:57.468 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:10:57.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:11:02.562 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:11:02.563 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 14 -eq 48 ] 2023-10-30T14:11:02.563 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.14 2023-10-30T14:11:02.563 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.14 2023-10-30T14:11:02.564 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.14 2023-10-30T14:11:02.564 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.14 2023-10-30T14:11:02.564 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:11:08.303 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:11:08.304 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.15 2023-10-30T14:11:08.305 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 15 -eq 48 ] 2023-10-30T14:11:08.305 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.15 2023-10-30T14:11:08.305 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.15 2023-10-30T14:11:08.305 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.15 2023-10-30T14:11:08.305 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:11:14.598 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:11:14.598 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 16 -eq 48 ] 2023-10-30T14:11:14.598 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.16 2023-10-30T14:11:14.599 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.16 2023-10-30T14:11:14.599 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.16 2023-10-30T14:11:14.599 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.16 2023-10-30T14:11:14.600 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:11:18.567 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:11:18.567 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 17 -eq 48 ] 2023-10-30T14:11:18.567 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.17 2023-10-30T14:11:18.568 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.17 2023-10-30T14:11:18.568 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.17 2023-10-30T14:11:18.568 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.17 2023-10-30T14:11:18.569 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:11:22.066 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:11:22.066 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 18 -eq 48 ] 2023-10-30T14:11:22.066 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.18 2023-10-30T14:11:22.066 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.18 2023-10-30T14:11:22.067 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.18 2023-10-30T14:11:22.067 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.18 2023-10-30T14:11:22.068 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:11:27.246 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:11:27.246 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 19 -eq 48 ] 2023-10-30T14:11:27.247 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.19 2023-10-30T14:11:27.247 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.19 2023-10-30T14:11:27.247 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.19 2023-10-30T14:11:27.248 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.19 2023-10-30T14:11:27.248 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:11:27.590 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:11:27.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:11:37.915 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:11:37.916 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 20 -eq 48 ] 2023-10-30T14:11:37.916 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.20 2023-10-30T14:11:37.916 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.20 2023-10-30T14:11:37.916 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.20 2023-10-30T14:11:37.917 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.20 2023-10-30T14:11:37.917 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:11:42.814 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:11:42.814 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.21 2023-10-30T14:11:42.815 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 21 -eq 48 ] 2023-10-30T14:11:42.815 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.21 2023-10-30T14:11:42.815 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.21 2023-10-30T14:11:42.815 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.21 2023-10-30T14:11:42.816 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:11:52.457 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:11:52.457 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 22 -eq 48 ] 2023-10-30T14:11:52.458 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.22 2023-10-30T14:11:52.458 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.22 2023-10-30T14:11:52.459 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.22 2023-10-30T14:11:52.459 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.22 2023-10-30T14:11:52.459 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:11:56.245 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:11:56.245 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 23 -eq 48 ] 2023-10-30T14:11:56.246 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.23 2023-10-30T14:11:56.246 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.23 2023-10-30T14:11:56.247 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.23 2023-10-30T14:11:56.247 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.23 2023-10-30T14:11:56.247 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:11:57.703 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:11:57.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:11:58.866 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:11:59.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.023+0000 7fadb9ef1640 1 Processor -- start 2023-10-30T14:11:59.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.023+0000 7fadb9ef1640 1 -- start start 2023-10-30T14:11:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.027+0000 7fadb9ef1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fadb406f7a0 0x7fadb406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:11:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.027+0000 7fadb9ef1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb4066cf0 con 0x7fadb40c6bd0 2023-10-30T14:11:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.027+0000 7fadb9ef1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb40c70d0 con 0x7fadb4071110 2023-10-30T14:11:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.027+0000 7fadb9ef1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb40700c0 con 0x7fadb406f7a0 2023-10-30T14:11:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.027+0000 7fadb37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fadb406f7a0 0x7fadb406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:11:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.027+0000 7fadb37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fadb406f7a0 0x7fadb406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46882/0 (socket says 172.21.15.46:46882) 2023-10-30T14:11:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.027+0000 7fadabfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fadb40c6bd0 0x7fadb4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51118/0 (socket says 172.21.15.46:51118) 2023-10-30T14:11:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.027+0000 7fadb37fe640 1 -- 172.21.15.46:0/3485856995 learned_addr learned my addr 172.21.15.46:0/3485856995 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:11:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.027+0000 7fadb37fe640 1 -- 172.21.15.46:0/3485856995 >> v1:172.21.15.154:6789/0 conn(0x7fadb4071110 legacy=0x7fadb4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:11:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.027+0000 7fadb37fe640 1 -- 172.21.15.46:0/3485856995 >> v1:172.21.15.46:6789/0 conn(0x7fadb40c6bd0 legacy=0x7fadb4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:11:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.027+0000 7fadb2ffd640 1 -- 172.21.15.46:0/3485856995 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155409163 0 0) 0x7fadb4066cf0 con 0x7fadb40c6bd0 2023-10-30T14:11:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.027+0000 7fadb37fe640 1 -- 172.21.15.46:0/3485856995 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb40c7390 con 0x7fadb406f7a0 2023-10-30T14:11:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb37fe640 1 --2- 172.21.15.46:0/3485856995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb406f7a0 0x7fadb406fb80 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fada4009870 tx=0x7fada401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90f69ed61acf8c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:11:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb2ffd640 1 -- 172.21.15.46:0/3485856995 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada40a8a30 con 0x7fadb406f7a0 2023-10-30T14:11:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb2ffd640 1 -- 172.21.15.46:0/3485856995 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada40a8b90 con 0x7fadb406f7a0 2023-10-30T14:11:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb2ffd640 1 -- 172.21.15.46:0/3485856995 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada40b0440 con 0x7fadb406f7a0 2023-10-30T14:11:59.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3485856995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb406f7a0 msgr2=0x7fadb406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:11:59.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb9ef1640 1 --2- 172.21.15.46:0/3485856995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb406f7a0 0x7fadb406fb80 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fada4009870 tx=0x7fada401ed30 comp rx=0 tx=0).stop 2023-10-30T14:11:59.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3485856995 shutdown_connections 2023-10-30T14:11:59.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb9ef1640 1 --2- 172.21.15.46:0/3485856995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb406f7a0 0x7fadb406fb80 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3485856995 >> 172.21.15.46:0/3485856995 conn(0x7fadb40c7910 msgr2=0x7fadb40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:11:59.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3485856995 shutdown_connections 2023-10-30T14:11:59.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3485856995 wait complete. 2023-10-30T14:11:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb9ef1640 1 Processor -- start 2023-10-30T14:11:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.031+0000 7fadb9ef1640 1 -- start start 2023-10-30T14:11:59.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb9ef1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb406f7a0 0x7fadb4152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:11:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb9ef1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadb4071110 0x7fadb41de540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:11:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb9ef1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadb40c6bd0 0x7fadb41ed010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:11:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb9ef1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb41ecbc0 con 0x7fadb406f7a0 2023-10-30T14:11:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb9ef1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb41f02b0 con 0x7fadb4071110 2023-10-30T14:11:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb9ef1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb41f0590 con 0x7fadb40c6bd0 2023-10-30T14:11:59.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb406f7a0 0x7fadb4152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:11:59.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb406f7a0 0x7fadb4152350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46896/0 (socket says 172.21.15.46:46896) 2023-10-30T14:11:59.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb37fe640 1 -- 172.21.15.46:0/3887727188 learned_addr learned my addr 172.21.15.46:0/3887727188 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:11:59.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadabfff640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadb4071110 0x7fadb41de540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:11:59.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb3fff640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadb40c6bd0 0x7fadb41ed010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:11:59.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb37fe640 1 -- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadb40c6bd0 msgr2=0x7fadb41ed010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:11:59.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb37fe640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadb40c6bd0 0x7fadb41ed010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb37fe640 1 -- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadb4071110 msgr2=0x7fadb41de540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:11:59.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb37fe640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadb4071110 0x7fadb41de540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb37fe640 1 -- 172.21.15.46:0/3887727188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada40094d0 con 0x7fadb406f7a0 2023-10-30T14:11:59.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb3fff640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadb40c6bd0 0x7fadb41ed010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:11:59.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb37fe640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb406f7a0 0x7fadb4152350 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fada40b3040 tx=0x7fada401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:11:59.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb0ff9640 1 -- 172.21.15.46:0/3887727188 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada40a8a30 con 0x7fadb406f7a0 2023-10-30T14:11:59.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb0ff9640 1 -- 172.21.15.46:0/3887727188 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada40bb480 con 0x7fadb406f7a0 2023-10-30T14:11:59.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb0ff9640 1 -- 172.21.15.46:0/3887727188 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada40bb850 con 0x7fadb406f7a0 2023-10-30T14:11:59.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3887727188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb41f0870 con 0x7fadb406f7a0 2023-10-30T14:11:59.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3887727188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb41f0e30 con 0x7fadb406f7a0 2023-10-30T14:11:59.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3887727188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb404e170 con 0x7fadb406f7a0 2023-10-30T14:11:59.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.035+0000 7fadb0ff9640 1 -- 172.21.15.46:0/3887727188 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fada40bb9b0 con 0x7fadb406f7a0 2023-10-30T14:11:59.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.039+0000 7fadb0ff9640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad940664a0 0x7fad94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:11:59.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.039+0000 7fadb0ff9640 1 -- 172.21.15.46:0/3887727188 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7fada4122db0 con 0x7fadb406f7a0 2023-10-30T14:11:59.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.039+0000 7fadabfff640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad940664a0 0x7fad94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:11:59.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.039+0000 7fadabfff640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad940664a0 0x7fad94068960 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fadb40b0650 tx=0x7fad9c007bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:11:59.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.043+0000 7fadb0ff9640 1 -- 172.21.15.46:0/3887727188 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fada4127070 con 0x7fadb406f7a0 2023-10-30T14:11:59.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3887727188 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadb40c8c10 con 0x7fadb406f7a0 2023-10-30T14:11:59.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb0ff9640 1 -- 172.21.15.46:0/3887727188 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+6520 (secure 0 0 0) 0x7fada40b5070 con 0x7fadb406f7a0 2023-10-30T14:11:59.245 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:11:59.245 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8982,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":208}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:27.888611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:11:59.245 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 208 2023-10-30T14:11:59.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad940664a0 msgr2=0x7fad94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:11:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad940664a0 0x7fad94068960 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fadb40b0650 tx=0x7fad9c007bc0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3887727188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb406f7a0 msgr2=0x7fadb4152350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:11:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb406f7a0 0x7fadb4152350 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fada40b3040 tx=0x7fada401bad0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3887727188 shutdown_connections 2023-10-30T14:11:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad940664a0 0x7fad94068960 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadb40c6bd0 0x7fadb41ed010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadb4071110 0x7fadb41de540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 --2- 172.21.15.46:0/3887727188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb406f7a0 0x7fadb4152350 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3887727188 >> 172.21.15.46:0/3887727188 conn(0x7fadb40c7910 msgr2=0x7fadb40c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:11:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3887727188 shutdown_connections 2023-10-30T14:11:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.239+0000 7fadb9ef1640 1 -- 172.21.15.46:0/3887727188 wait complete. 2023-10-30T14:11:59.265 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=1) 2023-10-30T14:11:59.266 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:11:59.266 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:11:59.289 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:11:59.289 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:11:59.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae1550f640 1 Processor -- start 2023-10-30T14:11:59.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae1550f640 1 -- start start 2023-10-30T14:11:59.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae1550f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae1015d6f0 0x7fae1015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:11:59.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae1550f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae100cf160 0x7fae100cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:11:59.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae1550f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae100cb800 con 0x7fae101538a0 2023-10-30T14:11:59.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae1550f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae100cb330 con 0x7fae1015d6f0 2023-10-30T14:11:59.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae1550f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae100cad70 con 0x7fae100cf160 2023-10-30T14:11:59.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae0effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fae101538a0 0x7fae100ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51124/0 (socket says 172.21.15.46:51124) 2023-10-30T14:11:59.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae0effd640 1 -- 172.21.15.46:0/3876310519 learned_addr learned my addr 172.21.15.46:0/3876310519 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:11:59.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae07fff640 1 --2- 172.21.15.46:0/3876310519 >> v2:172.21.15.46:3300/0 conn(0x7fae1015d6f0 0x7fae1015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:11:59.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae0f7fe640 1 --2- 172.21.15.46:0/3876310519 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae100cf160 0x7fae100cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:11:59.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae0e7fc640 1 -- 172.21.15.46:0/3876310519 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935648020 0 0) 0x7fae100cb800 con 0x7fae101538a0 2023-10-30T14:11:59.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae0e7fc640 1 -- 172.21.15.46:0/3876310519 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadfc003530 con 0x7fae101538a0 2023-10-30T14:11:59.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae07fff640 1 -- 172.21.15.46:0/3876310519 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae100cf160 msgr2=0x7fae100cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:11:59.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae07fff640 1 --2- 172.21.15.46:0/3876310519 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae100cf160 0x7fae100cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae07fff640 1 -- 172.21.15.46:0/3876310519 >> v1:172.21.15.46:6789/0 conn(0x7fae101538a0 legacy=0x7fae100ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:11:59.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.467+0000 7fae07fff640 1 -- 172.21.15.46:0/3876310519 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae100cb050 con 0x7fae1015d6f0 2023-10-30T14:11:59.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.471+0000 7fae07fff640 1 --2- 172.21.15.46:0/3876310519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae1015d6f0 0x7fae1015a090 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fadf80097c0 tx=0x7fadf801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee50466df7426c62 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:11:59.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.471+0000 7fae0e7fc640 1 -- 172.21.15.46:0/3876310519 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf80a8970 con 0x7fae1015d6f0 2023-10-30T14:11:59.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.471+0000 7fae0e7fc640 1 -- 172.21.15.46:0/3876310519 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf80a8ad0 con 0x7fae1015d6f0 2023-10-30T14:11:59.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.471+0000 7fae0e7fc640 1 -- 172.21.15.46:0/3876310519 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf80a8e60 con 0x7fae1015d6f0 2023-10-30T14:11:59.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.471+0000 7fae1550f640 1 -- 172.21.15.46:0/3876310519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae1015d6f0 msgr2=0x7fae1015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:11:59.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.471+0000 7fae1550f640 1 --2- 172.21.15.46:0/3876310519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae1015d6f0 0x7fae1015a090 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fadf80097c0 tx=0x7fadf801ec70 comp rx=0 tx=0).stop 2023-10-30T14:11:59.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.471+0000 7fae1550f640 1 -- 172.21.15.46:0/3876310519 shutdown_connections 2023-10-30T14:11:59.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.471+0000 7fae1550f640 1 --2- 172.21.15.46:0/3876310519 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae100cf160 0x7fae100cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.471+0000 7fae1550f640 1 --2- 172.21.15.46:0/3876310519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae1015d6f0 0x7fae1015a090 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.471+0000 7fae1550f640 1 -- 172.21.15.46:0/3876310519 >> 172.21.15.46:0/3876310519 conn(0x7fae1006bfe0 msgr2=0x7fae1006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:11:59.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.471+0000 7fae1550f640 1 -- 172.21.15.46:0/3876310519 shutdown_connections 2023-10-30T14:11:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.471+0000 7fae1550f640 1 -- 172.21.15.46:0/3876310519 wait complete. 2023-10-30T14:11:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae1550f640 1 Processor -- start 2023-10-30T14:11:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae1550f640 1 -- start start 2023-10-30T14:11:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae1550f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100cf160 0x7fae10143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:11:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae1550f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae101538a0 0x7fae10157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:11:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae1550f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae1015d6f0 0x7fae101efaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:11:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae1550f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae101f26e0 con 0x7fae100cf160 2023-10-30T14:11:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae1550f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae100b3590 con 0x7fae101538a0 2023-10-30T14:11:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae1550f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae100b32b0 con 0x7fae1015d6f0 2023-10-30T14:11:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae0effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100cf160 0x7fae10143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:11:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae0effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100cf160 0x7fae10143bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46914/0 (socket says 172.21.15.46:46914) 2023-10-30T14:11:59.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae0effd640 1 -- 172.21.15.46:0/3419113718 learned_addr learned my addr 172.21.15.46:0/3419113718 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:11:59.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae0f7fe640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae1015d6f0 0x7fae101efaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:11:59.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae07fff640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae101538a0 0x7fae10157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:11:59.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae0effd640 1 -- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae1015d6f0 msgr2=0x7fae101efaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:11:59.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae0effd640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae1015d6f0 0x7fae101efaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae0effd640 1 -- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae101538a0 msgr2=0x7fae10157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:11:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae0effd640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae101538a0 0x7fae10157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae0effd640 1 -- 172.21.15.46:0/3419113718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf8009420 con 0x7fae100cf160 2023-10-30T14:11:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae0f7fe640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae1015d6f0 0x7fae101efaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:11:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae0effd640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100cf160 0x7fae10143bd0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fae0000c500 tx=0x7fae0000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:11:59.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae077fe640 1 -- 172.21.15.46:0/3419113718 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae00005cf0 con 0x7fae100cf160 2023-10-30T14:11:59.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae077fe640 1 -- 172.21.15.46:0/3419113718 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae00005e50 con 0x7fae100cf160 2023-10-30T14:11:59.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae1550f640 1 -- 172.21.15.46:0/3419113718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae100cead0 con 0x7fae100cf160 2023-10-30T14:11:59.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae077fe640 1 -- 172.21.15.46:0/3419113718 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae000072f0 con 0x7fae100cf160 2023-10-30T14:11:59.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae1550f640 1 -- 172.21.15.46:0/3419113718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae101f28f0 con 0x7fae100cf160 2023-10-30T14:11:59.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.475+0000 7fae1550f640 1 -- 172.21.15.46:0/3419113718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae100b2a50 con 0x7fae100cf160 2023-10-30T14:11:59.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.479+0000 7fae077fe640 1 -- 172.21.15.46:0/3419113718 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fae00007450 con 0x7fae100cf160 2023-10-30T14:11:59.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.479+0000 7fae077fe640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fadf00664a0 0x7fadf0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:11:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.479+0000 7fae077fe640 1 -- 172.21.15.46:0/3419113718 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7fae00083d80 con 0x7fae100cf160 2023-10-30T14:11:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.479+0000 7fae07fff640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fadf00664a0 0x7fadf0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:11:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.479+0000 7fae07fff640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fadf00664a0 0x7fadf0068960 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fadf80b2000 tx=0x7fadf8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:11:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.483+0000 7fae077fe640 1 -- 172.21.15.46:0/3419113718 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fae000509a0 con 0x7fae100cf160 2023-10-30T14:11:59.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.671+0000 7fae1550f640 1 -- 172.21.15.46:0/3419113718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae1004e170 con 0x7fae100cf160 2023-10-30T14:11:59.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae077fe640 1 -- 172.21.15.46:0/3419113718 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+6520 (secure 0 0 0) 0x7fae00017070 con 0x7fae100cf160 2023-10-30T14:11:59.680 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:11:59.680 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8982,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":208}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:27.888611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:11:59.681 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 208 2023-10-30T14:11:59.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae1550f640 1 -- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fadf00664a0 msgr2=0x7fadf0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:11:59.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae1550f640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fadf00664a0 0x7fadf0068960 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fadf80b2000 tx=0x7fadf8008040 comp rx=0 tx=0).stop 2023-10-30T14:11:59.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae1550f640 1 -- 172.21.15.46:0/3419113718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100cf160 msgr2=0x7fae10143bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:11:59.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae1550f640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100cf160 0x7fae10143bd0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fae0000c500 tx=0x7fae0000c9c0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae1550f640 1 -- 172.21.15.46:0/3419113718 shutdown_connections 2023-10-30T14:11:59.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae1550f640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fadf00664a0 0x7fadf0068960 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae1550f640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae1015d6f0 0x7fae101efaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae1550f640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae101538a0 0x7fae10157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae1550f640 1 --2- 172.21.15.46:0/3419113718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae100cf160 0x7fae10143bd0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:11:59.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae1550f640 1 -- 172.21.15.46:0/3419113718 >> 172.21.15.46:0/3419113718 conn(0x7fae1006bfe0 msgr2=0x7fae1006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:11:59.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae1550f640 1 -- 172.21.15.46:0/3419113718 shutdown_connections 2023-10-30T14:11:59.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:11:59.675+0000 7fae1550f640 1 -- 172.21.15.46:0/3419113718 wait complete. 2023-10-30T14:11:59.701 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:12:01.702 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:01.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.851+0000 7f4e25a9e640 1 Processor -- start 2023-10-30T14:12:01.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.851+0000 7f4e25a9e640 1 -- start start 2023-10-30T14:12:01.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.851+0000 7f4e25a9e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4e200c6f60 0x7f4e201566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:01.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.851+0000 7f4e25a9e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e200c0a50 con 0x7f4e200c05d0 2023-10-30T14:12:01.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.851+0000 7f4e25a9e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2006f6e0 con 0x7f4e200c6b80 2023-10-30T14:12:01.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.851+0000 7f4e25a9e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2006f9c0 con 0x7f4e200c6f60 2023-10-30T14:12:01.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4e200c05d0 0x7f4e2013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51126/0 (socket says 172.21.15.46:51126) 2023-10-30T14:12:01.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1effd640 1 -- 172.21.15.46:0/2150745240 learned_addr learned my addr 172.21.15.46:0/2150745240 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:01.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1ffff640 1 --2- 172.21.15.46:0/2150745240 >> v2:172.21.15.154:3300/0 conn(0x7f4e200c6f60 0x7f4e201566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:01.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075579552 0 0) 0x7f4e200c0a50 con 0x7f4e200c05d0 2023-10-30T14:12:01.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e08003530 con 0x7f4e200c05d0 2023-10-30T14:12:01.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4132744937 0 0) 0x7f4e08003530 con 0x7f4e200c05d0 2023-10-30T14:12:01.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 >> v2:172.21.15.154:3300/0 conn(0x7f4e200c6f60 msgr2=0x7f4e201566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:01.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 --2- 172.21.15.46:0/2150745240 >> v2:172.21.15.154:3300/0 conn(0x7f4e200c6f60 0x7f4e201566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:01.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 >> v1:172.21.15.154:6789/0 conn(0x7f4e200c6b80 legacy=0x7f4e200bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:01.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2006fca0 con 0x7f4e200c05d0 2023-10-30T14:12:01.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f4e14001680 con 0x7f4e200c05d0 2023-10-30T14:12:01.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 >> v1:172.21.15.46:6789/0 conn(0x7f4e200c05d0 legacy=0x7f4e2013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:01.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 --2- 172.21.15.46:0/2150745240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e08003980 0x7f4e08003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:01.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 --2- 172.21.15.46:0/2150745240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e08004380 0x7f4e08004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:01.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 --2- 172.21.15.46:0/2150745240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e08005050 0x7f4e0800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:01.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2006f9c0 con 0x7f4e08003980 2023-10-30T14:12:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0800e1c0 con 0x7f4e08005050 2023-10-30T14:12:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0800e380 con 0x7f4e08004380 2023-10-30T14:12:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1f7fe640 1 --2- 172.21.15.46:0/2150745240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e08003980 0x7f4e08003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1ffff640 1 --2- 172.21.15.46:0/2150745240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e08005050 0x7f4e0800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1effd640 1 --2- 172.21.15.46:0/2150745240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e08004380 0x7f4e08004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1f7fe640 1 -- 172.21.15.46:0/2150745240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e08004380 msgr2=0x7f4e08004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1f7fe640 1 --2- 172.21.15.46:0/2150745240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e08004380 0x7f4e08004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1f7fe640 1 -- 172.21.15.46:0/2150745240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e08005050 msgr2=0x7f4e0800dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1f7fe640 1 --2- 172.21.15.46:0/2150745240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e08005050 0x7f4e0800dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1f7fe640 1 -- 172.21.15.46:0/2150745240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e08003530 con 0x7f4e08003980 2023-10-30T14:12:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.855+0000 7f4e1ffff640 1 -- 172.21.15.46:0/2150745240 reap_dead start 2023-10-30T14:12:01.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e1f7fe640 1 --2- 172.21.15.46:0/2150745240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e08003980 0x7f4e08003d60 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f4e10001860 tx=0x7f4e1001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d49685f67b7673 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:01.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e100ac640 con 0x7f4e08003980 2023-10-30T14:12:01.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e100ac7a0 con 0x7f4e08003980 2023-10-30T14:12:01.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e1e7fc640 1 -- 172.21.15.46:0/2150745240 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e100acb90 con 0x7f4e08003980 2023-10-30T14:12:01.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e25a9e640 1 -- 172.21.15.46:0/2150745240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e08003980 msgr2=0x7f4e08003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:01.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e25a9e640 1 --2- 172.21.15.46:0/2150745240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e08003980 0x7f4e08003d60 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f4e10001860 tx=0x7f4e1001c4e0 comp rx=0 tx=0).stop 2023-10-30T14:12:01.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e25a9e640 1 -- 172.21.15.46:0/2150745240 shutdown_connections 2023-10-30T14:12:01.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e25a9e640 1 --2- 172.21.15.46:0/2150745240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e08003980 0x7f4e08003d60 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:01.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e25a9e640 1 -- 172.21.15.46:0/2150745240 >> 172.21.15.46:0/2150745240 conn(0x7f4e200c7910 msgr2=0x7f4e200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:01.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e25a9e640 1 -- 172.21.15.46:0/2150745240 shutdown_connections 2023-10-30T14:12:01.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e25a9e640 1 -- 172.21.15.46:0/2150745240 wait complete. 2023-10-30T14:12:01.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e25a9e640 1 Processor -- start 2023-10-30T14:12:01.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.859+0000 7f4e25a9e640 1 -- start start 2023-10-30T14:12:01.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e25a9e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e08003980 0x7f4e201ebf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:01.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e25a9e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e2013b5b0 0x7f4e201589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:01.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e25a9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e20154e60 0x7f4e20158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:01.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e25a9e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e20139050 con 0x7f4e20154e60 2023-10-30T14:12:01.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e25a9e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e20155610 con 0x7f4e08003980 2023-10-30T14:12:01.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e25a9e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e200ca7a0 con 0x7f4e2013b5b0 2023-10-30T14:12:01.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e1ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e20154e60 0x7f4e20158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:01.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e1ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e20154e60 0x7f4e20158f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46934/0 (socket says 172.21.15.46:46934) 2023-10-30T14:12:01.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e1ffff640 1 -- 172.21.15.46:0/4081645536 learned_addr learned my addr 172.21.15.46:0/4081645536 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:01.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e1f7fe640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e08003980 0x7f4e201ebf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:01.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e1effd640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e2013b5b0 0x7f4e201589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:01.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e1ffff640 1 -- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e2013b5b0 msgr2=0x7f4e201589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:01.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e1ffff640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e2013b5b0 0x7f4e201589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:01.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e1ffff640 1 -- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e08003980 msgr2=0x7f4e201ebf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:01.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e1ffff640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e08003980 0x7f4e201ebf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:01.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e1ffff640 1 -- 172.21.15.46:0/4081645536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e200caa40 con 0x7f4e20154e60 2023-10-30T14:12:01.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e1effd640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e2013b5b0 0x7f4e201589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:12:01.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e1ffff640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e20154e60 0x7f4e20158f30 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f4e200b1190 tx=0x7f4e1800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:01.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e07fff640 1 -- 172.21.15.46:0/4081645536 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1800e040 con 0x7f4e20154e60 2023-10-30T14:12:01.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e07fff640 1 -- 172.21.15.46:0/4081645536 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1800f040 con 0x7f4e20154e60 2023-10-30T14:12:01.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e07fff640 1 -- 172.21.15.46:0/4081645536 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18015430 con 0x7f4e20154e60 2023-10-30T14:12:01.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e25a9e640 1 -- 172.21.15.46:0/4081645536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e20155860 con 0x7f4e20154e60 2023-10-30T14:12:01.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e25a9e640 1 -- 172.21.15.46:0/4081645536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e20155e20 con 0x7f4e20154e60 2023-10-30T14:12:01.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.863+0000 7f4e25a9e640 1 -- 172.21.15.46:0/4081645536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2004e170 con 0x7f4e20154e60 2023-10-30T14:12:01.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.867+0000 7f4e07fff640 1 -- 172.21.15.46:0/4081645536 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e18015590 con 0x7f4e20154e60 2023-10-30T14:12:01.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.867+0000 7f4e07fff640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e0c0664a0 0x7f4e0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:01.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.867+0000 7f4e07fff640 1 -- 172.21.15.46:0/4081645536 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f4e18084ee0 con 0x7f4e20154e60 2023-10-30T14:12:01.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.867+0000 7f4e1f7fe640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e0c0664a0 0x7f4e0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:01.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.867+0000 7f4e1f7fe640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e0c0664a0 0x7f4e0c068960 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f4e100ae000 tx=0x7f4e100a6070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:01.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:01.871+0000 7f4e07fff640 1 -- 172.21.15.46:0/4081645536 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e180519c0 con 0x7f4e20154e60 2023-10-30T14:12:02.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 -- 172.21.15.46:0/4081645536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e20157450 con 0x7f4e20154e60 2023-10-30T14:12:02.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e07fff640 1 -- 172.21.15.46:0/4081645536 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f4e18054fe0 con 0x7f4e20154e60 2023-10-30T14:12:02.069 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:02.069 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8982,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":208}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:27.888611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:02.069 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 208 2023-10-30T14:12:02.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 -- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e0c0664a0 msgr2=0x7f4e0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:02.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e0c0664a0 0x7f4e0c068960 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f4e100ae000 tx=0x7f4e100a6070 comp rx=0 tx=0).stop 2023-10-30T14:12:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 -- 172.21.15.46:0/4081645536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e20154e60 msgr2=0x7f4e20158f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e20154e60 0x7f4e20158f30 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f4e200b1190 tx=0x7f4e1800f580 comp rx=0 tx=0).stop 2023-10-30T14:12:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 -- 172.21.15.46:0/4081645536 shutdown_connections 2023-10-30T14:12:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e0c0664a0 0x7f4e0c068960 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:02.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e20154e60 0x7f4e20158f30 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:02.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e2013b5b0 0x7f4e201589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:02.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 --2- 172.21.15.46:0/4081645536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e08003980 0x7f4e201ebf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:02.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 -- 172.21.15.46:0/4081645536 >> 172.21.15.46:0/4081645536 conn(0x7f4e200c7910 msgr2=0x7f4e20137460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:02.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 -- 172.21.15.46:0/4081645536 shutdown_connections 2023-10-30T14:12:02.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:02.063+0000 7f4e25a9e640 1 -- 172.21.15.46:0/4081645536 wait complete. 2023-10-30T14:12:02.090 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:12:04.091 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:04.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.239+0000 7f233e846640 1 Processor -- start 2023-10-30T14:12:04.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.239+0000 7f233e846640 1 -- start start 2023-10-30T14:12:04.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.239+0000 7f233e846640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2338159e60 0x7f233806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:04.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.239+0000 7f233e846640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f233806a720 0x7f233806ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:04.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.239+0000 7f233e846640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233806fa80 con 0x7f23380bfec0 2023-10-30T14:12:04.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.239+0000 7f233e846640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2338071290 con 0x7f233806a720 2023-10-30T14:12:04.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.239+0000 7f233e846640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2338071570 con 0x7f2338159e60 2023-10-30T14:12:04.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.243+0000 7f233cde3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f233806a720 0x7f233806ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:04.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.243+0000 7f233cde3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f233806a720 0x7f233806ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46936/0 (socket says 172.21.15.46:46936) 2023-10-30T14:12:04.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.243+0000 7f233cde3640 1 -- 172.21.15.46:0/520769979 learned_addr learned my addr 172.21.15.46:0/520769979 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:04.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.243+0000 7f2337fff640 1 --2- 172.21.15.46:0/520769979 >> v2:172.21.15.154:3300/0 conn(0x7f2338159e60 0x7f233806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:04.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.243+0000 7f233cde3640 1 -- 172.21.15.46:0/520769979 >> v2:172.21.15.154:3300/0 conn(0x7f2338159e60 msgr2=0x7f233806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:04.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.243+0000 7f233cde3640 1 --2- 172.21.15.46:0/520769979 >> v2:172.21.15.154:3300/0 conn(0x7f2338159e60 0x7f233806f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:04.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.243+0000 7f233cde3640 1 -- 172.21.15.46:0/520769979 >> v1:172.21.15.154:6789/0 conn(0x7f23380bfec0 legacy=0x7f23380c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:04.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.243+0000 7f233cde3640 1 -- 172.21.15.46:0/520769979 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2338071850 con 0x7f233806a720 2023-10-30T14:12:04.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.243+0000 7f233cde3640 1 --2- 172.21.15.46:0/520769979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f233806a720 0x7f233806ab90 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f23380c8950 tx=0x7f233001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69feba507cfac1bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:04.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.243+0000 7f2336ffd640 1 -- 172.21.15.46:0/520769979 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23300af400 con 0x7f233806a720 2023-10-30T14:12:04.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.243+0000 7f2336ffd640 1 -- 172.21.15.46:0/520769979 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23300ab050 con 0x7f233806a720 2023-10-30T14:12:04.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.243+0000 7f2336ffd640 1 -- 172.21.15.46:0/520769979 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23300af780 con 0x7f233806a720 2023-10-30T14:12:04.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 -- 172.21.15.46:0/520769979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f233806a720 msgr2=0x7f233806ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:04.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 --2- 172.21.15.46:0/520769979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f233806a720 0x7f233806ab90 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f23380c8950 tx=0x7f233001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:12:04.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 -- 172.21.15.46:0/520769979 shutdown_connections 2023-10-30T14:12:04.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 --2- 172.21.15.46:0/520769979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f233806a720 0x7f233806ab90 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:04.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 --2- 172.21.15.46:0/520769979 >> v2:172.21.15.154:3300/0 conn(0x7f2338159e60 0x7f233806f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:04.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 -- 172.21.15.46:0/520769979 >> 172.21.15.46:0/520769979 conn(0x7f23380c7910 msgr2=0x7f23380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:04.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 -- 172.21.15.46:0/520769979 shutdown_connections 2023-10-30T14:12:04.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 -- 172.21.15.46:0/520769979 wait complete. 2023-10-30T14:12:04.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 Processor -- start 2023-10-30T14:12:04.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 -- start start 2023-10-30T14:12:04.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f233806a720 0x7f23380ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23380bfec0 0x7f233814b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2338159e60 0x7f233814b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2338151260 con 0x7f233806a720 2023-10-30T14:12:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23381f3100 con 0x7f23380bfec0 2023-10-30T14:12:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233e846640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23381f24f0 con 0x7f2338159e60 2023-10-30T14:12:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f2337fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f233806a720 0x7f23380ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f2337fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f233806a720 0x7f23380ae0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46952/0 (socket says 172.21.15.46:46952) 2023-10-30T14:12:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f2337fff640 1 -- 172.21.15.46:0/2134102852 learned_addr learned my addr 172.21.15.46:0/2134102852 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f233cde3640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2338159e60 0x7f233814b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.251+0000 7f23377fe640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23380bfec0 0x7f233814b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f2337fff640 1 -- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2338159e60 msgr2=0x7f233814b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f2337fff640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2338159e60 0x7f233814b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f2337fff640 1 -- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23380bfec0 msgr2=0x7f233814b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f2337fff640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23380bfec0 0x7f233814b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f2337fff640 1 -- 172.21.15.46:0/2134102852 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2330009b30 con 0x7f233806a720 2023-10-30T14:12:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f23377fe640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23380bfec0 0x7f233814b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:12:04.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f2337fff640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f233806a720 0x7f23380ae0a0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f232800e870 tx=0x7f232800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f2334ff9640 1 -- 172.21.15.46:0/2134102852 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328009510 con 0x7f233806a720 2023-10-30T14:12:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f2334ff9640 1 -- 172.21.15.46:0/2134102852 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232800eec0 con 0x7f233806a720 2023-10-30T14:12:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f2334ff9640 1 -- 172.21.15.46:0/2134102852 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232800cae0 con 0x7f233806a720 2023-10-30T14:12:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f233e846640 1 -- 172.21.15.46:0/2134102852 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23381f27d0 con 0x7f233806a720 2023-10-30T14:12:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f233e846640 1 -- 172.21.15.46:0/2134102852 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23381f2d90 con 0x7f233806a720 2023-10-30T14:12:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f233e846640 1 -- 172.21.15.46:0/2134102852 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23381f2f50 con 0x7f233806a720 2023-10-30T14:12:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f2334ff9640 1 -- 172.21.15.46:0/2134102852 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f232800cc40 con 0x7f233806a720 2023-10-30T14:12:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.255+0000 7f2334ff9640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23240664a0 0x7f2324068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.259+0000 7f2334ff9640 1 -- 172.21.15.46:0/2134102852 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f23280847d0 con 0x7f233806a720 2023-10-30T14:12:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.259+0000 7f23377fe640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23240664a0 0x7f2324068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.259+0000 7f23377fe640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23240664a0 0x7f2324068960 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f23381e9790 tx=0x7f232c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.263+0000 7f2334ff9640 1 -- 172.21.15.46:0/2134102852 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23280513f0 con 0x7f233806a720 2023-10-30T14:12:04.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 -- 172.21.15.46:0/2134102852 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f233804e170 con 0x7f233806a720 2023-10-30T14:12:04.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f2334ff9640 1 -- 172.21.15.46:0/2134102852 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f2328054a10 con 0x7f233806a720 2023-10-30T14:12:04.452 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:04.453 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8982,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":208}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:27.888611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:04.453 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 208 2023-10-30T14:12:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 -- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23240664a0 msgr2=0x7f2324068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23240664a0 0x7f2324068960 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f23381e9790 tx=0x7f232c009040 comp rx=0 tx=0).stop 2023-10-30T14:12:04.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 -- 172.21.15.46:0/2134102852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f233806a720 msgr2=0x7f23380ae0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:04.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f233806a720 0x7f23380ae0a0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f232800e870 tx=0x7f232800ec70 comp rx=0 tx=0).stop 2023-10-30T14:12:04.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 -- 172.21.15.46:0/2134102852 shutdown_connections 2023-10-30T14:12:04.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23240664a0 0x7f2324068960 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:04.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2338159e60 0x7f233814b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:04.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23380bfec0 0x7f233814b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:04.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 --2- 172.21.15.46:0/2134102852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f233806a720 0x7f23380ae0a0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:04.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 -- 172.21.15.46:0/2134102852 >> 172.21.15.46:0/2134102852 conn(0x7f23380c7910 msgr2=0x7f23380c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:04.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 -- 172.21.15.46:0/2134102852 shutdown_connections 2023-10-30T14:12:04.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:04.451+0000 7f233e846640 1 -- 172.21.15.46:0/2134102852 wait complete. 2023-10-30T14:12:04.474 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:12:06.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:06.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49f14f1640 1 Processor -- start 2023-10-30T14:12:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49f14f1640 1 -- start start 2023-10-30T14:12:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49f14f1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f49ec0c90d0 0x7f49ec0cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49f14f1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f49ec0cf4d0 0x7f49ec15d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49f14f1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ec15ce30 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49f14f1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ec159880 con 0x7f49ec0cf4d0 2023-10-30T14:12:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49f14f1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ec159ae0 con 0x7f49ec0c90d0 2023-10-30T14:12:06.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49eaffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f49ec0c8cf0 0x7f49ec0c9e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56940/0 (socket says 172.21.15.46:56940) 2023-10-30T14:12:06.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49eaffd640 1 -- 172.21.15.46:0/3202535256 learned_addr learned my addr 172.21.15.46:0/3202535256 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:06.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49eb7fe640 1 --2- 172.21.15.46:0/3202535256 >> v2:172.21.15.46:3300/0 conn(0x7f49ec0cf4d0 0x7f49ec15d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:06.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49e3fff640 1 --2- 172.21.15.46:0/3202535256 >> v2:172.21.15.154:3300/0 conn(0x7f49ec0c90d0 0x7f49ec0cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:06.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49eb7fe640 1 -- 172.21.15.46:0/3202535256 >> v2:172.21.15.154:3300/0 conn(0x7f49ec0c90d0 msgr2=0x7f49ec0cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:06.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49eb7fe640 1 --2- 172.21.15.46:0/3202535256 >> v2:172.21.15.154:3300/0 conn(0x7f49ec0c90d0 0x7f49ec0cef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:06.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49eb7fe640 1 -- 172.21.15.46:0/3202535256 >> v1:172.21.15.46:6789/0 conn(0x7f49ec0c8cf0 legacy=0x7f49ec0c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:06.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.631+0000 7f49eb7fe640 1 -- 172.21.15.46:0/3202535256 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec159dc0 con 0x7f49ec0cf4d0 2023-10-30T14:12:06.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49eb7fe640 1 --2- 172.21.15.46:0/3202535256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49ec0cf4d0 0x7f49ec15d2c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f49ec06cb60 tx=0x7f49e401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31b0bb11ccfb912 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:06.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49ea7fc640 1 -- 172.21.15.46:0/3202535256 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e40a59a0 con 0x7f49ec0cf4d0 2023-10-30T14:12:06.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49ea7fc640 1 -- 172.21.15.46:0/3202535256 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e40a5b00 con 0x7f49ec0cf4d0 2023-10-30T14:12:06.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49ea7fc640 1 -- 172.21.15.46:0/3202535256 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e40a5e90 con 0x7f49ec0cf4d0 2023-10-30T14:12:06.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49f14f1640 1 -- 172.21.15.46:0/3202535256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49ec0cf4d0 msgr2=0x7f49ec15d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:06.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49f14f1640 1 --2- 172.21.15.46:0/3202535256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49ec0cf4d0 0x7f49ec15d2c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f49ec06cb60 tx=0x7f49e401c8e0 comp rx=0 tx=0).stop 2023-10-30T14:12:06.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49f14f1640 1 -- 172.21.15.46:0/3202535256 shutdown_connections 2023-10-30T14:12:06.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49f14f1640 1 --2- 172.21.15.46:0/3202535256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49ec0cf4d0 0x7f49ec15d2c0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:06.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49f14f1640 1 --2- 172.21.15.46:0/3202535256 >> v2:172.21.15.154:3300/0 conn(0x7f49ec0c90d0 0x7f49ec0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:06.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49f14f1640 1 -- 172.21.15.46:0/3202535256 >> 172.21.15.46:0/3202535256 conn(0x7f49ec06be40 msgr2=0x7f49ec06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:06.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49f14f1640 1 -- 172.21.15.46:0/3202535256 shutdown_connections 2023-10-30T14:12:06.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49f14f1640 1 -- 172.21.15.46:0/3202535256 wait complete. 2023-10-30T14:12:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49f14f1640 1 Processor -- start 2023-10-30T14:12:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.635+0000 7f49f14f1640 1 -- start start 2023-10-30T14:12:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49f14f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49ec0c8cf0 0x7f49ec156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:06.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49f14f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49ec0c90d0 0x7f49ec1f1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:06.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49f14f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49ec0cf4d0 0x7f49ec1541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:06.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49f14f1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec0c36a0 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49f14f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec0cd8e0 con 0x7f49ec0cf4d0 2023-10-30T14:12:06.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49f14f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec0c2c50 con 0x7f49ec0c90d0 2023-10-30T14:12:06.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49eaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49ec0c8cf0 0x7f49ec156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:06.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49eaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49ec0c8cf0 0x7f49ec156e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35720/0 (socket says 172.21.15.46:35720) 2023-10-30T14:12:06.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49eaffd640 1 -- 172.21.15.46:0/3568250173 learned_addr learned my addr 172.21.15.46:0/3568250173 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:06.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49e3fff640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49ec0c90d0 0x7f49ec1f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:06.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49eb7fe640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49ec0cf4d0 0x7f49ec1541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:06.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49eaffd640 1 -- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49ec0c90d0 msgr2=0x7f49ec1f1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:06.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49eaffd640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49ec0c90d0 0x7f49ec1f1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:06.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49eaffd640 1 -- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49ec0cf4d0 msgr2=0x7f49ec1541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:06.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49eaffd640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49ec0cf4d0 0x7f49ec1541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:06.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49eaffd640 1 -- 172.21.15.46:0/3568250173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e4009a70 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49e3fff640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49ec0c90d0 0x7f49ec1f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:12:06.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49eb7fe640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49ec0cf4d0 0x7f49ec1541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:12:06.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49eaffd640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49ec0c8cf0 0x7f49ec156e30 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00c370 tx=0x7f49dc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:06.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49e37fe640 1 -- 172.21.15.46:0/3568250173 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49dc005cf0 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49e37fe640 1 -- 172.21.15.46:0/3568250173 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49dc005e50 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49e37fe640 1 -- 172.21.15.46:0/3568250173 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49dc0072f0 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49f14f1640 1 -- 172.21.15.46:0/3568250173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ec1f5a60 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49f14f1640 1 -- 172.21.15.46:0/3568250173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ec1557f0 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.639+0000 7f49f14f1640 1 -- 172.21.15.46:0/3568250173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ec04e170 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.643+0000 7f49e37fe640 1 -- 172.21.15.46:0/3568250173 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f49dc007450 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.643+0000 7f49e37fe640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49cc0664d0 0x7f49cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.643+0000 7f49e37fe640 1 -- 172.21.15.46:0/3568250173 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f49dc083cf0 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.643+0000 7f49e3fff640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49cc0664d0 0x7f49cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.643+0000 7f49e3fff640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49cc0664d0 0x7f49cc068990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0c8a10 tx=0x7f49d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:06.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.647+0000 7f49e37fe640 1 -- 172.21.15.46:0/3568250173 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f49dc050860 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.835+0000 7f49f14f1640 1 -- 172.21.15.46:0/3568250173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49ec0ca9d0 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49e37fe640 1 -- 172.21.15.46:0/3568250173 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f49dc017070 con 0x7f49ec0c8cf0 2023-10-30T14:12:06.840 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:06.840 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8982,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":208}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:27.888611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:06.840 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 208 2023-10-30T14:12:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49f14f1640 1 -- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49cc0664d0 msgr2=0x7f49cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:06.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49f14f1640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49cc0664d0 0x7f49cc068990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0c8a10 tx=0x7f49d4008040 comp rx=0 tx=0).stop 2023-10-30T14:12:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49f14f1640 1 -- 172.21.15.46:0/3568250173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49ec0c8cf0 msgr2=0x7f49ec156e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49f14f1640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49ec0c8cf0 0x7f49ec156e30 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00c370 tx=0x7f49dc00c830 comp rx=0 tx=0).stop 2023-10-30T14:12:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49f14f1640 1 -- 172.21.15.46:0/3568250173 shutdown_connections 2023-10-30T14:12:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49f14f1640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f49cc0664d0 0x7f49cc068990 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49f14f1640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49ec0cf4d0 0x7f49ec1541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49f14f1640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49ec0c90d0 0x7f49ec1f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49f14f1640 1 --2- 172.21.15.46:0/3568250173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49ec0c8cf0 0x7f49ec156e30 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49f14f1640 1 -- 172.21.15.46:0/3568250173 >> 172.21.15.46:0/3568250173 conn(0x7f49ec06be40 msgr2=0x7f49ec0b3570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49f14f1640 1 -- 172.21.15.46:0/3568250173 shutdown_connections 2023-10-30T14:12:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:06.839+0000 7f49f14f1640 1 -- 172.21.15.46:0/3568250173 wait complete. 2023-10-30T14:12:06.861 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:12:08.862 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f948d89c640 1 Processor -- start 2023-10-30T14:12:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f948d89c640 1 -- start start 2023-10-30T14:12:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f948d89c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f948806f7a0 0x7f948806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f948d89c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94880c6bd0 0x7f9488069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f948d89c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f948806a3d0 con 0x7f9488071110 2023-10-30T14:12:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f948d89c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94880700c0 con 0x7f94880c6bd0 2023-10-30T14:12:09.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f948d89c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9488066cf0 con 0x7f948806f7a0 2023-10-30T14:12:09.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f947ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94880c6bd0 0x7f9488069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:09.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f94877fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9488071110 0x7f9488158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56950/0 (socket says 172.21.15.46:56950) 2023-10-30T14:12:09.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f94877fe640 1 -- 172.21.15.46:0/352886265 learned_addr learned my addr 172.21.15.46:0/352886265 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:09.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f9486ffd640 1 --2- 172.21.15.46:0/352886265 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f948806f7a0 0x7f948806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:09.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f947ffff640 1 -- 172.21.15.46:0/352886265 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f948806f7a0 msgr2=0x7f948806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:09.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f947ffff640 1 --2- 172.21.15.46:0/352886265 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f948806f7a0 0x7f948806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:09.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f947ffff640 1 -- 172.21.15.46:0/352886265 >> v1:172.21.15.46:6789/0 conn(0x7f9488071110 legacy=0x7f9488158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:09.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f947ffff640 1 -- 172.21.15.46:0/352886265 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94880c70d0 con 0x7f94880c6bd0 2023-10-30T14:12:09.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.019+0000 7f94867fc640 1 -- 172.21.15.46:0/352886265 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256766908 0 0) 0x7f948806a3d0 con 0x7f9488071110 2023-10-30T14:12:09.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f947ffff640 1 --2- 172.21.15.46:0/352886265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94880c6bd0 0x7f9488069e90 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f94700097c0 tx=0x7f947001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771e66ca7d651877 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:09.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f94867fc640 1 -- 172.21.15.46:0/352886265 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94700a8970 con 0x7f94880c6bd0 2023-10-30T14:12:09.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f94867fc640 1 -- 172.21.15.46:0/352886265 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94700a8ad0 con 0x7f94880c6bd0 2023-10-30T14:12:09.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f94867fc640 1 -- 172.21.15.46:0/352886265 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94700a8e60 con 0x7f94880c6bd0 2023-10-30T14:12:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f948d89c640 1 -- 172.21.15.46:0/352886265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94880c6bd0 msgr2=0x7f9488069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f948d89c640 1 --2- 172.21.15.46:0/352886265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94880c6bd0 0x7f9488069e90 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f94700097c0 tx=0x7f947001ec70 comp rx=0 tx=0).stop 2023-10-30T14:12:09.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f948d89c640 1 -- 172.21.15.46:0/352886265 shutdown_connections 2023-10-30T14:12:09.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f948d89c640 1 --2- 172.21.15.46:0/352886265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94880c6bd0 0x7f9488069e90 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:09.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f948d89c640 1 --2- 172.21.15.46:0/352886265 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f948806f7a0 0x7f948806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:09.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f948d89c640 1 -- 172.21.15.46:0/352886265 >> 172.21.15.46:0/352886265 conn(0x7f94880c7910 msgr2=0x7f94880c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:09.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f948d89c640 1 -- 172.21.15.46:0/352886265 shutdown_connections 2023-10-30T14:12:09.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f948d89c640 1 -- 172.21.15.46:0/352886265 wait complete. 2023-10-30T14:12:09.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.023+0000 7f948d89c640 1 Processor -- start 2023-10-30T14:12:09.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f948d89c640 1 -- start start 2023-10-30T14:12:09.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f948d89c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f948806f7a0 0x7f94881592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:09.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f948d89c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9488071110 0x7f94880c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:09.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f948d89c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94880c6bd0 0x7f94880c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:09.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f948d89c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94881ed6d0 con 0x7f94880c6bd0 2023-10-30T14:12:09.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f948d89c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94880cab10 con 0x7f948806f7a0 2023-10-30T14:12:09.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f948d89c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94881ec800 con 0x7f9488071110 2023-10-30T14:12:09.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f94877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94880c6bd0 0x7f94880c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:09.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f94877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94880c6bd0 0x7f94880c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35728/0 (socket says 172.21.15.46:35728) 2023-10-30T14:12:09.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f94877fe640 1 -- 172.21.15.46:0/2974377309 learned_addr learned my addr 172.21.15.46:0/2974377309 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:09.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f947ffff640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9488071110 0x7f94880c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:09.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f9486ffd640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f948806f7a0 0x7f94881592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:09.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f94877fe640 1 -- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9488071110 msgr2=0x7f94880c3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:09.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f94877fe640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9488071110 0x7f94880c3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:09.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f94877fe640 1 -- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f948806f7a0 msgr2=0x7f94881592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:09.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f94877fe640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f948806f7a0 0x7f94881592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:09.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f94877fe640 1 -- 172.21.15.46:0/2974377309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9470009420 con 0x7f94880c6bd0 2023-10-30T14:12:09.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f9486ffd640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f948806f7a0 0x7f94881592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:12:09.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f94877fe640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94880c6bd0 0x7f94880c9ec0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f948000c780 tx=0x7f948000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:09.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f9484ff9640 1 -- 172.21.15.46:0/2974377309 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948000ce90 con 0x7f94880c6bd0 2023-10-30T14:12:09.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f9484ff9640 1 -- 172.21.15.46:0/2974377309 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948000a640 con 0x7f94880c6bd0 2023-10-30T14:12:09.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f9484ff9640 1 -- 172.21.15.46:0/2974377309 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948000a9d0 con 0x7f94880c6bd0 2023-10-30T14:12:09.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f948d89c640 1 -- 172.21.15.46:0/2974377309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94881ecac0 con 0x7f94880c6bd0 2023-10-30T14:12:09.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f948d89c640 1 -- 172.21.15.46:0/2974377309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94881ed080 con 0x7f94880c6bd0 2023-10-30T14:12:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.027+0000 7f948d89c640 1 -- 172.21.15.46:0/2974377309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9488070330 con 0x7f94880c6bd0 2023-10-30T14:12:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.031+0000 7f9484ff9640 1 -- 172.21.15.46:0/2974377309 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f948000ab30 con 0x7f94880c6bd0 2023-10-30T14:12:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.031+0000 7f9484ff9640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94680664a0 0x7f9468068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:09.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.031+0000 7f9484ff9640 1 -- 172.21.15.46:0/2974377309 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7f9480083ca0 con 0x7f94880c6bd0 2023-10-30T14:12:09.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.031+0000 7f9486ffd640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94680664a0 0x7f9468068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:09.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.031+0000 7f9486ffd640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94680664a0 0x7f9468068960 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9478009b10 tx=0x7f9478008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:09.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.035+0000 7f9484ff9640 1 -- 172.21.15.46:0/2974377309 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9488070330 con 0x7f94880c6bd0 2023-10-30T14:12:09.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 -- 172.21.15.46:0/2974377309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f948815aca0 con 0x7f94880c6bd0 2023-10-30T14:12:09.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f9484ff9640 1 -- 172.21.15.46:0/2974377309 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+6520 (secure 0 0 0) 0x7f9480018030 con 0x7f94880c6bd0 2023-10-30T14:12:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:09.229 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8982,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":208}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:27.888611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 208 2023-10-30T14:12:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 -- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94680664a0 msgr2=0x7f9468068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94680664a0 0x7f9468068960 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9478009b10 tx=0x7f9478008040 comp rx=0 tx=0).stop 2023-10-30T14:12:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 -- 172.21.15.46:0/2974377309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94880c6bd0 msgr2=0x7f94880c9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94880c6bd0 0x7f94880c9ec0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f948000c780 tx=0x7f948000cc40 comp rx=0 tx=0).stop 2023-10-30T14:12:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 -- 172.21.15.46:0/2974377309 shutdown_connections 2023-10-30T14:12:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94680664a0 0x7f9468068960 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94880c6bd0 0x7f94880c9ec0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9488071110 0x7f94880c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 --2- 172.21.15.46:0/2974377309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f948806f7a0 0x7f94881592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 -- 172.21.15.46:0/2974377309 >> 172.21.15.46:0/2974377309 conn(0x7f94880c7910 msgr2=0x7f94880c5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:09.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 -- 172.21.15.46:0/2974377309 shutdown_connections 2023-10-30T14:12:09.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:09.227+0000 7f948d89c640 1 -- 172.21.15.46:0/2974377309 wait complete. 2023-10-30T14:12:09.250 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:12:11.251 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.403+0000 7fd8e7b94640 1 Processor -- start 2023-10-30T14:12:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.403+0000 7fd8e7b94640 1 -- start start 2023-10-30T14:12:11.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.403+0000 7fd8e7b94640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8e006f7a0 0x7fd8e006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:11.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.403+0000 7fd8e7b94640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd8e0071110 0x7fd8e0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:11.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.403+0000 7fd8e7b94640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e0066cf0 con 0x7fd8e00c6bd0 2023-10-30T14:12:11.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.403+0000 7fd8e7b94640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e00700c0 con 0x7fd8e006f7a0 2023-10-30T14:12:11.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.403+0000 7fd8e7b94640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e00c70d0 con 0x7fd8e0071110 2023-10-30T14:12:11.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.403+0000 7fd8e5930640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8e006f7a0 0x7fd8e006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:11.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.403+0000 7fd8e5930640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8e006f7a0 0x7fd8e006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35734/0 (socket says 172.21.15.46:35734) 2023-10-30T14:12:11.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.403+0000 7fd8e5930640 1 -- 172.21.15.46:0/2245512365 learned_addr learned my addr 172.21.15.46:0/2245512365 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:11.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.407+0000 7fd8e6131640 1 --2- 172.21.15.46:0/2245512365 >> v2:172.21.15.154:3300/0 conn(0x7fd8e0071110 0x7fd8e0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:11.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.407+0000 7fd8e5930640 1 -- 172.21.15.46:0/2245512365 >> v2:172.21.15.154:3300/0 conn(0x7fd8e0071110 msgr2=0x7fd8e0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:11.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.407+0000 7fd8e5930640 1 --2- 172.21.15.46:0/2245512365 >> v2:172.21.15.154:3300/0 conn(0x7fd8e0071110 0x7fd8e0158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:11.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.407+0000 7fd8e5930640 1 -- 172.21.15.46:0/2245512365 >> v1:172.21.15.154:6789/0 conn(0x7fd8e00c6bd0 legacy=0x7fd8e0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:11.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.407+0000 7fd8e5930640 1 -- 172.21.15.46:0/2245512365 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e00c7390 con 0x7fd8e006f7a0 2023-10-30T14:12:11.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.407+0000 7fd8e5930640 1 --2- 172.21.15.46:0/2245512365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e006f7a0 0x7fd8e006fb80 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fd8e006cdd0 tx=0x7fd8dc01ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becb43812a4f98d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:11.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.407+0000 7fd8e492e640 1 -- 172.21.15.46:0/2245512365 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc0a89b0 con 0x7fd8e006f7a0 2023-10-30T14:12:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.407+0000 7fd8e492e640 1 -- 172.21.15.46:0/2245512365 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc0a8b10 con 0x7fd8e006f7a0 2023-10-30T14:12:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.407+0000 7fd8e492e640 1 -- 172.21.15.46:0/2245512365 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc0a8ea0 con 0x7fd8e006f7a0 2023-10-30T14:12:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.407+0000 7fd8e7b94640 1 -- 172.21.15.46:0/2245512365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e006f7a0 msgr2=0x7fd8e006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.407+0000 7fd8e7b94640 1 --2- 172.21.15.46:0/2245512365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e006f7a0 0x7fd8e006fb80 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fd8e006cdd0 tx=0x7fd8dc01ecb0 comp rx=0 tx=0).stop 2023-10-30T14:12:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 -- 172.21.15.46:0/2245512365 shutdown_connections 2023-10-30T14:12:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 --2- 172.21.15.46:0/2245512365 >> v2:172.21.15.154:3300/0 conn(0x7fd8e0071110 0x7fd8e0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 --2- 172.21.15.46:0/2245512365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e006f7a0 0x7fd8e006fb80 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 -- 172.21.15.46:0/2245512365 >> 172.21.15.46:0/2245512365 conn(0x7fd8e00c7910 msgr2=0x7fd8e00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 -- 172.21.15.46:0/2245512365 shutdown_connections 2023-10-30T14:12:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 -- 172.21.15.46:0/2245512365 wait complete. 2023-10-30T14:12:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 Processor -- start 2023-10-30T14:12:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 -- start start 2023-10-30T14:12:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e006f7a0 0x7fd8e00c0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8e0071110 0x7fd8e01efa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8e00c6bd0 0x7fd8e00c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:11.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e01503e0 con 0x7fd8e006f7a0 2023-10-30T14:12:11.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e01ef5d0 con 0x7fd8e0071110 2023-10-30T14:12:11.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e7b94640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e00c5ff0 con 0x7fd8e00c6bd0 2023-10-30T14:12:11.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e5930640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e006f7a0 0x7fd8e00c0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:11.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e5930640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e006f7a0 0x7fd8e00c0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35740/0 (socket says 172.21.15.46:35740) 2023-10-30T14:12:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e5930640 1 -- 172.21.15.46:0/1058071283 learned_addr learned my addr 172.21.15.46:0/1058071283 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e512f640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8e0071110 0x7fd8e01efa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e6131640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8e00c6bd0 0x7fd8e00c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e5930640 1 -- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8e00c6bd0 msgr2=0x7fd8e00c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e5930640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8e00c6bd0 0x7fd8e00c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e5930640 1 -- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8e0071110 msgr2=0x7fd8e01efa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e5930640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8e0071110 0x7fd8e01efa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e5930640 1 -- 172.21.15.46:0/1058071283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc0094d0 con 0x7fd8e006f7a0 2023-10-30T14:12:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e512f640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8e0071110 0x7fd8e01efa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:12:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.411+0000 7fd8e5930640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e006f7a0 0x7fd8e00c0650 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc0b3040 tx=0x7fd8dc01ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.415+0000 7fd8d67fc640 1 -- 172.21.15.46:0/1058071283 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc0ba600 con 0x7fd8e006f7a0 2023-10-30T14:12:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.415+0000 7fd8d67fc640 1 -- 172.21.15.46:0/1058071283 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc0ba760 con 0x7fd8e006f7a0 2023-10-30T14:12:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.415+0000 7fd8e7b94640 1 -- 172.21.15.46:0/1058071283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e01550f0 con 0x7fd8e006f7a0 2023-10-30T14:12:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.415+0000 7fd8d67fc640 1 -- 172.21.15.46:0/1058071283 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc0bab10 con 0x7fd8e006f7a0 2023-10-30T14:12:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.415+0000 7fd8e7b94640 1 -- 172.21.15.46:0/1058071283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e01f3260 con 0x7fd8e006f7a0 2023-10-30T14:12:11.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.415+0000 7fd8e7b94640 1 -- 172.21.15.46:0/1058071283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e0070330 con 0x7fd8e006f7a0 2023-10-30T14:12:11.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.415+0000 7fd8d67fc640 1 -- 172.21.15.46:0/1058071283 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd8dc0bac70 con 0x7fd8e006f7a0 2023-10-30T14:12:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.415+0000 7fd8d67fc640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8c40664a0 0x7fd8c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.415+0000 7fd8d67fc640 1 -- 172.21.15.46:0/1058071283 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7fd8dc122c90 con 0x7fd8e006f7a0 2023-10-30T14:12:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.415+0000 7fd8e512f640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8c40664a0 0x7fd8c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.419+0000 7fd8e512f640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8c40664a0 0x7fd8c4068960 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00c98b0 tx=0x7fd8d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.419+0000 7fd8d67fc640 1 -- 172.21.15.46:0/1058071283 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd8e0070330 con 0x7fd8e006f7a0 2023-10-30T14:12:11.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.607+0000 7fd8e7b94640 1 -- 172.21.15.46:0/1058071283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8e0070330 con 0x7fd8e006f7a0 2023-10-30T14:12:11.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.607+0000 7fd8d67fc640 1 -- 172.21.15.46:0/1058071283 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+6520 (secure 0 0 0) 0x7fd8e0070330 con 0x7fd8e006f7a0 2023-10-30T14:12:11.610 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:11.613 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8982,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":208}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:27.888611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 208 2023-10-30T14:12:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.611+0000 7fd8e7b94640 1 -- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8c40664a0 msgr2=0x7fd8c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.611+0000 7fd8e7b94640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8c40664a0 0x7fd8c4068960 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00c98b0 tx=0x7fd8d0009040 comp rx=0 tx=0).stop 2023-10-30T14:12:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.611+0000 7fd8e7b94640 1 -- 172.21.15.46:0/1058071283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e006f7a0 msgr2=0x7fd8e00c0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.611+0000 7fd8e7b94640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e006f7a0 0x7fd8e00c0650 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc0b3040 tx=0x7fd8dc01ba50 comp rx=0 tx=0).stop 2023-10-30T14:12:11.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.611+0000 7fd8e7b94640 1 -- 172.21.15.46:0/1058071283 shutdown_connections 2023-10-30T14:12:11.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.611+0000 7fd8e7b94640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd8c40664a0 0x7fd8c4068960 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:11.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.611+0000 7fd8e7b94640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8e00c6bd0 0x7fd8e00c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:11.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.611+0000 7fd8e7b94640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8e0071110 0x7fd8e01efa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:11.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.611+0000 7fd8e7b94640 1 --2- 172.21.15.46:0/1058071283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e006f7a0 0x7fd8e00c0650 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:11.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.611+0000 7fd8e7b94640 1 -- 172.21.15.46:0/1058071283 >> 172.21.15.46:0/1058071283 conn(0x7fd8e00c7910 msgr2=0x7fd8e006a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:11.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.611+0000 7fd8e7b94640 1 -- 172.21.15.46:0/1058071283 shutdown_connections 2023-10-30T14:12:11.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:11.611+0000 7fd8e7b94640 1 -- 172.21.15.46:0/1058071283 wait complete. 2023-10-30T14:12:11.631 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:12:13.633 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:13.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.787+0000 7fbd7ae9e640 1 Processor -- start 2023-10-30T14:12:13.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.787+0000 7fbd7ae9e640 1 -- start start 2023-10-30T14:12:13.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.787+0000 7fbd7ae9e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd740c6bd0 0x7fbd7406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:13.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.787+0000 7fbd7ae9e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbd74071110 0x7fbd74158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.787+0000 7fbd7ae9e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd74066cf0 con 0x7fbd7406f7a0 2023-10-30T14:12:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.787+0000 7fbd7ae9e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd740700c0 con 0x7fbd74071110 2023-10-30T14:12:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.787+0000 7fbd7ae9e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd740c70d0 con 0x7fbd740c6bd0 2023-10-30T14:12:13.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.787+0000 7fbd78c3a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbd7406f7a0 0x7fbd74069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56964/0 (socket says 172.21.15.46:56964) 2023-10-30T14:12:13.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.787+0000 7fbd78c3a640 1 -- 172.21.15.46:0/2600909974 learned_addr learned my addr 172.21.15.46:0/2600909974 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:13.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.787+0000 7fbd73fff640 1 --2- 172.21.15.46:0/2600909974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd740c6bd0 0x7fbd7406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:13.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.787+0000 7fbd7943b640 1 --2- 172.21.15.46:0/2600909974 >> v2:172.21.15.154:3300/0 conn(0x7fbd74071110 0x7fbd74158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:13.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.791+0000 7fbd737fe640 1 -- 172.21.15.46:0/2600909974 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604325925 0 0) 0x7fbd74066cf0 con 0x7fbd7406f7a0 2023-10-30T14:12:13.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.791+0000 7fbd737fe640 1 -- 172.21.15.46:0/2600909974 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd64003530 con 0x7fbd7406f7a0 2023-10-30T14:12:13.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.791+0000 7fbd7943b640 1 -- 172.21.15.46:0/2600909974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd740c6bd0 msgr2=0x7fbd7406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:13.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.791+0000 7fbd7943b640 1 --2- 172.21.15.46:0/2600909974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd740c6bd0 0x7fbd7406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:13.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.791+0000 7fbd7943b640 1 -- 172.21.15.46:0/2600909974 >> v1:172.21.15.46:6789/0 conn(0x7fbd7406f7a0 legacy=0x7fbd74069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:13.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.791+0000 7fbd737fe640 1 -- 172.21.15.46:0/2600909974 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1850623102 0 0) 0x7fbd64003530 con 0x7fbd7406f7a0 2023-10-30T14:12:13.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.791+0000 7fbd7943b640 1 -- 172.21.15.46:0/2600909974 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd740c7390 con 0x7fbd74071110 2023-10-30T14:12:13.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.791+0000 7fbd7943b640 1 --2- 172.21.15.46:0/2600909974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd74071110 0x7fbd74158110 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c009d90 tx=0x7fbd6c01f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2614441b60ef9c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:13.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.791+0000 7fbd737fe640 1 -- 172.21.15.46:0/2600909974 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd68001680 con 0x7fbd74071110 2023-10-30T14:12:13.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd737fe640 1 -- 172.21.15.46:0/2600909974 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6c0a8bb0 con 0x7fbd74071110 2023-10-30T14:12:13.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd737fe640 1 -- 172.21.15.46:0/2600909974 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c0b0440 con 0x7fbd74071110 2023-10-30T14:12:13.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/2600909974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd74071110 msgr2=0x7fbd74158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:13.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 --2- 172.21.15.46:0/2600909974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd74071110 0x7fbd74158110 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c009d90 tx=0x7fbd6c01f060 comp rx=0 tx=0).stop 2023-10-30T14:12:13.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/2600909974 shutdown_connections 2023-10-30T14:12:13.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 --2- 172.21.15.46:0/2600909974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd74071110 0x7fbd74158110 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:13.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 --2- 172.21.15.46:0/2600909974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd740c6bd0 0x7fbd7406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/2600909974 >> 172.21.15.46:0/2600909974 conn(0x7fbd740c7910 msgr2=0x7fbd740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/2600909974 shutdown_connections 2023-10-30T14:12:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/2600909974 wait complete. 2023-10-30T14:12:13.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 Processor -- start 2023-10-30T14:12:13.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 -- start start 2023-10-30T14:12:13.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd7406f7a0 0x7fbd741eed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:13.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd74071110 0x7fbd741f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:13.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd740c6bd0 0x7fbd7415b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:13.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd741f3280 con 0x7fbd74071110 2023-10-30T14:12:13.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7415ae50 con 0x7fbd7406f7a0 2023-10-30T14:12:13.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd7ae9e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7415b820 con 0x7fbd740c6bd0 2023-10-30T14:12:13.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd73fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd74071110 0x7fbd741f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:13.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd73fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd74071110 0x7fbd741f2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35756/0 (socket says 172.21.15.46:35756) 2023-10-30T14:12:13.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.795+0000 7fbd73fff640 1 -- 172.21.15.46:0/3516997394 learned_addr learned my addr 172.21.15.46:0/3516997394 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:13.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd78c3a640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd7406f7a0 0x7fbd741eed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:13.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd7943b640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd740c6bd0 0x7fbd7415b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:13.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd73fff640 1 -- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd740c6bd0 msgr2=0x7fbd7415b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:13.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd73fff640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd740c6bd0 0x7fbd7415b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:13.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd73fff640 1 -- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd7406f7a0 msgr2=0x7fbd741eed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:13.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd73fff640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd7406f7a0 0x7fbd741eed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:13.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd73fff640 1 -- 172.21.15.46:0/3516997394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6c009a70 con 0x7fbd74071110 2023-10-30T14:12:13.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd73fff640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd74071110 0x7fbd741f2350 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fbd6000e930 tx=0x7fbd6000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:13.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd717fa640 1 -- 172.21.15.46:0/3516997394 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd60009690 con 0x7fbd74071110 2023-10-30T14:12:13.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd717fa640 1 -- 172.21.15.46:0/3516997394 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6000ca00 con 0x7fbd74071110 2023-10-30T14:12:13.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd717fa640 1 -- 172.21.15.46:0/3516997394 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6000cd70 con 0x7fbd74071110 2023-10-30T14:12:13.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/3516997394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd741f29e0 con 0x7fbd74071110 2023-10-30T14:12:13.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/3516997394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd741f2fa0 con 0x7fbd74071110 2023-10-30T14:12:13.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/3516997394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd7415afe0 con 0x7fbd74071110 2023-10-30T14:12:13.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd717fa640 1 -- 172.21.15.46:0/3516997394 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbd60013470 con 0x7fbd74071110 2023-10-30T14:12:13.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.799+0000 7fbd717fa640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbd580664d0 0x7fbd58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:13.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.803+0000 7fbd717fa640 1 -- 172.21.15.46:0/3516997394 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8004+0+0 (secure 0 0 0) 0x7fbd60084900 con 0x7fbd74071110 2023-10-30T14:12:13.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.803+0000 7fbd78c3a640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbd580664d0 0x7fbd58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:13.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.803+0000 7fbd78c3a640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbd580664d0 0x7fbd58068990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fbd68001ec0 tx=0x7fbd680019c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:13.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.803+0000 7fbd717fa640 1 -- 172.21.15.46:0/3516997394 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbd600515f0 con 0x7fbd74071110 2023-10-30T14:12:13.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.995+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/3516997394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd7404e170 con 0x7fbd74071110 2023-10-30T14:12:13.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.995+0000 7fbd717fa640 1 -- 172.21.15.46:0/3516997394 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+6520 (secure 0 0 0) 0x7fbd60021090 con 0x7fbd74071110 2023-10-30T14:12:13.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:13.999 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8982,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":208}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:10:27.888611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8655},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8655":{"gid":8655,"name":"e","rank":1,"incarnation":196,"state":"up:active","state_seq":24,"addr":"172.21.15.46:6837/3044281789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":3044281789},{"type":"v1","addr":"172.21.15.46:6837","nonce":3044281789}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 208 2023-10-30T14:12:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.999+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbd580664d0 msgr2=0x7fbd58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:14.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.999+0000 7fbd7ae9e640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbd580664d0 0x7fbd58068990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fbd68001ec0 tx=0x7fbd680019c0 comp rx=0 tx=0).stop 2023-10-30T14:12:14.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.999+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/3516997394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd74071110 msgr2=0x7fbd741f2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:14.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.999+0000 7fbd7ae9e640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd74071110 0x7fbd741f2350 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fbd6000e930 tx=0x7fbd6000edf0 comp rx=0 tx=0).stop 2023-10-30T14:12:14.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.999+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/3516997394 shutdown_connections 2023-10-30T14:12:14.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.999+0000 7fbd7ae9e640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbd580664d0 0x7fbd58068990 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:14.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.999+0000 7fbd7ae9e640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd740c6bd0 0x7fbd7415b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:14.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.999+0000 7fbd7ae9e640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd74071110 0x7fbd741f2350 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:14.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.999+0000 7fbd7ae9e640 1 --2- 172.21.15.46:0/3516997394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd7406f7a0 0x7fbd741eed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:14.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.999+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/3516997394 >> 172.21.15.46:0/3516997394 conn(0x7fbd740c7910 msgr2=0x7fbd740b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:14.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.999+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/3516997394 shutdown_connections 2023-10-30T14:12:14.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:13.999+0000 7fbd7ae9e640 1 -- 172.21.15.46:0/3516997394 wait complete. 2023-10-30T14:12:14.020 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:12:16.021 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:16.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f93a7640 1 Processor -- start 2023-10-30T14:12:16.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f93a7640 1 -- start start 2023-10-30T14:12:16.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f93a7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f97f41538f0 0x7f97f4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:16.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f93a7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97f415cd50 0x7f97f40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:16.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f93a7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f415d300 con 0x7f97f40cefa0 2023-10-30T14:12:16.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f93a7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f415d5e0 con 0x7f97f41538f0 2023-10-30T14:12:16.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f93a7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f415d8c0 con 0x7f97f415cd50 2023-10-30T14:12:16.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f8ba6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f97f40cefa0 0x7f97f40cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56970/0 (socket says 172.21.15.46:56970) 2023-10-30T14:12:16.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f8ba6640 1 -- 172.21.15.46:0/719277363 learned_addr learned my addr 172.21.15.46:0/719277363 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:16.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f37fe640 1 --2- 172.21.15.46:0/719277363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97f415cd50 0x7f97f40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:16.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f3fff640 1 --2- 172.21.15.46:0/719277363 >> v2:172.21.15.154:3300/0 conn(0x7f97f41538f0 0x7f97f4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:16.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f2ffd640 1 -- 172.21.15.46:0/719277363 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400450031 0 0) 0x7f97f415d300 con 0x7f97f40cefa0 2023-10-30T14:12:16.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f2ffd640 1 -- 172.21.15.46:0/719277363 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97dc003530 con 0x7f97f40cefa0 2023-10-30T14:12:16.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f37fe640 1 -- 172.21.15.46:0/719277363 >> v2:172.21.15.154:3300/0 conn(0x7f97f41538f0 msgr2=0x7f97f4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f37fe640 1 --2- 172.21.15.46:0/719277363 >> v2:172.21.15.154:3300/0 conn(0x7f97f41538f0 0x7f97f4159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f37fe640 1 -- 172.21.15.46:0/719277363 >> v1:172.21.15.46:6789/0 conn(0x7f97f40cefa0 legacy=0x7f97f40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f37fe640 1 -- 172.21.15.46:0/719277363 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f415dba0 con 0x7f97f415cd50 2023-10-30T14:12:16.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.175+0000 7f97f3fff640 1 --2- 172.21.15.46:0/719277363 >> v2:172.21.15.154:3300/0 conn(0x7f97f41538f0 0x7f97f4159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:12:16.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.179+0000 7f97f37fe640 1 --2- 172.21.15.46:0/719277363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97f415cd50 0x7f97f40c9f90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f97e80097c0 tx=0x7f97e801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=905d69c008203120 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.179+0000 7f97f2ffd640 1 -- 172.21.15.46:0/719277363 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e80a8970 con 0x7f97f415cd50 2023-10-30T14:12:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.179+0000 7f97f2ffd640 1 -- 172.21.15.46:0/719277363 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e80a8ad0 con 0x7f97f415cd50 2023-10-30T14:12:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.179+0000 7f97f2ffd640 1 -- 172.21.15.46:0/719277363 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e80a8e40 con 0x7f97f415cd50 2023-10-30T14:12:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.179+0000 7f97f93a7640 1 -- 172.21.15.46:0/719277363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97f415cd50 msgr2=0x7f97f40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:16.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.179+0000 7f97f93a7640 1 --2- 172.21.15.46:0/719277363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97f415cd50 0x7f97f40c9f90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f97e80097c0 tx=0x7f97e801ec70 comp rx=0 tx=0).stop 2023-10-30T14:12:16.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.179+0000 7f97f93a7640 1 -- 172.21.15.46:0/719277363 shutdown_connections 2023-10-30T14:12:16.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.179+0000 7f97f93a7640 1 --2- 172.21.15.46:0/719277363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97f415cd50 0x7f97f40c9f90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.179+0000 7f97f93a7640 1 --2- 172.21.15.46:0/719277363 >> v2:172.21.15.154:3300/0 conn(0x7f97f41538f0 0x7f97f4159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.179+0000 7f97f93a7640 1 -- 172.21.15.46:0/719277363 >> 172.21.15.46:0/719277363 conn(0x7f97f406a640 msgr2=0x7f97f4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:16.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.179+0000 7f97f93a7640 1 -- 172.21.15.46:0/719277363 shutdown_connections 2023-10-30T14:12:16.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.179+0000 7f97f93a7640 1 -- 172.21.15.46:0/719277363 wait complete. 2023-10-30T14:12:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f93a7640 1 Processor -- start 2023-10-30T14:12:16.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f93a7640 1 -- start start 2023-10-30T14:12:16.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f93a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f40cefa0 0x7f97f415a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:16.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f93a7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97f41538f0 0x7f97f413fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:16.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f93a7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97f415cd50 0x7f97f4158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:16.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f93a7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f4140180 con 0x7f97f40cefa0 2023-10-30T14:12:16.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f93a7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f41ee7d0 con 0x7f97f41538f0 2023-10-30T14:12:16.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f93a7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f41eea90 con 0x7f97f415cd50 2023-10-30T14:12:16.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f40cefa0 0x7f97f415a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:16.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f40cefa0 0x7f97f415a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35764/0 (socket says 172.21.15.46:35764) 2023-10-30T14:12:16.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f3fff640 1 -- 172.21.15.46:0/184367454 learned_addr learned my addr 172.21.15.46:0/184367454 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:16.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f8ba6640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97f415cd50 0x7f97f4158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:16.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f37fe640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97f41538f0 0x7f97f413fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:16.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f3fff640 1 -- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97f415cd50 msgr2=0x7f97f4158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:16.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f3fff640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97f415cd50 0x7f97f4158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f3fff640 1 -- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97f41538f0 msgr2=0x7f97f413fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:16.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f3fff640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97f41538f0 0x7f97f413fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f3fff640 1 -- 172.21.15.46:0/184367454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4009420 con 0x7f97f40cefa0 2023-10-30T14:12:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f3fff640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f40cefa0 0x7f97f415a920 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f97e400ebe0 tx=0x7f97e400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f0ff9640 1 -- 172.21.15.46:0/184367454 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e400eee0 con 0x7f97f40cefa0 2023-10-30T14:12:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f0ff9640 1 -- 172.21.15.46:0/184367454 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e400cc30 con 0x7f97f40cefa0 2023-10-30T14:12:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f0ff9640 1 -- 172.21.15.46:0/184367454 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e4013470 con 0x7f97f40cefa0 2023-10-30T14:12:16.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f93a7640 1 -- 172.21.15.46:0/184367454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e8009420 con 0x7f97f40cefa0 2023-10-30T14:12:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f93a7640 1 -- 172.21.15.46:0/184367454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f41ef050 con 0x7f97f40cefa0 2023-10-30T14:12:16.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.183+0000 7f97f93a7640 1 -- 172.21.15.46:0/184367454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f404e170 con 0x7f97f40cefa0 2023-10-30T14:12:16.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.187+0000 7f97f0ff9640 1 -- 172.21.15.46:0/184367454 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f97e40135d0 con 0x7f97f40cefa0 2023-10-30T14:12:16.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.187+0000 7f97f0ff9640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97e00664a0 0x7f97e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:16.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.187+0000 7f97f0ff9640 1 -- 172.21.15.46:0/184367454 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f97e4084a60 con 0x7f97f40cefa0 2023-10-30T14:12:16.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.187+0000 7f97f37fe640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97e00664a0 0x7f97e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:16.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.187+0000 7f97f37fe640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97e00664a0 0x7f97e0068960 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f97e80b2000 tx=0x7f97e80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:16.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.191+0000 7f97f0ff9640 1 -- 172.21.15.46:0/184367454 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97e4051580 con 0x7f97f40cefa0 2023-10-30T14:12:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.379+0000 7f97f93a7640 1 -- 172.21.15.46:0/184367454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97f41f1c00 con 0x7f97f40cefa0 2023-10-30T14:12:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f0ff9640 1 -- 172.21.15.46:0/184367454 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5726 (secure 0 0 0) 0x7f97e4021090 con 0x7f97f40cefa0 2023-10-30T14:12:16.384 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:16.384 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:14.616094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 209 2023-10-30T14:12:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f93a7640 1 -- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97e00664a0 msgr2=0x7f97e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f93a7640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97e00664a0 0x7f97e0068960 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f97e80b2000 tx=0x7f97e80aa070 comp rx=0 tx=0).stop 2023-10-30T14:12:16.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f93a7640 1 -- 172.21.15.46:0/184367454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f40cefa0 msgr2=0x7f97f415a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f93a7640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f40cefa0 0x7f97f415a920 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f97e400ebe0 tx=0x7f97e400c520 comp rx=0 tx=0).stop 2023-10-30T14:12:16.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f93a7640 1 -- 172.21.15.46:0/184367454 shutdown_connections 2023-10-30T14:12:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f93a7640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97e00664a0 0x7f97e0068960 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f93a7640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97f415cd50 0x7f97f4158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f93a7640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97f41538f0 0x7f97f413fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f93a7640 1 --2- 172.21.15.46:0/184367454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97f40cefa0 0x7f97f415a920 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f93a7640 1 -- 172.21.15.46:0/184367454 >> 172.21.15.46:0/184367454 conn(0x7f97f406a640 msgr2=0x7f97f40c9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f93a7640 1 -- 172.21.15.46:0/184367454 shutdown_connections 2023-10-30T14:12:16.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.383+0000 7f97f93a7640 1 -- 172.21.15.46:0/184367454 wait complete. 2023-10-30T14:12:16.405 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T14:12:16.406 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:12:16.406 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:16.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.547+0000 7f04e8832640 1 Processor -- start 2023-10-30T14:12:16.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.547+0000 7f04e8832640 1 -- start start 2023-10-30T14:12:16.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.547+0000 7f04e8832640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f04e4159e60 0x7f04e406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:16.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.547+0000 7f04e8832640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f04e40bfec0 0x7f04e40c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:16.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.547+0000 7f04e8832640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e406fa80 con 0x7f04e406a720 2023-10-30T14:12:16.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.547+0000 7f04e8832640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e4071290 con 0x7f04e4159e60 2023-10-30T14:12:16.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.547+0000 7f04e8832640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e4071570 con 0x7f04e40bfec0 2023-10-30T14:12:16.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.547+0000 7f04e259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f04e4159e60 0x7f04e406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:16.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.547+0000 7f04e259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f04e4159e60 0x7f04e406f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52194/0 (socket says 172.21.15.46:52194) 2023-10-30T14:12:16.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.547+0000 7f04e259c640 1 -- 172.21.15.46:0/2439108767 learned_addr learned my addr 172.21.15.46:0/2439108767 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:16.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.547+0000 7f04dbfff640 1 --2- 172.21.15.46:0/2439108767 >> v2:172.21.15.154:3300/0 conn(0x7f04e40bfec0 0x7f04e40c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:16.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.551+0000 7f04e259c640 1 -- 172.21.15.46:0/2439108767 >> v2:172.21.15.154:3300/0 conn(0x7f04e40bfec0 msgr2=0x7f04e40c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:16.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.551+0000 7f04e259c640 1 --2- 172.21.15.46:0/2439108767 >> v2:172.21.15.154:3300/0 conn(0x7f04e40bfec0 0x7f04e40c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.551+0000 7f04e259c640 1 -- 172.21.15.46:0/2439108767 >> v1:172.21.15.154:6789/0 conn(0x7f04e406a720 legacy=0x7f04e4155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:16.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.551+0000 7f04e259c640 1 -- 172.21.15.46:0/2439108767 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e4071850 con 0x7f04e4159e60 2023-10-30T14:12:16.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.551+0000 7f04e259c640 1 --2- 172.21.15.46:0/2439108767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e4159e60 0x7f04e406f540 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f04e406f200 tx=0x7f04d401fdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ad70b9d0d3d4bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:16.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.551+0000 7f04e1d9b640 1 -- 172.21.15.46:0/2439108767 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d40a9ac0 con 0x7f04e4159e60 2023-10-30T14:12:16.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.551+0000 7f04e1d9b640 1 -- 172.21.15.46:0/2439108767 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d40a9c20 con 0x7f04e4159e60 2023-10-30T14:12:16.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.551+0000 7f04e1d9b640 1 -- 172.21.15.46:0/2439108767 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d40b1440 con 0x7f04e4159e60 2023-10-30T14:12:16.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 -- 172.21.15.46:0/2439108767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e4159e60 msgr2=0x7f04e406f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:16.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 --2- 172.21.15.46:0/2439108767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e4159e60 0x7f04e406f540 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f04e406f200 tx=0x7f04d401fdc0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 -- 172.21.15.46:0/2439108767 shutdown_connections 2023-10-30T14:12:16.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 --2- 172.21.15.46:0/2439108767 >> v2:172.21.15.154:3300/0 conn(0x7f04e40bfec0 0x7f04e40c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 --2- 172.21.15.46:0/2439108767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e4159e60 0x7f04e406f540 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 -- 172.21.15.46:0/2439108767 >> 172.21.15.46:0/2439108767 conn(0x7f04e40c7910 msgr2=0x7f04e40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:16.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 -- 172.21.15.46:0/2439108767 shutdown_connections 2023-10-30T14:12:16.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 -- 172.21.15.46:0/2439108767 wait complete. 2023-10-30T14:12:16.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 Processor -- start 2023-10-30T14:12:16.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 -- start start 2023-10-30T14:12:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e406a720 0x7f04e414f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:16.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e40bfec0 0x7f04e41520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:16.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e4159e60 0x7f04e4152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:16.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e4151690 con 0x7f04e40bfec0 2023-10-30T14:12:16.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e41f0e20 con 0x7f04e406a720 2023-10-30T14:12:16.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e8832640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e4150480 con 0x7f04e4159e60 2023-10-30T14:12:16.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04dbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e40bfec0 0x7f04e41520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:16.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04dbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e40bfec0 0x7f04e41520e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52206/0 (socket says 172.21.15.46:52206) 2023-10-30T14:12:16.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04dbfff640 1 -- 172.21.15.46:0/3893434409 learned_addr learned my addr 172.21.15.46:0/3893434409 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:16.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e2d9d640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e4159e60 0x7f04e4152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:16.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04e259c640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e406a720 0x7f04e414f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:16.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04dbfff640 1 -- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e4159e60 msgr2=0x7f04e4152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04dbfff640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e4159e60 0x7f04e4152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04dbfff640 1 -- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e406a720 msgr2=0x7f04e414f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04dbfff640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e406a720 0x7f04e414f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.555+0000 7f04dbfff640 1 -- 172.21.15.46:0/3893434409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d40094d0 con 0x7f04e40bfec0 2023-10-30T14:12:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.559+0000 7f04dbfff640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e40bfec0 0x7f04e41520e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f04cc00d800 tx=0x7f04cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.559+0000 7f04daffd640 1 -- 172.21.15.46:0/3893434409 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc00b6c0 con 0x7f04e40bfec0 2023-10-30T14:12:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.559+0000 7f04daffd640 1 -- 172.21.15.46:0/3893434409 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04cc00b820 con 0x7f04e40bfec0 2023-10-30T14:12:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.559+0000 7f04daffd640 1 -- 172.21.15.46:0/3893434409 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc00bb90 con 0x7f04e40bfec0 2023-10-30T14:12:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.559+0000 7f04e8832640 1 -- 172.21.15.46:0/3893434409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e4150760 con 0x7f04e40bfec0 2023-10-30T14:12:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.559+0000 7f04e8832640 1 -- 172.21.15.46:0/3893434409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e4151bc0 con 0x7f04e40bfec0 2023-10-30T14:12:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.559+0000 7f04e8832640 1 -- 172.21.15.46:0/3893434409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e404e170 con 0x7f04e40bfec0 2023-10-30T14:12:16.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.559+0000 7f04daffd640 1 -- 172.21.15.46:0/3893434409 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f04cc00bcf0 con 0x7f04e40bfec0 2023-10-30T14:12:16.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.559+0000 7f04daffd640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04c40664a0 0x7f04c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:16.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.559+0000 7f04daffd640 1 -- 172.21.15.46:0/3893434409 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f04cc084c40 con 0x7f04e40bfec0 2023-10-30T14:12:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.559+0000 7f04e259c640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04c40664a0 0x7f04c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.563+0000 7f04e259c640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04c40664a0 0x7f04c4068960 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f04d40b3000 tx=0x7f04d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.563+0000 7f04daffd640 1 -- 172.21.15.46:0/3893434409 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f04cc051780 con 0x7f04e40bfec0 2023-10-30T14:12:16.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.755+0000 7f04e8832640 1 -- 172.21.15.46:0/3893434409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04e41e0df0 con 0x7f04e40bfec0 2023-10-30T14:12:16.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.755+0000 7f04daffd640 1 -- 172.21.15.46:0/3893434409 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5726 (secure 0 0 0) 0x7f04cc054da0 con 0x7f04e40bfec0 2023-10-30T14:12:16.759 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:16.759 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:14.616094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:16.760 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 209 2023-10-30T14:12:16.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.759+0000 7f04e8832640 1 -- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04c40664a0 msgr2=0x7f04c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:16.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.759+0000 7f04e8832640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04c40664a0 0x7f04c4068960 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f04d40b3000 tx=0x7f04d4008040 comp rx=0 tx=0).stop 2023-10-30T14:12:16.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.759+0000 7f04e8832640 1 -- 172.21.15.46:0/3893434409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e40bfec0 msgr2=0x7f04e41520e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:16.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.759+0000 7f04e8832640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e40bfec0 0x7f04e41520e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f04cc00d800 tx=0x7f04cc00dcc0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.759+0000 7f04e8832640 1 -- 172.21.15.46:0/3893434409 shutdown_connections 2023-10-30T14:12:16.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.759+0000 7f04e8832640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04c40664a0 0x7f04c4068960 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.759+0000 7f04e8832640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e4159e60 0x7f04e4152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.759+0000 7f04e8832640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e40bfec0 0x7f04e41520e0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.759+0000 7f04e8832640 1 --2- 172.21.15.46:0/3893434409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e406a720 0x7f04e414f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:16.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.759+0000 7f04e8832640 1 -- 172.21.15.46:0/3893434409 >> 172.21.15.46:0/3893434409 conn(0x7f04e40c7910 msgr2=0x7f04e4158d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:16.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.759+0000 7f04e8832640 1 -- 172.21.15.46:0/3893434409 shutdown_connections 2023-10-30T14:12:16.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:16.759+0000 7f04e8832640 1 -- 172.21.15.46:0/3893434409 wait complete. 2023-10-30T14:12:16.781 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 209, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 209, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8826": { "addr": "172.21.15.46:6836/66265765", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 66265765, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 66265765, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 8826, "incarnation": 203, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 17 }, "gid_8982": { "addr": "172.21.15.46:6833/2455118950", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2455118950, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2455118950, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8982, "incarnation": 209, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:12:14.616094+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8826, "mds_1": 8982 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:12:18.781 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.935+0000 7ffa12d1d640 1 Processor -- start 2023-10-30T14:12:18.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.935+0000 7ffa12d1d640 1 -- start start 2023-10-30T14:12:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.935+0000 7ffa12d1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0c1538a0 0x7ffa0c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.935+0000 7ffa12d1d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffa0c0cf160 0x7ffa0c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.935+0000 7ffa12d1d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0c0cb800 con 0x7ffa0c0ca0e0 2023-10-30T14:12:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.935+0000 7ffa12d1d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa0c15dc30 con 0x7ffa0c0cf160 2023-10-30T14:12:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.935+0000 7ffa12d1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c0cb330 con 0x7ffa0c1538a0 2023-10-30T14:12:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.935+0000 7ffa10ab9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0c1538a0 0x7ffa0c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.935+0000 7ffa112ba640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffa0c0cf160 0x7ffa0c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.935+0000 7ffa0bfff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ffa0c0ca0e0 0x7ffa0c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:52894/0 (socket says 172.21.15.46:52894) 2023-10-30T14:12:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.935+0000 7ffa0bfff640 1 -- 172.21.15.46:0/2420709854 learned_addr learned my addr 172.21.15.46:0/2420709854 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.939+0000 7ffa10ab9640 1 -- 172.21.15.46:0/2420709854 >> v2:172.21.15.154:3300/0 conn(0x7ffa0c0cf160 msgr2=0x7ffa0c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.939+0000 7ffa10ab9640 1 --2- 172.21.15.46:0/2420709854 >> v2:172.21.15.154:3300/0 conn(0x7ffa0c0cf160 0x7ffa0c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.939+0000 7ffa10ab9640 1 -- 172.21.15.46:0/2420709854 >> v1:172.21.15.154:6789/0 conn(0x7ffa0c0ca0e0 legacy=0x7ffa0c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.939+0000 7ffa10ab9640 1 -- 172.21.15.46:0/2420709854 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c0cad70 con 0x7ffa0c1538a0 2023-10-30T14:12:18.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.939+0000 7ffa112ba640 1 --2- 172.21.15.46:0/2420709854 >> v2:172.21.15.154:3300/0 conn(0x7ffa0c0cf160 0x7ffa0c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:12:18.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.939+0000 7ffa10ab9640 1 --2- 172.21.15.46:0/2420709854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0c1538a0 0x7ffa0c15d6f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffa000097c0 tx=0x7ffa0001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b49b1dd595282a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:18.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.939+0000 7ffa0b7fe640 1 -- 172.21.15.46:0/2420709854 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa000a8970 con 0x7ffa0c1538a0 2023-10-30T14:12:18.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa0b7fe640 1 -- 172.21.15.46:0/2420709854 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa000a8ad0 con 0x7ffa0c1538a0 2023-10-30T14:12:18.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa0b7fe640 1 -- 172.21.15.46:0/2420709854 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa000a8e60 con 0x7ffa0c1538a0 2023-10-30T14:12:18.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2420709854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0c1538a0 msgr2=0x7ffa0c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:18.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 --2- 172.21.15.46:0/2420709854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0c1538a0 0x7ffa0c15d6f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffa000097c0 tx=0x7ffa0001ec70 comp rx=0 tx=0).stop 2023-10-30T14:12:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2420709854 shutdown_connections 2023-10-30T14:12:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 --2- 172.21.15.46:0/2420709854 >> v2:172.21.15.154:3300/0 conn(0x7ffa0c0cf160 0x7ffa0c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 --2- 172.21.15.46:0/2420709854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0c1538a0 0x7ffa0c15d6f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2420709854 >> 172.21.15.46:0/2420709854 conn(0x7ffa0c06bfe0 msgr2=0x7ffa0c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2420709854 shutdown_connections 2023-10-30T14:12:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2420709854 wait complete. 2023-10-30T14:12:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 Processor -- start 2023-10-30T14:12:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 -- start start 2023-10-30T14:12:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa0c0ca0e0 0x7ffa0c158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c0cf160 0x7ffa0c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0c1538a0 0x7ffa0c0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c1f36b0 con 0x7ffa0c0cf160 2023-10-30T14:12:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c157a90 con 0x7ffa0c0ca0e0 2023-10-30T14:12:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa12d1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c14d790 con 0x7ffa0c1538a0 2023-10-30T14:12:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa0bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c0cf160 0x7ffa0c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:18.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa0bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c0cf160 0x7ffa0c0ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52220/0 (socket says 172.21.15.46:52220) 2023-10-30T14:12:18.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.943+0000 7ffa0bfff640 1 -- 172.21.15.46:0/2500364858 learned_addr learned my addr 172.21.15.46:0/2500364858 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:18.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa112ba640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0c1538a0 0x7ffa0c0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:18.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa10ab9640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa0c0ca0e0 0x7ffa0c158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:18.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa0bfff640 1 -- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0c1538a0 msgr2=0x7ffa0c0cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:18.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa0bfff640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0c1538a0 0x7ffa0c0cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:18.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa0bfff640 1 -- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa0c0ca0e0 msgr2=0x7ffa0c158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:18.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa0bfff640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa0c0ca0e0 0x7ffa0c158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:18.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa0bfff640 1 -- 172.21.15.46:0/2500364858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa04009a70 con 0x7ffa0c0cf160 2023-10-30T14:12:18.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa0bfff640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c0cf160 0x7ffa0c0ce590 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800c500 tx=0x7ff9f800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:18.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa097fa640 1 -- 172.21.15.46:0/2500364858 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8005cf0 con 0x7ffa0c0cf160 2023-10-30T14:12:18.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa097fa640 1 -- 172.21.15.46:0/2500364858 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f8005e50 con 0x7ffa0c0cf160 2023-10-30T14:12:18.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa097fa640 1 -- 172.21.15.46:0/2500364858 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f80072f0 con 0x7ffa0c0cf160 2023-10-30T14:12:18.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2500364858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa00009420 con 0x7ffa0c0cf160 2023-10-30T14:12:18.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2500364858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0c1f2d10 con 0x7ffa0c0cf160 2023-10-30T14:12:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2500364858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0c0b2a50 con 0x7ffa0c0cf160 2023-10-30T14:12:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.947+0000 7ffa097fa640 1 -- 172.21.15.46:0/2500364858 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff9f8007450 con 0x7ffa0c0cf160 2023-10-30T14:12:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.951+0000 7ffa097fa640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9f00664a0 0x7ff9f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.951+0000 7ffa097fa640 1 -- 172.21.15.46:0/2500364858 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7ff9f8083ca0 con 0x7ffa0c0cf160 2023-10-30T14:12:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.951+0000 7ffa10ab9640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9f00664a0 0x7ff9f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.951+0000 7ffa10ab9640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9f00664a0 0x7ff9f0068960 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ffa000b2000 tx=0x7ffa000aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:18.955+0000 7ffa097fa640 1 -- 172.21.15.46:0/2500364858 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9f8050870 con 0x7ffa0c0cf160 2023-10-30T14:12:19.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.143+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2500364858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa0c04e170 con 0x7ffa0c0cf160 2023-10-30T14:12:19.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.143+0000 7ffa097fa640 1 -- 172.21.15.46:0/2500364858 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5726 (secure 0 0 0) 0x7ff9f8017070 con 0x7ffa0c0cf160 2023-10-30T14:12:19.146 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:19.146 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:14.616094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:19.146 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 209 2023-10-30T14:12:19.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.147+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9f00664a0 msgr2=0x7ff9f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.147+0000 7ffa12d1d640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9f00664a0 0x7ff9f0068960 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ffa000b2000 tx=0x7ffa000aa070 comp rx=0 tx=0).stop 2023-10-30T14:12:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.147+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2500364858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c0cf160 msgr2=0x7ffa0c0ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.147+0000 7ffa12d1d640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c0cf160 0x7ffa0c0ce590 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800c500 tx=0x7ff9f800c9c0 comp rx=0 tx=0).stop 2023-10-30T14:12:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.147+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2500364858 shutdown_connections 2023-10-30T14:12:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.147+0000 7ffa12d1d640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff9f00664a0 0x7ff9f0068960 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:19.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.147+0000 7ffa12d1d640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa0c1538a0 0x7ffa0c0cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.147+0000 7ffa12d1d640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c0cf160 0x7ffa0c0ce590 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.147+0000 7ffa12d1d640 1 --2- 172.21.15.46:0/2500364858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa0c0ca0e0 0x7ffa0c158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.147+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2500364858 >> 172.21.15.46:0/2500364858 conn(0x7ffa0c06bfe0 msgr2=0x7ffa0c1510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.147+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2500364858 shutdown_connections 2023-10-30T14:12:19.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:19.147+0000 7ffa12d1d640 1 -- 172.21.15.46:0/2500364858 wait complete. 2023-10-30T14:12:19.167 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:12:21.168 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:21.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0f01e09640 1 Processor -- start 2023-10-30T14:12:21.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0f01e09640 1 -- start start 2023-10-30T14:12:21.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0f01e09640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0efc1538a0 0x7f0efc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0f01e09640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0efc0cb800 con 0x7f0efc0ca0e0 2023-10-30T14:12:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0f01e09640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0efc15dc30 con 0x7f0efc0cf160 2023-10-30T14:12:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0f01e09640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0efc0cb330 con 0x7f0efc1538a0 2023-10-30T14:12:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0efaffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0efc0ca0e0 0x7f0efc15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48498/0 (socket says 172.21.15.46:48498) 2023-10-30T14:12:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0efaffd640 1 -- 172.21.15.46:0/2905853625 learned_addr learned my addr 172.21.15.46:0/2905853625 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0efb7fe640 1 --2- 172.21.15.46:0/2905853625 >> v2:172.21.15.154:3300/0 conn(0x7f0efc1538a0 0x7f0efc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:21.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917463407 0 0) 0x7f0efc0cb800 con 0x7f0efc0ca0e0 2023-10-30T14:12:21.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee4003530 con 0x7f0efc0ca0e0 2023-10-30T14:12:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 209975140 0 0) 0x7f0ee4003530 con 0x7f0efc0ca0e0 2023-10-30T14:12:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 >> v2:172.21.15.154:3300/0 conn(0x7f0efc1538a0 msgr2=0x7f0efc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0efa7fc640 1 --2- 172.21.15.46:0/2905853625 >> v2:172.21.15.154:3300/0 conn(0x7f0efc1538a0 0x7f0efc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 >> v1:172.21.15.154:6789/0 conn(0x7f0efc0cf160 legacy=0x7f0efc164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc0cad70 con 0x7f0efc0ca0e0 2023-10-30T14:12:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.335+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f0ef0001680 con 0x7f0efc0ca0e0 2023-10-30T14:12:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 >> v1:172.21.15.46:6789/0 conn(0x7f0efc0ca0e0 legacy=0x7f0efc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efa7fc640 1 --2- 172.21.15.46:0/2905853625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee4003980 0x7f0ee4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efa7fc640 1 --2- 172.21.15.46:0/2905853625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee40047d0 0x7f0ee4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efa7fc640 1 --2- 172.21.15.46:0/2905853625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee4005190 0x7f0ee400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0cb330 con 0x7f0ee4005190 2023-10-30T14:12:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee400c040 con 0x7f0ee40047d0 2023-10-30T14:12:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee400c200 con 0x7f0ee4003980 2023-10-30T14:12:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efbfff640 1 --2- 172.21.15.46:0/2905853625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee4005190 0x7f0ee400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efaffd640 1 --2- 172.21.15.46:0/2905853625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee40047d0 0x7f0ee4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efb7fe640 1 --2- 172.21.15.46:0/2905853625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee4003980 0x7f0ee4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efbfff640 1 -- 172.21.15.46:0/2905853625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee4003980 msgr2=0x7f0ee4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efbfff640 1 --2- 172.21.15.46:0/2905853625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ee4003980 0x7f0ee4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efbfff640 1 -- 172.21.15.46:0/2905853625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee40047d0 msgr2=0x7f0ee4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efbfff640 1 --2- 172.21.15.46:0/2905853625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee40047d0 0x7f0ee4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efbfff640 1 -- 172.21.15.46:0/2905853625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee4003530 con 0x7f0ee4005190 2023-10-30T14:12:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efbfff640 1 -- 172.21.15.46:0/2905853625 reap_dead start 2023-10-30T14:12:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efbfff640 1 --2- 172.21.15.46:0/2905853625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee4005190 0x7f0ee400b990 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4001cd0 tx=0x7f0ef401c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76290b31289b5f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4003b80 con 0x7f0ee4005190 2023-10-30T14:12:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef4003d20 con 0x7f0ee4005190 2023-10-30T14:12:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.339+0000 7f0efa7fc640 1 -- 172.21.15.46:0/2905853625 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4005310 con 0x7f0ee4005190 2023-10-30T14:12:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 -- 172.21.15.46:0/2905853625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee4005190 msgr2=0x7f0ee400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 --2- 172.21.15.46:0/2905853625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee4005190 0x7f0ee400b990 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4001cd0 tx=0x7f0ef401c5c0 comp rx=0 tx=0).stop 2023-10-30T14:12:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 -- 172.21.15.46:0/2905853625 shutdown_connections 2023-10-30T14:12:21.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 --2- 172.21.15.46:0/2905853625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ee4005190 0x7f0ee400b990 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:21.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 -- 172.21.15.46:0/2905853625 >> 172.21.15.46:0/2905853625 conn(0x7f0efc06bfe0 msgr2=0x7f0efc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:21.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 -- 172.21.15.46:0/2905853625 shutdown_connections 2023-10-30T14:12:21.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 -- 172.21.15.46:0/2905853625 wait complete. 2023-10-30T14:12:21.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 Processor -- start 2023-10-30T14:12:21.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 -- start start 2023-10-30T14:12:21.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee4005190 0x7f0efc0c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:21.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0efc152e90 0x7f0efc15cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:21.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0efc0c71e0 0x7f0efc0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc15c8d0 con 0x7f0efc152e90 2023-10-30T14:12:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc15af20 con 0x7f0ee4005190 2023-10-30T14:12:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0f01e09640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc15bfa0 con 0x7f0efc0c71e0 2023-10-30T14:12:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0efaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0efc152e90 0x7f0efc15cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0efaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0efc152e90 0x7f0efc15cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52236/0 (socket says 172.21.15.46:52236) 2023-10-30T14:12:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.343+0000 7f0efaffd640 1 -- 172.21.15.46:0/3274671376 learned_addr learned my addr 172.21.15.46:0/3274671376 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0efbfff640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0efc0c71e0 0x7f0efc0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0efb7fe640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee4005190 0x7f0efc0c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0efaffd640 1 -- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0efc0c71e0 msgr2=0x7f0efc0b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0efaffd640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0efc0c71e0 0x7f0efc0b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0efaffd640 1 -- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee4005190 msgr2=0x7f0efc0c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0efaffd640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee4005190 0x7f0efc0c6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0efaffd640 1 -- 172.21.15.46:0/3274671376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc15c280 con 0x7f0efc152e90 2023-10-30T14:12:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0efb7fe640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee4005190 0x7f0efc0c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:12:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0efaffd640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0efc152e90 0x7f0efc15cd60 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0001a00 tx=0x7f0ef0007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0ee3fff640 1 -- 172.21.15.46:0/3274671376 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef000f940 con 0x7f0efc152e90 2023-10-30T14:12:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0ee3fff640 1 -- 172.21.15.46:0/3274671376 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef000fac0 con 0x7f0efc152e90 2023-10-30T14:12:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0ee3fff640 1 -- 172.21.15.46:0/3274671376 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef000fe70 con 0x7f0efc152e90 2023-10-30T14:12:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0f01e09640 1 -- 172.21.15.46:0/3274671376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0efc15c560 con 0x7f0efc152e90 2023-10-30T14:12:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0f01e09640 1 -- 172.21.15.46:0/3274671376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0efc15d2a0 con 0x7f0efc152e90 2023-10-30T14:12:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0f01e09640 1 -- 172.21.15.46:0/3274671376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0efc04e170 con 0x7f0efc152e90 2023-10-30T14:12:21.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0ee3fff640 1 -- 172.21.15.46:0/3274671376 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ef001d460 con 0x7f0efc152e90 2023-10-30T14:12:21.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0ee3fff640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ee80664a0 0x7f0ee8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:21.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.347+0000 7f0ee3fff640 1 -- 172.21.15.46:0/3274671376 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f0ef0084970 con 0x7f0efc152e90 2023-10-30T14:12:21.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.351+0000 7f0efb7fe640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ee80664a0 0x7f0ee8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:21.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.351+0000 7f0efb7fe640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ee80664a0 0x7f0ee8068960 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009b60 tx=0x7f0eec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:21.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.355+0000 7f0ee3fff640 1 -- 172.21.15.46:0/3274671376 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0ef0051400 con 0x7f0efc152e90 2023-10-30T14:12:21.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.611+0000 7f0f01e09640 1 -- 172.21.15.46:0/3274671376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0efc15ca10 con 0x7f0efc152e90 2023-10-30T14:12:21.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.611+0000 7f0ee3fff640 1 -- 172.21.15.46:0/3274671376 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5726 (secure 0 0 0) 0x7f0ef0054a20 con 0x7f0efc152e90 2023-10-30T14:12:21.614 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:21.614 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:14.616094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 209 2023-10-30T14:12:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.615+0000 7f0f01e09640 1 -- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ee80664a0 msgr2=0x7f0ee8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.615+0000 7f0f01e09640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ee80664a0 0x7f0ee8068960 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009b60 tx=0x7f0eec008040 comp rx=0 tx=0).stop 2023-10-30T14:12:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.615+0000 7f0f01e09640 1 -- 172.21.15.46:0/3274671376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0efc152e90 msgr2=0x7f0efc15cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.615+0000 7f0f01e09640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0efc152e90 0x7f0efc15cd60 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0001a00 tx=0x7f0ef0007720 comp rx=0 tx=0).stop 2023-10-30T14:12:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.615+0000 7f0f01e09640 1 -- 172.21.15.46:0/3274671376 shutdown_connections 2023-10-30T14:12:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.615+0000 7f0f01e09640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0ee80664a0 0x7f0ee8068960 secure :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009b60 tx=0x7f0eec008040 comp rx=0 tx=0).stop 2023-10-30T14:12:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.615+0000 7f0f01e09640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0efc0c71e0 0x7f0efc0b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.615+0000 7f0f01e09640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0efc152e90 0x7f0efc15cd60 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.615+0000 7f0f01e09640 1 --2- 172.21.15.46:0/3274671376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee4005190 0x7f0efc0c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.615+0000 7f0f01e09640 1 -- 172.21.15.46:0/3274671376 >> 172.21.15.46:0/3274671376 conn(0x7f0efc06bfe0 msgr2=0x7f0efc0cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.615+0000 7f0f01e09640 1 -- 172.21.15.46:0/3274671376 shutdown_connections 2023-10-30T14:12:21.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:21.615+0000 7f0f01e09640 1 -- 172.21.15.46:0/3274671376 wait complete. 2023-10-30T14:12:21.638 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:12:23.639 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:23.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.787+0000 7fa85ebc5640 1 Processor -- start 2023-10-30T14:12:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.787+0000 7fa85ebc5640 1 -- start start 2023-10-30T14:12:23.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.787+0000 7fa85ebc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa85815d6f0 0x7fa85815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.787+0000 7fa85ebc5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa8580cf160 0x7fa8580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.787+0000 7fa85ebc5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8580cb800 con 0x7fa8581538a0 2023-10-30T14:12:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.787+0000 7fa85ebc5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8580cb330 con 0x7fa8580cf160 2023-10-30T14:12:23.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.787+0000 7fa85ebc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8580cad70 con 0x7fa85815d6f0 2023-10-30T14:12:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.791+0000 7fa85c961640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa8581538a0 0x7fa8580ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:52916/0 (socket says 172.21.15.46:52916) 2023-10-30T14:12:23.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.791+0000 7fa857fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa85815d6f0 0x7fa85815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.791+0000 7fa85d162640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa8580cf160 0x7fa8580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.791+0000 7fa85c961640 1 -- 172.21.15.46:0/3126368614 learned_addr learned my addr 172.21.15.46:0/3126368614 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.791+0000 7fa8577fe640 1 -- 172.21.15.46:0/3126368614 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060602376 0 0) 0x7fa8580cb800 con 0x7fa8581538a0 2023-10-30T14:12:23.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.791+0000 7fa8577fe640 1 -- 172.21.15.46:0/3126368614 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa848003530 con 0x7fa8581538a0 2023-10-30T14:12:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.791+0000 7fa857fff640 1 -- 172.21.15.46:0/3126368614 >> v2:172.21.15.154:3300/0 conn(0x7fa8580cf160 msgr2=0x7fa8580cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.791+0000 7fa857fff640 1 --2- 172.21.15.46:0/3126368614 >> v2:172.21.15.154:3300/0 conn(0x7fa8580cf160 0x7fa8580cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.791+0000 7fa857fff640 1 -- 172.21.15.46:0/3126368614 >> v1:172.21.15.154:6789/0 conn(0x7fa8581538a0 legacy=0x7fa8580ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.791+0000 7fa857fff640 1 -- 172.21.15.46:0/3126368614 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8580cb050 con 0x7fa85815d6f0 2023-10-30T14:12:23.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.791+0000 7fa85d162640 1 --2- 172.21.15.46:0/3126368614 >> v2:172.21.15.154:3300/0 conn(0x7fa8580cf160 0x7fa8580cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:12:23.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa857fff640 1 --2- 172.21.15.46:0/3126368614 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa85815d6f0 0x7fa85815a090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa8440097c0 tx=0x7fa84401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9938b6644fedcbd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa8577fe640 1 -- 172.21.15.46:0/3126368614 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8440a8970 con 0x7fa85815d6f0 2023-10-30T14:12:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa8577fe640 1 -- 172.21.15.46:0/3126368614 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8440a8ad0 con 0x7fa85815d6f0 2023-10-30T14:12:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa8577fe640 1 -- 172.21.15.46:0/3126368614 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8440a8e40 con 0x7fa85815d6f0 2023-10-30T14:12:23.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3126368614 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa85815d6f0 msgr2=0x7fa85815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa85ebc5640 1 --2- 172.21.15.46:0/3126368614 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa85815d6f0 0x7fa85815a090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa8440097c0 tx=0x7fa84401ec70 comp rx=0 tx=0).stop 2023-10-30T14:12:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3126368614 shutdown_connections 2023-10-30T14:12:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa85ebc5640 1 --2- 172.21.15.46:0/3126368614 >> v2:172.21.15.154:3300/0 conn(0x7fa8580cf160 0x7fa8580cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa85ebc5640 1 --2- 172.21.15.46:0/3126368614 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa85815d6f0 0x7fa85815a090 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3126368614 >> 172.21.15.46:0/3126368614 conn(0x7fa85806bfe0 msgr2=0x7fa85806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3126368614 shutdown_connections 2023-10-30T14:12:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3126368614 wait complete. 2023-10-30T14:12:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.795+0000 7fa85ebc5640 1 Processor -- start 2023-10-30T14:12:23.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85ebc5640 1 -- start start 2023-10-30T14:12:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85ebc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8580cf160 0x7fa8581f2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85ebc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8581538a0 0x7fa85814cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:23.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85ebc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa85815d6f0 0x7fa85814d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85ebc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8581f4370 con 0x7fa8580cf160 2023-10-30T14:12:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85ebc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8581f3e50 con 0x7fa85815d6f0 2023-10-30T14:12:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85ebc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa85815b920 con 0x7fa8581538a0 2023-10-30T14:12:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85c961640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8580cf160 0x7fa8581f2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:23.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85c961640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8580cf160 0x7fa8581f2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52248/0 (socket says 172.21.15.46:52248) 2023-10-30T14:12:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa857fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8581538a0 0x7fa85814cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85c961640 1 -- 172.21.15.46:0/3420237536 learned_addr learned my addr 172.21.15.46:0/3420237536 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:23.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85d162640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa85815d6f0 0x7fa85814d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85c961640 1 -- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8581538a0 msgr2=0x7fa85814cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85c961640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8581538a0 0x7fa85814cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85c961640 1 -- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa85815d6f0 msgr2=0x7fa85814d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85c961640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa85815d6f0 0x7fa85814d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85c961640 1 -- 172.21.15.46:0/3420237536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa850009a70 con 0x7fa8580cf160 2023-10-30T14:12:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85c961640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8580cf160 0x7fa8581f2f00 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00c4d0 tx=0x7fa84c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa8557fa640 1 -- 172.21.15.46:0/3420237536 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c005cf0 con 0x7fa8580cf160 2023-10-30T14:12:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa8557fa640 1 -- 172.21.15.46:0/3420237536 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84c005e50 con 0x7fa8580cf160 2023-10-30T14:12:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3420237536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa844009420 con 0x7fa8580cf160 2023-10-30T14:12:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa8557fa640 1 -- 172.21.15.46:0/3420237536 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c007270 con 0x7fa8580cf160 2023-10-30T14:12:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.799+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3420237536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8581f3890 con 0x7fa8580cf160 2023-10-30T14:12:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.803+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3420237536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8580b2a50 con 0x7fa8580cf160 2023-10-30T14:12:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.803+0000 7fa8557fa640 1 -- 172.21.15.46:0/3420237536 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa84c0073d0 con 0x7fa8580cf160 2023-10-30T14:12:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.803+0000 7fa8557fa640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa83c0664a0 0x7fa83c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.803+0000 7fa8557fa640 1 -- 172.21.15.46:0/3420237536 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7fa84c083d90 con 0x7fa8580cf160 2023-10-30T14:12:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.803+0000 7fa857fff640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa83c0664a0 0x7fa83c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:23.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.807+0000 7fa857fff640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa83c0664a0 0x7fa83c068960 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa8440b2000 tx=0x7fa8440aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:23.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:23.807+0000 7fa8557fa640 1 -- 172.21.15.46:0/3420237536 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa84c0508d0 con 0x7fa8580cf160 2023-10-30T14:12:24.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3420237536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa85804e170 con 0x7fa8580cf160 2023-10-30T14:12:24.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa8557fa640 1 -- 172.21.15.46:0/3420237536 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5726 (secure 0 0 0) 0x7fa84c017070 con 0x7fa8580cf160 2023-10-30T14:12:24.076 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:24.076 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:14.616094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:24.077 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 209 2023-10-30T14:12:24.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa83c0664a0 msgr2=0x7fa83c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:24.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa83c0664a0 0x7fa83c068960 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa8440b2000 tx=0x7fa8440aa070 comp rx=0 tx=0).stop 2023-10-30T14:12:24.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3420237536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8580cf160 msgr2=0x7fa8581f2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:24.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8580cf160 0x7fa8581f2f00 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00c4d0 tx=0x7fa84c00c990 comp rx=0 tx=0).stop 2023-10-30T14:12:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3420237536 shutdown_connections 2023-10-30T14:12:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa83c0664a0 0x7fa83c068960 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa85815d6f0 0x7fa85814d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8581538a0 0x7fa85814cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 --2- 172.21.15.46:0/3420237536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8580cf160 0x7fa8581f2f00 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:24.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3420237536 >> 172.21.15.46:0/3420237536 conn(0x7fa85806bfe0 msgr2=0x7fa85815c010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:24.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3420237536 shutdown_connections 2023-10-30T14:12:24.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:24.075+0000 7fa85ebc5640 1 -- 172.21.15.46:0/3420237536 wait complete. 2023-10-30T14:12:24.100 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:12:26.102 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:26.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f6fc4640 1 Processor -- start 2023-10-30T14:12:26.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f6fc4640 1 -- start start 2023-10-30T14:12:26.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f6fc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f04f00c05d0 0x7f04f013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:26.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f6fc4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f00c0a50 con 0x7f04f00c6f60 2023-10-30T14:12:26.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f6fc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f006f6e0 con 0x7f04f00c6b80 2023-10-30T14:12:26.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f6fc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04f006f9c0 con 0x7f04f00c05d0 2023-10-30T14:12:26.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f5561640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f04f00c6f60 0x7f04f01566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48502/0 (socket says 172.21.15.46:48502) 2023-10-30T14:12:26.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f5561640 1 -- 172.21.15.46:0/154459530 learned_addr learned my addr 172.21.15.46:0/154459530 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:26.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04effff640 1 --2- 172.21.15.46:0/154459530 >> v2:172.21.15.154:3300/0 conn(0x7f04f00c05d0 0x7f04f013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:26.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487211275 0 0) 0x7f04f00c0a50 con 0x7f04f00c6f60 2023-10-30T14:12:26.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e0003530 con 0x7f04f00c6f60 2023-10-30T14:12:26.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1425819887 0 0) 0x7f04e0003530 con 0x7f04f00c6f60 2023-10-30T14:12:26.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 >> v2:172.21.15.154:3300/0 conn(0x7f04f00c05d0 msgr2=0x7f04f013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:26.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 --2- 172.21.15.46:0/154459530 >> v2:172.21.15.154:3300/0 conn(0x7f04f00c05d0 0x7f04f013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:26.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 >> v1:172.21.15.154:6789/0 conn(0x7f04f00c6b80 legacy=0x7f04f00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:26.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f006fca0 con 0x7f04f00c6f60 2023-10-30T14:12:26.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f04e8001a00 con 0x7f04f00c6f60 2023-10-30T14:12:26.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 >> v1:172.21.15.46:6789/0 conn(0x7f04f00c6f60 legacy=0x7f04f01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:26.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 --2- 172.21.15.46:0/154459530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0003980 0x7f04e0003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:26.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 --2- 172.21.15.46:0/154459530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e0004380 0x7f04e0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:26.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 --2- 172.21.15.46:0/154459530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e0005050 0x7f04e000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:26.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f006f9c0 con 0x7f04e0003980 2023-10-30T14:12:26.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e000e1c0 con 0x7f04e0004380 2023-10-30T14:12:26.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e000e380 con 0x7f04e0005050 2023-10-30T14:12:26.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f4d60640 1 --2- 172.21.15.46:0/154459530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0003980 0x7f04e0003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:26.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04effff640 1 --2- 172.21.15.46:0/154459530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e0004380 0x7f04e0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:26.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f5561640 1 --2- 172.21.15.46:0/154459530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e0005050 0x7f04e000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:26.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f4d60640 1 -- 172.21.15.46:0/154459530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e0005050 msgr2=0x7f04e000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:26.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f4d60640 1 --2- 172.21.15.46:0/154459530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e0005050 0x7f04e000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:26.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f4d60640 1 -- 172.21.15.46:0/154459530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e0004380 msgr2=0x7f04e0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:26.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f4d60640 1 --2- 172.21.15.46:0/154459530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e0004380 0x7f04e0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:26.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f4d60640 1 -- 172.21.15.46:0/154459530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e0003530 con 0x7f04e0003980 2023-10-30T14:12:26.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.271+0000 7f04f5561640 1 -- 172.21.15.46:0/154459530 reap_dead start 2023-10-30T14:12:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.275+0000 7f04f4d60640 1 --2- 172.21.15.46:0/154459530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0003980 0x7f04e0003d60 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f04e4001860 tx=0x7f04e401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2215124c2274d43a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.275+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e40ac640 con 0x7f04e0003980 2023-10-30T14:12:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.275+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e40ac7a0 con 0x7f04e0003980 2023-10-30T14:12:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.275+0000 7f04ef7fe640 1 -- 172.21.15.46:0/154459530 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e40acb90 con 0x7f04e0003980 2023-10-30T14:12:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.275+0000 7f04f6fc4640 1 -- 172.21.15.46:0/154459530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0003980 msgr2=0x7f04e0003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.275+0000 7f04f6fc4640 1 --2- 172.21.15.46:0/154459530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0003980 0x7f04e0003d60 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f04e4001860 tx=0x7f04e401c4e0 comp rx=0 tx=0).stop 2023-10-30T14:12:26.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.275+0000 7f04f6fc4640 1 -- 172.21.15.46:0/154459530 shutdown_connections 2023-10-30T14:12:26.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.275+0000 7f04f6fc4640 1 --2- 172.21.15.46:0/154459530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04e0003980 0x7f04e0003d60 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:26.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.275+0000 7f04f6fc4640 1 -- 172.21.15.46:0/154459530 >> 172.21.15.46:0/154459530 conn(0x7f04f00c7910 msgr2=0x7f04f00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:26.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.275+0000 7f04f6fc4640 1 -- 172.21.15.46:0/154459530 shutdown_connections 2023-10-30T14:12:26.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.275+0000 7f04f6fc4640 1 -- 172.21.15.46:0/154459530 wait complete. 2023-10-30T14:12:26.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04f6fc4640 1 Processor -- start 2023-10-30T14:12:26.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04f6fc4640 1 -- start start 2023-10-30T14:12:26.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04f6fc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e0003980 0x7f04f01f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:26.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04f6fc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04f01edd70 0x7f04f00c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04f6fc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f0069e90 0x7f04f006a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04f6fc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f00ca140 con 0x7f04f01edd70 2023-10-30T14:12:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04f6fc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f00c8c10 con 0x7f04f0069e90 2023-10-30T14:12:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04f6fc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f01ee1d0 con 0x7f04e0003980 2023-10-30T14:12:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04f01edd70 0x7f04f00c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04f01edd70 0x7f04f00c9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52266/0 (socket says 172.21.15.46:52266) 2023-10-30T14:12:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04effff640 1 -- 172.21.15.46:0/2525737292 learned_addr learned my addr 172.21.15.46:0/2525737292 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04f4d60640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e0003980 0x7f04f01f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04f5561640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f0069e90 0x7f04f006a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04effff640 1 -- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e0003980 msgr2=0x7f04f01f1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:26.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04effff640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e0003980 0x7f04f01f1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:26.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04effff640 1 -- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f0069e90 msgr2=0x7f04f006a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:26.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04effff640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f0069e90 0x7f04f006a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:26.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04effff640 1 -- 172.21.15.46:0/2525737292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f00ca570 con 0x7f04f01edd70 2023-10-30T14:12:26.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04effff640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04f01edd70 0x7f04f00c9c00 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0119b0 tx=0x7f04dc011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:26.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04ed7fa640 1 -- 172.21.15.46:0/2525737292 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc004240 con 0x7f04f01edd70 2023-10-30T14:12:26.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04ed7fa640 1 -- 172.21.15.46:0/2525737292 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc0043a0 con 0x7f04f01edd70 2023-10-30T14:12:26.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04f6fc4640 1 -- 172.21.15.46:0/2525737292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f00ca850 con 0x7f04f01edd70 2023-10-30T14:12:26.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.279+0000 7f04ed7fa640 1 -- 172.21.15.46:0/2525737292 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc00fb00 con 0x7f04f01edd70 2023-10-30T14:12:26.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.283+0000 7f04f6fc4640 1 -- 172.21.15.46:0/2525737292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f01f12e0 con 0x7f04f01edd70 2023-10-30T14:12:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.283+0000 7f04f6fc4640 1 -- 172.21.15.46:0/2525737292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f004e170 con 0x7f04f01edd70 2023-10-30T14:12:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.283+0000 7f04ed7fa640 1 -- 172.21.15.46:0/2525737292 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f04dc00fc60 con 0x7f04f01edd70 2023-10-30T14:12:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.283+0000 7f04ed7fa640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04d40664a0 0x7f04d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.283+0000 7f04ed7fa640 1 -- 172.21.15.46:0/2525737292 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f04dc084e40 con 0x7f04f01edd70 2023-10-30T14:12:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.283+0000 7f04f4d60640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04d40664a0 0x7f04d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.287+0000 7f04f4d60640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04d40664a0 0x7f04d4068960 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f04e40ae000 tx=0x7f04e40a6070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:26.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.287+0000 7f04ed7fa640 1 -- 172.21.15.46:0/2525737292 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f04dc051960 con 0x7f04f01edd70 2023-10-30T14:12:26.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 -- 172.21.15.46:0/2525737292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04f0071dc0 con 0x7f04f01edd70 2023-10-30T14:12:26.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04ed7fa640 1 -- 172.21.15.46:0/2525737292 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+5728 (secure 0 0 0) 0x7f04dc023020 con 0x7f04f01edd70 2023-10-30T14:12:26.480 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:26.481 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":210,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:24.575078+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:resolve","state_seq":31,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:26.481 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 210 2023-10-30T14:12:26.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 -- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04d40664a0 msgr2=0x7f04d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:26.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04d40664a0 0x7f04d4068960 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f04e40ae000 tx=0x7f04e40a6070 comp rx=0 tx=0).stop 2023-10-30T14:12:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 -- 172.21.15.46:0/2525737292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04f01edd70 msgr2=0x7f04f00c9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04f01edd70 0x7f04f00c9c00 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0119b0 tx=0x7f04dc011e70 comp rx=0 tx=0).stop 2023-10-30T14:12:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 -- 172.21.15.46:0/2525737292 shutdown_connections 2023-10-30T14:12:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f04d40664a0 0x7f04d4068960 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f0069e90 0x7f04f006a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04f01edd70 0x7f04f00c9c00 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 --2- 172.21.15.46:0/2525737292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e0003980 0x7f04f01f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 -- 172.21.15.46:0/2525737292 >> 172.21.15.46:0/2525737292 conn(0x7f04f00c7910 msgr2=0x7f04f01373f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 -- 172.21.15.46:0/2525737292 shutdown_connections 2023-10-30T14:12:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:26.479+0000 7f04f6fc4640 1 -- 172.21.15.46:0/2525737292 wait complete. 2023-10-30T14:12:26.502 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:12:27.783 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:12:27.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:12:28.502 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.655+0000 7f236c3aa640 1 Processor -- start 2023-10-30T14:12:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.655+0000 7f236c3aa640 1 -- start start 2023-10-30T14:12:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.655+0000 7f236c3aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2364153bf0 0x7f2364071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.655+0000 7f236c3aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23640c6fe0 0x7f23640c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.655+0000 7f236c3aa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23640700b0 con 0x7f23640c79e0 2023-10-30T14:12:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.655+0000 7f236c3aa640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23640c6810 con 0x7f2364153bf0 2023-10-30T14:12:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.655+0000 7f236c3aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2364159900 con 0x7f23640c6fe0 2023-10-30T14:12:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.659+0000 7f236a146640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2364153bf0 0x7f2364071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.659+0000 7f236a146640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2364153bf0 0x7f2364071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33500/0 (socket says 172.21.15.46:33500) 2023-10-30T14:12:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.659+0000 7f236a146640 1 -- 172.21.15.46:0/3126223802 learned_addr learned my addr 172.21.15.46:0/3126223802 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.659+0000 7f2369945640 1 --2- 172.21.15.46:0/3126223802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23640c6fe0 0x7f23640c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.659+0000 7f236a146640 1 -- 172.21.15.46:0/3126223802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23640c6fe0 msgr2=0x7f23640c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.659+0000 7f236a146640 1 --2- 172.21.15.46:0/3126223802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23640c6fe0 0x7f23640c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.659+0000 7f236a146640 1 -- 172.21.15.46:0/3126223802 >> v1:172.21.15.154:6789/0 conn(0x7f23640c79e0 legacy=0x7f23640cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.659+0000 7f236a146640 1 -- 172.21.15.46:0/3126223802 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23640c6530 con 0x7f2364153bf0 2023-10-30T14:12:28.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.659+0000 7f236a146640 1 --2- 172.21.15.46:0/3126223802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2364153bf0 0x7f2364071260 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f2360009870 tx=0x7f236001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17370d93e1dd023f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:28.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.659+0000 7f2369144640 1 -- 172.21.15.46:0/3126223802 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23600a89b0 con 0x7f2364153bf0 2023-10-30T14:12:28.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.659+0000 7f2369144640 1 -- 172.21.15.46:0/3126223802 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23600a8b10 con 0x7f2364153bf0 2023-10-30T14:12:28.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.659+0000 7f2369144640 1 -- 172.21.15.46:0/3126223802 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23600a8ea0 con 0x7f2364153bf0 2023-10-30T14:12:28.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 -- 172.21.15.46:0/3126223802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2364153bf0 msgr2=0x7f2364071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:28.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 --2- 172.21.15.46:0/3126223802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2364153bf0 0x7f2364071260 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f2360009870 tx=0x7f236001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:12:28.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 -- 172.21.15.46:0/3126223802 shutdown_connections 2023-10-30T14:12:28.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 --2- 172.21.15.46:0/3126223802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f23640c6fe0 0x7f23640c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:28.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 --2- 172.21.15.46:0/3126223802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2364153bf0 0x7f2364071260 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:28.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 -- 172.21.15.46:0/3126223802 >> 172.21.15.46:0/3126223802 conn(0x7f236406c6e0 msgr2=0x7f2364069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:28.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 -- 172.21.15.46:0/3126223802 shutdown_connections 2023-10-30T14:12:28.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 -- 172.21.15.46:0/3126223802 wait complete. 2023-10-30T14:12:28.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 Processor -- start 2023-10-30T14:12:28.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 -- start start 2023-10-30T14:12:28.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23640c6fe0 0x7f23641504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:28.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23640c79e0 0x7f23641509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:28.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2364153bf0 0x7f23641f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:28.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236414e2b0 con 0x7f23640c6fe0 2023-10-30T14:12:28.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23640c5260 con 0x7f23640c79e0 2023-10-30T14:12:28.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236c3aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23640c5520 con 0x7f2364153bf0 2023-10-30T14:12:28.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236a146640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23640c6fe0 0x7f23641504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:28.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236a146640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23640c6fe0 0x7f23641504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33506/0 (socket says 172.21.15.46:33506) 2023-10-30T14:12:28.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236a146640 1 -- 172.21.15.46:0/983920806 learned_addr learned my addr 172.21.15.46:0/983920806 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:28.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f2369945640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23640c79e0 0x7f23641509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:28.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236a947640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2364153bf0 0x7f23641f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:28.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236a146640 1 -- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2364153bf0 msgr2=0x7f23641f2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:28.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236a146640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2364153bf0 0x7f23641f2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:28.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236a146640 1 -- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23640c79e0 msgr2=0x7f23641509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:28.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236a146640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23640c79e0 0x7f23641509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:28.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.663+0000 7f236a146640 1 -- 172.21.15.46:0/983920806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23600094d0 con 0x7f23640c6fe0 2023-10-30T14:12:28.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.667+0000 7f236a146640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23640c6fe0 0x7f23641504a0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f23600b3040 tx=0x7f236001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:28.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.667+0000 7f235affd640 1 -- 172.21.15.46:0/983920806 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23600ba600 con 0x7f23640c6fe0 2023-10-30T14:12:28.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.667+0000 7f235affd640 1 -- 172.21.15.46:0/983920806 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23600ba780 con 0x7f23640c6fe0 2023-10-30T14:12:28.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.667+0000 7f235affd640 1 -- 172.21.15.46:0/983920806 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23600bab30 con 0x7f23640c6fe0 2023-10-30T14:12:28.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.667+0000 7f236c3aa640 1 -- 172.21.15.46:0/983920806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236414ef00 con 0x7f23640c6fe0 2023-10-30T14:12:28.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.667+0000 7f236c3aa640 1 -- 172.21.15.46:0/983920806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236414df40 con 0x7f23640c6fe0 2023-10-30T14:12:28.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.667+0000 7f236c3aa640 1 -- 172.21.15.46:0/983920806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236404e170 con 0x7f23640c6fe0 2023-10-30T14:12:28.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.667+0000 7f235affd640 1 -- 172.21.15.46:0/983920806 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f23600bac90 con 0x7f23640c6fe0 2023-10-30T14:12:28.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.667+0000 7f235affd640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23480664a0 0x7f2348068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:28.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.667+0000 7f235affd640 1 -- 172.21.15.46:0/983920806 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f2360122e30 con 0x7f23640c6fe0 2023-10-30T14:12:28.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.667+0000 7f2369945640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23480664a0 0x7f2348068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:28.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.671+0000 7f2369945640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23480664a0 0x7f2348068960 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f23640c26a0 tx=0x7f2354008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:28.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.671+0000 7f235affd640 1 -- 172.21.15.46:0/983920806 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23600ef830 con 0x7f23640c6fe0 2023-10-30T14:12:28.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.859+0000 7f236c3aa640 1 -- 172.21.15.46:0/983920806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f236404e170 con 0x7f23640c6fe0 2023-10-30T14:12:28.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f235affd640 1 -- 172.21.15.46:0/983920806 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+5728 (secure 0 0 0) 0x7f23600b5070 con 0x7f23640c6fe0 2023-10-30T14:12:28.864 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:28.864 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":210,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:24.575078+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:resolve","state_seq":31,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:28.864 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 210 2023-10-30T14:12:28.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f236c3aa640 1 -- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23480664a0 msgr2=0x7f2348068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:28.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f236c3aa640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23480664a0 0x7f2348068960 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f23640c26a0 tx=0x7f2354008040 comp rx=0 tx=0).stop 2023-10-30T14:12:28.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f236c3aa640 1 -- 172.21.15.46:0/983920806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23640c6fe0 msgr2=0x7f23641504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:28.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f236c3aa640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23640c6fe0 0x7f23641504a0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f23600b3040 tx=0x7f236001ba90 comp rx=0 tx=0).stop 2023-10-30T14:12:28.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f236c3aa640 1 -- 172.21.15.46:0/983920806 shutdown_connections 2023-10-30T14:12:28.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f236c3aa640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f23480664a0 0x7f2348068960 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:28.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f236c3aa640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2364153bf0 0x7f23641f2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:28.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f236c3aa640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23640c79e0 0x7f23641509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:28.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f236c3aa640 1 --2- 172.21.15.46:0/983920806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23640c6fe0 0x7f23641504a0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:28.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f236c3aa640 1 -- 172.21.15.46:0/983920806 >> 172.21.15.46:0/983920806 conn(0x7f236406c6e0 msgr2=0x7f2364158e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:28.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f236c3aa640 1 -- 172.21.15.46:0/983920806 shutdown_connections 2023-10-30T14:12:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:28.863+0000 7f236c3aa640 1 -- 172.21.15.46:0/983920806 wait complete. 2023-10-30T14:12:28.885 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 210, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 210, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8826": { "addr": "172.21.15.46:6836/66265765", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 66265765, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 66265765, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 8826, "incarnation": 203, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 17 }, "gid_8982": { "addr": "172.21.15.46:6833/2455118950", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2455118950, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2455118950, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8982, "incarnation": 209, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:resolve", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:12:24.575078+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8826, "mds_1": 8982 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:12:30.886 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:31.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.031+0000 7f10e5209640 1 Processor -- start 2023-10-30T14:12:31.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.031+0000 7f10e5209640 1 -- start start 2023-10-30T14:12:31.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10e5209640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f10e0151800 0x7f10e006e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:31.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10e5209640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e00c1ee0 0x7f10e00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:31.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10e5209640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e006e090 con 0x7f10e00c9230 2023-10-30T14:12:31.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10e5209640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e006dc70 con 0x7f10e0151800 2023-10-30T14:12:31.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10e5209640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00c8a50 con 0x7f10e00c1ee0 2023-10-30T14:12:31.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10df59e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f10e00c9230 0x7f10e00cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44642/0 (socket says 172.21.15.46:44642) 2023-10-30T14:12:31.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10df59e640 1 -- 172.21.15.46:0/1792204149 learned_addr learned my addr 172.21.15.46:0/1792204149 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:31.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10ded9d640 1 --2- 172.21.15.46:0/1792204149 >> v2:172.21.15.154:3300/0 conn(0x7f10e0151800 0x7f10e006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:31.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10d7fff640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e00c1ee0 0x7f10e00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:31.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237460720 0 0) 0x7f10e006e090 con 0x7f10e00c9230 2023-10-30T14:12:31.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc003530 con 0x7f10e00c9230 2023-10-30T14:12:31.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1199858926 0 0) 0x7f10cc003530 con 0x7f10e00c9230 2023-10-30T14:12:31.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e00c1ee0 msgr2=0x7f10e00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:31.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e00c1ee0 0x7f10e00c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:31.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 >> v2:172.21.15.154:3300/0 conn(0x7f10e0151800 msgr2=0x7f10e006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:31.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 --2- 172.21.15.46:0/1792204149 >> v2:172.21.15.154:3300/0 conn(0x7f10e0151800 0x7f10e006e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:31.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e00c3b10 con 0x7f10e00c9230 2023-10-30T14:12:31.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f10d8001a20 con 0x7f10e00c9230 2023-10-30T14:12:31.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 >> v1:172.21.15.46:6789/0 conn(0x7f10e00c9230 legacy=0x7f10e00cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:31.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e00c1ee0 0x7f10cc003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:31.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10cc003f00 0x7f10cc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:31.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10cc004d10 0x7f10cc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:31.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e006dc70 con 0x7f10cc003f00 2023-10-30T14:12:31.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00c8a50 con 0x7f10cc004d10 2023-10-30T14:12:31.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc00b990 con 0x7f10e00c1ee0 2023-10-30T14:12:31.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10d7fff640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10cc003f00 0x7f10cc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:31.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10ded9d640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e00c1ee0 0x7f10cc003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:31.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10df59e640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10cc004d10 0x7f10cc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:31.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10d7fff640 1 -- 172.21.15.46:0/1792204149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e00c1ee0 msgr2=0x7f10cc003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:31.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10d7fff640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e00c1ee0 0x7f10cc003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:31.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10d7fff640 1 -- 172.21.15.46:0/1792204149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10cc004d10 msgr2=0x7f10cc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:31.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10d7fff640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10cc004d10 0x7f10cc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10d7fff640 1 -- 172.21.15.46:0/1792204149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc003530 con 0x7f10cc003f00 2023-10-30T14:12:31.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.035+0000 7f10df59e640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10cc004d10 0x7f10cc005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:12:31.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.039+0000 7f10d7fff640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10cc003f00 0x7f10cc0047d0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009ab0 tx=0x7f10c801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195f122dfbc926f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:31.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.039+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c80a5620 con 0x7f10cc003f00 2023-10-30T14:12:31.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.039+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c80a5780 con 0x7f10cc003f00 2023-10-30T14:12:31.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.039+0000 7f10de59c640 1 -- 172.21.15.46:0/1792204149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c80a5b30 con 0x7f10cc003f00 2023-10-30T14:12:31.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.039+0000 7f10e5209640 1 -- 172.21.15.46:0/1792204149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10cc003f00 msgr2=0x7f10cc0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:31.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.039+0000 7f10e5209640 1 --2- 172.21.15.46:0/1792204149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10cc003f00 0x7f10cc0047d0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009ab0 tx=0x7f10c801c9d0 comp rx=0 tx=0).stop 2023-10-30T14:12:31.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.039+0000 7f10df59e640 1 -- 172.21.15.46:0/1792204149 reap_dead start 2023-10-30T14:12:31.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.039+0000 7f10e5209640 1 -- 172.21.15.46:0/1792204149 shutdown_connections 2023-10-30T14:12:31.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.039+0000 7f10e5209640 1 -- 172.21.15.46:0/1792204149 >> 172.21.15.46:0/1792204149 conn(0x7f10e0071ea0 msgr2=0x7f10e0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:31.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.039+0000 7f10e5209640 1 -- 172.21.15.46:0/1792204149 shutdown_connections 2023-10-30T14:12:31.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.039+0000 7f10e5209640 1 -- 172.21.15.46:0/1792204149 wait complete. 2023-10-30T14:12:31.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10e5209640 1 Processor -- start 2023-10-30T14:12:31.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10e5209640 1 -- start start 2023-10-30T14:12:31.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10e5209640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10cc003f00 0x7f10e01e5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:31.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10e5209640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e014ecf0 0x7f10e00cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:31.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10e5209640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e00cb2b0 0x7f10e01e2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:31.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10e5209640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e01e5e30 con 0x7f10e014ecf0 2023-10-30T14:12:31.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10e5209640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e01ddce0 con 0x7f10e00cb2b0 2023-10-30T14:12:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10e5209640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00c5c20 con 0x7f10cc003f00 2023-10-30T14:12:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e014ecf0 0x7f10e00cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e014ecf0 0x7f10e00cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33528/0 (socket says 172.21.15.46:33528) 2023-10-30T14:12:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10d7fff640 1 -- 172.21.15.46:0/1032763733 learned_addr learned my addr 172.21.15.46:0/1032763733 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10ded9d640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10cc003f00 0x7f10e01e5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10df59e640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e00cb2b0 0x7f10e01e2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10d7fff640 1 -- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10cc003f00 msgr2=0x7f10e01e5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10d7fff640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10cc003f00 0x7f10e01e5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10d7fff640 1 -- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e00cb2b0 msgr2=0x7f10e01e2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10d7fff640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e00cb2b0 0x7f10e01e2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:31.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10d7fff640 1 -- 172.21.15.46:0/1032763733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e00c4d30 con 0x7f10e014ecf0 2023-10-30T14:12:31.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10df59e640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e00cb2b0 0x7f10e01e2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:12:31.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10d7fff640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e014ecf0 0x7f10e00cad70 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009ab0 tx=0x7f10cc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:31.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10d77fe640 1 -- 172.21.15.46:0/1032763733 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c80ad850 con 0x7f10e014ecf0 2023-10-30T14:12:31.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10d77fe640 1 -- 172.21.15.46:0/1032763733 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c80ad9b0 con 0x7f10e014ecf0 2023-10-30T14:12:31.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10d77fe640 1 -- 172.21.15.46:0/1032763733 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c80add20 con 0x7f10e014ecf0 2023-10-30T14:12:31.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10e5209640 1 -- 172.21.15.46:0/1032763733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e00c5010 con 0x7f10e014ecf0 2023-10-30T14:12:31.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.043+0000 7f10e5209640 1 -- 172.21.15.46:0/1032763733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e00c55d0 con 0x7f10e014ecf0 2023-10-30T14:12:31.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.047+0000 7f10e5209640 1 -- 172.21.15.46:0/1032763733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e004e170 con 0x7f10e014ecf0 2023-10-30T14:12:31.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.047+0000 7f10d77fe640 1 -- 172.21.15.46:0/1032763733 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f10c80b4040 con 0x7f10e014ecf0 2023-10-30T14:12:31.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.047+0000 7f10d77fe640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f10c00664d0 0x7f10c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:31.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.047+0000 7f10d77fe640 1 -- 172.21.15.46:0/1032763733 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f10c811ee40 con 0x7f10e014ecf0 2023-10-30T14:12:31.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.047+0000 7f10ded9d640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f10c00664d0 0x7f10c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:31.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.047+0000 7f10ded9d640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f10c00664d0 0x7f10c0068990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009c60 tx=0x7f10d0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:31.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.051+0000 7f10d77fe640 1 -- 172.21.15.46:0/1032763733 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f10c80eb960 con 0x7f10e014ecf0 2023-10-30T14:12:31.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.243+0000 7f10e5209640 1 -- 172.21.15.46:0/1032763733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10e00c6de0 con 0x7f10e014ecf0 2023-10-30T14:12:31.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.243+0000 7f10d77fe640 1 -- 172.21.15.46:0/1032763733 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f10c80b2020 con 0x7f10e014ecf0 2023-10-30T14:12:31.247 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:31.248 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:30.024025+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:31.248 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 211 2023-10-30T14:12:31.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.247+0000 7f10e5209640 1 -- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f10c00664d0 msgr2=0x7f10c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:31.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.247+0000 7f10e5209640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f10c00664d0 0x7f10c0068990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009c60 tx=0x7f10d0009190 comp rx=0 tx=0).stop 2023-10-30T14:12:31.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.247+0000 7f10e5209640 1 -- 172.21.15.46:0/1032763733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e014ecf0 msgr2=0x7f10e00cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.247+0000 7f10e5209640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e014ecf0 0x7f10e00cad70 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009ab0 tx=0x7f10cc005800 comp rx=0 tx=0).stop 2023-10-30T14:12:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.247+0000 7f10e5209640 1 -- 172.21.15.46:0/1032763733 shutdown_connections 2023-10-30T14:12:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.247+0000 7f10e5209640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f10c00664d0 0x7f10c0068990 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.247+0000 7f10e5209640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e00cb2b0 0x7f10e01e2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:31.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.247+0000 7f10e5209640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e014ecf0 0x7f10e00cad70 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.247+0000 7f10e5209640 1 --2- 172.21.15.46:0/1032763733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10cc003f00 0x7f10e01e5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.247+0000 7f10e5209640 1 -- 172.21.15.46:0/1032763733 >> 172.21.15.46:0/1032763733 conn(0x7f10e0071ea0 msgr2=0x7f10e006b140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.247+0000 7f10e5209640 1 -- 172.21.15.46:0/1032763733 shutdown_connections 2023-10-30T14:12:31.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:31.247+0000 7f10e5209640 1 -- 172.21.15.46:0/1032763733 wait complete. 2023-10-30T14:12:31.269 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:12:33.269 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f895c679640 1 Processor -- start 2023-10-30T14:12:33.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f895c679640 1 -- start start 2023-10-30T14:12:33.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f895c679640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89540c6bd0 0x7f895406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:33.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f895c679640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8954071110 0x7f8954158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:33.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f895c679640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8954066cf0 con 0x7f895406f7a0 2023-10-30T14:12:33.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f895c679640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89540700c0 con 0x7f8954071110 2023-10-30T14:12:33.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f895c679640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89540c70d0 con 0x7f89540c6bd0 2023-10-30T14:12:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f895a415640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f895406f7a0 0x7f8954069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44644/0 (socket says 172.21.15.46:44644) 2023-10-30T14:12:33.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f895a415640 1 -- 172.21.15.46:0/1763908572 learned_addr learned my addr 172.21.15.46:0/1763908572 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:33.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959c14640 1 --2- 172.21.15.46:0/1763908572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89540c6bd0 0x7f895406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:33.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f895ac16640 1 --2- 172.21.15.46:0/1763908572 >> v2:172.21.15.154:3300/0 conn(0x7f8954071110 0x7f8954158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:33.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235916359 0 0) 0x7f8954066cf0 con 0x7f895406f7a0 2023-10-30T14:12:33.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8940003530 con 0x7f895406f7a0 2023-10-30T14:12:33.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 767649103 0 0) 0x7f8940003530 con 0x7f895406f7a0 2023-10-30T14:12:33.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89540c6bd0 msgr2=0x7f895406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:33.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 --2- 172.21.15.46:0/1763908572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89540c6bd0 0x7f895406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:33.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 >> v2:172.21.15.154:3300/0 conn(0x7f8954071110 msgr2=0x7f8954158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:33.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 --2- 172.21.15.46:0/1763908572 >> v2:172.21.15.154:3300/0 conn(0x7f8954071110 0x7f8954158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:33.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89540c7390 con 0x7f895406f7a0 2023-10-30T14:12:33.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f8950001680 con 0x7f895406f7a0 2023-10-30T14:12:33.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 >> v1:172.21.15.46:6789/0 conn(0x7f895406f7a0 legacy=0x7f8954069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:33.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 --2- 172.21.15.46:0/1763908572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89540c6bd0 0x7f8940003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:33.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 --2- 172.21.15.46:0/1763908572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8940003f00 0x7f89400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 --2- 172.21.15.46:0/1763908572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940004d10 0x7f8940005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89540700c0 con 0x7f8940004d10 2023-10-30T14:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89540c70d0 con 0x7f8940003f00 2023-10-30T14:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.423+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894000b9c0 con 0x7f89540c6bd0 2023-10-30T14:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.427+0000 7f895ac16640 1 --2- 172.21.15.46:0/1763908572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940004d10 0x7f8940005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.427+0000 7f8959c14640 1 --2- 172.21.15.46:0/1763908572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8940003f00 0x7f89400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.427+0000 7f895a415640 1 --2- 172.21.15.46:0/1763908572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89540c6bd0 0x7f8940003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.427+0000 7f895ac16640 1 -- 172.21.15.46:0/1763908572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89540c6bd0 msgr2=0x7f8940003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.427+0000 7f895ac16640 1 --2- 172.21.15.46:0/1763908572 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f89540c6bd0 0x7f8940003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.427+0000 7f895ac16640 1 -- 172.21.15.46:0/1763908572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8940003f00 msgr2=0x7f89400047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.427+0000 7f895ac16640 1 --2- 172.21.15.46:0/1763908572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8940003f00 0x7f89400047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.427+0000 7f895ac16640 1 -- 172.21.15.46:0/1763908572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8940003530 con 0x7f8940004d10 2023-10-30T14:12:33.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.427+0000 7f895ac16640 1 --2- 172.21.15.46:0/1763908572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940004d10 0x7f8940005190 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f894c009fc0 tx=0x7f894c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64d7d89f9a743ca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.427+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894c0a5cb0 con 0x7f8940004d10 2023-10-30T14:12:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.427+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894c00f040 con 0x7f8940004d10 2023-10-30T14:12:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.427+0000 7f8959413640 1 -- 172.21.15.46:0/1763908572 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894c0ad680 con 0x7f8940004d10 2023-10-30T14:12:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 -- 172.21.15.46:0/1763908572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940004d10 msgr2=0x7f8940005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 --2- 172.21.15.46:0/1763908572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940004d10 0x7f8940005190 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f894c009fc0 tx=0x7f894c01cca0 comp rx=0 tx=0).stop 2023-10-30T14:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895ac16640 1 -- 172.21.15.46:0/1763908572 reap_dead start 2023-10-30T14:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 -- 172.21.15.46:0/1763908572 shutdown_connections 2023-10-30T14:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 -- 172.21.15.46:0/1763908572 >> 172.21.15.46:0/1763908572 conn(0x7f89540c7910 msgr2=0x7f89540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 -- 172.21.15.46:0/1763908572 shutdown_connections 2023-10-30T14:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 -- 172.21.15.46:0/1763908572 wait complete. 2023-10-30T14:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 Processor -- start 2023-10-30T14:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 -- start start 2023-10-30T14:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8940004d10 0x7f895415af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f895414ef80 0x7f895415b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8954158e70 0x7f89541592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895415b9f0 con 0x7f8954158e70 2023-10-30T14:12:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89541eaee0 con 0x7f8940004d10 2023-10-30T14:12:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895c679640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89541eb410 con 0x7f895414ef80 2023-10-30T14:12:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895ac16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8954158e70 0x7f89541592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895ac16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8954158e70 0x7f89541592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33548/0 (socket says 172.21.15.46:33548) 2023-10-30T14:12:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895ac16640 1 -- 172.21.15.46:0/146443506 learned_addr learned my addr 172.21.15.46:0/146443506 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895a415640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8940004d10 0x7f895415af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f8959c14640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f895414ef80 0x7f895415b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895ac16640 1 -- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f895414ef80 msgr2=0x7f895415b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895ac16640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f895414ef80 0x7f895415b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895ac16640 1 -- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8940004d10 msgr2=0x7f895415af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895ac16640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8940004d10 0x7f895415af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.431+0000 7f895ac16640 1 -- 172.21.15.46:0/146443506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89541589f0 con 0x7f8954158e70 2023-10-30T14:12:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.435+0000 7f895ac16640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8954158e70 0x7f89541592f0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f894c01c710 tx=0x7f894c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.435+0000 7f894affd640 1 -- 172.21.15.46:0/146443506 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894c0039a0 con 0x7f8954158e70 2023-10-30T14:12:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.435+0000 7f894affd640 1 -- 172.21.15.46:0/146443506 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894c003b20 con 0x7f8954158e70 2023-10-30T14:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.435+0000 7f894affd640 1 -- 172.21.15.46:0/146443506 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894c0ae890 con 0x7f8954158e70 2023-10-30T14:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.435+0000 7f895c679640 1 -- 172.21.15.46:0/146443506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8954151680 con 0x7f8954158e70 2023-10-30T14:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.435+0000 7f895c679640 1 -- 172.21.15.46:0/146443506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f895415a790 con 0x7f8954158e70 2023-10-30T14:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.435+0000 7f895c679640 1 -- 172.21.15.46:0/146443506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f895404e170 con 0x7f8954158e70 2023-10-30T14:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.435+0000 7f894affd640 1 -- 172.21.15.46:0/146443506 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f894c0b4040 con 0x7f8954158e70 2023-10-30T14:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.435+0000 7f894affd640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89380664c0 0x7f8938068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.435+0000 7f894affd640 1 -- 172.21.15.46:0/146443506 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f894c11e290 con 0x7f8954158e70 2023-10-30T14:12:33.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.435+0000 7f895a415640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89380664c0 0x7f8938068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:33.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.439+0000 7f895a415640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89380664c0 0x7f8938068980 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8950001060 tx=0x7f8950007900 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:33.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.439+0000 7f894affd640 1 -- 172.21.15.46:0/146443506 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f894c0eaf80 con 0x7f8954158e70 2023-10-30T14:12:33.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.627+0000 7f895c679640 1 -- 172.21.15.46:0/146443506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8954158620 con 0x7f8954158e70 2023-10-30T14:12:33.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.627+0000 7f894affd640 1 -- 172.21.15.46:0/146443506 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f894c0b2020 con 0x7f8954158e70 2023-10-30T14:12:33.630 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:33.631 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:30.024025+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:33.632 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 211 2023-10-30T14:12:33.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.631+0000 7f895c679640 1 -- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89380664c0 msgr2=0x7f8938068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:33.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.631+0000 7f895c679640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89380664c0 0x7f8938068980 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8950001060 tx=0x7f8950007900 comp rx=0 tx=0).stop 2023-10-30T14:12:33.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.631+0000 7f895c679640 1 -- 172.21.15.46:0/146443506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8954158e70 msgr2=0x7f89541592f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:33.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.631+0000 7f895c679640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8954158e70 0x7f89541592f0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f894c01c710 tx=0x7f894c009750 comp rx=0 tx=0).stop 2023-10-30T14:12:33.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.631+0000 7f895c679640 1 -- 172.21.15.46:0/146443506 shutdown_connections 2023-10-30T14:12:33.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.631+0000 7f895c679640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89380664c0 0x7f8938068980 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:33.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.631+0000 7f895c679640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8954158e70 0x7f89541592f0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:33.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.631+0000 7f895c679640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f895414ef80 0x7f895415b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:33.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.631+0000 7f895c679640 1 --2- 172.21.15.46:0/146443506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8940004d10 0x7f895415af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:33.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.631+0000 7f895c679640 1 -- 172.21.15.46:0/146443506 >> 172.21.15.46:0/146443506 conn(0x7f89540c7910 msgr2=0x7f89540c26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:33.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.631+0000 7f895c679640 1 -- 172.21.15.46:0/146443506 shutdown_connections 2023-10-30T14:12:33.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:33.631+0000 7f895c679640 1 -- 172.21.15.46:0/146443506 wait complete. 2023-10-30T14:12:33.652 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:12:35.652 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:35.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.803+0000 7f5d931d3640 1 Processor -- start 2023-10-30T14:12:35.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.803+0000 7f5d931d3640 1 -- start start 2023-10-30T14:12:35.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.803+0000 7f5d931d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d8c069e90 0x7f5d8c06a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:35.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.803+0000 7f5d931d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c153d80 con 0x7f5d8c1539a0 2023-10-30T14:12:35.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.803+0000 7f5d931d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c0c5c50 con 0x7f5d8c070a00 2023-10-30T14:12:35.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.803+0000 7f5d931d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c06fbb0 con 0x7f5d8c069e90 2023-10-30T14:12:35.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d90f6f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5d8c1539a0 0x7f5d8c15a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44656/0 (socket says 172.21.15.46:44656) 2023-10-30T14:12:35.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d90f6f640 1 -- 172.21.15.46:0/3947420523 learned_addr learned my addr 172.21.15.46:0/3947420523 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:35.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d91770640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d8c069e90 0x7f5d8c06a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:35.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400628239 0 0) 0x7f5d8c153d80 con 0x7f5d8c1539a0 2023-10-30T14:12:35.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d7c003530 con 0x7f5d8c1539a0 2023-10-30T14:12:35.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759537141 0 0) 0x7f5d8c0c5c50 con 0x7f5d8c070a00 2023-10-30T14:12:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c153d80 con 0x7f5d8c070a00 2023-10-30T14:12:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 95810960 0 0) 0x7f5d7c003530 con 0x7f5d8c1539a0 2023-10-30T14:12:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d8c069e90 msgr2=0x7f5d8c06a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d8c069e90 0x7f5d8c06a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 >> v1:172.21.15.154:6789/0 conn(0x7f5d8c070a00 legacy=0x7f5d8c070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:35.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c0c5f30 con 0x7f5d8c1539a0 2023-10-30T14:12:35.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f5d80001680 con 0x7f5d8c1539a0 2023-10-30T14:12:35.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 >> v1:172.21.15.46:6789/0 conn(0x7f5d8c1539a0 legacy=0x7f5d8c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:35.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d7c003b40 0x7f5d7c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:35.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d7c0047d0 0x7f5d7c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:35.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d7c005190 0x7f5d7c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c06fbb0 con 0x7f5d7c005190 2023-10-30T14:12:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c00c000 con 0x7f5d7c0047d0 2023-10-30T14:12:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c00c1c0 con 0x7f5d7c003b40 2023-10-30T14:12:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d91770640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d8c069e90 0x7f5d8c06a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:12:35.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d90f6f640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d7c003b40 0x7f5d7c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:35.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d91770640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d7c005190 0x7f5d7c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:35.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d8bfff640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d7c0047d0 0x7f5d7c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:35.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d91770640 1 -- 172.21.15.46:0/3947420523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d7c003b40 msgr2=0x7f5d7c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:35.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d91770640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d7c003b40 0x7f5d7c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:35.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d91770640 1 -- 172.21.15.46:0/3947420523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d7c0047d0 msgr2=0x7f5d7c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:35.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d91770640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d7c0047d0 0x7f5d7c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:35.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.807+0000 7f5d91770640 1 -- 172.21.15.46:0/3947420523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c003530 con 0x7f5d7c005190 2023-10-30T14:12:35.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d91770640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d7c005190 0x7f5d7c00b990 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c06a270 tx=0x7f5d8401cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a026a220137a64 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:35.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d840a5d50 con 0x7f5d7c005190 2023-10-30T14:12:35.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8400f040 con 0x7f5d7c005190 2023-10-30T14:12:35.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d8b7fe640 1 -- 172.21.15.46:0/3947420523 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d84003bc0 con 0x7f5d7c005190 2023-10-30T14:12:35.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d931d3640 1 -- 172.21.15.46:0/3947420523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d7c005190 msgr2=0x7f5d7c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:35.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d931d3640 1 --2- 172.21.15.46:0/3947420523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d7c005190 0x7f5d7c00b990 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c06a270 tx=0x7f5d8401cd00 comp rx=0 tx=0).stop 2023-10-30T14:12:35.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d91770640 1 -- 172.21.15.46:0/3947420523 reap_dead start 2023-10-30T14:12:35.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d931d3640 1 -- 172.21.15.46:0/3947420523 shutdown_connections 2023-10-30T14:12:35.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d931d3640 1 -- 172.21.15.46:0/3947420523 >> 172.21.15.46:0/3947420523 conn(0x7f5d8c0c7910 msgr2=0x7f5d8c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:35.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d931d3640 1 -- 172.21.15.46:0/3947420523 shutdown_connections 2023-10-30T14:12:35.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d931d3640 1 -- 172.21.15.46:0/3947420523 wait complete. 2023-10-30T14:12:35.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d931d3640 1 Processor -- start 2023-10-30T14:12:35.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.811+0000 7f5d931d3640 1 -- start start 2023-10-30T14:12:35.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d931d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d7c005190 0x7f5d8c1499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:35.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d931d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d8c14a6d0 0x7f5d8c14ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:35.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d931d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d8c1553d0 0x7f5d8c155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:35.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d931d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c1f3d50 con 0x7f5d7c005190 2023-10-30T14:12:35.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d931d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c1f31d0 con 0x7f5d8c14a6d0 2023-10-30T14:12:35.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d931d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c1f34b0 con 0x7f5d8c1553d0 2023-10-30T14:12:35.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d90f6f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d7c005190 0x7f5d8c1499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:35.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d90f6f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d7c005190 0x7f5d8c1499c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33568/0 (socket says 172.21.15.46:33568) 2023-10-30T14:12:35.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d90f6f640 1 -- 172.21.15.46:0/3884890321 learned_addr learned my addr 172.21.15.46:0/3884890321 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:35.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d91770640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d8c1553d0 0x7f5d8c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:35.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d8bfff640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d8c14a6d0 0x7f5d8c14ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:35.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d90f6f640 1 -- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d8c1553d0 msgr2=0x7f5d8c155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:35.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d90f6f640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d8c1553d0 0x7f5d8c155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:35.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d90f6f640 1 -- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d8c14a6d0 msgr2=0x7f5d8c14ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:35.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d90f6f640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d8c14a6d0 0x7f5d8c14ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:35.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d90f6f640 1 -- 172.21.15.46:0/3884890321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c1f3790 con 0x7f5d7c005190 2023-10-30T14:12:35.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d91770640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d8c1553d0 0x7f5d8c155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:12:35.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d90f6f640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d7c005190 0x7f5d8c1499c0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f5d80001c50 tx=0x7f5d800074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:35.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d897fa640 1 -- 172.21.15.46:0/3884890321 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80001090 con 0x7f5d7c005190 2023-10-30T14:12:35.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d897fa640 1 -- 172.21.15.46:0/3884890321 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8000f780 con 0x7f5d7c005190 2023-10-30T14:12:35.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d897fa640 1 -- 172.21.15.46:0/3884890321 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8000fb30 con 0x7f5d7c005190 2023-10-30T14:12:35.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d931d3640 1 -- 172.21.15.46:0/3884890321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8c1f3a70 con 0x7f5d7c005190 2023-10-30T14:12:35.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d931d3640 1 -- 172.21.15.46:0/3884890321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8c0c1ad0 con 0x7f5d7c005190 2023-10-30T14:12:35.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.815+0000 7f5d931d3640 1 -- 172.21.15.46:0/3884890321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8c1541f0 con 0x7f5d7c005190 2023-10-30T14:12:35.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.819+0000 7f5d897fa640 1 -- 172.21.15.46:0/3884890321 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5d8001d460 con 0x7f5d7c005190 2023-10-30T14:12:35.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.819+0000 7f5d897fa640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d700664a0 0x7f5d70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:35.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.819+0000 7f5d897fa640 1 -- 172.21.15.46:0/3884890321 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f5d80084c10 con 0x7f5d7c005190 2023-10-30T14:12:35.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.819+0000 7f5d8bfff640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d700664a0 0x7f5d70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:35.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.819+0000 7f5d8bfff640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d700664a0 0x7f5d70068960 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c157400 tx=0x7f5d78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:35.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:35.823+0000 7f5d897fa640 1 -- 172.21.15.46:0/3884890321 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5d8c1541f0 con 0x7f5d7c005190 2023-10-30T14:12:36.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.011+0000 7f5d931d3640 1 -- 172.21.15.46:0/3884890321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d8c1541f0 con 0x7f5d7c005190 2023-10-30T14:12:36.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.011+0000 7f5d897fa640 1 -- 172.21.15.46:0/3884890321 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f5d8c1541f0 con 0x7f5d7c005190 2023-10-30T14:12:36.014 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:36.015 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":212,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:35.647379+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:36.015 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 212 2023-10-30T14:12:36.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.015+0000 7f5d931d3640 1 -- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d700664a0 msgr2=0x7f5d70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:36.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.015+0000 7f5d931d3640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d700664a0 0x7f5d70068960 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c157400 tx=0x7f5d78009040 comp rx=0 tx=0).stop 2023-10-30T14:12:36.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.015+0000 7f5d931d3640 1 -- 172.21.15.46:0/3884890321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d7c005190 msgr2=0x7f5d8c1499c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:36.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.015+0000 7f5d931d3640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d7c005190 0x7f5d8c1499c0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f5d80001c50 tx=0x7f5d800074f0 comp rx=0 tx=0).stop 2023-10-30T14:12:36.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.015+0000 7f5d931d3640 1 -- 172.21.15.46:0/3884890321 shutdown_connections 2023-10-30T14:12:36.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.015+0000 7f5d931d3640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5d700664a0 0x7f5d70068960 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:36.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.015+0000 7f5d931d3640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d8c1553d0 0x7f5d8c155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:36.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.015+0000 7f5d931d3640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d8c14a6d0 0x7f5d8c14ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:36.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.015+0000 7f5d931d3640 1 --2- 172.21.15.46:0/3884890321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d7c005190 0x7f5d8c1499c0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:36.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.015+0000 7f5d931d3640 1 -- 172.21.15.46:0/3884890321 >> 172.21.15.46:0/3884890321 conn(0x7f5d8c0c7910 msgr2=0x7f5d8c0c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:36.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.015+0000 7f5d931d3640 1 -- 172.21.15.46:0/3884890321 shutdown_connections 2023-10-30T14:12:36.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:36.015+0000 7f5d931d3640 1 -- 172.21.15.46:0/3884890321 wait complete. 2023-10-30T14:12:36.036 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:12:38.037 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:38.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0c794640 1 Processor -- start 2023-10-30T14:12:38.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0c794640 1 -- start start 2023-10-30T14:12:38.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0c794640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4e040c05d0 0x7f4e0413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:38.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0c794640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e040c0a50 con 0x7f4e040c6b80 2023-10-30T14:12:38.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0c794640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0406f6e0 con 0x7f4e040c6f60 2023-10-30T14:12:38.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0c794640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0406f9c0 con 0x7f4e040c05d0 2023-10-30T14:12:38.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0a530640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4e040c6b80 0x7f4e040bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43290/0 (socket says 172.21.15.46:43290) 2023-10-30T14:12:38.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0a530640 1 -- 172.21.15.46:0/2515580666 learned_addr learned my addr 172.21.15.46:0/2515580666 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:38.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e09d2f640 1 --2- 172.21.15.46:0/2515580666 >> v2:172.21.15.154:3300/0 conn(0x7f4e040c05d0 0x7f4e0413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:38.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960362766 0 0) 0x7f4e040c0a50 con 0x7f4e040c6b80 2023-10-30T14:12:38.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df0005740 con 0x7f4e040c6b80 2023-10-30T14:12:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2203447660 0 0) 0x7f4df0005740 con 0x7f4e040c6b80 2023-10-30T14:12:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 >> v2:172.21.15.154:3300/0 conn(0x7f4e040c05d0 msgr2=0x7f4e0413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 --2- 172.21.15.46:0/2515580666 >> v2:172.21.15.154:3300/0 conn(0x7f4e040c05d0 0x7f4e0413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 >> v1:172.21.15.154:6789/0 conn(0x7f4e040c6f60 legacy=0x7f4e041566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:38.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0406fca0 con 0x7f4e040c6b80 2023-10-30T14:12:38.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f4e00001680 con 0x7f4e040c6b80 2023-10-30T14:12:38.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 >> v1:172.21.15.46:6789/0 conn(0x7f4e040c6b80 legacy=0x7f4e040bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:38.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 --2- 172.21.15.46:0/2515580666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4df0005b90 0x7f4df0005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:38.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 --2- 172.21.15.46:0/2515580666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4df00069e0 0x7f4df0006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:38.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 --2- 172.21.15.46:0/2515580666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df00073a0 0x7f4df000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:38.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0406f9c0 con 0x7f4df00073a0 2023-10-30T14:12:38.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df000e240 con 0x7f4df0005b90 2023-10-30T14:12:38.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df000e400 con 0x7f4df00069e0 2023-10-30T14:12:38.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0ad31640 1 --2- 172.21.15.46:0/2515580666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df00073a0 0x7f4df000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:38.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e0a530640 1 --2- 172.21.15.46:0/2515580666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4df0005b90 0x7f4df0005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:38.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.191+0000 7f4e09d2f640 1 --2- 172.21.15.46:0/2515580666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4df00069e0 0x7f4df0006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:38.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0ad31640 1 -- 172.21.15.46:0/2515580666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4df00069e0 msgr2=0x7f4df0006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:38.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0ad31640 1 --2- 172.21.15.46:0/2515580666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4df00069e0 0x7f4df0006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:38.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0ad31640 1 -- 172.21.15.46:0/2515580666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4df0005b90 msgr2=0x7f4df0005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:38.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0ad31640 1 --2- 172.21.15.46:0/2515580666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4df0005b90 0x7f4df0005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:38.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0ad31640 1 -- 172.21.15.46:0/2515580666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0005740 con 0x7f4df00073a0 2023-10-30T14:12:38.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0ad31640 1 -- 172.21.15.46:0/2515580666 reap_dead start 2023-10-30T14:12:38.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0ad31640 1 --2- 172.21.15.46:0/2515580666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df00073a0 0x7f4df000dba0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc001e70 tx=0x7f4dfc01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306fa74422dedaa3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:38.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc003b80 con 0x7f4df00073a0 2023-10-30T14:12:38.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dfc003d00 con 0x7f4df00073a0 2023-10-30T14:12:38.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0952e640 1 -- 172.21.15.46:0/2515580666 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc005260 con 0x7f4df00073a0 2023-10-30T14:12:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0c794640 1 -- 172.21.15.46:0/2515580666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df00073a0 msgr2=0x7f4df000dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0c794640 1 --2- 172.21.15.46:0/2515580666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df00073a0 0x7f4df000dba0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc001e70 tx=0x7f4dfc01c760 comp rx=0 tx=0).stop 2023-10-30T14:12:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0c794640 1 -- 172.21.15.46:0/2515580666 shutdown_connections 2023-10-30T14:12:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0c794640 1 --2- 172.21.15.46:0/2515580666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df00073a0 0x7f4df000dba0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0c794640 1 -- 172.21.15.46:0/2515580666 >> 172.21.15.46:0/2515580666 conn(0x7f4e040c7910 msgr2=0x7f4e040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:38.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0c794640 1 -- 172.21.15.46:0/2515580666 shutdown_connections 2023-10-30T14:12:38.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.195+0000 7f4e0c794640 1 -- 172.21.15.46:0/2515580666 wait complete. 2023-10-30T14:12:38.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e0c794640 1 Processor -- start 2023-10-30T14:12:38.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e0c794640 1 -- start start 2023-10-30T14:12:38.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e0c794640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4df00073a0 0x7f4e041553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:38.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e0c794640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e041543e0 0x7f4e0413dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:38.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e0c794640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e04153380 0x7f4e04152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:38.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e0c794640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e040b04e0 con 0x7f4e041543e0 2023-10-30T14:12:38.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e0c794640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e040c9ae0 con 0x7f4e04153380 2023-10-30T14:12:38.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e0c794640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e040c9dc0 con 0x7f4df00073a0 2023-10-30T14:12:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e09d2f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e041543e0 0x7f4e0413dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e09d2f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e041543e0 0x7f4e0413dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34846/0 (socket says 172.21.15.46:34846) 2023-10-30T14:12:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e09d2f640 1 -- 172.21.15.46:0/3378555282 learned_addr learned my addr 172.21.15.46:0/3378555282 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e0a530640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4df00073a0 0x7f4e041553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e0ad31640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e04153380 0x7f4e04152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e09d2f640 1 -- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4df00073a0 msgr2=0x7f4e041553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e09d2f640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4df00073a0 0x7f4e041553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e09d2f640 1 -- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e04153380 msgr2=0x7f4e04152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e09d2f640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e04153380 0x7f4e04152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e09d2f640 1 -- 172.21.15.46:0/3378555282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e040ca0a0 con 0x7f4e041543e0 2023-10-30T14:12:38.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e09d2f640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e041543e0 0x7f4e0413dad0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f4df4011960 tx=0x7f4df4011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:38.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4dfaffd640 1 -- 172.21.15.46:0/3378555282 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4011e90 con 0x7f4e041543e0 2023-10-30T14:12:38.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4dfaffd640 1 -- 172.21.15.46:0/3378555282 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df4004240 con 0x7f4e041543e0 2023-10-30T14:12:38.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4e0c794640 1 -- 172.21.15.46:0/3378555282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e040ca380 con 0x7f4e041543e0 2023-10-30T14:12:38.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.199+0000 7f4dfaffd640 1 -- 172.21.15.46:0/3378555282 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df400f820 con 0x7f4e041543e0 2023-10-30T14:12:38.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.203+0000 7f4e0c794640 1 -- 172.21.15.46:0/3378555282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e040ca940 con 0x7f4e041543e0 2023-10-30T14:12:38.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.203+0000 7f4e0c794640 1 -- 172.21.15.46:0/3378555282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e040cab00 con 0x7f4e041543e0 2023-10-30T14:12:38.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.203+0000 7f4dfaffd640 1 -- 172.21.15.46:0/3378555282 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4df400f980 con 0x7f4e041543e0 2023-10-30T14:12:38.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.203+0000 7f4dfaffd640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4de80664a0 0x7f4de8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:38.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.203+0000 7f4dfaffd640 1 -- 172.21.15.46:0/3378555282 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f4df4084d60 con 0x7f4e041543e0 2023-10-30T14:12:38.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.203+0000 7f4e0a530640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4de80664a0 0x7f4de8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:38.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.207+0000 7f4e0a530640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4de80664a0 0x7f4de8068960 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4e00007d30 tx=0x7f4e00000ee0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:38.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.207+0000 7f4dfaffd640 1 -- 172.21.15.46:0/3378555282 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4df40517f0 con 0x7f4e041543e0 2023-10-30T14:12:38.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 -- 172.21.15.46:0/3378555282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e0404e170 con 0x7f4e041543e0 2023-10-30T14:12:38.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4dfaffd640 1 -- 172.21.15.46:0/3378555282 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f4df4023020 con 0x7f4e041543e0 2023-10-30T14:12:38.404 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:38.404 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":212,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:35.647379+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:38.405 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 212 2023-10-30T14:12:38.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 -- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4de80664a0 msgr2=0x7f4de8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:38.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4de80664a0 0x7f4de8068960 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4e00007d30 tx=0x7f4e00000ee0 comp rx=0 tx=0).stop 2023-10-30T14:12:38.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 -- 172.21.15.46:0/3378555282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e041543e0 msgr2=0x7f4e0413dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:38.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e041543e0 0x7f4e0413dad0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f4df4011960 tx=0x7f4df4011d60 comp rx=0 tx=0).stop 2023-10-30T14:12:38.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 -- 172.21.15.46:0/3378555282 shutdown_connections 2023-10-30T14:12:38.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4de80664a0 0x7f4de8068960 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:38.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e04153380 0x7f4e04152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:38.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e041543e0 0x7f4e0413dad0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:38.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 --2- 172.21.15.46:0/3378555282 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4df00073a0 0x7f4e041553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:38.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 -- 172.21.15.46:0/3378555282 >> 172.21.15.46:0/3378555282 conn(0x7f4e040c7910 msgr2=0x7f4e0413bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:38.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 -- 172.21.15.46:0/3378555282 shutdown_connections 2023-10-30T14:12:38.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:38.403+0000 7f4e0c794640 1 -- 172.21.15.46:0/3378555282 wait complete. 2023-10-30T14:12:38.425 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:12:40.427 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:40.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.579+0000 7fd63d30f640 1 Processor -- start 2023-10-30T14:12:40.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.579+0000 7fd63d30f640 1 -- start start 2023-10-30T14:12:40.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.579+0000 7fd63d30f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd63806e470 0x7fd63806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:40.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.579+0000 7fd63d30f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63806afa0 con 0x7fd6380c8490 2023-10-30T14:12:40.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.579+0000 7fd63d30f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd638154150 con 0x7fd638153d70 2023-10-30T14:12:40.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.579+0000 7fd63d30f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63806d780 con 0x7fd63806e470 2023-10-30T14:12:40.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.579+0000 7fd6377fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd6380c8490 0x7fd6380c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43302/0 (socket says 172.21.15.46:43302) 2023-10-30T14:12:40.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.579+0000 7fd6377fe640 1 -- 172.21.15.46:0/3850129421 learned_addr learned my addr 172.21.15.46:0/3850129421 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:40.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd62ffff640 1 --2- 172.21.15.46:0/3850129421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd63806e470 0x7fd63806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:40.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853678648 0 0) 0x7fd63806afa0 con 0x7fd6380c8490 2023-10-30T14:12:40.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd624003530 con 0x7fd6380c8490 2023-10-30T14:12:40.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481318993 0 0) 0x7fd638154150 con 0x7fd638153d70 2023-10-30T14:12:40.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd63806afa0 con 0x7fd638153d70 2023-10-30T14:12:40.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1753074558 0 0) 0x7fd624003530 con 0x7fd6380c8490 2023-10-30T14:12:40.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd63806e470 msgr2=0x7fd63806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:40.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 --2- 172.21.15.46:0/3850129421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd63806e470 0x7fd63806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:40.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 >> v1:172.21.15.154:6789/0 conn(0x7fd638153d70 legacy=0x7fd6380cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:40.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63806e030 con 0x7fd6380c8490 2023-10-30T14:12:40.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fd630001a00 con 0x7fd6380c8490 2023-10-30T14:12:40.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 >> v1:172.21.15.46:6789/0 conn(0x7fd6380c8490 legacy=0x7fd6380c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:40.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 --2- 172.21.15.46:0/3850129421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd624003b40 0x7fd624003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:40.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 --2- 172.21.15.46:0/3850129421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6240047d0 0x7fd624004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:40.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 --2- 172.21.15.46:0/3850129421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd63806e470 0x7fd624005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:40.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63806d780 con 0x7fd624003b40 2023-10-30T14:12:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62400ba40 con 0x7fd6240047d0 2023-10-30T14:12:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62400bbd0 con 0x7fd63806e470 2023-10-30T14:12:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd636ffd640 1 --2- 172.21.15.46:0/3850129421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd624003b40 0x7fd624003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd62ffff640 1 --2- 172.21.15.46:0/3850129421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6240047d0 0x7fd624004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd6377fe640 1 --2- 172.21.15.46:0/3850129421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd63806e470 0x7fd624005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd636ffd640 1 -- 172.21.15.46:0/3850129421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd63806e470 msgr2=0x7fd624005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd636ffd640 1 --2- 172.21.15.46:0/3850129421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd63806e470 0x7fd624005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd636ffd640 1 -- 172.21.15.46:0/3850129421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6240047d0 msgr2=0x7fd624004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd636ffd640 1 --2- 172.21.15.46:0/3850129421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6240047d0 0x7fd624004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.583+0000 7fd636ffd640 1 -- 172.21.15.46:0/3850129421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd624003530 con 0x7fd624003b40 2023-10-30T14:12:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd636ffd640 1 --2- 172.21.15.46:0/3850129421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd624003b40 0x7fd624003f40 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fd6280018d0 tx=0x7fd62801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b89738c4bef20c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6280052d0 con 0x7fd624003b40 2023-10-30T14:12:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd628005450 con 0x7fd624003b40 2023-10-30T14:12:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd6367fc640 1 -- 172.21.15.46:0/3850129421 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628003840 con 0x7fd624003b40 2023-10-30T14:12:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd63d30f640 1 -- 172.21.15.46:0/3850129421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd624003b40 msgr2=0x7fd624003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd63d30f640 1 --2- 172.21.15.46:0/3850129421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd624003b40 0x7fd624003f40 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fd6280018d0 tx=0x7fd62801c4d0 comp rx=0 tx=0).stop 2023-10-30T14:12:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd6377fe640 1 -- 172.21.15.46:0/3850129421 reap_dead start 2023-10-30T14:12:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd63d30f640 1 -- 172.21.15.46:0/3850129421 shutdown_connections 2023-10-30T14:12:40.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd63d30f640 1 -- 172.21.15.46:0/3850129421 >> 172.21.15.46:0/3850129421 conn(0x7fd638071ea0 msgr2=0x7fd638065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:40.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd63d30f640 1 -- 172.21.15.46:0/3850129421 shutdown_connections 2023-10-30T14:12:40.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd63d30f640 1 -- 172.21.15.46:0/3850129421 wait complete. 2023-10-30T14:12:40.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd63d30f640 1 Processor -- start 2023-10-30T14:12:40.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd63d30f640 1 -- start start 2023-10-30T14:12:40.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.587+0000 7fd63d30f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd624003b40 0x7fd638156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:40.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd63d30f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd638155d90 0x7fd638149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:40.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd63d30f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6380cc930 0x7fd6380ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:40.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd63d30f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63815a540 con 0x7fd6380cc930 2023-10-30T14:12:40.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd63d30f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63814aea0 con 0x7fd624003b40 2023-10-30T14:12:40.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd63d30f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63814dda0 con 0x7fd638155d90 2023-10-30T14:12:40.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd6377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6380cc930 0x7fd6380ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:40.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd6377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6380cc930 0x7fd6380ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34858/0 (socket says 172.21.15.46:34858) 2023-10-30T14:12:40.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd6377fe640 1 -- 172.21.15.46:0/1677633178 learned_addr learned my addr 172.21.15.46:0/1677633178 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:40.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd636ffd640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd624003b40 0x7fd638156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:40.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd62ffff640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd638155d90 0x7fd638149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:40.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd6377fe640 1 -- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd638155d90 msgr2=0x7fd638149280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:40.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd6377fe640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd638155d90 0x7fd638149280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:40.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd6377fe640 1 -- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd624003b40 msgr2=0x7fd638156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:40.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd6377fe640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd624003b40 0x7fd638156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:40.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd6377fe640 1 -- 172.21.15.46:0/1677633178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6381f3d90 con 0x7fd6380cc930 2023-10-30T14:12:40.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd62ffff640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd638155d90 0x7fd638149280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:12:40.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd6377fe640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6380cc930 0x7fd6380ccdb0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fd6380700c0 tx=0x7fd63000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:40.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd62f7fe640 1 -- 172.21.15.46:0/1677633178 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630007aa0 con 0x7fd6380cc930 2023-10-30T14:12:40.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd62f7fe640 1 -- 172.21.15.46:0/1677633178 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63000edc0 con 0x7fd6380cc930 2023-10-30T14:12:40.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd62f7fe640 1 -- 172.21.15.46:0/1677633178 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630013660 con 0x7fd6380cc930 2023-10-30T14:12:40.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd63d30f640 1 -- 172.21.15.46:0/1677633178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd638156340 con 0x7fd6380cc930 2023-10-30T14:12:40.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd63d30f640 1 -- 172.21.15.46:0/1677633178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6381f4050 con 0x7fd6380cc930 2023-10-30T14:12:40.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd63d30f640 1 -- 172.21.15.46:0/1677633178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63804e170 con 0x7fd6380cc930 2023-10-30T14:12:40.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.591+0000 7fd62f7fe640 1 -- 172.21.15.46:0/1677633178 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6300137c0 con 0x7fd6380cc930 2023-10-30T14:12:40.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.595+0000 7fd62f7fe640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6180664a0 0x7fd618068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:40.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.595+0000 7fd62f7fe640 1 -- 172.21.15.46:0/1677633178 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7fd630084c80 con 0x7fd6380cc930 2023-10-30T14:12:40.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.595+0000 7fd636ffd640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6180664a0 0x7fd618068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:40.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.595+0000 7fd636ffd640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6180664a0 0x7fd618068960 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd6280ad000 tx=0x7fd6280a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:40.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.599+0000 7fd62f7fe640 1 -- 172.21.15.46:0/1677633178 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd630051740 con 0x7fd6380cc930 2023-10-30T14:12:40.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 -- 172.21.15.46:0/1677633178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd63814e6c0 con 0x7fd6380cc930 2023-10-30T14:12:40.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd62f7fe640 1 -- 172.21.15.46:0/1677633178 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+5727 (secure 0 0 0) 0x7fd630054d60 con 0x7fd6380cc930 2023-10-30T14:12:40.832 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:40.833 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":212,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:35.647379+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:40.833 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 212 2023-10-30T14:12:40.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 -- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6180664a0 msgr2=0x7fd618068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:40.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6180664a0 0x7fd618068960 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd6280ad000 tx=0x7fd6280a7000 comp rx=0 tx=0).stop 2023-10-30T14:12:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 -- 172.21.15.46:0/1677633178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6380cc930 msgr2=0x7fd6380ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6380cc930 0x7fd6380ccdb0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fd6380700c0 tx=0x7fd63000e530 comp rx=0 tx=0).stop 2023-10-30T14:12:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 -- 172.21.15.46:0/1677633178 shutdown_connections 2023-10-30T14:12:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6180664a0 0x7fd618068960 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6380cc930 0x7fd6380ccdb0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:40.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd638155d90 0x7fd638149280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:40.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 --2- 172.21.15.46:0/1677633178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd624003b40 0x7fd638156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:40.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 -- 172.21.15.46:0/1677633178 >> 172.21.15.46:0/1677633178 conn(0x7fd638071ea0 msgr2=0x7fd6381507b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:40.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 -- 172.21.15.46:0/1677633178 shutdown_connections 2023-10-30T14:12:40.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:40.831+0000 7fd63d30f640 1 -- 172.21.15.46:0/1677633178 wait complete. 2023-10-30T14:12:40.854 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 212, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 212, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8826": { "addr": "172.21.15.46:6836/66265765", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 66265765, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 66265765, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 8826, "incarnation": 203, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 17 }, "gid_8982": { "addr": "172.21.15.46:6833/2455118950", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2455118950, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2455118950, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8982, "incarnation": 209, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:rejoin", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:12:35.647379+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8826, "mds_1": 8982 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:12:42.855 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:43.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.007+0000 7f06d906b640 1 Processor -- start 2023-10-30T14:12:43.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.007+0000 7f06d906b640 1 -- start start 2023-10-30T14:12:43.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06d906b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f06d4157010 0x7f06d40715c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06d906b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f06d40c8460 0x7f06d4070650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06d906b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06d4159810 0x7f06d4071bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06d906b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d40c21b0 con 0x7f06d40c8460 2023-10-30T14:12:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06d906b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d40c1530 con 0x7f06d4157010 2023-10-30T14:12:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06d906b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d40c0f70 con 0x7f06d4159810 2023-10-30T14:12:43.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06cbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f06d40c8460 0x7f06d4070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:43.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06cbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f06d40c8460 0x7f06d4070650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34870/0 (socket says 172.21.15.46:34870) 2023-10-30T14:12:43.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06cbfff640 1 -- 172.21.15.46:0/1882182999 learned_addr learned my addr 172.21.15.46:0/1882182999 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:43.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06d359e640 1 --2- 172.21.15.46:0/1882182999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06d4159810 0x7f06d4071bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:43.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06d2d9d640 1 --2- 172.21.15.46:0/1882182999 >> v2:172.21.15.154:3300/0 conn(0x7f06d4157010 0x7f06d40715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:43.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06cbfff640 1 -- 172.21.15.46:0/1882182999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06d4159810 msgr2=0x7f06d4071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:43.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06cbfff640 1 --2- 172.21.15.46:0/1882182999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06d4159810 0x7f06d4071bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:43.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06cbfff640 1 -- 172.21.15.46:0/1882182999 >> v2:172.21.15.154:3300/0 conn(0x7f06d4157010 msgr2=0x7f06d40715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:43.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06cbfff640 1 --2- 172.21.15.46:0/1882182999 >> v2:172.21.15.154:3300/0 conn(0x7f06d4157010 0x7f06d40715c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:43.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06cbfff640 1 -- 172.21.15.46:0/1882182999 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d40c1250 con 0x7f06d40c8460 2023-10-30T14:12:43.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.011+0000 7f06d2d9d640 1 --2- 172.21.15.46:0/1882182999 >> v2:172.21.15.154:3300/0 conn(0x7f06d4157010 0x7f06d40715c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:12:43.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06cbfff640 1 --2- 172.21.15.46:0/1882182999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06d40c8460 0x7f06d4070650 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f06d406d550 tx=0x7f06bc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938359f65b03879c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:43.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d259c640 1 -- 172.21.15.46:0/1882182999 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc0a89b0 con 0x7f06d40c8460 2023-10-30T14:12:43.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d259c640 1 -- 172.21.15.46:0/1882182999 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06bc0a8b10 con 0x7f06d40c8460 2023-10-30T14:12:43.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d259c640 1 -- 172.21.15.46:0/1882182999 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc0a8ea0 con 0x7f06d40c8460 2023-10-30T14:12:43.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d906b640 1 -- 172.21.15.46:0/1882182999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06d40c8460 msgr2=0x7f06d4070650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:43.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d906b640 1 --2- 172.21.15.46:0/1882182999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06d40c8460 0x7f06d4070650 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f06d406d550 tx=0x7f06bc01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:12:43.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d906b640 1 -- 172.21.15.46:0/1882182999 shutdown_connections 2023-10-30T14:12:43.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d906b640 1 --2- 172.21.15.46:0/1882182999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06d4159810 0x7f06d4071bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:43.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d906b640 1 --2- 172.21.15.46:0/1882182999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06d40c8460 0x7f06d4070650 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:43.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d906b640 1 --2- 172.21.15.46:0/1882182999 >> v2:172.21.15.154:3300/0 conn(0x7f06d4157010 0x7f06d40715c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:43.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d906b640 1 -- 172.21.15.46:0/1882182999 >> 172.21.15.46:0/1882182999 conn(0x7f06d40cbeb0 msgr2=0x7f06d40cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:43.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d906b640 1 -- 172.21.15.46:0/1882182999 shutdown_connections 2023-10-30T14:12:43.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d906b640 1 -- 172.21.15.46:0/1882182999 wait complete. 2023-10-30T14:12:43.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d906b640 1 Processor -- start 2023-10-30T14:12:43.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.015+0000 7f06d906b640 1 -- start start 2023-10-30T14:12:43.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d906b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06d40c8460 0x7f06d4155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:43.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d906b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06d4157010 0x7f06d414fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:43.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d906b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d4159810 0x7f06d40ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:43.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d906b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d40cbd40 con 0x7f06d40c8460 2023-10-30T14:12:43.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d906b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d41f2f80 con 0x7f06d4159810 2023-10-30T14:12:43.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d906b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d40cba60 con 0x7f06d4157010 2023-10-30T14:12:43.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06d40c8460 0x7f06d4155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:43.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06d40c8460 0x7f06d4155610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34880/0 (socket says 172.21.15.46:34880) 2023-10-30T14:12:43.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d2d9d640 1 -- 172.21.15.46:0/3985250701 learned_addr learned my addr 172.21.15.46:0/3985250701 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:43.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d359e640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d4159810 0x7f06d40ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:43.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06cbfff640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06d4157010 0x7f06d414fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:43.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d2d9d640 1 -- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06d4157010 msgr2=0x7f06d414fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:43.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d2d9d640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06d4157010 0x7f06d414fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:43.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d2d9d640 1 -- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d4159810 msgr2=0x7f06d40ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:43.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d2d9d640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d4159810 0x7f06d40ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:43.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d2d9d640 1 -- 172.21.15.46:0/3985250701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06bc0094d0 con 0x7f06d40c8460 2023-10-30T14:12:43.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d2d9d640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06d40c8460 0x7f06d4155610 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f06c400c7c0 tx=0x7f06c400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:43.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06cb7fe640 1 -- 172.21.15.46:0/3985250701 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4007a30 con 0x7f06d40c8460 2023-10-30T14:12:43.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06cb7fe640 1 -- 172.21.15.46:0/3985250701 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c4007b90 con 0x7f06d40c8460 2023-10-30T14:12:43.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06cb7fe640 1 -- 172.21.15.46:0/3985250701 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c400ce70 con 0x7f06d40c8460 2023-10-30T14:12:43.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d906b640 1 -- 172.21.15.46:0/3985250701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d41f21c0 con 0x7f06d40c8460 2023-10-30T14:12:43.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d906b640 1 -- 172.21.15.46:0/3985250701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d41f2780 con 0x7f06d40c8460 2023-10-30T14:12:43.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06d906b640 1 -- 172.21.15.46:0/3985250701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d404e170 con 0x7f06d40c8460 2023-10-30T14:12:43.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.019+0000 7f06cb7fe640 1 -- 172.21.15.46:0/3985250701 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06c40113f0 con 0x7f06d40c8460 2023-10-30T14:12:43.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.023+0000 7f06cb7fe640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06b40664a0 0x7f06b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:43.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.023+0000 7f06cb7fe640 1 -- 172.21.15.46:0/3985250701 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f06c4084a40 con 0x7f06d40c8460 2023-10-30T14:12:43.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.023+0000 7f06cbfff640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06b40664a0 0x7f06b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:43.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.023+0000 7f06cbfff640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06b40664a0 0x7f06b4068960 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f06bc0b2000 tx=0x7f06bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:43.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.027+0000 7f06cb7fe640 1 -- 172.21.15.46:0/3985250701 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06c4051440 con 0x7f06d40c8460 2023-10-30T14:12:43.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.215+0000 7f06d906b640 1 -- 172.21.15.46:0/3985250701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06d4153230 con 0x7f06d40c8460 2023-10-30T14:12:43.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.215+0000 7f06cb7fe640 1 -- 172.21.15.46:0/3985250701 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f06c4054a60 con 0x7f06d40c8460 2023-10-30T14:12:43.218 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:43.218 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:40.757705+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:43.219 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 213 2023-10-30T14:12:43.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.215+0000 7f06d906b640 1 -- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06b40664a0 msgr2=0x7f06b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:43.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.215+0000 7f06d906b640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06b40664a0 0x7f06b4068960 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f06bc0b2000 tx=0x7f06bc008040 comp rx=0 tx=0).stop 2023-10-30T14:12:43.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.215+0000 7f06d906b640 1 -- 172.21.15.46:0/3985250701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06d40c8460 msgr2=0x7f06d4155610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:43.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.215+0000 7f06d906b640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06d40c8460 0x7f06d4155610 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f06c400c7c0 tx=0x7f06c400cc80 comp rx=0 tx=0).stop 2023-10-30T14:12:43.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.219+0000 7f06d906b640 1 -- 172.21.15.46:0/3985250701 shutdown_connections 2023-10-30T14:12:43.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.219+0000 7f06d906b640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f06b40664a0 0x7f06b4068960 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:43.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.219+0000 7f06d906b640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d4159810 0x7f06d40ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:43.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.219+0000 7f06d906b640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06d4157010 0x7f06d414fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:43.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.219+0000 7f06d906b640 1 --2- 172.21.15.46:0/3985250701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06d40c8460 0x7f06d4155610 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:43.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.219+0000 7f06d906b640 1 -- 172.21.15.46:0/3985250701 >> 172.21.15.46:0/3985250701 conn(0x7f06d40cbeb0 msgr2=0x7f06d40cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:43.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.219+0000 7f06d906b640 1 -- 172.21.15.46:0/3985250701 shutdown_connections 2023-10-30T14:12:43.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:43.219+0000 7f06d906b640 1 -- 172.21.15.46:0/3985250701 wait complete. 2023-10-30T14:12:43.239 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=8655 2023-10-30T14:12:43.239 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.e 2023-10-30T14:12:48.240 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T14:12:48.241 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T14:12:48.241 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T14:12:48.245 INFO:tasks.ceph.mds.e:Started 2023-10-30T14:12:48.245 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:48.348 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T14:12:48.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f3c91640 1 Processor -- start 2023-10-30T14:12:48.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f3c91640 1 -- start start 2023-10-30T14:12:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f3c91640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb9ec06f7a0 0x7fb9ec06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f3c91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c6bd0 0x7fb9ec069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f3c91640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ec06a3d0 con 0x7fb9ec071110 2023-10-30T14:12:48.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f3c91640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9ec0700c0 con 0x7fb9ec06f7a0 2023-10-30T14:12:48.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f3c91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec066cf0 con 0x7fb9ec0c6bd0 2023-10-30T14:12:48.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f222e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb9ec071110 0x7fb9ec158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32778/0 (socket says 172.21.15.46:32778) 2023-10-30T14:12:48.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f222e640 1 -- 172.21.15.46:0/1913402435 learned_addr learned my addr 172.21.15.46:0/1913402435 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:12:48.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f122c640 1 --2- 172.21.15.46:0/1913402435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c6bd0 0x7fb9ec069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:48.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f1a2d640 1 --2- 172.21.15.46:0/1913402435 >> v2:172.21.15.154:3300/0 conn(0x7fb9ec06f7a0 0x7fb9ec06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:48.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f0a2b640 1 -- 172.21.15.46:0/1913402435 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846920562 0 0) 0x7fb9ec06a3d0 con 0x7fb9ec071110 2023-10-30T14:12:48.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f0a2b640 1 -- 172.21.15.46:0/1913402435 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d8003530 con 0x7fb9ec071110 2023-10-30T14:12:48.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f122c640 1 -- 172.21.15.46:0/1913402435 >> v2:172.21.15.154:3300/0 conn(0x7fb9ec06f7a0 msgr2=0x7fb9ec06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:48.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f122c640 1 --2- 172.21.15.46:0/1913402435 >> v2:172.21.15.154:3300/0 conn(0x7fb9ec06f7a0 0x7fb9ec06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:48.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f122c640 1 -- 172.21.15.46:0/1913402435 >> v1:172.21.15.46:6789/0 conn(0x7fb9ec071110 legacy=0x7fb9ec158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:48.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f0a2b640 1 -- 172.21.15.46:0/1913402435 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1959876996 0 0) 0x7fb9d8003530 con 0x7fb9ec071110 2023-10-30T14:12:48.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f122c640 1 -- 172.21.15.46:0/1913402435 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec0c70d0 con 0x7fb9ec0c6bd0 2023-10-30T14:12:48.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.399+0000 7fb9f1a2d640 1 --2- 172.21.15.46:0/1913402435 >> v2:172.21.15.154:3300/0 conn(0x7fb9ec06f7a0 0x7fb9ec06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:12:48.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.403+0000 7fb9f122c640 1 --2- 172.21.15.46:0/1913402435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c6bd0 0x7fb9ec069e90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0097c0 tx=0x7fb9dc01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=536a857f9f1c3184 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:48.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.403+0000 7fb9f0a2b640 1 -- 172.21.15.46:0/1913402435 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc0a8970 con 0x7fb9ec0c6bd0 2023-10-30T14:12:48.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.403+0000 7fb9f0a2b640 1 -- 172.21.15.46:0/1913402435 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc0a8ad0 con 0x7fb9ec0c6bd0 2023-10-30T14:12:48.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.403+0000 7fb9f0a2b640 1 -- 172.21.15.46:0/1913402435 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc0a8e40 con 0x7fb9ec0c6bd0 2023-10-30T14:12:48.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.403+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1913402435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c6bd0 msgr2=0x7fb9ec069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:48.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.403+0000 7fb9f3c91640 1 --2- 172.21.15.46:0/1913402435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c6bd0 0x7fb9ec069e90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0097c0 tx=0x7fb9dc01ec70 comp rx=0 tx=0).stop 2023-10-30T14:12:48.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.403+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1913402435 shutdown_connections 2023-10-30T14:12:48.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.403+0000 7fb9f3c91640 1 --2- 172.21.15.46:0/1913402435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c6bd0 0x7fb9ec069e90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:48.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.403+0000 7fb9f3c91640 1 --2- 172.21.15.46:0/1913402435 >> v2:172.21.15.154:3300/0 conn(0x7fb9ec06f7a0 0x7fb9ec06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:48.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.403+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1913402435 >> 172.21.15.46:0/1913402435 conn(0x7fb9ec0c7910 msgr2=0x7fb9ec0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:48.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.403+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1913402435 shutdown_connections 2023-10-30T14:12:48.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.403+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1913402435 wait complete. 2023-10-30T14:12:48.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f3c91640 1 Processor -- start 2023-10-30T14:12:48.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f3c91640 1 -- start start 2023-10-30T14:12:48.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f3c91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec06f7a0 0x7fb9ec14fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:48.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f3c91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec071110 0x7fb9ec1ef270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:48.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f3c91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c6bd0 0x7fb9ec15a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:48.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f3c91640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec0afeb0 con 0x7fb9ec071110 2023-10-30T14:12:48.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f3c91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec1ef7b0 con 0x7fb9ec06f7a0 2023-10-30T14:12:48.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f3c91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec15a6d0 con 0x7fb9ec0c6bd0 2023-10-30T14:12:48.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f122c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec071110 0x7fb9ec1ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:48.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f122c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec071110 0x7fb9ec1ef270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34666/0 (socket says 172.21.15.46:34666) 2023-10-30T14:12:48.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f122c640 1 -- 172.21.15.46:0/1880606149 learned_addr learned my addr 172.21.15.46:0/1880606149 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:48.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f222e640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c6bd0 0x7fb9ec15a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:48.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f1a2d640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec06f7a0 0x7fb9ec14fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:48.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f122c640 1 -- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c6bd0 msgr2=0x7fb9ec15a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:48.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f122c640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c6bd0 0x7fb9ec15a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:48.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f122c640 1 -- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec06f7a0 msgr2=0x7fb9ec14fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:48.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f122c640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec06f7a0 0x7fb9ec14fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:48.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f122c640 1 -- 172.21.15.46:0/1880606149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e8009420 con 0x7fb9ec071110 2023-10-30T14:12:48.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f122c640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec071110 0x7fb9ec1ef270 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc000c10 tx=0x7fb9dc01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:48.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9e27fc640 1 -- 172.21.15.46:0/1880606149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc0be070 con 0x7fb9ec071110 2023-10-30T14:12:48.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9e27fc640 1 -- 172.21.15.46:0/1880606149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc004250 con 0x7fb9ec071110 2023-10-30T14:12:48.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1880606149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc009420 con 0x7fb9ec071110 2023-10-30T14:12:48.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9e27fc640 1 -- 172.21.15.46:0/1880606149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc003620 con 0x7fb9ec071110 2023-10-30T14:12:48.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.407+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1880606149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ec1f1e30 con 0x7fb9ec071110 2023-10-30T14:12:48.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.411+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1880606149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ec04e170 con 0x7fb9ec071110 2023-10-30T14:12:48.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.411+0000 7fb9e27fc640 1 -- 172.21.15.46:0/1880606149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9dc0b7040 con 0x7fb9ec071110 2023-10-30T14:12:48.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.411+0000 7fb9e27fc640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9d00664a0 0x7fb9d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:48.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.411+0000 7fb9e27fc640 1 -- 172.21.15.46:0/1880606149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7fb9dc1214e0 con 0x7fb9ec071110 2023-10-30T14:12:48.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.411+0000 7fb9f1a2d640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9d00664a0 0x7fb9d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:48.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.411+0000 7fb9f1a2d640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9d00664a0 0x7fb9d0068960 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009be0 tx=0x7fb9e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:48.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.415+0000 7fb9e27fc640 1 -- 172.21.15.46:0/1880606149 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9dc0ee000 con 0x7fb9ec071110 2023-10-30T14:12:48.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.607+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1880606149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9ec0c4200 con 0x7fb9ec071110 2023-10-30T14:12:48.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.607+0000 7fb9e27fc640 1 -- 172.21.15.46:0/1880606149 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+5727 (secure 0 0 0) 0x7fb9dc0f1620 con 0x7fb9ec071110 2023-10-30T14:12:48.611 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:48.611 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:40.757705+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:48.611 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 213 2023-10-30T14:12:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.611+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9d00664a0 msgr2=0x7fb9d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.611+0000 7fb9f3c91640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9d00664a0 0x7fb9d0068960 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009be0 tx=0x7fb9e8008040 comp rx=0 tx=0).stop 2023-10-30T14:12:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.611+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1880606149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec071110 msgr2=0x7fb9ec1ef270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.611+0000 7fb9f3c91640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec071110 0x7fb9ec1ef270 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc000c10 tx=0x7fb9dc01ba10 comp rx=0 tx=0).stop 2023-10-30T14:12:48.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.611+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1880606149 shutdown_connections 2023-10-30T14:12:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.611+0000 7fb9f3c91640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9d00664a0 0x7fb9d0068960 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.611+0000 7fb9f3c91640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9ec0c6bd0 0x7fb9ec15a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.611+0000 7fb9f3c91640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9ec071110 0x7fb9ec1ef270 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.611+0000 7fb9f3c91640 1 --2- 172.21.15.46:0/1880606149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9ec06f7a0 0x7fb9ec14fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:48.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.611+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1880606149 >> 172.21.15.46:0/1880606149 conn(0x7fb9ec0c7910 msgr2=0x7fb9ec0ca670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:48.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.611+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1880606149 shutdown_connections 2023-10-30T14:12:48.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:48.611+0000 7fb9f3c91640 1 -- 172.21.15.46:0/1880606149 wait complete. 2023-10-30T14:12:48.632 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T14:12:49.410 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:12:49.411 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 24 -eq 48 ] 2023-10-30T14:12:49.411 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.24 2023-10-30T14:12:49.411 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.24 2023-10-30T14:12:49.412 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.24 2023-10-30T14:12:49.412 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.24 2023-10-30T14:12:49.412 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:12:50.633 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:12:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.787+0000 7f11fd33e640 1 Processor -- start 2023-10-30T14:12:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.787+0000 7f11fd33e640 1 -- start start 2023-10-30T14:12:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.787+0000 7f11fd33e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11f80c55e0 0x7f11f806a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.787+0000 7f11fd33e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f80b4a70 0x7f11f8159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.787+0000 7f11fd33e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f80cc200 con 0x7f11f80c68b0 2023-10-30T14:12:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.787+0000 7f11fd33e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f80c8fa0 con 0x7f11f80c55e0 2023-10-30T14:12:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.787+0000 7f11fd33e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f8159bb0 con 0x7f11f80b4a70 2023-10-30T14:12:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.787+0000 7f11f6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11f80c55e0 0x7f11f806a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.791+0000 7f11f6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11f80c55e0 0x7f11f806a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34670/0 (socket says 172.21.15.46:34670) 2023-10-30T14:12:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.791+0000 7f11f6ffd640 1 -- 172.21.15.46:0/2202853855 learned_addr learned my addr 172.21.15.46:0/2202853855 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:50.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.791+0000 7f11f77fe640 1 --2- 172.21.15.46:0/2202853855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f80b4a70 0x7f11f8159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.791+0000 7f11f6ffd640 1 -- 172.21.15.46:0/2202853855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f80b4a70 msgr2=0x7f11f8159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:12:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.791+0000 7f11f6ffd640 1 --2- 172.21.15.46:0/2202853855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f80b4a70 0x7f11f8159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.791+0000 7f11f6ffd640 1 -- 172.21.15.46:0/2202853855 >> v1:172.21.15.154:6789/0 conn(0x7f11f80c68b0 legacy=0x7f11f80caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.791+0000 7f11f6ffd640 1 -- 172.21.15.46:0/2202853855 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f80c9df0 con 0x7f11f80c55e0 2023-10-30T14:12:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.791+0000 7f11f6ffd640 1 --2- 172.21.15.46:0/2202853855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f80c55e0 0x7f11f806a3a0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f11e8009870 tx=0x7f11e801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60825b261a547639 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:50.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.791+0000 7f11f5ffb640 1 -- 172.21.15.46:0/2202853855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e80ab600 con 0x7f11f80c55e0 2023-10-30T14:12:50.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11f5ffb640 1 -- 172.21.15.46:0/2202853855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e80a7050 con 0x7f11f80c55e0 2023-10-30T14:12:50.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11f5ffb640 1 -- 172.21.15.46:0/2202853855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e80ab980 con 0x7f11f80c55e0 2023-10-30T14:12:50.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11fd33e640 1 -- 172.21.15.46:0/2202853855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f80c55e0 msgr2=0x7f11f806a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:50.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11fd33e640 1 --2- 172.21.15.46:0/2202853855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f80c55e0 0x7f11f806a3a0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f11e8009870 tx=0x7f11e801c540 comp rx=0 tx=0).stop 2023-10-30T14:12:50.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11fd33e640 1 -- 172.21.15.46:0/2202853855 shutdown_connections 2023-10-30T14:12:50.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11fd33e640 1 --2- 172.21.15.46:0/2202853855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f80b4a70 0x7f11f8159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:50.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11fd33e640 1 --2- 172.21.15.46:0/2202853855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f80c55e0 0x7f11f806a3a0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:50.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11fd33e640 1 -- 172.21.15.46:0/2202853855 >> 172.21.15.46:0/2202853855 conn(0x7f11f806bed0 msgr2=0x7f11f806c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11fd33e640 1 -- 172.21.15.46:0/2202853855 shutdown_connections 2023-10-30T14:12:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11fd33e640 1 -- 172.21.15.46:0/2202853855 wait complete. 2023-10-30T14:12:50.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11fd33e640 1 Processor -- start 2023-10-30T14:12:50.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11fd33e640 1 -- start start 2023-10-30T14:12:50.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.795+0000 7f11fd33e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f80b4a70 0x7f11f81552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:50.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11fd33e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11f80c55e0 0x7f11f8152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:50.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11fd33e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f80c68b0 0x7f11f81f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11fd33e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f80c27b0 con 0x7f11f80c68b0 2023-10-30T14:12:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11fd33e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f80cce70 con 0x7f11f80c55e0 2023-10-30T14:12:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11fd33e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f80cd110 con 0x7f11f80b4a70 2023-10-30T14:12:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11f77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f80c68b0 0x7f11f81f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11f6ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f80b4a70 0x7f11f81552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:50.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11f77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f80c68b0 0x7f11f81f1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34676/0 (socket says 172.21.15.46:34676) 2023-10-30T14:12:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11f77fe640 1 -- 172.21.15.46:0/2475609452 learned_addr learned my addr 172.21.15.46:0/2475609452 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:12:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11f67fc640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11f80c55e0 0x7f11f8152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11f77fe640 1 -- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f80b4a70 msgr2=0x7f11f81552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:50.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11f77fe640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f80b4a70 0x7f11f81552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11f77fe640 1 -- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11f80c55e0 msgr2=0x7f11f8152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11f77fe640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11f80c55e0 0x7f11f8152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11f77fe640 1 -- 172.21.15.46:0/2475609452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e80094d0 con 0x7f11f80c68b0 2023-10-30T14:12:50.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11f77fe640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f80c68b0 0x7f11f81f1a60 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f11f000efc0 tx=0x7f11f000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11df7fe640 1 -- 172.21.15.46:0/2475609452 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0014480 con 0x7f11f80c68b0 2023-10-30T14:12:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11df7fe640 1 -- 172.21.15.46:0/2475609452 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f000f040 con 0x7f11f80c68b0 2023-10-30T14:12:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11fd33e640 1 -- 172.21.15.46:0/2475609452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f80cd3f0 con 0x7f11f80c68b0 2023-10-30T14:12:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11df7fe640 1 -- 172.21.15.46:0/2475609452 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0014820 con 0x7f11f80c68b0 2023-10-30T14:12:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11fd33e640 1 -- 172.21.15.46:0/2475609452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f80ceb20 con 0x7f11f80c68b0 2023-10-30T14:12:50.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11fd33e640 1 -- 172.21.15.46:0/2475609452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f804e170 con 0x7f11f80c68b0 2023-10-30T14:12:50.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.799+0000 7f11df7fe640 1 -- 172.21.15.46:0/2475609452 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f11f0014980 con 0x7f11f80c68b0 2023-10-30T14:12:50.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.803+0000 7f11df7fe640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11e40664a0 0x7f11e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:12:50.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.803+0000 7f11df7fe640 1 -- 172.21.15.46:0/2475609452 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f11f00853a0 con 0x7f11f80c68b0 2023-10-30T14:12:50.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.803+0000 7f11f6ffd640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11e40664a0 0x7f11e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:12:50.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.803+0000 7f11f6ffd640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11e40664a0 0x7f11e4068960 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f11f806b8e0 tx=0x7f11e80a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:12:50.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:50.807+0000 7f11df7fe640 1 -- 172.21.15.46:0/2475609452 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f11f0051e50 con 0x7f11f80c68b0 2023-10-30T14:12:51.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.003+0000 7f11fd33e640 1 -- 172.21.15.46:0/2475609452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11f80ceff0 con 0x7f11f80c68b0 2023-10-30T14:12:51.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.003+0000 7f11df7fe640 1 -- 172.21.15.46:0/2475609452 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 214 v214) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f11f0018020 con 0x7f11f80c68b0 2023-10-30T14:12:51.007 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:12:51.008 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":214,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9135,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:12:40.757705+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:12:51.008 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 214 2023-10-30T14:12:51.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.007+0000 7f11fd33e640 1 -- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11e40664a0 msgr2=0x7f11e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:51.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.007+0000 7f11fd33e640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11e40664a0 0x7f11e4068960 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f11f806b8e0 tx=0x7f11e80a6000 comp rx=0 tx=0).stop 2023-10-30T14:12:51.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.007+0000 7f11fd33e640 1 -- 172.21.15.46:0/2475609452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f80c68b0 msgr2=0x7f11f81f1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:12:51.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.007+0000 7f11fd33e640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f80c68b0 0x7f11f81f1a60 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f11f000efc0 tx=0x7f11f000c490 comp rx=0 tx=0).stop 2023-10-30T14:12:51.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.007+0000 7f11fd33e640 1 -- 172.21.15.46:0/2475609452 shutdown_connections 2023-10-30T14:12:51.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.007+0000 7f11fd33e640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f11e40664a0 0x7f11e4068960 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:51.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.007+0000 7f11fd33e640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11f80c68b0 0x7f11f81f1a60 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:51.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.007+0000 7f11fd33e640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11f80c55e0 0x7f11f8152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:51.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.007+0000 7f11fd33e640 1 --2- 172.21.15.46:0/2475609452 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11f80b4a70 0x7f11f81552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:12:51.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.007+0000 7f11fd33e640 1 -- 172.21.15.46:0/2475609452 >> 172.21.15.46:0/2475609452 conn(0x7f11f806bed0 msgr2=0x7f11f80c2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:12:51.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.007+0000 7f11fd33e640 1 -- 172.21.15.46:0/2475609452 shutdown_connections 2023-10-30T14:12:51.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:12:51.007+0000 7f11fd33e640 1 -- 172.21.15.46:0/2475609452 wait complete. 2023-10-30T14:12:51.028 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T14:12:51.029 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 110 secs before thrashing 2023-10-30T14:12:57.633 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:12:57.633 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 25 -eq 48 ] 2023-10-30T14:12:57.633 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.25 2023-10-30T14:12:57.634 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.25 2023-10-30T14:12:57.634 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.25 2023-10-30T14:12:57.634 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.25 2023-10-30T14:12:57.635 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:12:57.855 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:12:57.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:13:01.891 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:01.891 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.26 2023-10-30T14:13:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 26 -eq 48 ] 2023-10-30T14:13:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.26 2023-10-30T14:13:01.892 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.26 2023-10-30T14:13:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.26 2023-10-30T14:13:01.893 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:05.195 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:05.196 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 27 -eq 48 ] 2023-10-30T14:13:05.196 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.27 2023-10-30T14:13:05.196 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.27 2023-10-30T14:13:05.197 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.27 2023-10-30T14:13:05.197 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.27 2023-10-30T14:13:05.197 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:09.498 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:09.499 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 28 -eq 48 ] 2023-10-30T14:13:09.499 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.28 2023-10-30T14:13:09.499 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.28 2023-10-30T14:13:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.28 2023-10-30T14:13:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.28 2023-10-30T14:13:09.500 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:12.766 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:12.767 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.29 2023-10-30T14:13:12.767 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 29 -eq 48 ] 2023-10-30T14:13:12.768 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.29 2023-10-30T14:13:12.768 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.29 2023-10-30T14:13:12.768 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.29 2023-10-30T14:13:12.768 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:16.379 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:16.379 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 30 -eq 48 ] 2023-10-30T14:13:16.379 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.30 2023-10-30T14:13:16.380 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.30 2023-10-30T14:13:16.380 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.30 2023-10-30T14:13:16.380 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.30 2023-10-30T14:13:16.380 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:22.054 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:22.054 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 31 -eq 48 ] 2023-10-30T14:13:22.055 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.31 2023-10-30T14:13:22.055 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.31 2023-10-30T14:13:22.055 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.31 2023-10-30T14:13:22.056 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.31 2023-10-30T14:13:22.056 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:27.913 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:13:27.918 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:13:28.208 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:28.208 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 32 -eq 48 ] 2023-10-30T14:13:28.208 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.32 2023-10-30T14:13:28.209 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.32 2023-10-30T14:13:28.209 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.32 2023-10-30T14:13:28.210 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.32 2023-10-30T14:13:28.210 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:31.564 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:31.565 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.33 2023-10-30T14:13:31.565 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 33 -eq 48 ] 2023-10-30T14:13:31.565 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.33 2023-10-30T14:13:31.566 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.33 2023-10-30T14:13:31.566 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.33 2023-10-30T14:13:31.566 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:38.255 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:38.255 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.34 2023-10-30T14:13:38.256 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 34 -eq 48 ] 2023-10-30T14:13:38.256 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.34 2023-10-30T14:13:38.256 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.34 2023-10-30T14:13:38.257 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.34 2023-10-30T14:13:38.257 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:44.759 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:44.760 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.35 2023-10-30T14:13:44.760 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 35 -eq 48 ] 2023-10-30T14:13:44.760 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.35 2023-10-30T14:13:44.761 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.35 2023-10-30T14:13:44.761 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.35 2023-10-30T14:13:44.761 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:48.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:48.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 36 -eq 48 ] 2023-10-30T14:13:48.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.36 2023-10-30T14:13:48.723 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.36 2023-10-30T14:13:48.724 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.36 2023-10-30T14:13:48.724 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.36 2023-10-30T14:13:48.724 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:54.327 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:54.327 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 37 -eq 48 ] 2023-10-30T14:13:54.328 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.37 2023-10-30T14:13:54.328 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.37 2023-10-30T14:13:54.328 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.37 2023-10-30T14:13:54.329 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.37 2023-10-30T14:13:54.329 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:57.927 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:13:57.927 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 38 -eq 48 ] 2023-10-30T14:13:57.927 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.38 2023-10-30T14:13:57.928 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.38 2023-10-30T14:13:57.928 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.38 2023-10-30T14:13:57.929 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.38 2023-10-30T14:13:57.929 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:13:57.964 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:13:57.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:14:03.502 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:14:03.503 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.39 2023-10-30T14:14:03.504 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 39 -eq 48 ] 2023-10-30T14:14:03.504 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.39 2023-10-30T14:14:03.504 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.39 2023-10-30T14:14:03.504 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.39 2023-10-30T14:14:03.504 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:14:10.714 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:14:10.714 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 40 -eq 48 ] 2023-10-30T14:14:10.715 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.40 2023-10-30T14:14:10.715 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.40 2023-10-30T14:14:10.715 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.40 2023-10-30T14:14:10.716 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.40 2023-10-30T14:14:10.716 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:14:15.811 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:14:15.812 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.41 2023-10-30T14:14:15.813 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 41 -eq 48 ] 2023-10-30T14:14:15.813 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.41 2023-10-30T14:14:15.813 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.41 2023-10-30T14:14:15.813 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.41 2023-10-30T14:14:15.814 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:14:21.586 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:14:21.587 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.42 2023-10-30T14:14:21.587 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 42 -eq 48 ] 2023-10-30T14:14:21.588 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.42 2023-10-30T14:14:21.588 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.42 2023-10-30T14:14:21.588 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.42 2023-10-30T14:14:21.588 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:14:27.686 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:14:27.687 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.43 2023-10-30T14:14:27.687 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 43 -eq 48 ] 2023-10-30T14:14:27.687 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.43 2023-10-30T14:14:27.688 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.43 2023-10-30T14:14:27.688 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.43 2023-10-30T14:14:28.027 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:14:28.032 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:14:28.408 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:14:32.809 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:14:32.810 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 44 -eq 48 ] 2023-10-30T14:14:32.810 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.44 2023-10-30T14:14:32.810 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.44 2023-10-30T14:14:32.811 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.44 2023-10-30T14:14:32.811 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.44 2023-10-30T14:14:32.811 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:14:39.131 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:14:39.131 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 45 -eq 48 ] 2023-10-30T14:14:39.132 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.45 2023-10-30T14:14:39.132 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.45 2023-10-30T14:14:39.132 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.45 2023-10-30T14:14:39.133 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.45 2023-10-30T14:14:39.133 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:14:41.029 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:41.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.182+0000 7fc033ca9640 1 Processor -- start 2023-10-30T14:14:41.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.182+0000 7fc033ca9640 1 -- start start 2023-10-30T14:14:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc033ca9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc02c0c05d0 0x7fc02c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc033ca9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc02c0c6f60 0x7fc02c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc033ca9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c0c0a50 con 0x7fc02c0c6b80 2023-10-30T14:14:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc033ca9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc02c06f6e0 con 0x7fc02c0c6f60 2023-10-30T14:14:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc033ca9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c06f9c0 con 0x7fc02c0c05d0 2023-10-30T14:14:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc031a45640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc02c0c6b80 0x7fc02c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37638/0 (socket says 172.21.15.46:37638) 2023-10-30T14:14:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc031a45640 1 -- 172.21.15.46:0/427850327 learned_addr learned my addr 172.21.15.46:0/427850327 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:14:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc032246640 1 --2- 172.21.15.46:0/427850327 >> v2:172.21.15.46:3300/0 conn(0x7fc02c0c6f60 0x7fc02c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc031244640 1 --2- 172.21.15.46:0/427850327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc02c0c05d0 0x7fc02c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc032246640 1 -- 172.21.15.46:0/427850327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc02c0c05d0 msgr2=0x7fc02c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc032246640 1 --2- 172.21.15.46:0/427850327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc02c0c05d0 0x7fc02c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc032246640 1 -- 172.21.15.46:0/427850327 >> v1:172.21.15.46:6789/0 conn(0x7fc02c0c6b80 legacy=0x7fc02c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc032246640 1 -- 172.21.15.46:0/427850327 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c06fca0 con 0x7fc02c0c6f60 2023-10-30T14:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.186+0000 7fc030a43640 1 -- 172.21.15.46:0/427850327 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834273901 0 0) 0x7fc02c0c0a50 con 0x7fc02c0c6b80 2023-10-30T14:14:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc032246640 1 --2- 172.21.15.46:0/427850327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc02c0c6f60 0x7fc02c13f2d0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fc024009d90 tx=0x7fc02401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47dc66ea8e610f98 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc030a43640 1 -- 172.21.15.46:0/427850327 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0240a5920 con 0x7fc02c0c6f60 2023-10-30T14:14:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc030a43640 1 -- 172.21.15.46:0/427850327 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0240a5a80 con 0x7fc02c0c6f60 2023-10-30T14:14:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc030a43640 1 -- 172.21.15.46:0/427850327 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0240a5e10 con 0x7fc02c0c6f60 2023-10-30T14:14:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 -- 172.21.15.46:0/427850327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc02c0c6f60 msgr2=0x7fc02c13f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 --2- 172.21.15.46:0/427850327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc02c0c6f60 0x7fc02c13f2d0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fc024009d90 tx=0x7fc02401c950 comp rx=0 tx=0).stop 2023-10-30T14:14:41.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 -- 172.21.15.46:0/427850327 shutdown_connections 2023-10-30T14:14:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 --2- 172.21.15.46:0/427850327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc02c0c6f60 0x7fc02c13f2d0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 --2- 172.21.15.46:0/427850327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc02c0c05d0 0x7fc02c13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 -- 172.21.15.46:0/427850327 >> 172.21.15.46:0/427850327 conn(0x7fc02c0c7910 msgr2=0x7fc02c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 -- 172.21.15.46:0/427850327 shutdown_connections 2023-10-30T14:14:41.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 -- 172.21.15.46:0/427850327 wait complete. 2023-10-30T14:14:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 Processor -- start 2023-10-30T14:14:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 -- start start 2023-10-30T14:14:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc02c0c05d0 0x7fc02c0ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc02c0c6b80 0x7fc02c151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02c0c6f60 0x7fc02c150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c0b0570 con 0x7fc02c0c05d0 2023-10-30T14:14:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c0c98a0 con 0x7fc02c0c6f60 2023-10-30T14:14:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc033ca9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c13a460 con 0x7fc02c0c6b80 2023-10-30T14:14:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.190+0000 7fc031a45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc02c0c05d0 0x7fc02c0ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc031a45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc02c0c05d0 0x7fc02c0ca640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34326/0 (socket says 172.21.15.46:34326) 2023-10-30T14:14:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc031a45640 1 -- 172.21.15.46:0/3496172332 learned_addr learned my addr 172.21.15.46:0/3496172332 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc031244640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc02c0c6b80 0x7fc02c151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc032246640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02c0c6f60 0x7fc02c150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc031a45640 1 -- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc02c0c6b80 msgr2=0x7fc02c151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc031a45640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc02c0c6b80 0x7fc02c151300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc031a45640 1 -- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02c0c6f60 msgr2=0x7fc02c150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc031a45640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02c0c6f60 0x7fc02c150150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc031a45640 1 -- 172.21.15.46:0/3496172332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc024009a70 con 0x7fc02c0c05d0 2023-10-30T14:14:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc032246640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02c0c6f60 0x7fc02c150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc031a45640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc02c0c05d0 0x7fc02c0ca640 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fc0280011f0 tx=0x7fc02800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc0227fc640 1 -- 172.21.15.46:0/3496172332 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0280074e0 con 0x7fc02c0c05d0 2023-10-30T14:14:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc0227fc640 1 -- 172.21.15.46:0/3496172332 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc028007640 con 0x7fc02c0c05d0 2023-10-30T14:14:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc0227fc640 1 -- 172.21.15.46:0/3496172332 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc028004a10 con 0x7fc02c0c05d0 2023-10-30T14:14:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc033ca9640 1 -- 172.21.15.46:0/3496172332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02c0c95c0 con 0x7fc02c0c05d0 2023-10-30T14:14:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc033ca9640 1 -- 172.21.15.46:0/3496172332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02c1f5230 con 0x7fc02c0c05d0 2023-10-30T14:14:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc033ca9640 1 -- 172.21.15.46:0/3496172332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02c0b03c0 con 0x7fc02c0c05d0 2023-10-30T14:14:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc0227fc640 1 -- 172.21.15.46:0/3496172332 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc028004b70 con 0x7fc02c0c05d0 2023-10-30T14:14:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc0227fc640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0100664d0 0x7fc010068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.194+0000 7fc0227fc640 1 -- 172.21.15.46:0/3496172332 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7fc0280838f0 con 0x7fc02c0c05d0 2023-10-30T14:14:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.198+0000 7fc031244640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0100664d0 0x7fc010068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.198+0000 7fc031244640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0100664d0 0x7fc010068990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc02c138aa0 tx=0x7fc01c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.202+0000 7fc0227fc640 1 -- 172.21.15.46:0/3496172332 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc0280504c0 con 0x7fc02c0c05d0 2023-10-30T14:14:41.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.390+0000 7fc033ca9640 1 -- 172.21.15.46:0/3496172332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc02c04e170 con 0x7fc02c0c05d0 2023-10-30T14:14:41.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc0227fc640 1 -- 172.21.15.46:0/3496172332 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+6517 (secure 0 0 0) 0x7fc028018090 con 0x7fc02c0c05d0 2023-10-30T14:14:41.396 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:14:41.396 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9135,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:14:17.893242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:41.397 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 217 2023-10-30T14:14:41.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc033ca9640 1 -- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0100664d0 msgr2=0x7fc010068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc033ca9640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0100664d0 0x7fc010068990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc02c138aa0 tx=0x7fc01c008040 comp rx=0 tx=0).stop 2023-10-30T14:14:41.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc033ca9640 1 -- 172.21.15.46:0/3496172332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc02c0c05d0 msgr2=0x7fc02c0ca640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc033ca9640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc02c0c05d0 0x7fc02c0ca640 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fc0280011f0 tx=0x7fc02800c9b0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc033ca9640 1 -- 172.21.15.46:0/3496172332 shutdown_connections 2023-10-30T14:14:41.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc033ca9640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc0100664d0 0x7fc010068990 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc033ca9640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02c0c6f60 0x7fc02c150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc033ca9640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc02c0c6b80 0x7fc02c151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc033ca9640 1 --2- 172.21.15.46:0/3496172332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc02c0c05d0 0x7fc02c0ca640 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc033ca9640 1 -- 172.21.15.46:0/3496172332 >> 172.21.15.46:0/3496172332 conn(0x7fc02c0c7910 msgr2=0x7fc02c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc033ca9640 1 -- 172.21.15.46:0/3496172332 shutdown_connections 2023-10-30T14:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.394+0000 7fc033ca9640 1 -- 172.21.15.46:0/3496172332 wait complete. 2023-10-30T14:14:41.416 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T14:14:41.417 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:14:41.417 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:14:41.440 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:14:41.441 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:41.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.622+0000 7f35fee32640 1 Processor -- start 2023-10-30T14:14:41.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.622+0000 7f35fee32640 1 -- start start 2023-10-30T14:14:41.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.622+0000 7f35fee32640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35f806f7a0 0x7f35f806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.622+0000 7f35fee32640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f35f8071110 0x7f35f8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.622+0000 7f35fee32640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f8066cf0 con 0x7f35f80c6bd0 2023-10-30T14:14:41.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.622+0000 7f35fee32640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f80700c0 con 0x7f35f8071110 2023-10-30T14:14:41.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.622+0000 7f35fee32640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80c70d0 con 0x7f35f806f7a0 2023-10-30T14:14:41.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.626+0000 7f35fd3cf640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f35f8071110 0x7f35f8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.626+0000 7f35f7fff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f35f80c6bd0 0x7f35f8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:45200/0 (socket says 172.21.15.46:45200) 2023-10-30T14:14:41.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.626+0000 7f35f7fff640 1 -- 172.21.15.46:0/3473988584 learned_addr learned my addr 172.21.15.46:0/3473988584 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:14:41.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.626+0000 7f35fcbce640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35f806f7a0 0x7f35f806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.626+0000 7f35fd3cf640 1 -- 172.21.15.46:0/3473988584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35f806f7a0 msgr2=0x7f35f806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:41.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.626+0000 7f35fd3cf640 1 --2- 172.21.15.46:0/3473988584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35f806f7a0 0x7f35f806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.626+0000 7f35fd3cf640 1 -- 172.21.15.46:0/3473988584 >> v1:172.21.15.154:6789/0 conn(0x7f35f80c6bd0 legacy=0x7f35f8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.626+0000 7f35fd3cf640 1 -- 172.21.15.46:0/3473988584 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f80c7390 con 0x7f35f8071110 2023-10-30T14:14:41.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.626+0000 7f35fd3cf640 1 --2- 172.21.15.46:0/3473988584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f8071110 0x7f35f8158110 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009d90 tx=0x7f35f001f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc6f0874496edda7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35f77fe640 1 -- 172.21.15.46:0/3473988584 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f00af600 con 0x7f35f8071110 2023-10-30T14:14:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35f77fe640 1 -- 172.21.15.46:0/3473988584 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f00ab050 con 0x7f35f8071110 2023-10-30T14:14:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35f77fe640 1 -- 172.21.15.46:0/3473988584 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f00af980 con 0x7f35f8071110 2023-10-30T14:14:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 -- 172.21.15.46:0/3473988584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f8071110 msgr2=0x7f35f8158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 --2- 172.21.15.46:0/3473988584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f8071110 0x7f35f8158110 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009d90 tx=0x7f35f001f130 comp rx=0 tx=0).stop 2023-10-30T14:14:41.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 -- 172.21.15.46:0/3473988584 shutdown_connections 2023-10-30T14:14:41.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 --2- 172.21.15.46:0/3473988584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f8071110 0x7f35f8158110 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 --2- 172.21.15.46:0/3473988584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35f806f7a0 0x7f35f806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 -- 172.21.15.46:0/3473988584 >> 172.21.15.46:0/3473988584 conn(0x7f35f80c7910 msgr2=0x7f35f80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:41.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 -- 172.21.15.46:0/3473988584 shutdown_connections 2023-10-30T14:14:41.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 -- 172.21.15.46:0/3473988584 wait complete. 2023-10-30T14:14:41.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 Processor -- start 2023-10-30T14:14:41.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 -- start start 2023-10-30T14:14:41.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35f806f7a0 0x7f35f81ece40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35f8071110 0x7f35f815a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f80c6bd0 0x7f35f815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f815a5a0 con 0x7f35f8071110 2023-10-30T14:14:41.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f8158ce0 con 0x7f35f80c6bd0 2023-10-30T14:14:41.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fee32640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f8158fc0 con 0x7f35f806f7a0 2023-10-30T14:14:41.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35f7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35f8071110 0x7f35f815a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35f7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35f8071110 0x7f35f815a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34336/0 (socket says 172.21.15.46:34336) 2023-10-30T14:14:41.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35f7fff640 1 -- 172.21.15.46:0/3729562217 learned_addr learned my addr 172.21.15.46:0/3729562217 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:41.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fcbce640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35f806f7a0 0x7f35f81ece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.630+0000 7f35fd3cf640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f80c6bd0 0x7f35f815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35f7fff640 1 -- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35f806f7a0 msgr2=0x7f35f81ece40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35f7fff640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35f806f7a0 0x7f35f81ece40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35f7fff640 1 -- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f80c6bd0 msgr2=0x7f35f815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35f7fff640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f80c6bd0 0x7f35f815aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35f7fff640 1 -- 172.21.15.46:0/3729562217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f0009a70 con 0x7f35f8071110 2023-10-30T14:14:41.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35f7fff640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35f8071110 0x7f35f815a060 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f35e400c420 tx=0x7f35e400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:41.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35f57fa640 1 -- 172.21.15.46:0/3729562217 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4005cf0 con 0x7f35f8071110 2023-10-30T14:14:41.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35f57fa640 1 -- 172.21.15.46:0/3729562217 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e4005e50 con 0x7f35f8071110 2023-10-30T14:14:41.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35fee32640 1 -- 172.21.15.46:0/3729562217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f81592a0 con 0x7f35f8071110 2023-10-30T14:14:41.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35f57fa640 1 -- 172.21.15.46:0/3729562217 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e40072f0 con 0x7f35f8071110 2023-10-30T14:14:41.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35fee32640 1 -- 172.21.15.46:0/3729562217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f8159860 con 0x7f35f8071110 2023-10-30T14:14:41.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35fee32640 1 -- 172.21.15.46:0/3729562217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f804e170 con 0x7f35f8071110 2023-10-30T14:14:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35f57fa640 1 -- 172.21.15.46:0/3729562217 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f35e4007450 con 0x7f35f8071110 2023-10-30T14:14:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35f57fa640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35dc0664a0 0x7f35dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.634+0000 7f35f57fa640 1 -- 172.21.15.46:0/3729562217 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f35e4083d70 con 0x7f35f8071110 2023-10-30T14:14:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.638+0000 7f35fcbce640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35dc0664a0 0x7f35dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.638+0000 7f35fcbce640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35dc0664a0 0x7f35dc068960 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f35ec009ac0 tx=0x7f35ec009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:41.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.638+0000 7f35f57fa640 1 -- 172.21.15.46:0/3729562217 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f35e40508b0 con 0x7f35f8071110 2023-10-30T14:14:41.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.834+0000 7f35fee32640 1 -- 172.21.15.46:0/3729562217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35f806a800 con 0x7f35f8071110 2023-10-30T14:14:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.834+0000 7f35f57fa640 1 -- 172.21.15.46:0/3729562217 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+6517 (secure 0 0 0) 0x7f35e4017030 con 0x7f35f8071110 2023-10-30T14:14:41.839 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:14:41.839 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9135,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:14:17.893242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 217 2023-10-30T14:14:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.838+0000 7f35fee32640 1 -- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35dc0664a0 msgr2=0x7f35dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.838+0000 7f35fee32640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35dc0664a0 0x7f35dc068960 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f35ec009ac0 tx=0x7f35ec009190 comp rx=0 tx=0).stop 2023-10-30T14:14:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.838+0000 7f35fee32640 1 -- 172.21.15.46:0/3729562217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35f8071110 msgr2=0x7f35f815a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.838+0000 7f35fee32640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35f8071110 0x7f35f815a060 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f35e400c420 tx=0x7f35e400c8e0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.838+0000 7f35fee32640 1 -- 172.21.15.46:0/3729562217 shutdown_connections 2023-10-30T14:14:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.838+0000 7f35fee32640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35dc0664a0 0x7f35dc068960 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.838+0000 7f35fee32640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f80c6bd0 0x7f35f815aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.838+0000 7f35fee32640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35f8071110 0x7f35f815a060 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.838+0000 7f35fee32640 1 --2- 172.21.15.46:0/3729562217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35f806f7a0 0x7f35f81ece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:41.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.838+0000 7f35fee32640 1 -- 172.21.15.46:0/3729562217 >> 172.21.15.46:0/3729562217 conn(0x7f35f80c7910 msgr2=0x7f35f80c97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:41.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.838+0000 7f35fee32640 1 -- 172.21.15.46:0/3729562217 shutdown_connections 2023-10-30T14:14:41.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:41.838+0000 7f35fee32640 1 -- 172.21.15.46:0/3729562217 wait complete. 2023-10-30T14:14:41.860 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:14:43.861 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:44.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef2e565640 1 Processor -- start 2023-10-30T14:14:44.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef2e565640 1 -- start start 2023-10-30T14:14:44.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef2e565640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fef280cf160 0x7fef280cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef2e565640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef280cb800 con 0x7fef2815d6f0 2023-10-30T14:14:44.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef2e565640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef280cb330 con 0x7fef281538a0 2023-10-30T14:14:44.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef2e565640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef280cad70 con 0x7fef280cf160 2023-10-30T14:14:44.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef277fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fef2815d6f0 0x7fef2815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37644/0 (socket says 172.21.15.46:37644) 2023-10-30T14:14:44.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef277fe640 1 -- 172.21.15.46:0/647884211 learned_addr learned my addr 172.21.15.46:0/647884211 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:14:44.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef2cb02640 1 --2- 172.21.15.46:0/647884211 >> v2:172.21.15.154:3300/0 conn(0x7fef280cf160 0x7fef280cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482288781 0 0) 0x7fef280cb800 con 0x7fef2815d6f0 2023-10-30T14:14:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef10003530 con 0x7fef2815d6f0 2023-10-30T14:14:44.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4124179327 0 0) 0x7fef10003530 con 0x7fef2815d6f0 2023-10-30T14:14:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 >> v2:172.21.15.154:3300/0 conn(0x7fef280cf160 msgr2=0x7fef280cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 --2- 172.21.15.46:0/647884211 >> v2:172.21.15.154:3300/0 conn(0x7fef280cf160 0x7fef280cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 >> v1:172.21.15.154:6789/0 conn(0x7fef281538a0 legacy=0x7fef280ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef280cb050 con 0x7fef2815d6f0 2023-10-30T14:14:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fef1c001680 con 0x7fef2815d6f0 2023-10-30T14:14:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 >> v1:172.21.15.46:6789/0 conn(0x7fef2815d6f0 legacy=0x7fef2815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 --2- 172.21.15.46:0/647884211 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef10003980 0x7fef10003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 --2- 172.21.15.46:0/647884211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef100047d0 0x7fef10004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 --2- 172.21.15.46:0/647884211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef10005190 0x7fef1000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef280cad70 con 0x7fef100047d0 2023-10-30T14:14:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1000c040 con 0x7fef10005190 2023-10-30T14:14:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1000c200 con 0x7fef10003980 2023-10-30T14:14:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef277fe640 1 --2- 172.21.15.46:0/647884211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef100047d0 0x7fef10004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef27fff640 1 --2- 172.21.15.46:0/647884211 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef10003980 0x7fef10003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.010+0000 7fef2cb02640 1 --2- 172.21.15.46:0/647884211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef10005190 0x7fef1000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef277fe640 1 -- 172.21.15.46:0/647884211 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef10003980 msgr2=0x7fef10003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef277fe640 1 --2- 172.21.15.46:0/647884211 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef10003980 0x7fef10003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef277fe640 1 -- 172.21.15.46:0/647884211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef10005190 msgr2=0x7fef1000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef277fe640 1 --2- 172.21.15.46:0/647884211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef10005190 0x7fef1000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef277fe640 1 -- 172.21.15.46:0/647884211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef10003530 con 0x7fef100047d0 2023-10-30T14:14:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef2cb02640 1 -- 172.21.15.46:0/647884211 reap_dead start 2023-10-30T14:14:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef277fe640 1 --2- 172.21.15.46:0/647884211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef100047d0 0x7fef10004c50 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00c240 tx=0x7fef1c01cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e5b4ca9decff0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:44.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c0a5b00 con 0x7fef100047d0 2023-10-30T14:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1c0a5c60 con 0x7fef100047d0 2023-10-30T14:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef26ffd640 1 -- 172.21.15.46:0/647884211 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c005b20 con 0x7fef100047d0 2023-10-30T14:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef2e565640 1 -- 172.21.15.46:0/647884211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef100047d0 msgr2=0x7fef10004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef2e565640 1 --2- 172.21.15.46:0/647884211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef100047d0 0x7fef10004c50 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00c240 tx=0x7fef1c01cc00 comp rx=0 tx=0).stop 2023-10-30T14:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef2e565640 1 -- 172.21.15.46:0/647884211 shutdown_connections 2023-10-30T14:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef2e565640 1 --2- 172.21.15.46:0/647884211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef100047d0 0x7fef10004c50 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef2e565640 1 -- 172.21.15.46:0/647884211 >> 172.21.15.46:0/647884211 conn(0x7fef2806bfe0 msgr2=0x7fef2806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef2e565640 1 -- 172.21.15.46:0/647884211 shutdown_connections 2023-10-30T14:14:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.014+0000 7fef2e565640 1 -- 172.21.15.46:0/647884211 wait complete. 2023-10-30T14:14:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2e565640 1 Processor -- start 2023-10-30T14:14:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2e565640 1 -- start start 2023-10-30T14:14:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2e565640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef100047d0 0x7fef281f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2e565640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef280661f0 0x7fef281f0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2e565640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef281f12b0 0x7fef2814fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2e565640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef281f0950 con 0x7fef281f12b0 2023-10-30T14:14:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2e565640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef28158590 con 0x7fef280661f0 2023-10-30T14:14:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2e565640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef281f3520 con 0x7fef100047d0 2023-10-30T14:14:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2cb02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef281f12b0 0x7fef2814fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2cb02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef281f12b0 0x7fef2814fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34342/0 (socket says 172.21.15.46:34342) 2023-10-30T14:14:44.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2cb02640 1 -- 172.21.15.46:0/1044792139 learned_addr learned my addr 172.21.15.46:0/1044792139 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:44.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef27fff640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef100047d0 0x7fef281f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef277fe640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef280661f0 0x7fef281f0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2cb02640 1 -- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef100047d0 msgr2=0x7fef281f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2cb02640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef100047d0 0x7fef281f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2cb02640 1 -- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef280661f0 msgr2=0x7fef281f0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2cb02640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef280661f0 0x7fef281f0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2cb02640 1 -- 172.21.15.46:0/1044792139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef281f1730 con 0x7fef281f12b0 2023-10-30T14:14:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef277fe640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef280661f0 0x7fef281f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2cb02640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef281f12b0 0x7fef2814fb00 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fef20007c20 tx=0x7fef200115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef24ff9640 1 -- 172.21.15.46:0/1044792139 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2000e070 con 0x7fef281f12b0 2023-10-30T14:14:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef24ff9640 1 -- 172.21.15.46:0/1044792139 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef20007e90 con 0x7fef281f12b0 2023-10-30T14:14:44.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef24ff9640 1 -- 172.21.15.46:0/1044792139 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef20011de0 con 0x7fef281f12b0 2023-10-30T14:14:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.018+0000 7fef2e565640 1 -- 172.21.15.46:0/1044792139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef281587c0 con 0x7fef281f12b0 2023-10-30T14:14:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.022+0000 7fef2e565640 1 -- 172.21.15.46:0/1044792139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef28158d80 con 0x7fef281f12b0 2023-10-30T14:14:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.022+0000 7fef2e565640 1 -- 172.21.15.46:0/1044792139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef281f0a90 con 0x7fef281f12b0 2023-10-30T14:14:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.022+0000 7fef24ff9640 1 -- 172.21.15.46:0/1044792139 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fef200039a0 con 0x7fef281f12b0 2023-10-30T14:14:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.022+0000 7fef24ff9640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fef140664a0 0x7fef14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.022+0000 7fef24ff9640 1 -- 172.21.15.46:0/1044792139 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7fef20084fe0 con 0x7fef281f12b0 2023-10-30T14:14:44.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.022+0000 7fef27fff640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fef140664a0 0x7fef14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:44.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.026+0000 7fef27fff640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fef140664a0 0x7fef14068960 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fef18000c10 tx=0x7fef18009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:44.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.026+0000 7fef24ff9640 1 -- 172.21.15.46:0/1044792139 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fef281f0a90 con 0x7fef281f12b0 2023-10-30T14:14:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 -- 172.21.15.46:0/1044792139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef281f0a90 con 0x7fef281f12b0 2023-10-30T14:14:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef24ff9640 1 -- 172.21.15.46:0/1044792139 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+6517 (secure 0 0 0) 0x7fef281f0a90 con 0x7fef281f12b0 2023-10-30T14:14:44.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:14:44.221 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9135,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:14:17.893242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 217 2023-10-30T14:14:44.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 -- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fef140664a0 msgr2=0x7fef14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fef140664a0 0x7fef14068960 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fef18000c10 tx=0x7fef18009040 comp rx=0 tx=0).stop 2023-10-30T14:14:44.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 -- 172.21.15.46:0/1044792139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef281f12b0 msgr2=0x7fef2814fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:44.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef281f12b0 0x7fef2814fb00 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fef20007c20 tx=0x7fef200115a0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 -- 172.21.15.46:0/1044792139 shutdown_connections 2023-10-30T14:14:44.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fef140664a0 0x7fef14068960 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef281f12b0 0x7fef2814fb00 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef280661f0 0x7fef281f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 --2- 172.21.15.46:0/1044792139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef100047d0 0x7fef281f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 -- 172.21.15.46:0/1044792139 >> 172.21.15.46:0/1044792139 conn(0x7fef2806bfe0 msgr2=0x7fef280c4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 -- 172.21.15.46:0/1044792139 shutdown_connections 2023-10-30T14:14:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:44.218+0000 7fef2e565640 1 -- 172.21.15.46:0/1044792139 wait complete. 2023-10-30T14:14:44.242 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:14:46.244 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:46.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.394+0000 7faa054ea640 1 Processor -- start 2023-10-30T14:14:46.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.394+0000 7faa054ea640 1 -- start start 2023-10-30T14:14:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7faa054ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faa00153d70 0x7faa0006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7faa054ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa0006e470 0x7faa000cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7faa054ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faa000c8490 0x7faa000cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7faa054ea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0006d780 con 0x7faa000c8490 2023-10-30T14:14:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7faa054ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0006afa0 con 0x7faa00153d70 2023-10-30T14:14:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7faa054ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa00154150 con 0x7faa0006e470 2023-10-30T14:14:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7fa9ff7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faa000c8490 0x7faa000cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7fa9ff7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faa000c8490 0x7faa000cd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34344/0 (socket says 172.21.15.46:34344) 2023-10-30T14:14:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7fa9ff7fe640 1 -- 172.21.15.46:0/2712853882 learned_addr learned my addr 172.21.15.46:0/2712853882 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7fa9feffd640 1 --2- 172.21.15.46:0/2712853882 >> v2:172.21.15.154:3300/0 conn(0x7faa00153d70 0x7faa0006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7fa9f7fff640 1 --2- 172.21.15.46:0/2712853882 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa0006e470 0x7faa000cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7fa9ff7fe640 1 -- 172.21.15.46:0/2712853882 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa0006e470 msgr2=0x7faa000cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:46.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7fa9ff7fe640 1 --2- 172.21.15.46:0/2712853882 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa0006e470 0x7faa000cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7fa9ff7fe640 1 -- 172.21.15.46:0/2712853882 >> v2:172.21.15.154:3300/0 conn(0x7faa00153d70 msgr2=0x7faa0006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:46.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7fa9ff7fe640 1 --2- 172.21.15.46:0/2712853882 >> v2:172.21.15.154:3300/0 conn(0x7faa00153d70 0x7faa0006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7fa9ff7fe640 1 -- 172.21.15.46:0/2712853882 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0006e030 con 0x7faa000c8490 2023-10-30T14:14:46.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.398+0000 7fa9feffd640 1 --2- 172.21.15.46:0/2712853882 >> v2:172.21.15.154:3300/0 conn(0x7faa00153d70 0x7faa0006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:46.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7fa9ff7fe640 1 --2- 172.21.15.46:0/2712853882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000c8490 0x7faa000cd340 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7faa0006d180 tx=0x7fa9f801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5edf8713d0bd8a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:46.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7fa9fe7fc640 1 -- 172.21.15.46:0/2712853882 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f80a5820 con 0x7faa000c8490 2023-10-30T14:14:46.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7fa9fe7fc640 1 -- 172.21.15.46:0/2712853882 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f80a5980 con 0x7faa000c8490 2023-10-30T14:14:46.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7fa9fe7fc640 1 -- 172.21.15.46:0/2712853882 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f80a5d10 con 0x7faa000c8490 2023-10-30T14:14:46.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7faa054ea640 1 -- 172.21.15.46:0/2712853882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000c8490 msgr2=0x7faa000cd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:46.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7faa054ea640 1 --2- 172.21.15.46:0/2712853882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000c8490 0x7faa000cd340 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7faa0006d180 tx=0x7fa9f801c9a0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7faa054ea640 1 -- 172.21.15.46:0/2712853882 shutdown_connections 2023-10-30T14:14:46.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7faa054ea640 1 --2- 172.21.15.46:0/2712853882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000c8490 0x7faa000cd340 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7faa054ea640 1 --2- 172.21.15.46:0/2712853882 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa0006e470 0x7faa000cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7faa054ea640 1 --2- 172.21.15.46:0/2712853882 >> v2:172.21.15.154:3300/0 conn(0x7faa00153d70 0x7faa0006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7faa054ea640 1 -- 172.21.15.46:0/2712853882 >> 172.21.15.46:0/2712853882 conn(0x7faa00071ea0 msgr2=0x7faa00065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:46.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7faa054ea640 1 -- 172.21.15.46:0/2712853882 shutdown_connections 2023-10-30T14:14:46.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7faa054ea640 1 -- 172.21.15.46:0/2712853882 wait complete. 2023-10-30T14:14:46.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7faa054ea640 1 Processor -- start 2023-10-30T14:14:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.402+0000 7faa054ea640 1 -- start start 2023-10-30T14:14:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7faa054ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0006e470 0x7faa00156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7faa054ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa000c8490 0x7faa000cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7faa054ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa00153d70 0x7faa000b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7faa054ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa000adae0 con 0x7faa0006e470 2023-10-30T14:14:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7faa054ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa000ae660 con 0x7faa000c8490 2023-10-30T14:14:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7faa054ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa001f3210 con 0x7faa00153d70 2023-10-30T14:14:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9feffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0006e470 0x7faa00156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9feffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0006e470 0x7faa00156bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34348/0 (socket says 172.21.15.46:34348) 2023-10-30T14:14:46.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9feffd640 1 -- 172.21.15.46:0/763600729 learned_addr learned my addr 172.21.15.46:0/763600729 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:46.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9f7fff640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa000c8490 0x7faa000cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9ff7fe640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa00153d70 0x7faa000b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9feffd640 1 -- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa00153d70 msgr2=0x7faa000b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:46.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9feffd640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa00153d70 0x7faa000b0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9feffd640 1 -- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa000c8490 msgr2=0x7faa000cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9feffd640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa000c8490 0x7faa000cc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9feffd640 1 -- 172.21.15.46:0/763600729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f8009b30 con 0x7faa0006e470 2023-10-30T14:14:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9ff7fe640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa00153d70 0x7faa000b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9f7fff640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa000c8490 0x7faa000cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9feffd640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0006e470 0x7faa00156bb0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000d830 tx=0x7fa9f000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9f77fe640 1 -- 172.21.15.46:0/763600729 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f000b660 con 0x7faa0006e470 2023-10-30T14:14:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9f77fe640 1 -- 172.21.15.46:0/763600729 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f000b7c0 con 0x7faa0006e470 2023-10-30T14:14:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9f77fe640 1 -- 172.21.15.46:0/763600729 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f000bb70 con 0x7faa0006e470 2023-10-30T14:14:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7faa054ea640 1 -- 172.21.15.46:0/763600729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa000add30 con 0x7faa0006e470 2023-10-30T14:14:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7faa054ea640 1 -- 172.21.15.46:0/763600729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa000ae2f0 con 0x7faa0006e470 2023-10-30T14:14:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7faa054ea640 1 -- 172.21.15.46:0/763600729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa000ae4b0 con 0x7faa0006e470 2023-10-30T14:14:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.406+0000 7fa9f77fe640 1 -- 172.21.15.46:0/763600729 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa9f001e460 con 0x7faa0006e470 2023-10-30T14:14:46.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.410+0000 7fa9f77fe640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9e00664a0 0x7fa9e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:46.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.410+0000 7fa9f77fe640 1 -- 172.21.15.46:0/763600729 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7fa9f0084f00 con 0x7faa0006e470 2023-10-30T14:14:46.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.410+0000 7fa9f7fff640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9e00664a0 0x7fa9e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:46.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.410+0000 7fa9f7fff640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9e00664a0 0x7fa9e0068960 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7faa001486e0 tx=0x7fa9e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:46.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.414+0000 7fa9f77fe640 1 -- 172.21.15.46:0/763600729 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa9f0051990 con 0x7faa0006e470 2023-10-30T14:14:46.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.602+0000 7faa054ea640 1 -- 172.21.15.46:0/763600729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa0004e170 con 0x7faa0006e470 2023-10-30T14:14:46.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.602+0000 7fa9f77fe640 1 -- 172.21.15.46:0/763600729 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+6517 (secure 0 0 0) 0x7fa9f0082e60 con 0x7faa0006e470 2023-10-30T14:14:46.606 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:14:46.606 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9135,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:14:17.893242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 217 2023-10-30T14:14:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.606+0000 7faa054ea640 1 -- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9e00664a0 msgr2=0x7fa9e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.606+0000 7faa054ea640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9e00664a0 0x7fa9e0068960 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7faa001486e0 tx=0x7fa9e8008040 comp rx=0 tx=0).stop 2023-10-30T14:14:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.606+0000 7faa054ea640 1 -- 172.21.15.46:0/763600729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0006e470 msgr2=0x7faa00156bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:46.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.606+0000 7faa054ea640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0006e470 0x7faa00156bb0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000d830 tx=0x7fa9f000dcf0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.606+0000 7faa054ea640 1 -- 172.21.15.46:0/763600729 shutdown_connections 2023-10-30T14:14:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.606+0000 7faa054ea640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9e00664a0 0x7fa9e0068960 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.606+0000 7faa054ea640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa00153d70 0x7faa000b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.606+0000 7faa054ea640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa000c8490 0x7faa000cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.606+0000 7faa054ea640 1 --2- 172.21.15.46:0/763600729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0006e470 0x7faa00156bb0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:46.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.606+0000 7faa054ea640 1 -- 172.21.15.46:0/763600729 >> 172.21.15.46:0/763600729 conn(0x7faa00071ea0 msgr2=0x7faa000c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:46.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.606+0000 7faa054ea640 1 -- 172.21.15.46:0/763600729 shutdown_connections 2023-10-30T14:14:46.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:46.606+0000 7faa054ea640 1 -- 172.21.15.46:0/763600729 wait complete. 2023-10-30T14:14:46.627 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:14:48.629 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:48.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f186d8d6640 1 Processor -- start 2023-10-30T14:14:48.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f186d8d6640 1 -- start start 2023-10-30T14:14:48.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f186d8d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f18681538a0 0x7f186815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:48.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f186d8d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18680ca0e0 0x7f186815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:48.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f186d8d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f18680cf160 0x7f18680cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:48.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f186d8d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18680cb800 con 0x7f18680cf160 2023-10-30T14:14:48.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f186d8d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186815dc30 con 0x7f18681538a0 2023-10-30T14:14:48.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f186d8d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18680cb330 con 0x7f18680ca0e0 2023-10-30T14:14:48.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f18677fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f18680cf160 0x7f18680cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:48.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f18677fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f18680cf160 0x7f18680cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46498/0 (socket says 172.21.15.46:46498) 2023-10-30T14:14:48.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f18677fe640 1 -- 172.21.15.46:0/766948761 learned_addr learned my addr 172.21.15.46:0/766948761 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:48.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f1866ffd640 1 --2- 172.21.15.46:0/766948761 >> v2:172.21.15.154:3300/0 conn(0x7f18681538a0 0x7f186815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:48.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f18667fc640 1 --2- 172.21.15.46:0/766948761 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18680ca0e0 0x7f186815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:48.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f18677fe640 1 -- 172.21.15.46:0/766948761 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18680ca0e0 msgr2=0x7f186815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:48.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f18677fe640 1 --2- 172.21.15.46:0/766948761 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18680ca0e0 0x7f186815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f18677fe640 1 -- 172.21.15.46:0/766948761 >> v2:172.21.15.154:3300/0 conn(0x7f18681538a0 msgr2=0x7f186815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:48.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f18677fe640 1 --2- 172.21.15.46:0/766948761 >> v2:172.21.15.154:3300/0 conn(0x7f18681538a0 0x7f186815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f18677fe640 1 -- 172.21.15.46:0/766948761 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18680cad70 con 0x7f18680cf160 2023-10-30T14:14:48.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.782+0000 7f1866ffd640 1 --2- 172.21.15.46:0/766948761 >> v2:172.21.15.154:3300/0 conn(0x7f18681538a0 0x7f186815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:48.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f18677fe640 1 --2- 172.21.15.46:0/766948761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18680cf160 0x7f18680cf590 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f18680cfd10 tx=0x7f186001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb8e0cc727c2ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:48.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f1865ffb640 1 -- 172.21.15.46:0/766948761 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18600af400 con 0x7f18680cf160 2023-10-30T14:14:48.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f1865ffb640 1 -- 172.21.15.46:0/766948761 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18600ab050 con 0x7f18680cf160 2023-10-30T14:14:48.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f1865ffb640 1 -- 172.21.15.46:0/766948761 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18600af780 con 0x7f18680cf160 2023-10-30T14:14:48.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f186d8d6640 1 -- 172.21.15.46:0/766948761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18680cf160 msgr2=0x7f18680cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:48.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f186d8d6640 1 --2- 172.21.15.46:0/766948761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18680cf160 0x7f18680cf590 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f18680cfd10 tx=0x7f186001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f186d8d6640 1 -- 172.21.15.46:0/766948761 shutdown_connections 2023-10-30T14:14:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f186d8d6640 1 --2- 172.21.15.46:0/766948761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18680cf160 0x7f18680cf590 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f186d8d6640 1 --2- 172.21.15.46:0/766948761 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18680ca0e0 0x7f186815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f186d8d6640 1 --2- 172.21.15.46:0/766948761 >> v2:172.21.15.154:3300/0 conn(0x7f18681538a0 0x7f186815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f186d8d6640 1 -- 172.21.15.46:0/766948761 >> 172.21.15.46:0/766948761 conn(0x7f186806bfe0 msgr2=0x7f186806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f186d8d6640 1 -- 172.21.15.46:0/766948761 shutdown_connections 2023-10-30T14:14:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.786+0000 7f186d8d6640 1 -- 172.21.15.46:0/766948761 wait complete. 2023-10-30T14:14:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f186d8d6640 1 Processor -- start 2023-10-30T14:14:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f186d8d6640 1 -- start start 2023-10-30T14:14:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f186d8d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18680ca0e0 0x7f18681e2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f186d8d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18680cf160 0x7f18680cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:48.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f186d8d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18681538a0 0x7f18680b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:48.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f186d8d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18681e31e0 con 0x7f18680cf160 2023-10-30T14:14:48.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f186d8d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18681e34b0 con 0x7f18681538a0 2023-10-30T14:14:48.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f186d8d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18681ef390 con 0x7f18680ca0e0 2023-10-30T14:14:48.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f18667fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18680cf160 0x7f18680cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:48.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f18667fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18680cf160 0x7f18680cd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46500/0 (socket says 172.21.15.46:46500) 2023-10-30T14:14:48.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f18667fc640 1 -- 172.21.15.46:0/1947512578 learned_addr learned my addr 172.21.15.46:0/1947512578 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:48.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f1866ffd640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18680ca0e0 0x7f18681e2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:48.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f18677fe640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18681538a0 0x7f18680b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:48.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f18667fc640 1 -- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18680ca0e0 msgr2=0x7f18681e2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:48.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f18667fc640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18680ca0e0 0x7f18681e2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f18667fc640 1 -- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18681538a0 msgr2=0x7f18680b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:48.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f18667fc640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18681538a0 0x7f18680b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f18667fc640 1 -- 172.21.15.46:0/1947512578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1860009b30 con 0x7f18680cf160 2023-10-30T14:14:48.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f1866ffd640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18680ca0e0 0x7f18681e2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:48.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f18667fc640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18680cf160 0x7f18680cd610 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f185c00d800 tx=0x7f185c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:48.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f184ffff640 1 -- 172.21.15.46:0/1947512578 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0096a0 con 0x7f18680cf160 2023-10-30T14:14:48.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f184ffff640 1 -- 172.21.15.46:0/1947512578 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c009800 con 0x7f18680cf160 2023-10-30T14:14:48.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f184ffff640 1 -- 172.21.15.46:0/1947512578 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c009b70 con 0x7f18680cf160 2023-10-30T14:14:48.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f186d8d6640 1 -- 172.21.15.46:0/1947512578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18680b3240 con 0x7f18680cf160 2023-10-30T14:14:48.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.790+0000 7f186d8d6640 1 -- 172.21.15.46:0/1947512578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18680b3800 con 0x7f18680cf160 2023-10-30T14:14:48.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.794+0000 7f186d8d6640 1 -- 172.21.15.46:0/1947512578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18680b2a50 con 0x7f18680cf160 2023-10-30T14:14:48.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.794+0000 7f184ffff640 1 -- 172.21.15.46:0/1947512578 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f185c009cd0 con 0x7f18680cf160 2023-10-30T14:14:48.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.794+0000 7f184ffff640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f18540664a0 0x7f1854068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:48.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.794+0000 7f184ffff640 1 -- 172.21.15.46:0/1947512578 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f185c084e90 con 0x7f18680cf160 2023-10-30T14:14:48.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.794+0000 7f1866ffd640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f18540664a0 0x7f1854068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:48.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.794+0000 7f1866ffd640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f18540664a0 0x7f1854068960 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f185800b990 tx=0x7f1858008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:48.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.798+0000 7f184ffff640 1 -- 172.21.15.46:0/1947512578 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f185c0519b0 con 0x7f18680cf160 2023-10-30T14:14:48.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 -- 172.21.15.46:0/1947512578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f186804e170 con 0x7f18680cf160 2023-10-30T14:14:48.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f184ffff640 1 -- 172.21.15.46:0/1947512578 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+6517 (secure 0 0 0) 0x7f185c086070 con 0x7f18680cf160 2023-10-30T14:14:48.992 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:14:48.993 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9135,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:14:17.893242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:48.993 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 217 2023-10-30T14:14:48.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 -- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f18540664a0 msgr2=0x7f1854068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:48.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f18540664a0 0x7f1854068960 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f185800b990 tx=0x7f1858008040 comp rx=0 tx=0).stop 2023-10-30T14:14:48.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 -- 172.21.15.46:0/1947512578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18680cf160 msgr2=0x7f18680cd610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:48.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18680cf160 0x7f18680cd610 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f185c00d800 tx=0x7f185c00dcc0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 -- 172.21.15.46:0/1947512578 shutdown_connections 2023-10-30T14:14:48.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f18540664a0 0x7f1854068960 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18681538a0 0x7f18680b4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18680cf160 0x7f18680cd610 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 --2- 172.21.15.46:0/1947512578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18680ca0e0 0x7f18681e2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:48.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 -- 172.21.15.46:0/1947512578 >> 172.21.15.46:0/1947512578 conn(0x7f186806bfe0 msgr2=0x7f1868151880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:48.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 -- 172.21.15.46:0/1947512578 shutdown_connections 2023-10-30T14:14:48.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:48.990+0000 7f186d8d6640 1 -- 172.21.15.46:0/1947512578 wait complete. 2023-10-30T14:14:49.013 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:14:51.014 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:51.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.166+0000 7f8433d94640 1 Processor -- start 2023-10-30T14:14:51.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.166+0000 7f8433d94640 1 -- start start 2023-10-30T14:14:51.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.166+0000 7f8433d94640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0c6bd0 0x7f842c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:51.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.166+0000 7f8433d94640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842c066cf0 con 0x7f842c071110 2023-10-30T14:14:51.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.166+0000 7f8433d94640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842c0c70d0 con 0x7f842c06f7a0 2023-10-30T14:14:51.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.166+0000 7f8433d94640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842c0700c0 con 0x7f842c0c6bd0 2023-10-30T14:14:51.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.166+0000 7f8432331640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f842c071110 0x7f842c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49042/0 (socket says 172.21.15.46:49042) 2023-10-30T14:14:51.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.166+0000 7f8432331640 1 -- 172.21.15.46:0/3193730524 learned_addr learned my addr 172.21.15.46:0/3193730524 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:14:51.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.166+0000 7f843132f640 1 --2- 172.21.15.46:0/3193730524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0c6bd0 0x7f842c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:51.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.170+0000 7f8430b2e640 1 -- 172.21.15.46:0/3193730524 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483788300 0 0) 0x7f842c066cf0 con 0x7f842c071110 2023-10-30T14:14:51.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.170+0000 7f8430b2e640 1 -- 172.21.15.46:0/3193730524 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8418003530 con 0x7f842c071110 2023-10-30T14:14:51.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.170+0000 7f8430b2e640 1 -- 172.21.15.46:0/3193730524 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436802631 0 0) 0x7f842c0c70d0 con 0x7f842c06f7a0 2023-10-30T14:14:51.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.170+0000 7f8430b2e640 1 -- 172.21.15.46:0/3193730524 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f842c066cf0 con 0x7f842c06f7a0 2023-10-30T14:14:51.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.170+0000 7f843132f640 1 -- 172.21.15.46:0/3193730524 >> v1:172.21.15.154:6789/0 conn(0x7f842c06f7a0 legacy=0x7f842c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.170+0000 7f843132f640 1 -- 172.21.15.46:0/3193730524 >> v1:172.21.15.46:6789/0 conn(0x7f842c071110 legacy=0x7f842c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.170+0000 7f843132f640 1 -- 172.21.15.46:0/3193730524 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c0c7390 con 0x7f842c0c6bd0 2023-10-30T14:14:51.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.170+0000 7f843132f640 1 --2- 172.21.15.46:0/3193730524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0c6bd0 0x7f842c06fb80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f841c0097f0 tx=0x7f841c01ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a956291a09f9120 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:51.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.170+0000 7f8430b2e640 1 -- 172.21.15.46:0/3193730524 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c0a89b0 con 0x7f842c0c6bd0 2023-10-30T14:14:51.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.170+0000 7f8430b2e640 1 -- 172.21.15.46:0/3193730524 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841c0a8b10 con 0x7f842c0c6bd0 2023-10-30T14:14:51.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.170+0000 7f8430b2e640 1 -- 172.21.15.46:0/3193730524 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c0a8e80 con 0x7f842c0c6bd0 2023-10-30T14:14:51.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 -- 172.21.15.46:0/3193730524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0c6bd0 msgr2=0x7f842c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 --2- 172.21.15.46:0/3193730524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0c6bd0 0x7f842c06fb80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f841c0097f0 tx=0x7f841c01ecb0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 -- 172.21.15.46:0/3193730524 shutdown_connections 2023-10-30T14:14:51.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 --2- 172.21.15.46:0/3193730524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0c6bd0 0x7f842c06fb80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 -- 172.21.15.46:0/3193730524 >> 172.21.15.46:0/3193730524 conn(0x7f842c0c7910 msgr2=0x7f842c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:51.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 -- 172.21.15.46:0/3193730524 shutdown_connections 2023-10-30T14:14:51.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 -- 172.21.15.46:0/3193730524 wait complete. 2023-10-30T14:14:51.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 Processor -- start 2023-10-30T14:14:51.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 -- start start 2023-10-30T14:14:51.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c06f7a0 0x7f842c1f0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:51.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f842c071110 0x7f842c14ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:51.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0c6bd0 0x7f842c15b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:51.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c15bc10 con 0x7f842c06f7a0 2023-10-30T14:14:51.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c158650 con 0x7f842c071110 2023-10-30T14:14:51.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8433d94640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842c1f0f80 con 0x7f842c0c6bd0 2023-10-30T14:14:51.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8431b30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c06f7a0 0x7f842c1f0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:51.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8431b30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c06f7a0 0x7f842c1f0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46506/0 (socket says 172.21.15.46:46506) 2023-10-30T14:14:51.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8431b30640 1 -- 172.21.15.46:0/2174666047 learned_addr learned my addr 172.21.15.46:0/2174666047 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:51.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f8432331640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0c6bd0 0x7f842c15b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:51.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.174+0000 7f843132f640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f842c071110 0x7f842c14ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:51.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f8431b30640 1 -- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0c6bd0 msgr2=0x7f842c15b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f8431b30640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0c6bd0 0x7f842c15b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f8431b30640 1 -- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f842c071110 msgr2=0x7f842c14ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f8431b30640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f842c071110 0x7f842c14ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f8431b30640 1 -- 172.21.15.46:0/2174666047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841c009420 con 0x7f842c06f7a0 2023-10-30T14:14:51.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f8431b30640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c06f7a0 0x7f842c1f0a40 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f842800c490 tx=0x7f842800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:51.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f84227fc640 1 -- 172.21.15.46:0/2174666047 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84280052d0 con 0x7f842c06f7a0 2023-10-30T14:14:51.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f84227fc640 1 -- 172.21.15.46:0/2174666047 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8428005450 con 0x7f842c06f7a0 2023-10-30T14:14:51.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f84227fc640 1 -- 172.21.15.46:0/2174666047 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8428012870 con 0x7f842c06f7a0 2023-10-30T14:14:51.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f8433d94640 1 -- 172.21.15.46:0/2174666047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842c157db0 con 0x7f842c06f7a0 2023-10-30T14:14:51.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f8433d94640 1 -- 172.21.15.46:0/2174666047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842c158370 con 0x7f842c06f7a0 2023-10-30T14:14:51.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f8433d94640 1 -- 172.21.15.46:0/2174666047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842c04e170 con 0x7f842c06f7a0 2023-10-30T14:14:51.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f84227fc640 1 -- 172.21.15.46:0/2174666047 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f84280129d0 con 0x7f842c06f7a0 2023-10-30T14:14:51.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f84227fc640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84100664a0 0x7f8410068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:51.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f84227fc640 1 -- 172.21.15.46:0/2174666047 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f8428084290 con 0x7f842c06f7a0 2023-10-30T14:14:51.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.178+0000 7f843132f640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84100664a0 0x7f8410068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:51.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.182+0000 7f843132f640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84100664a0 0x7f8410068960 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f841c0b2000 tx=0x7f841c0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:51.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.182+0000 7f84227fc640 1 -- 172.21.15.46:0/2174666047 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8428050d40 con 0x7f842c06f7a0 2023-10-30T14:14:51.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.374+0000 7f8433d94640 1 -- 172.21.15.46:0/2174666047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f842c14d930 con 0x7f842c06f7a0 2023-10-30T14:14:51.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.374+0000 7f84227fc640 1 -- 172.21.15.46:0/2174666047 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+6517 (secure 0 0 0) 0x7f8428054360 con 0x7f842c06f7a0 2023-10-30T14:14:51.378 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:14:51.378 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9135,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:14:17.893242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:51.378 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 217 2023-10-30T14:14:51.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.378+0000 7f8433d94640 1 -- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84100664a0 msgr2=0x7f8410068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.378+0000 7f8433d94640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84100664a0 0x7f8410068960 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f841c0b2000 tx=0x7f841c0aa070 comp rx=0 tx=0).stop 2023-10-30T14:14:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.378+0000 7f8433d94640 1 -- 172.21.15.46:0/2174666047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c06f7a0 msgr2=0x7f842c1f0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.378+0000 7f8433d94640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c06f7a0 0x7f842c1f0a40 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f842800c490 tx=0x7f842800c950 comp rx=0 tx=0).stop 2023-10-30T14:14:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.378+0000 7f8433d94640 1 -- 172.21.15.46:0/2174666047 shutdown_connections 2023-10-30T14:14:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.378+0000 7f8433d94640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84100664a0 0x7f8410068960 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.378+0000 7f8433d94640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0c6bd0 0x7f842c15b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.378+0000 7f8433d94640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f842c071110 0x7f842c14ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.378+0000 7f8433d94640 1 --2- 172.21.15.46:0/2174666047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c06f7a0 0x7f842c1f0a40 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.378+0000 7f8433d94640 1 -- 172.21.15.46:0/2174666047 >> 172.21.15.46:0/2174666047 conn(0x7f842c0c7910 msgr2=0x7f842c0c2790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.378+0000 7f8433d94640 1 -- 172.21.15.46:0/2174666047 shutdown_connections 2023-10-30T14:14:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:51.378+0000 7f8433d94640 1 -- 172.21.15.46:0/2174666047 wait complete. 2023-10-30T14:14:51.399 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:14:53.400 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:53.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.549+0000 7f6a196d6640 1 Processor -- start 2023-10-30T14:14:53.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.549+0000 7f6a196d6640 1 -- start start 2023-10-30T14:14:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a196d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a1406f7a0 0x7f6a1406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a196d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a14066cf0 con 0x7f6a140c6bd0 2023-10-30T14:14:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a196d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a140c70d0 con 0x7f6a14071110 2023-10-30T14:14:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a196d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a140700c0 con 0x7f6a1406f7a0 2023-10-30T14:14:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a0bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6a140c6bd0 0x7f6a14069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49044/0 (socket says 172.21.15.46:49044) 2023-10-30T14:14:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a0bfff640 1 -- 172.21.15.46:0/3654996666 learned_addr learned my addr 172.21.15.46:0/3654996666 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:14:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a12ffd640 1 --2- 172.21.15.46:0/3654996666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a1406f7a0 0x7f6a1406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159967935 0 0) 0x7f6a14066cf0 con 0x7f6a140c6bd0 2023-10-30T14:14:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a00003530 con 0x7f6a140c6bd0 2023-10-30T14:14:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2393088984 0 0) 0x7f6a00003530 con 0x7f6a140c6bd0 2023-10-30T14:14:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a1406f7a0 msgr2=0x7f6a1406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 --2- 172.21.15.46:0/3654996666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a1406f7a0 0x7f6a1406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 >> v1:172.21.15.154:6789/0 conn(0x7f6a14071110 legacy=0x7f6a14158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a140c7390 con 0x7f6a140c6bd0 2023-10-30T14:14:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f69fc001680 con 0x7f6a140c6bd0 2023-10-30T14:14:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 >> v1:172.21.15.46:6789/0 conn(0x7f6a140c6bd0 legacy=0x7f6a14069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 --2- 172.21.15.46:0/3654996666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a1406f7a0 0x7f6a000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 --2- 172.21.15.46:0/3654996666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a00003ef0 0x7f6a000047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 --2- 172.21.15.46:0/3654996666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a00004d10 0x7f6a00005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a140700c0 con 0x7f6a00004d10 2023-10-30T14:14:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0000b9f0 con 0x7f6a00003ef0 2023-10-30T14:14:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0000bbb0 con 0x7f6a1406f7a0 2023-10-30T14:14:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a137fe640 1 --2- 172.21.15.46:0/3654996666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a00004d10 0x7f6a00005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a0bfff640 1 --2- 172.21.15.46:0/3654996666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a00003ef0 0x7f6a000047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a12ffd640 1 --2- 172.21.15.46:0/3654996666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a1406f7a0 0x7f6a000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a137fe640 1 -- 172.21.15.46:0/3654996666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a1406f7a0 msgr2=0x7f6a000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a137fe640 1 --2- 172.21.15.46:0/3654996666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a1406f7a0 0x7f6a000039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a137fe640 1 -- 172.21.15.46:0/3654996666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a00003ef0 msgr2=0x7f6a000047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a137fe640 1 --2- 172.21.15.46:0/3654996666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a00003ef0 0x7f6a000047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.553+0000 7f6a137fe640 1 -- 172.21.15.46:0/3654996666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a00003530 con 0x7f6a00004d10 2023-10-30T14:14:53.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.557+0000 7f6a137fe640 1 --2- 172.21.15.46:0/3654996666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a00004d10 0x7f6a00005190 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c001f60 tx=0x7f6a0c01c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10d9a3aa3de4dce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:53.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.557+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0a5750 con 0x7f6a00004d10 2023-10-30T14:14:53.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.557+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c0a58b0 con 0x7f6a00004d10 2023-10-30T14:14:53.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.557+0000 7f6a127fc640 1 -- 172.21.15.46:0/3654996666 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0a5ca0 con 0x7f6a00004d10 2023-10-30T14:14:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.557+0000 7f6a196d6640 1 -- 172.21.15.46:0/3654996666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a00004d10 msgr2=0x7f6a00005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.557+0000 7f6a196d6640 1 --2- 172.21.15.46:0/3654996666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a00004d10 0x7f6a00005190 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c001f60 tx=0x7f6a0c01c860 comp rx=0 tx=0).stop 2023-10-30T14:14:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.557+0000 7f6a137fe640 1 -- 172.21.15.46:0/3654996666 reap_dead start 2023-10-30T14:14:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.557+0000 7f6a196d6640 1 -- 172.21.15.46:0/3654996666 shutdown_connections 2023-10-30T14:14:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.557+0000 7f6a196d6640 1 -- 172.21.15.46:0/3654996666 >> 172.21.15.46:0/3654996666 conn(0x7f6a140c7910 msgr2=0x7f6a140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.557+0000 7f6a196d6640 1 -- 172.21.15.46:0/3654996666 shutdown_connections 2023-10-30T14:14:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.557+0000 7f6a196d6640 1 -- 172.21.15.46:0/3654996666 wait complete. 2023-10-30T14:14:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a196d6640 1 Processor -- start 2023-10-30T14:14:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a196d6640 1 -- start start 2023-10-30T14:14:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a196d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a00004d10 0x7f6a1414f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a196d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a141ea410 0x7f6a14150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a196d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a1414fb80 0x7f6a140c1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a196d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a140c2ce0 con 0x7f6a00004d10 2023-10-30T14:14:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a196d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a141500d0 con 0x7f6a1414fb80 2023-10-30T14:14:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a196d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a141503b0 con 0x7f6a141ea410 2023-10-30T14:14:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a12ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a00004d10 0x7f6a1414f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a12ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a00004d10 0x7f6a1414f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46522/0 (socket says 172.21.15.46:46522) 2023-10-30T14:14:53.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a12ffd640 1 -- 172.21.15.46:0/789067924 learned_addr learned my addr 172.21.15.46:0/789067924 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:53.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a0bfff640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a141ea410 0x7f6a14150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a137fe640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a1414fb80 0x7f6a140c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a12ffd640 1 -- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a141ea410 msgr2=0x7f6a14150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:53.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a12ffd640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a141ea410 0x7f6a14150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a12ffd640 1 -- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a1414fb80 msgr2=0x7f6a140c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a12ffd640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a1414fb80 0x7f6a140c1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a12ffd640 1 -- 172.21.15.46:0/789067924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a14150d80 con 0x7f6a00004d10 2023-10-30T14:14:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a137fe640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a1414fb80 0x7f6a140c1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:14:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a12ffd640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a00004d10 0x7f6a1414f5f0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f6a04011a00 tx=0x7f6a04011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a0b7fe640 1 -- 172.21.15.46:0/789067924 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0400faf0 con 0x7f6a00004d10 2023-10-30T14:14:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a0b7fe640 1 -- 172.21.15.46:0/789067924 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0400fc50 con 0x7f6a00004d10 2023-10-30T14:14:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a196d6640 1 -- 172.21.15.46:0/789067924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a140c41b0 con 0x7f6a00004d10 2023-10-30T14:14:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a0b7fe640 1 -- 172.21.15.46:0/789067924 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a04014440 con 0x7f6a00004d10 2023-10-30T14:14:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a196d6640 1 -- 172.21.15.46:0/789067924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a140c2670 con 0x7f6a00004d10 2023-10-30T14:14:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.561+0000 7f6a196d6640 1 -- 172.21.15.46:0/789067924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1404e170 con 0x7f6a00004d10 2023-10-30T14:14:53.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.565+0000 7f6a0b7fe640 1 -- 172.21.15.46:0/789067924 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6a040145a0 con 0x7f6a00004d10 2023-10-30T14:14:53.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.565+0000 7f6a0b7fe640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69f40664a0 0x7f69f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.565+0000 7f6a0b7fe640 1 -- 172.21.15.46:0/789067924 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8090+0+0 (secure 0 0 0) 0x7f6a040850e0 con 0x7f6a00004d10 2023-10-30T14:14:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.565+0000 7f6a0bfff640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69f40664a0 0x7f69f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.565+0000 7f6a0bfff640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69f40664a0 0x7f69f4068960 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6a14148390 tx=0x7f69fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:53.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.569+0000 7f6a0b7fe640 1 -- 172.21.15.46:0/789067924 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6a04051c30 con 0x7f6a00004d10 2023-10-30T14:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.761+0000 7f6a196d6640 1 -- 172.21.15.46:0/789067924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a141483c0 con 0x7f6a00004d10 2023-10-30T14:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.761+0000 7f6a0b7fe640 1 -- 172.21.15.46:0/789067924 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+6517 (secure 0 0 0) 0x7f6a04055250 con 0x7f6a00004d10 2023-10-30T14:14:53.765 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:14:53.766 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9135,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:14:17.893242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8826,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8826":{"gid":8826,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.46:6836/66265765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":66265765},{"type":"v1","addr":"172.21.15.46:6836","nonce":66265765}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 217 2023-10-30T14:14:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.761+0000 7f6a196d6640 1 -- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69f40664a0 msgr2=0x7f69f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.761+0000 7f6a196d6640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69f40664a0 0x7f69f4068960 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6a14148390 tx=0x7f69fc009040 comp rx=0 tx=0).stop 2023-10-30T14:14:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.761+0000 7f6a196d6640 1 -- 172.21.15.46:0/789067924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a00004d10 msgr2=0x7f6a1414f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:53.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.761+0000 7f6a196d6640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a00004d10 0x7f6a1414f5f0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f6a04011a00 tx=0x7f6a04011ec0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.761+0000 7f6a196d6640 1 -- 172.21.15.46:0/789067924 shutdown_connections 2023-10-30T14:14:53.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.761+0000 7f6a196d6640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f69f40664a0 0x7f69f4068960 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.761+0000 7f6a196d6640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a1414fb80 0x7f6a140c1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.761+0000 7f6a196d6640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a141ea410 0x7f6a14150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.761+0000 7f6a196d6640 1 --2- 172.21.15.46:0/789067924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a00004d10 0x7f6a1414f5f0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:53.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.761+0000 7f6a196d6640 1 -- 172.21.15.46:0/789067924 >> 172.21.15.46:0/789067924 conn(0x7f6a140c7910 msgr2=0x7f6a140c5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:53.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.765+0000 7f6a196d6640 1 -- 172.21.15.46:0/789067924 shutdown_connections 2023-10-30T14:14:53.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:53.765+0000 7f6a196d6640 1 -- 172.21.15.46:0/789067924 wait complete. 2023-10-30T14:14:53.787 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:14:55.788 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.941+0000 7fc9b9194640 1 Processor -- start 2023-10-30T14:14:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.941+0000 7fc9b9194640 1 -- start start 2023-10-30T14:14:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.941+0000 7fc9b9194640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9b41538a0 0x7fc9b415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.941+0000 7fc9b9194640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9b40cf160 0x7fc9b40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:55.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.941+0000 7fc9b9194640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b40cb800 con 0x7fc9b40ca0e0 2023-10-30T14:14:55.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.941+0000 7fc9b9194640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b415dc30 con 0x7fc9b40cf160 2023-10-30T14:14:55.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.941+0000 7fc9b9194640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b40cb330 con 0x7fc9b41538a0 2023-10-30T14:14:55.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b359e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9b40cf160 0x7fc9b40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:55.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b359e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9b40cf160 0x7fc9b40cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46538/0 (socket says 172.21.15.46:46538) 2023-10-30T14:14:55.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b359e640 1 -- 172.21.15.46:0/1519413499 learned_addr learned my addr 172.21.15.46:0/1519413499 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:55.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b259c640 1 --1- 172.21.15.46:0/1519413499 >> v1:172.21.15.46:6789/0 conn(0x7fc9b40ca0e0 0x7fc9b415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49054/0 (socket says 172.21.15.46:49054) 2023-10-30T14:14:55.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b2d9d640 1 --2- 172.21.15.46:0/1519413499 >> v2:172.21.15.154:3300/0 conn(0x7fc9b41538a0 0x7fc9b415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:55.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b359e640 1 -- 172.21.15.46:0/1519413499 >> v2:172.21.15.154:3300/0 conn(0x7fc9b41538a0 msgr2=0x7fc9b415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:55.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b359e640 1 --2- 172.21.15.46:0/1519413499 >> v2:172.21.15.154:3300/0 conn(0x7fc9b41538a0 0x7fc9b415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:55.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b359e640 1 -- 172.21.15.46:0/1519413499 >> v1:172.21.15.46:6789/0 conn(0x7fc9b40ca0e0 legacy=0x7fc9b415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:55.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b359e640 1 -- 172.21.15.46:0/1519413499 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b40cad70 con 0x7fc9b40cf160 2023-10-30T14:14:55.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b1d9b640 1 -- 172.21.15.46:0/1519413499 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830571005 0 0) 0x7fc9b40cb800 con 0x7fc9b40ca0e0 2023-10-30T14:14:55.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b359e640 1 --2- 172.21.15.46:0/1519413499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b40cf160 0x7fc9b40cf590 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40cfd10 tx=0x7fc9ac01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afc63dc1c0eae192 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:55.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b1d9b640 1 -- 172.21.15.46:0/1519413499 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac0af400 con 0x7fc9b40cf160 2023-10-30T14:14:55.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b1d9b640 1 -- 172.21.15.46:0/1519413499 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ac0ab050 con 0x7fc9b40cf160 2023-10-30T14:14:55.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.945+0000 7fc9b1d9b640 1 -- 172.21.15.46:0/1519413499 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac0af780 con 0x7fc9b40cf160 2023-10-30T14:14:55.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 -- 172.21.15.46:0/1519413499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b40cf160 msgr2=0x7fc9b40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:55.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 --2- 172.21.15.46:0/1519413499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b40cf160 0x7fc9b40cf590 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40cfd10 tx=0x7fc9ac01f1c0 comp rx=0 tx=0).stop 2023-10-30T14:14:55.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 -- 172.21.15.46:0/1519413499 shutdown_connections 2023-10-30T14:14:55.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 --2- 172.21.15.46:0/1519413499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b40cf160 0x7fc9b40cf590 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:55.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 --2- 172.21.15.46:0/1519413499 >> v2:172.21.15.154:3300/0 conn(0x7fc9b41538a0 0x7fc9b415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:55.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 -- 172.21.15.46:0/1519413499 >> 172.21.15.46:0/1519413499 conn(0x7fc9b406bfe0 msgr2=0x7fc9b406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:55.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 -- 172.21.15.46:0/1519413499 shutdown_connections 2023-10-30T14:14:55.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 -- 172.21.15.46:0/1519413499 wait complete. 2023-10-30T14:14:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 Processor -- start 2023-10-30T14:14:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 -- start start 2023-10-30T14:14:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9b40ca0e0 0x7fc9b41e33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b40cf160 0x7fc9b41e3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b41538a0 0x7fc9b40ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:55.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b41e4750 con 0x7fc9b40cf160 2023-10-30T14:14:55.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b41f3290 con 0x7fc9b41538a0 2023-10-30T14:14:55.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b9194640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b41e5020 con 0x7fc9b40ca0e0 2023-10-30T14:14:55.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b40cf160 0x7fc9b41e3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:55.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b40cf160 0x7fc9b41e3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46540/0 (socket says 172.21.15.46:46540) 2023-10-30T14:14:55.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b259c640 1 -- 172.21.15.46:0/269075332 learned_addr learned my addr 172.21.15.46:0/269075332 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:55.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b359e640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b41538a0 0x7fc9b40ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:55.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.949+0000 7fc9b2d9d640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9b40ca0e0 0x7fc9b41e33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:55.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc9b259c640 1 -- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9b40ca0e0 msgr2=0x7fc9b41e33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:55.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc9b259c640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9b40ca0e0 0x7fc9b41e33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:55.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc9b259c640 1 -- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b41538a0 msgr2=0x7fc9b40ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:55.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc9b259c640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b41538a0 0x7fc9b40ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:55.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc9b259c640 1 -- 172.21.15.46:0/269075332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac009b30 con 0x7fc9b40cf160 2023-10-30T14:14:55.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc9b259c640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b40cf160 0x7fc9b41e3930 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fc9a800c4b0 tx=0x7fc9a800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:55.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc99b7fe640 1 -- 172.21.15.46:0/269075332 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a80074e0 con 0x7fc9b40cf160 2023-10-30T14:14:55.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc99b7fe640 1 -- 172.21.15.46:0/269075332 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a8007660 con 0x7fc9b40cf160 2023-10-30T14:14:55.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc99b7fe640 1 -- 172.21.15.46:0/269075332 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8004a30 con 0x7fc9b40cf160 2023-10-30T14:14:55.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc9b9194640 1 -- 172.21.15.46:0/269075332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b41e5a20 con 0x7fc9b40cf160 2023-10-30T14:14:55.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc9b9194640 1 -- 172.21.15.46:0/269075332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b41e5740 con 0x7fc9b40cf160 2023-10-30T14:14:55.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc9b9194640 1 -- 172.21.15.46:0/269075332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b40b2a50 con 0x7fc9b40cf160 2023-10-30T14:14:55.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc99b7fe640 1 -- 172.21.15.46:0/269075332 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc9a8004b90 con 0x7fc9b40cf160 2023-10-30T14:14:55.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc99b7fe640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9a00664d0 0x7fc9a0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:55.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.953+0000 7fc99b7fe640 1 -- 172.21.15.46:0/269075332 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7fc9a8083910 con 0x7fc9b40cf160 2023-10-30T14:14:55.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.957+0000 7fc9b2d9d640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9a00664d0 0x7fc9a0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:55.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.957+0000 7fc9b2d9d640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9a00664d0 0x7fc9a0068990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4007a60 tx=0x7fc9a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:55.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:55.957+0000 7fc99b7fe640 1 -- 172.21.15.46:0/269075332 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc9a8050390 con 0x7fc9b40cf160 2023-10-30T14:14:56.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.149+0000 7fc9b9194640 1 -- 172.21.15.46:0/269075332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9b4159870 con 0x7fc9b40cf160 2023-10-30T14:14:56.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc99b7fe640 1 -- 172.21.15.46:0/269075332 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fc9b4159870 con 0x7fc9b40cf160 2023-10-30T14:14:56.155 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:14:56.156 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":218,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:14:54.671027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:56.156 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 218 2023-10-30T14:14:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc9b9194640 1 -- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9a00664d0 msgr2=0x7fc9a0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc9b9194640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9a00664d0 0x7fc9a0068990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4007a60 tx=0x7fc9a4008040 comp rx=0 tx=0).stop 2023-10-30T14:14:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc9b9194640 1 -- 172.21.15.46:0/269075332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b40cf160 msgr2=0x7fc9b41e3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc9b9194640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b40cf160 0x7fc9b41e3930 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fc9a800c4b0 tx=0x7fc9a800c970 comp rx=0 tx=0).stop 2023-10-30T14:14:56.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc9b9194640 1 -- 172.21.15.46:0/269075332 shutdown_connections 2023-10-30T14:14:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc9b9194640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc9a00664d0 0x7fc9a0068990 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc9b9194640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b41538a0 0x7fc9b40ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc9b9194640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9b40cf160 0x7fc9b41e3930 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc9b9194640 1 --2- 172.21.15.46:0/269075332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9b40ca0e0 0x7fc9b41e33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc9b9194640 1 -- 172.21.15.46:0/269075332 >> 172.21.15.46:0/269075332 conn(0x7fc9b406bfe0 msgr2=0x7fc9b4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc9b9194640 1 -- 172.21.15.46:0/269075332 shutdown_connections 2023-10-30T14:14:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.153+0000 7fc9b9194640 1 -- 172.21.15.46:0/269075332 wait complete. 2023-10-30T14:14:56.177 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T14:14:56.177 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:14:56.178 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1f731d640 1 Processor -- start 2023-10-30T14:14:56.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1f731d640 1 -- start start 2023-10-30T14:14:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1f731d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0071110 0x7fd1f006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1f731d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f0066cf0 con 0x7fd1f00c6bd0 2023-10-30T14:14:56.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1f731d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f00c70d0 con 0x7fd1f006f7a0 2023-10-30T14:14:56.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1f731d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f00700c0 con 0x7fd1f0071110 2023-10-30T14:14:56.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1f48b8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd1f00c6bd0 0x7fd1f0158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49064/0 (socket says 172.21.15.46:49064) 2023-10-30T14:14:56.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1f48b8640 1 -- 172.21.15.46:0/2952581604 learned_addr learned my addr 172.21.15.46:0/2952581604 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:14:56.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1f58ba640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0071110 0x7fd1f006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281187729 0 0) 0x7fd1f0066cf0 con 0x7fd1f00c6bd0 2023-10-30T14:14:56.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1dc003530 con 0x7fd1f00c6bd0 2023-10-30T14:14:56.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1310765211 0 0) 0x7fd1dc003530 con 0x7fd1f00c6bd0 2023-10-30T14:14:56.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0071110 msgr2=0x7fd1f006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:56.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1e7fff640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0071110 0x7fd1f006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 >> v1:172.21.15.154:6789/0 conn(0x7fd1f006f7a0 legacy=0x7fd1f0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.317+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f00c7390 con 0x7fd1f00c6bd0 2023-10-30T14:14:56.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fd1e0001680 con 0x7fd1f00c6bd0 2023-10-30T14:14:56.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 >> v1:172.21.15.46:6789/0 conn(0x7fd1f00c6bd0 legacy=0x7fd1f0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1e7fff640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc0039b0 0x7fd1dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1e7fff640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0071110 0x7fd1dc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1e7fff640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc004d10 0x7fd1dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f00700c0 con 0x7fd1dc004d10 2023-10-30T14:14:56.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc00bad0 con 0x7fd1dc0039b0 2023-10-30T14:14:56.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc00bc90 con 0x7fd1f0071110 2023-10-30T14:14:56.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1f58ba640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc004d10 0x7fd1dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1f50b9640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc0039b0 0x7fd1dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1f48b8640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0071110 0x7fd1dc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1f58ba640 1 -- 172.21.15.46:0/2952581604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0071110 msgr2=0x7fd1dc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:56.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1f58ba640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0071110 0x7fd1dc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1f58ba640 1 -- 172.21.15.46:0/2952581604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc0039b0 msgr2=0x7fd1dc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:56.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1f58ba640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc0039b0 0x7fd1dc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1f58ba640 1 -- 172.21.15.46:0/2952581604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc003530 con 0x7fd1dc004d10 2023-10-30T14:14:56.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1f50b9640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1dc0039b0 0x7fd1dc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:14:56.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1f58ba640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc004d10 0x7fd1dc005190 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8009f90 tx=0x7fd1e801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4232c6281e150a10 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:56.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e80a5c30 con 0x7fd1dc004d10 2023-10-30T14:14:56.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e800f040 con 0x7fd1dc004d10 2023-10-30T14:14:56.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.321+0000 7fd1e7fff640 1 -- 172.21.15.46:0/2952581604 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e80ad440 con 0x7fd1dc004d10 2023-10-30T14:14:56.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 -- 172.21.15.46:0/2952581604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc004d10 msgr2=0x7fd1dc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 --2- 172.21.15.46:0/2952581604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc004d10 0x7fd1dc005190 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8009f90 tx=0x7fd1e801cc20 comp rx=0 tx=0).stop 2023-10-30T14:14:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f58ba640 1 -- 172.21.15.46:0/2952581604 reap_dead start 2023-10-30T14:14:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 -- 172.21.15.46:0/2952581604 shutdown_connections 2023-10-30T14:14:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 -- 172.21.15.46:0/2952581604 >> 172.21.15.46:0/2952581604 conn(0x7fd1f00c7910 msgr2=0x7fd1f00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 -- 172.21.15.46:0/2952581604 shutdown_connections 2023-10-30T14:14:56.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 -- 172.21.15.46:0/2952581604 wait complete. 2023-10-30T14:14:56.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 Processor -- start 2023-10-30T14:14:56.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 -- start start 2023-10-30T14:14:56.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc004d10 0x7fd1f0154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1f01539c0 0x7fd1f01ee390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0158040 0x7fd1f00af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f01ee8d0 con 0x7fd1dc004d10 2023-10-30T14:14:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f01548b0 con 0x7fd1f01539c0 2023-10-30T14:14:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f731d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f0153e20 con 0x7fd1f0158040 2023-10-30T14:14:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f50b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc004d10 0x7fd1f0154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f50b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc004d10 0x7fd1f0154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46560/0 (socket says 172.21.15.46:46560) 2023-10-30T14:14:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f50b9640 1 -- 172.21.15.46:0/327932417 learned_addr learned my addr 172.21.15.46:0/327932417 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f48b8640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1f01539c0 0x7fd1f01ee390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.325+0000 7fd1f58ba640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0158040 0x7fd1f00af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1f50b9640 1 -- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0158040 msgr2=0x7fd1f00af120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1f50b9640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0158040 0x7fd1f00af120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1f50b9640 1 -- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1f01539c0 msgr2=0x7fd1f01ee390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1f50b9640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1f01539c0 0x7fd1f01ee390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1f50b9640 1 -- 172.21.15.46:0/327932417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f0152570 con 0x7fd1dc004d10 2023-10-30T14:14:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1f50b9640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc004d10 0x7fd1f0154d40 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00e4b0 tx=0x7fd1ec00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1e5ffb640 1 -- 172.21.15.46:0/327932417 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec0053a0 con 0x7fd1dc004d10 2023-10-30T14:14:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1e5ffb640 1 -- 172.21.15.46:0/327932417 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1ec003600 con 0x7fd1dc004d10 2023-10-30T14:14:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1f731d640 1 -- 172.21.15.46:0/327932417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f0152850 con 0x7fd1dc004d10 2023-10-30T14:14:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1e5ffb640 1 -- 172.21.15.46:0/327932417 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec003990 con 0x7fd1dc004d10 2023-10-30T14:14:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1f731d640 1 -- 172.21.15.46:0/327932417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f0152e10 con 0x7fd1dc004d10 2023-10-30T14:14:56.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1f731d640 1 -- 172.21.15.46:0/327932417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f004e170 con 0x7fd1dc004d10 2023-10-30T14:14:56.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1e5ffb640 1 -- 172.21.15.46:0/327932417 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd1ec003af0 con 0x7fd1dc004d10 2023-10-30T14:14:56.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1e5ffb640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1d40664a0 0x7fd1d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:56.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1e5ffb640 1 -- 172.21.15.46:0/327932417 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7fd1ec0845e0 con 0x7fd1dc004d10 2023-10-30T14:14:56.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.329+0000 7fd1f48b8640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1d40664a0 0x7fd1d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:56.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.333+0000 7fd1f48b8640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1d40664a0 0x7fd1d4068960 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0156ee0 tx=0x7fd1e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:56.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.333+0000 7fd1e5ffb640 1 -- 172.21.15.46:0/327932417 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd1ec051040 con 0x7fd1dc004d10 2023-10-30T14:14:56.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.525+0000 7fd1f731d640 1 -- 172.21.15.46:0/327932417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1f0066510 con 0x7fd1dc004d10 2023-10-30T14:14:56.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.525+0000 7fd1e5ffb640 1 -- 172.21.15.46:0/327932417 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fd1ec054660 con 0x7fd1dc004d10 2023-10-30T14:14:56.530 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:14:56.530 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":218,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:14:54.671027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:56.530 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 218 2023-10-30T14:14:56.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.529+0000 7fd1f731d640 1 -- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1d40664a0 msgr2=0x7fd1d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.529+0000 7fd1f731d640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1d40664a0 0x7fd1d4068960 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0156ee0 tx=0x7fd1e0009040 comp rx=0 tx=0).stop 2023-10-30T14:14:56.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.529+0000 7fd1f731d640 1 -- 172.21.15.46:0/327932417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc004d10 msgr2=0x7fd1f0154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:56.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.529+0000 7fd1f731d640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc004d10 0x7fd1f0154d40 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00e4b0 tx=0x7fd1ec00e970 comp rx=0 tx=0).stop 2023-10-30T14:14:56.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.529+0000 7fd1f731d640 1 -- 172.21.15.46:0/327932417 shutdown_connections 2023-10-30T14:14:56.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.529+0000 7fd1f731d640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1d40664a0 0x7fd1d4068960 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.529+0000 7fd1f731d640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1f0158040 0x7fd1f00af120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.529+0000 7fd1f731d640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1f01539c0 0x7fd1f01ee390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.529+0000 7fd1f731d640 1 --2- 172.21.15.46:0/327932417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1dc004d10 0x7fd1f0154d40 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:56.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.529+0000 7fd1f731d640 1 -- 172.21.15.46:0/327932417 >> 172.21.15.46:0/327932417 conn(0x7fd1f00c7910 msgr2=0x7fd1f006a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:56.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.529+0000 7fd1f731d640 1 -- 172.21.15.46:0/327932417 shutdown_connections 2023-10-30T14:14:56.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:56.529+0000 7fd1f731d640 1 -- 172.21.15.46:0/327932417 wait complete. 2023-10-30T14:14:56.552 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 218, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 218, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8982": { "addr": "172.21.15.46:6833/2455118950", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2455118950, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2455118950, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8982, "incarnation": 209, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 37 }, "gid_9135": { "addr": "172.21.15.46:6837/681817982", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 681817982, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 681817982, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9135, "incarnation": 218, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:14:54.671027+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9135, "mds_1": 8982 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:14:58.167 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:14:58.171 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:14:58.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:14:58.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda445d3640 1 Processor -- start 2023-10-30T14:14:58.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda445d3640 1 -- start start 2023-10-30T14:14:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda445d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fda3c0c6b80 0x7fda3c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda445d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda3c0c6f60 0x7fda3c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:58.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda445d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c0c0a50 con 0x7fda3c0c8430 2023-10-30T14:14:58.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda445d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3c06f6e0 con 0x7fda3c0c6b80 2023-10-30T14:14:58.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda445d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c06f9c0 con 0x7fda3c0c6f60 2023-10-30T14:14:58.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda4236f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fda3c0c6b80 0x7fda3c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:58.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda41b6e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda3c0c6f60 0x7fda3c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:58.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda42b70640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fda3c0c8430 0x7fda3c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:52020/0 (socket says 172.21.15.46:52020) 2023-10-30T14:14:58.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda42b70640 1 -- 172.21.15.46:0/3878887548 learned_addr learned my addr 172.21.15.46:0/3878887548 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:14:58.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda4136d640 1 -- 172.21.15.46:0/3878887548 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390427254 0 0) 0x7fda3c0c0a50 con 0x7fda3c0c8430 2023-10-30T14:14:58.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda4136d640 1 -- 172.21.15.46:0/3878887548 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda28005740 con 0x7fda3c0c8430 2023-10-30T14:14:58.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda4236f640 1 -- 172.21.15.46:0/3878887548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda3c0c6f60 msgr2=0x7fda3c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:14:58.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda4236f640 1 --2- 172.21.15.46:0/3878887548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda3c0c6f60 0x7fda3c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda4236f640 1 -- 172.21.15.46:0/3878887548 >> v1:172.21.15.154:6789/0 conn(0x7fda3c0c8430 legacy=0x7fda3c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:58.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda4236f640 1 -- 172.21.15.46:0/3878887548 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c06fca0 con 0x7fda3c0c6b80 2023-10-30T14:14:58.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.705+0000 7fda41b6e640 1 --2- 172.21.15.46:0/3878887548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda3c0c6f60 0x7fda3c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:14:58.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.709+0000 7fda4236f640 1 --2- 172.21.15.46:0/3878887548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0c6b80 0x7fda3c0bfec0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fda380097c0 tx=0x7fda3801c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5909d397441f1239 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:58.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.709+0000 7fda4136d640 1 -- 172.21.15.46:0/3878887548 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda380ab600 con 0x7fda3c0c6b80 2023-10-30T14:14:58.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.709+0000 7fda4136d640 1 -- 172.21.15.46:0/3878887548 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda380a7050 con 0x7fda3c0c6b80 2023-10-30T14:14:58.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.709+0000 7fda4136d640 1 -- 172.21.15.46:0/3878887548 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda380ab980 con 0x7fda3c0c6b80 2023-10-30T14:14:58.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.709+0000 7fda445d3640 1 -- 172.21.15.46:0/3878887548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0c6b80 msgr2=0x7fda3c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:58.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.709+0000 7fda445d3640 1 --2- 172.21.15.46:0/3878887548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0c6b80 0x7fda3c0bfec0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fda380097c0 tx=0x7fda3801c3f0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.709+0000 7fda445d3640 1 -- 172.21.15.46:0/3878887548 shutdown_connections 2023-10-30T14:14:58.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.709+0000 7fda445d3640 1 --2- 172.21.15.46:0/3878887548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda3c0c6f60 0x7fda3c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.709+0000 7fda445d3640 1 --2- 172.21.15.46:0/3878887548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0c6b80 0x7fda3c0bfec0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.709+0000 7fda445d3640 1 -- 172.21.15.46:0/3878887548 >> 172.21.15.46:0/3878887548 conn(0x7fda3c0c7910 msgr2=0x7fda3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:58.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.709+0000 7fda445d3640 1 -- 172.21.15.46:0/3878887548 shutdown_connections 2023-10-30T14:14:58.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.709+0000 7fda445d3640 1 -- 172.21.15.46:0/3878887548 wait complete. 2023-10-30T14:14:58.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda445d3640 1 Processor -- start 2023-10-30T14:14:58.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda445d3640 1 -- start start 2023-10-30T14:14:58.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda445d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c0c6b80 0x7fda3c0c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:58.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda445d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0c6f60 0x7fda3c1f57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:58.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda445d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda3c0c8430 0x7fda3c1589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:58.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda445d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1f53b0 con 0x7fda3c0c6b80 2023-10-30T14:14:58.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda445d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c13aac0 con 0x7fda3c0c6f60 2023-10-30T14:14:58.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda445d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1f5060 con 0x7fda3c0c8430 2023-10-30T14:14:58.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda4236f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c0c6b80 0x7fda3c0c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:58.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda4236f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c0c6b80 0x7fda3c0c9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42616/0 (socket says 172.21.15.46:42616) 2023-10-30T14:14:58.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda4236f640 1 -- 172.21.15.46:0/4050944228 learned_addr learned my addr 172.21.15.46:0/4050944228 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:14:58.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda42b70640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda3c0c8430 0x7fda3c1589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:58.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda41b6e640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0c6f60 0x7fda3c1f57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:58.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda4236f640 1 -- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda3c0c8430 msgr2=0x7fda3c1589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:58.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda4236f640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda3c0c8430 0x7fda3c1589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda4236f640 1 -- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0c6f60 msgr2=0x7fda3c1f57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda4236f640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0c6f60 0x7fda3c1f57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda4236f640 1 -- 172.21.15.46:0/4050944228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c009420 con 0x7fda3c0c6b80 2023-10-30T14:14:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.713+0000 7fda4236f640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c0c6b80 0x7fda3c0c9ff0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fda380097c0 tx=0x7fda3801ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.717+0000 7fda32ffd640 1 -- 172.21.15.46:0/4050944228 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda380b7650 con 0x7fda3c0c6b80 2023-10-30T14:14:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.717+0000 7fda32ffd640 1 -- 172.21.15.46:0/4050944228 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda380b77d0 con 0x7fda3c0c6b80 2023-10-30T14:14:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.717+0000 7fda32ffd640 1 -- 172.21.15.46:0/4050944228 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda380b7b80 con 0x7fda3c0c6b80 2023-10-30T14:14:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.717+0000 7fda445d3640 1 -- 172.21.15.46:0/4050944228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda38009420 con 0x7fda3c0c6b80 2023-10-30T14:14:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.717+0000 7fda445d3640 1 -- 172.21.15.46:0/4050944228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c1f1fd0 con 0x7fda3c0c6b80 2023-10-30T14:14:58.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.717+0000 7fda445d3640 1 -- 172.21.15.46:0/4050944228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c04e170 con 0x7fda3c0c6b80 2023-10-30T14:14:58.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.717+0000 7fda32ffd640 1 -- 172.21.15.46:0/4050944228 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fda380b7ce0 con 0x7fda3c0c6b80 2023-10-30T14:14:58.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.717+0000 7fda32ffd640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda200664d0 0x7fda20068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:14:58.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.717+0000 7fda32ffd640 1 -- 172.21.15.46:0/4050944228 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7fda3811f070 con 0x7fda3c0c6b80 2023-10-30T14:14:58.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.717+0000 7fda41b6e640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda200664d0 0x7fda20068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:14:58.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.721+0000 7fda41b6e640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda200664d0 0x7fda20068990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fda3c159ac0 tx=0x7fda2c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:14:58.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.721+0000 7fda32ffd640 1 -- 172.21.15.46:0/4050944228 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fda380ebb80 con 0x7fda3c0c6b80 2023-10-30T14:14:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 -- 172.21.15.46:0/4050944228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda3c13e790 con 0x7fda3c0c6b80 2023-10-30T14:14:58.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda32ffd640 1 -- 172.21.15.46:0/4050944228 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fda380bb090 con 0x7fda3c0c6b80 2023-10-30T14:14:58.916 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:14:58.916 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":218,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:14:54.671027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:14:58.917 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 218 2023-10-30T14:14:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 -- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda200664d0 msgr2=0x7fda20068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda200664d0 0x7fda20068990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fda3c159ac0 tx=0x7fda2c009190 comp rx=0 tx=0).stop 2023-10-30T14:14:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 -- 172.21.15.46:0/4050944228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c0c6b80 msgr2=0x7fda3c0c9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:14:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c0c6b80 0x7fda3c0c9ff0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fda380097c0 tx=0x7fda3801ac30 comp rx=0 tx=0).stop 2023-10-30T14:14:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 -- 172.21.15.46:0/4050944228 shutdown_connections 2023-10-30T14:14:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fda200664d0 0x7fda20068990 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda3c0c8430 0x7fda3c1589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0c6f60 0x7fda3c1f57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 --2- 172.21.15.46:0/4050944228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c0c6b80 0x7fda3c0c9ff0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:14:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 -- 172.21.15.46:0/4050944228 >> 172.21.15.46:0/4050944228 conn(0x7fda3c0c7910 msgr2=0x7fda3c0c5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:14:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 -- 172.21.15.46:0/4050944228 shutdown_connections 2023-10-30T14:14:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:14:58.913+0000 7fda445d3640 1 -- 172.21.15.46:0/4050944228 wait complete. 2023-10-30T14:14:58.938 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:00.939 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:01.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f3470646640 1 Processor -- start 2023-10-30T14:15:01.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f3470646640 1 -- start start 2023-10-30T14:15:01.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f3470646640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f346815d6f0 0x7f346815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f3470646640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34680cb800 con 0x7f34680cf160 2023-10-30T14:15:01.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f3470646640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34680cb330 con 0x7f34681538a0 2023-10-30T14:15:01.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f3470646640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34680cad70 con 0x7f346815d6f0 2023-10-30T14:15:01.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f346ebe3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f34680cf160 0x7f3468164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56276/0 (socket says 172.21.15.46:56276) 2023-10-30T14:15:01.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f346ebe3640 1 -- 172.21.15.46:0/1105626302 learned_addr learned my addr 172.21.15.46:0/1105626302 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:01.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f346dbe1640 1 --2- 172.21.15.46:0/1105626302 >> v2:172.21.15.154:3300/0 conn(0x7f346815d6f0 0x7f346815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f346d3e0640 1 -- 172.21.15.46:0/1105626302 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479468726 0 0) 0x7f34680cb800 con 0x7f34680cf160 2023-10-30T14:15:01.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f346d3e0640 1 -- 172.21.15.46:0/1105626302 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3454003530 con 0x7f34680cf160 2023-10-30T14:15:01.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f346dbe1640 1 -- 172.21.15.46:0/1105626302 >> v1:172.21.15.154:6789/0 conn(0x7f34681538a0 legacy=0x7f34680ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f346dbe1640 1 -- 172.21.15.46:0/1105626302 >> v1:172.21.15.46:6789/0 conn(0x7f34680cf160 legacy=0x7f3468164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.085+0000 7f346dbe1640 1 -- 172.21.15.46:0/1105626302 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34680cb050 con 0x7f346815d6f0 2023-10-30T14:15:01.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.089+0000 7f346dbe1640 1 --2- 172.21.15.46:0/1105626302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f346815d6f0 0x7f346815a090 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f34580097c0 tx=0x7f345801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=457bb5f9e3e2947d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:01.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.089+0000 7f346d3e0640 1 -- 172.21.15.46:0/1105626302 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34580a8970 con 0x7f346815d6f0 2023-10-30T14:15:01.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.089+0000 7f346d3e0640 1 -- 172.21.15.46:0/1105626302 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34580a8ad0 con 0x7f346815d6f0 2023-10-30T14:15:01.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.089+0000 7f346d3e0640 1 -- 172.21.15.46:0/1105626302 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34580a8e40 con 0x7f346815d6f0 2023-10-30T14:15:01.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.089+0000 7f3470646640 1 -- 172.21.15.46:0/1105626302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f346815d6f0 msgr2=0x7f346815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.089+0000 7f3470646640 1 --2- 172.21.15.46:0/1105626302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f346815d6f0 0x7f346815a090 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f34580097c0 tx=0x7f345801ec70 comp rx=0 tx=0).stop 2023-10-30T14:15:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.089+0000 7f3470646640 1 -- 172.21.15.46:0/1105626302 shutdown_connections 2023-10-30T14:15:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.089+0000 7f3470646640 1 --2- 172.21.15.46:0/1105626302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f346815d6f0 0x7f346815a090 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.089+0000 7f3470646640 1 -- 172.21.15.46:0/1105626302 >> 172.21.15.46:0/1105626302 conn(0x7f346806bfe0 msgr2=0x7f346806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.089+0000 7f3470646640 1 -- 172.21.15.46:0/1105626302 shutdown_connections 2023-10-30T14:15:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.089+0000 7f3470646640 1 -- 172.21.15.46:0/1105626302 wait complete. 2023-10-30T14:15:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f3470646640 1 Processor -- start 2023-10-30T14:15:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f3470646640 1 -- start start 2023-10-30T14:15:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f3470646640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680cf160 0x7f34681e2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f3470646640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34681538a0 0x7f34681e60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f3470646640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f346815d6f0 0x7f34681e3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f3470646640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34681e3500 con 0x7f34680cf160 2023-10-30T14:15:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f3470646640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34681f3120 con 0x7f346815d6f0 2023-10-30T14:15:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f3470646640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34680c6c00 con 0x7f34681538a0 2023-10-30T14:15:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f346e3e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680cf160 0x7f34681e2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f346e3e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680cf160 0x7f34681e2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42620/0 (socket says 172.21.15.46:42620) 2023-10-30T14:15:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f346e3e2640 1 -- 172.21.15.46:0/4044135079 learned_addr learned my addr 172.21.15.46:0/4044135079 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f346ebe3640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f346815d6f0 0x7f34681e3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f346dbe1640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34681538a0 0x7f34681e60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f346e3e2640 1 -- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34681538a0 msgr2=0x7f34681e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f346e3e2640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34681538a0 0x7f34681e60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f346e3e2640 1 -- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f346815d6f0 msgr2=0x7f34681e3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f346e3e2640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f346815d6f0 0x7f34681e3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f346e3e2640 1 -- 172.21.15.46:0/4044135079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3458009420 con 0x7f34680cf160 2023-10-30T14:15:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f346e3e2640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680cf160 0x7f34681e2e70 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f346400c530 tx=0x7f346400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f345effd640 1 -- 172.21.15.46:0/4044135079 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464005220 con 0x7f34680cf160 2023-10-30T14:15:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f345effd640 1 -- 172.21.15.46:0/4044135079 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3464005380 con 0x7f34680cf160 2023-10-30T14:15:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f345effd640 1 -- 172.21.15.46:0/4044135079 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464007270 con 0x7f34680cf160 2023-10-30T14:15:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f3470646640 1 -- 172.21.15.46:0/4044135079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34681f2550 con 0x7f34680cf160 2023-10-30T14:15:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.093+0000 7f3470646640 1 -- 172.21.15.46:0/4044135079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34681f2ad0 con 0x7f34680cf160 2023-10-30T14:15:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.097+0000 7f3470646640 1 -- 172.21.15.46:0/4044135079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346804e170 con 0x7f34680cf160 2023-10-30T14:15:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.097+0000 7f345effd640 1 -- 172.21.15.46:0/4044135079 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34640073d0 con 0x7f34680cf160 2023-10-30T14:15:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.097+0000 7f345effd640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f344c0664d0 0x7f344c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.097+0000 7f345effd640 1 -- 172.21.15.46:0/4044135079 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f34640840c0 con 0x7f34680cf160 2023-10-30T14:15:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.097+0000 7f346dbe1640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f344c0664d0 0x7f344c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:01.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.097+0000 7f346dbe1640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f344c0664d0 0x7f344c068990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f34580b2000 tx=0x7f34580aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:01.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.101+0000 7f345effd640 1 -- 172.21.15.46:0/4044135079 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3464050c30 con 0x7f34680cf160 2023-10-30T14:15:01.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.353+0000 7f3470646640 1 -- 172.21.15.46:0/4044135079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34680ca9b0 con 0x7f34680cf160 2023-10-30T14:15:01.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f345effd640 1 -- 172.21.15.46:0/4044135079 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f3464054250 con 0x7f34680cf160 2023-10-30T14:15:01.360 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:01.361 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":218,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:14:54.671027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:01.361 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 218 2023-10-30T14:15:01.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f3470646640 1 -- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f344c0664d0 msgr2=0x7f344c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f3470646640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f344c0664d0 0x7f344c068990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f34580b2000 tx=0x7f34580aa070 comp rx=0 tx=0).stop 2023-10-30T14:15:01.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f3470646640 1 -- 172.21.15.46:0/4044135079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680cf160 msgr2=0x7f34681e2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f3470646640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680cf160 0x7f34681e2e70 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f346400c530 tx=0x7f346400c9f0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f3470646640 1 -- 172.21.15.46:0/4044135079 shutdown_connections 2023-10-30T14:15:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f3470646640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f344c0664d0 0x7f344c068990 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f3470646640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f346815d6f0 0x7f34681e3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f3470646640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f34681538a0 0x7f34681e60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f3470646640 1 --2- 172.21.15.46:0/4044135079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680cf160 0x7f34681e2e70 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:01.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f3470646640 1 -- 172.21.15.46:0/4044135079 >> 172.21.15.46:0/4044135079 conn(0x7f346806bfe0 msgr2=0x7f34680c5070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:01.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f3470646640 1 -- 172.21.15.46:0/4044135079 shutdown_connections 2023-10-30T14:15:01.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:01.357+0000 7f3470646640 1 -- 172.21.15.46:0/4044135079 wait complete. 2023-10-30T14:15:01.383 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:03.384 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:03.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.533+0000 7f29b553e640 1 Processor -- start 2023-10-30T14:15:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.533+0000 7f29b553e640 1 -- start start 2023-10-30T14:15:03.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.533+0000 7f29b553e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29b0153d60 0x7f29b00cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:03.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.533+0000 7f29b553e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b00ccb10 con 0x7f29b006d180 2023-10-30T14:15:03.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.533+0000 7f29b553e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b006ccf0 con 0x7f29b00ccd30 2023-10-30T14:15:03.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.533+0000 7f29b553e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b015a540 con 0x7f29b0153d60 2023-10-30T14:15:03.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.533+0000 7f29affff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f29b006d180 0x7f29b01642a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56280/0 (socket says 172.21.15.46:56280) 2023-10-30T14:15:03.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.533+0000 7f29affff640 1 -- 172.21.15.46:0/749400673 learned_addr learned my addr 172.21.15.46:0/749400673 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:03.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.533+0000 7f29aeffd640 1 --2- 172.21.15.46:0/749400673 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29b0153d60 0x7f29b00cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:03.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.533+0000 7f29adffb640 1 -- 172.21.15.46:0/749400673 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331657231 0 0) 0x7f29a80014a0 con 0x7f29b006d180 2023-10-30T14:15:03.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.537+0000 7f29adffb640 1 -- 172.21.15.46:0/749400673 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2998003530 con 0x7f29b006d180 2023-10-30T14:15:03.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.537+0000 7f29aeffd640 1 -- 172.21.15.46:0/749400673 >> v1:172.21.15.154:6789/0 conn(0x7f29b00ccd30 legacy=0x7f29b00cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:03.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.537+0000 7f29aeffd640 1 -- 172.21.15.46:0/749400673 >> v1:172.21.15.46:6789/0 conn(0x7f29b006d180 legacy=0x7f29b01642a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:03.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.537+0000 7f29aeffd640 1 -- 172.21.15.46:0/749400673 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b00cb4c0 con 0x7f29b0153d60 2023-10-30T14:15:03.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.537+0000 7f29aeffd640 1 --2- 172.21.15.46:0/749400673 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29b0153d60 0x7f29b00cc5d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f29a00097f0 tx=0x7f29a001ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e43197e5572e3d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:03.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.537+0000 7f29adffb640 1 -- 172.21.15.46:0/749400673 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a00a89b0 con 0x7f29b0153d60 2023-10-30T14:15:03.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.537+0000 7f29adffb640 1 -- 172.21.15.46:0/749400673 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a00a8b10 con 0x7f29b0153d60 2023-10-30T14:15:03.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.537+0000 7f29adffb640 1 -- 172.21.15.46:0/749400673 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a00a8e80 con 0x7f29b0153d60 2023-10-30T14:15:03.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.541+0000 7f29b553e640 1 -- 172.21.15.46:0/749400673 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29b0153d60 msgr2=0x7f29b00cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.541+0000 7f29b553e640 1 --2- 172.21.15.46:0/749400673 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29b0153d60 0x7f29b00cc5d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f29a00097f0 tx=0x7f29a001ecb0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.541+0000 7f29b553e640 1 -- 172.21.15.46:0/749400673 shutdown_connections 2023-10-30T14:15:03.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.541+0000 7f29b553e640 1 --2- 172.21.15.46:0/749400673 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29b0153d60 0x7f29b00cc5d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.541+0000 7f29b553e640 1 -- 172.21.15.46:0/749400673 >> 172.21.15.46:0/749400673 conn(0x7f29b006e840 msgr2=0x7f29b006ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:03.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.541+0000 7f29b553e640 1 -- 172.21.15.46:0/749400673 shutdown_connections 2023-10-30T14:15:03.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.541+0000 7f29b553e640 1 -- 172.21.15.46:0/749400673 wait complete. 2023-10-30T14:15:03.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.541+0000 7f29b553e640 1 Processor -- start 2023-10-30T14:15:03.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.541+0000 7f29b553e640 1 -- start start 2023-10-30T14:15:03.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29b553e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29b006d180 0x7f29b014a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:03.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29b553e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b00ccd30 0x7f29b014a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:03.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29b553e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29b0153d60 0x7f29b0146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:03.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29b553e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b0148620 con 0x7f29b0153d60 2023-10-30T14:15:03.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29b553e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b0148ef0 con 0x7f29b00ccd30 2023-10-30T14:15:03.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29b553e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b0149f80 con 0x7f29b006d180 2023-10-30T14:15:03.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29affff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29b0153d60 0x7f29b0146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:03.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29affff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29b0153d60 0x7f29b0146d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42628/0 (socket says 172.21.15.46:42628) 2023-10-30T14:15:03.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29affff640 1 -- 172.21.15.46:0/375577372 learned_addr learned my addr 172.21.15.46:0/375577372 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:03.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29aeffd640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29b006d180 0x7f29b014a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:03.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29ae7fc640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b00ccd30 0x7f29b014a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:03.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29affff640 1 -- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29b006d180 msgr2=0x7f29b014a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:03.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29affff640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29b006d180 0x7f29b014a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29affff640 1 -- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b00ccd30 msgr2=0x7f29b014a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:03.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29affff640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b00ccd30 0x7f29b014a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29affff640 1 -- 172.21.15.46:0/375577372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a0009420 con 0x7f29b0153d60 2023-10-30T14:15:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29ae7fc640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b00ccd30 0x7f29b014a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29aeffd640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29b006d180 0x7f29b014a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:03.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29affff640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29b0153d60 0x7f29b0146d60 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f29a800cb40 tx=0x7f29a800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29977fe640 1 -- 172.21.15.46:0/375577372 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a800aca0 con 0x7f29b0153d60 2023-10-30T14:15:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29977fe640 1 -- 172.21.15.46:0/375577372 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a800ae00 con 0x7f29b0153d60 2023-10-30T14:15:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29977fe640 1 -- 172.21.15.46:0/375577372 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8012620 con 0x7f29b0153d60 2023-10-30T14:15:03.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29b553e640 1 -- 172.21.15.46:0/375577372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b0149370 con 0x7f29b0153d60 2023-10-30T14:15:03.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29b553e640 1 -- 172.21.15.46:0/375577372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b0149930 con 0x7f29b0153d60 2023-10-30T14:15:03.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.545+0000 7f29b553e640 1 -- 172.21.15.46:0/375577372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b014a110 con 0x7f29b0153d60 2023-10-30T14:15:03.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.549+0000 7f29977fe640 1 -- 172.21.15.46:0/375577372 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29a8007070 con 0x7f29b0153d60 2023-10-30T14:15:03.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.549+0000 7f29977fe640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f299c0664a0 0x7f299c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:03.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.549+0000 7f29977fe640 1 -- 172.21.15.46:0/375577372 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f29a8084770 con 0x7f29b0153d60 2023-10-30T14:15:03.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.549+0000 7f29aeffd640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f299c0664a0 0x7f299c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:03.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.553+0000 7f29aeffd640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f299c0664a0 0x7f299c068960 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f29a00b2000 tx=0x7f29a00aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:03.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.553+0000 7f29977fe640 1 -- 172.21.15.46:0/375577372 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29a80511d0 con 0x7f29b0153d60 2023-10-30T14:15:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.845+0000 7f29b553e640 1 -- 172.21.15.46:0/375577372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29b006ca50 con 0x7f29b0153d60 2023-10-30T14:15:03.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.845+0000 7f29977fe640 1 -- 172.21.15.46:0/375577372 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f29a8001c20 con 0x7f29b0153d60 2023-10-30T14:15:03.849 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:03.849 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":218,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:14:54.671027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:03.849 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 218 2023-10-30T14:15:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.849+0000 7f29b553e640 1 -- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f299c0664a0 msgr2=0x7f299c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.849+0000 7f29b553e640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f299c0664a0 0x7f299c068960 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f29a00b2000 tx=0x7f29a00aa070 comp rx=0 tx=0).stop 2023-10-30T14:15:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.849+0000 7f29b553e640 1 -- 172.21.15.46:0/375577372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29b0153d60 msgr2=0x7f29b0146d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.849+0000 7f29b553e640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29b0153d60 0x7f29b0146d60 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f29a800cb40 tx=0x7f29a800a060 comp rx=0 tx=0).stop 2023-10-30T14:15:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.849+0000 7f29b553e640 1 -- 172.21.15.46:0/375577372 shutdown_connections 2023-10-30T14:15:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.849+0000 7f29b553e640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f299c0664a0 0x7f299c068960 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.849+0000 7f29b553e640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29b0153d60 0x7f29b0146d60 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.849+0000 7f29b553e640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b00ccd30 0x7f29b014a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.849+0000 7f29b553e640 1 --2- 172.21.15.46:0/375577372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29b006d180 0x7f29b014a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.849+0000 7f29b553e640 1 -- 172.21.15.46:0/375577372 >> 172.21.15.46:0/375577372 conn(0x7f29b006e840 msgr2=0x7f29b006a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.849+0000 7f29b553e640 1 -- 172.21.15.46:0/375577372 shutdown_connections 2023-10-30T14:15:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:03.849+0000 7f29b553e640 1 -- 172.21.15.46:0/375577372 wait complete. 2023-10-30T14:15:03.879 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:05.880 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5074609640 1 Processor -- start 2023-10-30T14:15:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5074609640 1 -- start start 2023-10-30T14:15:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5074609640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f506c15d6f0 0x7f506c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5074609640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f506c0cf160 0x7f506c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5074609640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506c0cb800 con 0x7f506c1538a0 2023-10-30T14:15:06.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5074609640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506c0cb330 con 0x7f506c0cf160 2023-10-30T14:15:06.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5074609640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506c0cad70 con 0x7f506c15d6f0 2023-10-30T14:15:06.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5072ba6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f506c0cf160 0x7f506c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:06.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5072ba6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f506c0cf160 0x7f506c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42644/0 (socket says 172.21.15.46:42644) 2023-10-30T14:15:06.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5072ba6640 1 -- 172.21.15.46:0/3609594843 learned_addr learned my addr 172.21.15.46:0/3609594843 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:06.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5071ba4640 1 --2- 172.21.15.46:0/3609594843 >> v2:172.21.15.154:3300/0 conn(0x7f506c15d6f0 0x7f506c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:06.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5072ba6640 1 -- 172.21.15.46:0/3609594843 >> v2:172.21.15.154:3300/0 conn(0x7f506c15d6f0 msgr2=0x7f506c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:06.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5072ba6640 1 --2- 172.21.15.46:0/3609594843 >> v2:172.21.15.154:3300/0 conn(0x7f506c15d6f0 0x7f506c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5072ba6640 1 -- 172.21.15.46:0/3609594843 >> v1:172.21.15.154:6789/0 conn(0x7f506c1538a0 legacy=0x7f506c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:06.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.033+0000 7f5072ba6640 1 -- 172.21.15.46:0/3609594843 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c0cb050 con 0x7f506c0cf160 2023-10-30T14:15:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.037+0000 7f5072ba6640 1 --2- 172.21.15.46:0/3609594843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f506c0cf160 0x7f506c0cf590 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f506c0b30a0 tx=0x7f506401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66aae0409e2785b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.037+0000 7f50713a3640 1 -- 172.21.15.46:0/3609594843 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50640af400 con 0x7f506c0cf160 2023-10-30T14:15:06.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.037+0000 7f50713a3640 1 -- 172.21.15.46:0/3609594843 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50640ab050 con 0x7f506c0cf160 2023-10-30T14:15:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.037+0000 7f50713a3640 1 -- 172.21.15.46:0/3609594843 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50640af780 con 0x7f506c0cf160 2023-10-30T14:15:06.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.037+0000 7f5074609640 1 -- 172.21.15.46:0/3609594843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f506c0cf160 msgr2=0x7f506c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:06.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.037+0000 7f5074609640 1 --2- 172.21.15.46:0/3609594843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f506c0cf160 0x7f506c0cf590 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f506c0b30a0 tx=0x7f506401f1c0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.037+0000 7f5074609640 1 -- 172.21.15.46:0/3609594843 shutdown_connections 2023-10-30T14:15:06.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.037+0000 7f5074609640 1 --2- 172.21.15.46:0/3609594843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f506c0cf160 0x7f506c0cf590 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.037+0000 7f5074609640 1 --2- 172.21.15.46:0/3609594843 >> v2:172.21.15.154:3300/0 conn(0x7f506c15d6f0 0x7f506c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.037+0000 7f5074609640 1 -- 172.21.15.46:0/3609594843 >> 172.21.15.46:0/3609594843 conn(0x7f506c06bfe0 msgr2=0x7f506c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.037+0000 7f5074609640 1 -- 172.21.15.46:0/3609594843 shutdown_connections 2023-10-30T14:15:06.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.037+0000 7f5074609640 1 -- 172.21.15.46:0/3609594843 wait complete. 2023-10-30T14:15:06.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5074609640 1 Processor -- start 2023-10-30T14:15:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5074609640 1 -- start start 2023-10-30T14:15:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5074609640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f506c0cf160 0x7f506c14eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5074609640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f506c1538a0 0x7f506c14f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5074609640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f506c15d6f0 0x7f506c1f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:06.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5074609640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c0b43c0 con 0x7f506c15d6f0 2023-10-30T14:15:06.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5074609640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c1f33b0 con 0x7f506c0cf160 2023-10-30T14:15:06.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5074609640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506c0b4660 con 0x7f506c1538a0 2023-10-30T14:15:06.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5072ba6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f506c15d6f0 0x7f506c1f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:06.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5072ba6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f506c15d6f0 0x7f506c1f2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42654/0 (socket says 172.21.15.46:42654) 2023-10-30T14:15:06.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5072ba6640 1 -- 172.21.15.46:0/2778205612 learned_addr learned my addr 172.21.15.46:0/2778205612 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:06.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5071ba4640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f506c1538a0 0x7f506c14f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:06.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f50723a5640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f506c0cf160 0x7f506c14eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:06.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5072ba6640 1 -- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f506c1538a0 msgr2=0x7f506c14f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:06.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5072ba6640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f506c1538a0 0x7f506c14f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5072ba6640 1 -- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f506c0cf160 msgr2=0x7f506c14eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:06.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5072ba6640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f506c0cf160 0x7f506c14eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5072ba6640 1 -- 172.21.15.46:0/2778205612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5064009b30 con 0x7f506c15d6f0 2023-10-30T14:15:06.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5072ba6640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f506c15d6f0 0x7f506c1f2760 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f506401cdf0 tx=0x7f506401bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:06.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5062ffd640 1 -- 172.21.15.46:0/2778205612 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50640bf070 con 0x7f506c15d6f0 2023-10-30T14:15:06.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5062ffd640 1 -- 172.21.15.46:0/2778205612 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50640af9e0 con 0x7f506c15d6f0 2023-10-30T14:15:06.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5062ffd640 1 -- 172.21.15.46:0/2778205612 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50640afd50 con 0x7f506c15d6f0 2023-10-30T14:15:06.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5074609640 1 -- 172.21.15.46:0/2778205612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506c0b4940 con 0x7f506c15d6f0 2023-10-30T14:15:06.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5074609640 1 -- 172.21.15.46:0/2778205612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506c0b4f00 con 0x7f506c15d6f0 2023-10-30T14:15:06.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.041+0000 7f5074609640 1 -- 172.21.15.46:0/2778205612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506c0b2a50 con 0x7f506c15d6f0 2023-10-30T14:15:06.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.045+0000 7f5062ffd640 1 -- 172.21.15.46:0/2778205612 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f50640b6070 con 0x7f506c15d6f0 2023-10-30T14:15:06.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.045+0000 7f5062ffd640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50500664a0 0x7f5050068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:06.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.045+0000 7f5062ffd640 1 -- 172.21.15.46:0/2778205612 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f5064122090 con 0x7f506c15d6f0 2023-10-30T14:15:06.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.045+0000 7f50723a5640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50500664a0 0x7f5050068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:06.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.045+0000 7f50723a5640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50500664a0 0x7f5050068960 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f5068001810 tx=0x7f5068009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:06.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.049+0000 7f5062ffd640 1 -- 172.21.15.46:0/2778205612 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f50640eed50 con 0x7f506c15d6f0 2023-10-30T14:15:06.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.241+0000 7f5074609640 1 -- 172.21.15.46:0/2778205612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f506c04e170 con 0x7f506c15d6f0 2023-10-30T14:15:06.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.241+0000 7f5062ffd640 1 -- 172.21.15.46:0/2778205612 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f50640f2370 con 0x7f506c15d6f0 2023-10-30T14:15:06.246 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:06.247 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:05.587015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:resolve","state_seq":36,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 219 2023-10-30T14:15:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.245+0000 7f5074609640 1 -- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50500664a0 msgr2=0x7f5050068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:06.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.245+0000 7f5074609640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50500664a0 0x7f5050068960 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f5068001810 tx=0x7f5068009040 comp rx=0 tx=0).stop 2023-10-30T14:15:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.245+0000 7f5074609640 1 -- 172.21.15.46:0/2778205612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f506c15d6f0 msgr2=0x7f506c1f2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.245+0000 7f5074609640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f506c15d6f0 0x7f506c1f2760 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f506401cdf0 tx=0x7f506401bf60 comp rx=0 tx=0).stop 2023-10-30T14:15:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.245+0000 7f5074609640 1 -- 172.21.15.46:0/2778205612 shutdown_connections 2023-10-30T14:15:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.245+0000 7f5074609640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f50500664a0 0x7f5050068960 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.245+0000 7f5074609640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f506c15d6f0 0x7f506c1f2760 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.245+0000 7f5074609640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f506c1538a0 0x7f506c14f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.245+0000 7f5074609640 1 --2- 172.21.15.46:0/2778205612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f506c0cf160 0x7f506c14eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:06.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.245+0000 7f5074609640 1 -- 172.21.15.46:0/2778205612 >> 172.21.15.46:0/2778205612 conn(0x7f506c06bfe0 msgr2=0x7f506c15b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:06.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.245+0000 7f5074609640 1 -- 172.21.15.46:0/2778205612 shutdown_connections 2023-10-30T14:15:06.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:06.245+0000 7f5074609640 1 -- 172.21.15.46:0/2778205612 wait complete. 2023-10-30T14:15:06.267 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:08.269 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:08.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9f9c4640 1 Processor -- start 2023-10-30T14:15:08.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9f9c4640 1 -- start start 2023-10-30T14:15:08.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9f9c4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0a980cf160 0x7f0a980cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9f9c4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a980cb800 con 0x7f0a981538a0 2023-10-30T14:15:08.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9f9c4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a980cb330 con 0x7f0a9815d6f0 2023-10-30T14:15:08.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9f9c4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a980cad70 con 0x7f0a980cf160 2023-10-30T14:15:08.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9d760640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0a981538a0 0x7f0a980ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52624/0 (socket says 172.21.15.46:52624) 2023-10-30T14:15:08.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9d760640 1 -- 172.21.15.46:0/1310170474 learned_addr learned my addr 172.21.15.46:0/1310170474 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:08.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9df61640 1 --2- 172.21.15.46:0/1310170474 >> v2:172.21.15.154:3300/0 conn(0x7f0a980cf160 0x7f0a980cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282920497 0 0) 0x7f0a980cb800 con 0x7f0a981538a0 2023-10-30T14:15:08.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a84003530 con 0x7f0a981538a0 2023-10-30T14:15:08.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094405162 0 0) 0x7f0a980cb330 con 0x7f0a9815d6f0 2023-10-30T14:15:08.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a980cb800 con 0x7f0a9815d6f0 2023-10-30T14:15:08.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 403987073 0 0) 0x7f0a84003530 con 0x7f0a981538a0 2023-10-30T14:15:08.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 >> v2:172.21.15.154:3300/0 conn(0x7f0a980cf160 msgr2=0x7f0a980cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:08.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 --2- 172.21.15.46:0/1310170474 >> v2:172.21.15.154:3300/0 conn(0x7f0a980cf160 0x7f0a980cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 >> v1:172.21.15.154:6789/0 conn(0x7f0a9815d6f0 legacy=0x7f0a9815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:08.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a980cb050 con 0x7f0a981538a0 2023-10-30T14:15:08.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f0a94001680 con 0x7f0a981538a0 2023-10-30T14:15:08.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 >> v1:172.21.15.46:6789/0 conn(0x7f0a981538a0 legacy=0x7f0a980ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:08.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 --2- 172.21.15.46:0/1310170474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a84003b60 0x7f0a84003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 --2- 172.21.15.46:0/1310170474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a840047d0 0x7f0a84004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 --2- 172.21.15.46:0/1310170474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a84005190 0x7f0a8400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980cad70 con 0x7f0a84005190 2023-10-30T14:15:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8400bfb0 con 0x7f0a84003b60 2023-10-30T14:15:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8400c170 con 0x7f0a840047d0 2023-10-30T14:15:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9df61640 1 --2- 172.21.15.46:0/1310170474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a84005190 0x7f0a8400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9d760640 1 --2- 172.21.15.46:0/1310170474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a84003b60 0x7f0a84003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9cf5f640 1 --2- 172.21.15.46:0/1310170474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a840047d0 0x7f0a84004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9df61640 1 -- 172.21.15.46:0/1310170474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a840047d0 msgr2=0x7f0a84004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9df61640 1 --2- 172.21.15.46:0/1310170474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a840047d0 0x7f0a84004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9df61640 1 -- 172.21.15.46:0/1310170474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a84003b60 msgr2=0x7f0a84003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9df61640 1 --2- 172.21.15.46:0/1310170474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a84003b60 0x7f0a84003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.421+0000 7f0a9df61640 1 -- 172.21.15.46:0/1310170474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a84003530 con 0x7f0a84005190 2023-10-30T14:15:08.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.425+0000 7f0a9df61640 1 -- 172.21.15.46:0/1310170474 reap_dead start 2023-10-30T14:15:08.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.425+0000 7f0a9df61640 1 --2- 172.21.15.46:0/1310170474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a84005190 0x7f0a8400b990 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009fc0 tx=0x7f0a9001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e83374c97413de server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:08.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.425+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900a5cb0 con 0x7f0a84005190 2023-10-30T14:15:08.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.425+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9000f040 con 0x7f0a84005190 2023-10-30T14:15:08.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.425+0000 7f0a8ffff640 1 -- 172.21.15.46:0/1310170474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90003be0 con 0x7f0a84005190 2023-10-30T14:15:08.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.425+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/1310170474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a84005190 msgr2=0x7f0a8400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:08.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.425+0000 7f0a9f9c4640 1 --2- 172.21.15.46:0/1310170474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a84005190 0x7f0a8400b990 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009fc0 tx=0x7f0a9001cca0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.425+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/1310170474 shutdown_connections 2023-10-30T14:15:08.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.425+0000 7f0a9f9c4640 1 --2- 172.21.15.46:0/1310170474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a84005190 0x7f0a8400b990 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.425+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/1310170474 >> 172.21.15.46:0/1310170474 conn(0x7f0a9806bfe0 msgr2=0x7f0a9806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:08.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.425+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/1310170474 shutdown_connections 2023-10-30T14:15:08.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.425+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/1310170474 wait complete. 2023-10-30T14:15:08.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9f9c4640 1 Processor -- start 2023-10-30T14:15:08.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9f9c4640 1 -- start start 2023-10-30T14:15:08.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9f9c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a84005190 0x7f0a98065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9f9c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a9814db50 0x7f0a980c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9f9c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980c8760 0x7f0a980ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9f9c4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980c7d90 con 0x7f0a980c8760 2023-10-30T14:15:08.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9f9c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980c8be0 con 0x7f0a84005190 2023-10-30T14:15:08.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9f9c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a981f2c70 con 0x7f0a9814db50 2023-10-30T14:15:08.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9df61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980c8760 0x7f0a980ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9df61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980c8760 0x7f0a980ce000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34802/0 (socket says 172.21.15.46:34802) 2023-10-30T14:15:08.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9df61640 1 -- 172.21.15.46:0/3347314754 learned_addr learned my addr 172.21.15.46:0/3347314754 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:08.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9d760640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a84005190 0x7f0a98065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9cf5f640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a9814db50 0x7f0a980c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9df61640 1 -- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a9814db50 msgr2=0x7f0a980c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:08.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9df61640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a9814db50 0x7f0a980c8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9df61640 1 -- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a84005190 msgr2=0x7f0a98065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:08.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9df61640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a84005190 0x7f0a98065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9df61640 1 -- 172.21.15.46:0/3347314754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a980b49f0 con 0x7f0a980c8760 2023-10-30T14:15:08.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9d760640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a84005190 0x7f0a98065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:08.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9cf5f640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a9814db50 0x7f0a980c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:08.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9df61640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980c8760 0x7f0a980ce000 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f0a900987d0 tx=0x7f0a90009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:08.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a8dffb640 1 -- 172.21.15.46:0/3347314754 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900aecb0 con 0x7f0a980c8760 2023-10-30T14:15:08.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a8dffb640 1 -- 172.21.15.46:0/3347314754 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a900aee30 con 0x7f0a980c8760 2023-10-30T14:15:08.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/3347314754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a980b4cd0 con 0x7f0a980c8760 2023-10-30T14:15:08.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.429+0000 7f0a8dffb640 1 -- 172.21.15.46:0/3347314754 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90003be0 con 0x7f0a980c8760 2023-10-30T14:15:08.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.433+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/3347314754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a980c7a20 con 0x7f0a980c8760 2023-10-30T14:15:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.433+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/3347314754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a9804e170 con 0x7f0a980c8760 2023-10-30T14:15:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.433+0000 7f0a8dffb640 1 -- 172.21.15.46:0/3347314754 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0a90003d40 con 0x7f0a980c8760 2023-10-30T14:15:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.433+0000 7f0a8dffb640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a7c0664a0 0x7f0a7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.433+0000 7f0a8dffb640 1 -- 172.21.15.46:0/3347314754 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f0a900a6e40 con 0x7f0a980c8760 2023-10-30T14:15:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.433+0000 7f0a9d760640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a7c0664a0 0x7f0a7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.437+0000 7f0a9d760640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a7c0664a0 0x7f0a7c068960 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f0a94001060 tx=0x7f0a94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.437+0000 7f0a8dffb640 1 -- 172.21.15.46:0/3347314754 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0a900ec8b0 con 0x7f0a980c8760 2023-10-30T14:15:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.625+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/3347314754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a98158e50 con 0x7f0a980c8760 2023-10-30T14:15:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.625+0000 7f0a8dffb640 1 -- 172.21.15.46:0/3347314754 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f0a900be070 con 0x7f0a980c8760 2023-10-30T14:15:08.631 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:08.631 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:05.587015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:resolve","state_seq":36,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:08.631 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 219 2023-10-30T14:15:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.629+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a7c0664a0 msgr2=0x7f0a7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.629+0000 7f0a9f9c4640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a7c0664a0 0x7f0a7c068960 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f0a94001060 tx=0x7f0a94009040 comp rx=0 tx=0).stop 2023-10-30T14:15:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.629+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/3347314754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980c8760 msgr2=0x7f0a980ce000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.629+0000 7f0a9f9c4640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980c8760 0x7f0a980ce000 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f0a900987d0 tx=0x7f0a90009750 comp rx=0 tx=0).stop 2023-10-30T14:15:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.629+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/3347314754 shutdown_connections 2023-10-30T14:15:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.629+0000 7f0a9f9c4640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a7c0664a0 0x7f0a7c068960 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.629+0000 7f0a9f9c4640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a980c8760 0x7f0a980ce000 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.629+0000 7f0a9f9c4640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a9814db50 0x7f0a980c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.629+0000 7f0a9f9c4640 1 --2- 172.21.15.46:0/3347314754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a84005190 0x7f0a98065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.629+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/3347314754 >> 172.21.15.46:0/3347314754 conn(0x7f0a9806bfe0 msgr2=0x7f0a98157540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:08.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.629+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/3347314754 shutdown_connections 2023-10-30T14:15:08.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:08.629+0000 7f0a9f9c4640 1 -- 172.21.15.46:0/3347314754 wait complete. 2023-10-30T14:15:08.652 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 219, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 219, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8982": { "addr": "172.21.15.46:6833/2455118950", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2455118950, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2455118950, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8982, "incarnation": 209, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 37 }, "gid_9135": { "addr": "172.21.15.46:6837/681817982", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 681817982, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 681817982, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9135, "incarnation": 218, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:resolve", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:15:05.587015+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9135, "mds_1": 8982 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:15:10.653 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e36017640 1 Processor -- start 2023-10-30T14:15:10.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e36017640 1 -- start start 2023-10-30T14:15:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e36017640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e301538a0 0x7f4e3015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e36017640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4e300ca0e0 0x7f4e3015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e36017640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e300cb800 con 0x7f4e300cf160 2023-10-30T14:15:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e36017640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e3015dc30 con 0x7f4e300ca0e0 2023-10-30T14:15:10.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e36017640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e300cb330 con 0x7f4e301538a0 2023-10-30T14:15:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e2f7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e301538a0 0x7f4e3015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e2effd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4e300ca0e0 0x7f4e3015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:10.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e2ffff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f4e300cf160 0x7f4e300cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:39236/0 (socket says 172.21.15.46:39236) 2023-10-30T14:15:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e2ffff640 1 -- 172.21.15.46:0/4291930381 learned_addr learned my addr 172.21.15.46:0/4291930381 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e2effd640 1 -- 172.21.15.46:0/4291930381 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e301538a0 msgr2=0x7f4e3015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e2effd640 1 --2- 172.21.15.46:0/4291930381 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e301538a0 0x7f4e3015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e2effd640 1 -- 172.21.15.46:0/4291930381 >> v1:172.21.15.154:6789/0 conn(0x7f4e300cf160 legacy=0x7f4e300cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:10.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.805+0000 7f4e2effd640 1 -- 172.21.15.46:0/4291930381 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e300cad70 con 0x7f4e300ca0e0 2023-10-30T14:15:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.809+0000 7f4e2effd640 1 --2- 172.21.15.46:0/4291930381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300ca0e0 0x7f4e3015a090 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4e240097c0 tx=0x7f4e2401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b28e3858640dbaa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.809+0000 7f4e2e7fc640 1 -- 172.21.15.46:0/4291930381 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e240a8970 con 0x7f4e300ca0e0 2023-10-30T14:15:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e2e7fc640 1 -- 172.21.15.46:0/4291930381 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e240a8ad0 con 0x7f4e300ca0e0 2023-10-30T14:15:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e2e7fc640 1 -- 172.21.15.46:0/4291930381 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e240a8e60 con 0x7f4e300ca0e0 2023-10-30T14:15:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 -- 172.21.15.46:0/4291930381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300ca0e0 msgr2=0x7f4e3015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 --2- 172.21.15.46:0/4291930381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300ca0e0 0x7f4e3015a090 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4e240097c0 tx=0x7f4e2401ec70 comp rx=0 tx=0).stop 2023-10-30T14:15:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 -- 172.21.15.46:0/4291930381 shutdown_connections 2023-10-30T14:15:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 --2- 172.21.15.46:0/4291930381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300ca0e0 0x7f4e3015a090 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 --2- 172.21.15.46:0/4291930381 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e301538a0 0x7f4e3015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 -- 172.21.15.46:0/4291930381 >> 172.21.15.46:0/4291930381 conn(0x7f4e3006bfe0 msgr2=0x7f4e3006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 -- 172.21.15.46:0/4291930381 shutdown_connections 2023-10-30T14:15:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 -- 172.21.15.46:0/4291930381 wait complete. 2023-10-30T14:15:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 Processor -- start 2023-10-30T14:15:10.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 -- start start 2023-10-30T14:15:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300ca0e0 0x7f4e300b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300cf160 0x7f4e30148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e301538a0 0x7f4e30149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e30158070 con 0x7f4e301538a0 2023-10-30T14:15:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e301578a0 con 0x7f4e300cf160 2023-10-30T14:15:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e36017640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e300b4630 con 0x7f4e300ca0e0 2023-10-30T14:15:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e2ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e301538a0 0x7f4e30149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e2ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e301538a0 0x7f4e30149410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34818/0 (socket says 172.21.15.46:34818) 2023-10-30T14:15:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.813+0000 7f4e2ffff640 1 -- 172.21.15.46:0/466178519 learned_addr learned my addr 172.21.15.46:0/466178519 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e2f7fe640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300ca0e0 0x7f4e300b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e2effd640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300cf160 0x7f4e30148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e2ffff640 1 -- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300ca0e0 msgr2=0x7f4e300b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e2ffff640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300ca0e0 0x7f4e300b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e2ffff640 1 -- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300cf160 msgr2=0x7f4e30148ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e2ffff640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300cf160 0x7f4e30148ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e2ffff640 1 -- 172.21.15.46:0/466178519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24009420 con 0x7f4e301538a0 2023-10-30T14:15:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e2effd640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300cf160 0x7f4e30148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:10.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e2ffff640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e301538a0 0x7f4e30149410 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f4e2800ca70 tx=0x7f4e2800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e17fff640 1 -- 172.21.15.46:0/466178519 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2800abd0 con 0x7f4e301538a0 2023-10-30T14:15:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e17fff640 1 -- 172.21.15.46:0/466178519 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2800ad30 con 0x7f4e301538a0 2023-10-30T14:15:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e17fff640 1 -- 172.21.15.46:0/466178519 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28012600 con 0x7f4e301538a0 2023-10-30T14:15:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e36017640 1 -- 172.21.15.46:0/466178519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e301591f0 con 0x7f4e301538a0 2023-10-30T14:15:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e36017640 1 -- 172.21.15.46:0/466178519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e30159680 con 0x7f4e301538a0 2023-10-30T14:15:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e36017640 1 -- 172.21.15.46:0/466178519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e300b2a50 con 0x7f4e301538a0 2023-10-30T14:15:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.817+0000 7f4e17fff640 1 -- 172.21.15.46:0/466178519 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e28007070 con 0x7f4e301538a0 2023-10-30T14:15:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.821+0000 7f4e17fff640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e1c0664d0 0x7f4e1c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.821+0000 7f4e17fff640 1 -- 172.21.15.46:0/466178519 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f4e28084680 con 0x7f4e301538a0 2023-10-30T14:15:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.821+0000 7f4e2f7fe640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e1c0664d0 0x7f4e1c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.821+0000 7f4e2f7fe640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e1c0664d0 0x7f4e1c068990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4e20009ac0 tx=0x7f4e20009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:10.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:10.825+0000 7f4e17fff640 1 -- 172.21.15.46:0/466178519 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e28051190 con 0x7f4e301538a0 2023-10-30T14:15:11.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.013+0000 7f4e36017640 1 -- 172.21.15.46:0/466178519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e30159870 con 0x7f4e301538a0 2023-10-30T14:15:11.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e17fff640 1 -- 172.21.15.46:0/466178519 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f4e30159870 con 0x7f4e301538a0 2023-10-30T14:15:11.019 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:11.020 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:05.587015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:resolve","state_seq":36,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:11.020 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 219 2023-10-30T14:15:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e36017640 1 -- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e1c0664d0 msgr2=0x7f4e1c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e36017640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e1c0664d0 0x7f4e1c068990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4e20009ac0 tx=0x7f4e20009190 comp rx=0 tx=0).stop 2023-10-30T14:15:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e36017640 1 -- 172.21.15.46:0/466178519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e301538a0 msgr2=0x7f4e30149410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e36017640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e301538a0 0x7f4e30149410 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f4e2800ca70 tx=0x7f4e2800cf30 comp rx=0 tx=0).stop 2023-10-30T14:15:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e36017640 1 -- 172.21.15.46:0/466178519 shutdown_connections 2023-10-30T14:15:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e36017640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e1c0664d0 0x7f4e1c068990 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e36017640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e301538a0 0x7f4e30149410 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e36017640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300cf160 0x7f4e30148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e36017640 1 --2- 172.21.15.46:0/466178519 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e300ca0e0 0x7f4e300b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e36017640 1 -- 172.21.15.46:0/466178519 >> 172.21.15.46:0/466178519 conn(0x7f4e3006bfe0 msgr2=0x7f4e3006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e36017640 1 -- 172.21.15.46:0/466178519 shutdown_connections 2023-10-30T14:15:11.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:11.017+0000 7f4e36017640 1 -- 172.21.15.46:0/466178519 wait complete. 2023-10-30T14:15:11.041 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:13.041 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:13.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.193+0000 7fe63d90e640 1 Processor -- start 2023-10-30T14:15:13.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.193+0000 7fe63d90e640 1 -- start start 2023-10-30T14:15:13.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.193+0000 7fe63d90e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380c6b80 0x7fe6380bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:13.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.193+0000 7fe63d90e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6380c0a50 con 0x7fe63813f4a0 2023-10-30T14:15:13.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.193+0000 7fe63d90e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63806f6e0 con 0x7fe6380c6f60 2023-10-30T14:15:13.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.193+0000 7fe63d90e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63806f9c0 con 0x7fe6380c6b80 2023-10-30T14:15:13.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.193+0000 7fe6377fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe63813f4a0 0x7fe6381566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52638/0 (socket says 172.21.15.46:52638) 2023-10-30T14:15:13.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.193+0000 7fe6377fe640 1 -- 172.21.15.46:0/401076109 learned_addr learned my addr 172.21.15.46:0/401076109 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:13.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.193+0000 7fe636ffd640 1 --2- 172.21.15.46:0/401076109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380c6b80 0x7fe6380bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:13.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.193+0000 7fe635ffb640 1 -- 172.21.15.46:0/401076109 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358440588 0 0) 0x7fe6380c0a50 con 0x7fe63813f4a0 2023-10-30T14:15:13.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.193+0000 7fe635ffb640 1 -- 172.21.15.46:0/401076109 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe620005740 con 0x7fe63813f4a0 2023-10-30T14:15:13.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.197+0000 7fe635ffb640 1 -- 172.21.15.46:0/401076109 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351266974 0 0) 0x7fe63806f6e0 con 0x7fe6380c6f60 2023-10-30T14:15:13.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.197+0000 7fe635ffb640 1 -- 172.21.15.46:0/401076109 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6380c0a50 con 0x7fe6380c6f60 2023-10-30T14:15:13.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.197+0000 7fe635ffb640 1 -- 172.21.15.46:0/401076109 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1236380413 0 0) 0x7fe620005740 con 0x7fe63813f4a0 2023-10-30T14:15:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.197+0000 7fe636ffd640 1 -- 172.21.15.46:0/401076109 >> v1:172.21.15.154:6789/0 conn(0x7fe6380c6f60 legacy=0x7fe63813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.197+0000 7fe636ffd640 1 -- 172.21.15.46:0/401076109 >> v1:172.21.15.46:6789/0 conn(0x7fe63813f4a0 legacy=0x7fe6381566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.197+0000 7fe636ffd640 1 -- 172.21.15.46:0/401076109 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63806fca0 con 0x7fe6380c6b80 2023-10-30T14:15:13.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.197+0000 7fe636ffd640 1 --2- 172.21.15.46:0/401076109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380c6b80 0x7fe6380bfec0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe6280097f0 tx=0x7fe62801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3336d6d15dc804e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:13.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.197+0000 7fe635ffb640 1 -- 172.21.15.46:0/401076109 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630001a00 con 0x7fe6380c6b80 2023-10-30T14:15:13.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.197+0000 7fe635ffb640 1 -- 172.21.15.46:0/401076109 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6280a7050 con 0x7fe6380c6b80 2023-10-30T14:15:13.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.197+0000 7fe635ffb640 1 -- 172.21.15.46:0/401076109 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6280ab400 con 0x7fe6380c6b80 2023-10-30T14:15:13.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 -- 172.21.15.46:0/401076109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380c6b80 msgr2=0x7fe6380bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 --2- 172.21.15.46:0/401076109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380c6b80 0x7fe6380bfec0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe6280097f0 tx=0x7fe62801c3e0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 -- 172.21.15.46:0/401076109 shutdown_connections 2023-10-30T14:15:13.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 --2- 172.21.15.46:0/401076109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380c6b80 0x7fe6380bfec0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 -- 172.21.15.46:0/401076109 >> 172.21.15.46:0/401076109 conn(0x7fe6380c7910 msgr2=0x7fe6380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:13.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 -- 172.21.15.46:0/401076109 shutdown_connections 2023-10-30T14:15:13.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 -- 172.21.15.46:0/401076109 wait complete. 2023-10-30T14:15:13.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 Processor -- start 2023-10-30T14:15:13.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 -- start start 2023-10-30T14:15:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380c6b80 0x7fe638156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6380c6f60 0x7fe638158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:13.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe63813f4a0 0x7fe638152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6381f8010 con 0x7fe63813f4a0 2023-10-30T14:15:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6381546d0 con 0x7fe6380c6f60 2023-10-30T14:15:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe63d90e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6381549b0 con 0x7fe6380c6b80 2023-10-30T14:15:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe6377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe63813f4a0 0x7fe638152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe6377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe63813f4a0 0x7fe638152f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34832/0 (socket says 172.21.15.46:34832) 2023-10-30T14:15:13.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe6377fe640 1 -- 172.21.15.46:0/2216296195 learned_addr learned my addr 172.21.15.46:0/2216296195 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:13.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.201+0000 7fe636ffd640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380c6b80 0x7fe638156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:13.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe6367fc640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6380c6f60 0x7fe638158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:13.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe6377fe640 1 -- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380c6b80 msgr2=0x7fe638156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe6377fe640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380c6b80 0x7fe638156250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe6377fe640 1 -- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6380c6f60 msgr2=0x7fe638158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe6377fe640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6380c6f60 0x7fe638158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe6377fe640 1 -- 172.21.15.46:0/2216296195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe628009420 con 0x7fe63813f4a0 2023-10-30T14:15:13.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe6377fe640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe63813f4a0 0x7fe638152f80 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fe630005c10 tx=0x7fe63000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:13.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe61ffff640 1 -- 172.21.15.46:0/2216296195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63000ce30 con 0x7fe63813f4a0 2023-10-30T14:15:13.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe61ffff640 1 -- 172.21.15.46:0/2216296195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe630012400 con 0x7fe63813f4a0 2023-10-30T14:15:13.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe61ffff640 1 -- 172.21.15.46:0/2216296195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6300127f0 con 0x7fe63813f4a0 2023-10-30T14:15:13.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe63d90e640 1 -- 172.21.15.46:0/2216296195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe638154c90 con 0x7fe63813f4a0 2023-10-30T14:15:13.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe63d90e640 1 -- 172.21.15.46:0/2216296195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe638155250 con 0x7fe63813f4a0 2023-10-30T14:15:13.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe63d90e640 1 -- 172.21.15.46:0/2216296195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6381f81c0 con 0x7fe63813f4a0 2023-10-30T14:15:13.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe61ffff640 1 -- 172.21.15.46:0/2216296195 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe630012950 con 0x7fe63813f4a0 2023-10-30T14:15:13.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe61ffff640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe6240664a0 0x7fe624068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:13.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.205+0000 7fe61ffff640 1 -- 172.21.15.46:0/2216296195 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7fe630084f60 con 0x7fe63813f4a0 2023-10-30T14:15:13.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.209+0000 7fe636ffd640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe6240664a0 0x7fe624068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:13.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.209+0000 7fe636ffd640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe6240664a0 0x7fe624068960 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe6280ad000 tx=0x7fe6280a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:13.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.209+0000 7fe61ffff640 1 -- 172.21.15.46:0/2216296195 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe6381f81c0 con 0x7fe63813f4a0 2023-10-30T14:15:13.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.401+0000 7fe63d90e640 1 -- 172.21.15.46:0/2216296195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe63804e170 con 0x7fe63813f4a0 2023-10-30T14:15:13.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.401+0000 7fe61ffff640 1 -- 172.21.15.46:0/2216296195 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fe630055470 con 0x7fe63813f4a0 2023-10-30T14:15:13.406 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:13.407 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:11.024289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:13.407 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 220 2023-10-30T14:15:13.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.405+0000 7fe63d90e640 1 -- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe6240664a0 msgr2=0x7fe624068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.405+0000 7fe63d90e640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe6240664a0 0x7fe624068960 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe6280ad000 tx=0x7fe6280a6000 comp rx=0 tx=0).stop 2023-10-30T14:15:13.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.405+0000 7fe63d90e640 1 -- 172.21.15.46:0/2216296195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe63813f4a0 msgr2=0x7fe638152f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.405+0000 7fe63d90e640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe63813f4a0 0x7fe638152f80 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fe630005c10 tx=0x7fe63000a430 comp rx=0 tx=0).stop 2023-10-30T14:15:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.405+0000 7fe63d90e640 1 -- 172.21.15.46:0/2216296195 shutdown_connections 2023-10-30T14:15:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.405+0000 7fe63d90e640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe6240664a0 0x7fe624068960 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.405+0000 7fe63d90e640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe63813f4a0 0x7fe638152f80 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.405+0000 7fe63d90e640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6380c6f60 0x7fe638158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.405+0000 7fe63d90e640 1 --2- 172.21.15.46:0/2216296195 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380c6b80 0x7fe638156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.405+0000 7fe63d90e640 1 -- 172.21.15.46:0/2216296195 >> 172.21.15.46:0/2216296195 conn(0x7fe6380c7910 msgr2=0x7fe6380afe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.405+0000 7fe63d90e640 1 -- 172.21.15.46:0/2216296195 shutdown_connections 2023-10-30T14:15:13.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:13.405+0000 7fe63d90e640 1 -- 172.21.15.46:0/2216296195 wait complete. 2023-10-30T14:15:13.427 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:15.429 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:15.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad6a209640 1 Processor -- start 2023-10-30T14:15:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad6a209640 1 -- start start 2023-10-30T14:15:15.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad6a209640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad641538f0 0x7fad64159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:15.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad6a209640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad6415cd50 0x7fad640c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:15.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad6a209640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fad640cefa0 0x7fad640cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:15.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad6a209640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad6415d300 con 0x7fad6415cd50 2023-10-30T14:15:15.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad6a209640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad6415d5e0 con 0x7fad640cefa0 2023-10-30T14:15:15.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad6a209640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6415d8c0 con 0x7fad641538f0 2023-10-30T14:15:15.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad68a06640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad6415cd50 0x7fad640c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:15.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad68a06640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad6415cd50 0x7fad640c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34844/0 (socket says 172.21.15.46:34844) 2023-10-30T14:15:15.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad68a06640 1 -- 172.21.15.46:0/1654579291 learned_addr learned my addr 172.21.15.46:0/1654579291 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:15.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad69207640 1 --2- 172.21.15.46:0/1654579291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad641538f0 0x7fad64159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:15.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad69a08640 1 --2- 172.21.15.46:0/1654579291 >> v2:172.21.15.154:3300/0 conn(0x7fad640cefa0 0x7fad640cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:15.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad68a06640 1 -- 172.21.15.46:0/1654579291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad641538f0 msgr2=0x7fad64159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:15.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad68a06640 1 --2- 172.21.15.46:0/1654579291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad641538f0 0x7fad64159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:15.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad68a06640 1 -- 172.21.15.46:0/1654579291 >> v2:172.21.15.154:3300/0 conn(0x7fad640cefa0 msgr2=0x7fad640cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:15.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad68a06640 1 --2- 172.21.15.46:0/1654579291 >> v2:172.21.15.154:3300/0 conn(0x7fad640cefa0 0x7fad640cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:15.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.581+0000 7fad68a06640 1 -- 172.21.15.46:0/1654579291 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6415dba0 con 0x7fad6415cd50 2023-10-30T14:15:15.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad68a06640 1 --2- 172.21.15.46:0/1654579291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad6415cd50 0x7fad640c9f90 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fad640b1ae0 tx=0x7fad5401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef48499d28aecfc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:15.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad5bfff640 1 -- 172.21.15.46:0/1654579291 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad540a89b0 con 0x7fad6415cd50 2023-10-30T14:15:15.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad5bfff640 1 -- 172.21.15.46:0/1654579291 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad540a8b10 con 0x7fad6415cd50 2023-10-30T14:15:15.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad5bfff640 1 -- 172.21.15.46:0/1654579291 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad540a8ea0 con 0x7fad6415cd50 2023-10-30T14:15:15.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad6a209640 1 -- 172.21.15.46:0/1654579291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad6415cd50 msgr2=0x7fad640c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:15.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad6a209640 1 --2- 172.21.15.46:0/1654579291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad6415cd50 0x7fad640c9f90 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fad640b1ae0 tx=0x7fad5401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:15:15.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad6a209640 1 -- 172.21.15.46:0/1654579291 shutdown_connections 2023-10-30T14:15:15.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad6a209640 1 --2- 172.21.15.46:0/1654579291 >> v2:172.21.15.154:3300/0 conn(0x7fad640cefa0 0x7fad640cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:15.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad6a209640 1 --2- 172.21.15.46:0/1654579291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad6415cd50 0x7fad640c9f90 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:15.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad6a209640 1 --2- 172.21.15.46:0/1654579291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad641538f0 0x7fad64159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:15.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad6a209640 1 -- 172.21.15.46:0/1654579291 >> 172.21.15.46:0/1654579291 conn(0x7fad6406a640 msgr2=0x7fad64065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:15.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad6a209640 1 -- 172.21.15.46:0/1654579291 shutdown_connections 2023-10-30T14:15:15.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad6a209640 1 -- 172.21.15.46:0/1654579291 wait complete. 2023-10-30T14:15:15.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.585+0000 7fad6a209640 1 Processor -- start 2023-10-30T14:15:15.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad6a209640 1 -- start start 2023-10-30T14:15:15.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad6a209640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad640cefa0 0x7fad64147ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:15.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad6a209640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad641538f0 0x7fad64157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:15.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad6a209640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad6415cd50 0x7fad64157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:15.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad6a209640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6415c3a0 con 0x7fad641538f0 2023-10-30T14:15:15.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad6a209640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6415c6f0 con 0x7fad640cefa0 2023-10-30T14:15:15.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad6a209640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6415b4a0 con 0x7fad6415cd50 2023-10-30T14:15:15.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad68a06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad641538f0 0x7fad64157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:15.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad68a06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad641538f0 0x7fad64157a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34854/0 (socket says 172.21.15.46:34854) 2023-10-30T14:15:15.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad68a06640 1 -- 172.21.15.46:0/3784025095 learned_addr learned my addr 172.21.15.46:0/3784025095 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:15.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad69207640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad640cefa0 0x7fad64147ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:15.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad69a08640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad6415cd50 0x7fad64157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:15.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad68a06640 1 -- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad6415cd50 msgr2=0x7fad64157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:15.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad68a06640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad6415cd50 0x7fad64157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:15.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad68a06640 1 -- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad640cefa0 msgr2=0x7fad64147ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:15.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad68a06640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad640cefa0 0x7fad64147ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:15.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad68a06640 1 -- 172.21.15.46:0/3784025095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad540094d0 con 0x7fad641538f0 2023-10-30T14:15:15.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad69a08640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad6415cd50 0x7fad64157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:15.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad68a06640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad641538f0 0x7fad64157a60 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fad540b3040 tx=0x7fad5401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:15.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad59ffb640 1 -- 172.21.15.46:0/3784025095 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad540b9620 con 0x7fad641538f0 2023-10-30T14:15:15.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad59ffb640 1 -- 172.21.15.46:0/3784025095 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad540b9780 con 0x7fad641538f0 2023-10-30T14:15:15.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad59ffb640 1 -- 172.21.15.46:0/3784025095 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad540b9b30 con 0x7fad641538f0 2023-10-30T14:15:15.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad6a209640 1 -- 172.21.15.46:0/3784025095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad6415a920 con 0x7fad641538f0 2023-10-30T14:15:15.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad6a209640 1 -- 172.21.15.46:0/3784025095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6415aee0 con 0x7fad641538f0 2023-10-30T14:15:15.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.589+0000 7fad6a209640 1 -- 172.21.15.46:0/3784025095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad64159740 con 0x7fad641538f0 2023-10-30T14:15:15.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.593+0000 7fad59ffb640 1 -- 172.21.15.46:0/3784025095 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fad540b7070 con 0x7fad641538f0 2023-10-30T14:15:15.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.593+0000 7fad59ffb640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad480664a0 0x7fad48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:15.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.593+0000 7fad59ffb640 1 -- 172.21.15.46:0/3784025095 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7fad541226b0 con 0x7fad641538f0 2023-10-30T14:15:15.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.593+0000 7fad69207640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad480664a0 0x7fad48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:15.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.593+0000 7fad69207640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad480664a0 0x7fad48068960 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fad60007a60 tx=0x7fad60008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:15.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.597+0000 7fad59ffb640 1 -- 172.21.15.46:0/3784025095 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fad64159740 con 0x7fad641538f0 2023-10-30T14:15:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.785+0000 7fad6a209640 1 -- 172.21.15.46:0/3784025095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad64159740 con 0x7fad641538f0 2023-10-30T14:15:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.785+0000 7fad59ffb640 1 -- 172.21.15.46:0/3784025095 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fad64159740 con 0x7fad641538f0 2023-10-30T14:15:15.791 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:15.792 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:11.024289+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:15.792 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 220 2023-10-30T14:15:15.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.789+0000 7fad6a209640 1 -- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad480664a0 msgr2=0x7fad48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:15.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.789+0000 7fad6a209640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad480664a0 0x7fad48068960 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fad60007a60 tx=0x7fad60008040 comp rx=0 tx=0).stop 2023-10-30T14:15:15.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.789+0000 7fad6a209640 1 -- 172.21.15.46:0/3784025095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad641538f0 msgr2=0x7fad64157a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:15.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.789+0000 7fad6a209640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad641538f0 0x7fad64157a60 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fad540b3040 tx=0x7fad5401ba90 comp rx=0 tx=0).stop 2023-10-30T14:15:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.789+0000 7fad6a209640 1 -- 172.21.15.46:0/3784025095 shutdown_connections 2023-10-30T14:15:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.789+0000 7fad6a209640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fad480664a0 0x7fad48068960 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.789+0000 7fad6a209640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad6415cd50 0x7fad64157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.789+0000 7fad6a209640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad641538f0 0x7fad64157a60 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:15.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.789+0000 7fad6a209640 1 --2- 172.21.15.46:0/3784025095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad640cefa0 0x7fad64147ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.789+0000 7fad6a209640 1 -- 172.21.15.46:0/3784025095 >> 172.21.15.46:0/3784025095 conn(0x7fad6406a640 msgr2=0x7fad640c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.789+0000 7fad6a209640 1 -- 172.21.15.46:0/3784025095 shutdown_connections 2023-10-30T14:15:15.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:15.789+0000 7fad6a209640 1 -- 172.21.15.46:0/3784025095 wait complete. 2023-10-30T14:15:15.813 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:17.814 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:17.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.965+0000 7fe85ab94640 1 Processor -- start 2023-10-30T14:15:17.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.965+0000 7fe85ab94640 1 -- start start 2023-10-30T14:15:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.965+0000 7fe85ab94640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe854150920 0x7fe85406d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.965+0000 7fe85ab94640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8540c68b0 con 0x7fe8540cb7e0 2023-10-30T14:15:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.965+0000 7fe85ab94640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe854150d00 con 0x7fe8540c2570 2023-10-30T14:15:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.965+0000 7fe85ab94640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85406d960 con 0x7fe854150920 2023-10-30T14:15:17.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.969+0000 7fe858930640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe854150920 0x7fe85406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:17.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.969+0000 7fe859131640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe8540cb7e0 0x7fe854156680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34676/0 (socket says 172.21.15.46:34676) 2023-10-30T14:15:17.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.969+0000 7fe859131640 1 -- 172.21.15.46:0/808796837 learned_addr learned my addr 172.21.15.46:0/808796837 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:17.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.969+0000 7fe858930640 1 -- 172.21.15.46:0/808796837 >> v1:172.21.15.154:6789/0 conn(0x7fe8540c2570 legacy=0x7fe8540cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:17.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.969+0000 7fe858930640 1 -- 172.21.15.46:0/808796837 >> v1:172.21.15.46:6789/0 conn(0x7fe8540cb7e0 legacy=0x7fe854156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:17.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.969+0000 7fe8537fe640 1 -- 172.21.15.46:0/808796837 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767541746 0 0) 0x7fe8540c68b0 con 0x7fe8540cb7e0 2023-10-30T14:15:17.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.969+0000 7fe858930640 1 -- 172.21.15.46:0/808796837 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8540c2e90 con 0x7fe854150920 2023-10-30T14:15:17.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.969+0000 7fe858930640 1 --2- 172.21.15.46:0/808796837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe854150920 0x7fe85406d420 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fe8480097c0 tx=0x7fe84801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931b6cb80e91f7a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:17.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.969+0000 7fe8537fe640 1 -- 172.21.15.46:0/808796837 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8480ab600 con 0x7fe854150920 2023-10-30T14:15:17.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.969+0000 7fe8537fe640 1 -- 172.21.15.46:0/808796837 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8480a7050 con 0x7fe854150920 2023-10-30T14:15:17.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.969+0000 7fe8537fe640 1 -- 172.21.15.46:0/808796837 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8480ab980 con 0x7fe854150920 2023-10-30T14:15:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 -- 172.21.15.46:0/808796837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe854150920 msgr2=0x7fe85406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 --2- 172.21.15.46:0/808796837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe854150920 0x7fe85406d420 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fe8480097c0 tx=0x7fe84801c480 comp rx=0 tx=0).stop 2023-10-30T14:15:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 -- 172.21.15.46:0/808796837 shutdown_connections 2023-10-30T14:15:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 --2- 172.21.15.46:0/808796837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe854150920 0x7fe85406d420 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 -- 172.21.15.46:0/808796837 >> 172.21.15.46:0/808796837 conn(0x7fe85406dfd0 msgr2=0x7fe85406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 -- 172.21.15.46:0/808796837 shutdown_connections 2023-10-30T14:15:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 -- 172.21.15.46:0/808796837 wait complete. 2023-10-30T14:15:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 Processor -- start 2023-10-30T14:15:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 -- start start 2023-10-30T14:15:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8540c2570 0x7fe8541556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8540cb7e0 0x7fe85414faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe854150920 0x7fe854146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8541f2570 con 0x7fe8540cb7e0 2023-10-30T14:15:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8541f2850 con 0x7fe8540c2570 2023-10-30T14:15:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe85ab94640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8541f2b30 con 0x7fe854150920 2023-10-30T14:15:17.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe853fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8540cb7e0 0x7fe85414faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:17.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe853fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8540cb7e0 0x7fe85414faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42666/0 (socket says 172.21.15.46:42666) 2023-10-30T14:15:17.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe853fff640 1 -- 172.21.15.46:0/692502320 learned_addr learned my addr 172.21.15.46:0/692502320 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:17.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe858930640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8540c2570 0x7fe8541556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:17.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe859131640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe854150920 0x7fe854146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:17.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe853fff640 1 -- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe854150920 msgr2=0x7fe854146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe853fff640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe854150920 0x7fe854146800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe853fff640 1 -- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8540c2570 msgr2=0x7fe8541556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe853fff640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8540c2570 0x7fe8541556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.973+0000 7fe853fff640 1 -- 172.21.15.46:0/692502320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe848009420 con 0x7fe8540cb7e0 2023-10-30T14:15:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.977+0000 7fe853fff640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8540cb7e0 0x7fe85414faa0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fe84000c500 tx=0x7fe84000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.977+0000 7fe8517fa640 1 -- 172.21.15.46:0/692502320 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840005220 con 0x7fe8540cb7e0 2023-10-30T14:15:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.977+0000 7fe8517fa640 1 -- 172.21.15.46:0/692502320 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe840005380 con 0x7fe8540cb7e0 2023-10-30T14:15:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.977+0000 7fe8517fa640 1 -- 172.21.15.46:0/692502320 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840007270 con 0x7fe8540cb7e0 2023-10-30T14:15:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.977+0000 7fe85ab94640 1 -- 172.21.15.46:0/692502320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85414f2a0 con 0x7fe8540cb7e0 2023-10-30T14:15:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.977+0000 7fe85ab94640 1 -- 172.21.15.46:0/692502320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8540710b0 con 0x7fe8540cb7e0 2023-10-30T14:15:17.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.977+0000 7fe85ab94640 1 -- 172.21.15.46:0/692502320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85404e170 con 0x7fe8540cb7e0 2023-10-30T14:15:17.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.977+0000 7fe8517fa640 1 -- 172.21.15.46:0/692502320 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8400073d0 con 0x7fe8540cb7e0 2023-10-30T14:15:17.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.977+0000 7fe8517fa640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe8380664d0 0x7fe838068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.977+0000 7fe8517fa640 1 -- 172.21.15.46:0/692502320 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7fe840084130 con 0x7fe8540cb7e0 2023-10-30T14:15:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.977+0000 7fe858930640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe8380664d0 0x7fe838068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.981+0000 7fe858930640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe8380664d0 0x7fe838068990 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe8480ad000 tx=0x7fe848008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:17.981+0000 7fe8517fa640 1 -- 172.21.15.46:0/692502320 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe840050ba0 con 0x7fe8540cb7e0 2023-10-30T14:15:18.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.173+0000 7fe85ab94640 1 -- 172.21.15.46:0/692502320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe85414f750 con 0x7fe8540cb7e0 2023-10-30T14:15:18.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.173+0000 7fe8517fa640 1 -- 172.21.15.46:0/692502320 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fe8400541c0 con 0x7fe8540cb7e0 2023-10-30T14:15:18.178 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:18.178 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:16.701981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 221 2023-10-30T14:15:18.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.177+0000 7fe85ab94640 1 -- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe8380664d0 msgr2=0x7fe838068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:18.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.177+0000 7fe85ab94640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe8380664d0 0x7fe838068990 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe8480ad000 tx=0x7fe848008040 comp rx=0 tx=0).stop 2023-10-30T14:15:18.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.177+0000 7fe85ab94640 1 -- 172.21.15.46:0/692502320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8540cb7e0 msgr2=0x7fe85414faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:18.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.177+0000 7fe85ab94640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8540cb7e0 0x7fe85414faa0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fe84000c500 tx=0x7fe84000c9c0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.177+0000 7fe85ab94640 1 -- 172.21.15.46:0/692502320 shutdown_connections 2023-10-30T14:15:18.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.177+0000 7fe85ab94640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe8380664d0 0x7fe838068990 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.177+0000 7fe85ab94640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe854150920 0x7fe854146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.177+0000 7fe85ab94640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8540cb7e0 0x7fe85414faa0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.177+0000 7fe85ab94640 1 --2- 172.21.15.46:0/692502320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8540c2570 0x7fe8541556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:18.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.177+0000 7fe85ab94640 1 -- 172.21.15.46:0/692502320 >> 172.21.15.46:0/692502320 conn(0x7fe85406dfd0 msgr2=0x7fe85414cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.177+0000 7fe85ab94640 1 -- 172.21.15.46:0/692502320 shutdown_connections 2023-10-30T14:15:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:18.177+0000 7fe85ab94640 1 -- 172.21.15.46:0/692502320 wait complete. 2023-10-30T14:15:18.199 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:15:20.200 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:20.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.349+0000 7f9279491640 1 Processor -- start 2023-10-30T14:15:20.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.349+0000 7f9279491640 1 -- start start 2023-10-30T14:15:20.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.353+0000 7f9279491640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9274071110 0x7f927406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:20.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.353+0000 7f9279491640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9274066cf0 con 0x7f92740c6bd0 2023-10-30T14:15:20.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.353+0000 7f9279491640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92740c70d0 con 0x7f927406f7a0 2023-10-30T14:15:20.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.353+0000 7f9279491640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92740700c0 con 0x7f9274071110 2023-10-30T14:15:20.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.353+0000 7f926bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f92740c6bd0 0x7f9274158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34678/0 (socket says 172.21.15.46:34678) 2023-10-30T14:15:20.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.353+0000 7f926bfff640 1 -- 172.21.15.46:0/3622448541 learned_addr learned my addr 172.21.15.46:0/3622448541 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:20.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.353+0000 7f92737fe640 1 --2- 172.21.15.46:0/3622448541 >> v2:172.21.15.46:3300/0 conn(0x7f9274071110 0x7f927406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:20.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.353+0000 7f92737fe640 1 -- 172.21.15.46:0/3622448541 >> v1:172.21.15.154:6789/0 conn(0x7f927406f7a0 legacy=0x7f9274069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:20.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.353+0000 7f92737fe640 1 -- 172.21.15.46:0/3622448541 >> v1:172.21.15.46:6789/0 conn(0x7f92740c6bd0 legacy=0x7f9274158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:20.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.353+0000 7f92737fe640 1 -- 172.21.15.46:0/3622448541 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92740c7390 con 0x7f9274071110 2023-10-30T14:15:20.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.353+0000 7f92727fc640 1 -- 172.21.15.46:0/3622448541 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617941276 0 0) 0x7f9274066cf0 con 0x7f92740c6bd0 2023-10-30T14:15:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.353+0000 7f92737fe640 1 --2- 172.21.15.46:0/3622448541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9274071110 0x7f927406fb80 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f926c009d90 tx=0x7f926c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeae02260968d15a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f92727fc640 1 -- 172.21.15.46:0/3622448541 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c0af600 con 0x7f9274071110 2023-10-30T14:15:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f92727fc640 1 -- 172.21.15.46:0/3622448541 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c0ab050 con 0x7f9274071110 2023-10-30T14:15:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f92727fc640 1 -- 172.21.15.46:0/3622448541 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c0af980 con 0x7f9274071110 2023-10-30T14:15:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 -- 172.21.15.46:0/3622448541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9274071110 msgr2=0x7f927406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 --2- 172.21.15.46:0/3622448541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9274071110 0x7f927406fb80 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f926c009d90 tx=0x7f926c01f130 comp rx=0 tx=0).stop 2023-10-30T14:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 -- 172.21.15.46:0/3622448541 shutdown_connections 2023-10-30T14:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 --2- 172.21.15.46:0/3622448541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9274071110 0x7f927406fb80 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 -- 172.21.15.46:0/3622448541 >> 172.21.15.46:0/3622448541 conn(0x7f92740c7910 msgr2=0x7f92740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 -- 172.21.15.46:0/3622448541 shutdown_connections 2023-10-30T14:15:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 -- 172.21.15.46:0/3622448541 wait complete. 2023-10-30T14:15:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 Processor -- start 2023-10-30T14:15:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 -- start start 2023-10-30T14:15:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f927406f7a0 0x7f92741ecbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9274071110 0x7f92741f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92740c6bd0 0x7f927414edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92740c6730 con 0x7f9274071110 2023-10-30T14:15:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92741ed100 con 0x7f92740c6bd0 2023-10-30T14:15:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f9279491640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92740c6450 con 0x7f927406f7a0 2023-10-30T14:15:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f926bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9274071110 0x7f92741f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:20.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f926bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9274071110 0x7f92741f23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42678/0 (socket says 172.21.15.46:42678) 2023-10-30T14:15:20.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.357+0000 7f926bfff640 1 -- 172.21.15.46:0/3630143014 learned_addr learned my addr 172.21.15.46:0/3630143014 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:20.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f92737fe640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92740c6bd0 0x7f927414edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:20.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f9272ffd640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f927406f7a0 0x7f92741ecbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:20.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f926bfff640 1 -- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f927406f7a0 msgr2=0x7f92741ecbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:20.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f926bfff640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f927406f7a0 0x7f92741ecbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:20.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f926bfff640 1 -- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92740c6bd0 msgr2=0x7f927414edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:20.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f926bfff640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92740c6bd0 0x7f927414edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:20.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f926bfff640 1 -- 172.21.15.46:0/3630143014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c009a70 con 0x7f9274071110 2023-10-30T14:15:20.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f926bfff640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9274071110 0x7f92741f23b0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f925c00c4b0 tx=0x7f925c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:20.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f926b7fe640 1 -- 172.21.15.46:0/3630143014 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c0074e0 con 0x7f9274071110 2023-10-30T14:15:20.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f926b7fe640 1 -- 172.21.15.46:0/3630143014 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925c007660 con 0x7f9274071110 2023-10-30T14:15:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f926b7fe640 1 -- 172.21.15.46:0/3630143014 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c004a30 con 0x7f9274071110 2023-10-30T14:15:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f9279491640 1 -- 172.21.15.46:0/3630143014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9274154ea0 con 0x7f9274071110 2023-10-30T14:15:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f9279491640 1 -- 172.21.15.46:0/3630143014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9274151e10 con 0x7f9274071110 2023-10-30T14:15:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f9279491640 1 -- 172.21.15.46:0/3630143014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92741550e0 con 0x7f9274071110 2023-10-30T14:15:20.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f926b7fe640 1 -- 172.21.15.46:0/3630143014 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f925c004b90 con 0x7f9274071110 2023-10-30T14:15:20.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f926b7fe640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92540664d0 0x7f9254068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:20.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f926b7fe640 1 -- 172.21.15.46:0/3630143014 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f925c083a10 con 0x7f9274071110 2023-10-30T14:15:20.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.361+0000 7f9272ffd640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92540664d0 0x7f9254068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:20.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.365+0000 7f9272ffd640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92540664d0 0x7f9254068990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9264000f70 tx=0x7f9264009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:20.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.365+0000 7f926b7fe640 1 -- 172.21.15.46:0/3630143014 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92741550e0 con 0x7f9274071110 2023-10-30T14:15:20.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 -- 172.21.15.46:0/3630143014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92741531c0 con 0x7f9274071110 2023-10-30T14:15:20.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f926b7fe640 1 -- 172.21.15.46:0/3630143014 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f92741531c0 con 0x7f9274071110 2023-10-30T14:15:20.561 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:20.561 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:16.701981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:20.561 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 221 2023-10-30T14:15:20.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 -- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92540664d0 msgr2=0x7f9254068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:20.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92540664d0 0x7f9254068990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9264000f70 tx=0x7f9264009040 comp rx=0 tx=0).stop 2023-10-30T14:15:20.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 -- 172.21.15.46:0/3630143014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9274071110 msgr2=0x7f92741f23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:20.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9274071110 0x7f92741f23b0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f925c00c4b0 tx=0x7f925c00c970 comp rx=0 tx=0).stop 2023-10-30T14:15:20.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 -- 172.21.15.46:0/3630143014 shutdown_connections 2023-10-30T14:15:20.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92540664d0 0x7f9254068990 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:20.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92740c6bd0 0x7f927414edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:20.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9274071110 0x7f92741f23b0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:20.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 --2- 172.21.15.46:0/3630143014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f927406f7a0 0x7f92741ecbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 -- 172.21.15.46:0/3630143014 >> 172.21.15.46:0/3630143014 conn(0x7f92740c7910 msgr2=0x7f92740c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 -- 172.21.15.46:0/3630143014 shutdown_connections 2023-10-30T14:15:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:20.557+0000 7f9279491640 1 -- 172.21.15.46:0/3630143014 wait complete. 2023-10-30T14:15:20.583 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 221, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 221, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8982": { "addr": "172.21.15.46:6833/2455118950", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 2455118950, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 2455118950, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8982, "incarnation": 209, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 37 }, "gid_9135": { "addr": "172.21.15.46:6837/681817982", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 681817982, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 681817982, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9135, "incarnation": 218, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 40 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:15:16.701981+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9135, "mds_1": 8982 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:15:22.584 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:22.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.733+0000 7f6d87780640 1 Processor -- start 2023-10-30T14:15:22.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.733+0000 7f6d87780640 1 -- start start 2023-10-30T14:15:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d87780640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d800c1ee0 0x7f6d8006e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d87780640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6d800c8260 0x7f6d800cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d87780640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8006e090 con 0x7f6d80151800 2023-10-30T14:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d87780640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8006dc70 con 0x7f6d800c8260 2023-10-30T14:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d87780640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800c8a50 con 0x7f6d800c1ee0 2023-10-30T14:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d8551c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6d80151800 0x7f6d800c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34682/0 (socket says 172.21.15.46:34682) 2023-10-30T14:15:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d8551c640 1 -- 172.21.15.46:0/3764533104 learned_addr learned my addr 172.21.15.46:0/3764533104 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d84d1b640 1 --2- 172.21.15.46:0/3764533104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d800c1ee0 0x7f6d8006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d85d1d640 1 --2- 172.21.15.46:0/3764533104 >> v2:172.21.15.154:3300/0 conn(0x7f6d800c8260 0x7f6d800cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898175953 0 0) 0x7f6d8006e090 con 0x7f6d80151800 2023-10-30T14:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d6c003530 con 0x7f6d80151800 2023-10-30T14:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3648483048 0 0) 0x7f6d6c003530 con 0x7f6d80151800 2023-10-30T14:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d800c1ee0 msgr2=0x7f6d8006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:22.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 --2- 172.21.15.46:0/3764533104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d800c1ee0 0x7f6d8006e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:22.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 >> v2:172.21.15.154:3300/0 conn(0x7f6d800c8260 msgr2=0x7f6d800cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:22.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 --2- 172.21.15.46:0/3764533104 >> v2:172.21.15.154:3300/0 conn(0x7f6d800c8260 0x7f6d800cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:22.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d800c3b10 con 0x7f6d80151800 2023-10-30T14:15:22.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f6d7c0016a0 con 0x7f6d80151800 2023-10-30T14:15:22.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 >> v1:172.21.15.46:6789/0 conn(0x7f6d80151800 legacy=0x7f6d800c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 --2- 172.21.15.46:0/3764533104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d6c003980 0x7f6d6c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 --2- 172.21.15.46:0/3764533104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d6c0047d0 0x7f6d6c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 --2- 172.21.15.46:0/3764533104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d800c1ee0 0x7f6d6c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8006dc70 con 0x7f6d6c0047d0 2023-10-30T14:15:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800c8a50 con 0x7f6d6c003980 2023-10-30T14:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c00b990 con 0x7f6d800c1ee0 2023-10-30T14:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d84d1b640 1 --2- 172.21.15.46:0/3764533104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d6c0047d0 0x7f6d6c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d8551c640 1 --2- 172.21.15.46:0/3764533104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d6c003980 0x7f6d6c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d85d1d640 1 --2- 172.21.15.46:0/3764533104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d800c1ee0 0x7f6d6c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d84d1b640 1 -- 172.21.15.46:0/3764533104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d800c1ee0 msgr2=0x7f6d6c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d84d1b640 1 --2- 172.21.15.46:0/3764533104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d800c1ee0 0x7f6d6c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d84d1b640 1 -- 172.21.15.46:0/3764533104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d6c003980 msgr2=0x7f6d6c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d84d1b640 1 --2- 172.21.15.46:0/3764533104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d6c003980 0x7f6d6c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.737+0000 7f6d84d1b640 1 -- 172.21.15.46:0/3764533104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c003530 con 0x7f6d6c0047d0 2023-10-30T14:15:22.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d84d1b640 1 --2- 172.21.15.46:0/3764533104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d6c0047d0 0x7f6d6c004c50 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009ab0 tx=0x7f6d7001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc07a230425a26c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:22.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d700a5540 con 0x7f6d6c0047d0 2023-10-30T14:15:22.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d700a56a0 con 0x7f6d6c0047d0 2023-10-30T14:15:22.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d77fff640 1 -- 172.21.15.46:0/3764533104 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d700a5a10 con 0x7f6d6c0047d0 2023-10-30T14:15:22.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d87780640 1 -- 172.21.15.46:0/3764533104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d6c0047d0 msgr2=0x7f6d6c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:22.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d87780640 1 --2- 172.21.15.46:0/3764533104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d6c0047d0 0x7f6d6c004c50 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009ab0 tx=0x7f6d7001ca30 comp rx=0 tx=0).stop 2023-10-30T14:15:22.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d85d1d640 1 -- 172.21.15.46:0/3764533104 reap_dead start 2023-10-30T14:15:22.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d87780640 1 -- 172.21.15.46:0/3764533104 shutdown_connections 2023-10-30T14:15:22.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d87780640 1 -- 172.21.15.46:0/3764533104 >> 172.21.15.46:0/3764533104 conn(0x7f6d80071ea0 msgr2=0x7f6d80065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:22.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d87780640 1 -- 172.21.15.46:0/3764533104 shutdown_connections 2023-10-30T14:15:22.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d87780640 1 -- 172.21.15.46:0/3764533104 wait complete. 2023-10-30T14:15:22.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d87780640 1 Processor -- start 2023-10-30T14:15:22.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.741+0000 7f6d87780640 1 -- start start 2023-10-30T14:15:22.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d87780640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d6c0047d0 0x7f6d801479d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:22.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d87780640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d8006b150 0x7f6d8013e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:22.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d87780640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d8006b5b0 0x7f6d8013ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:22.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d87780640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d801449d0 con 0x7f6d6c0047d0 2023-10-30T14:15:22.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d87780640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8006c000 con 0x7f6d8006b5b0 2023-10-30T14:15:22.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d87780640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d801443e0 con 0x7f6d8006b150 2023-10-30T14:15:22.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d8551c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d6c0047d0 0x7f6d801479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:22.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d8551c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d6c0047d0 0x7f6d801479d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42696/0 (socket says 172.21.15.46:42696) 2023-10-30T14:15:22.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d8551c640 1 -- 172.21.15.46:0/3495477808 learned_addr learned my addr 172.21.15.46:0/3495477808 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:22.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d84d1b640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d8006b150 0x7f6d8013e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:22.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d85d1d640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d8006b5b0 0x7f6d8013ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:22.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d8551c640 1 -- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d8006b150 msgr2=0x7f6d8013e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:22.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d8551c640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d8006b150 0x7f6d8013e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:22.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d8551c640 1 -- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d8006b5b0 msgr2=0x7f6d8013ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:22.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d8551c640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d8006b5b0 0x7f6d8013ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:22.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d8551c640 1 -- 172.21.15.46:0/3495477808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d801587d0 con 0x7f6d6c0047d0 2023-10-30T14:15:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d85d1d640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d8006b5b0 0x7f6d8013ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d84d1b640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d8006b150 0x7f6d8013e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d8551c640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d6c0047d0 0x7f6d801479d0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c012510 tx=0x7f6d7c0129d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d75ffb640 1 -- 172.21.15.46:0/3495477808 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c014810 con 0x7f6d6c0047d0 2023-10-30T14:15:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d75ffb640 1 -- 172.21.15.46:0/3495477808 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7c014970 con 0x7f6d6c0047d0 2023-10-30T14:15:22.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d87780640 1 -- 172.21.15.46:0/3495477808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d801e3110 con 0x7f6d6c0047d0 2023-10-30T14:15:22.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d75ffb640 1 -- 172.21.15.46:0/3495477808 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c014d20 con 0x7f6d6c0047d0 2023-10-30T14:15:22.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d87780640 1 -- 172.21.15.46:0/3495477808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d801445f0 con 0x7f6d6c0047d0 2023-10-30T14:15:22.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.745+0000 7f6d87780640 1 -- 172.21.15.46:0/3495477808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d80144b10 con 0x7f6d6c0047d0 2023-10-30T14:15:22.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.749+0000 7f6d75ffb640 1 -- 172.21.15.46:0/3495477808 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6d7c0153e0 con 0x7f6d6c0047d0 2023-10-30T14:15:22.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.749+0000 7f6d75ffb640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d640664d0 0x7f6d64068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:22.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.749+0000 7f6d75ffb640 1 -- 172.21.15.46:0/3495477808 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f6d7c0872b0 con 0x7f6d6c0047d0 2023-10-30T14:15:22.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.749+0000 7f6d84d1b640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d640664d0 0x7f6d64068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:22.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.749+0000 7f6d84d1b640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d640664d0 0x7f6d64068990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f6d700af000 tx=0x7f6d700046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:22.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.753+0000 7f6d75ffb640 1 -- 172.21.15.46:0/3495477808 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6d7c053c00 con 0x7f6d6c0047d0 2023-10-30T14:15:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 -- 172.21.15.46:0/3495477808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d801e35c0 con 0x7f6d6c0047d0 2023-10-30T14:15:22.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d75ffb640 1 -- 172.21.15.46:0/3495477808 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f6d7c01c020 con 0x7f6d6c0047d0 2023-10-30T14:15:22.949 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:22.949 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:21.950322+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:22.949 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 222 2023-10-30T14:15:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 -- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d640664d0 msgr2=0x7f6d64068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d640664d0 0x7f6d64068990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f6d700af000 tx=0x7f6d700046d0 comp rx=0 tx=0).stop 2023-10-30T14:15:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 -- 172.21.15.46:0/3495477808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d6c0047d0 msgr2=0x7f6d801479d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:22.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d6c0047d0 0x7f6d801479d0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c012510 tx=0x7f6d7c0129d0 comp rx=0 tx=0).stop 2023-10-30T14:15:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 -- 172.21.15.46:0/3495477808 shutdown_connections 2023-10-30T14:15:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6d640664d0 0x7f6d64068990 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d8006b5b0 0x7f6d8013ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d8006b150 0x7f6d8013e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 --2- 172.21.15.46:0/3495477808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d6c0047d0 0x7f6d801479d0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:22.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 -- 172.21.15.46:0/3495477808 >> 172.21.15.46:0/3495477808 conn(0x7f6d80071ea0 msgr2=0x7f6d8006ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:22.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 -- 172.21.15.46:0/3495477808 shutdown_connections 2023-10-30T14:15:22.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:22.945+0000 7f6d87780640 1 -- 172.21.15.46:0/3495477808 wait complete. 2023-10-30T14:15:22.970 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=8826 2023-10-30T14:15:22.970 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.c 2023-10-30T14:15:24.971 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T14:15:24.972 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T14:15:24.972 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T14:15:24.976 INFO:tasks.ceph.mds.c:Started 2023-10-30T14:15:24.977 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:25.072 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-10-30T14:15:25.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.129+0000 7f7b3c66b640 1 Processor -- start 2023-10-30T14:15:25.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.129+0000 7f7b3c66b640 1 -- start start 2023-10-30T14:15:25.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.129+0000 7f7b3c66b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7b340c6bd0 0x7f7b3406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:25.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.129+0000 7f7b3c66b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b34071110 0x7f7b34158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:25.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.129+0000 7f7b3c66b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b34066cf0 con 0x7f7b3406f7a0 2023-10-30T14:15:25.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.129+0000 7f7b3c66b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b340700c0 con 0x7f7b340c6bd0 2023-10-30T14:15:25.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.129+0000 7f7b3c66b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b340c70d0 con 0x7f7b34071110 2023-10-30T14:15:25.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.129+0000 7f7b3ac08640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b34071110 0x7f7b34158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:25.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.129+0000 7f7b39c06640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7b340c6bd0 0x7f7b3406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:25.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.129+0000 7f7b3ac08640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b34071110 0x7f7b34158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:49938/0 (socket says 172.21.15.46:49938) 2023-10-30T14:15:25.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.129+0000 7f7b3ac08640 1 -- 172.21.15.46:0/2103710567 learned_addr learned my addr 172.21.15.46:0/2103710567 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:25.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.133+0000 7f7b3ac08640 1 -- 172.21.15.46:0/2103710567 >> v2:172.21.15.154:3300/0 conn(0x7f7b340c6bd0 msgr2=0x7f7b3406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:25.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.133+0000 7f7b3ac08640 1 --2- 172.21.15.46:0/2103710567 >> v2:172.21.15.154:3300/0 conn(0x7f7b340c6bd0 0x7f7b3406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:25.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.133+0000 7f7b3ac08640 1 -- 172.21.15.46:0/2103710567 >> v1:172.21.15.154:6789/0 conn(0x7f7b3406f7a0 legacy=0x7f7b34069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:25.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.133+0000 7f7b3ac08640 1 -- 172.21.15.46:0/2103710567 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b340c7390 con 0x7f7b34071110 2023-10-30T14:15:25.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.133+0000 7f7b3ac08640 1 --2- 172.21.15.46:0/2103710567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b34071110 0x7f7b34158110 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009e50 tx=0x7f7b2c01f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=177a8c4dff38d248 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:25.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.133+0000 7f7b39405640 1 -- 172.21.15.46:0/2103710567 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c0af400 con 0x7f7b34071110 2023-10-30T14:15:25.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.133+0000 7f7b39405640 1 -- 172.21.15.46:0/2103710567 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2c0ab050 con 0x7f7b34071110 2023-10-30T14:15:25.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.133+0000 7f7b39405640 1 -- 172.21.15.46:0/2103710567 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c0af780 con 0x7f7b34071110 2023-10-30T14:15:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 -- 172.21.15.46:0/2103710567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b34071110 msgr2=0x7f7b34158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 --2- 172.21.15.46:0/2103710567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b34071110 0x7f7b34158110 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009e50 tx=0x7f7b2c01f1f0 comp rx=0 tx=0).stop 2023-10-30T14:15:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 -- 172.21.15.46:0/2103710567 shutdown_connections 2023-10-30T14:15:25.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 --2- 172.21.15.46:0/2103710567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b34071110 0x7f7b34158110 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:25.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 --2- 172.21.15.46:0/2103710567 >> v2:172.21.15.154:3300/0 conn(0x7f7b340c6bd0 0x7f7b3406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:25.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 -- 172.21.15.46:0/2103710567 >> 172.21.15.46:0/2103710567 conn(0x7f7b340c7910 msgr2=0x7f7b340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:25.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 -- 172.21.15.46:0/2103710567 shutdown_connections 2023-10-30T14:15:25.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 -- 172.21.15.46:0/2103710567 wait complete. 2023-10-30T14:15:25.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 Processor -- start 2023-10-30T14:15:25.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 -- start start 2023-10-30T14:15:25.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b3406f7a0 0x7f7b34152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:25.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b34071110 0x7f7b340c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:25.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b340c6bd0 0x7f7b3414d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:25.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b341e2e30 con 0x7f7b34071110 2023-10-30T14:15:25.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b341e20a0 con 0x7f7b3406f7a0 2023-10-30T14:15:25.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3c66b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b341eeb70 con 0x7f7b340c6bd0 2023-10-30T14:15:25.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b39c06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b34071110 0x7f7b340c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:25.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b39c06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b34071110 0x7f7b340c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42714/0 (socket says 172.21.15.46:42714) 2023-10-30T14:15:25.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b39c06640 1 -- 172.21.15.46:0/964330127 learned_addr learned my addr 172.21.15.46:0/964330127 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:25.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3ac08640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b340c6bd0 0x7f7b3414d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:25.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b3a407640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b3406f7a0 0x7f7b34152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:25.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b39c06640 1 -- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b340c6bd0 msgr2=0x7f7b3414d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:25.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b39c06640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b340c6bd0 0x7f7b3414d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:25.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b39c06640 1 -- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b3406f7a0 msgr2=0x7f7b34152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:25.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b39c06640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b3406f7a0 0x7f7b34152200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:25.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.137+0000 7f7b39c06640 1 -- 172.21.15.46:0/964330127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c009b30 con 0x7f7b34071110 2023-10-30T14:15:25.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.141+0000 7f7b39c06640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b34071110 0x7f7b340c5760 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f7b2400e900 tx=0x7f7b2400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:25.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.141+0000 7f7b2affd640 1 -- 172.21.15.46:0/964330127 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b24009690 con 0x7f7b34071110 2023-10-30T14:15:25.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.141+0000 7f7b2affd640 1 -- 172.21.15.46:0/964330127 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2400ca00 con 0x7f7b34071110 2023-10-30T14:15:25.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.141+0000 7f7b2affd640 1 -- 172.21.15.46:0/964330127 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2400cd90 con 0x7f7b34071110 2023-10-30T14:15:25.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.141+0000 7f7b3c66b640 1 -- 172.21.15.46:0/964330127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b341eee50 con 0x7f7b34071110 2023-10-30T14:15:25.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.141+0000 7f7b3c66b640 1 -- 172.21.15.46:0/964330127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b341ef410 con 0x7f7b34071110 2023-10-30T14:15:25.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.141+0000 7f7b3c66b640 1 -- 172.21.15.46:0/964330127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b341e2fe0 con 0x7f7b34071110 2023-10-30T14:15:25.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.141+0000 7f7b2affd640 1 -- 172.21.15.46:0/964330127 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7b24013470 con 0x7f7b34071110 2023-10-30T14:15:25.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.141+0000 7f7b2affd640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b180664a0 0x7f7b18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:25.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.141+0000 7f7b2affd640 1 -- 172.21.15.46:0/964330127 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f7b240849d0 con 0x7f7b34071110 2023-10-30T14:15:25.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.141+0000 7f7b3a407640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b180664a0 0x7f7b18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:25.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.145+0000 7f7b3a407640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b180664a0 0x7f7b18068960 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7b30000f70 tx=0x7f7b30009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:25.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.145+0000 7f7b2affd640 1 -- 172.21.15.46:0/964330127 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7b240514e0 con 0x7f7b34071110 2023-10-30T14:15:25.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 -- 172.21.15.46:0/964330127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b3404e170 con 0x7f7b34071110 2023-10-30T14:15:25.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b2affd640 1 -- 172.21.15.46:0/964330127 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f7b24017090 con 0x7f7b34071110 2023-10-30T14:15:25.345 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:25.345 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:21.950322+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:25.345 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 222 2023-10-30T14:15:25.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 -- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b180664a0 msgr2=0x7f7b18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:25.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b180664a0 0x7f7b18068960 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7b30000f70 tx=0x7f7b30009040 comp rx=0 tx=0).stop 2023-10-30T14:15:25.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 -- 172.21.15.46:0/964330127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b34071110 msgr2=0x7f7b340c5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:25.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b34071110 0x7f7b340c5760 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f7b2400e900 tx=0x7f7b2400edc0 comp rx=0 tx=0).stop 2023-10-30T14:15:25.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 -- 172.21.15.46:0/964330127 shutdown_connections 2023-10-30T14:15:25.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b180664a0 0x7f7b18068960 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b340c6bd0 0x7f7b3414d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b34071110 0x7f7b340c5760 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 --2- 172.21.15.46:0/964330127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b3406f7a0 0x7f7b34152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 -- 172.21.15.46:0/964330127 >> 172.21.15.46:0/964330127 conn(0x7f7b340c7910 msgr2=0x7f7b340c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:25.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 -- 172.21.15.46:0/964330127 shutdown_connections 2023-10-30T14:15:25.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:25.341+0000 7f7b3c66b640 1 -- 172.21.15.46:0/964330127 wait complete. 2023-10-30T14:15:25.366 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T14:15:27.368 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:27.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.517+0000 7f92c8446640 1 Processor -- start 2023-10-30T14:15:27.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c8446640 1 -- start start 2023-10-30T14:15:27.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c8446640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92c00c6bd0 0x7f92c006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:27.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c8446640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f92c0071110 0x7f92c0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:27.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c8446640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c0066cf0 con 0x7f92c006f7a0 2023-10-30T14:15:27.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c8446640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c00700c0 con 0x7f92c0071110 2023-10-30T14:15:27.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c8446640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00c70d0 con 0x7f92c00c6bd0 2023-10-30T14:15:27.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c59e1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92c00c6bd0 0x7f92c006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:27.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c69e3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f92c0071110 0x7f92c0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:27.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c59e1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92c00c6bd0 0x7f92c006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:43098/0 (socket says 172.21.15.46:43098) 2023-10-30T14:15:27.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c59e1640 1 -- 172.21.15.46:0/3423573194 learned_addr learned my addr 172.21.15.46:0/3423573194 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:27.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c69e3640 1 -- 172.21.15.46:0/3423573194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92c00c6bd0 msgr2=0x7f92c006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:27.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c69e3640 1 --2- 172.21.15.46:0/3423573194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92c00c6bd0 0x7f92c006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:27.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c69e3640 1 -- 172.21.15.46:0/3423573194 >> v1:172.21.15.154:6789/0 conn(0x7f92c006f7a0 legacy=0x7f92c0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:27.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c69e3640 1 -- 172.21.15.46:0/3423573194 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00c7390 con 0x7f92c0071110 2023-10-30T14:15:27.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.521+0000 7f92c59e1640 1 --2- 172.21.15.46:0/3423573194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92c00c6bd0 0x7f92c006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:15:27.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.525+0000 7f92c69e3640 1 --2- 172.21.15.46:0/3423573194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92c0071110 0x7f92c0158110 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009d90 tx=0x7f92b801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3c25b2f8adb0385 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:27.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.525+0000 7f92c51e0640 1 -- 172.21.15.46:0/3423573194 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b80af600 con 0x7f92c0071110 2023-10-30T14:15:27.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.525+0000 7f92c51e0640 1 -- 172.21.15.46:0/3423573194 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b80ab050 con 0x7f92c0071110 2023-10-30T14:15:27.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.525+0000 7f92c51e0640 1 -- 172.21.15.46:0/3423573194 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b80af980 con 0x7f92c0071110 2023-10-30T14:15:27.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.525+0000 7f92c8446640 1 -- 172.21.15.46:0/3423573194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92c0071110 msgr2=0x7f92c0158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:27.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.525+0000 7f92c8446640 1 --2- 172.21.15.46:0/3423573194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92c0071110 0x7f92c0158110 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009d90 tx=0x7f92b801f130 comp rx=0 tx=0).stop 2023-10-30T14:15:27.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.525+0000 7f92c8446640 1 -- 172.21.15.46:0/3423573194 shutdown_connections 2023-10-30T14:15:27.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.525+0000 7f92c8446640 1 --2- 172.21.15.46:0/3423573194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92c0071110 0x7f92c0158110 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:27.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.525+0000 7f92c8446640 1 --2- 172.21.15.46:0/3423573194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92c00c6bd0 0x7f92c006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:27.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.525+0000 7f92c8446640 1 -- 172.21.15.46:0/3423573194 >> 172.21.15.46:0/3423573194 conn(0x7f92c00c7910 msgr2=0x7f92c00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:27.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.525+0000 7f92c8446640 1 -- 172.21.15.46:0/3423573194 shutdown_connections 2023-10-30T14:15:27.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.525+0000 7f92c8446640 1 -- 172.21.15.46:0/3423573194 wait complete. 2023-10-30T14:15:27.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c8446640 1 Processor -- start 2023-10-30T14:15:27.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c8446640 1 -- start start 2023-10-30T14:15:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c8446640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92c006f7a0 0x7f92c015a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c8446640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92c0071110 0x7f92c01f2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c8446640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92c00c6bd0 0x7f92c01588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c8446640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c0156f10 con 0x7f92c00c6bd0 2023-10-30T14:15:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c8446640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c015b850 con 0x7f92c0071110 2023-10-30T14:15:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c8446640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c006aad0 con 0x7f92c006f7a0 2023-10-30T14:15:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c69e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92c00c6bd0 0x7f92c01588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:27.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c69e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92c00c6bd0 0x7f92c01588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55266/0 (socket says 172.21.15.46:55266) 2023-10-30T14:15:27.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c69e3640 1 -- 172.21.15.46:0/2241614364 learned_addr learned my addr 172.21.15.46:0/2241614364 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:27.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c61e2640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92c006f7a0 0x7f92c015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:27.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c59e1640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92c0071110 0x7f92c01f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:27.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c69e3640 1 -- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92c006f7a0 msgr2=0x7f92c015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c69e3640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92c006f7a0 0x7f92c015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c69e3640 1 -- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92c0071110 msgr2=0x7f92c01f2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c69e3640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92c0071110 0x7f92c01f2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c69e3640 1 -- 172.21.15.46:0/2241614364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b00094d0 con 0x7f92c00c6bd0 2023-10-30T14:15:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c69e3640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92c00c6bd0 0x7f92c01588a0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f92b80b2040 tx=0x7f92b801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92b6ffd640 1 -- 172.21.15.46:0/2241614364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8009450 con 0x7f92c00c6bd0 2023-10-30T14:15:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92b6ffd640 1 -- 172.21.15.46:0/2241614364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b80b9410 con 0x7f92c00c6bd0 2023-10-30T14:15:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92b6ffd640 1 -- 172.21.15.46:0/2241614364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b80b97c0 con 0x7f92c00c6bd0 2023-10-30T14:15:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c8446640 1 -- 172.21.15.46:0/2241614364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b8009a70 con 0x7f92c00c6bd0 2023-10-30T14:15:27.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.529+0000 7f92c8446640 1 -- 172.21.15.46:0/2241614364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c01f2fe0 con 0x7f92c00c6bd0 2023-10-30T14:15:27.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.533+0000 7f92c8446640 1 -- 172.21.15.46:0/2241614364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c015ba00 con 0x7f92c00c6bd0 2023-10-30T14:15:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.533+0000 7f92b6ffd640 1 -- 172.21.15.46:0/2241614364 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f92b80b6050 con 0x7f92c00c6bd0 2023-10-30T14:15:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.533+0000 7f92b6ffd640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92a40664a0 0x7f92a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.533+0000 7f92b6ffd640 1 -- 172.21.15.46:0/2241614364 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f92b8122bf0 con 0x7f92c00c6bd0 2023-10-30T14:15:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.533+0000 7f92c61e2640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92a40664a0 0x7f92a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.533+0000 7f92c61e2640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92a40664a0 0x7f92a4068960 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f92bc007fd0 tx=0x7f92bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:27.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.537+0000 7f92b6ffd640 1 -- 172.21.15.46:0/2241614364 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92c015ba00 con 0x7f92c00c6bd0 2023-10-30T14:15:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.725+0000 7f92c8446640 1 -- 172.21.15.46:0/2241614364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92c015ba00 con 0x7f92c00c6bd0 2023-10-30T14:15:27.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.725+0000 7f92b6ffd640 1 -- 172.21.15.46:0/2241614364 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f92c015ba00 con 0x7f92c00c6bd0 2023-10-30T14:15:27.730 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:27.730 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:21.950322+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:27.731 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 223 2023-10-30T14:15:27.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.729+0000 7f92c8446640 1 -- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92a40664a0 msgr2=0x7f92a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:27.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.729+0000 7f92c8446640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92a40664a0 0x7f92a4068960 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f92bc007fd0 tx=0x7f92bc009040 comp rx=0 tx=0).stop 2023-10-30T14:15:27.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.729+0000 7f92c8446640 1 -- 172.21.15.46:0/2241614364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92c00c6bd0 msgr2=0x7f92c01588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:27.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.729+0000 7f92c8446640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92c00c6bd0 0x7f92c01588a0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f92b80b2040 tx=0x7f92b801bed0 comp rx=0 tx=0).stop 2023-10-30T14:15:27.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.729+0000 7f92c8446640 1 -- 172.21.15.46:0/2241614364 shutdown_connections 2023-10-30T14:15:27.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.729+0000 7f92c8446640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f92a40664a0 0x7f92a4068960 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:27.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.729+0000 7f92c8446640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92c00c6bd0 0x7f92c01588a0 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:27.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.729+0000 7f92c8446640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92c0071110 0x7f92c01f2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:27.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.729+0000 7f92c8446640 1 --2- 172.21.15.46:0/2241614364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92c006f7a0 0x7f92c015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:27.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.729+0000 7f92c8446640 1 -- 172.21.15.46:0/2241614364 >> 172.21.15.46:0/2241614364 conn(0x7f92c00c7910 msgr2=0x7f92c00c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:27.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.729+0000 7f92c8446640 1 -- 172.21.15.46:0/2241614364 shutdown_connections 2023-10-30T14:15:27.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:27.729+0000 7f92c8446640 1 -- 172.21.15.46:0/2241614364 wait complete. 2023-10-30T14:15:27.751 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T14:15:27.751 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 19 secs before thrashing 2023-10-30T14:15:28.279 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:15:28.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:15:28.906 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T14:15:28.905+0000 7efe6f0c7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:15:28.906 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T14:15:28.905+0000 7efe6f0c7640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:15:37.270 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:15:37.270 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.46 2023-10-30T14:15:37.271 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 46 -eq 48 ] 2023-10-30T14:15:37.271 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.46 2023-10-30T14:15:37.271 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.46 2023-10-30T14:15:37.271 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.46 2023-10-30T14:15:37.271 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:15:44.335 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:15:44.336 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.47 2023-10-30T14:15:44.336 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 47 -eq 48 ] 2023-10-30T14:15:44.336 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.47 2023-10-30T14:15:44.337 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.47 2023-10-30T14:15:44.337 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.47 2023-10-30T14:15:44.337 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:15:46.753 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:46.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4aa4d6640 1 Processor -- start 2023-10-30T14:15:46.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4aa4d6640 1 -- start start 2023-10-30T14:15:46.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4aa4d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a41538a0 0x7fa4a415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:46.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4aa4d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40cb800 con 0x7fa4a40ca0e0 2023-10-30T14:15:46.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4aa4d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a415dc30 con 0x7fa4a40cf160 2023-10-30T14:15:46.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4aa4d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a40cb330 con 0x7fa4a41538a0 2023-10-30T14:15:46.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa49bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa4a40ca0e0 0x7fa4a415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39328/0 (socket says 172.21.15.46:39328) 2023-10-30T14:15:46.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa49bfff640 1 -- 172.21.15.46:0/3287641752 learned_addr learned my addr 172.21.15.46:0/3287641752 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:46.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a3fff640 1 --2- 172.21.15.46:0/3287641752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a41538a0 0x7fa4a415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:46.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794405748 0 0) 0x7fa4a40cb800 con 0x7fa4a40ca0e0 2023-10-30T14:15:46.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa490003530 con 0x7fa4a40ca0e0 2023-10-30T14:15:46.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1590849922 0 0) 0x7fa490003530 con 0x7fa4a40ca0e0 2023-10-30T14:15:46.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a41538a0 msgr2=0x7fa4a415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:46.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 --2- 172.21.15.46:0/3287641752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a41538a0 0x7fa4a415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:46.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 >> v1:172.21.15.154:6789/0 conn(0x7fa4a40cf160 legacy=0x7fa4a4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:46.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a40cad70 con 0x7fa4a40ca0e0 2023-10-30T14:15:46.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fa48c001680 con 0x7fa4a40ca0e0 2023-10-30T14:15:46.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 >> v1:172.21.15.46:6789/0 conn(0x7fa4a40ca0e0 legacy=0x7fa4a415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:46.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 --2- 172.21.15.46:0/3287641752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4900039b0 0x7fa490003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:46.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 --2- 172.21.15.46:0/3287641752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900047d0 0x7fa490004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:46.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 --2- 172.21.15.46:0/3287641752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a41538a0 0x7fa490005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:46.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a40cb330 con 0x7fa4900039b0 2023-10-30T14:15:46.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49000bad0 con 0x7fa4900047d0 2023-10-30T14:15:46.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49000bc90 con 0x7fa4a41538a0 2023-10-30T14:15:46.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a3fff640 1 --2- 172.21.15.46:0/3287641752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4900039b0 0x7fa490003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:46.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa49bfff640 1 --2- 172.21.15.46:0/3287641752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900047d0 0x7fa490004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:46.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a8a73640 1 --2- 172.21.15.46:0/3287641752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a41538a0 0x7fa490005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:46.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a3fff640 1 -- 172.21.15.46:0/3287641752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a41538a0 msgr2=0x7fa490005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:46.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a3fff640 1 --2- 172.21.15.46:0/3287641752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a41538a0 0x7fa490005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:46.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a3fff640 1 -- 172.21.15.46:0/3287641752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900047d0 msgr2=0x7fa490004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:46.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a3fff640 1 --2- 172.21.15.46:0/3287641752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900047d0 0x7fa490004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:46.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.933+0000 7fa4a3fff640 1 -- 172.21.15.46:0/3287641752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa490003530 con 0x7fa4900039b0 2023-10-30T14:15:46.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.937+0000 7fa4a3fff640 1 --2- 172.21.15.46:0/3287641752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4900039b0 0x7fa490003dd0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fa494009ab0 tx=0x7fa49401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0649fffdaf8b011 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:46.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.937+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4940a6500 con 0x7fa4900039b0 2023-10-30T14:15:46.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.937+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4940a6660 con 0x7fa4900039b0 2023-10-30T14:15:46.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.937+0000 7fa4a37fe640 1 -- 172.21.15.46:0/3287641752 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4940a6a10 con 0x7fa4900039b0 2023-10-30T14:15:46.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.937+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/3287641752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4900039b0 msgr2=0x7fa490003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:46.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.937+0000 7fa4aa4d6640 1 --2- 172.21.15.46:0/3287641752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4900039b0 0x7fa490003dd0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fa494009ab0 tx=0x7fa49401ca10 comp rx=0 tx=0).stop 2023-10-30T14:15:46.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.937+0000 7fa4a8a73640 1 -- 172.21.15.46:0/3287641752 reap_dead start 2023-10-30T14:15:46.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.937+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/3287641752 shutdown_connections 2023-10-30T14:15:46.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.937+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/3287641752 >> 172.21.15.46:0/3287641752 conn(0x7fa4a406bfe0 msgr2=0x7fa4a406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:46.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.937+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/3287641752 shutdown_connections 2023-10-30T14:15:46.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.937+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/3287641752 wait complete. 2023-10-30T14:15:46.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4aa4d6640 1 Processor -- start 2023-10-30T14:15:46.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4aa4d6640 1 -- start start 2023-10-30T14:15:46.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4aa4d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900039b0 0x7fa4a40cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:46.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4aa4d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a41e5700 0x7fa4a40ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4aa4d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a40cea60 0x7fa4a40b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4aa4d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a41efea0 con 0x7fa4a41e5700 2023-10-30T14:15:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4aa4d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a41f03d0 con 0x7fa4900039b0 2023-10-30T14:15:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4aa4d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a40cd450 con 0x7fa4a40cea60 2023-10-30T14:15:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa49bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a41e5700 0x7fa4a40ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:46.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa49bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a41e5700 0x7fa4a40ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48092/0 (socket says 172.21.15.46:48092) 2023-10-30T14:15:46.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa49bfff640 1 -- 172.21.15.46:0/2083370483 learned_addr learned my addr 172.21.15.46:0/2083370483 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:46.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4a3fff640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900039b0 0x7fa4a40cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:46.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4a8a73640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a40cea60 0x7fa4a40b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:46.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa49bfff640 1 -- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a40cea60 msgr2=0x7fa4a40b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:46.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa49bfff640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a40cea60 0x7fa4a40b3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa49bfff640 1 -- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900039b0 msgr2=0x7fa4a40cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa49bfff640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900039b0 0x7fa4a40cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa49bfff640 1 -- 172.21.15.46:0/2083370483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a40cd710 con 0x7fa4a41e5700 2023-10-30T14:15:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4a3fff640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900039b0 0x7fa4a40cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa49bfff640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a41e5700 0x7fa4a40ce0d0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fa48c011810 tx=0x7fa48c011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4a17fa640 1 -- 172.21.15.46:0/2083370483 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c00fb60 con 0x7fa4a41e5700 2023-10-30T14:15:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4a17fa640 1 -- 172.21.15.46:0/2083370483 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48c00fce0 con 0x7fa4a41e5700 2023-10-30T14:15:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4a17fa640 1 -- 172.21.15.46:0/2083370483 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c015650 con 0x7fa4a41e5700 2023-10-30T14:15:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/2083370483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a41f00f0 con 0x7fa4a41e5700 2023-10-30T14:15:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/2083370483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a40b35b0 con 0x7fa4a41e5700 2023-10-30T14:15:46.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.941+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/2083370483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a404e170 con 0x7fa4a41e5700 2023-10-30T14:15:46.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.945+0000 7fa4a17fa640 1 -- 172.21.15.46:0/2083370483 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa48c0157b0 con 0x7fa4a41e5700 2023-10-30T14:15:46.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.945+0000 7fa4a17fa640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4840664d0 0x7fa484068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:46.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.945+0000 7fa4a17fa640 1 -- 172.21.15.46:0/2083370483 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7fa48c087b40 con 0x7fa4a41e5700 2023-10-30T14:15:46.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.945+0000 7fa4a3fff640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4840664d0 0x7fa484068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:46.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.945+0000 7fa4a3fff640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4840664d0 0x7fa484068990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fa4940af000 tx=0x7fa4940046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:46.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:46.949+0000 7fa4a17fa640 1 -- 172.21.15.46:0/2083370483 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa48c0545a0 con 0x7fa4a41e5700 2023-10-30T14:15:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.137+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/2083370483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4a40b1dd0 con 0x7fa4a41e5700 2023-10-30T14:15:47.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4a17fa640 1 -- 172.21.15.46:0/2083370483 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fa48c057bc0 con 0x7fa4a41e5700 2023-10-30T14:15:47.144 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:47.144 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:32.954977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:47.145 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 224 2023-10-30T14:15:47.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4840664d0 msgr2=0x7fa484068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4aa4d6640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4840664d0 0x7fa484068990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fa4940af000 tx=0x7fa4940046d0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/2083370483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a41e5700 msgr2=0x7fa4a40ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4aa4d6640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a41e5700 0x7fa4a40ce0d0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fa48c011810 tx=0x7fa48c011cd0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/2083370483 shutdown_connections 2023-10-30T14:15:47.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4aa4d6640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4840664d0 0x7fa484068990 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4aa4d6640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4a40cea60 0x7fa4a40b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4aa4d6640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4a41e5700 0x7fa4a40ce0d0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4aa4d6640 1 --2- 172.21.15.46:0/2083370483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900039b0 0x7fa4a40cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/2083370483 >> 172.21.15.46:0/2083370483 conn(0x7fa4a406bfe0 msgr2=0x7fa4a4150f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:47.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/2083370483 shutdown_connections 2023-10-30T14:15:47.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.141+0000 7fa4aa4d6640 1 -- 172.21.15.46:0/2083370483 wait complete. 2023-10-30T14:15:47.171 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=1) 2023-10-30T14:15:47.171 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T14:15:47.171 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:15:47.200 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T14:15:47.200 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:47.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a5450f640 1 Processor -- start 2023-10-30T14:15:47.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a5450f640 1 -- start start 2023-10-30T14:15:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a5450f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a4c0cc0b0 0x7f9a4c06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a5450f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a4c0b1ea0 0x7f9a4c0cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a5450f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4c06ade0 con 0x7f9a4c153c30 2023-10-30T14:15:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a5450f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4c1596e0 con 0x7f9a4c0cc0b0 2023-10-30T14:15:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a5450f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c1598f0 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a51aaa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a4c0cc0b0 0x7f9a4c06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a51aaa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a4c0cc0b0 0x7f9a4c06a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48106/0 (socket says 172.21.15.46:48106) 2023-10-30T14:15:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a51aaa640 1 -- 172.21.15.46:0/960237369 learned_addr learned my addr 172.21.15.46:0/960237369 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:47.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a52aac640 1 --2- 172.21.15.46:0/960237369 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a4c0b1ea0 0x7f9a4c0cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:47.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a51aaa640 1 -- 172.21.15.46:0/960237369 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a4c0b1ea0 msgr2=0x7f9a4c0cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:47.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a51aaa640 1 --2- 172.21.15.46:0/960237369 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a4c0b1ea0 0x7f9a4c0cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a51aaa640 1 -- 172.21.15.46:0/960237369 >> v1:172.21.15.154:6789/0 conn(0x7f9a4c153c30 legacy=0x7f9a4c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:47.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.381+0000 7f9a51aaa640 1 -- 172.21.15.46:0/960237369 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c159b90 con 0x7f9a4c0cc0b0 2023-10-30T14:15:47.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.385+0000 7f9a51aaa640 1 --2- 172.21.15.46:0/960237369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a4c0cc0b0 0x7f9a4c06a8a0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c006fd0 tx=0x7f9a3c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02dbb1f73fc4993 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:47.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.385+0000 7f9a512a9640 1 -- 172.21.15.46:0/960237369 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c0a8a10 con 0x7f9a4c0cc0b0 2023-10-30T14:15:47.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.385+0000 7f9a512a9640 1 -- 172.21.15.46:0/960237369 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c0a8b70 con 0x7f9a4c0cc0b0 2023-10-30T14:15:47.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.385+0000 7f9a512a9640 1 -- 172.21.15.46:0/960237369 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c0b0440 con 0x7f9a4c0cc0b0 2023-10-30T14:15:47.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.385+0000 7f9a5450f640 1 -- 172.21.15.46:0/960237369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a4c0cc0b0 msgr2=0x7f9a4c06a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:47.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.385+0000 7f9a5450f640 1 --2- 172.21.15.46:0/960237369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a4c0cc0b0 0x7f9a4c06a8a0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c006fd0 tx=0x7f9a3c01ed10 comp rx=0 tx=0).stop 2023-10-30T14:15:47.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.385+0000 7f9a5450f640 1 -- 172.21.15.46:0/960237369 shutdown_connections 2023-10-30T14:15:47.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.385+0000 7f9a5450f640 1 --2- 172.21.15.46:0/960237369 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a4c0b1ea0 0x7f9a4c0cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.385+0000 7f9a5450f640 1 --2- 172.21.15.46:0/960237369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a4c0cc0b0 0x7f9a4c06a8a0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.385+0000 7f9a5450f640 1 -- 172.21.15.46:0/960237369 >> 172.21.15.46:0/960237369 conn(0x7f9a4c06bf40 msgr2=0x7f9a4c06c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:47.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.385+0000 7f9a5450f640 1 -- 172.21.15.46:0/960237369 shutdown_connections 2023-10-30T14:15:47.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.385+0000 7f9a5450f640 1 -- 172.21.15.46:0/960237369 wait complete. 2023-10-30T14:15:47.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a5450f640 1 Processor -- start 2023-10-30T14:15:47.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a5450f640 1 -- start start 2023-10-30T14:15:47.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a5450f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a4c0b1ea0 0x7f9a4c0b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:47.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a5450f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a4c0cc0b0 0x7f9a4c1420f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:47.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a5450f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a4c153c30 0x7f9a4c151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:47.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a5450f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c152b80 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a5450f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c1f3230 con 0x7f9a4c0cc0b0 2023-10-30T14:15:47.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a5450f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c1f23a0 con 0x7f9a4c153c30 2023-10-30T14:15:47.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a522ab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a4c0b1ea0 0x7f9a4c0b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:47.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a522ab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a4c0b1ea0 0x7f9a4c0b3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48120/0 (socket says 172.21.15.46:48120) 2023-10-30T14:15:47.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a522ab640 1 -- 172.21.15.46:0/2178380289 learned_addr learned my addr 172.21.15.46:0/2178380289 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:47.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a51aaa640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a4c0cc0b0 0x7f9a4c1420f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:47.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a52aac640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a4c153c30 0x7f9a4c151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:47.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a522ab640 1 -- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a4c153c30 msgr2=0x7f9a4c151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:47.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a522ab640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a4c153c30 0x7f9a4c151740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a522ab640 1 -- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a4c0cc0b0 msgr2=0x7f9a4c1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:47.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a522ab640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a4c0cc0b0 0x7f9a4c1420f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a522ab640 1 -- 172.21.15.46:0/2178380289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c009480 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a522ab640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a4c0b1ea0 0x7f9a4c0b3e80 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800c3c0 tx=0x7f9a4800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:47.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a42ffd640 1 -- 172.21.15.46:0/2178380289 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a480052d0 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a42ffd640 1 -- 172.21.15.46:0/2178380289 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a48005430 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a42ffd640 1 -- 172.21.15.46:0/2178380289 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a48003840 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a5450f640 1 -- 172.21.15.46:0/2178380289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4c1f2620 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a5450f640 1 -- 172.21.15.46:0/2178380289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4c1f2be0 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.389+0000 7f9a5450f640 1 -- 172.21.15.46:0/2178380289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4c04e170 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.393+0000 7f9a42ffd640 1 -- 172.21.15.46:0/2178380289 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9a480039a0 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.393+0000 7f9a42ffd640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9a300664a0 0x7f9a30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.393+0000 7f9a42ffd640 1 -- 172.21.15.46:0/2178380289 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f9a48083cb0 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.393+0000 7f9a51aaa640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9a300664a0 0x7f9a30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.393+0000 7f9a51aaa640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9a300664a0 0x7f9a30068960 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0b2000 tx=0x7f9a3c0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.397+0000 7f9a42ffd640 1 -- 172.21.15.46:0/2178380289 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9a48050680 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.589+0000 7f9a5450f640 1 -- 172.21.15.46:0/2178380289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a4c06ade0 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.589+0000 7f9a42ffd640 1 -- 172.21.15.46:0/2178380289 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f9a4c06ade0 con 0x7f9a4c0b1ea0 2023-10-30T14:15:47.594 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:47.594 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:32.954977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 224 2023-10-30T14:15:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.589+0000 7f9a5450f640 1 -- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9a300664a0 msgr2=0x7f9a30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.589+0000 7f9a5450f640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9a300664a0 0x7f9a30068960 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0b2000 tx=0x7f9a3c0aa070 comp rx=0 tx=0).stop 2023-10-30T14:15:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.589+0000 7f9a5450f640 1 -- 172.21.15.46:0/2178380289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a4c0b1ea0 msgr2=0x7f9a4c0b3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.589+0000 7f9a5450f640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a4c0b1ea0 0x7f9a4c0b3e80 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800c3c0 tx=0x7f9a4800c880 comp rx=0 tx=0).stop 2023-10-30T14:15:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.589+0000 7f9a5450f640 1 -- 172.21.15.46:0/2178380289 shutdown_connections 2023-10-30T14:15:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.589+0000 7f9a5450f640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9a300664a0 0x7f9a30068960 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.589+0000 7f9a5450f640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a4c153c30 0x7f9a4c151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.589+0000 7f9a5450f640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a4c0cc0b0 0x7f9a4c1420f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.589+0000 7f9a5450f640 1 --2- 172.21.15.46:0/2178380289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a4c0b1ea0 0x7f9a4c0b3e80 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:47.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.589+0000 7f9a5450f640 1 -- 172.21.15.46:0/2178380289 >> 172.21.15.46:0/2178380289 conn(0x7f9a4c06bf40 msgr2=0x7f9a4c157a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:47.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.593+0000 7f9a5450f640 1 -- 172.21.15.46:0/2178380289 shutdown_connections 2023-10-30T14:15:47.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:47.593+0000 7f9a5450f640 1 -- 172.21.15.46:0/2178380289 wait complete. 2023-10-30T14:15:47.615 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:15:49.616 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fecbb640 1 Processor -- start 2023-10-30T14:15:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fecbb640 1 -- start start 2023-10-30T14:15:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fecbb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3f8153890 0x7fe3f8159630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fecbb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe3f8069e90 0x7fe3f806a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fecbb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f80c54d0 con 0x7fe3f806ab20 2023-10-30T14:15:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fecbb640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f80c1cd0 con 0x7fe3f8069e90 2023-10-30T14:15:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fecbb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80c5a90 con 0x7fe3f8153890 2023-10-30T14:15:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fd4b8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe3f806ab20 0x7fe3f80c5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39342/0 (socket says 172.21.15.46:39342) 2023-10-30T14:15:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fd4b8640 1 -- 172.21.15.46:0/1406618346 learned_addr learned my addr 172.21.15.46:0/1406618346 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fe4ba640 1 --2- 172.21.15.46:0/1406618346 >> v2:172.21.15.154:3300/0 conn(0x7fe3f8069e90 0x7fe3f806a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fdcb9640 1 --2- 172.21.15.46:0/1406618346 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3f8153890 0x7fe3f8159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:49.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fccb7640 1 -- 172.21.15.46:0/1406618346 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023370814 0 0) 0x7fe3f80c54d0 con 0x7fe3f806ab20 2023-10-30T14:15:49.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fccb7640 1 -- 172.21.15.46:0/1406618346 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e4003530 con 0x7fe3f806ab20 2023-10-30T14:15:49.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fe4ba640 1 -- 172.21.15.46:0/1406618346 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3f8153890 msgr2=0x7fe3f8159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:49.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fe4ba640 1 --2- 172.21.15.46:0/1406618346 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3f8153890 0x7fe3f8159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:49.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fccb7640 1 -- 172.21.15.46:0/1406618346 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3891251595 0 0) 0x7fe3e4003530 con 0x7fe3f806ab20 2023-10-30T14:15:49.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fe4ba640 1 -- 172.21.15.46:0/1406618346 >> v1:172.21.15.46:6789/0 conn(0x7fe3f806ab20 legacy=0x7fe3f80c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:49.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.769+0000 7fe3fe4ba640 1 -- 172.21.15.46:0/1406618346 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f8159b70 con 0x7fe3f8069e90 2023-10-30T14:15:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.773+0000 7fe3fe4ba640 1 --2- 172.21.15.46:0/1406618346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3f8069e90 0x7fe3f806a320 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0009d90 tx=0x7fe3f001c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b9723e1910d841b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.773+0000 7fe3fccb7640 1 -- 172.21.15.46:0/1406618346 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8001680 con 0x7fe3f8069e90 2023-10-30T14:15:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.773+0000 7fe3fccb7640 1 -- 172.21.15.46:0/1406618346 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f00a5560 con 0x7fe3f8069e90 2023-10-30T14:15:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.773+0000 7fe3fccb7640 1 -- 172.21.15.46:0/1406618346 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f00a58f0 con 0x7fe3f8069e90 2023-10-30T14:15:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.773+0000 7fe3fecbb640 1 -- 172.21.15.46:0/1406618346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3f8069e90 msgr2=0x7fe3f806a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.773+0000 7fe3fecbb640 1 --2- 172.21.15.46:0/1406618346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3f8069e90 0x7fe3f806a320 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0009d90 tx=0x7fe3f001c870 comp rx=0 tx=0).stop 2023-10-30T14:15:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.773+0000 7fe3fecbb640 1 -- 172.21.15.46:0/1406618346 shutdown_connections 2023-10-30T14:15:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.773+0000 7fe3fecbb640 1 --2- 172.21.15.46:0/1406618346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3f8069e90 0x7fe3f806a320 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.773+0000 7fe3fecbb640 1 --2- 172.21.15.46:0/1406618346 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3f8153890 0x7fe3f8159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.773+0000 7fe3fecbb640 1 -- 172.21.15.46:0/1406618346 >> 172.21.15.46:0/1406618346 conn(0x7fe3f80c75d0 msgr2=0x7fe3f80c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.773+0000 7fe3fecbb640 1 -- 172.21.15.46:0/1406618346 shutdown_connections 2023-10-30T14:15:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.773+0000 7fe3fecbb640 1 -- 172.21.15.46:0/1406618346 wait complete. 2023-10-30T14:15:49.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fecbb640 1 Processor -- start 2023-10-30T14:15:49.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fecbb640 1 -- start start 2023-10-30T14:15:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fecbb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f8069e90 0x7fe3f814f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fecbb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3f806ab20 0x7fe3f8146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fecbb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3f8153890 0x7fe3f8146b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fecbb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f8147670 con 0x7fe3f8069e90 2023-10-30T14:15:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fecbb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f8152400 con 0x7fe3f8153890 2023-10-30T14:15:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fecbb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f8152120 con 0x7fe3f806ab20 2023-10-30T14:15:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fdcb9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f8069e90 0x7fe3f814f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fdcb9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f8069e90 0x7fe3f814f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48122/0 (socket says 172.21.15.46:48122) 2023-10-30T14:15:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fdcb9640 1 -- 172.21.15.46:0/4201122732 learned_addr learned my addr 172.21.15.46:0/4201122732 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fd4b8640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3f806ab20 0x7fe3f8146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fe4ba640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3f8153890 0x7fe3f8146b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fdcb9640 1 -- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3f806ab20 msgr2=0x7fe3f8146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fdcb9640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3f806ab20 0x7fe3f8146640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fdcb9640 1 -- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3f8153890 msgr2=0x7fe3f8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fdcb9640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3f8153890 0x7fe3f8146b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fdcb9640 1 -- 172.21.15.46:0/4201122732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f0009a70 con 0x7fe3f8069e90 2023-10-30T14:15:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fe4ba640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3f8153890 0x7fe3f8146b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fdcb9640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f8069e90 0x7fe3f814f0e0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400e9b0 tx=0x7fe3f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3ee7fc640 1 -- 172.21.15.46:0/4201122732 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4009670 con 0x7fe3f8069e90 2023-10-30T14:15:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3ee7fc640 1 -- 172.21.15.46:0/4201122732 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f400c9a0 con 0x7fe3f8069e90 2023-10-30T14:15:49.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3ee7fc640 1 -- 172.21.15.46:0/4201122732 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f400cd10 con 0x7fe3f8069e90 2023-10-30T14:15:49.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fecbb640 1 -- 172.21.15.46:0/4201122732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f81f2cc0 con 0x7fe3f8069e90 2023-10-30T14:15:49.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fecbb640 1 -- 172.21.15.46:0/4201122732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f81f3280 con 0x7fe3f8069e90 2023-10-30T14:15:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.777+0000 7fe3fecbb640 1 -- 172.21.15.46:0/4201122732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f804e170 con 0x7fe3f8069e90 2023-10-30T14:15:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.781+0000 7fe3ee7fc640 1 -- 172.21.15.46:0/4201122732 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe3f4013470 con 0x7fe3f8069e90 2023-10-30T14:15:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.781+0000 7fe3ee7fc640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3dc0664d0 0x7fe3dc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.781+0000 7fe3ee7fc640 1 -- 172.21.15.46:0/4201122732 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7fe3f4084e80 con 0x7fe3f8069e90 2023-10-30T14:15:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.781+0000 7fe3fd4b8640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3dc0664d0 0x7fe3dc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.781+0000 7fe3fd4b8640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3dc0664d0 0x7fe3dc068990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8001ba0 tx=0x7fe3e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.785+0000 7fe3ee7fc640 1 -- 172.21.15.46:0/4201122732 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe3f40518e0 con 0x7fe3f8069e90 2023-10-30T14:15:49.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.973+0000 7fe3fecbb640 1 -- 172.21.15.46:0/4201122732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3f80ca0d0 con 0x7fe3f8069e90 2023-10-30T14:15:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.973+0000 7fe3ee7fc640 1 -- 172.21.15.46:0/4201122732 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fe3f4021020 con 0x7fe3f8069e90 2023-10-30T14:15:49.978 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:49.979 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:32.954977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 224 2023-10-30T14:15:49.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.977+0000 7fe3fecbb640 1 -- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3dc0664d0 msgr2=0x7fe3dc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:49.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.977+0000 7fe3fecbb640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3dc0664d0 0x7fe3dc068990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8001ba0 tx=0x7fe3e8009040 comp rx=0 tx=0).stop 2023-10-30T14:15:49.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.977+0000 7fe3fecbb640 1 -- 172.21.15.46:0/4201122732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f8069e90 msgr2=0x7fe3f814f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:49.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.977+0000 7fe3fecbb640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f8069e90 0x7fe3f814f0e0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400e9b0 tx=0x7fe3f400ee70 comp rx=0 tx=0).stop 2023-10-30T14:15:49.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.977+0000 7fe3fecbb640 1 -- 172.21.15.46:0/4201122732 shutdown_connections 2023-10-30T14:15:49.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.977+0000 7fe3fecbb640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe3dc0664d0 0x7fe3dc068990 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:49.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.977+0000 7fe3fecbb640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3f8153890 0x7fe3f8146b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:49.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.977+0000 7fe3fecbb640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3f806ab20 0x7fe3f8146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:49.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.977+0000 7fe3fecbb640 1 --2- 172.21.15.46:0/4201122732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3f8069e90 0x7fe3f814f0e0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:49.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.977+0000 7fe3fecbb640 1 -- 172.21.15.46:0/4201122732 >> 172.21.15.46:0/4201122732 conn(0x7fe3f80c75d0 msgr2=0x7fe3f80c4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:49.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.977+0000 7fe3fecbb640 1 -- 172.21.15.46:0/4201122732 shutdown_connections 2023-10-30T14:15:49.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:49.977+0000 7fe3fecbb640 1 -- 172.21.15.46:0/4201122732 wait complete. 2023-10-30T14:15:49.999 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:15:52.000 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:52.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f3549b63640 1 Processor -- start 2023-10-30T14:15:52.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f3549b63640 1 -- start start 2023-10-30T14:15:52.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f3549b63640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f35441538f0 0x7f3544159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:52.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f3549b63640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354415cd50 0x7f35440c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:52.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f3549b63640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354415d300 con 0x7f35440cefa0 2023-10-30T14:15:52.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f3549b63640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f354415d5e0 con 0x7f35441538f0 2023-10-30T14:15:52.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f3549b63640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f354415d8c0 con 0x7f354415cd50 2023-10-30T14:15:52.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f3549362640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f35440cefa0 0x7f35440cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39346/0 (socket says 172.21.15.46:39346) 2023-10-30T14:15:52.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f3549362640 1 -- 172.21.15.46:0/260519053 learned_addr learned my addr 172.21.15.46:0/260519053 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f3543fff640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354415cd50 0x7f35440c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f3548b61640 1 --2- 172.21.15.46:0/260519053 >> v2:172.21.15.154:3300/0 conn(0x7f35441538f0 0x7f3544159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483525126 0 0) 0x7f354415d300 con 0x7f35440cefa0 2023-10-30T14:15:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.153+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3534003530 con 0x7f35440cefa0 2023-10-30T14:15:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3593896075 0 0) 0x7f3534003530 con 0x7f35440cefa0 2023-10-30T14:15:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354415cd50 msgr2=0x7f35440c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354415cd50 0x7f35440c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 >> v2:172.21.15.154:3300/0 conn(0x7f35441538f0 msgr2=0x7f3544159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 --2- 172.21.15.46:0/260519053 >> v2:172.21.15.154:3300/0 conn(0x7f35441538f0 0x7f3544159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354415dba0 con 0x7f35440cefa0 2023-10-30T14:15:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f353c001a00 con 0x7f35440cefa0 2023-10-30T14:15:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 >> v1:172.21.15.46:6789/0 conn(0x7f35440cefa0 legacy=0x7f35440cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354415cd50 0x7f3534003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534003f00 0x7f35340047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3534004d10 0x7f3534005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354415d5e0 con 0x7f3534003f00 2023-10-30T14:15:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354415d8c0 con 0x7f3534004d10 2023-10-30T14:15:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353400b9c0 con 0x7f354415cd50 2023-10-30T14:15:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f3543fff640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534003f00 0x7f35340047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f3548b61640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354415cd50 0x7f3534003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f3549362640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3534004d10 0x7f3534005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f3543fff640 1 -- 172.21.15.46:0/260519053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354415cd50 msgr2=0x7f3534003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f3543fff640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354415cd50 0x7f3534003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f3543fff640 1 -- 172.21.15.46:0/260519053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3534004d10 msgr2=0x7f3534005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f3543fff640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3534004d10 0x7f3534005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f3543fff640 1 -- 172.21.15.46:0/260519053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3534003530 con 0x7f3534003f00 2023-10-30T14:15:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f3549362640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3534004d10 0x7f3534005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:15:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.157+0000 7f3543fff640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534003f00 0x7f35340047d0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f3530009ae0 tx=0x7f353001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81de0efc2bdd9999 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35300a5640 con 0x7f3534003f00 2023-10-30T14:15:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35300a57a0 con 0x7f3534003f00 2023-10-30T14:15:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f35437fe640 1 -- 172.21.15.46:0/260519053 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35300a5b50 con 0x7f3534003f00 2023-10-30T14:15:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 -- 172.21.15.46:0/260519053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534003f00 msgr2=0x7f35340047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 --2- 172.21.15.46:0/260519053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534003f00 0x7f35340047d0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f3530009ae0 tx=0x7f353001ca50 comp rx=0 tx=0).stop 2023-10-30T14:15:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549362640 1 -- 172.21.15.46:0/260519053 reap_dead start 2023-10-30T14:15:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 -- 172.21.15.46:0/260519053 shutdown_connections 2023-10-30T14:15:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 -- 172.21.15.46:0/260519053 >> 172.21.15.46:0/260519053 conn(0x7f354406a640 msgr2=0x7f3544065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 -- 172.21.15.46:0/260519053 shutdown_connections 2023-10-30T14:15:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 -- 172.21.15.46:0/260519053 wait complete. 2023-10-30T14:15:52.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 Processor -- start 2023-10-30T14:15:52.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 -- start start 2023-10-30T14:15:52.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534003f00 0x7f354415c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:52.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354414fa40 0x7f35440cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:52.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35440b2f50 0x7f35440ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354414eea0 con 0x7f3534003f00 2023-10-30T14:15:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35441eebd0 con 0x7f35440b2f50 2023-10-30T14:15:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3549b63640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35440cec10 con 0x7f354414fa40 2023-10-30T14:15:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3548b61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534003f00 0x7f354415c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3548b61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534003f00 0x7f354415c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48130/0 (socket says 172.21.15.46:48130) 2023-10-30T14:15:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.161+0000 7f3548b61640 1 -- 172.21.15.46:0/1347067665 learned_addr learned my addr 172.21.15.46:0/1347067665 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f3543fff640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354414fa40 0x7f35440cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f3549362640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35440b2f50 0x7f35440ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f3548b61640 1 -- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354414fa40 msgr2=0x7f35440cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f3548b61640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354414fa40 0x7f35440cdf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f3548b61640 1 -- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35440b2f50 msgr2=0x7f35440ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f3548b61640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35440b2f50 0x7f35440ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f3548b61640 1 -- 172.21.15.46:0/1347067665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354414e4e0 con 0x7f3534003f00 2023-10-30T14:15:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f3549362640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35440b2f50 0x7f35440ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:52.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f3548b61640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534003f00 0x7f354415c240 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f3538011aa0 tx=0x7f3538011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:52.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f35417fa640 1 -- 172.21.15.46:0/1347067665 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353800fb80 con 0x7f3534003f00 2023-10-30T14:15:52.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f35417fa640 1 -- 172.21.15.46:0/1347067665 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353800fce0 con 0x7f3534003f00 2023-10-30T14:15:52.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f3549b63640 1 -- 172.21.15.46:0/1347067665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3544148730 con 0x7f3534003f00 2023-10-30T14:15:52.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f35417fa640 1 -- 172.21.15.46:0/1347067665 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3538014680 con 0x7f3534003f00 2023-10-30T14:15:52.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f3549b63640 1 -- 172.21.15.46:0/1347067665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f354414dc40 con 0x7f3534003f00 2023-10-30T14:15:52.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f3549b63640 1 -- 172.21.15.46:0/1347067665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354404e170 con 0x7f3534003f00 2023-10-30T14:15:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f35417fa640 1 -- 172.21.15.46:0/1347067665 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f35380147e0 con 0x7f3534003f00 2023-10-30T14:15:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f35417fa640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35280664d0 0x7f3528068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.165+0000 7f35417fa640 1 -- 172.21.15.46:0/1347067665 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f35380850f0 con 0x7f3534003f00 2023-10-30T14:15:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.169+0000 7f3543fff640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35280664d0 0x7f3528068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.169+0000 7f3543fff640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35280664d0 0x7f3528068990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f35300af000 tx=0x7f35300046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.169+0000 7f35417fa640 1 -- 172.21.15.46:0/1347067665 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3538051bd0 con 0x7f3534003f00 2023-10-30T14:15:52.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 -- 172.21.15.46:0/1347067665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3544153120 con 0x7f3534003f00 2023-10-30T14:15:52.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f35417fa640 1 -- 172.21.15.46:0/1347067665 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f35380551f0 con 0x7f3534003f00 2023-10-30T14:15:52.369 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:52.369 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:32.954977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:52.370 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 224 2023-10-30T14:15:52.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 -- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35280664d0 msgr2=0x7f3528068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:52.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35280664d0 0x7f3528068990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f35300af000 tx=0x7f35300046d0 comp rx=0 tx=0).stop 2023-10-30T14:15:52.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 -- 172.21.15.46:0/1347067665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534003f00 msgr2=0x7f354415c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:52.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534003f00 0x7f354415c240 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f3538011aa0 tx=0x7f3538011f60 comp rx=0 tx=0).stop 2023-10-30T14:15:52.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 -- 172.21.15.46:0/1347067665 shutdown_connections 2023-10-30T14:15:52.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35280664d0 0x7f3528068990 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:52.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35440b2f50 0x7f35440ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:52.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f354414fa40 0x7f35440cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:52.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 --2- 172.21.15.46:0/1347067665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534003f00 0x7f354415c240 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:52.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 -- 172.21.15.46:0/1347067665 >> 172.21.15.46:0/1347067665 conn(0x7f354406a640 msgr2=0x7f35440c52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:52.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 -- 172.21.15.46:0/1347067665 shutdown_connections 2023-10-30T14:15:52.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:52.365+0000 7f3549b63640 1 -- 172.21.15.46:0/1347067665 wait complete. 2023-10-30T14:15:52.391 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:15:54.392 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.541+0000 7f6005654640 1 Processor -- start 2023-10-30T14:15:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.541+0000 7f6005654640 1 -- start start 2023-10-30T14:15:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.541+0000 7f6005654640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60001538a0 0x7f600015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.541+0000 7f6005654640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60000cf160 0x7f60000cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.541+0000 7f6005654640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60000cb800 con 0x7f60000ca0e0 2023-10-30T14:15:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.541+0000 7f6005654640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f600015dc30 con 0x7f60001538a0 2023-10-30T14:15:54.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.541+0000 7f6005654640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60000cb330 con 0x7f60000cf160 2023-10-30T14:15:54.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.545+0000 7f5fff7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60000cf160 0x7f60000cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:54.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.545+0000 7f5ffeffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60001538a0 0x7f600015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:54.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.545+0000 7f5ff7fff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f60000ca0e0 0x7f600015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:39190/0 (socket says 172.21.15.46:39190) 2023-10-30T14:15:54.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.545+0000 7f5ff7fff640 1 -- 172.21.15.46:0/2280923208 learned_addr learned my addr 172.21.15.46:0/2280923208 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.545+0000 7f5ffe7fc640 1 -- 172.21.15.46:0/2280923208 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364277315 0 0) 0x7f60000cb800 con 0x7f60000ca0e0 2023-10-30T14:15:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.545+0000 7f5ffe7fc640 1 -- 172.21.15.46:0/2280923208 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fec003530 con 0x7f60000ca0e0 2023-10-30T14:15:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.545+0000 7f5fff7fe640 1 -- 172.21.15.46:0/2280923208 >> v2:172.21.15.154:3300/0 conn(0x7f60001538a0 msgr2=0x7f600015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.545+0000 7f5fff7fe640 1 --2- 172.21.15.46:0/2280923208 >> v2:172.21.15.154:3300/0 conn(0x7f60001538a0 0x7f600015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.545+0000 7f5fff7fe640 1 -- 172.21.15.46:0/2280923208 >> v1:172.21.15.154:6789/0 conn(0x7f60000ca0e0 legacy=0x7f600015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.545+0000 7f5fff7fe640 1 -- 172.21.15.46:0/2280923208 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60000cad70 con 0x7f60000cf160 2023-10-30T14:15:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.545+0000 7f5fff7fe640 1 --2- 172.21.15.46:0/2280923208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60000cf160 0x7f60000cf590 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8009d90 tx=0x7f5ff801f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2271c58a2cf95e85 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f5ffe7fc640 1 -- 172.21.15.46:0/2280923208 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff80af600 con 0x7f60000cf160 2023-10-30T14:15:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f5ffe7fc640 1 -- 172.21.15.46:0/2280923208 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff80ab050 con 0x7f60000cf160 2023-10-30T14:15:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f5ffe7fc640 1 -- 172.21.15.46:0/2280923208 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff80af960 con 0x7f60000cf160 2023-10-30T14:15:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 -- 172.21.15.46:0/2280923208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60000cf160 msgr2=0x7f60000cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 --2- 172.21.15.46:0/2280923208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60000cf160 0x7f60000cf590 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8009d90 tx=0x7f5ff801f130 comp rx=0 tx=0).stop 2023-10-30T14:15:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 -- 172.21.15.46:0/2280923208 shutdown_connections 2023-10-30T14:15:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 --2- 172.21.15.46:0/2280923208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60000cf160 0x7f60000cf590 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 --2- 172.21.15.46:0/2280923208 >> v2:172.21.15.154:3300/0 conn(0x7f60001538a0 0x7f600015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 -- 172.21.15.46:0/2280923208 >> 172.21.15.46:0/2280923208 conn(0x7f600006bfe0 msgr2=0x7f600006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 -- 172.21.15.46:0/2280923208 shutdown_connections 2023-10-30T14:15:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 -- 172.21.15.46:0/2280923208 wait complete. 2023-10-30T14:15:54.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 Processor -- start 2023-10-30T14:15:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 -- start start 2023-10-30T14:15:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60000ca0e0 0x7f6000145f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60000cf160 0x7f60001ef210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60001538a0 0x7f6000147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60001f2f70 con 0x7f60000ca0e0 2023-10-30T14:15:54.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60001464a0 con 0x7f60001538a0 2023-10-30T14:15:54.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.549+0000 7f6005654640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6000158c70 con 0x7f60000cf160 2023-10-30T14:15:54.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ffeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60000ca0e0 0x7f6000145f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:54.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ffeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60000ca0e0 0x7f6000145f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48146/0 (socket says 172.21.15.46:48146) 2023-10-30T14:15:54.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ffeffd640 1 -- 172.21.15.46:0/4182847753 learned_addr learned my addr 172.21.15.46:0/4182847753 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:54.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ff7fff640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60000cf160 0x7f60001ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:54.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5fff7fe640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60001538a0 0x7f6000147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:54.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ffeffd640 1 -- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60000cf160 msgr2=0x7f60001ef210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:54.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ffeffd640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60000cf160 0x7f60001ef210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:54.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ffeffd640 1 -- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60001538a0 msgr2=0x7f6000147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:54.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ffeffd640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60001538a0 0x7f6000147bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:54.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ffeffd640 1 -- 172.21.15.46:0/4182847753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff8009a70 con 0x7f60000ca0e0 2023-10-30T14:15:54.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ffeffd640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60000ca0e0 0x7f6000145f60 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f5ff000e870 tx=0x7f5ff000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:54.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ff77fe640 1 -- 172.21.15.46:0/4182847753 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0009510 con 0x7f60000ca0e0 2023-10-30T14:15:54.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ff77fe640 1 -- 172.21.15.46:0/4182847753 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff000eec0 con 0x7f60000ca0e0 2023-10-30T14:15:54.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f6005654640 1 -- 172.21.15.46:0/4182847753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60000c72f0 con 0x7f60000ca0e0 2023-10-30T14:15:54.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ff77fe640 1 -- 172.21.15.46:0/4182847753 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff000cb40 con 0x7f60000ca0e0 2023-10-30T14:15:54.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f6005654640 1 -- 172.21.15.46:0/4182847753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60001f28b0 con 0x7f60000ca0e0 2023-10-30T14:15:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f6005654640 1 -- 172.21.15.46:0/4182847753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60000b2a50 con 0x7f60000ca0e0 2023-10-30T14:15:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ff77fe640 1 -- 172.21.15.46:0/4182847753 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5ff000cca0 con 0x7f60000ca0e0 2023-10-30T14:15:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.553+0000 7f5ff77fe640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5fe0066520 0x7f5fe00689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:54.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.557+0000 7f5ff77fe640 1 -- 172.21.15.46:0/4182847753 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f5ff0084ac0 con 0x7f60000ca0e0 2023-10-30T14:15:54.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.557+0000 7f5ff7fff640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5fe0066520 0x7f5fe00689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:54.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.557+0000 7f5ff7fff640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5fe0066520 0x7f5fe00689e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8007fd0 tx=0x7f5fe8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:54.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.561+0000 7f5ff77fe640 1 -- 172.21.15.46:0/4182847753 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5ff00516e0 con 0x7f60000ca0e0 2023-10-30T14:15:54.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 -- 172.21.15.46:0/4182847753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f600004e170 con 0x7f60000ca0e0 2023-10-30T14:15:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f5ff77fe640 1 -- 172.21.15.46:0/4182847753 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f5ff0021090 con 0x7f60000ca0e0 2023-10-30T14:15:54.756 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:54.757 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:32.954977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 224 2023-10-30T14:15:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 -- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5fe0066520 msgr2=0x7f5fe00689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5fe0066520 0x7f5fe00689e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8007fd0 tx=0x7f5fe8009040 comp rx=0 tx=0).stop 2023-10-30T14:15:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 -- 172.21.15.46:0/4182847753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60000ca0e0 msgr2=0x7f6000145f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60000ca0e0 0x7f6000145f60 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f5ff000e870 tx=0x7f5ff000ec70 comp rx=0 tx=0).stop 2023-10-30T14:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 -- 172.21.15.46:0/4182847753 shutdown_connections 2023-10-30T14:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5fe0066520 0x7f5fe00689e0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60001538a0 0x7f6000147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60000cf160 0x7f60001ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 --2- 172.21.15.46:0/4182847753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60000ca0e0 0x7f6000145f60 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 -- 172.21.15.46:0/4182847753 >> 172.21.15.46:0/4182847753 conn(0x7f600006bfe0 msgr2=0x7f60000c9620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 -- 172.21.15.46:0/4182847753 shutdown_connections 2023-10-30T14:15:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:54.753+0000 7f6005654640 1 -- 172.21.15.46:0/4182847753 wait complete. 2023-10-30T14:15:54.778 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:15:56.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:57.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:56.929+0000 7ff44ca46640 1 Processor -- start 2023-10-30T14:15:57.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:56.929+0000 7ff44ca46640 1 -- start start 2023-10-30T14:15:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:56.929+0000 7ff44ca46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4480720e0 0x7ff4480c6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:56.929+0000 7ff44ca46640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff44816a510 0x7ff4480bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:56.929+0000 7ff44ca46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4480724c0 con 0x7ff4480c0400 2023-10-30T14:15:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:56.929+0000 7ff44ca46640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff44806fe90 con 0x7ff44816a510 2023-10-30T14:15:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:56.929+0000 7ff44ca46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481666a0 con 0x7ff4480720e0 2023-10-30T14:15:57.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:56.929+0000 7ff446d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff4480c0400 0x7ff4480c5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33688/0 (socket says 172.21.15.46:33688) 2023-10-30T14:15:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:56.929+0000 7ff446d9d640 1 -- 172.21.15.46:0/2534076470 learned_addr learned my addr 172.21.15.46:0/2534076470 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:56.929+0000 7ff43ffff640 1 --2- 172.21.15.46:0/2534076470 >> v2:172.21.15.154:3300/0 conn(0x7ff44816a510 0x7ff4480bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:56.929+0000 7ff44659c640 1 --2- 172.21.15.46:0/2534076470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4480720e0 0x7ff4480c6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:57.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:56.929+0000 7ff445d9b640 1 -- 172.21.15.46:0/2534076470 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471678205 0 0) 0x7ff4480724c0 con 0x7ff4480c0400 2023-10-30T14:15:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.113+0000 7ff445d9b640 1 -- 172.21.15.46:0/2534076470 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff434003510 con 0x7ff4480c0400 2023-10-30T14:15:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.113+0000 7ff43ffff640 1 -- 172.21.15.46:0/2534076470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4480720e0 msgr2=0x7ff4480c6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.113+0000 7ff43ffff640 1 --2- 172.21.15.46:0/2534076470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4480720e0 0x7ff4480c6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.113+0000 7ff43ffff640 1 -- 172.21.15.46:0/2534076470 >> v1:172.21.15.46:6789/0 conn(0x7ff4480c0400 legacy=0x7ff4480c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.113+0000 7ff43ffff640 1 -- 172.21.15.46:0/2534076470 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff448166980 con 0x7ff44816a510 2023-10-30T14:15:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.113+0000 7ff44659c640 1 --2- 172.21.15.46:0/2534076470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4480720e0 0x7ff4480c6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:15:57.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.117+0000 7ff43ffff640 1 --2- 172.21.15.46:0/2534076470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff44816a510 0x7ff4480bfec0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff430008840 tx=0x7ff43001c570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b68ef2d6eb98ff99 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.117+0000 7ff445d9b640 1 -- 172.21.15.46:0/2534076470 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4300a9650 con 0x7ff44816a510 2023-10-30T14:15:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.117+0000 7ff445d9b640 1 -- 172.21.15.46:0/2534076470 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4300a97b0 con 0x7ff44816a510 2023-10-30T14:15:57.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.117+0000 7ff445d9b640 1 -- 172.21.15.46:0/2534076470 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4300a9b40 con 0x7ff44816a510 2023-10-30T14:15:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.117+0000 7ff44ca46640 1 -- 172.21.15.46:0/2534076470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff44816a510 msgr2=0x7ff4480bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.117+0000 7ff44ca46640 1 --2- 172.21.15.46:0/2534076470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff44816a510 0x7ff4480bfec0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff430008840 tx=0x7ff43001c570 comp rx=0 tx=0).stop 2023-10-30T14:15:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.117+0000 7ff44ca46640 1 -- 172.21.15.46:0/2534076470 shutdown_connections 2023-10-30T14:15:57.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.117+0000 7ff44ca46640 1 --2- 172.21.15.46:0/2534076470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff44816a510 0x7ff4480bfec0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.117+0000 7ff44ca46640 1 --2- 172.21.15.46:0/2534076470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4480720e0 0x7ff4480c6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.117+0000 7ff44ca46640 1 -- 172.21.15.46:0/2534076470 >> 172.21.15.46:0/2534076470 conn(0x7ff4480c7910 msgr2=0x7ff4480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.117+0000 7ff44ca46640 1 -- 172.21.15.46:0/2534076470 shutdown_connections 2023-10-30T14:15:57.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.117+0000 7ff44ca46640 1 -- 172.21.15.46:0/2534076470 wait complete. 2023-10-30T14:15:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff44ca46640 1 Processor -- start 2023-10-30T14:15:57.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff44ca46640 1 -- start start 2023-10-30T14:15:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff44ca46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4480720e0 0x7ff4480c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff44ca46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480c0400 0x7ff4480c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff44ca46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff44816a510 0x7ff448158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:57.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff44ca46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff448167b30 con 0x7ff4480c0400 2023-10-30T14:15:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff44ca46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff448167870 con 0x7ff4480720e0 2023-10-30T14:15:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff44ca46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481fef70 con 0x7ff44816a510 2023-10-30T14:15:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff43ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480c0400 0x7ff4480c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff43ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480c0400 0x7ff4480c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56606/0 (socket says 172.21.15.46:56606) 2023-10-30T14:15:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff43ffff640 1 -- 172.21.15.46:0/3119525174 learned_addr learned my addr 172.21.15.46:0/3119525174 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff446d9d640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff44816a510 0x7ff448158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff44659c640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4480720e0 0x7ff4480c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff43ffff640 1 -- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff44816a510 msgr2=0x7ff448158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff43ffff640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff44816a510 0x7ff448158640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff43ffff640 1 -- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4480720e0 msgr2=0x7ff4480c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff43ffff640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4480720e0 0x7ff4480c92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff43ffff640 1 -- 172.21.15.46:0/3119525174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43800d8c0 con 0x7ff4480c0400 2023-10-30T14:15:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff43ffff640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480c0400 0x7ff4480c97f0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7ff4300ab040 tx=0x7ff430008180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff43effd640 1 -- 172.21.15.46:0/3119525174 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4300be680 con 0x7ff4480c0400 2023-10-30T14:15:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff43effd640 1 -- 172.21.15.46:0/3119525174 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4300b7070 con 0x7ff4480c0400 2023-10-30T14:15:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff43effd640 1 -- 172.21.15.46:0/3119525174 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4300be9e0 con 0x7ff4480c0400 2023-10-30T14:15:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff44ca46640 1 -- 172.21.15.46:0/3119525174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4300084a0 con 0x7ff4480c0400 2023-10-30T14:15:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.121+0000 7ff44ca46640 1 -- 172.21.15.46:0/3119525174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4480c8d90 con 0x7ff4480c0400 2023-10-30T14:15:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.125+0000 7ff44ca46640 1 -- 172.21.15.46:0/3119525174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4480b0c70 con 0x7ff4480c0400 2023-10-30T14:15:57.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.125+0000 7ff43effd640 1 -- 172.21.15.46:0/3119525174 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4300bed50 con 0x7ff4480c0400 2023-10-30T14:15:57.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.125+0000 7ff43effd640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff4280664a0 0x7ff428068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.125+0000 7ff43effd640 1 -- 172.21.15.46:0/3119525174 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7ff4300b9070 con 0x7ff4480c0400 2023-10-30T14:15:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.125+0000 7ff44659c640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff4280664a0 0x7ff428068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.125+0000 7ff44659c640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff4280664a0 0x7ff428068960 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff438015340 tx=0x7ff43800f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:57.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.129+0000 7ff43effd640 1 -- 172.21.15.46:0/3119525174 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4480b0c70 con 0x7ff4480c0400 2023-10-30T14:15:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 -- 172.21.15.46:0/3119525174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4480b0c70 con 0x7ff4480c0400 2023-10-30T14:15:57.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff43effd640 1 -- 172.21.15.46:0/3119525174 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+6523 (secure 0 0 0) 0x7ff4480b0c70 con 0x7ff4480c0400 2023-10-30T14:15:57.329 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:57.329 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:32.954977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 224 2023-10-30T14:15:57.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 -- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff4280664a0 msgr2=0x7ff428068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:57.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff4280664a0 0x7ff428068960 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff438015340 tx=0x7ff43800f040 comp rx=0 tx=0).stop 2023-10-30T14:15:57.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 -- 172.21.15.46:0/3119525174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480c0400 msgr2=0x7ff4480c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:57.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480c0400 0x7ff4480c97f0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7ff4300ab040 tx=0x7ff430008180 comp rx=0 tx=0).stop 2023-10-30T14:15:57.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 -- 172.21.15.46:0/3119525174 shutdown_connections 2023-10-30T14:15:57.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff4280664a0 0x7ff428068960 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:57.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff44816a510 0x7ff448158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:57.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480c0400 0x7ff4480c97f0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:57.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 --2- 172.21.15.46:0/3119525174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4480720e0 0x7ff4480c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:57.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 -- 172.21.15.46:0/3119525174 >> 172.21.15.46:0/3119525174 conn(0x7ff4480c7910 msgr2=0x7ff44806b590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:57.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 -- 172.21.15.46:0/3119525174 shutdown_connections 2023-10-30T14:15:57.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:57.325+0000 7ff44ca46640 1 -- 172.21.15.46:0/3119525174 wait complete. 2023-10-30T14:15:57.350 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:15:58.335 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:15:58.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:15:59.351 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:15:59.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.501+0000 7f151f5f5640 1 Processor -- start 2023-10-30T14:15:59.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.501+0000 7f151f5f5640 1 -- start start 2023-10-30T14:15:59.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.501+0000 7f151f5f5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1518070740 0x7f15180c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:59.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.501+0000 7f151f5f5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f151815e700 0x7f151806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:59.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.501+0000 7f151f5f5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15180c5810 con 0x7f15181580e0 2023-10-30T14:15:59.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.501+0000 7f151f5f5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151815eb90 con 0x7f1518070740 2023-10-30T14:15:59.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.501+0000 7f151f5f5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15180c6bb0 con 0x7f151815e700 2023-10-30T14:15:59.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.505+0000 7f151db92640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f151815e700 0x7f151806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:59.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.505+0000 7f151cb90640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1518070740 0x7f15180c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:59.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.505+0000 7f151d391640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f15181580e0 0x7f1518069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:41236/0 (socket says 172.21.15.46:41236) 2023-10-30T14:15:59.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.505+0000 7f151d391640 1 -- 172.21.15.46:0/1077170601 learned_addr learned my addr 172.21.15.46:0/1077170601 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:15:59.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.505+0000 7f151db92640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f151815e700 0x7f151806a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:55706/0 (socket says 172.21.15.46:55706) 2023-10-30T14:15:59.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.505+0000 7f151cb90640 1 -- 172.21.15.46:0/1077170601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f151815e700 msgr2=0x7f151806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:15:59.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.505+0000 7f151cb90640 1 --2- 172.21.15.46:0/1077170601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f151815e700 0x7f151806a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:59.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.505+0000 7f151cb90640 1 -- 172.21.15.46:0/1077170601 >> v1:172.21.15.154:6789/0 conn(0x7f15181580e0 legacy=0x7f1518069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:59.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.505+0000 7f151cb90640 1 -- 172.21.15.46:0/1077170601 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15180c6e70 con 0x7f1518070740 2023-10-30T14:15:59.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151cb90640 1 --2- 172.21.15.46:0/1077170601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518070740 0x7f15180c6540 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f15080097c0 tx=0x7f150801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b440cd7743fc37a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:59.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f150ffff640 1 -- 172.21.15.46:0/1077170601 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15080ab600 con 0x7f1518070740 2023-10-30T14:15:59.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f150ffff640 1 -- 172.21.15.46:0/1077170601 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15080a7050 con 0x7f1518070740 2023-10-30T14:15:59.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f150ffff640 1 -- 172.21.15.46:0/1077170601 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15080ab980 con 0x7f1518070740 2023-10-30T14:15:59.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 -- 172.21.15.46:0/1077170601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518070740 msgr2=0x7f15180c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:59.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 --2- 172.21.15.46:0/1077170601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518070740 0x7f15180c6540 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f15080097c0 tx=0x7f150801c480 comp rx=0 tx=0).stop 2023-10-30T14:15:59.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 -- 172.21.15.46:0/1077170601 shutdown_connections 2023-10-30T14:15:59.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 --2- 172.21.15.46:0/1077170601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f151815e700 0x7f151806a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 --2- 172.21.15.46:0/1077170601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518070740 0x7f15180c6540 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 -- 172.21.15.46:0/1077170601 >> 172.21.15.46:0/1077170601 conn(0x7f15180c7910 msgr2=0x7f15180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 -- 172.21.15.46:0/1077170601 shutdown_connections 2023-10-30T14:15:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 -- 172.21.15.46:0/1077170601 wait complete. 2023-10-30T14:15:59.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 Processor -- start 2023-10-30T14:15:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 -- start start 2023-10-30T14:15:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518070740 0x7f15180c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15181580e0 0x7f15180ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f151815e700 0x7f1518160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15180c1a80 con 0x7f151815e700 2023-10-30T14:15:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15180c0f10 con 0x7f1518070740 2023-10-30T14:15:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151f5f5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15180c11e0 con 0x7f15181580e0 2023-10-30T14:15:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.509+0000 7f151db92640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f151815e700 0x7f1518160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151db92640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f151815e700 0x7f1518160bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56620/0 (socket says 172.21.15.46:56620) 2023-10-30T14:15:59.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151db92640 1 -- 172.21.15.46:0/1585670421 learned_addr learned my addr 172.21.15.46:0/1585670421 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:15:59.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151d391640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518070740 0x7f15180c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:59.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151cb90640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15181580e0 0x7f15180ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:59.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151db92640 1 -- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15181580e0 msgr2=0x7f15180ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:59.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151db92640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15181580e0 0x7f15180ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:59.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151db92640 1 -- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518070740 msgr2=0x7f15180c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:59.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151db92640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518070740 0x7f15180c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:59.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151db92640 1 -- 172.21.15.46:0/1585670421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1508009420 con 0x7f151815e700 2023-10-30T14:15:59.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151d391640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518070740 0x7f15180c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:15:59.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151db92640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f151815e700 0x7f1518160bc0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f151000efc0 tx=0x7f151000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f150dffb640 1 -- 172.21.15.46:0/1585670421 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510009ba0 con 0x7f151815e700 2023-10-30T14:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f150dffb640 1 -- 172.21.15.46:0/1585670421 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151000f040 con 0x7f151815e700 2023-10-30T14:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151f5f5640 1 -- 172.21.15.46:0/1585670421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15180c14c0 con 0x7f151815e700 2023-10-30T14:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f150dffb640 1 -- 172.21.15.46:0/1585670421 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15100146c0 con 0x7f151815e700 2023-10-30T14:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151f5f5640 1 -- 172.21.15.46:0/1585670421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15181603c0 con 0x7f151815e700 2023-10-30T14:15:59.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f151f5f5640 1 -- 172.21.15.46:0/1585670421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151804e170 con 0x7f151815e700 2023-10-30T14:15:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.513+0000 7f150dffb640 1 -- 172.21.15.46:0/1585670421 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1510014820 con 0x7f151815e700 2023-10-30T14:15:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.517+0000 7f150dffb640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f14fc0664a0 0x7f14fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:15:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.517+0000 7f150dffb640 1 -- 172.21.15.46:0/1585670421 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7f1510085780 con 0x7f151815e700 2023-10-30T14:15:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.517+0000 7f151d391640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f14fc0664a0 0x7f14fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:15:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.517+0000 7f151d391640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f14fc0664a0 0x7f14fc068960 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f1514005890 tx=0x7f1514009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:15:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.521+0000 7f150dffb640 1 -- 172.21.15.46:0/1585670421 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1510052150 con 0x7f151815e700 2023-10-30T14:15:59.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.709+0000 7f151f5f5640 1 -- 172.21.15.46:0/1585670421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15181683e0 con 0x7f151815e700 2023-10-30T14:15:59.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.709+0000 7f150dffb640 1 -- 172.21.15.46:0/1585670421 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f1510018070 con 0x7f151815e700 2023-10-30T14:15:59.714 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:15:59.714 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:32.954977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:15:59.714 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 224 2023-10-30T14:15:59.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.709+0000 7f151f5f5640 1 -- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f14fc0664a0 msgr2=0x7f14fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.709+0000 7f151f5f5640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f14fc0664a0 0x7f14fc068960 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f1514005890 tx=0x7f1514009040 comp rx=0 tx=0).stop 2023-10-30T14:15:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.709+0000 7f151f5f5640 1 -- 172.21.15.46:0/1585670421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f151815e700 msgr2=0x7f1518160bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:15:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.709+0000 7f151f5f5640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f151815e700 0x7f1518160bc0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f151000efc0 tx=0x7f151000c490 comp rx=0 tx=0).stop 2023-10-30T14:15:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.713+0000 7f151f5f5640 1 -- 172.21.15.46:0/1585670421 shutdown_connections 2023-10-30T14:15:59.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.713+0000 7f151f5f5640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f14fc0664a0 0x7f14fc068960 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.713+0000 7f151f5f5640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f151815e700 0x7f1518160bc0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.713+0000 7f151f5f5640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15181580e0 0x7f15180ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.713+0000 7f151f5f5640 1 --2- 172.21.15.46:0/1585670421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1518070740 0x7f15180c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:15:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.713+0000 7f151f5f5640 1 -- 172.21.15.46:0/1585670421 >> 172.21.15.46:0/1585670421 conn(0x7f15180c7910 msgr2=0x7f151814e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:15:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.713+0000 7f151f5f5640 1 -- 172.21.15.46:0/1585670421 shutdown_connections 2023-10-30T14:15:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:15:59.713+0000 7f151f5f5640 1 -- 172.21.15.46:0/1585670421 wait complete. 2023-10-30T14:15:59.735 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:16:01.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:01.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff25e691640 1 Processor -- start 2023-10-30T14:16:01.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff25e691640 1 -- start start 2023-10-30T14:16:01.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff25e691640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff2581538a0 0x7ff25815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:01.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff25e691640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2580ca0e0 0x7ff25815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:01.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff25e691640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2580cf160 0x7ff2580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:01.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff25e691640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2580cb800 con 0x7ff2580ca0e0 2023-10-30T14:16:01.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff25e691640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25815dc30 con 0x7ff2581538a0 2023-10-30T14:16:01.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff25e691640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580cb330 con 0x7ff2580cf160 2023-10-30T14:16:01.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff2577fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2580ca0e0 0x7ff25815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:01.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff2577fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2580ca0e0 0x7ff25815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56634/0 (socket says 172.21.15.46:56634) 2023-10-30T14:16:01.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff2577fe640 1 -- 172.21.15.46:0/1560635908 learned_addr learned my addr 172.21.15.46:0/1560635908 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:01.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff25cc2e640 1 --2- 172.21.15.46:0/1560635908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2580cf160 0x7ff2580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:01.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff257fff640 1 --2- 172.21.15.46:0/1560635908 >> v2:172.21.15.154:3300/0 conn(0x7ff2581538a0 0x7ff25815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff2577fe640 1 -- 172.21.15.46:0/1560635908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2580cf160 msgr2=0x7ff2580cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff2577fe640 1 --2- 172.21.15.46:0/1560635908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2580cf160 0x7ff2580cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff2577fe640 1 -- 172.21.15.46:0/1560635908 >> v2:172.21.15.154:3300/0 conn(0x7ff2581538a0 msgr2=0x7ff25815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff2577fe640 1 --2- 172.21.15.46:0/1560635908 >> v2:172.21.15.154:3300/0 conn(0x7ff2581538a0 0x7ff25815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff2577fe640 1 -- 172.21.15.46:0/1560635908 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2580cad70 con 0x7ff2580ca0e0 2023-10-30T14:16:01.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.889+0000 7ff25cc2e640 1 --2- 172.21.15.46:0/1560635908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2580cf160 0x7ff2580cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:01.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff2577fe640 1 --2- 172.21.15.46:0/1560635908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2580ca0e0 0x7ff25815a090 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7ff2580cfd10 tx=0x7ff24c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e563e4c06ed166 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:01.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff256ffd640 1 -- 172.21.15.46:0/1560635908 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c0a89b0 con 0x7ff2580ca0e0 2023-10-30T14:16:01.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff256ffd640 1 -- 172.21.15.46:0/1560635908 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c0a8b10 con 0x7ff2580ca0e0 2023-10-30T14:16:01.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff256ffd640 1 -- 172.21.15.46:0/1560635908 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c0a8ea0 con 0x7ff2580ca0e0 2023-10-30T14:16:01.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff25e691640 1 -- 172.21.15.46:0/1560635908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2580ca0e0 msgr2=0x7ff25815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:01.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff25e691640 1 --2- 172.21.15.46:0/1560635908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2580ca0e0 0x7ff25815a090 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7ff2580cfd10 tx=0x7ff24c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:16:01.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff25e691640 1 -- 172.21.15.46:0/1560635908 shutdown_connections 2023-10-30T14:16:01.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff25e691640 1 --2- 172.21.15.46:0/1560635908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2580cf160 0x7ff2580cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:01.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff25e691640 1 --2- 172.21.15.46:0/1560635908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2580ca0e0 0x7ff25815a090 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:01.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff25e691640 1 --2- 172.21.15.46:0/1560635908 >> v2:172.21.15.154:3300/0 conn(0x7ff2581538a0 0x7ff25815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:01.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff25e691640 1 -- 172.21.15.46:0/1560635908 >> 172.21.15.46:0/1560635908 conn(0x7ff25806bfe0 msgr2=0x7ff25806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:01.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff25e691640 1 -- 172.21.15.46:0/1560635908 shutdown_connections 2023-10-30T14:16:01.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.893+0000 7ff25e691640 1 -- 172.21.15.46:0/1560635908 wait complete. 2023-10-30T14:16:01.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff25e691640 1 Processor -- start 2023-10-30T14:16:01.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff25e691640 1 -- start start 2023-10-30T14:16:01.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff25e691640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2580ca0e0 0x7ff258148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:01.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff25e691640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2580cf160 0x7ff258146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:01.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff25e691640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2581538a0 0x7ff258147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:01.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff25e691640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580cef70 con 0x7ff2580ca0e0 2023-10-30T14:16:01.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff25e691640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580ce3f0 con 0x7ff2581538a0 2023-10-30T14:16:01.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff25e691640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25815ac60 con 0x7ff2580cf160 2023-10-30T14:16:01.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff257fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2580ca0e0 0x7ff258148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:01.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff257fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2580ca0e0 0x7ff258148f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56644/0 (socket says 172.21.15.46:56644) 2023-10-30T14:16:01.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff257fff640 1 -- 172.21.15.46:0/1250410140 learned_addr learned my addr 172.21.15.46:0/1250410140 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:01.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff2577fe640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2580cf160 0x7ff258146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:01.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff25cc2e640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2581538a0 0x7ff258147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:01.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff257fff640 1 -- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2580cf160 msgr2=0x7ff258146e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:01.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff257fff640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2580cf160 0x7ff258146e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:01.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff257fff640 1 -- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2581538a0 msgr2=0x7ff258147390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:01.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff257fff640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2581538a0 0x7ff258147390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:01.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff257fff640 1 -- 172.21.15.46:0/1250410140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0094d0 con 0x7ff2580ca0e0 2023-10-30T14:16:01.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff257fff640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2580ca0e0 0x7ff258148f60 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7ff24800d5f0 tx=0x7ff24800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:01.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff254ff9640 1 -- 172.21.15.46:0/1250410140 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2480125f0 con 0x7ff2580ca0e0 2023-10-30T14:16:01.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff254ff9640 1 -- 172.21.15.46:0/1250410140 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff248012750 con 0x7ff2580ca0e0 2023-10-30T14:16:01.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff25e691640 1 -- 172.21.15.46:0/1250410140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2580ce640 con 0x7ff2580ca0e0 2023-10-30T14:16:01.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff254ff9640 1 -- 172.21.15.46:0/1250410140 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff248012ae0 con 0x7ff2580ca0e0 2023-10-30T14:16:01.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.897+0000 7ff25e691640 1 -- 172.21.15.46:0/1250410140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2580cec00 con 0x7ff2580ca0e0 2023-10-30T14:16:01.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.901+0000 7ff25e691640 1 -- 172.21.15.46:0/1250410140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2580b2a50 con 0x7ff2580ca0e0 2023-10-30T14:16:01.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.901+0000 7ff254ff9640 1 -- 172.21.15.46:0/1250410140 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff248012c40 con 0x7ff2580ca0e0 2023-10-30T14:16:01.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.901+0000 7ff254ff9640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff2440664a0 0x7ff244068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:01.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.901+0000 7ff254ff9640 1 -- 172.21.15.46:0/1250410140 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7ff248084cb0 con 0x7ff2580ca0e0 2023-10-30T14:16:01.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.901+0000 7ff2577fe640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff2440664a0 0x7ff244068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:01.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.901+0000 7ff2577fe640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff2440664a0 0x7ff244068960 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0b2000 tx=0x7ff24c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:01.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:01.905+0000 7ff254ff9640 1 -- 172.21.15.46:0/1250410140 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff2480517a0 con 0x7ff2580ca0e0 2023-10-30T14:16:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.117+0000 7ff25e691640 1 -- 172.21.15.46:0/1250410140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff25804e170 con 0x7ff2580ca0e0 2023-10-30T14:16:02.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.117+0000 7ff254ff9640 1 -- 172.21.15.46:0/1250410140 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+6523 (secure 0 0 0) 0x7ff248007380 con 0x7ff2580ca0e0 2023-10-30T14:16:02.123 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:02.123 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:32.954977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:02.123 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 224 2023-10-30T14:16:02.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.121+0000 7ff25e691640 1 -- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff2440664a0 msgr2=0x7ff244068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.121+0000 7ff25e691640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff2440664a0 0x7ff244068960 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0b2000 tx=0x7ff24c008040 comp rx=0 tx=0).stop 2023-10-30T14:16:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.121+0000 7ff25e691640 1 -- 172.21.15.46:0/1250410140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2580ca0e0 msgr2=0x7ff258148f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.121+0000 7ff25e691640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2580ca0e0 0x7ff258148f60 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7ff24800d5f0 tx=0x7ff24800dab0 comp rx=0 tx=0).stop 2023-10-30T14:16:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.121+0000 7ff25e691640 1 -- 172.21.15.46:0/1250410140 shutdown_connections 2023-10-30T14:16:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.121+0000 7ff25e691640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff2440664a0 0x7ff244068960 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.121+0000 7ff25e691640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2581538a0 0x7ff258147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.121+0000 7ff25e691640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2580cf160 0x7ff258146e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.121+0000 7ff25e691640 1 --2- 172.21.15.46:0/1250410140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2580ca0e0 0x7ff258148f60 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.121+0000 7ff25e691640 1 -- 172.21.15.46:0/1250410140 >> 172.21.15.46:0/1250410140 conn(0x7ff25806bfe0 msgr2=0x7ff25806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.121+0000 7ff25e691640 1 -- 172.21.15.46:0/1250410140 shutdown_connections 2023-10-30T14:16:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:02.121+0000 7ff25e691640 1 -- 172.21.15.46:0/1250410140 wait complete. 2023-10-30T14:16:02.144 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:16:04.145 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:04.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca86409640 1 Processor -- start 2023-10-30T14:16:04.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca86409640 1 -- start start 2023-10-30T14:16:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca86409640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fca8015d6f0 0x7fca8015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca86409640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca800cf160 0x7fca800cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca86409640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca800cb800 con 0x7fca801538a0 2023-10-30T14:16:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca86409640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca800cb330 con 0x7fca8015d6f0 2023-10-30T14:16:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca86409640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca800cad70 con 0x7fca800cf160 2023-10-30T14:16:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fca801538a0 0x7fca800ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33692/0 (socket says 172.21.15.46:33692) 2023-10-30T14:16:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7ffff640 1 -- 172.21.15.46:0/3886338731 learned_addr learned my addr 172.21.15.46:0/3886338731 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:16:04.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7f7fe640 1 --2- 172.21.15.46:0/3886338731 >> v2:172.21.15.154:3300/0 conn(0x7fca8015d6f0 0x7fca8015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:04.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca849a6640 1 --2- 172.21.15.46:0/3886338731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca800cf160 0x7fca800cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372147580 0 0) 0x7fca800cb800 con 0x7fca801538a0 2023-10-30T14:16:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca68003530 con 0x7fca801538a0 2023-10-30T14:16:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 267605453 0 0) 0x7fca68003530 con 0x7fca801538a0 2023-10-30T14:16:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca800cf160 msgr2=0x7fca800cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 --2- 172.21.15.46:0/3886338731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca800cf160 0x7fca800cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:04.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 >> v2:172.21.15.154:3300/0 conn(0x7fca8015d6f0 msgr2=0x7fca8015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:04.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 --2- 172.21.15.46:0/3886338731 >> v2:172.21.15.154:3300/0 conn(0x7fca8015d6f0 0x7fca8015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:04.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca800cb050 con 0x7fca801538a0 2023-10-30T14:16:04.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fca70001680 con 0x7fca801538a0 2023-10-30T14:16:04.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 >> v1:172.21.15.46:6789/0 conn(0x7fca801538a0 legacy=0x7fca800ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:04.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 --2- 172.21.15.46:0/3886338731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca68003980 0x7fca68003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:04.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 --2- 172.21.15.46:0/3886338731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680047d0 0x7fca68004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:04.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 --2- 172.21.15.46:0/3886338731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca800cf160 0x7fca68005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:04.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca800cb330 con 0x7fca680047d0 2023-10-30T14:16:04.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca800cad70 con 0x7fca68003980 2023-10-30T14:16:04.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.297+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6800b990 con 0x7fca800cf160 2023-10-30T14:16:04.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.301+0000 7fca7f7fe640 1 --2- 172.21.15.46:0/3886338731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680047d0 0x7fca68004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:04.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.301+0000 7fca7ffff640 1 --2- 172.21.15.46:0/3886338731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca68003980 0x7fca68003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:04.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.301+0000 7fca849a6640 1 --2- 172.21.15.46:0/3886338731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca800cf160 0x7fca68005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:04.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.301+0000 7fca7f7fe640 1 -- 172.21.15.46:0/3886338731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca800cf160 msgr2=0x7fca68005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:04.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.301+0000 7fca7f7fe640 1 --2- 172.21.15.46:0/3886338731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca800cf160 0x7fca68005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:04.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.301+0000 7fca7f7fe640 1 -- 172.21.15.46:0/3886338731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca68003980 msgr2=0x7fca68003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:04.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.301+0000 7fca7f7fe640 1 --2- 172.21.15.46:0/3886338731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca68003980 0x7fca68003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:04.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.301+0000 7fca7f7fe640 1 -- 172.21.15.46:0/3886338731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca68003530 con 0x7fca680047d0 2023-10-30T14:16:04.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.301+0000 7fca7f7fe640 1 --2- 172.21.15.46:0/3886338731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680047d0 0x7fca68004c50 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fca74009ae0 tx=0x7fca7401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e25c9cecd3205e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.301+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca740a55a0 con 0x7fca680047d0 2023-10-30T14:16:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.301+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca740a5700 con 0x7fca680047d0 2023-10-30T14:16:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.301+0000 7fca7effd640 1 -- 172.21.15.46:0/3886338731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca740a5ab0 con 0x7fca680047d0 2023-10-30T14:16:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 -- 172.21.15.46:0/3886338731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680047d0 msgr2=0x7fca68004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 --2- 172.21.15.46:0/3886338731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca680047d0 0x7fca68004c50 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fca74009ae0 tx=0x7fca7401ca90 comp rx=0 tx=0).stop 2023-10-30T14:16:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca849a6640 1 -- 172.21.15.46:0/3886338731 reap_dead start 2023-10-30T14:16:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 -- 172.21.15.46:0/3886338731 shutdown_connections 2023-10-30T14:16:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 -- 172.21.15.46:0/3886338731 >> 172.21.15.46:0/3886338731 conn(0x7fca8006bfe0 msgr2=0x7fca8006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 -- 172.21.15.46:0/3886338731 shutdown_connections 2023-10-30T14:16:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 -- 172.21.15.46:0/3886338731 wait complete. 2023-10-30T14:16:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 Processor -- start 2023-10-30T14:16:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 -- start start 2023-10-30T14:16:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680047d0 0x7fca8015bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca8015c260 0x7fca80158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca80159060 0x7fca801f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca800cd0b0 con 0x7fca80159060 2023-10-30T14:16:04.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8015b2c0 con 0x7fca680047d0 2023-10-30T14:16:04.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca86409640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca801efd80 con 0x7fca8015c260 2023-10-30T14:16:04.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca849a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca80159060 0x7fca801f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:04.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca849a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca80159060 0x7fca801f2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56660/0 (socket says 172.21.15.46:56660) 2023-10-30T14:16:04.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca849a6640 1 -- 172.21.15.46:0/2043221160 learned_addr learned my addr 172.21.15.46:0/2043221160 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca7f7fe640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca8015c260 0x7fca80158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.305+0000 7fca7ffff640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680047d0 0x7fca8015bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca849a6640 1 -- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca8015c260 msgr2=0x7fca80158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca849a6640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca8015c260 0x7fca80158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca849a6640 1 -- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680047d0 msgr2=0x7fca8015bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca849a6640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680047d0 0x7fca8015bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca849a6640 1 -- 172.21.15.46:0/2043221160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8015a920 con 0x7fca80159060 2023-10-30T14:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca7f7fe640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca8015c260 0x7fca80158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca849a6640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca80159060 0x7fca801f2380 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fca80072ce0 tx=0x7fca7800f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca7cff9640 1 -- 172.21.15.46:0/2043221160 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7800e040 con 0x7fca80159060 2023-10-30T14:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca7cff9640 1 -- 172.21.15.46:0/2043221160 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7800f040 con 0x7fca80159060 2023-10-30T14:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca7cff9640 1 -- 172.21.15.46:0/2043221160 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78015430 con 0x7fca80159060 2023-10-30T14:16:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca86409640 1 -- 172.21.15.46:0/2043221160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8015c6a0 con 0x7fca80159060 2023-10-30T14:16:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca86409640 1 -- 172.21.15.46:0/2043221160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8015b510 con 0x7fca80159060 2023-10-30T14:16:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca86409640 1 -- 172.21.15.46:0/2043221160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8015b700 con 0x7fca80159060 2023-10-30T14:16:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca7cff9640 1 -- 172.21.15.46:0/2043221160 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fca78015590 con 0x7fca80159060 2023-10-30T14:16:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca7cff9640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca6c0664d0 0x7fca6c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.309+0000 7fca7cff9640 1 -- 172.21.15.46:0/2043221160 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8176+0+0 (secure 0 0 0) 0x7fca78084ea0 con 0x7fca80159060 2023-10-30T14:16:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.313+0000 7fca7ffff640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca6c0664d0 0x7fca6c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.313+0000 7fca7ffff640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca6c0664d0 0x7fca6c068990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fca70001ec0 tx=0x7fca70009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.313+0000 7fca7cff9640 1 -- 172.21.15.46:0/2043221160 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fca78051900 con 0x7fca80159060 2023-10-30T14:16:04.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.501+0000 7fca86409640 1 -- 172.21.15.46:0/2043221160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca8004e170 con 0x7fca80159060 2023-10-30T14:16:04.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca7cff9640 1 -- 172.21.15.46:0/2043221160 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fca78054f20 con 0x7fca80159060 2023-10-30T14:16:04.509 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:04.510 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:15:32.954977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":8982},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8982":{"gid":8982,"name":"a","rank":1,"incarnation":209,"state":"up:active","state_seq":37,"addr":"172.21.15.46:6833/2455118950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2455118950},{"type":"v1","addr":"172.21.15.46:6833","nonce":2455118950}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:04.511 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 224 2023-10-30T14:16:04.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca86409640 1 -- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca6c0664d0 msgr2=0x7fca6c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:04.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca86409640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca6c0664d0 0x7fca6c068990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fca70001ec0 tx=0x7fca70009040 comp rx=0 tx=0).stop 2023-10-30T14:16:04.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca86409640 1 -- 172.21.15.46:0/2043221160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca80159060 msgr2=0x7fca801f2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:04.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca86409640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca80159060 0x7fca801f2380 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fca80072ce0 tx=0x7fca7800f1b0 comp rx=0 tx=0).stop 2023-10-30T14:16:04.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca86409640 1 -- 172.21.15.46:0/2043221160 shutdown_connections 2023-10-30T14:16:04.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca86409640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fca6c0664d0 0x7fca6c068990 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:04.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca86409640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca80159060 0x7fca801f2380 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:04.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca86409640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca8015c260 0x7fca80158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:04.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca86409640 1 --2- 172.21.15.46:0/2043221160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680047d0 0x7fca8015bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:04.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca86409640 1 -- 172.21.15.46:0/2043221160 >> 172.21.15.46:0/2043221160 conn(0x7fca8006bfe0 msgr2=0x7fca80151870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:04.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca86409640 1 -- 172.21.15.46:0/2043221160 shutdown_connections 2023-10-30T14:16:04.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:04.505+0000 7fca86409640 1 -- 172.21.15.46:0/2043221160 wait complete. 2023-10-30T14:16:04.529 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:16:06.530 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:06.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.681+0000 7fce99868640 1 Processor -- start 2023-10-30T14:16:06.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.681+0000 7fce99868640 1 -- start start 2023-10-30T14:16:06.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.681+0000 7fce99868640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fce9406f7a0 0x7fce9406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:06.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.681+0000 7fce99868640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce94066cf0 con 0x7fce940c6bd0 2023-10-30T14:16:06.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.681+0000 7fce99868640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce940c70d0 con 0x7fce94071110 2023-10-30T14:16:06.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.681+0000 7fce99868640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce940700c0 con 0x7fce9406f7a0 2023-10-30T14:16:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.681+0000 7fce92ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fce9406f7a0 0x7fce9406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.681+0000 7fce92ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fce9406f7a0 0x7fce9406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44590/0 (socket says 172.21.15.46:44590) 2023-10-30T14:16:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.681+0000 7fce8bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fce940c6bd0 0x7fce94069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57732/0 (socket says 172.21.15.46:57732) 2023-10-30T14:16:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.681+0000 7fce92ffd640 1 -- 172.21.15.46:0/1847077581 learned_addr learned my addr 172.21.15.46:0/1847077581 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:06.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.685+0000 7fce92ffd640 1 -- 172.21.15.46:0/1847077581 >> v1:172.21.15.154:6789/0 conn(0x7fce94071110 legacy=0x7fce94158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:06.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.685+0000 7fce92ffd640 1 -- 172.21.15.46:0/1847077581 >> v1:172.21.15.46:6789/0 conn(0x7fce940c6bd0 legacy=0x7fce94069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:06.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.685+0000 7fce92ffd640 1 -- 172.21.15.46:0/1847077581 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce940c7390 con 0x7fce9406f7a0 2023-10-30T14:16:06.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.685+0000 7fce92ffd640 1 --2- 172.21.15.46:0/1847077581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce9406f7a0 0x7fce9406fb80 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fce84009870 tx=0x7fce8401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a80a9e2602155fe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:06.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.685+0000 7fce927fc640 1 -- 172.21.15.46:0/1847077581 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce840a8a30 con 0x7fce9406f7a0 2023-10-30T14:16:06.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.685+0000 7fce927fc640 1 -- 172.21.15.46:0/1847077581 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce840a8b90 con 0x7fce9406f7a0 2023-10-30T14:16:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.685+0000 7fce927fc640 1 -- 172.21.15.46:0/1847077581 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce840b0440 con 0x7fce9406f7a0 2023-10-30T14:16:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.685+0000 7fce99868640 1 -- 172.21.15.46:0/1847077581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce9406f7a0 msgr2=0x7fce9406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.685+0000 7fce99868640 1 --2- 172.21.15.46:0/1847077581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce9406f7a0 0x7fce9406fb80 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fce84009870 tx=0x7fce8401ed30 comp rx=0 tx=0).stop 2023-10-30T14:16:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 -- 172.21.15.46:0/1847077581 shutdown_connections 2023-10-30T14:16:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 --2- 172.21.15.46:0/1847077581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce9406f7a0 0x7fce9406fb80 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:06.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 -- 172.21.15.46:0/1847077581 >> 172.21.15.46:0/1847077581 conn(0x7fce940c7910 msgr2=0x7fce940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:06.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 -- 172.21.15.46:0/1847077581 shutdown_connections 2023-10-30T14:16:06.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 -- 172.21.15.46:0/1847077581 wait complete. 2023-10-30T14:16:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 Processor -- start 2023-10-30T14:16:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 -- start start 2023-10-30T14:16:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce9406f7a0 0x7fce94154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:06.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce94071110 0x7fce941e3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:06.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce940c6bd0 0x7fce941e3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:06.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce941e2bc0 con 0x7fce94071110 2023-10-30T14:16:06.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce941ef6d0 con 0x7fce9406f7a0 2023-10-30T14:16:06.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce99868640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce941efc00 con 0x7fce940c6bd0 2023-10-30T14:16:06.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce8bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce94071110 0x7fce941e3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:06.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce8bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce94071110 0x7fce941e3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44596/0 (socket says 172.21.15.46:44596) 2023-10-30T14:16:06.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce8bfff640 1 -- 172.21.15.46:0/3004477898 learned_addr learned my addr 172.21.15.46:0/3004477898 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:06.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce92ffd640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce9406f7a0 0x7fce94154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce937fe640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce940c6bd0 0x7fce941e3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce8bfff640 1 -- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce940c6bd0 msgr2=0x7fce941e3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce8bfff640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce940c6bd0 0x7fce941e3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce8bfff640 1 -- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce9406f7a0 msgr2=0x7fce94154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce8bfff640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce9406f7a0 0x7fce94154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce8bfff640 1 -- 172.21.15.46:0/3004477898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce840094d0 con 0x7fce94071110 2023-10-30T14:16:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce8bfff640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce94071110 0x7fce941e3010 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00c500 tx=0x7fce7c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.689+0000 7fce90ff9640 1 -- 172.21.15.46:0/3004477898 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c007500 con 0x7fce94071110 2023-10-30T14:16:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.693+0000 7fce90ff9640 1 -- 172.21.15.46:0/3004477898 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c007660 con 0x7fce94071110 2023-10-30T14:16:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.693+0000 7fce99868640 1 -- 172.21.15.46:0/3004477898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce941ef920 con 0x7fce94071110 2023-10-30T14:16:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.693+0000 7fce90ff9640 1 -- 172.21.15.46:0/3004477898 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c004a10 con 0x7fce94071110 2023-10-30T14:16:06.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.693+0000 7fce99868640 1 -- 172.21.15.46:0/3004477898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce941e2090 con 0x7fce94071110 2023-10-30T14:16:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.693+0000 7fce99868640 1 -- 172.21.15.46:0/3004477898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce941e2d30 con 0x7fce94071110 2023-10-30T14:16:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.693+0000 7fce90ff9640 1 -- 172.21.15.46:0/3004477898 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fce7c004b70 con 0x7fce94071110 2023-10-30T14:16:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.693+0000 7fce90ff9640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fce740664f0 0x7fce740689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.693+0000 7fce90ff9640 1 -- 172.21.15.46:0/3004477898 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7fce7c083e90 con 0x7fce94071110 2023-10-30T14:16:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.693+0000 7fce92ffd640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fce740664f0 0x7fce740689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:06.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.697+0000 7fce92ffd640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fce740664f0 0x7fce740689b0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fce840b2000 tx=0x7fce84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:06.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.697+0000 7fce90ff9640 1 -- 172.21.15.46:0/3004477898 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fce7c0509d0 con 0x7fce94071110 2023-10-30T14:16:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.889+0000 7fce99868640 1 -- 172.21.15.46:0/3004477898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce9404e170 con 0x7fce94071110 2023-10-30T14:16:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.889+0000 7fce90ff9640 1 -- 172.21.15.46:0/3004477898 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fce7c018070 con 0x7fce94071110 2023-10-30T14:16:06.894 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:06.895 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":225,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:04.689367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 225 2023-10-30T14:16:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.893+0000 7fce99868640 1 -- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fce740664f0 msgr2=0x7fce740689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.893+0000 7fce99868640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fce740664f0 0x7fce740689b0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fce840b2000 tx=0x7fce84008040 comp rx=0 tx=0).stop 2023-10-30T14:16:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.893+0000 7fce99868640 1 -- 172.21.15.46:0/3004477898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce94071110 msgr2=0x7fce941e3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.893+0000 7fce99868640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce94071110 0x7fce941e3010 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00c500 tx=0x7fce7c00c9c0 comp rx=0 tx=0).stop 2023-10-30T14:16:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.893+0000 7fce99868640 1 -- 172.21.15.46:0/3004477898 shutdown_connections 2023-10-30T14:16:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.893+0000 7fce99868640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fce740664f0 0x7fce740689b0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.893+0000 7fce99868640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce940c6bd0 0x7fce941e3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.893+0000 7fce99868640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce94071110 0x7fce941e3010 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.893+0000 7fce99868640 1 --2- 172.21.15.46:0/3004477898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce9406f7a0 0x7fce94154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.893+0000 7fce99868640 1 -- 172.21.15.46:0/3004477898 >> 172.21.15.46:0/3004477898 conn(0x7fce940c7910 msgr2=0x7fce940c3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.893+0000 7fce99868640 1 -- 172.21.15.46:0/3004477898 shutdown_connections 2023-10-30T14:16:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:06.893+0000 7fce99868640 1 -- 172.21.15.46:0/3004477898 wait complete. 2023-10-30T14:16:06.916 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T14:16:06.917 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:16:06.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe866d9d640 1 Processor -- start 2023-10-30T14:16:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe866d9d640 1 -- start start 2023-10-30T14:16:07.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe866d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe86015d6f0 0x7fe86015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe866d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8600cf160 0x7fe8600cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe866d9d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8600cb800 con 0x7fe8601538a0 2023-10-30T14:16:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe866d9d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8600cb330 con 0x7fe8600cf160 2023-10-30T14:16:07.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe866d9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8600cad70 con 0x7fe86015d6f0 2023-10-30T14:16:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe86533a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8600cf160 0x7fe8600cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe86533a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8600cf160 0x7fe8600cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44602/0 (socket says 172.21.15.46:44602) 2023-10-30T14:16:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe86533a640 1 -- 172.21.15.46:0/2454972150 learned_addr learned my addr 172.21.15.46:0/2454972150 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe85ffff640 1 --2- 172.21.15.46:0/2454972150 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe86015d6f0 0x7fe86015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe86533a640 1 -- 172.21.15.46:0/2454972150 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe86015d6f0 msgr2=0x7fe86015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe86533a640 1 --2- 172.21.15.46:0/2454972150 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe86015d6f0 0x7fe86015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe86533a640 1 -- 172.21.15.46:0/2454972150 >> v1:172.21.15.154:6789/0 conn(0x7fe8601538a0 legacy=0x7fe8600ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:07.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.057+0000 7fe86533a640 1 -- 172.21.15.46:0/2454972150 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8600cb050 con 0x7fe8600cf160 2023-10-30T14:16:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.061+0000 7fe86533a640 1 --2- 172.21.15.46:0/2454972150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8600cf160 0x7fe8600cf590 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fe858009e50 tx=0x7fe85801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1a8b44e8f14389 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.061+0000 7fe85f7fe640 1 -- 172.21.15.46:0/2454972150 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8580af400 con 0x7fe8600cf160 2023-10-30T14:16:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.061+0000 7fe85f7fe640 1 -- 172.21.15.46:0/2454972150 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8580ab050 con 0x7fe8600cf160 2023-10-30T14:16:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.061+0000 7fe85f7fe640 1 -- 172.21.15.46:0/2454972150 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8580af780 con 0x7fe8600cf160 2023-10-30T14:16:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.061+0000 7fe866d9d640 1 -- 172.21.15.46:0/2454972150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8600cf160 msgr2=0x7fe8600cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:07.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.061+0000 7fe866d9d640 1 --2- 172.21.15.46:0/2454972150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8600cf160 0x7fe8600cf590 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fe858009e50 tx=0x7fe85801f1f0 comp rx=0 tx=0).stop 2023-10-30T14:16:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.061+0000 7fe866d9d640 1 -- 172.21.15.46:0/2454972150 shutdown_connections 2023-10-30T14:16:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.061+0000 7fe866d9d640 1 --2- 172.21.15.46:0/2454972150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8600cf160 0x7fe8600cf590 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.061+0000 7fe866d9d640 1 --2- 172.21.15.46:0/2454972150 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe86015d6f0 0x7fe86015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:07.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.061+0000 7fe866d9d640 1 -- 172.21.15.46:0/2454972150 >> 172.21.15.46:0/2454972150 conn(0x7fe86006bfe0 msgr2=0x7fe86006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.061+0000 7fe866d9d640 1 -- 172.21.15.46:0/2454972150 shutdown_connections 2023-10-30T14:16:07.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.061+0000 7fe866d9d640 1 -- 172.21.15.46:0/2454972150 wait complete. 2023-10-30T14:16:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe866d9d640 1 Processor -- start 2023-10-30T14:16:07.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe866d9d640 1 -- start start 2023-10-30T14:16:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe866d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8600cf160 0x7fe8600c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe866d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8601538a0 0x7fe8600cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe866d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe86015d6f0 0x7fe8600cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:07.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe866d9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8600c6c30 con 0x7fe8600cf160 2023-10-30T14:16:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe866d9d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe860159130 con 0x7fe8601538a0 2023-10-30T14:16:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe866d9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8600ccfb0 con 0x7fe86015d6f0 2023-10-30T14:16:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe864b39640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8600cf160 0x7fe8600c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe864b39640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8600cf160 0x7fe8600c66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44608/0 (socket says 172.21.15.46:44608) 2023-10-30T14:16:07.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe864b39640 1 -- 172.21.15.46:0/3869999465 learned_addr learned my addr 172.21.15.46:0/3869999465 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe86533a640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe86015d6f0 0x7fe8600cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe85ffff640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8601538a0 0x7fe8600cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe864b39640 1 -- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe86015d6f0 msgr2=0x7fe8600cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe864b39640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe86015d6f0 0x7fe8600cdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:07.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe864b39640 1 -- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8601538a0 msgr2=0x7fe8600cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe864b39640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8601538a0 0x7fe8600cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe864b39640 1 -- 172.21.15.46:0/3869999465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe858009b30 con 0x7fe8600cf160 2023-10-30T14:16:07.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe85ffff640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8601538a0 0x7fe8600cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:07.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe864b39640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8600cf160 0x7fe8600c66f0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fe85400c500 tx=0x7fe85400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe85d7fa640 1 -- 172.21.15.46:0/3869999465 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854005220 con 0x7fe8600cf160 2023-10-30T14:16:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe85d7fa640 1 -- 172.21.15.46:0/3869999465 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe854005380 con 0x7fe8600cf160 2023-10-30T14:16:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.065+0000 7fe85d7fa640 1 -- 172.21.15.46:0/3869999465 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854007270 con 0x7fe8600cf160 2023-10-30T14:16:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.069+0000 7fe866d9d640 1 -- 172.21.15.46:0/3869999465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8601588d0 con 0x7fe8600cf160 2023-10-30T14:16:07.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.069+0000 7fe866d9d640 1 -- 172.21.15.46:0/3869999465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe860158e50 con 0x7fe8600cf160 2023-10-30T14:16:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.069+0000 7fe866d9d640 1 -- 172.21.15.46:0/3869999465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8600b18e0 con 0x7fe8600cf160 2023-10-30T14:16:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.069+0000 7fe85d7fa640 1 -- 172.21.15.46:0/3869999465 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8540073d0 con 0x7fe8600cf160 2023-10-30T14:16:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.069+0000 7fe85d7fa640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe844066520 0x7fe8440689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.069+0000 7fe85d7fa640 1 -- 172.21.15.46:0/3869999465 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7fe854083df0 con 0x7fe8600cf160 2023-10-30T14:16:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.069+0000 7fe85ffff640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe844066520 0x7fe8440689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.073+0000 7fe85ffff640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe844066520 0x7fe8440689e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe860066ad0 tx=0x7fe84c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.073+0000 7fe85d7fa640 1 -- 172.21.15.46:0/3869999465 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe854050850 con 0x7fe8600cf160 2023-10-30T14:16:07.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.269+0000 7fe866d9d640 1 -- 172.21.15.46:0/3869999465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8600b2a50 con 0x7fe8600cf160 2023-10-30T14:16:07.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.269+0000 7fe85d7fa640 1 -- 172.21.15.46:0/3869999465 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fe854053e70 con 0x7fe8600cf160 2023-10-30T14:16:07.275 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:07.275 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":225,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:04.689367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:07.275 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 225 2023-10-30T14:16:07.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.273+0000 7fe866d9d640 1 -- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe844066520 msgr2=0x7fe8440689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:07.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.273+0000 7fe866d9d640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe844066520 0x7fe8440689e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe860066ad0 tx=0x7fe84c008040 comp rx=0 tx=0).stop 2023-10-30T14:16:07.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.273+0000 7fe866d9d640 1 -- 172.21.15.46:0/3869999465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8600cf160 msgr2=0x7fe8600c66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:07.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.273+0000 7fe866d9d640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8600cf160 0x7fe8600c66f0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fe85400c500 tx=0x7fe85400c9c0 comp rx=0 tx=0).stop 2023-10-30T14:16:07.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.273+0000 7fe866d9d640 1 -- 172.21.15.46:0/3869999465 shutdown_connections 2023-10-30T14:16:07.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.273+0000 7fe866d9d640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe844066520 0x7fe8440689e0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:07.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.273+0000 7fe866d9d640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe86015d6f0 0x7fe8600cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:07.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.273+0000 7fe866d9d640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8601538a0 0x7fe8600cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:07.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.273+0000 7fe866d9d640 1 --2- 172.21.15.46:0/3869999465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8600cf160 0x7fe8600c66f0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:07.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.273+0000 7fe866d9d640 1 -- 172.21.15.46:0/3869999465 >> 172.21.15.46:0/3869999465 conn(0x7fe86006bfe0 msgr2=0x7fe86015b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:07.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.273+0000 7fe866d9d640 1 -- 172.21.15.46:0/3869999465 shutdown_connections 2023-10-30T14:16:07.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:07.273+0000 7fe866d9d640 1 -- 172.21.15.46:0/3869999465 wait complete. 2023-10-30T14:16:07.296 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 225, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 225, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9135": { "addr": "172.21.15.46:6837/681817982", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 681817982, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 681817982, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 9135, "incarnation": 218, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 42 }, "gid_9276": { "addr": "172.21.15.46:6836/2239735286", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 2239735286, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 2239735286, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9276, "incarnation": 225, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:16:04.689367+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9135, "mds_1": 9276 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:16:09.297 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:09.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd29b3e640 1 Processor -- start 2023-10-30T14:16:09.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd29b3e640 1 -- start start 2023-10-30T14:16:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd29b3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efd2415d6f0 0x7efd2415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd29b3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd240cb800 con 0x7efd241538a0 2023-10-30T14:16:09.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd29b3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd240cb330 con 0x7efd240cf160 2023-10-30T14:16:09.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd29b3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd240cad70 con 0x7efd2415d6f0 2023-10-30T14:16:09.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd237fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efd241538a0 0x7efd240ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57738/0 (socket says 172.21.15.46:57738) 2023-10-30T14:16:09.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd237fe640 1 -- 172.21.15.46:0/3168298513 learned_addr learned my addr 172.21.15.46:0/3168298513 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:16:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd22ffd640 1 --2- 172.21.15.46:0/3168298513 >> v2:172.21.15.154:3300/0 conn(0x7efd2415d6f0 0x7efd2415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548889664 0 0) 0x7efd240cb800 con 0x7efd241538a0 2023-10-30T14:16:09.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd0c003530 con 0x7efd241538a0 2023-10-30T14:16:09.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287141627 0 0) 0x7efd240cb330 con 0x7efd240cf160 2023-10-30T14:16:09.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd240cb800 con 0x7efd240cf160 2023-10-30T14:16:09.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1606260724 0 0) 0x7efd0c003530 con 0x7efd241538a0 2023-10-30T14:16:09.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 >> v2:172.21.15.154:3300/0 conn(0x7efd2415d6f0 msgr2=0x7efd2415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:09.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 --2- 172.21.15.46:0/3168298513 >> v2:172.21.15.154:3300/0 conn(0x7efd2415d6f0 0x7efd2415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:09.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 >> v1:172.21.15.154:6789/0 conn(0x7efd240cf160 legacy=0x7efd24164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:09.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd240cb050 con 0x7efd241538a0 2023-10-30T14:16:09.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7efd14001680 con 0x7efd241538a0 2023-10-30T14:16:09.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 >> v1:172.21.15.46:6789/0 conn(0x7efd241538a0 legacy=0x7efd240ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:09.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 --2- 172.21.15.46:0/3168298513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd0c003b10 0x7efd0c003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:09.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 --2- 172.21.15.46:0/3168298513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 0x7efd0c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:09.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 --2- 172.21.15.46:0/3168298513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd0c005190 0x7efd0c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:09.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd240cad70 con 0x7efd0c0047d0 2023-10-30T14:16:09.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c00bf80 con 0x7efd0c005190 2023-10-30T14:16:09.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c00c140 con 0x7efd0c003b10 2023-10-30T14:16:09.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd22ffd640 1 --2- 172.21.15.46:0/3168298513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 0x7efd0c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:09.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd23fff640 1 --2- 172.21.15.46:0/3168298513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd0c005190 0x7efd0c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:09.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd237fe640 1 --2- 172.21.15.46:0/3168298513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd0c003b10 0x7efd0c003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:09.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd22ffd640 1 -- 172.21.15.46:0/3168298513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd0c003b10 msgr2=0x7efd0c003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:09.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd22ffd640 1 --2- 172.21.15.46:0/3168298513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd0c003b10 0x7efd0c003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:09.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd22ffd640 1 -- 172.21.15.46:0/3168298513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd0c005190 msgr2=0x7efd0c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:09.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd22ffd640 1 --2- 172.21.15.46:0/3168298513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd0c005190 0x7efd0c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:09.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.449+0000 7efd22ffd640 1 -- 172.21.15.46:0/3168298513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0c003530 con 0x7efd0c0047d0 2023-10-30T14:16:09.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd23fff640 1 --2- 172.21.15.46:0/3168298513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd0c005190 0x7efd0c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:09.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd23fff640 1 -- 172.21.15.46:0/3168298513 reap_dead start 2023-10-30T14:16:09.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd22ffd640 1 --2- 172.21.15.46:0/3168298513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 0x7efd0c004c50 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7efd18009ab0 tx=0x7efd1801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70032205fb795bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd180a5620 con 0x7efd0c0047d0 2023-10-30T14:16:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd180a5780 con 0x7efd0c0047d0 2023-10-30T14:16:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd227fc640 1 -- 172.21.15.46:0/3168298513 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd180a5b30 con 0x7efd0c0047d0 2023-10-30T14:16:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd29b3e640 1 -- 172.21.15.46:0/3168298513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 msgr2=0x7efd0c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:09.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd29b3e640 1 --2- 172.21.15.46:0/3168298513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 0x7efd0c004c50 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7efd18009ab0 tx=0x7efd1801ca30 comp rx=0 tx=0).stop 2023-10-30T14:16:09.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd29b3e640 1 -- 172.21.15.46:0/3168298513 shutdown_connections 2023-10-30T14:16:09.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd29b3e640 1 --2- 172.21.15.46:0/3168298513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 0x7efd0c004c50 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:09.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd29b3e640 1 -- 172.21.15.46:0/3168298513 >> 172.21.15.46:0/3168298513 conn(0x7efd2406bfe0 msgr2=0x7efd2406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:09.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd29b3e640 1 -- 172.21.15.46:0/3168298513 shutdown_connections 2023-10-30T14:16:09.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.453+0000 7efd29b3e640 1 -- 172.21.15.46:0/3168298513 wait complete. 2023-10-30T14:16:09.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd29b3e640 1 Processor -- start 2023-10-30T14:16:09.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd29b3e640 1 -- start start 2023-10-30T14:16:09.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd29b3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 0x7efd240c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:09.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd29b3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd2415c000 0x7efd241e34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd29b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd24159320 0x7efd241f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd29b3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd241e47f0 con 0x7efd0c0047d0 2023-10-30T14:16:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd29b3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd240c8ec0 con 0x7efd2415c000 2023-10-30T14:16:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd29b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd240b4710 con 0x7efd24159320 2023-10-30T14:16:09.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 0x7efd240c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:09.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 0x7efd240c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44626/0 (socket says 172.21.15.46:44626) 2023-10-30T14:16:09.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd237fe640 1 -- 172.21.15.46:0/3084156790 learned_addr learned my addr 172.21.15.46:0/3084156790 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:09.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd23fff640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd24159320 0x7efd241f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:09.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd22ffd640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd2415c000 0x7efd241e34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:09.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd237fe640 1 -- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd24159320 msgr2=0x7efd241f28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd237fe640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd24159320 0x7efd241f28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd237fe640 1 -- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd2415c000 msgr2=0x7efd241e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd237fe640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd2415c000 0x7efd241e34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd237fe640 1 -- 172.21.15.46:0/3084156790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd241f3410 con 0x7efd0c0047d0 2023-10-30T14:16:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd22ffd640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd2415c000 0x7efd241e34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd23fff640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd24159320 0x7efd241f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd237fe640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 0x7efd240c8980 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7efd14011980 tx=0x7efd14011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd0bfff640 1 -- 172.21.15.46:0/3084156790 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1400fc20 con 0x7efd0c0047d0 2023-10-30T14:16:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd0bfff640 1 -- 172.21.15.46:0/3084156790 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1400fdc0 con 0x7efd0c0047d0 2023-10-30T14:16:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd29b3e640 1 -- 172.21.15.46:0/3084156790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd241f36f0 con 0x7efd0c0047d0 2023-10-30T14:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.457+0000 7efd0bfff640 1 -- 172.21.15.46:0/3084156790 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14015650 con 0x7efd0c0047d0 2023-10-30T14:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.461+0000 7efd29b3e640 1 -- 172.21.15.46:0/3084156790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd240b4430 con 0x7efd0c0047d0 2023-10-30T14:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.461+0000 7efd29b3e640 1 -- 172.21.15.46:0/3084156790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd2404e170 con 0x7efd0c0047d0 2023-10-30T14:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.461+0000 7efd0bfff640 1 -- 172.21.15.46:0/3084156790 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efd140157b0 con 0x7efd0c0047d0 2023-10-30T14:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.461+0000 7efd0bfff640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efd100664d0 0x7efd10068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.461+0000 7efd0bfff640 1 -- 172.21.15.46:0/3084156790 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7efd14087e80 con 0x7efd0c0047d0 2023-10-30T14:16:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.461+0000 7efd22ffd640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efd100664d0 0x7efd10068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.465+0000 7efd22ffd640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efd100664d0 0x7efd10068990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7efd18009ab0 tx=0x7efd18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.465+0000 7efd0bfff640 1 -- 172.21.15.46:0/3084156790 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efd140547c0 con 0x7efd0c0047d0 2023-10-30T14:16:09.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.653+0000 7efd29b3e640 1 -- 172.21.15.46:0/3084156790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd240b3f30 con 0x7efd0c0047d0 2023-10-30T14:16:09.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd0bfff640 1 -- 172.21.15.46:0/3084156790 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5729 (secure 0 0 0) 0x7efd14057de0 con 0x7efd0c0047d0 2023-10-30T14:16:09.661 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:09.661 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":225,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:04.689367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:09.662 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 225 2023-10-30T14:16:09.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd29b3e640 1 -- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efd100664d0 msgr2=0x7efd10068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:09.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd29b3e640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efd100664d0 0x7efd10068990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7efd18009ab0 tx=0x7efd18008040 comp rx=0 tx=0).stop 2023-10-30T14:16:09.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd29b3e640 1 -- 172.21.15.46:0/3084156790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 msgr2=0x7efd240c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:09.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd29b3e640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 0x7efd240c8980 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7efd14011980 tx=0x7efd14011e40 comp rx=0 tx=0).stop 2023-10-30T14:16:09.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd29b3e640 1 -- 172.21.15.46:0/3084156790 shutdown_connections 2023-10-30T14:16:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd29b3e640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efd100664d0 0x7efd10068990 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd29b3e640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd24159320 0x7efd241f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd29b3e640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd2415c000 0x7efd241e34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd29b3e640 1 --2- 172.21.15.46:0/3084156790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0047d0 0x7efd240c8980 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd29b3e640 1 -- 172.21.15.46:0/3084156790 >> 172.21.15.46:0/3084156790 conn(0x7efd2406bfe0 msgr2=0x7efd2406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd29b3e640 1 -- 172.21.15.46:0/3084156790 shutdown_connections 2023-10-30T14:16:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:09.657+0000 7efd29b3e640 1 -- 172.21.15.46:0/3084156790 wait complete. 2023-10-30T14:16:09.682 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:16:11.682 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.833+0000 7fcd59417640 1 Processor -- start 2023-10-30T14:16:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.833+0000 7fcd59417640 1 -- start start 2023-10-30T14:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.833+0000 7fcd59417640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd541538f0 0x7fcd54159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.833+0000 7fcd59417640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd540ca6a0 con 0x7fcd5415cd50 2023-10-30T14:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.833+0000 7fcd59417640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd5415d300 con 0x7fcd540cefa0 2023-10-30T14:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.833+0000 7fcd59417640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5415d5e0 con 0x7fcd541538f0 2023-10-30T14:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.833+0000 7fcd537fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcd5415cd50 0x7fcd540c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57740/0 (socket says 172.21.15.46:57740) 2023-10-30T14:16:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.833+0000 7fcd537fe640 1 -- 172.21.15.46:0/1227476311 learned_addr learned my addr 172.21.15.46:0/1227476311 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:16:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.833+0000 7fcd53fff640 1 --2- 172.21.15.46:0/1227476311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd541538f0 0x7fcd54159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.837+0000 7fcd52ffd640 1 -- 172.21.15.46:0/1227476311 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106924482 0 0) 0x7fcd540ca6a0 con 0x7fcd5415cd50 2023-10-30T14:16:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.837+0000 7fcd52ffd640 1 -- 172.21.15.46:0/1227476311 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd3c003530 con 0x7fcd5415cd50 2023-10-30T14:16:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.837+0000 7fcd53fff640 1 -- 172.21.15.46:0/1227476311 >> v1:172.21.15.154:6789/0 conn(0x7fcd540cefa0 legacy=0x7fcd541640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.837+0000 7fcd52ffd640 1 -- 172.21.15.46:0/1227476311 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220513914 0 0) 0x7fcd5415d300 con 0x7fcd540cefa0 2023-10-30T14:16:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.837+0000 7fcd53fff640 1 -- 172.21.15.46:0/1227476311 >> v1:172.21.15.46:6789/0 conn(0x7fcd5415cd50 legacy=0x7fcd540c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.837+0000 7fcd53fff640 1 -- 172.21.15.46:0/1227476311 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5415d8c0 con 0x7fcd541538f0 2023-10-30T14:16:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.837+0000 7fcd53fff640 1 --2- 172.21.15.46:0/1227476311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd541538f0 0x7fcd54159fb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcd440097f0 tx=0x7fcd4401ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52f42b0dec9df7bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.837+0000 7fcd52ffd640 1 -- 172.21.15.46:0/1227476311 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd440a89b0 con 0x7fcd541538f0 2023-10-30T14:16:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.837+0000 7fcd52ffd640 1 -- 172.21.15.46:0/1227476311 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd440a8b10 con 0x7fcd541538f0 2023-10-30T14:16:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd52ffd640 1 -- 172.21.15.46:0/1227476311 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd440a8e80 con 0x7fcd541538f0 2023-10-30T14:16:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 -- 172.21.15.46:0/1227476311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd541538f0 msgr2=0x7fcd54159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 --2- 172.21.15.46:0/1227476311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd541538f0 0x7fcd54159fb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcd440097f0 tx=0x7fcd4401ecb0 comp rx=0 tx=0).stop 2023-10-30T14:16:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 -- 172.21.15.46:0/1227476311 shutdown_connections 2023-10-30T14:16:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 --2- 172.21.15.46:0/1227476311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd541538f0 0x7fcd54159fb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 -- 172.21.15.46:0/1227476311 >> 172.21.15.46:0/1227476311 conn(0x7fcd5406a640 msgr2=0x7fcd54065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 -- 172.21.15.46:0/1227476311 shutdown_connections 2023-10-30T14:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 -- 172.21.15.46:0/1227476311 wait complete. 2023-10-30T14:16:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 Processor -- start 2023-10-30T14:16:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 -- start start 2023-10-30T14:16:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd540cefa0 0x7fcd540ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd541538f0 0x7fcd541ef790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5415cd50 0x7fcd541efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5415b9e0 con 0x7fcd541538f0 2023-10-30T14:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd54159660 con 0x7fcd5415cd50 2023-10-30T14:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.841+0000 7fcd59417640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd541ef300 con 0x7fcd540cefa0 2023-10-30T14:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd537fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd541538f0 0x7fcd541ef790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd537fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd541538f0 0x7fcd541ef790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44630/0 (socket says 172.21.15.46:44630) 2023-10-30T14:16:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd537fe640 1 -- 172.21.15.46:0/2155530787 learned_addr learned my addr 172.21.15.46:0/2155530787 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd53fff640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd540cefa0 0x7fcd540ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd58c16640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5415cd50 0x7fcd541efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd537fe640 1 -- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd540cefa0 msgr2=0x7fcd540ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd537fe640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd540cefa0 0x7fcd540ce6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd537fe640 1 -- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5415cd50 msgr2=0x7fcd541efcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd537fe640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5415cd50 0x7fcd541efcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd537fe640 1 -- 172.21.15.46:0/2155530787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd44009420 con 0x7fcd541538f0 2023-10-30T14:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd537fe640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd541538f0 0x7fcd541ef790 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800c3f0 tx=0x7fcd4800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:11.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd50ff9640 1 -- 172.21.15.46:0/2155530787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48005cf0 con 0x7fcd541538f0 2023-10-30T14:16:11.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd50ff9640 1 -- 172.21.15.46:0/2155530787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd48005e50 con 0x7fcd541538f0 2023-10-30T14:16:11.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd50ff9640 1 -- 172.21.15.46:0/2155530787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd480072f0 con 0x7fcd541538f0 2023-10-30T14:16:11.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd59417640 1 -- 172.21.15.46:0/2155530787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd540b3050 con 0x7fcd541538f0 2023-10-30T14:16:11.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd59417640 1 -- 172.21.15.46:0/2155530787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd541590a0 con 0x7fcd541538f0 2023-10-30T14:16:11.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd59417640 1 -- 172.21.15.46:0/2155530787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd541ef0d0 con 0x7fcd541538f0 2023-10-30T14:16:11.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.845+0000 7fcd50ff9640 1 -- 172.21.15.46:0/2155530787 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd48007450 con 0x7fcd541538f0 2023-10-30T14:16:11.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.849+0000 7fcd50ff9640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd400664d0 0x7fcd40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.849+0000 7fcd50ff9640 1 -- 172.21.15.46:0/2155530787 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7fcd48083d50 con 0x7fcd541538f0 2023-10-30T14:16:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.849+0000 7fcd53fff640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd400664d0 0x7fcd40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.849+0000 7fcd53fff640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd400664d0 0x7fcd40068990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcd440b2000 tx=0x7fcd440aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:11.853+0000 7fcd50ff9640 1 -- 172.21.15.46:0/2155530787 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd541ef0d0 con 0x7fcd541538f0 2023-10-30T14:16:12.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.077+0000 7fcd59417640 1 -- 172.21.15.46:0/2155530787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd541ef0d0 con 0x7fcd541538f0 2023-10-30T14:16:12.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.077+0000 7fcd50ff9640 1 -- 172.21.15.46:0/2155530787 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fcd541ef0d0 con 0x7fcd541538f0 2023-10-30T14:16:12.083 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:12.083 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":225,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:04.689367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:12.083 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 225 2023-10-30T14:16:12.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.081+0000 7fcd59417640 1 -- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd400664d0 msgr2=0x7fcd40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:12.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.081+0000 7fcd59417640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd400664d0 0x7fcd40068990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcd440b2000 tx=0x7fcd440aa070 comp rx=0 tx=0).stop 2023-10-30T14:16:12.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.081+0000 7fcd59417640 1 -- 172.21.15.46:0/2155530787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd541538f0 msgr2=0x7fcd541ef790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:12.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.081+0000 7fcd59417640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd541538f0 0x7fcd541ef790 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800c3f0 tx=0x7fcd4800c8b0 comp rx=0 tx=0).stop 2023-10-30T14:16:12.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.081+0000 7fcd59417640 1 -- 172.21.15.46:0/2155530787 shutdown_connections 2023-10-30T14:16:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.081+0000 7fcd59417640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd400664d0 0x7fcd40068990 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.081+0000 7fcd59417640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5415cd50 0x7fcd541efcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.081+0000 7fcd59417640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd541538f0 0x7fcd541ef790 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.081+0000 7fcd59417640 1 --2- 172.21.15.46:0/2155530787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd540cefa0 0x7fcd540ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:12.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.081+0000 7fcd59417640 1 -- 172.21.15.46:0/2155530787 >> 172.21.15.46:0/2155530787 conn(0x7fcd5406a640 msgr2=0x7fcd540caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:12.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.081+0000 7fcd59417640 1 -- 172.21.15.46:0/2155530787 shutdown_connections 2023-10-30T14:16:12.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:12.081+0000 7fcd59417640 1 -- 172.21.15.46:0/2155530787 wait complete. 2023-10-30T14:16:12.105 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:16:14.105 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7e56d6640 1 Processor -- start 2023-10-30T14:16:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7e56d6640 1 -- start start 2023-10-30T14:16:14.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7e56d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7e00c8cf0 0x7fc7e00c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:14.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7e56d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e00c90d0 0x7fc7e00ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:14.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7e56d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc7e00cef90 0x7fc7e00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:14.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7e56d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e015ce30 con 0x7fc7e00c8cf0 2023-10-30T14:16:14.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7e56d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e0159880 con 0x7fc7e00cef90 2023-10-30T14:16:14.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7e56d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e0159ae0 con 0x7fc7e00c90d0 2023-10-30T14:16:14.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7deffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7e00c8cf0 0x7fc7e00c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:14.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7deffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7e00c8cf0 0x7fc7e00c9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44638/0 (socket says 172.21.15.46:44638) 2023-10-30T14:16:14.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7deffd640 1 -- 172.21.15.46:0/3160657395 learned_addr learned my addr 172.21.15.46:0/3160657395 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:14.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7de7fc640 1 --2- 172.21.15.46:0/3160657395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e00c90d0 0x7fc7e00ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:14.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7df7fe640 1 --2- 172.21.15.46:0/3160657395 >> v2:172.21.15.154:3300/0 conn(0x7fc7e00cef90 0x7fc7e00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:14.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7deffd640 1 -- 172.21.15.46:0/3160657395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e00c90d0 msgr2=0x7fc7e00ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:14.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7deffd640 1 --2- 172.21.15.46:0/3160657395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e00c90d0 0x7fc7e00ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:14.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7deffd640 1 -- 172.21.15.46:0/3160657395 >> v2:172.21.15.154:3300/0 conn(0x7fc7e00cef90 msgr2=0x7fc7e00cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:14.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7deffd640 1 --2- 172.21.15.46:0/3160657395 >> v2:172.21.15.154:3300/0 conn(0x7fc7e00cef90 0x7fc7e00cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:14.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7deffd640 1 -- 172.21.15.46:0/3160657395 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e0159dc0 con 0x7fc7e00c8cf0 2023-10-30T14:16:14.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.249+0000 7fc7df7fe640 1 --2- 172.21.15.46:0/3160657395 >> v2:172.21.15.154:3300/0 conn(0x7fc7e00cef90 0x7fc7e00cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7deffd640 1 --2- 172.21.15.46:0/3160657395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e00c8cf0 0x7fc7e00c9e00 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fc7e006c9f0 tx=0x7fc7d001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3cc23bcef4fd7d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7ddffb640 1 -- 172.21.15.46:0/3160657395 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d00aa5f0 con 0x7fc7e00c8cf0 2023-10-30T14:16:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7ddffb640 1 -- 172.21.15.46:0/3160657395 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d00aa750 con 0x7fc7e00c8cf0 2023-10-30T14:16:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7ddffb640 1 -- 172.21.15.46:0/3160657395 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d00aaae0 con 0x7fc7e00c8cf0 2023-10-30T14:16:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7e56d6640 1 -- 172.21.15.46:0/3160657395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e00c8cf0 msgr2=0x7fc7e00c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7e56d6640 1 --2- 172.21.15.46:0/3160657395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e00c8cf0 0x7fc7e00c9e00 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fc7e006c9f0 tx=0x7fc7d001c4d0 comp rx=0 tx=0).stop 2023-10-30T14:16:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7e56d6640 1 -- 172.21.15.46:0/3160657395 shutdown_connections 2023-10-30T14:16:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7e56d6640 1 --2- 172.21.15.46:0/3160657395 >> v2:172.21.15.154:3300/0 conn(0x7fc7e00cef90 0x7fc7e00cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7e56d6640 1 --2- 172.21.15.46:0/3160657395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e00c90d0 0x7fc7e00ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7e56d6640 1 --2- 172.21.15.46:0/3160657395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e00c8cf0 0x7fc7e00c9e00 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7e56d6640 1 -- 172.21.15.46:0/3160657395 >> 172.21.15.46:0/3160657395 conn(0x7fc7e006be40 msgr2=0x7fc7e006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:14.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7e56d6640 1 -- 172.21.15.46:0/3160657395 shutdown_connections 2023-10-30T14:16:14.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.253+0000 7fc7e56d6640 1 -- 172.21.15.46:0/3160657395 wait complete. 2023-10-30T14:16:14.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7e56d6640 1 Processor -- start 2023-10-30T14:16:14.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7e56d6640 1 -- start start 2023-10-30T14:16:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7e56d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e00c8cf0 0x7fc7e0155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7e56d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e00c90d0 0x7fc7e01541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7e56d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e00cef90 0x7fc7e0156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7e56d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e01f5060 con 0x7fc7e00c90d0 2023-10-30T14:16:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7e56d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e01562c0 con 0x7fc7e00c8cf0 2023-10-30T14:16:14.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7e56d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e00b1ef0 con 0x7fc7e00cef90 2023-10-30T14:16:14.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7de7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e00c90d0 0x7fc7e01541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:14.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7de7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e00c90d0 0x7fc7e01541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44654/0 (socket says 172.21.15.46:44654) 2023-10-30T14:16:14.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7de7fc640 1 -- 172.21.15.46:0/65011712 learned_addr learned my addr 172.21.15.46:0/65011712 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:14.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7deffd640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e00c8cf0 0x7fc7e0155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:14.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7df7fe640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e00cef90 0x7fc7e0156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:14.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7de7fc640 1 -- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e00cef90 msgr2=0x7fc7e0156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:14.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7de7fc640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e00cef90 0x7fc7e0156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:14.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7de7fc640 1 -- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e00c8cf0 msgr2=0x7fc7e0155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:14.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7de7fc640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e00c8cf0 0x7fc7e0155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:14.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7de7fc640 1 -- 172.21.15.46:0/65011712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d00094d0 con 0x7fc7e00c90d0 2023-10-30T14:16:14.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7df7fe640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e00cef90 0x7fc7e0156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:14.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7de7fc640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e00c90d0 0x7fc7e01541f0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400e8b0 tx=0x7fc7d400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:14.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7c77fe640 1 -- 172.21.15.46:0/65011712 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d400c940 con 0x7fc7e00c90d0 2023-10-30T14:16:14.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7c77fe640 1 -- 172.21.15.46:0/65011712 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d400caa0 con 0x7fc7e00c90d0 2023-10-30T14:16:14.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7c77fe640 1 -- 172.21.15.46:0/65011712 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d400ce30 con 0x7fc7e00c90d0 2023-10-30T14:16:14.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7e56d6640 1 -- 172.21.15.46:0/65011712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e00b21b0 con 0x7fc7e00c90d0 2023-10-30T14:16:14.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.257+0000 7fc7e56d6640 1 -- 172.21.15.46:0/65011712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e01f5520 con 0x7fc7e00c90d0 2023-10-30T14:16:14.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.261+0000 7fc7e56d6640 1 -- 172.21.15.46:0/65011712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e004e170 con 0x7fc7e00c90d0 2023-10-30T14:16:14.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.261+0000 7fc7c77fe640 1 -- 172.21.15.46:0/65011712 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7d4013470 con 0x7fc7e00c90d0 2023-10-30T14:16:14.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.261+0000 7fc7c77fe640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7cc0664a0 0x7fc7cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:14.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.261+0000 7fc7c77fe640 1 -- 172.21.15.46:0/65011712 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7fc7d40849c0 con 0x7fc7e00c90d0 2023-10-30T14:16:14.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.261+0000 7fc7deffd640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7cc0664a0 0x7fc7cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:14.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.261+0000 7fc7deffd640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7cc0664a0 0x7fc7cc068960 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00ae000 tx=0x7fc7d00b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:14.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.265+0000 7fc7c77fe640 1 -- 172.21.15.46:0/65011712 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7d40514d0 con 0x7fc7e00c90d0 2023-10-30T14:16:14.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7e56d6640 1 -- 172.21.15.46:0/65011712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7e01574d0 con 0x7fc7e00c90d0 2023-10-30T14:16:14.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7c77fe640 1 -- 172.21.15.46:0/65011712 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fc7d4017090 con 0x7fc7e00c90d0 2023-10-30T14:16:14.485 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:14.486 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":225,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:04.689367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:14.486 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 225 2023-10-30T14:16:14.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7e56d6640 1 -- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7cc0664a0 msgr2=0x7fc7cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:14.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7e56d6640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7cc0664a0 0x7fc7cc068960 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00ae000 tx=0x7fc7d00b5040 comp rx=0 tx=0).stop 2023-10-30T14:16:14.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7e56d6640 1 -- 172.21.15.46:0/65011712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e00c90d0 msgr2=0x7fc7e01541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:14.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7e56d6640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e00c90d0 0x7fc7e01541f0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400e8b0 tx=0x7fc7d400ed70 comp rx=0 tx=0).stop 2023-10-30T14:16:14.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7e56d6640 1 -- 172.21.15.46:0/65011712 shutdown_connections 2023-10-30T14:16:14.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7e56d6640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc7cc0664a0 0x7fc7cc068960 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:14.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7e56d6640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7e00cef90 0x7fc7e0156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:14.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7e56d6640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7e00c90d0 0x7fc7e01541f0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:14.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7e56d6640 1 --2- 172.21.15.46:0/65011712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7e00c8cf0 0x7fc7e0155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:14.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7e56d6640 1 -- 172.21.15.46:0/65011712 >> 172.21.15.46:0/65011712 conn(0x7fc7e006be40 msgr2=0x7fc7e015b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:14.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.481+0000 7fc7e56d6640 1 -- 172.21.15.46:0/65011712 shutdown_connections 2023-10-30T14:16:14.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:14.485+0000 7fc7e56d6640 1 -- 172.21.15.46:0/65011712 wait complete. 2023-10-30T14:16:14.508 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:16:16.509 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:16.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.660+0000 7f7c13646640 1 Processor -- start 2023-10-30T14:16:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.660+0000 7f7c13646640 1 -- start start 2023-10-30T14:16:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.660+0000 7f7c13646640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7c0c153d70 0x7f7c0c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.660+0000 7f7c13646640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c0c8490 0x7f7c0c0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.660+0000 7f7c13646640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c06afa0 con 0x7f7c0c06e470 2023-10-30T14:16:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.660+0000 7f7c13646640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c0c06d780 con 0x7f7c0c153d70 2023-10-30T14:16:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.660+0000 7f7c13646640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c154150 con 0x7f7c0c0c8490 2023-10-30T14:16:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.660+0000 7f7c10be1640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7c0c06e470 0x7f7c0c0cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34644/0 (socket says 172.21.15.46:34644) 2023-10-30T14:16:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.660+0000 7f7c10be1640 1 -- 172.21.15.46:0/3650391557 learned_addr learned my addr 172.21.15.46:0/3650391557 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:16:16.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c11be3640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c0c8490 0x7f7c0c0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:16.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c113e2640 1 --2- 172.21.15.46:0/3650391557 >> v2:172.21.15.154:3300/0 conn(0x7f7c0c153d70 0x7f7c0c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:16.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398992324 0 0) 0x7f7c0c06afa0 con 0x7f7c0c06e470 2023-10-30T14:16:16.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bf8003530 con 0x7f7c0c06e470 2023-10-30T14:16:16.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3436295436 0 0) 0x7f7bf8003530 con 0x7f7c0c06e470 2023-10-30T14:16:16.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c0c8490 msgr2=0x7f7c0c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:16.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c0c8490 0x7f7c0c0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:16.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 >> v2:172.21.15.154:3300/0 conn(0x7f7c0c153d70 msgr2=0x7f7c0c06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:16.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 --2- 172.21.15.46:0/3650391557 >> v2:172.21.15.154:3300/0 conn(0x7f7c0c153d70 0x7f7c0c06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c06e030 con 0x7f7c0c06e470 2023-10-30T14:16:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f7bfc001680 con 0x7f7c0c06e470 2023-10-30T14:16:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 >> v1:172.21.15.46:6789/0 conn(0x7f7c0c06e470 legacy=0x7f7c0c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf8003980 0x7f7bf8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c0c8490 0x7f7bf80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:16.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf8004d10 0x7f7bf8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:16.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c06d780 con 0x7f7bf8003980 2023-10-30T14:16:16.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c154150 con 0x7f7bf8004d10 2023-10-30T14:16:16.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf800b990 con 0x7f7c0c0c8490 2023-10-30T14:16:16.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c113e2640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf8003980 0x7f7bf8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:16.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c10be1640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c0c8490 0x7f7bf80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:16.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c11be3640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf8004d10 0x7f7bf8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:16.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c113e2640 1 -- 172.21.15.46:0/3650391557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c0c8490 msgr2=0x7f7bf80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:16.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c113e2640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c0c8490 0x7f7bf80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:16.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c113e2640 1 -- 172.21.15.46:0/3650391557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf8004d10 msgr2=0x7f7bf8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c113e2640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf8004d10 0x7f7bf8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c113e2640 1 -- 172.21.15.46:0/3650391557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf8003530 con 0x7f7bf8003980 2023-10-30T14:16:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.664+0000 7f7c10be1640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c0c8490 0x7f7bf80047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:16.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c113e2640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf8003980 0x7f7bf8003da0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009ae0 tx=0x7f7c0801ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd25194ec8afdf1b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:16.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c080a5630 con 0x7f7bf8003980 2023-10-30T14:16:16.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c080a5790 con 0x7f7bf8003980 2023-10-30T14:16:16.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c03fff640 1 -- 172.21.15.46:0/3650391557 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c080a5b40 con 0x7f7bf8003980 2023-10-30T14:16:16.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c13646640 1 -- 172.21.15.46:0/3650391557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf8003980 msgr2=0x7f7bf8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c13646640 1 --2- 172.21.15.46:0/3650391557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf8003980 0x7f7bf8003da0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009ae0 tx=0x7f7c0801ca60 comp rx=0 tx=0).stop 2023-10-30T14:16:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c11be3640 1 -- 172.21.15.46:0/3650391557 reap_dead start 2023-10-30T14:16:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c13646640 1 -- 172.21.15.46:0/3650391557 shutdown_connections 2023-10-30T14:16:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c13646640 1 -- 172.21.15.46:0/3650391557 >> 172.21.15.46:0/3650391557 conn(0x7f7c0c071ea0 msgr2=0x7f7c0c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c13646640 1 -- 172.21.15.46:0/3650391557 shutdown_connections 2023-10-30T14:16:16.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c13646640 1 -- 172.21.15.46:0/3650391557 wait complete. 2023-10-30T14:16:16.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c13646640 1 Processor -- start 2023-10-30T14:16:16.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.668+0000 7f7c13646640 1 -- start start 2023-10-30T14:16:16.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c13646640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf8003980 0x7f7c0c1446c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:16.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c13646640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c1490b0 0x7f7c0c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c13646640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0c1494f0 0x7f7c0c0ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c13646640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0ae460 con 0x7f7bf8003980 2023-10-30T14:16:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c13646640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c1f2310 con 0x7f7c0c1494f0 2023-10-30T14:16:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c13646640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c1f25b0 con 0x7f7c0c1490b0 2023-10-30T14:16:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c113e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf8003980 0x7f7c0c1446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c113e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf8003980 0x7f7c0c1446c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59746/0 (socket says 172.21.15.46:59746) 2023-10-30T14:16:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c113e2640 1 -- 172.21.15.46:0/353321419 learned_addr learned my addr 172.21.15.46:0/353321419 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c10be1640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c1490b0 0x7f7c0c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c11be3640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0c1494f0 0x7f7c0c0ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c113e2640 1 -- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c1490b0 msgr2=0x7f7c0c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c113e2640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c1490b0 0x7f7c0c0ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c113e2640 1 -- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0c1494f0 msgr2=0x7f7c0c0ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c113e2640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0c1494f0 0x7f7c0c0ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c113e2640 1 -- 172.21.15.46:0/353321419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c147d20 con 0x7f7bf8003980 2023-10-30T14:16:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c113e2640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf8003980 0x7f7c0c1446c0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009ae0 tx=0x7f7bf8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c01ffb640 1 -- 172.21.15.46:0/353321419 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c080be070 con 0x7f7bf8003980 2023-10-30T14:16:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c01ffb640 1 -- 172.21.15.46:0/353321419 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0800f000 con 0x7f7bf8003980 2023-10-30T14:16:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c01ffb640 1 -- 172.21.15.46:0/353321419 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08004220 con 0x7f7bf8003980 2023-10-30T14:16:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c13646640 1 -- 172.21.15.46:0/353321419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0c148000 con 0x7f7bf8003980 2023-10-30T14:16:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c13646640 1 -- 172.21.15.46:0/353321419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0c1485c0 con 0x7f7bf8003980 2023-10-30T14:16:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c13646640 1 -- 172.21.15.46:0/353321419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0c04e170 con 0x7f7bf8003980 2023-10-30T14:16:16.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.672+0000 7f7c01ffb640 1 -- 172.21.15.46:0/353321419 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7c080b4040 con 0x7f7bf8003980 2023-10-30T14:16:16.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.676+0000 7f7c01ffb640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7bf00664d0 0x7f7bf0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:16.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.676+0000 7f7c01ffb640 1 -- 172.21.15.46:0/353321419 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7f7c0811e2b0 con 0x7f7bf8003980 2023-10-30T14:16:16.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.676+0000 7f7c10be1640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7bf00664d0 0x7f7bf0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:16.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.676+0000 7f7c10be1640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7bf00664d0 0x7f7bf0068990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c14d730 tx=0x7f7bfc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.680+0000 7f7c01ffb640 1 -- 172.21.15.46:0/353321419 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7c080eac80 con 0x7f7bf8003980 2023-10-30T14:16:16.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 -- 172.21.15.46:0/353321419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c0c06c2a0 con 0x7f7bf8003980 2023-10-30T14:16:16.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c01ffb640 1 -- 172.21.15.46:0/353321419 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f7c080ee2a0 con 0x7f7bf8003980 2023-10-30T14:16:16.877 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:16.877 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":226,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:14.615378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:resolve","state_seq":14,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:16.877 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 226 2023-10-30T14:16:16.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 -- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7bf00664d0 msgr2=0x7f7bf0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:16.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7bf00664d0 0x7f7bf0068990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c14d730 tx=0x7f7bfc009040 comp rx=0 tx=0).stop 2023-10-30T14:16:16.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 -- 172.21.15.46:0/353321419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf8003980 msgr2=0x7f7c0c1446c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:16.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf8003980 0x7f7c0c1446c0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009ae0 tx=0x7f7bf8005800 comp rx=0 tx=0).stop 2023-10-30T14:16:16.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 -- 172.21.15.46:0/353321419 shutdown_connections 2023-10-30T14:16:16.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7bf00664d0 0x7f7bf0068990 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:16.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0c1494f0 0x7f7c0c0ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:16.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0c1490b0 0x7f7c0c0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:16.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 --2- 172.21.15.46:0/353321419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bf8003980 0x7f7c0c1446c0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:16.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 -- 172.21.15.46:0/353321419 >> 172.21.15.46:0/353321419 conn(0x7f7c0c071ea0 msgr2=0x7f7c0c14f410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:16.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 -- 172.21.15.46:0/353321419 shutdown_connections 2023-10-30T14:16:16.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:16.872+0000 7f7c13646640 1 -- 172.21.15.46:0/353321419 wait complete. 2023-10-30T14:16:16.898 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:16:18.899 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:19.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.052+0000 7f43af93e640 1 Processor -- start 2023-10-30T14:16:19.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.052+0000 7f43af93e640 1 -- start start 2023-10-30T14:16:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.052+0000 7f43af93e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f43a80c6fe0 0x7f43a8071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.052+0000 7f43af93e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43a80c5f60 0x7f43a80cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.052+0000 7f43af93e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a80700b0 con 0x7f43a8153bf0 2023-10-30T14:16:19.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.052+0000 7f43af93e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a80c6810 con 0x7f43a80c6fe0 2023-10-30T14:16:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.052+0000 7f43af93e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a8159900 con 0x7f43a80c5f60 2023-10-30T14:16:19.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.056+0000 7f43adedb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43a80c5f60 0x7f43a80cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:19.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.056+0000 7f43aced9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f43a80c6fe0 0x7f43a8071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:19.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.056+0000 7f43adedb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43a80c5f60 0x7f43a80cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:60870/0 (socket says 172.21.15.46:60870) 2023-10-30T14:16:19.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.056+0000 7f43aced9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f43a80c6fe0 0x7f43a8071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:60134/0 (socket says 172.21.15.46:60134) 2023-10-30T14:16:19.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.056+0000 7f43ad6da640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f43a8153bf0 0x7f43a80c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:40478/0 (socket says 172.21.15.46:40478) 2023-10-30T14:16:19.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.056+0000 7f43adedb640 1 -- 172.21.15.46:0/565258086 learned_addr learned my addr 172.21.15.46:0/565258086 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:19.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.056+0000 7f43aced9640 1 -- 172.21.15.46:0/565258086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43a80c5f60 msgr2=0x7f43a80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:19.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.056+0000 7f43aced9640 1 --2- 172.21.15.46:0/565258086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43a80c5f60 0x7f43a80cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:19.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.056+0000 7f43aced9640 1 -- 172.21.15.46:0/565258086 >> v1:172.21.15.154:6789/0 conn(0x7f43a8153bf0 legacy=0x7f43a80c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:19.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.056+0000 7f43aced9640 1 -- 172.21.15.46:0/565258086 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a80c6530 con 0x7f43a80c6fe0 2023-10-30T14:16:19.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.056+0000 7f43adedb640 1 --2- 172.21.15.46:0/565258086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43a80c5f60 0x7f43a80cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:16:19.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.056+0000 7f43aced9640 1 --2- 172.21.15.46:0/565258086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43a80c6fe0 0x7f43a8071260 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f43980097c0 tx=0x7f439801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c1c7f72557625bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f439ffff640 1 -- 172.21.15.46:0/565258086 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43980a8970 con 0x7f43a80c6fe0 2023-10-30T14:16:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f439ffff640 1 -- 172.21.15.46:0/565258086 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43980a8ad0 con 0x7f43a80c6fe0 2023-10-30T14:16:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f439ffff640 1 -- 172.21.15.46:0/565258086 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43980a8e60 con 0x7f43a80c6fe0 2023-10-30T14:16:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 -- 172.21.15.46:0/565258086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43a80c6fe0 msgr2=0x7f43a8071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 --2- 172.21.15.46:0/565258086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43a80c6fe0 0x7f43a8071260 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f43980097c0 tx=0x7f439801ec70 comp rx=0 tx=0).stop 2023-10-30T14:16:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 -- 172.21.15.46:0/565258086 shutdown_connections 2023-10-30T14:16:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 --2- 172.21.15.46:0/565258086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43a80c5f60 0x7f43a80cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 --2- 172.21.15.46:0/565258086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43a80c6fe0 0x7f43a8071260 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 -- 172.21.15.46:0/565258086 >> 172.21.15.46:0/565258086 conn(0x7f43a806c6e0 msgr2=0x7f43a8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 -- 172.21.15.46:0/565258086 shutdown_connections 2023-10-30T14:16:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 -- 172.21.15.46:0/565258086 wait complete. 2023-10-30T14:16:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 Processor -- start 2023-10-30T14:16:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 -- start start 2023-10-30T14:16:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43a80c5f60 0x7f43a81e4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43a80c6fe0 0x7f43a81e4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.060+0000 7f43af93e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43a8153bf0 0x7f43a8149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43af93e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a814f5b0 con 0x7f43a8153bf0 2023-10-30T14:16:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43af93e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a814eac0 con 0x7f43a80c6fe0 2023-10-30T14:16:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43af93e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a81e3cb0 con 0x7f43a80c5f60 2023-10-30T14:16:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43adedb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43a8153bf0 0x7f43a8149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:19.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43adedb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43a8153bf0 0x7f43a8149dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59756/0 (socket says 172.21.15.46:59756) 2023-10-30T14:16:19.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43adedb640 1 -- 172.21.15.46:0/3520422728 learned_addr learned my addr 172.21.15.46:0/3520422728 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:19.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43aced9640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43a80c6fe0 0x7f43a81e4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:19.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43ad6da640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43a80c5f60 0x7f43a81e4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:19.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43adedb640 1 -- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43a80c5f60 msgr2=0x7f43a81e4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:19.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43adedb640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43a80c5f60 0x7f43a81e4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:19.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43adedb640 1 -- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43a80c6fe0 msgr2=0x7f43a81e4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:19.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43adedb640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43a80c6fe0 0x7f43a81e4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:19.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43adedb640 1 -- 172.21.15.46:0/3520422728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a0009b30 con 0x7f43a8153bf0 2023-10-30T14:16:19.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43adedb640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43a8153bf0 0x7f43a8149dd0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009e70 tx=0x7f43a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:19.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f439dffb640 1 -- 172.21.15.46:0/3520422728 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a0018070 con 0x7f43a8153bf0 2023-10-30T14:16:19.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f439dffb640 1 -- 172.21.15.46:0/3520422728 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a000f040 con 0x7f43a8153bf0 2023-10-30T14:16:19.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43af93e640 1 -- 172.21.15.46:0/3520422728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4398009420 con 0x7f43a8153bf0 2023-10-30T14:16:19.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f439dffb640 1 -- 172.21.15.46:0/3520422728 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a000ce00 con 0x7f43a8153bf0 2023-10-30T14:16:19.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43af93e640 1 -- 172.21.15.46:0/3520422728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a814ed10 con 0x7f43a8153bf0 2023-10-30T14:16:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f43af93e640 1 -- 172.21.15.46:0/3520422728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a804e170 con 0x7f43a8153bf0 2023-10-30T14:16:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.064+0000 7f439dffb640 1 -- 172.21.15.46:0/3520422728 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f43a0014480 con 0x7f43a8153bf0 2023-10-30T14:16:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.068+0000 7f439dffb640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f438c0664a0 0x7f438c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.068+0000 7f439dffb640 1 -- 172.21.15.46:0/3520422728 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7f43a0084af0 con 0x7f43a8153bf0 2023-10-30T14:16:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.068+0000 7f43ad6da640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f438c0664a0 0x7f438c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.068+0000 7f43ad6da640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f438c0664a0 0x7f438c068960 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f43a4000f70 tx=0x7f43a4007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.072+0000 7f439dffb640 1 -- 172.21.15.46:0/3520422728 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f43a00514c0 con 0x7f43a8153bf0 2023-10-30T14:16:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.260+0000 7f43af93e640 1 -- 172.21.15.46:0/3520422728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43a81e3df0 con 0x7f43a8153bf0 2023-10-30T14:16:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.260+0000 7f439dffb640 1 -- 172.21.15.46:0/3520422728 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f43a0054ae0 con 0x7f43a8153bf0 2023-10-30T14:16:19.266 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:19.266 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":226,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:14.615378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:resolve","state_seq":14,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 226 2023-10-30T14:16:19.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.260+0000 7f43af93e640 1 -- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f438c0664a0 msgr2=0x7f438c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:19.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.260+0000 7f43af93e640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f438c0664a0 0x7f438c068960 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f43a4000f70 tx=0x7f43a4007c90 comp rx=0 tx=0).stop 2023-10-30T14:16:19.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.260+0000 7f43af93e640 1 -- 172.21.15.46:0/3520422728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43a8153bf0 msgr2=0x7f43a8149dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:19.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.260+0000 7f43af93e640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43a8153bf0 0x7f43a8149dd0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009e70 tx=0x7f43a000c490 comp rx=0 tx=0).stop 2023-10-30T14:16:19.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.260+0000 7f43af93e640 1 -- 172.21.15.46:0/3520422728 shutdown_connections 2023-10-30T14:16:19.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.260+0000 7f43af93e640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f438c0664a0 0x7f438c068960 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:19.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.260+0000 7f43af93e640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43a8153bf0 0x7f43a8149dd0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:19.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.260+0000 7f43af93e640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43a80c6fe0 0x7f43a81e4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:19.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.260+0000 7f43af93e640 1 --2- 172.21.15.46:0/3520422728 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43a80c5f60 0x7f43a81e4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:19.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.260+0000 7f43af93e640 1 -- 172.21.15.46:0/3520422728 >> 172.21.15.46:0/3520422728 conn(0x7f43a806c6e0 msgr2=0x7f43a8159000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:19.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.264+0000 7f43af93e640 1 -- 172.21.15.46:0/3520422728 shutdown_connections 2023-10-30T14:16:19.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:19.264+0000 7f43af93e640 1 -- 172.21.15.46:0/3520422728 wait complete. 2023-10-30T14:16:19.287 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 226, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 226, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9135": { "addr": "172.21.15.46:6837/681817982", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 681817982, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 681817982, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 9135, "incarnation": 218, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 42 }, "gid_9276": { "addr": "172.21.15.46:6836/2239735286", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 2239735286, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 2239735286, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9276, "incarnation": 225, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:resolve", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:16:14.615378+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9135, "mds_1": 9276 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:16:21.289 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:21.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f6214646640 1 Processor -- start 2023-10-30T14:16:21.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f6214646640 1 -- start start 2023-10-30T14:16:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f6214646640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f620c15d6f0 0x7f620c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f6214646640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620c0cb800 con 0x7f620c1538a0 2023-10-30T14:16:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f6214646640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620c0cb330 con 0x7f620c0cf160 2023-10-30T14:16:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f6214646640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0cad70 con 0x7f620c15d6f0 2023-10-30T14:16:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62123e2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f620c1538a0 0x7f620c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34658/0 (socket says 172.21.15.46:34658) 2023-10-30T14:16:21.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62123e2640 1 -- 172.21.15.46:0/3804140990 learned_addr learned my addr 172.21.15.46:0/3804140990 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:16:21.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f6211be1640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f620c15d6f0 0x7f620c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:21.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054790678 0 0) 0x7f620c0cb800 con 0x7f620c1538a0 2023-10-30T14:16:21.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f8003530 con 0x7f620c1538a0 2023-10-30T14:16:21.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943834475 0 0) 0x7f620c0cb330 con 0x7f620c0cf160 2023-10-30T14:16:21.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f620c0cb800 con 0x7f620c0cf160 2023-10-30T14:16:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 538044664 0 0) 0x7f61f8003530 con 0x7f620c1538a0 2023-10-30T14:16:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f620c15d6f0 msgr2=0x7f620c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f620c15d6f0 0x7f620c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:21.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 >> v1:172.21.15.154:6789/0 conn(0x7f620c0cf160 legacy=0x7f620c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c0cb050 con 0x7f620c1538a0 2023-10-30T14:16:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f6208001680 con 0x7f620c1538a0 2023-10-30T14:16:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 >> v1:172.21.15.46:6789/0 conn(0x7f620c1538a0 legacy=0x7f620c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61f8003b40 0x7f61f8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61f80047d0 0x7f61f8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f620c15d6f0 0x7f61f8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0cad70 con 0x7f61f80047d0 2023-10-30T14:16:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f800ba70 con 0x7f61f8003b40 2023-10-30T14:16:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.440+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f800bc30 con 0x7f620c15d6f0 2023-10-30T14:16:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f6211be1640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61f80047d0 0x7f61f8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:21.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f62123e2640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61f8003b40 0x7f61f8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:21.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f6212be3640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f620c15d6f0 0x7f61f8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:21.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f6211be1640 1 -- 172.21.15.46:0/3804140990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f620c15d6f0 msgr2=0x7f61f8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:21.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f6211be1640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f620c15d6f0 0x7f61f8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:21.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f6211be1640 1 -- 172.21.15.46:0/3804140990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61f8003b40 msgr2=0x7f61f8003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:21.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f6211be1640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61f8003b40 0x7f61f8003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:21.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f6211be1640 1 -- 172.21.15.46:0/3804140990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8003530 con 0x7f61f80047d0 2023-10-30T14:16:21.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f6212be3640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f620c15d6f0 0x7f61f8005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:21.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f6211be1640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61f80047d0 0x7f61f8004c50 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009ae0 tx=0x7f61fc01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c38b133ab1340e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:21.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0a5530 con 0x7f61f80047d0 2023-10-30T14:16:21.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61fc0a5690 con 0x7f61f80047d0 2023-10-30T14:16:21.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.444+0000 7f62113e0640 1 -- 172.21.15.46:0/3804140990 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0a5a40 con 0x7f61f80047d0 2023-10-30T14:16:21.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 -- 172.21.15.46:0/3804140990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61f80047d0 msgr2=0x7f61f8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:21.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 --2- 172.21.15.46:0/3804140990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61f80047d0 0x7f61f8004c50 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009ae0 tx=0x7f61fc01ca50 comp rx=0 tx=0).stop 2023-10-30T14:16:21.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6212be3640 1 -- 172.21.15.46:0/3804140990 reap_dead start 2023-10-30T14:16:21.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 -- 172.21.15.46:0/3804140990 shutdown_connections 2023-10-30T14:16:21.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 -- 172.21.15.46:0/3804140990 >> 172.21.15.46:0/3804140990 conn(0x7f620c06bfe0 msgr2=0x7f620c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:21.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 -- 172.21.15.46:0/3804140990 shutdown_connections 2023-10-30T14:16:21.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 -- 172.21.15.46:0/3804140990 wait complete. 2023-10-30T14:16:21.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 Processor -- start 2023-10-30T14:16:21.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 -- start start 2023-10-30T14:16:21.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61f80047d0 0x7f620c0b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:21.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f620c0c9370 0x7f620c1e3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:21.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f620c0c7af0 0x7f620c1e4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:21.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c0c90b0 con 0x7f620c0c7af0 2023-10-30T14:16:21.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c1e2ae0 con 0x7f620c0c9370 2023-10-30T14:16:21.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6214646640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620c1e3940 con 0x7f61f80047d0 2023-10-30T14:16:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6212be3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f620c0c7af0 0x7f620c1e4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6212be3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f620c0c7af0 0x7f620c1e4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59764/0 (socket says 172.21.15.46:59764) 2023-10-30T14:16:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6212be3640 1 -- 172.21.15.46:0/2341192788 learned_addr learned my addr 172.21.15.46:0/2341192788 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6211be1640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f620c0c9370 0x7f620c1e3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f62123e2640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61f80047d0 0x7f620c0b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6212be3640 1 -- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61f80047d0 msgr2=0x7f620c0b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6212be3640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61f80047d0 0x7f620c0b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6212be3640 1 -- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f620c0c9370 msgr2=0x7f620c1e3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6212be3640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f620c0c9370 0x7f620c1e3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.448+0000 7f6212be3640 1 -- 172.21.15.46:0/2341192788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c1e2d30 con 0x7f620c0c7af0 2023-10-30T14:16:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.452+0000 7f6212be3640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f620c0c7af0 0x7f620c1e4490 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f6204007e70 tx=0x7f620400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.452+0000 7f6202ffd640 1 -- 172.21.15.46:0/2341192788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6204006b60 con 0x7f620c0c7af0 2023-10-30T14:16:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.452+0000 7f6202ffd640 1 -- 172.21.15.46:0/2341192788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6204006cc0 con 0x7f620c0c7af0 2023-10-30T14:16:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.452+0000 7f6214646640 1 -- 172.21.15.46:0/2341192788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620c1e3010 con 0x7f620c0c7af0 2023-10-30T14:16:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.452+0000 7f6202ffd640 1 -- 172.21.15.46:0/2341192788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6204003bc0 con 0x7f620c0c7af0 2023-10-30T14:16:21.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.452+0000 7f6214646640 1 -- 172.21.15.46:0/2341192788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620c1e35d0 con 0x7f620c0c7af0 2023-10-30T14:16:21.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.452+0000 7f6214646640 1 -- 172.21.15.46:0/2341192788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620c04e170 con 0x7f620c0c7af0 2023-10-30T14:16:21.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.452+0000 7f6202ffd640 1 -- 172.21.15.46:0/2341192788 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6204003d20 con 0x7f620c0c7af0 2023-10-30T14:16:21.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.452+0000 7f6202ffd640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f61f00664d0 0x7f61f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:21.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.452+0000 7f6202ffd640 1 -- 172.21.15.46:0/2341192788 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7f6204084320 con 0x7f620c0c7af0 2023-10-30T14:16:21.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.452+0000 7f62123e2640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f61f00664d0 0x7f61f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:21.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.456+0000 7f62123e2640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f61f00664d0 0x7f61f0068990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6208001060 tx=0x7f6208009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:21.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.456+0000 7f6202ffd640 1 -- 172.21.15.46:0/2341192788 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6204050cf0 con 0x7f620c0c7af0 2023-10-30T14:16:21.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.648+0000 7f6214646640 1 -- 172.21.15.46:0/2341192788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f620c1e3a80 con 0x7f620c0c7af0 2023-10-30T14:16:21.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.648+0000 7f6202ffd640 1 -- 172.21.15.46:0/2341192788 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f6204054310 con 0x7f620c0c7af0 2023-10-30T14:16:21.654 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:21.655 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:20.112642+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:21.655 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 227 2023-10-30T14:16:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.652+0000 7f6214646640 1 -- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f61f00664d0 msgr2=0x7f61f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.652+0000 7f6214646640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f61f00664d0 0x7f61f0068990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6208001060 tx=0x7f6208009040 comp rx=0 tx=0).stop 2023-10-30T14:16:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.652+0000 7f6214646640 1 -- 172.21.15.46:0/2341192788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f620c0c7af0 msgr2=0x7f620c1e4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.652+0000 7f6214646640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f620c0c7af0 0x7f620c1e4490 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f6204007e70 tx=0x7f620400eb90 comp rx=0 tx=0).stop 2023-10-30T14:16:21.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.652+0000 7f6214646640 1 -- 172.21.15.46:0/2341192788 shutdown_connections 2023-10-30T14:16:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.652+0000 7f6214646640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f61f00664d0 0x7f61f0068990 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.652+0000 7f6214646640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f620c0c7af0 0x7f620c1e4490 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.652+0000 7f6214646640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f620c0c9370 0x7f620c1e3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:21.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.652+0000 7f6214646640 1 --2- 172.21.15.46:0/2341192788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61f80047d0 0x7f620c0b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:21.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.652+0000 7f6214646640 1 -- 172.21.15.46:0/2341192788 >> 172.21.15.46:0/2341192788 conn(0x7f620c06bfe0 msgr2=0x7f620c15b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:21.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.652+0000 7f6214646640 1 -- 172.21.15.46:0/2341192788 shutdown_connections 2023-10-30T14:16:21.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:21.652+0000 7f6214646640 1 -- 172.21.15.46:0/2341192788 wait complete. 2023-10-30T14:16:21.676 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:16:23.677 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:23.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.828+0000 7f0902380640 1 Processor -- start 2023-10-30T14:16:23.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.828+0000 7f0902380640 1 -- start start 2023-10-30T14:16:23.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.828+0000 7f0902380640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08fc071470 0x7f08fc158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:23.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.828+0000 7f0902380640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08fc071a70 0x7f08fc159530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:23.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.828+0000 7f0902380640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08fc0c6050 con 0x7f08fc153200 2023-10-30T14:16:23.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.828+0000 7f0902380640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08fc0c6e50 con 0x7f08fc071470 2023-10-30T14:16:23.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.828+0000 7f0902380640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0c62a0 con 0x7f08fc071a70 2023-10-30T14:16:23.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.828+0000 7f08fb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08fc071470 0x7f08fc158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:23.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.828+0000 7f08fb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08fc071470 0x7f08fc158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59774/0 (socket says 172.21.15.46:59774) 2023-10-30T14:16:23.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.828+0000 7f08fb7fe640 1 -- 172.21.15.46:0/593056965 learned_addr learned my addr 172.21.15.46:0/593056965 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.828+0000 7f090091d640 1 --2- 172.21.15.46:0/593056965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08fc071a70 0x7f08fc159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.832+0000 7f08fb7fe640 1 -- 172.21.15.46:0/593056965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08fc071a70 msgr2=0x7f08fc159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.832+0000 7f08fb7fe640 1 --2- 172.21.15.46:0/593056965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08fc071a70 0x7f08fc159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:23.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.832+0000 7f08fb7fe640 1 -- 172.21.15.46:0/593056965 >> v1:172.21.15.46:6789/0 conn(0x7f08fc153200 legacy=0x7f08fc0c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:23.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.832+0000 7f08fb7fe640 1 -- 172.21.15.46:0/593056965 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc0c70f0 con 0x7f08fc071470 2023-10-30T14:16:23.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.832+0000 7f090091d640 1 --2- 172.21.15.46:0/593056965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08fc071a70 0x7f08fc159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:23.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.832+0000 7f08fb7fe640 1 --2- 172.21.15.46:0/593056965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08fc071470 0x7f08fc158ff0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009870 tx=0x7f08f001cfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d5d768670d138f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:23.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.832+0000 7f08faffd640 1 -- 172.21.15.46:0/593056965 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f00a9620 con 0x7f08fc071470 2023-10-30T14:16:23.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.832+0000 7f08faffd640 1 -- 172.21.15.46:0/593056965 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f00a9780 con 0x7f08fc071470 2023-10-30T14:16:23.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.832+0000 7f08faffd640 1 -- 172.21.15.46:0/593056965 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f00a9b10 con 0x7f08fc071470 2023-10-30T14:16:23.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 -- 172.21.15.46:0/593056965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08fc071470 msgr2=0x7f08fc158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:23.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 --2- 172.21.15.46:0/593056965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08fc071470 0x7f08fc158ff0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009870 tx=0x7f08f001cfa0 comp rx=0 tx=0).stop 2023-10-30T14:16:23.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 -- 172.21.15.46:0/593056965 shutdown_connections 2023-10-30T14:16:23.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 --2- 172.21.15.46:0/593056965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08fc071a70 0x7f08fc159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:23.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 --2- 172.21.15.46:0/593056965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08fc071470 0x7f08fc158ff0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:23.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 -- 172.21.15.46:0/593056965 >> 172.21.15.46:0/593056965 conn(0x7f08fc06d770 msgr2=0x7f08fc06db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:23.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 -- 172.21.15.46:0/593056965 shutdown_connections 2023-10-30T14:16:23.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 -- 172.21.15.46:0/593056965 wait complete. 2023-10-30T14:16:23.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 Processor -- start 2023-10-30T14:16:23.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 -- start start 2023-10-30T14:16:23.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08fc071470 0x7f08fc14fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:23.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08fc071a70 0x7f08fc1f9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:23.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08fc153200 0x7f08fc1f9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:23.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc1541c0 con 0x7f08fc071a70 2023-10-30T14:16:23.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc1586a0 con 0x7f08fc071470 2023-10-30T14:16:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f0902380640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc1588f0 con 0x7f08fc153200 2023-10-30T14:16:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f08fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08fc071a70 0x7f08fc1f9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f08fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08fc071a70 0x7f08fc1f9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59782/0 (socket says 172.21.15.46:59782) 2023-10-30T14:16:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f08fb7fe640 1 -- 172.21.15.46:0/1160412189 learned_addr learned my addr 172.21.15.46:0/1160412189 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f08fbfff640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08fc071470 0x7f08fc14fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:23.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f090091d640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08fc153200 0x7f08fc1f9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:23.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f08fb7fe640 1 -- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08fc153200 msgr2=0x7f08fc1f9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:23.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f08fb7fe640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08fc153200 0x7f08fc1f9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:23.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f08fb7fe640 1 -- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08fc071470 msgr2=0x7f08fc14fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:23.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f08fb7fe640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08fc071470 0x7f08fc14fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:23.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.836+0000 7f08fb7fe640 1 -- 172.21.15.46:0/1160412189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f00094d0 con 0x7f08fc071a70 2023-10-30T14:16:23.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.840+0000 7f08fb7fe640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08fc071a70 0x7f08fc1f9680 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f08f001abf0 tx=0x7f08f00192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:23.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.840+0000 7f08f8ff9640 1 -- 172.21.15.46:0/1160412189 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f00bb3f0 con 0x7f08fc071a70 2023-10-30T14:16:23.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.840+0000 7f08f8ff9640 1 -- 172.21.15.46:0/1160412189 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f00bb550 con 0x7f08fc071a70 2023-10-30T14:16:23.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.840+0000 7f0902380640 1 -- 172.21.15.46:0/1160412189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08fc158bd0 con 0x7f08fc071a70 2023-10-30T14:16:23.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.840+0000 7f08f8ff9640 1 -- 172.21.15.46:0/1160412189 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f00bb900 con 0x7f08fc071a70 2023-10-30T14:16:23.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.840+0000 7f0902380640 1 -- 172.21.15.46:0/1160412189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08fc153c00 con 0x7f08fc071a70 2023-10-30T14:16:23.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.840+0000 7f0902380640 1 -- 172.21.15.46:0/1160412189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08fc074250 con 0x7f08fc071a70 2023-10-30T14:16:23.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.840+0000 7f08f8ff9640 1 -- 172.21.15.46:0/1160412189 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f08f00bba60 con 0x7f08fc071a70 2023-10-30T14:16:23.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.840+0000 7f08f8ff9640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f08e80664a0 0x7f08e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:23.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.840+0000 7f08f8ff9640 1 -- 172.21.15.46:0/1160412189 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7f08f011ede0 con 0x7f08fc071a70 2023-10-30T14:16:23.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.844+0000 7f08fbfff640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f08e80664a0 0x7f08e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:23.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.844+0000 7f08fbfff640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f08e80664a0 0x7f08e8068960 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f08ec007fd0 tx=0x7f08ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:23.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:23.844+0000 7f08f8ff9640 1 -- 172.21.15.46:0/1160412189 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f08f00eb7b0 con 0x7f08fc071a70 2023-10-30T14:16:24.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f0902380640 1 -- 172.21.15.46:0/1160412189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08fc0c6800 con 0x7f08fc071a70 2023-10-30T14:16:24.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f08f8ff9640 1 -- 172.21.15.46:0/1160412189 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f08f00eedd0 con 0x7f08fc071a70 2023-10-30T14:16:24.041 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:24.042 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:20.112642+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:24.042 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 227 2023-10-30T14:16:24.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f0902380640 1 -- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f08e80664a0 msgr2=0x7f08e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:24.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f0902380640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f08e80664a0 0x7f08e8068960 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f08ec007fd0 tx=0x7f08ec009040 comp rx=0 tx=0).stop 2023-10-30T14:16:24.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f0902380640 1 -- 172.21.15.46:0/1160412189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08fc071a70 msgr2=0x7f08fc1f9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:24.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f0902380640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08fc071a70 0x7f08fc1f9680 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f08f001abf0 tx=0x7f08f00192e0 comp rx=0 tx=0).stop 2023-10-30T14:16:24.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f0902380640 1 -- 172.21.15.46:0/1160412189 shutdown_connections 2023-10-30T14:16:24.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f0902380640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f08e80664a0 0x7f08e8068960 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:24.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f0902380640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08fc153200 0x7f08fc1f9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:24.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f0902380640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08fc071a70 0x7f08fc1f9680 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:24.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f0902380640 1 --2- 172.21.15.46:0/1160412189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08fc071470 0x7f08fc14fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:24.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f0902380640 1 -- 172.21.15.46:0/1160412189 >> 172.21.15.46:0/1160412189 conn(0x7f08fc06d770 msgr2=0x7f08fc14e010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:24.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.036+0000 7f0902380640 1 -- 172.21.15.46:0/1160412189 shutdown_connections 2023-10-30T14:16:24.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:24.040+0000 7f0902380640 1 -- 172.21.15.46:0/1160412189 wait complete. 2023-10-30T14:16:24.066 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:16:26.067 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:26.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.216+0000 7f48b150e640 1 Processor -- start 2023-10-30T14:16:26.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.216+0000 7f48b150e640 1 -- start start 2023-10-30T14:16:26.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.216+0000 7f48b150e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48ac06f7a0 0x7f48ac06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:26.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.216+0000 7f48b150e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f48ac0c6bd0 0x7f48ac069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:26.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.216+0000 7f48b150e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac06a3d0 con 0x7f48ac071110 2023-10-30T14:16:26.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.216+0000 7f48b150e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ac0700c0 con 0x7f48ac06f7a0 2023-10-30T14:16:26.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.216+0000 7f48b150e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ac066cf0 con 0x7f48ac0c6bd0 2023-10-30T14:16:26.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.216+0000 7f48aaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48ac06f7a0 0x7f48ac06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:26.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.216+0000 7f48aaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48ac06f7a0 0x7f48ac06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59786/0 (socket says 172.21.15.46:59786) 2023-10-30T14:16:26.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.216+0000 7f48aaffd640 1 -- 172.21.15.46:0/3235656597 learned_addr learned my addr 172.21.15.46:0/3235656597 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:26.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.216+0000 7f48a3fff640 1 --2- 172.21.15.46:0/3235656597 >> v2:172.21.15.154:3300/0 conn(0x7f48ac0c6bd0 0x7f48ac069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:26.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48aaffd640 1 -- 172.21.15.46:0/3235656597 >> v2:172.21.15.154:3300/0 conn(0x7f48ac0c6bd0 msgr2=0x7f48ac069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:26.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48aaffd640 1 --2- 172.21.15.46:0/3235656597 >> v2:172.21.15.154:3300/0 conn(0x7f48ac0c6bd0 0x7f48ac069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:26.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48aaffd640 1 -- 172.21.15.46:0/3235656597 >> v1:172.21.15.46:6789/0 conn(0x7f48ac071110 legacy=0x7f48ac158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:26.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48aaffd640 1 -- 172.21.15.46:0/3235656597 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac0c70d0 con 0x7f48ac06f7a0 2023-10-30T14:16:26.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48aa7fc640 1 -- 172.21.15.46:0/3235656597 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615168504 0 0) 0x7f48ac06a3d0 con 0x7f48ac071110 2023-10-30T14:16:26.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48aaffd640 1 --2- 172.21.15.46:0/3235656597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48ac06f7a0 0x7f48ac06fb80 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f48ac06f1c0 tx=0x7f489c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19277324f5539326 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:26.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48aa7fc640 1 -- 172.21.15.46:0/3235656597 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c0a89f0 con 0x7f48ac06f7a0 2023-10-30T14:16:26.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48aa7fc640 1 -- 172.21.15.46:0/3235656597 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489c0a8b50 con 0x7f48ac06f7a0 2023-10-30T14:16:26.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48aa7fc640 1 -- 172.21.15.46:0/3235656597 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c0b0440 con 0x7f48ac06f7a0 2023-10-30T14:16:26.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48b150e640 1 -- 172.21.15.46:0/3235656597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48ac06f7a0 msgr2=0x7f48ac06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:26.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48b150e640 1 --2- 172.21.15.46:0/3235656597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48ac06f7a0 0x7f48ac06fb80 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f48ac06f1c0 tx=0x7f489c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:16:26.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48b150e640 1 -- 172.21.15.46:0/3235656597 shutdown_connections 2023-10-30T14:16:26.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48b150e640 1 --2- 172.21.15.46:0/3235656597 >> v2:172.21.15.154:3300/0 conn(0x7f48ac0c6bd0 0x7f48ac069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:26.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48b150e640 1 --2- 172.21.15.46:0/3235656597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48ac06f7a0 0x7f48ac06fb80 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:26.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48b150e640 1 -- 172.21.15.46:0/3235656597 >> 172.21.15.46:0/3235656597 conn(0x7f48ac0c7910 msgr2=0x7f48ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:26.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.220+0000 7f48b150e640 1 -- 172.21.15.46:0/3235656597 shutdown_connections 2023-10-30T14:16:26.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48b150e640 1 -- 172.21.15.46:0/3235656597 wait complete. 2023-10-30T14:16:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48b150e640 1 Processor -- start 2023-10-30T14:16:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48b150e640 1 -- start start 2023-10-30T14:16:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48b150e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48ac06f7a0 0x7f48ac0af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48b150e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48ac071110 0x7f48ac1eed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48b150e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48ac0c6bd0 0x7f48ac1ef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48b150e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac158540 con 0x7f48ac071110 2023-10-30T14:16:26.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48b150e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac1534b0 con 0x7f48ac0c6bd0 2023-10-30T14:16:26.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48b150e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac158260 con 0x7f48ac06f7a0 2023-10-30T14:16:26.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48ac071110 0x7f48ac1eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:26.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48ac071110 0x7f48ac1eed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59788/0 (socket says 172.21.15.46:59788) 2023-10-30T14:16:26.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48a3fff640 1 -- 172.21.15.46:0/2527612397 learned_addr learned my addr 172.21.15.46:0/2527612397 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:26.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48aaffd640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48ac06f7a0 0x7f48ac0af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:26.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48ab7fe640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48ac0c6bd0 0x7f48ac1ef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:26.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48a3fff640 1 -- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48ac06f7a0 msgr2=0x7f48ac0af090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:26.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48a3fff640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48ac06f7a0 0x7f48ac0af090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:26.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48a3fff640 1 -- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48ac0c6bd0 msgr2=0x7f48ac1ef2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48a3fff640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48ac0c6bd0 0x7f48ac1ef2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48a3fff640 1 -- 172.21.15.46:0/2527612397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c0094d0 con 0x7f48ac071110 2023-10-30T14:16:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48ab7fe640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48ac0c6bd0 0x7f48ac1ef2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48a3fff640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48ac071110 0x7f48ac1eed70 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f489400d5d0 tx=0x7f489400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48a37fe640 1 -- 172.21.15.46:0/2527612397 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4894004240 con 0x7f48ac071110 2023-10-30T14:16:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48a37fe640 1 -- 172.21.15.46:0/2527612397 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48940043a0 con 0x7f48ac071110 2023-10-30T14:16:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48a37fe640 1 -- 172.21.15.46:0/2527612397 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4894003840 con 0x7f48ac071110 2023-10-30T14:16:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.224+0000 7f48b150e640 1 -- 172.21.15.46:0/2527612397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ac157930 con 0x7f48ac071110 2023-10-30T14:16:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.228+0000 7f48b150e640 1 -- 172.21.15.46:0/2527612397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ac157ef0 con 0x7f48ac071110 2023-10-30T14:16:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.228+0000 7f48b150e640 1 -- 172.21.15.46:0/2527612397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ac04e170 con 0x7f48ac071110 2023-10-30T14:16:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.228+0000 7f48a37fe640 1 -- 172.21.15.46:0/2527612397 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f48940039a0 con 0x7f48ac071110 2023-10-30T14:16:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.228+0000 7f48a37fe640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f488c0664a0 0x7f488c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.228+0000 7f48a37fe640 1 -- 172.21.15.46:0/2527612397 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7f48940849f0 con 0x7f48ac071110 2023-10-30T14:16:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.228+0000 7f48aaffd640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f488c0664a0 0x7f488c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.228+0000 7f48aaffd640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f488c0664a0 0x7f488c068960 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f489c0b2000 tx=0x7f489c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:26.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.232+0000 7f48a37fe640 1 -- 172.21.15.46:0/2527612397 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4894051420 con 0x7f48ac071110 2023-10-30T14:16:26.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.420+0000 7f48b150e640 1 -- 172.21.15.46:0/2527612397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48ac0b06c0 con 0x7f48ac071110 2023-10-30T14:16:26.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48a37fe640 1 -- 172.21.15.46:0/2527612397 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f4894054a40 con 0x7f48ac071110 2023-10-30T14:16:26.428 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:26.428 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":228,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:26.038434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:26.428 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 228 2023-10-30T14:16:26.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48b150e640 1 -- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f488c0664a0 msgr2=0x7f488c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:26.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48b150e640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f488c0664a0 0x7f488c068960 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f489c0b2000 tx=0x7f489c009040 comp rx=0 tx=0).stop 2023-10-30T14:16:26.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48b150e640 1 -- 172.21.15.46:0/2527612397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48ac071110 msgr2=0x7f48ac1eed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:26.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48b150e640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48ac071110 0x7f48ac1eed70 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f489400d5d0 tx=0x7f489400da90 comp rx=0 tx=0).stop 2023-10-30T14:16:26.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48b150e640 1 -- 172.21.15.46:0/2527612397 shutdown_connections 2023-10-30T14:16:26.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48b150e640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f488c0664a0 0x7f488c068960 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:26.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48b150e640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48ac0c6bd0 0x7f48ac1ef2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:26.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48b150e640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48ac071110 0x7f48ac1eed70 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:26.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48b150e640 1 --2- 172.21.15.46:0/2527612397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48ac06f7a0 0x7f48ac0af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:26.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48b150e640 1 -- 172.21.15.46:0/2527612397 >> 172.21.15.46:0/2527612397 conn(0x7f48ac0c7910 msgr2=0x7f48ac0c8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:26.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48b150e640 1 -- 172.21.15.46:0/2527612397 shutdown_connections 2023-10-30T14:16:26.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:26.424+0000 7f48b150e640 1 -- 172.21.15.46:0/2527612397 wait complete. 2023-10-30T14:16:26.449 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:16:28.399 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:16:28.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:16:28.450 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:28.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.600+0000 7faf0f2c9640 1 Processor -- start 2023-10-30T14:16:28.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.600+0000 7faf0f2c9640 1 -- start start 2023-10-30T14:16:28.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.600+0000 7faf0f2c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faf08070740 0x7faf080c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:28.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.600+0000 7faf0f2c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf0815e700 0x7faf0806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:28.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.600+0000 7faf0f2c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf080c5810 con 0x7faf081580e0 2023-10-30T14:16:28.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.600+0000 7faf0f2c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf0815eb70 con 0x7faf0815e700 2023-10-30T14:16:28.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.600+0000 7faf0f2c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf080c6bb0 con 0x7faf08070740 2023-10-30T14:16:28.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faf0d866640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf0815e700 0x7faf0806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:28.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faf0d866640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf0815e700 0x7faf0806a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57660/0 (socket says 172.21.15.46:57660) 2023-10-30T14:16:28.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faf0d866640 1 -- 172.21.15.46:0/2564256605 learned_addr learned my addr 172.21.15.46:0/2564256605 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:28.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faf0c864640 1 --2- 172.21.15.46:0/2564256605 >> v2:172.21.15.154:3300/0 conn(0x7faf08070740 0x7faf080c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:28.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faf0d866640 1 -- 172.21.15.46:0/2564256605 >> v2:172.21.15.154:3300/0 conn(0x7faf08070740 msgr2=0x7faf080c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:28.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faf0d866640 1 --2- 172.21.15.46:0/2564256605 >> v2:172.21.15.154:3300/0 conn(0x7faf08070740 0x7faf080c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:28.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faf0d866640 1 -- 172.21.15.46:0/2564256605 >> v1:172.21.15.154:6789/0 conn(0x7faf081580e0 legacy=0x7faf08069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:28.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faf0d866640 1 -- 172.21.15.46:0/2564256605 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf080c6e70 con 0x7faf0815e700 2023-10-30T14:16:28.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faf0c864640 1 --2- 172.21.15.46:0/2564256605 >> v2:172.21.15.154:3300/0 conn(0x7faf08070740 0x7faf080c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:28.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faf0d866640 1 --2- 172.21.15.46:0/2564256605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0815e700 0x7faf0806a5a0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7faf0806b1a0 tx=0x7faf0001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ebd53ac24aea66 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:28.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faefffff640 1 -- 172.21.15.46:0/2564256605 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf000a5780 con 0x7faf0815e700 2023-10-30T14:16:28.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faefffff640 1 -- 172.21.15.46:0/2564256605 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf000a58e0 con 0x7faf0815e700 2023-10-30T14:16:28.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.604+0000 7faefffff640 1 -- 172.21.15.46:0/2564256605 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf000a5c70 con 0x7faf0815e700 2023-10-30T14:16:28.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 -- 172.21.15.46:0/2564256605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0815e700 msgr2=0x7faf0806a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:28.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 --2- 172.21.15.46:0/2564256605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0815e700 0x7faf0806a5a0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7faf0806b1a0 tx=0x7faf0001c9d0 comp rx=0 tx=0).stop 2023-10-30T14:16:28.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 -- 172.21.15.46:0/2564256605 shutdown_connections 2023-10-30T14:16:28.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 --2- 172.21.15.46:0/2564256605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0815e700 0x7faf0806a5a0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:28.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 --2- 172.21.15.46:0/2564256605 >> v2:172.21.15.154:3300/0 conn(0x7faf08070740 0x7faf080c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:28.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 -- 172.21.15.46:0/2564256605 >> 172.21.15.46:0/2564256605 conn(0x7faf080c7910 msgr2=0x7faf080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:28.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 -- 172.21.15.46:0/2564256605 shutdown_connections 2023-10-30T14:16:28.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 -- 172.21.15.46:0/2564256605 wait complete. 2023-10-30T14:16:28.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 Processor -- start 2023-10-30T14:16:28.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 -- start start 2023-10-30T14:16:28.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf08070740 0x7faf081fa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:28.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf081580e0 0x7faf0813ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:28.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf0815e700 0x7faf081455f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:28.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf080b0a80 con 0x7faf081580e0 2023-10-30T14:16:28.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf080c8d10 con 0x7faf0815e700 2023-10-30T14:16:28.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0f2c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf08145160 con 0x7faf08070740 2023-10-30T14:16:28.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0c864640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf081580e0 0x7faf0813ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:28.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0c864640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf081580e0 0x7faf0813ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57664/0 (socket says 172.21.15.46:57664) 2023-10-30T14:16:28.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0c864640 1 -- 172.21.15.46:0/4001741976 learned_addr learned my addr 172.21.15.46:0/4001741976 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:28.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0d866640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf0815e700 0x7faf081455f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0d065640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf08070740 0x7faf081fa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0c864640 1 -- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf08070740 msgr2=0x7faf081fa0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0c864640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf08070740 0x7faf081fa0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0c864640 1 -- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf0815e700 msgr2=0x7faf081455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0c864640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf0815e700 0x7faf081455f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:28.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.608+0000 7faf0c864640 1 -- 172.21.15.46:0/4001741976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf00009b30 con 0x7faf081580e0 2023-10-30T14:16:28.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.612+0000 7faf0c864640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf081580e0 0x7faf0813ec90 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7faef800e840 tx=0x7faef800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:28.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.612+0000 7faefdffb640 1 -- 172.21.15.46:0/4001741976 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef800c870 con 0x7faf081580e0 2023-10-30T14:16:28.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.612+0000 7faefdffb640 1 -- 172.21.15.46:0/4001741976 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef800c9d0 con 0x7faf081580e0 2023-10-30T14:16:28.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.612+0000 7faefdffb640 1 -- 172.21.15.46:0/4001741976 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef800cd60 con 0x7faf081580e0 2023-10-30T14:16:28.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.612+0000 7faf0f2c9640 1 -- 172.21.15.46:0/4001741976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf081fa5f0 con 0x7faf081580e0 2023-10-30T14:16:28.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.612+0000 7faf0f2c9640 1 -- 172.21.15.46:0/4001741976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf08148c40 con 0x7faf081580e0 2023-10-30T14:16:28.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.612+0000 7faf0f2c9640 1 -- 172.21.15.46:0/4001741976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0804e170 con 0x7faf081580e0 2023-10-30T14:16:28.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.612+0000 7faefdffb640 1 -- 172.21.15.46:0/4001741976 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faef8013410 con 0x7faf081580e0 2023-10-30T14:16:28.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.612+0000 7faefdffb640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faeec0664d0 0x7faeec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.612+0000 7faefdffb640 1 -- 172.21.15.46:0/4001741976 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7faef8084570 con 0x7faf081580e0 2023-10-30T14:16:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.616+0000 7faf0d065640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faeec0664d0 0x7faeec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.616+0000 7faf0d065640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faeec0664d0 0x7faeec068990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7faf04000f70 tx=0x7faf04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.616+0000 7faefdffb640 1 -- 172.21.15.46:0/4001741976 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faef8051050 con 0x7faf081580e0 2023-10-30T14:16:28.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.808+0000 7faf0f2c9640 1 -- 172.21.15.46:0/4001741976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf081452a0 con 0x7faf081580e0 2023-10-30T14:16:28.814 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:28.815 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":228,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:26.038434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:28.815 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 228 2023-10-30T14:16:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.808+0000 7faefdffb640 1 -- 172.21.15.46:0/4001741976 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+5730 (secure 0 0 0) 0x7faef8054670 con 0x7faf081580e0 2023-10-30T14:16:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.812+0000 7faf0f2c9640 1 -- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faeec0664d0 msgr2=0x7faeec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.812+0000 7faf0f2c9640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faeec0664d0 0x7faeec068990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7faf04000f70 tx=0x7faf04009040 comp rx=0 tx=0).stop 2023-10-30T14:16:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.812+0000 7faf0f2c9640 1 -- 172.21.15.46:0/4001741976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf081580e0 msgr2=0x7faf0813ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.812+0000 7faf0f2c9640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf081580e0 0x7faf0813ec90 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7faef800e840 tx=0x7faef800ed00 comp rx=0 tx=0).stop 2023-10-30T14:16:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.812+0000 7faf0f2c9640 1 -- 172.21.15.46:0/4001741976 shutdown_connections 2023-10-30T14:16:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.812+0000 7faf0f2c9640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7faeec0664d0 0x7faeec068990 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.812+0000 7faf0f2c9640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf0815e700 0x7faf081455f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.812+0000 7faf0f2c9640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf081580e0 0x7faf0813ec90 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:28.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.812+0000 7faf0f2c9640 1 --2- 172.21.15.46:0/4001741976 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf08070740 0x7faf081fa0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:28.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.812+0000 7faf0f2c9640 1 -- 172.21.15.46:0/4001741976 >> 172.21.15.46:0/4001741976 conn(0x7faf080c7910 msgr2=0x7faf080c7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:28.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.812+0000 7faf0f2c9640 1 -- 172.21.15.46:0/4001741976 shutdown_connections 2023-10-30T14:16:28.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:28.812+0000 7faf0f2c9640 1 -- 172.21.15.46:0/4001741976 wait complete. 2023-10-30T14:16:28.892 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:16:30.893 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:31.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe993172640 1 Processor -- start 2023-10-30T14:16:31.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe993172640 1 -- start start 2023-10-30T14:16:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe993172640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe98c1538a0 0x7fe98c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe993172640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe98c0cf160 0x7fe98c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe993172640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c0cb800 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe993172640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe98c15dc30 con 0x7fe98c1538a0 2023-10-30T14:16:31.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe993172640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe98c0cb330 con 0x7fe98c0cf160 2023-10-30T14:16:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe990f0e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe98c1538a0 0x7fe98c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe990f0e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe98c1538a0 0x7fe98c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57680/0 (socket says 172.21.15.46:57680) 2023-10-30T14:16:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe990f0e640 1 -- 172.21.15.46:0/2242781152 learned_addr learned my addr 172.21.15.46:0/2242781152 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:31.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe98bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe98c0ca0e0 0x7fe98c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40896/0 (socket says 172.21.15.46:40896) 2023-10-30T14:16:31.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe99170f640 1 --2- 172.21.15.46:0/2242781152 >> v2:172.21.15.154:3300/0 conn(0x7fe98c0cf160 0x7fe98c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:31.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe990f0e640 1 -- 172.21.15.46:0/2242781152 >> v2:172.21.15.154:3300/0 conn(0x7fe98c0cf160 msgr2=0x7fe98c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:31.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe990f0e640 1 --2- 172.21.15.46:0/2242781152 >> v2:172.21.15.154:3300/0 conn(0x7fe98c0cf160 0x7fe98c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:31.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe990f0e640 1 -- 172.21.15.46:0/2242781152 >> v1:172.21.15.46:6789/0 conn(0x7fe98c0ca0e0 legacy=0x7fe98c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:31.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe990f0e640 1 -- 172.21.15.46:0/2242781152 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98c0cad70 con 0x7fe98c1538a0 2023-10-30T14:16:31.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.044+0000 7fe98b7fe640 1 -- 172.21.15.46:0/2242781152 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239658208 0 0) 0x7fe98c0cb800 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.048+0000 7fe990f0e640 1 --2- 172.21.15.46:0/2242781152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98c1538a0 0x7fe98c15d6f0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0b5330 tx=0x7fe98001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ddd915b6375d85 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:31.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.048+0000 7fe98b7fe640 1 -- 172.21.15.46:0/2242781152 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9800a89f0 con 0x7fe98c1538a0 2023-10-30T14:16:31.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.048+0000 7fe98b7fe640 1 -- 172.21.15.46:0/2242781152 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9800a8b50 con 0x7fe98c1538a0 2023-10-30T14:16:31.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.048+0000 7fe98b7fe640 1 -- 172.21.15.46:0/2242781152 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9800b0440 con 0x7fe98c1538a0 2023-10-30T14:16:31.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.048+0000 7fe993172640 1 -- 172.21.15.46:0/2242781152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98c1538a0 msgr2=0x7fe98c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:31.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.048+0000 7fe993172640 1 --2- 172.21.15.46:0/2242781152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98c1538a0 0x7fe98c15d6f0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0b5330 tx=0x7fe98001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:16:31.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.048+0000 7fe993172640 1 -- 172.21.15.46:0/2242781152 shutdown_connections 2023-10-30T14:16:31.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.048+0000 7fe993172640 1 --2- 172.21.15.46:0/2242781152 >> v2:172.21.15.154:3300/0 conn(0x7fe98c0cf160 0x7fe98c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:31.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.048+0000 7fe993172640 1 --2- 172.21.15.46:0/2242781152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98c1538a0 0x7fe98c15d6f0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:31.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.048+0000 7fe993172640 1 -- 172.21.15.46:0/2242781152 >> 172.21.15.46:0/2242781152 conn(0x7fe98c06bfe0 msgr2=0x7fe98c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:31.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.048+0000 7fe993172640 1 -- 172.21.15.46:0/2242781152 shutdown_connections 2023-10-30T14:16:31.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.048+0000 7fe993172640 1 -- 172.21.15.46:0/2242781152 wait complete. 2023-10-30T14:16:31.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe993172640 1 Processor -- start 2023-10-30T14:16:31.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe993172640 1 -- start start 2023-10-30T14:16:31.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe993172640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98c0ca0e0 0x7fe98c15b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:31.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe993172640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe98c0cf160 0x7fe98c0b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:31.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe993172640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe98c1538a0 0x7fe98c1ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe993172640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c14ecd0 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe993172640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c14e750 con 0x7fe98c1538a0 2023-10-30T14:16:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe993172640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c1f3380 con 0x7fe98c0cf160 2023-10-30T14:16:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe990f0e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98c0ca0e0 0x7fe98c15b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe990f0e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98c0ca0e0 0x7fe98c15b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57688/0 (socket says 172.21.15.46:57688) 2023-10-30T14:16:31.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe990f0e640 1 -- 172.21.15.46:0/448790772 learned_addr learned my addr 172.21.15.46:0/448790772 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:31.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe98bfff640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe98c0cf160 0x7fe98c0b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:31.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe99170f640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe98c1538a0 0x7fe98c1ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:31.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe990f0e640 1 -- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe98c0cf160 msgr2=0x7fe98c0b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:31.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe990f0e640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe98c0cf160 0x7fe98c0b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:31.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe990f0e640 1 -- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe98c1538a0 msgr2=0x7fe98c1ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:31.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe990f0e640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe98c1538a0 0x7fe98c1ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:31.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe990f0e640 1 -- 172.21.15.46:0/448790772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9800094d0 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe990f0e640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98c0ca0e0 0x7fe98c15b060 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fe98001b900 tx=0x7fe98001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:31.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe9897fa640 1 -- 172.21.15.46:0/448790772 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9800ba600 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe9897fa640 1 -- 172.21.15.46:0/448790772 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9800ba760 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe9897fa640 1 -- 172.21.15.46:0/448790772 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9800bab10 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe993172640 1 -- 172.21.15.46:0/448790772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe98c14e9f0 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.052+0000 7fe993172640 1 -- 172.21.15.46:0/448790772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe98c0b1b60 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.056+0000 7fe993172640 1 -- 172.21.15.46:0/448790772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe98c0b2a50 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.056+0000 7fe9897fa640 1 -- 172.21.15.46:0/448790772 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe9800bac70 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.056+0000 7fe9897fa640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe9700664a0 0x7fe970068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.056+0000 7fe9897fa640 1 -- 172.21.15.46:0/448790772 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7fe980122ef0 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.056+0000 7fe98bfff640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe9700664a0 0x7fe970068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.056+0000 7fe98bfff640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe9700664a0 0x7fe970068960 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fe98c1597c0 tx=0x7fe978007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.060+0000 7fe9897fa640 1 -- 172.21.15.46:0/448790772 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe9800ef8c0 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.264+0000 7fe993172640 1 -- 172.21.15.46:0/448790772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe98c1ef280 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.264+0000 7fe9897fa640 1 -- 172.21.15.46:0/448790772 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fe98c1ef280 con 0x7fe98c0ca0e0 2023-10-30T14:16:31.270 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:31.270 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":228,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:26.038434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:31.270 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 228 2023-10-30T14:16:31.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.264+0000 7fe993172640 1 -- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe9700664a0 msgr2=0x7fe970068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:31.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.264+0000 7fe993172640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe9700664a0 0x7fe970068960 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fe98c1597c0 tx=0x7fe978007c10 comp rx=0 tx=0).stop 2023-10-30T14:16:31.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.264+0000 7fe993172640 1 -- 172.21.15.46:0/448790772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98c0ca0e0 msgr2=0x7fe98c15b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:31.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.264+0000 7fe993172640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98c0ca0e0 0x7fe98c15b060 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fe98001b900 tx=0x7fe98001ba90 comp rx=0 tx=0).stop 2023-10-30T14:16:31.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.268+0000 7fe993172640 1 -- 172.21.15.46:0/448790772 shutdown_connections 2023-10-30T14:16:31.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.268+0000 7fe993172640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe9700664a0 0x7fe970068960 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:31.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.268+0000 7fe993172640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe98c1538a0 0x7fe98c1ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:31.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.268+0000 7fe993172640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe98c0cf160 0x7fe98c0b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:31.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.268+0000 7fe993172640 1 --2- 172.21.15.46:0/448790772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe98c0ca0e0 0x7fe98c15b060 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:31.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.268+0000 7fe993172640 1 -- 172.21.15.46:0/448790772 >> 172.21.15.46:0/448790772 conn(0x7fe98c06bfe0 msgr2=0x7fe98c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:31.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.268+0000 7fe993172640 1 -- 172.21.15.46:0/448790772 shutdown_connections 2023-10-30T14:16:31.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:31.268+0000 7fe993172640 1 -- 172.21.15.46:0/448790772 wait complete. 2023-10-30T14:16:31.291 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 228, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 228, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9135": { "addr": "172.21.15.46:6837/681817982", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 681817982, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 681817982, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 9135, "incarnation": 218, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 42 }, "gid_9276": { "addr": "172.21.15.46:6836/2239735286", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 2239735286, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 2239735286, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9276, "incarnation": 225, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:rejoin", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:16:26.038434+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9135, "mds_1": 9276 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:16:33.293 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:33.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.444+0000 7f6462c88640 1 Processor -- start 2023-10-30T14:16:33.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.444+0000 7f6462c88640 1 -- start start 2023-10-30T14:16:33.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.444+0000 7f6462c88640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f645c0cefa0 0x7f645c0cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:33.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.444+0000 7f6462c88640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c15a6c0 con 0x7f645c15cd50 2023-10-30T14:16:33.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.444+0000 7f6462c88640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c0ca6a0 con 0x7f645c1538f0 2023-10-30T14:16:33.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.444+0000 7f6462c88640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f645c0cf8f0 con 0x7f645c0cefa0 2023-10-30T14:16:33.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.444+0000 7f6462487640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f645c0cefa0 0x7f645c0cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:33.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.444+0000 7f6462487640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f645c0cefa0 0x7f645c0cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57698/0 (socket says 172.21.15.46:57698) 2023-10-30T14:16:33.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.444+0000 7f6462487640 1 -- 172.21.15.46:0/1084577944 learned_addr learned my addr 172.21.15.46:0/1084577944 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.444+0000 7f6462487640 1 -- 172.21.15.46:0/1084577944 >> v1:172.21.15.154:6789/0 conn(0x7f645c1538f0 legacy=0x7f645c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.444+0000 7f6462487640 1 -- 172.21.15.46:0/1084577944 >> v1:172.21.15.46:6789/0 conn(0x7f645c15cd50 legacy=0x7f645c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.444+0000 7f6462487640 1 -- 172.21.15.46:0/1084577944 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645c15d300 con 0x7f645c0cefa0 2023-10-30T14:16:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.448+0000 7f6462487640 1 --2- 172.21.15.46:0/1084577944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f645c0cefa0 0x7f645c0cf3b0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f6454009e50 tx=0x7f645401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44bdd25a81846ac5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.448+0000 7f6460c84640 1 -- 172.21.15.46:0/1084577944 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64540af400 con 0x7f645c0cefa0 2023-10-30T14:16:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.448+0000 7f6460c84640 1 -- 172.21.15.46:0/1084577944 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64540ab050 con 0x7f645c0cefa0 2023-10-30T14:16:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.448+0000 7f6460c84640 1 -- 172.21.15.46:0/1084577944 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64540af780 con 0x7f645c0cefa0 2023-10-30T14:16:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.448+0000 7f6462c88640 1 -- 172.21.15.46:0/1084577944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f645c0cefa0 msgr2=0x7f645c0cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:33.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.448+0000 7f6462c88640 1 --2- 172.21.15.46:0/1084577944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f645c0cefa0 0x7f645c0cf3b0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f6454009e50 tx=0x7f645401f1f0 comp rx=0 tx=0).stop 2023-10-30T14:16:33.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.448+0000 7f6462c88640 1 -- 172.21.15.46:0/1084577944 shutdown_connections 2023-10-30T14:16:33.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.448+0000 7f6462c88640 1 --2- 172.21.15.46:0/1084577944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f645c0cefa0 0x7f645c0cf3b0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.448+0000 7f6462c88640 1 -- 172.21.15.46:0/1084577944 >> 172.21.15.46:0/1084577944 conn(0x7f645c06a640 msgr2=0x7f645c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.448+0000 7f6462c88640 1 -- 172.21.15.46:0/1084577944 shutdown_connections 2023-10-30T14:16:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.448+0000 7f6462c88640 1 -- 172.21.15.46:0/1084577944 wait complete. 2023-10-30T14:16:33.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6462c88640 1 Processor -- start 2023-10-30T14:16:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6462c88640 1 -- start start 2023-10-30T14:16:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6462c88640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f645c0cefa0 0x7f645c1f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6462c88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f645c1538f0 0x7f645c148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6462c88640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f645c15cd50 0x7f645c13c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6462c88640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c148a40 con 0x7f645c1538f0 2023-10-30T14:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6462c88640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c13f2b0 con 0x7f645c0cefa0 2023-10-30T14:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6462c88640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645c146bd0 con 0x7f645c15cd50 2023-10-30T14:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6461485640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f645c1538f0 0x7f645c148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6461485640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f645c1538f0 0x7f645c148f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57702/0 (socket says 172.21.15.46:57702) 2023-10-30T14:16:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6461485640 1 -- 172.21.15.46:0/4292556781 learned_addr learned my addr 172.21.15.46:0/4292556781 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6462487640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f645c15cd50 0x7f645c13c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6461c86640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f645c0cefa0 0x7f645c1f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6461485640 1 -- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f645c15cd50 msgr2=0x7f645c13c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6461485640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f645c15cd50 0x7f645c13c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6461485640 1 -- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f645c0cefa0 msgr2=0x7f645c1f24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6461485640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f645c0cefa0 0x7f645c1f24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6461485640 1 -- 172.21.15.46:0/4292556781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6454009b30 con 0x7f645c1538f0 2023-10-30T14:16:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6461c86640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f645c0cefa0 0x7f645c1f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6461485640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f645c1538f0 0x7f645c148f20 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f644c00c3f0 tx=0x7f644c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f64527fc640 1 -- 172.21.15.46:0/4292556781 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c0052d0 con 0x7f645c1538f0 2023-10-30T14:16:33.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f64527fc640 1 -- 172.21.15.46:0/4292556781 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644c005430 con 0x7f645c1538f0 2023-10-30T14:16:33.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f64527fc640 1 -- 172.21.15.46:0/4292556781 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c003840 con 0x7f645c1538f0 2023-10-30T14:16:33.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6462c88640 1 -- 172.21.15.46:0/4292556781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645c146eb0 con 0x7f645c1538f0 2023-10-30T14:16:33.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6462c88640 1 -- 172.21.15.46:0/4292556781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645c151540 con 0x7f645c1538f0 2023-10-30T14:16:33.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.452+0000 7f6462c88640 1 -- 172.21.15.46:0/4292556781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645c1470a0 con 0x7f645c1538f0 2023-10-30T14:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.456+0000 7f64527fc640 1 -- 172.21.15.46:0/4292556781 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f644c0039a0 con 0x7f645c1538f0 2023-10-30T14:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.456+0000 7f64527fc640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64400664d0 0x7f6440068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.456+0000 7f64527fc640 1 -- 172.21.15.46:0/4292556781 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7f644c083df0 con 0x7f645c1538f0 2023-10-30T14:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.456+0000 7f6461c86640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64400664d0 0x7f6440068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.456+0000 7f6461c86640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64400664d0 0x7f6440068990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f6458000f70 tx=0x7f6458009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.460+0000 7f64527fc640 1 -- 172.21.15.46:0/4292556781 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f644c050870 con 0x7f645c1538f0 2023-10-30T14:16:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 -- 172.21.15.46:0/4292556781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f645c04e170 con 0x7f645c1538f0 2023-10-30T14:16:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f64527fc640 1 -- 172.21.15.46:0/4292556781 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f644c017020 con 0x7f645c1538f0 2023-10-30T14:16:33.653 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:33.653 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:31.202372+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:33.654 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 229 2023-10-30T14:16:33.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 -- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64400664d0 msgr2=0x7f6440068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:33.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64400664d0 0x7f6440068990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f6458000f70 tx=0x7f6458009040 comp rx=0 tx=0).stop 2023-10-30T14:16:33.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 -- 172.21.15.46:0/4292556781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f645c1538f0 msgr2=0x7f645c148f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:33.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f645c1538f0 0x7f645c148f20 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f644c00c3f0 tx=0x7f644c00c8b0 comp rx=0 tx=0).stop 2023-10-30T14:16:33.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 -- 172.21.15.46:0/4292556781 shutdown_connections 2023-10-30T14:16:33.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64400664d0 0x7f6440068990 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:33.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f645c15cd50 0x7f645c13c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:33.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f645c1538f0 0x7f645c148f20 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:33.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 --2- 172.21.15.46:0/4292556781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f645c0cefa0 0x7f645c1f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:33.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 -- 172.21.15.46:0/4292556781 >> 172.21.15.46:0/4292556781 conn(0x7f645c06a640 msgr2=0x7f645c06bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:33.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 -- 172.21.15.46:0/4292556781 shutdown_connections 2023-10-30T14:16:33.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:33.648+0000 7f6462c88640 1 -- 172.21.15.46:0/4292556781 wait complete. 2023-10-30T14:16:33.674 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=8982 2023-10-30T14:16:33.674 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.a 2023-10-30T14:16:38.675 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T14:16:38.676 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T14:16:38.676 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T14:16:38.680 INFO:tasks.ceph.mds.a:Started 2023-10-30T14:16:38.680 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:16:38.778 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-10-30T14:16:38.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed5346b640 1 Processor -- start 2023-10-30T14:16:38.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed5346b640 1 -- start start 2023-10-30T14:16:38.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed5346b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fed4c15d6f0 0x7fed4c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:38.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed5346b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4c0cf160 0x7fed4c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:38.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed5346b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4c0cb800 con 0x7fed4c1538a0 2023-10-30T14:16:38.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed5346b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed4c0cb330 con 0x7fed4c15d6f0 2023-10-30T14:16:38.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed5346b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c0cad70 con 0x7fed4c0cf160 2023-10-30T14:16:38.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed51207640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fed4c1538a0 0x7fed4c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51926/0 (socket says 172.21.15.46:51926) 2023-10-30T14:16:38.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed51207640 1 -- 172.21.15.46:0/1799031342 learned_addr learned my addr 172.21.15.46:0/1799031342 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:16:38.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed50a06640 1 --2- 172.21.15.46:0/1799031342 >> v2:172.21.15.154:3300/0 conn(0x7fed4c15d6f0 0x7fed4c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:38.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed51a08640 1 --2- 172.21.15.46:0/1799031342 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4c0cf160 0x7fed4c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:38.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed43fff640 1 -- 172.21.15.46:0/1799031342 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3313471683 0 0) 0x7fed4c0cb800 con 0x7fed4c1538a0 2023-10-30T14:16:38.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed43fff640 1 -- 172.21.15.46:0/1799031342 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed38003530 con 0x7fed4c1538a0 2023-10-30T14:16:38.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed43fff640 1 -- 172.21.15.46:0/1799031342 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2828967709 0 0) 0x7fed38003530 con 0x7fed4c1538a0 2023-10-30T14:16:38.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed50a06640 1 -- 172.21.15.46:0/1799031342 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4c0cf160 msgr2=0x7fed4c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:16:38.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed50a06640 1 --2- 172.21.15.46:0/1799031342 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4c0cf160 0x7fed4c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:38.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed50a06640 1 -- 172.21.15.46:0/1799031342 >> v1:172.21.15.46:6789/0 conn(0x7fed4c1538a0 legacy=0x7fed4c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:38.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.832+0000 7fed50a06640 1 -- 172.21.15.46:0/1799031342 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c0cb050 con 0x7fed4c15d6f0 2023-10-30T14:16:38.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.836+0000 7fed50a06640 1 --2- 172.21.15.46:0/1799031342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4c15d6f0 0x7fed4c15a090 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fed3c0097c0 tx=0x7fed3c01eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c99f203b3580e15b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:38.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.836+0000 7fed43fff640 1 -- 172.21.15.46:0/1799031342 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48001680 con 0x7fed4c15d6f0 2023-10-30T14:16:38.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.836+0000 7fed43fff640 1 -- 172.21.15.46:0/1799031342 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c0a85d0 con 0x7fed4c15d6f0 2023-10-30T14:16:38.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.836+0000 7fed43fff640 1 -- 172.21.15.46:0/1799031342 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c0a8940 con 0x7fed4c15d6f0 2023-10-30T14:16:38.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.836+0000 7fed5346b640 1 -- 172.21.15.46:0/1799031342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4c15d6f0 msgr2=0x7fed4c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:38.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.836+0000 7fed5346b640 1 --2- 172.21.15.46:0/1799031342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4c15d6f0 0x7fed4c15a090 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fed3c0097c0 tx=0x7fed3c01eb90 comp rx=0 tx=0).stop 2023-10-30T14:16:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.836+0000 7fed5346b640 1 -- 172.21.15.46:0/1799031342 shutdown_connections 2023-10-30T14:16:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.836+0000 7fed5346b640 1 --2- 172.21.15.46:0/1799031342 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4c0cf160 0x7fed4c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.836+0000 7fed5346b640 1 --2- 172.21.15.46:0/1799031342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4c15d6f0 0x7fed4c15a090 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.836+0000 7fed5346b640 1 -- 172.21.15.46:0/1799031342 >> 172.21.15.46:0/1799031342 conn(0x7fed4c06bfe0 msgr2=0x7fed4c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:38.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.836+0000 7fed5346b640 1 -- 172.21.15.46:0/1799031342 shutdown_connections 2023-10-30T14:16:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.836+0000 7fed5346b640 1 -- 172.21.15.46:0/1799031342 wait complete. 2023-10-30T14:16:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed5346b640 1 Processor -- start 2023-10-30T14:16:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed5346b640 1 -- start start 2023-10-30T14:16:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed5346b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4c0cf160 0x7fed4c1440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:38.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed5346b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4c1538a0 0x7fed4c1ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed5346b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4c15d6f0 0x7fed4c1efb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed5346b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c1f34f0 con 0x7fed4c1538a0 2023-10-30T14:16:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed5346b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c158e80 con 0x7fed4c15d6f0 2023-10-30T14:16:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed5346b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c144610 con 0x7fed4c0cf160 2023-10-30T14:16:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed50a06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4c1538a0 0x7fed4c1ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed50a06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4c1538a0 0x7fed4c1ef600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33148/0 (socket says 172.21.15.46:33148) 2023-10-30T14:16:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed50a06640 1 -- 172.21.15.46:0/2266170123 learned_addr learned my addr 172.21.15.46:0/2266170123 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:16:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed51207640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4c0cf160 0x7fed4c1440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed51a08640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4c15d6f0 0x7fed4c1efb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:38.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed50a06640 1 -- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4c0cf160 msgr2=0x7fed4c1440d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed50a06640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4c0cf160 0x7fed4c1440d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed50a06640 1 -- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4c15d6f0 msgr2=0x7fed4c1efb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed50a06640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4c15d6f0 0x7fed4c1efb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed50a06640 1 -- 172.21.15.46:0/2266170123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c009420 con 0x7fed4c1538a0 2023-10-30T14:16:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed51a08640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4c15d6f0 0x7fed4c1efb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:16:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed50a06640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4c1538a0 0x7fed4c1ef600 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fed3c01d240 tx=0x7fed3c01b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed41ffb640 1 -- 172.21.15.46:0/2266170123 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c0a9c60 con 0x7fed4c1538a0 2023-10-30T14:16:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed41ffb640 1 -- 172.21.15.46:0/2266170123 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c0a9dc0 con 0x7fed4c1538a0 2023-10-30T14:16:38.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed41ffb640 1 -- 172.21.15.46:0/2266170123 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c003820 con 0x7fed4c1538a0 2023-10-30T14:16:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed5346b640 1 -- 172.21.15.46:0/2266170123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4c1f3250 con 0x7fed4c1538a0 2023-10-30T14:16:38.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.840+0000 7fed5346b640 1 -- 172.21.15.46:0/2266170123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4c152770 con 0x7fed4c1538a0 2023-10-30T14:16:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.844+0000 7fed5346b640 1 -- 172.21.15.46:0/2266170123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4c0b2a50 con 0x7fed4c1538a0 2023-10-30T14:16:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.844+0000 7fed41ffb640 1 -- 172.21.15.46:0/2266170123 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed3c0b7040 con 0x7fed4c1538a0 2023-10-30T14:16:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.844+0000 7fed41ffb640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed300664a0 0x7fed30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:16:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.844+0000 7fed41ffb640 1 -- 172.21.15.46:0/2266170123 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7fed3c121f00 con 0x7fed4c1538a0 2023-10-30T14:16:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.844+0000 7fed51207640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed300664a0 0x7fed30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:16:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.848+0000 7fed51207640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed300664a0 0x7fed30068960 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fed48007a10 tx=0x7fed48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:16:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:38.848+0000 7fed41ffb640 1 -- 172.21.15.46:0/2266170123 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fed3c0ee960 con 0x7fed4c1538a0 2023-10-30T14:16:39.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.040+0000 7fed5346b640 1 -- 172.21.15.46:0/2266170123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed4c067a30 con 0x7fed4c1538a0 2023-10-30T14:16:39.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed41ffb640 1 -- 172.21.15.46:0/2266170123 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+6522 (secure 0 0 0) 0x7fed3c0b5070 con 0x7fed4c1538a0 2023-10-30T14:16:39.048 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:16:39.048 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9426,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:31.202372+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:16:39.048 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 230 2023-10-30T14:16:39.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed5346b640 1 -- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed300664a0 msgr2=0x7fed30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed5346b640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed300664a0 0x7fed30068960 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fed48007a10 tx=0x7fed48009040 comp rx=0 tx=0).stop 2023-10-30T14:16:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed5346b640 1 -- 172.21.15.46:0/2266170123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4c1538a0 msgr2=0x7fed4c1ef600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:16:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed5346b640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4c1538a0 0x7fed4c1ef600 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fed3c01d240 tx=0x7fed3c01b930 comp rx=0 tx=0).stop 2023-10-30T14:16:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed5346b640 1 -- 172.21.15.46:0/2266170123 shutdown_connections 2023-10-30T14:16:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed5346b640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed300664a0 0x7fed30068960 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed5346b640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4c15d6f0 0x7fed4c1efb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed5346b640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4c1538a0 0x7fed4c1ef600 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed5346b640 1 --2- 172.21.15.46:0/2266170123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4c0cf160 0x7fed4c1440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:16:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed5346b640 1 -- 172.21.15.46:0/2266170123 >> 172.21.15.46:0/2266170123 conn(0x7fed4c06bfe0 msgr2=0x7fed4c0cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:16:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed5346b640 1 -- 172.21.15.46:0/2266170123 shutdown_connections 2023-10-30T14:16:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:16:39.044+0000 7fed5346b640 1 -- 172.21.15.46:0/2266170123 wait complete. 2023-10-30T14:16:39.069 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T14:16:39.069 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 109 secs before thrashing 2023-10-30T14:16:39.842 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:16:39.842 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 48 -eq 48 ] 2023-10-30T14:16:39.845 INFO:tasks.workunit.client.0.smithi046.stderr:+ continue 2023-10-30T14:16:39.845 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 49 -eq 48 ] 2023-10-30T14:16:39.846 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.49 2023-10-30T14:16:39.847 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.49 2023-10-30T14:16:39.847 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.49 2023-10-30T14:16:39.847 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.49 2023-10-30T14:16:39.847 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:16:45.549 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:16:45.549 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 50 -eq 48 ] 2023-10-30T14:16:45.550 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.50 2023-10-30T14:16:45.550 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.50 2023-10-30T14:16:45.551 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.50 2023-10-30T14:16:45.551 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.50 2023-10-30T14:16:45.551 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:16:49.975 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:16:49.975 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 51 -eq 48 ] 2023-10-30T14:16:49.976 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.51 2023-10-30T14:16:49.976 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.51 2023-10-30T14:16:49.976 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.51 2023-10-30T14:16:49.977 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.51 2023-10-30T14:16:49.977 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:16:53.480 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:16:53.481 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.52 2023-10-30T14:16:53.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 52 -eq 48 ] 2023-10-30T14:16:53.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.52 2023-10-30T14:16:53.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.52 2023-10-30T14:16:53.482 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.52 2023-10-30T14:16:53.483 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:16:58.416 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:16:58.417 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.53 2023-10-30T14:16:58.417 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 53 -eq 48 ] 2023-10-30T14:16:58.418 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.53 2023-10-30T14:16:58.418 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.53 2023-10-30T14:16:58.418 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.53 2023-10-30T14:16:58.418 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:16:59.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:16:59.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:17:04.301 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:17:04.301 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 54 -eq 48 ] 2023-10-30T14:17:04.302 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.54 2023-10-30T14:17:04.302 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.54 2023-10-30T14:17:04.303 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.54 2023-10-30T14:17:04.303 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.54 2023-10-30T14:17:04.303 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:17:07.923 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:17:07.924 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 55 -eq 48 ] 2023-10-30T14:17:07.924 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.55 2023-10-30T14:17:07.924 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.55 2023-10-30T14:17:07.925 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.55 2023-10-30T14:17:07.925 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.55 2023-10-30T14:17:07.925 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:17:14.941 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:17:14.941 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 56 -eq 48 ] 2023-10-30T14:17:14.942 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.56 2023-10-30T14:17:14.942 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.56 2023-10-30T14:17:14.942 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.56 2023-10-30T14:17:14.943 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.56 2023-10-30T14:17:14.943 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:17:20.698 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:17:20.699 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 57 -eq 48 ] 2023-10-30T14:17:20.699 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.57 2023-10-30T14:17:20.699 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.57 2023-10-30T14:17:20.700 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.57 2023-10-30T14:17:20.700 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.57 2023-10-30T14:17:20.700 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:17:26.342 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:17:26.343 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 58 -eq 48 ] 2023-10-30T14:17:26.343 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.58 2023-10-30T14:17:26.343 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.58 2023-10-30T14:17:26.344 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.58 2023-10-30T14:17:26.344 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.58 2023-10-30T14:17:26.344 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:17:29.732 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:17:29.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:17:31.005 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:17:31.006 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 59 -eq 48 ] 2023-10-30T14:17:31.006 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.59 2023-10-30T14:17:31.006 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.59 2023-10-30T14:17:31.007 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.59 2023-10-30T14:17:31.007 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.59 2023-10-30T14:17:31.007 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:17:37.449 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:17:37.449 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.60 2023-10-30T14:17:37.450 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 60 -eq 48 ] 2023-10-30T14:17:37.450 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.60 2023-10-30T14:17:37.451 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.60 2023-10-30T14:17:37.451 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.60 2023-10-30T14:17:37.451 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:17:43.484 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:17:43.484 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 61 -eq 48 ] 2023-10-30T14:17:43.485 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.61 2023-10-30T14:17:43.485 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.61 2023-10-30T14:17:43.486 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.61 2023-10-30T14:17:43.486 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.61 2023-10-30T14:17:43.486 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:17:52.346 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:17:52.347 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.62 2023-10-30T14:17:52.347 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 62 -eq 48 ] 2023-10-30T14:17:52.347 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.62 2023-10-30T14:17:52.348 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.62 2023-10-30T14:17:52.348 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.62 2023-10-30T14:17:52.348 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:17:57.722 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:17:57.722 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 63 -eq 48 ] 2023-10-30T14:17:57.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.63 2023-10-30T14:17:57.723 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.63 2023-10-30T14:17:57.723 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.63 2023-10-30T14:17:57.724 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.63 2023-10-30T14:17:57.724 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:17:59.813 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:17:59.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:18:03.189 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:18:03.189 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 64 -eq 48 ] 2023-10-30T14:18:03.189 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.64 2023-10-30T14:18:03.189 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.64 2023-10-30T14:18:03.190 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.64 2023-10-30T14:18:03.190 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.64 2023-10-30T14:18:03.191 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:18:08.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:18:08.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 65 -eq 48 ] 2023-10-30T14:18:08.117 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.65 2023-10-30T14:18:08.118 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.65 2023-10-30T14:18:08.118 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.65 2023-10-30T14:18:08.118 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.65 2023-10-30T14:18:08.119 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:18:15.127 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:18:15.128 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.66 2023-10-30T14:18:15.129 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 66 -eq 48 ] 2023-10-30T14:18:15.129 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.66 2023-10-30T14:18:15.129 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.66 2023-10-30T14:18:15.129 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.66 2023-10-30T14:18:15.129 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:18:22.138 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:18:22.138 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 67 -eq 48 ] 2023-10-30T14:18:22.138 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.67 2023-10-30T14:18:22.139 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.67 2023-10-30T14:18:22.139 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.67 2023-10-30T14:18:22.139 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.67 2023-10-30T14:18:22.140 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:18:28.070 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:28.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc150e17640 1 Processor -- start 2023-10-30T14:18:28.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc150e17640 1 -- start start 2023-10-30T14:18:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc150e17640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc14c1538a0 0x7fc14c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc150e17640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc14c0ca0e0 0x7fc14c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc150e17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc14c0cf160 0x7fc14c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc150e17640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc14c0cb800 con 0x7fc14c1538a0 2023-10-30T14:18:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc150e17640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc14c15dc30 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc150e17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c0cb330 con 0x7fc14c0cf160 2023-10-30T14:18:28.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc14a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc14c1538a0 0x7fc14c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:28.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc14a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc14c1538a0 0x7fc14c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35220/0 (socket says 172.21.15.46:35220) 2023-10-30T14:18:28.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc14a59c640 1 -- 172.21.15.46:0/3825043583 learned_addr learned my addr 172.21.15.46:0/3825043583 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:28.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc14ad9d640 1 --2- 172.21.15.46:0/3825043583 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc14c0cf160 0x7fc14c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:28.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc149d9b640 1 --2- 172.21.15.46:0/3825043583 >> v2:172.21.15.154:3300/0 conn(0x7fc14c0ca0e0 0x7fc14c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:28.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc14a59c640 1 -- 172.21.15.46:0/3825043583 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc14c0cf160 msgr2=0x7fc14c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:28.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc14a59c640 1 --2- 172.21.15.46:0/3825043583 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc14c0cf160 0x7fc14c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc14a59c640 1 -- 172.21.15.46:0/3825043583 >> v2:172.21.15.154:3300/0 conn(0x7fc14c0ca0e0 msgr2=0x7fc14c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:28.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc14a59c640 1 --2- 172.21.15.46:0/3825043583 >> v2:172.21.15.154:3300/0 conn(0x7fc14c0ca0e0 0x7fc14c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.219+0000 7fc14a59c640 1 -- 172.21.15.46:0/3825043583 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c0cad70 con 0x7fc14c1538a0 2023-10-30T14:18:28.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc14a59c640 1 --2- 172.21.15.46:0/3825043583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc14c1538a0 0x7fc14c15d6f0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0b30a0 tx=0x7fc13c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ba420975a24e43 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:28.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc14959a640 1 -- 172.21.15.46:0/3825043583 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0a89b0 con 0x7fc14c1538a0 2023-10-30T14:18:28.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc14959a640 1 -- 172.21.15.46:0/3825043583 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c0a8b10 con 0x7fc14c1538a0 2023-10-30T14:18:28.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc14959a640 1 -- 172.21.15.46:0/3825043583 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0a8ea0 con 0x7fc14c1538a0 2023-10-30T14:18:28.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc150e17640 1 -- 172.21.15.46:0/3825043583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc14c1538a0 msgr2=0x7fc14c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:28.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc150e17640 1 --2- 172.21.15.46:0/3825043583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc14c1538a0 0x7fc14c15d6f0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0b30a0 tx=0x7fc13c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc150e17640 1 -- 172.21.15.46:0/3825043583 shutdown_connections 2023-10-30T14:18:28.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc150e17640 1 --2- 172.21.15.46:0/3825043583 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc14c0cf160 0x7fc14c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc150e17640 1 --2- 172.21.15.46:0/3825043583 >> v2:172.21.15.154:3300/0 conn(0x7fc14c0ca0e0 0x7fc14c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc150e17640 1 --2- 172.21.15.46:0/3825043583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc14c1538a0 0x7fc14c15d6f0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc150e17640 1 -- 172.21.15.46:0/3825043583 >> 172.21.15.46:0/3825043583 conn(0x7fc14c06bfe0 msgr2=0x7fc14c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:28.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc150e17640 1 -- 172.21.15.46:0/3825043583 shutdown_connections 2023-10-30T14:18:28.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.223+0000 7fc150e17640 1 -- 172.21.15.46:0/3825043583 wait complete. 2023-10-30T14:18:28.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc150e17640 1 Processor -- start 2023-10-30T14:18:28.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc150e17640 1 -- start start 2023-10-30T14:18:28.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc150e17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc14c0ca0e0 0x7fc14c15b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:28.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc150e17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc14c0cf160 0x7fc14c1f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:28.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc150e17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc14c1538a0 0x7fc14c0ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:28.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc150e17640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c15aec0 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc150e17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c150dc0 con 0x7fc14c1538a0 2023-10-30T14:18:28.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc150e17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c0b46b0 con 0x7fc14c0cf160 2023-10-30T14:18:28.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc14a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc14c0ca0e0 0x7fc14c15b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:28.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc14a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc14c0ca0e0 0x7fc14c15b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35226/0 (socket says 172.21.15.46:35226) 2023-10-30T14:18:28.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc14a59c640 1 -- 172.21.15.46:0/3979446482 learned_addr learned my addr 172.21.15.46:0/3979446482 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:28.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc149d9b640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc14c0cf160 0x7fc14c1f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:28.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc14ad9d640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc14c1538a0 0x7fc14c0ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:28.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc14a59c640 1 -- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc14c0cf160 msgr2=0x7fc14c1f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:28.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc14a59c640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc14c0cf160 0x7fc14c1f2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc14a59c640 1 -- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc14c1538a0 msgr2=0x7fc14c0ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:28.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc14a59c640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc14c1538a0 0x7fc14c0ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc14a59c640 1 -- 172.21.15.46:0/3979446482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c0094d0 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc14ad9d640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc14c1538a0 0x7fc14c0ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:28.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc14a59c640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc14c0ca0e0 0x7fc14c15b350 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0b3040 tx=0x7fc13c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:28.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc13affd640 1 -- 172.21.15.46:0/3979446482 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0ba600 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc13affd640 1 -- 172.21.15.46:0/3979446482 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c0ba760 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc150e17640 1 -- 172.21.15.46:0/3979446482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14c15a8b0 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc13affd640 1 -- 172.21.15.46:0/3979446482 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0bab10 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.227+0000 7fc150e17640 1 -- 172.21.15.46:0/3979446482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14c0b4c30 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.231+0000 7fc150e17640 1 -- 172.21.15.46:0/3979446482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc14c0b2a50 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.231+0000 7fc13affd640 1 -- 172.21.15.46:0/3979446482 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc13c0bac70 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.231+0000 7fc13affd640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc12c0664a0 0x7fc12c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:28.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.231+0000 7fc13affd640 1 -- 172.21.15.46:0/3979446482 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7fc13c122f20 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.231+0000 7fc149d9b640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc12c0664a0 0x7fc12c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:28.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.231+0000 7fc149d9b640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc12c0664a0 0x7fc12c068960 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc14c146180 tx=0x7fc140008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.235+0000 7fc13affd640 1 -- 172.21.15.46:0/3979446482 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc13c0ef860 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.439+0000 7fc150e17640 1 -- 172.21.15.46:0/3979446482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc14c04e170 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.439+0000 7fc13affd640 1 -- 172.21.15.46:0/3979446482 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fc13c0b5090 con 0x7fc14c0ca0e0 2023-10-30T14:18:28.443 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:28.443 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9426,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:41.742417+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:28.443 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 231 2023-10-30T14:18:28.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.443+0000 7fc150e17640 1 -- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc12c0664a0 msgr2=0x7fc12c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:28.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.443+0000 7fc150e17640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc12c0664a0 0x7fc12c068960 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc14c146180 tx=0x7fc140008040 comp rx=0 tx=0).stop 2023-10-30T14:18:28.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.443+0000 7fc150e17640 1 -- 172.21.15.46:0/3979446482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc14c0ca0e0 msgr2=0x7fc14c15b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:28.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.443+0000 7fc150e17640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc14c0ca0e0 0x7fc14c15b350 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0b3040 tx=0x7fc13c01ba90 comp rx=0 tx=0).stop 2023-10-30T14:18:28.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.443+0000 7fc150e17640 1 -- 172.21.15.46:0/3979446482 shutdown_connections 2023-10-30T14:18:28.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.443+0000 7fc150e17640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc12c0664a0 0x7fc12c068960 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.443+0000 7fc150e17640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc14c1538a0 0x7fc14c0ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.443+0000 7fc150e17640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc14c0cf160 0x7fc14c1f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.443+0000 7fc150e17640 1 --2- 172.21.15.46:0/3979446482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc14c0ca0e0 0x7fc14c15b350 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.443+0000 7fc150e17640 1 -- 172.21.15.46:0/3979446482 >> 172.21.15.46:0/3979446482 conn(0x7fc14c06bfe0 msgr2=0x7fc14c1524c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:28.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.443+0000 7fc150e17640 1 -- 172.21.15.46:0/3979446482 shutdown_connections 2023-10-30T14:18:28.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.443+0000 7fc150e17640 1 -- 172.21.15.46:0/3979446482 wait complete. 2023-10-30T14:18:28.463 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T14:18:28.464 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:18:28.464 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:18:28.485 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:18:28.486 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:28.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.667+0000 7ff5b7d2a640 1 Processor -- start 2023-10-30T14:18:28.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.667+0000 7ff5b7d2a640 1 -- start start 2023-10-30T14:18:28.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.671+0000 7ff5b7d2a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5b00c6bd0 0x7ff5b006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:28.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.671+0000 7ff5b7d2a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b0066cf0 con 0x7ff5b0071110 2023-10-30T14:18:28.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.671+0000 7ff5b7d2a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b00c70d0 con 0x7ff5b006f7a0 2023-10-30T14:18:28.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.671+0000 7ff5b7d2a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b00700c0 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.671+0000 7ff5b52c5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5b00c6bd0 0x7ff5b006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:28.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.671+0000 7ff5b62c7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff5b0071110 0x7ff5b0158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44974/0 (socket says 172.21.15.46:44974) 2023-10-30T14:18:28.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.671+0000 7ff5b62c7640 1 -- 172.21.15.46:0/1483078122 learned_addr learned my addr 172.21.15.46:0/1483078122 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:18:28.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.671+0000 7ff5b52c5640 1 -- 172.21.15.46:0/1483078122 >> v1:172.21.15.154:6789/0 conn(0x7ff5b006f7a0 legacy=0x7ff5b0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:28.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.671+0000 7ff5b52c5640 1 -- 172.21.15.46:0/1483078122 >> v1:172.21.15.46:6789/0 conn(0x7ff5b0071110 legacy=0x7ff5b0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:28.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.671+0000 7ff5b52c5640 1 -- 172.21.15.46:0/1483078122 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b00c7390 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.671+0000 7ff5b52c5640 1 --2- 172.21.15.46:0/1483078122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b00c6bd0 0x7ff5b006fb80 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00097c0 tx=0x7ff5a001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c53fd758b07853 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.671+0000 7ff5b4ac4640 1 -- 172.21.15.46:0/1483078122 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a00a8970 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b4ac4640 1 -- 172.21.15.46:0/1483078122 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a00a8ad0 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b4ac4640 1 -- 172.21.15.46:0/1483078122 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a00a8e60 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/1483078122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b00c6bd0 msgr2=0x7ff5b006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:28.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 --2- 172.21.15.46:0/1483078122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b00c6bd0 0x7ff5b006fb80 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00097c0 tx=0x7ff5a001ec70 comp rx=0 tx=0).stop 2023-10-30T14:18:28.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/1483078122 shutdown_connections 2023-10-30T14:18:28.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 --2- 172.21.15.46:0/1483078122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b00c6bd0 0x7ff5b006fb80 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/1483078122 >> 172.21.15.46:0/1483078122 conn(0x7ff5b00c7910 msgr2=0x7ff5b00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:28.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/1483078122 shutdown_connections 2023-10-30T14:18:28.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/1483078122 wait complete. 2023-10-30T14:18:28.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 Processor -- start 2023-10-30T14:18:28.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 -- start start 2023-10-30T14:18:28.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5b006f7a0 0x7ff5b01e2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:28.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5b0071110 0x7ff5b00c0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:28.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b00c6bd0 0x7ff5b00c0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:28.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b0153bf0 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b00c00f0 con 0x7ff5b006f7a0 2023-10-30T14:18:28.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b7d2a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b01e3330 con 0x7ff5b0071110 2023-10-30T14:18:28.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b62c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b00c6bd0 0x7ff5b00c0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:28.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b62c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b00c6bd0 0x7ff5b00c0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35234/0 (socket says 172.21.15.46:35234) 2023-10-30T14:18:28.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b62c7640 1 -- 172.21.15.46:0/3084190657 learned_addr learned my addr 172.21.15.46:0/3084190657 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:28.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b52c5640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5b0071110 0x7ff5b00c0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:28.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.675+0000 7ff5b5ac6640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5b006f7a0 0x7ff5b01e2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:28.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5b62c7640 1 -- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5b0071110 msgr2=0x7ff5b00c0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:28.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5b62c7640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5b0071110 0x7ff5b00c0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5b62c7640 1 -- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5b006f7a0 msgr2=0x7ff5b01e2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5b62c7640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5b006f7a0 0x7ff5b01e2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5b62c7640 1 -- 172.21.15.46:0/3084190657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a0009420 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5b5ac6640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5b006f7a0 0x7ff5b01e2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5b62c7640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b00c6bd0 0x7ff5b00c0ac0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7ff5a800cab0 tx=0x7ff5a800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:28.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5a67fc640 1 -- 172.21.15.46:0/3084190657 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a800ac30 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5a67fc640 1 -- 172.21.15.46:0/3084190657 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a800ad90 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5a67fc640 1 -- 172.21.15.46:0/3084190657 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8012600 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/3084190657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b01e4550 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/3084190657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b0153630 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/3084190657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b00bfec0 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5a67fc640 1 -- 172.21.15.46:0/3084190657 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5a8012760 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5a67fc640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5940664a0 0x7ff594068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:28.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.679+0000 7ff5a67fc640 1 -- 172.21.15.46:0/3084190657 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7ff5a80846c0 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.683+0000 7ff5b5ac6640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5940664a0 0x7ff594068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:28.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.683+0000 7ff5b5ac6640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5940664a0 0x7ff594068960 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac000f70 tx=0x7ff5ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:28.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.683+0000 7ff5a67fc640 1 -- 172.21.15.46:0/3084190657 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5b00bfec0 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/3084190657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5b00bfec0 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5a67fc640 1 -- 172.21.15.46:0/3084190657 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+6523 (secure 0 0 0) 0x7ff5a8051140 con 0x7ff5b00c6bd0 2023-10-30T14:18:28.881 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:28.881 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9426,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:41.742417+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 231 2023-10-30T14:18:28.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5940664a0 msgr2=0x7ff594068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:28.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5940664a0 0x7ff594068960 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac000f70 tx=0x7ff5ac009040 comp rx=0 tx=0).stop 2023-10-30T14:18:28.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/3084190657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b00c6bd0 msgr2=0x7ff5b00c0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:28.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b00c6bd0 0x7ff5b00c0ac0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7ff5a800cab0 tx=0x7ff5a800cf70 comp rx=0 tx=0).stop 2023-10-30T14:18:28.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/3084190657 shutdown_connections 2023-10-30T14:18:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff5940664a0 0x7ff594068960 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b00c6bd0 0x7ff5b00c0ac0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5b0071110 0x7ff5b00c0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 --2- 172.21.15.46:0/3084190657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5b006f7a0 0x7ff5b01e2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/3084190657 >> 172.21.15.46:0/3084190657 conn(0x7ff5b00c7910 msgr2=0x7ff5b00c3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/3084190657 shutdown_connections 2023-10-30T14:18:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:28.879+0000 7ff5b7d2a640 1 -- 172.21.15.46:0/3084190657 wait complete. 2023-10-30T14:18:28.902 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:18:29.870 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:18:29.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:18:30.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:31.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.055+0000 7f9d00780640 1 Processor -- start 2023-10-30T14:18:31.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.055+0000 7f9d00780640 1 -- start start 2023-10-30T14:18:31.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.055+0000 7f9d00780640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9cf806f7a0 0x7f9cf806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.055+0000 7f9d00780640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9cf8071110 0x7f9cf8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.055+0000 7f9d00780640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf8066cf0 con 0x7f9cf80c6bd0 2023-10-30T14:18:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.055+0000 7f9d00780640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf80700c0 con 0x7f9cf8071110 2023-10-30T14:18:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.055+0000 7f9d00780640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf80c70d0 con 0x7f9cf806f7a0 2023-10-30T14:18:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfdd1b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9cf80c6bd0 0x7f9cf8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44986/0 (socket says 172.21.15.46:44986) 2023-10-30T14:18:31.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfdd1b640 1 -- 172.21.15.46:0/2185924640 learned_addr learned my addr 172.21.15.46:0/2185924640 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:18:31.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfed1d640 1 --2- 172.21.15.46:0/2185924640 >> v2:172.21.15.46:3300/0 conn(0x7f9cf8071110 0x7f9cf8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:31.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfe51c640 1 --2- 172.21.15.46:0/2185924640 >> v2:172.21.15.154:3300/0 conn(0x7f9cf806f7a0 0x7f9cf806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:31.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfed1d640 1 -- 172.21.15.46:0/2185924640 >> v2:172.21.15.154:3300/0 conn(0x7f9cf806f7a0 msgr2=0x7f9cf806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:31.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfed1d640 1 --2- 172.21.15.46:0/2185924640 >> v2:172.21.15.154:3300/0 conn(0x7f9cf806f7a0 0x7f9cf806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:31.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfed1d640 1 -- 172.21.15.46:0/2185924640 >> v1:172.21.15.46:6789/0 conn(0x7f9cf80c6bd0 legacy=0x7f9cf8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:31.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfed1d640 1 -- 172.21.15.46:0/2185924640 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf80c7390 con 0x7f9cf8071110 2023-10-30T14:18:31.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfd51a640 1 -- 172.21.15.46:0/2185924640 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021290820 0 0) 0x7f9cf8066cf0 con 0x7f9cf80c6bd0 2023-10-30T14:18:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfed1d640 1 --2- 172.21.15.46:0/2185924640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8071110 0x7f9cf8158110 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f9cf806c830 tx=0x7f9cf001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3f9631743f8d18 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:31.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfd51a640 1 -- 172.21.15.46:0/2185924640 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf00b0440 con 0x7f9cf8071110 2023-10-30T14:18:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfd51a640 1 -- 172.21.15.46:0/2185924640 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf00b05a0 con 0x7f9cf8071110 2023-10-30T14:18:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.059+0000 7f9cfd51a640 1 -- 172.21.15.46:0/2185924640 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf00b0930 con 0x7f9cf8071110 2023-10-30T14:18:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 -- 172.21.15.46:0/2185924640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8071110 msgr2=0x7f9cf8158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:31.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 --2- 172.21.15.46:0/2185924640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8071110 0x7f9cf8158110 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f9cf806c830 tx=0x7f9cf001f0f0 comp rx=0 tx=0).stop 2023-10-30T14:18:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 -- 172.21.15.46:0/2185924640 shutdown_connections 2023-10-30T14:18:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 --2- 172.21.15.46:0/2185924640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf8071110 0x7f9cf8158110 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 --2- 172.21.15.46:0/2185924640 >> v2:172.21.15.154:3300/0 conn(0x7f9cf806f7a0 0x7f9cf806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 -- 172.21.15.46:0/2185924640 >> 172.21.15.46:0/2185924640 conn(0x7f9cf80c7910 msgr2=0x7f9cf80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 -- 172.21.15.46:0/2185924640 shutdown_connections 2023-10-30T14:18:31.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 -- 172.21.15.46:0/2185924640 wait complete. 2023-10-30T14:18:31.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 Processor -- start 2023-10-30T14:18:31.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 -- start start 2023-10-30T14:18:31.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf806f7a0 0x7f9cf81f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:31.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cf8071110 0x7f9cf8152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:31.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9cf80c6bd0 0x7f9cf8154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:31.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf81f2e70 con 0x7f9cf806f7a0 2023-10-30T14:18:31.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf81e63a0 con 0x7f9cf8071110 2023-10-30T14:18:31.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9d00780640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf81f2bf0 con 0x7f9cf80c6bd0 2023-10-30T14:18:31.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9cfe51c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf806f7a0 0x7f9cf81f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9cfe51c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf806f7a0 0x7f9cf81f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35256/0 (socket says 172.21.15.46:35256) 2023-10-30T14:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9cfe51c640 1 -- 172.21.15.46:0/837628947 learned_addr learned my addr 172.21.15.46:0/837628947 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9cfed1d640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9cf80c6bd0 0x7f9cf8154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.063+0000 7f9cfdd1b640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cf8071110 0x7f9cf8152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:31.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9cfe51c640 1 -- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9cf80c6bd0 msgr2=0x7f9cf8154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:31.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9cfe51c640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9cf80c6bd0 0x7f9cf8154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:31.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9cfe51c640 1 -- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cf8071110 msgr2=0x7f9cf8152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:31.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9cfe51c640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cf8071110 0x7f9cf8152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:31.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9cfe51c640 1 -- 172.21.15.46:0/837628947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf0009a70 con 0x7f9cf806f7a0 2023-10-30T14:18:31.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9cfe51c640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf806f7a0 0x7f9cf81f22b0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400e8a0 tx=0x7f9cf400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:31.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9ceeffd640 1 -- 172.21.15.46:0/837628947 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4009510 con 0x7f9cf806f7a0 2023-10-30T14:18:31.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9ceeffd640 1 -- 172.21.15.46:0/837628947 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf400c940 con 0x7f9cf806f7a0 2023-10-30T14:18:31.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9d00780640 1 -- 172.21.15.46:0/837628947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf8154db0 con 0x7f9cf806f7a0 2023-10-30T14:18:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9ceeffd640 1 -- 172.21.15.46:0/837628947 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf400ccb0 con 0x7f9cf806f7a0 2023-10-30T14:18:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9d00780640 1 -- 172.21.15.46:0/837628947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf80c3ff0 con 0x7f9cf806f7a0 2023-10-30T14:18:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9d00780640 1 -- 172.21.15.46:0/837628947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf8158610 con 0x7f9cf806f7a0 2023-10-30T14:18:31.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9ceeffd640 1 -- 172.21.15.46:0/837628947 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9cf4013470 con 0x7f9cf806f7a0 2023-10-30T14:18:31.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.067+0000 7f9ceeffd640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9cdc0664a0 0x7f9cdc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:31.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.071+0000 7f9ceeffd640 1 -- 172.21.15.46:0/837628947 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7f9cf4084d10 con 0x7f9cf806f7a0 2023-10-30T14:18:31.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.071+0000 7f9cfdd1b640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9cdc0664a0 0x7f9cdc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:31.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.071+0000 7f9cfdd1b640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9cdc0664a0 0x7f9cdc068960 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f9cf80c7e60 tx=0x7f9ce8007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:31.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.075+0000 7f9ceeffd640 1 -- 172.21.15.46:0/837628947 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9cf4051740 con 0x7f9cf806f7a0 2023-10-30T14:18:31.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.263+0000 7f9d00780640 1 -- 172.21.15.46:0/837628947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cf804e170 con 0x7f9cf806f7a0 2023-10-30T14:18:31.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.263+0000 7f9ceeffd640 1 -- 172.21.15.46:0/837628947 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f9cf4021090 con 0x7f9cf806f7a0 2023-10-30T14:18:31.268 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:31.268 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9426,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:41.742417+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:31.268 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 231 2023-10-30T14:18:31.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.267+0000 7f9d00780640 1 -- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9cdc0664a0 msgr2=0x7f9cdc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:31.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.267+0000 7f9d00780640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9cdc0664a0 0x7f9cdc068960 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f9cf80c7e60 tx=0x7f9ce8007b80 comp rx=0 tx=0).stop 2023-10-30T14:18:31.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.267+0000 7f9d00780640 1 -- 172.21.15.46:0/837628947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf806f7a0 msgr2=0x7f9cf81f22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:31.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.267+0000 7f9d00780640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf806f7a0 0x7f9cf81f22b0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400e8a0 tx=0x7f9cf400eca0 comp rx=0 tx=0).stop 2023-10-30T14:18:31.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.267+0000 7f9d00780640 1 -- 172.21.15.46:0/837628947 shutdown_connections 2023-10-30T14:18:31.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.267+0000 7f9d00780640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9cdc0664a0 0x7f9cdc068960 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:31.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.267+0000 7f9d00780640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9cf80c6bd0 0x7f9cf8154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:31.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.267+0000 7f9d00780640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cf8071110 0x7f9cf8152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:31.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.267+0000 7f9d00780640 1 --2- 172.21.15.46:0/837628947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf806f7a0 0x7f9cf81f22b0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:31.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.267+0000 7f9d00780640 1 -- 172.21.15.46:0/837628947 >> 172.21.15.46:0/837628947 conn(0x7f9cf80c7910 msgr2=0x7f9cf80c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:31.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.267+0000 7f9d00780640 1 -- 172.21.15.46:0/837628947 shutdown_connections 2023-10-30T14:18:31.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:31.267+0000 7f9d00780640 1 -- 172.21.15.46:0/837628947 wait complete. 2023-10-30T14:18:31.289 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:18:33.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:33.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f6892809640 1 Processor -- start 2023-10-30T14:18:33.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f6892809640 1 -- start start 2023-10-30T14:18:33.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f6892809640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f688c06f7a0 0x7f688c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:33.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f6892809640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f688c0c6bd0 0x7f688c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:33.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f6892809640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f688c071110 0x7f688c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:33.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f6892809640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688c0700c0 con 0x7f688c071110 2023-10-30T14:18:33.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f6892809640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688c06a3d0 con 0x7f688c06f7a0 2023-10-30T14:18:33.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f6892809640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688c066cf0 con 0x7f688c0c6bd0 2023-10-30T14:18:33.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f6890da6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f688c071110 0x7f688c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:33.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f6890da6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f688c071110 0x7f688c158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35266/0 (socket says 172.21.15.46:35266) 2023-10-30T14:18:33.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f6890da6640 1 -- 172.21.15.46:0/3009767064 learned_addr learned my addr 172.21.15.46:0/3009767064 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:33.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f688b7fe640 1 --2- 172.21.15.46:0/3009767064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f688c0c6bd0 0x7f688c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:33.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.443+0000 7f688bfff640 1 --2- 172.21.15.46:0/3009767064 >> v2:172.21.15.154:3300/0 conn(0x7f688c06f7a0 0x7f688c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:33.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6890da6640 1 -- 172.21.15.46:0/3009767064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f688c0c6bd0 msgr2=0x7f688c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:33.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6890da6640 1 --2- 172.21.15.46:0/3009767064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f688c0c6bd0 0x7f688c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:33.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6890da6640 1 -- 172.21.15.46:0/3009767064 >> v2:172.21.15.154:3300/0 conn(0x7f688c06f7a0 msgr2=0x7f688c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:33.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6890da6640 1 --2- 172.21.15.46:0/3009767064 >> v2:172.21.15.154:3300/0 conn(0x7f688c06f7a0 0x7f688c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:33.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6890da6640 1 -- 172.21.15.46:0/3009767064 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c0c70d0 con 0x7f688c071110 2023-10-30T14:18:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6890da6640 1 --2- 172.21.15.46:0/3009767064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f688c071110 0x7f688c158110 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f688c06c830 tx=0x7f688401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747cc2fc03d91093 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f688affd640 1 -- 172.21.15.46:0/3009767064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68840af400 con 0x7f688c071110 2023-10-30T14:18:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f688affd640 1 -- 172.21.15.46:0/3009767064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68840ab050 con 0x7f688c071110 2023-10-30T14:18:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f688affd640 1 -- 172.21.15.46:0/3009767064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68840af780 con 0x7f688c071110 2023-10-30T14:18:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6892809640 1 -- 172.21.15.46:0/3009767064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f688c071110 msgr2=0x7f688c158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6892809640 1 --2- 172.21.15.46:0/3009767064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f688c071110 0x7f688c158110 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f688c06c830 tx=0x7f688401f1c0 comp rx=0 tx=0).stop 2023-10-30T14:18:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6892809640 1 -- 172.21.15.46:0/3009767064 shutdown_connections 2023-10-30T14:18:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6892809640 1 --2- 172.21.15.46:0/3009767064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f688c071110 0x7f688c158110 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6892809640 1 --2- 172.21.15.46:0/3009767064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f688c0c6bd0 0x7f688c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6892809640 1 --2- 172.21.15.46:0/3009767064 >> v2:172.21.15.154:3300/0 conn(0x7f688c06f7a0 0x7f688c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6892809640 1 -- 172.21.15.46:0/3009767064 >> 172.21.15.46:0/3009767064 conn(0x7f688c0c7910 msgr2=0x7f688c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.447+0000 7f6892809640 1 -- 172.21.15.46:0/3009767064 shutdown_connections 2023-10-30T14:18:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f6892809640 1 -- 172.21.15.46:0/3009767064 wait complete. 2023-10-30T14:18:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f6892809640 1 Processor -- start 2023-10-30T14:18:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f6892809640 1 -- start start 2023-10-30T14:18:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f6892809640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f688c06f7a0 0x7f688c1f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f6892809640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f688c071110 0x7f688c14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f6892809640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f688c0c6bd0 0x7f688c149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f6892809640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688c14cc40 con 0x7f688c071110 2023-10-30T14:18:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f6892809640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688c1f3230 con 0x7f688c06f7a0 2023-10-30T14:18:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f6892809640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688c1f76f0 con 0x7f688c0c6bd0 2023-10-30T14:18:33.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f688b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f688c071110 0x7f688c14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:33.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f688b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f688c071110 0x7f688c14aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35268/0 (socket says 172.21.15.46:35268) 2023-10-30T14:18:33.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f688b7fe640 1 -- 172.21.15.46:0/1223348368 learned_addr learned my addr 172.21.15.46:0/1223348368 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:33.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f688bfff640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f688c06f7a0 0x7f688c1f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:33.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f6890da6640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f688c0c6bd0 0x7f688c149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f688b7fe640 1 -- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f688c0c6bd0 msgr2=0x7f688c149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f688b7fe640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f688c0c6bd0 0x7f688c149970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f688b7fe640 1 -- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f688c06f7a0 msgr2=0x7f688c1f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f688b7fe640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f688c06f7a0 0x7f688c1f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f688b7fe640 1 -- 172.21.15.46:0/1223348368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6884009b30 con 0x7f688c071110 2023-10-30T14:18:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f6890da6640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f688c0c6bd0 0x7f688c149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.451+0000 7f688b7fe640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f688c071110 0x7f688c14aa90 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f688000e870 tx=0x7f688000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.455+0000 7f6888ff9640 1 -- 172.21.15.46:0/1223348368 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6880009510 con 0x7f688c071110 2023-10-30T14:18:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.455+0000 7f6888ff9640 1 -- 172.21.15.46:0/1223348368 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688000eec0 con 0x7f688c071110 2023-10-30T14:18:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.455+0000 7f6892809640 1 -- 172.21.15.46:0/1223348368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688c14b630 con 0x7f688c071110 2023-10-30T14:18:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.455+0000 7f6888ff9640 1 -- 172.21.15.46:0/1223348368 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688000cb40 con 0x7f688c071110 2023-10-30T14:18:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.455+0000 7f6892809640 1 -- 172.21.15.46:0/1223348368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688c1f3720 con 0x7f688c071110 2023-10-30T14:18:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.455+0000 7f6892809640 1 -- 172.21.15.46:0/1223348368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f688c06ad80 con 0x7f688c071110 2023-10-30T14:18:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.455+0000 7f6888ff9640 1 -- 172.21.15.46:0/1223348368 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f688000cca0 con 0x7f688c071110 2023-10-30T14:18:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.455+0000 7f6888ff9640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68780664d0 0x7f6878068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.459+0000 7f6888ff9640 1 -- 172.21.15.46:0/1223348368 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7f6880084950 con 0x7f688c071110 2023-10-30T14:18:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.459+0000 7f688bfff640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68780664d0 0x7f6878068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.459+0000 7f688bfff640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68780664d0 0x7f6878068990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f687c0045f0 tx=0x7f687c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.463+0000 7f6888ff9640 1 -- 172.21.15.46:0/1223348368 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f68800513d0 con 0x7f688c071110 2023-10-30T14:18:33.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.651+0000 7f6892809640 1 -- 172.21.15.46:0/1223348368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f688c04e170 con 0x7f688c071110 2023-10-30T14:18:33.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.651+0000 7f6888ff9640 1 -- 172.21.15.46:0/1223348368 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f6880017090 con 0x7f688c071110 2023-10-30T14:18:33.655 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:33.656 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9426,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:41.742417+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:33.656 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 231 2023-10-30T14:18:33.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.655+0000 7f6892809640 1 -- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68780664d0 msgr2=0x7f6878068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:33.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.655+0000 7f6892809640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68780664d0 0x7f6878068990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f687c0045f0 tx=0x7f687c008040 comp rx=0 tx=0).stop 2023-10-30T14:18:33.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.655+0000 7f6892809640 1 -- 172.21.15.46:0/1223348368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f688c071110 msgr2=0x7f688c14aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:33.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.655+0000 7f6892809640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f688c071110 0x7f688c14aa90 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f688000e870 tx=0x7f688000ec70 comp rx=0 tx=0).stop 2023-10-30T14:18:33.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.655+0000 7f6892809640 1 -- 172.21.15.46:0/1223348368 shutdown_connections 2023-10-30T14:18:33.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.655+0000 7f6892809640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f68780664d0 0x7f6878068990 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:33.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.655+0000 7f6892809640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f688c0c6bd0 0x7f688c149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:33.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.655+0000 7f6892809640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f688c071110 0x7f688c14aa90 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:33.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.655+0000 7f6892809640 1 --2- 172.21.15.46:0/1223348368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f688c06f7a0 0x7f688c1f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:33.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.655+0000 7f6892809640 1 -- 172.21.15.46:0/1223348368 >> 172.21.15.46:0/1223348368 conn(0x7f688c0c7910 msgr2=0x7f688c0c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:33.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.655+0000 7f6892809640 1 -- 172.21.15.46:0/1223348368 shutdown_connections 2023-10-30T14:18:33.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:33.655+0000 7f6892809640 1 -- 172.21.15.46:0/1223348368 wait complete. 2023-10-30T14:18:33.677 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:18:35.678 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:35.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.831+0000 7f8a1eb65640 1 Processor -- start 2023-10-30T14:18:35.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.831+0000 7f8a1eb65640 1 -- start start 2023-10-30T14:18:35.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.831+0000 7f8a1eb65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a18070c90 0x7f8a18165130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:35.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.831+0000 7f8a1eb65640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180c2f20 con 0x7f8a18156f80 2023-10-30T14:18:35.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.831+0000 7f8a1eb65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180c85c0 con 0x7f8a18070650 2023-10-30T14:18:35.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.831+0000 7f8a1eb65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a18157360 con 0x7f8a18070c90 2023-10-30T14:18:35.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.831+0000 7f8a1c901640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8a18156f80 0x7f8a18159880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45002/0 (socket says 172.21.15.46:45002) 2023-10-30T14:18:35.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.831+0000 7f8a1c901640 1 -- 172.21.15.46:0/3340270823 learned_addr learned my addr 172.21.15.46:0/3340270823 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:18:35.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.831+0000 7f8a1d102640 1 --2- 172.21.15.46:0/3340270823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a18070c90 0x7f8a18165130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:35.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.831+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536455147 0 0) 0x7f8a180c2f20 con 0x7f8a18156f80 2023-10-30T14:18:35.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.831+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a08003530 con 0x7f8a18156f80 2023-10-30T14:18:35.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1471527162 0 0) 0x7f8a08003530 con 0x7f8a18156f80 2023-10-30T14:18:35.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a18070c90 msgr2=0x7f8a18165130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:35.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 --2- 172.21.15.46:0/3340270823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a18070c90 0x7f8a18165130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:35.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 >> v1:172.21.15.154:6789/0 conn(0x7f8a18070650 legacy=0x7f8a18071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:35.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a18071230 con 0x7f8a18156f80 2023-10-30T14:18:35.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f8a0c001680 con 0x7f8a18156f80 2023-10-30T14:18:35.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 >> v1:172.21.15.46:6789/0 conn(0x7f8a18156f80 legacy=0x7f8a18159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:35.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 --2- 172.21.15.46:0/3340270823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a080039b0 0x7f8a08003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:35.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 --2- 172.21.15.46:0/3340270823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a18070c90 0x7f8a080047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:35.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 --2- 172.21.15.46:0/3340270823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a08004d10 0x7f8a08005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:35.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a18157360 con 0x7f8a08004d10 2023-10-30T14:18:35.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0800b990 con 0x7f8a080039b0 2023-10-30T14:18:35.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0800bb50 con 0x7f8a18070c90 2023-10-30T14:18:35.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a1d102640 1 --2- 172.21.15.46:0/3340270823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a08004d10 0x7f8a08005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:35.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a1c901640 1 --2- 172.21.15.46:0/3340270823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a080039b0 0x7f8a08003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:35.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a17fff640 1 --2- 172.21.15.46:0/3340270823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a18070c90 0x7f8a080047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:35.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a1d102640 1 -- 172.21.15.46:0/3340270823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a18070c90 msgr2=0x7f8a080047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:35.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a1d102640 1 --2- 172.21.15.46:0/3340270823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a18070c90 0x7f8a080047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:35.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a1d102640 1 -- 172.21.15.46:0/3340270823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a080039b0 msgr2=0x7f8a08003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:35.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a1d102640 1 --2- 172.21.15.46:0/3340270823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a080039b0 0x7f8a08003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:35.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.835+0000 7f8a1d102640 1 -- 172.21.15.46:0/3340270823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a08003530 con 0x7f8a08004d10 2023-10-30T14:18:35.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a1d102640 1 --2- 172.21.15.46:0/3340270823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a08004d10 0x7f8a08005190 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009fc0 tx=0x7f8a1001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4540eba23d93c85 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:35.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a100a5b60 con 0x7f8a08004d10 2023-10-30T14:18:35.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1000f040 con 0x7f8a08004d10 2023-10-30T14:18:35.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a177fe640 1 -- 172.21.15.46:0/3340270823 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a100ad440 con 0x7f8a08004d10 2023-10-30T14:18:35.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a1eb65640 1 -- 172.21.15.46:0/3340270823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a08004d10 msgr2=0x7f8a08005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:35.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a1eb65640 1 --2- 172.21.15.46:0/3340270823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a08004d10 0x7f8a08005190 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009fc0 tx=0x7f8a1001cc60 comp rx=0 tx=0).stop 2023-10-30T14:18:35.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a1d102640 1 -- 172.21.15.46:0/3340270823 reap_dead start 2023-10-30T14:18:35.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a1eb65640 1 -- 172.21.15.46:0/3340270823 shutdown_connections 2023-10-30T14:18:35.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a1eb65640 1 -- 172.21.15.46:0/3340270823 >> 172.21.15.46:0/3340270823 conn(0x7f8a180cbeb0 msgr2=0x7f8a180cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:35.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a1eb65640 1 -- 172.21.15.46:0/3340270823 shutdown_connections 2023-10-30T14:18:35.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a1eb65640 1 -- 172.21.15.46:0/3340270823 wait complete. 2023-10-30T14:18:35.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a1eb65640 1 Processor -- start 2023-10-30T14:18:35.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a1eb65640 1 -- start start 2023-10-30T14:18:35.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a1eb65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a08004d10 0x7f8a181e0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:35.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.839+0000 7f8a1eb65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a181ee880 0x7f8a180ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:35.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1eb65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a180cae60 0x7f8a1814a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1eb65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a181df900 con 0x7f8a180cae60 2023-10-30T14:18:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1eb65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1814bc20 con 0x7f8a181ee880 2023-10-30T14:18:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1eb65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a181e17f0 con 0x7f8a08004d10 2023-10-30T14:18:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1d102640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a180cae60 0x7f8a1814a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:35.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1d102640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a180cae60 0x7f8a1814a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35282/0 (socket says 172.21.15.46:35282) 2023-10-30T14:18:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1d102640 1 -- 172.21.15.46:0/209388193 learned_addr learned my addr 172.21.15.46:0/209388193 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1c901640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a08004d10 0x7f8a181e0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a17fff640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a181ee880 0x7f8a180ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1d102640 1 -- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a08004d10 msgr2=0x7f8a181e0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:35.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1d102640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a08004d10 0x7f8a181e0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:35.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1d102640 1 -- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a181ee880 msgr2=0x7f8a180ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:35.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1d102640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a181ee880 0x7f8a180ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:35.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1d102640 1 -- 172.21.15.46:0/209388193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a181dfb10 con 0x7f8a180cae60 2023-10-30T14:18:35.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a17fff640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a181ee880 0x7f8a180ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:35.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1c901640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a08004d10 0x7f8a181e0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:35.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1d102640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a180cae60 0x7f8a1814a8d0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f8a10002240 tx=0x7f8a10009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:35.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a157fa640 1 -- 172.21.15.46:0/209388193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a100039a0 con 0x7f8a180cae60 2023-10-30T14:18:35.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a157fa640 1 -- 172.21.15.46:0/209388193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a10003b00 con 0x7f8a180cae60 2023-10-30T14:18:35.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a157fa640 1 -- 172.21.15.46:0/209388193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10003e70 con 0x7f8a180cae60 2023-10-30T14:18:35.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1eb65640 1 -- 172.21.15.46:0/209388193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a181dfdf0 con 0x7f8a180cae60 2023-10-30T14:18:35.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1eb65640 1 -- 172.21.15.46:0/209388193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a181e03b0 con 0x7f8a180cae60 2023-10-30T14:18:35.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a1eb65640 1 -- 172.21.15.46:0/209388193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a18072270 con 0x7f8a180cae60 2023-10-30T14:18:35.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.843+0000 7f8a157fa640 1 -- 172.21.15.46:0/209388193 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8a100b4040 con 0x7f8a180cae60 2023-10-30T14:18:35.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.847+0000 7f8a157fa640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89fc0664d0 0x7f89fc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:35.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.847+0000 7f8a157fa640 1 -- 172.21.15.46:0/209388193 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7f8a1011e130 con 0x7f8a180cae60 2023-10-30T14:18:35.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.847+0000 7f8a1c901640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89fc0664d0 0x7f89fc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:35.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.847+0000 7f8a1c901640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89fc0664d0 0x7f89fc068990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c001cc0 tx=0x7f8a0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:35.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:35.851+0000 7f8a157fa640 1 -- 172.21.15.46:0/209388193 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8a100eab80 con 0x7f8a180cae60 2023-10-30T14:18:36.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.039+0000 7f8a1eb65640 1 -- 172.21.15.46:0/209388193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a1804e170 con 0x7f8a180cae60 2023-10-30T14:18:36.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a157fa640 1 -- 172.21.15.46:0/209388193 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f8a100b2070 con 0x7f8a180cae60 2023-10-30T14:18:36.044 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:36.045 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9426,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:41.742417+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:36.045 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 231 2023-10-30T14:18:36.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a1eb65640 1 -- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89fc0664d0 msgr2=0x7f89fc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:36.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a1eb65640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89fc0664d0 0x7f89fc068990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c001cc0 tx=0x7f8a0c009040 comp rx=0 tx=0).stop 2023-10-30T14:18:36.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a1eb65640 1 -- 172.21.15.46:0/209388193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a180cae60 msgr2=0x7f8a1814a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:36.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a1eb65640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a180cae60 0x7f8a1814a8d0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f8a10002240 tx=0x7f8a10009750 comp rx=0 tx=0).stop 2023-10-30T14:18:36.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a1eb65640 1 -- 172.21.15.46:0/209388193 shutdown_connections 2023-10-30T14:18:36.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a1eb65640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f89fc0664d0 0x7f89fc068990 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:36.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a1eb65640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a180cae60 0x7f8a1814a8d0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:36.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a1eb65640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a181ee880 0x7f8a180ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:36.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a1eb65640 1 --2- 172.21.15.46:0/209388193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a08004d10 0x7f8a181e0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:36.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a1eb65640 1 -- 172.21.15.46:0/209388193 >> 172.21.15.46:0/209388193 conn(0x7f8a180cbeb0 msgr2=0x7f8a18153860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:36.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a1eb65640 1 -- 172.21.15.46:0/209388193 shutdown_connections 2023-10-30T14:18:36.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:36.043+0000 7f8a1eb65640 1 -- 172.21.15.46:0/209388193 wait complete. 2023-10-30T14:18:36.066 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:18:38.067 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.215+0000 7f4095fa1640 1 Processor -- start 2023-10-30T14:18:38.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.215+0000 7f4095fa1640 1 -- start start 2023-10-30T14:18:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f4095fa1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4090156f80 0x7f4090159880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f4095fa1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4090070c90 0x7f4090164120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f4095fa1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40900c2f20 con 0x7f4090070650 2023-10-30T14:18:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f4095fa1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4090157360 con 0x7f4090070c90 2023-10-30T14:18:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f4095fa1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40900c85c0 con 0x7f4090156f80 2023-10-30T14:18:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f408ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4090070c90 0x7f4090164120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f408ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4090070c90 0x7f4090164120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38760/0 (socket says 172.21.15.46:38760) 2023-10-30T14:18:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f408ffff640 1 -- 172.21.15.46:0/2722507995 learned_addr learned my addr 172.21.15.46:0/2722507995 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f408f7fe640 1 --2- 172.21.15.46:0/2722507995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4090156f80 0x7f4090159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f408ffff640 1 -- 172.21.15.46:0/2722507995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4090156f80 msgr2=0x7f4090159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f408ffff640 1 --2- 172.21.15.46:0/2722507995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4090156f80 0x7f4090159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f408ffff640 1 -- 172.21.15.46:0/2722507995 >> v1:172.21.15.154:6789/0 conn(0x7f4090070650 legacy=0x7f4090071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f408ffff640 1 -- 172.21.15.46:0/2722507995 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4090071230 con 0x7f4090070c90 2023-10-30T14:18:38.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.219+0000 7f408ffff640 1 --2- 172.21.15.46:0/2722507995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4090070c90 0x7f4090164120 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f4088009e50 tx=0x7f408801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a8ca82c26f0eee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:38.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f408e7fc640 1 -- 172.21.15.46:0/2722507995 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40880a58a0 con 0x7f4090070c90 2023-10-30T14:18:38.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f408e7fc640 1 -- 172.21.15.46:0/2722507995 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40880a5a00 con 0x7f4090070c90 2023-10-30T14:18:38.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f408e7fc640 1 -- 172.21.15.46:0/2722507995 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40880a5d90 con 0x7f4090070c90 2023-10-30T14:18:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 -- 172.21.15.46:0/2722507995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4090070c90 msgr2=0x7f4090164120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 --2- 172.21.15.46:0/2722507995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4090070c90 0x7f4090164120 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f4088009e50 tx=0x7f408801ca10 comp rx=0 tx=0).stop 2023-10-30T14:18:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 -- 172.21.15.46:0/2722507995 shutdown_connections 2023-10-30T14:18:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 --2- 172.21.15.46:0/2722507995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4090070c90 0x7f4090164120 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 --2- 172.21.15.46:0/2722507995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4090156f80 0x7f4090159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 -- 172.21.15.46:0/2722507995 >> 172.21.15.46:0/2722507995 conn(0x7f40900cbeb0 msgr2=0x7f40900cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 -- 172.21.15.46:0/2722507995 shutdown_connections 2023-10-30T14:18:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 -- 172.21.15.46:0/2722507995 wait complete. 2023-10-30T14:18:38.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 Processor -- start 2023-10-30T14:18:38.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 -- start start 2023-10-30T14:18:38.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4090070650 0x7f409006b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4090070c90 0x7f4090158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4090156f80 0x7f40901457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4090164290 con 0x7f4090070c90 2023-10-30T14:18:38.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4090157680 con 0x7f4090156f80 2023-10-30T14:18:38.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f4095fa1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4090156080 con 0x7f4090070650 2023-10-30T14:18:38.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f408effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4090070c90 0x7f4090158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f408effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4090070c90 0x7f4090158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38774/0 (socket says 172.21.15.46:38774) 2023-10-30T14:18:38.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.223+0000 7f408effd640 1 -- 172.21.15.46:0/2591791524 learned_addr learned my addr 172.21.15.46:0/2591791524 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:38.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f408ffff640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4090156f80 0x7f40901457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f408f7fe640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4090070650 0x7f409006b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f408effd640 1 -- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4090070650 msgr2=0x7f409006b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:38.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f408effd640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4090070650 0x7f409006b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f408effd640 1 -- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4090156f80 msgr2=0x7f40901457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:38.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f408effd640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4090156f80 0x7f40901457c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f408effd640 1 -- 172.21.15.46:0/2591791524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4088009b30 con 0x7f4090070c90 2023-10-30T14:18:38.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f408effd640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4090070c90 0x7f4090158140 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f408400c390 tx=0x7f408400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:38.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f4077fff640 1 -- 172.21.15.46:0/2591791524 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40840052d0 con 0x7f4090070c90 2023-10-30T14:18:38.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f4077fff640 1 -- 172.21.15.46:0/2591791524 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4084005430 con 0x7f4090070c90 2023-10-30T14:18:38.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f4077fff640 1 -- 172.21.15.46:0/2591791524 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4084003820 con 0x7f4090070c90 2023-10-30T14:18:38.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f4095fa1640 1 -- 172.21.15.46:0/2591791524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4090157960 con 0x7f4090070c90 2023-10-30T14:18:38.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f4095fa1640 1 -- 172.21.15.46:0/2591791524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409006bae0 con 0x7f4090070c90 2023-10-30T14:18:38.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f4095fa1640 1 -- 172.21.15.46:0/2591791524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40901574e0 con 0x7f4090070c90 2023-10-30T14:18:38.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f4077fff640 1 -- 172.21.15.46:0/2591791524 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4084003980 con 0x7f4090070c90 2023-10-30T14:18:38.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f4077fff640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f407c0664d0 0x7f407c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:38.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.227+0000 7f4077fff640 1 -- 172.21.15.46:0/2591791524 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7f4084083cb0 con 0x7f4090070c90 2023-10-30T14:18:38.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.231+0000 7f408f7fe640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f407c0664d0 0x7f407c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:38.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.231+0000 7f408f7fe640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f407c0664d0 0x7f407c068990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4080009b60 tx=0x7f4080009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:38.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.231+0000 7f4077fff640 1 -- 172.21.15.46:0/2591791524 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f40901574e0 con 0x7f4090070c90 2023-10-30T14:18:38.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 -- 172.21.15.46:0/2591791524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40901574e0 con 0x7f4090070c90 2023-10-30T14:18:38.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4077fff640 1 -- 172.21.15.46:0/2591791524 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f40901574e0 con 0x7f4090070c90 2023-10-30T14:18:38.424 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:38.425 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9426,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:41.742417+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:38.425 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 231 2023-10-30T14:18:38.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 -- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f407c0664d0 msgr2=0x7f407c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:38.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f407c0664d0 0x7f407c068990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4080009b60 tx=0x7f4080009190 comp rx=0 tx=0).stop 2023-10-30T14:18:38.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 -- 172.21.15.46:0/2591791524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4090070c90 msgr2=0x7f4090158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:38.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4090070c90 0x7f4090158140 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f408400c390 tx=0x7f408400c850 comp rx=0 tx=0).stop 2023-10-30T14:18:38.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 -- 172.21.15.46:0/2591791524 shutdown_connections 2023-10-30T14:18:38.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f407c0664d0 0x7f407c068990 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4090156f80 0x7f40901457c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4090070c90 0x7f4090158140 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 --2- 172.21.15.46:0/2591791524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4090070650 0x7f409006b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:38.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 -- 172.21.15.46:0/2591791524 >> 172.21.15.46:0/2591791524 conn(0x7f40900cbeb0 msgr2=0x7f409006a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:38.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 -- 172.21.15.46:0/2591791524 shutdown_connections 2023-10-30T14:18:38.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:38.423+0000 7f4095fa1640 1 -- 172.21.15.46:0/2591791524 wait complete. 2023-10-30T14:18:38.446 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:18:40.447 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:40.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.599+0000 7fb6f4c2f640 1 Processor -- start 2023-10-30T14:18:40.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.599+0000 7fb6f4c2f640 1 -- start start 2023-10-30T14:18:40.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.599+0000 7fb6f4c2f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6f01538f0 0x7fb6f0159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:40.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.599+0000 7fb6f4c2f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb6f015cd50 0x7fb6f00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:40.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.599+0000 7fb6f4c2f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb6f00cefa0 0x7fb6f00cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:40.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.599+0000 7fb6f4c2f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f015d300 con 0x7fb6f00cefa0 2023-10-30T14:18:40.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.599+0000 7fb6f4c2f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f015d5e0 con 0x7fb6f015cd50 2023-10-30T14:18:40.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.599+0000 7fb6f4c2f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f015d8c0 con 0x7fb6f01538f0 2023-10-30T14:18:40.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6effff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb6f00cefa0 0x7fb6f00cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:40.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6effff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb6f00cefa0 0x7fb6f00cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38788/0 (socket says 172.21.15.46:38788) 2023-10-30T14:18:40.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6effff640 1 -- 172.21.15.46:0/2454411564 learned_addr learned my addr 172.21.15.46:0/2454411564 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:40.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6ef7fe640 1 --2- 172.21.15.46:0/2454411564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6f01538f0 0x7fb6f0159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:40.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6eeffd640 1 --2- 172.21.15.46:0/2454411564 >> v2:172.21.15.154:3300/0 conn(0x7fb6f015cd50 0x7fb6f00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:40.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6effff640 1 -- 172.21.15.46:0/2454411564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6f01538f0 msgr2=0x7fb6f0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:40.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6effff640 1 --2- 172.21.15.46:0/2454411564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6f01538f0 0x7fb6f0159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:40.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6effff640 1 -- 172.21.15.46:0/2454411564 >> v2:172.21.15.154:3300/0 conn(0x7fb6f015cd50 msgr2=0x7fb6f00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:40.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6effff640 1 --2- 172.21.15.46:0/2454411564 >> v2:172.21.15.154:3300/0 conn(0x7fb6f015cd50 0x7fb6f00c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:40.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6effff640 1 -- 172.21.15.46:0/2454411564 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f015dba0 con 0x7fb6f00cefa0 2023-10-30T14:18:40.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6eeffd640 1 --2- 172.21.15.46:0/2454411564 >> v2:172.21.15.154:3300/0 conn(0x7fb6f015cd50 0x7fb6f00c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:40.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6effff640 1 --2- 172.21.15.46:0/2454411564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f00cefa0 0x7fb6f00cf3f0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0066410 tx=0x7fb6e801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5365ab792223d81e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:40.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6ee7fc640 1 -- 172.21.15.46:0/2454411564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e80af400 con 0x7fb6f00cefa0 2023-10-30T14:18:40.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6ee7fc640 1 -- 172.21.15.46:0/2454411564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e80ab050 con 0x7fb6f00cefa0 2023-10-30T14:18:40.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.603+0000 7fb6ee7fc640 1 -- 172.21.15.46:0/2454411564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e80af780 con 0x7fb6f00cefa0 2023-10-30T14:18:40.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/2454411564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f00cefa0 msgr2=0x7fb6f00cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:40.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 --2- 172.21.15.46:0/2454411564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f00cefa0 0x7fb6f00cf3f0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0066410 tx=0x7fb6e801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:18:40.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/2454411564 shutdown_connections 2023-10-30T14:18:40.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 --2- 172.21.15.46:0/2454411564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f00cefa0 0x7fb6f00cf3f0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:40.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 --2- 172.21.15.46:0/2454411564 >> v2:172.21.15.154:3300/0 conn(0x7fb6f015cd50 0x7fb6f00c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 --2- 172.21.15.46:0/2454411564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6f01538f0 0x7fb6f0159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/2454411564 >> 172.21.15.46:0/2454411564 conn(0x7fb6f006a640 msgr2=0x7fb6f0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/2454411564 shutdown_connections 2023-10-30T14:18:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/2454411564 wait complete. 2023-10-30T14:18:40.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 Processor -- start 2023-10-30T14:18:40.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 -- start start 2023-10-30T14:18:40.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6f00cefa0 0x7fb6f0144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6f01538f0 0x7fb6f0152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f015cd50 0x7fb6f0145530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f01effd0 con 0x7fb6f015cd50 2023-10-30T14:18:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f0144d90 con 0x7fb6f00cefa0 2023-10-30T14:18:40.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6f4c2f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f01efa50 con 0x7fb6f01538f0 2023-10-30T14:18:40.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f015cd50 0x7fb6f0145530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:40.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f015cd50 0x7fb6f0145530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38794/0 (socket says 172.21.15.46:38794) 2023-10-30T14:18:40.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6effff640 1 -- 172.21.15.46:0/175671656 learned_addr learned my addr 172.21.15.46:0/175671656 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:40.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6ef7fe640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6f00cefa0 0x7fb6f0144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:40.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.607+0000 7fb6eeffd640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6f01538f0 0x7fb6f0152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:40.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6effff640 1 -- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6f01538f0 msgr2=0x7fb6f0152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:40.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6effff640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6f01538f0 0x7fb6f0152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:40.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6effff640 1 -- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6f00cefa0 msgr2=0x7fb6f0144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:40.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6effff640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6f00cefa0 0x7fb6f0144640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:40.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6effff640 1 -- 172.21.15.46:0/175671656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e8009b30 con 0x7fb6f015cd50 2023-10-30T14:18:40.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6effff640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f015cd50 0x7fb6f0145530 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fb6e80b2040 tx=0x7fb6e801bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6d7fff640 1 -- 172.21.15.46:0/175671656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e80afd30 con 0x7fb6f015cd50 2023-10-30T14:18:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6d7fff640 1 -- 172.21.15.46:0/175671656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e8009510 con 0x7fb6f015cd50 2023-10-30T14:18:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6d7fff640 1 -- 172.21.15.46:0/175671656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e80ba660 con 0x7fb6f015cd50 2023-10-30T14:18:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/175671656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f01efcf0 con 0x7fb6f015cd50 2023-10-30T14:18:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/175671656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f0146b80 con 0x7fb6f015cd50 2023-10-30T14:18:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/175671656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f0146d70 con 0x7fb6f015cd50 2023-10-30T14:18:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6d7fff640 1 -- 172.21.15.46:0/175671656 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb6e80b6040 con 0x7fb6f015cd50 2023-10-30T14:18:40.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6d7fff640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb6dc0664a0 0x7fb6dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:40.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.611+0000 7fb6d7fff640 1 -- 172.21.15.46:0/175671656 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7fb6e8122a90 con 0x7fb6f015cd50 2023-10-30T14:18:40.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.615+0000 7fb6ef7fe640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb6dc0664a0 0x7fb6dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:40.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.615+0000 7fb6ef7fe640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb6dc0664a0 0x7fb6dc068960 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0009ae0 tx=0x7fb6e0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:40.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.615+0000 7fb6d7fff640 1 -- 172.21.15.46:0/175671656 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb6f0146d70 con 0x7fb6f015cd50 2023-10-30T14:18:40.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.807+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/175671656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6f0146d70 con 0x7fb6f015cd50 2023-10-30T14:18:40.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.807+0000 7fb6d7fff640 1 -- 172.21.15.46:0/175671656 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fb6f0146d70 con 0x7fb6f015cd50 2023-10-30T14:18:40.811 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:40.812 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9426,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:41.742417+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:40.812 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 231 2023-10-30T14:18:40.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.811+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb6dc0664a0 msgr2=0x7fb6dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:40.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.811+0000 7fb6f4c2f640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb6dc0664a0 0x7fb6dc068960 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0009ae0 tx=0x7fb6e0009190 comp rx=0 tx=0).stop 2023-10-30T14:18:40.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.811+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/175671656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f015cd50 msgr2=0x7fb6f0145530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:40.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.811+0000 7fb6f4c2f640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f015cd50 0x7fb6f0145530 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fb6e80b2040 tx=0x7fb6e801bf60 comp rx=0 tx=0).stop 2023-10-30T14:18:40.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.811+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/175671656 shutdown_connections 2023-10-30T14:18:40.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.811+0000 7fb6f4c2f640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb6dc0664a0 0x7fb6dc068960 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:40.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.811+0000 7fb6f4c2f640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6f015cd50 0x7fb6f0145530 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:40.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.811+0000 7fb6f4c2f640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6f01538f0 0x7fb6f0152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:40.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.811+0000 7fb6f4c2f640 1 --2- 172.21.15.46:0/175671656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6f00cefa0 0x7fb6f0144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:40.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.811+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/175671656 >> 172.21.15.46:0/175671656 conn(0x7fb6f006a640 msgr2=0x7fb6f00cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:40.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.811+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/175671656 shutdown_connections 2023-10-30T14:18:40.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:40.811+0000 7fb6f4c2f640 1 -- 172.21.15.46:0/175671656 wait complete. 2023-10-30T14:18:40.833 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:18:42.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:42.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90f4964640 1 Processor -- start 2023-10-30T14:18:42.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90f4964640 1 -- start start 2023-10-30T14:18:42.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90f4964640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90f015cd50 0x7f90f00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:42.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90f4964640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f00cefa0 0x7f90f00cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:42.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90f4964640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f015a6c0 con 0x7f90f01538f0 2023-10-30T14:18:42.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90f4964640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f015d300 con 0x7f90f015cd50 2023-10-30T14:18:42.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90f4964640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f015d5e0 con 0x7f90f00cefa0 2023-10-30T14:18:42.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90ee59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90f015cd50 0x7f90f00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:42.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90ee59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90f015cd50 0x7f90f00c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38800/0 (socket says 172.21.15.46:38800) 2023-10-30T14:18:42.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90ee59c640 1 -- 172.21.15.46:0/3498519823 learned_addr learned my addr 172.21.15.46:0/3498519823 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:42.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90ef59e640 1 --2- 172.21.15.46:0/3498519823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f00cefa0 0x7f90f00cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:42.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90ee59c640 1 -- 172.21.15.46:0/3498519823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f00cefa0 msgr2=0x7f90f00cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:42.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90ee59c640 1 --2- 172.21.15.46:0/3498519823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f00cefa0 0x7f90f00cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:42.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90ee59c640 1 -- 172.21.15.46:0/3498519823 >> v1:172.21.15.154:6789/0 conn(0x7f90f01538f0 legacy=0x7f90f0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:42.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.987+0000 7f90ee59c640 1 -- 172.21.15.46:0/3498519823 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f015d8c0 con 0x7f90f015cd50 2023-10-30T14:18:42.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.991+0000 7f90ee59c640 1 --2- 172.21.15.46:0/3498519823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f015cd50 0x7f90f00c9f90 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f90e4009870 tx=0x7f90e401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28af7d1a8fbf6631 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:42.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.991+0000 7f90edd9b640 1 -- 172.21.15.46:0/3498519823 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e40a8a30 con 0x7f90f015cd50 2023-10-30T14:18:42.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.991+0000 7f90edd9b640 1 -- 172.21.15.46:0/3498519823 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e40a8b90 con 0x7f90f015cd50 2023-10-30T14:18:42.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.991+0000 7f90edd9b640 1 -- 172.21.15.46:0/3498519823 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e40b0440 con 0x7f90f015cd50 2023-10-30T14:18:42.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.991+0000 7f90f4964640 1 -- 172.21.15.46:0/3498519823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f015cd50 msgr2=0x7f90f00c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:42.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.991+0000 7f90f4964640 1 --2- 172.21.15.46:0/3498519823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f015cd50 0x7f90f00c9f90 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f90e4009870 tx=0x7f90e401ed30 comp rx=0 tx=0).stop 2023-10-30T14:18:42.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.991+0000 7f90f4964640 1 -- 172.21.15.46:0/3498519823 shutdown_connections 2023-10-30T14:18:42.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.991+0000 7f90f4964640 1 --2- 172.21.15.46:0/3498519823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f00cefa0 0x7f90f00cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:42.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.991+0000 7f90f4964640 1 --2- 172.21.15.46:0/3498519823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f015cd50 0x7f90f00c9f90 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:42.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.991+0000 7f90f4964640 1 -- 172.21.15.46:0/3498519823 >> 172.21.15.46:0/3498519823 conn(0x7f90f006a640 msgr2=0x7f90f0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:42.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.991+0000 7f90f4964640 1 -- 172.21.15.46:0/3498519823 shutdown_connections 2023-10-30T14:18:42.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.991+0000 7f90f4964640 1 -- 172.21.15.46:0/3498519823 wait complete. 2023-10-30T14:18:42.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90f4964640 1 Processor -- start 2023-10-30T14:18:42.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90f4964640 1 -- start start 2023-10-30T14:18:42.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90f4964640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90f00cefa0 0x7f90f01e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:42.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90f4964640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f01538f0 0x7f90f01f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:42.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90f4964640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f015cd50 0x7f90f013be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:42.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90f4964640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f01f3700 con 0x7f90f01538f0 2023-10-30T14:18:42.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90f4964640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f01e5bd0 con 0x7f90f00cefa0 2023-10-30T14:18:42.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90f4964640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f01e5650 con 0x7f90f015cd50 2023-10-30T14:18:42.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90ee59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f01538f0 0x7f90f01f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90ee59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f01538f0 0x7f90f01f31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38808/0 (socket says 172.21.15.46:38808) 2023-10-30T14:18:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90ee59c640 1 -- 172.21.15.46:0/1614837002 learned_addr learned my addr 172.21.15.46:0/1614837002 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90ef59e640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f015cd50 0x7f90f013be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90eed9d640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90f00cefa0 0x7f90f01e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90ee59c640 1 -- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f015cd50 msgr2=0x7f90f013be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90ee59c640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f015cd50 0x7f90f013be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90ee59c640 1 -- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90f00cefa0 msgr2=0x7f90f01e3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90ee59c640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90f00cefa0 0x7f90f01e3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90ee59c640 1 -- 172.21.15.46:0/1614837002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e40094d0 con 0x7f90f01538f0 2023-10-30T14:18:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90ee59c640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f01538f0 0x7f90f01f31c0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f90e40b3040 tx=0x7f90e401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90d77fe640 1 -- 172.21.15.46:0/1614837002 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e40be070 con 0x7f90f01538f0 2023-10-30T14:18:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90d77fe640 1 -- 172.21.15.46:0/1614837002 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e40ba400 con 0x7f90f01538f0 2023-10-30T14:18:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90d77fe640 1 -- 172.21.15.46:0/1614837002 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e40ba7d0 con 0x7f90f01538f0 2023-10-30T14:18:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90f4964640 1 -- 172.21.15.46:0/1614837002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f01ef3e0 con 0x7f90f01538f0 2023-10-30T14:18:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90f4964640 1 -- 172.21.15.46:0/1614837002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f01ef680 con 0x7f90f01538f0 2023-10-30T14:18:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.995+0000 7f90f4964640 1 -- 172.21.15.46:0/1614837002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f01ef240 con 0x7f90f01538f0 2023-10-30T14:18:43.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.999+0000 7f90d77fe640 1 -- 172.21.15.46:0/1614837002 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f90e40c3410 con 0x7f90f01538f0 2023-10-30T14:18:43.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.999+0000 7f90d77fe640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90dc0664a0 0x7f90dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:43.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.999+0000 7f90d77fe640 1 -- 172.21.15.46:0/1614837002 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8262+0+0 (secure 0 0 0) 0x7f90e41224d0 con 0x7f90f01538f0 2023-10-30T14:18:43.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.999+0000 7f90eed9d640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90dc0664a0 0x7f90dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:43.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:42.999+0000 7f90eed9d640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90dc0664a0 0x7f90dc068960 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f90e0000f70 tx=0x7f90e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:43.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.003+0000 7f90d77fe640 1 -- 172.21.15.46:0/1614837002 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f90f01ef240 con 0x7f90f01538f0 2023-10-30T14:18:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 -- 172.21.15.46:0/1614837002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90f01ef240 con 0x7f90f01538f0 2023-10-30T14:18:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90d77fe640 1 -- 172.21.15.46:0/1614837002 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f90f01ef240 con 0x7f90f01538f0 2023-10-30T14:18:43.197 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:43.197 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9426,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:16:41.742417+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9135,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9135":{"gid":9135,"name":"e","rank":0,"incarnation":218,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6837/681817982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":681817982},{"type":"v1","addr":"172.21.15.46:6837","nonce":681817982}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 231 2023-10-30T14:18:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 -- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90dc0664a0 msgr2=0x7f90dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90dc0664a0 0x7f90dc068960 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f90e0000f70 tx=0x7f90e0009040 comp rx=0 tx=0).stop 2023-10-30T14:18:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 -- 172.21.15.46:0/1614837002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f01538f0 msgr2=0x7f90f01f31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f01538f0 0x7f90f01f31c0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f90e40b3040 tx=0x7f90e401bad0 comp rx=0 tx=0).stop 2023-10-30T14:18:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 -- 172.21.15.46:0/1614837002 shutdown_connections 2023-10-30T14:18:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90dc0664a0 0x7f90dc068960 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f015cd50 0x7f90f013be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f01538f0 0x7f90f01f31c0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 --2- 172.21.15.46:0/1614837002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90f00cefa0 0x7f90f01e3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 -- 172.21.15.46:0/1614837002 >> 172.21.15.46:0/1614837002 conn(0x7f90f006a640 msgr2=0x7f90f00c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 -- 172.21.15.46:0/1614837002 shutdown_connections 2023-10-30T14:18:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:43.195+0000 7f90f4964640 1 -- 172.21.15.46:0/1614837002 wait complete. 2023-10-30T14:18:43.218 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:18:45.219 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:45.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.371+0000 7fbb80c79640 1 Processor -- start 2023-10-30T14:18:45.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.371+0000 7fbb80c79640 1 -- start start 2023-10-30T14:18:45.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.371+0000 7fbb80c79640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb7c0c05d0 0x7fbb7c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:45.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.371+0000 7fbb80c79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb7c0c6f60 0x7fbb7c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:45.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.371+0000 7fbb80c79640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7c0c0a50 con 0x7fbb7c0c6b80 2023-10-30T14:18:45.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.371+0000 7fbb80c79640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb7c06f6e0 con 0x7fbb7c0c05d0 2023-10-30T14:18:45.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.371+0000 7fbb80c79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c06f9c0 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.371+0000 7fbb79d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb7c0c05d0 0x7fbb7c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:45.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.371+0000 7fbb7a59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbb7c0c6b80 0x7fbb7c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50258/0 (socket says 172.21.15.46:50258) 2023-10-30T14:18:45.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.371+0000 7fbb7a59c640 1 -- 172.21.15.46:0/340723071 learned_addr learned my addr 172.21.15.46:0/340723071 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:18:45.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.371+0000 7fbb7ad9d640 1 --2- 172.21.15.46:0/340723071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb7c0c6f60 0x7fbb7c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:45.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb79d9b640 1 -- 172.21.15.46:0/340723071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb7c0c6f60 msgr2=0x7fbb7c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:45.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb79d9b640 1 --2- 172.21.15.46:0/340723071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb7c0c6f60 0x7fbb7c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb79d9b640 1 -- 172.21.15.46:0/340723071 >> v1:172.21.15.46:6789/0 conn(0x7fbb7c0c6b80 legacy=0x7fbb7c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:45.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb79d9b640 1 -- 172.21.15.46:0/340723071 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7c06fca0 con 0x7fbb7c0c05d0 2023-10-30T14:18:45.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb7959a640 1 -- 172.21.15.46:0/340723071 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205375831 0 0) 0x7fbb7c0c0a50 con 0x7fbb7c0c6b80 2023-10-30T14:18:45.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb79d9b640 1 --2- 172.21.15.46:0/340723071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb7c0c05d0 0x7fbb7c13ed90 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fbb700097c0 tx=0x7fbb7001e690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a7ba37d6c83d1e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:45.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb7959a640 1 -- 172.21.15.46:0/340723071 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb700ab670 con 0x7fbb7c0c05d0 2023-10-30T14:18:45.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb7959a640 1 -- 172.21.15.46:0/340723071 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb700ab7d0 con 0x7fbb7c0c05d0 2023-10-30T14:18:45.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb7959a640 1 -- 172.21.15.46:0/340723071 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb700abb60 con 0x7fbb7c0c05d0 2023-10-30T14:18:45.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb80c79640 1 -- 172.21.15.46:0/340723071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb7c0c05d0 msgr2=0x7fbb7c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:45.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb80c79640 1 --2- 172.21.15.46:0/340723071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb7c0c05d0 0x7fbb7c13ed90 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fbb700097c0 tx=0x7fbb7001e690 comp rx=0 tx=0).stop 2023-10-30T14:18:45.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb80c79640 1 -- 172.21.15.46:0/340723071 shutdown_connections 2023-10-30T14:18:45.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb80c79640 1 --2- 172.21.15.46:0/340723071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb7c0c6f60 0x7fbb7c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.375+0000 7fbb80c79640 1 --2- 172.21.15.46:0/340723071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb7c0c05d0 0x7fbb7c13ed90 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb80c79640 1 -- 172.21.15.46:0/340723071 >> 172.21.15.46:0/340723071 conn(0x7fbb7c0c7910 msgr2=0x7fbb7c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:45.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb80c79640 1 -- 172.21.15.46:0/340723071 shutdown_connections 2023-10-30T14:18:45.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb80c79640 1 -- 172.21.15.46:0/340723071 wait complete. 2023-10-30T14:18:45.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb80c79640 1 Processor -- start 2023-10-30T14:18:45.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb80c79640 1 -- start start 2023-10-30T14:18:45.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb80c79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb7c0c05d0 0x7fbb7c138fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:45.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb80c79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb7c0c6b80 0x7fbb7c133e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:45.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb80c79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb7c0c6f60 0x7fbb7c13c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:45.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb80c79640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c1f5ed0 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb80c79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c0cab90 con 0x7fbb7c0c6b80 2023-10-30T14:18:45.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb80c79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c139520 con 0x7fbb7c0c05d0 2023-10-30T14:18:45.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb7ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb7c0c6f60 0x7fbb7c13c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:45.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb7ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb7c0c6f60 0x7fbb7c13c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38826/0 (socket says 172.21.15.46:38826) 2023-10-30T14:18:45.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb7ad9d640 1 -- 172.21.15.46:0/566423878 learned_addr learned my addr 172.21.15.46:0/566423878 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:45.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb79d9b640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb7c0c6b80 0x7fbb7c133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:45.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb7a59c640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb7c0c05d0 0x7fbb7c138fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:45.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb7ad9d640 1 -- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb7c0c05d0 msgr2=0x7fbb7c138fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:45.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb7ad9d640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb7c0c05d0 0x7fbb7c138fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb7ad9d640 1 -- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb7c0c6b80 msgr2=0x7fbb7c133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:45.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb7ad9d640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb7c0c6b80 0x7fbb7c133e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.379+0000 7fbb7ad9d640 1 -- 172.21.15.46:0/566423878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb70009420 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.383+0000 7fbb7ad9d640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb7c0c6f60 0x7fbb7c13c810 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fbb7400efc0 tx=0x7fbb7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:45.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.383+0000 7fbb6affd640 1 -- 172.21.15.46:0/566423878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb74009ba0 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.383+0000 7fbb6affd640 1 -- 172.21.15.46:0/566423878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7400f040 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.383+0000 7fbb80c79640 1 -- 172.21.15.46:0/566423878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb7c13e210 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.383+0000 7fbb6affd640 1 -- 172.21.15.46:0/566423878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb740146c0 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.383+0000 7fbb80c79640 1 -- 172.21.15.46:0/566423878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb7c0ca8b0 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.383+0000 7fbb80c79640 1 -- 172.21.15.46:0/566423878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7c0ca710 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.383+0000 7fbb6affd640 1 -- 172.21.15.46:0/566423878 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb74014820 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.383+0000 7fbb6affd640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbb5c0664a0 0x7fbb5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.383+0000 7fbb6affd640 1 -- 172.21.15.46:0/566423878 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7fbb74085640 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.383+0000 7fbb7a59c640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbb5c0664a0 0x7fbb5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:45.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.387+0000 7fbb7a59c640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbb5c0664a0 0x7fbb5c068960 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c000f70 tx=0x7fbb6c007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:45.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.387+0000 7fbb6affd640 1 -- 172.21.15.46:0/566423878 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb7c0ca710 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.575+0000 7fbb80c79640 1 -- 172.21.15.46:0/566423878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb7c1f6010 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.575+0000 7fbb6affd640 1 -- 172.21.15.46:0/566423878 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fbb7c1f6010 con 0x7fbb7c0c6f60 2023-10-30T14:18:45.579 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:45.579 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:18:44.741668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:45.580 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 232 2023-10-30T14:18:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.579+0000 7fbb80c79640 1 -- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbb5c0664a0 msgr2=0x7fbb5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.579+0000 7fbb80c79640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbb5c0664a0 0x7fbb5c068960 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c000f70 tx=0x7fbb6c007b60 comp rx=0 tx=0).stop 2023-10-30T14:18:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.579+0000 7fbb80c79640 1 -- 172.21.15.46:0/566423878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb7c0c6f60 msgr2=0x7fbb7c13c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:45.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.579+0000 7fbb80c79640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb7c0c6f60 0x7fbb7c13c810 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fbb7400efc0 tx=0x7fbb7400c490 comp rx=0 tx=0).stop 2023-10-30T14:18:45.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.579+0000 7fbb80c79640 1 -- 172.21.15.46:0/566423878 shutdown_connections 2023-10-30T14:18:45.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.579+0000 7fbb80c79640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbb5c0664a0 0x7fbb5c068960 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.579+0000 7fbb80c79640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb7c0c6f60 0x7fbb7c13c810 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.579+0000 7fbb80c79640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb7c0c6b80 0x7fbb7c133e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.579+0000 7fbb80c79640 1 --2- 172.21.15.46:0/566423878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb7c0c05d0 0x7fbb7c138fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.579+0000 7fbb80c79640 1 -- 172.21.15.46:0/566423878 >> 172.21.15.46:0/566423878 conn(0x7fbb7c0c7910 msgr2=0x7fbb7c13ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:45.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.579+0000 7fbb80c79640 1 -- 172.21.15.46:0/566423878 shutdown_connections 2023-10-30T14:18:45.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.579+0000 7fbb80c79640 1 -- 172.21.15.46:0/566423878 wait complete. 2023-10-30T14:18:45.601 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T14:18:45.602 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:18:45.602 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:45.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f036b66b640 1 Processor -- start 2023-10-30T14:18:45.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f036b66b640 1 -- start start 2023-10-30T14:18:45.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f036b66b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0364153d70 0x7f036406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:45.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f036b66b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f03640c8490 0x7f03640c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:45.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f036b66b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036406afa0 con 0x7f036406e470 2023-10-30T14:18:45.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f036b66b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f036406d780 con 0x7f03640c8490 2023-10-30T14:18:45.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f036b66b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0364154150 con 0x7f0364153d70 2023-10-30T14:18:45.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f0368c06640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f036406e470 0x7f03640cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50266/0 (socket says 172.21.15.46:50266) 2023-10-30T14:18:45.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f0369c08640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f03640c8490 0x7f03640c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:45.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f0368c06640 1 -- 172.21.15.46:0/4094698035 learned_addr learned my addr 172.21.15.46:0/4094698035 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:18:45.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f0369407640 1 --2- 172.21.15.46:0/4094698035 >> v2:172.21.15.154:3300/0 conn(0x7f0364153d70 0x7f036406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:45.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f0369c08640 1 -- 172.21.15.46:0/4094698035 >> v2:172.21.15.154:3300/0 conn(0x7f0364153d70 msgr2=0x7f036406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:45.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f0369c08640 1 --2- 172.21.15.46:0/4094698035 >> v2:172.21.15.154:3300/0 conn(0x7f0364153d70 0x7f036406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f0369c08640 1 -- 172.21.15.46:0/4094698035 >> v1:172.21.15.46:6789/0 conn(0x7f036406e470 legacy=0x7f03640cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:45.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f0369c08640 1 -- 172.21.15.46:0/4094698035 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036406e030 con 0x7f03640c8490 2023-10-30T14:18:45.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.743+0000 7f035bfff640 1 -- 172.21.15.46:0/4094698035 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502340451 0 0) 0x7f036406afa0 con 0x7f036406e470 2023-10-30T14:18:45.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.747+0000 7f0369c08640 1 --2- 172.21.15.46:0/4094698035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03640c8490 0x7f03640c8cc0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f036406d180 tx=0x7f035c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9c70d7a0f721f04 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:45.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.747+0000 7f035bfff640 1 -- 172.21.15.46:0/4094698035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c0a59a0 con 0x7f03640c8490 2023-10-30T14:18:45.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.747+0000 7f035bfff640 1 -- 172.21.15.46:0/4094698035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035c0a5b00 con 0x7f03640c8490 2023-10-30T14:18:45.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.747+0000 7f035bfff640 1 -- 172.21.15.46:0/4094698035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c0a5e90 con 0x7f03640c8490 2023-10-30T14:18:45.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.747+0000 7f036b66b640 1 -- 172.21.15.46:0/4094698035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03640c8490 msgr2=0x7f03640c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:45.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.747+0000 7f036b66b640 1 --2- 172.21.15.46:0/4094698035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03640c8490 0x7f03640c8cc0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f036406d180 tx=0x7f035c01c8e0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.747+0000 7f036b66b640 1 -- 172.21.15.46:0/4094698035 shutdown_connections 2023-10-30T14:18:45.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.747+0000 7f036b66b640 1 --2- 172.21.15.46:0/4094698035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03640c8490 0x7f03640c8cc0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.747+0000 7f036b66b640 1 --2- 172.21.15.46:0/4094698035 >> v2:172.21.15.154:3300/0 conn(0x7f0364153d70 0x7f036406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.747+0000 7f036b66b640 1 -- 172.21.15.46:0/4094698035 >> 172.21.15.46:0/4094698035 conn(0x7f0364071ea0 msgr2=0x7f0364065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:45.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.747+0000 7f036b66b640 1 -- 172.21.15.46:0/4094698035 shutdown_connections 2023-10-30T14:18:45.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.747+0000 7f036b66b640 1 -- 172.21.15.46:0/4094698035 wait complete. 2023-10-30T14:18:45.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f036b66b640 1 Processor -- start 2023-10-30T14:18:45.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f036b66b640 1 -- start start 2023-10-30T14:18:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f036b66b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f036406e470 0x7f0364144900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f036b66b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03640c8490 0x7f03640cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f036b66b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0364153d70 0x7f03641388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f036b66b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036415a880 con 0x7f0364153d70 2023-10-30T14:18:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f036b66b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03640cb0f0 con 0x7f03640c8490 2023-10-30T14:18:45.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f036b66b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0364144f90 con 0x7f036406e470 2023-10-30T14:18:45.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0369c08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0364153d70 0x7f03641388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:45.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0369c08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0364153d70 0x7f03641388e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38846/0 (socket says 172.21.15.46:38846) 2023-10-30T14:18:45.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0369c08640 1 -- 172.21.15.46:0/2014037485 learned_addr learned my addr 172.21.15.46:0/2014037485 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:45.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0369407640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f036406e470 0x7f0364144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:45.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0368c06640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03640c8490 0x7f03640cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:45.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0369c08640 1 -- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f036406e470 msgr2=0x7f0364144900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:45.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0369c08640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f036406e470 0x7f0364144900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0369c08640 1 -- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03640c8490 msgr2=0x7f03640cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:45.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0369c08640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03640c8490 0x7f03640cb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0369c08640 1 -- 172.21.15.46:0/2014037485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035c009a70 con 0x7f0364153d70 2023-10-30T14:18:45.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0368c06640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03640c8490 0x7f03640cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:45.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0369c08640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0364153d70 0x7f03641388e0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f035c0194f0 tx=0x7f035c019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:45.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0359ffb640 1 -- 172.21.15.46:0/2014037485 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c0add70 con 0x7f0364153d70 2023-10-30T14:18:45.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0359ffb640 1 -- 172.21.15.46:0/2014037485 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035c009450 con 0x7f0364153d70 2023-10-30T14:18:45.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f0359ffb640 1 -- 172.21.15.46:0/2014037485 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c0b9690 con 0x7f0364153d70 2023-10-30T14:18:45.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f036b66b640 1 -- 172.21.15.46:0/2014037485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0364143380 con 0x7f0364153d70 2023-10-30T14:18:45.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f036b66b640 1 -- 172.21.15.46:0/2014037485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0364142dc0 con 0x7f0364153d70 2023-10-30T14:18:45.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.751+0000 7f036b66b640 1 -- 172.21.15.46:0/2014037485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03640c71b0 con 0x7f0364153d70 2023-10-30T14:18:45.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.755+0000 7f0359ffb640 1 -- 172.21.15.46:0/2014037485 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f035c0b97f0 con 0x7f0364153d70 2023-10-30T14:18:45.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.755+0000 7f0359ffb640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03480664a0 0x7f0348068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:45.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.755+0000 7f0359ffb640 1 -- 172.21.15.46:0/2014037485 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f035c1203c0 con 0x7f0364153d70 2023-10-30T14:18:45.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.755+0000 7f0369407640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03480664a0 0x7f0348068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:45.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.755+0000 7f0369407640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03480664a0 0x7f0348068960 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0360009a50 tx=0x7f0360008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:45.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.759+0000 7f0359ffb640 1 -- 172.21.15.46:0/2014037485 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f035c0ece60 con 0x7f0364153d70 2023-10-30T14:18:45.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.951+0000 7f036b66b640 1 -- 172.21.15.46:0/2014037485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f036404e170 con 0x7f0364153d70 2023-10-30T14:18:45.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f0359ffb640 1 -- 172.21.15.46:0/2014037485 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f035c0f0480 con 0x7f0364153d70 2023-10-30T14:18:45.956 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:45.956 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:18:44.741668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:45.957 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 232 2023-10-30T14:18:45.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f036b66b640 1 -- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03480664a0 msgr2=0x7f0348068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:45.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f036b66b640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03480664a0 0x7f0348068960 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0360009a50 tx=0x7f0360008040 comp rx=0 tx=0).stop 2023-10-30T14:18:45.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f036b66b640 1 -- 172.21.15.46:0/2014037485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0364153d70 msgr2=0x7f03641388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:45.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f036b66b640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0364153d70 0x7f03641388e0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f035c0194f0 tx=0x7f035c019680 comp rx=0 tx=0).stop 2023-10-30T14:18:45.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f036b66b640 1 -- 172.21.15.46:0/2014037485 shutdown_connections 2023-10-30T14:18:45.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f036b66b640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03480664a0 0x7f0348068960 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f036b66b640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0364153d70 0x7f03641388e0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f036b66b640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03640c8490 0x7f03640cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f036b66b640 1 --2- 172.21.15.46:0/2014037485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f036406e470 0x7f0364144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:45.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f036b66b640 1 -- 172.21.15.46:0/2014037485 >> 172.21.15.46:0/2014037485 conn(0x7f0364071ea0 msgr2=0x7f0364151d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:45.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f036b66b640 1 -- 172.21.15.46:0/2014037485 shutdown_connections 2023-10-30T14:18:45.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:45.955+0000 7f036b66b640 1 -- 172.21.15.46:0/2014037485 wait complete. 2023-10-30T14:18:45.978 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 232, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 232, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9276": { "addr": "172.21.15.46:6836/2239735286", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 2239735286, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 2239735286, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9276, "incarnation": 225, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 20 }, "gid_9426": { "addr": "172.21.15.46:6833/3764280378", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 3764280378, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 3764280378, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9426, "incarnation": 232, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 86, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:18:44.741668+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9426, "mds_1": 9276 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:18:47.980 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.131+0000 7f4cce2dc640 1 Processor -- start 2023-10-30T14:18:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4cce2dc640 1 -- start start 2023-10-30T14:18:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4cce2dc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cc815cd50 0x7f4cc80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4cce2dc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cc80cefa0 0x7f4cc80cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4cce2dc640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc815a6c0 con 0x7f4cc81538f0 2023-10-30T14:18:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4cce2dc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc815d300 con 0x7f4cc80cefa0 2023-10-30T14:18:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4cce2dc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc815d5e0 con 0x7f4cc815cd50 2023-10-30T14:18:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4ccdadb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cc80cefa0 0x7f4cc80cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4ccdadb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cc80cefa0 0x7f4cc80cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37518/0 (socket says 172.21.15.46:37518) 2023-10-30T14:18:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4ccdadb640 1 -- 172.21.15.46:0/2951068366 learned_addr learned my addr 172.21.15.46:0/2951068366 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4cccad9640 1 --2- 172.21.15.46:0/2951068366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cc815cd50 0x7f4cc80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4ccdadb640 1 -- 172.21.15.46:0/2951068366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cc815cd50 msgr2=0x7f4cc80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4ccdadb640 1 --2- 172.21.15.46:0/2951068366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cc815cd50 0x7f4cc80c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4ccdadb640 1 -- 172.21.15.46:0/2951068366 >> v1:172.21.15.154:6789/0 conn(0x7f4cc81538f0 legacy=0x7f4cc8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:48.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.135+0000 7f4ccdadb640 1 -- 172.21.15.46:0/2951068366 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc815d8c0 con 0x7f4cc80cefa0 2023-10-30T14:18:48.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4ccdadb640 1 --2- 172.21.15.46:0/2951068366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cc80cefa0 0x7f4cc80cf3f0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009e50 tx=0x7f4cc001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=677dfdb9e343382b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:48.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4cbffff640 1 -- 172.21.15.46:0/2951068366 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc00af400 con 0x7f4cc80cefa0 2023-10-30T14:18:48.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4cbffff640 1 -- 172.21.15.46:0/2951068366 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc00ab050 con 0x7f4cc80cefa0 2023-10-30T14:18:48.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4cbffff640 1 -- 172.21.15.46:0/2951068366 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc00af780 con 0x7f4cc80cefa0 2023-10-30T14:18:48.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4cce2dc640 1 -- 172.21.15.46:0/2951068366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cc80cefa0 msgr2=0x7f4cc80cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:48.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4cce2dc640 1 --2- 172.21.15.46:0/2951068366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cc80cefa0 0x7f4cc80cf3f0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009e50 tx=0x7f4cc001f1f0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4cce2dc640 1 -- 172.21.15.46:0/2951068366 shutdown_connections 2023-10-30T14:18:48.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4cce2dc640 1 --2- 172.21.15.46:0/2951068366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cc80cefa0 0x7f4cc80cf3f0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4cce2dc640 1 --2- 172.21.15.46:0/2951068366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cc815cd50 0x7f4cc80c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4cce2dc640 1 -- 172.21.15.46:0/2951068366 >> 172.21.15.46:0/2951068366 conn(0x7f4cc806a640 msgr2=0x7f4cc8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:48.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4cce2dc640 1 -- 172.21.15.46:0/2951068366 shutdown_connections 2023-10-30T14:18:48.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4cce2dc640 1 -- 172.21.15.46:0/2951068366 wait complete. 2023-10-30T14:18:48.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.139+0000 7f4cce2dc640 1 Processor -- start 2023-10-30T14:18:48.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cce2dc640 1 -- start start 2023-10-30T14:18:48.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cce2dc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cc80cefa0 0x7f4cc81f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:48.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cce2dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cc81538f0 0x7f4cc8158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:48.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cce2dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cc815cd50 0x7f4cc80b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:48.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cce2dc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc815c320 con 0x7f4cc81538f0 2023-10-30T14:18:48.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cce2dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc80c6480 con 0x7f4cc815cd50 2023-10-30T14:18:48.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cce2dc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc81582d0 con 0x7f4cc80cefa0 2023-10-30T14:18:48.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cccad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cc81538f0 0x7f4cc8158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:48.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4ccd2da640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cc80cefa0 0x7f4cc81f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:48.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cccad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cc81538f0 0x7f4cc8158760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37526/0 (socket says 172.21.15.46:37526) 2023-10-30T14:18:48.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cccad9640 1 -- 172.21.15.46:0/93052935 learned_addr learned my addr 172.21.15.46:0/93052935 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:48.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4ccdadb640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cc815cd50 0x7f4cc80b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:48.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cccad9640 1 -- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cc80cefa0 msgr2=0x7f4cc81f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:48.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cccad9640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cc80cefa0 0x7f4cc81f2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cccad9640 1 -- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cc815cd50 msgr2=0x7f4cc80b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:48.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cccad9640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cc815cd50 0x7f4cc80b3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cccad9640 1 -- 172.21.15.46:0/93052935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc0009b30 con 0x7f4cc81538f0 2023-10-30T14:18:48.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4ccdadb640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cc815cd50 0x7f4cc80b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:48.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cccad9640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cc81538f0 0x7f4cc8158760 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800d5f0 tx=0x7f4cb800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:48.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cbdffb640 1 -- 172.21.15.46:0/93052935 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80125f0 con 0x7f4cc81538f0 2023-10-30T14:18:48.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cbdffb640 1 -- 172.21.15.46:0/93052935 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb8012750 con 0x7f4cc81538f0 2023-10-30T14:18:48.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cbdffb640 1 -- 172.21.15.46:0/93052935 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8012ae0 con 0x7f4cc81538f0 2023-10-30T14:18:48.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cce2dc640 1 -- 172.21.15.46:0/93052935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc81f3260 con 0x7f4cc81538f0 2023-10-30T14:18:48.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cce2dc640 1 -- 172.21.15.46:0/93052935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc8158ca0 con 0x7f4cc81538f0 2023-10-30T14:18:48.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.143+0000 7f4cce2dc640 1 -- 172.21.15.46:0/93052935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc804e170 con 0x7f4cc81538f0 2023-10-30T14:18:48.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.147+0000 7f4cbdffb640 1 -- 172.21.15.46:0/93052935 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4cb8012c40 con 0x7f4cc81538f0 2023-10-30T14:18:48.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.147+0000 7f4cbdffb640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4cac0664a0 0x7f4cac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:48.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.147+0000 7f4cbdffb640 1 -- 172.21.15.46:0/93052935 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f4cb8084d20 con 0x7f4cc81538f0 2023-10-30T14:18:48.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.147+0000 7f4ccd2da640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4cac0664a0 0x7f4cac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:48.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.147+0000 7f4ccd2da640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4cac0664a0 0x7f4cac068960 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4007fd0 tx=0x7f4cc4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:48.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.151+0000 7f4cbdffb640 1 -- 172.21.15.46:0/93052935 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4cb80516a0 con 0x7f4cc81538f0 2023-10-30T14:18:48.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.339+0000 7f4cce2dc640 1 -- 172.21.15.46:0/93052935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cc80b2290 con 0x7f4cc81538f0 2023-10-30T14:18:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cbdffb640 1 -- 172.21.15.46:0/93052935 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f4cb8054cc0 con 0x7f4cc81538f0 2023-10-30T14:18:48.344 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:48.344 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:18:44.741668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 232 2023-10-30T14:18:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cce2dc640 1 -- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4cac0664a0 msgr2=0x7f4cac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cce2dc640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4cac0664a0 0x7f4cac068960 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4007fd0 tx=0x7f4cc4009040 comp rx=0 tx=0).stop 2023-10-30T14:18:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cce2dc640 1 -- 172.21.15.46:0/93052935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cc81538f0 msgr2=0x7f4cc8158760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cce2dc640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cc81538f0 0x7f4cc8158760 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800d5f0 tx=0x7f4cb800dab0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cce2dc640 1 -- 172.21.15.46:0/93052935 shutdown_connections 2023-10-30T14:18:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cce2dc640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4cac0664a0 0x7f4cac068960 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cce2dc640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cc815cd50 0x7f4cc80b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cce2dc640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cc81538f0 0x7f4cc8158760 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cce2dc640 1 --2- 172.21.15.46:0/93052935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cc80cefa0 0x7f4cc81f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cce2dc640 1 -- 172.21.15.46:0/93052935 >> 172.21.15.46:0/93052935 conn(0x7f4cc806a640 msgr2=0x7f4cc80cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cce2dc640 1 -- 172.21.15.46:0/93052935 shutdown_connections 2023-10-30T14:18:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:48.343+0000 7f4cce2dc640 1 -- 172.21.15.46:0/93052935 wait complete. 2023-10-30T14:18:48.366 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:18:50.366 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:50.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.519+0000 7f6b8906b640 1 Processor -- start 2023-10-30T14:18:50.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.519+0000 7f6b8906b640 1 -- start start 2023-10-30T14:18:50.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.519+0000 7f6b8906b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b841538a0 0x7f6b8415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:50.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.519+0000 7f6b8906b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b840ca0e0 0x7f6b8415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:50.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.519+0000 7f6b8906b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b840cb800 con 0x7f6b840cf160 2023-10-30T14:18:50.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.519+0000 7f6b8906b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b8415dc30 con 0x7f6b840ca0e0 2023-10-30T14:18:50.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.519+0000 7f6b8906b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840cb330 con 0x7f6b841538a0 2023-10-30T14:18:50.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.519+0000 7f6b8259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b840ca0e0 0x7f6b8415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:50.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.519+0000 7f6b8259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b840ca0e0 0x7f6b8415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37540/0 (socket says 172.21.15.46:37540) 2023-10-30T14:18:50.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.519+0000 7f6b8259c640 1 -- 172.21.15.46:0/1315290474 learned_addr learned my addr 172.21.15.46:0/1315290474 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:50.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.519+0000 7f6b82d9d640 1 --2- 172.21.15.46:0/1315290474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b841538a0 0x7f6b8415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:50.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.523+0000 7f6b8259c640 1 -- 172.21.15.46:0/1315290474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b841538a0 msgr2=0x7f6b8415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:50.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.523+0000 7f6b8259c640 1 --2- 172.21.15.46:0/1315290474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b841538a0 0x7f6b8415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:50.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.523+0000 7f6b8259c640 1 -- 172.21.15.46:0/1315290474 >> v1:172.21.15.154:6789/0 conn(0x7f6b840cf160 legacy=0x7f6b840cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:50.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.523+0000 7f6b8259c640 1 -- 172.21.15.46:0/1315290474 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b840cad70 con 0x7f6b840ca0e0 2023-10-30T14:18:50.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.523+0000 7f6b8259c640 1 --2- 172.21.15.46:0/1315290474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca0e0 0x7f6b8415a090 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009870 tx=0x7f6b7801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8573f2d4eaf53a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b81d9b640 1 -- 172.21.15.46:0/1315290474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b780a8a30 con 0x7f6b840ca0e0 2023-10-30T14:18:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b81d9b640 1 -- 172.21.15.46:0/1315290474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b780a8b90 con 0x7f6b840ca0e0 2023-10-30T14:18:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b81d9b640 1 -- 172.21.15.46:0/1315290474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b780b0440 con 0x7f6b840ca0e0 2023-10-30T14:18:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b8906b640 1 -- 172.21.15.46:0/1315290474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca0e0 msgr2=0x7f6b8415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b8906b640 1 --2- 172.21.15.46:0/1315290474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca0e0 0x7f6b8415a090 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009870 tx=0x7f6b7801ed30 comp rx=0 tx=0).stop 2023-10-30T14:18:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b8906b640 1 -- 172.21.15.46:0/1315290474 shutdown_connections 2023-10-30T14:18:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b8906b640 1 --2- 172.21.15.46:0/1315290474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca0e0 0x7f6b8415a090 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b8906b640 1 --2- 172.21.15.46:0/1315290474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b841538a0 0x7f6b8415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b8906b640 1 -- 172.21.15.46:0/1315290474 >> 172.21.15.46:0/1315290474 conn(0x7f6b8406bfe0 msgr2=0x7f6b8406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b8906b640 1 -- 172.21.15.46:0/1315290474 shutdown_connections 2023-10-30T14:18:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b8906b640 1 -- 172.21.15.46:0/1315290474 wait complete. 2023-10-30T14:18:50.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b8906b640 1 Processor -- start 2023-10-30T14:18:50.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.527+0000 7f6b8906b640 1 -- start start 2023-10-30T14:18:50.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b8906b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca0e0 0x7f6b840ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:50.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b8906b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b840cf160 0x7f6b841442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:50.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b8906b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b841538a0 0x7f6b84144830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:50.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b8906b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b841e2fd0 con 0x7f6b840ca0e0 2023-10-30T14:18:50.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b8906b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840d0010 con 0x7f6b841538a0 2023-10-30T14:18:50.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b8906b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b84145ce0 con 0x7f6b840cf160 2023-10-30T14:18:50.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b82d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca0e0 0x7f6b840ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:50.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b82d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca0e0 0x7f6b840ce450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37546/0 (socket says 172.21.15.46:37546) 2023-10-30T14:18:50.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b82d9d640 1 -- 172.21.15.46:0/1307390980 learned_addr learned my addr 172.21.15.46:0/1307390980 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:50.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b8259c640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b840cf160 0x7f6b841442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:50.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b8359e640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b841538a0 0x7f6b84144830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:50.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b82d9d640 1 -- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b840cf160 msgr2=0x7f6b841442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:50.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b82d9d640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b840cf160 0x7f6b841442f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:50.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b82d9d640 1 -- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b841538a0 msgr2=0x7f6b84144830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:50.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b82d9d640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b841538a0 0x7f6b84144830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:50.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b82d9d640 1 -- 172.21.15.46:0/1307390980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b780094d0 con 0x7f6b840ca0e0 2023-10-30T14:18:50.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b82d9d640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca0e0 0x7f6b840ce450 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400e8a0 tx=0x7f6b7400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:50.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b6b7fe640 1 -- 172.21.15.46:0/1307390980 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b74009530 con 0x7f6b840ca0e0 2023-10-30T14:18:50.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b6b7fe640 1 -- 172.21.15.46:0/1307390980 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7400c940 con 0x7f6b840ca0e0 2023-10-30T14:18:50.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b6b7fe640 1 -- 172.21.15.46:0/1307390980 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7400ccd0 con 0x7f6b840ca0e0 2023-10-30T14:18:50.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b8906b640 1 -- 172.21.15.46:0/1307390980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b84145fc0 con 0x7f6b840ca0e0 2023-10-30T14:18:50.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b8906b640 1 -- 172.21.15.46:0/1307390980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b84143af0 con 0x7f6b840ca0e0 2023-10-30T14:18:50.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b8906b640 1 -- 172.21.15.46:0/1307390980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b840b2a50 con 0x7f6b840ca0e0 2023-10-30T14:18:50.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.531+0000 7f6b6b7fe640 1 -- 172.21.15.46:0/1307390980 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b74013470 con 0x7f6b840ca0e0 2023-10-30T14:18:50.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.535+0000 7f6b6b7fe640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b700664a0 0x7f6b70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:50.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.535+0000 7f6b6b7fe640 1 -- 172.21.15.46:0/1307390980 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f6b74084980 con 0x7f6b840ca0e0 2023-10-30T14:18:50.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.535+0000 7f6b8259c640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b700664a0 0x7f6b70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:50.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.535+0000 7f6b8259c640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b700664a0 0x7f6b70068960 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f6b780b2000 tx=0x7f6b78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:50.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.539+0000 7f6b6b7fe640 1 -- 172.21.15.46:0/1307390980 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6b74051440 con 0x7f6b840ca0e0 2023-10-30T14:18:50.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.735+0000 7f6b8906b640 1 -- 172.21.15.46:0/1307390980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b8404e170 con 0x7f6b840ca0e0 2023-10-30T14:18:50.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.735+0000 7f6b6b7fe640 1 -- 172.21.15.46:0/1307390980 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f6b74017090 con 0x7f6b840ca0e0 2023-10-30T14:18:50.740 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:50.740 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:18:44.741668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:50.740 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 232 2023-10-30T14:18:50.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.739+0000 7f6b8906b640 1 -- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b700664a0 msgr2=0x7f6b70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:50.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.739+0000 7f6b8906b640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b700664a0 0x7f6b70068960 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f6b780b2000 tx=0x7f6b78009040 comp rx=0 tx=0).stop 2023-10-30T14:18:50.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.739+0000 7f6b8906b640 1 -- 172.21.15.46:0/1307390980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca0e0 msgr2=0x7f6b840ce450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:50.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.739+0000 7f6b8906b640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca0e0 0x7f6b840ce450 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400e8a0 tx=0x7f6b7400ed60 comp rx=0 tx=0).stop 2023-10-30T14:18:50.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.739+0000 7f6b8906b640 1 -- 172.21.15.46:0/1307390980 shutdown_connections 2023-10-30T14:18:50.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.739+0000 7f6b8906b640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b700664a0 0x7f6b70068960 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.739+0000 7f6b8906b640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b841538a0 0x7f6b84144830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.739+0000 7f6b8906b640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b840cf160 0x7f6b841442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.739+0000 7f6b8906b640 1 --2- 172.21.15.46:0/1307390980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca0e0 0x7f6b840ce450 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.739+0000 7f6b8906b640 1 -- 172.21.15.46:0/1307390980 >> 172.21.15.46:0/1307390980 conn(0x7f6b8406bfe0 msgr2=0x7f6b841526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.739+0000 7f6b8906b640 1 -- 172.21.15.46:0/1307390980 shutdown_connections 2023-10-30T14:18:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:50.739+0000 7f6b8906b640 1 -- 172.21.15.46:0/1307390980 wait complete. 2023-10-30T14:18:50.763 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:18:52.764 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:52.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.915+0000 7f3594c46640 1 Processor -- start 2023-10-30T14:18:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.915+0000 7f3594c46640 1 -- start start 2023-10-30T14:18:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f3594c46640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f35900cf160 0x7f35900cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f3594c46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900cb800 con 0x7f35901538a0 2023-10-30T14:18:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f3594c46640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900cb330 con 0x7f359015d6f0 2023-10-30T14:18:52.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f3594c46640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35900cad70 con 0x7f35900cf160 2023-10-30T14:18:52.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f358e59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f35901538a0 0x7f35900ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60068/0 (socket says 172.21.15.46:60068) 2023-10-30T14:18:52.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f358e59c640 1 -- 172.21.15.46:0/2487167356 learned_addr learned my addr 172.21.15.46:0/2487167356 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:18:52.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f358ed9d640 1 --2- 172.21.15.46:0/2487167356 >> v2:172.21.15.154:3300/0 conn(0x7f35900cf160 0x7f35900cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:52.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f358d59a640 1 -- 172.21.15.46:0/2487167356 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959714726 0 0) 0x7f35900cb800 con 0x7f35901538a0 2023-10-30T14:18:52.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f358d59a640 1 -- 172.21.15.46:0/2487167356 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3578003530 con 0x7f35901538a0 2023-10-30T14:18:52.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f358d59a640 1 -- 172.21.15.46:0/2487167356 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037367376 0 0) 0x7f35900cb330 con 0x7f359015d6f0 2023-10-30T14:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f358d59a640 1 -- 172.21.15.46:0/2487167356 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35900cb800 con 0x7f359015d6f0 2023-10-30T14:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f358ed9d640 1 -- 172.21.15.46:0/2487167356 >> v1:172.21.15.154:6789/0 conn(0x7f359015d6f0 legacy=0x7f359015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f358ed9d640 1 -- 172.21.15.46:0/2487167356 >> v1:172.21.15.46:6789/0 conn(0x7f35901538a0 legacy=0x7f35900ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f358d59a640 1 -- 172.21.15.46:0/2487167356 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3150216094 0 0) 0x7f3578003530 con 0x7f35901538a0 2023-10-30T14:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.919+0000 7f358ed9d640 1 -- 172.21.15.46:0/2487167356 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35900cb050 con 0x7f35900cf160 2023-10-30T14:18:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.923+0000 7f358ed9d640 1 --2- 172.21.15.46:0/2487167356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35900cf160 0x7f35900cf590 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3588009d90 tx=0x7f358801f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a7afc6496b6079b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.923+0000 7f358d59a640 1 -- 172.21.15.46:0/2487167356 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580001680 con 0x7f35900cf160 2023-10-30T14:18:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.923+0000 7f358d59a640 1 -- 172.21.15.46:0/2487167356 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35880a8bf0 con 0x7f35900cf160 2023-10-30T14:18:52.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.923+0000 7f358d59a640 1 -- 172.21.15.46:0/2487167356 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35880b0440 con 0x7f35900cf160 2023-10-30T14:18:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.923+0000 7f3594c46640 1 -- 172.21.15.46:0/2487167356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35900cf160 msgr2=0x7f35900cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.923+0000 7f3594c46640 1 --2- 172.21.15.46:0/2487167356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35900cf160 0x7f35900cf590 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3588009d90 tx=0x7f358801f060 comp rx=0 tx=0).stop 2023-10-30T14:18:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.923+0000 7f3594c46640 1 -- 172.21.15.46:0/2487167356 shutdown_connections 2023-10-30T14:18:52.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.923+0000 7f3594c46640 1 --2- 172.21.15.46:0/2487167356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35900cf160 0x7f35900cf590 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.923+0000 7f3594c46640 1 -- 172.21.15.46:0/2487167356 >> 172.21.15.46:0/2487167356 conn(0x7f359006bfe0 msgr2=0x7f359006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.923+0000 7f3594c46640 1 -- 172.21.15.46:0/2487167356 shutdown_connections 2023-10-30T14:18:52.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.923+0000 7f3594c46640 1 -- 172.21.15.46:0/2487167356 wait complete. 2023-10-30T14:18:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f3594c46640 1 Processor -- start 2023-10-30T14:18:52.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f3594c46640 1 -- start start 2023-10-30T14:18:52.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f3594c46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35900cf160 0x7f359014fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f3594c46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35901538a0 0x7f35900b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f3594c46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f359015d6f0 0x7f35901f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f3594c46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3590150c00 con 0x7f35901538a0 2023-10-30T14:18:52.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f3594c46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359014e7c0 con 0x7f35900cf160 2023-10-30T14:18:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f3594c46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3590150470 con 0x7f359015d6f0 2023-10-30T14:18:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f358dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35901538a0 0x7f35900b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f358dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35901538a0 0x7f35900b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37552/0 (socket says 172.21.15.46:37552) 2023-10-30T14:18:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f358dd9b640 1 -- 172.21.15.46:0/2301437533 learned_addr learned my addr 172.21.15.46:0/2301437533 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:52.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f358e59c640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35900cf160 0x7f359014fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f358ed9d640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f359015d6f0 0x7f35901f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f358dd9b640 1 -- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f359015d6f0 msgr2=0x7f35901f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f358dd9b640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f359015d6f0 0x7f35901f29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f358dd9b640 1 -- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35900cf160 msgr2=0x7f359014fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:52.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f358dd9b640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35900cf160 0x7f359014fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f358dd9b640 1 -- 172.21.15.46:0/2301437533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3588009a70 con 0x7f35901538a0 2023-10-30T14:18:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f358ed9d640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f359015d6f0 0x7f35901f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:52.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f358dd9b640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35901538a0 0x7f35900b31a0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f358400c430 tx=0x7f358400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f357effd640 1 -- 172.21.15.46:0/2301437533 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35840052d0 con 0x7f35901538a0 2023-10-30T14:18:52.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f357effd640 1 -- 172.21.15.46:0/2301437533 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3584005450 con 0x7f35901538a0 2023-10-30T14:18:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f357effd640 1 -- 172.21.15.46:0/2301437533 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584003840 con 0x7f35901538a0 2023-10-30T14:18:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f3594c46640 1 -- 172.21.15.46:0/2301437533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3590150040 con 0x7f35901538a0 2023-10-30T14:18:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.927+0000 7f3594c46640 1 -- 172.21.15.46:0/2301437533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359014ea10 con 0x7f35901538a0 2023-10-30T14:18:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.931+0000 7f3594c46640 1 -- 172.21.15.46:0/2301437533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3590159840 con 0x7f35901538a0 2023-10-30T14:18:52.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.931+0000 7f357effd640 1 -- 172.21.15.46:0/2301437533 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f35840039a0 con 0x7f35901538a0 2023-10-30T14:18:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.931+0000 7f357effd640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35700664a0 0x7f3570068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.931+0000 7f357effd640 1 -- 172.21.15.46:0/2301437533 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f3584083ed0 con 0x7f35901538a0 2023-10-30T14:18:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.931+0000 7f358e59c640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35700664a0 0x7f3570068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:52.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.931+0000 7f358e59c640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35700664a0 0x7f3570068960 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3580001ec0 tx=0x7f3580001a60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:52.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:52.935+0000 7f357effd640 1 -- 172.21.15.46:0/2301437533 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3584050900 con 0x7f35901538a0 2023-10-30T14:18:53.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.135+0000 7f3594c46640 1 -- 172.21.15.46:0/2301437533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f359004e170 con 0x7f35901538a0 2023-10-30T14:18:53.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.135+0000 7f357effd640 1 -- 172.21.15.46:0/2301437533 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f3584017020 con 0x7f35901538a0 2023-10-30T14:18:53.138 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:53.139 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:18:44.741668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:53.139 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 232 2023-10-30T14:18:53.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.139+0000 7f3594c46640 1 -- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35700664a0 msgr2=0x7f3570068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:53.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.139+0000 7f3594c46640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35700664a0 0x7f3570068960 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3580001ec0 tx=0x7f3580001a60 comp rx=0 tx=0).stop 2023-10-30T14:18:53.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.139+0000 7f3594c46640 1 -- 172.21.15.46:0/2301437533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35901538a0 msgr2=0x7f35900b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:53.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.139+0000 7f3594c46640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35901538a0 0x7f35900b31a0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f358400c430 tx=0x7f358400c8f0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.139+0000 7f3594c46640 1 -- 172.21.15.46:0/2301437533 shutdown_connections 2023-10-30T14:18:53.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.139+0000 7f3594c46640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35700664a0 0x7f3570068960 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.139+0000 7f3594c46640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f359015d6f0 0x7f35901f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.139+0000 7f3594c46640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35901538a0 0x7f35900b31a0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.139+0000 7f3594c46640 1 --2- 172.21.15.46:0/2301437533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35900cf160 0x7f359014fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:53.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.139+0000 7f3594c46640 1 -- 172.21.15.46:0/2301437533 >> 172.21.15.46:0/2301437533 conn(0x7f359006bfe0 msgr2=0x7f35900c4870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:53.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.139+0000 7f3594c46640 1 -- 172.21.15.46:0/2301437533 shutdown_connections 2023-10-30T14:18:53.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:53.139+0000 7f3594c46640 1 -- 172.21.15.46:0/2301437533 wait complete. 2023-10-30T14:18:53.163 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:18:55.164 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:55.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9db2c9640 1 Processor -- start 2023-10-30T14:18:55.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9db2c9640 1 -- start start 2023-10-30T14:18:55.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9db2c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb9d41580e0 0x7fb9d4070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:55.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9db2c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d40c6540 0x7fb9d415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:55.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9db2c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d40c5810 con 0x7fb9d4069e90 2023-10-30T14:18:55.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9db2c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d40c6960 con 0x7fb9d41580e0 2023-10-30T14:18:55.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9db2c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d40c6bb0 con 0x7fb9d40c6540 2023-10-30T14:18:55.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9d9866640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb9d4069e90 0x7fb9d406a320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60076/0 (socket says 172.21.15.46:60076) 2023-10-30T14:18:55.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9d9866640 1 -- 172.21.15.46:0/90137115 learned_addr learned my addr 172.21.15.46:0/90137115 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:18:55.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9d9065640 1 --2- 172.21.15.46:0/90137115 >> v2:172.21.15.154:3300/0 conn(0x7fb9d41580e0 0x7fb9d4070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:55.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9d8864640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d40c6540 0x7fb9d415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:55.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508082118 0 0) 0x7fb9d40c5810 con 0x7fb9d4069e90 2023-10-30T14:18:55.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c0003530 con 0x7fb9d4069e90 2023-10-30T14:18:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3945832942 0 0) 0x7fb9c0003530 con 0x7fb9d4069e90 2023-10-30T14:18:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d40c6540 msgr2=0x7fb9d415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9cbfff640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d40c6540 0x7fb9d415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 >> v2:172.21.15.154:3300/0 conn(0x7fb9d41580e0 msgr2=0x7fb9d4070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9cbfff640 1 --2- 172.21.15.46:0/90137115 >> v2:172.21.15.154:3300/0 conn(0x7fb9d41580e0 0x7fb9d4070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d40c6e70 con 0x7fb9d4069e90 2023-10-30T14:18:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fb9cc001a00 con 0x7fb9d4069e90 2023-10-30T14:18:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 >> v1:172.21.15.46:6789/0 conn(0x7fb9d4069e90 legacy=0x7fb9d406a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9cbfff640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9c0003980 0x7fb9c0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.307+0000 7fb9cbfff640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00047d0 0x7fb9c0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9cbfff640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d40c6540 0x7fb9c0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d40c6960 con 0x7fb9c00047d0 2023-10-30T14:18:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d40c6bb0 con 0x7fb9c0003980 2023-10-30T14:18:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c000b990 con 0x7fb9d40c6540 2023-10-30T14:18:55.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9d8864640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00047d0 0x7fb9c0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:55.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9d9065640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9c0003980 0x7fb9c0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:55.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9d9866640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d40c6540 0x7fb9c0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:55.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9d8864640 1 -- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d40c6540 msgr2=0x7fb9c0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:55.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9d8864640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d40c6540 0x7fb9c0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9d8864640 1 -- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9c0003980 msgr2=0x7fb9c0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9d8864640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9c0003980 0x7fb9c0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9d8864640 1 -- 172.21.15.46:0/90137115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c0003530 con 0x7fb9c00047d0 2023-10-30T14:18:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9d9065640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9c0003980 0x7fb9c0003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9d9866640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d40c6540 0x7fb9c0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:18:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9d8864640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00047d0 0x7fb9c0004c50 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4009ae0 tx=0x7fb9c401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebbc684e8dda1622 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c40a5560 con 0x7fb9c00047d0 2023-10-30T14:18:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c40a56c0 con 0x7fb9c00047d0 2023-10-30T14:18:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.311+0000 7fb9cbfff640 1 -- 172.21.15.46:0/90137115 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c40a5a70 con 0x7fb9c00047d0 2023-10-30T14:18:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 -- 172.21.15.46:0/90137115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00047d0 msgr2=0x7fb9c0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 --2- 172.21.15.46:0/90137115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00047d0 0x7fb9c0004c50 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4009ae0 tx=0x7fb9c401ca50 comp rx=0 tx=0).stop 2023-10-30T14:18:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d9866640 1 -- 172.21.15.46:0/90137115 reap_dead start 2023-10-30T14:18:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 -- 172.21.15.46:0/90137115 shutdown_connections 2023-10-30T14:18:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 -- 172.21.15.46:0/90137115 >> 172.21.15.46:0/90137115 conn(0x7fb9d40c7910 msgr2=0x7fb9d40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 -- 172.21.15.46:0/90137115 shutdown_connections 2023-10-30T14:18:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 -- 172.21.15.46:0/90137115 wait complete. 2023-10-30T14:18:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 Processor -- start 2023-10-30T14:18:55.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 -- start start 2023-10-30T14:18:55.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00047d0 0x7fb9d4070200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:55.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9d41599f0 0x7fb9d4159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:55.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d41521f0 0x7fb9d4147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:55.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d4153eb0 con 0x7fb9c00047d0 2023-10-30T14:18:55.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d415a060 con 0x7fb9d41599f0 2023-10-30T14:18:55.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9db2c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d415a340 con 0x7fb9d41521f0 2023-10-30T14:18:55.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d9065640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00047d0 0x7fb9d4070200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:55.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d9065640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00047d0 0x7fb9d4070200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37582/0 (socket says 172.21.15.46:37582) 2023-10-30T14:18:55.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d9065640 1 -- 172.21.15.46:0/521476145 learned_addr learned my addr 172.21.15.46:0/521476145 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:55.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d8864640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9d41599f0 0x7fb9d4159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:55.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d9866640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d41521f0 0x7fb9d4147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:55.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d9065640 1 -- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d41521f0 msgr2=0x7fb9d4147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:55.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d9065640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d41521f0 0x7fb9d4147ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:55.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d9065640 1 -- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9d41599f0 msgr2=0x7fb9d4159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d9065640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9d41599f0 0x7fb9d4159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d9065640 1 -- 172.21.15.46:0/521476145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d4161060 con 0x7fb9c00047d0 2023-10-30T14:18:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d8864640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9d41599f0 0x7fb9d4159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.315+0000 7fb9d9065640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00047d0 0x7fb9d4070200 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00118d0 tx=0x7fb9d0011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.319+0000 7fb9c9ffb640 1 -- 172.21.15.46:0/521476145 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0004220 con 0x7fb9c00047d0 2023-10-30T14:18:55.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.319+0000 7fb9c9ffb640 1 -- 172.21.15.46:0/521476145 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d0004380 con 0x7fb9c00047d0 2023-10-30T14:18:55.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.319+0000 7fb9c9ffb640 1 -- 172.21.15.46:0/521476145 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d000fbd0 con 0x7fb9c00047d0 2023-10-30T14:18:55.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.319+0000 7fb9db2c9640 1 -- 172.21.15.46:0/521476145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d4161340 con 0x7fb9c00047d0 2023-10-30T14:18:55.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.319+0000 7fb9db2c9640 1 -- 172.21.15.46:0/521476145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d4158720 con 0x7fb9c00047d0 2023-10-30T14:18:55.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.319+0000 7fb9db2c9640 1 -- 172.21.15.46:0/521476145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d4160ec0 con 0x7fb9c00047d0 2023-10-30T14:18:55.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.319+0000 7fb9c9ffb640 1 -- 172.21.15.46:0/521476145 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9d000fd30 con 0x7fb9c00047d0 2023-10-30T14:18:55.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.323+0000 7fb9c9ffb640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9b80664a0 0x7fb9b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:55.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.323+0000 7fb9c9ffb640 1 -- 172.21.15.46:0/521476145 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7fb9d0084f40 con 0x7fb9c00047d0 2023-10-30T14:18:55.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.323+0000 7fb9d8864640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9b80664a0 0x7fb9b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:55.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.323+0000 7fb9d8864640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9b80664a0 0x7fb9b8068960 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40af000 tx=0x7fb9c40046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:55.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.323+0000 7fb9c9ffb640 1 -- 172.21.15.46:0/521476145 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9d4160ec0 con 0x7fb9c00047d0 2023-10-30T14:18:55.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.515+0000 7fb9db2c9640 1 -- 172.21.15.46:0/521476145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9d4160ec0 con 0x7fb9c00047d0 2023-10-30T14:18:55.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.515+0000 7fb9c9ffb640 1 -- 172.21.15.46:0/521476145 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fb9d4160ec0 con 0x7fb9c00047d0 2023-10-30T14:18:55.519 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:55.519 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:18:44.741668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:55.520 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 232 2023-10-30T14:18:55.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.519+0000 7fb9db2c9640 1 -- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9b80664a0 msgr2=0x7fb9b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:55.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.519+0000 7fb9db2c9640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9b80664a0 0x7fb9b8068960 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40af000 tx=0x7fb9c40046d0 comp rx=0 tx=0).stop 2023-10-30T14:18:55.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.519+0000 7fb9db2c9640 1 -- 172.21.15.46:0/521476145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00047d0 msgr2=0x7fb9d4070200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:55.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.519+0000 7fb9db2c9640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00047d0 0x7fb9d4070200 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00118d0 tx=0x7fb9d0011d90 comp rx=0 tx=0).stop 2023-10-30T14:18:55.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.519+0000 7fb9db2c9640 1 -- 172.21.15.46:0/521476145 shutdown_connections 2023-10-30T14:18:55.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.519+0000 7fb9db2c9640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9b80664a0 0x7fb9b8068960 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:55.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.519+0000 7fb9db2c9640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d41521f0 0x7fb9d4147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:55.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.519+0000 7fb9db2c9640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9d41599f0 0x7fb9d4159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:55.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.519+0000 7fb9db2c9640 1 --2- 172.21.15.46:0/521476145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9c00047d0 0x7fb9d4070200 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:55.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.519+0000 7fb9db2c9640 1 -- 172.21.15.46:0/521476145 >> 172.21.15.46:0/521476145 conn(0x7fb9d40c7910 msgr2=0x7fb9d40c0340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:55.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.519+0000 7fb9db2c9640 1 -- 172.21.15.46:0/521476145 shutdown_connections 2023-10-30T14:18:55.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:55.519+0000 7fb9db2c9640 1 -- 172.21.15.46:0/521476145 wait complete. 2023-10-30T14:18:55.540 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:18:57.541 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:18:57.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.691+0000 7f61261d3640 1 Processor -- start 2023-10-30T14:18:57.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.691+0000 7f61261d3640 1 -- start start 2023-10-30T14:18:57.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.695+0000 7f61261d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f61201538a0 0x7f612015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:57.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.695+0000 7f61261d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61200cb800 con 0x7f61200cf160 2023-10-30T14:18:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.695+0000 7f61261d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612015dc30 con 0x7f61200ca0e0 2023-10-30T14:18:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.695+0000 7f61261d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61200cb330 con 0x7f61201538a0 2023-10-30T14:18:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.695+0000 7f611ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f61200cf160 0x7f6120164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46562/0 (socket says 172.21.15.46:46562) 2023-10-30T14:18:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.695+0000 7f611ffff640 1 -- 172.21.15.46:0/3135572766 learned_addr learned my addr 172.21.15.46:0/3135572766 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:18:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.695+0000 7f611f7fe640 1 --2- 172.21.15.46:0/3135572766 >> v2:172.21.15.46:3300/0 conn(0x7f61201538a0 0x7f612015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.695+0000 7f611f7fe640 1 -- 172.21.15.46:0/3135572766 >> v1:172.21.15.154:6789/0 conn(0x7f61200ca0e0 legacy=0x7f612015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:18:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.695+0000 7f611f7fe640 1 -- 172.21.15.46:0/3135572766 >> v1:172.21.15.46:6789/0 conn(0x7f61200cf160 legacy=0x7f6120164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.695+0000 7f611f7fe640 1 -- 172.21.15.46:0/3135572766 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61200cad70 con 0x7f61201538a0 2023-10-30T14:18:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.695+0000 7f611e7fc640 1 -- 172.21.15.46:0/3135572766 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931729391 0 0) 0x7f61200cb800 con 0x7f61200cf160 2023-10-30T14:18:57.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f611f7fe640 1 --2- 172.21.15.46:0/3135572766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61201538a0 0x7f612015d6f0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f61100097c0 tx=0x7f611001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d58ad20dc3b54e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:57.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f611e7fc640 1 -- 172.21.15.46:0/3135572766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61100a8970 con 0x7f61201538a0 2023-10-30T14:18:57.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f611e7fc640 1 -- 172.21.15.46:0/3135572766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61100a8ad0 con 0x7f61201538a0 2023-10-30T14:18:57.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f611e7fc640 1 -- 172.21.15.46:0/3135572766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61100a8e60 con 0x7f61201538a0 2023-10-30T14:18:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f61261d3640 1 -- 172.21.15.46:0/3135572766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61201538a0 msgr2=0x7f612015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f61261d3640 1 --2- 172.21.15.46:0/3135572766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61201538a0 0x7f612015d6f0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f61100097c0 tx=0x7f611001ec70 comp rx=0 tx=0).stop 2023-10-30T14:18:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f61261d3640 1 -- 172.21.15.46:0/3135572766 shutdown_connections 2023-10-30T14:18:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f61261d3640 1 --2- 172.21.15.46:0/3135572766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61201538a0 0x7f612015d6f0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f61261d3640 1 -- 172.21.15.46:0/3135572766 >> 172.21.15.46:0/3135572766 conn(0x7f612006bfe0 msgr2=0x7f612006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f61261d3640 1 -- 172.21.15.46:0/3135572766 shutdown_connections 2023-10-30T14:18:57.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f61261d3640 1 -- 172.21.15.46:0/3135572766 wait complete. 2023-10-30T14:18:57.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f61261d3640 1 Processor -- start 2023-10-30T14:18:57.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.699+0000 7f61261d3640 1 -- start start 2023-10-30T14:18:57.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f61261d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61200ca0e0 0x7f6120158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:57.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f61261d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61200cf160 0x7f61200b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:57.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f61261d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61201538a0 0x7f61200b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:57.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f61261d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6120157da0 con 0x7f61200cf160 2023-10-30T14:18:57.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f61261d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61201578a0 con 0x7f61200ca0e0 2023-10-30T14:18:57.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f61261d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61200c45f0 con 0x7f61201538a0 2023-10-30T14:18:57.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f611effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61200cf160 0x7f61200b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:57.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f611effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61200cf160 0x7f61200b3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40524/0 (socket says 172.21.15.46:40524) 2023-10-30T14:18:57.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f611effd640 1 -- 172.21.15.46:0/2275800946 learned_addr learned my addr 172.21.15.46:0/2275800946 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:18:57.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f611f7fe640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61200ca0e0 0x7f6120158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:57.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f611ffff640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61201538a0 0x7f61200b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:57.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f611effd640 1 -- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61201538a0 msgr2=0x7f61200b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:57.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f611effd640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61201538a0 0x7f61200b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f611effd640 1 -- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61200ca0e0 msgr2=0x7f6120158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f611effd640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61200ca0e0 0x7f6120158ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f611effd640 1 -- 172.21.15.46:0/2275800946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6110009420 con 0x7f61200cf160 2023-10-30T14:18:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f611f7fe640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61200ca0e0 0x7f6120158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:18:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f611effd640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61200cf160 0x7f61200b3690 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f611400c3f0 tx=0x7f611400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f6107fff640 1 -- 172.21.15.46:0/2275800946 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61140052d0 con 0x7f61200cf160 2023-10-30T14:18:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f6107fff640 1 -- 172.21.15.46:0/2275800946 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6114005430 con 0x7f61200cf160 2023-10-30T14:18:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f6107fff640 1 -- 172.21.15.46:0/2275800946 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6114003840 con 0x7f61200cf160 2023-10-30T14:18:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f61261d3640 1 -- 172.21.15.46:0/2275800946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61200cee50 con 0x7f61200cf160 2023-10-30T14:18:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f61261d3640 1 -- 172.21.15.46:0/2275800946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61201575c0 con 0x7f61200cf160 2023-10-30T14:18:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.703+0000 7f61261d3640 1 -- 172.21.15.46:0/2275800946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612004e170 con 0x7f61200cf160 2023-10-30T14:18:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.707+0000 7f6107fff640 1 -- 172.21.15.46:0/2275800946 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f61140039a0 con 0x7f61200cf160 2023-10-30T14:18:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.707+0000 7f6107fff640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f610c0664d0 0x7f610c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:18:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.707+0000 7f6107fff640 1 -- 172.21.15.46:0/2275800946 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f6114083d40 con 0x7f61200cf160 2023-10-30T14:18:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.707+0000 7f611f7fe640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f610c0664d0 0x7f610c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:18:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.707+0000 7f611f7fe640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f610c0664d0 0x7f610c068990 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f61100b2000 tx=0x7f6110008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:18:57.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.711+0000 7f6107fff640 1 -- 172.21.15.46:0/2275800946 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6114050690 con 0x7f61200cf160 2023-10-30T14:18:57.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.903+0000 7f61261d3640 1 -- 172.21.15.46:0/2275800946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f612015dc30 con 0x7f61200cf160 2023-10-30T14:18:57.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.903+0000 7f6107fff640 1 -- 172.21.15.46:0/2275800946 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+5734 (secure 0 0 0) 0x7f612015dc30 con 0x7f61200cf160 2023-10-30T14:18:57.906 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:18:57.907 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":233,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:18:54.997384+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:resolve","state_seq":36,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:18:57.908 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 233 2023-10-30T14:18:57.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.907+0000 7f61261d3640 1 -- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f610c0664d0 msgr2=0x7f610c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.907+0000 7f61261d3640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f610c0664d0 0x7f610c068990 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f61100b2000 tx=0x7f6110008040 comp rx=0 tx=0).stop 2023-10-30T14:18:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.907+0000 7f61261d3640 1 -- 172.21.15.46:0/2275800946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61200cf160 msgr2=0x7f61200b3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:18:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.907+0000 7f61261d3640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61200cf160 0x7f61200b3690 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f611400c3f0 tx=0x7f611400c8b0 comp rx=0 tx=0).stop 2023-10-30T14:18:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.907+0000 7f61261d3640 1 -- 172.21.15.46:0/2275800946 shutdown_connections 2023-10-30T14:18:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.907+0000 7f61261d3640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f610c0664d0 0x7f610c068990 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.907+0000 7f61261d3640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61201538a0 0x7f61200b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.907+0000 7f61261d3640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61200cf160 0x7f61200b3690 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.907+0000 7f61261d3640 1 --2- 172.21.15.46:0/2275800946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61200ca0e0 0x7f6120158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:18:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.907+0000 7f61261d3640 1 -- 172.21.15.46:0/2275800946 >> 172.21.15.46:0/2275800946 conn(0x7f612006bfe0 msgr2=0x7f6120152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:18:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.907+0000 7f61261d3640 1 -- 172.21.15.46:0/2275800946 shutdown_connections 2023-10-30T14:18:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:18:57.907+0000 7f61261d3640 1 -- 172.21.15.46:0/2275800946 wait complete. 2023-10-30T14:18:57.928 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 233, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 233, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9276": { "addr": "172.21.15.46:6836/2239735286", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 2239735286, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 2239735286, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9276, "incarnation": 225, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 20 }, "gid_9426": { "addr": "172.21.15.46:6833/3764280378", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 3764280378, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 3764280378, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9426, "incarnation": 232, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:resolve", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 86, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:18:54.997384+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9426, "mds_1": 9276 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:18:59.924 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:18:59.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:18:59.929 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:00.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.123+0000 7f7068f1e640 1 Processor -- start 2023-10-30T14:19:00.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.123+0000 7f7068f1e640 1 -- start start 2023-10-30T14:19:00.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.127+0000 7f7068f1e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70640cefa0 0x7f70640cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:00.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.127+0000 7f7068f1e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706415a6c0 con 0x7f706415cd50 2023-10-30T14:19:00.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.127+0000 7f7068f1e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70640ca6a0 con 0x7f70641538f0 2023-10-30T14:19:00.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.127+0000 7f7068f1e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70640cf8f0 con 0x7f70640cefa0 2023-10-30T14:19:00.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.127+0000 7f7062ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f706415cd50 0x7f70640c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46564/0 (socket says 172.21.15.46:46564) 2023-10-30T14:19:00.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.127+0000 7f7062ffd640 1 -- 172.21.15.46:0/3761917232 learned_addr learned my addr 172.21.15.46:0/3761917232 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:19:00.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.127+0000 7f7063fff640 1 --2- 172.21.15.46:0/3761917232 >> v2:172.21.15.46:3300/0 conn(0x7f70640cefa0 0x7f70640cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:00.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.127+0000 7f7063fff640 1 -- 172.21.15.46:0/3761917232 >> v1:172.21.15.154:6789/0 conn(0x7f70641538f0 legacy=0x7f7064159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:00.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.127+0000 7f7063fff640 1 -- 172.21.15.46:0/3761917232 >> v1:172.21.15.46:6789/0 conn(0x7f706415cd50 legacy=0x7f70640c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:00.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.127+0000 7f7063fff640 1 -- 172.21.15.46:0/3761917232 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706415d300 con 0x7f70640cefa0 2023-10-30T14:19:00.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.127+0000 7f70627fc640 1 -- 172.21.15.46:0/3761917232 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204194602 0 0) 0x7f706415a6c0 con 0x7f706415cd50 2023-10-30T14:19:00.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f7063fff640 1 --2- 172.21.15.46:0/3761917232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640cefa0 0x7f70640cf3b0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f705c009d90 tx=0x7f705c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdfa2d536f85a4b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:00.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f70627fc640 1 -- 172.21.15.46:0/3761917232 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c0af600 con 0x7f70640cefa0 2023-10-30T14:19:00.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f70627fc640 1 -- 172.21.15.46:0/3761917232 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705c0ab050 con 0x7f70640cefa0 2023-10-30T14:19:00.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f70627fc640 1 -- 172.21.15.46:0/3761917232 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c0af980 con 0x7f70640cefa0 2023-10-30T14:19:00.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f7068f1e640 1 -- 172.21.15.46:0/3761917232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640cefa0 msgr2=0x7f70640cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:00.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f7068f1e640 1 --2- 172.21.15.46:0/3761917232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640cefa0 0x7f70640cf3b0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f705c009d90 tx=0x7f705c01f130 comp rx=0 tx=0).stop 2023-10-30T14:19:00.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f7068f1e640 1 -- 172.21.15.46:0/3761917232 shutdown_connections 2023-10-30T14:19:00.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f7068f1e640 1 --2- 172.21.15.46:0/3761917232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640cefa0 0x7f70640cf3b0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:00.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f7068f1e640 1 -- 172.21.15.46:0/3761917232 >> 172.21.15.46:0/3761917232 conn(0x7f706406a640 msgr2=0x7f7064065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:00.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f7068f1e640 1 -- 172.21.15.46:0/3761917232 shutdown_connections 2023-10-30T14:19:00.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f7068f1e640 1 -- 172.21.15.46:0/3761917232 wait complete. 2023-10-30T14:19:00.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f7068f1e640 1 Processor -- start 2023-10-30T14:19:00.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.131+0000 7f7068f1e640 1 -- start start 2023-10-30T14:19:00.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f7068f1e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640cefa0 0x7f70640b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:00.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f7068f1e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70641538f0 0x7f706415a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:00.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f7068f1e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f706415cd50 0x7f7064144900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:00.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f7068f1e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706415c5c0 con 0x7f70640cefa0 2023-10-30T14:19:00.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f7068f1e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7064152b60 con 0x7f70641538f0 2023-10-30T14:19:00.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f7068f1e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70640b4460 con 0x7f706415cd50 2023-10-30T14:19:00.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f70637fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640cefa0 0x7f70640b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:00.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f70637fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640cefa0 0x7f70640b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40546/0 (socket says 172.21.15.46:40546) 2023-10-30T14:19:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f70637fe640 1 -- 172.21.15.46:0/143285284 learned_addr learned my addr 172.21.15.46:0/143285284 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:19:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f7062ffd640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70641538f0 0x7f706415a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f7063fff640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f706415cd50 0x7f7064144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f70637fe640 1 -- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f706415cd50 msgr2=0x7f7064144900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f70637fe640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f706415cd50 0x7f7064144900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f70637fe640 1 -- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70641538f0 msgr2=0x7f706415a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f70637fe640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70641538f0 0x7f706415a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f70637fe640 1 -- 172.21.15.46:0/143285284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c009a70 con 0x7f70640cefa0 2023-10-30T14:19:00.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f70637fe640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640cefa0 0x7f70640b4f00 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f705400c500 tx=0x7f705400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:00.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f704bfff640 1 -- 172.21.15.46:0/143285284 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054005220 con 0x7f70640cefa0 2023-10-30T14:19:00.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f704bfff640 1 -- 172.21.15.46:0/143285284 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7054005380 con 0x7f70640cefa0 2023-10-30T14:19:00.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f704bfff640 1 -- 172.21.15.46:0/143285284 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054007270 con 0x7f70640cefa0 2023-10-30T14:19:00.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f7068f1e640 1 -- 172.21.15.46:0/143285284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70640b3240 con 0x7f70640cefa0 2023-10-30T14:19:00.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f7068f1e640 1 -- 172.21.15.46:0/143285284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70640b49e0 con 0x7f70640cefa0 2023-10-30T14:19:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f7068f1e640 1 -- 172.21.15.46:0/143285284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706404e170 con 0x7f70640cefa0 2023-10-30T14:19:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.135+0000 7f704bfff640 1 -- 172.21.15.46:0/143285284 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f70540073d0 con 0x7f70640cefa0 2023-10-30T14:19:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.139+0000 7f704bfff640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f70500664d0 0x7f7050068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.139+0000 7f704bfff640 1 -- 172.21.15.46:0/143285284 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f7054084210 con 0x7f70640cefa0 2023-10-30T14:19:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.139+0000 7f7062ffd640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f70500664d0 0x7f7050068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.139+0000 7f7062ffd640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f70500664d0 0x7f7050068990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f706406c850 tx=0x7f7058007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:00.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.143+0000 7f704bfff640 1 -- 172.21.15.46:0/143285284 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7054050ba0 con 0x7f70640cefa0 2023-10-30T14:19:00.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.331+0000 7f7068f1e640 1 -- 172.21.15.46:0/143285284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70640b2290 con 0x7f70640cefa0 2023-10-30T14:19:00.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.331+0000 7f704bfff640 1 -- 172.21.15.46:0/143285284 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+5734 (secure 0 0 0) 0x7f70540541c0 con 0x7f70640cefa0 2023-10-30T14:19:00.336 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:19:00.336 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":233,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:18:54.997384+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:resolve","state_seq":36,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:00.336 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 233 2023-10-30T14:19:00.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.335+0000 7f7068f1e640 1 -- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f70500664d0 msgr2=0x7f7050068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:00.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.335+0000 7f7068f1e640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f70500664d0 0x7f7050068990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f706406c850 tx=0x7f7058007d30 comp rx=0 tx=0).stop 2023-10-30T14:19:00.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.335+0000 7f7068f1e640 1 -- 172.21.15.46:0/143285284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640cefa0 msgr2=0x7f70640b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.335+0000 7f7068f1e640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640cefa0 0x7f70640b4f00 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f705400c500 tx=0x7f705400c9c0 comp rx=0 tx=0).stop 2023-10-30T14:19:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.335+0000 7f7068f1e640 1 -- 172.21.15.46:0/143285284 shutdown_connections 2023-10-30T14:19:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.335+0000 7f7068f1e640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f70500664d0 0x7f7050068990 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.335+0000 7f7068f1e640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f706415cd50 0x7f7064144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:00.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.335+0000 7f7068f1e640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70641538f0 0x7f706415a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.335+0000 7f7068f1e640 1 --2- 172.21.15.46:0/143285284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640cefa0 0x7f70640b4f00 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.335+0000 7f7068f1e640 1 -- 172.21.15.46:0/143285284 >> 172.21.15.46:0/143285284 conn(0x7f706406a640 msgr2=0x7f70640cd920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.335+0000 7f7068f1e640 1 -- 172.21.15.46:0/143285284 shutdown_connections 2023-10-30T14:19:00.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:00.335+0000 7f7068f1e640 1 -- 172.21.15.46:0/143285284 wait complete. 2023-10-30T14:19:00.357 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:19:02.358 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.503+0000 7f17adfa1640 1 Processor -- start 2023-10-30T14:19:02.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.503+0000 7f17adfa1640 1 -- start start 2023-10-30T14:19:02.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.503+0000 7f17adfa1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f17a8071110 0x7f17a806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:02.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.503+0000 7f17adfa1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a8066cf0 con 0x7f17a80c6bd0 2023-10-30T14:19:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.503+0000 7f17adfa1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a80c70d0 con 0x7f17a806f7a0 2023-10-30T14:19:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.503+0000 7f17adfa1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a80700c0 con 0x7f17a8071110 2023-10-30T14:19:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.503+0000 7f17a6ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f17a80c6bd0 0x7f17a8158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46570/0 (socket says 172.21.15.46:46570) 2023-10-30T14:19:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.503+0000 7f17a6ffd640 1 -- 172.21.15.46:0/4100751008 learned_addr learned my addr 172.21.15.46:0/4100751008 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:19:02.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.503+0000 7f17a7fff640 1 --2- 172.21.15.46:0/4100751008 >> v2:172.21.15.154:3300/0 conn(0x7f17a8071110 0x7f17a806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.503+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069999853 0 0) 0x7f17a8066cf0 con 0x7f17a80c6bd0 2023-10-30T14:19:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.503+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1790003530 con 0x7f17a80c6bd0 2023-10-30T14:19:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217881488 0 0) 0x7f17a80c70d0 con 0x7f17a806f7a0 2023-10-30T14:19:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a8066cf0 con 0x7f17a806f7a0 2023-10-30T14:19:02.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 521145308 0 0) 0x7f1790003530 con 0x7f17a80c6bd0 2023-10-30T14:19:02.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 >> v2:172.21.15.154:3300/0 conn(0x7f17a8071110 msgr2=0x7f17a806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:02.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 --2- 172.21.15.46:0/4100751008 >> v2:172.21.15.154:3300/0 conn(0x7f17a8071110 0x7f17a806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:02.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 >> v1:172.21.15.154:6789/0 conn(0x7f17a806f7a0 legacy=0x7f17a8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:02.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a80c7390 con 0x7f17a80c6bd0 2023-10-30T14:19:02.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f179c001680 con 0x7f17a80c6bd0 2023-10-30T14:19:02.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 >> v1:172.21.15.46:6789/0 conn(0x7f17a80c6bd0 legacy=0x7f17a8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:02.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 --2- 172.21.15.46:0/4100751008 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1790003b10 0x7f1790003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:02.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 --2- 172.21.15.46:0/4100751008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17900047d0 0x7f1790004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:02.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 --2- 172.21.15.46:0/4100751008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1790005190 0x7f179000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:02.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80700c0 con 0x7f1790005190 2023-10-30T14:19:02.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179000bf80 con 0x7f17900047d0 2023-10-30T14:19:02.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179000c140 con 0x7f1790003b10 2023-10-30T14:19:02.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a7fff640 1 --2- 172.21.15.46:0/4100751008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1790005190 0x7f179000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:02.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a77fe640 1 --2- 172.21.15.46:0/4100751008 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1790003b10 0x7f1790003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:02.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a6ffd640 1 --2- 172.21.15.46:0/4100751008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17900047d0 0x7f1790004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:02.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a7fff640 1 -- 172.21.15.46:0/4100751008 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1790003b10 msgr2=0x7f1790003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:02.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a7fff640 1 --2- 172.21.15.46:0/4100751008 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1790003b10 0x7f1790003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:02.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a7fff640 1 -- 172.21.15.46:0/4100751008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17900047d0 msgr2=0x7f1790004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:02.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a7fff640 1 --2- 172.21.15.46:0/4100751008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17900047d0 0x7f1790004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:02.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a7fff640 1 -- 172.21.15.46:0/4100751008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1790003530 con 0x7f1790005190 2023-10-30T14:19:02.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a6ffd640 1 --2- 172.21.15.46:0/4100751008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17900047d0 0x7f1790004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:19:02.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a7fff640 1 -- 172.21.15.46:0/4100751008 reap_dead start 2023-10-30T14:19:02.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.507+0000 7f17a7fff640 1 --2- 172.21.15.46:0/4100751008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1790005190 0x7f179000b990 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009f90 tx=0x7f17a001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6de04be58d1e34 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:02.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a00a5c70 con 0x7f1790005190 2023-10-30T14:19:02.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a000f040 con 0x7f1790005190 2023-10-30T14:19:02.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17a67fc640 1 -- 172.21.15.46:0/4100751008 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a00039a0 con 0x7f1790005190 2023-10-30T14:19:02.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 -- 172.21.15.46:0/4100751008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1790005190 msgr2=0x7f179000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:02.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 --2- 172.21.15.46:0/4100751008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1790005190 0x7f179000b990 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009f90 tx=0x7f17a001cc60 comp rx=0 tx=0).stop 2023-10-30T14:19:02.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 -- 172.21.15.46:0/4100751008 shutdown_connections 2023-10-30T14:19:02.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 --2- 172.21.15.46:0/4100751008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1790005190 0x7f179000b990 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:02.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 -- 172.21.15.46:0/4100751008 >> 172.21.15.46:0/4100751008 conn(0x7f17a80c7910 msgr2=0x7f17a80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:02.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 -- 172.21.15.46:0/4100751008 shutdown_connections 2023-10-30T14:19:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 -- 172.21.15.46:0/4100751008 wait complete. 2023-10-30T14:19:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 Processor -- start 2023-10-30T14:19:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 -- start start 2023-10-30T14:19:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1790005190 0x7f17a814ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17a8153bd0 0x7f17a814bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17a814c420 0x7f17a80c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a81488d0 con 0x7f17a8153bd0 2023-10-30T14:19:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a81edb30 con 0x7f17a814c420 2023-10-30T14:19:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.511+0000 7f17adfa1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a81eeb00 con 0x7f1790005190 2023-10-30T14:19:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17a6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17a8153bd0 0x7f17a814bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17a6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17a8153bd0 0x7f17a814bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40564/0 (socket says 172.21.15.46:40564) 2023-10-30T14:19:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17a6ffd640 1 -- 172.21.15.46:0/660066573 learned_addr learned my addr 172.21.15.46:0/660066573 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:19:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17a77fe640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1790005190 0x7f17a814ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17a7fff640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17a814c420 0x7f17a80c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17a6ffd640 1 -- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1790005190 msgr2=0x7f17a814ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17a6ffd640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1790005190 0x7f17a814ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17a6ffd640 1 -- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17a814c420 msgr2=0x7f17a80c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17a6ffd640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17a814c420 0x7f17a80c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:02.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17a6ffd640 1 -- 172.21.15.46:0/660066573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a81eb370 con 0x7f17a8153bd0 2023-10-30T14:19:02.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17a6ffd640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17a8153bd0 0x7f17a814bee0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f179c00e7b0 tx=0x7f179c00ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:02.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f178ffff640 1 -- 172.21.15.46:0/660066573 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c00c660 con 0x7f17a8153bd0 2023-10-30T14:19:02.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f178ffff640 1 -- 172.21.15.46:0/660066573 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179c00c7c0 con 0x7f17a8153bd0 2023-10-30T14:19:02.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17adfa1640 1 -- 172.21.15.46:0/660066573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a81edd40 con 0x7f17a8153bd0 2023-10-30T14:19:02.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f178ffff640 1 -- 172.21.15.46:0/660066573 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c00cb30 con 0x7f17a8153bd0 2023-10-30T14:19:02.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17adfa1640 1 -- 172.21.15.46:0/660066573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a81ee300 con 0x7f17a8153bd0 2023-10-30T14:19:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f17adfa1640 1 -- 172.21.15.46:0/660066573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a804e170 con 0x7f17a8153bd0 2023-10-30T14:19:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.515+0000 7f178ffff640 1 -- 172.21.15.46:0/660066573 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f179c00cc90 con 0x7f17a8153bd0 2023-10-30T14:19:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.519+0000 7f178ffff640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17940664d0 0x7f1794068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.519+0000 7f178ffff640 1 -- 172.21.15.46:0/660066573 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f179c0843b0 con 0x7f17a8153bd0 2023-10-30T14:19:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.519+0000 7f17a77fe640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17940664d0 0x7f1794068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:02.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.519+0000 7f17a77fe640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17940664d0 0x7f1794068990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1798007fd0 tx=0x7f1798009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:02.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.519+0000 7f178ffff640 1 -- 172.21.15.46:0/660066573 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f179c050ca0 con 0x7f17a8153bd0 2023-10-30T14:19:02.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 -- 172.21.15.46:0/660066573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17a80c3a00 con 0x7f17a8153bd0 2023-10-30T14:19:02.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f178ffff640 1 -- 172.21.15.46:0/660066573 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f179c0542c0 con 0x7f17a8153bd0 2023-10-30T14:19:02.713 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:19:02.713 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:00.812761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:02.713 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 234 2023-10-30T14:19:02.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 -- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17940664d0 msgr2=0x7f1794068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17940664d0 0x7f1794068990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1798007fd0 tx=0x7f1798009040 comp rx=0 tx=0).stop 2023-10-30T14:19:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 -- 172.21.15.46:0/660066573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17a8153bd0 msgr2=0x7f17a814bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17a8153bd0 0x7f17a814bee0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f179c00e7b0 tx=0x7f179c00ebb0 comp rx=0 tx=0).stop 2023-10-30T14:19:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 -- 172.21.15.46:0/660066573 shutdown_connections 2023-10-30T14:19:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f17940664d0 0x7f1794068990 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17a814c420 0x7f17a80c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17a8153bd0 0x7f17a814bee0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 --2- 172.21.15.46:0/660066573 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1790005190 0x7f17a814ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 -- 172.21.15.46:0/660066573 >> 172.21.15.46:0/660066573 conn(0x7f17a80c7910 msgr2=0x7f17a80c41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 -- 172.21.15.46:0/660066573 shutdown_connections 2023-10-30T14:19:02.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:02.711+0000 7f17adfa1640 1 -- 172.21.15.46:0/660066573 wait complete. 2023-10-30T14:19:02.734 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:19:04.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:04.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f505633e640 1 Processor -- start 2023-10-30T14:19:04.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f505633e640 1 -- start start 2023-10-30T14:19:04.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f505633e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5050153d70 0x7f505006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:04.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f505633e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f505006afa0 con 0x7f50500c8490 2023-10-30T14:19:04.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f505633e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5050154150 con 0x7f505006e470 2023-10-30T14:19:04.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f505633e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f505006d780 con 0x7f5050153d70 2023-10-30T14:19:04.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f50548db640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f50500c8490 0x7f50500c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46586/0 (socket says 172.21.15.46:46586) 2023-10-30T14:19:04.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f50548db640 1 -- 172.21.15.46:0/2138673142 learned_addr learned my addr 172.21.15.46:0/2138673142 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:19:04.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f504ffff640 1 --2- 172.21.15.46:0/2138673142 >> v2:172.21.15.154:3300/0 conn(0x7f5050153d70 0x7f505006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:04.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f504effd640 1 -- 172.21.15.46:0/2138673142 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532292136 0 0) 0x7f505006afa0 con 0x7f50500c8490 2023-10-30T14:19:04.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f504effd640 1 -- 172.21.15.46:0/2138673142 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5038003530 con 0x7f50500c8490 2023-10-30T14:19:04.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f504effd640 1 -- 172.21.15.46:0/2138673142 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4282500970 0 0) 0x7f5038003530 con 0x7f50500c8490 2023-10-30T14:19:04.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f504ffff640 1 -- 172.21.15.46:0/2138673142 >> v1:172.21.15.154:6789/0 conn(0x7f505006e470 legacy=0x7f50500cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:04.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f504ffff640 1 -- 172.21.15.46:0/2138673142 >> v1:172.21.15.46:6789/0 conn(0x7f50500c8490 legacy=0x7f50500c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:04.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.891+0000 7f504ffff640 1 -- 172.21.15.46:0/2138673142 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505006e030 con 0x7f5050153d70 2023-10-30T14:19:04.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.895+0000 7f504ffff640 1 --2- 172.21.15.46:0/2138673142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5050153d70 0x7f505006ca80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f50400097c0 tx=0x7f504001c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c66ec3d88a3fc21d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:04.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.895+0000 7f504effd640 1 -- 172.21.15.46:0/2138673142 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048001a00 con 0x7f5050153d70 2023-10-30T14:19:04.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.895+0000 7f504effd640 1 -- 172.21.15.46:0/2138673142 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50400a7050 con 0x7f5050153d70 2023-10-30T14:19:04.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.895+0000 7f504effd640 1 -- 172.21.15.46:0/2138673142 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50400ab620 con 0x7f5050153d70 2023-10-30T14:19:04.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.895+0000 7f505633e640 1 -- 172.21.15.46:0/2138673142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5050153d70 msgr2=0x7f505006ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:04.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.895+0000 7f505633e640 1 --2- 172.21.15.46:0/2138673142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5050153d70 0x7f505006ca80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f50400097c0 tx=0x7f504001c370 comp rx=0 tx=0).stop 2023-10-30T14:19:04.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.895+0000 7f505633e640 1 -- 172.21.15.46:0/2138673142 shutdown_connections 2023-10-30T14:19:04.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.895+0000 7f505633e640 1 --2- 172.21.15.46:0/2138673142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5050153d70 0x7f505006ca80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:04.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.895+0000 7f505633e640 1 -- 172.21.15.46:0/2138673142 >> 172.21.15.46:0/2138673142 conn(0x7f5050071ea0 msgr2=0x7f5050065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:04.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.895+0000 7f505633e640 1 -- 172.21.15.46:0/2138673142 shutdown_connections 2023-10-30T14:19:04.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.895+0000 7f505633e640 1 -- 172.21.15.46:0/2138673142 wait complete. 2023-10-30T14:19:04.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f505633e640 1 Processor -- start 2023-10-30T14:19:04.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f505633e640 1 -- start start 2023-10-30T14:19:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f505633e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f505006e470 0x7f505014f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f505633e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50500c8490 0x7f50501475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f505633e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050153d70 0x7f5050147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f505633e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505013f450 con 0x7f5050153d70 2023-10-30T14:19:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f505633e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50501470c0 con 0x7f505006e470 2023-10-30T14:19:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f505633e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50501f64c0 con 0x7f50500c8490 2023-10-30T14:19:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f50548db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050153d70 0x7f5050147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f50548db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050153d70 0x7f5050147ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40570/0 (socket says 172.21.15.46:40570) 2023-10-30T14:19:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f50548db640 1 -- 172.21.15.46:0/2093197039 learned_addr learned my addr 172.21.15.46:0/2093197039 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:19:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f504ffff640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f505006e470 0x7f505014f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f504f7fe640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50500c8490 0x7f50501475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f50548db640 1 -- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50500c8490 msgr2=0x7f50501475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f50548db640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50500c8490 0x7f50501475a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f50548db640 1 -- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f505006e470 msgr2=0x7f505014f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f50548db640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f505006e470 0x7f505014f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f50548db640 1 -- 172.21.15.46:0/2093197039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5040009420 con 0x7f5050153d70 2023-10-30T14:19:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f504ffff640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f505006e470 0x7f505014f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:19:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f50548db640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050153d70 0x7f5050147ae0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f50480018a0 tx=0x7f504800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f504cff9640 1 -- 172.21.15.46:0/2093197039 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048012400 con 0x7f5050153d70 2023-10-30T14:19:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f504cff9640 1 -- 172.21.15.46:0/2093197039 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5048012560 con 0x7f5050153d70 2023-10-30T14:19:04.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f505633e640 1 -- 172.21.15.46:0/2093197039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50501498e0 con 0x7f5050153d70 2023-10-30T14:19:04.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f504cff9640 1 -- 172.21.15.46:0/2093197039 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048012910 con 0x7f5050153d70 2023-10-30T14:19:04.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.899+0000 7f505633e640 1 -- 172.21.15.46:0/2093197039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50501490d0 con 0x7f5050153d70 2023-10-30T14:19:04.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.903+0000 7f505633e640 1 -- 172.21.15.46:0/2093197039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50501472c0 con 0x7f5050153d70 2023-10-30T14:19:04.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.903+0000 7f504cff9640 1 -- 172.21.15.46:0/2093197039 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5048012a70 con 0x7f5050153d70 2023-10-30T14:19:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.903+0000 7f504cff9640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f503c066470 0x7f503c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.903+0000 7f504cff9640 1 -- 172.21.15.46:0/2093197039 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f50480852c0 con 0x7f5050153d70 2023-10-30T14:19:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.903+0000 7f504ffff640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f503c066470 0x7f503c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.903+0000 7f504ffff640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f503c066470 0x7f503c068930 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f50400ad000 tx=0x7f50400a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:04.907+0000 7f504cff9640 1 -- 172.21.15.46:0/2093197039 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f50501472c0 con 0x7f5050153d70 2023-10-30T14:19:05.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 -- 172.21.15.46:0/2093197039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50501472c0 con 0x7f5050153d70 2023-10-30T14:19:05.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f504cff9640 1 -- 172.21.15.46:0/2093197039 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f50501472c0 con 0x7f5050153d70 2023-10-30T14:19:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:19:05.101 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:00.812761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:05.101 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 234 2023-10-30T14:19:05.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 -- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f503c066470 msgr2=0x7f503c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:05.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f503c066470 0x7f503c068930 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f50400ad000 tx=0x7f50400a6000 comp rx=0 tx=0).stop 2023-10-30T14:19:05.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 -- 172.21.15.46:0/2093197039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050153d70 msgr2=0x7f5050147ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:05.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050153d70 0x7f5050147ae0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f50480018a0 tx=0x7f504800a430 comp rx=0 tx=0).stop 2023-10-30T14:19:05.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 -- 172.21.15.46:0/2093197039 shutdown_connections 2023-10-30T14:19:05.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f503c066470 0x7f503c068930 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:05.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050153d70 0x7f5050147ae0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:05.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50500c8490 0x7f50501475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:05.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 --2- 172.21.15.46:0/2093197039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f505006e470 0x7f505014f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:05.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 -- 172.21.15.46:0/2093197039 >> 172.21.15.46:0/2093197039 conn(0x7f5050071ea0 msgr2=0x7f5050150220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:05.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 -- 172.21.15.46:0/2093197039 shutdown_connections 2023-10-30T14:19:05.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:05.099+0000 7f505633e640 1 -- 172.21.15.46:0/2093197039 wait complete. 2023-10-30T14:19:05.122 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:19:07.124 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:07.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f474c691640 1 Processor -- start 2023-10-30T14:19:07.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f474c691640 1 -- start start 2023-10-30T14:19:07.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f474c691640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f474406f7a0 0x7f474406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:07.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f474c691640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4744066cf0 con 0x7f47440c6bd0 2023-10-30T14:19:07.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f474c691640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47440c70d0 con 0x7f4744071110 2023-10-30T14:19:07.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f474c691640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47440700c0 con 0x7f474406f7a0 2023-10-30T14:19:07.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f474a42d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f474406f7a0 0x7f474406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:07.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f4749c2c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f47440c6bd0 0x7f4744069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55102/0 (socket says 172.21.15.46:55102) 2023-10-30T14:19:07.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f4749c2c640 1 -- 172.21.15.46:0/1581262694 learned_addr learned my addr 172.21.15.46:0/1581262694 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:19:07.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f474a42d640 1 -- 172.21.15.46:0/1581262694 >> v1:172.21.15.154:6789/0 conn(0x7f4744071110 legacy=0x7f4744158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:07.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f474a42d640 1 -- 172.21.15.46:0/1581262694 >> v1:172.21.15.46:6789/0 conn(0x7f47440c6bd0 legacy=0x7f4744069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:07.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f474a42d640 1 -- 172.21.15.46:0/1581262694 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47440c7390 con 0x7f474406f7a0 2023-10-30T14:19:07.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.271+0000 7f474942b640 1 -- 172.21.15.46:0/1581262694 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695656463 0 0) 0x7f4744066cf0 con 0x7f47440c6bd0 2023-10-30T14:19:07.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.275+0000 7f474a42d640 1 --2- 172.21.15.46:0/1581262694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f474406f7a0 0x7f474406fb80 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f47400097c0 tx=0x7f474001ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54fbf77980b3c65 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:07.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.275+0000 7f474942b640 1 -- 172.21.15.46:0/1581262694 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47400a8930 con 0x7f474406f7a0 2023-10-30T14:19:07.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.275+0000 7f474942b640 1 -- 172.21.15.46:0/1581262694 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47400a8a90 con 0x7f474406f7a0 2023-10-30T14:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.275+0000 7f474942b640 1 -- 172.21.15.46:0/1581262694 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47400a8e20 con 0x7f474406f7a0 2023-10-30T14:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.275+0000 7f474c691640 1 -- 172.21.15.46:0/1581262694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f474406f7a0 msgr2=0x7f474406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.275+0000 7f474c691640 1 --2- 172.21.15.46:0/1581262694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f474406f7a0 0x7f474406fb80 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f47400097c0 tx=0x7f474001ec30 comp rx=0 tx=0).stop 2023-10-30T14:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.275+0000 7f474c691640 1 -- 172.21.15.46:0/1581262694 shutdown_connections 2023-10-30T14:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.275+0000 7f474c691640 1 --2- 172.21.15.46:0/1581262694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f474406f7a0 0x7f474406fb80 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.275+0000 7f474c691640 1 -- 172.21.15.46:0/1581262694 >> 172.21.15.46:0/1581262694 conn(0x7f47440c7910 msgr2=0x7f47440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.275+0000 7f474c691640 1 -- 172.21.15.46:0/1581262694 shutdown_connections 2023-10-30T14:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.275+0000 7f474c691640 1 -- 172.21.15.46:0/1581262694 wait complete. 2023-10-30T14:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474c691640 1 Processor -- start 2023-10-30T14:19:07.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474c691640 1 -- start start 2023-10-30T14:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474c691640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f474406f7a0 0x7f4744156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474c691640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4744071110 0x7f474414bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474c691640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47440c6bd0 0x7f474414c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474c691640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474414b1c0 con 0x7f47440c6bd0 2023-10-30T14:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474c691640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4744146720 con 0x7f4744071110 2023-10-30T14:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474c691640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4744155d60 con 0x7f474406f7a0 2023-10-30T14:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474ac2e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47440c6bd0 0x7f474414c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474ac2e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47440c6bd0 0x7f474414c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45942/0 (socket says 172.21.15.46:45942) 2023-10-30T14:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474ac2e640 1 -- 172.21.15.46:0/1710216738 learned_addr learned my addr 172.21.15.46:0/1710216738 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f4749c2c640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4744071110 0x7f474414bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:07.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474a42d640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f474406f7a0 0x7f4744156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:07.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474ac2e640 1 -- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f474406f7a0 msgr2=0x7f4744156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:07.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474ac2e640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f474406f7a0 0x7f4744156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:07.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474ac2e640 1 -- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4744071110 msgr2=0x7f474414bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:07.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474ac2e640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4744071110 0x7f474414bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474ac2e640 1 -- 172.21.15.46:0/1710216738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4740009420 con 0x7f47440c6bd0 2023-10-30T14:19:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f4749c2c640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4744071110 0x7f474414bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:19:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f474ac2e640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47440c6bd0 0x7f474414c0b0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f473c00b970 tx=0x7f473c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f473affd640 1 -- 172.21.15.46:0/1710216738 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c00cb40 con 0x7f47440c6bd0 2023-10-30T14:19:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f473affd640 1 -- 172.21.15.46:0/1710216738 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c00cca0 con 0x7f47440c6bd0 2023-10-30T14:19:07.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.279+0000 7f473affd640 1 -- 172.21.15.46:0/1710216738 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c014680 con 0x7f47440c6bd0 2023-10-30T14:19:07.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.283+0000 7f474c691640 1 -- 172.21.15.46:0/1710216738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474414a540 con 0x7f47440c6bd0 2023-10-30T14:19:07.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.283+0000 7f474c691640 1 -- 172.21.15.46:0/1710216738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474414ab00 con 0x7f47440c6bd0 2023-10-30T14:19:07.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.283+0000 7f474c691640 1 -- 172.21.15.46:0/1710216738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474414b300 con 0x7f47440c6bd0 2023-10-30T14:19:07.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.283+0000 7f473affd640 1 -- 172.21.15.46:0/1710216738 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f473c01c360 con 0x7f47440c6bd0 2023-10-30T14:19:07.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.283+0000 7f473affd640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4728066520 0x7f47280689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.283+0000 7f473affd640 1 -- 172.21.15.46:0/1710216738 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f473c0847c0 con 0x7f47440c6bd0 2023-10-30T14:19:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.283+0000 7f474a42d640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4728066520 0x7f47280689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.287+0000 7f474a42d640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4728066520 0x7f47280689e0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f47400b2000 tx=0x7f47400aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.287+0000 7f473affd640 1 -- 172.21.15.46:0/1710216738 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f474414b300 con 0x7f47440c6bd0 2023-10-30T14:19:07.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.474+0000 7f474c691640 1 -- 172.21.15.46:0/1710216738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f474414b300 con 0x7f47440c6bd0 2023-10-30T14:19:07.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.474+0000 7f473affd640 1 -- 172.21.15.46:0/1710216738 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f474414b300 con 0x7f47440c6bd0 2023-10-30T14:19:07.480 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:19:07.480 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":235,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:06.771914+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:07.480 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 235 2023-10-30T14:19:07.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.478+0000 7f474c691640 1 -- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4728066520 msgr2=0x7f47280689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:07.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.478+0000 7f474c691640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4728066520 0x7f47280689e0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f47400b2000 tx=0x7f47400aa070 comp rx=0 tx=0).stop 2023-10-30T14:19:07.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.478+0000 7f474c691640 1 -- 172.21.15.46:0/1710216738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47440c6bd0 msgr2=0x7f474414c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:07.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.478+0000 7f474c691640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47440c6bd0 0x7f474414c0b0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f473c00b970 tx=0x7f473c00be30 comp rx=0 tx=0).stop 2023-10-30T14:19:07.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.478+0000 7f474c691640 1 -- 172.21.15.46:0/1710216738 shutdown_connections 2023-10-30T14:19:07.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.478+0000 7f474c691640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4728066520 0x7f47280689e0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.478+0000 7f474c691640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47440c6bd0 0x7f474414c0b0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.478+0000 7f474c691640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4744071110 0x7f474414bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.478+0000 7f474c691640 1 --2- 172.21.15.46:0/1710216738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f474406f7a0 0x7f4744156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.478+0000 7f474c691640 1 -- 172.21.15.46:0/1710216738 >> 172.21.15.46:0/1710216738 conn(0x7f47440c7910 msgr2=0x7f47440c92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:07.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.478+0000 7f474c691640 1 -- 172.21.15.46:0/1710216738 shutdown_connections 2023-10-30T14:19:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:07.478+0000 7f474c691640 1 -- 172.21.15.46:0/1710216738 wait complete. 2023-10-30T14:19:07.501 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:19:09.502 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:09.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.650+0000 7f9ea39aa640 1 Processor -- start 2023-10-30T14:19:09.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.650+0000 7f9ea39aa640 1 -- start start 2023-10-30T14:19:09.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea39aa640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9e9c0c1ee0 0x7f9e9c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:09.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea39aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e9c0c8260 0x7f9e9c0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:09.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea39aa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c06e090 con 0x7f9e9c151800 2023-10-30T14:19:09.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea39aa640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9c06dc70 con 0x7f9e9c0c1ee0 2023-10-30T14:19:09.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea39aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c0c8a50 con 0x7f9e9c0c8260 2023-10-30T14:19:09.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea1746640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9e9c151800 0x7f9e9c0c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55110/0 (socket says 172.21.15.46:55110) 2023-10-30T14:19:09.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea1746640 1 -- 172.21.15.46:0/2927148151 learned_addr learned my addr 172.21.15.46:0/2927148151 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:19:09.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea0f45640 1 --2- 172.21.15.46:0/2927148151 >> v2:172.21.15.154:3300/0 conn(0x7f9e9c0c1ee0 0x7f9e9c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:09.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea1f47640 1 --2- 172.21.15.46:0/2927148151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e9c0c8260 0x7f9e9c0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:09.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265733849 0 0) 0x7f9e9c06e090 con 0x7f9e9c151800 2023-10-30T14:19:09.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e88003530 con 0x7f9e9c151800 2023-10-30T14:19:09.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 842651872 0 0) 0x7f9e88003530 con 0x7f9e9c151800 2023-10-30T14:19:09.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e9c0c8260 msgr2=0x7f9e9c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:09.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 --2- 172.21.15.46:0/2927148151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e9c0c8260 0x7f9e9c0cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:09.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 >> v2:172.21.15.154:3300/0 conn(0x7f9e9c0c1ee0 msgr2=0x7f9e9c06e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:09.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 --2- 172.21.15.46:0/2927148151 >> v2:172.21.15.154:3300/0 conn(0x7f9e9c0c1ee0 0x7f9e9c06e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:09.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c0c3b10 con 0x7f9e9c151800 2023-10-30T14:19:09.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f9e980016a0 con 0x7f9e9c151800 2023-10-30T14:19:09.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 >> v1:172.21.15.46:6789/0 conn(0x7f9e9c151800 legacy=0x7f9e9c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:09.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 --2- 172.21.15.46:0/2927148151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e9c0c8260 0x7f9e88003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:09.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 --2- 172.21.15.46:0/2927148151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e88003f00 0x7f9e880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:09.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 --2- 172.21.15.46:0/2927148151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e88004d10 0x7f9e88005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:09.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c06dc70 con 0x7f9e88004d10 2023-10-30T14:19:09.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c0c8a50 con 0x7f9e88003f00 2023-10-30T14:19:09.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8800b990 con 0x7f9e9c0c8260 2023-10-30T14:19:09.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea1f47640 1 --2- 172.21.15.46:0/2927148151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e88004d10 0x7f9e88005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:09.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea0f45640 1 --2- 172.21.15.46:0/2927148151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e88003f00 0x7f9e880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea1746640 1 --2- 172.21.15.46:0/2927148151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e9c0c8260 0x7f9e88003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea1f47640 1 -- 172.21.15.46:0/2927148151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e9c0c8260 msgr2=0x7f9e88003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea1f47640 1 --2- 172.21.15.46:0/2927148151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e9c0c8260 0x7f9e88003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea1f47640 1 -- 172.21.15.46:0/2927148151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e88003f00 msgr2=0x7f9e880047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea1f47640 1 --2- 172.21.15.46:0/2927148151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e88003f00 0x7f9e880047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.654+0000 7f9ea1f47640 1 -- 172.21.15.46:0/2927148151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e88003530 con 0x7f9e88004d10 2023-10-30T14:19:09.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.658+0000 7f9ea1f47640 1 --2- 172.21.15.46:0/2927148151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e88004d10 0x7f9e88005190 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009fc0 tx=0x7f9e9401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c8f94c19abb401 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:09.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.658+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e940a5c70 con 0x7f9e88004d10 2023-10-30T14:19:09.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.658+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9400f040 con 0x7f9e88004d10 2023-10-30T14:19:09.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.658+0000 7f9e93fff640 1 -- 172.21.15.46:0/2927148151 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e940ad440 con 0x7f9e88004d10 2023-10-30T14:19:09.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.658+0000 7f9ea39aa640 1 -- 172.21.15.46:0/2927148151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e88004d10 msgr2=0x7f9e88005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:09.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.658+0000 7f9ea39aa640 1 --2- 172.21.15.46:0/2927148151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e88004d10 0x7f9e88005190 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009fc0 tx=0x7f9e9401cc00 comp rx=0 tx=0).stop 2023-10-30T14:19:09.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.658+0000 7f9ea1f47640 1 -- 172.21.15.46:0/2927148151 reap_dead start 2023-10-30T14:19:09.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.658+0000 7f9ea39aa640 1 -- 172.21.15.46:0/2927148151 shutdown_connections 2023-10-30T14:19:09.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.658+0000 7f9ea39aa640 1 -- 172.21.15.46:0/2927148151 >> 172.21.15.46:0/2927148151 conn(0x7f9e9c071ea0 msgr2=0x7f9e9c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:09.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.658+0000 7f9ea39aa640 1 -- 172.21.15.46:0/2927148151 shutdown_connections 2023-10-30T14:19:09.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.658+0000 7f9ea39aa640 1 -- 172.21.15.46:0/2927148151 wait complete. 2023-10-30T14:19:09.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea39aa640 1 Processor -- start 2023-10-30T14:19:09.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea39aa640 1 -- start start 2023-10-30T14:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea39aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e88004d10 0x7f9e9c149270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea39aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e9c1e3870 0x7f9e9c149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea39aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9c1e3cd0 0x7f9e9c06f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea39aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c14b290 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea39aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c1e2c40 con 0x7f9e9c1e3870 2023-10-30T14:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea39aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c1e4150 con 0x7f9e88004d10 2023-10-30T14:19:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea1f47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9c1e3cd0 0x7f9e9c06f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea1f47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9c1e3cd0 0x7f9e9c06f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45972/0 (socket says 172.21.15.46:45972) 2023-10-30T14:19:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea1f47640 1 -- 172.21.15.46:0/890928568 learned_addr learned my addr 172.21.15.46:0/890928568 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:19:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea1746640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e88004d10 0x7f9e9c149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea0f45640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e9c1e3870 0x7f9e9c149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea1f47640 1 -- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e88004d10 msgr2=0x7f9e9c149270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea1f47640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e88004d10 0x7f9e9c149270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea1f47640 1 -- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e9c1e3870 msgr2=0x7f9e9c149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea1f47640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e9c1e3870 0x7f9e9c149940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea1f47640 1 -- 172.21.15.46:0/890928568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c1e4430 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea1746640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e88004d10 0x7f9e9c149270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:19:09.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea1f47640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9c1e3cd0 0x7f9e9c06f0c0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f9e940b0040 tx=0x7f9e94009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:09.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9e91ffb640 1 -- 172.21.15.46:0/890928568 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e940bc070 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9e91ffb640 1 -- 172.21.15.46:0/890928568 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e940039a0 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9e91ffb640 1 -- 172.21.15.46:0/890928568 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94003b30 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea39aa640 1 -- 172.21.15.46:0/890928568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9c1f0670 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea39aa640 1 -- 172.21.15.46:0/890928568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9c1f0c30 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.662+0000 7f9ea39aa640 1 -- 172.21.15.46:0/890928568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9c04e170 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.666+0000 7f9e91ffb640 1 -- 172.21.15.46:0/890928568 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e940b4040 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.666+0000 7f9e91ffb640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9e800664a0 0x7f9e80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.666+0000 7f9e91ffb640 1 -- 172.21.15.46:0/890928568 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f9e9411ebc0 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.666+0000 7f9ea1746640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9e800664a0 0x7f9e80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.666+0000 7f9ea1746640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9e800664a0 0x7f9e80068960 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f9e98000c10 tx=0x7f9e98009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.670+0000 7f9e91ffb640 1 -- 172.21.15.46:0/890928568 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e940eb560 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.862+0000 7f9ea39aa640 1 -- 172.21.15.46:0/890928568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e9c0c6de0 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.862+0000 7f9e91ffb640 1 -- 172.21.15.46:0/890928568 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f9e940b2030 con 0x7f9e9c1e3cd0 2023-10-30T14:19:09.868 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:19:09.868 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":235,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:06.771914+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:09.868 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 235 2023-10-30T14:19:09.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.866+0000 7f9ea39aa640 1 -- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9e800664a0 msgr2=0x7f9e80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:09.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.866+0000 7f9ea39aa640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9e800664a0 0x7f9e80068960 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f9e98000c10 tx=0x7f9e98009040 comp rx=0 tx=0).stop 2023-10-30T14:19:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.866+0000 7f9ea39aa640 1 -- 172.21.15.46:0/890928568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9c1e3cd0 msgr2=0x7f9e9c06f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.866+0000 7f9ea39aa640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9c1e3cd0 0x7f9e9c06f0c0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f9e940b0040 tx=0x7f9e94009750 comp rx=0 tx=0).stop 2023-10-30T14:19:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.866+0000 7f9ea39aa640 1 -- 172.21.15.46:0/890928568 shutdown_connections 2023-10-30T14:19:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.866+0000 7f9ea39aa640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9e800664a0 0x7f9e80068960 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:09.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.866+0000 7f9ea39aa640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9c1e3cd0 0x7f9e9c06f0c0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:09.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.866+0000 7f9ea39aa640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e9c1e3870 0x7f9e9c149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:09.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.866+0000 7f9ea39aa640 1 --2- 172.21.15.46:0/890928568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e88004d10 0x7f9e9c149270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:09.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.866+0000 7f9ea39aa640 1 -- 172.21.15.46:0/890928568 >> 172.21.15.46:0/890928568 conn(0x7f9e9c071ea0 msgr2=0x7f9e9c0c4030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:09.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.866+0000 7f9ea39aa640 1 -- 172.21.15.46:0/890928568 shutdown_connections 2023-10-30T14:19:09.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:09.866+0000 7f9ea39aa640 1 -- 172.21.15.46:0/890928568 wait complete. 2023-10-30T14:19:09.889 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 235, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 235, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9276": { "addr": "172.21.15.46:6836/2239735286", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 2239735286, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 2239735286, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9276, "incarnation": 225, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 20 }, "gid_9426": { "addr": "172.21.15.46:6833/3764280378", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 3764280378, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 3764280378, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9426, "incarnation": 232, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 40 } }, "last_failure": 0, "last_failure_osd_epoch": 86, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:19:06.771914+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9426, "mds_1": 9276 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:19:11.890 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:12.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.042+0000 7f22ad365640 1 Processor -- start 2023-10-30T14:19:12.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.042+0000 7f22ad365640 1 -- start start 2023-10-30T14:19:12.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22ad365640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a80cf160 0x7f22a80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:12.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22ad365640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80cb800 con 0x7f22a815d6f0 2023-10-30T14:19:12.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22ad365640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80cb330 con 0x7f22a81538a0 2023-10-30T14:19:12.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22ad365640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a80cad70 con 0x7f22a80cf160 2023-10-30T14:19:12.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f229ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f22a815d6f0 0x7f22a815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55124/0 (socket says 172.21.15.46:55124) 2023-10-30T14:19:12.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f229ffff640 1 -- 172.21.15.46:0/1521353933 learned_addr learned my addr 172.21.15.46:0/1521353933 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:19:12.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22a77fe640 1 --2- 172.21.15.46:0/1521353933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a80cf160 0x7f22a80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:12.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22a67fc640 1 -- 172.21.15.46:0/1521353933 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539040000 0 0) 0x7f22a80cb800 con 0x7f22a815d6f0 2023-10-30T14:19:12.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22a67fc640 1 -- 172.21.15.46:0/1521353933 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2294003530 con 0x7f22a815d6f0 2023-10-30T14:19:12.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22a67fc640 1 -- 172.21.15.46:0/1521353933 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684790771 0 0) 0x7f22a80cb330 con 0x7f22a81538a0 2023-10-30T14:19:12.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22a67fc640 1 -- 172.21.15.46:0/1521353933 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a80cb800 con 0x7f22a81538a0 2023-10-30T14:19:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22a67fc640 1 -- 172.21.15.46:0/1521353933 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2412776262 0 0) 0x7f2294003530 con 0x7f22a815d6f0 2023-10-30T14:19:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22a77fe640 1 -- 172.21.15.46:0/1521353933 >> v1:172.21.15.154:6789/0 conn(0x7f22a81538a0 legacy=0x7f22a80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22a77fe640 1 -- 172.21.15.46:0/1521353933 >> v1:172.21.15.46:6789/0 conn(0x7f22a815d6f0 legacy=0x7f22a815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.046+0000 7f22a77fe640 1 -- 172.21.15.46:0/1521353933 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a80cb050 con 0x7f22a80cf160 2023-10-30T14:19:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.050+0000 7f22a77fe640 1 --2- 172.21.15.46:0/1521353933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a80cf160 0x7f22a80cf590 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22a0009dc0 tx=0x7f22a001f080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cab01730b2d05c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.050+0000 7f22a67fc640 1 -- 172.21.15.46:0/1521353933 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290001680 con 0x7f22a80cf160 2023-10-30T14:19:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.050+0000 7f22a67fc640 1 -- 172.21.15.46:0/1521353933 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a00a8c10 con 0x7f22a80cf160 2023-10-30T14:19:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.050+0000 7f22a67fc640 1 -- 172.21.15.46:0/1521353933 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a00b0440 con 0x7f22a80cf160 2023-10-30T14:19:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.050+0000 7f22ad365640 1 -- 172.21.15.46:0/1521353933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a80cf160 msgr2=0x7f22a80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.050+0000 7f22ad365640 1 --2- 172.21.15.46:0/1521353933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a80cf160 0x7f22a80cf590 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22a0009dc0 tx=0x7f22a001f080 comp rx=0 tx=0).stop 2023-10-30T14:19:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.050+0000 7f22ad365640 1 -- 172.21.15.46:0/1521353933 shutdown_connections 2023-10-30T14:19:12.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.050+0000 7f22ad365640 1 --2- 172.21.15.46:0/1521353933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a80cf160 0x7f22a80cf590 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:12.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.050+0000 7f22ad365640 1 -- 172.21.15.46:0/1521353933 >> 172.21.15.46:0/1521353933 conn(0x7f22a806bfe0 msgr2=0x7f22a806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:12.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.050+0000 7f22ad365640 1 -- 172.21.15.46:0/1521353933 shutdown_connections 2023-10-30T14:19:12.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.050+0000 7f22ad365640 1 -- 172.21.15.46:0/1521353933 wait complete. 2023-10-30T14:19:12.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22ad365640 1 Processor -- start 2023-10-30T14:19:12.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22ad365640 1 -- start start 2023-10-30T14:19:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22ad365640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22a80cf160 0x7f22a80ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22ad365640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a81538a0 0x7f22a80ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22ad365640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22a815d6f0 0x7f22a80b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22ad365640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a814d790 con 0x7f22a815d6f0 2023-10-30T14:19:12.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22ad365640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a8157f70 con 0x7f22a80cf160 2023-10-30T14:19:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22ad365640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a80b31a0 con 0x7f22a81538a0 2023-10-30T14:19:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22a77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22a815d6f0 0x7f22a80b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22a77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22a815d6f0 0x7f22a80b38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45978/0 (socket says 172.21.15.46:45978) 2023-10-30T14:19:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22a77fe640 1 -- 172.21.15.46:0/2469839795 learned_addr learned my addr 172.21.15.46:0/2469839795 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:19:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f229ffff640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a81538a0 0x7f22a80ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22a6ffd640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22a80cf160 0x7f22a80ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22a77fe640 1 -- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a81538a0 msgr2=0x7f22a80ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22a77fe640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a81538a0 0x7f22a80ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22a77fe640 1 -- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22a80cf160 msgr2=0x7f22a80ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22a77fe640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22a80cf160 0x7f22a80ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:12.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22a77fe640 1 -- 172.21.15.46:0/2469839795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a0009a70 con 0x7f22a815d6f0 2023-10-30T14:19:12.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22a6ffd640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22a80cf160 0x7f22a80ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:19:12.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22a77fe640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22a815d6f0 0x7f22a80b38d0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f22a0009eb0 tx=0x7f22a001be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f229f7fe640 1 -- 172.21.15.46:0/2469839795 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a0009450 con 0x7f22a815d6f0 2023-10-30T14:19:12.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f229f7fe640 1 -- 172.21.15.46:0/2469839795 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a00bc450 con 0x7f22a815d6f0 2023-10-30T14:19:12.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f229f7fe640 1 -- 172.21.15.46:0/2469839795 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a00bc7c0 con 0x7f22a815d6f0 2023-10-30T14:19:12.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22ad365640 1 -- 172.21.15.46:0/2469839795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a80b3440 con 0x7f22a815d6f0 2023-10-30T14:19:12.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.054+0000 7f22ad365640 1 -- 172.21.15.46:0/2469839795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a8158180 con 0x7f22a815d6f0 2023-10-30T14:19:12.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.058+0000 7f22ad365640 1 -- 172.21.15.46:0/2469839795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a815db00 con 0x7f22a815d6f0 2023-10-30T14:19:12.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.058+0000 7f229f7fe640 1 -- 172.21.15.46:0/2469839795 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f22a00bc920 con 0x7f22a815d6f0 2023-10-30T14:19:12.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.058+0000 7f229f7fe640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f22880664a0 0x7f2288068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:12.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.058+0000 7f229f7fe640 1 -- 172.21.15.46:0/2469839795 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f22a0123210 con 0x7f22a815d6f0 2023-10-30T14:19:12.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.058+0000 7f22a6ffd640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f22880664a0 0x7f2288068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:12.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.058+0000 7f22a6ffd640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f22880664a0 0x7f2288068960 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f22980011f0 tx=0x7f2298009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:12.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.062+0000 7f229f7fe640 1 -- 172.21.15.46:0/2469839795 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f22a815db00 con 0x7f22a815d6f0 2023-10-30T14:19:12.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.274+0000 7f22ad365640 1 -- 172.21.15.46:0/2469839795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22a815db00 con 0x7f22a815d6f0 2023-10-30T14:19:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.274+0000 7f229f7fe640 1 -- 172.21.15.46:0/2469839795 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f22a815db00 con 0x7f22a815d6f0 2023-10-30T14:19:12.279 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:19:12.279 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":235,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:06.771914+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:12.280 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 235 2023-10-30T14:19:12.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.278+0000 7f22ad365640 1 -- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f22880664a0 msgr2=0x7f2288068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:12.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.278+0000 7f22ad365640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f22880664a0 0x7f2288068960 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f22980011f0 tx=0x7f2298009040 comp rx=0 tx=0).stop 2023-10-30T14:19:12.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.278+0000 7f22ad365640 1 -- 172.21.15.46:0/2469839795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22a815d6f0 msgr2=0x7f22a80b38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:12.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.278+0000 7f22ad365640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22a815d6f0 0x7f22a80b38d0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f22a0009eb0 tx=0x7f22a001be20 comp rx=0 tx=0).stop 2023-10-30T14:19:12.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.278+0000 7f22ad365640 1 -- 172.21.15.46:0/2469839795 shutdown_connections 2023-10-30T14:19:12.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.278+0000 7f22ad365640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f22880664a0 0x7f2288068960 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:12.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.278+0000 7f22ad365640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22a815d6f0 0x7f22a80b38d0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:12.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.278+0000 7f22ad365640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a81538a0 0x7f22a80ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:12.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.278+0000 7f22ad365640 1 --2- 172.21.15.46:0/2469839795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22a80cf160 0x7f22a80ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:12.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.278+0000 7f22ad365640 1 -- 172.21.15.46:0/2469839795 >> 172.21.15.46:0/2469839795 conn(0x7f22a806bfe0 msgr2=0x7f22a80c5540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:12.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.278+0000 7f22ad365640 1 -- 172.21.15.46:0/2469839795 shutdown_connections 2023-10-30T14:19:12.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:12.278+0000 7f22ad365640 1 -- 172.21.15.46:0/2469839795 wait complete. 2023-10-30T14:19:12.300 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:19:14.300 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:14.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.450+0000 7f3b527f8640 1 Processor -- start 2023-10-30T14:19:14.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.450+0000 7f3b527f8640 1 -- start start 2023-10-30T14:19:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b527f8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3b4c06e470 0x7f3b4c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b527f8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3b4c0c8490 0x7f3b4c0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b527f8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c06afa0 con 0x7f3b4c153d70 2023-10-30T14:19:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b527f8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b4c06d780 con 0x7f3b4c06e470 2023-10-30T14:19:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b527f8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b4c154150 con 0x7f3b4c0c8490 2023-10-30T14:19:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b4bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3b4c153d70 0x7f3b4c0cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55134/0 (socket says 172.21.15.46:55134) 2023-10-30T14:19:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b4b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3b4c06e470 0x7f3b4c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b4bfff640 1 -- 172.21.15.46:0/3946396538 learned_addr learned my addr 172.21.15.46:0/3946396538 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:19:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b50d95640 1 --2- 172.21.15.46:0/3946396538 >> v2:172.21.15.154:3300/0 conn(0x7f3b4c0c8490 0x7f3b4c0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:14.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/3946396538 >> v2:172.21.15.154:3300/0 conn(0x7f3b4c0c8490 msgr2=0x7f3b4c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:14.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/3946396538 >> v2:172.21.15.154:3300/0 conn(0x7f3b4c0c8490 0x7f3b4c0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:14.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/3946396538 >> v1:172.21.15.46:6789/0 conn(0x7f3b4c153d70 legacy=0x7f3b4c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.454+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/3946396538 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c06e030 con 0x7f3b4c06e470 2023-10-30T14:19:14.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/3946396538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c06e470 0x7f3b4c06ca80 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0cddf0 tx=0x7f3b4001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d067e8af6a1e777e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:14.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b4affd640 1 -- 172.21.15.46:0/3946396538 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b400a7070 con 0x7f3b4c06e470 2023-10-30T14:19:14.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b4affd640 1 -- 172.21.15.46:0/3946396538 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b400ab430 con 0x7f3b4c06e470 2023-10-30T14:19:14.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b4affd640 1 -- 172.21.15.46:0/3946396538 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b400ab7c0 con 0x7f3b4c06e470 2023-10-30T14:19:14.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b527f8640 1 -- 172.21.15.46:0/3946396538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c06e470 msgr2=0x7f3b4c06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:14.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b527f8640 1 --2- 172.21.15.46:0/3946396538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c06e470 0x7f3b4c06ca80 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0cddf0 tx=0x7f3b4001c430 comp rx=0 tx=0).stop 2023-10-30T14:19:14.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b527f8640 1 -- 172.21.15.46:0/3946396538 shutdown_connections 2023-10-30T14:19:14.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b527f8640 1 --2- 172.21.15.46:0/3946396538 >> v2:172.21.15.154:3300/0 conn(0x7f3b4c0c8490 0x7f3b4c0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b527f8640 1 --2- 172.21.15.46:0/3946396538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c06e470 0x7f3b4c06ca80 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b527f8640 1 -- 172.21.15.46:0/3946396538 >> 172.21.15.46:0/3946396538 conn(0x7f3b4c071ea0 msgr2=0x7f3b4c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b527f8640 1 -- 172.21.15.46:0/3946396538 shutdown_connections 2023-10-30T14:19:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b527f8640 1 -- 172.21.15.46:0/3946396538 wait complete. 2023-10-30T14:19:14.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b527f8640 1 Processor -- start 2023-10-30T14:19:14.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.458+0000 7f3b527f8640 1 -- start start 2023-10-30T14:19:14.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b527f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b4c06e470 0x7f3b4c146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:14.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b527f8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c0c8490 0x7f3b4c154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:14.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b527f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b4c153d70 0x7f3b4c143d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:14.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b527f8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c1564b0 con 0x7f3b4c0c8490 2023-10-30T14:19:14.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b527f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c0cc670 con 0x7f3b4c153d70 2023-10-30T14:19:14.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b527f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c0cc8a0 con 0x7f3b4c06e470 2023-10-30T14:19:14.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b4b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c0c8490 0x7f3b4c154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:14.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b4b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c0c8490 0x7f3b4c154290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46002/0 (socket says 172.21.15.46:46002) 2023-10-30T14:19:14.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/3398247704 learned_addr learned my addr 172.21.15.46:0/3398247704 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:19:14.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b4bfff640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b4c06e470 0x7f3b4c146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:14.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b50d95640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b4c153d70 0x7f3b4c143d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:14.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b4c06e470 msgr2=0x7f3b4c146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:14.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b4c06e470 0x7f3b4c146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:14.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b4c153d70 msgr2=0x7f3b4c143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b4c153d70 0x7f3b4c143d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/3398247704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b40009420 con 0x7f3b4c0c8490 2023-10-30T14:19:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b50d95640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b4c153d70 0x7f3b4c143d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:19:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c0c8490 0x7f3b4c154290 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f3b40019040 tx=0x7f3b4001ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b48ff9640 1 -- 172.21.15.46:0/3398247704 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b400b5810 con 0x7f3b4c0c8490 2023-10-30T14:19:14.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b48ff9640 1 -- 172.21.15.46:0/3398247704 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b400b5990 con 0x7f3b4c0c8490 2023-10-30T14:19:14.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b48ff9640 1 -- 172.21.15.46:0/3398247704 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b400b5d40 con 0x7f3b4c0c8490 2023-10-30T14:19:14.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b527f8640 1 -- 172.21.15.46:0/3398247704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4c0cb680 con 0x7f3b4c0c8490 2023-10-30T14:19:14.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b527f8640 1 -- 172.21.15.46:0/3398247704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4c144a20 con 0x7f3b4c0c8490 2023-10-30T14:19:14.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b527f8640 1 -- 172.21.15.46:0/3398247704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b4c04e170 con 0x7f3b4c0c8490 2023-10-30T14:19:14.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.462+0000 7f3b48ff9640 1 -- 172.21.15.46:0/3398247704 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3b400b4390 con 0x7f3b4c0c8490 2023-10-30T14:19:14.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.466+0000 7f3b48ff9640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3b380664a0 0x7f3b38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:14.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.466+0000 7f3b48ff9640 1 -- 172.21.15.46:0/3398247704 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f3b4011f0f0 con 0x7f3b4c0c8490 2023-10-30T14:19:14.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.466+0000 7f3b4bfff640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3b380664a0 0x7f3b38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:14.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.466+0000 7f3b4bfff640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3b380664a0 0x7f3b38068960 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c005890 tx=0x7f3b3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:14.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.470+0000 7f3b48ff9640 1 -- 172.21.15.46:0/3398247704 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3b400ebbb0 con 0x7f3b4c0c8490 2023-10-30T14:19:14.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.658+0000 7f3b527f8640 1 -- 172.21.15.46:0/3398247704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b4c163fd0 con 0x7f3b4c0c8490 2023-10-30T14:19:14.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.658+0000 7f3b48ff9640 1 -- 172.21.15.46:0/3398247704 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 236 v236) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f3b400c4090 con 0x7f3b4c0c8490 2023-10-30T14:19:14.663 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:19:14.664 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":236,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:12.129796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:14.664 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 236 2023-10-30T14:19:14.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.662+0000 7f3b527f8640 1 -- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3b380664a0 msgr2=0x7f3b38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:14.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.662+0000 7f3b527f8640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3b380664a0 0x7f3b38068960 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c005890 tx=0x7f3b3c009040 comp rx=0 tx=0).stop 2023-10-30T14:19:14.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.662+0000 7f3b527f8640 1 -- 172.21.15.46:0/3398247704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c0c8490 msgr2=0x7f3b4c154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:14.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.662+0000 7f3b527f8640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c0c8490 0x7f3b4c154290 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f3b40019040 tx=0x7f3b4001ac70 comp rx=0 tx=0).stop 2023-10-30T14:19:14.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.662+0000 7f3b527f8640 1 -- 172.21.15.46:0/3398247704 shutdown_connections 2023-10-30T14:19:14.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.662+0000 7f3b527f8640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3b380664a0 0x7f3b38068960 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:14.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.662+0000 7f3b527f8640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b4c153d70 0x7f3b4c143d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:14.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.662+0000 7f3b527f8640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c0c8490 0x7f3b4c154290 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:14.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.662+0000 7f3b527f8640 1 --2- 172.21.15.46:0/3398247704 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b4c06e470 0x7f3b4c146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:14.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.662+0000 7f3b527f8640 1 -- 172.21.15.46:0/3398247704 >> 172.21.15.46:0/3398247704 conn(0x7f3b4c071ea0 msgr2=0x7f3b4c0c76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:14.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.662+0000 7f3b527f8640 1 -- 172.21.15.46:0/3398247704 shutdown_connections 2023-10-30T14:19:14.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:14.662+0000 7f3b527f8640 1 -- 172.21.15.46:0/3398247704 wait complete. 2023-10-30T14:19:14.685 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=9135 2023-10-30T14:19:14.685 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.e 2023-10-30T14:19:18.982 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-10-30T14:19:18.978+0000 7f456f162640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:19:18.983 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-10-30T14:19:18.978+0000 7f456f162640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:19:19.821 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:19:19.822 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.68 2023-10-30T14:19:19.823 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 68 -eq 48 ] 2023-10-30T14:19:19.823 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.68 2023-10-30T14:19:19.823 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.68 2023-10-30T14:19:19.823 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.68 2023-10-30T14:19:19.823 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:19:22.686 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T14:19:22.686 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T14:19:22.686 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T14:19:22.691 INFO:tasks.ceph.mds.e:Started 2023-10-30T14:19:22.691 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:19:22.778 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T14:19:22.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9ee0d03640 1 Processor -- start 2023-10-30T14:19:22.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9ee0d03640 1 -- start start 2023-10-30T14:19:22.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9ee0d03640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9edc1538f0 0x7f9edc159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:22.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9ee0d03640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9edc15cd50 0x7f9edc0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:22.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9ee0d03640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9edc15d300 con 0x7f9edc0cefa0 2023-10-30T14:19:22.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9ee0d03640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9edc15d5e0 con 0x7f9edc1538f0 2023-10-30T14:19:22.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9ee0d03640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9edc15d8c0 con 0x7f9edc15cd50 2023-10-30T14:19:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9edb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9edc1538f0 0x7f9edc159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9edb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9edc1538f0 0x7f9edc159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56384/0 (socket says 172.21.15.46:56384) 2023-10-30T14:19:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9edb7fe640 1 -- 172.21.15.46:0/22220621 learned_addr learned my addr 172.21.15.46:0/22220621 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:19:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9edaffd640 1 --2- 172.21.15.46:0/22220621 >> v2:172.21.15.154:3300/0 conn(0x7f9edc15cd50 0x7f9edc0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:22.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9edb7fe640 1 -- 172.21.15.46:0/22220621 >> v2:172.21.15.154:3300/0 conn(0x7f9edc15cd50 msgr2=0x7f9edc0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:19:22.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9edb7fe640 1 --2- 172.21.15.46:0/22220621 >> v2:172.21.15.154:3300/0 conn(0x7f9edc15cd50 0x7f9edc0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:22.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9edb7fe640 1 -- 172.21.15.46:0/22220621 >> v1:172.21.15.154:6789/0 conn(0x7f9edc0cefa0 legacy=0x7f9edc0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:22.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.834+0000 7f9edb7fe640 1 -- 172.21.15.46:0/22220621 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc15dba0 con 0x7f9edc1538f0 2023-10-30T14:19:22.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9edb7fe640 1 --2- 172.21.15.46:0/22220621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9edc1538f0 0x7f9edc159fb0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f9edc072e90 tx=0x7f9ecc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1175ba104a3e8dd5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:22.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9eda7fc640 1 -- 172.21.15.46:0/22220621 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc0a89f0 con 0x7f9edc1538f0 2023-10-30T14:19:22.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9eda7fc640 1 -- 172.21.15.46:0/22220621 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc0a8b50 con 0x7f9edc1538f0 2023-10-30T14:19:22.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9eda7fc640 1 -- 172.21.15.46:0/22220621 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc0b0440 con 0x7f9edc1538f0 2023-10-30T14:19:22.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9ee0d03640 1 -- 172.21.15.46:0/22220621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9edc1538f0 msgr2=0x7f9edc159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:22.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9ee0d03640 1 --2- 172.21.15.46:0/22220621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9edc1538f0 0x7f9edc159fb0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f9edc072e90 tx=0x7f9ecc01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:19:22.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9ee0d03640 1 -- 172.21.15.46:0/22220621 shutdown_connections 2023-10-30T14:19:22.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9ee0d03640 1 --2- 172.21.15.46:0/22220621 >> v2:172.21.15.154:3300/0 conn(0x7f9edc15cd50 0x7f9edc0c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:22.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9ee0d03640 1 --2- 172.21.15.46:0/22220621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9edc1538f0 0x7f9edc159fb0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:22.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9ee0d03640 1 -- 172.21.15.46:0/22220621 >> 172.21.15.46:0/22220621 conn(0x7f9edc06a640 msgr2=0x7f9edc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9ee0d03640 1 -- 172.21.15.46:0/22220621 shutdown_connections 2023-10-30T14:19:22.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9ee0d03640 1 -- 172.21.15.46:0/22220621 wait complete. 2023-10-30T14:19:22.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.838+0000 7f9ee0d03640 1 Processor -- start 2023-10-30T14:19:22.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ee0d03640 1 -- start start 2023-10-30T14:19:22.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ee0d03640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9edc0cefa0 0x7f9edc1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:22.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ee0d03640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9edc1538f0 0x7f9edc145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ee0d03640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9edc15cd50 0x7f9edc1f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ee0d03640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc152ad0 con 0x7f9edc0cefa0 2023-10-30T14:19:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ee0d03640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc146490 con 0x7f9edc1538f0 2023-10-30T14:19:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ee0d03640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc157ff0 con 0x7f9edc15cd50 2023-10-30T14:19:22.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9edc0cefa0 0x7f9edc1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9edc0cefa0 0x7f9edc1573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56386/0 (socket says 172.21.15.46:56386) 2023-10-30T14:19:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9edb7fe640 1 -- 172.21.15.46:0/1432821571 learned_addr learned my addr 172.21.15.46:0/1432821571 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:19:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9edaffd640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9edc1538f0 0x7f9edc145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9edbfff640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9edc15cd50 0x7f9edc1f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:22.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9edb7fe640 1 -- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9edc15cd50 msgr2=0x7f9edc1f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:22.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9edb7fe640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9edc15cd50 0x7f9edc1f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:22.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9edb7fe640 1 -- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9edc1538f0 msgr2=0x7f9edc145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:22.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9edb7fe640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9edc1538f0 0x7f9edc145f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:22.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9edb7fe640 1 -- 172.21.15.46:0/1432821571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc0094d0 con 0x7f9edc0cefa0 2023-10-30T14:19:22.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9edb7fe640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9edc0cefa0 0x7f9edc1573a0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc01b900 tx=0x7f9ecc01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ec3fff640 1 -- 172.21.15.46:0/1432821571 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc0ba600 con 0x7f9edc0cefa0 2023-10-30T14:19:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ec3fff640 1 -- 172.21.15.46:0/1432821571 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc0ba760 con 0x7f9edc0cefa0 2023-10-30T14:19:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ec3fff640 1 -- 172.21.15.46:0/1432821571 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc0bab10 con 0x7f9edc0cefa0 2023-10-30T14:19:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ee0d03640 1 -- 172.21.15.46:0/1432821571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9edc157a30 con 0x7f9edc0cefa0 2023-10-30T14:19:22.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ee0d03640 1 -- 172.21.15.46:0/1432821571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9edc152480 con 0x7f9edc0cefa0 2023-10-30T14:19:22.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.842+0000 7f9ee0d03640 1 -- 172.21.15.46:0/1432821571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9edc04e170 con 0x7f9edc0cefa0 2023-10-30T14:19:22.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.846+0000 7f9ec3fff640 1 -- 172.21.15.46:0/1432821571 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ecc0bac70 con 0x7f9edc0cefa0 2023-10-30T14:19:22.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.846+0000 7f9ec3fff640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9ec80664d0 0x7f9ec8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:19:22.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.846+0000 7f9ec3fff640 1 -- 172.21.15.46:0/1432821571 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f9ecc122f60 con 0x7f9edc0cefa0 2023-10-30T14:19:22.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.846+0000 7f9edaffd640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9ec80664d0 0x7f9ec8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:19:22.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.846+0000 7f9edaffd640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9ec80664d0 0x7f9ec8068990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0b1790 tx=0x7f9ed0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:19:22.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:22.850+0000 7f9ec3fff640 1 -- 172.21.15.46:0/1432821571 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ecc0ef970 con 0x7f9edc0cefa0 2023-10-30T14:19:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.050+0000 7f9ee0d03640 1 -- 172.21.15.46:0/1432821571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9edc150ed0 con 0x7f9edc0cefa0 2023-10-30T14:19:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.050+0000 7f9ec3fff640 1 -- 172.21.15.46:0/1432821571 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f9ecc0b5070 con 0x7f9edc0cefa0 2023-10-30T14:19:23.053 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:19:23.053 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9594,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:12.129796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:19:23.054 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 237 2023-10-30T14:19:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.050+0000 7f9ee0d03640 1 -- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9ec80664d0 msgr2=0x7f9ec8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.050+0000 7f9ee0d03640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9ec80664d0 0x7f9ec8068990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0b1790 tx=0x7f9ed0008040 comp rx=0 tx=0).stop 2023-10-30T14:19:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.050+0000 7f9ee0d03640 1 -- 172.21.15.46:0/1432821571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9edc0cefa0 msgr2=0x7f9edc1573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:19:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.050+0000 7f9ee0d03640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9edc0cefa0 0x7f9edc1573a0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc01b900 tx=0x7f9ecc01ba90 comp rx=0 tx=0).stop 2023-10-30T14:19:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.050+0000 7f9ee0d03640 1 -- 172.21.15.46:0/1432821571 shutdown_connections 2023-10-30T14:19:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.050+0000 7f9ee0d03640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f9ec80664d0 0x7f9ec8068990 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:23.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.050+0000 7f9ee0d03640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9edc15cd50 0x7f9edc1f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:23.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.050+0000 7f9ee0d03640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9edc1538f0 0x7f9edc145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:23.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.050+0000 7f9ee0d03640 1 --2- 172.21.15.46:0/1432821571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9edc0cefa0 0x7f9edc1573a0 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:19:23.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.050+0000 7f9ee0d03640 1 -- 172.21.15.46:0/1432821571 >> 172.21.15.46:0/1432821571 conn(0x7f9edc06a640 msgr2=0x7f9edc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:19:23.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.054+0000 7f9ee0d03640 1 -- 172.21.15.46:0/1432821571 shutdown_connections 2023-10-30T14:19:23.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:19:23.054+0000 7f9ee0d03640 1 -- 172.21.15.46:0/1432821571 wait complete. 2023-10-30T14:19:23.077 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T14:19:23.078 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 63 secs before thrashing 2023-10-30T14:19:25.680 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:19:25.681 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 69 -eq 48 ] 2023-10-30T14:19:25.681 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.69 2023-10-30T14:19:25.681 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.69 2023-10-30T14:19:25.682 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.69 2023-10-30T14:19:25.682 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.69 2023-10-30T14:19:25.682 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:19:30.062 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:19:30.067 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:19:34.938 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:19:34.939 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.70 2023-10-30T14:19:34.939 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 70 -eq 48 ] 2023-10-30T14:19:34.940 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.70 2023-10-30T14:19:34.940 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.70 2023-10-30T14:19:34.940 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.70 2023-10-30T14:19:34.940 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:19:42.711 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:19:42.711 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 71 -eq 48 ] 2023-10-30T14:19:42.711 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.71 2023-10-30T14:19:42.712 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.71 2023-10-30T14:19:42.712 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.71 2023-10-30T14:19:42.712 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.71 2023-10-30T14:19:42.713 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:19:49.201 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:19:49.201 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 72 -eq 48 ] 2023-10-30T14:19:49.202 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.72 2023-10-30T14:19:49.202 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.72 2023-10-30T14:19:49.203 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.72 2023-10-30T14:19:49.203 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.72 2023-10-30T14:19:49.203 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:19:56.793 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:19:56.794 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 73 -eq 48 ] 2023-10-30T14:19:56.794 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.73 2023-10-30T14:19:56.794 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.73 2023-10-30T14:19:56.795 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.73 2023-10-30T14:19:56.795 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.73 2023-10-30T14:19:56.795 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:20:00.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:20:00.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:20:03.422 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:20:03.423 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.74 2023-10-30T14:20:03.424 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 74 -eq 48 ] 2023-10-30T14:20:03.424 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.74 2023-10-30T14:20:03.424 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.74 2023-10-30T14:20:03.425 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.74 2023-10-30T14:20:03.425 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:20:08.542 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:20:08.542 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 75 -eq 48 ] 2023-10-30T14:20:08.542 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.75 2023-10-30T14:20:08.543 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.75 2023-10-30T14:20:08.543 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.75 2023-10-30T14:20:08.543 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.75 2023-10-30T14:20:08.543 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:20:16.564 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:20:16.565 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 76 -eq 48 ] 2023-10-30T14:20:16.565 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.76 2023-10-30T14:20:16.565 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.76 2023-10-30T14:20:16.566 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.76 2023-10-30T14:20:16.566 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.76 2023-10-30T14:20:16.566 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:20:24.346 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:20:24.346 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 77 -eq 48 ] 2023-10-30T14:20:24.346 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.77 2023-10-30T14:20:24.347 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.77 2023-10-30T14:20:24.347 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.77 2023-10-30T14:20:24.348 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.77 2023-10-30T14:20:24.348 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:20:26.078 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:26.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.218+0000 7ff86c6a9640 1 Processor -- start 2023-10-30T14:20:26.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.218+0000 7ff86c6a9640 1 -- start start 2023-10-30T14:20:26.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.222+0000 7ff86c6a9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8640c6b80 0x7ff8640bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:26.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.222+0000 7ff86c6a9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8640c0a50 con 0x7ff8640c6f60 2023-10-30T14:20:26.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.222+0000 7ff86c6a9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86406f6e0 con 0x7ff86413f4a0 2023-10-30T14:20:26.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.222+0000 7ff86c6a9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86406f9c0 con 0x7ff8640c6b80 2023-10-30T14:20:26.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.222+0000 7ff86a445640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8640c6b80 0x7ff8640bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:26.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.222+0000 7ff869c44640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff8640c6f60 0x7ff86413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44360/0 (socket says 172.21.15.46:44360) 2023-10-30T14:20:26.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.222+0000 7ff869c44640 1 -- 172.21.15.46:0/1240285476 learned_addr learned my addr 172.21.15.46:0/1240285476 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:20:26.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.222+0000 7ff86a445640 1 -- 172.21.15.46:0/1240285476 >> v1:172.21.15.154:6789/0 conn(0x7ff86413f4a0 legacy=0x7ff8641566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:26.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.222+0000 7ff86a445640 1 -- 172.21.15.46:0/1240285476 >> v1:172.21.15.46:6789/0 conn(0x7ff8640c6f60 legacy=0x7ff86413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:26.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.222+0000 7ff86a445640 1 -- 172.21.15.46:0/1240285476 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86406fca0 con 0x7ff8640c6b80 2023-10-30T14:20:26.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.222+0000 7ff869443640 1 -- 172.21.15.46:0/1240285476 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820212382 0 0) 0x7ff8640c0a50 con 0x7ff8640c6f60 2023-10-30T14:20:26.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.222+0000 7ff86a445640 1 --2- 172.21.15.46:0/1240285476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8640c6b80 0x7ff8640bfec0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7ff8600097c0 tx=0x7ff86001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e20899c9bd3e169 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:26.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff869443640 1 -- 172.21.15.46:0/1240285476 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8600a8050 con 0x7ff8640c6b80 2023-10-30T14:20:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff869443640 1 -- 172.21.15.46:0/1240285476 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8600ac5f0 con 0x7ff8640c6b80 2023-10-30T14:20:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff869443640 1 -- 172.21.15.46:0/1240285476 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8600ac980 con 0x7ff8640c6b80 2023-10-30T14:20:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 -- 172.21.15.46:0/1240285476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8640c6b80 msgr2=0x7ff8640bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 --2- 172.21.15.46:0/1240285476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8640c6b80 0x7ff8640bfec0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7ff8600097c0 tx=0x7ff86001c480 comp rx=0 tx=0).stop 2023-10-30T14:20:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 -- 172.21.15.46:0/1240285476 shutdown_connections 2023-10-30T14:20:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 --2- 172.21.15.46:0/1240285476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8640c6b80 0x7ff8640bfec0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 -- 172.21.15.46:0/1240285476 >> 172.21.15.46:0/1240285476 conn(0x7ff8640c7910 msgr2=0x7ff8640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 -- 172.21.15.46:0/1240285476 shutdown_connections 2023-10-30T14:20:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 -- 172.21.15.46:0/1240285476 wait complete. 2023-10-30T14:20:26.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 Processor -- start 2023-10-30T14:20:26.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 -- start start 2023-10-30T14:20:26.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8640c6b80 0x7ff8641f07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8640c6f60 0x7ff864158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86413f4a0 0x7ff864153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86413c970 con 0x7ff86413f4a0 2023-10-30T14:20:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8641efdc0 con 0x7ff8640c6b80 2023-10-30T14:20:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.226+0000 7ff86c6a9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8641f4210 con 0x7ff8640c6f60 2023-10-30T14:20:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86ac46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86413f4a0 0x7ff864153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86ac46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86413f4a0 0x7ff864153240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45370/0 (socket says 172.21.15.46:45370) 2023-10-30T14:20:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86ac46640 1 -- 172.21.15.46:0/412780689 learned_addr learned my addr 172.21.15.46:0/412780689 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff869c44640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8640c6f60 0x7ff864158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86a445640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8640c6b80 0x7ff8641f07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86ac46640 1 -- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8640c6f60 msgr2=0x7ff864158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86ac46640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8640c6f60 0x7ff864158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86ac46640 1 -- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8640c6b80 msgr2=0x7ff8641f07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86ac46640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8640c6b80 0x7ff8641f07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86ac46640 1 -- 172.21.15.46:0/412780689 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff860009420 con 0x7ff86413f4a0 2023-10-30T14:20:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86ac46640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86413f4a0 0x7ff864153240 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7ff85c00c730 tx=0x7ff85c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff85affd640 1 -- 172.21.15.46:0/412780689 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c003b80 con 0x7ff86413f4a0 2023-10-30T14:20:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff85affd640 1 -- 172.21.15.46:0/412780689 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85c003ce0 con 0x7ff86413f4a0 2023-10-30T14:20:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86c6a9640 1 -- 172.21.15.46:0/412780689 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8641f0d10 con 0x7ff86413f4a0 2023-10-30T14:20:26.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff85affd640 1 -- 172.21.15.46:0/412780689 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c005310 con 0x7ff86413f4a0 2023-10-30T14:20:26.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86c6a9640 1 -- 172.21.15.46:0/412780689 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86413c690 con 0x7ff86413f4a0 2023-10-30T14:20:26.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff86c6a9640 1 -- 172.21.15.46:0/412780689 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86404e170 con 0x7ff86413f4a0 2023-10-30T14:20:26.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff85affd640 1 -- 172.21.15.46:0/412780689 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff85c005680 con 0x7ff86413f4a0 2023-10-30T14:20:26.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff85affd640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8480664d0 0x7ff848068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:26.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.230+0000 7ff85affd640 1 -- 172.21.15.46:0/412780689 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7ff85c084830 con 0x7ff86413f4a0 2023-10-30T14:20:26.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.234+0000 7ff86a445640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8480664d0 0x7ff848068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:26.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.234+0000 7ff86a445640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8480664d0 0x7ff848068990 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff8600ae000 tx=0x7ff8600a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:26.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.234+0000 7ff85affd640 1 -- 172.21.15.46:0/412780689 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff85c051240 con 0x7ff86413f4a0 2023-10-30T14:20:26.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 -- 172.21.15.46:0/412780689 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff864156950 con 0x7ff86413f4a0 2023-10-30T14:20:26.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff85affd640 1 -- 172.21.15.46:0/412780689 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+6523 (secure 0 0 0) 0x7ff85c054860 con 0x7ff86413f4a0 2023-10-30T14:20:26.437 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:26.437 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9594,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:23.041684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:26.437 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 238 2023-10-30T14:20:26.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 -- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8480664d0 msgr2=0x7ff848068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:26.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8480664d0 0x7ff848068990 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff8600ae000 tx=0x7ff8600a7000 comp rx=0 tx=0).stop 2023-10-30T14:20:26.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 -- 172.21.15.46:0/412780689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86413f4a0 msgr2=0x7ff864153240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:26.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86413f4a0 0x7ff864153240 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7ff85c00c730 tx=0x7ff85c00cbf0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 -- 172.21.15.46:0/412780689 shutdown_connections 2023-10-30T14:20:26.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff8480664d0 0x7ff848068990 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86413f4a0 0x7ff864153240 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8640c6f60 0x7ff864158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 --2- 172.21.15.46:0/412780689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8640c6b80 0x7ff8641f07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 -- 172.21.15.46:0/412780689 >> 172.21.15.46:0/412780689 conn(0x7ff8640c7910 msgr2=0x7ff864135b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:26.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 -- 172.21.15.46:0/412780689 shutdown_connections 2023-10-30T14:20:26.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.434+0000 7ff86c6a9640 1 -- 172.21.15.46:0/412780689 wait complete. 2023-10-30T14:20:26.457 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=1) 2023-10-30T14:20:26.458 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:20:26.458 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:20:26.484 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:20:26.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:26.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8407bf8640 1 Processor -- start 2023-10-30T14:20:26.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8407bf8640 1 -- start start 2023-10-30T14:20:26.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8407bf8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8400151800 0x7f840006e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:26.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8407bf8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f84000c1ee0 0x7f84000c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8407bf8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840006e090 con 0x7f84000c9230 2023-10-30T14:20:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8407bf8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f840006dc70 con 0x7f8400151800 2023-10-30T14:20:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8407bf8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84000c8a50 con 0x7f84000c1ee0 2023-10-30T14:20:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8405994640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8400151800 0x7f840006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8405994640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8400151800 0x7f840006e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52908/0 (socket says 172.21.15.46:52908) 2023-10-30T14:20:26.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8405994640 1 -- 172.21.15.46:0/1669654077 learned_addr learned my addr 172.21.15.46:0/1669654077 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:26.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8406195640 1 --1- 172.21.15.46:0/1669654077 >> v1:172.21.15.46:6789/0 conn(0x7f84000c9230 0x7f84000cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57520/0 (socket says 172.21.15.46:57520) 2023-10-30T14:20:26.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8405193640 1 --2- 172.21.15.46:0/1669654077 >> v2:172.21.15.154:3300/0 conn(0x7f84000c1ee0 0x7f84000c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:26.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8405994640 1 -- 172.21.15.46:0/1669654077 >> v2:172.21.15.154:3300/0 conn(0x7f84000c1ee0 msgr2=0x7f84000c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:26.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.666+0000 7f8405994640 1 --2- 172.21.15.46:0/1669654077 >> v2:172.21.15.154:3300/0 conn(0x7f84000c1ee0 0x7f84000c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8405994640 1 -- 172.21.15.46:0/1669654077 >> v1:172.21.15.46:6789/0 conn(0x7f84000c9230 legacy=0x7f84000cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:26.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8405994640 1 -- 172.21.15.46:0/1669654077 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84000c3b10 con 0x7f8400151800 2023-10-30T14:20:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8405994640 1 --2- 172.21.15.46:0/1669654077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8400151800 0x7f840006e470 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f840006e9b0 tx=0x7f83fc01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8495d3feff0d77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8404992640 1 -- 172.21.15.46:0/1669654077 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc0aa430 con 0x7f8400151800 2023-10-30T14:20:26.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8404992640 1 -- 172.21.15.46:0/1669654077 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83fc0aa590 con 0x7f8400151800 2023-10-30T14:20:26.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8404992640 1 -- 172.21.15.46:0/1669654077 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc0aa920 con 0x7f8400151800 2023-10-30T14:20:26.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8407bf8640 1 -- 172.21.15.46:0/1669654077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8400151800 msgr2=0x7f840006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:26.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8407bf8640 1 --2- 172.21.15.46:0/1669654077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8400151800 0x7f840006e470 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f840006e9b0 tx=0x7f83fc01c4d0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8407bf8640 1 -- 172.21.15.46:0/1669654077 shutdown_connections 2023-10-30T14:20:26.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8407bf8640 1 --2- 172.21.15.46:0/1669654077 >> v2:172.21.15.154:3300/0 conn(0x7f84000c1ee0 0x7f84000c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8407bf8640 1 --2- 172.21.15.46:0/1669654077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8400151800 0x7f840006e470 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8407bf8640 1 -- 172.21.15.46:0/1669654077 >> 172.21.15.46:0/1669654077 conn(0x7f8400071ea0 msgr2=0x7f8400065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:26.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8407bf8640 1 -- 172.21.15.46:0/1669654077 shutdown_connections 2023-10-30T14:20:26.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.670+0000 7f8407bf8640 1 -- 172.21.15.46:0/1669654077 wait complete. 2023-10-30T14:20:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8407bf8640 1 Processor -- start 2023-10-30T14:20:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8407bf8640 1 -- start start 2023-10-30T14:20:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8407bf8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84000c1ee0 0x7f840014af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8407bf8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84000c9230 0x7f840014c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8407bf8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8400151800 0x7f84001505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8407bf8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840014dd10 con 0x7f84000c9230 2023-10-30T14:20:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8407bf8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84001f1160 con 0x7f8400151800 2023-10-30T14:20:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8407bf8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84000c65e0 con 0x7f84000c1ee0 2023-10-30T14:20:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8405193640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84000c9230 0x7f840014c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8405193640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84000c9230 0x7f840014c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52924/0 (socket says 172.21.15.46:52924) 2023-10-30T14:20:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8405193640 1 -- 172.21.15.46:0/3090129241 learned_addr learned my addr 172.21.15.46:0/3090129241 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8405994640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84000c1ee0 0x7f840014af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8406195640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8400151800 0x7f84001505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8405193640 1 -- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84000c1ee0 msgr2=0x7f840014af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8405193640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84000c1ee0 0x7f840014af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8405193640 1 -- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8400151800 msgr2=0x7f84001505e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8405193640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8400151800 0x7f84001505e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8405193640 1 -- 172.21.15.46:0/3090129241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83fc0094d0 con 0x7f84000c9230 2023-10-30T14:20:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8405193640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84000c9230 0x7f840014c910 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f83f000d5d0 tx=0x7f83f000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f83f67fc640 1 -- 172.21.15.46:0/3090129241 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f0004240 con 0x7f84000c9230 2023-10-30T14:20:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f83f67fc640 1 -- 172.21.15.46:0/3090129241 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f00043a0 con 0x7f84000c9230 2023-10-30T14:20:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f83f67fc640 1 -- 172.21.15.46:0/3090129241 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f0003840 con 0x7f84000c9230 2023-10-30T14:20:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8407bf8640 1 -- 172.21.15.46:0/3090129241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84001f1420 con 0x7f84000c9230 2023-10-30T14:20:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.674+0000 7f8407bf8640 1 -- 172.21.15.46:0/3090129241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f840014be30 con 0x7f84000c9230 2023-10-30T14:20:26.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.678+0000 7f8407bf8640 1 -- 172.21.15.46:0/3090129241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f840004e170 con 0x7f84000c9230 2023-10-30T14:20:26.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.678+0000 7f83f67fc640 1 -- 172.21.15.46:0/3090129241 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83f00039a0 con 0x7f84000c9230 2023-10-30T14:20:26.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.678+0000 7f83f67fc640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83e40664a0 0x7f83e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:26.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.678+0000 7f83f67fc640 1 -- 172.21.15.46:0/3090129241 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f83f0084b30 con 0x7f84000c9230 2023-10-30T14:20:26.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.678+0000 7f8405994640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83e40664a0 0x7f83e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:26.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.678+0000 7f8405994640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83e40664a0 0x7f83e4068960 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f83fc0ae000 tx=0x7f83fc0b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:26.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.682+0000 7f83f67fc640 1 -- 172.21.15.46:0/3090129241 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f83f0051420 con 0x7f84000c9230 2023-10-30T14:20:26.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.870+0000 7f8407bf8640 1 -- 172.21.15.46:0/3090129241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84000c6de0 con 0x7f84000c9230 2023-10-30T14:20:26.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.870+0000 7f83f67fc640 1 -- 172.21.15.46:0/3090129241 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f83f0054a40 con 0x7f84000c9230 2023-10-30T14:20:26.875 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:26.875 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9594,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:23.041684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:26.876 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 238 2023-10-30T14:20:26.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.874+0000 7f8407bf8640 1 -- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83e40664a0 msgr2=0x7f83e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:26.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.874+0000 7f8407bf8640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83e40664a0 0x7f83e4068960 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f83fc0ae000 tx=0x7f83fc0b5040 comp rx=0 tx=0).stop 2023-10-30T14:20:26.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.874+0000 7f8407bf8640 1 -- 172.21.15.46:0/3090129241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84000c9230 msgr2=0x7f840014c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:26.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.874+0000 7f8407bf8640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84000c9230 0x7f840014c910 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f83f000d5d0 tx=0x7f83f000da90 comp rx=0 tx=0).stop 2023-10-30T14:20:26.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.874+0000 7f8407bf8640 1 -- 172.21.15.46:0/3090129241 shutdown_connections 2023-10-30T14:20:26.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.874+0000 7f8407bf8640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f83e40664a0 0x7f83e4068960 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.874+0000 7f8407bf8640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8400151800 0x7f84001505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.874+0000 7f8407bf8640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84000c9230 0x7f840014c910 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.874+0000 7f8407bf8640 1 --2- 172.21.15.46:0/3090129241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84000c1ee0 0x7f840014af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:26.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.874+0000 7f8407bf8640 1 -- 172.21.15.46:0/3090129241 >> 172.21.15.46:0/3090129241 conn(0x7f8400071ea0 msgr2=0x7f84000cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:26.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.874+0000 7f8407bf8640 1 -- 172.21.15.46:0/3090129241 shutdown_connections 2023-10-30T14:20:26.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:26.874+0000 7f8407bf8640 1 -- 172.21.15.46:0/3090129241 wait complete. 2023-10-30T14:20:26.897 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:20:28.898 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.050+0000 7f516a10f640 1 Processor -- start 2023-10-30T14:20:29.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.050+0000 7f516a10f640 1 -- start start 2023-10-30T14:20:29.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.050+0000 7f516a10f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5164153bf0 0x7f5164071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:29.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.050+0000 7f516a10f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f51640c5f60 0x7f51640cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:29.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.050+0000 7f516a10f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51640700b0 con 0x7f51640c6fe0 2023-10-30T14:20:29.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.050+0000 7f516a10f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51640c6810 con 0x7f51640c5f60 2023-10-30T14:20:29.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.050+0000 7f516a10f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5164159900 con 0x7f5164153bf0 2023-10-30T14:20:29.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f5162ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f51640c6fe0 0x7f51640c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57524/0 (socket says 172.21.15.46:57524) 2023-10-30T14:20:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f5162ffd640 1 -- 172.21.15.46:0/3282250111 learned_addr learned my addr 172.21.15.46:0/3282250111 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:20:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51637fe640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5164153bf0 0x7f5164071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f5163fff640 1 --2- 172.21.15.46:0/3282250111 >> v2:172.21.15.154:3300/0 conn(0x7f51640c5f60 0x7f51640cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178245022 0 0) 0x7f51640700b0 con 0x7f51640c6fe0 2023-10-30T14:20:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f514c003530 con 0x7f51640c6fe0 2023-10-30T14:20:29.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1233261797 0 0) 0x7f514c003530 con 0x7f51640c6fe0 2023-10-30T14:20:29.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5164153bf0 msgr2=0x7f5164071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:29.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5164153bf0 0x7f5164071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:29.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 >> v2:172.21.15.154:3300/0 conn(0x7f51640c5f60 msgr2=0x7f51640cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:29.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 --2- 172.21.15.46:0/3282250111 >> v2:172.21.15.154:3300/0 conn(0x7f51640c5f60 0x7f51640cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:29.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51640c6530 con 0x7f51640c6fe0 2023-10-30T14:20:29.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f5158001680 con 0x7f51640c6fe0 2023-10-30T14:20:29.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 >> v1:172.21.15.46:6789/0 conn(0x7f51640c6fe0 legacy=0x7f51640c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:29.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5164153bf0 0x7f514c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:29.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f514c003f00 0x7f514c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:29.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f514c004d10 0x7f514c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:29.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51640c6810 con 0x7f514c004d10 2023-10-30T14:20:29.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5164159900 con 0x7f514c003f00 2023-10-30T14:20:29.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c00b9c0 con 0x7f5164153bf0 2023-10-30T14:20:29.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f5163fff640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f514c004d10 0x7f514c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:29.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f51637fe640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5164153bf0 0x7f514c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:29.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f5162ffd640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f514c003f00 0x7f514c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:29.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f5163fff640 1 -- 172.21.15.46:0/3282250111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5164153bf0 msgr2=0x7f514c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:29.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f5163fff640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5164153bf0 0x7f514c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:29.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f5163fff640 1 -- 172.21.15.46:0/3282250111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f514c003f00 msgr2=0x7f514c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f5163fff640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f514c003f00 0x7f514c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f5163fff640 1 -- 172.21.15.46:0/3282250111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c003530 con 0x7f514c004d10 2023-10-30T14:20:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.054+0000 7f5162ffd640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f514c003f00 0x7f514c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:29.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.058+0000 7f5163fff640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f514c004d10 0x7f514c005190 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f515c009fc0 tx=0x7f515c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9d452c7ddf29d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:29.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.058+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c0a5cb0 con 0x7f514c004d10 2023-10-30T14:20:29.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.058+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c00f040 con 0x7f514c004d10 2023-10-30T14:20:29.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.058+0000 7f51627fc640 1 -- 172.21.15.46:0/3282250111 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c0ad680 con 0x7f514c004d10 2023-10-30T14:20:29.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.058+0000 7f516a10f640 1 -- 172.21.15.46:0/3282250111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f514c004d10 msgr2=0x7f514c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:29.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.058+0000 7f516a10f640 1 --2- 172.21.15.46:0/3282250111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f514c004d10 0x7f514c005190 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f515c009fc0 tx=0x7f515c01cca0 comp rx=0 tx=0).stop 2023-10-30T14:20:29.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.058+0000 7f5163fff640 1 -- 172.21.15.46:0/3282250111 reap_dead start 2023-10-30T14:20:29.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.058+0000 7f516a10f640 1 -- 172.21.15.46:0/3282250111 shutdown_connections 2023-10-30T14:20:29.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.058+0000 7f516a10f640 1 -- 172.21.15.46:0/3282250111 >> 172.21.15.46:0/3282250111 conn(0x7f516406c6e0 msgr2=0x7f5164069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:29.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.058+0000 7f516a10f640 1 -- 172.21.15.46:0/3282250111 shutdown_connections 2023-10-30T14:20:29.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.058+0000 7f516a10f640 1 -- 172.21.15.46:0/3282250111 wait complete. 2023-10-30T14:20:29.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f516a10f640 1 Processor -- start 2023-10-30T14:20:29.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f516a10f640 1 -- start start 2023-10-30T14:20:29.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f516a10f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f514c004d10 0x7f516414bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:29.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f516a10f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51640c43f0 0x7f516414c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:29.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f516a10f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51641f3020 0x7f51641f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:29.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f516a10f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51640c41f0 con 0x7f51641f3020 2023-10-30T14:20:29.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f516a10f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51641f71d0 con 0x7f51640c43f0 2023-10-30T14:20:29.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f516a10f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51641f6630 con 0x7f514c004d10 2023-10-30T14:20:29.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f5163fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51641f3020 0x7f51641f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:29.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f5163fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51641f3020 0x7f51641f34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52952/0 (socket says 172.21.15.46:52952) 2023-10-30T14:20:29.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f5163fff640 1 -- 172.21.15.46:0/3440260220 learned_addr learned my addr 172.21.15.46:0/3440260220 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:29.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f5162ffd640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51640c43f0 0x7f516414c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:29.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f51637fe640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f514c004d10 0x7f516414bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:29.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f5163fff640 1 -- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f514c004d10 msgr2=0x7f516414bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:29.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f5163fff640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f514c004d10 0x7f516414bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:29.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f5163fff640 1 -- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51640c43f0 msgr2=0x7f516414c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f5163fff640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51640c43f0 0x7f516414c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f5163fff640 1 -- 172.21.15.46:0/3440260220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51641f39e0 con 0x7f51641f3020 2023-10-30T14:20:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f5163fff640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51641f3020 0x7f51641f34a0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f515c002240 tx=0x7f515c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f514bfff640 1 -- 172.21.15.46:0/3440260220 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c0039a0 con 0x7f51641f3020 2023-10-30T14:20:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f514bfff640 1 -- 172.21.15.46:0/3440260220 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c003b00 con 0x7f51641f3020 2023-10-30T14:20:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f516a10f640 1 -- 172.21.15.46:0/3440260220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51640c4970 con 0x7f51641f3020 2023-10-30T14:20:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f514bfff640 1 -- 172.21.15.46:0/3440260220 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c003e70 con 0x7f51641f3020 2023-10-30T14:20:29.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.062+0000 7f516a10f640 1 -- 172.21.15.46:0/3440260220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51641f6b80 con 0x7f51641f3020 2023-10-30T14:20:29.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.066+0000 7f516a10f640 1 -- 172.21.15.46:0/3440260220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516404e170 con 0x7f51641f3020 2023-10-30T14:20:29.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.066+0000 7f514bfff640 1 -- 172.21.15.46:0/3440260220 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f515c0b4040 con 0x7f51641f3020 2023-10-30T14:20:29.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.066+0000 7f514bfff640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51500664a0 0x7f5150068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:29.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.066+0000 7f514bfff640 1 -- 172.21.15.46:0/3440260220 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f515c11e970 con 0x7f51641f3020 2023-10-30T14:20:29.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.066+0000 7f51637fe640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51500664a0 0x7f5150068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:29.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.070+0000 7f51637fe640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51500664a0 0x7f5150068960 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f5154009b20 tx=0x7f5154008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:29.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.070+0000 7f514bfff640 1 -- 172.21.15.46:0/3440260220 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f515c0eb310 con 0x7f51641f3020 2023-10-30T14:20:29.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 -- 172.21.15.46:0/3440260220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51641f7310 con 0x7f51641f3020 2023-10-30T14:20:29.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f514bfff640 1 -- 172.21.15.46:0/3440260220 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f515c0b2020 con 0x7f51641f3020 2023-10-30T14:20:29.265 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:29.265 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9594,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:23.041684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:29.265 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 238 2023-10-30T14:20:29.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 -- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51500664a0 msgr2=0x7f5150068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:29.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51500664a0 0x7f5150068960 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f5154009b20 tx=0x7f5154008040 comp rx=0 tx=0).stop 2023-10-30T14:20:29.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 -- 172.21.15.46:0/3440260220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51641f3020 msgr2=0x7f51641f34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:29.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51641f3020 0x7f51641f34a0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f515c002240 tx=0x7f515c009750 comp rx=0 tx=0).stop 2023-10-30T14:20:29.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 -- 172.21.15.46:0/3440260220 shutdown_connections 2023-10-30T14:20:29.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51500664a0 0x7f5150068960 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:29.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51641f3020 0x7f51641f34a0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:29.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51640c43f0 0x7f516414c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:29.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 --2- 172.21.15.46:0/3440260220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f514c004d10 0x7f516414bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:29.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 -- 172.21.15.46:0/3440260220 >> 172.21.15.46:0/3440260220 conn(0x7f516406c6e0 msgr2=0x7f5164158f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:29.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 -- 172.21.15.46:0/3440260220 shutdown_connections 2023-10-30T14:20:29.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:29.262+0000 7f516a10f640 1 -- 172.21.15.46:0/3440260220 wait complete. 2023-10-30T14:20:29.286 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:20:30.281 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:20:30.286 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:20:31.288 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:31.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa89d86b640 1 Processor -- start 2023-10-30T14:20:31.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa89d86b640 1 -- start start 2023-10-30T14:20:31.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa89d86b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa89815d6f0 0x7fa89815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:31.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa89d86b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8980cb800 con 0x7fa8981538a0 2023-10-30T14:20:31.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa89d86b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8980cb330 con 0x7fa8980cf160 2023-10-30T14:20:31.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa89d86b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8980cad70 con 0x7fa89815d6f0 2023-10-30T14:20:31.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa896ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa8981538a0 0x7fa8980ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57528/0 (socket says 172.21.15.46:57528) 2023-10-30T14:20:31.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa896ffd640 1 -- 172.21.15.46:0/3413179475 learned_addr learned my addr 172.21.15.46:0/3413179475 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:20:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa88ffff640 1 --2- 172.21.15.46:0/3413179475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa89815d6f0 0x7fa89815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291449790 0 0) 0x7fa8980cb800 con 0x7fa8981538a0 2023-10-30T14:20:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa884003530 con 0x7fa8981538a0 2023-10-30T14:20:31.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238928947 0 0) 0x7fa8980cb330 con 0x7fa8980cf160 2023-10-30T14:20:31.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8980cb800 con 0x7fa8980cf160 2023-10-30T14:20:31.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1321023548 0 0) 0x7fa884003530 con 0x7fa8981538a0 2023-10-30T14:20:31.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa89815d6f0 msgr2=0x7fa89815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:31.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 --2- 172.21.15.46:0/3413179475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa89815d6f0 0x7fa89815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:31.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 >> v1:172.21.15.154:6789/0 conn(0x7fa8980cf160 legacy=0x7fa898164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8980cb050 con 0x7fa8981538a0 2023-10-30T14:20:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fa888001680 con 0x7fa8981538a0 2023-10-30T14:20:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 >> v1:172.21.15.46:6789/0 conn(0x7fa8981538a0 legacy=0x7fa8980ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:31.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 --2- 172.21.15.46:0/3413179475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa884003b40 0x7fa884003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:31.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 --2- 172.21.15.46:0/3413179475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa89815d6f0 0x7fa8840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:31.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.446+0000 7fa8967fc640 1 --2- 172.21.15.46:0/3413179475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa884004d10 0x7fa884005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:31.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8980cad70 con 0x7fa884004d10 2023-10-30T14:20:31.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88400ba70 con 0x7fa884003b40 2023-10-30T14:20:31.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88400bc30 con 0x7fa89815d6f0 2023-10-30T14:20:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8977fe640 1 --2- 172.21.15.46:0/3413179475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa884004d10 0x7fa884005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa896ffd640 1 --2- 172.21.15.46:0/3413179475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa884003b40 0x7fa884003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa88ffff640 1 --2- 172.21.15.46:0/3413179475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa89815d6f0 0x7fa8840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8977fe640 1 -- 172.21.15.46:0/3413179475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa89815d6f0 msgr2=0x7fa8840047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8977fe640 1 --2- 172.21.15.46:0/3413179475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa89815d6f0 0x7fa8840047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8977fe640 1 -- 172.21.15.46:0/3413179475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa884003b40 msgr2=0x7fa884003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8977fe640 1 --2- 172.21.15.46:0/3413179475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa884003b40 0x7fa884003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8977fe640 1 -- 172.21.15.46:0/3413179475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa884003530 con 0x7fa884004d10 2023-10-30T14:20:31.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8977fe640 1 --2- 172.21.15.46:0/3413179475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa884004d10 0x7fa884005190 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fa890001fd0 tx=0x7fa89001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a570d3808f1e0e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:31.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890003b80 con 0x7fa884004d10 2023-10-30T14:20:31.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa890003d00 con 0x7fa884004d10 2023-10-30T14:20:31.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.450+0000 7fa8967fc640 1 -- 172.21.15.46:0/3413179475 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890005310 con 0x7fa884004d10 2023-10-30T14:20:31.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 -- 172.21.15.46:0/3413179475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa884004d10 msgr2=0x7fa884005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:31.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 --2- 172.21.15.46:0/3413179475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa884004d10 0x7fa884005190 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fa890001fd0 tx=0x7fa89001c8e0 comp rx=0 tx=0).stop 2023-10-30T14:20:31.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa8977fe640 1 -- 172.21.15.46:0/3413179475 reap_dead start 2023-10-30T14:20:31.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 -- 172.21.15.46:0/3413179475 shutdown_connections 2023-10-30T14:20:31.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 -- 172.21.15.46:0/3413179475 >> 172.21.15.46:0/3413179475 conn(0x7fa89806bfe0 msgr2=0x7fa89806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:31.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 -- 172.21.15.46:0/3413179475 shutdown_connections 2023-10-30T14:20:31.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 -- 172.21.15.46:0/3413179475 wait complete. 2023-10-30T14:20:31.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 Processor -- start 2023-10-30T14:20:31.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 -- start start 2023-10-30T14:20:31.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa884004d10 0x7fa8981ef930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:31.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8980cd950 0x7fa898149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:31.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8981f2450 0x7fa8981f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:31.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8980ce150 con 0x7fa884004d10 2023-10-30T14:20:31.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8981efe70 con 0x7fa8981f2450 2023-10-30T14:20:31.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa89d86b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8980ce3a0 con 0x7fa8980cd950 2023-10-30T14:20:31.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa896ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa884004d10 0x7fa8981ef930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:31.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa896ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa884004d10 0x7fa8981ef930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52978/0 (socket says 172.21.15.46:52978) 2023-10-30T14:20:31.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa896ffd640 1 -- 172.21.15.46:0/3342771229 learned_addr learned my addr 172.21.15.46:0/3342771229 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:31.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa88ffff640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8980cd950 0x7fa898149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:31.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa8977fe640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8981f2450 0x7fa8981f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:31.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa896ffd640 1 -- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8980cd950 msgr2=0x7fa898149160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:31.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa896ffd640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8980cd950 0x7fa898149160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:31.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa896ffd640 1 -- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8981f2450 msgr2=0x7fa8981f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:31.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa896ffd640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8981f2450 0x7fa8981f28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:31.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.454+0000 7fa896ffd640 1 -- 172.21.15.46:0/3342771229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8980ce680 con 0x7fa884004d10 2023-10-30T14:20:31.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.458+0000 7fa896ffd640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa884004d10 0x7fa8981ef930 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fa8880072c0 tx=0x7fa888007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:31.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.458+0000 7fa894ff9640 1 -- 172.21.15.46:0/3342771229 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88800f9c0 con 0x7fa884004d10 2023-10-30T14:20:31.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.458+0000 7fa894ff9640 1 -- 172.21.15.46:0/3342771229 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88800fb60 con 0x7fa884004d10 2023-10-30T14:20:31.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.458+0000 7fa894ff9640 1 -- 172.21.15.46:0/3342771229 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8880133f0 con 0x7fa884004d10 2023-10-30T14:20:31.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.458+0000 7fa89d86b640 1 -- 172.21.15.46:0/3342771229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8980ce960 con 0x7fa884004d10 2023-10-30T14:20:31.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.458+0000 7fa89d86b640 1 -- 172.21.15.46:0/3342771229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8981496a0 con 0x7fa884004d10 2023-10-30T14:20:31.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.458+0000 7fa89d86b640 1 -- 172.21.15.46:0/3342771229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8980ccfb0 con 0x7fa884004d10 2023-10-30T14:20:31.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.458+0000 7fa894ff9640 1 -- 172.21.15.46:0/3342771229 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa888013550 con 0x7fa884004d10 2023-10-30T14:20:31.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.458+0000 7fa894ff9640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8780664d0 0x7fa878068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:31.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.458+0000 7fa894ff9640 1 -- 172.21.15.46:0/3342771229 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7fa888084c30 con 0x7fa884004d10 2023-10-30T14:20:31.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.458+0000 7fa88ffff640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8780664d0 0x7fa878068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:31.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.462+0000 7fa88ffff640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8780664d0 0x7fa878068990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa898152900 tx=0x7fa880008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:31.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.462+0000 7fa894ff9640 1 -- 172.21.15.46:0/3342771229 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa8980ccfb0 con 0x7fa884004d10 2023-10-30T14:20:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 -- 172.21.15.46:0/3342771229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8980cd320 con 0x7fa884004d10 2023-10-30T14:20:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa894ff9640 1 -- 172.21.15.46:0/3342771229 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fa8980cd320 con 0x7fa884004d10 2023-10-30T14:20:31.657 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:31.658 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9594,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:23.041684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:31.658 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 238 2023-10-30T14:20:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 -- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8780664d0 msgr2=0x7fa878068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8780664d0 0x7fa878068990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa898152900 tx=0x7fa880008040 comp rx=0 tx=0).stop 2023-10-30T14:20:31.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 -- 172.21.15.46:0/3342771229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa884004d10 msgr2=0x7fa8981ef930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:31.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa884004d10 0x7fa8981ef930 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fa8880072c0 tx=0x7fa888007780 comp rx=0 tx=0).stop 2023-10-30T14:20:31.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 -- 172.21.15.46:0/3342771229 shutdown_connections 2023-10-30T14:20:31.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8780664d0 0x7fa878068990 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:31.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8981f2450 0x7fa8981f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8980cd950 0x7fa898149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 --2- 172.21.15.46:0/3342771229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa884004d10 0x7fa8981ef930 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 -- 172.21.15.46:0/3342771229 >> 172.21.15.46:0/3342771229 conn(0x7fa89806bfe0 msgr2=0x7fa898151500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 -- 172.21.15.46:0/3342771229 shutdown_connections 2023-10-30T14:20:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:31.654+0000 7fa89d86b640 1 -- 172.21.15.46:0/3342771229 wait complete. 2023-10-30T14:20:31.679 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:20:33.680 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.830+0000 7fe2a8b65640 1 Processor -- start 2023-10-30T14:20:33.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.830+0000 7fe2a8b65640 1 -- start start 2023-10-30T14:20:33.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.830+0000 7fe2a8b65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2a4153bf0 0x7fe2a4071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:33.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.830+0000 7fe2a8b65640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe2a40c5f60 0x7fe2a40cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:33.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a8b65640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a40700b0 con 0x7fe2a40c6fe0 2023-10-30T14:20:33.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a8b65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a40c6810 con 0x7fe2a4153bf0 2023-10-30T14:20:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a8b65640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a4159900 con 0x7fe2a40c5f60 2023-10-30T14:20:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2a4153bf0 0x7fe2a4071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2a4153bf0 0x7fe2a4071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52990/0 (socket says 172.21.15.46:52990) 2023-10-30T14:20:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a259c640 1 -- 172.21.15.46:0/1826444317 learned_addr learned my addr 172.21.15.46:0/1826444317 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a2d9d640 1 --2- 172.21.15.46:0/1826444317 >> v2:172.21.15.154:3300/0 conn(0x7fe2a40c5f60 0x7fe2a40cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a259c640 1 -- 172.21.15.46:0/1826444317 >> v2:172.21.15.154:3300/0 conn(0x7fe2a40c5f60 msgr2=0x7fe2a40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:33.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a259c640 1 --2- 172.21.15.46:0/1826444317 >> v2:172.21.15.154:3300/0 conn(0x7fe2a40c5f60 0x7fe2a40cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:33.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a259c640 1 -- 172.21.15.46:0/1826444317 >> v1:172.21.15.46:6789/0 conn(0x7fe2a40c6fe0 legacy=0x7fe2a40c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:33.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a259c640 1 -- 172.21.15.46:0/1826444317 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a40c6530 con 0x7fe2a4153bf0 2023-10-30T14:20:33.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a259c640 1 --2- 172.21.15.46:0/1826444317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a4153bf0 0x7fe2a4071260 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fe2a406ce90 tx=0x7fe29401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b86bd3d27caf33b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:33.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a1d9b640 1 -- 172.21.15.46:0/1826444317 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2940a89f0 con 0x7fe2a4153bf0 2023-10-30T14:20:33.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a1d9b640 1 -- 172.21.15.46:0/1826444317 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2940a8b50 con 0x7fe2a4153bf0 2023-10-30T14:20:33.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.834+0000 7fe2a1d9b640 1 -- 172.21.15.46:0/1826444317 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2940b0440 con 0x7fe2a4153bf0 2023-10-30T14:20:33.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 -- 172.21.15.46:0/1826444317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a4153bf0 msgr2=0x7fe2a4071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:33.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 --2- 172.21.15.46:0/1826444317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a4153bf0 0x7fe2a4071260 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fe2a406ce90 tx=0x7fe29401ecf0 comp rx=0 tx=0).stop 2023-10-30T14:20:33.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 -- 172.21.15.46:0/1826444317 shutdown_connections 2023-10-30T14:20:33.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 --2- 172.21.15.46:0/1826444317 >> v2:172.21.15.154:3300/0 conn(0x7fe2a40c5f60 0x7fe2a40cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:33.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 --2- 172.21.15.46:0/1826444317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a4153bf0 0x7fe2a4071260 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:33.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 -- 172.21.15.46:0/1826444317 >> 172.21.15.46:0/1826444317 conn(0x7fe2a406c6e0 msgr2=0x7fe2a4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:33.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 -- 172.21.15.46:0/1826444317 shutdown_connections 2023-10-30T14:20:33.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 -- 172.21.15.46:0/1826444317 wait complete. 2023-10-30T14:20:33.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 Processor -- start 2023-10-30T14:20:33.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 -- start start 2023-10-30T14:20:33.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a40c5f60 0x7fe2a414dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:33.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2a40c6fe0 0x7fe2a40c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:33.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2a4153bf0 0x7fe2a41eee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:33.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a414e810 con 0x7fe2a40c5f60 2023-10-30T14:20:33.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a414e570 con 0x7fe2a4153bf0 2023-10-30T14:20:33.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a8b65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a41f32c0 con 0x7fe2a40c6fe0 2023-10-30T14:20:33.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a40c5f60 0x7fe2a414dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:33.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a40c5f60 0x7fe2a414dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52994/0 (socket says 172.21.15.46:52994) 2023-10-30T14:20:33.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a259c640 1 -- 172.21.15.46:0/2376967535 learned_addr learned my addr 172.21.15.46:0/2376967535 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:33.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a2d9d640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2a4153bf0 0x7fe2a41eee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:33.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe29bfff640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2a40c6fe0 0x7fe2a40c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:33.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a259c640 1 -- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2a40c6fe0 msgr2=0x7fe2a40c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:33.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a259c640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2a40c6fe0 0x7fe2a40c7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:33.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a259c640 1 -- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2a4153bf0 msgr2=0x7fe2a41eee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:33.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a259c640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2a4153bf0 0x7fe2a41eee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:33.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.838+0000 7fe2a259c640 1 -- 172.21.15.46:0/2376967535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2940094d0 con 0x7fe2a40c5f60 2023-10-30T14:20:33.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.842+0000 7fe2a259c640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a40c5f60 0x7fe2a414dea0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fe29401b900 tx=0x7fe29401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:33.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.842+0000 7fe29affd640 1 -- 172.21.15.46:0/2376967535 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2940a89f0 con 0x7fe2a40c5f60 2023-10-30T14:20:33.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.842+0000 7fe29affd640 1 -- 172.21.15.46:0/2376967535 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2940b9410 con 0x7fe2a40c5f60 2023-10-30T14:20:33.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.842+0000 7fe29affd640 1 -- 172.21.15.46:0/2376967535 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2940b97c0 con 0x7fe2a40c5f60 2023-10-30T14:20:33.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.842+0000 7fe2a8b65640 1 -- 172.21.15.46:0/2376967535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a41f2500 con 0x7fe2a40c5f60 2023-10-30T14:20:33.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.842+0000 7fe2a8b65640 1 -- 172.21.15.46:0/2376967535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a41f2ac0 con 0x7fe2a40c5f60 2023-10-30T14:20:33.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.842+0000 7fe2a8b65640 1 -- 172.21.15.46:0/2376967535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a404e170 con 0x7fe2a40c5f60 2023-10-30T14:20:33.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.842+0000 7fe29affd640 1 -- 172.21.15.46:0/2376967535 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe2940b7040 con 0x7fe2a40c5f60 2023-10-30T14:20:33.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.842+0000 7fe29affd640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe2840664a0 0x7fe284068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:33.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.842+0000 7fe29affd640 1 -- 172.21.15.46:0/2376967535 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7fe294122430 con 0x7fe2a40c5f60 2023-10-30T14:20:33.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.846+0000 7fe29bfff640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe2840664a0 0x7fe284068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.846+0000 7fe29bfff640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe2840664a0 0x7fe284068960 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe2a414ddc0 tx=0x7fe28c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:33.850+0000 7fe29affd640 1 -- 172.21.15.46:0/2376967535 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe2940eee40 con 0x7fe2a40c5f60 2023-10-30T14:20:34.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.034+0000 7fe2a8b65640 1 -- 172.21.15.46:0/2376967535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2a414d8a0 con 0x7fe2a40c5f60 2023-10-30T14:20:34.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.034+0000 7fe29affd640 1 -- 172.21.15.46:0/2376967535 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fe2940b5070 con 0x7fe2a40c5f60 2023-10-30T14:20:34.039 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:34.039 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9594,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:23.041684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:34.040 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 238 2023-10-30T14:20:34.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.038+0000 7fe2a8b65640 1 -- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe2840664a0 msgr2=0x7fe284068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:34.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.038+0000 7fe2a8b65640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe2840664a0 0x7fe284068960 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe2a414ddc0 tx=0x7fe28c009040 comp rx=0 tx=0).stop 2023-10-30T14:20:34.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.038+0000 7fe2a8b65640 1 -- 172.21.15.46:0/2376967535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a40c5f60 msgr2=0x7fe2a414dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:34.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.038+0000 7fe2a8b65640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a40c5f60 0x7fe2a414dea0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fe29401b900 tx=0x7fe29401ba90 comp rx=0 tx=0).stop 2023-10-30T14:20:34.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.038+0000 7fe2a8b65640 1 -- 172.21.15.46:0/2376967535 shutdown_connections 2023-10-30T14:20:34.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.038+0000 7fe2a8b65640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe2840664a0 0x7fe284068960 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:34.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.038+0000 7fe2a8b65640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2a4153bf0 0x7fe2a41eee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:34.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.038+0000 7fe2a8b65640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2a40c6fe0 0x7fe2a40c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:34.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.038+0000 7fe2a8b65640 1 --2- 172.21.15.46:0/2376967535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2a40c5f60 0x7fe2a414dea0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:34.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.038+0000 7fe2a8b65640 1 -- 172.21.15.46:0/2376967535 >> 172.21.15.46:0/2376967535 conn(0x7fe2a406c6e0 msgr2=0x7fe2a4159110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:34.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.038+0000 7fe2a8b65640 1 -- 172.21.15.46:0/2376967535 shutdown_connections 2023-10-30T14:20:34.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:34.038+0000 7fe2a8b65640 1 -- 172.21.15.46:0/2376967535 wait complete. 2023-10-30T14:20:34.061 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:20:36.062 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:36.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb032bf5640 1 Processor -- start 2023-10-30T14:20:36.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb032bf5640 1 -- start start 2023-10-30T14:20:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb032bf5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb02c1538a0 0x7fb02c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb032bf5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb02c0ca0e0 0x7fb02c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb032bf5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02c0cb800 con 0x7fb02c0cf160 2023-10-30T14:20:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb032bf5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02c15dc30 con 0x7fb02c0ca0e0 2023-10-30T14:20:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb032bf5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02c0cb330 con 0x7fb02c1538a0 2023-10-30T14:20:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb02bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb02c0ca0e0 0x7fb02c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb02bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb02c0ca0e0 0x7fb02c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53000/0 (socket says 172.21.15.46:53000) 2023-10-30T14:20:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb02bfff640 1 -- 172.21.15.46:0/3455616438 learned_addr learned my addr 172.21.15.46:0/3455616438 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb030991640 1 --2- 172.21.15.46:0/3455616438 >> v2:172.21.15.154:3300/0 conn(0x7fb02c1538a0 0x7fb02c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:36.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb02bfff640 1 -- 172.21.15.46:0/3455616438 >> v2:172.21.15.154:3300/0 conn(0x7fb02c1538a0 msgr2=0x7fb02c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:36.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb02bfff640 1 --2- 172.21.15.46:0/3455616438 >> v2:172.21.15.154:3300/0 conn(0x7fb02c1538a0 0x7fb02c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:36.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb02bfff640 1 -- 172.21.15.46:0/3455616438 >> v1:172.21.15.154:6789/0 conn(0x7fb02c0cf160 legacy=0x7fb02c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:36.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.214+0000 7fb02bfff640 1 -- 172.21.15.46:0/3455616438 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c0cad70 con 0x7fb02c0ca0e0 2023-10-30T14:20:36.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.218+0000 7fb02bfff640 1 --2- 172.21.15.46:0/3455616438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c0ca0e0 0x7fb02c15a090 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0cfd10 tx=0x7fb01801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b67e02db3bfcc11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:36.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.218+0000 7fb02b7fe640 1 -- 172.21.15.46:0/3455616438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0180a89f0 con 0x7fb02c0ca0e0 2023-10-30T14:20:36.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.218+0000 7fb02b7fe640 1 -- 172.21.15.46:0/3455616438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0180a8b50 con 0x7fb02c0ca0e0 2023-10-30T14:20:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.218+0000 7fb02b7fe640 1 -- 172.21.15.46:0/3455616438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0180b0440 con 0x7fb02c0ca0e0 2023-10-30T14:20:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.218+0000 7fb032bf5640 1 -- 172.21.15.46:0/3455616438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c0ca0e0 msgr2=0x7fb02c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.218+0000 7fb032bf5640 1 --2- 172.21.15.46:0/3455616438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c0ca0e0 0x7fb02c15a090 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0cfd10 tx=0x7fb01801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:20:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.218+0000 7fb032bf5640 1 -- 172.21.15.46:0/3455616438 shutdown_connections 2023-10-30T14:20:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.218+0000 7fb032bf5640 1 --2- 172.21.15.46:0/3455616438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c0ca0e0 0x7fb02c15a090 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.218+0000 7fb032bf5640 1 --2- 172.21.15.46:0/3455616438 >> v2:172.21.15.154:3300/0 conn(0x7fb02c1538a0 0x7fb02c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:36.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.218+0000 7fb032bf5640 1 -- 172.21.15.46:0/3455616438 >> 172.21.15.46:0/3455616438 conn(0x7fb02c06bfe0 msgr2=0x7fb02c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:36.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.218+0000 7fb032bf5640 1 -- 172.21.15.46:0/3455616438 shutdown_connections 2023-10-30T14:20:36.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.218+0000 7fb032bf5640 1 -- 172.21.15.46:0/3455616438 wait complete. 2023-10-30T14:20:36.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb032bf5640 1 Processor -- start 2023-10-30T14:20:36.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb032bf5640 1 -- start start 2023-10-30T14:20:36.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb032bf5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb02c0ca0e0 0x7fb02c158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:36.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb032bf5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb02c0cf160 0x7fb02c0ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:36.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb032bf5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1538a0 0x7fb02c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:36.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb032bf5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c158210 con 0x7fb02c1538a0 2023-10-30T14:20:36.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb032bf5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0cd7b0 con 0x7fb02c0ca0e0 2023-10-30T14:20:36.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb032bf5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0cda50 con 0x7fb02c0cf160 2023-10-30T14:20:36.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb031192640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1538a0 0x7fb02c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:36.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb031192640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1538a0 0x7fb02c0ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53004/0 (socket says 172.21.15.46:53004) 2023-10-30T14:20:36.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb031192640 1 -- 172.21.15.46:0/3186460142 learned_addr learned my addr 172.21.15.46:0/3186460142 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:36.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb030991640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb02c0ca0e0 0x7fb02c158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:36.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb02bfff640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb02c0cf160 0x7fb02c0ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:36.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb031192640 1 -- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb02c0cf160 msgr2=0x7fb02c0ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb031192640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb02c0cf160 0x7fb02c0ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb031192640 1 -- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb02c0ca0e0 msgr2=0x7fb02c158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb031192640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb02c0ca0e0 0x7fb02c158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb031192640 1 -- 172.21.15.46:0/3186460142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180094d0 con 0x7fb02c1538a0 2023-10-30T14:20:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb02bfff640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb02c0cf160 0x7fb02c0ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb030991640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb02c0ca0e0 0x7fb02c158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb031192640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1538a0 0x7fb02c0ce700 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fb02400c6c0 tx=0x7fb02400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb0297fa640 1 -- 172.21.15.46:0/3186460142 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024003b80 con 0x7fb02c1538a0 2023-10-30T14:20:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb0297fa640 1 -- 172.21.15.46:0/3186460142 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb024003ce0 con 0x7fb02c1538a0 2023-10-30T14:20:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb0297fa640 1 -- 172.21.15.46:0/3186460142 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0240052f0 con 0x7fb02c1538a0 2023-10-30T14:20:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb032bf5640 1 -- 172.21.15.46:0/3186460142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02c1f0020 con 0x7fb02c1538a0 2023-10-30T14:20:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.222+0000 7fb032bf5640 1 -- 172.21.15.46:0/3186460142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02c158460 con 0x7fb02c1538a0 2023-10-30T14:20:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.226+0000 7fb032bf5640 1 -- 172.21.15.46:0/3186460142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02c0b2a50 con 0x7fb02c1538a0 2023-10-30T14:20:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.226+0000 7fb0297fa640 1 -- 172.21.15.46:0/3186460142 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb024005450 con 0x7fb02c1538a0 2023-10-30T14:20:36.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.226+0000 7fb0297fa640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0100664a0 0x7fb010068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:36.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.226+0000 7fb0297fa640 1 -- 172.21.15.46:0/3186460142 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7fb0240847c0 con 0x7fb02c1538a0 2023-10-30T14:20:36.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.226+0000 7fb030991640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0100664a0 0x7fb010068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:36.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.230+0000 7fb030991640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0100664a0 0x7fb010068960 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fb02000b9f0 tx=0x7fb020008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.230+0000 7fb0297fa640 1 -- 172.21.15.46:0/3186460142 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb0240510b0 con 0x7fb02c1538a0 2023-10-30T14:20:36.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.422+0000 7fb032bf5640 1 -- 172.21.15.46:0/3186460142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb02c159860 con 0x7fb02c1538a0 2023-10-30T14:20:36.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.422+0000 7fb0297fa640 1 -- 172.21.15.46:0/3186460142 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fb0240546d0 con 0x7fb02c1538a0 2023-10-30T14:20:36.427 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:36.427 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9594,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:23.041684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:36.427 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 238 2023-10-30T14:20:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.426+0000 7fb032bf5640 1 -- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0100664a0 msgr2=0x7fb010068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.426+0000 7fb032bf5640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0100664a0 0x7fb010068960 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fb02000b9f0 tx=0x7fb020008040 comp rx=0 tx=0).stop 2023-10-30T14:20:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.426+0000 7fb032bf5640 1 -- 172.21.15.46:0/3186460142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1538a0 msgr2=0x7fb02c0ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.426+0000 7fb032bf5640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1538a0 0x7fb02c0ce700 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fb02400c6c0 tx=0x7fb02400cb80 comp rx=0 tx=0).stop 2023-10-30T14:20:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.426+0000 7fb032bf5640 1 -- 172.21.15.46:0/3186460142 shutdown_connections 2023-10-30T14:20:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.426+0000 7fb032bf5640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb0100664a0 0x7fb010068960 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.426+0000 7fb032bf5640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1538a0 0x7fb02c0ce700 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.426+0000 7fb032bf5640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb02c0cf160 0x7fb02c0ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.426+0000 7fb032bf5640 1 --2- 172.21.15.46:0/3186460142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb02c0ca0e0 0x7fb02c158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.426+0000 7fb032bf5640 1 -- 172.21.15.46:0/3186460142 >> 172.21.15.46:0/3186460142 conn(0x7fb02c06bfe0 msgr2=0x7fb02c152270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.426+0000 7fb032bf5640 1 -- 172.21.15.46:0/3186460142 shutdown_connections 2023-10-30T14:20:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:36.426+0000 7fb032bf5640 1 -- 172.21.15.46:0/3186460142 wait complete. 2023-10-30T14:20:36.448 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:20:38.449 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:38.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5ce151d640 1 Processor -- start 2023-10-30T14:20:38.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5ce151d640 1 -- start start 2023-10-30T14:20:38.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5ce151d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cdc151800 0x7f5cdc06e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5ce151d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5cdc0c1ee0 0x7f5cdc0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5ce151d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0c9230 0x7f5cdc0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5ce151d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cdc06dc70 con 0x7f5cdc151800 2023-10-30T14:20:38.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5ce151d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cdc06e090 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5ce151d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc0c8a50 con 0x7f5cdc0c9230 2023-10-30T14:20:38.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5cdaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cdc151800 0x7f5cdc06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5cdaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cdc151800 0x7f5cdc06e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39420/0 (socket says 172.21.15.46:39420) 2023-10-30T14:20:38.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5cdaffd640 1 -- 172.21.15.46:0/2153263073 learned_addr learned my addr 172.21.15.46:0/2153263073 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5cdb7fe640 1 --2- 172.21.15.46:0/2153263073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0c9230 0x7f5cdc0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5cd3fff640 1 --2- 172.21.15.46:0/2153263073 >> v2:172.21.15.154:3300/0 conn(0x7f5cdc0c1ee0 0x7f5cdc0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5cdaffd640 1 -- 172.21.15.46:0/2153263073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0c9230 msgr2=0x7f5cdc0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5cdaffd640 1 --2- 172.21.15.46:0/2153263073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0c9230 0x7f5cdc0cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5cdaffd640 1 -- 172.21.15.46:0/2153263073 >> v2:172.21.15.154:3300/0 conn(0x7f5cdc0c1ee0 msgr2=0x7f5cdc0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:38.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5cdaffd640 1 --2- 172.21.15.46:0/2153263073 >> v2:172.21.15.154:3300/0 conn(0x7f5cdc0c1ee0 0x7f5cdc0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5cdaffd640 1 -- 172.21.15.46:0/2153263073 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc0c3b10 con 0x7f5cdc151800 2023-10-30T14:20:38.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5cd3fff640 1 --2- 172.21.15.46:0/2153263073 >> v2:172.21.15.154:3300/0 conn(0x7f5cdc0c1ee0 0x7f5cdc0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:38.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.601+0000 7f5cdb7fe640 1 --2- 172.21.15.46:0/2153263073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0c9230 0x7f5cdc0cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5cdaffd640 1 --2- 172.21.15.46:0/2153263073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cdc151800 0x7f5cdc06e470 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc06e9b0 tx=0x7f5ccc01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f1f73171728452 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5cda7fc640 1 -- 172.21.15.46:0/2153263073 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc0aa5f0 con 0x7f5cdc151800 2023-10-30T14:20:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5cda7fc640 1 -- 172.21.15.46:0/2153263073 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc0aa750 con 0x7f5cdc151800 2023-10-30T14:20:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5cda7fc640 1 -- 172.21.15.46:0/2153263073 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc0aaae0 con 0x7f5cdc151800 2023-10-30T14:20:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5ce151d640 1 -- 172.21.15.46:0/2153263073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cdc151800 msgr2=0x7f5cdc06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:38.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5ce151d640 1 --2- 172.21.15.46:0/2153263073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cdc151800 0x7f5cdc06e470 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc06e9b0 tx=0x7f5ccc01c4d0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5ce151d640 1 -- 172.21.15.46:0/2153263073 shutdown_connections 2023-10-30T14:20:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5ce151d640 1 --2- 172.21.15.46:0/2153263073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0c9230 0x7f5cdc0cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5ce151d640 1 --2- 172.21.15.46:0/2153263073 >> v2:172.21.15.154:3300/0 conn(0x7f5cdc0c1ee0 0x7f5cdc0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5ce151d640 1 --2- 172.21.15.46:0/2153263073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cdc151800 0x7f5cdc06e470 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5ce151d640 1 -- 172.21.15.46:0/2153263073 >> 172.21.15.46:0/2153263073 conn(0x7f5cdc071ea0 msgr2=0x7f5cdc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5ce151d640 1 -- 172.21.15.46:0/2153263073 shutdown_connections 2023-10-30T14:20:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.605+0000 7f5ce151d640 1 -- 172.21.15.46:0/2153263073 wait complete. 2023-10-30T14:20:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5ce151d640 1 Processor -- start 2023-10-30T14:20:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5ce151d640 1 -- start start 2023-10-30T14:20:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5ce151d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cdc0c1ee0 0x7f5cdc143d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5ce151d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0c9230 0x7f5cdc0c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5ce151d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cdc151800 0x7f5cdc1f0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5ce151d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc14cc70 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5ce151d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc1f0960 con 0x7f5cdc151800 2023-10-30T14:20:38.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5ce151d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc14bb50 con 0x7f5cdc0c9230 2023-10-30T14:20:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cdaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cdc0c1ee0 0x7f5cdc143d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cdaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cdc0c1ee0 0x7f5cdc143d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39428/0 (socket says 172.21.15.46:39428) 2023-10-30T14:20:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cdaffd640 1 -- 172.21.15.46:0/3148979495 learned_addr learned my addr 172.21.15.46:0/3148979495 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cd3fff640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0c9230 0x7f5cdc0c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cdb7fe640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cdc151800 0x7f5cdc1f0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cdaffd640 1 -- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0c9230 msgr2=0x7f5cdc0c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cdaffd640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0c9230 0x7f5cdc0c7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cdaffd640 1 -- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cdc151800 msgr2=0x7f5cdc1f0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cdaffd640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cdc151800 0x7f5cdc1f0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cdaffd640 1 -- 172.21.15.46:0/3148979495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc0094d0 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cdaffd640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cdc0c1ee0 0x7f5cdc143d10 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc01ab80 tx=0x7f5ccc01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cd37fe640 1 -- 172.21.15.46:0/3148979495 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc0b3800 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cd37fe640 1 -- 172.21.15.46:0/3148979495 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc0b3980 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5cd37fe640 1 -- 172.21.15.46:0/3148979495 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc0b3d30 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5ce151d640 1 -- 172.21.15.46:0/3148979495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cdc14be30 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5ce151d640 1 -- 172.21.15.46:0/3148979495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc14c3f0 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.609+0000 7f5ce151d640 1 -- 172.21.15.46:0/3148979495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cdc04e170 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.613+0000 7f5cd37fe640 1 -- 172.21.15.46:0/3148979495 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5ccc0b2380 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.613+0000 7f5cd37fe640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5cbc0664a0 0x7f5cbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.613+0000 7f5cd37fe640 1 -- 172.21.15.46:0/3148979495 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8348+0+0 (secure 0 0 0) 0x7f5ccc0a7070 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.613+0000 7f5cd3fff640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5cbc0664a0 0x7f5cbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.613+0000 7f5cd3fff640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5cbc0664a0 0x7f5cbc068960 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc14ebb0 tx=0x7f5cc4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.617+0000 7f5cd37fe640 1 -- 172.21.15.46:0/3148979495 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5ccc0eb6b0 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.805+0000 7f5ce151d640 1 -- 172.21.15.46:0/3148979495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cdc0c6de0 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.805+0000 7f5cd37fe640 1 -- 172.21.15.46:0/3148979495 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f5ccc0c3020 con 0x7f5cdc0c1ee0 2023-10-30T14:20:38.812 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:38.812 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9594,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:19:23.041684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9276},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9276":{"gid":9276,"name":"c","rank":1,"incarnation":225,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6836/2239735286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2239735286},{"type":"v1","addr":"172.21.15.46:6836","nonce":2239735286}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:38.812 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 238 2023-10-30T14:20:38.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.809+0000 7f5ce151d640 1 -- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5cbc0664a0 msgr2=0x7f5cbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.809+0000 7f5ce151d640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5cbc0664a0 0x7f5cbc068960 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc14ebb0 tx=0x7f5cc4008040 comp rx=0 tx=0).stop 2023-10-30T14:20:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.809+0000 7f5ce151d640 1 -- 172.21.15.46:0/3148979495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cdc0c1ee0 msgr2=0x7f5cdc143d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.809+0000 7f5ce151d640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cdc0c1ee0 0x7f5cdc143d10 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc01ab80 tx=0x7f5ccc01ad10 comp rx=0 tx=0).stop 2023-10-30T14:20:38.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.809+0000 7f5ce151d640 1 -- 172.21.15.46:0/3148979495 shutdown_connections 2023-10-30T14:20:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.809+0000 7f5ce151d640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5cbc0664a0 0x7f5cbc068960 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.809+0000 7f5ce151d640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cdc151800 0x7f5cdc1f0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.809+0000 7f5ce151d640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0c9230 0x7f5cdc0c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.809+0000 7f5ce151d640 1 --2- 172.21.15.46:0/3148979495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cdc0c1ee0 0x7f5cdc143d10 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.809+0000 7f5ce151d640 1 -- 172.21.15.46:0/3148979495 >> 172.21.15.46:0/3148979495 conn(0x7f5cdc071ea0 msgr2=0x7f5cdc156ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:38.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.809+0000 7f5ce151d640 1 -- 172.21.15.46:0/3148979495 shutdown_connections 2023-10-30T14:20:38.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:38.809+0000 7f5ce151d640 1 -- 172.21.15.46:0/3148979495 wait complete. 2023-10-30T14:20:38.833 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:20:40.835 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.985+0000 7f81cf06b640 1 Processor -- start 2023-10-30T14:20:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.985+0000 7f81cf06b640 1 -- start start 2023-10-30T14:20:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.985+0000 7f81cf06b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f81c8153bf0 0x7f81c8071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81cf06b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c80c5f60 0x7f81c80cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81cf06b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c80700b0 con 0x7f81c80c6fe0 2023-10-30T14:20:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81cf06b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c80c6810 con 0x7f81c8153bf0 2023-10-30T14:20:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81cf06b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8159900 con 0x7f81c80c5f60 2023-10-30T14:20:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81cce07640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f81c8153bf0 0x7f81c8071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81cce07640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f81c8153bf0 0x7f81c8071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:47028/0 (socket says 172.21.15.46:47028) 2023-10-30T14:20:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81cce07640 1 -- 172.21.15.46:0/3724899242 learned_addr learned my addr 172.21.15.46:0/3724899242 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81cd608640 1 --2- 172.21.15.46:0/3724899242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c80c5f60 0x7f81c80cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81cce07640 1 -- 172.21.15.46:0/3724899242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c80c5f60 msgr2=0x7f81c80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81cce07640 1 --2- 172.21.15.46:0/3724899242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c80c5f60 0x7f81c80cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81cce07640 1 -- 172.21.15.46:0/3724899242 >> v1:172.21.15.154:6789/0 conn(0x7f81c80c6fe0 legacy=0x7f81c80c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81cce07640 1 -- 172.21.15.46:0/3724899242 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c80c6530 con 0x7f81c8153bf0 2023-10-30T14:20:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.989+0000 7f81c77fe640 1 -- 172.21.15.46:0/3724899242 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413756562 0 0) 0x7f81c80700b0 con 0x7f81c80c6fe0 2023-10-30T14:20:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81cce07640 1 --2- 172.21.15.46:0/3724899242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c8153bf0 0x7f81c8071260 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009870 tx=0x7f81bc01ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=753f0e9428b1b34a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81c77fe640 1 -- 172.21.15.46:0/3724899242 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0a8a30 con 0x7f81c8153bf0 2023-10-30T14:20:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81c77fe640 1 -- 172.21.15.46:0/3724899242 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc0a8b90 con 0x7f81c8153bf0 2023-10-30T14:20:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81c77fe640 1 -- 172.21.15.46:0/3724899242 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0b0440 con 0x7f81c8153bf0 2023-10-30T14:20:40.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81cf06b640 1 -- 172.21.15.46:0/3724899242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c8153bf0 msgr2=0x7f81c8071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:40.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81cf06b640 1 --2- 172.21.15.46:0/3724899242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c8153bf0 0x7f81c8071260 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009870 tx=0x7f81bc01ed30 comp rx=0 tx=0).stop 2023-10-30T14:20:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81cf06b640 1 -- 172.21.15.46:0/3724899242 shutdown_connections 2023-10-30T14:20:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81cf06b640 1 --2- 172.21.15.46:0/3724899242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c80c5f60 0x7f81c80cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81cf06b640 1 --2- 172.21.15.46:0/3724899242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c8153bf0 0x7f81c8071260 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81cf06b640 1 -- 172.21.15.46:0/3724899242 >> 172.21.15.46:0/3724899242 conn(0x7f81c806c6e0 msgr2=0x7f81c8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:40.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81cf06b640 1 -- 172.21.15.46:0/3724899242 shutdown_connections 2023-10-30T14:20:40.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81cf06b640 1 -- 172.21.15.46:0/3724899242 wait complete. 2023-10-30T14:20:41.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81cf06b640 1 Processor -- start 2023-10-30T14:20:41.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.993+0000 7f81cf06b640 1 -- start start 2023-10-30T14:20:41.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81cf06b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c80c5f60 0x7f81c80ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:41.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81cf06b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80c6fe0 0x7f81c814e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:41.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81cf06b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8153bf0 0x7f81c814e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:41.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81cf06b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c814d1f0 con 0x7f81c80c6fe0 2023-10-30T14:20:41.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81cf06b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c81eefd0 con 0x7f81c80c5f60 2023-10-30T14:20:41.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81cf06b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c81efdf0 con 0x7f81c8153bf0 2023-10-30T14:20:41.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80c6fe0 0x7f81c814e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:41.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80c6fe0 0x7f81c814e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39442/0 (socket says 172.21.15.46:39442) 2023-10-30T14:20:41.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c7fff640 1 -- 172.21.15.46:0/2828172690 learned_addr learned my addr 172.21.15.46:0/2828172690 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:41.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81cce07640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c80c5f60 0x7f81c80ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:41.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81cd608640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8153bf0 0x7f81c814e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:41.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c7fff640 1 -- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8153bf0 msgr2=0x7f81c814e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:41.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c7fff640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8153bf0 0x7f81c814e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c7fff640 1 -- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c80c5f60 msgr2=0x7f81c80ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c7fff640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c80c5f60 0x7f81c80ca790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c7fff640 1 -- 172.21.15.46:0/2828172690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc0094d0 con 0x7f81c80c6fe0 2023-10-30T14:20:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c7fff640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80c6fe0 0x7f81c814e050 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f81b400c4d0 tx=0x7f81b400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c57fa640 1 -- 172.21.15.46:0/2828172690 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b4005480 con 0x7f81c80c6fe0 2023-10-30T14:20:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c57fa640 1 -- 172.21.15.46:0/2828172690 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b4003600 con 0x7f81c80c6fe0 2023-10-30T14:20:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c57fa640 1 -- 172.21.15.46:0/2828172690 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b40039b0 con 0x7f81c80c6fe0 2023-10-30T14:20:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81cf06b640 1 -- 172.21.15.46:0/2828172690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c814f120 con 0x7f81c80c6fe0 2023-10-30T14:20:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81cf06b640 1 -- 172.21.15.46:0/2828172690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c81ef4c0 con 0x7f81c80c6fe0 2023-10-30T14:20:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81cf06b640 1 -- 172.21.15.46:0/2828172690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c804e170 con 0x7f81c80c6fe0 2023-10-30T14:20:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:40.997+0000 7f81c57fa640 1 -- 172.21.15.46:0/2828172690 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f81b4003b10 con 0x7f81c80c6fe0 2023-10-30T14:20:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.001+0000 7f81c57fa640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81ac0664a0 0x7f81ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.001+0000 7f81c57fa640 1 -- 172.21.15.46:0/2828172690 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f81b4083fa0 con 0x7f81c80c6fe0 2023-10-30T14:20:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.001+0000 7f81cce07640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81ac0664a0 0x7f81ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.001+0000 7f81cce07640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81ac0664a0 0x7f81ac068960 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0b2000 tx=0x7f81bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.005+0000 7f81c57fa640 1 -- 172.21.15.46:0/2828172690 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f81b40509a0 con 0x7f81c80c6fe0 2023-10-30T14:20:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.197+0000 7f81cf06b640 1 -- 172.21.15.46:0/2828172690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81c81590f0 con 0x7f81c80c6fe0 2023-10-30T14:20:41.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.197+0000 7f81c57fa640 1 -- 172.21.15.46:0/2828172690 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f81b4017020 con 0x7f81c80c6fe0 2023-10-30T14:20:41.203 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:41.203 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:20:39.772439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:41.204 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 239 2023-10-30T14:20:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.201+0000 7f81cf06b640 1 -- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81ac0664a0 msgr2=0x7f81ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.201+0000 7f81cf06b640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81ac0664a0 0x7f81ac068960 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0b2000 tx=0x7f81bc009040 comp rx=0 tx=0).stop 2023-10-30T14:20:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.201+0000 7f81cf06b640 1 -- 172.21.15.46:0/2828172690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80c6fe0 msgr2=0x7f81c814e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.201+0000 7f81cf06b640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80c6fe0 0x7f81c814e050 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f81b400c4d0 tx=0x7f81b400c990 comp rx=0 tx=0).stop 2023-10-30T14:20:41.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.201+0000 7f81cf06b640 1 -- 172.21.15.46:0/2828172690 shutdown_connections 2023-10-30T14:20:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.201+0000 7f81cf06b640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81ac0664a0 0x7f81ac068960 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.201+0000 7f81cf06b640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81c8153bf0 0x7f81c814e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.201+0000 7f81cf06b640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80c6fe0 0x7f81c814e050 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.201+0000 7f81cf06b640 1 --2- 172.21.15.46:0/2828172690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c80c5f60 0x7f81c80ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.201+0000 7f81cf06b640 1 -- 172.21.15.46:0/2828172690 >> 172.21.15.46:0/2828172690 conn(0x7f81c806c6e0 msgr2=0x7f81c8151ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.201+0000 7f81cf06b640 1 -- 172.21.15.46:0/2828172690 shutdown_connections 2023-10-30T14:20:41.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.201+0000 7f81cf06b640 1 -- 172.21.15.46:0/2828172690 wait complete. 2023-10-30T14:20:41.225 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T14:20:41.225 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:20:41.226 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.365+0000 7f8ef1846640 1 Processor -- start 2023-10-30T14:20:41.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.365+0000 7f8ef1846640 1 -- start start 2023-10-30T14:20:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.365+0000 7f8ef1846640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec15d6f0 0x7f8eec15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:41.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.365+0000 7f8ef1846640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eec0cb800 con 0x7f8eec1538a0 2023-10-30T14:20:41.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.365+0000 7f8ef1846640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eec0cb330 con 0x7f8eec0cf160 2023-10-30T14:20:41.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.365+0000 7f8ef1846640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec0cad70 con 0x7f8eec15d6f0 2023-10-30T14:20:41.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.365+0000 7f8eeaffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8eec1538a0 0x7f8eec0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36830/0 (socket says 172.21.15.46:36830) 2023-10-30T14:20:41.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.365+0000 7f8eeaffd640 1 -- 172.21.15.46:0/3601128625 learned_addr learned my addr 172.21.15.46:0/3601128625 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:20:41.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8ee3fff640 1 --2- 172.21.15.46:0/3601128625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec15d6f0 0x7f8eec15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:41.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058802209 0 0) 0x7f8eec0cb800 con 0x7f8eec1538a0 2023-10-30T14:20:41.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed8003530 con 0x7f8eec1538a0 2023-10-30T14:20:41.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220217902 0 0) 0x7f8eec0cb330 con 0x7f8eec0cf160 2023-10-30T14:20:41.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eec0cb800 con 0x7f8eec0cf160 2023-10-30T14:20:41.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2290076005 0 0) 0x7f8ed8003530 con 0x7f8eec1538a0 2023-10-30T14:20:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec15d6f0 msgr2=0x7f8eec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 --2- 172.21.15.46:0/3601128625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec15d6f0 0x7f8eec15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 >> v1:172.21.15.154:6789/0 conn(0x7f8eec0cf160 legacy=0x7f8eec164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec0cb050 con 0x7f8eec1538a0 2023-10-30T14:20:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f8edc001680 con 0x7f8eec1538a0 2023-10-30T14:20:41.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 >> v1:172.21.15.46:6789/0 conn(0x7f8eec1538a0 legacy=0x7f8eec0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:41.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 --2- 172.21.15.46:0/3601128625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8003b40 0x7f8ed8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:41.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 --2- 172.21.15.46:0/3601128625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed80047d0 0x7f8ed8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:41.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 --2- 172.21.15.46:0/3601128625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec15d6f0 0x7f8ed8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:41.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec0cad70 con 0x7f8ed8003b40 2023-10-30T14:20:41.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed800ba40 con 0x7f8ed80047d0 2023-10-30T14:20:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed800bc00 con 0x7f8eec15d6f0 2023-10-30T14:20:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eeaffd640 1 --2- 172.21.15.46:0/3601128625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8003b40 0x7f8ed8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8ee3fff640 1 --2- 172.21.15.46:0/3601128625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed80047d0 0x7f8ed8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eeb7fe640 1 --2- 172.21.15.46:0/3601128625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec15d6f0 0x7f8ed8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:41.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eeaffd640 1 -- 172.21.15.46:0/3601128625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec15d6f0 msgr2=0x7f8ed8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:41.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eeaffd640 1 --2- 172.21.15.46:0/3601128625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec15d6f0 0x7f8ed8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eeaffd640 1 -- 172.21.15.46:0/3601128625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed80047d0 msgr2=0x7f8ed8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:41.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eeaffd640 1 --2- 172.21.15.46:0/3601128625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed80047d0 0x7f8ed8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.369+0000 7f8eeaffd640 1 -- 172.21.15.46:0/3601128625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed8003530 con 0x7f8ed8003b40 2023-10-30T14:20:41.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8eeaffd640 1 --2- 172.21.15.46:0/3601128625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8003b40 0x7f8ed8003f40 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f8edc001ec0 tx=0x7f8edc01cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e105f44917e46a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:41.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc0058e0 con 0x7f8ed8003b40 2023-10-30T14:20:41.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8edc005a60 con 0x7f8ed8003b40 2023-10-30T14:20:41.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8eea7fc640 1 -- 172.21.15.46:0/3601128625 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc005e10 con 0x7f8ed8003b40 2023-10-30T14:20:41.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8ef1846640 1 -- 172.21.15.46:0/3601128625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8003b40 msgr2=0x7f8ed8003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:41.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8ef1846640 1 --2- 172.21.15.46:0/3601128625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8003b40 0x7f8ed8003f40 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f8edc001ec0 tx=0x7f8edc01cb80 comp rx=0 tx=0).stop 2023-10-30T14:20:41.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8eeb7fe640 1 -- 172.21.15.46:0/3601128625 reap_dead start 2023-10-30T14:20:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8ef1846640 1 -- 172.21.15.46:0/3601128625 shutdown_connections 2023-10-30T14:20:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8ef1846640 1 -- 172.21.15.46:0/3601128625 >> 172.21.15.46:0/3601128625 conn(0x7f8eec06bfe0 msgr2=0x7f8eec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8ef1846640 1 -- 172.21.15.46:0/3601128625 shutdown_connections 2023-10-30T14:20:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8ef1846640 1 -- 172.21.15.46:0/3601128625 wait complete. 2023-10-30T14:20:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8ef1846640 1 Processor -- start 2023-10-30T14:20:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.373+0000 7f8ef1846640 1 -- start start 2023-10-30T14:20:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ef1846640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8003b40 0x7f8eec0b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ef1846640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8eec1465e0 0x7f8eec1e5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ef1846640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec1f03c0 0x7f8eec149450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ef1846640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec146a20 con 0x7f8ed8003b40 2023-10-30T14:20:41.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ef1846640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec144e80 con 0x7f8eec1465e0 2023-10-30T14:20:41.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ef1846640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec140b50 con 0x7f8eec1f03c0 2023-10-30T14:20:41.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8eeaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8003b40 0x7f8eec0b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:41.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8eeaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8003b40 0x7f8eec0b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39456/0 (socket says 172.21.15.46:39456) 2023-10-30T14:20:41.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8eeaffd640 1 -- 172.21.15.46:0/1914735337 learned_addr learned my addr 172.21.15.46:0/1914735337 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:41.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ee3fff640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8eec1465e0 0x7f8eec1e5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:41.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8eeb7fe640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec1f03c0 0x7f8eec149450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:41.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8eeaffd640 1 -- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec1f03c0 msgr2=0x7f8eec149450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:41.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8eeaffd640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec1f03c0 0x7f8eec149450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8eeaffd640 1 -- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8eec1465e0 msgr2=0x7f8eec1e5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:41.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8eeaffd640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8eec1465e0 0x7f8eec1e5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8eeaffd640 1 -- 172.21.15.46:0/1914735337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec145e90 con 0x7f8ed8003b40 2023-10-30T14:20:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ee3fff640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8eec1465e0 0x7f8eec1e5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8eeaffd640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8003b40 0x7f8eec0b3b20 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f8edc01c9b0 tx=0x7f8ed8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ee8ff9640 1 -- 172.21.15.46:0/1914735337 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc0a5d50 con 0x7f8ed8003b40 2023-10-30T14:20:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ee8ff9640 1 -- 172.21.15.46:0/1914735337 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8edc0047f0 con 0x7f8ed8003b40 2023-10-30T14:20:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ee8ff9640 1 -- 172.21.15.46:0/1914735337 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc004b60 con 0x7f8ed8003b40 2023-10-30T14:20:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ef1846640 1 -- 172.21.15.46:0/1914735337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eec146c20 con 0x7f8ed8003b40 2023-10-30T14:20:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ef1846640 1 -- 172.21.15.46:0/1914735337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eec1453b0 con 0x7f8ed8003b40 2023-10-30T14:20:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ef1846640 1 -- 172.21.15.46:0/1914735337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eec04e170 con 0x7f8ed8003b40 2023-10-30T14:20:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.377+0000 7f8ee8ff9640 1 -- 172.21.15.46:0/1914735337 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8edc004cc0 con 0x7f8ed8003b40 2023-10-30T14:20:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.381+0000 7f8ee8ff9640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ecc0664a0 0x7f8ecc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.381+0000 7f8ee8ff9640 1 -- 172.21.15.46:0/1914735337 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f8edc11e1d0 con 0x7f8ed8003b40 2023-10-30T14:20:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.381+0000 7f8ee3fff640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ecc0664a0 0x7f8ecc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.381+0000 7f8ee3fff640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ecc0664a0 0x7f8ecc068960 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f8eec1f2420 tx=0x7f8ed4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.385+0000 7f8ee8ff9640 1 -- 172.21.15.46:0/1914735337 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8edc0eab50 con 0x7f8ed8003b40 2023-10-30T14:20:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 -- 172.21.15.46:0/1914735337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8eec0b1dd0 con 0x7f8ed8003b40 2023-10-30T14:20:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ee8ff9640 1 -- 172.21.15.46:0/1914735337 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f8edc0be090 con 0x7f8ed8003b40 2023-10-30T14:20:41.577 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:41.577 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:20:39.772439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 239 2023-10-30T14:20:41.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 -- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ecc0664a0 msgr2=0x7f8ecc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:41.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ecc0664a0 0x7f8ecc068960 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f8eec1f2420 tx=0x7f8ed4008040 comp rx=0 tx=0).stop 2023-10-30T14:20:41.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 -- 172.21.15.46:0/1914735337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8003b40 msgr2=0x7f8eec0b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:41.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8003b40 0x7f8eec0b3b20 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f8edc01c9b0 tx=0x7f8ed8005800 comp rx=0 tx=0).stop 2023-10-30T14:20:41.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 -- 172.21.15.46:0/1914735337 shutdown_connections 2023-10-30T14:20:41.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ecc0664a0 0x7f8ecc068960 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8eec1f03c0 0x7f8eec149450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8eec1465e0 0x7f8eec1e5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 --2- 172.21.15.46:0/1914735337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed8003b40 0x7f8eec0b3b20 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:41.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 -- 172.21.15.46:0/1914735337 >> 172.21.15.46:0/1914735337 conn(0x7f8eec06bfe0 msgr2=0x7f8eec0c5310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:41.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 -- 172.21.15.46:0/1914735337 shutdown_connections 2023-10-30T14:20:41.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:41.573+0000 7f8ef1846640 1 -- 172.21.15.46:0/1914735337 wait complete. 2023-10-30T14:20:41.598 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 239, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 239, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9426": { "addr": "172.21.15.46:6833/3764280378", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 3764280378, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 3764280378, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 9426, "incarnation": 232, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 42 }, "gid_9594": { "addr": "172.21.15.46:6837/470164358", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 470164358, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 470164358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9594, "incarnation": 239, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 87, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:20:39.772439+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9426, "mds_1": 9594 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:20:43.600 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:43.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.749+0000 7f296a72a640 1 Processor -- start 2023-10-30T14:20:43.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.749+0000 7f296a72a640 1 -- start start 2023-10-30T14:20:43.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f296a72a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29640c6b80 0x7f29640bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:43.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f296a72a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29640c6f60 0x7f29640c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:43.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f296a72a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29640c0a50 con 0x7f29640c8430 2023-10-30T14:20:43.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f296a72a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f296406f6e0 con 0x7f29640c6f60 2023-10-30T14:20:43.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f296a72a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296406f9c0 con 0x7f29640c6b80 2023-10-30T14:20:43.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f29637fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29640c6f60 0x7f29640c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:43.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f29637fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29640c6f60 0x7f29640c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39466/0 (socket says 172.21.15.46:39466) 2023-10-30T14:20:43.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f29637fe640 1 -- 172.21.15.46:0/2678938372 learned_addr learned my addr 172.21.15.46:0/2678938372 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:43.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f2963fff640 1 --2- 172.21.15.46:0/2678938372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29640c6b80 0x7f29640bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:43.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f29637fe640 1 -- 172.21.15.46:0/2678938372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29640c6b80 msgr2=0x7f29640bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:43.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f29637fe640 1 --2- 172.21.15.46:0/2678938372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29640c6b80 0x7f29640bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f29637fe640 1 -- 172.21.15.46:0/2678938372 >> v1:172.21.15.154:6789/0 conn(0x7f29640c8430 legacy=0x7f296413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:43.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.753+0000 7f29637fe640 1 -- 172.21.15.46:0/2678938372 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296406fca0 con 0x7f29640c6f60 2023-10-30T14:20:43.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f29637fe640 1 --2- 172.21.15.46:0/2678938372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29640c6f60 0x7f29640c0400 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f2958009870 tx=0x7f295801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74cea5f702ab1e75 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:43.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f2962ffd640 1 -- 172.21.15.46:0/2678938372 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29580ab650 con 0x7f29640c6f60 2023-10-30T14:20:43.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f2962ffd640 1 -- 172.21.15.46:0/2678938372 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29580ab7b0 con 0x7f29640c6f60 2023-10-30T14:20:43.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f2962ffd640 1 -- 172.21.15.46:0/2678938372 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29580abb40 con 0x7f29640c6f60 2023-10-30T14:20:43.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f296a72a640 1 -- 172.21.15.46:0/2678938372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29640c6f60 msgr2=0x7f29640c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:43.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f296a72a640 1 --2- 172.21.15.46:0/2678938372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29640c6f60 0x7f29640c0400 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f2958009870 tx=0x7f295801c540 comp rx=0 tx=0).stop 2023-10-30T14:20:43.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f296a72a640 1 -- 172.21.15.46:0/2678938372 shutdown_connections 2023-10-30T14:20:43.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f296a72a640 1 --2- 172.21.15.46:0/2678938372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29640c6f60 0x7f29640c0400 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f296a72a640 1 --2- 172.21.15.46:0/2678938372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29640c6b80 0x7f29640bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f296a72a640 1 -- 172.21.15.46:0/2678938372 >> 172.21.15.46:0/2678938372 conn(0x7f29640c7910 msgr2=0x7f29640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:43.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f296a72a640 1 -- 172.21.15.46:0/2678938372 shutdown_connections 2023-10-30T14:20:43.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f296a72a640 1 -- 172.21.15.46:0/2678938372 wait complete. 2023-10-30T14:20:43.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f296a72a640 1 Processor -- start 2023-10-30T14:20:43.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f296a72a640 1 -- start start 2023-10-30T14:20:43.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f296a72a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29640c6b80 0x7f2964150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:43.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.757+0000 7f296a72a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29640c6f60 0x7f29641359d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:43.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f296a72a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29640c8430 0x7f2964135f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:43.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f296a72a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2964150870 con 0x7f29640c6f60 2023-10-30T14:20:43.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f296a72a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29641577a0 con 0x7f29640c8430 2023-10-30T14:20:43.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f296a72a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29641f0840 con 0x7f29640c6b80 2023-10-30T14:20:43.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f29637fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29640c6f60 0x7f29641359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:43.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f29637fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29640c6f60 0x7f29641359d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39480/0 (socket says 172.21.15.46:39480) 2023-10-30T14:20:43.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f29637fe640 1 -- 172.21.15.46:0/3063300652 learned_addr learned my addr 172.21.15.46:0/3063300652 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:43.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f2963fff640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29640c6b80 0x7f2964150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:43.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f2968cc7640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29640c8430 0x7f2964135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:43.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f29637fe640 1 -- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29640c6b80 msgr2=0x7f2964150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:43.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f29637fe640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29640c6b80 0x7f2964150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f29637fe640 1 -- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29640c8430 msgr2=0x7f2964135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:43.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f29637fe640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29640c8430 0x7f2964135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f29637fe640 1 -- 172.21.15.46:0/3063300652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29580094d0 con 0x7f29640c6f60 2023-10-30T14:20:43.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f2968cc7640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29640c8430 0x7f2964135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:43.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f29637fe640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29640c6f60 0x7f29641359d0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f295801a170 tx=0x7f295801ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:43.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f2960ff9640 1 -- 172.21.15.46:0/3063300652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29580aa980 con 0x7f29640c6f60 2023-10-30T14:20:43.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f2960ff9640 1 -- 172.21.15.46:0/3063300652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29580aab00 con 0x7f29640c6f60 2023-10-30T14:20:43.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f2960ff9640 1 -- 172.21.15.46:0/3063300652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29580aaeb0 con 0x7f29640c6f60 2023-10-30T14:20:43.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f296a72a640 1 -- 172.21.15.46:0/3063300652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2964150e40 con 0x7f29640c6f60 2023-10-30T14:20:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f296a72a640 1 -- 172.21.15.46:0/3063300652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29641eff10 con 0x7f29640c6f60 2023-10-30T14:20:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f296a72a640 1 -- 172.21.15.46:0/3063300652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29641540d0 con 0x7f29640c6f60 2023-10-30T14:20:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.761+0000 7f2960ff9640 1 -- 172.21.15.46:0/3063300652 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29580b43f0 con 0x7f29640c6f60 2023-10-30T14:20:43.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.765+0000 7f2960ff9640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29500664a0 0x7f2950068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.765+0000 7f2960ff9640 1 -- 172.21.15.46:0/3063300652 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f295811f6e0 con 0x7f29640c6f60 2023-10-30T14:20:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.765+0000 7f2963fff640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29500664a0 0x7f2950068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.765+0000 7f2963fff640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29500664a0 0x7f2950068960 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2954006aa0 tx=0x7f2954008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.765+0000 7f2960ff9640 1 -- 172.21.15.46:0/3063300652 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29580ec110 con 0x7f29640c6f60 2023-10-30T14:20:43.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.957+0000 7f296a72a640 1 -- 172.21.15.46:0/3063300652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29641f09b0 con 0x7f29640c6f60 2023-10-30T14:20:43.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.957+0000 7f2960ff9640 1 -- 172.21.15.46:0/3063300652 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f29580ef730 con 0x7f29640c6f60 2023-10-30T14:20:43.963 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:43.963 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:20:39.772439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:43.964 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 239 2023-10-30T14:20:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.961+0000 7f296a72a640 1 -- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29500664a0 msgr2=0x7f2950068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.961+0000 7f296a72a640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29500664a0 0x7f2950068960 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2954006aa0 tx=0x7f2954008040 comp rx=0 tx=0).stop 2023-10-30T14:20:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.961+0000 7f296a72a640 1 -- 172.21.15.46:0/3063300652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29640c6f60 msgr2=0x7f29641359d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.961+0000 7f296a72a640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29640c6f60 0x7f29641359d0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f295801a170 tx=0x7f295801ad80 comp rx=0 tx=0).stop 2023-10-30T14:20:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.961+0000 7f296a72a640 1 -- 172.21.15.46:0/3063300652 shutdown_connections 2023-10-30T14:20:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.961+0000 7f296a72a640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29500664a0 0x7f2950068960 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.961+0000 7f296a72a640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29640c8430 0x7f2964135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.961+0000 7f296a72a640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29640c6f60 0x7f29641359d0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.961+0000 7f296a72a640 1 --2- 172.21.15.46:0/3063300652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29640c6b80 0x7f2964150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.961+0000 7f296a72a640 1 -- 172.21.15.46:0/3063300652 >> 172.21.15.46:0/3063300652 conn(0x7f29640c7910 msgr2=0x7f29640c0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.961+0000 7f296a72a640 1 -- 172.21.15.46:0/3063300652 shutdown_connections 2023-10-30T14:20:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:43.961+0000 7f296a72a640 1 -- 172.21.15.46:0/3063300652 wait complete. 2023-10-30T14:20:43.985 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:45.986 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:46.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b2be4640 1 Processor -- start 2023-10-30T14:20:46.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b2be4640 1 -- start start 2023-10-30T14:20:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b2be4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90ac1538f0 0x7f90ac159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b2be4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f90ac15cd50 0x7f90ac0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b2be4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90ac0cefa0 0x7f90ac0cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b2be4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ac15d300 con 0x7f90ac0cefa0 2023-10-30T14:20:46.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b2be4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ac15d5e0 con 0x7f90ac15cd50 2023-10-30T14:20:46.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b2be4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac15d8c0 con 0x7f90ac1538f0 2023-10-30T14:20:46.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b23e3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90ac0cefa0 0x7f90ac0cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:46.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b23e3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90ac0cefa0 0x7f90ac0cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39494/0 (socket says 172.21.15.46:39494) 2023-10-30T14:20:46.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b23e3640 1 -- 172.21.15.46:0/270752858 learned_addr learned my addr 172.21.15.46:0/270752858 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:46.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b13e1640 1 --2- 172.21.15.46:0/270752858 >> v2:172.21.15.154:3300/0 conn(0x7f90ac15cd50 0x7f90ac0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:46.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.133+0000 7f90b1be2640 1 --2- 172.21.15.46:0/270752858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90ac1538f0 0x7f90ac159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:46.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.137+0000 7f90b23e3640 1 -- 172.21.15.46:0/270752858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90ac1538f0 msgr2=0x7f90ac159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:46.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.137+0000 7f90b23e3640 1 --2- 172.21.15.46:0/270752858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90ac1538f0 0x7f90ac159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.137+0000 7f90b23e3640 1 -- 172.21.15.46:0/270752858 >> v2:172.21.15.154:3300/0 conn(0x7f90ac15cd50 msgr2=0x7f90ac0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:46.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.137+0000 7f90b23e3640 1 --2- 172.21.15.46:0/270752858 >> v2:172.21.15.154:3300/0 conn(0x7f90ac15cd50 0x7f90ac0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.137+0000 7f90b23e3640 1 -- 172.21.15.46:0/270752858 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac15dba0 con 0x7f90ac0cefa0 2023-10-30T14:20:46.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.137+0000 7f90b23e3640 1 --2- 172.21.15.46:0/270752858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac0cefa0 0x7f90ac0cf3f0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f90ac066410 tx=0x7f90a401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0631b51b3cb9f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:46.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.137+0000 7f90b0be0640 1 -- 172.21.15.46:0/270752858 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40af400 con 0x7f90ac0cefa0 2023-10-30T14:20:46.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.137+0000 7f90b0be0640 1 -- 172.21.15.46:0/270752858 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a40ab050 con 0x7f90ac0cefa0 2023-10-30T14:20:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.137+0000 7f90b0be0640 1 -- 172.21.15.46:0/270752858 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40af780 con 0x7f90ac0cefa0 2023-10-30T14:20:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.141+0000 7f90b2be4640 1 -- 172.21.15.46:0/270752858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac0cefa0 msgr2=0x7f90ac0cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.141+0000 7f90b2be4640 1 --2- 172.21.15.46:0/270752858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac0cefa0 0x7f90ac0cf3f0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f90ac066410 tx=0x7f90a401f1c0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.141+0000 7f90b2be4640 1 -- 172.21.15.46:0/270752858 shutdown_connections 2023-10-30T14:20:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.141+0000 7f90b2be4640 1 --2- 172.21.15.46:0/270752858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac0cefa0 0x7f90ac0cf3f0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.141+0000 7f90b2be4640 1 --2- 172.21.15.46:0/270752858 >> v2:172.21.15.154:3300/0 conn(0x7f90ac15cd50 0x7f90ac0c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.141+0000 7f90b2be4640 1 --2- 172.21.15.46:0/270752858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90ac1538f0 0x7f90ac159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.141+0000 7f90b2be4640 1 -- 172.21.15.46:0/270752858 >> 172.21.15.46:0/270752858 conn(0x7f90ac06a640 msgr2=0x7f90ac065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:46.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.141+0000 7f90b2be4640 1 -- 172.21.15.46:0/270752858 shutdown_connections 2023-10-30T14:20:46.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.141+0000 7f90b2be4640 1 -- 172.21.15.46:0/270752858 wait complete. 2023-10-30T14:20:46.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.141+0000 7f90b2be4640 1 Processor -- start 2023-10-30T14:20:46.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.141+0000 7f90b2be4640 1 -- start start 2023-10-30T14:20:46.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b2be4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90ac0cefa0 0x7f90ac151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:46.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b2be4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac1538f0 0x7f90ac152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:46.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b2be4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90ac15cd50 0x7f90ac0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:46.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b2be4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac151e40 con 0x7f90ac1538f0 2023-10-30T14:20:46.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b2be4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac145780 con 0x7f90ac15cd50 2023-10-30T14:20:46.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b2be4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac147510 con 0x7f90ac0cefa0 2023-10-30T14:20:46.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b13e1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac1538f0 0x7f90ac152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:46.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b13e1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac1538f0 0x7f90ac152490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39510/0 (socket says 172.21.15.46:39510) 2023-10-30T14:20:46.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b13e1640 1 -- 172.21.15.46:0/1607844122 learned_addr learned my addr 172.21.15.46:0/1607844122 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:46.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b1be2640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90ac0cefa0 0x7f90ac151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:46.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b23e3640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90ac15cd50 0x7f90ac0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:46.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b13e1640 1 -- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90ac0cefa0 msgr2=0x7f90ac151900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:46.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b13e1640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90ac0cefa0 0x7f90ac151900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b13e1640 1 -- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90ac15cd50 msgr2=0x7f90ac0ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:46.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b13e1640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90ac15cd50 0x7f90ac0ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b13e1640 1 -- 172.21.15.46:0/1607844122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a4009b30 con 0x7f90ac1538f0 2023-10-30T14:20:46.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b13e1640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac1538f0 0x7f90ac152490 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f909c00d620 tx=0x7f909c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:46.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90a27fc640 1 -- 172.21.15.46:0/1607844122 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c00b620 con 0x7f90ac1538f0 2023-10-30T14:20:46.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90a27fc640 1 -- 172.21.15.46:0/1607844122 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909c00b780 con 0x7f90ac1538f0 2023-10-30T14:20:46.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b2be4640 1 -- 172.21.15.46:0/1607844122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ac145a00 con 0x7f90ac1538f0 2023-10-30T14:20:46.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90a27fc640 1 -- 172.21.15.46:0/1607844122 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909c00bb10 con 0x7f90ac1538f0 2023-10-30T14:20:46.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b2be4640 1 -- 172.21.15.46:0/1607844122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ac147a70 con 0x7f90ac1538f0 2023-10-30T14:20:46.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90b2be4640 1 -- 172.21.15.46:0/1607844122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ac04e170 con 0x7f90ac1538f0 2023-10-30T14:20:46.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.145+0000 7f90a27fc640 1 -- 172.21.15.46:0/1607844122 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f909c00bc70 con 0x7f90ac1538f0 2023-10-30T14:20:46.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.149+0000 7f90a27fc640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90900664d0 0x7f9090068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:46.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.149+0000 7f90a27fc640 1 -- 172.21.15.46:0/1607844122 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f909c084d10 con 0x7f90ac1538f0 2023-10-30T14:20:46.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.149+0000 7f90b1be2640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90900664d0 0x7f9090068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:46.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.149+0000 7f90b1be2640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90900664d0 0x7f9090068990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f90a80045f0 tx=0x7f90a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.153+0000 7f90a27fc640 1 -- 172.21.15.46:0/1607844122 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f909c051740 con 0x7f90ac1538f0 2023-10-30T14:20:46.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.349+0000 7f90b2be4640 1 -- 172.21.15.46:0/1607844122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90ac15c9a0 con 0x7f90ac1538f0 2023-10-30T14:20:46.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.349+0000 7f90a27fc640 1 -- 172.21.15.46:0/1607844122 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f909c054d60 con 0x7f90ac1538f0 2023-10-30T14:20:46.354 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:46.354 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:20:39.772439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:46.354 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 239 2023-10-30T14:20:46.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.349+0000 7f90b2be4640 1 -- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90900664d0 msgr2=0x7f9090068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:46.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.353+0000 7f90b2be4640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90900664d0 0x7f9090068990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f90a80045f0 tx=0x7f90a8008040 comp rx=0 tx=0).stop 2023-10-30T14:20:46.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.353+0000 7f90b2be4640 1 -- 172.21.15.46:0/1607844122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac1538f0 msgr2=0x7f90ac152490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:46.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.353+0000 7f90b2be4640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac1538f0 0x7f90ac152490 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f909c00d620 tx=0x7f909c00dae0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.353+0000 7f90b2be4640 1 -- 172.21.15.46:0/1607844122 shutdown_connections 2023-10-30T14:20:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.353+0000 7f90b2be4640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90900664d0 0x7f9090068990 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.353+0000 7f90b2be4640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90ac15cd50 0x7f90ac0ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.353+0000 7f90b2be4640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90ac1538f0 0x7f90ac152490 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.353+0000 7f90b2be4640 1 --2- 172.21.15.46:0/1607844122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90ac0cefa0 0x7f90ac151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.353+0000 7f90b2be4640 1 -- 172.21.15.46:0/1607844122 >> 172.21.15.46:0/1607844122 conn(0x7f90ac06a640 msgr2=0x7f90ac0cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.353+0000 7f90b2be4640 1 -- 172.21.15.46:0/1607844122 shutdown_connections 2023-10-30T14:20:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:46.353+0000 7f90b2be4640 1 -- 172.21.15.46:0/1607844122 wait complete. 2023-10-30T14:20:46.383 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:48.384 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.541+0000 7f09fb646640 1 Processor -- start 2023-10-30T14:20:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.541+0000 7f09fb646640 1 -- start start 2023-10-30T14:20:48.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.541+0000 7f09fb646640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f09f4070740 0x7f09f40c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:48.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.541+0000 7f09fb646640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09f415e700 0x7f09f406a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:48.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.541+0000 7f09fb646640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f40c5810 con 0x7f09f41580e0 2023-10-30T14:20:48.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.541+0000 7f09fb646640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f415eb90 con 0x7f09f4070740 2023-10-30T14:20:48.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.541+0000 7f09fb646640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f40c6bb0 con 0x7f09f415e700 2023-10-30T14:20:48.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.541+0000 7f09f8be1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f09f4070740 0x7f09f40c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:48.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.541+0000 7f09f9be3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09f415e700 0x7f09f406a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:48.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.541+0000 7f09f8be1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f09f4070740 0x7f09f40c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:34964/0 (socket says 172.21.15.46:34964) 2023-10-30T14:20:48.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.541+0000 7f09f8be1640 1 -- 172.21.15.46:0/1335003265 learned_addr learned my addr 172.21.15.46:0/1335003265 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:48.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.545+0000 7f09f8be1640 1 -- 172.21.15.46:0/1335003265 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09f415e700 msgr2=0x7f09f406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:48.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.545+0000 7f09f8be1640 1 --2- 172.21.15.46:0/1335003265 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09f415e700 0x7f09f406a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:48.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.545+0000 7f09f8be1640 1 -- 172.21.15.46:0/1335003265 >> v1:172.21.15.154:6789/0 conn(0x7f09f41580e0 legacy=0x7f09f4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:48.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.545+0000 7f09f8be1640 1 -- 172.21.15.46:0/1335003265 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f40c6e70 con 0x7f09f4070740 2023-10-30T14:20:48.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.545+0000 7f09f9be3640 1 --2- 172.21.15.46:0/1335003265 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09f415e700 0x7f09f406a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:20:48.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.545+0000 7f09f8be1640 1 --2- 172.21.15.46:0/1335003265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09f4070740 0x7f09f40c6540 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f09e4009870 tx=0x7f09e401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=693e6227d8444a1d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:48.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09ebfff640 1 -- 172.21.15.46:0/1335003265 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e40a7070 con 0x7f09f4070740 2023-10-30T14:20:48.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09ebfff640 1 -- 172.21.15.46:0/1335003265 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e40ab430 con 0x7f09f4070740 2023-10-30T14:20:48.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09ebfff640 1 -- 172.21.15.46:0/1335003265 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e40ab7c0 con 0x7f09f4070740 2023-10-30T14:20:48.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09fb646640 1 -- 172.21.15.46:0/1335003265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09f4070740 msgr2=0x7f09f40c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:48.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09fb646640 1 --2- 172.21.15.46:0/1335003265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09f4070740 0x7f09f40c6540 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f09e4009870 tx=0x7f09e401c540 comp rx=0 tx=0).stop 2023-10-30T14:20:48.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09fb646640 1 -- 172.21.15.46:0/1335003265 shutdown_connections 2023-10-30T14:20:48.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09fb646640 1 --2- 172.21.15.46:0/1335003265 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09f415e700 0x7f09f406a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:48.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09fb646640 1 --2- 172.21.15.46:0/1335003265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09f4070740 0x7f09f40c6540 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:48.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09fb646640 1 -- 172.21.15.46:0/1335003265 >> 172.21.15.46:0/1335003265 conn(0x7f09f40c7910 msgr2=0x7f09f40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:48.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09fb646640 1 -- 172.21.15.46:0/1335003265 shutdown_connections 2023-10-30T14:20:48.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09fb646640 1 -- 172.21.15.46:0/1335003265 wait complete. 2023-10-30T14:20:48.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09fb646640 1 Processor -- start 2023-10-30T14:20:48.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.549+0000 7f09fb646640 1 -- start start 2023-10-30T14:20:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09fb646640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f4070740 0x7f09f41fa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09fb646640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09f41580e0 0x7f09f4155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:48.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09fb646640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09f415e700 0x7f09f4160110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:48.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09fb646640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f41fa830 con 0x7f09f4070740 2023-10-30T14:20:48.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09fb646640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f41f6370 con 0x7f09f41580e0 2023-10-30T14:20:48.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09fb646640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f41547b0 con 0x7f09f415e700 2023-10-30T14:20:48.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09f93e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f4070740 0x7f09f41fa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:48.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09f9be3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09f415e700 0x7f09f4160110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:48.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09f93e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f4070740 0x7f09f41fa2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52504/0 (socket says 172.21.15.46:52504) 2023-10-30T14:20:48.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09f8be1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09f41580e0 0x7f09f4155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:48.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09f93e2640 1 -- 172.21.15.46:0/3014465133 learned_addr learned my addr 172.21.15.46:0/3014465133 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:48.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09f93e2640 1 -- 172.21.15.46:0/3014465133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09f415e700 msgr2=0x7f09f4160110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:48.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09f93e2640 1 --2- 172.21.15.46:0/3014465133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09f415e700 0x7f09f4160110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:48.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09f93e2640 1 -- 172.21.15.46:0/3014465133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09f41580e0 msgr2=0x7f09f4155b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:48.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09f93e2640 1 --2- 172.21.15.46:0/3014465133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09f41580e0 0x7f09f4155b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:48.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09f93e2640 1 -- 172.21.15.46:0/3014465133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec009a70 con 0x7f09f4070740 2023-10-30T14:20:48.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09f93e2640 1 --2- 172.21.15.46:0/3014465133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f4070740 0x7f09f41fa2f0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f09f000c420 tx=0x7f09f000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:48.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09e9ffb640 1 -- 172.21.15.46:0/3014465133 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f00052d0 con 0x7f09f4070740 2023-10-30T14:20:48.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09e9ffb640 1 -- 172.21.15.46:0/3014465133 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f0005430 con 0x7f09f4070740 2023-10-30T14:20:48.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09fb646640 1 -- 172.21.15.46:0/3014465133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e40094d0 con 0x7f09f4070740 2023-10-30T14:20:48.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09e9ffb640 1 -- 172.21.15.46:0/3014465133 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0007260 con 0x7f09f4070740 2023-10-30T14:20:48.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09fb646640 1 -- 172.21.15.46:0/3014465133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f41f6910 con 0x7f09f4070740 2023-10-30T14:20:48.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.553+0000 7f09fb646640 1 -- 172.21.15.46:0/3014465133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f404e170 con 0x7f09f4070740 2023-10-30T14:20:48.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.557+0000 7f09e9ffb640 1 -- 172.21.15.46:0/3014465133 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09f00073c0 con 0x7f09f4070740 2023-10-30T14:20:48.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.557+0000 7f09e9ffb640 1 --2- 172.21.15.46:0/3014465133 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f09d80664a0 0x7f09d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:48.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.557+0000 7f09e9ffb640 1 -- 172.21.15.46:0/3014465133 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f09f0083d10 con 0x7f09f4070740 2023-10-30T14:20:48.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.557+0000 7f09f8be1640 1 --2- 172.21.15.46:0/3014465133 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f09d80664a0 0x7f09d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:48.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.557+0000 7f09f8be1640 1 --2- 172.21.15.46:0/3014465133 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f09d80664a0 0x7f09d8068960 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f09e40af000 tx=0x7f09e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:48.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.561+0000 7f09e9ffb640 1 -- 172.21.15.46:0/3014465133 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f09f00506b0 con 0x7f09f4070740 2023-10-30T14:20:48.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.765+0000 7f09fb646640 1 -- 172.21.15.46:0/3014465133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09f406aed0 con 0x7f09f4070740 2023-10-30T14:20:48.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.765+0000 7f09e9ffb640 1 -- 172.21.15.46:0/3014465133 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f09f0053cd0 con 0x7f09f4070740 2023-10-30T14:20:48.770 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:48.770 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:20:39.772439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:48.770 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 239 2023-10-30T14:20:48.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.769+0000 7f09fb646640 1 -- 172.21.15.46:0/3014465133 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f09d80664a0 msgr2=0x7f09d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:48.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.769+0000 7f09fb646640 1 --2- 172.21.15.46:0/3014465133 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f09d80664a0 0x7f09d8068960 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f09e40af000 tx=0x7f09e4008040 comp rx=0 tx=0).stop 2023-10-30T14:20:48.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.769+0000 7f09fb646640 1 -- 172.21.15.46:0/3014465133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f4070740 msgr2=0x7f09f41fa2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:48.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.769+0000 7f09fb646640 1 --2- 172.21.15.46:0/3014465133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f4070740 0x7f09f41fa2f0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f09f000c420 tx=0x7f09f000c820 comp rx=0 tx=0).stop 2023-10-30T14:20:48.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.769+0000 7f09fb646640 1 -- 172.21.15.46:0/3014465133 shutdown_connections 2023-10-30T14:20:48.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.769+0000 7f09fb646640 1 --2- 172.21.15.46:0/3014465133 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f09d80664a0 0x7f09d8068960 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:48.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.769+0000 7f09fb646640 1 --2- 172.21.15.46:0/3014465133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09f415e700 0x7f09f4160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:48.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.769+0000 7f09fb646640 1 --2- 172.21.15.46:0/3014465133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09f41580e0 0x7f09f4155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:48.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.769+0000 7f09fb646640 1 --2- 172.21.15.46:0/3014465133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f4070740 0x7f09f41fa2f0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:48.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.769+0000 7f09fb646640 1 -- 172.21.15.46:0/3014465133 >> 172.21.15.46:0/3014465133 conn(0x7f09f40c7910 msgr2=0x7f09f4156f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:48.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.769+0000 7f09fb646640 1 -- 172.21.15.46:0/3014465133 shutdown_connections 2023-10-30T14:20:48.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:48.769+0000 7f09fb646640 1 -- 172.21.15.46:0/3014465133 wait complete. 2023-10-30T14:20:48.792 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:50.793 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:50.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc241aea640 1 Processor -- start 2023-10-30T14:20:50.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc241aea640 1 -- start start 2023-10-30T14:20:50.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc241aea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc23c06f7a0 0x7fc23c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:50.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc241aea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc23c0c6bd0 0x7fc23c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:50.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc241aea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc23c071110 0x7fc23c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:50.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc241aea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23c0700c0 con 0x7fc23c06f7a0 2023-10-30T14:20:50.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc241aea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23c06a3d0 con 0x7fc23c071110 2023-10-30T14:20:50.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc241aea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c066cf0 con 0x7fc23c0c6bd0 2023-10-30T14:20:50.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc23b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc23c06f7a0 0x7fc23c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:50.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc23b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc23c06f7a0 0x7fc23c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52520/0 (socket says 172.21.15.46:52520) 2023-10-30T14:20:50.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc23b7fe640 1 -- 172.21.15.46:0/3272295922 learned_addr learned my addr 172.21.15.46:0/3272295922 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:50.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc23bfff640 1 --2- 172.21.15.46:0/3272295922 >> v2:172.21.15.154:3300/0 conn(0x7fc23c071110 0x7fc23c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:50.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.937+0000 7fc23affd640 1 --2- 172.21.15.46:0/3272295922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc23c0c6bd0 0x7fc23c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:50.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc23b7fe640 1 -- 172.21.15.46:0/3272295922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc23c0c6bd0 msgr2=0x7fc23c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:50.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc23b7fe640 1 --2- 172.21.15.46:0/3272295922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc23c0c6bd0 0x7fc23c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc23b7fe640 1 -- 172.21.15.46:0/3272295922 >> v2:172.21.15.154:3300/0 conn(0x7fc23c071110 msgr2=0x7fc23c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:50.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc23b7fe640 1 --2- 172.21.15.46:0/3272295922 >> v2:172.21.15.154:3300/0 conn(0x7fc23c071110 0x7fc23c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc23b7fe640 1 -- 172.21.15.46:0/3272295922 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c0c70d0 con 0x7fc23c06f7a0 2023-10-30T14:20:50.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc23b7fe640 1 --2- 172.21.15.46:0/3272295922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc23c06f7a0 0x7fc23c06fb80 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fc23c06c7d0 tx=0x7fc22c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e677b02b5dc2abf7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:50.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc23a7fc640 1 -- 172.21.15.46:0/3272295922 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0a89b0 con 0x7fc23c06f7a0 2023-10-30T14:20:50.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc23a7fc640 1 -- 172.21.15.46:0/3272295922 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c0a8b10 con 0x7fc23c06f7a0 2023-10-30T14:20:50.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc23a7fc640 1 -- 172.21.15.46:0/3272295922 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0a8ea0 con 0x7fc23c06f7a0 2023-10-30T14:20:50.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc241aea640 1 -- 172.21.15.46:0/3272295922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc23c06f7a0 msgr2=0x7fc23c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:50.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc241aea640 1 --2- 172.21.15.46:0/3272295922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc23c06f7a0 0x7fc23c06fb80 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fc23c06c7d0 tx=0x7fc22c01ecf0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc241aea640 1 -- 172.21.15.46:0/3272295922 shutdown_connections 2023-10-30T14:20:50.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc241aea640 1 --2- 172.21.15.46:0/3272295922 >> v2:172.21.15.154:3300/0 conn(0x7fc23c071110 0x7fc23c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc241aea640 1 --2- 172.21.15.46:0/3272295922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc23c0c6bd0 0x7fc23c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc241aea640 1 --2- 172.21.15.46:0/3272295922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc23c06f7a0 0x7fc23c06fb80 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.941+0000 7fc241aea640 1 -- 172.21.15.46:0/3272295922 >> 172.21.15.46:0/3272295922 conn(0x7fc23c0c7910 msgr2=0x7fc23c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:50.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc241aea640 1 -- 172.21.15.46:0/3272295922 shutdown_connections 2023-10-30T14:20:50.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc241aea640 1 -- 172.21.15.46:0/3272295922 wait complete. 2023-10-30T14:20:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc241aea640 1 Processor -- start 2023-10-30T14:20:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc241aea640 1 -- start start 2023-10-30T14:20:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc241aea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc23c06f7a0 0x7fc23c0c0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc241aea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc23c071110 0x7fc23c0c05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc241aea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc23c0c6bd0 0x7fc23c1496c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc241aea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c14df90 con 0x7fc23c06f7a0 2023-10-30T14:20:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc241aea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c14f880 con 0x7fc23c0c6bd0 2023-10-30T14:20:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc241aea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c14fb60 con 0x7fc23c071110 2023-10-30T14:20:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc23b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc23c06f7a0 0x7fc23c0c0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc23b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc23c06f7a0 0x7fc23c0c0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52522/0 (socket says 172.21.15.46:52522) 2023-10-30T14:20:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc23b7fe640 1 -- 172.21.15.46:0/844045797 learned_addr learned my addr 172.21.15.46:0/844045797 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc23bfff640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc23c0c6bd0 0x7fc23c1496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc23affd640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc23c071110 0x7fc23c0c05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc23b7fe640 1 -- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc23c071110 msgr2=0x7fc23c0c05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc23b7fe640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc23c071110 0x7fc23c0c05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc23b7fe640 1 -- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc23c0c6bd0 msgr2=0x7fc23c1496c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:50.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc23b7fe640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc23c0c6bd0 0x7fc23c1496c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:50.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc23b7fe640 1 -- 172.21.15.46:0/844045797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c0094d0 con 0x7fc23c06f7a0 2023-10-30T14:20:50.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc23bfff640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc23c0c6bd0 0x7fc23c1496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:20:50.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc23b7fe640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc23c06f7a0 0x7fc23c0c0070 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fc22c01d3a0 tx=0x7fc22c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:50.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.945+0000 7fc223fff640 1 -- 172.21.15.46:0/844045797 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0ba600 con 0x7fc23c06f7a0 2023-10-30T14:20:50.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.949+0000 7fc223fff640 1 -- 172.21.15.46:0/844045797 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c0ba760 con 0x7fc23c06f7a0 2023-10-30T14:20:50.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.949+0000 7fc223fff640 1 -- 172.21.15.46:0/844045797 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0bab10 con 0x7fc23c06f7a0 2023-10-30T14:20:50.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.949+0000 7fc241aea640 1 -- 172.21.15.46:0/844045797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c14e230 con 0x7fc23c06f7a0 2023-10-30T14:20:50.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.949+0000 7fc241aea640 1 -- 172.21.15.46:0/844045797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c14fe40 con 0x7fc23c06f7a0 2023-10-30T14:20:50.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.949+0000 7fc241aea640 1 -- 172.21.15.46:0/844045797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c14e6e0 con 0x7fc23c06f7a0 2023-10-30T14:20:50.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.949+0000 7fc223fff640 1 -- 172.21.15.46:0/844045797 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc22c0bac70 con 0x7fc23c06f7a0 2023-10-30T14:20:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.949+0000 7fc223fff640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc2280664a0 0x7fc228068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.949+0000 7fc223fff640 1 -- 172.21.15.46:0/844045797 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7fc22c122ff0 con 0x7fc23c06f7a0 2023-10-30T14:20:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.949+0000 7fc23affd640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc2280664a0 0x7fc228068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.953+0000 7fc23affd640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc2280664a0 0x7fc228068960 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0afaa0 tx=0x7fc230008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:50.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:50.953+0000 7fc223fff640 1 -- 172.21.15.46:0/844045797 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc23c14e6e0 con 0x7fc23c06f7a0 2023-10-30T14:20:51.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.145+0000 7fc241aea640 1 -- 172.21.15.46:0/844045797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc23c14e6e0 con 0x7fc23c06f7a0 2023-10-30T14:20:51.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.145+0000 7fc223fff640 1 -- 172.21.15.46:0/844045797 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+5731 (secure 0 0 0) 0x7fc23c14e6e0 con 0x7fc23c06f7a0 2023-10-30T14:20:51.151 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:51.152 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:20:50.223268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:resolve","state_seq":23,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:51.152 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 240 2023-10-30T14:20:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.149+0000 7fc241aea640 1 -- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc2280664a0 msgr2=0x7fc228068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.149+0000 7fc241aea640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc2280664a0 0x7fc228068960 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0afaa0 tx=0x7fc230008040 comp rx=0 tx=0).stop 2023-10-30T14:20:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.149+0000 7fc241aea640 1 -- 172.21.15.46:0/844045797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc23c06f7a0 msgr2=0x7fc23c0c0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.149+0000 7fc241aea640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc23c06f7a0 0x7fc23c0c0070 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fc22c01d3a0 tx=0x7fc22c01ba90 comp rx=0 tx=0).stop 2023-10-30T14:20:51.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.149+0000 7fc241aea640 1 -- 172.21.15.46:0/844045797 shutdown_connections 2023-10-30T14:20:51.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.149+0000 7fc241aea640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc2280664a0 0x7fc228068960 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:51.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.149+0000 7fc241aea640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc23c0c6bd0 0x7fc23c1496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:51.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.149+0000 7fc241aea640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc23c071110 0x7fc23c0c05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:51.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.149+0000 7fc241aea640 1 --2- 172.21.15.46:0/844045797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc23c06f7a0 0x7fc23c0c0070 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:51.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.149+0000 7fc241aea640 1 -- 172.21.15.46:0/844045797 >> 172.21.15.46:0/844045797 conn(0x7fc23c0c7910 msgr2=0x7fc23c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:51.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.149+0000 7fc241aea640 1 -- 172.21.15.46:0/844045797 shutdown_connections 2023-10-30T14:20:51.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:51.149+0000 7fc241aea640 1 -- 172.21.15.46:0/844045797 wait complete. 2023-10-30T14:20:51.172 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:53.172 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:53.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.321+0000 7f8432eac640 1 Processor -- start 2023-10-30T14:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8432eac640 1 -- start start 2023-10-30T14:20:53.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8432eac640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f842c0cf1a0 0x7f842c15a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8432eac640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0cb4c0 0x7f842c0c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8432eac640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f842c0cf580 0x7f842c1535c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8432eac640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f842c15d7a0 con 0x7f842c0cf1a0 2023-10-30T14:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8432eac640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f842c15dbc0 con 0x7f842c0cf580 2023-10-30T14:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8432eac640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842c15a660 con 0x7f842c0cb4c0 2023-10-30T14:20:53.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8431eaa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f842c0cf1a0 0x7f842c15a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:53.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8431eaa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f842c0cf1a0 0x7f842c15a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52538/0 (socket says 172.21.15.46:52538) 2023-10-30T14:20:53.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8431eaa640 1 -- 172.21.15.46:0/1593764615 learned_addr learned my addr 172.21.15.46:0/1593764615 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:53.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f84326ab640 1 --2- 172.21.15.46:0/1593764615 >> v2:172.21.15.154:3300/0 conn(0x7f842c0cf580 0x7f842c1535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:53.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f84316a9640 1 --2- 172.21.15.46:0/1593764615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0cb4c0 0x7f842c0c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:53.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8431eaa640 1 -- 172.21.15.46:0/1593764615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0cb4c0 msgr2=0x7f842c0c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:53.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8431eaa640 1 --2- 172.21.15.46:0/1593764615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0cb4c0 0x7f842c0c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8431eaa640 1 -- 172.21.15.46:0/1593764615 >> v2:172.21.15.154:3300/0 conn(0x7f842c0cf580 msgr2=0x7f842c1535c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:53.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8431eaa640 1 --2- 172.21.15.46:0/1593764615 >> v2:172.21.15.154:3300/0 conn(0x7f842c0cf580 0x7f842c1535c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.325+0000 7f8431eaa640 1 -- 172.21.15.46:0/1593764615 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c0ca8f0 con 0x7f842c0cf1a0 2023-10-30T14:20:53.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8431eaa640 1 --2- 172.21.15.46:0/1593764615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c0cf1a0 0x7f842c15a120 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f842c0cfd40 tx=0x7f842801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0b88c476536cf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8430ea8640 1 -- 172.21.15.46:0/1593764615 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84280a89b0 con 0x7f842c0cf1a0 2023-10-30T14:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8430ea8640 1 -- 172.21.15.46:0/1593764615 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84280a8b10 con 0x7f842c0cf1a0 2023-10-30T14:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8430ea8640 1 -- 172.21.15.46:0/1593764615 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84280a8ea0 con 0x7f842c0cf1a0 2023-10-30T14:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8432eac640 1 -- 172.21.15.46:0/1593764615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c0cf1a0 msgr2=0x7f842c15a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8432eac640 1 --2- 172.21.15.46:0/1593764615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c0cf1a0 0x7f842c15a120 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f842c0cfd40 tx=0x7f842801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8432eac640 1 -- 172.21.15.46:0/1593764615 shutdown_connections 2023-10-30T14:20:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8432eac640 1 --2- 172.21.15.46:0/1593764615 >> v2:172.21.15.154:3300/0 conn(0x7f842c0cf580 0x7f842c1535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8432eac640 1 --2- 172.21.15.46:0/1593764615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0cb4c0 0x7f842c0c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8432eac640 1 --2- 172.21.15.46:0/1593764615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c0cf1a0 0x7f842c15a120 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8432eac640 1 -- 172.21.15.46:0/1593764615 >> 172.21.15.46:0/1593764615 conn(0x7f842c06be40 msgr2=0x7f842c06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8432eac640 1 -- 172.21.15.46:0/1593764615 shutdown_connections 2023-10-30T14:20:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8432eac640 1 -- 172.21.15.46:0/1593764615 wait complete. 2023-10-30T14:20:53.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8432eac640 1 Processor -- start 2023-10-30T14:20:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.329+0000 7f8432eac640 1 -- start start 2023-10-30T14:20:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f8432eac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f842c0cb4c0 0x7f842c1f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f8432eac640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0cf1a0 0x7f842c1f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f8432eac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c0cf580 0x7f842c0b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f8432eac640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c0c81a0 con 0x7f842c0cf580 2023-10-30T14:20:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f8432eac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c15a040 con 0x7f842c0cb4c0 2023-10-30T14:20:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f8432eac640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842c1f20d0 con 0x7f842c0cf1a0 2023-10-30T14:20:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84326ab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c0cf580 0x7f842c0b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84326ab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c0cf580 0x7f842c0b4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52548/0 (socket says 172.21.15.46:52548) 2023-10-30T14:20:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84326ab640 1 -- 172.21.15.46:0/2428548101 learned_addr learned my addr 172.21.15.46:0/2428548101 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84316a9640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0cf1a0 0x7f842c1f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f8431eaa640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f842c0cb4c0 0x7f842c1f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84326ab640 1 -- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0cf1a0 msgr2=0x7f842c1f2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84326ab640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0cf1a0 0x7f842c1f2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84326ab640 1 -- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f842c0cb4c0 msgr2=0x7f842c1f2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:53.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84326ab640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f842c0cb4c0 0x7f842c1f2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84326ab640 1 -- 172.21.15.46:0/2428548101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84280094d0 con 0x7f842c0cf580 2023-10-30T14:20:53.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84326ab640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c0cf580 0x7f842c0b4290 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f8424007e60 tx=0x7f842400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:53.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84227fc640 1 -- 172.21.15.46:0/2428548101 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424014420 con 0x7f842c0cf580 2023-10-30T14:20:53.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84227fc640 1 -- 172.21.15.46:0/2428548101 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84240145a0 con 0x7f842c0cf580 2023-10-30T14:20:53.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f8432eac640 1 -- 172.21.15.46:0/2428548101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842c0c9000 con 0x7f842c0cf580 2023-10-30T14:20:53.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84227fc640 1 -- 172.21.15.46:0/2428548101 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424014950 con 0x7f842c0cf580 2023-10-30T14:20:53.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f8432eac640 1 -- 172.21.15.46:0/2428548101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842c15a250 con 0x7f842c0cf580 2023-10-30T14:20:53.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f8432eac640 1 -- 172.21.15.46:0/2428548101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842c1f2240 con 0x7f842c0cf580 2023-10-30T14:20:53.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.333+0000 7f84227fc640 1 -- 172.21.15.46:0/2428548101 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8424014ab0 con 0x7f842c0cf580 2023-10-30T14:20:53.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.337+0000 7f84227fc640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84100664a0 0x7f8410068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:53.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.337+0000 7f84227fc640 1 -- 172.21.15.46:0/2428548101 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f8424085300 con 0x7f842c0cf580 2023-10-30T14:20:53.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.337+0000 7f8431eaa640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84100664a0 0x7f8410068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:53.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.337+0000 7f8431eaa640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84100664a0 0x7f8410068960 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f84280b2000 tx=0x7f8428008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:53.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.341+0000 7f84227fc640 1 -- 172.21.15.46:0/2428548101 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f842c1f2240 con 0x7f842c0cf580 2023-10-30T14:20:53.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 -- 172.21.15.46:0/2428548101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f842c1f2240 con 0x7f842c0cf580 2023-10-30T14:20:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f84227fc640 1 -- 172.21.15.46:0/2428548101 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+5731 (secure 0 0 0) 0x7f842c1f2240 con 0x7f842c0cf580 2023-10-30T14:20:53.534 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:53.534 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:20:50.223268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:resolve","state_seq":23,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 240 2023-10-30T14:20:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 -- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84100664a0 msgr2=0x7f8410068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84100664a0 0x7f8410068960 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f84280b2000 tx=0x7f8428008040 comp rx=0 tx=0).stop 2023-10-30T14:20:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 -- 172.21.15.46:0/2428548101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c0cf580 msgr2=0x7f842c0b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c0cf580 0x7f842c0b4290 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f8424007e60 tx=0x7f842400e4b0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 -- 172.21.15.46:0/2428548101 shutdown_connections 2023-10-30T14:20:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f84100664a0 0x7f8410068960 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f842c0cf580 0x7f842c0b4290 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f842c0cf1a0 0x7f842c1f2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 --2- 172.21.15.46:0/2428548101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f842c0cb4c0 0x7f842c1f2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 -- 172.21.15.46:0/2428548101 >> 172.21.15.46:0/2428548101 conn(0x7f842c06be40 msgr2=0x7f842c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 -- 172.21.15.46:0/2428548101 shutdown_connections 2023-10-30T14:20:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:53.529+0000 7f8432eac640 1 -- 172.21.15.46:0/2428548101 wait complete. 2023-10-30T14:20:53.555 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 240, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 240, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9426": { "addr": "172.21.15.46:6833/3764280378", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 3764280378, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 3764280378, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 9426, "incarnation": 232, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 42 }, "gid_9594": { "addr": "172.21.15.46:6837/470164358", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 470164358, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 470164358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9594, "incarnation": 239, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:resolve", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 87, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:20:50.223268+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9426, "mds_1": 9594 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:20:55.555 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:55.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.705+0000 7efde5196640 1 Processor -- start 2023-10-30T14:20:55.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efde5196640 1 -- start start 2023-10-30T14:20:55.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efde5196640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efde01538f0 0x7efde0159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:55.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efde5196640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efde015cd50 0x7efde00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:55.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efde5196640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde015d300 con 0x7efde00cefa0 2023-10-30T14:20:55.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efde5196640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde015d5e0 con 0x7efde01538f0 2023-10-30T14:20:55.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efde5196640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde015d8c0 con 0x7efde015cd50 2023-10-30T14:20:55.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efddffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efde01538f0 0x7efde0159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:55.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efddffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efde01538f0 0x7efde0159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52550/0 (socket says 172.21.15.46:52550) 2023-10-30T14:20:55.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efde4995640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efde00cefa0 0x7efde00cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58750/0 (socket says 172.21.15.46:58750) 2023-10-30T14:20:55.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efddffff640 1 -- 172.21.15.46:0/3693479823 learned_addr learned my addr 172.21.15.46:0/3693479823 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:55.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efddf7fe640 1 --2- 172.21.15.46:0/3693479823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efde015cd50 0x7efde00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:55.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efddffff640 1 -- 172.21.15.46:0/3693479823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efde015cd50 msgr2=0x7efde00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:55.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efddffff640 1 --2- 172.21.15.46:0/3693479823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efde015cd50 0x7efde00c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:55.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efddffff640 1 -- 172.21.15.46:0/3693479823 >> v1:172.21.15.46:6789/0 conn(0x7efde00cefa0 legacy=0x7efde00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:55.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efddffff640 1 -- 172.21.15.46:0/3693479823 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde015dba0 con 0x7efde01538f0 2023-10-30T14:20:55.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.709+0000 7efddeffd640 1 -- 172.21.15.46:0/3693479823 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465346298 0 0) 0x7efde015d300 con 0x7efde00cefa0 2023-10-30T14:20:55.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efddffff640 1 --2- 172.21.15.46:0/3693479823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde01538f0 0x7efde0159fb0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009870 tx=0x7efdd001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=493f1f7279580054 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:55.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efddeffd640 1 -- 172.21.15.46:0/3693479823 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd00a8a30 con 0x7efde01538f0 2023-10-30T14:20:55.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efddeffd640 1 -- 172.21.15.46:0/3693479823 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd00a8b90 con 0x7efde01538f0 2023-10-30T14:20:55.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efddeffd640 1 -- 172.21.15.46:0/3693479823 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd00b0440 con 0x7efde01538f0 2023-10-30T14:20:55.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efde5196640 1 -- 172.21.15.46:0/3693479823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde01538f0 msgr2=0x7efde0159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:55.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efde5196640 1 --2- 172.21.15.46:0/3693479823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde01538f0 0x7efde0159fb0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009870 tx=0x7efdd001ed30 comp rx=0 tx=0).stop 2023-10-30T14:20:55.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efde5196640 1 -- 172.21.15.46:0/3693479823 shutdown_connections 2023-10-30T14:20:55.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efde5196640 1 --2- 172.21.15.46:0/3693479823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efde015cd50 0x7efde00c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:55.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efde5196640 1 --2- 172.21.15.46:0/3693479823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde01538f0 0x7efde0159fb0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:55.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efde5196640 1 -- 172.21.15.46:0/3693479823 >> 172.21.15.46:0/3693479823 conn(0x7efde006a640 msgr2=0x7efde0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:55.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efde5196640 1 -- 172.21.15.46:0/3693479823 shutdown_connections 2023-10-30T14:20:55.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efde5196640 1 -- 172.21.15.46:0/3693479823 wait complete. 2023-10-30T14:20:55.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.713+0000 7efde5196640 1 Processor -- start 2023-10-30T14:20:55.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efde5196640 1 -- start start 2023-10-30T14:20:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efde5196640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde00cefa0 0x7efde00b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efde5196640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde01538f0 0x7efde01efb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efde5196640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efde015cd50 0x7efde0147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efde5196640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde01f3480 con 0x7efde01538f0 2023-10-30T14:20:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efde5196640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde01ef0d0 con 0x7efde00cefa0 2023-10-30T14:20:55.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efde5196640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde01e6410 con 0x7efde015cd50 2023-10-30T14:20:55.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde01538f0 0x7efde01efb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:55.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde01538f0 0x7efde01efb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52554/0 (socket says 172.21.15.46:52554) 2023-10-30T14:20:55.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddf7fe640 1 -- 172.21.15.46:0/3556980261 learned_addr learned my addr 172.21.15.46:0/3556980261 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:55.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddffff640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde00cefa0 0x7efde00b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:55.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efde4995640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efde015cd50 0x7efde0147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:55.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddf7fe640 1 -- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efde015cd50 msgr2=0x7efde0147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:55.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddf7fe640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efde015cd50 0x7efde0147bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:55.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddf7fe640 1 -- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde00cefa0 msgr2=0x7efde00b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:55.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddf7fe640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde00cefa0 0x7efde00b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:55.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddf7fe640 1 -- 172.21.15.46:0/3556980261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd00094d0 con 0x7efde01538f0 2023-10-30T14:20:55.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddf7fe640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde01538f0 0x7efde01efb90 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7efdd400d5f0 tx=0x7efdd400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:55.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddcff9640 1 -- 172.21.15.46:0/3556980261 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd40125f0 con 0x7efde01538f0 2023-10-30T14:20:55.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddcff9640 1 -- 172.21.15.46:0/3556980261 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd4012750 con 0x7efde01538f0 2023-10-30T14:20:55.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efde5196640 1 -- 172.21.15.46:0/3556980261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde01e66f0 con 0x7efde01538f0 2023-10-30T14:20:55.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efddcff9640 1 -- 172.21.15.46:0/3556980261 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd4012ae0 con 0x7efde01538f0 2023-10-30T14:20:55.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efde5196640 1 -- 172.21.15.46:0/3556980261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde01ef390 con 0x7efde01538f0 2023-10-30T14:20:55.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.717+0000 7efde5196640 1 -- 172.21.15.46:0/3556980261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde004e170 con 0x7efde01538f0 2023-10-30T14:20:55.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.721+0000 7efddcff9640 1 -- 172.21.15.46:0/3556980261 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efdd4012c40 con 0x7efde01538f0 2023-10-30T14:20:55.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.721+0000 7efddcff9640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efdcc0664a0 0x7efdcc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:55.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.721+0000 7efddcff9640 1 -- 172.21.15.46:0/3556980261 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7efdd4084c70 con 0x7efde01538f0 2023-10-30T14:20:55.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.721+0000 7efddffff640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efdcc0664a0 0x7efdcc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:55.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.721+0000 7efddffff640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efdcc0664a0 0x7efdcc068960 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7efdd00b2000 tx=0x7efdd0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:55.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.725+0000 7efddcff9640 1 -- 172.21.15.46:0/3556980261 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efdd4051730 con 0x7efde01538f0 2023-10-30T14:20:55.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 -- 172.21.15.46:0/3556980261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efde0159a60 con 0x7efde01538f0 2023-10-30T14:20:55.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efddcff9640 1 -- 172.21.15.46:0/3556980261 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+5731 (secure 0 0 0) 0x7efdd4054d50 con 0x7efde01538f0 2023-10-30T14:20:55.926 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:55.926 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:20:50.223268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:resolve","state_seq":23,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:55.927 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 240 2023-10-30T14:20:55.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 -- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efdcc0664a0 msgr2=0x7efdcc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:55.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efdcc0664a0 0x7efdcc068960 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7efdd00b2000 tx=0x7efdd0009040 comp rx=0 tx=0).stop 2023-10-30T14:20:55.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 -- 172.21.15.46:0/3556980261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde01538f0 msgr2=0x7efde01efb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:55.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde01538f0 0x7efde01efb90 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7efdd400d5f0 tx=0x7efdd400dab0 comp rx=0 tx=0).stop 2023-10-30T14:20:55.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 -- 172.21.15.46:0/3556980261 shutdown_connections 2023-10-30T14:20:55.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efdcc0664a0 0x7efdcc068960 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:55.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efde015cd50 0x7efde0147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:55.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde01538f0 0x7efde01efb90 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:55.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 --2- 172.21.15.46:0/3556980261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde00cefa0 0x7efde00b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:55.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 -- 172.21.15.46:0/3556980261 >> 172.21.15.46:0/3556980261 conn(0x7efde006a640 msgr2=0x7efde00caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:55.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 -- 172.21.15.46:0/3556980261 shutdown_connections 2023-10-30T14:20:55.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:55.921+0000 7efde5196640 1 -- 172.21.15.46:0/3556980261 wait complete. 2023-10-30T14:20:55.950 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:20:57.951 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:20:58.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd16a11d640 1 Processor -- start 2023-10-30T14:20:58.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd16a11d640 1 -- start start 2023-10-30T14:20:58.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd16a11d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd1640c6bd0 0x7fd16406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:58.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd16a11d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd164071110 0x7fd164158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:58.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd16a11d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd164066cf0 con 0x7fd16406f7a0 2023-10-30T14:20:58.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd16a11d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1640700c0 con 0x7fd1640c6bd0 2023-10-30T14:20:58.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd16a11d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1640c70d0 con 0x7fd164071110 2023-10-30T14:20:58.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd1637fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd16406f7a0 0x7fd164069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33718/0 (socket says 172.21.15.46:33718) 2023-10-30T14:20:58.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd1637fe640 1 -- 172.21.15.46:0/2755570398 learned_addr learned my addr 172.21.15.46:0/2755570398 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:20:58.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd162ffd640 1 --2- 172.21.15.46:0/2755570398 >> v2:172.21.15.46:3300/0 conn(0x7fd1640c6bd0 0x7fd16406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:58.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd163fff640 1 --2- 172.21.15.46:0/2755570398 >> v2:172.21.15.154:3300/0 conn(0x7fd164071110 0x7fd164158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd162ffd640 1 -- 172.21.15.46:0/2755570398 >> v2:172.21.15.154:3300/0 conn(0x7fd164071110 msgr2=0x7fd164158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:20:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd162ffd640 1 --2- 172.21.15.46:0/2755570398 >> v2:172.21.15.154:3300/0 conn(0x7fd164071110 0x7fd164158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd162ffd640 1 -- 172.21.15.46:0/2755570398 >> v1:172.21.15.46:6789/0 conn(0x7fd16406f7a0 legacy=0x7fd164069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.101+0000 7fd162ffd640 1 -- 172.21.15.46:0/2755570398 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1640c7390 con 0x7fd1640c6bd0 2023-10-30T14:20:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.105+0000 7fd162ffd640 1 --2- 172.21.15.46:0/2755570398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1640c6bd0 0x7fd16406fb80 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fd16406cdd0 tx=0x7fd15801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2272e14cc168e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.105+0000 7fd1627fc640 1 -- 172.21.15.46:0/2755570398 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1580a8950 con 0x7fd1640c6bd0 2023-10-30T14:20:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.105+0000 7fd1627fc640 1 -- 172.21.15.46:0/2755570398 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1580a8ab0 con 0x7fd1640c6bd0 2023-10-30T14:20:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.105+0000 7fd1627fc640 1 -- 172.21.15.46:0/2755570398 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1580a8e40 con 0x7fd1640c6bd0 2023-10-30T14:20:58.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.105+0000 7fd16a11d640 1 -- 172.21.15.46:0/2755570398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1640c6bd0 msgr2=0x7fd16406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:58.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.105+0000 7fd16a11d640 1 --2- 172.21.15.46:0/2755570398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1640c6bd0 0x7fd16406fb80 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fd16406cdd0 tx=0x7fd15801ec50 comp rx=0 tx=0).stop 2023-10-30T14:20:58.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.105+0000 7fd16a11d640 1 -- 172.21.15.46:0/2755570398 shutdown_connections 2023-10-30T14:20:58.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.105+0000 7fd16a11d640 1 --2- 172.21.15.46:0/2755570398 >> v2:172.21.15.154:3300/0 conn(0x7fd164071110 0x7fd164158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:58.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.105+0000 7fd16a11d640 1 --2- 172.21.15.46:0/2755570398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1640c6bd0 0x7fd16406fb80 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:58.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.105+0000 7fd16a11d640 1 -- 172.21.15.46:0/2755570398 >> 172.21.15.46:0/2755570398 conn(0x7fd1640c7910 msgr2=0x7fd1640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:58.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.105+0000 7fd16a11d640 1 -- 172.21.15.46:0/2755570398 shutdown_connections 2023-10-30T14:20:58.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.105+0000 7fd16a11d640 1 -- 172.21.15.46:0/2755570398 wait complete. 2023-10-30T14:20:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd16a11d640 1 Processor -- start 2023-10-30T14:20:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd16a11d640 1 -- start start 2023-10-30T14:20:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd16a11d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd16406f7a0 0x7fd1640c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:58.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd16a11d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd164071110 0x7fd1641545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:58.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd16a11d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1640c6bd0 0x7fd1641eeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:58.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd16a11d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641eea20 con 0x7fd164071110 2023-10-30T14:20:58.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd16a11d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641e2f20 con 0x7fd1640c6bd0 2023-10-30T14:20:58.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd16a11d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641f2200 con 0x7fd16406f7a0 2023-10-30T14:20:58.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd162ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd164071110 0x7fd1641545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:58.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd162ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd164071110 0x7fd1641545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54048/0 (socket says 172.21.15.46:54048) 2023-10-30T14:20:58.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd162ffd640 1 -- 172.21.15.46:0/3949028978 learned_addr learned my addr 172.21.15.46:0/3949028978 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:20:58.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd163fff640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1640c6bd0 0x7fd1641eeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:58.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd1637fe640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd16406f7a0 0x7fd1640c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:58.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd162ffd640 1 -- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd16406f7a0 msgr2=0x7fd1640c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:58.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd162ffd640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd16406f7a0 0x7fd1640c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:58.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd162ffd640 1 -- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1640c6bd0 msgr2=0x7fd1641eeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:58.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd162ffd640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1640c6bd0 0x7fd1641eeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:58.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd162ffd640 1 -- 172.21.15.46:0/3949028978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd158009420 con 0x7fd164071110 2023-10-30T14:20:58.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.109+0000 7fd162ffd640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd164071110 0x7fd1641545e0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fd15801b860 tx=0x7fd15801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:58.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.113+0000 7fd14bfff640 1 -- 172.21.15.46:0/3949028978 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1580bb640 con 0x7fd164071110 2023-10-30T14:20:58.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.113+0000 7fd14bfff640 1 -- 172.21.15.46:0/3949028978 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1580bb7a0 con 0x7fd164071110 2023-10-30T14:20:58.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.113+0000 7fd14bfff640 1 -- 172.21.15.46:0/3949028978 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1580bbb10 con 0x7fd164071110 2023-10-30T14:20:58.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.113+0000 7fd16a11d640 1 -- 172.21.15.46:0/3949028978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1641f24e0 con 0x7fd164071110 2023-10-30T14:20:58.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.113+0000 7fd16a11d640 1 -- 172.21.15.46:0/3949028978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1641f2aa0 con 0x7fd164071110 2023-10-30T14:20:58.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.113+0000 7fd16a11d640 1 -- 172.21.15.46:0/3949028978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1641eeb90 con 0x7fd164071110 2023-10-30T14:20:58.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.113+0000 7fd14bfff640 1 -- 172.21.15.46:0/3949028978 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd1580b7040 con 0x7fd164071110 2023-10-30T14:20:58.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.113+0000 7fd14bfff640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1500664a0 0x7fd150068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:20:58.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.113+0000 7fd14bfff640 1 -- 172.21.15.46:0/3949028978 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7fd158122d50 con 0x7fd164071110 2023-10-30T14:20:58.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.113+0000 7fd1637fe640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1500664a0 0x7fd150068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:20:58.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.117+0000 7fd1637fe640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1500664a0 0x7fd150068960 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fd154005890 tx=0x7fd154009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:20:58.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.117+0000 7fd14bfff640 1 -- 172.21.15.46:0/3949028978 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd1580ef5e0 con 0x7fd164071110 2023-10-30T14:20:58.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.305+0000 7fd16a11d640 1 -- 172.21.15.46:0/3949028978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd16404e170 con 0x7fd164071110 2023-10-30T14:20:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd14bfff640 1 -- 172.21.15.46:0/3949028978 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fd1580b5070 con 0x7fd164071110 2023-10-30T14:20:58.312 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:20:58.312 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":241,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:20:55.886549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:20:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 241 2023-10-30T14:20:58.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd16a11d640 1 -- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1500664a0 msgr2=0x7fd150068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:58.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd16a11d640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1500664a0 0x7fd150068960 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fd154005890 tx=0x7fd154009040 comp rx=0 tx=0).stop 2023-10-30T14:20:58.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd16a11d640 1 -- 172.21.15.46:0/3949028978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd164071110 msgr2=0x7fd1641545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:20:58.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd16a11d640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd164071110 0x7fd1641545e0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fd15801b860 tx=0x7fd15801b9f0 comp rx=0 tx=0).stop 2023-10-30T14:20:58.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd16a11d640 1 -- 172.21.15.46:0/3949028978 shutdown_connections 2023-10-30T14:20:58.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd16a11d640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd1500664a0 0x7fd150068960 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:58.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd16a11d640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1640c6bd0 0x7fd1641eeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:58.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd16a11d640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd164071110 0x7fd1641545e0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:58.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd16a11d640 1 --2- 172.21.15.46:0/3949028978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd16406f7a0 0x7fd1640c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:20:58.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd16a11d640 1 -- 172.21.15.46:0/3949028978 >> 172.21.15.46:0/3949028978 conn(0x7fd1640c7910 msgr2=0x7fd1640c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:20:58.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd16a11d640 1 -- 172.21.15.46:0/3949028978 shutdown_connections 2023-10-30T14:20:58.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:20:58.309+0000 7fd16a11d640 1 -- 172.21.15.46:0/3949028978 wait complete. 2023-10-30T14:20:58.333 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:21:00.334 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:00.335 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:21:00.341 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:21:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.485+0000 7f51ec46b640 1 Processor -- start 2023-10-30T14:21:00.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.485+0000 7f51ec46b640 1 -- start start 2023-10-30T14:21:00.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.485+0000 7f51ec46b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f51e40c6b80 0x7f51e40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:00.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.485+0000 7f51ec46b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c6f60 0x7f51e40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:00.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.485+0000 7f51ec46b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e40c0a50 con 0x7f51e40c8430 2023-10-30T14:21:00.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.485+0000 7f51ec46b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e406f6e0 con 0x7f51e40c6b80 2023-10-30T14:21:00.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.485+0000 7f51ec46b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e406f9c0 con 0x7f51e40c6f60 2023-10-30T14:21:00.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.489+0000 7f51e9a06640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c6f60 0x7f51e40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:00.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.489+0000 7f51ea207640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f51e40c6b80 0x7f51e40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:00.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.489+0000 7f51e9a06640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c6f60 0x7f51e40c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:37984/0 (socket says 172.21.15.46:37984) 2023-10-30T14:21:00.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.489+0000 7f51e9a06640 1 -- 172.21.15.46:0/2336802872 learned_addr learned my addr 172.21.15.46:0/2336802872 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:21:00.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.489+0000 7f51eaa08640 1 --1- 172.21.15.46:0/2336802872 >> v1:172.21.15.154:6789/0 conn(0x7f51e40c8430 0x7f51e413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:51264/0 (socket says 172.21.15.46:51264) 2023-10-30T14:21:00.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.489+0000 7f51ea207640 1 -- 172.21.15.46:0/2336802872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c6f60 msgr2=0x7f51e40c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:00.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.489+0000 7f51ea207640 1 --2- 172.21.15.46:0/2336802872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c6f60 0x7f51e40c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:00.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.489+0000 7f51ea207640 1 -- 172.21.15.46:0/2336802872 >> v1:172.21.15.154:6789/0 conn(0x7f51e40c8430 legacy=0x7f51e413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:00.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.489+0000 7f51ea207640 1 -- 172.21.15.46:0/2336802872 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e406fca0 con 0x7f51e40c6b80 2023-10-30T14:21:00.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.489+0000 7f51e9a06640 1 --2- 172.21.15.46:0/2336802872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c6f60 0x7f51e40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:21:00.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.489+0000 7f51ea207640 1 --2- 172.21.15.46:0/2336802872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51e40c6b80 0x7f51e40bfec0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f51e00097c0 tx=0x7f51e001c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1526d79afd8efb27 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:00.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51e9205640 1 -- 172.21.15.46:0/2336802872 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e00ab600 con 0x7f51e40c6b80 2023-10-30T14:21:00.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51e9205640 1 -- 172.21.15.46:0/2336802872 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e00a7050 con 0x7f51e40c6b80 2023-10-30T14:21:00.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51e9205640 1 -- 172.21.15.46:0/2336802872 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e00ab980 con 0x7f51e40c6b80 2023-10-30T14:21:00.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 -- 172.21.15.46:0/2336802872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51e40c6b80 msgr2=0x7f51e40bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:00.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 --2- 172.21.15.46:0/2336802872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51e40c6b80 0x7f51e40bfec0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f51e00097c0 tx=0x7f51e001c3f0 comp rx=0 tx=0).stop 2023-10-30T14:21:00.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 -- 172.21.15.46:0/2336802872 shutdown_connections 2023-10-30T14:21:00.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 --2- 172.21.15.46:0/2336802872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c6f60 0x7f51e40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:00.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 --2- 172.21.15.46:0/2336802872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51e40c6b80 0x7f51e40bfec0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:00.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 -- 172.21.15.46:0/2336802872 >> 172.21.15.46:0/2336802872 conn(0x7f51e40c7910 msgr2=0x7f51e40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:00.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 -- 172.21.15.46:0/2336802872 shutdown_connections 2023-10-30T14:21:00.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 -- 172.21.15.46:0/2336802872 wait complete. 2023-10-30T14:21:00.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 Processor -- start 2023-10-30T14:21:00.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 -- start start 2023-10-30T14:21:00.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51e40c6b80 0x7f51e4157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:00.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51e40c6f60 0x7f51e41f5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:00.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c8430 0x7f51e413cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:00.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e415fb20 con 0x7f51e40c6f60 2023-10-30T14:21:00.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e41f52d0 con 0x7f51e40c6b80 2023-10-30T14:21:00.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51ec46b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e415fd50 con 0x7f51e40c8430 2023-10-30T14:21:00.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51e9a06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51e40c6f60 0x7f51e41f5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:00.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51e9a06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51e40c6f60 0x7f51e41f5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54056/0 (socket says 172.21.15.46:54056) 2023-10-30T14:21:00.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.493+0000 7f51e9a06640 1 -- 172.21.15.46:0/3205394973 learned_addr learned my addr 172.21.15.46:0/3205394973 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:21:00.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51ea207640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51e40c6b80 0x7f51e4157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:00.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51eaa08640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c8430 0x7f51e413cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:00.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51e9a06640 1 -- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c8430 msgr2=0x7f51e413cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:00.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51e9a06640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c8430 0x7f51e413cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:00.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51e9a06640 1 -- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51e40c6b80 msgr2=0x7f51e4157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:00.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51e9a06640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51e40c6b80 0x7f51e4157e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:00.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51e9a06640 1 -- 172.21.15.46:0/3205394973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40094d0 con 0x7f51e40c6f60 2023-10-30T14:21:00.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51eaa08640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c8430 0x7f51e413cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:21:00.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51ea207640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51e40c6b80 0x7f51e4157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:21:00.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51e9a06640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51e40c6f60 0x7f51e41f5d20 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f51d400e9d0 tx=0x7f51d400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:00.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51daffd640 1 -- 172.21.15.46:0/3205394973 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d400ca60 con 0x7f51e40c6f60 2023-10-30T14:21:00.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51daffd640 1 -- 172.21.15.46:0/3205394973 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d400cbc0 con 0x7f51e40c6f60 2023-10-30T14:21:00.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51daffd640 1 -- 172.21.15.46:0/3205394973 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d4013470 con 0x7f51e40c6f60 2023-10-30T14:21:00.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51ec46b640 1 -- 172.21.15.46:0/3205394973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e0009420 con 0x7f51e40c6f60 2023-10-30T14:21:00.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51ec46b640 1 -- 172.21.15.46:0/3205394973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e413b980 con 0x7f51e40c6f60 2023-10-30T14:21:00.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51ec46b640 1 -- 172.21.15.46:0/3205394973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e404e170 con 0x7f51e40c6f60 2023-10-30T14:21:00.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51daffd640 1 -- 172.21.15.46:0/3205394973 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51d40135d0 con 0x7f51e40c6f60 2023-10-30T14:21:00.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51daffd640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51c80664a0 0x7f51c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:00.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.497+0000 7f51daffd640 1 -- 172.21.15.46:0/3205394973 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f51d4084cf0 con 0x7f51e40c6f60 2023-10-30T14:21:00.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.501+0000 7f51ea207640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51c80664a0 0x7f51c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:00.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.501+0000 7f51ea207640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51c80664a0 0x7f51c8068960 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f51e00ad000 tx=0x7f51e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:00.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.501+0000 7f51daffd640 1 -- 172.21.15.46:0/3205394973 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51d4051610 con 0x7f51e40c6f60 2023-10-30T14:21:00.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 -- 172.21.15.46:0/3205394973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51e415ed80 con 0x7f51e40c6f60 2023-10-30T14:21:00.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51daffd640 1 -- 172.21.15.46:0/3205394973 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f51d4021090 con 0x7f51e40c6f60 2023-10-30T14:21:00.697 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:21:00.697 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":241,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:20:55.886549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:00.698 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 241 2023-10-30T14:21:00.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 -- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51c80664a0 msgr2=0x7f51c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:00.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51c80664a0 0x7f51c8068960 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f51e00ad000 tx=0x7f51e0008040 comp rx=0 tx=0).stop 2023-10-30T14:21:00.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 -- 172.21.15.46:0/3205394973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51e40c6f60 msgr2=0x7f51e41f5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:00.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51e40c6f60 0x7f51e41f5d20 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f51d400e9d0 tx=0x7f51d400ee90 comp rx=0 tx=0).stop 2023-10-30T14:21:00.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 -- 172.21.15.46:0/3205394973 shutdown_connections 2023-10-30T14:21:00.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f51c80664a0 0x7f51c8068960 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:00.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51e40c8430 0x7f51e413cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:00.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51e40c6f60 0x7f51e41f5d20 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:00.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 --2- 172.21.15.46:0/3205394973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51e40c6b80 0x7f51e4157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:00.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 -- 172.21.15.46:0/3205394973 >> 172.21.15.46:0/3205394973 conn(0x7f51e40c7910 msgr2=0x7f51e40c1590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:00.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 -- 172.21.15.46:0/3205394973 shutdown_connections 2023-10-30T14:21:00.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:00.693+0000 7f51ec46b640 1 -- 172.21.15.46:0/3205394973 wait complete. 2023-10-30T14:21:00.719 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:21:02.719 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:02.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.869+0000 7f8244009640 1 Processor -- start 2023-10-30T14:21:02.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.869+0000 7f8244009640 1 -- start start 2023-10-30T14:21:02.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f8244009640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f823c0c05d0 0x7f823c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:02.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f8244009640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f823c0c6f60 0x7f823c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f8244009640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823c0c0a50 con 0x7f823c0c6b80 2023-10-30T14:21:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f8244009640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f823c06f6e0 con 0x7f823c0c6f60 2023-10-30T14:21:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f8244009640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823c06f9c0 con 0x7f823c0c05d0 2023-10-30T14:21:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f82425a6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f823c0c6f60 0x7f823c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f82425a6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f823c0c6f60 0x7f823c13f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54066/0 (socket says 172.21.15.46:54066) 2023-10-30T14:21:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f82425a6640 1 -- 172.21.15.46:0/229422174 learned_addr learned my addr 172.21.15.46:0/229422174 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:21:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f82415a4640 1 --2- 172.21.15.46:0/229422174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f823c0c05d0 0x7f823c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f82425a6640 1 -- 172.21.15.46:0/229422174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f823c0c05d0 msgr2=0x7f823c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f82425a6640 1 --2- 172.21.15.46:0/229422174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f823c0c05d0 0x7f823c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f82425a6640 1 -- 172.21.15.46:0/229422174 >> v1:172.21.15.46:6789/0 conn(0x7f823c0c6b80 legacy=0x7f823c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:02.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.873+0000 7f82425a6640 1 -- 172.21.15.46:0/229422174 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c06fca0 con 0x7f823c0c6f60 2023-10-30T14:21:02.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f82425a6640 1 --2- 172.21.15.46:0/229422174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f823c0c6f60 0x7f823c13f2d0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f8234009e50 tx=0x7f823401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8017529ef6a544 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f8240da3640 1 -- 172.21.15.46:0/229422174 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82340a57f0 con 0x7f823c0c6f60 2023-10-30T14:21:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f8240da3640 1 -- 172.21.15.46:0/229422174 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82340a5950 con 0x7f823c0c6f60 2023-10-30T14:21:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f8240da3640 1 -- 172.21.15.46:0/229422174 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82340a5ce0 con 0x7f823c0c6f60 2023-10-30T14:21:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f8244009640 1 -- 172.21.15.46:0/229422174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f823c0c6f60 msgr2=0x7f823c13f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f8244009640 1 --2- 172.21.15.46:0/229422174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f823c0c6f60 0x7f823c13f2d0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f8234009e50 tx=0x7f823401ca10 comp rx=0 tx=0).stop 2023-10-30T14:21:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f8244009640 1 -- 172.21.15.46:0/229422174 shutdown_connections 2023-10-30T14:21:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f8244009640 1 --2- 172.21.15.46:0/229422174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f823c0c6f60 0x7f823c13f2d0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f8244009640 1 --2- 172.21.15.46:0/229422174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f823c0c05d0 0x7f823c13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f8244009640 1 -- 172.21.15.46:0/229422174 >> 172.21.15.46:0/229422174 conn(0x7f823c0c7910 msgr2=0x7f823c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f8244009640 1 -- 172.21.15.46:0/229422174 shutdown_connections 2023-10-30T14:21:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f8244009640 1 -- 172.21.15.46:0/229422174 wait complete. 2023-10-30T14:21:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.877+0000 7f8244009640 1 Processor -- start 2023-10-30T14:21:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f8244009640 1 -- start start 2023-10-30T14:21:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f8244009640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f823c0c05d0 0x7f823c0af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f8244009640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f823c0c6b80 0x7f823c0af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f8244009640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f823c0c6f60 0x7f823c0c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f8244009640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c13e680 con 0x7f823c0c6f60 2023-10-30T14:21:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f8244009640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c155910 con 0x7f823c0c05d0 2023-10-30T14:21:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f8244009640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823c13c820 con 0x7f823c0c6b80 2023-10-30T14:21:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82425a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f823c0c6f60 0x7f823c0c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82425a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f823c0c6f60 0x7f823c0c8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54080/0 (socket says 172.21.15.46:54080) 2023-10-30T14:21:02.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82425a6640 1 -- 172.21.15.46:0/3535768645 learned_addr learned my addr 172.21.15.46:0/3535768645 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:21:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f8241da5640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f823c0c05d0 0x7f823c0af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82415a4640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f823c0c6b80 0x7f823c0af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82425a6640 1 -- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f823c0c6b80 msgr2=0x7f823c0af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82425a6640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f823c0c6b80 0x7f823c0af720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82425a6640 1 -- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f823c0c05d0 msgr2=0x7f823c0af1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82425a6640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f823c0c05d0 0x7f823c0af1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82425a6640 1 -- 172.21.15.46:0/3535768645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8234009b30 con 0x7f823c0c6f60 2023-10-30T14:21:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82425a6640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f823c0c6f60 0x7f823c0c8ca0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f823401a640 tx=0x7f82340197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:02.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82327fc640 1 -- 172.21.15.46:0/3535768645 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82340b8880 con 0x7f823c0c6f60 2023-10-30T14:21:02.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82327fc640 1 -- 172.21.15.46:0/3535768645 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82340b89e0 con 0x7f823c0c6f60 2023-10-30T14:21:02.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f8244009640 1 -- 172.21.15.46:0/3535768645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f823c06ad90 con 0x7f823c0c6f60 2023-10-30T14:21:02.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f82327fc640 1 -- 172.21.15.46:0/3535768645 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82340b8d90 con 0x7f823c0c6f60 2023-10-30T14:21:02.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f8244009640 1 -- 172.21.15.46:0/3535768645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f823c13ca30 con 0x7f823c0c6f60 2023-10-30T14:21:02.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.881+0000 7f8244009640 1 -- 172.21.15.46:0/3535768645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f823c13e7c0 con 0x7f823c0c6f60 2023-10-30T14:21:02.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.885+0000 7f82327fc640 1 -- 172.21.15.46:0/3535768645 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82340c3460 con 0x7f823c0c6f60 2023-10-30T14:21:02.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.885+0000 7f82327fc640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82200664a0 0x7f8220068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:02.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.885+0000 7f82327fc640 1 -- 172.21.15.46:0/3535768645 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f8234120370 con 0x7f823c0c6f60 2023-10-30T14:21:02.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.885+0000 7f8241da5640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82200664a0 0x7f8220068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:02.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.885+0000 7f8241da5640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82200664a0 0x7f8220068960 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8238000f70 tx=0x7f8238009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:02.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:02.889+0000 7f82327fc640 1 -- 172.21.15.46:0/3535768645 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82340ecef0 con 0x7f823c0c6f60 2023-10-30T14:21:03.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 -- 172.21.15.46:0/3535768645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f823c04e170 con 0x7f823c0c6f60 2023-10-30T14:21:03.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f82327fc640 1 -- 172.21.15.46:0/3535768645 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f82340f0510 con 0x7f823c0c6f60 2023-10-30T14:21:03.093 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:21:03.093 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:21:00.940934+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:03.093 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 242 2023-10-30T14:21:03.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 -- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82200664a0 msgr2=0x7f8220068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:03.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82200664a0 0x7f8220068960 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8238000f70 tx=0x7f8238009040 comp rx=0 tx=0).stop 2023-10-30T14:21:03.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 -- 172.21.15.46:0/3535768645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f823c0c6f60 msgr2=0x7f823c0c8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f823c0c6f60 0x7f823c0c8ca0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f823401a640 tx=0x7f82340197b0 comp rx=0 tx=0).stop 2023-10-30T14:21:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 -- 172.21.15.46:0/3535768645 shutdown_connections 2023-10-30T14:21:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f82200664a0 0x7f8220068960 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f823c0c6f60 0x7f823c0c8ca0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f823c0c6b80 0x7f823c0af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:03.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 --2- 172.21.15.46:0/3535768645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f823c0c05d0 0x7f823c0af1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:03.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 -- 172.21.15.46:0/3535768645 >> 172.21.15.46:0/3535768645 conn(0x7f823c0c7910 msgr2=0x7f823c0c1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:03.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 -- 172.21.15.46:0/3535768645 shutdown_connections 2023-10-30T14:21:03.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:03.089+0000 7f8244009640 1 -- 172.21.15.46:0/3535768645 wait complete. 2023-10-30T14:21:03.115 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:21:05.115 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:05.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.265+0000 7f94aa9c5640 1 Processor -- start 2023-10-30T14:21:05.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.265+0000 7f94aa9c5640 1 -- start start 2023-10-30T14:21:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.265+0000 7f94aa9c5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f94a40c05d0 0x7f94a413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.265+0000 7f94aa9c5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94a40c6f60 0x7f94a413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.265+0000 7f94aa9c5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a40c0a50 con 0x7f94a40c6b80 2023-10-30T14:21:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.265+0000 7f94aa9c5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a406f6e0 con 0x7f94a40c6f60 2023-10-30T14:21:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.265+0000 7f94aa9c5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a406f9c0 con 0x7f94a40c05d0 2023-10-30T14:21:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.265+0000 7f94a3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f94a40c6b80 0x7f94a40bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33726/0 (socket says 172.21.15.46:33726) 2023-10-30T14:21:05.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.265+0000 7f94a8f62640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94a40c6f60 0x7f94a413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:05.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.265+0000 7f94a3fff640 1 -- 172.21.15.46:0/1389673177 learned_addr learned my addr 172.21.15.46:0/1389673177 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:21:05.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.265+0000 7f94a37fe640 1 --2- 172.21.15.46:0/1389673177 >> v2:172.21.15.154:3300/0 conn(0x7f94a40c05d0 0x7f94a413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:05.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94a8f62640 1 -- 172.21.15.46:0/1389673177 >> v2:172.21.15.154:3300/0 conn(0x7f94a40c05d0 msgr2=0x7f94a413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:05.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94a8f62640 1 --2- 172.21.15.46:0/1389673177 >> v2:172.21.15.154:3300/0 conn(0x7f94a40c05d0 0x7f94a413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:05.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94a8f62640 1 -- 172.21.15.46:0/1389673177 >> v1:172.21.15.46:6789/0 conn(0x7f94a40c6b80 legacy=0x7f94a40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:05.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94a8f62640 1 -- 172.21.15.46:0/1389673177 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a406fca0 con 0x7f94a40c6f60 2023-10-30T14:21:05.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94a2ffd640 1 -- 172.21.15.46:0/1389673177 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694124797 0 0) 0x7f94a40c0a50 con 0x7f94a40c6b80 2023-10-30T14:21:05.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94a8f62640 1 --2- 172.21.15.46:0/1389673177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94a40c6f60 0x7f94a413f2d0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f94a406c7e0 tx=0x7f949c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3059643ab8219d4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:05.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94a2ffd640 1 -- 172.21.15.46:0/1389673177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c0a58e0 con 0x7f94a40c6f60 2023-10-30T14:21:05.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94a2ffd640 1 -- 172.21.15.46:0/1389673177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949c0a5a40 con 0x7f94a40c6f60 2023-10-30T14:21:05.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94a2ffd640 1 -- 172.21.15.46:0/1389673177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c0a5dd0 con 0x7f94a40c6f60 2023-10-30T14:21:05.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94aa9c5640 1 -- 172.21.15.46:0/1389673177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94a40c6f60 msgr2=0x7f94a413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:05.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94aa9c5640 1 --2- 172.21.15.46:0/1389673177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94a40c6f60 0x7f94a413f2d0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f94a406c7e0 tx=0x7f949c01c910 comp rx=0 tx=0).stop 2023-10-30T14:21:05.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94aa9c5640 1 -- 172.21.15.46:0/1389673177 shutdown_connections 2023-10-30T14:21:05.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94aa9c5640 1 --2- 172.21.15.46:0/1389673177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94a40c6f60 0x7f94a413f2d0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:05.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94aa9c5640 1 --2- 172.21.15.46:0/1389673177 >> v2:172.21.15.154:3300/0 conn(0x7f94a40c05d0 0x7f94a413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:05.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94aa9c5640 1 -- 172.21.15.46:0/1389673177 >> 172.21.15.46:0/1389673177 conn(0x7f94a40c7910 msgr2=0x7f94a40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:05.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94aa9c5640 1 -- 172.21.15.46:0/1389673177 shutdown_connections 2023-10-30T14:21:05.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.269+0000 7f94aa9c5640 1 -- 172.21.15.46:0/1389673177 wait complete. 2023-10-30T14:21:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94aa9c5640 1 Processor -- start 2023-10-30T14:21:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94aa9c5640 1 -- start start 2023-10-30T14:21:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94aa9c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94a40c05d0 0x7f94a41593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94aa9c5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a40c6b80 0x7f94a41fa020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94aa9c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94a40c6f60 0x7f94a4139e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94aa9c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a413d5b0 con 0x7f94a40c05d0 2023-10-30T14:21:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94aa9c5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a41fa560 con 0x7f94a40c6b80 2023-10-30T14:21:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94aa9c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a413c950 con 0x7f94a40c6f60 2023-10-30T14:21:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94a40c05d0 0x7f94a41593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94a40c05d0 0x7f94a41593c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54102/0 (socket says 172.21.15.46:54102) 2023-10-30T14:21:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a3fff640 1 -- 172.21.15.46:0/3199983378 learned_addr learned my addr 172.21.15.46:0/3199983378 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:21:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a37fe640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a40c6b80 0x7f94a41fa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a8f62640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94a40c6f60 0x7f94a4139e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a3fff640 1 -- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94a40c6f60 msgr2=0x7f94a4139e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a3fff640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94a40c6f60 0x7f94a4139e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a3fff640 1 -- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a40c6b80 msgr2=0x7f94a41fa020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:05.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a3fff640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a40c6b80 0x7f94a41fa020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:05.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a3fff640 1 -- 172.21.15.46:0/3199983378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949c009a70 con 0x7f94a40c05d0 2023-10-30T14:21:05.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a3fff640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94a40c05d0 0x7f94a41593c0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f94940011f0 tx=0x7f949400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:05.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a0ff9640 1 -- 172.21.15.46:0/3199983378 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94940074e0 con 0x7f94a40c05d0 2023-10-30T14:21:05.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.273+0000 7f94a0ff9640 1 -- 172.21.15.46:0/3199983378 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9494007640 con 0x7f94a40c05d0 2023-10-30T14:21:05.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.277+0000 7f94aa9c5640 1 -- 172.21.15.46:0/3199983378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a413cc10 con 0x7f94a40c05d0 2023-10-30T14:21:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.277+0000 7f94a0ff9640 1 -- 172.21.15.46:0/3199983378 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9494004a10 con 0x7f94a40c05d0 2023-10-30T14:21:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.277+0000 7f94aa9c5640 1 -- 172.21.15.46:0/3199983378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a413d1d0 con 0x7f94a40c05d0 2023-10-30T14:21:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.277+0000 7f94aa9c5640 1 -- 172.21.15.46:0/3199983378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a404e170 con 0x7f94a40c05d0 2023-10-30T14:21:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.277+0000 7f94a0ff9640 1 -- 172.21.15.46:0/3199983378 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9494004b70 con 0x7f94a40c05d0 2023-10-30T14:21:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.277+0000 7f94a0ff9640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94900664a0 0x7f9490068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.277+0000 7f94a0ff9640 1 -- 172.21.15.46:0/3199983378 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f9494083ff0 con 0x7f94a40c05d0 2023-10-30T14:21:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.277+0000 7f94a37fe640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94900664a0 0x7f9490068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.281+0000 7f94a37fe640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94900664a0 0x7f9490068960 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f94a4151ac0 tx=0x7f9498008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.281+0000 7f94a0ff9640 1 -- 172.21.15.46:0/3199983378 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94940b5090 con 0x7f94a40c05d0 2023-10-30T14:21:05.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 -- 172.21.15.46:0/3199983378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94a413c210 con 0x7f94a40c05d0 2023-10-30T14:21:05.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94a0ff9640 1 -- 172.21.15.46:0/3199983378 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f9494018030 con 0x7f94a40c05d0 2023-10-30T14:21:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:21:05.477 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:21:00.940934+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:05.478 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 242 2023-10-30T14:21:05.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 -- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94900664a0 msgr2=0x7f9490068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:05.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94900664a0 0x7f9490068960 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f94a4151ac0 tx=0x7f9498008040 comp rx=0 tx=0).stop 2023-10-30T14:21:05.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 -- 172.21.15.46:0/3199983378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94a40c05d0 msgr2=0x7f94a41593c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:05.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94a40c05d0 0x7f94a41593c0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f94940011f0 tx=0x7f949400c9b0 comp rx=0 tx=0).stop 2023-10-30T14:21:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 -- 172.21.15.46:0/3199983378 shutdown_connections 2023-10-30T14:21:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f94900664a0 0x7f9490068960 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f94a40c6f60 0x7f94a4139e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a40c6b80 0x7f94a41fa020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 --2- 172.21.15.46:0/3199983378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94a40c05d0 0x7f94a41593c0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:05.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 -- 172.21.15.46:0/3199983378 >> 172.21.15.46:0/3199983378 conn(0x7f94a40c7910 msgr2=0x7f94a40bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:05.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 -- 172.21.15.46:0/3199983378 shutdown_connections 2023-10-30T14:21:05.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:05.473+0000 7f94aa9c5640 1 -- 172.21.15.46:0/3199983378 wait complete. 2023-10-30T14:21:05.499 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 242, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 242, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9426": { "addr": "172.21.15.46:6833/3764280378", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 3764280378, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 3764280378, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 9426, "incarnation": 232, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 42 }, "gid_9594": { "addr": "172.21.15.46:6837/470164358", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 470164358, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 470164358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9594, "incarnation": 239, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:rejoin", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 87, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:21:00.940934+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9426, "mds_1": 9594 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:21:07.500 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:07.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc745cc9640 1 Processor -- start 2023-10-30T14:21:07.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc745cc9640 1 -- start start 2023-10-30T14:21:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc745cc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7400c1ee0 0x7fc74006e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc745cc9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc7400c8260 0x7fc7400cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc745cc9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc74006e090 con 0x7fc740151800 2023-10-30T14:21:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc745cc9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc74006dc70 con 0x7fc7400c8260 2023-10-30T14:21:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc745cc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7400c8a50 con 0x7fc7400c1ee0 2023-10-30T14:21:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc73effd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7400c1ee0 0x7fc74006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc73ffff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc7400c8260 0x7fc7400cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc73effd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7400c1ee0 0x7fc74006e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:33844/0 (socket says 172.21.15.46:33844) 2023-10-30T14:21:07.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc73effd640 1 -- 172.21.15.46:0/1156749419 learned_addr learned my addr 172.21.15.46:0/1156749419 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:21:07.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc73effd640 1 -- 172.21.15.46:0/1156749419 >> v2:172.21.15.154:3300/0 conn(0x7fc7400c8260 msgr2=0x7fc7400cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:07.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc73effd640 1 --2- 172.21.15.46:0/1156749419 >> v2:172.21.15.154:3300/0 conn(0x7fc7400c8260 0x7fc7400cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:07.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc73effd640 1 -- 172.21.15.46:0/1156749419 >> v1:172.21.15.154:6789/0 conn(0x7fc740151800 legacy=0x7fc7400c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:07.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.649+0000 7fc73effd640 1 -- 172.21.15.46:0/1156749419 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7400c3b10 con 0x7fc7400c1ee0 2023-10-30T14:21:07.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.653+0000 7fc73effd640 1 --2- 172.21.15.46:0/1156749419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7400c1ee0 0x7fc74006e470 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc734009870 tx=0x7fc73401c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da48b9d9502375c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.653+0000 7fc73e7fc640 1 -- 172.21.15.46:0/1156749419 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7340ab620 con 0x7fc7400c1ee0 2023-10-30T14:21:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.653+0000 7fc73e7fc640 1 -- 172.21.15.46:0/1156749419 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7340ab780 con 0x7fc7400c1ee0 2023-10-30T14:21:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.653+0000 7fc73e7fc640 1 -- 172.21.15.46:0/1156749419 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7340abb10 con 0x7fc7400c1ee0 2023-10-30T14:21:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.653+0000 7fc745cc9640 1 -- 172.21.15.46:0/1156749419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7400c1ee0 msgr2=0x7fc74006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.653+0000 7fc745cc9640 1 --2- 172.21.15.46:0/1156749419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7400c1ee0 0x7fc74006e470 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc734009870 tx=0x7fc73401c510 comp rx=0 tx=0).stop 2023-10-30T14:21:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.653+0000 7fc745cc9640 1 -- 172.21.15.46:0/1156749419 shutdown_connections 2023-10-30T14:21:07.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.653+0000 7fc745cc9640 1 --2- 172.21.15.46:0/1156749419 >> v2:172.21.15.154:3300/0 conn(0x7fc7400c8260 0x7fc7400cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:07.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.653+0000 7fc745cc9640 1 --2- 172.21.15.46:0/1156749419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7400c1ee0 0x7fc74006e470 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:07.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.653+0000 7fc745cc9640 1 -- 172.21.15.46:0/1156749419 >> 172.21.15.46:0/1156749419 conn(0x7fc740071ea0 msgr2=0x7fc740065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:07.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.653+0000 7fc745cc9640 1 -- 172.21.15.46:0/1156749419 shutdown_connections 2023-10-30T14:21:07.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.653+0000 7fc745cc9640 1 -- 172.21.15.46:0/1156749419 wait complete. 2023-10-30T14:21:07.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc745cc9640 1 Processor -- start 2023-10-30T14:21:07.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc745cc9640 1 -- start start 2023-10-30T14:21:07.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc745cc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7400c1ee0 0x7fc7400b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:07.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc745cc9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7400c8260 0x7fc7401436e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:07.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc745cc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc740151800 0x7fc740145ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:07.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc745cc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc740143fe0 con 0x7fc740151800 2023-10-30T14:21:07.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc745cc9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7401f2e30 con 0x7fc7400c8260 2023-10-30T14:21:07.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc745cc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7401f2220 con 0x7fc7400c1ee0 2023-10-30T14:21:07.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc73ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc740151800 0x7fc740145ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc73effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7400c8260 0x7fc7401436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc73ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc740151800 0x7fc740145ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35262/0 (socket says 172.21.15.46:35262) 2023-10-30T14:21:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc73ffff640 1 -- 172.21.15.46:0/2476424966 learned_addr learned my addr 172.21.15.46:0/2476424966 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:21:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc73f7fe640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7400c1ee0 0x7fc7400b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc73ffff640 1 -- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7400c1ee0 msgr2=0x7fc7400b1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc73ffff640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7400c1ee0 0x7fc7400b1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc73ffff640 1 -- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7400c8260 msgr2=0x7fc7401436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc73ffff640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7400c8260 0x7fc7401436e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc73ffff640 1 -- 172.21.15.46:0/2476424966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7340094d0 con 0x7fc740151800 2023-10-30T14:21:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc73ffff640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc740151800 0x7fc740145ac0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fc73800efc0 tx=0x7fc73800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc727fff640 1 -- 172.21.15.46:0/2476424966 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738009ba0 con 0x7fc740151800 2023-10-30T14:21:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc727fff640 1 -- 172.21.15.46:0/2476424966 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73800f040 con 0x7fc740151800 2023-10-30T14:21:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc745cc9640 1 -- 172.21.15.46:0/2476424966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7401f2500 con 0x7fc740151800 2023-10-30T14:21:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc727fff640 1 -- 172.21.15.46:0/2476424966 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7380146c0 con 0x7fc740151800 2023-10-30T14:21:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.657+0000 7fc745cc9640 1 -- 172.21.15.46:0/2476424966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7401f2ac0 con 0x7fc740151800 2023-10-30T14:21:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.661+0000 7fc745cc9640 1 -- 172.21.15.46:0/2476424966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7401f2fa0 con 0x7fc740151800 2023-10-30T14:21:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.661+0000 7fc727fff640 1 -- 172.21.15.46:0/2476424966 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc738014820 con 0x7fc740151800 2023-10-30T14:21:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.661+0000 7fc727fff640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc72c0664a0 0x7fc72c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.661+0000 7fc727fff640 1 -- 172.21.15.46:0/2476424966 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7fc738085290 con 0x7fc740151800 2023-10-30T14:21:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.661+0000 7fc73f7fe640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc72c0664a0 0x7fc72c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.661+0000 7fc73f7fe640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc72c0664a0 0x7fc72c068960 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc730007fd0 tx=0x7fc730009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:07.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.665+0000 7fc727fff640 1 -- 172.21.15.46:0/2476424966 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc738051cf0 con 0x7fc740151800 2023-10-30T14:21:07.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.861+0000 7fc745cc9640 1 -- 172.21.15.46:0/2476424966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc74004e170 con 0x7fc740151800 2023-10-30T14:21:07.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.861+0000 7fc727fff640 1 -- 172.21.15.46:0/2476424966 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fc738018020 con 0x7fc740151800 2023-10-30T14:21:07.867 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:21:07.867 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:21:06.915846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:07.868 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 243 2023-10-30T14:21:07.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.865+0000 7fc745cc9640 1 -- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc72c0664a0 msgr2=0x7fc72c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:07.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.865+0000 7fc745cc9640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc72c0664a0 0x7fc72c068960 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc730007fd0 tx=0x7fc730009040 comp rx=0 tx=0).stop 2023-10-30T14:21:07.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.865+0000 7fc745cc9640 1 -- 172.21.15.46:0/2476424966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc740151800 msgr2=0x7fc740145ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:07.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.865+0000 7fc745cc9640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc740151800 0x7fc740145ac0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fc73800efc0 tx=0x7fc73800c490 comp rx=0 tx=0).stop 2023-10-30T14:21:07.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.865+0000 7fc745cc9640 1 -- 172.21.15.46:0/2476424966 shutdown_connections 2023-10-30T14:21:07.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.865+0000 7fc745cc9640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc72c0664a0 0x7fc72c068960 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:07.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.865+0000 7fc745cc9640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc740151800 0x7fc740145ac0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:07.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.865+0000 7fc745cc9640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7400c8260 0x7fc7401436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:07.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.865+0000 7fc745cc9640 1 --2- 172.21.15.46:0/2476424966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7400c1ee0 0x7fc7400b1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:07.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.865+0000 7fc745cc9640 1 -- 172.21.15.46:0/2476424966 >> 172.21.15.46:0/2476424966 conn(0x7fc740071ea0 msgr2=0x7fc74014f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:07.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.865+0000 7fc745cc9640 1 -- 172.21.15.46:0/2476424966 shutdown_connections 2023-10-30T14:21:07.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:07.865+0000 7fc745cc9640 1 -- 172.21.15.46:0/2476424966 wait complete. 2023-10-30T14:21:07.888 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=1, replacing gid=9276 2023-10-30T14:21:07.889 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.c 2023-10-30T14:21:07.889 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T14:21:07.889 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T14:21:07.889 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T14:21:07.893 INFO:tasks.ceph.mds.c:Started 2023-10-30T14:21:07.893 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:07.981 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-10-30T14:21:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5b2b72640 1 Processor -- start 2023-10-30T14:21:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5b2b72640 1 -- start start 2023-10-30T14:21:08.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5b2b72640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff5ac1538a0 0x7ff5ac15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5b2b72640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5ac0ca0e0 0x7ff5ac15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5b2b72640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5ac0cb800 con 0x7ff5ac0cf160 2023-10-30T14:21:08.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5b2b72640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5ac15dc30 con 0x7ff5ac0ca0e0 2023-10-30T14:21:08.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5b2b72640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5ac0cb330 con 0x7ff5ac1538a0 2023-10-30T14:21:08.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5abfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5ac0ca0e0 0x7ff5ac15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5abfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff5ac0ca0e0 0x7ff5ac15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35278/0 (socket says 172.21.15.46:35278) 2023-10-30T14:21:08.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5abfff640 1 -- 172.21.15.46:0/2314569641 learned_addr learned my addr 172.21.15.46:0/2314569641 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:21:08.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5b090e640 1 --2- 172.21.15.46:0/2314569641 >> v2:172.21.15.154:3300/0 conn(0x7ff5ac1538a0 0x7ff5ac15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5abfff640 1 -- 172.21.15.46:0/2314569641 >> v2:172.21.15.154:3300/0 conn(0x7ff5ac1538a0 msgr2=0x7ff5ac15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5abfff640 1 --2- 172.21.15.46:0/2314569641 >> v2:172.21.15.154:3300/0 conn(0x7ff5ac1538a0 0x7ff5ac15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5abfff640 1 -- 172.21.15.46:0/2314569641 >> v1:172.21.15.154:6789/0 conn(0x7ff5ac0cf160 legacy=0x7ff5ac0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.033+0000 7ff5abfff640 1 -- 172.21.15.46:0/2314569641 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac0cad70 con 0x7ff5ac0ca0e0 2023-10-30T14:21:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.037+0000 7ff5abfff640 1 --2- 172.21.15.46:0/2314569641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ac0ca0e0 0x7ff5ac15a090 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0cfd10 tx=0x7ff59801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02d3431ceca127b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.037+0000 7ff5ab7fe640 1 -- 172.21.15.46:0/2314569641 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5980a89f0 con 0x7ff5ac0ca0e0 2023-10-30T14:21:08.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.037+0000 7ff5ab7fe640 1 -- 172.21.15.46:0/2314569641 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5980a8b50 con 0x7ff5ac0ca0e0 2023-10-30T14:21:08.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.037+0000 7ff5ab7fe640 1 -- 172.21.15.46:0/2314569641 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5980b0440 con 0x7ff5ac0ca0e0 2023-10-30T14:21:08.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.037+0000 7ff5b2b72640 1 -- 172.21.15.46:0/2314569641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ac0ca0e0 msgr2=0x7ff5ac15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.037+0000 7ff5b2b72640 1 --2- 172.21.15.46:0/2314569641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ac0ca0e0 0x7ff5ac15a090 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0cfd10 tx=0x7ff59801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.037+0000 7ff5b2b72640 1 -- 172.21.15.46:0/2314569641 shutdown_connections 2023-10-30T14:21:08.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.037+0000 7ff5b2b72640 1 --2- 172.21.15.46:0/2314569641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ac0ca0e0 0x7ff5ac15a090 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.037+0000 7ff5b2b72640 1 --2- 172.21.15.46:0/2314569641 >> v2:172.21.15.154:3300/0 conn(0x7ff5ac1538a0 0x7ff5ac15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.037+0000 7ff5b2b72640 1 -- 172.21.15.46:0/2314569641 >> 172.21.15.46:0/2314569641 conn(0x7ff5ac06bfe0 msgr2=0x7ff5ac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:08.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.037+0000 7ff5b2b72640 1 -- 172.21.15.46:0/2314569641 shutdown_connections 2023-10-30T14:21:08.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.037+0000 7ff5b2b72640 1 -- 172.21.15.46:0/2314569641 wait complete. 2023-10-30T14:21:08.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b2b72640 1 Processor -- start 2023-10-30T14:21:08.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b2b72640 1 -- start start 2023-10-30T14:21:08.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b2b72640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5ac0ca0e0 0x7ff5ac140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b2b72640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5ac0cf160 0x7ff5ac0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b2b72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ac1538a0 0x7ff5ac0cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b2b72640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac1e4380 con 0x7ff5ac1538a0 2023-10-30T14:21:08.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b2b72640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac158620 con 0x7ff5ac0cf160 2023-10-30T14:21:08.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b2b72640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac159690 con 0x7ff5ac0ca0e0 2023-10-30T14:21:08.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b110f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ac1538a0 0x7ff5ac0cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b110f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ac1538a0 0x7ff5ac0cd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35294/0 (socket says 172.21.15.46:35294) 2023-10-30T14:21:08.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b110f640 1 -- 172.21.15.46:0/3907210670 learned_addr learned my addr 172.21.15.46:0/3907210670 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:21:08.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5abfff640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5ac0cf160 0x7ff5ac0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b090e640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5ac0ca0e0 0x7ff5ac140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b110f640 1 -- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5ac0ca0e0 msgr2=0x7ff5ac140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b110f640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5ac0ca0e0 0x7ff5ac140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b110f640 1 -- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5ac0cf160 msgr2=0x7ff5ac0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b110f640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5ac0cf160 0x7ff5ac0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b110f640 1 -- 172.21.15.46:0/3907210670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5980094d0 con 0x7ff5ac1538a0 2023-10-30T14:21:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b090e640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5ac0ca0e0 0x7ff5ac140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:21:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5abfff640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5ac0cf160 0x7ff5ac0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:21:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b110f640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ac1538a0 0x7ff5ac0cd4f0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400c6f0 tx=0x7ff5a400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:08.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5a97fa640 1 -- 172.21.15.46:0/3907210670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4003980 con 0x7ff5ac1538a0 2023-10-30T14:21:08.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5a97fa640 1 -- 172.21.15.46:0/3907210670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a4003ae0 con 0x7ff5ac1538a0 2023-10-30T14:21:08.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5a97fa640 1 -- 172.21.15.46:0/3907210670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4003e90 con 0x7ff5ac1538a0 2023-10-30T14:21:08.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b2b72640 1 -- 172.21.15.46:0/3907210670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ac1e51e0 con 0x7ff5ac1538a0 2023-10-30T14:21:08.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.041+0000 7ff5b2b72640 1 -- 172.21.15.46:0/3907210670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ac1e5610 con 0x7ff5ac1538a0 2023-10-30T14:21:08.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.045+0000 7ff5b2b72640 1 -- 172.21.15.46:0/3907210670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ac0b2a50 con 0x7ff5ac1538a0 2023-10-30T14:21:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.045+0000 7ff5a97fa640 1 -- 172.21.15.46:0/3907210670 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5a400a850 con 0x7ff5ac1538a0 2023-10-30T14:21:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.045+0000 7ff5a97fa640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff590066520 0x7ff5900689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.045+0000 7ff5a97fa640 1 -- 172.21.15.46:0/3907210670 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7ff5a4084660 con 0x7ff5ac1538a0 2023-10-30T14:21:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.045+0000 7ff5b090e640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff590066520 0x7ff5900689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.045+0000 7ff5b090e640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff590066520 0x7ff5900689e0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0cf070 tx=0x7ff5a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.049+0000 7ff5a97fa640 1 -- 172.21.15.46:0/3907210670 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5a4051090 con 0x7ff5ac1538a0 2023-10-30T14:21:08.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.237+0000 7ff5b2b72640 1 -- 172.21.15.46:0/3907210670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5ac158790 con 0x7ff5ac1538a0 2023-10-30T14:21:08.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5a97fa640 1 -- 172.21.15.46:0/3907210670 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+5730 (secure 0 0 0) 0x7ff5a40546b0 con 0x7ff5ac1538a0 2023-10-30T14:21:08.244 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:21:08.244 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:21:06.915846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:08.244 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 243 2023-10-30T14:21:08.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5b2b72640 1 -- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff590066520 msgr2=0x7ff5900689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5b2b72640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff590066520 0x7ff5900689e0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0cf070 tx=0x7ff5a0008040 comp rx=0 tx=0).stop 2023-10-30T14:21:08.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5b2b72640 1 -- 172.21.15.46:0/3907210670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ac1538a0 msgr2=0x7ff5ac0cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:08.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5b2b72640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ac1538a0 0x7ff5ac0cd4f0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400c6f0 tx=0x7ff5a400cbb0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5b2b72640 1 -- 172.21.15.46:0/3907210670 shutdown_connections 2023-10-30T14:21:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5b2b72640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ff590066520 0x7ff5900689e0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5b2b72640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ac1538a0 0x7ff5ac0cd4f0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5b2b72640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5ac0cf160 0x7ff5ac0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5b2b72640 1 --2- 172.21.15.46:0/3907210670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5ac0ca0e0 0x7ff5ac140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5b2b72640 1 -- 172.21.15.46:0/3907210670 >> 172.21.15.46:0/3907210670 conn(0x7ff5ac06bfe0 msgr2=0x7ff5ac0c5210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5b2b72640 1 -- 172.21.15.46:0/3907210670 shutdown_connections 2023-10-30T14:21:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:08.241+0000 7ff5b2b72640 1 -- 172.21.15.46:0/3907210670 wait complete. 2023-10-30T14:21:08.265 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T14:21:10.267 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:21:10.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.417+0000 7f303d846640 1 Processor -- start 2023-10-30T14:21:10.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.417+0000 7f303d846640 1 -- start start 2023-10-30T14:21:10.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f303d846640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30380c6b80 0x7f30380bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:10.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f303d846640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30380c6f60 0x7f30380c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:10.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f303d846640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f30380c8430 0x7f303813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:10.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f303d846640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30380c0a50 con 0x7f30380c6b80 2023-10-30T14:21:10.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f303d846640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f303806f6e0 con 0x7f30380c8430 2023-10-30T14:21:10.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f303d846640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303806f9c0 con 0x7f30380c6f60 2023-10-30T14:21:10.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f3036ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30380c6b80 0x7f30380bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:10.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f3036ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30380c6b80 0x7f30380bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35296/0 (socket says 172.21.15.46:35296) 2023-10-30T14:21:10.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f3036ffd640 1 -- 172.21.15.46:0/3994124444 learned_addr learned my addr 172.21.15.46:0/3994124444 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:21:10.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f30377fe640 1 --2- 172.21.15.46:0/3994124444 >> v2:172.21.15.154:3300/0 conn(0x7f30380c8430 0x7f303813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:10.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f302ffff640 1 --2- 172.21.15.46:0/3994124444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30380c6f60 0x7f30380c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:10.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f3036ffd640 1 -- 172.21.15.46:0/3994124444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30380c6f60 msgr2=0x7f30380c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:10.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f3036ffd640 1 --2- 172.21.15.46:0/3994124444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30380c6f60 0x7f30380c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f3036ffd640 1 -- 172.21.15.46:0/3994124444 >> v2:172.21.15.154:3300/0 conn(0x7f30380c8430 msgr2=0x7f303813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:21:10.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f3036ffd640 1 --2- 172.21.15.46:0/3994124444 >> v2:172.21.15.154:3300/0 conn(0x7f30380c8430 0x7f303813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.421+0000 7f3036ffd640 1 -- 172.21.15.46:0/3994124444 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303806fca0 con 0x7f30380c6b80 2023-10-30T14:21:10.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f3036ffd640 1 --2- 172.21.15.46:0/3994124444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380c6b80 0x7f30380bfec0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f30380b0860 tx=0x7f302801e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50ae5744893661 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:10.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f30367fc640 1 -- 172.21.15.46:0/3994124444 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30280a7840 con 0x7f30380c6b80 2023-10-30T14:21:10.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f30367fc640 1 -- 172.21.15.46:0/3994124444 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30280a79a0 con 0x7f30380c6b80 2023-10-30T14:21:10.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f30367fc640 1 -- 172.21.15.46:0/3994124444 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30280a7d30 con 0x7f30380c6b80 2023-10-30T14:21:10.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f303d846640 1 -- 172.21.15.46:0/3994124444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380c6b80 msgr2=0x7f30380bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:10.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f303d846640 1 --2- 172.21.15.46:0/3994124444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380c6b80 0x7f30380bfec0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f30380b0860 tx=0x7f302801e8a0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f303d846640 1 -- 172.21.15.46:0/3994124444 shutdown_connections 2023-10-30T14:21:10.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f303d846640 1 --2- 172.21.15.46:0/3994124444 >> v2:172.21.15.154:3300/0 conn(0x7f30380c8430 0x7f303813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f303d846640 1 --2- 172.21.15.46:0/3994124444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30380c6f60 0x7f30380c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f303d846640 1 --2- 172.21.15.46:0/3994124444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380c6b80 0x7f30380bfec0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f303d846640 1 -- 172.21.15.46:0/3994124444 >> 172.21.15.46:0/3994124444 conn(0x7f30380c7910 msgr2=0x7f30380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f303d846640 1 -- 172.21.15.46:0/3994124444 shutdown_connections 2023-10-30T14:21:10.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f303d846640 1 -- 172.21.15.46:0/3994124444 wait complete. 2023-10-30T14:21:10.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f303d846640 1 Processor -- start 2023-10-30T14:21:10.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.425+0000 7f303d846640 1 -- start start 2023-10-30T14:21:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f303d846640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30380c6b80 0x7f303813d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f303d846640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30380c6f60 0x7f3038158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f303d846640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380c8430 0x7f3038154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f303d846640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3038157ef0 con 0x7f30380c8430 2023-10-30T14:21:10.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f303d846640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3038138520 con 0x7f30380c6f60 2023-10-30T14:21:10.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f303d846640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30380c8d20 con 0x7f30380c6b80 2023-10-30T14:21:10.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f30377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380c8430 0x7f3038154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:10.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f30377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380c8430 0x7f3038154d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35310/0 (socket says 172.21.15.46:35310) 2023-10-30T14:21:10.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f30377fe640 1 -- 172.21.15.46:0/1879496040 learned_addr learned my addr 172.21.15.46:0/1879496040 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:21:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f302ffff640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30380c6f60 0x7f3038158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f3036ffd640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30380c6b80 0x7f303813d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f30377fe640 1 -- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30380c6b80 msgr2=0x7f303813d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f30377fe640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30380c6b80 0x7f303813d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f30377fe640 1 -- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30380c6f60 msgr2=0x7f3038158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f30377fe640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30380c6f60 0x7f3038158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f30377fe640 1 -- 172.21.15.46:0/1879496040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30280094d0 con 0x7f30380c8430 2023-10-30T14:21:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f30377fe640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380c8430 0x7f3038154d60 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f303000efc0 tx=0x7f303000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f3034ff9640 1 -- 172.21.15.46:0/1879496040 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030009ba0 con 0x7f30380c8430 2023-10-30T14:21:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f3034ff9640 1 -- 172.21.15.46:0/1879496040 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303000f040 con 0x7f30380c8430 2023-10-30T14:21:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f3034ff9640 1 -- 172.21.15.46:0/1879496040 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30300146c0 con 0x7f30380c8430 2023-10-30T14:21:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f303d846640 1 -- 172.21.15.46:0/1879496040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30380c8fe0 con 0x7f30380c8430 2023-10-30T14:21:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f303d846640 1 -- 172.21.15.46:0/1879496040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3038157830 con 0x7f30380c8430 2023-10-30T14:21:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f303d846640 1 -- 172.21.15.46:0/1879496040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303804e170 con 0x7f30380c8430 2023-10-30T14:21:10.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.429+0000 7f3034ff9640 1 -- 172.21.15.46:0/1879496040 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3030014820 con 0x7f30380c8430 2023-10-30T14:21:10.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.433+0000 7f3034ff9640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30180664a0 0x7f3018068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:21:10.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.433+0000 7f3034ff9640 1 -- 172.21.15.46:0/1879496040 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f3030085320 con 0x7f30380c8430 2023-10-30T14:21:10.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.433+0000 7f3036ffd640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30180664a0 0x7f3018068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:21:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.433+0000 7f3036ffd640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30180664a0 0x7f3018068960 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f30280b1000 tx=0x7f30280a9070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:21:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.437+0000 7f3034ff9640 1 -- 172.21.15.46:0/1879496040 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3030051cf0 con 0x7f30380c8430 2023-10-30T14:21:10.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.625+0000 7f303d846640 1 -- 172.21.15.46:0/1879496040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30380c1730 con 0x7f30380c8430 2023-10-30T14:21:10.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f3034ff9640 1 -- 172.21.15.46:0/1879496040 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f3030018020 con 0x7f30380c8430 2023-10-30T14:21:10.632 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:21:10.632 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":244,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":244}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:21:06.915846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:21:10.633 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 244 2023-10-30T14:21:10.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f303d846640 1 -- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30180664a0 msgr2=0x7f3018068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:10.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f303d846640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30180664a0 0x7f3018068960 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f30280b1000 tx=0x7f30280a9070 comp rx=0 tx=0).stop 2023-10-30T14:21:10.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f303d846640 1 -- 172.21.15.46:0/1879496040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380c8430 msgr2=0x7f3038154d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:21:10.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f303d846640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380c8430 0x7f3038154d60 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f303000efc0 tx=0x7f303000c490 comp rx=0 tx=0).stop 2023-10-30T14:21:10.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f303d846640 1 -- 172.21.15.46:0/1879496040 shutdown_connections 2023-10-30T14:21:10.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f303d846640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30180664a0 0x7f3018068960 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f303d846640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380c8430 0x7f3038154d60 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f303d846640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30380c6f60 0x7f3038158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f303d846640 1 --2- 172.21.15.46:0/1879496040 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30380c6b80 0x7f303813d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:21:10.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f303d846640 1 -- 172.21.15.46:0/1879496040 >> 172.21.15.46:0/1879496040 conn(0x7f30380c7910 msgr2=0x7f30380c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:21:10.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f303d846640 1 -- 172.21.15.46:0/1879496040 shutdown_connections 2023-10-30T14:21:10.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:21:10.629+0000 7f303d846640 1 -- 172.21.15.46:0/1879496040 wait complete. 2023-10-30T14:21:10.653 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T14:21:10.654 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 75 secs before thrashing 2023-10-30T14:21:18.479 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:21:18.479 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 78 -eq 48 ] 2023-10-30T14:21:18.480 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.78 2023-10-30T14:21:18.480 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.78 2023-10-30T14:21:18.480 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.78 2023-10-30T14:21:18.481 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.78 2023-10-30T14:21:18.481 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:21:23.991 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:21:23.991 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.79 2023-10-30T14:21:23.992 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 79 -eq 48 ] 2023-10-30T14:21:23.992 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.79 2023-10-30T14:21:23.992 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.79 2023-10-30T14:21:23.992 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.79 2023-10-30T14:21:23.992 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:21:30.119 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:21:30.119 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 80 -eq 48 ] 2023-10-30T14:21:30.120 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.80 2023-10-30T14:21:30.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.80 2023-10-30T14:21:30.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.80 2023-10-30T14:21:30.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.80 2023-10-30T14:21:30.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:21:30.449 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:21:30.453 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:21:36.641 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:21:36.642 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.81 2023-10-30T14:21:36.642 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 81 -eq 48 ] 2023-10-30T14:21:36.643 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.81 2023-10-30T14:21:36.643 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.81 2023-10-30T14:21:36.643 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.81 2023-10-30T14:21:36.643 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:21:42.269 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:21:42.269 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 82 -eq 48 ] 2023-10-30T14:21:42.270 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.82 2023-10-30T14:21:42.270 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.82 2023-10-30T14:21:42.270 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.82 2023-10-30T14:21:42.271 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.82 2023-10-30T14:21:42.271 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:21:49.087 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:21:49.087 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 83 -eq 48 ] 2023-10-30T14:21:49.088 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.83 2023-10-30T14:21:49.088 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.83 2023-10-30T14:21:49.088 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.83 2023-10-30T14:21:49.088 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.83 2023-10-30T14:21:49.089 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:21:55.656 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:21:55.657 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 84 -eq 48 ] 2023-10-30T14:21:55.657 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.84 2023-10-30T14:21:55.657 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.84 2023-10-30T14:21:55.658 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.84 2023-10-30T14:21:55.658 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.84 2023-10-30T14:21:55.658 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:22:00.570 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:22:00.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:22:00.611 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-10-30T14:22:00.605+0000 7f457316a640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 67829) UID: 0 2023-10-30T14:22:00.611 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T14:22:00.605+0000 7ff1a0b7f640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 67829) UID: 0 2023-10-30T14:22:00.612 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T14:22:00.605+0000 7fc73d350640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 67829) UID: 0 2023-10-30T14:22:00.891 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:22:00.892 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 85 -eq 48 ] 2023-10-30T14:22:00.892 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.85 2023-10-30T14:22:00.892 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.85 2023-10-30T14:22:00.893 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.85 2023-10-30T14:22:00.893 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.85 2023-10-30T14:22:00.893 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:22:08.666 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:22:08.666 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 86 -eq 48 ] 2023-10-30T14:22:08.667 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.86 2023-10-30T14:22:08.667 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.86 2023-10-30T14:22:08.667 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.86 2023-10-30T14:22:08.667 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.86 2023-10-30T14:22:08.668 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:22:23.177 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:22:23.178 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.87 2023-10-30T14:22:23.178 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 87 -eq 48 ] 2023-10-30T14:22:23.179 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.87 2023-10-30T14:22:23.179 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.87 2023-10-30T14:22:23.179 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.87 2023-10-30T14:22:23.179 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:22:25.654 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f77654640 1 Processor -- start 2023-10-30T14:22:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f77654640 1 -- start start 2023-10-30T14:22:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f77654640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0f701538a0 0x7f0f7015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f77654640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f700cf160 0x7f0f700cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f77654640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f700cb800 con 0x7f0f700ca0e0 2023-10-30T14:22:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f77654640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7015dc30 con 0x7f0f701538a0 2023-10-30T14:22:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f77654640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f700cb330 con 0x7f0f700cf160 2023-10-30T14:22:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f74bef640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0f700ca0e0 0x7f0f7015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60778/0 (socket says 172.21.15.46:60778) 2023-10-30T14:22:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f74bef640 1 -- 172.21.15.46:0/2144372620 learned_addr learned my addr 172.21.15.46:0/2144372620 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f75bf1640 1 --2- 172.21.15.46:0/2144372620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f700cf160 0x7f0f700cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f753f0640 1 --2- 172.21.15.46:0/2144372620 >> v2:172.21.15.154:3300/0 conn(0x7f0f701538a0 0x7f0f7015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f67fff640 1 -- 172.21.15.46:0/2144372620 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243608331 0 0) 0x7f0f700cb800 con 0x7f0f700ca0e0 2023-10-30T14:22:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f67fff640 1 -- 172.21.15.46:0/2144372620 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f5c003530 con 0x7f0f700ca0e0 2023-10-30T14:22:25.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f75bf1640 1 -- 172.21.15.46:0/2144372620 >> v2:172.21.15.154:3300/0 conn(0x7f0f701538a0 msgr2=0x7f0f7015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:25.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f67fff640 1 -- 172.21.15.46:0/2144372620 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3885675472 0 0) 0x7f0f5c003530 con 0x7f0f700ca0e0 2023-10-30T14:22:25.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f75bf1640 1 --2- 172.21.15.46:0/2144372620 >> v2:172.21.15.154:3300/0 conn(0x7f0f701538a0 0x7f0f7015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f75bf1640 1 -- 172.21.15.46:0/2144372620 >> v1:172.21.15.46:6789/0 conn(0x7f0f700ca0e0 legacy=0x7f0f7015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:25.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.852+0000 7f0f75bf1640 1 -- 172.21.15.46:0/2144372620 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f700cad70 con 0x7f0f700cf160 2023-10-30T14:22:25.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.856+0000 7f0f75bf1640 1 --2- 172.21.15.46:0/2144372620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f700cf160 0x7f0f700cf590 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0f68009d90 tx=0x7f0f6801f060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d441618ccff9b16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:25.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.856+0000 7f0f67fff640 1 -- 172.21.15.46:0/2144372620 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60001680 con 0x7f0f700cf160 2023-10-30T14:22:25.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.856+0000 7f0f67fff640 1 -- 172.21.15.46:0/2144372620 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f680a8bb0 con 0x7f0f700cf160 2023-10-30T14:22:25.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.856+0000 7f0f67fff640 1 -- 172.21.15.46:0/2144372620 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f680b0440 con 0x7f0f700cf160 2023-10-30T14:22:25.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.856+0000 7f0f77654640 1 -- 172.21.15.46:0/2144372620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f700cf160 msgr2=0x7f0f700cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:25.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.856+0000 7f0f77654640 1 --2- 172.21.15.46:0/2144372620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f700cf160 0x7f0f700cf590 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0f68009d90 tx=0x7f0f6801f060 comp rx=0 tx=0).stop 2023-10-30T14:22:25.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.856+0000 7f0f77654640 1 -- 172.21.15.46:0/2144372620 shutdown_connections 2023-10-30T14:22:25.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.856+0000 7f0f77654640 1 --2- 172.21.15.46:0/2144372620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f700cf160 0x7f0f700cf590 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.856+0000 7f0f77654640 1 --2- 172.21.15.46:0/2144372620 >> v2:172.21.15.154:3300/0 conn(0x7f0f701538a0 0x7f0f7015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.856+0000 7f0f77654640 1 -- 172.21.15.46:0/2144372620 >> 172.21.15.46:0/2144372620 conn(0x7f0f7006bfe0 msgr2=0x7f0f7006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:25.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.856+0000 7f0f77654640 1 -- 172.21.15.46:0/2144372620 shutdown_connections 2023-10-30T14:22:25.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.856+0000 7f0f77654640 1 -- 172.21.15.46:0/2144372620 wait complete. 2023-10-30T14:22:25.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f77654640 1 Processor -- start 2023-10-30T14:22:25.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f77654640 1 -- start start 2023-10-30T14:22:25.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f77654640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f700ca0e0 0x7f0f7014b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:25.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f77654640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f700cf160 0x7f0f70147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:25.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f77654640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f701538a0 0x7f0f701f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:25.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f77654640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f700c9490 con 0x7f0f700cf160 2023-10-30T14:22:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f77654640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7014ba50 con 0x7f0f700ca0e0 2023-10-30T14:22:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f77654640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f700c8cf0 con 0x7f0f701538a0 2023-10-30T14:22:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f74bef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f700cf160 0x7f0f70147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f74bef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f700cf160 0x7f0f70147f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34946/0 (socket says 172.21.15.46:34946) 2023-10-30T14:22:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f74bef640 1 -- 172.21.15.46:0/4048552860 learned_addr learned my addr 172.21.15.46:0/4048552860 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:25.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f753f0640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f700ca0e0 0x7f0f7014b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:25.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f75bf1640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f701538a0 0x7f0f701f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:25.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f74bef640 1 -- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f701538a0 msgr2=0x7f0f701f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:25.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f74bef640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f701538a0 0x7f0f701f2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f74bef640 1 -- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f700ca0e0 msgr2=0x7f0f7014b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f74bef640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f700ca0e0 0x7f0f7014b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f74bef640 1 -- 172.21.15.46:0/4048552860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f68009a70 con 0x7f0f700cf160 2023-10-30T14:22:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f74bef640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f700cf160 0x7f0f70147f30 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000c750 tx=0x7f0f6000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f65ffb640 1 -- 172.21.15.46:0/4048552860 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60005ae0 con 0x7f0f700cf160 2023-10-30T14:22:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f65ffb640 1 -- 172.21.15.46:0/4048552860 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f60005c40 con 0x7f0f700cf160 2023-10-30T14:22:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f65ffb640 1 -- 172.21.15.46:0/4048552860 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60007070 con 0x7f0f700cf160 2023-10-30T14:22:25.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.860+0000 7f0f77654640 1 -- 172.21.15.46:0/4048552860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f700c8860 con 0x7f0f700cf160 2023-10-30T14:22:25.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.864+0000 7f0f77654640 1 -- 172.21.15.46:0/4048552860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7014d0b0 con 0x7f0f700cf160 2023-10-30T14:22:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.864+0000 7f0f77654640 1 -- 172.21.15.46:0/4048552860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f700b2a50 con 0x7f0f700cf160 2023-10-30T14:22:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.864+0000 7f0f65ffb640 1 -- 172.21.15.46:0/4048552860 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0f600071d0 con 0x7f0f700cf160 2023-10-30T14:22:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.864+0000 7f0f65ffb640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f540664a0 0x7f0f54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.864+0000 7f0f65ffb640 1 -- 172.21.15.46:0/4048552860 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f0f60084820 con 0x7f0f700cf160 2023-10-30T14:22:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.864+0000 7f0f753f0640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f540664a0 0x7f0f54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.868+0000 7f0f753f0640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f540664a0 0x7f0f54068960 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009a50 tx=0x7f0f6c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:25.868+0000 7f0f65ffb640 1 -- 172.21.15.46:0/4048552860 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0f600511d0 con 0x7f0f700cf160 2023-10-30T14:22:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f77654640 1 -- 172.21.15.46:0/4048552860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f7004e170 con 0x7f0f700cf160 2023-10-30T14:22:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f65ffb640 1 -- 172.21.15.46:0/4048552860 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f0f600547f0 con 0x7f0f700cf160 2023-10-30T14:22:26.066 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:26.066 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":244}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:21:22.847601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 245 2023-10-30T14:22:26.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f77654640 1 -- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f540664a0 msgr2=0x7f0f54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:26.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f77654640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f540664a0 0x7f0f54068960 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009a50 tx=0x7f0f6c008040 comp rx=0 tx=0).stop 2023-10-30T14:22:26.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f77654640 1 -- 172.21.15.46:0/4048552860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f700cf160 msgr2=0x7f0f70147f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:26.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f77654640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f700cf160 0x7f0f70147f30 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000c750 tx=0x7f0f6000cc10 comp rx=0 tx=0).stop 2023-10-30T14:22:26.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f77654640 1 -- 172.21.15.46:0/4048552860 shutdown_connections 2023-10-30T14:22:26.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f77654640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f540664a0 0x7f0f54068960 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:26.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f77654640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f701538a0 0x7f0f701f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:26.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f77654640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f700cf160 0x7f0f70147f30 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:26.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f77654640 1 --2- 172.21.15.46:0/4048552860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f700ca0e0 0x7f0f7014b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:26.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f77654640 1 -- 172.21.15.46:0/4048552860 >> 172.21.15.46:0/4048552860 conn(0x7f0f7006bfe0 msgr2=0x7f0f70066310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:26.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.060+0000 7f0f77654640 1 -- 172.21.15.46:0/4048552860 shutdown_connections 2023-10-30T14:22:26.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.064+0000 7f0f77654640 1 -- 172.21.15.46:0/4048552860 wait complete. 2023-10-30T14:22:26.090 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T14:22:26.090 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T14:22:26.090 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:22:26.115 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T14:22:26.116 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:26.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.300+0000 7f28f8b65640 1 Processor -- start 2023-10-30T14:22:26.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.300+0000 7f28f8b65640 1 -- start start 2023-10-30T14:22:26.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f8b65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28f41538a0 0x7f28f415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:26.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f8b65640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f40cb800 con 0x7f28f40ca0e0 2023-10-30T14:22:26.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f8b65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f415dc30 con 0x7f28f40cf160 2023-10-30T14:22:26.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f8b65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40cb330 con 0x7f28f41538a0 2023-10-30T14:22:26.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f1d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f28f40ca0e0 0x7f28f415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60794/0 (socket says 172.21.15.46:60794) 2023-10-30T14:22:26.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f1d9b640 1 -- 172.21.15.46:0/1863379904 learned_addr learned my addr 172.21.15.46:0/1863379904 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:26.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f259c640 1 --2- 172.21.15.46:0/1863379904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28f41538a0 0x7f28f415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:26.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f159a640 1 -- 172.21.15.46:0/1863379904 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420806219 0 0) 0x7f28f40cb800 con 0x7f28f40ca0e0 2023-10-30T14:22:26.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f159a640 1 -- 172.21.15.46:0/1863379904 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28dc003530 con 0x7f28f40ca0e0 2023-10-30T14:22:26.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f159a640 1 -- 172.21.15.46:0/1863379904 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330613079 0 0) 0x7f28ec0014a0 con 0x7f28f40cf160 2023-10-30T14:22:26.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f159a640 1 -- 172.21.15.46:0/1863379904 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f40cb800 con 0x7f28f40cf160 2023-10-30T14:22:26.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f159a640 1 -- 172.21.15.46:0/1863379904 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3868333674 0 0) 0x7f28dc003530 con 0x7f28f40ca0e0 2023-10-30T14:22:26.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f259c640 1 -- 172.21.15.46:0/1863379904 >> v1:172.21.15.154:6789/0 conn(0x7f28f40cf160 legacy=0x7f28f4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:26.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f259c640 1 -- 172.21.15.46:0/1863379904 >> v1:172.21.15.46:6789/0 conn(0x7f28f40ca0e0 legacy=0x7f28f415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:26.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.304+0000 7f28f259c640 1 -- 172.21.15.46:0/1863379904 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f40cad70 con 0x7f28f41538a0 2023-10-30T14:22:26.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f259c640 1 --2- 172.21.15.46:0/1863379904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28f41538a0 0x7f28f415d6f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f28e40097f0 tx=0x7f28e401ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfa3c64933d1bf83 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f159a640 1 -- 172.21.15.46:0/1863379904 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8001680 con 0x7f28f41538a0 2023-10-30T14:22:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f159a640 1 -- 172.21.15.46:0/1863379904 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e40a8650 con 0x7f28f41538a0 2023-10-30T14:22:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f159a640 1 -- 172.21.15.46:0/1863379904 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e40a89c0 con 0x7f28f41538a0 2023-10-30T14:22:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f8b65640 1 -- 172.21.15.46:0/1863379904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28f41538a0 msgr2=0x7f28f415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f8b65640 1 --2- 172.21.15.46:0/1863379904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28f41538a0 0x7f28f415d6f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f28e40097f0 tx=0x7f28e401ebd0 comp rx=0 tx=0).stop 2023-10-30T14:22:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f8b65640 1 -- 172.21.15.46:0/1863379904 shutdown_connections 2023-10-30T14:22:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f8b65640 1 --2- 172.21.15.46:0/1863379904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28f41538a0 0x7f28f415d6f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f8b65640 1 -- 172.21.15.46:0/1863379904 >> 172.21.15.46:0/1863379904 conn(0x7f28f406bfe0 msgr2=0x7f28f406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f8b65640 1 -- 172.21.15.46:0/1863379904 shutdown_connections 2023-10-30T14:22:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f8b65640 1 -- 172.21.15.46:0/1863379904 wait complete. 2023-10-30T14:22:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f8b65640 1 Processor -- start 2023-10-30T14:22:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.308+0000 7f28f8b65640 1 -- start start 2023-10-30T14:22:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f8b65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28f40ca0e0 0x7f28f40b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f8b65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f40cf160 0x7f28f41e34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f8b65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f41538a0 0x7f28f41409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f8b65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f41596e0 con 0x7f28f41538a0 2023-10-30T14:22:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f8b65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f4157560 con 0x7f28f40cf160 2023-10-30T14:22:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f8b65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f4159250 con 0x7f28f40ca0e0 2023-10-30T14:22:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f41538a0 0x7f28f41409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f41538a0 0x7f28f41409f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34960/0 (socket says 172.21.15.46:34960) 2023-10-30T14:22:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f2d9d640 1 -- 172.21.15.46:0/858113539 learned_addr learned my addr 172.21.15.46:0/858113539 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f1d9b640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f40cf160 0x7f28f41e34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f259c640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28f40ca0e0 0x7f28f40b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:26.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f2d9d640 1 -- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28f40ca0e0 msgr2=0x7f28f40b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f2d9d640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28f40ca0e0 0x7f28f40b4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f2d9d640 1 -- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f40cf160 msgr2=0x7f28f41e34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f2d9d640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f40cf160 0x7f28f41e34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f2d9d640 1 -- 172.21.15.46:0/858113539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e4009420 con 0x7f28f41538a0 2023-10-30T14:22:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f1d9b640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f40cf160 0x7f28f41e34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f2d9d640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f41538a0 0x7f28f41409f0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00c770 tx=0x7f28ec00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28e2ffd640 1 -- 172.21.15.46:0/858113539 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec003b80 con 0x7f28f41538a0 2023-10-30T14:22:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28e2ffd640 1 -- 172.21.15.46:0/858113539 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ec003d00 con 0x7f28f41538a0 2023-10-30T14:22:26.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f8b65640 1 -- 172.21.15.46:0/858113539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f4157790 con 0x7f28f41538a0 2023-10-30T14:22:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28e2ffd640 1 -- 172.21.15.46:0/858113539 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec00aa30 con 0x7f28f41538a0 2023-10-30T14:22:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f8b65640 1 -- 172.21.15.46:0/858113539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f41e57e0 con 0x7f28f41538a0 2023-10-30T14:22:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.312+0000 7f28f8b65640 1 -- 172.21.15.46:0/858113539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f404e170 con 0x7f28f41538a0 2023-10-30T14:22:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.316+0000 7f28e2ffd640 1 -- 172.21.15.46:0/858113539 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f28ec00ab90 con 0x7f28f41538a0 2023-10-30T14:22:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.316+0000 7f28e2ffd640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28d40664a0 0x7f28d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.316+0000 7f28e2ffd640 1 -- 172.21.15.46:0/858113539 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f28ec084550 con 0x7f28f41538a0 2023-10-30T14:22:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.316+0000 7f28f259c640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28d40664a0 0x7f28d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.316+0000 7f28f259c640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28d40664a0 0x7f28d4068960 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f28e40b2000 tx=0x7f28e40aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.320+0000 7f28e2ffd640 1 -- 172.21.15.46:0/858113539 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f28ec050ee0 con 0x7f28f41538a0 2023-10-30T14:22:26.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.512+0000 7f28f8b65640 1 -- 172.21.15.46:0/858113539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28f415dc30 con 0x7f28f41538a0 2023-10-30T14:22:26.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.512+0000 7f28e2ffd640 1 -- 172.21.15.46:0/858113539 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f28f415dc30 con 0x7f28f41538a0 2023-10-30T14:22:26.519 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:26.519 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":244}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:21:22.847601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:26.519 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 245 2023-10-30T14:22:26.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.516+0000 7f28f8b65640 1 -- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28d40664a0 msgr2=0x7f28d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.516+0000 7f28f8b65640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28d40664a0 0x7f28d4068960 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f28e40b2000 tx=0x7f28e40aa070 comp rx=0 tx=0).stop 2023-10-30T14:22:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.516+0000 7f28f8b65640 1 -- 172.21.15.46:0/858113539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f41538a0 msgr2=0x7f28f41409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.516+0000 7f28f8b65640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f41538a0 0x7f28f41409f0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00c770 tx=0x7f28ec00cc30 comp rx=0 tx=0).stop 2023-10-30T14:22:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.516+0000 7f28f8b65640 1 -- 172.21.15.46:0/858113539 shutdown_connections 2023-10-30T14:22:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.516+0000 7f28f8b65640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f28d40664a0 0x7f28d4068960 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.516+0000 7f28f8b65640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f41538a0 0x7f28f41409f0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.516+0000 7f28f8b65640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f40cf160 0x7f28f41e34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.516+0000 7f28f8b65640 1 --2- 172.21.15.46:0/858113539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28f40ca0e0 0x7f28f40b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.516+0000 7f28f8b65640 1 -- 172.21.15.46:0/858113539 >> 172.21.15.46:0/858113539 conn(0x7f28f406bfe0 msgr2=0x7f28f4142dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.516+0000 7f28f8b65640 1 -- 172.21.15.46:0/858113539 shutdown_connections 2023-10-30T14:22:26.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:26.516+0000 7f28f8b65640 1 -- 172.21.15.46:0/858113539 wait complete. 2023-10-30T14:22:26.540 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:22:28.542 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:28.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.680+0000 7f7b2b217640 1 Processor -- start 2023-10-30T14:22:28.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2b217640 1 -- start start 2023-10-30T14:22:28.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2b217640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b240cefa0 0x7f7b240cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:28.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2b217640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2415a6c0 con 0x7f7b2415cd50 2023-10-30T14:22:28.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2b217640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b240ca6a0 con 0x7f7b241538f0 2023-10-30T14:22:28.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2b217640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2415d300 con 0x7f7b240cefa0 2023-10-30T14:22:28.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29a14640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7b2415cd50 0x7f7b240c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59634/0 (socket says 172.21.15.46:59634) 2023-10-30T14:22:28.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29a14640 1 -- 172.21.15.46:0/2463232522 learned_addr learned my addr 172.21.15.46:0/2463232522 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:28.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2aa16640 1 --2- 172.21.15.46:0/2463232522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b240cefa0 0x7f7b240cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:28.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597039776 0 0) 0x7f7b2415a6c0 con 0x7f7b2415cd50 2023-10-30T14:22:28.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b10003530 con 0x7f7b2415cd50 2023-10-30T14:22:28.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728898212 0 0) 0x7f7b240ca6a0 con 0x7f7b241538f0 2023-10-30T14:22:28.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b2415a6c0 con 0x7f7b241538f0 2023-10-30T14:22:28.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3117930924 0 0) 0x7f7b10003530 con 0x7f7b2415cd50 2023-10-30T14:22:28.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b240cefa0 msgr2=0x7f7b240cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:28.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 --2- 172.21.15.46:0/2463232522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b240cefa0 0x7f7b240cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:28.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 >> v1:172.21.15.154:6789/0 conn(0x7f7b241538f0 legacy=0x7f7b24159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:28.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2415d5e0 con 0x7f7b2415cd50 2023-10-30T14:22:28.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f7b14001680 con 0x7f7b2415cd50 2023-10-30T14:22:28.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 >> v1:172.21.15.46:6789/0 conn(0x7f7b2415cd50 legacy=0x7f7b240c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:28.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 --2- 172.21.15.46:0/2463232522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b10003b40 0x7f7b10003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:28.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 --2- 172.21.15.46:0/2463232522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b240cefa0 0x7f7b100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:28.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 --2- 172.21.15.46:0/2463232522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b10004d10 0x7f7b10005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:28.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2415d300 con 0x7f7b10004d10 2023-10-30T14:22:28.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1000ba40 con 0x7f7b10003b40 2023-10-30T14:22:28.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1000bbb0 con 0x7f7b240cefa0 2023-10-30T14:22:28.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b29a14640 1 --2- 172.21.15.46:0/2463232522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b240cefa0 0x7f7b100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:28.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2aa16640 1 --2- 172.21.15.46:0/2463232522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b10004d10 0x7f7b10005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:28.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2a215640 1 --2- 172.21.15.46:0/2463232522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b10003b40 0x7f7b10003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:28.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2aa16640 1 -- 172.21.15.46:0/2463232522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b240cefa0 msgr2=0x7f7b100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2aa16640 1 --2- 172.21.15.46:0/2463232522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b240cefa0 0x7f7b100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2aa16640 1 -- 172.21.15.46:0/2463232522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b10003b40 msgr2=0x7f7b10003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2aa16640 1 --2- 172.21.15.46:0/2463232522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b10003b40 0x7f7b10003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.684+0000 7f7b2aa16640 1 -- 172.21.15.46:0/2463232522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b10003530 con 0x7f7b10004d10 2023-10-30T14:22:28.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.688+0000 7f7b2aa16640 1 --2- 172.21.15.46:0/2463232522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b10004d10 0x7f7b10005190 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c009fc0 tx=0x7f7b1c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d74af12995268e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:28.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.688+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c0a5c70 con 0x7f7b10004d10 2023-10-30T14:22:28.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.688+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c00f040 con 0x7f7b10004d10 2023-10-30T14:22:28.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.688+0000 7f7b29213640 1 -- 172.21.15.46:0/2463232522 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c0ad440 con 0x7f7b10004d10 2023-10-30T14:22:28.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.688+0000 7f7b2b217640 1 -- 172.21.15.46:0/2463232522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b10004d10 msgr2=0x7f7b10005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:28.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.688+0000 7f7b2b217640 1 --2- 172.21.15.46:0/2463232522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b10004d10 0x7f7b10005190 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c009fc0 tx=0x7f7b1c01cc60 comp rx=0 tx=0).stop 2023-10-30T14:22:28.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.688+0000 7f7b2aa16640 1 -- 172.21.15.46:0/2463232522 reap_dead start 2023-10-30T14:22:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.688+0000 7f7b2b217640 1 -- 172.21.15.46:0/2463232522 shutdown_connections 2023-10-30T14:22:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.688+0000 7f7b2b217640 1 -- 172.21.15.46:0/2463232522 >> 172.21.15.46:0/2463232522 conn(0x7f7b2406a640 msgr2=0x7f7b24065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.688+0000 7f7b2b217640 1 -- 172.21.15.46:0/2463232522 shutdown_connections 2023-10-30T14:22:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.688+0000 7f7b2b217640 1 -- 172.21.15.46:0/2463232522 wait complete. 2023-10-30T14:22:28.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2b217640 1 Processor -- start 2023-10-30T14:22:28.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2b217640 1 -- start start 2023-10-30T14:22:28.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2b217640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b10004d10 0x7f7b241573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:28.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2b217640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b240b3a20 0x7f7b240b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:28.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2b217640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b2415bb40 0x7f7b2413e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:28.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2b217640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b24159950 con 0x7f7b2415bb40 2023-10-30T14:22:28.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2b217640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b24158d50 con 0x7f7b240b3a20 2023-10-30T14:22:28.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2b217640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2414d960 con 0x7f7b10004d10 2023-10-30T14:22:28.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2aa16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b2415bb40 0x7f7b2413e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:28.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2aa16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b2415bb40 0x7f7b2413e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47184/0 (socket says 172.21.15.46:47184) 2023-10-30T14:22:28.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2aa16640 1 -- 172.21.15.46:0/1966845116 learned_addr learned my addr 172.21.15.46:0/1966845116 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:28.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b29a14640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b240b3a20 0x7f7b240b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:28.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2a215640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b10004d10 0x7f7b241573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:28.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2aa16640 1 -- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b10004d10 msgr2=0x7f7b241573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:28.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2aa16640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b10004d10 0x7f7b241573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:28.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2aa16640 1 -- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b240b3a20 msgr2=0x7f7b240b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:28.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2aa16640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b240b3a20 0x7f7b240b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:28.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.692+0000 7f7b2aa16640 1 -- 172.21.15.46:0/1966845116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b24159280 con 0x7f7b2415bb40 2023-10-30T14:22:28.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.696+0000 7f7b2aa16640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b2415bb40 0x7f7b2413e920 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0b0040 tx=0x7f7b1c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:28.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.696+0000 7f7b1affd640 1 -- 172.21.15.46:0/1966845116 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c0bc070 con 0x7f7b2415bb40 2023-10-30T14:22:28.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.696+0000 7f7b1affd640 1 -- 172.21.15.46:0/1966845116 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c0039a0 con 0x7f7b2415bb40 2023-10-30T14:22:28.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.696+0000 7f7b1affd640 1 -- 172.21.15.46:0/1966845116 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c003b30 con 0x7f7b2415bb40 2023-10-30T14:22:28.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.696+0000 7f7b2b217640 1 -- 172.21.15.46:0/1966845116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b240c64b0 con 0x7f7b2415bb40 2023-10-30T14:22:28.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.696+0000 7f7b2b217640 1 -- 172.21.15.46:0/1966845116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b240b4900 con 0x7f7b2415bb40 2023-10-30T14:22:28.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.696+0000 7f7b2b217640 1 -- 172.21.15.46:0/1966845116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b240b4af0 con 0x7f7b2415bb40 2023-10-30T14:22:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.696+0000 7f7b1affd640 1 -- 172.21.15.46:0/1966845116 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7b1c0b4040 con 0x7f7b2415bb40 2023-10-30T14:22:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.696+0000 7f7b1affd640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b080664d0 0x7f7b08068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:28.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.700+0000 7f7b1affd640 1 -- 172.21.15.46:0/1966845116 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f7b1c11eb20 con 0x7f7b2415bb40 2023-10-30T14:22:28.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.700+0000 7f7b2a215640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b080664d0 0x7f7b08068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:28.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.700+0000 7f7b2a215640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b080664d0 0x7f7b08068990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f7b20007fd0 tx=0x7f7b20009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:28.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.704+0000 7f7b1affd640 1 -- 172.21.15.46:0/1966845116 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7b1c0eb670 con 0x7f7b2415bb40 2023-10-30T14:22:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.896+0000 7f7b2b217640 1 -- 172.21.15.46:0/1966845116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b2404e170 con 0x7f7b2415bb40 2023-10-30T14:22:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.896+0000 7f7b1affd640 1 -- 172.21.15.46:0/1966845116 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f7b1c0b2070 con 0x7f7b2415bb40 2023-10-30T14:22:28.902 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:28.903 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":244}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:21:22.847601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:28.903 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 245 2023-10-30T14:22:28.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.900+0000 7f7b2b217640 1 -- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b080664d0 msgr2=0x7f7b08068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:28.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.900+0000 7f7b2b217640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b080664d0 0x7f7b08068990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f7b20007fd0 tx=0x7f7b20009040 comp rx=0 tx=0).stop 2023-10-30T14:22:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.900+0000 7f7b2b217640 1 -- 172.21.15.46:0/1966845116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b2415bb40 msgr2=0x7f7b2413e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.900+0000 7f7b2b217640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b2415bb40 0x7f7b2413e920 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0b0040 tx=0x7f7b1c009750 comp rx=0 tx=0).stop 2023-10-30T14:22:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.900+0000 7f7b2b217640 1 -- 172.21.15.46:0/1966845116 shutdown_connections 2023-10-30T14:22:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.900+0000 7f7b2b217640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b080664d0 0x7f7b08068990 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.900+0000 7f7b2b217640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b2415bb40 0x7f7b2413e920 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.900+0000 7f7b2b217640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b240b3a20 0x7f7b240b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.900+0000 7f7b2b217640 1 --2- 172.21.15.46:0/1966845116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b10004d10 0x7f7b241573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.900+0000 7f7b2b217640 1 -- 172.21.15.46:0/1966845116 >> 172.21.15.46:0/1966845116 conn(0x7f7b2406a640 msgr2=0x7f7b240c9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.900+0000 7f7b2b217640 1 -- 172.21.15.46:0/1966845116 shutdown_connections 2023-10-30T14:22:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:28.900+0000 7f7b2b217640 1 -- 172.21.15.46:0/1966845116 wait complete. 2023-10-30T14:22:28.923 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:22:30.924 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:31.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.064+0000 7fb33993e640 1 Processor -- start 2023-10-30T14:22:31.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.064+0000 7fb33993e640 1 -- start start 2023-10-30T14:22:31.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.064+0000 7fb33993e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb3340c05d0 0x7fb33413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:31.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.064+0000 7fb33993e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3340c0a50 con 0x7fb3340c6f60 2023-10-30T14:22:31.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.064+0000 7fb33993e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33406f6e0 con 0x7fb3340c6b80 2023-10-30T14:22:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.064+0000 7fb33993e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33406f9c0 con 0x7fb3340c05d0 2023-10-30T14:22:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.064+0000 7fb3337fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb3340c6f60 0x7fb3341566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59644/0 (socket says 172.21.15.46:59644) 2023-10-30T14:22:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.064+0000 7fb3337fe640 1 -- 172.21.15.46:0/844897593 learned_addr learned my addr 172.21.15.46:0/844897593 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:31.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb32bfff640 1 --2- 172.21.15.46:0/844897593 >> v2:172.21.15.154:3300/0 conn(0x7fb3340c05d0 0x7fb33413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:31.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134621775 0 0) 0x7fb3340c0a50 con 0x7fb3340c6f60 2023-10-30T14:22:31.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb320003530 con 0x7fb3340c6f60 2023-10-30T14:22:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468633714 0 0) 0x7fb33406f6e0 con 0x7fb3340c6b80 2023-10-30T14:22:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3340c0a50 con 0x7fb3340c6b80 2023-10-30T14:22:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1791773446 0 0) 0x7fb320003530 con 0x7fb3340c6f60 2023-10-30T14:22:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 >> v2:172.21.15.154:3300/0 conn(0x7fb3340c05d0 msgr2=0x7fb33413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 --2- 172.21.15.46:0/844897593 >> v2:172.21.15.154:3300/0 conn(0x7fb3340c05d0 0x7fb33413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 >> v1:172.21.15.154:6789/0 conn(0x7fb3340c6b80 legacy=0x7fb3340bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:31.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33406fca0 con 0x7fb3340c6f60 2023-10-30T14:22:31.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fb32c001a00 con 0x7fb3340c6f60 2023-10-30T14:22:31.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 >> v1:172.21.15.46:6789/0 conn(0x7fb3340c6f60 legacy=0x7fb3341566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:31.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 --2- 172.21.15.46:0/844897593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb320003b10 0x7fb320003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:31.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 --2- 172.21.15.46:0/844897593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 0x7fb320004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:31.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 --2- 172.21.15.46:0/844897593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb320005190 0x7fb32000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:31.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33406f9c0 con 0x7fb3200047d0 2023-10-30T14:22:31.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32000c010 con 0x7fb320005190 2023-10-30T14:22:31.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32000c1d0 con 0x7fb320003b10 2023-10-30T14:22:31.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb32bfff640 1 --2- 172.21.15.46:0/844897593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 0x7fb320004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:31.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3337fe640 1 --2- 172.21.15.46:0/844897593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb320005190 0x7fb32000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:31.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb332ffd640 1 --2- 172.21.15.46:0/844897593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb320003b10 0x7fb320003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:31.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb32bfff640 1 -- 172.21.15.46:0/844897593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb320003b10 msgr2=0x7fb320003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:31.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb32bfff640 1 --2- 172.21.15.46:0/844897593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb320003b10 0x7fb320003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb32bfff640 1 -- 172.21.15.46:0/844897593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb320005190 msgr2=0x7fb32000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:31.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb32bfff640 1 --2- 172.21.15.46:0/844897593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb320005190 0x7fb32000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb3337fe640 1 -- 172.21.15.46:0/844897593 reap_dead start 2023-10-30T14:22:31.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.068+0000 7fb32bfff640 1 -- 172.21.15.46:0/844897593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb320003530 con 0x7fb3200047d0 2023-10-30T14:22:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb32bfff640 1 --2- 172.21.15.46:0/844897593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 0x7fb320004c50 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009670 tx=0x7fb31c01ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c587d12c194f55d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c0a7840 con 0x7fb3200047d0 2023-10-30T14:22:31.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c0a79a0 con 0x7fb3200047d0 2023-10-30T14:22:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb3327fc640 1 -- 172.21.15.46:0/844897593 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c0a7d50 con 0x7fb3200047d0 2023-10-30T14:22:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb33993e640 1 -- 172.21.15.46:0/844897593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 msgr2=0x7fb320004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb33993e640 1 --2- 172.21.15.46:0/844897593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 0x7fb320004c50 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009670 tx=0x7fb31c01ec40 comp rx=0 tx=0).stop 2023-10-30T14:22:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb33993e640 1 -- 172.21.15.46:0/844897593 shutdown_connections 2023-10-30T14:22:31.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb33993e640 1 --2- 172.21.15.46:0/844897593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 0x7fb320004c50 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb33993e640 1 -- 172.21.15.46:0/844897593 >> 172.21.15.46:0/844897593 conn(0x7fb3340c7910 msgr2=0x7fb3340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb33993e640 1 -- 172.21.15.46:0/844897593 shutdown_connections 2023-10-30T14:22:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb33993e640 1 -- 172.21.15.46:0/844897593 wait complete. 2023-10-30T14:22:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb33993e640 1 Processor -- start 2023-10-30T14:22:31.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.072+0000 7fb33993e640 1 -- start start 2023-10-30T14:22:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb33993e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 0x7fb334152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb33993e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3341538f0 0x7fb334156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb33993e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb334154eb0 0x7fb334156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb33993e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb334155520 con 0x7fb3200047d0 2023-10-30T14:22:31.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb33993e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb334153190 con 0x7fb334154eb0 2023-10-30T14:22:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb33993e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3341ea500 con 0x7fb3341538f0 2023-10-30T14:22:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb332ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 0x7fb334152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb332ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 0x7fb334152c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47194/0 (socket says 172.21.15.46:47194) 2023-10-30T14:22:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb332ffd640 1 -- 172.21.15.46:0/3010983691 learned_addr learned my addr 172.21.15.46:0/3010983691 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:31.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb3337fe640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb334154eb0 0x7fb334156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb32bfff640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3341538f0 0x7fb334156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb332ffd640 1 -- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3341538f0 msgr2=0x7fb334156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb332ffd640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3341538f0 0x7fb334156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb332ffd640 1 -- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb334154eb0 msgr2=0x7fb334156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:31.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb332ffd640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb334154eb0 0x7fb334156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb332ffd640 1 -- 172.21.15.46:0/3010983691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3341ea7c0 con 0x7fb3200047d0 2023-10-30T14:22:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb32bfff640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3341538f0 0x7fb334156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:22:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb332ffd640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 0x7fb334152c50 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fb3240114c0 tx=0x7fb324011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb330ff9640 1 -- 172.21.15.46:0/3010983691 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324015820 con 0x7fb3200047d0 2023-10-30T14:22:31.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb330ff9640 1 -- 172.21.15.46:0/3010983691 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb324015980 con 0x7fb3200047d0 2023-10-30T14:22:31.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb330ff9640 1 -- 172.21.15.46:0/3010983691 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324015d30 con 0x7fb3200047d0 2023-10-30T14:22:31.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb33993e640 1 -- 172.21.15.46:0/3010983691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3341eaaa0 con 0x7fb3200047d0 2023-10-30T14:22:31.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb33993e640 1 -- 172.21.15.46:0/3010983691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3341eb060 con 0x7fb3200047d0 2023-10-30T14:22:31.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb33993e640 1 -- 172.21.15.46:0/3010983691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33404e170 con 0x7fb3200047d0 2023-10-30T14:22:31.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.076+0000 7fb330ff9640 1 -- 172.21.15.46:0/3010983691 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb32401f460 con 0x7fb3200047d0 2023-10-30T14:22:31.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.080+0000 7fb330ff9640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3140664a0 0x7fb314068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.080+0000 7fb330ff9640 1 -- 172.21.15.46:0/3010983691 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7fb3240874c0 con 0x7fb3200047d0 2023-10-30T14:22:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.080+0000 7fb32bfff640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3140664a0 0x7fb314068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.080+0000 7fb32bfff640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3140664a0 0x7fb314068960 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0b1000 tx=0x7fb31c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:31.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.084+0000 7fb330ff9640 1 -- 172.21.15.46:0/3010983691 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb324053e70 con 0x7fb3200047d0 2023-10-30T14:22:31.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.276+0000 7fb33993e640 1 -- 172.21.15.46:0/3010983691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb334154070 con 0x7fb3200047d0 2023-10-30T14:22:31.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.276+0000 7fb330ff9640 1 -- 172.21.15.46:0/3010983691 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fb324058e60 con 0x7fb3200047d0 2023-10-30T14:22:31.282 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:31.283 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":244}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:21:22.847601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 245 2023-10-30T14:22:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.280+0000 7fb33993e640 1 -- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3140664a0 msgr2=0x7fb314068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.280+0000 7fb33993e640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3140664a0 0x7fb314068960 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0b1000 tx=0x7fb31c008040 comp rx=0 tx=0).stop 2023-10-30T14:22:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.280+0000 7fb33993e640 1 -- 172.21.15.46:0/3010983691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 msgr2=0x7fb334152c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.280+0000 7fb33993e640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 0x7fb334152c50 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fb3240114c0 tx=0x7fb324011980 comp rx=0 tx=0).stop 2023-10-30T14:22:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.280+0000 7fb33993e640 1 -- 172.21.15.46:0/3010983691 shutdown_connections 2023-10-30T14:22:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.280+0000 7fb33993e640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb3140664a0 0x7fb314068960 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.280+0000 7fb33993e640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb334154eb0 0x7fb334156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.280+0000 7fb33993e640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3341538f0 0x7fb334156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.280+0000 7fb33993e640 1 --2- 172.21.15.46:0/3010983691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3200047d0 0x7fb334152c50 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.280+0000 7fb33993e640 1 -- 172.21.15.46:0/3010983691 >> 172.21.15.46:0/3010983691 conn(0x7fb3340c7910 msgr2=0x7fb3340c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.280+0000 7fb33993e640 1 -- 172.21.15.46:0/3010983691 shutdown_connections 2023-10-30T14:22:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:31.280+0000 7fb33993e640 1 -- 172.21.15.46:0/3010983691 wait complete. 2023-10-30T14:22:31.304 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:22:33.305 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:33.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29fcd9d640 1 Processor -- start 2023-10-30T14:22:33.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29fcd9d640 1 -- start start 2023-10-30T14:22:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29fcd9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29f81538a0 0x7f29f815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29fcd9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80ca0e0 0x7f29f815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29fcd9d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f80cb800 con 0x7f29f80cf160 2023-10-30T14:22:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29fcd9d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f815dc30 con 0x7f29f81538a0 2023-10-30T14:22:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29fcd9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f80cb330 con 0x7f29f80ca0e0 2023-10-30T14:22:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29f659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29f81538a0 0x7f29f815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29f659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29f81538a0 0x7f29f815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47206/0 (socket says 172.21.15.46:47206) 2023-10-30T14:22:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29f659c640 1 -- 172.21.15.46:0/896420743 learned_addr learned my addr 172.21.15.46:0/896420743 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29effff640 1 --2- 172.21.15.46:0/896420743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80ca0e0 0x7f29f815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29f659c640 1 -- 172.21.15.46:0/896420743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80ca0e0 msgr2=0x7f29f815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29f659c640 1 --2- 172.21.15.46:0/896420743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80ca0e0 0x7f29f815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29f659c640 1 -- 172.21.15.46:0/896420743 >> v1:172.21.15.46:6789/0 conn(0x7f29f80cf160 legacy=0x7f29f80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.444+0000 7f29f659c640 1 -- 172.21.15.46:0/896420743 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f80cad70 con 0x7f29f81538a0 2023-10-30T14:22:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.448+0000 7f29f659c640 1 --2- 172.21.15.46:0/896420743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 0x7f29f815d6f0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f29e8009870 tx=0x7f29e801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48693e8b496e860d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.448+0000 7f29f5d9b640 1 -- 172.21.15.46:0/896420743 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e80a8a30 con 0x7f29f81538a0 2023-10-30T14:22:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.448+0000 7f29f5d9b640 1 -- 172.21.15.46:0/896420743 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e80a8b90 con 0x7f29f81538a0 2023-10-30T14:22:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.448+0000 7f29f5d9b640 1 -- 172.21.15.46:0/896420743 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e80b0440 con 0x7f29f81538a0 2023-10-30T14:22:33.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.448+0000 7f29fcd9d640 1 -- 172.21.15.46:0/896420743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 msgr2=0x7f29f815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.448+0000 7f29fcd9d640 1 --2- 172.21.15.46:0/896420743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 0x7f29f815d6f0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f29e8009870 tx=0x7f29e801ed30 comp rx=0 tx=0).stop 2023-10-30T14:22:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.448+0000 7f29fcd9d640 1 -- 172.21.15.46:0/896420743 shutdown_connections 2023-10-30T14:22:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.448+0000 7f29fcd9d640 1 --2- 172.21.15.46:0/896420743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80ca0e0 0x7f29f815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.448+0000 7f29fcd9d640 1 --2- 172.21.15.46:0/896420743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 0x7f29f815d6f0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:33.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.448+0000 7f29fcd9d640 1 -- 172.21.15.46:0/896420743 >> 172.21.15.46:0/896420743 conn(0x7f29f806bfe0 msgr2=0x7f29f806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.448+0000 7f29fcd9d640 1 -- 172.21.15.46:0/896420743 shutdown_connections 2023-10-30T14:22:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.448+0000 7f29fcd9d640 1 -- 172.21.15.46:0/896420743 wait complete. 2023-10-30T14:22:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29fcd9d640 1 Processor -- start 2023-10-30T14:22:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29fcd9d640 1 -- start start 2023-10-30T14:22:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29fcd9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f80ca0e0 0x7f29f815b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29fcd9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 0x7f29f81e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29fcd9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 0x7f29f81e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29fcd9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f81e5580 con 0x7f29f81538a0 2023-10-30T14:22:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29fcd9d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f815bc60 con 0x7f29f80ca0e0 2023-10-30T14:22:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29fcd9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f81e4670 con 0x7f29f80cf160 2023-10-30T14:22:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29f6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 0x7f29f81e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29f6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 0x7f29f81e5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47214/0 (socket says 172.21.15.46:47214) 2023-10-30T14:22:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29f6d9d640 1 -- 172.21.15.46:0/3776031243 learned_addr learned my addr 172.21.15.46:0/3776031243 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29effff640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 0x7f29f81e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29f659c640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f80ca0e0 0x7f29f815b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29f6d9d640 1 -- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 msgr2=0x7f29f81e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29f6d9d640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 0x7f29f81e4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29f6d9d640 1 -- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f80ca0e0 msgr2=0x7f29f815b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29f6d9d640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f80ca0e0 0x7f29f815b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29f6d9d640 1 -- 172.21.15.46:0/3776031243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e80094d0 con 0x7f29f81538a0 2023-10-30T14:22:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29f6d9d640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 0x7f29f81e5040 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f29f000c720 tx=0x7f29f000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29eeffd640 1 -- 172.21.15.46:0/3776031243 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0003b80 con 0x7f29f81538a0 2023-10-30T14:22:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29eeffd640 1 -- 172.21.15.46:0/3776031243 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f0003ce0 con 0x7f29f81538a0 2023-10-30T14:22:33.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29eeffd640 1 -- 172.21.15.46:0/3776031243 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f000aa90 con 0x7f29f81538a0 2023-10-30T14:22:33.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29fcd9d640 1 -- 172.21.15.46:0/3776031243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f8158840 con 0x7f29f81538a0 2023-10-30T14:22:33.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.452+0000 7f29fcd9d640 1 -- 172.21.15.46:0/3776031243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f8158ae0 con 0x7f29f81538a0 2023-10-30T14:22:33.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.456+0000 7f29fcd9d640 1 -- 172.21.15.46:0/3776031243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f80b2a50 con 0x7f29f81538a0 2023-10-30T14:22:33.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.456+0000 7f29eeffd640 1 -- 172.21.15.46:0/3776031243 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29f000abf0 con 0x7f29f81538a0 2023-10-30T14:22:33.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.456+0000 7f29eeffd640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29d80664a0 0x7f29d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:33.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.456+0000 7f29eeffd640 1 -- 172.21.15.46:0/3776031243 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7f29f00846a0 con 0x7f29f81538a0 2023-10-30T14:22:33.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.456+0000 7f29f659c640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29d80664a0 0x7f29d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:33.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.456+0000 7f29f659c640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29d80664a0 0x7f29d8068960 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f29e80b2000 tx=0x7f29e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.460+0000 7f29eeffd640 1 -- 172.21.15.46:0/3776031243 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29f0050fe0 con 0x7f29f81538a0 2023-10-30T14:22:33.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 -- 172.21.15.46:0/3776031243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29f81e4820 con 0x7f29f81538a0 2023-10-30T14:22:33.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29eeffd640 1 -- 172.21.15.46:0/3776031243 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f29f81e4820 con 0x7f29f81538a0 2023-10-30T14:22:33.657 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:33.658 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":244}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:21:22.847601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:33.658 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 245 2023-10-30T14:22:33.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 -- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29d80664a0 msgr2=0x7f29d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:33.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29d80664a0 0x7f29d8068960 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f29e80b2000 tx=0x7f29e8009040 comp rx=0 tx=0).stop 2023-10-30T14:22:33.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 -- 172.21.15.46:0/3776031243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 msgr2=0x7f29f81e5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:33.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 0x7f29f81e5040 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f29f000c720 tx=0x7f29f000cbe0 comp rx=0 tx=0).stop 2023-10-30T14:22:33.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 -- 172.21.15.46:0/3776031243 shutdown_connections 2023-10-30T14:22:33.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f29d80664a0 0x7f29d8068960 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:33.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29f81538a0 0x7f29f81e5040 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:33.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29f80cf160 0x7f29f81e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:33.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 --2- 172.21.15.46:0/3776031243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f80ca0e0 0x7f29f815b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:33.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 -- 172.21.15.46:0/3776031243 >> 172.21.15.46:0/3776031243 conn(0x7f29f806bfe0 msgr2=0x7f29f8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:33.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 -- 172.21.15.46:0/3776031243 shutdown_connections 2023-10-30T14:22:33.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:33.652+0000 7f29fcd9d640 1 -- 172.21.15.46:0/3776031243 wait complete. 2023-10-30T14:22:33.679 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:22:35.680 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:35.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21f009640 1 Processor -- start 2023-10-30T14:22:35.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21f009640 1 -- start start 2023-10-30T14:22:35.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21f009640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb2180c6c20 0x7fb21815a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:35.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21f009640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2180c3b30 con 0x7fb2180c6840 2023-10-30T14:22:35.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21f009640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2180c3df0 con 0x7fb2180c3640 2023-10-30T14:22:35.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21f009640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2180c2aa0 con 0x7fb2180c6c20 2023-10-30T14:22:35.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21e007640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb2180c6840 0x7fb218153730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59662/0 (socket says 172.21.15.46:59662) 2023-10-30T14:22:35.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21e007640 1 -- 172.21.15.46:0/3150579740 learned_addr learned my addr 172.21.15.46:0/3150579740 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21e808640 1 --2- 172.21.15.46:0/3150579740 >> v2:172.21.15.154:3300/0 conn(0x7fb2180c6c20 0x7fb21815a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21d005640 1 -- 172.21.15.46:0/3150579740 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749904377 0 0) 0x7fb2180c3b30 con 0x7fb2180c6840 2023-10-30T14:22:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21d005640 1 -- 172.21.15.46:0/3150579740 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb204005740 con 0x7fb2180c6840 2023-10-30T14:22:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21e808640 1 -- 172.21.15.46:0/3150579740 >> v1:172.21.15.154:6789/0 conn(0x7fb2180c3640 legacy=0x7fb21815ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:35.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21e808640 1 -- 172.21.15.46:0/3150579740 >> v1:172.21.15.46:6789/0 conn(0x7fb2180c6840 legacy=0x7fb218153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:35.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.820+0000 7fb21e808640 1 -- 172.21.15.46:0/3150579740 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb218159ba0 con 0x7fb2180c6c20 2023-10-30T14:22:35.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.824+0000 7fb21e808640 1 --2- 172.21.15.46:0/3150579740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2180c6c20 0x7fb21815a680 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb210009d90 tx=0x7fb21001c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87f17be4de361489 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:35.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.824+0000 7fb21d005640 1 -- 172.21.15.46:0/3150579740 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2100a5710 con 0x7fb2180c6c20 2023-10-30T14:22:35.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.824+0000 7fb21d005640 1 -- 172.21.15.46:0/3150579740 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2100a5870 con 0x7fb2180c6c20 2023-10-30T14:22:35.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.824+0000 7fb21d005640 1 -- 172.21.15.46:0/3150579740 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2100a5be0 con 0x7fb2180c6c20 2023-10-30T14:22:35.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.824+0000 7fb21f009640 1 -- 172.21.15.46:0/3150579740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2180c6c20 msgr2=0x7fb21815a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:35.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.824+0000 7fb21f009640 1 --2- 172.21.15.46:0/3150579740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2180c6c20 0x7fb21815a680 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb210009d90 tx=0x7fb21001c950 comp rx=0 tx=0).stop 2023-10-30T14:22:35.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.824+0000 7fb21f009640 1 -- 172.21.15.46:0/3150579740 shutdown_connections 2023-10-30T14:22:35.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.824+0000 7fb21f009640 1 --2- 172.21.15.46:0/3150579740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2180c6c20 0x7fb21815a680 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:35.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.824+0000 7fb21f009640 1 -- 172.21.15.46:0/3150579740 >> 172.21.15.46:0/3150579740 conn(0x7fb2180c75d0 msgr2=0x7fb2180c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:35.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.824+0000 7fb21f009640 1 -- 172.21.15.46:0/3150579740 shutdown_connections 2023-10-30T14:22:35.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.824+0000 7fb21f009640 1 -- 172.21.15.46:0/3150579740 wait complete. 2023-10-30T14:22:35.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21f009640 1 Processor -- start 2023-10-30T14:22:35.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21f009640 1 -- start start 2023-10-30T14:22:35.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21f009640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2180c3640 0x7fb21814bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:35.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21f009640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2180c6840 0x7fb218070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:35.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21f009640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2180c6c20 0x7fb2181f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:35.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21f009640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb218070970 con 0x7fb2180c6c20 2023-10-30T14:22:35.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21f009640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2180702b0 con 0x7fb2180c6840 2023-10-30T14:22:35.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21f009640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2181523a0 con 0x7fb2180c3640 2023-10-30T14:22:35.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21e808640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2180c6c20 0x7fb2181f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:35.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21e808640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2180c6c20 0x7fb2181f3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47218/0 (socket says 172.21.15.46:47218) 2023-10-30T14:22:35.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21e808640 1 -- 172.21.15.46:0/2169413927 learned_addr learned my addr 172.21.15.46:0/2169413927 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:35.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21d806640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2180c6840 0x7fb218070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:35.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21e007640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2180c3640 0x7fb21814bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:35.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21e808640 1 -- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2180c3640 msgr2=0x7fb21814bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:35.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21e808640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2180c3640 0x7fb21814bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:35.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21e808640 1 -- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2180c6840 msgr2=0x7fb218070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:35.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21e808640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2180c6840 0x7fb218070eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:35.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21e808640 1 -- 172.21.15.46:0/2169413927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb210009a70 con 0x7fb2180c6c20 2023-10-30T14:22:35.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21e808640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2180c6c20 0x7fb2181f3990 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fb210019560 tx=0x7fb2100196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:35.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb20effd640 1 -- 172.21.15.46:0/2169413927 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2100ad440 con 0x7fb2180c6c20 2023-10-30T14:22:35.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb20effd640 1 -- 172.21.15.46:0/2169413927 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2100ad5a0 con 0x7fb2180c6c20 2023-10-30T14:22:35.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb20effd640 1 -- 172.21.15.46:0/2169413927 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2100ad950 con 0x7fb2180c6c20 2023-10-30T14:22:35.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21f009640 1 -- 172.21.15.46:0/2169413927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2181514b0 con 0x7fb2180c6c20 2023-10-30T14:22:35.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.828+0000 7fb21f009640 1 -- 172.21.15.46:0/2169413927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb218151a70 con 0x7fb2180c6c20 2023-10-30T14:22:35.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.832+0000 7fb21f009640 1 -- 172.21.15.46:0/2169413927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21804e170 con 0x7fb2180c6c20 2023-10-30T14:22:35.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.832+0000 7fb20effd640 1 -- 172.21.15.46:0/2169413927 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb2100b4070 con 0x7fb2180c6c20 2023-10-30T14:22:35.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.832+0000 7fb20effd640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb1fc0664a0 0x7fb1fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:35.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.832+0000 7fb20effd640 1 -- 172.21.15.46:0/2169413927 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7fb21011f360 con 0x7fb2180c6c20 2023-10-30T14:22:35.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.832+0000 7fb21e007640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb1fc0664a0 0x7fb1fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:35.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.832+0000 7fb21e007640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb1fc0664a0 0x7fb1fc068960 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb214007fd0 tx=0x7fb214009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:35.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:35.836+0000 7fb20effd640 1 -- 172.21.15.46:0/2169413927 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb2100ebd30 con 0x7fb2180c6c20 2023-10-30T14:22:36.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 -- 172.21.15.46:0/2169413927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2181526e0 con 0x7fb2180c6c20 2023-10-30T14:22:36.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb20effd640 1 -- 172.21.15.46:0/2169413927 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fb2100b2090 con 0x7fb2180c6c20 2023-10-30T14:22:36.033 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:36.033 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":244}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:21:22.847601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:36.033 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 245 2023-10-30T14:22:36.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 -- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb1fc0664a0 msgr2=0x7fb1fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:36.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb1fc0664a0 0x7fb1fc068960 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb214007fd0 tx=0x7fb214009040 comp rx=0 tx=0).stop 2023-10-30T14:22:36.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 -- 172.21.15.46:0/2169413927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2180c6c20 msgr2=0x7fb2181f3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:36.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2180c6c20 0x7fb2181f3990 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fb210019560 tx=0x7fb2100196f0 comp rx=0 tx=0).stop 2023-10-30T14:22:36.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 -- 172.21.15.46:0/2169413927 shutdown_connections 2023-10-30T14:22:36.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb1fc0664a0 0x7fb1fc068960 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:36.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2180c6c20 0x7fb2181f3990 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:36.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2180c6840 0x7fb218070eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:36.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 --2- 172.21.15.46:0/2169413927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2180c3640 0x7fb21814bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:36.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 -- 172.21.15.46:0/2169413927 >> 172.21.15.46:0/2169413927 conn(0x7fb2180c75d0 msgr2=0x7fb21814c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:36.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 -- 172.21.15.46:0/2169413927 shutdown_connections 2023-10-30T14:22:36.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:36.028+0000 7fb21f009640 1 -- 172.21.15.46:0/2169413927 wait complete. 2023-10-30T14:22:36.054 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:22:38.055 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:38.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd0446b640 1 Processor -- start 2023-10-30T14:22:38.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd0446b640 1 -- start start 2023-10-30T14:22:38.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd0446b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efcfc159e60 0x7efcfc06f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:38.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd0446b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcfc06fa80 con 0x7efcfc0bfec0 2023-10-30T14:22:38.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd0446b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcfc071290 con 0x7efcfc06a720 2023-10-30T14:22:38.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd0446b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcfc071570 con 0x7efcfc159e60 2023-10-30T14:22:38.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01a06640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efcfc0bfec0 0x7efcfc0c1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43518/0 (socket says 172.21.15.46:43518) 2023-10-30T14:22:38.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01a06640 1 -- 172.21.15.46:0/2731425908 learned_addr learned my addr 172.21.15.46:0/2731425908 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd02207640 1 --2- 172.21.15.46:0/2731425908 >> v2:172.21.15.154:3300/0 conn(0x7efcfc159e60 0x7efcfc06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365144506 0 0) 0x7efcfc06fa80 con 0x7efcfc0bfec0 2023-10-30T14:22:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce8003530 con 0x7efcfc0bfec0 2023-10-30T14:22:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4108569341 0 0) 0x7efce8003530 con 0x7efcfc0bfec0 2023-10-30T14:22:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 >> v2:172.21.15.154:3300/0 conn(0x7efcfc159e60 msgr2=0x7efcfc06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:38.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 --2- 172.21.15.46:0/2731425908 >> v2:172.21.15.154:3300/0 conn(0x7efcfc159e60 0x7efcfc06f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:38.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 >> v1:172.21.15.154:6789/0 conn(0x7efcfc06a720 legacy=0x7efcfc160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:38.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc071850 con 0x7efcfc0bfec0 2023-10-30T14:22:38.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7efcec001680 con 0x7efcfc0bfec0 2023-10-30T14:22:38.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 >> v1:172.21.15.46:6789/0 conn(0x7efcfc0bfec0 legacy=0x7efcfc0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:38.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 --2- 172.21.15.46:0/2731425908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce8003980 0x7efce8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:38.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 --2- 172.21.15.46:0/2731425908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce80047d0 0x7efce8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:38.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 --2- 172.21.15.46:0/2731425908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efce8005190 0x7efce800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:38.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc071570 con 0x7efce80047d0 2023-10-30T14:22:38.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce800c040 con 0x7efce8003980 2023-10-30T14:22:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce800c200 con 0x7efce8005190 2023-10-30T14:22:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd01a06640 1 --2- 172.21.15.46:0/2731425908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce80047d0 0x7efce8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd02207640 1 --2- 172.21.15.46:0/2731425908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce8003980 0x7efce8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.196+0000 7efd02a08640 1 --2- 172.21.15.46:0/2731425908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efce8005190 0x7efce800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.200+0000 7efd01a06640 1 -- 172.21.15.46:0/2731425908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efce8005190 msgr2=0x7efce800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.200+0000 7efd01a06640 1 --2- 172.21.15.46:0/2731425908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efce8005190 0x7efce800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.200+0000 7efd01a06640 1 -- 172.21.15.46:0/2731425908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce8003980 msgr2=0x7efce8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.200+0000 7efd01a06640 1 --2- 172.21.15.46:0/2731425908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce8003980 0x7efce8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.200+0000 7efd01a06640 1 -- 172.21.15.46:0/2731425908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce8003530 con 0x7efce80047d0 2023-10-30T14:22:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.200+0000 7efd02a08640 1 -- 172.21.15.46:0/2731425908 reap_dead start 2023-10-30T14:22:38.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.200+0000 7efd01a06640 1 --2- 172.21.15.46:0/2731425908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce80047d0 0x7efce8004c50 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7efcec00c240 tx=0x7efcec01cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4b0872bf071d1e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:38.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.200+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec0a5b00 con 0x7efce80047d0 2023-10-30T14:22:38.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.200+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcec0a5c60 con 0x7efce80047d0 2023-10-30T14:22:38.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.200+0000 7efd01205640 1 -- 172.21.15.46:0/2731425908 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec005b20 con 0x7efce80047d0 2023-10-30T14:22:38.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.200+0000 7efd0446b640 1 -- 172.21.15.46:0/2731425908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce80047d0 msgr2=0x7efce8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:38.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.200+0000 7efd0446b640 1 --2- 172.21.15.46:0/2731425908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce80047d0 0x7efce8004c50 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7efcec00c240 tx=0x7efcec01cc00 comp rx=0 tx=0).stop 2023-10-30T14:22:38.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 -- 172.21.15.46:0/2731425908 shutdown_connections 2023-10-30T14:22:38.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 --2- 172.21.15.46:0/2731425908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efce80047d0 0x7efce8004c50 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:38.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 -- 172.21.15.46:0/2731425908 >> 172.21.15.46:0/2731425908 conn(0x7efcfc0c7910 msgr2=0x7efcfc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:38.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 -- 172.21.15.46:0/2731425908 shutdown_connections 2023-10-30T14:22:38.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 -- 172.21.15.46:0/2731425908 wait complete. 2023-10-30T14:22:38.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 Processor -- start 2023-10-30T14:22:38.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 -- start start 2023-10-30T14:22:38.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce80047d0 0x7efcfc0b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:38.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efcfc156e50 0x7efcfc0c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:38.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcfc1ef510 0x7efcfc1ef990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:38.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc1545a0 con 0x7efcfc1ef510 2023-10-30T14:22:38.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc1eede0 con 0x7efce80047d0 2023-10-30T14:22:38.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd0446b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc153870 con 0x7efcfc156e50 2023-10-30T14:22:38.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd02a08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcfc1ef510 0x7efcfc1ef990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:38.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd02a08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcfc1ef510 0x7efcfc1ef990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58478/0 (socket says 172.21.15.46:58478) 2023-10-30T14:22:38.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd02a08640 1 -- 172.21.15.46:0/3672514796 learned_addr learned my addr 172.21.15.46:0/3672514796 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:38.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd01a06640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efcfc156e50 0x7efcfc0c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd02207640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce80047d0 0x7efcfc0b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd02a08640 1 -- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efcfc156e50 msgr2=0x7efcfc0c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd02a08640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efcfc156e50 0x7efcfc0c9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd02a08640 1 -- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce80047d0 msgr2=0x7efcfc0b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd02a08640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce80047d0 0x7efcfc0b0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.204+0000 7efd02a08640 1 -- 172.21.15.46:0/3672514796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc153b30 con 0x7efcfc1ef510 2023-10-30T14:22:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.208+0000 7efd02a08640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcfc1ef510 0x7efcfc1ef990 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7efcfc06f3b0 tx=0x7efcf400fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.208+0000 7efcf2ffd640 1 -- 172.21.15.46:0/3672514796 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf400fed0 con 0x7efcfc1ef510 2023-10-30T14:22:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.208+0000 7efcf2ffd640 1 -- 172.21.15.46:0/3672514796 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf4003b90 con 0x7efcfc1ef510 2023-10-30T14:22:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.208+0000 7efcf2ffd640 1 -- 172.21.15.46:0/3672514796 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf40050d0 con 0x7efcfc1ef510 2023-10-30T14:22:38.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.208+0000 7efd0446b640 1 -- 172.21.15.46:0/3672514796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcfc153e10 con 0x7efcfc1ef510 2023-10-30T14:22:38.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.208+0000 7efd0446b640 1 -- 172.21.15.46:0/3672514796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcfc1eeff0 con 0x7efcfc1ef510 2023-10-30T14:22:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.208+0000 7efd0446b640 1 -- 172.21.15.46:0/3672514796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcfc1542c0 con 0x7efcfc1ef510 2023-10-30T14:22:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.208+0000 7efcf2ffd640 1 -- 172.21.15.46:0/3672514796 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efcf4005230 con 0x7efcfc1ef510 2023-10-30T14:22:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.212+0000 7efcf2ffd640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efce00664a0 0x7efce0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.212+0000 7efcf2ffd640 1 -- 172.21.15.46:0/3672514796 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8434+0+0 (secure 0 0 0) 0x7efcf4084890 con 0x7efcfc1ef510 2023-10-30T14:22:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.212+0000 7efd02207640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efce00664a0 0x7efce0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.212+0000 7efd02207640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efce00664a0 0x7efce0068960 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7efcfc069010 tx=0x7efcf8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:38.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.216+0000 7efcf2ffd640 1 -- 172.21.15.46:0/3672514796 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efcf4051130 con 0x7efcfc1ef510 2023-10-30T14:22:38.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.404+0000 7efd0446b640 1 -- 172.21.15.46:0/3672514796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcfc04e170 con 0x7efcfc1ef510 2023-10-30T14:22:38.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.404+0000 7efcf2ffd640 1 -- 172.21.15.46:0/3672514796 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+6523 (secure 0 0 0) 0x7efcf4054750 con 0x7efcfc1ef510 2023-10-30T14:22:38.412 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:38.412 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9735,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":244}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:21:22.847601+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9426,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9426":{"gid":9426,"name":"a","rank":0,"incarnation":232,"state":"up:active","state_seq":42,"addr":"172.21.15.46:6833/3764280378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3764280378},{"type":"v1","addr":"172.21.15.46:6833","nonce":3764280378}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:38.412 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 245 2023-10-30T14:22:38.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.408+0000 7efd0446b640 1 -- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efce00664a0 msgr2=0x7efce0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:38.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.408+0000 7efd0446b640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efce00664a0 0x7efce0068960 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7efcfc069010 tx=0x7efcf8009190 comp rx=0 tx=0).stop 2023-10-30T14:22:38.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.408+0000 7efd0446b640 1 -- 172.21.15.46:0/3672514796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcfc1ef510 msgr2=0x7efcfc1ef990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:38.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.408+0000 7efd0446b640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcfc1ef510 0x7efcfc1ef990 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7efcfc06f3b0 tx=0x7efcf400fc80 comp rx=0 tx=0).stop 2023-10-30T14:22:38.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.408+0000 7efd0446b640 1 -- 172.21.15.46:0/3672514796 shutdown_connections 2023-10-30T14:22:38.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.408+0000 7efd0446b640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efce00664a0 0x7efce0068960 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:38.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.408+0000 7efd0446b640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcfc1ef510 0x7efcfc1ef990 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:38.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.408+0000 7efd0446b640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efcfc156e50 0x7efcfc0c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:38.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.408+0000 7efd0446b640 1 --2- 172.21.15.46:0/3672514796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efce80047d0 0x7efcfc0b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:38.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.408+0000 7efd0446b640 1 -- 172.21.15.46:0/3672514796 >> 172.21.15.46:0/3672514796 conn(0x7efcfc0c7910 msgr2=0x7efcfc0ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:38.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.408+0000 7efd0446b640 1 -- 172.21.15.46:0/3672514796 shutdown_connections 2023-10-30T14:22:38.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:38.408+0000 7efd0446b640 1 -- 172.21.15.46:0/3672514796 wait complete. 2023-10-30T14:22:38.433 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T14:22:40.433 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:40.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.572+0000 7fa9b113e640 1 Processor -- start 2023-10-30T14:22:40.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.572+0000 7fa9b113e640 1 -- start start 2023-10-30T14:22:40.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.572+0000 7fa9b113e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa9ac15d6f0 0x7fa9ac15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:40.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.572+0000 7fa9b113e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac0cb800 con 0x7fa9ac0cf160 2023-10-30T14:22:40.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.572+0000 7fa9b113e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac0cb330 con 0x7fa9ac1538a0 2023-10-30T14:22:40.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.572+0000 7fa9b113e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ac0cad70 con 0x7fa9ac15d6f0 2023-10-30T14:22:40.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.572+0000 7fa9ab59e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa9ac0cf160 0x7fa9ac164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43526/0 (socket says 172.21.15.46:43526) 2023-10-30T14:22:40.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.576+0000 7fa9ab59e640 1 -- 172.21.15.46:0/3912839130 learned_addr learned my addr 172.21.15.46:0/3912839130 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:40.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.576+0000 7fa9aa59c640 1 --2- 172.21.15.46:0/3912839130 >> v2:172.21.15.154:3300/0 conn(0x7fa9ac15d6f0 0x7fa9ac15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:40.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.576+0000 7fa9a9d9b640 1 -- 172.21.15.46:0/3912839130 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260338812 0 0) 0x7fa9ac0cb800 con 0x7fa9ac0cf160 2023-10-30T14:22:40.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.576+0000 7fa9a9d9b640 1 -- 172.21.15.46:0/3912839130 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa994003530 con 0x7fa9ac0cf160 2023-10-30T14:22:40.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.576+0000 7fa9aa59c640 1 -- 172.21.15.46:0/3912839130 >> v1:172.21.15.154:6789/0 conn(0x7fa9ac1538a0 legacy=0x7fa9ac0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:40.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.576+0000 7fa9aa59c640 1 -- 172.21.15.46:0/3912839130 >> v1:172.21.15.46:6789/0 conn(0x7fa9ac0cf160 legacy=0x7fa9ac164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:40.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.576+0000 7fa9a9d9b640 1 -- 172.21.15.46:0/3912839130 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2681608638 0 0) 0x7fa994003530 con 0x7fa9ac0cf160 2023-10-30T14:22:40.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.576+0000 7fa9aa59c640 1 -- 172.21.15.46:0/3912839130 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac0cb050 con 0x7fa9ac15d6f0 2023-10-30T14:22:40.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.576+0000 7fa9aa59c640 1 --2- 172.21.15.46:0/3912839130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ac15d6f0 0x7fa9ac15a090 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fa9a00097c0 tx=0x7fa9a001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e03e74b447c9902f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:40.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9a9d9b640 1 -- 172.21.15.46:0/3912839130 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a00a8970 con 0x7fa9ac15d6f0 2023-10-30T14:22:40.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9a9d9b640 1 -- 172.21.15.46:0/3912839130 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a00a8ad0 con 0x7fa9ac15d6f0 2023-10-30T14:22:40.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9a9d9b640 1 -- 172.21.15.46:0/3912839130 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a00a8e40 con 0x7fa9ac15d6f0 2023-10-30T14:22:40.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 -- 172.21.15.46:0/3912839130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ac15d6f0 msgr2=0x7fa9ac15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:40.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 --2- 172.21.15.46:0/3912839130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ac15d6f0 0x7fa9ac15a090 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fa9a00097c0 tx=0x7fa9a001ec70 comp rx=0 tx=0).stop 2023-10-30T14:22:40.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 -- 172.21.15.46:0/3912839130 shutdown_connections 2023-10-30T14:22:40.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 --2- 172.21.15.46:0/3912839130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ac15d6f0 0x7fa9ac15a090 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:40.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 -- 172.21.15.46:0/3912839130 >> 172.21.15.46:0/3912839130 conn(0x7fa9ac06bfe0 msgr2=0x7fa9ac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:40.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 -- 172.21.15.46:0/3912839130 shutdown_connections 2023-10-30T14:22:40.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 -- 172.21.15.46:0/3912839130 wait complete. 2023-10-30T14:22:40.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 Processor -- start 2023-10-30T14:22:40.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 -- start start 2023-10-30T14:22:40.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9ac0cf160 0x7fa9ac150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:40.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9ac1538a0 0x7fa9ac150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:40.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ac15d6f0 0x7fa9ac0b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:40.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0b4960 con 0x7fa9ac1538a0 2023-10-30T14:22:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac15a3c0 con 0x7fa9ac15d6f0 2023-10-30T14:22:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.580+0000 7fa9b113e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac157520 con 0x7fa9ac0cf160 2023-10-30T14:22:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9aa59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9ac1538a0 0x7fa9ac150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9aa59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9ac1538a0 0x7fa9ac150750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58494/0 (socket says 172.21.15.46:58494) 2023-10-30T14:22:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9aa59c640 1 -- 172.21.15.46:0/2693304285 learned_addr learned my addr 172.21.15.46:0/2693304285 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9aad9d640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9ac0cf160 0x7fa9ac150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9ab59e640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ac15d6f0 0x7fa9ac0b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:40.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9aa59c640 1 -- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9ac0cf160 msgr2=0x7fa9ac150210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:40.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9aa59c640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9ac0cf160 0x7fa9ac150210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:40.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9aa59c640 1 -- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ac15d6f0 msgr2=0x7fa9ac0b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:40.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9aa59c640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ac15d6f0 0x7fa9ac0b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:40.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9aa59c640 1 -- 172.21.15.46:0/2693304285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a0009420 con 0x7fa9ac1538a0 2023-10-30T14:22:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9aa59c640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9ac1538a0 0x7fa9ac150750 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fa9a001c8a0 tx=0x7fa9a001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9937fe640 1 -- 172.21.15.46:0/2693304285 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a00044d0 con 0x7fa9ac1538a0 2023-10-30T14:22:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9937fe640 1 -- 172.21.15.46:0/2693304285 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a0003620 con 0x7fa9ac1538a0 2023-10-30T14:22:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9937fe640 1 -- 172.21.15.46:0/2693304285 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0003990 con 0x7fa9ac1538a0 2023-10-30T14:22:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9b113e640 1 -- 172.21.15.46:0/2693304285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac159470 con 0x7fa9ac1538a0 2023-10-30T14:22:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9b113e640 1 -- 172.21.15.46:0/2693304285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac1ef870 con 0x7fa9ac1538a0 2023-10-30T14:22:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9b113e640 1 -- 172.21.15.46:0/2693304285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ac159600 con 0x7fa9ac1538a0 2023-10-30T14:22:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.584+0000 7fa9937fe640 1 -- 172.21.15.46:0/2693304285 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa9a00b7040 con 0x7fa9ac1538a0 2023-10-30T14:22:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.588+0000 7fa9937fe640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9980664d0 0x7fa998068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.588+0000 7fa9937fe640 1 -- 172.21.15.46:0/2693304285 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa9a01220d0 con 0x7fa9ac1538a0 2023-10-30T14:22:40.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.588+0000 7fa9aad9d640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9980664d0 0x7fa998068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:40.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.588+0000 7fa9aad9d640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9980664d0 0x7fa998068990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa99c007fd0 tx=0x7fa99c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:40.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.592+0000 7fa9937fe640 1 -- 172.21.15.46:0/2693304285 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa9a00ee9c0 con 0x7fa9ac1538a0 2023-10-30T14:22:40.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.784+0000 7fa9b113e640 1 -- 172.21.15.46:0/2693304285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9ac04e170 con 0x7fa9ac1538a0 2023-10-30T14:22:40.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.784+0000 7fa9937fe640 1 -- 172.21.15.46:0/2693304285 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fa9a00f1fe0 con 0x7fa9ac1538a0 2023-10-30T14:22:40.790 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:40.790 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:22:39.814079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:40.791 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 246 2023-10-30T14:22:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.788+0000 7fa9b113e640 1 -- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9980664d0 msgr2=0x7fa998068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.788+0000 7fa9b113e640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9980664d0 0x7fa998068990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa99c007fd0 tx=0x7fa99c009040 comp rx=0 tx=0).stop 2023-10-30T14:22:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.788+0000 7fa9b113e640 1 -- 172.21.15.46:0/2693304285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9ac1538a0 msgr2=0x7fa9ac150750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:40.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.788+0000 7fa9b113e640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9ac1538a0 0x7fa9ac150750 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fa9a001c8a0 tx=0x7fa9a001ba10 comp rx=0 tx=0).stop 2023-10-30T14:22:40.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.788+0000 7fa9b113e640 1 -- 172.21.15.46:0/2693304285 shutdown_connections 2023-10-30T14:22:40.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.788+0000 7fa9b113e640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9980664d0 0x7fa998068990 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:40.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.788+0000 7fa9b113e640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ac15d6f0 0x7fa9ac0b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:40.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.788+0000 7fa9b113e640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9ac1538a0 0x7fa9ac150750 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:40.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.788+0000 7fa9b113e640 1 --2- 172.21.15.46:0/2693304285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9ac0cf160 0x7fa9ac150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.788+0000 7fa9b113e640 1 -- 172.21.15.46:0/2693304285 >> 172.21.15.46:0/2693304285 conn(0x7fa9ac06bfe0 msgr2=0x7fa9ac151450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.788+0000 7fa9b113e640 1 -- 172.21.15.46:0/2693304285 shutdown_connections 2023-10-30T14:22:40.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.788+0000 7fa9b113e640 1 -- 172.21.15.46:0/2693304285 wait complete. 2023-10-30T14:22:40.819 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T14:22:40.819 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:22:40.820 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.956+0000 7f267d8ea640 1 Processor -- start 2023-10-30T14:22:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.956+0000 7f267d8ea640 1 -- start start 2023-10-30T14:22:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f267d8ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f267815d6f0 0x7f267815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:40.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f267d8ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26780cb800 con 0x7f26781538a0 2023-10-30T14:22:40.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f267d8ea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26780cb330 con 0x7f26780cf160 2023-10-30T14:22:40.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f267d8ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26780cad70 con 0x7f267815d6f0 2023-10-30T14:22:40.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f2676ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f26781538a0 0x7f26780ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43528/0 (socket says 172.21.15.46:43528) 2023-10-30T14:22:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f2676ffd640 1 -- 172.21.15.46:0/1216990986 learned_addr learned my addr 172.21.15.46:0/1216990986 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f266ffff640 1 --2- 172.21.15.46:0/1216990986 >> v2:172.21.15.154:3300/0 conn(0x7f267815d6f0 0x7f267815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f26767fc640 1 -- 172.21.15.46:0/1216990986 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906520072 0 0) 0x7f26780cb800 con 0x7f26781538a0 2023-10-30T14:22:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f26767fc640 1 -- 172.21.15.46:0/1216990986 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2664003530 con 0x7f26781538a0 2023-10-30T14:22:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f26767fc640 1 -- 172.21.15.46:0/1216990986 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1192169295 0 0) 0x7f2664003530 con 0x7f26781538a0 2023-10-30T14:22:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f266ffff640 1 -- 172.21.15.46:0/1216990986 >> v1:172.21.15.154:6789/0 conn(0x7f26780cf160 legacy=0x7f2678164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f266ffff640 1 -- 172.21.15.46:0/1216990986 >> v1:172.21.15.46:6789/0 conn(0x7f26781538a0 legacy=0x7f26780ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.960+0000 7f266ffff640 1 -- 172.21.15.46:0/1216990986 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26780cb050 con 0x7f267815d6f0 2023-10-30T14:22:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.964+0000 7f266ffff640 1 --2- 172.21.15.46:0/1216990986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f267815d6f0 0x7f267815a090 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f26600097c0 tx=0x7f266001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5587104dd663fee0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.964+0000 7f26767fc640 1 -- 172.21.15.46:0/1216990986 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2668001680 con 0x7f267815d6f0 2023-10-30T14:22:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.964+0000 7f26767fc640 1 -- 172.21.15.46:0/1216990986 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26600a8610 con 0x7f267815d6f0 2023-10-30T14:22:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.964+0000 7f26767fc640 1 -- 172.21.15.46:0/1216990986 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26600a8980 con 0x7f267815d6f0 2023-10-30T14:22:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.964+0000 7f267d8ea640 1 -- 172.21.15.46:0/1216990986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f267815d6f0 msgr2=0x7f267815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.964+0000 7f267d8ea640 1 --2- 172.21.15.46:0/1216990986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f267815d6f0 0x7f267815a090 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f26600097c0 tx=0x7f266001eb90 comp rx=0 tx=0).stop 2023-10-30T14:22:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.964+0000 7f267d8ea640 1 -- 172.21.15.46:0/1216990986 shutdown_connections 2023-10-30T14:22:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.964+0000 7f267d8ea640 1 --2- 172.21.15.46:0/1216990986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f267815d6f0 0x7f267815a090 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.964+0000 7f267d8ea640 1 -- 172.21.15.46:0/1216990986 >> 172.21.15.46:0/1216990986 conn(0x7f267806bfe0 msgr2=0x7f267806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.964+0000 7f267d8ea640 1 -- 172.21.15.46:0/1216990986 shutdown_connections 2023-10-30T14:22:40.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.964+0000 7f267d8ea640 1 -- 172.21.15.46:0/1216990986 wait complete. 2023-10-30T14:22:40.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.964+0000 7f267d8ea640 1 Processor -- start 2023-10-30T14:22:40.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f267d8ea640 1 -- start start 2023-10-30T14:22:40.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f267d8ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26780cf160 0x7f26780cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:40.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f267d8ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26781538a0 0x7f26780cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:40.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f267d8ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f267815d6f0 0x7f2678157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:40.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f267d8ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26780cdfb0 con 0x7f267815d6f0 2023-10-30T14:22:40.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f267d8ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26781efbb0 con 0x7f26780cf160 2023-10-30T14:22:40.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f267d8ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26781f00e0 con 0x7f26781538a0 2023-10-30T14:22:40.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f26777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f267815d6f0 0x7f2678157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:40.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f26777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f267815d6f0 0x7f2678157bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58500/0 (socket says 172.21.15.46:58500) 2023-10-30T14:22:40.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f26777fe640 1 -- 172.21.15.46:0/1184013479 learned_addr learned my addr 172.21.15.46:0/1184013479 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:40.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f266ffff640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26781538a0 0x7f26780cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:40.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f2676ffd640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26780cf160 0x7f26780cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:40.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f26777fe640 1 -- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26781538a0 msgr2=0x7f26780cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:40.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f26777fe640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26781538a0 0x7f26780cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:40.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f26777fe640 1 -- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26780cf160 msgr2=0x7f26780cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:40.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f26777fe640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26780cf160 0x7f26780cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:40.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f26777fe640 1 -- 172.21.15.46:0/1184013479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2660009420 con 0x7f267815d6f0 2023-10-30T14:22:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f26777fe640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f267815d6f0 0x7f2678157bc0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f267000c760 tx=0x7f267000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f2674ff9640 1 -- 172.21.15.46:0/1184013479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2670003b80 con 0x7f267815d6f0 2023-10-30T14:22:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f2674ff9640 1 -- 172.21.15.46:0/1184013479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2670003ce0 con 0x7f267815d6f0 2023-10-30T14:22:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f2674ff9640 1 -- 172.21.15.46:0/1184013479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267000aa90 con 0x7f267815d6f0 2023-10-30T14:22:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f267d8ea640 1 -- 172.21.15.46:0/1184013479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26781efe00 con 0x7f267815d6f0 2023-10-30T14:22:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f267d8ea640 1 -- 172.21.15.46:0/1184013479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26780ce610 con 0x7f267815d6f0 2023-10-30T14:22:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.968+0000 7f267d8ea640 1 -- 172.21.15.46:0/1184013479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26780ce800 con 0x7f267815d6f0 2023-10-30T14:22:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.972+0000 7f2674ff9640 1 -- 172.21.15.46:0/1184013479 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f267000abf0 con 0x7f267815d6f0 2023-10-30T14:22:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.972+0000 7f2674ff9640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f26580664a0 0x7f2658068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:40.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.972+0000 7f2674ff9640 1 -- 172.21.15.46:0/1184013479 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f2670084530 con 0x7f267815d6f0 2023-10-30T14:22:40.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.972+0000 7f2676ffd640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f26580664a0 0x7f2658068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:40.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.972+0000 7f2676ffd640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f26580664a0 0x7f2658068960 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f2668001ec0 tx=0x7f2668001a60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:40.976+0000 7f2674ff9640 1 -- 172.21.15.46:0/1184013479 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2670050f40 con 0x7f267815d6f0 2023-10-30T14:22:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.180+0000 7f267d8ea640 1 -- 172.21.15.46:0/1184013479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f267804e170 con 0x7f267815d6f0 2023-10-30T14:22:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.180+0000 7f2674ff9640 1 -- 172.21.15.46:0/1184013479 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f2670054560 con 0x7f267815d6f0 2023-10-30T14:22:41.188 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:41.188 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:22:39.814079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 246 2023-10-30T14:22:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.184+0000 7f267d8ea640 1 -- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f26580664a0 msgr2=0x7f2658068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.184+0000 7f267d8ea640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f26580664a0 0x7f2658068960 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f2668001ec0 tx=0x7f2668001a60 comp rx=0 tx=0).stop 2023-10-30T14:22:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.184+0000 7f267d8ea640 1 -- 172.21.15.46:0/1184013479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f267815d6f0 msgr2=0x7f2678157bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.184+0000 7f267d8ea640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f267815d6f0 0x7f2678157bc0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f267000c760 tx=0x7f267000cc20 comp rx=0 tx=0).stop 2023-10-30T14:22:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.184+0000 7f267d8ea640 1 -- 172.21.15.46:0/1184013479 shutdown_connections 2023-10-30T14:22:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.184+0000 7f267d8ea640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f26580664a0 0x7f2658068960 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.184+0000 7f267d8ea640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f267815d6f0 0x7f2678157bc0 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.184+0000 7f267d8ea640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26781538a0 0x7f26780cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.184+0000 7f267d8ea640 1 --2- 172.21.15.46:0/1184013479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26780cf160 0x7f26780cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.184+0000 7f267d8ea640 1 -- 172.21.15.46:0/1184013479 >> 172.21.15.46:0/1184013479 conn(0x7f267806bfe0 msgr2=0x7f26780c4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.184+0000 7f267d8ea640 1 -- 172.21.15.46:0/1184013479 shutdown_connections 2023-10-30T14:22:41.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:41.184+0000 7f267d8ea640 1 -- 172.21.15.46:0/1184013479 wait complete. 2023-10-30T14:22:41.210 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 246, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 246, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9594": { "addr": "172.21.15.46:6837/470164358", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 470164358, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 470164358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9594, "incarnation": 239, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 29 }, "gid_9735": { "addr": "172.21.15.46:6836/3574404756", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3574404756, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3574404756, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9735, "incarnation": 246, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 88, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:22:39.814079+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9735, "mds_1": 9594 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:22:43.211 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.348+0000 7f62c9c17640 1 Processor -- start 2023-10-30T14:22:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.348+0000 7f62c9c17640 1 -- start start 2023-10-30T14:22:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c9c17640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f62c415d6f0 0x7f62c415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c9c17640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c40cb800 con 0x7f62c41538a0 2023-10-30T14:22:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c9c17640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c40cb330 con 0x7f62c40cf160 2023-10-30T14:22:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c9c17640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c40cad70 con 0x7f62c415d6f0 2023-10-30T14:22:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f62c41538a0 0x7f62c40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43538/0 (socket says 172.21.15.46:43538) 2023-10-30T14:22:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c37fe640 1 -- 172.21.15.46:0/3152877096 learned_addr learned my addr 172.21.15.46:0/3152877096 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c2ffd640 1 --2- 172.21.15.46:0/3152877096 >> v2:172.21.15.154:3300/0 conn(0x7f62c415d6f0 0x7f62c415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801709486 0 0) 0x7f62c40cb800 con 0x7f62c41538a0 2023-10-30T14:22:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ac003530 con 0x7f62c41538a0 2023-10-30T14:22:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1772226052 0 0) 0x7f62ac003530 con 0x7f62c41538a0 2023-10-30T14:22:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 >> v2:172.21.15.154:3300/0 conn(0x7f62c415d6f0 msgr2=0x7f62c415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 --2- 172.21.15.46:0/3152877096 >> v2:172.21.15.154:3300/0 conn(0x7f62c415d6f0 0x7f62c415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 >> v1:172.21.15.154:6789/0 conn(0x7f62c40cf160 legacy=0x7f62c4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:43.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c40cb050 con 0x7f62c41538a0 2023-10-30T14:22:43.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f62b4001680 con 0x7f62c41538a0 2023-10-30T14:22:43.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 >> v1:172.21.15.46:6789/0 conn(0x7f62c41538a0 legacy=0x7f62c40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:43.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 --2- 172.21.15.46:0/3152877096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62ac003980 0x7f62ac003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:43.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 --2- 172.21.15.46:0/3152877096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 0x7f62ac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:43.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 --2- 172.21.15.46:0/3152877096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62ac005190 0x7f62ac00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:43.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c40cad70 con 0x7f62ac0047d0 2023-10-30T14:22:43.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac00c040 con 0x7f62ac005190 2023-10-30T14:22:43.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac00c200 con 0x7f62ac003980 2023-10-30T14:22:43.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c2ffd640 1 --2- 172.21.15.46:0/3152877096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 0x7f62ac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:43.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c37fe640 1 --2- 172.21.15.46:0/3152877096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62ac003980 0x7f62ac003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:43.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c3fff640 1 --2- 172.21.15.46:0/3152877096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62ac005190 0x7f62ac00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:43.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c2ffd640 1 -- 172.21.15.46:0/3152877096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62ac003980 msgr2=0x7f62ac003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:43.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c2ffd640 1 --2- 172.21.15.46:0/3152877096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62ac003980 0x7f62ac003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:43.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c2ffd640 1 -- 172.21.15.46:0/3152877096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62ac005190 msgr2=0x7f62ac00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:43.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c2ffd640 1 --2- 172.21.15.46:0/3152877096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62ac005190 0x7f62ac00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:43.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c2ffd640 1 -- 172.21.15.46:0/3152877096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac003530 con 0x7f62ac0047d0 2023-10-30T14:22:43.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.352+0000 7f62c3fff640 1 -- 172.21.15.46:0/3152877096 reap_dead start 2023-10-30T14:22:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.356+0000 7f62c2ffd640 1 --2- 172.21.15.46:0/3152877096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 0x7f62ac004c50 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f62b8009ae0 tx=0x7f62b801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a49ae868dd2a4a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.356+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b80a5640 con 0x7f62ac0047d0 2023-10-30T14:22:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.356+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b80a57a0 con 0x7f62ac0047d0 2023-10-30T14:22:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.356+0000 7f62c27fc640 1 -- 172.21.15.46:0/3152877096 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b80a5b50 con 0x7f62ac0047d0 2023-10-30T14:22:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.356+0000 7f62c9c17640 1 -- 172.21.15.46:0/3152877096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 msgr2=0x7f62ac004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.356+0000 7f62c9c17640 1 --2- 172.21.15.46:0/3152877096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 0x7f62ac004c50 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f62b8009ae0 tx=0x7f62b801ca50 comp rx=0 tx=0).stop 2023-10-30T14:22:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.356+0000 7f62c9c17640 1 -- 172.21.15.46:0/3152877096 shutdown_connections 2023-10-30T14:22:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.356+0000 7f62c9c17640 1 --2- 172.21.15.46:0/3152877096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 0x7f62ac004c50 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.356+0000 7f62c9c17640 1 -- 172.21.15.46:0/3152877096 >> 172.21.15.46:0/3152877096 conn(0x7f62c406bfe0 msgr2=0x7f62c406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.356+0000 7f62c9c17640 1 -- 172.21.15.46:0/3152877096 shutdown_connections 2023-10-30T14:22:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.356+0000 7f62c9c17640 1 -- 172.21.15.46:0/3152877096 wait complete. 2023-10-30T14:22:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c9c17640 1 Processor -- start 2023-10-30T14:22:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c9c17640 1 -- start start 2023-10-30T14:22:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c9c17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 0x7f62c41f0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c9c17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c41f01b0 0x7f62c40b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c9c17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c4158790 0x7f62c41f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c9c17640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c41e55b0 con 0x7f62ac0047d0 2023-10-30T14:22:43.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c9c17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c41ef4c0 con 0x7f62c41f01b0 2023-10-30T14:22:43.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c9c17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c41f3400 con 0x7f62c4158790 2023-10-30T14:22:43.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 0x7f62c41f0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:43.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 0x7f62c41f0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58512/0 (socket says 172.21.15.46:58512) 2023-10-30T14:22:43.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c37fe640 1 -- 172.21.15.46:0/4074113306 learned_addr learned my addr 172.21.15.46:0/4074113306 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:43.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c2ffd640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c41f01b0 0x7f62c40b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:43.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c3fff640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c4158790 0x7f62c41f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:43.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c37fe640 1 -- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c4158790 msgr2=0x7f62c41f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:43.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c37fe640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c4158790 0x7f62c41f2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:43.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c37fe640 1 -- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c41f01b0 msgr2=0x7f62c40b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:43.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c37fe640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c41f01b0 0x7f62c40b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:43.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c37fe640 1 -- 172.21.15.46:0/4074113306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c41f36e0 con 0x7f62ac0047d0 2023-10-30T14:22:43.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c3fff640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c4158790 0x7f62c41f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:43.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c37fe640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 0x7f62c41f0930 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f62b400e7b0 tx=0x7f62b400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:43.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62abfff640 1 -- 172.21.15.46:0/4074113306 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b400c8d0 con 0x7f62ac0047d0 2023-10-30T14:22:43.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62abfff640 1 -- 172.21.15.46:0/4074113306 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b400ca30 con 0x7f62ac0047d0 2023-10-30T14:22:43.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62abfff640 1 -- 172.21.15.46:0/4074113306 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b400cde0 con 0x7f62ac0047d0 2023-10-30T14:22:43.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c9c17640 1 -- 172.21.15.46:0/4074113306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c41ef6d0 con 0x7f62ac0047d0 2023-10-30T14:22:43.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.360+0000 7f62c9c17640 1 -- 172.21.15.46:0/4074113306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c41efc90 con 0x7f62ac0047d0 2023-10-30T14:22:43.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.364+0000 7f62c9c17640 1 -- 172.21.15.46:0/4074113306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c41efe80 con 0x7f62ac0047d0 2023-10-30T14:22:43.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.364+0000 7f62abfff640 1 -- 172.21.15.46:0/4074113306 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f62b401d460 con 0x7f62ac0047d0 2023-10-30T14:22:43.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.364+0000 7f62abfff640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62b00664d0 0x7f62b0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:43.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.364+0000 7f62abfff640 1 -- 172.21.15.46:0/4074113306 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f62b4084a50 con 0x7f62ac0047d0 2023-10-30T14:22:43.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.364+0000 7f62c2ffd640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62b00664d0 0x7f62b0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:43.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.364+0000 7f62c2ffd640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62b00664d0 0x7f62b0068990 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f62b80af000 tx=0x7f62b80046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:43.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.372+0000 7f62abfff640 1 -- 172.21.15.46:0/4074113306 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f62b4051290 con 0x7f62ac0047d0 2023-10-30T14:22:43.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.564+0000 7f62c9c17640 1 -- 172.21.15.46:0/4074113306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62c404e170 con 0x7f62ac0047d0 2023-10-30T14:22:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.564+0000 7f62abfff640 1 -- 172.21.15.46:0/4074113306 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f62b40548b0 con 0x7f62ac0047d0 2023-10-30T14:22:43.571 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:43.571 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:22:39.814079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 246 2023-10-30T14:22:43.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.568+0000 7f62c9c17640 1 -- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62b00664d0 msgr2=0x7f62b0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:43.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.568+0000 7f62c9c17640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62b00664d0 0x7f62b0068990 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f62b80af000 tx=0x7f62b80046d0 comp rx=0 tx=0).stop 2023-10-30T14:22:43.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.568+0000 7f62c9c17640 1 -- 172.21.15.46:0/4074113306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 msgr2=0x7f62c41f0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:43.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.568+0000 7f62c9c17640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 0x7f62c41f0930 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f62b400e7b0 tx=0x7f62b400ec70 comp rx=0 tx=0).stop 2023-10-30T14:22:43.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.568+0000 7f62c9c17640 1 -- 172.21.15.46:0/4074113306 shutdown_connections 2023-10-30T14:22:43.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.568+0000 7f62c9c17640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f62b00664d0 0x7f62b0068990 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:43.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.568+0000 7f62c9c17640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c4158790 0x7f62c41f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:43.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.568+0000 7f62c9c17640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c41f01b0 0x7f62c40b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:43.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.568+0000 7f62c9c17640 1 --2- 172.21.15.46:0/4074113306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac0047d0 0x7f62c41f0930 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:43.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.568+0000 7f62c9c17640 1 -- 172.21.15.46:0/4074113306 >> 172.21.15.46:0/4074113306 conn(0x7f62c406bfe0 msgr2=0x7f62c406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:43.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.568+0000 7f62c9c17640 1 -- 172.21.15.46:0/4074113306 shutdown_connections 2023-10-30T14:22:43.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:43.568+0000 7f62c9c17640 1 -- 172.21.15.46:0/4074113306 wait complete. 2023-10-30T14:22:43.592 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:45.593 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:45.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.764+0000 7fdc0fb1d640 1 Processor -- start 2023-10-30T14:22:45.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.764+0000 7fdc0fb1d640 1 -- start start 2023-10-30T14:22:45.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0fb1d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdc081538a0 0x7fdc0815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:45.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0fb1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc080cf160 0x7fdc080cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:45.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0fb1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc080cb800 con 0x7fdc080ca0e0 2023-10-30T14:22:45.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0fb1d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0815dc30 con 0x7fdc081538a0 2023-10-30T14:22:45.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0fb1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc080cb330 con 0x7fdc080cf160 2023-10-30T14:22:45.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0d0b8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdc080ca0e0 0x7fdc0815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43554/0 (socket says 172.21.15.46:43554) 2023-10-30T14:22:45.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0d0b8640 1 -- 172.21.15.46:0/2225766314 learned_addr learned my addr 172.21.15.46:0/2225766314 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:45.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0d8b9640 1 --2- 172.21.15.46:0/2225766314 >> v2:172.21.15.154:3300/0 conn(0x7fdc081538a0 0x7fdc0815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:45.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0e0ba640 1 --2- 172.21.15.46:0/2225766314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc080cf160 0x7fdc080cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:45.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0c8b7640 1 -- 172.21.15.46:0/2225766314 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845134192 0 0) 0x7fdc080cb800 con 0x7fdc080ca0e0 2023-10-30T14:22:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0c8b7640 1 -- 172.21.15.46:0/2225766314 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf4003530 con 0x7fdc080ca0e0 2023-10-30T14:22:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0d8b9640 1 -- 172.21.15.46:0/2225766314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc080cf160 msgr2=0x7fdc080cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0d8b9640 1 --2- 172.21.15.46:0/2225766314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc080cf160 0x7fdc080cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0d8b9640 1 -- 172.21.15.46:0/2225766314 >> v1:172.21.15.46:6789/0 conn(0x7fdc080ca0e0 legacy=0x7fdc0815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:45.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0c8b7640 1 -- 172.21.15.46:0/2225766314 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 553035322 0 0) 0x7fdbf4003530 con 0x7fdc080ca0e0 2023-10-30T14:22:45.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.768+0000 7fdc0d8b9640 1 -- 172.21.15.46:0/2225766314 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc080cad70 con 0x7fdc081538a0 2023-10-30T14:22:45.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.772+0000 7fdc0d8b9640 1 --2- 172.21.15.46:0/2225766314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc081538a0 0x7fdc0815d6f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fdc040097c0 tx=0x7fdc0401eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6c941e932eb8a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:45.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.772+0000 7fdc0c8b7640 1 -- 172.21.15.46:0/2225766314 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8001680 con 0x7fdc081538a0 2023-10-30T14:22:45.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.772+0000 7fdc0c8b7640 1 -- 172.21.15.46:0/2225766314 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc040a85d0 con 0x7fdc081538a0 2023-10-30T14:22:45.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.772+0000 7fdc0c8b7640 1 -- 172.21.15.46:0/2225766314 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc040a8940 con 0x7fdc081538a0 2023-10-30T14:22:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.772+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/2225766314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc081538a0 msgr2=0x7fdc0815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.772+0000 7fdc0fb1d640 1 --2- 172.21.15.46:0/2225766314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc081538a0 0x7fdc0815d6f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fdc040097c0 tx=0x7fdc0401eb90 comp rx=0 tx=0).stop 2023-10-30T14:22:45.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.772+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/2225766314 shutdown_connections 2023-10-30T14:22:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.772+0000 7fdc0fb1d640 1 --2- 172.21.15.46:0/2225766314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc080cf160 0x7fdc080cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.772+0000 7fdc0fb1d640 1 --2- 172.21.15.46:0/2225766314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc081538a0 0x7fdc0815d6f0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.772+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/2225766314 >> 172.21.15.46:0/2225766314 conn(0x7fdc0806bfe0 msgr2=0x7fdc0806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.772+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/2225766314 shutdown_connections 2023-10-30T14:22:45.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.772+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/2225766314 wait complete. 2023-10-30T14:22:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0fb1d640 1 Processor -- start 2023-10-30T14:22:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0fb1d640 1 -- start start 2023-10-30T14:22:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0fb1d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc080ca0e0 0x7fdc080cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:45.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0fb1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc080cf160 0x7fdc08065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0fb1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc081538a0 0x7fdc08145b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0fb1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc081585a0 con 0x7fdc080cf160 2023-10-30T14:22:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0fb1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc080ce7e0 con 0x7fdc081538a0 2023-10-30T14:22:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0fb1d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc080ced50 con 0x7fdc080ca0e0 2023-10-30T14:22:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0d0b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc080cf160 0x7fdc08065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0d0b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc080cf160 0x7fdc08065ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58522/0 (socket says 172.21.15.46:58522) 2023-10-30T14:22:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0d0b8640 1 -- 172.21.15.46:0/1666673793 learned_addr learned my addr 172.21.15.46:0/1666673793 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0d8b9640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc080ca0e0 0x7fdc080cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.776+0000 7fdc0e0ba640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc081538a0 0x7fdc08145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.780+0000 7fdc0d0b8640 1 -- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc080ca0e0 msgr2=0x7fdc080cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.780+0000 7fdc0d0b8640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc080ca0e0 0x7fdc080cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.780+0000 7fdc0d0b8640 1 -- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc081538a0 msgr2=0x7fdc08145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.780+0000 7fdc0d0b8640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc081538a0 0x7fdc08145b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:45.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.780+0000 7fdc0d0b8640 1 -- 172.21.15.46:0/1666673793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc04009420 con 0x7fdc080cf160 2023-10-30T14:22:45.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.780+0000 7fdc0d0b8640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc080cf160 0x7fdc08065ce0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800c780 tx=0x7fdbf800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.780+0000 7fdbfe7fc640 1 -- 172.21.15.46:0/1666673793 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8005ae0 con 0x7fdc080cf160 2023-10-30T14:22:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.780+0000 7fdbfe7fc640 1 -- 172.21.15.46:0/1666673793 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf8005c40 con 0x7fdc080cf160 2023-10-30T14:22:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.780+0000 7fdbfe7fc640 1 -- 172.21.15.46:0/1666673793 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8007070 con 0x7fdc080cf160 2023-10-30T14:22:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.780+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/1666673793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc080cd430 con 0x7fdc080cf160 2023-10-30T14:22:45.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.780+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/1666673793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc08159110 con 0x7fdc080cf160 2023-10-30T14:22:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.780+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/1666673793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc080b2a50 con 0x7fdc080cf160 2023-10-30T14:22:45.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.784+0000 7fdbfe7fc640 1 -- 172.21.15.46:0/1666673793 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdbf80071f0 con 0x7fdc080cf160 2023-10-30T14:22:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.784+0000 7fdbfe7fc640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbec0664a0 0x7fdbec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.784+0000 7fdbfe7fc640 1 -- 172.21.15.46:0/1666673793 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fdbf8083ff0 con 0x7fdc080cf160 2023-10-30T14:22:45.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.784+0000 7fdc0d8b9640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbec0664a0 0x7fdbec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:45.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.784+0000 7fdc0d8b9640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbec0664a0 0x7fdbec068960 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fdc040b2000 tx=0x7fdc040aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:45.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:45.792+0000 7fdbfe7fc640 1 -- 172.21.15.46:0/1666673793 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdbf80b5090 con 0x7fdc080cf160 2023-10-30T14:22:46.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.004+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/1666673793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc080b4c50 con 0x7fdc080cf160 2023-10-30T14:22:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.004+0000 7fdbfe7fc640 1 -- 172.21.15.46:0/1666673793 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+5729 (secure 0 0 0) 0x7fdbf8017070 con 0x7fdc080cf160 2023-10-30T14:22:46.010 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:46.010 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:22:39.814079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 246 2023-10-30T14:22:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.008+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbec0664a0 msgr2=0x7fdbec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.008+0000 7fdc0fb1d640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbec0664a0 0x7fdbec068960 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fdc040b2000 tx=0x7fdc040aa070 comp rx=0 tx=0).stop 2023-10-30T14:22:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.008+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/1666673793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc080cf160 msgr2=0x7fdc08065ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.008+0000 7fdc0fb1d640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc080cf160 0x7fdc08065ce0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800c780 tx=0x7fdbf800cc40 comp rx=0 tx=0).stop 2023-10-30T14:22:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.008+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/1666673793 shutdown_connections 2023-10-30T14:22:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.008+0000 7fdc0fb1d640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fdbec0664a0 0x7fdbec068960 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.008+0000 7fdc0fb1d640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc081538a0 0x7fdc08145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.008+0000 7fdc0fb1d640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc080cf160 0x7fdc08065ce0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.008+0000 7fdc0fb1d640 1 --2- 172.21.15.46:0/1666673793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc080ca0e0 0x7fdc080cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.008+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/1666673793 >> 172.21.15.46:0/1666673793 conn(0x7fdc0806bfe0 msgr2=0x7fdc081e2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.008+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/1666673793 shutdown_connections 2023-10-30T14:22:46.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:46.008+0000 7fdc0fb1d640 1 -- 172.21.15.46:0/1666673793 wait complete. 2023-10-30T14:22:46.040 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:48.042 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:48.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.188+0000 7f7e0749f640 1 Processor -- start 2023-10-30T14:22:48.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.188+0000 7f7e0749f640 1 -- start start 2023-10-30T14:22:48.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.188+0000 7f7e0749f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7e0015cd50 0x7f7e000c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:48.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.188+0000 7f7e0749f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0015a6c0 con 0x7f7e000cefa0 2023-10-30T14:22:48.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.188+0000 7f7e0749f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0015d300 con 0x7f7e001538f0 2023-10-30T14:22:48.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.188+0000 7f7e0749f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0015d5e0 con 0x7f7e0015cd50 2023-10-30T14:22:48.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.188+0000 7f7e06c9e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7e000cefa0 0x7f7e001640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35488/0 (socket says 172.21.15.46:35488) 2023-10-30T14:22:48.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.188+0000 7f7e06c9e640 1 -- 172.21.15.46:0/2371702592 learned_addr learned my addr 172.21.15.46:0/2371702592 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:48.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.188+0000 7f7e05c9c640 1 --2- 172.21.15.46:0/2371702592 >> v2:172.21.15.154:3300/0 conn(0x7f7e0015cd50 0x7f7e000c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:48.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.192+0000 7f7e0549b640 1 -- 172.21.15.46:0/2371702592 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008117574 0 0) 0x7f7e0015a6c0 con 0x7f7e000cefa0 2023-10-30T14:22:48.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.192+0000 7f7e0549b640 1 -- 172.21.15.46:0/2371702592 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dec003530 con 0x7f7e000cefa0 2023-10-30T14:22:48.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.192+0000 7f7e05c9c640 1 -- 172.21.15.46:0/2371702592 >> v1:172.21.15.154:6789/0 conn(0x7f7e001538f0 legacy=0x7f7e00159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:48.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.192+0000 7f7e05c9c640 1 -- 172.21.15.46:0/2371702592 >> v1:172.21.15.46:6789/0 conn(0x7f7e000cefa0 legacy=0x7f7e001640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:48.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.192+0000 7f7e05c9c640 1 -- 172.21.15.46:0/2371702592 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0015d8c0 con 0x7f7e0015cd50 2023-10-30T14:22:48.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.196+0000 7f7e05c9c640 1 --2- 172.21.15.46:0/2371702592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e0015cd50 0x7f7e000c9f90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7df00097c0 tx=0x7f7df001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8275cd9dfe479b09 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:48.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.196+0000 7f7e0549b640 1 -- 172.21.15.46:0/2371702592 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df00a8970 con 0x7f7e0015cd50 2023-10-30T14:22:48.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.196+0000 7f7e0549b640 1 -- 172.21.15.46:0/2371702592 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df00a8ad0 con 0x7f7e0015cd50 2023-10-30T14:22:48.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.196+0000 7f7e0549b640 1 -- 172.21.15.46:0/2371702592 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df00a8e60 con 0x7f7e0015cd50 2023-10-30T14:22:48.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.196+0000 7f7e0749f640 1 -- 172.21.15.46:0/2371702592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e0015cd50 msgr2=0x7f7e000c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:48.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.196+0000 7f7e0749f640 1 --2- 172.21.15.46:0/2371702592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e0015cd50 0x7f7e000c9f90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7df00097c0 tx=0x7f7df001ec70 comp rx=0 tx=0).stop 2023-10-30T14:22:48.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.196+0000 7f7e0749f640 1 -- 172.21.15.46:0/2371702592 shutdown_connections 2023-10-30T14:22:48.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.196+0000 7f7e0749f640 1 --2- 172.21.15.46:0/2371702592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e0015cd50 0x7f7e000c9f90 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:48.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.196+0000 7f7e0749f640 1 -- 172.21.15.46:0/2371702592 >> 172.21.15.46:0/2371702592 conn(0x7f7e0006a640 msgr2=0x7f7e00065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:48.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.196+0000 7f7e0749f640 1 -- 172.21.15.46:0/2371702592 shutdown_connections 2023-10-30T14:22:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.196+0000 7f7e0749f640 1 -- 172.21.15.46:0/2371702592 wait complete. 2023-10-30T14:22:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e0749f640 1 Processor -- start 2023-10-30T14:22:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e0749f640 1 -- start start 2023-10-30T14:22:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e0749f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e000cefa0 0x7f7e001f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e0749f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e001538f0 0x7f7e001453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e0749f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e0015cd50 0x7f7e000cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e0749f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0015bbe0 con 0x7f7e0015cd50 2023-10-30T14:22:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e0749f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0014db50 con 0x7f7e000cefa0 2023-10-30T14:22:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e0749f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0014e080 con 0x7f7e001538f0 2023-10-30T14:22:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e06c9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e0015cd50 0x7f7e000cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:48.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e06c9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e0015cd50 0x7f7e000cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49716/0 (socket says 172.21.15.46:49716) 2023-10-30T14:22:48.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e06c9e640 1 -- 172.21.15.46:0/3961840193 learned_addr learned my addr 172.21.15.46:0/3961840193 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:48.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e0649d640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e000cefa0 0x7f7e001f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:48.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e05c9c640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e001538f0 0x7f7e001453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:48.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e06c9e640 1 -- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e001538f0 msgr2=0x7f7e001453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:48.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e06c9e640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e001538f0 0x7f7e001453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:48.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e06c9e640 1 -- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e000cefa0 msgr2=0x7f7e001f3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:48.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e06c9e640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e000cefa0 0x7f7e001f3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:48.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e06c9e640 1 -- 172.21.15.46:0/3961840193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df0009420 con 0x7f7e0015cd50 2023-10-30T14:22:48.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.200+0000 7f7e06c9e640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e0015cd50 0x7f7e000cdf90 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f7df800cb70 tx=0x7f7df800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:48.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.204+0000 7f7df6ffd640 1 -- 172.21.15.46:0/3961840193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df800ad40 con 0x7f7e0015cd50 2023-10-30T14:22:48.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.204+0000 7f7df6ffd640 1 -- 172.21.15.46:0/3961840193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df8012400 con 0x7f7e0015cd50 2023-10-30T14:22:48.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.204+0000 7f7df6ffd640 1 -- 172.21.15.46:0/3961840193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df800aea0 con 0x7f7e0015cd50 2023-10-30T14:22:48.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.204+0000 7f7e0749f640 1 -- 172.21.15.46:0/3961840193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e001f2bc0 con 0x7f7e0015cd50 2023-10-30T14:22:48.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.204+0000 7f7e0749f640 1 -- 172.21.15.46:0/3961840193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e000b4480 con 0x7f7e0015cd50 2023-10-30T14:22:48.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.204+0000 7f7e0749f640 1 -- 172.21.15.46:0/3961840193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e001f2d50 con 0x7f7e0015cd50 2023-10-30T14:22:48.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.204+0000 7f7df6ffd640 1 -- 172.21.15.46:0/3961840193 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7df8007070 con 0x7f7e0015cd50 2023-10-30T14:22:48.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.204+0000 7f7df6ffd640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7de40664d0 0x7f7de4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:48.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.204+0000 7f7df6ffd640 1 -- 172.21.15.46:0/3961840193 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f7df80848a0 con 0x7f7e0015cd50 2023-10-30T14:22:48.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.208+0000 7f7e0649d640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7de40664d0 0x7f7de4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:48.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.208+0000 7f7e0649d640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7de40664d0 0x7f7de4068990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc007fd0 tx=0x7f7dfc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:48.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.212+0000 7f7df6ffd640 1 -- 172.21.15.46:0/3961840193 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7df8051310 con 0x7f7e0015cd50 2023-10-30T14:22:48.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 -- 172.21.15.46:0/3961840193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e0004e170 con 0x7f7e0015cd50 2023-10-30T14:22:48.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7df6ffd640 1 -- 172.21.15.46:0/3961840193 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f7df8017020 con 0x7f7e0015cd50 2023-10-30T14:22:48.437 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:48.437 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:22:39.814079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:48.438 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 246 2023-10-30T14:22:48.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 -- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7de40664d0 msgr2=0x7f7de4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:48.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7de40664d0 0x7f7de4068990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc007fd0 tx=0x7f7dfc009040 comp rx=0 tx=0).stop 2023-10-30T14:22:48.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 -- 172.21.15.46:0/3961840193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e0015cd50 msgr2=0x7f7e000cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:48.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e0015cd50 0x7f7e000cdf90 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f7df800cb70 tx=0x7f7df800a430 comp rx=0 tx=0).stop 2023-10-30T14:22:48.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 -- 172.21.15.46:0/3961840193 shutdown_connections 2023-10-30T14:22:48.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7de40664d0 0x7f7de4068990 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:48.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e0015cd50 0x7f7e000cdf90 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:48.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e001538f0 0x7f7e001453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:48.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 --2- 172.21.15.46:0/3961840193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e000cefa0 0x7f7e001f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:48.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 -- 172.21.15.46:0/3961840193 >> 172.21.15.46:0/3961840193 conn(0x7f7e0006a640 msgr2=0x7f7e000c48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:48.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 -- 172.21.15.46:0/3961840193 shutdown_connections 2023-10-30T14:22:48.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:48.432+0000 7f7e0749f640 1 -- 172.21.15.46:0/3961840193 wait complete. 2023-10-30T14:22:48.462 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:50.463 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:50.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.604+0000 7f8f96b3e640 1 Processor -- start 2023-10-30T14:22:50.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.604+0000 7f8f96b3e640 1 -- start start 2023-10-30T14:22:50.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f96b3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8f9015d6f0 0x7f8f9015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:50.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f96b3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8f900cf160 0x7f8f900cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:50.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f96b3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f900cb800 con 0x7f8f901538a0 2023-10-30T14:22:50.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f96b3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f900cb330 con 0x7f8f900cf160 2023-10-30T14:22:50.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f96b3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f900cad70 con 0x7f8f9015d6f0 2023-10-30T14:22:50.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f948da640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8f901538a0 0x7f8f900ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35498/0 (socket says 172.21.15.46:35498) 2023-10-30T14:22:50.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f948da640 1 -- 172.21.15.46:0/1511658818 learned_addr learned my addr 172.21.15.46:0/1511658818 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:50.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f950db640 1 --2- 172.21.15.46:0/1511658818 >> v2:172.21.15.46:3300/0 conn(0x7f8f900cf160 0x7f8f900cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:50.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f8ffff640 1 --2- 172.21.15.46:0/1511658818 >> v2:172.21.15.154:3300/0 conn(0x7f8f9015d6f0 0x7f8f9015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:50.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f950db640 1 -- 172.21.15.46:0/1511658818 >> v2:172.21.15.154:3300/0 conn(0x7f8f9015d6f0 msgr2=0x7f8f9015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:50.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f950db640 1 --2- 172.21.15.46:0/1511658818 >> v2:172.21.15.154:3300/0 conn(0x7f8f9015d6f0 0x7f8f9015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:50.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f950db640 1 -- 172.21.15.46:0/1511658818 >> v1:172.21.15.46:6789/0 conn(0x7f8f901538a0 legacy=0x7f8f900ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:50.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.608+0000 7f8f950db640 1 -- 172.21.15.46:0/1511658818 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f900cb050 con 0x7f8f900cf160 2023-10-30T14:22:50.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f950db640 1 --2- 172.21.15.46:0/1511658818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f900cf160 0x7f8f900cf590 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f8f900b30a0 tx=0x7f8f8801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=891bc52657ebc72a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:50.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f8f7fe640 1 -- 172.21.15.46:0/1511658818 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f880b0440 con 0x7f8f900cf160 2023-10-30T14:22:50.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f8f7fe640 1 -- 172.21.15.46:0/1511658818 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f880b05a0 con 0x7f8f900cf160 2023-10-30T14:22:50.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f8f7fe640 1 -- 172.21.15.46:0/1511658818 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f880b0930 con 0x7f8f900cf160 2023-10-30T14:22:50.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f96b3e640 1 -- 172.21.15.46:0/1511658818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f900cf160 msgr2=0x7f8f900cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:50.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f96b3e640 1 --2- 172.21.15.46:0/1511658818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f900cf160 0x7f8f900cf590 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f8f900b30a0 tx=0x7f8f8801f0f0 comp rx=0 tx=0).stop 2023-10-30T14:22:50.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f96b3e640 1 -- 172.21.15.46:0/1511658818 shutdown_connections 2023-10-30T14:22:50.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f96b3e640 1 --2- 172.21.15.46:0/1511658818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f900cf160 0x7f8f900cf590 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:50.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f96b3e640 1 --2- 172.21.15.46:0/1511658818 >> v2:172.21.15.154:3300/0 conn(0x7f8f9015d6f0 0x7f8f9015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:50.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f96b3e640 1 -- 172.21.15.46:0/1511658818 >> 172.21.15.46:0/1511658818 conn(0x7f8f9006bfe0 msgr2=0x7f8f9006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:50.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f96b3e640 1 -- 172.21.15.46:0/1511658818 shutdown_connections 2023-10-30T14:22:50.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f96b3e640 1 -- 172.21.15.46:0/1511658818 wait complete. 2023-10-30T14:22:50.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f96b3e640 1 Processor -- start 2023-10-30T14:22:50.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.612+0000 7f8f96b3e640 1 -- start start 2023-10-30T14:22:50.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f96b3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f900cf160 0x7f8f9014dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:50.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f96b3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f901538a0 0x7f8f901591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:50.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f96b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f9015d6f0 0x7f8f901e5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:50.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f96b3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f901e4e60 con 0x7f8f900cf160 2023-10-30T14:22:50.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f96b3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f901596f0 con 0x7f8f901538a0 2023-10-30T14:22:50.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f96b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f90158480 con 0x7f8f9015d6f0 2023-10-30T14:22:50.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f948da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f900cf160 0x7f8f9014dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:50.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f948da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f900cf160 0x7f8f9014dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49730/0 (socket says 172.21.15.46:49730) 2023-10-30T14:22:50.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f948da640 1 -- 172.21.15.46:0/3677871660 learned_addr learned my addr 172.21.15.46:0/3677871660 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:50.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f8ffff640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f901538a0 0x7f8f901591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:50.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f950db640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f9015d6f0 0x7f8f901e5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:50.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f948da640 1 -- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f9015d6f0 msgr2=0x7f8f901e5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:50.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f948da640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f9015d6f0 0x7f8f901e5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:50.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f948da640 1 -- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f901538a0 msgr2=0x7f8f901591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:50.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f948da640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f901538a0 0x7f8f901591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:50.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f948da640 1 -- 172.21.15.46:0/3677871660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f88009a70 con 0x7f8f900cf160 2023-10-30T14:22:50.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f948da640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f900cf160 0x7f8f9014dc50 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400c530 tx=0x7f8f8400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:50.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f8d7fa640 1 -- 172.21.15.46:0/3677871660 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84005cf0 con 0x7f8f900cf160 2023-10-30T14:22:50.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f8d7fa640 1 -- 172.21.15.46:0/3677871660 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f84005e50 con 0x7f8f900cf160 2023-10-30T14:22:50.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f96b3e640 1 -- 172.21.15.46:0/3677871660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f90158d20 con 0x7f8f900cf160 2023-10-30T14:22:50.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f8d7fa640 1 -- 172.21.15.46:0/3677871660 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84007270 con 0x7f8f900cf160 2023-10-30T14:22:50.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f96b3e640 1 -- 172.21.15.46:0/3677871660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f901589b0 con 0x7f8f900cf160 2023-10-30T14:22:50.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.616+0000 7f8f96b3e640 1 -- 172.21.15.46:0/3677871660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f900b2a50 con 0x7f8f900cf160 2023-10-30T14:22:50.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.620+0000 7f8f8d7fa640 1 -- 172.21.15.46:0/3677871660 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f840073d0 con 0x7f8f900cf160 2023-10-30T14:22:50.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.620+0000 7f8f8d7fa640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8f740664a0 0x7f8f74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:50.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.620+0000 7f8f8d7fa640 1 -- 172.21.15.46:0/3677871660 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f8f84083da0 con 0x7f8f900cf160 2023-10-30T14:22:50.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.620+0000 7f8f8ffff640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8f740664a0 0x7f8f74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:50.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.620+0000 7f8f8ffff640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8f740664a0 0x7f8f74068960 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f8f900c5540 tx=0x7f8f7c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:50.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.624+0000 7f8f8d7fa640 1 -- 172.21.15.46:0/3677871660 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f840507b0 con 0x7f8f900cf160 2023-10-30T14:22:50.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.816+0000 7f8f96b3e640 1 -- 172.21.15.46:0/3677871660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f900cc470 con 0x7f8f900cf160 2023-10-30T14:22:50.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.820+0000 7f8f8d7fa640 1 -- 172.21.15.46:0/3677871660 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+5729 (secure 0 0 0) 0x7f8f84017070 con 0x7f8f900cf160 2023-10-30T14:22:50.825 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:50.826 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:22:39.814079+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:50.826 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 246 2023-10-30T14:22:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.824+0000 7f8f96b3e640 1 -- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8f740664a0 msgr2=0x7f8f74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:50.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.824+0000 7f8f96b3e640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8f740664a0 0x7f8f74068960 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f8f900c5540 tx=0x7f8f7c009190 comp rx=0 tx=0).stop 2023-10-30T14:22:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.824+0000 7f8f96b3e640 1 -- 172.21.15.46:0/3677871660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f900cf160 msgr2=0x7f8f9014dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.824+0000 7f8f96b3e640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f900cf160 0x7f8f9014dc50 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400c530 tx=0x7f8f8400c9f0 comp rx=0 tx=0).stop 2023-10-30T14:22:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.824+0000 7f8f96b3e640 1 -- 172.21.15.46:0/3677871660 shutdown_connections 2023-10-30T14:22:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.824+0000 7f8f96b3e640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8f740664a0 0x7f8f74068960 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:50.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.824+0000 7f8f96b3e640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f9015d6f0 0x7f8f901e5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:50.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.824+0000 7f8f96b3e640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f901538a0 0x7f8f901591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:50.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.824+0000 7f8f96b3e640 1 --2- 172.21.15.46:0/3677871660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f900cf160 0x7f8f9014dc50 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:50.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.824+0000 7f8f96b3e640 1 -- 172.21.15.46:0/3677871660 >> 172.21.15.46:0/3677871660 conn(0x7f8f9006bfe0 msgr2=0x7f8f900c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:50.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.824+0000 7f8f96b3e640 1 -- 172.21.15.46:0/3677871660 shutdown_connections 2023-10-30T14:22:50.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:50.824+0000 7f8f96b3e640 1 -- 172.21.15.46:0/3677871660 wait complete. 2023-10-30T14:22:50.858 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:52.859 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:53.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.000+0000 7fccf7479640 1 Processor -- start 2023-10-30T14:22:53.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.000+0000 7fccf7479640 1 -- start start 2023-10-30T14:22:53.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf7479640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf01538a0 0x7fccf015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:53.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf7479640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf00cb800 con 0x7fccf00ca0e0 2023-10-30T14:22:53.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf7479640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf015dc30 con 0x7fccf00cf160 2023-10-30T14:22:53.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf7479640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf00cb330 con 0x7fccf01538a0 2023-10-30T14:22:53.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf4a14640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fccf00ca0e0 0x7fccf015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35510/0 (socket says 172.21.15.46:35510) 2023-10-30T14:22:53.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf4a14640 1 -- 172.21.15.46:0/3068410834 learned_addr learned my addr 172.21.15.46:0/3068410834 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:22:53.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf5215640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf01538a0 0x7fccf015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:53.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694053708 0 0) 0x7fccf00cb800 con 0x7fccf00ca0e0 2023-10-30T14:22:53.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccdc003530 con 0x7fccf00ca0e0 2023-10-30T14:22:53.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147085353 0 0) 0x7fcce80014a0 con 0x7fccf00cf160 2023-10-30T14:22:53.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccf00cb800 con 0x7fccf00cf160 2023-10-30T14:22:53.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 425812023 0 0) 0x7fccdc003530 con 0x7fccf00ca0e0 2023-10-30T14:22:53.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf01538a0 msgr2=0x7fccf015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:53.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf01538a0 0x7fccf015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:53.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 >> v1:172.21.15.154:6789/0 conn(0x7fccf00cf160 legacy=0x7fccf0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:53.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf00cad70 con 0x7fccf00ca0e0 2023-10-30T14:22:53.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fcce0001680 con 0x7fccf00ca0e0 2023-10-30T14:22:53.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 >> v1:172.21.15.46:6789/0 conn(0x7fccf00ca0e0 legacy=0x7fccf015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:53.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccdc003b40 0x7fccdc003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:53.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccdc0047d0 0x7fccdc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:53.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf01538a0 0x7fccdc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:53.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf00cb330 con 0x7fccdc003b40 2023-10-30T14:22:53.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc00ba40 con 0x7fccdc0047d0 2023-10-30T14:22:53.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc00bbd0 con 0x7fccf01538a0 2023-10-30T14:22:53.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf5215640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccdc003b40 0x7fccdc003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:53.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf4a14640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccdc0047d0 0x7fccdc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:53.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf5a16640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf01538a0 0x7fccdc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:53.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf5215640 1 -- 172.21.15.46:0/3068410834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf01538a0 msgr2=0x7fccdc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:53.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf5215640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf01538a0 0x7fccdc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:53.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf5215640 1 -- 172.21.15.46:0/3068410834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccdc0047d0 msgr2=0x7fccdc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:53.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf5215640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccdc0047d0 0x7fccdc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:53.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf5215640 1 -- 172.21.15.46:0/3068410834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc003530 con 0x7fccdc003b40 2023-10-30T14:22:53.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.004+0000 7fccf5a16640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf01538a0 0x7fccdc005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:53.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.008+0000 7fccf5215640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccdc003b40 0x7fccdc003f40 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fccec009ae0 tx=0x7fccec01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08d0559d283ae5e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:53.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.008+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec0a55e0 con 0x7fccdc003b40 2023-10-30T14:22:53.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.008+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce8002100 con 0x7fccdc003b40 2023-10-30T14:22:53.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.008+0000 7fcce7fff640 1 -- 172.21.15.46:0/3068410834 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec0a5980 con 0x7fccdc003b40 2023-10-30T14:22:53.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.008+0000 7fccf7479640 1 -- 172.21.15.46:0/3068410834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccdc003b40 msgr2=0x7fccdc003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:53.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.008+0000 7fccf7479640 1 --2- 172.21.15.46:0/3068410834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccdc003b40 0x7fccdc003f40 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fccec009ae0 tx=0x7fccec01c9f0 comp rx=0 tx=0).stop 2023-10-30T14:22:53.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.008+0000 7fccf5a16640 1 -- 172.21.15.46:0/3068410834 reap_dead start 2023-10-30T14:22:53.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.008+0000 7fccf7479640 1 -- 172.21.15.46:0/3068410834 shutdown_connections 2023-10-30T14:22:53.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.008+0000 7fccf7479640 1 -- 172.21.15.46:0/3068410834 >> 172.21.15.46:0/3068410834 conn(0x7fccf006bfe0 msgr2=0x7fccf006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:53.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.008+0000 7fccf7479640 1 -- 172.21.15.46:0/3068410834 shutdown_connections 2023-10-30T14:22:53.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.008+0000 7fccf7479640 1 -- 172.21.15.46:0/3068410834 wait complete. 2023-10-30T14:22:53.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf7479640 1 Processor -- start 2023-10-30T14:22:53.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf7479640 1 -- start start 2023-10-30T14:22:53.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf7479640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccdc003b40 0x7fccf01e3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:53.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf7479640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccf0157d60 0x7fccf01e5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:53.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf7479640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf014e500 0x7fccf015d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:53.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf7479640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf01f2440 con 0x7fccdc003b40 2023-10-30T14:22:53.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf7479640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf01eef20 con 0x7fccf0157d60 2023-10-30T14:22:53.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf7479640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf01f2630 con 0x7fccf014e500 2023-10-30T14:22:53.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf5215640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccdc003b40 0x7fccf01e3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:53.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf5215640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccdc003b40 0x7fccf01e3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49740/0 (socket says 172.21.15.46:49740) 2023-10-30T14:22:53.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf5215640 1 -- 172.21.15.46:0/682188314 learned_addr learned my addr 172.21.15.46:0/682188314 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:53.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf5a16640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf014e500 0x7fccf015d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:53.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf4a14640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccf0157d60 0x7fccf01e5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:53.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf5215640 1 -- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf014e500 msgr2=0x7fccf015d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:53.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf5215640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf014e500 0x7fccf015d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:53.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf5215640 1 -- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccf0157d60 msgr2=0x7fccf01e5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:53.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf5215640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccf0157d60 0x7fccf01e5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:53.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf5215640 1 -- 172.21.15.46:0/682188314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf01ef1e0 con 0x7fccdc003b40 2023-10-30T14:22:53.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf4a14640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccf0157d60 0x7fccf01e5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:53.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf5215640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccdc003b40 0x7fccf01e3720 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fccec009ae0 tx=0x7fccdc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:53.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fcce5ffb640 1 -- 172.21.15.46:0/682188314 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec004220 con 0x7fccdc003b40 2023-10-30T14:22:53.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fcce5ffb640 1 -- 172.21.15.46:0/682188314 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccec004380 con 0x7fccdc003b40 2023-10-30T14:22:53.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf7479640 1 -- 172.21.15.46:0/682188314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf01ef4c0 con 0x7fccdc003b40 2023-10-30T14:22:53.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fcce5ffb640 1 -- 172.21.15.46:0/682188314 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec003820 con 0x7fccdc003b40 2023-10-30T14:22:53.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.012+0000 7fccf7479640 1 -- 172.21.15.46:0/682188314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf0066240 con 0x7fccdc003b40 2023-10-30T14:22:53.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.016+0000 7fccf7479640 1 -- 172.21.15.46:0/682188314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf0159850 con 0x7fccdc003b40 2023-10-30T14:22:53.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.016+0000 7fcce5ffb640 1 -- 172.21.15.46:0/682188314 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fccec0b4040 con 0x7fccdc003b40 2023-10-30T14:22:53.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.016+0000 7fcce5ffb640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fccd40664a0 0x7fccd4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:53.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.016+0000 7fcce5ffb640 1 -- 172.21.15.46:0/682188314 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fccec11ede0 con 0x7fccdc003b40 2023-10-30T14:22:53.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.016+0000 7fccf4a14640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fccd40664a0 0x7fccd4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:53.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.020+0000 7fccf4a14640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fccd40664a0 0x7fccd4068960 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fccf01e3410 tx=0x7fcce0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:53.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.020+0000 7fcce5ffb640 1 -- 172.21.15.46:0/682188314 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fccec0eb7f0 con 0x7fccdc003b40 2023-10-30T14:22:53.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.220+0000 7fccf7479640 1 -- 172.21.15.46:0/682188314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccf004e170 con 0x7fccdc003b40 2023-10-30T14:22:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.220+0000 7fcce5ffb640 1 -- 172.21.15.46:0/682188314 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+5731 (secure 0 0 0) 0x7fccec0eee10 con 0x7fccdc003b40 2023-10-30T14:22:53.227 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:53.227 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:22:51.327203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:resolve","state_seq":27,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 247 2023-10-30T14:22:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.224+0000 7fccf7479640 1 -- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fccd40664a0 msgr2=0x7fccd4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.224+0000 7fccf7479640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fccd40664a0 0x7fccd4068960 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fccf01e3410 tx=0x7fcce0009040 comp rx=0 tx=0).stop 2023-10-30T14:22:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.224+0000 7fccf7479640 1 -- 172.21.15.46:0/682188314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccdc003b40 msgr2=0x7fccf01e3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.224+0000 7fccf7479640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccdc003b40 0x7fccf01e3720 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fccec009ae0 tx=0x7fccdc005800 comp rx=0 tx=0).stop 2023-10-30T14:22:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.224+0000 7fccf7479640 1 -- 172.21.15.46:0/682188314 shutdown_connections 2023-10-30T14:22:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.224+0000 7fccf7479640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fccd40664a0 0x7fccd4068960 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.224+0000 7fccf7479640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccf014e500 0x7fccf015d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.224+0000 7fccf7479640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccf0157d60 0x7fccf01e5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.224+0000 7fccf7479640 1 --2- 172.21.15.46:0/682188314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccdc003b40 0x7fccf01e3720 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.224+0000 7fccf7479640 1 -- 172.21.15.46:0/682188314 >> 172.21.15.46:0/682188314 conn(0x7fccf006bfe0 msgr2=0x7fccf0151a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.224+0000 7fccf7479640 1 -- 172.21.15.46:0/682188314 shutdown_connections 2023-10-30T14:22:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:53.224+0000 7fccf7479640 1 -- 172.21.15.46:0/682188314 wait complete. 2023-10-30T14:22:53.249 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 247, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 247, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9594": { "addr": "172.21.15.46:6837/470164358", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 470164358, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 470164358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9594, "incarnation": 239, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 29 }, "gid_9735": { "addr": "172.21.15.46:6836/3574404756", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3574404756, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3574404756, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9735, "incarnation": 246, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:resolve", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 88, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:22:51.327203+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9735, "mds_1": 9594 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:22:55.251 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:55.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.388+0000 7fc3dbcd6640 1 Processor -- start 2023-10-30T14:22:55.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.388+0000 7fc3dbcd6640 1 -- start start 2023-10-30T14:22:55.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3dbcd6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d41538a0 0x7fc3d415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:55.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3dbcd6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3d40ca0e0 0x7fc3d415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:55.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3dbcd6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d40cb800 con 0x7fc3d40cf160 2023-10-30T14:22:55.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3dbcd6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d415dc30 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3dbcd6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d40cb330 con 0x7fc3d41538a0 2023-10-30T14:22:55.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3d9271640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3d40ca0e0 0x7fc3d415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:55.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3d9271640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3d40ca0e0 0x7fc3d415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49742/0 (socket says 172.21.15.46:49742) 2023-10-30T14:22:55.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3d9271640 1 -- 172.21.15.46:0/2275984980 learned_addr learned my addr 172.21.15.46:0/2275984980 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:55.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3d9a72640 1 --2- 172.21.15.46:0/2275984980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d41538a0 0x7fc3d415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:55.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3d9271640 1 -- 172.21.15.46:0/2275984980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d41538a0 msgr2=0x7fc3d415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:55.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3d9271640 1 --2- 172.21.15.46:0/2275984980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d41538a0 0x7fc3d415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:55.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3d9271640 1 -- 172.21.15.46:0/2275984980 >> v1:172.21.15.46:6789/0 conn(0x7fc3d40cf160 legacy=0x7fc3d40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:55.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.392+0000 7fc3d9271640 1 -- 172.21.15.46:0/2275984980 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d40cad70 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3d9271640 1 --2- 172.21.15.46:0/2275984980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d40ca0e0 0x7fc3d415a090 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009870 tx=0x7fc3c401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e8dc6286197b92 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:55.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3d8a70640 1 -- 172.21.15.46:0/2275984980 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c40a8a30 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3d8a70640 1 -- 172.21.15.46:0/2275984980 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c40a8b90 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3d8a70640 1 -- 172.21.15.46:0/2275984980 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c40b0440 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/2275984980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d40ca0e0 msgr2=0x7fc3d415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:55.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 --2- 172.21.15.46:0/2275984980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d40ca0e0 0x7fc3d415a090 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009870 tx=0x7fc3c401ed30 comp rx=0 tx=0).stop 2023-10-30T14:22:55.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/2275984980 shutdown_connections 2023-10-30T14:22:55.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 --2- 172.21.15.46:0/2275984980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d40ca0e0 0x7fc3d415a090 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:55.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 --2- 172.21.15.46:0/2275984980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d41538a0 0x7fc3d415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:55.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/2275984980 >> 172.21.15.46:0/2275984980 conn(0x7fc3d406bfe0 msgr2=0x7fc3d406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:55.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/2275984980 shutdown_connections 2023-10-30T14:22:55.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/2275984980 wait complete. 2023-10-30T14:22:55.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 Processor -- start 2023-10-30T14:22:55.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 -- start start 2023-10-30T14:22:55.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d40ca0e0 0x7fc3d4147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:55.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d40cf160 0x7fc3d415c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:55.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d41538a0 0x7fc3d415ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:55.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d41f0c50 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d41436f0 con 0x7fc3d41538a0 2023-10-30T14:22:55.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.396+0000 7fc3dbcd6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d415acf0 con 0x7fc3d40cf160 2023-10-30T14:22:55.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3d9a72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d40ca0e0 0x7fc3d4147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:55.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3d9a72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d40ca0e0 0x7fc3d4147e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49746/0 (socket says 172.21.15.46:49746) 2023-10-30T14:22:55.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3d9a72640 1 -- 172.21.15.46:0/1840694765 learned_addr learned my addr 172.21.15.46:0/1840694765 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:55.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3da273640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d41538a0 0x7fc3d415ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:55.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3d9271640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d40cf160 0x7fc3d415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:55.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3d9a72640 1 -- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d40cf160 msgr2=0x7fc3d415c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:55.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3d9a72640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d40cf160 0x7fc3d415c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:55.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3d9a72640 1 -- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d41538a0 msgr2=0x7fc3d415ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:55.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3d9a72640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d41538a0 0x7fc3d415ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:55.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3d9a72640 1 -- 172.21.15.46:0/1840694765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c40094d0 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3d9271640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d40cf160 0x7fc3d415c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:55.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3da273640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d41538a0 0x7fc3d415ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:22:55.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3d9a72640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d40ca0e0 0x7fc3d4147e90 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000d5f0 tx=0x7fc3d000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:55.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3ca7fc640 1 -- 172.21.15.46:0/1840694765 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d00125f0 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3ca7fc640 1 -- 172.21.15.46:0/1840694765 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d0012750 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/1840694765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d415afd0 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3ca7fc640 1 -- 172.21.15.46:0/1840694765 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0012ae0 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/1840694765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d415b590 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/1840694765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d40b2a50 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.400+0000 7fc3ca7fc640 1 -- 172.21.15.46:0/1840694765 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3d0012c40 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.404+0000 7fc3ca7fc640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3b80664a0 0x7fc3b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:55.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.404+0000 7fc3ca7fc640 1 -- 172.21.15.46:0/1840694765 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc3d0084ee0 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.404+0000 7fc3d9271640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3b80664a0 0x7fc3b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:55.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.404+0000 7fc3d9271640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3b80664a0 0x7fc3b8068960 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fc3c40b2000 tx=0x7fc3c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:55.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.408+0000 7fc3ca7fc640 1 -- 172.21.15.46:0/1840694765 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc3d0051780 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/1840694765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3d404e170 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3ca7fc640 1 -- 172.21.15.46:0/1840694765 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+5731 (secure 0 0 0) 0x7fc3d0054da0 con 0x7fc3d40ca0e0 2023-10-30T14:22:55.609 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:55.609 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:22:51.327203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:resolve","state_seq":27,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:55.610 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 247 2023-10-30T14:22:55.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3b80664a0 msgr2=0x7fc3b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:55.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3b80664a0 0x7fc3b8068960 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fc3c40b2000 tx=0x7fc3c4008040 comp rx=0 tx=0).stop 2023-10-30T14:22:55.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/1840694765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d40ca0e0 msgr2=0x7fc3d4147e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:55.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d40ca0e0 0x7fc3d4147e90 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000d5f0 tx=0x7fc3d000dab0 comp rx=0 tx=0).stop 2023-10-30T14:22:55.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/1840694765 shutdown_connections 2023-10-30T14:22:55.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3b80664a0 0x7fc3b8068960 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:55.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d41538a0 0x7fc3d415ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:55.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3d40cf160 0x7fc3d415c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:55.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 --2- 172.21.15.46:0/1840694765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3d40ca0e0 0x7fc3d4147e90 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:55.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/1840694765 >> 172.21.15.46:0/1840694765 conn(0x7fc3d406bfe0 msgr2=0x7fc3d40cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:55.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/1840694765 shutdown_connections 2023-10-30T14:22:55.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:55.604+0000 7fc3dbcd6640 1 -- 172.21.15.46:0/1840694765 wait complete. 2023-10-30T14:22:55.631 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:22:57.632 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:22:57.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa538fc5640 1 Processor -- start 2023-10-30T14:22:57.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa538fc5640 1 -- start start 2023-10-30T14:22:57.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa538fc5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5341539a0 0x7fa53415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:57.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa538fc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa534069e90 0x7fa53406a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:57.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa538fc5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53415a8c0 con 0x7fa534070a00 2023-10-30T14:22:57.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa538fc5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53406fbb0 con 0x7fa5341539a0 2023-10-30T14:22:57.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa538fc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa534153d80 con 0x7fa534069e90 2023-10-30T14:22:57.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa53259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5341539a0 0x7fa53415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:57.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa53259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5341539a0 0x7fa53415a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57526/0 (socket says 172.21.15.46:57526) 2023-10-30T14:22:57.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa53259c640 1 -- 172.21.15.46:0/1168270704 learned_addr learned my addr 172.21.15.46:0/1168270704 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:57.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa532d9d640 1 --2- 172.21.15.46:0/1168270704 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa534069e90 0x7fa53406a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:57.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa53259c640 1 -- 172.21.15.46:0/1168270704 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa534069e90 msgr2=0x7fa53406a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:22:57.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa53259c640 1 --2- 172.21.15.46:0/1168270704 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa534069e90 0x7fa53406a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:57.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa53259c640 1 -- 172.21.15.46:0/1168270704 >> v1:172.21.15.154:6789/0 conn(0x7fa534070a00 legacy=0x7fa534070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:57.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.772+0000 7fa53259c640 1 -- 172.21.15.46:0/1168270704 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5340c5c50 con 0x7fa5341539a0 2023-10-30T14:22:57.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.776+0000 7fa53259c640 1 --2- 172.21.15.46:0/1168270704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5341539a0 0x7fa53415a380 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fa524009870 tx=0x7fa52401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962c501915ce57cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:57.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.776+0000 7fa531d9b640 1 -- 172.21.15.46:0/1168270704 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5240ad640 con 0x7fa5341539a0 2023-10-30T14:22:57.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.776+0000 7fa531d9b640 1 -- 172.21.15.46:0/1168270704 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5240ad7a0 con 0x7fa5341539a0 2023-10-30T14:22:57.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.776+0000 7fa531d9b640 1 -- 172.21.15.46:0/1168270704 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5240adb30 con 0x7fa5341539a0 2023-10-30T14:22:57.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.776+0000 7fa538fc5640 1 -- 172.21.15.46:0/1168270704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5341539a0 msgr2=0x7fa53415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:57.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.776+0000 7fa538fc5640 1 --2- 172.21.15.46:0/1168270704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5341539a0 0x7fa53415a380 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fa524009870 tx=0x7fa52401c480 comp rx=0 tx=0).stop 2023-10-30T14:22:57.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.776+0000 7fa538fc5640 1 -- 172.21.15.46:0/1168270704 shutdown_connections 2023-10-30T14:22:57.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.776+0000 7fa538fc5640 1 --2- 172.21.15.46:0/1168270704 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa534069e90 0x7fa53406a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:57.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.776+0000 7fa538fc5640 1 --2- 172.21.15.46:0/1168270704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5341539a0 0x7fa53415a380 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:57.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.776+0000 7fa538fc5640 1 -- 172.21.15.46:0/1168270704 >> 172.21.15.46:0/1168270704 conn(0x7fa5340c7910 msgr2=0x7fa5340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:57.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.776+0000 7fa538fc5640 1 -- 172.21.15.46:0/1168270704 shutdown_connections 2023-10-30T14:22:57.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.776+0000 7fa538fc5640 1 -- 172.21.15.46:0/1168270704 wait complete. 2023-10-30T14:22:57.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa538fc5640 1 Processor -- start 2023-10-30T14:22:57.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa538fc5640 1 -- start start 2023-10-30T14:22:57.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa538fc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa534069e90 0x7fa53414c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:57.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa538fc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa534070a00 0x7fa5340ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:57.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa538fc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5341539a0 0x7fa53414cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa538fc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53414ff00 con 0x7fa5341539a0 2023-10-30T14:22:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa538fc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53414b210 con 0x7fa534069e90 2023-10-30T14:22:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa538fc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53414c600 con 0x7fa534070a00 2023-10-30T14:22:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa532d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5341539a0 0x7fa53414cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:57.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa532d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5341539a0 0x7fa53414cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57542/0 (socket says 172.21.15.46:57542) 2023-10-30T14:22:57.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa532d9d640 1 -- 172.21.15.46:0/372172578 learned_addr learned my addr 172.21.15.46:0/372172578 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:22:57.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa52bfff640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa534070a00 0x7fa5340ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:57.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa53259c640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa534069e90 0x7fa53414c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:57.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa532d9d640 1 -- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa534070a00 msgr2=0x7fa5340ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:57.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa532d9d640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa534070a00 0x7fa5340ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:57.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa532d9d640 1 -- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa534069e90 msgr2=0x7fa53414c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:57.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa532d9d640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa534069e90 0x7fa53414c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:57.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa532d9d640 1 -- 172.21.15.46:0/372172578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5240094d0 con 0x7fa5341539a0 2023-10-30T14:22:57.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa532d9d640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5341539a0 0x7fa53414cf90 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00efc0 tx=0x7fa52c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:57.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa52affd640 1 -- 172.21.15.46:0/372172578 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c009ba0 con 0x7fa5341539a0 2023-10-30T14:22:57.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa52affd640 1 -- 172.21.15.46:0/372172578 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52c00f040 con 0x7fa5341539a0 2023-10-30T14:22:57.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa52affd640 1 -- 172.21.15.46:0/372172578 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c0146c0 con 0x7fa5341539a0 2023-10-30T14:22:57.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa538fc5640 1 -- 172.21.15.46:0/372172578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5341500f0 con 0x7fa5341539a0 2023-10-30T14:22:57.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.780+0000 7fa538fc5640 1 -- 172.21.15.46:0/372172578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5340c0e70 con 0x7fa5341539a0 2023-10-30T14:22:57.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.784+0000 7fa538fc5640 1 -- 172.21.15.46:0/372172578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53414b380 con 0x7fa5341539a0 2023-10-30T14:22:57.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.784+0000 7fa52affd640 1 -- 172.21.15.46:0/372172578 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa52c014820 con 0x7fa5341539a0 2023-10-30T14:22:57.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.784+0000 7fa52affd640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5140664f0 0x7fa5140689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:22:57.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.784+0000 7fa52affd640 1 -- 172.21.15.46:0/372172578 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa52c0852f0 con 0x7fa5341539a0 2023-10-30T14:22:57.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.784+0000 7fa53259c640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5140664f0 0x7fa5140689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:22:57.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.784+0000 7fa53259c640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5140664f0 0x7fa5140689b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa5240af000 tx=0x7fa524008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:22:57.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:57.788+0000 7fa52affd640 1 -- 172.21.15.46:0/372172578 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa53414b380 con 0x7fa5341539a0 2023-10-30T14:22:58.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.000+0000 7fa538fc5640 1 -- 172.21.15.46:0/372172578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa53414b380 con 0x7fa5341539a0 2023-10-30T14:22:58.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.000+0000 7fa52affd640 1 -- 172.21.15.46:0/372172578 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fa53414b380 con 0x7fa5341539a0 2023-10-30T14:22:58.008 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:22:58.008 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":248,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:22:56.905227+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:22:58.008 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 248 2023-10-30T14:22:58.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.004+0000 7fa538fc5640 1 -- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5140664f0 msgr2=0x7fa5140689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:58.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.004+0000 7fa538fc5640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5140664f0 0x7fa5140689b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa5240af000 tx=0x7fa524008040 comp rx=0 tx=0).stop 2023-10-30T14:22:58.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.004+0000 7fa538fc5640 1 -- 172.21.15.46:0/372172578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5341539a0 msgr2=0x7fa53414cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:22:58.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.004+0000 7fa538fc5640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5341539a0 0x7fa53414cf90 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00efc0 tx=0x7fa52c00c490 comp rx=0 tx=0).stop 2023-10-30T14:22:58.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.004+0000 7fa538fc5640 1 -- 172.21.15.46:0/372172578 shutdown_connections 2023-10-30T14:22:58.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.004+0000 7fa538fc5640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa5140664f0 0x7fa5140689b0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:58.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.004+0000 7fa538fc5640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5341539a0 0x7fa53414cf90 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:58.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.004+0000 7fa538fc5640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa534070a00 0x7fa5340ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:58.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.004+0000 7fa538fc5640 1 --2- 172.21.15.46:0/372172578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa534069e90 0x7fa53414c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:22:58.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.004+0000 7fa538fc5640 1 -- 172.21.15.46:0/372172578 >> 172.21.15.46:0/372172578 conn(0x7fa5340c7910 msgr2=0x7fa534151890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:22:58.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.004+0000 7fa538fc5640 1 -- 172.21.15.46:0/372172578 shutdown_connections 2023-10-30T14:22:58.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:22:58.004+0000 7fa538fc5640 1 -- 172.21.15.46:0/372172578 wait complete. 2023-10-30T14:22:58.029 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:23:00.030 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:00.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910f0d6640 1 Processor -- start 2023-10-30T14:23:00.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910f0d6640 1 -- start start 2023-10-30T14:23:00.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910f0d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91081538a0 0x7f910815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:00.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910f0d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f91080cf160 0x7f91080cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:00.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910f0d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91080cb800 con 0x7f91080ca0e0 2023-10-30T14:23:00.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910f0d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f910815dc30 con 0x7f91080cf160 2023-10-30T14:23:00.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910f0d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91080cb330 con 0x7f91081538a0 2023-10-30T14:23:00.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910d673640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f91080cf160 0x7f91080cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:00.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910ce72640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91081538a0 0x7f910815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:00.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f9107fff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f91080ca0e0 0x7f910815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:52348/0 (socket says 172.21.15.46:52348) 2023-10-30T14:23:00.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f9107fff640 1 -- 172.21.15.46:0/3333245380 learned_addr learned my addr 172.21.15.46:0/3333245380 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:23:00.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910d673640 1 --2- 172.21.15.46:0/3333245380 >> v2:172.21.15.154:3300/0 conn(0x7f91080cf160 0x7f91080cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:42664/0 (socket says 172.21.15.46:42664) 2023-10-30T14:23:00.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910ce72640 1 -- 172.21.15.46:0/3333245380 >> v2:172.21.15.154:3300/0 conn(0x7f91080cf160 msgr2=0x7f91080cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:00.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910ce72640 1 --2- 172.21.15.46:0/3333245380 >> v2:172.21.15.154:3300/0 conn(0x7f91080cf160 0x7f91080cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:00.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910ce72640 1 -- 172.21.15.46:0/3333245380 >> v1:172.21.15.154:6789/0 conn(0x7f91080ca0e0 legacy=0x7f910815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:00.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910ce72640 1 -- 172.21.15.46:0/3333245380 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91080cad70 con 0x7f91081538a0 2023-10-30T14:23:00.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.172+0000 7f910d673640 1 --2- 172.21.15.46:0/3333245380 >> v2:172.21.15.154:3300/0 conn(0x7f91080cf160 0x7f91080cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:23:00.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.176+0000 7f910ce72640 1 --2- 172.21.15.46:0/3333245380 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91081538a0 0x7f910815d6f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0097c0 tx=0x7f90fc01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b86697a7cd4598e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:00.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.176+0000 7f91077fe640 1 -- 172.21.15.46:0/3333245380 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc0a8970 con 0x7f91081538a0 2023-10-30T14:23:00.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.176+0000 7f91077fe640 1 -- 172.21.15.46:0/3333245380 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90fc0a8ad0 con 0x7f91081538a0 2023-10-30T14:23:00.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.176+0000 7f91077fe640 1 -- 172.21.15.46:0/3333245380 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc0a8e60 con 0x7f91081538a0 2023-10-30T14:23:00.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.176+0000 7f910f0d6640 1 -- 172.21.15.46:0/3333245380 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91081538a0 msgr2=0x7f910815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:00.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.176+0000 7f910f0d6640 1 --2- 172.21.15.46:0/3333245380 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91081538a0 0x7f910815d6f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0097c0 tx=0x7f90fc01ec70 comp rx=0 tx=0).stop 2023-10-30T14:23:00.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.176+0000 7f910f0d6640 1 -- 172.21.15.46:0/3333245380 shutdown_connections 2023-10-30T14:23:00.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.176+0000 7f910f0d6640 1 --2- 172.21.15.46:0/3333245380 >> v2:172.21.15.154:3300/0 conn(0x7f91080cf160 0x7f91080cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.176+0000 7f910f0d6640 1 --2- 172.21.15.46:0/3333245380 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91081538a0 0x7f910815d6f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.176+0000 7f910f0d6640 1 -- 172.21.15.46:0/3333245380 >> 172.21.15.46:0/3333245380 conn(0x7f910806bfe0 msgr2=0x7f910806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f910f0d6640 1 -- 172.21.15.46:0/3333245380 shutdown_connections 2023-10-30T14:23:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f910f0d6640 1 -- 172.21.15.46:0/3333245380 wait complete. 2023-10-30T14:23:00.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f910f0d6640 1 Processor -- start 2023-10-30T14:23:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f910f0d6640 1 -- start start 2023-10-30T14:23:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f910f0d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91080ca0e0 0x7f9108158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f910f0d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080cf160 0x7f91080ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f910f0d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91081538a0 0x7f9108143c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f910f0d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91081f00f0 con 0x7f91080cf160 2023-10-30T14:23:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f910f0d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9108159380 con 0x7f91081538a0 2023-10-30T14:23:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f910f0d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91081ef520 con 0x7f91080ca0e0 2023-10-30T14:23:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f9107fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080cf160 0x7f91080ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f9107fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080cf160 0x7f91080ce390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57558/0 (socket says 172.21.15.46:57558) 2023-10-30T14:23:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f9107fff640 1 -- 172.21.15.46:0/44600463 learned_addr learned my addr 172.21.15.46:0/44600463 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:00.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f910d673640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91081538a0 0x7f9108143c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:00.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f910ce72640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91080ca0e0 0x7f9108158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:00.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f9107fff640 1 -- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91080ca0e0 msgr2=0x7f9108158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:00.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f9107fff640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91080ca0e0 0x7f9108158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:00.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f9107fff640 1 -- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91081538a0 msgr2=0x7f9108143c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f9107fff640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91081538a0 0x7f9108143c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f9107fff640 1 -- 172.21.15.46:0/44600463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9100009a70 con 0x7f91080cf160 2023-10-30T14:23:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.180+0000 7f9107fff640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080cf160 0x7f91080ce390 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f90f400c420 tx=0x7f90f400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.184+0000 7f91057fa640 1 -- 172.21.15.46:0/44600463 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4005cf0 con 0x7f91080cf160 2023-10-30T14:23:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.184+0000 7f910f0d6640 1 -- 172.21.15.46:0/44600463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc009420 con 0x7f91080cf160 2023-10-30T14:23:00.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.184+0000 7f91057fa640 1 -- 172.21.15.46:0/44600463 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f4005e50 con 0x7f91080cf160 2023-10-30T14:23:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.184+0000 7f91057fa640 1 -- 172.21.15.46:0/44600463 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f40072f0 con 0x7f91080cf160 2023-10-30T14:23:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.184+0000 7f910f0d6640 1 -- 172.21.15.46:0/44600463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91081efaa0 con 0x7f91080cf160 2023-10-30T14:23:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.184+0000 7f910f0d6640 1 -- 172.21.15.46:0/44600463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91080b2a50 con 0x7f91080cf160 2023-10-30T14:23:00.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.184+0000 7f91057fa640 1 -- 172.21.15.46:0/44600463 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f90f4007450 con 0x7f91080cf160 2023-10-30T14:23:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.184+0000 7f91057fa640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90ec0664a0 0x7f90ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.184+0000 7f91057fa640 1 -- 172.21.15.46:0/44600463 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f90f4083f60 con 0x7f91080cf160 2023-10-30T14:23:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.184+0000 7f910ce72640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90ec0664a0 0x7f90ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:00.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.188+0000 7f910ce72640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90ec0664a0 0x7f90ec068960 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0b2000 tx=0x7f90fc0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:00.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.188+0000 7f91057fa640 1 -- 172.21.15.46:0/44600463 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f90f40508f0 con 0x7f91080cf160 2023-10-30T14:23:00.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.380+0000 7f910f0d6640 1 -- 172.21.15.46:0/44600463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f910804e170 con 0x7f91080cf160 2023-10-30T14:23:00.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.380+0000 7f91057fa640 1 -- 172.21.15.46:0/44600463 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f90f4017020 con 0x7f91080cf160 2023-10-30T14:23:00.386 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:00.386 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":248,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:22:56.905227+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:00.387 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 248 2023-10-30T14:23:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.384+0000 7f910f0d6640 1 -- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90ec0664a0 msgr2=0x7f90ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:00.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.384+0000 7f910f0d6640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90ec0664a0 0x7f90ec068960 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0b2000 tx=0x7f90fc0aa070 comp rx=0 tx=0).stop 2023-10-30T14:23:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.384+0000 7f910f0d6640 1 -- 172.21.15.46:0/44600463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080cf160 msgr2=0x7f91080ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.384+0000 7f910f0d6640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080cf160 0x7f91080ce390 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f90f400c420 tx=0x7f90f400c8e0 comp rx=0 tx=0).stop 2023-10-30T14:23:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.384+0000 7f910f0d6640 1 -- 172.21.15.46:0/44600463 shutdown_connections 2023-10-30T14:23:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.384+0000 7f910f0d6640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f90ec0664a0 0x7f90ec068960 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.384+0000 7f910f0d6640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91081538a0 0x7f9108143c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:00.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.384+0000 7f910f0d6640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080cf160 0x7f91080ce390 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:00.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.384+0000 7f910f0d6640 1 --2- 172.21.15.46:0/44600463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91080ca0e0 0x7f9108158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:00.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.384+0000 7f910f0d6640 1 -- 172.21.15.46:0/44600463 >> 172.21.15.46:0/44600463 conn(0x7f910806bfe0 msgr2=0x7f9108152730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:00.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.384+0000 7f910f0d6640 1 -- 172.21.15.46:0/44600463 shutdown_connections 2023-10-30T14:23:00.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:00.384+0000 7f910f0d6640 1 -- 172.21.15.46:0/44600463 wait complete. 2023-10-30T14:23:00.407 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:23:02.409 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:02.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb45aa640 1 Processor -- start 2023-10-30T14:23:02.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb45aa640 1 -- start start 2023-10-30T14:23:02.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb45aa640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7fac151800 0x7f7fac06e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:02.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb45aa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac06e090 con 0x7f7fac0c1ee0 2023-10-30T14:23:02.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb45aa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac0c8a50 con 0x7f7fac0c8260 2023-10-30T14:23:02.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb45aa640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fac06dc70 con 0x7f7fac151800 2023-10-30T14:23:02.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1b45640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7fac0c1ee0 0x7f7fac0c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47270/0 (socket says 172.21.15.46:47270) 2023-10-30T14:23:02.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1b45640 1 -- 172.21.15.46:0/4293958714 learned_addr learned my addr 172.21.15.46:0/4293958714 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:23:02.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb2346640 1 --2- 172.21.15.46:0/4293958714 >> v2:172.21.15.154:3300/0 conn(0x7f7fac151800 0x7f7fac06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:02.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328266735 0 0) 0x7f7fac06e090 con 0x7f7fac0c1ee0 2023-10-30T14:23:02.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f98003530 con 0x7f7fac0c1ee0 2023-10-30T14:23:02.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1196054393 0 0) 0x7f7f98003530 con 0x7f7fac0c1ee0 2023-10-30T14:23:02.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 >> v2:172.21.15.154:3300/0 conn(0x7f7fac151800 msgr2=0x7f7fac06e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:02.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 --2- 172.21.15.46:0/4293958714 >> v2:172.21.15.154:3300/0 conn(0x7f7fac151800 0x7f7fac06e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:02.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 >> v1:172.21.15.154:6789/0 conn(0x7f7fac0c8260 legacy=0x7f7fac0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:02.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac0c3b10 con 0x7f7fac0c1ee0 2023-10-30T14:23:02.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f7f9c001680 con 0x7f7fac0c1ee0 2023-10-30T14:23:02.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 >> v1:172.21.15.46:6789/0 conn(0x7f7fac0c1ee0 legacy=0x7f7fac0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:02.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 --2- 172.21.15.46:0/4293958714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f98003980 0x7f7f98003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:02.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 --2- 172.21.15.46:0/4293958714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f980047d0 0x7f7f98004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 --2- 172.21.15.46:0/4293958714 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f98005190 0x7f7f9800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac06dc70 con 0x7f7f98003980 2023-10-30T14:23:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9800c030 con 0x7f7f980047d0 2023-10-30T14:23:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9800c1f0 con 0x7f7f98005190 2023-10-30T14:23:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.548+0000 7f7fb2346640 1 --2- 172.21.15.46:0/4293958714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f98003980 0x7f7f98003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb1b45640 1 --2- 172.21.15.46:0/4293958714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f980047d0 0x7f7f98004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb2b47640 1 --2- 172.21.15.46:0/4293958714 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f98005190 0x7f7f9800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb2346640 1 -- 172.21.15.46:0/4293958714 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f98005190 msgr2=0x7f7f9800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb2346640 1 --2- 172.21.15.46:0/4293958714 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f98005190 0x7f7f9800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb2346640 1 -- 172.21.15.46:0/4293958714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f980047d0 msgr2=0x7f7f98004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb2346640 1 --2- 172.21.15.46:0/4293958714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f980047d0 0x7f7f98004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:02.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb2346640 1 -- 172.21.15.46:0/4293958714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f98003530 con 0x7f7f98003980 2023-10-30T14:23:02.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb2b47640 1 -- 172.21.15.46:0/4293958714 reap_dead start 2023-10-30T14:23:02.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb2346640 1 --2- 172.21.15.46:0/4293958714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f98003980 0x7f7f98003da0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009ae0 tx=0x7f7fa801ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824774cfb97043a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:02.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa80a5520 con 0x7f7f98003980 2023-10-30T14:23:02.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa80a5680 con 0x7f7f98003980 2023-10-30T14:23:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb1344640 1 -- 172.21.15.46:0/4293958714 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa80a5a30 con 0x7f7f98003980 2023-10-30T14:23:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb45aa640 1 -- 172.21.15.46:0/4293958714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f98003980 msgr2=0x7f7f98003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.552+0000 7f7fb45aa640 1 --2- 172.21.15.46:0/4293958714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f98003980 0x7f7f98003da0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009ae0 tx=0x7f7fa801ca40 comp rx=0 tx=0).stop 2023-10-30T14:23:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 -- 172.21.15.46:0/4293958714 shutdown_connections 2023-10-30T14:23:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 --2- 172.21.15.46:0/4293958714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f98003980 0x7f7f98003da0 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 -- 172.21.15.46:0/4293958714 >> 172.21.15.46:0/4293958714 conn(0x7f7fac071ea0 msgr2=0x7f7fac065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 -- 172.21.15.46:0/4293958714 shutdown_connections 2023-10-30T14:23:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 -- 172.21.15.46:0/4293958714 wait complete. 2023-10-30T14:23:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 Processor -- start 2023-10-30T14:23:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 -- start start 2023-10-30T14:23:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f98003980 0x7f7fac143190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fac1f5810 0x7f7fac1e7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fac06b9f0 0x7f7fac06be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac06b560 con 0x7f7fac06b9f0 2023-10-30T14:23:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1f3260 con 0x7f7f98003980 2023-10-30T14:23:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb45aa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1436d0 con 0x7f7fac1f5810 2023-10-30T14:23:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb2b47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fac06b9f0 0x7f7fac06be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb2346640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f98003980 0x7f7fac143190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb2b47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fac06b9f0 0x7f7fac06be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57576/0 (socket says 172.21.15.46:57576) 2023-10-30T14:23:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb2b47640 1 -- 172.21.15.46:0/681449320 learned_addr learned my addr 172.21.15.46:0/681449320 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:02.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb1b45640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fac1f5810 0x7f7fac1e7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:02.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb2b47640 1 -- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fac1f5810 msgr2=0x7f7fac1e7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:02.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb2b47640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fac1f5810 0x7f7fac1e7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:02.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb2b47640 1 -- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f98003980 msgr2=0x7f7fac143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:02.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb2b47640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f98003980 0x7f7fac143190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:02.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb2b47640 1 -- 172.21.15.46:0/681449320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac06c3b0 con 0x7f7fac06b9f0 2023-10-30T14:23:02.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb1b45640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fac1f5810 0x7f7fac1e7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:02.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.556+0000 7f7fb2b47640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fac06b9f0 0x7f7fac06be70 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0700c0 tx=0x7f7fa400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:02.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.560+0000 7f7fa2ffd640 1 -- 172.21.15.46:0/681449320 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4003980 con 0x7f7fac06b9f0 2023-10-30T14:23:02.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.560+0000 7f7fa2ffd640 1 -- 172.21.15.46:0/681449320 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa4003ae0 con 0x7f7fac06b9f0 2023-10-30T14:23:02.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.560+0000 7f7fa2ffd640 1 -- 172.21.15.46:0/681449320 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4003e90 con 0x7f7fac06b9f0 2023-10-30T14:23:02.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.560+0000 7f7fb45aa640 1 -- 172.21.15.46:0/681449320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac1f26e0 con 0x7f7fac06b9f0 2023-10-30T14:23:02.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.560+0000 7f7fb45aa640 1 -- 172.21.15.46:0/681449320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac1f2ca0 con 0x7f7fac06b9f0 2023-10-30T14:23:02.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.560+0000 7f7fb45aa640 1 -- 172.21.15.46:0/681449320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fac04e170 con 0x7f7fac06b9f0 2023-10-30T14:23:02.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.560+0000 7f7fa2ffd640 1 -- 172.21.15.46:0/681449320 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7fa40050d0 con 0x7f7fac06b9f0 2023-10-30T14:23:02.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.560+0000 7f7fa2ffd640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f900664a0 0x7f7f90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:02.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.560+0000 7f7fa2ffd640 1 -- 172.21.15.46:0/681449320 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f7fa4084610 con 0x7f7fac06b9f0 2023-10-30T14:23:02.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.564+0000 7f7fb2346640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f900664a0 0x7f7f90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:02.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.564+0000 7f7fb2346640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f900664a0 0x7f7f90068960 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80af000 tx=0x7f7fa80046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:02.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.568+0000 7f7fa2ffd640 1 -- 172.21.15.46:0/681449320 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7fa4050f40 con 0x7f7fac06b9f0 2023-10-30T14:23:02.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 -- 172.21.15.46:0/681449320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fac06e250 con 0x7f7fac06b9f0 2023-10-30T14:23:02.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fa2ffd640 1 -- 172.21.15.46:0/681449320 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f7fa4054560 con 0x7f7fac06b9f0 2023-10-30T14:23:02.765 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:02.766 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":248,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:22:56.905227+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:02.766 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 248 2023-10-30T14:23:02.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 -- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f900664a0 msgr2=0x7f7f90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:02.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f900664a0 0x7f7f90068960 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80af000 tx=0x7f7fa80046d0 comp rx=0 tx=0).stop 2023-10-30T14:23:02.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 -- 172.21.15.46:0/681449320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fac06b9f0 msgr2=0x7f7fac06be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:02.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fac06b9f0 0x7f7fac06be70 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0700c0 tx=0x7f7fa400ec10 comp rx=0 tx=0).stop 2023-10-30T14:23:02.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 -- 172.21.15.46:0/681449320 shutdown_connections 2023-10-30T14:23:02.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f900664a0 0x7f7f90068960 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:02.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7fac06b9f0 0x7f7fac06be70 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:02.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fac1f5810 0x7f7fac1e7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:02.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 --2- 172.21.15.46:0/681449320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f98003980 0x7f7fac143190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:02.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 -- 172.21.15.46:0/681449320 >> 172.21.15.46:0/681449320 conn(0x7f7fac071ea0 msgr2=0x7f7fac14fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:02.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 -- 172.21.15.46:0/681449320 shutdown_connections 2023-10-30T14:23:02.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:02.760+0000 7f7fb45aa640 1 -- 172.21.15.46:0/681449320 wait complete. 2023-10-30T14:23:02.787 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:23:04.788 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:04.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.928+0000 7f7434491640 1 Processor -- start 2023-10-30T14:23:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.928+0000 7f7434491640 1 -- start start 2023-10-30T14:23:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f7434491640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f742c1538a0 0x7f742c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f7434491640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f742c0cf160 0x7f742c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f7434491640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742c0cb800 con 0x7f742c0ca0e0 2023-10-30T14:23:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f7434491640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f742c15dc30 con 0x7f742c1538a0 2023-10-30T14:23:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f7434491640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f742c0cb330 con 0x7f742c0cf160 2023-10-30T14:23:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f743222d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f742c1538a0 0x7f742c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:04.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f743222d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f742c1538a0 0x7f742c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57582/0 (socket says 172.21.15.46:57582) 2023-10-30T14:23:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f743222d640 1 -- 172.21.15.46:0/635524756 learned_addr learned my addr 172.21.15.46:0/635524756 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f7432a2e640 1 --2- 172.21.15.46:0/635524756 >> v2:172.21.15.154:3300/0 conn(0x7f742c0cf160 0x7f742c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f743222d640 1 -- 172.21.15.46:0/635524756 >> v2:172.21.15.154:3300/0 conn(0x7f742c0cf160 msgr2=0x7f742c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f743222d640 1 --2- 172.21.15.46:0/635524756 >> v2:172.21.15.154:3300/0 conn(0x7f742c0cf160 0x7f742c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f743222d640 1 -- 172.21.15.46:0/635524756 >> v1:172.21.15.154:6789/0 conn(0x7f742c0ca0e0 legacy=0x7f742c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f743222d640 1 -- 172.21.15.46:0/635524756 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c0cad70 con 0x7f742c1538a0 2023-10-30T14:23:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.932+0000 7f743222d640 1 --2- 172.21.15.46:0/635524756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f742c1538a0 0x7f742c15d6f0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f742c0b5330 tx=0x7f742801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7e3da3ec0c1ea1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:04.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f743122b640 1 -- 172.21.15.46:0/635524756 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74280a89b0 con 0x7f742c1538a0 2023-10-30T14:23:04.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f743122b640 1 -- 172.21.15.46:0/635524756 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74280a8b10 con 0x7f742c1538a0 2023-10-30T14:23:04.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f743122b640 1 -- 172.21.15.46:0/635524756 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74280a8ea0 con 0x7f742c1538a0 2023-10-30T14:23:04.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 -- 172.21.15.46:0/635524756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f742c1538a0 msgr2=0x7f742c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 --2- 172.21.15.46:0/635524756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f742c1538a0 0x7f742c15d6f0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f742c0b5330 tx=0x7f742801ecb0 comp rx=0 tx=0).stop 2023-10-30T14:23:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 -- 172.21.15.46:0/635524756 shutdown_connections 2023-10-30T14:23:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 --2- 172.21.15.46:0/635524756 >> v2:172.21.15.154:3300/0 conn(0x7f742c0cf160 0x7f742c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 --2- 172.21.15.46:0/635524756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f742c1538a0 0x7f742c15d6f0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 -- 172.21.15.46:0/635524756 >> 172.21.15.46:0/635524756 conn(0x7f742c06bfe0 msgr2=0x7f742c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 -- 172.21.15.46:0/635524756 shutdown_connections 2023-10-30T14:23:04.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 -- 172.21.15.46:0/635524756 wait complete. 2023-10-30T14:23:04.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 Processor -- start 2023-10-30T14:23:04.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 -- start start 2023-10-30T14:23:04.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c0ca0e0 0x7f742c0c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:04.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f742c0cf160 0x7f742c157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:04.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c1538a0 0x7f742c1e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:04.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c1ef4c0 con 0x7f742c0cf160 2023-10-30T14:23:04.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c0c9180 con 0x7f742c0ca0e0 2023-10-30T14:23:04.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.936+0000 7f7434491640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742c1ef730 con 0x7f742c1538a0 2023-10-30T14:23:04.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7431a2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f742c0cf160 0x7f742c157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:04.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7431a2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f742c0cf160 0x7f742c157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57588/0 (socket says 172.21.15.46:57588) 2023-10-30T14:23:04.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7431a2c640 1 -- 172.21.15.46:0/2367432354 learned_addr learned my addr 172.21.15.46:0/2367432354 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:04.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f743222d640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c0ca0e0 0x7f742c0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:04.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7432a2e640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c1538a0 0x7f742c1e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:04.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7431a2c640 1 -- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c1538a0 msgr2=0x7f742c1e5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:04.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7431a2c640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c1538a0 0x7f742c1e5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:04.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7431a2c640 1 -- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c0ca0e0 msgr2=0x7f742c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:04.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7431a2c640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c0ca0e0 0x7f742c0c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:04.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7431a2c640 1 -- 172.21.15.46:0/2367432354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74280094d0 con 0x7f742c0cf160 2023-10-30T14:23:04.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7431a2c640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f742c0cf160 0x7f742c157560 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f741c00b410 tx=0x7f741c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7422ffd640 1 -- 172.21.15.46:0/2367432354 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c004020 con 0x7f742c0cf160 2023-10-30T14:23:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7422ffd640 1 -- 172.21.15.46:0/2367432354 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c004180 con 0x7f742c0cf160 2023-10-30T14:23:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7434491640 1 -- 172.21.15.46:0/2367432354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742c1efa10 con 0x7f742c0cf160 2023-10-30T14:23:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7422ffd640 1 -- 172.21.15.46:0/2367432354 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c011630 con 0x7f742c0cf160 2023-10-30T14:23:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7434491640 1 -- 172.21.15.46:0/2367432354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742c0c9460 con 0x7f742c0cf160 2023-10-30T14:23:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.940+0000 7f7434491640 1 -- 172.21.15.46:0/2367432354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742c0b2a50 con 0x7f742c0cf160 2023-10-30T14:23:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.944+0000 7f7422ffd640 1 -- 172.21.15.46:0/2367432354 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f741c011790 con 0x7f742c0cf160 2023-10-30T14:23:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.944+0000 7f7422ffd640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f74100664a0 0x7f7410068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:04.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.944+0000 7f7422ffd640 1 -- 172.21.15.46:0/2367432354 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f741c0846b0 con 0x7f742c0cf160 2023-10-30T14:23:04.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.944+0000 7f743222d640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f74100664a0 0x7f7410068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:04.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.944+0000 7f743222d640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f74100664a0 0x7f7410068960 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f74280b2000 tx=0x7f7428009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:04.948+0000 7f7422ffd640 1 -- 172.21.15.46:0/2367432354 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f741c051000 con 0x7f742c0cf160 2023-10-30T14:23:05.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 -- 172.21.15.46:0/2367432354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f742c1eff30 con 0x7f742c0cf160 2023-10-30T14:23:05.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7422ffd640 1 -- 172.21.15.46:0/2367432354 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f742c1eff30 con 0x7f742c0cf160 2023-10-30T14:23:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:05.144 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":249,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:02.881530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:05.145 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 249 2023-10-30T14:23:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 -- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f74100664a0 msgr2=0x7f7410068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f74100664a0 0x7f7410068960 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f74280b2000 tx=0x7f7428009040 comp rx=0 tx=0).stop 2023-10-30T14:23:05.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 -- 172.21.15.46:0/2367432354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f742c0cf160 msgr2=0x7f742c157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:05.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f742c0cf160 0x7f742c157560 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f741c00b410 tx=0x7f741c00b8d0 comp rx=0 tx=0).stop 2023-10-30T14:23:05.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 -- 172.21.15.46:0/2367432354 shutdown_connections 2023-10-30T14:23:05.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f74100664a0 0x7f7410068960 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:05.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c1538a0 0x7f742c1e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:05.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f742c0cf160 0x7f742c157560 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:05.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 --2- 172.21.15.46:0/2367432354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c0ca0e0 0x7f742c0c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:05.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 -- 172.21.15.46:0/2367432354 >> 172.21.15.46:0/2367432354 conn(0x7f742c06bfe0 msgr2=0x7f742c1519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:05.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 -- 172.21.15.46:0/2367432354 shutdown_connections 2023-10-30T14:23:05.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:05.140+0000 7f7434491640 1 -- 172.21.15.46:0/2367432354 wait complete. 2023-10-30T14:23:05.166 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 249, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 249, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9594": { "addr": "172.21.15.46:6837/470164358", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 470164358, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 470164358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9594, "incarnation": 239, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 29 }, "gid_9735": { "addr": "172.21.15.46:6836/3574404756", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3574404756, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3574404756, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9735, "incarnation": 246, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 88, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:23:02.881530+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9735, "mds_1": 9594 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:23:07.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.304+0000 7f331e51d640 1 Processor -- start 2023-10-30T14:23:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.304+0000 7f331e51d640 1 -- start start 2023-10-30T14:23:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.304+0000 7f331e51d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f331806f7a0 0x7f331806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.304+0000 7f331e51d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f33180c6bd0 0x7f3318069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.304+0000 7f331e51d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331806a3d0 con 0x7f3318071110 2023-10-30T14:23:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.304+0000 7f331e51d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33180700c0 con 0x7f331806f7a0 2023-10-30T14:23:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.304+0000 7f331e51d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3318066cf0 con 0x7f33180c6bd0 2023-10-30T14:23:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.304+0000 7f331caba640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3318071110 0x7f3318158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38832/0 (socket says 172.21.15.46:38832) 2023-10-30T14:23:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.304+0000 7f331caba640 1 -- 172.21.15.46:0/4039999925 learned_addr learned my addr 172.21.15.46:0/4039999925 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:23:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.304+0000 7f3317fff640 1 --2- 172.21.15.46:0/4039999925 >> v2:172.21.15.46:3300/0 conn(0x7f331806f7a0 0x7f331806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.304+0000 7f33177fe640 1 --2- 172.21.15.46:0/4039999925 >> v2:172.21.15.154:3300/0 conn(0x7f33180c6bd0 0x7f3318069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.308+0000 7f3316ffd640 1 -- 172.21.15.46:0/4039999925 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266693408 0 0) 0x7f331806a3d0 con 0x7f3318071110 2023-10-30T14:23:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.308+0000 7f3316ffd640 1 -- 172.21.15.46:0/4039999925 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3300003530 con 0x7f3318071110 2023-10-30T14:23:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.308+0000 7f3317fff640 1 -- 172.21.15.46:0/4039999925 >> v2:172.21.15.154:3300/0 conn(0x7f33180c6bd0 msgr2=0x7f3318069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.308+0000 7f3317fff640 1 --2- 172.21.15.46:0/4039999925 >> v2:172.21.15.154:3300/0 conn(0x7f33180c6bd0 0x7f3318069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.308+0000 7f3317fff640 1 -- 172.21.15.46:0/4039999925 >> v1:172.21.15.46:6789/0 conn(0x7f3318071110 legacy=0x7f3318158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.308+0000 7f3317fff640 1 -- 172.21.15.46:0/4039999925 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33180c70d0 con 0x7f331806f7a0 2023-10-30T14:23:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.308+0000 7f33177fe640 1 --2- 172.21.15.46:0/4039999925 >> v2:172.21.15.154:3300/0 conn(0x7f33180c6bd0 0x7f3318069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.308+0000 7f3317fff640 1 --2- 172.21.15.46:0/4039999925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331806f7a0 0x7f331806fb80 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f331806f1c0 tx=0x7f330801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d389d316ae2251 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.308+0000 7f3316ffd640 1 -- 172.21.15.46:0/4039999925 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33080a8950 con 0x7f331806f7a0 2023-10-30T14:23:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.308+0000 7f3316ffd640 1 -- 172.21.15.46:0/4039999925 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33080a8ab0 con 0x7f331806f7a0 2023-10-30T14:23:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.308+0000 7f3316ffd640 1 -- 172.21.15.46:0/4039999925 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33080a8e40 con 0x7f331806f7a0 2023-10-30T14:23:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 -- 172.21.15.46:0/4039999925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331806f7a0 msgr2=0x7f331806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 --2- 172.21.15.46:0/4039999925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331806f7a0 0x7f331806fb80 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f331806f1c0 tx=0x7f330801ec50 comp rx=0 tx=0).stop 2023-10-30T14:23:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 -- 172.21.15.46:0/4039999925 shutdown_connections 2023-10-30T14:23:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 --2- 172.21.15.46:0/4039999925 >> v2:172.21.15.154:3300/0 conn(0x7f33180c6bd0 0x7f3318069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 --2- 172.21.15.46:0/4039999925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331806f7a0 0x7f331806fb80 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 -- 172.21.15.46:0/4039999925 >> 172.21.15.46:0/4039999925 conn(0x7f33180c7910 msgr2=0x7f33180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:07.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 -- 172.21.15.46:0/4039999925 shutdown_connections 2023-10-30T14:23:07.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 -- 172.21.15.46:0/4039999925 wait complete. 2023-10-30T14:23:07.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 Processor -- start 2023-10-30T14:23:07.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 -- start start 2023-10-30T14:23:07.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331806f7a0 0x7f3318157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:07.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3318071110 0x7f331814bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:07.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33180c6bd0 0x7f331814a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:07.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331814d070 con 0x7f331806f7a0 2023-10-30T14:23:07.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3318157650 con 0x7f33180c6bd0 2023-10-30T14:23:07.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331e51d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33181463e0 con 0x7f3318071110 2023-10-30T14:23:07.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f3317fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331806f7a0 0x7f3318157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:07.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f3317fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331806f7a0 0x7f3318157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51884/0 (socket says 172.21.15.46:51884) 2023-10-30T14:23:07.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f3317fff640 1 -- 172.21.15.46:0/1569207335 learned_addr learned my addr 172.21.15.46:0/1569207335 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:07.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f331caba640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33180c6bd0 0x7f331814a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:07.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f33177fe640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3318071110 0x7f331814bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:07.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f3317fff640 1 -- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3318071110 msgr2=0x7f331814bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:07.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f3317fff640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3318071110 0x7f331814bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:07.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f3317fff640 1 -- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33180c6bd0 msgr2=0x7f331814a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:07.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f3317fff640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33180c6bd0 0x7f331814a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:07.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.312+0000 7f3317fff640 1 -- 172.21.15.46:0/1569207335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3308009420 con 0x7f331806f7a0 2023-10-30T14:23:07.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.316+0000 7f3317fff640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331806f7a0 0x7f3318157110 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f33080b3040 tx=0x7f330801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:07.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.316+0000 7f3314ff9640 1 -- 172.21.15.46:0/1569207335 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33080b0dd0 con 0x7f331806f7a0 2023-10-30T14:23:07.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.316+0000 7f3314ff9640 1 -- 172.21.15.46:0/1569207335 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33080b9430 con 0x7f331806f7a0 2023-10-30T14:23:07.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.316+0000 7f3314ff9640 1 -- 172.21.15.46:0/1569207335 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33080b97e0 con 0x7f331806f7a0 2023-10-30T14:23:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.316+0000 7f331e51d640 1 -- 172.21.15.46:0/1569207335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331814c9b0 con 0x7f331806f7a0 2023-10-30T14:23:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.316+0000 7f331e51d640 1 -- 172.21.15.46:0/1569207335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331814b250 con 0x7f331806f7a0 2023-10-30T14:23:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.316+0000 7f331e51d640 1 -- 172.21.15.46:0/1569207335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331814c810 con 0x7f331806f7a0 2023-10-30T14:23:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.316+0000 7f3314ff9640 1 -- 172.21.15.46:0/1569207335 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33080b7040 con 0x7f331806f7a0 2023-10-30T14:23:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.320+0000 7f3314ff9640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33040664d0 0x7f3304068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.320+0000 7f3314ff9640 1 -- 172.21.15.46:0/1569207335 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f3308122440 con 0x7f331806f7a0 2023-10-30T14:23:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.320+0000 7f33177fe640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33040664d0 0x7f3304068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.320+0000 7f33177fe640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33040664d0 0x7f3304068990 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f33181587f0 tx=0x7f330c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.324+0000 7f3314ff9640 1 -- 172.21.15.46:0/1569207335 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f331814c810 con 0x7f331806f7a0 2023-10-30T14:23:07.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 -- 172.21.15.46:0/1569207335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f331814c810 con 0x7f331806f7a0 2023-10-30T14:23:07.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f3314ff9640 1 -- 172.21.15.46:0/1569207335 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f331814c810 con 0x7f331806f7a0 2023-10-30T14:23:07.518 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:07.518 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":249,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:02.881530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:07.518 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 249 2023-10-30T14:23:07.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 -- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33040664d0 msgr2=0x7f3304068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:07.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33040664d0 0x7f3304068990 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f33181587f0 tx=0x7f330c008040 comp rx=0 tx=0).stop 2023-10-30T14:23:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 -- 172.21.15.46:0/1569207335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331806f7a0 msgr2=0x7f3318157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331806f7a0 0x7f3318157110 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f33080b3040 tx=0x7f330801b9f0 comp rx=0 tx=0).stop 2023-10-30T14:23:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 -- 172.21.15.46:0/1569207335 shutdown_connections 2023-10-30T14:23:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33040664d0 0x7f3304068990 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:07.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33180c6bd0 0x7f331814a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:07.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3318071110 0x7f331814bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:07.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 --2- 172.21.15.46:0/1569207335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331806f7a0 0x7f3318157110 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:07.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 -- 172.21.15.46:0/1569207335 >> 172.21.15.46:0/1569207335 conn(0x7f33180c7910 msgr2=0x7f33180c7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:07.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 -- 172.21.15.46:0/1569207335 shutdown_connections 2023-10-30T14:23:07.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:07.512+0000 7f331e51d640 1 -- 172.21.15.46:0/1569207335 wait complete. 2023-10-30T14:23:07.539 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:23:09.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.676+0000 7fa86d365640 1 Processor -- start 2023-10-30T14:23:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa86d365640 1 -- start start 2023-10-30T14:23:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa86d365640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8680c6b80 0x7fa8680bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa86d365640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa86813f4a0 0x7fa8680c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa86d365640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8680c0a50 con 0x7fa8680c6f60 2023-10-30T14:23:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa86d365640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86806f6e0 con 0x7fa8680c6b80 2023-10-30T14:23:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa86d365640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86806f9c0 con 0x7fa86813f4a0 2023-10-30T14:23:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa866ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8680c6b80 0x7fa8680bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa866ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8680c6b80 0x7fa8680bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51886/0 (socket says 172.21.15.46:51886) 2023-10-30T14:23:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa866ffd640 1 -- 172.21.15.46:0/1484190137 learned_addr learned my addr 172.21.15.46:0/1484190137 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa8677fe640 1 --2- 172.21.15.46:0/1484190137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa86813f4a0 0x7fa8680c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa866ffd640 1 -- 172.21.15.46:0/1484190137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa86813f4a0 msgr2=0x7fa8680c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa866ffd640 1 --2- 172.21.15.46:0/1484190137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa86813f4a0 0x7fa8680c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa866ffd640 1 -- 172.21.15.46:0/1484190137 >> v1:172.21.15.154:6789/0 conn(0x7fa8680c6f60 legacy=0x7fa86813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.680+0000 7fa866ffd640 1 -- 172.21.15.46:0/1484190137 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86806fca0 con 0x7fa8680c6b80 2023-10-30T14:23:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa866ffd640 1 --2- 172.21.15.46:0/1484190137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680c6b80 0x7fa8680bfec0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fa858009870 tx=0x7fa85801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5376d0c76d2fcc61 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa8667fc640 1 -- 172.21.15.46:0/1484190137 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8580ab650 con 0x7fa8680c6b80 2023-10-30T14:23:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa8667fc640 1 -- 172.21.15.46:0/1484190137 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8580ab7b0 con 0x7fa8680c6b80 2023-10-30T14:23:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa8667fc640 1 -- 172.21.15.46:0/1484190137 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8580abb40 con 0x7fa8680c6b80 2023-10-30T14:23:09.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa86d365640 1 -- 172.21.15.46:0/1484190137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680c6b80 msgr2=0x7fa8680bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:09.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa86d365640 1 --2- 172.21.15.46:0/1484190137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680c6b80 0x7fa8680bfec0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fa858009870 tx=0x7fa85801c540 comp rx=0 tx=0).stop 2023-10-30T14:23:09.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa86d365640 1 -- 172.21.15.46:0/1484190137 shutdown_connections 2023-10-30T14:23:09.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa86d365640 1 --2- 172.21.15.46:0/1484190137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa86813f4a0 0x7fa8680c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:09.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa86d365640 1 --2- 172.21.15.46:0/1484190137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680c6b80 0x7fa8680bfec0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:09.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa86d365640 1 -- 172.21.15.46:0/1484190137 >> 172.21.15.46:0/1484190137 conn(0x7fa8680c7910 msgr2=0x7fa8680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:09.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa86d365640 1 -- 172.21.15.46:0/1484190137 shutdown_connections 2023-10-30T14:23:09.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa86d365640 1 -- 172.21.15.46:0/1484190137 wait complete. 2023-10-30T14:23:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa86d365640 1 Processor -- start 2023-10-30T14:23:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.684+0000 7fa86d365640 1 -- start start 2023-10-30T14:23:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa86d365640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8680c6b80 0x7fa8681544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa86d365640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680c6f60 0x7fa8681328e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa86d365640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa86813f4a0 0x7fa868159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa86d365640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa868155050 con 0x7fa8680c6f60 2023-10-30T14:23:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa86d365640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86813e6f0 con 0x7fa8680c6b80 2023-10-30T14:23:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa86d365640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa868154a90 con 0x7fa86813f4a0 2023-10-30T14:23:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680c6f60 0x7fa8681328e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680c6f60 0x7fa8681328e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51892/0 (socket says 172.21.15.46:51892) 2023-10-30T14:23:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85ffff640 1 -- 172.21.15.46:0/2861512680 learned_addr learned my addr 172.21.15.46:0/2861512680 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa866ffd640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8680c6b80 0x7fa8681544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa8677fe640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa86813f4a0 0x7fa868159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85ffff640 1 -- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa86813f4a0 msgr2=0x7fa868159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85ffff640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa86813f4a0 0x7fa868159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85ffff640 1 -- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8680c6b80 msgr2=0x7fa8681544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85ffff640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8680c6b80 0x7fa8681544a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85ffff640 1 -- 172.21.15.46:0/2861512680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8580094d0 con 0x7fa8680c6f60 2023-10-30T14:23:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa866ffd640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8680c6b80 0x7fa8681544a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85ffff640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680c6f60 0x7fa8681328e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fa85000c500 tx=0x7fa85000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85f7fe640 1 -- 172.21.15.46:0/2861512680 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa850005220 con 0x7fa8680c6f60 2023-10-30T14:23:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85f7fe640 1 -- 172.21.15.46:0/2861512680 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa850005380 con 0x7fa8680c6f60 2023-10-30T14:23:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85f7fe640 1 -- 172.21.15.46:0/2861512680 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa850007270 con 0x7fa8680c6f60 2023-10-30T14:23:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa86d365640 1 -- 172.21.15.46:0/2861512680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa868154d70 con 0x7fa8680c6f60 2023-10-30T14:23:09.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa86d365640 1 -- 172.21.15.46:0/2861512680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa868132e20 con 0x7fa8680c6f60 2023-10-30T14:23:09.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa86d365640 1 -- 172.21.15.46:0/2861512680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86804e170 con 0x7fa8680c6f60 2023-10-30T14:23:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.688+0000 7fa85f7fe640 1 -- 172.21.15.46:0/2861512680 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa8500073d0 con 0x7fa8680c6f60 2023-10-30T14:23:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.692+0000 7fa85f7fe640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8480664a0 0x7fa848068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.692+0000 7fa85f7fe640 1 -- 172.21.15.46:0/2861512680 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa850084310 con 0x7fa8680c6f60 2023-10-30T14:23:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.692+0000 7fa866ffd640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8480664a0 0x7fa848068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.692+0000 7fa866ffd640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8480664a0 0x7fa848068960 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa8580af000 tx=0x7fa8580a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.696+0000 7fa85f7fe640 1 -- 172.21.15.46:0/2861512680 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa850050bf0 con 0x7fa8680c6f60 2023-10-30T14:23:09.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 -- 172.21.15.46:0/2861512680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa868071ae0 con 0x7fa8680c6f60 2023-10-30T14:23:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa85f7fe640 1 -- 172.21.15.46:0/2861512680 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 250 v250) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fa850054210 con 0x7fa8680c6f60 2023-10-30T14:23:09.892 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:09.893 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":250,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":250,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:08.177482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 250 2023-10-30T14:23:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 -- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8480664a0 msgr2=0x7fa848068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8480664a0 0x7fa848068960 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa8580af000 tx=0x7fa8580a7000 comp rx=0 tx=0).stop 2023-10-30T14:23:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 -- 172.21.15.46:0/2861512680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680c6f60 msgr2=0x7fa8681328e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680c6f60 0x7fa8681328e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fa85000c500 tx=0x7fa85000c9c0 comp rx=0 tx=0).stop 2023-10-30T14:23:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 -- 172.21.15.46:0/2861512680 shutdown_connections 2023-10-30T14:23:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8480664a0 0x7fa848068960 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa86813f4a0 0x7fa868159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680c6f60 0x7fa8681328e0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 --2- 172.21.15.46:0/2861512680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8680c6b80 0x7fa8681544a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 -- 172.21.15.46:0/2861512680 >> 172.21.15.46:0/2861512680 conn(0x7fa8680c7910 msgr2=0x7fa868153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 -- 172.21.15.46:0/2861512680 shutdown_connections 2023-10-30T14:23:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:09.888+0000 7fa86d365640 1 -- 172.21.15.46:0/2861512680 wait complete. 2023-10-30T14:23:09.914 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=9426 2023-10-30T14:23:09.914 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.a 2023-10-30T14:23:09.914 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T14:23:09.914 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T14:23:09.915 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T14:23:09.918 INFO:tasks.ceph.mds.a:Started 2023-10-30T14:23:09.918 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:10.009 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-10-30T14:23:10.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3cad6640 1 Processor -- start 2023-10-30T14:23:10.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3cad6640 1 -- start start 2023-10-30T14:23:10.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3cad6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e381538a0 0x7f4e3815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:10.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3cad6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e380cb800 con 0x7f4e380cf160 2023-10-30T14:23:10.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3cad6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3815dc30 con 0x7f4e380ca0e0 2023-10-30T14:23:10.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3cad6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e380cb330 con 0x7f4e381538a0 2023-10-30T14:23:10.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e36d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4e380cf160 0x7f4e38164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38852/0 (socket says 172.21.15.46:38852) 2023-10-30T14:23:10.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e36d9d640 1 -- 172.21.15.46:0/2289457591 learned_addr learned my addr 172.21.15.46:0/2289457591 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:23:10.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3659c640 1 --2- 172.21.15.46:0/2289457591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e381538a0 0x7f4e3815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:10.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3559a640 1 -- 172.21.15.46:0/2289457591 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267830588 0 0) 0x7f4e380cb800 con 0x7f4e380cf160 2023-10-30T14:23:10.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3559a640 1 -- 172.21.15.46:0/2289457591 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e20003530 con 0x7f4e380cf160 2023-10-30T14:23:10.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3559a640 1 -- 172.21.15.46:0/2289457591 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020866019 0 0) 0x7f4e2c001120 con 0x7f4e380ca0e0 2023-10-30T14:23:10.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3559a640 1 -- 172.21.15.46:0/2289457591 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e380cb800 con 0x7f4e380ca0e0 2023-10-30T14:23:10.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3559a640 1 -- 172.21.15.46:0/2289457591 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1970970746 0 0) 0x7f4e20003530 con 0x7f4e380cf160 2023-10-30T14:23:10.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3659c640 1 -- 172.21.15.46:0/2289457591 >> v1:172.21.15.154:6789/0 conn(0x7f4e380ca0e0 legacy=0x7f4e3815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:10.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3659c640 1 -- 172.21.15.46:0/2289457591 >> v1:172.21.15.46:6789/0 conn(0x7f4e380cf160 legacy=0x7f4e38164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:10.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.060+0000 7f4e3659c640 1 -- 172.21.15.46:0/2289457591 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e380cad70 con 0x7f4e381538a0 2023-10-30T14:23:10.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.064+0000 7f4e3659c640 1 --2- 172.21.15.46:0/2289457591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e381538a0 0x7f4e3815d6f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4e280097f0 tx=0x7f4e2801ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c704455c36bd7d39 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:10.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.064+0000 7f4e3559a640 1 -- 172.21.15.46:0/2289457591 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30001930 con 0x7f4e381538a0 2023-10-30T14:23:10.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.064+0000 7f4e3559a640 1 -- 172.21.15.46:0/2289457591 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e280a8650 con 0x7f4e381538a0 2023-10-30T14:23:10.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.064+0000 7f4e3559a640 1 -- 172.21.15.46:0/2289457591 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e280a89e0 con 0x7f4e381538a0 2023-10-30T14:23:10.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.064+0000 7f4e3cad6640 1 -- 172.21.15.46:0/2289457591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e381538a0 msgr2=0x7f4e3815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:10.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.064+0000 7f4e3cad6640 1 --2- 172.21.15.46:0/2289457591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e381538a0 0x7f4e3815d6f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4e280097f0 tx=0x7f4e2801ebd0 comp rx=0 tx=0).stop 2023-10-30T14:23:10.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 -- 172.21.15.46:0/2289457591 shutdown_connections 2023-10-30T14:23:10.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 --2- 172.21.15.46:0/2289457591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e381538a0 0x7f4e3815d6f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:10.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 -- 172.21.15.46:0/2289457591 >> 172.21.15.46:0/2289457591 conn(0x7f4e3806bfe0 msgr2=0x7f4e3806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:10.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 -- 172.21.15.46:0/2289457591 shutdown_connections 2023-10-30T14:23:10.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 -- 172.21.15.46:0/2289457591 wait complete. 2023-10-30T14:23:10.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 Processor -- start 2023-10-30T14:23:10.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 -- start start 2023-10-30T14:23:10.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380ca0e0 0x7f4e381f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:10.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e380cf160 0x7f4e380b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:10.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e381538a0 0x7f4e38157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:10.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e381e3670 con 0x7f4e381538a0 2023-10-30T14:23:10.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e381f34e0 con 0x7f4e380cf160 2023-10-30T14:23:10.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3cad6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e381e5630 con 0x7f4e380ca0e0 2023-10-30T14:23:10.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e36d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e381538a0 0x7f4e38157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:10.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e3659c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380ca0e0 0x7f4e381f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:10.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e36d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e381538a0 0x7f4e38157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51908/0 (socket says 172.21.15.46:51908) 2023-10-30T14:23:10.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e35d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e380cf160 0x7f4e380b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:10.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e36d9d640 1 -- 172.21.15.46:0/711006733 learned_addr learned my addr 172.21.15.46:0/711006733 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:10.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e36d9d640 1 -- 172.21.15.46:0/711006733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380ca0e0 msgr2=0x7f4e381f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:10.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e36d9d640 1 --2- 172.21.15.46:0/711006733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380ca0e0 0x7f4e381f3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:10.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e36d9d640 1 -- 172.21.15.46:0/711006733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e380cf160 msgr2=0x7f4e380b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:10.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e36d9d640 1 --2- 172.21.15.46:0/711006733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e380cf160 0x7f4e380b44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:10.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.068+0000 7f4e36d9d640 1 -- 172.21.15.46:0/711006733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e28009420 con 0x7f4e381538a0 2023-10-30T14:23:10.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.072+0000 7f4e36d9d640 1 --2- 172.21.15.46:0/711006733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e381538a0 0x7f4e38157520 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000cc20 tx=0x7f4e3000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:10.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.072+0000 7f4e26ffd640 1 -- 172.21.15.46:0/711006733 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3000ce30 con 0x7f4e381538a0 2023-10-30T14:23:10.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.072+0000 7f4e26ffd640 1 -- 172.21.15.46:0/711006733 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e30012400 con 0x7f4e381538a0 2023-10-30T14:23:10.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.072+0000 7f4e3cad6640 1 -- 172.21.15.46:0/711006733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e381f3eb0 con 0x7f4e381538a0 2023-10-30T14:23:10.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.072+0000 7f4e26ffd640 1 -- 172.21.15.46:0/711006733 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e300127b0 con 0x7f4e381538a0 2023-10-30T14:23:10.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.072+0000 7f4e3cad6640 1 -- 172.21.15.46:0/711006733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e381e30b0 con 0x7f4e381538a0 2023-10-30T14:23:10.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.072+0000 7f4e3cad6640 1 -- 172.21.15.46:0/711006733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e38073c40 con 0x7f4e381538a0 2023-10-30T14:23:10.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.072+0000 7f4e26ffd640 1 -- 172.21.15.46:0/711006733 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e30007070 con 0x7f4e381538a0 2023-10-30T14:23:10.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.076+0000 7f4e26ffd640 1 --2- 172.21.15.46:0/711006733 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e180664a0 0x7f4e18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:10.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.076+0000 7f4e26ffd640 1 -- 172.21.15.46:0/711006733 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f4e30084aa0 con 0x7f4e381538a0 2023-10-30T14:23:10.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.076+0000 7f4e3659c640 1 --2- 172.21.15.46:0/711006733 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e180664a0 0x7f4e18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:10.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.076+0000 7f4e3659c640 1 --2- 172.21.15.46:0/711006733 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e180664a0 0x7f4e18068960 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4e280b2000 tx=0x7f4e280aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:10.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.080+0000 7f4e26ffd640 1 -- 172.21.15.46:0/711006733 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e300513f0 con 0x7f4e381538a0 2023-10-30T14:23:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.272+0000 7f4e3cad6640 1 -- 172.21.15.46:0/711006733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e3804e170 con 0x7f4e381538a0 2023-10-30T14:23:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.272+0000 7f4e26ffd640 1 -- 172.21.15.46:0/711006733 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 250 v250) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f4e30017070 con 0x7f4e381538a0 2023-10-30T14:23:10.278 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:10.278 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":250,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":250,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:08.177482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:10.278 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 250 2023-10-30T14:23:10.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.272+0000 7f4e3cad6640 1 -- 172.21.15.46:0/711006733 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e180664a0 msgr2=0x7f4e18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.272+0000 7f4e3cad6640 1 --2- 172.21.15.46:0/711006733 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e180664a0 0x7f4e18068960 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4e280b2000 tx=0x7f4e280aa070 comp rx=0 tx=0).stop 2023-10-30T14:23:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.272+0000 7f4e3cad6640 1 -- 172.21.15.46:0/711006733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e381538a0 msgr2=0x7f4e38157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.272+0000 7f4e3cad6640 1 --2- 172.21.15.46:0/711006733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e381538a0 0x7f4e38157520 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000cc20 tx=0x7f4e3000a430 comp rx=0 tx=0).stop 2023-10-30T14:23:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.272+0000 7f4e3cad6640 1 -- 172.21.15.46:0/711006733 shutdown_connections 2023-10-30T14:23:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.272+0000 7f4e3cad6640 1 --2- 172.21.15.46:0/711006733 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4e180664a0 0x7f4e18068960 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.272+0000 7f4e3cad6640 1 --2- 172.21.15.46:0/711006733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e381538a0 0x7f4e38157520 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.272+0000 7f4e3cad6640 1 --2- 172.21.15.46:0/711006733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e380cf160 0x7f4e380b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.272+0000 7f4e3cad6640 1 --2- 172.21.15.46:0/711006733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380ca0e0 0x7f4e381f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.272+0000 7f4e3cad6640 1 -- 172.21.15.46:0/711006733 >> 172.21.15.46:0/711006733 conn(0x7f4e3806bfe0 msgr2=0x7f4e38151d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.276+0000 7f4e3cad6640 1 -- 172.21.15.46:0/711006733 shutdown_connections 2023-10-30T14:23:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:10.276+0000 7f4e3cad6640 1 -- 172.21.15.46:0/711006733 wait complete. 2023-10-30T14:23:10.299 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T14:23:12.300 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f883070f640 1 Processor -- start 2023-10-30T14:23:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f883070f640 1 -- start start 2023-10-30T14:23:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f883070f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8828157080 0x7f8828071b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f883070f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88280c8600 0x7f8828072090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f883070f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88280c8da0 con 0x7f88280c7990 2023-10-30T14:23:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f883070f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8828157460 con 0x7f8828157080 2023-10-30T14:23:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f883070f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882806cb60 con 0x7f88280c8600 2023-10-30T14:23:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882dcaa640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f88280c7990 0x7f88280c7ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38854/0 (socket says 172.21.15.46:38854) 2023-10-30T14:23:12.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882dcaa640 1 -- 172.21.15.46:0/4245168447 learned_addr learned my addr 172.21.15.46:0/4245168447 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:23:12.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882e4ab640 1 --2- 172.21.15.46:0/4245168447 >> v2:172.21.15.154:3300/0 conn(0x7f8828157080 0x7f8828071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:12.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882ecac640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88280c8600 0x7f8828072090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:12.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647210458 0 0) 0x7f88280c8da0 con 0x7f88280c7990 2023-10-30T14:23:12.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8814003530 con 0x7f88280c7990 2023-10-30T14:23:12.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 224498180 0 0) 0x7f8814003530 con 0x7f88280c7990 2023-10-30T14:23:12.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88280c8600 msgr2=0x7f8828072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:12.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88280c8600 0x7f8828072090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:12.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 >> v2:172.21.15.154:3300/0 conn(0x7f8828157080 msgr2=0x7f8828071b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:12.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 --2- 172.21.15.46:0/4245168447 >> v2:172.21.15.154:3300/0 conn(0x7f8828157080 0x7f8828071b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:12.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882806ce20 con 0x7f88280c7990 2023-10-30T14:23:12.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f8818001680 con 0x7f88280c7990 2023-10-30T14:23:12.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 >> v1:172.21.15.46:6789/0 conn(0x7f88280c7990 legacy=0x7f88280c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:12.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8814003980 0x7f8814003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:12.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88280c8600 0x7f88140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:12.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814004d10 0x7f8814005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:12.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8828157460 con 0x7f8814003980 2023-10-30T14:23:12.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882806cb60 con 0x7f8814004d10 2023-10-30T14:23:12.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881400ba10 con 0x7f88280c8600 2023-10-30T14:23:12.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882e4ab640 1 --2- 172.21.15.46:0/4245168447 >> v2:172.21.15.154:3300/0 conn(0x7f8828157080 0x7f8828071b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:23:12.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882ecac640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814004d10 0x7f8814005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:12.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882e4ab640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8814003980 0x7f8814003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:12.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882dcaa640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88280c8600 0x7f88140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:12.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882e4ab640 1 -- 172.21.15.46:0/4245168447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88280c8600 msgr2=0x7f88140047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:12.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882e4ab640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88280c8600 0x7f88140047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:12.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882e4ab640 1 -- 172.21.15.46:0/4245168447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814004d10 msgr2=0x7f8814005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:12.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882e4ab640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814004d10 0x7f8814005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:12.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882e4ab640 1 -- 172.21.15.46:0/4245168447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8814003530 con 0x7f8814003980 2023-10-30T14:23:12.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.440+0000 7f882dcaa640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88280c8600 0x7f88140047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:12.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.444+0000 7f882e4ab640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8814003980 0x7f8814003da0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f8824009c90 tx=0x7f882401cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875028f3a222c73a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:12.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.444+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824004020 con 0x7f8814003980 2023-10-30T14:23:12.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.444+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8824004180 con 0x7f8814003980 2023-10-30T14:23:12.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.444+0000 7f882d4a9640 1 -- 172.21.15.46:0/4245168447 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88240a57e0 con 0x7f8814003980 2023-10-30T14:23:12.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.444+0000 7f883070f640 1 -- 172.21.15.46:0/4245168447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8814003980 msgr2=0x7f8814003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:12.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.444+0000 7f883070f640 1 --2- 172.21.15.46:0/4245168447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8814003980 0x7f8814003da0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f8824009c90 tx=0x7f882401cc50 comp rx=0 tx=0).stop 2023-10-30T14:23:12.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.444+0000 7f882ecac640 1 -- 172.21.15.46:0/4245168447 reap_dead start 2023-10-30T14:23:12.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.444+0000 7f883070f640 1 -- 172.21.15.46:0/4245168447 shutdown_connections 2023-10-30T14:23:12.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.444+0000 7f883070f640 1 -- 172.21.15.46:0/4245168447 >> 172.21.15.46:0/4245168447 conn(0x7f8828069e90 msgr2=0x7f882806a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:12.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.444+0000 7f883070f640 1 -- 172.21.15.46:0/4245168447 shutdown_connections 2023-10-30T14:23:12.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.444+0000 7f883070f640 1 -- 172.21.15.46:0/4245168447 wait complete. 2023-10-30T14:23:12.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f883070f640 1 Processor -- start 2023-10-30T14:23:12.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f883070f640 1 -- start start 2023-10-30T14:23:12.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f883070f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814003980 0x7f8828157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:12.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f883070f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8828153e50 0x7f882814f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:12.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f883070f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f882814f710 0x7f882814fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:12.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f883070f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88281f2b70 con 0x7f8828153e50 2023-10-30T14:23:12.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f883070f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88281f1f10 con 0x7f8814003980 2023-10-30T14:23:12.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f883070f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8828151030 con 0x7f882814f710 2023-10-30T14:23:12.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f882dcaa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8828153e50 0x7f882814f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:12.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f882dcaa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8828153e50 0x7f882814f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51936/0 (socket says 172.21.15.46:51936) 2023-10-30T14:23:12.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f882dcaa640 1 -- 172.21.15.46:0/3144542828 learned_addr learned my addr 172.21.15.46:0/3144542828 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:12.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f882e4ab640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814003980 0x7f8828157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:12.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f882ecac640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f882814f710 0x7f882814fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:12.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f882dcaa640 1 -- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f882814f710 msgr2=0x7f882814fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:12.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f882dcaa640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f882814f710 0x7f882814fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:12.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f882dcaa640 1 -- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814003980 msgr2=0x7f8828157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f882dcaa640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814003980 0x7f8828157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f882dcaa640 1 -- 172.21.15.46:0/3144542828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88281f21d0 con 0x7f8828153e50 2023-10-30T14:23:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f882e4ab640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814003980 0x7f8828157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f882dcaa640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8828153e50 0x7f882814f1d0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f881800e780 tx=0x7f881800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f881effd640 1 -- 172.21.15.46:0/3144542828 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881800ee30 con 0x7f8828153e50 2023-10-30T14:23:12.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f881effd640 1 -- 172.21.15.46:0/3144542828 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881800c8d0 con 0x7f8828153e50 2023-10-30T14:23:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f881effd640 1 -- 172.21.15.46:0/3144542828 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881800cc60 con 0x7f8828153e50 2023-10-30T14:23:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f883070f640 1 -- 172.21.15.46:0/3144542828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88281f24b0 con 0x7f8828153e50 2023-10-30T14:23:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.448+0000 7f883070f640 1 -- 172.21.15.46:0/3144542828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8828154de0 con 0x7f8828153e50 2023-10-30T14:23:12.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.452+0000 7f883070f640 1 -- 172.21.15.46:0/3144542828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882814faf0 con 0x7f8828153e50 2023-10-30T14:23:12.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.452+0000 7f881effd640 1 -- 172.21.15.46:0/3144542828 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8818013420 con 0x7f8828153e50 2023-10-30T14:23:12.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.452+0000 7f881effd640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f880c0664d0 0x7f880c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:12.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.452+0000 7f881effd640 1 -- 172.21.15.46:0/3144542828 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f88180845f0 con 0x7f8828153e50 2023-10-30T14:23:12.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.452+0000 7f882e4ab640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f880c0664d0 0x7f880c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:12.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.452+0000 7f882e4ab640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f880c0664d0 0x7f880c068990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f88240ae000 tx=0x7f88240046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:12.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.456+0000 7f881effd640 1 -- 172.21.15.46:0/3144542828 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f882814faf0 con 0x7f8828153e50 2023-10-30T14:23:12.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.664+0000 7f883070f640 1 -- 172.21.15.46:0/3144542828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f882814faf0 con 0x7f8828153e50 2023-10-30T14:23:12.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f881effd640 1 -- 172.21.15.46:0/3144542828 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 251 v251) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f882814faf0 con 0x7f8828153e50 2023-10-30T14:23:12.672 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:12.672 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":251,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9882,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":251}],"filesystems":[{"mdsmap":{"epoch":250,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:08.177482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:12.673 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 251 2023-10-30T14:23:12.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f883070f640 1 -- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f880c0664d0 msgr2=0x7f880c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:12.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f883070f640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f880c0664d0 0x7f880c068990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f88240ae000 tx=0x7f88240046d0 comp rx=0 tx=0).stop 2023-10-30T14:23:12.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f883070f640 1 -- 172.21.15.46:0/3144542828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8828153e50 msgr2=0x7f882814f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:12.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f883070f640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8828153e50 0x7f882814f1d0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f881800e780 tx=0x7f881800ec40 comp rx=0 tx=0).stop 2023-10-30T14:23:12.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f883070f640 1 -- 172.21.15.46:0/3144542828 shutdown_connections 2023-10-30T14:23:12.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f883070f640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f880c0664d0 0x7f880c068990 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:12.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f883070f640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f882814f710 0x7f882814fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:12.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f883070f640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8828153e50 0x7f882814f1d0 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:12.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f883070f640 1 --2- 172.21.15.46:0/3144542828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8814003980 0x7f8828157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:12.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f883070f640 1 -- 172.21.15.46:0/3144542828 >> 172.21.15.46:0/3144542828 conn(0x7f8828069e90 msgr2=0x7f8828155bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:12.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f883070f640 1 -- 172.21.15.46:0/3144542828 shutdown_connections 2023-10-30T14:23:12.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:12.668+0000 7f883070f640 1 -- 172.21.15.46:0/3144542828 wait complete. 2023-10-30T14:23:12.693 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T14:23:12.693 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 29 secs before thrashing 2023-10-30T14:23:18.866 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T14:23:18.860+0000 7fc739348640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:23:18.866 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T14:23:18.860+0000 7fc739348640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:23:20.593 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:23:20.593 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 88 -eq 48 ] 2023-10-30T14:23:20.594 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.88 2023-10-30T14:23:20.594 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.88 2023-10-30T14:23:20.594 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.88 2023-10-30T14:23:20.594 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.88 2023-10-30T14:23:20.595 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:23:26.349 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:23:26.350 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.89 2023-10-30T14:23:26.350 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 89 -eq 48 ] 2023-10-30T14:23:26.351 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.89 2023-10-30T14:23:26.351 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.89 2023-10-30T14:23:26.351 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.89 2023-10-30T14:23:26.351 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:23:33.973 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:23:33.973 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 90 -eq 48 ] 2023-10-30T14:23:33.974 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.90 2023-10-30T14:23:33.974 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking v0.90 2023-10-30T14:23:33.974 INFO:tasks.workunit.client.0.smithi046.stdout:checking v0.90 2023-10-30T14:23:33.975 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd .snap/v0.90 2023-10-30T14:23:33.975 INFO:tasks.workunit.client.0.smithi046.stderr:+ git diff --exit-code 2023-10-30T14:23:40.934 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd ../.. 2023-10-30T14:23:40.934 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 1 -eq 48 ] 2023-10-30T14:23:40.934 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.1 2023-10-30T14:23:40.935 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.1 2023-10-30T14:23:40.958 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 2 -eq 48 ] 2023-10-30T14:23:40.959 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.2 2023-10-30T14:23:40.959 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.2 2023-10-30T14:23:40.969 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 3 -eq 48 ] 2023-10-30T14:23:40.969 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.3 2023-10-30T14:23:40.969 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.3 2023-10-30T14:23:40.980 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 4 -eq 48 ] 2023-10-30T14:23:40.980 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.4 2023-10-30T14:23:40.980 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.4 2023-10-30T14:23:40.991 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 5 -eq 48 ] 2023-10-30T14:23:40.991 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.5 2023-10-30T14:23:40.991 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.5 2023-10-30T14:23:41.002 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 6 -eq 48 ] 2023-10-30T14:23:41.002 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.6 2023-10-30T14:23:41.003 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.6 2023-10-30T14:23:41.013 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 7 -eq 48 ] 2023-10-30T14:23:41.013 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.7 2023-10-30T14:23:41.013 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.7 2023-10-30T14:23:41.024 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 8 -eq 48 ] 2023-10-30T14:23:41.024 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.8 2023-10-30T14:23:41.024 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.8 2023-10-30T14:23:41.035 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 9 -eq 48 ] 2023-10-30T14:23:41.035 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.9 2023-10-30T14:23:41.036 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.9 2023-10-30T14:23:41.046 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 10 -eq 48 ] 2023-10-30T14:23:41.046 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.10 2023-10-30T14:23:41.046 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.10 2023-10-30T14:23:41.056 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 11 -eq 48 ] 2023-10-30T14:23:41.057 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.11 2023-10-30T14:23:41.057 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.11 2023-10-30T14:23:41.067 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 12 -eq 48 ] 2023-10-30T14:23:41.067 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.12 2023-10-30T14:23:41.067 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.12 2023-10-30T14:23:41.078 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 13 -eq 48 ] 2023-10-30T14:23:41.078 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.13 2023-10-30T14:23:41.078 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.13 2023-10-30T14:23:41.088 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 14 -eq 48 ] 2023-10-30T14:23:41.089 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.14 2023-10-30T14:23:41.089 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.14 2023-10-30T14:23:41.099 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 15 -eq 48 ] 2023-10-30T14:23:41.099 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.15 2023-10-30T14:23:41.099 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.15 2023-10-30T14:23:41.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 16 -eq 48 ] 2023-10-30T14:23:41.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.16 2023-10-30T14:23:41.110 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.16 2023-10-30T14:23:41.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 17 -eq 48 ] 2023-10-30T14:23:41.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.17 2023-10-30T14:23:41.121 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.17 2023-10-30T14:23:41.131 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 18 -eq 48 ] 2023-10-30T14:23:41.131 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.18 2023-10-30T14:23:41.131 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.18 2023-10-30T14:23:41.141 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 19 -eq 48 ] 2023-10-30T14:23:41.141 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.19 2023-10-30T14:23:41.142 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.19 2023-10-30T14:23:41.152 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 20 -eq 48 ] 2023-10-30T14:23:41.152 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.20 2023-10-30T14:23:41.152 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.20 2023-10-30T14:23:41.163 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 21 -eq 48 ] 2023-10-30T14:23:41.163 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.21 2023-10-30T14:23:41.163 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.21 2023-10-30T14:23:41.174 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 22 -eq 48 ] 2023-10-30T14:23:41.174 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.22 2023-10-30T14:23:41.174 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.22 2023-10-30T14:23:41.184 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 23 -eq 48 ] 2023-10-30T14:23:41.184 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.23 2023-10-30T14:23:41.184 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.23 2023-10-30T14:23:41.194 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 24 -eq 48 ] 2023-10-30T14:23:41.194 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.24 2023-10-30T14:23:41.195 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.24 2023-10-30T14:23:41.205 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 25 -eq 48 ] 2023-10-30T14:23:41.205 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.25 2023-10-30T14:23:41.206 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.25 2023-10-30T14:23:41.217 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 26 -eq 48 ] 2023-10-30T14:23:41.217 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.26 2023-10-30T14:23:41.217 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.26 2023-10-30T14:23:41.228 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 27 -eq 48 ] 2023-10-30T14:23:41.228 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.27 2023-10-30T14:23:41.229 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.27 2023-10-30T14:23:41.241 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 28 -eq 48 ] 2023-10-30T14:23:41.241 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.28 2023-10-30T14:23:41.241 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.28 2023-10-30T14:23:41.252 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 29 -eq 48 ] 2023-10-30T14:23:41.253 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.29 2023-10-30T14:23:41.253 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.29 2023-10-30T14:23:41.695 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.832+0000 7f98e16ea640 1 Processor -- start 2023-10-30T14:23:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.832+0000 7f98e16ea640 1 -- start start 2023-10-30T14:23:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.832+0000 7f98e16ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98dc0c6b80 0x7f98dc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.836+0000 7f98e16ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc0c0a50 con 0x7f98dc0c6f60 2023-10-30T14:23:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.836+0000 7f98e16ea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc06f6e0 con 0x7f98dc13f4a0 2023-10-30T14:23:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.836+0000 7f98e16ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc06f9c0 con 0x7f98dc0c6b80 2023-10-30T14:23:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.836+0000 7f98d3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f98dc0c6f60 0x7f98dc13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44886/0 (socket says 172.21.15.46:44886) 2023-10-30T14:23:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.836+0000 7f98d3fff640 1 -- 172.21.15.46:0/2472158813 learned_addr learned my addr 172.21.15.46:0/2472158813 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:23:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.836+0000 7f98daffd640 1 --2- 172.21.15.46:0/2472158813 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98dc0c6b80 0x7f98dc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.836+0000 7f98da7fc640 1 -- 172.21.15.46:0/2472158813 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472338706 0 0) 0x7f98dc0c0a50 con 0x7f98dc0c6f60 2023-10-30T14:23:41.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.836+0000 7f98da7fc640 1 -- 172.21.15.46:0/2472158813 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c8003530 con 0x7f98dc0c6f60 2023-10-30T14:23:41.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.836+0000 7f98daffd640 1 -- 172.21.15.46:0/2472158813 >> v1:172.21.15.154:6789/0 conn(0x7f98dc13f4a0 legacy=0x7f98dc1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:41.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.836+0000 7f98daffd640 1 -- 172.21.15.46:0/2472158813 >> v1:172.21.15.46:6789/0 conn(0x7f98dc0c6f60 legacy=0x7f98dc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:41.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.836+0000 7f98daffd640 1 -- 172.21.15.46:0/2472158813 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc06fca0 con 0x7f98dc0c6b80 2023-10-30T14:23:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.836+0000 7f98daffd640 1 --2- 172.21.15.46:0/2472158813 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98dc0c6b80 0x7f98dc0bfec0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0097f0 tx=0x7f98cc01c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40bb635299ff3658 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98da7fc640 1 -- 172.21.15.46:0/2472158813 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc0ac630 con 0x7f98dc0c6b80 2023-10-30T14:23:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98da7fc640 1 -- 172.21.15.46:0/2472158813 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc0ac790 con 0x7f98dc0c6b80 2023-10-30T14:23:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98da7fc640 1 -- 172.21.15.46:0/2472158813 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc0acb20 con 0x7f98dc0c6b80 2023-10-30T14:23:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 -- 172.21.15.46:0/2472158813 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98dc0c6b80 msgr2=0x7f98dc0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 --2- 172.21.15.46:0/2472158813 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98dc0c6b80 0x7f98dc0bfec0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0097f0 tx=0x7f98cc01c4c0 comp rx=0 tx=0).stop 2023-10-30T14:23:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 -- 172.21.15.46:0/2472158813 shutdown_connections 2023-10-30T14:23:41.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 --2- 172.21.15.46:0/2472158813 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98dc0c6b80 0x7f98dc0bfec0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:41.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 -- 172.21.15.46:0/2472158813 >> 172.21.15.46:0/2472158813 conn(0x7f98dc0c7910 msgr2=0x7f98dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:41.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 -- 172.21.15.46:0/2472158813 shutdown_connections 2023-10-30T14:23:41.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 -- 172.21.15.46:0/2472158813 wait complete. 2023-10-30T14:23:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 Processor -- start 2023-10-30T14:23:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 -- start start 2023-10-30T14:23:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98dc0c6b80 0x7f98dc0c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98dc0c6f60 0x7f98dc138620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98dc13f4a0 0x7f98dc138b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:41.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc151500 con 0x7f98dc0c6f60 2023-10-30T14:23:41.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc1517b0 con 0x7f98dc0c6b80 2023-10-30T14:23:41.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.840+0000 7f98e16ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc150f80 con 0x7f98dc13f4a0 2023-10-30T14:23:41.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98dc0c6f60 0x7f98dc138620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:41.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98dc0c6f60 0x7f98dc138620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47456/0 (socket says 172.21.15.46:47456) 2023-10-30T14:23:41.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d3fff640 1 -- 172.21.15.46:0/268260748 learned_addr learned my addr 172.21.15.46:0/268260748 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:41.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98daffd640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98dc0c6b80 0x7f98dc0c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:41.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98db7fe640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98dc13f4a0 0x7f98dc138b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d3fff640 1 -- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98dc13f4a0 msgr2=0x7f98dc138b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d3fff640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98dc13f4a0 0x7f98dc138b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d3fff640 1 -- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98dc0c6b80 msgr2=0x7f98dc0c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d3fff640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98dc0c6b80 0x7f98dc0c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d3fff640 1 -- 172.21.15.46:0/268260748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc009420 con 0x7f98dc0c6f60 2023-10-30T14:23:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d3fff640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98dc0c6f60 0x7f98dc138620 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f98c4001ec0 tx=0x7f98c400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d37fe640 1 -- 172.21.15.46:0/268260748 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4005d50 con 0x7f98dc0c6f60 2023-10-30T14:23:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d37fe640 1 -- 172.21.15.46:0/268260748 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c4007070 con 0x7f98dc0c6f60 2023-10-30T14:23:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d37fe640 1 -- 172.21.15.46:0/268260748 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40073e0 con 0x7f98dc0c6f60 2023-10-30T14:23:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98e16ea640 1 -- 172.21.15.46:0/268260748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98dc151220 con 0x7f98dc0c6f60 2023-10-30T14:23:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98e16ea640 1 -- 172.21.15.46:0/268260748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc1f6530 con 0x7f98dc0c6f60 2023-10-30T14:23:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98e16ea640 1 -- 172.21.15.46:0/268260748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc1f6080 con 0x7f98dc0c6f60 2023-10-30T14:23:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.844+0000 7f98d37fe640 1 -- 172.21.15.46:0/268260748 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f98c4007540 con 0x7f98dc0c6f60 2023-10-30T14:23:41.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.848+0000 7f98d37fe640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98bc0664d0 0x7f98bc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:41.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.848+0000 7f98d37fe640 1 -- 172.21.15.46:0/268260748 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f98c4083f30 con 0x7f98dc0c6f60 2023-10-30T14:23:41.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.848+0000 7f98daffd640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98bc0664d0 0x7f98bc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:41.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.848+0000 7f98daffd640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98bc0664d0 0x7f98bc068990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0ae000 tx=0x7f98cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:41.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:41.852+0000 7f98d37fe640 1 -- 172.21.15.46:0/268260748 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f98c40508e0 con 0x7f98dc0c6f60 2023-10-30T14:23:42.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.039+0000 7f98e16ea640 1 -- 172.21.15.46:0/268260748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98dc04e170 con 0x7f98dc0c6f60 2023-10-30T14:23:42.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.039+0000 7f98d37fe640 1 -- 172.21.15.46:0/268260748 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f98c4017030 con 0x7f98dc0c6f60 2023-10-30T14:23:42.046 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:42.047 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":252,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9882,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":251}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:19.162295+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:42.047 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 252 2023-10-30T14:23:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.043+0000 7f98e16ea640 1 -- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98bc0664d0 msgr2=0x7f98bc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.043+0000 7f98e16ea640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98bc0664d0 0x7f98bc068990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0ae000 tx=0x7f98cc008040 comp rx=0 tx=0).stop 2023-10-30T14:23:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.043+0000 7f98e16ea640 1 -- 172.21.15.46:0/268260748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98dc0c6f60 msgr2=0x7f98dc138620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.043+0000 7f98e16ea640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98dc0c6f60 0x7f98dc138620 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f98c4001ec0 tx=0x7f98c400c960 comp rx=0 tx=0).stop 2023-10-30T14:23:42.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.043+0000 7f98e16ea640 1 -- 172.21.15.46:0/268260748 shutdown_connections 2023-10-30T14:23:42.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.043+0000 7f98e16ea640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f98bc0664d0 0x7f98bc068990 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.043+0000 7f98e16ea640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98dc13f4a0 0x7f98dc138b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.043+0000 7f98e16ea640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98dc0c6f60 0x7f98dc138620 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.043+0000 7f98e16ea640 1 --2- 172.21.15.46:0/268260748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98dc0c6b80 0x7f98dc0c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.043+0000 7f98e16ea640 1 -- 172.21.15.46:0/268260748 >> 172.21.15.46:0/268260748 conn(0x7f98dc0c7910 msgr2=0x7f98dc139170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:42.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.043+0000 7f98e16ea640 1 -- 172.21.15.46:0/268260748 shutdown_connections 2023-10-30T14:23:42.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.043+0000 7f98e16ea640 1 -- 172.21.15.46:0/268260748 wait complete. 2023-10-30T14:23:42.067 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=1) 2023-10-30T14:23:42.068 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:23:42.068 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:23:42.093 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:23:42.094 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:42.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.271+0000 7fd67b78d640 1 Processor -- start 2023-10-30T14:23:42.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.271+0000 7fd67b78d640 1 -- start start 2023-10-30T14:23:42.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd67b78d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6741538a0 0x7fd67415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:42.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd67b78d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6740cf160 0x7fd6740cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:42.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd67b78d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6740cb800 con 0x7fd6740ca0e0 2023-10-30T14:23:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd67b78d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67415dc30 con 0x7fd6740cf160 2023-10-30T14:23:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd67b78d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6740cb330 con 0x7fd6741538a0 2023-10-30T14:23:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd679d2a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6740cf160 0x7fd6740cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:42.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd679d2a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6740cf160 0x7fd6740cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47460/0 (socket says 172.21.15.46:47460) 2023-10-30T14:23:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd679d2a640 1 -- 172.21.15.46:0/1923175466 learned_addr learned my addr 172.21.15.46:0/1923175466 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd679529640 1 --2- 172.21.15.46:0/1923175466 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6741538a0 0x7fd67415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd679d2a640 1 -- 172.21.15.46:0/1923175466 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6741538a0 msgr2=0x7fd67415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd679d2a640 1 --2- 172.21.15.46:0/1923175466 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6741538a0 0x7fd67415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd679d2a640 1 -- 172.21.15.46:0/1923175466 >> v1:172.21.15.154:6789/0 conn(0x7fd6740ca0e0 legacy=0x7fd67415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd679d2a640 1 -- 172.21.15.46:0/1923175466 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6740cad70 con 0x7fd6740cf160 2023-10-30T14:23:42.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.275+0000 7fd679529640 1 --2- 172.21.15.46:0/1923175466 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6741538a0 0x7fd67415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd679d2a640 1 --2- 172.21.15.46:0/1923175466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6740cf160 0x7fd6740cf590 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009e50 tx=0x7fd66c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9449b38e1431f4b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd66bfff640 1 -- 172.21.15.46:0/1923175466 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c0af400 con 0x7fd6740cf160 2023-10-30T14:23:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd66bfff640 1 -- 172.21.15.46:0/1923175466 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c0ab050 con 0x7fd6740cf160 2023-10-30T14:23:42.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd66bfff640 1 -- 172.21.15.46:0/1923175466 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c0af780 con 0x7fd6740cf160 2023-10-30T14:23:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd67b78d640 1 -- 172.21.15.46:0/1923175466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6740cf160 msgr2=0x7fd6740cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd67b78d640 1 --2- 172.21.15.46:0/1923175466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6740cf160 0x7fd6740cf590 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009e50 tx=0x7fd66c01f1f0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd67b78d640 1 -- 172.21.15.46:0/1923175466 shutdown_connections 2023-10-30T14:23:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd67b78d640 1 --2- 172.21.15.46:0/1923175466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6740cf160 0x7fd6740cf590 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd67b78d640 1 --2- 172.21.15.46:0/1923175466 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6741538a0 0x7fd67415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd67b78d640 1 -- 172.21.15.46:0/1923175466 >> 172.21.15.46:0/1923175466 conn(0x7fd67406bfe0 msgr2=0x7fd67406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd67b78d640 1 -- 172.21.15.46:0/1923175466 shutdown_connections 2023-10-30T14:23:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd67b78d640 1 -- 172.21.15.46:0/1923175466 wait complete. 2023-10-30T14:23:42.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd67b78d640 1 Processor -- start 2023-10-30T14:23:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.279+0000 7fd67b78d640 1 -- start start 2023-10-30T14:23:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd67b78d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6740ca0e0 0x7fd6740ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd67b78d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6740cf160 0x7fd67415c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd67b78d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6741538a0 0x7fd674144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:42.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd67b78d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6741442e0 con 0x7fd6741538a0 2023-10-30T14:23:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd67b78d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd674143fe0 con 0x7fd6740cf160 2023-10-30T14:23:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd67b78d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67414c570 con 0x7fd6740ca0e0 2023-10-30T14:23:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd679d2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6741538a0 0x7fd674144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd679d2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6741538a0 0x7fd674144770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47464/0 (socket says 172.21.15.46:47464) 2023-10-30T14:23:42.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd679d2a640 1 -- 172.21.15.46:0/415741528 learned_addr learned my addr 172.21.15.46:0/415741528 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd679529640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6740ca0e0 0x7fd6740ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd678d28640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6740cf160 0x7fd67415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd679d2a640 1 -- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6740ca0e0 msgr2=0x7fd6740ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd679d2a640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6740ca0e0 0x7fd6740ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd679d2a640 1 -- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6740cf160 msgr2=0x7fd67415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd679d2a640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6740cf160 0x7fd67415c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd679d2a640 1 -- 172.21.15.46:0/415741528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c009b30 con 0x7fd6741538a0 2023-10-30T14:23:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd679d2a640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6741538a0 0x7fd674144770 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0b2040 tx=0x7fd66c01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd669ffb640 1 -- 172.21.15.46:0/415741528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c0afd30 con 0x7fd6741538a0 2023-10-30T14:23:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd669ffb640 1 -- 172.21.15.46:0/415741528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c0afe90 con 0x7fd6741538a0 2023-10-30T14:23:42.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd67b78d640 1 -- 172.21.15.46:0/415741528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67414c850 con 0x7fd6741538a0 2023-10-30T14:23:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd669ffb640 1 -- 172.21.15.46:0/415741528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c0ba680 con 0x7fd6741538a0 2023-10-30T14:23:42.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd67b78d640 1 -- 172.21.15.46:0/415741528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67414ce10 con 0x7fd6741538a0 2023-10-30T14:23:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd67b78d640 1 -- 172.21.15.46:0/415741528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6740b2a50 con 0x7fd6741538a0 2023-10-30T14:23:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.283+0000 7fd669ffb640 1 -- 172.21.15.46:0/415741528 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd66c0b6040 con 0x7fd6741538a0 2023-10-30T14:23:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.287+0000 7fd669ffb640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6580664a0 0x7fd658068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:42.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.287+0000 7fd669ffb640 1 -- 172.21.15.46:0/415741528 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fd66c122c30 con 0x7fd6741538a0 2023-10-30T14:23:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.287+0000 7fd679529640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6580664a0 0x7fd658068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.287+0000 7fd679529640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6580664a0 0x7fd658068960 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd67000ba10 tx=0x7fd670008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:42.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.291+0000 7fd669ffb640 1 -- 172.21.15.46:0/415741528 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd66c0ef770 con 0x7fd6741538a0 2023-10-30T14:23:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.479+0000 7fd67b78d640 1 -- 172.21.15.46:0/415741528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd674144490 con 0x7fd6741538a0 2023-10-30T14:23:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.479+0000 7fd669ffb640 1 -- 172.21.15.46:0/415741528 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fd674144490 con 0x7fd6741538a0 2023-10-30T14:23:42.486 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:42.486 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":252,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9882,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":251}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:19.162295+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:42.487 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 252 2023-10-30T14:23:42.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.483+0000 7fd67b78d640 1 -- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6580664a0 msgr2=0x7fd658068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.483+0000 7fd67b78d640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6580664a0 0x7fd658068960 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd67000ba10 tx=0x7fd670008040 comp rx=0 tx=0).stop 2023-10-30T14:23:42.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.483+0000 7fd67b78d640 1 -- 172.21.15.46:0/415741528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6741538a0 msgr2=0x7fd674144770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.483+0000 7fd67b78d640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6741538a0 0x7fd674144770 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0b2040 tx=0x7fd66c01bf90 comp rx=0 tx=0).stop 2023-10-30T14:23:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.483+0000 7fd67b78d640 1 -- 172.21.15.46:0/415741528 shutdown_connections 2023-10-30T14:23:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.483+0000 7fd67b78d640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd6580664a0 0x7fd658068960 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.483+0000 7fd67b78d640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6741538a0 0x7fd674144770 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.483+0000 7fd67b78d640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6740cf160 0x7fd67415c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.483+0000 7fd67b78d640 1 --2- 172.21.15.46:0/415741528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6740ca0e0 0x7fd6740ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:42.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.483+0000 7fd67b78d640 1 -- 172.21.15.46:0/415741528 >> 172.21.15.46:0/415741528 conn(0x7fd67406bfe0 msgr2=0x7fd674151320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:42.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.483+0000 7fd67b78d640 1 -- 172.21.15.46:0/415741528 shutdown_connections 2023-10-30T14:23:42.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:42.483+0000 7fd67b78d640 1 -- 172.21.15.46:0/415741528 wait complete. 2023-10-30T14:23:42.507 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:23:44.509 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f194d7a0640 1 Processor -- start 2023-10-30T14:23:44.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f194d7a0640 1 -- start start 2023-10-30T14:23:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f194d7a0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f19480717b0 0x7f194815ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f194d7a0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f19480c6290 0x7f1948161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f194d7a0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194815b060 con 0x7f1948071b90 2023-10-30T14:23:44.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f194d7a0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1948157530 con 0x7f19480717b0 2023-10-30T14:23:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f194d7a0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19480c4ab0 con 0x7f19480c6290 2023-10-30T14:23:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f1947fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f19480717b0 0x7f194815ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f19477fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1948071b90 0x7f19481513c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44892/0 (socket says 172.21.15.46:44892) 2023-10-30T14:23:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f19477fe640 1 -- 172.21.15.46:0/1178071914 learned_addr learned my addr 172.21.15.46:0/1178071914 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:23:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f194cf9f640 1 --2- 172.21.15.46:0/1178071914 >> v2:172.21.15.154:3300/0 conn(0x7f19480c6290 0x7f1948161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f1947fff640 1 -- 172.21.15.46:0/1178071914 >> v2:172.21.15.154:3300/0 conn(0x7f19480c6290 msgr2=0x7f1948161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f1947fff640 1 --2- 172.21.15.46:0/1178071914 >> v2:172.21.15.154:3300/0 conn(0x7f19480c6290 0x7f1948161eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f1947fff640 1 -- 172.21.15.46:0/1178071914 >> v1:172.21.15.46:6789/0 conn(0x7f1948071b90 legacy=0x7f19481513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f1947fff640 1 -- 172.21.15.46:0/1178071914 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19480c4d90 con 0x7f19480717b0 2023-10-30T14:23:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.647+0000 7f194cf9f640 1 --2- 172.21.15.46:0/1178071914 >> v2:172.21.15.154:3300/0 conn(0x7f19480c6290 0x7f1948161eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:44.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.651+0000 7f1947fff640 1 --2- 172.21.15.46:0/1178071914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19480717b0 0x7f194815ab20 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f194806c3d0 tx=0x7f193801ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1fc0df8ae06e38 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.651+0000 7f1946ffd640 1 -- 172.21.15.46:0/1178071914 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19380a8930 con 0x7f19480717b0 2023-10-30T14:23:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.651+0000 7f1946ffd640 1 -- 172.21.15.46:0/1178071914 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19380a8a90 con 0x7f19480717b0 2023-10-30T14:23:44.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.651+0000 7f1946ffd640 1 -- 172.21.15.46:0/1178071914 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19380a8e20 con 0x7f19480717b0 2023-10-30T14:23:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.651+0000 7f194d7a0640 1 -- 172.21.15.46:0/1178071914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19480717b0 msgr2=0x7f194815ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.651+0000 7f194d7a0640 1 --2- 172.21.15.46:0/1178071914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19480717b0 0x7f194815ab20 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f194806c3d0 tx=0x7f193801ec30 comp rx=0 tx=0).stop 2023-10-30T14:23:44.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.651+0000 7f194d7a0640 1 -- 172.21.15.46:0/1178071914 shutdown_connections 2023-10-30T14:23:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.651+0000 7f194d7a0640 1 --2- 172.21.15.46:0/1178071914 >> v2:172.21.15.154:3300/0 conn(0x7f19480c6290 0x7f1948161eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.651+0000 7f194d7a0640 1 --2- 172.21.15.46:0/1178071914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19480717b0 0x7f194815ab20 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.651+0000 7f194d7a0640 1 -- 172.21.15.46:0/1178071914 >> 172.21.15.46:0/1178071914 conn(0x7f19480c75d0 msgr2=0x7f19480c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.651+0000 7f194d7a0640 1 -- 172.21.15.46:0/1178071914 shutdown_connections 2023-10-30T14:23:44.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.651+0000 7f194d7a0640 1 -- 172.21.15.46:0/1178071914 wait complete. 2023-10-30T14:23:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194d7a0640 1 Processor -- start 2023-10-30T14:23:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194d7a0640 1 -- start start 2023-10-30T14:23:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194d7a0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19480717b0 0x7f194814b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194d7a0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1948071b90 0x7f1948159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194d7a0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19480c6290 0x7f19481f2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194d7a0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194815ab70 con 0x7f19480c6290 2023-10-30T14:23:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194d7a0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19481eedc0 con 0x7f1948071b90 2023-10-30T14:23:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194d7a0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19480bcef0 con 0x7f19480717b0 2023-10-30T14:23:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194cf9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19480c6290 0x7f19481f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194cf9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19480c6290 0x7f19481f2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47468/0 (socket says 172.21.15.46:47468) 2023-10-30T14:23:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194cf9f640 1 -- 172.21.15.46:0/3354110402 learned_addr learned my addr 172.21.15.46:0/3354110402 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f19477fe640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1948071b90 0x7f1948159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f1947fff640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19480717b0 0x7f194814b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194cf9f640 1 -- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19480717b0 msgr2=0x7f194814b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194cf9f640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19480717b0 0x7f194814b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194cf9f640 1 -- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1948071b90 msgr2=0x7f1948159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194cf9f640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1948071b90 0x7f1948159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194cf9f640 1 -- 172.21.15.46:0/3354110402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19380093b0 con 0x7f19480c6290 2023-10-30T14:23:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f1947fff640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19480717b0 0x7f194814b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194cf9f640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19480c6290 0x7f19481f2330 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f1940007df0 tx=0x7f194000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:44.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f1944ff9640 1 -- 172.21.15.46:0/3354110402 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19400039a0 con 0x7f19480c6290 2023-10-30T14:23:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f1944ff9640 1 -- 172.21.15.46:0/3354110402 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194000f040 con 0x7f19480c6290 2023-10-30T14:23:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f1944ff9640 1 -- 172.21.15.46:0/3354110402 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940003d40 con 0x7f19480c6290 2023-10-30T14:23:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194d7a0640 1 -- 172.21.15.46:0/3354110402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19481eefd0 con 0x7f19480c6290 2023-10-30T14:23:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.655+0000 7f194d7a0640 1 -- 172.21.15.46:0/3354110402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19481ef590 con 0x7f19480c6290 2023-10-30T14:23:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.659+0000 7f194d7a0640 1 -- 172.21.15.46:0/3354110402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19481ef7b0 con 0x7f19480c6290 2023-10-30T14:23:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.659+0000 7f1944ff9640 1 -- 172.21.15.46:0/3354110402 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1940004030 con 0x7f19480c6290 2023-10-30T14:23:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.659+0000 7f1944ff9640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19340664a0 0x7f1934068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.659+0000 7f1944ff9640 1 -- 172.21.15.46:0/3354110402 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f1940084c90 con 0x7f19480c6290 2023-10-30T14:23:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.659+0000 7f1947fff640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19340664a0 0x7f1934068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.663+0000 7f1947fff640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19340664a0 0x7f1934068960 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f19380b2000 tx=0x7f1938008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:44.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.663+0000 7f1944ff9640 1 -- 172.21.15.46:0/3354110402 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f19400515e0 con 0x7f19480c6290 2023-10-30T14:23:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.855+0000 7f194d7a0640 1 -- 172.21.15.46:0/3354110402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19481576f0 con 0x7f19480c6290 2023-10-30T14:23:44.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.855+0000 7f1944ff9640 1 -- 172.21.15.46:0/3354110402 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f194001a070 con 0x7f19480c6290 2023-10-30T14:23:44.862 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:44.862 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":252,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9882,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":251}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:19.162295+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:44.862 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 252 2023-10-30T14:23:44.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.859+0000 7f194d7a0640 1 -- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19340664a0 msgr2=0x7f1934068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.859+0000 7f194d7a0640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19340664a0 0x7f1934068960 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f19380b2000 tx=0x7f1938008040 comp rx=0 tx=0).stop 2023-10-30T14:23:44.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.859+0000 7f194d7a0640 1 -- 172.21.15.46:0/3354110402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19480c6290 msgr2=0x7f19481f2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:44.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.859+0000 7f194d7a0640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19480c6290 0x7f19481f2330 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f1940007df0 tx=0x7f194000d4a0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.859+0000 7f194d7a0640 1 -- 172.21.15.46:0/3354110402 shutdown_connections 2023-10-30T14:23:44.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.859+0000 7f194d7a0640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f19340664a0 0x7f1934068960 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.859+0000 7f194d7a0640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19480c6290 0x7f19481f2330 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.859+0000 7f194d7a0640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1948071b90 0x7f1948159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.859+0000 7f194d7a0640 1 --2- 172.21.15.46:0/3354110402 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19480717b0 0x7f194814b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:44.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.859+0000 7f194d7a0640 1 -- 172.21.15.46:0/3354110402 >> 172.21.15.46:0/3354110402 conn(0x7f19480c75d0 msgr2=0x7f194806ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:44.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.859+0000 7f194d7a0640 1 -- 172.21.15.46:0/3354110402 shutdown_connections 2023-10-30T14:23:44.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:44.859+0000 7f194d7a0640 1 -- 172.21.15.46:0/3354110402 wait complete. 2023-10-30T14:23:44.883 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:23:46.885 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:47.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.143+0000 7f15763f5640 1 Processor -- start 2023-10-30T14:23:47.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.147+0000 7f15763f5640 1 -- start start 2023-10-30T14:23:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.147+0000 7f15763f5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1570070740 0x7f15700c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.147+0000 7f15763f5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f157015e700 0x7f157006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.147+0000 7f15763f5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15700c5810 con 0x7f15701580e0 2023-10-30T14:23:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.147+0000 7f15763f5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f157015eb70 con 0x7f157015e700 2023-10-30T14:23:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.147+0000 7f15763f5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15700c6bb0 con 0x7f1570070740 2023-10-30T14:23:47.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.147+0000 7f156ffff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f15701580e0 0x7f1570069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:41860/0 (socket says 172.21.15.46:41860) 2023-10-30T14:23:47.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.147+0000 7f156ffff640 1 -- 172.21.15.46:0/1869320941 learned_addr learned my addr 172.21.15.46:0/1869320941 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:23:47.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.147+0000 7f156f7fe640 1 --2- 172.21.15.46:0/1869320941 >> v2:172.21.15.154:3300/0 conn(0x7f1570070740 0x7f15700c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.151+0000 7f1574992640 1 --2- 172.21.15.46:0/1869320941 >> v2:172.21.15.46:3300/0 conn(0x7f157015e700 0x7f157006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.151+0000 7f156effd640 1 -- 172.21.15.46:0/1869320941 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726276954 0 0) 0x7f15700c5810 con 0x7f15701580e0 2023-10-30T14:23:47.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.151+0000 7f156effd640 1 -- 172.21.15.46:0/1869320941 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1558003530 con 0x7f15701580e0 2023-10-30T14:23:47.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.151+0000 7f156f7fe640 1 -- 172.21.15.46:0/1869320941 >> v2:172.21.15.46:3300/0 conn(0x7f157015e700 msgr2=0x7f157006a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:47.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.151+0000 7f156effd640 1 -- 172.21.15.46:0/1869320941 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 728325746 0 0) 0x7f1558003530 con 0x7f15701580e0 2023-10-30T14:23:47.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.151+0000 7f156f7fe640 1 --2- 172.21.15.46:0/1869320941 >> v2:172.21.15.46:3300/0 conn(0x7f157015e700 0x7f157006a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.151+0000 7f156f7fe640 1 -- 172.21.15.46:0/1869320941 >> v1:172.21.15.154:6789/0 conn(0x7f15701580e0 legacy=0x7f1570069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:47.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.151+0000 7f156f7fe640 1 -- 172.21.15.46:0/1869320941 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15700c6e70 con 0x7f1570070740 2023-10-30T14:23:47.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.151+0000 7f1574992640 1 --2- 172.21.15.46:0/1869320941 >> v2:172.21.15.46:3300/0 conn(0x7f157015e700 0x7f157006a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:23:47.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.159+0000 7f156f7fe640 1 --2- 172.21.15.46:0/1869320941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1570070740 0x7f15700c6540 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f157006b1a0 tx=0x7f156801c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fdaf285df6667b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.159+0000 7f156effd640 1 -- 172.21.15.46:0/1869320941 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1564001680 con 0x7f1570070740 2023-10-30T14:23:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.159+0000 7f156effd640 1 -- 172.21.15.46:0/1869320941 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15680a5560 con 0x7f1570070740 2023-10-30T14:23:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.159+0000 7f156effd640 1 -- 172.21.15.46:0/1869320941 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15680a58f0 con 0x7f1570070740 2023-10-30T14:23:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.163+0000 7f15763f5640 1 -- 172.21.15.46:0/1869320941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1570070740 msgr2=0x7f15700c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.163+0000 7f15763f5640 1 --2- 172.21.15.46:0/1869320941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1570070740 0x7f15700c6540 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f157006b1a0 tx=0x7f156801c850 comp rx=0 tx=0).stop 2023-10-30T14:23:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.163+0000 7f15763f5640 1 -- 172.21.15.46:0/1869320941 shutdown_connections 2023-10-30T14:23:47.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.163+0000 7f15763f5640 1 --2- 172.21.15.46:0/1869320941 >> v2:172.21.15.46:3300/0 conn(0x7f157015e700 0x7f157006a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.163+0000 7f15763f5640 1 --2- 172.21.15.46:0/1869320941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1570070740 0x7f15700c6540 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.163+0000 7f15763f5640 1 -- 172.21.15.46:0/1869320941 >> 172.21.15.46:0/1869320941 conn(0x7f15700c7910 msgr2=0x7f15700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.163+0000 7f15763f5640 1 -- 172.21.15.46:0/1869320941 shutdown_connections 2023-10-30T14:23:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.167+0000 7f15763f5640 1 -- 172.21.15.46:0/1869320941 wait complete. 2023-10-30T14:23:47.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.167+0000 7f15763f5640 1 Processor -- start 2023-10-30T14:23:47.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.167+0000 7f15763f5640 1 -- start start 2023-10-30T14:23:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f15763f5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1570070740 0x7f15701420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f15763f5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15701580e0 0x7f157013c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f15763f5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f157015e700 0x7f157013c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f15763f5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15700c8f90 con 0x7f15701580e0 2023-10-30T14:23:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f15763f5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15700c9760 con 0x7f1570070740 2023-10-30T14:23:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f15763f5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15700ca320 con 0x7f157015e700 2023-10-30T14:23:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f156ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1570070740 0x7f15701420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f156ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1570070740 0x7f15701420b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39678/0 (socket says 172.21.15.46:39678) 2023-10-30T14:23:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f156ffff640 1 -- 172.21.15.46:0/760583908 learned_addr learned my addr 172.21.15.46:0/760583908 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:47.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f1574992640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f157015e700 0x7f157013c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f156f7fe640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15701580e0 0x7f157013c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f1574992640 1 -- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1570070740 msgr2=0x7f15701420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:47.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f1574992640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1570070740 0x7f15701420b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f1574992640 1 -- 172.21.15.46:0/760583908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15701580e0 msgr2=0x7f157013c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f1574992640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15701580e0 0x7f157013c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.171+0000 7f1574992640 1 -- 172.21.15.46:0/760583908 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1568009a70 con 0x7f157015e700 2023-10-30T14:23:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.175+0000 7f1574992640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f157015e700 0x7f157013c8e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f156000c5f0 tx=0x7f156000cab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:47.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.175+0000 7f156cff9640 1 -- 172.21.15.46:0/760583908 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1560011800 con 0x7f157015e700 2023-10-30T14:23:47.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.175+0000 7f156cff9640 1 -- 172.21.15.46:0/760583908 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1560011980 con 0x7f157015e700 2023-10-30T14:23:47.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.175+0000 7f156cff9640 1 -- 172.21.15.46:0/760583908 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1560011d30 con 0x7f157015e700 2023-10-30T14:23:47.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.175+0000 7f15763f5640 1 -- 172.21.15.46:0/760583908 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15700caa10 con 0x7f157015e700 2023-10-30T14:23:47.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.175+0000 7f15763f5640 1 -- 172.21.15.46:0/760583908 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1570140ed0 con 0x7f157015e700 2023-10-30T14:23:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.179+0000 7f15763f5640 1 -- 172.21.15.46:0/760583908 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15700caba0 con 0x7f157015e700 2023-10-30T14:23:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.179+0000 7f156cff9640 1 -- 172.21.15.46:0/760583908 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1560022350 con 0x7f157015e700 2023-10-30T14:23:47.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.179+0000 7f156cff9640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f155c0664f0 0x7f155c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:47.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.179+0000 7f156cff9640 1 -- 172.21.15.46:0/760583908 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f1560084980 con 0x7f157015e700 2023-10-30T14:23:47.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.183+0000 7f156ffff640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f155c0664f0 0x7f155c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:47.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.183+0000 7f156ffff640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f155c0664f0 0x7f155c0689b0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f1564001510 tx=0x7f15640014a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.191+0000 7f156cff9640 1 -- 172.21.15.46:0/760583908 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1560051310 con 0x7f157015e700 2023-10-30T14:23:47.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.583+0000 7f15763f5640 1 -- 172.21.15.46:0/760583908 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f157004e170 con 0x7f157015e700 2023-10-30T14:23:47.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.583+0000 7f156cff9640 1 -- 172.21.15.46:0/760583908 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f1560054930 con 0x7f157015e700 2023-10-30T14:23:47.587 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:47.588 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":252,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9882,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":251}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:19.162295+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:47.588 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 252 2023-10-30T14:23:47.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.587+0000 7f15763f5640 1 -- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f155c0664f0 msgr2=0x7f155c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.587+0000 7f15763f5640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f155c0664f0 0x7f155c0689b0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f1564001510 tx=0x7f15640014a0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.587+0000 7f15763f5640 1 -- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f157015e700 msgr2=0x7f157013c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.587+0000 7f15763f5640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f157015e700 0x7f157013c8e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f156000c5f0 tx=0x7f156000cab0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.587+0000 7f15763f5640 1 -- 172.21.15.46:0/760583908 shutdown_connections 2023-10-30T14:23:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.587+0000 7f15763f5640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f155c0664f0 0x7f155c0689b0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.587+0000 7f15763f5640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f157015e700 0x7f157013c8e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.587+0000 7f15763f5640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15701580e0 0x7f157013c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.587+0000 7f15763f5640 1 --2- 172.21.15.46:0/760583908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1570070740 0x7f15701420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.587+0000 7f15763f5640 1 -- 172.21.15.46:0/760583908 >> 172.21.15.46:0/760583908 conn(0x7f15700c7910 msgr2=0x7f1570155d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.587+0000 7f15763f5640 1 -- 172.21.15.46:0/760583908 shutdown_connections 2023-10-30T14:23:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:47.587+0000 7f15763f5640 1 -- 172.21.15.46:0/760583908 wait complete. 2023-10-30T14:23:47.628 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:23:49.629 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa95890e640 1 Processor -- start 2023-10-30T14:23:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa95890e640 1 -- start start 2023-10-30T14:23:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa95890e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa954150920 0x7fa95406d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa95890e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa9540cb7e0 0x7fa954156680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa95890e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9540c68b0 con 0x7fa9540c2570 2023-10-30T14:23:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa95890e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa954150d00 con 0x7fa9540cb7e0 2023-10-30T14:23:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa95890e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95406d960 con 0x7fa954150920 2023-10-30T14:23:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa952d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa9540cb7e0 0x7fa954156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa952d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa9540cb7e0 0x7fa954156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48602/0 (socket says 172.21.15.46:48602) 2023-10-30T14:23:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa952d9d640 1 -- 172.21.15.46:0/998421063 learned_addr learned my addr 172.21.15.46:0/998421063 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa95259c640 1 --2- 172.21.15.46:0/998421063 >> v2:172.21.15.154:3300/0 conn(0x7fa954150920 0x7fa95406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:49.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa952d9d640 1 -- 172.21.15.46:0/998421063 >> v2:172.21.15.154:3300/0 conn(0x7fa954150920 msgr2=0x7fa95406d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:49.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa952d9d640 1 --2- 172.21.15.46:0/998421063 >> v2:172.21.15.154:3300/0 conn(0x7fa954150920 0x7fa95406d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa952d9d640 1 -- 172.21.15.46:0/998421063 >> v1:172.21.15.154:6789/0 conn(0x7fa9540c2570 legacy=0x7fa9540cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:49.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.771+0000 7fa952d9d640 1 -- 172.21.15.46:0/998421063 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9540c2e90 con 0x7fa9540cb7e0 2023-10-30T14:23:49.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.775+0000 7fa952d9d640 1 --2- 172.21.15.46:0/998421063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9540cb7e0 0x7fa954156680 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fa95406feb0 tx=0x7fa94c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a96ddf0032facf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.775+0000 7fa951d9b640 1 -- 172.21.15.46:0/998421063 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c0a5780 con 0x7fa9540cb7e0 2023-10-30T14:23:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.775+0000 7fa951d9b640 1 -- 172.21.15.46:0/998421063 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94c0a58e0 con 0x7fa9540cb7e0 2023-10-30T14:23:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.775+0000 7fa951d9b640 1 -- 172.21.15.46:0/998421063 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c0a5c70 con 0x7fa9540cb7e0 2023-10-30T14:23:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.775+0000 7fa95890e640 1 -- 172.21.15.46:0/998421063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9540cb7e0 msgr2=0x7fa954156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.775+0000 7fa95890e640 1 --2- 172.21.15.46:0/998421063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9540cb7e0 0x7fa954156680 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fa95406feb0 tx=0x7fa94c01c9d0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.775+0000 7fa95890e640 1 -- 172.21.15.46:0/998421063 shutdown_connections 2023-10-30T14:23:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.775+0000 7fa95890e640 1 --2- 172.21.15.46:0/998421063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9540cb7e0 0x7fa954156680 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.775+0000 7fa95890e640 1 --2- 172.21.15.46:0/998421063 >> v2:172.21.15.154:3300/0 conn(0x7fa954150920 0x7fa95406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.775+0000 7fa95890e640 1 -- 172.21.15.46:0/998421063 >> 172.21.15.46:0/998421063 conn(0x7fa95406dfd0 msgr2=0x7fa95406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.775+0000 7fa95890e640 1 -- 172.21.15.46:0/998421063 shutdown_connections 2023-10-30T14:23:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.775+0000 7fa95890e640 1 -- 172.21.15.46:0/998421063 wait complete. 2023-10-30T14:23:49.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95890e640 1 Processor -- start 2023-10-30T14:23:49.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95890e640 1 -- start start 2023-10-30T14:23:49.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95890e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9540c2570 0x7fa954071900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:49.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95890e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9540cb7e0 0x7fa954151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:49.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95890e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa954150920 0x7fa9541515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95890e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95406a7f0 con 0x7fa9540c2570 2023-10-30T14:23:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95890e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa954159730 con 0x7fa954150920 2023-10-30T14:23:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95890e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95406ecc0 con 0x7fa9540cb7e0 2023-10-30T14:23:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9540c2570 0x7fa954071900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9540c2570 0x7fa954071900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48614/0 (socket says 172.21.15.46:48614) 2023-10-30T14:23:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95259c640 1 -- 172.21.15.46:0/464706792 learned_addr learned my addr 172.21.15.46:0/464706792 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa94bfff640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9540cb7e0 0x7fa954151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa952d9d640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa954150920 0x7fa9541515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95259c640 1 -- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9540cb7e0 msgr2=0x7fa954151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95259c640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9540cb7e0 0x7fa954151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95259c640 1 -- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa954150920 msgr2=0x7fa9541515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95259c640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa954150920 0x7fa9541515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95259c640 1 -- 172.21.15.46:0/464706792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c009b30 con 0x7fa9540c2570 2023-10-30T14:23:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95259c640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9540c2570 0x7fa954071900 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fa94400d800 tx=0x7fa94400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa94affd640 1 -- 172.21.15.46:0/464706792 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94400b6c0 con 0x7fa9540c2570 2023-10-30T14:23:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa94affd640 1 -- 172.21.15.46:0/464706792 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94400b820 con 0x7fa9540c2570 2023-10-30T14:23:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95890e640 1 -- 172.21.15.46:0/464706792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa954069f30 con 0x7fa9540c2570 2023-10-30T14:23:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa94affd640 1 -- 172.21.15.46:0/464706792 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94400bb90 con 0x7fa9540c2570 2023-10-30T14:23:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.779+0000 7fa95890e640 1 -- 172.21.15.46:0/464706792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa954158e00 con 0x7fa9540c2570 2023-10-30T14:23:49.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.783+0000 7fa95890e640 1 -- 172.21.15.46:0/464706792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa954072560 con 0x7fa9540c2570 2023-10-30T14:23:49.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.783+0000 7fa94affd640 1 -- 172.21.15.46:0/464706792 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa94400bcf0 con 0x7fa9540c2570 2023-10-30T14:23:49.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.783+0000 7fa94affd640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9340664d0 0x7fa934068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.783+0000 7fa94affd640 1 -- 172.21.15.46:0/464706792 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fa944084d60 con 0x7fa9540c2570 2023-10-30T14:23:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.787+0000 7fa94bfff640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9340664d0 0x7fa934068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.787+0000 7fa94bfff640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9340664d0 0x7fa934068990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fa954156b60 tx=0x7fa93c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.791+0000 7fa94affd640 1 -- 172.21.15.46:0/464706792 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa944051780 con 0x7fa9540c2570 2023-10-30T14:23:49.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.983+0000 7fa95890e640 1 -- 172.21.15.46:0/464706792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa95404e170 con 0x7fa9540c2570 2023-10-30T14:23:49.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa94affd640 1 -- 172.21.15.46:0/464706792 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) v1 ==== 95+0+6523 (secure 0 0 0) 0x7fa944054da0 con 0x7fa9540c2570 2023-10-30T14:23:49.988 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:49.988 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":252,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9882,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":251}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:19.162295+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:49.988 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 252 2023-10-30T14:23:49.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa95890e640 1 -- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9340664d0 msgr2=0x7fa934068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:49.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa95890e640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9340664d0 0x7fa934068990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fa954156b60 tx=0x7fa93c009040 comp rx=0 tx=0).stop 2023-10-30T14:23:49.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa95890e640 1 -- 172.21.15.46:0/464706792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9540c2570 msgr2=0x7fa954071900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:49.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa95890e640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9540c2570 0x7fa954071900 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fa94400d800 tx=0x7fa94400dcc0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa95890e640 1 -- 172.21.15.46:0/464706792 shutdown_connections 2023-10-30T14:23:49.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa95890e640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa9340664d0 0x7fa934068990 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa95890e640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa954150920 0x7fa9541515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa95890e640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9540cb7e0 0x7fa954151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa95890e640 1 --2- 172.21.15.46:0/464706792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9540c2570 0x7fa954071900 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:49.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa95890e640 1 -- 172.21.15.46:0/464706792 >> 172.21.15.46:0/464706792 conn(0x7fa95406dfd0 msgr2=0x7fa9540ca330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:49.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa95890e640 1 -- 172.21.15.46:0/464706792 shutdown_connections 2023-10-30T14:23:49.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:49.987+0000 7fa95890e640 1 -- 172.21.15.46:0/464706792 wait complete. 2023-10-30T14:23:50.010 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:23:52.011 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:52.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.151+0000 7f647d217640 1 Processor -- start 2023-10-30T14:23:52.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.151+0000 7f647d217640 1 -- start start 2023-10-30T14:23:52.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647d217640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f64781538a0 0x7f647815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:52.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647d217640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64780ca0e0 0x7f647815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:52.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647d217640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64780cf160 0x7f64780cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:52.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647d217640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64780cb800 con 0x7f64780cf160 2023-10-30T14:23:52.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647d217640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647815dc30 con 0x7f64781538a0 2023-10-30T14:23:52.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647d217640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64780cb330 con 0x7f64780ca0e0 2023-10-30T14:23:52.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647759e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64780cf160 0x7f64780cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:52.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f6476d9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f64781538a0 0x7f647815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:52.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647759e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64780cf160 0x7f64780cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48624/0 (socket says 172.21.15.46:48624) 2023-10-30T14:23:52.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647759e640 1 -- 172.21.15.46:0/4262481175 learned_addr learned my addr 172.21.15.46:0/4262481175 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:52.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647659c640 1 --2- 172.21.15.46:0/4262481175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64780ca0e0 0x7f647815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:52.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647759e640 1 -- 172.21.15.46:0/4262481175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64780ca0e0 msgr2=0x7f647815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:52.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647759e640 1 --2- 172.21.15.46:0/4262481175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64780ca0e0 0x7f647815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647759e640 1 -- 172.21.15.46:0/4262481175 >> v2:172.21.15.154:3300/0 conn(0x7f64781538a0 msgr2=0x7f647815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:52.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647759e640 1 --2- 172.21.15.46:0/4262481175 >> v2:172.21.15.154:3300/0 conn(0x7f64781538a0 0x7f647815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.155+0000 7f647759e640 1 -- 172.21.15.46:0/4262481175 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64780cad70 con 0x7f64780cf160 2023-10-30T14:23:52.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f647759e640 1 --2- 172.21.15.46:0/4262481175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64780cf160 0x7f64780cf590 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f64780cfd10 tx=0x7f647001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d1ed5ca92741eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f6475d9b640 1 -- 172.21.15.46:0/4262481175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700af400 con 0x7f64780cf160 2023-10-30T14:23:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f6475d9b640 1 -- 172.21.15.46:0/4262481175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64700ab050 con 0x7f64780cf160 2023-10-30T14:23:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f6475d9b640 1 -- 172.21.15.46:0/4262481175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700af780 con 0x7f64780cf160 2023-10-30T14:23:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f647d217640 1 -- 172.21.15.46:0/4262481175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64780cf160 msgr2=0x7f64780cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f647d217640 1 --2- 172.21.15.46:0/4262481175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64780cf160 0x7f64780cf590 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f64780cfd10 tx=0x7f647001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f647d217640 1 -- 172.21.15.46:0/4262481175 shutdown_connections 2023-10-30T14:23:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f647d217640 1 --2- 172.21.15.46:0/4262481175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64780cf160 0x7f64780cf590 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f647d217640 1 --2- 172.21.15.46:0/4262481175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64780ca0e0 0x7f647815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f647d217640 1 --2- 172.21.15.46:0/4262481175 >> v2:172.21.15.154:3300/0 conn(0x7f64781538a0 0x7f647815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f647d217640 1 -- 172.21.15.46:0/4262481175 >> 172.21.15.46:0/4262481175 conn(0x7f647806bfe0 msgr2=0x7f647806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f647d217640 1 -- 172.21.15.46:0/4262481175 shutdown_connections 2023-10-30T14:23:52.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.159+0000 7f647d217640 1 -- 172.21.15.46:0/4262481175 wait complete. 2023-10-30T14:23:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.163+0000 7f647d217640 1 Processor -- start 2023-10-30T14:23:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.163+0000 7f647d217640 1 -- start start 2023-10-30T14:23:52.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.163+0000 7f647d217640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64780ca0e0 0x7f64781e2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:52.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.163+0000 7f647d217640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64780cf160 0x7f64780cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:52.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.163+0000 7f647d217640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64781538a0 0x7f64780b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:52.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.163+0000 7f647d217640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64781e31e0 con 0x7f64781538a0 2023-10-30T14:23:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.163+0000 7f647d217640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64781e34b0 con 0x7f64780ca0e0 2023-10-30T14:23:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.163+0000 7f647d217640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64781ef390 con 0x7f64780cf160 2023-10-30T14:23:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.163+0000 7f647759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64781538a0 0x7f64780b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f6476d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64780ca0e0 0x7f64781e2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647659c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64780cf160 0x7f64780cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64781538a0 0x7f64780b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48640/0 (socket says 172.21.15.46:48640) 2023-10-30T14:23:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647659c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64780cf160 0x7f64780cd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:56088/0 (socket says 172.21.15.46:56088) 2023-10-30T14:23:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f6476d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64780ca0e0 0x7f64781e2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:39712/0 (socket says 172.21.15.46:39712) 2023-10-30T14:23:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647759e640 1 -- 172.21.15.46:0/2136870941 learned_addr learned my addr 172.21.15.46:0/2136870941 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647759e640 1 -- 172.21.15.46:0/2136870941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64780cf160 msgr2=0x7f64780cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647759e640 1 --2- 172.21.15.46:0/2136870941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64780cf160 0x7f64780cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647759e640 1 -- 172.21.15.46:0/2136870941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64780ca0e0 msgr2=0x7f64781e2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647759e640 1 --2- 172.21.15.46:0/2136870941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64780ca0e0 0x7f64781e2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647759e640 1 -- 172.21.15.46:0/2136870941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6470009b30 con 0x7f64781538a0 2023-10-30T14:23:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647759e640 1 --2- 172.21.15.46:0/2136870941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64781538a0 0x7f64780b4000 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f64700b2040 tx=0x7f647001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:52.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f645f7fe640 1 -- 172.21.15.46:0/2136870941 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700afc60 con 0x7f64781538a0 2023-10-30T14:23:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f645f7fe640 1 -- 172.21.15.46:0/2136870941 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64700afde0 con 0x7f64781538a0 2023-10-30T14:23:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f645f7fe640 1 -- 172.21.15.46:0/2136870941 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700ba660 con 0x7f64781538a0 2023-10-30T14:23:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647d217640 1 -- 172.21.15.46:0/2136870941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64780b3240 con 0x7f64781538a0 2023-10-30T14:23:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647d217640 1 -- 172.21.15.46:0/2136870941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64780b3800 con 0x7f64781538a0 2023-10-30T14:23:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f647d217640 1 -- 172.21.15.46:0/2136870941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64780b2a50 con 0x7f64781538a0 2023-10-30T14:23:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.167+0000 7f645f7fe640 1 -- 172.21.15.46:0/2136870941 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64700b6040 con 0x7f64781538a0 2023-10-30T14:23:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.171+0000 7f645f7fe640 1 --2- 172.21.15.46:0/2136870941 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64640664a0 0x7f6464068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.171+0000 7f645f7fe640 1 -- 172.21.15.46:0/2136870941 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8592+0+0 (secure 0 0 0) 0x7f6470122890 con 0x7f64781538a0 2023-10-30T14:23:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.171+0000 7f6476d9d640 1 --2- 172.21.15.46:0/2136870941 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64640664a0 0x7f6464068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:52.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.171+0000 7f6476d9d640 1 --2- 172.21.15.46:0/2136870941 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64640664a0 0x7f6464068960 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f6468009ac0 tx=0x7f6468009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:52.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.175+0000 7f645f7fe640 1 -- 172.21.15.46:0/2136870941 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64700ef2c0 con 0x7f64781538a0 2023-10-30T14:23:52.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 -- 172.21.15.46:0/2136870941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f647804e170 con 0x7f64781538a0 2023-10-30T14:23:52.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f645f7fe640 1 -- 172.21.15.46:0/2136870941 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f6470009510 con 0x7f64781538a0 2023-10-30T14:23:52.377 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:52.378 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":252,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9882,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":251}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:19.162295+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:52.378 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 252 2023-10-30T14:23:52.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 -- 172.21.15.46:0/2136870941 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64640664a0 msgr2=0x7f6464068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:52.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 --2- 172.21.15.46:0/2136870941 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64640664a0 0x7f6464068960 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f6468009ac0 tx=0x7f6468009190 comp rx=0 tx=0).stop 2023-10-30T14:23:52.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 -- 172.21.15.46:0/2136870941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64781538a0 msgr2=0x7f64780b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:52.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 --2- 172.21.15.46:0/2136870941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64781538a0 0x7f64780b4000 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f64700b2040 tx=0x7f647001bf60 comp rx=0 tx=0).stop 2023-10-30T14:23:52.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 -- 172.21.15.46:0/2136870941 shutdown_connections 2023-10-30T14:23:52.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 --2- 172.21.15.46:0/2136870941 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f64640664a0 0x7f6464068960 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 --2- 172.21.15.46:0/2136870941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64781538a0 0x7f64780b4000 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 --2- 172.21.15.46:0/2136870941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64780cf160 0x7f64780cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 --2- 172.21.15.46:0/2136870941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64780ca0e0 0x7f64781e2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 -- 172.21.15.46:0/2136870941 >> 172.21.15.46:0/2136870941 conn(0x7f647806bfe0 msgr2=0x7f6478151880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 -- 172.21.15.46:0/2136870941 shutdown_connections 2023-10-30T14:23:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:52.375+0000 7f647d217640 1 -- 172.21.15.46:0/2136870941 wait complete. 2023-10-30T14:23:52.398 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:23:54.399 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:54.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.539+0000 7f1a719c5640 1 Processor -- start 2023-10-30T14:23:54.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.539+0000 7f1a719c5640 1 -- start start 2023-10-30T14:23:54.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a719c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a6c159e60 0x7f1a6c06f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:54.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a719c5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a6c0bfec0 0x7f1a6c0c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:54.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a719c5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c06fa80 con 0x7f1a6c06a720 2023-10-30T14:23:54.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a719c5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6c071290 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a719c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c071570 con 0x7f1a6c159e60 2023-10-30T14:23:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a6affd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a6c159e60 0x7f1a6c06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a63fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a6c0bfec0 0x7f1a6c0c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a6b7fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f1a6c06a720 0x7f1a6c155d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:41874/0 (socket says 172.21.15.46:41874) 2023-10-30T14:23:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a6b7fe640 1 -- 172.21.15.46:0/2568347165 learned_addr learned my addr 172.21.15.46:0/2568347165 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:23:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a6affd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a6c159e60 0x7f1a6c06f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:56102/0 (socket says 172.21.15.46:56102) 2023-10-30T14:23:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a6a7fc640 1 -- 172.21.15.46:0/2568347165 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336125264 0 0) 0x7f1a6c06fa80 con 0x7f1a6c06a720 2023-10-30T14:23:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a6a7fc640 1 -- 172.21.15.46:0/2568347165 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a58003530 con 0x7f1a6c06a720 2023-10-30T14:23:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a63fff640 1 -- 172.21.15.46:0/2568347165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a6c159e60 msgr2=0x7f1a6c06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:54.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a63fff640 1 --2- 172.21.15.46:0/2568347165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a6c159e60 0x7f1a6c06f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a63fff640 1 -- 172.21.15.46:0/2568347165 >> v1:172.21.15.154:6789/0 conn(0x7f1a6c06a720 legacy=0x7f1a6c155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a63fff640 1 -- 172.21.15.46:0/2568347165 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c071850 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.543+0000 7f1a6affd640 1 --2- 172.21.15.46:0/2568347165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a6c159e60 0x7f1a6c06f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:23:54.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.547+0000 7f1a63fff640 1 --2- 172.21.15.46:0/2568347165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a6c0bfec0 0x7f1a6c0c1a40 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1a540097c0 tx=0x7f1a5401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec3e436c268385ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.547+0000 7f1a6a7fc640 1 -- 172.21.15.46:0/2568347165 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a540a8970 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.547+0000 7f1a6a7fc640 1 -- 172.21.15.46:0/2568347165 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a540a8ad0 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.547+0000 7f1a6a7fc640 1 -- 172.21.15.46:0/2568347165 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a540a8e40 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.547+0000 7f1a719c5640 1 -- 172.21.15.46:0/2568347165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a6c0bfec0 msgr2=0x7f1a6c0c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.547+0000 7f1a719c5640 1 --2- 172.21.15.46:0/2568347165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a6c0bfec0 0x7f1a6c0c1a40 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1a540097c0 tx=0x7f1a5401ec70 comp rx=0 tx=0).stop 2023-10-30T14:23:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.547+0000 7f1a719c5640 1 -- 172.21.15.46:0/2568347165 shutdown_connections 2023-10-30T14:23:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.547+0000 7f1a719c5640 1 --2- 172.21.15.46:0/2568347165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a6c0bfec0 0x7f1a6c0c1a40 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.547+0000 7f1a719c5640 1 --2- 172.21.15.46:0/2568347165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a6c159e60 0x7f1a6c06f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.547+0000 7f1a719c5640 1 -- 172.21.15.46:0/2568347165 >> 172.21.15.46:0/2568347165 conn(0x7f1a6c0c7910 msgr2=0x7f1a6c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.547+0000 7f1a719c5640 1 -- 172.21.15.46:0/2568347165 shutdown_connections 2023-10-30T14:23:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.547+0000 7f1a719c5640 1 -- 172.21.15.46:0/2568347165 wait complete. 2023-10-30T14:23:54.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a719c5640 1 Processor -- start 2023-10-30T14:23:54.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a719c5640 1 -- start start 2023-10-30T14:23:54.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a719c5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a6c06a720 0x7f1a6c1f2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:54.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a719c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a6c0bfec0 0x7f1a6c0c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:54.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a719c5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a6c159e60 0x7f1a6c14f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:54.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a719c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1f0370 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a719c5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1537c0 con 0x7f1a6c159e60 2023-10-30T14:23:54.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a719c5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c154d00 con 0x7f1a6c06a720 2023-10-30T14:23:54.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a63fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a6c0bfec0 0x7f1a6c0c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:54.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a6b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a6c159e60 0x7f1a6c14f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a63fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a6c0bfec0 0x7f1a6c0c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48648/0 (socket says 172.21.15.46:48648) 2023-10-30T14:23:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a63fff640 1 -- 172.21.15.46:0/3880216535 learned_addr learned my addr 172.21.15.46:0/3880216535 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a6affd640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a6c06a720 0x7f1a6c1f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a63fff640 1 -- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a6c06a720 msgr2=0x7f1a6c1f2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a63fff640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a6c06a720 0x7f1a6c1f2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a63fff640 1 -- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a6c159e60 msgr2=0x7f1a6c14f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a63fff640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a6c159e60 0x7f1a6c14f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a63fff640 1 -- 172.21.15.46:0/3880216535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c00a5c0 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a63fff640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a6c0bfec0 0x7f1a6c0c8e70 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f1a54000c10 tx=0x7f1a5401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:54.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.551+0000 7f1a68ff9640 1 -- 172.21.15.46:0/3880216535 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a540044d0 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.555+0000 7f1a68ff9640 1 -- 172.21.15.46:0/3880216535 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a54003620 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.555+0000 7f1a68ff9640 1 -- 172.21.15.46:0/3880216535 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54003990 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.555+0000 7f1a719c5640 1 -- 172.21.15.46:0/3880216535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a54009420 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.555+0000 7f1a719c5640 1 -- 172.21.15.46:0/3880216535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c150740 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.555+0000 7f1a719c5640 1 -- 172.21.15.46:0/3880216535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6c1f04b0 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.555+0000 7f1a68ff9640 1 -- 172.21.15.46:0/3880216535 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1a540b7040 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.555+0000 7f1a68ff9640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a4c0664d0 0x7f1a4c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.559+0000 7f1a68ff9640 1 -- 172.21.15.46:0/3880216535 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8592+0+0 (secure 0 0 0) 0x7f1a54121f80 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.559+0000 7f1a6affd640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a4c0664d0 0x7f1a4c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:54.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.559+0000 7f1a6affd640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a4c0664d0 0x7f1a4c068990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00a590 tx=0x7f1a5c00a2a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:54.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.563+0000 7f1a68ff9640 1 -- 172.21.15.46:0/3880216535 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a6c1f04b0 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.747+0000 7f1a719c5640 1 -- 172.21.15.46:0/3880216535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a6c1f04b0 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a68ff9640 1 -- 172.21.15.46:0/3880216535 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) v1 ==== 95+0+6523 (secure 0 0 0) 0x7f1a6c1f04b0 con 0x7f1a6c0bfec0 2023-10-30T14:23:54.752 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:54.753 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":252,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":9882,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":251}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:19.162295+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9594},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9594":{"gid":9594,"name":"e","rank":1,"incarnation":239,"state":"up:active","state_seq":29,"addr":"172.21.15.46:6837/470164358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":470164358},{"type":"v1","addr":"172.21.15.46:6837","nonce":470164358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:54.753 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 252 2023-10-30T14:23:54.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a719c5640 1 -- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a4c0664d0 msgr2=0x7f1a4c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a719c5640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a4c0664d0 0x7f1a4c068990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00a590 tx=0x7f1a5c00a2a0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a719c5640 1 -- 172.21.15.46:0/3880216535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a6c0bfec0 msgr2=0x7f1a6c0c8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:54.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a719c5640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a6c0bfec0 0x7f1a6c0c8e70 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f1a54000c10 tx=0x7f1a5401ba10 comp rx=0 tx=0).stop 2023-10-30T14:23:54.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a719c5640 1 -- 172.21.15.46:0/3880216535 shutdown_connections 2023-10-30T14:23:54.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a719c5640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1a4c0664d0 0x7f1a4c068990 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a719c5640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a6c159e60 0x7f1a6c14f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a719c5640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a6c0bfec0 0x7f1a6c0c8e70 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a719c5640 1 --2- 172.21.15.46:0/3880216535 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a6c06a720 0x7f1a6c1f2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a719c5640 1 -- 172.21.15.46:0/3880216535 >> 172.21.15.46:0/3880216535 conn(0x7f1a6c0c7910 msgr2=0x7f1a6c0b0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a719c5640 1 -- 172.21.15.46:0/3880216535 shutdown_connections 2023-10-30T14:23:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:54.751+0000 7f1a719c5640 1 -- 172.21.15.46:0/3880216535 wait complete. 2023-10-30T14:23:54.773 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T14:23:54.864 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 30 -eq 48 ] 2023-10-30T14:23:54.865 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.30 2023-10-30T14:23:54.865 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.30 2023-10-30T14:23:56.774 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.915+0000 7fcf36f3e640 1 Processor -- start 2023-10-30T14:23:56.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.915+0000 7fcf36f3e640 1 -- start start 2023-10-30T14:23:56.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf36f3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcf301538a0 0x7fcf3015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:56.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf36f3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf300cf160 0x7fcf300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:56.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf36f3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf300cb800 con 0x7fcf300ca0e0 2023-10-30T14:23:56.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf36f3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3015dc30 con 0x7fcf301538a0 2023-10-30T14:23:56.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf36f3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300cb330 con 0x7fcf300cf160 2023-10-30T14:23:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf34cda640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcf301538a0 0x7fcf3015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf354db640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf300cf160 0x7fcf300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:56.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf354db640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf300cf160 0x7fcf300cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:50990/0 (socket says 172.21.15.46:50990) 2023-10-30T14:23:56.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf354db640 1 -- 172.21.15.46:0/1462269907 learned_addr learned my addr 172.21.15.46:0/1462269907 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:56.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf34cda640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcf301538a0 0x7fcf3015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:51592/0 (socket says 172.21.15.46:51592) 2023-10-30T14:23:56.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf354db640 1 -- 172.21.15.46:0/1462269907 >> v2:172.21.15.154:3300/0 conn(0x7fcf301538a0 msgr2=0x7fcf3015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:56.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf354db640 1 --2- 172.21.15.46:0/1462269907 >> v2:172.21.15.154:3300/0 conn(0x7fcf301538a0 0x7fcf3015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf354db640 1 -- 172.21.15.46:0/1462269907 >> v1:172.21.15.154:6789/0 conn(0x7fcf300ca0e0 legacy=0x7fcf3015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.919+0000 7fcf354db640 1 -- 172.21.15.46:0/1462269907 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf300cad70 con 0x7fcf300cf160 2023-10-30T14:23:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf354db640 1 --2- 172.21.15.46:0/1462269907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf300cf160 0x7fcf300cf590 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009e50 tx=0x7fcf2801f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ab68affc1e9744 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf2f7fe640 1 -- 172.21.15.46:0/1462269907 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf280af400 con 0x7fcf300cf160 2023-10-30T14:23:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf2f7fe640 1 -- 172.21.15.46:0/1462269907 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf280ab050 con 0x7fcf300cf160 2023-10-30T14:23:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf2f7fe640 1 -- 172.21.15.46:0/1462269907 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf280af780 con 0x7fcf300cf160 2023-10-30T14:23:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1462269907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf300cf160 msgr2=0x7fcf300cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf36f3e640 1 --2- 172.21.15.46:0/1462269907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf300cf160 0x7fcf300cf590 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009e50 tx=0x7fcf2801f1f0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1462269907 shutdown_connections 2023-10-30T14:23:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf36f3e640 1 --2- 172.21.15.46:0/1462269907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf300cf160 0x7fcf300cf590 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf36f3e640 1 --2- 172.21.15.46:0/1462269907 >> v2:172.21.15.154:3300/0 conn(0x7fcf301538a0 0x7fcf3015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1462269907 >> 172.21.15.46:0/1462269907 conn(0x7fcf3006bfe0 msgr2=0x7fcf3006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1462269907 shutdown_connections 2023-10-30T14:23:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1462269907 wait complete. 2023-10-30T14:23:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf36f3e640 1 Processor -- start 2023-10-30T14:23:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.923+0000 7fcf36f3e640 1 -- start start 2023-10-30T14:23:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf36f3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf300ca0e0 0x7fcf300ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf36f3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf300cf160 0x7fcf30142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf36f3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf301538a0 0x7fcf30157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf36f3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300ce820 con 0x7fcf301538a0 2023-10-30T14:23:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf36f3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300ceae0 con 0x7fcf300cf160 2023-10-30T14:23:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf36f3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300cedc0 con 0x7fcf300ca0e0 2023-10-30T14:23:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf354db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf301538a0 0x7fcf30157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf354db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf301538a0 0x7fcf30157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42326/0 (socket says 172.21.15.46:42326) 2023-10-30T14:23:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf354db640 1 -- 172.21.15.46:0/1891410062 learned_addr learned my addr 172.21.15.46:0/1891410062 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf2ffff640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf300cf160 0x7fcf30142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf34cda640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf300ca0e0 0x7fcf300ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf354db640 1 -- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf300ca0e0 msgr2=0x7fcf300ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:56.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf354db640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf300ca0e0 0x7fcf300ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf354db640 1 -- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf300cf160 msgr2=0x7fcf30142550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:56.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf354db640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf300cf160 0x7fcf30142550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:56.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf354db640 1 -- 172.21.15.46:0/1891410062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf28009b30 con 0x7fcf301538a0 2023-10-30T14:23:56.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf354db640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf301538a0 0x7fcf30157520 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7fcf2801ce20 tx=0x7fcf2801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf2d7fa640 1 -- 172.21.15.46:0/1891410062 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf280afc80 con 0x7fcf301538a0 2023-10-30T14:23:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf2d7fa640 1 -- 172.21.15.46:0/1891410062 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf280afde0 con 0x7fcf301538a0 2023-10-30T14:23:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf2d7fa640 1 -- 172.21.15.46:0/1891410062 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf280ba660 con 0x7fcf301538a0 2023-10-30T14:23:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1891410062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf301efb80 con 0x7fcf301538a0 2023-10-30T14:23:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1891410062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf301f0140 con 0x7fcf301538a0 2023-10-30T14:23:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1891410062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf300b2a50 con 0x7fcf301538a0 2023-10-30T14:23:56.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.927+0000 7fcf2d7fa640 1 -- 172.21.15.46:0/1891410062 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcf280b6040 con 0x7fcf301538a0 2023-10-30T14:23:56.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.931+0000 7fcf2d7fa640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcf140664a0 0x7fcf14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:56.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.931+0000 7fcf2d7fa640 1 -- 172.21.15.46:0/1891410062 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8650+0+0 (secure 0 0 0) 0x7fcf28122a00 con 0x7fcf301538a0 2023-10-30T14:23:56.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.931+0000 7fcf34cda640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcf140664a0 0x7fcf14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:56.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.931+0000 7fcf34cda640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcf140664a0 0x7fcf14068960 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009ac0 tx=0x7fcf24009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:56.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:56.935+0000 7fcf2d7fa640 1 -- 172.21.15.46:0/1891410062 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcf280ef420 con 0x7fcf301538a0 2023-10-30T14:23:57.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.123+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1891410062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf301f0330 con 0x7fcf301538a0 2023-10-30T14:23:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.123+0000 7fcf2d7fa640 1 -- 172.21.15.46:0/1891410062 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fcf28009510 con 0x7fcf301538a0 2023-10-30T14:23:57.127 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:57.127 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":253,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:54.843855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 253 2023-10-30T14:23:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.127+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcf140664a0 msgr2=0x7fcf14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.127+0000 7fcf36f3e640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcf140664a0 0x7fcf14068960 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009ac0 tx=0x7fcf24009190 comp rx=0 tx=0).stop 2023-10-30T14:23:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.127+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1891410062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf301538a0 msgr2=0x7fcf30157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.127+0000 7fcf36f3e640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf301538a0 0x7fcf30157520 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7fcf2801ce20 tx=0x7fcf2801bf90 comp rx=0 tx=0).stop 2023-10-30T14:23:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.127+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1891410062 shutdown_connections 2023-10-30T14:23:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.127+0000 7fcf36f3e640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcf140664a0 0x7fcf14068960 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.127+0000 7fcf36f3e640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf301538a0 0x7fcf30157520 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.127+0000 7fcf36f3e640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf300cf160 0x7fcf30142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.127+0000 7fcf36f3e640 1 --2- 172.21.15.46:0/1891410062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf300ca0e0 0x7fcf300ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.127+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1891410062 >> 172.21.15.46:0/1891410062 conn(0x7fcf3006bfe0 msgr2=0x7fcf300c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:57.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.127+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1891410062 shutdown_connections 2023-10-30T14:23:57.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.127+0000 7fcf36f3e640 1 -- 172.21.15.46:0/1891410062 wait complete. 2023-10-30T14:23:57.149 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T14:23:57.150 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:23:57.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.291+0000 7f60ba565640 1 Processor -- start 2023-10-30T14:23:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.291+0000 7f60ba565640 1 -- start start 2023-10-30T14:23:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60ba565640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60b41538a0 0x7f60b415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60ba565640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60b40cf160 0x7f60b40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60ba565640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b40cb800 con 0x7f60b40ca0e0 2023-10-30T14:23:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60ba565640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b415dc30 con 0x7f60b40cf160 2023-10-30T14:23:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60ba565640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b40cb330 con 0x7f60b41538a0 2023-10-30T14:23:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60b8b02640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60b40cf160 0x7f60b40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60b8b02640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60b40cf160 0x7f60b40cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42338/0 (socket says 172.21.15.46:42338) 2023-10-30T14:23:57.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60b8b02640 1 -- 172.21.15.46:0/1342922586 learned_addr learned my addr 172.21.15.46:0/1342922586 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:57.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60b3fff640 1 --2- 172.21.15.46:0/1342922586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60b41538a0 0x7f60b415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:57.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60b8b02640 1 -- 172.21.15.46:0/1342922586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60b41538a0 msgr2=0x7f60b415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:57.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60b8b02640 1 --2- 172.21.15.46:0/1342922586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60b41538a0 0x7f60b415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60b8b02640 1 -- 172.21.15.46:0/1342922586 >> v1:172.21.15.154:6789/0 conn(0x7f60b40ca0e0 legacy=0x7f60b415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.295+0000 7f60b8b02640 1 -- 172.21.15.46:0/1342922586 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b40cad70 con 0x7f60b40cf160 2023-10-30T14:23:57.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60b8b02640 1 --2- 172.21.15.46:0/1342922586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b40cf160 0x7f60b40cf590 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009e50 tx=0x7f60ac01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cfa1136b71932cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:57.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60b2ffd640 1 -- 172.21.15.46:0/1342922586 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0af400 con 0x7f60b40cf160 2023-10-30T14:23:57.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60b2ffd640 1 -- 172.21.15.46:0/1342922586 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac0ab050 con 0x7f60b40cf160 2023-10-30T14:23:57.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60b2ffd640 1 -- 172.21.15.46:0/1342922586 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0af780 con 0x7f60b40cf160 2023-10-30T14:23:57.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60ba565640 1 -- 172.21.15.46:0/1342922586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b40cf160 msgr2=0x7f60b40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60ba565640 1 --2- 172.21.15.46:0/1342922586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b40cf160 0x7f60b40cf590 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009e50 tx=0x7f60ac01f1f0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60ba565640 1 -- 172.21.15.46:0/1342922586 shutdown_connections 2023-10-30T14:23:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60ba565640 1 --2- 172.21.15.46:0/1342922586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b40cf160 0x7f60b40cf590 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60ba565640 1 --2- 172.21.15.46:0/1342922586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60b41538a0 0x7f60b415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60ba565640 1 -- 172.21.15.46:0/1342922586 >> 172.21.15.46:0/1342922586 conn(0x7f60b406bfe0 msgr2=0x7f60b406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60ba565640 1 -- 172.21.15.46:0/1342922586 shutdown_connections 2023-10-30T14:23:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60ba565640 1 -- 172.21.15.46:0/1342922586 wait complete. 2023-10-30T14:23:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60ba565640 1 Processor -- start 2023-10-30T14:23:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60ba565640 1 -- start start 2023-10-30T14:23:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.299+0000 7f60ba565640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60b40ca0e0 0x7f60b415a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60ba565640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60b40cf160 0x7f60b415b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60ba565640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b41538a0 0x7f60b415b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60ba565640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41e3960 con 0x7f60b41538a0 2023-10-30T14:23:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60ba565640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41e4750 con 0x7f60b40cf160 2023-10-30T14:23:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60ba565640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41e3ba0 con 0x7f60b40ca0e0 2023-10-30T14:23:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b8b02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b41538a0 0x7f60b415b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b8b02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b41538a0 0x7f60b415b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42346/0 (socket says 172.21.15.46:42346) 2023-10-30T14:23:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b8b02640 1 -- 172.21.15.46:0/3744965146 learned_addr learned my addr 172.21.15.46:0/3744965146 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b37fe640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60b40cf160 0x7f60b415b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b3fff640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60b40ca0e0 0x7f60b415a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b8b02640 1 -- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60b40ca0e0 msgr2=0x7f60b415a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b8b02640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60b40ca0e0 0x7f60b415a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b8b02640 1 -- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60b40cf160 msgr2=0x7f60b415b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b8b02640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60b40cf160 0x7f60b415b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b8b02640 1 -- 172.21.15.46:0/3744965146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac009b30 con 0x7f60b41538a0 2023-10-30T14:23:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b37fe640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60b40cf160 0x7f60b415b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b8b02640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b41538a0 0x7f60b415b840 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f60ac01ce20 tx=0x7f60ac01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b0ff9640 1 -- 172.21.15.46:0/3744965146 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0afc80 con 0x7f60b41538a0 2023-10-30T14:23:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b0ff9640 1 -- 172.21.15.46:0/3744965146 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac0afde0 con 0x7f60b41538a0 2023-10-30T14:23:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60ba565640 1 -- 172.21.15.46:0/3744965146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b41e3430 con 0x7f60b41538a0 2023-10-30T14:23:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b0ff9640 1 -- 172.21.15.46:0/3744965146 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0ba660 con 0x7f60b41538a0 2023-10-30T14:23:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60ba565640 1 -- 172.21.15.46:0/3744965146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b41e3db0 con 0x7f60b41538a0 2023-10-30T14:23:57.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60ba565640 1 -- 172.21.15.46:0/3744965146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b40b2a50 con 0x7f60b41538a0 2023-10-30T14:23:57.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.303+0000 7f60b0ff9640 1 -- 172.21.15.46:0/3744965146 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60ac0b6040 con 0x7f60b41538a0 2023-10-30T14:23:57.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.307+0000 7f60b0ff9640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60a00664a0 0x7f60a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:57.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.307+0000 7f60b0ff9640 1 -- 172.21.15.46:0/3744965146 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8650+0+0 (secure 0 0 0) 0x7f60ac1235c0 con 0x7f60b41538a0 2023-10-30T14:23:57.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.307+0000 7f60b3fff640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60a00664a0 0x7f60a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:57.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.307+0000 7f60b3fff640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60a00664a0 0x7f60a0068960 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009a50 tx=0x7f60a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.307+0000 7f60b0ff9640 1 -- 172.21.15.46:0/3744965146 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60ac0f0060 con 0x7f60b41538a0 2023-10-30T14:23:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.503+0000 7f60ba565640 1 -- 172.21.15.46:0/3744965146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60b41e4890 con 0x7f60b41538a0 2023-10-30T14:23:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60b0ff9640 1 -- 172.21.15.46:0/3744965146 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f60b41e4890 con 0x7f60b41538a0 2023-10-30T14:23:57.508 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:57.508 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":253,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:54.843855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 253 2023-10-30T14:23:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60ba565640 1 -- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60a00664a0 msgr2=0x7f60a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60ba565640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60a00664a0 0x7f60a0068960 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009a50 tx=0x7f60a4008040 comp rx=0 tx=0).stop 2023-10-30T14:23:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60ba565640 1 -- 172.21.15.46:0/3744965146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b41538a0 msgr2=0x7f60b415b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60ba565640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b41538a0 0x7f60b415b840 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f60ac01ce20 tx=0x7f60ac01bf90 comp rx=0 tx=0).stop 2023-10-30T14:23:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60ba565640 1 -- 172.21.15.46:0/3744965146 shutdown_connections 2023-10-30T14:23:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60ba565640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f60a00664a0 0x7f60a0068960 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60ba565640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60b41538a0 0x7f60b415b840 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60ba565640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60b40cf160 0x7f60b415b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60ba565640 1 --2- 172.21.15.46:0/3744965146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60b40ca0e0 0x7f60b415a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60ba565640 1 -- 172.21.15.46:0/3744965146 >> 172.21.15.46:0/3744965146 conn(0x7f60b406bfe0 msgr2=0x7f60b40c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60ba565640 1 -- 172.21.15.46:0/3744965146 shutdown_connections 2023-10-30T14:23:57.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:57.507+0000 7f60ba565640 1 -- 172.21.15.46:0/3744965146 wait complete. 2023-10-30T14:23:57.531 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 253, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 253, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9735": { "addr": "172.21.15.46:6836/3574404756", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3574404756, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3574404756, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 9735, "incarnation": 246, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 33 }, "gid_9882": { "addr": "172.21.15.46:6833/3394097058", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 3394097058, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 3394097058, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9882, "incarnation": 253, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 91, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:23:54.843855+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9735, "mds_1": 9882 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:23:59.532 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:23:59.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7ebd91d640 1 Processor -- start 2023-10-30T14:23:59.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7ebd91d640 1 -- start start 2023-10-30T14:23:59.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7ebd91d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7eb80c6bd0 0x7f7eb806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7ebd91d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eb8071110 0x7f7eb8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7ebd91d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb8066cf0 con 0x7f7eb806f7a0 2023-10-30T14:23:59.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7ebd91d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb80700c0 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7ebd91d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb80c70d0 con 0x7f7eb8071110 2023-10-30T14:23:59.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7eb6ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7eb806f7a0 0x7f7eb8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37410/0 (socket says 172.21.15.46:37410) 2023-10-30T14:23:59.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7eb6ffd640 1 -- 172.21.15.46:0/2535354216 learned_addr learned my addr 172.21.15.46:0/2535354216 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:23:59.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7eaffff640 1 --2- 172.21.15.46:0/2535354216 >> v2:172.21.15.46:3300/0 conn(0x7f7eb80c6bd0 0x7f7eb806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7eb77fe640 1 --2- 172.21.15.46:0/2535354216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eb8071110 0x7f7eb8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7eaffff640 1 -- 172.21.15.46:0/2535354216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eb8071110 msgr2=0x7f7eb8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:23:59.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7eaffff640 1 --2- 172.21.15.46:0/2535354216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eb8071110 0x7f7eb8158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7eaffff640 1 -- 172.21.15.46:0/2535354216 >> v1:172.21.15.46:6789/0 conn(0x7f7eb806f7a0 legacy=0x7f7eb8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:59.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7eaffff640 1 -- 172.21.15.46:0/2535354216 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb80c7390 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.675+0000 7f7eb67fc640 1 -- 172.21.15.46:0/2535354216 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558237242 0 0) 0x7f7eb8066cf0 con 0x7f7eb806f7a0 2023-10-30T14:23:59.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.679+0000 7f7eaffff640 1 --2- 172.21.15.46:0/2535354216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb80c6bd0 0x7f7eb806fb80 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f7ea00097c0 tx=0x7f7ea001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1867ca682dffe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:59.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.679+0000 7f7eb67fc640 1 -- 172.21.15.46:0/2535354216 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea00a8970 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.679+0000 7f7eb67fc640 1 -- 172.21.15.46:0/2535354216 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea00a8ad0 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.679+0000 7f7eb67fc640 1 -- 172.21.15.46:0/2535354216 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea00a8e60 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.679+0000 7f7ebd91d640 1 -- 172.21.15.46:0/2535354216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb80c6bd0 msgr2=0x7f7eb806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:59.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.679+0000 7f7ebd91d640 1 --2- 172.21.15.46:0/2535354216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb80c6bd0 0x7f7eb806fb80 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f7ea00097c0 tx=0x7f7ea001ec70 comp rx=0 tx=0).stop 2023-10-30T14:23:59.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.679+0000 7f7ebd91d640 1 -- 172.21.15.46:0/2535354216 shutdown_connections 2023-10-30T14:23:59.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.679+0000 7f7ebd91d640 1 --2- 172.21.15.46:0/2535354216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eb8071110 0x7f7eb8158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.679+0000 7f7ebd91d640 1 --2- 172.21.15.46:0/2535354216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb80c6bd0 0x7f7eb806fb80 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.679+0000 7f7ebd91d640 1 -- 172.21.15.46:0/2535354216 >> 172.21.15.46:0/2535354216 conn(0x7f7eb80c7910 msgr2=0x7f7eb80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:59.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.679+0000 7f7ebd91d640 1 -- 172.21.15.46:0/2535354216 shutdown_connections 2023-10-30T14:23:59.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.679+0000 7f7ebd91d640 1 -- 172.21.15.46:0/2535354216 wait complete. 2023-10-30T14:23:59.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7ebd91d640 1 Processor -- start 2023-10-30T14:23:59.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7ebd91d640 1 -- start start 2023-10-30T14:23:59.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7ebd91d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eb806f7a0 0x7f7eb80c0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7ebd91d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8071110 0x7f7eb814f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7ebd91d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb80c6bd0 0x7f7eb814fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7ebd91d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb80c9460 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7ebd91d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb81ef330 con 0x7f7eb8071110 2023-10-30T14:23:59.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7ebd91d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb81e4170 con 0x7f7eb806f7a0 2023-10-30T14:23:59.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb80c6bd0 0x7f7eb814fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb80c6bd0 0x7f7eb814fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42370/0 (socket says 172.21.15.46:42370) 2023-10-30T14:23:59.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb77fe640 1 -- 172.21.15.46:0/4125991368 learned_addr learned my addr 172.21.15.46:0/4125991368 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:23:59.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eaffff640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8071110 0x7f7eb814f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb6ffd640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eb806f7a0 0x7f7eb80c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb77fe640 1 -- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eb806f7a0 msgr2=0x7f7eb80c0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:59.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb77fe640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eb806f7a0 0x7f7eb80c0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb77fe640 1 -- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8071110 msgr2=0x7f7eb814f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:59.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb77fe640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8071110 0x7f7eb814f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb77fe640 1 -- 172.21.15.46:0/4125991368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea0009420 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb6ffd640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eb806f7a0 0x7f7eb80c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:23:59.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb77fe640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb80c6bd0 0x7f7eb814fcd0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0007e40 tx=0x7f7eb000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:59.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb4ff9640 1 -- 172.21.15.46:0/4125991368 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0014620 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb4ff9640 1 -- 172.21.15.46:0/4125991368 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb000ee90 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7eb4ff9640 1 -- 172.21.15.46:0/4125991368 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb00149e0 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7ebd91d640 1 -- 172.21.15.46:0/4125991368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb81e4450 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.683+0000 7f7ebd91d640 1 -- 172.21.15.46:0/4125991368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb80c08b0 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.687+0000 7f7ebd91d640 1 -- 172.21.15.46:0/4125991368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb80c8f50 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.687+0000 7f7eb4ff9640 1 -- 172.21.15.46:0/4125991368 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7eb0014b40 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.687+0000 7f7eb4ff9640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7e980664d0 0x7f7e98068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:23:59.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.687+0000 7f7eb4ff9640 1 -- 172.21.15.46:0/4125991368 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8650+0+0 (secure 0 0 0) 0x7f7eb0085430 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.687+0000 7f7eb6ffd640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7e980664d0 0x7f7e98068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:23:59.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.687+0000 7f7eb6ffd640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7e980664d0 0x7f7e98068990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8000f70 tx=0x7f7ea8007cb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:23:59.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.691+0000 7f7eb4ff9640 1 -- 172.21.15.46:0/4125991368 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7eb80c8f50 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.923+0000 7f7ebd91d640 1 -- 172.21.15.46:0/4125991368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7eb80c8f50 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.923+0000 7f7eb4ff9640 1 -- 172.21.15.46:0/4125991368 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+5732 (secure 0 0 0) 0x7f7eb80c8f50 con 0x7f7eb80c6bd0 2023-10-30T14:23:59.926 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:23:59.926 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":253,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:54.843855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:23:59.927 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 253 2023-10-30T14:23:59.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.927+0000 7f7ebd91d640 1 -- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7e980664d0 msgr2=0x7f7e98068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:59.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.927+0000 7f7ebd91d640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7e980664d0 0x7f7e98068990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8000f70 tx=0x7f7ea8007cb0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.927+0000 7f7ebd91d640 1 -- 172.21.15.46:0/4125991368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb80c6bd0 msgr2=0x7f7eb814fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:23:59.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.927+0000 7f7ebd91d640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb80c6bd0 0x7f7eb814fcd0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0007e40 tx=0x7f7eb000e4b0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.927+0000 7f7ebd91d640 1 -- 172.21.15.46:0/4125991368 shutdown_connections 2023-10-30T14:23:59.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.927+0000 7f7ebd91d640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7e980664d0 0x7f7e98068990 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.927+0000 7f7ebd91d640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eb80c6bd0 0x7f7eb814fcd0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.927+0000 7f7ebd91d640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eb8071110 0x7f7eb814f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.927+0000 7f7ebd91d640 1 --2- 172.21.15.46:0/4125991368 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eb806f7a0 0x7f7eb80c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:23:59.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.927+0000 7f7ebd91d640 1 -- 172.21.15.46:0/4125991368 >> 172.21.15.46:0/4125991368 conn(0x7f7eb80c7910 msgr2=0x7f7eb806a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:23:59.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.927+0000 7f7ebd91d640 1 -- 172.21.15.46:0/4125991368 shutdown_connections 2023-10-30T14:23:59.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:23:59.927+0000 7f7ebd91d640 1 -- 172.21.15.46:0/4125991368 wait complete. 2023-10-30T14:23:59.961 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:01.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc37df0f640 1 Processor -- start 2023-10-30T14:24:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc37df0f640 1 -- start start 2023-10-30T14:24:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc37df0f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc3781538a0 0x7fc37815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc37df0f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3780cf160 0x7fc3780cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc37df0f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3780cb800 con 0x7fc3780ca0e0 2023-10-30T14:24:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc37df0f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc37815dc30 con 0x7fc3780cf160 2023-10-30T14:24:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc37df0f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3780cb330 con 0x7fc3781538a0 2023-10-30T14:24:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc377fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3780cf160 0x7fc3780cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc377fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc3780cf160 0x7fc3780cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42374/0 (socket says 172.21.15.46:42374) 2023-10-30T14:24:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc377fff640 1 -- 172.21.15.46:0/1092484904 learned_addr learned my addr 172.21.15.46:0/1092484904 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc3777fe640 1 --2- 172.21.15.46:0/1092484904 >> v2:172.21.15.154:3300/0 conn(0x7fc3781538a0 0x7fc37815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc377fff640 1 -- 172.21.15.46:0/1092484904 >> v2:172.21.15.154:3300/0 conn(0x7fc3781538a0 msgr2=0x7fc37815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc377fff640 1 --2- 172.21.15.46:0/1092484904 >> v2:172.21.15.154:3300/0 conn(0x7fc3781538a0 0x7fc37815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.111+0000 7fc377fff640 1 -- 172.21.15.46:0/1092484904 >> v1:172.21.15.154:6789/0 conn(0x7fc3780ca0e0 legacy=0x7fc37815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.115+0000 7fc377fff640 1 -- 172.21.15.46:0/1092484904 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780cad70 con 0x7fc3780cf160 2023-10-30T14:24:02.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.115+0000 7fc377fff640 1 --2- 172.21.15.46:0/1092484904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3780cf160 0x7fc3780cf590 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fc3780cfd10 tx=0x7fc37001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f96a214167c6e04 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.115+0000 7fc3767fc640 1 -- 172.21.15.46:0/1092484904 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3700af400 con 0x7fc3780cf160 2023-10-30T14:24:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.115+0000 7fc3767fc640 1 -- 172.21.15.46:0/1092484904 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3700ab050 con 0x7fc3780cf160 2023-10-30T14:24:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.115+0000 7fc3767fc640 1 -- 172.21.15.46:0/1092484904 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3700af780 con 0x7fc3780cf160 2023-10-30T14:24:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.115+0000 7fc37df0f640 1 -- 172.21.15.46:0/1092484904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3780cf160 msgr2=0x7fc3780cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.115+0000 7fc37df0f640 1 --2- 172.21.15.46:0/1092484904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3780cf160 0x7fc3780cf590 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fc3780cfd10 tx=0x7fc37001f1c0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.115+0000 7fc37df0f640 1 -- 172.21.15.46:0/1092484904 shutdown_connections 2023-10-30T14:24:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.115+0000 7fc37df0f640 1 --2- 172.21.15.46:0/1092484904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3780cf160 0x7fc3780cf590 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.115+0000 7fc37df0f640 1 --2- 172.21.15.46:0/1092484904 >> v2:172.21.15.154:3300/0 conn(0x7fc3781538a0 0x7fc37815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.115+0000 7fc37df0f640 1 -- 172.21.15.46:0/1092484904 >> 172.21.15.46:0/1092484904 conn(0x7fc37806bfe0 msgr2=0x7fc37806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.115+0000 7fc37df0f640 1 -- 172.21.15.46:0/1092484904 shutdown_connections 2023-10-30T14:24:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc37df0f640 1 -- 172.21.15.46:0/1092484904 wait complete. 2023-10-30T14:24:02.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc37df0f640 1 Processor -- start 2023-10-30T14:24:02.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc37df0f640 1 -- start start 2023-10-30T14:24:02.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc37df0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3780ca0e0 0x7fc378159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:02.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc37df0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3780cf160 0x7fc3781480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:02.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc37df0f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3781538a0 0x7fc3781ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:02.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc37df0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc378146880 con 0x7fc3780cf160 2023-10-30T14:24:02.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc37df0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780b4b60 con 0x7fc3780ca0e0 2023-10-30T14:24:02.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc37df0f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3781e2ae0 con 0x7fc3781538a0 2023-10-30T14:24:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc376ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3780cf160 0x7fc3781480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc3777fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3780ca0e0 0x7fc378159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc376ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3780cf160 0x7fc3781480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42384/0 (socket says 172.21.15.46:42384) 2023-10-30T14:24:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc376ffd640 1 -- 172.21.15.46:0/2270450194 learned_addr learned my addr 172.21.15.46:0/2270450194 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc377fff640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3781538a0 0x7fc3781ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc376ffd640 1 -- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3781538a0 msgr2=0x7fc3781ef130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc376ffd640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3781538a0 0x7fc3781ef130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc376ffd640 1 -- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3780ca0e0 msgr2=0x7fc378159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc376ffd640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3780ca0e0 0x7fc378159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc376ffd640 1 -- 172.21.15.46:0/2270450194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc370009b30 con 0x7fc3780cf160 2023-10-30T14:24:02.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc377fff640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3781538a0 0x7fc3781ef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:24:02.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.119+0000 7fc376ffd640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3780cf160 0x7fc3781480f0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00c3c0 tx=0x7fc36c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:02.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.123+0000 7fc35ffff640 1 -- 172.21.15.46:0/2270450194 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c0052d0 con 0x7fc3780cf160 2023-10-30T14:24:02.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.123+0000 7fc35ffff640 1 -- 172.21.15.46:0/2270450194 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c005430 con 0x7fc3780cf160 2023-10-30T14:24:02.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.123+0000 7fc35ffff640 1 -- 172.21.15.46:0/2270450194 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c003840 con 0x7fc3780cf160 2023-10-30T14:24:02.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.123+0000 7fc37df0f640 1 -- 172.21.15.46:0/2270450194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc378159540 con 0x7fc3780cf160 2023-10-30T14:24:02.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.123+0000 7fc37df0f640 1 -- 172.21.15.46:0/2270450194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3781e2cf0 con 0x7fc3780cf160 2023-10-30T14:24:02.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.123+0000 7fc37df0f640 1 -- 172.21.15.46:0/2270450194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3780b2a50 con 0x7fc3780cf160 2023-10-30T14:24:02.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.123+0000 7fc35ffff640 1 -- 172.21.15.46:0/2270450194 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc36c0039a0 con 0x7fc3780cf160 2023-10-30T14:24:02.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.123+0000 7fc35ffff640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3640664a0 0x7fc364068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:02.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.123+0000 7fc35ffff640 1 -- 172.21.15.46:0/2270450194 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8694+0+0 (secure 0 0 0) 0x7fc36c083ea0 con 0x7fc3780cf160 2023-10-30T14:24:02.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.123+0000 7fc3777fe640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3640664a0 0x7fc364068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:02.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.127+0000 7fc3777fe640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3640664a0 0x7fc364068960 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc368007a60 tx=0x7fc368008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:02.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.127+0000 7fc35ffff640 1 -- 172.21.15.46:0/2270450194 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc36c050860 con 0x7fc3780cf160 2023-10-30T14:24:02.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.339+0000 7fc37df0f640 1 -- 172.21.15.46:0/2270450194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3780b4ca0 con 0x7fc3780cf160 2023-10-30T14:24:02.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.339+0000 7fc35ffff640 1 -- 172.21.15.46:0/2270450194 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fc3780b4ca0 con 0x7fc3780cf160 2023-10-30T14:24:02.342 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:02.342 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":253,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:54.843855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:02.342 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 253 2023-10-30T14:24:02.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.343+0000 7fc37df0f640 1 -- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3640664a0 msgr2=0x7fc364068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:02.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.343+0000 7fc37df0f640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3640664a0 0x7fc364068960 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc368007a60 tx=0x7fc368008040 comp rx=0 tx=0).stop 2023-10-30T14:24:02.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.343+0000 7fc37df0f640 1 -- 172.21.15.46:0/2270450194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3780cf160 msgr2=0x7fc3781480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:02.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.343+0000 7fc37df0f640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3780cf160 0x7fc3781480f0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00c3c0 tx=0x7fc36c00c880 comp rx=0 tx=0).stop 2023-10-30T14:24:02.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.343+0000 7fc37df0f640 1 -- 172.21.15.46:0/2270450194 shutdown_connections 2023-10-30T14:24:02.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.343+0000 7fc37df0f640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fc3640664a0 0x7fc364068960 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.343+0000 7fc37df0f640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3781538a0 0x7fc3781ef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.343+0000 7fc37df0f640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3780cf160 0x7fc3781480f0 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.343+0000 7fc37df0f640 1 --2- 172.21.15.46:0/2270450194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3780ca0e0 0x7fc378159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:02.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.343+0000 7fc37df0f640 1 -- 172.21.15.46:0/2270450194 >> 172.21.15.46:0/2270450194 conn(0x7fc37806bfe0 msgr2=0x7fc37806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:02.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.343+0000 7fc37df0f640 1 -- 172.21.15.46:0/2270450194 shutdown_connections 2023-10-30T14:24:02.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:02.343+0000 7fc37df0f640 1 -- 172.21.15.46:0/2270450194 wait complete. 2023-10-30T14:24:02.365 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:04.366 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc2829c640 1 Processor -- start 2023-10-30T14:24:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc2829c640 1 -- start start 2023-10-30T14:24:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc2829c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2015d6f0 0x7fbc2015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc2829c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbc200cf160 0x7fbc200cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc2829c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc200cb800 con 0x7fbc201538a0 2023-10-30T14:24:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc2829c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc200cb330 con 0x7fbc200cf160 2023-10-30T14:24:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc2829c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc200cad70 con 0x7fbc2015d6f0 2023-10-30T14:24:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc26839640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbc200cf160 0x7fbc200cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc25837640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2015d6f0 0x7fbc2015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc25837640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2015d6f0 0x7fbc2015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:51050/0 (socket says 172.21.15.46:51050) 2023-10-30T14:24:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc26839640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbc200cf160 0x7fbc200cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:51640/0 (socket says 172.21.15.46:51640) 2023-10-30T14:24:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc26038640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fbc201538a0 0x7fbc200ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:32896/0 (socket says 172.21.15.46:32896) 2023-10-30T14:24:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.515+0000 7fbc25837640 1 -- 172.21.15.46:0/1189915569 learned_addr learned my addr 172.21.15.46:0/1189915569 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.519+0000 7fbc26839640 1 -- 172.21.15.46:0/1189915569 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2015d6f0 msgr2=0x7fbc2015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.519+0000 7fbc26839640 1 --2- 172.21.15.46:0/1189915569 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2015d6f0 0x7fbc2015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.519+0000 7fbc26839640 1 -- 172.21.15.46:0/1189915569 >> v1:172.21.15.154:6789/0 conn(0x7fbc201538a0 legacy=0x7fbc200ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.519+0000 7fbc26839640 1 -- 172.21.15.46:0/1189915569 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc200cb050 con 0x7fbc200cf160 2023-10-30T14:24:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.519+0000 7fbc25036640 1 -- 172.21.15.46:0/1189915569 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690771374 0 0) 0x7fbc200cb800 con 0x7fbc201538a0 2023-10-30T14:24:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.519+0000 7fbc25837640 1 --2- 172.21.15.46:0/1189915569 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2015d6f0 0x7fbc2015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:24:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.519+0000 7fbc26839640 1 --2- 172.21.15.46:0/1189915569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc200cf160 0x7fbc200cf590 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fbc18009d90 tx=0x7fbc1801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2b9c505d9881c51 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:04.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.519+0000 7fbc25036640 1 -- 172.21.15.46:0/1189915569 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc180af600 con 0x7fbc200cf160 2023-10-30T14:24:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.519+0000 7fbc25036640 1 -- 172.21.15.46:0/1189915569 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc180ab050 con 0x7fbc200cf160 2023-10-30T14:24:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.519+0000 7fbc25036640 1 -- 172.21.15.46:0/1189915569 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc180af980 con 0x7fbc200cf160 2023-10-30T14:24:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 -- 172.21.15.46:0/1189915569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc200cf160 msgr2=0x7fbc200cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 --2- 172.21.15.46:0/1189915569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc200cf160 0x7fbc200cf590 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fbc18009d90 tx=0x7fbc1801f130 comp rx=0 tx=0).stop 2023-10-30T14:24:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 -- 172.21.15.46:0/1189915569 shutdown_connections 2023-10-30T14:24:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 --2- 172.21.15.46:0/1189915569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc200cf160 0x7fbc200cf590 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 --2- 172.21.15.46:0/1189915569 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2015d6f0 0x7fbc2015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 -- 172.21.15.46:0/1189915569 >> 172.21.15.46:0/1189915569 conn(0x7fbc2006bfe0 msgr2=0x7fbc2006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 -- 172.21.15.46:0/1189915569 shutdown_connections 2023-10-30T14:24:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 -- 172.21.15.46:0/1189915569 wait complete. 2023-10-30T14:24:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 Processor -- start 2023-10-30T14:24:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 -- start start 2023-10-30T14:24:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc200cf160 0x7fbc2014db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc201538a0 0x7fbc2014ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2015d6f0 0x7fbc2015b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc201e36e0 con 0x7fbc200cf160 2023-10-30T14:24:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc20150b00 con 0x7fbc201538a0 2023-10-30T14:24:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc2829c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc201e2b60 con 0x7fbc2015d6f0 2023-10-30T14:24:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc26038640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc200cf160 0x7fbc2014db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc26038640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc200cf160 0x7fbc2014db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42394/0 (socket says 172.21.15.46:42394) 2023-10-30T14:24:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.523+0000 7fbc26038640 1 -- 172.21.15.46:0/2980536907 learned_addr learned my addr 172.21.15.46:0/2980536907 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc25837640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc201538a0 0x7fbc2014ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc26839640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2015d6f0 0x7fbc2015b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc26038640 1 -- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2015d6f0 msgr2=0x7fbc2015b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:04.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc26038640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2015d6f0 0x7fbc2015b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc26038640 1 -- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc201538a0 msgr2=0x7fbc2014ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc26038640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc201538a0 0x7fbc2014ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc26038640 1 -- 172.21.15.46:0/2980536907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc100094d0 con 0x7fbc200cf160 2023-10-30T14:24:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc26038640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc200cf160 0x7fbc2014db10 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0011f0 tx=0x7fbc1c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:04.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc16ffd640 1 -- 172.21.15.46:0/2980536907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c0074e0 con 0x7fbc200cf160 2023-10-30T14:24:04.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc16ffd640 1 -- 172.21.15.46:0/2980536907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc1c007640 con 0x7fbc200cf160 2023-10-30T14:24:04.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc16ffd640 1 -- 172.21.15.46:0/2980536907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c004a10 con 0x7fbc200cf160 2023-10-30T14:24:04.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc2829c640 1 -- 172.21.15.46:0/2980536907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc18009a70 con 0x7fbc200cf160 2023-10-30T14:24:04.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc2829c640 1 -- 172.21.15.46:0/2980536907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc201e3120 con 0x7fbc200cf160 2023-10-30T14:24:04.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc2829c640 1 -- 172.21.15.46:0/2980536907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc200b2a50 con 0x7fbc200cf160 2023-10-30T14:24:04.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.527+0000 7fbc16ffd640 1 -- 172.21.15.46:0/2980536907 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbc1c004b70 con 0x7fbc200cf160 2023-10-30T14:24:04.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.531+0000 7fbc16ffd640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbc040664d0 0x7fbc04068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:04.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.531+0000 7fbc16ffd640 1 -- 172.21.15.46:0/2980536907 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8694+0+0 (secure 0 0 0) 0x7fbc1c083af0 con 0x7fbc200cf160 2023-10-30T14:24:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.531+0000 7fbc25837640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbc040664d0 0x7fbc04068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:04.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.531+0000 7fbc25837640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbc040664d0 0x7fbc04068990 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fbc200c5200 tx=0x7fbc10008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:04.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.535+0000 7fbc16ffd640 1 -- 172.21.15.46:0/2980536907 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbc1c0504b0 con 0x7fbc200cf160 2023-10-30T14:24:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.743+0000 7fbc2829c640 1 -- 172.21.15.46:0/2980536907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc2004e170 con 0x7fbc200cf160 2023-10-30T14:24:04.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc16ffd640 1 -- 172.21.15.46:0/2980536907 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+5732 (secure 0 0 0) 0x7fbc1c018090 con 0x7fbc200cf160 2023-10-30T14:24:04.749 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:04.749 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":253,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:23:54.843855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:04.749 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 253 2023-10-30T14:24:04.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc2829c640 1 -- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbc040664d0 msgr2=0x7fbc04068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:04.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc2829c640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbc040664d0 0x7fbc04068990 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fbc200c5200 tx=0x7fbc10008040 comp rx=0 tx=0).stop 2023-10-30T14:24:04.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc2829c640 1 -- 172.21.15.46:0/2980536907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc200cf160 msgr2=0x7fbc2014db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:04.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc2829c640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc200cf160 0x7fbc2014db10 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0011f0 tx=0x7fbc1c00c9b0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc2829c640 1 -- 172.21.15.46:0/2980536907 shutdown_connections 2023-10-30T14:24:04.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc2829c640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbc040664d0 0x7fbc04068990 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc2829c640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2015d6f0 0x7fbc2015b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc2829c640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc201538a0 0x7fbc2014ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc2829c640 1 --2- 172.21.15.46:0/2980536907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc200cf160 0x7fbc2014db10 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:04.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc2829c640 1 -- 172.21.15.46:0/2980536907 >> 172.21.15.46:0/2980536907 conn(0x7fbc2006bfe0 msgr2=0x7fbc200ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:04.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc2829c640 1 -- 172.21.15.46:0/2980536907 shutdown_connections 2023-10-30T14:24:04.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:04.747+0000 7fbc2829c640 1 -- 172.21.15.46:0/2980536907 wait complete. 2023-10-30T14:24:04.772 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:06.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.919+0000 7f8c1b394640 1 Processor -- start 2023-10-30T14:24:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.919+0000 7f8c1b394640 1 -- start start 2023-10-30T14:24:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1b394640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c140c2570 0x7f8c140cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1b394640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c140c29d0 0x7f8c140cb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1b394640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c140c68b0 con 0x7f8c14150920 2023-10-30T14:24:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1b394640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c14150d00 con 0x7f8c140c2570 2023-10-30T14:24:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1b394640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c140c2e90 con 0x7f8c140c29d0 2023-10-30T14:24:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1892f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c140c2570 0x7f8c140cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:06.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1892f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c140c2570 0x7f8c140cb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58468/0 (socket says 172.21.15.46:58468) 2023-10-30T14:24:06.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1892f640 1 -- 172.21.15.46:0/1479653885 learned_addr learned my addr 172.21.15.46:0/1479653885 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:06.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c19931640 1 --2- 172.21.15.46:0/1479653885 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c140c29d0 0x7f8c140cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:06.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1892f640 1 -- 172.21.15.46:0/1479653885 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c140c29d0 msgr2=0x7f8c140cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:06.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1892f640 1 --2- 172.21.15.46:0/1479653885 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c140c29d0 0x7f8c140cb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:06.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1892f640 1 -- 172.21.15.46:0/1479653885 >> v1:172.21.15.154:6789/0 conn(0x7f8c14150920 legacy=0x7f8c1406d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:06.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1892f640 1 -- 172.21.15.46:0/1479653885 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c140c6b00 con 0x7f8c140c2570 2023-10-30T14:24:06.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c19931640 1 --2- 172.21.15.46:0/1479653885 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c140c29d0 0x7f8c140cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:24:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.923+0000 7f8c1892f640 1 --2- 172.21.15.46:0/1479653885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c140c2570 0x7f8c140cb0d0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400bb10 tx=0x7f8c0401eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2324f6fb13127ce7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c0bfff640 1 -- 172.21.15.46:0/1479653885 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040a7b40 con 0x7f8c140c2570 2023-10-30T14:24:06.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c0bfff640 1 -- 172.21.15.46:0/1479653885 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c040a7ca0 con 0x7f8c140c2570 2023-10-30T14:24:06.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c0bfff640 1 -- 172.21.15.46:0/1479653885 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040af660 con 0x7f8c140c2570 2023-10-30T14:24:06.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 -- 172.21.15.46:0/1479653885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c140c2570 msgr2=0x7f8c140cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:06.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 --2- 172.21.15.46:0/1479653885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c140c2570 0x7f8c140cb0d0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400bb10 tx=0x7f8c0401eb80 comp rx=0 tx=0).stop 2023-10-30T14:24:06.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 -- 172.21.15.46:0/1479653885 shutdown_connections 2023-10-30T14:24:06.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 --2- 172.21.15.46:0/1479653885 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c140c29d0 0x7f8c140cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:06.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 --2- 172.21.15.46:0/1479653885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c140c2570 0x7f8c140cb0d0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:06.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 -- 172.21.15.46:0/1479653885 >> 172.21.15.46:0/1479653885 conn(0x7f8c1406dfd0 msgr2=0x7f8c1406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:06.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 -- 172.21.15.46:0/1479653885 shutdown_connections 2023-10-30T14:24:06.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 -- 172.21.15.46:0/1479653885 wait complete. 2023-10-30T14:24:06.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 Processor -- start 2023-10-30T14:24:06.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 -- start start 2023-10-30T14:24:06.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c140c2570 0x7f8c141f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:06.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c140c29d0 0x7f8c141f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:06.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c14150920 0x7f8c14160380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:06.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c14142610 con 0x7f8c14150920 2023-10-30T14:24:06.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c14072450 con 0x7f8c140c2570 2023-10-30T14:24:06.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c1b394640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c14071e90 con 0x7f8c140c29d0 2023-10-30T14:24:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c19931640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c14150920 0x7f8c14160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c19931640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c14150920 0x7f8c14160380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58470/0 (socket says 172.21.15.46:58470) 2023-10-30T14:24:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.927+0000 7f8c19931640 1 -- 172.21.15.46:0/2005928252 learned_addr learned my addr 172.21.15.46:0/2005928252 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c1892f640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c140c29d0 0x7f8c141f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c19130640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c140c2570 0x7f8c141f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:06.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c19931640 1 -- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c140c29d0 msgr2=0x7f8c141f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:06.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c19931640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c140c29d0 0x7f8c141f3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:06.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c19931640 1 -- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c140c2570 msgr2=0x7f8c141f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:06.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c19931640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c140c2570 0x7f8c141f5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:06.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c19931640 1 -- 172.21.15.46:0/2005928252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0400b770 con 0x7f8c14150920 2023-10-30T14:24:06.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c1892f640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c140c29d0 0x7f8c141f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:24:06.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c19931640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c14150920 0x7f8c14160380 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c007ec0 tx=0x7f8c0c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:06.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c09ffb640 1 -- 172.21.15.46:0/2005928252 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c003ba0 con 0x7f8c14150920 2023-10-30T14:24:06.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c09ffb640 1 -- 172.21.15.46:0/2005928252 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0c00f040 con 0x7f8c14150920 2023-10-30T14:24:06.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c09ffb640 1 -- 172.21.15.46:0/2005928252 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c00de80 con 0x7f8c14150920 2023-10-30T14:24:06.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c1b394640 1 -- 172.21.15.46:0/2005928252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c14072170 con 0x7f8c14150920 2023-10-30T14:24:06.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c1b394640 1 -- 172.21.15.46:0/2005928252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c141f57c0 con 0x7f8c14150920 2023-10-30T14:24:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c1b394640 1 -- 172.21.15.46:0/2005928252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1404e170 con 0x7f8c14150920 2023-10-30T14:24:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.931+0000 7f8c09ffb640 1 -- 172.21.15.46:0/2005928252 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8c0c004030 con 0x7f8c14150920 2023-10-30T14:24:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.935+0000 7f8c09ffb640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8bf80664a0 0x7f8bf8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.935+0000 7f8c09ffb640 1 -- 172.21.15.46:0/2005928252 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8694+0+0 (secure 0 0 0) 0x7f8c0c084d30 con 0x7f8c14150920 2023-10-30T14:24:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.935+0000 7f8c19130640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8bf80664a0 0x7f8bf8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.935+0000 7f8c19130640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8bf80664a0 0x7f8bf8068960 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f8c10000f70 tx=0x7f8c10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:06.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:06.939+0000 7f8c09ffb640 1 -- 172.21.15.46:0/2005928252 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8c0c051640 con 0x7f8c14150920 2023-10-30T14:24:07.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.127+0000 7f8c1b394640 1 -- 172.21.15.46:0/2005928252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c1406f010 con 0x7f8c14150920 2023-10-30T14:24:07.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.127+0000 7f8c09ffb640 1 -- 172.21.15.46:0/2005928252 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 254 v254) v1 ==== 95+0+5734 (secure 0 0 0) 0x7f8c0c01a020 con 0x7f8c14150920 2023-10-30T14:24:07.130 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:07.130 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":254,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":254,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:05.056866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:resolve","state_seq":15,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:07.131 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 254 2023-10-30T14:24:07.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.131+0000 7f8c1b394640 1 -- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8bf80664a0 msgr2=0x7f8bf8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:07.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.131+0000 7f8c1b394640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8bf80664a0 0x7f8bf8068960 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f8c10000f70 tx=0x7f8c10009040 comp rx=0 tx=0).stop 2023-10-30T14:24:07.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.131+0000 7f8c1b394640 1 -- 172.21.15.46:0/2005928252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c14150920 msgr2=0x7f8c14160380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:07.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.131+0000 7f8c1b394640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c14150920 0x7f8c14160380 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c007ec0 tx=0x7f8c0c00d4a0 comp rx=0 tx=0).stop 2023-10-30T14:24:07.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.131+0000 7f8c1b394640 1 -- 172.21.15.46:0/2005928252 shutdown_connections 2023-10-30T14:24:07.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.131+0000 7f8c1b394640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8bf80664a0 0x7f8bf8068960 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:07.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.131+0000 7f8c1b394640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c14150920 0x7f8c14160380 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:07.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.131+0000 7f8c1b394640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c140c29d0 0x7f8c141f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:07.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.131+0000 7f8c1b394640 1 --2- 172.21.15.46:0/2005928252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c140c2570 0x7f8c141f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:07.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.131+0000 7f8c1b394640 1 -- 172.21.15.46:0/2005928252 >> 172.21.15.46:0/2005928252 conn(0x7f8c1406dfd0 msgr2=0x7f8c1414d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:07.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.131+0000 7f8c1b394640 1 -- 172.21.15.46:0/2005928252 shutdown_connections 2023-10-30T14:24:07.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:07.131+0000 7f8c1b394640 1 -- 172.21.15.46:0/2005928252 wait complete. 2023-10-30T14:24:07.152 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:09.153 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:09.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.303+0000 7fe0e0217640 1 Processor -- start 2023-10-30T14:24:09.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.303+0000 7fe0e0217640 1 -- start start 2023-10-30T14:24:09.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0e0217640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe0d80c05d0 0x7fe0d813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:09.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0e0217640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe0d80c6f60 0x7fe0d813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:09.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0e0217640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d80c0a50 con 0x7fe0d80c6b80 2023-10-30T14:24:09.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0e0217640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d806f6e0 con 0x7fe0d80c6f60 2023-10-30T14:24:09.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0e0217640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d806f9c0 con 0x7fe0d80c05d0 2023-10-30T14:24:09.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0ddfb3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe0d80c6b80 0x7fe0d80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44760/0 (socket says 172.21.15.46:44760) 2023-10-30T14:24:09.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0ddfb3640 1 -- 172.21.15.46:0/679212792 learned_addr learned my addr 172.21.15.46:0/679212792 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:09.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0de7b4640 1 --2- 172.21.15.46:0/679212792 >> v2:172.21.15.46:3300/0 conn(0x7fe0d80c6f60 0x7fe0d813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:09.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0dd7b2640 1 --2- 172.21.15.46:0/679212792 >> v2:172.21.15.154:3300/0 conn(0x7fe0d80c05d0 0x7fe0d813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:09.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0dcfb1640 1 -- 172.21.15.46:0/679212792 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153191957 0 0) 0x7fe0d80c0a50 con 0x7fe0d80c6b80 2023-10-30T14:24:09.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0dcfb1640 1 -- 172.21.15.46:0/679212792 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c4003530 con 0x7fe0d80c6b80 2023-10-30T14:24:09.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0de7b4640 1 -- 172.21.15.46:0/679212792 >> v2:172.21.15.154:3300/0 conn(0x7fe0d80c05d0 msgr2=0x7fe0d813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:09.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0de7b4640 1 --2- 172.21.15.46:0/679212792 >> v2:172.21.15.154:3300/0 conn(0x7fe0d80c05d0 0x7fe0d813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0de7b4640 1 -- 172.21.15.46:0/679212792 >> v1:172.21.15.46:6789/0 conn(0x7fe0d80c6b80 legacy=0x7fe0d80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.307+0000 7fe0de7b4640 1 -- 172.21.15.46:0/679212792 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d806fca0 con 0x7fe0d80c6f60 2023-10-30T14:24:09.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0de7b4640 1 --2- 172.21.15.46:0/679212792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d80c6f60 0x7fe0d813f2d0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7fe0d806c7e0 tx=0x7fe0d001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4963acffb858123b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:09.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0dcfb1640 1 -- 172.21.15.46:0/679212792 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00a58e0 con 0x7fe0d80c6f60 2023-10-30T14:24:09.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0dcfb1640 1 -- 172.21.15.46:0/679212792 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d00a5a40 con 0x7fe0d80c6f60 2023-10-30T14:24:09.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0dcfb1640 1 -- 172.21.15.46:0/679212792 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00a5dd0 con 0x7fe0d80c6f60 2023-10-30T14:24:09.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0e0217640 1 -- 172.21.15.46:0/679212792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d80c6f60 msgr2=0x7fe0d813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0e0217640 1 --2- 172.21.15.46:0/679212792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d80c6f60 0x7fe0d813f2d0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7fe0d806c7e0 tx=0x7fe0d001c910 comp rx=0 tx=0).stop 2023-10-30T14:24:09.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0e0217640 1 -- 172.21.15.46:0/679212792 shutdown_connections 2023-10-30T14:24:09.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0e0217640 1 --2- 172.21.15.46:0/679212792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d80c6f60 0x7fe0d813f2d0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0e0217640 1 --2- 172.21.15.46:0/679212792 >> v2:172.21.15.154:3300/0 conn(0x7fe0d80c05d0 0x7fe0d813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0e0217640 1 -- 172.21.15.46:0/679212792 >> 172.21.15.46:0/679212792 conn(0x7fe0d80c7910 msgr2=0x7fe0d80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:09.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0e0217640 1 -- 172.21.15.46:0/679212792 shutdown_connections 2023-10-30T14:24:09.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0e0217640 1 -- 172.21.15.46:0/679212792 wait complete. 2023-10-30T14:24:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0e0217640 1 Processor -- start 2023-10-30T14:24:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.311+0000 7fe0e0217640 1 -- start start 2023-10-30T14:24:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0e0217640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0d80c05d0 0x7fe0d80c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0e0217640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d80c6b80 0x7fe0d81ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0e0217640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d80c6f60 0x7fe0d81efe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0e0217640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d80c9840 con 0x7fe0d80c6f60 2023-10-30T14:24:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0e0217640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d813ab90 con 0x7fe0d80c05d0 2023-10-30T14:24:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0e0217640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d80c90b0 con 0x7fe0d80c6b80 2023-10-30T14:24:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0de7b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d80c6f60 0x7fe0d81efe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:09.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0de7b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d80c6f60 0x7fe0d81efe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58488/0 (socket says 172.21.15.46:58488) 2023-10-30T14:24:09.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0de7b4640 1 -- 172.21.15.46:0/2157182019 learned_addr learned my addr 172.21.15.46:0/2157182019 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:09.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0ddfb3640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0d80c05d0 0x7fe0d80c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:09.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0dd7b2640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d80c6b80 0x7fe0d81ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:09.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0de7b4640 1 -- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d80c6b80 msgr2=0x7fe0d81ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0de7b4640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d80c6b80 0x7fe0d81ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0de7b4640 1 -- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0d80c05d0 msgr2=0x7fe0d80c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0de7b4640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0d80c05d0 0x7fe0d80c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0de7b4640 1 -- 172.21.15.46:0/2157182019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d0009a70 con 0x7fe0d80c6f60 2023-10-30T14:24:09.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0de7b4640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d80c6f60 0x7fe0d81efe80 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0019520 tx=0x7fe0d00196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0ce7fc640 1 -- 172.21.15.46:0/2157182019 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00ad650 con 0x7fe0d80c6f60 2023-10-30T14:24:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0ce7fc640 1 -- 172.21.15.46:0/2157182019 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d00ad7b0 con 0x7fe0d80c6f60 2023-10-30T14:24:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0ce7fc640 1 -- 172.21.15.46:0/2157182019 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00adb40 con 0x7fe0d80c6f60 2023-10-30T14:24:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0e0217640 1 -- 172.21.15.46:0/2157182019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d81f03c0 con 0x7fe0d80c6f60 2023-10-30T14:24:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0e0217640 1 -- 172.21.15.46:0/2157182019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d813a5d0 con 0x7fe0d80c6f60 2023-10-30T14:24:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0e0217640 1 -- 172.21.15.46:0/2157182019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d80c9260 con 0x7fe0d80c6f60 2023-10-30T14:24:09.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.315+0000 7fe0ce7fc640 1 -- 172.21.15.46:0/2157182019 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe0d00adca0 con 0x7fe0d80c6f60 2023-10-30T14:24:09.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.319+0000 7fe0ce7fc640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0bc0664a0 0x7fe0bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:09.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.319+0000 7fe0ce7fc640 1 -- 172.21.15.46:0/2157182019 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8694+0+0 (secure 0 0 0) 0x7fe0d011fe30 con 0x7fe0d80c6f60 2023-10-30T14:24:09.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.319+0000 7fe0ddfb3640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0bc0664a0 0x7fe0bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:09.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.319+0000 7fe0ddfb3640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0bc0664a0 0x7fe0bc068960 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4007fd0 tx=0x7fe0d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:09.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.323+0000 7fe0ce7fc640 1 -- 172.21.15.46:0/2157182019 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe0d80c9260 con 0x7fe0d80c6f60 2023-10-30T14:24:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 -- 172.21.15.46:0/2157182019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0d80c9260 con 0x7fe0d80c6f60 2023-10-30T14:24:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0ce7fc640 1 -- 172.21.15.46:0/2157182019 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 254 v254) v1 ==== 95+0+5734 (secure 0 0 0) 0x7fe0d80c9260 con 0x7fe0d80c6f60 2023-10-30T14:24:09.513 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:09.514 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":254,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":254,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:05.056866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:resolve","state_seq":15,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:09.514 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 254 2023-10-30T14:24:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 -- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0bc0664a0 msgr2=0x7fe0bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0bc0664a0 0x7fe0bc068960 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4007fd0 tx=0x7fe0d4009040 comp rx=0 tx=0).stop 2023-10-30T14:24:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 -- 172.21.15.46:0/2157182019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d80c6f60 msgr2=0x7fe0d81efe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d80c6f60 0x7fe0d81efe80 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0019520 tx=0x7fe0d00196b0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 -- 172.21.15.46:0/2157182019 shutdown_connections 2023-10-30T14:24:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe0bc0664a0 0x7fe0bc068960 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0d80c6f60 0x7fe0d81efe80 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe0d80c6b80 0x7fe0d81ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 --2- 172.21.15.46:0/2157182019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0d80c05d0 0x7fe0d80c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 -- 172.21.15.46:0/2157182019 >> 172.21.15.46:0/2157182019 conn(0x7fe0d80c7910 msgr2=0x7fe0d8136e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 -- 172.21.15.46:0/2157182019 shutdown_connections 2023-10-30T14:24:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:09.511+0000 7fe0e0217640 1 -- 172.21.15.46:0/2157182019 wait complete. 2023-10-30T14:24:09.537 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 254, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 254, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9735": { "addr": "172.21.15.46:6836/3574404756", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3574404756, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3574404756, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 9735, "incarnation": 246, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 33 }, "gid_9882": { "addr": "172.21.15.46:6833/3394097058", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 3394097058, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 3394097058, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9882, "incarnation": 253, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:resolve", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 91, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:24:05.056866+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9735, "mds_1": 9882 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:24:11.538 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c162dc640 1 Processor -- start 2023-10-30T14:24:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c162dc640 1 -- start start 2023-10-30T14:24:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c162dc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c101538f0 0x7f0c10159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c162dc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c1015cd50 0x7f0c100c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c162dc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c1015d300 con 0x7f0c100cefa0 2023-10-30T14:24:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c162dc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c1015d5e0 con 0x7f0c101538f0 2023-10-30T14:24:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c162dc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1015d8c0 con 0x7f0c1015cd50 2023-10-30T14:24:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c152da640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c101538f0 0x7f0c10159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c15adb640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0c100cefa0 0x7f0c100cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44776/0 (socket says 172.21.15.46:44776) 2023-10-30T14:24:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c15adb640 1 -- 172.21.15.46:0/3406890738 learned_addr learned my addr 172.21.15.46:0/3406890738 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c14ad9640 1 --2- 172.21.15.46:0/3406890738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c1015cd50 0x7f0c100c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c152da640 1 -- 172.21.15.46:0/3406890738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c1015cd50 msgr2=0x7f0c100c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c152da640 1 --2- 172.21.15.46:0/3406890738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c1015cd50 0x7f0c100c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c152da640 1 -- 172.21.15.46:0/3406890738 >> v1:172.21.15.46:6789/0 conn(0x7f0c100cefa0 legacy=0x7f0c100cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c152da640 1 -- 172.21.15.46:0/3406890738 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1015dba0 con 0x7f0c101538f0 2023-10-30T14:24:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c07fff640 1 -- 172.21.15.46:0/3406890738 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205755160 0 0) 0x7f0c1015d300 con 0x7f0c100cefa0 2023-10-30T14:24:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.691+0000 7f0c14ad9640 1 --2- 172.21.15.46:0/3406890738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c1015cd50 0x7f0c100c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:24:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.695+0000 7f0c152da640 1 --2- 172.21.15.46:0/3406890738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c101538f0 0x7f0c10159fb0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0097c0 tx=0x7f0c0c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89898eb125fd32e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.695+0000 7f0c07fff640 1 -- 172.21.15.46:0/3406890738 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c0a8970 con 0x7f0c101538f0 2023-10-30T14:24:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.695+0000 7f0c07fff640 1 -- 172.21.15.46:0/3406890738 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0c0a8ad0 con 0x7f0c101538f0 2023-10-30T14:24:11.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.695+0000 7f0c07fff640 1 -- 172.21.15.46:0/3406890738 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c0a8e60 con 0x7f0c101538f0 2023-10-30T14:24:11.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.695+0000 7f0c162dc640 1 -- 172.21.15.46:0/3406890738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c101538f0 msgr2=0x7f0c10159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:11.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.695+0000 7f0c162dc640 1 --2- 172.21.15.46:0/3406890738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c101538f0 0x7f0c10159fb0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0097c0 tx=0x7f0c0c01ec70 comp rx=0 tx=0).stop 2023-10-30T14:24:11.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.695+0000 7f0c162dc640 1 -- 172.21.15.46:0/3406890738 shutdown_connections 2023-10-30T14:24:11.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.695+0000 7f0c162dc640 1 --2- 172.21.15.46:0/3406890738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c1015cd50 0x7f0c100c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.695+0000 7f0c162dc640 1 --2- 172.21.15.46:0/3406890738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c101538f0 0x7f0c10159fb0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.695+0000 7f0c162dc640 1 -- 172.21.15.46:0/3406890738 >> 172.21.15.46:0/3406890738 conn(0x7f0c1006a640 msgr2=0x7f0c10065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:11.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.695+0000 7f0c162dc640 1 -- 172.21.15.46:0/3406890738 shutdown_connections 2023-10-30T14:24:11.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.695+0000 7f0c162dc640 1 -- 172.21.15.46:0/3406890738 wait complete. 2023-10-30T14:24:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c162dc640 1 Processor -- start 2023-10-30T14:24:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c162dc640 1 -- start start 2023-10-30T14:24:11.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c162dc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c100cefa0 0x7f0c1015c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:11.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c162dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c101538f0 0x7f0c10147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:11.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c162dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1015cd50 0x7f0c100ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:11.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c162dc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c101467b0 con 0x7f0c1015cd50 2023-10-30T14:24:11.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c162dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1015aaa0 con 0x7f0c101538f0 2023-10-30T14:24:11.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c162dc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1015c3b0 con 0x7f0c100cefa0 2023-10-30T14:24:11.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c15adb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1015cd50 0x7f0c100ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:11.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c15adb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1015cd50 0x7f0c100ce1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58516/0 (socket says 172.21.15.46:58516) 2023-10-30T14:24:11.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c15adb640 1 -- 172.21.15.46:0/2556847906 learned_addr learned my addr 172.21.15.46:0/2556847906 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:11.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c152da640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c100cefa0 0x7f0c1015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:11.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c14ad9640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c101538f0 0x7f0c10147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:11.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c15adb640 1 -- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c100cefa0 msgr2=0x7f0c1015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:11.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c15adb640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c100cefa0 0x7f0c1015c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c15adb640 1 -- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c101538f0 msgr2=0x7f0c10147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:11.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c15adb640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c101538f0 0x7f0c10147200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c15adb640 1 -- 172.21.15.46:0/2556847906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c009420 con 0x7f0c1015cd50 2023-10-30T14:24:11.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c15adb640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1015cd50 0x7f0c100ce1c0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f0c08001570 tx=0x7f0c0800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:11.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c05ffb640 1 -- 172.21.15.46:0/2556847906 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0800a640 con 0x7f0c1015cd50 2023-10-30T14:24:11.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c05ffb640 1 -- 172.21.15.46:0/2556847906 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0800a7c0 con 0x7f0c1015cd50 2023-10-30T14:24:11.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c162dc640 1 -- 172.21.15.46:0/2556847906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c10147740 con 0x7f0c1015cd50 2023-10-30T14:24:11.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c05ffb640 1 -- 172.21.15.46:0/2556847906 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0800ab70 con 0x7f0c1015cd50 2023-10-30T14:24:11.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c162dc640 1 -- 172.21.15.46:0/2556847906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1015b040 con 0x7f0c1015cd50 2023-10-30T14:24:11.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.699+0000 7f0c162dc640 1 -- 172.21.15.46:0/2556847906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c1004e170 con 0x7f0c1015cd50 2023-10-30T14:24:11.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.703+0000 7f0c05ffb640 1 -- 172.21.15.46:0/2556847906 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c0800acd0 con 0x7f0c1015cd50 2023-10-30T14:24:11.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.703+0000 7f0c05ffb640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0bf40664a0 0x7f0bf4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:11.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.703+0000 7f0c05ffb640 1 -- 172.21.15.46:0/2556847906 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8650+0+0 (secure 0 0 0) 0x7f0c080842f0 con 0x7f0c1015cd50 2023-10-30T14:24:11.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.703+0000 7f0c152da640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0bf40664a0 0x7f0bf4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:11.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.703+0000 7f0c152da640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0bf40664a0 0x7f0bf4068960 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0b2000 tx=0x7f0c0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:11.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.707+0000 7f0c05ffb640 1 -- 172.21.15.46:0/2556847906 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0c08050b40 con 0x7f0c1015cd50 2023-10-30T14:24:11.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.895+0000 7f0c162dc640 1 -- 172.21.15.46:0/2556847906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c100b2290 con 0x7f0c1015cd50 2023-10-30T14:24:11.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.895+0000 7f0c05ffb640 1 -- 172.21.15.46:0/2556847906 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 255 v255) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f0c08018020 con 0x7f0c1015cd50 2023-10-30T14:24:11.899 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:11.900 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":255,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":255,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:10.956351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:11.900 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 255 2023-10-30T14:24:11.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.899+0000 7f0c162dc640 1 -- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0bf40664a0 msgr2=0x7f0bf4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:11.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.899+0000 7f0c162dc640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0bf40664a0 0x7f0bf4068960 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0b2000 tx=0x7f0c0c008040 comp rx=0 tx=0).stop 2023-10-30T14:24:11.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.899+0000 7f0c162dc640 1 -- 172.21.15.46:0/2556847906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1015cd50 msgr2=0x7f0c100ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:11.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.899+0000 7f0c162dc640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1015cd50 0x7f0c100ce1c0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f0c08001570 tx=0x7f0c0800cca0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.899+0000 7f0c162dc640 1 -- 172.21.15.46:0/2556847906 shutdown_connections 2023-10-30T14:24:11.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.899+0000 7f0c162dc640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0bf40664a0 0x7f0bf4068960 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.899+0000 7f0c162dc640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c1015cd50 0x7f0c100ce1c0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.899+0000 7f0c162dc640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c101538f0 0x7f0c10147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.899+0000 7f0c162dc640 1 --2- 172.21.15.46:0/2556847906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0c100cefa0 0x7f0c1015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:11.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.899+0000 7f0c162dc640 1 -- 172.21.15.46:0/2556847906 >> 172.21.15.46:0/2556847906 conn(0x7f0c1006a640 msgr2=0x7f0c100caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:11.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.899+0000 7f0c162dc640 1 -- 172.21.15.46:0/2556847906 shutdown_connections 2023-10-30T14:24:11.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:11.899+0000 7f0c162dc640 1 -- 172.21.15.46:0/2556847906 wait complete. 2023-10-30T14:24:11.922 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:13.923 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:14.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.075+0000 7fa87f71d640 1 Processor -- start 2023-10-30T14:24:14.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.075+0000 7fa87f71d640 1 -- start start 2023-10-30T14:24:14.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.079+0000 7fa87f71d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa878153d70 0x7fa87806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:14.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.079+0000 7fa87f71d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87806afa0 con 0x7fa87806e470 2023-10-30T14:24:14.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.079+0000 7fa87f71d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa878154150 con 0x7fa8780c8490 2023-10-30T14:24:14.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.079+0000 7fa87f71d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa87806d780 con 0x7fa878153d70 2023-10-30T14:24:14.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.079+0000 7fa87ccb8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa87806e470 0x7fa8780cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44778/0 (socket says 172.21.15.46:44778) 2023-10-30T14:24:14.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.079+0000 7fa87ccb8640 1 -- 172.21.15.46:0/4095334379 learned_addr learned my addr 172.21.15.46:0/4095334379 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:14.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.079+0000 7fa87d4b9640 1 --2- 172.21.15.46:0/4095334379 >> v2:172.21.15.46:3300/0 conn(0x7fa878153d70 0x7fa87806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:14.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.079+0000 7fa87d4b9640 1 -- 172.21.15.46:0/4095334379 >> v1:172.21.15.154:6789/0 conn(0x7fa8780c8490 legacy=0x7fa8780c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.079+0000 7fa87d4b9640 1 -- 172.21.15.46:0/4095334379 >> v1:172.21.15.46:6789/0 conn(0x7fa87806e470 legacy=0x7fa8780cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.079+0000 7fa87d4b9640 1 -- 172.21.15.46:0/4095334379 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87806e030 con 0x7fa878153d70 2023-10-30T14:24:14.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.079+0000 7fa86ffff640 1 -- 172.21.15.46:0/4095334379 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268291949 0 0) 0x7fa87806afa0 con 0x7fa87806e470 2023-10-30T14:24:14.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87d4b9640 1 --2- 172.21.15.46:0/4095334379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa878153d70 0x7fa87806ca80 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fa8740097c0 tx=0x7fa87401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54dfd0d2908f7201 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:14.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa86ffff640 1 -- 172.21.15.46:0/4095334379 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8740ab650 con 0x7fa878153d70 2023-10-30T14:24:14.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa86ffff640 1 -- 172.21.15.46:0/4095334379 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8740ab7b0 con 0x7fa878153d70 2023-10-30T14:24:14.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa86ffff640 1 -- 172.21.15.46:0/4095334379 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8740abb40 con 0x7fa878153d70 2023-10-30T14:24:14.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 -- 172.21.15.46:0/4095334379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa878153d70 msgr2=0x7fa87806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 --2- 172.21.15.46:0/4095334379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa878153d70 0x7fa87806ca80 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fa8740097c0 tx=0x7fa87401c450 comp rx=0 tx=0).stop 2023-10-30T14:24:14.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 -- 172.21.15.46:0/4095334379 shutdown_connections 2023-10-30T14:24:14.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 --2- 172.21.15.46:0/4095334379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa878153d70 0x7fa87806ca80 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 -- 172.21.15.46:0/4095334379 >> 172.21.15.46:0/4095334379 conn(0x7fa878071ea0 msgr2=0x7fa878065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:14.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 -- 172.21.15.46:0/4095334379 shutdown_connections 2023-10-30T14:24:14.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 -- 172.21.15.46:0/4095334379 wait complete. 2023-10-30T14:24:14.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 Processor -- start 2023-10-30T14:24:14.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 -- start start 2023-10-30T14:24:14.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa87806e470 0x7fa878141b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:14.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8780c8490 0x7fa878142050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:14.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153d70 0x7fa8781f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:14.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8780cc430 con 0x7fa87806e470 2023-10-30T14:24:14.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa878149c40 con 0x7fa8780c8490 2023-10-30T14:24:14.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.083+0000 7fa87f71d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8780cbeb0 con 0x7fa878153d70 2023-10-30T14:24:14.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87d4b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa87806e470 0x7fa878141b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:14.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87d4b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa87806e470 0x7fa878141b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58526/0 (socket says 172.21.15.46:58526) 2023-10-30T14:24:14.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87d4b9640 1 -- 172.21.15.46:0/3498321254 learned_addr learned my addr 172.21.15.46:0/3498321254 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:14.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87ccb8640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8780c8490 0x7fa878142050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:14.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87dcba640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153d70 0x7fa8781f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:14.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87d4b9640 1 -- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153d70 msgr2=0x7fa8781f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87d4b9640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153d70 0x7fa8781f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87d4b9640 1 -- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8780c8490 msgr2=0x7fa878142050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87d4b9640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8780c8490 0x7fa878142050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87d4b9640 1 -- 172.21.15.46:0/3498321254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa874009420 con 0x7fa87806e470 2023-10-30T14:24:14.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87ccb8640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8780c8490 0x7fa878142050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:24:14.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87d4b9640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa87806e470 0x7fa878141b10 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fa8740ad040 tx=0x7fa87401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:14.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa86dffb640 1 -- 172.21.15.46:0/3498321254 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8740b5810 con 0x7fa87806e470 2023-10-30T14:24:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa86dffb640 1 -- 172.21.15.46:0/3498321254 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8740b5990 con 0x7fa87806e470 2023-10-30T14:24:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87f71d640 1 -- 172.21.15.46:0/3498321254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa878149070 con 0x7fa87806e470 2023-10-30T14:24:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa86dffb640 1 -- 172.21.15.46:0/3498321254 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8740b5d40 con 0x7fa87806e470 2023-10-30T14:24:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87f71d640 1 -- 172.21.15.46:0/3498321254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa878149310 con 0x7fa87806e470 2023-10-30T14:24:14.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa87f71d640 1 -- 172.21.15.46:0/3498321254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87804e170 con 0x7fa87806e470 2023-10-30T14:24:14.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.087+0000 7fa86dffb640 1 -- 172.21.15.46:0/3498321254 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa8740bf410 con 0x7fa87806e470 2023-10-30T14:24:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.091+0000 7fa86dffb640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa85c0664a0 0x7fa85c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.091+0000 7fa86dffb640 1 -- 172.21.15.46:0/3498321254 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8650+0+0 (secure 0 0 0) 0x7fa87411f830 con 0x7fa87806e470 2023-10-30T14:24:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.091+0000 7fa87ccb8640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa85c0664a0 0x7fa85c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.091+0000 7fa87ccb8640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa85c0664a0 0x7fa85c068960 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fa868007fd0 tx=0x7fa868007c80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:14.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.095+0000 7fa86dffb640 1 -- 172.21.15.46:0/3498321254 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa8740ec110 con 0x7fa87806e470 2023-10-30T14:24:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.283+0000 7fa87f71d640 1 -- 172.21.15.46:0/3498321254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa87806c2a0 con 0x7fa87806e470 2023-10-30T14:24:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.283+0000 7fa86dffb640 1 -- 172.21.15.46:0/3498321254 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 255 v255) v1 ==== 95+0+5736 (secure 0 0 0) 0x7fa8740ef730 con 0x7fa87806e470 2023-10-30T14:24:14.287 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:14.288 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":255,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":255,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:10.956351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:14.288 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 255 2023-10-30T14:24:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.287+0000 7fa87f71d640 1 -- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa85c0664a0 msgr2=0x7fa85c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.287+0000 7fa87f71d640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa85c0664a0 0x7fa85c068960 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fa868007fd0 tx=0x7fa868007c80 comp rx=0 tx=0).stop 2023-10-30T14:24:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.287+0000 7fa87f71d640 1 -- 172.21.15.46:0/3498321254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa87806e470 msgr2=0x7fa878141b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.287+0000 7fa87f71d640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa87806e470 0x7fa878141b10 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fa8740ad040 tx=0x7fa87401ac90 comp rx=0 tx=0).stop 2023-10-30T14:24:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.287+0000 7fa87f71d640 1 -- 172.21.15.46:0/3498321254 shutdown_connections 2023-10-30T14:24:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.287+0000 7fa87f71d640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa85c0664a0 0x7fa85c068960 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.287+0000 7fa87f71d640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153d70 0x7fa8781f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.287+0000 7fa87f71d640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8780c8490 0x7fa878142050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.287+0000 7fa87f71d640 1 --2- 172.21.15.46:0/3498321254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa87806e470 0x7fa878141b10 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.287+0000 7fa87f71d640 1 -- 172.21.15.46:0/3498321254 >> 172.21.15.46:0/3498321254 conn(0x7fa878071ea0 msgr2=0x7fa878150d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.287+0000 7fa87f71d640 1 -- 172.21.15.46:0/3498321254 shutdown_connections 2023-10-30T14:24:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:14.287+0000 7fa87f71d640 1 -- 172.21.15.46:0/3498321254 wait complete. 2023-10-30T14:24:14.309 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:16.310 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:16.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb56f96640 1 Processor -- start 2023-10-30T14:24:16.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb56f96640 1 -- start start 2023-10-30T14:24:16.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb56f96640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feb501538f0 0x7feb50159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb56f96640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb500ca6a0 con 0x7feb500cefa0 2023-10-30T14:24:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb56f96640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5015d300 con 0x7feb5015cd50 2023-10-30T14:24:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb56f96640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb5015d5e0 con 0x7feb501538f0 2023-10-30T14:24:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb56795640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7feb500cefa0 0x7feb501640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60312/0 (socket says 172.21.15.46:60312) 2023-10-30T14:24:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb56795640 1 -- 172.21.15.46:0/2576579355 learned_addr learned my addr 172.21.15.46:0/2576579355 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb55f94640 1 --2- 172.21.15.46:0/2576579355 >> v2:172.21.15.154:3300/0 conn(0x7feb501538f0 0x7feb50159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb54f92640 1 -- 172.21.15.46:0/2576579355 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530063280 0 0) 0x7feb500ca6a0 con 0x7feb500cefa0 2023-10-30T14:24:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb54f92640 1 -- 172.21.15.46:0/2576579355 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb3c003530 con 0x7feb500cefa0 2023-10-30T14:24:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb54f92640 1 -- 172.21.15.46:0/2576579355 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 73995261 0 0) 0x7feb3c003530 con 0x7feb500cefa0 2023-10-30T14:24:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb55f94640 1 -- 172.21.15.46:0/2576579355 >> v1:172.21.15.154:6789/0 conn(0x7feb5015cd50 legacy=0x7feb500c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb55f94640 1 -- 172.21.15.46:0/2576579355 >> v1:172.21.15.46:6789/0 conn(0x7feb500cefa0 legacy=0x7feb501640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.463+0000 7feb55f94640 1 -- 172.21.15.46:0/2576579355 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5015d8c0 con 0x7feb501538f0 2023-10-30T14:24:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.467+0000 7feb55f94640 1 --2- 172.21.15.46:0/2576579355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb501538f0 0x7feb50159fb0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0097c0 tx=0x7feb4c01eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2836db61b254108 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.467+0000 7feb54f92640 1 -- 172.21.15.46:0/2576579355 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb48001a00 con 0x7feb501538f0 2023-10-30T14:24:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.467+0000 7feb54f92640 1 -- 172.21.15.46:0/2576579355 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4c0a8610 con 0x7feb501538f0 2023-10-30T14:24:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.467+0000 7feb54f92640 1 -- 172.21.15.46:0/2576579355 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c0a8980 con 0x7feb501538f0 2023-10-30T14:24:16.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.467+0000 7feb56f96640 1 -- 172.21.15.46:0/2576579355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb501538f0 msgr2=0x7feb50159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.467+0000 7feb56f96640 1 --2- 172.21.15.46:0/2576579355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb501538f0 0x7feb50159fb0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0097c0 tx=0x7feb4c01eb90 comp rx=0 tx=0).stop 2023-10-30T14:24:16.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.467+0000 7feb56f96640 1 -- 172.21.15.46:0/2576579355 shutdown_connections 2023-10-30T14:24:16.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.467+0000 7feb56f96640 1 --2- 172.21.15.46:0/2576579355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb501538f0 0x7feb50159fb0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.467+0000 7feb56f96640 1 -- 172.21.15.46:0/2576579355 >> 172.21.15.46:0/2576579355 conn(0x7feb5006a640 msgr2=0x7feb50065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:16.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.467+0000 7feb56f96640 1 -- 172.21.15.46:0/2576579355 shutdown_connections 2023-10-30T14:24:16.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.467+0000 7feb56f96640 1 -- 172.21.15.46:0/2576579355 wait complete. 2023-10-30T14:24:16.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb56f96640 1 Processor -- start 2023-10-30T14:24:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb56f96640 1 -- start start 2023-10-30T14:24:16.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb56f96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb500cefa0 0x7feb50151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:16.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb56f96640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb501538f0 0x7feb50147990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb56f96640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb5015cd50 0x7feb501f2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb56f96640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5015a890 con 0x7feb500cefa0 2023-10-30T14:24:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb56f96640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb50140a60 con 0x7feb5015cd50 2023-10-30T14:24:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb56f96640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb50140d40 con 0x7feb501538f0 2023-10-30T14:24:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb55f94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb500cefa0 0x7feb50151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb55f94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb500cefa0 0x7feb50151020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38156/0 (socket says 172.21.15.46:38156) 2023-10-30T14:24:16.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb55f94640 1 -- 172.21.15.46:0/2569408277 learned_addr learned my addr 172.21.15.46:0/2569408277 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:16.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb56795640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb5015cd50 0x7feb501f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:16.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb55793640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb501538f0 0x7feb50147990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:16.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb55f94640 1 -- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb501538f0 msgr2=0x7feb50147990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb55f94640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb501538f0 0x7feb50147990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb55f94640 1 -- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb5015cd50 msgr2=0x7feb501f2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb55f94640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb5015cd50 0x7feb501f2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb55f94640 1 -- 172.21.15.46:0/2569408277 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c009420 con 0x7feb500cefa0 2023-10-30T14:24:16.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb55f94640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb500cefa0 0x7feb50151020 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7feb4c01b7a0 tx=0x7feb4c01b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:16.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb467fc640 1 -- 172.21.15.46:0/2569408277 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c0bf070 con 0x7feb500cefa0 2023-10-30T14:24:16.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb467fc640 1 -- 172.21.15.46:0/2569408277 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4c006e20 con 0x7feb500cefa0 2023-10-30T14:24:16.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb467fc640 1 -- 172.21.15.46:0/2569408277 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c0b08b0 con 0x7feb500cefa0 2023-10-30T14:24:16.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.471+0000 7feb56f96640 1 -- 172.21.15.46:0/2569408277 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb50141020 con 0x7feb500cefa0 2023-10-30T14:24:16.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.475+0000 7feb56f96640 1 -- 172.21.15.46:0/2569408277 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5015ab30 con 0x7feb500cefa0 2023-10-30T14:24:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.475+0000 7feb56f96640 1 -- 172.21.15.46:0/2569408277 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5004e170 con 0x7feb500cefa0 2023-10-30T14:24:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.475+0000 7feb467fc640 1 -- 172.21.15.46:0/2569408277 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feb4c0b7040 con 0x7feb500cefa0 2023-10-30T14:24:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.475+0000 7feb467fc640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb340664d0 0x7feb34068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.475+0000 7feb467fc640 1 -- 172.21.15.46:0/2569408277 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8650+0+0 (secure 0 0 0) 0x7feb4c121910 con 0x7feb500cefa0 2023-10-30T14:24:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.475+0000 7feb55793640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb340664d0 0x7feb34068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.475+0000 7feb55793640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb340664d0 0x7feb34068990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7feb501481f0 tx=0x7feb40009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:16.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.479+0000 7feb467fc640 1 -- 172.21.15.46:0/2569408277 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feb4c0ee160 con 0x7feb500cefa0 2023-10-30T14:24:16.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.671+0000 7feb56f96640 1 -- 172.21.15.46:0/2569408277 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb50157180 con 0x7feb500cefa0 2023-10-30T14:24:16.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.671+0000 7feb467fc640 1 -- 172.21.15.46:0/2569408277 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 255 v255) v1 ==== 95+0+5736 (secure 0 0 0) 0x7feb4c0ae340 con 0x7feb500cefa0 2023-10-30T14:24:16.675 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:16.675 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":255,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":255,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:10.956351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:16.676 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 255 2023-10-30T14:24:16.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.675+0000 7feb56f96640 1 -- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb340664d0 msgr2=0x7feb34068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.675+0000 7feb56f96640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb340664d0 0x7feb34068990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7feb501481f0 tx=0x7feb40009040 comp rx=0 tx=0).stop 2023-10-30T14:24:16.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.675+0000 7feb56f96640 1 -- 172.21.15.46:0/2569408277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb500cefa0 msgr2=0x7feb50151020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:16.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.675+0000 7feb56f96640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb500cefa0 0x7feb50151020 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7feb4c01b7a0 tx=0x7feb4c01b930 comp rx=0 tx=0).stop 2023-10-30T14:24:16.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.675+0000 7feb56f96640 1 -- 172.21.15.46:0/2569408277 shutdown_connections 2023-10-30T14:24:16.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.675+0000 7feb56f96640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7feb340664d0 0x7feb34068990 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.675+0000 7feb56f96640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb5015cd50 0x7feb501f2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.675+0000 7feb56f96640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb501538f0 0x7feb50147990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.675+0000 7feb56f96640 1 --2- 172.21.15.46:0/2569408277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb500cefa0 0x7feb50151020 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.675+0000 7feb56f96640 1 -- 172.21.15.46:0/2569408277 >> 172.21.15.46:0/2569408277 conn(0x7feb5006a640 msgr2=0x7feb500cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.675+0000 7feb56f96640 1 -- 172.21.15.46:0/2569408277 shutdown_connections 2023-10-30T14:24:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:16.675+0000 7feb56f96640 1 -- 172.21.15.46:0/2569408277 wait complete. 2023-10-30T14:24:16.696 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:18.697 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:18.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd534a68640 1 Processor -- start 2023-10-30T14:24:18.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd534a68640 1 -- start start 2023-10-30T14:24:18.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd534a68640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53006f7a0 0x7fd53006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:18.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd534a68640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd530071110 0x7fd530158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:18.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd534a68640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd530066cf0 con 0x7fd5300c6bd0 2023-10-30T14:24:18.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd534a68640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5300700c0 con 0x7fd530071110 2023-10-30T14:24:18.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd534a68640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300c70d0 con 0x7fd53006f7a0 2023-10-30T14:24:18.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd52dd9b640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd5300c6bd0 0x7fd530069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:41418/0 (socket says 172.21.15.46:41418) 2023-10-30T14:24:18.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd52dd9b640 1 -- 172.21.15.46:0/2906926203 learned_addr learned my addr 172.21.15.46:0/2906926203 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:18.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd52e59c640 1 --2- 172.21.15.46:0/2906926203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53006f7a0 0x7fd53006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:18.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd52ed9d640 1 --2- 172.21.15.46:0/2906926203 >> v2:172.21.15.154:3300/0 conn(0x7fd530071110 0x7fd530158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:18.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd52d59a640 1 -- 172.21.15.46:0/2906926203 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254918963 0 0) 0x7fd530066cf0 con 0x7fd5300c6bd0 2023-10-30T14:24:18.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd52e59c640 1 -- 172.21.15.46:0/2906926203 >> v2:172.21.15.154:3300/0 conn(0x7fd530071110 msgr2=0x7fd530158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:18.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd52e59c640 1 --2- 172.21.15.46:0/2906926203 >> v2:172.21.15.154:3300/0 conn(0x7fd530071110 0x7fd530158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:18.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd52e59c640 1 -- 172.21.15.46:0/2906926203 >> v1:172.21.15.154:6789/0 conn(0x7fd5300c6bd0 legacy=0x7fd530069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:18.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.851+0000 7fd52e59c640 1 -- 172.21.15.46:0/2906926203 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5300c7390 con 0x7fd53006f7a0 2023-10-30T14:24:18.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.855+0000 7fd52e59c640 1 --2- 172.21.15.46:0/2906926203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53006f7a0 0x7fd53006fb80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5200097c0 tx=0x7fd52001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91728f692a829016 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:18.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.855+0000 7fd52d59a640 1 -- 172.21.15.46:0/2906926203 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5200a8970 con 0x7fd53006f7a0 2023-10-30T14:24:18.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.855+0000 7fd52d59a640 1 -- 172.21.15.46:0/2906926203 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5200a8ad0 con 0x7fd53006f7a0 2023-10-30T14:24:18.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.855+0000 7fd52d59a640 1 -- 172.21.15.46:0/2906926203 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5200a8e60 con 0x7fd53006f7a0 2023-10-30T14:24:18.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.855+0000 7fd534a68640 1 -- 172.21.15.46:0/2906926203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53006f7a0 msgr2=0x7fd53006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:18.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.855+0000 7fd534a68640 1 --2- 172.21.15.46:0/2906926203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53006f7a0 0x7fd53006fb80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5200097c0 tx=0x7fd52001ec70 comp rx=0 tx=0).stop 2023-10-30T14:24:18.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.855+0000 7fd534a68640 1 -- 172.21.15.46:0/2906926203 shutdown_connections 2023-10-30T14:24:18.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.855+0000 7fd534a68640 1 --2- 172.21.15.46:0/2906926203 >> v2:172.21.15.154:3300/0 conn(0x7fd530071110 0x7fd530158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:18.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.855+0000 7fd534a68640 1 --2- 172.21.15.46:0/2906926203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53006f7a0 0x7fd53006fb80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:18.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.855+0000 7fd534a68640 1 -- 172.21.15.46:0/2906926203 >> 172.21.15.46:0/2906926203 conn(0x7fd5300c7910 msgr2=0x7fd5300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.855+0000 7fd534a68640 1 -- 172.21.15.46:0/2906926203 shutdown_connections 2023-10-30T14:24:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.855+0000 7fd534a68640 1 -- 172.21.15.46:0/2906926203 wait complete. 2023-10-30T14:24:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd534a68640 1 Processor -- start 2023-10-30T14:24:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd534a68640 1 -- start start 2023-10-30T14:24:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd534a68640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53006f7a0 0x7fd5301ee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:18.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd534a68640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd530071110 0x7fd5300ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:18.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd534a68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5300c6bd0 0x7fd5300aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:18.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd534a68640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300afd20 con 0x7fd530071110 2023-10-30T14:24:18.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd534a68640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300ad8d0 con 0x7fd5300c6bd0 2023-10-30T14:24:18.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd534a68640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300c8410 con 0x7fd53006f7a0 2023-10-30T14:24:18.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd52dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd530071110 0x7fd5300ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:18.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd52dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd530071110 0x7fd5300ae880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38160/0 (socket says 172.21.15.46:38160) 2023-10-30T14:24:18.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd52dd9b640 1 -- 172.21.15.46:0/4155498770 learned_addr learned my addr 172.21.15.46:0/4155498770 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:18.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd52e59c640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53006f7a0 0x7fd5301ee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:18.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd52ed9d640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5300c6bd0 0x7fd5300aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:18.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd52dd9b640 1 -- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53006f7a0 msgr2=0x7fd5301ee9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:18.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd52dd9b640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53006f7a0 0x7fd5301ee9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:18.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd52dd9b640 1 -- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5300c6bd0 msgr2=0x7fd5300aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:18.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd52dd9b640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5300c6bd0 0x7fd5300aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:18.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd52dd9b640 1 -- 172.21.15.46:0/4155498770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd520009420 con 0x7fd530071110 2023-10-30T14:24:18.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd52dd9b640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd530071110 0x7fd5300ae880 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fd5240011f0 tx=0x7fd52400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:18.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd51effd640 1 -- 172.21.15.46:0/4155498770 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5240074e0 con 0x7fd530071110 2023-10-30T14:24:18.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd51effd640 1 -- 172.21.15.46:0/4155498770 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd524007640 con 0x7fd530071110 2023-10-30T14:24:18.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd51effd640 1 -- 172.21.15.46:0/4155498770 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd524004a10 con 0x7fd530071110 2023-10-30T14:24:18.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.859+0000 7fd534a68640 1 -- 172.21.15.46:0/4155498770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5301ef2b0 con 0x7fd530071110 2023-10-30T14:24:18.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.863+0000 7fd534a68640 1 -- 172.21.15.46:0/4155498770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5300adb00 con 0x7fd530071110 2023-10-30T14:24:18.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.863+0000 7fd534a68640 1 -- 172.21.15.46:0/4155498770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53004e170 con 0x7fd530071110 2023-10-30T14:24:18.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.863+0000 7fd51effd640 1 -- 172.21.15.46:0/4155498770 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd524004b70 con 0x7fd530071110 2023-10-30T14:24:18.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.863+0000 7fd51effd640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5100664a0 0x7fd510068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:18.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.863+0000 7fd51effd640 1 -- 172.21.15.46:0/4155498770 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8650+0+0 (secure 0 0 0) 0x7fd524083c30 con 0x7fd530071110 2023-10-30T14:24:18.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.863+0000 7fd52e59c640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5100664a0 0x7fd510068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:18.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.867+0000 7fd52e59c640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5100664a0 0x7fd510068960 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd5200b2000 tx=0x7fd5200aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:18.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:18.867+0000 7fd51effd640 1 -- 172.21.15.46:0/4155498770 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd5240504a0 con 0x7fd530071110 2023-10-30T14:24:19.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.059+0000 7fd534a68640 1 -- 172.21.15.46:0/4155498770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd530070900 con 0x7fd530071110 2023-10-30T14:24:19.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.059+0000 7fd51effd640 1 -- 172.21.15.46:0/4155498770 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 256 v256) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fd524018030 con 0x7fd530071110 2023-10-30T14:24:19.062 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:19.063 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":256,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":256,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:16.890912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:19.063 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 256 2023-10-30T14:24:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.063+0000 7fd534a68640 1 -- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5100664a0 msgr2=0x7fd510068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.063+0000 7fd534a68640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5100664a0 0x7fd510068960 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd5200b2000 tx=0x7fd5200aa070 comp rx=0 tx=0).stop 2023-10-30T14:24:19.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.063+0000 7fd534a68640 1 -- 172.21.15.46:0/4155498770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd530071110 msgr2=0x7fd5300ae880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.063+0000 7fd534a68640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd530071110 0x7fd5300ae880 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fd5240011f0 tx=0x7fd52400c9b0 comp rx=0 tx=0).stop 2023-10-30T14:24:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.063+0000 7fd534a68640 1 -- 172.21.15.46:0/4155498770 shutdown_connections 2023-10-30T14:24:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.063+0000 7fd534a68640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd5100664a0 0x7fd510068960 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.063+0000 7fd534a68640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5300c6bd0 0x7fd5300aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:19.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.063+0000 7fd534a68640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd530071110 0x7fd5300ae880 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.063+0000 7fd534a68640 1 --2- 172.21.15.46:0/4155498770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53006f7a0 0x7fd5301ee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.063+0000 7fd534a68640 1 -- 172.21.15.46:0/4155498770 >> 172.21.15.46:0/4155498770 conn(0x7fd5300c7910 msgr2=0x7fd5300ca400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.063+0000 7fd534a68640 1 -- 172.21.15.46:0/4155498770 shutdown_connections 2023-10-30T14:24:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:19.063+0000 7fd534a68640 1 -- 172.21.15.46:0/4155498770 wait complete. 2023-10-30T14:24:19.084 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:24:21.085 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:21.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c2af1640 1 Processor -- start 2023-10-30T14:24:21.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c2af1640 1 -- start start 2023-10-30T14:24:21.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c2af1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72bc1538a0 0x7f72bc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:21.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c2af1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72bc0ca0e0 0x7f72bc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:21.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c2af1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72bc0cb800 con 0x7f72bc0cf160 2023-10-30T14:24:21.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c2af1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72bc15dc30 con 0x7f72bc1538a0 2023-10-30T14:24:21.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c2af1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc0cb330 con 0x7f72bc0ca0e0 2023-10-30T14:24:21.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c088d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72bc1538a0 0x7f72bc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:21.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c108e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f72bc0cf160 0x7f72bc0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60314/0 (socket says 172.21.15.46:60314) 2023-10-30T14:24:21.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c108e640 1 -- 172.21.15.46:0/279475464 learned_addr learned my addr 172.21.15.46:0/279475464 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:21.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72bbfff640 1 --2- 172.21.15.46:0/279475464 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72bc0ca0e0 0x7f72bc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:21.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72bb7fe640 1 -- 172.21.15.46:0/279475464 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843527129 0 0) 0x7f72bc0cb800 con 0x7f72bc0cf160 2023-10-30T14:24:21.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c088d640 1 -- 172.21.15.46:0/279475464 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72bc0ca0e0 msgr2=0x7f72bc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:21.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c088d640 1 --2- 172.21.15.46:0/279475464 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72bc0ca0e0 0x7f72bc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:21.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c088d640 1 -- 172.21.15.46:0/279475464 >> v1:172.21.15.46:6789/0 conn(0x7f72bc0cf160 legacy=0x7f72bc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:21.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.239+0000 7f72c088d640 1 -- 172.21.15.46:0/279475464 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72bc0cad70 con 0x7f72bc1538a0 2023-10-30T14:24:21.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.243+0000 7f72c088d640 1 --2- 172.21.15.46:0/279475464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72bc1538a0 0x7f72bc15d6f0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f72b00097c0 tx=0x7f72b001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259ca9ee0a3e92ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:21.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.243+0000 7f72bb7fe640 1 -- 172.21.15.46:0/279475464 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b00a8970 con 0x7f72bc1538a0 2023-10-30T14:24:21.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.243+0000 7f72bb7fe640 1 -- 172.21.15.46:0/279475464 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b00a8ad0 con 0x7f72bc1538a0 2023-10-30T14:24:21.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.243+0000 7f72bb7fe640 1 -- 172.21.15.46:0/279475464 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b00a8e60 con 0x7f72bc1538a0 2023-10-30T14:24:21.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.243+0000 7f72c2af1640 1 -- 172.21.15.46:0/279475464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72bc1538a0 msgr2=0x7f72bc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:21.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.243+0000 7f72c2af1640 1 --2- 172.21.15.46:0/279475464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72bc1538a0 0x7f72bc15d6f0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f72b00097c0 tx=0x7f72b001ec70 comp rx=0 tx=0).stop 2023-10-30T14:24:21.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.243+0000 7f72c2af1640 1 -- 172.21.15.46:0/279475464 shutdown_connections 2023-10-30T14:24:21.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.243+0000 7f72c2af1640 1 --2- 172.21.15.46:0/279475464 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72bc0ca0e0 0x7f72bc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:21.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.243+0000 7f72c2af1640 1 --2- 172.21.15.46:0/279475464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72bc1538a0 0x7f72bc15d6f0 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:21.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.243+0000 7f72c2af1640 1 -- 172.21.15.46:0/279475464 >> 172.21.15.46:0/279475464 conn(0x7f72bc06bfe0 msgr2=0x7f72bc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:21.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.243+0000 7f72c2af1640 1 -- 172.21.15.46:0/279475464 shutdown_connections 2023-10-30T14:24:21.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.243+0000 7f72c2af1640 1 -- 172.21.15.46:0/279475464 wait complete. 2023-10-30T14:24:21.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c2af1640 1 Processor -- start 2023-10-30T14:24:21.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c2af1640 1 -- start start 2023-10-30T14:24:21.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c2af1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72bc0ca0e0 0x7f72bc140530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:21.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c2af1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72bc0cf160 0x7f72bc146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:21.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c2af1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72bc1538a0 0x7f72bc147110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:21.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c2af1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc0ce8c0 con 0x7f72bc0cf160 2023-10-30T14:24:21.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c2af1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc14eed0 con 0x7f72bc1538a0 2023-10-30T14:24:21.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c2af1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc1f3290 con 0x7f72bc0ca0e0 2023-10-30T14:24:21.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72bc0cf160 0x7f72bc146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:21.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72bc0cf160 0x7f72bc146bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38184/0 (socket says 172.21.15.46:38184) 2023-10-30T14:24:21.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72bbfff640 1 -- 172.21.15.46:0/1047315923 learned_addr learned my addr 172.21.15.46:0/1047315923 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:21.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c088d640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72bc0ca0e0 0x7f72bc140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:21.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c108e640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72bc1538a0 0x7f72bc147110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:21.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72bbfff640 1 -- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72bc0ca0e0 msgr2=0x7f72bc140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:21.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72bbfff640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72bc0ca0e0 0x7f72bc140530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:21.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72bbfff640 1 -- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72bc1538a0 msgr2=0x7f72bc147110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:21.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72bbfff640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72bc1538a0 0x7f72bc147110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:21.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72bbfff640 1 -- 172.21.15.46:0/1047315923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b0009420 con 0x7f72bc0cf160 2023-10-30T14:24:21.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c108e640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72bc1538a0 0x7f72bc147110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:24:21.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72bbfff640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72bc0cf160 0x7f72bc146bd0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f72a800e930 tx=0x7f72a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:21.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72b97fa640 1 -- 172.21.15.46:0/1047315923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8009690 con 0x7f72bc0cf160 2023-10-30T14:24:21.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72b97fa640 1 -- 172.21.15.46:0/1047315923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a800c9a0 con 0x7f72bc0cf160 2023-10-30T14:24:21.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72b97fa640 1 -- 172.21.15.46:0/1047315923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a800cd10 con 0x7f72bc0cf160 2023-10-30T14:24:21.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c2af1640 1 -- 172.21.15.46:0/1047315923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72bc1f2430 con 0x7f72bc0cf160 2023-10-30T14:24:21.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.247+0000 7f72c2af1640 1 -- 172.21.15.46:0/1047315923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72bc1f29f0 con 0x7f72bc0cf160 2023-10-30T14:24:21.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.251+0000 7f72c2af1640 1 -- 172.21.15.46:0/1047315923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72bc0b2a50 con 0x7f72bc0cf160 2023-10-30T14:24:21.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.251+0000 7f72b97fa640 1 -- 172.21.15.46:0/1047315923 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f72a8013470 con 0x7f72bc0cf160 2023-10-30T14:24:21.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.251+0000 7f72b97fa640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72a00664a0 0x7f72a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:21.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.251+0000 7f72b97fa640 1 -- 172.21.15.46:0/1047315923 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8650+0+0 (secure 0 0 0) 0x7f72a8084ca0 con 0x7f72bc0cf160 2023-10-30T14:24:21.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.251+0000 7f72c088d640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72a00664a0 0x7f72a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:21.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.251+0000 7f72c088d640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72a00664a0 0x7f72a0068960 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f72b00b2000 tx=0x7f72b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:21.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.255+0000 7f72b97fa640 1 -- 172.21.15.46:0/1047315923 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f72a80515d0 con 0x7f72bc0cf160 2023-10-30T14:24:21.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 -- 172.21.15.46:0/1047315923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72bc04e170 con 0x7f72bc0cf160 2023-10-30T14:24:21.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72b97fa640 1 -- 172.21.15.46:0/1047315923 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 256 v256) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f72a8021020 con 0x7f72bc0cf160 2023-10-30T14:24:21.449 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:21.449 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":256,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":256,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:16.890912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:21.449 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 256 2023-10-30T14:24:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 -- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72a00664a0 msgr2=0x7f72a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72a00664a0 0x7f72a0068960 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f72b00b2000 tx=0x7f72b0008040 comp rx=0 tx=0).stop 2023-10-30T14:24:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 -- 172.21.15.46:0/1047315923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72bc0cf160 msgr2=0x7f72bc146bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72bc0cf160 0x7f72bc146bd0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f72a800e930 tx=0x7f72a800edf0 comp rx=0 tx=0).stop 2023-10-30T14:24:21.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 -- 172.21.15.46:0/1047315923 shutdown_connections 2023-10-30T14:24:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f72a00664a0 0x7f72a0068960 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72bc1538a0 0x7f72bc147110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72bc0cf160 0x7f72bc146bd0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 --2- 172.21.15.46:0/1047315923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72bc0ca0e0 0x7f72bc140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 -- 172.21.15.46:0/1047315923 >> 172.21.15.46:0/1047315923 conn(0x7f72bc06bfe0 msgr2=0x7f72bc1527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 -- 172.21.15.46:0/1047315923 shutdown_connections 2023-10-30T14:24:21.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:21.447+0000 7f72c2af1640 1 -- 172.21.15.46:0/1047315923 wait complete. 2023-10-30T14:24:21.472 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 256, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 256, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_9735": { "addr": "172.21.15.46:6836/3574404756", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6834", "nonce": 3574404756, "type": "v2" }, { "addr": "172.21.15.46:6836", "nonce": 3574404756, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 9735, "incarnation": 246, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 33 }, "gid_9882": { "addr": "172.21.15.46:6833/3394097058", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 3394097058, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 3394097058, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9882, "incarnation": 253, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:rejoin", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 91, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:24:16.890912+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 9735, "mds_1": 9882 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:24:23.473 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:23.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c94a08640 1 Processor -- start 2023-10-30T14:24:23.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c94a08640 1 -- start start 2023-10-30T14:24:23.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c94a08640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c9015cd50 0x7f5c900c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:23.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c94a08640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9015a6c0 con 0x7f5c901538f0 2023-10-30T14:24:23.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c94a08640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9015d300 con 0x7f5c900cefa0 2023-10-30T14:24:23.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c94a08640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9015d5e0 con 0x7f5c9015cd50 2023-10-30T14:24:23.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5c901538f0 0x7f5c90159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60330/0 (socket says 172.21.15.46:60330) 2023-10-30T14:24:23.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8ed9d640 1 -- 172.21.15.46:0/3134888501 learned_addr learned my addr 172.21.15.46:0/3134888501 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:23.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8e59c640 1 --2- 172.21.15.46:0/3134888501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c9015cd50 0x7f5c900c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:23.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958504722 0 0) 0x7f5c9015a6c0 con 0x7f5c901538f0 2023-10-30T14:24:23.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c78003530 con 0x7f5c901538f0 2023-10-30T14:24:23.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1529132917 0 0) 0x7f5c78003530 con 0x7f5c901538f0 2023-10-30T14:24:23.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c9015cd50 msgr2=0x7f5c900c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:23.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 --2- 172.21.15.46:0/3134888501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c9015cd50 0x7f5c900c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:23.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 >> v1:172.21.15.154:6789/0 conn(0x7f5c900cefa0 legacy=0x7f5c901640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:23.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9015d8c0 con 0x7f5c901538f0 2023-10-30T14:24:23.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f5c80001680 con 0x7f5c901538f0 2023-10-30T14:24:23.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 >> v1:172.21.15.46:6789/0 conn(0x7f5c901538f0 legacy=0x7f5c90159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:23.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 --2- 172.21.15.46:0/3134888501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c780039b0 0x7f5c78003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:23.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 --2- 172.21.15.46:0/3134888501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c9015cd50 0x7f5c780047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:23.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 --2- 172.21.15.46:0/3134888501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c78004d10 0x7f5c78005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:23.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9015d5e0 con 0x7f5c780039b0 2023-10-30T14:24:23.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7800b9c0 con 0x7f5c78004d10 2023-10-30T14:24:23.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7800bb80 con 0x7f5c9015cd50 2023-10-30T14:24:23.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8ed9d640 1 --2- 172.21.15.46:0/3134888501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c780039b0 0x7f5c78003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:23.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8f59e640 1 --2- 172.21.15.46:0/3134888501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c78004d10 0x7f5c78005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:23.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8e59c640 1 --2- 172.21.15.46:0/3134888501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c9015cd50 0x7f5c780047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:23.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8ed9d640 1 -- 172.21.15.46:0/3134888501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c9015cd50 msgr2=0x7f5c780047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:23.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8ed9d640 1 --2- 172.21.15.46:0/3134888501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c9015cd50 0x7f5c780047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:23.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8ed9d640 1 -- 172.21.15.46:0/3134888501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c78004d10 msgr2=0x7f5c78005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:23.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8ed9d640 1 --2- 172.21.15.46:0/3134888501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c78004d10 0x7f5c78005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:23.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.627+0000 7f5c8ed9d640 1 -- 172.21.15.46:0/3134888501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c78003530 con 0x7f5c780039b0 2023-10-30T14:24:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.631+0000 7f5c8ed9d640 1 --2- 172.21.15.46:0/3134888501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c780039b0 0x7f5c78003d90 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f5c8000c210 tx=0x7f5c8001cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b814968abeff6f31 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.631+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c80005ae0 con 0x7f5c780039b0 2023-10-30T14:24:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.631+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c80005c60 con 0x7f5c780039b0 2023-10-30T14:24:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.631+0000 7f5c8dd9b640 1 -- 172.21.15.46:0/3134888501 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c800a58a0 con 0x7f5c780039b0 2023-10-30T14:24:23.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.631+0000 7f5c94a08640 1 -- 172.21.15.46:0/3134888501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c780039b0 msgr2=0x7f5c78003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.631+0000 7f5c94a08640 1 --2- 172.21.15.46:0/3134888501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c780039b0 0x7f5c78003d90 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f5c8000c210 tx=0x7f5c8001cba0 comp rx=0 tx=0).stop 2023-10-30T14:24:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.631+0000 7f5c8f59e640 1 -- 172.21.15.46:0/3134888501 reap_dead start 2023-10-30T14:24:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.631+0000 7f5c94a08640 1 -- 172.21.15.46:0/3134888501 shutdown_connections 2023-10-30T14:24:23.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.631+0000 7f5c94a08640 1 -- 172.21.15.46:0/3134888501 >> 172.21.15.46:0/3134888501 conn(0x7f5c9006a640 msgr2=0x7f5c90065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.631+0000 7f5c94a08640 1 -- 172.21.15.46:0/3134888501 shutdown_connections 2023-10-30T14:24:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.631+0000 7f5c94a08640 1 -- 172.21.15.46:0/3134888501 wait complete. 2023-10-30T14:24:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c94a08640 1 Processor -- start 2023-10-30T14:24:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c94a08640 1 -- start start 2023-10-30T14:24:23.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c94a08640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c780039b0 0x7f5c90149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c94a08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9015bf30 0x7f5c901ee750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c94a08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c900cdf60 0x7f5c90145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c94a08640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c901f25f0 con 0x7f5c900cdf60 2023-10-30T14:24:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c94a08640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c900ce980 con 0x7f5c9015bf30 2023-10-30T14:24:23.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c94a08640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c900cec60 con 0x7f5c780039b0 2023-10-30T14:24:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c8f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c900cdf60 0x7f5c90145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c8e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9015bf30 0x7f5c901ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c8f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c900cdf60 0x7f5c90145710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38198/0 (socket says 172.21.15.46:38198) 2023-10-30T14:24:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c8f59e640 1 -- 172.21.15.46:0/3337989506 learned_addr learned my addr 172.21.15.46:0/3337989506 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:23.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c8e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9015bf30 0x7f5c901ee750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:43280/0 (socket says 172.21.15.46:43280) 2023-10-30T14:24:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c8ed9d640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c780039b0 0x7f5c90149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c8f59e640 1 -- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c780039b0 msgr2=0x7f5c90149050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c8f59e640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c780039b0 0x7f5c90149050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c8f59e640 1 -- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9015bf30 msgr2=0x7f5c901ee750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:23.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c8f59e640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9015bf30 0x7f5c901ee750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c8f59e640 1 -- 172.21.15.46:0/3337989506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c901f1cc0 con 0x7f5c900cdf60 2023-10-30T14:24:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c8f59e640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c900cdf60 0x7f5c90145710 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f5c90066a40 tx=0x7f5c8800ae00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c777fe640 1 -- 172.21.15.46:0/3337989506 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c88006e30 con 0x7f5c900cdf60 2023-10-30T14:24:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c777fe640 1 -- 172.21.15.46:0/3337989506 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c88003980 con 0x7f5c900cdf60 2023-10-30T14:24:23.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c94a08640 1 -- 172.21.15.46:0/3337989506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c901f1fa0 con 0x7f5c900cdf60 2023-10-30T14:24:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c777fe640 1 -- 172.21.15.46:0/3337989506 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c88003d30 con 0x7f5c900cdf60 2023-10-30T14:24:23.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.635+0000 7f5c94a08640 1 -- 172.21.15.46:0/3337989506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c901f2a80 con 0x7f5c900cdf60 2023-10-30T14:24:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.639+0000 7f5c94a08640 1 -- 172.21.15.46:0/3337989506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9004e170 con 0x7f5c900cdf60 2023-10-30T14:24:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.639+0000 7f5c777fe640 1 -- 172.21.15.46:0/3337989506 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5c8800f7c0 con 0x7f5c900cdf60 2023-10-30T14:24:23.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.639+0000 7f5c777fe640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c7c0664a0 0x7f5c7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.639+0000 7f5c777fe640 1 -- 172.21.15.46:0/3337989506 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8650+0+0 (secure 0 0 0) 0x7f5c88087330 con 0x7f5c900cdf60 2023-10-30T14:24:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.639+0000 7f5c8ed9d640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c7c0664a0 0x7f5c7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.639+0000 7f5c8ed9d640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c7c0664a0 0x7f5c7c068960 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f5c800ab000 tx=0x7f5c800a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:23.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.643+0000 7f5c777fe640 1 -- 172.21.15.46:0/3337989506 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5c88053c90 con 0x7f5c900cdf60 2023-10-30T14:24:23.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.831+0000 7f5c94a08640 1 -- 172.21.15.46:0/3337989506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c900b2290 con 0x7f5c900cdf60 2023-10-30T14:24:23.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c777fe640 1 -- 172.21.15.46:0/3337989506 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 257 v257) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f5c8801b090 con 0x7f5c900cdf60 2023-10-30T14:24:23.836 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:23.836 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":257,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:22.592577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:23.837 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 257 2023-10-30T14:24:23.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c94a08640 1 -- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c7c0664a0 msgr2=0x7f5c7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:23.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c94a08640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c7c0664a0 0x7f5c7c068960 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f5c800ab000 tx=0x7f5c800a7070 comp rx=0 tx=0).stop 2023-10-30T14:24:23.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c94a08640 1 -- 172.21.15.46:0/3337989506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c900cdf60 msgr2=0x7f5c90145710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:23.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c94a08640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c900cdf60 0x7f5c90145710 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f5c90066a40 tx=0x7f5c8800ae00 comp rx=0 tx=0).stop 2023-10-30T14:24:23.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c94a08640 1 -- 172.21.15.46:0/3337989506 shutdown_connections 2023-10-30T14:24:23.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c94a08640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f5c7c0664a0 0x7f5c7c068960 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:23.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c94a08640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c900cdf60 0x7f5c90145710 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:23.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c94a08640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9015bf30 0x7f5c901ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:23.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c94a08640 1 --2- 172.21.15.46:0/3337989506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c780039b0 0x7f5c90149050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:23.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c94a08640 1 -- 172.21.15.46:0/3337989506 >> 172.21.15.46:0/3337989506 conn(0x7f5c9006a640 msgr2=0x7f5c90159170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:23.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c94a08640 1 -- 172.21.15.46:0/3337989506 shutdown_connections 2023-10-30T14:24:23.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:23.835+0000 7f5c94a08640 1 -- 172.21.15.46:0/3337989506 wait complete. 2023-10-30T14:24:23.859 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=9594 2023-10-30T14:24:23.859 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.e 2023-10-30T14:24:23.867 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T14:24:23.863+0000 7fc739348640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:24:27.860 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T14:24:27.861 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T14:24:27.861 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T14:24:27.865 INFO:tasks.ceph.mds.e:Started 2023-10-30T14:24:27.865 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 31 -eq 48 ] 2023-10-30T14:24:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.31 2023-10-30T14:24:27.984 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.31 2023-10-30T14:24:27.990 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T14:24:28.006 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 32 -eq 48 ] 2023-10-30T14:24:28.006 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.32 2023-10-30T14:24:28.006 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.32 2023-10-30T14:24:28.018 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 33 -eq 48 ] 2023-10-30T14:24:28.018 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.33 2023-10-30T14:24:28.019 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.33 2023-10-30T14:24:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 34 -eq 48 ] 2023-10-30T14:24:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.34 2023-10-30T14:24:28.031 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.34 2023-10-30T14:24:28.045 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 35 -eq 48 ] 2023-10-30T14:24:28.045 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.35 2023-10-30T14:24:28.045 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.35 2023-10-30T14:24:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.055+0000 7f7f5f074640 1 Processor -- start 2023-10-30T14:24:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.055+0000 7f7f5f074640 1 -- start start 2023-10-30T14:24:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.055+0000 7f7f5f074640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f581538f0 0x7f7f58159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.055+0000 7f7f5f074640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f580ca6a0 con 0x7f7f5815cd50 2023-10-30T14:24:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.059+0000 7f7f5f074640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5815d300 con 0x7f7f580cefa0 2023-10-30T14:24:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.059+0000 7f7f5f074640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5815d5e0 con 0x7f7f581538f0 2023-10-30T14:24:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.059+0000 7f7f5d871640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7f5815cd50 0x7f7f580c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40416/0 (socket says 172.21.15.46:40416) 2023-10-30T14:24:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.059+0000 7f7f5d871640 1 -- 172.21.15.46:0/3898091628 learned_addr learned my addr 172.21.15.46:0/3898091628 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.059+0000 7f7f5e072640 1 --2- 172.21.15.46:0/3898091628 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f581538f0 0x7f7f58159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.059+0000 7f7f5d070640 1 -- 172.21.15.46:0/3898091628 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296420383 0 0) 0x7f7f580ca6a0 con 0x7f7f5815cd50 2023-10-30T14:24:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.059+0000 7f7f5d070640 1 -- 172.21.15.46:0/3898091628 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f44003530 con 0x7f7f5815cd50 2023-10-30T14:24:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.059+0000 7f7f5e072640 1 -- 172.21.15.46:0/3898091628 >> v1:172.21.15.154:6789/0 conn(0x7f7f580cefa0 legacy=0x7f7f581640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.059+0000 7f7f5e072640 1 -- 172.21.15.46:0/3898091628 >> v1:172.21.15.46:6789/0 conn(0x7f7f5815cd50 legacy=0x7f7f580c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.059+0000 7f7f5e072640 1 -- 172.21.15.46:0/3898091628 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5815d8c0 con 0x7f7f581538f0 2023-10-30T14:24:28.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.059+0000 7f7f5e072640 1 --2- 172.21.15.46:0/3898091628 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f581538f0 0x7f7f58159fb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7f540097f0 tx=0x7f7f5401ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b167491822452c19 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5d070640 1 -- 172.21.15.46:0/3898091628 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f540a89b0 con 0x7f7f581538f0 2023-10-30T14:24:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5d070640 1 -- 172.21.15.46:0/3898091628 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f540a8b10 con 0x7f7f581538f0 2023-10-30T14:24:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5d070640 1 -- 172.21.15.46:0/3898091628 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f540a8e80 con 0x7f7f581538f0 2023-10-30T14:24:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 -- 172.21.15.46:0/3898091628 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f581538f0 msgr2=0x7f7f58159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 --2- 172.21.15.46:0/3898091628 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f581538f0 0x7f7f58159fb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7f540097f0 tx=0x7f7f5401ecb0 comp rx=0 tx=0).stop 2023-10-30T14:24:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 -- 172.21.15.46:0/3898091628 shutdown_connections 2023-10-30T14:24:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 --2- 172.21.15.46:0/3898091628 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f581538f0 0x7f7f58159fb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 -- 172.21.15.46:0/3898091628 >> 172.21.15.46:0/3898091628 conn(0x7f7f5806a640 msgr2=0x7f7f58065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 -- 172.21.15.46:0/3898091628 shutdown_connections 2023-10-30T14:24:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 -- 172.21.15.46:0/3898091628 wait complete. 2023-10-30T14:24:28.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 Processor -- start 2023-10-30T14:24:28.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 -- start start 2023-10-30T14:24:28.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f580cefa0 0x7f7f5815c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:28.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f581538f0 0x7f7f58146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5815cd50 0x7f7f58146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f580b5000 con 0x7f7f5815cd50 2023-10-30T14:24:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f580b4a20 con 0x7f7f581538f0 2023-10-30T14:24:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.063+0000 7f7f5f074640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f58147eb0 con 0x7f7f580cefa0 2023-10-30T14:24:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5e873640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5815cd50 0x7f7f58146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5e873640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5815cd50 0x7f7f58146bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33562/0 (socket says 172.21.15.46:33562) 2023-10-30T14:24:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5e873640 1 -- 172.21.15.46:0/3121233781 learned_addr learned my addr 172.21.15.46:0/3121233781 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5d871640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f581538f0 0x7f7f58146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5e072640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f580cefa0 0x7f7f5815c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5e873640 1 -- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f580cefa0 msgr2=0x7f7f5815c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5e873640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f580cefa0 0x7f7f5815c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5e873640 1 -- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f581538f0 msgr2=0x7f7f58146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5e873640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f581538f0 0x7f7f58146690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5e873640 1 -- 172.21.15.46:0/3121233781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f54009420 con 0x7f7f5815cd50 2023-10-30T14:24:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5e873640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5815cd50 0x7f7f58146bd0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f7f5000c770 tx=0x7f7f5000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f4effd640 1 -- 172.21.15.46:0/3121233781 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f50003b80 con 0x7f7f5815cd50 2023-10-30T14:24:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f4effd640 1 -- 172.21.15.46:0/3121233781 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f50003ce0 con 0x7f7f5815cd50 2023-10-30T14:24:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f4effd640 1 -- 172.21.15.46:0/3121233781 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5000aa90 con 0x7f7f5815cd50 2023-10-30T14:24:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5f074640 1 -- 172.21.15.46:0/3121233781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f58148150 con 0x7f7f5815cd50 2023-10-30T14:24:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5f074640 1 -- 172.21.15.46:0/3121233781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f580b4c90 con 0x7f7f5815cd50 2023-10-30T14:24:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f5f074640 1 -- 172.21.15.46:0/3121233781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f580b4e20 con 0x7f7f5815cd50 2023-10-30T14:24:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.067+0000 7f7f4effd640 1 -- 172.21.15.46:0/3121233781 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7f5000abf0 con 0x7f7f5815cd50 2023-10-30T14:24:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.071+0000 7f7f4effd640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f3c0664d0 0x7f7f3c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.071+0000 7f7f4effd640 1 -- 172.21.15.46:0/3121233781 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8650+0+0 (secure 0 0 0) 0x7f7f500847f0 con 0x7f7f5815cd50 2023-10-30T14:24:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.071+0000 7f7f5e072640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f3c0664d0 0x7f7f3c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.071+0000 7f7f5e072640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f3c0664d0 0x7f7f3c068990 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f7f540b2000 tx=0x7f7f540aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.075+0000 7f7f4effd640 1 -- 172.21.15.46:0/3121233781 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7f50051060 con 0x7f7f5815cd50 2023-10-30T14:24:28.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.275+0000 7f7f5f074640 1 -- 172.21.15.46:0/3121233781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f5815db70 con 0x7f7f5815cd50 2023-10-30T14:24:28.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.275+0000 7f7f4effd640 1 -- 172.21.15.46:0/3121233781 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+6526 (secure 0 0 0) 0x7f7f50054680 con 0x7f7f5815cd50 2023-10-30T14:24:28.280 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:28.280 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":10020,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:22.592577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:28.280 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 258 2023-10-30T14:24:28.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.279+0000 7f7f5f074640 1 -- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f3c0664d0 msgr2=0x7f7f3c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:28.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.279+0000 7f7f5f074640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f3c0664d0 0x7f7f3c068990 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f7f540b2000 tx=0x7f7f540aa070 comp rx=0 tx=0).stop 2023-10-30T14:24:28.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.279+0000 7f7f5f074640 1 -- 172.21.15.46:0/3121233781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5815cd50 msgr2=0x7f7f58146bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:28.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.279+0000 7f7f5f074640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5815cd50 0x7f7f58146bd0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f7f5000c770 tx=0x7f7f5000cc30 comp rx=0 tx=0).stop 2023-10-30T14:24:28.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.279+0000 7f7f5f074640 1 -- 172.21.15.46:0/3121233781 shutdown_connections 2023-10-30T14:24:28.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.279+0000 7f7f5f074640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7f3c0664d0 0x7f7f3c068990 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:28.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.279+0000 7f7f5f074640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f5815cd50 0x7f7f58146bd0 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:28.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.279+0000 7f7f5f074640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f581538f0 0x7f7f58146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:28.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.279+0000 7f7f5f074640 1 --2- 172.21.15.46:0/3121233781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f580cefa0 0x7f7f5815c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:28.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.279+0000 7f7f5f074640 1 -- 172.21.15.46:0/3121233781 >> 172.21.15.46:0/3121233781 conn(0x7f7f5806a640 msgr2=0x7f7f580caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:28.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.279+0000 7f7f5f074640 1 -- 172.21.15.46:0/3121233781 shutdown_connections 2023-10-30T14:24:28.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:28.279+0000 7f7f5f074640 1 -- 172.21.15.46:0/3121233781 wait complete. 2023-10-30T14:24:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 36 -eq 48 ] 2023-10-30T14:24:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.36 2023-10-30T14:24:28.297 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.36 2023-10-30T14:24:28.301 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T14:24:28.301 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 19 secs before thrashing 2023-10-30T14:24:28.310 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 37 -eq 48 ] 2023-10-30T14:24:28.310 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.37 2023-10-30T14:24:28.310 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.37 2023-10-30T14:24:28.322 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 38 -eq 48 ] 2023-10-30T14:24:28.323 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.38 2023-10-30T14:24:28.323 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.38 2023-10-30T14:24:28.336 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 39 -eq 48 ] 2023-10-30T14:24:28.336 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.39 2023-10-30T14:24:28.336 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.39 2023-10-30T14:24:28.349 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 40 -eq 48 ] 2023-10-30T14:24:28.349 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.40 2023-10-30T14:24:28.350 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.40 2023-10-30T14:24:28.362 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 41 -eq 48 ] 2023-10-30T14:24:28.362 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.41 2023-10-30T14:24:28.363 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.41 2023-10-30T14:24:28.376 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 42 -eq 48 ] 2023-10-30T14:24:28.376 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.42 2023-10-30T14:24:28.377 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.42 2023-10-30T14:24:28.389 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 43 -eq 48 ] 2023-10-30T14:24:28.389 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.43 2023-10-30T14:24:28.389 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.43 2023-10-30T14:24:28.402 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 44 -eq 48 ] 2023-10-30T14:24:28.402 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.44 2023-10-30T14:24:28.402 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.44 2023-10-30T14:24:28.414 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 45 -eq 48 ] 2023-10-30T14:24:28.414 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.45 2023-10-30T14:24:28.415 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.45 2023-10-30T14:24:28.427 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 46 -eq 48 ] 2023-10-30T14:24:28.428 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.46 2023-10-30T14:24:28.428 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.46 2023-10-30T14:24:28.442 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 47 -eq 48 ] 2023-10-30T14:24:28.442 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.47 2023-10-30T14:24:28.442 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.47 2023-10-30T14:24:28.454 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 48 -eq 48 ] 2023-10-30T14:24:28.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ continue 2023-10-30T14:24:28.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 49 -eq 48 ] 2023-10-30T14:24:28.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.49 2023-10-30T14:24:28.455 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.49 2023-10-30T14:24:28.467 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 50 -eq 48 ] 2023-10-30T14:24:28.467 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.50 2023-10-30T14:24:28.467 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.50 2023-10-30T14:24:28.480 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 51 -eq 48 ] 2023-10-30T14:24:28.481 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.51 2023-10-30T14:24:28.481 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.51 2023-10-30T14:24:28.494 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 52 -eq 48 ] 2023-10-30T14:24:28.495 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.52 2023-10-30T14:24:28.495 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.52 2023-10-30T14:24:28.507 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 53 -eq 48 ] 2023-10-30T14:24:28.507 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.53 2023-10-30T14:24:28.507 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.53 2023-10-30T14:24:28.521 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 54 -eq 48 ] 2023-10-30T14:24:28.521 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.54 2023-10-30T14:24:28.521 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.54 2023-10-30T14:24:28.535 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 55 -eq 48 ] 2023-10-30T14:24:28.535 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.55 2023-10-30T14:24:28.535 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.55 2023-10-30T14:24:28.776 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 56 -eq 48 ] 2023-10-30T14:24:28.777 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.56 2023-10-30T14:24:28.777 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.56 2023-10-30T14:24:28.918 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 57 -eq 48 ] 2023-10-30T14:24:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.57 2023-10-30T14:24:28.919 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.57 2023-10-30T14:24:28.935 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 58 -eq 48 ] 2023-10-30T14:24:28.935 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.58 2023-10-30T14:24:28.935 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.58 2023-10-30T14:24:28.948 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 59 -eq 48 ] 2023-10-30T14:24:28.948 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.59 2023-10-30T14:24:28.948 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.59 2023-10-30T14:24:28.961 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 60 -eq 48 ] 2023-10-30T14:24:28.961 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.60 2023-10-30T14:24:28.961 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.60 2023-10-30T14:24:28.974 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 61 -eq 48 ] 2023-10-30T14:24:28.974 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.61 2023-10-30T14:24:28.975 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.61 2023-10-30T14:24:28.987 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 62 -eq 48 ] 2023-10-30T14:24:28.987 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.62 2023-10-30T14:24:28.987 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.62 2023-10-30T14:24:28.999 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 63 -eq 48 ] 2023-10-30T14:24:29.000 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.63 2023-10-30T14:24:29.000 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.63 2023-10-30T14:24:29.013 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 64 -eq 48 ] 2023-10-30T14:24:29.013 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.64 2023-10-30T14:24:29.013 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.64 2023-10-30T14:24:29.026 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 65 -eq 48 ] 2023-10-30T14:24:29.026 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.65 2023-10-30T14:24:29.026 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.65 2023-10-30T14:24:29.038 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 66 -eq 48 ] 2023-10-30T14:24:29.039 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.66 2023-10-30T14:24:29.039 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.66 2023-10-30T14:24:29.052 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 67 -eq 48 ] 2023-10-30T14:24:29.052 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.67 2023-10-30T14:24:29.052 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.67 2023-10-30T14:24:29.066 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 68 -eq 48 ] 2023-10-30T14:24:29.067 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.68 2023-10-30T14:24:29.067 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.68 2023-10-30T14:24:29.079 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 69 -eq 48 ] 2023-10-30T14:24:29.080 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.69 2023-10-30T14:24:29.080 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.69 2023-10-30T14:24:29.092 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 70 -eq 48 ] 2023-10-30T14:24:29.092 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.70 2023-10-30T14:24:29.093 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.70 2023-10-30T14:24:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 71 -eq 48 ] 2023-10-30T14:24:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.71 2023-10-30T14:24:29.106 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.71 2023-10-30T14:24:29.119 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 72 -eq 48 ] 2023-10-30T14:24:29.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.72 2023-10-30T14:24:29.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.72 2023-10-30T14:24:29.132 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 73 -eq 48 ] 2023-10-30T14:24:29.133 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.73 2023-10-30T14:24:29.133 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.73 2023-10-30T14:24:29.147 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 74 -eq 48 ] 2023-10-30T14:24:29.147 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.74 2023-10-30T14:24:29.147 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.74 2023-10-30T14:24:29.161 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 75 -eq 48 ] 2023-10-30T14:24:29.162 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.75 2023-10-30T14:24:29.162 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.75 2023-10-30T14:24:29.181 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 76 -eq 48 ] 2023-10-30T14:24:29.181 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.76 2023-10-30T14:24:29.182 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.76 2023-10-30T14:24:29.195 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 77 -eq 48 ] 2023-10-30T14:24:29.195 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.77 2023-10-30T14:24:29.195 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.77 2023-10-30T14:24:29.208 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 78 -eq 48 ] 2023-10-30T14:24:29.209 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.78 2023-10-30T14:24:29.209 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.78 2023-10-30T14:24:29.224 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 79 -eq 48 ] 2023-10-30T14:24:29.224 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.79 2023-10-30T14:24:29.224 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.79 2023-10-30T14:24:29.237 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 80 -eq 48 ] 2023-10-30T14:24:29.237 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.80 2023-10-30T14:24:29.237 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.80 2023-10-30T14:24:29.249 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 81 -eq 48 ] 2023-10-30T14:24:29.250 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.81 2023-10-30T14:24:29.250 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.81 2023-10-30T14:24:29.261 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 82 -eq 48 ] 2023-10-30T14:24:29.262 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.82 2023-10-30T14:24:29.262 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.82 2023-10-30T14:24:29.275 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 83 -eq 48 ] 2023-10-30T14:24:29.275 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.83 2023-10-30T14:24:29.275 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.83 2023-10-30T14:24:29.288 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 84 -eq 48 ] 2023-10-30T14:24:29.288 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.84 2023-10-30T14:24:29.288 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.84 2023-10-30T14:24:29.300 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 85 -eq 48 ] 2023-10-30T14:24:29.300 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.85 2023-10-30T14:24:29.301 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.85 2023-10-30T14:24:29.312 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 86 -eq 48 ] 2023-10-30T14:24:29.313 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.86 2023-10-30T14:24:29.313 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.86 2023-10-30T14:24:29.325 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 87 -eq 48 ] 2023-10-30T14:24:29.325 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.87 2023-10-30T14:24:29.325 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.87 2023-10-30T14:24:29.340 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 88 -eq 48 ] 2023-10-30T14:24:29.340 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.88 2023-10-30T14:24:29.340 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.88 2023-10-30T14:24:29.355 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 89 -eq 48 ] 2023-10-30T14:24:29.355 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.89 2023-10-30T14:24:29.355 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.89 2023-10-30T14:24:29.368 INFO:tasks.workunit.client.0.smithi046.stderr:+ [ 90 -eq 48 ] 2023-10-30T14:24:29.369 INFO:tasks.workunit.client.0.smithi046.stderr:+ ver=v0.90 2023-10-30T14:24:29.369 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/v0.90 2023-10-30T14:24:29.382 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo OK 2023-10-30T14:24:29.383 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T14:24:29.383 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:24:29.383 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:24:36.264 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:24:36.269 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:24:36.315 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-10-30T14:24:36.311+0000 7fdb0c1d8640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 69176) UID: 0 2023-10-30T14:24:36.316 INFO:tasks.ceph.mds.c.smithi046.stderr:2023-10-30T14:24:36.311+0000 7fc73d350640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 69176) UID: 0 2023-10-30T14:24:36.316 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T14:24:36.315+0000 7f12c16c3640 -1 received signal: Hangup from (PID: 69176) UID: 0 2023-10-30T14:24:47.303 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:47.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b424dc640 1 Processor -- start 2023-10-30T14:24:47.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b424dc640 1 -- start start 2023-10-30T14:24:47.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b424dc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 0x7f7b3c15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:47.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b424dc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b3c15b060 con 0x7f7b3c0c6290 2023-10-30T14:24:47.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b424dc640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b3c0c4ab0 con 0x7f7b3c071b90 2023-10-30T14:24:47.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b424dc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c157530 con 0x7f7b3c0717b0 2023-10-30T14:24:47.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b41cdb640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7b3c0c6290 0x7f7b3c161eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44012/0 (socket says 172.21.15.46:44012) 2023-10-30T14:24:47.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b41cdb640 1 -- 172.21.15.46:0/2095222856 learned_addr learned my addr 172.21.15.46:0/2095222856 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:47.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b414da640 1 --2- 172.21.15.46:0/2095222856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 0x7f7b3c15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:47.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b33fff640 1 -- 172.21.15.46:0/2095222856 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155177061 0 0) 0x7f7b3c15b060 con 0x7f7b3c0c6290 2023-10-30T14:24:47.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b33fff640 1 -- 172.21.15.46:0/2095222856 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b28003530 con 0x7f7b3c0c6290 2023-10-30T14:24:47.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b414da640 1 -- 172.21.15.46:0/2095222856 >> v1:172.21.15.154:6789/0 conn(0x7f7b3c071b90 legacy=0x7f7b3c1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:47.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b414da640 1 -- 172.21.15.46:0/2095222856 >> v1:172.21.15.46:6789/0 conn(0x7f7b3c0c6290 legacy=0x7f7b3c161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:47.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.459+0000 7f7b414da640 1 -- 172.21.15.46:0/2095222856 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3c0c4d90 con 0x7f7b3c0717b0 2023-10-30T14:24:47.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.463+0000 7f7b414da640 1 --2- 172.21.15.46:0/2095222856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 0x7f7b3c15ab20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7b380097d0 tx=0x7f7b3801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa7ecdb42c24b23 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:47.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.467+0000 7f7b33fff640 1 -- 172.21.15.46:0/2095222856 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b380a8990 con 0x7f7b3c0717b0 2023-10-30T14:24:47.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.467+0000 7f7b33fff640 1 -- 172.21.15.46:0/2095222856 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b380a8af0 con 0x7f7b3c0717b0 2023-10-30T14:24:47.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.467+0000 7f7b33fff640 1 -- 172.21.15.46:0/2095222856 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b380a8e80 con 0x7f7b3c0717b0 2023-10-30T14:24:47.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.467+0000 7f7b424dc640 1 -- 172.21.15.46:0/2095222856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 msgr2=0x7f7b3c15ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:47.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.467+0000 7f7b424dc640 1 --2- 172.21.15.46:0/2095222856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 0x7f7b3c15ab20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7b380097d0 tx=0x7f7b3801ec90 comp rx=0 tx=0).stop 2023-10-30T14:24:47.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.467+0000 7f7b424dc640 1 -- 172.21.15.46:0/2095222856 shutdown_connections 2023-10-30T14:24:47.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.467+0000 7f7b424dc640 1 --2- 172.21.15.46:0/2095222856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 0x7f7b3c15ab20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:47.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.467+0000 7f7b424dc640 1 -- 172.21.15.46:0/2095222856 >> 172.21.15.46:0/2095222856 conn(0x7f7b3c0c75d0 msgr2=0x7f7b3c0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:47.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.467+0000 7f7b424dc640 1 -- 172.21.15.46:0/2095222856 shutdown_connections 2023-10-30T14:24:47.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.467+0000 7f7b424dc640 1 -- 172.21.15.46:0/2095222856 wait complete. 2023-10-30T14:24:47.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.467+0000 7f7b424dc640 1 Processor -- start 2023-10-30T14:24:47.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.467+0000 7f7b424dc640 1 -- start start 2023-10-30T14:24:47.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b424dc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 0x7f7b3c070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:47.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b424dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b3c071b90 0x7f7b3c1f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:47.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b424dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3c0c6290 0x7f7b3c0c25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:47.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b424dc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c070ba0 con 0x7f7b3c0c6290 2023-10-30T14:24:47.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b424dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1f21a0 con 0x7f7b3c071b90 2023-10-30T14:24:47.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b424dc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c06ff20 con 0x7f7b3c0717b0 2023-10-30T14:24:47.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b41cdb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3c0c6290 0x7f7b3c0c25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:47.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b414da640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 0x7f7b3c070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:47.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b414da640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 0x7f7b3c070fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:51242/0 (socket says 172.21.15.46:51242) 2023-10-30T14:24:47.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b41cdb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3c0c6290 0x7f7b3c0c25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42238/0 (socket says 172.21.15.46:42238) 2023-10-30T14:24:47.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b414da640 1 -- 172.21.15.46:0/1909199834 learned_addr learned my addr 172.21.15.46:0/1909199834 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:47.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b40cd9640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b3c071b90 0x7f7b3c1f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:47.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b41cdb640 1 -- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 msgr2=0x7f7b3c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:47.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b41cdb640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 0x7f7b3c070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:47.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b41cdb640 1 -- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b3c071b90 msgr2=0x7f7b3c1f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:47.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b41cdb640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b3c071b90 0x7f7b3c1f1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:47.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b41cdb640 1 -- 172.21.15.46:0/1909199834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b380093b0 con 0x7f7b3c0c6290 2023-10-30T14:24:47.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b414da640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 0x7f7b3c070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:24:47.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b41cdb640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3c0c6290 0x7f7b3c0c25e0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400cab0 tx=0x7f7b3400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:47.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b31ffb640 1 -- 172.21.15.46:0/1909199834 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3400ac30 con 0x7f7b3c0c6290 2023-10-30T14:24:47.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b31ffb640 1 -- 172.21.15.46:0/1909199834 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3400ad90 con 0x7f7b3c0c6290 2023-10-30T14:24:47.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b31ffb640 1 -- 172.21.15.46:0/1909199834 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34012600 con 0x7f7b3c0c6290 2023-10-30T14:24:47.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b424dc640 1 -- 172.21.15.46:0/1909199834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3c070800 con 0x7f7b3c0c6290 2023-10-30T14:24:47.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b424dc640 1 -- 172.21.15.46:0/1909199834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3c06f960 con 0x7f7b3c0c6290 2023-10-30T14:24:47.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.471+0000 7f7b424dc640 1 -- 172.21.15.46:0/1909199834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3c070ce0 con 0x7f7b3c0c6290 2023-10-30T14:24:47.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.475+0000 7f7b31ffb640 1 -- 172.21.15.46:0/1909199834 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7b34007070 con 0x7f7b3c0c6290 2023-10-30T14:24:47.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.475+0000 7f7b31ffb640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b200664a0 0x7f7b20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:47.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.475+0000 7f7b31ffb640 1 -- 172.21.15.46:0/1909199834 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8678+0+0 (secure 0 0 0) 0x7f7b340848d0 con 0x7f7b3c0c6290 2023-10-30T14:24:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.475+0000 7f7b414da640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b200664a0 0x7f7b20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.475+0000 7f7b414da640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b200664a0 0x7f7b20068960 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f7b3801ecd0 tx=0x7f7b38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.483+0000 7f7b31ffb640 1 -- 172.21.15.46:0/1909199834 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7b3c070ce0 con 0x7f7b3c0c6290 2023-10-30T14:24:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 -- 172.21.15.46:0/1909199834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b3c070ce0 con 0x7f7b3c0c6290 2023-10-30T14:24:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b31ffb640 1 -- 172.21.15.46:0/1909199834 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+6527 (secure 0 0 0) 0x7f7b3c070ce0 con 0x7f7b3c0c6290 2023-10-30T14:24:47.677 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:47.677 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":10020,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":258}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:32.905080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:47.678 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 259 2023-10-30T14:24:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 -- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b200664a0 msgr2=0x7f7b20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b200664a0 0x7f7b20068960 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f7b3801ecd0 tx=0x7f7b38008040 comp rx=0 tx=0).stop 2023-10-30T14:24:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 -- 172.21.15.46:0/1909199834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3c0c6290 msgr2=0x7f7b3c0c25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3c0c6290 0x7f7b3c0c25e0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400cab0 tx=0x7f7b3400cf70 comp rx=0 tx=0).stop 2023-10-30T14:24:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 -- 172.21.15.46:0/1909199834 shutdown_connections 2023-10-30T14:24:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f7b200664a0 0x7f7b20068960 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b3c0c6290 0x7f7b3c0c25e0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b3c071b90 0x7f7b3c1f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 --2- 172.21.15.46:0/1909199834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0717b0 0x7f7b3c070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 -- 172.21.15.46:0/1909199834 >> 172.21.15.46:0/1909199834 conn(0x7f7b3c0c75d0 msgr2=0x7f7b3c06d400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 -- 172.21.15.46:0/1909199834 shutdown_connections 2023-10-30T14:24:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.675+0000 7f7b424dc640 1 -- 172.21.15.46:0/1909199834 wait complete. 2023-10-30T14:24:47.699 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T14:24:47.699 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:24:47.700 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:24:47.721 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:24:47.721 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409f29a640 1 Processor -- start 2023-10-30T14:24:47.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409f29a640 1 -- start start 2023-10-30T14:24:47.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409f29a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40981538a0 0x7f409815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:47.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409f29a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f40980ca0e0 0x7f409815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:47.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409f29a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40980cb800 con 0x7f40980cf160 2023-10-30T14:24:47.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409f29a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f409815dc30 con 0x7f40980ca0e0 2023-10-30T14:24:47.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409f29a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40980cb330 con 0x7f40981538a0 2023-10-30T14:24:47.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409d837640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f40980cf160 0x7f40980cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:58970/0 (socket says 172.21.15.46:58970) 2023-10-30T14:24:47.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409d837640 1 -- 172.21.15.46:0/892336890 learned_addr learned my addr 172.21.15.46:0/892336890 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:47.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409c835640 1 --2- 172.21.15.46:0/892336890 >> v2:172.21.15.154:3300/0 conn(0x7f40980ca0e0 0x7f409815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:47.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409d036640 1 --2- 172.21.15.46:0/892336890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40981538a0 0x7f409815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:47.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f408ffff640 1 -- 172.21.15.46:0/892336890 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342178127 0 0) 0x7f40980cb800 con 0x7f40980cf160 2023-10-30T14:24:47.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f408ffff640 1 -- 172.21.15.46:0/892336890 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4084003530 con 0x7f40980cf160 2023-10-30T14:24:47.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409d036640 1 -- 172.21.15.46:0/892336890 >> v2:172.21.15.154:3300/0 conn(0x7f40980ca0e0 msgr2=0x7f409815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:47.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409d036640 1 --2- 172.21.15.46:0/892336890 >> v2:172.21.15.154:3300/0 conn(0x7f40980ca0e0 0x7f409815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:47.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409d036640 1 -- 172.21.15.46:0/892336890 >> v1:172.21.15.154:6789/0 conn(0x7f40980cf160 legacy=0x7f40980cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:47.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409d036640 1 -- 172.21.15.46:0/892336890 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40980cad70 con 0x7f40981538a0 2023-10-30T14:24:47.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.911+0000 7f409c835640 1 --2- 172.21.15.46:0/892336890 >> v2:172.21.15.154:3300/0 conn(0x7f40980ca0e0 0x7f409815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:24:47.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.915+0000 7f409d036640 1 --2- 172.21.15.46:0/892336890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40981538a0 0x7f409815d6f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f40940097c0 tx=0x7f409401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1f452edf7128686 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:47.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.915+0000 7f408ffff640 1 -- 172.21.15.46:0/892336890 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40940a8970 con 0x7f40981538a0 2023-10-30T14:24:47.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.915+0000 7f408ffff640 1 -- 172.21.15.46:0/892336890 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40940a8ad0 con 0x7f40981538a0 2023-10-30T14:24:47.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.915+0000 7f408ffff640 1 -- 172.21.15.46:0/892336890 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40940a8e60 con 0x7f40981538a0 2023-10-30T14:24:47.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.915+0000 7f409f29a640 1 -- 172.21.15.46:0/892336890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40981538a0 msgr2=0x7f409815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:47.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.915+0000 7f409f29a640 1 --2- 172.21.15.46:0/892336890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40981538a0 0x7f409815d6f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f40940097c0 tx=0x7f409401ec70 comp rx=0 tx=0).stop 2023-10-30T14:24:47.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.915+0000 7f409f29a640 1 -- 172.21.15.46:0/892336890 shutdown_connections 2023-10-30T14:24:47.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.915+0000 7f409f29a640 1 --2- 172.21.15.46:0/892336890 >> v2:172.21.15.154:3300/0 conn(0x7f40980ca0e0 0x7f409815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:47.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.915+0000 7f409f29a640 1 --2- 172.21.15.46:0/892336890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40981538a0 0x7f409815d6f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:47.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.915+0000 7f409f29a640 1 -- 172.21.15.46:0/892336890 >> 172.21.15.46:0/892336890 conn(0x7f409806bfe0 msgr2=0x7f409806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:47.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.915+0000 7f409f29a640 1 -- 172.21.15.46:0/892336890 shutdown_connections 2023-10-30T14:24:47.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.915+0000 7f409f29a640 1 -- 172.21.15.46:0/892336890 wait complete. 2023-10-30T14:24:47.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409f29a640 1 Processor -- start 2023-10-30T14:24:47.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409f29a640 1 -- start start 2023-10-30T14:24:47.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409f29a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40980ca0e0 0x7f409815d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:47.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409f29a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40980cf160 0x7f40980c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:47.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409f29a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40981538a0 0x7f40980ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:47.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409f29a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409815b2c0 con 0x7f40981538a0 2023-10-30T14:24:47.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409f29a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40980c8f20 con 0x7f40980ca0e0 2023-10-30T14:24:47.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409f29a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f409815bb80 con 0x7f40980cf160 2023-10-30T14:24:47.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409d036640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40980ca0e0 0x7f409815d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:47.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409c835640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40980cf160 0x7f40980c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:47.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409d036640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40980ca0e0 0x7f409815d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:38654/0 (socket says 172.21.15.46:38654) 2023-10-30T14:24:47.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409d036640 1 -- 172.21.15.46:0/3166499062 learned_addr learned my addr 172.21.15.46:0/3166499062 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:47.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409d837640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40981538a0 0x7f40980ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:47.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409c835640 1 -- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40980ca0e0 msgr2=0x7f409815d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:47.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409c835640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40980ca0e0 0x7f409815d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:47.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409c835640 1 -- 172.21.15.46:0/3166499062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40981538a0 msgr2=0x7f40980ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:47.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409c835640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40981538a0 0x7f40980ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:47.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409c835640 1 -- 172.21.15.46:0/3166499062 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4088009420 con 0x7f40980cf160 2023-10-30T14:24:47.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409d036640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40980ca0e0 0x7f409815d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:24:47.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409c835640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40980cf160 0x7f40980c91e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f408800e9b0 tx=0x7f408800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:47.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f408dffb640 1 -- 172.21.15.46:0/3166499062 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408800c9a0 con 0x7f40980cf160 2023-10-30T14:24:47.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f408dffb640 1 -- 172.21.15.46:0/3166499062 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408800cb00 con 0x7f40980cf160 2023-10-30T14:24:47.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409f29a640 1 -- 172.21.15.46:0/3166499062 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4094009420 con 0x7f40980cf160 2023-10-30T14:24:47.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f408dffb640 1 -- 172.21.15.46:0/3166499062 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4088013470 con 0x7f40980cf160 2023-10-30T14:24:47.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.919+0000 7f409f29a640 1 -- 172.21.15.46:0/3166499062 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409815b7d0 con 0x7f40980cf160 2023-10-30T14:24:47.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.923+0000 7f409f29a640 1 -- 172.21.15.46:0/3166499062 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40980b2a50 con 0x7f40980cf160 2023-10-30T14:24:47.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.923+0000 7f408dffb640 1 -- 172.21.15.46:0/3166499062 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f40880135d0 con 0x7f40980cf160 2023-10-30T14:24:47.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.923+0000 7f408dffb640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f407c0664d0 0x7f407c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.923+0000 7f408dffb640 1 -- 172.21.15.46:0/3166499062 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8678+0+0 (secure 0 0 0) 0x7f4088084c90 con 0x7f40980cf160 2023-10-30T14:24:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.923+0000 7f409d036640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f407c0664d0 0x7f407c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.923+0000 7f409d036640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f407c0664d0 0x7f407c068990 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f409401ecb0 tx=0x7f40940aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:47.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:47.927+0000 7f408dffb640 1 -- 172.21.15.46:0/3166499062 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4088051570 con 0x7f40980cf160 2023-10-30T14:24:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.131+0000 7f409f29a640 1 -- 172.21.15.46:0/3166499062 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f409804e170 con 0x7f40980cf160 2023-10-30T14:24:48.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.131+0000 7f408dffb640 1 -- 172.21.15.46:0/3166499062 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+6527 (secure 0 0 0) 0x7f4088021090 con 0x7f40980cf160 2023-10-30T14:24:48.134 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:48.135 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":10020,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":258}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:32.905080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 259 2023-10-30T14:24:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.135+0000 7f409f29a640 1 -- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f407c0664d0 msgr2=0x7f407c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.135+0000 7f409f29a640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f407c0664d0 0x7f407c068990 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f409401ecb0 tx=0x7f40940aa070 comp rx=0 tx=0).stop 2023-10-30T14:24:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.135+0000 7f409f29a640 1 -- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40980cf160 msgr2=0x7f40980c91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.135+0000 7f409f29a640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40980cf160 0x7f40980c91e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f408800e9b0 tx=0x7f408800ee70 comp rx=0 tx=0).stop 2023-10-30T14:24:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.135+0000 7f409f29a640 1 -- 172.21.15.46:0/3166499062 shutdown_connections 2023-10-30T14:24:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.135+0000 7f409f29a640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f407c0664d0 0x7f407c068990 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.135+0000 7f409f29a640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40981538a0 0x7f40980ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.135+0000 7f409f29a640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40980cf160 0x7f40980c91e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.135+0000 7f409f29a640 1 --2- 172.21.15.46:0/3166499062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40980ca0e0 0x7f409815d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.135+0000 7f409f29a640 1 -- 172.21.15.46:0/3166499062 >> 172.21.15.46:0/3166499062 conn(0x7f409806bfe0 msgr2=0x7f4098152680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.135+0000 7f409f29a640 1 -- 172.21.15.46:0/3166499062 shutdown_connections 2023-10-30T14:24:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:48.135+0000 7f409f29a640 1 -- 172.21.15.46:0/3166499062 wait complete. 2023-10-30T14:24:48.157 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:24:50.158 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:50.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da2a06640 1 Processor -- start 2023-10-30T14:24:50.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da2a06640 1 -- start start 2023-10-30T14:24:50.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da2a06640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d9c1538f0 0x7f1d9c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:50.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da2a06640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1d9c0cefa0 0x7f1d9c0cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:50.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da2a06640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c0ca6a0 con 0x7f1d9c15cd50 2023-10-30T14:24:50.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da2a06640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9c15d300 con 0x7f1d9c0cefa0 2023-10-30T14:24:50.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da2a06640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c15d5e0 con 0x7f1d9c1538f0 2023-10-30T14:24:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da1a04640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d9c1538f0 0x7f1d9c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da2205640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1d9c0cefa0 0x7f1d9c0cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da1203640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f1d9c15cd50 0x7f1d9c0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:58982/0 (socket says 172.21.15.46:58982) 2023-10-30T14:24:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da1203640 1 -- 172.21.15.46:0/221738048 learned_addr learned my addr 172.21.15.46:0/221738048 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da1a04640 1 -- 172.21.15.46:0/221738048 >> v2:172.21.15.154:3300/0 conn(0x7f1d9c0cefa0 msgr2=0x7f1d9c0cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:50.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da1a04640 1 --2- 172.21.15.46:0/221738048 >> v2:172.21.15.154:3300/0 conn(0x7f1d9c0cefa0 0x7f1d9c0cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:50.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da1a04640 1 -- 172.21.15.46:0/221738048 >> v1:172.21.15.154:6789/0 conn(0x7f1d9c15cd50 legacy=0x7f1d9c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:50.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.303+0000 7f1da1a04640 1 -- 172.21.15.46:0/221738048 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c15d8c0 con 0x7f1d9c1538f0 2023-10-30T14:24:50.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.307+0000 7f1da1a04640 1 --2- 172.21.15.46:0/221738048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d9c1538f0 0x7f1d9c159fb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0097c0 tx=0x7f1d8c01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ba52036fd38d980 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:50.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.307+0000 7f1da0a02640 1 -- 172.21.15.46:0/221738048 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c0a8970 con 0x7f1d9c1538f0 2023-10-30T14:24:50.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.307+0000 7f1da0a02640 1 -- 172.21.15.46:0/221738048 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8c0a8ad0 con 0x7f1d9c1538f0 2023-10-30T14:24:50.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.307+0000 7f1da0a02640 1 -- 172.21.15.46:0/221738048 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c0a8e60 con 0x7f1d9c1538f0 2023-10-30T14:24:50.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.307+0000 7f1da2a06640 1 -- 172.21.15.46:0/221738048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d9c1538f0 msgr2=0x7f1d9c159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.307+0000 7f1da2a06640 1 --2- 172.21.15.46:0/221738048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d9c1538f0 0x7f1d9c159fb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0097c0 tx=0x7f1d8c01ec70 comp rx=0 tx=0).stop 2023-10-30T14:24:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.307+0000 7f1da2a06640 1 -- 172.21.15.46:0/221738048 shutdown_connections 2023-10-30T14:24:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.307+0000 7f1da2a06640 1 --2- 172.21.15.46:0/221738048 >> v2:172.21.15.154:3300/0 conn(0x7f1d9c0cefa0 0x7f1d9c0cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.307+0000 7f1da2a06640 1 --2- 172.21.15.46:0/221738048 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d9c1538f0 0x7f1d9c159fb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.307+0000 7f1da2a06640 1 -- 172.21.15.46:0/221738048 >> 172.21.15.46:0/221738048 conn(0x7f1d9c06a640 msgr2=0x7f1d9c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.307+0000 7f1da2a06640 1 -- 172.21.15.46:0/221738048 shutdown_connections 2023-10-30T14:24:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.307+0000 7f1da2a06640 1 -- 172.21.15.46:0/221738048 wait complete. 2023-10-30T14:24:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da2a06640 1 Processor -- start 2023-10-30T14:24:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da2a06640 1 -- start start 2023-10-30T14:24:50.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da2a06640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d9c0cefa0 0x7f1d9c0c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da2a06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d9c1538f0 0x7f1d9c1f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da2a06640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d9c15cd50 0x7f1d9c1f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da2a06640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c0c55d0 con 0x7f1d9c1538f0 2023-10-30T14:24:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da2a06640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c146a80 con 0x7f1d9c0cefa0 2023-10-30T14:24:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da2a06640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c0b3b40 con 0x7f1d9c15cd50 2023-10-30T14:24:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da1203640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d9c1538f0 0x7f1d9c1f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:50.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da1203640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d9c1538f0 0x7f1d9c1f2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42254/0 (socket says 172.21.15.46:42254) 2023-10-30T14:24:50.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da1203640 1 -- 172.21.15.46:0/1435175808 learned_addr learned my addr 172.21.15.46:0/1435175808 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:50.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da1a04640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d9c0cefa0 0x7f1d9c0c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:50.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da2205640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d9c15cd50 0x7f1d9c1f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:50.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da1203640 1 -- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d9c15cd50 msgr2=0x7f1d9c1f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:50.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da1203640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d9c15cd50 0x7f1d9c1f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da1203640 1 -- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d9c0cefa0 msgr2=0x7f1d9c0c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da1203640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d9c0cefa0 0x7f1d9c0c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da1203640 1 -- 172.21.15.46:0/1435175808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c009420 con 0x7f1d9c1538f0 2023-10-30T14:24:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da1203640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d9c1538f0 0x7f1d9c1f2d80 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800c400 tx=0x7f1d9800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1d927fc640 1 -- 172.21.15.46:0/1435175808 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d98005cf0 con 0x7f1d9c1538f0 2023-10-30T14:24:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1d927fc640 1 -- 172.21.15.46:0/1435175808 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d98005e50 con 0x7f1d9c1538f0 2023-10-30T14:24:50.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1d927fc640 1 -- 172.21.15.46:0/1435175808 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d980072d0 con 0x7f1d9c1538f0 2023-10-30T14:24:50.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da2a06640 1 -- 172.21.15.46:0/1435175808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9c0c6490 con 0x7f1d9c1538f0 2023-10-30T14:24:50.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.311+0000 7f1da2a06640 1 -- 172.21.15.46:0/1435175808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9c1f2860 con 0x7f1d9c1538f0 2023-10-30T14:24:50.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.315+0000 7f1da2a06640 1 -- 172.21.15.46:0/1435175808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9c1f2a80 con 0x7f1d9c1538f0 2023-10-30T14:24:50.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.315+0000 7f1d927fc640 1 -- 172.21.15.46:0/1435175808 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1d98007430 con 0x7f1d9c1538f0 2023-10-30T14:24:50.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.315+0000 7f1d927fc640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1d800664a0 0x7f1d80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:50.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.315+0000 7f1d927fc640 1 -- 172.21.15.46:0/1435175808 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8662+0+0 (secure 0 0 0) 0x7f1d98083f50 con 0x7f1d9c1538f0 2023-10-30T14:24:50.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.315+0000 7f1da1a04640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1d800664a0 0x7f1d80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.315+0000 7f1da1a04640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1d800664a0 0x7f1d80068960 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0b2000 tx=0x7f1d8c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.319+0000 7f1d927fc640 1 -- 172.21.15.46:0/1435175808 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1d98050840 con 0x7f1d9c1538f0 2023-10-30T14:24:50.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.511+0000 7f1da2a06640 1 -- 172.21.15.46:0/1435175808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d9c04e170 con 0x7f1d9c1538f0 2023-10-30T14:24:50.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.511+0000 7f1d927fc640 1 -- 172.21.15.46:0/1435175808 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+6527 (secure 0 0 0) 0x7f1d98017020 con 0x7f1d9c1538f0 2023-10-30T14:24:50.514 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:50.514 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":10020,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":258}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:32.905080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 259 2023-10-30T14:24:50.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.515+0000 7f1da2a06640 1 -- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1d800664a0 msgr2=0x7f1d80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:50.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.515+0000 7f1da2a06640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1d800664a0 0x7f1d80068960 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0b2000 tx=0x7f1d8c008040 comp rx=0 tx=0).stop 2023-10-30T14:24:50.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.515+0000 7f1da2a06640 1 -- 172.21.15.46:0/1435175808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d9c1538f0 msgr2=0x7f1d9c1f2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:50.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.515+0000 7f1da2a06640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d9c1538f0 0x7f1d9c1f2d80 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800c400 tx=0x7f1d9800c8c0 comp rx=0 tx=0).stop 2023-10-30T14:24:50.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.515+0000 7f1da2a06640 1 -- 172.21.15.46:0/1435175808 shutdown_connections 2023-10-30T14:24:50.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.515+0000 7f1da2a06640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f1d800664a0 0x7f1d80068960 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:50.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.515+0000 7f1da2a06640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d9c15cd50 0x7f1d9c1f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:50.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.515+0000 7f1da2a06640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d9c1538f0 0x7f1d9c1f2d80 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:50.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.515+0000 7f1da2a06640 1 --2- 172.21.15.46:0/1435175808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d9c0cefa0 0x7f1d9c0c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:50.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.515+0000 7f1da2a06640 1 -- 172.21.15.46:0/1435175808 >> 172.21.15.46:0/1435175808 conn(0x7f1d9c06a640 msgr2=0x7f1d9c0c9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:50.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.515+0000 7f1da2a06640 1 -- 172.21.15.46:0/1435175808 shutdown_connections 2023-10-30T14:24:50.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:50.515+0000 7f1da2a06640 1 -- 172.21.15.46:0/1435175808 wait complete. 2023-10-30T14:24:50.537 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:24:52.538 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:52.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7c5f5640 1 Processor -- start 2023-10-30T14:24:52.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7c5f5640 1 -- start start 2023-10-30T14:24:52.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7c5f5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0f74157080 0x7f0f74071b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:52.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7c5f5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f740c8da0 con 0x7f0f740c7990 2023-10-30T14:24:52.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7c5f5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7406cb60 con 0x7f0f740c8600 2023-10-30T14:24:52.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7c5f5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f74157460 con 0x7f0f74157080 2023-10-30T14:24:52.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f79b90640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0f740c7990 0x7f0f740c7ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44020/0 (socket says 172.21.15.46:44020) 2023-10-30T14:24:52.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f79b90640 1 -- 172.21.15.46:0/1643096277 learned_addr learned my addr 172.21.15.46:0/1643096277 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:52.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7a391640 1 --2- 172.21.15.46:0/1643096277 >> v2:172.21.15.154:3300/0 conn(0x7f0f74157080 0x7f0f74071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7938f640 1 -- 172.21.15.46:0/1643096277 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829049127 0 0) 0x7f0f740c8da0 con 0x7f0f740c7990 2023-10-30T14:24:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7938f640 1 -- 172.21.15.46:0/1643096277 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f60003530 con 0x7f0f740c7990 2023-10-30T14:24:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7938f640 1 -- 172.21.15.46:0/1643096277 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 525756287 0 0) 0x7f0f60003530 con 0x7f0f740c7990 2023-10-30T14:24:52.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7a391640 1 -- 172.21.15.46:0/1643096277 >> v1:172.21.15.154:6789/0 conn(0x7f0f740c8600 legacy=0x7f0f74164220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:52.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7a391640 1 -- 172.21.15.46:0/1643096277 >> v1:172.21.15.46:6789/0 conn(0x7f0f740c7990 legacy=0x7f0f740c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:52.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.679+0000 7f0f7a391640 1 -- 172.21.15.46:0/1643096277 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7406ce20 con 0x7f0f74157080 2023-10-30T14:24:52.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.683+0000 7f0f7a391640 1 --2- 172.21.15.46:0/1643096277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f74157080 0x7f0f74071b50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0f700097c0 tx=0x7f0f7001c280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a790a3b0f3959d3c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:52.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.683+0000 7f0f7938f640 1 -- 172.21.15.46:0/1643096277 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64001680 con 0x7f0f74157080 2023-10-30T14:24:52.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.683+0000 7f0f7938f640 1 -- 172.21.15.46:0/1643096277 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f700a7050 con 0x7f0f74157080 2023-10-30T14:24:52.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.683+0000 7f0f7938f640 1 -- 172.21.15.46:0/1643096277 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f700ab600 con 0x7f0f74157080 2023-10-30T14:24:52.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.683+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/1643096277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f74157080 msgr2=0x7f0f74071b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:52.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.683+0000 7f0f7c5f5640 1 --2- 172.21.15.46:0/1643096277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f74157080 0x7f0f74071b50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0f700097c0 tx=0x7f0f7001c280 comp rx=0 tx=0).stop 2023-10-30T14:24:52.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.683+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/1643096277 shutdown_connections 2023-10-30T14:24:52.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.683+0000 7f0f7c5f5640 1 --2- 172.21.15.46:0/1643096277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f74157080 0x7f0f74071b50 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:52.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.683+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/1643096277 >> 172.21.15.46:0/1643096277 conn(0x7f0f74069e90 msgr2=0x7f0f7406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:52.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.683+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/1643096277 shutdown_connections 2023-10-30T14:24:52.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.687+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/1643096277 wait complete. 2023-10-30T14:24:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.687+0000 7f0f7c5f5640 1 Processor -- start 2023-10-30T14:24:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.687+0000 7f0f7c5f5640 1 -- start start 2023-10-30T14:24:52.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.687+0000 7f0f7c5f5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f740c7990 0x7f0f74154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:52.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.687+0000 7f0f7c5f5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f740c8600 0x7f0f740cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.687+0000 7f0f7c5f5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f74157080 0x7f0f740cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.687+0000 7f0f7c5f5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f741f2140 con 0x7f0f740c8600 2023-10-30T14:24:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.687+0000 7f0f7c5f5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f741f63d0 con 0x7f0f740c7990 2023-10-30T14:24:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.687+0000 7f0f7c5f5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f741f2350 con 0x7f0f74157080 2023-10-30T14:24:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f79b90640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f740c8600 0x7f0f740cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f79b90640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f740c8600 0x7f0f740cb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42268/0 (socket says 172.21.15.46:42268) 2023-10-30T14:24:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f79b90640 1 -- 172.21.15.46:0/3153334972 learned_addr learned my addr 172.21.15.46:0/3153334972 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f7a391640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f740c7990 0x7f0f74154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f7ab92640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f74157080 0x7f0f740cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f79b90640 1 -- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f74157080 msgr2=0x7f0f740cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f79b90640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f74157080 0x7f0f740cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:52.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f79b90640 1 -- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f740c7990 msgr2=0x7f0f74154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:52.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f79b90640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f740c7990 0x7f0f74154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:52.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f79b90640 1 -- 172.21.15.46:0/3153334972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f70009420 con 0x7f0f740c8600 2023-10-30T14:24:52.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f79b90640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f740c8600 0x7f0f740cb3d0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f0f6400c750 tx=0x7f0f6400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:52.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f6affd640 1 -- 172.21.15.46:0/3153334972 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64005ae0 con 0x7f0f740c8600 2023-10-30T14:24:52.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f6affd640 1 -- 172.21.15.46:0/3153334972 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f64005c40 con 0x7f0f740c8600 2023-10-30T14:24:52.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/3153334972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f741f2630 con 0x7f0f740c8600 2023-10-30T14:24:52.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f6affd640 1 -- 172.21.15.46:0/3153334972 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f640072b0 con 0x7f0f740c8600 2023-10-30T14:24:52.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/3153334972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f741f2bf0 con 0x7f0f740c8600 2023-10-30T14:24:52.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/3153334972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7404e170 con 0x7f0f740c8600 2023-10-30T14:24:52.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.691+0000 7f0f6affd640 1 -- 172.21.15.46:0/3153334972 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0f64007410 con 0x7f0f740c8600 2023-10-30T14:24:52.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.695+0000 7f0f6affd640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f580664d0 0x7f0f58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:52.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.695+0000 7f0f6affd640 1 -- 172.21.15.46:0/3153334972 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8662+0+0 (secure 0 0 0) 0x7f0f64084220 con 0x7f0f740c8600 2023-10-30T14:24:52.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.695+0000 7f0f7a391640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f580664d0 0x7f0f58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:52.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.695+0000 7f0f7a391640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f580664d0 0x7f0f58068990 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f0f700ad000 tx=0x7f0f700a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:52.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.699+0000 7f0f6affd640 1 -- 172.21.15.46:0/3153334972 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0f64050a80 con 0x7f0f740c8600 2023-10-30T14:24:52.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.887+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/3153334972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f7406fb90 con 0x7f0f740c8600 2023-10-30T14:24:52.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.887+0000 7f0f6affd640 1 -- 172.21.15.46:0/3153334972 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+6527 (secure 0 0 0) 0x7f0f64017070 con 0x7f0f740c8600 2023-10-30T14:24:52.891 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:52.891 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":10020,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":258}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:32.905080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:52.892 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 259 2023-10-30T14:24:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.891+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f580664d0 msgr2=0x7f0f58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.891+0000 7f0f7c5f5640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f580664d0 0x7f0f58068990 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f0f700ad000 tx=0x7f0f700a6000 comp rx=0 tx=0).stop 2023-10-30T14:24:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.891+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/3153334972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f740c8600 msgr2=0x7f0f740cb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.891+0000 7f0f7c5f5640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f740c8600 0x7f0f740cb3d0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f0f6400c750 tx=0x7f0f6400cc10 comp rx=0 tx=0).stop 2023-10-30T14:24:52.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.891+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/3153334972 shutdown_connections 2023-10-30T14:24:52.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.891+0000 7f0f7c5f5640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0f580664d0 0x7f0f58068990 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:52.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.891+0000 7f0f7c5f5640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f74157080 0x7f0f740cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:52.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.891+0000 7f0f7c5f5640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f740c8600 0x7f0f740cb3d0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:52.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.891+0000 7f0f7c5f5640 1 --2- 172.21.15.46:0/3153334972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f740c7990 0x7f0f74154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:52.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.891+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/3153334972 >> 172.21.15.46:0/3153334972 conn(0x7f0f74069e90 msgr2=0x7f0f7406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:52.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.891+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/3153334972 shutdown_connections 2023-10-30T14:24:52.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:52.891+0000 7f0f7c5f5640 1 -- 172.21.15.46:0/3153334972 wait complete. 2023-10-30T14:24:52.913 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:24:54.913 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd358e09640 1 Processor -- start 2023-10-30T14:24:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd358e09640 1 -- start start 2023-10-30T14:24:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd358e09640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3540c6bd0 0x7fd35406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd358e09640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd354071110 0x7fd354158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd358e09640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd354066cf0 con 0x7fd35406f7a0 2023-10-30T14:24:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd358e09640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3540700c0 con 0x7fd3540c6bd0 2023-10-30T14:24:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd358e09640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3540c70d0 con 0x7fd354071110 2023-10-30T14:24:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd34bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3540c6bd0 0x7fd35406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd34bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3540c6bd0 0x7fd35406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42282/0 (socket says 172.21.15.46:42282) 2023-10-30T14:24:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd34bfff640 1 -- 172.21.15.46:0/2717383912 learned_addr learned my addr 172.21.15.46:0/2717383912 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd352d9d640 1 --2- 172.21.15.46:0/2717383912 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd354071110 0x7fd354158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd34bfff640 1 -- 172.21.15.46:0/2717383912 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd354071110 msgr2=0x7fd354158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd34bfff640 1 --2- 172.21.15.46:0/2717383912 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd354071110 0x7fd354158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd34bfff640 1 -- 172.21.15.46:0/2717383912 >> v1:172.21.15.46:6789/0 conn(0x7fd35406f7a0 legacy=0x7fd354069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd34bfff640 1 -- 172.21.15.46:0/2717383912 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3540c7390 con 0x7fd3540c6bd0 2023-10-30T14:24:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.059+0000 7fd351d9b640 1 -- 172.21.15.46:0/2717383912 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204761512 0 0) 0x7fd354066cf0 con 0x7fd35406f7a0 2023-10-30T14:24:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.063+0000 7fd34bfff640 1 --2- 172.21.15.46:0/2717383912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3540c6bd0 0x7fd35406fb80 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fd33c009870 tx=0x7fd33c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6409e0f9fc72a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.063+0000 7fd351d9b640 1 -- 172.21.15.46:0/2717383912 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c0a8a30 con 0x7fd3540c6bd0 2023-10-30T14:24:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.063+0000 7fd351d9b640 1 -- 172.21.15.46:0/2717383912 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c0a8b90 con 0x7fd3540c6bd0 2023-10-30T14:24:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.063+0000 7fd351d9b640 1 -- 172.21.15.46:0/2717383912 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c0b0440 con 0x7fd3540c6bd0 2023-10-30T14:24:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.063+0000 7fd358e09640 1 -- 172.21.15.46:0/2717383912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3540c6bd0 msgr2=0x7fd35406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.063+0000 7fd358e09640 1 --2- 172.21.15.46:0/2717383912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3540c6bd0 0x7fd35406fb80 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fd33c009870 tx=0x7fd33c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:24:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.063+0000 7fd358e09640 1 -- 172.21.15.46:0/2717383912 shutdown_connections 2023-10-30T14:24:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.063+0000 7fd358e09640 1 --2- 172.21.15.46:0/2717383912 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd354071110 0x7fd354158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.063+0000 7fd358e09640 1 --2- 172.21.15.46:0/2717383912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3540c6bd0 0x7fd35406fb80 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.063+0000 7fd358e09640 1 -- 172.21.15.46:0/2717383912 >> 172.21.15.46:0/2717383912 conn(0x7fd3540c7910 msgr2=0x7fd3540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.063+0000 7fd358e09640 1 -- 172.21.15.46:0/2717383912 shutdown_connections 2023-10-30T14:24:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.063+0000 7fd358e09640 1 -- 172.21.15.46:0/2717383912 wait complete. 2023-10-30T14:24:55.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd358e09640 1 Processor -- start 2023-10-30T14:24:55.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd358e09640 1 -- start start 2023-10-30T14:24:55.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd358e09640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd35406f7a0 0x7fd3541565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:55.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd358e09640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd354071110 0x7fd3540c07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:55.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd358e09640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3540c6bd0 0x7fd3540c0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd358e09640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3540c1b70 con 0x7fd3540c6bd0 2023-10-30T14:24:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd358e09640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3540bfff0 con 0x7fd354071110 2023-10-30T14:24:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd358e09640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3541efad0 con 0x7fd35406f7a0 2023-10-30T14:24:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd352d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3540c6bd0 0x7fd3540c0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd352d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3540c6bd0 0x7fd3540c0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42290/0 (socket says 172.21.15.46:42290) 2023-10-30T14:24:55.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd352d9d640 1 -- 172.21.15.46:0/2287333291 learned_addr learned my addr 172.21.15.46:0/2287333291 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:55.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd34bfff640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd354071110 0x7fd3540c07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:55.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd35259c640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd35406f7a0 0x7fd3541565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:55.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd352d9d640 1 -- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd35406f7a0 msgr2=0x7fd3541565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:55.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd352d9d640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd35406f7a0 0x7fd3541565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:55.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd352d9d640 1 -- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd354071110 msgr2=0x7fd3540c07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:55.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd352d9d640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd354071110 0x7fd3540c07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:55.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd352d9d640 1 -- 172.21.15.46:0/2287333291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0094d0 con 0x7fd3540c6bd0 2023-10-30T14:24:55.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd352d9d640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3540c6bd0 0x7fd3540c0cf0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fd34c007e40 tx=0x7fd34c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:55.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd34affd640 1 -- 172.21.15.46:0/2287333291 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c014620 con 0x7fd3540c6bd0 2023-10-30T14:24:55.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd34affd640 1 -- 172.21.15.46:0/2287333291 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c00ee90 con 0x7fd3540c6bd0 2023-10-30T14:24:55.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd34affd640 1 -- 172.21.15.46:0/2287333291 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c0149e0 con 0x7fd3540c6bd0 2023-10-30T14:24:55.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd358e09640 1 -- 172.21.15.46:0/2287333291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3540c15b0 con 0x7fd3540c6bd0 2023-10-30T14:24:55.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.067+0000 7fd358e09640 1 -- 172.21.15.46:0/2287333291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3540c0290 con 0x7fd3540c6bd0 2023-10-30T14:24:55.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.071+0000 7fd358e09640 1 -- 172.21.15.46:0/2287333291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3540c0480 con 0x7fd3540c6bd0 2023-10-30T14:24:55.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.071+0000 7fd34affd640 1 -- 172.21.15.46:0/2287333291 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd34c014b40 con 0x7fd3540c6bd0 2023-10-30T14:24:55.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.071+0000 7fd34affd640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3340664d0 0x7fd334068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:55.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.071+0000 7fd34affd640 1 -- 172.21.15.46:0/2287333291 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8662+0+0 (secure 0 0 0) 0x7fd34c085300 con 0x7fd3540c6bd0 2023-10-30T14:24:55.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.071+0000 7fd35259c640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3340664d0 0x7fd334068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:55.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.071+0000 7fd35259c640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3340664d0 0x7fd334068990 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fd344007fd0 tx=0x7fd344009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.079+0000 7fd34affd640 1 -- 172.21.15.46:0/2287333291 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd34c051bf0 con 0x7fd3540c6bd0 2023-10-30T14:24:55.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.271+0000 7fd358e09640 1 -- 172.21.15.46:0/2287333291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd35404e170 con 0x7fd3540c6bd0 2023-10-30T14:24:55.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.271+0000 7fd34affd640 1 -- 172.21.15.46:0/2287333291 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+6527 (secure 0 0 0) 0x7fd34c055210 con 0x7fd3540c6bd0 2023-10-30T14:24:55.275 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:55.275 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":10020,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":258}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:32.905080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:55.275 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 259 2023-10-30T14:24:55.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.275+0000 7fd358e09640 1 -- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3340664d0 msgr2=0x7fd334068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:55.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.275+0000 7fd358e09640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3340664d0 0x7fd334068990 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fd344007fd0 tx=0x7fd344009040 comp rx=0 tx=0).stop 2023-10-30T14:24:55.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.275+0000 7fd358e09640 1 -- 172.21.15.46:0/2287333291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3540c6bd0 msgr2=0x7fd3540c0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:55.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.275+0000 7fd358e09640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3540c6bd0 0x7fd3540c0cf0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fd34c007e40 tx=0x7fd34c00e4b0 comp rx=0 tx=0).stop 2023-10-30T14:24:55.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.275+0000 7fd358e09640 1 -- 172.21.15.46:0/2287333291 shutdown_connections 2023-10-30T14:24:55.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.275+0000 7fd358e09640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd3340664d0 0x7fd334068990 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:55.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.275+0000 7fd358e09640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3540c6bd0 0x7fd3540c0cf0 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:55.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.275+0000 7fd358e09640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd354071110 0x7fd3540c07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:55.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.275+0000 7fd358e09640 1 --2- 172.21.15.46:0/2287333291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd35406f7a0 0x7fd3541565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:55.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.275+0000 7fd358e09640 1 -- 172.21.15.46:0/2287333291 >> 172.21.15.46:0/2287333291 conn(0x7fd3540c7910 msgr2=0x7fd3540c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:55.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.275+0000 7fd358e09640 1 -- 172.21.15.46:0/2287333291 shutdown_connections 2023-10-30T14:24:55.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:55.275+0000 7fd358e09640 1 -- 172.21.15.46:0/2287333291 wait complete. 2023-10-30T14:24:55.296 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:24:57.297 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:57.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d31e79640 1 Processor -- start 2023-10-30T14:24:57.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d31e79640 1 -- start start 2023-10-30T14:24:57.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d31e79640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d2c1538a0 0x7f3d2c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:57.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d31e79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d2c0cf160 0x7f3d2c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:57.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d31e79640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2c0cb800 con 0x7f3d2c0ca0e0 2023-10-30T14:24:57.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d31e79640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d2c15dc30 con 0x7f3d2c1538a0 2023-10-30T14:24:57.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d31e79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0cb330 con 0x7f3d2c0cf160 2023-10-30T14:24:57.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d2affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3d2c0ca0e0 0x7f3d2c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38848/0 (socket says 172.21.15.46:38848) 2023-10-30T14:24:57.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d2affd640 1 -- 172.21.15.46:0/537059339 learned_addr learned my addr 172.21.15.46:0/537059339 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:24:57.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d2b7fe640 1 --2- 172.21.15.46:0/537059339 >> v2:172.21.15.154:3300/0 conn(0x7f3d2c1538a0 0x7f3d2c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:57.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d2bfff640 1 --2- 172.21.15.46:0/537059339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d2c0cf160 0x7f3d2c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:57.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d2a7fc640 1 -- 172.21.15.46:0/537059339 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535051814 0 0) 0x7f3d2c0cb800 con 0x7f3d2c0ca0e0 2023-10-30T14:24:57.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.439+0000 7f3d2a7fc640 1 -- 172.21.15.46:0/537059339 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d14003530 con 0x7f3d2c0ca0e0 2023-10-30T14:24:57.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.443+0000 7f3d2b7fe640 1 -- 172.21.15.46:0/537059339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d2c0cf160 msgr2=0x7f3d2c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:57.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.443+0000 7f3d2b7fe640 1 --2- 172.21.15.46:0/537059339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d2c0cf160 0x7f3d2c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:57.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.443+0000 7f3d2b7fe640 1 -- 172.21.15.46:0/537059339 >> v1:172.21.15.46:6789/0 conn(0x7f3d2c0ca0e0 legacy=0x7f3d2c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:57.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.443+0000 7f3d2b7fe640 1 -- 172.21.15.46:0/537059339 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c0cad70 con 0x7f3d2c1538a0 2023-10-30T14:24:57.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.443+0000 7f3d2b7fe640 1 --2- 172.21.15.46:0/537059339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c1538a0 0x7f3d2c15d6f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0097c0 tx=0x7f3d1c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=262903c604ca13ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:57.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.443+0000 7f3d2a7fc640 1 -- 172.21.15.46:0/537059339 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0a8970 con 0x7f3d2c1538a0 2023-10-30T14:24:57.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.443+0000 7f3d2a7fc640 1 -- 172.21.15.46:0/537059339 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0a8ad0 con 0x7f3d2c1538a0 2023-10-30T14:24:57.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.443+0000 7f3d2a7fc640 1 -- 172.21.15.46:0/537059339 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0a8e40 con 0x7f3d2c1538a0 2023-10-30T14:24:57.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 -- 172.21.15.46:0/537059339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c1538a0 msgr2=0x7f3d2c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:57.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 --2- 172.21.15.46:0/537059339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c1538a0 0x7f3d2c15d6f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0097c0 tx=0x7f3d1c01ec70 comp rx=0 tx=0).stop 2023-10-30T14:24:57.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 -- 172.21.15.46:0/537059339 shutdown_connections 2023-10-30T14:24:57.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 --2- 172.21.15.46:0/537059339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d2c0cf160 0x7f3d2c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:57.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 --2- 172.21.15.46:0/537059339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c1538a0 0x7f3d2c15d6f0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:57.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 -- 172.21.15.46:0/537059339 >> 172.21.15.46:0/537059339 conn(0x7f3d2c06bfe0 msgr2=0x7f3d2c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:57.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 -- 172.21.15.46:0/537059339 shutdown_connections 2023-10-30T14:24:57.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 -- 172.21.15.46:0/537059339 wait complete. 2023-10-30T14:24:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 Processor -- start 2023-10-30T14:24:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 -- start start 2023-10-30T14:24:57.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0ca0e0 0x7f3d2c157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:57.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d2c0cf160 0x7f3d2c1f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:57.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d2c1538a0 0x7f3d2c1efb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:57.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c157560 con 0x7f3d2c1538a0 2023-10-30T14:24:57.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c1f2a70 con 0x7f3d2c0ca0e0 2023-10-30T14:24:57.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d31e79640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c1f2d50 con 0x7f3d2c0cf160 2023-10-30T14:24:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d2bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d2c1538a0 0x7f3d2c1efb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d2bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d2c1538a0 0x7f3d2c1efb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56676/0 (socket says 172.21.15.46:56676) 2023-10-30T14:24:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d2bfff640 1 -- 172.21.15.46:0/81096075 learned_addr learned my addr 172.21.15.46:0/81096075 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d2b7fe640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0ca0e0 0x7f3d2c157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.447+0000 7f3d2affd640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d2c0cf160 0x7f3d2c1f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d2bfff640 1 -- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d2c0cf160 msgr2=0x7f3d2c1f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d2bfff640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d2c0cf160 0x7f3d2c1f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d2bfff640 1 -- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0ca0e0 msgr2=0x7f3d2c157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d2bfff640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0ca0e0 0x7f3d2c157f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d2bfff640 1 -- 172.21.15.46:0/81096075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c009420 con 0x7f3d2c1538a0 2023-10-30T14:24:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d2bfff640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d2c1538a0 0x7f3d2c1efb50 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400efc0 tx=0x7f3d2400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d13fff640 1 -- 172.21.15.46:0/81096075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24009b10 con 0x7f3d2c1538a0 2023-10-30T14:24:57.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d13fff640 1 -- 172.21.15.46:0/81096075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2400f040 con 0x7f3d2c1538a0 2023-10-30T14:24:57.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d31e79640 1 -- 172.21.15.46:0/81096075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2c0c91e0 con 0x7f3d2c1538a0 2023-10-30T14:24:57.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d13fff640 1 -- 172.21.15.46:0/81096075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d240146c0 con 0x7f3d2c1538a0 2023-10-30T14:24:57.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d31e79640 1 -- 172.21.15.46:0/81096075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2c1577b0 con 0x7f3d2c1538a0 2023-10-30T14:24:57.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d31e79640 1 -- 172.21.15.46:0/81096075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d2c0b2a50 con 0x7f3d2c1538a0 2023-10-30T14:24:57.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d13fff640 1 -- 172.21.15.46:0/81096075 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3d24014820 con 0x7f3d2c1538a0 2023-10-30T14:24:57.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d13fff640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d180664a0 0x7f3d18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:57.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d13fff640 1 -- 172.21.15.46:0/81096075 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8662+0+0 (secure 0 0 0) 0x7f3d240858c0 con 0x7f3d2c1538a0 2023-10-30T14:24:57.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.451+0000 7f3d2b7fe640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d180664a0 0x7f3d18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:57.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.455+0000 7f3d2b7fe640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d180664a0 0x7f3d18068960 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0b2000 tx=0x7f3d1c0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:57.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.459+0000 7f3d13fff640 1 -- 172.21.15.46:0/81096075 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3d24052080 con 0x7f3d2c1538a0 2023-10-30T14:24:57.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d31e79640 1 -- 172.21.15.46:0/81096075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d2c1f2ec0 con 0x7f3d2c1538a0 2023-10-30T14:24:57.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d13fff640 1 -- 172.21.15.46:0/81096075 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+6527 (secure 0 0 0) 0x7f3d24018090 con 0x7f3d2c1538a0 2023-10-30T14:24:57.661 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:24:57.662 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":10020,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":258}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:32.905080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9735,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9735":{"gid":9735,"name":"c","rank":0,"incarnation":246,"state":"up:active","state_seq":33,"addr":"172.21.15.46:6836/3574404756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3574404756},{"type":"v1","addr":"172.21.15.46:6836","nonce":3574404756}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:24:57.662 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 259 2023-10-30T14:24:57.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d31e79640 1 -- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d180664a0 msgr2=0x7f3d18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:57.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d31e79640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d180664a0 0x7f3d18068960 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0b2000 tx=0x7f3d1c0aa070 comp rx=0 tx=0).stop 2023-10-30T14:24:57.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d31e79640 1 -- 172.21.15.46:0/81096075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d2c1538a0 msgr2=0x7f3d2c1efb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:57.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d31e79640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d2c1538a0 0x7f3d2c1efb50 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400efc0 tx=0x7f3d2400c490 comp rx=0 tx=0).stop 2023-10-30T14:24:57.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d31e79640 1 -- 172.21.15.46:0/81096075 shutdown_connections 2023-10-30T14:24:57.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d31e79640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d180664a0 0x7f3d18068960 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:57.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d31e79640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d2c1538a0 0x7f3d2c1efb50 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:57.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d31e79640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d2c0cf160 0x7f3d2c1f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:57.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d31e79640 1 --2- 172.21.15.46:0/81096075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d2c0ca0e0 0x7f3d2c157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:57.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d31e79640 1 -- 172.21.15.46:0/81096075 >> 172.21.15.46:0/81096075 conn(0x7f3d2c06bfe0 msgr2=0x7f3d2c1516e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:57.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.659+0000 7f3d31e79640 1 -- 172.21.15.46:0/81096075 shutdown_connections 2023-10-30T14:24:57.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:57.663+0000 7f3d31e79640 1 -- 172.21.15.46:0/81096075 wait complete. 2023-10-30T14:24:57.683 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T14:24:59.685 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:24:59.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305c99d640 1 Processor -- start 2023-10-30T14:24:59.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305c99d640 1 -- start start 2023-10-30T14:24:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305c99d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3058153d70 0x7f305806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305c99d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f30580c8490 0x7f30580c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305c99d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f305806afa0 con 0x7f305806e470 2023-10-30T14:24:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305c99d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f305806d780 con 0x7f3058153d70 2023-10-30T14:24:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305c99d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3058154150 con 0x7f30580c8490 2023-10-30T14:24:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3058153d70 0x7f305806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3058153d70 0x7f305806ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56680/0 (socket says 172.21.15.46:56680) 2023-10-30T14:24:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305659c640 1 -- 172.21.15.46:0/698365861 learned_addr learned my addr 172.21.15.46:0/698365861 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f3056d9d640 1 --2- 172.21.15.46:0/698365861 >> v2:172.21.15.154:3300/0 conn(0x7f30580c8490 0x7f30580c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305659c640 1 -- 172.21.15.46:0/698365861 >> v2:172.21.15.154:3300/0 conn(0x7f30580c8490 msgr2=0x7f30580c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305659c640 1 --2- 172.21.15.46:0/698365861 >> v2:172.21.15.154:3300/0 conn(0x7f30580c8490 0x7f30580c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305659c640 1 -- 172.21.15.46:0/698365861 >> v1:172.21.15.154:6789/0 conn(0x7f305806e470 legacy=0x7f30580cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:24:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.827+0000 7f305659c640 1 -- 172.21.15.46:0/698365861 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305806e030 con 0x7f3058153d70 2023-10-30T14:24:59.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.831+0000 7f305659c640 1 --2- 172.21.15.46:0/698365861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3058153d70 0x7f305806ca80 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f30580cddf0 tx=0x7f304801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c5497955a1751d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:59.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.831+0000 7f3055d9b640 1 -- 172.21.15.46:0/698365861 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30480aa5f0 con 0x7f3058153d70 2023-10-30T14:24:59.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.831+0000 7f3055d9b640 1 -- 172.21.15.46:0/698365861 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30480aa750 con 0x7f3058153d70 2023-10-30T14:24:59.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.831+0000 7f3055d9b640 1 -- 172.21.15.46:0/698365861 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30480aaae0 con 0x7f3058153d70 2023-10-30T14:24:59.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.831+0000 7f305c99d640 1 -- 172.21.15.46:0/698365861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3058153d70 msgr2=0x7f305806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:59.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.831+0000 7f305c99d640 1 --2- 172.21.15.46:0/698365861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3058153d70 0x7f305806ca80 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f30580cddf0 tx=0x7f304801c490 comp rx=0 tx=0).stop 2023-10-30T14:24:59.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 -- 172.21.15.46:0/698365861 shutdown_connections 2023-10-30T14:24:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 --2- 172.21.15.46:0/698365861 >> v2:172.21.15.154:3300/0 conn(0x7f30580c8490 0x7f30580c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 --2- 172.21.15.46:0/698365861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3058153d70 0x7f305806ca80 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 -- 172.21.15.46:0/698365861 >> 172.21.15.46:0/698365861 conn(0x7f3058071ea0 msgr2=0x7f3058065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:24:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 -- 172.21.15.46:0/698365861 shutdown_connections 2023-10-30T14:24:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 -- 172.21.15.46:0/698365861 wait complete. 2023-10-30T14:24:59.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 Processor -- start 2023-10-30T14:24:59.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 -- start start 2023-10-30T14:24:59.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305806e470 0x7f3058145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:59.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30580c8490 0x7f30580cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:59.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3058153d70 0x7f30580cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:59.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3058146940 con 0x7f3058153d70 2023-10-30T14:24:59.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3058149b00 con 0x7f305806e470 2023-10-30T14:24:59.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305c99d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3058154b30 con 0x7f30580c8490 2023-10-30T14:24:59.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f3056d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3058153d70 0x7f30580cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:59.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f3056d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3058153d70 0x7f30580cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56688/0 (socket says 172.21.15.46:56688) 2023-10-30T14:24:59.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f3056d9d640 1 -- 172.21.15.46:0/1359121311 learned_addr learned my addr 172.21.15.46:0/1359121311 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:24:59.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f305659c640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305806e470 0x7f3058145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:59.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f304ffff640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30580c8490 0x7f30580cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:59.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f3056d9d640 1 -- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30580c8490 msgr2=0x7f30580cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:59.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f3056d9d640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30580c8490 0x7f30580cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:59.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f3056d9d640 1 -- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305806e470 msgr2=0x7f3058145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:24:59.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f3056d9d640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305806e470 0x7f3058145cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:24:59.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.835+0000 7f3056d9d640 1 -- 172.21.15.46:0/1359121311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30480094d0 con 0x7f3058153d70 2023-10-30T14:24:59.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.839+0000 7f3056d9d640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3058153d70 0x7f30580cb1b0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f3050007e40 tx=0x7f305000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:59.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.839+0000 7f304effd640 1 -- 172.21.15.46:0/1359121311 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050014620 con 0x7f3058153d70 2023-10-30T14:24:59.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.839+0000 7f304effd640 1 -- 172.21.15.46:0/1359121311 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305000ee90 con 0x7f3058153d70 2023-10-30T14:24:59.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.839+0000 7f305c99d640 1 -- 172.21.15.46:0/1359121311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3058149790 con 0x7f3058153d70 2023-10-30T14:24:59.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.839+0000 7f304effd640 1 -- 172.21.15.46:0/1359121311 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30500149e0 con 0x7f3058153d70 2023-10-30T14:24:59.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.839+0000 7f305c99d640 1 -- 172.21.15.46:0/1359121311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30581453c0 con 0x7f3058153d70 2023-10-30T14:24:59.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.839+0000 7f305c99d640 1 -- 172.21.15.46:0/1359121311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3058149c40 con 0x7f3058153d70 2023-10-30T14:24:59.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.839+0000 7f304effd640 1 -- 172.21.15.46:0/1359121311 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3050014b40 con 0x7f3058153d70 2023-10-30T14:24:59.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.839+0000 7f304effd640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30380664a0 0x7f3038068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:24:59.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.839+0000 7f304effd640 1 -- 172.21.15.46:0/1359121311 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8662+0+0 (secure 0 0 0) 0x7f3050085330 con 0x7f3058153d70 2023-10-30T14:24:59.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.839+0000 7f305659c640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30380664a0 0x7f3038068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:24:59.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.843+0000 7f305659c640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30380664a0 0x7f3038068960 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f30480ae000 tx=0x7f30480b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:24:59.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:24:59.843+0000 7f304effd640 1 -- 172.21.15.46:0/1359121311 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3058149c40 con 0x7f3058153d70 2023-10-30T14:25:00.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 -- 172.21.15.46:0/1359121311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3058149c40 con 0x7f3058153d70 2023-10-30T14:25:00.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f304effd640 1 -- 172.21.15.46:0/1359121311 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f3058149c40 con 0x7f3058153d70 2023-10-30T14:25:00.045 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:00.045 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":260,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:59.863388+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:00.046 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 260 2023-10-30T14:25:00.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 -- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30380664a0 msgr2=0x7f3038068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30380664a0 0x7f3038068960 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f30480ae000 tx=0x7f30480b5040 comp rx=0 tx=0).stop 2023-10-30T14:25:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 -- 172.21.15.46:0/1359121311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3058153d70 msgr2=0x7f30580cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3058153d70 0x7f30580cb1b0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f3050007e40 tx=0x7f305000e4b0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 -- 172.21.15.46:0/1359121311 shutdown_connections 2023-10-30T14:25:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f30380664a0 0x7f3038068960 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3058153d70 0x7f30580cb1b0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30580c8490 0x7f30580cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 --2- 172.21.15.46:0/1359121311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305806e470 0x7f3058145cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 -- 172.21.15.46:0/1359121311 >> 172.21.15.46:0/1359121311 conn(0x7f3058071ea0 msgr2=0x7f3058151ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:00.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 -- 172.21.15.46:0/1359121311 shutdown_connections 2023-10-30T14:25:00.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.043+0000 7f305c99d640 1 -- 172.21.15.46:0/1359121311 wait complete. 2023-10-30T14:25:00.066 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T14:25:00.066 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T14:25:00.067 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33518dc640 1 Processor -- start 2023-10-30T14:25:00.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33518dc640 1 -- start start 2023-10-30T14:25:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33518dc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c0cefa0 0x7f334c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33518dc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c15a6c0 con 0x7f334c15cd50 2023-10-30T14:25:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33518dc640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c0ca6a0 con 0x7f334c1538f0 2023-10-30T14:25:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33518dc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334c15d300 con 0x7f334c0cefa0 2023-10-30T14:25:00.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f334c15cd50 0x7f334c0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38854/0 (socket says 172.21.15.46:38854) 2023-10-30T14:25:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334bfff640 1 -- 172.21.15.46:0/3250917723 learned_addr learned my addr 172.21.15.46:0/3250917723 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:25:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33510db640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c0cefa0 0x7f334c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820044711 0 0) 0x7f334c15a6c0 con 0x7f334c15cd50 2023-10-30T14:25:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f333c003530 con 0x7f334c15cd50 2023-10-30T14:25:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941693555 0 0) 0x7f334c0ca6a0 con 0x7f334c1538f0 2023-10-30T14:25:00.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f334c15a6c0 con 0x7f334c1538f0 2023-10-30T14:25:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1977504825 0 0) 0x7f333c003530 con 0x7f334c15cd50 2023-10-30T14:25:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c0cefa0 msgr2=0x7f334c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c0cefa0 0x7f334c0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 >> v1:172.21.15.154:6789/0 conn(0x7f334c1538f0 legacy=0x7f334c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:00.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c15d5e0 con 0x7f334c15cd50 2023-10-30T14:25:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f3338001680 con 0x7f334c15cd50 2023-10-30T14:25:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33510db640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c0cefa0 0x7f334c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:25:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 >> v1:172.21.15.46:6789/0 conn(0x7f334c15cd50 legacy=0x7f334c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c0cefa0 0x7f333c003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f333c004080 0x7f333c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f333c004d10 0x7f333c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c15d300 con 0x7f333c004d10 2023-10-30T14:25:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c00ba40 con 0x7f333c004080 2023-10-30T14:25:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f333c00bc00 con 0x7f334c0cefa0 2023-10-30T14:25:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33510db640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f333c004d10 0x7f333c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f334bfff640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f333c004080 0x7f333c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33508da640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c0cefa0 0x7f333c003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33510db640 1 -- 172.21.15.46:0/3250917723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c0cefa0 msgr2=0x7f333c003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33510db640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c0cefa0 0x7f333c003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33510db640 1 -- 172.21.15.46:0/3250917723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f333c004080 msgr2=0x7f333c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33510db640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f333c004080 0x7f333c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.211+0000 7f33510db640 1 -- 172.21.15.46:0/3250917723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344009a70 con 0x7f333c004d10 2023-10-30T14:25:00.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.215+0000 7f33510db640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f333c004d10 0x7f333c005190 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f334400e350 tx=0x7f334401cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac8b28751641dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:00.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.215+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33440a5df0 con 0x7f333c004d10 2023-10-30T14:25:00.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.215+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334400f040 con 0x7f333c004d10 2023-10-30T14:25:00.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.215+0000 7f334b7fe640 1 -- 172.21.15.46:0/3250917723 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3344003ba0 con 0x7f333c004d10 2023-10-30T14:25:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.215+0000 7f33518dc640 1 -- 172.21.15.46:0/3250917723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f333c004d10 msgr2=0x7f333c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.215+0000 7f33518dc640 1 --2- 172.21.15.46:0/3250917723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f333c004d10 0x7f333c005190 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f334400e350 tx=0x7f334401cde0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.215+0000 7f33510db640 1 -- 172.21.15.46:0/3250917723 reap_dead start 2023-10-30T14:25:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.215+0000 7f33518dc640 1 -- 172.21.15.46:0/3250917723 shutdown_connections 2023-10-30T14:25:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.215+0000 7f33518dc640 1 -- 172.21.15.46:0/3250917723 >> 172.21.15.46:0/3250917723 conn(0x7f334c06a640 msgr2=0x7f334c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:00.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.215+0000 7f33518dc640 1 -- 172.21.15.46:0/3250917723 shutdown_connections 2023-10-30T14:25:00.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.215+0000 7f33518dc640 1 -- 172.21.15.46:0/3250917723 wait complete. 2023-10-30T14:25:00.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33518dc640 1 Processor -- start 2023-10-30T14:25:00.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33518dc640 1 -- start start 2023-10-30T14:25:00.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33518dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f333c004d10 0x7f334c0c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:00.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33518dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f334c159620 0x7f334c15c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:00.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33518dc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c1ef100 0x7f334c0c8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33518dc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c0c8550 con 0x7f333c004d10 2023-10-30T14:25:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33518dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c0c7630 con 0x7f334c159620 2023-10-30T14:25:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33518dc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334c0c78d0 con 0x7f334c1ef100 2023-10-30T14:25:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33508da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f333c004d10 0x7f334c0c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33508da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f333c004d10 0x7f334c0c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56706/0 (socket says 172.21.15.46:56706) 2023-10-30T14:25:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33508da640 1 -- 172.21.15.46:0/392729552 learned_addr learned my addr 172.21.15.46:0/392729552 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f334bfff640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f334c159620 0x7f334c15c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33510db640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c1ef100 0x7f334c0c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33508da640 1 -- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c1ef100 msgr2=0x7f334c0c8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33508da640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c1ef100 0x7f334c0c8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33508da640 1 -- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f334c159620 msgr2=0x7f334c15c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33508da640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f334c159620 0x7f334c15c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33508da640 1 -- 172.21.15.46:0/392729552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c0c7bb0 con 0x7f333c004d10 2023-10-30T14:25:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33508da640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f333c004d10 0x7f334c0c6be0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f334000e7b0 tx=0x7f334000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33497fa640 1 -- 172.21.15.46:0/392729552 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334000ee60 con 0x7f333c004d10 2023-10-30T14:25:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33497fa640 1 -- 172.21.15.46:0/392729552 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33400052e0 con 0x7f333c004d10 2023-10-30T14:25:00.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33497fa640 1 -- 172.21.15.46:0/392729552 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334000c6c0 con 0x7f333c004d10 2023-10-30T14:25:00.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33518dc640 1 -- 172.21.15.46:0/392729552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334c0c7e90 con 0x7f333c004d10 2023-10-30T14:25:00.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.219+0000 7f33518dc640 1 -- 172.21.15.46:0/392729552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334c15c620 con 0x7f333c004d10 2023-10-30T14:25:00.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.223+0000 7f33518dc640 1 -- 172.21.15.46:0/392729552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334c04e170 con 0x7f333c004d10 2023-10-30T14:25:00.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.223+0000 7f33497fa640 1 -- 172.21.15.46:0/392729552 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f334000c820 con 0x7f333c004d10 2023-10-30T14:25:00.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.223+0000 7f33497fa640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33300664a0 0x7f3330068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.223+0000 7f33497fa640 1 -- 172.21.15.46:0/392729552 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f3340084990 con 0x7f333c004d10 2023-10-30T14:25:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.223+0000 7f334bfff640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33300664a0 0x7f3330068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.223+0000 7f334bfff640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33300664a0 0x7f3330068960 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f3338001060 tx=0x7f3338009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:00.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.227+0000 7f33497fa640 1 -- 172.21.15.46:0/392729552 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3340051200 con 0x7f333c004d10 2023-10-30T14:25:00.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 -- 172.21.15.46:0/392729552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f334c157180 con 0x7f333c004d10 2023-10-30T14:25:00.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33497fa640 1 -- 172.21.15.46:0/392729552 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f3340054820 con 0x7f333c004d10 2023-10-30T14:25:00.420 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:00.421 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":260,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:59.863388+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:00.421 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 260 2023-10-30T14:25:00.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 -- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33300664a0 msgr2=0x7f3330068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:00.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33300664a0 0x7f3330068960 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f3338001060 tx=0x7f3338009040 comp rx=0 tx=0).stop 2023-10-30T14:25:00.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 -- 172.21.15.46:0/392729552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f333c004d10 msgr2=0x7f334c0c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:00.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f333c004d10 0x7f334c0c6be0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f334000e7b0 tx=0x7f334000ec70 comp rx=0 tx=0).stop 2023-10-30T14:25:00.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 -- 172.21.15.46:0/392729552 shutdown_connections 2023-10-30T14:25:00.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f33300664a0 0x7f3330068960 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f334c1ef100 0x7f334c0c8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f334c159620 0x7f334c15c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 --2- 172.21.15.46:0/392729552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f333c004d10 0x7f334c0c6be0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:00.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 -- 172.21.15.46:0/392729552 >> 172.21.15.46:0/392729552 conn(0x7f334c06a640 msgr2=0x7f334c0c5670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:00.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 -- 172.21.15.46:0/392729552 shutdown_connections 2023-10-30T14:25:00.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:00.419+0000 7f33518dc640 1 -- 172.21.15.46:0/392729552 wait complete. 2023-10-30T14:25:00.443 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 260, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 260, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_10020": { "addr": "172.21.15.46:6837/2187461695", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2187461695, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2187461695, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 10020, "incarnation": 260, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_9882": { "addr": "172.21.15.46:6833/3394097058", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 3394097058, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 3394097058, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9882, "incarnation": 253, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 102, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:24:59.863388+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 10020, "mds_1": 9882 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:25:02.444 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:02.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f36148ad640 1 Processor -- start 2023-10-30T14:25:02.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f36148ad640 1 -- start start 2023-10-30T14:25:02.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f36148ad640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f361015cd50 0x7f36100c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:02.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f36148ad640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36100cefa0 0x7f36100cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:02.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f36148ad640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361015a6c0 con 0x7f36101538f0 2023-10-30T14:25:02.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f36148ad640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36100cf8f0 con 0x7f36100cefa0 2023-10-30T14:25:02.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f36148ad640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361015d300 con 0x7f361015cd50 2023-10-30T14:25:02.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f360ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36100cefa0 0x7f36100cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:02.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f360ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36100cefa0 0x7f36100cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56708/0 (socket says 172.21.15.46:56708) 2023-10-30T14:25:02.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f360ffff640 1 -- 172.21.15.46:0/2904044867 learned_addr learned my addr 172.21.15.46:0/2904044867 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:02.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f360effd640 1 --2- 172.21.15.46:0/2904044867 >> v2:172.21.15.154:3300/0 conn(0x7f361015cd50 0x7f36100c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:02.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f360ffff640 1 -- 172.21.15.46:0/2904044867 >> v2:172.21.15.154:3300/0 conn(0x7f361015cd50 msgr2=0x7f36100c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:02.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f360ffff640 1 --2- 172.21.15.46:0/2904044867 >> v2:172.21.15.154:3300/0 conn(0x7f361015cd50 0x7f36100c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:02.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f360ffff640 1 -- 172.21.15.46:0/2904044867 >> v1:172.21.15.154:6789/0 conn(0x7f36101538f0 legacy=0x7f3610159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:02.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.587+0000 7f360ffff640 1 -- 172.21.15.46:0/2904044867 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361015d5e0 con 0x7f36100cefa0 2023-10-30T14:25:02.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.591+0000 7f360ffff640 1 --2- 172.21.15.46:0/2904044867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36100cefa0 0x7f36100cf3b0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f3610072e90 tx=0x7f360801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990e99e0b6f90695 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:02.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.591+0000 7f360e7fc640 1 -- 172.21.15.46:0/2904044867 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080af400 con 0x7f36100cefa0 2023-10-30T14:25:02.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.591+0000 7f360e7fc640 1 -- 172.21.15.46:0/2904044867 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36080ab050 con 0x7f36100cefa0 2023-10-30T14:25:02.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.591+0000 7f360e7fc640 1 -- 172.21.15.46:0/2904044867 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080af780 con 0x7f36100cefa0 2023-10-30T14:25:02.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.591+0000 7f36148ad640 1 -- 172.21.15.46:0/2904044867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36100cefa0 msgr2=0x7f36100cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:02.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.591+0000 7f36148ad640 1 --2- 172.21.15.46:0/2904044867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36100cefa0 0x7f36100cf3b0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f3610072e90 tx=0x7f360801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:25:02.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.591+0000 7f36148ad640 1 -- 172.21.15.46:0/2904044867 shutdown_connections 2023-10-30T14:25:02.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.591+0000 7f36148ad640 1 --2- 172.21.15.46:0/2904044867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36100cefa0 0x7f36100cf3b0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:02.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.591+0000 7f36148ad640 1 --2- 172.21.15.46:0/2904044867 >> v2:172.21.15.154:3300/0 conn(0x7f361015cd50 0x7f36100c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:02.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.591+0000 7f36148ad640 1 -- 172.21.15.46:0/2904044867 >> 172.21.15.46:0/2904044867 conn(0x7f361006a640 msgr2=0x7f3610065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:02.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.591+0000 7f36148ad640 1 -- 172.21.15.46:0/2904044867 shutdown_connections 2023-10-30T14:25:02.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.591+0000 7f36148ad640 1 -- 172.21.15.46:0/2904044867 wait complete. 2023-10-30T14:25:02.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f36148ad640 1 Processor -- start 2023-10-30T14:25:02.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f36148ad640 1 -- start start 2023-10-30T14:25:02.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f36148ad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36100cefa0 0x7f36101573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:02.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f36148ad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36101538f0 0x7f36100b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:02.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f36148ad640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f361015cd50 0x7f36101f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:02.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f36148ad640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36100b3050 con 0x7f36100cefa0 2023-10-30T14:25:02.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f36148ad640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36101578e0 con 0x7f36101538f0 2023-10-30T14:25:02.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f36148ad640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361015ac60 con 0x7f361015cd50 2023-10-30T14:25:02.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f360f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36100cefa0 0x7f36101573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:02.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f360f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36100cefa0 0x7f36101573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56710/0 (socket says 172.21.15.46:56710) 2023-10-30T14:25:02.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f360f7fe640 1 -- 172.21.15.46:0/282504593 learned_addr learned my addr 172.21.15.46:0/282504593 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:02.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f360effd640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36101538f0 0x7f36100b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:02.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f360ffff640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f361015cd50 0x7f36101f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:02.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f360f7fe640 1 -- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f361015cd50 msgr2=0x7f36101f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:02.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f360f7fe640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f361015cd50 0x7f36101f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:02.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f360f7fe640 1 -- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36101538f0 msgr2=0x7f36100b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:02.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f360f7fe640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36101538f0 0x7f36100b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:02.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f360f7fe640 1 -- 172.21.15.46:0/282504593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3608009b30 con 0x7f36100cefa0 2023-10-30T14:25:02.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f360f7fe640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36100cefa0 0x7f36101573a0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f360000c530 tx=0x7f360000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:02.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f35f7fff640 1 -- 172.21.15.46:0/282504593 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600005220 con 0x7f36100cefa0 2023-10-30T14:25:02.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f35f7fff640 1 -- 172.21.15.46:0/282504593 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3600005380 con 0x7f36100cefa0 2023-10-30T14:25:02.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f36148ad640 1 -- 172.21.15.46:0/282504593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361015af40 con 0x7f36100cefa0 2023-10-30T14:25:02.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.595+0000 7f35f7fff640 1 -- 172.21.15.46:0/282504593 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600007270 con 0x7f36100cefa0 2023-10-30T14:25:02.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.599+0000 7f36148ad640 1 -- 172.21.15.46:0/282504593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36100b3580 con 0x7f36100cefa0 2023-10-30T14:25:02.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.599+0000 7f36148ad640 1 -- 172.21.15.46:0/282504593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361004e170 con 0x7f36100cefa0 2023-10-30T14:25:02.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.599+0000 7f35f7fff640 1 -- 172.21.15.46:0/282504593 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f36000073d0 con 0x7f36100cefa0 2023-10-30T14:25:02.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.599+0000 7f35f7fff640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35fc0664d0 0x7f35fc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:02.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.599+0000 7f35f7fff640 1 -- 172.21.15.46:0/282504593 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f3600084410 con 0x7f36100cefa0 2023-10-30T14:25:02.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.599+0000 7f360effd640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35fc0664d0 0x7f35fc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:02.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.603+0000 7f360effd640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35fc0664d0 0x7f35fc068990 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f361013f4e0 tx=0x7f3604008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:02.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.603+0000 7f35f7fff640 1 -- 172.21.15.46:0/282504593 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3600050c40 con 0x7f36100cefa0 2023-10-30T14:25:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.795+0000 7f36148ad640 1 -- 172.21.15.46:0/282504593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36100b2290 con 0x7f36100cefa0 2023-10-30T14:25:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.795+0000 7f35f7fff640 1 -- 172.21.15.46:0/282504593 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f3600054260 con 0x7f36100cefa0 2023-10-30T14:25:02.799 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:02.799 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":260,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:59.863388+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 260 2023-10-30T14:25:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.799+0000 7f36148ad640 1 -- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35fc0664d0 msgr2=0x7f35fc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.799+0000 7f36148ad640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35fc0664d0 0x7f35fc068990 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f361013f4e0 tx=0x7f3604008040 comp rx=0 tx=0).stop 2023-10-30T14:25:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.799+0000 7f36148ad640 1 -- 172.21.15.46:0/282504593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36100cefa0 msgr2=0x7f36101573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.799+0000 7f36148ad640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36100cefa0 0x7f36101573a0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f360000c530 tx=0x7f360000c9f0 comp rx=0 tx=0).stop 2023-10-30T14:25:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.799+0000 7f36148ad640 1 -- 172.21.15.46:0/282504593 shutdown_connections 2023-10-30T14:25:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.799+0000 7f36148ad640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f35fc0664d0 0x7f35fc068990 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:02.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.799+0000 7f36148ad640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f361015cd50 0x7f36101f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:02.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.799+0000 7f36148ad640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36101538f0 0x7f36100b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:02.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.799+0000 7f36148ad640 1 --2- 172.21.15.46:0/282504593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36100cefa0 0x7f36101573a0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:02.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.799+0000 7f36148ad640 1 -- 172.21.15.46:0/282504593 >> 172.21.15.46:0/282504593 conn(0x7f361006a640 msgr2=0x7f36100c5620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:02.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.799+0000 7f36148ad640 1 -- 172.21.15.46:0/282504593 shutdown_connections 2023-10-30T14:25:02.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:02.799+0000 7f36148ad640 1 -- 172.21.15.46:0/282504593 wait complete. 2023-10-30T14:25:02.820 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:25:04.821 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:04.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.983+0000 7f76dbf65640 1 Processor -- start 2023-10-30T14:25:04.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76dbf65640 1 -- start start 2023-10-30T14:25:04.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76dbf65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76d41538a0 0x7f76d415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:04.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76dbf65640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f76d40cf160 0x7f76d40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:04.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76dbf65640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d40cb800 con 0x7f76d40ca0e0 2023-10-30T14:25:04.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76dbf65640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d415dc30 con 0x7f76d40cf160 2023-10-30T14:25:04.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76dbf65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40cb330 con 0x7f76d41538a0 2023-10-30T14:25:04.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76d9500640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f76d40ca0e0 0x7f76d415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38856/0 (socket says 172.21.15.46:38856) 2023-10-30T14:25:04.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76d9500640 1 -- 172.21.15.46:0/3578239608 learned_addr learned my addr 172.21.15.46:0/3578239608 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:25:04.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76da502640 1 --2- 172.21.15.46:0/3578239608 >> v2:172.21.15.154:3300/0 conn(0x7f76d40cf160 0x7f76d40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:04.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76d9d01640 1 --2- 172.21.15.46:0/3578239608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76d41538a0 0x7f76d415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:04.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76d8cff640 1 -- 172.21.15.46:0/3578239608 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200232009 0 0) 0x7f76d40cb800 con 0x7f76d40ca0e0 2023-10-30T14:25:04.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76d8cff640 1 -- 172.21.15.46:0/3578239608 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c0003530 con 0x7f76d40ca0e0 2023-10-30T14:25:04.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76da502640 1 -- 172.21.15.46:0/3578239608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76d41538a0 msgr2=0x7f76d415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:04.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76da502640 1 --2- 172.21.15.46:0/3578239608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76d41538a0 0x7f76d415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:04.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76da502640 1 -- 172.21.15.46:0/3578239608 >> v1:172.21.15.46:6789/0 conn(0x7f76d40ca0e0 legacy=0x7f76d415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:04.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76da502640 1 -- 172.21.15.46:0/3578239608 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d40cad70 con 0x7f76d40cf160 2023-10-30T14:25:04.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.987+0000 7f76d9d01640 1 --2- 172.21.15.46:0/3578239608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76d41538a0 0x7f76d415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:25:04.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.991+0000 7f76da502640 1 --2- 172.21.15.46:0/3578239608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d40cf160 0x7f76d40cf590 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009d90 tx=0x7f76cc01f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75919b08fe6e2c71 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:04.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.995+0000 7f76d8cff640 1 -- 172.21.15.46:0/3578239608 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0af600 con 0x7f76d40cf160 2023-10-30T14:25:04.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.995+0000 7f76d8cff640 1 -- 172.21.15.46:0/3578239608 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc0ab050 con 0x7f76d40cf160 2023-10-30T14:25:04.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.995+0000 7f76d8cff640 1 -- 172.21.15.46:0/3578239608 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0af980 con 0x7f76d40cf160 2023-10-30T14:25:04.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.995+0000 7f76dbf65640 1 -- 172.21.15.46:0/3578239608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d40cf160 msgr2=0x7f76d40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.995+0000 7f76dbf65640 1 --2- 172.21.15.46:0/3578239608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d40cf160 0x7f76d40cf590 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009d90 tx=0x7f76cc01f130 comp rx=0 tx=0).stop 2023-10-30T14:25:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.995+0000 7f76dbf65640 1 -- 172.21.15.46:0/3578239608 shutdown_connections 2023-10-30T14:25:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.995+0000 7f76dbf65640 1 --2- 172.21.15.46:0/3578239608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d40cf160 0x7f76d40cf590 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.995+0000 7f76dbf65640 1 --2- 172.21.15.46:0/3578239608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76d41538a0 0x7f76d415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.995+0000 7f76dbf65640 1 -- 172.21.15.46:0/3578239608 >> 172.21.15.46:0/3578239608 conn(0x7f76d406bfe0 msgr2=0x7f76d406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:04.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.995+0000 7f76dbf65640 1 -- 172.21.15.46:0/3578239608 shutdown_connections 2023-10-30T14:25:04.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.995+0000 7f76dbf65640 1 -- 172.21.15.46:0/3578239608 wait complete. 2023-10-30T14:25:04.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.995+0000 7f76dbf65640 1 Processor -- start 2023-10-30T14:25:05.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76dbf65640 1 -- start start 2023-10-30T14:25:05.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76dbf65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d40ca0e0 0x7f76d41f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:05.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76dbf65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76d40cf160 0x7f76d41527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:05.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76dbf65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76d41538a0 0x7f76d4157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:05.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76dbf65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d414c2b0 con 0x7f76d41538a0 2023-10-30T14:25:05.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76dbf65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d41f2b40 con 0x7f76d40ca0e0 2023-10-30T14:25:05.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76dbf65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d41f2e20 con 0x7f76d40cf160 2023-10-30T14:25:05.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76da502640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76d41538a0 0x7f76d4157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:05.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76da502640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76d41538a0 0x7f76d4157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56720/0 (socket says 172.21.15.46:56720) 2023-10-30T14:25:05.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76da502640 1 -- 172.21.15.46:0/1871420570 learned_addr learned my addr 172.21.15.46:0/1871420570 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:05.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76d9d01640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d40ca0e0 0x7f76d41f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:05.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76d9500640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76d40cf160 0x7f76d41527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:05.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76da502640 1 -- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76d40cf160 msgr2=0x7f76d41527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:05.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76da502640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76d40cf160 0x7f76d41527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:05.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76da502640 1 -- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d40ca0e0 msgr2=0x7f76d41f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:05.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76da502640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d40ca0e0 0x7f76d41f2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:05.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76da502640 1 -- 172.21.15.46:0/1871420570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d0009420 con 0x7f76d41538a0 2023-10-30T14:25:05.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76da502640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76d41538a0 0x7f76d4157fd0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0b2040 tx=0x7f76cc01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:05.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76ca7fc640 1 -- 172.21.15.46:0/1871420570 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc009450 con 0x7f76d41538a0 2023-10-30T14:25:05.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76ca7fc640 1 -- 172.21.15.46:0/1871420570 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc0b9410 con 0x7f76d41538a0 2023-10-30T14:25:05.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76ca7fc640 1 -- 172.21.15.46:0/1871420570 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0b97c0 con 0x7f76d41538a0 2023-10-30T14:25:05.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76dbf65640 1 -- 172.21.15.46:0/1871420570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc009a70 con 0x7f76d41538a0 2023-10-30T14:25:05.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:04.999+0000 7f76dbf65640 1 -- 172.21.15.46:0/1871420570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d41574f0 con 0x7f76d41538a0 2023-10-30T14:25:05.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.003+0000 7f76dbf65640 1 -- 172.21.15.46:0/1871420570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d40b2a50 con 0x7f76d41538a0 2023-10-30T14:25:05.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.003+0000 7f76ca7fc640 1 -- 172.21.15.46:0/1871420570 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f76cc0b6050 con 0x7f76d41538a0 2023-10-30T14:25:05.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.003+0000 7f76ca7fc640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76b80664a0 0x7f76b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:05.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.003+0000 7f76ca7fc640 1 -- 172.21.15.46:0/1871420570 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f76cc122de0 con 0x7f76d41538a0 2023-10-30T14:25:05.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.003+0000 7f76d9d01640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76b80664a0 0x7f76b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:05.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.007+0000 7f76d9d01640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76b80664a0 0x7f76b8068960 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009c50 tx=0x7f76d0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:05.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.007+0000 7f76ca7fc640 1 -- 172.21.15.46:0/1871420570 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f76cc0ef6c0 con 0x7f76d41538a0 2023-10-30T14:25:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.211+0000 7f76dbf65640 1 -- 172.21.15.46:0/1871420570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76d404e170 con 0x7f76d41538a0 2023-10-30T14:25:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.211+0000 7f76ca7fc640 1 -- 172.21.15.46:0/1871420570 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f76cc0b4070 con 0x7f76d41538a0 2023-10-30T14:25:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:05.213 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":260,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:59.863388+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:05.213 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 260 2023-10-30T14:25:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.211+0000 7f76dbf65640 1 -- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76b80664a0 msgr2=0x7f76b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.211+0000 7f76dbf65640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76b80664a0 0x7f76b8068960 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009c50 tx=0x7f76d0009190 comp rx=0 tx=0).stop 2023-10-30T14:25:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.211+0000 7f76dbf65640 1 -- 172.21.15.46:0/1871420570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76d41538a0 msgr2=0x7f76d4157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.211+0000 7f76dbf65640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76d41538a0 0x7f76d4157fd0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0b2040 tx=0x7f76cc01bed0 comp rx=0 tx=0).stop 2023-10-30T14:25:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.211+0000 7f76dbf65640 1 -- 172.21.15.46:0/1871420570 shutdown_connections 2023-10-30T14:25:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.211+0000 7f76dbf65640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f76b80664a0 0x7f76b8068960 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.211+0000 7f76dbf65640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76d41538a0 0x7f76d4157fd0 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:05.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.211+0000 7f76dbf65640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f76d40cf160 0x7f76d41527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:05.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.211+0000 7f76dbf65640 1 --2- 172.21.15.46:0/1871420570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d40ca0e0 0x7f76d41f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:05.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.211+0000 7f76dbf65640 1 -- 172.21.15.46:0/1871420570 >> 172.21.15.46:0/1871420570 conn(0x7f76d406bfe0 msgr2=0x7f76d4066410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:05.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.215+0000 7f76dbf65640 1 -- 172.21.15.46:0/1871420570 shutdown_connections 2023-10-30T14:25:05.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:05.215+0000 7f76dbf65640 1 -- 172.21.15.46:0/1871420570 wait complete. 2023-10-30T14:25:05.238 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:25:07.239 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:07.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd7150f640 1 Processor -- start 2023-10-30T14:25:07.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd7150f640 1 -- start start 2023-10-30T14:25:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd7150f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd6c1538a0 0x7fcd6c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd7150f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcd6c0ca0e0 0x7fcd6c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd7150f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd6c0cf160 0x7fcd6c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd7150f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd6c0cb800 con 0x7fcd6c0cf160 2023-10-30T14:25:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd7150f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd6c15dc30 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd7150f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c0cb330 con 0x7fcd6c1538a0 2023-10-30T14:25:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd6b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd6c0cf160 0x7fcd6c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd6b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd6c0cf160 0x7fcd6c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59432/0 (socket says 172.21.15.46:59432) 2023-10-30T14:25:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd6b7fe640 1 -- 172.21.15.46:0/2608016483 learned_addr learned my addr 172.21.15.46:0/2608016483 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd63fff640 1 --2- 172.21.15.46:0/2608016483 >> v2:172.21.15.154:3300/0 conn(0x7fcd6c0ca0e0 0x7fcd6c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.395+0000 7fcd6affd640 1 --2- 172.21.15.46:0/2608016483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd6c1538a0 0x7fcd6c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.399+0000 7fcd6b7fe640 1 -- 172.21.15.46:0/2608016483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd6c1538a0 msgr2=0x7fcd6c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.399+0000 7fcd6b7fe640 1 --2- 172.21.15.46:0/2608016483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd6c1538a0 0x7fcd6c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.399+0000 7fcd6b7fe640 1 -- 172.21.15.46:0/2608016483 >> v2:172.21.15.154:3300/0 conn(0x7fcd6c0ca0e0 msgr2=0x7fcd6c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.399+0000 7fcd6b7fe640 1 --2- 172.21.15.46:0/2608016483 >> v2:172.21.15.154:3300/0 conn(0x7fcd6c0ca0e0 0x7fcd6c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.399+0000 7fcd6b7fe640 1 -- 172.21.15.46:0/2608016483 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c0cad70 con 0x7fcd6c0cf160 2023-10-30T14:25:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.399+0000 7fcd6b7fe640 1 --2- 172.21.15.46:0/2608016483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd6c0cf160 0x7fcd6c0cf590 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0ccd10 tx=0x7fcd6401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ea5910bf51dd89 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.399+0000 7fcd6a7fc640 1 -- 172.21.15.46:0/2608016483 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd640af400 con 0x7fcd6c0cf160 2023-10-30T14:25:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.399+0000 7fcd6a7fc640 1 -- 172.21.15.46:0/2608016483 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd640ab050 con 0x7fcd6c0cf160 2023-10-30T14:25:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.399+0000 7fcd6a7fc640 1 -- 172.21.15.46:0/2608016483 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd640af780 con 0x7fcd6c0cf160 2023-10-30T14:25:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.403+0000 7fcd7150f640 1 -- 172.21.15.46:0/2608016483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd6c0cf160 msgr2=0x7fcd6c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.403+0000 7fcd7150f640 1 --2- 172.21.15.46:0/2608016483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd6c0cf160 0x7fcd6c0cf590 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0ccd10 tx=0x7fcd6401f1c0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.403+0000 7fcd7150f640 1 -- 172.21.15.46:0/2608016483 shutdown_connections 2023-10-30T14:25:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.403+0000 7fcd7150f640 1 --2- 172.21.15.46:0/2608016483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd6c0cf160 0x7fcd6c0cf590 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.403+0000 7fcd7150f640 1 --2- 172.21.15.46:0/2608016483 >> v2:172.21.15.154:3300/0 conn(0x7fcd6c0ca0e0 0x7fcd6c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.403+0000 7fcd7150f640 1 --2- 172.21.15.46:0/2608016483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd6c1538a0 0x7fcd6c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.403+0000 7fcd7150f640 1 -- 172.21.15.46:0/2608016483 >> 172.21.15.46:0/2608016483 conn(0x7fcd6c06bfe0 msgr2=0x7fcd6c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.403+0000 7fcd7150f640 1 -- 172.21.15.46:0/2608016483 shutdown_connections 2023-10-30T14:25:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.403+0000 7fcd7150f640 1 -- 172.21.15.46:0/2608016483 wait complete. 2023-10-30T14:25:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.403+0000 7fcd7150f640 1 Processor -- start 2023-10-30T14:25:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.403+0000 7fcd7150f640 1 -- start start 2023-10-30T14:25:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.407+0000 7fcd7150f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd6c0ca0e0 0x7fcd6c1efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.407+0000 7fcd7150f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd6c0cf160 0x7fcd6c1e5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.407+0000 7fcd7150f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd6c1538a0 0x7fcd6c1e2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.407+0000 7fcd7150f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c15afa0 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.407+0000 7fcd7150f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c15a9e0 con 0x7fcd6c1538a0 2023-10-30T14:25:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.407+0000 7fcd7150f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c15acc0 con 0x7fcd6c0cf160 2023-10-30T14:25:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd6affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd6c0ca0e0 0x7fcd6c1efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd6affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd6c0ca0e0 0x7fcd6c1efb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59446/0 (socket says 172.21.15.46:59446) 2023-10-30T14:25:07.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd6affd640 1 -- 172.21.15.46:0/3040754717 learned_addr learned my addr 172.21.15.46:0/3040754717 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:07.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd6b7fe640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd6c1538a0 0x7fcd6c1e2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:07.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd63fff640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd6c0cf160 0x7fcd6c1e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:07.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd6affd640 1 -- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd6c0cf160 msgr2=0x7fcd6c1e5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:07.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd6affd640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd6c0cf160 0x7fcd6c1e5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd6affd640 1 -- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd6c1538a0 msgr2=0x7fcd6c1e2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:07.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd6affd640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd6c1538a0 0x7fcd6c1e2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd6affd640 1 -- 172.21.15.46:0/3040754717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd64009b30 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd6b7fe640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd6c1538a0 0x7fcd6c1e2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:25:07.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd6affd640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd6c0ca0e0 0x7fcd6c1efb60 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c00d5f0 tx=0x7fcd5c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd637fe640 1 -- 172.21.15.46:0/3040754717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c0125f0 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd637fe640 1 -- 172.21.15.46:0/3040754717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5c012750 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd637fe640 1 -- 172.21.15.46:0/3040754717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c012ae0 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd7150f640 1 -- 172.21.15.46:0/3040754717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd6c1ef080 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd7150f640 1 -- 172.21.15.46:0/3040754717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd6c1ef640 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd7150f640 1 -- 172.21.15.46:0/3040754717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6c1ef800 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.411+0000 7fcd637fe640 1 -- 172.21.15.46:0/3040754717 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd5c012c40 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.415+0000 7fcd637fe640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd4c0664a0 0x7fcd4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.415+0000 7fcd637fe640 1 -- 172.21.15.46:0/3040754717 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fcd5c084f40 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.415+0000 7fcd63fff640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd4c0664a0 0x7fcd4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:07.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.415+0000 7fcd63fff640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd4c0664a0 0x7fcd4c068960 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c146860 tx=0x7fcd54008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:07.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.423+0000 7fcd637fe640 1 -- 172.21.15.46:0/3040754717 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd5c051800 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.615+0000 7fcd7150f640 1 -- 172.21.15.46:0/3040754717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd6c0b2a50 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd637fe640 1 -- 172.21.15.46:0/3040754717 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) v1 ==== 95+0+5736 (secure 0 0 0) 0x7fcd5c054e20 con 0x7fcd6c0ca0e0 2023-10-30T14:25:07.621 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:07.621 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":260,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:59.863388+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:07.622 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 260 2023-10-30T14:25:07.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd7150f640 1 -- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd4c0664a0 msgr2=0x7fcd4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:07.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd7150f640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd4c0664a0 0x7fcd4c068960 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c146860 tx=0x7fcd54008040 comp rx=0 tx=0).stop 2023-10-30T14:25:07.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd7150f640 1 -- 172.21.15.46:0/3040754717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd6c0ca0e0 msgr2=0x7fcd6c1efb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:07.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd7150f640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd6c0ca0e0 0x7fcd6c1efb60 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c00d5f0 tx=0x7fcd5c00dab0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd7150f640 1 -- 172.21.15.46:0/3040754717 shutdown_connections 2023-10-30T14:25:07.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd7150f640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fcd4c0664a0 0x7fcd4c068960 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd7150f640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd6c1538a0 0x7fcd6c1e2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd7150f640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd6c0cf160 0x7fcd6c1e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd7150f640 1 --2- 172.21.15.46:0/3040754717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd6c0ca0e0 0x7fcd6c1efb60 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:07.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd7150f640 1 -- 172.21.15.46:0/3040754717 >> 172.21.15.46:0/3040754717 conn(0x7fcd6c06bfe0 msgr2=0x7fcd6c1526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:07.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd7150f640 1 -- 172.21.15.46:0/3040754717 shutdown_connections 2023-10-30T14:25:07.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:07.619+0000 7fcd7150f640 1 -- 172.21.15.46:0/3040754717 wait complete. 2023-10-30T14:25:07.643 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:25:09.644 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:09.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.791+0000 7f2c1af0e640 1 Processor -- start 2023-10-30T14:25:09.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.791+0000 7f2c1af0e640 1 -- start start 2023-10-30T14:25:09.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.791+0000 7f2c1af0e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 0x7f2c1415a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:09.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.791+0000 7f2c1af0e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2c1406a270 0x7f2c140c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:09.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.791+0000 7f2c1af0e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c140c6c10 con 0x7f2c1406cb80 2023-10-30T14:25:09.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.791+0000 7f2c1af0e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c140c8570 con 0x7f2c1406a270 2023-10-30T14:25:09.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.791+0000 7f2c1af0e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c140c6ef0 con 0x7f2c14069e90 2023-10-30T14:25:09.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.795+0000 7f2c13fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2c1406a270 0x7f2c140c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:09.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.795+0000 7f2c18caa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 0x7f2c1415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:09.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.795+0000 7f2c18caa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 0x7f2c1415a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:54018/0 (socket says 172.21.15.46:54018) 2023-10-30T14:25:09.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.795+0000 7f2c18caa640 1 -- 172.21.15.46:0/3189385335 learned_addr learned my addr 172.21.15.46:0/3189385335 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:09.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.795+0000 7f2c194ab640 1 --1- 172.21.15.46:0/3189385335 >> v1:172.21.15.154:6789/0 conn(0x7f2c1406cb80 0x7f2c1406d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:36892/0 (socket says 172.21.15.46:36892) 2023-10-30T14:25:09.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.795+0000 7f2c13fff640 1 -- 172.21.15.46:0/3189385335 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 msgr2=0x7f2c1415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:09.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.795+0000 7f2c13fff640 1 --2- 172.21.15.46:0/3189385335 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 0x7f2c1415a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:09.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.795+0000 7f2c13fff640 1 -- 172.21.15.46:0/3189385335 >> v1:172.21.15.154:6789/0 conn(0x7f2c1406cb80 legacy=0x7f2c1406d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:09.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.795+0000 7f2c13fff640 1 -- 172.21.15.46:0/3189385335 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c140c71d0 con 0x7f2c1406a270 2023-10-30T14:25:09.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.795+0000 7f2c18caa640 1 --2- 172.21.15.46:0/3189385335 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 0x7f2c1415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:25:09.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.799+0000 7f2c13fff640 1 --2- 172.21.15.46:0/3189385335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1406a270 0x7f2c140c9cb0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2c000097c0 tx=0x7f2c0001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4a42bd2b9e46a7a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:09.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.799+0000 7f2c137fe640 1 -- 172.21.15.46:0/3189385335 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c000ab600 con 0x7f2c1406a270 2023-10-30T14:25:09.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.799+0000 7f2c137fe640 1 -- 172.21.15.46:0/3189385335 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c000a7050 con 0x7f2c1406a270 2023-10-30T14:25:09.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.799+0000 7f2c137fe640 1 -- 172.21.15.46:0/3189385335 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c000ab980 con 0x7f2c1406a270 2023-10-30T14:25:09.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.799+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3189385335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1406a270 msgr2=0x7f2c140c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:09.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.799+0000 7f2c1af0e640 1 --2- 172.21.15.46:0/3189385335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1406a270 0x7f2c140c9cb0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2c000097c0 tx=0x7f2c0001c480 comp rx=0 tx=0).stop 2023-10-30T14:25:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.799+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3189385335 shutdown_connections 2023-10-30T14:25:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.799+0000 7f2c1af0e640 1 --2- 172.21.15.46:0/3189385335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1406a270 0x7f2c140c9cb0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.799+0000 7f2c1af0e640 1 --2- 172.21.15.46:0/3189385335 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 0x7f2c1415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.799+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3189385335 >> 172.21.15.46:0/3189385335 conn(0x7f2c1406dfd0 msgr2=0x7f2c1406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:09.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.799+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3189385335 shutdown_connections 2023-10-30T14:25:09.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.799+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3189385335 wait complete. 2023-10-30T14:25:09.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c1af0e640 1 Processor -- start 2023-10-30T14:25:09.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c1af0e640 1 -- start start 2023-10-30T14:25:09.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c1af0e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 0x7f2c1406b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:09.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c1af0e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1406a270 0x7f2c1414b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c1af0e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1406cb80 0x7f2c1414bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c1af0e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c141f4a30 con 0x7f2c1406a270 2023-10-30T14:25:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c1af0e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1414d010 con 0x7f2c1406cb80 2023-10-30T14:25:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c1af0e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c141f4c40 con 0x7f2c14069e90 2023-10-30T14:25:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c13fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1406a270 0x7f2c1414b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c13fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1406a270 0x7f2c1414b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59458/0 (socket says 172.21.15.46:59458) 2023-10-30T14:25:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c13fff640 1 -- 172.21.15.46:0/3461628556 learned_addr learned my addr 172.21.15.46:0/3461628556 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c18caa640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 0x7f2c1406b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:09.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c194ab640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1406cb80 0x7f2c1414bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c18caa640 1 -- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1406cb80 msgr2=0x7f2c1414bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c18caa640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1406cb80 0x7f2c1414bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c18caa640 1 -- 172.21.15.46:0/3461628556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1406a270 msgr2=0x7f2c1414b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c18caa640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1406a270 0x7f2c1414b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.803+0000 7f2c18caa640 1 -- 172.21.15.46:0/3461628556 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c080094d0 con 0x7f2c14069e90 2023-10-30T14:25:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.807+0000 7f2c18caa640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 0x7f2c1406b200 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2c0800e910 tx=0x7f2c0800edd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.807+0000 7f2c117fa640 1 -- 172.21.15.46:0/3461628556 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0800ca00 con 0x7f2c14069e90 2023-10-30T14:25:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.807+0000 7f2c117fa640 1 -- 172.21.15.46:0/3461628556 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0800cb60 con 0x7f2c14069e90 2023-10-30T14:25:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.807+0000 7f2c117fa640 1 -- 172.21.15.46:0/3461628556 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c08004030 con 0x7f2c14069e90 2023-10-30T14:25:09.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.807+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3461628556 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c00009420 con 0x7f2c14069e90 2023-10-30T14:25:09.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.807+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3461628556 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c141f5200 con 0x7f2c14069e90 2023-10-30T14:25:09.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.807+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3461628556 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c141475b0 con 0x7f2c14069e90 2023-10-30T14:25:09.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.811+0000 7f2c117fa640 1 -- 172.21.15.46:0/3461628556 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c08004190 con 0x7f2c14069e90 2023-10-30T14:25:09.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.811+0000 7f2c117fa640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2bf80664a0 0x7f2bf8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:09.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.811+0000 7f2c117fa640 1 -- 172.21.15.46:0/3461628556 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f2c08085100 con 0x7f2c14069e90 2023-10-30T14:25:09.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.811+0000 7f2c13fff640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2bf80664a0 0x7f2bf8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:09.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.811+0000 7f2c13fff640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2bf80664a0 0x7f2bf8068960 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f2c0001ab30 tx=0x7f2c00008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:09.819+0000 7f2c117fa640 1 -- 172.21.15.46:0/3461628556 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c08051a10 con 0x7f2c14069e90 2023-10-30T14:25:10.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.067+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3461628556 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c141475b0 con 0x7f2c14069e90 2023-10-30T14:25:10.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.067+0000 7f2c117fa640 1 -- 172.21.15.46:0/3461628556 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f2c141475b0 con 0x7f2c14069e90 2023-10-30T14:25:10.070 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:10.070 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":260,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:24:59.863388+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:10.071 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 260 2023-10-30T14:25:10.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.071+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2bf80664a0 msgr2=0x7f2bf8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:10.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.071+0000 7f2c1af0e640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2bf80664a0 0x7f2bf8068960 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f2c0001ab30 tx=0x7f2c00008040 comp rx=0 tx=0).stop 2023-10-30T14:25:10.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.071+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 msgr2=0x7f2c1406b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:10.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.071+0000 7f2c1af0e640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 0x7f2c1406b200 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2c0800e910 tx=0x7f2c0800edd0 comp rx=0 tx=0).stop 2023-10-30T14:25:10.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.071+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3461628556 shutdown_connections 2023-10-30T14:25:10.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.071+0000 7f2c1af0e640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2bf80664a0 0x7f2bf8068960 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:10.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.071+0000 7f2c1af0e640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1406cb80 0x7f2c1414bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:10.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.071+0000 7f2c1af0e640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1406a270 0x7f2c1414b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:10.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.071+0000 7f2c1af0e640 1 --2- 172.21.15.46:0/3461628556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c14069e90 0x7f2c1406b200 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:10.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.071+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3461628556 >> 172.21.15.46:0/3461628556 conn(0x7f2c1406dfd0 msgr2=0x7f2c14071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:10.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.071+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3461628556 shutdown_connections 2023-10-30T14:25:10.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:10.071+0000 7f2c1af0e640 1 -- 172.21.15.46:0/3461628556 wait complete. 2023-10-30T14:25:10.095 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:25:12.096 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.235+0000 7fbebf9d3640 1 Processor -- start 2023-10-30T14:25:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebf9d3640 1 -- start start 2023-10-30T14:25:12.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebf9d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbeb8069e90 0x7fbeb815a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:12.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebf9d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbeb806a270 0x7fbeb80c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:12.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebf9d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb80c6c10 con 0x7fbeb806cb80 2023-10-30T14:25:12.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebf9d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb80c8570 con 0x7fbeb8069e90 2023-10-30T14:25:12.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebf9d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb80c6ef0 con 0x7fbeb806a270 2023-10-30T14:25:12.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebd76f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbeb8069e90 0x7fbeb815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:12.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebd76f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbeb8069e90 0x7fbeb815a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59466/0 (socket says 172.21.15.46:59466) 2023-10-30T14:25:12.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebd76f640 1 -- 172.21.15.46:0/3156330017 learned_addr learned my addr 172.21.15.46:0/3156330017 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:12.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebcf6e640 1 --2- 172.21.15.46:0/3156330017 >> v2:172.21.15.154:3300/0 conn(0x7fbeb806a270 0x7fbeb80c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:12.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebd76f640 1 -- 172.21.15.46:0/3156330017 >> v2:172.21.15.154:3300/0 conn(0x7fbeb806a270 msgr2=0x7fbeb80c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:12.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebd76f640 1 --2- 172.21.15.46:0/3156330017 >> v2:172.21.15.154:3300/0 conn(0x7fbeb806a270 0x7fbeb80c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:12.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebd76f640 1 -- 172.21.15.46:0/3156330017 >> v1:172.21.15.46:6789/0 conn(0x7fbeb806cb80 legacy=0x7fbeb806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:12.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.239+0000 7fbebd76f640 1 -- 172.21.15.46:0/3156330017 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb80c71d0 con 0x7fbeb8069e90 2023-10-30T14:25:12.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbebd76f640 1 --2- 172.21.15.46:0/3156330017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8069e90 0x7fbeb815a410 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fbeb806feb0 tx=0x7fbeb401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a68c1467d66504 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:12.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbeaffff640 1 -- 172.21.15.46:0/3156330017 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40ab5f0 con 0x7fbeb8069e90 2023-10-30T14:25:12.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbeaffff640 1 -- 172.21.15.46:0/3156330017 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb40ab750 con 0x7fbeb8069e90 2023-10-30T14:25:12.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbeaffff640 1 -- 172.21.15.46:0/3156330017 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40abae0 con 0x7fbeb8069e90 2023-10-30T14:25:12.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbebf9d3640 1 -- 172.21.15.46:0/3156330017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8069e90 msgr2=0x7fbeb815a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:12.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbebf9d3640 1 --2- 172.21.15.46:0/3156330017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8069e90 0x7fbeb815a410 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fbeb806feb0 tx=0x7fbeb401c500 comp rx=0 tx=0).stop 2023-10-30T14:25:12.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbebf9d3640 1 -- 172.21.15.46:0/3156330017 shutdown_connections 2023-10-30T14:25:12.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbebf9d3640 1 --2- 172.21.15.46:0/3156330017 >> v2:172.21.15.154:3300/0 conn(0x7fbeb806a270 0x7fbeb80c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:12.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbebf9d3640 1 --2- 172.21.15.46:0/3156330017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8069e90 0x7fbeb815a410 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:12.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbebf9d3640 1 -- 172.21.15.46:0/3156330017 >> 172.21.15.46:0/3156330017 conn(0x7fbeb806dfd0 msgr2=0x7fbeb806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:12.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbebf9d3640 1 -- 172.21.15.46:0/3156330017 shutdown_connections 2023-10-30T14:25:12.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbebf9d3640 1 -- 172.21.15.46:0/3156330017 wait complete. 2023-10-30T14:25:12.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbebf9d3640 1 Processor -- start 2023-10-30T14:25:12.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.243+0000 7fbebf9d3640 1 -- start start 2023-10-30T14:25:12.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebf9d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8069e90 0x7fbeb814d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:12.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebf9d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb806a270 0x7fbeb81f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:12.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebf9d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbeb806cb80 0x7fbeb8148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:12.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebf9d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb81554a0 con 0x7fbeb8069e90 2023-10-30T14:25:12.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebf9d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb81f1a90 con 0x7fbeb806a270 2023-10-30T14:25:12.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebf9d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb81546e0 con 0x7fbeb806cb80 2023-10-30T14:25:12.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebd76f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8069e90 0x7fbeb814d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:12.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebd76f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8069e90 0x7fbeb814d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59468/0 (socket says 172.21.15.46:59468) 2023-10-30T14:25:12.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebd76f640 1 -- 172.21.15.46:0/1869739278 learned_addr learned my addr 172.21.15.46:0/1869739278 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:12.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebcf6e640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb806a270 0x7fbeb81f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:12.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebdf70640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbeb806cb80 0x7fbeb8148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:12.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebd76f640 1 -- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbeb806cb80 msgr2=0x7fbeb8148af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:12.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebd76f640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbeb806cb80 0x7fbeb8148af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:12.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebd76f640 1 -- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb806a270 msgr2=0x7fbeb81f2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:12.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebd76f640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb806a270 0x7fbeb81f2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:12.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebd76f640 1 -- 172.21.15.46:0/1869739278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb40094d0 con 0x7fbeb8069e90 2023-10-30T14:25:12.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebdf70640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbeb806cb80 0x7fbeb8148af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:25:12.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebcf6e640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb806a270 0x7fbeb81f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:25:12.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebd76f640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8069e90 0x7fbeb814d840 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4019110 tx=0x7fbeb401ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:12.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbeadffb640 1 -- 172.21.15.46:0/1869739278 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40b4800 con 0x7fbeb8069e90 2023-10-30T14:25:12.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbeadffb640 1 -- 172.21.15.46:0/1869739278 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb40b4960 con 0x7fbeb8069e90 2023-10-30T14:25:12.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbeadffb640 1 -- 172.21.15.46:0/1869739278 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40b4cf0 con 0x7fbeb8069e90 2023-10-30T14:25:12.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebf9d3640 1 -- 172.21.15.46:0/1869739278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb81556f0 con 0x7fbeb8069e90 2023-10-30T14:25:12.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebf9d3640 1 -- 172.21.15.46:0/1869739278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb81f2030 con 0x7fbeb8069e90 2023-10-30T14:25:12.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.247+0000 7fbebf9d3640 1 -- 172.21.15.46:0/1869739278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb804e170 con 0x7fbeb8069e90 2023-10-30T14:25:12.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.251+0000 7fbeadffb640 1 -- 172.21.15.46:0/1869739278 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbeb40b3380 con 0x7fbeb8069e90 2023-10-30T14:25:12.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.251+0000 7fbeadffb640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe9c066470 0x7fbe9c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:12.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.251+0000 7fbeadffb640 1 -- 172.21.15.46:0/1869739278 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fbeb40a8050 con 0x7fbeb8069e90 2023-10-30T14:25:12.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.251+0000 7fbebcf6e640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe9c066470 0x7fbe9c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:12.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.251+0000 7fbebcf6e640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe9c066470 0x7fbe9c068930 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8150990 tx=0x7fbea8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:12.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.255+0000 7fbeadffb640 1 -- 172.21.15.46:0/1869739278 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbeb40eb990 con 0x7fbeb8069e90 2023-10-30T14:25:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.443+0000 7fbebf9d3640 1 -- 172.21.15.46:0/1869739278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbeb806ef60 con 0x7fbeb8069e90 2023-10-30T14:25:12.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.443+0000 7fbeadffb640 1 -- 172.21.15.46:0/1869739278 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 261 v261) v1 ==== 95+0+5738 (secure 0 0 0) 0x7fbeb40c4070 con 0x7fbeb8069e90 2023-10-30T14:25:12.447 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:12.447 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":261,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":261,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:10.631600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:resolve","state_seq":12,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:12.447 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 261 2023-10-30T14:25:12.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.447+0000 7fbebf9d3640 1 -- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe9c066470 msgr2=0x7fbe9c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:12.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.447+0000 7fbebf9d3640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe9c066470 0x7fbe9c068930 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8150990 tx=0x7fbea8008040 comp rx=0 tx=0).stop 2023-10-30T14:25:12.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.447+0000 7fbebf9d3640 1 -- 172.21.15.46:0/1869739278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8069e90 msgr2=0x7fbeb814d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:12.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.447+0000 7fbebf9d3640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8069e90 0x7fbeb814d840 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4019110 tx=0x7fbeb401ad40 comp rx=0 tx=0).stop 2023-10-30T14:25:12.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.447+0000 7fbebf9d3640 1 -- 172.21.15.46:0/1869739278 shutdown_connections 2023-10-30T14:25:12.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.447+0000 7fbebf9d3640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbe9c066470 0x7fbe9c068930 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:12.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.447+0000 7fbebf9d3640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbeb806cb80 0x7fbeb8148af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:12.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.447+0000 7fbebf9d3640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb806a270 0x7fbeb81f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:12.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.447+0000 7fbebf9d3640 1 --2- 172.21.15.46:0/1869739278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8069e90 0x7fbeb814d840 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:12.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.447+0000 7fbebf9d3640 1 -- 172.21.15.46:0/1869739278 >> 172.21.15.46:0/1869739278 conn(0x7fbeb806dfd0 msgr2=0x7fbeb80ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:12.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.447+0000 7fbebf9d3640 1 -- 172.21.15.46:0/1869739278 shutdown_connections 2023-10-30T14:25:12.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:12.447+0000 7fbebf9d3640 1 -- 172.21.15.46:0/1869739278 wait complete. 2023-10-30T14:25:12.468 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 261, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 261, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_10020": { "addr": "172.21.15.46:6837/2187461695", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2187461695, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2187461695, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 10020, "incarnation": 260, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:resolve", "state_seq": 12 }, "gid_9882": { "addr": "172.21.15.46:6833/3394097058", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 3394097058, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 3394097058, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9882, "incarnation": 253, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 102, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:25:10.631600+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 10020, "mds_1": 9882 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:25:14.469 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.611+0000 7fd0da609640 1 Processor -- start 2023-10-30T14:25:14.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.611+0000 7fd0da609640 1 -- start start 2023-10-30T14:25:14.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0da609640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0d4151800 0x7fd0d406e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:14.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0da609640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0d40c8260 0x7fd0d40cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:14.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0da609640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d406e090 con 0x7fd0d40c1ee0 2023-10-30T14:25:14.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0da609640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d406dc70 con 0x7fd0d4151800 2023-10-30T14:25:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0da609640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d40c8a50 con 0x7fd0d40c8260 2023-10-30T14:25:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0d3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0d4151800 0x7fd0d406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0d3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0d4151800 0x7fd0d406e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59482/0 (socket says 172.21.15.46:59482) 2023-10-30T14:25:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0d3fff640 1 -- 172.21.15.46:0/214234800 learned_addr learned my addr 172.21.15.46:0/214234800 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:14.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0d8ba6640 1 --2- 172.21.15.46:0/214234800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0d40c8260 0x7fd0d40cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:14.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0d3fff640 1 -- 172.21.15.46:0/214234800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0d40c8260 msgr2=0x7fd0d40cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:14.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0d3fff640 1 --2- 172.21.15.46:0/214234800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0d40c8260 0x7fd0d40cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:14.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0d3fff640 1 -- 172.21.15.46:0/214234800 >> v1:172.21.15.154:6789/0 conn(0x7fd0d40c1ee0 legacy=0x7fd0d40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:14.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.615+0000 7fd0d3fff640 1 -- 172.21.15.46:0/214234800 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40c3b10 con 0x7fd0d4151800 2023-10-30T14:25:14.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0d3fff640 1 --2- 172.21.15.46:0/214234800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d4151800 0x7fd0d406e470 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4009870 tx=0x7fd0c401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c291632323c4e92f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0d2ffd640 1 -- 172.21.15.46:0/214234800 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c40ab620 con 0x7fd0d4151800 2023-10-30T14:25:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0d2ffd640 1 -- 172.21.15.46:0/214234800 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c40ab780 con 0x7fd0d4151800 2023-10-30T14:25:14.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0d2ffd640 1 -- 172.21.15.46:0/214234800 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c40abb10 con 0x7fd0d4151800 2023-10-30T14:25:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 -- 172.21.15.46:0/214234800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d4151800 msgr2=0x7fd0d406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:14.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 --2- 172.21.15.46:0/214234800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d4151800 0x7fd0d406e470 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4009870 tx=0x7fd0c401c510 comp rx=0 tx=0).stop 2023-10-30T14:25:14.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 -- 172.21.15.46:0/214234800 shutdown_connections 2023-10-30T14:25:14.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 --2- 172.21.15.46:0/214234800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0d40c8260 0x7fd0d40cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:14.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 --2- 172.21.15.46:0/214234800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d4151800 0x7fd0d406e470 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:14.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 -- 172.21.15.46:0/214234800 >> 172.21.15.46:0/214234800 conn(0x7fd0d4071ea0 msgr2=0x7fd0d4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:14.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 -- 172.21.15.46:0/214234800 shutdown_connections 2023-10-30T14:25:14.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 -- 172.21.15.46:0/214234800 wait complete. 2023-10-30T14:25:14.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 Processor -- start 2023-10-30T14:25:14.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 -- start start 2023-10-30T14:25:14.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0d40c1ee0 0x7fd0d41fab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:14.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d40c8260 0x7fd0d40c4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:14.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0d4151800 0x7fd0d4161480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:14.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d41486e0 con 0x7fd0d40c8260 2023-10-30T14:25:14.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d41fa0e0 con 0x7fd0d4151800 2023-10-30T14:25:14.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.619+0000 7fd0da609640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d41fa6a0 con 0x7fd0d40c1ee0 2023-10-30T14:25:14.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d40c8260 0x7fd0d40c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:14.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d40c8260 0x7fd0d40c4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59486/0 (socket says 172.21.15.46:59486) 2023-10-30T14:25:14.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d37fe640 1 -- 172.21.15.46:0/1172003326 learned_addr learned my addr 172.21.15.46:0/1172003326 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:14.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d3fff640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0d40c1ee0 0x7fd0d41fab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:14.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d8ba6640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0d4151800 0x7fd0d4161480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:14.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d37fe640 1 -- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0d40c1ee0 msgr2=0x7fd0d41fab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:14.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d37fe640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0d40c1ee0 0x7fd0d41fab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:14.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d37fe640 1 -- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0d4151800 msgr2=0x7fd0d4161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:14.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d37fe640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0d4151800 0x7fd0d4161480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:14.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d37fe640 1 -- 172.21.15.46:0/1172003326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c40094d0 con 0x7fd0d40c8260 2023-10-30T14:25:14.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d8ba6640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0d4151800 0x7fd0d4161480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:25:14.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d3fff640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0d40c1ee0 0x7fd0d41fab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:25:14.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d37fe640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d40c8260 0x7fd0d40c4230 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800c3c0 tx=0x7fd0c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:14.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d0ff9640 1 -- 172.21.15.46:0/1172003326 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80052d0 con 0x7fd0d40c8260 2023-10-30T14:25:14.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d0ff9640 1 -- 172.21.15.46:0/1172003326 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c8005430 con 0x7fd0d40c8260 2023-10-30T14:25:14.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d0ff9640 1 -- 172.21.15.46:0/1172003326 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8003820 con 0x7fd0d40c8260 2023-10-30T14:25:14.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0da609640 1 -- 172.21.15.46:0/1172003326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d41fb070 con 0x7fd0d40c8260 2023-10-30T14:25:14.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0da609640 1 -- 172.21.15.46:0/1172003326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d40c52e0 con 0x7fd0d40c8260 2023-10-30T14:25:14.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0da609640 1 -- 172.21.15.46:0/1172003326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d404e170 con 0x7fd0d40c8260 2023-10-30T14:25:14.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.623+0000 7fd0d0ff9640 1 -- 172.21.15.46:0/1172003326 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0c8003980 con 0x7fd0d40c8260 2023-10-30T14:25:14.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.627+0000 7fd0d0ff9640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0c00664a0 0x7fd0c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:14.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.627+0000 7fd0d0ff9640 1 -- 172.21.15.46:0/1172003326 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fd0c8084060 con 0x7fd0d40c8260 2023-10-30T14:25:14.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.627+0000 7fd0d3fff640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0c00664a0 0x7fd0c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:14.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.627+0000 7fd0d3fff640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0c00664a0 0x7fd0c0068960 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40af000 tx=0x7fd0c40a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:14.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.631+0000 7fd0d0ff9640 1 -- 172.21.15.46:0/1172003326 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0c80b5090 con 0x7fd0d40c8260 2023-10-30T14:25:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 -- 172.21.15.46:0/1172003326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0d414ee10 con 0x7fd0d40c8260 2023-10-30T14:25:14.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0d0ff9640 1 -- 172.21.15.46:0/1172003326 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 261 v261) v1 ==== 95+0+5738 (secure 0 0 0) 0x7fd0c8017070 con 0x7fd0d40c8260 2023-10-30T14:25:14.821 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:14.821 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":261,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":261,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:10.631600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:resolve","state_seq":12,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:14.822 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 261 2023-10-30T14:25:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 -- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0c00664a0 msgr2=0x7fd0c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0c00664a0 0x7fd0c0068960 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40af000 tx=0x7fd0c40a7000 comp rx=0 tx=0).stop 2023-10-30T14:25:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 -- 172.21.15.46:0/1172003326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d40c8260 msgr2=0x7fd0d40c4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d40c8260 0x7fd0d40c4230 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800c3c0 tx=0x7fd0c800c880 comp rx=0 tx=0).stop 2023-10-30T14:25:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 -- 172.21.15.46:0/1172003326 shutdown_connections 2023-10-30T14:25:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd0c00664a0 0x7fd0c0068960 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0d4151800 0x7fd0d4161480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d40c8260 0x7fd0d40c4230 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 --2- 172.21.15.46:0/1172003326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0d40c1ee0 0x7fd0d41fab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 -- 172.21.15.46:0/1172003326 >> 172.21.15.46:0/1172003326 conn(0x7fd0d4071ea0 msgr2=0x7fd0d40cc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 -- 172.21.15.46:0/1172003326 shutdown_connections 2023-10-30T14:25:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:14.819+0000 7fd0da609640 1 -- 172.21.15.46:0/1172003326 wait complete. 2023-10-30T14:25:14.842 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:25:16.843 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:16.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec55b3e640 1 Processor -- start 2023-10-30T14:25:16.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec55b3e640 1 -- start start 2023-10-30T14:25:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec55b3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fec50071110 0x7fec5006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:16.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec55b3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec50066cf0 con 0x7fec500c6bd0 2023-10-30T14:25:16.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec55b3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec500c70d0 con 0x7fec5006f7a0 2023-10-30T14:25:16.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec55b3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec500700c0 con 0x7fec50071110 2023-10-30T14:25:16.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec4ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fec50071110 0x7fec5006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:16.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec4ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fec50071110 0x7fec5006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56272/0 (socket says 172.21.15.46:56272) 2023-10-30T14:25:16.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec4ffff640 1 -- 172.21.15.46:0/612119599 learned_addr learned my addr 172.21.15.46:0/612119599 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:16.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec4effd640 1 --1- 172.21.15.46:0/612119599 >> v1:172.21.15.46:6789/0 conn(0x7fec500c6bd0 0x7fec50158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37098/0 (socket says 172.21.15.46:37098) 2023-10-30T14:25:16.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec4ffff640 1 -- 172.21.15.46:0/612119599 >> v1:172.21.15.154:6789/0 conn(0x7fec5006f7a0 legacy=0x7fec50069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec4ffff640 1 -- 172.21.15.46:0/612119599 >> v1:172.21.15.46:6789/0 conn(0x7fec500c6bd0 legacy=0x7fec50158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec4ffff640 1 -- 172.21.15.46:0/612119599 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec500c7390 con 0x7fec50071110 2023-10-30T14:25:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.986+0000 7fec4e7fc640 1 -- 172.21.15.46:0/612119599 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364905117 0 0) 0x7fec50066cf0 con 0x7fec500c6bd0 2023-10-30T14:25:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.990+0000 7fec4ffff640 1 --2- 172.21.15.46:0/612119599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec50071110 0x7fec5006fb80 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fec48009e50 tx=0x7fec4801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11559156cc1bdb95 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.990+0000 7fec4e7fc640 1 -- 172.21.15.46:0/612119599 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec480af400 con 0x7fec50071110 2023-10-30T14:25:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.990+0000 7fec4e7fc640 1 -- 172.21.15.46:0/612119599 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec480ab050 con 0x7fec50071110 2023-10-30T14:25:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.990+0000 7fec4e7fc640 1 -- 172.21.15.46:0/612119599 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec480af780 con 0x7fec50071110 2023-10-30T14:25:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.990+0000 7fec55b3e640 1 -- 172.21.15.46:0/612119599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec50071110 msgr2=0x7fec5006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.990+0000 7fec55b3e640 1 --2- 172.21.15.46:0/612119599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec50071110 0x7fec5006fb80 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fec48009e50 tx=0x7fec4801f1f0 comp rx=0 tx=0).stop 2023-10-30T14:25:16.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.990+0000 7fec55b3e640 1 -- 172.21.15.46:0/612119599 shutdown_connections 2023-10-30T14:25:16.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.990+0000 7fec55b3e640 1 --2- 172.21.15.46:0/612119599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec50071110 0x7fec5006fb80 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:16.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.990+0000 7fec55b3e640 1 -- 172.21.15.46:0/612119599 >> 172.21.15.46:0/612119599 conn(0x7fec500c7910 msgr2=0x7fec500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:16.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.990+0000 7fec55b3e640 1 -- 172.21.15.46:0/612119599 shutdown_connections 2023-10-30T14:25:16.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.990+0000 7fec55b3e640 1 -- 172.21.15.46:0/612119599 wait complete. 2023-10-30T14:25:16.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.990+0000 7fec55b3e640 1 Processor -- start 2023-10-30T14:25:16.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec55b3e640 1 -- start start 2023-10-30T14:25:16.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec55b3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec5006f7a0 0x7fec501582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:16.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec55b3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec50071110 0x7fec5014d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:16.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec55b3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec500c6bd0 0x7fec50156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:16.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec55b3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec50157210 con 0x7fec500c6bd0 2023-10-30T14:25:16.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec55b3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec50157550 con 0x7fec5006f7a0 2023-10-30T14:25:16.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec55b3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5014d6a0 con 0x7fec50071110 2023-10-30T14:25:16.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec4ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec500c6bd0 0x7fec50156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:16.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec4ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec500c6bd0 0x7fec50156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56284/0 (socket says 172.21.15.46:56284) 2023-10-30T14:25:16.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec4ffff640 1 -- 172.21.15.46:0/1134883684 learned_addr learned my addr 172.21.15.46:0/1134883684 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:16.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec4effd640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec50071110 0x7fec5014d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:17.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec4f7fe640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec5006f7a0 0x7fec501582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:17.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec4ffff640 1 -- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec50071110 msgr2=0x7fec5014d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:17.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec4ffff640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec50071110 0x7fec5014d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:17.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec4ffff640 1 -- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec5006f7a0 msgr2=0x7fec501582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:17.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec4ffff640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec5006f7a0 0x7fec501582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:17.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec4ffff640 1 -- 172.21.15.46:0/1134883684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec48009b30 con 0x7fec500c6bd0 2023-10-30T14:25:17.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec4f7fe640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec5006f7a0 0x7fec501582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:25:17.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec4ffff640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec500c6bd0 0x7fec50156cd0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fec480b2040 tx=0x7fec4801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:17.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec37fff640 1 -- 172.21.15.46:0/1134883684 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec480afde0 con 0x7fec500c6bd0 2023-10-30T14:25:17.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec37fff640 1 -- 172.21.15.46:0/1134883684 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec48009510 con 0x7fec500c6bd0 2023-10-30T14:25:17.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec37fff640 1 -- 172.21.15.46:0/1134883684 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec480ba660 con 0x7fec500c6bd0 2023-10-30T14:25:17.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec55b3e640 1 -- 172.21.15.46:0/1134883684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec501553a0 con 0x7fec500c6bd0 2023-10-30T14:25:17.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec55b3e640 1 -- 172.21.15.46:0/1134883684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec50155960 con 0x7fec500c6bd0 2023-10-30T14:25:17.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.994+0000 7fec55b3e640 1 -- 172.21.15.46:0/1134883684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5004e170 con 0x7fec500c6bd0 2023-10-30T14:25:17.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.998+0000 7fec37fff640 1 -- 172.21.15.46:0/1134883684 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fec480b6040 con 0x7fec500c6bd0 2023-10-30T14:25:17.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.998+0000 7fec37fff640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fec3c0664d0 0x7fec3c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:17.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.998+0000 7fec37fff640 1 -- 172.21.15.46:0/1134883684 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fec481229a0 con 0x7fec500c6bd0 2023-10-30T14:25:17.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.998+0000 7fec4f7fe640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fec3c0664d0 0x7fec3c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:17.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:16.998+0000 7fec4f7fe640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fec3c0664d0 0x7fec3c068990 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fec40000f70 tx=0x7fec40009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:17.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.002+0000 7fec37fff640 1 -- 172.21.15.46:0/1134883684 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fec480ef2a0 con 0x7fec500c6bd0 2023-10-30T14:25:17.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.190+0000 7fec55b3e640 1 -- 172.21.15.46:0/1134883684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec500c7e90 con 0x7fec500c6bd0 2023-10-30T14:25:17.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec37fff640 1 -- 172.21.15.46:0/1134883684 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 262 v262) v1 ==== 95+0+5740 (secure 0 0 0) 0x7fec480f28c0 con 0x7fec500c6bd0 2023-10-30T14:25:17.196 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:17.196 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":262,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:15.947369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:17.196 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 262 2023-10-30T14:25:17.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec55b3e640 1 -- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fec3c0664d0 msgr2=0x7fec3c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:17.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec55b3e640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fec3c0664d0 0x7fec3c068990 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fec40000f70 tx=0x7fec40009040 comp rx=0 tx=0).stop 2023-10-30T14:25:17.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec55b3e640 1 -- 172.21.15.46:0/1134883684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec500c6bd0 msgr2=0x7fec50156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:17.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec55b3e640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec500c6bd0 0x7fec50156cd0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fec480b2040 tx=0x7fec4801bf90 comp rx=0 tx=0).stop 2023-10-30T14:25:17.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec55b3e640 1 -- 172.21.15.46:0/1134883684 shutdown_connections 2023-10-30T14:25:17.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec55b3e640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fec3c0664d0 0x7fec3c068990 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:17.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec55b3e640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec500c6bd0 0x7fec50156cd0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:17.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec55b3e640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec50071110 0x7fec5014d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:17.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec55b3e640 1 --2- 172.21.15.46:0/1134883684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec5006f7a0 0x7fec501582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:17.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec55b3e640 1 -- 172.21.15.46:0/1134883684 >> 172.21.15.46:0/1134883684 conn(0x7fec500c7910 msgr2=0x7fec500c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:17.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec55b3e640 1 -- 172.21.15.46:0/1134883684 shutdown_connections 2023-10-30T14:25:17.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:17.194+0000 7fec55b3e640 1 -- 172.21.15.46:0/1134883684 wait complete. 2023-10-30T14:25:17.217 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:25:19.219 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:19.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a4205640 1 Processor -- start 2023-10-30T14:25:19.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a4205640 1 -- start start 2023-10-30T14:25:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a4205640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe19c151800 0x7fe19c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a4205640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe19c0c1ee0 0x7fe19c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a4205640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19c06e090 con 0x7fe19c0c9230 2023-10-30T14:25:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a4205640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe19c06dc70 con 0x7fe19c0c1ee0 2023-10-30T14:25:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a4205640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0c8a50 con 0x7fe19c151800 2023-10-30T14:25:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a27a2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe19c0c9230 0x7fe19c0cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37104/0 (socket says 172.21.15.46:37104) 2023-10-30T14:25:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a27a2640 1 -- 172.21.15.46:0/1523587050 learned_addr learned my addr 172.21.15.46:0/1523587050 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:25:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a17a0640 1 --2- 172.21.15.46:0/1523587050 >> v2:172.21.15.154:3300/0 conn(0x7fe19c0c1ee0 0x7fe19c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a1fa1640 1 --2- 172.21.15.46:0/1523587050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe19c151800 0x7fe19c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a0f9f640 1 -- 172.21.15.46:0/1523587050 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394414122 0 0) 0x7fe19c06e090 con 0x7fe19c0c9230 2023-10-30T14:25:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a0f9f640 1 -- 172.21.15.46:0/1523587050 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe188003530 con 0x7fe19c0c9230 2023-10-30T14:25:19.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a0f9f640 1 -- 172.21.15.46:0/1523587050 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278666338 0 0) 0x7fe188003530 con 0x7fe19c0c9230 2023-10-30T14:25:19.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a17a0640 1 -- 172.21.15.46:0/1523587050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe19c151800 msgr2=0x7fe19c06e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:19.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a17a0640 1 --2- 172.21.15.46:0/1523587050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe19c151800 0x7fe19c06e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:19.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a17a0640 1 -- 172.21.15.46:0/1523587050 >> v1:172.21.15.46:6789/0 conn(0x7fe19c0c9230 legacy=0x7fe19c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:19.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a17a0640 1 -- 172.21.15.46:0/1523587050 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c0c3b10 con 0x7fe19c0c1ee0 2023-10-30T14:25:19.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.362+0000 7fe1a1fa1640 1 --2- 172.21.15.46:0/1523587050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe19c151800 0x7fe19c06e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:25:19.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.366+0000 7fe1a17a0640 1 --2- 172.21.15.46:0/1523587050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe19c0c1ee0 0x7fe19c0c8cf0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0097c0 tx=0x7fe18c01c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=101169340dde572a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:19.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.366+0000 7fe1a0f9f640 1 -- 172.21.15.46:0/1523587050 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194001a20 con 0x7fe19c0c1ee0 2023-10-30T14:25:19.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.366+0000 7fe1a0f9f640 1 -- 172.21.15.46:0/1523587050 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18c0a7050 con 0x7fe19c0c1ee0 2023-10-30T14:25:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.366+0000 7fe1a0f9f640 1 -- 172.21.15.46:0/1523587050 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c0ab600 con 0x7fe19c0c1ee0 2023-10-30T14:25:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.366+0000 7fe1a4205640 1 -- 172.21.15.46:0/1523587050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe19c0c1ee0 msgr2=0x7fe19c0c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.366+0000 7fe1a4205640 1 --2- 172.21.15.46:0/1523587050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe19c0c1ee0 0x7fe19c0c8cf0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0097c0 tx=0x7fe18c01c370 comp rx=0 tx=0).stop 2023-10-30T14:25:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.366+0000 7fe1a4205640 1 -- 172.21.15.46:0/1523587050 shutdown_connections 2023-10-30T14:25:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.366+0000 7fe1a4205640 1 --2- 172.21.15.46:0/1523587050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe19c0c1ee0 0x7fe19c0c8cf0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.366+0000 7fe1a4205640 1 --2- 172.21.15.46:0/1523587050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe19c151800 0x7fe19c06e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.366+0000 7fe1a4205640 1 -- 172.21.15.46:0/1523587050 >> 172.21.15.46:0/1523587050 conn(0x7fe19c071ea0 msgr2=0x7fe19c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.366+0000 7fe1a4205640 1 -- 172.21.15.46:0/1523587050 shutdown_connections 2023-10-30T14:25:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a4205640 1 -- 172.21.15.46:0/1523587050 wait complete. 2023-10-30T14:25:19.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a4205640 1 Processor -- start 2023-10-30T14:25:19.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a4205640 1 -- start start 2023-10-30T14:25:19.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a4205640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe19c0c1ee0 0x7fe19c14b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:19.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a4205640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe19c0c9230 0x7fe19c1f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:19.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a4205640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe19c151800 0x7fe19c146c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:19.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a4205640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c155440 con 0x7fe19c151800 2023-10-30T14:25:19.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a4205640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c1f6110 con 0x7fe19c0c1ee0 2023-10-30T14:25:19.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a4205640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c155690 con 0x7fe19c0c9230 2023-10-30T14:25:19.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a27a2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe19c151800 0x7fe19c146c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:19.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a27a2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe19c151800 0x7fe19c146c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56296/0 (socket says 172.21.15.46:56296) 2023-10-30T14:25:19.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a27a2640 1 -- 172.21.15.46:0/14994389 learned_addr learned my addr 172.21.15.46:0/14994389 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:19.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a17a0640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe19c0c9230 0x7fe19c1f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:19.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a1fa1640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe19c0c1ee0 0x7fe19c14b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:19.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a27a2640 1 -- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe19c0c9230 msgr2=0x7fe19c1f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:19.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a27a2640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe19c0c9230 0x7fe19c1f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:19.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a27a2640 1 -- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe19c0c1ee0 msgr2=0x7fe19c14b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:19.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a27a2640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe19c0c1ee0 0x7fe19c14b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:19.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a27a2640 1 -- 172.21.15.46:0/14994389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe198009420 con 0x7fe19c151800 2023-10-30T14:25:19.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a27a2640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe19c151800 0x7fe19c146c90 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fe19400caf0 tx=0x7fe1940020a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:19.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1927fc640 1 -- 172.21.15.46:0/14994389 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19400a700 con 0x7fe19c151800 2023-10-30T14:25:19.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1927fc640 1 -- 172.21.15.46:0/14994389 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19400a880 con 0x7fe19c151800 2023-10-30T14:25:19.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1927fc640 1 -- 172.21.15.46:0/14994389 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19400ac30 con 0x7fe19c151800 2023-10-30T14:25:19.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.370+0000 7fe1a4205640 1 -- 172.21.15.46:0/14994389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18c009420 con 0x7fe19c151800 2023-10-30T14:25:19.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.374+0000 7fe1a4205640 1 -- 172.21.15.46:0/14994389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19c1f2220 con 0x7fe19c151800 2023-10-30T14:25:19.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.374+0000 7fe1a4205640 1 -- 172.21.15.46:0/14994389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19c1f2410 con 0x7fe19c151800 2023-10-30T14:25:19.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.374+0000 7fe1927fc640 1 -- 172.21.15.46:0/14994389 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe1940070d0 con 0x7fe19c151800 2023-10-30T14:25:19.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.374+0000 7fe1927fc640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe1800664a0 0x7fe180068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:19.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.374+0000 7fe1927fc640 1 -- 172.21.15.46:0/14994389 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fe194084230 con 0x7fe19c151800 2023-10-30T14:25:19.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.374+0000 7fe1a1fa1640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe1800664a0 0x7fe180068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:19.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.378+0000 7fe1a1fa1640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe1800664a0 0x7fe180068960 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fe198009c80 tx=0x7fe198008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:19.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.378+0000 7fe1927fc640 1 -- 172.21.15.46:0/14994389 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe1940509d0 con 0x7fe19c151800 2023-10-30T14:25:19.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.570+0000 7fe1a4205640 1 -- 172.21.15.46:0/14994389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe19c04e170 con 0x7fe19c151800 2023-10-30T14:25:19.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.570+0000 7fe1927fc640 1 -- 172.21.15.46:0/14994389 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 262 v262) v1 ==== 95+0+5740 (secure 0 0 0) 0x7fe194018070 con 0x7fe19c151800 2023-10-30T14:25:19.574 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:19.575 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":262,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:15.947369+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:19.575 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 262 2023-10-30T14:25:19.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.574+0000 7fe1a4205640 1 -- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe1800664a0 msgr2=0x7fe180068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:19.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.574+0000 7fe1a4205640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe1800664a0 0x7fe180068960 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fe198009c80 tx=0x7fe198008040 comp rx=0 tx=0).stop 2023-10-30T14:25:19.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.574+0000 7fe1a4205640 1 -- 172.21.15.46:0/14994389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe19c151800 msgr2=0x7fe19c146c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:19.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.574+0000 7fe1a4205640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe19c151800 0x7fe19c146c90 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fe19400caf0 tx=0x7fe1940020a0 comp rx=0 tx=0).stop 2023-10-30T14:25:19.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.574+0000 7fe1a4205640 1 -- 172.21.15.46:0/14994389 shutdown_connections 2023-10-30T14:25:19.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.574+0000 7fe1a4205640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fe1800664a0 0x7fe180068960 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:19.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.574+0000 7fe1a4205640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe19c151800 0x7fe19c146c90 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:19.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.574+0000 7fe1a4205640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe19c0c9230 0x7fe19c1f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:19.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.574+0000 7fe1a4205640 1 --2- 172.21.15.46:0/14994389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe19c0c1ee0 0x7fe19c14b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:19.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.574+0000 7fe1a4205640 1 -- 172.21.15.46:0/14994389 >> 172.21.15.46:0/14994389 conn(0x7fe19c071ea0 msgr2=0x7fe19c0c60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:19.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.574+0000 7fe1a4205640 1 -- 172.21.15.46:0/14994389 shutdown_connections 2023-10-30T14:25:19.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:19.574+0000 7fe1a4205640 1 -- 172.21.15.46:0/14994389 wait complete. 2023-10-30T14:25:19.595 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:25:21.596 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:21.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.734+0000 7f2f7b29e640 1 Processor -- start 2023-10-30T14:25:21.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.734+0000 7f2f7b29e640 1 -- start start 2023-10-30T14:25:21.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7b29e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f7415d6f0 0x7f2f7415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:21.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7b29e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f740cb800 con 0x7f2f741538a0 2023-10-30T14:25:21.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7b29e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f740cb330 con 0x7f2f740cf160 2023-10-30T14:25:21.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7b29e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740cad70 con 0x7f2f7415d6f0 2023-10-30T14:25:21.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7903a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2f741538a0 0x7f2f740ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37118/0 (socket says 172.21.15.46:37118) 2023-10-30T14:25:21.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7903a640 1 -- 172.21.15.46:0/93225520 learned_addr learned my addr 172.21.15.46:0/93225520 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:25:21.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f78839640 1 --2- 172.21.15.46:0/93225520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f7415d6f0 0x7f2f7415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:21.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049027665 0 0) 0x7f2f740cb800 con 0x7f2f741538a0 2023-10-30T14:25:21.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f60003530 con 0x7f2f741538a0 2023-10-30T14:25:21.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 20145340 0 0) 0x7f2f60003530 con 0x7f2f741538a0 2023-10-30T14:25:21.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f7415d6f0 msgr2=0x7f2f7415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:21.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 --2- 172.21.15.46:0/93225520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f7415d6f0 0x7f2f7415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:21.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 >> v1:172.21.15.154:6789/0 conn(0x7f2f740cf160 legacy=0x7f2f74164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:21.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f740cb050 con 0x7f2f741538a0 2023-10-30T14:25:21.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f2f70001680 con 0x7f2f741538a0 2023-10-30T14:25:21.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 >> v1:172.21.15.46:6789/0 conn(0x7f2f741538a0 legacy=0x7f2f740ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:21.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 --2- 172.21.15.46:0/93225520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f7415d6f0 0x7f2f600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:21.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 --2- 172.21.15.46:0/93225520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f60003f30 0x7f2f600047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 --2- 172.21.15.46:0/93225520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f60004d10 0x7f2f60005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740cad70 con 0x7f2f60004d10 2023-10-30T14:25:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6000bb00 con 0x7f2f60003f30 2023-10-30T14:25:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6000bcc0 con 0x7f2f7415d6f0 2023-10-30T14:25:21.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7983b640 1 --2- 172.21.15.46:0/93225520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f60004d10 0x7f2f60005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f78839640 1 --2- 172.21.15.46:0/93225520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f60003f30 0x7f2f600047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7903a640 1 --2- 172.21.15.46:0/93225520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f7415d6f0 0x7f2f600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7983b640 1 -- 172.21.15.46:0/93225520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f7415d6f0 msgr2=0x7f2f600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:21.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7983b640 1 --2- 172.21.15.46:0/93225520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f7415d6f0 0x7f2f600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:21.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7983b640 1 -- 172.21.15.46:0/93225520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f60003f30 msgr2=0x7f2f600047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:21.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7983b640 1 --2- 172.21.15.46:0/93225520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f60003f30 0x7f2f600047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:21.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.738+0000 7f2f7983b640 1 -- 172.21.15.46:0/93225520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f60003530 con 0x7f2f60004d10 2023-10-30T14:25:21.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.742+0000 7f2f7983b640 1 --2- 172.21.15.46:0/93225520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f60004d10 0x7f2f60005190 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c001e70 tx=0x7f2f6c01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed26152509f23ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.742+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c003b80 con 0x7f2f60004d10 2023-10-30T14:25:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.742+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6c003ce0 con 0x7f2f60004d10 2023-10-30T14:25:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.742+0000 7f2f6bfff640 1 -- 172.21.15.46:0/93225520 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c0052c0 con 0x7f2f60004d10 2023-10-30T14:25:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.742+0000 7f2f7b29e640 1 -- 172.21.15.46:0/93225520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f60004d10 msgr2=0x7f2f60005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.742+0000 7f2f7b29e640 1 --2- 172.21.15.46:0/93225520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f60004d10 0x7f2f60005190 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c001e70 tx=0x7f2f6c01c760 comp rx=0 tx=0).stop 2023-10-30T14:25:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.742+0000 7f2f7983b640 1 -- 172.21.15.46:0/93225520 reap_dead start 2023-10-30T14:25:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7b29e640 1 -- 172.21.15.46:0/93225520 shutdown_connections 2023-10-30T14:25:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7b29e640 1 -- 172.21.15.46:0/93225520 >> 172.21.15.46:0/93225520 conn(0x7f2f7406bfe0 msgr2=0x7f2f7406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7b29e640 1 -- 172.21.15.46:0/93225520 shutdown_connections 2023-10-30T14:25:21.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7b29e640 1 -- 172.21.15.46:0/93225520 wait complete. 2023-10-30T14:25:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7b29e640 1 Processor -- start 2023-10-30T14:25:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7b29e640 1 -- start start 2023-10-30T14:25:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7b29e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f60004d10 0x7f2f741efe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7b29e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7414d690 0x7f2f7414df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7b29e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f7415b1b0 0x7f2f7414e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7b29e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f74150bf0 con 0x7f2f7414d690 2023-10-30T14:25:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7b29e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f741ef3e0 con 0x7f2f7415b1b0 2023-10-30T14:25:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7b29e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740b50e0 con 0x7f2f60004d10 2023-10-30T14:25:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f78839640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7414d690 0x7f2f7414df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f78839640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7414d690 0x7f2f7414df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56326/0 (socket says 172.21.15.46:56326) 2023-10-30T14:25:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f78839640 1 -- 172.21.15.46:0/2740085237 learned_addr learned my addr 172.21.15.46:0/2740085237 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:21.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7903a640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f60004d10 0x7f2f741efe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:21.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f7983b640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f7415b1b0 0x7f2f7414e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:21.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f78839640 1 -- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f60004d10 msgr2=0x7f2f741efe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:21.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f78839640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f60004d10 0x7f2f741efe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:21.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f78839640 1 -- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f7415b1b0 msgr2=0x7f2f7414e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:21.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f78839640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f7415b1b0 0x7f2f7414e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:21.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.746+0000 7f2f78839640 1 -- 172.21.15.46:0/2740085237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f741f33f0 con 0x7f2f7414d690 2023-10-30T14:25:21.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.750+0000 7f2f78839640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7414d690 0x7f2f7414df80 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f2f640119d0 tx=0x7f2f64011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:21.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.750+0000 7f2f69ffb640 1 -- 172.21.15.46:0/2740085237 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400fb50 con 0x7f2f7414d690 2023-10-30T14:25:21.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.750+0000 7f2f69ffb640 1 -- 172.21.15.46:0/2740085237 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6400fcb0 con 0x7f2f7414d690 2023-10-30T14:25:21.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.750+0000 7f2f69ffb640 1 -- 172.21.15.46:0/2740085237 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64014440 con 0x7f2f7414d690 2023-10-30T14:25:21.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.750+0000 7f2f7b29e640 1 -- 172.21.15.46:0/2740085237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f741ef680 con 0x7f2f7414d690 2023-10-30T14:25:21.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.750+0000 7f2f7b29e640 1 -- 172.21.15.46:0/2740085237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f741f36d0 con 0x7f2f7414d690 2023-10-30T14:25:21.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.750+0000 7f2f7b29e640 1 -- 172.21.15.46:0/2740085237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7404e170 con 0x7f2f7414d690 2023-10-30T14:25:21.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.750+0000 7f2f69ffb640 1 -- 172.21.15.46:0/2740085237 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2f640145a0 con 0x7f2f7414d690 2023-10-30T14:25:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.750+0000 7f2f69ffb640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2f580664d0 0x7f2f58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.750+0000 7f2f69ffb640 1 -- 172.21.15.46:0/2740085237 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f2f640852c0 con 0x7f2f7414d690 2023-10-30T14:25:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.754+0000 7f2f7903a640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2f580664d0 0x7f2f58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.754+0000 7f2f7903a640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2f580664d0 0x7f2f58068990 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f2f70007d50 tx=0x7f2f70009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:21.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.758+0000 7f2f69ffb640 1 -- 172.21.15.46:0/2740085237 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2f64051c80 con 0x7f2f7414d690 2023-10-30T14:25:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.950+0000 7f2f7b29e640 1 -- 172.21.15.46:0/2740085237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f740b53a0 con 0x7f2f7414d690 2023-10-30T14:25:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.950+0000 7f2f69ffb640 1 -- 172.21.15.46:0/2740085237 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+5737 (secure 0 0 0) 0x7f2f64022020 con 0x7f2f7414d690 2023-10-30T14:25:21.955 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:21.956 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:21.170937+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 263 2023-10-30T14:25:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.954+0000 7f2f7b29e640 1 -- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2f580664d0 msgr2=0x7f2f58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.954+0000 7f2f7b29e640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2f580664d0 0x7f2f58068990 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f2f70007d50 tx=0x7f2f70009040 comp rx=0 tx=0).stop 2023-10-30T14:25:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.954+0000 7f2f7b29e640 1 -- 172.21.15.46:0/2740085237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7414d690 msgr2=0x7f2f7414df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.954+0000 7f2f7b29e640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7414d690 0x7f2f7414df80 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f2f640119d0 tx=0x7f2f64011e90 comp rx=0 tx=0).stop 2023-10-30T14:25:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.954+0000 7f2f7b29e640 1 -- 172.21.15.46:0/2740085237 shutdown_connections 2023-10-30T14:25:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.954+0000 7f2f7b29e640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2f580664d0 0x7f2f58068990 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.954+0000 7f2f7b29e640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f7415b1b0 0x7f2f7414e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.954+0000 7f2f7b29e640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7414d690 0x7f2f7414df80 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.954+0000 7f2f7b29e640 1 --2- 172.21.15.46:0/2740085237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2f60004d10 0x7f2f741efe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.954+0000 7f2f7b29e640 1 -- 172.21.15.46:0/2740085237 >> 172.21.15.46:0/2740085237 conn(0x7f2f7406bfe0 msgr2=0x7f2f741510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.954+0000 7f2f7b29e640 1 -- 172.21.15.46:0/2740085237 shutdown_connections 2023-10-30T14:25:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:21.954+0000 7f2f7b29e640 1 -- 172.21.15.46:0/2740085237 wait complete. 2023-10-30T14:25:21.977 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:25:23.978 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:24.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.118+0000 7f8f14454640 1 Processor -- start 2023-10-30T14:25:24.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.118+0000 7f8f14454640 1 -- start start 2023-10-30T14:25:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f14454640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8f0c15d6f0 0x7f8f0c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f14454640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0cf160 0x7f8f0c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f14454640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0c0cb800 con 0x7f8f0c1538a0 2023-10-30T14:25:24.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f14454640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f0c0cb330 con 0x7f8f0c15d6f0 2023-10-30T14:25:24.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f14454640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c0cad70 con 0x7f8f0c0cf160 2023-10-30T14:25:24.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f121f0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8f0c1538a0 0x7f8f0c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37126/0 (socket says 172.21.15.46:37126) 2023-10-30T14:25:24.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f121f0640 1 -- 172.21.15.46:0/627023294 learned_addr learned my addr 172.21.15.46:0/627023294 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:25:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f119ef640 1 --2- 172.21.15.46:0/627023294 >> v2:172.21.15.154:3300/0 conn(0x7f8f0c15d6f0 0x7f8f0c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f129f1640 1 --2- 172.21.15.46:0/627023294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0cf160 0x7f8f0c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435554607 0 0) 0x7f8f0c0cb800 con 0x7f8f0c1538a0 2023-10-30T14:25:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef8003530 con 0x7f8f0c1538a0 2023-10-30T14:25:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2299855867 0 0) 0x7f8ef8003530 con 0x7f8f0c1538a0 2023-10-30T14:25:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0cf160 msgr2=0x7f8f0c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 --2- 172.21.15.46:0/627023294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0cf160 0x7f8f0c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 >> v2:172.21.15.154:3300/0 conn(0x7f8f0c15d6f0 msgr2=0x7f8f0c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 --2- 172.21.15.46:0/627023294 >> v2:172.21.15.154:3300/0 conn(0x7f8f0c15d6f0 0x7f8f0c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c0cb050 con 0x7f8f0c1538a0 2023-10-30T14:25:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f8f08001680 con 0x7f8f0c1538a0 2023-10-30T14:25:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 >> v1:172.21.15.46:6789/0 conn(0x7f8f0c1538a0 legacy=0x7f8f0c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 --2- 172.21.15.46:0/627023294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0cf160 0x7f8ef8003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 --2- 172.21.15.46:0/627023294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef8003f00 0x7f8ef80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 --2- 172.21.15.46:0/627023294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef8004d10 0x7f8ef8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c0cb330 con 0x7f8ef8004d10 2023-10-30T14:25:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c0cad70 con 0x7f8ef8003f00 2023-10-30T14:25:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef800b9c0 con 0x7f8f0c0cf160 2023-10-30T14:25:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f129f1640 1 --2- 172.21.15.46:0/627023294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef8004d10 0x7f8ef8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f119ef640 1 --2- 172.21.15.46:0/627023294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef8003f00 0x7f8ef80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f121f0640 1 --2- 172.21.15.46:0/627023294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0cf160 0x7f8ef8003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f129f1640 1 -- 172.21.15.46:0/627023294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0cf160 msgr2=0x7f8ef8003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f129f1640 1 --2- 172.21.15.46:0/627023294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0cf160 0x7f8ef8003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f129f1640 1 -- 172.21.15.46:0/627023294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef8003f00 msgr2=0x7f8ef80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:24.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f129f1640 1 --2- 172.21.15.46:0/627023294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef8003f00 0x7f8ef80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:24.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.122+0000 7f8f129f1640 1 -- 172.21.15.46:0/627023294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef8003530 con 0x7f8ef8004d10 2023-10-30T14:25:24.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.126+0000 7f8f129f1640 1 --2- 172.21.15.46:0/627023294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef8004d10 0x7f8ef8005190 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f8f04009fc0 tx=0x7f8f0401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa64b33cc8d680f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:24.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.126+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f040a5c70 con 0x7f8ef8004d10 2023-10-30T14:25:24.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.126+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0400f040 con 0x7f8ef8004d10 2023-10-30T14:25:24.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.126+0000 7f8f111ee640 1 -- 172.21.15.46:0/627023294 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f040ad440 con 0x7f8ef8004d10 2023-10-30T14:25:24.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.126+0000 7f8f14454640 1 -- 172.21.15.46:0/627023294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef8004d10 msgr2=0x7f8ef8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:24.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.126+0000 7f8f14454640 1 --2- 172.21.15.46:0/627023294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ef8004d10 0x7f8ef8005190 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f8f04009fc0 tx=0x7f8f0401cc60 comp rx=0 tx=0).stop 2023-10-30T14:25:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.126+0000 7f8f129f1640 1 -- 172.21.15.46:0/627023294 reap_dead start 2023-10-30T14:25:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.126+0000 7f8f14454640 1 -- 172.21.15.46:0/627023294 shutdown_connections 2023-10-30T14:25:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.126+0000 7f8f14454640 1 -- 172.21.15.46:0/627023294 >> 172.21.15.46:0/627023294 conn(0x7f8f0c06bfe0 msgr2=0x7f8f0c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.126+0000 7f8f14454640 1 -- 172.21.15.46:0/627023294 shutdown_connections 2023-10-30T14:25:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.126+0000 7f8f14454640 1 -- 172.21.15.46:0/627023294 wait complete. 2023-10-30T14:25:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f14454640 1 Processor -- start 2023-10-30T14:25:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f14454640 1 -- start start 2023-10-30T14:25:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f14454640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef8004d10 0x7f8f0c14dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f14454640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0c0c64d0 0x7f8f0c14c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f14454640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0c7c70 0x7f8f0c14c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f14454640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c1578f0 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f14454640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c1575c0 con 0x7f8ef8004d10 2023-10-30T14:25:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f14454640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c157190 con 0x7f8f0c0c7c70 2023-10-30T14:25:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f119ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0c0c64d0 0x7f8f0c14c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f121f0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef8004d10 0x7f8f0c14dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f119ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0c0c64d0 0x7f8f0c14c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56344/0 (socket says 172.21.15.46:56344) 2023-10-30T14:25:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f119ef640 1 -- 172.21.15.46:0/566488699 learned_addr learned my addr 172.21.15.46:0/566488699 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f129f1640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0c7c70 0x7f8f0c14c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f119ef640 1 -- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0c7c70 msgr2=0x7f8f0c14c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f119ef640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0c7c70 0x7f8f0c14c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f119ef640 1 -- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef8004d10 msgr2=0x7f8f0c14dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f119ef640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef8004d10 0x7f8f0c14dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f119ef640 1 -- 172.21.15.46:0/566488699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c0c8f10 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f119ef640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0c0c64d0 0x7f8f0c14c080 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0116a0 tx=0x7f8efc011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f02ffd640 1 -- 172.21.15.46:0/566488699 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc00f9c0 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f02ffd640 1 -- 172.21.15.46:0/566488699 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc00fb20 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f14454640 1 -- 172.21.15.46:0/566488699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0c0c91f0 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f02ffd640 1 -- 172.21.15.46:0/566488699 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc00fe90 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f14454640 1 -- 172.21.15.46:0/566488699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0c0c97b0 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.130+0000 7f8f14454640 1 -- 172.21.15.46:0/566488699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0c157730 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.134+0000 7f8f02ffd640 1 -- 172.21.15.46:0/566488699 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8efc015460 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.134+0000 7f8f02ffd640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ef00664d0 0x7f8ef0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.134+0000 7f8f02ffd640 1 -- 172.21.15.46:0/566488699 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f8efc085220 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.134+0000 7f8f121f0640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ef00664d0 0x7f8ef0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.134+0000 7f8f121f0640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ef00664d0 0x7f8ef0068990 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f8f08000fc0 tx=0x7f8f08007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.138+0000 7f8f02ffd640 1 -- 172.21.15.46:0/566488699 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f0c157730 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.342+0000 7f8f14454640 1 -- 172.21.15.46:0/566488699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f0c157730 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.342+0000 7f8f02ffd640 1 -- 172.21.15.46:0/566488699 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+5737 (secure 0 0 0) 0x7f8f0c157730 con 0x7f8f0c0c64d0 2023-10-30T14:25:24.348 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:24.348 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:21.170937+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:24.348 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 263 2023-10-30T14:25:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.346+0000 7f8f14454640 1 -- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ef00664d0 msgr2=0x7f8ef0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:24.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.346+0000 7f8f14454640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ef00664d0 0x7f8ef0068990 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f8f08000fc0 tx=0x7f8f08007040 comp rx=0 tx=0).stop 2023-10-30T14:25:24.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.346+0000 7f8f14454640 1 -- 172.21.15.46:0/566488699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0c0c64d0 msgr2=0x7f8f0c14c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:24.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.346+0000 7f8f14454640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0c0c64d0 0x7f8f0c14c080 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0116a0 tx=0x7f8efc011b60 comp rx=0 tx=0).stop 2023-10-30T14:25:24.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.346+0000 7f8f14454640 1 -- 172.21.15.46:0/566488699 shutdown_connections 2023-10-30T14:25:24.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.346+0000 7f8f14454640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8ef00664d0 0x7f8ef0068990 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:24.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.346+0000 7f8f14454640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f0c0c7c70 0x7f8f0c14c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:24.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.346+0000 7f8f14454640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0c0c64d0 0x7f8f0c14c080 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:24.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.346+0000 7f8f14454640 1 --2- 172.21.15.46:0/566488699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef8004d10 0x7f8f0c14dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:24.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.346+0000 7f8f14454640 1 -- 172.21.15.46:0/566488699 >> 172.21.15.46:0/566488699 conn(0x7f8f0c06bfe0 msgr2=0x7f8f0c0ccfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:24.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.346+0000 7f8f14454640 1 -- 172.21.15.46:0/566488699 shutdown_connections 2023-10-30T14:25:24.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:24.346+0000 7f8f14454640 1 -- 172.21.15.46:0/566488699 wait complete. 2023-10-30T14:25:24.369 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 263, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:23:19.337396+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 263, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_10020": { "addr": "172.21.15.46:6837/2187461695", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6835", "nonce": 2187461695, "type": "v2" }, { "addr": "172.21.15.46:6837", "nonce": 2187461695, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 10020, "incarnation": 260, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 16 }, "gid_9882": { "addr": "172.21.15.46:6833/3394097058", "addrs": { "addrvec": [ { "addr": "172.21.15.46:6832", "nonce": 3394097058, "type": "v2" }, { "addr": "172.21.15.46:6833", "nonce": 3394097058, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9882, "incarnation": 253, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 102, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T14:25:21.170937+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 10020, "mds_1": 9882 } } } ], "standbys": [ { "addr": "172.21.15.154:6838/892930607", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 892930607, "type": "v2" }, { "addr": "172.21.15.154:6838", "nonce": 892930607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6839/2803385213", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6837", "nonce": 2803385213, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2803385213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.154:6835/3159009654", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 3159009654, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 3159009654, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 102, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5939, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T14:25:26.370 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:26.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.510+0000 7fd4efbc5640 1 Processor -- start 2023-10-30T14:25:26.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.510+0000 7fd4efbc5640 1 -- start start 2023-10-30T14:25:26.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.510+0000 7fd4efbc5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd4e81538a0 0x7fd4e815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:26.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.510+0000 7fd4efbc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4e80ca0e0 0x7fd4e815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:26.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.510+0000 7fd4efbc5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e80cb800 con 0x7fd4e80cf160 2023-10-30T14:25:26.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.510+0000 7fd4efbc5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e815dc30 con 0x7fd4e81538a0 2023-10-30T14:25:26.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.510+0000 7fd4efbc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e80cb330 con 0x7fd4e80ca0e0 2023-10-30T14:25:26.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.514+0000 7fd4ed961640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd4e81538a0 0x7fd4e815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:26.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.514+0000 7fd4ed160640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4e80ca0e0 0x7fd4e815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:26.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.514+0000 7fd4ee162640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd4e80cf160 0x7fd4e80cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:39406/0 (socket says 172.21.15.46:39406) 2023-10-30T14:25:26.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.514+0000 7fd4ee162640 1 -- 172.21.15.46:0/186437484 learned_addr learned my addr 172.21.15.46:0/186437484 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:25:26.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.514+0000 7fd4ed961640 1 -- 172.21.15.46:0/186437484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4e80ca0e0 msgr2=0x7fd4e815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:26.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.514+0000 7fd4ed961640 1 --2- 172.21.15.46:0/186437484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4e80ca0e0 0x7fd4e815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:26.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.514+0000 7fd4ed961640 1 -- 172.21.15.46:0/186437484 >> v1:172.21.15.154:6789/0 conn(0x7fd4e80cf160 legacy=0x7fd4e80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:26.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.514+0000 7fd4ed961640 1 -- 172.21.15.46:0/186437484 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e80cad70 con 0x7fd4e81538a0 2023-10-30T14:25:26.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4ed961640 1 --2- 172.21.15.46:0/186437484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e81538a0 0x7fd4e815d6f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40097c0 tx=0x7fd4e401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b795815d126cc562 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:26.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4ec95f640 1 -- 172.21.15.46:0/186437484 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e40a8970 con 0x7fd4e81538a0 2023-10-30T14:25:26.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4ec95f640 1 -- 172.21.15.46:0/186437484 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e40a8ad0 con 0x7fd4e81538a0 2023-10-30T14:25:26.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4ec95f640 1 -- 172.21.15.46:0/186437484 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e40a8e60 con 0x7fd4e81538a0 2023-10-30T14:25:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 -- 172.21.15.46:0/186437484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e81538a0 msgr2=0x7fd4e815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:26.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 --2- 172.21.15.46:0/186437484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e81538a0 0x7fd4e815d6f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40097c0 tx=0x7fd4e401ec70 comp rx=0 tx=0).stop 2023-10-30T14:25:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 -- 172.21.15.46:0/186437484 shutdown_connections 2023-10-30T14:25:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 --2- 172.21.15.46:0/186437484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4e80ca0e0 0x7fd4e815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 --2- 172.21.15.46:0/186437484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e81538a0 0x7fd4e815d6f0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:26.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 -- 172.21.15.46:0/186437484 >> 172.21.15.46:0/186437484 conn(0x7fd4e806bfe0 msgr2=0x7fd4e806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:26.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 -- 172.21.15.46:0/186437484 shutdown_connections 2023-10-30T14:25:26.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 -- 172.21.15.46:0/186437484 wait complete. 2023-10-30T14:25:26.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 Processor -- start 2023-10-30T14:25:26.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 -- start start 2023-10-30T14:25:26.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80ca0e0 0x7fd4e80ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:26.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4e80cf160 0x7fd4e8158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:26.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4e81538a0 0x7fd4e814cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:26.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e81ef540 con 0x7fd4e80cf160 2023-10-30T14:25:26.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e81f0510 con 0x7fd4e80ca0e0 2023-10-30T14:25:26.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.518+0000 7fd4efbc5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e80666b0 con 0x7fd4e81538a0 2023-10-30T14:25:26.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4ed160640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4e80cf160 0x7fd4e8158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:26.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4ed160640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4e80cf160 0x7fd4e8158250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53254/0 (socket says 172.21.15.46:53254) 2023-10-30T14:25:26.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4ed160640 1 -- 172.21.15.46:0/1263959144 learned_addr learned my addr 172.21.15.46:0/1263959144 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:26.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4ed961640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80ca0e0 0x7fd4e80ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:26.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4ee162640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4e81538a0 0x7fd4e814cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:26.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4ed160640 1 -- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4e81538a0 msgr2=0x7fd4e814cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:26.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4ed160640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4e81538a0 0x7fd4e814cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:26.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4ed160640 1 -- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80ca0e0 msgr2=0x7fd4e80ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:26.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4ed160640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80ca0e0 0x7fd4e80ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:26.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4ed160640 1 -- 172.21.15.46:0/1263959144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e4009420 con 0x7fd4e80cf160 2023-10-30T14:25:26.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4ee162640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4e81538a0 0x7fd4e814cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:25:26.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4ed160640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4e80cf160 0x7fd4e8158250 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800e930 tx=0x7fd4d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:26.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4de7fc640 1 -- 172.21.15.46:0/1263959144 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8009690 con 0x7fd4e80cf160 2023-10-30T14:25:26.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4de7fc640 1 -- 172.21.15.46:0/1263959144 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d800ca00 con 0x7fd4e80cf160 2023-10-30T14:25:26.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4de7fc640 1 -- 172.21.15.46:0/1263959144 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d800cd70 con 0x7fd4e80cf160 2023-10-30T14:25:26.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4efbc5640 1 -- 172.21.15.46:0/1263959144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e81ef750 con 0x7fd4e80cf160 2023-10-30T14:25:26.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4efbc5640 1 -- 172.21.15.46:0/1263959144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e81efd10 con 0x7fd4e80cf160 2023-10-30T14:25:26.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4efbc5640 1 -- 172.21.15.46:0/1263959144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e80b2a50 con 0x7fd4e80cf160 2023-10-30T14:25:26.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.522+0000 7fd4de7fc640 1 -- 172.21.15.46:0/1263959144 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd4d8013470 con 0x7fd4e80cf160 2023-10-30T14:25:26.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.526+0000 7fd4de7fc640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd4cc0664a0 0x7fd4cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:26.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.526+0000 7fd4de7fc640 1 -- 172.21.15.46:0/1263959144 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fd4d8084c70 con 0x7fd4e80cf160 2023-10-30T14:25:26.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.526+0000 7fd4ed961640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd4cc0664a0 0x7fd4cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:26.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.526+0000 7fd4ed961640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd4cc0664a0 0x7fd4cc068960 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40b2000 tx=0x7fd4e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.530+0000 7fd4de7fc640 1 -- 172.21.15.46:0/1263959144 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd4d8051610 con 0x7fd4e80cf160 2023-10-30T14:25:26.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.722+0000 7fd4efbc5640 1 -- 172.21.15.46:0/1263959144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4e804e170 con 0x7fd4e80cf160 2023-10-30T14:25:26.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.722+0000 7fd4de7fc640 1 -- 172.21.15.46:0/1263959144 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+5737 (secure 0 0 0) 0x7fd4d8021090 con 0x7fd4e80cf160 2023-10-30T14:25:26.727 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:26.727 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:21.170937+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:26.727 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 263 2023-10-30T14:25:26.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.726+0000 7fd4efbc5640 1 -- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd4cc0664a0 msgr2=0x7fd4cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:26.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.726+0000 7fd4efbc5640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd4cc0664a0 0x7fd4cc068960 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40b2000 tx=0x7fd4e4008040 comp rx=0 tx=0).stop 2023-10-30T14:25:26.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.726+0000 7fd4efbc5640 1 -- 172.21.15.46:0/1263959144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4e80cf160 msgr2=0x7fd4e8158250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:26.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.726+0000 7fd4efbc5640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4e80cf160 0x7fd4e8158250 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800e930 tx=0x7fd4d800edf0 comp rx=0 tx=0).stop 2023-10-30T14:25:26.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.726+0000 7fd4efbc5640 1 -- 172.21.15.46:0/1263959144 shutdown_connections 2023-10-30T14:25:26.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.726+0000 7fd4efbc5640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd4cc0664a0 0x7fd4cc068960 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:26.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.726+0000 7fd4efbc5640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4e81538a0 0x7fd4e814cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:26.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.726+0000 7fd4efbc5640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4e80cf160 0x7fd4e8158250 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:26.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.726+0000 7fd4efbc5640 1 --2- 172.21.15.46:0/1263959144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80ca0e0 0x7fd4e80ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:26.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.726+0000 7fd4efbc5640 1 -- 172.21.15.46:0/1263959144 >> 172.21.15.46:0/1263959144 conn(0x7fd4e806bfe0 msgr2=0x7fd4e814db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:26.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.726+0000 7fd4efbc5640 1 -- 172.21.15.46:0/1263959144 shutdown_connections 2023-10-30T14:25:26.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:26.726+0000 7fd4efbc5640 1 -- 172.21.15.46:0/1263959144 wait complete. 2023-10-30T14:25:26.747 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:25:28.748 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:28.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8ddc19d640 1 Processor -- start 2023-10-30T14:25:28.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8ddc19d640 1 -- start start 2023-10-30T14:25:28.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8ddc19d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd41538a0 0x7f8dd415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:28.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8ddc19d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8dd40ca0e0 0x7f8dd415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:28.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8ddc19d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd40cb800 con 0x7f8dd40cf160 2023-10-30T14:25:28.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8ddc19d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd415dc30 con 0x7f8dd40ca0e0 2023-10-30T14:25:28.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8ddc19d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd40cb330 con 0x7f8dd41538a0 2023-10-30T14:25:28.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8dd9738640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8dd40ca0e0 0x7f8dd415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:28.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8dd9738640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8dd40ca0e0 0x7f8dd415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.46:40486/0 (socket says 172.21.15.46:40486) 2023-10-30T14:25:28.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8dda73a640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f8dd40cf160 0x7f8dd40cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:39416/0 (socket says 172.21.15.46:39416) 2023-10-30T14:25:28.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8dd9738640 1 -- 172.21.15.46:0/1254226328 learned_addr learned my addr 172.21.15.46:0/1254226328 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:28.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8dd9f39640 1 --2- 172.21.15.46:0/1254226328 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd41538a0 0x7f8dd415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:28.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8dd9738640 1 -- 172.21.15.46:0/1254226328 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd41538a0 msgr2=0x7f8dd415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:28.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8dd9738640 1 --2- 172.21.15.46:0/1254226328 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd41538a0 0x7f8dd415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:28.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8dd9738640 1 -- 172.21.15.46:0/1254226328 >> v1:172.21.15.154:6789/0 conn(0x7f8dd40cf160 legacy=0x7f8dd40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:28.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.890+0000 7f8dd9738640 1 -- 172.21.15.46:0/1254226328 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd40cad70 con 0x7f8dd40ca0e0 2023-10-30T14:25:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.894+0000 7f8dd9738640 1 --2- 172.21.15.46:0/1254226328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd40ca0e0 0x7f8dd415a090 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009870 tx=0x7f8dc401ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d421788e7269a53 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.894+0000 7f8dd8f37640 1 -- 172.21.15.46:0/1254226328 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc40a8a30 con 0x7f8dd40ca0e0 2023-10-30T14:25:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.894+0000 7f8dd8f37640 1 -- 172.21.15.46:0/1254226328 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc40a8b90 con 0x7f8dd40ca0e0 2023-10-30T14:25:28.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.894+0000 7f8dd8f37640 1 -- 172.21.15.46:0/1254226328 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc40b0440 con 0x7f8dd40ca0e0 2023-10-30T14:25:28.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.894+0000 7f8ddc19d640 1 -- 172.21.15.46:0/1254226328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd40ca0e0 msgr2=0x7f8dd415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:28.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.894+0000 7f8ddc19d640 1 --2- 172.21.15.46:0/1254226328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd40ca0e0 0x7f8dd415a090 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009870 tx=0x7f8dc401ed30 comp rx=0 tx=0).stop 2023-10-30T14:25:28.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.894+0000 7f8ddc19d640 1 -- 172.21.15.46:0/1254226328 shutdown_connections 2023-10-30T14:25:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.894+0000 7f8ddc19d640 1 --2- 172.21.15.46:0/1254226328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd40ca0e0 0x7f8dd415a090 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.894+0000 7f8ddc19d640 1 --2- 172.21.15.46:0/1254226328 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd41538a0 0x7f8dd415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.894+0000 7f8ddc19d640 1 -- 172.21.15.46:0/1254226328 >> 172.21.15.46:0/1254226328 conn(0x7f8dd406bfe0 msgr2=0x7f8dd406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.894+0000 7f8ddc19d640 1 -- 172.21.15.46:0/1254226328 shutdown_connections 2023-10-30T14:25:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.894+0000 7f8ddc19d640 1 -- 172.21.15.46:0/1254226328 wait complete. 2023-10-30T14:25:28.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8ddc19d640 1 Processor -- start 2023-10-30T14:25:28.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8ddc19d640 1 -- start start 2023-10-30T14:25:28.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8ddc19d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd40ca0e0 0x7f8dd41e5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:28.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8ddc19d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd40cf160 0x7f8dd41e4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8ddc19d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd41538a0 0x7f8dd41e3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8ddc19d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd41e2b70 con 0x7f8dd41538a0 2023-10-30T14:25:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8ddc19d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd40b36d0 con 0x7f8dd40cf160 2023-10-30T14:25:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8ddc19d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd40b39b0 con 0x7f8dd40ca0e0 2023-10-30T14:25:28.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dda73a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd41538a0 0x7f8dd41e3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dd9738640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd40cf160 0x7f8dd41e4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dd9f39640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd40ca0e0 0x7f8dd41e5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dda73a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd41538a0 0x7f8dd41e3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53256/0 (socket says 172.21.15.46:53256) 2023-10-30T14:25:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dda73a640 1 -- 172.21.15.46:0/3975130187 learned_addr learned my addr 172.21.15.46:0/3975130187 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dda73a640 1 -- 172.21.15.46:0/3975130187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd40ca0e0 msgr2=0x7f8dd41e5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dda73a640 1 --2- 172.21.15.46:0/3975130187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd40ca0e0 0x7f8dd41e5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dda73a640 1 -- 172.21.15.46:0/3975130187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd40cf160 msgr2=0x7f8dd41e4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dda73a640 1 --2- 172.21.15.46:0/3975130187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd40cf160 0x7f8dd41e4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dda73a640 1 -- 172.21.15.46:0/3975130187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc40094d0 con 0x7f8dd41538a0 2023-10-30T14:25:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dda73a640 1 --2- 172.21.15.46:0/3975130187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd41538a0 0x7f8dd41e3000 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00c730 tx=0x7f8dcc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dca7fc640 1 -- 172.21.15.46:0/3975130187 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc00ce40 con 0x7f8dd41538a0 2023-10-30T14:25:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dca7fc640 1 -- 172.21.15.46:0/3975130187 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dcc00a640 con 0x7f8dd41538a0 2023-10-30T14:25:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.898+0000 7f8dca7fc640 1 -- 172.21.15.46:0/3975130187 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc00a9b0 con 0x7f8dd41538a0 2023-10-30T14:25:28.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.902+0000 7f8ddc19d640 1 -- 172.21.15.46:0/3975130187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd40b3c90 con 0x7f8dd41538a0 2023-10-30T14:25:28.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.902+0000 7f8ddc19d640 1 -- 172.21.15.46:0/3975130187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd41e64c0 con 0x7f8dd41538a0 2023-10-30T14:25:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.902+0000 7f8ddc19d640 1 -- 172.21.15.46:0/3975130187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd40b2a50 con 0x7f8dd41538a0 2023-10-30T14:25:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.902+0000 7f8dca7fc640 1 -- 172.21.15.46:0/3975130187 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8dcc00ab10 con 0x7f8dd41538a0 2023-10-30T14:25:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.902+0000 7f8dca7fc640 1 --2- 172.21.15.46:0/3975130187 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8db80664a0 0x7f8db8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.902+0000 7f8dca7fc640 1 -- 172.21.15.46:0/3975130187 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f8dcc084110 con 0x7f8dd41538a0 2023-10-30T14:25:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.902+0000 7f8dd9f39640 1 --2- 172.21.15.46:0/3975130187 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8db80664a0 0x7f8db8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.906+0000 7f8dd9f39640 1 --2- 172.21.15.46:0/3975130187 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8db80664a0 0x7f8db8068960 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0009ac0 tx=0x7f8dd0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:28.906+0000 7f8dca7fc640 1 -- 172.21.15.46:0/3975130187 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8dcc050a80 con 0x7f8dd41538a0 2023-10-30T14:25:29.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 -- 172.21.15.46:0/3975130187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dd4066960 con 0x7f8dd41538a0 2023-10-30T14:25:29.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8dca7fc640 1 -- 172.21.15.46:0/3975130187 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+5743 (secure 0 0 0) 0x7f8dcc017020 con 0x7f8dd41538a0 2023-10-30T14:25:29.125 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:29.125 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:26.947574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:clientreplay","state_seq":18,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:29.126 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 264 2023-10-30T14:25:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 -- 172.21.15.46:0/3975130187 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8db80664a0 msgr2=0x7f8db8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 --2- 172.21.15.46:0/3975130187 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8db80664a0 0x7f8db8068960 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0009ac0 tx=0x7f8dd0009190 comp rx=0 tx=0).stop 2023-10-30T14:25:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 -- 172.21.15.46:0/3975130187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd41538a0 msgr2=0x7f8dd41e3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:29.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 --2- 172.21.15.46:0/3975130187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd41538a0 0x7f8dd41e3000 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00c730 tx=0x7f8dcc00cbf0 comp rx=0 tx=0).stop 2023-10-30T14:25:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 -- 172.21.15.46:0/3975130187 shutdown_connections 2023-10-30T14:25:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 --2- 172.21.15.46:0/3975130187 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8db80664a0 0x7f8db8068960 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 --2- 172.21.15.46:0/3975130187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dd41538a0 0x7f8dd41e3000 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 --2- 172.21.15.46:0/3975130187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd40cf160 0x7f8dd41e4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:29.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 --2- 172.21.15.46:0/3975130187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd40ca0e0 0x7f8dd41e5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:29.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 -- 172.21.15.46:0/3975130187 >> 172.21.15.46:0/3975130187 conn(0x7f8dd406bfe0 msgr2=0x7f8dd40cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:29.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 -- 172.21.15.46:0/3975130187 shutdown_connections 2023-10-30T14:25:29.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:29.122+0000 7f8ddc19d640 1 -- 172.21.15.46:0/3975130187 wait complete. 2023-10-30T14:25:29.146 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:25:31.147 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.286+0000 7fb24099d640 1 Processor -- start 2023-10-30T14:25:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.286+0000 7fb24099d640 1 -- start start 2023-10-30T14:25:31.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.286+0000 7fb24099d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb23c1538a0 0x7fb23c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:31.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.286+0000 7fb24099d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23c0cb800 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb24099d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23c15dc30 con 0x7fb23c0cf160 2023-10-30T14:25:31.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb24099d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c0cb330 con 0x7fb23c1538a0 2023-10-30T14:25:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb239d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb23c0ca0e0 0x7fb23c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38684/0 (socket says 172.21.15.46:38684) 2023-10-30T14:25:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb239d9b640 1 -- 172.21.15.46:0/159867101 learned_addr learned my addr 172.21.15.46:0/159867101 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:25:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb23a59c640 1 --2- 172.21.15.46:0/159867101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb23c1538a0 0x7fb23c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:31.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb23959a640 1 -- 172.21.15.46:0/159867101 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045403464 0 0) 0x7fb23c0cb800 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb23959a640 1 -- 172.21.15.46:0/159867101 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb224003530 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb23959a640 1 -- 172.21.15.46:0/159867101 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647285175 0 0) 0x7fb2340014a0 con 0x7fb23c0cf160 2023-10-30T14:25:31.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb23959a640 1 -- 172.21.15.46:0/159867101 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb23c0cb800 con 0x7fb23c0cf160 2023-10-30T14:25:31.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb23959a640 1 -- 172.21.15.46:0/159867101 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 832695551 0 0) 0x7fb224003530 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb23a59c640 1 -- 172.21.15.46:0/159867101 >> v1:172.21.15.154:6789/0 conn(0x7fb23c0cf160 legacy=0x7fb23c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:31.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb23a59c640 1 -- 172.21.15.46:0/159867101 >> v1:172.21.15.46:6789/0 conn(0x7fb23c0ca0e0 legacy=0x7fb23c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:31.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb23a59c640 1 -- 172.21.15.46:0/159867101 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c0cad70 con 0x7fb23c1538a0 2023-10-30T14:25:31.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.290+0000 7fb23a59c640 1 --2- 172.21.15.46:0/159867101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb23c1538a0 0x7fb23c15d6f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0097f0 tx=0x7fb22c01ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34578979eca7e441 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:31.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb23959a640 1 -- 172.21.15.46:0/159867101 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230001680 con 0x7fb23c1538a0 2023-10-30T14:25:31.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb23959a640 1 -- 172.21.15.46:0/159867101 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c0a8650 con 0x7fb23c1538a0 2023-10-30T14:25:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb23959a640 1 -- 172.21.15.46:0/159867101 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c0a89c0 con 0x7fb23c1538a0 2023-10-30T14:25:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 -- 172.21.15.46:0/159867101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb23c1538a0 msgr2=0x7fb23c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 --2- 172.21.15.46:0/159867101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb23c1538a0 0x7fb23c15d6f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0097f0 tx=0x7fb22c01ebd0 comp rx=0 tx=0).stop 2023-10-30T14:25:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 -- 172.21.15.46:0/159867101 shutdown_connections 2023-10-30T14:25:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 --2- 172.21.15.46:0/159867101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb23c1538a0 0x7fb23c15d6f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 -- 172.21.15.46:0/159867101 >> 172.21.15.46:0/159867101 conn(0x7fb23c06bfe0 msgr2=0x7fb23c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 -- 172.21.15.46:0/159867101 shutdown_connections 2023-10-30T14:25:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 -- 172.21.15.46:0/159867101 wait complete. 2023-10-30T14:25:31.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 Processor -- start 2023-10-30T14:25:31.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 -- start start 2023-10-30T14:25:31.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb23c0ca0e0 0x7fb23c158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:31.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0cf160 0x7fb23c1e6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:31.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb23c1538a0 0x7fb23c0b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:31.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c0b44d0 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c158880 con 0x7fb23c0cf160 2023-10-30T14:25:31.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.294+0000 7fb24099d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c1f0680 con 0x7fb23c1538a0 2023-10-30T14:25:31.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb23a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb23c0ca0e0 0x7fb23c158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:31.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb23a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb23c0ca0e0 0x7fb23c158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53268/0 (socket says 172.21.15.46:53268) 2023-10-30T14:25:31.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb23a59c640 1 -- 172.21.15.46:0/1248685094 learned_addr learned my addr 172.21.15.46:0/1248685094 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:31.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb239d9b640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0cf160 0x7fb23c1e6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:31.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb23ad9d640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb23c1538a0 0x7fb23c0b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:31.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb23a59c640 1 -- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb23c1538a0 msgr2=0x7fb23c0b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:31.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb23a59c640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb23c1538a0 0x7fb23c0b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:31.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb23a59c640 1 -- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0cf160 msgr2=0x7fb23c1e6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:31.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb23a59c640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0cf160 0x7fb23c1e6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:31.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb23a59c640 1 -- 172.21.15.46:0/1248685094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c009420 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb23a59c640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb23c0ca0e0 0x7fb23c158340 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fb22c01c800 tx=0x7fb22c01b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:31.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb22affd640 1 -- 172.21.15.46:0/1248685094 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c0a8bd0 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb22affd640 1 -- 172.21.15.46:0/1248685094 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c0a8d30 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb22affd640 1 -- 172.21.15.46:0/1248685094 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c0a8ea0 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb24099d640 1 -- 172.21.15.46:0/1248685094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23c0b35e0 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb24099d640 1 -- 172.21.15.46:0/1248685094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23c0b3ba0 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb24099d640 1 -- 172.21.15.46:0/1248685094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23c159850 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.298+0000 7fb22affd640 1 -- 172.21.15.46:0/1248685094 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb22c003820 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.302+0000 7fb22affd640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb21c0664a0 0x7fb21c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:31.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.302+0000 7fb22affd640 1 -- 172.21.15.46:0/1248685094 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fb22c122650 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.302+0000 7fb239d9b640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb21c0664a0 0x7fb21c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:31.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.302+0000 7fb239d9b640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb21c0664a0 0x7fb21c068960 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb23c0b5330 tx=0x7fb23000f5a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:31.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.306+0000 7fb22affd640 1 -- 172.21.15.46:0/1248685094 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb22c0eefe0 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.494+0000 7fb24099d640 1 -- 172.21.15.46:0/1248685094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb23c04e170 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.494+0000 7fb22affd640 1 -- 172.21.15.46:0/1248685094 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+5743 (secure 0 0 0) 0x7fb22c0f2600 con 0x7fb23c0ca0e0 2023-10-30T14:25:31.499 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:31.499 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:26.947574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:clientreplay","state_seq":18,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:31.500 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 264 2023-10-30T14:25:31.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.498+0000 7fb24099d640 1 -- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb21c0664a0 msgr2=0x7fb21c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:31.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.498+0000 7fb24099d640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb21c0664a0 0x7fb21c068960 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb23c0b5330 tx=0x7fb23000f5a0 comp rx=0 tx=0).stop 2023-10-30T14:25:31.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.498+0000 7fb24099d640 1 -- 172.21.15.46:0/1248685094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb23c0ca0e0 msgr2=0x7fb23c158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:31.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.498+0000 7fb24099d640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb23c0ca0e0 0x7fb23c158340 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fb22c01c800 tx=0x7fb22c01b970 comp rx=0 tx=0).stop 2023-10-30T14:25:31.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.498+0000 7fb24099d640 1 -- 172.21.15.46:0/1248685094 shutdown_connections 2023-10-30T14:25:31.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.498+0000 7fb24099d640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb21c0664a0 0x7fb21c068960 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:31.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.498+0000 7fb24099d640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb23c1538a0 0x7fb23c0b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.498+0000 7fb24099d640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0cf160 0x7fb23c1e6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.498+0000 7fb24099d640 1 --2- 172.21.15.46:0/1248685094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb23c0ca0e0 0x7fb23c158340 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.498+0000 7fb24099d640 1 -- 172.21.15.46:0/1248685094 >> 172.21.15.46:0/1248685094 conn(0x7fb23c06bfe0 msgr2=0x7fb23c151ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.498+0000 7fb24099d640 1 -- 172.21.15.46:0/1248685094 shutdown_connections 2023-10-30T14:25:31.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:31.498+0000 7fb24099d640 1 -- 172.21.15.46:0/1248685094 wait complete. 2023-10-30T14:25:31.519 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T14:25:33.521 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:33.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.662+0000 7efed2617640 1 Processor -- start 2023-10-30T14:25:33.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.662+0000 7efed2617640 1 -- start start 2023-10-30T14:25:33.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.662+0000 7efed2617640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efecc1538a0 0x7efecc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:33.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.662+0000 7efed2617640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efecc0ca0e0 0x7efecc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:33.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.662+0000 7efed2617640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efecc0cb800 con 0x7efecc0cf160 2023-10-30T14:25:33.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.662+0000 7efed2617640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efecc15dc30 con 0x7efecc0ca0e0 2023-10-30T14:25:33.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.662+0000 7efed2617640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efecc0cb330 con 0x7efecc1538a0 2023-10-30T14:25:33.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecbfff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efecc1538a0 0x7efecc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:33.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecbfff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efecc1538a0 0x7efecc15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:42706/0 (socket says 172.21.15.46:42706) 2023-10-30T14:25:33.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecbfff640 1 -- 172.21.15.46:0/3946812060 learned_addr learned my addr 172.21.15.46:0/3946812060 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:33.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecb7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efecc0ca0e0 0x7efecc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:33.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecaffd640 1 -- 172.21.15.46:0/3946812060 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480748204 0 0) 0x7efecc0cb800 con 0x7efecc0cf160 2023-10-30T14:25:33.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecaffd640 1 -- 172.21.15.46:0/3946812060 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb4003530 con 0x7efecc0cf160 2023-10-30T14:25:33.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecbfff640 1 -- 172.21.15.46:0/3946812060 >> v2:172.21.15.154:3300/0 conn(0x7efecc0ca0e0 msgr2=0x7efecc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:33.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecbfff640 1 --2- 172.21.15.46:0/3946812060 >> v2:172.21.15.154:3300/0 conn(0x7efecc0ca0e0 0x7efecc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:33.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecbfff640 1 -- 172.21.15.46:0/3946812060 >> v1:172.21.15.154:6789/0 conn(0x7efecc0cf160 legacy=0x7efecc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:33.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecbfff640 1 -- 172.21.15.46:0/3946812060 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc0cad70 con 0x7efecc1538a0 2023-10-30T14:25:33.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecb7fe640 1 --2- 172.21.15.46:0/3946812060 >> v2:172.21.15.154:3300/0 conn(0x7efecc0ca0e0 0x7efecc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:25:33.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecbfff640 1 --2- 172.21.15.46:0/3946812060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efecc1538a0 0x7efecc15d6f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efebc009870 tx=0x7efebc01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c3e9150ae73d9a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:33.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.666+0000 7efecaffd640 1 -- 172.21.15.46:0/3946812060 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efebc0a8a30 con 0x7efecc1538a0 2023-10-30T14:25:33.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.670+0000 7efecaffd640 1 -- 172.21.15.46:0/3946812060 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efebc0a8b90 con 0x7efecc1538a0 2023-10-30T14:25:33.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.670+0000 7efecaffd640 1 -- 172.21.15.46:0/3946812060 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efebc0b0440 con 0x7efecc1538a0 2023-10-30T14:25:33.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.670+0000 7efed2617640 1 -- 172.21.15.46:0/3946812060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efecc1538a0 msgr2=0x7efecc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:33.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.670+0000 7efed2617640 1 --2- 172.21.15.46:0/3946812060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efecc1538a0 0x7efecc15d6f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efebc009870 tx=0x7efebc01ed30 comp rx=0 tx=0).stop 2023-10-30T14:25:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.670+0000 7efed2617640 1 -- 172.21.15.46:0/3946812060 shutdown_connections 2023-10-30T14:25:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.670+0000 7efed2617640 1 --2- 172.21.15.46:0/3946812060 >> v2:172.21.15.154:3300/0 conn(0x7efecc0ca0e0 0x7efecc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.670+0000 7efed2617640 1 --2- 172.21.15.46:0/3946812060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efecc1538a0 0x7efecc15d6f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.670+0000 7efed2617640 1 -- 172.21.15.46:0/3946812060 >> 172.21.15.46:0/3946812060 conn(0x7efecc06bfe0 msgr2=0x7efecc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:33.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.670+0000 7efed2617640 1 -- 172.21.15.46:0/3946812060 shutdown_connections 2023-10-30T14:25:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.670+0000 7efed2617640 1 -- 172.21.15.46:0/3946812060 wait complete. 2023-10-30T14:25:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.670+0000 7efed2617640 1 Processor -- start 2023-10-30T14:25:33.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.670+0000 7efed2617640 1 -- start start 2023-10-30T14:25:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed2617640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efecc0ca0e0 0x7efecc0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:33.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed2617640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efecc0cf160 0x7efecc1eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed2617640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc1538a0 0x7efecc1448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed2617640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc0b51a0 con 0x7efecc1538a0 2023-10-30T14:25:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed2617640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc0c56b0 con 0x7efecc0cf160 2023-10-30T14:25:33.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed2617640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efecc1f2c40 con 0x7efecc0ca0e0 2023-10-30T14:25:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed0bb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc1538a0 0x7efecc1448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efecb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efecc0cf160 0x7efecc1eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed0bb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc1538a0 0x7efecc1448c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53272/0 (socket says 172.21.15.46:53272) 2023-10-30T14:25:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed0bb4640 1 -- 172.21.15.46:0/2720769178 learned_addr learned my addr 172.21.15.46:0/2720769178 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:33.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efecbfff640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efecc0ca0e0 0x7efecc0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed0bb4640 1 -- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efecc0ca0e0 msgr2=0x7efecc0b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed0bb4640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efecc0ca0e0 0x7efecc0b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed0bb4640 1 -- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efecc0cf160 msgr2=0x7efecc1eec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed0bb4640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efecc0cf160 0x7efecc1eec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:33.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed0bb4640 1 -- 172.21.15.46:0/2720769178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec0009420 con 0x7efecc1538a0 2023-10-30T14:25:33.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed0bb4640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc1538a0 0x7efecc1448c0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7efec400c760 tx=0x7efec400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:33.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efec8ff9640 1 -- 172.21.15.46:0/2720769178 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec4003b80 con 0x7efecc1538a0 2023-10-30T14:25:33.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efec8ff9640 1 -- 172.21.15.46:0/2720769178 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec4003ce0 con 0x7efecc1538a0 2023-10-30T14:25:33.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed2617640 1 -- 172.21.15.46:0/2720769178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efebc0094d0 con 0x7efecc1538a0 2023-10-30T14:25:33.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efec8ff9640 1 -- 172.21.15.46:0/2720769178 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec400aa90 con 0x7efecc1538a0 2023-10-30T14:25:33.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed2617640 1 -- 172.21.15.46:0/2720769178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efecc1f30d0 con 0x7efecc1538a0 2023-10-30T14:25:33.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.674+0000 7efed2617640 1 -- 172.21.15.46:0/2720769178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efecc0b2a50 con 0x7efecc1538a0 2023-10-30T14:25:33.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.678+0000 7efec8ff9640 1 -- 172.21.15.46:0/2720769178 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efec400abf0 con 0x7efecc1538a0 2023-10-30T14:25:33.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.678+0000 7efec8ff9640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efeb80664a0 0x7efeb8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:33.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.678+0000 7efec8ff9640 1 -- 172.21.15.46:0/2720769178 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7efec4084800 con 0x7efecc1538a0 2023-10-30T14:25:33.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.678+0000 7efecbfff640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efeb80664a0 0x7efeb8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:33.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.678+0000 7efecbfff640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efeb80664a0 0x7efeb8068960 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7efebc0b2000 tx=0x7efebc0aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:33.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.682+0000 7efec8ff9640 1 -- 172.21.15.46:0/2720769178 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efec4051050 con 0x7efecc1538a0 2023-10-30T14:25:33.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 -- 172.21.15.46:0/2720769178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efecc04e170 con 0x7efecc1538a0 2023-10-30T14:25:33.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efec8ff9640 1 -- 172.21.15.46:0/2720769178 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 265 v265) v1 ==== 95+0+5737 (secure 0 0 0) 0x7efec4054670 con 0x7efecc1538a0 2023-10-30T14:25:33.873 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:33.874 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":265,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":265,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:32.264566+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:33.874 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 265 2023-10-30T14:25:33.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 -- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efeb80664a0 msgr2=0x7efeb8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:33.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efeb80664a0 0x7efeb8068960 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7efebc0b2000 tx=0x7efebc0aa070 comp rx=0 tx=0).stop 2023-10-30T14:25:33.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 -- 172.21.15.46:0/2720769178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc1538a0 msgr2=0x7efecc1448c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:33.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc1538a0 0x7efecc1448c0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7efec400c760 tx=0x7efec400cc20 comp rx=0 tx=0).stop 2023-10-30T14:25:33.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 -- 172.21.15.46:0/2720769178 shutdown_connections 2023-10-30T14:25:33.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efeb80664a0 0x7efeb8068960 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:33.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efecc1538a0 0x7efecc1448c0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:33.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efecc0cf160 0x7efecc1eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:33.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 --2- 172.21.15.46:0/2720769178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efecc0ca0e0 0x7efecc0b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:33.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 -- 172.21.15.46:0/2720769178 >> 172.21.15.46:0/2720769178 conn(0x7efecc06bfe0 msgr2=0x7efecc0c4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:33.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 -- 172.21.15.46:0/2720769178 shutdown_connections 2023-10-30T14:25:33.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:33.870+0000 7efed2617640 1 -- 172.21.15.46:0/2720769178 wait complete. 2023-10-30T14:25:33.895 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=9735 2023-10-30T14:25:33.895 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.c 2023-10-30T14:25:33.896 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T14:25:33.896 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T14:25:33.896 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T14:25:33.900 INFO:tasks.ceph.mds.c:Started 2023-10-30T14:25:33.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:33.990 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-10-30T14:25:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.042+0000 7f0036a17640 1 Processor -- start 2023-10-30T14:25:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.042+0000 7f0036a17640 1 -- start start 2023-10-30T14:25:34.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0036a17640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f003015cd50 0x7f00300c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:34.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0036a17640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f00300cefa0 0x7f00300cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:34.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0036a17640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003015a6c0 con 0x7f00301538f0 2023-10-30T14:25:34.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0036a17640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00300cf8f0 con 0x7f003015cd50 2023-10-30T14:25:34.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0036a17640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003015d300 con 0x7f00300cefa0 2023-10-30T14:25:34.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0035214640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f003015cd50 0x7f00300c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:34.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0035214640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f003015cd50 0x7f00300c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53306/0 (socket says 172.21.15.46:53306) 2023-10-30T14:25:34.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0035a15640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f00301538f0 0x7f0030159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38708/0 (socket says 172.21.15.46:38708) 2023-10-30T14:25:34.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0035214640 1 -- 172.21.15.46:0/139706738 learned_addr learned my addr 172.21.15.46:0/139706738 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:34.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0036216640 1 --2- 172.21.15.46:0/139706738 >> v2:172.21.15.154:3300/0 conn(0x7f00300cefa0 0x7f00300cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:34.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0035214640 1 -- 172.21.15.46:0/139706738 >> v2:172.21.15.154:3300/0 conn(0x7f00300cefa0 msgr2=0x7f00300cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:25:34.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0035214640 1 --2- 172.21.15.46:0/139706738 >> v2:172.21.15.154:3300/0 conn(0x7f00300cefa0 0x7f00300cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:34.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0035214640 1 -- 172.21.15.46:0/139706738 >> v1:172.21.15.46:6789/0 conn(0x7f00301538f0 legacy=0x7f0030159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:34.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0035214640 1 -- 172.21.15.46:0/139706738 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003015d5e0 con 0x7f003015cd50 2023-10-30T14:25:34.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.046+0000 7f0036216640 1 --2- 172.21.15.46:0/139706738 >> v2:172.21.15.154:3300/0 conn(0x7f00300cefa0 0x7f00300cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:25:34.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0035214640 1 --2- 172.21.15.46:0/139706738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f003015cd50 0x7f00300c9f90 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f0030066a40 tx=0x7f002001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd9d8d73aef0b358 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:34.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0034a13640 1 -- 172.21.15.46:0/139706738 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00200a89f0 con 0x7f003015cd50 2023-10-30T14:25:34.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0034a13640 1 -- 172.21.15.46:0/139706738 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00200a8b50 con 0x7f003015cd50 2023-10-30T14:25:34.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0034a13640 1 -- 172.21.15.46:0/139706738 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00200b0440 con 0x7f003015cd50 2023-10-30T14:25:34.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 -- 172.21.15.46:0/139706738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f003015cd50 msgr2=0x7f00300c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:34.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 --2- 172.21.15.46:0/139706738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f003015cd50 0x7f00300c9f90 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f0030066a40 tx=0x7f002001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:25:34.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 -- 172.21.15.46:0/139706738 shutdown_connections 2023-10-30T14:25:34.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 --2- 172.21.15.46:0/139706738 >> v2:172.21.15.154:3300/0 conn(0x7f00300cefa0 0x7f00300cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:34.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 --2- 172.21.15.46:0/139706738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f003015cd50 0x7f00300c9f90 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:34.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 -- 172.21.15.46:0/139706738 >> 172.21.15.46:0/139706738 conn(0x7f003006a640 msgr2=0x7f0030065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:34.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 -- 172.21.15.46:0/139706738 shutdown_connections 2023-10-30T14:25:34.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 -- 172.21.15.46:0/139706738 wait complete. 2023-10-30T14:25:34.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 Processor -- start 2023-10-30T14:25:34.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 -- start start 2023-10-30T14:25:34.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00300cefa0 0x7f00300b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:34.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00301538f0 0x7f00300b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:34.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f003015cd50 0x7f003013e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:34.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00300b43f0 con 0x7f00300cefa0 2023-10-30T14:25:34.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.050+0000 7f0036a17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00300b5290 con 0x7f00301538f0 2023-10-30T14:25:34.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0036a17640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0030159770 con 0x7f003015cd50 2023-10-30T14:25:34.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0035a15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00300cefa0 0x7f00300b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:34.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0035a15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00300cefa0 0x7f00300b4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53310/0 (socket says 172.21.15.46:53310) 2023-10-30T14:25:34.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0035a15640 1 -- 172.21.15.46:0/2270363924 learned_addr learned my addr 172.21.15.46:0/2270363924 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:34.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0035214640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00301538f0 0x7f00300b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:34.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0036216640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f003015cd50 0x7f003013e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:34.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0035a15640 1 -- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f003015cd50 msgr2=0x7f003013e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:34.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0035a15640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f003015cd50 0x7f003013e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:34.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0035a15640 1 -- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00301538f0 msgr2=0x7f00300b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:34.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0035a15640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00301538f0 0x7f00300b3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0035a15640 1 -- 172.21.15.46:0/2270363924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00200094d0 con 0x7f00300cefa0 2023-10-30T14:25:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0035a15640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00300cefa0 0x7f00300b4ca0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f002c00c4c0 tx=0x7f002c00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f00267fc640 1 -- 172.21.15.46:0/2270363924 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c007480 con 0x7f00300cefa0 2023-10-30T14:25:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f00267fc640 1 -- 172.21.15.46:0/2270363924 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c0075e0 con 0x7f00300cefa0 2023-10-30T14:25:34.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0036a17640 1 -- 172.21.15.46:0/2270363924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00300c5560 con 0x7f00300cefa0 2023-10-30T14:25:34.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f00267fc640 1 -- 172.21.15.46:0/2270363924 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c004a10 con 0x7f00300cefa0 2023-10-30T14:25:34.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0036a17640 1 -- 172.21.15.46:0/2270363924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00300c5800 con 0x7f00300cefa0 2023-10-30T14:25:34.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f0036a17640 1 -- 172.21.15.46:0/2270363924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003004e170 con 0x7f00300cefa0 2023-10-30T14:25:34.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.054+0000 7f00267fc640 1 -- 172.21.15.46:0/2270363924 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f002c004b70 con 0x7f00300cefa0 2023-10-30T14:25:34.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.058+0000 7f00267fc640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00140664d0 0x7f0014068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:34.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.058+0000 7f00267fc640 1 -- 172.21.15.46:0/2270363924 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f002c083e30 con 0x7f00300cefa0 2023-10-30T14:25:34.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.058+0000 7f0035214640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00140664d0 0x7f0014068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:34.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.058+0000 7f0035214640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00140664d0 0x7f0014068990 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f00200b2000 tx=0x7f0020009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:34.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.062+0000 7f00267fc640 1 -- 172.21.15.46:0/2270363924 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f002c0507a0 con 0x7f00300cefa0 2023-10-30T14:25:34.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.250+0000 7f0036a17640 1 -- 172.21.15.46:0/2270363924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00300b2290 con 0x7f00300cefa0 2023-10-30T14:25:34.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.250+0000 7f00267fc640 1 -- 172.21.15.46:0/2270363924 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 265 v265) v1 ==== 95+0+5737 (secure 0 0 0) 0x7f002c018020 con 0x7f00300cefa0 2023-10-30T14:25:34.254 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:34.255 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":265,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":265,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:32.264566+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:34.255 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 265 2023-10-30T14:25:34.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.254+0000 7f0036a17640 1 -- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00140664d0 msgr2=0x7f0014068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:34.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.254+0000 7f0036a17640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00140664d0 0x7f0014068990 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f00200b2000 tx=0x7f0020009040 comp rx=0 tx=0).stop 2023-10-30T14:25:34.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.254+0000 7f0036a17640 1 -- 172.21.15.46:0/2270363924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00300cefa0 msgr2=0x7f00300b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:34.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.254+0000 7f0036a17640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00300cefa0 0x7f00300b4ca0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f002c00c4c0 tx=0x7f002c00c980 comp rx=0 tx=0).stop 2023-10-30T14:25:34.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.254+0000 7f0036a17640 1 -- 172.21.15.46:0/2270363924 shutdown_connections 2023-10-30T14:25:34.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.254+0000 7f0036a17640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f00140664d0 0x7f0014068990 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:34.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.254+0000 7f0036a17640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f003015cd50 0x7f003013e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:34.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.254+0000 7f0036a17640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00301538f0 0x7f00300b3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:34.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.254+0000 7f0036a17640 1 --2- 172.21.15.46:0/2270363924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00300cefa0 0x7f00300b4ca0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:34.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.254+0000 7f0036a17640 1 -- 172.21.15.46:0/2270363924 >> 172.21.15.46:0/2270363924 conn(0x7f003006a640 msgr2=0x7f00300cd560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:34.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.254+0000 7f0036a17640 1 -- 172.21.15.46:0/2270363924 shutdown_connections 2023-10-30T14:25:34.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:34.254+0000 7f0036a17640 1 -- 172.21.15.46:0/2270363924 wait complete. 2023-10-30T14:25:34.275 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T14:25:36.276 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:25:36.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f615ecd6640 1 Processor -- start 2023-10-30T14:25:36.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f615ecd6640 1 -- start start 2023-10-30T14:25:36.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f615ecd6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f61580c6bd0 0x7f615806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:36.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f615ecd6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6158066cf0 con 0x7f6158071110 2023-10-30T14:25:36.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f615ecd6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61580c70d0 con 0x7f615806f7a0 2023-10-30T14:25:36.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f615ecd6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61580700c0 con 0x7f61580c6bd0 2023-10-30T14:25:36.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f6157fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f61580c6bd0 0x7f615806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:36.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f6157fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f61580c6bd0 0x7f615806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48020/0 (socket says 172.21.15.46:48020) 2023-10-30T14:25:36.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f6157fff640 1 -- 172.21.15.46:0/2071007095 learned_addr learned my addr 172.21.15.46:0/2071007095 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f6157fff640 1 -- 172.21.15.46:0/2071007095 >> v1:172.21.15.154:6789/0 conn(0x7f615806f7a0 legacy=0x7f6158069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f6157fff640 1 -- 172.21.15.46:0/2071007095 >> v1:172.21.15.46:6789/0 conn(0x7f6158071110 legacy=0x7f6158158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f6157fff640 1 -- 172.21.15.46:0/2071007095 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61580c7390 con 0x7f61580c6bd0 2023-10-30T14:25:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.422+0000 7f61577fe640 1 -- 172.21.15.46:0/2071007095 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508135630 0 0) 0x7f6158066cf0 con 0x7f6158071110 2023-10-30T14:25:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.426+0000 7f6157fff640 1 --2- 172.21.15.46:0/2071007095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61580c6bd0 0x7f615806fb80 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f6144009870 tx=0x7f614401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c53ad73a2158d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.426+0000 7f61577fe640 1 -- 172.21.15.46:0/2071007095 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61440a8a30 con 0x7f61580c6bd0 2023-10-30T14:25:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.426+0000 7f61577fe640 1 -- 172.21.15.46:0/2071007095 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61440a8b90 con 0x7f61580c6bd0 2023-10-30T14:25:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.426+0000 7f61577fe640 1 -- 172.21.15.46:0/2071007095 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61440b0440 con 0x7f61580c6bd0 2023-10-30T14:25:36.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.426+0000 7f615ecd6640 1 -- 172.21.15.46:0/2071007095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61580c6bd0 msgr2=0x7f615806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:36.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.426+0000 7f615ecd6640 1 --2- 172.21.15.46:0/2071007095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61580c6bd0 0x7f615806fb80 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f6144009870 tx=0x7f614401ed30 comp rx=0 tx=0).stop 2023-10-30T14:25:36.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.426+0000 7f615ecd6640 1 -- 172.21.15.46:0/2071007095 shutdown_connections 2023-10-30T14:25:36.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.426+0000 7f615ecd6640 1 --2- 172.21.15.46:0/2071007095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61580c6bd0 0x7f615806fb80 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:36.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.426+0000 7f615ecd6640 1 -- 172.21.15.46:0/2071007095 >> 172.21.15.46:0/2071007095 conn(0x7f61580c7910 msgr2=0x7f61580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:36.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.426+0000 7f615ecd6640 1 -- 172.21.15.46:0/2071007095 shutdown_connections 2023-10-30T14:25:36.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.426+0000 7f615ecd6640 1 -- 172.21.15.46:0/2071007095 wait complete. 2023-10-30T14:25:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f615ecd6640 1 Processor -- start 2023-10-30T14:25:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f615ecd6640 1 -- start start 2023-10-30T14:25:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f615ecd6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615806f7a0 0x7f6158154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f615ecd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158071110 0x7f61580c2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f615ecd6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61580c6bd0 0x7f61580c2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f615ecd6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61581584c0 con 0x7f6158071110 2023-10-30T14:25:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f615ecd6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61580ad960 con 0x7f615806f7a0 2023-10-30T14:25:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f615ecd6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6158157c60 con 0x7f61580c6bd0 2023-10-30T14:25:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f6157fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158071110 0x7f61580c2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f6157fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158071110 0x7f61580c2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48036/0 (socket says 172.21.15.46:48036) 2023-10-30T14:25:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f6157fff640 1 -- 172.21.15.46:0/2288895840 learned_addr learned my addr 172.21.15.46:0/2288895840 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:25:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f615ca72640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615806f7a0 0x7f6158154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f615d273640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61580c6bd0 0x7f61580c2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f6157fff640 1 -- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61580c6bd0 msgr2=0x7f61580c2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f6157fff640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61580c6bd0 0x7f61580c2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f6157fff640 1 -- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615806f7a0 msgr2=0x7f6158154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f6157fff640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615806f7a0 0x7f6158154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f6157fff640 1 -- 172.21.15.46:0/2288895840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61440094d0 con 0x7f6158071110 2023-10-30T14:25:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f6157fff640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158071110 0x7f61580c2640 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f61440b3040 tx=0x7f614401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f61557fa640 1 -- 172.21.15.46:0/2288895840 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61440b9620 con 0x7f6158071110 2023-10-30T14:25:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f61557fa640 1 -- 172.21.15.46:0/2288895840 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61440b97c0 con 0x7f6158071110 2023-10-30T14:25:36.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f615ecd6640 1 -- 172.21.15.46:0/2288895840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61581efd20 con 0x7f6158071110 2023-10-30T14:25:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f61557fa640 1 -- 172.21.15.46:0/2288895840 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61440b9b70 con 0x7f6158071110 2023-10-30T14:25:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.430+0000 7f615ecd6640 1 -- 172.21.15.46:0/2288895840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61581581e0 con 0x7f6158071110 2023-10-30T14:25:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.434+0000 7f615ecd6640 1 -- 172.21.15.46:0/2288895840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615804e170 con 0x7f6158071110 2023-10-30T14:25:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.434+0000 7f61557fa640 1 -- 172.21.15.46:0/2288895840 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f61440b7070 con 0x7f6158071110 2023-10-30T14:25:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.434+0000 7f61557fa640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f613c0664a0 0x7f613c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:25:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.434+0000 7f61557fa640 1 -- 172.21.15.46:0/2288895840 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f6144122a60 con 0x7f6158071110 2023-10-30T14:25:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.434+0000 7f615ca72640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f613c0664a0 0x7f613c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:25:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.434+0000 7f615ca72640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f613c0664a0 0x7f613c068960 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f614c007fd0 tx=0x7f614c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:25:36.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.438+0000 7f61557fa640 1 -- 172.21.15.46:0/2288895840 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f61440ef240 con 0x7f6158071110 2023-10-30T14:25:36.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 -- 172.21.15.46:0/2288895840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61580c1c80 con 0x7f6158071110 2023-10-30T14:25:36.633 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:25:36.633 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":266,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":10158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3415299388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3415299388},{"type":"v1","addr":"172.21.15.46:6836","nonce":3415299388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":265,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:32.264566+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:25:36.634 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 266 2023-10-30T14:25:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f61557fa640 1 -- 172.21.15.46:0/2288895840 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 266 v266) v1 ==== 95+0+6530 (secure 0 0 0) 0x7f61440b5030 con 0x7f6158071110 2023-10-30T14:25:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 -- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f613c0664a0 msgr2=0x7f613c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f613c0664a0 0x7f613c068960 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f614c007fd0 tx=0x7f614c009040 comp rx=0 tx=0).stop 2023-10-30T14:25:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 -- 172.21.15.46:0/2288895840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158071110 msgr2=0x7f61580c2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:25:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158071110 0x7f61580c2640 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f61440b3040 tx=0x7f614401bad0 comp rx=0 tx=0).stop 2023-10-30T14:25:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 -- 172.21.15.46:0/2288895840 shutdown_connections 2023-10-30T14:25:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f613c0664a0 0x7f613c068960 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61580c6bd0 0x7f61580c2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6158071110 0x7f61580c2640 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 --2- 172.21.15.46:0/2288895840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615806f7a0 0x7f6158154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:25:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 -- 172.21.15.46:0/2288895840 >> 172.21.15.46:0/2288895840 conn(0x7f61580c7910 msgr2=0x7f61580c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:25:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 -- 172.21.15.46:0/2288895840 shutdown_connections 2023-10-30T14:25:36.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:25:36.630+0000 7f615ecd6640 1 -- 172.21.15.46:0/2288895840 wait complete. 2023-10-30T14:25:36.780 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T14:25:36.780 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 89 secs before thrashing 2023-10-30T14:25:38.159 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T14:25:38.154+0000 7f12bd6bb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T14:25:38.159 INFO:tasks.ceph.mds.e.smithi046.stderr:2023-10-30T14:25:38.154+0000 7f12bd6bb640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T14:26:16.111 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-hardlink.sh... 2023-10-30T14:26:16.112 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-hardlink.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-hardlink.sh 2023-10-30T14:26:18.758 INFO:tasks.workunit.client.0.smithi046.stderr:+ set -e 2023-10-30T14:26:18.759 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir 1 2 2023-10-30T14:26:18.763 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo asdf 2023-10-30T14:26:18.763 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo asdf 2023-10-30T14:26:18.764 INFO:tasks.workunit.client.0.smithi046.stderr:+ ln 1/file1 2/file1 2023-10-30T14:26:18.774 INFO:tasks.workunit.client.0.smithi046.stderr:+ ln 1/file2 2/file2 2023-10-30T14:26:18.776 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir 2/.snap/s1 2023-10-30T14:26:18.806 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo qwer 2023-10-30T14:26:18.811 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep asdf 2/.snap/s1/file1 2023-10-30T14:26:18.832 INFO:tasks.workunit.client.0.smithi046.stdout:asdf 2023-10-30T14:26:18.832 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm -f 1/file2 2023-10-30T14:26:18.834 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep asdf 2/.snap/s1/file2 2023-10-30T14:26:18.859 INFO:tasks.workunit.client.0.smithi046.stdout:asdf 2023-10-30T14:26:18.859 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm -f 2/file2 2023-10-30T14:26:18.864 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep asdf 2/.snap/s1/file2 2023-10-30T14:26:18.864 INFO:tasks.workunit.client.0.smithi046.stdout:asdf 2023-10-30T14:26:18.864 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir 2/.snap/s1 2023-10-30T14:26:18.880 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm -rf 1 2 2023-10-30T14:26:18.925 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo OK 2023-10-30T14:26:18.927 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T14:26:18.927 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:26:18.927 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:26:18.962 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-intodir.sh... 2023-10-30T14:26:18.963 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-intodir.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-intodir.sh 2023-10-30T14:26:22.925 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir foo 2023-10-30T14:26:22.927 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir foo/.snap/one 2023-10-30T14:26:22.940 INFO:tasks.workunit.client.0.smithi046.stderr:+ touch bar 2023-10-30T14:26:22.952 INFO:tasks.workunit.client.0.smithi046.stderr:+ mv bar foo 2023-10-30T14:26:22.964 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T14:26:23.726 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir baz 2023-10-30T14:26:23.728 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir baz/.snap/two 2023-10-30T14:26:23.736 INFO:tasks.workunit.client.0.smithi046.stderr:+ mv baz foo 2023-10-30T14:26:23.740 INFO:tasks.workunit.client.0.smithi046.stderr:+ sync 2023-10-30T14:26:24.023 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir foo/baz/.snap/two 2023-10-30T14:26:24.033 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir foo/.snap/one 2023-10-30T14:26:24.045 INFO:tasks.workunit.client.0.smithi046.stderr:+ rm -r foo 2023-10-30T14:26:24.063 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo OK 2023-10-30T14:26:24.064 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T14:26:24.064 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:26:24.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:26:24.098 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-multiple-capsnaps.sh... 2023-10-30T14:26:24.098 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-multiple-capsnaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh 2023-10-30T14:26:27.925 INFO:tasks.workunit.client.0.smithi046.stderr:+ set -e 2023-10-30T14:26:27.925 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo asdf 2023-10-30T14:26:27.925 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/1 2023-10-30T14:26:27.943 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 777 a 2023-10-30T14:26:27.980 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/2 2023-10-30T14:26:28.343 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo qwer 2023-10-30T14:26:28.391 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/3 2023-10-30T14:26:28.405 INFO:tasks.workunit.client.0.smithi046.stderr:+ chmod 666 a 2023-10-30T14:26:28.406 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/4 2023-10-30T14:26:28.451 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo zxcv 2023-10-30T14:26:28.458 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir .snap/5 2023-10-30T14:26:28.930 INFO:tasks.workunit.client.0.smithi046.stderr:+ ls -al .snap/1/a .snap/2/a .snap/3/a .snap/4/a .snap/5/a 2023-10-30T14:26:28.947 INFO:tasks.workunit.client.0.smithi046.stdout:-rw-rw-r-- 1 ubuntu ubuntu 5 Oct 30 14:26 .snap/1/a 2023-10-30T14:26:28.947 INFO:tasks.workunit.client.0.smithi046.stdout:-rwxrwxrwx 1 ubuntu ubuntu 5 Oct 30 14:26 .snap/2/a 2023-10-30T14:26:28.947 INFO:tasks.workunit.client.0.smithi046.stdout:-rwxrwxrwx 1 ubuntu ubuntu 5 Oct 30 14:26 .snap/3/a 2023-10-30T14:26:28.947 INFO:tasks.workunit.client.0.smithi046.stdout:-rw-rw-rw- 1 ubuntu ubuntu 5 Oct 30 14:26 .snap/4/a 2023-10-30T14:26:28.948 INFO:tasks.workunit.client.0.smithi046.stdout:-rw-rw-rw- 1 ubuntu ubuntu 5 Oct 30 14:26 .snap/5/a 2023-10-30T14:26:28.948 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep asdf .snap/1/a 2023-10-30T14:26:28.956 INFO:tasks.workunit.client.0.smithi046.stdout:asdf 2023-10-30T14:26:28.956 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/1/a 2023-10-30T14:26:28.957 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep Size: 5 2023-10-30T14:26:28.965 INFO:tasks.workunit.client.0.smithi046.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-10-30T14:26:28.966 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep asdf .snap/2/a 2023-10-30T14:26:28.975 INFO:tasks.workunit.client.0.smithi046.stdout:asdf 2023-10-30T14:26:28.975 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/2/a 2023-10-30T14:26:28.976 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep Size: 5 2023-10-30T14:26:28.976 INFO:tasks.workunit.client.0.smithi046.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-10-30T14:26:28.978 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/2/a 2023-10-30T14:26:28.980 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep -- -rwxrwxrwx 2023-10-30T14:26:28.982 INFO:tasks.workunit.client.0.smithi046.stdout:Access: (0777/-rwxrwxrwx) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2023-10-30T14:26:28.983 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep qwer .snap/3/a 2023-10-30T14:26:28.996 INFO:tasks.workunit.client.0.smithi046.stdout:qwer 2023-10-30T14:26:28.997 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/3/a 2023-10-30T14:26:29.001 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep Size: 5 2023-10-30T14:26:29.003 INFO:tasks.workunit.client.0.smithi046.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-10-30T14:26:29.004 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/3/a 2023-10-30T14:26:29.004 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep -- -rwxrwxrwx 2023-10-30T14:26:29.011 INFO:tasks.workunit.client.0.smithi046.stdout:Access: (0777/-rwxrwxrwx) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2023-10-30T14:26:29.011 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep qwer .snap/4/a 2023-10-30T14:26:29.015 INFO:tasks.workunit.client.0.smithi046.stdout:qwer 2023-10-30T14:26:29.016 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/4/a 2023-10-30T14:26:29.019 INFO:tasks.workunit.client.0.smithi046.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-10-30T14:26:29.019 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep Size: 5 2023-10-30T14:26:29.020 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/4/a 2023-10-30T14:26:29.024 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep -- -rw-rw-rw- 2023-10-30T14:26:29.026 INFO:tasks.workunit.client.0.smithi046.stdout:Access: (0666/-rw-rw-rw-) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2023-10-30T14:26:29.026 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep zxcv .snap/5/a 2023-10-30T14:26:29.030 INFO:tasks.workunit.client.0.smithi046.stdout:zxcv 2023-10-30T14:26:29.032 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep Size: 5 2023-10-30T14:26:29.036 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/5/a 2023-10-30T14:26:29.039 INFO:tasks.workunit.client.0.smithi046.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-10-30T14:26:29.043 INFO:tasks.workunit.client.0.smithi046.stdout:Access: (0666/-rw-rw-rw-) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2023-10-30T14:26:29.043 INFO:tasks.workunit.client.0.smithi046.stderr:+ stat .snap/5/a 2023-10-30T14:26:29.043 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep -- -rw-rw-rw- 2023-10-30T14:26:29.048 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir .snap/1 .snap/2 .snap/3 .snap/4 .snap/5 2023-10-30T14:26:29.120 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo OK 2023-10-30T14:26:29.121 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T14:26:29.122 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:26:29.122 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:26:29.173 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-name-limits.sh... 2023-10-30T14:26:29.173 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-name-limits.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-name-limits.sh 2023-10-30T14:26:29.617 INFO:tasks.workunit.client.0.smithi046.stderr:rmdir: failed to remove 'd1/.snap/*': No such file or directory 2023-10-30T14:26:29.620 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T14:26:29.621 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:26:29.622 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:26:29.660 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-parents.sh... 2023-10-30T14:26:29.661 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-parents.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-parents.sh 2023-10-30T14:26:29.720 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo making directory tree and files 2023-10-30T14:26:29.721 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir -p 1/a/b/c/ 2023-10-30T14:26:29.721 INFO:tasks.workunit.client.0.smithi046.stdout:making directory tree and files 2023-10-30T14:26:29.725 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo i'm file1 2023-10-30T14:26:29.726 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo i'm file2 2023-10-30T14:26:29.726 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo i'm file3 2023-10-30T14:26:29.727 INFO:tasks.workunit.client.0.smithi046.stdout:snapshotting 2023-10-30T14:26:29.727 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo snapshotting 2023-10-30T14:26:29.727 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir 1/.snap/foosnap1 2023-10-30T14:26:29.750 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir 2 2023-10-30T14:26:29.753 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo moving tree 2023-10-30T14:26:29.753 INFO:tasks.workunit.client.0.smithi046.stderr:+ mv 1/a 2 2023-10-30T14:26:29.753 INFO:tasks.workunit.client.0.smithi046.stdout:moving tree 2023-10-30T14:26:29.758 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo checking snapshot contains tree... 2023-10-30T14:26:29.758 INFO:tasks.workunit.client.0.smithi046.stdout:checking snapshot contains tree... 2023-10-30T14:26:29.759 INFO:tasks.workunit.client.0.smithi046.stderr:+ find 1/.snap/foosnap1 2023-10-30T14:26:29.759 INFO:tasks.workunit.client.0.smithi046.stderr:+ wc -w 2023-10-30T14:26:29.785 INFO:tasks.workunit.client.0.smithi046.stderr:+ dir1=7 2023-10-30T14:26:29.786 INFO:tasks.workunit.client.0.smithi046.stderr:+ find 2/ 2023-10-30T14:26:29.786 INFO:tasks.workunit.client.0.smithi046.stderr:+ wc -w 2023-10-30T14:26:29.793 INFO:tasks.workunit.client.0.smithi046.stderr:+ dir2=7 2023-10-30T14:26:29.793 INFO:tasks.workunit.client.0.smithi046.stdout:Success! 2023-10-30T14:26:29.793 INFO:tasks.workunit.client.0.smithi046.stderr:+ test 7==7 2023-10-30T14:26:29.794 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo Success! 2023-10-30T14:26:29.794 INFO:tasks.workunit.client.0.smithi046.stdout:adding folder and file to tree... 2023-10-30T14:26:29.794 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo adding folder and file to tree... 2023-10-30T14:26:29.795 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir 2/a/b/c/d 2023-10-30T14:26:29.796 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo i'm file 4! 2023-10-30T14:26:29.797 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo snapshotting tree 2 2023-10-30T14:26:29.797 INFO:tasks.workunit.client.0.smithi046.stdout:snapshotting tree 2 2023-10-30T14:26:29.797 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir 2/.snap/barsnap2 2023-10-30T14:26:29.815 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo comparing snapshots 2023-10-30T14:26:29.815 INFO:tasks.workunit.client.0.smithi046.stdout:comparing snapshots 2023-10-30T14:26:29.817 INFO:tasks.workunit.client.0.smithi046.stderr:+ find 1/.snap/foosnap1/ -maxdepth 2 2023-10-30T14:26:29.817 INFO:tasks.workunit.client.0.smithi046.stderr:+ wc -w 2023-10-30T14:26:29.824 INFO:tasks.workunit.client.0.smithi046.stderr:+ dir1=4 2023-10-30T14:26:29.825 INFO:tasks.workunit.client.0.smithi046.stderr:+ find 2/.snap/barsnap2/ -maxdepth 2 2023-10-30T14:26:29.825 INFO:tasks.workunit.client.0.smithi046.stderr:+ wc -w 2023-10-30T14:26:30.586 INFO:tasks.workunit.client.0.smithi046.stderr:+ dir2=4 2023-10-30T14:26:30.586 INFO:tasks.workunit.client.0.smithi046.stderr:+ test 4==4 2023-10-30T14:26:30.587 INFO:tasks.workunit.client.0.smithi046.stdout:Success! 2023-10-30T14:26:30.587 INFO:tasks.workunit.client.0.smithi046.stdout:moving subtree to first folder 2023-10-30T14:26:30.587 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo Success! 2023-10-30T14:26:30.588 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo moving subtree to first folder 2023-10-30T14:26:30.588 INFO:tasks.workunit.client.0.smithi046.stderr:+ mv 2/a/b/c 1 2023-10-30T14:26:30.590 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo comparing snapshots and new tree 2023-10-30T14:26:30.590 INFO:tasks.workunit.client.0.smithi046.stdout:comparing snapshots and new tree 2023-10-30T14:26:30.591 INFO:tasks.workunit.client.0.smithi046.stderr:+ find 1/ 2023-10-30T14:26:30.592 INFO:tasks.workunit.client.0.smithi046.stderr:+ wc -w 2023-10-30T14:26:31.295 INFO:tasks.workunit.client.0.smithi046.stderr:+ dir1=5 2023-10-30T14:26:31.299 INFO:tasks.workunit.client.0.smithi046.stderr:+ find 2/.snap/barsnap2/a/b/c 2023-10-30T14:26:31.300 INFO:tasks.workunit.client.0.smithi046.stderr:+ wc -w 2023-10-30T14:26:31.932 INFO:tasks.workunit.client.0.smithi046.stdout:Success! 2023-10-30T14:26:31.933 INFO:tasks.workunit.client.0.smithi046.stderr:+ dir2=4 2023-10-30T14:26:31.933 INFO:tasks.workunit.client.0.smithi046.stderr:+ test 5==4 2023-10-30T14:26:31.933 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo Success! 2023-10-30T14:26:31.934 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir 1/.snap/foosnap1 2023-10-30T14:26:31.964 INFO:tasks.workunit.client.0.smithi046.stderr:+ rmdir 2/.snap/barsnap2 2023-10-30T14:26:31.984 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo OK 2023-10-30T14:26:31.984 INFO:tasks.workunit.client.0.smithi046.stdout:OK 2023-10-30T14:26:31.988 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T14:26:31.988 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T14:26:32.307 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-realm-split.sh... 2023-10-30T14:26:32.308 DEBUG:teuthology.orchestra.run.smithi046:workunit test fs/snaps/snaptest-realm-split.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-split.sh 2023-10-30T14:26:32.929 INFO:tasks.workunit.client.0.smithi046.stderr:+ set -e 2023-10-30T14:26:32.930 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir -p 1/a 2023-10-30T14:26:32.934 INFO:tasks.workunit.client.0.smithi046.stderr:+ exec 2023-10-30T14:26:32.935 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo -n a 2023-10-30T14:26:32.935 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir 1/.snap/s1 2023-10-30T14:26:33.062 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo -n b 2023-10-30T14:26:33.070 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir 2 2023-10-30T14:26:33.074 INFO:tasks.workunit.client.0.smithi046.stderr:+ mv 1/a 2 2023-10-30T14:26:33.082 INFO:tasks.workunit.client.0.smithi046.stderr:+ mkdir 2/.snap/s2 2023-10-30T14:26:33.122 INFO:tasks.workunit.client.0.smithi046.stderr:+ echo -n c 2023-10-30T14:26:33.513 INFO:tasks.workunit.client.0.smithi046.stderr:+ exec 2023-10-30T14:26:33.514 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep ^a$ 1/.snap/s1/a/file1 2023-10-30T14:26:33.521 INFO:tasks.workunit.client.0.smithi046.stdout:a 2023-10-30T14:26:33.521 INFO:tasks.workunit.client.0.smithi046.stderr:+ grep ^ab$ 2/.snap/s2/a/file1 2023-10-30T14:26:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:26:33.770 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-10-30T14:26:33.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-30T14:26:34.774 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on smithi046 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-split.sh' 2023-10-30T14:26:34.838 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2d97ed70c57841a19ce6f9750f8134ee Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on smithi046 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-split.sh' 2023-10-30T14:26:34.841 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-10-30T14:26:34.851 INFO:tasks.mds_thrash:joining mds_thrasher 2023-10-30T14:26:34.853 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 1 2023-10-30T14:26:34.854 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2023-10-30T14:26:34.854 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 34 2023-10-30T14:26:34.855 INFO:tasks.mds_thrash:done joining 2023-10-30T14:26:34.855 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-30T14:26:34.873 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-30T14:26:34.874 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T14:26:34.874 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T14:26:34.913 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-10-30T14:26:34.913 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T14:26:34.964 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:26:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.150+0000 7f4a8753e640 1 Processor -- start 2023-10-30T14:26:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.150+0000 7f4a8753e640 1 -- start start 2023-10-30T14:26:35.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a8753e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4a801538a0 0x7f4a8015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a8753e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4a800ca0e0 0x7f4a8015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a8753e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a800cb800 con 0x7f4a800cf160 2023-10-30T14:26:35.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a8753e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a8015dc30 con 0x7f4a800ca0e0 2023-10-30T14:26:35.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a8753e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a800cb330 con 0x7f4a801538a0 2023-10-30T14:26:35.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a84ad9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4a800ca0e0 0x7f4a8015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a84ad9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4a800ca0e0 0x7f4a8015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56576/0 (socket says 172.21.15.46:56576) 2023-10-30T14:26:35.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a84ad9640 1 -- 172.21.15.46:0/2355245510 learned_addr learned my addr 172.21.15.46:0/2355245510 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:35.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a852da640 1 --2- 172.21.15.46:0/2355245510 >> v2:172.21.15.154:3300/0 conn(0x7f4a801538a0 0x7f4a8015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a84ad9640 1 -- 172.21.15.46:0/2355245510 >> v2:172.21.15.154:3300/0 conn(0x7f4a801538a0 msgr2=0x7f4a8015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:35.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a84ad9640 1 --2- 172.21.15.46:0/2355245510 >> v2:172.21.15.154:3300/0 conn(0x7f4a801538a0 0x7f4a8015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a84ad9640 1 -- 172.21.15.46:0/2355245510 >> v1:172.21.15.154:6789/0 conn(0x7f4a800cf160 legacy=0x7f4a800cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a84ad9640 1 -- 172.21.15.46:0/2355245510 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a800cad70 con 0x7f4a800ca0e0 2023-10-30T14:26:35.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.154+0000 7f4a852da640 1 --2- 172.21.15.46:0/2355245510 >> v2:172.21.15.154:3300/0 conn(0x7f4a801538a0 0x7f4a8015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:35.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a84ad9640 1 --2- 172.21.15.46:0/2355245510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a800ca0e0 0x7f4a8015a090 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f4a800cfd10 tx=0x7f4a7001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d554ee6da0925f77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:35.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a77fff640 1 -- 172.21.15.46:0/2355245510 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a700a89f0 con 0x7f4a800ca0e0 2023-10-30T14:26:35.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a77fff640 1 -- 172.21.15.46:0/2355245510 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a700a8b50 con 0x7f4a800ca0e0 2023-10-30T14:26:35.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a77fff640 1 -- 172.21.15.46:0/2355245510 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a700b0440 con 0x7f4a800ca0e0 2023-10-30T14:26:35.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a8753e640 1 -- 172.21.15.46:0/2355245510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a800ca0e0 msgr2=0x7f4a8015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a8753e640 1 --2- 172.21.15.46:0/2355245510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a800ca0e0 0x7f4a8015a090 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f4a800cfd10 tx=0x7f4a7001ecf0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a8753e640 1 -- 172.21.15.46:0/2355245510 shutdown_connections 2023-10-30T14:26:35.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a8753e640 1 --2- 172.21.15.46:0/2355245510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a800ca0e0 0x7f4a8015a090 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a8753e640 1 --2- 172.21.15.46:0/2355245510 >> v2:172.21.15.154:3300/0 conn(0x7f4a801538a0 0x7f4a8015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a8753e640 1 -- 172.21.15.46:0/2355245510 >> 172.21.15.46:0/2355245510 conn(0x7f4a8006bfe0 msgr2=0x7f4a8006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:35.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a8753e640 1 -- 172.21.15.46:0/2355245510 shutdown_connections 2023-10-30T14:26:35.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a8753e640 1 -- 172.21.15.46:0/2355245510 wait complete. 2023-10-30T14:26:35.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a8753e640 1 Processor -- start 2023-10-30T14:26:35.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.158+0000 7f4a8753e640 1 -- start start 2023-10-30T14:26:35.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a8753e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a800ca0e0 0x7f4a801ef570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a8753e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a800cf160 0x7f4a800cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a8753e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a801538a0 0x7f4a801585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a8753e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a800b3110 con 0x7f4a800ca0e0 2023-10-30T14:26:35.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a8753e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a801efab0 con 0x7f4a801538a0 2023-10-30T14:26:35.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a8753e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a800c4420 con 0x7f4a800cf160 2023-10-30T14:26:35.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a852da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a800ca0e0 0x7f4a801ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a852da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a800ca0e0 0x7f4a801ef570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56580/0 (socket says 172.21.15.46:56580) 2023-10-30T14:26:35.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a852da640 1 -- 172.21.15.46:0/4076618515 learned_addr learned my addr 172.21.15.46:0/4076618515 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:35.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a85adb640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a801538a0 0x7f4a801585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a84ad9640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a800cf160 0x7f4a800cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a852da640 1 -- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a800cf160 msgr2=0x7f4a800cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a852da640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a800cf160 0x7f4a800cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a852da640 1 -- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a801538a0 msgr2=0x7f4a801585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a852da640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a801538a0 0x7f4a801585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a852da640 1 -- 172.21.15.46:0/4076618515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a700094d0 con 0x7f4a800ca0e0 2023-10-30T14:26:35.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a852da640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a800ca0e0 0x7f4a801ef570 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00d800 tx=0x7f4a7c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:35.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a75ffb640 1 -- 172.21.15.46:0/4076618515 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c0096c0 con 0x7f4a800ca0e0 2023-10-30T14:26:35.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a75ffb640 1 -- 172.21.15.46:0/4076618515 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7c009820 con 0x7f4a800ca0e0 2023-10-30T14:26:35.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a8753e640 1 -- 172.21.15.46:0/4076618515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a801efd20 con 0x7f4a800ca0e0 2023-10-30T14:26:35.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a75ffb640 1 -- 172.21.15.46:0/4076618515 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c009bd0 con 0x7f4a800ca0e0 2023-10-30T14:26:35.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a8753e640 1 -- 172.21.15.46:0/4076618515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a801f02c0 con 0x7f4a800ca0e0 2023-10-30T14:26:35.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.162+0000 7f4a8753e640 1 -- 172.21.15.46:0/4076618515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a800b2a50 con 0x7f4a800ca0e0 2023-10-30T14:26:35.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.166+0000 7f4a75ffb640 1 -- 172.21.15.46:0/4076618515 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4a7c009d30 con 0x7f4a800ca0e0 2023-10-30T14:26:35.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.166+0000 7f4a75ffb640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4a640664d0 0x7f4a64068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.166+0000 7f4a75ffb640 1 -- 172.21.15.46:0/4076618515 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 8748+0+0 (secure 0 0 0) 0x7f4a7c085730 con 0x7f4a800ca0e0 2023-10-30T14:26:35.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.166+0000 7f4a84ad9640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4a640664d0 0x7f4a64068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.166+0000 7f4a84ad9640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4a640664d0 0x7f4a64068990 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f4a700b2000 tx=0x7f4a70008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.170+0000 7f4a75ffb640 1 -- 172.21.15.46:0/4076618515 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4a7c051f20 con 0x7f4a800ca0e0 2023-10-30T14:26:35.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.370+0000 7f4a8753e640 1 -- 172.21.15.46:0/4076618515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a80159860 con 0x7f4a800ca0e0 2023-10-30T14:26:35.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.370+0000 7f4a75ffb640 1 -- 172.21.15.46:0/4076618515 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+21184 (secure 0 0 0) 0x7f4a7c055540 con 0x7f4a800ca0e0 2023-10-30T14:26:35.375 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:26:35.376 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":109,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T14:26:34.886261+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":703,"snap_epoch":108,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.13","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6836/3574404756":"2023-10-31T14:24:59.863368+0000","172.21.15.46:6834/3574404756":"2023-10-31T14:24:59.863368+0000","172.21.15.46:6837/470164358":"2023-10-31T14:23:54.843840+0000","172.21.15.46:6835/470164358":"2023-10-31T14:23:54.843840+0000","172.21.15.46:6833/3764280378":"2023-10-31T14:22:39.814064+0000","172.21.15.46:6832/3764280378":"2023-10-31T14:22:39.814064+0000","172.21.15.46:6836/2239735286":"2023-10-31T14:20:39.772410+0000","172.21.15.46:6834/2197729559":"2023-10-31T13:51:39.216877+0000","172.21.15.46:6833/190134548":"2023-10-31T13:50:04.191476+0000","172.21.15.46:6832/190134548":"2023-10-31T13:50:04.191476+0000","172.21.15.46:6833/157453225":"2023-10-31T14:00:29.382692+0000","172.21.15.46:6837/1135482564":"2023-10-31T14:01:49.404286+0000","172.21.15.46:6837/3154171789":"2023-10-31T14:07:14.520406+0000","172.21.15.46:6836/3037060424":"2023-10-31T13:44:39.066140+0000","172.21.15.46:6835/1135482564":"2023-10-31T14:01:49.404286+0000","172.21.15.46:6835/2572113428":"2023-10-31T13:52:59.238225+0000","172.21.15.46:6833/2075693104":"2023-10-31T14:05:39.493074+0000","172.21.15.46:6836/2197729559":"2023-10-31T13:51:39.216877+0000","172.21.15.46:6835/994489214":"2023-10-31T13:25:13.695228+0000","172.21.15.46:6832/2075693104":"2023-10-31T14:05:39.493074+0000","172.21.15.46:6832/4044310929":"2023-10-31T13:41:14.000569+0000","172.21.15.46:6837/126953235":"2023-10-31T13:39:08.955190+0000","172.21.15.46:6832/875791418":"2023-10-31T13:26:28.713838+0000","172.21.15.46:6835/126953235":"2023-10-31T13:39:08.955190+0000","172.21.15.46:6837/3112266825":"2023-10-31T13:57:44.339823+0000","172.21.15.46:6836/3955931693":"2023-10-31T13:36:53.920781+0000","172.21.15.46:6834/3955931693":"2023-10-31T13:36:53.920781+0000","172.21.15.46:6835/681817982":"2023-10-31T14:18:44.741650+0000","172.21.15.46:6836/275749119":"2023-10-31T13:30:28.793688+0000","172.21.15.46:6837/2572113428":"2023-10-31T13:52:59.238225+0000","172.21.15.154:6834/3109262047":"2023-10-31T13:29:08.773403+0000","172.21.15.46:6834/2239735286":"2023-10-31T14:20:39.772410+0000","172.21.15.46:6833/2455118950":"2023-10-31T14:16:04.689323+0000","172.21.15.46:6837/994489214":"2023-10-31T13:25:13.695228+0000","172.21.15.46:6837/3044281789":"2023-10-31T14:12:14.616071+0000","172.21.15.46:6835/1315246492":"2023-10-31T13:31:23.809908+0000","172.21.15.46:6835/3112266825":"2023-10-31T13:57:44.339823+0000","172.21.15.46:6834/128865315":"2023-10-31T14:08:44.543673+0000","172.21.15.46:6833/603884510":"2023-10-31T13:33:43.868205+0000","172.21.15.46:6835/2546436963":"2023-10-31T13:47:24.113508+0000","172.21.15.46:6837/1315246492":"2023-10-31T13:31:23.809908+0000","172.21.15.154:6834/651441434":"2023-10-31T13:35:03.892131+0000","172.21.15.46:6833/2807435195":"2023-10-31T13:54:39.265053+0000","172.21.15.46:6834/711135361":"2023-10-31T13:55:44.283031+0000","172.21.15.154:6835/651441434":"2023-10-31T13:35:03.892131+0000","172.21.15.46:6836/711135361":"2023-10-31T13:55:44.283031+0000","172.21.15.46:6837/2546436963":"2023-10-31T13:47:24.113508+0000","172.21.15.46:6833/875791418":"2023-10-31T13:26:28.713838+0000","172.21.15.154:6835/3109262047":"2023-10-31T13:29:08.773403+0000","172.21.15.46:6832/2807435195":"2023-10-31T13:54:39.265053+0000","172.21.15.46:6832/157453225":"2023-10-31T14:00:29.382692+0000","172.21.15.46:6832/603884510":"2023-10-31T13:33:43.868205+0000","172.21.15.46:6834/3734066164":"2023-10-31T14:04:24.459794+0000","172.21.15.46:6833/4044310929":"2023-10-31T13:41:14.000569+0000","172.21.15.46:6834/66265765":"2023-10-31T14:14:54.671010+0000","172.21.15.46:6836/3734066164":"2023-10-31T14:04:24.459794+0000","172.21.15.46:6836/128865315":"2023-10-31T14:08:44.543673+0000","172.21.15.46:6832/2455118950":"2023-10-31T14:16:04.689323+0000","172.21.15.46:6835/3154171789":"2023-10-31T14:07:14.520406+0000","172.21.15.46:6832/4078242540":"2023-10-31T14:09:49.576202+0000","172.21.15.46:6834/3037060424":"2023-10-31T13:44:39.066140+0000","172.21.15.46:6833/4078242540":"2023-10-31T14:09:49.576202+0000","172.21.15.46:6835/3044281789":"2023-10-31T14:12:14.616071+0000","172.21.15.46:6836/66265765":"2023-10-31T14:14:54.671010+0000","172.21.15.46:6834/275749119":"2023-10-31T13:30:28.793688+0000","172.21.15.46:6837/681817982":"2023-10-31T14:18:44.741650+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":690,"length":14}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":686,"length":4}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:26:35.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.374+0000 7f4a8753e640 1 -- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4a640664d0 msgr2=0x7f4a64068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.374+0000 7f4a8753e640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4a640664d0 0x7f4a64068990 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f4a700b2000 tx=0x7f4a70008040 comp rx=0 tx=0).stop 2023-10-30T14:26:35.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.374+0000 7f4a8753e640 1 -- 172.21.15.46:0/4076618515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a800ca0e0 msgr2=0x7f4a801ef570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.374+0000 7f4a8753e640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a800ca0e0 0x7f4a801ef570 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00d800 tx=0x7f4a7c00dcc0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.374+0000 7f4a8753e640 1 -- 172.21.15.46:0/4076618515 shutdown_connections 2023-10-30T14:26:35.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.374+0000 7f4a8753e640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4a640664d0 0x7f4a64068990 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.374+0000 7f4a8753e640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a801538a0 0x7f4a801585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.374+0000 7f4a8753e640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a800cf160 0x7f4a800cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.374+0000 7f4a8753e640 1 --2- 172.21.15.46:0/4076618515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a800ca0e0 0x7f4a801ef570 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.374+0000 7f4a8753e640 1 -- 172.21.15.46:0/4076618515 >> 172.21.15.46:0/4076618515 conn(0x7f4a8006bfe0 msgr2=0x7f4a80148dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:35.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.374+0000 7f4a8753e640 1 -- 172.21.15.46:0/4076618515 shutdown_connections 2023-10-30T14:26:35.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.374+0000 7f4a8753e640 1 -- 172.21.15.46:0/4076618515 wait complete. 2023-10-30T14:26:35.398 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:23:06.788949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:23:17.648827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:23:18.326967+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 703, 'snap_epoch': 108, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T14:26:35.399 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T14:26:35.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f478b59e640 1 Processor -- start 2023-10-30T14:26:35.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f478b59e640 1 -- start start 2023-10-30T14:26:35.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f478b59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478c153c20 0x7f478c15a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f478b59e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f478c0cb4c0 0x7f478c0cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f478b59e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478c15a6b0 con 0x7f478c153840 2023-10-30T14:26:35.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f478b59e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f478c15dbc0 con 0x7f478c0cb4c0 2023-10-30T14:26:35.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f478b59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478c15d730 con 0x7f478c153c20 2023-10-30T14:26:35.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f478a59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f478c153840 0x7f478c0c3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54928/0 (socket says 172.21.15.46:54928) 2023-10-30T14:26:35.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f478a59c640 1 -- 172.21.15.46:0/1987590965 learned_addr learned my addr 172.21.15.46:0/1987590965 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:35.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f4789d9b640 1 --2- 172.21.15.46:0/1987590965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478c153c20 0x7f478c15a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f478ad9d640 1 --2- 172.21.15.46:0/1987590965 >> v2:172.21.15.154:3300/0 conn(0x7f478c0cb4c0 0x7f478c0cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f478959a640 1 -- 172.21.15.46:0/1987590965 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361717865 0 0) 0x7f477c001120 con 0x7f478c153840 2023-10-30T14:26:35.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f478959a640 1 -- 172.21.15.46:0/1987590965 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4774003530 con 0x7f478c153840 2023-10-30T14:26:35.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f4789d9b640 1 -- 172.21.15.46:0/1987590965 >> v2:172.21.15.154:3300/0 conn(0x7f478c0cb4c0 msgr2=0x7f478c0cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:35.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f4789d9b640 1 --2- 172.21.15.46:0/1987590965 >> v2:172.21.15.154:3300/0 conn(0x7f478c0cb4c0 0x7f478c0cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f4789d9b640 1 -- 172.21.15.46:0/1987590965 >> v1:172.21.15.46:6789/0 conn(0x7f478c153840 legacy=0x7f478c0c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.546+0000 7f4789d9b640 1 -- 172.21.15.46:0/1987590965 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c0ca8f0 con 0x7f478c153c20 2023-10-30T14:26:35.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.554+0000 7f4789d9b640 1 --2- 172.21.15.46:0/1987590965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478c153c20 0x7f478c15a170 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f47800097c0 tx=0x7f478001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d96948b3b8a68f88 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:35.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.554+0000 7f478959a640 1 -- 172.21.15.46:0/1987590965 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47800a8970 con 0x7f478c153c20 2023-10-30T14:26:35.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.554+0000 7f478959a640 1 -- 172.21.15.46:0/1987590965 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47800a8ad0 con 0x7f478c153c20 2023-10-30T14:26:35.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.554+0000 7f478959a640 1 -- 172.21.15.46:0/1987590965 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47800a8e60 con 0x7f478c153c20 2023-10-30T14:26:35.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.554+0000 7f478b59e640 1 -- 172.21.15.46:0/1987590965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478c153c20 msgr2=0x7f478c15a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.554+0000 7f478b59e640 1 --2- 172.21.15.46:0/1987590965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478c153c20 0x7f478c15a170 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f47800097c0 tx=0x7f478001ec70 comp rx=0 tx=0).stop 2023-10-30T14:26:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.554+0000 7f478b59e640 1 -- 172.21.15.46:0/1987590965 shutdown_connections 2023-10-30T14:26:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.554+0000 7f478b59e640 1 --2- 172.21.15.46:0/1987590965 >> v2:172.21.15.154:3300/0 conn(0x7f478c0cb4c0 0x7f478c0cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.554+0000 7f478b59e640 1 --2- 172.21.15.46:0/1987590965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478c153c20 0x7f478c15a170 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.554+0000 7f478b59e640 1 -- 172.21.15.46:0/1987590965 >> 172.21.15.46:0/1987590965 conn(0x7f478c06be40 msgr2=0x7f478c06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:35.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.554+0000 7f478b59e640 1 -- 172.21.15.46:0/1987590965 shutdown_connections 2023-10-30T14:26:35.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.554+0000 7f478b59e640 1 -- 172.21.15.46:0/1987590965 wait complete. 2023-10-30T14:26:35.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478b59e640 1 Processor -- start 2023-10-30T14:26:35.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478b59e640 1 -- start start 2023-10-30T14:26:35.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c0cb4c0 0x7f478c154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478b59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f478c153840 0x7f478c15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478b59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478c153c20 0x7f478c1ef030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478b59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c154b10 con 0x7f478c0cb4c0 2023-10-30T14:26:35.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478b59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c148e40 con 0x7f478c153840 2023-10-30T14:26:35.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478b59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478c149090 con 0x7f478c153c20 2023-10-30T14:26:35.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c0cb4c0 0x7f478c154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c0cb4c0 0x7f478c154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56596/0 (socket says 172.21.15.46:56596) 2023-10-30T14:26:35.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478a59c640 1 -- 172.21.15.46:0/431139886 learned_addr learned my addr 172.21.15.46:0/431139886 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:35.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f4789d9b640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f478c153840 0x7f478c15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478ad9d640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478c153c20 0x7f478c1ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478a59c640 1 -- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478c153c20 msgr2=0x7f478c1ef030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478a59c640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478c153c20 0x7f478c1ef030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478a59c640 1 -- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f478c153840 msgr2=0x7f478c15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478a59c640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f478c153840 0x7f478c15bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478a59c640 1 -- 172.21.15.46:0/431139886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4780009420 con 0x7f478c0cb4c0 2023-10-30T14:26:35.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478a59c640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c0cb4c0 0x7f478c154060 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f477c00c4d0 tx=0x7f477c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:35.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f477affd640 1 -- 172.21.15.46:0/431139886 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477c005cf0 con 0x7f478c0cb4c0 2023-10-30T14:26:35.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f477affd640 1 -- 172.21.15.46:0/431139886 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477c005e50 con 0x7f478c0cb4c0 2023-10-30T14:26:35.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f477affd640 1 -- 172.21.15.46:0/431139886 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477c007290 con 0x7f478c0cb4c0 2023-10-30T14:26:35.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478b59e640 1 -- 172.21.15.46:0/431139886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478c149370 con 0x7f478c0cb4c0 2023-10-30T14:26:35.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.558+0000 7f478b59e640 1 -- 172.21.15.46:0/431139886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478c157c30 con 0x7f478c0cb4c0 2023-10-30T14:26:35.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.562+0000 7f478b59e640 1 -- 172.21.15.46:0/431139886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478c04e9c0 con 0x7f478c0cb4c0 2023-10-30T14:26:35.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.562+0000 7f477affd640 1 -- 172.21.15.46:0/431139886 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f477c0073f0 con 0x7f478c0cb4c0 2023-10-30T14:26:35.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.562+0000 7f477affd640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f476c0664a0 0x7f476c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.562+0000 7f477affd640 1 -- 172.21.15.46:0/431139886 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 8748+0+0 (secure 0 0 0) 0x7f477c083ea0 con 0x7f478c0cb4c0 2023-10-30T14:26:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.562+0000 7f4789d9b640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f476c0664a0 0x7f476c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.562+0000 7f4789d9b640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f476c0664a0 0x7f476c068960 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f47800b2000 tx=0x7f47800aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:35.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.566+0000 7f477affd640 1 -- 172.21.15.46:0/431139886 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f477c0507d0 con 0x7f478c0cb4c0 2023-10-30T14:26:35.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.742+0000 7f478b59e640 1 -- 172.21.15.46:0/431139886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f478c15a6b0 con 0x7f478c0cb4c0 2023-10-30T14:26:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.742+0000 7f477affd640 1 -- 172.21.15.46:0/431139886 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7f478c15a6b0 con 0x7f478c0cb4c0 2023-10-30T14:26:35.748 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-10-30T14:26:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.746+0000 7f478b59e640 1 -- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f476c0664a0 msgr2=0x7f476c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.746+0000 7f478b59e640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f476c0664a0 0x7f476c068960 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f47800b2000 tx=0x7f47800aa070 comp rx=0 tx=0).stop 2023-10-30T14:26:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.746+0000 7f478b59e640 1 -- 172.21.15.46:0/431139886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c0cb4c0 msgr2=0x7f478c154060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.746+0000 7f478b59e640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c0cb4c0 0x7f478c154060 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f477c00c4d0 tx=0x7f477c00c990 comp rx=0 tx=0).stop 2023-10-30T14:26:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.746+0000 7f478b59e640 1 -- 172.21.15.46:0/431139886 shutdown_connections 2023-10-30T14:26:35.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.746+0000 7f478b59e640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f476c0664a0 0x7f476c068960 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.746+0000 7f478b59e640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f478c153c20 0x7f478c1ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.746+0000 7f478b59e640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f478c153840 0x7f478c15bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.746+0000 7f478b59e640 1 --2- 172.21.15.46:0/431139886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c0cb4c0 0x7f478c154060 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.746+0000 7f478b59e640 1 -- 172.21.15.46:0/431139886 >> 172.21.15.46:0/431139886 conn(0x7f478c06be40 msgr2=0x7f478c0b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.746+0000 7f478b59e640 1 -- 172.21.15.46:0/431139886 shutdown_connections 2023-10-30T14:26:35.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.746+0000 7f478b59e640 1 -- 172.21.15.46:0/431139886 wait complete. 2023-10-30T14:26:35.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T14:26:35.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d1e972640 1 Processor -- start 2023-10-30T14:26:35.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d1e972640 1 -- start start 2023-10-30T14:26:35.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d1e972640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d18153980 0x7f3d180caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d1e972640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1815a920 con 0x7f3d18159930 2023-10-30T14:26:35.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d1e972640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d180cb930 con 0x7f3d180cb1e0 2023-10-30T14:26:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d1e972640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d18159dc0 con 0x7f3d18153980 2023-10-30T14:26:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d177fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3d18159930 0x7f3d180cf1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54938/0 (socket says 172.21.15.46:54938) 2023-10-30T14:26:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d177fe640 1 -- 172.21.15.46:0/726938411 learned_addr learned my addr 172.21.15.46:0/726938411 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d17fff640 1 --2- 172.21.15.46:0/726938411 >> v2:172.21.15.154:3300/0 conn(0x7f3d18153980 0x7f3d180caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d16ffd640 1 -- 172.21.15.46:0/726938411 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808134606 0 0) 0x7f3d1815a920 con 0x7f3d18159930 2023-10-30T14:26:35.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d16ffd640 1 -- 172.21.15.46:0/726938411 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d00003530 con 0x7f3d18159930 2023-10-30T14:26:35.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d17fff640 1 -- 172.21.15.46:0/726938411 >> v1:172.21.15.154:6789/0 conn(0x7f3d180cb1e0 legacy=0x7f3d1815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d17fff640 1 -- 172.21.15.46:0/726938411 >> v1:172.21.15.46:6789/0 conn(0x7f3d18159930 legacy=0x7f3d180cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.942+0000 7f3d17fff640 1 -- 172.21.15.46:0/726938411 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1815a080 con 0x7f3d18153980 2023-10-30T14:26:35.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.950+0000 7f3d17fff640 1 --2- 172.21.15.46:0/726938411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d18153980 0x7f3d180caca0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f3d080097c0 tx=0x7f3d0801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=227ef2db6ef8eafc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:35.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.950+0000 7f3d16ffd640 1 -- 172.21.15.46:0/726938411 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d080ad660 con 0x7f3d18153980 2023-10-30T14:26:35.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.950+0000 7f3d16ffd640 1 -- 172.21.15.46:0/726938411 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d080ad7c0 con 0x7f3d18153980 2023-10-30T14:26:35.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.950+0000 7f3d16ffd640 1 -- 172.21.15.46:0/726938411 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d080adb30 con 0x7f3d18153980 2023-10-30T14:26:35.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.950+0000 7f3d1e972640 1 -- 172.21.15.46:0/726938411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d18153980 msgr2=0x7f3d180caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.950+0000 7f3d1e972640 1 --2- 172.21.15.46:0/726938411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d18153980 0x7f3d180caca0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f3d080097c0 tx=0x7f3d0801c450 comp rx=0 tx=0).stop 2023-10-30T14:26:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.950+0000 7f3d1e972640 1 -- 172.21.15.46:0/726938411 shutdown_connections 2023-10-30T14:26:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.950+0000 7f3d1e972640 1 --2- 172.21.15.46:0/726938411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d18153980 0x7f3d180caca0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.950+0000 7f3d1e972640 1 -- 172.21.15.46:0/726938411 >> 172.21.15.46:0/726938411 conn(0x7f3d1806c0f0 msgr2=0x7f3d1806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.950+0000 7f3d1e972640 1 -- 172.21.15.46:0/726938411 shutdown_connections 2023-10-30T14:26:35.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.950+0000 7f3d1e972640 1 -- 172.21.15.46:0/726938411 wait complete. 2023-10-30T14:26:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d1e972640 1 Processor -- start 2023-10-30T14:26:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d1e972640 1 -- start start 2023-10-30T14:26:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d1e972640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d180cb1e0 0x7f3d18147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d1e972640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d18153980 0x7f3d18148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d1e972640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815bb60 0x7f3d1815bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d1e972640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d180cd7e0 con 0x7f3d180cb1e0 2023-10-30T14:26:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d1e972640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d180cd500 con 0x7f3d1815bb60 2023-10-30T14:26:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d1e972640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1815c520 con 0x7f3d18153980 2023-10-30T14:26:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d180cb1e0 0x7f3d18147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d180cb1e0 0x7f3d18147ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56608/0 (socket says 172.21.15.46:56608) 2023-10-30T14:26:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d17fff640 1 -- 172.21.15.46:0/2722776019 learned_addr learned my addr 172.21.15.46:0/2722776019 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d177fe640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d18153980 0x7f3d18148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d1cf0f640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815bb60 0x7f3d1815bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d17fff640 1 -- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d18153980 msgr2=0x7f3d18148000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d17fff640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d18153980 0x7f3d18148000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d17fff640 1 -- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815bb60 msgr2=0x7f3d1815bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d17fff640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815bb60 0x7f3d1815bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.954+0000 7f3d17fff640 1 -- 172.21.15.46:0/2722776019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d08009420 con 0x7f3d180cb1e0 2023-10-30T14:26:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.958+0000 7f3d17fff640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d180cb1e0 0x7f3d18147ac0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f3d0801a080 tx=0x7f3d0801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.958+0000 7f3d14ff9640 1 -- 172.21.15.46:0/2722776019 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08004230 con 0x7f3d180cb1e0 2023-10-30T14:26:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.958+0000 7f3d14ff9640 1 -- 172.21.15.46:0/2722776019 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d08004390 con 0x7f3d180cb1e0 2023-10-30T14:26:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.958+0000 7f3d1e972640 1 -- 172.21.15.46:0/2722776019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1815b620 con 0x7f3d180cb1e0 2023-10-30T14:26:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.958+0000 7f3d14ff9640 1 -- 172.21.15.46:0/2722776019 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08003820 con 0x7f3d180cb1e0 2023-10-30T14:26:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.958+0000 7f3d1e972640 1 -- 172.21.15.46:0/2722776019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d180b3df0 con 0x7f3d180cb1e0 2023-10-30T14:26:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.958+0000 7f3d1e972640 1 -- 172.21.15.46:0/2722776019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1804e170 con 0x7f3d180cb1e0 2023-10-30T14:26:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.958+0000 7f3d14ff9640 1 -- 172.21.15.46:0/2722776019 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3d08003980 con 0x7f3d180cb1e0 2023-10-30T14:26:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.962+0000 7f3d14ff9640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d040664a0 0x7f3d04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.962+0000 7f3d14ff9640 1 -- 172.21.15.46:0/2722776019 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 8748+0+0 (secure 0 0 0) 0x7f3d0811f740 con 0x7f3d180cb1e0 2023-10-30T14:26:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.962+0000 7f3d177fe640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d040664a0 0x7f3d04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.962+0000 7f3d177fe640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d040664a0 0x7f3d04068960 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c007fd0 tx=0x7f3d0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:35.962+0000 7f3d14ff9640 1 -- 172.21.15.46:0/2722776019 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3d080ec070 con 0x7f3d180cb1e0 2023-10-30T14:26:36.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 -- 172.21.15.46:0/2722776019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3d18157580 con 0x7f3d180cb1e0 2023-10-30T14:26:36.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d14ff9640 1 -- 172.21.15.46:0/2722776019 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3d080b2020 con 0x7f3d180cb1e0 2023-10-30T14:26:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-10-30T14:26:36.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 -- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d040664a0 msgr2=0x7f3d04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d040664a0 0x7f3d04068960 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c007fd0 tx=0x7f3d0c009040 comp rx=0 tx=0).stop 2023-10-30T14:26:36.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 -- 172.21.15.46:0/2722776019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d180cb1e0 msgr2=0x7f3d18147ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d180cb1e0 0x7f3d18147ac0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f3d0801a080 tx=0x7f3d0801ac90 comp rx=0 tx=0).stop 2023-10-30T14:26:36.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 -- 172.21.15.46:0/2722776019 shutdown_connections 2023-10-30T14:26:36.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d040664a0 0x7f3d04068960 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815bb60 0x7f3d1815bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d18153980 0x7f3d18148000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 --2- 172.21.15.46:0/2722776019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d180cb1e0 0x7f3d18147ac0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 -- 172.21.15.46:0/2722776019 >> 172.21.15.46:0/2722776019 conn(0x7f3d1806c0f0 msgr2=0x7f3d181579f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:36.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 -- 172.21.15.46:0/2722776019 shutdown_connections 2023-10-30T14:26:36.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.158+0000 7f3d1e972640 1 -- 172.21.15.46:0/2722776019 wait complete. 2023-10-30T14:26:36.192 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T14:26:36.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.338+0000 7f6af93d3640 1 Processor -- start 2023-10-30T14:26:36.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.338+0000 7f6af93d3640 1 -- start start 2023-10-30T14:26:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.338+0000 7f6af93d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6af4153980 0x7f6af40caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.338+0000 7f6af93d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af415a920 con 0x7f6af4159930 2023-10-30T14:26:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.338+0000 7f6af93d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af40cb930 con 0x7f6af40cb1e0 2023-10-30T14:26:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.338+0000 7f6af93d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af4159dc0 con 0x7f6af4153980 2023-10-30T14:26:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.338+0000 7f6af2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6af4153980 0x7f6af40caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.338+0000 7f6af27fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6af4159930 0x7f6af40cf1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54950/0 (socket says 172.21.15.46:54950) 2023-10-30T14:26:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.338+0000 7f6af27fc640 1 -- 172.21.15.46:0/3628882677 learned_addr learned my addr 172.21.15.46:0/3628882677 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:36.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.338+0000 7f6af2ffd640 1 -- 172.21.15.46:0/3628882677 >> v1:172.21.15.154:6789/0 conn(0x7f6af40cb1e0 legacy=0x7f6af415d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.338+0000 7f6af2ffd640 1 -- 172.21.15.46:0/3628882677 >> v1:172.21.15.46:6789/0 conn(0x7f6af4159930 legacy=0x7f6af40cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.338+0000 7f6af2ffd640 1 -- 172.21.15.46:0/3628882677 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af415a080 con 0x7f6af4153980 2023-10-30T14:26:36.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.342+0000 7f6af2ffd640 1 --2- 172.21.15.46:0/3628882677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4153980 0x7f6af40caca0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40097c0 tx=0x7f6ae401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73577577186597f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:36.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.342+0000 7f6af1ffb640 1 -- 172.21.15.46:0/3628882677 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40ad660 con 0x7f6af4153980 2023-10-30T14:26:36.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.342+0000 7f6af1ffb640 1 -- 172.21.15.46:0/3628882677 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae40ad7c0 con 0x7f6af4153980 2023-10-30T14:26:36.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.342+0000 7f6af1ffb640 1 -- 172.21.15.46:0/3628882677 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40adb50 con 0x7f6af4153980 2023-10-30T14:26:36.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.342+0000 7f6af93d3640 1 -- 172.21.15.46:0/3628882677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4153980 msgr2=0x7f6af40caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.342+0000 7f6af93d3640 1 --2- 172.21.15.46:0/3628882677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4153980 0x7f6af40caca0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40097c0 tx=0x7f6ae401c450 comp rx=0 tx=0).stop 2023-10-30T14:26:36.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.342+0000 7f6af93d3640 1 -- 172.21.15.46:0/3628882677 shutdown_connections 2023-10-30T14:26:36.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.342+0000 7f6af93d3640 1 --2- 172.21.15.46:0/3628882677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4153980 0x7f6af40caca0 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.342+0000 7f6af93d3640 1 -- 172.21.15.46:0/3628882677 >> 172.21.15.46:0/3628882677 conn(0x7f6af406c0f0 msgr2=0x7f6af406c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:36.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.342+0000 7f6af93d3640 1 -- 172.21.15.46:0/3628882677 shutdown_connections 2023-10-30T14:26:36.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.342+0000 7f6af93d3640 1 -- 172.21.15.46:0/3628882677 wait complete. 2023-10-30T14:26:36.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af93d3640 1 Processor -- start 2023-10-30T14:26:36.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af93d3640 1 -- start start 2023-10-30T14:26:36.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af93d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6af40cb1e0 0x7f6af41553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af93d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4153980 0x7f6af40ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af93d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af41f1e00 0x7f6af41f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af93d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af41f27f0 con 0x7f6af4153980 2023-10-30T14:26:36.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af93d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af40cf120 con 0x7f6af41f1e00 2023-10-30T14:26:36.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af93d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af40cf680 con 0x7f6af40cb1e0 2023-10-30T14:26:36.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4153980 0x7f6af40ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4153980 0x7f6af40ce3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56622/0 (socket says 172.21.15.46:56622) 2023-10-30T14:26:36.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af27fc640 1 -- 172.21.15.46:0/1731959474 learned_addr learned my addr 172.21.15.46:0/1731959474 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:36.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af2ffd640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6af40cb1e0 0x7f6af41553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af37fe640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af41f1e00 0x7f6af41f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af27fc640 1 -- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6af40cb1e0 msgr2=0x7f6af41553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af27fc640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6af40cb1e0 0x7f6af41553e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af27fc640 1 -- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af41f1e00 msgr2=0x7f6af41f22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af27fc640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af41f1e00 0x7f6af41f22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af27fc640 1 -- 172.21.15.46:0/1731959474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae4009420 con 0x7f6af4153980 2023-10-30T14:26:36.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.346+0000 7f6af27fc640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4153980 0x7f6af40ce3d0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800c530 tx=0x7f6ae800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:36.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.350+0000 7f6adb7fe640 1 -- 172.21.15.46:0/1731959474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8005cf0 con 0x7f6af4153980 2023-10-30T14:26:36.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.350+0000 7f6adb7fe640 1 -- 172.21.15.46:0/1731959474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae8005e50 con 0x7f6af4153980 2023-10-30T14:26:36.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.350+0000 7f6adb7fe640 1 -- 172.21.15.46:0/1731959474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8007270 con 0x7f6af4153980 2023-10-30T14:26:36.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.350+0000 7f6af93d3640 1 -- 172.21.15.46:0/1731959474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af40cf3a0 con 0x7f6af4153980 2023-10-30T14:26:36.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.350+0000 7f6af93d3640 1 -- 172.21.15.46:0/1731959474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af41569a0 con 0x7f6af4153980 2023-10-30T14:26:36.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.350+0000 7f6af93d3640 1 -- 172.21.15.46:0/1731959474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af40cf850 con 0x7f6af4153980 2023-10-30T14:26:36.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.350+0000 7f6adb7fe640 1 -- 172.21.15.46:0/1731959474 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ae80073d0 con 0x7f6af4153980 2023-10-30T14:26:36.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.350+0000 7f6adb7fe640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6ae00664a0 0x7f6ae0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.350+0000 7f6adb7fe640 1 -- 172.21.15.46:0/1731959474 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 8748+0+0 (secure 0 0 0) 0x7f6ae8083eb0 con 0x7f6af4153980 2023-10-30T14:26:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.354+0000 7f6af2ffd640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6ae00664a0 0x7f6ae0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.354+0000 7f6af2ffd640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6ae00664a0 0x7f6ae0068960 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40af000 tx=0x7f6ae40a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.358+0000 7f6adb7fe640 1 -- 172.21.15.46:0/1731959474 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ae80507b0 con 0x7f6af4153980 2023-10-30T14:26:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.518+0000 7f6af93d3640 1 -- 172.21.15.46:0/1731959474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6af404e170 con 0x7f6af4153980 2023-10-30T14:26:36.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.518+0000 7f6adb7fe640 1 -- 172.21.15.46:0/1731959474 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6ae8017070 con 0x7f6af4153980 2023-10-30T14:26:36.522 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-10-30T14:26:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.522+0000 7f6af93d3640 1 -- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6ae00664a0 msgr2=0x7f6ae0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.522+0000 7f6af93d3640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6ae00664a0 0x7f6ae0068960 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40af000 tx=0x7f6ae40a7000 comp rx=0 tx=0).stop 2023-10-30T14:26:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.522+0000 7f6af93d3640 1 -- 172.21.15.46:0/1731959474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4153980 msgr2=0x7f6af40ce3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.522+0000 7f6af93d3640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4153980 0x7f6af40ce3d0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800c530 tx=0x7f6ae800c9f0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.522+0000 7f6af93d3640 1 -- 172.21.15.46:0/1731959474 shutdown_connections 2023-10-30T14:26:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.522+0000 7f6af93d3640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6ae00664a0 0x7f6ae0068960 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.522+0000 7f6af93d3640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af41f1e00 0x7f6af41f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.522+0000 7f6af93d3640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af4153980 0x7f6af40ce3d0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.522+0000 7f6af93d3640 1 --2- 172.21.15.46:0/1731959474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6af40cb1e0 0x7f6af41553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.522+0000 7f6af93d3640 1 -- 172.21.15.46:0/1731959474 >> 172.21.15.46:0/1731959474 conn(0x7f6af406c0f0 msgr2=0x7f6af41582c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.522+0000 7f6af93d3640 1 -- 172.21.15.46:0/1731959474 shutdown_connections 2023-10-30T14:26:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.522+0000 7f6af93d3640 1 -- 172.21.15.46:0/1731959474 wait complete. 2023-10-30T14:26:36.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-30T14:26:36.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8cee291640 1 Processor -- start 2023-10-30T14:26:36.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8cee291640 1 -- start start 2023-10-30T14:26:36.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8cee291640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce8153c60 0x7f8ce80ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8cee291640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce80cb780 con 0x7f8ce80c4430 2023-10-30T14:26:36.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8cee291640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce815a2e0 con 0x7f8ce8159830 2023-10-30T14:26:36.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8cee291640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce8154040 con 0x7f8ce8153c60 2023-10-30T14:26:36.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce77fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8ce80c4430 0x7f8ce80cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37994/0 (socket says 172.21.15.46:37994) 2023-10-30T14:26:36.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce77fe640 1 -- 172.21.15.46:0/502431470 learned_addr learned my addr 172.21.15.46:0/502431470 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:36.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce7fff640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce8153c60 0x7f8ce80ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490176188 0 0) 0x7f8ce80cb780 con 0x7f8ce80c4430 2023-10-30T14:26:36.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd0003530 con 0x7f8ce80c4430 2023-10-30T14:26:36.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268009356 0 0) 0x7f8ce815a2e0 con 0x7f8ce8159830 2023-10-30T14:26:36.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ce80cb780 con 0x7f8ce8159830 2023-10-30T14:26:36.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 425875852 0 0) 0x7f8cd0003530 con 0x7f8ce80c4430 2023-10-30T14:26:36.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce8153c60 msgr2=0x7f8ce80ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:36.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce8153c60 0x7f8ce80ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 >> v1:172.21.15.154:6789/0 conn(0x7f8ce8159830 legacy=0x7f8ce80cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce815a5c0 con 0x7f8ce80c4430 2023-10-30T14:26:36.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f8cdc001680 con 0x7f8ce80c4430 2023-10-30T14:26:36.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce7fff640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce8153c60 0x7f8ce80ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:26:36.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 >> v1:172.21.15.46:6789/0 conn(0x7f8ce80c4430 legacy=0x7f8ce80cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cd0003b40 0x7f8cd0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cd00047d0 0x7f8cd0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce8153c60 0x7f8cd0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce8154040 con 0x7f8cd00047d0 2023-10-30T14:26:36.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd000b9c0 con 0x7f8cd0003b40 2023-10-30T14:26:36.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd000bb80 con 0x7f8ce8153c60 2023-10-30T14:26:36.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce77fe640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cd00047d0 0x7f8cd0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce7fff640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cd0003b40 0x7f8cd0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8cec82e640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce8153c60 0x7f8cd0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce77fe640 1 -- 172.21.15.46:0/502431470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce8153c60 msgr2=0x7f8cd0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:36.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce77fe640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce8153c60 0x7f8cd0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce77fe640 1 -- 172.21.15.46:0/502431470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cd0003b40 msgr2=0x7f8cd0003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:36.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce77fe640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cd0003b40 0x7f8cd0003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.698+0000 7f8ce77fe640 1 -- 172.21.15.46:0/502431470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd8009420 con 0x7f8cd00047d0 2023-10-30T14:26:36.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.702+0000 7f8ce77fe640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cd00047d0 0x7f8cd0004c50 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc001ec0 tx=0x7f8cdc01cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad28f83384dcd832 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:36.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.702+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc0058e0 con 0x7f8cd00047d0 2023-10-30T14:26:36.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.702+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cdc005a60 con 0x7f8cd00047d0 2023-10-30T14:26:36.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.702+0000 7f8ce6ffd640 1 -- 172.21.15.46:0/502431470 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cdc005e10 con 0x7f8cd00047d0 2023-10-30T14:26:36.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.702+0000 7f8cee291640 1 -- 172.21.15.46:0/502431470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cd00047d0 msgr2=0x7f8cd0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.702+0000 7f8cee291640 1 --2- 172.21.15.46:0/502431470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cd00047d0 0x7f8cd0004c50 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc001ec0 tx=0x7f8cdc01cb20 comp rx=0 tx=0).stop 2023-10-30T14:26:36.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.702+0000 7f8cec82e640 1 -- 172.21.15.46:0/502431470 reap_dead start 2023-10-30T14:26:36.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.702+0000 7f8cee291640 1 -- 172.21.15.46:0/502431470 shutdown_connections 2023-10-30T14:26:36.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.702+0000 7f8cee291640 1 -- 172.21.15.46:0/502431470 >> 172.21.15.46:0/502431470 conn(0x7f8ce806bfe0 msgr2=0x7f8ce806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:36.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.702+0000 7f8cee291640 1 -- 172.21.15.46:0/502431470 shutdown_connections 2023-10-30T14:26:36.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.702+0000 7f8cee291640 1 -- 172.21.15.46:0/502431470 wait complete. 2023-10-30T14:26:36.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cee291640 1 Processor -- start 2023-10-30T14:26:36.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cee291640 1 -- start start 2023-10-30T14:26:36.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cee291640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cd00047d0 0x7f8ce8149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cee291640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce80cf140 0x7f8ce80cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cee291640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ce81f34e0 0x7f8ce81f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cee291640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce81f30c0 con 0x7f8ce81f34e0 2023-10-30T14:26:36.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cee291640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce81f3ea0 con 0x7f8ce80cf140 2023-10-30T14:26:36.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cee291640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce80ce170 con 0x7f8cd00047d0 2023-10-30T14:26:36.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cec82e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ce81f34e0 0x7f8ce81f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cec82e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ce81f34e0 0x7f8ce81f3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43518/0 (socket says 172.21.15.46:43518) 2023-10-30T14:26:36.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cec82e640 1 -- 172.21.15.46:0/4182020370 learned_addr learned my addr 172.21.15.46:0/4182020370 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:36.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8ce7fff640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cd00047d0 0x7f8ce8149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8ce77fe640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce80cf140 0x7f8ce80cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cec82e640 1 -- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cd00047d0 msgr2=0x7f8ce8149470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cec82e640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cd00047d0 0x7f8ce8149470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cec82e640 1 -- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce80cf140 msgr2=0x7f8ce80cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cec82e640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce80cf140 0x7f8ce80cf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cec82e640 1 -- 172.21.15.46:0/4182020370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce81f6420 con 0x7f8ce81f34e0 2023-10-30T14:26:36.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8ce77fe640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce80cf140 0x7f8ce80cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:36.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cec82e640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ce81f34e0 0x7f8ce81f3960 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8157460 tx=0x7f8ce0011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:36.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8ce4ff9640 1 -- 172.21.15.46:0/4182020370 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce000fce0 con 0x7f8ce81f34e0 2023-10-30T14:26:36.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8ce4ff9640 1 -- 172.21.15.46:0/4182020370 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce000fe40 con 0x7f8ce81f34e0 2023-10-30T14:26:36.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8ce4ff9640 1 -- 172.21.15.46:0/4182020370 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0015650 con 0x7f8ce81f34e0 2023-10-30T14:26:36.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cee291640 1 -- 172.21.15.46:0/4182020370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce81f66c0 con 0x7f8ce81f34e0 2023-10-30T14:26:36.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.706+0000 7f8cee291640 1 -- 172.21.15.46:0/4182020370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce81f6c80 con 0x7f8ce81f34e0 2023-10-30T14:26:36.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.710+0000 7f8cee291640 1 -- 172.21.15.46:0/4182020370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce81f3200 con 0x7f8ce81f34e0 2023-10-30T14:26:36.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.710+0000 7f8ce4ff9640 1 -- 172.21.15.46:0/4182020370 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ce00157b0 con 0x7f8ce81f34e0 2023-10-30T14:26:36.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.710+0000 7f8ce4ff9640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8cd40664a0 0x7f8cd4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:36.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.710+0000 7f8ce4ff9640 1 -- 172.21.15.46:0/4182020370 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 8748+0+0 (secure 0 0 0) 0x7f8ce0087e60 con 0x7f8ce81f34e0 2023-10-30T14:26:36.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.710+0000 7f8ce7fff640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8cd40664a0 0x7f8cd4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:36.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.710+0000 7f8ce7fff640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8cd40664a0 0x7f8cd4068960 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8009c80 tx=0x7f8cd8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:36.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.714+0000 7f8ce4ff9640 1 -- 172.21.15.46:0/4182020370 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8ce81f3200 con 0x7f8ce81f34e0 2023-10-30T14:26:36.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 -- 172.21.15.46:0/4182020370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8ce81f3200 con 0x7f8ce81f34e0 2023-10-30T14:26:36.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8ce4ff9640 1 -- 172.21.15.46:0/4182020370 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 66 entries v109) v1 ==== 81+0+4002 (secure 0 0 0) 0x7f8ce81f3200 con 0x7f8ce81f34e0 2023-10-30T14:26:36.890 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6836/3574404756 2023-10-31T14:24:59.863368+0000 2023-10-30T14:26:36.890 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3574404756 2023-10-31T14:24:59.863368+0000 2023-10-30T14:26:36.891 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6837/470164358 2023-10-31T14:23:54.843840+0000 2023-10-30T14:26:36.891 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/470164358 2023-10-31T14:23:54.843840+0000 2023-10-30T14:26:36.891 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/3764280378 2023-10-31T14:22:39.814064+0000 2023-10-30T14:26:36.891 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3764280378 2023-10-31T14:22:39.814064+0000 2023-10-30T14:26:36.892 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6836/2239735286 2023-10-31T14:20:39.772410+0000 2023-10-30T14:26:36.892 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2197729559 2023-10-31T13:51:39.216877+0000 2023-10-30T14:26:36.892 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/190134548 2023-10-31T13:50:04.191476+0000 2023-10-30T14:26:36.892 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/190134548 2023-10-31T13:50:04.191476+0000 2023-10-30T14:26:36.893 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/157453225 2023-10-31T14:00:29.382692+0000 2023-10-30T14:26:36.893 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6837/1135482564 2023-10-31T14:01:49.404286+0000 2023-10-30T14:26:36.893 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6837/3154171789 2023-10-31T14:07:14.520406+0000 2023-10-30T14:26:36.893 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6836/3037060424 2023-10-31T13:44:39.066140+0000 2023-10-30T14:26:36.894 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/1135482564 2023-10-31T14:01:49.404286+0000 2023-10-30T14:26:36.894 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/2572113428 2023-10-31T13:52:59.238225+0000 2023-10-30T14:26:36.894 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/2075693104 2023-10-31T14:05:39.493074+0000 2023-10-30T14:26:36.895 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6836/2197729559 2023-10-31T13:51:39.216877+0000 2023-10-30T14:26:36.895 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/994489214 2023-10-31T13:25:13.695228+0000 2023-10-30T14:26:36.895 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2075693104 2023-10-31T14:05:39.493074+0000 2023-10-30T14:26:36.895 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/4044310929 2023-10-31T13:41:14.000569+0000 2023-10-30T14:26:36.896 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6837/126953235 2023-10-31T13:39:08.955190+0000 2023-10-30T14:26:36.896 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/875791418 2023-10-31T13:26:28.713838+0000 2023-10-30T14:26:36.896 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/126953235 2023-10-31T13:39:08.955190+0000 2023-10-30T14:26:36.896 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6837/3112266825 2023-10-31T13:57:44.339823+0000 2023-10-30T14:26:36.897 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6836/3955931693 2023-10-31T13:36:53.920781+0000 2023-10-30T14:26:36.897 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3955931693 2023-10-31T13:36:53.920781+0000 2023-10-30T14:26:36.897 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/681817982 2023-10-31T14:18:44.741650+0000 2023-10-30T14:26:36.897 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6836/275749119 2023-10-31T13:30:28.793688+0000 2023-10-30T14:26:36.898 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6837/2572113428 2023-10-31T13:52:59.238225+0000 2023-10-30T14:26:36.898 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.154:6834/3109262047 2023-10-31T13:29:08.773403+0000 2023-10-30T14:26:36.898 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2239735286 2023-10-31T14:20:39.772410+0000 2023-10-30T14:26:36.898 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/2455118950 2023-10-31T14:16:04.689323+0000 2023-10-30T14:26:36.899 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6837/994489214 2023-10-31T13:25:13.695228+0000 2023-10-30T14:26:36.899 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6837/3044281789 2023-10-31T14:12:14.616071+0000 2023-10-30T14:26:36.899 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/1315246492 2023-10-31T13:31:23.809908+0000 2023-10-30T14:26:36.899 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/3112266825 2023-10-31T13:57:44.339823+0000 2023-10-30T14:26:36.900 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/128865315 2023-10-31T14:08:44.543673+0000 2023-10-30T14:26:36.900 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/603884510 2023-10-31T13:33:43.868205+0000 2023-10-30T14:26:36.900 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/2546436963 2023-10-31T13:47:24.113508+0000 2023-10-30T14:26:36.900 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6837/1315246492 2023-10-31T13:31:23.809908+0000 2023-10-30T14:26:36.901 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.154:6834/651441434 2023-10-31T13:35:03.892131+0000 2023-10-30T14:26:36.901 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/2807435195 2023-10-31T13:54:39.265053+0000 2023-10-30T14:26:36.901 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/711135361 2023-10-31T13:55:44.283031+0000 2023-10-30T14:26:36.901 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.154:6835/651441434 2023-10-31T13:35:03.892131+0000 2023-10-30T14:26:36.902 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6836/711135361 2023-10-31T13:55:44.283031+0000 2023-10-30T14:26:36.902 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6837/2546436963 2023-10-31T13:47:24.113508+0000 2023-10-30T14:26:36.902 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/875791418 2023-10-31T13:26:28.713838+0000 2023-10-30T14:26:36.902 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.154:6835/3109262047 2023-10-31T13:29:08.773403+0000 2023-10-30T14:26:36.903 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2807435195 2023-10-31T13:54:39.265053+0000 2023-10-30T14:26:36.903 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/157453225 2023-10-31T14:00:29.382692+0000 2023-10-30T14:26:36.903 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/603884510 2023-10-31T13:33:43.868205+0000 2023-10-30T14:26:36.903 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3734066164 2023-10-31T14:04:24.459794+0000 2023-10-30T14:26:36.904 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/4044310929 2023-10-31T13:41:14.000569+0000 2023-10-30T14:26:36.904 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/66265765 2023-10-31T14:14:54.671010+0000 2023-10-30T14:26:36.904 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6836/3734066164 2023-10-31T14:04:24.459794+0000 2023-10-30T14:26:36.904 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6836/128865315 2023-10-31T14:08:44.543673+0000 2023-10-30T14:26:36.905 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2455118950 2023-10-31T14:16:04.689323+0000 2023-10-30T14:26:36.905 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/3154171789 2023-10-31T14:07:14.520406+0000 2023-10-30T14:26:36.905 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/4078242540 2023-10-31T14:09:49.576202+0000 2023-10-30T14:26:36.905 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3037060424 2023-10-31T13:44:39.066140+0000 2023-10-30T14:26:36.905 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/4078242540 2023-10-31T14:09:49.576202+0000 2023-10-30T14:26:36.906 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/3044281789 2023-10-31T14:12:14.616071+0000 2023-10-30T14:26:36.906 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6836/66265765 2023-10-31T14:14:54.671010+0000 2023-10-30T14:26:36.906 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/275749119 2023-10-31T13:30:28.793688+0000 2023-10-30T14:26:36.906 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6837/681817982 2023-10-31T14:18:44.741650+0000 2023-10-30T14:26:36.907 INFO:teuthology.orchestra.run.smithi046.stderr:listed 66 entries 2023-10-30T14:26:36.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 -- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8cd40664a0 msgr2=0x7f8cd4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8cd40664a0 0x7f8cd4068960 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8009c80 tx=0x7f8cd8009190 comp rx=0 tx=0).stop 2023-10-30T14:26:36.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 -- 172.21.15.46:0/4182020370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ce81f34e0 msgr2=0x7f8ce81f3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:36.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ce81f34e0 0x7f8ce81f3960 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8157460 tx=0x7f8ce0011ee0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 -- 172.21.15.46:0/4182020370 shutdown_connections 2023-10-30T14:26:36.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8cd40664a0 0x7f8cd4068960 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ce81f34e0 0x7f8ce81f3960 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce80cf140 0x7f8ce80cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 --2- 172.21.15.46:0/4182020370 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cd00047d0 0x7f8ce8149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:36.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 -- 172.21.15.46:0/4182020370 >> 172.21.15.46:0/4182020370 conn(0x7f8ce806bfe0 msgr2=0x7f8ce80cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:36.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 -- 172.21.15.46:0/4182020370 shutdown_connections 2023-10-30T14:26:36.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:36.886+0000 7f8cee291640 1 -- 172.21.15.46:0/4182020370 wait complete. 2023-10-30T14:26:36.912 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-30T14:26:36.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:26:36.912 DEBUG:teuthology.orchestra.run.smithi046:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-30T14:26:37.028 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2023-10-30T14:26:37.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:26:37.029 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-30T14:26:37.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:26:37.068 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-10-30T14:26:37.165 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-30T14:26:37.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:26:37.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-30T14:26:37.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T14:26:37.189 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2023-10-30T14:26:37.406 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-30T14:26:37.416 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T14:26:37.416 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:26:37.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc687a1640 1 Processor -- start 2023-10-30T14:26:37.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc687a1640 1 -- start start 2023-10-30T14:26:37.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc687a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc600cf160 0x7efc600cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc687a1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600cb800 con 0x7efc6015d6f0 2023-10-30T14:26:37.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc687a1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600cb330 con 0x7efc601538a0 2023-10-30T14:26:37.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc687a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc600cad70 con 0x7efc600cf160 2023-10-30T14:26:37.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc65d3c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efc6015d6f0 0x7efc6015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38006/0 (socket says 172.21.15.46:38006) 2023-10-30T14:26:37.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc65d3c640 1 -- 172.21.15.46:0/1887593617 learned_addr learned my addr 172.21.15.46:0/1887593617 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:37.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc66d3e640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc600cf160 0x7efc600cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628053670 0 0) 0x7efc600cb800 con 0x7efc6015d6f0 2023-10-30T14:26:37.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc4c003530 con 0x7efc6015d6f0 2023-10-30T14:26:37.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490334044 0 0) 0x7efc600cb330 con 0x7efc601538a0 2023-10-30T14:26:37.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc600cb800 con 0x7efc601538a0 2023-10-30T14:26:37.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1670099364 0 0) 0x7efc4c003530 con 0x7efc6015d6f0 2023-10-30T14:26:37.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc600cf160 msgr2=0x7efc600cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:37.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc600cf160 0x7efc600cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 >> v1:172.21.15.154:6789/0 conn(0x7efc601538a0 legacy=0x7efc600ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:37.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc600cb050 con 0x7efc6015d6f0 2023-10-30T14:26:37.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7efc50001680 con 0x7efc6015d6f0 2023-10-30T14:26:37.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 >> v1:172.21.15.46:6789/0 conn(0x7efc6015d6f0 legacy=0x7efc6015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:37.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc4c003b40 0x7efc4c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc4c0047d0 0x7efc4c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc600cf160 0x7efc4c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc600cad70 con 0x7efc4c0047d0 2023-10-30T14:26:37.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c00ba40 con 0x7efc4c003b40 2023-10-30T14:26:37.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c00bbd0 con 0x7efc600cf160 2023-10-30T14:26:37.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc65d3c640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc4c0047d0 0x7efc4c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc66d3e640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc600cf160 0x7efc4c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6653d640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc4c003b40 0x7efc4c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc65d3c640 1 -- 172.21.15.46:0/1887593617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc600cf160 msgr2=0x7efc4c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:37.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc65d3c640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc600cf160 0x7efc4c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc65d3c640 1 -- 172.21.15.46:0/1887593617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc4c003b40 msgr2=0x7efc4c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:37.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc65d3c640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc4c003b40 0x7efc4c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc65d3c640 1 -- 172.21.15.46:0/1887593617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c003530 con 0x7efc4c0047d0 2023-10-30T14:26:37.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.558+0000 7efc6653d640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc4c003b40 0x7efc4c003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:26:37.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.562+0000 7efc65d3c640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc4c0047d0 0x7efc4c004c50 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7efc5000c160 tx=0x7efc5001f2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5612b00c717ea800 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:37.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.562+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc500a8d00 con 0x7efc4c0047d0 2023-10-30T14:26:37.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.562+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc500a8e80 con 0x7efc4c0047d0 2023-10-30T14:26:37.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.562+0000 7efc6553b640 1 -- 172.21.15.46:0/1887593617 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc50005b20 con 0x7efc4c0047d0 2023-10-30T14:26:37.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.562+0000 7efc687a1640 1 -- 172.21.15.46:0/1887593617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc4c0047d0 msgr2=0x7efc4c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:37.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.562+0000 7efc687a1640 1 --2- 172.21.15.46:0/1887593617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc4c0047d0 0x7efc4c004c50 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7efc5000c160 tx=0x7efc5001f2d0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.562+0000 7efc66d3e640 1 -- 172.21.15.46:0/1887593617 reap_dead start 2023-10-30T14:26:37.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.562+0000 7efc687a1640 1 -- 172.21.15.46:0/1887593617 shutdown_connections 2023-10-30T14:26:37.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.562+0000 7efc687a1640 1 -- 172.21.15.46:0/1887593617 >> 172.21.15.46:0/1887593617 conn(0x7efc6006bfe0 msgr2=0x7efc6006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:37.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.562+0000 7efc687a1640 1 -- 172.21.15.46:0/1887593617 shutdown_connections 2023-10-30T14:26:37.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.562+0000 7efc687a1640 1 -- 172.21.15.46:0/1887593617 wait complete. 2023-10-30T14:26:37.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.566+0000 7efc687a1640 1 Processor -- start 2023-10-30T14:26:37.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.566+0000 7efc687a1640 1 -- start start 2023-10-30T14:26:37.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.566+0000 7efc687a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc4c0047d0 0x7efc6014e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.566+0000 7efc687a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc6015b6c0 0x7efc6014f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.566+0000 7efc687a1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc6014fbf0 0x7efc6014c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.566+0000 7efc687a1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6014f240 con 0x7efc4c0047d0 2023-10-30T14:26:37.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.566+0000 7efc687a1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc60150050 con 0x7efc6014fbf0 2023-10-30T14:26:37.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.566+0000 7efc687a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc601f2d60 con 0x7efc6015b6c0 2023-10-30T14:26:37.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc6653d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc4c0047d0 0x7efc6014e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc6653d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc4c0047d0 0x7efc6014e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43532/0 (socket says 172.21.15.46:43532) 2023-10-30T14:26:37.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc6653d640 1 -- 172.21.15.46:0/3988598462 learned_addr learned my addr 172.21.15.46:0/3988598462 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:37.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc66d3e640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc6014fbf0 0x7efc6014c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc65d3c640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc6015b6c0 0x7efc6014f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc6653d640 1 -- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc6015b6c0 msgr2=0x7efc6014f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:37.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc6653d640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc6015b6c0 0x7efc6014f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc6653d640 1 -- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc6014fbf0 msgr2=0x7efc6014c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:37.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc6653d640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc6014fbf0 0x7efc6014c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc6653d640 1 -- 172.21.15.46:0/3988598462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc601f3020 con 0x7efc4c0047d0 2023-10-30T14:26:37.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc66d3e640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc6014fbf0 0x7efc6014c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc6653d640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc4c0047d0 0x7efc6014e9c0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7efc5c001ec0 tx=0x7efc5c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc56ffd640 1 -- 172.21.15.46:0/3988598462 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c00bac0 con 0x7efc4c0047d0 2023-10-30T14:26:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc56ffd640 1 -- 172.21.15.46:0/3988598462 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5c00bc20 con 0x7efc4c0047d0 2023-10-30T14:26:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc687a1640 1 -- 172.21.15.46:0/3988598462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc600c6570 con 0x7efc4c0047d0 2023-10-30T14:26:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc56ffd640 1 -- 172.21.15.46:0/3988598462 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c00de50 con 0x7efc4c0047d0 2023-10-30T14:26:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc687a1640 1 -- 172.21.15.46:0/3988598462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc600c6b30 con 0x7efc4c0047d0 2023-10-30T14:26:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc687a1640 1 -- 172.21.15.46:0/3988598462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6004e170 con 0x7efc4c0047d0 2023-10-30T14:26:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.570+0000 7efc56ffd640 1 -- 172.21.15.46:0/3988598462 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efc5c007020 con 0x7efc4c0047d0 2023-10-30T14:26:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.574+0000 7efc56ffd640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efc440664a0 0x7efc44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.574+0000 7efc56ffd640 1 -- 172.21.15.46:0/3988598462 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 8748+0+0 (secure 0 0 0) 0x7efc5c084990 con 0x7efc4c0047d0 2023-10-30T14:26:37.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.574+0000 7efc65d3c640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efc440664a0 0x7efc44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.574+0000 7efc65d3c640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efc440664a0 0x7efc44068960 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7efc500ad000 tx=0x7efc50001a00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:37.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.578+0000 7efc56ffd640 1 -- 172.21.15.46:0/3988598462 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efc5c051210 con 0x7efc4c0047d0 2023-10-30T14:26:37.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.730+0000 7efc687a1640 1 -- 172.21.15.46:0/3988598462 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc60159b70 con 0x7efc440664a0 2023-10-30T14:26:37.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.754+0000 7efc56ffd640 1 -- 172.21.15.46:0/3988598462 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+295003 (secure 0 0 0) 0x7efc500a8d80 con 0x7efc440664a0 2023-10-30T14:26:37.760 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:26:37.763 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T14:26:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.758+0000 7efc687a1640 1 -- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efc440664a0 msgr2=0x7efc44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.758+0000 7efc687a1640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efc440664a0 0x7efc44068960 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7efc500ad000 tx=0x7efc50001a00 comp rx=0 tx=0).stop 2023-10-30T14:26:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.758+0000 7efc687a1640 1 -- 172.21.15.46:0/3988598462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc4c0047d0 msgr2=0x7efc6014e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:37.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.758+0000 7efc687a1640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc4c0047d0 0x7efc6014e9c0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7efc5c001ec0 tx=0x7efc5c00dc00 comp rx=0 tx=0).stop 2023-10-30T14:26:37.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.758+0000 7efc687a1640 1 -- 172.21.15.46:0/3988598462 shutdown_connections 2023-10-30T14:26:37.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.758+0000 7efc687a1640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7efc440664a0 0x7efc44068960 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.758+0000 7efc687a1640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc6014fbf0 0x7efc6014c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.758+0000 7efc687a1640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc6015b6c0 0x7efc6014f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.758+0000 7efc687a1640 1 --2- 172.21.15.46:0/3988598462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc4c0047d0 0x7efc6014e9c0 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.758+0000 7efc687a1640 1 -- 172.21.15.46:0/3988598462 >> 172.21.15.46:0/3988598462 conn(0x7efc6006bfe0 msgr2=0x7efc6015aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.758+0000 7efc687a1640 1 -- 172.21.15.46:0/3988598462 shutdown_connections 2023-10-30T14:26:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.758+0000 7efc687a1640 1 -- 172.21.15.46:0/3988598462 wait complete. 2023-10-30T14:26:37.781 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":2007,"stamp":"2023-10-30T14:26:36.126675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1284301339,"num_objects":3948,"num_object_clones":0,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":20155,"num_read_kb":14487316,"num_write":119346,"num_write_kb":3804643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170087,"ondisk_log_size":170087,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3917396,"kb_used_data":2519216,"kb_used_omap":12,"kb_used_meta":1398067,"kb_avail":751057324,"statfs":{"total":773094113280,"available":769082699776,"internally_reserved":0,"allocated":2579677184,"data_stored":2574617905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1431621176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1662389,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":58,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001605"},"pg_stats":[{"pgid":"3.3f","version":"103'2152","reported_seq":4910,"reported_epoch":107,"state":"active+clean+snaptrim","last_fresh":"2023-10-30T14:26:31.405824+0000","last_change":"2023-10-30T14:26:31.405864+0000","last_active":"2023-10-30T14:26:31.405824+0000","last_peered":"2023-10-30T14:26:31.405824+0000","last_clean":"2023-10-30T14:26:31.405824+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T14:26:31.405824+0000","last_undegraded":"2023-10-30T14:26:31.405824+0000","last_fullsized":"2023-10-30T14:26:31.405824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2152,"log_dups_size":0,"ondisk_log_size":2152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:03:07.458179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20174655,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":124,"num_read_kb":10662,"num_write":1276,"num_write_kb":15944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"3.3e","version":"103'1964","reported_seq":4779,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.147502+0000","last_change":"2023-10-30T14:26:29.072778+0000","last_active":"2023-10-30T14:26:30.147502+0000","last_peered":"2023-10-30T14:26:30.147502+0000","last_clean":"2023-10-30T14:26:30.147502+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T14:26:30.147502+0000","last_undegraded":"2023-10-30T14:26:30.147502+0000","last_fullsized":"2023-10-30T14:26:30.147502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1964,"log_dups_size":0,"ondisk_log_size":1964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:58:36.734493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42379451400000001,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":133,"num_read_kb":21158,"num_write":1230,"num_write_kb":16620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"3.3d","version":"103'2090","reported_seq":4870,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.242867+0000","last_change":"2023-10-30T14:26:31.242904+0000","last_active":"2023-10-30T14:26:31.242867+0000","last_peered":"2023-10-30T14:26:31.242867+0000","last_clean":"2023-10-30T14:26:31.242867+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T14:26:31.242867+0000","last_undegraded":"2023-10-30T14:26:31.242867+0000","last_fullsized":"2023-10-30T14:26:31.242867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:41.150471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35104858900000002,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":129,"num_read_kb":29825,"num_write":1270,"num_write_kb":28412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"3.3c","version":"103'2021","reported_seq":4859,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.894425+0000","last_change":"2023-10-30T14:26:33.286784+0000","last_active":"2023-10-30T14:26:34.894425+0000","last_peered":"2023-10-30T14:26:34.894425+0000","last_clean":"2023-10-30T14:26:34.894425+0000","last_became_active":"2023-10-30T13:24:00.704829+0000","last_became_peered":"2023-10-30T13:24:00.704829+0000","last_unstale":"2023-10-30T14:26:34.894425+0000","last_undegraded":"2023-10-30T14:26:34.894425+0000","last_fullsized":"2023-10-30T14:26:34.894425+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2021,"log_dups_size":0,"ondisk_log_size":2021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:28.307918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.361435862,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":124,"num_read_kb":7939,"num_write":1184,"num_write_kb":24239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.3b","version":"103'2001","reported_seq":4802,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.158385+0000","last_change":"2023-10-30T14:26:31.158421+0000","last_active":"2023-10-30T14:26:31.158385+0000","last_peered":"2023-10-30T14:26:31.158385+0000","last_clean":"2023-10-30T14:26:31.158385+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T14:26:31.158385+0000","last_undegraded":"2023-10-30T14:26:31.158385+0000","last_fullsized":"2023-10-30T14:26:31.158385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2001,"log_dups_size":0,"ondisk_log_size":2001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:39.126803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.18308748399999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":41266,"num_write":1244,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"3.3a","version":"103'2205","reported_seq":4940,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.602121+0000","last_change":"2023-10-30T14:26:28.487481+0000","last_active":"2023-10-30T14:26:30.602121+0000","last_peered":"2023-10-30T14:26:30.602121+0000","last_clean":"2023-10-30T14:26:30.602121+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T14:26:30.602121+0000","last_undegraded":"2023-10-30T14:26:30.602121+0000","last_fullsized":"2023-10-30T14:26:30.602121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2205,"log_dups_size":0,"ondisk_log_size":2205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:40.491215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.14173730100000001,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":141,"num_read_kb":27429,"num_write":1290,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"3.39","version":"103'2138","reported_seq":4934,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.323849+0000","last_change":"2023-10-30T14:26:31.324108+0000","last_active":"2023-10-30T14:26:31.323849+0000","last_peered":"2023-10-30T14:26:31.323849+0000","last_clean":"2023-10-30T14:26:31.323849+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T14:26:31.323849+0000","last_undegraded":"2023-10-30T14:26:31.323849+0000","last_fullsized":"2023-10-30T14:26:31.323849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2138,"log_dups_size":0,"ondisk_log_size":2138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:49.843722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.349248691,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":146,"num_read_kb":25110,"num_write":1316,"num_write_kb":20043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"3.38","version":"103'2099","reported_seq":4854,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.034968+0000","last_change":"2023-10-30T14:26:31.035012+0000","last_active":"2023-10-30T14:26:31.034968+0000","last_peered":"2023-10-30T14:26:31.034968+0000","last_clean":"2023-10-30T14:26:31.034968+0000","last_became_active":"2023-10-30T13:24:00.989938+0000","last_became_peered":"2023-10-30T13:24:00.989938+0000","last_unstale":"2023-10-30T14:26:31.034968+0000","last_undegraded":"2023-10-30T14:26:31.034968+0000","last_fullsized":"2023-10-30T14:26:31.034968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:35.910150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.094879888999999995,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":124,"num_read_kb":25628,"num_write":1232,"num_write_kb":20540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"3.37","version":"103'2126","reported_seq":4901,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.229138+0000","last_change":"2023-10-30T14:26:31.229181+0000","last_active":"2023-10-30T14:26:31.229138+0000","last_peered":"2023-10-30T14:26:31.229138+0000","last_clean":"2023-10-30T14:26:31.229138+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T14:26:31.229138+0000","last_undegraded":"2023-10-30T14:26:31.229138+0000","last_fullsized":"2023-10-30T14:26:31.229138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2126,"log_dups_size":0,"ondisk_log_size":2126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:42:58.889203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.25097449300000002,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":24548,"num_write":1310,"num_write_kb":19821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"3.36","version":"103'2060","reported_seq":4882,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.505970+0000","last_change":"2023-10-30T14:26:31.506091+0000","last_active":"2023-10-30T14:26:31.505970+0000","last_peered":"2023-10-30T14:26:31.505970+0000","last_clean":"2023-10-30T14:26:31.505970+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T14:26:31.505970+0000","last_undegraded":"2023-10-30T14:26:31.505970+0000","last_fullsized":"2023-10-30T14:26:31.505970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2060,"log_dups_size":0,"ondisk_log_size":2060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:40:10.106446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.60487880000000005,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":148,"num_read_kb":64678,"num_write":1273,"num_write_kb":32495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2ae","length":"9"}]},{"pgid":"3.35","version":"105'1999","reported_seq":4829,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.404244+0000","last_change":"2023-10-30T14:26:31.404282+0000","last_active":"2023-10-30T14:26:31.404244+0000","last_peered":"2023-10-30T14:26:31.404244+0000","last_clean":"2023-10-30T14:26:31.404244+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T14:26:31.404244+0000","last_undegraded":"2023-10-30T14:26:31.404244+0000","last_fullsized":"2023-10-30T14:26:31.404244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1999,"log_dups_size":0,"ondisk_log_size":1999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:37:07.300154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50371527999999999,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":139,"num_read_kb":25550,"num_write":1254,"num_write_kb":16915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2ae","length":"8"}]},{"pgid":"3.34","version":"103'2073","reported_seq":4865,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.182015+0000","last_change":"2023-10-30T14:26:31.182171+0000","last_active":"2023-10-30T14:26:31.182015+0000","last_peered":"2023-10-30T14:26:31.182015+0000","last_clean":"2023-10-30T14:26:31.182015+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T14:26:31.182015+0000","last_undegraded":"2023-10-30T14:26:31.182015+0000","last_fullsized":"2023-10-30T14:26:31.182015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:01:21.539767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.27800623400000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":142,"num_read_kb":52725,"num_write":1240,"num_write_kb":42463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"3.33","version":"103'2219","reported_seq":4971,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.348005+0000","last_change":"2023-10-30T14:26:31.348042+0000","last_active":"2023-10-30T14:26:31.348005+0000","last_peered":"2023-10-30T14:26:31.348005+0000","last_clean":"2023-10-30T14:26:31.348005+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T14:26:31.348005+0000","last_undegraded":"2023-10-30T14:26:31.348005+0000","last_fullsized":"2023-10-30T14:26:31.348005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2219,"log_dups_size":0,"ondisk_log_size":2219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:51:49.601891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40741854599999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":46739,"num_write":1326,"num_write_kb":33562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"3.32","version":"103'2145","reported_seq":4942,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.151304+0000","last_change":"2023-10-30T14:26:31.151348+0000","last_active":"2023-10-30T14:26:31.151304+0000","last_peered":"2023-10-30T14:26:31.151304+0000","last_clean":"2023-10-30T14:26:31.151304+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T14:26:31.151304+0000","last_undegraded":"2023-10-30T14:26:31.151304+0000","last_fullsized":"2023-10-30T14:26:31.151304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2145,"log_dups_size":0,"ondisk_log_size":2145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:46.312532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.259903049,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":149,"num_read_kb":13007,"num_write":1312,"num_write_kb":16233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"3.31","version":"103'2258","reported_seq":5019,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.146736+0000","last_change":"2023-10-30T14:26:29.112256+0000","last_active":"2023-10-30T14:26:30.146736+0000","last_peered":"2023-10-30T14:26:30.146736+0000","last_clean":"2023-10-30T14:26:30.146736+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T14:26:30.146736+0000","last_undegraded":"2023-10-30T14:26:30.146736+0000","last_fullsized":"2023-10-30T14:26:30.146736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2258,"log_dups_size":0,"ondisk_log_size":2258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:54.270851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.462737495,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":161,"num_read_kb":14756,"num_write":1377,"num_write_kb":22909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"3.30","version":"103'2085","reported_seq":4881,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.601996+0000","last_change":"2023-10-30T14:26:28.492885+0000","last_active":"2023-10-30T14:26:30.601996+0000","last_peered":"2023-10-30T14:26:30.601996+0000","last_clean":"2023-10-30T14:26:30.601996+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T14:26:30.601996+0000","last_undegraded":"2023-10-30T14:26:30.601996+0000","last_fullsized":"2023-10-30T14:26:30.601996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2085,"log_dups_size":0,"ondisk_log_size":2085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:45:32.212694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.14735420599999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":149,"num_read_kb":40211,"num_write":1268,"num_write_kb":23485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"3.2f","version":"104'2338","reported_seq":5089,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.602896+0000","last_change":"2023-10-30T14:26:28.500358+0000","last_active":"2023-10-30T14:26:30.602896+0000","last_peered":"2023-10-30T14:26:30.602896+0000","last_clean":"2023-10-30T14:26:30.602896+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T14:26:30.602896+0000","last_undegraded":"2023-10-30T14:26:30.602896+0000","last_fullsized":"2023-10-30T14:26:30.602896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2338,"log_dups_size":0,"ondisk_log_size":2338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:33:17.494453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.146315691,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":170,"num_read_kb":39932,"num_write":1446,"num_write_kb":30153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"3.2e","version":"103'1987","reported_seq":4842,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.895081+0000","last_change":"2023-10-30T14:26:33.267575+0000","last_active":"2023-10-30T14:26:34.895081+0000","last_peered":"2023-10-30T14:26:34.895081+0000","last_clean":"2023-10-30T14:26:34.895081+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T14:26:34.895081+0000","last_undegraded":"2023-10-30T14:26:34.895081+0000","last_fullsized":"2023-10-30T14:26:34.895081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1987,"log_dups_size":0,"ondisk_log_size":1987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:03.867577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.34246037800000001,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":137,"num_read_kb":13706,"num_write":1196,"num_write_kb":12790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2d","version":"103'2087","reported_seq":4863,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.211317+0000","last_change":"2023-10-30T14:26:31.211355+0000","last_active":"2023-10-30T14:26:31.211317+0000","last_peered":"2023-10-30T14:26:31.211317+0000","last_clean":"2023-10-30T14:26:31.211317+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T14:26:31.211317+0000","last_undegraded":"2023-10-30T14:26:31.211317+0000","last_fullsized":"2023-10-30T14:26:31.211317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2087,"log_dups_size":0,"ondisk_log_size":2087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:09:19.169501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.233393294,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":125,"num_read_kb":30178,"num_write":1282,"num_write_kb":24413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"3.2c","version":"103'2210","reported_seq":5000,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.480584+0000","last_change":"2023-10-30T14:26:31.480632+0000","last_active":"2023-10-30T14:26:31.480584+0000","last_peered":"2023-10-30T14:26:31.480584+0000","last_clean":"2023-10-30T14:26:31.480584+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T14:26:31.480584+0000","last_undegraded":"2023-10-30T14:26:31.480584+0000","last_fullsized":"2023-10-30T14:26:31.480584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:50:05.961878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.57258774599999995,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":148,"num_read_kb":30111,"num_write":1332,"num_write_kb":26597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2ae","length":"9"}]},{"pgid":"3.2b","version":"103'2218","reported_seq":4955,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.225503+0000","last_change":"2023-10-30T14:26:31.225546+0000","last_active":"2023-10-30T14:26:31.225503+0000","last_peered":"2023-10-30T14:26:31.225503+0000","last_clean":"2023-10-30T14:26:31.225503+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T14:26:31.225503+0000","last_undegraded":"2023-10-30T14:26:31.225503+0000","last_fullsized":"2023-10-30T14:26:31.225503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2218,"log_dups_size":0,"ondisk_log_size":2218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:13.743237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.333760422,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":127,"num_read_kb":10449,"num_write":1298,"num_write_kb":11653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"3.2a","version":"103'2269","reported_seq":5007,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.602927+0000","last_change":"2023-10-30T14:26:28.495192+0000","last_active":"2023-10-30T14:26:30.602927+0000","last_peered":"2023-10-30T14:26:30.602927+0000","last_clean":"2023-10-30T14:26:30.602927+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T14:26:30.602927+0000","last_undegraded":"2023-10-30T14:26:30.602927+0000","last_fullsized":"2023-10-30T14:26:30.602927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2269,"log_dups_size":0,"ondisk_log_size":2269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:13.951505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.14835188599999999,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":144,"num_read_kb":30506,"num_write":1376,"num_write_kb":26822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"3.29","version":"103'2106","reported_seq":5013,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.178044+0000","last_change":"2023-10-30T14:26:31.178095+0000","last_active":"2023-10-30T14:26:31.178044+0000","last_peered":"2023-10-30T14:26:31.178044+0000","last_clean":"2023-10-30T14:26:31.178044+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T14:26:31.178044+0000","last_undegraded":"2023-10-30T14:26:31.178044+0000","last_fullsized":"2023-10-30T14:26:31.178044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2106,"log_dups_size":0,"ondisk_log_size":2106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:24:06.652678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.23788753600000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":255,"num_read_kb":42144,"num_write":1294,"num_write_kb":25263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"3.28","version":"103'2167","reported_seq":4966,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.601534+0000","last_change":"2023-10-30T14:26:28.490377+0000","last_active":"2023-10-30T14:26:30.601534+0000","last_peered":"2023-10-30T14:26:30.601534+0000","last_clean":"2023-10-30T14:26:30.601534+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T14:26:30.601534+0000","last_undegraded":"2023-10-30T14:26:30.601534+0000","last_fullsized":"2023-10-30T14:26:30.601534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2167,"log_dups_size":0,"ondisk_log_size":2167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:49:32.111076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.144462494,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":164,"num_read_kb":14461,"num_write":1342,"num_write_kb":12663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"3.27","version":"103'2051","reported_seq":4835,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.467460+0000","last_change":"2023-10-30T14:26:31.467505+0000","last_active":"2023-10-30T14:26:31.467460+0000","last_peered":"2023-10-30T14:26:31.467460+0000","last_clean":"2023-10-30T14:26:31.467460+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T14:26:31.467460+0000","last_undegraded":"2023-10-30T14:26:31.467460+0000","last_fullsized":"2023-10-30T14:26:31.467460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2051,"log_dups_size":0,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:05:52.100105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.52636625299999995,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":133,"num_read_kb":25337,"num_write":1220,"num_write_kb":15637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"3.26","version":"103'2135","reported_seq":4970,"reported_epoch":107,"state":"active+clean+snaptrim","last_fresh":"2023-10-30T14:26:31.505889+0000","last_change":"2023-10-30T14:26:31.505921+0000","last_active":"2023-10-30T14:26:31.505889+0000","last_peered":"2023-10-30T14:26:31.505889+0000","last_clean":"2023-10-30T14:26:31.505889+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T14:26:31.505889+0000","last_undegraded":"2023-10-30T14:26:31.505889+0000","last_fullsized":"2023-10-30T14:26:31.505889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:27.058123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.53742728900000003,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":160,"num_read_kb":14034,"num_write":1340,"num_write_kb":12359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2ae","length":"8"}]},{"pgid":"3.25","version":"103'2083","reported_seq":4869,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.147335+0000","last_change":"2023-10-30T14:26:29.095721+0000","last_active":"2023-10-30T14:26:30.147335+0000","last_peered":"2023-10-30T14:26:30.147335+0000","last_clean":"2023-10-30T14:26:30.147335+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T14:26:30.147335+0000","last_undegraded":"2023-10-30T14:26:30.147335+0000","last_fullsized":"2023-10-30T14:26:30.147335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:12.858564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.445938259,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":128,"num_read_kb":22518,"num_write":1277,"num_write_kb":32107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"3.24","version":"103'2025","reported_seq":4794,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:30.941416+0000","last_change":"2023-10-30T14:26:30.941399+0000","last_active":"2023-10-30T14:26:30.941416+0000","last_peered":"2023-10-30T14:26:30.941416+0000","last_clean":"2023-10-30T14:26:30.941416+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T14:26:30.941416+0000","last_undegraded":"2023-10-30T14:26:30.941416+0000","last_fullsized":"2023-10-30T14:26:30.941416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2025,"log_dups_size":0,"ondisk_log_size":2025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:47:48.502233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.73649854000000003,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":129,"num_read_kb":17081,"num_write":1202,"num_write_kb":15468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"3.23","version":"103'2131","reported_seq":4901,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.167796+0000","last_change":"2023-10-30T14:26:31.167835+0000","last_active":"2023-10-30T14:26:31.167796+0000","last_peered":"2023-10-30T14:26:31.167796+0000","last_clean":"2023-10-30T14:26:31.167796+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T14:26:31.167796+0000","last_undegraded":"2023-10-30T14:26:31.167796+0000","last_fullsized":"2023-10-30T14:26:31.167796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2131,"log_dups_size":0,"ondisk_log_size":2131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:07.264909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.27606367199999998,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":125,"num_read_kb":13655,"num_write":1323,"num_write_kb":17357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"3.22","version":"103'2073","reported_seq":4879,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.603421+0000","last_change":"2023-10-30T14:26:28.490431+0000","last_active":"2023-10-30T14:26:30.603421+0000","last_peered":"2023-10-30T14:26:30.603421+0000","last_clean":"2023-10-30T14:26:30.603421+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T14:26:30.603421+0000","last_undegraded":"2023-10-30T14:26:30.603421+0000","last_fullsized":"2023-10-30T14:26:30.603421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:21.447484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.14455268900000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":140,"num_read_kb":17880,"num_write":1311,"num_write_kb":13902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"3.21","version":"108'2369","reported_seq":5183,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.894065+0000","last_change":"2023-10-30T14:26:33.231592+0000","last_active":"2023-10-30T14:26:34.894065+0000","last_peered":"2023-10-30T14:26:34.894065+0000","last_clean":"2023-10-30T14:26:34.894065+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T14:26:34.894065+0000","last_undegraded":"2023-10-30T14:26:34.894065+0000","last_fullsized":"2023-10-30T14:26:34.894065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2369,"log_dups_size":0,"ondisk_log_size":2369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:00:56.118846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30670995400000001,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":165,"num_read_kb":40193,"num_write":1443,"num_write_kb":35290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.20","version":"105'2296","reported_seq":5075,"reported_epoch":107,"state":"active+clean+snaptrim","last_fresh":"2023-10-30T14:26:31.438526+0000","last_change":"2023-10-30T14:26:31.438565+0000","last_active":"2023-10-30T14:26:31.438526+0000","last_peered":"2023-10-30T14:26:31.438526+0000","last_clean":"2023-10-30T14:26:31.438526+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T14:26:31.438526+0000","last_undegraded":"2023-10-30T14:26:31.438526+0000","last_fullsized":"2023-10-30T14:26:31.438526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2296,"log_dups_size":0,"ondisk_log_size":2296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:34:17.435850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.43994382799999998,"stat_sum":{"num_bytes":10,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":168,"num_read_kb":24860,"num_write":1382,"num_write_kb":20782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2ae","length":"8"}]},{"pgid":"3.1a","version":"103'2103","reported_seq":4893,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.139091+0000","last_change":"2023-10-30T14:26:31.139130+0000","last_active":"2023-10-30T14:26:31.139091+0000","last_peered":"2023-10-30T14:26:31.139091+0000","last_clean":"2023-10-30T14:26:31.139091+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T14:26:31.139091+0000","last_undegraded":"2023-10-30T14:26:31.139091+0000","last_fullsized":"2023-10-30T14:26:31.139091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:48:24.469934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.235557462,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":146,"num_read_kb":30802,"num_write":1268,"num_write_kb":21419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"2.1b","version":"103'1028","reported_seq":1332,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:29.954946+0000","last_change":"2023-10-30T13:23:18.963933+0000","last_active":"2023-10-30T14:26:29.954946+0000","last_peered":"2023-10-30T14:26:29.954946+0000","last_clean":"2023-10-30T14:26:29.954946+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T14:26:29.954946+0000","last_undegraded":"2023-10-30T14:26:29.954946+0000","last_fullsized":"2023-10-30T14:26:29.954946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1028,"log_dups_size":0,"ondisk_log_size":1028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:48:58.196567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497506,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":222,"num_read_kb":501605,"num_write":1094,"num_write_kb":95785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"103'2148","reported_seq":4898,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.075925+0000","last_change":"2023-10-30T14:26:31.075970+0000","last_active":"2023-10-30T14:26:31.075925+0000","last_peered":"2023-10-30T14:26:31.075925+0000","last_clean":"2023-10-30T14:26:31.075925+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T14:26:31.075925+0000","last_undegraded":"2023-10-30T14:26:31.075925+0000","last_fullsized":"2023-10-30T14:26:31.075925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:01.414300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.183980063,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":17524,"num_write":1294,"num_write_kb":15816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"2.1a","version":"103'1692","reported_seq":2123,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:29.899004+0000","last_change":"2023-10-30T13:23:18.976615+0000","last_active":"2023-10-30T14:26:29.899004+0000","last_peered":"2023-10-30T14:26:29.899004+0000","last_clean":"2023-10-30T14:26:29.899004+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T14:26:29.899004+0000","last_undegraded":"2023-10-30T14:26:29.899004+0000","last_fullsized":"2023-10-30T14:26:29.899004+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1692,"log_dups_size":0,"ondisk_log_size":1692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:59.048396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29694842,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":392,"num_read_kb":410068,"num_write":1792,"num_write_kb":89526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"104'1557","reported_seq":2908,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.940416+0000","last_change":"2023-10-30T13:23:18.977357+0000","last_active":"2023-10-30T14:26:30.940416+0000","last_peered":"2023-10-30T14:26:30.940416+0000","last_clean":"2023-10-30T14:26:30.940416+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T14:26:30.940416+0000","last_undegraded":"2023-10-30T14:26:30.940416+0000","last_fullsized":"2023-10-30T14:26:30.940416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1557,"log_dups_size":0,"ondisk_log_size":1557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:53.502681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60855928,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":1296,"num_read_kb":556671,"num_write":1596,"num_write_kb":94675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"103'2314","reported_seq":5094,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.070741+0000","last_change":"2023-10-30T14:26:31.070781+0000","last_active":"2023-10-30T14:26:31.070741+0000","last_peered":"2023-10-30T14:26:31.070741+0000","last_clean":"2023-10-30T14:26:31.070741+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T14:26:31.070741+0000","last_undegraded":"2023-10-30T14:26:31.070741+0000","last_fullsized":"2023-10-30T14:26:31.070741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2314,"log_dups_size":0,"ondisk_log_size":2314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:51:54.566301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.130565722,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":161,"num_read_kb":17698,"num_write":1449,"num_write_kb":26122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"2.18","version":"106'939","reported_seq":1282,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.602635+0000","last_change":"2023-10-30T13:23:18.989299+0000","last_active":"2023-10-30T14:26:30.602635+0000","last_peered":"2023-10-30T14:26:30.602635+0000","last_clean":"2023-10-30T14:26:30.602635+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T14:26:30.602635+0000","last_undegraded":"2023-10-30T14:26:30.602635+0000","last_fullsized":"2023-10-30T14:26:30.602635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":939,"log_dups_size":0,"ondisk_log_size":939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:44.510184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52619898,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":312,"num_read_kb":563254,"num_write":1045,"num_write_kb":95040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"105'2142","reported_seq":4936,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.146681+0000","last_change":"2023-10-30T14:26:29.006860+0000","last_active":"2023-10-30T14:26:30.146681+0000","last_peered":"2023-10-30T14:26:30.146681+0000","last_clean":"2023-10-30T14:26:30.146681+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T14:26:30.146681+0000","last_undegraded":"2023-10-30T14:26:30.146681+0000","last_fullsized":"2023-10-30T14:26:30.146681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:09:58.393359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.359708678,"stat_sum":{"num_bytes":10,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":152,"num_read_kb":51949,"num_write":1320,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"2.17","version":"103'801","reported_seq":1131,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:29.899746+0000","last_change":"2023-10-30T13:23:18.989515+0000","last_active":"2023-10-30T14:26:29.899746+0000","last_peered":"2023-10-30T14:26:29.899746+0000","last_clean":"2023-10-30T14:26:29.899746+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T14:26:29.899746+0000","last_undegraded":"2023-10-30T14:26:29.899746+0000","last_fullsized":"2023-10-30T14:26:29.899746+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":801,"log_dups_size":0,"ondisk_log_size":801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:35.133762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943130,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":319,"num_read_kb":417394,"num_write":916,"num_write_kb":71613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"103'2444","reported_seq":5110,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.602179+0000","last_change":"2023-10-30T14:26:28.496922+0000","last_active":"2023-10-30T14:26:30.602179+0000","last_peered":"2023-10-30T14:26:30.602179+0000","last_clean":"2023-10-30T14:26:30.602179+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T14:26:30.602179+0000","last_undegraded":"2023-10-30T14:26:30.602179+0000","last_fullsized":"2023-10-30T14:26:30.602179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2444,"log_dups_size":0,"ondisk_log_size":2444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:42.984741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.146963134,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":152,"num_read_kb":10055,"num_write":1351,"num_write_kb":13264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"2.16","version":"103'849","reported_seq":1150,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-30T14:26:28.349915+0000","last_change":"2023-10-30T13:23:18.991454+0000","last_active":"2023-10-30T14:26:28.349915+0000","last_peered":"2023-10-30T14:26:28.349915+0000","last_clean":"2023-10-30T14:26:28.349915+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T14:26:28.349915+0000","last_undegraded":"2023-10-30T14:26:28.349915+0000","last_fullsized":"2023-10-30T14:26:28.349915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:13.450999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":291,"num_read_kb":402257,"num_write":947,"num_write_kb":71494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"103'2079","reported_seq":4870,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.389096+0000","last_change":"2023-10-30T14:26:31.389134+0000","last_active":"2023-10-30T14:26:31.389096+0000","last_peered":"2023-10-30T14:26:31.389096+0000","last_clean":"2023-10-30T14:26:31.389096+0000","last_became_active":"2023-10-30T13:24:00.990019+0000","last_became_peered":"2023-10-30T13:24:00.990019+0000","last_unstale":"2023-10-30T14:26:31.389096+0000","last_undegraded":"2023-10-30T14:26:31.389096+0000","last_fullsized":"2023-10-30T14:26:31.389096+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2079,"log_dups_size":0,"ondisk_log_size":2079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:54:43.880762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.49683508799999998,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":132,"num_read_kb":30385,"num_write":1263,"num_write_kb":18224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"2.15","version":"105'994","reported_seq":1250,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-30T14:26:29.007706+0000","last_change":"2023-10-30T13:23:18.982337+0000","last_active":"2023-10-30T14:26:29.007706+0000","last_peered":"2023-10-30T14:26:29.007706+0000","last_clean":"2023-10-30T14:26:29.007706+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T14:26:29.007706+0000","last_undegraded":"2023-10-30T14:26:29.007706+0000","last_fullsized":"2023-10-30T14:26:29.007706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":994,"log_dups_size":0,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:09:39.996740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":208,"num_read_kb":224450,"num_write":1056,"num_write_kb":46971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"103'2138","reported_seq":4914,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.232612+0000","last_change":"2023-10-30T14:26:31.232678+0000","last_active":"2023-10-30T14:26:31.232612+0000","last_peered":"2023-10-30T14:26:31.232612+0000","last_clean":"2023-10-30T14:26:31.232612+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T14:26:31.232612+0000","last_undegraded":"2023-10-30T14:26:31.232612+0000","last_fullsized":"2023-10-30T14:26:31.232612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2138,"log_dups_size":0,"ondisk_log_size":2138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:05.791418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.32825118800000003,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":141,"num_read_kb":34327,"num_write":1294,"num_write_kb":27493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"2.14","version":"103'993","reported_seq":1304,"reported_epoch":107,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.900495+0000","last_change":"2023-10-30T13:23:18.969882+0000","last_active":"2023-10-30T14:26:30.900495+0000","last_peered":"2023-10-30T14:26:30.900495+0000","last_clean":"2023-10-30T14:26:30.900495+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T14:26:30.900495+0000","last_undegraded":"2023-10-30T14:26:30.900495+0000","last_fullsized":"2023-10-30T14:26:30.900495+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":993,"log_dups_size":0,"ondisk_log_size":993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:17.093448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":290,"num_read_kb":355466,"num_write":1082,"num_write_kb":67489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"103'2160","reported_seq":4926,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.290922+0000","last_change":"2023-10-30T14:26:31.290963+0000","last_active":"2023-10-30T14:26:31.290922+0000","last_peered":"2023-10-30T14:26:31.290922+0000","last_clean":"2023-10-30T14:26:31.290922+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T14:26:31.290922+0000","last_undegraded":"2023-10-30T14:26:31.290922+0000","last_fullsized":"2023-10-30T14:26:31.290922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2160,"log_dups_size":0,"ondisk_log_size":2160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:34.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31630479500000003,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":143,"num_read_kb":29369,"num_write":1266,"num_write_kb":21737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"2.13","version":"103'701","reported_seq":1067,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.855757+0000","last_change":"2023-10-30T13:23:18.971632+0000","last_active":"2023-10-30T14:26:30.855757+0000","last_peered":"2023-10-30T14:26:30.855757+0000","last_clean":"2023-10-30T14:26:30.855757+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T14:26:30.855757+0000","last_undegraded":"2023-10-30T14:26:30.855757+0000","last_fullsized":"2023-10-30T14:26:30.855757+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:54.428499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331682,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":392,"num_read_kb":480410,"num_write":807,"num_write_kb":87841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"103'2142","reported_seq":4947,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.189211+0000","last_change":"2023-10-30T14:26:31.189246+0000","last_active":"2023-10-30T14:26:31.189211+0000","last_peered":"2023-10-30T14:26:31.189211+0000","last_clean":"2023-10-30T14:26:31.189211+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T14:26:31.189211+0000","last_undegraded":"2023-10-30T14:26:31.189211+0000","last_fullsized":"2023-10-30T14:26:31.189211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:33:52.119936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.29757573399999998,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":151,"num_read_kb":31378,"num_write":1306,"num_write_kb":33173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"2.12","version":"103'851","reported_seq":1107,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-30T14:26:28.346054+0000","last_change":"2023-10-30T13:23:18.978420+0000","last_active":"2023-10-30T14:26:28.346054+0000","last_peered":"2023-10-30T14:26:28.346054+0000","last_clean":"2023-10-30T14:26:28.346054+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T14:26:28.346054+0000","last_undegraded":"2023-10-30T14:26:28.346054+0000","last_fullsized":"2023-10-30T14:26:28.346054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:10:44.511282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53927368,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":208,"num_read_kb":306859,"num_write":914,"num_write_kb":62457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"103'2282","reported_seq":5070,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.894056+0000","last_change":"2023-10-30T14:26:33.299265+0000","last_active":"2023-10-30T14:26:34.894056+0000","last_peered":"2023-10-30T14:26:34.894056+0000","last_clean":"2023-10-30T14:26:34.894056+0000","last_became_active":"2023-10-30T13:24:00.706894+0000","last_became_peered":"2023-10-30T13:24:00.706894+0000","last_unstale":"2023-10-30T14:26:34.894056+0000","last_undegraded":"2023-10-30T14:26:34.894056+0000","last_fullsized":"2023-10-30T14:26:34.894056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2282,"log_dups_size":0,"ondisk_log_size":2282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:58:25.208615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.3738494,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":149,"num_read_kb":1539,"num_write":1375,"num_write_kb":12931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.11","version":"103'727","reported_seq":999,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:29.898574+0000","last_change":"2023-10-30T13:23:18.972707+0000","last_active":"2023-10-30T14:26:29.898574+0000","last_peered":"2023-10-30T14:26:29.898574+0000","last_clean":"2023-10-30T14:26:29.898574+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T14:26:29.898574+0000","last_undegraded":"2023-10-30T14:26:29.898574+0000","last_fullsized":"2023-10-30T14:26:29.898574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":727,"log_dups_size":0,"ondisk_log_size":727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":234,"num_read_kb":298146,"num_write":829,"num_write_kb":54985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"103'2091","reported_seq":4920,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.438547+0000","last_change":"2023-10-30T14:26:31.439481+0000","last_active":"2023-10-30T14:26:31.438547+0000","last_peered":"2023-10-30T14:26:31.438547+0000","last_clean":"2023-10-30T14:26:31.438547+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T14:26:31.438547+0000","last_undegraded":"2023-10-30T14:26:31.438547+0000","last_fullsized":"2023-10-30T14:26:31.438547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2091,"log_dups_size":0,"ondisk_log_size":2091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:13.302003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.538756546,"stat_sum":{"num_bytes":0,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":148,"num_read_kb":43506,"num_write":1375,"num_write_kb":26417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2ae","length":"9"}]},{"pgid":"2.10","version":"103'825","reported_seq":1169,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.924671+0000","last_change":"2023-10-30T13:23:18.948296+0000","last_active":"2023-10-30T14:26:32.924671+0000","last_peered":"2023-10-30T14:26:32.924671+0000","last_clean":"2023-10-30T14:26:32.924671+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T14:26:32.924671+0000","last_undegraded":"2023-10-30T14:26:32.924671+0000","last_fullsized":"2023-10-30T14:26:32.924671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:55.308329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":481,"num_read_kb":229594,"num_write":1003,"num_write_kb":42855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"105'2329","reported_seq":5104,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.349211+0000","last_change":"2023-10-30T14:26:31.350510+0000","last_active":"2023-10-30T14:26:31.349211+0000","last_peered":"2023-10-30T14:26:31.349211+0000","last_clean":"2023-10-30T14:26:31.349211+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T14:26:31.349211+0000","last_undegraded":"2023-10-30T14:26:31.349211+0000","last_fullsized":"2023-10-30T14:26:31.349211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2329,"log_dups_size":0,"ondisk_log_size":2329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:52:53.481213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.37434098500000001,"stat_sum":{"num_bytes":10,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":166,"num_read_kb":25446,"num_write":1447,"num_write_kb":32435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"2.f","version":"103'995","reported_seq":1275,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:29.898179+0000","last_change":"2023-10-30T13:23:18.962343+0000","last_active":"2023-10-30T14:26:29.898179+0000","last_peered":"2023-10-30T14:26:29.898179+0000","last_clean":"2023-10-30T14:26:29.898179+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T14:26:29.898179+0000","last_undegraded":"2023-10-30T14:26:29.898179+0000","last_fullsized":"2023-10-30T14:26:29.898179+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":995,"log_dups_size":0,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:36.082164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":214,"num_read_kb":373366,"num_write":1150,"num_write_kb":71086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"103'2232","reported_seq":4968,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.405792+0000","last_change":"2023-10-30T14:26:31.405833+0000","last_active":"2023-10-30T14:26:31.405792+0000","last_peered":"2023-10-30T14:26:31.405792+0000","last_clean":"2023-10-30T14:26:31.405792+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T14:26:31.405792+0000","last_undegraded":"2023-10-30T14:26:31.405792+0000","last_fullsized":"2023-10-30T14:26:31.405792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2232,"log_dups_size":0,"ondisk_log_size":2232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:20.681496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.43066016899999998,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":142,"num_read_kb":18968,"num_write":1278,"num_write_kb":24747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"2.e","version":"105'626","reported_seq":936,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.925265+0000","last_change":"2023-10-30T13:23:18.949081+0000","last_active":"2023-10-30T14:26:32.925265+0000","last_peered":"2023-10-30T14:26:32.925265+0000","last_clean":"2023-10-30T14:26:32.925265+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T14:26:32.925265+0000","last_undegraded":"2023-10-30T14:26:32.925265+0000","last_fullsized":"2023-10-30T14:26:32.925265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:00.356885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":280,"num_read_kb":492842,"num_write":679,"num_write_kb":83547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"103'2085","reported_seq":4901,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.269360+0000","last_change":"2023-10-30T14:26:31.269503+0000","last_active":"2023-10-30T14:26:31.269360+0000","last_peered":"2023-10-30T14:26:31.269360+0000","last_clean":"2023-10-30T14:26:31.269360+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T14:26:31.269360+0000","last_undegraded":"2023-10-30T14:26:31.269360+0000","last_fullsized":"2023-10-30T14:26:31.269360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2085,"log_dups_size":0,"ondisk_log_size":2085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:48:07.930338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.28710528200000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":152,"num_read_kb":61046,"num_write":1310,"num_write_kb":36653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"2.d","version":"103'731","reported_seq":1021,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-30T14:26:28.649843+0000","last_change":"2023-10-30T13:23:18.970780+0000","last_active":"2023-10-30T14:26:28.649843+0000","last_peered":"2023-10-30T14:26:28.649843+0000","last_clean":"2023-10-30T14:26:28.649843+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T14:26:28.649843+0000","last_undegraded":"2023-10-30T14:26:28.649843+0000","last_fullsized":"2023-10-30T14:26:28.649843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:38:51.113629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364699,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":233,"num_read_kb":370768,"num_write":815,"num_write_kb":71238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"103'2230","reported_seq":5027,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.602081+0000","last_change":"2023-10-30T14:26:28.494245+0000","last_active":"2023-10-30T14:26:30.602081+0000","last_peered":"2023-10-30T14:26:30.602081+0000","last_clean":"2023-10-30T14:26:30.602081+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T14:26:30.602081+0000","last_undegraded":"2023-10-30T14:26:30.602081+0000","last_fullsized":"2023-10-30T14:26:30.602081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2230,"log_dups_size":0,"ondisk_log_size":2230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:21:20.903439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.147841474,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":178,"num_read_kb":29706,"num_write":1375,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"2.c","version":"103'1078","reported_seq":1359,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.925585+0000","last_change":"2023-10-30T13:23:18.977920+0000","last_active":"2023-10-30T14:26:32.925585+0000","last_peered":"2023-10-30T14:26:32.925585+0000","last_clean":"2023-10-30T14:26:32.925585+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T14:26:32.925585+0000","last_undegraded":"2023-10-30T14:26:32.925585+0000","last_fullsized":"2023-10-30T14:26:32.925585+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:07:59.139948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50332230,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":210,"num_read_kb":406971,"num_write":1089,"num_write_kb":71553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"103'2029","reported_seq":4827,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.377911+0000","last_change":"2023-10-30T14:26:31.377951+0000","last_active":"2023-10-30T14:26:31.377911+0000","last_peered":"2023-10-30T14:26:31.377911+0000","last_clean":"2023-10-30T14:26:31.377911+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T14:26:31.377911+0000","last_undegraded":"2023-10-30T14:26:31.377911+0000","last_fullsized":"2023-10-30T14:26:31.377911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:42.017596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40298152700000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":122,"num_read_kb":1096,"num_write":1268,"num_write_kb":17447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"1.0","version":"16'117","reported_seq":283,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.859281+0000","last_change":"2023-10-30T13:23:08.208031+0000","last_active":"2023-10-30T14:26:30.859281+0000","last_peered":"2023-10-30T14:26:30.859281+0000","last_clean":"2023-10-30T14:26:30.859281+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T14:26:30.859281+0000","last_undegraded":"2023-10-30T14:26:30.859281+0000","last_fullsized":"2023-10-30T14:26:30.859281+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_clean_scrub_stamp":"2023-10-30T13:23:07.179866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:22.691367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"103'848","reported_seq":1182,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-30T14:26:28.345671+0000","last_change":"2023-10-30T13:23:18.979328+0000","last_active":"2023-10-30T14:26:28.345671+0000","last_peered":"2023-10-30T14:26:28.345671+0000","last_clean":"2023-10-30T14:26:28.345671+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T14:26:28.345671+0000","last_undegraded":"2023-10-30T14:26:28.345671+0000","last_fullsized":"2023-10-30T14:26:28.345671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":848,"log_dups_size":0,"ondisk_log_size":848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:04.942593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331714,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":319,"num_read_kb":439640,"num_write":936,"num_write_kb":75464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"103'2192","reported_seq":4922,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.429084+0000","last_change":"2023-10-30T14:26:31.429121+0000","last_active":"2023-10-30T14:26:31.429084+0000","last_peered":"2023-10-30T14:26:31.429084+0000","last_clean":"2023-10-30T14:26:31.429084+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T14:26:31.429084+0000","last_undegraded":"2023-10-30T14:26:31.429084+0000","last_fullsized":"2023-10-30T14:26:31.429084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2192,"log_dups_size":0,"ondisk_log_size":2192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:50:29.776722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.48831134300000001,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":127,"num_read_kb":26261,"num_write":1294,"num_write_kb":24882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"2.0","version":"103'1186","reported_seq":1496,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.859034+0000","last_change":"2023-10-30T13:23:18.985318+0000","last_active":"2023-10-30T14:26:30.859034+0000","last_peered":"2023-10-30T14:26:30.859034+0000","last_clean":"2023-10-30T14:26:30.859034+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T14:26:30.859034+0000","last_undegraded":"2023-10-30T14:26:30.859034+0000","last_fullsized":"2023-10-30T14:26:30.859034+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1186,"log_dups_size":0,"ondisk_log_size":1186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:16.754084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":256,"num_read_kb":488007,"num_write":1246,"num_write_kb":83351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"103'1973","reported_seq":4784,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.113701+0000","last_change":"2023-10-30T14:26:31.113739+0000","last_active":"2023-10-30T14:26:31.113701+0000","last_peered":"2023-10-30T14:26:31.113701+0000","last_clean":"2023-10-30T14:26:31.113701+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T14:26:31.113701+0000","last_undegraded":"2023-10-30T14:26:31.113701+0000","last_fullsized":"2023-10-30T14:26:31.113701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1973,"log_dups_size":0,"ondisk_log_size":1973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:00:22.152754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.22231293899999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":129,"num_read_kb":13045,"num_write":1222,"num_write_kb":11637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"2.1","version":"108'1950","reported_seq":3194,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.981824+0000","last_change":"2023-10-30T13:23:18.949565+0000","last_active":"2023-10-30T14:26:33.981824+0000","last_peered":"2023-10-30T14:26:33.981824+0000","last_clean":"2023-10-30T14:26:33.981824+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T14:26:33.981824+0000","last_undegraded":"2023-10-30T14:26:33.981824+0000","last_fullsized":"2023-10-30T14:26:33.981824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1950,"log_dups_size":0,"ondisk_log_size":1950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:55:46.029368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40060913,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":1167,"num_read_kb":359975,"num_write":2020,"num_write_kb":69890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"103'2255","reported_seq":5004,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.147114+0000","last_change":"2023-10-30T14:26:29.028605+0000","last_active":"2023-10-30T14:26:30.147114+0000","last_peered":"2023-10-30T14:26:30.147114+0000","last_clean":"2023-10-30T14:26:30.147114+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T14:26:30.147114+0000","last_undegraded":"2023-10-30T14:26:30.147114+0000","last_fullsized":"2023-10-30T14:26:30.147114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2255,"log_dups_size":0,"ondisk_log_size":2255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:34.388506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.381779967,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":145,"num_read_kb":55105,"num_write":1354,"num_write_kb":28005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"2.2","version":"103'1488","reported_seq":1790,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-30T14:26:28.346160+0000","last_change":"2023-10-30T13:23:18.984535+0000","last_active":"2023-10-30T14:26:28.346160+0000","last_peered":"2023-10-30T14:26:28.346160+0000","last_clean":"2023-10-30T14:26:28.346160+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T14:26:28.346160+0000","last_undegraded":"2023-10-30T14:26:28.346160+0000","last_fullsized":"2023-10-30T14:26:28.346160+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1488,"log_dups_size":0,"ondisk_log_size":1488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:28.405405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":235,"num_read_kb":509467,"num_write":1537,"num_write_kb":83943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"103'2200","reported_seq":4972,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.173309+0000","last_change":"2023-10-30T14:26:31.173350+0000","last_active":"2023-10-30T14:26:31.173309+0000","last_peered":"2023-10-30T14:26:31.173309+0000","last_clean":"2023-10-30T14:26:31.173309+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T14:26:31.173309+0000","last_undegraded":"2023-10-30T14:26:31.173309+0000","last_fullsized":"2023-10-30T14:26:31.173309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2200,"log_dups_size":0,"ondisk_log_size":2200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:01.000314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.26946742800000001,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":143,"num_read_kb":13239,"num_write":1306,"num_write_kb":17120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"2.4","version":"103'610","reported_seq":917,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-30T14:26:28.648620+0000","last_change":"2023-10-30T13:23:18.958210+0000","last_active":"2023-10-30T14:26:28.648620+0000","last_peered":"2023-10-30T14:26:28.648620+0000","last_clean":"2023-10-30T14:26:28.648620+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T14:26:28.648620+0000","last_undegraded":"2023-10-30T14:26:28.648620+0000","last_fullsized":"2023-10-30T14:26:28.648620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:34.653005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":230,"num_read_kb":520969,"num_write":671,"num_write_kb":95828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"103'2173","reported_seq":4928,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.602524+0000","last_change":"2023-10-30T14:26:28.500331+0000","last_active":"2023-10-30T14:26:30.602524+0000","last_peered":"2023-10-30T14:26:30.602524+0000","last_clean":"2023-10-30T14:26:30.602524+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T14:26:30.602524+0000","last_undegraded":"2023-10-30T14:26:30.602524+0000","last_fullsized":"2023-10-30T14:26:30.602524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:49.634910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.14913711299999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":60418,"num_write":1286,"num_write_kb":36950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"2.5","version":"103'1046","reported_seq":1328,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.860494+0000","last_change":"2023-10-30T13:23:18.985185+0000","last_active":"2023-10-30T14:26:30.860494+0000","last_peered":"2023-10-30T14:26:30.860494+0000","last_clean":"2023-10-30T14:26:30.860494+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T14:26:30.860494+0000","last_undegraded":"2023-10-30T14:26:30.860494+0000","last_fullsized":"2023-10-30T14:26:30.860494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:36.230072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44105804,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":214,"num_read_kb":377318,"num_write":1088,"num_write_kb":69045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"103'2202","reported_seq":4972,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.147058+0000","last_change":"2023-10-30T14:26:29.045893+0000","last_active":"2023-10-30T14:26:30.147058+0000","last_peered":"2023-10-30T14:26:30.147058+0000","last_clean":"2023-10-30T14:26:30.147058+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T14:26:30.147058+0000","last_undegraded":"2023-10-30T14:26:30.147058+0000","last_fullsized":"2023-10-30T14:26:30.147058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:12:50.574169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.39742323000000002,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":157,"num_read_kb":35362,"num_write":1332,"num_write_kb":31772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"2.6","version":"104'1125","reported_seq":1415,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-30T14:26:28.649135+0000","last_change":"2023-10-30T13:23:18.978085+0000","last_active":"2023-10-30T14:26:28.649135+0000","last_peered":"2023-10-30T14:26:28.649135+0000","last_clean":"2023-10-30T14:26:28.649135+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T14:26:28.649135+0000","last_undegraded":"2023-10-30T14:26:28.649135+0000","last_fullsized":"2023-10-30T14:26:28.649135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1125,"log_dups_size":0,"ondisk_log_size":1125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:47.588972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":247,"num_read_kb":365780,"num_write":1778,"num_write_kb":65023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"103'2077","reported_seq":4909,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.895121+0000","last_change":"2023-10-30T14:26:33.252060+0000","last_active":"2023-10-30T14:26:34.895121+0000","last_peered":"2023-10-30T14:26:34.895121+0000","last_clean":"2023-10-30T14:26:34.895121+0000","last_became_active":"2023-10-30T13:24:01.001412+0000","last_became_peered":"2023-10-30T13:24:01.001412+0000","last_unstale":"2023-10-30T14:26:34.895121+0000","last_undegraded":"2023-10-30T14:26:34.895121+0000","last_fullsized":"2023-10-30T14:26:34.895121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2077,"log_dups_size":0,"ondisk_log_size":2077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:37:55.891565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.32700652899999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":137,"num_read_kb":5811,"num_write":1214,"num_write_kb":16143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.7","version":"103'468","reported_seq":771,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:29.900569+0000","last_change":"2023-10-30T13:23:18.970566+0000","last_active":"2023-10-30T14:26:29.900569+0000","last_peered":"2023-10-30T14:26:29.900569+0000","last_clean":"2023-10-30T14:26:29.900569+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T14:26:29.900569+0000","last_undegraded":"2023-10-30T14:26:29.900569+0000","last_fullsized":"2023-10-30T14:26:29.900569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:03.851889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":281,"num_read_kb":442743,"num_write":555,"num_write_kb":70797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"103'2179","reported_seq":4959,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.286645+0000","last_change":"2023-10-30T14:26:31.286682+0000","last_active":"2023-10-30T14:26:31.286645+0000","last_peered":"2023-10-30T14:26:31.286645+0000","last_clean":"2023-10-30T14:26:31.286645+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T14:26:31.286645+0000","last_undegraded":"2023-10-30T14:26:31.286645+0000","last_fullsized":"2023-10-30T14:26:31.286645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:45.467912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.39475043599999998,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":144,"num_read_kb":25445,"num_write":1326,"num_write_kb":20705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"2.8","version":"103'544","reported_seq":807,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.858776+0000","last_change":"2023-10-30T13:23:18.986394+0000","last_active":"2023-10-30T14:26:30.858776+0000","last_peered":"2023-10-30T14:26:30.858776+0000","last_clean":"2023-10-30T14:26:30.858776+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T14:26:30.858776+0000","last_undegraded":"2023-10-30T14:26:30.858776+0000","last_fullsized":"2023-10-30T14:26:30.858776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:59.986981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":236,"num_read_kb":274544,"num_write":603,"num_write_kb":42177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"103'2113","reported_seq":4909,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.114115+0000","last_change":"2023-10-30T14:26:31.114154+0000","last_active":"2023-10-30T14:26:31.114115+0000","last_peered":"2023-10-30T14:26:31.114115+0000","last_clean":"2023-10-30T14:26:31.114115+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T14:26:31.114115+0000","last_undegraded":"2023-10-30T14:26:31.114115+0000","last_fullsized":"2023-10-30T14:26:31.114115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:33:55.510897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.211043338,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":144,"num_read_kb":34655,"num_write":1307,"num_write_kb":20329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"2.9","version":"103'1061","reported_seq":1328,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-30T14:26:28.649571+0000","last_change":"2023-10-30T13:23:18.963905+0000","last_active":"2023-10-30T14:26:28.649571+0000","last_peered":"2023-10-30T14:26:28.649571+0000","last_clean":"2023-10-30T14:26:28.649571+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T14:26:28.649571+0000","last_undegraded":"2023-10-30T14:26:28.649571+0000","last_fullsized":"2023-10-30T14:26:28.649571+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1061,"log_dups_size":0,"ondisk_log_size":1061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":237,"num_read_kb":249991,"num_write":1213,"num_write_kb":43538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"103'2263","reported_seq":4993,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.200814+0000","last_change":"2023-10-30T14:26:31.200850+0000","last_active":"2023-10-30T14:26:31.200814+0000","last_peered":"2023-10-30T14:26:31.200814+0000","last_clean":"2023-10-30T14:26:31.200814+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T14:26:31.200814+0000","last_undegraded":"2023-10-30T14:26:31.200814+0000","last_fullsized":"2023-10-30T14:26:31.200814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2263,"log_dups_size":0,"ondisk_log_size":2263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:23.681284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.26051332799999999,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":129,"num_read_kb":1227,"num_write":1352,"num_write_kb":12536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"2.a","version":"103'1473","reported_seq":1805,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-30T14:26:28.648313+0000","last_change":"2023-10-30T13:23:18.952302+0000","last_active":"2023-10-30T14:26:28.648313+0000","last_peered":"2023-10-30T14:26:28.648313+0000","last_clean":"2023-10-30T14:26:28.648313+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T14:26:28.648313+0000","last_undegraded":"2023-10-30T14:26:28.648313+0000","last_fullsized":"2023-10-30T14:26:28.648313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:12.827417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":321,"num_read_kb":438308,"num_write":2001,"num_write_kb":98279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"103'2090","reported_seq":4873,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.306316+0000","last_change":"2023-10-30T14:26:31.306337+0000","last_active":"2023-10-30T14:26:31.306316+0000","last_peered":"2023-10-30T14:26:31.306316+0000","last_clean":"2023-10-30T14:26:31.306316+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T14:26:31.306316+0000","last_undegraded":"2023-10-30T14:26:31.306316+0000","last_fullsized":"2023-10-30T14:26:31.306316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:48:21.894364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.36574155699999999,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":141,"num_read_kb":11852,"num_write":1237,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2ae","length":"5"}]},{"pgid":"2.b","version":"103'1064","reported_seq":1378,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.859947+0000","last_change":"2023-10-30T13:23:18.984912+0000","last_active":"2023-10-30T14:26:30.859947+0000","last_peered":"2023-10-30T14:26:30.859947+0000","last_clean":"2023-10-30T14:26:30.859947+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T14:26:30.859947+0000","last_undegraded":"2023-10-30T14:26:30.859947+0000","last_fullsized":"2023-10-30T14:26:30.859947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:27:16.205042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":296,"num_read_kb":429426,"num_write":1159,"num_write_kb":75926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"103'2153","reported_seq":4944,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.373423+0000","last_change":"2023-10-30T14:26:31.373458+0000","last_active":"2023-10-30T14:26:31.373423+0000","last_peered":"2023-10-30T14:26:31.373423+0000","last_clean":"2023-10-30T14:26:31.373423+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T14:26:31.373423+0000","last_undegraded":"2023-10-30T14:26:31.373423+0000","last_fullsized":"2023-10-30T14:26:31.373423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2153,"log_dups_size":0,"ondisk_log_size":2153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:13:23.098088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.47353502400000003,"stat_sum":{"num_bytes":5,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":137,"num_read_kb":40109,"num_write":1287,"num_write_kb":25283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2ae","length":"8"}]},{"pgid":"3.1d","version":"103'2173","reported_seq":4917,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.602108+0000","last_change":"2023-10-30T14:26:28.500321+0000","last_active":"2023-10-30T14:26:30.602108+0000","last_peered":"2023-10-30T14:26:30.602108+0000","last_clean":"2023-10-30T14:26:30.602108+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T14:26:30.602108+0000","last_undegraded":"2023-10-30T14:26:30.602108+0000","last_fullsized":"2023-10-30T14:26:30.602108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:36:48.108413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.14834739899999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":129,"num_read_kb":12975,"num_write":1318,"num_write_kb":20116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"2.1c","version":"103'734","reported_seq":1020,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.859690+0000","last_change":"2023-10-30T13:23:18.993372+0000","last_active":"2023-10-30T14:26:30.859690+0000","last_peered":"2023-10-30T14:26:30.859690+0000","last_clean":"2023-10-30T14:26:30.859690+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T14:26:30.859690+0000","last_undegraded":"2023-10-30T14:26:30.859690+0000","last_fullsized":"2023-10-30T14:26:30.859690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":734,"log_dups_size":0,"ondisk_log_size":734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:41.370678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":304,"num_read_kb":264369,"num_write":877,"num_write_kb":48007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"103'2635","reported_seq":5256,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.602822+0000","last_change":"2023-10-30T14:26:28.492092+0000","last_active":"2023-10-30T14:26:30.602822+0000","last_peered":"2023-10-30T14:26:30.602822+0000","last_clean":"2023-10-30T14:26:30.602822+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T14:26:30.602822+0000","last_undegraded":"2023-10-30T14:26:30.602822+0000","last_fullsized":"2023-10-30T14:26:30.602822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2635,"log_dups_size":0,"ondisk_log_size":2635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:29:50.938506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.145909233,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":171,"num_read_kb":40337,"num_write":1456,"num_write_kb":34441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"2.1d","version":"103'1179","reported_seq":1477,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.860310+0000","last_change":"2023-10-30T13:23:18.987633+0000","last_active":"2023-10-30T14:26:30.860310+0000","last_peered":"2023-10-30T14:26:30.860310+0000","last_clean":"2023-10-30T14:26:30.860310+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T14:26:30.860310+0000","last_undegraded":"2023-10-30T14:26:30.860310+0000","last_fullsized":"2023-10-30T14:26:30.860310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1179,"log_dups_size":0,"ondisk_log_size":1179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:09:33.324898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":255,"num_read_kb":502436,"num_write":1236,"num_write_kb":99955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"103'2021","reported_seq":4836,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:31.331003+0000","last_change":"2023-10-30T14:26:31.331040+0000","last_active":"2023-10-30T14:26:31.331003+0000","last_peered":"2023-10-30T14:26:31.331003+0000","last_clean":"2023-10-30T14:26:31.331003+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T14:26:31.331003+0000","last_undegraded":"2023-10-30T14:26:31.331003+0000","last_fullsized":"2023-10-30T14:26:31.331003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2021,"log_dups_size":0,"ondisk_log_size":2021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:08:17.137347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.43894956899999998,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":13189,"num_write":1249,"num_write_kb":15084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2ae","length":"6"}]},{"pgid":"2.1e","version":"103'779","reported_seq":1030,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:29.955372+0000","last_change":"2023-10-30T13:23:18.958470+0000","last_active":"2023-10-30T14:26:29.955372+0000","last_peered":"2023-10-30T14:26:29.955372+0000","last_clean":"2023-10-30T14:26:29.955372+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T14:26:29.955372+0000","last_undegraded":"2023-10-30T14:26:29.955372+0000","last_fullsized":"2023-10-30T14:26:29.955372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":779,"log_dups_size":0,"ondisk_log_size":779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:21.507744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":159,"num_read_kb":369044,"num_write":787,"num_write_kb":79251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"103'2191","reported_seq":5079,"reported_epoch":107,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-30T14:26:30.941207+0000","last_change":"2023-10-30T14:26:30.941191+0000","last_active":"2023-10-30T14:26:30.941207+0000","last_peered":"2023-10-30T14:26:30.941207+0000","last_clean":"2023-10-30T14:26:30.941207+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T14:26:30.941207+0000","last_undegraded":"2023-10-30T14:26:30.941207+0000","last_fullsized":"2023-10-30T14:26:30.941207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2191,"log_dups_size":0,"ondisk_log_size":2191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:21:12.818309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.64452194299999999,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":262,"num_read_kb":42307,"num_write":1336,"num_write_kb":30630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2ae","length":"4"}]},{"pgid":"2.1f","version":"103'1010","reported_seq":1326,"reported_epoch":106,"state":"active+clean","last_fresh":"2023-10-30T14:26:30.855988+0000","last_change":"2023-10-30T13:23:18.980952+0000","last_active":"2023-10-30T14:26:30.855988+0000","last_peered":"2023-10-30T14:26:30.855988+0000","last_clean":"2023-10-30T14:26:30.855988+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T14:26:30.855988+0000","last_undegraded":"2023-10-30T14:26:30.855988+0000","last_fullsized":"2023-10-30T14:26:30.855988+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1010,"log_dups_size":0,"ondisk_log_size":1010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:44.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777798,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":313,"num_read_kb":340671,"num_write":1085,"num_write_kb":63851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":35,"num_objects":2377,"num_object_clones":0,"num_object_copies":4754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":9387,"num_read_kb":1724407,"num_write":83581,"num_write_kb":1459747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":85,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138018,"ondisk_log_size":138018,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1283710936,"num_objects":1569,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1569,"num_whiteouts":0,"num_read":10652,"num_read_kb":12762809,"num_write":35606,"num_write_kb":2342480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2567577600,"data_stored":2567421165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31952,"ondisk_log_size":31952,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575613,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590928,"kb_used_data":389060,"kb_used_omap":1,"kb_used_meta":201854,"kb_avail":93780912,"statfs":{"total":96636764160,"available":96031653888,"internally_reserved":0,"allocated":398397440,"data_stored":397766040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":206698955},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.746,"15min":0.80500000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":1.6339999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.82999999999999996,"15min":0.91400000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.286,"5min":1.3160000000000001,"15min":1.448},"last":1.7250000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.85999999999999999,"15min":0.88300000000000001},"min":{"1min":0.46300000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0840000000000001,"5min":2.452,"15min":2.452},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.78600000000000003,"15min":0.84199999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.47399999999999998},"max":{"1min":1.028,"5min":1.198,"15min":1.3520000000000001},"last":1.1619999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.78600000000000003,"15min":0.82799999999999996},"min":{"1min":0.48999999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.169,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.88,"15min":0.95099999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.024,"5min":1.3009999999999999,"15min":1.5569999999999999},"last":0.94299999999999995}]},{"osd":3,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.294,"5min":1.405,"15min":1.405},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.028,"15min":1.004},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.21,"5min":1.4610000000000001,"15min":1.5269999999999999},"last":0.747}]},{"osd":4,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.378,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.2010000000000001,"5min":1.4350000000000001,"15min":1.732},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0309999999999999,"15min":1.03},"min":{"1min":0.495,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.266,"5min":1.4890000000000001,"15min":1.673},"last":0.97799999999999998}]},{"osd":5,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.1439999999999999,"15min":1.0800000000000001},"min":{"1min":0.42999999999999999,"5min":0.375,"15min":0.375},"max":{"1min":1.3720000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1120000000000001,"15min":1.1619999999999999},"min":{"1min":0.57099999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7809999999999999},"last":0.90200000000000002}]},{"osd":6,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.095,"15min":1.1060000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.456,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.151,"15min":1.1930000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3779999999999999,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.65700000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575616,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501196,"kb_used_data":339644,"kb_used_omap":1,"kb_used_meta":161534,"kb_avail":93870644,"statfs":{"total":96636764160,"available":96123539456,"internally_reserved":0,"allocated":347795456,"data_stored":347161219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":165411271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.75700000000000001,"15min":0.80100000000000005},"min":{"1min":0.51100000000000001,"5min":0.46700000000000003,"15min":0.375},"max":{"1min":0.89600000000000002,"5min":1.306,"15min":1.363},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.88900000000000001,"15min":0.95299999999999996},"min":{"1min":0.53500000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3979999999999999,"5min":1.4099999999999999,"15min":1.4370000000000001},"last":0.746}]},{"osd":1,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.72299999999999998,"15min":0.77400000000000002},"min":{"1min":0.54100000000000004,"5min":0.46999999999999997,"15min":0.46800000000000003},"max":{"1min":0.875,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.47399999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.127,"5min":1.706,"15min":1.706},"last":0.48399999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.81299999999999994,"15min":0.82999999999999996},"min":{"1min":0.57699999999999996,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.1679999999999999,"5min":1.29,"15min":1.571},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.872,"5min":0.96899999999999997,"15min":0.93700000000000006},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.286,"5min":5.4480000000000004,"15min":5.4480000000000004},"last":0.64800000000000002}]},{"osd":3,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0229999999999999,"15min":1.0329999999999999},"min":{"1min":0.68000000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.103,"15min":1.1160000000000001},"min":{"1min":0.68899999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.23,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.89100000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.90300000000000002,"15min":0.90700000000000003},"min":{"1min":0.70799999999999996,"5min":0.434,"15min":0.434},"max":{"1min":1.276,"5min":1.3640000000000001,"15min":1.4339999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.002,"15min":1.0309999999999999},"min":{"1min":0.71999999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.153,"5min":1.595,"15min":1.595},"last":1.0860000000000001}]},{"osd":5,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.159,"15min":1.03},"min":{"1min":0.77400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.2889999999999999,"5min":3.0219999999999998,"15min":3.0219999999999998},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.0089999999999999,"15min":1.0509999999999999},"min":{"1min":0.75600000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.135,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":0.85499999999999998}]},{"osd":7,"last update":"Mon Oct 30 14:26:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.204,"15min":1.196},"min":{"1min":1.048,"5min":0.58699999999999997,"15min":0.52600000000000002},"max":{"1min":1.3740000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1599999999999999,"15min":1.1359999999999999},"min":{"1min":0.83399999999999996,"5min":0.752,"15min":0.63700000000000001},"max":{"1min":1.3440000000000001,"5min":2.806,"15min":2.806},"last":1.0029999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575613,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565188,"kb_used_data":367544,"kb_used_omap":1,"kb_used_meta":197630,"kb_avail":93806652,"statfs":{"total":96636764160,"available":96058011648,"internally_reserved":0,"allocated":376365056,"data_stored":375730744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202373574},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72399999999999998,"15min":0.86499999999999999},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.40999999999999998},"max":{"1min":1.0860000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.95099999999999996,"15min":0.95799999999999996},"min":{"1min":0.60499999999999998,"5min":0.58699999999999997,"15min":0.57199999999999995},"max":{"1min":1.2410000000000001,"5min":1.617,"15min":1.9219999999999999},"last":0.84399999999999997}]},{"osd":1,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.81999999999999995,"15min":0.84099999999999997},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.77000000000000002,"15min":0.83799999999999997},"min":{"1min":0.52800000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.97099999999999997,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.88100000000000001,"15min":0.88500000000000001},"min":{"1min":0.59699999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.2589999999999999,"5min":1.331,"15min":1.3999999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81299999999999994,"15min":0.84799999999999998},"min":{"1min":0.59999999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.331,"5min":1.331,"15min":1.796},"last":0.90300000000000002}]},{"osd":3,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0049999999999999,"15min":1.006},"min":{"1min":0.63700000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":1.093},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92000000000000004,"15min":0.999},"min":{"1min":0.71899999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.1930000000000001,"5min":1.3460000000000001,"15min":2.0030000000000001},"last":1.1439999999999999}]},{"osd":4,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.88400000000000001,"15min":0.879},"min":{"1min":0.56899999999999995,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.2729999999999999,"5min":2.8170000000000002,"15min":2.8170000000000002},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.159,"5min":1.024,"15min":1.03},"min":{"1min":0.86799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.6299999999999999,"5min":2.2400000000000002,"15min":2.262},"last":1.2509999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.0880000000000001,"15min":1.1299999999999999},"min":{"1min":0.80900000000000005,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.1499999999999999,"5min":2.3090000000000002,"15min":2.7949999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.0229999999999999,"15min":1.056},"min":{"1min":0.71899999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.49,"5min":2.161,"15min":2.1629999999999998},"last":1.3009999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:26:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0209999999999999,"15min":1.091},"min":{"1min":0.78100000000000003,"5min":0.317,"15min":0.317},"max":{"1min":1.3440000000000001,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.381},{"interface":"front","average":{"1min":1.149,"5min":1.1299999999999999,"15min":1.153},"min":{"1min":0.95099999999999996,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.4019999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.1970000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575613,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463760,"kb_used_data":266052,"kb_used_omap":1,"kb_used_meta":197694,"kb_avail":93908080,"statfs":{"total":96636764160,"available":96161873920,"internally_reserved":0,"allocated":272437248,"data_stored":271810678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202439110},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"min":{"1min":0.48799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1259999999999999,"5min":1.3500000000000001,"15min":1.446},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.71099999999999997,"15min":0.82299999999999995},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.875,"5min":1.659,"15min":1.659},"last":0.71199999999999997}]},{"osd":1,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.56799999999999995,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.141,"5min":1.589,"15min":1.589},"last":2.2000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.84399999999999997,"15min":0.84699999999999998},"min":{"1min":0.58999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.075,"5min":1.532,"15min":1.532},"last":1.831}]},{"osd":2,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.76100000000000001,"15min":0.85999999999999999},"min":{"1min":0.55800000000000005,"5min":0.5,"15min":0.5},"max":{"1min":0.94899999999999995,"5min":1.607,"15min":1.607},"last":1.296},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84699999999999998,"15min":0.92200000000000004},"min":{"1min":0.57199999999999995,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.218,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.196}]},{"osd":3,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.99099999999999999,"15min":0.97499999999999998},"min":{"1min":0.66300000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.242,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":2.008},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.099,"15min":1.0780000000000001},"min":{"1min":0.879,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3480000000000001,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.9279999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.98899999999999999,"15min":1.042},"min":{"1min":0.77000000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.3009999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.391},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.65400000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.1279999999999999,"5min":1.3220000000000001,"15min":1.3280000000000001},"last":1.456}]},{"osd":6,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.97899999999999998,"15min":0.96099999999999997},"min":{"1min":0.78800000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.3100000000000001,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.042,"15min":1.0700000000000001},"min":{"1min":0.78300000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.238,"5min":1.677,"15min":1.728},"last":1.534}]},{"osd":7,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.137,"15min":1.1040000000000001},"min":{"1min":1.0820000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.417,"5min":1.4750000000000001,"15min":1.518},"last":2.1190000000000002},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.204,"15min":1.1799999999999999},"min":{"1min":1.056,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4670000000000001,"5min":1.587,"15min":1.587},"last":2.2810000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575611,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589328,"kb_used_data":381952,"kb_used_omap":1,"kb_used_meta":207358,"kb_avail":93782512,"statfs":{"total":96636764160,"available":96033292288,"internally_reserved":0,"allocated":391118848,"data_stored":390482370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":212335048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38100000000000001,"15min":0.40600000000000003},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71799999999999997,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.308},"max":{"1min":0.747,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.432}]},{"osd":1,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":0.76500000000000001,"5min":1.403,"15min":1.403},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.51700000000000002,"15min":0.54800000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65800000000000003,"5min":1.752,"15min":1.752},"last":0.60799999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56399999999999995,"15min":0.59999999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69499999999999995,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.41699999999999998,"5min":0.29999999999999999,"15min":0.29299999999999998},"max":{"1min":0.78000000000000003,"5min":1.73,"15min":1.73},"last":0.69699999999999995}]},{"osd":4,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.80300000000000005},"min":{"1min":0.53600000000000003,"5min":0.379,"15min":0.379},"max":{"1min":1.6519999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.80400000000000005,"15min":0.82399999999999995},"min":{"1min":0.55800000000000005,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.679,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.89200000000000002}]},{"osd":5,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77400000000000002,"15min":0.83299999999999996},"min":{"1min":0.61499999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98899999999999999,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.81000000000000005,"15min":0.85699999999999998},"min":{"1min":0.58399999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.071,"5min":1.486,"15min":1.486},"last":0.63500000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.85199999999999998,"15min":0.88400000000000001},"min":{"1min":0.55600000000000005,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.048,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.878,"5min":0.91500000000000004,"15min":0.93999999999999995},"min":{"1min":0.71999999999999997,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0409999999999999,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.224}]},{"osd":7,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83199999999999996,"15min":0.86199999999999999},"min":{"1min":0.64200000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.1100000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.748},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79100000000000004,"15min":0.82599999999999996},"min":{"1min":0.56299999999999994,"5min":0.501,"15min":0.501},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.93400000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299400,"kb_used_data":200252,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94072440,"statfs":{"total":96636764160,"available":96330178560,"internally_reserved":0,"allocated":205058048,"data_stored":204414271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101513670},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.433,"15min":0.45800000000000002},"min":{"1min":0.254,"5min":0.246,"15min":0.23899999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.89300000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":0.94599999999999995,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.42199999999999999,"15min":0.44700000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72999999999999998,"5min":1.956,"15min":1.956},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.36199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.153}]},{"osd":3,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81999999999999995,"5min":1.988,"15min":1.988},"last":1.2010000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49099999999999999,"15min":0.51800000000000002},"min":{"1min":0.32300000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.90100000000000002,"5min":2.335,"15min":2.335},"last":0.67900000000000005}]},{"osd":4,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.60899999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5409999999999999,"5min":2.302,"15min":2.302},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84299999999999997,"15min":0.84099999999999997},"min":{"1min":0.625,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6879999999999999,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.91200000000000003}]},{"osd":5,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.77100000000000002,"15min":0.80600000000000005},"min":{"1min":0.64500000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82699999999999996,"15min":0.84499999999999997},"min":{"1min":0.73099999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.587,"5min":1.875,"15min":1.875},"last":0.85299999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.81599999999999995,"15min":0.82799999999999996},"min":{"1min":0.61299999999999999,"5min":0.502,"15min":0.48099999999999998},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84499999999999997,"15min":0.86799999999999999},"min":{"1min":0.60999999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.429,"5min":1.921,"15min":1.921},"last":1.2130000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.86099999999999999,"15min":0.86399999999999999},"min":{"1min":0.66100000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.131,"5min":4.5739999999999998,"15min":4.5739999999999998},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.88900000000000001,"15min":0.91700000000000004},"min":{"1min":0.64300000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6080000000000001,"5min":2.375,"15min":2.375},"last":1.1659999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575614,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490624,"kb_used_data":329016,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":93881216,"statfs":{"total":96636764160,"available":96134365184,"internally_reserved":0,"allocated":336912384,"data_stored":336288270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165476806},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.438,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.504,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.51600000000000001},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.47799999999999998,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.443}]},{"osd":2,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.46999999999999997,"15min":0.48399999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54800000000000004,"5min":1.954,"15min":1.954},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.45800000000000002,"15min":0.47799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63200000000000001,"5min":2.274,"15min":2.274},"last":0.497}]},{"osd":3,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53600000000000003,"15min":0.54900000000000004},"min":{"1min":0.35599999999999998,"5min":0.35399999999999998,"15min":0.35099999999999998},"max":{"1min":0.60099999999999998,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.436,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.70199999999999996,"5min":2.625,"15min":2.625},"last":0.55100000000000005}]},{"osd":4,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.80200000000000005,"15min":0.83999999999999997},"min":{"1min":0.56499999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93300000000000005,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.82299999999999995,"15min":0.88},"min":{"1min":0.629,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.91200000000000003,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.77500000000000002}]},{"osd":5,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82399999999999995,"15min":0.878},"min":{"1min":0.58099999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94799999999999995,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.85099999999999998,"15min":0.90100000000000002},"min":{"1min":0.60399999999999998,"5min":0.499,"15min":0.499},"max":{"1min":0.879,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.81100000000000005}]},{"osd":6,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.91000000000000003,"15min":0.84199999999999997},"min":{"1min":0.73799999999999999,"5min":0.624,"15min":0.56499999999999995},"max":{"1min":1.0289999999999999,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.81299999999999994,"15min":0.89700000000000002},"min":{"1min":0.56200000000000006,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.95799999999999996,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.874}]},{"osd":7,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.80600000000000005,"15min":0.82999999999999996},"min":{"1min":0.59599999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.95999999999999996,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.82799999999999996,"15min":0.85199999999999998},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93799999999999994,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.72199999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575611,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416972,"kb_used_data":245696,"kb_used_omap":1,"kb_used_meta":171262,"kb_avail":93954868,"statfs":{"total":96636764160,"available":96209784832,"internally_reserved":0,"allocated":251592704,"data_stored":250964313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175372742},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.38100000000000001,"15min":0.40500000000000003},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54400000000000004,"5min":0.66100000000000003,"15min":0.94999999999999996},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.49299999999999999,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":1.0229999999999999},"last":0.52600000000000002}]},{"osd":2,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.45300000000000001,"15min":0.46700000000000003},"min":{"1min":0.307,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":1.05,"15min":1.05},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.47099999999999997,"15min":0.498},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.29099999999999998},"max":{"1min":0.59699999999999998,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52900000000000003,"15min":0.54900000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.34499999999999997},"max":{"1min":0.73299999999999998,"5min":1.236,"15min":1.236},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51600000000000001,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.74399999999999999,"5min":1.054,"15min":1.054},"last":0.56200000000000006}]},{"osd":4,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.78600000000000003,"15min":0.80300000000000005},"min":{"1min":0.65900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":0.88800000000000001,"5min":1.103,"15min":1.29},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76600000000000001,"15min":0.81000000000000005},"min":{"1min":0.59299999999999997,"5min":0.53800000000000003,"15min":0.48399999999999999},"max":{"1min":0.95999999999999996,"5min":1.012,"15min":1.2210000000000001},"last":0.63900000000000001}]},{"osd":5,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.78700000000000003,"15min":0.81200000000000006},"min":{"1min":0.64700000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.93899999999999995,"5min":1.0509999999999999,"15min":1.347},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.81599999999999995,"15min":0.85199999999999998},"min":{"1min":0.63,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.95899999999999996,"5min":1.2130000000000001,"15min":1.3939999999999999},"last":0.73099999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81499999999999995,"15min":0.84399999999999997},"min":{"1min":0.64500000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.95799999999999996,"5min":1.222,"15min":1.486},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79700000000000004,"15min":0.83399999999999996},"min":{"1min":0.64400000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0149999999999999,"5min":1.113,"15min":1.1619999999999999},"last":0.82299999999999995}]},{"osd":7,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.81699999999999995,"15min":0.85099999999999998},"min":{"1min":0.625,"5min":0.625,"15min":0.58599999999999997},"max":{"1min":0.97599999999999998,"5min":1.2509999999999999,"15min":1.29},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84199999999999997,"15min":0.872},"min":{"1min":0.70799999999999996,"5min":0.61799999999999999,"15min":0.57299999999999995},"max":{"1min":0.96099999999999997,"5min":1.173,"15min":1.4419999999999999},"last":0.76500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249647104,"data_stored":249627338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335572992,"data_stored":335549765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203661312,"data_stored":203639529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389746688,"data_stored":389725232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":271077376,"data_stored":271063047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":375005184,"data_stored":374984132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":346419200,"data_stored":346404076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":396447744,"data_stored":396428046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:26:37.789 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:26:37.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a74cc9640 1 Processor -- start 2023-10-30T14:26:37.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a74cc9640 1 -- start start 2023-10-30T14:26:37.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a74cc9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a701538a0 0x7f0a7015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a74cc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a700ca0e0 0x7f0a7015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a74cc9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a700cb800 con 0x7f0a700cf160 2023-10-30T14:26:37.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a74cc9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a7015dc30 con 0x7f0a701538a0 2023-10-30T14:26:37.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a74cc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a700cb330 con 0x7f0a700ca0e0 2023-10-30T14:26:37.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a6e59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a701538a0 0x7f0a7015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a6ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0a700cf160 0x7f0a700cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38010/0 (socket says 172.21.15.46:38010) 2023-10-30T14:26:37.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a6ed9d640 1 -- 172.21.15.46:0/2066017157 learned_addr learned my addr 172.21.15.46:0/2066017157 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:37.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a6e59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a701538a0 0x7f0a7015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43536/0 (socket says 172.21.15.46:43536) 2023-10-30T14:26:37.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a6dd9b640 1 --2- 172.21.15.46:0/2066017157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a700ca0e0 0x7f0a7015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a6e59c640 1 -- 172.21.15.46:0/2066017157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a700ca0e0 msgr2=0x7f0a7015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:37.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a6e59c640 1 --2- 172.21.15.46:0/2066017157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a700ca0e0 0x7f0a7015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a6e59c640 1 -- 172.21.15.46:0/2066017157 >> v1:172.21.15.46:6789/0 conn(0x7f0a700cf160 legacy=0x7f0a700cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a6e59c640 1 -- 172.21.15.46:0/2066017157 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a700cad70 con 0x7f0a701538a0 2023-10-30T14:26:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a6dd9b640 1 --2- 172.21.15.46:0/2066017157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a700ca0e0 0x7f0a7015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.966+0000 7f0a6d59a640 1 -- 172.21.15.46:0/2066017157 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968677745 0 0) 0x7f0a700cb800 con 0x7f0a700cf160 2023-10-30T14:26:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.970+0000 7f0a6e59c640 1 --2- 172.21.15.46:0/2066017157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a701538a0 0x7f0a7015d6f0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f0a600097c0 tx=0x7f0a6001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4246fad2fadfc493 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.970+0000 7f0a6d59a640 1 -- 172.21.15.46:0/2066017157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a600a8970 con 0x7f0a701538a0 2023-10-30T14:26:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.970+0000 7f0a6d59a640 1 -- 172.21.15.46:0/2066017157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a600a8ad0 con 0x7f0a701538a0 2023-10-30T14:26:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.970+0000 7f0a6d59a640 1 -- 172.21.15.46:0/2066017157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a600a8e60 con 0x7f0a701538a0 2023-10-30T14:26:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.970+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2066017157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a701538a0 msgr2=0x7f0a7015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.970+0000 7f0a74cc9640 1 --2- 172.21.15.46:0/2066017157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a701538a0 0x7f0a7015d6f0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f0a600097c0 tx=0x7f0a6001ec70 comp rx=0 tx=0).stop 2023-10-30T14:26:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.970+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2066017157 shutdown_connections 2023-10-30T14:26:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.970+0000 7f0a74cc9640 1 --2- 172.21.15.46:0/2066017157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a700ca0e0 0x7f0a7015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.970+0000 7f0a74cc9640 1 --2- 172.21.15.46:0/2066017157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a701538a0 0x7f0a7015d6f0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.970+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2066017157 >> 172.21.15.46:0/2066017157 conn(0x7f0a7006bfe0 msgr2=0x7f0a7006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.970+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2066017157 shutdown_connections 2023-10-30T14:26:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.970+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2066017157 wait complete. 2023-10-30T14:26:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a74cc9640 1 Processor -- start 2023-10-30T14:26:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a74cc9640 1 -- start start 2023-10-30T14:26:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a74cc9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a700ca0e0 0x7f0a701e4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a74cc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a700cf160 0x7f0a701e3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a74cc9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a701538a0 0x7f0a70157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a74cc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a701e4490 con 0x7f0a700cf160 2023-10-30T14:26:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a74cc9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a701e56c0 con 0x7f0a701538a0 2023-10-30T14:26:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a74cc9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a700b4c00 con 0x7f0a700ca0e0 2023-10-30T14:26:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a6dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a700cf160 0x7f0a701e3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a6dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a700cf160 0x7f0a701e3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43542/0 (socket says 172.21.15.46:43542) 2023-10-30T14:26:37.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a6dd9b640 1 -- 172.21.15.46:0/2713974824 learned_addr learned my addr 172.21.15.46:0/2713974824 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:37.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a6ed9d640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a701538a0 0x7f0a70157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a6e59c640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a700ca0e0 0x7f0a701e4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a6dd9b640 1 -- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a700ca0e0 msgr2=0x7f0a701e4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:37.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a6dd9b640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a700ca0e0 0x7f0a701e4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a6dd9b640 1 -- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a701538a0 msgr2=0x7f0a70157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:37.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a6dd9b640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a701538a0 0x7f0a70157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:37.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a6dd9b640 1 -- 172.21.15.46:0/2713974824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60009420 con 0x7f0a700cf160 2023-10-30T14:26:37.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a6dd9b640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a700cf160 0x7f0a701e3750 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400e810 tx=0x7f0a6400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:37.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a5effd640 1 -- 172.21.15.46:0/2713974824 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6400c870 con 0x7f0a700cf160 2023-10-30T14:26:37.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a5effd640 1 -- 172.21.15.46:0/2713974824 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6400c9d0 con 0x7f0a700cf160 2023-10-30T14:26:37.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2713974824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a701e2f80 con 0x7f0a700cf160 2023-10-30T14:26:37.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a5effd640 1 -- 172.21.15.46:0/2713974824 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6400cd60 con 0x7f0a700cf160 2023-10-30T14:26:37.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.974+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2713974824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a700b5130 con 0x7f0a700cf160 2023-10-30T14:26:37.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.978+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2713974824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a700b2a50 con 0x7f0a700cf160 2023-10-30T14:26:37.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.978+0000 7f0a5effd640 1 -- 172.21.15.46:0/2713974824 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0a64013410 con 0x7f0a700cf160 2023-10-30T14:26:37.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.978+0000 7f0a5effd640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a500664a0 0x7f0a50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.978+0000 7f0a5effd640 1 -- 172.21.15.46:0/2713974824 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 8748+0+0 (secure 0 0 0) 0x7f0a64084830 con 0x7f0a700cf160 2023-10-30T14:26:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.978+0000 7f0a6e59c640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a500664a0 0x7f0a50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.982+0000 7f0a6e59c640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a500664a0 0x7f0a50068960 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0a600b2000 tx=0x7f0a60008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:37.982+0000 7f0a5effd640 1 -- 172.21.15.46:0/2713974824 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0a640510b0 con 0x7f0a700cf160 2023-10-30T14:26:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.138+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2713974824 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0a701e4270 con 0x7f0a500664a0 2023-10-30T14:26:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.162+0000 7f0a5effd640 1 -- 172.21.15.46:0/2713974824 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294423 (secure 0 0 0) 0x7f0a600b0440 con 0x7f0a500664a0 2023-10-30T14:26:38.167 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:26:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T14:26:38.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.166+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a500664a0 msgr2=0x7f0a50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.166+0000 7f0a74cc9640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a500664a0 0x7f0a50068960 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0a600b2000 tx=0x7f0a60008040 comp rx=0 tx=0).stop 2023-10-30T14:26:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.166+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2713974824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a700cf160 msgr2=0x7f0a701e3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.166+0000 7f0a74cc9640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a700cf160 0x7f0a701e3750 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400e810 tx=0x7f0a6400ecd0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.166+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2713974824 shutdown_connections 2023-10-30T14:26:38.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.166+0000 7f0a74cc9640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f0a500664a0 0x7f0a50068960 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.166+0000 7f0a74cc9640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a701538a0 0x7f0a70157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.166+0000 7f0a74cc9640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a700cf160 0x7f0a701e3750 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.166+0000 7f0a74cc9640 1 --2- 172.21.15.46:0/2713974824 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a700ca0e0 0x7f0a701e4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.166+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2713974824 >> 172.21.15.46:0/2713974824 conn(0x7f0a7006bfe0 msgr2=0x7f0a70151d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:38.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.166+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2713974824 shutdown_connections 2023-10-30T14:26:38.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.166+0000 7f0a74cc9640 1 -- 172.21.15.46:0/2713974824 wait complete. 2023-10-30T14:26:38.187 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":2008,"stamp":"2023-10-30T14:26:38.127326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1284924434,"num_objects":3948,"num_object_clones":2,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":20157,"num_read_kb":14487318,"num_write":119381,"num_write_kb":3805271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170124,"ondisk_log_size":170124,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3927216,"kb_used_data":2520724,"kb_used_omap":12,"kb_used_meta":1406387,"kb_avail":751047504,"statfs":{"total":773094113280,"available":769072644096,"internally_reserved":0,"allocated":2581221376,"data_stored":2576042778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1440140856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2285484,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":93,"num_write_kb":2287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.234424"},"pg_stats":[{"pgid":"3.3f","version":"103'2152","reported_seq":4963,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.727766+0000","last_change":"2023-10-30T14:26:34.688745+0000","last_active":"2023-10-30T14:26:35.727766+0000","last_peered":"2023-10-30T14:26:35.727766+0000","last_clean":"2023-10-30T14:26:35.727766+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T14:26:35.727766+0000","last_undegraded":"2023-10-30T14:26:35.727766+0000","last_fullsized":"2023-10-30T14:26:35.727766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2152,"log_dups_size":0,"ondisk_log_size":2152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:50.061774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.84725695499999998,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":124,"num_read_kb":10662,"num_write":1276,"num_write_kb":15944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.3e","version":"103'1964","reported_seq":4841,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.720358+0000","last_change":"2023-10-30T14:26:33.665197+0000","last_active":"2023-10-30T14:26:35.720358+0000","last_peered":"2023-10-30T14:26:35.720358+0000","last_clean":"2023-10-30T14:26:35.720358+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T14:26:35.720358+0000","last_undegraded":"2023-10-30T14:26:35.720358+0000","last_fullsized":"2023-10-30T14:26:35.720358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1964,"log_dups_size":0,"ondisk_log_size":1964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:04:48.747855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.406256491,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":133,"num_read_kb":21158,"num_write":1230,"num_write_kb":16620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.3d","version":"103'2090","reported_seq":4921,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.721385+0000","last_change":"2023-10-30T14:26:34.547863+0000","last_active":"2023-10-30T14:26:35.721385+0000","last_peered":"2023-10-30T14:26:35.721385+0000","last_clean":"2023-10-30T14:26:35.721385+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T14:26:35.721385+0000","last_undegraded":"2023-10-30T14:26:35.721385+0000","last_fullsized":"2023-10-30T14:26:35.721385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:55:03.386638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.030559456,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":129,"num_read_kb":29825,"num_write":1270,"num_write_kb":28412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.3c","version":"103'2021","reported_seq":4859,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.894425+0000","last_change":"2023-10-30T14:26:33.286784+0000","last_active":"2023-10-30T14:26:34.894425+0000","last_peered":"2023-10-30T14:26:34.894425+0000","last_clean":"2023-10-30T14:26:34.894425+0000","last_became_active":"2023-10-30T13:24:00.704829+0000","last_became_peered":"2023-10-30T13:24:00.704829+0000","last_unstale":"2023-10-30T14:26:34.894425+0000","last_undegraded":"2023-10-30T14:26:34.894425+0000","last_fullsized":"2023-10-30T14:26:34.894425+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2021,"log_dups_size":0,"ondisk_log_size":2021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:28.307918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.361435862,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":124,"num_read_kb":7939,"num_write":1184,"num_write_kb":24239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.3b","version":"103'2001","reported_seq":4857,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723057+0000","last_change":"2023-10-30T14:26:34.648843+0000","last_active":"2023-10-30T14:26:35.723057+0000","last_peered":"2023-10-30T14:26:35.723057+0000","last_clean":"2023-10-30T14:26:35.723057+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T14:26:35.723057+0000","last_undegraded":"2023-10-30T14:26:35.723057+0000","last_fullsized":"2023-10-30T14:26:35.723057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2001,"log_dups_size":0,"ondisk_log_size":2001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:36.619771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.80853723300000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":41266,"num_write":1244,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.3a","version":"103'2205","reported_seq":5002,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724501+0000","last_change":"2023-10-30T14:26:33.897829+0000","last_active":"2023-10-30T14:26:35.724501+0000","last_peered":"2023-10-30T14:26:35.724501+0000","last_clean":"2023-10-30T14:26:35.724501+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T14:26:35.724501+0000","last_undegraded":"2023-10-30T14:26:35.724501+0000","last_fullsized":"2023-10-30T14:26:35.724501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2205,"log_dups_size":0,"ondisk_log_size":2205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:03.177100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.94867293500000005,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":141,"num_read_kb":27429,"num_write":1290,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.39","version":"103'2138","reported_seq":4985,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.722859+0000","last_change":"2023-10-30T14:26:34.555126+0000","last_active":"2023-10-30T14:26:35.722859+0000","last_peered":"2023-10-30T14:26:35.722859+0000","last_clean":"2023-10-30T14:26:35.722859+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T14:26:35.722859+0000","last_undegraded":"2023-10-30T14:26:35.722859+0000","last_fullsized":"2023-10-30T14:26:35.722859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2138,"log_dups_size":0,"ondisk_log_size":2138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:25.464527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.71511574600000005,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":146,"num_read_kb":25110,"num_write":1316,"num_write_kb":20043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.38","version":"103'2099","reported_seq":4909,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.297117+0000","last_change":"2023-10-30T14:26:36.369916+0000","last_active":"2023-10-30T14:26:36.297117+0000","last_peered":"2023-10-30T14:26:36.297117+0000","last_clean":"2023-10-30T14:26:36.297117+0000","last_became_active":"2023-10-30T13:24:00.989938+0000","last_became_peered":"2023-10-30T13:24:00.989938+0000","last_unstale":"2023-10-30T14:26:36.297117+0000","last_undegraded":"2023-10-30T14:26:36.297117+0000","last_fullsized":"2023-10-30T14:26:36.297117+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:03.578741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.90238370000000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":124,"num_read_kb":25628,"num_write":1232,"num_write_kb":20540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.37","version":"103'2126","reported_seq":4956,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.727278+0000","last_change":"2023-10-30T14:26:34.654560+0000","last_active":"2023-10-30T14:26:35.727278+0000","last_peered":"2023-10-30T14:26:35.727278+0000","last_clean":"2023-10-30T14:26:35.727278+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T14:26:35.727278+0000","last_undegraded":"2023-10-30T14:26:35.727278+0000","last_fullsized":"2023-10-30T14:26:35.727278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2126,"log_dups_size":0,"ondisk_log_size":2126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:56.758220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81280227199999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":24548,"num_write":1310,"num_write_kb":19821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.36","version":"103'2060","reported_seq":4921,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.899341+0000","last_change":"2023-10-30T14:26:33.465676+0000","last_active":"2023-10-30T14:26:34.899341+0000","last_peered":"2023-10-30T14:26:34.899341+0000","last_clean":"2023-10-30T14:26:34.899341+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T14:26:34.899341+0000","last_undegraded":"2023-10-30T14:26:34.899341+0000","last_fullsized":"2023-10-30T14:26:34.899341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2060,"log_dups_size":0,"ondisk_log_size":2060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:02.527162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.53668512800000001,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":148,"num_read_kb":64678,"num_write":1273,"num_write_kb":32495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.35","version":"105'1999","reported_seq":4872,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.899012+0000","last_change":"2023-10-30T14:26:33.399684+0000","last_active":"2023-10-30T14:26:34.899012+0000","last_peered":"2023-10-30T14:26:34.899012+0000","last_clean":"2023-10-30T14:26:34.899012+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T14:26:34.899012+0000","last_undegraded":"2023-10-30T14:26:34.899012+0000","last_fullsized":"2023-10-30T14:26:34.899012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1999,"log_dups_size":0,"ondisk_log_size":1999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:10:40.091601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.471346551,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":139,"num_read_kb":25550,"num_write":1254,"num_write_kb":16915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.34","version":"103'2073","reported_seq":4916,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.725794+0000","last_change":"2023-10-30T14:26:33.495643+0000","last_active":"2023-10-30T14:26:35.725794+0000","last_peered":"2023-10-30T14:26:35.725794+0000","last_clean":"2023-10-30T14:26:35.725794+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T14:26:35.725794+0000","last_undegraded":"2023-10-30T14:26:35.725794+0000","last_fullsized":"2023-10-30T14:26:35.725794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:07:16.016021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.54183594099999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":142,"num_read_kb":52725,"num_write":1240,"num_write_kb":42463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.33","version":"103'2219","reported_seq":5026,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.452367+0000","last_change":"2023-10-30T14:26:36.506702+0000","last_active":"2023-10-30T14:26:36.452367+0000","last_peered":"2023-10-30T14:26:36.452367+0000","last_clean":"2023-10-30T14:26:36.452367+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T14:26:36.452367+0000","last_undegraded":"2023-10-30T14:26:36.452367+0000","last_fullsized":"2023-10-30T14:26:36.452367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2219,"log_dups_size":0,"ondisk_log_size":2219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:07.521185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.038919041,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":46739,"num_write":1326,"num_write_kb":33562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.32","version":"103'2145","reported_seq":4993,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.721177+0000","last_change":"2023-10-30T14:26:34.440977+0000","last_active":"2023-10-30T14:26:35.721177+0000","last_peered":"2023-10-30T14:26:35.721177+0000","last_clean":"2023-10-30T14:26:35.721177+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T14:26:35.721177+0000","last_undegraded":"2023-10-30T14:26:35.721177+0000","last_fullsized":"2023-10-30T14:26:35.721177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2145,"log_dups_size":0,"ondisk_log_size":2145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:09:13.743893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.97549857799999995,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":149,"num_read_kb":13007,"num_write":1312,"num_write_kb":16233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.31","version":"103'2258","reported_seq":5081,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.718848+0000","last_change":"2023-10-30T14:26:33.691289+0000","last_active":"2023-10-30T14:26:35.718848+0000","last_peered":"2023-10-30T14:26:35.718848+0000","last_clean":"2023-10-30T14:26:35.718848+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T14:26:35.718848+0000","last_undegraded":"2023-10-30T14:26:35.718848+0000","last_fullsized":"2023-10-30T14:26:35.718848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2258,"log_dups_size":0,"ondisk_log_size":2258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:20:29.170495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.432871758,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":161,"num_read_kb":14756,"num_write":1377,"num_write_kb":22909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.30","version":"103'2085","reported_seq":4943,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723221+0000","last_change":"2023-10-30T14:26:33.787493+0000","last_active":"2023-10-30T14:26:35.723221+0000","last_peered":"2023-10-30T14:26:35.723221+0000","last_clean":"2023-10-30T14:26:35.723221+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T14:26:35.723221+0000","last_undegraded":"2023-10-30T14:26:35.723221+0000","last_fullsized":"2023-10-30T14:26:35.723221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2085,"log_dups_size":0,"ondisk_log_size":2085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:57.806855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.83845289999999995,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":149,"num_read_kb":40211,"num_write":1268,"num_write_kb":23485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2f","version":"104'2338","reported_seq":5151,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724465+0000","last_change":"2023-10-30T14:26:34.060688+0000","last_active":"2023-10-30T14:26:35.724465+0000","last_peered":"2023-10-30T14:26:35.724465+0000","last_clean":"2023-10-30T14:26:35.724465+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T14:26:35.724465+0000","last_undegraded":"2023-10-30T14:26:35.724465+0000","last_fullsized":"2023-10-30T14:26:35.724465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2338,"log_dups_size":0,"ondisk_log_size":2338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:47:21.386319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.109948846,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":170,"num_read_kb":39932,"num_write":1446,"num_write_kb":30153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2e","version":"103'1987","reported_seq":4842,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.895081+0000","last_change":"2023-10-30T14:26:33.267575+0000","last_active":"2023-10-30T14:26:34.895081+0000","last_peered":"2023-10-30T14:26:34.895081+0000","last_clean":"2023-10-30T14:26:34.895081+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T14:26:34.895081+0000","last_undegraded":"2023-10-30T14:26:34.895081+0000","last_fullsized":"2023-10-30T14:26:34.895081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1987,"log_dups_size":0,"ondisk_log_size":1987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:03.867577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.34246037800000001,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":137,"num_read_kb":13706,"num_write":1196,"num_write_kb":12790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2d","version":"103'2087","reported_seq":4919,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.725816+0000","last_change":"2023-10-30T14:26:34.604839+0000","last_active":"2023-10-30T14:26:35.725816+0000","last_peered":"2023-10-30T14:26:35.725816+0000","last_clean":"2023-10-30T14:26:35.725816+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T14:26:35.725816+0000","last_undegraded":"2023-10-30T14:26:35.725816+0000","last_fullsized":"2023-10-30T14:26:35.725816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2087,"log_dups_size":0,"ondisk_log_size":2087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:35:33.855326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.76369308499999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":125,"num_read_kb":30178,"num_write":1282,"num_write_kb":24413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2c","version":"103'2210","reported_seq":5039,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.899583+0000","last_change":"2023-10-30T14:26:33.395186+0000","last_active":"2023-10-30T14:26:34.899583+0000","last_peered":"2023-10-30T14:26:34.899583+0000","last_clean":"2023-10-30T14:26:34.899583+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T14:26:34.899583+0000","last_undegraded":"2023-10-30T14:26:34.899583+0000","last_fullsized":"2023-10-30T14:26:34.899583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:31.161754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.46668427699999998,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":148,"num_read_kb":30111,"num_write":1332,"num_write_kb":26597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2b","version":"103'2218","reported_seq":5006,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.722043+0000","last_change":"2023-10-30T14:26:34.474353+0000","last_active":"2023-10-30T14:26:35.722043+0000","last_peered":"2023-10-30T14:26:35.722043+0000","last_clean":"2023-10-30T14:26:35.722043+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T14:26:35.722043+0000","last_undegraded":"2023-10-30T14:26:35.722043+0000","last_fullsized":"2023-10-30T14:26:35.722043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2218,"log_dups_size":0,"ondisk_log_size":2218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:01.333450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.008362585,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":127,"num_read_kb":10449,"num_write":1298,"num_write_kb":11653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2a","version":"103'2269","reported_seq":5069,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724749+0000","last_change":"2023-10-30T14:26:33.893164+0000","last_active":"2023-10-30T14:26:35.724749+0000","last_peered":"2023-10-30T14:26:35.724749+0000","last_clean":"2023-10-30T14:26:35.724749+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T14:26:35.724749+0000","last_undegraded":"2023-10-30T14:26:35.724749+0000","last_fullsized":"2023-10-30T14:26:35.724749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2269,"log_dups_size":0,"ondisk_log_size":2269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:42.538423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.94381626399999996,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":144,"num_read_kb":30506,"num_write":1376,"num_write_kb":26822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.29","version":"108'2111","reported_seq":5073,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.405991+0000","last_change":"2023-10-30T14:26:36.452306+0000","last_active":"2023-10-30T14:26:36.405991+0000","last_peered":"2023-10-30T14:26:36.405991+0000","last_clean":"2023-10-30T14:26:36.405991+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T14:26:36.405991+0000","last_undegraded":"2023-10-30T14:26:36.405991+0000","last_fullsized":"2023-10-30T14:26:36.405991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2111,"log_dups_size":0,"ondisk_log_size":2111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:47.201749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98440143999999996,"stat_sum":{"num_bytes":6,"num_objects":38,"num_object_clones":2,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":257,"num_read_kb":42146,"num_write":1297,"num_write_kb":25266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.28","version":"103'2167","reported_seq":5028,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724912+0000","last_change":"2023-10-30T14:26:33.846175+0000","last_active":"2023-10-30T14:26:35.724912+0000","last_peered":"2023-10-30T14:26:35.724912+0000","last_clean":"2023-10-30T14:26:35.724912+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T14:26:35.724912+0000","last_undegraded":"2023-10-30T14:26:35.724912+0000","last_fullsized":"2023-10-30T14:26:35.724912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2167,"log_dups_size":0,"ondisk_log_size":2167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:34:46.488680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89723890399999995,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":164,"num_read_kb":14461,"num_write":1342,"num_write_kb":12663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.27","version":"103'2051","reported_seq":4890,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.472159+0000","last_change":"2023-10-30T14:26:36.518474+0000","last_active":"2023-10-30T14:26:36.472159+0000","last_peered":"2023-10-30T14:26:36.472159+0000","last_clean":"2023-10-30T14:26:36.472159+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T14:26:36.472159+0000","last_undegraded":"2023-10-30T14:26:36.472159+0000","last_fullsized":"2023-10-30T14:26:36.472159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2051,"log_dups_size":0,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:14.252516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0495642949999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":133,"num_read_kb":25337,"num_write":1220,"num_write_kb":15637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.26","version":"103'2135","reported_seq":5011,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.899304+0000","last_change":"2023-10-30T14:26:33.433647+0000","last_active":"2023-10-30T14:26:34.899304+0000","last_peered":"2023-10-30T14:26:34.899304+0000","last_clean":"2023-10-30T14:26:34.899304+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T14:26:34.899304+0000","last_undegraded":"2023-10-30T14:26:34.899304+0000","last_fullsized":"2023-10-30T14:26:34.899304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:13:23.542810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.504458776,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":160,"num_read_kb":14034,"num_write":1340,"num_write_kb":12359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.25","version":"103'2083","reported_seq":4931,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.719163+0000","last_change":"2023-10-30T14:26:33.601252+0000","last_active":"2023-10-30T14:26:35.719163+0000","last_peered":"2023-10-30T14:26:35.719163+0000","last_clean":"2023-10-30T14:26:35.719163+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T14:26:35.719163+0000","last_undegraded":"2023-10-30T14:26:35.719163+0000","last_fullsized":"2023-10-30T14:26:35.719163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:23:40.438800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.34262146900000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":128,"num_read_kb":22518,"num_write":1277,"num_write_kb":32107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.24","version":"103'2025","reported_seq":4853,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.506833+0000","last_change":"2023-10-30T14:26:36.555781+0000","last_active":"2023-10-30T14:26:36.506833+0000","last_peered":"2023-10-30T14:26:36.506833+0000","last_clean":"2023-10-30T14:26:36.506833+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T14:26:36.506833+0000","last_undegraded":"2023-10-30T14:26:36.506833+0000","last_fullsized":"2023-10-30T14:26:36.506833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2025,"log_dups_size":0,"ondisk_log_size":2025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:34.247640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.068284585,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":129,"num_read_kb":17081,"num_write":1202,"num_write_kb":15468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.23","version":"103'2131","reported_seq":4952,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.722225+0000","last_change":"2023-10-30T14:26:34.496718+0000","last_active":"2023-10-30T14:26:35.722225+0000","last_peered":"2023-10-30T14:26:35.722225+0000","last_clean":"2023-10-30T14:26:35.722225+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T14:26:35.722225+0000","last_undegraded":"2023-10-30T14:26:35.722225+0000","last_fullsized":"2023-10-30T14:26:35.722225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2131,"log_dups_size":0,"ondisk_log_size":2131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:53.519748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.030536272,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":125,"num_read_kb":13655,"num_write":1323,"num_write_kb":17357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.22","version":"103'2073","reported_seq":4941,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.727070+0000","last_change":"2023-10-30T14:26:33.934508+0000","last_active":"2023-10-30T14:26:35.727070+0000","last_peered":"2023-10-30T14:26:35.727070+0000","last_clean":"2023-10-30T14:26:35.727070+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T14:26:35.727070+0000","last_undegraded":"2023-10-30T14:26:35.727070+0000","last_fullsized":"2023-10-30T14:26:35.727070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:25:07.586673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98488935700000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":140,"num_read_kb":17880,"num_write":1311,"num_write_kb":13902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.21","version":"108'2369","reported_seq":5183,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.894065+0000","last_change":"2023-10-30T14:26:33.231592+0000","last_active":"2023-10-30T14:26:34.894065+0000","last_peered":"2023-10-30T14:26:34.894065+0000","last_clean":"2023-10-30T14:26:34.894065+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T14:26:34.894065+0000","last_undegraded":"2023-10-30T14:26:34.894065+0000","last_fullsized":"2023-10-30T14:26:34.894065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2369,"log_dups_size":0,"ondisk_log_size":2369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:00:56.118846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30670995400000001,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":165,"num_read_kb":40193,"num_write":1443,"num_write_kb":35290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.20","version":"108'2297","reported_seq":5117,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.898291+0000","last_change":"2023-10-30T14:26:33.357836+0000","last_active":"2023-10-30T14:26:34.898291+0000","last_peered":"2023-10-30T14:26:34.898291+0000","last_clean":"2023-10-30T14:26:34.898291+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T14:26:34.898291+0000","last_undegraded":"2023-10-30T14:26:34.898291+0000","last_fullsized":"2023-10-30T14:26:34.898291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2297,"log_dups_size":0,"ondisk_log_size":2297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:53:07.770089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42993166100000002,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":168,"num_read_kb":24860,"num_write":1383,"num_write_kb":20782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.1a","version":"103'2103","reported_seq":4944,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724893+0000","last_change":"2023-10-30T14:26:33.546412+0000","last_active":"2023-10-30T14:26:35.724893+0000","last_peered":"2023-10-30T14:26:35.724893+0000","last_clean":"2023-10-30T14:26:35.724893+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T14:26:35.724893+0000","last_undegraded":"2023-10-30T14:26:35.724893+0000","last_fullsized":"2023-10-30T14:26:35.724893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:27.773478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.59246732999999996,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":146,"num_read_kb":30802,"num_write":1268,"num_write_kb":21419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1b","version":"103'1028","reported_seq":1334,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.328825+0000","last_change":"2023-10-30T13:23:18.963933+0000","last_active":"2023-10-30T14:26:33.328825+0000","last_peered":"2023-10-30T14:26:33.328825+0000","last_clean":"2023-10-30T14:26:33.328825+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T14:26:33.328825+0000","last_undegraded":"2023-10-30T14:26:33.328825+0000","last_fullsized":"2023-10-30T14:26:33.328825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1028,"log_dups_size":0,"ondisk_log_size":1028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:48:58.196567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497506,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":222,"num_read_kb":501605,"num_write":1094,"num_write_kb":95785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"103'2148","reported_seq":4953,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.722408+0000","last_change":"2023-10-30T14:26:34.609800+0000","last_active":"2023-10-30T14:26:35.722408+0000","last_peered":"2023-10-30T14:26:35.722408+0000","last_clean":"2023-10-30T14:26:35.722408+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T14:26:35.722408+0000","last_undegraded":"2023-10-30T14:26:35.722408+0000","last_fullsized":"2023-10-30T14:26:35.722408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:24:55.845740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.1434485830000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":17524,"num_write":1294,"num_write_kb":15816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1a","version":"108'1693","reported_seq":2126,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.988999+0000","last_change":"2023-10-30T13:23:18.976615+0000","last_active":"2023-10-30T14:26:33.988999+0000","last_peered":"2023-10-30T14:26:33.988999+0000","last_clean":"2023-10-30T14:26:33.988999+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T14:26:33.988999+0000","last_undegraded":"2023-10-30T14:26:33.988999+0000","last_fullsized":"2023-10-30T14:26:33.988999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1693,"log_dups_size":0,"ondisk_log_size":1693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:59.048396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29694842,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":392,"num_read_kb":410068,"num_write":1792,"num_write_kb":89526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"104'1557","reported_seq":2910,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.394516+0000","last_change":"2023-10-30T13:23:18.977357+0000","last_active":"2023-10-30T14:26:33.394516+0000","last_peered":"2023-10-30T14:26:33.394516+0000","last_clean":"2023-10-30T14:26:33.394516+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T14:26:33.394516+0000","last_undegraded":"2023-10-30T14:26:33.394516+0000","last_fullsized":"2023-10-30T14:26:33.394516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1557,"log_dups_size":0,"ondisk_log_size":1557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:53.502681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60855928,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":1296,"num_read_kb":556671,"num_write":1596,"num_write_kb":94675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"103'2314","reported_seq":5149,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.308697+0000","last_change":"2023-10-30T14:26:36.357910+0000","last_active":"2023-10-30T14:26:36.308697+0000","last_peered":"2023-10-30T14:26:36.308697+0000","last_clean":"2023-10-30T14:26:36.308697+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T14:26:36.308697+0000","last_undegraded":"2023-10-30T14:26:36.308697+0000","last_fullsized":"2023-10-30T14:26:36.308697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2314,"log_dups_size":0,"ondisk_log_size":2314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:17.422938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89032229399999996,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":161,"num_read_kb":17698,"num_write":1449,"num_write_kb":26122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.18","version":"108'961","reported_seq":1307,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.770599+0000","last_change":"2023-10-30T13:23:18.989299+0000","last_active":"2023-10-30T14:26:33.770599+0000","last_peered":"2023-10-30T14:26:33.770599+0000","last_clean":"2023-10-30T14:26:33.770599+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T14:26:33.770599+0000","last_undegraded":"2023-10-30T14:26:33.770599+0000","last_fullsized":"2023-10-30T14:26:33.770599+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:44.510184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53243017,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":312,"num_read_kb":563254,"num_write":1067,"num_write_kb":95660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"108'2143","reported_seq":4999,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.718793+0000","last_change":"2023-10-30T14:26:33.629921+0000","last_active":"2023-10-30T14:26:35.718793+0000","last_peered":"2023-10-30T14:26:35.718793+0000","last_clean":"2023-10-30T14:26:35.718793+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T14:26:35.718793+0000","last_undegraded":"2023-10-30T14:26:35.718793+0000","last_fullsized":"2023-10-30T14:26:35.718793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2143,"log_dups_size":0,"ondisk_log_size":2143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:40:42.318115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.36920391600000002,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":152,"num_read_kb":51949,"num_write":1321,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.17","version":"103'801","reported_seq":1133,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.928642+0000","last_change":"2023-10-30T13:23:18.989515+0000","last_active":"2023-10-30T14:26:32.928642+0000","last_peered":"2023-10-30T14:26:32.928642+0000","last_clean":"2023-10-30T14:26:32.928642+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T14:26:32.928642+0000","last_undegraded":"2023-10-30T14:26:32.928642+0000","last_fullsized":"2023-10-30T14:26:32.928642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":801,"log_dups_size":0,"ondisk_log_size":801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:35.133762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943130,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":319,"num_read_kb":417394,"num_write":916,"num_write_kb":71613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"103'2444","reported_seq":5172,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724775+0000","last_change":"2023-10-30T14:26:33.980471+0000","last_active":"2023-10-30T14:26:35.724775+0000","last_peered":"2023-10-30T14:26:35.724775+0000","last_clean":"2023-10-30T14:26:35.724775+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T14:26:35.724775+0000","last_undegraded":"2023-10-30T14:26:35.724775+0000","last_fullsized":"2023-10-30T14:26:35.724775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2444,"log_dups_size":0,"ondisk_log_size":2444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:40.292098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0302270280000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":152,"num_read_kb":10055,"num_write":1351,"num_write_kb":13264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.16","version":"103'849","reported_seq":1153,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.950393+0000","last_change":"2023-10-30T13:23:18.991454+0000","last_active":"2023-10-30T14:26:32.950393+0000","last_peered":"2023-10-30T14:26:32.950393+0000","last_clean":"2023-10-30T14:26:32.950393+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T14:26:32.950393+0000","last_undegraded":"2023-10-30T14:26:32.950393+0000","last_fullsized":"2023-10-30T14:26:32.950393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:13.450999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":291,"num_read_kb":402257,"num_write":947,"num_write_kb":71494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"103'2079","reported_seq":4921,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.721729+0000","last_change":"2023-10-30T14:26:34.598085+0000","last_active":"2023-10-30T14:26:35.721729+0000","last_peered":"2023-10-30T14:26:35.721729+0000","last_clean":"2023-10-30T14:26:35.721729+0000","last_became_active":"2023-10-30T13:24:00.990019+0000","last_became_peered":"2023-10-30T13:24:00.990019+0000","last_unstale":"2023-10-30T14:26:35.721729+0000","last_undegraded":"2023-10-30T14:26:35.721729+0000","last_fullsized":"2023-10-30T14:26:35.721729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2079,"log_dups_size":0,"ondisk_log_size":2079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:06:56.464105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.1322467139999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":132,"num_read_kb":30385,"num_write":1263,"num_write_kb":18224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.15","version":"108'995","reported_seq":1254,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.022661+0000","last_change":"2023-10-30T13:23:18.982337+0000","last_active":"2023-10-30T14:26:34.022661+0000","last_peered":"2023-10-30T14:26:34.022661+0000","last_clean":"2023-10-30T14:26:34.022661+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T14:26:34.022661+0000","last_undegraded":"2023-10-30T14:26:34.022661+0000","last_fullsized":"2023-10-30T14:26:34.022661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":995,"log_dups_size":0,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:09:39.996740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":208,"num_read_kb":224450,"num_write":1057,"num_write_kb":46972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"103'2138","reported_seq":4966,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.725948+0000","last_change":"2023-10-30T14:26:33.534876+0000","last_active":"2023-10-30T14:26:35.725948+0000","last_peered":"2023-10-30T14:26:35.725948+0000","last_clean":"2023-10-30T14:26:35.725948+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T14:26:35.725948+0000","last_undegraded":"2023-10-30T14:26:35.725948+0000","last_fullsized":"2023-10-30T14:26:35.725948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2138,"log_dups_size":0,"ondisk_log_size":2138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:17.987617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.58089010799999996,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":141,"num_read_kb":34327,"num_write":1294,"num_write_kb":27493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.14","version":"107'994","reported_seq":1307,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.928561+0000","last_change":"2023-10-30T13:23:18.969882+0000","last_active":"2023-10-30T14:26:32.928561+0000","last_peered":"2023-10-30T14:26:32.928561+0000","last_clean":"2023-10-30T14:26:32.928561+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T14:26:32.928561+0000","last_undegraded":"2023-10-30T14:26:32.928561+0000","last_fullsized":"2023-10-30T14:26:32.928561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":994,"log_dups_size":0,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:17.093448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":290,"num_read_kb":355466,"num_write":1083,"num_write_kb":67490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"108'2162","reported_seq":4979,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723783+0000","last_change":"2023-10-30T14:26:34.585946+0000","last_active":"2023-10-30T14:26:35.723783+0000","last_peered":"2023-10-30T14:26:35.723783+0000","last_clean":"2023-10-30T14:26:35.723783+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T14:26:35.723783+0000","last_undegraded":"2023-10-30T14:26:35.723783+0000","last_fullsized":"2023-10-30T14:26:35.723783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2162,"log_dups_size":0,"ondisk_log_size":2162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:03:21.700327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.74553484199999998,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":143,"num_read_kb":29369,"num_write":1268,"num_write_kb":21738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.13","version":"103'701","reported_seq":1069,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.466470+0000","last_change":"2023-10-30T13:23:18.971632+0000","last_active":"2023-10-30T14:26:33.466470+0000","last_peered":"2023-10-30T14:26:33.466470+0000","last_clean":"2023-10-30T14:26:33.466470+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T14:26:33.466470+0000","last_undegraded":"2023-10-30T14:26:33.466470+0000","last_fullsized":"2023-10-30T14:26:33.466470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:54.428499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331682,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":392,"num_read_kb":480410,"num_write":807,"num_write_kb":87841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"103'2142","reported_seq":4998,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.721389+0000","last_change":"2023-10-30T14:26:34.422972+0000","last_active":"2023-10-30T14:26:35.721389+0000","last_peered":"2023-10-30T14:26:35.721389+0000","last_clean":"2023-10-30T14:26:35.721389+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T14:26:35.721389+0000","last_undegraded":"2023-10-30T14:26:35.721389+0000","last_fullsized":"2023-10-30T14:26:35.721389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:47:44.081448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.95729191499999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":151,"num_read_kb":31378,"num_write":1306,"num_write_kb":33173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.12","version":"103'851","reported_seq":1110,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.949624+0000","last_change":"2023-10-30T13:23:18.978420+0000","last_active":"2023-10-30T14:26:32.949624+0000","last_peered":"2023-10-30T14:26:32.949624+0000","last_clean":"2023-10-30T14:26:32.949624+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T14:26:32.949624+0000","last_undegraded":"2023-10-30T14:26:32.949624+0000","last_fullsized":"2023-10-30T14:26:32.949624+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:10:44.511282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53927368,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":208,"num_read_kb":306859,"num_write":914,"num_write_kb":62457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"103'2282","reported_seq":5070,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.894056+0000","last_change":"2023-10-30T14:26:33.299265+0000","last_active":"2023-10-30T14:26:34.894056+0000","last_peered":"2023-10-30T14:26:34.894056+0000","last_clean":"2023-10-30T14:26:34.894056+0000","last_became_active":"2023-10-30T13:24:00.706894+0000","last_became_peered":"2023-10-30T13:24:00.706894+0000","last_unstale":"2023-10-30T14:26:34.894056+0000","last_undegraded":"2023-10-30T14:26:34.894056+0000","last_fullsized":"2023-10-30T14:26:34.894056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2282,"log_dups_size":0,"ondisk_log_size":2282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:58:25.208615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.3738494,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":149,"num_read_kb":1539,"num_write":1375,"num_write_kb":12931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.11","version":"103'727","reported_seq":1001,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.927895+0000","last_change":"2023-10-30T13:23:18.972707+0000","last_active":"2023-10-30T14:26:32.927895+0000","last_peered":"2023-10-30T14:26:32.927895+0000","last_clean":"2023-10-30T14:26:32.927895+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T14:26:32.927895+0000","last_undegraded":"2023-10-30T14:26:32.927895+0000","last_fullsized":"2023-10-30T14:26:32.927895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":727,"log_dups_size":0,"ondisk_log_size":727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":234,"num_read_kb":298146,"num_write":829,"num_write_kb":54985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"103'2091","reported_seq":4959,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.899515+0000","last_change":"2023-10-30T14:26:33.432572+0000","last_active":"2023-10-30T14:26:34.899515+0000","last_peered":"2023-10-30T14:26:34.899515+0000","last_clean":"2023-10-30T14:26:34.899515+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T14:26:34.899515+0000","last_undegraded":"2023-10-30T14:26:34.899515+0000","last_fullsized":"2023-10-30T14:26:34.899515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2091,"log_dups_size":0,"ondisk_log_size":2091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:47:50.528827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50412874500000004,"stat_sum":{"num_bytes":0,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":148,"num_read_kb":43506,"num_write":1375,"num_write_kb":26417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.10","version":"103'825","reported_seq":1169,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.924671+0000","last_change":"2023-10-30T13:23:18.948296+0000","last_active":"2023-10-30T14:26:32.924671+0000","last_peered":"2023-10-30T14:26:32.924671+0000","last_clean":"2023-10-30T14:26:32.924671+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T14:26:32.924671+0000","last_undegraded":"2023-10-30T14:26:32.924671+0000","last_fullsized":"2023-10-30T14:26:32.924671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:55.308329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":481,"num_read_kb":229594,"num_write":1003,"num_write_kb":42855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"108'2330","reported_seq":5156,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723033+0000","last_change":"2023-10-30T14:26:34.549537+0000","last_active":"2023-10-30T14:26:35.723033+0000","last_peered":"2023-10-30T14:26:35.723033+0000","last_clean":"2023-10-30T14:26:35.723033+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T14:26:35.723033+0000","last_undegraded":"2023-10-30T14:26:35.723033+0000","last_fullsized":"2023-10-30T14:26:35.723033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2330,"log_dups_size":0,"ondisk_log_size":2330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:07.630163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70955538100000004,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":166,"num_read_kb":25446,"num_write":1448,"num_write_kb":32435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.f","version":"103'995","reported_seq":1278,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.725769+0000","last_change":"2023-10-30T13:23:18.962343+0000","last_active":"2023-10-30T14:26:35.725769+0000","last_peered":"2023-10-30T14:26:35.725769+0000","last_clean":"2023-10-30T14:26:35.725769+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T14:26:35.725769+0000","last_undegraded":"2023-10-30T14:26:35.725769+0000","last_fullsized":"2023-10-30T14:26:35.725769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":995,"log_dups_size":0,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:36.082164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":214,"num_read_kb":373366,"num_write":1150,"num_write_kb":71086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"103'2232","reported_seq":5019,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.726899+0000","last_change":"2023-10-30T14:26:34.736009+0000","last_active":"2023-10-30T14:26:35.726899+0000","last_peered":"2023-10-30T14:26:35.726899+0000","last_clean":"2023-10-30T14:26:35.726899+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T14:26:35.726899+0000","last_undegraded":"2023-10-30T14:26:35.726899+0000","last_fullsized":"2023-10-30T14:26:35.726899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2232,"log_dups_size":0,"ondisk_log_size":2232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:19.838741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89413299700000004,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":142,"num_read_kb":18968,"num_write":1278,"num_write_kb":24747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.e","version":"105'626","reported_seq":936,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.925265+0000","last_change":"2023-10-30T13:23:18.949081+0000","last_active":"2023-10-30T14:26:32.925265+0000","last_peered":"2023-10-30T14:26:32.925265+0000","last_clean":"2023-10-30T14:26:32.925265+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T14:26:32.925265+0000","last_undegraded":"2023-10-30T14:26:32.925265+0000","last_fullsized":"2023-10-30T14:26:32.925265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:00.356885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":280,"num_read_kb":492842,"num_write":679,"num_write_kb":83547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"103'2085","reported_seq":4957,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.727597+0000","last_change":"2023-10-30T14:26:34.726845+0000","last_active":"2023-10-30T14:26:35.727597+0000","last_peered":"2023-10-30T14:26:35.727597+0000","last_clean":"2023-10-30T14:26:35.727597+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T14:26:35.727597+0000","last_undegraded":"2023-10-30T14:26:35.727597+0000","last_fullsized":"2023-10-30T14:26:35.727597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2085,"log_dups_size":0,"ondisk_log_size":2085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:53.224813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.88483857600000004,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":152,"num_read_kb":61046,"num_write":1310,"num_write_kb":36653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.d","version":"103'731","reported_seq":1024,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.258765+0000","last_change":"2023-10-30T13:23:18.970780+0000","last_active":"2023-10-30T14:26:33.258765+0000","last_peered":"2023-10-30T14:26:33.258765+0000","last_clean":"2023-10-30T14:26:33.258765+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T14:26:33.258765+0000","last_undegraded":"2023-10-30T14:26:33.258765+0000","last_fullsized":"2023-10-30T14:26:33.258765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:38:51.113629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364699,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":233,"num_read_kb":370768,"num_write":815,"num_write_kb":71238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"103'2230","reported_seq":5089,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723892+0000","last_change":"2023-10-30T14:26:33.934502+0000","last_active":"2023-10-30T14:26:35.723892+0000","last_peered":"2023-10-30T14:26:35.723892+0000","last_clean":"2023-10-30T14:26:35.723892+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T14:26:35.723892+0000","last_undegraded":"2023-10-30T14:26:35.723892+0000","last_fullsized":"2023-10-30T14:26:35.723892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2230,"log_dups_size":0,"ondisk_log_size":2230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:46.397667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98441546899999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":178,"num_read_kb":29706,"num_write":1375,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.c","version":"103'1078","reported_seq":1359,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.925585+0000","last_change":"2023-10-30T13:23:18.977920+0000","last_active":"2023-10-30T14:26:32.925585+0000","last_peered":"2023-10-30T14:26:32.925585+0000","last_clean":"2023-10-30T14:26:32.925585+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T14:26:32.925585+0000","last_undegraded":"2023-10-30T14:26:32.925585+0000","last_fullsized":"2023-10-30T14:26:32.925585+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:07:59.139948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50332230,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":210,"num_read_kb":406971,"num_write":1089,"num_write_kb":71553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"103'2029","reported_seq":4878,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.725143+0000","last_change":"2023-10-30T14:26:34.688764+0000","last_active":"2023-10-30T14:26:35.725143+0000","last_peered":"2023-10-30T14:26:35.725143+0000","last_clean":"2023-10-30T14:26:35.725143+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T14:26:35.725143+0000","last_undegraded":"2023-10-30T14:26:35.725143+0000","last_fullsized":"2023-10-30T14:26:35.725143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:07.116174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.84782311300000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":122,"num_read_kb":1096,"num_write":1268,"num_write_kb":17447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"1.0","version":"16'117","reported_seq":285,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.839992+0000","last_change":"2023-10-30T13:23:08.208031+0000","last_active":"2023-10-30T14:26:33.839992+0000","last_peered":"2023-10-30T14:26:33.839992+0000","last_clean":"2023-10-30T14:26:33.839992+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T14:26:33.839992+0000","last_undegraded":"2023-10-30T14:26:33.839992+0000","last_fullsized":"2023-10-30T14:26:33.839992+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_clean_scrub_stamp":"2023-10-30T13:23:07.179866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:22.691367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"108'849","reported_seq":1186,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.991014+0000","last_change":"2023-10-30T13:23:18.979328+0000","last_active":"2023-10-30T14:26:33.991014+0000","last_peered":"2023-10-30T14:26:33.991014+0000","last_clean":"2023-10-30T14:26:33.991014+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T14:26:33.991014+0000","last_undegraded":"2023-10-30T14:26:33.991014+0000","last_fullsized":"2023-10-30T14:26:33.991014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:04.942593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331714,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":319,"num_read_kb":439640,"num_write":936,"num_write_kb":75464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"103'2192","reported_seq":4977,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.417782+0000","last_change":"2023-10-30T14:26:36.472084+0000","last_active":"2023-10-30T14:26:36.417782+0000","last_peered":"2023-10-30T14:26:36.417782+0000","last_clean":"2023-10-30T14:26:36.417782+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T14:26:36.417782+0000","last_undegraded":"2023-10-30T14:26:36.417782+0000","last_fullsized":"2023-10-30T14:26:36.417782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2192,"log_dups_size":0,"ondisk_log_size":2192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:48:10.995137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.003812562,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":127,"num_read_kb":26261,"num_write":1294,"num_write_kb":24882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.0","version":"103'1186","reported_seq":1498,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.839806+0000","last_change":"2023-10-30T13:23:18.985318+0000","last_active":"2023-10-30T14:26:33.839806+0000","last_peered":"2023-10-30T14:26:33.839806+0000","last_clean":"2023-10-30T14:26:33.839806+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T14:26:33.839806+0000","last_undegraded":"2023-10-30T14:26:33.839806+0000","last_fullsized":"2023-10-30T14:26:33.839806+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1186,"log_dups_size":0,"ondisk_log_size":1186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:16.754084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":256,"num_read_kb":488007,"num_write":1246,"num_write_kb":83351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"103'1973","reported_seq":4835,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.718436+0000","last_change":"2023-10-30T14:26:34.386961+0000","last_active":"2023-10-30T14:26:35.718436+0000","last_peered":"2023-10-30T14:26:35.718436+0000","last_clean":"2023-10-30T14:26:35.718436+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T14:26:35.718436+0000","last_undegraded":"2023-10-30T14:26:35.718436+0000","last_fullsized":"2023-10-30T14:26:35.718436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1973,"log_dups_size":0,"ondisk_log_size":1973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:03.616902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.92159720499999997,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":129,"num_read_kb":13045,"num_write":1222,"num_write_kb":11637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1","version":"108'1950","reported_seq":3194,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.981824+0000","last_change":"2023-10-30T13:23:18.949565+0000","last_active":"2023-10-30T14:26:33.981824+0000","last_peered":"2023-10-30T14:26:33.981824+0000","last_clean":"2023-10-30T14:26:33.981824+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T14:26:33.981824+0000","last_undegraded":"2023-10-30T14:26:33.981824+0000","last_fullsized":"2023-10-30T14:26:33.981824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1950,"log_dups_size":0,"ondisk_log_size":1950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:55:46.029368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40060913,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":1167,"num_read_kb":359975,"num_write":2020,"num_write_kb":69890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"103'2255","reported_seq":5066,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.718578+0000","last_change":"2023-10-30T14:26:33.665105+0000","last_active":"2023-10-30T14:26:35.718578+0000","last_peered":"2023-10-30T14:26:35.718578+0000","last_clean":"2023-10-30T14:26:35.718578+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T14:26:35.718578+0000","last_undegraded":"2023-10-30T14:26:35.718578+0000","last_fullsized":"2023-10-30T14:26:35.718578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2255,"log_dups_size":0,"ondisk_log_size":2255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:37:34.538070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40657480699999998,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":145,"num_read_kb":55105,"num_write":1354,"num_write_kb":28005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.2","version":"103'1488","reported_seq":1793,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.949707+0000","last_change":"2023-10-30T13:23:18.984535+0000","last_active":"2023-10-30T14:26:32.949707+0000","last_peered":"2023-10-30T14:26:32.949707+0000","last_clean":"2023-10-30T14:26:32.949707+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T14:26:32.949707+0000","last_undegraded":"2023-10-30T14:26:32.949707+0000","last_fullsized":"2023-10-30T14:26:32.949707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1488,"log_dups_size":0,"ondisk_log_size":1488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:28.405405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":235,"num_read_kb":509467,"num_write":1537,"num_write_kb":83943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"103'2200","reported_seq":5023,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724420+0000","last_change":"2023-10-30T14:26:33.550991+0000","last_active":"2023-10-30T14:26:35.724420+0000","last_peered":"2023-10-30T14:26:35.724420+0000","last_clean":"2023-10-30T14:26:35.724420+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T14:26:35.724420+0000","last_undegraded":"2023-10-30T14:26:35.724420+0000","last_fullsized":"2023-10-30T14:26:35.724420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2200,"log_dups_size":0,"ondisk_log_size":2200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:34.572699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.59694472600000004,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":143,"num_read_kb":13239,"num_write":1306,"num_write_kb":17120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.4","version":"103'610","reported_seq":920,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.258720+0000","last_change":"2023-10-30T13:23:18.958210+0000","last_active":"2023-10-30T14:26:33.258720+0000","last_peered":"2023-10-30T14:26:33.258720+0000","last_clean":"2023-10-30T14:26:33.258720+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T14:26:33.258720+0000","last_undegraded":"2023-10-30T14:26:33.258720+0000","last_fullsized":"2023-10-30T14:26:33.258720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:34.653005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":230,"num_read_kb":520969,"num_write":671,"num_write_kb":95828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"103'2173","reported_seq":4990,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.726608+0000","last_change":"2023-10-30T14:26:34.025524+0000","last_active":"2023-10-30T14:26:35.726608+0000","last_peered":"2023-10-30T14:26:35.726608+0000","last_clean":"2023-10-30T14:26:35.726608+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T14:26:35.726608+0000","last_undegraded":"2023-10-30T14:26:35.726608+0000","last_fullsized":"2023-10-30T14:26:35.726608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:45.716921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.075105449,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":60418,"num_write":1286,"num_write_kb":36950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.5","version":"103'1046","reported_seq":1330,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.841399+0000","last_change":"2023-10-30T13:23:18.985185+0000","last_active":"2023-10-30T14:26:33.841399+0000","last_peered":"2023-10-30T14:26:33.841399+0000","last_clean":"2023-10-30T14:26:33.841399+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T14:26:33.841399+0000","last_undegraded":"2023-10-30T14:26:33.841399+0000","last_fullsized":"2023-10-30T14:26:33.841399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:36.230072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44105804,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":214,"num_read_kb":377318,"num_write":1088,"num_write_kb":69045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"103'2202","reported_seq":5034,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.718743+0000","last_change":"2023-10-30T14:26:33.629915+0000","last_active":"2023-10-30T14:26:35.718743+0000","last_peered":"2023-10-30T14:26:35.718743+0000","last_clean":"2023-10-30T14:26:35.718743+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T14:26:35.718743+0000","last_undegraded":"2023-10-30T14:26:35.718743+0000","last_fullsized":"2023-10-30T14:26:35.718743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:15.644068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.36741958800000002,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":157,"num_read_kb":35362,"num_write":1332,"num_write_kb":31772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.6","version":"107'1126","reported_seq":1419,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.259087+0000","last_change":"2023-10-30T13:23:18.978085+0000","last_active":"2023-10-30T14:26:33.259087+0000","last_peered":"2023-10-30T14:26:33.259087+0000","last_clean":"2023-10-30T14:26:33.259087+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T14:26:33.259087+0000","last_undegraded":"2023-10-30T14:26:33.259087+0000","last_fullsized":"2023-10-30T14:26:33.259087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1126,"log_dups_size":0,"ondisk_log_size":1126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:47.588972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":247,"num_read_kb":365780,"num_write":1781,"num_write_kb":65025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"103'2077","reported_seq":4909,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.895121+0000","last_change":"2023-10-30T14:26:33.252060+0000","last_active":"2023-10-30T14:26:34.895121+0000","last_peered":"2023-10-30T14:26:34.895121+0000","last_clean":"2023-10-30T14:26:34.895121+0000","last_became_active":"2023-10-30T13:24:01.001412+0000","last_became_peered":"2023-10-30T13:24:01.001412+0000","last_unstale":"2023-10-30T14:26:34.895121+0000","last_undegraded":"2023-10-30T14:26:34.895121+0000","last_fullsized":"2023-10-30T14:26:34.895121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2077,"log_dups_size":0,"ondisk_log_size":2077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:37:55.891565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.32700652899999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":137,"num_read_kb":5811,"num_write":1214,"num_write_kb":16143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.7","version":"103'468","reported_seq":773,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.928722+0000","last_change":"2023-10-30T13:23:18.970566+0000","last_active":"2023-10-30T14:26:32.928722+0000","last_peered":"2023-10-30T14:26:32.928722+0000","last_clean":"2023-10-30T14:26:32.928722+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T14:26:32.928722+0000","last_undegraded":"2023-10-30T14:26:32.928722+0000","last_fullsized":"2023-10-30T14:26:32.928722+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:03.851889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":281,"num_read_kb":442743,"num_write":555,"num_write_kb":70797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"103'2179","reported_seq":5010,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.720013+0000","last_change":"2023-10-30T14:26:34.649272+0000","last_active":"2023-10-30T14:26:35.720013+0000","last_peered":"2023-10-30T14:26:35.720013+0000","last_clean":"2023-10-30T14:26:35.720013+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T14:26:35.720013+0000","last_undegraded":"2023-10-30T14:26:35.720013+0000","last_fullsized":"2023-10-30T14:26:35.720013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:08.185639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.183697322,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":144,"num_read_kb":25445,"num_write":1326,"num_write_kb":20705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.8","version":"103'544","reported_seq":809,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.838454+0000","last_change":"2023-10-30T13:23:18.986394+0000","last_active":"2023-10-30T14:26:33.838454+0000","last_peered":"2023-10-30T14:26:33.838454+0000","last_clean":"2023-10-30T14:26:33.838454+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T14:26:33.838454+0000","last_undegraded":"2023-10-30T14:26:33.838454+0000","last_fullsized":"2023-10-30T14:26:33.838454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:59.986981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":236,"num_read_kb":274544,"num_write":603,"num_write_kb":42177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"103'2113","reported_seq":4960,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724178+0000","last_change":"2023-10-30T14:26:33.516141+0000","last_active":"2023-10-30T14:26:35.724178+0000","last_peered":"2023-10-30T14:26:35.724178+0000","last_clean":"2023-10-30T14:26:35.724178+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T14:26:35.724178+0000","last_undegraded":"2023-10-30T14:26:35.724178+0000","last_fullsized":"2023-10-30T14:26:35.724178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:05.930508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.56267233699999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":144,"num_read_kb":34655,"num_write":1307,"num_write_kb":20329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.9","version":"103'1061","reported_seq":1331,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.258600+0000","last_change":"2023-10-30T13:23:18.963905+0000","last_active":"2023-10-30T14:26:33.258600+0000","last_peered":"2023-10-30T14:26:33.258600+0000","last_clean":"2023-10-30T14:26:33.258600+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T14:26:33.258600+0000","last_undegraded":"2023-10-30T14:26:33.258600+0000","last_fullsized":"2023-10-30T14:26:33.258600+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1061,"log_dups_size":0,"ondisk_log_size":1061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":237,"num_read_kb":249991,"num_write":1213,"num_write_kb":43538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"103'2263","reported_seq":5048,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.357975+0000","last_change":"2023-10-30T14:26:36.405912+0000","last_active":"2023-10-30T14:26:36.357975+0000","last_peered":"2023-10-30T14:26:36.357975+0000","last_clean":"2023-10-30T14:26:36.357975+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T14:26:36.357975+0000","last_undegraded":"2023-10-30T14:26:36.357975+0000","last_fullsized":"2023-10-30T14:26:36.357975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2263,"log_dups_size":0,"ondisk_log_size":2263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:29:31.853578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.93815086199999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":129,"num_read_kb":1227,"num_write":1352,"num_write_kb":12536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.a","version":"103'1473","reported_seq":1808,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.258472+0000","last_change":"2023-10-30T13:23:18.952302+0000","last_active":"2023-10-30T14:26:33.258472+0000","last_peered":"2023-10-30T14:26:33.258472+0000","last_clean":"2023-10-30T14:26:33.258472+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T14:26:33.258472+0000","last_undegraded":"2023-10-30T14:26:33.258472+0000","last_fullsized":"2023-10-30T14:26:33.258472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:12.827417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":321,"num_read_kb":438308,"num_write":2001,"num_write_kb":98279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"103'2090","reported_seq":4928,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.369993+0000","last_change":"2023-10-30T14:26:36.417706+0000","last_active":"2023-10-30T14:26:36.369993+0000","last_peered":"2023-10-30T14:26:36.369993+0000","last_clean":"2023-10-30T14:26:36.369993+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T14:26:36.369993+0000","last_undegraded":"2023-10-30T14:26:36.369993+0000","last_fullsized":"2023-10-30T14:26:36.369993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:50:45.036636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.94968995099999998,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":141,"num_read_kb":11852,"num_write":1237,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.b","version":"103'1064","reported_seq":1380,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.840499+0000","last_change":"2023-10-30T13:23:18.984912+0000","last_active":"2023-10-30T14:26:33.840499+0000","last_peered":"2023-10-30T14:26:33.840499+0000","last_clean":"2023-10-30T14:26:33.840499+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T14:26:33.840499+0000","last_undegraded":"2023-10-30T14:26:33.840499+0000","last_fullsized":"2023-10-30T14:26:33.840499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:27:16.205042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":296,"num_read_kb":429426,"num_write":1159,"num_write_kb":75926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"103'2153","reported_seq":4987,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.898995+0000","last_change":"2023-10-30T14:26:33.347987+0000","last_active":"2023-10-30T14:26:34.898995+0000","last_peered":"2023-10-30T14:26:34.898995+0000","last_clean":"2023-10-30T14:26:34.898995+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T14:26:34.898995+0000","last_undegraded":"2023-10-30T14:26:34.898995+0000","last_fullsized":"2023-10-30T14:26:34.898995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2153,"log_dups_size":0,"ondisk_log_size":2153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:25.445527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.419978554,"stat_sum":{"num_bytes":5,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":137,"num_read_kb":40109,"num_write":1287,"num_write_kb":25283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.1d","version":"103'2173","reported_seq":4979,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723792+0000","last_change":"2023-10-30T14:26:34.023924+0000","last_active":"2023-10-30T14:26:35.723792+0000","last_peered":"2023-10-30T14:26:35.723792+0000","last_clean":"2023-10-30T14:26:35.723792+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T14:26:35.723792+0000","last_undegraded":"2023-10-30T14:26:35.723792+0000","last_fullsized":"2023-10-30T14:26:35.723792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:40:49.561271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.073619836,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":129,"num_read_kb":12975,"num_write":1318,"num_write_kb":20116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1c","version":"103'734","reported_seq":1022,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.840725+0000","last_change":"2023-10-30T13:23:18.993372+0000","last_active":"2023-10-30T14:26:33.840725+0000","last_peered":"2023-10-30T14:26:33.840725+0000","last_clean":"2023-10-30T14:26:33.840725+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T14:26:33.840725+0000","last_undegraded":"2023-10-30T14:26:33.840725+0000","last_fullsized":"2023-10-30T14:26:33.840725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":734,"log_dups_size":0,"ondisk_log_size":734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:41.370678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":304,"num_read_kb":264369,"num_write":877,"num_write_kb":48007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"103'2635","reported_seq":5318,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723624+0000","last_change":"2023-10-30T14:26:33.971887+0000","last_active":"2023-10-30T14:26:35.723624+0000","last_peered":"2023-10-30T14:26:35.723624+0000","last_clean":"2023-10-30T14:26:35.723624+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T14:26:35.723624+0000","last_undegraded":"2023-10-30T14:26:35.723624+0000","last_fullsized":"2023-10-30T14:26:35.723624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2635,"log_dups_size":0,"ondisk_log_size":2635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:02.739787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.021840552,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":171,"num_read_kb":40337,"num_write":1456,"num_write_kb":34441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1d","version":"103'1179","reported_seq":1479,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.840962+0000","last_change":"2023-10-30T13:23:18.987633+0000","last_active":"2023-10-30T14:26:33.840962+0000","last_peered":"2023-10-30T14:26:33.840962+0000","last_clean":"2023-10-30T14:26:33.840962+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T14:26:33.840962+0000","last_undegraded":"2023-10-30T14:26:33.840962+0000","last_fullsized":"2023-10-30T14:26:33.840962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1179,"log_dups_size":0,"ondisk_log_size":1179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:09:33.324898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":255,"num_read_kb":502436,"num_write":1236,"num_write_kb":99955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"103'2021","reported_seq":4887,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.721600+0000","last_change":"2023-10-30T14:26:34.532204+0000","last_active":"2023-10-30T14:26:35.721600+0000","last_peered":"2023-10-30T14:26:35.721600+0000","last_clean":"2023-10-30T14:26:35.721600+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T14:26:35.721600+0000","last_undegraded":"2023-10-30T14:26:35.721600+0000","last_fullsized":"2023-10-30T14:26:35.721600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2021,"log_dups_size":0,"ondisk_log_size":2021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:06.800217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.015373858,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":13189,"num_write":1249,"num_write_kb":15084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1e","version":"103'779","reported_seq":1032,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.329244+0000","last_change":"2023-10-30T13:23:18.958470+0000","last_active":"2023-10-30T14:26:33.329244+0000","last_peered":"2023-10-30T14:26:33.329244+0000","last_clean":"2023-10-30T14:26:33.329244+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T14:26:33.329244+0000","last_undegraded":"2023-10-30T14:26:33.329244+0000","last_fullsized":"2023-10-30T14:26:33.329244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":779,"log_dups_size":0,"ondisk_log_size":779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:21.507744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":159,"num_read_kb":369044,"num_write":787,"num_write_kb":79251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"103'2191","reported_seq":5138,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.518541+0000","last_change":"2023-10-30T14:26:36.567193+0000","last_active":"2023-10-30T14:26:36.518541+0000","last_peered":"2023-10-30T14:26:36.518541+0000","last_clean":"2023-10-30T14:26:36.518541+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T14:26:36.518541+0000","last_undegraded":"2023-10-30T14:26:36.518541+0000","last_fullsized":"2023-10-30T14:26:36.518541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2191,"log_dups_size":0,"ondisk_log_size":2191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:03:26.764254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.043790548,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":262,"num_read_kb":42307,"num_write":1336,"num_write_kb":30630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1f","version":"103'1010","reported_seq":1328,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.516670+0000","last_change":"2023-10-30T13:23:18.980952+0000","last_active":"2023-10-30T14:26:33.516670+0000","last_peered":"2023-10-30T14:26:33.516670+0000","last_clean":"2023-10-30T14:26:33.516670+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T14:26:33.516670+0000","last_undegraded":"2023-10-30T14:26:33.516670+0000","last_fullsized":"2023-10-30T14:26:33.516670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1010,"log_dups_size":0,"ondisk_log_size":1010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:44.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777798,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":313,"num_read_kb":340671,"num_write":1085,"num_write_kb":63851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":11,"num_objects":2377,"num_object_clones":2,"num_object_copies":4754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":9389,"num_read_kb":1724409,"num_write":83589,"num_write_kb":1459751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138028,"ondisk_log_size":138028,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1284334055,"num_objects":1569,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1569,"num_whiteouts":0,"num_read":10652,"num_read_kb":12762809,"num_write":35633,"num_write_kb":2343104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2568822784,"data_stored":2568668110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31979,"ondisk_log_size":31979,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575614,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590972,"kb_used_data":389104,"kb_used_omap":1,"kb_used_meta":201854,"kb_avail":93780868,"statfs":{"total":96636764160,"available":96031608832,"internally_reserved":0,"allocated":398442496,"data_stored":397794175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":206698955},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.746,"15min":0.80500000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":1.6339999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.82999999999999996,"15min":0.91400000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.286,"5min":1.3160000000000001,"15min":1.448},"last":1.7250000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.85999999999999999,"15min":0.88300000000000001},"min":{"1min":0.46300000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0840000000000001,"5min":2.452,"15min":2.452},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.78600000000000003,"15min":0.84199999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.47399999999999998},"max":{"1min":1.028,"5min":1.198,"15min":1.3520000000000001},"last":1.1619999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.78600000000000003,"15min":0.82799999999999996},"min":{"1min":0.48999999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.169,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.88,"15min":0.95099999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.024,"5min":1.3009999999999999,"15min":1.5569999999999999},"last":0.94299999999999995}]},{"osd":3,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.294,"5min":1.405,"15min":1.405},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.028,"15min":1.004},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.21,"5min":1.4610000000000001,"15min":1.5269999999999999},"last":0.747}]},{"osd":4,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.378,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.2010000000000001,"5min":1.4350000000000001,"15min":1.732},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0309999999999999,"15min":1.03},"min":{"1min":0.495,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.266,"5min":1.4890000000000001,"15min":1.673},"last":0.97799999999999998}]},{"osd":5,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.1439999999999999,"15min":1.0800000000000001},"min":{"1min":0.42999999999999999,"5min":0.375,"15min":0.375},"max":{"1min":1.3720000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1120000000000001,"15min":1.1619999999999999},"min":{"1min":0.57099999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7809999999999999},"last":0.90200000000000002}]},{"osd":6,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.095,"15min":1.1060000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.456,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.151,"15min":1.1930000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3779999999999999,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.65700000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575617,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505380,"kb_used_data":339672,"kb_used_omap":1,"kb_used_meta":165694,"kb_avail":93866460,"statfs":{"total":96636764160,"available":96119255040,"internally_reserved":0,"allocated":347824128,"data_stored":347179842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169671111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.75700000000000001,"15min":0.80100000000000005},"min":{"1min":0.51100000000000001,"5min":0.46700000000000003,"15min":0.375},"max":{"1min":0.89600000000000002,"5min":1.306,"15min":1.363},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.88900000000000001,"15min":0.95299999999999996},"min":{"1min":0.53500000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3979999999999999,"5min":1.4099999999999999,"15min":1.4370000000000001},"last":0.625}]},{"osd":1,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.72299999999999998,"15min":0.77400000000000002},"min":{"1min":0.54100000000000004,"5min":0.46999999999999997,"15min":0.46800000000000003},"max":{"1min":0.875,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":3.452},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.47399999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.127,"5min":1.706,"15min":1.706},"last":0.85299999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.81299999999999994,"15min":0.82999999999999996},"min":{"1min":0.57699999999999996,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.1679999999999999,"5min":1.29,"15min":1.571},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.872,"5min":0.96899999999999997,"15min":0.93700000000000006},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.286,"5min":5.4480000000000004,"15min":5.4480000000000004},"last":3.6459999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0229999999999999,"15min":1.0329999999999999},"min":{"1min":0.68000000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":3.5169999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.103,"15min":1.1160000000000001},"min":{"1min":0.68899999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.23,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":3.5830000000000002}]},{"osd":4,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.90300000000000002,"15min":0.90700000000000003},"min":{"1min":0.70799999999999996,"5min":0.434,"15min":0.434},"max":{"1min":1.276,"5min":1.3640000000000001,"15min":1.4339999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.002,"15min":1.0309999999999999},"min":{"1min":0.71999999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.153,"5min":1.595,"15min":1.595},"last":0.443}]},{"osd":5,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.159,"15min":1.03},"min":{"1min":0.77400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.2889999999999999,"5min":3.0219999999999998,"15min":3.0219999999999998},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.0089999999999999,"15min":1.0509999999999999},"min":{"1min":0.75600000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.135,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":0.58199999999999996}]},{"osd":7,"last update":"Mon Oct 30 14:26:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.204,"15min":1.196},"min":{"1min":1.048,"5min":0.58699999999999997,"15min":0.52600000000000002},"max":{"1min":1.3740000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1599999999999999,"15min":1.1359999999999999},"min":{"1min":0.83399999999999996,"5min":0.752,"15min":0.63700000000000001},"max":{"1min":1.3440000000000001,"5min":2.806,"15min":2.806},"last":0.505}]}]},{"osd":5,"up_from":13,"seq":55834575614,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565832,"kb_used_data":368188,"kb_used_omap":1,"kb_used_meta":197630,"kb_avail":93806008,"statfs":{"total":96636764160,"available":96057352192,"internally_reserved":0,"allocated":377024512,"data_stored":376381993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202373574},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72399999999999998,"15min":0.86499999999999999},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.40999999999999998},"max":{"1min":1.0860000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.95099999999999996,"15min":0.95799999999999996},"min":{"1min":0.60499999999999998,"5min":0.58699999999999997,"15min":0.57199999999999995},"max":{"1min":1.2410000000000001,"5min":1.617,"15min":1.9219999999999999},"last":0.66200000000000003}]},{"osd":1,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.81999999999999995,"15min":0.84099999999999997},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":1.911},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.77000000000000002,"15min":0.83799999999999997},"min":{"1min":0.52800000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.97099999999999997,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.58199999999999996}]},{"osd":2,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.88100000000000001,"15min":0.88500000000000001},"min":{"1min":0.59699999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.2589999999999999,"5min":1.331,"15min":1.3999999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81299999999999994,"15min":0.84799999999999998},"min":{"1min":0.59999999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.331,"5min":1.331,"15min":1.796},"last":0.746}]},{"osd":3,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0049999999999999,"15min":1.006},"min":{"1min":0.63700000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92000000000000004,"15min":0.999},"min":{"1min":0.71899999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.1930000000000001,"5min":1.3460000000000001,"15min":2.0030000000000001},"last":0.79100000000000004}]},{"osd":4,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.88400000000000001,"15min":0.879},"min":{"1min":0.56899999999999995,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.2729999999999999,"5min":2.8170000000000002,"15min":2.8170000000000002},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.159,"5min":1.024,"15min":1.03},"min":{"1min":0.86799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.6299999999999999,"5min":2.2400000000000002,"15min":2.262},"last":0.72199999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.0880000000000001,"15min":1.1299999999999999},"min":{"1min":0.80900000000000005,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.1499999999999999,"5min":2.3090000000000002,"15min":2.7949999999999999},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.097,"5min":1.0229999999999999,"15min":1.056},"min":{"1min":0.71899999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.49,"5min":2.161,"15min":2.1629999999999998},"last":0.78000000000000003}]},{"osd":7,"last update":"Mon Oct 30 14:26:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0209999999999999,"15min":1.091},"min":{"1min":0.78100000000000003,"5min":0.317,"15min":0.317},"max":{"1min":1.3440000000000001,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.02},{"interface":"front","average":{"1min":1.149,"5min":1.1299999999999999,"15min":1.153},"min":{"1min":0.95099999999999996,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.4019999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.1100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575614,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464424,"kb_used_data":266716,"kb_used_omap":1,"kb_used_meta":197694,"kb_avail":93907416,"statfs":{"total":96636764160,"available":96161193984,"internally_reserved":0,"allocated":273117184,"data_stored":272461938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202439110},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"min":{"1min":0.48799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1259999999999999,"5min":1.3500000000000001,"15min":1.446},"last":0.439},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.71099999999999997,"15min":0.82299999999999995},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.875,"5min":1.659,"15min":1.659},"last":0.61899999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.56799999999999995,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.141,"5min":1.589,"15min":1.589},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.84399999999999997,"15min":0.84699999999999998},"min":{"1min":0.58999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.075,"5min":1.532,"15min":1.532},"last":0.49199999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.76100000000000001,"15min":0.85999999999999999},"min":{"1min":0.55800000000000005,"5min":0.5,"15min":0.5},"max":{"1min":0.94899999999999995,"5min":1.607,"15min":1.607},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84699999999999998,"15min":0.92200000000000004},"min":{"1min":0.57199999999999995,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.218,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.99099999999999999,"15min":0.97499999999999998},"min":{"1min":0.66300000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.242,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.099,"15min":1.0780000000000001},"min":{"1min":0.879,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3480000000000001,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.98899999999999999,"15min":1.042},"min":{"1min":0.77000000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.3009999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.65400000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.1279999999999999,"5min":1.3220000000000001,"15min":1.3280000000000001},"last":0.316}]},{"osd":6,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.97899999999999998,"15min":0.96099999999999997},"min":{"1min":0.78800000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.3100000000000001,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.042,"15min":1.0700000000000001},"min":{"1min":0.78300000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.238,"5min":1.677,"15min":1.728},"last":0.51800000000000002}]},{"osd":7,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.137,"15min":1.1040000000000001},"min":{"1min":1.0820000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.417,"5min":1.4750000000000001,"15min":1.518},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.204,"15min":1.1799999999999999},"min":{"1min":1.056,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4670000000000001,"5min":1.587,"15min":1.587},"last":0.33600000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575612,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589364,"kb_used_data":381992,"kb_used_omap":1,"kb_used_meta":207358,"kb_avail":93782476,"statfs":{"total":96636764160,"available":96033255424,"internally_reserved":0,"allocated":391159808,"data_stored":390501706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":212335048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38100000000000001,"15min":0.40600000000000003},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71799999999999997,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.308},"max":{"1min":0.747,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.47399999999999998}]},{"osd":1,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":0.76500000000000001,"5min":1.403,"15min":1.403},"last":1.895},{"interface":"front","average":{"1min":0.438,"5min":0.51700000000000002,"15min":0.54800000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65800000000000003,"5min":1.752,"15min":1.752},"last":1.8420000000000001}]},{"osd":2,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56399999999999995,"15min":0.59999999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69499999999999995,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.41699999999999998,"5min":0.29999999999999999,"15min":0.29299999999999998},"max":{"1min":0.78000000000000003,"5min":1.73,"15min":1.73},"last":0.56999999999999995}]},{"osd":4,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.80300000000000005},"min":{"1min":0.53600000000000003,"5min":0.379,"15min":0.379},"max":{"1min":1.6519999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.80400000000000005,"15min":0.82399999999999995},"min":{"1min":0.55800000000000005,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.679,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77400000000000002,"15min":0.83299999999999996},"min":{"1min":0.61499999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98899999999999999,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.81000000000000005,"15min":0.85699999999999998},"min":{"1min":0.58399999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.071,"5min":1.486,"15min":1.486},"last":0.81799999999999995}]},{"osd":6,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.85199999999999998,"15min":0.88400000000000001},"min":{"1min":0.55600000000000005,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.048,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.042},{"interface":"front","average":{"1min":0.878,"5min":0.91500000000000004,"15min":0.93999999999999995},"min":{"1min":0.71999999999999997,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0409999999999999,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.077}]},{"osd":7,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83199999999999996,"15min":0.86199999999999999},"min":{"1min":0.64200000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.1100000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79100000000000004,"15min":0.82599999999999996},"min":{"1min":0.56299999999999994,"5min":0.501,"15min":0.501},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.76200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299400,"kb_used_data":200252,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94072440,"statfs":{"total":96636764160,"available":96330178560,"internally_reserved":0,"allocated":205058048,"data_stored":204414271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101513670},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.433,"15min":0.45800000000000002},"min":{"1min":0.254,"5min":0.246,"15min":0.23899999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.89300000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":0.94599999999999995,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.42199999999999999,"15min":0.44700000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72999999999999998,"5min":1.956,"15min":1.956},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.36199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.153}]},{"osd":3,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81999999999999995,"5min":1.988,"15min":1.988},"last":1.2010000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49099999999999999,"15min":0.51800000000000002},"min":{"1min":0.32300000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.90100000000000002,"5min":2.335,"15min":2.335},"last":0.67900000000000005}]},{"osd":4,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.60899999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5409999999999999,"5min":2.302,"15min":2.302},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84299999999999997,"15min":0.84099999999999997},"min":{"1min":0.625,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6879999999999999,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.91200000000000003}]},{"osd":5,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.77100000000000002,"15min":0.80600000000000005},"min":{"1min":0.64500000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82699999999999996,"15min":0.84499999999999997},"min":{"1min":0.73099999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.587,"5min":1.875,"15min":1.875},"last":0.85299999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.81599999999999995,"15min":0.82799999999999996},"min":{"1min":0.61299999999999999,"5min":0.502,"15min":0.48099999999999998},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84499999999999997,"15min":0.86799999999999999},"min":{"1min":0.60999999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.429,"5min":1.921,"15min":1.921},"last":1.2130000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.86099999999999999,"15min":0.86399999999999999},"min":{"1min":0.66100000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.131,"5min":4.5739999999999998,"15min":4.5739999999999998},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.88900000000000001,"15min":0.91700000000000004},"min":{"1min":0.64300000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6080000000000001,"5min":2.375,"15min":2.375},"last":1.1659999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575615,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490668,"kb_used_data":329060,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":93881172,"statfs":{"total":96636764160,"available":96134320128,"internally_reserved":0,"allocated":336957440,"data_stored":336316405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165476806},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.438,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.504,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.51600000000000001},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.47799999999999998,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.41499999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.46999999999999997,"15min":0.48399999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54800000000000004,"5min":1.954,"15min":1.954},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.45800000000000002,"15min":0.47799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63200000000000001,"5min":2.274,"15min":2.274},"last":5.7039999999999997}]},{"osd":3,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53600000000000003,"15min":0.54900000000000004},"min":{"1min":0.35599999999999998,"5min":0.35399999999999998,"15min":0.35099999999999998},"max":{"1min":0.60099999999999998,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":1.3220000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.436,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.70199999999999996,"5min":2.625,"15min":2.625},"last":1.3600000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.80200000000000005,"15min":0.83999999999999997},"min":{"1min":0.56499999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93300000000000005,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.82299999999999995,"15min":0.88},"min":{"1min":0.629,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.91200000000000003,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.752}]},{"osd":5,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82399999999999995,"15min":0.878},"min":{"1min":0.58099999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94799999999999995,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.85099999999999998,"15min":0.90100000000000002},"min":{"1min":0.60399999999999998,"5min":0.499,"15min":0.499},"max":{"1min":0.879,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.52200000000000002}]},{"osd":6,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.91000000000000003,"15min":0.84199999999999997},"min":{"1min":0.73799999999999999,"5min":0.624,"15min":0.56499999999999995},"max":{"1min":1.0289999999999999,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.81299999999999994,"15min":0.89700000000000002},"min":{"1min":0.56200000000000006,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.95799999999999996,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.56399999999999995}]},{"osd":7,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.80600000000000005,"15min":0.82999999999999996},"min":{"1min":0.59599999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.95999999999999996,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":0.626},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.82799999999999996,"15min":0.85199999999999998},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93799999999999994,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.66900000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575612,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421176,"kb_used_data":245740,"kb_used_omap":1,"kb_used_meta":175422,"kb_avail":93950664,"statfs":{"total":96636764160,"available":96205479936,"internally_reserved":0,"allocated":251637760,"data_stored":250992448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179632582},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.38100000000000001,"15min":0.40500000000000003},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54400000000000004,"5min":0.66100000000000003,"15min":0.94999999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.49299999999999999,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":1.0229999999999999},"last":0.65800000000000003}]},{"osd":2,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.45300000000000001,"15min":0.46700000000000003},"min":{"1min":0.307,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":1.05,"15min":1.05},"last":1.302},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.47099999999999997,"15min":0.498},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.29099999999999998},"max":{"1min":0.59699999999999998,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.83599999999999997}]},{"osd":3,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52900000000000003,"15min":0.54900000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.34499999999999997},"max":{"1min":0.73299999999999998,"5min":1.236,"15min":1.236},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51600000000000001,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.74399999999999999,"5min":1.054,"15min":1.054},"last":1.5800000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.78600000000000003,"15min":0.80300000000000005},"min":{"1min":0.65900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":0.88800000000000001,"5min":1.103,"15min":1.29},"last":1.385},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76600000000000001,"15min":0.81000000000000005},"min":{"1min":0.59299999999999997,"5min":0.53800000000000003,"15min":0.48399999999999999},"max":{"1min":0.95999999999999996,"5min":1.012,"15min":1.2210000000000001},"last":0.42599999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.78700000000000003,"15min":0.81200000000000006},"min":{"1min":0.64700000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.93899999999999995,"5min":1.0509999999999999,"15min":1.347},"last":1.7709999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.81599999999999995,"15min":0.85199999999999998},"min":{"1min":0.63,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.95899999999999996,"5min":1.2130000000000001,"15min":1.3939999999999999},"last":1.8779999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81499999999999995,"15min":0.84399999999999997},"min":{"1min":0.64500000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.95799999999999996,"5min":1.222,"15min":1.486},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79700000000000004,"15min":0.83399999999999996},"min":{"1min":0.64400000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0149999999999999,"5min":1.113,"15min":1.1619999999999999},"last":1.8109999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.81699999999999995,"15min":0.85099999999999998},"min":{"1min":0.625,"5min":0.625,"15min":0.58599999999999997},"max":{"1min":0.97599999999999998,"5min":1.2509999999999999,"15min":1.29},"last":1.8440000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84199999999999997,"15min":0.872},"min":{"1min":0.70799999999999996,"5min":0.61799999999999999,"15min":0.57299999999999995},"max":{"1min":0.96099999999999997,"5min":1.173,"15min":1.4419999999999999},"last":0.47999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249647104,"data_stored":249627338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335572992,"data_stored":335549765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203661312,"data_stored":203639529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389746688,"data_stored":389725939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":271699968,"data_stored":271686166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":375627776,"data_stored":375607251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":346419200,"data_stored":346404076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":396447744,"data_stored":396428046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":11,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:26:38.193 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T14:26:38.194 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:26:38.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.370+0000 7fd93aead640 1 Processor -- start 2023-10-30T14:26:38.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.370+0000 7fd93aead640 1 -- start start 2023-10-30T14:26:38.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd93aead640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9341538f0 0x7fd934159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd93aead640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9340ca6a0 con 0x7fd9340cefa0 2023-10-30T14:26:38.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd93aead640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93415d300 con 0x7fd93415cd50 2023-10-30T14:26:38.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd93aead640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93415d5e0 con 0x7fd9341538f0 2023-10-30T14:26:38.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd93a6ac640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd9340cefa0 0x7fd9341640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38020/0 (socket says 172.21.15.46:38020) 2023-10-30T14:26:38.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd93a6ac640 1 -- 172.21.15.46:0/3301184136 learned_addr learned my addr 172.21.15.46:0/3301184136 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:38.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd939eab640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9341538f0 0x7fd934159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606225223 0 0) 0x7fd9340ca6a0 con 0x7fd9340cefa0 2023-10-30T14:26:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd920003530 con 0x7fd9340cefa0 2023-10-30T14:26:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815173844 0 0) 0x7fd93415d300 con 0x7fd93415cd50 2023-10-30T14:26:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9340ca6a0 con 0x7fd93415cd50 2023-10-30T14:26:38.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3777823702 0 0) 0x7fd920003530 con 0x7fd9340cefa0 2023-10-30T14:26:38.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9341538f0 msgr2=0x7fd934159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:38.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9341538f0 0x7fd934159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 >> v1:172.21.15.154:6789/0 conn(0x7fd93415cd50 legacy=0x7fd9340c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93415d8c0 con 0x7fd9340cefa0 2023-10-30T14:26:38.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7fd92c001a00 con 0x7fd9340cefa0 2023-10-30T14:26:38.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd939eab640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9341538f0 0x7fd934159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:26:38.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 >> v1:172.21.15.46:6789/0 conn(0x7fd9340cefa0 legacy=0x7fd9341640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd920003b40 0x7fd920003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9200047d0 0x7fd920004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9341538f0 0x7fd920005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93415d5e0 con 0x7fd920003b40 2023-10-30T14:26:38.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92000b9c0 con 0x7fd9200047d0 2023-10-30T14:26:38.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92000bb80 con 0x7fd9341538f0 2023-10-30T14:26:38.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd939eab640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd920003b40 0x7fd920003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd9396aa640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9200047d0 0x7fd920004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd93a6ac640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9341538f0 0x7fd920005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd939eab640 1 -- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9341538f0 msgr2=0x7fd920005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd939eab640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9341538f0 0x7fd920005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd939eab640 1 -- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9200047d0 msgr2=0x7fd920004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd939eab640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9200047d0 0x7fd920004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd939eab640 1 -- 172.21.15.46:0/3301184136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd930009420 con 0x7fd920003b40 2023-10-30T14:26:38.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.374+0000 7fd9396aa640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9200047d0 0x7fd920004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.378+0000 7fd939eab640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd920003b40 0x7fd920003f40 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fd930009c80 tx=0x7fd93001cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e86de6bb6cea4b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.378+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9300a5850 con 0x7fd920003b40 2023-10-30T14:26:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.378+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9300a59b0 con 0x7fd920003b40 2023-10-30T14:26:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.378+0000 7fd938ea9640 1 -- 172.21.15.46:0/3301184136 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9300a5d60 con 0x7fd920003b40 2023-10-30T14:26:38.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.378+0000 7fd93aead640 1 -- 172.21.15.46:0/3301184136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd920003b40 msgr2=0x7fd920003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.378+0000 7fd93aead640 1 --2- 172.21.15.46:0/3301184136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd920003b40 0x7fd920003f40 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fd930009c80 tx=0x7fd93001cb90 comp rx=0 tx=0).stop 2023-10-30T14:26:38.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.378+0000 7fd93a6ac640 1 -- 172.21.15.46:0/3301184136 reap_dead start 2023-10-30T14:26:38.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.378+0000 7fd93aead640 1 -- 172.21.15.46:0/3301184136 shutdown_connections 2023-10-30T14:26:38.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.378+0000 7fd93aead640 1 -- 172.21.15.46:0/3301184136 >> 172.21.15.46:0/3301184136 conn(0x7fd93406a640 msgr2=0x7fd934065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:38.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.378+0000 7fd93aead640 1 -- 172.21.15.46:0/3301184136 shutdown_connections 2023-10-30T14:26:38.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.378+0000 7fd93aead640 1 -- 172.21.15.46:0/3301184136 wait complete. 2023-10-30T14:26:38.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd93aead640 1 Processor -- start 2023-10-30T14:26:38.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd93aead640 1 -- start start 2023-10-30T14:26:38.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd93aead640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd920003b40 0x7fd9341448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd93aead640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd934145c20 0x7fd934147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd93aead640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9340b43d0 0x7fd9340b4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd93aead640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9340b3980 con 0x7fd920003b40 2023-10-30T14:26:38.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd93aead640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9340b4dc0 con 0x7fd934145c20 2023-10-30T14:26:38.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd93aead640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd934144e20 con 0x7fd9340b43d0 2023-10-30T14:26:38.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd939eab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd920003b40 0x7fd9341448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd939eab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd920003b40 0x7fd9341448e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43558/0 (socket says 172.21.15.46:43558) 2023-10-30T14:26:38.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd939eab640 1 -- 172.21.15.46:0/849874921 learned_addr learned my addr 172.21.15.46:0/849874921 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:38.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd9396aa640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd934145c20 0x7fd934147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd93a6ac640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9340b43d0 0x7fd9340b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd939eab640 1 -- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9340b43d0 msgr2=0x7fd9340b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd939eab640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9340b43d0 0x7fd9340b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd939eab640 1 -- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd934145c20 msgr2=0x7fd934147ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd939eab640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd934145c20 0x7fd934147ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd939eab640 1 -- 172.21.15.46:0/849874921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd934143250 con 0x7fd920003b40 2023-10-30T14:26:38.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd9396aa640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd934145c20 0x7fd934147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:38.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd939eab640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd920003b40 0x7fd9341448e0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fd930009c80 tx=0x7fd920005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:38.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd92a7fc640 1 -- 172.21.15.46:0/849874921 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9300ad850 con 0x7fd920003b40 2023-10-30T14:26:38.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd92a7fc640 1 -- 172.21.15.46:0/849874921 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9300ad9b0 con 0x7fd920003b40 2023-10-30T14:26:38.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd93aead640 1 -- 172.21.15.46:0/849874921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd934143530 con 0x7fd920003b40 2023-10-30T14:26:38.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd92a7fc640 1 -- 172.21.15.46:0/849874921 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9300add60 con 0x7fd920003b40 2023-10-30T14:26:38.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd93aead640 1 -- 172.21.15.46:0/849874921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9340b3bd0 con 0x7fd920003b40 2023-10-30T14:26:38.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.382+0000 7fd93aead640 1 -- 172.21.15.46:0/849874921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93404e170 con 0x7fd920003b40 2023-10-30T14:26:38.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.386+0000 7fd92a7fc640 1 -- 172.21.15.46:0/849874921 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd9300b4040 con 0x7fd920003b40 2023-10-30T14:26:38.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.386+0000 7fd92a7fc640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd9180664a0 0x7fd918068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.386+0000 7fd92a7fc640 1 -- 172.21.15.46:0/849874921 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 8748+0+0 (secure 0 0 0) 0x7fd93011e900 con 0x7fd920003b40 2023-10-30T14:26:38.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.386+0000 7fd9396aa640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd9180664a0 0x7fd918068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.386+0000 7fd9396aa640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd9180664a0 0x7fd918068960 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd93415cad0 tx=0x7fd924009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:38.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.390+0000 7fd92a7fc640 1 -- 172.21.15.46:0/849874921 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd9300eb0e0 con 0x7fd920003b40 2023-10-30T14:26:38.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.546+0000 7fd93aead640 1 -- 172.21.15.46:0/849874921 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd934159a90 con 0x7fd9180664a0 2023-10-30T14:26:38.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.566+0000 7fd92a7fc640 1 -- 172.21.15.46:0/849874921 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294423 (secure 0 0 0) 0x7fd9240163f0 con 0x7fd9180664a0 2023-10-30T14:26:38.574 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:26:38.576 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T14:26:38.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.570+0000 7fd93aead640 1 -- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd9180664a0 msgr2=0x7fd918068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.570+0000 7fd93aead640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd9180664a0 0x7fd918068960 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd93415cad0 tx=0x7fd924009040 comp rx=0 tx=0).stop 2023-10-30T14:26:38.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.570+0000 7fd93aead640 1 -- 172.21.15.46:0/849874921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd920003b40 msgr2=0x7fd9341448e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.570+0000 7fd93aead640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd920003b40 0x7fd9341448e0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fd930009c80 tx=0x7fd920005800 comp rx=0 tx=0).stop 2023-10-30T14:26:38.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.570+0000 7fd93aead640 1 -- 172.21.15.46:0/849874921 shutdown_connections 2023-10-30T14:26:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.570+0000 7fd93aead640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fd9180664a0 0x7fd918068960 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.570+0000 7fd93aead640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9340b43d0 0x7fd9340b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.570+0000 7fd93aead640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd934145c20 0x7fd934147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.570+0000 7fd93aead640 1 --2- 172.21.15.46:0/849874921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd920003b40 0x7fd9341448e0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.570+0000 7fd93aead640 1 -- 172.21.15.46:0/849874921 >> 172.21.15.46:0/849874921 conn(0x7fd93406a640 msgr2=0x7fd934065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:38.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.570+0000 7fd93aead640 1 -- 172.21.15.46:0/849874921 shutdown_connections 2023-10-30T14:26:38.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.570+0000 7fd93aead640 1 -- 172.21.15.46:0/849874921 wait complete. 2023-10-30T14:26:38.595 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":2008,"stamp":"2023-10-30T14:26:38.127326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1284924434,"num_objects":3948,"num_object_clones":2,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":20157,"num_read_kb":14487318,"num_write":119381,"num_write_kb":3805271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170124,"ondisk_log_size":170124,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3927216,"kb_used_data":2520724,"kb_used_omap":12,"kb_used_meta":1406387,"kb_avail":751047504,"statfs":{"total":773094113280,"available":769072644096,"internally_reserved":0,"allocated":2581221376,"data_stored":2576042778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1440140856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2285484,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":93,"num_write_kb":2287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.234424"},"pg_stats":[{"pgid":"3.3f","version":"103'2152","reported_seq":4963,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.727766+0000","last_change":"2023-10-30T14:26:34.688745+0000","last_active":"2023-10-30T14:26:35.727766+0000","last_peered":"2023-10-30T14:26:35.727766+0000","last_clean":"2023-10-30T14:26:35.727766+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T14:26:35.727766+0000","last_undegraded":"2023-10-30T14:26:35.727766+0000","last_fullsized":"2023-10-30T14:26:35.727766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2152,"log_dups_size":0,"ondisk_log_size":2152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:50.061774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.84725695499999998,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":124,"num_read_kb":10662,"num_write":1276,"num_write_kb":15944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.3e","version":"103'1964","reported_seq":4841,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.720358+0000","last_change":"2023-10-30T14:26:33.665197+0000","last_active":"2023-10-30T14:26:35.720358+0000","last_peered":"2023-10-30T14:26:35.720358+0000","last_clean":"2023-10-30T14:26:35.720358+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T14:26:35.720358+0000","last_undegraded":"2023-10-30T14:26:35.720358+0000","last_fullsized":"2023-10-30T14:26:35.720358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1964,"log_dups_size":0,"ondisk_log_size":1964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:04:48.747855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.406256491,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":133,"num_read_kb":21158,"num_write":1230,"num_write_kb":16620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.3d","version":"103'2090","reported_seq":4921,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.721385+0000","last_change":"2023-10-30T14:26:34.547863+0000","last_active":"2023-10-30T14:26:35.721385+0000","last_peered":"2023-10-30T14:26:35.721385+0000","last_clean":"2023-10-30T14:26:35.721385+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T14:26:35.721385+0000","last_undegraded":"2023-10-30T14:26:35.721385+0000","last_fullsized":"2023-10-30T14:26:35.721385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:55:03.386638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.030559456,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":129,"num_read_kb":29825,"num_write":1270,"num_write_kb":28412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.3c","version":"103'2021","reported_seq":4859,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.894425+0000","last_change":"2023-10-30T14:26:33.286784+0000","last_active":"2023-10-30T14:26:34.894425+0000","last_peered":"2023-10-30T14:26:34.894425+0000","last_clean":"2023-10-30T14:26:34.894425+0000","last_became_active":"2023-10-30T13:24:00.704829+0000","last_became_peered":"2023-10-30T13:24:00.704829+0000","last_unstale":"2023-10-30T14:26:34.894425+0000","last_undegraded":"2023-10-30T14:26:34.894425+0000","last_fullsized":"2023-10-30T14:26:34.894425+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2021,"log_dups_size":0,"ondisk_log_size":2021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:28.307918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.361435862,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":124,"num_read_kb":7939,"num_write":1184,"num_write_kb":24239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.3b","version":"103'2001","reported_seq":4857,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723057+0000","last_change":"2023-10-30T14:26:34.648843+0000","last_active":"2023-10-30T14:26:35.723057+0000","last_peered":"2023-10-30T14:26:35.723057+0000","last_clean":"2023-10-30T14:26:35.723057+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T14:26:35.723057+0000","last_undegraded":"2023-10-30T14:26:35.723057+0000","last_fullsized":"2023-10-30T14:26:35.723057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2001,"log_dups_size":0,"ondisk_log_size":2001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:36.619771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.80853723300000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":41266,"num_write":1244,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.3a","version":"103'2205","reported_seq":5002,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724501+0000","last_change":"2023-10-30T14:26:33.897829+0000","last_active":"2023-10-30T14:26:35.724501+0000","last_peered":"2023-10-30T14:26:35.724501+0000","last_clean":"2023-10-30T14:26:35.724501+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T14:26:35.724501+0000","last_undegraded":"2023-10-30T14:26:35.724501+0000","last_fullsized":"2023-10-30T14:26:35.724501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2205,"log_dups_size":0,"ondisk_log_size":2205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:03.177100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.94867293500000005,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":141,"num_read_kb":27429,"num_write":1290,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.39","version":"103'2138","reported_seq":4985,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.722859+0000","last_change":"2023-10-30T14:26:34.555126+0000","last_active":"2023-10-30T14:26:35.722859+0000","last_peered":"2023-10-30T14:26:35.722859+0000","last_clean":"2023-10-30T14:26:35.722859+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T14:26:35.722859+0000","last_undegraded":"2023-10-30T14:26:35.722859+0000","last_fullsized":"2023-10-30T14:26:35.722859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2138,"log_dups_size":0,"ondisk_log_size":2138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:25.464527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.71511574600000005,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":146,"num_read_kb":25110,"num_write":1316,"num_write_kb":20043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.38","version":"103'2099","reported_seq":4909,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.297117+0000","last_change":"2023-10-30T14:26:36.369916+0000","last_active":"2023-10-30T14:26:36.297117+0000","last_peered":"2023-10-30T14:26:36.297117+0000","last_clean":"2023-10-30T14:26:36.297117+0000","last_became_active":"2023-10-30T13:24:00.989938+0000","last_became_peered":"2023-10-30T13:24:00.989938+0000","last_unstale":"2023-10-30T14:26:36.297117+0000","last_undegraded":"2023-10-30T14:26:36.297117+0000","last_fullsized":"2023-10-30T14:26:36.297117+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:03.578741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.90238370000000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":124,"num_read_kb":25628,"num_write":1232,"num_write_kb":20540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.37","version":"103'2126","reported_seq":4956,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.727278+0000","last_change":"2023-10-30T14:26:34.654560+0000","last_active":"2023-10-30T14:26:35.727278+0000","last_peered":"2023-10-30T14:26:35.727278+0000","last_clean":"2023-10-30T14:26:35.727278+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T14:26:35.727278+0000","last_undegraded":"2023-10-30T14:26:35.727278+0000","last_fullsized":"2023-10-30T14:26:35.727278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2126,"log_dups_size":0,"ondisk_log_size":2126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:56.758220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81280227199999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":24548,"num_write":1310,"num_write_kb":19821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.36","version":"103'2060","reported_seq":4921,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.899341+0000","last_change":"2023-10-30T14:26:33.465676+0000","last_active":"2023-10-30T14:26:34.899341+0000","last_peered":"2023-10-30T14:26:34.899341+0000","last_clean":"2023-10-30T14:26:34.899341+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T14:26:34.899341+0000","last_undegraded":"2023-10-30T14:26:34.899341+0000","last_fullsized":"2023-10-30T14:26:34.899341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2060,"log_dups_size":0,"ondisk_log_size":2060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:02.527162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.53668512800000001,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":148,"num_read_kb":64678,"num_write":1273,"num_write_kb":32495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.35","version":"105'1999","reported_seq":4872,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.899012+0000","last_change":"2023-10-30T14:26:33.399684+0000","last_active":"2023-10-30T14:26:34.899012+0000","last_peered":"2023-10-30T14:26:34.899012+0000","last_clean":"2023-10-30T14:26:34.899012+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T14:26:34.899012+0000","last_undegraded":"2023-10-30T14:26:34.899012+0000","last_fullsized":"2023-10-30T14:26:34.899012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1999,"log_dups_size":0,"ondisk_log_size":1999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:10:40.091601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.471346551,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":139,"num_read_kb":25550,"num_write":1254,"num_write_kb":16915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.34","version":"103'2073","reported_seq":4916,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.725794+0000","last_change":"2023-10-30T14:26:33.495643+0000","last_active":"2023-10-30T14:26:35.725794+0000","last_peered":"2023-10-30T14:26:35.725794+0000","last_clean":"2023-10-30T14:26:35.725794+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T14:26:35.725794+0000","last_undegraded":"2023-10-30T14:26:35.725794+0000","last_fullsized":"2023-10-30T14:26:35.725794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:07:16.016021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.54183594099999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":142,"num_read_kb":52725,"num_write":1240,"num_write_kb":42463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.33","version":"103'2219","reported_seq":5026,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.452367+0000","last_change":"2023-10-30T14:26:36.506702+0000","last_active":"2023-10-30T14:26:36.452367+0000","last_peered":"2023-10-30T14:26:36.452367+0000","last_clean":"2023-10-30T14:26:36.452367+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T14:26:36.452367+0000","last_undegraded":"2023-10-30T14:26:36.452367+0000","last_fullsized":"2023-10-30T14:26:36.452367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2219,"log_dups_size":0,"ondisk_log_size":2219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:07.521185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.038919041,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":46739,"num_write":1326,"num_write_kb":33562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.32","version":"103'2145","reported_seq":4993,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.721177+0000","last_change":"2023-10-30T14:26:34.440977+0000","last_active":"2023-10-30T14:26:35.721177+0000","last_peered":"2023-10-30T14:26:35.721177+0000","last_clean":"2023-10-30T14:26:35.721177+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T14:26:35.721177+0000","last_undegraded":"2023-10-30T14:26:35.721177+0000","last_fullsized":"2023-10-30T14:26:35.721177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2145,"log_dups_size":0,"ondisk_log_size":2145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:09:13.743893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.97549857799999995,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":149,"num_read_kb":13007,"num_write":1312,"num_write_kb":16233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.31","version":"103'2258","reported_seq":5081,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.718848+0000","last_change":"2023-10-30T14:26:33.691289+0000","last_active":"2023-10-30T14:26:35.718848+0000","last_peered":"2023-10-30T14:26:35.718848+0000","last_clean":"2023-10-30T14:26:35.718848+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T14:26:35.718848+0000","last_undegraded":"2023-10-30T14:26:35.718848+0000","last_fullsized":"2023-10-30T14:26:35.718848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2258,"log_dups_size":0,"ondisk_log_size":2258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:20:29.170495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.432871758,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":161,"num_read_kb":14756,"num_write":1377,"num_write_kb":22909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.30","version":"103'2085","reported_seq":4943,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723221+0000","last_change":"2023-10-30T14:26:33.787493+0000","last_active":"2023-10-30T14:26:35.723221+0000","last_peered":"2023-10-30T14:26:35.723221+0000","last_clean":"2023-10-30T14:26:35.723221+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T14:26:35.723221+0000","last_undegraded":"2023-10-30T14:26:35.723221+0000","last_fullsized":"2023-10-30T14:26:35.723221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2085,"log_dups_size":0,"ondisk_log_size":2085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:57.806855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.83845289999999995,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":149,"num_read_kb":40211,"num_write":1268,"num_write_kb":23485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2f","version":"104'2338","reported_seq":5151,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724465+0000","last_change":"2023-10-30T14:26:34.060688+0000","last_active":"2023-10-30T14:26:35.724465+0000","last_peered":"2023-10-30T14:26:35.724465+0000","last_clean":"2023-10-30T14:26:35.724465+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T14:26:35.724465+0000","last_undegraded":"2023-10-30T14:26:35.724465+0000","last_fullsized":"2023-10-30T14:26:35.724465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2338,"log_dups_size":0,"ondisk_log_size":2338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:47:21.386319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.109948846,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":170,"num_read_kb":39932,"num_write":1446,"num_write_kb":30153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2e","version":"103'1987","reported_seq":4842,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.895081+0000","last_change":"2023-10-30T14:26:33.267575+0000","last_active":"2023-10-30T14:26:34.895081+0000","last_peered":"2023-10-30T14:26:34.895081+0000","last_clean":"2023-10-30T14:26:34.895081+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T14:26:34.895081+0000","last_undegraded":"2023-10-30T14:26:34.895081+0000","last_fullsized":"2023-10-30T14:26:34.895081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1987,"log_dups_size":0,"ondisk_log_size":1987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:03.867577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.34246037800000001,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":137,"num_read_kb":13706,"num_write":1196,"num_write_kb":12790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2d","version":"103'2087","reported_seq":4919,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.725816+0000","last_change":"2023-10-30T14:26:34.604839+0000","last_active":"2023-10-30T14:26:35.725816+0000","last_peered":"2023-10-30T14:26:35.725816+0000","last_clean":"2023-10-30T14:26:35.725816+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T14:26:35.725816+0000","last_undegraded":"2023-10-30T14:26:35.725816+0000","last_fullsized":"2023-10-30T14:26:35.725816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2087,"log_dups_size":0,"ondisk_log_size":2087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:35:33.855326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.76369308499999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":125,"num_read_kb":30178,"num_write":1282,"num_write_kb":24413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2c","version":"103'2210","reported_seq":5039,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.899583+0000","last_change":"2023-10-30T14:26:33.395186+0000","last_active":"2023-10-30T14:26:34.899583+0000","last_peered":"2023-10-30T14:26:34.899583+0000","last_clean":"2023-10-30T14:26:34.899583+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T14:26:34.899583+0000","last_undegraded":"2023-10-30T14:26:34.899583+0000","last_fullsized":"2023-10-30T14:26:34.899583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:31.161754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.46668427699999998,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":148,"num_read_kb":30111,"num_write":1332,"num_write_kb":26597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2b","version":"103'2218","reported_seq":5006,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.722043+0000","last_change":"2023-10-30T14:26:34.474353+0000","last_active":"2023-10-30T14:26:35.722043+0000","last_peered":"2023-10-30T14:26:35.722043+0000","last_clean":"2023-10-30T14:26:35.722043+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T14:26:35.722043+0000","last_undegraded":"2023-10-30T14:26:35.722043+0000","last_fullsized":"2023-10-30T14:26:35.722043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2218,"log_dups_size":0,"ondisk_log_size":2218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:01.333450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.008362585,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":127,"num_read_kb":10449,"num_write":1298,"num_write_kb":11653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.2a","version":"103'2269","reported_seq":5069,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724749+0000","last_change":"2023-10-30T14:26:33.893164+0000","last_active":"2023-10-30T14:26:35.724749+0000","last_peered":"2023-10-30T14:26:35.724749+0000","last_clean":"2023-10-30T14:26:35.724749+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T14:26:35.724749+0000","last_undegraded":"2023-10-30T14:26:35.724749+0000","last_fullsized":"2023-10-30T14:26:35.724749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2269,"log_dups_size":0,"ondisk_log_size":2269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:42.538423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.94381626399999996,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":144,"num_read_kb":30506,"num_write":1376,"num_write_kb":26822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.29","version":"108'2111","reported_seq":5073,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.405991+0000","last_change":"2023-10-30T14:26:36.452306+0000","last_active":"2023-10-30T14:26:36.405991+0000","last_peered":"2023-10-30T14:26:36.405991+0000","last_clean":"2023-10-30T14:26:36.405991+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T14:26:36.405991+0000","last_undegraded":"2023-10-30T14:26:36.405991+0000","last_fullsized":"2023-10-30T14:26:36.405991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2111,"log_dups_size":0,"ondisk_log_size":2111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:47.201749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98440143999999996,"stat_sum":{"num_bytes":6,"num_objects":38,"num_object_clones":2,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":257,"num_read_kb":42146,"num_write":1297,"num_write_kb":25266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.28","version":"103'2167","reported_seq":5028,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724912+0000","last_change":"2023-10-30T14:26:33.846175+0000","last_active":"2023-10-30T14:26:35.724912+0000","last_peered":"2023-10-30T14:26:35.724912+0000","last_clean":"2023-10-30T14:26:35.724912+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T14:26:35.724912+0000","last_undegraded":"2023-10-30T14:26:35.724912+0000","last_fullsized":"2023-10-30T14:26:35.724912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2167,"log_dups_size":0,"ondisk_log_size":2167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:34:46.488680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89723890399999995,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":164,"num_read_kb":14461,"num_write":1342,"num_write_kb":12663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.27","version":"103'2051","reported_seq":4890,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.472159+0000","last_change":"2023-10-30T14:26:36.518474+0000","last_active":"2023-10-30T14:26:36.472159+0000","last_peered":"2023-10-30T14:26:36.472159+0000","last_clean":"2023-10-30T14:26:36.472159+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T14:26:36.472159+0000","last_undegraded":"2023-10-30T14:26:36.472159+0000","last_fullsized":"2023-10-30T14:26:36.472159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2051,"log_dups_size":0,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:14.252516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0495642949999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":133,"num_read_kb":25337,"num_write":1220,"num_write_kb":15637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.26","version":"103'2135","reported_seq":5011,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.899304+0000","last_change":"2023-10-30T14:26:33.433647+0000","last_active":"2023-10-30T14:26:34.899304+0000","last_peered":"2023-10-30T14:26:34.899304+0000","last_clean":"2023-10-30T14:26:34.899304+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T14:26:34.899304+0000","last_undegraded":"2023-10-30T14:26:34.899304+0000","last_fullsized":"2023-10-30T14:26:34.899304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:13:23.542810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.504458776,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":160,"num_read_kb":14034,"num_write":1340,"num_write_kb":12359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.25","version":"103'2083","reported_seq":4931,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.719163+0000","last_change":"2023-10-30T14:26:33.601252+0000","last_active":"2023-10-30T14:26:35.719163+0000","last_peered":"2023-10-30T14:26:35.719163+0000","last_clean":"2023-10-30T14:26:35.719163+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T14:26:35.719163+0000","last_undegraded":"2023-10-30T14:26:35.719163+0000","last_fullsized":"2023-10-30T14:26:35.719163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:23:40.438800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.34262146900000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":128,"num_read_kb":22518,"num_write":1277,"num_write_kb":32107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.24","version":"103'2025","reported_seq":4853,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.506833+0000","last_change":"2023-10-30T14:26:36.555781+0000","last_active":"2023-10-30T14:26:36.506833+0000","last_peered":"2023-10-30T14:26:36.506833+0000","last_clean":"2023-10-30T14:26:36.506833+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T14:26:36.506833+0000","last_undegraded":"2023-10-30T14:26:36.506833+0000","last_fullsized":"2023-10-30T14:26:36.506833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2025,"log_dups_size":0,"ondisk_log_size":2025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:34.247640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.068284585,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":129,"num_read_kb":17081,"num_write":1202,"num_write_kb":15468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.23","version":"103'2131","reported_seq":4952,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.722225+0000","last_change":"2023-10-30T14:26:34.496718+0000","last_active":"2023-10-30T14:26:35.722225+0000","last_peered":"2023-10-30T14:26:35.722225+0000","last_clean":"2023-10-30T14:26:35.722225+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T14:26:35.722225+0000","last_undegraded":"2023-10-30T14:26:35.722225+0000","last_fullsized":"2023-10-30T14:26:35.722225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2131,"log_dups_size":0,"ondisk_log_size":2131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:53.519748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.030536272,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":125,"num_read_kb":13655,"num_write":1323,"num_write_kb":17357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.22","version":"103'2073","reported_seq":4941,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.727070+0000","last_change":"2023-10-30T14:26:33.934508+0000","last_active":"2023-10-30T14:26:35.727070+0000","last_peered":"2023-10-30T14:26:35.727070+0000","last_clean":"2023-10-30T14:26:35.727070+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T14:26:35.727070+0000","last_undegraded":"2023-10-30T14:26:35.727070+0000","last_fullsized":"2023-10-30T14:26:35.727070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:25:07.586673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98488935700000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":140,"num_read_kb":17880,"num_write":1311,"num_write_kb":13902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.21","version":"108'2369","reported_seq":5183,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.894065+0000","last_change":"2023-10-30T14:26:33.231592+0000","last_active":"2023-10-30T14:26:34.894065+0000","last_peered":"2023-10-30T14:26:34.894065+0000","last_clean":"2023-10-30T14:26:34.894065+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T14:26:34.894065+0000","last_undegraded":"2023-10-30T14:26:34.894065+0000","last_fullsized":"2023-10-30T14:26:34.894065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2369,"log_dups_size":0,"ondisk_log_size":2369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:00:56.118846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30670995400000001,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":165,"num_read_kb":40193,"num_write":1443,"num_write_kb":35290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.20","version":"108'2297","reported_seq":5117,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.898291+0000","last_change":"2023-10-30T14:26:33.357836+0000","last_active":"2023-10-30T14:26:34.898291+0000","last_peered":"2023-10-30T14:26:34.898291+0000","last_clean":"2023-10-30T14:26:34.898291+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T14:26:34.898291+0000","last_undegraded":"2023-10-30T14:26:34.898291+0000","last_fullsized":"2023-10-30T14:26:34.898291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2297,"log_dups_size":0,"ondisk_log_size":2297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:53:07.770089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42993166100000002,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":168,"num_read_kb":24860,"num_write":1383,"num_write_kb":20782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.1a","version":"103'2103","reported_seq":4944,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724893+0000","last_change":"2023-10-30T14:26:33.546412+0000","last_active":"2023-10-30T14:26:35.724893+0000","last_peered":"2023-10-30T14:26:35.724893+0000","last_clean":"2023-10-30T14:26:35.724893+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T14:26:35.724893+0000","last_undegraded":"2023-10-30T14:26:35.724893+0000","last_fullsized":"2023-10-30T14:26:35.724893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:27.773478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.59246732999999996,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":146,"num_read_kb":30802,"num_write":1268,"num_write_kb":21419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1b","version":"103'1028","reported_seq":1334,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.328825+0000","last_change":"2023-10-30T13:23:18.963933+0000","last_active":"2023-10-30T14:26:33.328825+0000","last_peered":"2023-10-30T14:26:33.328825+0000","last_clean":"2023-10-30T14:26:33.328825+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T14:26:33.328825+0000","last_undegraded":"2023-10-30T14:26:33.328825+0000","last_fullsized":"2023-10-30T14:26:33.328825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1028,"log_dups_size":0,"ondisk_log_size":1028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:48:58.196567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497506,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":222,"num_read_kb":501605,"num_write":1094,"num_write_kb":95785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"103'2148","reported_seq":4953,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.722408+0000","last_change":"2023-10-30T14:26:34.609800+0000","last_active":"2023-10-30T14:26:35.722408+0000","last_peered":"2023-10-30T14:26:35.722408+0000","last_clean":"2023-10-30T14:26:35.722408+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T14:26:35.722408+0000","last_undegraded":"2023-10-30T14:26:35.722408+0000","last_fullsized":"2023-10-30T14:26:35.722408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:24:55.845740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.1434485830000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":17524,"num_write":1294,"num_write_kb":15816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1a","version":"108'1693","reported_seq":2126,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.988999+0000","last_change":"2023-10-30T13:23:18.976615+0000","last_active":"2023-10-30T14:26:33.988999+0000","last_peered":"2023-10-30T14:26:33.988999+0000","last_clean":"2023-10-30T14:26:33.988999+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T14:26:33.988999+0000","last_undegraded":"2023-10-30T14:26:33.988999+0000","last_fullsized":"2023-10-30T14:26:33.988999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1693,"log_dups_size":0,"ondisk_log_size":1693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:59.048396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29694842,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":392,"num_read_kb":410068,"num_write":1792,"num_write_kb":89526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"104'1557","reported_seq":2910,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.394516+0000","last_change":"2023-10-30T13:23:18.977357+0000","last_active":"2023-10-30T14:26:33.394516+0000","last_peered":"2023-10-30T14:26:33.394516+0000","last_clean":"2023-10-30T14:26:33.394516+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T14:26:33.394516+0000","last_undegraded":"2023-10-30T14:26:33.394516+0000","last_fullsized":"2023-10-30T14:26:33.394516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1557,"log_dups_size":0,"ondisk_log_size":1557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:53.502681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60855928,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":1296,"num_read_kb":556671,"num_write":1596,"num_write_kb":94675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"103'2314","reported_seq":5149,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.308697+0000","last_change":"2023-10-30T14:26:36.357910+0000","last_active":"2023-10-30T14:26:36.308697+0000","last_peered":"2023-10-30T14:26:36.308697+0000","last_clean":"2023-10-30T14:26:36.308697+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T14:26:36.308697+0000","last_undegraded":"2023-10-30T14:26:36.308697+0000","last_fullsized":"2023-10-30T14:26:36.308697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2314,"log_dups_size":0,"ondisk_log_size":2314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:17.422938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89032229399999996,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":161,"num_read_kb":17698,"num_write":1449,"num_write_kb":26122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.18","version":"108'961","reported_seq":1307,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.770599+0000","last_change":"2023-10-30T13:23:18.989299+0000","last_active":"2023-10-30T14:26:33.770599+0000","last_peered":"2023-10-30T14:26:33.770599+0000","last_clean":"2023-10-30T14:26:33.770599+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T14:26:33.770599+0000","last_undegraded":"2023-10-30T14:26:33.770599+0000","last_fullsized":"2023-10-30T14:26:33.770599+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:44.510184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53243017,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":312,"num_read_kb":563254,"num_write":1067,"num_write_kb":95660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"108'2143","reported_seq":4999,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.718793+0000","last_change":"2023-10-30T14:26:33.629921+0000","last_active":"2023-10-30T14:26:35.718793+0000","last_peered":"2023-10-30T14:26:35.718793+0000","last_clean":"2023-10-30T14:26:35.718793+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T14:26:35.718793+0000","last_undegraded":"2023-10-30T14:26:35.718793+0000","last_fullsized":"2023-10-30T14:26:35.718793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2143,"log_dups_size":0,"ondisk_log_size":2143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:40:42.318115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.36920391600000002,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":152,"num_read_kb":51949,"num_write":1321,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.17","version":"103'801","reported_seq":1133,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.928642+0000","last_change":"2023-10-30T13:23:18.989515+0000","last_active":"2023-10-30T14:26:32.928642+0000","last_peered":"2023-10-30T14:26:32.928642+0000","last_clean":"2023-10-30T14:26:32.928642+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T14:26:32.928642+0000","last_undegraded":"2023-10-30T14:26:32.928642+0000","last_fullsized":"2023-10-30T14:26:32.928642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":801,"log_dups_size":0,"ondisk_log_size":801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:35.133762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943130,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":319,"num_read_kb":417394,"num_write":916,"num_write_kb":71613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"103'2444","reported_seq":5172,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724775+0000","last_change":"2023-10-30T14:26:33.980471+0000","last_active":"2023-10-30T14:26:35.724775+0000","last_peered":"2023-10-30T14:26:35.724775+0000","last_clean":"2023-10-30T14:26:35.724775+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T14:26:35.724775+0000","last_undegraded":"2023-10-30T14:26:35.724775+0000","last_fullsized":"2023-10-30T14:26:35.724775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2444,"log_dups_size":0,"ondisk_log_size":2444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:40.292098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0302270280000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":152,"num_read_kb":10055,"num_write":1351,"num_write_kb":13264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.16","version":"103'849","reported_seq":1153,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.950393+0000","last_change":"2023-10-30T13:23:18.991454+0000","last_active":"2023-10-30T14:26:32.950393+0000","last_peered":"2023-10-30T14:26:32.950393+0000","last_clean":"2023-10-30T14:26:32.950393+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T14:26:32.950393+0000","last_undegraded":"2023-10-30T14:26:32.950393+0000","last_fullsized":"2023-10-30T14:26:32.950393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:13.450999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":291,"num_read_kb":402257,"num_write":947,"num_write_kb":71494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"103'2079","reported_seq":4921,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.721729+0000","last_change":"2023-10-30T14:26:34.598085+0000","last_active":"2023-10-30T14:26:35.721729+0000","last_peered":"2023-10-30T14:26:35.721729+0000","last_clean":"2023-10-30T14:26:35.721729+0000","last_became_active":"2023-10-30T13:24:00.990019+0000","last_became_peered":"2023-10-30T13:24:00.990019+0000","last_unstale":"2023-10-30T14:26:35.721729+0000","last_undegraded":"2023-10-30T14:26:35.721729+0000","last_fullsized":"2023-10-30T14:26:35.721729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2079,"log_dups_size":0,"ondisk_log_size":2079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:06:56.464105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.1322467139999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":132,"num_read_kb":30385,"num_write":1263,"num_write_kb":18224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.15","version":"108'995","reported_seq":1254,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.022661+0000","last_change":"2023-10-30T13:23:18.982337+0000","last_active":"2023-10-30T14:26:34.022661+0000","last_peered":"2023-10-30T14:26:34.022661+0000","last_clean":"2023-10-30T14:26:34.022661+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T14:26:34.022661+0000","last_undegraded":"2023-10-30T14:26:34.022661+0000","last_fullsized":"2023-10-30T14:26:34.022661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":995,"log_dups_size":0,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:09:39.996740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":208,"num_read_kb":224450,"num_write":1057,"num_write_kb":46972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"103'2138","reported_seq":4966,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.725948+0000","last_change":"2023-10-30T14:26:33.534876+0000","last_active":"2023-10-30T14:26:35.725948+0000","last_peered":"2023-10-30T14:26:35.725948+0000","last_clean":"2023-10-30T14:26:35.725948+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T14:26:35.725948+0000","last_undegraded":"2023-10-30T14:26:35.725948+0000","last_fullsized":"2023-10-30T14:26:35.725948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2138,"log_dups_size":0,"ondisk_log_size":2138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:17.987617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.58089010799999996,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":141,"num_read_kb":34327,"num_write":1294,"num_write_kb":27493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.14","version":"107'994","reported_seq":1307,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.928561+0000","last_change":"2023-10-30T13:23:18.969882+0000","last_active":"2023-10-30T14:26:32.928561+0000","last_peered":"2023-10-30T14:26:32.928561+0000","last_clean":"2023-10-30T14:26:32.928561+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T14:26:32.928561+0000","last_undegraded":"2023-10-30T14:26:32.928561+0000","last_fullsized":"2023-10-30T14:26:32.928561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":994,"log_dups_size":0,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:17.093448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":290,"num_read_kb":355466,"num_write":1083,"num_write_kb":67490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"108'2162","reported_seq":4979,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723783+0000","last_change":"2023-10-30T14:26:34.585946+0000","last_active":"2023-10-30T14:26:35.723783+0000","last_peered":"2023-10-30T14:26:35.723783+0000","last_clean":"2023-10-30T14:26:35.723783+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T14:26:35.723783+0000","last_undegraded":"2023-10-30T14:26:35.723783+0000","last_fullsized":"2023-10-30T14:26:35.723783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2162,"log_dups_size":0,"ondisk_log_size":2162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:03:21.700327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.74553484199999998,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":143,"num_read_kb":29369,"num_write":1268,"num_write_kb":21738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.13","version":"103'701","reported_seq":1069,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.466470+0000","last_change":"2023-10-30T13:23:18.971632+0000","last_active":"2023-10-30T14:26:33.466470+0000","last_peered":"2023-10-30T14:26:33.466470+0000","last_clean":"2023-10-30T14:26:33.466470+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T14:26:33.466470+0000","last_undegraded":"2023-10-30T14:26:33.466470+0000","last_fullsized":"2023-10-30T14:26:33.466470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:54.428499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331682,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":392,"num_read_kb":480410,"num_write":807,"num_write_kb":87841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"103'2142","reported_seq":4998,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.721389+0000","last_change":"2023-10-30T14:26:34.422972+0000","last_active":"2023-10-30T14:26:35.721389+0000","last_peered":"2023-10-30T14:26:35.721389+0000","last_clean":"2023-10-30T14:26:35.721389+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T14:26:35.721389+0000","last_undegraded":"2023-10-30T14:26:35.721389+0000","last_fullsized":"2023-10-30T14:26:35.721389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:47:44.081448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.95729191499999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":151,"num_read_kb":31378,"num_write":1306,"num_write_kb":33173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.12","version":"103'851","reported_seq":1110,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.949624+0000","last_change":"2023-10-30T13:23:18.978420+0000","last_active":"2023-10-30T14:26:32.949624+0000","last_peered":"2023-10-30T14:26:32.949624+0000","last_clean":"2023-10-30T14:26:32.949624+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T14:26:32.949624+0000","last_undegraded":"2023-10-30T14:26:32.949624+0000","last_fullsized":"2023-10-30T14:26:32.949624+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:10:44.511282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53927368,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":208,"num_read_kb":306859,"num_write":914,"num_write_kb":62457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"103'2282","reported_seq":5070,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.894056+0000","last_change":"2023-10-30T14:26:33.299265+0000","last_active":"2023-10-30T14:26:34.894056+0000","last_peered":"2023-10-30T14:26:34.894056+0000","last_clean":"2023-10-30T14:26:34.894056+0000","last_became_active":"2023-10-30T13:24:00.706894+0000","last_became_peered":"2023-10-30T13:24:00.706894+0000","last_unstale":"2023-10-30T14:26:34.894056+0000","last_undegraded":"2023-10-30T14:26:34.894056+0000","last_fullsized":"2023-10-30T14:26:34.894056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2282,"log_dups_size":0,"ondisk_log_size":2282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:58:25.208615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.3738494,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":149,"num_read_kb":1539,"num_write":1375,"num_write_kb":12931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.11","version":"103'727","reported_seq":1001,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.927895+0000","last_change":"2023-10-30T13:23:18.972707+0000","last_active":"2023-10-30T14:26:32.927895+0000","last_peered":"2023-10-30T14:26:32.927895+0000","last_clean":"2023-10-30T14:26:32.927895+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T14:26:32.927895+0000","last_undegraded":"2023-10-30T14:26:32.927895+0000","last_fullsized":"2023-10-30T14:26:32.927895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":727,"log_dups_size":0,"ondisk_log_size":727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":234,"num_read_kb":298146,"num_write":829,"num_write_kb":54985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"103'2091","reported_seq":4959,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.899515+0000","last_change":"2023-10-30T14:26:33.432572+0000","last_active":"2023-10-30T14:26:34.899515+0000","last_peered":"2023-10-30T14:26:34.899515+0000","last_clean":"2023-10-30T14:26:34.899515+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T14:26:34.899515+0000","last_undegraded":"2023-10-30T14:26:34.899515+0000","last_fullsized":"2023-10-30T14:26:34.899515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2091,"log_dups_size":0,"ondisk_log_size":2091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:47:50.528827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50412874500000004,"stat_sum":{"num_bytes":0,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":148,"num_read_kb":43506,"num_write":1375,"num_write_kb":26417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.10","version":"103'825","reported_seq":1169,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.924671+0000","last_change":"2023-10-30T13:23:18.948296+0000","last_active":"2023-10-30T14:26:32.924671+0000","last_peered":"2023-10-30T14:26:32.924671+0000","last_clean":"2023-10-30T14:26:32.924671+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T14:26:32.924671+0000","last_undegraded":"2023-10-30T14:26:32.924671+0000","last_fullsized":"2023-10-30T14:26:32.924671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:55.308329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":481,"num_read_kb":229594,"num_write":1003,"num_write_kb":42855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"108'2330","reported_seq":5156,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723033+0000","last_change":"2023-10-30T14:26:34.549537+0000","last_active":"2023-10-30T14:26:35.723033+0000","last_peered":"2023-10-30T14:26:35.723033+0000","last_clean":"2023-10-30T14:26:35.723033+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T14:26:35.723033+0000","last_undegraded":"2023-10-30T14:26:35.723033+0000","last_fullsized":"2023-10-30T14:26:35.723033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2330,"log_dups_size":0,"ondisk_log_size":2330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:07.630163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70955538100000004,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":166,"num_read_kb":25446,"num_write":1448,"num_write_kb":32435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.f","version":"103'995","reported_seq":1278,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.725769+0000","last_change":"2023-10-30T13:23:18.962343+0000","last_active":"2023-10-30T14:26:35.725769+0000","last_peered":"2023-10-30T14:26:35.725769+0000","last_clean":"2023-10-30T14:26:35.725769+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T14:26:35.725769+0000","last_undegraded":"2023-10-30T14:26:35.725769+0000","last_fullsized":"2023-10-30T14:26:35.725769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":995,"log_dups_size":0,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:36.082164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":214,"num_read_kb":373366,"num_write":1150,"num_write_kb":71086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"103'2232","reported_seq":5019,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.726899+0000","last_change":"2023-10-30T14:26:34.736009+0000","last_active":"2023-10-30T14:26:35.726899+0000","last_peered":"2023-10-30T14:26:35.726899+0000","last_clean":"2023-10-30T14:26:35.726899+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T14:26:35.726899+0000","last_undegraded":"2023-10-30T14:26:35.726899+0000","last_fullsized":"2023-10-30T14:26:35.726899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2232,"log_dups_size":0,"ondisk_log_size":2232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:19.838741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89413299700000004,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":142,"num_read_kb":18968,"num_write":1278,"num_write_kb":24747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.e","version":"105'626","reported_seq":936,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.925265+0000","last_change":"2023-10-30T13:23:18.949081+0000","last_active":"2023-10-30T14:26:32.925265+0000","last_peered":"2023-10-30T14:26:32.925265+0000","last_clean":"2023-10-30T14:26:32.925265+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T14:26:32.925265+0000","last_undegraded":"2023-10-30T14:26:32.925265+0000","last_fullsized":"2023-10-30T14:26:32.925265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:00.356885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":280,"num_read_kb":492842,"num_write":679,"num_write_kb":83547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"103'2085","reported_seq":4957,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.727597+0000","last_change":"2023-10-30T14:26:34.726845+0000","last_active":"2023-10-30T14:26:35.727597+0000","last_peered":"2023-10-30T14:26:35.727597+0000","last_clean":"2023-10-30T14:26:35.727597+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T14:26:35.727597+0000","last_undegraded":"2023-10-30T14:26:35.727597+0000","last_fullsized":"2023-10-30T14:26:35.727597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2085,"log_dups_size":0,"ondisk_log_size":2085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:53.224813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.88483857600000004,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":152,"num_read_kb":61046,"num_write":1310,"num_write_kb":36653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.d","version":"103'731","reported_seq":1024,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.258765+0000","last_change":"2023-10-30T13:23:18.970780+0000","last_active":"2023-10-30T14:26:33.258765+0000","last_peered":"2023-10-30T14:26:33.258765+0000","last_clean":"2023-10-30T14:26:33.258765+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T14:26:33.258765+0000","last_undegraded":"2023-10-30T14:26:33.258765+0000","last_fullsized":"2023-10-30T14:26:33.258765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:38:51.113629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364699,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":233,"num_read_kb":370768,"num_write":815,"num_write_kb":71238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"103'2230","reported_seq":5089,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723892+0000","last_change":"2023-10-30T14:26:33.934502+0000","last_active":"2023-10-30T14:26:35.723892+0000","last_peered":"2023-10-30T14:26:35.723892+0000","last_clean":"2023-10-30T14:26:35.723892+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T14:26:35.723892+0000","last_undegraded":"2023-10-30T14:26:35.723892+0000","last_fullsized":"2023-10-30T14:26:35.723892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2230,"log_dups_size":0,"ondisk_log_size":2230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:46.397667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98441546899999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":178,"num_read_kb":29706,"num_write":1375,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.c","version":"103'1078","reported_seq":1359,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.925585+0000","last_change":"2023-10-30T13:23:18.977920+0000","last_active":"2023-10-30T14:26:32.925585+0000","last_peered":"2023-10-30T14:26:32.925585+0000","last_clean":"2023-10-30T14:26:32.925585+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T14:26:32.925585+0000","last_undegraded":"2023-10-30T14:26:32.925585+0000","last_fullsized":"2023-10-30T14:26:32.925585+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:07:59.139948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50332230,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":210,"num_read_kb":406971,"num_write":1089,"num_write_kb":71553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"103'2029","reported_seq":4878,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.725143+0000","last_change":"2023-10-30T14:26:34.688764+0000","last_active":"2023-10-30T14:26:35.725143+0000","last_peered":"2023-10-30T14:26:35.725143+0000","last_clean":"2023-10-30T14:26:35.725143+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T14:26:35.725143+0000","last_undegraded":"2023-10-30T14:26:35.725143+0000","last_fullsized":"2023-10-30T14:26:35.725143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:07.116174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.84782311300000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":122,"num_read_kb":1096,"num_write":1268,"num_write_kb":17447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"1.0","version":"16'117","reported_seq":285,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.839992+0000","last_change":"2023-10-30T13:23:08.208031+0000","last_active":"2023-10-30T14:26:33.839992+0000","last_peered":"2023-10-30T14:26:33.839992+0000","last_clean":"2023-10-30T14:26:33.839992+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T14:26:33.839992+0000","last_undegraded":"2023-10-30T14:26:33.839992+0000","last_fullsized":"2023-10-30T14:26:33.839992+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_clean_scrub_stamp":"2023-10-30T13:23:07.179866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:22.691367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"108'849","reported_seq":1186,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.991014+0000","last_change":"2023-10-30T13:23:18.979328+0000","last_active":"2023-10-30T14:26:33.991014+0000","last_peered":"2023-10-30T14:26:33.991014+0000","last_clean":"2023-10-30T14:26:33.991014+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T14:26:33.991014+0000","last_undegraded":"2023-10-30T14:26:33.991014+0000","last_fullsized":"2023-10-30T14:26:33.991014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:04.942593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331714,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":319,"num_read_kb":439640,"num_write":936,"num_write_kb":75464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"103'2192","reported_seq":4977,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.417782+0000","last_change":"2023-10-30T14:26:36.472084+0000","last_active":"2023-10-30T14:26:36.417782+0000","last_peered":"2023-10-30T14:26:36.417782+0000","last_clean":"2023-10-30T14:26:36.417782+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T14:26:36.417782+0000","last_undegraded":"2023-10-30T14:26:36.417782+0000","last_fullsized":"2023-10-30T14:26:36.417782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2192,"log_dups_size":0,"ondisk_log_size":2192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:48:10.995137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.003812562,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":127,"num_read_kb":26261,"num_write":1294,"num_write_kb":24882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.0","version":"103'1186","reported_seq":1498,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.839806+0000","last_change":"2023-10-30T13:23:18.985318+0000","last_active":"2023-10-30T14:26:33.839806+0000","last_peered":"2023-10-30T14:26:33.839806+0000","last_clean":"2023-10-30T14:26:33.839806+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T14:26:33.839806+0000","last_undegraded":"2023-10-30T14:26:33.839806+0000","last_fullsized":"2023-10-30T14:26:33.839806+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1186,"log_dups_size":0,"ondisk_log_size":1186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:16.754084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":256,"num_read_kb":488007,"num_write":1246,"num_write_kb":83351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"103'1973","reported_seq":4835,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.718436+0000","last_change":"2023-10-30T14:26:34.386961+0000","last_active":"2023-10-30T14:26:35.718436+0000","last_peered":"2023-10-30T14:26:35.718436+0000","last_clean":"2023-10-30T14:26:35.718436+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T14:26:35.718436+0000","last_undegraded":"2023-10-30T14:26:35.718436+0000","last_fullsized":"2023-10-30T14:26:35.718436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1973,"log_dups_size":0,"ondisk_log_size":1973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:03.616902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.92159720499999997,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":129,"num_read_kb":13045,"num_write":1222,"num_write_kb":11637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1","version":"108'1950","reported_seq":3194,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.981824+0000","last_change":"2023-10-30T13:23:18.949565+0000","last_active":"2023-10-30T14:26:33.981824+0000","last_peered":"2023-10-30T14:26:33.981824+0000","last_clean":"2023-10-30T14:26:33.981824+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T14:26:33.981824+0000","last_undegraded":"2023-10-30T14:26:33.981824+0000","last_fullsized":"2023-10-30T14:26:33.981824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1950,"log_dups_size":0,"ondisk_log_size":1950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:55:46.029368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40060913,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":1167,"num_read_kb":359975,"num_write":2020,"num_write_kb":69890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"103'2255","reported_seq":5066,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.718578+0000","last_change":"2023-10-30T14:26:33.665105+0000","last_active":"2023-10-30T14:26:35.718578+0000","last_peered":"2023-10-30T14:26:35.718578+0000","last_clean":"2023-10-30T14:26:35.718578+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T14:26:35.718578+0000","last_undegraded":"2023-10-30T14:26:35.718578+0000","last_fullsized":"2023-10-30T14:26:35.718578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2255,"log_dups_size":0,"ondisk_log_size":2255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:37:34.538070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40657480699999998,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":145,"num_read_kb":55105,"num_write":1354,"num_write_kb":28005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.2","version":"103'1488","reported_seq":1793,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.949707+0000","last_change":"2023-10-30T13:23:18.984535+0000","last_active":"2023-10-30T14:26:32.949707+0000","last_peered":"2023-10-30T14:26:32.949707+0000","last_clean":"2023-10-30T14:26:32.949707+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T14:26:32.949707+0000","last_undegraded":"2023-10-30T14:26:32.949707+0000","last_fullsized":"2023-10-30T14:26:32.949707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1488,"log_dups_size":0,"ondisk_log_size":1488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:28.405405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":235,"num_read_kb":509467,"num_write":1537,"num_write_kb":83943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"103'2200","reported_seq":5023,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724420+0000","last_change":"2023-10-30T14:26:33.550991+0000","last_active":"2023-10-30T14:26:35.724420+0000","last_peered":"2023-10-30T14:26:35.724420+0000","last_clean":"2023-10-30T14:26:35.724420+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T14:26:35.724420+0000","last_undegraded":"2023-10-30T14:26:35.724420+0000","last_fullsized":"2023-10-30T14:26:35.724420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2200,"log_dups_size":0,"ondisk_log_size":2200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:34.572699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.59694472600000004,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":143,"num_read_kb":13239,"num_write":1306,"num_write_kb":17120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.4","version":"103'610","reported_seq":920,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.258720+0000","last_change":"2023-10-30T13:23:18.958210+0000","last_active":"2023-10-30T14:26:33.258720+0000","last_peered":"2023-10-30T14:26:33.258720+0000","last_clean":"2023-10-30T14:26:33.258720+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T14:26:33.258720+0000","last_undegraded":"2023-10-30T14:26:33.258720+0000","last_fullsized":"2023-10-30T14:26:33.258720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:34.653005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":230,"num_read_kb":520969,"num_write":671,"num_write_kb":95828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"103'2173","reported_seq":4990,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.726608+0000","last_change":"2023-10-30T14:26:34.025524+0000","last_active":"2023-10-30T14:26:35.726608+0000","last_peered":"2023-10-30T14:26:35.726608+0000","last_clean":"2023-10-30T14:26:35.726608+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T14:26:35.726608+0000","last_undegraded":"2023-10-30T14:26:35.726608+0000","last_fullsized":"2023-10-30T14:26:35.726608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:45.716921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.075105449,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":60418,"num_write":1286,"num_write_kb":36950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.5","version":"103'1046","reported_seq":1330,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.841399+0000","last_change":"2023-10-30T13:23:18.985185+0000","last_active":"2023-10-30T14:26:33.841399+0000","last_peered":"2023-10-30T14:26:33.841399+0000","last_clean":"2023-10-30T14:26:33.841399+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T14:26:33.841399+0000","last_undegraded":"2023-10-30T14:26:33.841399+0000","last_fullsized":"2023-10-30T14:26:33.841399+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:36.230072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44105804,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":214,"num_read_kb":377318,"num_write":1088,"num_write_kb":69045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"103'2202","reported_seq":5034,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.718743+0000","last_change":"2023-10-30T14:26:33.629915+0000","last_active":"2023-10-30T14:26:35.718743+0000","last_peered":"2023-10-30T14:26:35.718743+0000","last_clean":"2023-10-30T14:26:35.718743+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T14:26:35.718743+0000","last_undegraded":"2023-10-30T14:26:35.718743+0000","last_fullsized":"2023-10-30T14:26:35.718743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:15.644068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.36741958800000002,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":157,"num_read_kb":35362,"num_write":1332,"num_write_kb":31772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.6","version":"107'1126","reported_seq":1419,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.259087+0000","last_change":"2023-10-30T13:23:18.978085+0000","last_active":"2023-10-30T14:26:33.259087+0000","last_peered":"2023-10-30T14:26:33.259087+0000","last_clean":"2023-10-30T14:26:33.259087+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T14:26:33.259087+0000","last_undegraded":"2023-10-30T14:26:33.259087+0000","last_fullsized":"2023-10-30T14:26:33.259087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1126,"log_dups_size":0,"ondisk_log_size":1126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:47.588972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":247,"num_read_kb":365780,"num_write":1781,"num_write_kb":65025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"103'2077","reported_seq":4909,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.895121+0000","last_change":"2023-10-30T14:26:33.252060+0000","last_active":"2023-10-30T14:26:34.895121+0000","last_peered":"2023-10-30T14:26:34.895121+0000","last_clean":"2023-10-30T14:26:34.895121+0000","last_became_active":"2023-10-30T13:24:01.001412+0000","last_became_peered":"2023-10-30T13:24:01.001412+0000","last_unstale":"2023-10-30T14:26:34.895121+0000","last_undegraded":"2023-10-30T14:26:34.895121+0000","last_fullsized":"2023-10-30T14:26:34.895121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2077,"log_dups_size":0,"ondisk_log_size":2077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:37:55.891565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.32700652899999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":137,"num_read_kb":5811,"num_write":1214,"num_write_kb":16143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.7","version":"103'468","reported_seq":773,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:32.928722+0000","last_change":"2023-10-30T13:23:18.970566+0000","last_active":"2023-10-30T14:26:32.928722+0000","last_peered":"2023-10-30T14:26:32.928722+0000","last_clean":"2023-10-30T14:26:32.928722+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T14:26:32.928722+0000","last_undegraded":"2023-10-30T14:26:32.928722+0000","last_fullsized":"2023-10-30T14:26:32.928722+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:03.851889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":281,"num_read_kb":442743,"num_write":555,"num_write_kb":70797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"103'2179","reported_seq":5010,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.720013+0000","last_change":"2023-10-30T14:26:34.649272+0000","last_active":"2023-10-30T14:26:35.720013+0000","last_peered":"2023-10-30T14:26:35.720013+0000","last_clean":"2023-10-30T14:26:35.720013+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T14:26:35.720013+0000","last_undegraded":"2023-10-30T14:26:35.720013+0000","last_fullsized":"2023-10-30T14:26:35.720013+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:08.185639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.183697322,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":144,"num_read_kb":25445,"num_write":1326,"num_write_kb":20705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.8","version":"103'544","reported_seq":809,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.838454+0000","last_change":"2023-10-30T13:23:18.986394+0000","last_active":"2023-10-30T14:26:33.838454+0000","last_peered":"2023-10-30T14:26:33.838454+0000","last_clean":"2023-10-30T14:26:33.838454+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T14:26:33.838454+0000","last_undegraded":"2023-10-30T14:26:33.838454+0000","last_fullsized":"2023-10-30T14:26:33.838454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:59.986981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":236,"num_read_kb":274544,"num_write":603,"num_write_kb":42177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"103'2113","reported_seq":4960,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.724178+0000","last_change":"2023-10-30T14:26:33.516141+0000","last_active":"2023-10-30T14:26:35.724178+0000","last_peered":"2023-10-30T14:26:35.724178+0000","last_clean":"2023-10-30T14:26:35.724178+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T14:26:35.724178+0000","last_undegraded":"2023-10-30T14:26:35.724178+0000","last_fullsized":"2023-10-30T14:26:35.724178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:05.930508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.56267233699999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":144,"num_read_kb":34655,"num_write":1307,"num_write_kb":20329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.9","version":"103'1061","reported_seq":1331,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.258600+0000","last_change":"2023-10-30T13:23:18.963905+0000","last_active":"2023-10-30T14:26:33.258600+0000","last_peered":"2023-10-30T14:26:33.258600+0000","last_clean":"2023-10-30T14:26:33.258600+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T14:26:33.258600+0000","last_undegraded":"2023-10-30T14:26:33.258600+0000","last_fullsized":"2023-10-30T14:26:33.258600+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1061,"log_dups_size":0,"ondisk_log_size":1061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":237,"num_read_kb":249991,"num_write":1213,"num_write_kb":43538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"103'2263","reported_seq":5048,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.357975+0000","last_change":"2023-10-30T14:26:36.405912+0000","last_active":"2023-10-30T14:26:36.357975+0000","last_peered":"2023-10-30T14:26:36.357975+0000","last_clean":"2023-10-30T14:26:36.357975+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T14:26:36.357975+0000","last_undegraded":"2023-10-30T14:26:36.357975+0000","last_fullsized":"2023-10-30T14:26:36.357975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2263,"log_dups_size":0,"ondisk_log_size":2263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:29:31.853578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.93815086199999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":129,"num_read_kb":1227,"num_write":1352,"num_write_kb":12536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.a","version":"103'1473","reported_seq":1808,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.258472+0000","last_change":"2023-10-30T13:23:18.952302+0000","last_active":"2023-10-30T14:26:33.258472+0000","last_peered":"2023-10-30T14:26:33.258472+0000","last_clean":"2023-10-30T14:26:33.258472+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T14:26:33.258472+0000","last_undegraded":"2023-10-30T14:26:33.258472+0000","last_fullsized":"2023-10-30T14:26:33.258472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:21:12.827417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":321,"num_read_kb":438308,"num_write":2001,"num_write_kb":98279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"103'2090","reported_seq":4928,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.369993+0000","last_change":"2023-10-30T14:26:36.417706+0000","last_active":"2023-10-30T14:26:36.369993+0000","last_peered":"2023-10-30T14:26:36.369993+0000","last_clean":"2023-10-30T14:26:36.369993+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T14:26:36.369993+0000","last_undegraded":"2023-10-30T14:26:36.369993+0000","last_fullsized":"2023-10-30T14:26:36.369993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:50:45.036636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.94968995099999998,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":141,"num_read_kb":11852,"num_write":1237,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.b","version":"103'1064","reported_seq":1380,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.840499+0000","last_change":"2023-10-30T13:23:18.984912+0000","last_active":"2023-10-30T14:26:33.840499+0000","last_peered":"2023-10-30T14:26:33.840499+0000","last_clean":"2023-10-30T14:26:33.840499+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T14:26:33.840499+0000","last_undegraded":"2023-10-30T14:26:33.840499+0000","last_fullsized":"2023-10-30T14:26:33.840499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:27:16.205042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":296,"num_read_kb":429426,"num_write":1159,"num_write_kb":75926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"103'2153","reported_seq":4987,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.898995+0000","last_change":"2023-10-30T14:26:33.347987+0000","last_active":"2023-10-30T14:26:34.898995+0000","last_peered":"2023-10-30T14:26:34.898995+0000","last_clean":"2023-10-30T14:26:34.898995+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T14:26:34.898995+0000","last_undegraded":"2023-10-30T14:26:34.898995+0000","last_fullsized":"2023-10-30T14:26:34.898995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2153,"log_dups_size":0,"ondisk_log_size":2153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:25.445527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.419978554,"stat_sum":{"num_bytes":5,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":137,"num_read_kb":40109,"num_write":1287,"num_write_kb":25283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"3.1d","version":"103'2173","reported_seq":4979,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723792+0000","last_change":"2023-10-30T14:26:34.023924+0000","last_active":"2023-10-30T14:26:35.723792+0000","last_peered":"2023-10-30T14:26:35.723792+0000","last_clean":"2023-10-30T14:26:35.723792+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T14:26:35.723792+0000","last_undegraded":"2023-10-30T14:26:35.723792+0000","last_fullsized":"2023-10-30T14:26:35.723792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:40:49.561271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.073619836,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":129,"num_read_kb":12975,"num_write":1318,"num_write_kb":20116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1c","version":"103'734","reported_seq":1022,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.840725+0000","last_change":"2023-10-30T13:23:18.993372+0000","last_active":"2023-10-30T14:26:33.840725+0000","last_peered":"2023-10-30T14:26:33.840725+0000","last_clean":"2023-10-30T14:26:33.840725+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T14:26:33.840725+0000","last_undegraded":"2023-10-30T14:26:33.840725+0000","last_fullsized":"2023-10-30T14:26:33.840725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":734,"log_dups_size":0,"ondisk_log_size":734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:41.370678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":304,"num_read_kb":264369,"num_write":877,"num_write_kb":48007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"103'2635","reported_seq":5318,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723624+0000","last_change":"2023-10-30T14:26:33.971887+0000","last_active":"2023-10-30T14:26:35.723624+0000","last_peered":"2023-10-30T14:26:35.723624+0000","last_clean":"2023-10-30T14:26:35.723624+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T14:26:35.723624+0000","last_undegraded":"2023-10-30T14:26:35.723624+0000","last_fullsized":"2023-10-30T14:26:35.723624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2635,"log_dups_size":0,"ondisk_log_size":2635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:02.739787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.021840552,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":171,"num_read_kb":40337,"num_write":1456,"num_write_kb":34441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1d","version":"103'1179","reported_seq":1479,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.840962+0000","last_change":"2023-10-30T13:23:18.987633+0000","last_active":"2023-10-30T14:26:33.840962+0000","last_peered":"2023-10-30T14:26:33.840962+0000","last_clean":"2023-10-30T14:26:33.840962+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T14:26:33.840962+0000","last_undegraded":"2023-10-30T14:26:33.840962+0000","last_fullsized":"2023-10-30T14:26:33.840962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1179,"log_dups_size":0,"ondisk_log_size":1179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:09:33.324898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":255,"num_read_kb":502436,"num_write":1236,"num_write_kb":99955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"103'2021","reported_seq":4887,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.721600+0000","last_change":"2023-10-30T14:26:34.532204+0000","last_active":"2023-10-30T14:26:35.721600+0000","last_peered":"2023-10-30T14:26:35.721600+0000","last_clean":"2023-10-30T14:26:35.721600+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T14:26:35.721600+0000","last_undegraded":"2023-10-30T14:26:35.721600+0000","last_fullsized":"2023-10-30T14:26:35.721600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2021,"log_dups_size":0,"ondisk_log_size":2021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:06.800217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.015373858,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":13189,"num_write":1249,"num_write_kb":15084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1e","version":"103'779","reported_seq":1032,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.329244+0000","last_change":"2023-10-30T13:23:18.958470+0000","last_active":"2023-10-30T14:26:33.329244+0000","last_peered":"2023-10-30T14:26:33.329244+0000","last_clean":"2023-10-30T14:26:33.329244+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T14:26:33.329244+0000","last_undegraded":"2023-10-30T14:26:33.329244+0000","last_fullsized":"2023-10-30T14:26:33.329244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":779,"log_dups_size":0,"ondisk_log_size":779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:21.507744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":159,"num_read_kb":369044,"num_write":787,"num_write_kb":79251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"103'2191","reported_seq":5138,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.518541+0000","last_change":"2023-10-30T14:26:36.567193+0000","last_active":"2023-10-30T14:26:36.518541+0000","last_peered":"2023-10-30T14:26:36.518541+0000","last_clean":"2023-10-30T14:26:36.518541+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T14:26:36.518541+0000","last_undegraded":"2023-10-30T14:26:36.518541+0000","last_fullsized":"2023-10-30T14:26:36.518541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2191,"log_dups_size":0,"ondisk_log_size":2191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:03:26.764254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.043790548,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":262,"num_read_kb":42307,"num_write":1336,"num_write_kb":30630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b2","length":"e"}]},{"pgid":"2.1f","version":"103'1010","reported_seq":1328,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-30T14:26:33.516670+0000","last_change":"2023-10-30T13:23:18.980952+0000","last_active":"2023-10-30T14:26:33.516670+0000","last_peered":"2023-10-30T14:26:33.516670+0000","last_clean":"2023-10-30T14:26:33.516670+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T14:26:33.516670+0000","last_undegraded":"2023-10-30T14:26:33.516670+0000","last_fullsized":"2023-10-30T14:26:33.516670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1010,"log_dups_size":0,"ondisk_log_size":1010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:44.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777798,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":313,"num_read_kb":340671,"num_write":1085,"num_write_kb":63851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":11,"num_objects":2377,"num_object_clones":2,"num_object_copies":4754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":9389,"num_read_kb":1724409,"num_write":83589,"num_write_kb":1459751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138028,"ondisk_log_size":138028,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1284334055,"num_objects":1569,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1569,"num_whiteouts":0,"num_read":10652,"num_read_kb":12762809,"num_write":35633,"num_write_kb":2343104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2568822784,"data_stored":2568668110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31979,"ondisk_log_size":31979,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575614,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590972,"kb_used_data":389104,"kb_used_omap":1,"kb_used_meta":201854,"kb_avail":93780868,"statfs":{"total":96636764160,"available":96031608832,"internally_reserved":0,"allocated":398442496,"data_stored":397794175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":206698955},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.746,"15min":0.80500000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":1.6339999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.82999999999999996,"15min":0.91400000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.286,"5min":1.3160000000000001,"15min":1.448},"last":1.7250000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.85999999999999999,"15min":0.88300000000000001},"min":{"1min":0.46300000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0840000000000001,"5min":2.452,"15min":2.452},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.78600000000000003,"15min":0.84199999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.47399999999999998},"max":{"1min":1.028,"5min":1.198,"15min":1.3520000000000001},"last":1.1619999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.78600000000000003,"15min":0.82799999999999996},"min":{"1min":0.48999999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.169,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.88,"15min":0.95099999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.024,"5min":1.3009999999999999,"15min":1.5569999999999999},"last":0.94299999999999995}]},{"osd":3,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.294,"5min":1.405,"15min":1.405},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.028,"15min":1.004},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.21,"5min":1.4610000000000001,"15min":1.5269999999999999},"last":0.747}]},{"osd":4,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.378,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.2010000000000001,"5min":1.4350000000000001,"15min":1.732},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0309999999999999,"15min":1.03},"min":{"1min":0.495,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.266,"5min":1.4890000000000001,"15min":1.673},"last":0.97799999999999998}]},{"osd":5,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.1439999999999999,"15min":1.0800000000000001},"min":{"1min":0.42999999999999999,"5min":0.375,"15min":0.375},"max":{"1min":1.3720000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1120000000000001,"15min":1.1619999999999999},"min":{"1min":0.57099999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7809999999999999},"last":0.90200000000000002}]},{"osd":6,"last update":"Mon Oct 30 14:25:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.095,"15min":1.1060000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.456,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.151,"15min":1.1930000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3779999999999999,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.65700000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575617,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505380,"kb_used_data":339672,"kb_used_omap":1,"kb_used_meta":165694,"kb_avail":93866460,"statfs":{"total":96636764160,"available":96119255040,"internally_reserved":0,"allocated":347824128,"data_stored":347179842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169671111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.75700000000000001,"15min":0.80100000000000005},"min":{"1min":0.51100000000000001,"5min":0.46700000000000003,"15min":0.375},"max":{"1min":0.89600000000000002,"5min":1.306,"15min":1.363},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.88900000000000001,"15min":0.95299999999999996},"min":{"1min":0.53500000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3979999999999999,"5min":1.4099999999999999,"15min":1.4370000000000001},"last":0.625}]},{"osd":1,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.72299999999999998,"15min":0.77400000000000002},"min":{"1min":0.54100000000000004,"5min":0.46999999999999997,"15min":0.46800000000000003},"max":{"1min":0.875,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":3.452},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.47399999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.127,"5min":1.706,"15min":1.706},"last":0.85299999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.81299999999999994,"15min":0.82999999999999996},"min":{"1min":0.57699999999999996,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.1679999999999999,"5min":1.29,"15min":1.571},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.872,"5min":0.96899999999999997,"15min":0.93700000000000006},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.286,"5min":5.4480000000000004,"15min":5.4480000000000004},"last":3.6459999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0229999999999999,"15min":1.0329999999999999},"min":{"1min":0.68000000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":3.5169999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.103,"15min":1.1160000000000001},"min":{"1min":0.68899999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.23,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":3.5830000000000002}]},{"osd":4,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.90300000000000002,"15min":0.90700000000000003},"min":{"1min":0.70799999999999996,"5min":0.434,"15min":0.434},"max":{"1min":1.276,"5min":1.3640000000000001,"15min":1.4339999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.002,"15min":1.0309999999999999},"min":{"1min":0.71999999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.153,"5min":1.595,"15min":1.595},"last":0.443}]},{"osd":5,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.159,"15min":1.03},"min":{"1min":0.77400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.2889999999999999,"5min":3.0219999999999998,"15min":3.0219999999999998},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.0089999999999999,"15min":1.0509999999999999},"min":{"1min":0.75600000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.135,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":0.58199999999999996}]},{"osd":7,"last update":"Mon Oct 30 14:26:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.204,"15min":1.196},"min":{"1min":1.048,"5min":0.58699999999999997,"15min":0.52600000000000002},"max":{"1min":1.3740000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1599999999999999,"15min":1.1359999999999999},"min":{"1min":0.83399999999999996,"5min":0.752,"15min":0.63700000000000001},"max":{"1min":1.3440000000000001,"5min":2.806,"15min":2.806},"last":0.505}]}]},{"osd":5,"up_from":13,"seq":55834575614,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565832,"kb_used_data":368188,"kb_used_omap":1,"kb_used_meta":197630,"kb_avail":93806008,"statfs":{"total":96636764160,"available":96057352192,"internally_reserved":0,"allocated":377024512,"data_stored":376381993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202373574},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72399999999999998,"15min":0.86499999999999999},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.40999999999999998},"max":{"1min":1.0860000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.95099999999999996,"15min":0.95799999999999996},"min":{"1min":0.60499999999999998,"5min":0.58699999999999997,"15min":0.57199999999999995},"max":{"1min":1.2410000000000001,"5min":1.617,"15min":1.9219999999999999},"last":0.66200000000000003}]},{"osd":1,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.81999999999999995,"15min":0.84099999999999997},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":1.911},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.77000000000000002,"15min":0.83799999999999997},"min":{"1min":0.52800000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.97099999999999997,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.58199999999999996}]},{"osd":2,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.88100000000000001,"15min":0.88500000000000001},"min":{"1min":0.59699999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.2589999999999999,"5min":1.331,"15min":1.3999999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81299999999999994,"15min":0.84799999999999998},"min":{"1min":0.59999999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.331,"5min":1.331,"15min":1.796},"last":0.746}]},{"osd":3,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0049999999999999,"15min":1.006},"min":{"1min":0.63700000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92000000000000004,"15min":0.999},"min":{"1min":0.71899999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.1930000000000001,"5min":1.3460000000000001,"15min":2.0030000000000001},"last":0.79100000000000004}]},{"osd":4,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.88400000000000001,"15min":0.879},"min":{"1min":0.56899999999999995,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.2729999999999999,"5min":2.8170000000000002,"15min":2.8170000000000002},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.159,"5min":1.024,"15min":1.03},"min":{"1min":0.86799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.6299999999999999,"5min":2.2400000000000002,"15min":2.262},"last":0.72199999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.0880000000000001,"15min":1.1299999999999999},"min":{"1min":0.80900000000000005,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.1499999999999999,"5min":2.3090000000000002,"15min":2.7949999999999999},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.097,"5min":1.0229999999999999,"15min":1.056},"min":{"1min":0.71899999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.49,"5min":2.161,"15min":2.1629999999999998},"last":0.78000000000000003}]},{"osd":7,"last update":"Mon Oct 30 14:26:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0209999999999999,"15min":1.091},"min":{"1min":0.78100000000000003,"5min":0.317,"15min":0.317},"max":{"1min":1.3440000000000001,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.02},{"interface":"front","average":{"1min":1.149,"5min":1.1299999999999999,"15min":1.153},"min":{"1min":0.95099999999999996,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.4019999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.1100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575614,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464424,"kb_used_data":266716,"kb_used_omap":1,"kb_used_meta":197694,"kb_avail":93907416,"statfs":{"total":96636764160,"available":96161193984,"internally_reserved":0,"allocated":273117184,"data_stored":272461938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202439110},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"min":{"1min":0.48799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1259999999999999,"5min":1.3500000000000001,"15min":1.446},"last":0.439},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.71099999999999997,"15min":0.82299999999999995},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.875,"5min":1.659,"15min":1.659},"last":0.61899999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.56799999999999995,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.141,"5min":1.589,"15min":1.589},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.84399999999999997,"15min":0.84699999999999998},"min":{"1min":0.58999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.075,"5min":1.532,"15min":1.532},"last":0.49199999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.76100000000000001,"15min":0.85999999999999999},"min":{"1min":0.55800000000000005,"5min":0.5,"15min":0.5},"max":{"1min":0.94899999999999995,"5min":1.607,"15min":1.607},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84699999999999998,"15min":0.92200000000000004},"min":{"1min":0.57199999999999995,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.218,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.99099999999999999,"15min":0.97499999999999998},"min":{"1min":0.66300000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.242,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.099,"15min":1.0780000000000001},"min":{"1min":0.879,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3480000000000001,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.98899999999999999,"15min":1.042},"min":{"1min":0.77000000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.3009999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.65400000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.1279999999999999,"5min":1.3220000000000001,"15min":1.3280000000000001},"last":0.316}]},{"osd":6,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.97899999999999998,"15min":0.96099999999999997},"min":{"1min":0.78800000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.3100000000000001,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.042,"15min":1.0700000000000001},"min":{"1min":0.78300000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.238,"5min":1.677,"15min":1.728},"last":0.51800000000000002}]},{"osd":7,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.137,"15min":1.1040000000000001},"min":{"1min":1.0820000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.417,"5min":1.4750000000000001,"15min":1.518},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.204,"15min":1.1799999999999999},"min":{"1min":1.056,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4670000000000001,"5min":1.587,"15min":1.587},"last":0.33600000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575612,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589364,"kb_used_data":381992,"kb_used_omap":1,"kb_used_meta":207358,"kb_avail":93782476,"statfs":{"total":96636764160,"available":96033255424,"internally_reserved":0,"allocated":391159808,"data_stored":390501706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":212335048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38100000000000001,"15min":0.40600000000000003},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71799999999999997,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.308},"max":{"1min":0.747,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.47399999999999998}]},{"osd":1,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":0.76500000000000001,"5min":1.403,"15min":1.403},"last":1.895},{"interface":"front","average":{"1min":0.438,"5min":0.51700000000000002,"15min":0.54800000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65800000000000003,"5min":1.752,"15min":1.752},"last":1.8420000000000001}]},{"osd":2,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56399999999999995,"15min":0.59999999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69499999999999995,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.41699999999999998,"5min":0.29999999999999999,"15min":0.29299999999999998},"max":{"1min":0.78000000000000003,"5min":1.73,"15min":1.73},"last":0.56999999999999995}]},{"osd":4,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.80300000000000005},"min":{"1min":0.53600000000000003,"5min":0.379,"15min":0.379},"max":{"1min":1.6519999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.80400000000000005,"15min":0.82399999999999995},"min":{"1min":0.55800000000000005,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.679,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77400000000000002,"15min":0.83299999999999996},"min":{"1min":0.61499999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98899999999999999,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.81000000000000005,"15min":0.85699999999999998},"min":{"1min":0.58399999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.071,"5min":1.486,"15min":1.486},"last":0.81799999999999995}]},{"osd":6,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.85199999999999998,"15min":0.88400000000000001},"min":{"1min":0.55600000000000005,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.048,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.042},{"interface":"front","average":{"1min":0.878,"5min":0.91500000000000004,"15min":0.93999999999999995},"min":{"1min":0.71999999999999997,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0409999999999999,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.077}]},{"osd":7,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83199999999999996,"15min":0.86199999999999999},"min":{"1min":0.64200000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.1100000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79100000000000004,"15min":0.82599999999999996},"min":{"1min":0.56299999999999994,"5min":0.501,"15min":0.501},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.76200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299400,"kb_used_data":200252,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94072440,"statfs":{"total":96636764160,"available":96330178560,"internally_reserved":0,"allocated":205058048,"data_stored":204414271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101513670},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.433,"15min":0.45800000000000002},"min":{"1min":0.254,"5min":0.246,"15min":0.23899999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.89300000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":0.94599999999999995,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.42199999999999999,"15min":0.44700000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72999999999999998,"5min":1.956,"15min":1.956},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.36199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":1.153}]},{"osd":3,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81999999999999995,"5min":1.988,"15min":1.988},"last":1.2010000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49099999999999999,"15min":0.51800000000000002},"min":{"1min":0.32300000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.90100000000000002,"5min":2.335,"15min":2.335},"last":0.67900000000000005}]},{"osd":4,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.60899999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5409999999999999,"5min":2.302,"15min":2.302},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84299999999999997,"15min":0.84099999999999997},"min":{"1min":0.625,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6879999999999999,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.91200000000000003}]},{"osd":5,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.77100000000000002,"15min":0.80600000000000005},"min":{"1min":0.64500000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82699999999999996,"15min":0.84499999999999997},"min":{"1min":0.73099999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.587,"5min":1.875,"15min":1.875},"last":0.85299999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.81599999999999995,"15min":0.82799999999999996},"min":{"1min":0.61299999999999999,"5min":0.502,"15min":0.48099999999999998},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84499999999999997,"15min":0.86799999999999999},"min":{"1min":0.60999999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.429,"5min":1.921,"15min":1.921},"last":1.2130000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.86099999999999999,"15min":0.86399999999999999},"min":{"1min":0.66100000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.131,"5min":4.5739999999999998,"15min":4.5739999999999998},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.88900000000000001,"15min":0.91700000000000004},"min":{"1min":0.64300000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6080000000000001,"5min":2.375,"15min":2.375},"last":1.1659999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575615,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490668,"kb_used_data":329060,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":93881172,"statfs":{"total":96636764160,"available":96134320128,"internally_reserved":0,"allocated":336957440,"data_stored":336316405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165476806},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.438,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.504,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.51600000000000001},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.47799999999999998,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.41499999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.46999999999999997,"15min":0.48399999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54800000000000004,"5min":1.954,"15min":1.954},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.45800000000000002,"15min":0.47799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63200000000000001,"5min":2.274,"15min":2.274},"last":5.7039999999999997}]},{"osd":3,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53600000000000003,"15min":0.54900000000000004},"min":{"1min":0.35599999999999998,"5min":0.35399999999999998,"15min":0.35099999999999998},"max":{"1min":0.60099999999999998,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":1.3220000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.436,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.70199999999999996,"5min":2.625,"15min":2.625},"last":1.3600000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.80200000000000005,"15min":0.83999999999999997},"min":{"1min":0.56499999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93300000000000005,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.82299999999999995,"15min":0.88},"min":{"1min":0.629,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.91200000000000003,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.752}]},{"osd":5,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82399999999999995,"15min":0.878},"min":{"1min":0.58099999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94799999999999995,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.85099999999999998,"15min":0.90100000000000002},"min":{"1min":0.60399999999999998,"5min":0.499,"15min":0.499},"max":{"1min":0.879,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.52200000000000002}]},{"osd":6,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.91000000000000003,"15min":0.84199999999999997},"min":{"1min":0.73799999999999999,"5min":0.624,"15min":0.56499999999999995},"max":{"1min":1.0289999999999999,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.81299999999999994,"15min":0.89700000000000002},"min":{"1min":0.56200000000000006,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.95799999999999996,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.56399999999999995}]},{"osd":7,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.80600000000000005,"15min":0.82999999999999996},"min":{"1min":0.59599999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.95999999999999996,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":0.626},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.82799999999999996,"15min":0.85199999999999998},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93799999999999994,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.66900000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575612,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421176,"kb_used_data":245740,"kb_used_omap":1,"kb_used_meta":175422,"kb_avail":93950664,"statfs":{"total":96636764160,"available":96205479936,"internally_reserved":0,"allocated":251637760,"data_stored":250992448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179632582},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.38100000000000001,"15min":0.40500000000000003},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54400000000000004,"5min":0.66100000000000003,"15min":0.94999999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.49299999999999999,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":1.0229999999999999},"last":0.65800000000000003}]},{"osd":2,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.45300000000000001,"15min":0.46700000000000003},"min":{"1min":0.307,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":1.05,"15min":1.05},"last":1.302},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.47099999999999997,"15min":0.498},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.29099999999999998},"max":{"1min":0.59699999999999998,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.83599999999999997}]},{"osd":3,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52900000000000003,"15min":0.54900000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.34499999999999997},"max":{"1min":0.73299999999999998,"5min":1.236,"15min":1.236},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51600000000000001,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.74399999999999999,"5min":1.054,"15min":1.054},"last":1.5800000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.78600000000000003,"15min":0.80300000000000005},"min":{"1min":0.65900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":0.88800000000000001,"5min":1.103,"15min":1.29},"last":1.385},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76600000000000001,"15min":0.81000000000000005},"min":{"1min":0.59299999999999997,"5min":0.53800000000000003,"15min":0.48399999999999999},"max":{"1min":0.95999999999999996,"5min":1.012,"15min":1.2210000000000001},"last":0.42599999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.78700000000000003,"15min":0.81200000000000006},"min":{"1min":0.64700000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.93899999999999995,"5min":1.0509999999999999,"15min":1.347},"last":1.7709999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.81599999999999995,"15min":0.85199999999999998},"min":{"1min":0.63,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.95899999999999996,"5min":1.2130000000000001,"15min":1.3939999999999999},"last":1.8779999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81499999999999995,"15min":0.84399999999999997},"min":{"1min":0.64500000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.95799999999999996,"5min":1.222,"15min":1.486},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79700000000000004,"15min":0.83399999999999996},"min":{"1min":0.64400000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0149999999999999,"5min":1.113,"15min":1.1619999999999999},"last":1.8109999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.81699999999999995,"15min":0.85099999999999998},"min":{"1min":0.625,"5min":0.625,"15min":0.58599999999999997},"max":{"1min":0.97599999999999998,"5min":1.2509999999999999,"15min":1.29},"last":1.8440000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84199999999999997,"15min":0.872},"min":{"1min":0.70799999999999996,"5min":0.61799999999999999,"15min":0.57299999999999995},"max":{"1min":0.96099999999999997,"5min":1.173,"15min":1.4419999999999999},"last":0.47999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249647104,"data_stored":249627338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335572992,"data_stored":335549765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203661312,"data_stored":203639529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389746688,"data_stored":389725939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":271699968,"data_stored":271686166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":375627776,"data_stored":375607251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":346419200,"data_stored":346404076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":396447744,"data_stored":396428046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":11,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:26:38.601 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:26:38.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.778+0000 7f562c7d3640 1 Processor -- start 2023-10-30T14:26:38.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.778+0000 7f562c7d3640 1 -- start start 2023-10-30T14:26:38.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562c7d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56241538a0 0x7f562415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562c7d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56240cf160 0x7f56240cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562c7d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56240cb800 con 0x7f56240ca0e0 2023-10-30T14:26:38.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562c7d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562415dc30 con 0x7f56240cf160 2023-10-30T14:26:38.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562c7d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56240cb330 con 0x7f56241538a0 2023-10-30T14:26:38.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562ad70640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56240cf160 0x7f56240cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562ad70640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56240cf160 0x7f56240cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43562/0 (socket says 172.21.15.46:43562) 2023-10-30T14:26:38.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562ad70640 1 -- 172.21.15.46:0/3713359281 learned_addr learned my addr 172.21.15.46:0/3713359281 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:38.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f5629d6e640 1 --1- 172.21.15.46:0/3713359281 >> v1:172.21.15.46:6789/0 conn(0x7f56240ca0e0 0x7f562415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38032/0 (socket says 172.21.15.46:38032) 2023-10-30T14:26:38.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562a56f640 1 --2- 172.21.15.46:0/3713359281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56241538a0 0x7f562415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562ad70640 1 -- 172.21.15.46:0/3713359281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56241538a0 msgr2=0x7f562415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:38.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562ad70640 1 --2- 172.21.15.46:0/3713359281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56241538a0 0x7f562415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562ad70640 1 -- 172.21.15.46:0/3713359281 >> v1:172.21.15.46:6789/0 conn(0x7f56240ca0e0 legacy=0x7f562415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562ad70640 1 -- 172.21.15.46:0/3713359281 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56240cad70 con 0x7f56240cf160 2023-10-30T14:26:38.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.782+0000 7f562956d640 1 -- 172.21.15.46:0/3713359281 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228919915 0 0) 0x7f56240cb800 con 0x7f56240ca0e0 2023-10-30T14:26:38.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.786+0000 7f562ad70640 1 --2- 172.21.15.46:0/3713359281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56240cf160 0x7f56240cf590 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f561c009e50 tx=0x7f561c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfc7b952a3b5334 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:38.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.786+0000 7f562956d640 1 -- 172.21.15.46:0/3713359281 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c0af400 con 0x7f56240cf160 2023-10-30T14:26:38.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.786+0000 7f562956d640 1 -- 172.21.15.46:0/3713359281 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c0ab050 con 0x7f56240cf160 2023-10-30T14:26:38.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.786+0000 7f562956d640 1 -- 172.21.15.46:0/3713359281 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c0af780 con 0x7f56240cf160 2023-10-30T14:26:38.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.786+0000 7f562c7d3640 1 -- 172.21.15.46:0/3713359281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56240cf160 msgr2=0x7f56240cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.786+0000 7f562c7d3640 1 --2- 172.21.15.46:0/3713359281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56240cf160 0x7f56240cf590 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f561c009e50 tx=0x7f561c01f1f0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.786+0000 7f562c7d3640 1 -- 172.21.15.46:0/3713359281 shutdown_connections 2023-10-30T14:26:38.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.786+0000 7f562c7d3640 1 --2- 172.21.15.46:0/3713359281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56240cf160 0x7f56240cf590 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.786+0000 7f562c7d3640 1 --2- 172.21.15.46:0/3713359281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56241538a0 0x7f562415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.786+0000 7f562c7d3640 1 -- 172.21.15.46:0/3713359281 >> 172.21.15.46:0/3713359281 conn(0x7f562406bfe0 msgr2=0x7f562406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:38.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.786+0000 7f562c7d3640 1 -- 172.21.15.46:0/3713359281 shutdown_connections 2023-10-30T14:26:38.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.786+0000 7f562c7d3640 1 -- 172.21.15.46:0/3713359281 wait complete. 2023-10-30T14:26:38.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562c7d3640 1 Processor -- start 2023-10-30T14:26:38.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562c7d3640 1 -- start start 2023-10-30T14:26:38.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562c7d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56240ca0e0 0x7f56240b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562c7d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56240cf160 0x7f562415bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562c7d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56241538a0 0x7f562415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562c7d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562415b840 con 0x7f56240cf160 2023-10-30T14:26:38.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562c7d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562415a950 con 0x7f56240ca0e0 2023-10-30T14:26:38.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562c7d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562415ac30 con 0x7f56241538a0 2023-10-30T14:26:38.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f5629d6e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56240cf160 0x7f562415bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f5629d6e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56240cf160 0x7f562415bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43564/0 (socket says 172.21.15.46:43564) 2023-10-30T14:26:38.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f5629d6e640 1 -- 172.21.15.46:0/2273986093 learned_addr learned my addr 172.21.15.46:0/2273986093 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:38.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562a56f640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56240ca0e0 0x7f56240b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562ad70640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56241538a0 0x7f562415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f5629d6e640 1 -- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56241538a0 msgr2=0x7f562415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f5629d6e640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56241538a0 0x7f562415c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f5629d6e640 1 -- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56240ca0e0 msgr2=0x7f56240b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f5629d6e640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56240ca0e0 0x7f56240b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f5629d6e640 1 -- 172.21.15.46:0/2273986093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c009b30 con 0x7f56240cf160 2023-10-30T14:26:38.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562ad70640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56241538a0 0x7f562415c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f5629d6e640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56240cf160 0x7f562415bd60 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f56140011f0 tx=0x7f561400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f561affd640 1 -- 172.21.15.46:0/2273986093 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56140074e0 con 0x7f56240cf160 2023-10-30T14:26:38.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f561affd640 1 -- 172.21.15.46:0/2273986093 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5614007640 con 0x7f56240cf160 2023-10-30T14:26:38.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562c7d3640 1 -- 172.21.15.46:0/2273986093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562415af10 con 0x7f56240cf160 2023-10-30T14:26:38.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f561affd640 1 -- 172.21.15.46:0/2273986093 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5614004a10 con 0x7f56240cf160 2023-10-30T14:26:38.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.790+0000 7f562c7d3640 1 -- 172.21.15.46:0/2273986093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562415b4d0 con 0x7f56240cf160 2023-10-30T14:26:38.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.794+0000 7f562c7d3640 1 -- 172.21.15.46:0/2273986093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56240b2a50 con 0x7f56240cf160 2023-10-30T14:26:38.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.794+0000 7f561affd640 1 -- 172.21.15.46:0/2273986093 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5614004b70 con 0x7f56240cf160 2023-10-30T14:26:38.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.794+0000 7f561affd640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56080664a0 0x7f5608068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:38.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.794+0000 7f561affd640 1 -- 172.21.15.46:0/2273986093 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 8748+0+0 (secure 0 0 0) 0x7f5614083f00 con 0x7f56240cf160 2023-10-30T14:26:38.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.794+0000 7f562a56f640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56080664a0 0x7f5608068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.798+0000 7f562a56f640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56080664a0 0x7f5608068960 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f56200079c0 tx=0x7f5620008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:38.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.798+0000 7f561affd640 1 -- 172.21.15.46:0/2273986093 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5614050890 con 0x7f56240cf160 2023-10-30T14:26:38.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.958+0000 7f562c7d3640 1 -- 172.21.15.46:0/2273986093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f562404e170 con 0x7f56240cf160 2023-10-30T14:26:38.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f561affd640 1 -- 172.21.15.46:0/2273986093 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+21184 (secure 0 0 0) 0x7f5614018070 con 0x7f56240cf160 2023-10-30T14:26:38.965 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:26:38.965 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":109,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T14:26:34.886261+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":703,"snap_epoch":108,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.13","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6836/3574404756":"2023-10-31T14:24:59.863368+0000","172.21.15.46:6834/3574404756":"2023-10-31T14:24:59.863368+0000","172.21.15.46:6837/470164358":"2023-10-31T14:23:54.843840+0000","172.21.15.46:6835/470164358":"2023-10-31T14:23:54.843840+0000","172.21.15.46:6833/3764280378":"2023-10-31T14:22:39.814064+0000","172.21.15.46:6832/3764280378":"2023-10-31T14:22:39.814064+0000","172.21.15.46:6836/2239735286":"2023-10-31T14:20:39.772410+0000","172.21.15.46:6834/2197729559":"2023-10-31T13:51:39.216877+0000","172.21.15.46:6833/190134548":"2023-10-31T13:50:04.191476+0000","172.21.15.46:6832/190134548":"2023-10-31T13:50:04.191476+0000","172.21.15.46:6833/157453225":"2023-10-31T14:00:29.382692+0000","172.21.15.46:6837/1135482564":"2023-10-31T14:01:49.404286+0000","172.21.15.46:6837/3154171789":"2023-10-31T14:07:14.520406+0000","172.21.15.46:6836/3037060424":"2023-10-31T13:44:39.066140+0000","172.21.15.46:6835/1135482564":"2023-10-31T14:01:49.404286+0000","172.21.15.46:6835/2572113428":"2023-10-31T13:52:59.238225+0000","172.21.15.46:6833/2075693104":"2023-10-31T14:05:39.493074+0000","172.21.15.46:6836/2197729559":"2023-10-31T13:51:39.216877+0000","172.21.15.46:6835/994489214":"2023-10-31T13:25:13.695228+0000","172.21.15.46:6832/2075693104":"2023-10-31T14:05:39.493074+0000","172.21.15.46:6832/4044310929":"2023-10-31T13:41:14.000569+0000","172.21.15.46:6837/126953235":"2023-10-31T13:39:08.955190+0000","172.21.15.46:6832/875791418":"2023-10-31T13:26:28.713838+0000","172.21.15.46:6835/126953235":"2023-10-31T13:39:08.955190+0000","172.21.15.46:6837/3112266825":"2023-10-31T13:57:44.339823+0000","172.21.15.46:6836/3955931693":"2023-10-31T13:36:53.920781+0000","172.21.15.46:6834/3955931693":"2023-10-31T13:36:53.920781+0000","172.21.15.46:6835/681817982":"2023-10-31T14:18:44.741650+0000","172.21.15.46:6836/275749119":"2023-10-31T13:30:28.793688+0000","172.21.15.46:6837/2572113428":"2023-10-31T13:52:59.238225+0000","172.21.15.154:6834/3109262047":"2023-10-31T13:29:08.773403+0000","172.21.15.46:6834/2239735286":"2023-10-31T14:20:39.772410+0000","172.21.15.46:6833/2455118950":"2023-10-31T14:16:04.689323+0000","172.21.15.46:6837/994489214":"2023-10-31T13:25:13.695228+0000","172.21.15.46:6837/3044281789":"2023-10-31T14:12:14.616071+0000","172.21.15.46:6835/1315246492":"2023-10-31T13:31:23.809908+0000","172.21.15.46:6835/3112266825":"2023-10-31T13:57:44.339823+0000","172.21.15.46:6834/128865315":"2023-10-31T14:08:44.543673+0000","172.21.15.46:6833/603884510":"2023-10-31T13:33:43.868205+0000","172.21.15.46:6835/2546436963":"2023-10-31T13:47:24.113508+0000","172.21.15.46:6837/1315246492":"2023-10-31T13:31:23.809908+0000","172.21.15.154:6834/651441434":"2023-10-31T13:35:03.892131+0000","172.21.15.46:6833/2807435195":"2023-10-31T13:54:39.265053+0000","172.21.15.46:6834/711135361":"2023-10-31T13:55:44.283031+0000","172.21.15.154:6835/651441434":"2023-10-31T13:35:03.892131+0000","172.21.15.46:6836/711135361":"2023-10-31T13:55:44.283031+0000","172.21.15.46:6837/2546436963":"2023-10-31T13:47:24.113508+0000","172.21.15.46:6833/875791418":"2023-10-31T13:26:28.713838+0000","172.21.15.154:6835/3109262047":"2023-10-31T13:29:08.773403+0000","172.21.15.46:6832/2807435195":"2023-10-31T13:54:39.265053+0000","172.21.15.46:6832/157453225":"2023-10-31T14:00:29.382692+0000","172.21.15.46:6832/603884510":"2023-10-31T13:33:43.868205+0000","172.21.15.46:6834/3734066164":"2023-10-31T14:04:24.459794+0000","172.21.15.46:6833/4044310929":"2023-10-31T13:41:14.000569+0000","172.21.15.46:6834/66265765":"2023-10-31T14:14:54.671010+0000","172.21.15.46:6836/3734066164":"2023-10-31T14:04:24.459794+0000","172.21.15.46:6836/128865315":"2023-10-31T14:08:44.543673+0000","172.21.15.46:6832/2455118950":"2023-10-31T14:16:04.689323+0000","172.21.15.46:6835/3154171789":"2023-10-31T14:07:14.520406+0000","172.21.15.46:6832/4078242540":"2023-10-31T14:09:49.576202+0000","172.21.15.46:6834/3037060424":"2023-10-31T13:44:39.066140+0000","172.21.15.46:6833/4078242540":"2023-10-31T14:09:49.576202+0000","172.21.15.46:6835/3044281789":"2023-10-31T14:12:14.616071+0000","172.21.15.46:6836/66265765":"2023-10-31T14:14:54.671010+0000","172.21.15.46:6834/275749119":"2023-10-31T13:30:28.793688+0000","172.21.15.46:6837/681817982":"2023-10-31T14:18:44.741650+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":690,"length":14}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":686,"length":4}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:26:38.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f562c7d3640 1 -- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56080664a0 msgr2=0x7f5608068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f562c7d3640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56080664a0 0x7f5608068960 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f56200079c0 tx=0x7f5620008040 comp rx=0 tx=0).stop 2023-10-30T14:26:38.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f562c7d3640 1 -- 172.21.15.46:0/2273986093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56240cf160 msgr2=0x7f562415bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:38.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f562c7d3640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56240cf160 0x7f562415bd60 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f56140011f0 tx=0x7f561400c9b0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f562c7d3640 1 -- 172.21.15.46:0/2273986093 shutdown_connections 2023-10-30T14:26:38.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f562c7d3640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f56080664a0 0x7f5608068960 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f562c7d3640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56241538a0 0x7f562415c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f562c7d3640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56240cf160 0x7f562415bd60 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f562c7d3640 1 --2- 172.21.15.46:0/2273986093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56240ca0e0 0x7f56240b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f562c7d3640 1 -- 172.21.15.46:0/2273986093 >> 172.21.15.46:0/2273986093 conn(0x7f562406bfe0 msgr2=0x7f56241523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:38.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f562c7d3640 1 -- 172.21.15.46:0/2273986093 shutdown_connections 2023-10-30T14:26:38.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:38.962+0000 7f562c7d3640 1 -- 172.21.15.46:0/2273986093 wait complete. 2023-10-30T14:26:39.987 INFO:tasks.ceph:Scrubbing osd.0 2023-10-30T14:26:39.987 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:26:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.130+0000 7f4212091640 1 Processor -- start 2023-10-30T14:26:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.130+0000 7f4212091640 1 -- start start 2023-10-30T14:26:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f4212091640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f420c1536d0 0x7f420c0cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f4212091640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f420c15d9d0 0x7f420c0c4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f4212091640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420c0c4540 con 0x7f420c159760 2023-10-30T14:26:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f4212091640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f420c15a210 con 0x7f420c1536d0 2023-10-30T14:26:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f4212091640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420c15a4f0 con 0x7f420c15d9d0 2023-10-30T14:26:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f420bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f420c159760 0x7f420c0cf090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38048/0 (socket says 172.21.15.46:38048) 2023-10-30T14:26:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f420bfff640 1 -- 172.21.15.46:0/4205576622 learned_addr learned my addr 172.21.15.46:0/4205576622 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f420b7fe640 1 --2- 172.21.15.46:0/4205576622 >> v2:172.21.15.154:3300/0 conn(0x7f420c1536d0 0x7f420c0cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f420affd640 1 --2- 172.21.15.46:0/4205576622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f420c15d9d0 0x7f420c0c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f420a7fc640 1 -- 172.21.15.46:0/4205576622 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903238219 0 0) 0x7f420c0c4540 con 0x7f420c159760 2023-10-30T14:26:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f420a7fc640 1 -- 172.21.15.46:0/4205576622 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f4003530 con 0x7f420c159760 2023-10-30T14:26:40.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f420b7fe640 1 -- 172.21.15.46:0/4205576622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f420c15d9d0 msgr2=0x7f420c0c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:40.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f420a7fc640 1 -- 172.21.15.46:0/4205576622 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4044861571 0 0) 0x7f41f4003530 con 0x7f420c159760 2023-10-30T14:26:40.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f420b7fe640 1 --2- 172.21.15.46:0/4205576622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f420c15d9d0 0x7f420c0c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f420b7fe640 1 -- 172.21.15.46:0/4205576622 >> v1:172.21.15.46:6789/0 conn(0x7f420c159760 legacy=0x7f420c0cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f420b7fe640 1 -- 172.21.15.46:0/4205576622 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c15cef0 con 0x7f420c1536d0 2023-10-30T14:26:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.134+0000 7f420affd640 1 --2- 172.21.15.46:0/4205576622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f420c15d9d0 0x7f420c0c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:26:40.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.138+0000 7f420b7fe640 1 --2- 172.21.15.46:0/4205576622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f420c1536d0 0x7f420c0cb3c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0097c0 tx=0x7f41fc01c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65510fb7c47b21b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f420a7fc640 1 -- 172.21.15.46:0/4205576622 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204001a00 con 0x7f420c1536d0 2023-10-30T14:26:40.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f420a7fc640 1 -- 172.21.15.46:0/4205576622 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41fc0a7050 con 0x7f420c1536d0 2023-10-30T14:26:40.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f420a7fc640 1 -- 172.21.15.46:0/4205576622 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0ab620 con 0x7f420c1536d0 2023-10-30T14:26:40.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 -- 172.21.15.46:0/4205576622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f420c1536d0 msgr2=0x7f420c0cb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 --2- 172.21.15.46:0/4205576622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f420c1536d0 0x7f420c0cb3c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0097c0 tx=0x7f41fc01c3a0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 -- 172.21.15.46:0/4205576622 shutdown_connections 2023-10-30T14:26:40.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 --2- 172.21.15.46:0/4205576622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f420c15d9d0 0x7f420c0c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 --2- 172.21.15.46:0/4205576622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f420c1536d0 0x7f420c0cb3c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 -- 172.21.15.46:0/4205576622 >> 172.21.15.46:0/4205576622 conn(0x7f420c06bf70 msgr2=0x7f420c06c380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:40.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 -- 172.21.15.46:0/4205576622 shutdown_connections 2023-10-30T14:26:40.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 -- 172.21.15.46:0/4205576622 wait complete. 2023-10-30T14:26:40.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 Processor -- start 2023-10-30T14:26:40.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 -- start start 2023-10-30T14:26:40.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f420c1536d0 0x7f420c15c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f420c159760 0x7f420c152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c15d9d0 0x7f420c1529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c15afd0 con 0x7f420c15d9d0 2023-10-30T14:26:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0cd3f0 con 0x7f420c1536d0 2023-10-30T14:26:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.142+0000 7f4212091640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420c156ea0 con 0x7f420c159760 2023-10-30T14:26:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f420bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c15d9d0 0x7f420c1529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f420affd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f420c159760 0x7f420c152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f420b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f420c1536d0 0x7f420c15c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f420bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c15d9d0 0x7f420c1529a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43570/0 (socket says 172.21.15.46:43570) 2023-10-30T14:26:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f420bfff640 1 -- 172.21.15.46:0/939238241 learned_addr learned my addr 172.21.15.46:0/939238241 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f420bfff640 1 -- 172.21.15.46:0/939238241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f420c159760 msgr2=0x7f420c152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f420bfff640 1 --2- 172.21.15.46:0/939238241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f420c159760 0x7f420c152460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f420bfff640 1 -- 172.21.15.46:0/939238241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f420c1536d0 msgr2=0x7f420c15c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f420bfff640 1 --2- 172.21.15.46:0/939238241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f420c1536d0 0x7f420c15c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f420bfff640 1 -- 172.21.15.46:0/939238241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4200009420 con 0x7f420c15d9d0 2023-10-30T14:26:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f420bfff640 1 --2- 172.21.15.46:0/939238241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c15d9d0 0x7f420c1529a0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f420400cc70 tx=0x7f420400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f41f3fff640 1 -- 172.21.15.46:0/939238241 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420400ce80 con 0x7f420c15d9d0 2023-10-30T14:26:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f41f3fff640 1 -- 172.21.15.46:0/939238241 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4204012400 con 0x7f420c15d9d0 2023-10-30T14:26:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f4212091640 1 -- 172.21.15.46:0/939238241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41fc009420 con 0x7f420c15d9d0 2023-10-30T14:26:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f41f3fff640 1 -- 172.21.15.46:0/939238241 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42040127d0 con 0x7f420c15d9d0 2023-10-30T14:26:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f4212091640 1 -- 172.21.15.46:0/939238241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f420c15b4c0 con 0x7f420c15d9d0 2023-10-30T14:26:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f4212091640 1 -- 172.21.15.46:0/939238241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f420c0cf7a0 con 0x7f420c15d9d0 2023-10-30T14:26:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.146+0000 7f41f3fff640 1 -- 172.21.15.46:0/939238241 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4204007070 con 0x7f420c15d9d0 2023-10-30T14:26:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.150+0000 7f41f3fff640 1 --2- 172.21.15.46:0/939238241 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41f80664a0 0x7f41f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.150+0000 7f41f3fff640 1 -- 172.21.15.46:0/939238241 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f4204085390 con 0x7f420c15d9d0 2023-10-30T14:26:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.150+0000 7f41f3fff640 1 --2- 172.21.15.46:0/939238241 >> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] conn(0x7f41f8070480 0x7f41f8072920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.150+0000 7f41f3fff640 1 -- 172.21.15.46:0/939238241 --> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41f8072ff0 con 0x7f41f8070480 2023-10-30T14:26:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.150+0000 7f41f3fff640 1 -- 172.21.15.46:0/939238241 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4204085700 con 0x7f420c15d9d0 2023-10-30T14:26:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.150+0000 7f420b7fe640 1 --2- 172.21.15.46:0/939238241 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41f80664a0 0x7f41f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.150+0000 7f420affd640 1 --2- 172.21.15.46:0/939238241 >> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] conn(0x7f41f8070480 0x7f41f8072920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.150+0000 7f420affd640 1 --2- 172.21.15.46:0/939238241 >> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] conn(0x7f41f8070480 0x7f41f8072920 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.150+0000 7f420b7fe640 1 --2- 172.21.15.46:0/939238241 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41f80664a0 0x7f41f8068960 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0ad000 tx=0x7f41fc0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.150+0000 7f41f3fff640 1 -- 172.21.15.46:0/939238241 <== osd.0 v2:172.21.15.46:6809/1936524547 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f41f8072ff0 con 0x7f41f8070480 2023-10-30T14:26:40.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.162+0000 7f4212091640 1 -- 172.21.15.46:0/939238241 --> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f420c15b6e0 con 0x7f41f8070480 2023-10-30T14:26:40.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.162+0000 7f41f3fff640 1 -- 172.21.15.46:0/939238241 <== osd.0 v2:172.21.15.46:6809/1936524547 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f420c15b6e0 con 0x7f41f8070480 2023-10-30T14:26:40.168 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-10-30T14:26:40.168 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:26:40.169 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-10-30T14:26:40.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.166+0000 7f4212091640 1 -- 172.21.15.46:0/939238241 >> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] conn(0x7f41f8070480 msgr2=0x7f41f8072920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.166+0000 7f4212091640 1 --2- 172.21.15.46:0/939238241 >> [v2:172.21.15.46:6809/1936524547,v1:172.21.15.46:6811/1936524547] conn(0x7f41f8070480 0x7f41f8072920 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.166+0000 7f4212091640 1 -- 172.21.15.46:0/939238241 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41f80664a0 msgr2=0x7f41f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.166+0000 7f4212091640 1 --2- 172.21.15.46:0/939238241 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f41f80664a0 0x7f41f8068960 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0ad000 tx=0x7f41fc0a6000 comp rx=0 tx=0).stop 2023-10-30T14:26:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.166+0000 7f4212091640 1 -- 172.21.15.46:0/939238241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c15d9d0 msgr2=0x7f420c1529a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.166+0000 7f4212091640 1 --2- 172.21.15.46:0/939238241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c15d9d0 0x7f420c1529a0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f420400cc70 tx=0x7f420400a430 comp rx=0 tx=0).stop 2023-10-30T14:26:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.166+0000 7f420bfff640 1 -- 172.21.15.46:0/939238241 reap_dead start 2023-10-30T14:26:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.166+0000 7f4212091640 1 -- 172.21.15.46:0/939238241 shutdown_connections 2023-10-30T14:26:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.166+0000 7f4212091640 1 -- 172.21.15.46:0/939238241 >> 172.21.15.46:0/939238241 conn(0x7f420c06bf70 msgr2=0x7f420c0b20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.166+0000 7f4212091640 1 -- 172.21.15.46:0/939238241 shutdown_connections 2023-10-30T14:26:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.166+0000 7f4212091640 1 -- 172.21.15.46:0/939238241 wait complete. 2023-10-30T14:26:40.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-30T14:26:40.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.330+0000 7ffae72c9640 1 Processor -- start 2023-10-30T14:26:40.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.330+0000 7ffae72c9640 1 -- start start 2023-10-30T14:26:40.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.330+0000 7ffae72c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffae0153c60 0x7ffae00ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.330+0000 7ffae72c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffae0159830 0x7ffae0159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.330+0000 7ffae72c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae00cb780 con 0x7ffae00c4430 2023-10-30T14:26:40.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.330+0000 7ffae72c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae0154040 con 0x7ffae0153c60 2023-10-30T14:26:40.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.330+0000 7ffae72c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae015a2e0 con 0x7ffae0159830 2023-10-30T14:26:40.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.330+0000 7ffae5065640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ffae0153c60 0x7ffae00ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.330+0000 7ffae4864640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffae00c4430 0x7ffae00cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38052/0 (socket says 172.21.15.46:38052) 2023-10-30T14:26:40.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.330+0000 7ffae4864640 1 -- 172.21.15.46:0/1523286617 learned_addr learned my addr 172.21.15.46:0/1523286617 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:40.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.330+0000 7ffae5866640 1 --2- 172.21.15.46:0/1523286617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffae0159830 0x7ffae0159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.330+0000 7ffae5065640 1 -- 172.21.15.46:0/1523286617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffae0159830 msgr2=0x7ffae0159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:40.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.334+0000 7ffae5065640 1 --2- 172.21.15.46:0/1523286617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffae0159830 0x7ffae0159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.334+0000 7ffae5065640 1 -- 172.21.15.46:0/1523286617 >> v1:172.21.15.46:6789/0 conn(0x7ffae00c4430 legacy=0x7ffae00cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.334+0000 7ffae5065640 1 -- 172.21.15.46:0/1523286617 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae015a5c0 con 0x7ffae0153c60 2023-10-30T14:26:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.334+0000 7ffad7fff640 1 -- 172.21.15.46:0/1523286617 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280753272 0 0) 0x7ffae00cb780 con 0x7ffae00c4430 2023-10-30T14:26:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.334+0000 7ffae5866640 1 --2- 172.21.15.46:0/1523286617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffae0159830 0x7ffae0159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:40.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.334+0000 7ffae5065640 1 --2- 172.21.15.46:0/1523286617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffae0153c60 0x7ffae00ca360 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0097c0 tx=0x7ffadc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bef50cc298775ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.334+0000 7ffad7fff640 1 -- 172.21.15.46:0/1523286617 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc0ab600 con 0x7ffae0153c60 2023-10-30T14:26:40.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.334+0000 7ffad7fff640 1 -- 172.21.15.46:0/1523286617 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffadc0a7050 con 0x7ffae0153c60 2023-10-30T14:26:40.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.334+0000 7ffad7fff640 1 -- 172.21.15.46:0/1523286617 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc0ab980 con 0x7ffae0153c60 2023-10-30T14:26:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.334+0000 7ffae72c9640 1 -- 172.21.15.46:0/1523286617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffae0153c60 msgr2=0x7ffae00ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.334+0000 7ffae72c9640 1 --2- 172.21.15.46:0/1523286617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffae0153c60 0x7ffae00ca360 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0097c0 tx=0x7ffadc01c480 comp rx=0 tx=0).stop 2023-10-30T14:26:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 -- 172.21.15.46:0/1523286617 shutdown_connections 2023-10-30T14:26:40.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 --2- 172.21.15.46:0/1523286617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffae0159830 0x7ffae0159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 --2- 172.21.15.46:0/1523286617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffae0153c60 0x7ffae00ca360 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 -- 172.21.15.46:0/1523286617 >> 172.21.15.46:0/1523286617 conn(0x7ffae006bfe0 msgr2=0x7ffae006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:40.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 -- 172.21.15.46:0/1523286617 shutdown_connections 2023-10-30T14:26:40.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 -- 172.21.15.46:0/1523286617 wait complete. 2023-10-30T14:26:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 Processor -- start 2023-10-30T14:26:40.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 -- start start 2023-10-30T14:26:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae00c4430 0x7ffae00cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffae0153c60 0x7ffae014a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffae0159830 0x7ffae0148ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae014cdb0 con 0x7ffae0159830 2023-10-30T14:26:40.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae014c2a0 con 0x7ffae00c4430 2023-10-30T14:26:40.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae72c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae0154180 con 0x7ffae0153c60 2023-10-30T14:26:40.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae5866640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffae0159830 0x7ffae0148ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae5866640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffae0159830 0x7ffae0148ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43584/0 (socket says 172.21.15.46:43584) 2023-10-30T14:26:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae5866640 1 -- 172.21.15.46:0/573974639 learned_addr learned my addr 172.21.15.46:0/573974639 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae5065640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae00c4430 0x7ffae00cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae4864640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffae0153c60 0x7ffae014a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae5866640 1 -- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffae0153c60 msgr2=0x7ffae014a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae5866640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffae0153c60 0x7ffae014a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae5866640 1 -- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae00c4430 msgr2=0x7ffae00cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae5866640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae00c4430 0x7ffae00cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae5866640 1 -- 172.21.15.46:0/573974639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffadc009420 con 0x7ffae0159830 2023-10-30T14:26:40.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.338+0000 7ffae5866640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffae0159830 0x7ffae0148ec0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7ffad8007e20 tx=0x7ffad800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.342+0000 7ffad5ffb640 1 -- 172.21.15.46:0/573974639 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8003ba0 con 0x7ffae0159830 2023-10-30T14:26:40.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.342+0000 7ffad5ffb640 1 -- 172.21.15.46:0/573974639 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad800f040 con 0x7ffae0159830 2023-10-30T14:26:40.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.342+0000 7ffad5ffb640 1 -- 172.21.15.46:0/573974639 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad800de80 con 0x7ffae0159830 2023-10-30T14:26:40.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.342+0000 7ffae72c9640 1 -- 172.21.15.46:0/573974639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae00c93a0 con 0x7ffae0159830 2023-10-30T14:26:40.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.342+0000 7ffae72c9640 1 -- 172.21.15.46:0/573974639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae0154690 con 0x7ffae0159830 2023-10-30T14:26:40.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.342+0000 7ffae72c9640 1 -- 172.21.15.46:0/573974639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae004e170 con 0x7ffae0159830 2023-10-30T14:26:40.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.342+0000 7ffad5ffb640 1 -- 172.21.15.46:0/573974639 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffad8004030 con 0x7ffae0159830 2023-10-30T14:26:40.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.346+0000 7ffad5ffb640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffac40664d0 0x7ffac4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.346+0000 7ffad5ffb640 1 -- 172.21.15.46:0/573974639 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7ffad8084c90 con 0x7ffae0159830 2023-10-30T14:26:40.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.346+0000 7ffae5065640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffac40664d0 0x7ffac4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.346+0000 7ffae5065640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffac40664d0 0x7ffac4068990 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0ad000 tx=0x7ffadc0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.350+0000 7ffad5ffb640 1 -- 172.21.15.46:0/573974639 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffad80515b0 con 0x7ffae0159830 2023-10-30T14:26:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.510+0000 7ffae72c9640 1 -- 172.21.15.46:0/573974639 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7ffae00cbf90 con 0x7ffac40664d0 2023-10-30T14:26:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.510+0000 7ffad5ffb640 1 -- 172.21.15.46:0/573974639 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ffadc004380 con 0x7ffac40664d0 2023-10-30T14:26:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 0 to deep-scrub 2023-10-30T14:26:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.514+0000 7ffae72c9640 1 -- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffac40664d0 msgr2=0x7ffac4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.514+0000 7ffae72c9640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffac40664d0 0x7ffac4068990 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0ad000 tx=0x7ffadc0a6000 comp rx=0 tx=0).stop 2023-10-30T14:26:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.514+0000 7ffae72c9640 1 -- 172.21.15.46:0/573974639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffae0159830 msgr2=0x7ffae0148ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.514+0000 7ffae72c9640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffae0159830 0x7ffae0148ec0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7ffad8007e20 tx=0x7ffad800d4a0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.514+0000 7ffae72c9640 1 -- 172.21.15.46:0/573974639 shutdown_connections 2023-10-30T14:26:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.514+0000 7ffae72c9640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7ffac40664d0 0x7ffac4068990 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.514+0000 7ffae72c9640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffae0159830 0x7ffae0148ec0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.514+0000 7ffae72c9640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffae0153c60 0x7ffae014a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.514+0000 7ffae72c9640 1 --2- 172.21.15.46:0/573974639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae00c4430 0x7ffae00cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.514+0000 7ffae72c9640 1 -- 172.21.15.46:0/573974639 >> 172.21.15.46:0/573974639 conn(0x7ffae006bfe0 msgr2=0x7ffae0158620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.514+0000 7ffae72c9640 1 -- 172.21.15.46:0/573974639 shutdown_connections 2023-10-30T14:26:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.514+0000 7ffae72c9640 1 -- 172.21.15.46:0/573974639 wait complete. 2023-10-30T14:26:40.537 INFO:tasks.ceph:Scrubbing osd.1 2023-10-30T14:26:40.537 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:26:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01c4dc4640 1 Processor -- start 2023-10-30T14:26:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01c4dc4640 1 -- start start 2023-10-30T14:26:40.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01c4dc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f01c015d9d0 0x7f01c00cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01c4dc4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c015a210 con 0x7f01c00c4000 2023-10-30T14:26:40.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01c4dc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c015a4f0 con 0x7f01c01536d0 2023-10-30T14:26:40.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01c4dc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01c015cef0 con 0x7f01c015d9d0 2023-10-30T14:26:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f01c00c4000 0x7f01c00cf090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38060/0 (socket says 172.21.15.46:38060) 2023-10-30T14:26:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bed9d640 1 -- 172.21.15.46:0/1200826890 learned_addr learned my addr 172.21.15.46:0/1200826890 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:40.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bdd9b640 1 --2- 172.21.15.46:0/1200826890 >> v2:172.21.15.154:3300/0 conn(0x7f01c015d9d0 0x7f01c00cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742268067 0 0) 0x7f01c015a210 con 0x7f01c00c4000 2023-10-30T14:26:40.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a8003530 con 0x7f01c00c4000 2023-10-30T14:26:40.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4235665002 0 0) 0x7f01a8003530 con 0x7f01c00c4000 2023-10-30T14:26:40.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 >> v2:172.21.15.154:3300/0 conn(0x7f01c015d9d0 msgr2=0x7f01c00cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:40.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 --2- 172.21.15.46:0/1200826890 >> v2:172.21.15.154:3300/0 conn(0x7f01c015d9d0 0x7f01c00cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 >> v1:172.21.15.154:6789/0 conn(0x7f01c01536d0 legacy=0x7f01c0159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c015d1d0 con 0x7f01c00c4000 2023-10-30T14:26:40.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f01b8001a00 con 0x7f01c00c4000 2023-10-30T14:26:40.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bdd9b640 1 --2- 172.21.15.46:0/1200826890 >> v2:172.21.15.154:3300/0 conn(0x7f01c015d9d0 0x7f01c00cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:26:40.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 >> v1:172.21.15.46:6789/0 conn(0x7f01c00c4000 legacy=0x7f01c00cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 --2- 172.21.15.46:0/1200826890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01a8003980 0x7f01a8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 --2- 172.21.15.46:0/1200826890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a80047d0 0x7f01a8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 --2- 172.21.15.46:0/1200826890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01a8005190 0x7f01a800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c015cef0 con 0x7f01a80047d0 2023-10-30T14:26:40.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a800c030 con 0x7f01a8005190 2023-10-30T14:26:40.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a800c1f0 con 0x7f01a8003980 2023-10-30T14:26:40.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bdd9b640 1 --2- 172.21.15.46:0/1200826890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a80047d0 0x7f01a8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bed9d640 1 --2- 172.21.15.46:0/1200826890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01a8005190 0x7f01a800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01be59c640 1 --2- 172.21.15.46:0/1200826890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01a8003980 0x7f01a8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bdd9b640 1 -- 172.21.15.46:0/1200826890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01a8003980 msgr2=0x7f01a8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:40.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bdd9b640 1 --2- 172.21.15.46:0/1200826890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01a8003980 0x7f01a8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bdd9b640 1 -- 172.21.15.46:0/1200826890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01a8005190 msgr2=0x7f01a800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:40.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bdd9b640 1 --2- 172.21.15.46:0/1200826890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01a8005190 0x7f01a800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bdd9b640 1 -- 172.21.15.46:0/1200826890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a8003530 con 0x7f01a80047d0 2023-10-30T14:26:40.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.686+0000 7f01bed9d640 1 -- 172.21.15.46:0/1200826890 reap_dead start 2023-10-30T14:26:40.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.690+0000 7f01bdd9b640 1 --2- 172.21.15.46:0/1200826890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a80047d0 0x7f01a8004c50 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009cb0 tx=0x7f01b401cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39157a197db115b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.690+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b40a57a0 con 0x7f01a80047d0 2023-10-30T14:26:40.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.690+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b40a5900 con 0x7f01a80047d0 2023-10-30T14:26:40.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.690+0000 7f01bd59a640 1 -- 172.21.15.46:0/1200826890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b40a5cf0 con 0x7f01a80047d0 2023-10-30T14:26:40.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.690+0000 7f01c4dc4640 1 -- 172.21.15.46:0/1200826890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a80047d0 msgr2=0x7f01a8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.690+0000 7f01c4dc4640 1 --2- 172.21.15.46:0/1200826890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a80047d0 0x7f01a8004c50 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009cb0 tx=0x7f01b401cbf0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.690+0000 7f01c4dc4640 1 -- 172.21.15.46:0/1200826890 shutdown_connections 2023-10-30T14:26:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.690+0000 7f01c4dc4640 1 --2- 172.21.15.46:0/1200826890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a80047d0 0x7f01a8004c50 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.690+0000 7f01c4dc4640 1 -- 172.21.15.46:0/1200826890 >> 172.21.15.46:0/1200826890 conn(0x7f01c006bf70 msgr2=0x7f01c006c380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:40.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.690+0000 7f01c4dc4640 1 -- 172.21.15.46:0/1200826890 shutdown_connections 2023-10-30T14:26:40.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.690+0000 7f01c4dc4640 1 -- 172.21.15.46:0/1200826890 wait complete. 2023-10-30T14:26:40.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01c4dc4640 1 Processor -- start 2023-10-30T14:26:40.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01c4dc4640 1 -- start start 2023-10-30T14:26:40.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01c4dc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01a80047d0 0x7f01c0158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01c4dc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c00b3e70 0x7f01c01f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01c4dc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01c01f2fd0 0x7f01c00c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01c4dc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c014df60 con 0x7f01c00b3e70 2023-10-30T14:26:40.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01c4dc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01ef960 con 0x7f01a80047d0 2023-10-30T14:26:40.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01c4dc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01f3450 con 0x7f01c01f2fd0 2023-10-30T14:26:40.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01bdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c00b3e70 0x7f01c01f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01bdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c00b3e70 0x7f01c01f2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43608/0 (socket says 172.21.15.46:43608) 2023-10-30T14:26:40.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01bdd9b640 1 -- 172.21.15.46:0/2008127439 learned_addr learned my addr 172.21.15.46:0/2008127439 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:40.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01bed9d640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01c01f2fd0 0x7f01c00c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01be59c640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01a80047d0 0x7f01c0158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01bdd9b640 1 -- 172.21.15.46:0/2008127439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01c01f2fd0 msgr2=0x7f01c00c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01bdd9b640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f01c01f2fd0 0x7f01c00c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01bdd9b640 1 -- 172.21.15.46:0/2008127439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01a80047d0 msgr2=0x7f01c0158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01bdd9b640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01a80047d0 0x7f01c0158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01bdd9b640 1 -- 172.21.15.46:0/2008127439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c00b45b0 con 0x7f01c00b3e70 2023-10-30T14:26:40.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01be59c640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01a80047d0 0x7f01c0158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:40.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01bdd9b640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c00b3e70 0x7f01c01f2a90 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009cb0 tx=0x7f01b4019990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01aeffd640 1 -- 172.21.15.46:0/2008127439 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b40b8860 con 0x7f01c00b3e70 2023-10-30T14:26:40.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01aeffd640 1 -- 172.21.15.46:0/2008127439 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b40b89e0 con 0x7f01c00b3e70 2023-10-30T14:26:40.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01aeffd640 1 -- 172.21.15.46:0/2008127439 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b40b8d90 con 0x7f01c00b3e70 2023-10-30T14:26:40.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01c4dc4640 1 -- 172.21.15.46:0/2008127439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c00b50c0 con 0x7f01c00b3e70 2023-10-30T14:26:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.694+0000 7f01c4dc4640 1 -- 172.21.15.46:0/2008127439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c01efb70 con 0x7f01c00b3e70 2023-10-30T14:26:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.698+0000 7f01c4dc4640 1 -- 172.21.15.46:0/2008127439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f01c00c4450 con 0x7f01c00b3e70 2023-10-30T14:26:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.698+0000 7f01aeffd640 1 -- 172.21.15.46:0/2008127439 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f01b40b4070 con 0x7f01c00b3e70 2023-10-30T14:26:40.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.698+0000 7f01aeffd640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01a00664d0 0x7f01a0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.698+0000 7f01aeffd640 1 -- 172.21.15.46:0/2008127439 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f01b411f820 con 0x7f01c00b3e70 2023-10-30T14:26:40.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.698+0000 7f01aeffd640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] conn(0x7f01a00704b0 0x7f01a0072950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.698+0000 7f01aeffd640 1 -- 172.21.15.46:0/2008127439 --> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01a0073020 con 0x7f01a00704b0 2023-10-30T14:26:40.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.698+0000 7f01aeffd640 1 -- 172.21.15.46:0/2008127439 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01b4121e00 con 0x7f01c00b3e70 2023-10-30T14:26:40.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.698+0000 7f01be59c640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01a00664d0 0x7f01a0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.698+0000 7f01bed9d640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] conn(0x7f01a00704b0 0x7f01a0072950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.698+0000 7f01bed9d640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] conn(0x7f01a00704b0 0x7f01a0072950 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.702+0000 7f01be59c640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01a00664d0 0x7f01a0068990 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f01b0000c10 tx=0x7f01b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.702+0000 7f01aeffd640 1 -- 172.21.15.46:0/2008127439 <== osd.1 v2:172.21.15.46:6800/1107843451 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f01a0073020 con 0x7f01a00704b0 2023-10-30T14:26:40.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01c4dc4640 1 -- 172.21.15.46:0/2008127439 --> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f01c015d4b0 con 0x7f01a00704b0 2023-10-30T14:26:40.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01aeffd640 1 -- 172.21.15.46:0/2008127439 <== osd.1 v2:172.21.15.46:6800/1107843451 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f01b80054a0 con 0x7f01a00704b0 2023-10-30T14:26:40.719 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-10-30T14:26:40.719 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:26:40.720 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-10-30T14:26:40.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01c4dc4640 1 -- 172.21.15.46:0/2008127439 >> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] conn(0x7f01a00704b0 msgr2=0x7f01a0072950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01c4dc4640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.46:6800/1107843451,v1:172.21.15.46:6801/1107843451] conn(0x7f01a00704b0 0x7f01a0072950 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01c4dc4640 1 -- 172.21.15.46:0/2008127439 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01a00664d0 msgr2=0x7f01a0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01c4dc4640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f01a00664d0 0x7f01a0068990 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f01b0000c10 tx=0x7f01b0009040 comp rx=0 tx=0).stop 2023-10-30T14:26:40.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01c4dc4640 1 -- 172.21.15.46:0/2008127439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c00b3e70 msgr2=0x7f01c01f2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01c4dc4640 1 --2- 172.21.15.46:0/2008127439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c00b3e70 0x7f01c01f2a90 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009cb0 tx=0x7f01b4019990 comp rx=0 tx=0).stop 2023-10-30T14:26:40.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01bed9d640 1 -- 172.21.15.46:0/2008127439 reap_dead start 2023-10-30T14:26:40.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01c4dc4640 1 -- 172.21.15.46:0/2008127439 shutdown_connections 2023-10-30T14:26:40.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01c4dc4640 1 -- 172.21.15.46:0/2008127439 >> 172.21.15.46:0/2008127439 conn(0x7f01c006bf70 msgr2=0x7f01c01525d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:40.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01c4dc4640 1 -- 172.21.15.46:0/2008127439 shutdown_connections 2023-10-30T14:26:40.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.714+0000 7f01c4dc4640 1 -- 172.21.15.46:0/2008127439 wait complete. 2023-10-30T14:26:40.737 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-30T14:26:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.882+0000 7f6b8b70f640 1 Processor -- start 2023-10-30T14:26:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.882+0000 7f6b8b70f640 1 -- start start 2023-10-30T14:26:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.882+0000 7f6b8b70f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b840ca360 0x7f6b840cac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.882+0000 7f6b8b70f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b840cb780 con 0x7f6b840c4430 2023-10-30T14:26:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.882+0000 7f6b8b70f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b840ca7f0 con 0x7f6b84153c60 2023-10-30T14:26:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.882+0000 7f6b8b70f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b84154040 con 0x7f6b840ca360 2023-10-30T14:26:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.886+0000 7f6b89cac640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b840ca360 0x7f6b840cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.886+0000 7f6b88caa640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6b840c4430 0x7f6b84159830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38074/0 (socket says 172.21.15.46:38074) 2023-10-30T14:26:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.886+0000 7f6b89cac640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6b840ca360 0x7f6b840cac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43620/0 (socket says 172.21.15.46:43620) 2023-10-30T14:26:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.886+0000 7f6b88caa640 1 -- 172.21.15.46:0/2444741329 learned_addr learned my addr 172.21.15.46:0/2444741329 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.886+0000 7f6b89cac640 1 -- 172.21.15.46:0/2444741329 >> v1:172.21.15.154:6789/0 conn(0x7f6b84153c60 legacy=0x7f6b840cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.886+0000 7f6b89cac640 1 -- 172.21.15.46:0/2444741329 >> v1:172.21.15.46:6789/0 conn(0x7f6b840c4430 legacy=0x7f6b84159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.886+0000 7f6b89cac640 1 -- 172.21.15.46:0/2444741329 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8415a2e0 con 0x7f6b840ca360 2023-10-30T14:26:40.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.886+0000 7f6b89cac640 1 --2- 172.21.15.46:0/2444741329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca360 0x7f6b840cac40 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009d90 tx=0x7f6b7c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6967a1be7fbaf0ee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.886+0000 7f6b7bfff640 1 -- 172.21.15.46:0/2444741329 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0a58f0 con 0x7f6b840ca360 2023-10-30T14:26:40.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.886+0000 7f6b7bfff640 1 -- 172.21.15.46:0/2444741329 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c0a5a50 con 0x7f6b840ca360 2023-10-30T14:26:40.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.886+0000 7f6b7bfff640 1 -- 172.21.15.46:0/2444741329 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0a5de0 con 0x7f6b840ca360 2023-10-30T14:26:40.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2444741329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca360 msgr2=0x7f6b840cac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 --2- 172.21.15.46:0/2444741329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca360 0x7f6b840cac40 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009d90 tx=0x7f6b7c01c920 comp rx=0 tx=0).stop 2023-10-30T14:26:40.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2444741329 shutdown_connections 2023-10-30T14:26:40.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 --2- 172.21.15.46:0/2444741329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b840ca360 0x7f6b840cac40 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2444741329 >> 172.21.15.46:0/2444741329 conn(0x7f6b8406bfe0 msgr2=0x7f6b8406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:40.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2444741329 shutdown_connections 2023-10-30T14:26:40.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2444741329 wait complete. 2023-10-30T14:26:40.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 Processor -- start 2023-10-30T14:26:40.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 -- start start 2023-10-30T14:26:40.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b840c4430 0x7f6b840ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b840ca360 0x7f6b84148370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b84153c60 0x7f6b84140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840cde70 con 0x7f6b84153c60 2023-10-30T14:26:40.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b841405d0 con 0x7f6b840c4430 2023-10-30T14:26:40.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b8b70f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b84151020 con 0x7f6b840ca360 2023-10-30T14:26:40.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b89cac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b84153c60 0x7f6b84140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b89cac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b84153c60 0x7f6b84140090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43622/0 (socket says 172.21.15.46:43622) 2023-10-30T14:26:40.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b89cac640 1 -- 172.21.15.46:0/2530262923 learned_addr learned my addr 172.21.15.46:0/2530262923 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:40.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b894ab640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b840c4430 0x7f6b840ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b88caa640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b840ca360 0x7f6b84148370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b89cac640 1 -- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b840ca360 msgr2=0x7f6b84148370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b89cac640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b840ca360 0x7f6b84148370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b89cac640 1 -- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b840c4430 msgr2=0x7f6b840ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:40.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b89cac640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b840c4430 0x7f6b840ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:40.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b89cac640 1 -- 172.21.15.46:0/2530262923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c009a70 con 0x7f6b84153c60 2023-10-30T14:26:40.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.890+0000 7f6b88caa640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b840ca360 0x7f6b84148370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:40.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.894+0000 7f6b89cac640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b84153c60 0x7f6b84140090 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0b0040 tx=0x7f6b7c0196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.894+0000 7f6b79ffb640 1 -- 172.21.15.46:0/2530262923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0b9860 con 0x7f6b84153c60 2023-10-30T14:26:40.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.894+0000 7f6b79ffb640 1 -- 172.21.15.46:0/2530262923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c0b99c0 con 0x7f6b84153c60 2023-10-30T14:26:40.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.894+0000 7f6b79ffb640 1 -- 172.21.15.46:0/2530262923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0b9d70 con 0x7f6b84153c60 2023-10-30T14:26:40.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.894+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2530262923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b841448a0 con 0x7f6b84153c60 2023-10-30T14:26:40.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.894+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2530262923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b840cedf0 con 0x7f6b84153c60 2023-10-30T14:26:40.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.894+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2530262923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b840cdfb0 con 0x7f6b84153c60 2023-10-30T14:26:40.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.894+0000 7f6b79ffb640 1 -- 172.21.15.46:0/2530262923 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b7c0c4460 con 0x7f6b84153c60 2023-10-30T14:26:40.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.898+0000 7f6b79ffb640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b680664a0 0x7f6b68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:40.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.898+0000 7f6b79ffb640 1 -- 172.21.15.46:0/2530262923 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f6b7c11ff70 con 0x7f6b84153c60 2023-10-30T14:26:40.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.898+0000 7f6b894ab640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b680664a0 0x7f6b68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:40.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.898+0000 7f6b894ab640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b680664a0 0x7f6b68068960 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f6b80007fd0 tx=0x7f6b80009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:40.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:40.898+0000 7f6b79ffb640 1 -- 172.21.15.46:0/2530262923 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6b840cdfb0 con 0x7f6b84153c60 2023-10-30T14:26:41.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.058+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2530262923 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b840cbf90 con 0x7f6b680664a0 2023-10-30T14:26:41.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.058+0000 7f6b79ffb640 1 -- 172.21.15.46:0/2530262923 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6b80001450 con 0x7f6b680664a0 2023-10-30T14:26:41.062 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 1 to deep-scrub 2023-10-30T14:26:41.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.062+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b680664a0 msgr2=0x7f6b68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.062+0000 7f6b8b70f640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b680664a0 0x7f6b68068960 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f6b80007fd0 tx=0x7f6b80009040 comp rx=0 tx=0).stop 2023-10-30T14:26:41.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.062+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2530262923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b84153c60 msgr2=0x7f6b84140090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.062+0000 7f6b8b70f640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b84153c60 0x7f6b84140090 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0b0040 tx=0x7f6b7c0196c0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.062+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2530262923 shutdown_connections 2023-10-30T14:26:41.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.062+0000 7f6b8b70f640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f6b680664a0 0x7f6b68068960 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.062+0000 7f6b8b70f640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b84153c60 0x7f6b84140090 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.062+0000 7f6b8b70f640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b840ca360 0x7f6b84148370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.062+0000 7f6b8b70f640 1 --2- 172.21.15.46:0/2530262923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b840c4430 0x7f6b840ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.062+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2530262923 >> 172.21.15.46:0/2530262923 conn(0x7f6b8406bfe0 msgr2=0x7f6b84151ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:41.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.062+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2530262923 shutdown_connections 2023-10-30T14:26:41.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.062+0000 7f6b8b70f640 1 -- 172.21.15.46:0/2530262923 wait complete. 2023-10-30T14:26:41.085 INFO:tasks.ceph:Scrubbing osd.2 2023-10-30T14:26:41.085 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:26:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.230+0000 7f3d7882f640 1 Processor -- start 2023-10-30T14:26:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.230+0000 7f3d7882f640 1 -- start start 2023-10-30T14:26:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d7882f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d74153a80 0x7f3d7415a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d7882f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d740c4100 0x7f3d7415d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d7882f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7415d1b0 con 0x7f3d740cb1c0 2023-10-30T14:26:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d7882f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d740cb770 con 0x7f3d74153a80 2023-10-30T14:26:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d7882f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7415a740 con 0x7f3d740c4100 2023-10-30T14:26:41.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d73fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3d740cb1c0 0x7f3d740ceeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38082/0 (socket says 172.21.15.46:38082) 2023-10-30T14:26:41.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d73fff640 1 -- 172.21.15.46:0/203337973 learned_addr learned my addr 172.21.15.46:0/203337973 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:41.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d737fe640 1 --2- 172.21.15.46:0/203337973 >> v2:172.21.15.154:3300/0 conn(0x7f3d74153a80 0x7f3d7415a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d72ffd640 1 --2- 172.21.15.46:0/203337973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d740c4100 0x7f3d7415d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d727fc640 1 -- 172.21.15.46:0/203337973 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506475473 0 0) 0x7f3d7415d1b0 con 0x7f3d740cb1c0 2023-10-30T14:26:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d727fc640 1 -- 172.21.15.46:0/203337973 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d5c003530 con 0x7f3d740cb1c0 2023-10-30T14:26:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d737fe640 1 -- 172.21.15.46:0/203337973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d740c4100 msgr2=0x7f3d7415d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d737fe640 1 --2- 172.21.15.46:0/203337973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d740c4100 0x7f3d7415d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d737fe640 1 -- 172.21.15.46:0/203337973 >> v1:172.21.15.46:6789/0 conn(0x7f3d740cb1c0 legacy=0x7f3d740ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.234+0000 7f3d737fe640 1 -- 172.21.15.46:0/203337973 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d740ca6e0 con 0x7f3d74153a80 2023-10-30T14:26:41.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d737fe640 1 --2- 172.21.15.46:0/203337973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d74153a80 0x7f3d7415a200 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3d640097c0 tx=0x7f3d6401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ffec7181f89863e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d727fc640 1 -- 172.21.15.46:0/203337973 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d640a8970 con 0x7f3d74153a80 2023-10-30T14:26:41.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d727fc640 1 -- 172.21.15.46:0/203337973 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d640a8ad0 con 0x7f3d74153a80 2023-10-30T14:26:41.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d727fc640 1 -- 172.21.15.46:0/203337973 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d640a8e60 con 0x7f3d74153a80 2023-10-30T14:26:41.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d7882f640 1 -- 172.21.15.46:0/203337973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d74153a80 msgr2=0x7f3d7415a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d7882f640 1 --2- 172.21.15.46:0/203337973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d74153a80 0x7f3d7415a200 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3d640097c0 tx=0x7f3d6401ec70 comp rx=0 tx=0).stop 2023-10-30T14:26:41.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d7882f640 1 -- 172.21.15.46:0/203337973 shutdown_connections 2023-10-30T14:26:41.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d7882f640 1 --2- 172.21.15.46:0/203337973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d740c4100 0x7f3d7415d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d7882f640 1 --2- 172.21.15.46:0/203337973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d74153a80 0x7f3d7415a200 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d7882f640 1 -- 172.21.15.46:0/203337973 >> 172.21.15.46:0/203337973 conn(0x7f3d740719c0 msgr2=0x7f3d7406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d7882f640 1 -- 172.21.15.46:0/203337973 shutdown_connections 2023-10-30T14:26:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d7882f640 1 -- 172.21.15.46:0/203337973 wait complete. 2023-10-30T14:26:41.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d7882f640 1 Processor -- start 2023-10-30T14:26:41.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.238+0000 7f3d7882f640 1 -- start start 2023-10-30T14:26:41.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d7882f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d740c4100 0x7f3d741eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d7882f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d740cb1c0 0x7f3d74159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d7882f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d74153a80 0x7f3d741e5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d7882f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d741ee7f0 con 0x7f3d740cb1c0 2023-10-30T14:26:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d7882f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d741ee200 con 0x7f3d740c4100 2023-10-30T14:26:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d7882f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d741ef150 con 0x7f3d74153a80 2023-10-30T14:26:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d72ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d740cb1c0 0x7f3d74159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d72ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d740cb1c0 0x7f3d74159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43638/0 (socket says 172.21.15.46:43638) 2023-10-30T14:26:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d72ffd640 1 -- 172.21.15.46:0/3883784061 learned_addr learned my addr 172.21.15.46:0/3883784061 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d73fff640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d74153a80 0x7f3d741e5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d737fe640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d740c4100 0x7f3d741eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d72ffd640 1 -- 172.21.15.46:0/3883784061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d74153a80 msgr2=0x7f3d741e5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d72ffd640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d74153a80 0x7f3d741e5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d72ffd640 1 -- 172.21.15.46:0/3883784061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d740c4100 msgr2=0x7f3d741eec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d72ffd640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d740c4100 0x7f3d741eec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d72ffd640 1 -- 172.21.15.46:0/3883784061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d64009420 con 0x7f3d740cb1c0 2023-10-30T14:26:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d72ffd640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d740cb1c0 0x7f3d74159f80 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800e900 tx=0x7f3d6800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d5bfff640 1 -- 172.21.15.46:0/3883784061 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68009690 con 0x7f3d740cb1c0 2023-10-30T14:26:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d5bfff640 1 -- 172.21.15.46:0/3883784061 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6800ca00 con 0x7f3d740cb1c0 2023-10-30T14:26:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d5bfff640 1 -- 172.21.15.46:0/3883784061 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6800cd90 con 0x7f3d740cb1c0 2023-10-30T14:26:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d7882f640 1 -- 172.21.15.46:0/3883784061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d740b1970 con 0x7f3d740cb1c0 2023-10-30T14:26:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d7882f640 1 -- 172.21.15.46:0/3883784061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d740b4d00 con 0x7f3d740cb1c0 2023-10-30T14:26:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.242+0000 7f3d7882f640 1 -- 172.21.15.46:0/3883784061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3d740ca400 con 0x7f3d740cb1c0 2023-10-30T14:26:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.246+0000 7f3d5bfff640 1 -- 172.21.15.46:0/3883784061 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3d68013470 con 0x7f3d740cb1c0 2023-10-30T14:26:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.246+0000 7f3d5bfff640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d600664d0 0x7f3d60068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.246+0000 7f3d5bfff640 1 -- 172.21.15.46:0/3883784061 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f3d68084ff0 con 0x7f3d740cb1c0 2023-10-30T14:26:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.246+0000 7f3d5bfff640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] conn(0x7f3d600704b0 0x7f3d60072950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.246+0000 7f3d5bfff640 1 -- 172.21.15.46:0/3883784061 --> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d60073020 con 0x7f3d600704b0 2023-10-30T14:26:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.246+0000 7f3d5bfff640 1 -- 172.21.15.46:0/3883784061 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d680875d0 con 0x7f3d740cb1c0 2023-10-30T14:26:41.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.246+0000 7f3d737fe640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d600664d0 0x7f3d60068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.246+0000 7f3d73fff640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] conn(0x7f3d600704b0 0x7f3d60072950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.246+0000 7f3d73fff640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] conn(0x7f3d600704b0 0x7f3d60072950 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.246+0000 7f3d737fe640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d600664d0 0x7f3d60068990 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f3d640b2000 tx=0x7f3d64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.246+0000 7f3d5bfff640 1 -- 172.21.15.46:0/3883784061 <== osd.2 v2:172.21.15.46:6808/3816077541 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f3d60073020 con 0x7f3d600704b0 2023-10-30T14:26:41.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.270+0000 7f3d7882f640 1 -- 172.21.15.46:0/3883784061 --> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3d740ca160 con 0x7f3d600704b0 2023-10-30T14:26:41.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.270+0000 7f3d5bfff640 1 -- 172.21.15.46:0/3883784061 <== osd.2 v2:172.21.15.46:6808/3816077541 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f3d6c001fb0 con 0x7f3d600704b0 2023-10-30T14:26:41.275 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-10-30T14:26:41.276 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:26:41.276 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-10-30T14:26:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.270+0000 7f3d7882f640 1 -- 172.21.15.46:0/3883784061 >> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] conn(0x7f3d600704b0 msgr2=0x7f3d60072950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.270+0000 7f3d7882f640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.46:6808/3816077541,v1:172.21.15.46:6810/3816077541] conn(0x7f3d600704b0 0x7f3d60072950 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.270+0000 7f3d7882f640 1 -- 172.21.15.46:0/3883784061 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d600664d0 msgr2=0x7f3d60068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.274+0000 7f3d7882f640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d600664d0 0x7f3d60068990 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f3d640b2000 tx=0x7f3d64008040 comp rx=0 tx=0).stop 2023-10-30T14:26:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.274+0000 7f3d7882f640 1 -- 172.21.15.46:0/3883784061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d740cb1c0 msgr2=0x7f3d74159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.274+0000 7f3d7882f640 1 --2- 172.21.15.46:0/3883784061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d740cb1c0 0x7f3d74159f80 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800e900 tx=0x7f3d6800edc0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.274+0000 7f3d73fff640 1 -- 172.21.15.46:0/3883784061 reap_dead start 2023-10-30T14:26:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.274+0000 7f3d7882f640 1 -- 172.21.15.46:0/3883784061 shutdown_connections 2023-10-30T14:26:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.274+0000 7f3d7882f640 1 -- 172.21.15.46:0/3883784061 >> 172.21.15.46:0/3883784061 conn(0x7f3d740719c0 msgr2=0x7f3d74150560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.274+0000 7f3d7882f640 1 -- 172.21.15.46:0/3883784061 shutdown_connections 2023-10-30T14:26:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.274+0000 7f3d7882f640 1 -- 172.21.15.46:0/3883784061 wait complete. 2023-10-30T14:26:41.299 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-30T14:26:41.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.438+0000 7fa4edd3e640 1 Processor -- start 2023-10-30T14:26:41.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.438+0000 7fa4edd3e640 1 -- start start 2023-10-30T14:26:41.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4edd3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4e80c4430 0x7fa4e80ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4edd3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa4e8159830 0x7fa4e8159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4edd3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e80cb780 con 0x7fa4e8153c60 2023-10-30T14:26:41.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4edd3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e8154040 con 0x7fa4e8159830 2023-10-30T14:26:41.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4edd3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e815a2e0 con 0x7fa4e80c4430 2023-10-30T14:26:41.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4e7fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa4e8159830 0x7fa4e8159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4e6ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4e80c4430 0x7fa4e80ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4e77fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa4e8153c60 0x7fa4e80cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:54702/0 (socket says 172.21.15.46:54702) 2023-10-30T14:26:41.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4e77fe640 1 -- 172.21.15.46:0/2986275354 learned_addr learned my addr 172.21.15.46:0/2986275354 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:41.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4e7fff640 1 -- 172.21.15.46:0/2986275354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4e80c4430 msgr2=0x7fa4e80ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:41.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4e7fff640 1 --2- 172.21.15.46:0/2986275354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4e80c4430 0x7fa4e80ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4e7fff640 1 -- 172.21.15.46:0/2986275354 >> v1:172.21.15.154:6789/0 conn(0x7fa4e8153c60 legacy=0x7fa4e80cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.442+0000 7fa4e7fff640 1 -- 172.21.15.46:0/2986275354 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e815a5c0 con 0x7fa4e8159830 2023-10-30T14:26:41.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4e7fff640 1 --2- 172.21.15.46:0/2986275354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e8159830 0x7fa4e8159ce0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009d90 tx=0x7fa4e001c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cb39bb9549ca7de server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4e67fc640 1 -- 172.21.15.46:0/2986275354 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e00a5900 con 0x7fa4e8159830 2023-10-30T14:26:41.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4e67fc640 1 -- 172.21.15.46:0/2986275354 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e00a5a60 con 0x7fa4e8159830 2023-10-30T14:26:41.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4e67fc640 1 -- 172.21.15.46:0/2986275354 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e00a5df0 con 0x7fa4e8159830 2023-10-30T14:26:41.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4edd3e640 1 -- 172.21.15.46:0/2986275354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e8159830 msgr2=0x7fa4e8159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4edd3e640 1 --2- 172.21.15.46:0/2986275354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e8159830 0x7fa4e8159ce0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009d90 tx=0x7fa4e001c950 comp rx=0 tx=0).stop 2023-10-30T14:26:41.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4edd3e640 1 -- 172.21.15.46:0/2986275354 shutdown_connections 2023-10-30T14:26:41.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4edd3e640 1 --2- 172.21.15.46:0/2986275354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e8159830 0x7fa4e8159ce0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4edd3e640 1 --2- 172.21.15.46:0/2986275354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4e80c4430 0x7fa4e80ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4edd3e640 1 -- 172.21.15.46:0/2986275354 >> 172.21.15.46:0/2986275354 conn(0x7fa4e806bfe0 msgr2=0x7fa4e806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:41.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4edd3e640 1 -- 172.21.15.46:0/2986275354 shutdown_connections 2023-10-30T14:26:41.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4edd3e640 1 -- 172.21.15.46:0/2986275354 wait complete. 2023-10-30T14:26:41.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4edd3e640 1 Processor -- start 2023-10-30T14:26:41.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.446+0000 7fa4edd3e640 1 -- start start 2023-10-30T14:26:41.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4edd3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4e80c4430 0x7fa4e81f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4edd3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e8153c60 0x7fa4e80cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4edd3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e8159830 0x7fa4e80cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4edd3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e81f64d0 con 0x7fa4e8159830 2023-10-30T14:26:41.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4edd3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e80b46f0 con 0x7fa4e8153c60 2023-10-30T14:26:41.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4edd3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e80b4260 con 0x7fa4e80c4430 2023-10-30T14:26:41.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e8159830 0x7fa4e80cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e8159830 0x7fa4e80cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43652/0 (socket says 172.21.15.46:43652) 2023-10-30T14:26:41.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4e7fff640 1 -- 172.21.15.46:0/3100499097 learned_addr learned my addr 172.21.15.46:0/3100499097 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:41.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4e77fe640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4e80c4430 0x7fa4e81f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4e6ffd640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e8153c60 0x7fa4e80cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4e7fff640 1 -- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4e80c4430 msgr2=0x7fa4e81f57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4e7fff640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4e80c4430 0x7fa4e81f57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4e7fff640 1 -- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e8153c60 msgr2=0x7fa4e80cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4e7fff640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e8153c60 0x7fa4e80cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4e7fff640 1 -- 172.21.15.46:0/3100499097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e0009a70 con 0x7fa4e8159830 2023-10-30T14:26:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4e7fff640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e8159830 0x7fa4e80cdc60 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00b0040 tx=0x7fa4e00196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4cffff640 1 -- 172.21.15.46:0/3100499097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e00b9860 con 0x7fa4e8159830 2023-10-30T14:26:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4cffff640 1 -- 172.21.15.46:0/3100499097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e00a58f0 con 0x7fa4e8159830 2023-10-30T14:26:41.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4edd3e640 1 -- 172.21.15.46:0/3100499097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e81f61f0 con 0x7fa4e8159830 2023-10-30T14:26:41.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4cffff640 1 -- 172.21.15.46:0/3100499097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e00b9c00 con 0x7fa4e8159830 2023-10-30T14:26:41.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4edd3e640 1 -- 172.21.15.46:0/3100499097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e80b3ef0 con 0x7fa4e8159830 2023-10-30T14:26:41.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.450+0000 7fa4edd3e640 1 -- 172.21.15.46:0/3100499097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e80b3a70 con 0x7fa4e8159830 2023-10-30T14:26:41.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.454+0000 7fa4cffff640 1 -- 172.21.15.46:0/3100499097 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4e00c4460 con 0x7fa4e8159830 2023-10-30T14:26:41.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.454+0000 7fa4cffff640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4d40664a0 0x7fa4d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.454+0000 7fa4cffff640 1 -- 172.21.15.46:0/3100499097 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fa4e011fce0 con 0x7fa4e8159830 2023-10-30T14:26:41.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.454+0000 7fa4e77fe640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4d40664a0 0x7fa4d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.454+0000 7fa4e77fe640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4d40664a0 0x7fa4d4068960 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8005890 tx=0x7fa4d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.458+0000 7fa4cffff640 1 -- 172.21.15.46:0/3100499097 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4e80b3a70 con 0x7fa4e8159830 2023-10-30T14:26:41.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.610+0000 7fa4edd3e640 1 -- 172.21.15.46:0/3100499097 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4e80cbf90 con 0x7fa4d40664a0 2023-10-30T14:26:41.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4cffff640 1 -- 172.21.15.46:0/3100499097 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa4d8001510 con 0x7fa4d40664a0 2023-10-30T14:26:41.616 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 2 to deep-scrub 2023-10-30T14:26:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4edd3e640 1 -- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4d40664a0 msgr2=0x7fa4d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4edd3e640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4d40664a0 0x7fa4d4068960 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8005890 tx=0x7fa4d8009040 comp rx=0 tx=0).stop 2023-10-30T14:26:41.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4edd3e640 1 -- 172.21.15.46:0/3100499097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e8159830 msgr2=0x7fa4e80cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4edd3e640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e8159830 0x7fa4e80cdc60 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00b0040 tx=0x7fa4e00196f0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4edd3e640 1 -- 172.21.15.46:0/3100499097 shutdown_connections 2023-10-30T14:26:41.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4edd3e640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa4d40664a0 0x7fa4d4068960 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4edd3e640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e8159830 0x7fa4e80cdc60 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4edd3e640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e8153c60 0x7fa4e80cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4edd3e640 1 --2- 172.21.15.46:0/3100499097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4e80c4430 0x7fa4e81f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4edd3e640 1 -- 172.21.15.46:0/3100499097 >> 172.21.15.46:0/3100499097 conn(0x7fa4e806bfe0 msgr2=0x7fa4e8151c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:41.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4edd3e640 1 -- 172.21.15.46:0/3100499097 shutdown_connections 2023-10-30T14:26:41.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.614+0000 7fa4edd3e640 1 -- 172.21.15.46:0/3100499097 wait complete. 2023-10-30T14:26:41.638 INFO:tasks.ceph:Scrubbing osd.3 2023-10-30T14:26:41.638 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:26:41.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02b5046640 1 Processor -- start 2023-10-30T14:26:41.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02b5046640 1 -- start start 2023-10-30T14:26:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02b5046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b014cf40 0x7f02b0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02b5046640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f02b00c8430 0x7f02b00c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02b5046640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b015a270 con 0x7f02b015a700 2023-10-30T14:26:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02b5046640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b006c7c0 con 0x7f02b00c8430 2023-10-30T14:26:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02b5046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b006a3d0 con 0x7f02b014cf40 2023-10-30T14:26:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02aed9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b014cf40 0x7f02b0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02ae59c640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f02b015a700 0x7f02b0156490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:54706/0 (socket says 172.21.15.46:54706) 2023-10-30T14:26:41.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02ae59c640 1 -- 172.21.15.46:0/3963974858 learned_addr learned my addr 172.21.15.46:0/3963974858 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:41.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02af59e640 1 --2- 172.21.15.46:0/3963974858 >> v2:172.21.15.154:3300/0 conn(0x7f02b00c8430 0x7f02b00c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02add9b640 1 -- 172.21.15.46:0/3963974858 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504991392 0 0) 0x7f02b015a270 con 0x7f02b015a700 2023-10-30T14:26:41.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02add9b640 1 -- 172.21.15.46:0/3963974858 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0298005740 con 0x7f02b015a700 2023-10-30T14:26:41.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02aed9d640 1 -- 172.21.15.46:0/3963974858 >> v2:172.21.15.154:3300/0 conn(0x7f02b00c8430 msgr2=0x7f02b00c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:41.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02aed9d640 1 --2- 172.21.15.46:0/3963974858 >> v2:172.21.15.154:3300/0 conn(0x7f02b00c8430 0x7f02b00c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02aed9d640 1 -- 172.21.15.46:0/3963974858 >> v1:172.21.15.154:6789/0 conn(0x7f02b015a700 legacy=0x7f02b0156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.782+0000 7f02aed9d640 1 -- 172.21.15.46:0/3963974858 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b0156ed0 con 0x7f02b014cf40 2023-10-30T14:26:41.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.786+0000 7f02aed9d640 1 --2- 172.21.15.46:0/3963974858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b014cf40 0x7f02b0069e90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f02a00097c0 tx=0x7f02a001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7334e66cdf3620ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.786+0000 7f02add9b640 1 -- 172.21.15.46:0/3963974858 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a00ab600 con 0x7f02b014cf40 2023-10-30T14:26:41.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.786+0000 7f02add9b640 1 -- 172.21.15.46:0/3963974858 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a00a7050 con 0x7f02b014cf40 2023-10-30T14:26:41.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.786+0000 7f02add9b640 1 -- 172.21.15.46:0/3963974858 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a00ab960 con 0x7f02b014cf40 2023-10-30T14:26:41.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 -- 172.21.15.46:0/3963974858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b014cf40 msgr2=0x7f02b0069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 --2- 172.21.15.46:0/3963974858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b014cf40 0x7f02b0069e90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f02a00097c0 tx=0x7f02a001c480 comp rx=0 tx=0).stop 2023-10-30T14:26:41.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 -- 172.21.15.46:0/3963974858 shutdown_connections 2023-10-30T14:26:41.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 --2- 172.21.15.46:0/3963974858 >> v2:172.21.15.154:3300/0 conn(0x7f02b00c8430 0x7f02b00c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 --2- 172.21.15.46:0/3963974858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b014cf40 0x7f02b0069e90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 -- 172.21.15.46:0/3963974858 >> 172.21.15.46:0/3963974858 conn(0x7f02b00c7910 msgr2=0x7f02b00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:41.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 -- 172.21.15.46:0/3963974858 shutdown_connections 2023-10-30T14:26:41.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 -- 172.21.15.46:0/3963974858 wait complete. 2023-10-30T14:26:41.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 Processor -- start 2023-10-30T14:26:41.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 -- start start 2023-10-30T14:26:41.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b00c8430 0x7f02b01530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02b014cf40 0x7f02b0146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b015a700 0x7f02b0148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0155d20 con 0x7f02b015a700 2023-10-30T14:26:41.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0152240 con 0x7f02b014cf40 2023-10-30T14:26:41.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.790+0000 7f02b5046640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01553f0 con 0x7f02b00c8430 2023-10-30T14:26:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02af59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b015a700 0x7f02b0148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02aed9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b00c8430 0x7f02b01530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02af59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b015a700 0x7f02b0148e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43654/0 (socket says 172.21.15.46:43654) 2023-10-30T14:26:41.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02af59e640 1 -- 172.21.15.46:0/3045177465 learned_addr learned my addr 172.21.15.46:0/3045177465 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:41.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02ae59c640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02b014cf40 0x7f02b0146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02af59e640 1 -- 172.21.15.46:0/3045177465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b00c8430 msgr2=0x7f02b01530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02af59e640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02b00c8430 0x7f02b01530e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02af59e640 1 -- 172.21.15.46:0/3045177465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02b014cf40 msgr2=0x7f02b0146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02af59e640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02b014cf40 0x7f02b0146960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02af59e640 1 -- 172.21.15.46:0/3045177465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a0009420 con 0x7f02b015a700 2023-10-30T14:26:41.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02af59e640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b015a700 0x7f02b0148e90 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f02a800efc0 tx=0x7f02a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02977fe640 1 -- 172.21.15.46:0/3045177465 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8009b20 con 0x7f02b015a700 2023-10-30T14:26:41.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02977fe640 1 -- 172.21.15.46:0/3045177465 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a800f040 con 0x7f02b015a700 2023-10-30T14:26:41.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02977fe640 1 -- 172.21.15.46:0/3045177465 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a80146c0 con 0x7f02b015a700 2023-10-30T14:26:41.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02b5046640 1 -- 172.21.15.46:0/3045177465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b01556d0 con 0x7f02b015a700 2023-10-30T14:26:41.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02b5046640 1 -- 172.21.15.46:0/3045177465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b0152520 con 0x7f02b015a700 2023-10-30T14:26:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02b5046640 1 -- 172.21.15.46:0/3045177465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f02b014d320 con 0x7f02b015a700 2023-10-30T14:26:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.794+0000 7f02977fe640 1 -- 172.21.15.46:0/3045177465 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02a8014820 con 0x7f02b015a700 2023-10-30T14:26:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.798+0000 7f02977fe640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f029c0664a0 0x7f029c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.798+0000 7f02977fe640 1 -- 172.21.15.46:0/3045177465 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f02a8085800 con 0x7f02b015a700 2023-10-30T14:26:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.798+0000 7f02977fe640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] conn(0x7f029c070480 0x7f029c072920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.798+0000 7f02977fe640 1 -- 172.21.15.46:0/3045177465 --> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f029c072ff0 con 0x7f029c070480 2023-10-30T14:26:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.798+0000 7f02977fe640 1 -- 172.21.15.46:0/3045177465 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02a8087de0 con 0x7f02b015a700 2023-10-30T14:26:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.798+0000 7f02aed9d640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f029c0664a0 0x7f029c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.798+0000 7f02ae59c640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] conn(0x7f029c070480 0x7f029c072920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.798+0000 7f02ae59c640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] conn(0x7f029c070480 0x7f029c072920 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.798+0000 7f02aed9d640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f029c0664a0 0x7f029c068960 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f02a00ad000 tx=0x7f02a00a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.798+0000 7f02977fe640 1 -- 172.21.15.46:0/3045177465 <== osd.3 v2:172.21.15.46:6824/2013670008 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f029c072ff0 con 0x7f029c070480 2023-10-30T14:26:41.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.810+0000 7f02b5046640 1 -- 172.21.15.46:0/3045177465 --> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f02b0159f00 con 0x7f029c070480 2023-10-30T14:26:41.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.810+0000 7f02977fe640 1 -- 172.21.15.46:0/3045177465 <== osd.3 v2:172.21.15.46:6824/2013670008 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f02a4001520 con 0x7f029c070480 2023-10-30T14:26:41.816 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-10-30T14:26:41.816 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:26:41.817 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-10-30T14:26:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.814+0000 7f02b5046640 1 -- 172.21.15.46:0/3045177465 >> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] conn(0x7f029c070480 msgr2=0x7f029c072920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.814+0000 7f02b5046640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.46:6824/2013670008,v1:172.21.15.46:6825/2013670008] conn(0x7f029c070480 0x7f029c072920 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.814+0000 7f02b5046640 1 -- 172.21.15.46:0/3045177465 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f029c0664a0 msgr2=0x7f029c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.814+0000 7f02b5046640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f029c0664a0 0x7f029c068960 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f02a00ad000 tx=0x7f02a00a6000 comp rx=0 tx=0).stop 2023-10-30T14:26:41.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.814+0000 7f02b5046640 1 -- 172.21.15.46:0/3045177465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b015a700 msgr2=0x7f02b0148e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.814+0000 7f02b5046640 1 --2- 172.21.15.46:0/3045177465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b015a700 0x7f02b0148e90 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f02a800efc0 tx=0x7f02a800c490 comp rx=0 tx=0).stop 2023-10-30T14:26:41.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.814+0000 7f02af59e640 1 -- 172.21.15.46:0/3045177465 reap_dead start 2023-10-30T14:26:41.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.814+0000 7f02b5046640 1 -- 172.21.15.46:0/3045177465 shutdown_connections 2023-10-30T14:26:41.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.814+0000 7f02b5046640 1 -- 172.21.15.46:0/3045177465 >> 172.21.15.46:0/3045177465 conn(0x7f02b00c7910 msgr2=0x7f02b00703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:41.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.814+0000 7f02b5046640 1 -- 172.21.15.46:0/3045177465 shutdown_connections 2023-10-30T14:26:41.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.814+0000 7f02b5046640 1 -- 172.21.15.46:0/3045177465 wait complete. 2023-10-30T14:26:41.835 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-30T14:26:41.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.982+0000 7f42c02ea640 1 Processor -- start 2023-10-30T14:26:41.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.982+0000 7f42c02ea640 1 -- start start 2023-10-30T14:26:41.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.982+0000 7f42c02ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f42b8151910 0x7f42b8158120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.982+0000 7f42c02ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42b806a970 0x7f42b80c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.982+0000 7f42c02ea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b806a4e0 con 0x7f42b806adb0 2023-10-30T14:26:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.982+0000 7f42c02ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b8158660 con 0x7f42b8151910 2023-10-30T14:26:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.982+0000 7f42c02ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80c6c50 con 0x7f42b806a970 2023-10-30T14:26:41.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.986+0000 7f42be086640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f42b8151910 0x7f42b8158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.986+0000 7f42bd885640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42b806a970 0x7f42b80c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.986+0000 7f42be887640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f42b806adb0 0x7f42b80c5d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:54720/0 (socket says 172.21.15.46:54720) 2023-10-30T14:26:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.986+0000 7f42be887640 1 -- 172.21.15.46:0/3924780845 learned_addr learned my addr 172.21.15.46:0/3924780845 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.986+0000 7f42be086640 1 -- 172.21.15.46:0/3924780845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42b806a970 msgr2=0x7f42b80c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:41.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.986+0000 7f42be086640 1 --2- 172.21.15.46:0/3924780845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42b806a970 0x7f42b80c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.986+0000 7f42be086640 1 -- 172.21.15.46:0/3924780845 >> v1:172.21.15.154:6789/0 conn(0x7f42b806adb0 legacy=0x7f42b80c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.986+0000 7f42be086640 1 -- 172.21.15.46:0/3924780845 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b80c6f30 con 0x7f42b8151910 2023-10-30T14:26:41.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.990+0000 7f42be086640 1 --2- 172.21.15.46:0/3924780845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42b8151910 0x7f42b8158120 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f42b40097c0 tx=0x7f42b401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be98e23453f5115f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.990+0000 7f42bd084640 1 -- 172.21.15.46:0/3924780845 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b40a8970 con 0x7f42b8151910 2023-10-30T14:26:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.990+0000 7f42bd084640 1 -- 172.21.15.46:0/3924780845 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b40a8ad0 con 0x7f42b8151910 2023-10-30T14:26:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.990+0000 7f42bd084640 1 -- 172.21.15.46:0/3924780845 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b40a8e60 con 0x7f42b8151910 2023-10-30T14:26:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.990+0000 7f42c02ea640 1 -- 172.21.15.46:0/3924780845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42b8151910 msgr2=0x7f42b8158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.990+0000 7f42c02ea640 1 --2- 172.21.15.46:0/3924780845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42b8151910 0x7f42b8158120 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f42b40097c0 tx=0x7f42b401ec70 comp rx=0 tx=0).stop 2023-10-30T14:26:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.990+0000 7f42c02ea640 1 -- 172.21.15.46:0/3924780845 shutdown_connections 2023-10-30T14:26:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.990+0000 7f42c02ea640 1 --2- 172.21.15.46:0/3924780845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42b806a970 0x7f42b80c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.990+0000 7f42c02ea640 1 --2- 172.21.15.46:0/3924780845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42b8151910 0x7f42b8158120 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.990+0000 7f42c02ea640 1 -- 172.21.15.46:0/3924780845 >> 172.21.15.46:0/3924780845 conn(0x7f42b80c7910 msgr2=0x7f42b80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.990+0000 7f42c02ea640 1 -- 172.21.15.46:0/3924780845 shutdown_connections 2023-10-30T14:26:41.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.990+0000 7f42c02ea640 1 -- 172.21.15.46:0/3924780845 wait complete. 2023-10-30T14:26:42.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42c02ea640 1 Processor -- start 2023-10-30T14:26:42.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42c02ea640 1 -- start start 2023-10-30T14:26:42.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42c02ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42b806a970 0x7f42b8156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42c02ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42b806adb0 0x7f42b80c3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42c02ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42b8151910 0x7f42b8146160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42c02ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b8157430 con 0x7f42b806adb0 2023-10-30T14:26:42.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42c02ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80c3c70 con 0x7f42b8151910 2023-10-30T14:26:42.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42c02ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80c3990 con 0x7f42b806a970 2023-10-30T14:26:42.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42bd885640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42b806adb0 0x7f42b80c3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42be086640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42b806a970 0x7f42b8156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42bd885640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42b806adb0 0x7f42b80c3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43666/0 (socket says 172.21.15.46:43666) 2023-10-30T14:26:42.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42bd885640 1 -- 172.21.15.46:0/1426581556 learned_addr learned my addr 172.21.15.46:0/1426581556 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:42.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42be887640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42b8151910 0x7f42b8146160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42bd885640 1 -- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42b806a970 msgr2=0x7f42b8156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42bd885640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42b806a970 0x7f42b8156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42bd885640 1 -- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42b8151910 msgr2=0x7f42b8146160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42bd885640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42b8151910 0x7f42b8146160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42bd885640 1 -- 172.21.15.46:0/1426581556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b4009420 con 0x7f42b806adb0 2023-10-30T14:26:42.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.994+0000 7f42bd885640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42b806adb0 0x7f42b80c3170 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f42a800e870 tx=0x7f42a800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.998+0000 7f42aeffd640 1 -- 172.21.15.46:0/1426581556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8009510 con 0x7f42b806adb0 2023-10-30T14:26:42.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.998+0000 7f42aeffd640 1 -- 172.21.15.46:0/1426581556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a800eec0 con 0x7f42b806adb0 2023-10-30T14:26:42.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.998+0000 7f42aeffd640 1 -- 172.21.15.46:0/1426581556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a800cb40 con 0x7f42b806adb0 2023-10-30T14:26:42.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.998+0000 7f42c02ea640 1 -- 172.21.15.46:0/1426581556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b81555b0 con 0x7f42b806adb0 2023-10-30T14:26:42.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.998+0000 7f42c02ea640 1 -- 172.21.15.46:0/1426581556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b8155b70 con 0x7f42b806adb0 2023-10-30T14:26:42.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.998+0000 7f42c02ea640 1 -- 172.21.15.46:0/1426581556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b804e170 con 0x7f42b806adb0 2023-10-30T14:26:42.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.998+0000 7f42aeffd640 1 -- 172.21.15.46:0/1426581556 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f42a800cca0 con 0x7f42b806adb0 2023-10-30T14:26:42.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.998+0000 7f42aeffd640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f429c0664d0 0x7f429c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.998+0000 7f42aeffd640 1 -- 172.21.15.46:0/1426581556 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f42a8084b00 con 0x7f42b806adb0 2023-10-30T14:26:42.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:41.998+0000 7f42be086640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f429c0664d0 0x7f429c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.002+0000 7f42be086640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f429c0664d0 0x7f429c068990 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f42b40b2000 tx=0x7f42b40aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.002+0000 7f42aeffd640 1 -- 172.21.15.46:0/1426581556 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f42a8051420 con 0x7f42b806adb0 2023-10-30T14:26:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 -- 172.21.15.46:0/1426581556 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f42b8069e90 con 0x7f429c0664d0 2023-10-30T14:26:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42aeffd640 1 -- 172.21.15.46:0/1426581556 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f42b40c0090 con 0x7f429c0664d0 2023-10-30T14:26:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 3 to deep-scrub 2023-10-30T14:26:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 -- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f429c0664d0 msgr2=0x7f429c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f429c0664d0 0x7f429c068990 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f42b40b2000 tx=0x7f42b40aa070 comp rx=0 tx=0).stop 2023-10-30T14:26:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 -- 172.21.15.46:0/1426581556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42b806adb0 msgr2=0x7f42b80c3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42b806adb0 0x7f42b80c3170 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f42a800e870 tx=0x7f42a800ec70 comp rx=0 tx=0).stop 2023-10-30T14:26:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 -- 172.21.15.46:0/1426581556 shutdown_connections 2023-10-30T14:26:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f429c0664d0 0x7f429c068990 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42b8151910 0x7f42b8146160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42b806adb0 0x7f42b80c3170 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 --2- 172.21.15.46:0/1426581556 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f42b806a970 0x7f42b8156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 -- 172.21.15.46:0/1426581556 >> 172.21.15.46:0/1426581556 conn(0x7f42b80c7910 msgr2=0x7f42b80c2800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:42.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 -- 172.21.15.46:0/1426581556 shutdown_connections 2023-10-30T14:26:42.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.162+0000 7f42c02ea640 1 -- 172.21.15.46:0/1426581556 wait complete. 2023-10-30T14:26:42.187 INFO:tasks.ceph:Scrubbing osd.4 2023-10-30T14:26:42.187 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:26:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.330+0000 7fbf4050f640 1 Processor -- start 2023-10-30T14:26:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.330+0000 7fbf4050f640 1 -- start start 2023-10-30T14:26:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.330+0000 7fbf4050f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf381517f0 0x7fbf38157620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.330+0000 7fbf4050f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf380c6490 0x7fbf380c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.330+0000 7fbf4050f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3806c810 con 0x7fbf38158b10 2023-10-30T14:26:42.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.330+0000 7fbf4050f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3815bbf0 con 0x7fbf381517f0 2023-10-30T14:26:42.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.330+0000 7fbf4050f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf38071ed0 con 0x7fbf380c6490 2023-10-30T14:26:42.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3e2ab640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf381517f0 0x7fbf38157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3e2ab640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbf381517f0 0x7fbf38157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43680/0 (socket says 172.21.15.46:43680) 2023-10-30T14:26:42.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3e2ab640 1 -- 172.21.15.46:0/1482058987 learned_addr learned my addr 172.21.15.46:0/1482058987 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:42.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3daaa640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbf38158b10 0x7fbf3815ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38092/0 (socket says 172.21.15.46:38092) 2023-10-30T14:26:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3eaac640 1 --2- 172.21.15.46:0/1482058987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf380c6490 0x7fbf380c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3e2ab640 1 -- 172.21.15.46:0/1482058987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf380c6490 msgr2=0x7fbf380c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3e2ab640 1 --2- 172.21.15.46:0/1482058987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf380c6490 0x7fbf380c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3e2ab640 1 -- 172.21.15.46:0/1482058987 >> v1:172.21.15.46:6789/0 conn(0x7fbf38158b10 legacy=0x7fbf3815ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3e2ab640 1 -- 172.21.15.46:0/1482058987 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3815b630 con 0x7fbf381517f0 2023-10-30T14:26:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3d2a9640 1 -- 172.21.15.46:0/1482058987 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858775032 0 0) 0x7fbf3806c810 con 0x7fbf38158b10 2023-10-30T14:26:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3e2ab640 1 --2- 172.21.15.46:0/1482058987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf381517f0 0x7fbf38157620 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009870 tx=0x7fbf3401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b669a2e8c94b11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3d2a9640 1 -- 172.21.15.46:0/1482058987 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf340ab620 con 0x7fbf381517f0 2023-10-30T14:26:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3d2a9640 1 -- 172.21.15.46:0/1482058987 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf340ab780 con 0x7fbf381517f0 2023-10-30T14:26:42.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.334+0000 7fbf3d2a9640 1 -- 172.21.15.46:0/1482058987 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf340abb10 con 0x7fbf381517f0 2023-10-30T14:26:42.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 -- 172.21.15.46:0/1482058987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf381517f0 msgr2=0x7fbf38157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 --2- 172.21.15.46:0/1482058987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf381517f0 0x7fbf38157620 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009870 tx=0x7fbf3401c510 comp rx=0 tx=0).stop 2023-10-30T14:26:42.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 -- 172.21.15.46:0/1482058987 shutdown_connections 2023-10-30T14:26:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 --2- 172.21.15.46:0/1482058987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf380c6490 0x7fbf380c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 --2- 172.21.15.46:0/1482058987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf381517f0 0x7fbf38157620 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 -- 172.21.15.46:0/1482058987 >> 172.21.15.46:0/1482058987 conn(0x7fbf380c7910 msgr2=0x7fbf380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:42.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 -- 172.21.15.46:0/1482058987 shutdown_connections 2023-10-30T14:26:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 -- 172.21.15.46:0/1482058987 wait complete. 2023-10-30T14:26:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 Processor -- start 2023-10-30T14:26:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 -- start start 2023-10-30T14:26:42.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf380c6490 0x7fbf380c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf381517f0 0x7fbf381f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf38158b10 0x7fbf38147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf38147360 con 0x7fbf381517f0 2023-10-30T14:26:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380c46f0 con 0x7fbf380c6490 2023-10-30T14:26:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf4050f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf38161d10 con 0x7fbf38158b10 2023-10-30T14:26:42.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf3daaa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf381517f0 0x7fbf381f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf3daaa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf381517f0 0x7fbf381f57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43696/0 (socket says 172.21.15.46:43696) 2023-10-30T14:26:42.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf3daaa640 1 -- 172.21.15.46:0/2351370698 learned_addr learned my addr 172.21.15.46:0/2351370698 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:42.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf3e2ab640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf380c6490 0x7fbf380c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf3eaac640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf38158b10 0x7fbf38147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf3daaa640 1 -- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf38158b10 msgr2=0x7fbf38147840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf3daaa640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbf38158b10 0x7fbf38147840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.351 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf3daaa640 1 -- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf380c6490 msgr2=0x7fbf380c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf3daaa640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf380c6490 0x7fbf380c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf3daaa640 1 -- 172.21.15.46:0/2351370698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf340094d0 con 0x7fbf381517f0 2023-10-30T14:26:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.338+0000 7fbf3e2ab640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf380c6490 0x7fbf380c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf3daaa640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf381517f0 0x7fbf381f57c0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800c480 tx=0x7fbf2800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf2effd640 1 -- 172.21.15.46:0/2351370698 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf280074e0 con 0x7fbf381517f0 2023-10-30T14:26:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf2effd640 1 -- 172.21.15.46:0/2351370698 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf28007660 con 0x7fbf381517f0 2023-10-30T14:26:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf2effd640 1 -- 172.21.15.46:0/2351370698 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf28004a30 con 0x7fbf381517f0 2023-10-30T14:26:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf4050f640 1 -- 172.21.15.46:0/2351370698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf381f6100 con 0x7fbf381517f0 2023-10-30T14:26:42.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf4050f640 1 -- 172.21.15.46:0/2351370698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf380c31c0 con 0x7fbf381517f0 2023-10-30T14:26:42.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf4050f640 1 -- 172.21.15.46:0/2351370698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbf380c5f70 con 0x7fbf381517f0 2023-10-30T14:26:42.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf2effd640 1 -- 172.21.15.46:0/2351370698 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbf28004b90 con 0x7fbf381517f0 2023-10-30T14:26:42.354 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf2effd640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf1c0664d0 0x7fbf1c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf2effd640 1 -- 172.21.15.46:0/2351370698 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fbf28083b90 con 0x7fbf381517f0 2023-10-30T14:26:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf2effd640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] conn(0x7fbf1c0704b0 0x7fbf1c072950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf3e2ab640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf1c0664d0 0x7fbf1c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf2effd640 1 -- 172.21.15.46:0/2351370698 --> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf1c073020 con 0x7fbf1c0704b0 2023-10-30T14:26:42.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf2effd640 1 -- 172.21.15.46:0/2351370698 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf280882a0 con 0x7fbf381517f0 2023-10-30T14:26:42.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.342+0000 7fbf3eaac640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] conn(0x7fbf1c0704b0 0x7fbf1c072950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.346+0000 7fbf3e2ab640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf1c0664d0 0x7fbf1c068990 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fbf340af000 tx=0x7fbf340a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.346+0000 7fbf3eaac640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] conn(0x7fbf1c0704b0 0x7fbf1c072950 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.346+0000 7fbf2effd640 1 -- 172.21.15.46:0/2351370698 <== osd.4 v2:172.21.15.154:6800/1011303130 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fbf1c073020 con 0x7fbf1c0704b0 2023-10-30T14:26:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.358+0000 7fbf4050f640 1 -- 172.21.15.46:0/2351370698 --> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbf380c5c90 con 0x7fbf1c0704b0 2023-10-30T14:26:42.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.362+0000 7fbf2effd640 1 -- 172.21.15.46:0/2351370698 <== osd.4 v2:172.21.15.154:6800/1011303130 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fbf30009b70 con 0x7fbf1c0704b0 2023-10-30T14:26:42.364 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-10-30T14:26:42.365 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:26:42.365 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-10-30T14:26:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.362+0000 7fbf4050f640 1 -- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] conn(0x7fbf1c0704b0 msgr2=0x7fbf1c072950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.362+0000 7fbf4050f640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:6800/1011303130,v1:172.21.15.154:6802/1011303130] conn(0x7fbf1c0704b0 0x7fbf1c072950 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.362+0000 7fbf4050f640 1 -- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf1c0664d0 msgr2=0x7fbf1c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.362+0000 7fbf4050f640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fbf1c0664d0 0x7fbf1c068990 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fbf340af000 tx=0x7fbf340a7000 comp rx=0 tx=0).stop 2023-10-30T14:26:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.362+0000 7fbf4050f640 1 -- 172.21.15.46:0/2351370698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf381517f0 msgr2=0x7fbf381f57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.362+0000 7fbf4050f640 1 --2- 172.21.15.46:0/2351370698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf381517f0 0x7fbf381f57c0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800c480 tx=0x7fbf2800c940 comp rx=0 tx=0).stop 2023-10-30T14:26:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.362+0000 7fbf3eaac640 1 -- 172.21.15.46:0/2351370698 reap_dead start 2023-10-30T14:26:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.362+0000 7fbf4050f640 1 -- 172.21.15.46:0/2351370698 shutdown_connections 2023-10-30T14:26:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.362+0000 7fbf4050f640 1 -- 172.21.15.46:0/2351370698 >> 172.21.15.46:0/2351370698 conn(0x7fbf380c7910 msgr2=0x7fbf3806faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.362+0000 7fbf4050f640 1 -- 172.21.15.46:0/2351370698 shutdown_connections 2023-10-30T14:26:42.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.362+0000 7fbf4050f640 1 -- 172.21.15.46:0/2351370698 wait complete. 2023-10-30T14:26:42.383 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-30T14:26:42.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f5014d3e640 1 Processor -- start 2023-10-30T14:26:42.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f5014d3e640 1 -- start start 2023-10-30T14:26:42.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f5014d3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5010153c60 0x7f50100ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f5014d3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f50100c4430 0x7f50100cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f5014d3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50100cb780 con 0x7f5010159830 2023-10-30T14:26:42.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f5014d3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5010154040 con 0x7f5010153c60 2023-10-30T14:26:42.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f5014d3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50100cc9f0 con 0x7f50100c4430 2023-10-30T14:26:42.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f500e59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5010153c60 0x7f50100ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f500e59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5010153c60 0x7f50100ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43704/0 (socket says 172.21.15.46:43704) 2023-10-30T14:26:42.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f500ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5010159830 0x7f50100cac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38108/0 (socket says 172.21.15.46:38108) 2023-10-30T14:26:42.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f500e59c640 1 -- 172.21.15.46:0/1097954002 learned_addr learned my addr 172.21.15.46:0/1097954002 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:42.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f500dd9b640 1 --2- 172.21.15.46:0/1097954002 >> v2:172.21.15.154:3300/0 conn(0x7f50100c4430 0x7f50100cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f500e59c640 1 -- 172.21.15.46:0/1097954002 >> v2:172.21.15.154:3300/0 conn(0x7f50100c4430 msgr2=0x7f50100cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:42.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f500e59c640 1 --2- 172.21.15.46:0/1097954002 >> v2:172.21.15.154:3300/0 conn(0x7f50100c4430 0x7f50100cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f500e59c640 1 -- 172.21.15.46:0/1097954002 >> v1:172.21.15.46:6789/0 conn(0x7f5010159830 legacy=0x7f50100cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.526+0000 7f500e59c640 1 -- 172.21.15.46:0/1097954002 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501015a2e0 con 0x7f5010153c60 2023-10-30T14:26:42.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.530+0000 7f500e59c640 1 --2- 172.21.15.46:0/1097954002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5010153c60 0x7f50100ca360 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f50100b1950 tx=0x7f500001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a81d51f39ccbf7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.530+0000 7f500d59a640 1 -- 172.21.15.46:0/1097954002 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50000ab600 con 0x7f5010153c60 2023-10-30T14:26:42.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.530+0000 7f500d59a640 1 -- 172.21.15.46:0/1097954002 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50000a7050 con 0x7f5010153c60 2023-10-30T14:26:42.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.530+0000 7f500d59a640 1 -- 172.21.15.46:0/1097954002 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50000ab980 con 0x7f5010153c60 2023-10-30T14:26:42.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.530+0000 7f5014d3e640 1 -- 172.21.15.46:0/1097954002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5010153c60 msgr2=0x7f50100ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.530+0000 7f5014d3e640 1 --2- 172.21.15.46:0/1097954002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5010153c60 0x7f50100ca360 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f50100b1950 tx=0x7f500001c500 comp rx=0 tx=0).stop 2023-10-30T14:26:42.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.530+0000 7f5014d3e640 1 -- 172.21.15.46:0/1097954002 shutdown_connections 2023-10-30T14:26:42.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.530+0000 7f5014d3e640 1 --2- 172.21.15.46:0/1097954002 >> v2:172.21.15.154:3300/0 conn(0x7f50100c4430 0x7f50100cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.530+0000 7f5014d3e640 1 --2- 172.21.15.46:0/1097954002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5010153c60 0x7f50100ca360 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.530+0000 7f5014d3e640 1 -- 172.21.15.46:0/1097954002 >> 172.21.15.46:0/1097954002 conn(0x7f501006bfe0 msgr2=0x7f501006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:42.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.530+0000 7f5014d3e640 1 -- 172.21.15.46:0/1097954002 shutdown_connections 2023-10-30T14:26:42.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.530+0000 7f5014d3e640 1 -- 172.21.15.46:0/1097954002 wait complete. 2023-10-30T14:26:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f5014d3e640 1 Processor -- start 2023-10-30T14:26:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f5014d3e640 1 -- start start 2023-10-30T14:26:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f5014d3e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50100c4430 0x7f50100cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f5014d3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5010153c60 0x7f501014a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f5014d3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5010159830 0x7f50101f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f5014d3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501014d6e0 con 0x7f5010159830 2023-10-30T14:26:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f5014d3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501014b360 con 0x7f5010153c60 2023-10-30T14:26:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f5014d3e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501014cd40 con 0x7f50100c4430 2023-10-30T14:26:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f500ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5010159830 0x7f50101f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f500ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5010159830 0x7f50101f5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43708/0 (socket says 172.21.15.46:43708) 2023-10-30T14:26:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f500ed9d640 1 -- 172.21.15.46:0/2511954982 learned_addr learned my addr 172.21.15.46:0/2511954982 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f500dd9b640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5010153c60 0x7f501014a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f500e59c640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50100c4430 0x7f50100cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f500ed9d640 1 -- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50100c4430 msgr2=0x7f50100cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f500ed9d640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50100c4430 0x7f50100cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f500ed9d640 1 -- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5010153c60 msgr2=0x7f501014a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f500ed9d640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5010153c60 0x7f501014a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f500ed9d640 1 -- 172.21.15.46:0/2511954982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50000094d0 con 0x7f5010159830 2023-10-30T14:26:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f500ed9d640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5010159830 0x7f50101f5850 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f500800c700 tx=0x7f500800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f4ffeffd640 1 -- 172.21.15.46:0/2511954982 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500800ce10 con 0x7f5010159830 2023-10-30T14:26:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f4ffeffd640 1 -- 172.21.15.46:0/2511954982 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500800a640 con 0x7f5010159830 2023-10-30T14:26:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f5014d3e640 1 -- 172.21.15.46:0/2511954982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501014d020 con 0x7f5010159830 2023-10-30T14:26:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f4ffeffd640 1 -- 172.21.15.46:0/2511954982 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500800a9b0 con 0x7f5010159830 2023-10-30T14:26:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.534+0000 7f5014d3e640 1 -- 172.21.15.46:0/2511954982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501014b640 con 0x7f5010159830 2023-10-30T14:26:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.538+0000 7f5014d3e640 1 -- 172.21.15.46:0/2511954982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501014d820 con 0x7f5010159830 2023-10-30T14:26:42.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.538+0000 7f4ffeffd640 1 -- 172.21.15.46:0/2511954982 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f500800ab10 con 0x7f5010159830 2023-10-30T14:26:42.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.538+0000 7f4ffeffd640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4ff00664a0 0x7f4ff0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.538+0000 7f4ffeffd640 1 -- 172.21.15.46:0/2511954982 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f5008084160 con 0x7f5010159830 2023-10-30T14:26:42.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.538+0000 7f500e59c640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4ff00664a0 0x7f4ff0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.542+0000 7f500e59c640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4ff00664a0 0x7f4ff0068960 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f50000ad000 tx=0x7f5000008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.542+0000 7f4ffeffd640 1 -- 172.21.15.46:0/2511954982 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f501014d820 con 0x7f5010159830 2023-10-30T14:26:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.714+0000 7f5014d3e640 1 -- 172.21.15.46:0/2511954982 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f50100cbf90 con 0x7f4ff00664a0 2023-10-30T14:26:42.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.714+0000 7f4ffeffd640 1 -- 172.21.15.46:0/2511954982 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f50000ab6e0 con 0x7f4ff00664a0 2023-10-30T14:26:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 4 to deep-scrub 2023-10-30T14:26:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.718+0000 7f5014d3e640 1 -- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4ff00664a0 msgr2=0x7f4ff0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.718+0000 7f5014d3e640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4ff00664a0 0x7f4ff0068960 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f50000ad000 tx=0x7f5000008040 comp rx=0 tx=0).stop 2023-10-30T14:26:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.718+0000 7f5014d3e640 1 -- 172.21.15.46:0/2511954982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5010159830 msgr2=0x7f50101f5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.718+0000 7f5014d3e640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5010159830 0x7f50101f5850 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f500800c700 tx=0x7f500800cbc0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.718+0000 7f5014d3e640 1 -- 172.21.15.46:0/2511954982 shutdown_connections 2023-10-30T14:26:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.718+0000 7f5014d3e640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f4ff00664a0 0x7f4ff0068960 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.718+0000 7f5014d3e640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5010159830 0x7f50101f5850 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.718+0000 7f5014d3e640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5010153c60 0x7f501014a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.718+0000 7f5014d3e640 1 --2- 172.21.15.46:0/2511954982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50100c4430 0x7f50100cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.718+0000 7f5014d3e640 1 -- 172.21.15.46:0/2511954982 >> 172.21.15.46:0/2511954982 conn(0x7f501006bfe0 msgr2=0x7f5010151e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.718+0000 7f5014d3e640 1 -- 172.21.15.46:0/2511954982 shutdown_connections 2023-10-30T14:26:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.718+0000 7f5014d3e640 1 -- 172.21.15.46:0/2511954982 wait complete. 2023-10-30T14:26:42.740 INFO:tasks.ceph:Scrubbing osd.5 2023-10-30T14:26:42.741 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:26:42.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.894+0000 7f63f0780640 1 Processor -- start 2023-10-30T14:26:42.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.894+0000 7f63f0780640 1 -- start start 2023-10-30T14:26:42.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.894+0000 7f63f0780640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f63e815d9d0 0x7f63e80cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.894+0000 7f63f0780640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e815a210 con 0x7f63e80c4000 2023-10-30T14:26:42.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.894+0000 7f63f0780640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e815a4f0 con 0x7f63e81536d0 2023-10-30T14:26:42.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.894+0000 7f63f0780640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e815cef0 con 0x7f63e815d9d0 2023-10-30T14:26:42.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.898+0000 7f63edd1b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f63e815d9d0 0x7f63e80cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.898+0000 7f63eed1d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f63e80c4000 0x7f63e80cf090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38114/0 (socket says 172.21.15.46:38114) 2023-10-30T14:26:42.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.898+0000 7f63eed1d640 1 -- 172.21.15.46:0/689841852 learned_addr learned my addr 172.21.15.46:0/689841852 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.898+0000 7f63edd1b640 1 -- 172.21.15.46:0/689841852 >> v1:172.21.15.154:6789/0 conn(0x7f63e81536d0 legacy=0x7f63e8159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.898+0000 7f63edd1b640 1 -- 172.21.15.46:0/689841852 >> v1:172.21.15.46:6789/0 conn(0x7f63e80c4000 legacy=0x7f63e80cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.898+0000 7f63edd1b640 1 -- 172.21.15.46:0/689841852 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e815d1d0 con 0x7f63e815d9d0 2023-10-30T14:26:42.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.898+0000 7f63ed51a640 1 -- 172.21.15.46:0/689841852 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869745699 0 0) 0x7f63e815a210 con 0x7f63e80c4000 2023-10-30T14:26:42.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.898+0000 7f63edd1b640 1 --2- 172.21.15.46:0/689841852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e815d9d0 0x7f63e80cb3c0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f63d80097c0 tx=0x7f63d801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682a6b8a514c66fb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.898+0000 7f63ed51a640 1 -- 172.21.15.46:0/689841852 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d80a8050 con 0x7f63e815d9d0 2023-10-30T14:26:42.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.902+0000 7f63ed51a640 1 -- 172.21.15.46:0/689841852 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d80ac5f0 con 0x7f63e815d9d0 2023-10-30T14:26:42.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.902+0000 7f63ed51a640 1 -- 172.21.15.46:0/689841852 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d80ac980 con 0x7f63e815d9d0 2023-10-30T14:26:42.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.902+0000 7f63f0780640 1 -- 172.21.15.46:0/689841852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e815d9d0 msgr2=0x7f63e80cb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.902+0000 7f63f0780640 1 --2- 172.21.15.46:0/689841852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e815d9d0 0x7f63e80cb3c0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f63d80097c0 tx=0x7f63d801c480 comp rx=0 tx=0).stop 2023-10-30T14:26:42.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.902+0000 7f63f0780640 1 -- 172.21.15.46:0/689841852 shutdown_connections 2023-10-30T14:26:42.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.902+0000 7f63f0780640 1 --2- 172.21.15.46:0/689841852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e815d9d0 0x7f63e80cb3c0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.902+0000 7f63f0780640 1 -- 172.21.15.46:0/689841852 >> 172.21.15.46:0/689841852 conn(0x7f63e806bf70 msgr2=0x7f63e806c380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.902+0000 7f63f0780640 1 -- 172.21.15.46:0/689841852 shutdown_connections 2023-10-30T14:26:42.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.902+0000 7f63f0780640 1 -- 172.21.15.46:0/689841852 wait complete. 2023-10-30T14:26:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63f0780640 1 Processor -- start 2023-10-30T14:26:42.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63f0780640 1 -- start start 2023-10-30T14:26:42.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63f0780640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63e80c4000 0x7f63e814c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63f0780640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e81536d0 0x7f63e814c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63f0780640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e815d9d0 0x7f63e815bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63f0780640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e8158c50 con 0x7f63e81536d0 2023-10-30T14:26:42.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63f0780640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e81f6240 con 0x7f63e80c4000 2023-10-30T14:26:42.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63f0780640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e81f6520 con 0x7f63e815d9d0 2023-10-30T14:26:42.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63edd1b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e81536d0 0x7f63e814c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63edd1b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e81536d0 0x7f63e814c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43728/0 (socket says 172.21.15.46:43728) 2023-10-30T14:26:42.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63edd1b640 1 -- 172.21.15.46:0/2814162426 learned_addr learned my addr 172.21.15.46:0/2814162426 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:42.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63ee51c640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63e80c4000 0x7f63e814c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63edd1b640 1 -- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e815d9d0 msgr2=0x7f63e815bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63edd1b640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e815d9d0 0x7f63e815bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63edd1b640 1 -- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63e80c4000 msgr2=0x7f63e814c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63edd1b640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63e80c4000 0x7f63e814c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63edd1b640 1 -- 172.21.15.46:0/2814162426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d8009420 con 0x7f63e81536d0 2023-10-30T14:26:42.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63edd1b640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e81536d0 0x7f63e814c8e0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f63d801a0b0 tx=0x7f63d801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63deffd640 1 -- 172.21.15.46:0/2814162426 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d80b8660 con 0x7f63e81536d0 2023-10-30T14:26:42.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63deffd640 1 -- 172.21.15.46:0/2814162426 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d80b87e0 con 0x7f63e81536d0 2023-10-30T14:26:42.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63deffd640 1 -- 172.21.15.46:0/2814162426 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d80b8b90 con 0x7f63e81536d0 2023-10-30T14:26:42.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63f0780640 1 -- 172.21.15.46:0/2814162426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e8159550 con 0x7f63e81536d0 2023-10-30T14:26:42.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.906+0000 7f63f0780640 1 -- 172.21.15.46:0/2814162426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e80c9120 con 0x7f63e81536d0 2023-10-30T14:26:42.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.910+0000 7f63f0780640 1 -- 172.21.15.46:0/2814162426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f63e8153ab0 con 0x7f63e81536d0 2023-10-30T14:26:42.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.910+0000 7f63deffd640 1 -- 172.21.15.46:0/2814162426 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f63d80b8cf0 con 0x7f63e81536d0 2023-10-30T14:26:42.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.910+0000 7f63deffd640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63cc0664d0 0x7f63cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.910+0000 7f63deffd640 1 -- 172.21.15.46:0/2814162426 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f63d811eee0 con 0x7f63e81536d0 2023-10-30T14:26:42.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.910+0000 7f63ee51c640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63cc0664d0 0x7f63cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.910+0000 7f63deffd640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] conn(0x7f63cc0704b0 0x7f63cc072950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:42.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.910+0000 7f63deffd640 1 -- 172.21.15.46:0/2814162426 --> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63cc073020 con 0x7f63cc0704b0 2023-10-30T14:26:42.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.910+0000 7f63deffd640 1 -- 172.21.15.46:0/2814162426 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63d80a8020 con 0x7f63e81536d0 2023-10-30T14:26:42.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.910+0000 7f63eed1d640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] conn(0x7f63cc0704b0 0x7f63cc072950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:42.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.914+0000 7f63ee51c640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63cc0664d0 0x7f63cc068990 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f63e4007fd0 tx=0x7f63e4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.914+0000 7f63eed1d640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] conn(0x7f63cc0704b0 0x7f63cc072950 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:42.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.914+0000 7f63deffd640 1 -- 172.21.15.46:0/2814162426 <== osd.5 v2:172.21.15.154:6801/956300018 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f63cc073020 con 0x7f63cc0704b0 2023-10-30T14:26:42.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63f0780640 1 -- 172.21.15.46:0/2814162426 --> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f63e804e170 con 0x7f63cc0704b0 2023-10-30T14:26:42.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63deffd640 1 -- 172.21.15.46:0/2814162426 <== osd.5 v2:172.21.15.154:6801/956300018 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f63e804e170 con 0x7f63cc0704b0 2023-10-30T14:26:42.938 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-10-30T14:26:42.939 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:26:42.939 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-10-30T14:26:42.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63f0780640 1 -- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] conn(0x7f63cc0704b0 msgr2=0x7f63cc072950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63f0780640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:6801/956300018,v1:172.21.15.154:6803/956300018] conn(0x7f63cc0704b0 0x7f63cc072950 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63f0780640 1 -- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63cc0664d0 msgr2=0x7f63cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63f0780640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f63cc0664d0 0x7f63cc068990 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f63e4007fd0 tx=0x7f63e4009040 comp rx=0 tx=0).stop 2023-10-30T14:26:42.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63f0780640 1 -- 172.21.15.46:0/2814162426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e81536d0 msgr2=0x7f63e814c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:42.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63f0780640 1 --2- 172.21.15.46:0/2814162426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e81536d0 0x7f63e814c8e0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f63d801a0b0 tx=0x7f63d801acc0 comp rx=0 tx=0).stop 2023-10-30T14:26:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63eed1d640 1 -- 172.21.15.46:0/2814162426 reap_dead start 2023-10-30T14:26:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63f0780640 1 -- 172.21.15.46:0/2814162426 shutdown_connections 2023-10-30T14:26:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63f0780640 1 -- 172.21.15.46:0/2814162426 >> 172.21.15.46:0/2814162426 conn(0x7f63e806bf70 msgr2=0x7f63e8151270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:42.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63f0780640 1 -- 172.21.15.46:0/2814162426 shutdown_connections 2023-10-30T14:26:42.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:42.934+0000 7f63f0780640 1 -- 172.21.15.46:0/2814162426 wait complete. 2023-10-30T14:26:42.959 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-30T14:26:43.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae41009640 1 Processor -- start 2023-10-30T14:26:43.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae41009640 1 -- start start 2023-10-30T14:26:43.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae41009640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fae3c0cb050 0x7fae3c0cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae41009640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae3c0cab00 0x7fae3c0cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae41009640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c0cba30 con 0x7fae3c0c5550 2023-10-30T14:26:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae41009640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae3c0ca290 con 0x7fae3c0cab00 2023-10-30T14:26:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae41009640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae3c0c9cf0 con 0x7fae3c0cb050 2023-10-30T14:26:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae3bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fae3c0c5550 0x7fae3c0cef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38122/0 (socket says 172.21.15.46:38122) 2023-10-30T14:26:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae3bfff640 1 -- 172.21.15.46:0/4055003751 learned_addr learned my addr 172.21.15.46:0/4055003751 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:43.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae40808640 1 --2- 172.21.15.46:0/4055003751 >> v2:172.21.15.46:3300/0 conn(0x7fae3c0cab00 0x7fae3c0cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae3b7fe640 1 --2- 172.21.15.46:0/4055003751 >> v2:172.21.15.154:3300/0 conn(0x7fae3c0cb050 0x7fae3c0cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae3affd640 1 -- 172.21.15.46:0/4055003751 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535703156 0 0) 0x7fae3c0cba30 con 0x7fae3c0c5550 2023-10-30T14:26:43.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae3affd640 1 -- 172.21.15.46:0/4055003751 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae24003530 con 0x7fae3c0c5550 2023-10-30T14:26:43.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae40808640 1 -- 172.21.15.46:0/4055003751 >> v2:172.21.15.154:3300/0 conn(0x7fae3c0cb050 msgr2=0x7fae3c0cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:43.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae40808640 1 --2- 172.21.15.46:0/4055003751 >> v2:172.21.15.154:3300/0 conn(0x7fae3c0cb050 0x7fae3c0cbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae40808640 1 -- 172.21.15.46:0/4055003751 >> v1:172.21.15.46:6789/0 conn(0x7fae3c0c5550 legacy=0x7fae3c0cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.102+0000 7fae40808640 1 -- 172.21.15.46:0/4055003751 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c15d990 con 0x7fae3c0cab00 2023-10-30T14:26:43.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.106+0000 7fae40808640 1 --2- 172.21.15.46:0/4055003751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae3c0cab00 0x7fae3c0cc400 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fae3c0cffe0 tx=0x7fae3401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804a8d8f3bff6779 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.106+0000 7fae3affd640 1 -- 172.21.15.46:0/4055003751 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae340a58c0 con 0x7fae3c0cab00 2023-10-30T14:26:43.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.106+0000 7fae3affd640 1 -- 172.21.15.46:0/4055003751 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae340a5a20 con 0x7fae3c0cab00 2023-10-30T14:26:43.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.106+0000 7fae3affd640 1 -- 172.21.15.46:0/4055003751 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae340a5db0 con 0x7fae3c0cab00 2023-10-30T14:26:43.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.106+0000 7fae41009640 1 -- 172.21.15.46:0/4055003751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae3c0cab00 msgr2=0x7fae3c0cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.106+0000 7fae41009640 1 --2- 172.21.15.46:0/4055003751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae3c0cab00 0x7fae3c0cc400 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fae3c0cffe0 tx=0x7fae3401c910 comp rx=0 tx=0).stop 2023-10-30T14:26:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.106+0000 7fae41009640 1 -- 172.21.15.46:0/4055003751 shutdown_connections 2023-10-30T14:26:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.106+0000 7fae41009640 1 --2- 172.21.15.46:0/4055003751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae3c0cab00 0x7fae3c0cc400 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.106+0000 7fae41009640 1 --2- 172.21.15.46:0/4055003751 >> v2:172.21.15.154:3300/0 conn(0x7fae3c0cb050 0x7fae3c0cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.106+0000 7fae41009640 1 -- 172.21.15.46:0/4055003751 >> 172.21.15.46:0/4055003751 conn(0x7fae3c06bed0 msgr2=0x7fae3c06c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.106+0000 7fae41009640 1 -- 172.21.15.46:0/4055003751 shutdown_connections 2023-10-30T14:26:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.106+0000 7fae41009640 1 -- 172.21.15.46:0/4055003751 wait complete. 2023-10-30T14:26:43.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae41009640 1 Processor -- start 2023-10-30T14:26:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae41009640 1 -- start start 2023-10-30T14:26:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae41009640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae3c0c5550 0x7fae3c1f20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae41009640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0cab00 0x7fae3c159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae41009640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae3c0cb050 0x7fae3c157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae41009640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c06a280 con 0x7fae3c0c5550 2023-10-30T14:26:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae41009640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c1f2630 con 0x7fae3c0cb050 2023-10-30T14:26:43.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae41009640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c0c6da0 con 0x7fae3c0cab00 2023-10-30T14:26:43.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae3bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae3c0c5550 0x7fae3c1f20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae3bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae3c0c5550 0x7fae3c1f20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43744/0 (socket says 172.21.15.46:43744) 2023-10-30T14:26:43.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae3bfff640 1 -- 172.21.15.46:0/1571057618 learned_addr learned my addr 172.21.15.46:0/1571057618 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae3b7fe640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0cab00 0x7fae3c159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae40808640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae3c0cb050 0x7fae3c157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae3bfff640 1 -- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0cab00 msgr2=0x7fae3c159ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae3bfff640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0cab00 0x7fae3c159ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae3bfff640 1 -- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae3c0cb050 msgr2=0x7fae3c157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae3bfff640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae3c0cb050 0x7fae3c157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae3bfff640 1 -- 172.21.15.46:0/1571057618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae34009a70 con 0x7fae3c0c5550 2023-10-30T14:26:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae3bfff640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae3c0c5550 0x7fae3c1f20f0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00c4d0 tx=0x7fae2c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae38ff9640 1 -- 172.21.15.46:0/1571057618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c005cf0 con 0x7fae3c0c5550 2023-10-30T14:26:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae38ff9640 1 -- 172.21.15.46:0/1571057618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2c005e50 con 0x7fae3c0c5550 2023-10-30T14:26:43.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae41009640 1 -- 172.21.15.46:0/1571057618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c1574c0 con 0x7fae3c0c5550 2023-10-30T14:26:43.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae38ff9640 1 -- 172.21.15.46:0/1571057618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c007270 con 0x7fae3c0c5550 2023-10-30T14:26:43.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.110+0000 7fae41009640 1 -- 172.21.15.46:0/1571057618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c14d880 con 0x7fae3c0c5550 2023-10-30T14:26:43.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.114+0000 7fae41009640 1 -- 172.21.15.46:0/1571057618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c0c85c0 con 0x7fae3c0c5550 2023-10-30T14:26:43.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.114+0000 7fae38ff9640 1 -- 172.21.15.46:0/1571057618 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fae2c0073d0 con 0x7fae3c0c5550 2023-10-30T14:26:43.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.114+0000 7fae38ff9640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fae280664a0 0x7fae28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.114+0000 7fae38ff9640 1 -- 172.21.15.46:0/1571057618 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fae2c083f90 con 0x7fae3c0c5550 2023-10-30T14:26:43.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.114+0000 7fae3b7fe640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fae280664a0 0x7fae28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.114+0000 7fae3b7fe640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fae280664a0 0x7fae28068960 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fae3c14c280 tx=0x7fae30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.118+0000 7fae38ff9640 1 -- 172.21.15.46:0/1571057618 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fae2c050870 con 0x7fae3c0c5550 2023-10-30T14:26:43.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.278+0000 7fae41009640 1 -- 172.21.15.46:0/1571057618 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fae3c15d570 con 0x7fae280664a0 2023-10-30T14:26:43.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.278+0000 7fae38ff9640 1 -- 172.21.15.46:0/1571057618 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fae300059a0 con 0x7fae280664a0 2023-10-30T14:26:43.282 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 5 to deep-scrub 2023-10-30T14:26:43.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.282+0000 7fae41009640 1 -- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fae280664a0 msgr2=0x7fae28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.282+0000 7fae41009640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fae280664a0 0x7fae28068960 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fae3c14c280 tx=0x7fae30008040 comp rx=0 tx=0).stop 2023-10-30T14:26:43.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.282+0000 7fae41009640 1 -- 172.21.15.46:0/1571057618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae3c0c5550 msgr2=0x7fae3c1f20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.282+0000 7fae41009640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae3c0c5550 0x7fae3c1f20f0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00c4d0 tx=0x7fae2c00c990 comp rx=0 tx=0).stop 2023-10-30T14:26:43.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.282+0000 7fae41009640 1 -- 172.21.15.46:0/1571057618 shutdown_connections 2023-10-30T14:26:43.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.282+0000 7fae41009640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fae280664a0 0x7fae28068960 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.282+0000 7fae41009640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae3c0cb050 0x7fae3c157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.282+0000 7fae41009640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0cab00 0x7fae3c159ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.282+0000 7fae41009640 1 --2- 172.21.15.46:0/1571057618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae3c0c5550 0x7fae3c1f20f0 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.282+0000 7fae41009640 1 -- 172.21.15.46:0/1571057618 >> 172.21.15.46:0/1571057618 conn(0x7fae3c06bed0 msgr2=0x7fae3c14f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:43.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.282+0000 7fae41009640 1 -- 172.21.15.46:0/1571057618 shutdown_connections 2023-10-30T14:26:43.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.282+0000 7fae41009640 1 -- 172.21.15.46:0/1571057618 wait complete. 2023-10-30T14:26:43.304 INFO:tasks.ceph:Scrubbing osd.6 2023-10-30T14:26:43.304 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:26:43.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.446+0000 7fa7cf5d3640 1 Processor -- start 2023-10-30T14:26:43.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.446+0000 7fa7cf5d3640 1 -- start start 2023-10-30T14:26:43.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7cf5d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa7c80c6eb0 0x7fa7c80c2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7cf5d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c80c2ca0 0x7fa7c806f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7cf5d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c80c6a50 con 0x7fa7c8156ed0 2023-10-30T14:26:43.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7cf5d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7c80c6630 con 0x7fa7c80c6eb0 2023-10-30T14:26:43.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7cf5d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c80c3750 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7ccb6e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa7c80c6eb0 0x7fa7c80c2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7cdb70640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c80c2ca0 0x7fa7c806f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7cd36f640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa7c8156ed0 0x7fa7c80c43c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:54752/0 (socket says 172.21.15.46:54752) 2023-10-30T14:26:43.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7cd36f640 1 -- 172.21.15.46:0/3308854553 learned_addr learned my addr 172.21.15.46:0/3308854553 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:43.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7cdb70640 1 -- 172.21.15.46:0/3308854553 >> v2:172.21.15.154:3300/0 conn(0x7fa7c80c6eb0 msgr2=0x7fa7c80c2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:43.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7cdb70640 1 --2- 172.21.15.46:0/3308854553 >> v2:172.21.15.154:3300/0 conn(0x7fa7c80c6eb0 0x7fa7c80c2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7cdb70640 1 -- 172.21.15.46:0/3308854553 >> v1:172.21.15.154:6789/0 conn(0x7fa7c8156ed0 legacy=0x7fa7c80c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7cdb70640 1 -- 172.21.15.46:0/3308854553 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c81572b0 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.450+0000 7fa7ccb6e640 1 --2- 172.21.15.46:0/3308854553 >> v2:172.21.15.154:3300/0 conn(0x7fa7c80c6eb0 0x7fa7c80c2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:26:43.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7cdb70640 1 --2- 172.21.15.46:0/3308854553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c80c2ca0 0x7fa7c806f010 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009d90 tx=0x7fa7c001c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=565d561b3a0c55a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7bffff640 1 -- 172.21.15.46:0/3308854553 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00a58f0 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7bffff640 1 -- 172.21.15.46:0/3308854553 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c00a5a50 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7bffff640 1 -- 172.21.15.46:0/3308854553 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00a5de0 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/3308854553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c80c2ca0 msgr2=0x7fa7c806f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7cf5d3640 1 --2- 172.21.15.46:0/3308854553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c80c2ca0 0x7fa7c806f010 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009d90 tx=0x7fa7c001c920 comp rx=0 tx=0).stop 2023-10-30T14:26:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/3308854553 shutdown_connections 2023-10-30T14:26:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7cf5d3640 1 --2- 172.21.15.46:0/3308854553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c80c2ca0 0x7fa7c806f010 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7cf5d3640 1 --2- 172.21.15.46:0/3308854553 >> v2:172.21.15.154:3300/0 conn(0x7fa7c80c6eb0 0x7fa7c80c2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/3308854553 >> 172.21.15.46:0/3308854553 conn(0x7fa7c80ccf80 msgr2=0x7fa7c80cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:43.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/3308854553 shutdown_connections 2023-10-30T14:26:43.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/3308854553 wait complete. 2023-10-30T14:26:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7cf5d3640 1 Processor -- start 2023-10-30T14:26:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.454+0000 7fa7cf5d3640 1 -- start start 2023-10-30T14:26:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cf5d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7c80c2ca0 0x7fa7c814df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cf5d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7c80c6eb0 0x7fa7c8148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cf5d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c8156ed0 0x7fa7c814ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cf5d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c814dc60 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cf5d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c81f5300 con 0x7fa7c80c6eb0 2023-10-30T14:26:43.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cf5d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c81f5510 con 0x7fa7c8156ed0 2023-10-30T14:26:43.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cd36f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7c80c2ca0 0x7fa7c814df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cd36f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7c80c2ca0 0x7fa7c814df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43748/0 (socket says 172.21.15.46:43748) 2023-10-30T14:26:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cd36f640 1 -- 172.21.15.46:0/730248491 learned_addr learned my addr 172.21.15.46:0/730248491 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7ccb6e640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7c80c6eb0 0x7fa7c8148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cdb70640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c8156ed0 0x7fa7c814ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cd36f640 1 -- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c8156ed0 msgr2=0x7fa7c814ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cd36f640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c8156ed0 0x7fa7c814ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cd36f640 1 -- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7c80c6eb0 msgr2=0x7fa7c8148bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cd36f640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7c80c6eb0 0x7fa7c8148bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cd36f640 1 -- 172.21.15.46:0/730248491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b8009420 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cd36f640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7c80c2ca0 0x7fa7c814df00 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fa7c400c400 tx=0x7fa7c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7bdffb640 1 -- 172.21.15.46:0/730248491 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4005cf0 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7bdffb640 1 -- 172.21.15.46:0/730248491 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c4005e50 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7bdffb640 1 -- 172.21.15.46:0/730248491 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c40072d0 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/730248491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c0009a70 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/730248491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c81f5ad0 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/730248491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa7c80c6250 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.458+0000 7fa7bdffb640 1 -- 172.21.15.46:0/730248491 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7c4007430 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.462+0000 7fa7bdffb640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7ac0664d0 0x7fa7ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.462+0000 7fa7bdffb640 1 -- 172.21.15.46:0/730248491 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fa7c4083ec0 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.462+0000 7fa7bdffb640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] conn(0x7fa7ac0704b0 0x7fa7ac072950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.462+0000 7fa7bdffb640 1 -- 172.21.15.46:0/730248491 --> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7ac073020 con 0x7fa7ac0704b0 2023-10-30T14:26:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.462+0000 7fa7bdffb640 1 -- 172.21.15.46:0/730248491 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7c40864a0 con 0x7fa7c80c2ca0 2023-10-30T14:26:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.462+0000 7fa7ccb6e640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7ac0664d0 0x7fa7ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.462+0000 7fa7cdb70640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] conn(0x7fa7ac0704b0 0x7fa7ac072950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.462+0000 7fa7ccb6e640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7ac0664d0 0x7fa7ac068990 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fa7c814c5d0 tx=0x7fa7b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.462+0000 7fa7cdb70640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] conn(0x7fa7ac0704b0 0x7fa7ac072950 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.462+0000 7fa7bdffb640 1 -- 172.21.15.46:0/730248491 <== osd.6 v2:172.21.15.154:6824/1247354054 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fa7ac073020 con 0x7fa7ac0704b0 2023-10-30T14:26:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/730248491 --> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa7c80c3fd0 con 0x7fa7ac0704b0 2023-10-30T14:26:43.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7bdffb640 1 -- 172.21.15.46:0/730248491 <== osd.6 v2:172.21.15.154:6824/1247354054 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa7c0009450 con 0x7fa7ac0704b0 2023-10-30T14:26:43.550 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-10-30T14:26:43.550 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:26:43.550 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-10-30T14:26:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] conn(0x7fa7ac0704b0 msgr2=0x7fa7ac072950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7cf5d3640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:6824/1247354054,v1:172.21.15.154:6825/1247354054] conn(0x7fa7ac0704b0 0x7fa7ac072950 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7ac0664d0 msgr2=0x7fa7ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7cf5d3640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa7ac0664d0 0x7fa7ac068990 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fa7c814c5d0 tx=0x7fa7b8008040 comp rx=0 tx=0).stop 2023-10-30T14:26:43.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/730248491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7c80c2ca0 msgr2=0x7fa7c814df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7cf5d3640 1 --2- 172.21.15.46:0/730248491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7c80c2ca0 0x7fa7c814df00 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fa7c400c400 tx=0x7fa7c400c8c0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7cdb70640 1 -- 172.21.15.46:0/730248491 reap_dead start 2023-10-30T14:26:43.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/730248491 shutdown_connections 2023-10-30T14:26:43.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/730248491 >> 172.21.15.46:0/730248491 conn(0x7fa7c80ccf80 msgr2=0x7fa7c80cb680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:43.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/730248491 shutdown_connections 2023-10-30T14:26:43.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.478+0000 7fa7cf5d3640 1 -- 172.21.15.46:0/730248491 wait complete. 2023-10-30T14:26:43.555 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-30T14:26:43.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.694+0000 7f2b4a06b640 1 Processor -- start 2023-10-30T14:26:43.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.694+0000 7f2b4a06b640 1 -- start start 2023-10-30T14:26:43.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b4a06b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2b441570f0 0x7f2b4406ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b4a06b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4406f370 0x7f2b441640d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b4a06b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4406b3b0 con 0x7f2b440b2300 2023-10-30T14:26:43.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b4a06b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b440c38a0 con 0x7f2b441570f0 2023-10-30T14:26:43.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b4a06b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b440c3b20 con 0x7f2b4406f370 2023-10-30T14:26:43.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b43fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4406f370 0x7f2b441640d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b437fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2b441570f0 0x7f2b4406ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b43fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4406f370 0x7f2b441640d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:52904/0 (socket says 172.21.15.46:52904) 2023-10-30T14:26:43.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b43fff640 1 -- 172.21.15.46:0/625782361 learned_addr learned my addr 172.21.15.46:0/625782361 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:43.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b42ffd640 1 --1- 172.21.15.46:0/625782361 >> v1:172.21.15.154:6789/0 conn(0x7f2b440b2300 0x7f2b44071b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:54756/0 (socket says 172.21.15.46:54756) 2023-10-30T14:26:43.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b437fe640 1 -- 172.21.15.46:0/625782361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4406f370 msgr2=0x7f2b441640d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:43.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b437fe640 1 --2- 172.21.15.46:0/625782361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4406f370 0x7f2b441640d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b437fe640 1 -- 172.21.15.46:0/625782361 >> v1:172.21.15.154:6789/0 conn(0x7f2b440b2300 legacy=0x7f2b44071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.698+0000 7f2b437fe640 1 -- 172.21.15.46:0/625782361 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b440ca9f0 con 0x7f2b441570f0 2023-10-30T14:26:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b437fe640 1 --2- 172.21.15.46:0/625782361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b441570f0 0x7f2b4406ca30 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2b340097c0 tx=0x7f2b3401c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e1f05e074fdbd5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b427fc640 1 -- 172.21.15.46:0/625782361 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b340ab600 con 0x7f2b441570f0 2023-10-30T14:26:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b427fc640 1 -- 172.21.15.46:0/625782361 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b340a7050 con 0x7f2b441570f0 2023-10-30T14:26:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b427fc640 1 -- 172.21.15.46:0/625782361 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b340ab980 con 0x7f2b441570f0 2023-10-30T14:26:43.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b4a06b640 1 -- 172.21.15.46:0/625782361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b441570f0 msgr2=0x7f2b4406ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b4a06b640 1 --2- 172.21.15.46:0/625782361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b441570f0 0x7f2b4406ca30 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2b340097c0 tx=0x7f2b3401c480 comp rx=0 tx=0).stop 2023-10-30T14:26:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b4a06b640 1 -- 172.21.15.46:0/625782361 shutdown_connections 2023-10-30T14:26:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b4a06b640 1 --2- 172.21.15.46:0/625782361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4406f370 0x7f2b441640d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b4a06b640 1 --2- 172.21.15.46:0/625782361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b441570f0 0x7f2b4406ca30 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b4a06b640 1 -- 172.21.15.46:0/625782361 >> 172.21.15.46:0/625782361 conn(0x7f2b44069e90 msgr2=0x7f2b4406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b4a06b640 1 -- 172.21.15.46:0/625782361 shutdown_connections 2023-10-30T14:26:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b4a06b640 1 -- 172.21.15.46:0/625782361 wait complete. 2023-10-30T14:26:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b4a06b640 1 Processor -- start 2023-10-30T14:26:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.702+0000 7f2b4a06b640 1 -- start start 2023-10-30T14:26:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b4a06b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4406f370 0x7f2b4415ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b4a06b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b440b2300 0x7f2b44071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b4a06b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b441570f0 0x7f2b44072010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b4a06b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4415b370 con 0x7f2b4406f370 2023-10-30T14:26:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b4a06b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b441f60d0 con 0x7f2b440b2300 2023-10-30T14:26:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b4a06b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b440cd580 con 0x7f2b441570f0 2023-10-30T14:26:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b437fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4406f370 0x7f2b4415ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b437fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4406f370 0x7f2b4415ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43764/0 (socket says 172.21.15.46:43764) 2023-10-30T14:26:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b437fe640 1 -- 172.21.15.46:0/2347055908 learned_addr learned my addr 172.21.15.46:0/2347055908 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:43.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b42ffd640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b440b2300 0x7f2b44071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b43fff640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b441570f0 0x7f2b44072010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b437fe640 1 -- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b441570f0 msgr2=0x7f2b44072010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b437fe640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b441570f0 0x7f2b44072010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b437fe640 1 -- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b440b2300 msgr2=0x7f2b44071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b437fe640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b440b2300 0x7f2b44071ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b437fe640 1 -- 172.21.15.46:0/2347055908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b34009420 con 0x7f2b4406f370 2023-10-30T14:26:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b42ffd640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b440b2300 0x7f2b44071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b437fe640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4406f370 0x7f2b4415ce80 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f2b340097c0 tx=0x7f2b34019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b2bfff640 1 -- 172.21.15.46:0/2347055908 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b340b4430 con 0x7f2b4406f370 2023-10-30T14:26:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b2bfff640 1 -- 172.21.15.46:0/2347055908 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b340b4590 con 0x7f2b4406f370 2023-10-30T14:26:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b4a06b640 1 -- 172.21.15.46:0/2347055908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b441f5450 con 0x7f2b4406f370 2023-10-30T14:26:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b2bfff640 1 -- 172.21.15.46:0/2347055908 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b340b4940 con 0x7f2b4406f370 2023-10-30T14:26:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b4a06b640 1 -- 172.21.15.46:0/2347055908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b441f5a10 con 0x7f2b4406f370 2023-10-30T14:26:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.706+0000 7f2b4a06b640 1 -- 172.21.15.46:0/2347055908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4404e170 con 0x7f2b4406f370 2023-10-30T14:26:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.710+0000 7f2b2bfff640 1 -- 172.21.15.46:0/2347055908 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2b340b2040 con 0x7f2b4406f370 2023-10-30T14:26:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.710+0000 7f2b2bfff640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b300664a0 0x7f2b30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.710+0000 7f2b2bfff640 1 -- 172.21.15.46:0/2347055908 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f2b3411ea00 con 0x7f2b4406f370 2023-10-30T14:26:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.710+0000 7f2b42ffd640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b300664a0 0x7f2b30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.710+0000 7f2b42ffd640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b300664a0 0x7f2b30068960 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2b4415b8d0 tx=0x7f2b38007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:43.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.714+0000 7f2b2bfff640 1 -- 172.21.15.46:0/2347055908 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2b340eb230 con 0x7f2b4406f370 2023-10-30T14:26:43.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.870+0000 7f2b4a06b640 1 -- 172.21.15.46:0/2347055908 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b440c97f0 con 0x7f2b300664a0 2023-10-30T14:26:43.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.870+0000 7f2b2bfff640 1 -- 172.21.15.46:0/2347055908 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2b380074e0 con 0x7f2b300664a0 2023-10-30T14:26:43.875 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 6 to deep-scrub 2023-10-30T14:26:43.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.874+0000 7f2b4a06b640 1 -- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b300664a0 msgr2=0x7f2b30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.874+0000 7f2b4a06b640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b300664a0 0x7f2b30068960 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2b4415b8d0 tx=0x7f2b38007d50 comp rx=0 tx=0).stop 2023-10-30T14:26:43.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.874+0000 7f2b4a06b640 1 -- 172.21.15.46:0/2347055908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4406f370 msgr2=0x7f2b4415ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:43.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.874+0000 7f2b4a06b640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4406f370 0x7f2b4415ce80 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f2b340097c0 tx=0x7f2b34019220 comp rx=0 tx=0).stop 2023-10-30T14:26:43.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.874+0000 7f2b4a06b640 1 -- 172.21.15.46:0/2347055908 shutdown_connections 2023-10-30T14:26:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.874+0000 7f2b4a06b640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f2b300664a0 0x7f2b30068960 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.874+0000 7f2b4a06b640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b441570f0 0x7f2b44072010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.874+0000 7f2b4a06b640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b440b2300 0x7f2b44071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.874+0000 7f2b4a06b640 1 --2- 172.21.15.46:0/2347055908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4406f370 0x7f2b4415ce80 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:43.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.874+0000 7f2b4a06b640 1 -- 172.21.15.46:0/2347055908 >> 172.21.15.46:0/2347055908 conn(0x7f2b44069e90 msgr2=0x7f2b441554d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:43.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.874+0000 7f2b4a06b640 1 -- 172.21.15.46:0/2347055908 shutdown_connections 2023-10-30T14:26:43.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:43.874+0000 7f2b4a06b640 1 -- 172.21.15.46:0/2347055908 wait complete. 2023-10-30T14:26:43.897 INFO:tasks.ceph:Scrubbing osd.7 2023-10-30T14:26:43.897 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-30T14:26:44.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.042+0000 7fb98831d640 1 Processor -- start 2023-10-30T14:26:44.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb98831d640 1 -- start start 2023-10-30T14:26:44.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb98831d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb98015a700 0x7fb980156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb98831d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98015a270 con 0x7fb9801569d0 2023-10-30T14:26:44.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb98831d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98006c7c0 con 0x7fb98014cf40 2023-10-30T14:26:44.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb98831d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb980156ed0 con 0x7fb98015a700 2023-10-30T14:26:44.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb9868ba640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb9801569d0 0x7fb9800c6920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38134/0 (socket says 172.21.15.46:38134) 2023-10-30T14:26:44.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb9868ba640 1 -- 172.21.15.46:0/492751187 learned_addr learned my addr 172.21.15.46:0/492751187 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:44.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb9858b8640 1 --2- 172.21.15.46:0/492751187 >> v2:172.21.15.154:3300/0 conn(0x7fb98015a700 0x7fb980156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb9850b7640 1 -- 172.21.15.46:0/492751187 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926228635 0 0) 0x7fb98015a270 con 0x7fb9801569d0 2023-10-30T14:26:44.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb9850b7640 1 -- 172.21.15.46:0/492751187 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb96c003530 con 0x7fb9801569d0 2023-10-30T14:26:44.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb9858b8640 1 -- 172.21.15.46:0/492751187 >> v1:172.21.15.154:6789/0 conn(0x7fb98014cf40 legacy=0x7fb980069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb9858b8640 1 -- 172.21.15.46:0/492751187 >> v1:172.21.15.46:6789/0 conn(0x7fb9801569d0 legacy=0x7fb9800c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.046+0000 7fb9858b8640 1 -- 172.21.15.46:0/492751187 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9801571b0 con 0x7fb98015a700 2023-10-30T14:26:44.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.050+0000 7fb9858b8640 1 --2- 172.21.15.46:0/492751187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb98015a700 0x7fb980156490 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb9700097c0 tx=0x7fb97001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7fc2deb3a67d43 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:44.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.050+0000 7fb9850b7640 1 -- 172.21.15.46:0/492751187 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9700ab600 con 0x7fb98015a700 2023-10-30T14:26:44.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.050+0000 7fb9850b7640 1 -- 172.21.15.46:0/492751187 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9700a7050 con 0x7fb98015a700 2023-10-30T14:26:44.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.050+0000 7fb9850b7640 1 -- 172.21.15.46:0/492751187 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9700ab980 con 0x7fb98015a700 2023-10-30T14:26:44.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.050+0000 7fb98831d640 1 -- 172.21.15.46:0/492751187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb98015a700 msgr2=0x7fb980156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.050+0000 7fb98831d640 1 --2- 172.21.15.46:0/492751187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb98015a700 0x7fb980156490 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb9700097c0 tx=0x7fb97001c480 comp rx=0 tx=0).stop 2023-10-30T14:26:44.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.050+0000 7fb98831d640 1 -- 172.21.15.46:0/492751187 shutdown_connections 2023-10-30T14:26:44.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.050+0000 7fb98831d640 1 --2- 172.21.15.46:0/492751187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb98015a700 0x7fb980156490 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.050+0000 7fb98831d640 1 -- 172.21.15.46:0/492751187 >> 172.21.15.46:0/492751187 conn(0x7fb9800c7910 msgr2=0x7fb9800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.050+0000 7fb98831d640 1 -- 172.21.15.46:0/492751187 shutdown_connections 2023-10-30T14:26:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.050+0000 7fb98831d640 1 -- 172.21.15.46:0/492751187 wait complete. 2023-10-30T14:26:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb98831d640 1 Processor -- start 2023-10-30T14:26:44.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb98831d640 1 -- start start 2023-10-30T14:26:44.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb98831d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb98014cf40 0x7fb9801490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb98831d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9801569d0 0x7fb9801458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb98831d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98015a700 0x7fb980145df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb98831d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9801f64f0 con 0x7fb98014cf40 2023-10-30T14:26:44.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb98831d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800c6c90 con 0x7fb9801569d0 2023-10-30T14:26:44.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb98831d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9801f5690 con 0x7fb98015a700 2023-10-30T14:26:44.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb9860b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb98014cf40 0x7fb9801490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb9860b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb98014cf40 0x7fb9801490e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43774/0 (socket says 172.21.15.46:43774) 2023-10-30T14:26:44.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb9860b9640 1 -- 172.21.15.46:0/3939885290 learned_addr learned my addr 172.21.15.46:0/3939885290 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb9858b8640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9801569d0 0x7fb9801458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb9868ba640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98015a700 0x7fb980145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb9860b9640 1 -- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98015a700 msgr2=0x7fb980145df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb9860b9640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98015a700 0x7fb980145df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb9860b9640 1 -- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9801569d0 msgr2=0x7fb9801458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb9860b9640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9801569d0 0x7fb9801458b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb9860b9640 1 -- 172.21.15.46:0/3939885290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb970009420 con 0x7fb98014cf40 2023-10-30T14:26:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb9860b9640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb98014cf40 0x7fb9801490e0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00c420 tx=0x7fb97c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb976ffd640 1 -- 172.21.15.46:0/3939885290 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c0052d0 con 0x7fb98014cf40 2023-10-30T14:26:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb976ffd640 1 -- 172.21.15.46:0/3939885290 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c005430 con 0x7fb98014cf40 2023-10-30T14:26:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb98831d640 1 -- 172.21.15.46:0/3939885290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9801f5970 con 0x7fb98014cf40 2023-10-30T14:26:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb976ffd640 1 -- 172.21.15.46:0/3939885290 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c007260 con 0x7fb98014cf40 2023-10-30T14:26:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb98831d640 1 -- 172.21.15.46:0/3939885290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9801f5f30 con 0x7fb98014cf40 2023-10-30T14:26:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.054+0000 7fb98831d640 1 -- 172.21.15.46:0/3939885290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb98014d320 con 0x7fb98014cf40 2023-10-30T14:26:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.058+0000 7fb976ffd640 1 -- 172.21.15.46:0/3939885290 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb97c0073c0 con 0x7fb98014cf40 2023-10-30T14:26:44.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.058+0000 7fb976ffd640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9640664a0 0x7fb964068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.058+0000 7fb976ffd640 1 -- 172.21.15.46:0/3939885290 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fb97c083fd0 con 0x7fb98014cf40 2023-10-30T14:26:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.058+0000 7fb9858b8640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9640664a0 0x7fb964068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.058+0000 7fb976ffd640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] conn(0x7fb964070480 0x7fb964072920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.058+0000 7fb976ffd640 1 -- 172.21.15.46:0/3939885290 --> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb964072ff0 con 0x7fb964070480 2023-10-30T14:26:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.058+0000 7fb976ffd640 1 -- 172.21.15.46:0/3939885290 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb97c0865b0 con 0x7fb98014cf40 2023-10-30T14:26:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.058+0000 7fb9868ba640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] conn(0x7fb964070480 0x7fb964072920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.058+0000 7fb9858b8640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9640664a0 0x7fb964068960 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fb9700ad000 tx=0x7fb9700a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:44.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.058+0000 7fb9868ba640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] conn(0x7fb964070480 0x7fb964072920 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:44.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.062+0000 7fb976ffd640 1 -- 172.21.15.46:0/3939885290 <== osd.7 v2:172.21.15.154:6816/1343138186 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb964072ff0 con 0x7fb964070480 2023-10-30T14:26:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb98831d640 1 -- 172.21.15.46:0/3939885290 --> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb98004e170 con 0x7fb964070480 2023-10-30T14:26:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb976ffd640 1 -- 172.21.15.46:0/3939885290 <== osd.7 v2:172.21.15.154:6816/1343138186 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb98004e170 con 0x7fb964070480 2023-10-30T14:26:44.079 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-10-30T14:26:44.079 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T14:26:44.080 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-10-30T14:26:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb98831d640 1 -- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] conn(0x7fb964070480 msgr2=0x7fb964072920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb98831d640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:6816/1343138186,v1:172.21.15.154:6817/1343138186] conn(0x7fb964070480 0x7fb964072920 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb98831d640 1 -- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9640664a0 msgr2=0x7fb964068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb98831d640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fb9640664a0 0x7fb964068960 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fb9700ad000 tx=0x7fb9700a6000 comp rx=0 tx=0).stop 2023-10-30T14:26:44.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb98831d640 1 -- 172.21.15.46:0/3939885290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb98014cf40 msgr2=0x7fb9801490e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb98831d640 1 --2- 172.21.15.46:0/3939885290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb98014cf40 0x7fb9801490e0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00c420 tx=0x7fb97c00c8e0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb9868ba640 1 -- 172.21.15.46:0/3939885290 reap_dead start 2023-10-30T14:26:44.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb98831d640 1 -- 172.21.15.46:0/3939885290 shutdown_connections 2023-10-30T14:26:44.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb98831d640 1 -- 172.21.15.46:0/3939885290 >> 172.21.15.46:0/3939885290 conn(0x7fb9800c7910 msgr2=0x7fb9801555f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:44.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb98831d640 1 -- 172.21.15.46:0/3939885290 shutdown_connections 2023-10-30T14:26:44.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.074+0000 7fb98831d640 1 -- 172.21.15.46:0/3939885290 wait complete. 2023-10-30T14:26:44.097 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-30T14:26:44.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.238+0000 7fde0e92a640 1 Processor -- start 2023-10-30T14:26:44.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.238+0000 7fde0e92a640 1 -- start start 2023-10-30T14:26:44.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.238+0000 7fde0e92a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde080c6f60 0x7fde0813f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.238+0000 7fde0e92a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde080c08d0 0x7fde0813fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.238+0000 7fde0e92a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde08072030 con 0x7fde080c6b80 2023-10-30T14:26:44.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.238+0000 7fde0e92a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde08070010 con 0x7fde080c08d0 2023-10-30T14:26:44.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.238+0000 7fde0e92a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde08072260 con 0x7fde080c6f60 2023-10-30T14:26:44.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.238+0000 7fde07fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fde080c6b80 0x7fde08071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38142/0 (socket says 172.21.15.46:38142) 2023-10-30T14:26:44.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.238+0000 7fde07fff640 1 -- 172.21.15.46:0/3764418401 learned_addr learned my addr 172.21.15.46:0/3764418401 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:26:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.238+0000 7fde0cec7640 1 --2- 172.21.15.46:0/3764418401 >> v2:172.21.15.46:3300/0 conn(0x7fde080c08d0 0x7fde0813fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.238+0000 7fde077fe640 1 --2- 172.21.15.46:0/3764418401 >> v2:172.21.15.154:3300/0 conn(0x7fde080c6f60 0x7fde0813f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde06ffd640 1 -- 172.21.15.46:0/3764418401 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125640555 0 0) 0x7fde08072030 con 0x7fde080c6b80 2023-10-30T14:26:44.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde06ffd640 1 -- 172.21.15.46:0/3764418401 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf0005740 con 0x7fde080c6b80 2023-10-30T14:26:44.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde0cec7640 1 -- 172.21.15.46:0/3764418401 >> v2:172.21.15.154:3300/0 conn(0x7fde080c6f60 msgr2=0x7fde0813f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:44.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde0cec7640 1 --2- 172.21.15.46:0/3764418401 >> v2:172.21.15.154:3300/0 conn(0x7fde080c6f60 0x7fde0813f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde0cec7640 1 -- 172.21.15.46:0/3764418401 >> v1:172.21.15.46:6789/0 conn(0x7fde080c6b80 legacy=0x7fde08071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde0cec7640 1 -- 172.21.15.46:0/3764418401 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0806f6e0 con 0x7fde080c08d0 2023-10-30T14:26:44.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde0cec7640 1 --2- 172.21.15.46:0/3764418401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080c08d0 0x7fde0813fb40 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fde080b0850 tx=0x7fde0001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c09c8bb3c8d6e1e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:44.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde06ffd640 1 -- 172.21.15.46:0/3764418401 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde000a59a0 con 0x7fde080c08d0 2023-10-30T14:26:44.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde06ffd640 1 -- 172.21.15.46:0/3764418401 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde000a5b00 con 0x7fde080c08d0 2023-10-30T14:26:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde06ffd640 1 -- 172.21.15.46:0/3764418401 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde000a5e90 con 0x7fde080c08d0 2023-10-30T14:26:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde0e92a640 1 -- 172.21.15.46:0/3764418401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080c08d0 msgr2=0x7fde0813fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde0e92a640 1 --2- 172.21.15.46:0/3764418401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080c08d0 0x7fde0813fb40 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fde080b0850 tx=0x7fde0001c8e0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde0e92a640 1 -- 172.21.15.46:0/3764418401 shutdown_connections 2023-10-30T14:26:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde0e92a640 1 --2- 172.21.15.46:0/3764418401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080c08d0 0x7fde0813fb40 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde0e92a640 1 --2- 172.21.15.46:0/3764418401 >> v2:172.21.15.154:3300/0 conn(0x7fde080c6f60 0x7fde0813f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.242+0000 7fde0e92a640 1 -- 172.21.15.46:0/3764418401 >> 172.21.15.46:0/3764418401 conn(0x7fde080c7910 msgr2=0x7fde080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:44.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde0e92a640 1 -- 172.21.15.46:0/3764418401 shutdown_connections 2023-10-30T14:26:44.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde0e92a640 1 -- 172.21.15.46:0/3764418401 wait complete. 2023-10-30T14:26:44.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde0e92a640 1 Processor -- start 2023-10-30T14:26:44.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde0e92a640 1 -- start start 2023-10-30T14:26:44.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde0e92a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080c08d0 0x7fde080c2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde0e92a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde080c6b80 0x7fde080ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde0e92a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde080c6f60 0x7fde080c3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde0e92a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde080c2e90 con 0x7fde080c08d0 2023-10-30T14:26:44.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde0e92a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde080c2bb0 con 0x7fde080c6f60 2023-10-30T14:26:44.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde0e92a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde081edff0 con 0x7fde080c6b80 2023-10-30T14:26:44.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080c08d0 0x7fde080c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080c08d0 0x7fde080c2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43792/0 (socket says 172.21.15.46:43792) 2023-10-30T14:26:44.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde07fff640 1 -- 172.21.15.46:0/3373908536 learned_addr learned my addr 172.21.15.46:0/3373908536 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:44.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde0cec7640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde080c6f60 0x7fde080c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde077fe640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde080c6b80 0x7fde080ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde07fff640 1 -- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde080c6b80 msgr2=0x7fde080ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde07fff640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde080c6b80 0x7fde080ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde07fff640 1 -- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde080c6f60 msgr2=0x7fde080c3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde07fff640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde080c6f60 0x7fde080c3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde07fff640 1 -- 172.21.15.46:0/3373908536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde00009a70 con 0x7fde080c08d0 2023-10-30T14:26:44.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde077fe640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde080c6b80 0x7fde080ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:44.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde07fff640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080c08d0 0x7fde080c2210 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fddf800c4d0 tx=0x7fddf800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:44.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.246+0000 7fde04ff9640 1 -- 172.21.15.46:0/3373908536 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8005cf0 con 0x7fde080c08d0 2023-10-30T14:26:44.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.250+0000 7fde04ff9640 1 -- 172.21.15.46:0/3373908536 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf8005e50 con 0x7fde080c08d0 2023-10-30T14:26:44.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.250+0000 7fde04ff9640 1 -- 172.21.15.46:0/3373908536 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8007270 con 0x7fde080c08d0 2023-10-30T14:26:44.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.250+0000 7fde0e92a640 1 -- 172.21.15.46:0/3373908536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde080c2820 con 0x7fde080c08d0 2023-10-30T14:26:44.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.250+0000 7fde0e92a640 1 -- 172.21.15.46:0/3373908536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde080b0740 con 0x7fde080c08d0 2023-10-30T14:26:44.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.250+0000 7fde0e92a640 1 -- 172.21.15.46:0/3373908536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde080c3060 con 0x7fde080c08d0 2023-10-30T14:26:44.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.250+0000 7fde04ff9640 1 -- 172.21.15.46:0/3373908536 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fddf80073d0 con 0x7fde080c08d0 2023-10-30T14:26:44.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.250+0000 7fde04ff9640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddf40664d0 0x7fddf4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.250+0000 7fde04ff9640 1 -- 172.21.15.46:0/3373908536 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fddf8084040 con 0x7fde080c08d0 2023-10-30T14:26:44.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.250+0000 7fde077fe640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddf40664d0 0x7fddf4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.254+0000 7fde077fe640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddf40664d0 0x7fddf4068990 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fde080c1560 tx=0x7fddfc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:44.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.254+0000 7fde04ff9640 1 -- 172.21.15.46:0/3373908536 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fddf80b5090 con 0x7fde080c08d0 2023-10-30T14:26:44.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.410+0000 7fde0e92a640 1 -- 172.21.15.46:0/3373908536 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fde080c2d20 con 0x7fddf40664d0 2023-10-30T14:26:44.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde04ff9640 1 -- 172.21.15.46:0/3373908536 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fde080c2d20 con 0x7fddf40664d0 2023-10-30T14:26:44.416 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 7 to deep-scrub 2023-10-30T14:26:44.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde0e92a640 1 -- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddf40664d0 msgr2=0x7fddf4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde0e92a640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddf40664d0 0x7fddf4068990 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fde080c1560 tx=0x7fddfc008040 comp rx=0 tx=0).stop 2023-10-30T14:26:44.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde0e92a640 1 -- 172.21.15.46:0/3373908536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080c08d0 msgr2=0x7fde080c2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde0e92a640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080c08d0 0x7fde080c2210 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fddf800c4d0 tx=0x7fddf800c990 comp rx=0 tx=0).stop 2023-10-30T14:26:44.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde0e92a640 1 -- 172.21.15.46:0/3373908536 shutdown_connections 2023-10-30T14:26:44.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde0e92a640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fddf40664d0 0x7fddf4068990 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde0e92a640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde080c6f60 0x7fde080c3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde0e92a640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde080c6b80 0x7fde080ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde0e92a640 1 --2- 172.21.15.46:0/3373908536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde080c08d0 0x7fde080c2210 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde0e92a640 1 -- 172.21.15.46:0/3373908536 >> 172.21.15.46:0/3373908536 conn(0x7fde080c7910 msgr2=0x7fde08138da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:44.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde0e92a640 1 -- 172.21.15.46:0/3373908536 shutdown_connections 2023-10-30T14:26:44.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.414+0000 7fde0e92a640 1 -- 172.21.15.46:0/3373908536 wait complete. 2023-10-30T14:26:44.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:26:44.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c7dc4640 1 Processor -- start 2023-10-30T14:26:44.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c7dc4640 1 -- start start 2023-10-30T14:26:44.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c7dc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f21c01538a0 0x7f21c015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c7dc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21c00ca0e0 0x7f21c015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c7dc4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f21c00cf160 0x7f21c00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c7dc4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c00cb800 con 0x7f21c00cf160 2023-10-30T14:26:44.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c7dc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c015dc30 con 0x7f21c01538a0 2023-10-30T14:26:44.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c7dc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c00cb330 con 0x7f21c00ca0e0 2023-10-30T14:26:44.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c6361640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f21c00cf160 0x7f21c00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c6361640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f21c00cf160 0x7f21c00cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43806/0 (socket says 172.21.15.46:43806) 2023-10-30T14:26:44.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c6361640 1 -- 172.21.15.46:0/3059545899 learned_addr learned my addr 172.21.15.46:0/3059545899 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:44.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c5b60640 1 --2- 172.21.15.46:0/3059545899 >> v2:172.21.15.154:3300/0 conn(0x7f21c01538a0 0x7f21c015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c535f640 1 --2- 172.21.15.46:0/3059545899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21c00ca0e0 0x7f21c015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c6361640 1 -- 172.21.15.46:0/3059545899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21c00ca0e0 msgr2=0x7f21c015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:44.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c6361640 1 --2- 172.21.15.46:0/3059545899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21c00ca0e0 0x7f21c015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c6361640 1 -- 172.21.15.46:0/3059545899 >> v2:172.21.15.154:3300/0 conn(0x7f21c01538a0 msgr2=0x7f21c015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:26:44.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c6361640 1 --2- 172.21.15.46:0/3059545899 >> v2:172.21.15.154:3300/0 conn(0x7f21c01538a0 0x7f21c015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c6361640 1 -- 172.21.15.46:0/3059545899 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c00cad70 con 0x7f21c00cf160 2023-10-30T14:26:44.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.578+0000 7f21c535f640 1 --2- 172.21.15.46:0/3059545899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21c00ca0e0 0x7f21c015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T14:26:44.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c6361640 1 --2- 172.21.15.46:0/3059545899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21c00cf160 0x7f21c00cf590 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f21c00cfd10 tx=0x7f21b801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7275b7f3d32ac62f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:44.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c4b5e640 1 -- 172.21.15.46:0/3059545899 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b80af400 con 0x7f21c00cf160 2023-10-30T14:26:44.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c4b5e640 1 -- 172.21.15.46:0/3059545899 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b80ab050 con 0x7f21c00cf160 2023-10-30T14:26:44.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c4b5e640 1 -- 172.21.15.46:0/3059545899 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b80af780 con 0x7f21c00cf160 2023-10-30T14:26:44.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3059545899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21c00cf160 msgr2=0x7f21c00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c7dc4640 1 --2- 172.21.15.46:0/3059545899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21c00cf160 0x7f21c00cf590 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f21c00cfd10 tx=0x7f21b801f1c0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3059545899 shutdown_connections 2023-10-30T14:26:44.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c7dc4640 1 --2- 172.21.15.46:0/3059545899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21c00cf160 0x7f21c00cf590 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c7dc4640 1 --2- 172.21.15.46:0/3059545899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21c00ca0e0 0x7f21c015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c7dc4640 1 --2- 172.21.15.46:0/3059545899 >> v2:172.21.15.154:3300/0 conn(0x7f21c01538a0 0x7f21c015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3059545899 >> 172.21.15.46:0/3059545899 conn(0x7f21c006bfe0 msgr2=0x7f21c006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3059545899 shutdown_connections 2023-10-30T14:26:44.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.582+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3059545899 wait complete. 2023-10-30T14:26:44.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c7dc4640 1 Processor -- start 2023-10-30T14:26:44.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c7dc4640 1 -- start start 2023-10-30T14:26:44.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c7dc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21c00ca0e0 0x7f21c01592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c7dc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21c00cf160 0x7f21c01f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c7dc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21c01538a0 0x7f21c01f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c7dc4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c01e5ae0 con 0x7f21c00ca0e0 2023-10-30T14:26:44.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c7dc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c01e4720 con 0x7f21c01538a0 2023-10-30T14:26:44.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c7dc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c01f3760 con 0x7f21c00cf160 2023-10-30T14:26:44.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c5b60640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21c00ca0e0 0x7f21c01592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c5b60640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21c00ca0e0 0x7f21c01592a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43812/0 (socket says 172.21.15.46:43812) 2023-10-30T14:26:44.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c5b60640 1 -- 172.21.15.46:0/3114512199 learned_addr learned my addr 172.21.15.46:0/3114512199 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:26:44.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c6361640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21c01538a0 0x7f21c01f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c535f640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21c00cf160 0x7f21c01f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c5b60640 1 -- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21c00cf160 msgr2=0x7f21c01f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c5b60640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21c00cf160 0x7f21c01f2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c5b60640 1 -- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21c01538a0 msgr2=0x7f21c01f2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c5b60640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21c01538a0 0x7f21c01f2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c5b60640 1 -- 172.21.15.46:0/3114512199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b8009b30 con 0x7f21c00ca0e0 2023-10-30T14:26:44.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c6361640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21c01538a0 0x7f21c01f2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:26:44.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c5b60640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21c00ca0e0 0x7f21c01592a0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00d5f0 tx=0x7f21bc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:44.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21b67fc640 1 -- 172.21.15.46:0/3114512199 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0125f0 con 0x7f21c00ca0e0 2023-10-30T14:26:44.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21b67fc640 1 -- 172.21.15.46:0/3114512199 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc012750 con 0x7f21c00ca0e0 2023-10-30T14:26:44.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21b67fc640 1 -- 172.21.15.46:0/3114512199 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc012ae0 con 0x7f21c00ca0e0 2023-10-30T14:26:44.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3114512199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c00b33d0 con 0x7f21c00ca0e0 2023-10-30T14:26:44.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3114512199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c01e4160 con 0x7f21c00ca0e0 2023-10-30T14:26:44.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.586+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3114512199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c00b2a50 con 0x7f21c00ca0e0 2023-10-30T14:26:44.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.590+0000 7f21b67fc640 1 -- 172.21.15.46:0/3114512199 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f21bc012c40 con 0x7f21c00ca0e0 2023-10-30T14:26:44.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.590+0000 7f21b67fc640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21a40664a0 0x7f21a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:26:44.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.590+0000 7f21b67fc640 1 -- 172.21.15.46:0/3114512199 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f21bc084fd0 con 0x7f21c00ca0e0 2023-10-30T14:26:44.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.590+0000 7f21c535f640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21a40664a0 0x7f21a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:26:44.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.590+0000 7f21c535f640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21a40664a0 0x7f21a4068960 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f21c00b1da0 tx=0x7f21b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:26:44.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.594+0000 7f21b67fc640 1 -- 172.21.15.46:0/3114512199 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f21bc051800 con 0x7f21c00ca0e0 2023-10-30T14:26:44.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.746+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3114512199 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21c00cb050 con 0x7f21a40664a0 2023-10-30T14:26:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.770+0000 7f21b67fc640 1 -- 172.21.15.46:0/3114512199 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291977 (secure 0 0 0) 0x7f21b000c050 con 0x7f21a40664a0 2023-10-30T14:26:44.775 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:26:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T14:26:44.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.774+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21a40664a0 msgr2=0x7f21a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.774+0000 7f21c7dc4640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21a40664a0 0x7f21a4068960 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f21c00b1da0 tx=0x7f21b0008040 comp rx=0 tx=0).stop 2023-10-30T14:26:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.774+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3114512199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21c00ca0e0 msgr2=0x7f21c01592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:26:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.774+0000 7f21c7dc4640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21c00ca0e0 0x7f21c01592a0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00d5f0 tx=0x7f21bc00dab0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.774+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3114512199 shutdown_connections 2023-10-30T14:26:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.774+0000 7f21c7dc4640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f21a40664a0 0x7f21a4068960 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.774+0000 7f21c7dc4640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21c01538a0 0x7f21c01f2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.774+0000 7f21c7dc4640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21c00cf160 0x7f21c01f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.774+0000 7f21c7dc4640 1 --2- 172.21.15.46:0/3114512199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21c00ca0e0 0x7f21c01592a0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:26:44.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.774+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3114512199 >> 172.21.15.46:0/3114512199 conn(0x7f21c006bfe0 msgr2=0x7f21c0151880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:26:44.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.774+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3114512199 shutdown_connections 2023-10-30T14:26:44.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:26:44.774+0000 7f21c7dc4640 1 -- 172.21.15.46:0/3114512199 wait complete. 2023-10-30T14:26:44.796 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":2012,"stamp":"2023-10-30T14:26:44.129848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1285321698,"num_objects":3948,"num_object_clones":2,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":20157,"num_read_kb":14487318,"num_write":119396,"num_write_kb":3805667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170636,"ondisk_log_size":170636,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3932272,"kb_used_data":2521628,"kb_used_omap":12,"kb_used_meta":1410547,"kb_avail":751042448,"statfs":{"total":773094113280,"available":769067466752,"internally_reserved":0,"allocated":2582147072,"data_stored":2576910078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1444400696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":117,"apply_latency_ms":117,"commit_latency_ns":117000000,"apply_latency_ns":117000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1021051,"num_objects":-3,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":53,"num_write_kb":1026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.237203"},"pg_stats":[{"pgid":"3.3f","version":"109'2161","reported_seq":4974,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.907908+0000","last_change":"2023-10-30T14:26:34.688745+0000","last_active":"2023-10-30T14:26:39.907908+0000","last_peered":"2023-10-30T14:26:39.907908+0000","last_clean":"2023-10-30T14:26:39.907908+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T14:26:39.907908+0000","last_undegraded":"2023-10-30T14:26:39.907908+0000","last_fullsized":"2023-10-30T14:26:39.907908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2161,"log_dups_size":0,"ondisk_log_size":2161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:50.061774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.84725695499999998,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":124,"num_read_kb":10662,"num_write":1276,"num_write_kb":15944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"109'1971","reported_seq":4851,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.183422+0000","last_change":"2023-10-30T14:26:33.665197+0000","last_active":"2023-10-30T14:26:40.183422+0000","last_peered":"2023-10-30T14:26:40.183422+0000","last_clean":"2023-10-30T14:26:40.183422+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T14:26:40.183422+0000","last_undegraded":"2023-10-30T14:26:40.183422+0000","last_fullsized":"2023-10-30T14:26:40.183422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1971,"log_dups_size":0,"ondisk_log_size":1971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.406256491,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":133,"num_read_kb":21158,"num_write":1230,"num_write_kb":16620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"109'2095","reported_seq":4929,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.756604+0000","last_change":"2023-10-30T14:26:34.547863+0000","last_active":"2023-10-30T14:26:40.756604+0000","last_peered":"2023-10-30T14:26:40.756604+0000","last_clean":"2023-10-30T14:26:40.756604+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T14:26:40.756604+0000","last_undegraded":"2023-10-30T14:26:40.756604+0000","last_fullsized":"2023-10-30T14:26:40.756604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2095,"log_dups_size":0,"ondisk_log_size":2095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.030559456,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":129,"num_read_kb":29825,"num_write":1270,"num_write_kb":28412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"109'2030","reported_seq":4870,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754701+0000","last_change":"2023-10-30T14:26:33.286784+0000","last_active":"2023-10-30T14:26:40.754701+0000","last_peered":"2023-10-30T14:26:40.754701+0000","last_clean":"2023-10-30T14:26:40.754701+0000","last_became_active":"2023-10-30T13:24:00.704829+0000","last_became_peered":"2023-10-30T13:24:00.704829+0000","last_unstale":"2023-10-30T14:26:40.754701+0000","last_undegraded":"2023-10-30T14:26:40.754701+0000","last_fullsized":"2023-10-30T14:26:40.754701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2030,"log_dups_size":0,"ondisk_log_size":2030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:59:28.307918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.361435862,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":124,"num_read_kb":7939,"num_write":1184,"num_write_kb":24239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"109'2009","reported_seq":4867,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.903103+0000","last_change":"2023-10-30T14:26:34.648843+0000","last_active":"2023-10-30T14:26:39.903103+0000","last_peered":"2023-10-30T14:26:39.903103+0000","last_clean":"2023-10-30T14:26:39.903103+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T14:26:39.903103+0000","last_undegraded":"2023-10-30T14:26:39.903103+0000","last_fullsized":"2023-10-30T14:26:39.903103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2009,"log_dups_size":0,"ondisk_log_size":2009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:00:36.619771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.80853723300000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":41266,"num_write":1244,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"109'2209","reported_seq":5008,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.753370+0000","last_change":"2023-10-30T14:26:33.897829+0000","last_active":"2023-10-30T14:26:40.753370+0000","last_peered":"2023-10-30T14:26:40.753370+0000","last_clean":"2023-10-30T14:26:40.753370+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T14:26:40.753370+0000","last_undegraded":"2023-10-30T14:26:40.753370+0000","last_fullsized":"2023-10-30T14:26:40.753370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2209,"log_dups_size":0,"ondisk_log_size":2209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:03.177100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.94867293500000005,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":141,"num_read_kb":27429,"num_write":1290,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"109'2145","reported_seq":4994,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.902202+0000","last_change":"2023-10-30T14:26:34.555126+0000","last_active":"2023-10-30T14:26:39.902202+0000","last_peered":"2023-10-30T14:26:39.902202+0000","last_clean":"2023-10-30T14:26:39.902202+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T14:26:39.902202+0000","last_undegraded":"2023-10-30T14:26:39.902202+0000","last_fullsized":"2023-10-30T14:26:39.902202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2145,"log_dups_size":0,"ondisk_log_size":2145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:25.464527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.71511574600000005,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":146,"num_read_kb":25110,"num_write":1316,"num_write_kb":20043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"109'2109","reported_seq":4921,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893145+0000","last_change":"2023-10-30T14:26:36.369916+0000","last_active":"2023-10-30T14:26:39.893145+0000","last_peered":"2023-10-30T14:26:39.893145+0000","last_clean":"2023-10-30T14:26:39.893145+0000","last_became_active":"2023-10-30T13:24:00.989938+0000","last_became_peered":"2023-10-30T13:24:00.989938+0000","last_unstale":"2023-10-30T14:26:39.893145+0000","last_undegraded":"2023-10-30T14:26:39.893145+0000","last_fullsized":"2023-10-30T14:26:39.893145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2109,"log_dups_size":0,"ondisk_log_size":2109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:03.578741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.90238370000000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":124,"num_read_kb":25628,"num_write":1232,"num_write_kb":20540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"109'2137","reported_seq":4969,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.906989+0000","last_change":"2023-10-30T14:26:34.654560+0000","last_active":"2023-10-30T14:26:39.906989+0000","last_peered":"2023-10-30T14:26:39.906989+0000","last_clean":"2023-10-30T14:26:39.906989+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T14:26:39.906989+0000","last_undegraded":"2023-10-30T14:26:39.906989+0000","last_fullsized":"2023-10-30T14:26:39.906989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:56.758220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81280227199999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":24548,"num_write":1310,"num_write_kb":19821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"109'2073","reported_seq":4936,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.907671+0000","last_change":"2023-10-30T14:26:33.465676+0000","last_active":"2023-10-30T14:26:39.907671+0000","last_peered":"2023-10-30T14:26:39.907671+0000","last_clean":"2023-10-30T14:26:39.907671+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T14:26:39.907671+0000","last_undegraded":"2023-10-30T14:26:39.907671+0000","last_fullsized":"2023-10-30T14:26:39.907671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:02.527162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.53668512800000001,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":148,"num_read_kb":64678,"num_write":1273,"num_write_kb":32495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"109'2008","reported_seq":4883,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.906116+0000","last_change":"2023-10-30T14:26:33.399684+0000","last_active":"2023-10-30T14:26:39.906116+0000","last_peered":"2023-10-30T14:26:39.906116+0000","last_clean":"2023-10-30T14:26:39.906116+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T14:26:39.906116+0000","last_undegraded":"2023-10-30T14:26:39.906116+0000","last_fullsized":"2023-10-30T14:26:39.906116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2008,"log_dups_size":0,"ondisk_log_size":2008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:10:40.091601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.471346551,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":139,"num_read_kb":25550,"num_write":1254,"num_write_kb":16915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"109'2083","reported_seq":4928,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.764386+0000","last_change":"2023-10-30T14:26:33.495643+0000","last_active":"2023-10-30T14:26:40.764386+0000","last_peered":"2023-10-30T14:26:40.764386+0000","last_clean":"2023-10-30T14:26:40.764386+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T14:26:40.764386+0000","last_undegraded":"2023-10-30T14:26:40.764386+0000","last_fullsized":"2023-10-30T14:26:40.764386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:07:16.016021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.54183594099999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":142,"num_read_kb":52725,"num_write":1240,"num_write_kb":42463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"109'2223","reported_seq":5032,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893472+0000","last_change":"2023-10-30T14:26:36.506702+0000","last_active":"2023-10-30T14:26:39.893472+0000","last_peered":"2023-10-30T14:26:39.893472+0000","last_clean":"2023-10-30T14:26:39.893472+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T14:26:39.893472+0000","last_undegraded":"2023-10-30T14:26:39.893472+0000","last_fullsized":"2023-10-30T14:26:39.893472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:07.521185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.038919041,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":46739,"num_write":1326,"num_write_kb":33562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"109'2151","reported_seq":5003,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.755838+0000","last_change":"2023-10-30T14:26:34.440977+0000","last_active":"2023-10-30T14:26:40.755838+0000","last_peered":"2023-10-30T14:26:40.755838+0000","last_clean":"2023-10-30T14:26:40.755838+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T14:26:40.755838+0000","last_undegraded":"2023-10-30T14:26:40.755838+0000","last_fullsized":"2023-10-30T14:26:40.755838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2151,"log_dups_size":0,"ondisk_log_size":2151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.97549857799999995,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":149,"num_read_kb":13007,"num_write":1312,"num_write_kb":16233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"109'2267","reported_seq":5093,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.182798+0000","last_change":"2023-10-30T14:26:33.691289+0000","last_active":"2023-10-30T14:26:40.182798+0000","last_peered":"2023-10-30T14:26:40.182798+0000","last_clean":"2023-10-30T14:26:40.182798+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T14:26:40.182798+0000","last_undegraded":"2023-10-30T14:26:40.182798+0000","last_fullsized":"2023-10-30T14:26:40.182798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2267,"log_dups_size":0,"ondisk_log_size":2267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.432871758,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":161,"num_read_kb":14756,"num_write":1377,"num_write_kb":22909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"109'2090","reported_seq":4950,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754788+0000","last_change":"2023-10-30T14:26:33.787493+0000","last_active":"2023-10-30T14:26:40.754788+0000","last_peered":"2023-10-30T14:26:40.754788+0000","last_clean":"2023-10-30T14:26:40.754788+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T14:26:40.754788+0000","last_undegraded":"2023-10-30T14:26:40.754788+0000","last_fullsized":"2023-10-30T14:26:40.754788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:57.806855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.83845289999999995,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":149,"num_read_kb":40211,"num_write":1268,"num_write_kb":23485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"109'2348","reported_seq":5163,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.753302+0000","last_change":"2023-10-30T14:26:34.060688+0000","last_active":"2023-10-30T14:26:40.753302+0000","last_peered":"2023-10-30T14:26:40.753302+0000","last_clean":"2023-10-30T14:26:40.753302+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T14:26:40.753302+0000","last_undegraded":"2023-10-30T14:26:40.753302+0000","last_fullsized":"2023-10-30T14:26:40.753302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2348,"log_dups_size":0,"ondisk_log_size":2348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:47:21.386319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.109948846,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":170,"num_read_kb":39932,"num_write":1446,"num_write_kb":30153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"109'1996","reported_seq":4853,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.755097+0000","last_change":"2023-10-30T14:26:33.267575+0000","last_active":"2023-10-30T14:26:40.755097+0000","last_peered":"2023-10-30T14:26:40.755097+0000","last_clean":"2023-10-30T14:26:40.755097+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T14:26:40.755097+0000","last_undegraded":"2023-10-30T14:26:40.755097+0000","last_fullsized":"2023-10-30T14:26:40.755097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1996,"log_dups_size":0,"ondisk_log_size":1996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:03.867577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.34246037800000001,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":137,"num_read_kb":13706,"num_write":1196,"num_write_kb":12790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"109'2095","reported_seq":4929,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.905102+0000","last_change":"2023-10-30T14:26:34.604839+0000","last_active":"2023-10-30T14:26:39.905102+0000","last_peered":"2023-10-30T14:26:39.905102+0000","last_clean":"2023-10-30T14:26:39.905102+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T14:26:39.905102+0000","last_undegraded":"2023-10-30T14:26:39.905102+0000","last_fullsized":"2023-10-30T14:26:39.905102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2095,"log_dups_size":0,"ondisk_log_size":2095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:35:33.855326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.76369308499999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":125,"num_read_kb":30178,"num_write":1282,"num_write_kb":24413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"109'2215","reported_seq":5046,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.905189+0000","last_change":"2023-10-30T14:26:33.395186+0000","last_active":"2023-10-30T14:26:39.905189+0000","last_peered":"2023-10-30T14:26:39.905189+0000","last_clean":"2023-10-30T14:26:39.905189+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T14:26:39.905189+0000","last_undegraded":"2023-10-30T14:26:39.905189+0000","last_fullsized":"2023-10-30T14:26:39.905189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2215,"log_dups_size":0,"ondisk_log_size":2215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:31.161754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.46668427699999998,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":148,"num_read_kb":30111,"num_write":1332,"num_write_kb":26597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"109'2222","reported_seq":5013,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.756248+0000","last_change":"2023-10-30T14:26:34.474353+0000","last_active":"2023-10-30T14:26:40.756248+0000","last_peered":"2023-10-30T14:26:40.756248+0000","last_clean":"2023-10-30T14:26:40.756248+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T14:26:40.756248+0000","last_undegraded":"2023-10-30T14:26:40.756248+0000","last_fullsized":"2023-10-30T14:26:40.756248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2222,"log_dups_size":0,"ondisk_log_size":2222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.008362585,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":127,"num_read_kb":10449,"num_write":1298,"num_write_kb":11653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"109'2282","reported_seq":5084,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754139+0000","last_change":"2023-10-30T14:26:33.893164+0000","last_active":"2023-10-30T14:26:40.754139+0000","last_peered":"2023-10-30T14:26:40.754139+0000","last_clean":"2023-10-30T14:26:40.754139+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T14:26:40.754139+0000","last_undegraded":"2023-10-30T14:26:40.754139+0000","last_fullsized":"2023-10-30T14:26:40.754139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2282,"log_dups_size":0,"ondisk_log_size":2282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:42.538423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.94381626399999996,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":144,"num_read_kb":30506,"num_write":1376,"num_write_kb":26822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"109'2117","reported_seq":5081,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893182+0000","last_change":"2023-10-30T14:26:36.452306+0000","last_active":"2023-10-30T14:26:39.893182+0000","last_peered":"2023-10-30T14:26:39.893182+0000","last_clean":"2023-10-30T14:26:39.893182+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T14:26:39.893182+0000","last_undegraded":"2023-10-30T14:26:39.893182+0000","last_fullsized":"2023-10-30T14:26:39.893182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2117,"log_dups_size":0,"ondisk_log_size":2117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:47.201749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98440143999999996,"stat_sum":{"num_bytes":6,"num_objects":38,"num_object_clones":2,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":257,"num_read_kb":42146,"num_write":1297,"num_write_kb":25266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"109'2178","reported_seq":5041,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754854+0000","last_change":"2023-10-30T14:26:33.846175+0000","last_active":"2023-10-30T14:26:40.754854+0000","last_peered":"2023-10-30T14:26:40.754854+0000","last_clean":"2023-10-30T14:26:40.754854+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T14:26:40.754854+0000","last_undegraded":"2023-10-30T14:26:40.754854+0000","last_fullsized":"2023-10-30T14:26:40.754854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2178,"log_dups_size":0,"ondisk_log_size":2178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:34:46.488680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89723890399999995,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":164,"num_read_kb":14461,"num_write":1342,"num_write_kb":12663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"109'2057","reported_seq":4898,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893767+0000","last_change":"2023-10-30T14:26:36.518474+0000","last_active":"2023-10-30T14:26:39.893767+0000","last_peered":"2023-10-30T14:26:39.893767+0000","last_clean":"2023-10-30T14:26:39.893767+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T14:26:39.893767+0000","last_undegraded":"2023-10-30T14:26:39.893767+0000","last_fullsized":"2023-10-30T14:26:39.893767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2057,"log_dups_size":0,"ondisk_log_size":2057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:39:14.252516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0495642949999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":133,"num_read_kb":25337,"num_write":1220,"num_write_kb":15637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"109'2141","reported_seq":5019,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.908152+0000","last_change":"2023-10-30T14:26:33.433647+0000","last_active":"2023-10-30T14:26:39.908152+0000","last_peered":"2023-10-30T14:26:39.908152+0000","last_clean":"2023-10-30T14:26:39.908152+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T14:26:39.908152+0000","last_undegraded":"2023-10-30T14:26:39.908152+0000","last_fullsized":"2023-10-30T14:26:39.908152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2141,"log_dups_size":0,"ondisk_log_size":2141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:13:23.542810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.504458776,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":160,"num_read_kb":14034,"num_write":1340,"num_write_kb":12359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"109'2089","reported_seq":4940,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.183417+0000","last_change":"2023-10-30T14:26:33.601252+0000","last_active":"2023-10-30T14:26:40.183417+0000","last_peered":"2023-10-30T14:26:40.183417+0000","last_clean":"2023-10-30T14:26:40.183417+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T14:26:40.183417+0000","last_undegraded":"2023-10-30T14:26:40.183417+0000","last_fullsized":"2023-10-30T14:26:40.183417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2089,"log_dups_size":0,"ondisk_log_size":2089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.34262146900000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":128,"num_read_kb":22518,"num_write":1277,"num_write_kb":32107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"109'2030","reported_seq":4860,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893648+0000","last_change":"2023-10-30T14:26:36.555781+0000","last_active":"2023-10-30T14:26:39.893648+0000","last_peered":"2023-10-30T14:26:39.893648+0000","last_clean":"2023-10-30T14:26:39.893648+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T14:26:39.893648+0000","last_undegraded":"2023-10-30T14:26:39.893648+0000","last_fullsized":"2023-10-30T14:26:39.893648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2030,"log_dups_size":0,"ondisk_log_size":2030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:34.247640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.068284585,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":129,"num_read_kb":17081,"num_write":1202,"num_write_kb":15468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"109'2140","reported_seq":4964,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.756136+0000","last_change":"2023-10-30T14:26:34.496718+0000","last_active":"2023-10-30T14:26:40.756136+0000","last_peered":"2023-10-30T14:26:40.756136+0000","last_clean":"2023-10-30T14:26:40.756136+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T14:26:40.756136+0000","last_undegraded":"2023-10-30T14:26:40.756136+0000","last_fullsized":"2023-10-30T14:26:40.756136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2140,"log_dups_size":0,"ondisk_log_size":2140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.030536272,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":125,"num_read_kb":13655,"num_write":1323,"num_write_kb":17357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"109'2086","reported_seq":4956,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754485+0000","last_change":"2023-10-30T14:26:33.934508+0000","last_active":"2023-10-30T14:26:40.754485+0000","last_peered":"2023-10-30T14:26:40.754485+0000","last_clean":"2023-10-30T14:26:40.754485+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T14:26:40.754485+0000","last_undegraded":"2023-10-30T14:26:40.754485+0000","last_fullsized":"2023-10-30T14:26:40.754485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2086,"log_dups_size":0,"ondisk_log_size":2086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:25:07.586673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98488935700000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":140,"num_read_kb":17880,"num_write":1311,"num_write_kb":13902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"109'2377","reported_seq":5193,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754853+0000","last_change":"2023-10-30T14:26:33.231592+0000","last_active":"2023-10-30T14:26:40.754853+0000","last_peered":"2023-10-30T14:26:40.754853+0000","last_clean":"2023-10-30T14:26:40.754853+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T14:26:40.754853+0000","last_undegraded":"2023-10-30T14:26:40.754853+0000","last_fullsized":"2023-10-30T14:26:40.754853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2377,"log_dups_size":0,"ondisk_log_size":2377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:00:56.118846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30670995400000001,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":165,"num_read_kb":40193,"num_write":1443,"num_write_kb":35290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"109'2303","reported_seq":5125,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.904120+0000","last_change":"2023-10-30T14:26:33.357836+0000","last_active":"2023-10-30T14:26:39.904120+0000","last_peered":"2023-10-30T14:26:39.904120+0000","last_clean":"2023-10-30T14:26:39.904120+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T14:26:39.904120+0000","last_undegraded":"2023-10-30T14:26:39.904120+0000","last_fullsized":"2023-10-30T14:26:39.904120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2303,"log_dups_size":0,"ondisk_log_size":2303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:53:07.770089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42993166100000002,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":168,"num_read_kb":24860,"num_write":1383,"num_write_kb":20782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"109'2113","reported_seq":4956,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.762188+0000","last_change":"2023-10-30T14:26:33.546412+0000","last_active":"2023-10-30T14:26:40.762188+0000","last_peered":"2023-10-30T14:26:40.762188+0000","last_clean":"2023-10-30T14:26:40.762188+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T14:26:40.762188+0000","last_undegraded":"2023-10-30T14:26:40.762188+0000","last_fullsized":"2023-10-30T14:26:40.762188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:27.773478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.59246732999999996,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":146,"num_read_kb":30802,"num_write":1268,"num_write_kb":21419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"103'1028","reported_seq":1335,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.468163+0000","last_change":"2023-10-30T13:23:18.963933+0000","last_active":"2023-10-30T14:26:35.468163+0000","last_peered":"2023-10-30T14:26:35.468163+0000","last_clean":"2023-10-30T14:26:35.468163+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T14:26:35.468163+0000","last_undegraded":"2023-10-30T14:26:35.468163+0000","last_fullsized":"2023-10-30T14:26:35.468163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1028,"log_dups_size":0,"ondisk_log_size":1028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:48:58.196567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497506,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":222,"num_read_kb":501605,"num_write":1094,"num_write_kb":95785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"109'2155","reported_seq":4963,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.755848+0000","last_change":"2023-10-30T14:26:34.609800+0000","last_active":"2023-10-30T14:26:40.755848+0000","last_peered":"2023-10-30T14:26:40.755848+0000","last_clean":"2023-10-30T14:26:40.755848+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T14:26:40.755848+0000","last_undegraded":"2023-10-30T14:26:40.755848+0000","last_fullsized":"2023-10-30T14:26:40.755848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2155,"log_dups_size":0,"ondisk_log_size":2155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.1434485830000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":17524,"num_write":1294,"num_write_kb":15816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"108'1693","reported_seq":2127,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.898447+0000","last_change":"2023-10-30T13:23:18.976615+0000","last_active":"2023-10-30T14:26:34.898447+0000","last_peered":"2023-10-30T14:26:34.898447+0000","last_clean":"2023-10-30T14:26:34.898447+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T14:26:34.898447+0000","last_undegraded":"2023-10-30T14:26:34.898447+0000","last_fullsized":"2023-10-30T14:26:34.898447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1693,"log_dups_size":0,"ondisk_log_size":1693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:59.048396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29694842,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":392,"num_read_kb":410068,"num_write":1792,"num_write_kb":89526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"104'1557","reported_seq":2911,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.468125+0000","last_change":"2023-10-30T13:23:18.977357+0000","last_active":"2023-10-30T14:26:35.468125+0000","last_peered":"2023-10-30T14:26:35.468125+0000","last_clean":"2023-10-30T14:26:35.468125+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T14:26:35.468125+0000","last_undegraded":"2023-10-30T14:26:35.468125+0000","last_fullsized":"2023-10-30T14:26:35.468125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1557,"log_dups_size":0,"ondisk_log_size":1557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:53.502681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60855928,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":1296,"num_read_kb":556671,"num_write":1596,"num_write_kb":94675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"109'2327","reported_seq":5164,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893182+0000","last_change":"2023-10-30T14:26:36.357910+0000","last_active":"2023-10-30T14:26:39.893182+0000","last_peered":"2023-10-30T14:26:39.893182+0000","last_clean":"2023-10-30T14:26:39.893182+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T14:26:39.893182+0000","last_undegraded":"2023-10-30T14:26:39.893182+0000","last_fullsized":"2023-10-30T14:26:39.893182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2327,"log_dups_size":0,"ondisk_log_size":2327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:17.422938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89032229399999996,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":161,"num_read_kb":17698,"num_write":1449,"num_write_kb":26122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"109'970","reported_seq":1317,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:38.018514+0000","last_change":"2023-10-30T13:23:18.989299+0000","last_active":"2023-10-30T14:26:38.018514+0000","last_peered":"2023-10-30T14:26:38.018514+0000","last_clean":"2023-10-30T14:26:38.018514+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T14:26:38.018514+0000","last_undegraded":"2023-10-30T14:26:38.018514+0000","last_fullsized":"2023-10-30T14:26:38.018514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:44.510184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53640060,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":312,"num_read_kb":563254,"num_write":1076,"num_write_kb":96054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"109'2149","reported_seq":5008,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.182796+0000","last_change":"2023-10-30T14:26:33.629921+0000","last_active":"2023-10-30T14:26:40.182796+0000","last_peered":"2023-10-30T14:26:40.182796+0000","last_clean":"2023-10-30T14:26:40.182796+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T14:26:40.182796+0000","last_undegraded":"2023-10-30T14:26:40.182796+0000","last_fullsized":"2023-10-30T14:26:40.182796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2149,"log_dups_size":0,"ondisk_log_size":2149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.36920391600000002,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":152,"num_read_kb":51949,"num_write":1321,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"109'802","reported_seq":1135,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.965883+0000","last_change":"2023-10-30T13:23:18.989515+0000","last_active":"2023-10-30T14:26:36.965883+0000","last_peered":"2023-10-30T14:26:36.965883+0000","last_clean":"2023-10-30T14:26:36.965883+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T14:26:36.965883+0000","last_undegraded":"2023-10-30T14:26:36.965883+0000","last_fullsized":"2023-10-30T14:26:36.965883+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":802,"log_dups_size":0,"ondisk_log_size":802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:21:35.133762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943130,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":319,"num_read_kb":417394,"num_write":917,"num_write_kb":71614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"109'2454","reported_seq":5184,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.753585+0000","last_change":"2023-10-30T14:26:33.980471+0000","last_active":"2023-10-30T14:26:40.753585+0000","last_peered":"2023-10-30T14:26:40.753585+0000","last_clean":"2023-10-30T14:26:40.753585+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T14:26:40.753585+0000","last_undegraded":"2023-10-30T14:26:40.753585+0000","last_fullsized":"2023-10-30T14:26:40.753585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2454,"log_dups_size":0,"ondisk_log_size":2454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:40.292098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0302270280000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":152,"num_read_kb":10055,"num_write":1351,"num_write_kb":13264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"103'849","reported_seq":1154,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.726762+0000","last_change":"2023-10-30T13:23:18.991454+0000","last_active":"2023-10-30T14:26:35.726762+0000","last_peered":"2023-10-30T14:26:35.726762+0000","last_clean":"2023-10-30T14:26:35.726762+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T14:26:35.726762+0000","last_undegraded":"2023-10-30T14:26:35.726762+0000","last_fullsized":"2023-10-30T14:26:35.726762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:13.450999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":291,"num_read_kb":402257,"num_write":947,"num_write_kb":71494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"109'2085","reported_seq":4930,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.756546+0000","last_change":"2023-10-30T14:26:34.598085+0000","last_active":"2023-10-30T14:26:40.756546+0000","last_peered":"2023-10-30T14:26:40.756546+0000","last_clean":"2023-10-30T14:26:40.756546+0000","last_became_active":"2023-10-30T13:24:00.990019+0000","last_became_peered":"2023-10-30T13:24:00.990019+0000","last_unstale":"2023-10-30T14:26:40.756546+0000","last_undegraded":"2023-10-30T14:26:40.756546+0000","last_fullsized":"2023-10-30T14:26:40.756546+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2085,"log_dups_size":0,"ondisk_log_size":2085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.1322467139999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":132,"num_read_kb":30385,"num_write":1263,"num_write_kb":18224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"108'995","reported_seq":1256,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.183570+0000","last_change":"2023-10-30T13:23:18.982337+0000","last_active":"2023-10-30T14:26:40.183570+0000","last_peered":"2023-10-30T14:26:40.183570+0000","last_clean":"2023-10-30T14:26:40.183570+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T14:26:40.183570+0000","last_undegraded":"2023-10-30T14:26:40.183570+0000","last_fullsized":"2023-10-30T14:26:40.183570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":995,"log_dups_size":0,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":208,"num_read_kb":224450,"num_write":1057,"num_write_kb":46972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"109'2149","reported_seq":4979,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.764408+0000","last_change":"2023-10-30T14:26:33.534876+0000","last_active":"2023-10-30T14:26:40.764408+0000","last_peered":"2023-10-30T14:26:40.764408+0000","last_clean":"2023-10-30T14:26:40.764408+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T14:26:40.764408+0000","last_undegraded":"2023-10-30T14:26:40.764408+0000","last_fullsized":"2023-10-30T14:26:40.764408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2149,"log_dups_size":0,"ondisk_log_size":2149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:17.987617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.58089010799999996,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":141,"num_read_kb":34327,"num_write":1294,"num_write_kb":27493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"107'994","reported_seq":1308,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.899816+0000","last_change":"2023-10-30T13:23:18.969882+0000","last_active":"2023-10-30T14:26:34.899816+0000","last_peered":"2023-10-30T14:26:34.899816+0000","last_clean":"2023-10-30T14:26:34.899816+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T14:26:34.899816+0000","last_undegraded":"2023-10-30T14:26:34.899816+0000","last_fullsized":"2023-10-30T14:26:34.899816+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":994,"log_dups_size":0,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:17.093448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":290,"num_read_kb":355466,"num_write":1083,"num_write_kb":67490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"109'2165","reported_seq":4984,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.903886+0000","last_change":"2023-10-30T14:26:34.585946+0000","last_active":"2023-10-30T14:26:39.903886+0000","last_peered":"2023-10-30T14:26:39.903886+0000","last_clean":"2023-10-30T14:26:39.903886+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T14:26:39.903886+0000","last_undegraded":"2023-10-30T14:26:39.903886+0000","last_fullsized":"2023-10-30T14:26:39.903886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2165,"log_dups_size":0,"ondisk_log_size":2165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:03:21.700327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.74553484199999998,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":143,"num_read_kb":29369,"num_write":1268,"num_write_kb":21738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"103'701","reported_seq":1071,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.755623+0000","last_change":"2023-10-30T13:23:18.971632+0000","last_active":"2023-10-30T14:26:40.755623+0000","last_peered":"2023-10-30T14:26:40.755623+0000","last_clean":"2023-10-30T14:26:40.755623+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T14:26:40.755623+0000","last_undegraded":"2023-10-30T14:26:40.755623+0000","last_fullsized":"2023-10-30T14:26:40.755623+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331682,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":392,"num_read_kb":480410,"num_write":807,"num_write_kb":87841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"109'2150","reported_seq":5009,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.755471+0000","last_change":"2023-10-30T14:26:34.422972+0000","last_active":"2023-10-30T14:26:40.755471+0000","last_peered":"2023-10-30T14:26:40.755471+0000","last_clean":"2023-10-30T14:26:40.755471+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T14:26:40.755471+0000","last_undegraded":"2023-10-30T14:26:40.755471+0000","last_fullsized":"2023-10-30T14:26:40.755471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2150,"log_dups_size":0,"ondisk_log_size":2150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.95729191499999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":151,"num_read_kb":31378,"num_write":1306,"num_write_kb":33173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"103'851","reported_seq":1111,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.726707+0000","last_change":"2023-10-30T13:23:18.978420+0000","last_active":"2023-10-30T14:26:35.726707+0000","last_peered":"2023-10-30T14:26:35.726707+0000","last_clean":"2023-10-30T14:26:35.726707+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T14:26:35.726707+0000","last_undegraded":"2023-10-30T14:26:35.726707+0000","last_fullsized":"2023-10-30T14:26:35.726707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:10:44.511282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53927368,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":208,"num_read_kb":306859,"num_write":914,"num_write_kb":62457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"109'2293","reported_seq":5083,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754710+0000","last_change":"2023-10-30T14:26:33.299265+0000","last_active":"2023-10-30T14:26:40.754710+0000","last_peered":"2023-10-30T14:26:40.754710+0000","last_clean":"2023-10-30T14:26:40.754710+0000","last_became_active":"2023-10-30T13:24:00.706894+0000","last_became_peered":"2023-10-30T13:24:00.706894+0000","last_unstale":"2023-10-30T14:26:40.754710+0000","last_undegraded":"2023-10-30T14:26:40.754710+0000","last_fullsized":"2023-10-30T14:26:40.754710+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2293,"log_dups_size":0,"ondisk_log_size":2293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:58:25.208615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.3738494,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":149,"num_read_kb":1539,"num_write":1375,"num_write_kb":12931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"103'727","reported_seq":1002,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.898294+0000","last_change":"2023-10-30T13:23:18.972707+0000","last_active":"2023-10-30T14:26:34.898294+0000","last_peered":"2023-10-30T14:26:34.898294+0000","last_clean":"2023-10-30T14:26:34.898294+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T14:26:34.898294+0000","last_undegraded":"2023-10-30T14:26:34.898294+0000","last_fullsized":"2023-10-30T14:26:34.898294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":727,"log_dups_size":0,"ondisk_log_size":727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:11:08.140135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":234,"num_read_kb":298146,"num_write":829,"num_write_kb":54985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"109'2096","reported_seq":4966,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.904709+0000","last_change":"2023-10-30T14:26:33.432572+0000","last_active":"2023-10-30T14:26:39.904709+0000","last_peered":"2023-10-30T14:26:39.904709+0000","last_clean":"2023-10-30T14:26:39.904709+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T14:26:39.904709+0000","last_undegraded":"2023-10-30T14:26:39.904709+0000","last_fullsized":"2023-10-30T14:26:39.904709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2096,"log_dups_size":0,"ondisk_log_size":2096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:47:50.528827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50412874500000004,"stat_sum":{"num_bytes":0,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":148,"num_read_kb":43506,"num_write":1375,"num_write_kb":26417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"103'825","reported_seq":1170,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.894070+0000","last_change":"2023-10-30T13:23:18.948296+0000","last_active":"2023-10-30T14:26:34.894070+0000","last_peered":"2023-10-30T14:26:34.894070+0000","last_clean":"2023-10-30T14:26:34.894070+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T14:26:34.894070+0000","last_undegraded":"2023-10-30T14:26:34.894070+0000","last_fullsized":"2023-10-30T14:26:34.894070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:55.308329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":481,"num_read_kb":229594,"num_write":1003,"num_write_kb":42855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"109'2334","reported_seq":5162,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.902794+0000","last_change":"2023-10-30T14:26:34.549537+0000","last_active":"2023-10-30T14:26:39.902794+0000","last_peered":"2023-10-30T14:26:39.902794+0000","last_clean":"2023-10-30T14:26:39.902794+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T14:26:39.902794+0000","last_undegraded":"2023-10-30T14:26:39.902794+0000","last_fullsized":"2023-10-30T14:26:39.902794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2334,"log_dups_size":0,"ondisk_log_size":2334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:07.630163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70955538100000004,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":166,"num_read_kb":25446,"num_write":1448,"num_write_kb":32435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"103'995","reported_seq":1279,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.233490+0000","last_change":"2023-10-30T13:23:18.962343+0000","last_active":"2023-10-30T14:26:36.233490+0000","last_peered":"2023-10-30T14:26:36.233490+0000","last_clean":"2023-10-30T14:26:36.233490+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T14:26:36.233490+0000","last_undegraded":"2023-10-30T14:26:36.233490+0000","last_fullsized":"2023-10-30T14:26:36.233490+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":995,"log_dups_size":0,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:01:36.082164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":214,"num_read_kb":373366,"num_write":1150,"num_write_kb":71086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"109'2238","reported_seq":5027,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.906071+0000","last_change":"2023-10-30T14:26:34.736009+0000","last_active":"2023-10-30T14:26:39.906071+0000","last_peered":"2023-10-30T14:26:39.906071+0000","last_clean":"2023-10-30T14:26:39.906071+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T14:26:39.906071+0000","last_undegraded":"2023-10-30T14:26:39.906071+0000","last_fullsized":"2023-10-30T14:26:39.906071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2238,"log_dups_size":0,"ondisk_log_size":2238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:19.838741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89413299700000004,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":142,"num_read_kb":18968,"num_write":1278,"num_write_kb":24747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"105'626","reported_seq":937,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.895299+0000","last_change":"2023-10-30T13:23:18.949081+0000","last_active":"2023-10-30T14:26:34.895299+0000","last_peered":"2023-10-30T14:26:34.895299+0000","last_clean":"2023-10-30T14:26:34.895299+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T14:26:34.895299+0000","last_undegraded":"2023-10-30T14:26:34.895299+0000","last_fullsized":"2023-10-30T14:26:34.895299+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:00.356885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":280,"num_read_kb":492842,"num_write":679,"num_write_kb":83547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"109'2096","reported_seq":4970,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.907595+0000","last_change":"2023-10-30T14:26:34.726845+0000","last_active":"2023-10-30T14:26:39.907595+0000","last_peered":"2023-10-30T14:26:39.907595+0000","last_clean":"2023-10-30T14:26:39.907595+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T14:26:39.907595+0000","last_undegraded":"2023-10-30T14:26:39.907595+0000","last_fullsized":"2023-10-30T14:26:39.907595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2096,"log_dups_size":0,"ondisk_log_size":2096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:53.224813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.88483857600000004,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":152,"num_read_kb":61046,"num_write":1310,"num_write_kb":36653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"103'731","reported_seq":1026,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.183652+0000","last_change":"2023-10-30T13:23:18.970780+0000","last_active":"2023-10-30T14:26:40.183652+0000","last_peered":"2023-10-30T14:26:40.183652+0000","last_clean":"2023-10-30T14:26:40.183652+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T14:26:40.183652+0000","last_undegraded":"2023-10-30T14:26:40.183652+0000","last_fullsized":"2023-10-30T14:26:40.183652+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364699,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":233,"num_read_kb":370768,"num_write":815,"num_write_kb":71238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"109'2240","reported_seq":5101,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.752909+0000","last_change":"2023-10-30T14:26:33.934502+0000","last_active":"2023-10-30T14:26:40.752909+0000","last_peered":"2023-10-30T14:26:40.752909+0000","last_clean":"2023-10-30T14:26:40.752909+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T14:26:40.752909+0000","last_undegraded":"2023-10-30T14:26:40.752909+0000","last_fullsized":"2023-10-30T14:26:40.752909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2240,"log_dups_size":0,"ondisk_log_size":2240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:46.397667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98441546899999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":178,"num_read_kb":29706,"num_write":1375,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"103'1078","reported_seq":1360,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.894848+0000","last_change":"2023-10-30T13:23:18.977920+0000","last_active":"2023-10-30T14:26:34.894848+0000","last_peered":"2023-10-30T14:26:34.894848+0000","last_clean":"2023-10-30T14:26:34.894848+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T14:26:34.894848+0000","last_undegraded":"2023-10-30T14:26:34.894848+0000","last_fullsized":"2023-10-30T14:26:34.894848+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:07:59.139948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50332230,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":210,"num_read_kb":406971,"num_write":1089,"num_write_kb":71553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"109'2040","reported_seq":4891,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.904390+0000","last_change":"2023-10-30T14:26:34.688764+0000","last_active":"2023-10-30T14:26:39.904390+0000","last_peered":"2023-10-30T14:26:39.904390+0000","last_clean":"2023-10-30T14:26:39.904390+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T14:26:39.904390+0000","last_undegraded":"2023-10-30T14:26:39.904390+0000","last_fullsized":"2023-10-30T14:26:39.904390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:36:07.116174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.84782311300000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":122,"num_read_kb":1096,"num_write":1268,"num_write_kb":17447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":286,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.722977+0000","last_change":"2023-10-30T13:23:08.208031+0000","last_active":"2023-10-30T14:26:35.722977+0000","last_peered":"2023-10-30T14:26:35.722977+0000","last_clean":"2023-10-30T14:26:35.722977+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T14:26:35.722977+0000","last_undegraded":"2023-10-30T14:26:35.722977+0000","last_fullsized":"2023-10-30T14:26:35.722977+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:07.179866+0000","last_clean_scrub_stamp":"2023-10-30T13:23:07.179866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:22.691367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"108'849","reported_seq":1187,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723476+0000","last_change":"2023-10-30T13:23:18.979328+0000","last_active":"2023-10-30T14:26:35.723476+0000","last_peered":"2023-10-30T14:26:35.723476+0000","last_clean":"2023-10-30T14:26:35.723476+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T14:26:35.723476+0000","last_undegraded":"2023-10-30T14:26:35.723476+0000","last_fullsized":"2023-10-30T14:26:35.723476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:04.942593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331714,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":319,"num_read_kb":439640,"num_write":936,"num_write_kb":75464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"109'2198","reported_seq":4985,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.894132+0000","last_change":"2023-10-30T14:26:36.472084+0000","last_active":"2023-10-30T14:26:39.894132+0000","last_peered":"2023-10-30T14:26:39.894132+0000","last_clean":"2023-10-30T14:26:39.894132+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T14:26:39.894132+0000","last_undegraded":"2023-10-30T14:26:39.894132+0000","last_fullsized":"2023-10-30T14:26:39.894132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2198,"log_dups_size":0,"ondisk_log_size":2198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:48:10.995137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.003812562,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":127,"num_read_kb":26261,"num_write":1294,"num_write_kb":24882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"103'1186","reported_seq":1499,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.722877+0000","last_change":"2023-10-30T13:23:18.985318+0000","last_active":"2023-10-30T14:26:35.722877+0000","last_peered":"2023-10-30T14:26:35.722877+0000","last_clean":"2023-10-30T14:26:35.722877+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T14:26:35.722877+0000","last_undegraded":"2023-10-30T14:26:35.722877+0000","last_fullsized":"2023-10-30T14:26:35.722877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1186,"log_dups_size":0,"ondisk_log_size":1186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:16.754084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":256,"num_read_kb":488007,"num_write":1246,"num_write_kb":83351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"109'1979","reported_seq":4844,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.755060+0000","last_change":"2023-10-30T14:26:34.386961+0000","last_active":"2023-10-30T14:26:40.755060+0000","last_peered":"2023-10-30T14:26:40.755060+0000","last_clean":"2023-10-30T14:26:40.755060+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T14:26:40.755060+0000","last_undegraded":"2023-10-30T14:26:40.755060+0000","last_fullsized":"2023-10-30T14:26:40.755060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1979,"log_dups_size":0,"ondisk_log_size":1979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.92159720499999997,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":129,"num_read_kb":13045,"num_write":1222,"num_write_kb":11637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"108'1950","reported_seq":3195,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.894297+0000","last_change":"2023-10-30T13:23:18.949565+0000","last_active":"2023-10-30T14:26:34.894297+0000","last_peered":"2023-10-30T14:26:34.894297+0000","last_clean":"2023-10-30T14:26:34.894297+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T14:26:34.894297+0000","last_undegraded":"2023-10-30T14:26:34.894297+0000","last_fullsized":"2023-10-30T14:26:34.894297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1950,"log_dups_size":0,"ondisk_log_size":1950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:55:46.029368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40060913,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":1167,"num_read_kb":359975,"num_write":2020,"num_write_kb":69890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"109'2261","reported_seq":5075,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.183111+0000","last_change":"2023-10-30T14:26:33.665105+0000","last_active":"2023-10-30T14:26:40.183111+0000","last_peered":"2023-10-30T14:26:40.183111+0000","last_clean":"2023-10-30T14:26:40.183111+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T14:26:40.183111+0000","last_undegraded":"2023-10-30T14:26:40.183111+0000","last_fullsized":"2023-10-30T14:26:40.183111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2261,"log_dups_size":0,"ondisk_log_size":2261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40657480699999998,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":145,"num_read_kb":55105,"num_write":1354,"num_write_kb":28005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"103'1488","reported_seq":1794,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.726817+0000","last_change":"2023-10-30T13:23:18.984535+0000","last_active":"2023-10-30T14:26:35.726817+0000","last_peered":"2023-10-30T14:26:35.726817+0000","last_clean":"2023-10-30T14:26:35.726817+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T14:26:35.726817+0000","last_undegraded":"2023-10-30T14:26:35.726817+0000","last_fullsized":"2023-10-30T14:26:35.726817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1488,"log_dups_size":0,"ondisk_log_size":1488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:28.405405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":235,"num_read_kb":509467,"num_write":1537,"num_write_kb":83943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"109'2208","reported_seq":5033,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.763537+0000","last_change":"2023-10-30T14:26:33.550991+0000","last_active":"2023-10-30T14:26:40.763537+0000","last_peered":"2023-10-30T14:26:40.763537+0000","last_clean":"2023-10-30T14:26:40.763537+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T14:26:40.763537+0000","last_undegraded":"2023-10-30T14:26:40.763537+0000","last_fullsized":"2023-10-30T14:26:40.763537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2208,"log_dups_size":0,"ondisk_log_size":2208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:34.572699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.59694472600000004,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":143,"num_read_kb":13239,"num_write":1306,"num_write_kb":17120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"103'610","reported_seq":922,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.183381+0000","last_change":"2023-10-30T13:23:18.958210+0000","last_active":"2023-10-30T14:26:40.183381+0000","last_peered":"2023-10-30T14:26:40.183381+0000","last_clean":"2023-10-30T14:26:40.183381+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T14:26:40.183381+0000","last_undegraded":"2023-10-30T14:26:40.183381+0000","last_fullsized":"2023-10-30T14:26:40.183381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":230,"num_read_kb":520969,"num_write":671,"num_write_kb":95828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"109'2180","reported_seq":4999,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754358+0000","last_change":"2023-10-30T14:26:34.025524+0000","last_active":"2023-10-30T14:26:40.754358+0000","last_peered":"2023-10-30T14:26:40.754358+0000","last_clean":"2023-10-30T14:26:40.754358+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T14:26:40.754358+0000","last_undegraded":"2023-10-30T14:26:40.754358+0000","last_fullsized":"2023-10-30T14:26:40.754358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2180,"log_dups_size":0,"ondisk_log_size":2180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:45.716921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.075105449,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":60418,"num_write":1286,"num_write_kb":36950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"109'1047","reported_seq":1332,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:36.966097+0000","last_change":"2023-10-30T13:23:18.985185+0000","last_active":"2023-10-30T14:26:36.966097+0000","last_peered":"2023-10-30T14:26:36.966097+0000","last_clean":"2023-10-30T14:26:36.966097+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T14:26:36.966097+0000","last_undegraded":"2023-10-30T14:26:36.966097+0000","last_fullsized":"2023-10-30T14:26:36.966097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1047,"log_dups_size":0,"ondisk_log_size":1047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:36.230072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44106025,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":214,"num_read_kb":377318,"num_write":1089,"num_write_kb":69046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"109'2210","reported_seq":5045,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.183220+0000","last_change":"2023-10-30T14:26:33.629915+0000","last_active":"2023-10-30T14:26:40.183220+0000","last_peered":"2023-10-30T14:26:40.183220+0000","last_clean":"2023-10-30T14:26:40.183220+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T14:26:40.183220+0000","last_undegraded":"2023-10-30T14:26:40.183220+0000","last_fullsized":"2023-10-30T14:26:40.183220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.36741958800000002,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":157,"num_read_kb":35362,"num_write":1332,"num_write_kb":31772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"109'1127","reported_seq":1422,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.183586+0000","last_change":"2023-10-30T13:23:18.978085+0000","last_active":"2023-10-30T14:26:40.183586+0000","last_peered":"2023-10-30T14:26:40.183586+0000","last_clean":"2023-10-30T14:26:40.183586+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T14:26:40.183586+0000","last_undegraded":"2023-10-30T14:26:40.183586+0000","last_fullsized":"2023-10-30T14:26:40.183586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1127,"log_dups_size":0,"ondisk_log_size":1127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":247,"num_read_kb":365780,"num_write":1783,"num_write_kb":65025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"109'2088","reported_seq":4922,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754955+0000","last_change":"2023-10-30T14:26:33.252060+0000","last_active":"2023-10-30T14:26:40.754955+0000","last_peered":"2023-10-30T14:26:40.754955+0000","last_clean":"2023-10-30T14:26:40.754955+0000","last_became_active":"2023-10-30T13:24:01.001412+0000","last_became_peered":"2023-10-30T13:24:01.001412+0000","last_unstale":"2023-10-30T14:26:40.754955+0000","last_undegraded":"2023-10-30T14:26:40.754955+0000","last_fullsized":"2023-10-30T14:26:40.754955+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:37:55.891565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.32700652899999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":137,"num_read_kb":5811,"num_write":1214,"num_write_kb":16143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"103'468","reported_seq":774,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:34.899192+0000","last_change":"2023-10-30T13:23:18.970566+0000","last_active":"2023-10-30T14:26:34.899192+0000","last_peered":"2023-10-30T14:26:34.899192+0000","last_clean":"2023-10-30T14:26:34.899192+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T14:26:34.899192+0000","last_undegraded":"2023-10-30T14:26:34.899192+0000","last_fullsized":"2023-10-30T14:26:34.899192+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:03.851889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":281,"num_read_kb":442743,"num_write":555,"num_write_kb":70797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"109'2186","reported_seq":5020,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.755398+0000","last_change":"2023-10-30T14:26:34.649272+0000","last_active":"2023-10-30T14:26:40.755398+0000","last_peered":"2023-10-30T14:26:40.755398+0000","last_clean":"2023-10-30T14:26:40.755398+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T14:26:40.755398+0000","last_undegraded":"2023-10-30T14:26:40.755398+0000","last_fullsized":"2023-10-30T14:26:40.755398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2186,"log_dups_size":0,"ondisk_log_size":2186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.183697322,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":144,"num_read_kb":25445,"num_write":1326,"num_write_kb":20705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"103'544","reported_seq":810,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.722683+0000","last_change":"2023-10-30T13:23:18.986394+0000","last_active":"2023-10-30T14:26:35.722683+0000","last_peered":"2023-10-30T14:26:35.722683+0000","last_clean":"2023-10-30T14:26:35.722683+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T14:26:35.722683+0000","last_undegraded":"2023-10-30T14:26:35.722683+0000","last_fullsized":"2023-10-30T14:26:35.722683+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:59.986981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":236,"num_read_kb":274544,"num_write":603,"num_write_kb":42177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"109'2123","reported_seq":4972,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.764634+0000","last_change":"2023-10-30T14:26:33.516141+0000","last_active":"2023-10-30T14:26:40.764634+0000","last_peered":"2023-10-30T14:26:40.764634+0000","last_clean":"2023-10-30T14:26:40.764634+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T14:26:40.764634+0000","last_undegraded":"2023-10-30T14:26:40.764634+0000","last_fullsized":"2023-10-30T14:26:40.764634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2123,"log_dups_size":0,"ondisk_log_size":2123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:05.930508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.56267233699999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":144,"num_read_kb":34655,"num_write":1307,"num_write_kb":20329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"103'1061","reported_seq":1333,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.182981+0000","last_change":"2023-10-30T13:23:18.963905+0000","last_active":"2023-10-30T14:26:40.182981+0000","last_peered":"2023-10-30T14:26:40.182981+0000","last_clean":"2023-10-30T14:26:40.182981+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T14:26:40.182981+0000","last_undegraded":"2023-10-30T14:26:40.182981+0000","last_fullsized":"2023-10-30T14:26:40.182981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1061,"log_dups_size":0,"ondisk_log_size":1061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":237,"num_read_kb":249991,"num_write":1213,"num_write_kb":43538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"109'2271","reported_seq":5058,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893369+0000","last_change":"2023-10-30T14:26:36.405912+0000","last_active":"2023-10-30T14:26:39.893369+0000","last_peered":"2023-10-30T14:26:39.893369+0000","last_clean":"2023-10-30T14:26:39.893369+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T14:26:39.893369+0000","last_undegraded":"2023-10-30T14:26:39.893369+0000","last_fullsized":"2023-10-30T14:26:39.893369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2271,"log_dups_size":0,"ondisk_log_size":2271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:29:31.853578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.93815086199999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":129,"num_read_kb":1227,"num_write":1352,"num_write_kb":12536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"109'1474","reported_seq":1811,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.183268+0000","last_change":"2023-10-30T13:23:18.952302+0000","last_active":"2023-10-30T14:26:40.183268+0000","last_peered":"2023-10-30T14:26:40.183268+0000","last_clean":"2023-10-30T14:26:40.183268+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T14:26:40.183268+0000","last_undegraded":"2023-10-30T14:26:40.183268+0000","last_fullsized":"2023-10-30T14:26:40.183268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1474,"log_dups_size":0,"ondisk_log_size":1474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":321,"num_read_kb":438308,"num_write":2003,"num_write_kb":98279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"109'2096","reported_seq":4936,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893680+0000","last_change":"2023-10-30T14:26:36.417706+0000","last_active":"2023-10-30T14:26:39.893680+0000","last_peered":"2023-10-30T14:26:39.893680+0000","last_clean":"2023-10-30T14:26:39.893680+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T14:26:39.893680+0000","last_undegraded":"2023-10-30T14:26:39.893680+0000","last_fullsized":"2023-10-30T14:26:39.893680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2096,"log_dups_size":0,"ondisk_log_size":2096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:50:45.036636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.94968995099999998,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":141,"num_read_kb":11852,"num_write":1237,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"103'1064","reported_seq":1381,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723232+0000","last_change":"2023-10-30T13:23:18.984912+0000","last_active":"2023-10-30T14:26:35.723232+0000","last_peered":"2023-10-30T14:26:35.723232+0000","last_clean":"2023-10-30T14:26:35.723232+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T14:26:35.723232+0000","last_undegraded":"2023-10-30T14:26:35.723232+0000","last_fullsized":"2023-10-30T14:26:35.723232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:27:16.205042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":296,"num_read_kb":429426,"num_write":1159,"num_write_kb":75926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"109'2164","reported_seq":5000,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.904433+0000","last_change":"2023-10-30T14:26:33.347987+0000","last_active":"2023-10-30T14:26:39.904433+0000","last_peered":"2023-10-30T14:26:39.904433+0000","last_clean":"2023-10-30T14:26:39.904433+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T14:26:39.904433+0000","last_undegraded":"2023-10-30T14:26:39.904433+0000","last_fullsized":"2023-10-30T14:26:39.904433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2164,"log_dups_size":0,"ondisk_log_size":2164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:25.445527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.419978554,"stat_sum":{"num_bytes":5,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":137,"num_read_kb":40109,"num_write":1287,"num_write_kb":25283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"109'2180","reported_seq":4988,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.753501+0000","last_change":"2023-10-30T14:26:34.023924+0000","last_active":"2023-10-30T14:26:40.753501+0000","last_peered":"2023-10-30T14:26:40.753501+0000","last_clean":"2023-10-30T14:26:40.753501+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T14:26:40.753501+0000","last_undegraded":"2023-10-30T14:26:40.753501+0000","last_fullsized":"2023-10-30T14:26:40.753501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2180,"log_dups_size":0,"ondisk_log_size":2180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:40:49.561271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.073619836,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":129,"num_read_kb":12975,"num_write":1318,"num_write_kb":20116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"103'734","reported_seq":1023,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.723353+0000","last_change":"2023-10-30T13:23:18.993372+0000","last_active":"2023-10-30T14:26:35.723353+0000","last_peered":"2023-10-30T14:26:35.723353+0000","last_clean":"2023-10-30T14:26:35.723353+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T14:26:35.723353+0000","last_undegraded":"2023-10-30T14:26:35.723353+0000","last_fullsized":"2023-10-30T14:26:35.723353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":734,"log_dups_size":0,"ondisk_log_size":734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:41.370678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":304,"num_read_kb":264369,"num_write":877,"num_write_kb":48007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"109'2642","reported_seq":5327,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.753695+0000","last_change":"2023-10-30T14:26:33.971887+0000","last_active":"2023-10-30T14:26:40.753695+0000","last_peered":"2023-10-30T14:26:40.753695+0000","last_clean":"2023-10-30T14:26:40.753695+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T14:26:40.753695+0000","last_undegraded":"2023-10-30T14:26:40.753695+0000","last_fullsized":"2023-10-30T14:26:40.753695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2642,"log_dups_size":0,"ondisk_log_size":2642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:02.739787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.021840552,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":171,"num_read_kb":40337,"num_write":1456,"num_write_kb":34441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"103'1179","reported_seq":1480,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.725364+0000","last_change":"2023-10-30T13:23:18.987633+0000","last_active":"2023-10-30T14:26:35.725364+0000","last_peered":"2023-10-30T14:26:35.725364+0000","last_clean":"2023-10-30T14:26:35.725364+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T14:26:35.725364+0000","last_undegraded":"2023-10-30T14:26:35.725364+0000","last_fullsized":"2023-10-30T14:26:35.725364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1179,"log_dups_size":0,"ondisk_log_size":1179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:09:33.324898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":255,"num_read_kb":502436,"num_write":1236,"num_write_kb":99955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"109'2028","reported_seq":4897,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.756778+0000","last_change":"2023-10-30T14:26:34.532204+0000","last_active":"2023-10-30T14:26:40.756778+0000","last_peered":"2023-10-30T14:26:40.756778+0000","last_clean":"2023-10-30T14:26:40.756778+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T14:26:40.756778+0000","last_undegraded":"2023-10-30T14:26:40.756778+0000","last_fullsized":"2023-10-30T14:26:40.756778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2028,"log_dups_size":0,"ondisk_log_size":2028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.015373858,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":13189,"num_write":1249,"num_write_kb":15084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"103'779","reported_seq":1033,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-30T14:26:35.523404+0000","last_change":"2023-10-30T13:23:18.958470+0000","last_active":"2023-10-30T14:26:35.523404+0000","last_peered":"2023-10-30T14:26:35.523404+0000","last_clean":"2023-10-30T14:26:35.523404+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T14:26:35.523404+0000","last_undegraded":"2023-10-30T14:26:35.523404+0000","last_fullsized":"2023-10-30T14:26:35.523404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":779,"log_dups_size":0,"ondisk_log_size":779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:21.507744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":159,"num_read_kb":369044,"num_write":787,"num_write_kb":79251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"109'2192","reported_seq":5141,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.894212+0000","last_change":"2023-10-30T14:26:36.567193+0000","last_active":"2023-10-30T14:26:39.894212+0000","last_peered":"2023-10-30T14:26:39.894212+0000","last_clean":"2023-10-30T14:26:39.894212+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T14:26:39.894212+0000","last_undegraded":"2023-10-30T14:26:39.894212+0000","last_fullsized":"2023-10-30T14:26:39.894212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2192,"log_dups_size":0,"ondisk_log_size":2192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:03:26.764254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.043790548,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":262,"num_read_kb":42307,"num_write":1336,"num_write_kb":30630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"103'1010","reported_seq":1330,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.756373+0000","last_change":"2023-10-30T13:23:18.980952+0000","last_active":"2023-10-30T14:26:40.756373+0000","last_peered":"2023-10-30T14:26:40.756373+0000","last_clean":"2023-10-30T14:26:40.756373+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T14:26:40.756373+0000","last_undegraded":"2023-10-30T14:26:40.756373+0000","last_fullsized":"2023-10-30T14:26:40.756373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1010,"log_dups_size":0,"ondisk_log_size":1010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777798,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":313,"num_read_kb":340671,"num_write":1085,"num_write_kb":63851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":11,"num_objects":2377,"num_object_clones":2,"num_object_copies":4754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":9389,"num_read_kb":1724409,"num_write":83589,"num_write_kb":1459751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138527,"ondisk_log_size":138527,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1284731319,"num_objects":1569,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1569,"num_whiteouts":0,"num_read":10652,"num_read_kb":12762809,"num_write":35648,"num_write_kb":2343500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2569617408,"data_stored":2569462638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31992,"ondisk_log_size":31992,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575615,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590984,"kb_used_data":389120,"kb_used_omap":1,"kb_used_meta":201854,"kb_avail":93780856,"statfs":{"total":96636764160,"available":96031596544,"internally_reserved":0,"allocated":398458880,"data_stored":397803489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":206698955},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.73999999999999999,"15min":0.79700000000000004},"min":{"1min":0.41399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.82699999999999996,"15min":0.91000000000000003},"min":{"1min":0.68100000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.873}]},{"osd":1,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.80700000000000005,"15min":0.86599999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.288,"5min":2.452,"15min":2.452},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.79000000000000004,"15min":0.83999999999999997},"min":{"1min":0.56499999999999995,"5min":0.495,"15min":0.47399999999999998},"max":{"1min":3.7160000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.746}]},{"osd":2,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.75800000000000001,"15min":0.81999999999999995},"min":{"1min":0.45200000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.96299999999999997,"5min":1.169,"15min":1.4350000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.84799999999999998,"15min":0.93600000000000005},"min":{"1min":0.52700000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2210000000000001,"5min":1.2929999999999999,"15min":1.5569999999999999},"last":0.92000000000000004}]},{"osd":3,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94899999999999995,"15min":0.95099999999999996},"min":{"1min":0.60499999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.0790000000000002,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.026,"15min":1.0029999999999999},"min":{"1min":0.67000000000000004,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.7869999999999999,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":1.089}]},{"osd":4,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.91900000000000004,"15min":0.90200000000000002},"min":{"1min":0.63200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.6629999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.202,"5min":1.0660000000000001,"15min":1.042},"min":{"1min":0.84099999999999997,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":3.6899999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":1.014}]},{"osd":5,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.139,"15min":1.0880000000000001},"min":{"1min":0.75800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.4179999999999999,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.127,"15min":1.1599999999999999},"min":{"1min":0.88300000000000001,"5min":0.436,"15min":0.436},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":1.1140000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1379999999999999,"15min":1.1140000000000001},"min":{"1min":0.50800000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":6.3070000000000004,"5min":6.3070000000000004,"15min":6.3070000000000004},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.054,"5min":1.1279999999999999,"15min":1.181},"min":{"1min":0.65700000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.0469999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575618,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505392,"kb_used_data":339688,"kb_used_omap":1,"kb_used_meta":165694,"kb_avail":93866448,"statfs":{"total":96636764160,"available":96119242752,"internally_reserved":0,"allocated":347840512,"data_stored":347188935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169671111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.75700000000000001,"15min":0.80100000000000005},"min":{"1min":0.51100000000000001,"5min":0.46700000000000003,"15min":0.375},"max":{"1min":0.89600000000000002,"5min":1.306,"15min":1.363},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.88900000000000001,"15min":0.95299999999999996},"min":{"1min":0.53500000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3979999999999999,"5min":1.4099999999999999,"15min":1.4370000000000001},"last":0.996}]},{"osd":1,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.72299999999999998,"15min":0.77400000000000002},"min":{"1min":0.54100000000000004,"5min":0.46999999999999997,"15min":0.46800000000000003},"max":{"1min":0.875,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.47399999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.127,"5min":1.706,"15min":1.706},"last":0.73699999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.81299999999999994,"15min":0.82999999999999996},"min":{"1min":0.57699999999999996,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.1679999999999999,"5min":1.29,"15min":1.571},"last":0.876},{"interface":"front","average":{"1min":0.872,"5min":0.96899999999999997,"15min":0.93700000000000006},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.286,"5min":5.4480000000000004,"15min":5.4480000000000004},"last":1.125}]},{"osd":3,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0229999999999999,"15min":1.0329999999999999},"min":{"1min":0.68000000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.103,"15min":1.1160000000000001},"min":{"1min":0.68899999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.23,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":1.27}]},{"osd":4,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.90300000000000002,"15min":0.90700000000000003},"min":{"1min":0.70799999999999996,"5min":0.434,"15min":0.434},"max":{"1min":1.276,"5min":1.3640000000000001,"15min":1.4339999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.002,"15min":1.0309999999999999},"min":{"1min":0.71999999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.153,"5min":1.595,"15min":1.595},"last":1.0800000000000001}]},{"osd":5,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.159,"15min":1.03},"min":{"1min":0.77400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.2889999999999999,"5min":3.0219999999999998,"15min":3.0219999999999998},"last":1.3600000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.0089999999999999,"15min":1.0509999999999999},"min":{"1min":0.75600000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.135,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":1.2410000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:26:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.204,"15min":1.196},"min":{"1min":1.048,"5min":0.58699999999999997,"15min":0.52600000000000002},"max":{"1min":1.3740000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1599999999999999,"15min":1.1359999999999999},"min":{"1min":0.83399999999999996,"5min":0.752,"15min":0.63700000000000001},"max":{"1min":1.3440000000000001,"5min":2.806,"15min":2.806},"last":1.1679999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575615,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566236,"kb_used_data":368592,"kb_used_omap":1,"kb_used_meta":197630,"kb_avail":93805604,"statfs":{"total":96636764160,"available":96056938496,"internally_reserved":0,"allocated":377438208,"data_stored":376788129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202373574},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.72399999999999998,"15min":0.86499999999999999},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.40999999999999998},"max":{"1min":1.0860000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.95099999999999996,"15min":0.95799999999999996},"min":{"1min":0.60499999999999998,"5min":0.58699999999999997,"15min":0.57199999999999995},"max":{"1min":1.2410000000000001,"5min":1.617,"15min":1.9219999999999999},"last":1.1359999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.81999999999999995,"15min":0.84099999999999997},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.77000000000000002,"15min":0.83799999999999997},"min":{"1min":0.52800000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.97099999999999997,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.85399999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.88100000000000001,"15min":0.88500000000000001},"min":{"1min":0.59699999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.2589999999999999,"5min":1.331,"15min":1.3999999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81299999999999994,"15min":0.84799999999999998},"min":{"1min":0.59999999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.331,"5min":1.331,"15min":1.796},"last":0.67500000000000004}]},{"osd":3,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0049999999999999,"15min":1.006},"min":{"1min":0.63700000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92000000000000004,"15min":0.999},"min":{"1min":0.71899999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.1930000000000001,"5min":1.3460000000000001,"15min":2.0030000000000001},"last":1.2470000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.88400000000000001,"15min":0.879},"min":{"1min":0.56899999999999995,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.2729999999999999,"5min":2.8170000000000002,"15min":2.8170000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.159,"5min":1.024,"15min":1.03},"min":{"1min":0.86799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.6299999999999999,"5min":2.2400000000000002,"15min":2.262},"last":1.3400000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:26:00 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.0880000000000001,"15min":1.1299999999999999},"min":{"1min":0.80900000000000005,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.1499999999999999,"5min":2.3090000000000002,"15min":2.7949999999999999},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.0229999999999999,"15min":1.056},"min":{"1min":0.71899999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.49,"5min":2.161,"15min":2.1629999999999998},"last":1.0649999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:26:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0209999999999999,"15min":1.091},"min":{"1min":0.78100000000000003,"5min":0.317,"15min":0.317},"max":{"1min":1.3440000000000001,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.1299999999999999,"15min":1.153},"min":{"1min":0.95099999999999996,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.4019999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.29}]}]},{"osd":4,"up_from":13,"seq":55834575615,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464828,"kb_used_data":267120,"kb_used_omap":1,"kb_used_meta":197694,"kb_avail":93907012,"statfs":{"total":96636764160,"available":96160780288,"internally_reserved":0,"allocated":273530880,"data_stored":272868074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202439110},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"min":{"1min":0.48799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1259999999999999,"5min":1.3500000000000001,"15min":1.446},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.71099999999999997,"15min":0.82299999999999995},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.875,"5min":1.659,"15min":1.659},"last":0.86699999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.56799999999999995,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.141,"5min":1.589,"15min":1.589},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.84399999999999997,"15min":0.84699999999999998},"min":{"1min":0.58999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.075,"5min":1.532,"15min":1.532},"last":0.92900000000000005}]},{"osd":2,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.76100000000000001,"15min":0.85999999999999999},"min":{"1min":0.55800000000000005,"5min":0.5,"15min":0.5},"max":{"1min":0.94899999999999995,"5min":1.607,"15min":1.607},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84699999999999998,"15min":0.92200000000000004},"min":{"1min":0.57199999999999995,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.218,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.80900000000000005}]},{"osd":3,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.99099999999999999,"15min":0.97499999999999998},"min":{"1min":0.66300000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.242,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.099,"15min":1.0780000000000001},"min":{"1min":0.879,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3480000000000001,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.1120000000000001}]},{"osd":5,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.98899999999999999,"15min":1.042},"min":{"1min":0.77000000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.3009999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.65400000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.1279999999999999,"5min":1.3220000000000001,"15min":1.3280000000000001},"last":1.012}]},{"osd":6,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.97899999999999998,"15min":0.96099999999999997},"min":{"1min":0.78800000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.3100000000000001,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.042,"15min":1.0700000000000001},"min":{"1min":0.78300000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.238,"5min":1.677,"15min":1.728},"last":1.0820000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.137,"15min":1.1040000000000001},"min":{"1min":1.0820000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.417,"5min":1.4750000000000001,"15min":1.518},"last":1.167},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.204,"15min":1.1799999999999999},"min":{"1min":1.056,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4670000000000001,"5min":1.587,"15min":1.587},"last":1.2090000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575613,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593540,"kb_used_data":382008,"kb_used_omap":1,"kb_used_meta":211518,"kb_avail":93778300,"statfs":{"total":96636764160,"available":96028979200,"internally_reserved":0,"allocated":391176192,"data_stored":390510799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":216594888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38100000000000001,"15min":0.40600000000000003},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71799999999999997,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.308},"max":{"1min":0.747,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.47699999999999998}]},{"osd":1,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":0.76500000000000001,"5min":1.403,"15min":1.403},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.438,"5min":0.51700000000000002,"15min":0.54800000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65800000000000003,"5min":1.752,"15min":1.752},"last":0.55000000000000004}]},{"osd":2,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56399999999999995,"15min":0.59999999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69499999999999995,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.41699999999999998,"5min":0.29999999999999999,"15min":0.29299999999999998},"max":{"1min":0.78000000000000003,"5min":1.73,"15min":1.73},"last":0.65100000000000002}]},{"osd":4,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.80300000000000005},"min":{"1min":0.53600000000000003,"5min":0.379,"15min":0.379},"max":{"1min":1.6519999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.80400000000000005,"15min":0.82399999999999995},"min":{"1min":0.55800000000000005,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.679,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.76100000000000001}]},{"osd":5,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77400000000000002,"15min":0.83299999999999996},"min":{"1min":0.61499999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98899999999999999,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.81000000000000005,"15min":0.85699999999999998},"min":{"1min":0.58399999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.071,"5min":1.486,"15min":1.486},"last":0.70799999999999996}]},{"osd":6,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.85199999999999998,"15min":0.88400000000000001},"min":{"1min":0.55600000000000005,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.048,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.878,"5min":0.91500000000000004,"15min":0.93999999999999995},"min":{"1min":0.71999999999999997,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0409999999999999,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.88600000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83199999999999996,"15min":0.86199999999999999},"min":{"1min":0.64200000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.1100000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79100000000000004,"15min":0.82599999999999996},"min":{"1min":0.56299999999999994,"5min":0.501,"15min":0.501},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.72699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299416,"kb_used_data":200268,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94072424,"statfs":{"total":96636764160,"available":96330162176,"internally_reserved":0,"allocated":205074432,"data_stored":204423364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101513670},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.433,"15min":0.45800000000000002},"min":{"1min":0.254,"5min":0.246,"15min":0.23899999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.89300000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":0.94599999999999995,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":0.432}]},{"osd":1,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.42199999999999999,"15min":0.44700000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72999999999999998,"5min":1.956,"15min":1.956},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.36199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.55000000000000004}]},{"osd":3,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81999999999999995,"5min":1.988,"15min":1.988},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49099999999999999,"15min":0.51800000000000002},"min":{"1min":0.32300000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.90100000000000002,"5min":2.335,"15min":2.335},"last":0.498}]},{"osd":4,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.60899999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5409999999999999,"5min":2.302,"15min":2.302},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84299999999999997,"15min":0.84099999999999997},"min":{"1min":0.625,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6879999999999999,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.73299999999999998}]},{"osd":5,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.77100000000000002,"15min":0.80600000000000005},"min":{"1min":0.64500000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82699999999999996,"15min":0.84499999999999997},"min":{"1min":0.73099999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.587,"5min":1.875,"15min":1.875},"last":0.69099999999999995}]},{"osd":6,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.81599999999999995,"15min":0.82799999999999996},"min":{"1min":0.61299999999999999,"5min":0.502,"15min":0.48099999999999998},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84499999999999997,"15min":0.86799999999999999},"min":{"1min":0.60999999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.429,"5min":1.921,"15min":1.921},"last":0.67400000000000004}]},{"osd":7,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.86099999999999999,"15min":0.86399999999999999},"min":{"1min":0.66100000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.131,"5min":4.5739999999999998,"15min":4.5739999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.871,"5min":0.88900000000000001,"15min":0.91700000000000004},"min":{"1min":0.64300000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6080000000000001,"5min":2.375,"15min":2.375},"last":0.83999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575616,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490684,"kb_used_data":329076,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":93881156,"statfs":{"total":96636764160,"available":96134303744,"internally_reserved":0,"allocated":336973824,"data_stored":336325526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165476806},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.438,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.504,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.51600000000000001},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.47799999999999998,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.46999999999999997,"15min":0.48399999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54800000000000004,"5min":1.954,"15min":1.954},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.45800000000000002,"15min":0.47799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63200000000000001,"5min":2.274,"15min":2.274},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53600000000000003,"15min":0.54900000000000004},"min":{"1min":0.35599999999999998,"5min":0.35399999999999998,"15min":0.35099999999999998},"max":{"1min":0.60099999999999998,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.436,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.70199999999999996,"5min":2.625,"15min":2.625},"last":0.58299999999999996}]},{"osd":4,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.80200000000000005,"15min":0.83999999999999997},"min":{"1min":0.56499999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93300000000000005,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.82299999999999995,"15min":0.88},"min":{"1min":0.629,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.91200000000000003,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.94999999999999996}]},{"osd":5,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82399999999999995,"15min":0.878},"min":{"1min":0.58099999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94799999999999995,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.85099999999999998,"15min":0.90100000000000002},"min":{"1min":0.60399999999999998,"5min":0.499,"15min":0.499},"max":{"1min":0.879,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.72999999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.91000000000000003,"15min":0.84199999999999997},"min":{"1min":0.73799999999999999,"5min":0.624,"15min":0.56499999999999995},"max":{"1min":1.0289999999999999,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":1.01},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.81299999999999994,"15min":0.89700000000000002},"min":{"1min":0.56200000000000006,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.95799999999999996,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.98199999999999998}]},{"osd":7,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.80600000000000005,"15min":0.82999999999999996},"min":{"1min":0.59599999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.95999999999999996,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.82799999999999996,"15min":0.85199999999999998},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93799999999999994,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.88500000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575613,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421192,"kb_used_data":245756,"kb_used_omap":1,"kb_used_meta":175422,"kb_avail":93950648,"statfs":{"total":96636764160,"available":96205463552,"internally_reserved":0,"allocated":251654144,"data_stored":251001762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179632582},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.38100000000000001,"15min":0.40500000000000003},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54400000000000004,"5min":0.66100000000000003,"15min":0.94999999999999996},"last":0.313},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.49299999999999999,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":1.0229999999999999},"last":0.47599999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.45300000000000001,"15min":0.46700000000000003},"min":{"1min":0.307,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":1.05,"15min":1.05},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.47099999999999997,"15min":0.498},"min":{"1min":0.32900000000000001,"5min":0.30499999999999999,"15min":0.29099999999999998},"max":{"1min":0.59699999999999998,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.501}]},{"osd":3,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52900000000000003,"15min":0.54900000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.34499999999999997},"max":{"1min":0.73299999999999998,"5min":1.236,"15min":1.236},"last":0.436},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51600000000000001,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.74399999999999999,"5min":1.054,"15min":1.054},"last":0.45600000000000002}]},{"osd":4,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.78600000000000003,"15min":0.80300000000000005},"min":{"1min":0.65900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":0.88800000000000001,"5min":1.103,"15min":1.29},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76600000000000001,"15min":0.81000000000000005},"min":{"1min":0.59299999999999997,"5min":0.53800000000000003,"15min":0.48399999999999999},"max":{"1min":0.95999999999999996,"5min":1.012,"15min":1.2210000000000001},"last":0.68899999999999995}]},{"osd":5,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.78700000000000003,"15min":0.81200000000000006},"min":{"1min":0.64700000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.93899999999999995,"5min":1.0509999999999999,"15min":1.347},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.81599999999999995,"15min":0.85199999999999998},"min":{"1min":0.63,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.95899999999999996,"5min":1.2130000000000001,"15min":1.3939999999999999},"last":0.77600000000000002}]},{"osd":6,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81499999999999995,"15min":0.84399999999999997},"min":{"1min":0.64500000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.95799999999999996,"5min":1.222,"15min":1.486},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79700000000000004,"15min":0.83399999999999996},"min":{"1min":0.64400000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0149999999999999,"5min":1.113,"15min":1.1619999999999999},"last":0.89500000000000002}]},{"osd":7,"last update":"Mon Oct 30 14:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.81699999999999995,"15min":0.85099999999999998},"min":{"1min":0.625,"5min":0.625,"15min":0.58599999999999997},"max":{"1min":0.97599999999999998,"5min":1.2509999999999999,"15min":1.29},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84199999999999997,"15min":0.872},"min":{"1min":0.70799999999999996,"5min":0.61799999999999999,"15min":0.57299999999999995},"max":{"1min":0.96099999999999997,"5min":1.173,"15min":1.4419999999999999},"last":0.91300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249647104,"data_stored":249627559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335572992,"data_stored":335549765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203661312,"data_stored":203639529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389746688,"data_stored":389725939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":272097280,"data_stored":272083209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376025088,"data_stored":376004294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":346419200,"data_stored":346404076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":396447744,"data_stored":396428267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":11,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:26:44.839 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.839 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.840 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.840 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.840 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.841 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.841 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.841 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.842 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.842 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.843 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.843 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.843 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.844 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.844 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.844 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.845 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.845 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.845 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.846 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.846 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.846 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.847 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.847 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.847 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.848 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.848 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.849 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.849 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.849 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.850 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.850 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.850 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.851 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.851 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.851 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.852 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.852 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.852 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.853 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.853 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.853 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.854 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.854 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.854 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.855 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.855 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.855 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.856 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.856 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.856 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.857 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.857 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.857 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.858 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.858 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.858 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.859 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.859 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.859 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.860 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.860 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.861 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.861 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.861 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-30T13:23:07.179866+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=7, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.862 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.862 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.862 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.863 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.863 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.863 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.864 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.864 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.864 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.865 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.865 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.865 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.866 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.866 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.866 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.867 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.867 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.867 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.868 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.868 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.869 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.869 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.869 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.870 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.870 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.870 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.871 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.871 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.872 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.872 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.872 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.873 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:26:44.873 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T14:27:04.874 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:27:05.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9f59e640 1 Processor -- start 2023-10-30T14:27:05.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9f59e640 1 -- start start 2023-10-30T14:27:05.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9f59e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3da01538f0 0x7f3da0159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:05.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9f59e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da00ca6a0 con 0x7f3da00cefa0 2023-10-30T14:27:05.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9f59e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da015d300 con 0x7f3da015cd50 2023-10-30T14:27:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9f59e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da015d5e0 con 0x7f3da01538f0 2023-10-30T14:27:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3da00cefa0 0x7f3da01640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37850/0 (socket says 172.21.15.46:37850) 2023-10-30T14:27:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9ed9d640 1 -- 172.21.15.46:0/1148416635 learned_addr learned my addr 172.21.15.46:0/1148416635 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:27:05.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9e59c640 1 --2- 172.21.15.46:0/1148416635 >> v2:172.21.15.154:3300/0 conn(0x7f3da01538f0 0x7f3da0159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639063746 0 0) 0x7f3da00ca6a0 con 0x7f3da00cefa0 2023-10-30T14:27:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d88003530 con 0x7f3da00cefa0 2023-10-30T14:27:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859148661 0 0) 0x7f3da015d300 con 0x7f3da015cd50 2023-10-30T14:27:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da00ca6a0 con 0x7f3da015cd50 2023-10-30T14:27:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1335732680 0 0) 0x7f3d88003530 con 0x7f3da00cefa0 2023-10-30T14:27:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 >> v2:172.21.15.154:3300/0 conn(0x7f3da01538f0 msgr2=0x7f3da0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 --2- 172.21.15.46:0/1148416635 >> v2:172.21.15.154:3300/0 conn(0x7f3da01538f0 0x7f3da0159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 >> v1:172.21.15.154:6789/0 conn(0x7f3da015cd50 legacy=0x7f3da00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da015d8c0 con 0x7f3da00cefa0 2023-10-30T14:27:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f3d98001a00 con 0x7f3da00cefa0 2023-10-30T14:27:05.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 >> v1:172.21.15.46:6789/0 conn(0x7f3da00cefa0 legacy=0x7f3da01640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:05.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d88003b10 0x7f3d88003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:05.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d880047d0 0x7f3d88004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:05.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d88005190 0x7f3d8800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:05.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da015d5e0 con 0x7f3d880047d0 2023-10-30T14:27:05.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8800bf80 con 0x7f3d88003b10 2023-10-30T14:27:05.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8800c140 con 0x7f3d88005190 2023-10-30T14:27:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9dd9b640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d880047d0 0x7f3d88004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9e59c640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d88003b10 0x7f3d88003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.021+0000 7f3d9ed9d640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d88005190 0x7f3d8800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9dd9b640 1 -- 172.21.15.46:0/1148416635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d88005190 msgr2=0x7f3d8800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9dd9b640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d88005190 0x7f3d8800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9dd9b640 1 -- 172.21.15.46:0/1148416635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d88003b10 msgr2=0x7f3d88003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9dd9b640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d88003b10 0x7f3d88003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:05.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9dd9b640 1 -- 172.21.15.46:0/1148416635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d88003530 con 0x7f3d880047d0 2023-10-30T14:27:05.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9e59c640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d88003b10 0x7f3d88003f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:27:05.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9ed9d640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d88005190 0x7f3d8800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:27:05.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9ed9d640 1 -- 172.21.15.46:0/1148416635 reap_dead start 2023-10-30T14:27:05.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9dd9b640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d880047d0 0x7f3d88004c50 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f3d94007fd0 tx=0x7f3d9401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33510a084d8ff7e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:05.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d940052d0 con 0x7f3d880047d0 2023-10-30T14:27:05.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d940a7050 con 0x7f3d880047d0 2023-10-30T14:27:05.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9d59a640 1 -- 172.21.15.46:0/1148416635 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94003600 con 0x7f3d880047d0 2023-10-30T14:27:05.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9f59e640 1 -- 172.21.15.46:0/1148416635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d880047d0 msgr2=0x7f3d88004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:05.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.025+0000 7f3d9f59e640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d880047d0 0x7f3d88004c50 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f3d94007fd0 tx=0x7f3d9401c500 comp rx=0 tx=0).stop 2023-10-30T14:27:05.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 -- 172.21.15.46:0/1148416635 shutdown_connections 2023-10-30T14:27:05.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 --2- 172.21.15.46:0/1148416635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d880047d0 0x7f3d88004c50 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:05.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 -- 172.21.15.46:0/1148416635 >> 172.21.15.46:0/1148416635 conn(0x7f3da006a640 msgr2=0x7f3da0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:05.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 -- 172.21.15.46:0/1148416635 shutdown_connections 2023-10-30T14:27:05.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 -- 172.21.15.46:0/1148416635 wait complete. 2023-10-30T14:27:05.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 Processor -- start 2023-10-30T14:27:05.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 -- start start 2023-10-30T14:27:05.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d880047d0 0x7f3da01f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:05.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da014f690 0x7f3da01f18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:05.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3da00c8f90 0x7f3da01477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:05.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da00b38b0 con 0x7f3da014f690 2023-10-30T14:27:05.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da01f2770 con 0x7f3d880047d0 2023-10-30T14:27:05.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9f59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da014d920 con 0x7f3da00c8f90 2023-10-30T14:27:05.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da014f690 0x7f3da01f18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:05.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da014f690 0x7f3da01f18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57818/0 (socket says 172.21.15.46:57818) 2023-10-30T14:27:05.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9dd9b640 1 -- 172.21.15.46:0/2082369667 learned_addr learned my addr 172.21.15.46:0/2082369667 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:27:05.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9e59c640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d880047d0 0x7f3da01f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:05.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.029+0000 7f3d9ed9d640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3da00c8f90 0x7f3da01477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:05.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d9dd9b640 1 -- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3da00c8f90 msgr2=0x7f3da01477c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:05.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d9dd9b640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3da00c8f90 0x7f3da01477c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:05.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d9dd9b640 1 -- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d880047d0 msgr2=0x7f3da01f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:05.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d9dd9b640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d880047d0 0x7f3da01f2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:05.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d9dd9b640 1 -- 172.21.15.46:0/2082369667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da00b3ae0 con 0x7f3da014f690 2023-10-30T14:27:05.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d9e59c640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d880047d0 0x7f3da01f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:27:05.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d9dd9b640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da014f690 0x7f3da01f18b0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f3d94005040 tx=0x7f3d88005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:05.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d8effd640 1 -- 172.21.15.46:0/2082369667 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94003d40 con 0x7f3da014f690 2023-10-30T14:27:05.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d8effd640 1 -- 172.21.15.46:0/2082369667 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d940067e0 con 0x7f3da014f690 2023-10-30T14:27:05.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d8effd640 1 -- 172.21.15.46:0/2082369667 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94003ea0 con 0x7f3da014f690 2023-10-30T14:27:05.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d9f59e640 1 -- 172.21.15.46:0/2082369667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da014d360 con 0x7f3da014f690 2023-10-30T14:27:05.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d9f59e640 1 -- 172.21.15.46:0/2082369667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da01f2400 con 0x7f3da014f690 2023-10-30T14:27:05.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d9f59e640 1 -- 172.21.15.46:0/2082369667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da01f2590 con 0x7f3da014f690 2023-10-30T14:27:05.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.033+0000 7f3d8effd640 1 -- 172.21.15.46:0/2082369667 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3d94006b90 con 0x7f3da014f690 2023-10-30T14:27:05.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.037+0000 7f3d8effd640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d800664a0 0x7f3d80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:05.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.037+0000 7f3d8effd640 1 -- 172.21.15.46:0/2082369667 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f3d9411e1f0 con 0x7f3da014f690 2023-10-30T14:27:05.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.037+0000 7f3d9e59c640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d800664a0 0x7f3d80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:05.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.037+0000 7f3d9e59c640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d800664a0 0x7f3d80068960 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f3d90009b10 tx=0x7f3d90008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:05.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.041+0000 7f3d8effd640 1 -- 172.21.15.46:0/2082369667 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3d940ea990 con 0x7f3da014f690 2023-10-30T14:27:05.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.197+0000 7f3d9f59e640 1 -- 172.21.15.46:0/2082369667 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3da0159a90 con 0x7f3d800664a0 2023-10-30T14:27:05.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.217+0000 7f3d8effd640 1 -- 172.21.15.46:0/2082369667 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+292610 (secure 0 0 0) 0x7f3d90013a50 con 0x7f3d800664a0 2023-10-30T14:27:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:27:05.227 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T14:27:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.221+0000 7f3d9f59e640 1 -- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d800664a0 msgr2=0x7f3d80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.221+0000 7f3d9f59e640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d800664a0 0x7f3d80068960 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f3d90009b10 tx=0x7f3d90008040 comp rx=0 tx=0).stop 2023-10-30T14:27:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.221+0000 7f3d9f59e640 1 -- 172.21.15.46:0/2082369667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da014f690 msgr2=0x7f3da01f18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.221+0000 7f3d9f59e640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da014f690 0x7f3da01f18b0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f3d94005040 tx=0x7f3d88005800 comp rx=0 tx=0).stop 2023-10-30T14:27:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.221+0000 7f3d9f59e640 1 -- 172.21.15.46:0/2082369667 shutdown_connections 2023-10-30T14:27:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.221+0000 7f3d9f59e640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f3d800664a0 0x7f3d80068960 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.221+0000 7f3d9f59e640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3da00c8f90 0x7f3da01477c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.221+0000 7f3d9f59e640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da014f690 0x7f3da01f18b0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.221+0000 7f3d9f59e640 1 --2- 172.21.15.46:0/2082369667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d880047d0 0x7f3da01f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:05.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.221+0000 7f3d9f59e640 1 -- 172.21.15.46:0/2082369667 >> 172.21.15.46:0/2082369667 conn(0x7f3da006a640 msgr2=0x7f3da00c5590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.221+0000 7f3d9f59e640 1 -- 172.21.15.46:0/2082369667 shutdown_connections 2023-10-30T14:27:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:05.221+0000 7f3d9f59e640 1 -- 172.21.15.46:0/2082369667 wait complete. 2023-10-30T14:27:05.246 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":2022,"stamp":"2023-10-30T14:27:04.137149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1285691727,"num_objects":3948,"num_object_clones":2,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":20157,"num_read_kb":14487318,"num_write":119403,"num_write_kb":3806032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12730482,"num_omap_keys":22652,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172989,"ondisk_log_size":172989,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3945476,"kb_used_data":2522372,"kb_used_omap":12,"kb_used_meta":1423027,"kb_avail":751029244,"statfs":{"total":773094113280,"available":769053945856,"internally_reserved":0,"allocated":2582908928,"data_stored":2577659229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1457180216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":153021,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6843620,"num_omap_keys":12186,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004584"},"pg_stats":[{"pgid":"3.3f","version":"109'2161","reported_seq":4975,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.908021+0000","last_change":"2023-10-30T14:26:34.688745+0000","last_active":"2023-10-30T14:26:39.908021+0000","last_peered":"2023-10-30T14:26:39.908021+0000","last_clean":"2023-10-30T14:26:39.908021+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T14:26:39.908021+0000","last_undegraded":"2023-10-30T14:26:39.908021+0000","last_fullsized":"2023-10-30T14:26:39.908021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2161,"log_dups_size":0,"ondisk_log_size":2161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.84725695499999998,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":124,"num_read_kb":10662,"num_write":1276,"num_write_kb":15944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"109'1971","reported_seq":4851,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.183422+0000","last_change":"2023-10-30T14:26:33.665197+0000","last_active":"2023-10-30T14:26:40.183422+0000","last_peered":"2023-10-30T14:26:40.183422+0000","last_clean":"2023-10-30T14:26:40.183422+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T14:26:40.183422+0000","last_undegraded":"2023-10-30T14:26:40.183422+0000","last_fullsized":"2023-10-30T14:26:40.183422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":1971,"log_dups_size":0,"ondisk_log_size":1971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.406256491,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":133,"num_read_kb":21158,"num_write":1230,"num_write_kb":16620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"109'2095","reported_seq":4929,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.756604+0000","last_change":"2023-10-30T14:26:34.547863+0000","last_active":"2023-10-30T14:26:40.756604+0000","last_peered":"2023-10-30T14:26:40.756604+0000","last_clean":"2023-10-30T14:26:40.756604+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T14:26:40.756604+0000","last_undegraded":"2023-10-30T14:26:40.756604+0000","last_fullsized":"2023-10-30T14:26:40.756604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2095,"log_dups_size":0,"ondisk_log_size":2095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.030559456,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":129,"num_read_kb":29825,"num_write":1270,"num_write_kb":28412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"109'2030","reported_seq":4871,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754758+0000","last_change":"2023-10-30T14:26:33.286784+0000","last_active":"2023-10-30T14:26:40.754758+0000","last_peered":"2023-10-30T14:26:40.754758+0000","last_clean":"2023-10-30T14:26:40.754758+0000","last_became_active":"2023-10-30T13:24:00.704829+0000","last_became_peered":"2023-10-30T13:24:00.704829+0000","last_unstale":"2023-10-30T14:26:40.754758+0000","last_undegraded":"2023-10-30T14:26:40.754758+0000","last_fullsized":"2023-10-30T14:26:40.754758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2030,"log_dups_size":0,"ondisk_log_size":2030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.361435862,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":124,"num_read_kb":7939,"num_write":1184,"num_write_kb":24239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"109'2009","reported_seq":4868,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.903222+0000","last_change":"2023-10-30T14:26:34.648843+0000","last_active":"2023-10-30T14:26:39.903222+0000","last_peered":"2023-10-30T14:26:39.903222+0000","last_clean":"2023-10-30T14:26:39.903222+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T14:26:39.903222+0000","last_undegraded":"2023-10-30T14:26:39.903222+0000","last_fullsized":"2023-10-30T14:26:39.903222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2009,"log_dups_size":0,"ondisk_log_size":2009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.80853723300000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":41266,"num_write":1244,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"109'2209","reported_seq":5009,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754003+0000","last_change":"2023-10-30T14:26:33.897829+0000","last_active":"2023-10-30T14:26:40.754003+0000","last_peered":"2023-10-30T14:26:40.754003+0000","last_clean":"2023-10-30T14:26:40.754003+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T14:26:40.754003+0000","last_undegraded":"2023-10-30T14:26:40.754003+0000","last_fullsized":"2023-10-30T14:26:40.754003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2209,"log_dups_size":0,"ondisk_log_size":2209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.94867293500000005,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":141,"num_read_kb":27429,"num_write":1290,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"109'2145","reported_seq":4995,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.902346+0000","last_change":"2023-10-30T14:26:34.555126+0000","last_active":"2023-10-30T14:26:39.902346+0000","last_peered":"2023-10-30T14:26:39.902346+0000","last_clean":"2023-10-30T14:26:39.902346+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T14:26:39.902346+0000","last_undegraded":"2023-10-30T14:26:39.902346+0000","last_fullsized":"2023-10-30T14:26:39.902346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2145,"log_dups_size":0,"ondisk_log_size":2145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.71511574600000005,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":146,"num_read_kb":25110,"num_write":1316,"num_write_kb":20043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"109'2109","reported_seq":4922,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893200+0000","last_change":"2023-10-30T14:26:36.369916+0000","last_active":"2023-10-30T14:26:39.893200+0000","last_peered":"2023-10-30T14:26:39.893200+0000","last_clean":"2023-10-30T14:26:39.893200+0000","last_became_active":"2023-10-30T13:24:00.989938+0000","last_became_peered":"2023-10-30T13:24:00.989938+0000","last_unstale":"2023-10-30T14:26:39.893200+0000","last_undegraded":"2023-10-30T14:26:39.893200+0000","last_fullsized":"2023-10-30T14:26:39.893200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2109,"log_dups_size":0,"ondisk_log_size":2109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.90238370000000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":124,"num_read_kb":25628,"num_write":1232,"num_write_kb":20540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"109'2137","reported_seq":4970,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.907230+0000","last_change":"2023-10-30T14:26:34.654560+0000","last_active":"2023-10-30T14:26:39.907230+0000","last_peered":"2023-10-30T14:26:39.907230+0000","last_clean":"2023-10-30T14:26:39.907230+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T14:26:39.907230+0000","last_undegraded":"2023-10-30T14:26:39.907230+0000","last_fullsized":"2023-10-30T14:26:39.907230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81280227199999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":24548,"num_write":1310,"num_write_kb":19821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"109'2073","reported_seq":4937,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.907791+0000","last_change":"2023-10-30T14:26:33.465676+0000","last_active":"2023-10-30T14:26:39.907791+0000","last_peered":"2023-10-30T14:26:39.907791+0000","last_clean":"2023-10-30T14:26:39.907791+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T14:26:39.907791+0000","last_undegraded":"2023-10-30T14:26:39.907791+0000","last_fullsized":"2023-10-30T14:26:39.907791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.53668512800000001,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":148,"num_read_kb":64678,"num_write":1273,"num_write_kb":32495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"109'2008","reported_seq":4884,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.906222+0000","last_change":"2023-10-30T14:26:33.399684+0000","last_active":"2023-10-30T14:26:39.906222+0000","last_peered":"2023-10-30T14:26:39.906222+0000","last_clean":"2023-10-30T14:26:39.906222+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T14:26:39.906222+0000","last_undegraded":"2023-10-30T14:26:39.906222+0000","last_fullsized":"2023-10-30T14:26:39.906222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2008,"log_dups_size":0,"ondisk_log_size":2008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.471346551,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":139,"num_read_kb":25550,"num_write":1254,"num_write_kb":16915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"110'2118","reported_seq":4973,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:58.807440+0000","last_change":"2023-10-30T14:26:58.807556+0000","last_active":"2023-10-30T14:26:58.807440+0000","last_peered":"2023-10-30T14:26:58.807440+0000","last_clean":"2023-10-30T14:26:58.807440+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T14:26:58.807440+0000","last_undegraded":"2023-10-30T14:26:58.807440+0000","last_fullsized":"2023-10-30T14:26:58.807440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2118","last_scrub_stamp":"2023-10-30T14:26:58.807287+0000","last_deep_scrub":"110'2118","last_deep_scrub_stamp":"2023-10-30T14:26:58.807287+0000","last_clean_scrub_stamp":"2023-10-30T14:26:58.807287+0000","objects_scrubbed":35,"log_size":2118,"log_dups_size":0,"ondisk_log_size":2118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:18.713125+0000","scrub_duration":0.105301005,"objects_trimmed":0,"snaptrim_duration":0.54183594099999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":142,"num_read_kb":52725,"num_write":1240,"num_write_kb":42463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"109'2223","reported_seq":5033,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893525+0000","last_change":"2023-10-30T14:26:36.506702+0000","last_active":"2023-10-30T14:26:39.893525+0000","last_peered":"2023-10-30T14:26:39.893525+0000","last_clean":"2023-10-30T14:26:39.893525+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T14:26:39.893525+0000","last_undegraded":"2023-10-30T14:26:39.893525+0000","last_fullsized":"2023-10-30T14:26:39.893525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.038919041,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":46739,"num_write":1326,"num_write_kb":33562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"109'2151","reported_seq":5003,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.755838+0000","last_change":"2023-10-30T14:26:34.440977+0000","last_active":"2023-10-30T14:26:40.755838+0000","last_peered":"2023-10-30T14:26:40.755838+0000","last_clean":"2023-10-30T14:26:40.755838+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T14:26:40.755838+0000","last_undegraded":"2023-10-30T14:26:40.755838+0000","last_fullsized":"2023-10-30T14:26:40.755838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2151,"log_dups_size":0,"ondisk_log_size":2151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.97549857799999995,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":149,"num_read_kb":13007,"num_write":1312,"num_write_kb":16233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"109'2267","reported_seq":5093,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.182798+0000","last_change":"2023-10-30T14:26:33.691289+0000","last_active":"2023-10-30T14:26:40.182798+0000","last_peered":"2023-10-30T14:26:40.182798+0000","last_clean":"2023-10-30T14:26:40.182798+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T14:26:40.182798+0000","last_undegraded":"2023-10-30T14:26:40.182798+0000","last_fullsized":"2023-10-30T14:26:40.182798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2267,"log_dups_size":0,"ondisk_log_size":2267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.432871758,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":161,"num_read_kb":14756,"num_write":1377,"num_write_kb":22909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"109'2090","reported_seq":4951,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754926+0000","last_change":"2023-10-30T14:26:33.787493+0000","last_active":"2023-10-30T14:26:40.754926+0000","last_peered":"2023-10-30T14:26:40.754926+0000","last_clean":"2023-10-30T14:26:40.754926+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T14:26:40.754926+0000","last_undegraded":"2023-10-30T14:26:40.754926+0000","last_fullsized":"2023-10-30T14:26:40.754926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.83845289999999995,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":149,"num_read_kb":40211,"num_write":1268,"num_write_kb":23485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"109'2348","reported_seq":5164,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.753436+0000","last_change":"2023-10-30T14:26:34.060688+0000","last_active":"2023-10-30T14:26:40.753436+0000","last_peered":"2023-10-30T14:26:40.753436+0000","last_clean":"2023-10-30T14:26:40.753436+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T14:26:40.753436+0000","last_undegraded":"2023-10-30T14:26:40.753436+0000","last_fullsized":"2023-10-30T14:26:40.753436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2348,"log_dups_size":0,"ondisk_log_size":2348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.109948846,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":170,"num_read_kb":39932,"num_write":1446,"num_write_kb":30153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"110'2027","reported_seq":4894,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:59.058458+0000","last_change":"2023-10-30T14:26:59.058509+0000","last_active":"2023-10-30T14:26:59.058458+0000","last_peered":"2023-10-30T14:26:59.058458+0000","last_clean":"2023-10-30T14:26:59.058458+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T14:26:59.058458+0000","last_undegraded":"2023-10-30T14:26:59.058458+0000","last_fullsized":"2023-10-30T14:26:59.058458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2027","last_scrub_stamp":"2023-10-30T14:26:59.058377+0000","last_deep_scrub":"110'2027","last_deep_scrub_stamp":"2023-10-30T14:26:59.058377+0000","last_clean_scrub_stamp":"2023-10-30T14:26:59.058377+0000","objects_scrubbed":31,"log_size":2027,"log_dups_size":0,"ondisk_log_size":2027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:24:17.547389+0000","scrub_duration":0.077206206999999999,"objects_trimmed":0,"snaptrim_duration":0.34246037800000001,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":137,"num_read_kb":13706,"num_write":1196,"num_write_kb":12790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"109'2095","reported_seq":4930,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.905220+0000","last_change":"2023-10-30T14:26:34.604839+0000","last_active":"2023-10-30T14:26:39.905220+0000","last_peered":"2023-10-30T14:26:39.905220+0000","last_clean":"2023-10-30T14:26:39.905220+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T14:26:39.905220+0000","last_undegraded":"2023-10-30T14:26:39.905220+0000","last_fullsized":"2023-10-30T14:26:39.905220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2095,"log_dups_size":0,"ondisk_log_size":2095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.76369308499999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":125,"num_read_kb":30178,"num_write":1282,"num_write_kb":24413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"110'2253","reported_seq":5094,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.434812+0000","last_change":"2023-10-30T14:26:57.434908+0000","last_active":"2023-10-30T14:26:57.434812+0000","last_peered":"2023-10-30T14:26:57.434812+0000","last_clean":"2023-10-30T14:26:57.434812+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T14:26:57.434812+0000","last_undegraded":"2023-10-30T14:26:57.434812+0000","last_fullsized":"2023-10-30T14:26:57.434812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2253","last_scrub_stamp":"2023-10-30T14:26:57.434694+0000","last_deep_scrub":"110'2253","last_deep_scrub_stamp":"2023-10-30T14:26:57.434694+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.434694+0000","objects_scrubbed":38,"log_size":2253,"log_dups_size":0,"ondisk_log_size":2253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:26:38.794241+0000","scrub_duration":0.122965276,"objects_trimmed":0,"snaptrim_duration":0.46668427699999998,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":148,"num_read_kb":30111,"num_write":1332,"num_write_kb":26597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"109'2222","reported_seq":5013,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.756248+0000","last_change":"2023-10-30T14:26:34.474353+0000","last_active":"2023-10-30T14:26:40.756248+0000","last_peered":"2023-10-30T14:26:40.756248+0000","last_clean":"2023-10-30T14:26:40.756248+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T14:26:40.756248+0000","last_undegraded":"2023-10-30T14:26:40.756248+0000","last_fullsized":"2023-10-30T14:26:40.756248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2222,"log_dups_size":0,"ondisk_log_size":2222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.008362585,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":127,"num_read_kb":10449,"num_write":1298,"num_write_kb":11653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"109'2282","reported_seq":5085,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754280+0000","last_change":"2023-10-30T14:26:33.893164+0000","last_active":"2023-10-30T14:26:40.754280+0000","last_peered":"2023-10-30T14:26:40.754280+0000","last_clean":"2023-10-30T14:26:40.754280+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T14:26:40.754280+0000","last_undegraded":"2023-10-30T14:26:40.754280+0000","last_fullsized":"2023-10-30T14:26:40.754280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2282,"log_dups_size":0,"ondisk_log_size":2282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.94381626399999996,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":144,"num_read_kb":30506,"num_write":1376,"num_write_kb":26822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"109'2117","reported_seq":5082,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893234+0000","last_change":"2023-10-30T14:26:36.452306+0000","last_active":"2023-10-30T14:26:39.893234+0000","last_peered":"2023-10-30T14:26:39.893234+0000","last_clean":"2023-10-30T14:26:39.893234+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T14:26:39.893234+0000","last_undegraded":"2023-10-30T14:26:39.893234+0000","last_fullsized":"2023-10-30T14:26:39.893234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2117,"log_dups_size":0,"ondisk_log_size":2117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98440143999999996,"stat_sum":{"num_bytes":6,"num_objects":38,"num_object_clones":2,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":257,"num_read_kb":42146,"num_write":1297,"num_write_kb":25266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"109'2178","reported_seq":5042,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.755000+0000","last_change":"2023-10-30T14:26:33.846175+0000","last_active":"2023-10-30T14:26:40.755000+0000","last_peered":"2023-10-30T14:26:40.755000+0000","last_clean":"2023-10-30T14:26:40.755000+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T14:26:40.755000+0000","last_undegraded":"2023-10-30T14:26:40.755000+0000","last_fullsized":"2023-10-30T14:26:40.755000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2178,"log_dups_size":0,"ondisk_log_size":2178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89723890399999995,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":164,"num_read_kb":14461,"num_write":1342,"num_write_kb":12663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"109'2057","reported_seq":4899,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893814+0000","last_change":"2023-10-30T14:26:36.518474+0000","last_active":"2023-10-30T14:26:39.893814+0000","last_peered":"2023-10-30T14:26:39.893814+0000","last_clean":"2023-10-30T14:26:39.893814+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T14:26:39.893814+0000","last_undegraded":"2023-10-30T14:26:39.893814+0000","last_fullsized":"2023-10-30T14:26:39.893814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2057,"log_dups_size":0,"ondisk_log_size":2057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0495642949999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":133,"num_read_kb":25337,"num_write":1220,"num_write_kb":15637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"109'2141","reported_seq":5020,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.908255+0000","last_change":"2023-10-30T14:26:33.433647+0000","last_active":"2023-10-30T14:26:39.908255+0000","last_peered":"2023-10-30T14:26:39.908255+0000","last_clean":"2023-10-30T14:26:39.908255+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T14:26:39.908255+0000","last_undegraded":"2023-10-30T14:26:39.908255+0000","last_fullsized":"2023-10-30T14:26:39.908255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2141,"log_dups_size":0,"ondisk_log_size":2141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.504458776,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":160,"num_read_kb":14034,"num_write":1340,"num_write_kb":12359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"109'2089","reported_seq":4940,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.183417+0000","last_change":"2023-10-30T14:26:33.601252+0000","last_active":"2023-10-30T14:26:40.183417+0000","last_peered":"2023-10-30T14:26:40.183417+0000","last_clean":"2023-10-30T14:26:40.183417+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T14:26:40.183417+0000","last_undegraded":"2023-10-30T14:26:40.183417+0000","last_fullsized":"2023-10-30T14:26:40.183417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2089,"log_dups_size":0,"ondisk_log_size":2089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.34262146900000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":128,"num_read_kb":22518,"num_write":1277,"num_write_kb":32107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"110'2068","reported_seq":4908,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.691221+0000","last_change":"2023-10-30T14:26:52.691273+0000","last_active":"2023-10-30T14:26:52.691221+0000","last_peered":"2023-10-30T14:26:52.691221+0000","last_clean":"2023-10-30T14:26:52.691221+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T14:26:52.691221+0000","last_undegraded":"2023-10-30T14:26:52.691221+0000","last_fullsized":"2023-10-30T14:26:52.691221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2068","last_scrub_stamp":"2023-10-30T14:26:52.691147+0000","last_deep_scrub":"110'2068","last_deep_scrub_stamp":"2023-10-30T14:26:52.691147+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.691147+0000","objects_scrubbed":38,"log_size":2068,"log_dups_size":0,"ondisk_log_size":2068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:08.320506+0000","scrub_duration":0.079048169000000001,"objects_trimmed":0,"snaptrim_duration":1.068284585,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":129,"num_read_kb":17081,"num_write":1202,"num_write_kb":15468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"109'2140","reported_seq":4964,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.756136+0000","last_change":"2023-10-30T14:26:34.496718+0000","last_active":"2023-10-30T14:26:40.756136+0000","last_peered":"2023-10-30T14:26:40.756136+0000","last_clean":"2023-10-30T14:26:40.756136+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T14:26:40.756136+0000","last_undegraded":"2023-10-30T14:26:40.756136+0000","last_fullsized":"2023-10-30T14:26:40.756136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2140,"log_dups_size":0,"ondisk_log_size":2140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.030536272,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":125,"num_read_kb":13655,"num_write":1323,"num_write_kb":17357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"110'2124","reported_seq":5004,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.119325+0000","last_change":"2023-10-30T14:26:57.119416+0000","last_active":"2023-10-30T14:26:57.119325+0000","last_peered":"2023-10-30T14:26:57.119325+0000","last_clean":"2023-10-30T14:26:57.119325+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T14:26:57.119325+0000","last_undegraded":"2023-10-30T14:26:57.119325+0000","last_fullsized":"2023-10-30T14:26:57.119325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2124","last_scrub_stamp":"2023-10-30T14:26:57.119213+0000","last_deep_scrub":"110'2124","last_deep_scrub_stamp":"2023-10-30T14:26:57.119213+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.119213+0000","objects_scrubbed":38,"log_size":2124,"log_dups_size":0,"ondisk_log_size":2124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:10.120593+0000","scrub_duration":0.076576980000000003,"objects_trimmed":0,"snaptrim_duration":0.98488935700000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":140,"num_read_kb":17880,"num_write":1311,"num_write_kb":13902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"110'2418","reported_seq":5244,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.085217+0000","last_change":"2023-10-30T14:27:00.085296+0000","last_active":"2023-10-30T14:27:00.085217+0000","last_peered":"2023-10-30T14:27:00.085217+0000","last_clean":"2023-10-30T14:27:00.085217+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T14:27:00.085217+0000","last_undegraded":"2023-10-30T14:27:00.085217+0000","last_fullsized":"2023-10-30T14:27:00.085217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2418","last_scrub_stamp":"2023-10-30T14:27:00.085138+0000","last_deep_scrub":"110'2418","last_deep_scrub_stamp":"2023-10-30T14:27:00.085138+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.085138+0000","objects_scrubbed":41,"log_size":2418,"log_dups_size":0,"ondisk_log_size":2418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:00.491199+0000","scrub_duration":0.10095144,"objects_trimmed":0,"snaptrim_duration":0.30670995400000001,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":165,"num_read_kb":40193,"num_write":1443,"num_write_kb":35290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"109'2303","reported_seq":5126,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.904270+0000","last_change":"2023-10-30T14:26:33.357836+0000","last_active":"2023-10-30T14:26:39.904270+0000","last_peered":"2023-10-30T14:26:39.904270+0000","last_clean":"2023-10-30T14:26:39.904270+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T14:26:39.904270+0000","last_undegraded":"2023-10-30T14:26:39.904270+0000","last_fullsized":"2023-10-30T14:26:39.904270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2303,"log_dups_size":0,"ondisk_log_size":2303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42993166100000002,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":168,"num_read_kb":24860,"num_write":1383,"num_write_kb":20782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"110'2147","reported_seq":5000,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:50.868519+0000","last_change":"2023-10-30T14:26:50.868736+0000","last_active":"2023-10-30T14:26:50.868519+0000","last_peered":"2023-10-30T14:26:50.868519+0000","last_clean":"2023-10-30T14:26:50.868519+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T14:26:50.868519+0000","last_undegraded":"2023-10-30T14:26:50.868519+0000","last_fullsized":"2023-10-30T14:26:50.868519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2147","last_scrub_stamp":"2023-10-30T14:26:50.868292+0000","last_deep_scrub":"110'2147","last_deep_scrub_stamp":"2023-10-30T14:26:50.868292+0000","last_clean_scrub_stamp":"2023-10-30T14:26:50.868292+0000","objects_scrubbed":34,"log_size":2147,"log_dups_size":0,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:21.454895+0000","scrub_duration":0.095674789999999996,"objects_trimmed":0,"snaptrim_duration":0.59246732999999996,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":146,"num_read_kb":30802,"num_write":1268,"num_write_kb":21419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"110'1078","reported_seq":1396,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.791196+0000","last_change":"2023-10-30T14:26:42.791248+0000","last_active":"2023-10-30T14:26:42.791196+0000","last_peered":"2023-10-30T14:26:42.791196+0000","last_clean":"2023-10-30T14:26:42.791196+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T14:26:42.791196+0000","last_undegraded":"2023-10-30T14:26:42.791196+0000","last_fullsized":"2023-10-30T14:26:42.791196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1078","last_scrub_stamp":"2023-10-30T14:26:42.791118+0000","last_deep_scrub":"110'1078","last_deep_scrub_stamp":"2023-10-30T14:26:42.791118+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.791118+0000","objects_scrubbed":50,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:33.250671+0000","scrub_duration":0.285611593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497506,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":222,"num_read_kb":501605,"num_write":1094,"num_write_kb":95785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":358881,"num_omap_keys":701,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"109'2155","reported_seq":4963,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.755848+0000","last_change":"2023-10-30T14:26:34.609800+0000","last_active":"2023-10-30T14:26:40.755848+0000","last_peered":"2023-10-30T14:26:40.755848+0000","last_clean":"2023-10-30T14:26:40.755848+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T14:26:40.755848+0000","last_undegraded":"2023-10-30T14:26:40.755848+0000","last_fullsized":"2023-10-30T14:26:40.755848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2155,"log_dups_size":0,"ondisk_log_size":2155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.1434485830000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":17524,"num_write":1294,"num_write_kb":15816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"110'1745","reported_seq":2190,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.486755+0000","last_change":"2023-10-30T14:26:54.486806+0000","last_active":"2023-10-30T14:26:54.486755+0000","last_peered":"2023-10-30T14:26:54.486755+0000","last_clean":"2023-10-30T14:26:54.486755+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T14:26:54.486755+0000","last_undegraded":"2023-10-30T14:26:54.486755+0000","last_fullsized":"2023-10-30T14:26:54.486755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1745","last_scrub_stamp":"2023-10-30T14:26:54.486663+0000","last_deep_scrub":"110'1745","last_deep_scrub_stamp":"2023-10-30T14:26:54.486663+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.486663+0000","objects_scrubbed":52,"log_size":1745,"log_dups_size":0,"ondisk_log_size":1745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:54.960607+0000","scrub_duration":0.19504682200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29694842,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":392,"num_read_kb":410068,"num_write":1792,"num_write_kb":89526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":545048,"num_omap_keys":1059,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"110'1609","reported_seq":2975,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.816492+0000","last_change":"2023-10-30T14:26:46.816578+0000","last_active":"2023-10-30T14:26:46.816492+0000","last_peered":"2023-10-30T14:26:46.816492+0000","last_clean":"2023-10-30T14:26:46.816492+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T14:26:46.816492+0000","last_undegraded":"2023-10-30T14:26:46.816492+0000","last_fullsized":"2023-10-30T14:26:46.816492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1609","last_scrub_stamp":"2023-10-30T14:26:46.816372+0000","last_deep_scrub":"110'1609","last_deep_scrub_stamp":"2023-10-30T14:26:46.816372+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.816372+0000","objects_scrubbed":52,"log_size":1609,"log_dups_size":0,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:04:35.564593+0000","scrub_duration":0.28844654400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60855928,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":1296,"num_read_kb":556671,"num_write":1596,"num_write_kb":94675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":405715,"num_omap_keys":625,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"110'2379","reported_seq":5227,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.714048+0000","last_change":"2023-10-30T14:26:53.714120+0000","last_active":"2023-10-30T14:26:53.714048+0000","last_peered":"2023-10-30T14:26:53.714048+0000","last_clean":"2023-10-30T14:26:53.714048+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T14:26:53.714048+0000","last_undegraded":"2023-10-30T14:26:53.714048+0000","last_fullsized":"2023-10-30T14:26:53.714048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2379","last_scrub_stamp":"2023-10-30T14:26:53.713939+0000","last_deep_scrub":"110'2379","last_deep_scrub_stamp":"2023-10-30T14:26:53.713939+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.713939+0000","objects_scrubbed":52,"log_size":2379,"log_dups_size":0,"ondisk_log_size":2379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:15.405210+0000","scrub_duration":0.058831420000000002,"objects_trimmed":0,"snaptrim_duration":0.89032229399999996,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":161,"num_read_kb":17698,"num_write":1449,"num_write_kb":26122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"110'1032","reported_seq":1390,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:55.223361+0000","last_change":"2023-10-30T14:26:55.223448+0000","last_active":"2023-10-30T14:26:55.223361+0000","last_peered":"2023-10-30T14:26:55.223361+0000","last_clean":"2023-10-30T14:26:55.223361+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T14:26:55.223361+0000","last_undegraded":"2023-10-30T14:26:55.223361+0000","last_fullsized":"2023-10-30T14:26:55.223361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1032","last_scrub_stamp":"2023-10-30T14:26:55.223231+0000","last_deep_scrub":"110'1032","last_deep_scrub_stamp":"2023-10-30T14:26:55.223231+0000","last_clean_scrub_stamp":"2023-10-30T14:26:55.223231+0000","objects_scrubbed":57,"log_size":1032,"log_dups_size":0,"ondisk_log_size":1032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:18:36.091837+0000","scrub_duration":0.26442538300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54010089,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":312,"num_read_kb":563254,"num_write":1082,"num_write_kb":96418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":706737,"num_omap_keys":1228,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"110'2179","reported_seq":5047,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.255870+0000","last_change":"2023-10-30T14:27:00.255921+0000","last_active":"2023-10-30T14:27:00.255870+0000","last_peered":"2023-10-30T14:27:00.255870+0000","last_clean":"2023-10-30T14:27:00.255870+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T14:27:00.255870+0000","last_undegraded":"2023-10-30T14:27:00.255870+0000","last_fullsized":"2023-10-30T14:27:00.255870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2179","last_scrub_stamp":"2023-10-30T14:27:00.255786+0000","last_deep_scrub":"110'2179","last_deep_scrub_stamp":"2023-10-30T14:27:00.255786+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.255786+0000","objects_scrubbed":30,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:01.714268+0000","scrub_duration":0.037354261999999999,"objects_trimmed":0,"snaptrim_duration":0.36920391600000002,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":152,"num_read_kb":51949,"num_write":1321,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"110'865","reported_seq":1210,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:56.576908+0000","last_change":"2023-10-30T14:26:56.577007+0000","last_active":"2023-10-30T14:26:56.576908+0000","last_peered":"2023-10-30T14:26:56.576908+0000","last_clean":"2023-10-30T14:26:56.576908+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T14:26:56.576908+0000","last_undegraded":"2023-10-30T14:26:56.576908+0000","last_fullsized":"2023-10-30T14:26:56.576908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'865","last_scrub_stamp":"2023-10-30T14:26:56.576765+0000","last_deep_scrub":"110'865","last_deep_scrub_stamp":"2023-10-30T14:26:56.576765+0000","last_clean_scrub_stamp":"2023-10-30T14:26:56.576765+0000","objects_scrubbed":64,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:40.939877+0000","scrub_duration":0.28082476899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943130,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":319,"num_read_kb":417394,"num_write":917,"num_write_kb":71614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":395317,"num_omap_keys":751,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"110'2477","reported_seq":5216,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:58.107565+0000","last_change":"2023-10-30T14:26:58.107765+0000","last_active":"2023-10-30T14:26:58.107565+0000","last_peered":"2023-10-30T14:26:58.107565+0000","last_clean":"2023-10-30T14:26:58.107565+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T14:26:58.107565+0000","last_undegraded":"2023-10-30T14:26:58.107565+0000","last_fullsized":"2023-10-30T14:26:58.107565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2477","last_scrub_stamp":"2023-10-30T14:26:58.107390+0000","last_deep_scrub":"110'2477","last_deep_scrub_stamp":"2023-10-30T14:26:58.107390+0000","last_clean_scrub_stamp":"2023-10-30T14:26:58.107390+0000","objects_scrubbed":23,"log_size":2477,"log_dups_size":0,"ondisk_log_size":2477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:29.902228+0000","scrub_duration":0.078694307000000005,"objects_trimmed":0,"snaptrim_duration":1.0302270280000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":152,"num_read_kb":10055,"num_write":1351,"num_write_kb":13264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"110'901","reported_seq":1217,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:51.209972+0000","last_change":"2023-10-30T14:26:51.210091+0000","last_active":"2023-10-30T14:26:51.209972+0000","last_peered":"2023-10-30T14:26:51.209972+0000","last_clean":"2023-10-30T14:26:51.209972+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T14:26:51.209972+0000","last_undegraded":"2023-10-30T14:26:51.209972+0000","last_fullsized":"2023-10-30T14:26:51.209972+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'901","last_scrub_stamp":"2023-10-30T14:26:51.209745+0000","last_deep_scrub":"110'901","last_deep_scrub_stamp":"2023-10-30T14:26:51.209745+0000","last_clean_scrub_stamp":"2023-10-30T14:26:51.209745+0000","objects_scrubbed":52,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:29:33.421380+0000","scrub_duration":0.228163176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":291,"num_read_kb":402257,"num_write":947,"num_write_kb":71494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":417597,"num_omap_keys":787,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"109'2085","reported_seq":4930,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.756546+0000","last_change":"2023-10-30T14:26:34.598085+0000","last_active":"2023-10-30T14:26:40.756546+0000","last_peered":"2023-10-30T14:26:40.756546+0000","last_clean":"2023-10-30T14:26:40.756546+0000","last_became_active":"2023-10-30T13:24:00.990019+0000","last_became_peered":"2023-10-30T13:24:00.990019+0000","last_unstale":"2023-10-30T14:26:40.756546+0000","last_undegraded":"2023-10-30T14:26:40.756546+0000","last_fullsized":"2023-10-30T14:26:40.756546+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2085,"log_dups_size":0,"ondisk_log_size":2085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.1322467139999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":132,"num_read_kb":30385,"num_write":1263,"num_write_kb":18224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"110'1029","reported_seq":1299,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:55.421984+0000","last_change":"2023-10-30T14:26:55.422041+0000","last_active":"2023-10-30T14:26:55.421984+0000","last_peered":"2023-10-30T14:26:55.421984+0000","last_clean":"2023-10-30T14:26:55.421984+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T14:26:55.421984+0000","last_undegraded":"2023-10-30T14:26:55.421984+0000","last_fullsized":"2023-10-30T14:26:55.421984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1029","last_scrub_stamp":"2023-10-30T14:26:55.421890+0000","last_deep_scrub":"110'1029","last_deep_scrub_stamp":"2023-10-30T14:26:55.421890+0000","last_clean_scrub_stamp":"2023-10-30T14:26:55.421890+0000","objects_scrubbed":35,"log_size":1029,"log_dups_size":0,"ondisk_log_size":1029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:34:36.976816+0000","scrub_duration":0.131590227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":208,"num_read_kb":224450,"num_write":1057,"num_write_kb":46972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":295749,"num_omap_keys":457,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"110'2177","reported_seq":5017,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:48.799169+0000","last_change":"2023-10-30T14:26:48.799265+0000","last_active":"2023-10-30T14:26:48.799169+0000","last_peered":"2023-10-30T14:26:48.799169+0000","last_clean":"2023-10-30T14:26:48.799169+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T14:26:48.799169+0000","last_undegraded":"2023-10-30T14:26:48.799169+0000","last_fullsized":"2023-10-30T14:26:48.799169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2177","last_scrub_stamp":"2023-10-30T14:26:48.799002+0000","last_deep_scrub":"110'2177","last_deep_scrub_stamp":"2023-10-30T14:26:48.799002+0000","last_clean_scrub_stamp":"2023-10-30T14:26:48.799002+0000","objects_scrubbed":28,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:56:02.403730+0000","scrub_duration":0.092776059999999994,"objects_trimmed":0,"snaptrim_duration":0.58089010799999996,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":141,"num_read_kb":34327,"num_write":1294,"num_write_kb":27493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"110'1040","reported_seq":1364,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:47.590330+0000","last_change":"2023-10-30T14:26:47.590330+0000","last_active":"2023-10-30T14:26:47.590330+0000","last_peered":"2023-10-30T14:26:47.590330+0000","last_clean":"2023-10-30T14:26:47.590330+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T14:26:47.590330+0000","last_undegraded":"2023-10-30T14:26:47.590330+0000","last_fullsized":"2023-10-30T14:26:47.590330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1039","last_scrub_stamp":"2023-10-30T14:26:47.589869+0000","last_deep_scrub":"110'1039","last_deep_scrub_stamp":"2023-10-30T14:26:47.589869+0000","last_clean_scrub_stamp":"2023-10-30T14:26:47.589869+0000","objects_scrubbed":46,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:04:37.337840+0000","scrub_duration":0.243642899,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":290,"num_read_kb":355466,"num_write":1084,"num_write_kb":67491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":426651,"num_omap_keys":733,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"109'2165","reported_seq":4985,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.904020+0000","last_change":"2023-10-30T14:26:34.585946+0000","last_active":"2023-10-30T14:26:39.904020+0000","last_peered":"2023-10-30T14:26:39.904020+0000","last_clean":"2023-10-30T14:26:39.904020+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T14:26:39.904020+0000","last_undegraded":"2023-10-30T14:26:39.904020+0000","last_fullsized":"2023-10-30T14:26:39.904020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2165,"log_dups_size":0,"ondisk_log_size":2165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.74553484199999998,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":143,"num_read_kb":29369,"num_write":1268,"num_write_kb":21738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"110'758","reported_seq":1138,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.031518+0000","last_change":"2023-10-30T14:26:42.031593+0000","last_active":"2023-10-30T14:26:42.031518+0000","last_peered":"2023-10-30T14:26:42.031518+0000","last_clean":"2023-10-30T14:26:42.031518+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T14:26:42.031518+0000","last_undegraded":"2023-10-30T14:26:42.031518+0000","last_fullsized":"2023-10-30T14:26:42.031518+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'758","last_scrub_stamp":"2023-10-30T14:26:42.031416+0000","last_deep_scrub":"110'758","last_deep_scrub_stamp":"2023-10-30T14:26:42.031416+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.031416+0000","objects_scrubbed":57,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:32.795744+0000","scrub_duration":0.243287324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331682,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":392,"num_read_kb":480410,"num_write":807,"num_write_kb":87841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":384994,"num_omap_keys":754,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"110'2184","reported_seq":5052,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.780582+0000","last_change":"2023-10-30T14:26:52.780632+0000","last_active":"2023-10-30T14:26:52.780582+0000","last_peered":"2023-10-30T14:26:52.780582+0000","last_clean":"2023-10-30T14:26:52.780582+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T14:26:52.780582+0000","last_undegraded":"2023-10-30T14:26:52.780582+0000","last_fullsized":"2023-10-30T14:26:52.780582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2184","last_scrub_stamp":"2023-10-30T14:26:52.780505+0000","last_deep_scrub":"110'2184","last_deep_scrub_stamp":"2023-10-30T14:26:52.780505+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.780505+0000","objects_scrubbed":34,"log_size":2184,"log_dups_size":0,"ondisk_log_size":2184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:22.325362+0000","scrub_duration":0.086211091000000004,"objects_trimmed":0,"snaptrim_duration":0.95729191499999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":151,"num_read_kb":31378,"num_write":1306,"num_write_kb":33173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"110'899","reported_seq":1169,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.271896+0000","last_change":"2023-10-30T14:26:54.272052+0000","last_active":"2023-10-30T14:26:54.271896+0000","last_peered":"2023-10-30T14:26:54.271896+0000","last_clean":"2023-10-30T14:26:54.271896+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T14:26:54.271896+0000","last_undegraded":"2023-10-30T14:26:54.271896+0000","last_fullsized":"2023-10-30T14:26:54.271896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'899","last_scrub_stamp":"2023-10-30T14:26:54.271663+0000","last_deep_scrub":"110'899","last_deep_scrub_stamp":"2023-10-30T14:26:54.271663+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.271663+0000","objects_scrubbed":48,"log_size":899,"log_dups_size":0,"ondisk_log_size":899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:16.820077+0000","scrub_duration":0.294758195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53927368,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":208,"num_read_kb":306859,"num_write":914,"num_write_kb":62457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":467666,"num_omap_keys":812,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"109'2293","reported_seq":5085,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:41.616336+0000","last_change":"2023-10-30T14:26:33.299265+0000","last_active":"2023-10-30T14:26:41.616336+0000","last_peered":"2023-10-30T14:26:41.616336+0000","last_clean":"2023-10-30T14:26:41.616336+0000","last_became_active":"2023-10-30T13:24:00.706894+0000","last_became_peered":"2023-10-30T13:24:00.706894+0000","last_unstale":"2023-10-30T14:26:41.616336+0000","last_undegraded":"2023-10-30T14:26:41.616336+0000","last_fullsized":"2023-10-30T14:26:41.616336+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2293,"log_dups_size":0,"ondisk_log_size":2293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.3738494,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":149,"num_read_kb":1539,"num_write":1375,"num_write_kb":12931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"110'780","reported_seq":1067,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.584097+0000","last_change":"2023-10-30T14:26:46.584221+0000","last_active":"2023-10-30T14:26:46.584097+0000","last_peered":"2023-10-30T14:26:46.584097+0000","last_clean":"2023-10-30T14:26:46.584097+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T14:26:46.584097+0000","last_undegraded":"2023-10-30T14:26:46.584097+0000","last_fullsized":"2023-10-30T14:26:46.584097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'780","last_scrub_stamp":"2023-10-30T14:26:46.583879+0000","last_deep_scrub":"110'780","last_deep_scrub_stamp":"2023-10-30T14:26:46.583879+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.583879+0000","objects_scrubbed":53,"log_size":780,"log_dups_size":0,"ondisk_log_size":780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:52.998170+0000","scrub_duration":0.20502225800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":234,"num_read_kb":298146,"num_write":829,"num_write_kb":54985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":494954,"num_omap_keys":946,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"110'2149","reported_seq":5030,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.527465+0000","last_change":"2023-10-30T14:27:00.527596+0000","last_active":"2023-10-30T14:27:00.527465+0000","last_peered":"2023-10-30T14:27:00.527465+0000","last_clean":"2023-10-30T14:27:00.527465+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T14:27:00.527465+0000","last_undegraded":"2023-10-30T14:27:00.527465+0000","last_fullsized":"2023-10-30T14:27:00.527465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2149","last_scrub_stamp":"2023-10-30T14:27:00.527256+0000","last_deep_scrub":"110'2149","last_deep_scrub_stamp":"2023-10-30T14:27:00.527256+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.527256+0000","objects_scrubbed":53,"log_size":2149,"log_dups_size":0,"ondisk_log_size":2149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:55:50.223566+0000","scrub_duration":0.14114542399999999,"objects_trimmed":0,"snaptrim_duration":0.50412874500000004,"stat_sum":{"num_bytes":0,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":148,"num_read_kb":43506,"num_write":1375,"num_write_kb":26417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"110'876","reported_seq":1232,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.183186+0000","last_change":"2023-10-30T14:26:52.183292+0000","last_active":"2023-10-30T14:26:52.183186+0000","last_peered":"2023-10-30T14:26:52.183186+0000","last_clean":"2023-10-30T14:26:52.183186+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T14:26:52.183186+0000","last_undegraded":"2023-10-30T14:26:52.183186+0000","last_fullsized":"2023-10-30T14:26:52.183186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'876","last_scrub_stamp":"2023-10-30T14:26:52.183105+0000","last_deep_scrub":"110'876","last_deep_scrub_stamp":"2023-10-30T14:26:52.183105+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.183105+0000","objects_scrubbed":51,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:18:36.571108+0000","scrub_duration":0.119037509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":481,"num_read_kb":229594,"num_write":1003,"num_write_kb":42855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":532426,"num_omap_keys":823,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"110'2380","reported_seq":5218,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.002361+0000","last_change":"2023-10-30T14:26:46.002570+0000","last_active":"2023-10-30T14:26:46.002361+0000","last_peered":"2023-10-30T14:26:46.002361+0000","last_clean":"2023-10-30T14:26:46.002361+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T14:26:46.002361+0000","last_undegraded":"2023-10-30T14:26:46.002361+0000","last_fullsized":"2023-10-30T14:26:46.002361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2380","last_scrub_stamp":"2023-10-30T14:26:46.002134+0000","last_deep_scrub":"110'2380","last_deep_scrub_stamp":"2023-10-30T14:26:46.002134+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.002134+0000","objects_scrubbed":46,"log_size":2380,"log_dups_size":0,"ondisk_log_size":2380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:38.920284+0000","scrub_duration":0.14576676,"objects_trimmed":0,"snaptrim_duration":0.70955538100000004,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":166,"num_read_kb":25446,"num_write":1448,"num_write_kb":32435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"110'1046","reported_seq":1341,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.977448+0000","last_change":"2023-10-30T14:26:42.977518+0000","last_active":"2023-10-30T14:26:42.977448+0000","last_peered":"2023-10-30T14:26:42.977448+0000","last_clean":"2023-10-30T14:26:42.977448+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T14:26:42.977448+0000","last_undegraded":"2023-10-30T14:26:42.977448+0000","last_fullsized":"2023-10-30T14:26:42.977448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1046","last_scrub_stamp":"2023-10-30T14:26:42.977329+0000","last_deep_scrub":"110'1046","last_deep_scrub_stamp":"2023-10-30T14:26:42.977329+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.977329+0000","objects_scrubbed":51,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:00.448502+0000","scrub_duration":0.210468818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":214,"num_read_kb":373366,"num_write":1150,"num_write_kb":71086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":337344,"num_omap_keys":660,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"109'2238","reported_seq":5028,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.906190+0000","last_change":"2023-10-30T14:26:34.736009+0000","last_active":"2023-10-30T14:26:39.906190+0000","last_peered":"2023-10-30T14:26:39.906190+0000","last_clean":"2023-10-30T14:26:39.906190+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T14:26:39.906190+0000","last_undegraded":"2023-10-30T14:26:39.906190+0000","last_fullsized":"2023-10-30T14:26:39.906190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2238,"log_dups_size":0,"ondisk_log_size":2238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89413299700000004,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":142,"num_read_kb":18968,"num_write":1278,"num_write_kb":24747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"110'668","reported_seq":989,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:47.033482+0000","last_change":"2023-10-30T14:26:47.033575+0000","last_active":"2023-10-30T14:26:47.033482+0000","last_peered":"2023-10-30T14:26:47.033482+0000","last_clean":"2023-10-30T14:26:47.033482+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T14:26:47.033482+0000","last_undegraded":"2023-10-30T14:26:47.033482+0000","last_fullsized":"2023-10-30T14:26:47.033482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'668","last_scrub_stamp":"2023-10-30T14:26:47.033389+0000","last_deep_scrub":"110'668","last_deep_scrub_stamp":"2023-10-30T14:26:47.033389+0000","last_clean_scrub_stamp":"2023-10-30T14:26:47.033389+0000","objects_scrubbed":42,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:25.680237+0000","scrub_duration":0.095992737999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":280,"num_read_kb":492842,"num_write":679,"num_write_kb":83547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":470248,"num_omap_keys":762,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"109'2096","reported_seq":4971,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.907716+0000","last_change":"2023-10-30T14:26:34.726845+0000","last_active":"2023-10-30T14:26:39.907716+0000","last_peered":"2023-10-30T14:26:39.907716+0000","last_clean":"2023-10-30T14:26:39.907716+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T14:26:39.907716+0000","last_undegraded":"2023-10-30T14:26:39.907716+0000","last_fullsized":"2023-10-30T14:26:39.907716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2096,"log_dups_size":0,"ondisk_log_size":2096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.88483857600000004,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":152,"num_read_kb":61046,"num_write":1310,"num_write_kb":36653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"110'790","reported_seq":1095,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.425710+0000","last_change":"2023-10-30T14:26:52.425764+0000","last_active":"2023-10-30T14:26:52.425710+0000","last_peered":"2023-10-30T14:26:52.425710+0000","last_clean":"2023-10-30T14:26:52.425710+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T14:26:52.425710+0000","last_undegraded":"2023-10-30T14:26:52.425710+0000","last_fullsized":"2023-10-30T14:26:52.425710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'790","last_scrub_stamp":"2023-10-30T14:26:52.425638+0000","last_deep_scrub":"110'790","last_deep_scrub_stamp":"2023-10-30T14:26:52.425638+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.425638+0000","objects_scrubbed":60,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:57:40.747257+0000","scrub_duration":0.18100369799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364699,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":233,"num_read_kb":370768,"num_write":815,"num_write_kb":71238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":549758,"num_omap_keys":876,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"109'2240","reported_seq":5102,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.753065+0000","last_change":"2023-10-30T14:26:33.934502+0000","last_active":"2023-10-30T14:26:40.753065+0000","last_peered":"2023-10-30T14:26:40.753065+0000","last_clean":"2023-10-30T14:26:40.753065+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T14:26:40.753065+0000","last_undegraded":"2023-10-30T14:26:40.753065+0000","last_fullsized":"2023-10-30T14:26:40.753065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2240,"log_dups_size":0,"ondisk_log_size":2240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98441546899999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":178,"num_read_kb":29706,"num_write":1375,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"110'1121","reported_seq":1413,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.293606+0000","last_change":"2023-10-30T14:26:54.293672+0000","last_active":"2023-10-30T14:26:54.293606+0000","last_peered":"2023-10-30T14:26:54.293606+0000","last_clean":"2023-10-30T14:26:54.293606+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T14:26:54.293606+0000","last_undegraded":"2023-10-30T14:26:54.293606+0000","last_fullsized":"2023-10-30T14:26:54.293606+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1121","last_scrub_stamp":"2023-10-30T14:26:54.293526+0000","last_deep_scrub":"110'1121","last_deep_scrub_stamp":"2023-10-30T14:26:54.293526+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.293526+0000","objects_scrubbed":43,"log_size":1121,"log_dups_size":0,"ondisk_log_size":1121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:17.009305+0000","scrub_duration":0.241353279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50332230,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":210,"num_read_kb":406971,"num_write":1089,"num_write_kb":71553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":220904,"num_omap_keys":399,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"109'2040","reported_seq":4892,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.904507+0000","last_change":"2023-10-30T14:26:34.688764+0000","last_active":"2023-10-30T14:26:39.904507+0000","last_peered":"2023-10-30T14:26:39.904507+0000","last_clean":"2023-10-30T14:26:39.904507+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T14:26:39.904507+0000","last_undegraded":"2023-10-30T14:26:39.904507+0000","last_fullsized":"2023-10-30T14:26:39.904507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.84782311300000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":122,"num_read_kb":1096,"num_write":1268,"num_write_kb":17447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"110'119","reported_seq":297,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:47.898917+0000","last_change":"2023-10-30T14:26:47.899032+0000","last_active":"2023-10-30T14:26:47.898917+0000","last_peered":"2023-10-30T14:26:47.898917+0000","last_clean":"2023-10-30T14:26:47.898917+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T14:26:47.898917+0000","last_undegraded":"2023-10-30T14:26:47.898917+0000","last_fullsized":"2023-10-30T14:26:47.898917+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'119","last_scrub_stamp":"2023-10-30T14:26:47.898726+0000","last_deep_scrub":"110'119","last_deep_scrub_stamp":"2023-10-30T14:26:47.898726+0000","last_clean_scrub_stamp":"2023-10-30T14:26:47.898726+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:29.562342+0000","scrub_duration":0.0091457540000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"110'909","reported_seq":1258,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:50.195755+0000","last_change":"2023-10-30T14:26:50.195816+0000","last_active":"2023-10-30T14:26:50.195755+0000","last_peered":"2023-10-30T14:26:50.195755+0000","last_clean":"2023-10-30T14:26:50.195755+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T14:26:50.195755+0000","last_undegraded":"2023-10-30T14:26:50.195755+0000","last_fullsized":"2023-10-30T14:26:50.195755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'909","last_scrub_stamp":"2023-10-30T14:26:50.195649+0000","last_deep_scrub":"110'909","last_deep_scrub_stamp":"2023-10-30T14:26:50.195649+0000","last_clean_scrub_stamp":"2023-10-30T14:26:50.195649+0000","objects_scrubbed":60,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:12.567856+0000","scrub_duration":0.19057717900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331714,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":319,"num_read_kb":439640,"num_write":936,"num_write_kb":75464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":459843,"num_omap_keys":807,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"110'2244","reported_seq":5041,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.842968+0000","last_change":"2023-10-30T14:26:57.843044+0000","last_active":"2023-10-30T14:26:57.842968+0000","last_peered":"2023-10-30T14:26:57.842968+0000","last_clean":"2023-10-30T14:26:57.842968+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T14:26:57.842968+0000","last_undegraded":"2023-10-30T14:26:57.842968+0000","last_fullsized":"2023-10-30T14:26:57.842968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2244","last_scrub_stamp":"2023-10-30T14:26:57.842882+0000","last_deep_scrub":"110'2244","last_deep_scrub_stamp":"2023-10-30T14:26:57.842882+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.842882+0000","objects_scrubbed":46,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:34:34.719838+0000","scrub_duration":0.10386124300000001,"objects_trimmed":0,"snaptrim_duration":1.003812562,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":127,"num_read_kb":26261,"num_write":1294,"num_write_kb":24882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"110'1232","reported_seq":1555,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.159370+0000","last_change":"2023-10-30T14:26:53.159534+0000","last_active":"2023-10-30T14:26:53.159370+0000","last_peered":"2023-10-30T14:26:53.159370+0000","last_clean":"2023-10-30T14:26:53.159370+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T14:26:53.159370+0000","last_undegraded":"2023-10-30T14:26:53.159370+0000","last_fullsized":"2023-10-30T14:26:53.159370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1232","last_scrub_stamp":"2023-10-30T14:26:53.159107+0000","last_deep_scrub":"110'1232","last_deep_scrub_stamp":"2023-10-30T14:26:53.159107+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.159107+0000","objects_scrubbed":46,"log_size":1232,"log_dups_size":0,"ondisk_log_size":1232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:27.847424+0000","scrub_duration":0.24891354399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":256,"num_read_kb":488007,"num_write":1246,"num_write_kb":83351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":346758,"num_omap_keys":680,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"110'2013","reported_seq":4887,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:51.757304+0000","last_change":"2023-10-30T14:26:51.757396+0000","last_active":"2023-10-30T14:26:51.757304+0000","last_peered":"2023-10-30T14:26:51.757304+0000","last_clean":"2023-10-30T14:26:51.757304+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T14:26:51.757304+0000","last_undegraded":"2023-10-30T14:26:51.757304+0000","last_fullsized":"2023-10-30T14:26:51.757304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2013","last_scrub_stamp":"2023-10-30T14:26:51.757217+0000","last_deep_scrub":"110'2013","last_deep_scrub_stamp":"2023-10-30T14:26:51.757217+0000","last_clean_scrub_stamp":"2023-10-30T14:26:51.757217+0000","objects_scrubbed":34,"log_size":2013,"log_dups_size":0,"ondisk_log_size":2013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:49.616919+0000","scrub_duration":0.071006446000000001,"objects_trimmed":0,"snaptrim_duration":0.92159720499999997,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":129,"num_read_kb":13045,"num_write":1222,"num_write_kb":11637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"110'2000","reported_seq":3256,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:58.136825+0000","last_change":"2023-10-30T14:26:58.136871+0000","last_active":"2023-10-30T14:26:58.136825+0000","last_peered":"2023-10-30T14:26:58.136825+0000","last_clean":"2023-10-30T14:26:58.136825+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T14:26:58.136825+0000","last_undegraded":"2023-10-30T14:26:58.136825+0000","last_fullsized":"2023-10-30T14:26:58.136825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2000","last_scrub_stamp":"2023-10-30T14:26:58.136733+0000","last_deep_scrub":"110'2000","last_deep_scrub_stamp":"2023-10-30T14:26:58.136733+0000","last_clean_scrub_stamp":"2023-10-30T14:26:58.136733+0000","objects_scrubbed":51,"log_size":2000,"log_dups_size":0,"ondisk_log_size":2000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:54.997528+0000","scrub_duration":0.11551852899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40060913,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":1167,"num_read_kb":359975,"num_write":2020,"num_write_kb":69890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":329700,"num_omap_keys":585,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"110'2306","reported_seq":5129,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:56.314765+0000","last_change":"2023-10-30T14:26:56.314854+0000","last_active":"2023-10-30T14:26:56.314765+0000","last_peered":"2023-10-30T14:26:56.314765+0000","last_clean":"2023-10-30T14:26:56.314765+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T14:26:56.314765+0000","last_undegraded":"2023-10-30T14:26:56.314765+0000","last_fullsized":"2023-10-30T14:26:56.314765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2306","last_scrub_stamp":"2023-10-30T14:26:56.314673+0000","last_deep_scrub":"110'2306","last_deep_scrub_stamp":"2023-10-30T14:26:56.314673+0000","last_clean_scrub_stamp":"2023-10-30T14:26:56.314673+0000","objects_scrubbed":45,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:16:05.533537+0000","scrub_duration":0.059182301999999999,"objects_trimmed":0,"snaptrim_duration":0.40657480699999998,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":145,"num_read_kb":55105,"num_write":1354,"num_write_kb":28005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"110'1533","reported_seq":1849,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:45.029556+0000","last_change":"2023-10-30T14:26:45.029718+0000","last_active":"2023-10-30T14:26:45.029556+0000","last_peered":"2023-10-30T14:26:45.029556+0000","last_clean":"2023-10-30T14:26:45.029556+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T14:26:45.029556+0000","last_undegraded":"2023-10-30T14:26:45.029556+0000","last_fullsized":"2023-10-30T14:26:45.029556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1533","last_scrub_stamp":"2023-10-30T14:26:45.029381+0000","last_deep_scrub":"110'1533","last_deep_scrub_stamp":"2023-10-30T14:26:45.029381+0000","last_clean_scrub_stamp":"2023-10-30T14:26:45.029381+0000","objects_scrubbed":45,"log_size":1533,"log_dups_size":0,"ondisk_log_size":1533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:54.864240+0000","scrub_duration":0.21060546799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":235,"num_read_kb":509467,"num_write":1537,"num_write_kb":83943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":361192,"num_omap_keys":645,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"110'2242","reported_seq":5077,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.859618+0000","last_change":"2023-10-30T14:26:46.859846+0000","last_active":"2023-10-30T14:26:46.859618+0000","last_peered":"2023-10-30T14:26:46.859618+0000","last_clean":"2023-10-30T14:26:46.859618+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T14:26:46.859618+0000","last_undegraded":"2023-10-30T14:26:46.859618+0000","last_fullsized":"2023-10-30T14:26:46.859618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2242","last_scrub_stamp":"2023-10-30T14:26:46.859379+0000","last_deep_scrub":"110'2242","last_deep_scrub_stamp":"2023-10-30T14:26:46.859379+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.859379+0000","objects_scrubbed":34,"log_size":2242,"log_dups_size":0,"ondisk_log_size":2242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:58:59.480931+0000","scrub_duration":0.104056545,"objects_trimmed":0,"snaptrim_duration":0.59694472600000004,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":143,"num_read_kb":13239,"num_write":1306,"num_write_kb":17120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"110'663","reported_seq":985,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:50.346246+0000","last_change":"2023-10-30T14:26:50.346292+0000","last_active":"2023-10-30T14:26:50.346246+0000","last_peered":"2023-10-30T14:26:50.346246+0000","last_clean":"2023-10-30T14:26:50.346246+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T14:26:50.346246+0000","last_undegraded":"2023-10-30T14:26:50.346246+0000","last_fullsized":"2023-10-30T14:26:50.346246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'663","last_scrub_stamp":"2023-10-30T14:26:50.346173+0000","last_deep_scrub":"110'663","last_deep_scrub_stamp":"2023-10-30T14:26:50.346173+0000","last_clean_scrub_stamp":"2023-10-30T14:26:50.346173+0000","objects_scrubbed":53,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:23:57.407805+0000","scrub_duration":0.108375394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":230,"num_read_kb":520969,"num_write":671,"num_write_kb":95828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":375058,"num_omap_keys":621,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"109'2180","reported_seq":5000,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754491+0000","last_change":"2023-10-30T14:26:34.025524+0000","last_active":"2023-10-30T14:26:40.754491+0000","last_peered":"2023-10-30T14:26:40.754491+0000","last_clean":"2023-10-30T14:26:40.754491+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T14:26:40.754491+0000","last_undegraded":"2023-10-30T14:26:40.754491+0000","last_fullsized":"2023-10-30T14:26:40.754491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2180,"log_dups_size":0,"ondisk_log_size":2180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.075105449,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":60418,"num_write":1286,"num_write_kb":36950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"110'1084","reported_seq":1379,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:59.155850+0000","last_change":"2023-10-30T14:26:59.155993+0000","last_active":"2023-10-30T14:26:59.155850+0000","last_peered":"2023-10-30T14:26:59.155850+0000","last_clean":"2023-10-30T14:26:59.155850+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T14:26:59.155850+0000","last_undegraded":"2023-10-30T14:26:59.155850+0000","last_fullsized":"2023-10-30T14:26:59.155850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1084","last_scrub_stamp":"2023-10-30T14:26:59.155631+0000","last_deep_scrub":"110'1084","last_deep_scrub_stamp":"2023-10-30T14:26:59.155631+0000","last_clean_scrub_stamp":"2023-10-30T14:26:59.155631+0000","objects_scrubbed":38,"log_size":1084,"log_dups_size":0,"ondisk_log_size":1084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:20:05.652224+0000","scrub_duration":0.220506067,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44106025,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":214,"num_read_kb":377318,"num_write":1089,"num_write_kb":69046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":276186,"num_omap_keys":520,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"109'2210","reported_seq":5045,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.183220+0000","last_change":"2023-10-30T14:26:33.629915+0000","last_active":"2023-10-30T14:26:40.183220+0000","last_peered":"2023-10-30T14:26:40.183220+0000","last_clean":"2023-10-30T14:26:40.183220+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T14:26:40.183220+0000","last_undegraded":"2023-10-30T14:26:40.183220+0000","last_fullsized":"2023-10-30T14:26:40.183220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.36741958800000002,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":157,"num_read_kb":35362,"num_write":1332,"num_write_kb":31772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"110'1165","reported_seq":1469,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.451066+0000","last_change":"2023-10-30T14:26:54.451118+0000","last_active":"2023-10-30T14:26:54.451066+0000","last_peered":"2023-10-30T14:26:54.451066+0000","last_clean":"2023-10-30T14:26:54.451066+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T14:26:54.451066+0000","last_undegraded":"2023-10-30T14:26:54.451066+0000","last_fullsized":"2023-10-30T14:26:54.451066+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1165","last_scrub_stamp":"2023-10-30T14:26:54.450996+0000","last_deep_scrub":"110'1165","last_deep_scrub_stamp":"2023-10-30T14:26:54.450996+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.450996+0000","objects_scrubbed":39,"log_size":1165,"log_dups_size":0,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:18:07.890199+0000","scrub_duration":0.17568342300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":247,"num_read_kb":365780,"num_write":1783,"num_write_kb":65025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":395035,"num_omap_keys":693,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"109'2088","reported_seq":4924,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:41.616256+0000","last_change":"2023-10-30T14:26:33.252060+0000","last_active":"2023-10-30T14:26:41.616256+0000","last_peered":"2023-10-30T14:26:41.616256+0000","last_clean":"2023-10-30T14:26:41.616256+0000","last_became_active":"2023-10-30T13:24:01.001412+0000","last_became_peered":"2023-10-30T13:24:01.001412+0000","last_unstale":"2023-10-30T14:26:41.616256+0000","last_undegraded":"2023-10-30T14:26:41.616256+0000","last_fullsized":"2023-10-30T14:26:41.616256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.32700652899999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":137,"num_read_kb":5811,"num_write":1214,"num_write_kb":16143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"110'515","reported_seq":831,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.490148+0000","last_change":"2023-10-30T14:26:53.490236+0000","last_active":"2023-10-30T14:26:53.490148+0000","last_peered":"2023-10-30T14:26:53.490148+0000","last_clean":"2023-10-30T14:26:53.490148+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T14:26:53.490148+0000","last_undegraded":"2023-10-30T14:26:53.490148+0000","last_fullsized":"2023-10-30T14:26:53.490148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'515","last_scrub_stamp":"2023-10-30T14:26:53.490042+0000","last_deep_scrub":"110'515","last_deep_scrub_stamp":"2023-10-30T14:26:53.490042+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.490042+0000","objects_scrubbed":47,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:16.217409+0000","scrub_duration":0.22512389199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":281,"num_read_kb":442743,"num_write":555,"num_write_kb":70797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":243443,"num_omap_keys":457,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"110'2226","reported_seq":5069,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:59.865953+0000","last_change":"2023-10-30T14:26:59.866026+0000","last_active":"2023-10-30T14:26:59.865953+0000","last_peered":"2023-10-30T14:26:59.865953+0000","last_clean":"2023-10-30T14:26:59.865953+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T14:26:59.865953+0000","last_undegraded":"2023-10-30T14:26:59.865953+0000","last_fullsized":"2023-10-30T14:26:59.865953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2226","last_scrub_stamp":"2023-10-30T14:26:59.865860+0000","last_deep_scrub":"110'2226","last_deep_scrub_stamp":"2023-10-30T14:26:59.865860+0000","last_clean_scrub_stamp":"2023-10-30T14:26:59.865860+0000","objects_scrubbed":40,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:25.638448+0000","scrub_duration":0.053687321000000003,"objects_trimmed":0,"snaptrim_duration":1.183697322,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":144,"num_read_kb":25445,"num_write":1326,"num_write_kb":20705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"110'584","reported_seq":860,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.994203+0000","last_change":"2023-10-30T14:26:53.994349+0000","last_active":"2023-10-30T14:26:53.994203+0000","last_peered":"2023-10-30T14:26:53.994203+0000","last_clean":"2023-10-30T14:26:53.994203+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T14:26:53.994203+0000","last_undegraded":"2023-10-30T14:26:53.994203+0000","last_fullsized":"2023-10-30T14:26:53.994203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'584","last_scrub_stamp":"2023-10-30T14:26:53.993981+0000","last_deep_scrub":"110'584","last_deep_scrub_stamp":"2023-10-30T14:26:53.993981+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.993981+0000","objects_scrubbed":40,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:54.497876+0000","scrub_duration":0.12048839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":236,"num_read_kb":274544,"num_write":603,"num_write_kb":42177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":323567,"num_omap_keys":593,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"110'2168","reported_seq":5027,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:49.864350+0000","last_change":"2023-10-30T14:26:49.864486+0000","last_active":"2023-10-30T14:26:49.864350+0000","last_peered":"2023-10-30T14:26:49.864350+0000","last_clean":"2023-10-30T14:26:49.864350+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T14:26:49.864350+0000","last_undegraded":"2023-10-30T14:26:49.864350+0000","last_fullsized":"2023-10-30T14:26:49.864350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2168","last_scrub_stamp":"2023-10-30T14:26:49.864161+0000","last_deep_scrub":"110'2168","last_deep_scrub_stamp":"2023-10-30T14:26:49.864161+0000","last_clean_scrub_stamp":"2023-10-30T14:26:49.864161+0000","objects_scrubbed":45,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:54:18.553003+0000","scrub_duration":0.11066809399999999,"objects_trimmed":0,"snaptrim_duration":0.56267233699999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":144,"num_read_kb":34655,"num_write":1307,"num_write_kb":20329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"110'1112","reported_seq":1394,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:45.299966+0000","last_change":"2023-10-30T14:26:45.300020+0000","last_active":"2023-10-30T14:26:45.299966+0000","last_peered":"2023-10-30T14:26:45.299966+0000","last_clean":"2023-10-30T14:26:45.299966+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T14:26:45.299966+0000","last_undegraded":"2023-10-30T14:26:45.299966+0000","last_fullsized":"2023-10-30T14:26:45.299966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1112","last_scrub_stamp":"2023-10-30T14:26:45.299861+0000","last_deep_scrub":"110'1112","last_deep_scrub_stamp":"2023-10-30T14:26:45.299861+0000","last_clean_scrub_stamp":"2023-10-30T14:26:45.299861+0000","objects_scrubbed":51,"log_size":1112,"log_dups_size":0,"ondisk_log_size":1112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:30.314690+0000","scrub_duration":0.15631188700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":237,"num_read_kb":249991,"num_write":1213,"num_write_kb":43538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":609555,"num_omap_keys":1142,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"109'2271","reported_seq":5059,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893421+0000","last_change":"2023-10-30T14:26:36.405912+0000","last_active":"2023-10-30T14:26:39.893421+0000","last_peered":"2023-10-30T14:26:39.893421+0000","last_clean":"2023-10-30T14:26:39.893421+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T14:26:39.893421+0000","last_undegraded":"2023-10-30T14:26:39.893421+0000","last_fullsized":"2023-10-30T14:26:39.893421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2271,"log_dups_size":0,"ondisk_log_size":2271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.93815086199999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":129,"num_read_kb":1227,"num_write":1352,"num_write_kb":12536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"110'1527","reported_seq":1874,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.480258+0000","last_change":"2023-10-30T14:26:53.480338+0000","last_active":"2023-10-30T14:26:53.480258+0000","last_peered":"2023-10-30T14:26:53.480258+0000","last_clean":"2023-10-30T14:26:53.480258+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T14:26:53.480258+0000","last_undegraded":"2023-10-30T14:26:53.480258+0000","last_fullsized":"2023-10-30T14:26:53.480258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1527","last_scrub_stamp":"2023-10-30T14:26:53.480148+0000","last_deep_scrub":"110'1527","last_deep_scrub_stamp":"2023-10-30T14:26:53.480148+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.480148+0000","objects_scrubbed":54,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:31:02.686045+0000","scrub_duration":0.25185449900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":321,"num_read_kb":438308,"num_write":2003,"num_write_kb":98279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":359842,"num_omap_keys":684,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"110'2123","reported_seq":4973,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:55.746681+0000","last_change":"2023-10-30T14:26:55.746755+0000","last_active":"2023-10-30T14:26:55.746681+0000","last_peered":"2023-10-30T14:26:55.746681+0000","last_clean":"2023-10-30T14:26:55.746681+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T14:26:55.746681+0000","last_undegraded":"2023-10-30T14:26:55.746681+0000","last_fullsized":"2023-10-30T14:26:55.746681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2123","last_scrub_stamp":"2023-10-30T14:26:55.746576+0000","last_deep_scrub":"110'2123","last_deep_scrub_stamp":"2023-10-30T14:26:55.746576+0000","last_clean_scrub_stamp":"2023-10-30T14:26:55.746576+0000","objects_scrubbed":27,"log_size":2123,"log_dups_size":0,"ondisk_log_size":2123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:08:17.663937+0000","scrub_duration":0.039827926,"objects_trimmed":0,"snaptrim_duration":0.94968995099999998,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":141,"num_read_kb":11852,"num_write":1237,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"110'1108","reported_seq":1435,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.161003+0000","last_change":"2023-10-30T14:26:57.161075+0000","last_active":"2023-10-30T14:26:57.161003+0000","last_peered":"2023-10-30T14:26:57.161003+0000","last_clean":"2023-10-30T14:26:57.161003+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T14:26:57.161003+0000","last_undegraded":"2023-10-30T14:26:57.161003+0000","last_fullsized":"2023-10-30T14:26:57.161003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1108","last_scrub_stamp":"2023-10-30T14:26:57.160913+0000","last_deep_scrub":"110'1108","last_deep_scrub_stamp":"2023-10-30T14:26:57.160913+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.160913+0000","objects_scrubbed":44,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:56:50.729954+0000","scrub_duration":0.25105180199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":296,"num_read_kb":429426,"num_write":1159,"num_write_kb":75926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":378159,"num_omap_keys":718,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"109'2164","reported_seq":5001,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.904567+0000","last_change":"2023-10-30T14:26:33.347987+0000","last_active":"2023-10-30T14:26:39.904567+0000","last_peered":"2023-10-30T14:26:39.904567+0000","last_clean":"2023-10-30T14:26:39.904567+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T14:26:39.904567+0000","last_undegraded":"2023-10-30T14:26:39.904567+0000","last_fullsized":"2023-10-30T14:26:39.904567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2164,"log_dups_size":0,"ondisk_log_size":2164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.419978554,"stat_sum":{"num_bytes":5,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":137,"num_read_kb":40109,"num_write":1287,"num_write_kb":25283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"109'2180","reported_seq":4989,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.753661+0000","last_change":"2023-10-30T14:26:34.023924+0000","last_active":"2023-10-30T14:26:40.753661+0000","last_peered":"2023-10-30T14:26:40.753661+0000","last_clean":"2023-10-30T14:26:40.753661+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T14:26:40.753661+0000","last_undegraded":"2023-10-30T14:26:40.753661+0000","last_fullsized":"2023-10-30T14:26:40.753661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2180,"log_dups_size":0,"ondisk_log_size":2180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.073619836,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":129,"num_read_kb":12975,"num_write":1318,"num_write_kb":20116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"110'784","reported_seq":1084,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:56.157617+0000","last_change":"2023-10-30T14:26:56.157707+0000","last_active":"2023-10-30T14:26:56.157617+0000","last_peered":"2023-10-30T14:26:56.157617+0000","last_clean":"2023-10-30T14:26:56.157617+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T14:26:56.157617+0000","last_undegraded":"2023-10-30T14:26:56.157617+0000","last_fullsized":"2023-10-30T14:26:56.157617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'784","last_scrub_stamp":"2023-10-30T14:26:56.157430+0000","last_deep_scrub":"110'784","last_deep_scrub_stamp":"2023-10-30T14:26:56.157430+0000","last_clean_scrub_stamp":"2023-10-30T14:26:56.157430+0000","objects_scrubbed":50,"log_size":784,"log_dups_size":0,"ondisk_log_size":784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:19.819331+0000","scrub_duration":0.231260298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":304,"num_read_kb":264369,"num_write":877,"num_write_kb":48007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":477325,"num_omap_keys":851,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"109'2642","reported_seq":5328,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.753827+0000","last_change":"2023-10-30T14:26:33.971887+0000","last_active":"2023-10-30T14:26:40.753827+0000","last_peered":"2023-10-30T14:26:40.753827+0000","last_clean":"2023-10-30T14:26:40.753827+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T14:26:40.753827+0000","last_undegraded":"2023-10-30T14:26:40.753827+0000","last_fullsized":"2023-10-30T14:26:40.753827+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2642,"log_dups_size":0,"ondisk_log_size":2642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.021840552,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":171,"num_read_kb":40337,"num_write":1456,"num_write_kb":34441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"103'1179","reported_seq":1481,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.904903+0000","last_change":"2023-10-30T13:23:18.987633+0000","last_active":"2023-10-30T14:26:39.904903+0000","last_peered":"2023-10-30T14:26:39.904903+0000","last_clean":"2023-10-30T14:26:39.904903+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T14:26:39.904903+0000","last_undegraded":"2023-10-30T14:26:39.904903+0000","last_fullsized":"2023-10-30T14:26:39.904903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:17.917509+0000","last_clean_scrub_stamp":"2023-10-30T13:23:17.917509+0000","objects_scrubbed":0,"log_size":1179,"log_dups_size":0,"ondisk_log_size":1179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":255,"num_read_kb":502436,"num_write":1236,"num_write_kb":99955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"109'2028","reported_seq":4897,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.756778+0000","last_change":"2023-10-30T14:26:34.532204+0000","last_active":"2023-10-30T14:26:40.756778+0000","last_peered":"2023-10-30T14:26:40.756778+0000","last_clean":"2023-10-30T14:26:40.756778+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T14:26:40.756778+0000","last_undegraded":"2023-10-30T14:26:40.756778+0000","last_fullsized":"2023-10-30T14:26:40.756778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2028,"log_dups_size":0,"ondisk_log_size":2028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.015373858,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":13189,"num_write":1249,"num_write_kb":15084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"110'817","reported_seq":1081,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:45.554360+0000","last_change":"2023-10-30T14:26:45.554415+0000","last_active":"2023-10-30T14:26:45.554360+0000","last_peered":"2023-10-30T14:26:45.554360+0000","last_clean":"2023-10-30T14:26:45.554360+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T14:26:45.554360+0000","last_undegraded":"2023-10-30T14:26:45.554360+0000","last_fullsized":"2023-10-30T14:26:45.554360+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'817","last_scrub_stamp":"2023-10-30T14:26:45.554286+0000","last_deep_scrub":"110'817","last_deep_scrub_stamp":"2023-10-30T14:26:45.554286+0000","last_clean_scrub_stamp":"2023-10-30T14:26:45.554286+0000","objects_scrubbed":38,"log_size":817,"log_dups_size":0,"ondisk_log_size":817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:36.415824+0000","scrub_duration":0.061981296999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":159,"num_read_kb":369044,"num_write":787,"num_write_kb":79251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":283692,"num_omap_keys":523,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"110'2229","reported_seq":5188,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.797248+0000","last_change":"2023-10-30T14:27:00.797301+0000","last_active":"2023-10-30T14:27:00.797248+0000","last_peered":"2023-10-30T14:27:00.797248+0000","last_clean":"2023-10-30T14:27:00.797248+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T14:27:00.797248+0000","last_undegraded":"2023-10-30T14:27:00.797248+0000","last_fullsized":"2023-10-30T14:27:00.797248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2229","last_scrub_stamp":"2023-10-30T14:27:00.797179+0000","last_deep_scrub":"110'2229","last_deep_scrub_stamp":"2023-10-30T14:27:00.797179+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.797179+0000","objects_scrubbed":37,"log_size":2229,"log_dups_size":0,"ondisk_log_size":2229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:23.560124+0000","scrub_duration":0.084074704,"objects_trimmed":0,"snaptrim_duration":1.043790548,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":262,"num_read_kb":42307,"num_write":1336,"num_write_kb":30630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"110'1065","reported_seq":1395,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.937061+0000","last_change":"2023-10-30T14:26:42.937107+0000","last_active":"2023-10-30T14:26:42.937061+0000","last_peered":"2023-10-30T14:26:42.937061+0000","last_clean":"2023-10-30T14:26:42.937061+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T14:26:42.937061+0000","last_undegraded":"2023-10-30T14:26:42.937061+0000","last_fullsized":"2023-10-30T14:26:42.937061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1065","last_scrub_stamp":"2023-10-30T14:26:42.936998+0000","last_deep_scrub":"110'1065","last_deep_scrub_stamp":"2023-10-30T14:26:42.936998+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.936998+0000","objects_scrubbed":55,"log_size":1065,"log_dups_size":0,"ondisk_log_size":1065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:27.046930+0000","scrub_duration":0.131982027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777798,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":313,"num_read_kb":340671,"num_write":1085,"num_write_kb":63851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":501138,"num_omap_keys":760,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":11,"num_objects":2377,"num_object_clones":2,"num_object_copies":4754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":9389,"num_read_kb":1724409,"num_write":83589,"num_write_kb":1459751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139356,"ondisk_log_size":139356,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1285101348,"num_objects":1569,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1569,"num_whiteouts":0,"num_read":10652,"num_read_kb":12762809,"num_write":35655,"num_write_kb":2343865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12730482,"num_omap_keys":22652,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2570362880,"data_stored":2570202696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33514,"ondisk_log_size":33514,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575619,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590984,"kb_used_data":389120,"kb_used_omap":1,"kb_used_meta":201854,"kb_avail":93780856,"statfs":{"total":96636764160,"available":96031596544,"internally_reserved":0,"allocated":398458880,"data_stored":397803489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":206698955},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.73999999999999999,"15min":0.79700000000000004},"min":{"1min":0.41399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":1.236},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.82699999999999996,"15min":0.91000000000000003},"min":{"1min":0.68100000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.306}]},{"osd":1,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.80700000000000005,"15min":0.86599999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.288,"5min":2.452,"15min":2.452},"last":1.129},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.79000000000000004,"15min":0.83999999999999997},"min":{"1min":0.56499999999999995,"5min":0.495,"15min":0.47399999999999998},"max":{"1min":3.7160000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":1.2}]},{"osd":2,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.75800000000000001,"15min":0.81999999999999995},"min":{"1min":0.45200000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.96299999999999997,"5min":1.169,"15min":1.4350000000000001},"last":1.089},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.84799999999999998,"15min":0.93600000000000005},"min":{"1min":0.52700000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2210000000000001,"5min":1.2929999999999999,"15min":1.5569999999999999},"last":1.403}]},{"osd":3,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94899999999999995,"15min":0.95099999999999996},"min":{"1min":0.60499999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.0790000000000002,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":1.335},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.026,"15min":1.0029999999999999},"min":{"1min":0.67000000000000004,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.7869999999999999,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":1.2709999999999999}]},{"osd":4,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.91900000000000004,"15min":0.90200000000000002},"min":{"1min":0.63200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.6629999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.202,"5min":1.0660000000000001,"15min":1.042},"min":{"1min":0.84099999999999997,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":3.6899999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":0.746}]},{"osd":5,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.139,"15min":1.0880000000000001},"min":{"1min":0.75800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.4179999999999999,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":1.02},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.127,"15min":1.1599999999999999},"min":{"1min":0.88300000000000001,"5min":0.436,"15min":0.436},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.99299999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1379999999999999,"15min":1.1140000000000001},"min":{"1min":0.50800000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":6.3070000000000004,"5min":6.3070000000000004,"15min":6.3070000000000004},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.054,"5min":1.1279999999999999,"15min":1.181},"min":{"1min":0.65700000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.1579999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575622,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505392,"kb_used_data":339688,"kb_used_omap":1,"kb_used_meta":165694,"kb_avail":93866448,"statfs":{"total":96636764160,"available":96119242752,"internally_reserved":0,"allocated":347840512,"data_stored":347188935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169671111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.75700000000000001,"15min":0.80100000000000005},"min":{"1min":0.51100000000000001,"5min":0.46700000000000003,"15min":0.375},"max":{"1min":0.89600000000000002,"5min":1.306,"15min":1.363},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.88900000000000001,"15min":0.95299999999999996},"min":{"1min":0.53500000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3979999999999999,"5min":1.4099999999999999,"15min":1.4370000000000001},"last":1.018}]},{"osd":1,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.72299999999999998,"15min":0.77400000000000002},"min":{"1min":0.54100000000000004,"5min":0.46999999999999997,"15min":0.46800000000000003},"max":{"1min":0.875,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.47399999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.127,"5min":1.706,"15min":1.706},"last":0.68700000000000006}]},{"osd":2,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.81299999999999994,"15min":0.82999999999999996},"min":{"1min":0.57699999999999996,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.1679999999999999,"5min":1.29,"15min":1.571},"last":1.125},{"interface":"front","average":{"1min":0.872,"5min":0.96899999999999997,"15min":0.93700000000000006},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.286,"5min":5.4480000000000004,"15min":5.4480000000000004},"last":0.76900000000000002}]},{"osd":3,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0229999999999999,"15min":1.0329999999999999},"min":{"1min":0.68000000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.171},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.103,"15min":1.1160000000000001},"min":{"1min":0.68899999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.23,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.95299999999999996}]},{"osd":4,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.90300000000000002,"15min":0.90700000000000003},"min":{"1min":0.70799999999999996,"5min":0.434,"15min":0.434},"max":{"1min":1.276,"5min":1.3640000000000001,"15min":1.4339999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.002,"15min":1.0309999999999999},"min":{"1min":0.71999999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.153,"5min":1.595,"15min":1.595},"last":1.05}]},{"osd":5,"last update":"Mon Oct 30 14:26:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.159,"15min":1.03},"min":{"1min":0.77400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.2889999999999999,"5min":3.0219999999999998,"15min":3.0219999999999998},"last":1.077},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.0089999999999999,"15min":1.0509999999999999},"min":{"1min":0.75600000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.135,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":0.72199999999999998}]},{"osd":7,"last update":"Mon Oct 30 14:26:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.204,"15min":1.196},"min":{"1min":1.048,"5min":0.58699999999999997,"15min":0.52600000000000002},"max":{"1min":1.3740000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":1.22},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1599999999999999,"15min":1.1359999999999999},"min":{"1min":0.83399999999999996,"5min":0.752,"15min":0.63700000000000001},"max":{"1min":1.3440000000000001,"5min":2.806,"15min":2.806},"last":0.995}]}]},{"osd":5,"up_from":13,"seq":55834575619,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570756,"kb_used_data":368956,"kb_used_omap":1,"kb_used_meta":201790,"kb_avail":93801084,"statfs":{"total":96636764160,"available":96052310016,"internally_reserved":0,"allocated":377810944,"data_stored":377158158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206633414},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.84599999999999997},"min":{"1min":0.439,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1120000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.93700000000000006,"15min":0.95699999999999996},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.373,"5min":1.617,"15min":1.9219999999999999},"last":1.266}]},{"osd":1,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.81799999999999995,"15min":0.83899999999999997},"min":{"1min":0.48899999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.77100000000000002,"15min":0.83999999999999997},"min":{"1min":0.53700000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.774,"5min":4.774,"15min":4.774},"last":0.89600000000000002}]},{"osd":2,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85199999999999998,"15min":0.88100000000000001},"min":{"1min":0.58599999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.111,"5min":1.2589999999999999,"15min":1.331},"last":1.111},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.84699999999999998},"min":{"1min":0.501,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.1879999999999999,"5min":1.331,"15min":1.796},"last":1.1879999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.96999999999999997,"15min":1.0009999999999999},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3100000000000001,"5min":1.536,"15min":1.536},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.036,"5min":0.92600000000000005,"15min":0.99099999999999999},"min":{"1min":0.51800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":2.0030000000000001},"last":1.3520000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.88800000000000001,"15min":0.877},"min":{"1min":0.374,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.087,"5min":2.8170000000000002,"15min":2.8170000000000002},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.155,"5min":1.0609999999999999,"15min":1.036},"min":{"1min":0.45100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.4690000000000001,"5min":2.2400000000000002,"15min":2.262},"last":1.4299999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.071,"15min":1.1220000000000001},"min":{"1min":0.40300000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3380000000000001,"5min":2.3090000000000002,"15min":2.7949999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.038,"15min":1.054},"min":{"1min":0.38800000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.4350000000000001,"5min":2.161,"15min":2.1629999999999998},"last":1.399}]},{"osd":7,"last update":"Mon Oct 30 14:26:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0209999999999999,"15min":1.091},"min":{"1min":0.78100000000000003,"5min":0.317,"15min":0.317},"max":{"1min":1.3440000000000001,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.147},{"interface":"front","average":{"1min":1.149,"5min":1.1299999999999999,"15min":1.153},"min":{"1min":0.95099999999999996,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.4019999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.226}]}]},{"osd":4,"up_from":13,"seq":55834575619,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469348,"kb_used_data":267484,"kb_used_omap":1,"kb_used_meta":201854,"kb_avail":93902492,"statfs":{"total":96636764160,"available":96156151808,"internally_reserved":0,"allocated":273903616,"data_stored":273238103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206698950},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"min":{"1min":0.48799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1259999999999999,"5min":1.3500000000000001,"15min":1.446},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.71099999999999997,"15min":0.82299999999999995},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.875,"5min":1.659,"15min":1.659},"last":0.67000000000000004}]},{"osd":1,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.56799999999999995,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.141,"5min":1.589,"15min":1.589},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.84399999999999997,"15min":0.84699999999999998},"min":{"1min":0.58999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.075,"5min":1.532,"15min":1.532},"last":0.91400000000000003}]},{"osd":2,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.76100000000000001,"15min":0.85999999999999999},"min":{"1min":0.55800000000000005,"5min":0.5,"15min":0.5},"max":{"1min":0.94899999999999995,"5min":1.607,"15min":1.607},"last":0.627},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84699999999999998,"15min":0.92200000000000004},"min":{"1min":0.57199999999999995,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.218,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.53400000000000003}]},{"osd":3,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.99099999999999999,"15min":0.97499999999999998},"min":{"1min":0.66300000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.242,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.995},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.099,"15min":1.0780000000000001},"min":{"1min":0.879,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3480000000000001,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.089}]},{"osd":5,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.98899999999999999,"15min":1.042},"min":{"1min":0.77000000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.3009999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.65400000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.1279999999999999,"5min":1.3220000000000001,"15min":1.3280000000000001},"last":1.0329999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.97899999999999998,"15min":0.96099999999999997},"min":{"1min":0.78800000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.3100000000000001,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.042,"15min":1.0700000000000001},"min":{"1min":0.78300000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.238,"5min":1.677,"15min":1.728},"last":0.86699999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.137,"15min":1.1040000000000001},"min":{"1min":1.0820000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.417,"5min":1.4750000000000001,"15min":1.518},"last":1.143},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.204,"15min":1.1799999999999999},"min":{"1min":1.056,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4670000000000001,"5min":1.587,"15min":1.587},"last":1.24}]}]},{"osd":3,"up_from":13,"seq":55834575617,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593536,"kb_used_data":382008,"kb_used_omap":1,"kb_used_meta":211518,"kb_avail":93778304,"statfs":{"total":96636764160,"available":96028983296,"internally_reserved":0,"allocated":391176192,"data_stored":390510799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":216594888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38100000000000001,"15min":0.40600000000000003},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71799999999999997,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.308},"max":{"1min":0.747,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.42599999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":0.76500000000000001,"5min":1.403,"15min":1.403},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.51700000000000002,"15min":0.54800000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65800000000000003,"5min":1.752,"15min":1.752},"last":0.45900000000000002}]},{"osd":2,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56399999999999995,"15min":0.59999999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69499999999999995,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.41699999999999998,"5min":0.29999999999999999,"15min":0.29299999999999998},"max":{"1min":0.78000000000000003,"5min":1.73,"15min":1.73},"last":0.65800000000000003}]},{"osd":4,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.80300000000000005},"min":{"1min":0.53600000000000003,"5min":0.379,"15min":0.379},"max":{"1min":1.6519999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.80400000000000005,"15min":0.82399999999999995},"min":{"1min":0.55800000000000005,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.679,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.52500000000000002}]},{"osd":5,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77400000000000002,"15min":0.83299999999999996},"min":{"1min":0.61499999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98899999999999999,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.81000000000000005,"15min":0.85699999999999998},"min":{"1min":0.58399999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.071,"5min":1.486,"15min":1.486},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.85199999999999998,"15min":0.88400000000000001},"min":{"1min":0.55600000000000005,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.048,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.91500000000000004,"15min":0.93999999999999995},"min":{"1min":0.71999999999999997,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0409999999999999,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.99299999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:26:01 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83199999999999996,"15min":0.86199999999999999},"min":{"1min":0.64200000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.1100000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79100000000000004,"15min":0.82599999999999996},"min":{"1min":0.56299999999999994,"5min":0.501,"15min":0.501},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.68400000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299412,"kb_used_data":200268,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94072428,"statfs":{"total":96636764160,"available":96330166272,"internally_reserved":0,"allocated":205074432,"data_stored":204423364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101513670},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.433,"15min":0.45800000000000002},"min":{"1min":0.254,"5min":0.246,"15min":0.23899999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.89300000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":0.94599999999999995,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":0.58399999999999996}]},{"osd":1,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.42199999999999999,"15min":0.44700000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72999999999999998,"5min":1.956,"15min":1.956},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.36199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.53000000000000003}]},{"osd":3,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81999999999999995,"5min":1.988,"15min":1.988},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49099999999999999,"15min":0.51800000000000002},"min":{"1min":0.32300000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.90100000000000002,"5min":2.335,"15min":2.335},"last":0.497}]},{"osd":4,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.60899999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5409999999999999,"5min":2.302,"15min":2.302},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84299999999999997,"15min":0.84099999999999997},"min":{"1min":0.625,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6879999999999999,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.81499999999999995}]},{"osd":5,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.77100000000000002,"15min":0.80600000000000005},"min":{"1min":0.64500000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.755},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82699999999999996,"15min":0.84499999999999997},"min":{"1min":0.73099999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.587,"5min":1.875,"15min":1.875},"last":0.86699999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.81599999999999995,"15min":0.82799999999999996},"min":{"1min":0.61299999999999999,"5min":0.502,"15min":0.48099999999999998},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84499999999999997,"15min":0.86799999999999999},"min":{"1min":0.60999999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.429,"5min":1.921,"15min":1.921},"last":0.80500000000000005}]},{"osd":7,"last update":"Mon Oct 30 14:26:14 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.86099999999999999,"15min":0.86399999999999999},"min":{"1min":0.66100000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.131,"5min":4.5739999999999998,"15min":4.5739999999999998},"last":0.995},{"interface":"front","average":{"1min":0.871,"5min":0.88900000000000001,"15min":0.91700000000000004},"min":{"1min":0.64300000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6080000000000001,"5min":2.375,"15min":2.375},"last":0.83999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575620,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494860,"kb_used_data":329092,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93876980,"statfs":{"total":96636764160,"available":96130027520,"internally_reserved":0,"allocated":336990208,"data_stored":336334619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.438,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.504,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.51600000000000001},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.47799999999999998,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.69699999999999995}]},{"osd":2,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.46999999999999997,"15min":0.48399999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54800000000000004,"5min":1.954,"15min":1.954},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.45800000000000002,"15min":0.47799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63200000000000001,"5min":2.274,"15min":2.274},"last":0.54000000000000004}]},{"osd":3,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53600000000000003,"15min":0.54900000000000004},"min":{"1min":0.35599999999999998,"5min":0.35399999999999998,"15min":0.35099999999999998},"max":{"1min":0.60099999999999998,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.436,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.70199999999999996,"5min":2.625,"15min":2.625},"last":0.746}]},{"osd":4,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.80200000000000005,"15min":0.83999999999999997},"min":{"1min":0.56499999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93300000000000005,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.82299999999999995,"15min":0.88},"min":{"1min":0.629,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.91200000000000003,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":1.004}]},{"osd":5,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82399999999999995,"15min":0.878},"min":{"1min":0.58099999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94799999999999995,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.85099999999999998,"15min":0.90100000000000002},"min":{"1min":0.60399999999999998,"5min":0.499,"15min":0.499},"max":{"1min":0.879,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.85699999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.91000000000000003,"15min":0.84199999999999997},"min":{"1min":0.73799999999999999,"5min":0.624,"15min":0.56499999999999995},"max":{"1min":1.0289999999999999,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.81299999999999994,"15min":0.89700000000000002},"min":{"1min":0.56200000000000006,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.95799999999999996,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.81799999999999995}]},{"osd":7,"last update":"Mon Oct 30 14:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.80600000000000005,"15min":0.82999999999999996},"min":{"1min":0.59599999999999997,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.95999999999999996,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":1.085},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.82799999999999996,"15min":0.85199999999999998},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93799999999999994,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.046}]}]},{"osd":0,"up_from":13,"seq":55834575617,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421188,"kb_used_data":245756,"kb_used_omap":1,"kb_used_meta":175422,"kb_avail":93950652,"statfs":{"total":96636764160,"available":96205467648,"internally_reserved":0,"allocated":251654144,"data_stored":251001762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179632582},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.377,"15min":0.40100000000000002},"min":{"1min":0.26400000000000001,"5min":0.247,"15min":0.23499999999999999},"max":{"1min":0.61699999999999999,"5min":0.66100000000000003,"15min":0.94999999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48499999999999999,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65800000000000003,"5min":0.82299999999999995,"15min":1.0229999999999999},"last":0.49199999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.45400000000000001,"15min":0.46200000000000002},"min":{"1min":0.33400000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47699999999999998,"15min":0.496},"min":{"1min":0.38100000000000001,"5min":0.30499999999999999,"15min":0.29099999999999998},"max":{"1min":0.83599999999999997,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.51300000000000001}]},{"osd":3,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.91900000000000004,"5min":1.236,"15min":1.236},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51800000000000002,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78200000000000003,"15min":0.79700000000000004},"min":{"1min":0.58899999999999997,"5min":0.503,"15min":0.503},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.747,"15min":0.80000000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91200000000000003,"5min":1.012,"15min":1.2210000000000001},"last":0.71899999999999997}]},{"osd":5,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.78700000000000003,"15min":0.80900000000000005},"min":{"1min":0.61299999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.80700000000000005,"15min":0.84499999999999997},"min":{"1min":0.626,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.80900000000000005}]},{"osd":6,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.80800000000000005,"15min":0.83799999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.99399999999999999,"5min":1.222,"15min":1.486},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.80600000000000005,"15min":0.82899999999999996},"min":{"1min":0.66900000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.73799999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.82399999999999995,"15min":0.84699999999999998},"min":{"1min":0.70999999999999996,"5min":0.625,"15min":0.58599999999999997},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81799999999999995,"15min":0.86199999999999999},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0680000000000001,"5min":1.1559999999999999,"15min":1.4419999999999999},"last":0.92400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249647104,"data_stored":249627559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335572992,"data_stored":335549765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203661312,"data_stored":203639529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389746688,"data_stored":389725939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":272470016,"data_stored":272453238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376397824,"data_stored":376374323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":346419200,"data_stored":346404076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":396447744,"data_stored":396428267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":11,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:27:05.253 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.253 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.253 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.254 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.254 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.254 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.255 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.255 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.255 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.256 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.256 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.256 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.257 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.257 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.258 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.258 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.258 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.259 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.259 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.260 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.260 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.260 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.261 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.261 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.261 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.262 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.262 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.263 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.263 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.264 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.264 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.264 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.265 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.265 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.266 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.266 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.267 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.267 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.267 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.268 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.268 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.269 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-30T13:23:17.917509+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=17, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.269 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:05.269 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T14:27:11.366 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:27:11.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:27:25.270 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:27:25.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f4578454640 1 Processor -- start 2023-10-30T14:27:25.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f4578454640 1 -- start start 2023-10-30T14:27:25.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f4578454640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457015d6f0 0x7f457015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:25.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f4578454640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45700cb800 con 0x7f45701538a0 2023-10-30T14:27:25.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f4578454640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45700cb330 con 0x7f45700cf160 2023-10-30T14:27:25.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f4578454640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45700cad70 con 0x7f457015d6f0 2023-10-30T14:27:25.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45761f0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f45701538a0 0x7f45700ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55098/0 (socket says 172.21.15.46:55098) 2023-10-30T14:27:25.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45761f0640 1 -- 172.21.15.46:0/354391990 learned_addr learned my addr 172.21.15.46:0/354391990 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:27:25.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45759ef640 1 --2- 172.21.15.46:0/354391990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457015d6f0 0x7f457015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:25.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821592080 0 0) 0x7f45700cb800 con 0x7f45701538a0 2023-10-30T14:27:25.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455c003530 con 0x7f45701538a0 2023-10-30T14:27:25.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1501001119 0 0) 0x7f455c003530 con 0x7f45701538a0 2023-10-30T14:27:25.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457015d6f0 msgr2=0x7f457015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:25.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 --2- 172.21.15.46:0/354391990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457015d6f0 0x7f457015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:25.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 >> v1:172.21.15.154:6789/0 conn(0x7f45700cf160 legacy=0x7f4570164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:25.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45700cb050 con 0x7f45701538a0 2023-10-30T14:27:25.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f456c001680 con 0x7f45701538a0 2023-10-30T14:27:25.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 >> v1:172.21.15.46:6789/0 conn(0x7f45701538a0 legacy=0x7f45700ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:25.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 --2- 172.21.15.46:0/354391990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457015d6f0 0x7f455c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:25.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 --2- 172.21.15.46:0/354391990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f455c003f30 0x7f455c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:25.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 --2- 172.21.15.46:0/354391990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455c004d10 0x7f455c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:25.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45700cad70 con 0x7f455c004d10 2023-10-30T14:27:25.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c00bb00 con 0x7f455c003f30 2023-10-30T14:27:25.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.421+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455c00bcc0 con 0x7f457015d6f0 2023-10-30T14:27:25.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.425+0000 7f45759ef640 1 --2- 172.21.15.46:0/354391990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f455c003f30 0x7f455c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:25.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.425+0000 7f45761f0640 1 --2- 172.21.15.46:0/354391990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457015d6f0 0x7f455c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:25.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.425+0000 7f45769f1640 1 --2- 172.21.15.46:0/354391990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455c004d10 0x7f455c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:25.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.425+0000 7f45769f1640 1 -- 172.21.15.46:0/354391990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457015d6f0 msgr2=0x7f455c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:25.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.425+0000 7f45769f1640 1 --2- 172.21.15.46:0/354391990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457015d6f0 0x7f455c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:25.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.425+0000 7f45769f1640 1 -- 172.21.15.46:0/354391990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f455c003f30 msgr2=0x7f455c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:25.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.425+0000 7f45769f1640 1 --2- 172.21.15.46:0/354391990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f455c003f30 0x7f455c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:25.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.425+0000 7f45769f1640 1 -- 172.21.15.46:0/354391990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455c003530 con 0x7f455c004d10 2023-10-30T14:27:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.425+0000 7f45769f1640 1 --2- 172.21.15.46:0/354391990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455c004d10 0x7f455c005190 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f4568001e70 tx=0x7f456801c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ab6f2d1b73ee07 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.425+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568003b80 con 0x7f455c004d10 2023-10-30T14:27:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.425+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4568003ce0 con 0x7f455c004d10 2023-10-30T14:27:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.425+0000 7f45751ee640 1 -- 172.21.15.46:0/354391990 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45680052c0 con 0x7f455c004d10 2023-10-30T14:27:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 -- 172.21.15.46:0/354391990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455c004d10 msgr2=0x7f455c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 --2- 172.21.15.46:0/354391990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455c004d10 0x7f455c005190 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f4568001e70 tx=0x7f456801c760 comp rx=0 tx=0).stop 2023-10-30T14:27:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f45769f1640 1 -- 172.21.15.46:0/354391990 reap_dead start 2023-10-30T14:27:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 -- 172.21.15.46:0/354391990 shutdown_connections 2023-10-30T14:27:25.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 -- 172.21.15.46:0/354391990 >> 172.21.15.46:0/354391990 conn(0x7f457006bfe0 msgr2=0x7f457006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:25.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 -- 172.21.15.46:0/354391990 shutdown_connections 2023-10-30T14:27:25.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 -- 172.21.15.46:0/354391990 wait complete. 2023-10-30T14:27:25.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 Processor -- start 2023-10-30T14:27:25.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 -- start start 2023-10-30T14:27:25.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455c004d10 0x7f45701efe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:25.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457014d690 0x7f457014df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:25.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f457015b1b0 0x7f457014e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:25.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4570150bf0 con 0x7f455c004d10 2023-10-30T14:27:25.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45701ef3e0 con 0x7f457015b1b0 2023-10-30T14:27:25.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f4578454640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45700b50e0 con 0x7f457014d690 2023-10-30T14:27:25.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f45761f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455c004d10 0x7f45701efe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:25.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f45761f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455c004d10 0x7f45701efe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39202/0 (socket says 172.21.15.46:39202) 2023-10-30T14:27:25.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f45761f0640 1 -- 172.21.15.46:0/594203740 learned_addr learned my addr 172.21.15.46:0/594203740 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:27:25.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f45769f1640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f457015b1b0 0x7f457014e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:25.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.429+0000 7f45759ef640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457014d690 0x7f457014df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:25.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f45761f0640 1 -- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457014d690 msgr2=0x7f457014df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:25.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f45761f0640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457014d690 0x7f457014df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:25.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f45761f0640 1 -- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f457015b1b0 msgr2=0x7f457014e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f45761f0640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f457015b1b0 0x7f457014e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f45761f0640 1 -- 172.21.15.46:0/594203740 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45701f33f0 con 0x7f455c004d10 2023-10-30T14:27:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f45761f0640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455c004d10 0x7f45701efe80 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f456c007300 tx=0x7f456c0077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f4566ffd640 1 -- 172.21.15.46:0/594203740 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c00fa80 con 0x7f455c004d10 2023-10-30T14:27:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f4566ffd640 1 -- 172.21.15.46:0/594203740 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456c00fc20 con 0x7f455c004d10 2023-10-30T14:27:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f4566ffd640 1 -- 172.21.15.46:0/594203740 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c0133f0 con 0x7f455c004d10 2023-10-30T14:27:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f4578454640 1 -- 172.21.15.46:0/594203740 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45701ef680 con 0x7f455c004d10 2023-10-30T14:27:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f4578454640 1 -- 172.21.15.46:0/594203740 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45701f36d0 con 0x7f455c004d10 2023-10-30T14:27:25.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f4578454640 1 -- 172.21.15.46:0/594203740 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457004e170 con 0x7f455c004d10 2023-10-30T14:27:25.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.433+0000 7f4566ffd640 1 -- 172.21.15.46:0/594203740 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f456c013550 con 0x7f455c004d10 2023-10-30T14:27:25.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.437+0000 7f4566ffd640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45540664d0 0x7f4554068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:25.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.437+0000 7f4566ffd640 1 -- 172.21.15.46:0/594203740 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f456c084d50 con 0x7f455c004d10 2023-10-30T14:27:25.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.437+0000 7f45759ef640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45540664d0 0x7f4554068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:25.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.437+0000 7f45759ef640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45540664d0 0x7f4554068990 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f45701e3b80 tx=0x7f4560008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:25.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.441+0000 7f4566ffd640 1 -- 172.21.15.46:0/594203740 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f456c051610 con 0x7f455c004d10 2023-10-30T14:27:25.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.593+0000 7f4578454640 1 -- 172.21.15.46:0/594203740 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4570159b70 con 0x7f45540664d0 2023-10-30T14:27:25.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.617+0000 7f4566ffd640 1 -- 172.21.15.46:0/594203740 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294951 (secure 0 0 0) 0x7f4560009790 con 0x7f45540664d0 2023-10-30T14:27:25.624 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:27:25.626 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T14:27:25.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.621+0000 7f4578454640 1 -- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45540664d0 msgr2=0x7f4554068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:25.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.621+0000 7f4578454640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45540664d0 0x7f4554068990 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f45701e3b80 tx=0x7f4560008040 comp rx=0 tx=0).stop 2023-10-30T14:27:25.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.621+0000 7f4578454640 1 -- 172.21.15.46:0/594203740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455c004d10 msgr2=0x7f45701efe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:25.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.621+0000 7f4578454640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455c004d10 0x7f45701efe80 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f456c007300 tx=0x7f456c0077c0 comp rx=0 tx=0).stop 2023-10-30T14:27:25.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.621+0000 7f4578454640 1 -- 172.21.15.46:0/594203740 shutdown_connections 2023-10-30T14:27:25.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.621+0000 7f4578454640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f45540664d0 0x7f4554068990 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:25.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.621+0000 7f4578454640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f457015b1b0 0x7f457014e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:25.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.621+0000 7f4578454640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f457014d690 0x7f457014df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:25.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.621+0000 7f4578454640 1 --2- 172.21.15.46:0/594203740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455c004d10 0x7f45701efe80 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:25.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.621+0000 7f4578454640 1 -- 172.21.15.46:0/594203740 >> 172.21.15.46:0/594203740 conn(0x7f457006bfe0 msgr2=0x7f45701510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:25.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.621+0000 7f4578454640 1 -- 172.21.15.46:0/594203740 shutdown_connections 2023-10-30T14:27:25.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:25.621+0000 7f4578454640 1 -- 172.21.15.46:0/594203740 wait complete. 2023-10-30T14:27:25.645 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":2032,"stamp":"2023-10-30T14:27:24.144436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1285691727,"num_objects":3948,"num_object_clones":2,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":20157,"num_read_kb":14487318,"num_write":119403,"num_write_kb":3806032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13139950,"num_omap_keys":23453,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":174279,"ondisk_log_size":174279,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3957956,"kb_used_data":2522372,"kb_used_omap":12,"kb_used_meta":1435507,"kb_avail":751016764,"statfs":{"total":773094113280,"available":769041166336,"internally_reserved":0,"allocated":2582908928,"data_stored":2577659229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1469959736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004660"},"pg_stats":[{"pgid":"3.3f","version":"110'2193","reported_seq":5016,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:17.161673+0000","last_change":"2023-10-30T14:27:17.161738+0000","last_active":"2023-10-30T14:27:17.161673+0000","last_peered":"2023-10-30T14:27:17.161673+0000","last_clean":"2023-10-30T14:27:17.161673+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T14:27:17.161673+0000","last_undegraded":"2023-10-30T14:27:17.161673+0000","last_fullsized":"2023-10-30T14:27:17.161673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2193","last_scrub_stamp":"2023-10-30T14:27:17.161580+0000","last_deep_scrub":"110'2193","last_deep_scrub_stamp":"2023-10-30T14:27:17.161580+0000","last_clean_scrub_stamp":"2023-10-30T14:27:17.161580+0000","objects_scrubbed":32,"log_size":2193,"log_dups_size":0,"ondisk_log_size":2193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:00.032759+0000","scrub_duration":0.104386056,"objects_trimmed":0,"snaptrim_duration":0.84725695499999998,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":124,"num_read_kb":10662,"num_write":1276,"num_write_kb":15944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"110'2011","reported_seq":4900,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:06.287642+0000","last_change":"2023-10-30T14:27:06.287715+0000","last_active":"2023-10-30T14:27:06.287642+0000","last_peered":"2023-10-30T14:27:06.287642+0000","last_clean":"2023-10-30T14:27:06.287642+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T14:27:06.287642+0000","last_undegraded":"2023-10-30T14:27:06.287642+0000","last_fullsized":"2023-10-30T14:27:06.287642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2011","last_scrub_stamp":"2023-10-30T14:27:06.287549+0000","last_deep_scrub":"110'2011","last_deep_scrub_stamp":"2023-10-30T14:27:06.287549+0000","last_clean_scrub_stamp":"2023-10-30T14:27:06.287549+0000","objects_scrubbed":40,"log_size":2011,"log_dups_size":0,"ondisk_log_size":2011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:35:24.021901+0000","scrub_duration":0.052013427000000001,"objects_trimmed":0,"snaptrim_duration":0.406256491,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":133,"num_read_kb":21158,"num_write":1230,"num_write_kb":16620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"110'2124","reported_seq":4967,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:08.843187+0000","last_change":"2023-10-30T14:27:08.843240+0000","last_active":"2023-10-30T14:27:08.843187+0000","last_peered":"2023-10-30T14:27:08.843187+0000","last_clean":"2023-10-30T14:27:08.843187+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T14:27:08.843187+0000","last_undegraded":"2023-10-30T14:27:08.843187+0000","last_fullsized":"2023-10-30T14:27:08.843187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2124","last_scrub_stamp":"2023-10-30T14:27:08.843106+0000","last_deep_scrub":"110'2124","last_deep_scrub_stamp":"2023-10-30T14:27:08.843106+0000","last_clean_scrub_stamp":"2023-10-30T14:27:08.843106+0000","objects_scrubbed":29,"log_size":2124,"log_dups_size":0,"ondisk_log_size":2124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:20:20.621247+0000","scrub_duration":0.041951362999999998,"objects_trimmed":0,"snaptrim_duration":1.030559456,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":129,"num_read_kb":29825,"num_write":1270,"num_write_kb":28412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"110'2055","reported_seq":4905,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:07.083889+0000","last_change":"2023-10-30T14:27:07.083934+0000","last_active":"2023-10-30T14:27:07.083889+0000","last_peered":"2023-10-30T14:27:07.083889+0000","last_clean":"2023-10-30T14:27:07.083889+0000","last_became_active":"2023-10-30T13:24:00.704829+0000","last_became_peered":"2023-10-30T13:24:00.704829+0000","last_unstale":"2023-10-30T14:27:07.083889+0000","last_undegraded":"2023-10-30T14:27:07.083889+0000","last_fullsized":"2023-10-30T14:27:07.083889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2055","last_scrub_stamp":"2023-10-30T14:27:07.083807+0000","last_deep_scrub":"110'2055","last_deep_scrub_stamp":"2023-10-30T14:27:07.083807+0000","last_clean_scrub_stamp":"2023-10-30T14:27:07.083807+0000","objects_scrubbed":25,"log_size":2055,"log_dups_size":0,"ondisk_log_size":2055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:33.332360+0000","scrub_duration":0.034227107,"objects_trimmed":0,"snaptrim_duration":0.361435862,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":124,"num_read_kb":7939,"num_write":1184,"num_write_kb":24239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"109'2009","reported_seq":4868,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.903222+0000","last_change":"2023-10-30T14:26:34.648843+0000","last_active":"2023-10-30T14:26:39.903222+0000","last_peered":"2023-10-30T14:26:39.903222+0000","last_clean":"2023-10-30T14:26:39.903222+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T14:26:39.903222+0000","last_undegraded":"2023-10-30T14:26:39.903222+0000","last_fullsized":"2023-10-30T14:26:39.903222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2009,"log_dups_size":0,"ondisk_log_size":2009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.80853723300000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":41266,"num_write":1244,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"110'2251","reported_seq":5060,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:07.080029+0000","last_change":"2023-10-30T14:27:07.080296+0000","last_active":"2023-10-30T14:27:07.080029+0000","last_peered":"2023-10-30T14:27:07.080029+0000","last_clean":"2023-10-30T14:27:07.080029+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T14:27:07.080029+0000","last_undegraded":"2023-10-30T14:27:07.080029+0000","last_fullsized":"2023-10-30T14:27:07.080029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2251","last_scrub_stamp":"2023-10-30T14:27:07.079781+0000","last_deep_scrub":"110'2251","last_deep_scrub_stamp":"2023-10-30T14:27:07.079781+0000","last_clean_scrub_stamp":"2023-10-30T14:27:07.079781+0000","objects_scrubbed":42,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:47.980581+0000","scrub_duration":0.119106473,"objects_trimmed":0,"snaptrim_duration":0.94867293500000005,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":141,"num_read_kb":27429,"num_write":1290,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"110'2185","reported_seq":5044,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:12.196491+0000","last_change":"2023-10-30T14:27:12.196618+0000","last_active":"2023-10-30T14:27:12.196491+0000","last_peered":"2023-10-30T14:27:12.196491+0000","last_clean":"2023-10-30T14:27:12.196491+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T14:27:12.196491+0000","last_undegraded":"2023-10-30T14:27:12.196491+0000","last_fullsized":"2023-10-30T14:27:12.196491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2185","last_scrub_stamp":"2023-10-30T14:27:12.196356+0000","last_deep_scrub":"110'2185","last_deep_scrub_stamp":"2023-10-30T14:27:12.196356+0000","last_clean_scrub_stamp":"2023-10-30T14:27:12.196356+0000","objects_scrubbed":40,"log_size":2185,"log_dups_size":0,"ondisk_log_size":2185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:32.138507+0000","scrub_duration":0.11536558199999999,"objects_trimmed":0,"snaptrim_duration":0.71511574600000005,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":146,"num_read_kb":25110,"num_write":1316,"num_write_kb":20043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"110'2147","reported_seq":4969,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:20.681195+0000","last_change":"2023-10-30T14:27:20.681264+0000","last_active":"2023-10-30T14:27:20.681195+0000","last_peered":"2023-10-30T14:27:20.681195+0000","last_clean":"2023-10-30T14:27:20.681195+0000","last_became_active":"2023-10-30T13:24:00.989938+0000","last_became_peered":"2023-10-30T13:24:00.989938+0000","last_unstale":"2023-10-30T14:27:20.681195+0000","last_undegraded":"2023-10-30T14:27:20.681195+0000","last_fullsized":"2023-10-30T14:27:20.681195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2147","last_scrub_stamp":"2023-10-30T14:27:20.681102+0000","last_deep_scrub":"110'2147","last_deep_scrub_stamp":"2023-10-30T14:27:20.681102+0000","last_clean_scrub_stamp":"2023-10-30T14:27:20.681102+0000","objects_scrubbed":38,"log_size":2147,"log_dups_size":0,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:10:24.247496+0000","scrub_duration":0.050065129,"objects_trimmed":0,"snaptrim_duration":0.90238370000000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":124,"num_read_kb":25628,"num_write":1232,"num_write_kb":20540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"109'2137","reported_seq":4970,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.907230+0000","last_change":"2023-10-30T14:26:34.654560+0000","last_active":"2023-10-30T14:26:39.907230+0000","last_peered":"2023-10-30T14:26:39.907230+0000","last_clean":"2023-10-30T14:26:39.907230+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T14:26:39.907230+0000","last_undegraded":"2023-10-30T14:26:39.907230+0000","last_fullsized":"2023-10-30T14:26:39.907230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81280227199999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":24548,"num_write":1310,"num_write_kb":19821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"110'2110","reported_seq":4983,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:08.423266+0000","last_change":"2023-10-30T14:27:08.423413+0000","last_active":"2023-10-30T14:27:08.423266+0000","last_peered":"2023-10-30T14:27:08.423266+0000","last_clean":"2023-10-30T14:27:08.423266+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T14:27:08.423266+0000","last_undegraded":"2023-10-30T14:27:08.423266+0000","last_fullsized":"2023-10-30T14:27:08.423266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2110","last_scrub_stamp":"2023-10-30T14:27:08.423062+0000","last_deep_scrub":"110'2110","last_deep_scrub_stamp":"2023-10-30T14:27:08.423062+0000","last_clean_scrub_stamp":"2023-10-30T14:27:08.423062+0000","objects_scrubbed":37,"log_size":2110,"log_dups_size":0,"ondisk_log_size":2110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:05:31.451500+0000","scrub_duration":0.11051124900000001,"objects_trimmed":0,"snaptrim_duration":0.53668512800000001,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":148,"num_read_kb":64678,"num_write":1273,"num_write_kb":32495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"110'2046","reported_seq":4931,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:13.410232+0000","last_change":"2023-10-30T14:27:13.410351+0000","last_active":"2023-10-30T14:27:13.410232+0000","last_peered":"2023-10-30T14:27:13.410232+0000","last_clean":"2023-10-30T14:27:13.410232+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T14:27:13.410232+0000","last_undegraded":"2023-10-30T14:27:13.410232+0000","last_fullsized":"2023-10-30T14:27:13.410232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2046","last_scrub_stamp":"2023-10-30T14:27:13.410056+0000","last_deep_scrub":"110'2046","last_deep_scrub_stamp":"2023-10-30T14:27:13.410056+0000","last_clean_scrub_stamp":"2023-10-30T14:27:13.410056+0000","objects_scrubbed":38,"log_size":2046,"log_dups_size":0,"ondisk_log_size":2046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:58.219570+0000","scrub_duration":0.113389077,"objects_trimmed":0,"snaptrim_duration":0.471346551,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":139,"num_read_kb":25550,"num_write":1254,"num_write_kb":16915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"110'2118","reported_seq":4973,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:58.807440+0000","last_change":"2023-10-30T14:26:58.807556+0000","last_active":"2023-10-30T14:26:58.807440+0000","last_peered":"2023-10-30T14:26:58.807440+0000","last_clean":"2023-10-30T14:26:58.807440+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T14:26:58.807440+0000","last_undegraded":"2023-10-30T14:26:58.807440+0000","last_fullsized":"2023-10-30T14:26:58.807440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2118","last_scrub_stamp":"2023-10-30T14:26:58.807287+0000","last_deep_scrub":"110'2118","last_deep_scrub_stamp":"2023-10-30T14:26:58.807287+0000","last_clean_scrub_stamp":"2023-10-30T14:26:58.807287+0000","objects_scrubbed":35,"log_size":2118,"log_dups_size":0,"ondisk_log_size":2118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:18.713125+0000","scrub_duration":0.105301005,"objects_trimmed":0,"snaptrim_duration":0.54183594099999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":142,"num_read_kb":52725,"num_write":1240,"num_write_kb":42463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"110'2255","reported_seq":5074,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:16.707217+0000","last_change":"2023-10-30T14:27:16.707278+0000","last_active":"2023-10-30T14:27:16.707217+0000","last_peered":"2023-10-30T14:27:16.707217+0000","last_clean":"2023-10-30T14:27:16.707217+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T14:27:16.707217+0000","last_undegraded":"2023-10-30T14:27:16.707217+0000","last_fullsized":"2023-10-30T14:27:16.707217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2255","last_scrub_stamp":"2023-10-30T14:27:16.707128+0000","last_deep_scrub":"110'2255","last_deep_scrub_stamp":"2023-10-30T14:27:16.707128+0000","last_clean_scrub_stamp":"2023-10-30T14:27:16.707128+0000","objects_scrubbed":32,"log_size":2255,"log_dups_size":0,"ondisk_log_size":2255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:55.264805+0000","scrub_duration":0.077397046999999997,"objects_trimmed":0,"snaptrim_duration":1.038919041,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":46739,"num_write":1326,"num_write_kb":33562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"110'2198","reported_seq":5059,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:04.874664+0000","last_change":"2023-10-30T14:27:04.874740+0000","last_active":"2023-10-30T14:27:04.874664+0000","last_peered":"2023-10-30T14:27:04.874664+0000","last_clean":"2023-10-30T14:27:04.874664+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T14:27:04.874664+0000","last_undegraded":"2023-10-30T14:27:04.874664+0000","last_fullsized":"2023-10-30T14:27:04.874664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2198","last_scrub_stamp":"2023-10-30T14:27:04.874577+0000","last_deep_scrub":"110'2198","last_deep_scrub_stamp":"2023-10-30T14:27:04.874577+0000","last_clean_scrub_stamp":"2023-10-30T14:27:04.874577+0000","objects_scrubbed":47,"log_size":2198,"log_dups_size":0,"ondisk_log_size":2198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:18:28.985655+0000","scrub_duration":0.059905817,"objects_trimmed":0,"snaptrim_duration":0.97549857799999995,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":149,"num_read_kb":13007,"num_write":1312,"num_write_kb":16233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"110'2306","reported_seq":5141,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:10.340370+0000","last_change":"2023-10-30T14:27:10.340471+0000","last_active":"2023-10-30T14:27:10.340370+0000","last_peered":"2023-10-30T14:27:10.340370+0000","last_clean":"2023-10-30T14:27:10.340370+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T14:27:10.340370+0000","last_undegraded":"2023-10-30T14:27:10.340370+0000","last_fullsized":"2023-10-30T14:27:10.340370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2306","last_scrub_stamp":"2023-10-30T14:27:10.340294+0000","last_deep_scrub":"110'2306","last_deep_scrub_stamp":"2023-10-30T14:27:10.340294+0000","last_clean_scrub_stamp":"2023-10-30T14:27:10.340294+0000","objects_scrubbed":39,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:52:13.783347+0000","scrub_duration":0.101464054,"objects_trimmed":0,"snaptrim_duration":0.432871758,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":161,"num_read_kb":14756,"num_write":1377,"num_write_kb":22909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"110'2120","reported_seq":4990,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:12.050653+0000","last_change":"2023-10-30T14:27:12.050782+0000","last_active":"2023-10-30T14:27:12.050653+0000","last_peered":"2023-10-30T14:27:12.050653+0000","last_clean":"2023-10-30T14:27:12.050653+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T14:27:12.050653+0000","last_undegraded":"2023-10-30T14:27:12.050653+0000","last_fullsized":"2023-10-30T14:27:12.050653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2120","last_scrub_stamp":"2023-10-30T14:27:12.050452+0000","last_deep_scrub":"110'2120","last_deep_scrub_stamp":"2023-10-30T14:27:12.050452+0000","last_clean_scrub_stamp":"2023-10-30T14:27:12.050452+0000","objects_scrubbed":30,"log_size":2120,"log_dups_size":0,"ondisk_log_size":2120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:30:59.575797+0000","scrub_duration":0.088091438999999994,"objects_trimmed":0,"snaptrim_duration":0.83845289999999995,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":149,"num_read_kb":40211,"num_write":1268,"num_write_kb":23485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"110'2389","reported_seq":5214,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:13.135193+0000","last_change":"2023-10-30T14:27:13.135363+0000","last_active":"2023-10-30T14:27:13.135193+0000","last_peered":"2023-10-30T14:27:13.135193+0000","last_clean":"2023-10-30T14:27:13.135193+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T14:27:13.135193+0000","last_undegraded":"2023-10-30T14:27:13.135193+0000","last_fullsized":"2023-10-30T14:27:13.135193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2389","last_scrub_stamp":"2023-10-30T14:27:13.135025+0000","last_deep_scrub":"110'2389","last_deep_scrub_stamp":"2023-10-30T14:27:13.135025+0000","last_clean_scrub_stamp":"2023-10-30T14:27:13.135025+0000","objects_scrubbed":41,"log_size":2389,"log_dups_size":0,"ondisk_log_size":2389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:16.442511+0000","scrub_duration":0.126176548,"objects_trimmed":0,"snaptrim_duration":1.109948846,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":170,"num_read_kb":39932,"num_write":1446,"num_write_kb":30153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"110'2027","reported_seq":4894,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:59.058458+0000","last_change":"2023-10-30T14:26:59.058509+0000","last_active":"2023-10-30T14:26:59.058458+0000","last_peered":"2023-10-30T14:26:59.058458+0000","last_clean":"2023-10-30T14:26:59.058458+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T14:26:59.058458+0000","last_undegraded":"2023-10-30T14:26:59.058458+0000","last_fullsized":"2023-10-30T14:26:59.058458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2027","last_scrub_stamp":"2023-10-30T14:26:59.058377+0000","last_deep_scrub":"110'2027","last_deep_scrub_stamp":"2023-10-30T14:26:59.058377+0000","last_clean_scrub_stamp":"2023-10-30T14:26:59.058377+0000","objects_scrubbed":31,"log_size":2027,"log_dups_size":0,"ondisk_log_size":2027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:24:17.547389+0000","scrub_duration":0.077206206999999999,"objects_trimmed":0,"snaptrim_duration":0.34246037800000001,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":137,"num_read_kb":13706,"num_write":1196,"num_write_kb":12790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"109'2095","reported_seq":4930,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.905220+0000","last_change":"2023-10-30T14:26:34.604839+0000","last_active":"2023-10-30T14:26:39.905220+0000","last_peered":"2023-10-30T14:26:39.905220+0000","last_clean":"2023-10-30T14:26:39.905220+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T14:26:39.905220+0000","last_undegraded":"2023-10-30T14:26:39.905220+0000","last_fullsized":"2023-10-30T14:26:39.905220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2095,"log_dups_size":0,"ondisk_log_size":2095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.76369308499999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":125,"num_read_kb":30178,"num_write":1282,"num_write_kb":24413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"110'2253","reported_seq":5094,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.434812+0000","last_change":"2023-10-30T14:26:57.434908+0000","last_active":"2023-10-30T14:26:57.434812+0000","last_peered":"2023-10-30T14:26:57.434812+0000","last_clean":"2023-10-30T14:26:57.434812+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T14:26:57.434812+0000","last_undegraded":"2023-10-30T14:26:57.434812+0000","last_fullsized":"2023-10-30T14:26:57.434812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2253","last_scrub_stamp":"2023-10-30T14:26:57.434694+0000","last_deep_scrub":"110'2253","last_deep_scrub_stamp":"2023-10-30T14:26:57.434694+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.434694+0000","objects_scrubbed":38,"log_size":2253,"log_dups_size":0,"ondisk_log_size":2253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:26:38.794241+0000","scrub_duration":0.122965276,"objects_trimmed":0,"snaptrim_duration":0.46668427699999998,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":148,"num_read_kb":30111,"num_write":1332,"num_write_kb":26597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"110'2256","reported_seq":5056,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:11.917935+0000","last_change":"2023-10-30T14:27:11.918008+0000","last_active":"2023-10-30T14:27:11.917935+0000","last_peered":"2023-10-30T14:27:11.917935+0000","last_clean":"2023-10-30T14:27:11.917935+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T14:27:11.917935+0000","last_undegraded":"2023-10-30T14:27:11.917935+0000","last_fullsized":"2023-10-30T14:27:11.917935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2256","last_scrub_stamp":"2023-10-30T14:27:11.917846+0000","last_deep_scrub":"110'2256","last_deep_scrub_stamp":"2023-10-30T14:27:11.917846+0000","last_clean_scrub_stamp":"2023-10-30T14:27:11.917846+0000","objects_scrubbed":34,"log_size":2256,"log_dups_size":0,"ondisk_log_size":2256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:28.375501+0000","scrub_duration":0.083664374,"objects_trimmed":0,"snaptrim_duration":1.008362585,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":127,"num_read_kb":10449,"num_write":1298,"num_write_kb":11653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"109'2282","reported_seq":5085,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:40.754280+0000","last_change":"2023-10-30T14:26:33.893164+0000","last_active":"2023-10-30T14:26:40.754280+0000","last_peered":"2023-10-30T14:26:40.754280+0000","last_clean":"2023-10-30T14:26:40.754280+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T14:26:40.754280+0000","last_undegraded":"2023-10-30T14:26:40.754280+0000","last_fullsized":"2023-10-30T14:26:40.754280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2282,"log_dups_size":0,"ondisk_log_size":2282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.94381626399999996,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":144,"num_read_kb":30506,"num_write":1376,"num_write_kb":26822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"109'2117","reported_seq":5082,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.893234+0000","last_change":"2023-10-30T14:26:36.452306+0000","last_active":"2023-10-30T14:26:39.893234+0000","last_peered":"2023-10-30T14:26:39.893234+0000","last_clean":"2023-10-30T14:26:39.893234+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T14:26:39.893234+0000","last_undegraded":"2023-10-30T14:26:39.893234+0000","last_fullsized":"2023-10-30T14:26:39.893234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2117,"log_dups_size":0,"ondisk_log_size":2117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98440143999999996,"stat_sum":{"num_bytes":6,"num_objects":38,"num_object_clones":2,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":257,"num_read_kb":42146,"num_write":1297,"num_write_kb":25266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"110'2212","reported_seq":5085,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:05.052361+0000","last_change":"2023-10-30T14:27:05.052501+0000","last_active":"2023-10-30T14:27:05.052361+0000","last_peered":"2023-10-30T14:27:05.052361+0000","last_clean":"2023-10-30T14:27:05.052361+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T14:27:05.052361+0000","last_undegraded":"2023-10-30T14:27:05.052361+0000","last_fullsized":"2023-10-30T14:27:05.052361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2212","last_scrub_stamp":"2023-10-30T14:27:05.052163+0000","last_deep_scrub":"110'2212","last_deep_scrub_stamp":"2023-10-30T14:27:05.052163+0000","last_clean_scrub_stamp":"2023-10-30T14:27:05.052163+0000","objects_scrubbed":34,"log_size":2212,"log_dups_size":0,"ondisk_log_size":2212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:40.396490+0000","scrub_duration":0.091462254000000007,"objects_trimmed":0,"snaptrim_duration":0.89723890399999995,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":164,"num_read_kb":14461,"num_write":1342,"num_write_kb":12663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"110'2090","reported_seq":4941,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:07.710584+0000","last_change":"2023-10-30T14:27:07.710637+0000","last_active":"2023-10-30T14:27:07.710584+0000","last_peered":"2023-10-30T14:27:07.710584+0000","last_clean":"2023-10-30T14:27:07.710584+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T14:27:07.710584+0000","last_undegraded":"2023-10-30T14:27:07.710584+0000","last_fullsized":"2023-10-30T14:27:07.710584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2090","last_scrub_stamp":"2023-10-30T14:27:07.710502+0000","last_deep_scrub":"110'2090","last_deep_scrub_stamp":"2023-10-30T14:27:07.710502+0000","last_clean_scrub_stamp":"2023-10-30T14:27:07.710502+0000","objects_scrubbed":33,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:39.764424+0000","scrub_duration":0.081297077999999995,"objects_trimmed":0,"snaptrim_duration":1.0495642949999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":133,"num_read_kb":25337,"num_write":1220,"num_write_kb":15637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"110'2181","reported_seq":5069,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:03.420662+0000","last_change":"2023-10-30T14:27:03.420832+0000","last_active":"2023-10-30T14:27:03.420662+0000","last_peered":"2023-10-30T14:27:03.420662+0000","last_clean":"2023-10-30T14:27:03.420662+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T14:27:03.420662+0000","last_undegraded":"2023-10-30T14:27:03.420662+0000","last_fullsized":"2023-10-30T14:27:03.420662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2181","last_scrub_stamp":"2023-10-30T14:27:03.420503+0000","last_deep_scrub":"110'2181","last_deep_scrub_stamp":"2023-10-30T14:27:03.420503+0000","last_clean_scrub_stamp":"2023-10-30T14:27:03.420503+0000","objects_scrubbed":40,"log_size":2181,"log_dups_size":0,"ondisk_log_size":2181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:36:16.083580+0000","scrub_duration":0.11761008000000001,"objects_trimmed":0,"snaptrim_duration":0.504458776,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":160,"num_read_kb":14034,"num_write":1340,"num_write_kb":12359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"110'2127","reported_seq":4987,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:03.326222+0000","last_change":"2023-10-30T14:27:03.326282+0000","last_active":"2023-10-30T14:27:03.326222+0000","last_peered":"2023-10-30T14:27:03.326222+0000","last_clean":"2023-10-30T14:27:03.326222+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T14:27:03.326222+0000","last_undegraded":"2023-10-30T14:27:03.326222+0000","last_fullsized":"2023-10-30T14:27:03.326222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2127","last_scrub_stamp":"2023-10-30T14:27:03.326140+0000","last_deep_scrub":"110'2127","last_deep_scrub_stamp":"2023-10-30T14:27:03.326140+0000","last_clean_scrub_stamp":"2023-10-30T14:27:03.326140+0000","objects_scrubbed":38,"log_size":2127,"log_dups_size":0,"ondisk_log_size":2127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:46:52.407641+0000","scrub_duration":0.050416410000000002,"objects_trimmed":0,"snaptrim_duration":0.34262146900000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":128,"num_read_kb":22518,"num_write":1277,"num_write_kb":32107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"110'2068","reported_seq":4908,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.691221+0000","last_change":"2023-10-30T14:26:52.691273+0000","last_active":"2023-10-30T14:26:52.691221+0000","last_peered":"2023-10-30T14:26:52.691221+0000","last_clean":"2023-10-30T14:26:52.691221+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T14:26:52.691221+0000","last_undegraded":"2023-10-30T14:26:52.691221+0000","last_fullsized":"2023-10-30T14:26:52.691221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2068","last_scrub_stamp":"2023-10-30T14:26:52.691147+0000","last_deep_scrub":"110'2068","last_deep_scrub_stamp":"2023-10-30T14:26:52.691147+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.691147+0000","objects_scrubbed":38,"log_size":2068,"log_dups_size":0,"ondisk_log_size":2068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:08.320506+0000","scrub_duration":0.079048169000000001,"objects_trimmed":0,"snaptrim_duration":1.068284585,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":129,"num_read_kb":17081,"num_write":1202,"num_write_kb":15468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"110'2178","reported_seq":5011,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:16.911734+0000","last_change":"2023-10-30T14:27:16.911796+0000","last_active":"2023-10-30T14:27:16.911734+0000","last_peered":"2023-10-30T14:27:16.911734+0000","last_clean":"2023-10-30T14:27:16.911734+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T14:27:16.911734+0000","last_undegraded":"2023-10-30T14:27:16.911734+0000","last_fullsized":"2023-10-30T14:27:16.911734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2178","last_scrub_stamp":"2023-10-30T14:27:16.911656+0000","last_deep_scrub":"110'2178","last_deep_scrub_stamp":"2023-10-30T14:27:16.911656+0000","last_clean_scrub_stamp":"2023-10-30T14:27:16.911656+0000","objects_scrubbed":38,"log_size":2178,"log_dups_size":0,"ondisk_log_size":2178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:38:53.941111+0000","scrub_duration":0.046150844000000003,"objects_trimmed":0,"snaptrim_duration":1.030536272,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":125,"num_read_kb":13655,"num_write":1323,"num_write_kb":17357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"110'2124","reported_seq":5004,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.119325+0000","last_change":"2023-10-30T14:26:57.119416+0000","last_active":"2023-10-30T14:26:57.119325+0000","last_peered":"2023-10-30T14:26:57.119325+0000","last_clean":"2023-10-30T14:26:57.119325+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T14:26:57.119325+0000","last_undegraded":"2023-10-30T14:26:57.119325+0000","last_fullsized":"2023-10-30T14:26:57.119325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2124","last_scrub_stamp":"2023-10-30T14:26:57.119213+0000","last_deep_scrub":"110'2124","last_deep_scrub_stamp":"2023-10-30T14:26:57.119213+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.119213+0000","objects_scrubbed":38,"log_size":2124,"log_dups_size":0,"ondisk_log_size":2124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:10.120593+0000","scrub_duration":0.076576980000000003,"objects_trimmed":0,"snaptrim_duration":0.98488935700000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":140,"num_read_kb":17880,"num_write":1311,"num_write_kb":13902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"110'2418","reported_seq":5244,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.085217+0000","last_change":"2023-10-30T14:27:00.085296+0000","last_active":"2023-10-30T14:27:00.085217+0000","last_peered":"2023-10-30T14:27:00.085217+0000","last_clean":"2023-10-30T14:27:00.085217+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T14:27:00.085217+0000","last_undegraded":"2023-10-30T14:27:00.085217+0000","last_fullsized":"2023-10-30T14:27:00.085217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2418","last_scrub_stamp":"2023-10-30T14:27:00.085138+0000","last_deep_scrub":"110'2418","last_deep_scrub_stamp":"2023-10-30T14:27:00.085138+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.085138+0000","objects_scrubbed":41,"log_size":2418,"log_dups_size":0,"ondisk_log_size":2418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:00.491199+0000","scrub_duration":0.10095144,"objects_trimmed":0,"snaptrim_duration":0.30670995400000001,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":165,"num_read_kb":40193,"num_write":1443,"num_write_kb":35290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"110'2344","reported_seq":5176,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:02.425665+0000","last_change":"2023-10-30T14:27:02.425824+0000","last_active":"2023-10-30T14:27:02.425665+0000","last_peered":"2023-10-30T14:27:02.425665+0000","last_clean":"2023-10-30T14:27:02.425665+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T14:27:02.425665+0000","last_undegraded":"2023-10-30T14:27:02.425665+0000","last_fullsized":"2023-10-30T14:27:02.425665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2344","last_scrub_stamp":"2023-10-30T14:27:02.425448+0000","last_deep_scrub":"110'2344","last_deep_scrub_stamp":"2023-10-30T14:27:02.425448+0000","last_clean_scrub_stamp":"2023-10-30T14:27:02.425448+0000","objects_scrubbed":41,"log_size":2344,"log_dups_size":0,"ondisk_log_size":2344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:18.221218+0000","scrub_duration":0.11536735100000001,"objects_trimmed":0,"snaptrim_duration":0.42993166100000002,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":168,"num_read_kb":24860,"num_write":1383,"num_write_kb":20782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"110'2147","reported_seq":5000,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:50.868519+0000","last_change":"2023-10-30T14:26:50.868736+0000","last_active":"2023-10-30T14:26:50.868519+0000","last_peered":"2023-10-30T14:26:50.868519+0000","last_clean":"2023-10-30T14:26:50.868519+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T14:26:50.868519+0000","last_undegraded":"2023-10-30T14:26:50.868519+0000","last_fullsized":"2023-10-30T14:26:50.868519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2147","last_scrub_stamp":"2023-10-30T14:26:50.868292+0000","last_deep_scrub":"110'2147","last_deep_scrub_stamp":"2023-10-30T14:26:50.868292+0000","last_clean_scrub_stamp":"2023-10-30T14:26:50.868292+0000","objects_scrubbed":34,"log_size":2147,"log_dups_size":0,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:21.454895+0000","scrub_duration":0.095674789999999996,"objects_trimmed":0,"snaptrim_duration":0.59246732999999996,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":146,"num_read_kb":30802,"num_write":1268,"num_write_kb":21419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"110'1078","reported_seq":1396,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.791196+0000","last_change":"2023-10-30T14:26:42.791248+0000","last_active":"2023-10-30T14:26:42.791196+0000","last_peered":"2023-10-30T14:26:42.791196+0000","last_clean":"2023-10-30T14:26:42.791196+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T14:26:42.791196+0000","last_undegraded":"2023-10-30T14:26:42.791196+0000","last_fullsized":"2023-10-30T14:26:42.791196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1078","last_scrub_stamp":"2023-10-30T14:26:42.791118+0000","last_deep_scrub":"110'1078","last_deep_scrub_stamp":"2023-10-30T14:26:42.791118+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.791118+0000","objects_scrubbed":50,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:33.250671+0000","scrub_duration":0.285611593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497506,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":222,"num_read_kb":501605,"num_write":1094,"num_write_kb":95785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":358881,"num_omap_keys":701,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"110'2190","reported_seq":5007,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:10.871699+0000","last_change":"2023-10-30T14:27:10.871788+0000","last_active":"2023-10-30T14:27:10.871699+0000","last_peered":"2023-10-30T14:27:10.871699+0000","last_clean":"2023-10-30T14:27:10.871699+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T14:27:10.871699+0000","last_undegraded":"2023-10-30T14:27:10.871699+0000","last_fullsized":"2023-10-30T14:27:10.871699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2190","last_scrub_stamp":"2023-10-30T14:27:10.871590+0000","last_deep_scrub":"110'2190","last_deep_scrub_stamp":"2023-10-30T14:27:10.871590+0000","last_clean_scrub_stamp":"2023-10-30T14:27:10.871590+0000","objects_scrubbed":35,"log_size":2190,"log_dups_size":0,"ondisk_log_size":2190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:54:03.123430+0000","scrub_duration":0.083573112000000005,"objects_trimmed":0,"snaptrim_duration":1.1434485830000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":17524,"num_write":1294,"num_write_kb":15816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"110'1745","reported_seq":2190,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.486755+0000","last_change":"2023-10-30T14:26:54.486806+0000","last_active":"2023-10-30T14:26:54.486755+0000","last_peered":"2023-10-30T14:26:54.486755+0000","last_clean":"2023-10-30T14:26:54.486755+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T14:26:54.486755+0000","last_undegraded":"2023-10-30T14:26:54.486755+0000","last_fullsized":"2023-10-30T14:26:54.486755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1745","last_scrub_stamp":"2023-10-30T14:26:54.486663+0000","last_deep_scrub":"110'1745","last_deep_scrub_stamp":"2023-10-30T14:26:54.486663+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.486663+0000","objects_scrubbed":52,"log_size":1745,"log_dups_size":0,"ondisk_log_size":1745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:54.960607+0000","scrub_duration":0.19504682200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29694842,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":392,"num_read_kb":410068,"num_write":1792,"num_write_kb":89526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":545048,"num_omap_keys":1059,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"110'1609","reported_seq":2975,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.816492+0000","last_change":"2023-10-30T14:26:46.816578+0000","last_active":"2023-10-30T14:26:46.816492+0000","last_peered":"2023-10-30T14:26:46.816492+0000","last_clean":"2023-10-30T14:26:46.816492+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T14:26:46.816492+0000","last_undegraded":"2023-10-30T14:26:46.816492+0000","last_fullsized":"2023-10-30T14:26:46.816492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1609","last_scrub_stamp":"2023-10-30T14:26:46.816372+0000","last_deep_scrub":"110'1609","last_deep_scrub_stamp":"2023-10-30T14:26:46.816372+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.816372+0000","objects_scrubbed":52,"log_size":1609,"log_dups_size":0,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:04:35.564593+0000","scrub_duration":0.28844654400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60855928,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":1296,"num_read_kb":556671,"num_write":1596,"num_write_kb":94675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":405715,"num_omap_keys":625,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"110'2379","reported_seq":5227,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.714048+0000","last_change":"2023-10-30T14:26:53.714120+0000","last_active":"2023-10-30T14:26:53.714048+0000","last_peered":"2023-10-30T14:26:53.714048+0000","last_clean":"2023-10-30T14:26:53.714048+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T14:26:53.714048+0000","last_undegraded":"2023-10-30T14:26:53.714048+0000","last_fullsized":"2023-10-30T14:26:53.714048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2379","last_scrub_stamp":"2023-10-30T14:26:53.713939+0000","last_deep_scrub":"110'2379","last_deep_scrub_stamp":"2023-10-30T14:26:53.713939+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.713939+0000","objects_scrubbed":52,"log_size":2379,"log_dups_size":0,"ondisk_log_size":2379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:15.405210+0000","scrub_duration":0.058831420000000002,"objects_trimmed":0,"snaptrim_duration":0.89032229399999996,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":161,"num_read_kb":17698,"num_write":1449,"num_write_kb":26122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"110'1032","reported_seq":1390,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:55.223361+0000","last_change":"2023-10-30T14:26:55.223448+0000","last_active":"2023-10-30T14:26:55.223361+0000","last_peered":"2023-10-30T14:26:55.223361+0000","last_clean":"2023-10-30T14:26:55.223361+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T14:26:55.223361+0000","last_undegraded":"2023-10-30T14:26:55.223361+0000","last_fullsized":"2023-10-30T14:26:55.223361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1032","last_scrub_stamp":"2023-10-30T14:26:55.223231+0000","last_deep_scrub":"110'1032","last_deep_scrub_stamp":"2023-10-30T14:26:55.223231+0000","last_clean_scrub_stamp":"2023-10-30T14:26:55.223231+0000","objects_scrubbed":57,"log_size":1032,"log_dups_size":0,"ondisk_log_size":1032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:18:36.091837+0000","scrub_duration":0.26442538300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54010089,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":312,"num_read_kb":563254,"num_write":1082,"num_write_kb":96418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":706737,"num_omap_keys":1228,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"110'2179","reported_seq":5047,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.255870+0000","last_change":"2023-10-30T14:27:00.255921+0000","last_active":"2023-10-30T14:27:00.255870+0000","last_peered":"2023-10-30T14:27:00.255870+0000","last_clean":"2023-10-30T14:27:00.255870+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T14:27:00.255870+0000","last_undegraded":"2023-10-30T14:27:00.255870+0000","last_fullsized":"2023-10-30T14:27:00.255870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2179","last_scrub_stamp":"2023-10-30T14:27:00.255786+0000","last_deep_scrub":"110'2179","last_deep_scrub_stamp":"2023-10-30T14:27:00.255786+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.255786+0000","objects_scrubbed":30,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:01.714268+0000","scrub_duration":0.037354261999999999,"objects_trimmed":0,"snaptrim_duration":0.36920391600000002,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":152,"num_read_kb":51949,"num_write":1321,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"110'865","reported_seq":1210,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:56.576908+0000","last_change":"2023-10-30T14:26:56.577007+0000","last_active":"2023-10-30T14:26:56.576908+0000","last_peered":"2023-10-30T14:26:56.576908+0000","last_clean":"2023-10-30T14:26:56.576908+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T14:26:56.576908+0000","last_undegraded":"2023-10-30T14:26:56.576908+0000","last_fullsized":"2023-10-30T14:26:56.576908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'865","last_scrub_stamp":"2023-10-30T14:26:56.576765+0000","last_deep_scrub":"110'865","last_deep_scrub_stamp":"2023-10-30T14:26:56.576765+0000","last_clean_scrub_stamp":"2023-10-30T14:26:56.576765+0000","objects_scrubbed":64,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:40.939877+0000","scrub_duration":0.28082476899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943130,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":319,"num_read_kb":417394,"num_write":917,"num_write_kb":71614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":395317,"num_omap_keys":751,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"110'2477","reported_seq":5216,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:58.107565+0000","last_change":"2023-10-30T14:26:58.107765+0000","last_active":"2023-10-30T14:26:58.107565+0000","last_peered":"2023-10-30T14:26:58.107565+0000","last_clean":"2023-10-30T14:26:58.107565+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T14:26:58.107565+0000","last_undegraded":"2023-10-30T14:26:58.107565+0000","last_fullsized":"2023-10-30T14:26:58.107565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2477","last_scrub_stamp":"2023-10-30T14:26:58.107390+0000","last_deep_scrub":"110'2477","last_deep_scrub_stamp":"2023-10-30T14:26:58.107390+0000","last_clean_scrub_stamp":"2023-10-30T14:26:58.107390+0000","objects_scrubbed":23,"log_size":2477,"log_dups_size":0,"ondisk_log_size":2477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:29.902228+0000","scrub_duration":0.078694307000000005,"objects_trimmed":0,"snaptrim_duration":1.0302270280000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":152,"num_read_kb":10055,"num_write":1351,"num_write_kb":13264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"110'901","reported_seq":1217,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:51.209972+0000","last_change":"2023-10-30T14:26:51.210091+0000","last_active":"2023-10-30T14:26:51.209972+0000","last_peered":"2023-10-30T14:26:51.209972+0000","last_clean":"2023-10-30T14:26:51.209972+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T14:26:51.209972+0000","last_undegraded":"2023-10-30T14:26:51.209972+0000","last_fullsized":"2023-10-30T14:26:51.209972+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'901","last_scrub_stamp":"2023-10-30T14:26:51.209745+0000","last_deep_scrub":"110'901","last_deep_scrub_stamp":"2023-10-30T14:26:51.209745+0000","last_clean_scrub_stamp":"2023-10-30T14:26:51.209745+0000","objects_scrubbed":52,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:29:33.421380+0000","scrub_duration":0.228163176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":291,"num_read_kb":402257,"num_write":947,"num_write_kb":71494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":417597,"num_omap_keys":787,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"110'2117","reported_seq":4971,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:01.839258+0000","last_change":"2023-10-30T14:27:01.839312+0000","last_active":"2023-10-30T14:27:01.839258+0000","last_peered":"2023-10-30T14:27:01.839258+0000","last_clean":"2023-10-30T14:27:01.839258+0000","last_became_active":"2023-10-30T13:24:00.990019+0000","last_became_peered":"2023-10-30T13:24:00.990019+0000","last_unstale":"2023-10-30T14:27:01.839258+0000","last_undegraded":"2023-10-30T14:27:01.839258+0000","last_fullsized":"2023-10-30T14:27:01.839258+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2117","last_scrub_stamp":"2023-10-30T14:27:01.839175+0000","last_deep_scrub":"110'2117","last_deep_scrub_stamp":"2023-10-30T14:27:01.839175+0000","last_clean_scrub_stamp":"2023-10-30T14:27:01.839175+0000","objects_scrubbed":32,"log_size":2117,"log_dups_size":0,"ondisk_log_size":2117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:37:53.712537+0000","scrub_duration":0.043087937999999999,"objects_trimmed":0,"snaptrim_duration":1.1322467139999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":132,"num_read_kb":30385,"num_write":1263,"num_write_kb":18224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"110'1029","reported_seq":1299,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:55.421984+0000","last_change":"2023-10-30T14:26:55.422041+0000","last_active":"2023-10-30T14:26:55.421984+0000","last_peered":"2023-10-30T14:26:55.421984+0000","last_clean":"2023-10-30T14:26:55.421984+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T14:26:55.421984+0000","last_undegraded":"2023-10-30T14:26:55.421984+0000","last_fullsized":"2023-10-30T14:26:55.421984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1029","last_scrub_stamp":"2023-10-30T14:26:55.421890+0000","last_deep_scrub":"110'1029","last_deep_scrub_stamp":"2023-10-30T14:26:55.421890+0000","last_clean_scrub_stamp":"2023-10-30T14:26:55.421890+0000","objects_scrubbed":35,"log_size":1029,"log_dups_size":0,"ondisk_log_size":1029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:34:36.976816+0000","scrub_duration":0.131590227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":208,"num_read_kb":224450,"num_write":1057,"num_write_kb":46972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":295749,"num_omap_keys":457,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"110'2177","reported_seq":5017,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:48.799169+0000","last_change":"2023-10-30T14:26:48.799265+0000","last_active":"2023-10-30T14:26:48.799169+0000","last_peered":"2023-10-30T14:26:48.799169+0000","last_clean":"2023-10-30T14:26:48.799169+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T14:26:48.799169+0000","last_undegraded":"2023-10-30T14:26:48.799169+0000","last_fullsized":"2023-10-30T14:26:48.799169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2177","last_scrub_stamp":"2023-10-30T14:26:48.799002+0000","last_deep_scrub":"110'2177","last_deep_scrub_stamp":"2023-10-30T14:26:48.799002+0000","last_clean_scrub_stamp":"2023-10-30T14:26:48.799002+0000","objects_scrubbed":28,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:56:02.403730+0000","scrub_duration":0.092776059999999994,"objects_trimmed":0,"snaptrim_duration":0.58089010799999996,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":141,"num_read_kb":34327,"num_write":1294,"num_write_kb":27493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"110'1040","reported_seq":1364,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:47.590330+0000","last_change":"2023-10-30T14:26:47.590330+0000","last_active":"2023-10-30T14:26:47.590330+0000","last_peered":"2023-10-30T14:26:47.590330+0000","last_clean":"2023-10-30T14:26:47.590330+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T14:26:47.590330+0000","last_undegraded":"2023-10-30T14:26:47.590330+0000","last_fullsized":"2023-10-30T14:26:47.590330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1039","last_scrub_stamp":"2023-10-30T14:26:47.589869+0000","last_deep_scrub":"110'1039","last_deep_scrub_stamp":"2023-10-30T14:26:47.589869+0000","last_clean_scrub_stamp":"2023-10-30T14:26:47.589869+0000","objects_scrubbed":46,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:04:37.337840+0000","scrub_duration":0.243642899,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":290,"num_read_kb":355466,"num_write":1084,"num_write_kb":67491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":426651,"num_omap_keys":733,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"109'2165","reported_seq":4985,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.904020+0000","last_change":"2023-10-30T14:26:34.585946+0000","last_active":"2023-10-30T14:26:39.904020+0000","last_peered":"2023-10-30T14:26:39.904020+0000","last_clean":"2023-10-30T14:26:39.904020+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T14:26:39.904020+0000","last_undegraded":"2023-10-30T14:26:39.904020+0000","last_fullsized":"2023-10-30T14:26:39.904020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2165,"log_dups_size":0,"ondisk_log_size":2165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.74553484199999998,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":143,"num_read_kb":29369,"num_write":1268,"num_write_kb":21738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"110'758","reported_seq":1138,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.031518+0000","last_change":"2023-10-30T14:26:42.031593+0000","last_active":"2023-10-30T14:26:42.031518+0000","last_peered":"2023-10-30T14:26:42.031518+0000","last_clean":"2023-10-30T14:26:42.031518+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T14:26:42.031518+0000","last_undegraded":"2023-10-30T14:26:42.031518+0000","last_fullsized":"2023-10-30T14:26:42.031518+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'758","last_scrub_stamp":"2023-10-30T14:26:42.031416+0000","last_deep_scrub":"110'758","last_deep_scrub_stamp":"2023-10-30T14:26:42.031416+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.031416+0000","objects_scrubbed":57,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:32.795744+0000","scrub_duration":0.243287324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331682,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":392,"num_read_kb":480410,"num_write":807,"num_write_kb":87841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":384994,"num_omap_keys":754,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"110'2184","reported_seq":5052,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.780582+0000","last_change":"2023-10-30T14:26:52.780632+0000","last_active":"2023-10-30T14:26:52.780582+0000","last_peered":"2023-10-30T14:26:52.780582+0000","last_clean":"2023-10-30T14:26:52.780582+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T14:26:52.780582+0000","last_undegraded":"2023-10-30T14:26:52.780582+0000","last_fullsized":"2023-10-30T14:26:52.780582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2184","last_scrub_stamp":"2023-10-30T14:26:52.780505+0000","last_deep_scrub":"110'2184","last_deep_scrub_stamp":"2023-10-30T14:26:52.780505+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.780505+0000","objects_scrubbed":34,"log_size":2184,"log_dups_size":0,"ondisk_log_size":2184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:22.325362+0000","scrub_duration":0.086211091000000004,"objects_trimmed":0,"snaptrim_duration":0.95729191499999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":151,"num_read_kb":31378,"num_write":1306,"num_write_kb":33173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"110'899","reported_seq":1169,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.271896+0000","last_change":"2023-10-30T14:26:54.272052+0000","last_active":"2023-10-30T14:26:54.271896+0000","last_peered":"2023-10-30T14:26:54.271896+0000","last_clean":"2023-10-30T14:26:54.271896+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T14:26:54.271896+0000","last_undegraded":"2023-10-30T14:26:54.271896+0000","last_fullsized":"2023-10-30T14:26:54.271896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'899","last_scrub_stamp":"2023-10-30T14:26:54.271663+0000","last_deep_scrub":"110'899","last_deep_scrub_stamp":"2023-10-30T14:26:54.271663+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.271663+0000","objects_scrubbed":48,"log_size":899,"log_dups_size":0,"ondisk_log_size":899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:16.820077+0000","scrub_duration":0.294758195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53927368,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":208,"num_read_kb":306859,"num_write":914,"num_write_kb":62457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":467666,"num_omap_keys":812,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"110'2334","reported_seq":5135,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:09.227727+0000","last_change":"2023-10-30T14:27:09.227790+0000","last_active":"2023-10-30T14:27:09.227727+0000","last_peered":"2023-10-30T14:27:09.227727+0000","last_clean":"2023-10-30T14:27:09.227727+0000","last_became_active":"2023-10-30T13:24:00.706894+0000","last_became_peered":"2023-10-30T13:24:00.706894+0000","last_unstale":"2023-10-30T14:27:09.227727+0000","last_undegraded":"2023-10-30T14:27:09.227727+0000","last_fullsized":"2023-10-30T14:27:09.227727+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2334","last_scrub_stamp":"2023-10-30T14:27:09.227647+0000","last_deep_scrub":"110'2334","last_deep_scrub_stamp":"2023-10-30T14:27:09.227647+0000","last_clean_scrub_stamp":"2023-10-30T14:27:09.227647+0000","objects_scrubbed":41,"log_size":2334,"log_dups_size":0,"ondisk_log_size":2334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:45:55.621940+0000","scrub_duration":0.096202764999999996,"objects_trimmed":0,"snaptrim_duration":0.3738494,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":149,"num_read_kb":1539,"num_write":1375,"num_write_kb":12931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"110'780","reported_seq":1067,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.584097+0000","last_change":"2023-10-30T14:26:46.584221+0000","last_active":"2023-10-30T14:26:46.584097+0000","last_peered":"2023-10-30T14:26:46.584097+0000","last_clean":"2023-10-30T14:26:46.584097+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T14:26:46.584097+0000","last_undegraded":"2023-10-30T14:26:46.584097+0000","last_fullsized":"2023-10-30T14:26:46.584097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'780","last_scrub_stamp":"2023-10-30T14:26:46.583879+0000","last_deep_scrub":"110'780","last_deep_scrub_stamp":"2023-10-30T14:26:46.583879+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.583879+0000","objects_scrubbed":53,"log_size":780,"log_dups_size":0,"ondisk_log_size":780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:52.998170+0000","scrub_duration":0.20502225800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":234,"num_read_kb":298146,"num_write":829,"num_write_kb":54985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":494954,"num_omap_keys":946,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"110'2149","reported_seq":5030,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.527465+0000","last_change":"2023-10-30T14:27:00.527596+0000","last_active":"2023-10-30T14:27:00.527465+0000","last_peered":"2023-10-30T14:27:00.527465+0000","last_clean":"2023-10-30T14:27:00.527465+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T14:27:00.527465+0000","last_undegraded":"2023-10-30T14:27:00.527465+0000","last_fullsized":"2023-10-30T14:27:00.527465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2149","last_scrub_stamp":"2023-10-30T14:27:00.527256+0000","last_deep_scrub":"110'2149","last_deep_scrub_stamp":"2023-10-30T14:27:00.527256+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.527256+0000","objects_scrubbed":53,"log_size":2149,"log_dups_size":0,"ondisk_log_size":2149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:55:50.223566+0000","scrub_duration":0.14114542399999999,"objects_trimmed":0,"snaptrim_duration":0.50412874500000004,"stat_sum":{"num_bytes":0,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":148,"num_read_kb":43506,"num_write":1375,"num_write_kb":26417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"110'876","reported_seq":1232,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.183186+0000","last_change":"2023-10-30T14:26:52.183292+0000","last_active":"2023-10-30T14:26:52.183186+0000","last_peered":"2023-10-30T14:26:52.183186+0000","last_clean":"2023-10-30T14:26:52.183186+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T14:26:52.183186+0000","last_undegraded":"2023-10-30T14:26:52.183186+0000","last_fullsized":"2023-10-30T14:26:52.183186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'876","last_scrub_stamp":"2023-10-30T14:26:52.183105+0000","last_deep_scrub":"110'876","last_deep_scrub_stamp":"2023-10-30T14:26:52.183105+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.183105+0000","objects_scrubbed":51,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:18:36.571108+0000","scrub_duration":0.119037509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":481,"num_read_kb":229594,"num_write":1003,"num_write_kb":42855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":532426,"num_omap_keys":823,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"110'2380","reported_seq":5218,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.002361+0000","last_change":"2023-10-30T14:26:46.002570+0000","last_active":"2023-10-30T14:26:46.002361+0000","last_peered":"2023-10-30T14:26:46.002361+0000","last_clean":"2023-10-30T14:26:46.002361+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T14:26:46.002361+0000","last_undegraded":"2023-10-30T14:26:46.002361+0000","last_fullsized":"2023-10-30T14:26:46.002361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2380","last_scrub_stamp":"2023-10-30T14:26:46.002134+0000","last_deep_scrub":"110'2380","last_deep_scrub_stamp":"2023-10-30T14:26:46.002134+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.002134+0000","objects_scrubbed":46,"log_size":2380,"log_dups_size":0,"ondisk_log_size":2380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:38.920284+0000","scrub_duration":0.14576676,"objects_trimmed":0,"snaptrim_duration":0.70955538100000004,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":166,"num_read_kb":25446,"num_write":1448,"num_write_kb":32435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"110'1046","reported_seq":1341,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.977448+0000","last_change":"2023-10-30T14:26:42.977518+0000","last_active":"2023-10-30T14:26:42.977448+0000","last_peered":"2023-10-30T14:26:42.977448+0000","last_clean":"2023-10-30T14:26:42.977448+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T14:26:42.977448+0000","last_undegraded":"2023-10-30T14:26:42.977448+0000","last_fullsized":"2023-10-30T14:26:42.977448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1046","last_scrub_stamp":"2023-10-30T14:26:42.977329+0000","last_deep_scrub":"110'1046","last_deep_scrub_stamp":"2023-10-30T14:26:42.977329+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.977329+0000","objects_scrubbed":51,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:00.448502+0000","scrub_duration":0.210468818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":214,"num_read_kb":373366,"num_write":1150,"num_write_kb":71086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":337344,"num_omap_keys":660,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"109'2238","reported_seq":5028,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.906190+0000","last_change":"2023-10-30T14:26:34.736009+0000","last_active":"2023-10-30T14:26:39.906190+0000","last_peered":"2023-10-30T14:26:39.906190+0000","last_clean":"2023-10-30T14:26:39.906190+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T14:26:39.906190+0000","last_undegraded":"2023-10-30T14:26:39.906190+0000","last_fullsized":"2023-10-30T14:26:39.906190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2238,"log_dups_size":0,"ondisk_log_size":2238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89413299700000004,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":142,"num_read_kb":18968,"num_write":1278,"num_write_kb":24747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"110'668","reported_seq":989,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:47.033482+0000","last_change":"2023-10-30T14:26:47.033575+0000","last_active":"2023-10-30T14:26:47.033482+0000","last_peered":"2023-10-30T14:26:47.033482+0000","last_clean":"2023-10-30T14:26:47.033482+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T14:26:47.033482+0000","last_undegraded":"2023-10-30T14:26:47.033482+0000","last_fullsized":"2023-10-30T14:26:47.033482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'668","last_scrub_stamp":"2023-10-30T14:26:47.033389+0000","last_deep_scrub":"110'668","last_deep_scrub_stamp":"2023-10-30T14:26:47.033389+0000","last_clean_scrub_stamp":"2023-10-30T14:26:47.033389+0000","objects_scrubbed":42,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:25.680237+0000","scrub_duration":0.095992737999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":280,"num_read_kb":492842,"num_write":679,"num_write_kb":83547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":470248,"num_omap_keys":762,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"109'2096","reported_seq":4971,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.907716+0000","last_change":"2023-10-30T14:26:34.726845+0000","last_active":"2023-10-30T14:26:39.907716+0000","last_peered":"2023-10-30T14:26:39.907716+0000","last_clean":"2023-10-30T14:26:39.907716+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T14:26:39.907716+0000","last_undegraded":"2023-10-30T14:26:39.907716+0000","last_fullsized":"2023-10-30T14:26:39.907716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2096,"log_dups_size":0,"ondisk_log_size":2096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.88483857600000004,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":152,"num_read_kb":61046,"num_write":1310,"num_write_kb":36653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"110'790","reported_seq":1095,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.425710+0000","last_change":"2023-10-30T14:26:52.425764+0000","last_active":"2023-10-30T14:26:52.425710+0000","last_peered":"2023-10-30T14:26:52.425710+0000","last_clean":"2023-10-30T14:26:52.425710+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T14:26:52.425710+0000","last_undegraded":"2023-10-30T14:26:52.425710+0000","last_fullsized":"2023-10-30T14:26:52.425710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'790","last_scrub_stamp":"2023-10-30T14:26:52.425638+0000","last_deep_scrub":"110'790","last_deep_scrub_stamp":"2023-10-30T14:26:52.425638+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.425638+0000","objects_scrubbed":60,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:57:40.747257+0000","scrub_duration":0.18100369799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364699,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":233,"num_read_kb":370768,"num_write":815,"num_write_kb":71238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":549758,"num_omap_keys":876,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"110'2281","reported_seq":5152,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:14.156230+0000","last_change":"2023-10-30T14:27:14.156348+0000","last_active":"2023-10-30T14:27:14.156230+0000","last_peered":"2023-10-30T14:27:14.156230+0000","last_clean":"2023-10-30T14:27:14.156230+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T14:27:14.156230+0000","last_undegraded":"2023-10-30T14:27:14.156230+0000","last_fullsized":"2023-10-30T14:27:14.156230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2281","last_scrub_stamp":"2023-10-30T14:27:14.156044+0000","last_deep_scrub":"110'2281","last_deep_scrub_stamp":"2023-10-30T14:27:14.156044+0000","last_clean_scrub_stamp":"2023-10-30T14:27:14.156044+0000","objects_scrubbed":41,"log_size":2281,"log_dups_size":0,"ondisk_log_size":2281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:23:25.400045+0000","scrub_duration":0.105525299,"objects_trimmed":0,"snaptrim_duration":0.98441546899999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":178,"num_read_kb":29706,"num_write":1375,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"110'1121","reported_seq":1413,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.293606+0000","last_change":"2023-10-30T14:26:54.293672+0000","last_active":"2023-10-30T14:26:54.293606+0000","last_peered":"2023-10-30T14:26:54.293606+0000","last_clean":"2023-10-30T14:26:54.293606+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T14:26:54.293606+0000","last_undegraded":"2023-10-30T14:26:54.293606+0000","last_fullsized":"2023-10-30T14:26:54.293606+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1121","last_scrub_stamp":"2023-10-30T14:26:54.293526+0000","last_deep_scrub":"110'1121","last_deep_scrub_stamp":"2023-10-30T14:26:54.293526+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.293526+0000","objects_scrubbed":43,"log_size":1121,"log_dups_size":0,"ondisk_log_size":1121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:17.009305+0000","scrub_duration":0.241353279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50332230,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":210,"num_read_kb":406971,"num_write":1089,"num_write_kb":71553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":220904,"num_omap_keys":399,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"110'2078","reported_seq":4939,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:16.182932+0000","last_change":"2023-10-30T14:27:16.183032+0000","last_active":"2023-10-30T14:27:16.182932+0000","last_peered":"2023-10-30T14:27:16.182932+0000","last_clean":"2023-10-30T14:27:16.182932+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T14:27:16.182932+0000","last_undegraded":"2023-10-30T14:27:16.182932+0000","last_fullsized":"2023-10-30T14:27:16.182932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2078","last_scrub_stamp":"2023-10-30T14:27:16.182829+0000","last_deep_scrub":"110'2078","last_deep_scrub_stamp":"2023-10-30T14:27:16.182829+0000","last_clean_scrub_stamp":"2023-10-30T14:27:16.182829+0000","objects_scrubbed":38,"log_size":2078,"log_dups_size":0,"ondisk_log_size":2078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:14:39.775252+0000","scrub_duration":0.132528019,"objects_trimmed":0,"snaptrim_duration":0.84782311300000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":122,"num_read_kb":1096,"num_write":1268,"num_write_kb":17447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"110'119","reported_seq":297,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:47.898917+0000","last_change":"2023-10-30T14:26:47.899032+0000","last_active":"2023-10-30T14:26:47.898917+0000","last_peered":"2023-10-30T14:26:47.898917+0000","last_clean":"2023-10-30T14:26:47.898917+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T14:26:47.898917+0000","last_undegraded":"2023-10-30T14:26:47.898917+0000","last_fullsized":"2023-10-30T14:26:47.898917+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'119","last_scrub_stamp":"2023-10-30T14:26:47.898726+0000","last_deep_scrub":"110'119","last_deep_scrub_stamp":"2023-10-30T14:26:47.898726+0000","last_clean_scrub_stamp":"2023-10-30T14:26:47.898726+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:29.562342+0000","scrub_duration":0.0091457540000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"110'909","reported_seq":1258,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:50.195755+0000","last_change":"2023-10-30T14:26:50.195816+0000","last_active":"2023-10-30T14:26:50.195755+0000","last_peered":"2023-10-30T14:26:50.195755+0000","last_clean":"2023-10-30T14:26:50.195755+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T14:26:50.195755+0000","last_undegraded":"2023-10-30T14:26:50.195755+0000","last_fullsized":"2023-10-30T14:26:50.195755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'909","last_scrub_stamp":"2023-10-30T14:26:50.195649+0000","last_deep_scrub":"110'909","last_deep_scrub_stamp":"2023-10-30T14:26:50.195649+0000","last_clean_scrub_stamp":"2023-10-30T14:26:50.195649+0000","objects_scrubbed":60,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:12.567856+0000","scrub_duration":0.19057717900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331714,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":319,"num_read_kb":439640,"num_write":936,"num_write_kb":75464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":459843,"num_omap_keys":807,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"110'2244","reported_seq":5041,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.842968+0000","last_change":"2023-10-30T14:26:57.843044+0000","last_active":"2023-10-30T14:26:57.842968+0000","last_peered":"2023-10-30T14:26:57.842968+0000","last_clean":"2023-10-30T14:26:57.842968+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T14:26:57.842968+0000","last_undegraded":"2023-10-30T14:26:57.842968+0000","last_fullsized":"2023-10-30T14:26:57.842968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2244","last_scrub_stamp":"2023-10-30T14:26:57.842882+0000","last_deep_scrub":"110'2244","last_deep_scrub_stamp":"2023-10-30T14:26:57.842882+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.842882+0000","objects_scrubbed":46,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:34:34.719838+0000","scrub_duration":0.10386124300000001,"objects_trimmed":0,"snaptrim_duration":1.003812562,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":127,"num_read_kb":26261,"num_write":1294,"num_write_kb":24882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"110'1232","reported_seq":1555,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.159370+0000","last_change":"2023-10-30T14:26:53.159534+0000","last_active":"2023-10-30T14:26:53.159370+0000","last_peered":"2023-10-30T14:26:53.159370+0000","last_clean":"2023-10-30T14:26:53.159370+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T14:26:53.159370+0000","last_undegraded":"2023-10-30T14:26:53.159370+0000","last_fullsized":"2023-10-30T14:26:53.159370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1232","last_scrub_stamp":"2023-10-30T14:26:53.159107+0000","last_deep_scrub":"110'1232","last_deep_scrub_stamp":"2023-10-30T14:26:53.159107+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.159107+0000","objects_scrubbed":46,"log_size":1232,"log_dups_size":0,"ondisk_log_size":1232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:27.847424+0000","scrub_duration":0.24891354399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":256,"num_read_kb":488007,"num_write":1246,"num_write_kb":83351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":346758,"num_omap_keys":680,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"110'2013","reported_seq":4887,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:51.757304+0000","last_change":"2023-10-30T14:26:51.757396+0000","last_active":"2023-10-30T14:26:51.757304+0000","last_peered":"2023-10-30T14:26:51.757304+0000","last_clean":"2023-10-30T14:26:51.757304+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T14:26:51.757304+0000","last_undegraded":"2023-10-30T14:26:51.757304+0000","last_fullsized":"2023-10-30T14:26:51.757304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2013","last_scrub_stamp":"2023-10-30T14:26:51.757217+0000","last_deep_scrub":"110'2013","last_deep_scrub_stamp":"2023-10-30T14:26:51.757217+0000","last_clean_scrub_stamp":"2023-10-30T14:26:51.757217+0000","objects_scrubbed":34,"log_size":2013,"log_dups_size":0,"ondisk_log_size":2013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:49.616919+0000","scrub_duration":0.071006446000000001,"objects_trimmed":0,"snaptrim_duration":0.92159720499999997,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":129,"num_read_kb":13045,"num_write":1222,"num_write_kb":11637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"110'2000","reported_seq":3256,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:58.136825+0000","last_change":"2023-10-30T14:26:58.136871+0000","last_active":"2023-10-30T14:26:58.136825+0000","last_peered":"2023-10-30T14:26:58.136825+0000","last_clean":"2023-10-30T14:26:58.136825+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T14:26:58.136825+0000","last_undegraded":"2023-10-30T14:26:58.136825+0000","last_fullsized":"2023-10-30T14:26:58.136825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2000","last_scrub_stamp":"2023-10-30T14:26:58.136733+0000","last_deep_scrub":"110'2000","last_deep_scrub_stamp":"2023-10-30T14:26:58.136733+0000","last_clean_scrub_stamp":"2023-10-30T14:26:58.136733+0000","objects_scrubbed":51,"log_size":2000,"log_dups_size":0,"ondisk_log_size":2000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:54.997528+0000","scrub_duration":0.11551852899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40060913,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":1167,"num_read_kb":359975,"num_write":2020,"num_write_kb":69890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":329700,"num_omap_keys":585,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"110'2306","reported_seq":5129,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:56.314765+0000","last_change":"2023-10-30T14:26:56.314854+0000","last_active":"2023-10-30T14:26:56.314765+0000","last_peered":"2023-10-30T14:26:56.314765+0000","last_clean":"2023-10-30T14:26:56.314765+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T14:26:56.314765+0000","last_undegraded":"2023-10-30T14:26:56.314765+0000","last_fullsized":"2023-10-30T14:26:56.314765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2306","last_scrub_stamp":"2023-10-30T14:26:56.314673+0000","last_deep_scrub":"110'2306","last_deep_scrub_stamp":"2023-10-30T14:26:56.314673+0000","last_clean_scrub_stamp":"2023-10-30T14:26:56.314673+0000","objects_scrubbed":45,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:16:05.533537+0000","scrub_duration":0.059182301999999999,"objects_trimmed":0,"snaptrim_duration":0.40657480699999998,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":145,"num_read_kb":55105,"num_write":1354,"num_write_kb":28005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"110'1533","reported_seq":1849,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:45.029556+0000","last_change":"2023-10-30T14:26:45.029718+0000","last_active":"2023-10-30T14:26:45.029556+0000","last_peered":"2023-10-30T14:26:45.029556+0000","last_clean":"2023-10-30T14:26:45.029556+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T14:26:45.029556+0000","last_undegraded":"2023-10-30T14:26:45.029556+0000","last_fullsized":"2023-10-30T14:26:45.029556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1533","last_scrub_stamp":"2023-10-30T14:26:45.029381+0000","last_deep_scrub":"110'1533","last_deep_scrub_stamp":"2023-10-30T14:26:45.029381+0000","last_clean_scrub_stamp":"2023-10-30T14:26:45.029381+0000","objects_scrubbed":45,"log_size":1533,"log_dups_size":0,"ondisk_log_size":1533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:54.864240+0000","scrub_duration":0.21060546799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":235,"num_read_kb":509467,"num_write":1537,"num_write_kb":83943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":361192,"num_omap_keys":645,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"110'2242","reported_seq":5077,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.859618+0000","last_change":"2023-10-30T14:26:46.859846+0000","last_active":"2023-10-30T14:26:46.859618+0000","last_peered":"2023-10-30T14:26:46.859618+0000","last_clean":"2023-10-30T14:26:46.859618+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T14:26:46.859618+0000","last_undegraded":"2023-10-30T14:26:46.859618+0000","last_fullsized":"2023-10-30T14:26:46.859618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2242","last_scrub_stamp":"2023-10-30T14:26:46.859379+0000","last_deep_scrub":"110'2242","last_deep_scrub_stamp":"2023-10-30T14:26:46.859379+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.859379+0000","objects_scrubbed":34,"log_size":2242,"log_dups_size":0,"ondisk_log_size":2242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:58:59.480931+0000","scrub_duration":0.104056545,"objects_trimmed":0,"snaptrim_duration":0.59694472600000004,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":143,"num_read_kb":13239,"num_write":1306,"num_write_kb":17120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"110'663","reported_seq":985,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:50.346246+0000","last_change":"2023-10-30T14:26:50.346292+0000","last_active":"2023-10-30T14:26:50.346246+0000","last_peered":"2023-10-30T14:26:50.346246+0000","last_clean":"2023-10-30T14:26:50.346246+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T14:26:50.346246+0000","last_undegraded":"2023-10-30T14:26:50.346246+0000","last_fullsized":"2023-10-30T14:26:50.346246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'663","last_scrub_stamp":"2023-10-30T14:26:50.346173+0000","last_deep_scrub":"110'663","last_deep_scrub_stamp":"2023-10-30T14:26:50.346173+0000","last_clean_scrub_stamp":"2023-10-30T14:26:50.346173+0000","objects_scrubbed":53,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:23:57.407805+0000","scrub_duration":0.108375394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":230,"num_read_kb":520969,"num_write":671,"num_write_kb":95828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":375058,"num_omap_keys":621,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"110'2212","reported_seq":5041,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:20.112321+0000","last_change":"2023-10-30T14:27:20.112433+0000","last_active":"2023-10-30T14:27:20.112321+0000","last_peered":"2023-10-30T14:27:20.112321+0000","last_clean":"2023-10-30T14:27:20.112321+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T14:27:20.112321+0000","last_undegraded":"2023-10-30T14:27:20.112321+0000","last_fullsized":"2023-10-30T14:27:20.112321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2212","last_scrub_stamp":"2023-10-30T14:27:20.112126+0000","last_deep_scrub":"110'2212","last_deep_scrub_stamp":"2023-10-30T14:27:20.112126+0000","last_clean_scrub_stamp":"2023-10-30T14:27:20.112126+0000","objects_scrubbed":32,"log_size":2212,"log_dups_size":0,"ondisk_log_size":2212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:46.492910+0000","scrub_duration":0.087815049000000006,"objects_trimmed":0,"snaptrim_duration":1.075105449,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":60418,"num_write":1286,"num_write_kb":36950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"110'1084","reported_seq":1379,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:59.155850+0000","last_change":"2023-10-30T14:26:59.155993+0000","last_active":"2023-10-30T14:26:59.155850+0000","last_peered":"2023-10-30T14:26:59.155850+0000","last_clean":"2023-10-30T14:26:59.155850+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T14:26:59.155850+0000","last_undegraded":"2023-10-30T14:26:59.155850+0000","last_fullsized":"2023-10-30T14:26:59.155850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1084","last_scrub_stamp":"2023-10-30T14:26:59.155631+0000","last_deep_scrub":"110'1084","last_deep_scrub_stamp":"2023-10-30T14:26:59.155631+0000","last_clean_scrub_stamp":"2023-10-30T14:26:59.155631+0000","objects_scrubbed":38,"log_size":1084,"log_dups_size":0,"ondisk_log_size":1084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:20:05.652224+0000","scrub_duration":0.220506067,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44106025,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":214,"num_read_kb":377318,"num_write":1089,"num_write_kb":69046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":276186,"num_omap_keys":520,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"110'2247","reported_seq":5091,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:15.229805+0000","last_change":"2023-10-30T14:27:15.229873+0000","last_active":"2023-10-30T14:27:15.229805+0000","last_peered":"2023-10-30T14:27:15.229805+0000","last_clean":"2023-10-30T14:27:15.229805+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T14:27:15.229805+0000","last_undegraded":"2023-10-30T14:27:15.229805+0000","last_fullsized":"2023-10-30T14:27:15.229805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2247","last_scrub_stamp":"2023-10-30T14:27:15.229719+0000","last_deep_scrub":"110'2247","last_deep_scrub_stamp":"2023-10-30T14:27:15.229719+0000","last_clean_scrub_stamp":"2023-10-30T14:27:15.229719+0000","objects_scrubbed":37,"log_size":2247,"log_dups_size":0,"ondisk_log_size":2247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:06:33.811869+0000","scrub_duration":0.052142862999999998,"objects_trimmed":0,"snaptrim_duration":0.36741958800000002,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":157,"num_read_kb":35362,"num_write":1332,"num_write_kb":31772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"110'1165","reported_seq":1469,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.451066+0000","last_change":"2023-10-30T14:26:54.451118+0000","last_active":"2023-10-30T14:26:54.451066+0000","last_peered":"2023-10-30T14:26:54.451066+0000","last_clean":"2023-10-30T14:26:54.451066+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T14:26:54.451066+0000","last_undegraded":"2023-10-30T14:26:54.451066+0000","last_fullsized":"2023-10-30T14:26:54.451066+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1165","last_scrub_stamp":"2023-10-30T14:26:54.450996+0000","last_deep_scrub":"110'1165","last_deep_scrub_stamp":"2023-10-30T14:26:54.450996+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.450996+0000","objects_scrubbed":39,"log_size":1165,"log_dups_size":0,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:18:07.890199+0000","scrub_duration":0.17568342300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":247,"num_read_kb":365780,"num_write":1783,"num_write_kb":65025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":395035,"num_omap_keys":693,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"110'2123","reported_seq":4968,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:10.178670+0000","last_change":"2023-10-30T14:27:10.178735+0000","last_active":"2023-10-30T14:27:10.178670+0000","last_peered":"2023-10-30T14:27:10.178670+0000","last_clean":"2023-10-30T14:27:10.178670+0000","last_became_active":"2023-10-30T13:24:01.001412+0000","last_became_peered":"2023-10-30T13:24:01.001412+0000","last_unstale":"2023-10-30T14:27:10.178670+0000","last_undegraded":"2023-10-30T14:27:10.178670+0000","last_fullsized":"2023-10-30T14:27:10.178670+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2123","last_scrub_stamp":"2023-10-30T14:27:10.178582+0000","last_deep_scrub":"110'2123","last_deep_scrub_stamp":"2023-10-30T14:27:10.178582+0000","last_clean_scrub_stamp":"2023-10-30T14:27:10.178582+0000","objects_scrubbed":35,"log_size":2123,"log_dups_size":0,"ondisk_log_size":2123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:49.167738+0000","scrub_duration":0.079853908000000001,"objects_trimmed":0,"snaptrim_duration":0.32700652899999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":137,"num_read_kb":5811,"num_write":1214,"num_write_kb":16143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"110'515","reported_seq":831,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.490148+0000","last_change":"2023-10-30T14:26:53.490236+0000","last_active":"2023-10-30T14:26:53.490148+0000","last_peered":"2023-10-30T14:26:53.490148+0000","last_clean":"2023-10-30T14:26:53.490148+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T14:26:53.490148+0000","last_undegraded":"2023-10-30T14:26:53.490148+0000","last_fullsized":"2023-10-30T14:26:53.490148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'515","last_scrub_stamp":"2023-10-30T14:26:53.490042+0000","last_deep_scrub":"110'515","last_deep_scrub_stamp":"2023-10-30T14:26:53.490042+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.490042+0000","objects_scrubbed":47,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:16.217409+0000","scrub_duration":0.22512389199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":281,"num_read_kb":442743,"num_write":555,"num_write_kb":70797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":243443,"num_omap_keys":457,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"110'2226","reported_seq":5069,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:59.865953+0000","last_change":"2023-10-30T14:26:59.866026+0000","last_active":"2023-10-30T14:26:59.865953+0000","last_peered":"2023-10-30T14:26:59.865953+0000","last_clean":"2023-10-30T14:26:59.865953+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T14:26:59.865953+0000","last_undegraded":"2023-10-30T14:26:59.865953+0000","last_fullsized":"2023-10-30T14:26:59.865953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2226","last_scrub_stamp":"2023-10-30T14:26:59.865860+0000","last_deep_scrub":"110'2226","last_deep_scrub_stamp":"2023-10-30T14:26:59.865860+0000","last_clean_scrub_stamp":"2023-10-30T14:26:59.865860+0000","objects_scrubbed":40,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:25.638448+0000","scrub_duration":0.053687321000000003,"objects_trimmed":0,"snaptrim_duration":1.183697322,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":144,"num_read_kb":25445,"num_write":1326,"num_write_kb":20705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"110'584","reported_seq":860,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.994203+0000","last_change":"2023-10-30T14:26:53.994349+0000","last_active":"2023-10-30T14:26:53.994203+0000","last_peered":"2023-10-30T14:26:53.994203+0000","last_clean":"2023-10-30T14:26:53.994203+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T14:26:53.994203+0000","last_undegraded":"2023-10-30T14:26:53.994203+0000","last_fullsized":"2023-10-30T14:26:53.994203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'584","last_scrub_stamp":"2023-10-30T14:26:53.993981+0000","last_deep_scrub":"110'584","last_deep_scrub_stamp":"2023-10-30T14:26:53.993981+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.993981+0000","objects_scrubbed":40,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:54.497876+0000","scrub_duration":0.12048839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":236,"num_read_kb":274544,"num_write":603,"num_write_kb":42177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":323567,"num_omap_keys":593,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"110'2168","reported_seq":5027,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:49.864350+0000","last_change":"2023-10-30T14:26:49.864486+0000","last_active":"2023-10-30T14:26:49.864350+0000","last_peered":"2023-10-30T14:26:49.864350+0000","last_clean":"2023-10-30T14:26:49.864350+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T14:26:49.864350+0000","last_undegraded":"2023-10-30T14:26:49.864350+0000","last_fullsized":"2023-10-30T14:26:49.864350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2168","last_scrub_stamp":"2023-10-30T14:26:49.864161+0000","last_deep_scrub":"110'2168","last_deep_scrub_stamp":"2023-10-30T14:26:49.864161+0000","last_clean_scrub_stamp":"2023-10-30T14:26:49.864161+0000","objects_scrubbed":45,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:54:18.553003+0000","scrub_duration":0.11066809399999999,"objects_trimmed":0,"snaptrim_duration":0.56267233699999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":144,"num_read_kb":34655,"num_write":1307,"num_write_kb":20329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"110'1112","reported_seq":1394,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:45.299966+0000","last_change":"2023-10-30T14:26:45.300020+0000","last_active":"2023-10-30T14:26:45.299966+0000","last_peered":"2023-10-30T14:26:45.299966+0000","last_clean":"2023-10-30T14:26:45.299966+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T14:26:45.299966+0000","last_undegraded":"2023-10-30T14:26:45.299966+0000","last_fullsized":"2023-10-30T14:26:45.299966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1112","last_scrub_stamp":"2023-10-30T14:26:45.299861+0000","last_deep_scrub":"110'1112","last_deep_scrub_stamp":"2023-10-30T14:26:45.299861+0000","last_clean_scrub_stamp":"2023-10-30T14:26:45.299861+0000","objects_scrubbed":51,"log_size":1112,"log_dups_size":0,"ondisk_log_size":1112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:30.314690+0000","scrub_duration":0.15631188700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":237,"num_read_kb":249991,"num_write":1213,"num_write_kb":43538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":609555,"num_omap_keys":1142,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"110'2316","reported_seq":5113,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:18.616129+0000","last_change":"2023-10-30T14:27:18.616224+0000","last_active":"2023-10-30T14:27:18.616129+0000","last_peered":"2023-10-30T14:27:18.616129+0000","last_clean":"2023-10-30T14:27:18.616129+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T14:27:18.616129+0000","last_undegraded":"2023-10-30T14:27:18.616129+0000","last_fullsized":"2023-10-30T14:27:18.616129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2316","last_scrub_stamp":"2023-10-30T14:27:18.616043+0000","last_deep_scrub":"110'2316","last_deep_scrub_stamp":"2023-10-30T14:27:18.616043+0000","last_clean_scrub_stamp":"2023-10-30T14:27:18.616043+0000","objects_scrubbed":45,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:14:36.881982+0000","scrub_duration":0.061902202000000003,"objects_trimmed":0,"snaptrim_duration":0.93815086199999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":129,"num_read_kb":1227,"num_write":1352,"num_write_kb":12536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"110'1527","reported_seq":1874,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.480258+0000","last_change":"2023-10-30T14:26:53.480338+0000","last_active":"2023-10-30T14:26:53.480258+0000","last_peered":"2023-10-30T14:26:53.480258+0000","last_clean":"2023-10-30T14:26:53.480258+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T14:26:53.480258+0000","last_undegraded":"2023-10-30T14:26:53.480258+0000","last_fullsized":"2023-10-30T14:26:53.480258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1527","last_scrub_stamp":"2023-10-30T14:26:53.480148+0000","last_deep_scrub":"110'1527","last_deep_scrub_stamp":"2023-10-30T14:26:53.480148+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.480148+0000","objects_scrubbed":54,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:31:02.686045+0000","scrub_duration":0.25185449900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":321,"num_read_kb":438308,"num_write":2003,"num_write_kb":98279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":359842,"num_omap_keys":684,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"110'2123","reported_seq":4973,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:55.746681+0000","last_change":"2023-10-30T14:26:55.746755+0000","last_active":"2023-10-30T14:26:55.746681+0000","last_peered":"2023-10-30T14:26:55.746681+0000","last_clean":"2023-10-30T14:26:55.746681+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T14:26:55.746681+0000","last_undegraded":"2023-10-30T14:26:55.746681+0000","last_fullsized":"2023-10-30T14:26:55.746681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2123","last_scrub_stamp":"2023-10-30T14:26:55.746576+0000","last_deep_scrub":"110'2123","last_deep_scrub_stamp":"2023-10-30T14:26:55.746576+0000","last_clean_scrub_stamp":"2023-10-30T14:26:55.746576+0000","objects_scrubbed":27,"log_size":2123,"log_dups_size":0,"ondisk_log_size":2123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:08:17.663937+0000","scrub_duration":0.039827926,"objects_trimmed":0,"snaptrim_duration":0.94968995099999998,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":141,"num_read_kb":11852,"num_write":1237,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"110'1108","reported_seq":1435,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.161003+0000","last_change":"2023-10-30T14:26:57.161075+0000","last_active":"2023-10-30T14:26:57.161003+0000","last_peered":"2023-10-30T14:26:57.161003+0000","last_clean":"2023-10-30T14:26:57.161003+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T14:26:57.161003+0000","last_undegraded":"2023-10-30T14:26:57.161003+0000","last_fullsized":"2023-10-30T14:26:57.161003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1108","last_scrub_stamp":"2023-10-30T14:26:57.160913+0000","last_deep_scrub":"110'1108","last_deep_scrub_stamp":"2023-10-30T14:26:57.160913+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.160913+0000","objects_scrubbed":44,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:56:50.729954+0000","scrub_duration":0.25105180199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":296,"num_read_kb":429426,"num_write":1159,"num_write_kb":75926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":378159,"num_omap_keys":718,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"110'2194","reported_seq":5040,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:06.417252+0000","last_change":"2023-10-30T14:27:06.417354+0000","last_active":"2023-10-30T14:27:06.417252+0000","last_peered":"2023-10-30T14:27:06.417252+0000","last_clean":"2023-10-30T14:27:06.417252+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T14:27:06.417252+0000","last_undegraded":"2023-10-30T14:27:06.417252+0000","last_fullsized":"2023-10-30T14:27:06.417252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2194","last_scrub_stamp":"2023-10-30T14:27:06.417083+0000","last_deep_scrub":"110'2194","last_deep_scrub_stamp":"2023-10-30T14:27:06.417083+0000","last_clean_scrub_stamp":"2023-10-30T14:27:06.417083+0000","objects_scrubbed":30,"log_size":2194,"log_dups_size":0,"ondisk_log_size":2194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:32:26.701207+0000","scrub_duration":0.099883823999999996,"objects_trimmed":0,"snaptrim_duration":0.419978554,"stat_sum":{"num_bytes":5,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":137,"num_read_kb":40109,"num_write":1287,"num_write_kb":25283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"110'2213","reported_seq":5031,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:16.181195+0000","last_change":"2023-10-30T14:27:16.181270+0000","last_active":"2023-10-30T14:27:16.181195+0000","last_peered":"2023-10-30T14:27:16.181195+0000","last_clean":"2023-10-30T14:27:16.181195+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T14:27:16.181195+0000","last_undegraded":"2023-10-30T14:27:16.181195+0000","last_fullsized":"2023-10-30T14:27:16.181195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2213","last_scrub_stamp":"2023-10-30T14:27:16.181071+0000","last_deep_scrub":"110'2213","last_deep_scrub_stamp":"2023-10-30T14:27:16.181071+0000","last_clean_scrub_stamp":"2023-10-30T14:27:16.181071+0000","objects_scrubbed":33,"log_size":2213,"log_dups_size":0,"ondisk_log_size":2213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:59:49.845658+0000","scrub_duration":0.12957070300000001,"objects_trimmed":0,"snaptrim_duration":1.073619836,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":129,"num_read_kb":12975,"num_write":1318,"num_write_kb":20116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"110'784","reported_seq":1084,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:56.157617+0000","last_change":"2023-10-30T14:26:56.157707+0000","last_active":"2023-10-30T14:26:56.157617+0000","last_peered":"2023-10-30T14:26:56.157617+0000","last_clean":"2023-10-30T14:26:56.157617+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T14:26:56.157617+0000","last_undegraded":"2023-10-30T14:26:56.157617+0000","last_fullsized":"2023-10-30T14:26:56.157617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'784","last_scrub_stamp":"2023-10-30T14:26:56.157430+0000","last_deep_scrub":"110'784","last_deep_scrub_stamp":"2023-10-30T14:26:56.157430+0000","last_clean_scrub_stamp":"2023-10-30T14:26:56.157430+0000","objects_scrubbed":50,"log_size":784,"log_dups_size":0,"ondisk_log_size":784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:19.819331+0000","scrub_duration":0.231260298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":304,"num_read_kb":264369,"num_write":877,"num_write_kb":48007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":477325,"num_omap_keys":851,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"110'2679","reported_seq":5374,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:17.160156+0000","last_change":"2023-10-30T14:27:17.160251+0000","last_active":"2023-10-30T14:27:17.160156+0000","last_peered":"2023-10-30T14:27:17.160156+0000","last_clean":"2023-10-30T14:27:17.160156+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T14:27:17.160156+0000","last_undegraded":"2023-10-30T14:27:17.160156+0000","last_fullsized":"2023-10-30T14:27:17.160156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2679","last_scrub_stamp":"2023-10-30T14:27:17.160038+0000","last_deep_scrub":"110'2679","last_deep_scrub_stamp":"2023-10-30T14:27:17.160038+0000","last_clean_scrub_stamp":"2023-10-30T14:27:17.160038+0000","objects_scrubbed":37,"log_size":2679,"log_dups_size":0,"ondisk_log_size":2679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:38.223350+0000","scrub_duration":0.126936049,"objects_trimmed":0,"snaptrim_duration":1.021840552,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":171,"num_read_kb":40337,"num_write":1456,"num_write_kb":34441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"110'1224","reported_seq":1535,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:04.243739+0000","last_change":"2023-10-30T14:27:04.243861+0000","last_active":"2023-10-30T14:27:04.243739+0000","last_peered":"2023-10-30T14:27:04.243739+0000","last_clean":"2023-10-30T14:27:04.243739+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T14:27:04.243739+0000","last_undegraded":"2023-10-30T14:27:04.243739+0000","last_fullsized":"2023-10-30T14:27:04.243739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1224","last_scrub_stamp":"2023-10-30T14:27:04.243569+0000","last_deep_scrub":"110'1224","last_deep_scrub_stamp":"2023-10-30T14:27:04.243569+0000","last_clean_scrub_stamp":"2023-10-30T14:27:04.243569+0000","objects_scrubbed":45,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:57:40.587906+0000","scrub_duration":0.244476464,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":255,"num_read_kb":502436,"num_write":1236,"num_write_kb":99955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":409468,"num_omap_keys":801,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"110'2069","reported_seq":4947,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:09.913752+0000","last_change":"2023-10-30T14:27:09.913836+0000","last_active":"2023-10-30T14:27:09.913752+0000","last_peered":"2023-10-30T14:27:09.913752+0000","last_clean":"2023-10-30T14:27:09.913752+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T14:27:09.913752+0000","last_undegraded":"2023-10-30T14:27:09.913752+0000","last_fullsized":"2023-10-30T14:27:09.913752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2069","last_scrub_stamp":"2023-10-30T14:27:09.913657+0000","last_deep_scrub":"110'2069","last_deep_scrub_stamp":"2023-10-30T14:27:09.913657+0000","last_clean_scrub_stamp":"2023-10-30T14:27:09.913657+0000","objects_scrubbed":41,"log_size":2069,"log_dups_size":0,"ondisk_log_size":2069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:21:51.262361+0000","scrub_duration":0.091522568999999998,"objects_trimmed":0,"snaptrim_duration":1.015373858,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":13189,"num_write":1249,"num_write_kb":15084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"110'817","reported_seq":1081,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:45.554360+0000","last_change":"2023-10-30T14:26:45.554415+0000","last_active":"2023-10-30T14:26:45.554360+0000","last_peered":"2023-10-30T14:26:45.554360+0000","last_clean":"2023-10-30T14:26:45.554360+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T14:26:45.554360+0000","last_undegraded":"2023-10-30T14:26:45.554360+0000","last_fullsized":"2023-10-30T14:26:45.554360+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'817","last_scrub_stamp":"2023-10-30T14:26:45.554286+0000","last_deep_scrub":"110'817","last_deep_scrub_stamp":"2023-10-30T14:26:45.554286+0000","last_clean_scrub_stamp":"2023-10-30T14:26:45.554286+0000","objects_scrubbed":38,"log_size":817,"log_dups_size":0,"ondisk_log_size":817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:36.415824+0000","scrub_duration":0.061981296999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":159,"num_read_kb":369044,"num_write":787,"num_write_kb":79251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":283692,"num_omap_keys":523,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"110'2229","reported_seq":5188,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.797248+0000","last_change":"2023-10-30T14:27:00.797301+0000","last_active":"2023-10-30T14:27:00.797248+0000","last_peered":"2023-10-30T14:27:00.797248+0000","last_clean":"2023-10-30T14:27:00.797248+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T14:27:00.797248+0000","last_undegraded":"2023-10-30T14:27:00.797248+0000","last_fullsized":"2023-10-30T14:27:00.797248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2229","last_scrub_stamp":"2023-10-30T14:27:00.797179+0000","last_deep_scrub":"110'2229","last_deep_scrub_stamp":"2023-10-30T14:27:00.797179+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.797179+0000","objects_scrubbed":37,"log_size":2229,"log_dups_size":0,"ondisk_log_size":2229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:23.560124+0000","scrub_duration":0.084074704,"objects_trimmed":0,"snaptrim_duration":1.043790548,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":262,"num_read_kb":42307,"num_write":1336,"num_write_kb":30630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"110'1065","reported_seq":1395,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.937061+0000","last_change":"2023-10-30T14:26:42.937107+0000","last_active":"2023-10-30T14:26:42.937061+0000","last_peered":"2023-10-30T14:26:42.937061+0000","last_clean":"2023-10-30T14:26:42.937061+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T14:26:42.937061+0000","last_undegraded":"2023-10-30T14:26:42.937061+0000","last_fullsized":"2023-10-30T14:26:42.937061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1065","last_scrub_stamp":"2023-10-30T14:26:42.936998+0000","last_deep_scrub":"110'1065","last_deep_scrub_stamp":"2023-10-30T14:26:42.936998+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.936998+0000","objects_scrubbed":55,"log_size":1065,"log_dups_size":0,"ondisk_log_size":1065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:27.046930+0000","scrub_duration":0.131982027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777798,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":313,"num_read_kb":340671,"num_write":1085,"num_write_kb":63851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":501138,"num_omap_keys":760,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":11,"num_objects":2377,"num_object_clones":2,"num_object_copies":4754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":9389,"num_read_kb":1724409,"num_write":83589,"num_write_kb":1459751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140601,"ondisk_log_size":140601,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1285101348,"num_objects":1569,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1569,"num_whiteouts":0,"num_read":10652,"num_read_kb":12762809,"num_write":35655,"num_write_kb":2343865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13139950,"num_omap_keys":23453,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2570362880,"data_stored":2570202696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33559,"ondisk_log_size":33559,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575623,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595144,"kb_used_data":389120,"kb_used_omap":1,"kb_used_meta":206014,"kb_avail":93776696,"statfs":{"total":96636764160,"available":96027336704,"internally_reserved":0,"allocated":398458880,"data_stored":397803489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":210958795},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.73999999999999999,"15min":0.79700000000000004},"min":{"1min":0.41399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.82699999999999996,"15min":0.91000000000000003},"min":{"1min":0.68100000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.73699999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.80700000000000005,"15min":0.86599999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.288,"5min":2.452,"15min":2.452},"last":0.878},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.79000000000000004,"15min":0.83999999999999997},"min":{"1min":0.56499999999999995,"5min":0.495,"15min":0.47399999999999998},"max":{"1min":3.7160000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":1.107}]},{"osd":2,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.75800000000000001,"15min":0.81999999999999995},"min":{"1min":0.45200000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.96299999999999997,"5min":1.169,"15min":1.4350000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.84799999999999998,"15min":0.93600000000000005},"min":{"1min":0.52700000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2210000000000001,"5min":1.2929999999999999,"15min":1.5569999999999999},"last":1.202}]},{"osd":3,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94899999999999995,"15min":0.95099999999999996},"min":{"1min":0.60499999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.0790000000000002,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.026,"15min":1.0029999999999999},"min":{"1min":0.67000000000000004,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.7869999999999999,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":1.155}]},{"osd":4,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.91900000000000004,"15min":0.90200000000000002},"min":{"1min":0.63200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.6629999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.202,"5min":1.0660000000000001,"15min":1.042},"min":{"1min":0.84099999999999997,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":3.6899999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":1.0149999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.139,"15min":1.0880000000000001},"min":{"1min":0.75800000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.4179999999999999,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.127,"15min":1.1599999999999999},"min":{"1min":0.88300000000000001,"5min":0.436,"15min":0.436},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.68999999999999995}]},{"osd":6,"last update":"Mon Oct 30 14:26:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1379999999999999,"15min":1.1140000000000001},"min":{"1min":0.50800000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":6.3070000000000004,"5min":6.3070000000000004,"15min":6.3070000000000004},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.054,"5min":1.1279999999999999,"15min":1.181},"min":{"1min":0.65700000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.83299999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575626,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505392,"kb_used_data":339688,"kb_used_omap":1,"kb_used_meta":165694,"kb_avail":93866448,"statfs":{"total":96636764160,"available":96119242752,"internally_reserved":0,"allocated":347840512,"data_stored":347188935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169671111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.74099999999999999,"15min":0.79700000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.375},"max":{"1min":1.105,"5min":1.306,"15min":1.363},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.85499999999999998,"15min":0.94099999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.0389999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.72899999999999998,"15min":0.77700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.452,"5min":3.452,"15min":3.452},"last":0.63},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.748,"15min":0.77800000000000002},"min":{"1min":0.48399999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.163,"5min":1.706,"15min":1.706},"last":0.73199999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.80500000000000005,"15min":0.83599999999999997},"min":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.571},"last":0.878},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.99299999999999999,"15min":0.94699999999999995},"min":{"1min":0.52300000000000002,"5min":0.502,"15min":0.502},"max":{"1min":3.6459999999999999,"5min":5.4480000000000004,"15min":5.4480000000000004},"last":0.66900000000000004}]},{"osd":3,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.04,"15min":1.0369999999999999},"min":{"1min":0.752,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":3.5169999999999999,"5min":3.5169999999999999,"15min":3.5169999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.228,"5min":1.1240000000000001,"15min":1.1220000000000001},"min":{"1min":0.77700000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":3.5830000000000002,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":0.77700000000000002}]},{"osd":4,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.57199999999999995,"5min":0.434,"15min":0.434},"max":{"1min":1.2410000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.98899999999999999,"15min":1.026},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.452,"5min":1.595,"15min":1.595},"last":0.98499999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.1599999999999999,"15min":1.0429999999999999},"min":{"1min":0.59899999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.9590000000000001,"5min":3.0219999999999998,"15min":3.0219999999999998},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":1.0069999999999999,"15min":1.0429999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.835,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":1.087}]},{"osd":7,"last update":"Mon Oct 30 14:26:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.204,"15min":1.196},"min":{"1min":1.048,"5min":0.58699999999999997,"15min":0.52600000000000002},"max":{"1min":1.3740000000000001,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1599999999999999,"15min":1.1359999999999999},"min":{"1min":0.83399999999999996,"5min":0.752,"15min":0.63700000000000001},"max":{"1min":1.3440000000000001,"5min":2.806,"15min":2.806},"last":1.143}]}]},{"osd":5,"up_from":13,"seq":55834575623,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570756,"kb_used_data":368956,"kb_used_omap":1,"kb_used_meta":201790,"kb_avail":93801084,"statfs":{"total":96636764160,"available":96052310016,"internally_reserved":0,"allocated":377810944,"data_stored":377158158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206633414},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.84599999999999997},"min":{"1min":0.439,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1120000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.93700000000000006,"15min":0.95699999999999996},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.373,"5min":1.617,"15min":1.9219999999999999},"last":1.0249999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.81799999999999995,"15min":0.83899999999999997},"min":{"1min":0.48899999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.77100000000000002,"15min":0.83999999999999997},"min":{"1min":0.53700000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.774,"5min":4.774,"15min":4.774},"last":0.70299999999999996}]},{"osd":2,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85199999999999998,"15min":0.88100000000000001},"min":{"1min":0.58599999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.111,"5min":1.2589999999999999,"15min":1.331},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.84699999999999998},"min":{"1min":0.501,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.1879999999999999,"5min":1.331,"15min":1.796},"last":1.0720000000000001}]},{"osd":3,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.96999999999999997,"15min":1.0009999999999999},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3100000000000001,"5min":1.536,"15min":1.536},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.036,"5min":0.92600000000000005,"15min":0.99099999999999999},"min":{"1min":0.51800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":2.0030000000000001},"last":1.1160000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.88800000000000001,"15min":0.877},"min":{"1min":0.374,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.087,"5min":2.8170000000000002,"15min":2.8170000000000002},"last":1.145},{"interface":"front","average":{"1min":1.155,"5min":1.0609999999999999,"15min":1.036},"min":{"1min":0.45100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.4690000000000001,"5min":2.2400000000000002,"15min":2.262},"last":1.224}]},{"osd":6,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.071,"15min":1.1220000000000001},"min":{"1min":0.40300000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3380000000000001,"5min":2.3090000000000002,"15min":2.7949999999999999},"last":1.262},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.038,"15min":1.054},"min":{"1min":0.38800000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.4350000000000001,"5min":2.161,"15min":2.1629999999999998},"last":1.2929999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:26:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0209999999999999,"15min":1.091},"min":{"1min":0.78100000000000003,"5min":0.317,"15min":0.317},"max":{"1min":1.3440000000000001,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.321},{"interface":"front","average":{"1min":1.149,"5min":1.1299999999999999,"15min":1.153},"min":{"1min":0.95099999999999996,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.4019999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.3740000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575623,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469348,"kb_used_data":267484,"kb_used_omap":1,"kb_used_meta":201854,"kb_avail":93902492,"statfs":{"total":96636764160,"available":96156151808,"internally_reserved":0,"allocated":273903616,"data_stored":273238103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206698950},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.73599999999999999,"15min":0.80400000000000005},"min":{"1min":0.48799999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1259999999999999,"5min":1.3500000000000001,"15min":1.446},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.71099999999999997,"15min":0.82299999999999995},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.875,"5min":1.659,"15min":1.659},"last":0.87}]},{"osd":1,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.56799999999999995,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.141,"5min":1.589,"15min":1.589},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.84399999999999997,"15min":0.84699999999999998},"min":{"1min":0.58999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.075,"5min":1.532,"15min":1.532},"last":0.96299999999999997}]},{"osd":2,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.76100000000000001,"15min":0.85999999999999999},"min":{"1min":0.55800000000000005,"5min":0.5,"15min":0.5},"max":{"1min":0.94899999999999995,"5min":1.607,"15min":1.607},"last":0.625},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84699999999999998,"15min":0.92200000000000004},"min":{"1min":0.57199999999999995,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.218,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.73499999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.99099999999999999,"15min":0.97499999999999998},"min":{"1min":0.66300000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.242,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":1.071},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.099,"15min":1.0780000000000001},"min":{"1min":0.879,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3480000000000001,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":1.1439999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.98899999999999999,"15min":1.042},"min":{"1min":0.77000000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.3009999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.1930000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.65400000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.1279999999999999,"5min":1.3220000000000001,"15min":1.3280000000000001},"last":1.4099999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.97899999999999998,"15min":0.96099999999999997},"min":{"1min":0.78800000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.3100000000000001,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.042,"15min":1.0700000000000001},"min":{"1min":0.78300000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.238,"5min":1.677,"15min":1.728},"last":0.82699999999999996}]},{"osd":7,"last update":"Mon Oct 30 14:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.137,"15min":1.1040000000000001},"min":{"1min":1.0820000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.417,"5min":1.4750000000000001,"15min":1.518},"last":1.228},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.204,"15min":1.1799999999999999},"min":{"1min":1.056,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4670000000000001,"5min":1.587,"15min":1.587},"last":0.92700000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575621,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597696,"kb_used_data":382008,"kb_used_omap":1,"kb_used_meta":215678,"kb_avail":93774144,"statfs":{"total":96636764160,"available":96024723456,"internally_reserved":0,"allocated":391176192,"data_stored":390510799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":220854728},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.374,"15min":0.40300000000000002},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.72099999999999997,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.496},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.46700000000000003,"15min":0.48999999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.56399999999999995,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.59899999999999998}]},{"osd":1,"last update":"Mon Oct 30 14:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.47499999999999998,"15min":0.45600000000000002},"min":{"1min":0.379,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51100000000000001,"15min":0.54400000000000004},"min":{"1min":0.32800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.65500000000000003}]},{"osd":2,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54400000000000004,"15min":0.59199999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.66300000000000003,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.433,"5min":0.29999999999999999,"15min":0.29299999999999998},"max":{"1min":0.69699999999999995,"5min":1.73,"15min":1.73},"last":0.68000000000000005}]},{"osd":4,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.76600000000000001,"15min":0.79800000000000004},"min":{"1min":0.57699999999999996,"5min":0.379,"15min":0.379},"max":{"1min":0.98599999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.78100000000000003,"15min":0.81200000000000006},"min":{"1min":0.52500000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.89200000000000002,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.83699999999999997}]},{"osd":5,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.76100000000000001,"15min":0.81799999999999995},"min":{"1min":0.55700000000000005,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0209999999999999,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.75,"5min":0.78300000000000003,"15min":0.84499999999999997},"min":{"1min":0.58399999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.032,"5min":1.486,"15min":1.486},"last":0.93400000000000005}]},{"osd":6,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83999999999999997,"15min":0.88},"min":{"1min":0.625,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1830000000000001,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.90200000000000002,"15min":0.93600000000000005},"min":{"1min":0.72399999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.224,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.073}]},{"osd":7,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.80800000000000005,"15min":0.85599999999999998},"min":{"1min":0.65500000000000003,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.052,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.77900000000000003,"15min":0.81599999999999995},"min":{"1min":0.60899999999999999,"5min":0.501,"15min":0.501},"max":{"1min":0.93400000000000005,"5min":1.593,"15min":1.593},"last":0.88100000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299412,"kb_used_data":200268,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94072428,"statfs":{"total":96636764160,"available":96330166272,"internally_reserved":0,"allocated":205074432,"data_stored":204423364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101513670},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42299999999999999,"15min":0.45600000000000002},"min":{"1min":0.26100000000000001,"5min":0.246,"15min":0.23899999999999999},"max":{"1min":0.72599999999999998,"5min":0.747,"15min":0.89300000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.499,"15min":0.51300000000000001},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":0.63900000000000001,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":0.501}]},{"osd":1,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.41699999999999998,"15min":0.44600000000000001},"min":{"1min":0.307,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70099999999999996,"5min":1.956,"15min":1.956},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.35099999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.153,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.57999999999999996}]},{"osd":3,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55700000000000005,"15min":0.57399999999999995},"min":{"1min":0.40699999999999997,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.2010000000000001,"5min":1.988,"15min":1.988},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.502,"15min":0.51800000000000002},"min":{"1min":0.32600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69599999999999995,"5min":2.335,"15min":2.335},"last":0.68100000000000005}]},{"osd":4,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.80800000000000005,"15min":0.82599999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.085,"5min":2.302,"15min":2.302},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84299999999999997,"15min":0.83699999999999997},"min":{"1min":0.66900000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0169999999999999,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.83699999999999997}]},{"osd":5,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.77000000000000002,"15min":0.80400000000000005},"min":{"1min":0.59099999999999997,"5min":0.501,"15min":0.501},"max":{"1min":1.028,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.028},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.82999999999999996,"15min":0.84199999999999997},"min":{"1min":0.68400000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0940000000000001,"5min":1.875,"15min":1.875},"last":0.92400000000000004}]},{"osd":6,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.82299999999999995,"15min":0.82899999999999996},"min":{"1min":0.63100000000000001,"5min":0.502,"15min":0.48099999999999998},"max":{"1min":1.123,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.82899999999999996,"15min":0.86299999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.2130000000000001,"5min":1.921,"15min":1.921},"last":0.90100000000000002}]},{"osd":7,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85899999999999999,"15min":0.86099999999999999},"min":{"1min":0.66300000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1100000000000001,"5min":4.5739999999999998,"15min":4.5739999999999998},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.88500000000000001,"15min":0.91400000000000003},"min":{"1min":0.64700000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.202,"5min":2.375,"15min":2.375},"last":1.1419999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575624,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494860,"kb_used_data":329092,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93876980,"statfs":{"total":96636764160,"available":96130027520,"internally_reserved":0,"allocated":336990208,"data_stored":336334619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.45800000000000002},"min":{"1min":0.29999999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.72599999999999998,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.49099999999999999,"15min":0.51400000000000001},"min":{"1min":0.36799999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86799999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0469999999999999,"5min":1.954,"15min":1.954},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.50600000000000001,"15min":0.49299999999999999},"min":{"1min":0.28100000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":5.7039999999999997,"5min":5.7039999999999997,"15min":5.7039999999999997},"last":0.56000000000000005}]},{"osd":3,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55100000000000005,"15min":0.55000000000000004},"min":{"1min":0.44600000000000001,"5min":0.35499999999999998,"15min":0.35099999999999998},"max":{"1min":1.3220000000000001,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3600000000000001,"5min":2.625,"15min":2.625},"last":0.72999999999999998}]},{"osd":4,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.79200000000000004,"15min":0.82999999999999996},"min":{"1min":0.66000000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.92900000000000005,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.82199999999999995,"15min":0.872},"min":{"1min":0.67300000000000004,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0309999999999999,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.84199999999999997}]},{"osd":5,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.78800000000000003,"15min":0.86299999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.004,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.82099999999999995,"15min":0.88800000000000001},"min":{"1min":0.52200000000000002,"5min":0.499,"15min":0.499},"max":{"1min":1.167,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.749}]},{"osd":6,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.89900000000000002,"15min":0.84499999999999997},"min":{"1min":0.73999999999999999,"5min":0.624,"15min":0.56499999999999995},"max":{"1min":1.01,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.81000000000000005,"15min":0.88600000000000001},"min":{"1min":0.56399999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.99299999999999999,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.67100000000000004}]},{"osd":7,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.623,"5min":0.59599999999999997,"15min":0.56699999999999995},"max":{"1min":1.085,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.83999999999999997,"15min":0.84999999999999998},"min":{"1min":0.66900000000000004,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0660000000000001,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.92000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575621,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425348,"kb_used_data":245756,"kb_used_omap":1,"kb_used_meta":179582,"kb_avail":93946492,"statfs":{"total":96636764160,"available":96201207808,"internally_reserved":0,"allocated":251654144,"data_stored":251001762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":183892422},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.377,"15min":0.40100000000000002},"min":{"1min":0.26400000000000001,"5min":0.247,"15min":0.23499999999999999},"max":{"1min":0.61699999999999999,"5min":0.66100000000000003,"15min":0.94999999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48499999999999999,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65800000000000003,"5min":0.82299999999999995,"15min":1.0229999999999999},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.45400000000000001,"15min":0.46200000000000002},"min":{"1min":0.33400000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47699999999999998,"15min":0.496},"min":{"1min":0.38100000000000001,"5min":0.30499999999999999,"15min":0.29099999999999998},"max":{"1min":0.83599999999999997,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.51200000000000001}]},{"osd":3,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.91900000000000004,"5min":1.236,"15min":1.236},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51800000000000002,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":0.49099999999999999}]},{"osd":4,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78200000000000003,"15min":0.79700000000000004},"min":{"1min":0.58899999999999997,"5min":0.503,"15min":0.503},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.747,"15min":0.80000000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91200000000000003,"5min":1.012,"15min":1.2210000000000001},"last":0.92600000000000005}]},{"osd":5,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.78700000000000003,"15min":0.80900000000000005},"min":{"1min":0.61299999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.80700000000000005,"15min":0.84499999999999997},"min":{"1min":0.626,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.746}]},{"osd":6,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.80800000000000005,"15min":0.83799999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.99399999999999999,"5min":1.222,"15min":1.486},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.80600000000000005,"15min":0.82899999999999996},"min":{"1min":0.66900000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.90300000000000002}]},{"osd":7,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.82399999999999995,"15min":0.84699999999999998},"min":{"1min":0.70999999999999996,"5min":0.625,"15min":0.58599999999999997},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81799999999999995,"15min":0.86199999999999999},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0680000000000001,"5min":1.1559999999999999,"15min":1.4419999999999999},"last":0.83399999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249647104,"data_stored":249627559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335572992,"data_stored":335549765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203661312,"data_stored":203639529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389746688,"data_stored":389725939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":272470016,"data_stored":272453238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376397824,"data_stored":376374323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":346419200,"data_stored":346404076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":396447744,"data_stored":396428267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":11,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:27:25.651 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:25.651 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:25.652 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:25.652 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:25.652 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:25.653 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:25.653 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:25.654 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:25.655 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T14:27:41.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:27:41.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:27:45.656 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:27:45.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.805+0000 7f8e7e580640 1 Processor -- start 2023-10-30T14:27:45.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.805+0000 7f8e7e580640 1 -- start start 2023-10-30T14:27:45.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.805+0000 7f8e7e580640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e7806f7a0 0x7f8e7806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.805+0000 7f8e7e580640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8e780c6bd0 0x7f8e78069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.805+0000 7f8e7e580640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e78071110 0x7f8e78158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.805+0000 7f8e7e580640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e780700c0 con 0x7f8e7806f7a0 2023-10-30T14:27:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.805+0000 7f8e7e580640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e7806a3d0 con 0x7f8e780c6bd0 2023-10-30T14:27:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.805+0000 7f8e7e580640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e78066cf0 con 0x7f8e78071110 2023-10-30T14:27:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.805+0000 7f8e77fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e7806f7a0 0x7f8e7806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.805+0000 7f8e77fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8e7806f7a0 0x7f8e7806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55376/0 (socket says 172.21.15.46:55376) 2023-10-30T14:27:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.805+0000 7f8e77fff640 1 -- 172.21.15.46:0/3214384073 learned_addr learned my addr 172.21.15.46:0/3214384073 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:27:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e777fe640 1 --2- 172.21.15.46:0/3214384073 >> v2:172.21.15.154:3300/0 conn(0x7f8e780c6bd0 0x7f8e78069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e7cb1d640 1 --2- 172.21.15.46:0/3214384073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e78071110 0x7f8e78158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e77fff640 1 -- 172.21.15.46:0/3214384073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e78071110 msgr2=0x7f8e78158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:45.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e77fff640 1 --2- 172.21.15.46:0/3214384073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e78071110 0x7f8e78158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e77fff640 1 -- 172.21.15.46:0/3214384073 >> v2:172.21.15.154:3300/0 conn(0x7f8e780c6bd0 msgr2=0x7f8e78069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:27:45.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e77fff640 1 --2- 172.21.15.46:0/3214384073 >> v2:172.21.15.154:3300/0 conn(0x7f8e780c6bd0 0x7f8e78069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e77fff640 1 -- 172.21.15.46:0/3214384073 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e780c70d0 con 0x7f8e7806f7a0 2023-10-30T14:27:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e77fff640 1 --2- 172.21.15.46:0/3214384073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e7806f7a0 0x7f8e7806fb80 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f8e7806f1c0 tx=0x7f8e6801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af7f04feba526e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e76ffd640 1 -- 172.21.15.46:0/3214384073 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e680a89b0 con 0x7f8e7806f7a0 2023-10-30T14:27:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e76ffd640 1 -- 172.21.15.46:0/3214384073 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e680a8b10 con 0x7f8e7806f7a0 2023-10-30T14:27:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e76ffd640 1 -- 172.21.15.46:0/3214384073 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e680a8ea0 con 0x7f8e7806f7a0 2023-10-30T14:27:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e7e580640 1 -- 172.21.15.46:0/3214384073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e7806f7a0 msgr2=0x7f8e7806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.809+0000 7f8e7e580640 1 --2- 172.21.15.46:0/3214384073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e7806f7a0 0x7f8e7806fb80 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f8e7806f1c0 tx=0x7f8e6801ecf0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 -- 172.21.15.46:0/3214384073 shutdown_connections 2023-10-30T14:27:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 --2- 172.21.15.46:0/3214384073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e78071110 0x7f8e78158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 --2- 172.21.15.46:0/3214384073 >> v2:172.21.15.154:3300/0 conn(0x7f8e780c6bd0 0x7f8e78069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 --2- 172.21.15.46:0/3214384073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e7806f7a0 0x7f8e7806fb80 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 -- 172.21.15.46:0/3214384073 >> 172.21.15.46:0/3214384073 conn(0x7f8e780c7910 msgr2=0x7f8e780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 -- 172.21.15.46:0/3214384073 shutdown_connections 2023-10-30T14:27:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 -- 172.21.15.46:0/3214384073 wait complete. 2023-10-30T14:27:45.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 Processor -- start 2023-10-30T14:27:45.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 -- start start 2023-10-30T14:27:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e7806f7a0 0x7f8e7814cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e78071110 0x7f8e781f3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e780c6bd0 0x7f8e780aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e781f2c50 con 0x7f8e7806f7a0 2023-10-30T14:27:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e781f2900 con 0x7f8e780c6bd0 2023-10-30T14:27:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7e580640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7814d250 con 0x7f8e78071110 2023-10-30T14:27:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e77fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e7806f7a0 0x7f8e7814cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e77fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e7806f7a0 0x7f8e7814cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55388/0 (socket says 172.21.15.46:55388) 2023-10-30T14:27:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e77fff640 1 -- 172.21.15.46:0/3969647047 learned_addr learned my addr 172.21.15.46:0/3969647047 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:27:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e7cb1d640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e780c6bd0 0x7f8e780aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e777fe640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e78071110 0x7f8e781f3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e77fff640 1 -- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e78071110 msgr2=0x7f8e781f3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e77fff640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e78071110 0x7f8e781f3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e77fff640 1 -- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e780c6bd0 msgr2=0x7f8e780aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e77fff640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e780c6bd0 0x7f8e780aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.813+0000 7f8e77fff640 1 -- 172.21.15.46:0/3969647047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e680094d0 con 0x7f8e7806f7a0 2023-10-30T14:27:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.817+0000 7f8e77fff640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e7806f7a0 0x7f8e7814cd10 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f8e6801d3a0 tx=0x7f8e6801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.817+0000 7f8e74ff9640 1 -- 172.21.15.46:0/3969647047 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e680ba600 con 0x7f8e7806f7a0 2023-10-30T14:27:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.817+0000 7f8e74ff9640 1 -- 172.21.15.46:0/3969647047 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e680ba760 con 0x7f8e7806f7a0 2023-10-30T14:27:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.817+0000 7f8e7e580640 1 -- 172.21.15.46:0/3969647047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7814eb20 con 0x7f8e7806f7a0 2023-10-30T14:27:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.817+0000 7f8e74ff9640 1 -- 172.21.15.46:0/3969647047 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e680bab10 con 0x7f8e7806f7a0 2023-10-30T14:27:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.817+0000 7f8e7e580640 1 -- 172.21.15.46:0/3969647047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e781ef8b0 con 0x7f8e7806f7a0 2023-10-30T14:27:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.817+0000 7f8e7e580640 1 -- 172.21.15.46:0/3969647047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e781f2ec0 con 0x7f8e7806f7a0 2023-10-30T14:27:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.817+0000 7f8e74ff9640 1 -- 172.21.15.46:0/3969647047 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e680bac70 con 0x7f8e7806f7a0 2023-10-30T14:27:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.817+0000 7f8e74ff9640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e640664a0 0x7f8e64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:27:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.817+0000 7f8e74ff9640 1 -- 172.21.15.46:0/3969647047 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f8e681230d0 con 0x7f8e7806f7a0 2023-10-30T14:27:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.817+0000 7f8e777fe640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e640664a0 0x7f8e64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:27:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.821+0000 7f8e777fe640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e640664a0 0x7f8e64068960 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f8e781ecfa0 tx=0x7f8e6c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:27:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.821+0000 7f8e74ff9640 1 -- 172.21.15.46:0/3969647047 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e680ef990 con 0x7f8e7806f7a0 2023-10-30T14:27:45.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:45.977+0000 7f8e7e580640 1 -- 172.21.15.46:0/3969647047 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8e78158650 con 0x7f8e640664a0 2023-10-30T14:27:46.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.001+0000 7f8e74ff9640 1 -- 172.21.15.46:0/3969647047 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+295065 (secure 0 0 0) 0x7f8e6c00be80 con 0x7f8e640664a0 2023-10-30T14:27:46.007 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:27:46.009 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T14:27:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.005+0000 7f8e7e580640 1 -- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e640664a0 msgr2=0x7f8e64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.005+0000 7f8e7e580640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e640664a0 0x7f8e64068960 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f8e781ecfa0 tx=0x7f8e6c008040 comp rx=0 tx=0).stop 2023-10-30T14:27:46.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.005+0000 7f8e7e580640 1 -- 172.21.15.46:0/3969647047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e7806f7a0 msgr2=0x7f8e7814cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:27:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.005+0000 7f8e7e580640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e7806f7a0 0x7f8e7814cd10 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f8e6801d3a0 tx=0x7f8e6801ba90 comp rx=0 tx=0).stop 2023-10-30T14:27:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.005+0000 7f8e7e580640 1 -- 172.21.15.46:0/3969647047 shutdown_connections 2023-10-30T14:27:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.005+0000 7f8e7e580640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f8e640664a0 0x7f8e64068960 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.005+0000 7f8e7e580640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e780c6bd0 0x7f8e780aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.005+0000 7f8e7e580640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8e78071110 0x7f8e781f3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.005+0000 7f8e7e580640 1 --2- 172.21.15.46:0/3969647047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e7806f7a0 0x7f8e7814cd10 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:27:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.005+0000 7f8e7e580640 1 -- 172.21.15.46:0/3969647047 >> 172.21.15.46:0/3969647047 conn(0x7f8e780c7910 msgr2=0x7f8e78156e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:27:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.005+0000 7f8e7e580640 1 -- 172.21.15.46:0/3969647047 shutdown_connections 2023-10-30T14:27:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:27:46.005+0000 7f8e7e580640 1 -- 172.21.15.46:0/3969647047 wait complete. 2023-10-30T14:27:46.030 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":2042,"stamp":"2023-10-30T14:27:44.151516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1285691727,"num_objects":3948,"num_object_clones":2,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":20157,"num_read_kb":14487318,"num_write":119403,"num_write_kb":3806032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13139950,"num_omap_keys":23453,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":174514,"ondisk_log_size":174514,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3945476,"kb_used_data":2522372,"kb_used_omap":12,"kb_used_meta":1423027,"kb_avail":751029244,"statfs":{"total":773094113280,"available":769053945856,"internally_reserved":0,"allocated":2582908928,"data_stored":2577659229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1457180216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004482"},"pg_stats":[{"pgid":"3.3f","version":"110'2193","reported_seq":5016,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:17.161673+0000","last_change":"2023-10-30T14:27:17.161738+0000","last_active":"2023-10-30T14:27:17.161673+0000","last_peered":"2023-10-30T14:27:17.161673+0000","last_clean":"2023-10-30T14:27:17.161673+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T14:27:17.161673+0000","last_undegraded":"2023-10-30T14:27:17.161673+0000","last_fullsized":"2023-10-30T14:27:17.161673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2193","last_scrub_stamp":"2023-10-30T14:27:17.161580+0000","last_deep_scrub":"110'2193","last_deep_scrub_stamp":"2023-10-30T14:27:17.161580+0000","last_clean_scrub_stamp":"2023-10-30T14:27:17.161580+0000","objects_scrubbed":32,"log_size":2193,"log_dups_size":0,"ondisk_log_size":2193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:00.032759+0000","scrub_duration":0.104386056,"objects_trimmed":0,"snaptrim_duration":0.84725695499999998,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":124,"num_read_kb":10662,"num_write":1276,"num_write_kb":15944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"110'2011","reported_seq":4900,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:06.287642+0000","last_change":"2023-10-30T14:27:06.287715+0000","last_active":"2023-10-30T14:27:06.287642+0000","last_peered":"2023-10-30T14:27:06.287642+0000","last_clean":"2023-10-30T14:27:06.287642+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T14:27:06.287642+0000","last_undegraded":"2023-10-30T14:27:06.287642+0000","last_fullsized":"2023-10-30T14:27:06.287642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2011","last_scrub_stamp":"2023-10-30T14:27:06.287549+0000","last_deep_scrub":"110'2011","last_deep_scrub_stamp":"2023-10-30T14:27:06.287549+0000","last_clean_scrub_stamp":"2023-10-30T14:27:06.287549+0000","objects_scrubbed":40,"log_size":2011,"log_dups_size":0,"ondisk_log_size":2011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:35:24.021901+0000","scrub_duration":0.052013427000000001,"objects_trimmed":0,"snaptrim_duration":0.406256491,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":133,"num_read_kb":21158,"num_write":1230,"num_write_kb":16620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"110'2124","reported_seq":4967,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:08.843187+0000","last_change":"2023-10-30T14:27:08.843240+0000","last_active":"2023-10-30T14:27:08.843187+0000","last_peered":"2023-10-30T14:27:08.843187+0000","last_clean":"2023-10-30T14:27:08.843187+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T14:27:08.843187+0000","last_undegraded":"2023-10-30T14:27:08.843187+0000","last_fullsized":"2023-10-30T14:27:08.843187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2124","last_scrub_stamp":"2023-10-30T14:27:08.843106+0000","last_deep_scrub":"110'2124","last_deep_scrub_stamp":"2023-10-30T14:27:08.843106+0000","last_clean_scrub_stamp":"2023-10-30T14:27:08.843106+0000","objects_scrubbed":29,"log_size":2124,"log_dups_size":0,"ondisk_log_size":2124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:20:20.621247+0000","scrub_duration":0.041951362999999998,"objects_trimmed":0,"snaptrim_duration":1.030559456,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":129,"num_read_kb":29825,"num_write":1270,"num_write_kb":28412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"110'2055","reported_seq":4905,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:07.083889+0000","last_change":"2023-10-30T14:27:07.083934+0000","last_active":"2023-10-30T14:27:07.083889+0000","last_peered":"2023-10-30T14:27:07.083889+0000","last_clean":"2023-10-30T14:27:07.083889+0000","last_became_active":"2023-10-30T13:24:00.704829+0000","last_became_peered":"2023-10-30T13:24:00.704829+0000","last_unstale":"2023-10-30T14:27:07.083889+0000","last_undegraded":"2023-10-30T14:27:07.083889+0000","last_fullsized":"2023-10-30T14:27:07.083889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2055","last_scrub_stamp":"2023-10-30T14:27:07.083807+0000","last_deep_scrub":"110'2055","last_deep_scrub_stamp":"2023-10-30T14:27:07.083807+0000","last_clean_scrub_stamp":"2023-10-30T14:27:07.083807+0000","objects_scrubbed":25,"log_size":2055,"log_dups_size":0,"ondisk_log_size":2055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:33.332360+0000","scrub_duration":0.034227107,"objects_trimmed":0,"snaptrim_duration":0.361435862,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":124,"num_read_kb":7939,"num_write":1184,"num_write_kb":24239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"109'2009","reported_seq":4868,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.903222+0000","last_change":"2023-10-30T14:26:34.648843+0000","last_active":"2023-10-30T14:26:39.903222+0000","last_peered":"2023-10-30T14:26:39.903222+0000","last_clean":"2023-10-30T14:26:39.903222+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T14:26:39.903222+0000","last_undegraded":"2023-10-30T14:26:39.903222+0000","last_fullsized":"2023-10-30T14:26:39.903222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2009,"log_dups_size":0,"ondisk_log_size":2009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.80853723300000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":41266,"num_write":1244,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"110'2251","reported_seq":5060,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:07.080029+0000","last_change":"2023-10-30T14:27:07.080296+0000","last_active":"2023-10-30T14:27:07.080029+0000","last_peered":"2023-10-30T14:27:07.080029+0000","last_clean":"2023-10-30T14:27:07.080029+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T14:27:07.080029+0000","last_undegraded":"2023-10-30T14:27:07.080029+0000","last_fullsized":"2023-10-30T14:27:07.080029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2251","last_scrub_stamp":"2023-10-30T14:27:07.079781+0000","last_deep_scrub":"110'2251","last_deep_scrub_stamp":"2023-10-30T14:27:07.079781+0000","last_clean_scrub_stamp":"2023-10-30T14:27:07.079781+0000","objects_scrubbed":42,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:47.980581+0000","scrub_duration":0.119106473,"objects_trimmed":0,"snaptrim_duration":0.94867293500000005,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":141,"num_read_kb":27429,"num_write":1290,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"110'2185","reported_seq":5044,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:12.196491+0000","last_change":"2023-10-30T14:27:12.196618+0000","last_active":"2023-10-30T14:27:12.196491+0000","last_peered":"2023-10-30T14:27:12.196491+0000","last_clean":"2023-10-30T14:27:12.196491+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T14:27:12.196491+0000","last_undegraded":"2023-10-30T14:27:12.196491+0000","last_fullsized":"2023-10-30T14:27:12.196491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2185","last_scrub_stamp":"2023-10-30T14:27:12.196356+0000","last_deep_scrub":"110'2185","last_deep_scrub_stamp":"2023-10-30T14:27:12.196356+0000","last_clean_scrub_stamp":"2023-10-30T14:27:12.196356+0000","objects_scrubbed":40,"log_size":2185,"log_dups_size":0,"ondisk_log_size":2185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:32.138507+0000","scrub_duration":0.11536558199999999,"objects_trimmed":0,"snaptrim_duration":0.71511574600000005,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":146,"num_read_kb":25110,"num_write":1316,"num_write_kb":20043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"110'2147","reported_seq":4969,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:20.681195+0000","last_change":"2023-10-30T14:27:20.681264+0000","last_active":"2023-10-30T14:27:20.681195+0000","last_peered":"2023-10-30T14:27:20.681195+0000","last_clean":"2023-10-30T14:27:20.681195+0000","last_became_active":"2023-10-30T13:24:00.989938+0000","last_became_peered":"2023-10-30T13:24:00.989938+0000","last_unstale":"2023-10-30T14:27:20.681195+0000","last_undegraded":"2023-10-30T14:27:20.681195+0000","last_fullsized":"2023-10-30T14:27:20.681195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2147","last_scrub_stamp":"2023-10-30T14:27:20.681102+0000","last_deep_scrub":"110'2147","last_deep_scrub_stamp":"2023-10-30T14:27:20.681102+0000","last_clean_scrub_stamp":"2023-10-30T14:27:20.681102+0000","objects_scrubbed":38,"log_size":2147,"log_dups_size":0,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:10:24.247496+0000","scrub_duration":0.050065129,"objects_trimmed":0,"snaptrim_duration":0.90238370000000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":124,"num_read_kb":25628,"num_write":1232,"num_write_kb":20540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"110'2178","reported_seq":5020,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:23.177373+0000","last_change":"2023-10-30T14:27:23.177584+0000","last_active":"2023-10-30T14:27:23.177373+0000","last_peered":"2023-10-30T14:27:23.177373+0000","last_clean":"2023-10-30T14:27:23.177373+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T14:27:23.177373+0000","last_undegraded":"2023-10-30T14:27:23.177373+0000","last_fullsized":"2023-10-30T14:27:23.177373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2178","last_scrub_stamp":"2023-10-30T14:27:23.177181+0000","last_deep_scrub":"110'2178","last_deep_scrub_stamp":"2023-10-30T14:27:23.177181+0000","last_clean_scrub_stamp":"2023-10-30T14:27:23.177181+0000","objects_scrubbed":41,"log_size":2178,"log_dups_size":0,"ondisk_log_size":2178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:10:06.566411+0000","scrub_duration":0.124464532,"objects_trimmed":0,"snaptrim_duration":0.81280227199999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":24548,"num_write":1310,"num_write_kb":19821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"110'2110","reported_seq":4983,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:08.423266+0000","last_change":"2023-10-30T14:27:08.423413+0000","last_active":"2023-10-30T14:27:08.423266+0000","last_peered":"2023-10-30T14:27:08.423266+0000","last_clean":"2023-10-30T14:27:08.423266+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T14:27:08.423266+0000","last_undegraded":"2023-10-30T14:27:08.423266+0000","last_fullsized":"2023-10-30T14:27:08.423266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2110","last_scrub_stamp":"2023-10-30T14:27:08.423062+0000","last_deep_scrub":"110'2110","last_deep_scrub_stamp":"2023-10-30T14:27:08.423062+0000","last_clean_scrub_stamp":"2023-10-30T14:27:08.423062+0000","objects_scrubbed":37,"log_size":2110,"log_dups_size":0,"ondisk_log_size":2110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:05:31.451500+0000","scrub_duration":0.11051124900000001,"objects_trimmed":0,"snaptrim_duration":0.53668512800000001,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":148,"num_read_kb":64678,"num_write":1273,"num_write_kb":32495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"110'2046","reported_seq":4931,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:13.410232+0000","last_change":"2023-10-30T14:27:13.410351+0000","last_active":"2023-10-30T14:27:13.410232+0000","last_peered":"2023-10-30T14:27:13.410232+0000","last_clean":"2023-10-30T14:27:13.410232+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T14:27:13.410232+0000","last_undegraded":"2023-10-30T14:27:13.410232+0000","last_fullsized":"2023-10-30T14:27:13.410232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2046","last_scrub_stamp":"2023-10-30T14:27:13.410056+0000","last_deep_scrub":"110'2046","last_deep_scrub_stamp":"2023-10-30T14:27:13.410056+0000","last_clean_scrub_stamp":"2023-10-30T14:27:13.410056+0000","objects_scrubbed":38,"log_size":2046,"log_dups_size":0,"ondisk_log_size":2046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:58.219570+0000","scrub_duration":0.113389077,"objects_trimmed":0,"snaptrim_duration":0.471346551,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":139,"num_read_kb":25550,"num_write":1254,"num_write_kb":16915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"110'2118","reported_seq":4973,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:58.807440+0000","last_change":"2023-10-30T14:26:58.807556+0000","last_active":"2023-10-30T14:26:58.807440+0000","last_peered":"2023-10-30T14:26:58.807440+0000","last_clean":"2023-10-30T14:26:58.807440+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T14:26:58.807440+0000","last_undegraded":"2023-10-30T14:26:58.807440+0000","last_fullsized":"2023-10-30T14:26:58.807440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2118","last_scrub_stamp":"2023-10-30T14:26:58.807287+0000","last_deep_scrub":"110'2118","last_deep_scrub_stamp":"2023-10-30T14:26:58.807287+0000","last_clean_scrub_stamp":"2023-10-30T14:26:58.807287+0000","objects_scrubbed":35,"log_size":2118,"log_dups_size":0,"ondisk_log_size":2118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:18.713125+0000","scrub_duration":0.105301005,"objects_trimmed":0,"snaptrim_duration":0.54183594099999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":142,"num_read_kb":52725,"num_write":1240,"num_write_kb":42463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"110'2255","reported_seq":5074,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:16.707217+0000","last_change":"2023-10-30T14:27:16.707278+0000","last_active":"2023-10-30T14:27:16.707217+0000","last_peered":"2023-10-30T14:27:16.707217+0000","last_clean":"2023-10-30T14:27:16.707217+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T14:27:16.707217+0000","last_undegraded":"2023-10-30T14:27:16.707217+0000","last_fullsized":"2023-10-30T14:27:16.707217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2255","last_scrub_stamp":"2023-10-30T14:27:16.707128+0000","last_deep_scrub":"110'2255","last_deep_scrub_stamp":"2023-10-30T14:27:16.707128+0000","last_clean_scrub_stamp":"2023-10-30T14:27:16.707128+0000","objects_scrubbed":32,"log_size":2255,"log_dups_size":0,"ondisk_log_size":2255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:55.264805+0000","scrub_duration":0.077397046999999997,"objects_trimmed":0,"snaptrim_duration":1.038919041,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":46739,"num_write":1326,"num_write_kb":33562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"110'2198","reported_seq":5059,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:04.874664+0000","last_change":"2023-10-30T14:27:04.874740+0000","last_active":"2023-10-30T14:27:04.874664+0000","last_peered":"2023-10-30T14:27:04.874664+0000","last_clean":"2023-10-30T14:27:04.874664+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T14:27:04.874664+0000","last_undegraded":"2023-10-30T14:27:04.874664+0000","last_fullsized":"2023-10-30T14:27:04.874664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2198","last_scrub_stamp":"2023-10-30T14:27:04.874577+0000","last_deep_scrub":"110'2198","last_deep_scrub_stamp":"2023-10-30T14:27:04.874577+0000","last_clean_scrub_stamp":"2023-10-30T14:27:04.874577+0000","objects_scrubbed":47,"log_size":2198,"log_dups_size":0,"ondisk_log_size":2198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:18:28.985655+0000","scrub_duration":0.059905817,"objects_trimmed":0,"snaptrim_duration":0.97549857799999995,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":149,"num_read_kb":13007,"num_write":1312,"num_write_kb":16233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"110'2306","reported_seq":5141,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:10.340370+0000","last_change":"2023-10-30T14:27:10.340471+0000","last_active":"2023-10-30T14:27:10.340370+0000","last_peered":"2023-10-30T14:27:10.340370+0000","last_clean":"2023-10-30T14:27:10.340370+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T14:27:10.340370+0000","last_undegraded":"2023-10-30T14:27:10.340370+0000","last_fullsized":"2023-10-30T14:27:10.340370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2306","last_scrub_stamp":"2023-10-30T14:27:10.340294+0000","last_deep_scrub":"110'2306","last_deep_scrub_stamp":"2023-10-30T14:27:10.340294+0000","last_clean_scrub_stamp":"2023-10-30T14:27:10.340294+0000","objects_scrubbed":39,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:52:13.783347+0000","scrub_duration":0.101464054,"objects_trimmed":0,"snaptrim_duration":0.432871758,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":161,"num_read_kb":14756,"num_write":1377,"num_write_kb":22909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"110'2120","reported_seq":4990,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:12.050653+0000","last_change":"2023-10-30T14:27:12.050782+0000","last_active":"2023-10-30T14:27:12.050653+0000","last_peered":"2023-10-30T14:27:12.050653+0000","last_clean":"2023-10-30T14:27:12.050653+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T14:27:12.050653+0000","last_undegraded":"2023-10-30T14:27:12.050653+0000","last_fullsized":"2023-10-30T14:27:12.050653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2120","last_scrub_stamp":"2023-10-30T14:27:12.050452+0000","last_deep_scrub":"110'2120","last_deep_scrub_stamp":"2023-10-30T14:27:12.050452+0000","last_clean_scrub_stamp":"2023-10-30T14:27:12.050452+0000","objects_scrubbed":30,"log_size":2120,"log_dups_size":0,"ondisk_log_size":2120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:30:59.575797+0000","scrub_duration":0.088091438999999994,"objects_trimmed":0,"snaptrim_duration":0.83845289999999995,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":149,"num_read_kb":40211,"num_write":1268,"num_write_kb":23485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"110'2389","reported_seq":5214,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:13.135193+0000","last_change":"2023-10-30T14:27:13.135363+0000","last_active":"2023-10-30T14:27:13.135193+0000","last_peered":"2023-10-30T14:27:13.135193+0000","last_clean":"2023-10-30T14:27:13.135193+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T14:27:13.135193+0000","last_undegraded":"2023-10-30T14:27:13.135193+0000","last_fullsized":"2023-10-30T14:27:13.135193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2389","last_scrub_stamp":"2023-10-30T14:27:13.135025+0000","last_deep_scrub":"110'2389","last_deep_scrub_stamp":"2023-10-30T14:27:13.135025+0000","last_clean_scrub_stamp":"2023-10-30T14:27:13.135025+0000","objects_scrubbed":41,"log_size":2389,"log_dups_size":0,"ondisk_log_size":2389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:16.442511+0000","scrub_duration":0.126176548,"objects_trimmed":0,"snaptrim_duration":1.109948846,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":170,"num_read_kb":39932,"num_write":1446,"num_write_kb":30153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"110'2027","reported_seq":4894,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:59.058458+0000","last_change":"2023-10-30T14:26:59.058509+0000","last_active":"2023-10-30T14:26:59.058458+0000","last_peered":"2023-10-30T14:26:59.058458+0000","last_clean":"2023-10-30T14:26:59.058458+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T14:26:59.058458+0000","last_undegraded":"2023-10-30T14:26:59.058458+0000","last_fullsized":"2023-10-30T14:26:59.058458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2027","last_scrub_stamp":"2023-10-30T14:26:59.058377+0000","last_deep_scrub":"110'2027","last_deep_scrub_stamp":"2023-10-30T14:26:59.058377+0000","last_clean_scrub_stamp":"2023-10-30T14:26:59.058377+0000","objects_scrubbed":31,"log_size":2027,"log_dups_size":0,"ondisk_log_size":2027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:24:17.547389+0000","scrub_duration":0.077206206999999999,"objects_trimmed":0,"snaptrim_duration":0.34246037800000001,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":137,"num_read_kb":13706,"num_write":1196,"num_write_kb":12790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"110'2128","reported_seq":4972,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:29.091001+0000","last_change":"2023-10-30T14:27:29.091142+0000","last_active":"2023-10-30T14:27:29.091001+0000","last_peered":"2023-10-30T14:27:29.091001+0000","last_clean":"2023-10-30T14:27:29.091001+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T14:27:29.091001+0000","last_undegraded":"2023-10-30T14:27:29.091001+0000","last_fullsized":"2023-10-30T14:27:29.091001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2128","last_scrub_stamp":"2023-10-30T14:27:29.090772+0000","last_deep_scrub":"110'2128","last_deep_scrub_stamp":"2023-10-30T14:27:29.090772+0000","last_clean_scrub_stamp":"2023-10-30T14:27:29.090772+0000","objects_scrubbed":33,"log_size":2128,"log_dups_size":0,"ondisk_log_size":2128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:38:37.170898+0000","scrub_duration":0.089460595000000004,"objects_trimmed":0,"snaptrim_duration":0.76369308499999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":125,"num_read_kb":30178,"num_write":1282,"num_write_kb":24413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"110'2253","reported_seq":5094,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.434812+0000","last_change":"2023-10-30T14:26:57.434908+0000","last_active":"2023-10-30T14:26:57.434812+0000","last_peered":"2023-10-30T14:26:57.434812+0000","last_clean":"2023-10-30T14:26:57.434812+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T14:26:57.434812+0000","last_undegraded":"2023-10-30T14:26:57.434812+0000","last_fullsized":"2023-10-30T14:26:57.434812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2253","last_scrub_stamp":"2023-10-30T14:26:57.434694+0000","last_deep_scrub":"110'2253","last_deep_scrub_stamp":"2023-10-30T14:26:57.434694+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.434694+0000","objects_scrubbed":38,"log_size":2253,"log_dups_size":0,"ondisk_log_size":2253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:26:38.794241+0000","scrub_duration":0.122965276,"objects_trimmed":0,"snaptrim_duration":0.46668427699999998,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":148,"num_read_kb":30111,"num_write":1332,"num_write_kb":26597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"110'2256","reported_seq":5056,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:11.917935+0000","last_change":"2023-10-30T14:27:11.918008+0000","last_active":"2023-10-30T14:27:11.917935+0000","last_peered":"2023-10-30T14:27:11.917935+0000","last_clean":"2023-10-30T14:27:11.917935+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T14:27:11.917935+0000","last_undegraded":"2023-10-30T14:27:11.917935+0000","last_fullsized":"2023-10-30T14:27:11.917935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2256","last_scrub_stamp":"2023-10-30T14:27:11.917846+0000","last_deep_scrub":"110'2256","last_deep_scrub_stamp":"2023-10-30T14:27:11.917846+0000","last_clean_scrub_stamp":"2023-10-30T14:27:11.917846+0000","objects_scrubbed":34,"log_size":2256,"log_dups_size":0,"ondisk_log_size":2256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:28.375501+0000","scrub_duration":0.083664374,"objects_trimmed":0,"snaptrim_duration":1.008362585,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":127,"num_read_kb":10449,"num_write":1298,"num_write_kb":11653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"110'2330","reported_seq":5142,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:22.210913+0000","last_change":"2023-10-30T14:27:22.211067+0000","last_active":"2023-10-30T14:27:22.210913+0000","last_peered":"2023-10-30T14:27:22.210913+0000","last_clean":"2023-10-30T14:27:22.210913+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T14:27:22.210913+0000","last_undegraded":"2023-10-30T14:27:22.210913+0000","last_fullsized":"2023-10-30T14:27:22.210913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2330","last_scrub_stamp":"2023-10-30T14:27:22.210723+0000","last_deep_scrub":"110'2330","last_deep_scrub_stamp":"2023-10-30T14:27:22.210723+0000","last_clean_scrub_stamp":"2023-10-30T14:27:22.210723+0000","objects_scrubbed":48,"log_size":2330,"log_dups_size":0,"ondisk_log_size":2330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:38.233096+0000","scrub_duration":0.14657436700000001,"objects_trimmed":0,"snaptrim_duration":0.94381626399999996,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":144,"num_read_kb":30506,"num_write":1376,"num_write_kb":26822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"110'2155","reported_seq":5129,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:27.727917+0000","last_change":"2023-10-30T14:27:27.727989+0000","last_active":"2023-10-30T14:27:27.727917+0000","last_peered":"2023-10-30T14:27:27.727917+0000","last_clean":"2023-10-30T14:27:27.727917+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T14:27:27.727917+0000","last_undegraded":"2023-10-30T14:27:27.727917+0000","last_fullsized":"2023-10-30T14:27:27.727917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2155","last_scrub_stamp":"2023-10-30T14:27:27.727838+0000","last_deep_scrub":"110'2155","last_deep_scrub_stamp":"2023-10-30T14:27:27.727838+0000","last_clean_scrub_stamp":"2023-10-30T14:27:27.727838+0000","objects_scrubbed":38,"log_size":2155,"log_dups_size":0,"ondisk_log_size":2155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:27:08.100066+0000","scrub_duration":0.081191790999999999,"objects_trimmed":0,"snaptrim_duration":0.98440143999999996,"stat_sum":{"num_bytes":6,"num_objects":38,"num_object_clones":2,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":257,"num_read_kb":42146,"num_write":1297,"num_write_kb":25266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"110'2212","reported_seq":5085,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:05.052361+0000","last_change":"2023-10-30T14:27:05.052501+0000","last_active":"2023-10-30T14:27:05.052361+0000","last_peered":"2023-10-30T14:27:05.052361+0000","last_clean":"2023-10-30T14:27:05.052361+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T14:27:05.052361+0000","last_undegraded":"2023-10-30T14:27:05.052361+0000","last_fullsized":"2023-10-30T14:27:05.052361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2212","last_scrub_stamp":"2023-10-30T14:27:05.052163+0000","last_deep_scrub":"110'2212","last_deep_scrub_stamp":"2023-10-30T14:27:05.052163+0000","last_clean_scrub_stamp":"2023-10-30T14:27:05.052163+0000","objects_scrubbed":34,"log_size":2212,"log_dups_size":0,"ondisk_log_size":2212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:40.396490+0000","scrub_duration":0.091462254000000007,"objects_trimmed":0,"snaptrim_duration":0.89723890399999995,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":164,"num_read_kb":14461,"num_write":1342,"num_write_kb":12663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"110'2090","reported_seq":4941,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:07.710584+0000","last_change":"2023-10-30T14:27:07.710637+0000","last_active":"2023-10-30T14:27:07.710584+0000","last_peered":"2023-10-30T14:27:07.710584+0000","last_clean":"2023-10-30T14:27:07.710584+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T14:27:07.710584+0000","last_undegraded":"2023-10-30T14:27:07.710584+0000","last_fullsized":"2023-10-30T14:27:07.710584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2090","last_scrub_stamp":"2023-10-30T14:27:07.710502+0000","last_deep_scrub":"110'2090","last_deep_scrub_stamp":"2023-10-30T14:27:07.710502+0000","last_clean_scrub_stamp":"2023-10-30T14:27:07.710502+0000","objects_scrubbed":33,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:39.764424+0000","scrub_duration":0.081297077999999995,"objects_trimmed":0,"snaptrim_duration":1.0495642949999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":133,"num_read_kb":25337,"num_write":1220,"num_write_kb":15637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"110'2181","reported_seq":5069,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:03.420662+0000","last_change":"2023-10-30T14:27:03.420832+0000","last_active":"2023-10-30T14:27:03.420662+0000","last_peered":"2023-10-30T14:27:03.420662+0000","last_clean":"2023-10-30T14:27:03.420662+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T14:27:03.420662+0000","last_undegraded":"2023-10-30T14:27:03.420662+0000","last_fullsized":"2023-10-30T14:27:03.420662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2181","last_scrub_stamp":"2023-10-30T14:27:03.420503+0000","last_deep_scrub":"110'2181","last_deep_scrub_stamp":"2023-10-30T14:27:03.420503+0000","last_clean_scrub_stamp":"2023-10-30T14:27:03.420503+0000","objects_scrubbed":40,"log_size":2181,"log_dups_size":0,"ondisk_log_size":2181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:36:16.083580+0000","scrub_duration":0.11761008000000001,"objects_trimmed":0,"snaptrim_duration":0.504458776,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":160,"num_read_kb":14034,"num_write":1340,"num_write_kb":12359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"110'2127","reported_seq":4987,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:03.326222+0000","last_change":"2023-10-30T14:27:03.326282+0000","last_active":"2023-10-30T14:27:03.326222+0000","last_peered":"2023-10-30T14:27:03.326222+0000","last_clean":"2023-10-30T14:27:03.326222+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T14:27:03.326222+0000","last_undegraded":"2023-10-30T14:27:03.326222+0000","last_fullsized":"2023-10-30T14:27:03.326222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2127","last_scrub_stamp":"2023-10-30T14:27:03.326140+0000","last_deep_scrub":"110'2127","last_deep_scrub_stamp":"2023-10-30T14:27:03.326140+0000","last_clean_scrub_stamp":"2023-10-30T14:27:03.326140+0000","objects_scrubbed":38,"log_size":2127,"log_dups_size":0,"ondisk_log_size":2127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:46:52.407641+0000","scrub_duration":0.050416410000000002,"objects_trimmed":0,"snaptrim_duration":0.34262146900000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":128,"num_read_kb":22518,"num_write":1277,"num_write_kb":32107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"110'2068","reported_seq":4908,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.691221+0000","last_change":"2023-10-30T14:26:52.691273+0000","last_active":"2023-10-30T14:26:52.691221+0000","last_peered":"2023-10-30T14:26:52.691221+0000","last_clean":"2023-10-30T14:26:52.691221+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T14:26:52.691221+0000","last_undegraded":"2023-10-30T14:26:52.691221+0000","last_fullsized":"2023-10-30T14:26:52.691221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2068","last_scrub_stamp":"2023-10-30T14:26:52.691147+0000","last_deep_scrub":"110'2068","last_deep_scrub_stamp":"2023-10-30T14:26:52.691147+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.691147+0000","objects_scrubbed":38,"log_size":2068,"log_dups_size":0,"ondisk_log_size":2068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:08.320506+0000","scrub_duration":0.079048169000000001,"objects_trimmed":0,"snaptrim_duration":1.068284585,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":129,"num_read_kb":17081,"num_write":1202,"num_write_kb":15468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"110'2178","reported_seq":5011,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:16.911734+0000","last_change":"2023-10-30T14:27:16.911796+0000","last_active":"2023-10-30T14:27:16.911734+0000","last_peered":"2023-10-30T14:27:16.911734+0000","last_clean":"2023-10-30T14:27:16.911734+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T14:27:16.911734+0000","last_undegraded":"2023-10-30T14:27:16.911734+0000","last_fullsized":"2023-10-30T14:27:16.911734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2178","last_scrub_stamp":"2023-10-30T14:27:16.911656+0000","last_deep_scrub":"110'2178","last_deep_scrub_stamp":"2023-10-30T14:27:16.911656+0000","last_clean_scrub_stamp":"2023-10-30T14:27:16.911656+0000","objects_scrubbed":38,"log_size":2178,"log_dups_size":0,"ondisk_log_size":2178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:38:53.941111+0000","scrub_duration":0.046150844000000003,"objects_trimmed":0,"snaptrim_duration":1.030536272,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":125,"num_read_kb":13655,"num_write":1323,"num_write_kb":17357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"110'2124","reported_seq":5004,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.119325+0000","last_change":"2023-10-30T14:26:57.119416+0000","last_active":"2023-10-30T14:26:57.119325+0000","last_peered":"2023-10-30T14:26:57.119325+0000","last_clean":"2023-10-30T14:26:57.119325+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T14:26:57.119325+0000","last_undegraded":"2023-10-30T14:26:57.119325+0000","last_fullsized":"2023-10-30T14:26:57.119325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2124","last_scrub_stamp":"2023-10-30T14:26:57.119213+0000","last_deep_scrub":"110'2124","last_deep_scrub_stamp":"2023-10-30T14:26:57.119213+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.119213+0000","objects_scrubbed":38,"log_size":2124,"log_dups_size":0,"ondisk_log_size":2124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:10.120593+0000","scrub_duration":0.076576980000000003,"objects_trimmed":0,"snaptrim_duration":0.98488935700000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":140,"num_read_kb":17880,"num_write":1311,"num_write_kb":13902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"110'2418","reported_seq":5244,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.085217+0000","last_change":"2023-10-30T14:27:00.085296+0000","last_active":"2023-10-30T14:27:00.085217+0000","last_peered":"2023-10-30T14:27:00.085217+0000","last_clean":"2023-10-30T14:27:00.085217+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T14:27:00.085217+0000","last_undegraded":"2023-10-30T14:27:00.085217+0000","last_fullsized":"2023-10-30T14:27:00.085217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2418","last_scrub_stamp":"2023-10-30T14:27:00.085138+0000","last_deep_scrub":"110'2418","last_deep_scrub_stamp":"2023-10-30T14:27:00.085138+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.085138+0000","objects_scrubbed":41,"log_size":2418,"log_dups_size":0,"ondisk_log_size":2418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:00.491199+0000","scrub_duration":0.10095144,"objects_trimmed":0,"snaptrim_duration":0.30670995400000001,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":165,"num_read_kb":40193,"num_write":1443,"num_write_kb":35290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"110'2344","reported_seq":5176,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:02.425665+0000","last_change":"2023-10-30T14:27:02.425824+0000","last_active":"2023-10-30T14:27:02.425665+0000","last_peered":"2023-10-30T14:27:02.425665+0000","last_clean":"2023-10-30T14:27:02.425665+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T14:27:02.425665+0000","last_undegraded":"2023-10-30T14:27:02.425665+0000","last_fullsized":"2023-10-30T14:27:02.425665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2344","last_scrub_stamp":"2023-10-30T14:27:02.425448+0000","last_deep_scrub":"110'2344","last_deep_scrub_stamp":"2023-10-30T14:27:02.425448+0000","last_clean_scrub_stamp":"2023-10-30T14:27:02.425448+0000","objects_scrubbed":41,"log_size":2344,"log_dups_size":0,"ondisk_log_size":2344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:18.221218+0000","scrub_duration":0.11536735100000001,"objects_trimmed":0,"snaptrim_duration":0.42993166100000002,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":168,"num_read_kb":24860,"num_write":1383,"num_write_kb":20782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"110'2147","reported_seq":5000,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:50.868519+0000","last_change":"2023-10-30T14:26:50.868736+0000","last_active":"2023-10-30T14:26:50.868519+0000","last_peered":"2023-10-30T14:26:50.868519+0000","last_clean":"2023-10-30T14:26:50.868519+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T14:26:50.868519+0000","last_undegraded":"2023-10-30T14:26:50.868519+0000","last_fullsized":"2023-10-30T14:26:50.868519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2147","last_scrub_stamp":"2023-10-30T14:26:50.868292+0000","last_deep_scrub":"110'2147","last_deep_scrub_stamp":"2023-10-30T14:26:50.868292+0000","last_clean_scrub_stamp":"2023-10-30T14:26:50.868292+0000","objects_scrubbed":34,"log_size":2147,"log_dups_size":0,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:21.454895+0000","scrub_duration":0.095674789999999996,"objects_trimmed":0,"snaptrim_duration":0.59246732999999996,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":146,"num_read_kb":30802,"num_write":1268,"num_write_kb":21419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"110'1078","reported_seq":1396,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.791196+0000","last_change":"2023-10-30T14:26:42.791248+0000","last_active":"2023-10-30T14:26:42.791196+0000","last_peered":"2023-10-30T14:26:42.791196+0000","last_clean":"2023-10-30T14:26:42.791196+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T14:26:42.791196+0000","last_undegraded":"2023-10-30T14:26:42.791196+0000","last_fullsized":"2023-10-30T14:26:42.791196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1078","last_scrub_stamp":"2023-10-30T14:26:42.791118+0000","last_deep_scrub":"110'1078","last_deep_scrub_stamp":"2023-10-30T14:26:42.791118+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.791118+0000","objects_scrubbed":50,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:33.250671+0000","scrub_duration":0.285611593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497506,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":222,"num_read_kb":501605,"num_write":1094,"num_write_kb":95785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":358881,"num_omap_keys":701,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"110'2190","reported_seq":5007,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:10.871699+0000","last_change":"2023-10-30T14:27:10.871788+0000","last_active":"2023-10-30T14:27:10.871699+0000","last_peered":"2023-10-30T14:27:10.871699+0000","last_clean":"2023-10-30T14:27:10.871699+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T14:27:10.871699+0000","last_undegraded":"2023-10-30T14:27:10.871699+0000","last_fullsized":"2023-10-30T14:27:10.871699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2190","last_scrub_stamp":"2023-10-30T14:27:10.871590+0000","last_deep_scrub":"110'2190","last_deep_scrub_stamp":"2023-10-30T14:27:10.871590+0000","last_clean_scrub_stamp":"2023-10-30T14:27:10.871590+0000","objects_scrubbed":35,"log_size":2190,"log_dups_size":0,"ondisk_log_size":2190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:54:03.123430+0000","scrub_duration":0.083573112000000005,"objects_trimmed":0,"snaptrim_duration":1.1434485830000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":17524,"num_write":1294,"num_write_kb":15816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"110'1745","reported_seq":2190,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.486755+0000","last_change":"2023-10-30T14:26:54.486806+0000","last_active":"2023-10-30T14:26:54.486755+0000","last_peered":"2023-10-30T14:26:54.486755+0000","last_clean":"2023-10-30T14:26:54.486755+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T14:26:54.486755+0000","last_undegraded":"2023-10-30T14:26:54.486755+0000","last_fullsized":"2023-10-30T14:26:54.486755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1745","last_scrub_stamp":"2023-10-30T14:26:54.486663+0000","last_deep_scrub":"110'1745","last_deep_scrub_stamp":"2023-10-30T14:26:54.486663+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.486663+0000","objects_scrubbed":52,"log_size":1745,"log_dups_size":0,"ondisk_log_size":1745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:54.960607+0000","scrub_duration":0.19504682200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29694842,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":392,"num_read_kb":410068,"num_write":1792,"num_write_kb":89526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":545048,"num_omap_keys":1059,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"110'1609","reported_seq":2975,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.816492+0000","last_change":"2023-10-30T14:26:46.816578+0000","last_active":"2023-10-30T14:26:46.816492+0000","last_peered":"2023-10-30T14:26:46.816492+0000","last_clean":"2023-10-30T14:26:46.816492+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T14:26:46.816492+0000","last_undegraded":"2023-10-30T14:26:46.816492+0000","last_fullsized":"2023-10-30T14:26:46.816492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1609","last_scrub_stamp":"2023-10-30T14:26:46.816372+0000","last_deep_scrub":"110'1609","last_deep_scrub_stamp":"2023-10-30T14:26:46.816372+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.816372+0000","objects_scrubbed":52,"log_size":1609,"log_dups_size":0,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:04:35.564593+0000","scrub_duration":0.28844654400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60855928,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":1296,"num_read_kb":556671,"num_write":1596,"num_write_kb":94675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":405715,"num_omap_keys":625,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"110'2379","reported_seq":5227,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.714048+0000","last_change":"2023-10-30T14:26:53.714120+0000","last_active":"2023-10-30T14:26:53.714048+0000","last_peered":"2023-10-30T14:26:53.714048+0000","last_clean":"2023-10-30T14:26:53.714048+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T14:26:53.714048+0000","last_undegraded":"2023-10-30T14:26:53.714048+0000","last_fullsized":"2023-10-30T14:26:53.714048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2379","last_scrub_stamp":"2023-10-30T14:26:53.713939+0000","last_deep_scrub":"110'2379","last_deep_scrub_stamp":"2023-10-30T14:26:53.713939+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.713939+0000","objects_scrubbed":52,"log_size":2379,"log_dups_size":0,"ondisk_log_size":2379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:15.405210+0000","scrub_duration":0.058831420000000002,"objects_trimmed":0,"snaptrim_duration":0.89032229399999996,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":161,"num_read_kb":17698,"num_write":1449,"num_write_kb":26122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"110'1032","reported_seq":1390,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:55.223361+0000","last_change":"2023-10-30T14:26:55.223448+0000","last_active":"2023-10-30T14:26:55.223361+0000","last_peered":"2023-10-30T14:26:55.223361+0000","last_clean":"2023-10-30T14:26:55.223361+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T14:26:55.223361+0000","last_undegraded":"2023-10-30T14:26:55.223361+0000","last_fullsized":"2023-10-30T14:26:55.223361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1032","last_scrub_stamp":"2023-10-30T14:26:55.223231+0000","last_deep_scrub":"110'1032","last_deep_scrub_stamp":"2023-10-30T14:26:55.223231+0000","last_clean_scrub_stamp":"2023-10-30T14:26:55.223231+0000","objects_scrubbed":57,"log_size":1032,"log_dups_size":0,"ondisk_log_size":1032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:18:36.091837+0000","scrub_duration":0.26442538300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54010089,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":312,"num_read_kb":563254,"num_write":1082,"num_write_kb":96418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":706737,"num_omap_keys":1228,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"110'2179","reported_seq":5047,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.255870+0000","last_change":"2023-10-30T14:27:00.255921+0000","last_active":"2023-10-30T14:27:00.255870+0000","last_peered":"2023-10-30T14:27:00.255870+0000","last_clean":"2023-10-30T14:27:00.255870+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T14:27:00.255870+0000","last_undegraded":"2023-10-30T14:27:00.255870+0000","last_fullsized":"2023-10-30T14:27:00.255870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2179","last_scrub_stamp":"2023-10-30T14:27:00.255786+0000","last_deep_scrub":"110'2179","last_deep_scrub_stamp":"2023-10-30T14:27:00.255786+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.255786+0000","objects_scrubbed":30,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:01.714268+0000","scrub_duration":0.037354261999999999,"objects_trimmed":0,"snaptrim_duration":0.36920391600000002,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":152,"num_read_kb":51949,"num_write":1321,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"110'865","reported_seq":1210,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:56.576908+0000","last_change":"2023-10-30T14:26:56.577007+0000","last_active":"2023-10-30T14:26:56.576908+0000","last_peered":"2023-10-30T14:26:56.576908+0000","last_clean":"2023-10-30T14:26:56.576908+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T14:26:56.576908+0000","last_undegraded":"2023-10-30T14:26:56.576908+0000","last_fullsized":"2023-10-30T14:26:56.576908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'865","last_scrub_stamp":"2023-10-30T14:26:56.576765+0000","last_deep_scrub":"110'865","last_deep_scrub_stamp":"2023-10-30T14:26:56.576765+0000","last_clean_scrub_stamp":"2023-10-30T14:26:56.576765+0000","objects_scrubbed":64,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:40.939877+0000","scrub_duration":0.28082476899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943130,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":319,"num_read_kb":417394,"num_write":917,"num_write_kb":71614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":395317,"num_omap_keys":751,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"110'2477","reported_seq":5216,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:58.107565+0000","last_change":"2023-10-30T14:26:58.107765+0000","last_active":"2023-10-30T14:26:58.107565+0000","last_peered":"2023-10-30T14:26:58.107565+0000","last_clean":"2023-10-30T14:26:58.107565+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T14:26:58.107565+0000","last_undegraded":"2023-10-30T14:26:58.107565+0000","last_fullsized":"2023-10-30T14:26:58.107565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2477","last_scrub_stamp":"2023-10-30T14:26:58.107390+0000","last_deep_scrub":"110'2477","last_deep_scrub_stamp":"2023-10-30T14:26:58.107390+0000","last_clean_scrub_stamp":"2023-10-30T14:26:58.107390+0000","objects_scrubbed":23,"log_size":2477,"log_dups_size":0,"ondisk_log_size":2477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:29.902228+0000","scrub_duration":0.078694307000000005,"objects_trimmed":0,"snaptrim_duration":1.0302270280000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":152,"num_read_kb":10055,"num_write":1351,"num_write_kb":13264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"110'901","reported_seq":1217,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:51.209972+0000","last_change":"2023-10-30T14:26:51.210091+0000","last_active":"2023-10-30T14:26:51.209972+0000","last_peered":"2023-10-30T14:26:51.209972+0000","last_clean":"2023-10-30T14:26:51.209972+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T14:26:51.209972+0000","last_undegraded":"2023-10-30T14:26:51.209972+0000","last_fullsized":"2023-10-30T14:26:51.209972+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'901","last_scrub_stamp":"2023-10-30T14:26:51.209745+0000","last_deep_scrub":"110'901","last_deep_scrub_stamp":"2023-10-30T14:26:51.209745+0000","last_clean_scrub_stamp":"2023-10-30T14:26:51.209745+0000","objects_scrubbed":52,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:29:33.421380+0000","scrub_duration":0.228163176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":291,"num_read_kb":402257,"num_write":947,"num_write_kb":71494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":417597,"num_omap_keys":787,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"110'2117","reported_seq":4971,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:01.839258+0000","last_change":"2023-10-30T14:27:01.839312+0000","last_active":"2023-10-30T14:27:01.839258+0000","last_peered":"2023-10-30T14:27:01.839258+0000","last_clean":"2023-10-30T14:27:01.839258+0000","last_became_active":"2023-10-30T13:24:00.990019+0000","last_became_peered":"2023-10-30T13:24:00.990019+0000","last_unstale":"2023-10-30T14:27:01.839258+0000","last_undegraded":"2023-10-30T14:27:01.839258+0000","last_fullsized":"2023-10-30T14:27:01.839258+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2117","last_scrub_stamp":"2023-10-30T14:27:01.839175+0000","last_deep_scrub":"110'2117","last_deep_scrub_stamp":"2023-10-30T14:27:01.839175+0000","last_clean_scrub_stamp":"2023-10-30T14:27:01.839175+0000","objects_scrubbed":32,"log_size":2117,"log_dups_size":0,"ondisk_log_size":2117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:37:53.712537+0000","scrub_duration":0.043087937999999999,"objects_trimmed":0,"snaptrim_duration":1.1322467139999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":132,"num_read_kb":30385,"num_write":1263,"num_write_kb":18224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"110'1029","reported_seq":1299,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:55.421984+0000","last_change":"2023-10-30T14:26:55.422041+0000","last_active":"2023-10-30T14:26:55.421984+0000","last_peered":"2023-10-30T14:26:55.421984+0000","last_clean":"2023-10-30T14:26:55.421984+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T14:26:55.421984+0000","last_undegraded":"2023-10-30T14:26:55.421984+0000","last_fullsized":"2023-10-30T14:26:55.421984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1029","last_scrub_stamp":"2023-10-30T14:26:55.421890+0000","last_deep_scrub":"110'1029","last_deep_scrub_stamp":"2023-10-30T14:26:55.421890+0000","last_clean_scrub_stamp":"2023-10-30T14:26:55.421890+0000","objects_scrubbed":35,"log_size":1029,"log_dups_size":0,"ondisk_log_size":1029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:34:36.976816+0000","scrub_duration":0.131590227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":208,"num_read_kb":224450,"num_write":1057,"num_write_kb":46972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":295749,"num_omap_keys":457,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"110'2177","reported_seq":5017,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:48.799169+0000","last_change":"2023-10-30T14:26:48.799265+0000","last_active":"2023-10-30T14:26:48.799169+0000","last_peered":"2023-10-30T14:26:48.799169+0000","last_clean":"2023-10-30T14:26:48.799169+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T14:26:48.799169+0000","last_undegraded":"2023-10-30T14:26:48.799169+0000","last_fullsized":"2023-10-30T14:26:48.799169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2177","last_scrub_stamp":"2023-10-30T14:26:48.799002+0000","last_deep_scrub":"110'2177","last_deep_scrub_stamp":"2023-10-30T14:26:48.799002+0000","last_clean_scrub_stamp":"2023-10-30T14:26:48.799002+0000","objects_scrubbed":28,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:56:02.403730+0000","scrub_duration":0.092776059999999994,"objects_trimmed":0,"snaptrim_duration":0.58089010799999996,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":141,"num_read_kb":34327,"num_write":1294,"num_write_kb":27493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"110'1040","reported_seq":1364,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:47.590330+0000","last_change":"2023-10-30T14:26:47.590330+0000","last_active":"2023-10-30T14:26:47.590330+0000","last_peered":"2023-10-30T14:26:47.590330+0000","last_clean":"2023-10-30T14:26:47.590330+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T14:26:47.590330+0000","last_undegraded":"2023-10-30T14:26:47.590330+0000","last_fullsized":"2023-10-30T14:26:47.590330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1039","last_scrub_stamp":"2023-10-30T14:26:47.589869+0000","last_deep_scrub":"110'1039","last_deep_scrub_stamp":"2023-10-30T14:26:47.589869+0000","last_clean_scrub_stamp":"2023-10-30T14:26:47.589869+0000","objects_scrubbed":46,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:04:37.337840+0000","scrub_duration":0.243642899,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":290,"num_read_kb":355466,"num_write":1084,"num_write_kb":67491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":426651,"num_omap_keys":733,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"109'2165","reported_seq":4985,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:39.904020+0000","last_change":"2023-10-30T14:26:34.585946+0000","last_active":"2023-10-30T14:26:39.904020+0000","last_peered":"2023-10-30T14:26:39.904020+0000","last_clean":"2023-10-30T14:26:39.904020+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T14:26:39.904020+0000","last_undegraded":"2023-10-30T14:26:39.904020+0000","last_fullsized":"2023-10-30T14:26:39.904020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:23:18.926892+0000","last_clean_scrub_stamp":"2023-10-30T13:23:18.926892+0000","objects_scrubbed":0,"log_size":2165,"log_dups_size":0,"ondisk_log_size":2165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.74553484199999998,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":143,"num_read_kb":29369,"num_write":1268,"num_write_kb":21738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"110'758","reported_seq":1138,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.031518+0000","last_change":"2023-10-30T14:26:42.031593+0000","last_active":"2023-10-30T14:26:42.031518+0000","last_peered":"2023-10-30T14:26:42.031518+0000","last_clean":"2023-10-30T14:26:42.031518+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T14:26:42.031518+0000","last_undegraded":"2023-10-30T14:26:42.031518+0000","last_fullsized":"2023-10-30T14:26:42.031518+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'758","last_scrub_stamp":"2023-10-30T14:26:42.031416+0000","last_deep_scrub":"110'758","last_deep_scrub_stamp":"2023-10-30T14:26:42.031416+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.031416+0000","objects_scrubbed":57,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:32.795744+0000","scrub_duration":0.243287324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331682,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":392,"num_read_kb":480410,"num_write":807,"num_write_kb":87841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":384994,"num_omap_keys":754,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"110'2184","reported_seq":5052,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.780582+0000","last_change":"2023-10-30T14:26:52.780632+0000","last_active":"2023-10-30T14:26:52.780582+0000","last_peered":"2023-10-30T14:26:52.780582+0000","last_clean":"2023-10-30T14:26:52.780582+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T14:26:52.780582+0000","last_undegraded":"2023-10-30T14:26:52.780582+0000","last_fullsized":"2023-10-30T14:26:52.780582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2184","last_scrub_stamp":"2023-10-30T14:26:52.780505+0000","last_deep_scrub":"110'2184","last_deep_scrub_stamp":"2023-10-30T14:26:52.780505+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.780505+0000","objects_scrubbed":34,"log_size":2184,"log_dups_size":0,"ondisk_log_size":2184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:22.325362+0000","scrub_duration":0.086211091000000004,"objects_trimmed":0,"snaptrim_duration":0.95729191499999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":151,"num_read_kb":31378,"num_write":1306,"num_write_kb":33173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"110'899","reported_seq":1169,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.271896+0000","last_change":"2023-10-30T14:26:54.272052+0000","last_active":"2023-10-30T14:26:54.271896+0000","last_peered":"2023-10-30T14:26:54.271896+0000","last_clean":"2023-10-30T14:26:54.271896+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T14:26:54.271896+0000","last_undegraded":"2023-10-30T14:26:54.271896+0000","last_fullsized":"2023-10-30T14:26:54.271896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'899","last_scrub_stamp":"2023-10-30T14:26:54.271663+0000","last_deep_scrub":"110'899","last_deep_scrub_stamp":"2023-10-30T14:26:54.271663+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.271663+0000","objects_scrubbed":48,"log_size":899,"log_dups_size":0,"ondisk_log_size":899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:16.820077+0000","scrub_duration":0.294758195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53927368,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":208,"num_read_kb":306859,"num_write":914,"num_write_kb":62457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":467666,"num_omap_keys":812,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"110'2334","reported_seq":5135,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:09.227727+0000","last_change":"2023-10-30T14:27:09.227790+0000","last_active":"2023-10-30T14:27:09.227727+0000","last_peered":"2023-10-30T14:27:09.227727+0000","last_clean":"2023-10-30T14:27:09.227727+0000","last_became_active":"2023-10-30T13:24:00.706894+0000","last_became_peered":"2023-10-30T13:24:00.706894+0000","last_unstale":"2023-10-30T14:27:09.227727+0000","last_undegraded":"2023-10-30T14:27:09.227727+0000","last_fullsized":"2023-10-30T14:27:09.227727+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2334","last_scrub_stamp":"2023-10-30T14:27:09.227647+0000","last_deep_scrub":"110'2334","last_deep_scrub_stamp":"2023-10-30T14:27:09.227647+0000","last_clean_scrub_stamp":"2023-10-30T14:27:09.227647+0000","objects_scrubbed":41,"log_size":2334,"log_dups_size":0,"ondisk_log_size":2334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:45:55.621940+0000","scrub_duration":0.096202764999999996,"objects_trimmed":0,"snaptrim_duration":0.3738494,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":149,"num_read_kb":1539,"num_write":1375,"num_write_kb":12931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"110'780","reported_seq":1067,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.584097+0000","last_change":"2023-10-30T14:26:46.584221+0000","last_active":"2023-10-30T14:26:46.584097+0000","last_peered":"2023-10-30T14:26:46.584097+0000","last_clean":"2023-10-30T14:26:46.584097+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T14:26:46.584097+0000","last_undegraded":"2023-10-30T14:26:46.584097+0000","last_fullsized":"2023-10-30T14:26:46.584097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'780","last_scrub_stamp":"2023-10-30T14:26:46.583879+0000","last_deep_scrub":"110'780","last_deep_scrub_stamp":"2023-10-30T14:26:46.583879+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.583879+0000","objects_scrubbed":53,"log_size":780,"log_dups_size":0,"ondisk_log_size":780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:52.998170+0000","scrub_duration":0.20502225800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":234,"num_read_kb":298146,"num_write":829,"num_write_kb":54985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":494954,"num_omap_keys":946,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"110'2149","reported_seq":5030,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.527465+0000","last_change":"2023-10-30T14:27:00.527596+0000","last_active":"2023-10-30T14:27:00.527465+0000","last_peered":"2023-10-30T14:27:00.527465+0000","last_clean":"2023-10-30T14:27:00.527465+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T14:27:00.527465+0000","last_undegraded":"2023-10-30T14:27:00.527465+0000","last_fullsized":"2023-10-30T14:27:00.527465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2149","last_scrub_stamp":"2023-10-30T14:27:00.527256+0000","last_deep_scrub":"110'2149","last_deep_scrub_stamp":"2023-10-30T14:27:00.527256+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.527256+0000","objects_scrubbed":53,"log_size":2149,"log_dups_size":0,"ondisk_log_size":2149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:55:50.223566+0000","scrub_duration":0.14114542399999999,"objects_trimmed":0,"snaptrim_duration":0.50412874500000004,"stat_sum":{"num_bytes":0,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":148,"num_read_kb":43506,"num_write":1375,"num_write_kb":26417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"110'876","reported_seq":1232,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.183186+0000","last_change":"2023-10-30T14:26:52.183292+0000","last_active":"2023-10-30T14:26:52.183186+0000","last_peered":"2023-10-30T14:26:52.183186+0000","last_clean":"2023-10-30T14:26:52.183186+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T14:26:52.183186+0000","last_undegraded":"2023-10-30T14:26:52.183186+0000","last_fullsized":"2023-10-30T14:26:52.183186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'876","last_scrub_stamp":"2023-10-30T14:26:52.183105+0000","last_deep_scrub":"110'876","last_deep_scrub_stamp":"2023-10-30T14:26:52.183105+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.183105+0000","objects_scrubbed":51,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:18:36.571108+0000","scrub_duration":0.119037509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":481,"num_read_kb":229594,"num_write":1003,"num_write_kb":42855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":532426,"num_omap_keys":823,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"110'2380","reported_seq":5218,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.002361+0000","last_change":"2023-10-30T14:26:46.002570+0000","last_active":"2023-10-30T14:26:46.002361+0000","last_peered":"2023-10-30T14:26:46.002361+0000","last_clean":"2023-10-30T14:26:46.002361+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T14:26:46.002361+0000","last_undegraded":"2023-10-30T14:26:46.002361+0000","last_fullsized":"2023-10-30T14:26:46.002361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2380","last_scrub_stamp":"2023-10-30T14:26:46.002134+0000","last_deep_scrub":"110'2380","last_deep_scrub_stamp":"2023-10-30T14:26:46.002134+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.002134+0000","objects_scrubbed":46,"log_size":2380,"log_dups_size":0,"ondisk_log_size":2380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:38.920284+0000","scrub_duration":0.14576676,"objects_trimmed":0,"snaptrim_duration":0.70955538100000004,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":166,"num_read_kb":25446,"num_write":1448,"num_write_kb":32435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"110'1046","reported_seq":1341,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.977448+0000","last_change":"2023-10-30T14:26:42.977518+0000","last_active":"2023-10-30T14:26:42.977448+0000","last_peered":"2023-10-30T14:26:42.977448+0000","last_clean":"2023-10-30T14:26:42.977448+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T14:26:42.977448+0000","last_undegraded":"2023-10-30T14:26:42.977448+0000","last_fullsized":"2023-10-30T14:26:42.977448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1046","last_scrub_stamp":"2023-10-30T14:26:42.977329+0000","last_deep_scrub":"110'1046","last_deep_scrub_stamp":"2023-10-30T14:26:42.977329+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.977329+0000","objects_scrubbed":51,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:00.448502+0000","scrub_duration":0.210468818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":214,"num_read_kb":373366,"num_write":1150,"num_write_kb":71086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":337344,"num_omap_keys":660,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"110'2278","reported_seq":5077,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:36.022639+0000","last_change":"2023-10-30T14:27:36.022780+0000","last_active":"2023-10-30T14:27:36.022639+0000","last_peered":"2023-10-30T14:27:36.022639+0000","last_clean":"2023-10-30T14:27:36.022639+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T14:27:36.022639+0000","last_undegraded":"2023-10-30T14:27:36.022639+0000","last_fullsized":"2023-10-30T14:27:36.022639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2278","last_scrub_stamp":"2023-10-30T14:27:36.022470+0000","last_deep_scrub":"110'2278","last_deep_scrub_stamp":"2023-10-30T14:27:36.022470+0000","last_clean_scrub_stamp":"2023-10-30T14:27:36.022470+0000","objects_scrubbed":40,"log_size":2278,"log_dups_size":0,"ondisk_log_size":2278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:51.873473+0000","scrub_duration":0.121129692,"objects_trimmed":0,"snaptrim_duration":0.89413299700000004,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":142,"num_read_kb":18968,"num_write":1278,"num_write_kb":24747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"110'668","reported_seq":989,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:47.033482+0000","last_change":"2023-10-30T14:26:47.033575+0000","last_active":"2023-10-30T14:26:47.033482+0000","last_peered":"2023-10-30T14:26:47.033482+0000","last_clean":"2023-10-30T14:26:47.033482+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T14:26:47.033482+0000","last_undegraded":"2023-10-30T14:26:47.033482+0000","last_fullsized":"2023-10-30T14:26:47.033482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'668","last_scrub_stamp":"2023-10-30T14:26:47.033389+0000","last_deep_scrub":"110'668","last_deep_scrub_stamp":"2023-10-30T14:26:47.033389+0000","last_clean_scrub_stamp":"2023-10-30T14:26:47.033389+0000","objects_scrubbed":42,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:25.680237+0000","scrub_duration":0.095992737999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":280,"num_read_kb":492842,"num_write":679,"num_write_kb":83547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":470248,"num_omap_keys":762,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"110'2131","reported_seq":5015,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:33.067495+0000","last_change":"2023-10-30T14:27:33.067594+0000","last_active":"2023-10-30T14:27:33.067495+0000","last_peered":"2023-10-30T14:27:33.067495+0000","last_clean":"2023-10-30T14:27:33.067495+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T14:27:33.067495+0000","last_undegraded":"2023-10-30T14:27:33.067495+0000","last_fullsized":"2023-10-30T14:27:33.067495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2131","last_scrub_stamp":"2023-10-30T14:27:33.067332+0000","last_deep_scrub":"110'2131","last_deep_scrub_stamp":"2023-10-30T14:27:33.067332+0000","last_clean_scrub_stamp":"2023-10-30T14:27:33.067332+0000","objects_scrubbed":35,"log_size":2131,"log_dups_size":0,"ondisk_log_size":2131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:28:09.043194+0000","scrub_duration":0.10920801500000001,"objects_trimmed":0,"snaptrim_duration":0.88483857600000004,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":152,"num_read_kb":61046,"num_write":1310,"num_write_kb":36653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"110'790","reported_seq":1095,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.425710+0000","last_change":"2023-10-30T14:26:52.425764+0000","last_active":"2023-10-30T14:26:52.425710+0000","last_peered":"2023-10-30T14:26:52.425710+0000","last_clean":"2023-10-30T14:26:52.425710+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T14:26:52.425710+0000","last_undegraded":"2023-10-30T14:26:52.425710+0000","last_fullsized":"2023-10-30T14:26:52.425710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'790","last_scrub_stamp":"2023-10-30T14:26:52.425638+0000","last_deep_scrub":"110'790","last_deep_scrub_stamp":"2023-10-30T14:26:52.425638+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.425638+0000","objects_scrubbed":60,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:57:40.747257+0000","scrub_duration":0.18100369799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364699,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":233,"num_read_kb":370768,"num_write":815,"num_write_kb":71238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":549758,"num_omap_keys":876,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"110'2281","reported_seq":5152,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:14.156230+0000","last_change":"2023-10-30T14:27:14.156348+0000","last_active":"2023-10-30T14:27:14.156230+0000","last_peered":"2023-10-30T14:27:14.156230+0000","last_clean":"2023-10-30T14:27:14.156230+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T14:27:14.156230+0000","last_undegraded":"2023-10-30T14:27:14.156230+0000","last_fullsized":"2023-10-30T14:27:14.156230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2281","last_scrub_stamp":"2023-10-30T14:27:14.156044+0000","last_deep_scrub":"110'2281","last_deep_scrub_stamp":"2023-10-30T14:27:14.156044+0000","last_clean_scrub_stamp":"2023-10-30T14:27:14.156044+0000","objects_scrubbed":41,"log_size":2281,"log_dups_size":0,"ondisk_log_size":2281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:23:25.400045+0000","scrub_duration":0.105525299,"objects_trimmed":0,"snaptrim_duration":0.98441546899999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":178,"num_read_kb":29706,"num_write":1375,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"110'1121","reported_seq":1413,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.293606+0000","last_change":"2023-10-30T14:26:54.293672+0000","last_active":"2023-10-30T14:26:54.293606+0000","last_peered":"2023-10-30T14:26:54.293606+0000","last_clean":"2023-10-30T14:26:54.293606+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T14:26:54.293606+0000","last_undegraded":"2023-10-30T14:26:54.293606+0000","last_fullsized":"2023-10-30T14:26:54.293606+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1121","last_scrub_stamp":"2023-10-30T14:26:54.293526+0000","last_deep_scrub":"110'1121","last_deep_scrub_stamp":"2023-10-30T14:26:54.293526+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.293526+0000","objects_scrubbed":43,"log_size":1121,"log_dups_size":0,"ondisk_log_size":1121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:17.009305+0000","scrub_duration":0.241353279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50332230,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":210,"num_read_kb":406971,"num_write":1089,"num_write_kb":71553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":220904,"num_omap_keys":399,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"110'2078","reported_seq":4939,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:16.182932+0000","last_change":"2023-10-30T14:27:16.183032+0000","last_active":"2023-10-30T14:27:16.182932+0000","last_peered":"2023-10-30T14:27:16.182932+0000","last_clean":"2023-10-30T14:27:16.182932+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T14:27:16.182932+0000","last_undegraded":"2023-10-30T14:27:16.182932+0000","last_fullsized":"2023-10-30T14:27:16.182932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2078","last_scrub_stamp":"2023-10-30T14:27:16.182829+0000","last_deep_scrub":"110'2078","last_deep_scrub_stamp":"2023-10-30T14:27:16.182829+0000","last_clean_scrub_stamp":"2023-10-30T14:27:16.182829+0000","objects_scrubbed":38,"log_size":2078,"log_dups_size":0,"ondisk_log_size":2078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:14:39.775252+0000","scrub_duration":0.132528019,"objects_trimmed":0,"snaptrim_duration":0.84782311300000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":122,"num_read_kb":1096,"num_write":1268,"num_write_kb":17447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"110'119","reported_seq":298,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:47.899032+0000","last_change":"2023-10-30T14:26:47.899032+0000","last_active":"2023-10-30T14:26:47.899032+0000","last_peered":"2023-10-30T14:26:47.899032+0000","last_clean":"2023-10-30T14:26:47.899032+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T14:26:47.899032+0000","last_undegraded":"2023-10-30T14:26:47.899032+0000","last_fullsized":"2023-10-30T14:26:47.899032+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'119","last_scrub_stamp":"2023-10-30T14:26:47.898726+0000","last_deep_scrub":"110'119","last_deep_scrub_stamp":"2023-10-30T14:26:47.898726+0000","last_clean_scrub_stamp":"2023-10-30T14:26:47.898726+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:29.562342+0000","scrub_duration":0.0091457540000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"110'909","reported_seq":1258,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:50.195755+0000","last_change":"2023-10-30T14:26:50.195816+0000","last_active":"2023-10-30T14:26:50.195755+0000","last_peered":"2023-10-30T14:26:50.195755+0000","last_clean":"2023-10-30T14:26:50.195755+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T14:26:50.195755+0000","last_undegraded":"2023-10-30T14:26:50.195755+0000","last_fullsized":"2023-10-30T14:26:50.195755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'909","last_scrub_stamp":"2023-10-30T14:26:50.195649+0000","last_deep_scrub":"110'909","last_deep_scrub_stamp":"2023-10-30T14:26:50.195649+0000","last_clean_scrub_stamp":"2023-10-30T14:26:50.195649+0000","objects_scrubbed":60,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:12.567856+0000","scrub_duration":0.19057717900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331714,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":319,"num_read_kb":439640,"num_write":936,"num_write_kb":75464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":459843,"num_omap_keys":807,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"110'2244","reported_seq":5041,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.842968+0000","last_change":"2023-10-30T14:26:57.843044+0000","last_active":"2023-10-30T14:26:57.842968+0000","last_peered":"2023-10-30T14:26:57.842968+0000","last_clean":"2023-10-30T14:26:57.842968+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T14:26:57.842968+0000","last_undegraded":"2023-10-30T14:26:57.842968+0000","last_fullsized":"2023-10-30T14:26:57.842968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2244","last_scrub_stamp":"2023-10-30T14:26:57.842882+0000","last_deep_scrub":"110'2244","last_deep_scrub_stamp":"2023-10-30T14:26:57.842882+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.842882+0000","objects_scrubbed":46,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:34:34.719838+0000","scrub_duration":0.10386124300000001,"objects_trimmed":0,"snaptrim_duration":1.003812562,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":127,"num_read_kb":26261,"num_write":1294,"num_write_kb":24882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"110'1232","reported_seq":1556,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.159534+0000","last_change":"2023-10-30T14:26:53.159534+0000","last_active":"2023-10-30T14:26:53.159534+0000","last_peered":"2023-10-30T14:26:53.159534+0000","last_clean":"2023-10-30T14:26:53.159534+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T14:26:53.159534+0000","last_undegraded":"2023-10-30T14:26:53.159534+0000","last_fullsized":"2023-10-30T14:26:53.159534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1232","last_scrub_stamp":"2023-10-30T14:26:53.159107+0000","last_deep_scrub":"110'1232","last_deep_scrub_stamp":"2023-10-30T14:26:53.159107+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.159107+0000","objects_scrubbed":46,"log_size":1232,"log_dups_size":0,"ondisk_log_size":1232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:27.847424+0000","scrub_duration":0.24891354399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":256,"num_read_kb":488007,"num_write":1246,"num_write_kb":83351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":346758,"num_omap_keys":680,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"110'2013","reported_seq":4887,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:51.757304+0000","last_change":"2023-10-30T14:26:51.757396+0000","last_active":"2023-10-30T14:26:51.757304+0000","last_peered":"2023-10-30T14:26:51.757304+0000","last_clean":"2023-10-30T14:26:51.757304+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T14:26:51.757304+0000","last_undegraded":"2023-10-30T14:26:51.757304+0000","last_fullsized":"2023-10-30T14:26:51.757304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2013","last_scrub_stamp":"2023-10-30T14:26:51.757217+0000","last_deep_scrub":"110'2013","last_deep_scrub_stamp":"2023-10-30T14:26:51.757217+0000","last_clean_scrub_stamp":"2023-10-30T14:26:51.757217+0000","objects_scrubbed":34,"log_size":2013,"log_dups_size":0,"ondisk_log_size":2013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:49.616919+0000","scrub_duration":0.071006446000000001,"objects_trimmed":0,"snaptrim_duration":0.92159720499999997,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":129,"num_read_kb":13045,"num_write":1222,"num_write_kb":11637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"110'2000","reported_seq":3256,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:58.136825+0000","last_change":"2023-10-30T14:26:58.136871+0000","last_active":"2023-10-30T14:26:58.136825+0000","last_peered":"2023-10-30T14:26:58.136825+0000","last_clean":"2023-10-30T14:26:58.136825+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T14:26:58.136825+0000","last_undegraded":"2023-10-30T14:26:58.136825+0000","last_fullsized":"2023-10-30T14:26:58.136825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2000","last_scrub_stamp":"2023-10-30T14:26:58.136733+0000","last_deep_scrub":"110'2000","last_deep_scrub_stamp":"2023-10-30T14:26:58.136733+0000","last_clean_scrub_stamp":"2023-10-30T14:26:58.136733+0000","objects_scrubbed":51,"log_size":2000,"log_dups_size":0,"ondisk_log_size":2000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:54.997528+0000","scrub_duration":0.11551852899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40060913,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":1167,"num_read_kb":359975,"num_write":2020,"num_write_kb":69890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":329700,"num_omap_keys":585,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"110'2306","reported_seq":5130,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:56.314854+0000","last_change":"2023-10-30T14:26:56.314854+0000","last_active":"2023-10-30T14:26:56.314854+0000","last_peered":"2023-10-30T14:26:56.314854+0000","last_clean":"2023-10-30T14:26:56.314854+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T14:26:56.314854+0000","last_undegraded":"2023-10-30T14:26:56.314854+0000","last_fullsized":"2023-10-30T14:26:56.314854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2306","last_scrub_stamp":"2023-10-30T14:26:56.314673+0000","last_deep_scrub":"110'2306","last_deep_scrub_stamp":"2023-10-30T14:26:56.314673+0000","last_clean_scrub_stamp":"2023-10-30T14:26:56.314673+0000","objects_scrubbed":45,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:16:05.533537+0000","scrub_duration":0.059182301999999999,"objects_trimmed":0,"snaptrim_duration":0.40657480699999998,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":145,"num_read_kb":55105,"num_write":1354,"num_write_kb":28005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"110'1533","reported_seq":1849,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:45.029556+0000","last_change":"2023-10-30T14:26:45.029718+0000","last_active":"2023-10-30T14:26:45.029556+0000","last_peered":"2023-10-30T14:26:45.029556+0000","last_clean":"2023-10-30T14:26:45.029556+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T14:26:45.029556+0000","last_undegraded":"2023-10-30T14:26:45.029556+0000","last_fullsized":"2023-10-30T14:26:45.029556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1533","last_scrub_stamp":"2023-10-30T14:26:45.029381+0000","last_deep_scrub":"110'1533","last_deep_scrub_stamp":"2023-10-30T14:26:45.029381+0000","last_clean_scrub_stamp":"2023-10-30T14:26:45.029381+0000","objects_scrubbed":45,"log_size":1533,"log_dups_size":0,"ondisk_log_size":1533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:54.864240+0000","scrub_duration":0.21060546799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":235,"num_read_kb":509467,"num_write":1537,"num_write_kb":83943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":361192,"num_omap_keys":645,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"110'2242","reported_seq":5077,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.859618+0000","last_change":"2023-10-30T14:26:46.859846+0000","last_active":"2023-10-30T14:26:46.859618+0000","last_peered":"2023-10-30T14:26:46.859618+0000","last_clean":"2023-10-30T14:26:46.859618+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T14:26:46.859618+0000","last_undegraded":"2023-10-30T14:26:46.859618+0000","last_fullsized":"2023-10-30T14:26:46.859618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2242","last_scrub_stamp":"2023-10-30T14:26:46.859379+0000","last_deep_scrub":"110'2242","last_deep_scrub_stamp":"2023-10-30T14:26:46.859379+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.859379+0000","objects_scrubbed":34,"log_size":2242,"log_dups_size":0,"ondisk_log_size":2242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:58:59.480931+0000","scrub_duration":0.104056545,"objects_trimmed":0,"snaptrim_duration":0.59694472600000004,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":143,"num_read_kb":13239,"num_write":1306,"num_write_kb":17120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"110'663","reported_seq":985,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:50.346246+0000","last_change":"2023-10-30T14:26:50.346292+0000","last_active":"2023-10-30T14:26:50.346246+0000","last_peered":"2023-10-30T14:26:50.346246+0000","last_clean":"2023-10-30T14:26:50.346246+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T14:26:50.346246+0000","last_undegraded":"2023-10-30T14:26:50.346246+0000","last_fullsized":"2023-10-30T14:26:50.346246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'663","last_scrub_stamp":"2023-10-30T14:26:50.346173+0000","last_deep_scrub":"110'663","last_deep_scrub_stamp":"2023-10-30T14:26:50.346173+0000","last_clean_scrub_stamp":"2023-10-30T14:26:50.346173+0000","objects_scrubbed":53,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:23:57.407805+0000","scrub_duration":0.108375394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":230,"num_read_kb":520969,"num_write":671,"num_write_kb":95828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":375058,"num_omap_keys":621,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"110'2212","reported_seq":5041,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:20.112321+0000","last_change":"2023-10-30T14:27:20.112433+0000","last_active":"2023-10-30T14:27:20.112321+0000","last_peered":"2023-10-30T14:27:20.112321+0000","last_clean":"2023-10-30T14:27:20.112321+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T14:27:20.112321+0000","last_undegraded":"2023-10-30T14:27:20.112321+0000","last_fullsized":"2023-10-30T14:27:20.112321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2212","last_scrub_stamp":"2023-10-30T14:27:20.112126+0000","last_deep_scrub":"110'2212","last_deep_scrub_stamp":"2023-10-30T14:27:20.112126+0000","last_clean_scrub_stamp":"2023-10-30T14:27:20.112126+0000","objects_scrubbed":32,"log_size":2212,"log_dups_size":0,"ondisk_log_size":2212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:46.492910+0000","scrub_duration":0.087815049000000006,"objects_trimmed":0,"snaptrim_duration":1.075105449,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":60418,"num_write":1286,"num_write_kb":36950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"110'1084","reported_seq":1379,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:59.155850+0000","last_change":"2023-10-30T14:26:59.155993+0000","last_active":"2023-10-30T14:26:59.155850+0000","last_peered":"2023-10-30T14:26:59.155850+0000","last_clean":"2023-10-30T14:26:59.155850+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T14:26:59.155850+0000","last_undegraded":"2023-10-30T14:26:59.155850+0000","last_fullsized":"2023-10-30T14:26:59.155850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1084","last_scrub_stamp":"2023-10-30T14:26:59.155631+0000","last_deep_scrub":"110'1084","last_deep_scrub_stamp":"2023-10-30T14:26:59.155631+0000","last_clean_scrub_stamp":"2023-10-30T14:26:59.155631+0000","objects_scrubbed":38,"log_size":1084,"log_dups_size":0,"ondisk_log_size":1084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:20:05.652224+0000","scrub_duration":0.220506067,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44106025,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":214,"num_read_kb":377318,"num_write":1089,"num_write_kb":69046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":276186,"num_omap_keys":520,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"110'2247","reported_seq":5091,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:15.229805+0000","last_change":"2023-10-30T14:27:15.229873+0000","last_active":"2023-10-30T14:27:15.229805+0000","last_peered":"2023-10-30T14:27:15.229805+0000","last_clean":"2023-10-30T14:27:15.229805+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T14:27:15.229805+0000","last_undegraded":"2023-10-30T14:27:15.229805+0000","last_fullsized":"2023-10-30T14:27:15.229805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2247","last_scrub_stamp":"2023-10-30T14:27:15.229719+0000","last_deep_scrub":"110'2247","last_deep_scrub_stamp":"2023-10-30T14:27:15.229719+0000","last_clean_scrub_stamp":"2023-10-30T14:27:15.229719+0000","objects_scrubbed":37,"log_size":2247,"log_dups_size":0,"ondisk_log_size":2247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:06:33.811869+0000","scrub_duration":0.052142862999999998,"objects_trimmed":0,"snaptrim_duration":0.36741958800000002,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":157,"num_read_kb":35362,"num_write":1332,"num_write_kb":31772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"110'1165","reported_seq":1469,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.451066+0000","last_change":"2023-10-30T14:26:54.451118+0000","last_active":"2023-10-30T14:26:54.451066+0000","last_peered":"2023-10-30T14:26:54.451066+0000","last_clean":"2023-10-30T14:26:54.451066+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T14:26:54.451066+0000","last_undegraded":"2023-10-30T14:26:54.451066+0000","last_fullsized":"2023-10-30T14:26:54.451066+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1165","last_scrub_stamp":"2023-10-30T14:26:54.450996+0000","last_deep_scrub":"110'1165","last_deep_scrub_stamp":"2023-10-30T14:26:54.450996+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.450996+0000","objects_scrubbed":39,"log_size":1165,"log_dups_size":0,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:18:07.890199+0000","scrub_duration":0.17568342300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":247,"num_read_kb":365780,"num_write":1783,"num_write_kb":65025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":395035,"num_omap_keys":693,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"110'2123","reported_seq":4968,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:10.178670+0000","last_change":"2023-10-30T14:27:10.178735+0000","last_active":"2023-10-30T14:27:10.178670+0000","last_peered":"2023-10-30T14:27:10.178670+0000","last_clean":"2023-10-30T14:27:10.178670+0000","last_became_active":"2023-10-30T13:24:01.001412+0000","last_became_peered":"2023-10-30T13:24:01.001412+0000","last_unstale":"2023-10-30T14:27:10.178670+0000","last_undegraded":"2023-10-30T14:27:10.178670+0000","last_fullsized":"2023-10-30T14:27:10.178670+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2123","last_scrub_stamp":"2023-10-30T14:27:10.178582+0000","last_deep_scrub":"110'2123","last_deep_scrub_stamp":"2023-10-30T14:27:10.178582+0000","last_clean_scrub_stamp":"2023-10-30T14:27:10.178582+0000","objects_scrubbed":35,"log_size":2123,"log_dups_size":0,"ondisk_log_size":2123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:49.167738+0000","scrub_duration":0.079853908000000001,"objects_trimmed":0,"snaptrim_duration":0.32700652899999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":137,"num_read_kb":5811,"num_write":1214,"num_write_kb":16143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"110'515","reported_seq":831,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.490148+0000","last_change":"2023-10-30T14:26:53.490236+0000","last_active":"2023-10-30T14:26:53.490148+0000","last_peered":"2023-10-30T14:26:53.490148+0000","last_clean":"2023-10-30T14:26:53.490148+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T14:26:53.490148+0000","last_undegraded":"2023-10-30T14:26:53.490148+0000","last_fullsized":"2023-10-30T14:26:53.490148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'515","last_scrub_stamp":"2023-10-30T14:26:53.490042+0000","last_deep_scrub":"110'515","last_deep_scrub_stamp":"2023-10-30T14:26:53.490042+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.490042+0000","objects_scrubbed":47,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:16.217409+0000","scrub_duration":0.22512389199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":281,"num_read_kb":442743,"num_write":555,"num_write_kb":70797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":243443,"num_omap_keys":457,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"110'2226","reported_seq":5069,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:59.865953+0000","last_change":"2023-10-30T14:26:59.866026+0000","last_active":"2023-10-30T14:26:59.865953+0000","last_peered":"2023-10-30T14:26:59.865953+0000","last_clean":"2023-10-30T14:26:59.865953+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T14:26:59.865953+0000","last_undegraded":"2023-10-30T14:26:59.865953+0000","last_fullsized":"2023-10-30T14:26:59.865953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2226","last_scrub_stamp":"2023-10-30T14:26:59.865860+0000","last_deep_scrub":"110'2226","last_deep_scrub_stamp":"2023-10-30T14:26:59.865860+0000","last_clean_scrub_stamp":"2023-10-30T14:26:59.865860+0000","objects_scrubbed":40,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:25.638448+0000","scrub_duration":0.053687321000000003,"objects_trimmed":0,"snaptrim_duration":1.183697322,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":144,"num_read_kb":25445,"num_write":1326,"num_write_kb":20705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"110'584","reported_seq":860,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.994203+0000","last_change":"2023-10-30T14:26:53.994349+0000","last_active":"2023-10-30T14:26:53.994203+0000","last_peered":"2023-10-30T14:26:53.994203+0000","last_clean":"2023-10-30T14:26:53.994203+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T14:26:53.994203+0000","last_undegraded":"2023-10-30T14:26:53.994203+0000","last_fullsized":"2023-10-30T14:26:53.994203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'584","last_scrub_stamp":"2023-10-30T14:26:53.993981+0000","last_deep_scrub":"110'584","last_deep_scrub_stamp":"2023-10-30T14:26:53.993981+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.993981+0000","objects_scrubbed":40,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:54.497876+0000","scrub_duration":0.12048839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":236,"num_read_kb":274544,"num_write":603,"num_write_kb":42177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":323567,"num_omap_keys":593,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"110'2168","reported_seq":5027,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:49.864350+0000","last_change":"2023-10-30T14:26:49.864486+0000","last_active":"2023-10-30T14:26:49.864350+0000","last_peered":"2023-10-30T14:26:49.864350+0000","last_clean":"2023-10-30T14:26:49.864350+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T14:26:49.864350+0000","last_undegraded":"2023-10-30T14:26:49.864350+0000","last_fullsized":"2023-10-30T14:26:49.864350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2168","last_scrub_stamp":"2023-10-30T14:26:49.864161+0000","last_deep_scrub":"110'2168","last_deep_scrub_stamp":"2023-10-30T14:26:49.864161+0000","last_clean_scrub_stamp":"2023-10-30T14:26:49.864161+0000","objects_scrubbed":45,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:54:18.553003+0000","scrub_duration":0.11066809399999999,"objects_trimmed":0,"snaptrim_duration":0.56267233699999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":144,"num_read_kb":34655,"num_write":1307,"num_write_kb":20329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"110'1112","reported_seq":1394,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:45.299966+0000","last_change":"2023-10-30T14:26:45.300020+0000","last_active":"2023-10-30T14:26:45.299966+0000","last_peered":"2023-10-30T14:26:45.299966+0000","last_clean":"2023-10-30T14:26:45.299966+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T14:26:45.299966+0000","last_undegraded":"2023-10-30T14:26:45.299966+0000","last_fullsized":"2023-10-30T14:26:45.299966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1112","last_scrub_stamp":"2023-10-30T14:26:45.299861+0000","last_deep_scrub":"110'1112","last_deep_scrub_stamp":"2023-10-30T14:26:45.299861+0000","last_clean_scrub_stamp":"2023-10-30T14:26:45.299861+0000","objects_scrubbed":51,"log_size":1112,"log_dups_size":0,"ondisk_log_size":1112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:30.314690+0000","scrub_duration":0.15631188700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":237,"num_read_kb":249991,"num_write":1213,"num_write_kb":43538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":609555,"num_omap_keys":1142,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"110'2316","reported_seq":5113,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:18.616129+0000","last_change":"2023-10-30T14:27:18.616224+0000","last_active":"2023-10-30T14:27:18.616129+0000","last_peered":"2023-10-30T14:27:18.616129+0000","last_clean":"2023-10-30T14:27:18.616129+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T14:27:18.616129+0000","last_undegraded":"2023-10-30T14:27:18.616129+0000","last_fullsized":"2023-10-30T14:27:18.616129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2316","last_scrub_stamp":"2023-10-30T14:27:18.616043+0000","last_deep_scrub":"110'2316","last_deep_scrub_stamp":"2023-10-30T14:27:18.616043+0000","last_clean_scrub_stamp":"2023-10-30T14:27:18.616043+0000","objects_scrubbed":45,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:14:36.881982+0000","scrub_duration":0.061902202000000003,"objects_trimmed":0,"snaptrim_duration":0.93815086199999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":129,"num_read_kb":1227,"num_write":1352,"num_write_kb":12536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"110'1527","reported_seq":1874,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.480258+0000","last_change":"2023-10-30T14:26:53.480338+0000","last_active":"2023-10-30T14:26:53.480258+0000","last_peered":"2023-10-30T14:26:53.480258+0000","last_clean":"2023-10-30T14:26:53.480258+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T14:26:53.480258+0000","last_undegraded":"2023-10-30T14:26:53.480258+0000","last_fullsized":"2023-10-30T14:26:53.480258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1527","last_scrub_stamp":"2023-10-30T14:26:53.480148+0000","last_deep_scrub":"110'1527","last_deep_scrub_stamp":"2023-10-30T14:26:53.480148+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.480148+0000","objects_scrubbed":54,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:31:02.686045+0000","scrub_duration":0.25185449900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":321,"num_read_kb":438308,"num_write":2003,"num_write_kb":98279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":359842,"num_omap_keys":684,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"110'2123","reported_seq":4973,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:55.746681+0000","last_change":"2023-10-30T14:26:55.746755+0000","last_active":"2023-10-30T14:26:55.746681+0000","last_peered":"2023-10-30T14:26:55.746681+0000","last_clean":"2023-10-30T14:26:55.746681+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T14:26:55.746681+0000","last_undegraded":"2023-10-30T14:26:55.746681+0000","last_fullsized":"2023-10-30T14:26:55.746681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2123","last_scrub_stamp":"2023-10-30T14:26:55.746576+0000","last_deep_scrub":"110'2123","last_deep_scrub_stamp":"2023-10-30T14:26:55.746576+0000","last_clean_scrub_stamp":"2023-10-30T14:26:55.746576+0000","objects_scrubbed":27,"log_size":2123,"log_dups_size":0,"ondisk_log_size":2123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:08:17.663937+0000","scrub_duration":0.039827926,"objects_trimmed":0,"snaptrim_duration":0.94968995099999998,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":141,"num_read_kb":11852,"num_write":1237,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"110'1108","reported_seq":1435,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.161003+0000","last_change":"2023-10-30T14:26:57.161075+0000","last_active":"2023-10-30T14:26:57.161003+0000","last_peered":"2023-10-30T14:26:57.161003+0000","last_clean":"2023-10-30T14:26:57.161003+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T14:26:57.161003+0000","last_undegraded":"2023-10-30T14:26:57.161003+0000","last_fullsized":"2023-10-30T14:26:57.161003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1108","last_scrub_stamp":"2023-10-30T14:26:57.160913+0000","last_deep_scrub":"110'1108","last_deep_scrub_stamp":"2023-10-30T14:26:57.160913+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.160913+0000","objects_scrubbed":44,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:56:50.729954+0000","scrub_duration":0.25105180199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":296,"num_read_kb":429426,"num_write":1159,"num_write_kb":75926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":378159,"num_omap_keys":718,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"110'2194","reported_seq":5040,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:06.417252+0000","last_change":"2023-10-30T14:27:06.417354+0000","last_active":"2023-10-30T14:27:06.417252+0000","last_peered":"2023-10-30T14:27:06.417252+0000","last_clean":"2023-10-30T14:27:06.417252+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T14:27:06.417252+0000","last_undegraded":"2023-10-30T14:27:06.417252+0000","last_fullsized":"2023-10-30T14:27:06.417252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2194","last_scrub_stamp":"2023-10-30T14:27:06.417083+0000","last_deep_scrub":"110'2194","last_deep_scrub_stamp":"2023-10-30T14:27:06.417083+0000","last_clean_scrub_stamp":"2023-10-30T14:27:06.417083+0000","objects_scrubbed":30,"log_size":2194,"log_dups_size":0,"ondisk_log_size":2194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:32:26.701207+0000","scrub_duration":0.099883823999999996,"objects_trimmed":0,"snaptrim_duration":0.419978554,"stat_sum":{"num_bytes":5,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":137,"num_read_kb":40109,"num_write":1287,"num_write_kb":25283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"110'2213","reported_seq":5031,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:16.181195+0000","last_change":"2023-10-30T14:27:16.181270+0000","last_active":"2023-10-30T14:27:16.181195+0000","last_peered":"2023-10-30T14:27:16.181195+0000","last_clean":"2023-10-30T14:27:16.181195+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T14:27:16.181195+0000","last_undegraded":"2023-10-30T14:27:16.181195+0000","last_fullsized":"2023-10-30T14:27:16.181195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2213","last_scrub_stamp":"2023-10-30T14:27:16.181071+0000","last_deep_scrub":"110'2213","last_deep_scrub_stamp":"2023-10-30T14:27:16.181071+0000","last_clean_scrub_stamp":"2023-10-30T14:27:16.181071+0000","objects_scrubbed":33,"log_size":2213,"log_dups_size":0,"ondisk_log_size":2213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:59:49.845658+0000","scrub_duration":0.12957070300000001,"objects_trimmed":0,"snaptrim_duration":1.073619836,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":129,"num_read_kb":12975,"num_write":1318,"num_write_kb":20116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"110'784","reported_seq":1084,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:56.157617+0000","last_change":"2023-10-30T14:26:56.157707+0000","last_active":"2023-10-30T14:26:56.157617+0000","last_peered":"2023-10-30T14:26:56.157617+0000","last_clean":"2023-10-30T14:26:56.157617+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T14:26:56.157617+0000","last_undegraded":"2023-10-30T14:26:56.157617+0000","last_fullsized":"2023-10-30T14:26:56.157617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'784","last_scrub_stamp":"2023-10-30T14:26:56.157430+0000","last_deep_scrub":"110'784","last_deep_scrub_stamp":"2023-10-30T14:26:56.157430+0000","last_clean_scrub_stamp":"2023-10-30T14:26:56.157430+0000","objects_scrubbed":50,"log_size":784,"log_dups_size":0,"ondisk_log_size":784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:19.819331+0000","scrub_duration":0.231260298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":304,"num_read_kb":264369,"num_write":877,"num_write_kb":48007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":477325,"num_omap_keys":851,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"110'2679","reported_seq":5374,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:17.160156+0000","last_change":"2023-10-30T14:27:17.160251+0000","last_active":"2023-10-30T14:27:17.160156+0000","last_peered":"2023-10-30T14:27:17.160156+0000","last_clean":"2023-10-30T14:27:17.160156+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T14:27:17.160156+0000","last_undegraded":"2023-10-30T14:27:17.160156+0000","last_fullsized":"2023-10-30T14:27:17.160156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2679","last_scrub_stamp":"2023-10-30T14:27:17.160038+0000","last_deep_scrub":"110'2679","last_deep_scrub_stamp":"2023-10-30T14:27:17.160038+0000","last_clean_scrub_stamp":"2023-10-30T14:27:17.160038+0000","objects_scrubbed":37,"log_size":2679,"log_dups_size":0,"ondisk_log_size":2679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:38.223350+0000","scrub_duration":0.126936049,"objects_trimmed":0,"snaptrim_duration":1.021840552,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":171,"num_read_kb":40337,"num_write":1456,"num_write_kb":34441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"110'1224","reported_seq":1535,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:04.243739+0000","last_change":"2023-10-30T14:27:04.243861+0000","last_active":"2023-10-30T14:27:04.243739+0000","last_peered":"2023-10-30T14:27:04.243739+0000","last_clean":"2023-10-30T14:27:04.243739+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T14:27:04.243739+0000","last_undegraded":"2023-10-30T14:27:04.243739+0000","last_fullsized":"2023-10-30T14:27:04.243739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1224","last_scrub_stamp":"2023-10-30T14:27:04.243569+0000","last_deep_scrub":"110'1224","last_deep_scrub_stamp":"2023-10-30T14:27:04.243569+0000","last_clean_scrub_stamp":"2023-10-30T14:27:04.243569+0000","objects_scrubbed":45,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:57:40.587906+0000","scrub_duration":0.244476464,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":255,"num_read_kb":502436,"num_write":1236,"num_write_kb":99955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":409468,"num_omap_keys":801,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"110'2069","reported_seq":4947,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:09.913752+0000","last_change":"2023-10-30T14:27:09.913836+0000","last_active":"2023-10-30T14:27:09.913752+0000","last_peered":"2023-10-30T14:27:09.913752+0000","last_clean":"2023-10-30T14:27:09.913752+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T14:27:09.913752+0000","last_undegraded":"2023-10-30T14:27:09.913752+0000","last_fullsized":"2023-10-30T14:27:09.913752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2069","last_scrub_stamp":"2023-10-30T14:27:09.913657+0000","last_deep_scrub":"110'2069","last_deep_scrub_stamp":"2023-10-30T14:27:09.913657+0000","last_clean_scrub_stamp":"2023-10-30T14:27:09.913657+0000","objects_scrubbed":41,"log_size":2069,"log_dups_size":0,"ondisk_log_size":2069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:21:51.262361+0000","scrub_duration":0.091522568999999998,"objects_trimmed":0,"snaptrim_duration":1.015373858,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":13189,"num_write":1249,"num_write_kb":15084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"110'817","reported_seq":1082,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:45.554415+0000","last_change":"2023-10-30T14:26:45.554415+0000","last_active":"2023-10-30T14:26:45.554415+0000","last_peered":"2023-10-30T14:26:45.554415+0000","last_clean":"2023-10-30T14:26:45.554415+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T14:26:45.554415+0000","last_undegraded":"2023-10-30T14:26:45.554415+0000","last_fullsized":"2023-10-30T14:26:45.554415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'817","last_scrub_stamp":"2023-10-30T14:26:45.554286+0000","last_deep_scrub":"110'817","last_deep_scrub_stamp":"2023-10-30T14:26:45.554286+0000","last_clean_scrub_stamp":"2023-10-30T14:26:45.554286+0000","objects_scrubbed":38,"log_size":817,"log_dups_size":0,"ondisk_log_size":817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:36.415824+0000","scrub_duration":0.061981296999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":159,"num_read_kb":369044,"num_write":787,"num_write_kb":79251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":283692,"num_omap_keys":523,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"110'2229","reported_seq":5189,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.797301+0000","last_change":"2023-10-30T14:27:00.797301+0000","last_active":"2023-10-30T14:27:00.797301+0000","last_peered":"2023-10-30T14:27:00.797301+0000","last_clean":"2023-10-30T14:27:00.797301+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T14:27:00.797301+0000","last_undegraded":"2023-10-30T14:27:00.797301+0000","last_fullsized":"2023-10-30T14:27:00.797301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2229","last_scrub_stamp":"2023-10-30T14:27:00.797179+0000","last_deep_scrub":"110'2229","last_deep_scrub_stamp":"2023-10-30T14:27:00.797179+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.797179+0000","objects_scrubbed":37,"log_size":2229,"log_dups_size":0,"ondisk_log_size":2229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:23.560124+0000","scrub_duration":0.084074704,"objects_trimmed":0,"snaptrim_duration":1.043790548,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":262,"num_read_kb":42307,"num_write":1336,"num_write_kb":30630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"110'1065","reported_seq":1395,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.937061+0000","last_change":"2023-10-30T14:26:42.937107+0000","last_active":"2023-10-30T14:26:42.937061+0000","last_peered":"2023-10-30T14:26:42.937061+0000","last_clean":"2023-10-30T14:26:42.937061+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T14:26:42.937061+0000","last_undegraded":"2023-10-30T14:26:42.937061+0000","last_fullsized":"2023-10-30T14:26:42.937061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1065","last_scrub_stamp":"2023-10-30T14:26:42.936998+0000","last_deep_scrub":"110'1065","last_deep_scrub_stamp":"2023-10-30T14:26:42.936998+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.936998+0000","objects_scrubbed":55,"log_size":1065,"log_dups_size":0,"ondisk_log_size":1065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:27.046930+0000","scrub_duration":0.131982027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777798,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":313,"num_read_kb":340671,"num_write":1085,"num_write_kb":63851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":501138,"num_omap_keys":760,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":11,"num_objects":2377,"num_object_clones":2,"num_object_copies":4754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":9389,"num_read_kb":1724409,"num_write":83589,"num_write_kb":1459751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140836,"ondisk_log_size":140836,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1285101348,"num_objects":1569,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1569,"num_whiteouts":0,"num_read":10652,"num_read_kb":12762809,"num_write":35655,"num_write_kb":2343865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13139950,"num_omap_keys":23453,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2570362880,"data_stored":2570202696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33559,"ondisk_log_size":33559,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575627,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578504,"kb_used_data":389120,"kb_used_omap":1,"kb_used_meta":189374,"kb_avail":93793336,"statfs":{"total":96636764160,"available":96044376064,"internally_reserved":0,"allocated":398458880,"data_stored":397803489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":193919435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.75600000000000001,"15min":0.79800000000000004},"min":{"1min":0.50700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.236,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.85799999999999998,"15min":0.90900000000000003},"min":{"1min":0.627,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.306,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.0449999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.81100000000000005,"15min":0.86899999999999999},"min":{"1min":0.52700000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2130000000000001,"5min":2.452,"15min":2.452},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.82799999999999996,"15min":0.84999999999999998},"min":{"1min":0.61599999999999999,"5min":0.495,"15min":0.47399999999999998},"max":{"1min":1.29,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.61599999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.78500000000000003,"15min":0.82599999999999996},"min":{"1min":0.67300000000000004,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.175,"5min":1.175,"15min":1.4350000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.171,"5min":0.89700000000000002,"15min":0.94899999999999995},"min":{"1min":0.79600000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.403,"5min":1.403,"15min":1.5569999999999999},"last":0.97499999999999998}]},{"osd":3,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.95799999999999996,"15min":0.96099999999999997},"min":{"1min":0.83199999999999996,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.335,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.127,"5min":1.04,"15min":1.012},"min":{"1min":0.86199999999999999,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.4390000000000001,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":1.0940000000000001}]},{"osd":4,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.91000000000000003,"15min":0.89600000000000002},"min":{"1min":0.56899999999999995,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.202,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":1.01},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.085,"15min":1.046},"min":{"1min":0.746,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.3400000000000001,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":1.177}]},{"osd":5,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.1319999999999999,"15min":1.1000000000000001},"min":{"1min":0.755,"5min":0.375,"15min":0.375},"max":{"1min":1.5049999999999999,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.127,"15min":1.1479999999999999},"min":{"1min":0.68999999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.206,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":1.125}]},{"osd":6,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.1120000000000001,"15min":1.107},"min":{"1min":0.76600000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.256,"5min":6.3070000000000004,"15min":6.3070000000000004},"last":0.871},{"interface":"front","average":{"1min":1.206,"5min":1.135,"15min":1.1799999999999999},"min":{"1min":0.69899999999999995,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.9089999999999998,"5min":2.9089999999999998,"15min":2.9089999999999998},"last":1.147}]}]},{"osd":6,"up_from":13,"seq":55834575630,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505392,"kb_used_data":339688,"kb_used_omap":1,"kb_used_meta":165694,"kb_avail":93866448,"statfs":{"total":96636764160,"available":96119242752,"internally_reserved":0,"allocated":347840512,"data_stored":347188935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169671111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.74099999999999999,"15min":0.79700000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.375},"max":{"1min":1.105,"5min":1.306,"15min":1.363},"last":0.627},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.85499999999999998,"15min":0.94099999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.89800000000000002}]},{"osd":1,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.72899999999999998,"15min":0.77700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.452,"5min":3.452,"15min":3.452},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.748,"15min":0.77800000000000002},"min":{"1min":0.48399999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.163,"5min":1.706,"15min":1.706},"last":0.86299999999999999}]},{"osd":2,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.80500000000000005,"15min":0.83599999999999997},"min":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.571},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.99299999999999999,"15min":0.94699999999999995},"min":{"1min":0.52300000000000002,"5min":0.502,"15min":0.502},"max":{"1min":3.6459999999999999,"5min":5.4480000000000004,"15min":5.4480000000000004},"last":0.97699999999999998}]},{"osd":3,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.04,"15min":1.0369999999999999},"min":{"1min":0.752,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":3.5169999999999999,"5min":3.5169999999999999,"15min":3.5169999999999999},"last":1.054},{"interface":"front","average":{"1min":1.228,"5min":1.1240000000000001,"15min":1.1220000000000001},"min":{"1min":0.77700000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":3.5830000000000002,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":0.81000000000000005}]},{"osd":4,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.57199999999999995,"5min":0.434,"15min":0.434},"max":{"1min":1.2410000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.98899999999999999,"15min":1.026},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.452,"5min":1.595,"15min":1.595},"last":0.85099999999999998}]},{"osd":5,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.1599999999999999,"15min":1.0429999999999999},"min":{"1min":0.59899999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.9590000000000001,"5min":3.0219999999999998,"15min":3.0219999999999998},"last":1.1240000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":1.0069999999999999,"15min":1.0429999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.835,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":1.0189999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:27:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2410000000000001,"15min":1.1990000000000001},"min":{"1min":0.65300000000000002,"5min":0.58699999999999997,"15min":0.52600000000000002},"max":{"1min":2.0379999999999998,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":1.198},{"interface":"front","average":{"1min":1.034,"5min":1.1259999999999999,"15min":1.1279999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.9350000000000001,"5min":2.806,"15min":2.806},"last":1.0920000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575627,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570756,"kb_used_data":368956,"kb_used_omap":1,"kb_used_meta":201790,"kb_avail":93801084,"statfs":{"total":96636764160,"available":96052310016,"internally_reserved":0,"allocated":377810944,"data_stored":377158158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206633414},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.84599999999999997},"min":{"1min":0.439,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1120000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.93700000000000006,"15min":0.95699999999999996},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.373,"5min":1.617,"15min":1.9219999999999999},"last":1.0549999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.81799999999999995,"15min":0.83899999999999997},"min":{"1min":0.48899999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.3},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.77100000000000002,"15min":0.83999999999999997},"min":{"1min":0.53700000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.774,"5min":4.774,"15min":4.774},"last":0.78600000000000003}]},{"osd":2,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85199999999999998,"15min":0.88100000000000001},"min":{"1min":0.58599999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.111,"5min":1.2589999999999999,"15min":1.331},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.84699999999999998},"min":{"1min":0.501,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.1879999999999999,"5min":1.331,"15min":1.796},"last":0.92500000000000004}]},{"osd":3,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.96999999999999997,"15min":1.0009999999999999},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3100000000000001,"5min":1.536,"15min":1.536},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.036,"5min":0.92600000000000005,"15min":0.99099999999999999},"min":{"1min":0.51800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":2.0030000000000001},"last":0.85199999999999998}]},{"osd":4,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.88800000000000001,"15min":0.877},"min":{"1min":0.374,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.087,"5min":2.8170000000000002,"15min":2.8170000000000002},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.155,"5min":1.0609999999999999,"15min":1.036},"min":{"1min":0.45100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.4690000000000001,"5min":2.2400000000000002,"15min":2.262},"last":1.22}]},{"osd":6,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.071,"15min":1.1220000000000001},"min":{"1min":0.40300000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3380000000000001,"5min":2.3090000000000002,"15min":2.7949999999999999},"last":1.173},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.038,"15min":1.054},"min":{"1min":0.38800000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.4350000000000001,"5min":2.161,"15min":2.1629999999999998},"last":1.002}]},{"osd":7,"last update":"Mon Oct 30 14:27:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.026,"15min":1.079},"min":{"1min":0.33800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":1.391,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.133,"15min":1.147},"min":{"1min":0.436,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.587,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.0249999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469348,"kb_used_data":267484,"kb_used_omap":1,"kb_used_meta":201854,"kb_avail":93902492,"statfs":{"total":96636764160,"available":96156151808,"internally_reserved":0,"allocated":273903616,"data_stored":273238103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206698950},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.73599999999999999,"15min":0.80500000000000005},"min":{"1min":0.439,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.3029999999999999,"5min":1.3500000000000001,"15min":1.446},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.69199999999999995,"15min":0.81200000000000006},"min":{"1min":0.56599999999999995,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0940000000000001,"5min":1.659,"15min":1.659},"last":0.67600000000000005}]},{"osd":1,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.80100000000000005,"15min":0.79400000000000004},"min":{"1min":0.54000000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.86899999999999999,"15min":0.85399999999999998},"min":{"1min":0.49199999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.95399999999999996}]},{"osd":2,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.747,"15min":0.85299999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.296,"5min":1.607,"15min":1.607},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83199999999999996,"15min":0.90900000000000003},"min":{"1min":0.53400000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.196,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.77500000000000002}]},{"osd":3,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0029999999999999,"15min":0.97899999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.107,"15min":1.083},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":1.125}]},{"osd":5,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.99299999999999999,"15min":1.042},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.391,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.004,"5min":0.94399999999999995,"15min":0.96299999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.85499999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.97299999999999998,"15min":0.96499999999999997},"min":{"1min":0.42399999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.071,"5min":1.042,"15min":1.0700000000000001},"min":{"1min":0.51800000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.534,"5min":1.677,"15min":1.728},"last":1.0880000000000001}]},{"osd":7,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.133,"15min":1.1040000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.196},{"interface":"front","average":{"1min":1.181,"5min":1.1899999999999999,"15min":1.179},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.006}]}]},{"osd":3,"up_from":13,"seq":55834575625,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597696,"kb_used_data":382008,"kb_used_omap":1,"kb_used_meta":215678,"kb_avail":93774144,"statfs":{"total":96636764160,"available":96024723456,"internally_reserved":0,"allocated":391176192,"data_stored":390510799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":220854728},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.374,"15min":0.40300000000000002},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.72099999999999997,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.46700000000000003,"15min":0.48999999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.56399999999999995,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.41899999999999998}]},{"osd":1,"last update":"Mon Oct 30 14:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.47499999999999998,"15min":0.45600000000000002},"min":{"1min":0.379,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51100000000000001,"15min":0.54400000000000004},"min":{"1min":0.32800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.52600000000000002}]},{"osd":2,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54400000000000004,"15min":0.59199999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.66300000000000003,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.433,"5min":0.29999999999999999,"15min":0.29299999999999998},"max":{"1min":0.69699999999999995,"5min":1.73,"15min":1.73},"last":0.55300000000000005}]},{"osd":4,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.76600000000000001,"15min":0.79800000000000004},"min":{"1min":0.57699999999999996,"5min":0.379,"15min":0.379},"max":{"1min":0.98599999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.78100000000000003,"15min":0.81200000000000006},"min":{"1min":0.52500000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.89200000000000002,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.89900000000000002}]},{"osd":5,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.76100000000000001,"15min":0.81799999999999995},"min":{"1min":0.55700000000000005,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0209999999999999,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.78300000000000003,"15min":0.84499999999999997},"min":{"1min":0.58399999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.032,"5min":1.486,"15min":1.486},"last":0.79300000000000004}]},{"osd":6,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83999999999999997,"15min":0.88},"min":{"1min":0.625,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1830000000000001,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.90200000000000002,"15min":0.93600000000000005},"min":{"1min":0.72399999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.224,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.95699999999999996}]},{"osd":7,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.80800000000000005,"15min":0.85599999999999998},"min":{"1min":0.65500000000000003,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.052,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.77900000000000003,"15min":0.81599999999999995},"min":{"1min":0.60899999999999999,"5min":0.501,"15min":0.501},"max":{"1min":0.93400000000000005,"5min":1.593,"15min":1.593},"last":0.84699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303572,"kb_used_data":200268,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":94068268,"statfs":{"total":96636764160,"available":96325906432,"internally_reserved":0,"allocated":205074432,"data_stored":204423364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105773510},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42299999999999999,"15min":0.45600000000000002},"min":{"1min":0.26100000000000001,"5min":0.246,"15min":0.23899999999999999},"max":{"1min":0.72599999999999998,"5min":0.747,"15min":0.89300000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.499,"15min":0.51300000000000001},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":0.63900000000000001,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":0.51100000000000001}]},{"osd":1,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.41699999999999998,"15min":0.44600000000000001},"min":{"1min":0.307,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70099999999999996,"5min":1.956,"15min":1.956},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.35099999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.153,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.47099999999999997}]},{"osd":3,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55700000000000005,"15min":0.57399999999999995},"min":{"1min":0.40699999999999997,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.2010000000000001,"5min":1.988,"15min":1.988},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.502,"15min":0.51800000000000002},"min":{"1min":0.32600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69599999999999995,"5min":2.335,"15min":2.335},"last":0.48899999999999999}]},{"osd":4,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.80800000000000005,"15min":0.82599999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.085,"5min":2.302,"15min":2.302},"last":1.163},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84299999999999997,"15min":0.83699999999999997},"min":{"1min":0.66900000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0169999999999999,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.84499999999999997}]},{"osd":5,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.77000000000000002,"15min":0.80400000000000005},"min":{"1min":0.59099999999999997,"5min":0.501,"15min":0.501},"max":{"1min":1.028,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.82999999999999996,"15min":0.84199999999999997},"min":{"1min":0.68400000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0940000000000001,"5min":1.875,"15min":1.875},"last":1.0840000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.82299999999999995,"15min":0.82899999999999996},"min":{"1min":0.63100000000000001,"5min":0.502,"15min":0.48099999999999998},"max":{"1min":1.123,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.82899999999999996,"15min":0.86299999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.2130000000000001,"5min":1.921,"15min":1.921},"last":0.98799999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85899999999999999,"15min":0.86099999999999999},"min":{"1min":0.66300000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1100000000000001,"5min":4.5739999999999998,"15min":4.5739999999999998},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.88500000000000001,"15min":0.91400000000000003},"min":{"1min":0.64700000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.202,"5min":2.375,"15min":2.375},"last":1.04}]}]},{"osd":1,"up_from":13,"seq":55834575628,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494860,"kb_used_data":329092,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93876980,"statfs":{"total":96636764160,"available":96130027520,"internally_reserved":0,"allocated":336990208,"data_stored":336334619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.45800000000000002},"min":{"1min":0.29999999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.72599999999999998,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.49099999999999999,"15min":0.51400000000000001},"min":{"1min":0.36799999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86799999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0469999999999999,"5min":1.954,"15min":1.954},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.50600000000000001,"15min":0.49299999999999999},"min":{"1min":0.28100000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":5.7039999999999997,"5min":5.7039999999999997,"15min":5.7039999999999997},"last":0.48999999999999999}]},{"osd":3,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55100000000000005,"15min":0.55000000000000004},"min":{"1min":0.44600000000000001,"5min":0.35499999999999998,"15min":0.35099999999999998},"max":{"1min":1.3220000000000001,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3600000000000001,"5min":2.625,"15min":2.625},"last":0.68899999999999995}]},{"osd":4,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.79200000000000004,"15min":0.82999999999999996},"min":{"1min":0.66000000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.92900000000000005,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.82199999999999995,"15min":0.872},"min":{"1min":0.67300000000000004,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0309999999999999,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.77700000000000002}]},{"osd":5,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.78800000000000003,"15min":0.86299999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.004,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":0.753},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.82099999999999995,"15min":0.88800000000000001},"min":{"1min":0.52200000000000002,"5min":0.499,"15min":0.499},"max":{"1min":1.167,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.66700000000000004}]},{"osd":6,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.89900000000000002,"15min":0.84499999999999997},"min":{"1min":0.73999999999999999,"5min":0.624,"15min":0.56499999999999995},"max":{"1min":1.01,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":0.879},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.81000000000000005,"15min":0.88600000000000001},"min":{"1min":0.56399999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.99299999999999999,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.623,"5min":0.59599999999999997,"15min":0.56699999999999995},"max":{"1min":1.085,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.83999999999999997,"15min":0.84999999999999998},"min":{"1min":0.66900000000000004,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0660000000000001,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.96499999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575625,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425348,"kb_used_data":245756,"kb_used_omap":1,"kb_used_meta":179582,"kb_avail":93946492,"statfs":{"total":96636764160,"available":96201207808,"internally_reserved":0,"allocated":251654144,"data_stored":251001762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":183892422},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.377,"15min":0.40100000000000002},"min":{"1min":0.26400000000000001,"5min":0.247,"15min":0.23499999999999999},"max":{"1min":0.61699999999999999,"5min":0.66100000000000003,"15min":0.94999999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48499999999999999,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65800000000000003,"5min":0.82299999999999995,"15min":1.0229999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.45400000000000001,"15min":0.46200000000000002},"min":{"1min":0.33400000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47699999999999998,"15min":0.496},"min":{"1min":0.38100000000000001,"5min":0.30499999999999999,"15min":0.29099999999999998},"max":{"1min":0.83599999999999997,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.64600000000000002}]},{"osd":3,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.91900000000000004,"5min":1.236,"15min":1.236},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51800000000000002,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":0.68000000000000005}]},{"osd":4,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78200000000000003,"15min":0.79700000000000004},"min":{"1min":0.58899999999999997,"5min":0.503,"15min":0.503},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.747,"15min":0.80000000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91200000000000003,"5min":1.012,"15min":1.2210000000000001},"last":0.91100000000000003}]},{"osd":5,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.78700000000000003,"15min":0.80900000000000005},"min":{"1min":0.61299999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.80700000000000005,"15min":0.84499999999999997},"min":{"1min":0.626,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.0189999999999999}]},{"osd":6,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.80800000000000005,"15min":0.83799999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.99399999999999999,"5min":1.222,"15min":1.486},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.80600000000000005,"15min":0.82899999999999996},"min":{"1min":0.66900000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.89900000000000002}]},{"osd":7,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.82399999999999995,"15min":0.84699999999999998},"min":{"1min":0.70999999999999996,"5min":0.625,"15min":0.58599999999999997},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.054},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81799999999999995,"15min":0.86199999999999999},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0680000000000001,"5min":1.1559999999999999,"15min":1.4419999999999999},"last":1.0049999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249647104,"data_stored":249627559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335572992,"data_stored":335549765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203661312,"data_stored":203639529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389746688,"data_stored":389725939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":272470016,"data_stored":272453238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376397824,"data_stored":376374323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":346419200,"data_stored":346404076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":396447744,"data_stored":396428267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":11,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:27:46.036 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:46.037 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T13:23:18.926892+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=23, tm_sec=18, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=14, tm_min=26, tm_sec=38, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T14:27:46.037 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T14:28:06.039 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T14:28:06.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.189+0000 7f0366baa640 1 Processor -- start 2023-10-30T14:28:06.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.189+0000 7f0366baa640 1 -- start start 2023-10-30T14:28:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.193+0000 7f0366baa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0360069e90 0x7f036006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.193+0000 7f0366baa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0360153d80 con 0x7f0360070a00 2023-10-30T14:28:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.193+0000 7f0366baa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03600c5c50 con 0x7f03601539a0 2023-10-30T14:28:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.193+0000 7f0366baa640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f036006fbb0 con 0x7f0360069e90 2023-10-30T14:28:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.193+0000 7f0365147640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0360069e90 0x7f036006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.193+0000 7f035ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0360070a00 0x7f0360070e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46926/0 (socket says 172.21.15.46:46926) 2023-10-30T14:28:06.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.193+0000 7f035ffff640 1 -- 172.21.15.46:0/1262796738 learned_addr learned my addr 172.21.15.46:0/1262796738 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:28:06.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.193+0000 7f0365147640 1 -- 172.21.15.46:0/1262796738 >> v1:172.21.15.154:6789/0 conn(0x7f03601539a0 legacy=0x7f036015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.193+0000 7f0365147640 1 -- 172.21.15.46:0/1262796738 >> v1:172.21.15.46:6789/0 conn(0x7f0360070a00 legacy=0x7f0360070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.193+0000 7f0365147640 1 -- 172.21.15.46:0/1262796738 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03600c5f30 con 0x7f0360069e90 2023-10-30T14:28:06.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.193+0000 7f035f7fe640 1 -- 172.21.15.46:0/1262796738 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099947043 0 0) 0x7f0360153d80 con 0x7f0360070a00 2023-10-30T14:28:06.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.193+0000 7f0365147640 1 --2- 172.21.15.46:0/1262796738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0360069e90 0x7f036006a300 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f0358009d90 tx=0x7f035801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebbed1e4ca945d80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:06.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f035f7fe640 1 -- 172.21.15.46:0/1262796738 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03580a5860 con 0x7f0360069e90 2023-10-30T14:28:06.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f035f7fe640 1 -- 172.21.15.46:0/1262796738 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03580a59c0 con 0x7f0360069e90 2023-10-30T14:28:06.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f035f7fe640 1 -- 172.21.15.46:0/1262796738 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03580a5d50 con 0x7f0360069e90 2023-10-30T14:28:06.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 -- 172.21.15.46:0/1262796738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0360069e90 msgr2=0x7f036006a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 --2- 172.21.15.46:0/1262796738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0360069e90 0x7f036006a300 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f0358009d90 tx=0x7f035801c890 comp rx=0 tx=0).stop 2023-10-30T14:28:06.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 -- 172.21.15.46:0/1262796738 shutdown_connections 2023-10-30T14:28:06.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 --2- 172.21.15.46:0/1262796738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0360069e90 0x7f036006a300 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 -- 172.21.15.46:0/1262796738 >> 172.21.15.46:0/1262796738 conn(0x7f03600c7910 msgr2=0x7f03600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:06.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 -- 172.21.15.46:0/1262796738 shutdown_connections 2023-10-30T14:28:06.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 -- 172.21.15.46:0/1262796738 wait complete. 2023-10-30T14:28:06.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 Processor -- start 2023-10-30T14:28:06.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 -- start start 2023-10-30T14:28:06.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0360069e90 0x7f03600c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0360070a00 0x7f036014f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03601539a0 0x7f03601469a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03600c34b0 con 0x7f0360069e90 2023-10-30T14:28:06.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0360143060 con 0x7f0360070a00 2023-10-30T14:28:06.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0366baa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03600ca890 con 0x7f03601539a0 2023-10-30T14:28:06.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0364946640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0360069e90 0x7f03600c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0364946640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0360069e90 0x7f03600c2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45252/0 (socket says 172.21.15.46:45252) 2023-10-30T14:28:06.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.197+0000 7f0364946640 1 -- 172.21.15.46:0/3279795379 learned_addr learned my addr 172.21.15.46:0/3279795379 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:28:06.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f0365147640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03601539a0 0x7f03601469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f035ffff640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0360070a00 0x7f036014f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f0364946640 1 -- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03601539a0 msgr2=0x7f03601469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f0364946640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03601539a0 0x7f03601469a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f0364946640 1 -- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0360070a00 msgr2=0x7f036014f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f0364946640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0360070a00 0x7f036014f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f0364946640 1 -- 172.21.15.46:0/3279795379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0358009a70 con 0x7f0360069e90 2023-10-30T14:28:06.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f0365147640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03601539a0 0x7f03601469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:06.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f035ffff640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0360070a00 0x7f036014f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:06.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f0364946640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0360069e90 0x7f03600c2f70 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f035400c3f0 tx=0x7f035400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f035d7fa640 1 -- 172.21.15.46:0/3279795379 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03540052d0 con 0x7f0360069e90 2023-10-30T14:28:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f035d7fa640 1 -- 172.21.15.46:0/3279795379 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0354005430 con 0x7f0360069e90 2023-10-30T14:28:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f0366baa640 1 -- 172.21.15.46:0/3279795379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03600c4200 con 0x7f0360069e90 2023-10-30T14:28:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f035d7fa640 1 -- 172.21.15.46:0/3279795379 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354003840 con 0x7f0360069e90 2023-10-30T14:28:06.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f0366baa640 1 -- 172.21.15.46:0/3279795379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03600c3bb0 con 0x7f0360069e90 2023-10-30T14:28:06.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f0366baa640 1 -- 172.21.15.46:0/3279795379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036004e170 con 0x7f0360069e90 2023-10-30T14:28:06.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f035d7fa640 1 -- 172.21.15.46:0/3279795379 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f03540039a0 con 0x7f0360069e90 2023-10-30T14:28:06.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f035d7fa640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03440664d0 0x7f0344068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.201+0000 7f035d7fa640 1 -- 172.21.15.46:0/3279795379 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f0354084040 con 0x7f0360069e90 2023-10-30T14:28:06.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.205+0000 7f035ffff640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03440664d0 0x7f0344068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.205+0000 7f035ffff640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03440664d0 0x7f0344068990 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f03600ca0a0 tx=0x7f034c007c60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:06.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.205+0000 7f035d7fa640 1 -- 172.21.15.46:0/3279795379 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f03540b5090 con 0x7f0360069e90 2023-10-30T14:28:06.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.365+0000 7f0366baa640 1 -- 172.21.15.46:0/3279795379 --> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f03600c6ab0 con 0x7f03440664d0 2023-10-30T14:28:06.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.385+0000 7f035d7fa640 1 -- 172.21.15.46:0/3279795379 <== mgr.4102 v2:172.21.15.154:6832/313953570 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+295045 (secure 0 0 0) 0x7f034c007270 con 0x7f03440664d0 2023-10-30T14:28:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:28:06.396 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-10-30T14:28:06.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.389+0000 7f0366baa640 1 -- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03440664d0 msgr2=0x7f0344068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.389+0000 7f0366baa640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03440664d0 0x7f0344068990 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f03600ca0a0 tx=0x7f034c007c60 comp rx=0 tx=0).stop 2023-10-30T14:28:06.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.389+0000 7f0366baa640 1 -- 172.21.15.46:0/3279795379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0360069e90 msgr2=0x7f03600c2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.389+0000 7f0366baa640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0360069e90 0x7f03600c2f70 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f035400c3f0 tx=0x7f035400c8b0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.389+0000 7f0366baa640 1 -- 172.21.15.46:0/3279795379 shutdown_connections 2023-10-30T14:28:06.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.389+0000 7f0366baa640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f03440664d0 0x7f0344068990 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.389+0000 7f0366baa640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03601539a0 0x7f03601469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.389+0000 7f0366baa640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0360070a00 0x7f036014f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.389+0000 7f0366baa640 1 --2- 172.21.15.46:0/3279795379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0360069e90 0x7f03600c2f70 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.389+0000 7f0366baa640 1 -- 172.21.15.46:0/3279795379 >> 172.21.15.46:0/3279795379 conn(0x7f03600c7910 msgr2=0x7f036014e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:06.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.389+0000 7f0366baa640 1 -- 172.21.15.46:0/3279795379 shutdown_connections 2023-10-30T14:28:06.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.389+0000 7f0366baa640 1 -- 172.21.15.46:0/3279795379 wait complete. 2023-10-30T14:28:06.414 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":2053,"stamp":"2023-10-30T14:28:06.158894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1285691727,"num_objects":3948,"num_object_clones":2,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":20157,"num_read_kb":14487318,"num_write":119403,"num_write_kb":3806032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13139950,"num_omap_keys":23453,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":174582,"ondisk_log_size":174582,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3945476,"kb_used_data":2522372,"kb_used_omap":12,"kb_used_meta":1423027,"kb_avail":751029244,"statfs":{"total":773094113280,"available":769053945856,"internally_reserved":0,"allocated":2582908928,"data_stored":2577659229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1457180216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004090"},"pg_stats":[{"pgid":"3.3f","version":"110'2193","reported_seq":5017,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:17.161738+0000","last_change":"2023-10-30T14:27:17.161738+0000","last_active":"2023-10-30T14:27:17.161738+0000","last_peered":"2023-10-30T14:27:17.161738+0000","last_clean":"2023-10-30T14:27:17.161738+0000","last_became_active":"2023-10-30T13:23:19.718496+0000","last_became_peered":"2023-10-30T13:23:19.718496+0000","last_unstale":"2023-10-30T14:27:17.161738+0000","last_undegraded":"2023-10-30T14:27:17.161738+0000","last_fullsized":"2023-10-30T14:27:17.161738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2193","last_scrub_stamp":"2023-10-30T14:27:17.161580+0000","last_deep_scrub":"110'2193","last_deep_scrub_stamp":"2023-10-30T14:27:17.161580+0000","last_clean_scrub_stamp":"2023-10-30T14:27:17.161580+0000","objects_scrubbed":32,"log_size":2193,"log_dups_size":0,"ondisk_log_size":2193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:00.032759+0000","scrub_duration":0.104386056,"objects_trimmed":0,"snaptrim_duration":0.84725695499999998,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":124,"num_read_kb":10662,"num_write":1276,"num_write_kb":15944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"110'2011","reported_seq":4901,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:06.287715+0000","last_change":"2023-10-30T14:27:06.287715+0000","last_active":"2023-10-30T14:27:06.287715+0000","last_peered":"2023-10-30T14:27:06.287715+0000","last_clean":"2023-10-30T14:27:06.287715+0000","last_became_active":"2023-10-30T13:23:19.686863+0000","last_became_peered":"2023-10-30T13:23:19.686863+0000","last_unstale":"2023-10-30T14:27:06.287715+0000","last_undegraded":"2023-10-30T14:27:06.287715+0000","last_fullsized":"2023-10-30T14:27:06.287715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2011","last_scrub_stamp":"2023-10-30T14:27:06.287549+0000","last_deep_scrub":"110'2011","last_deep_scrub_stamp":"2023-10-30T14:27:06.287549+0000","last_clean_scrub_stamp":"2023-10-30T14:27:06.287549+0000","objects_scrubbed":40,"log_size":2011,"log_dups_size":0,"ondisk_log_size":2011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:35:24.021901+0000","scrub_duration":0.052013427000000001,"objects_trimmed":0,"snaptrim_duration":0.406256491,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":133,"num_read_kb":21158,"num_write":1230,"num_write_kb":16620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"110'2124","reported_seq":4968,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:08.843240+0000","last_change":"2023-10-30T14:27:08.843240+0000","last_active":"2023-10-30T14:27:08.843240+0000","last_peered":"2023-10-30T14:27:08.843240+0000","last_clean":"2023-10-30T14:27:08.843240+0000","last_became_active":"2023-10-30T13:23:19.686356+0000","last_became_peered":"2023-10-30T13:23:19.686356+0000","last_unstale":"2023-10-30T14:27:08.843240+0000","last_undegraded":"2023-10-30T14:27:08.843240+0000","last_fullsized":"2023-10-30T14:27:08.843240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2124","last_scrub_stamp":"2023-10-30T14:27:08.843106+0000","last_deep_scrub":"110'2124","last_deep_scrub_stamp":"2023-10-30T14:27:08.843106+0000","last_clean_scrub_stamp":"2023-10-30T14:27:08.843106+0000","objects_scrubbed":29,"log_size":2124,"log_dups_size":0,"ondisk_log_size":2124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:20:20.621247+0000","scrub_duration":0.041951362999999998,"objects_trimmed":0,"snaptrim_duration":1.030559456,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":129,"num_read_kb":29825,"num_write":1270,"num_write_kb":28412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"110'2055","reported_seq":4905,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:07.083889+0000","last_change":"2023-10-30T14:27:07.083934+0000","last_active":"2023-10-30T14:27:07.083889+0000","last_peered":"2023-10-30T14:27:07.083889+0000","last_clean":"2023-10-30T14:27:07.083889+0000","last_became_active":"2023-10-30T13:24:00.704829+0000","last_became_peered":"2023-10-30T13:24:00.704829+0000","last_unstale":"2023-10-30T14:27:07.083889+0000","last_undegraded":"2023-10-30T14:27:07.083889+0000","last_fullsized":"2023-10-30T14:27:07.083889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2055","last_scrub_stamp":"2023-10-30T14:27:07.083807+0000","last_deep_scrub":"110'2055","last_deep_scrub_stamp":"2023-10-30T14:27:07.083807+0000","last_clean_scrub_stamp":"2023-10-30T14:27:07.083807+0000","objects_scrubbed":25,"log_size":2055,"log_dups_size":0,"ondisk_log_size":2055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:33.332360+0000","scrub_duration":0.034227107,"objects_trimmed":0,"snaptrim_duration":0.361435862,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":124,"num_read_kb":7939,"num_write":1184,"num_write_kb":24239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"110'2044","reported_seq":4912,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:47.071955+0000","last_change":"2023-10-30T14:27:47.072096+0000","last_active":"2023-10-30T14:27:47.071955+0000","last_peered":"2023-10-30T14:27:47.071955+0000","last_clean":"2023-10-30T14:27:47.071955+0000","last_became_active":"2023-10-30T13:23:19.719140+0000","last_became_peered":"2023-10-30T13:23:19.719140+0000","last_unstale":"2023-10-30T14:27:47.071955+0000","last_undegraded":"2023-10-30T14:27:47.071955+0000","last_fullsized":"2023-10-30T14:27:47.071955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2044","last_scrub_stamp":"2023-10-30T14:27:47.071743+0000","last_deep_scrub":"110'2044","last_deep_scrub_stamp":"2023-10-30T14:27:47.071743+0000","last_clean_scrub_stamp":"2023-10-30T14:27:47.071743+0000","objects_scrubbed":35,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:52:37.062789+0000","scrub_duration":0.102987562,"objects_trimmed":0,"snaptrim_duration":0.80853723300000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":41266,"num_write":1244,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"110'2251","reported_seq":5060,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:07.080029+0000","last_change":"2023-10-30T14:27:07.080296+0000","last_active":"2023-10-30T14:27:07.080029+0000","last_peered":"2023-10-30T14:27:07.080029+0000","last_clean":"2023-10-30T14:27:07.080029+0000","last_became_active":"2023-10-30T13:23:19.702502+0000","last_became_peered":"2023-10-30T13:23:19.702502+0000","last_unstale":"2023-10-30T14:27:07.080029+0000","last_undegraded":"2023-10-30T14:27:07.080029+0000","last_fullsized":"2023-10-30T14:27:07.080029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2251","last_scrub_stamp":"2023-10-30T14:27:07.079781+0000","last_deep_scrub":"110'2251","last_deep_scrub_stamp":"2023-10-30T14:27:07.079781+0000","last_clean_scrub_stamp":"2023-10-30T14:27:07.079781+0000","objects_scrubbed":42,"log_size":2251,"log_dups_size":0,"ondisk_log_size":2251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:47.980581+0000","scrub_duration":0.119106473,"objects_trimmed":0,"snaptrim_duration":0.94867293500000005,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":141,"num_read_kb":27429,"num_write":1290,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"110'2185","reported_seq":5044,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:12.196491+0000","last_change":"2023-10-30T14:27:12.196618+0000","last_active":"2023-10-30T14:27:12.196491+0000","last_peered":"2023-10-30T14:27:12.196491+0000","last_clean":"2023-10-30T14:27:12.196491+0000","last_became_active":"2023-10-30T13:23:19.717548+0000","last_became_peered":"2023-10-30T13:23:19.717548+0000","last_unstale":"2023-10-30T14:27:12.196491+0000","last_undegraded":"2023-10-30T14:27:12.196491+0000","last_fullsized":"2023-10-30T14:27:12.196491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2185","last_scrub_stamp":"2023-10-30T14:27:12.196356+0000","last_deep_scrub":"110'2185","last_deep_scrub_stamp":"2023-10-30T14:27:12.196356+0000","last_clean_scrub_stamp":"2023-10-30T14:27:12.196356+0000","objects_scrubbed":40,"log_size":2185,"log_dups_size":0,"ondisk_log_size":2185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:32.138507+0000","scrub_duration":0.11536558199999999,"objects_trimmed":0,"snaptrim_duration":0.71511574600000005,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":146,"num_read_kb":25110,"num_write":1316,"num_write_kb":20043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"110'2147","reported_seq":4969,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:20.681195+0000","last_change":"2023-10-30T14:27:20.681264+0000","last_active":"2023-10-30T14:27:20.681195+0000","last_peered":"2023-10-30T14:27:20.681195+0000","last_clean":"2023-10-30T14:27:20.681195+0000","last_became_active":"2023-10-30T13:24:00.989938+0000","last_became_peered":"2023-10-30T13:24:00.989938+0000","last_unstale":"2023-10-30T14:27:20.681195+0000","last_undegraded":"2023-10-30T14:27:20.681195+0000","last_fullsized":"2023-10-30T14:27:20.681195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2147","last_scrub_stamp":"2023-10-30T14:27:20.681102+0000","last_deep_scrub":"110'2147","last_deep_scrub_stamp":"2023-10-30T14:27:20.681102+0000","last_clean_scrub_stamp":"2023-10-30T14:27:20.681102+0000","objects_scrubbed":38,"log_size":2147,"log_dups_size":0,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:10:24.247496+0000","scrub_duration":0.050065129,"objects_trimmed":0,"snaptrim_duration":0.90238370000000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":124,"num_read_kb":25628,"num_write":1232,"num_write_kb":20540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"110'2178","reported_seq":5020,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:23.177373+0000","last_change":"2023-10-30T14:27:23.177584+0000","last_active":"2023-10-30T14:27:23.177373+0000","last_peered":"2023-10-30T14:27:23.177373+0000","last_clean":"2023-10-30T14:27:23.177373+0000","last_became_active":"2023-10-30T13:23:19.723318+0000","last_became_peered":"2023-10-30T13:23:19.723318+0000","last_unstale":"2023-10-30T14:27:23.177373+0000","last_undegraded":"2023-10-30T14:27:23.177373+0000","last_fullsized":"2023-10-30T14:27:23.177373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2178","last_scrub_stamp":"2023-10-30T14:27:23.177181+0000","last_deep_scrub":"110'2178","last_deep_scrub_stamp":"2023-10-30T14:27:23.177181+0000","last_clean_scrub_stamp":"2023-10-30T14:27:23.177181+0000","objects_scrubbed":41,"log_size":2178,"log_dups_size":0,"ondisk_log_size":2178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:10:06.566411+0000","scrub_duration":0.124464532,"objects_trimmed":0,"snaptrim_duration":0.81280227199999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":24548,"num_write":1310,"num_write_kb":19821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"110'2110","reported_seq":4983,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:08.423266+0000","last_change":"2023-10-30T14:27:08.423413+0000","last_active":"2023-10-30T14:27:08.423266+0000","last_peered":"2023-10-30T14:27:08.423266+0000","last_clean":"2023-10-30T14:27:08.423266+0000","last_became_active":"2023-10-30T13:23:19.702535+0000","last_became_peered":"2023-10-30T13:23:19.702535+0000","last_unstale":"2023-10-30T14:27:08.423266+0000","last_undegraded":"2023-10-30T14:27:08.423266+0000","last_fullsized":"2023-10-30T14:27:08.423266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2110","last_scrub_stamp":"2023-10-30T14:27:08.423062+0000","last_deep_scrub":"110'2110","last_deep_scrub_stamp":"2023-10-30T14:27:08.423062+0000","last_clean_scrub_stamp":"2023-10-30T14:27:08.423062+0000","objects_scrubbed":37,"log_size":2110,"log_dups_size":0,"ondisk_log_size":2110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:05:31.451500+0000","scrub_duration":0.11051124900000001,"objects_trimmed":0,"snaptrim_duration":0.53668512800000001,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":148,"num_read_kb":64678,"num_write":1273,"num_write_kb":32495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"110'2046","reported_seq":4931,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:13.410232+0000","last_change":"2023-10-30T14:27:13.410351+0000","last_active":"2023-10-30T14:27:13.410232+0000","last_peered":"2023-10-30T14:27:13.410232+0000","last_clean":"2023-10-30T14:27:13.410232+0000","last_became_active":"2023-10-30T13:23:19.725298+0000","last_became_peered":"2023-10-30T13:23:19.725298+0000","last_unstale":"2023-10-30T14:27:13.410232+0000","last_undegraded":"2023-10-30T14:27:13.410232+0000","last_fullsized":"2023-10-30T14:27:13.410232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2046","last_scrub_stamp":"2023-10-30T14:27:13.410056+0000","last_deep_scrub":"110'2046","last_deep_scrub_stamp":"2023-10-30T14:27:13.410056+0000","last_clean_scrub_stamp":"2023-10-30T14:27:13.410056+0000","objects_scrubbed":38,"log_size":2046,"log_dups_size":0,"ondisk_log_size":2046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:58.219570+0000","scrub_duration":0.113389077,"objects_trimmed":0,"snaptrim_duration":0.471346551,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":139,"num_read_kb":25550,"num_write":1254,"num_write_kb":16915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"110'2118","reported_seq":4973,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:58.807440+0000","last_change":"2023-10-30T14:26:58.807556+0000","last_active":"2023-10-30T14:26:58.807440+0000","last_peered":"2023-10-30T14:26:58.807440+0000","last_clean":"2023-10-30T14:26:58.807440+0000","last_became_active":"2023-10-30T13:23:19.719710+0000","last_became_peered":"2023-10-30T13:23:19.719710+0000","last_unstale":"2023-10-30T14:26:58.807440+0000","last_undegraded":"2023-10-30T14:26:58.807440+0000","last_fullsized":"2023-10-30T14:26:58.807440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2118","last_scrub_stamp":"2023-10-30T14:26:58.807287+0000","last_deep_scrub":"110'2118","last_deep_scrub_stamp":"2023-10-30T14:26:58.807287+0000","last_clean_scrub_stamp":"2023-10-30T14:26:58.807287+0000","objects_scrubbed":35,"log_size":2118,"log_dups_size":0,"ondisk_log_size":2118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:18.713125+0000","scrub_duration":0.105301005,"objects_trimmed":0,"snaptrim_duration":0.54183594099999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":142,"num_read_kb":52725,"num_write":1240,"num_write_kb":42463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"110'2255","reported_seq":5074,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:16.707217+0000","last_change":"2023-10-30T14:27:16.707278+0000","last_active":"2023-10-30T14:27:16.707217+0000","last_peered":"2023-10-30T14:27:16.707217+0000","last_clean":"2023-10-30T14:27:16.707217+0000","last_became_active":"2023-10-30T13:23:19.692241+0000","last_became_peered":"2023-10-30T13:23:19.692241+0000","last_unstale":"2023-10-30T14:27:16.707217+0000","last_undegraded":"2023-10-30T14:27:16.707217+0000","last_fullsized":"2023-10-30T14:27:16.707217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2255","last_scrub_stamp":"2023-10-30T14:27:16.707128+0000","last_deep_scrub":"110'2255","last_deep_scrub_stamp":"2023-10-30T14:27:16.707128+0000","last_clean_scrub_stamp":"2023-10-30T14:27:16.707128+0000","objects_scrubbed":32,"log_size":2255,"log_dups_size":0,"ondisk_log_size":2255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:55.264805+0000","scrub_duration":0.077397046999999997,"objects_trimmed":0,"snaptrim_duration":1.038919041,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":46739,"num_write":1326,"num_write_kb":33562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"110'2198","reported_seq":5059,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:04.874664+0000","last_change":"2023-10-30T14:27:04.874740+0000","last_active":"2023-10-30T14:27:04.874664+0000","last_peered":"2023-10-30T14:27:04.874664+0000","last_clean":"2023-10-30T14:27:04.874664+0000","last_became_active":"2023-10-30T13:23:19.685104+0000","last_became_peered":"2023-10-30T13:23:19.685104+0000","last_unstale":"2023-10-30T14:27:04.874664+0000","last_undegraded":"2023-10-30T14:27:04.874664+0000","last_fullsized":"2023-10-30T14:27:04.874664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2198","last_scrub_stamp":"2023-10-30T14:27:04.874577+0000","last_deep_scrub":"110'2198","last_deep_scrub_stamp":"2023-10-30T14:27:04.874577+0000","last_clean_scrub_stamp":"2023-10-30T14:27:04.874577+0000","objects_scrubbed":47,"log_size":2198,"log_dups_size":0,"ondisk_log_size":2198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:18:28.985655+0000","scrub_duration":0.059905817,"objects_trimmed":0,"snaptrim_duration":0.97549857799999995,"stat_sum":{"num_bytes":0,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":149,"num_read_kb":13007,"num_write":1312,"num_write_kb":16233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"110'2306","reported_seq":5141,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:10.340370+0000","last_change":"2023-10-30T14:27:10.340471+0000","last_active":"2023-10-30T14:27:10.340370+0000","last_peered":"2023-10-30T14:27:10.340370+0000","last_clean":"2023-10-30T14:27:10.340370+0000","last_became_active":"2023-10-30T13:23:19.706473+0000","last_became_peered":"2023-10-30T13:23:19.706473+0000","last_unstale":"2023-10-30T14:27:10.340370+0000","last_undegraded":"2023-10-30T14:27:10.340370+0000","last_fullsized":"2023-10-30T14:27:10.340370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2306","last_scrub_stamp":"2023-10-30T14:27:10.340294+0000","last_deep_scrub":"110'2306","last_deep_scrub_stamp":"2023-10-30T14:27:10.340294+0000","last_clean_scrub_stamp":"2023-10-30T14:27:10.340294+0000","objects_scrubbed":39,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:52:13.783347+0000","scrub_duration":0.101464054,"objects_trimmed":0,"snaptrim_duration":0.432871758,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":161,"num_read_kb":14756,"num_write":1377,"num_write_kb":22909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"110'2120","reported_seq":4990,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:12.050653+0000","last_change":"2023-10-30T14:27:12.050782+0000","last_active":"2023-10-30T14:27:12.050653+0000","last_peered":"2023-10-30T14:27:12.050653+0000","last_clean":"2023-10-30T14:27:12.050653+0000","last_became_active":"2023-10-30T13:23:19.709327+0000","last_became_peered":"2023-10-30T13:23:19.709327+0000","last_unstale":"2023-10-30T14:27:12.050653+0000","last_undegraded":"2023-10-30T14:27:12.050653+0000","last_fullsized":"2023-10-30T14:27:12.050653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2120","last_scrub_stamp":"2023-10-30T14:27:12.050452+0000","last_deep_scrub":"110'2120","last_deep_scrub_stamp":"2023-10-30T14:27:12.050452+0000","last_clean_scrub_stamp":"2023-10-30T14:27:12.050452+0000","objects_scrubbed":30,"log_size":2120,"log_dups_size":0,"ondisk_log_size":2120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:30:59.575797+0000","scrub_duration":0.088091438999999994,"objects_trimmed":0,"snaptrim_duration":0.83845289999999995,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":149,"num_read_kb":40211,"num_write":1268,"num_write_kb":23485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"110'2389","reported_seq":5214,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:13.135193+0000","last_change":"2023-10-30T14:27:13.135363+0000","last_active":"2023-10-30T14:27:13.135193+0000","last_peered":"2023-10-30T14:27:13.135193+0000","last_clean":"2023-10-30T14:27:13.135193+0000","last_became_active":"2023-10-30T13:23:19.740683+0000","last_became_peered":"2023-10-30T13:23:19.740683+0000","last_unstale":"2023-10-30T14:27:13.135193+0000","last_undegraded":"2023-10-30T14:27:13.135193+0000","last_fullsized":"2023-10-30T14:27:13.135193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2389","last_scrub_stamp":"2023-10-30T14:27:13.135025+0000","last_deep_scrub":"110'2389","last_deep_scrub_stamp":"2023-10-30T14:27:13.135025+0000","last_clean_scrub_stamp":"2023-10-30T14:27:13.135025+0000","objects_scrubbed":41,"log_size":2389,"log_dups_size":0,"ondisk_log_size":2389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:16.442511+0000","scrub_duration":0.126176548,"objects_trimmed":0,"snaptrim_duration":1.109948846,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":170,"num_read_kb":39932,"num_write":1446,"num_write_kb":30153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"110'2027","reported_seq":4894,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:59.058458+0000","last_change":"2023-10-30T14:26:59.058509+0000","last_active":"2023-10-30T14:26:59.058458+0000","last_peered":"2023-10-30T14:26:59.058458+0000","last_clean":"2023-10-30T14:26:59.058458+0000","last_became_active":"2023-10-30T13:23:19.700763+0000","last_became_peered":"2023-10-30T13:23:19.700763+0000","last_unstale":"2023-10-30T14:26:59.058458+0000","last_undegraded":"2023-10-30T14:26:59.058458+0000","last_fullsized":"2023-10-30T14:26:59.058458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2027","last_scrub_stamp":"2023-10-30T14:26:59.058377+0000","last_deep_scrub":"110'2027","last_deep_scrub_stamp":"2023-10-30T14:26:59.058377+0000","last_clean_scrub_stamp":"2023-10-30T14:26:59.058377+0000","objects_scrubbed":31,"log_size":2027,"log_dups_size":0,"ondisk_log_size":2027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:24:17.547389+0000","scrub_duration":0.077206206999999999,"objects_trimmed":0,"snaptrim_duration":0.34246037800000001,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":137,"num_read_kb":13706,"num_write":1196,"num_write_kb":12790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"110'2128","reported_seq":4972,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:29.091001+0000","last_change":"2023-10-30T14:27:29.091142+0000","last_active":"2023-10-30T14:27:29.091001+0000","last_peered":"2023-10-30T14:27:29.091001+0000","last_clean":"2023-10-30T14:27:29.091001+0000","last_became_active":"2023-10-30T13:23:19.722231+0000","last_became_peered":"2023-10-30T13:23:19.722231+0000","last_unstale":"2023-10-30T14:27:29.091001+0000","last_undegraded":"2023-10-30T14:27:29.091001+0000","last_fullsized":"2023-10-30T14:27:29.091001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2128","last_scrub_stamp":"2023-10-30T14:27:29.090772+0000","last_deep_scrub":"110'2128","last_deep_scrub_stamp":"2023-10-30T14:27:29.090772+0000","last_clean_scrub_stamp":"2023-10-30T14:27:29.090772+0000","objects_scrubbed":33,"log_size":2128,"log_dups_size":0,"ondisk_log_size":2128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:38:37.170898+0000","scrub_duration":0.089460595000000004,"objects_trimmed":0,"snaptrim_duration":0.76369308499999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":125,"num_read_kb":30178,"num_write":1282,"num_write_kb":24413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"110'2253","reported_seq":5094,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.434812+0000","last_change":"2023-10-30T14:26:57.434908+0000","last_active":"2023-10-30T14:26:57.434812+0000","last_peered":"2023-10-30T14:26:57.434812+0000","last_clean":"2023-10-30T14:26:57.434812+0000","last_became_active":"2023-10-30T13:23:19.724399+0000","last_became_peered":"2023-10-30T13:23:19.724399+0000","last_unstale":"2023-10-30T14:26:57.434812+0000","last_undegraded":"2023-10-30T14:26:57.434812+0000","last_fullsized":"2023-10-30T14:26:57.434812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2253","last_scrub_stamp":"2023-10-30T14:26:57.434694+0000","last_deep_scrub":"110'2253","last_deep_scrub_stamp":"2023-10-30T14:26:57.434694+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.434694+0000","objects_scrubbed":38,"log_size":2253,"log_dups_size":0,"ondisk_log_size":2253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:26:38.794241+0000","scrub_duration":0.122965276,"objects_trimmed":0,"snaptrim_duration":0.46668427699999998,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":148,"num_read_kb":30111,"num_write":1332,"num_write_kb":26597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"110'2256","reported_seq":5056,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:11.917935+0000","last_change":"2023-10-30T14:27:11.918008+0000","last_active":"2023-10-30T14:27:11.917935+0000","last_peered":"2023-10-30T14:27:11.917935+0000","last_clean":"2023-10-30T14:27:11.917935+0000","last_became_active":"2023-10-30T13:23:19.721554+0000","last_became_peered":"2023-10-30T13:23:19.721554+0000","last_unstale":"2023-10-30T14:27:11.917935+0000","last_undegraded":"2023-10-30T14:27:11.917935+0000","last_fullsized":"2023-10-30T14:27:11.917935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2256","last_scrub_stamp":"2023-10-30T14:27:11.917846+0000","last_deep_scrub":"110'2256","last_deep_scrub_stamp":"2023-10-30T14:27:11.917846+0000","last_clean_scrub_stamp":"2023-10-30T14:27:11.917846+0000","objects_scrubbed":34,"log_size":2256,"log_dups_size":0,"ondisk_log_size":2256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:28.375501+0000","scrub_duration":0.083664374,"objects_trimmed":0,"snaptrim_duration":1.008362585,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":127,"num_read_kb":10449,"num_write":1298,"num_write_kb":11653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"110'2330","reported_seq":5142,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:22.210913+0000","last_change":"2023-10-30T14:27:22.211067+0000","last_active":"2023-10-30T14:27:22.210913+0000","last_peered":"2023-10-30T14:27:22.210913+0000","last_clean":"2023-10-30T14:27:22.210913+0000","last_became_active":"2023-10-30T13:23:19.724017+0000","last_became_peered":"2023-10-30T13:23:19.724017+0000","last_unstale":"2023-10-30T14:27:22.210913+0000","last_undegraded":"2023-10-30T14:27:22.210913+0000","last_fullsized":"2023-10-30T14:27:22.210913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2330","last_scrub_stamp":"2023-10-30T14:27:22.210723+0000","last_deep_scrub":"110'2330","last_deep_scrub_stamp":"2023-10-30T14:27:22.210723+0000","last_clean_scrub_stamp":"2023-10-30T14:27:22.210723+0000","objects_scrubbed":48,"log_size":2330,"log_dups_size":0,"ondisk_log_size":2330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:11:38.233096+0000","scrub_duration":0.14657436700000001,"objects_trimmed":0,"snaptrim_duration":0.94381626399999996,"stat_sum":{"num_bytes":0,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":144,"num_read_kb":30506,"num_write":1376,"num_write_kb":26822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"110'2155","reported_seq":5129,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:27.727917+0000","last_change":"2023-10-30T14:27:27.727989+0000","last_active":"2023-10-30T14:27:27.727917+0000","last_peered":"2023-10-30T14:27:27.727917+0000","last_clean":"2023-10-30T14:27:27.727917+0000","last_became_active":"2023-10-30T13:23:19.716456+0000","last_became_peered":"2023-10-30T13:23:19.716456+0000","last_unstale":"2023-10-30T14:27:27.727917+0000","last_undegraded":"2023-10-30T14:27:27.727917+0000","last_fullsized":"2023-10-30T14:27:27.727917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2155","last_scrub_stamp":"2023-10-30T14:27:27.727838+0000","last_deep_scrub":"110'2155","last_deep_scrub_stamp":"2023-10-30T14:27:27.727838+0000","last_clean_scrub_stamp":"2023-10-30T14:27:27.727838+0000","objects_scrubbed":38,"log_size":2155,"log_dups_size":0,"ondisk_log_size":2155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:27:08.100066+0000","scrub_duration":0.081191790999999999,"objects_trimmed":0,"snaptrim_duration":0.98440143999999996,"stat_sum":{"num_bytes":6,"num_objects":38,"num_object_clones":2,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":257,"num_read_kb":42146,"num_write":1297,"num_write_kb":25266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"110'2212","reported_seq":5085,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:05.052361+0000","last_change":"2023-10-30T14:27:05.052501+0000","last_active":"2023-10-30T14:27:05.052361+0000","last_peered":"2023-10-30T14:27:05.052361+0000","last_clean":"2023-10-30T14:27:05.052361+0000","last_became_active":"2023-10-30T13:23:19.699398+0000","last_became_peered":"2023-10-30T13:23:19.699398+0000","last_unstale":"2023-10-30T14:27:05.052361+0000","last_undegraded":"2023-10-30T14:27:05.052361+0000","last_fullsized":"2023-10-30T14:27:05.052361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2212","last_scrub_stamp":"2023-10-30T14:27:05.052163+0000","last_deep_scrub":"110'2212","last_deep_scrub_stamp":"2023-10-30T14:27:05.052163+0000","last_clean_scrub_stamp":"2023-10-30T14:27:05.052163+0000","objects_scrubbed":34,"log_size":2212,"log_dups_size":0,"ondisk_log_size":2212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:40.396490+0000","scrub_duration":0.091462254000000007,"objects_trimmed":0,"snaptrim_duration":0.89723890399999995,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":164,"num_read_kb":14461,"num_write":1342,"num_write_kb":12663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"110'2090","reported_seq":4941,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:07.710584+0000","last_change":"2023-10-30T14:27:07.710637+0000","last_active":"2023-10-30T14:27:07.710584+0000","last_peered":"2023-10-30T14:27:07.710584+0000","last_clean":"2023-10-30T14:27:07.710584+0000","last_became_active":"2023-10-30T13:23:19.716699+0000","last_became_peered":"2023-10-30T13:23:19.716699+0000","last_unstale":"2023-10-30T14:27:07.710584+0000","last_undegraded":"2023-10-30T14:27:07.710584+0000","last_fullsized":"2023-10-30T14:27:07.710584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2090","last_scrub_stamp":"2023-10-30T14:27:07.710502+0000","last_deep_scrub":"110'2090","last_deep_scrub_stamp":"2023-10-30T14:27:07.710502+0000","last_clean_scrub_stamp":"2023-10-30T14:27:07.710502+0000","objects_scrubbed":33,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:10:39.764424+0000","scrub_duration":0.081297077999999995,"objects_trimmed":0,"snaptrim_duration":1.0495642949999999,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":133,"num_read_kb":25337,"num_write":1220,"num_write_kb":15637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"110'2181","reported_seq":5069,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:03.420662+0000","last_change":"2023-10-30T14:27:03.420832+0000","last_active":"2023-10-30T14:27:03.420662+0000","last_peered":"2023-10-30T14:27:03.420662+0000","last_clean":"2023-10-30T14:27:03.420662+0000","last_became_active":"2023-10-30T13:23:19.705195+0000","last_became_peered":"2023-10-30T13:23:19.705195+0000","last_unstale":"2023-10-30T14:27:03.420662+0000","last_undegraded":"2023-10-30T14:27:03.420662+0000","last_fullsized":"2023-10-30T14:27:03.420662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2181","last_scrub_stamp":"2023-10-30T14:27:03.420503+0000","last_deep_scrub":"110'2181","last_deep_scrub_stamp":"2023-10-30T14:27:03.420503+0000","last_clean_scrub_stamp":"2023-10-30T14:27:03.420503+0000","objects_scrubbed":40,"log_size":2181,"log_dups_size":0,"ondisk_log_size":2181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:36:16.083580+0000","scrub_duration":0.11761008000000001,"objects_trimmed":0,"snaptrim_duration":0.504458776,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":160,"num_read_kb":14034,"num_write":1340,"num_write_kb":12359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"110'2127","reported_seq":4987,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:03.326222+0000","last_change":"2023-10-30T14:27:03.326282+0000","last_active":"2023-10-30T14:27:03.326222+0000","last_peered":"2023-10-30T14:27:03.326222+0000","last_clean":"2023-10-30T14:27:03.326222+0000","last_became_active":"2023-10-30T13:23:19.678291+0000","last_became_peered":"2023-10-30T13:23:19.678291+0000","last_unstale":"2023-10-30T14:27:03.326222+0000","last_undegraded":"2023-10-30T14:27:03.326222+0000","last_fullsized":"2023-10-30T14:27:03.326222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2127","last_scrub_stamp":"2023-10-30T14:27:03.326140+0000","last_deep_scrub":"110'2127","last_deep_scrub_stamp":"2023-10-30T14:27:03.326140+0000","last_clean_scrub_stamp":"2023-10-30T14:27:03.326140+0000","objects_scrubbed":38,"log_size":2127,"log_dups_size":0,"ondisk_log_size":2127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:46:52.407641+0000","scrub_duration":0.050416410000000002,"objects_trimmed":0,"snaptrim_duration":0.34262146900000001,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":128,"num_read_kb":22518,"num_write":1277,"num_write_kb":32107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"110'2068","reported_seq":4908,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.691221+0000","last_change":"2023-10-30T14:26:52.691273+0000","last_active":"2023-10-30T14:26:52.691221+0000","last_peered":"2023-10-30T14:26:52.691221+0000","last_clean":"2023-10-30T14:26:52.691221+0000","last_became_active":"2023-10-30T13:23:19.701896+0000","last_became_peered":"2023-10-30T13:23:19.701896+0000","last_unstale":"2023-10-30T14:26:52.691221+0000","last_undegraded":"2023-10-30T14:26:52.691221+0000","last_fullsized":"2023-10-30T14:26:52.691221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2068","last_scrub_stamp":"2023-10-30T14:26:52.691147+0000","last_deep_scrub":"110'2068","last_deep_scrub_stamp":"2023-10-30T14:26:52.691147+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.691147+0000","objects_scrubbed":38,"log_size":2068,"log_dups_size":0,"ondisk_log_size":2068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:07:08.320506+0000","scrub_duration":0.079048169000000001,"objects_trimmed":0,"snaptrim_duration":1.068284585,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":129,"num_read_kb":17081,"num_write":1202,"num_write_kb":15468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"110'2178","reported_seq":5011,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:16.911734+0000","last_change":"2023-10-30T14:27:16.911796+0000","last_active":"2023-10-30T14:27:16.911734+0000","last_peered":"2023-10-30T14:27:16.911734+0000","last_clean":"2023-10-30T14:27:16.911734+0000","last_became_active":"2023-10-30T13:23:19.687575+0000","last_became_peered":"2023-10-30T13:23:19.687575+0000","last_unstale":"2023-10-30T14:27:16.911734+0000","last_undegraded":"2023-10-30T14:27:16.911734+0000","last_fullsized":"2023-10-30T14:27:16.911734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2178","last_scrub_stamp":"2023-10-30T14:27:16.911656+0000","last_deep_scrub":"110'2178","last_deep_scrub_stamp":"2023-10-30T14:27:16.911656+0000","last_clean_scrub_stamp":"2023-10-30T14:27:16.911656+0000","objects_scrubbed":38,"log_size":2178,"log_dups_size":0,"ondisk_log_size":2178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:38:53.941111+0000","scrub_duration":0.046150844000000003,"objects_trimmed":0,"snaptrim_duration":1.030536272,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":125,"num_read_kb":13655,"num_write":1323,"num_write_kb":17357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"110'2124","reported_seq":5005,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.119416+0000","last_change":"2023-10-30T14:26:57.119416+0000","last_active":"2023-10-30T14:26:57.119416+0000","last_peered":"2023-10-30T14:26:57.119416+0000","last_clean":"2023-10-30T14:26:57.119416+0000","last_became_active":"2023-10-30T13:23:19.698666+0000","last_became_peered":"2023-10-30T13:23:19.698666+0000","last_unstale":"2023-10-30T14:26:57.119416+0000","last_undegraded":"2023-10-30T14:26:57.119416+0000","last_fullsized":"2023-10-30T14:26:57.119416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2124","last_scrub_stamp":"2023-10-30T14:26:57.119213+0000","last_deep_scrub":"110'2124","last_deep_scrub_stamp":"2023-10-30T14:26:57.119213+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.119213+0000","objects_scrubbed":38,"log_size":2124,"log_dups_size":0,"ondisk_log_size":2124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:32:10.120593+0000","scrub_duration":0.076576980000000003,"objects_trimmed":0,"snaptrim_duration":0.98488935700000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":140,"num_read_kb":17880,"num_write":1311,"num_write_kb":13902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"110'2418","reported_seq":5245,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.085296+0000","last_change":"2023-10-30T14:27:00.085296+0000","last_active":"2023-10-30T14:27:00.085296+0000","last_peered":"2023-10-30T14:27:00.085296+0000","last_clean":"2023-10-30T14:27:00.085296+0000","last_became_active":"2023-10-30T13:23:19.711034+0000","last_became_peered":"2023-10-30T13:23:19.711034+0000","last_unstale":"2023-10-30T14:27:00.085296+0000","last_undegraded":"2023-10-30T14:27:00.085296+0000","last_fullsized":"2023-10-30T14:27:00.085296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2418","last_scrub_stamp":"2023-10-30T14:27:00.085138+0000","last_deep_scrub":"110'2418","last_deep_scrub_stamp":"2023-10-30T14:27:00.085138+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.085138+0000","objects_scrubbed":41,"log_size":2418,"log_dups_size":0,"ondisk_log_size":2418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:00.491199+0000","scrub_duration":0.10095144,"objects_trimmed":0,"snaptrim_duration":0.30670995400000001,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":165,"num_read_kb":40193,"num_write":1443,"num_write_kb":35290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"110'2344","reported_seq":5177,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:02.425824+0000","last_change":"2023-10-30T14:27:02.425824+0000","last_active":"2023-10-30T14:27:02.425824+0000","last_peered":"2023-10-30T14:27:02.425824+0000","last_clean":"2023-10-30T14:27:02.425824+0000","last_became_active":"2023-10-30T13:23:19.693283+0000","last_became_peered":"2023-10-30T13:23:19.693283+0000","last_unstale":"2023-10-30T14:27:02.425824+0000","last_undegraded":"2023-10-30T14:27:02.425824+0000","last_fullsized":"2023-10-30T14:27:02.425824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2344","last_scrub_stamp":"2023-10-30T14:27:02.425448+0000","last_deep_scrub":"110'2344","last_deep_scrub_stamp":"2023-10-30T14:27:02.425448+0000","last_clean_scrub_stamp":"2023-10-30T14:27:02.425448+0000","objects_scrubbed":41,"log_size":2344,"log_dups_size":0,"ondisk_log_size":2344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:18.221218+0000","scrub_duration":0.11536735100000001,"objects_trimmed":0,"snaptrim_duration":0.42993166100000002,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":168,"num_read_kb":24860,"num_write":1383,"num_write_kb":20782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"110'2147","reported_seq":5000,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:50.868519+0000","last_change":"2023-10-30T14:26:50.868736+0000","last_active":"2023-10-30T14:26:50.868519+0000","last_peered":"2023-10-30T14:26:50.868519+0000","last_clean":"2023-10-30T14:26:50.868519+0000","last_became_active":"2023-10-30T13:23:19.702452+0000","last_became_peered":"2023-10-30T13:23:19.702452+0000","last_unstale":"2023-10-30T14:26:50.868519+0000","last_undegraded":"2023-10-30T14:26:50.868519+0000","last_fullsized":"2023-10-30T14:26:50.868519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2147","last_scrub_stamp":"2023-10-30T14:26:50.868292+0000","last_deep_scrub":"110'2147","last_deep_scrub_stamp":"2023-10-30T14:26:50.868292+0000","last_clean_scrub_stamp":"2023-10-30T14:26:50.868292+0000","objects_scrubbed":34,"log_size":2147,"log_dups_size":0,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:21.454895+0000","scrub_duration":0.095674789999999996,"objects_trimmed":0,"snaptrim_duration":0.59246732999999996,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":146,"num_read_kb":30802,"num_write":1268,"num_write_kb":21419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"110'1078","reported_seq":1396,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.791196+0000","last_change":"2023-10-30T14:26:42.791248+0000","last_active":"2023-10-30T14:26:42.791196+0000","last_peered":"2023-10-30T14:26:42.791196+0000","last_clean":"2023-10-30T14:26:42.791196+0000","last_became_active":"2023-10-30T13:23:18.963531+0000","last_became_peered":"2023-10-30T13:23:18.963531+0000","last_unstale":"2023-10-30T14:26:42.791196+0000","last_undegraded":"2023-10-30T14:26:42.791196+0000","last_fullsized":"2023-10-30T14:26:42.791196+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1078","last_scrub_stamp":"2023-10-30T14:26:42.791118+0000","last_deep_scrub":"110'1078","last_deep_scrub_stamp":"2023-10-30T14:26:42.791118+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.791118+0000","objects_scrubbed":50,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:04:33.250671+0000","scrub_duration":0.285611593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497506,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":222,"num_read_kb":501605,"num_write":1094,"num_write_kb":95785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":358881,"num_omap_keys":701,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"110'2190","reported_seq":5007,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:10.871699+0000","last_change":"2023-10-30T14:27:10.871788+0000","last_active":"2023-10-30T14:27:10.871699+0000","last_peered":"2023-10-30T14:27:10.871699+0000","last_clean":"2023-10-30T14:27:10.871699+0000","last_became_active":"2023-10-30T13:23:19.717519+0000","last_became_peered":"2023-10-30T13:23:19.717519+0000","last_unstale":"2023-10-30T14:27:10.871699+0000","last_undegraded":"2023-10-30T14:27:10.871699+0000","last_fullsized":"2023-10-30T14:27:10.871699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2190","last_scrub_stamp":"2023-10-30T14:27:10.871590+0000","last_deep_scrub":"110'2190","last_deep_scrub_stamp":"2023-10-30T14:27:10.871590+0000","last_clean_scrub_stamp":"2023-10-30T14:27:10.871590+0000","objects_scrubbed":35,"log_size":2190,"log_dups_size":0,"ondisk_log_size":2190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:54:03.123430+0000","scrub_duration":0.083573112000000005,"objects_trimmed":0,"snaptrim_duration":1.1434485830000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":131,"num_read_kb":17524,"num_write":1294,"num_write_kb":15816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"110'1745","reported_seq":2190,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.486755+0000","last_change":"2023-10-30T14:26:54.486806+0000","last_active":"2023-10-30T14:26:54.486755+0000","last_peered":"2023-10-30T14:26:54.486755+0000","last_clean":"2023-10-30T14:26:54.486755+0000","last_became_active":"2023-10-30T13:23:18.975877+0000","last_became_peered":"2023-10-30T13:23:18.975877+0000","last_unstale":"2023-10-30T14:26:54.486755+0000","last_undegraded":"2023-10-30T14:26:54.486755+0000","last_fullsized":"2023-10-30T14:26:54.486755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1745","last_scrub_stamp":"2023-10-30T14:26:54.486663+0000","last_deep_scrub":"110'1745","last_deep_scrub_stamp":"2023-10-30T14:26:54.486663+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.486663+0000","objects_scrubbed":52,"log_size":1745,"log_dups_size":0,"ondisk_log_size":1745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:24:54.960607+0000","scrub_duration":0.19504682200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29694842,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":392,"num_read_kb":410068,"num_write":1792,"num_write_kb":89526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":545048,"num_omap_keys":1059,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"110'1609","reported_seq":2975,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.816492+0000","last_change":"2023-10-30T14:26:46.816578+0000","last_active":"2023-10-30T14:26:46.816492+0000","last_peered":"2023-10-30T14:26:46.816492+0000","last_clean":"2023-10-30T14:26:46.816492+0000","last_became_active":"2023-10-30T13:23:18.976972+0000","last_became_peered":"2023-10-30T13:23:18.976972+0000","last_unstale":"2023-10-30T14:26:46.816492+0000","last_undegraded":"2023-10-30T14:26:46.816492+0000","last_fullsized":"2023-10-30T14:26:46.816492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1609","last_scrub_stamp":"2023-10-30T14:26:46.816372+0000","last_deep_scrub":"110'1609","last_deep_scrub_stamp":"2023-10-30T14:26:46.816372+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.816372+0000","objects_scrubbed":52,"log_size":1609,"log_dups_size":0,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:04:35.564593+0000","scrub_duration":0.28844654400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60855928,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":1296,"num_read_kb":556671,"num_write":1596,"num_write_kb":94675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":405715,"num_omap_keys":625,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"110'2379","reported_seq":5227,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.714048+0000","last_change":"2023-10-30T14:26:53.714120+0000","last_active":"2023-10-30T14:26:53.714048+0000","last_peered":"2023-10-30T14:26:53.714048+0000","last_clean":"2023-10-30T14:26:53.714048+0000","last_became_active":"2023-10-30T13:23:19.688855+0000","last_became_peered":"2023-10-30T13:23:19.688855+0000","last_unstale":"2023-10-30T14:26:53.714048+0000","last_undegraded":"2023-10-30T14:26:53.714048+0000","last_fullsized":"2023-10-30T14:26:53.714048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2379","last_scrub_stamp":"2023-10-30T14:26:53.713939+0000","last_deep_scrub":"110'2379","last_deep_scrub_stamp":"2023-10-30T14:26:53.713939+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.713939+0000","objects_scrubbed":52,"log_size":2379,"log_dups_size":0,"ondisk_log_size":2379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:15.405210+0000","scrub_duration":0.058831420000000002,"objects_trimmed":0,"snaptrim_duration":0.89032229399999996,"stat_sum":{"num_bytes":0,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":161,"num_read_kb":17698,"num_write":1449,"num_write_kb":26122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"110'1032","reported_seq":1390,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:55.223361+0000","last_change":"2023-10-30T14:26:55.223448+0000","last_active":"2023-10-30T14:26:55.223361+0000","last_peered":"2023-10-30T14:26:55.223361+0000","last_clean":"2023-10-30T14:26:55.223361+0000","last_became_active":"2023-10-30T13:23:18.988646+0000","last_became_peered":"2023-10-30T13:23:18.988646+0000","last_unstale":"2023-10-30T14:26:55.223361+0000","last_undegraded":"2023-10-30T14:26:55.223361+0000","last_fullsized":"2023-10-30T14:26:55.223361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1032","last_scrub_stamp":"2023-10-30T14:26:55.223231+0000","last_deep_scrub":"110'1032","last_deep_scrub_stamp":"2023-10-30T14:26:55.223231+0000","last_clean_scrub_stamp":"2023-10-30T14:26:55.223231+0000","objects_scrubbed":57,"log_size":1032,"log_dups_size":0,"ondisk_log_size":1032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:18:36.091837+0000","scrub_duration":0.26442538300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54010089,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":312,"num_read_kb":563254,"num_write":1082,"num_write_kb":96418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":706737,"num_omap_keys":1228,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"110'2179","reported_seq":5047,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.255870+0000","last_change":"2023-10-30T14:27:00.255921+0000","last_active":"2023-10-30T14:27:00.255870+0000","last_peered":"2023-10-30T14:27:00.255870+0000","last_clean":"2023-10-30T14:27:00.255870+0000","last_became_active":"2023-10-30T13:23:19.679040+0000","last_became_peered":"2023-10-30T13:23:19.679040+0000","last_unstale":"2023-10-30T14:27:00.255870+0000","last_undegraded":"2023-10-30T14:27:00.255870+0000","last_fullsized":"2023-10-30T14:27:00.255870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2179","last_scrub_stamp":"2023-10-30T14:27:00.255786+0000","last_deep_scrub":"110'2179","last_deep_scrub_stamp":"2023-10-30T14:27:00.255786+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.255786+0000","objects_scrubbed":30,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:01.714268+0000","scrub_duration":0.037354261999999999,"objects_trimmed":0,"snaptrim_duration":0.36920391600000002,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":152,"num_read_kb":51949,"num_write":1321,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"110'865","reported_seq":1210,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:56.576908+0000","last_change":"2023-10-30T14:26:56.577007+0000","last_active":"2023-10-30T14:26:56.576908+0000","last_peered":"2023-10-30T14:26:56.576908+0000","last_clean":"2023-10-30T14:26:56.576908+0000","last_became_active":"2023-10-30T13:23:18.988927+0000","last_became_peered":"2023-10-30T13:23:18.988927+0000","last_unstale":"2023-10-30T14:26:56.576908+0000","last_undegraded":"2023-10-30T14:26:56.576908+0000","last_fullsized":"2023-10-30T14:26:56.576908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'865","last_scrub_stamp":"2023-10-30T14:26:56.576765+0000","last_deep_scrub":"110'865","last_deep_scrub_stamp":"2023-10-30T14:26:56.576765+0000","last_clean_scrub_stamp":"2023-10-30T14:26:56.576765+0000","objects_scrubbed":64,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:40.939877+0000","scrub_duration":0.28082476899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943130,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":319,"num_read_kb":417394,"num_write":917,"num_write_kb":71614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":395317,"num_omap_keys":751,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"110'2477","reported_seq":5216,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:58.107565+0000","last_change":"2023-10-30T14:26:58.107765+0000","last_active":"2023-10-30T14:26:58.107565+0000","last_peered":"2023-10-30T14:26:58.107565+0000","last_clean":"2023-10-30T14:26:58.107565+0000","last_became_active":"2023-10-30T13:23:19.731502+0000","last_became_peered":"2023-10-30T13:23:19.731502+0000","last_unstale":"2023-10-30T14:26:58.107565+0000","last_undegraded":"2023-10-30T14:26:58.107565+0000","last_fullsized":"2023-10-30T14:26:58.107565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2477","last_scrub_stamp":"2023-10-30T14:26:58.107390+0000","last_deep_scrub":"110'2477","last_deep_scrub_stamp":"2023-10-30T14:26:58.107390+0000","last_clean_scrub_stamp":"2023-10-30T14:26:58.107390+0000","objects_scrubbed":23,"log_size":2477,"log_dups_size":0,"ondisk_log_size":2477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:29.902228+0000","scrub_duration":0.078694307000000005,"objects_trimmed":0,"snaptrim_duration":1.0302270280000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":152,"num_read_kb":10055,"num_write":1351,"num_write_kb":13264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"110'901","reported_seq":1217,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:51.209972+0000","last_change":"2023-10-30T14:26:51.210091+0000","last_active":"2023-10-30T14:26:51.209972+0000","last_peered":"2023-10-30T14:26:51.209972+0000","last_clean":"2023-10-30T14:26:51.209972+0000","last_became_active":"2023-10-30T13:23:18.990616+0000","last_became_peered":"2023-10-30T13:23:18.990616+0000","last_unstale":"2023-10-30T14:26:51.209972+0000","last_undegraded":"2023-10-30T14:26:51.209972+0000","last_fullsized":"2023-10-30T14:26:51.209972+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'901","last_scrub_stamp":"2023-10-30T14:26:51.209745+0000","last_deep_scrub":"110'901","last_deep_scrub_stamp":"2023-10-30T14:26:51.209745+0000","last_clean_scrub_stamp":"2023-10-30T14:26:51.209745+0000","objects_scrubbed":52,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:29:33.421380+0000","scrub_duration":0.228163176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":291,"num_read_kb":402257,"num_write":947,"num_write_kb":71494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":417597,"num_omap_keys":787,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"110'2117","reported_seq":4971,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:01.839258+0000","last_change":"2023-10-30T14:27:01.839312+0000","last_active":"2023-10-30T14:27:01.839258+0000","last_peered":"2023-10-30T14:27:01.839258+0000","last_clean":"2023-10-30T14:27:01.839258+0000","last_became_active":"2023-10-30T13:24:00.990019+0000","last_became_peered":"2023-10-30T13:24:00.990019+0000","last_unstale":"2023-10-30T14:27:01.839258+0000","last_undegraded":"2023-10-30T14:27:01.839258+0000","last_fullsized":"2023-10-30T14:27:01.839258+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2117","last_scrub_stamp":"2023-10-30T14:27:01.839175+0000","last_deep_scrub":"110'2117","last_deep_scrub_stamp":"2023-10-30T14:27:01.839175+0000","last_clean_scrub_stamp":"2023-10-30T14:27:01.839175+0000","objects_scrubbed":32,"log_size":2117,"log_dups_size":0,"ondisk_log_size":2117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:37:53.712537+0000","scrub_duration":0.043087937999999999,"objects_trimmed":0,"snaptrim_duration":1.1322467139999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":132,"num_read_kb":30385,"num_write":1263,"num_write_kb":18224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"110'1029","reported_seq":1299,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:55.421984+0000","last_change":"2023-10-30T14:26:55.422041+0000","last_active":"2023-10-30T14:26:55.421984+0000","last_peered":"2023-10-30T14:26:55.421984+0000","last_clean":"2023-10-30T14:26:55.421984+0000","last_became_active":"2023-10-30T13:23:18.982077+0000","last_became_peered":"2023-10-30T13:23:18.982077+0000","last_unstale":"2023-10-30T14:26:55.421984+0000","last_undegraded":"2023-10-30T14:26:55.421984+0000","last_fullsized":"2023-10-30T14:26:55.421984+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1029","last_scrub_stamp":"2023-10-30T14:26:55.421890+0000","last_deep_scrub":"110'1029","last_deep_scrub_stamp":"2023-10-30T14:26:55.421890+0000","last_clean_scrub_stamp":"2023-10-30T14:26:55.421890+0000","objects_scrubbed":35,"log_size":1029,"log_dups_size":0,"ondisk_log_size":1029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:34:36.976816+0000","scrub_duration":0.131590227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777888,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":208,"num_read_kb":224450,"num_write":1057,"num_write_kb":46972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":295749,"num_omap_keys":457,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"110'2177","reported_seq":5017,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:48.799169+0000","last_change":"2023-10-30T14:26:48.799265+0000","last_active":"2023-10-30T14:26:48.799169+0000","last_peered":"2023-10-30T14:26:48.799169+0000","last_clean":"2023-10-30T14:26:48.799169+0000","last_became_active":"2023-10-30T13:23:19.705771+0000","last_became_peered":"2023-10-30T13:23:19.705771+0000","last_unstale":"2023-10-30T14:26:48.799169+0000","last_undegraded":"2023-10-30T14:26:48.799169+0000","last_fullsized":"2023-10-30T14:26:48.799169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2177","last_scrub_stamp":"2023-10-30T14:26:48.799002+0000","last_deep_scrub":"110'2177","last_deep_scrub_stamp":"2023-10-30T14:26:48.799002+0000","last_clean_scrub_stamp":"2023-10-30T14:26:48.799002+0000","objects_scrubbed":28,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:56:02.403730+0000","scrub_duration":0.092776059999999994,"objects_trimmed":0,"snaptrim_duration":0.58089010799999996,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":141,"num_read_kb":34327,"num_write":1294,"num_write_kb":27493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"110'1040","reported_seq":1364,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:47.590330+0000","last_change":"2023-10-30T14:26:47.590330+0000","last_active":"2023-10-30T14:26:47.590330+0000","last_peered":"2023-10-30T14:26:47.590330+0000","last_clean":"2023-10-30T14:26:47.590330+0000","last_became_active":"2023-10-30T13:23:18.967101+0000","last_became_peered":"2023-10-30T13:23:18.967101+0000","last_unstale":"2023-10-30T14:26:47.590330+0000","last_undegraded":"2023-10-30T14:26:47.590330+0000","last_fullsized":"2023-10-30T14:26:47.590330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1039","last_scrub_stamp":"2023-10-30T14:26:47.589869+0000","last_deep_scrub":"110'1039","last_deep_scrub_stamp":"2023-10-30T14:26:47.589869+0000","last_clean_scrub_stamp":"2023-10-30T14:26:47.589869+0000","objects_scrubbed":46,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:04:37.337840+0000","scrub_duration":0.243642899,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":290,"num_read_kb":355466,"num_write":1084,"num_write_kb":67491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":426651,"num_omap_keys":733,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"110'2198","reported_seq":5027,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:45.082685+0000","last_change":"2023-10-30T14:27:45.082813+0000","last_active":"2023-10-30T14:27:45.082685+0000","last_peered":"2023-10-30T14:27:45.082685+0000","last_clean":"2023-10-30T14:27:45.082685+0000","last_became_active":"2023-10-30T13:23:19.727531+0000","last_became_peered":"2023-10-30T13:23:19.727531+0000","last_unstale":"2023-10-30T14:27:45.082685+0000","last_undegraded":"2023-10-30T14:27:45.082685+0000","last_fullsized":"2023-10-30T14:27:45.082685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2198","last_scrub_stamp":"2023-10-30T14:27:45.082475+0000","last_deep_scrub":"110'2198","last_deep_scrub_stamp":"2023-10-30T14:27:45.082475+0000","last_clean_scrub_stamp":"2023-10-30T14:27:45.082475+0000","objects_scrubbed":33,"log_size":2198,"log_dups_size":0,"ondisk_log_size":2198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:18.852743+0000","scrub_duration":0.092302616000000004,"objects_trimmed":0,"snaptrim_duration":0.74553484199999998,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":143,"num_read_kb":29369,"num_write":1268,"num_write_kb":21738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"110'758","reported_seq":1138,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.031518+0000","last_change":"2023-10-30T14:26:42.031593+0000","last_active":"2023-10-30T14:26:42.031518+0000","last_peered":"2023-10-30T14:26:42.031518+0000","last_clean":"2023-10-30T14:26:42.031518+0000","last_became_active":"2023-10-30T13:23:18.971179+0000","last_became_peered":"2023-10-30T13:23:18.971179+0000","last_unstale":"2023-10-30T14:26:42.031518+0000","last_undegraded":"2023-10-30T14:26:42.031518+0000","last_fullsized":"2023-10-30T14:26:42.031518+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'758","last_scrub_stamp":"2023-10-30T14:26:42.031416+0000","last_deep_scrub":"110'758","last_deep_scrub_stamp":"2023-10-30T14:26:42.031416+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.031416+0000","objects_scrubbed":57,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:32.795744+0000","scrub_duration":0.243287324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331682,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":392,"num_read_kb":480410,"num_write":807,"num_write_kb":87841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":384994,"num_omap_keys":754,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"110'2184","reported_seq":5052,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.780582+0000","last_change":"2023-10-30T14:26:52.780632+0000","last_active":"2023-10-30T14:26:52.780582+0000","last_peered":"2023-10-30T14:26:52.780582+0000","last_clean":"2023-10-30T14:26:52.780582+0000","last_became_active":"2023-10-30T13:23:19.717913+0000","last_became_peered":"2023-10-30T13:23:19.717913+0000","last_unstale":"2023-10-30T14:26:52.780582+0000","last_undegraded":"2023-10-30T14:26:52.780582+0000","last_fullsized":"2023-10-30T14:26:52.780582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2184","last_scrub_stamp":"2023-10-30T14:26:52.780505+0000","last_deep_scrub":"110'2184","last_deep_scrub_stamp":"2023-10-30T14:26:52.780505+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.780505+0000","objects_scrubbed":34,"log_size":2184,"log_dups_size":0,"ondisk_log_size":2184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:22.325362+0000","scrub_duration":0.086211091000000004,"objects_trimmed":0,"snaptrim_duration":0.95729191499999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":151,"num_read_kb":31378,"num_write":1306,"num_write_kb":33173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"110'899","reported_seq":1169,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.271896+0000","last_change":"2023-10-30T14:26:54.272052+0000","last_active":"2023-10-30T14:26:54.271896+0000","last_peered":"2023-10-30T14:26:54.271896+0000","last_clean":"2023-10-30T14:26:54.271896+0000","last_became_active":"2023-10-30T13:23:18.977661+0000","last_became_peered":"2023-10-30T13:23:18.977661+0000","last_unstale":"2023-10-30T14:26:54.271896+0000","last_undegraded":"2023-10-30T14:26:54.271896+0000","last_fullsized":"2023-10-30T14:26:54.271896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'899","last_scrub_stamp":"2023-10-30T14:26:54.271663+0000","last_deep_scrub":"110'899","last_deep_scrub_stamp":"2023-10-30T14:26:54.271663+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.271663+0000","objects_scrubbed":48,"log_size":899,"log_dups_size":0,"ondisk_log_size":899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:16.820077+0000","scrub_duration":0.294758195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53927368,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":208,"num_read_kb":306859,"num_write":914,"num_write_kb":62457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":467666,"num_omap_keys":812,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"110'2334","reported_seq":5135,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:09.227727+0000","last_change":"2023-10-30T14:27:09.227790+0000","last_active":"2023-10-30T14:27:09.227727+0000","last_peered":"2023-10-30T14:27:09.227727+0000","last_clean":"2023-10-30T14:27:09.227727+0000","last_became_active":"2023-10-30T13:24:00.706894+0000","last_became_peered":"2023-10-30T13:24:00.706894+0000","last_unstale":"2023-10-30T14:27:09.227727+0000","last_undegraded":"2023-10-30T14:27:09.227727+0000","last_fullsized":"2023-10-30T14:27:09.227727+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2334","last_scrub_stamp":"2023-10-30T14:27:09.227647+0000","last_deep_scrub":"110'2334","last_deep_scrub_stamp":"2023-10-30T14:27:09.227647+0000","last_clean_scrub_stamp":"2023-10-30T14:27:09.227647+0000","objects_scrubbed":41,"log_size":2334,"log_dups_size":0,"ondisk_log_size":2334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:45:55.621940+0000","scrub_duration":0.096202764999999996,"objects_trimmed":0,"snaptrim_duration":0.3738494,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":149,"num_read_kb":1539,"num_write":1375,"num_write_kb":12931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"110'780","reported_seq":1067,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.584097+0000","last_change":"2023-10-30T14:26:46.584221+0000","last_active":"2023-10-30T14:26:46.584097+0000","last_peered":"2023-10-30T14:26:46.584097+0000","last_clean":"2023-10-30T14:26:46.584097+0000","last_became_active":"2023-10-30T13:23:18.972069+0000","last_became_peered":"2023-10-30T13:23:18.972069+0000","last_unstale":"2023-10-30T14:26:46.584097+0000","last_undegraded":"2023-10-30T14:26:46.584097+0000","last_fullsized":"2023-10-30T14:26:46.584097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'780","last_scrub_stamp":"2023-10-30T14:26:46.583879+0000","last_deep_scrub":"110'780","last_deep_scrub_stamp":"2023-10-30T14:26:46.583879+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.583879+0000","objects_scrubbed":53,"log_size":780,"log_dups_size":0,"ondisk_log_size":780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:52.998170+0000","scrub_duration":0.20502225800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":234,"num_read_kb":298146,"num_write":829,"num_write_kb":54985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":494954,"num_omap_keys":946,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"110'2149","reported_seq":5030,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.527465+0000","last_change":"2023-10-30T14:27:00.527596+0000","last_active":"2023-10-30T14:27:00.527465+0000","last_peered":"2023-10-30T14:27:00.527465+0000","last_clean":"2023-10-30T14:27:00.527465+0000","last_became_active":"2023-10-30T13:23:19.693313+0000","last_became_peered":"2023-10-30T13:23:19.693313+0000","last_unstale":"2023-10-30T14:27:00.527465+0000","last_undegraded":"2023-10-30T14:27:00.527465+0000","last_fullsized":"2023-10-30T14:27:00.527465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2149","last_scrub_stamp":"2023-10-30T14:27:00.527256+0000","last_deep_scrub":"110'2149","last_deep_scrub_stamp":"2023-10-30T14:27:00.527256+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.527256+0000","objects_scrubbed":53,"log_size":2149,"log_dups_size":0,"ondisk_log_size":2149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:55:50.223566+0000","scrub_duration":0.14114542399999999,"objects_trimmed":0,"snaptrim_duration":0.50412874500000004,"stat_sum":{"num_bytes":0,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":148,"num_read_kb":43506,"num_write":1375,"num_write_kb":26417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"110'876","reported_seq":1232,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.183186+0000","last_change":"2023-10-30T14:26:52.183292+0000","last_active":"2023-10-30T14:26:52.183186+0000","last_peered":"2023-10-30T14:26:52.183186+0000","last_clean":"2023-10-30T14:26:52.183186+0000","last_became_active":"2023-10-30T13:23:18.947871+0000","last_became_peered":"2023-10-30T13:23:18.947871+0000","last_unstale":"2023-10-30T14:26:52.183186+0000","last_undegraded":"2023-10-30T14:26:52.183186+0000","last_fullsized":"2023-10-30T14:26:52.183186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'876","last_scrub_stamp":"2023-10-30T14:26:52.183105+0000","last_deep_scrub":"110'876","last_deep_scrub_stamp":"2023-10-30T14:26:52.183105+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.183105+0000","objects_scrubbed":51,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:18:36.571108+0000","scrub_duration":0.119037509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971632,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":481,"num_read_kb":229594,"num_write":1003,"num_write_kb":42855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":532426,"num_omap_keys":823,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"110'2380","reported_seq":5218,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.002361+0000","last_change":"2023-10-30T14:26:46.002570+0000","last_active":"2023-10-30T14:26:46.002361+0000","last_peered":"2023-10-30T14:26:46.002361+0000","last_clean":"2023-10-30T14:26:46.002361+0000","last_became_active":"2023-10-30T13:23:19.727874+0000","last_became_peered":"2023-10-30T13:23:19.727874+0000","last_unstale":"2023-10-30T14:26:46.002361+0000","last_undegraded":"2023-10-30T14:26:46.002361+0000","last_fullsized":"2023-10-30T14:26:46.002361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2380","last_scrub_stamp":"2023-10-30T14:26:46.002134+0000","last_deep_scrub":"110'2380","last_deep_scrub_stamp":"2023-10-30T14:26:46.002134+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.002134+0000","objects_scrubbed":46,"log_size":2380,"log_dups_size":0,"ondisk_log_size":2380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:38.920284+0000","scrub_duration":0.14576676,"objects_trimmed":0,"snaptrim_duration":0.70955538100000004,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":166,"num_read_kb":25446,"num_write":1448,"num_write_kb":32435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"110'1046","reported_seq":1341,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.977448+0000","last_change":"2023-10-30T14:26:42.977518+0000","last_active":"2023-10-30T14:26:42.977448+0000","last_peered":"2023-10-30T14:26:42.977448+0000","last_clean":"2023-10-30T14:26:42.977448+0000","last_became_active":"2023-10-30T13:23:18.960821+0000","last_became_peered":"2023-10-30T13:23:18.960821+0000","last_unstale":"2023-10-30T14:26:42.977448+0000","last_undegraded":"2023-10-30T14:26:42.977448+0000","last_fullsized":"2023-10-30T14:26:42.977448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1046","last_scrub_stamp":"2023-10-30T14:26:42.977329+0000","last_deep_scrub":"110'1046","last_deep_scrub_stamp":"2023-10-30T14:26:42.977329+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.977329+0000","objects_scrubbed":51,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:00.448502+0000","scrub_duration":0.210468818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":214,"num_read_kb":373366,"num_write":1150,"num_write_kb":71086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":337344,"num_omap_keys":660,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"110'2278","reported_seq":5077,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:36.022639+0000","last_change":"2023-10-30T14:27:36.022780+0000","last_active":"2023-10-30T14:27:36.022639+0000","last_peered":"2023-10-30T14:27:36.022639+0000","last_clean":"2023-10-30T14:27:36.022639+0000","last_became_active":"2023-10-30T13:23:19.720006+0000","last_became_peered":"2023-10-30T13:23:19.720006+0000","last_unstale":"2023-10-30T14:27:36.022639+0000","last_undegraded":"2023-10-30T14:27:36.022639+0000","last_fullsized":"2023-10-30T14:27:36.022639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2278","last_scrub_stamp":"2023-10-30T14:27:36.022470+0000","last_deep_scrub":"110'2278","last_deep_scrub_stamp":"2023-10-30T14:27:36.022470+0000","last_clean_scrub_stamp":"2023-10-30T14:27:36.022470+0000","objects_scrubbed":40,"log_size":2278,"log_dups_size":0,"ondisk_log_size":2278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:51.873473+0000","scrub_duration":0.121129692,"objects_trimmed":0,"snaptrim_duration":0.89413299700000004,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":142,"num_read_kb":18968,"num_write":1278,"num_write_kb":24747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"110'668","reported_seq":989,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:47.033482+0000","last_change":"2023-10-30T14:26:47.033575+0000","last_active":"2023-10-30T14:26:47.033482+0000","last_peered":"2023-10-30T14:26:47.033482+0000","last_clean":"2023-10-30T14:26:47.033482+0000","last_became_active":"2023-10-30T13:23:18.948755+0000","last_became_peered":"2023-10-30T13:23:18.948755+0000","last_unstale":"2023-10-30T14:26:47.033482+0000","last_undegraded":"2023-10-30T14:26:47.033482+0000","last_fullsized":"2023-10-30T14:26:47.033482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'668","last_scrub_stamp":"2023-10-30T14:26:47.033389+0000","last_deep_scrub":"110'668","last_deep_scrub_stamp":"2023-10-30T14:26:47.033389+0000","last_clean_scrub_stamp":"2023-10-30T14:26:47.033389+0000","objects_scrubbed":42,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:25.680237+0000","scrub_duration":0.095992737999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":280,"num_read_kb":492842,"num_write":679,"num_write_kb":83547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":470248,"num_omap_keys":762,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"110'2131","reported_seq":5015,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:33.067495+0000","last_change":"2023-10-30T14:27:33.067594+0000","last_active":"2023-10-30T14:27:33.067495+0000","last_peered":"2023-10-30T14:27:33.067495+0000","last_clean":"2023-10-30T14:27:33.067495+0000","last_became_active":"2023-10-30T13:23:19.722475+0000","last_became_peered":"2023-10-30T13:23:19.722475+0000","last_unstale":"2023-10-30T14:27:33.067495+0000","last_undegraded":"2023-10-30T14:27:33.067495+0000","last_fullsized":"2023-10-30T14:27:33.067495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2131","last_scrub_stamp":"2023-10-30T14:27:33.067332+0000","last_deep_scrub":"110'2131","last_deep_scrub_stamp":"2023-10-30T14:27:33.067332+0000","last_clean_scrub_stamp":"2023-10-30T14:27:33.067332+0000","objects_scrubbed":35,"log_size":2131,"log_dups_size":0,"ondisk_log_size":2131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:28:09.043194+0000","scrub_duration":0.10920801500000001,"objects_trimmed":0,"snaptrim_duration":0.88483857600000004,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":152,"num_read_kb":61046,"num_write":1310,"num_write_kb":36653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"110'790","reported_seq":1095,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:52.425710+0000","last_change":"2023-10-30T14:26:52.425764+0000","last_active":"2023-10-30T14:26:52.425710+0000","last_peered":"2023-10-30T14:26:52.425710+0000","last_clean":"2023-10-30T14:26:52.425710+0000","last_became_active":"2023-10-30T13:23:18.970490+0000","last_became_peered":"2023-10-30T13:23:18.970490+0000","last_unstale":"2023-10-30T14:26:52.425710+0000","last_undegraded":"2023-10-30T14:26:52.425710+0000","last_fullsized":"2023-10-30T14:26:52.425710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'790","last_scrub_stamp":"2023-10-30T14:26:52.425638+0000","last_deep_scrub":"110'790","last_deep_scrub_stamp":"2023-10-30T14:26:52.425638+0000","last_clean_scrub_stamp":"2023-10-30T14:26:52.425638+0000","objects_scrubbed":60,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:57:40.747257+0000","scrub_duration":0.18100369799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364699,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":233,"num_read_kb":370768,"num_write":815,"num_write_kb":71238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":549758,"num_omap_keys":876,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"110'2281","reported_seq":5152,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:14.156230+0000","last_change":"2023-10-30T14:27:14.156348+0000","last_active":"2023-10-30T14:27:14.156230+0000","last_peered":"2023-10-30T14:27:14.156230+0000","last_clean":"2023-10-30T14:27:14.156230+0000","last_became_active":"2023-10-30T13:23:19.699305+0000","last_became_peered":"2023-10-30T13:23:19.699305+0000","last_unstale":"2023-10-30T14:27:14.156230+0000","last_undegraded":"2023-10-30T14:27:14.156230+0000","last_fullsized":"2023-10-30T14:27:14.156230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2281","last_scrub_stamp":"2023-10-30T14:27:14.156044+0000","last_deep_scrub":"110'2281","last_deep_scrub_stamp":"2023-10-30T14:27:14.156044+0000","last_clean_scrub_stamp":"2023-10-30T14:27:14.156044+0000","objects_scrubbed":41,"log_size":2281,"log_dups_size":0,"ondisk_log_size":2281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:23:25.400045+0000","scrub_duration":0.105525299,"objects_trimmed":0,"snaptrim_duration":0.98441546899999999,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":178,"num_read_kb":29706,"num_write":1375,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"110'1121","reported_seq":1413,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.293606+0000","last_change":"2023-10-30T14:26:54.293672+0000","last_active":"2023-10-30T14:26:54.293606+0000","last_peered":"2023-10-30T14:26:54.293606+0000","last_clean":"2023-10-30T14:26:54.293606+0000","last_became_active":"2023-10-30T13:23:18.977555+0000","last_became_peered":"2023-10-30T13:23:18.977555+0000","last_unstale":"2023-10-30T14:26:54.293606+0000","last_undegraded":"2023-10-30T14:26:54.293606+0000","last_fullsized":"2023-10-30T14:26:54.293606+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1121","last_scrub_stamp":"2023-10-30T14:26:54.293526+0000","last_deep_scrub":"110'1121","last_deep_scrub_stamp":"2023-10-30T14:26:54.293526+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.293526+0000","objects_scrubbed":43,"log_size":1121,"log_dups_size":0,"ondisk_log_size":1121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:43:17.009305+0000","scrub_duration":0.241353279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50332230,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":210,"num_read_kb":406971,"num_write":1089,"num_write_kb":71553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":220904,"num_omap_keys":399,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"110'2078","reported_seq":4939,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:16.182932+0000","last_change":"2023-10-30T14:27:16.183032+0000","last_active":"2023-10-30T14:27:16.182932+0000","last_peered":"2023-10-30T14:27:16.182932+0000","last_clean":"2023-10-30T14:27:16.182932+0000","last_became_active":"2023-10-30T13:23:19.723144+0000","last_became_peered":"2023-10-30T13:23:19.723144+0000","last_unstale":"2023-10-30T14:27:16.182932+0000","last_undegraded":"2023-10-30T14:27:16.182932+0000","last_fullsized":"2023-10-30T14:27:16.182932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2078","last_scrub_stamp":"2023-10-30T14:27:16.182829+0000","last_deep_scrub":"110'2078","last_deep_scrub_stamp":"2023-10-30T14:27:16.182829+0000","last_clean_scrub_stamp":"2023-10-30T14:27:16.182829+0000","objects_scrubbed":38,"log_size":2078,"log_dups_size":0,"ondisk_log_size":2078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:14:39.775252+0000","scrub_duration":0.132528019,"objects_trimmed":0,"snaptrim_duration":0.84782311300000002,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":122,"num_read_kb":1096,"num_write":1268,"num_write_kb":17447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"110'119","reported_seq":298,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:47.899032+0000","last_change":"2023-10-30T14:26:47.899032+0000","last_active":"2023-10-30T14:26:47.899032+0000","last_peered":"2023-10-30T14:26:47.899032+0000","last_clean":"2023-10-30T14:26:47.899032+0000","last_became_active":"2023-10-30T13:23:08.207303+0000","last_became_peered":"2023-10-30T13:23:08.207303+0000","last_unstale":"2023-10-30T14:26:47.899032+0000","last_undegraded":"2023-10-30T14:26:47.899032+0000","last_fullsized":"2023-10-30T14:26:47.899032+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'119","last_scrub_stamp":"2023-10-30T14:26:47.898726+0000","last_deep_scrub":"110'119","last_deep_scrub_stamp":"2023-10-30T14:26:47.898726+0000","last_clean_scrub_stamp":"2023-10-30T14:26:47.898726+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:29.562342+0000","scrub_duration":0.0091457540000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"110'909","reported_seq":1259,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:50.195816+0000","last_change":"2023-10-30T14:26:50.195816+0000","last_active":"2023-10-30T14:26:50.195816+0000","last_peered":"2023-10-30T14:26:50.195816+0000","last_clean":"2023-10-30T14:26:50.195816+0000","last_became_active":"2023-10-30T13:23:18.977759+0000","last_became_peered":"2023-10-30T13:23:18.977759+0000","last_unstale":"2023-10-30T14:26:50.195816+0000","last_undegraded":"2023-10-30T14:26:50.195816+0000","last_fullsized":"2023-10-30T14:26:50.195816+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'909","last_scrub_stamp":"2023-10-30T14:26:50.195649+0000","last_deep_scrub":"110'909","last_deep_scrub_stamp":"2023-10-30T14:26:50.195649+0000","last_clean_scrub_stamp":"2023-10-30T14:26:50.195649+0000","objects_scrubbed":60,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:12.567856+0000","scrub_duration":0.19057717900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331714,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":319,"num_read_kb":439640,"num_write":936,"num_write_kb":75464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":459843,"num_omap_keys":807,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"110'2244","reported_seq":5042,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.843044+0000","last_change":"2023-10-30T14:26:57.843044+0000","last_active":"2023-10-30T14:26:57.843044+0000","last_peered":"2023-10-30T14:26:57.843044+0000","last_clean":"2023-10-30T14:26:57.843044+0000","last_became_active":"2023-10-30T13:23:19.710951+0000","last_became_peered":"2023-10-30T13:23:19.710951+0000","last_unstale":"2023-10-30T14:26:57.843044+0000","last_undegraded":"2023-10-30T14:26:57.843044+0000","last_fullsized":"2023-10-30T14:26:57.843044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2244","last_scrub_stamp":"2023-10-30T14:26:57.842882+0000","last_deep_scrub":"110'2244","last_deep_scrub_stamp":"2023-10-30T14:26:57.842882+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.842882+0000","objects_scrubbed":46,"log_size":2244,"log_dups_size":0,"ondisk_log_size":2244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:34:34.719838+0000","scrub_duration":0.10386124300000001,"objects_trimmed":0,"snaptrim_duration":1.003812562,"stat_sum":{"num_bytes":0,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":127,"num_read_kb":26261,"num_write":1294,"num_write_kb":24882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"110'1232","reported_seq":1556,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.159534+0000","last_change":"2023-10-30T14:26:53.159534+0000","last_active":"2023-10-30T14:26:53.159534+0000","last_peered":"2023-10-30T14:26:53.159534+0000","last_clean":"2023-10-30T14:26:53.159534+0000","last_became_active":"2023-10-30T13:23:18.984448+0000","last_became_peered":"2023-10-30T13:23:18.984448+0000","last_unstale":"2023-10-30T14:26:53.159534+0000","last_undegraded":"2023-10-30T14:26:53.159534+0000","last_fullsized":"2023-10-30T14:26:53.159534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1232","last_scrub_stamp":"2023-10-30T14:26:53.159107+0000","last_deep_scrub":"110'1232","last_deep_scrub_stamp":"2023-10-30T14:26:53.159107+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.159107+0000","objects_scrubbed":46,"log_size":1232,"log_dups_size":0,"ondisk_log_size":1232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:28:27.847424+0000","scrub_duration":0.24891354399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":256,"num_read_kb":488007,"num_write":1246,"num_write_kb":83351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":346758,"num_omap_keys":680,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"110'2013","reported_seq":4888,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:51.757396+0000","last_change":"2023-10-30T14:26:51.757396+0000","last_active":"2023-10-30T14:26:51.757396+0000","last_peered":"2023-10-30T14:26:51.757396+0000","last_clean":"2023-10-30T14:26:51.757396+0000","last_became_active":"2023-10-30T13:23:19.717510+0000","last_became_peered":"2023-10-30T13:23:19.717510+0000","last_unstale":"2023-10-30T14:26:51.757396+0000","last_undegraded":"2023-10-30T14:26:51.757396+0000","last_fullsized":"2023-10-30T14:26:51.757396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2013","last_scrub_stamp":"2023-10-30T14:26:51.757217+0000","last_deep_scrub":"110'2013","last_deep_scrub_stamp":"2023-10-30T14:26:51.757217+0000","last_clean_scrub_stamp":"2023-10-30T14:26:51.757217+0000","objects_scrubbed":34,"log_size":2013,"log_dups_size":0,"ondisk_log_size":2013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:49.616919+0000","scrub_duration":0.071006446000000001,"objects_trimmed":0,"snaptrim_duration":0.92159720499999997,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":129,"num_read_kb":13045,"num_write":1222,"num_write_kb":11637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"110'2000","reported_seq":3257,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:58.136871+0000","last_change":"2023-10-30T14:26:58.136871+0000","last_active":"2023-10-30T14:26:58.136871+0000","last_peered":"2023-10-30T14:26:58.136871+0000","last_clean":"2023-10-30T14:26:58.136871+0000","last_became_active":"2023-10-30T13:23:18.949096+0000","last_became_peered":"2023-10-30T13:23:18.949096+0000","last_unstale":"2023-10-30T14:26:58.136871+0000","last_undegraded":"2023-10-30T14:26:58.136871+0000","last_fullsized":"2023-10-30T14:26:58.136871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2000","last_scrub_stamp":"2023-10-30T14:26:58.136733+0000","last_deep_scrub":"110'2000","last_deep_scrub_stamp":"2023-10-30T14:26:58.136733+0000","last_clean_scrub_stamp":"2023-10-30T14:26:58.136733+0000","objects_scrubbed":51,"log_size":2000,"log_dups_size":0,"ondisk_log_size":2000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:54.997528+0000","scrub_duration":0.11551852899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40060913,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":1167,"num_read_kb":359975,"num_write":2020,"num_write_kb":69890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":329700,"num_omap_keys":585,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"110'2306","reported_seq":5130,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:56.314854+0000","last_change":"2023-10-30T14:26:56.314854+0000","last_active":"2023-10-30T14:26:56.314854+0000","last_peered":"2023-10-30T14:26:56.314854+0000","last_clean":"2023-10-30T14:26:56.314854+0000","last_became_active":"2023-10-30T13:23:19.677610+0000","last_became_peered":"2023-10-30T13:23:19.677610+0000","last_unstale":"2023-10-30T14:26:56.314854+0000","last_undegraded":"2023-10-30T14:26:56.314854+0000","last_fullsized":"2023-10-30T14:26:56.314854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2306","last_scrub_stamp":"2023-10-30T14:26:56.314673+0000","last_deep_scrub":"110'2306","last_deep_scrub_stamp":"2023-10-30T14:26:56.314673+0000","last_clean_scrub_stamp":"2023-10-30T14:26:56.314673+0000","objects_scrubbed":45,"log_size":2306,"log_dups_size":0,"ondisk_log_size":2306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:16:05.533537+0000","scrub_duration":0.059182301999999999,"objects_trimmed":0,"snaptrim_duration":0.40657480699999998,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":145,"num_read_kb":55105,"num_write":1354,"num_write_kb":28005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"110'1533","reported_seq":1850,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:45.029718+0000","last_change":"2023-10-30T14:26:45.029718+0000","last_active":"2023-10-30T14:26:45.029718+0000","last_peered":"2023-10-30T14:26:45.029718+0000","last_clean":"2023-10-30T14:26:45.029718+0000","last_became_active":"2023-10-30T13:23:18.983607+0000","last_became_peered":"2023-10-30T13:23:18.983607+0000","last_unstale":"2023-10-30T14:26:45.029718+0000","last_undegraded":"2023-10-30T14:26:45.029718+0000","last_fullsized":"2023-10-30T14:26:45.029718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1533","last_scrub_stamp":"2023-10-30T14:26:45.029381+0000","last_deep_scrub":"110'1533","last_deep_scrub_stamp":"2023-10-30T14:26:45.029381+0000","last_clean_scrub_stamp":"2023-10-30T14:26:45.029381+0000","objects_scrubbed":45,"log_size":1533,"log_dups_size":0,"ondisk_log_size":1533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:01:54.864240+0000","scrub_duration":0.21060546799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":235,"num_read_kb":509467,"num_write":1537,"num_write_kb":83943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":361192,"num_omap_keys":645,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"110'2242","reported_seq":5078,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:46.859846+0000","last_change":"2023-10-30T14:26:46.859846+0000","last_active":"2023-10-30T14:26:46.859846+0000","last_peered":"2023-10-30T14:26:46.859846+0000","last_clean":"2023-10-30T14:26:46.859846+0000","last_became_active":"2023-10-30T13:23:19.691093+0000","last_became_peered":"2023-10-30T13:23:19.691093+0000","last_unstale":"2023-10-30T14:26:46.859846+0000","last_undegraded":"2023-10-30T14:26:46.859846+0000","last_fullsized":"2023-10-30T14:26:46.859846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2242","last_scrub_stamp":"2023-10-30T14:26:46.859379+0000","last_deep_scrub":"110'2242","last_deep_scrub_stamp":"2023-10-30T14:26:46.859379+0000","last_clean_scrub_stamp":"2023-10-30T14:26:46.859379+0000","objects_scrubbed":34,"log_size":2242,"log_dups_size":0,"ondisk_log_size":2242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:58:59.480931+0000","scrub_duration":0.104056545,"objects_trimmed":0,"snaptrim_duration":0.59694472600000004,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":143,"num_read_kb":13239,"num_write":1306,"num_write_kb":17120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"110'663","reported_seq":986,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:50.346292+0000","last_change":"2023-10-30T14:26:50.346292+0000","last_active":"2023-10-30T14:26:50.346292+0000","last_peered":"2023-10-30T14:26:50.346292+0000","last_clean":"2023-10-30T14:26:50.346292+0000","last_became_active":"2023-10-30T13:23:18.957812+0000","last_became_peered":"2023-10-30T13:23:18.957812+0000","last_unstale":"2023-10-30T14:26:50.346292+0000","last_undegraded":"2023-10-30T14:26:50.346292+0000","last_fullsized":"2023-10-30T14:26:50.346292+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'663","last_scrub_stamp":"2023-10-30T14:26:50.346173+0000","last_deep_scrub":"110'663","last_deep_scrub_stamp":"2023-10-30T14:26:50.346173+0000","last_clean_scrub_stamp":"2023-10-30T14:26:50.346173+0000","objects_scrubbed":53,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:23:57.407805+0000","scrub_duration":0.108375394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":230,"num_read_kb":520969,"num_write":671,"num_write_kb":95828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":375058,"num_omap_keys":621,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"110'2212","reported_seq":5041,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:20.112321+0000","last_change":"2023-10-30T14:27:20.112433+0000","last_active":"2023-10-30T14:27:20.112321+0000","last_peered":"2023-10-30T14:27:20.112321+0000","last_clean":"2023-10-30T14:27:20.112321+0000","last_became_active":"2023-10-30T13:23:19.712032+0000","last_became_peered":"2023-10-30T13:23:19.712032+0000","last_unstale":"2023-10-30T14:27:20.112321+0000","last_undegraded":"2023-10-30T14:27:20.112321+0000","last_fullsized":"2023-10-30T14:27:20.112321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2212","last_scrub_stamp":"2023-10-30T14:27:20.112126+0000","last_deep_scrub":"110'2212","last_deep_scrub_stamp":"2023-10-30T14:27:20.112126+0000","last_clean_scrub_stamp":"2023-10-30T14:27:20.112126+0000","objects_scrubbed":32,"log_size":2212,"log_dups_size":0,"ondisk_log_size":2212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:46.492910+0000","scrub_duration":0.087815049000000006,"objects_trimmed":0,"snaptrim_duration":1.075105449,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":140,"num_read_kb":60418,"num_write":1286,"num_write_kb":36950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"110'1084","reported_seq":1379,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:59.155850+0000","last_change":"2023-10-30T14:26:59.155993+0000","last_active":"2023-10-30T14:26:59.155850+0000","last_peered":"2023-10-30T14:26:59.155850+0000","last_clean":"2023-10-30T14:26:59.155850+0000","last_became_active":"2023-10-30T13:23:18.984287+0000","last_became_peered":"2023-10-30T13:23:18.984287+0000","last_unstale":"2023-10-30T14:26:59.155850+0000","last_undegraded":"2023-10-30T14:26:59.155850+0000","last_fullsized":"2023-10-30T14:26:59.155850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1084","last_scrub_stamp":"2023-10-30T14:26:59.155631+0000","last_deep_scrub":"110'1084","last_deep_scrub_stamp":"2023-10-30T14:26:59.155631+0000","last_clean_scrub_stamp":"2023-10-30T14:26:59.155631+0000","objects_scrubbed":38,"log_size":1084,"log_dups_size":0,"ondisk_log_size":1084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:20:05.652224+0000","scrub_duration":0.220506067,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44106025,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":214,"num_read_kb":377318,"num_write":1089,"num_write_kb":69046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":276186,"num_omap_keys":520,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"110'2247","reported_seq":5092,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:15.229873+0000","last_change":"2023-10-30T14:27:15.229873+0000","last_active":"2023-10-30T14:27:15.229873+0000","last_peered":"2023-10-30T14:27:15.229873+0000","last_clean":"2023-10-30T14:27:15.229873+0000","last_became_active":"2023-10-30T13:23:19.679352+0000","last_became_peered":"2023-10-30T13:23:19.679352+0000","last_unstale":"2023-10-30T14:27:15.229873+0000","last_undegraded":"2023-10-30T14:27:15.229873+0000","last_fullsized":"2023-10-30T14:27:15.229873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2247","last_scrub_stamp":"2023-10-30T14:27:15.229719+0000","last_deep_scrub":"110'2247","last_deep_scrub_stamp":"2023-10-30T14:27:15.229719+0000","last_clean_scrub_stamp":"2023-10-30T14:27:15.229719+0000","objects_scrubbed":37,"log_size":2247,"log_dups_size":0,"ondisk_log_size":2247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T02:06:33.811869+0000","scrub_duration":0.052142862999999998,"objects_trimmed":0,"snaptrim_duration":0.36741958800000002,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":157,"num_read_kb":35362,"num_write":1332,"num_write_kb":31772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"110'1165","reported_seq":1469,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:54.451066+0000","last_change":"2023-10-30T14:26:54.451118+0000","last_active":"2023-10-30T14:26:54.451066+0000","last_peered":"2023-10-30T14:26:54.451066+0000","last_clean":"2023-10-30T14:26:54.451066+0000","last_became_active":"2023-10-30T13:23:18.977726+0000","last_became_peered":"2023-10-30T13:23:18.977726+0000","last_unstale":"2023-10-30T14:26:54.451066+0000","last_undegraded":"2023-10-30T14:26:54.451066+0000","last_fullsized":"2023-10-30T14:26:54.451066+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1165","last_scrub_stamp":"2023-10-30T14:26:54.450996+0000","last_deep_scrub":"110'1165","last_deep_scrub_stamp":"2023-10-30T14:26:54.450996+0000","last_clean_scrub_stamp":"2023-10-30T14:26:54.450996+0000","objects_scrubbed":39,"log_size":1165,"log_dups_size":0,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:18:07.890199+0000","scrub_duration":0.17568342300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":247,"num_read_kb":365780,"num_write":1783,"num_write_kb":65025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":395035,"num_omap_keys":693,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"110'2123","reported_seq":4968,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:10.178670+0000","last_change":"2023-10-30T14:27:10.178735+0000","last_active":"2023-10-30T14:27:10.178670+0000","last_peered":"2023-10-30T14:27:10.178670+0000","last_clean":"2023-10-30T14:27:10.178670+0000","last_became_active":"2023-10-30T13:24:01.001412+0000","last_became_peered":"2023-10-30T13:24:01.001412+0000","last_unstale":"2023-10-30T14:27:10.178670+0000","last_undegraded":"2023-10-30T14:27:10.178670+0000","last_fullsized":"2023-10-30T14:27:10.178670+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2123","last_scrub_stamp":"2023-10-30T14:27:10.178582+0000","last_deep_scrub":"110'2123","last_deep_scrub_stamp":"2023-10-30T14:27:10.178582+0000","last_clean_scrub_stamp":"2023-10-30T14:27:10.178582+0000","objects_scrubbed":35,"log_size":2123,"log_dups_size":0,"ondisk_log_size":2123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:49.167738+0000","scrub_duration":0.079853908000000001,"objects_trimmed":0,"snaptrim_duration":0.32700652899999999,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":137,"num_read_kb":5811,"num_write":1214,"num_write_kb":16143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"110'515","reported_seq":831,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.490148+0000","last_change":"2023-10-30T14:26:53.490236+0000","last_active":"2023-10-30T14:26:53.490148+0000","last_peered":"2023-10-30T14:26:53.490148+0000","last_clean":"2023-10-30T14:26:53.490148+0000","last_became_active":"2023-10-30T13:23:18.967828+0000","last_became_peered":"2023-10-30T13:23:18.967828+0000","last_unstale":"2023-10-30T14:26:53.490148+0000","last_undegraded":"2023-10-30T14:26:53.490148+0000","last_fullsized":"2023-10-30T14:26:53.490148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'515","last_scrub_stamp":"2023-10-30T14:26:53.490042+0000","last_deep_scrub":"110'515","last_deep_scrub_stamp":"2023-10-30T14:26:53.490042+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.490042+0000","objects_scrubbed":47,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:16.217409+0000","scrub_duration":0.22512389199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":281,"num_read_kb":442743,"num_write":555,"num_write_kb":70797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":243443,"num_omap_keys":457,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"110'2226","reported_seq":5069,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:59.865953+0000","last_change":"2023-10-30T14:26:59.866026+0000","last_active":"2023-10-30T14:26:59.865953+0000","last_peered":"2023-10-30T14:26:59.865953+0000","last_clean":"2023-10-30T14:26:59.865953+0000","last_became_active":"2023-10-30T13:23:19.685118+0000","last_became_peered":"2023-10-30T13:23:19.685118+0000","last_unstale":"2023-10-30T14:26:59.865953+0000","last_undegraded":"2023-10-30T14:26:59.865953+0000","last_fullsized":"2023-10-30T14:26:59.865953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2226","last_scrub_stamp":"2023-10-30T14:26:59.865860+0000","last_deep_scrub":"110'2226","last_deep_scrub_stamp":"2023-10-30T14:26:59.865860+0000","last_clean_scrub_stamp":"2023-10-30T14:26:59.865860+0000","objects_scrubbed":40,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:25.638448+0000","scrub_duration":0.053687321000000003,"objects_trimmed":0,"snaptrim_duration":1.183697322,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":144,"num_read_kb":25445,"num_write":1326,"num_write_kb":20705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"110'584","reported_seq":860,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.994203+0000","last_change":"2023-10-30T14:26:53.994349+0000","last_active":"2023-10-30T14:26:53.994203+0000","last_peered":"2023-10-30T14:26:53.994203+0000","last_clean":"2023-10-30T14:26:53.994203+0000","last_became_active":"2023-10-30T13:23:18.985252+0000","last_became_peered":"2023-10-30T13:23:18.985252+0000","last_unstale":"2023-10-30T14:26:53.994203+0000","last_undegraded":"2023-10-30T14:26:53.994203+0000","last_fullsized":"2023-10-30T14:26:53.994203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'584","last_scrub_stamp":"2023-10-30T14:26:53.993981+0000","last_deep_scrub":"110'584","last_deep_scrub_stamp":"2023-10-30T14:26:53.993981+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.993981+0000","objects_scrubbed":40,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:54.497876+0000","scrub_duration":0.12048839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":236,"num_read_kb":274544,"num_write":603,"num_write_kb":42177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":323567,"num_omap_keys":593,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"110'2168","reported_seq":5027,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:49.864350+0000","last_change":"2023-10-30T14:26:49.864486+0000","last_active":"2023-10-30T14:26:49.864350+0000","last_peered":"2023-10-30T14:26:49.864350+0000","last_clean":"2023-10-30T14:26:49.864350+0000","last_became_active":"2023-10-30T13:23:19.680473+0000","last_became_peered":"2023-10-30T13:23:19.680473+0000","last_unstale":"2023-10-30T14:26:49.864350+0000","last_undegraded":"2023-10-30T14:26:49.864350+0000","last_fullsized":"2023-10-30T14:26:49.864350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2168","last_scrub_stamp":"2023-10-30T14:26:49.864161+0000","last_deep_scrub":"110'2168","last_deep_scrub_stamp":"2023-10-30T14:26:49.864161+0000","last_clean_scrub_stamp":"2023-10-30T14:26:49.864161+0000","objects_scrubbed":45,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:54:18.553003+0000","scrub_duration":0.11066809399999999,"objects_trimmed":0,"snaptrim_duration":0.56267233699999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":144,"num_read_kb":34655,"num_write":1307,"num_write_kb":20329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"110'1112","reported_seq":1394,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:45.299966+0000","last_change":"2023-10-30T14:26:45.300020+0000","last_active":"2023-10-30T14:26:45.299966+0000","last_peered":"2023-10-30T14:26:45.299966+0000","last_clean":"2023-10-30T14:26:45.299966+0000","last_became_active":"2023-10-30T13:23:18.963094+0000","last_became_peered":"2023-10-30T13:23:18.963094+0000","last_unstale":"2023-10-30T14:26:45.299966+0000","last_undegraded":"2023-10-30T14:26:45.299966+0000","last_fullsized":"2023-10-30T14:26:45.299966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1112","last_scrub_stamp":"2023-10-30T14:26:45.299861+0000","last_deep_scrub":"110'1112","last_deep_scrub_stamp":"2023-10-30T14:26:45.299861+0000","last_clean_scrub_stamp":"2023-10-30T14:26:45.299861+0000","objects_scrubbed":51,"log_size":1112,"log_dups_size":0,"ondisk_log_size":1112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:30.314690+0000","scrub_duration":0.15631188700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":237,"num_read_kb":249991,"num_write":1213,"num_write_kb":43538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":609555,"num_omap_keys":1142,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"110'2316","reported_seq":5113,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:18.616129+0000","last_change":"2023-10-30T14:27:18.616224+0000","last_active":"2023-10-30T14:27:18.616129+0000","last_peered":"2023-10-30T14:27:18.616129+0000","last_clean":"2023-10-30T14:27:18.616129+0000","last_became_active":"2023-10-30T13:23:19.677706+0000","last_became_peered":"2023-10-30T13:23:19.677706+0000","last_unstale":"2023-10-30T14:27:18.616129+0000","last_undegraded":"2023-10-30T14:27:18.616129+0000","last_fullsized":"2023-10-30T14:27:18.616129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2316","last_scrub_stamp":"2023-10-30T14:27:18.616043+0000","last_deep_scrub":"110'2316","last_deep_scrub_stamp":"2023-10-30T14:27:18.616043+0000","last_clean_scrub_stamp":"2023-10-30T14:27:18.616043+0000","objects_scrubbed":45,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:14:36.881982+0000","scrub_duration":0.061902202000000003,"objects_trimmed":0,"snaptrim_duration":0.93815086199999997,"stat_sum":{"num_bytes":0,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":129,"num_read_kb":1227,"num_write":1352,"num_write_kb":12536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"110'1527","reported_seq":1874,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:53.480258+0000","last_change":"2023-10-30T14:26:53.480338+0000","last_active":"2023-10-30T14:26:53.480258+0000","last_peered":"2023-10-30T14:26:53.480258+0000","last_clean":"2023-10-30T14:26:53.480258+0000","last_became_active":"2023-10-30T13:23:18.951993+0000","last_became_peered":"2023-10-30T13:23:18.951993+0000","last_unstale":"2023-10-30T14:26:53.480258+0000","last_undegraded":"2023-10-30T14:26:53.480258+0000","last_fullsized":"2023-10-30T14:26:53.480258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1527","last_scrub_stamp":"2023-10-30T14:26:53.480148+0000","last_deep_scrub":"110'1527","last_deep_scrub_stamp":"2023-10-30T14:26:53.480148+0000","last_clean_scrub_stamp":"2023-10-30T14:26:53.480148+0000","objects_scrubbed":54,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:31:02.686045+0000","scrub_duration":0.25185449900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":321,"num_read_kb":438308,"num_write":2003,"num_write_kb":98279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":359842,"num_omap_keys":684,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"110'2123","reported_seq":4973,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:55.746681+0000","last_change":"2023-10-30T14:26:55.746755+0000","last_active":"2023-10-30T14:26:55.746681+0000","last_peered":"2023-10-30T14:26:55.746681+0000","last_clean":"2023-10-30T14:26:55.746681+0000","last_became_active":"2023-10-30T13:23:19.688218+0000","last_became_peered":"2023-10-30T13:23:19.688218+0000","last_unstale":"2023-10-30T14:26:55.746681+0000","last_undegraded":"2023-10-30T14:26:55.746681+0000","last_fullsized":"2023-10-30T14:26:55.746681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2123","last_scrub_stamp":"2023-10-30T14:26:55.746576+0000","last_deep_scrub":"110'2123","last_deep_scrub_stamp":"2023-10-30T14:26:55.746576+0000","last_clean_scrub_stamp":"2023-10-30T14:26:55.746576+0000","objects_scrubbed":27,"log_size":2123,"log_dups_size":0,"ondisk_log_size":2123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:08:17.663937+0000","scrub_duration":0.039827926,"objects_trimmed":0,"snaptrim_duration":0.94968995099999998,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":141,"num_read_kb":11852,"num_write":1237,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"110'1108","reported_seq":1435,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:57.161003+0000","last_change":"2023-10-30T14:26:57.161075+0000","last_active":"2023-10-30T14:26:57.161003+0000","last_peered":"2023-10-30T14:26:57.161003+0000","last_clean":"2023-10-30T14:26:57.161003+0000","last_became_active":"2023-10-30T13:23:18.984221+0000","last_became_peered":"2023-10-30T13:23:18.984221+0000","last_unstale":"2023-10-30T14:26:57.161003+0000","last_undegraded":"2023-10-30T14:26:57.161003+0000","last_fullsized":"2023-10-30T14:26:57.161003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1108","last_scrub_stamp":"2023-10-30T14:26:57.160913+0000","last_deep_scrub":"110'1108","last_deep_scrub_stamp":"2023-10-30T14:26:57.160913+0000","last_clean_scrub_stamp":"2023-10-30T14:26:57.160913+0000","objects_scrubbed":44,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:56:50.729954+0000","scrub_duration":0.25105180199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":296,"num_read_kb":429426,"num_write":1159,"num_write_kb":75926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":378159,"num_omap_keys":718,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"110'2194","reported_seq":5040,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:06.417252+0000","last_change":"2023-10-30T14:27:06.417354+0000","last_active":"2023-10-30T14:27:06.417252+0000","last_peered":"2023-10-30T14:27:06.417252+0000","last_clean":"2023-10-30T14:27:06.417252+0000","last_became_active":"2023-10-30T13:23:19.718009+0000","last_became_peered":"2023-10-30T13:23:19.718009+0000","last_unstale":"2023-10-30T14:27:06.417252+0000","last_undegraded":"2023-10-30T14:27:06.417252+0000","last_fullsized":"2023-10-30T14:27:06.417252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2194","last_scrub_stamp":"2023-10-30T14:27:06.417083+0000","last_deep_scrub":"110'2194","last_deep_scrub_stamp":"2023-10-30T14:27:06.417083+0000","last_clean_scrub_stamp":"2023-10-30T14:27:06.417083+0000","objects_scrubbed":30,"log_size":2194,"log_dups_size":0,"ondisk_log_size":2194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:32:26.701207+0000","scrub_duration":0.099883823999999996,"objects_trimmed":0,"snaptrim_duration":0.419978554,"stat_sum":{"num_bytes":5,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":137,"num_read_kb":40109,"num_write":1287,"num_write_kb":25283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"110'2213","reported_seq":5031,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:16.181195+0000","last_change":"2023-10-30T14:27:16.181270+0000","last_active":"2023-10-30T14:27:16.181195+0000","last_peered":"2023-10-30T14:27:16.181195+0000","last_clean":"2023-10-30T14:27:16.181195+0000","last_became_active":"2023-10-30T13:23:19.731690+0000","last_became_peered":"2023-10-30T13:23:19.731690+0000","last_unstale":"2023-10-30T14:27:16.181195+0000","last_undegraded":"2023-10-30T14:27:16.181195+0000","last_fullsized":"2023-10-30T14:27:16.181195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2213","last_scrub_stamp":"2023-10-30T14:27:16.181071+0000","last_deep_scrub":"110'2213","last_deep_scrub_stamp":"2023-10-30T14:27:16.181071+0000","last_clean_scrub_stamp":"2023-10-30T14:27:16.181071+0000","objects_scrubbed":33,"log_size":2213,"log_dups_size":0,"ondisk_log_size":2213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:59:49.845658+0000","scrub_duration":0.12957070300000001,"objects_trimmed":0,"snaptrim_duration":1.073619836,"stat_sum":{"num_bytes":0,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":129,"num_read_kb":12975,"num_write":1318,"num_write_kb":20116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"110'784","reported_seq":1084,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:56.157617+0000","last_change":"2023-10-30T14:26:56.157707+0000","last_active":"2023-10-30T14:26:56.157617+0000","last_peered":"2023-10-30T14:26:56.157617+0000","last_clean":"2023-10-30T14:26:56.157617+0000","last_became_active":"2023-10-30T13:23:18.992749+0000","last_became_peered":"2023-10-30T13:23:18.992749+0000","last_unstale":"2023-10-30T14:26:56.157617+0000","last_undegraded":"2023-10-30T14:26:56.157617+0000","last_fullsized":"2023-10-30T14:26:56.157617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'784","last_scrub_stamp":"2023-10-30T14:26:56.157430+0000","last_deep_scrub":"110'784","last_deep_scrub_stamp":"2023-10-30T14:26:56.157430+0000","last_clean_scrub_stamp":"2023-10-30T14:26:56.157430+0000","objects_scrubbed":50,"log_size":784,"log_dups_size":0,"ondisk_log_size":784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:19.819331+0000","scrub_duration":0.231260298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":304,"num_read_kb":264369,"num_write":877,"num_write_kb":48007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":477325,"num_omap_keys":851,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"110'2679","reported_seq":5374,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:17.160156+0000","last_change":"2023-10-30T14:27:17.160251+0000","last_active":"2023-10-30T14:27:17.160156+0000","last_peered":"2023-10-30T14:27:17.160156+0000","last_clean":"2023-10-30T14:27:17.160156+0000","last_became_active":"2023-10-30T13:23:19.731963+0000","last_became_peered":"2023-10-30T13:23:19.731963+0000","last_unstale":"2023-10-30T14:27:17.160156+0000","last_undegraded":"2023-10-30T14:27:17.160156+0000","last_fullsized":"2023-10-30T14:27:17.160156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2679","last_scrub_stamp":"2023-10-30T14:27:17.160038+0000","last_deep_scrub":"110'2679","last_deep_scrub_stamp":"2023-10-30T14:27:17.160038+0000","last_clean_scrub_stamp":"2023-10-30T14:27:17.160038+0000","objects_scrubbed":37,"log_size":2679,"log_dups_size":0,"ondisk_log_size":2679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:38.223350+0000","scrub_duration":0.126936049,"objects_trimmed":0,"snaptrim_duration":1.021840552,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":171,"num_read_kb":40337,"num_write":1456,"num_write_kb":34441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"110'1224","reported_seq":1535,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:04.243739+0000","last_change":"2023-10-30T14:27:04.243861+0000","last_active":"2023-10-30T14:27:04.243739+0000","last_peered":"2023-10-30T14:27:04.243739+0000","last_clean":"2023-10-30T14:27:04.243739+0000","last_became_active":"2023-10-30T13:23:18.986772+0000","last_became_peered":"2023-10-30T13:23:18.986772+0000","last_unstale":"2023-10-30T14:27:04.243739+0000","last_undegraded":"2023-10-30T14:27:04.243739+0000","last_fullsized":"2023-10-30T14:27:04.243739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1224","last_scrub_stamp":"2023-10-30T14:27:04.243569+0000","last_deep_scrub":"110'1224","last_deep_scrub_stamp":"2023-10-30T14:27:04.243569+0000","last_clean_scrub_stamp":"2023-10-30T14:27:04.243569+0000","objects_scrubbed":45,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:57:40.587906+0000","scrub_duration":0.244476464,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":255,"num_read_kb":502436,"num_write":1236,"num_write_kb":99955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":409468,"num_omap_keys":801,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"110'2069","reported_seq":4948,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:09.913836+0000","last_change":"2023-10-30T14:27:09.913836+0000","last_active":"2023-10-30T14:27:09.913836+0000","last_peered":"2023-10-30T14:27:09.913836+0000","last_clean":"2023-10-30T14:27:09.913836+0000","last_became_active":"2023-10-30T13:23:19.722085+0000","last_became_peered":"2023-10-30T13:23:19.722085+0000","last_unstale":"2023-10-30T14:27:09.913836+0000","last_undegraded":"2023-10-30T14:27:09.913836+0000","last_fullsized":"2023-10-30T14:27:09.913836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2069","last_scrub_stamp":"2023-10-30T14:27:09.913657+0000","last_deep_scrub":"110'2069","last_deep_scrub_stamp":"2023-10-30T14:27:09.913657+0000","last_clean_scrub_stamp":"2023-10-30T14:27:09.913657+0000","objects_scrubbed":41,"log_size":2069,"log_dups_size":0,"ondisk_log_size":2069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:21:51.262361+0000","scrub_duration":0.091522568999999998,"objects_trimmed":0,"snaptrim_duration":1.015373858,"stat_sum":{"num_bytes":0,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":144,"num_read_kb":13189,"num_write":1249,"num_write_kb":15084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"110'817","reported_seq":1082,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:45.554415+0000","last_change":"2023-10-30T14:26:45.554415+0000","last_active":"2023-10-30T14:26:45.554415+0000","last_peered":"2023-10-30T14:26:45.554415+0000","last_clean":"2023-10-30T14:26:45.554415+0000","last_became_active":"2023-10-30T13:23:18.958148+0000","last_became_peered":"2023-10-30T13:23:18.958148+0000","last_unstale":"2023-10-30T14:26:45.554415+0000","last_undegraded":"2023-10-30T14:26:45.554415+0000","last_fullsized":"2023-10-30T14:26:45.554415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'817","last_scrub_stamp":"2023-10-30T14:26:45.554286+0000","last_deep_scrub":"110'817","last_deep_scrub_stamp":"2023-10-30T14:26:45.554286+0000","last_clean_scrub_stamp":"2023-10-30T14:26:45.554286+0000","objects_scrubbed":38,"log_size":817,"log_dups_size":0,"ondisk_log_size":817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:36.415824+0000","scrub_duration":0.061981296999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":159,"num_read_kb":369044,"num_write":787,"num_write_kb":79251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":283692,"num_omap_keys":523,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"110'2229","reported_seq":5189,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:27:00.797301+0000","last_change":"2023-10-30T14:27:00.797301+0000","last_active":"2023-10-30T14:27:00.797301+0000","last_peered":"2023-10-30T14:27:00.797301+0000","last_clean":"2023-10-30T14:27:00.797301+0000","last_became_active":"2023-10-30T13:23:19.699505+0000","last_became_peered":"2023-10-30T13:23:19.699505+0000","last_unstale":"2023-10-30T14:27:00.797301+0000","last_undegraded":"2023-10-30T14:27:00.797301+0000","last_fullsized":"2023-10-30T14:27:00.797301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2229","last_scrub_stamp":"2023-10-30T14:27:00.797179+0000","last_deep_scrub":"110'2229","last_deep_scrub_stamp":"2023-10-30T14:27:00.797179+0000","last_clean_scrub_stamp":"2023-10-30T14:27:00.797179+0000","objects_scrubbed":37,"log_size":2229,"log_dups_size":0,"ondisk_log_size":2229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:23.560124+0000","scrub_duration":0.084074704,"objects_trimmed":0,"snaptrim_duration":1.043790548,"stat_sum":{"num_bytes":0,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":262,"num_read_kb":42307,"num_write":1336,"num_write_kb":30630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"110'1065","reported_seq":1396,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-30T14:26:42.937107+0000","last_change":"2023-10-30T14:26:42.937107+0000","last_active":"2023-10-30T14:26:42.937107+0000","last_peered":"2023-10-30T14:26:42.937107+0000","last_clean":"2023-10-30T14:26:42.937107+0000","last_became_active":"2023-10-30T13:23:18.980455+0000","last_became_peered":"2023-10-30T13:23:18.980455+0000","last_unstale":"2023-10-30T14:26:42.937107+0000","last_undegraded":"2023-10-30T14:26:42.937107+0000","last_fullsized":"2023-10-30T14:26:42.937107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1065","last_scrub_stamp":"2023-10-30T14:26:42.936998+0000","last_deep_scrub":"110'1065","last_deep_scrub_stamp":"2023-10-30T14:26:42.936998+0000","last_clean_scrub_stamp":"2023-10-30T14:26:42.936998+0000","objects_scrubbed":55,"log_size":1065,"log_dups_size":0,"ondisk_log_size":1065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:27.046930+0000","scrub_duration":0.131982027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777798,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":313,"num_read_kb":340671,"num_write":1085,"num_write_kb":63851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":501138,"num_omap_keys":760,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":11,"num_objects":2377,"num_object_clones":2,"num_object_copies":4754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":9389,"num_read_kb":1724409,"num_write":83589,"num_write_kb":1459751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140904,"ondisk_log_size":140904,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1285101348,"num_objects":1569,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1569,"num_whiteouts":0,"num_read":10652,"num_read_kb":12762809,"num_write":35655,"num_write_kb":2343865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1191,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13139950,"num_omap_keys":23453,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2570362880,"data_stored":2570202696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33559,"ondisk_log_size":33559,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575631,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578504,"kb_used_data":389120,"kb_used_omap":1,"kb_used_meta":189374,"kb_avail":93793336,"statfs":{"total":96636764160,"available":96044376064,"internally_reserved":0,"allocated":398458880,"data_stored":397803489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":193919435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.75600000000000001,"15min":0.79800000000000004},"min":{"1min":0.50700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.236,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.85799999999999998,"15min":0.90900000000000003},"min":{"1min":0.627,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.306,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.97899999999999998}]},{"osd":1,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.81100000000000005,"15min":0.86899999999999999},"min":{"1min":0.52700000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2130000000000001,"5min":2.452,"15min":2.452},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.82799999999999996,"15min":0.84999999999999998},"min":{"1min":0.61599999999999999,"5min":0.495,"15min":0.47399999999999998},"max":{"1min":1.29,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.78500000000000003,"15min":0.82599999999999996},"min":{"1min":0.67300000000000004,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.175,"5min":1.175,"15min":1.4350000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.171,"5min":0.89700000000000002,"15min":0.94899999999999995},"min":{"1min":0.79600000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.403,"5min":1.403,"15min":1.5569999999999999},"last":0.67700000000000005}]},{"osd":3,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.95799999999999996,"15min":0.96099999999999997},"min":{"1min":0.83199999999999996,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.335,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.127,"5min":1.04,"15min":1.012},"min":{"1min":0.86199999999999999,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.4390000000000001,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":0.755}]},{"osd":4,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.91000000000000003,"15min":0.89600000000000002},"min":{"1min":0.56899999999999995,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.202,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.085,"15min":1.046},"min":{"1min":0.746,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.3400000000000001,"5min":3.6899999999999999,"15min":3.6899999999999999},"last":1.149}]},{"osd":5,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.1319999999999999,"15min":1.1000000000000001},"min":{"1min":0.755,"5min":0.375,"15min":0.375},"max":{"1min":1.5049999999999999,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":1.004},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.127,"15min":1.1479999999999999},"min":{"1min":0.68999999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.206,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":1.1000000000000001}]},{"osd":6,"last update":"Mon Oct 30 14:27:38 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.1120000000000001,"15min":1.107},"min":{"1min":0.76600000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.256,"5min":6.3070000000000004,"15min":6.3070000000000004},"last":1.026},{"interface":"front","average":{"1min":1.206,"5min":1.135,"15min":1.1799999999999999},"min":{"1min":0.69899999999999995,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.9089999999999998,"5min":2.9089999999999998,"15min":2.9089999999999998},"last":0.70999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575634,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505392,"kb_used_data":339688,"kb_used_omap":1,"kb_used_meta":165694,"kb_avail":93866448,"statfs":{"total":96636764160,"available":96119242752,"internally_reserved":0,"allocated":347840512,"data_stored":347188935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169671111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.74099999999999999,"15min":0.79700000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.375},"max":{"1min":1.105,"5min":1.306,"15min":1.363},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.85499999999999998,"15min":0.94099999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.0669999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.72899999999999998,"15min":0.77700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.452,"5min":3.452,"15min":3.452},"last":1.103},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.748,"15min":0.77800000000000002},"min":{"1min":0.48399999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.163,"5min":1.706,"15min":1.706},"last":0.83799999999999997}]},{"osd":2,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.80500000000000005,"15min":0.83599999999999997},"min":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.571},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.99299999999999999,"15min":0.94699999999999995},"min":{"1min":0.52300000000000002,"5min":0.502,"15min":0.502},"max":{"1min":3.6459999999999999,"5min":5.4480000000000004,"15min":5.4480000000000004},"last":0.79300000000000004}]},{"osd":3,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.04,"15min":1.0369999999999999},"min":{"1min":0.752,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":3.5169999999999999,"5min":3.5169999999999999,"15min":3.5169999999999999},"last":1.236},{"interface":"front","average":{"1min":1.228,"5min":1.1240000000000001,"15min":1.1220000000000001},"min":{"1min":0.77700000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":3.5830000000000002,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":1.196}]},{"osd":4,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.57199999999999995,"5min":0.434,"15min":0.434},"max":{"1min":1.2410000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.98899999999999999,"15min":1.026},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.452,"5min":1.595,"15min":1.595},"last":0.94199999999999995}]},{"osd":5,"last update":"Mon Oct 30 14:27:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.1599999999999999,"15min":1.0429999999999999},"min":{"1min":0.59899999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.9590000000000001,"5min":3.0219999999999998,"15min":3.0219999999999998},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":1.0069999999999999,"15min":1.0429999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.835,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":0.67600000000000005}]},{"osd":7,"last update":"Mon Oct 30 14:27:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2410000000000001,"15min":1.1990000000000001},"min":{"1min":0.65300000000000002,"5min":0.58699999999999997,"15min":0.52600000000000002},"max":{"1min":2.0379999999999998,"5min":2.3940000000000001,"15min":2.3940000000000001},"last":1.147},{"interface":"front","average":{"1min":1.034,"5min":1.1259999999999999,"15min":1.1279999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.9350000000000001,"5min":2.806,"15min":2.806},"last":1.034}]}]},{"osd":5,"up_from":13,"seq":55834575631,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570756,"kb_used_data":368956,"kb_used_omap":1,"kb_used_meta":201790,"kb_avail":93801084,"statfs":{"total":96636764160,"available":96052310016,"internally_reserved":0,"allocated":377810944,"data_stored":377158158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206633414},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.84599999999999997},"min":{"1min":0.439,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1120000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.0900000000000001},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.93700000000000006,"15min":0.95699999999999996},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.373,"5min":1.617,"15min":1.9219999999999999},"last":1.2669999999999999}]},{"osd":1,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.81799999999999995,"15min":0.83899999999999997},"min":{"1min":0.48899999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.1739999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.77100000000000002,"15min":0.83999999999999997},"min":{"1min":0.53700000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.774,"5min":4.774,"15min":4.774},"last":0.98099999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85199999999999998,"15min":0.88100000000000001},"min":{"1min":0.58599999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.111,"5min":1.2589999999999999,"15min":1.331},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.84699999999999998},"min":{"1min":0.501,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.1879999999999999,"5min":1.331,"15min":1.796},"last":1.198}]},{"osd":3,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.96999999999999997,"15min":1.0009999999999999},"min":{"1min":0.42499999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3100000000000001,"5min":1.536,"15min":1.536},"last":1.323},{"interface":"front","average":{"1min":1.036,"5min":0.92600000000000005,"15min":0.99099999999999999},"min":{"1min":0.51800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":2.0030000000000001},"last":1.04}]},{"osd":4,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.88800000000000001,"15min":0.877},"min":{"1min":0.374,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.087,"5min":2.8170000000000002,"15min":2.8170000000000002},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.0609999999999999,"15min":1.036},"min":{"1min":0.45100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.4690000000000001,"5min":2.2400000000000002,"15min":2.262},"last":0.81399999999999995}]},{"osd":6,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.071,"15min":1.1220000000000001},"min":{"1min":0.40300000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3380000000000001,"5min":2.3090000000000002,"15min":2.7949999999999999},"last":1.143},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.038,"15min":1.054},"min":{"1min":0.38800000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.4350000000000001,"5min":2.161,"15min":2.1629999999999998},"last":1.109}]},{"osd":7,"last update":"Mon Oct 30 14:27:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.026,"15min":1.079},"min":{"1min":0.33800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":1.391,"5min":2.5289999999999999,"15min":2.5289999999999999},"last":1.069},{"interface":"front","average":{"1min":1.163,"5min":1.133,"15min":1.147},"min":{"1min":0.436,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.587,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.24}]}]},{"osd":4,"up_from":13,"seq":55834575631,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469348,"kb_used_data":267484,"kb_used_omap":1,"kb_used_meta":201854,"kb_avail":93902492,"statfs":{"total":96636764160,"available":96156151808,"internally_reserved":0,"allocated":273903616,"data_stored":273238103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206698950},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.73599999999999999,"15min":0.80500000000000005},"min":{"1min":0.439,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.3029999999999999,"5min":1.3500000000000001,"15min":1.446},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.69199999999999995,"15min":0.81200000000000006},"min":{"1min":0.56599999999999995,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0940000000000001,"5min":1.659,"15min":1.659},"last":0.78100000000000003}]},{"osd":1,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.80100000000000005,"15min":0.79400000000000004},"min":{"1min":0.54000000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.86899999999999999,"15min":0.85399999999999998},"min":{"1min":0.49199999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.84799999999999998}]},{"osd":2,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.747,"15min":0.85299999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.296,"5min":1.607,"15min":1.607},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.83199999999999996,"15min":0.90900000000000003},"min":{"1min":0.53400000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.196,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0029999999999999,"15min":0.97899999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.107,"15min":1.083},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":1.224}]},{"osd":5,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.99299999999999999,"15min":1.042},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.391,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.198},{"interface":"front","average":{"1min":1.004,"5min":0.94399999999999995,"15min":0.96299999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.97699999999999998}]},{"osd":6,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.97299999999999998,"15min":0.96499999999999997},"min":{"1min":0.42399999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":1.077},{"interface":"front","average":{"1min":1.071,"5min":1.042,"15min":1.0700000000000001},"min":{"1min":0.51800000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.534,"5min":1.677,"15min":1.728},"last":1.139}]},{"osd":7,"last update":"Mon Oct 30 14:27:25 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.133,"15min":1.1040000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.1899999999999999,"15min":1.179},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.024}]}]},{"osd":3,"up_from":13,"seq":55834575629,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597696,"kb_used_data":382008,"kb_used_omap":1,"kb_used_meta":215678,"kb_avail":93774144,"statfs":{"total":96636764160,"available":96024723456,"internally_reserved":0,"allocated":391176192,"data_stored":390510799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":220854728},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.374,"15min":0.40300000000000002},"min":{"1min":0.24199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.72099999999999997,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.46700000000000003,"15min":0.48999999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.56399999999999995,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.39800000000000002}]},{"osd":1,"last update":"Mon Oct 30 14:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.47499999999999998,"15min":0.45600000000000002},"min":{"1min":0.379,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51100000000000001,"15min":0.54400000000000004},"min":{"1min":0.32800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54400000000000004,"15min":0.59199999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.66300000000000003,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.503},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.433,"5min":0.29999999999999999,"15min":0.29299999999999998},"max":{"1min":0.69699999999999995,"5min":1.73,"15min":1.73},"last":0.55100000000000005}]},{"osd":4,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.76600000000000001,"15min":0.79800000000000004},"min":{"1min":0.57699999999999996,"5min":0.379,"15min":0.379},"max":{"1min":0.98599999999999999,"5min":1.6519999999999999,"15min":1.6519999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.78100000000000003,"15min":0.81200000000000006},"min":{"1min":0.52500000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.89200000000000002,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.84399999999999997}]},{"osd":5,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.76100000000000001,"15min":0.81799999999999995},"min":{"1min":0.55700000000000005,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0209999999999999,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.78300000000000003,"15min":0.84499999999999997},"min":{"1min":0.58399999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.032,"5min":1.486,"15min":1.486},"last":0.65100000000000002}]},{"osd":6,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83999999999999997,"15min":0.88},"min":{"1min":0.625,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1830000000000001,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.90200000000000002,"15min":0.93600000000000005},"min":{"1min":0.72399999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.224,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.86699999999999999}]},{"osd":7,"last update":"Mon Oct 30 14:27:01 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.80800000000000005,"15min":0.85599999999999998},"min":{"1min":0.65500000000000003,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.052,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.77900000000000003,"15min":0.81599999999999995},"min":{"1min":0.60899999999999999,"5min":0.501,"15min":0.501},"max":{"1min":0.93400000000000005,"5min":1.593,"15min":1.593},"last":0.69499999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303572,"kb_used_data":200268,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":94068268,"statfs":{"total":96636764160,"available":96325906432,"internally_reserved":0,"allocated":205074432,"data_stored":204423364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105773510},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42299999999999999,"15min":0.45600000000000002},"min":{"1min":0.26100000000000001,"5min":0.246,"15min":0.23899999999999999},"max":{"1min":0.72599999999999998,"5min":0.747,"15min":0.89300000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.499,"15min":0.51300000000000001},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":0.63900000000000001,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":0.502}]},{"osd":1,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.41699999999999998,"15min":0.44600000000000001},"min":{"1min":0.307,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70099999999999996,"5min":1.956,"15min":1.956},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.35099999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.153,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.53000000000000003}]},{"osd":3,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55700000000000005,"15min":0.57399999999999995},"min":{"1min":0.40699999999999997,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.2010000000000001,"5min":1.988,"15min":1.988},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.502,"15min":0.51800000000000002},"min":{"1min":0.32600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69599999999999995,"5min":2.335,"15min":2.335},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.80800000000000005,"15min":0.82599999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.085,"5min":2.302,"15min":2.302},"last":0.746},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84299999999999997,"15min":0.83699999999999997},"min":{"1min":0.66900000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0169999999999999,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.94199999999999995}]},{"osd":5,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.77000000000000002,"15min":0.80400000000000005},"min":{"1min":0.59099999999999997,"5min":0.501,"15min":0.501},"max":{"1min":1.028,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.82999999999999996,"15min":0.84199999999999997},"min":{"1min":0.68400000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0940000000000001,"5min":1.875,"15min":1.875},"last":0.82599999999999996}]},{"osd":6,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.82299999999999995,"15min":0.82899999999999996},"min":{"1min":0.63100000000000001,"5min":0.502,"15min":0.48099999999999998},"max":{"1min":1.123,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.82899999999999996,"15min":0.86299999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.2130000000000001,"5min":1.921,"15min":1.921},"last":0.70699999999999996}]},{"osd":7,"last update":"Mon Oct 30 14:27:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85899999999999999,"15min":0.86099999999999999},"min":{"1min":0.66300000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1100000000000001,"5min":4.5739999999999998,"15min":4.5739999999999998},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.88500000000000001,"15min":0.91400000000000003},"min":{"1min":0.64700000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.202,"5min":2.375,"15min":2.375},"last":0.89000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575633,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494860,"kb_used_data":329092,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93876980,"statfs":{"total":96636764160,"available":96130027520,"internally_reserved":0,"allocated":336990208,"data_stored":336334619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.45800000000000002},"min":{"1min":0.29999999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.72599999999999998,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.49099999999999999,"15min":0.51400000000000001},"min":{"1min":0.36799999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86799999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":0.46400000000000002}]},{"osd":2,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0469999999999999,"5min":1.954,"15min":1.954},"last":0.437},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.50600000000000001,"15min":0.49299999999999999},"min":{"1min":0.28100000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":5.7039999999999997,"5min":5.7039999999999997,"15min":5.7039999999999997},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55100000000000005,"15min":0.55000000000000004},"min":{"1min":0.44600000000000001,"5min":0.35499999999999998,"15min":0.35099999999999998},"max":{"1min":1.3220000000000001,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3600000000000001,"5min":2.625,"15min":2.625},"last":0.54700000000000004}]},{"osd":4,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.79200000000000004,"15min":0.82999999999999996},"min":{"1min":0.66000000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.92900000000000005,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.82199999999999995,"15min":0.872},"min":{"1min":0.67300000000000004,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0309999999999999,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.74099999999999999}]},{"osd":5,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.78800000000000003,"15min":0.86299999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.004,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.82099999999999995,"15min":0.88800000000000001},"min":{"1min":0.52200000000000002,"5min":0.499,"15min":0.499},"max":{"1min":1.167,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.70999999999999996}]},{"osd":6,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.89900000000000002,"15min":0.84499999999999997},"min":{"1min":0.73999999999999999,"5min":0.624,"15min":0.56499999999999995},"max":{"1min":1.01,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.81000000000000005,"15min":0.88600000000000001},"min":{"1min":0.56399999999999995,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.99299999999999999,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.98099999999999998}]},{"osd":7,"last update":"Mon Oct 30 14:27:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.623,"5min":0.59599999999999997,"15min":0.56699999999999995},"max":{"1min":1.085,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.875,"5min":0.83999999999999997,"15min":0.84999999999999998},"min":{"1min":0.66900000000000004,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0660000000000001,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.996}]}]},{"osd":0,"up_from":13,"seq":55834575629,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":425348,"kb_used_data":245756,"kb_used_omap":1,"kb_used_meta":179582,"kb_avail":93946492,"statfs":{"total":96636764160,"available":96201207808,"internally_reserved":0,"allocated":251654144,"data_stored":251001762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":183892422},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.377,"15min":0.40100000000000002},"min":{"1min":0.26400000000000001,"5min":0.247,"15min":0.23499999999999999},"max":{"1min":0.61699999999999999,"5min":0.66100000000000003,"15min":0.94999999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48499999999999999,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65800000000000003,"5min":0.82299999999999995,"15min":1.0229999999999999},"last":0.622}]},{"osd":2,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.45400000000000001,"15min":0.46200000000000002},"min":{"1min":0.33400000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47699999999999998,"15min":0.496},"min":{"1min":0.38100000000000001,"5min":0.30499999999999999,"15min":0.29099999999999998},"max":{"1min":0.83599999999999997,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.54100000000000004}]},{"osd":3,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.91900000000000004,"5min":1.236,"15min":1.236},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51800000000000002,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":0.65400000000000003}]},{"osd":4,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78200000000000003,"15min":0.79700000000000004},"min":{"1min":0.58899999999999997,"5min":0.503,"15min":0.503},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.747,"15min":0.80000000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91200000000000003,"5min":1.012,"15min":1.2210000000000001},"last":1.151}]},{"osd":5,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.78700000000000003,"15min":0.80900000000000005},"min":{"1min":0.61299999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.80700000000000005,"15min":0.84499999999999997},"min":{"1min":0.626,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.92000000000000004}]},{"osd":6,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.80800000000000005,"15min":0.83799999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.99399999999999999,"5min":1.222,"15min":1.486},"last":0.877},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.80600000000000005,"15min":0.82899999999999996},"min":{"1min":0.66900000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.024}]},{"osd":7,"last update":"Mon Oct 30 14:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.82399999999999995,"15min":0.84699999999999998},"min":{"1min":0.70999999999999996,"5min":0.625,"15min":0.58599999999999997},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81799999999999995,"15min":0.86199999999999999},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0680000000000001,"5min":1.1559999999999999,"15min":1.4419999999999999},"last":1.097}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249647104,"data_stored":249627559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335572992,"data_stored":335549765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203661312,"data_stored":203639529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389746688,"data_stored":389725939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":272470016,"data_stored":272453238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376397824,"data_stored":376374323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":346419200,"data_stored":346404076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":396447744,"data_stored":396428267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":11,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T14:28:06.422 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-30T14:28:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac2d70f640 1 Processor -- start 2023-10-30T14:28:06.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac2d70f640 1 -- start start 2023-10-30T14:28:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac2d70f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac281508c0 0x7fac280c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac2d70f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac280722a0 0x7fac280c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac2d70f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fac280bfec0 0x7fac280c0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac2d70f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac280c85b0 con 0x7fac280722a0 2023-10-30T14:28:06.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac2d70f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac280c0a30 con 0x7fac280bfec0 2023-10-30T14:28:06.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac2d70f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2806f650 con 0x7fac281508c0 2023-10-30T14:28:06.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac1ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac280722a0 0x7fac280c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac1ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac280722a0 0x7fac280c6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48980/0 (socket says 172.21.15.46:48980) 2023-10-30T14:28:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac1ffff640 1 -- 172.21.15.46:0/3175428829 learned_addr learned my addr 172.21.15.46:0/3175428829 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:28:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac26ffd640 1 --2- 172.21.15.46:0/3175428829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac281508c0 0x7fac280c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac277fe640 1 --2- 172.21.15.46:0/3175428829 >> v2:172.21.15.154:3300/0 conn(0x7fac280bfec0 0x7fac280c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac1ffff640 1 -- 172.21.15.46:0/3175428829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac281508c0 msgr2=0x7fac280c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac1ffff640 1 --2- 172.21.15.46:0/3175428829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac281508c0 0x7fac280c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac1ffff640 1 -- 172.21.15.46:0/3175428829 >> v2:172.21.15.154:3300/0 conn(0x7fac280bfec0 msgr2=0x7fac280c0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac1ffff640 1 --2- 172.21.15.46:0/3175428829 >> v2:172.21.15.154:3300/0 conn(0x7fac280bfec0 0x7fac280c0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.593+0000 7fac1ffff640 1 -- 172.21.15.46:0/3175428829 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2806f930 con 0x7fac280722a0 2023-10-30T14:28:06.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac1ffff640 1 --2- 172.21.15.46:0/3175428829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac280722a0 0x7fac280c6e60 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fac2806f1b0 tx=0x7fac1001e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea326148fa5c3c77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:06.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac267fc640 1 -- 172.21.15.46:0/3175428829 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac100ab400 con 0x7fac280722a0 2023-10-30T14:28:06.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac267fc640 1 -- 172.21.15.46:0/3175428829 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac100ab560 con 0x7fac280722a0 2023-10-30T14:28:06.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac267fc640 1 -- 172.21.15.46:0/3175428829 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac100ab8f0 con 0x7fac280722a0 2023-10-30T14:28:06.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac2d70f640 1 -- 172.21.15.46:0/3175428829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac280722a0 msgr2=0x7fac280c6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac2d70f640 1 --2- 172.21.15.46:0/3175428829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac280722a0 0x7fac280c6e60 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fac2806f1b0 tx=0x7fac1001e680 comp rx=0 tx=0).stop 2023-10-30T14:28:06.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac2d70f640 1 -- 172.21.15.46:0/3175428829 shutdown_connections 2023-10-30T14:28:06.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac2d70f640 1 --2- 172.21.15.46:0/3175428829 >> v2:172.21.15.154:3300/0 conn(0x7fac280bfec0 0x7fac280c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac2d70f640 1 --2- 172.21.15.46:0/3175428829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac280722a0 0x7fac280c6e60 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac2d70f640 1 --2- 172.21.15.46:0/3175428829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac281508c0 0x7fac280c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac2d70f640 1 -- 172.21.15.46:0/3175428829 >> 172.21.15.46:0/3175428829 conn(0x7fac280c7910 msgr2=0x7fac280c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:06.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac2d70f640 1 -- 172.21.15.46:0/3175428829 shutdown_connections 2023-10-30T14:28:06.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.597+0000 7fac2d70f640 1 -- 172.21.15.46:0/3175428829 wait complete. 2023-10-30T14:28:06.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac2d70f640 1 Processor -- start 2023-10-30T14:28:06.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac2d70f640 1 -- start start 2023-10-30T14:28:06.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac2d70f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac280722a0 0x7fac28154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac2d70f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac280bfec0 0x7fac28140720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac2d70f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac281508c0 0x7fac28155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac2d70f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac280ca3c0 con 0x7fac281508c0 2023-10-30T14:28:06.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac2d70f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac280c9f30 con 0x7fac280bfec0 2023-10-30T14:28:06.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac2d70f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2814c050 con 0x7fac280722a0 2023-10-30T14:28:06.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac281508c0 0x7fac28155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac281508c0 0x7fac28155030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48992/0 (socket says 172.21.15.46:48992) 2023-10-30T14:28:06.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac277fe640 1 -- 172.21.15.46:0/760760942 learned_addr learned my addr 172.21.15.46:0/760760942 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:28:06.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac26ffd640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac280722a0 0x7fac28154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac1ffff640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac280bfec0 0x7fac28140720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac277fe640 1 -- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac280722a0 msgr2=0x7fac28154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac277fe640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac280722a0 0x7fac28154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac277fe640 1 -- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac280bfec0 msgr2=0x7fac28140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac277fe640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac280bfec0 0x7fac28140720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac277fe640 1 -- 172.21.15.46:0/760760942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac100094d0 con 0x7fac281508c0 2023-10-30T14:28:06.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac26ffd640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac280722a0 0x7fac28154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:06.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac277fe640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac281508c0 0x7fac28155030 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7fac20007e90 tx=0x7fac2000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:06.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac1f7fe640 1 -- 172.21.15.46:0/760760942 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac20014420 con 0x7fac281508c0 2023-10-30T14:28:06.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac1f7fe640 1 -- 172.21.15.46:0/760760942 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac200145a0 con 0x7fac281508c0 2023-10-30T14:28:06.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac1f7fe640 1 -- 172.21.15.46:0/760760942 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac20014950 con 0x7fac281508c0 2023-10-30T14:28:06.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac2d70f640 1 -- 172.21.15.46:0/760760942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2814c330 con 0x7fac281508c0 2023-10-30T14:28:06.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac2d70f640 1 -- 172.21.15.46:0/760760942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac280c9bc0 con 0x7fac281508c0 2023-10-30T14:28:06.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.601+0000 7fac2d70f640 1 -- 172.21.15.46:0/760760942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac280ca0c0 con 0x7fac281508c0 2023-10-30T14:28:06.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.605+0000 7fac1f7fe640 1 -- 172.21.15.46:0/760760942 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fac20014ab0 con 0x7fac281508c0 2023-10-30T14:28:06.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.605+0000 7fac1f7fe640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fac080664a0 0x7fac08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.605+0000 7fac1f7fe640 1 -- 172.21.15.46:0/760760942 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fac200853e0 con 0x7fac281508c0 2023-10-30T14:28:06.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.605+0000 7fac26ffd640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fac080664a0 0x7fac08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.605+0000 7fac26ffd640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fac080664a0 0x7fac08068960 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fac18006fd0 tx=0x7fac18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:06.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.609+0000 7fac1f7fe640 1 -- 172.21.15.46:0/760760942 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fac20051b80 con 0x7fac281508c0 2023-10-30T14:28:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.765+0000 7fac2d70f640 1 -- 172.21.15.46:0/760760942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fac2804e170 con 0x7fac281508c0 2023-10-30T14:28:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.769+0000 7fac1f7fe640 1 -- 172.21.15.46:0/760760942 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) v1 ==== 123+0+0 (secure 0 0 0) 0x7fac200551a0 con 0x7fac281508c0 2023-10-30T14:28:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.769+0000 7fac1f7fe640 1 -- 172.21.15.46:0/760760942 <== mon.0 v2:172.21.15.46:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fac20051b80 con 0x7fac281508c0 2023-10-30T14:28:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.773+0000 7fac2d70f640 1 -- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fac080664a0 msgr2=0x7fac08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.773+0000 7fac2d70f640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fac080664a0 0x7fac08068960 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fac18006fd0 tx=0x7fac18008040 comp rx=0 tx=0).stop 2023-10-30T14:28:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.773+0000 7fac2d70f640 1 -- 172.21.15.46:0/760760942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac281508c0 msgr2=0x7fac28155030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.773+0000 7fac2d70f640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac281508c0 0x7fac28155030 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7fac20007e90 tx=0x7fac2000e4b0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.773+0000 7fac2d70f640 1 -- 172.21.15.46:0/760760942 shutdown_connections 2023-10-30T14:28:06.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.773+0000 7fac2d70f640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fac080664a0 0x7fac08068960 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.773+0000 7fac2d70f640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac281508c0 0x7fac28155030 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.773+0000 7fac2d70f640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac280bfec0 0x7fac28140720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.773+0000 7fac2d70f640 1 --2- 172.21.15.46:0/760760942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fac280722a0 0x7fac28154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.773+0000 7fac2d70f640 1 -- 172.21.15.46:0/760760942 >> 172.21.15.46:0/760760942 conn(0x7fac280c7910 msgr2=0x7fac2814faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:06.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.773+0000 7fac2d70f640 1 -- 172.21.15.46:0/760760942 shutdown_connections 2023-10-30T14:28:06.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.773+0000 7fac2d70f640 1 -- 172.21.15.46:0/760760942 wait complete. 2023-10-30T14:28:06.796 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-30T14:28:06.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:06.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7569ed6640 1 Processor -- start 2023-10-30T14:28:06.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7569ed6640 1 -- start start 2023-10-30T14:28:06.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7569ed6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f75641538a0 0x7f756415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7569ed6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f75640ca0e0 0x7f756415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7569ed6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75640cb800 con 0x7f75640cf160 2023-10-30T14:28:06.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7569ed6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f756415dc30 con 0x7f75640ca0e0 2023-10-30T14:28:06.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7569ed6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75640cb330 con 0x7f75641538a0 2023-10-30T14:28:06.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7562ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f75640ca0e0 0x7f756415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7563fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f75640cf160 0x7f75640cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41990/0 (socket says 172.21.15.46:41990) 2023-10-30T14:28:06.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7563fff640 1 -- 172.21.15.46:0/3240079009 learned_addr learned my addr 172.21.15.46:0/3240079009 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:28:06.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f75637fe640 1 --2- 172.21.15.46:0/3240079009 >> v2:172.21.15.154:3300/0 conn(0x7f75641538a0 0x7f756415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7562ffd640 1 -- 172.21.15.46:0/3240079009 >> v2:172.21.15.154:3300/0 conn(0x7f75641538a0 msgr2=0x7f756415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:06.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7562ffd640 1 --2- 172.21.15.46:0/3240079009 >> v2:172.21.15.154:3300/0 conn(0x7f75641538a0 0x7f756415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7562ffd640 1 -- 172.21.15.46:0/3240079009 >> v1:172.21.15.46:6789/0 conn(0x7f75640cf160 legacy=0x7f75640cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f7562ffd640 1 -- 172.21.15.46:0/3240079009 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75640cad70 con 0x7f75640ca0e0 2023-10-30T14:28:06.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f75627fc640 1 -- 172.21.15.46:0/3240079009 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057460146 0 0) 0x7f75640cb800 con 0x7f75640cf160 2023-10-30T14:28:06.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.937+0000 7f75637fe640 1 --2- 172.21.15.46:0/3240079009 >> v2:172.21.15.154:3300/0 conn(0x7f75641538a0 0x7f756415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:06.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f7562ffd640 1 --2- 172.21.15.46:0/3240079009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75640ca0e0 0x7f756415a090 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f75640cfd10 tx=0x7f755801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3be380cabad4e43 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:06.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f75627fc640 1 -- 172.21.15.46:0/3240079009 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75580a8950 con 0x7f75640ca0e0 2023-10-30T14:28:06.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f75627fc640 1 -- 172.21.15.46:0/3240079009 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f75580a8ab0 con 0x7f75640ca0e0 2023-10-30T14:28:06.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f75627fc640 1 -- 172.21.15.46:0/3240079009 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75580a8e60 con 0x7f75640ca0e0 2023-10-30T14:28:06.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f7569ed6640 1 -- 172.21.15.46:0/3240079009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75640ca0e0 msgr2=0x7f756415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f7569ed6640 1 --2- 172.21.15.46:0/3240079009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75640ca0e0 0x7f756415a090 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f75640cfd10 tx=0x7f755801ec50 comp rx=0 tx=0).stop 2023-10-30T14:28:06.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f7569ed6640 1 -- 172.21.15.46:0/3240079009 shutdown_connections 2023-10-30T14:28:06.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f7569ed6640 1 --2- 172.21.15.46:0/3240079009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75640ca0e0 0x7f756415a090 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f7569ed6640 1 --2- 172.21.15.46:0/3240079009 >> v2:172.21.15.154:3300/0 conn(0x7f75641538a0 0x7f756415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f7569ed6640 1 -- 172.21.15.46:0/3240079009 >> 172.21.15.46:0/3240079009 conn(0x7f756406bfe0 msgr2=0x7f756406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:06.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f7569ed6640 1 -- 172.21.15.46:0/3240079009 shutdown_connections 2023-10-30T14:28:06.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f7569ed6640 1 -- 172.21.15.46:0/3240079009 wait complete. 2023-10-30T14:28:06.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.941+0000 7f7569ed6640 1 Processor -- start 2023-10-30T14:28:06.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7569ed6640 1 -- start start 2023-10-30T14:28:06.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7569ed6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75640ca0e0 0x7f75640ce430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7569ed6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75640cf160 0x7f7564148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7569ed6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75641538a0 0x7f75641efb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7569ed6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75640b51c0 con 0x7f75641538a0 2023-10-30T14:28:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7569ed6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75640b4ee0 con 0x7f75640cf160 2023-10-30T14:28:06.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7569ed6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756415a9c0 con 0x7f75640ca0e0 2023-10-30T14:28:06.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7563fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75641538a0 0x7f75641efb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7563fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75641538a0 0x7f75641efb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49004/0 (socket says 172.21.15.46:49004) 2023-10-30T14:28:06.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7563fff640 1 -- 172.21.15.46:0/523066274 learned_addr learned my addr 172.21.15.46:0/523066274 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:28:06.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7562ffd640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75640cf160 0x7f7564148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f75637fe640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75640ca0e0 0x7f75640ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7563fff640 1 -- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75640ca0e0 msgr2=0x7f75640ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7563fff640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75640ca0e0 0x7f75640ce430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7563fff640 1 -- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75640cf160 msgr2=0x7f7564148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:06.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7563fff640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75640cf160 0x7f7564148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:06.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7563fff640 1 -- 172.21.15.46:0/523066274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7558009420 con 0x7f75641538a0 2023-10-30T14:28:06.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7562ffd640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75640cf160 0x7f7564148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:06.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7563fff640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75641538a0 0x7f75641efb40 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f755c001570 tx=0x7f755c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:06.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f754bfff640 1 -- 172.21.15.46:0/523066274 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c00cea0 con 0x7f75641538a0 2023-10-30T14:28:06.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f754bfff640 1 -- 172.21.15.46:0/523066274 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f755c00a5e0 con 0x7f75641538a0 2023-10-30T14:28:06.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7569ed6640 1 -- 172.21.15.46:0/523066274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f756415aca0 con 0x7f75641538a0 2023-10-30T14:28:06.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f754bfff640 1 -- 172.21.15.46:0/523066274 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c00a9b0 con 0x7f75641538a0 2023-10-30T14:28:06.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7569ed6640 1 -- 172.21.15.46:0/523066274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756415b260 con 0x7f75641538a0 2023-10-30T14:28:06.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.945+0000 7f7569ed6640 1 -- 172.21.15.46:0/523066274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75640b2a50 con 0x7f75641538a0 2023-10-30T14:28:06.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.949+0000 7f754bfff640 1 -- 172.21.15.46:0/523066274 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f755c00ab10 con 0x7f75641538a0 2023-10-30T14:28:06.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.949+0000 7f754bfff640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75500664a0 0x7f7550068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:06.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.949+0000 7f754bfff640 1 -- 172.21.15.46:0/523066274 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f755c0840b0 con 0x7f75641538a0 2023-10-30T14:28:06.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.949+0000 7f75637fe640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75500664a0 0x7f7550068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:06.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.949+0000 7f75637fe640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75500664a0 0x7f7550068960 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f755400ba10 tx=0x7f7554008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:06.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:06.953+0000 7f754bfff640 1 -- 172.21.15.46:0/523066274 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f755c050850 con 0x7f75641538a0 2023-10-30T14:28:07.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.145+0000 7f7569ed6640 1 -- 172.21.15.46:0/523066274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75641ef860 con 0x7f75641538a0 2023-10-30T14:28:07.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.145+0000 7f754bfff640 1 -- 172.21.15.46:0/523066274 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) v1 ==== 95+0+6531 (secure 0 0 0) 0x7f75641ef860 con 0x7f75641538a0 2023-10-30T14:28:07.151 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:28:07.152 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":267,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":10158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3415299388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3415299388},{"type":"v1","addr":"172.21.15.46:6836","nonce":3415299388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:37.949711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:07.152 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 267 2023-10-30T14:28:07.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.149+0000 7f7569ed6640 1 -- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75500664a0 msgr2=0x7f7550068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.149+0000 7f7569ed6640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75500664a0 0x7f7550068960 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f755400ba10 tx=0x7f7554008040 comp rx=0 tx=0).stop 2023-10-30T14:28:07.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.149+0000 7f7569ed6640 1 -- 172.21.15.46:0/523066274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75641538a0 msgr2=0x7f75641efb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.149+0000 7f7569ed6640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75641538a0 0x7f75641efb40 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f755c001570 tx=0x7f755c00cc50 comp rx=0 tx=0).stop 2023-10-30T14:28:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.149+0000 7f7569ed6640 1 -- 172.21.15.46:0/523066274 shutdown_connections 2023-10-30T14:28:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.149+0000 7f7569ed6640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f75500664a0 0x7f7550068960 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.149+0000 7f7569ed6640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75641538a0 0x7f75641efb40 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.149+0000 7f7569ed6640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75640cf160 0x7f7564148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.149+0000 7f7569ed6640 1 --2- 172.21.15.46:0/523066274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f75640ca0e0 0x7f75640ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.149+0000 7f7569ed6640 1 -- 172.21.15.46:0/523066274 >> 172.21.15.46:0/523066274 conn(0x7f756406bfe0 msgr2=0x7f75640c47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.149+0000 7f7569ed6640 1 -- 172.21.15.46:0/523066274 shutdown_connections 2023-10-30T14:28:07.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.149+0000 7f7569ed6640 1 -- 172.21.15.46:0/523066274 wait complete. 2023-10-30T14:28:07.173 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T14:28:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef495a1640 1 Processor -- start 2023-10-30T14:28:07.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef495a1640 1 -- start start 2023-10-30T14:28:07.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef495a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef44159e60 0x7fef4406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef495a1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef440bfec0 0x7fef440c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef495a1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef4406fa80 con 0x7fef4406a720 2023-10-30T14:28:07.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef495a1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef44071290 con 0x7fef440bfec0 2023-10-30T14:28:07.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef495a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef44071570 con 0x7fef44159e60 2023-10-30T14:28:07.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef3bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef440bfec0 0x7fef440c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef3bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef440bfec0 0x7fef440c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49016/0 (socket says 172.21.15.46:49016) 2023-10-30T14:28:07.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef3bfff640 1 -- 172.21.15.46:0/1005409419 learned_addr learned my addr 172.21.15.46:0/1005409419 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:28:07.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef42ffd640 1 --2- 172.21.15.46:0/1005409419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef44159e60 0x7fef4406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef3bfff640 1 -- 172.21.15.46:0/1005409419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef44159e60 msgr2=0x7fef4406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:07.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef3bfff640 1 --2- 172.21.15.46:0/1005409419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef44159e60 0x7fef4406f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef3bfff640 1 -- 172.21.15.46:0/1005409419 >> v1:172.21.15.154:6789/0 conn(0x7fef4406a720 legacy=0x7fef44155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.313+0000 7fef3bfff640 1 -- 172.21.15.46:0/1005409419 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef44071850 con 0x7fef440bfec0 2023-10-30T14:28:07.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.317+0000 7fef3bfff640 1 --2- 172.21.15.46:0/1005409419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef440bfec0 0x7fef440c1a40 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7fef2c009870 tx=0x7fef2c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6af391758dffdd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:07.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.317+0000 7fef427fc640 1 -- 172.21.15.46:0/1005409419 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0a8a30 con 0x7fef440bfec0 2023-10-30T14:28:07.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.317+0000 7fef427fc640 1 -- 172.21.15.46:0/1005409419 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fef2c0a8b90 con 0x7fef440bfec0 2023-10-30T14:28:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.317+0000 7fef427fc640 1 -- 172.21.15.46:0/1005409419 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0b0440 con 0x7fef440bfec0 2023-10-30T14:28:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.317+0000 7fef495a1640 1 -- 172.21.15.46:0/1005409419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef440bfec0 msgr2=0x7fef440c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.317+0000 7fef495a1640 1 --2- 172.21.15.46:0/1005409419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef440bfec0 0x7fef440c1a40 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7fef2c009870 tx=0x7fef2c01ed30 comp rx=0 tx=0).stop 2023-10-30T14:28:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.317+0000 7fef495a1640 1 -- 172.21.15.46:0/1005409419 shutdown_connections 2023-10-30T14:28:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.317+0000 7fef495a1640 1 --2- 172.21.15.46:0/1005409419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef440bfec0 0x7fef440c1a40 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.317+0000 7fef495a1640 1 --2- 172.21.15.46:0/1005409419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef44159e60 0x7fef4406f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.317+0000 7fef495a1640 1 -- 172.21.15.46:0/1005409419 >> 172.21.15.46:0/1005409419 conn(0x7fef440c7910 msgr2=0x7fef440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.317+0000 7fef495a1640 1 -- 172.21.15.46:0/1005409419 shutdown_connections 2023-10-30T14:28:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.317+0000 7fef495a1640 1 -- 172.21.15.46:0/1005409419 wait complete. 2023-10-30T14:28:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef495a1640 1 Processor -- start 2023-10-30T14:28:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef495a1640 1 -- start start 2023-10-30T14:28:07.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef495a1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef4406a720 0x7fef440c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef495a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef440bfec0 0x7fef44146e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef495a1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef44159e60 0x7fef441473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef495a1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef441577b0 con 0x7fef440bfec0 2023-10-30T14:28:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef495a1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef441f3650 con 0x7fef44159e60 2023-10-30T14:28:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef495a1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef441469d0 con 0x7fef4406a720 2023-10-30T14:28:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef3bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef440bfec0 0x7fef44146e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef3bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef440bfec0 0x7fef44146e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49030/0 (socket says 172.21.15.46:49030) 2023-10-30T14:28:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef3bfff640 1 -- 172.21.15.46:0/3060943618 learned_addr learned my addr 172.21.15.46:0/3060943618 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:28:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef437fe640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef44159e60 0x7fef441473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef42ffd640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef4406a720 0x7fef440c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef3bfff640 1 -- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef4406a720 msgr2=0x7fef440c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef3bfff640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef4406a720 0x7fef440c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef3bfff640 1 -- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef44159e60 msgr2=0x7fef441473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef3bfff640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef44159e60 0x7fef441473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef3bfff640 1 -- 172.21.15.46:0/3060943618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c0094d0 con 0x7fef440bfec0 2023-10-30T14:28:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef3bfff640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef440bfec0 0x7fef44146e60 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fef2c0b3040 tx=0x7fef2c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef3b7fe640 1 -- 172.21.15.46:0/3060943618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0be070 con 0x7fef440bfec0 2023-10-30T14:28:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef3b7fe640 1 -- 172.21.15.46:0/3060943618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fef2c0ba400 con 0x7fef440bfec0 2023-10-30T14:28:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef3b7fe640 1 -- 172.21.15.46:0/3060943618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0ba7f0 con 0x7fef440bfec0 2023-10-30T14:28:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef495a1640 1 -- 172.21.15.46:0/3060943618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef441f2af0 con 0x7fef440bfec0 2023-10-30T14:28:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.321+0000 7fef495a1640 1 -- 172.21.15.46:0/3060943618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef441f3090 con 0x7fef440bfec0 2023-10-30T14:28:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.325+0000 7fef495a1640 1 -- 172.21.15.46:0/3060943618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef441467a0 con 0x7fef440bfec0 2023-10-30T14:28:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.325+0000 7fef3b7fe640 1 -- 172.21.15.46:0/3060943618 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fef2c0c3410 con 0x7fef440bfec0 2023-10-30T14:28:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.325+0000 7fef3b7fe640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fef240664a0 0x7fef24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.325+0000 7fef3b7fe640 1 -- 172.21.15.46:0/3060943618 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fef2c122760 con 0x7fef440bfec0 2023-10-30T14:28:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.325+0000 7fef42ffd640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fef240664a0 0x7fef24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.325+0000 7fef42ffd640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fef240664a0 0x7fef24068960 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fef3400b930 tx=0x7fef34008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.329+0000 7fef3b7fe640 1 -- 172.21.15.46:0/3060943618 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fef441467a0 con 0x7fef440bfec0 2023-10-30T14:28:07.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.517+0000 7fef495a1640 1 -- 172.21.15.46:0/3060943618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef441467a0 con 0x7fef440bfec0 2023-10-30T14:28:07.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.517+0000 7fef3b7fe640 1 -- 172.21.15.46:0/3060943618 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) v1 ==== 95+0+6531 (secure 0 0 0) 0x7fef441467a0 con 0x7fef440bfec0 2023-10-30T14:28:07.523 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:28:07.523 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":267,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6838/892930607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":892930607},{"type":"v1","addr":"172.21.15.154:6838","nonce":892930607}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2803385213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6837","nonce":2803385213},{"type":"v1","addr":"172.21.15.154:6839","nonce":2803385213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5939,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3159009654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3159009654},{"type":"v1","addr":"172.21.15.154:6835","nonce":3159009654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":10158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6836/3415299388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3415299388},{"type":"v1","addr":"172.21.15.46:6836","nonce":3415299388}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:23:19.337396+0000","modified":"2023-10-30T14:25:37.949711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10020,"mds_1":9882},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_9882":{"gid":9882,"name":"a","rank":1,"incarnation":253,"state":"up:active","state_seq":21,"addr":"172.21.15.46:6833/3394097058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3394097058},{"type":"v1","addr":"172.21.15.46:6833","nonce":3394097058}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10020":{"gid":10020,"name":"e","rank":0,"incarnation":260,"state":"up:active","state_seq":20,"addr":"172.21.15.46:6837/2187461695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6835","nonce":2187461695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2187461695}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T14:28:07.523 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 267 2023-10-30T14:28:07.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.521+0000 7fef495a1640 1 -- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fef240664a0 msgr2=0x7fef24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.521+0000 7fef495a1640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fef240664a0 0x7fef24068960 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fef3400b930 tx=0x7fef34008040 comp rx=0 tx=0).stop 2023-10-30T14:28:07.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.521+0000 7fef495a1640 1 -- 172.21.15.46:0/3060943618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef440bfec0 msgr2=0x7fef44146e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.521+0000 7fef495a1640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef440bfec0 0x7fef44146e60 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fef2c0b3040 tx=0x7fef2c01bad0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.521+0000 7fef495a1640 1 -- 172.21.15.46:0/3060943618 shutdown_connections 2023-10-30T14:28:07.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.521+0000 7fef495a1640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fef240664a0 0x7fef24068960 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.521+0000 7fef495a1640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef44159e60 0x7fef441473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.521+0000 7fef495a1640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef440bfec0 0x7fef44146e60 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.521+0000 7fef495a1640 1 --2- 172.21.15.46:0/3060943618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef4406a720 0x7fef440c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.521+0000 7fef495a1640 1 -- 172.21.15.46:0/3060943618 >> 172.21.15.46:0/3060943618 conn(0x7fef440c7910 msgr2=0x7fef440ca040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:07.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.521+0000 7fef495a1640 1 -- 172.21.15.46:0/3060943618 shutdown_connections 2023-10-30T14:28:07.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.521+0000 7fef495a1640 1 -- 172.21.15.46:0/3060943618 wait complete. 2023-10-30T14:28:07.544 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T14:28:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f813c963640 1 Processor -- start 2023-10-30T14:28:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f813c963640 1 -- start start 2023-10-30T14:28:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f813c963640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f813815cd50 0x7f81380c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f813c963640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813815a6c0 con 0x7f81380cefa0 2023-10-30T14:28:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f813c963640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813815d300 con 0x7f81381538f0 2023-10-30T14:28:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f813c963640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813815d5e0 con 0x7f813815cd50 2023-10-30T14:28:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f8137fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f81380cefa0 0x7f81381640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42000/0 (socket says 172.21.15.46:42000) 2023-10-30T14:28:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f8137fff640 1 -- 172.21.15.46:0/1923714534 learned_addr learned my addr 172.21.15.46:0/1923714534 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:28:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f8136ffd640 1 --2- 172.21.15.46:0/1923714534 >> v2:172.21.15.154:3300/0 conn(0x7f813815cd50 0x7f81380c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778102868 0 0) 0x7f813815a6c0 con 0x7f81380cefa0 2023-10-30T14:28:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8120003530 con 0x7f81380cefa0 2023-10-30T14:28:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3069668986 0 0) 0x7f8120003530 con 0x7f81380cefa0 2023-10-30T14:28:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 >> v2:172.21.15.154:3300/0 conn(0x7f813815cd50 msgr2=0x7f81380c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 --2- 172.21.15.46:0/1923714534 >> v2:172.21.15.154:3300/0 conn(0x7f813815cd50 0x7f81380c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 >> v1:172.21.15.154:6789/0 conn(0x7f81381538f0 legacy=0x7f8138159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813815d8c0 con 0x7f81380cefa0 2023-10-30T14:28:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f8130001a00 con 0x7f81380cefa0 2023-10-30T14:28:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 >> v1:172.21.15.46:6789/0 conn(0x7f81380cefa0 legacy=0x7f81381640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 --2- 172.21.15.46:0/1923714534 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8120003980 0x7f8120003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 --2- 172.21.15.46:0/1923714534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 0x7f8120004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 --2- 172.21.15.46:0/1923714534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8120005050 0x7f812000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813815d5e0 con 0x7f8120004380 2023-10-30T14:28:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812000bf00 con 0x7f8120005050 2023-10-30T14:28:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812000c0c0 con 0x7f8120003980 2023-10-30T14:28:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f8136ffd640 1 --2- 172.21.15.46:0/1923714534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 0x7f8120004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f81377fe640 1 --2- 172.21.15.46:0/1923714534 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8120003980 0x7f8120003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f8137fff640 1 --2- 172.21.15.46:0/1923714534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8120005050 0x7f812000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f8136ffd640 1 -- 172.21.15.46:0/1923714534 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8120003980 msgr2=0x7f8120003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:07.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f8136ffd640 1 --2- 172.21.15.46:0/1923714534 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8120003980 0x7f8120003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f8136ffd640 1 -- 172.21.15.46:0/1923714534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8120005050 msgr2=0x7f812000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:07.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f8136ffd640 1 --2- 172.21.15.46:0/1923714534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8120005050 0x7f812000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f8136ffd640 1 -- 172.21.15.46:0/1923714534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8120003530 con 0x7f8120004380 2023-10-30T14:28:07.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.685+0000 7f8137fff640 1 -- 172.21.15.46:0/1923714534 reap_dead start 2023-10-30T14:28:07.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.689+0000 7f8136ffd640 1 --2- 172.21.15.46:0/1923714534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 0x7f8120004850 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f812c009ae0 tx=0x7f812c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede9a729561854ac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.689+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c0a5530 con 0x7f8120004380 2023-10-30T14:28:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.689+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f812c0a5690 con 0x7f8120004380 2023-10-30T14:28:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.689+0000 7f81367fc640 1 -- 172.21.15.46:0/1923714534 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c0a5a60 con 0x7f8120004380 2023-10-30T14:28:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.689+0000 7f813c963640 1 -- 172.21.15.46:0/1923714534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 msgr2=0x7f8120004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.689+0000 7f813c963640 1 --2- 172.21.15.46:0/1923714534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 0x7f8120004850 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f812c009ae0 tx=0x7f812c01ca50 comp rx=0 tx=0).stop 2023-10-30T14:28:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.689+0000 7f813c963640 1 -- 172.21.15.46:0/1923714534 shutdown_connections 2023-10-30T14:28:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.689+0000 7f813c963640 1 --2- 172.21.15.46:0/1923714534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 0x7f8120004850 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.689+0000 7f813c963640 1 -- 172.21.15.46:0/1923714534 >> 172.21.15.46:0/1923714534 conn(0x7f813806a640 msgr2=0x7f8138065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:07.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.689+0000 7f813c963640 1 -- 172.21.15.46:0/1923714534 shutdown_connections 2023-10-30T14:28:07.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.689+0000 7f813c963640 1 -- 172.21.15.46:0/1923714534 wait complete. 2023-10-30T14:28:07.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f813c963640 1 Processor -- start 2023-10-30T14:28:07.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f813c963640 1 -- start start 2023-10-30T14:28:07.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f813c963640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 0x7f8138152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f813c963640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8138157ad0 0x7f8138150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f813c963640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81380b3220 0x7f813815a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f813c963640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81381f3290 con 0x7f8120004380 2023-10-30T14:28:07.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f813c963640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81380b4dd0 con 0x7f8138157ad0 2023-10-30T14:28:07.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f813c963640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81381f29f0 con 0x7f81380b3220 2023-10-30T14:28:07.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f81377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 0x7f8138152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f81377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 0x7f8138152cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49040/0 (socket says 172.21.15.46:49040) 2023-10-30T14:28:07.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f81377fe640 1 -- 172.21.15.46:0/1113970386 learned_addr learned my addr 172.21.15.46:0/1113970386 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:28:07.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f8137fff640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81380b3220 0x7f813815a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f8136ffd640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8138157ad0 0x7f8138150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f81377fe640 1 -- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81380b3220 msgr2=0x7f813815a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f81377fe640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81380b3220 0x7f813815a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f81377fe640 1 -- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8138157ad0 msgr2=0x7f8138150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f81377fe640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8138157ad0 0x7f8138150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f81377fe640 1 -- 172.21.15.46:0/1113970386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81381582a0 con 0x7f8120004380 2023-10-30T14:28:07.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f81377fe640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 0x7f8138152cc0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f812800f770 tx=0x7f812800fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:07.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f811ffff640 1 -- 172.21.15.46:0/1113970386 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81280050d0 con 0x7f8120004380 2023-10-30T14:28:07.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f811ffff640 1 -- 172.21.15.46:0/1113970386 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f8128005230 con 0x7f8120004380 2023-10-30T14:28:07.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f813c963640 1 -- 172.21.15.46:0/1113970386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8138158580 con 0x7f8120004380 2023-10-30T14:28:07.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f811ffff640 1 -- 172.21.15.46:0/1113970386 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812800d730 con 0x7f8120004380 2023-10-30T14:28:07.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.693+0000 7f813c963640 1 -- 172.21.15.46:0/1113970386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81381f2f20 con 0x7f8120004380 2023-10-30T14:28:07.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.697+0000 7f813c963640 1 -- 172.21.15.46:0/1113970386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813804e170 con 0x7f8120004380 2023-10-30T14:28:07.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.697+0000 7f811ffff640 1 -- 172.21.15.46:0/1113970386 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f812800daa0 con 0x7f8120004380 2023-10-30T14:28:07.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.697+0000 7f811ffff640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81240664a0 0x7f8124068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:07.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.697+0000 7f811ffff640 1 -- 172.21.15.46:0/1113970386 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f8128084c40 con 0x7f8120004380 2023-10-30T14:28:07.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.697+0000 7f8136ffd640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81240664a0 0x7f8124068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:07.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.697+0000 7f8136ffd640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81240664a0 0x7f8124068960 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f812c0af000 tx=0x7f812c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:07.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.701+0000 7f811ffff640 1 -- 172.21.15.46:0/1113970386 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8128051470 con 0x7f8120004380 2023-10-30T14:28:07.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.861+0000 7f813c963640 1 -- 172.21.15.46:0/1113970386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8138159540 con 0x7f8120004380 2023-10-30T14:28:07.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f811ffff640 1 -- 172.21.15.46:0/1113970386 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+21139 (secure 0 0 0) 0x7f8128054a90 con 0x7f8120004380 2023-10-30T14:28:07.869 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-10-30T14:28:07.869 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":110,"fsid":"6f6cdb7c-b4be-4cc0-8e44-7b693b39b4d2","created":"2023-10-30T13:22:53.925149+0000","modified":"2023-10-30T14:26:39.887837+0000","last_up_change":"2023-10-30T13:23:05.814816+0000","last_in_change":"2023-10-30T13:22:57.133452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:23:06.788949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:23:17.648827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:23:18.326967+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":703,"snap_epoch":108,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de627c1f-8bc7-4f98-ae7d-2be818fb4024","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6811","nonce":1936524547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6815","nonce":1936524547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6822","nonce":1936524547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1936524547},{"type":"v1","addr":"172.21.15.46:6818","nonce":1936524547}]},"public_addr":"172.21.15.46:6811/1936524547","cluster_addr":"172.21.15.46:6815/1936524547","heartbeat_back_addr":"172.21.15.46:6822/1936524547","heartbeat_front_addr":"172.21.15.46:6818/1936524547","state":["exists","up"]},{"osd":1,"uuid":"e21e18dd-5538-4419-9f05-8ef2a6ce40d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6801","nonce":1107843451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6803","nonce":1107843451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6807","nonce":1107843451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1107843451},{"type":"v1","addr":"172.21.15.46:6805","nonce":1107843451}]},"public_addr":"172.21.15.46:6801/1107843451","cluster_addr":"172.21.15.46:6803/1107843451","heartbeat_back_addr":"172.21.15.46:6807/1107843451","heartbeat_front_addr":"172.21.15.46:6805/1107843451","state":["exists","up"]},{"osd":2,"uuid":"281c484d-79d3-4873-85c5-fb4fd3369397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6810","nonce":3816077541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6814","nonce":3816077541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6823","nonce":3816077541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3816077541},{"type":"v1","addr":"172.21.15.46:6819","nonce":3816077541}]},"public_addr":"172.21.15.46:6810/3816077541","cluster_addr":"172.21.15.46:6814/3816077541","heartbeat_back_addr":"172.21.15.46:6823/3816077541","heartbeat_front_addr":"172.21.15.46:6819/3816077541","state":["exists","up"]},{"osd":3,"uuid":"58d4f3f8-7966-4175-a940-907e8561511d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6825","nonce":2013670008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6827","nonce":2013670008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6831","nonce":2013670008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":2013670008},{"type":"v1","addr":"172.21.15.46:6829","nonce":2013670008}]},"public_addr":"172.21.15.46:6825/2013670008","cluster_addr":"172.21.15.46:6827/2013670008","heartbeat_back_addr":"172.21.15.46:6831/2013670008","heartbeat_front_addr":"172.21.15.46:6829/2013670008","state":["exists","up"]},{"osd":4,"uuid":"66d8f937-0303-4a2c-a493-54991edb7f32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6802","nonce":1011303130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6806","nonce":1011303130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6814","nonce":1011303130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1011303130},{"type":"v1","addr":"172.21.15.154:6810","nonce":1011303130}]},"public_addr":"172.21.15.154:6802/1011303130","cluster_addr":"172.21.15.154:6806/1011303130","heartbeat_back_addr":"172.21.15.154:6814/1011303130","heartbeat_front_addr":"172.21.15.154:6810/1011303130","state":["exists","up"]},{"osd":5,"uuid":"56cc0fc0-d1d1-4326-9b45-b806a903dd6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6803","nonce":956300018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6807","nonce":956300018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6815","nonce":956300018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":956300018},{"type":"v1","addr":"172.21.15.154:6811","nonce":956300018}]},"public_addr":"172.21.15.154:6803/956300018","cluster_addr":"172.21.15.154:6807/956300018","heartbeat_back_addr":"172.21.15.154:6815/956300018","heartbeat_front_addr":"172.21.15.154:6811/956300018","state":["exists","up"]},{"osd":6,"uuid":"b8f618d5-d080-47ef-b94c-2dbdaba5dfd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6825","nonce":1247354054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6827","nonce":1247354054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6831","nonce":1247354054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1247354054},{"type":"v1","addr":"172.21.15.154:6829","nonce":1247354054}]},"public_addr":"172.21.15.154:6825/1247354054","cluster_addr":"172.21.15.154:6827/1247354054","heartbeat_back_addr":"172.21.15.154:6831/1247354054","heartbeat_front_addr":"172.21.15.154:6829/1247354054","state":["exists","up"]},{"osd":7,"uuid":"dfcbd9f0-0ca4-4068-b367-021944895fc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6817","nonce":1343138186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6819","nonce":1343138186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6823","nonce":1343138186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1343138186},{"type":"v1","addr":"172.21.15.154:6821","nonce":1343138186}]},"public_addr":"172.21.15.154:6817/1343138186","cluster_addr":"172.21.15.154:6819/1343138186","heartbeat_back_addr":"172.21.15.154:6823/1343138186","heartbeat_front_addr":"172.21.15.154:6821/1343138186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.774116+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.884338+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.704088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.818474+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.672920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.680433+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.601033+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:23:02.610686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.13","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6836/3574404756":"2023-10-31T14:24:59.863368+0000","172.21.15.46:6834/3574404756":"2023-10-31T14:24:59.863368+0000","172.21.15.46:6837/470164358":"2023-10-31T14:23:54.843840+0000","172.21.15.46:6835/470164358":"2023-10-31T14:23:54.843840+0000","172.21.15.46:6833/3764280378":"2023-10-31T14:22:39.814064+0000","172.21.15.46:6832/3764280378":"2023-10-31T14:22:39.814064+0000","172.21.15.46:6836/2239735286":"2023-10-31T14:20:39.772410+0000","172.21.15.46:6834/2197729559":"2023-10-31T13:51:39.216877+0000","172.21.15.46:6833/190134548":"2023-10-31T13:50:04.191476+0000","172.21.15.46:6832/190134548":"2023-10-31T13:50:04.191476+0000","172.21.15.46:6833/157453225":"2023-10-31T14:00:29.382692+0000","172.21.15.46:6837/1135482564":"2023-10-31T14:01:49.404286+0000","172.21.15.46:6837/3154171789":"2023-10-31T14:07:14.520406+0000","172.21.15.46:6836/3037060424":"2023-10-31T13:44:39.066140+0000","172.21.15.46:6835/1135482564":"2023-10-31T14:01:49.404286+0000","172.21.15.46:6835/2572113428":"2023-10-31T13:52:59.238225+0000","172.21.15.46:6833/2075693104":"2023-10-31T14:05:39.493074+0000","172.21.15.46:6836/2197729559":"2023-10-31T13:51:39.216877+0000","172.21.15.46:6835/994489214":"2023-10-31T13:25:13.695228+0000","172.21.15.46:6832/2075693104":"2023-10-31T14:05:39.493074+0000","172.21.15.46:6832/4044310929":"2023-10-31T13:41:14.000569+0000","172.21.15.46:6837/126953235":"2023-10-31T13:39:08.955190+0000","172.21.15.46:6832/875791418":"2023-10-31T13:26:28.713838+0000","172.21.15.46:6835/126953235":"2023-10-31T13:39:08.955190+0000","172.21.15.46:6837/3112266825":"2023-10-31T13:57:44.339823+0000","172.21.15.46:6836/3955931693":"2023-10-31T13:36:53.920781+0000","172.21.15.46:6834/3955931693":"2023-10-31T13:36:53.920781+0000","172.21.15.46:6835/681817982":"2023-10-31T14:18:44.741650+0000","172.21.15.46:6836/275749119":"2023-10-31T13:30:28.793688+0000","172.21.15.46:6837/2572113428":"2023-10-31T13:52:59.238225+0000","172.21.15.154:6834/3109262047":"2023-10-31T13:29:08.773403+0000","172.21.15.46:6834/2239735286":"2023-10-31T14:20:39.772410+0000","172.21.15.46:6833/2455118950":"2023-10-31T14:16:04.689323+0000","172.21.15.46:6837/994489214":"2023-10-31T13:25:13.695228+0000","172.21.15.46:6837/3044281789":"2023-10-31T14:12:14.616071+0000","172.21.15.46:6835/1315246492":"2023-10-31T13:31:23.809908+0000","172.21.15.46:6835/3112266825":"2023-10-31T13:57:44.339823+0000","172.21.15.46:6834/128865315":"2023-10-31T14:08:44.543673+0000","172.21.15.46:6833/603884510":"2023-10-31T13:33:43.868205+0000","172.21.15.46:6835/2546436963":"2023-10-31T13:47:24.113508+0000","172.21.15.46:6837/1315246492":"2023-10-31T13:31:23.809908+0000","172.21.15.154:6834/651441434":"2023-10-31T13:35:03.892131+0000","172.21.15.46:6833/2807435195":"2023-10-31T13:54:39.265053+0000","172.21.15.46:6834/711135361":"2023-10-31T13:55:44.283031+0000","172.21.15.154:6835/651441434":"2023-10-31T13:35:03.892131+0000","172.21.15.46:6836/711135361":"2023-10-31T13:55:44.283031+0000","172.21.15.46:6837/2546436963":"2023-10-31T13:47:24.113508+0000","172.21.15.46:6833/875791418":"2023-10-31T13:26:28.713838+0000","172.21.15.154:6835/3109262047":"2023-10-31T13:29:08.773403+0000","172.21.15.46:6832/2807435195":"2023-10-31T13:54:39.265053+0000","172.21.15.46:6832/157453225":"2023-10-31T14:00:29.382692+0000","172.21.15.46:6832/603884510":"2023-10-31T13:33:43.868205+0000","172.21.15.46:6834/3734066164":"2023-10-31T14:04:24.459794+0000","172.21.15.46:6833/4044310929":"2023-10-31T13:41:14.000569+0000","172.21.15.46:6834/66265765":"2023-10-31T14:14:54.671010+0000","172.21.15.46:6836/3734066164":"2023-10-31T14:04:24.459794+0000","172.21.15.46:6836/128865315":"2023-10-31T14:08:44.543673+0000","172.21.15.46:6832/2455118950":"2023-10-31T14:16:04.689323+0000","172.21.15.46:6835/3154171789":"2023-10-31T14:07:14.520406+0000","172.21.15.46:6832/4078242540":"2023-10-31T14:09:49.576202+0000","172.21.15.46:6834/3037060424":"2023-10-31T13:44:39.066140+0000","172.21.15.46:6833/4078242540":"2023-10-31T14:09:49.576202+0000","172.21.15.46:6835/3044281789":"2023-10-31T14:12:14.616071+0000","172.21.15.46:6836/66265765":"2023-10-31T14:14:54.671010+0000","172.21.15.46:6834/275749119":"2023-10-31T13:30:28.793688+0000","172.21.15.46:6837/681817982":"2023-10-31T14:18:44.741650+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":690,"length":14}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T14:28:07.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f813c963640 1 -- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81240664a0 msgr2=0x7f8124068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f813c963640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81240664a0 0x7f8124068960 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f812c0af000 tx=0x7f812c008040 comp rx=0 tx=0).stop 2023-10-30T14:28:07.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f813c963640 1 -- 172.21.15.46:0/1113970386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 msgr2=0x7f8138152cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:07.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f813c963640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 0x7f8138152cc0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f812800f770 tx=0x7f812800fc30 comp rx=0 tx=0).stop 2023-10-30T14:28:07.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f813c963640 1 -- 172.21.15.46:0/1113970386 shutdown_connections 2023-10-30T14:28:07.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f813c963640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f81240664a0 0x7f8124068960 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f813c963640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81380b3220 0x7f813815a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f813c963640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8138157ad0 0x7f8138150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f813c963640 1 --2- 172.21.15.46:0/1113970386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8120004380 0x7f8138152cc0 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:07.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f813c963640 1 -- 172.21.15.46:0/1113970386 >> 172.21.15.46:0/1113970386 conn(0x7f813806a640 msgr2=0x7f81380c4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:07.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f813c963640 1 -- 172.21.15.46:0/1113970386 shutdown_connections 2023-10-30T14:28:07.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:07.865+0000 7f813c963640 1 -- 172.21.15.46:0/1113970386 wait complete. 2023-10-30T14:28:07.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-30T14:28:08.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed4e632640 1 Processor -- start 2023-10-30T14:28:08.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed4e632640 1 -- start start 2023-10-30T14:28:08.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed4e632640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 0x7fed48158120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:08.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed4e632640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fed4806a970 0x7fed480c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:08.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed4e632640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4806a4e0 con 0x7fed4806adb0 2023-10-30T14:28:08.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed4e632640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed48158660 con 0x7fed4806a970 2023-10-30T14:28:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed4e632640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed480c6c50 con 0x7fed48151910 2023-10-30T14:28:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed47fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 0x7fed48158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:08.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed47fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 0x7fed48158120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.46:60490/0 (socket says 172.21.15.46:60490) 2023-10-30T14:28:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed47fff640 1 -- 172.21.15.46:0/1962138476 learned_addr learned my addr 172.21.15.46:0/1962138476 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:28:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed477fe640 1 --2- 172.21.15.46:0/1962138476 >> v2:172.21.15.154:3300/0 conn(0x7fed4806a970 0x7fed480c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:08.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed47fff640 1 -- 172.21.15.46:0/1962138476 >> v2:172.21.15.154:3300/0 conn(0x7fed4806a970 msgr2=0x7fed480c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:08.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed47fff640 1 --2- 172.21.15.46:0/1962138476 >> v2:172.21.15.154:3300/0 conn(0x7fed4806a970 0x7fed480c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed47fff640 1 -- 172.21.15.46:0/1962138476 >> v1:172.21.15.154:6789/0 conn(0x7fed4806adb0 legacy=0x7fed480c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:08.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.033+0000 7fed47fff640 1 -- 172.21.15.46:0/1962138476 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed480c6f30 con 0x7fed48151910 2023-10-30T14:28:08.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.037+0000 7fed47fff640 1 --2- 172.21.15.46:0/1962138476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 0x7fed48158120 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fed38009870 tx=0x7fed3801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53cfd05d91b37018 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:08.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.037+0000 7fed46ffd640 1 -- 172.21.15.46:0/1962138476 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed380a8a30 con 0x7fed48151910 2023-10-30T14:28:08.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.037+0000 7fed46ffd640 1 -- 172.21.15.46:0/1962138476 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fed380a8b90 con 0x7fed48151910 2023-10-30T14:28:08.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.037+0000 7fed46ffd640 1 -- 172.21.15.46:0/1962138476 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed380b0440 con 0x7fed48151910 2023-10-30T14:28:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.037+0000 7fed4e632640 1 -- 172.21.15.46:0/1962138476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 msgr2=0x7fed48158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:08.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.037+0000 7fed4e632640 1 --2- 172.21.15.46:0/1962138476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 0x7fed48158120 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fed38009870 tx=0x7fed3801ed30 comp rx=0 tx=0).stop 2023-10-30T14:28:08.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.037+0000 7fed4e632640 1 -- 172.21.15.46:0/1962138476 shutdown_connections 2023-10-30T14:28:08.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.037+0000 7fed4e632640 1 --2- 172.21.15.46:0/1962138476 >> v2:172.21.15.154:3300/0 conn(0x7fed4806a970 0x7fed480c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.037+0000 7fed4e632640 1 --2- 172.21.15.46:0/1962138476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 0x7fed48158120 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.037+0000 7fed4e632640 1 -- 172.21.15.46:0/1962138476 >> 172.21.15.46:0/1962138476 conn(0x7fed480c7910 msgr2=0x7fed480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.037+0000 7fed4e632640 1 -- 172.21.15.46:0/1962138476 shutdown_connections 2023-10-30T14:28:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.037+0000 7fed4e632640 1 -- 172.21.15.46:0/1962138476 wait complete. 2023-10-30T14:28:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed4e632640 1 Processor -- start 2023-10-30T14:28:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed4e632640 1 -- start start 2023-10-30T14:28:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed4e632640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806a970 0x7fed4806f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed4e632640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4806adb0 0x7fed480c1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed4e632640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 0x7fed48148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed4e632640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed480bfec0 con 0x7fed4806a970 2023-10-30T14:28:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed4e632640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed480c0110 con 0x7fed4806adb0 2023-10-30T14:28:08.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed4e632640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed480c03f0 con 0x7fed48151910 2023-10-30T14:28:08.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed47fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806a970 0x7fed4806f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:08.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed47fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806a970 0x7fed4806f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49056/0 (socket says 172.21.15.46:49056) 2023-10-30T14:28:08.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed47fff640 1 -- 172.21.15.46:0/2276363204 learned_addr learned my addr 172.21.15.46:0/2276363204 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:28:08.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed477fe640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4806adb0 0x7fed480c1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:08.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed4cbcf640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 0x7fed48148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:08.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed47fff640 1 -- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 msgr2=0x7fed48148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:08.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed47fff640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 0x7fed48148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed47fff640 1 -- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4806adb0 msgr2=0x7fed480c1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:08.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed47fff640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4806adb0 0x7fed480c1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed47fff640 1 -- 172.21.15.46:0/2276363204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed380094d0 con 0x7fed4806a970 2023-10-30T14:28:08.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed4cbcf640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 0x7fed48148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T14:28:08.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed47fff640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806a970 0x7fed4806f7a0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fed380b3040 tx=0x7fed3801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:08.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed44ff9640 1 -- 172.21.15.46:0/2276363204 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed380b9620 con 0x7fed4806a970 2023-10-30T14:28:08.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed44ff9640 1 -- 172.21.15.46:0/2276363204 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fed380b97c0 con 0x7fed4806a970 2023-10-30T14:28:08.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed44ff9640 1 -- 172.21.15.46:0/2276363204 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed380b9b90 con 0x7fed4806a970 2023-10-30T14:28:08.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed4e632640 1 -- 172.21.15.46:0/2276363204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed480c06d0 con 0x7fed4806a970 2023-10-30T14:28:08.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.041+0000 7fed4e632640 1 -- 172.21.15.46:0/2276363204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed480c0c90 con 0x7fed4806a970 2023-10-30T14:28:08.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.045+0000 7fed4e632640 1 -- 172.21.15.46:0/2276363204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed480c0e80 con 0x7fed4806a970 2023-10-30T14:28:08.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.045+0000 7fed44ff9640 1 -- 172.21.15.46:0/2276363204 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed380b7070 con 0x7fed4806a970 2023-10-30T14:28:08.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.045+0000 7fed44ff9640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed340664a0 0x7fed34068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:08.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.045+0000 7fed44ff9640 1 -- 172.21.15.46:0/2276363204 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8720+0+0 (secure 0 0 0) 0x7fed38122af0 con 0x7fed4806a970 2023-10-30T14:28:08.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.045+0000 7fed477fe640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed340664a0 0x7fed34068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:08.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.045+0000 7fed477fe640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed340664a0 0x7fed34068960 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fed4814ff00 tx=0x7fed3c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:08.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.049+0000 7fed44ff9640 1 -- 172.21.15.46:0/2276363204 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fed380ef200 con 0x7fed4806a970 2023-10-30T14:28:08.245 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.241+0000 7fed4e632640 1 -- 172.21.15.46:0/2276363204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fed4804e170 con 0x7fed4806a970 2023-10-30T14:28:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.245+0000 7fed44ff9640 1 -- 172.21.15.46:0/2276363204 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v268) v1 ==== 161+0+0 (secure 0 0 0) 0x7fed380b5030 con 0x7fed4806a970 2023-10-30T14:28:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-30T14:28:08.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.245+0000 7fed4e632640 1 -- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed340664a0 msgr2=0x7fed34068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:08.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.245+0000 7fed4e632640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed340664a0 0x7fed34068960 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fed4814ff00 tx=0x7fed3c009190 comp rx=0 tx=0).stop 2023-10-30T14:28:08.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.245+0000 7fed4e632640 1 -- 172.21.15.46:0/2276363204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806a970 msgr2=0x7fed4806f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:08.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.245+0000 7fed4e632640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806a970 0x7fed4806f7a0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fed380b3040 tx=0x7fed3801bad0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.249+0000 7fed4e632640 1 -- 172.21.15.46:0/2276363204 shutdown_connections 2023-10-30T14:28:08.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.249+0000 7fed4e632640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fed340664a0 0x7fed34068960 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.249+0000 7fed4e632640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed48151910 0x7fed48148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.249+0000 7fed4e632640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4806adb0 0x7fed480c1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.249+0000 7fed4e632640 1 --2- 172.21.15.46:0/2276363204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed4806a970 0x7fed4806f7a0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.249+0000 7fed4e632640 1 -- 172.21.15.46:0/2276363204 >> 172.21.15.46:0/2276363204 conn(0x7fed480c7910 msgr2=0x7fed480c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:08.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.249+0000 7fed4e632640 1 -- 172.21.15.46:0/2276363204 shutdown_connections 2023-10-30T14:28:08.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.249+0000 7fed4e632640 1 -- 172.21.15.46:0/2276363204 wait complete. 2023-10-30T14:28:08.275 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-30T14:28:08.352 INFO:tasks.ceph.mds.e.smithi046.stdout:starting mds.e at 2023-10-30T14:28:08.359 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-10-30T14:28:08.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.425+0000 7fa87d088640 1 Processor -- start 2023-10-30T14:28:08.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.425+0000 7fa87d088640 1 -- start start 2023-10-30T14:28:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.425+0000 7fa87d088640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153840 0x7fa87815a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:08.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.425+0000 7fa87d088640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa878153c20 0x7fa8780c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.425+0000 7fa87d088640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87815a6b0 con 0x7fa8780cb4c0 2023-10-30T14:28:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.425+0000 7fa87d088640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa87815dbc0 con 0x7fa878153c20 2023-10-30T14:28:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.425+0000 7fa87d088640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87815d730 con 0x7fa878153840 2023-10-30T14:28:08.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.425+0000 7fa877fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153840 0x7fa87815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:08.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.425+0000 7fa8777fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa878153c20 0x7fa8780c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:08.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.425+0000 7fa87c887640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa8780cb4c0 0x7fa8780cef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:40600/0 (socket says 172.21.15.46:40600) 2023-10-30T14:28:08.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.425+0000 7fa87c887640 1 -- 172.21.15.46:0/4105590462 learned_addr learned my addr 172.21.15.46:0/4105590462 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:28:08.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.429+0000 7fa877fff640 1 -- 172.21.15.46:0/4105590462 >> v2:172.21.15.154:3300/0 conn(0x7fa878153c20 msgr2=0x7fa8780c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:08.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.429+0000 7fa877fff640 1 --2- 172.21.15.46:0/4105590462 >> v2:172.21.15.154:3300/0 conn(0x7fa878153c20 0x7fa8780c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.429+0000 7fa877fff640 1 -- 172.21.15.46:0/4105590462 >> v1:172.21.15.154:6789/0 conn(0x7fa8780cb4c0 legacy=0x7fa8780cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:08.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.429+0000 7fa877fff640 1 -- 172.21.15.46:0/4105590462 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8780ca8f0 con 0x7fa878153840 2023-10-30T14:28:08.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.429+0000 7fa877fff640 1 --2- 172.21.15.46:0/4105590462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153840 0x7fa87815a170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa8680097c0 tx=0x7fa86801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b88fb620e3e3b69 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:08.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.429+0000 7fa876ffd640 1 -- 172.21.15.46:0/4105590462 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8680a8970 con 0x7fa878153840 2023-10-30T14:28:08.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.429+0000 7fa876ffd640 1 -- 172.21.15.46:0/4105590462 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fa8680a8ad0 con 0x7fa878153840 2023-10-30T14:28:08.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.429+0000 7fa876ffd640 1 -- 172.21.15.46:0/4105590462 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8680a8e80 con 0x7fa878153840 2023-10-30T14:28:08.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.429+0000 7fa87d088640 1 -- 172.21.15.46:0/4105590462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153840 msgr2=0x7fa87815a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:08.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.429+0000 7fa87d088640 1 --2- 172.21.15.46:0/4105590462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153840 0x7fa87815a170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa8680097c0 tx=0x7fa86801ec70 comp rx=0 tx=0).stop 2023-10-30T14:28:08.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 -- 172.21.15.46:0/4105590462 shutdown_connections 2023-10-30T14:28:08.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 --2- 172.21.15.46:0/4105590462 >> v2:172.21.15.154:3300/0 conn(0x7fa878153c20 0x7fa8780c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 --2- 172.21.15.46:0/4105590462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153840 0x7fa87815a170 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 -- 172.21.15.46:0/4105590462 >> 172.21.15.46:0/4105590462 conn(0x7fa87806be40 msgr2=0x7fa87806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:08.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 -- 172.21.15.46:0/4105590462 shutdown_connections 2023-10-30T14:28:08.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 -- 172.21.15.46:0/4105590462 wait complete. 2023-10-30T14:28:08.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 Processor -- start 2023-10-30T14:28:08.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 -- start start 2023-10-30T14:28:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8780cb4c0 0x7fa87814cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa878153840 0x7fa878154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153c20 0x7fa878154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87814ab70 con 0x7fa878153840 2023-10-30T14:28:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8781543e0 con 0x7fa8780cb4c0 2023-10-30T14:28:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87d088640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87814a6e0 con 0x7fa878153c20 2023-10-30T14:28:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa8777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa878153840 0x7fa878154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa8777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa878153840 0x7fa878154a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49084/0 (socket says 172.21.15.46:49084) 2023-10-30T14:28:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa8777fe640 1 -- 172.21.15.46:0/546368299 learned_addr learned my addr 172.21.15.46:0/546368299 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:28:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa877fff640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8780cb4c0 0x7fa87814cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:08.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa87c887640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153c20 0x7fa878154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:08.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa8777fe640 1 -- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153c20 msgr2=0x7fa878154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:08.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa8777fe640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153c20 0x7fa878154f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa8777fe640 1 -- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8780cb4c0 msgr2=0x7fa87814cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:08.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa8777fe640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8780cb4c0 0x7fa87814cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:08.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.433+0000 7fa8777fe640 1 -- 172.21.15.46:0/546368299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa868009420 con 0x7fa878153840 2023-10-30T14:28:08.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.437+0000 7fa8777fe640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa878153840 0x7fa878154a40 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00e870 tx=0x7fa86c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:08.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.437+0000 7fa874ff9640 1 -- 172.21.15.46:0/546368299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c009510 con 0x7fa878153840 2023-10-30T14:28:08.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.437+0000 7fa874ff9640 1 -- 172.21.15.46:0/546368299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fa86c00eec0 con 0x7fa878153840 2023-10-30T14:28:08.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.437+0000 7fa874ff9640 1 -- 172.21.15.46:0/546368299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c00cb60 con 0x7fa878153840 2023-10-30T14:28:08.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.437+0000 7fa87d088640 1 -- 172.21.15.46:0/546368299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87814adc0 con 0x7fa878153840 2023-10-30T14:28:08.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.437+0000 7fa87d088640 1 -- 172.21.15.46:0/546368299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa878155af0 con 0x7fa878153840 2023-10-30T14:28:08.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.437+0000 7fa87d088640 1 -- 172.21.15.46:0/546368299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa878155cb0 con 0x7fa878153840 2023-10-30T14:28:08.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.437+0000 7fa874ff9640 1 -- 172.21.15.46:0/546368299 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa86c00ccc0 con 0x7fa878153840 2023-10-30T14:28:08.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.437+0000 7fa874ff9640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8640664a0 0x7fa864068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:08.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.437+0000 7fa874ff9640 1 -- 172.21.15.46:0/546368299 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8864+0+0 (secure 0 0 0) 0x7fa86c084d20 con 0x7fa878153840 2023-10-30T14:28:08.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.437+0000 7fa877fff640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8640664a0 0x7fa864068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:08.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.441+0000 7fa877fff640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8640664a0 0x7fa864068960 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa8680b2000 tx=0x7fa8680aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:08.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.441+0000 7fa874ff9640 1 -- 172.21.15.46:0/546368299 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa86c051490 con 0x7fa878153840 2023-10-30T14:28:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:08.633+0000 7fa87d088640 1 -- 172.21.15.46:0/546368299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa87814a9c0 con 0x7fa878153840 2023-10-30T14:28:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.245+0000 7fa874ff9640 1 -- 172.21.15.46:0/546368299 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v269) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa86c0210a0 con 0x7fa878153840 2023-10-30T14:28:09.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.249+0000 7fa87d088640 1 -- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8640664a0 msgr2=0x7fa864068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.249+0000 7fa87d088640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8640664a0 0x7fa864068960 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa8680b2000 tx=0x7fa8680aa070 comp rx=0 tx=0).stop 2023-10-30T14:28:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.249+0000 7fa87d088640 1 -- 172.21.15.46:0/546368299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa878153840 msgr2=0x7fa878154a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:09.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.249+0000 7fa87d088640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa878153840 0x7fa878154a40 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00e870 tx=0x7fa86c00ec70 comp rx=0 tx=0).stop 2023-10-30T14:28:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.249+0000 7fa87d088640 1 -- 172.21.15.46:0/546368299 shutdown_connections 2023-10-30T14:28:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.249+0000 7fa87d088640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7fa8640664a0 0x7fa864068960 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.249+0000 7fa87d088640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878153c20 0x7fa878154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.249+0000 7fa87d088640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa878153840 0x7fa878154a40 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.249+0000 7fa87d088640 1 --2- 172.21.15.46:0/546368299 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8780cb4c0 0x7fa87814cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.249+0000 7fa87d088640 1 -- 172.21.15.46:0/546368299 >> 172.21.15.46:0/546368299 conn(0x7fa87806be40 msgr2=0x7fa8780ccd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.249+0000 7fa87d088640 1 -- 172.21.15.46:0/546368299 shutdown_connections 2023-10-30T14:28:09.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.249+0000 7fa87d088640 1 -- 172.21.15.46:0/546368299 wait complete. 2023-10-30T14:28:09.278 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-30T14:28:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f979be46640 1 Processor -- start 2023-10-30T14:28:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f979be46640 1 -- start start 2023-10-30T14:28:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f979be46640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f97940c6eb0 0x7f97940c2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f979be46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97940c2ca0 0x7f979406f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f979be46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97940c6a50 con 0x7f9794156ed0 2023-10-30T14:28:09.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f979be46640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97940c6630 con 0x7f97940c6eb0 2023-10-30T14:28:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f979be46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97940c3750 con 0x7f97940c2ca0 2023-10-30T14:28:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9799be2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9794156ed0 0x7f97940c43c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42024/0 (socket says 172.21.15.46:42024) 2023-10-30T14:28:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9799be2640 1 -- 172.21.15.46:0/3701202540 learned_addr learned my addr 172.21.15.46:0/3701202540 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:28:09.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f97993e1640 1 --2- 172.21.15.46:0/3701202540 >> v2:172.21.15.154:3300/0 conn(0x7f97940c6eb0 0x7f97940c2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f979a3e3640 1 --2- 172.21.15.46:0/3701202540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97940c2ca0 0x7f979406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057605166 0 0) 0x7f97940c6a50 con 0x7f9794156ed0 2023-10-30T14:28:09.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9780003530 con 0x7f9794156ed0 2023-10-30T14:28:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3125888313 0 0) 0x7f9780003530 con 0x7f9794156ed0 2023-10-30T14:28:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97940c2ca0 msgr2=0x7f979406f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 --2- 172.21.15.46:0/3701202540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97940c2ca0 0x7f979406f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 >> v2:172.21.15.154:3300/0 conn(0x7f97940c6eb0 msgr2=0x7f97940c2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 --2- 172.21.15.46:0/3701202540 >> v2:172.21.15.154:3300/0 conn(0x7f97940c6eb0 0x7f97940c2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97941572b0 con 0x7f9794156ed0 2023-10-30T14:28:09.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 827834612 0 0) 0x7f9790001680 con 0x7f9794156ed0 2023-10-30T14:28:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 >> v1:172.21.15.46:6789/0 conn(0x7f9794156ed0 legacy=0x7f97940c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 --2- 172.21.15.46:0/3701202540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97940c2ca0 0x7f9780003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 --2- 172.21.15.46:0/3701202540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9780003f00 0x7f97800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 --2- 172.21.15.46:0/3701202540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9780004d10 0x7f9780005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97940c6630 con 0x7f9780003f00 2023-10-30T14:28:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97940c3750 con 0x7f9780004d10 2023-10-30T14:28:09.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978000b990 con 0x7f97940c2ca0 2023-10-30T14:28:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.425+0000 7f97993e1640 1 --2- 172.21.15.46:0/3701202540 >> v2:172.21.15.154:3300/0 conn(0x7f97940c6eb0 0x7f97940c2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T14:28:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.429+0000 7f9799be2640 1 --2- 172.21.15.46:0/3701202540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97940c2ca0 0x7f9780003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.429+0000 7f97993e1640 1 --2- 172.21.15.46:0/3701202540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9780003f00 0x7f97800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.429+0000 7f979a3e3640 1 --2- 172.21.15.46:0/3701202540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9780004d10 0x7f9780005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.429+0000 7f97993e1640 1 -- 172.21.15.46:0/3701202540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97940c2ca0 msgr2=0x7f9780003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:09.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.429+0000 7f97993e1640 1 --2- 172.21.15.46:0/3701202540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97940c2ca0 0x7f9780003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.429+0000 7f97993e1640 1 -- 172.21.15.46:0/3701202540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9780004d10 msgr2=0x7f9780005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T14:28:09.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.429+0000 7f97993e1640 1 --2- 172.21.15.46:0/3701202540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9780004d10 0x7f9780005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.429+0000 7f97993e1640 1 -- 172.21.15.46:0/3701202540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9780003530 con 0x7f9780003f00 2023-10-30T14:28:09.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.429+0000 7f97993e1640 1 --2- 172.21.15.46:0/3701202540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9780003f00 0x7f97800047d0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f9784009c80 tx=0x7f978401cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a78a2f7e249818 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:09.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.429+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840a58f0 con 0x7f9780003f00 2023-10-30T14:28:09.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.429+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f97840a5a50 con 0x7f9780003f00 2023-10-30T14:28:09.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.429+0000 7f9798be0640 1 -- 172.21.15.46:0/3701202540 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840a5e00 con 0x7f9780003f00 2023-10-30T14:28:09.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 -- 172.21.15.46:0/3701202540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9780003f00 msgr2=0x7f97800047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:09.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 --2- 172.21.15.46:0/3701202540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9780003f00 0x7f97800047d0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f9784009c80 tx=0x7f978401cbd0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979a3e3640 1 -- 172.21.15.46:0/3701202540 reap_dead start 2023-10-30T14:28:09.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 -- 172.21.15.46:0/3701202540 shutdown_connections 2023-10-30T14:28:09.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 -- 172.21.15.46:0/3701202540 >> 172.21.15.46:0/3701202540 conn(0x7f97940ccf80 msgr2=0x7f97940cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:09.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 -- 172.21.15.46:0/3701202540 shutdown_connections 2023-10-30T14:28:09.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 -- 172.21.15.46:0/3701202540 wait complete. 2023-10-30T14:28:09.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 Processor -- start 2023-10-30T14:28:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 -- start start 2023-10-30T14:28:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9780003f00 0x7f97941f2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97941536a0 0x7f97941598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9794159e00 0x7f979414ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979406b160 con 0x7f9794159e00 2023-10-30T14:28:09.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97941f3300 con 0x7f97941536a0 2023-10-30T14:28:09.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979be46640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979406c2b0 con 0x7f9780003f00 2023-10-30T14:28:09.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979a3e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9794159e00 0x7f979414ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979a3e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9794159e00 0x7f979414ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49102/0 (socket says 172.21.15.46:49102) 2023-10-30T14:28:09.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979a3e3640 1 -- 172.21.15.46:0/980408118 learned_addr learned my addr 172.21.15.46:0/980408118 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:28:09.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f9799be2640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9780003f00 0x7f97941f2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f97993e1640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97941536a0 0x7f97941598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979a3e3640 1 -- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9780003f00 msgr2=0x7f97941f2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:09.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979a3e3640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9780003f00 0x7f97941f2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979a3e3640 1 -- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97941536a0 msgr2=0x7f97941598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:09.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979a3e3640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97941536a0 0x7f97941598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:09.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.433+0000 7f979a3e3640 1 -- 172.21.15.46:0/980408118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9794159150 con 0x7f9794159e00 2023-10-30T14:28:09.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.437+0000 7f979a3e3640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9794159e00 0x7f979414ef40 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f97940caa10 tx=0x7f978c00f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:09.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.437+0000 7f978a7fc640 1 -- 172.21.15.46:0/980408118 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c00e040 con 0x7f9794159e00 2023-10-30T14:28:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.437+0000 7f978a7fc640 1 -- 172.21.15.46:0/980408118 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f978c00f040 con 0x7f9794159e00 2023-10-30T14:28:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.437+0000 7f978a7fc640 1 -- 172.21.15.46:0/980408118 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c015430 con 0x7f9794159e00 2023-10-30T14:28:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.437+0000 7f979be46640 1 -- 172.21.15.46:0/980408118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97941f3040 con 0x7f9794159e00 2023-10-30T14:28:09.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.437+0000 7f979be46640 1 -- 172.21.15.46:0/980408118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f979415ae60 con 0x7f9794159e00 2023-10-30T14:28:09.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.437+0000 7f979be46640 1 -- 172.21.15.46:0/980408118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979404e170 con 0x7f9794159e00 2023-10-30T14:28:09.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.437+0000 7f978a7fc640 1 -- 172.21.15.46:0/980408118 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f978c015590 con 0x7f9794159e00 2023-10-30T14:28:09.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.437+0000 7f978a7fc640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97780664d0 0x7f9778068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:09.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.437+0000 7f978a7fc640 1 -- 172.21.15.46:0/980408118 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8864+0+0 (secure 0 0 0) 0x7f978c085180 con 0x7f9794159e00 2023-10-30T14:28:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.437+0000 7f9799be2640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97780664d0 0x7f9778068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.441+0000 7f9799be2640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97780664d0 0x7f9778068990 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9790005180 tx=0x7f9790007900 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:09.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.441+0000 7f978a7fc640 1 -- 172.21.15.46:0/980408118 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f978c051890 con 0x7f9794159e00 2023-10-30T14:28:09.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:09.601+0000 7f979be46640 1 -- 172.21.15.46:0/980408118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f97940c3fa0 con 0x7f9794159e00 2023-10-30T14:28:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.249+0000 7f978a7fc640 1 -- 172.21.15.46:0/980408118 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v112) v1 ==== 181+0+0 (secure 0 0 0) 0x7f978c054eb0 con 0x7f9794159e00 2023-10-30T14:28:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2023-10-30T14:28:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.253+0000 7f979be46640 1 -- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97780664d0 msgr2=0x7f9778068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.253+0000 7f979be46640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97780664d0 0x7f9778068990 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9790005180 tx=0x7f9790007900 comp rx=0 tx=0).stop 2023-10-30T14:28:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.253+0000 7f979be46640 1 -- 172.21.15.46:0/980408118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9794159e00 msgr2=0x7f979414ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.253+0000 7f979be46640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9794159e00 0x7f979414ef40 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f97940caa10 tx=0x7f978c00f1b0 comp rx=0 tx=0).stop 2023-10-30T14:28:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.253+0000 7f979be46640 1 -- 172.21.15.46:0/980408118 shutdown_connections 2023-10-30T14:28:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.253+0000 7f979be46640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f97780664d0 0x7f9778068990 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.253+0000 7f979be46640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9794159e00 0x7f979414ef40 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.253+0000 7f979be46640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97941536a0 0x7f97941598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:10.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.253+0000 7f979be46640 1 --2- 172.21.15.46:0/980408118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9780003f00 0x7f97941f2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:10.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.253+0000 7f979be46640 1 -- 172.21.15.46:0/980408118 >> 172.21.15.46:0/980408118 conn(0x7f97940ccf80 msgr2=0x7f97940c9790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:10.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.253+0000 7f979be46640 1 -- 172.21.15.46:0/980408118 shutdown_connections 2023-10-30T14:28:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.253+0000 7f979be46640 1 -- 172.21.15.46:0/980408118 wait complete. 2023-10-30T14:28:10.299 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-30T14:28:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.457+0000 7f31ec6ea640 1 Processor -- start 2023-10-30T14:28:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.457+0000 7f31ec6ea640 1 -- start start 2023-10-30T14:28:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.457+0000 7f31ec6ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e414cf40 0x7f31e4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31ec6ea640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e415a270 con 0x7f31e40c8430 2023-10-30T14:28:10.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31ec6ea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e406c7c0 con 0x7f31e415a700 2023-10-30T14:28:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31ec6ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e406a3d0 con 0x7f31e414cf40 2023-10-30T14:28:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31eac87640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f31e40c8430 0x7f31e40c6920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42034/0 (socket says 172.21.15.46:42034) 2023-10-30T14:28:10.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31eac87640 1 -- 172.21.15.46:0/1146386858 learned_addr learned my addr 172.21.15.46:0/1146386858 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-10-30T14:28:10.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31e9c85640 1 --1- 172.21.15.46:0/1146386858 >> v1:172.21.15.154:6789/0 conn(0x7f31e415a700 0x7f31e4156490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.46:40614/0 (socket says 172.21.15.46:40614) 2023-10-30T14:28:10.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31ea486640 1 --2- 172.21.15.46:0/1146386858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e414cf40 0x7f31e4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:10.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31e9484640 1 -- 172.21.15.46:0/1146386858 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677490465 0 0) 0x7f31e406c7c0 con 0x7f31e415a700 2023-10-30T14:28:10.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31e9484640 1 -- 172.21.15.46:0/1146386858 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d0005740 con 0x7f31e415a700 2023-10-30T14:28:10.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31e9484640 1 -- 172.21.15.46:0/1146386858 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713253065 0 0) 0x7f31e415a270 con 0x7f31e40c8430 2023-10-30T14:28:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31ea486640 1 -- 172.21.15.46:0/1146386858 >> v1:172.21.15.154:6789/0 conn(0x7f31e415a700 legacy=0x7f31e4156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31ea486640 1 -- 172.21.15.46:0/1146386858 >> v1:172.21.15.46:6789/0 conn(0x7f31e40c8430 legacy=0x7f31e40c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31ea486640 1 -- 172.21.15.46:0/1146386858 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e4156ed0 con 0x7f31e414cf40 2023-10-30T14:28:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.461+0000 7f31ea486640 1 --2- 172.21.15.46:0/1146386858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e414cf40 0x7f31e4069e90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31e00097f0 tx=0x7f31e001c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3cc1351c6e405db server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31e9484640 1 -- 172.21.15.46:0/1146386858 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e00ab600 con 0x7f31e414cf40 2023-10-30T14:28:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31e9484640 1 -- 172.21.15.46:0/1146386858 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f31e00a7050 con 0x7f31e414cf40 2023-10-30T14:28:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31e9484640 1 -- 172.21.15.46:0/1146386858 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e00ab9a0 con 0x7f31e414cf40 2023-10-30T14:28:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 -- 172.21.15.46:0/1146386858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e414cf40 msgr2=0x7f31e4069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 --2- 172.21.15.46:0/1146386858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e414cf40 0x7f31e4069e90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31e00097f0 tx=0x7f31e001c4c0 comp rx=0 tx=0).stop 2023-10-30T14:28:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 -- 172.21.15.46:0/1146386858 shutdown_connections 2023-10-30T14:28:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 --2- 172.21.15.46:0/1146386858 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e414cf40 0x7f31e4069e90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 -- 172.21.15.46:0/1146386858 >> 172.21.15.46:0/1146386858 conn(0x7f31e40c7910 msgr2=0x7f31e40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 -- 172.21.15.46:0/1146386858 shutdown_connections 2023-10-30T14:28:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 -- 172.21.15.46:0/1146386858 wait complete. 2023-10-30T14:28:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 Processor -- start 2023-10-30T14:28:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 -- start start 2023-10-30T14:28:10.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e40c8430 0x7f31e4152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:10.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e414cf40 0x7f31e4155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:10.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e415a700 0x7f31e41f2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:10.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e41524e0 con 0x7f31e40c8430 2023-10-30T14:28:10.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e4152050 con 0x7f31e415a700 2023-10-30T14:28:10.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ec6ea640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e406a5b0 con 0x7f31e414cf40 2023-10-30T14:28:10.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ea486640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e40c8430 0x7f31e4152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:10.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ea486640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e40c8430 0x7f31e4152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49118/0 (socket says 172.21.15.46:49118) 2023-10-30T14:28:10.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.465+0000 7f31ea486640 1 -- 172.21.15.46:0/31362616 learned_addr learned my addr 172.21.15.46:0/31362616 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-10-30T14:28:10.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31e9c85640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e414cf40 0x7f31e4155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:10.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31eac87640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e415a700 0x7f31e41f2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:10.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31ea486640 1 -- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e414cf40 msgr2=0x7f31e4155ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:10.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31ea486640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e414cf40 0x7f31e4155ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:10.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31ea486640 1 -- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e415a700 msgr2=0x7f31e41f2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:10.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31ea486640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e415a700 0x7f31e41f2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:10.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31ea486640 1 -- 172.21.15.46:0/31362616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e0009420 con 0x7f31e40c8430 2023-10-30T14:28:10.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31ea486640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e40c8430 0x7f31e4152fa0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f31e00ae040 tx=0x7f31e0019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:10.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31daffd640 1 -- 172.21.15.46:0/31362616 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0004020 con 0x7f31e40c8430 2023-10-30T14:28:10.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31daffd640 1 -- 172.21.15.46:0/31362616 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f31e00041a0 con 0x7f31e40c8430 2023-10-30T14:28:10.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31ec6ea640 1 -- 172.21.15.46:0/31362616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e4156680 con 0x7f31e40c8430 2023-10-30T14:28:10.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31daffd640 1 -- 172.21.15.46:0/31362616 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0003620 con 0x7f31e40c8430 2023-10-30T14:28:10.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31ec6ea640 1 -- 172.21.15.46:0/31362616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e4150f20 con 0x7f31e40c8430 2023-10-30T14:28:10.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31ec6ea640 1 -- 172.21.15.46:0/31362616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e404e170 con 0x7f31e40c8430 2023-10-30T14:28:10.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.469+0000 7f31daffd640 1 -- 172.21.15.46:0/31362616 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f31e00b2040 con 0x7f31e40c8430 2023-10-30T14:28:10.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.473+0000 7f31daffd640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31c80664a0 0x7f31c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T14:28:10.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.473+0000 7f31daffd640 1 -- 172.21.15.46:0/31362616 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 8426+0+0 (secure 0 0 0) 0x7f31e011e8b0 con 0x7f31e40c8430 2023-10-30T14:28:10.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.473+0000 7f31e9c85640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31c80664a0 0x7f31c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T14:28:10.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.473+0000 7f31e9c85640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31c80664a0 0x7f31c8068960 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f31e4150b20 tx=0x7f31d4007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T14:28:10.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.477+0000 7f31daffd640 1 -- 172.21.15.46:0/31362616 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31e00eb290 con 0x7f31e40c8430 2023-10-30T14:28:10.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:10.633+0000 7f31ec6ea640 1 -- 172.21.15.46:0/31362616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f31e40b0c20 con 0x7f31e40c8430 2023-10-30T14:28:11.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.285+0000 7f31daffd640 1 -- 172.21.15.46:0/31362616 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v113) v1 ==== 169+0+0 (secure 0 0 0) 0x7f31e00ee8b0 con 0x7f31e40c8430 2023-10-30T14:28:11.291 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2023-10-30T14:28:11.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.289+0000 7f31ec6ea640 1 -- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31c80664a0 msgr2=0x7f31c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.289+0000 7f31ec6ea640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31c80664a0 0x7f31c8068960 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f31e4150b20 tx=0x7f31d4007d40 comp rx=0 tx=0).stop 2023-10-30T14:28:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.293+0000 7f31ec6ea640 1 -- 172.21.15.46:0/31362616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e40c8430 msgr2=0x7f31e4152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T14:28:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.293+0000 7f31ec6ea640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e40c8430 0x7f31e4152fa0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f31e00ae040 tx=0x7f31e0019260 comp rx=0 tx=0).stop 2023-10-30T14:28:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.293+0000 7f31ec6ea640 1 -- 172.21.15.46:0/31362616 shutdown_connections 2023-10-30T14:28:11.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.293+0000 7f31ec6ea640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:6832/313953570,v1:172.21.15.154:6833/313953570] conn(0x7f31c80664a0 0x7f31c8068960 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:11.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.293+0000 7f31ec6ea640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e415a700 0x7f31e41f2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:11.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.293+0000 7f31ec6ea640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e414cf40 0x7f31e4155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:11.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.293+0000 7f31ec6ea640 1 --2- 172.21.15.46:0/31362616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e40c8430 0x7f31e4152fa0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T14:28:11.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.293+0000 7f31ec6ea640 1 -- 172.21.15.46:0/31362616 >> 172.21.15.46:0/31362616 conn(0x7f31e40c7910 msgr2=0x7f31e406aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T14:28:11.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.293+0000 7f31ec6ea640 1 -- 172.21.15.46:0/31362616 shutdown_connections 2023-10-30T14:28:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-10-30T14:28:11.293+0000 7f31ec6ea640 1 -- 172.21.15.46:0/31362616 wait complete. 2023-10-30T14:28:11.339 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-30T14:28:11.339 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T14:28:11.340 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:11.402 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T14:28:11.402 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T14:28:11.403 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:11.424 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T14:28:11.425 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T14:28:11.425 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:11.484 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T14:28:11.484 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T14:28:11.485 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:11.503 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T14:28:11.504 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T14:28:11.504 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:11.518 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:28:11.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:28:11.560 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T14:28:11.560 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T14:28:11.561 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:11.617 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T14:28:11.617 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-30T14:28:11.618 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-30T14:28:11.619 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:11.714 INFO:tasks.ceph.osd.0:Stopped 2023-10-30T14:28:11.714 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-30T14:28:11.714 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:16.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-30T14:28:17.817 INFO:tasks.ceph.osd.1:Stopped 2023-10-30T14:28:17.818 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-30T14:28:17.818 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:17.913 INFO:tasks.ceph.osd.2:Stopped 2023-10-30T14:28:17.914 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-30T14:28:17.914 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:22.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-10-30T14:28:22.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-30T14:28:24.016 INFO:tasks.ceph.osd.3:Stopped 2023-10-30T14:28:24.017 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-30T14:28:24.017 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:28.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-10-30T14:28:28.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-10-30T14:28:30.119 INFO:tasks.ceph.osd.4:Stopped 2023-10-30T14:28:30.120 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-30T14:28:30.120 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:33.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-10-30T14:28:33.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-10-30T14:28:36.222 INFO:tasks.ceph.osd.5:Stopped 2023-10-30T14:28:36.223 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-30T14:28:36.223 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:36.312 INFO:tasks.ceph.osd.6:Stopped 2023-10-30T14:28:36.313 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-30T14:28:36.313 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:39.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-10-30T14:28:39.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-10-30T14:28:41.640 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:28:41.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:28:42.415 INFO:tasks.ceph.osd.7:Stopped 2023-10-30T14:28:42.416 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-30T14:28:42.416 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-30T14:28:42.417 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:42.471 INFO:tasks.ceph.mgr.y:Stopped 2023-10-30T14:28:42.472 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-30T14:28:42.472 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:42.502 INFO:tasks.ceph.mgr.x:Stopped 2023-10-30T14:28:42.502 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-30T14:28:42.503 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T14:28:42.503 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:42.528 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T14:28:42.528 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T14:28:42.529 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:42.566 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T14:28:42.566 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T14:28:42.566 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T14:28:42.630 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T14:28:42.631 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-30T14:28:42.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'Replacing daemon mds' | head -n 1 2023-10-30T14:28:42.738 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi046.front.sepia.ceph.com 2023-10-30T14:28:42.739 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-30T14:28:42.915 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi046.front.sepia.ceph.com 2023-10-30T14:28:42.915 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-30T14:28:43.143 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi046.front.sepia.ceph.com 2023-10-30T14:28:43.144 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-30T14:28:43.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi046.front.sepia.ceph.com 2023-10-30T14:28:43.330 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-30T14:28:43.532 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi154.front.sepia.ceph.com 2023-10-30T14:28:43.532 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-30T14:28:43.986 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi154.front.sepia.ceph.com 2023-10-30T14:28:43.987 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-30T14:28:44.246 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi154.front.sepia.ceph.com 2023-10-30T14:28:44.246 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-30T14:28:44.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-10-30T14:28:44.461 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi154.front.sepia.ceph.com 2023-10-30T14:28:44.461 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-30T14:28:44.655 INFO:tasks.ceph:Archiving mon data... 2023-10-30T14:28:44.657 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045/data/mon.a.tgz 2023-10-30T14:28:44.657 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2023-10-30T14:28:44.675 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.kCdrzjeKIl 2023-10-30T14:28:44.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.kCdrzjeKIl 2023-10-30T14:28:44.845 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.kCdrzjeKIl 2023-10-30T14:28:44.941 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.kCdrzjeKIl is 478KB 2023-10-30T14:28:45.010 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.kCdrzjeKIl 2023-10-30T14:28:45.019 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045/data/mon.b.tgz 2023-10-30T14:28:45.019 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2023-10-30T14:28:45.025 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.XnUak1T8x6 2023-10-30T14:28:45.026 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.XnUak1T8x6 2023-10-30T14:28:45.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.XnUak1T8x6 2023-10-30T14:28:45.242 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.XnUak1T8x6 is 487KB 2023-10-30T14:28:45.304 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.XnUak1T8x6 2023-10-30T14:28:45.309 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045/data/mon.c.tgz 2023-10-30T14:28:45.309 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2023-10-30T14:28:45.354 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.dUaEoyyiKT 2023-10-30T14:28:45.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.dUaEoyyiKT 2023-10-30T14:28:45.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.dUaEoyyiKT 2023-10-30T14:28:45.565 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.dUaEoyyiKT is 487KB 2023-10-30T14:28:45.628 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.dUaEoyyiKT 2023-10-30T14:28:45.633 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-30T14:28:45.634 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T14:28:45.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T14:28:45.694 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-30T14:28:45.694 DEBUG:teuthology.orchestra.run.smithi046:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T14:28:45.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T14:28:45.744 INFO:teuthology.orchestra.run.smithi046.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-30T14:28:45.774 INFO:teuthology.orchestra.run.smithi154.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-30T14:28:45.779 INFO:tasks.ceph:Archiving crash dumps... 2023-10-30T14:28:45.780 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045/remote/smithi046/crash 2023-10-30T14:28:45.781 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T14:28:45.817 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045/remote/smithi154/crash 2023-10-30T14:28:45.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T14:28:45.832 INFO:tasks.ceph:Shutting down logrotate 2023-10-30T14:28:45.832 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:28:45.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T14:28:45.920 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-30T14:28:45.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-30T14:28:45.971 INFO:tasks.ceph:Compressing logs... 2023-10-30T14:28:45.972 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T14:28:45.986 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T14:28:46.011 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46123.log 2023-10-30T14:28:46.012 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66860.log 2023-10-30T14:28:46.013 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52205.log 2023-10-30T14:28:46.013 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46123.log.gz 2023-10-30T14:28:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56860.log 2023-10-30T14:28:46.015 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66860.log.gz 2023-10-30T14:28:46.015 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49105.log 2023-10-30T14:28:46.017 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52205.log.gz 2023-10-30T14:28:46.017 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51275.log 2023-10-30T14:28:46.017 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56860.log.gz 2023-10-30T14:28:46.018 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46745.log 2023-10-30T14:28:46.019 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70389.log 2023-10-30T14:28:46.019 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46745.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51275.log.gz 2023-10-30T14:28:46.019 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70886.log 2023-10-30T14:28:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44371.log 2023-10-30T14:28:46.020 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70389.log.gz 2023-10-30T14:28:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43087.log 2023-10-30T14:28:46.021 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70886.log.gz 2023-10-30T14:28:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61929.log 2023-10-30T14:28:46.023 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58116.log 2023-10-30T14:28:46.024 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64546.log 2023-10-30T14:28:46.024 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61929.log.gz 2023-10-30T14:28:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58116.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.58116.log.gz -5 2023-10-30T14:28:46.026 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43817.log 2023-10-30T14:28:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42930.log 2023-10-30T14:28:46.027 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64546.log.gz 2023-10-30T14:28:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56648.log 2023-10-30T14:28:46.028 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49105.log.gz 2023-10-30T14:28:46.029 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59943.log 2023-10-30T14:28:46.029 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56648.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46745.log.gz 2023-10-30T14:28:46.030 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48147.log 2023-10-30T14:28:46.030 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56648.log.gz 2023-10-30T14:28:46.031 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55679.log 2023-10-30T14:28:46.031 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59943.log.gz 2023-10-30T14:28:46.032 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63807.log 2023-10-30T14:28:46.033 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58630.log 2023-10-30T14:28:46.033 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43817.log.gz 2023-10-30T14:28:46.033 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55679.log.gz 2023-10-30T14:28:46.034 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63807.log: gzip -5 --verbose 88.2% -- /var/log/ceph/ceph-client.admin.58462.log -- replaced with /var/log/ceph/ceph-client.admin.63807.log.gz 2023-10-30T14:28:46.034 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.036 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58630.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58630.log.gz 2023-10-30T14:28:46.036 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62852.log 2023-10-30T14:28:46.037 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44371.log.gz 2023-10-30T14:28:46.037 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69981.log 2023-10-30T14:28:46.037 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58462.log.gz 2023-10-30T14:28:46.038 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62852.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43087.log.gz 2023-10-30T14:28:46.038 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61364.log 2023-10-30T14:28:46.038 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62852.log.gz 2023-10-30T14:28:46.044 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69981.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69981.log.gz 2023-10-30T14:28:46.044 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63885.log 2023-10-30T14:28:46.044 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61364.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61364.log.gz 2023-10-30T14:28:46.045 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55954.log 2023-10-30T14:28:46.045 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48147.log.gz 2023-10-30T14:28:46.045 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63885.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63885.log.gz 2023-10-30T14:28:46.046 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65025.log 2023-10-30T14:28:46.046 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55954.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55954.log.gz 2023-10-30T14:28:46.046 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68173.log 2023-10-30T14:28:46.046 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57711.log 2023-10-30T14:28:46.047 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65025.log.gz 2023-10-30T14:28:46.047 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68173.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68173.log.gzgzip 2023-10-30T14:28:46.047 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68611.log 2023-10-30T14:28:46.048 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57711.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57711.log.gz -5 2023-10-30T14:28:46.048 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54279.log 2023-10-30T14:28:46.048 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42930.log.gz 2023-10-30T14:28:46.050 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38658.log 2023-10-30T14:28:46.050 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38682.log 2023-10-30T14:28:46.050 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38833.log 2023-10-30T14:28:46.051 INFO:teuthology.orchestra.run.smithi154.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38658.log.gz 2023-10-30T14:28:46.051 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38682.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-30T14:28:46.051 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38833.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-30T14:28:46.052 INFO:teuthology.orchestra.run.smithi154.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38682.log.gz 2023-10-30T14:28:46.052 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38754.log 2023-10-30T14:28:46.052 INFO:teuthology.orchestra.run.smithi154.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38833.log.gz 2023-10-30T14:28:46.052 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38730.log 2023-10-30T14:28:46.053 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38754.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-30T14:28:46.053 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38730.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38754.log.gz 2023-10-30T14:28:46.053 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38809.log 2023-10-30T14:28:46.053 INFO:teuthology.orchestra.run.smithi154.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38730.log.gz 2023-10-30T14:28:46.054 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38778.log 2023-10-30T14:28:46.054 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.38809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38706.log 2023-10-30T14:28:46.054 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38778.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-30T14:28:46.055 INFO:teuthology.orchestra.run.smithi154.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38809.log.gz 2023-10-30T14:28:46.055 INFO:teuthology.orchestra.run.smithi154.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38778.log.gz 2023-10-30T14:28:46.055 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38706.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-10-30T14:28:46.055 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mgr.x.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38706.log.gz 2023-10-30T14:28:46.056 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T14:28:46.056 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-30T14:28:46.056 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T14:28:46.056 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-30T14:28:46.059 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph.audit.log: 94.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T14:28:46.059 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-30T14:28:46.059 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mds.d.log: 94.4% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-10-30T14:28:46.060 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-30T14:28:46.064 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mds.b.log: 93.3% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T14:28:46.064 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-30T14:28:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68611.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68611.log.gz 2023-10-30T14:28:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63399.log 2023-10-30T14:28:46.068 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67178.log 2023-10-30T14:28:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63399.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.0.44485.log -- replaced with /var/log/ceph/ceph-client.admin.63399.log.gz 2023-10-30T14:28:46.069 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56999.log 2023-10-30T14:28:46.069 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67178.log.gz 2023-10-30T14:28:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.44485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.44485.log.gz 2023-10-30T14:28:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-30T14:28:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56999.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56999.log.gz 2023-10-30T14:28:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43954.log 2023-10-30T14:28:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61633.log 2023-10-30T14:28:46.071 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51381.log 2023-10-30T14:28:46.072 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43954.log.gz 2023-10-30T14:28:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61633.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.51247.log -- replaced with /var/log/ceph/ceph-client.admin.61633.log.gz 2023-10-30T14:28:46.072 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56620.log 2023-10-30T14:28:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64210.log 2023-10-30T14:28:46.073 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54279.log.gz 2023-10-30T14:28:46.073 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51247.log.gz 2023-10-30T14:28:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56620.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56620.log.gz 2023-10-30T14:28:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65281.log 2023-10-30T14:28:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64493.log 2023-10-30T14:28:46.075 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64210.log.gz 2023-10-30T14:28:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65281.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65281.log.gz 2023-10-30T14:28:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60156.log 2023-10-30T14:28:46.075 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51381.log.gz 2023-10-30T14:28:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46192.log 2023-10-30T14:28:46.076 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64493.log.gz 2023-10-30T14:28:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60156.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60156.log.gzgzip 2023-10-30T14:28:46.076 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59884.log 2023-10-30T14:28:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43009.log 2023-10-30T14:28:46.077 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46192.log.gz 2023-10-30T14:28:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59884.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59884.log.gz 2023-10-30T14:28:46.078 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69400.log 2023-10-30T14:28:46.078 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60528.log 2023-10-30T14:28:46.078 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9%/var/log/ceph/ceph-client.admin.69400.log: -- replaced with /var/log/ceph/ceph-client.admin.43009.log.gz 2023-10-30T14:28:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43876.log 2023-10-30T14:28:46.079 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69400.log.gz 2023-10-30T14:28:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59319.log 2023-10-30T14:28:46.079 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60528.log.gz 2023-10-30T14:28:46.080 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64792.log 2023-10-30T14:28:46.080 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69560.log 2023-10-30T14:28:46.080 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59319.log.gz 2023-10-30T14:28:46.082 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mon.b.log: 94.5% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-30T14:28:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64792.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.64792.log.gz --verbose 2023-10-30T14:28:46.087 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.53151.log 2023-10-30T14:28:46.088 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64896.log 2023-10-30T14:28:46.088 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69560.log.gz 2023-10-30T14:28:46.088 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61522.log 2023-10-30T14:28:46.088 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63155.log 2023-10-30T14:28:46.089 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64896.log.gz 2023-10-30T14:28:46.089 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48445.log 2023-10-30T14:28:46.089 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61522.log.gz 2023-10-30T14:28:46.090 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59290.log 2023-10-30T14:28:46.090 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63155.log.gz 2023-10-30T14:28:46.090 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49510.log 2023-10-30T14:28:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59153.log 2023-10-30T14:28:46.091 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59290.log.gz 2023-10-30T14:28:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51570.log 2023-10-30T14:28:46.091 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59153.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43876.log.gz 2023-10-30T14:28:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61988.log 2023-10-30T14:28:46.092 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53151.log.gz 2023-10-30T14:28:46.092 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48445.log.gz 87.8% 2023-10-30T14:28:46.093 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59153.log.gz 2023-10-30T14:28:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65426.log 2023-10-30T14:28:46.093 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42956.log 2023-10-30T14:28:46.093 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61988.log.gz 2023-10-30T14:28:46.094 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64298.log 2023-10-30T14:28:46.094 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65426.log.gz 2023-10-30T14:28:46.094 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67637.log 2023-10-30T14:28:46.095 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42956.log.gz 2023-10-30T14:28:46.095 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51622.log 2023-10-30T14:28:46.095 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64298.log.gz 2023-10-30T14:28:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69182.log 2023-10-30T14:28:46.096 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67637.log.gz 2023-10-30T14:28:46.096 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55598.log 2023-10-30T14:28:46.096 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51622.log.gz 2023-10-30T14:28:46.097 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49429.log 2023-10-30T14:28:46.097 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69182.log.gz 2023-10-30T14:28:46.097 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49510.log.gz 2023-10-30T14:28:46.098 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55598.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55598.log.gz 2023-10-30T14:28:46.098 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44776.log 2023-10-30T14:28:46.098 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51301.log 2023-10-30T14:28:46.104 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61849.log 2023-10-30T14:28:46.105 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44776.log.gz 2023-10-30T14:28:46.105 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61309.log 2023-10-30T14:28:46.106 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51301.log.gz 2023-10-30T14:28:46.106 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51570.log.gz 2023-10-30T14:28:46.106 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63689.log 2023-10-30T14:28:46.106 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61849.log.gz 2023-10-30T14:28:46.107 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48685.log 2023-10-30T14:28:46.107 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61309.log.gz 2023-10-30T14:28:46.107 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63689.log: gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.63689.log.gz 2023-10-30T14:28:46.108 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49264.log 2023-10-30T14:28:46.108 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55179.log 2023-10-30T14:28:46.108 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49429.log.gz 2023-10-30T14:28:46.109 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48685.log.gz 2023-10-30T14:28:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62175.log 2023-10-30T14:28:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57508.log 2023-10-30T14:28:46.109 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56752.log 2023-10-30T14:28:46.110 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62175.log.gz 2023-10-30T14:28:46.110 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50359.log 2023-10-30T14:28:46.110 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57508.log.gz 2023-10-30T14:28:46.111 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56752.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56752.log.gz 2023-10-30T14:28:46.111 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57929.log 2023-10-30T14:28:46.111 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60078.log 2023-10-30T14:28:46.111 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57929.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-30T14:28:46.112 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57929.log.gz 2023-10-30T14:28:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60078.log: gzip -5 --verbose -- 89.1% /var/log/ceph/ceph-client.admin.51832.log -- replaced with /var/log/ceph/ceph-client.admin.60078.log.gz 2023-10-30T14:28:46.112 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60826.log 2023-10-30T14:28:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58169.log 2023-10-30T14:28:46.113 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51832.log.gz 2023-10-30T14:28:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64818.log 2023-10-30T14:28:46.114 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60826.log.gz 2023-10-30T14:28:46.120 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66484.log 2023-10-30T14:28:46.120 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58169.log.gz 2023-10-30T14:28:46.121 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64818.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64818.log.gz 2023-10-30T14:28:46.121 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53883.log 2023-10-30T14:28:46.121 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48659.log 2023-10-30T14:28:46.121 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66484.log.gz 2023-10-30T14:28:46.122 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59530.log 2023-10-30T14:28:46.122 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53883.log.gz 2023-10-30T14:28:46.122 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65255.log 2023-10-30T14:28:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48016.log 2023-10-30T14:28:46.123 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59530.log.gz 2023-10-30T14:28:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65255.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49264.log.gz 2023-10-30T14:28:46.124 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose 87.7% -- -- replaced with /var/log/ceph/ceph-client.admin.65255.log.gz /var/log/ceph/ceph-client.admin.64978.log 2023-10-30T14:28:46.124 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.124 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68457.log 2023-10-30T14:28:46.124 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47085.log 2023-10-30T14:28:46.125 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64978.log.gz 2023-10-30T14:28:46.125 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48659.log.gz 2023-10-30T14:28:46.125 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50359.log.gz 2023-10-30T14:28:46.126 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59373.log 2023-10-30T14:28:46.126 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68457.log.gz 2023-10-30T14:28:46.126 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47085.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48016.log.gz 2023-10-30T14:28:46.127 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61767.log 2023-10-30T14:28:46.127 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47085.log.gz 2023-10-30T14:28:46.127 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62279.log 2023-10-30T14:28:46.127 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59373.log.gz 2023-10-30T14:28:46.128 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61767.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61767.log.gz 2023-10-30T14:28:46.128 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44136.log 2023-10-30T14:28:46.128 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59049.log 2023-10-30T14:28:46.129 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62279.log.gz 2023-10-30T14:28:46.129 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63859.log 2023-10-30T14:28:46.129 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44136.log.gz 2023-10-30T14:28:46.130 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63233.log 2023-10-30T14:28:46.130 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59049.log.gz 2023-10-30T14:28:46.136 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70912.log 2023-10-30T14:28:46.136 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63859.log.gz 2023-10-30T14:28:46.137 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52421.log 2023-10-30T14:28:46.137 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63233.log.gz 2023-10-30T14:28:46.137 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58865.log 2023-10-30T14:28:46.138 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70912.log.gz 2023-10-30T14:28:46.138 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70441.log 2023-10-30T14:28:46.138 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52421.log.gz 2023-10-30T14:28:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61222.log 2023-10-30T14:28:46.139 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58865.log.gz 2023-10-30T14:28:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62528.log 2023-10-30T14:28:46.140 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70441.log.gz 2023-10-30T14:28:46.140 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63181.log 2023-10-30T14:28:46.140 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61222.log.gz 2023-10-30T14:28:46.140 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53620.log 2023-10-30T14:28:46.141 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.62528.log.gz 2023-10-30T14:28:46.141 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63181.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63181.log.gz 2023-10-30T14:28:46.141 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48497.log 2023-10-30T14:28:46.142 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56700.log 2023-10-30T14:28:46.142 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48289.log 2023-10-30T14:28:46.142 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55179.log.gz 2023-10-30T14:28:46.143 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48497.log.gz 2023-10-30T14:28:46.143 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67981.log 2023-10-30T14:28:46.143 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56700.log.gz 2023-10-30T14:28:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54550.log 2023-10-30T14:28:46.144 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48289.log.gz 2023-10-30T14:28:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67981.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67981.log.gz 2023-10-30T14:28:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45652.log 2023-10-30T14:28:46.145 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53620.log.gz 2023-10-30T14:28:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60262.log 2023-10-30T14:28:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66076.log 2023-10-30T14:28:46.146 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54550.log.gz 2023-10-30T14:28:46.151 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63558.log 2023-10-30T14:28:46.151 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60262.log.gz 2023-10-30T14:28:46.151 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50973.log 2023-10-30T14:28:46.152 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66076.log.gz 2023-10-30T14:28:46.152 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59238.log 2023-10-30T14:28:46.152 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.63558.log.gz 2023-10-30T14:28:46.153 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45995.log 2023-10-30T14:28:46.153 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55572.log 2023-10-30T14:28:46.153 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59238.log.gz 2023-10-30T14:28:46.153 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45995.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45652.log.gz 2023-10-30T14:28:46.154 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56144.log 2023-10-30T14:28:46.154 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50973.log.gz 2023-10-30T14:28:46.154 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54121.log 2023-10-30T14:28:46.155 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55572.log.gz 2023-10-30T14:28:46.155 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56144.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56144.log.gz 2023-10-30T14:28:46.155 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43623.log 2023-10-30T14:28:46.155 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62043.log 2023-10-30T14:28:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43411.log 2023-10-30T14:28:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50548.log 2023-10-30T14:28:46.156 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62043.log.gz 2023-10-30T14:28:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43201.log 2023-10-30T14:28:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67689.log 2023-10-30T14:28:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65584.log 2023-10-30T14:28:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67663.log 2023-10-30T14:28:46.158 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67689.log.gz 2023-10-30T14:28:46.162 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57025.log 2023-10-30T14:28:46.162 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65584.log.gz 2023-10-30T14:28:46.163 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66346.log 2023-10-30T14:28:46.163 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67663.log.gz 2023-10-30T14:28:46.163 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57025.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57025.log.gz 2023-10-30T14:28:46.163 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65368.log 2023-10-30T14:28:46.164 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66346.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66346.log.gz -5 2023-10-30T14:28:46.164 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.68663.log 2023-10-30T14:28:46.164 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54436.log 2023-10-30T14:28:46.165 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65368.log.gz 2023-10-30T14:28:46.165 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68663.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45995.log.gz 2023-10-30T14:28:46.165 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57822.log 2023-10-30T14:28:46.165 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.68663.log.gz 2023-10-30T14:28:46.166 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54121.log.gz 2023-10-30T14:28:46.166 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64053.log 2023-10-30T14:28:46.166 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68303.log 2023-10-30T14:28:46.167 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57822.log.gz 2023-10-30T14:28:46.167 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68935.log 2023-10-30T14:28:46.167 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64053.log.gz 2023-10-30T14:28:46.168 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60104.log 2023-10-30T14:28:46.168 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68303.log.gz 2023-10-30T14:28:46.168 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68935.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68935.log.gz 2023-10-30T14:28:46.168 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51911.log 2023-10-30T14:28:46.169 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60104.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.60104.log.gz 2023-10-30T14:28:46.169 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67611.log 2023-10-30T14:28:46.173 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60450.log 2023-10-30T14:28:46.174 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64441.log 2023-10-30T14:28:46.174 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67611.log.gz 2023-10-30T14:28:46.174 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44646.log 2023-10-30T14:28:46.175 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60450.log.gz 2023-10-30T14:28:46.175 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64441.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64441.log.gz -5 2023-10-30T14:28:46.175 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55980.log 2023-10-30T14:28:46.175 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52930.log 2023-10-30T14:28:46.176 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55980.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43623.log.gz 2023-10-30T14:28:46.176 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64948.log 2023-10-30T14:28:46.176 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55980.log.gz 2023-10-30T14:28:46.177 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43411.log.gz 2023-10-30T14:28:46.177 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44032.log 2023-10-30T14:28:46.177 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50548.log.gz 2023-10-30T14:28:46.177 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64948.log: gzip -5 --verbose -- 89.3% /var/log/ceph/ceph-client.admin.61496.log -- replaced with /var/log/ceph/ceph-client.admin.64948.log.gz 2023-10-30T14:28:46.178 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.178 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57402.log 2023-10-30T14:28:46.179 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53068.log 2023-10-30T14:28:46.179 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61496.log.gz 2023-10-30T14:28:46.179 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57402.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57402.log.gz 2023-10-30T14:28:46.179 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53094.log 2023-10-30T14:28:46.180 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58971.log 2023-10-30T14:28:46.184 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58708.log 2023-10-30T14:28:46.184 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64415.log 2023-10-30T14:28:46.184 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58971.log.gz 2023-10-30T14:28:46.185 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44932.log 2023-10-30T14:28:46.185 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58708.log.gz 2023-10-30T14:28:46.185 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64415.log: gzip -5 87.9% --verbose -- replaced with /var/log/ceph/ceph-client.admin.64415.log.gz 2023-10-30T14:28:46.186 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.68121.log 2023-10-30T14:28:46.186 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68883.log 2023-10-30T14:28:46.186 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61418.log 2023-10-30T14:28:46.186 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68121.log.gz 2023-10-30T14:28:46.187 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66286.log 2023-10-30T14:28:46.187 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68883.log.gz 2023-10-30T14:28:46.187 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61418.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43201.log.gz 2023-10-30T14:28:46.188 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67478.log 2023-10-30T14:28:46.188 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61418.log.gz 2023-10-30T14:28:46.188 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54436.log.gz 2023-10-30T14:28:46.188 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51858.log 2023-10-30T14:28:46.189 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66286.log.gz 2023-10-30T14:28:46.189 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51911.log.gz 2023-10-30T14:28:46.189 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44032.log.gz 2023-10-30T14:28:46.190 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53068.log.gz 2023-10-30T14:28:46.190 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53094.log.gz 2023-10-30T14:28:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67478.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44932.log.gz 2023-10-30T14:28:46.191 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55902.log 2023-10-30T14:28:46.191 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67478.log.gz 2023-10-30T14:28:46.194 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68355.log 2023-10-30T14:28:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63729.log 2023-10-30T14:28:46.195 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55902.log.gz 2023-10-30T14:28:46.195 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58734.log 2023-10-30T14:28:46.196 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68355.log.gz 2023-10-30T14:28:46.196 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48042.log 2023-10-30T14:28:46.196 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63729.log.gz 2023-10-30T14:28:46.197 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58734.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58734.log.gz 2023-10-30T14:28:46.197 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45133.log 2023-10-30T14:28:46.197 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68066.log 2023-10-30T14:28:46.197 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48042.log.gz 2023-10-30T14:28:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47964.log 2023-10-30T14:28:46.198 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52930.log.gz 2023-10-30T14:28:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65751.log 2023-10-30T14:28:46.199 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68066.log.gz 2023-10-30T14:28:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43633.log 2023-10-30T14:28:46.199 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47964.log.gz 2023-10-30T14:28:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46528.log 2023-10-30T14:28:46.200 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65751.log.gz 2023-10-30T14:28:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55127.log 2023-10-30T14:28:46.204 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43633.log.gz 2023-10-30T14:28:46.204 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61715.log 2023-10-30T14:28:46.204 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55127.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51858.log.gz 2023-10-30T14:28:46.204 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46528.log.gz 2023-10-30T14:28:46.205 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43206.log 2023-10-30T14:28:46.205 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64738.log 2023-10-30T14:28:46.205 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61715.log.gz 2023-10-30T14:28:46.206 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43206.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-30T14:28:46.206 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64738.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43206.log.gz 2023-10-30T14:28:46.206 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55928.log 2023-10-30T14:28:46.206 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64738.log.gz 2023-10-30T14:28:46.207 INFO:teuthology.orchestra.run.smithi046.stderr: 95.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-30T14:28:46.207 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.2.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44646.log.gz 2023-10-30T14:28:46.207 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66993.log 2023-10-30T14:28:46.208 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48921.log 2023-10-30T14:28:46.208 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55928.log.gz 2023-10-30T14:28:46.208 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53463.log 2023-10-30T14:28:46.209 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66993.log.gz 2023-10-30T14:28:46.211 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63505.log 2023-10-30T14:28:46.212 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51596.log 2023-10-30T14:28:46.212 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55127.log.gz 2023-10-30T14:28:46.212 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57051.log 2023-10-30T14:28:46.212 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63505.log.gz 2023-10-30T14:28:46.213 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53463.log.gz 2023-10-30T14:28:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68277.log 2023-10-30T14:28:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44316.log 2023-10-30T14:28:46.214 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57051.log.gz 2023-10-30T14:28:46.214 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63425.log 2023-10-30T14:28:46.214 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68277.log.gz 2023-10-30T14:28:46.214 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66232.log 2023-10-30T14:28:46.215 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51596.log.gz 2023-10-30T14:28:46.215 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44802.log 2023-10-30T14:28:46.215 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63425.log.gz 2023-10-30T14:28:46.218 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66232.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.44316.log.gz 2023-10-30T14:28:46.218 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68804.log 2023-10-30T14:28:46.218 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66232.log.gz 2023-10-30T14:28:46.219 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64631.log 2023-10-30T14:28:46.219 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44802.log.gz 2023-10-30T14:28:46.219 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59974.log 2023-10-30T14:28:46.219 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68804.log.gz 2023-10-30T14:28:46.220 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45133.log.gz 2023-10-30T14:28:46.220 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64631.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.64631.log.gz 2023-10-30T14:28:46.220 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57131.log 2023-10-30T14:28:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55040.log 2023-10-30T14:28:46.221 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59974.log.gz 2023-10-30T14:28:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53936.log 2023-10-30T14:28:46.221 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57131.log.gz 2023-10-30T14:28:46.224 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44006.log 2023-10-30T14:28:46.224 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48921.log.gz 2023-10-30T14:28:46.224 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55040.log.gz 2023-10-30T14:28:46.224 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50658.log 2023-10-30T14:28:46.225 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53936.log.gz 2023-10-30T14:28:46.225 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66782.log 2023-10-30T14:28:46.225 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50658.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44006.log.gz 2023-10-30T14:28:46.226 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43414.log 2023-10-30T14:28:46.226 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50658.log.gz 2023-10-30T14:28:46.226 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48315.log 2023-10-30T14:28:46.227 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66782.log.gz 2023-10-30T14:28:46.227 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70085.log 2023-10-30T14:28:46.227 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43414.log.gz 2023-10-30T14:28:46.229 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48315.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48315.log.gz -5 2023-10-30T14:28:46.229 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48605.log 2023-10-30T14:28:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62121.log 2023-10-30T14:28:46.230 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70085.log.gz 2023-10-30T14:28:46.230 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60691.log 2023-10-30T14:28:46.231 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55653.log 2023-10-30T14:28:46.231 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62121.log.gz 2023-10-30T14:28:46.231 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48605.log.gz 2023-10-30T14:28:46.232 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57903.log 2023-10-30T14:28:46.232 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60691.log.gz 2023-10-30T14:28:46.232 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62665.log 2023-10-30T14:28:46.232 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55653.log.gz 2023-10-30T14:28:46.234 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57903.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.57903.log.gz --verbose 2023-10-30T14:28:46.235 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.43635.log 2023-10-30T14:28:46.235 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45268.log 2023-10-30T14:28:46.235 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62665.log.gz 2023-10-30T14:28:46.236 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70129.log 2023-10-30T14:28:46.236 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43635.log.gz 2023-10-30T14:28:46.236 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43631.log 2023-10-30T14:28:46.236 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45268.log.gz 2023-10-30T14:28:46.237 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51702.log 2023-10-30T14:28:46.237 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70129.log.gz 2023-10-30T14:28:46.239 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62961.log 2023-10-30T14:28:46.239 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66206.log 2023-10-30T14:28:46.240 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62961.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62961.log.gz 2023-10-30T14:28:46.240 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62613.log 2023-10-30T14:28:46.240 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66753.log 2023-10-30T14:28:46.240 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66206.log.gz 2023-10-30T14:28:46.241 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62613.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62613.log.gz -5 2023-10-30T14:28:46.241 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-30T14:28:46.243 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61392.log 2023-10-30T14:28:46.243 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66753.log.gz 2023-10-30T14:28:46.243 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57298.log 2023-10-30T14:28:46.244 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61392.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61392.log.gz 2023-10-30T14:28:46.244 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63610.log 2023-10-30T14:28:46.244 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60320.log 2023-10-30T14:28:46.244 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57298.log.gz 2023-10-30T14:28:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63610.log: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.63610.log.gz 2023-10-30T14:28:46.247 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60000.log 2023-10-30T14:28:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51433.log 2023-10-30T14:28:46.247 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60320.log.gz 2023-10-30T14:28:46.248 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60000.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43631.log.gz 2023-10-30T14:28:46.248 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56947.log 2023-10-30T14:28:46.248 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60000.log.gz 2023-10-30T14:28:46.248 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51702.log.gz 2023-10-30T14:28:46.249 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59803.log 2023-10-30T14:28:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56947.log: gzip -5 --verbose 88.8% -- -- replaced with /var/log/ceph/ceph-client.admin.56947.log.gz /var/log/ceph/ceph-client.admin.61548.log 2023-10-30T14:28:46.251 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59803.log: gzip -5 --verbose 88.0% -- -- replaced with /var/log/ceph/ceph-client.admin.59803.log.gz /var/log/ceph/ceph-client.admin.48180.log 2023-10-30T14:28:46.251 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60398.log 88.1% 2023-10-30T14:28:46.252 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61548.log.gz 2023-10-30T14:28:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48261.log 2023-10-30T14:28:46.254 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70834.log 2023-10-30T14:28:46.254 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60398.log.gz 2023-10-30T14:28:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66590.log 2023-10-30T14:28:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70834.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.70834.log.gz 2023-10-30T14:28:46.255 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43902.log 2023-10-30T14:28:46.256 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66590.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51433.log.gz 2023-10-30T14:28:46.256 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62475.log 2023-10-30T14:28:46.256 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66590.log.gz 2023-10-30T14:28:46.258 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43902.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48180.log.gz 2023-10-30T14:28:46.258 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57763.log 2023-10-30T14:28:46.259 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48261.log.gz 2023-10-30T14:28:46.259 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62930.log 2023-10-30T14:28:46.259 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62475.log.gz 2023-10-30T14:28:46.259 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57763.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57763.log.gz 2023-10-30T14:28:46.260 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43902.log.gz 2023-10-30T14:28:46.260 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43419.log 2023-10-30T14:28:46.261 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62930.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62930.log.gz 2023-10-30T14:28:46.262 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44828.log 2023-10-30T14:28:46.262 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59451.log 2023-10-30T14:28:46.262 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69013.log 2023-10-30T14:28:46.263 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48817.log 2023-10-30T14:28:46.264 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59451.log.gz 2023-10-30T14:28:46.264 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63009.log 2023-10-30T14:28:46.264 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69013.log.gz 2023-10-30T14:28:46.266 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52179.log 2023-10-30T14:28:46.266 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43419.log.gz 2023-10-30T14:28:46.266 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63009.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44828.log.gz 2023-10-30T14:28:46.267 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63009.log.gz 2023-10-30T14:28:46.267 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45104.log 2023-10-30T14:28:46.267 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48817.log.gz 2023-10-30T14:28:46.269 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65935.log 2023-10-30T14:28:46.269 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59101.log 2023-10-30T14:28:46.269 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65935.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65935.log.gz 2023-10-30T14:28:46.270 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57430.log 2023-10-30T14:28:46.270 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52179.log.gz 2023-10-30T14:28:46.271 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59101.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59101.log.gz 2023-10-30T14:28:46.271 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63344.log 2023-10-30T14:28:46.272 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67718.log 2023-10-30T14:28:46.272 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57430.log.gz 2023-10-30T14:28:46.272 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64870.log 2023-10-30T14:28:46.273 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63344.log.gz 2023-10-30T14:28:46.274 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65505.log 2023-10-30T14:28:46.275 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67718.log.gz 2023-10-30T14:28:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64870.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64870.log.gz 2023-10-30T14:28:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63373.log 2023-10-30T14:28:46.276 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65505.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65505.log.gz 2023-10-30T14:28:46.276 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67556.log 2023-10-30T14:28:46.277 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63373.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63373.log.gz 2023-10-30T14:28:46.278 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69638.log 2023-10-30T14:28:46.278 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50684.log 2023-10-30T14:28:46.278 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67556.log.gz 2023-10-30T14:28:46.279 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43630.log 2023-10-30T14:28:46.279 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69638.log.gz 2023-10-30T14:28:46.280 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42483.log 2023-10-30T14:28:46.280 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62746.log 2023-10-30T14:28:46.280 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51113.log 2023-10-30T14:28:46.282 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68007.log 2023-10-30T14:28:46.282 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62746.log.gz 2023-10-30T14:28:46.282 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62355.log 2023-10-30T14:28:46.284 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68007.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68007.log.gz 2023-10-30T14:28:46.284 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55338.log 2023-10-30T14:28:46.284 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50684.log.gz 2023-10-30T14:28:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56831.log 2023-10-30T14:28:46.285 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62355.log.gz 2023-10-30T14:28:46.285 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43630.log.gz 2023-10-30T14:28:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60288.log 2023-10-30T14:28:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44672.log 2023-10-30T14:28:46.287 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56831.log.gz 2023-10-30T14:28:46.288 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68749.log 2023-10-30T14:28:46.288 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42483.log.gz 2023-10-30T14:28:46.288 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.60288.log.gz 2023-10-30T14:28:46.290 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44672.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51113.log.gz 2023-10-30T14:28:46.290 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55338.log.gz 2023-10-30T14:28:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67955.log 2023-10-30T14:28:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68749.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68749.log.gz 2023-10-30T14:28:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55843.log 2023-10-30T14:28:46.292 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44672.log.gz 2023-10-30T14:28:46.292 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45104.log.gz 2023-10-30T14:28:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70704.log 2023-10-30T14:28:46.293 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67955.log.gz 2023-10-30T14:28:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44698.log 2023-10-30T14:28:46.295 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55843.log.gz 2023-10-30T14:28:46.295 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70704.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70704.log.gz 2023-10-30T14:28:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68533.log 2023-10-30T14:28:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46582.log 2023-10-30T14:28:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49758.log 2023-10-30T14:28:46.297 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44698.log.gz 2023-10-30T14:28:46.297 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68533.log.gz 2023-10-30T14:28:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66564.log 2023-10-30T14:28:46.299 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43216.log 2023-10-30T14:28:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66642.log 2023-10-30T14:28:46.300 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66564.log.gz 2023-10-30T14:28:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43216.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46582.log.gz 2023-10-30T14:28:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43165.log 2023-10-30T14:28:46.301 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63259.log 2023-10-30T14:28:46.301 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.66642.log.gz 2023-10-30T14:28:46.304 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59831.log 2023-10-30T14:28:46.304 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49758.log.gz 2023-10-30T14:28:46.304 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43216.log.gz 2023-10-30T14:28:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63259.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43165.log.gz 2023-10-30T14:28:46.305 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63259.log.gz 2023-10-30T14:28:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64344.log 2023-10-30T14:28:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61035.log 2023-10-30T14:28:46.306 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59831.log.gz 2023-10-30T14:28:46.306 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64344.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64344.log.gz 2023-10-30T14:28:46.306 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60606.log 2023-10-30T14:28:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61035.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61035.log.gz 2023-10-30T14:28:46.309 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58035.log 2023-10-30T14:28:46.309 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69208.log 2023-10-30T14:28:46.309 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60606.log.gz 2023-10-30T14:28:46.309 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47010.log 2023-10-30T14:28:46.310 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58035.log.gz 2023-10-30T14:28:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66020.log 2023-10-30T14:28:46.310 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69208.log.gz 2023-10-30T14:28:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53647.log 2023-10-30T14:28:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68909.log 2023-10-30T14:28:46.312 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66020.log.gz 2023-10-30T14:28:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50710.log 2023-10-30T14:28:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55014.log 2023-10-30T14:28:46.313 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68909.log.gz 2023-10-30T14:28:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51676.log 2023-10-30T14:28:46.314 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52448.log 2023-10-30T14:28:46.315 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43113.log 2023-10-30T14:28:46.315 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65532.log 2023-10-30T14:28:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42878.log 2023-10-30T14:28:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67452.log 2023-10-30T14:28:46.317 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65532.log.gz 2023-10-30T14:28:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49079.log 2023-10-30T14:28:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61962.log 2023-10-30T14:28:46.319 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67452.log.gz 2023-10-30T14:28:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56321.log 2023-10-30T14:28:46.320 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61962.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61962.log.gz 2023-10-30T14:28:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60905.log 2023-10-30T14:28:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43407.log 2023-10-30T14:28:46.322 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47010.log.gz 2023-10-30T14:28:46.322 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56321.log.gz 2023-10-30T14:28:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60905.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53647.log.gz 2023-10-30T14:28:46.324 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60905.log.gz 2023-10-30T14:28:46.324 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66727.log 2023-10-30T14:28:46.324 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66834.log 2023-10-30T14:28:46.326 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59582.log 2023-10-30T14:28:46.326 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66727.log.gz 2023-10-30T14:28:46.326 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66102.log 2023-10-30T14:28:46.327 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66834.log.gz 2023-10-30T14:28:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58495.log 2023-10-30T14:28:46.328 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59582.log.gz 2023-10-30T14:28:46.329 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44542.log 2023-10-30T14:28:46.329 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.66102.log.gz 2023-10-30T14:28:46.329 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68689.log 2023-10-30T14:28:46.330 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58495.log.gz 2023-10-30T14:28:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57659.log 2023-10-30T14:28:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56086.log 2023-10-30T14:28:46.332 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68689.log.gz 2023-10-30T14:28:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64657.log 2023-10-30T14:28:46.332 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57659.log.gz 2023-10-30T14:28:46.334 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56086.log: gzip -5 --verbose -- 87.8% /var/log/ceph/ceph-client.admin.64079.log -- replaced with /var/log/ceph/ceph-client.admin.56086.log.gz 2023-10-30T14:28:46.334 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.335 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64657.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64657.log.gz 2023-10-30T14:28:46.335 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56295.log 2023-10-30T14:28:46.335 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50710.log.gz 2023-10-30T14:28:46.335 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55014.log.gz 2023-10-30T14:28:46.336 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51676.log.gz 2023-10-30T14:28:46.336 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52448.log.gz 2023-10-30T14:28:46.336 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64079.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42878.log.gz 2023-10-30T14:28:46.336 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49079.log.gz 2023-10-30T14:28:46.338 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43113.log.gz 2023-10-30T14:28:46.339 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43407.log.gz 2023-10-30T14:28:46.339 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56973.log 2023-10-30T14:28:46.339 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64079.log.gz 2023-10-30T14:28:46.339 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44542.log.gz 2023-10-30T14:28:46.340 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48068.log 2023-10-30T14:28:46.340 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56295.log.gz 2023-10-30T14:28:46.340 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56973.log: gzip -5 89.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.56973.log.gz -- 2023-10-30T14:28:46.341 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.45242.log 2023-10-30T14:28:46.341 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65479.log 2023-10-30T14:28:46.341 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48068.log.gz 2023-10-30T14:28:46.343 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54988.log 88.3% 2023-10-30T14:28:46.343 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45242.log.gz 2023-10-30T14:28:46.344 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67851.log 2023-10-30T14:28:46.344 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65479.log.gz 2023-10-30T14:28:46.344 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61741.log 2023-10-30T14:28:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52017.log 2023-10-30T14:28:46.345 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67851.log.gz 2023-10-30T14:28:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61741.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61741.log.gz 2023-10-30T14:28:46.346 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58252.log 2023-10-30T14:28:46.347 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43035.log 2023-10-30T14:28:46.348 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59917.log 2023-10-30T14:28:46.348 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58252.log.gz 2023-10-30T14:28:46.348 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59075.log 2023-10-30T14:28:46.349 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43421.log 2023-10-30T14:28:46.349 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59917.log.gz 2023-10-30T14:28:46.349 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52635.log 2023-10-30T14:28:46.349 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59075.log.gz 2023-10-30T14:28:46.351 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63999.log 2023-10-30T14:28:46.351 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51487.log 2023-10-30T14:28:46.351 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44058.log 2023-10-30T14:28:46.352 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63999.log.gz 2023-10-30T14:28:46.352 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50385.log 2023-10-30T14:28:46.353 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59179.log 2023-10-30T14:28:46.354 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50385.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54988.log.gz 2023-10-30T14:28:46.354 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-30T14:28:46.355 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59179.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52635.log.gz 2023-10-30T14:28:46.355 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70007.log 2023-10-30T14:28:46.356 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59179.log.gz 2023-10-30T14:28:46.356 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52017.log.gz 2023-10-30T14:28:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65229.log 2023-10-30T14:28:46.357 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48947.log 2023-10-30T14:28:46.358 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70007.log.gz 2023-10-30T14:28:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65229.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.65229.log.gz 2023-10-30T14:28:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48550.log 2023-10-30T14:28:46.359 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46023.log 2023-10-30T14:28:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67310.log 2023-10-30T14:28:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62397.log 2023-10-30T14:28:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63918.log 2023-10-30T14:28:46.362 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67310.log.gz 2023-10-30T14:28:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49344.log 2023-10-30T14:28:46.363 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62397.log.gz 2023-10-30T14:28:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62639.log 2023-10-30T14:28:46.364 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63918.log.gz 2023-10-30T14:28:46.365 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48791.log 2023-10-30T14:28:46.366 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43035.log.gz 2023-10-30T14:28:46.366 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67338.log 2023-10-30T14:28:46.366 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62639.log.gz 2023-10-30T14:28:46.367 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43421.log.gz 2023-10-30T14:28:46.369 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48791.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44058.log.gz 2023-10-30T14:28:46.369 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51487.log.gz 2023-10-30T14:28:46.369 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50385.log.gz 2023-10-30T14:28:46.370 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50921.log 2023-10-30T14:28:46.370 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48947.log.gz 2023-10-30T14:28:46.370 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48550.log.gz 2023-10-30T14:28:46.371 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46023.log.gz 2023-10-30T14:28:46.371 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49344.log.gz 2023-10-30T14:28:46.371 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67338.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48791.log.gz 2023-10-30T14:28:46.371 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58787.log 2023-10-30T14:28:46.372 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.67338.log.gz 2023-10-30T14:28:46.372 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58410.log 2023-10-30T14:28:46.374 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50921.log.gz 2023-10-30T14:28:46.375 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48471.log 2023-10-30T14:28:46.375 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58787.log.gz 2023-10-30T14:28:46.375 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58410.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58410.log.gz 2023-10-30T14:28:46.376 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49370.log 2023-10-30T14:28:46.376 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50895.log 2023-10-30T14:28:46.376 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48471.log.gz 2023-10-30T14:28:46.376 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48973.log 2023-10-30T14:28:46.377 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48869.log 2023-10-30T14:28:46.377 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49370.log.gz 2023-10-30T14:28:46.379 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55705.log 2023-10-30T14:28:46.379 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50895.log.gz 2023-10-30T14:28:46.379 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52503.log 2023-10-30T14:28:46.380 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50865.log 2023-10-30T14:28:46.380 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55705.log.gz 2023-10-30T14:28:46.380 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67530.log 2023-10-30T14:28:46.380 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50865.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48973.log.gz 2023-10-30T14:28:46.381 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44215.log 2023-10-30T14:28:46.381 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48869.log.gz 2023-10-30T14:28:46.383 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67530.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52503.log.gz 2023-10-30T14:28:46.383 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67530.log.gz 2023-10-30T14:28:46.384 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70758.log 2023-10-30T14:28:46.384 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50865.log.gz 2023-10-30T14:28:46.384 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60502.log 2023-10-30T14:28:46.384 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67284.log 2023-10-30T14:28:46.385 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70758.log.gz 2023-10-30T14:28:46.385 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53831.log 2023-10-30T14:28:46.385 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60502.log.gz 2023-10-30T14:28:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50437.log 2023-10-30T14:28:46.387 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67284.log.gz 2023-10-30T14:28:46.388 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49001.log 2023-10-30T14:28:46.388 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.44215.log.gz 2023-10-30T14:28:46.388 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56425.log 2023-10-30T14:28:46.389 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59751.log 2023-10-30T14:28:46.390 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63755.log 2023-10-30T14:28:46.391 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56425.log.gz 2023-10-30T14:28:46.391 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59751.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53831.log.gz 2023-10-30T14:28:46.391 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54469.log 2023-10-30T14:28:46.392 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59751.log.gz 2023-10-30T14:28:46.392 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50437.log.gz 2023-10-30T14:28:46.392 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55627.log 2023-10-30T14:28:46.392 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63755.log.gz 2023-10-30T14:28:46.394 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61470.log 2023-10-30T14:28:46.394 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53009.log 2023-10-30T14:28:46.395 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55627.log.gz 2023-10-30T14:28:46.395 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61470.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61470.log.gz 2023-10-30T14:28:46.395 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68199.log 2023-10-30T14:28:46.396 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53009.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49001.log.gz 2023-10-30T14:28:46.396 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49131.log 2023-10-30T14:28:46.397 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54469.log.gz 2023-10-30T14:28:46.397 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49290.log 2023-10-30T14:28:46.398 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68199.log.gz 2023-10-30T14:28:46.398 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65777.log 2023-10-30T14:28:46.398 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63970.log 2023-10-30T14:28:46.400 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65777.log: gzip -5 88.7% --verbose -- replaced with /var/log/ceph/ceph-client.admin.53009.log.gz -- 2023-10-30T14:28:46.400 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.46554.log 2023-10-30T14:28:46.401 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65777.log.gz 2023-10-30T14:28:46.401 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49131.log.gz 2023-10-30T14:28:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58061.log 2023-10-30T14:28:46.401 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63970.log.gz 2023-10-30T14:28:46.403 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70519.log 2023-10-30T14:28:46.404 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68987.log 2023-10-30T14:28:46.404 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58061.log.gz 2023-10-30T14:28:46.404 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44188.log 2023-10-30T14:28:46.404 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70519.log.gz 2023-10-30T14:28:46.405 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68987.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68987.log.gz 2023-10-30T14:28:46.405 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56804.log 2023-10-30T14:28:46.407 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49290.log.gz 2023-10-30T14:28:46.407 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53120.log 2023-10-30T14:28:46.407 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46554.log.gz 2023-10-30T14:28:46.407 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62501.log 2023-10-30T14:28:46.408 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56804.log.gz 2023-10-30T14:28:46.408 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38261.log 2023-10-30T14:28:46.410 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62501.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62501.log.gz 2023-10-30T14:28:46.410 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60366.log 2023-10-30T14:28:46.410 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44188.log.gz 2023-10-30T14:28:46.410 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50819.log 2023-10-30T14:28:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60366.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60366.log.gz 2023-10-30T14:28:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60774.log 2023-10-30T14:28:46.411 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53120.log.gz 2023-10-30T14:28:46.413 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64922.log 2023-10-30T14:28:46.413 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43211.log 2023-10-30T14:28:46.414 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60774.log.gz 2023-10-30T14:28:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48419.log 2023-10-30T14:28:46.414 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64922.log.gz 2023-10-30T14:28:46.415 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50819.log.gz 2023-10-30T14:28:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61281.log 2023-10-30T14:28:46.416 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48419.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43211.log.gz 2023-10-30T14:28:46.417 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65610.log 2023-10-30T14:28:46.417 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61281.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38261.log.gz 2023-10-30T14:28:46.417 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48419.log.gz 2023-10-30T14:28:46.418 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61281.log.gz 2023-10-30T14:28:46.418 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51518.log 2023-10-30T14:28:46.419 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51407.log 2023-10-30T14:28:46.420 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65610.log.gz 2023-10-30T14:28:46.420 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62095.log 2023-10-30T14:28:46.420 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51518.log.gz 2023-10-30T14:28:46.421 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70363.log 2023-10-30T14:28:46.421 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51407.log.gz 2023-10-30T14:28:46.421 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61795.log 2023-10-30T14:28:46.422 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62095.log.gz 2023-10-30T14:28:46.423 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52609.log 2023-10-30T14:28:46.423 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70363.log.gz 2023-10-30T14:28:46.423 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61795.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61795.log.gz 2023-10-30T14:28:46.424 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57376.log 2023-10-30T14:28:46.424 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65881.log 2023-10-30T14:28:46.425 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56536.log 2023-10-30T14:28:46.425 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57376.log.gz 2023-10-30T14:28:46.426 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56192.log 2023-10-30T14:28:46.426 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65881.log.gz 2023-10-30T14:28:46.427 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56536.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56536.log.gz 2023-10-30T14:28:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45159.log 2023-10-30T14:28:46.428 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52609.log.gz 2023-10-30T14:28:46.428 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56192.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56192.log.gz 2023-10-30T14:28:46.429 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65961.log 2023-10-30T14:28:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53988.log 2023-10-30T14:28:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65961.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-30T14:28:46.431 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65961.log.gz 2023-10-30T14:28:46.431 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70207.log 2023-10-30T14:28:46.431 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45159.log.gz 2023-10-30T14:28:46.432 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.1.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53988.log.gz 2023-10-30T14:28:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59425.log 2023-10-30T14:28:46.433 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70207.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70207.log.gz 2023-10-30T14:28:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50632.log 2023-10-30T14:28:46.434 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63584.log 2023-10-30T14:28:46.434 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59425.log.gz 2023-10-30T14:28:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47938.log 2023-10-30T14:28:46.436 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56778.log 2023-10-30T14:28:46.436 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63584.log.gz 2023-10-30T14:28:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47938.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50632.log.gz 2023-10-30T14:28:46.437 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65340.log 2023-10-30T14:28:46.438 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68637.log 2023-10-30T14:28:46.439 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56778.log.gz 2023-10-30T14:28:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65340.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65340.log.gz 2023-10-30T14:28:46.439 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69612.log 2023-10-30T14:28:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58087.log 2023-10-30T14:28:46.441 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68637.log.gz 2023-10-30T14:28:46.441 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69338.log 2023-10-30T14:28:46.442 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69612.log.gz 2023-10-30T14:28:46.443 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58087.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58087.log.gz 2023-10-30T14:28:46.443 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55312.log 2023-10-30T14:28:46.443 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47938.log.gz 2023-10-30T14:28:46.444 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69338.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69338.log.gz 2023-10-30T14:28:46.444 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63479.log 2023-10-30T14:28:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60130.log 2023-10-30T14:28:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66047.log 2023-10-30T14:28:46.446 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63479.log.gz 2023-10-30T14:28:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65397.log 2023-10-30T14:28:46.447 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60130.log.gz 2023-10-30T14:28:46.447 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55312.log.gz 2023-10-30T14:28:46.448 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66047.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66047.log.gz 2023-10-30T14:28:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68961.log 2023-10-30T14:28:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51963.log 2023-10-30T14:28:46.449 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65397.log.gz 2023-10-30T14:28:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68961.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68961.log.gz 2023-10-30T14:28:46.450 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66808.log 2023-10-30T14:28:46.451 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44958.log 2023-10-30T14:28:46.451 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66808.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66808.log.gz 2023-10-30T14:28:46.452 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56241.log 2023-10-30T14:28:46.452 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44958.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51963.log.gz 2023-10-30T14:28:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54495.log 2023-10-30T14:28:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56241.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56241.log.gz 2023-10-30T14:28:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52069.log 2023-10-30T14:28:46.455 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65636.log 2023-10-30T14:28:46.455 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44958.log.gz 2023-10-30T14:28:46.456 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50762.log 2023-10-30T14:28:46.456 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54495.log.gz 2023-10-30T14:28:46.457 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57190.log 2023-10-30T14:28:46.457 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65636.log.gz 2023-10-30T14:28:46.458 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57216.log 2023-10-30T14:28:46.459 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52283.log 2023-10-30T14:28:46.459 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57190.log.gz 2023-10-30T14:28:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51051.log 2023-10-30T14:28:46.460 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57216.log.gz 2023-10-30T14:28:46.460 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52069.log.gz 2023-10-30T14:28:46.462 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52283.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50762.log.gz 2023-10-30T14:28:46.462 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45078.log 2023-10-30T14:28:46.462 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67423.log 2023-10-30T14:28:46.463 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56594.log 2023-10-30T14:28:46.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46686.log 2023-10-30T14:28:46.464 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67423.log.gz 2023-10-30T14:28:46.465 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62227.log 2023-10-30T14:28:46.465 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56594.log.gz 2023-10-30T14:28:46.465 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52283.log.gz 2023-10-30T14:28:46.467 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49701.log 2023-10-30T14:28:46.467 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51051.log.gz 2023-10-30T14:28:46.467 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55876.log 2023-10-30T14:28:46.468 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62227.log.gz 2023-10-30T14:28:46.468 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51780.log 2023-10-30T14:28:46.469 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55876.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55876.log.gz 2023-10-30T14:28:46.470 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52956.log 2023-10-30T14:28:46.470 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49701.log.gz 2023-10-30T14:28:46.470 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61600.log 2023-10-30T14:28:46.472 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55416.log 2023-10-30T14:28:46.472 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46686.log.gz 2023-10-30T14:28:46.472 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51780.log.gz 2023-10-30T14:28:46.473 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52956.log.gz 2023-10-30T14:28:46.473 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56484.log 2023-10-30T14:28:46.473 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61600.log.gz 2023-10-30T14:28:46.475 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42852.log 2023-10-30T14:28:46.475 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56484.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45078.log.gz 2023-10-30T14:28:46.475 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55416.log.gz 2023-10-30T14:28:46.476 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.56484.log.gz 2023-10-30T14:28:46.476 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57633.log 2023-10-30T14:28:46.476 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59399.log 2023-10-30T14:28:46.477 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57633.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57633.log.gzgzip 2023-10-30T14:28:46.477 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70649.log 2023-10-30T14:28:46.479 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52043.log 2023-10-30T14:28:46.479 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59399.log.gz 2023-10-30T14:28:46.479 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65314.log 2023-10-30T14:28:46.480 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70649.log.gz 2023-10-30T14:28:46.480 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51650.log 2023-10-30T14:28:46.480 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42852.log.gz 2023-10-30T14:28:46.480 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49166.log 2023-10-30T14:28:46.481 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65314.log.gz 2023-10-30T14:28:46.482 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8%/var/log/ceph/ceph-client.admin.51650.log: -- replaced with /var/log/ceph/ceph-client.admin.52043.log.gz 2023-10-30T14:28:46.482 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60989.log 2023-10-30T14:28:46.483 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49166.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51650.log.gz 2023-10-30T14:28:46.483 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43425.log 2023-10-30T14:28:46.483 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57482.log 2023-10-30T14:28:46.483 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60989.log.gz 2023-10-30T14:28:46.485 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43425.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49166.log.gz 2023-10-30T14:28:46.485 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48341.log 2023-10-30T14:28:46.485 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57482.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57482.log.gz 2023-10-30T14:28:46.486 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69742.log 2023-10-30T14:28:46.486 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60957.log 2023-10-30T14:28:46.487 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43425.log.gz 2023-10-30T14:28:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69742.log: gzip -5 --verbose -- 88.4% /var/log/ceph/ceph-client.admin.45216.log -- replaced with /var/log/ceph/ceph-client.admin.69742.log.gz 2023-10-30T14:28:46.488 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67230.log 2023-10-30T14:28:46.489 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60957.log.gz 2023-10-30T14:28:46.489 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49455.log 2023-10-30T14:28:46.490 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44854.log 2023-10-30T14:28:46.490 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67230.log.gz 2023-10-30T14:28:46.491 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46634.log 2023-10-30T14:28:46.491 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65177.log 2023-10-30T14:28:46.492 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67258.log 2023-10-30T14:28:46.493 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70545.log 2023-10-30T14:28:46.493 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65177.log.gz 2023-10-30T14:28:46.494 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57105.log 2023-10-30T14:28:46.495 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67258.log.gz 2023-10-30T14:28:46.495 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59264.log 2023-10-30T14:28:46.495 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70545.log.gz 2023-10-30T14:28:46.497 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45969.log 2023-10-30T14:28:46.497 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57105.log.gz 2023-10-30T14:28:46.497 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48341.log.gz 2023-10-30T14:28:46.498 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49238.log 2023-10-30T14:28:46.498 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45216.log.gz 2023-10-30T14:28:46.498 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59264.log.gz 2023-10-30T14:28:46.500 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45905.log 2023-10-30T14:28:46.500 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56451.log 2023-10-30T14:28:46.500 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55816.log 2023-10-30T14:28:46.502 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45714.log 2023-10-30T14:28:46.502 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56451.log.gz 2023-10-30T14:28:46.502 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55790.log 2023-10-30T14:28:46.503 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55816.log.gz 2023-10-30T14:28:46.503 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49455.log.gz 2023-10-30T14:28:46.505 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53437.log 2023-10-30T14:28:46.505 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44854.log.gz 2023-10-30T14:28:46.505 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69426.log 2023-10-30T14:28:46.506 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55790.log.gz 2023-10-30T14:28:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52815.log 2023-10-30T14:28:46.506 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45969.log.gz 2023-10-30T14:28:46.508 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68225.log 2023-10-30T14:28:46.509 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49238.log.gz 2023-10-30T14:28:46.509 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0%/var/log/ceph/ceph-client.admin.52815.log: -- replaced with /var/log/ceph/ceph-client.admin.69426.log.gz 2023-10-30T14:28:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43061.log 2023-10-30T14:28:46.509 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45905.log.gz 2023-10-30T14:28:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52878.log 2023-10-30T14:28:46.510 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68225.log.gz 2023-10-30T14:28:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43850.log 2023-10-30T14:28:46.512 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53568.log 2023-10-30T14:28:46.512 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62800.log 2023-10-30T14:28:46.512 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67765.log 2023-10-30T14:28:46.513 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62800.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45714.log.gz 2023-10-30T14:28:46.513 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64519.log 2023-10-30T14:28:46.513 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62800.log.gz 2023-10-30T14:28:46.515 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67765.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53437.log.gz 2023-10-30T14:28:46.515 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54842.log 2023-10-30T14:28:46.515 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67765.log.gz 2023-10-30T14:28:46.516 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71003.log 2023-10-30T14:28:46.516 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64519.log.gz 2023-10-30T14:28:46.516 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44162.log 2023-10-30T14:28:46.517 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43061.log.gz 2023-10-30T14:28:46.518 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50463.log 2023-10-30T14:28:46.518 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71003.log.gz 2023-10-30T14:28:46.519 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44162.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43850.log.gz 2023-10-30T14:28:46.519 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56267.log 2023-10-30T14:28:46.519 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60853.log 2023-10-30T14:28:46.521 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57537.log 2023-10-30T14:28:46.521 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56267.log.gz 2023-10-30T14:28:46.522 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60853.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60853.log.gz 2023-10-30T14:28:46.522 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53568.log.gz 2023-10-30T14:28:46.522 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48094.log 2023-10-30T14:28:46.523 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67903.log 2023-10-30T14:28:46.523 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54842.log.gz 2023-10-30T14:28:46.524 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57537.log.gz 2023-10-30T14:28:46.525 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52713.log 2023-10-30T14:28:46.525 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60476.log 2023-10-30T14:28:46.525 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67903.log.gz 2023-10-30T14:28:46.525 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52713.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44162.log.gz 2023-10-30T14:28:46.526 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63103.log 2023-10-30T14:28:46.527 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60476.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50463.log.gz 2023-10-30T14:28:46.528 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60476.log.gz 2023-10-30T14:28:46.528 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48633.log 2023-10-30T14:28:46.528 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63103.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48094.log.gz 2023-10-30T14:28:46.529 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63103.log.gz 2023-10-30T14:28:46.529 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51327.log 2023-10-30T14:28:46.529 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52309.log 2023-10-30T14:28:46.531 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49640.log 2023-10-30T14:28:46.531 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53035.log 2023-10-30T14:28:46.531 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52713.log.gz 2023-10-30T14:28:46.532 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56399.log 2023-10-30T14:28:46.532 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49640.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48633.log.gz 2023-10-30T14:28:46.532 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68382.log 2023-10-30T14:28:46.533 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59345.log 2023-10-30T14:28:46.534 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56399.log.gz 2023-10-30T14:28:46.534 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47039.log 2023-10-30T14:28:46.534 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68382.log.gz 2023-10-30T14:28:46.535 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59857.log 2023-10-30T14:28:46.536 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59345.log.gz 2023-10-30T14:28:46.536 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66966.log 2023-10-30T14:28:46.536 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51327.log.gz 2023-10-30T14:28:46.537 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54576.log 2023-10-30T14:28:46.537 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59857.log.gz 2023-10-30T14:28:46.538 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56906.log 2023-10-30T14:28:46.539 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66966.log.gz 2023-10-30T14:28:46.539 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54576.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49640.log.gz 2023-10-30T14:28:46.539 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54656.log 2023-10-30T14:28:46.539 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53035.log.gz 2023-10-30T14:28:46.541 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56906.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T14:28:46.542 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56906.log.gz 2023-10-30T14:28:46.542 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47039.log.gz 2023-10-30T14:28:46.542 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57789.log 2023-10-30T14:28:46.542 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52309.log.gz 2023-10-30T14:28:46.543 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52815.log.gz 2023-10-30T14:28:46.543 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48120.log 2023-10-30T14:28:46.543 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52878.log.gz 2023-10-30T14:28:46.545 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57789.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54576.log.gz 2023-10-30T14:28:46.546 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42791.log 2023-10-30T14:28:46.546 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57789.log.gz 2023-10-30T14:28:46.546 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48120.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54656.log.gz 2023-10-30T14:28:46.547 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67082.log 2023-10-30T14:28:46.547 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42791.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48120.log.gz 2023-10-30T14:28:46.547 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54708.log 2023-10-30T14:28:46.547 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46634.log.gz 2023-10-30T14:28:46.548 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42791.log.gz 2023-10-30T14:28:46.548 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67082.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67082.log.gz 2023-10-30T14:28:46.548 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48711.log 2023-10-30T14:28:46.551 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59127.log 2023-10-30T14:28:46.551 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54708.log.gz 2023-10-30T14:28:46.552 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48711.log: 88.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T14:28:46.552 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57685.log 2023-10-30T14:28:46.552 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48711.log.gz 2023-10-30T14:28:46.552 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69586.log 2023-10-30T14:28:46.553 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59127.log.gz 2023-10-30T14:28:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57685.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57685.log.gz 2023-10-30T14:28:46.553 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44514.log 2023-10-30T14:28:46.554 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61574.log 2023-10-30T14:28:46.554 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69586.log.gz 2023-10-30T14:28:46.556 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54014.log 2023-10-30T14:28:46.556 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61574.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61574.log.gz 2023-10-30T14:28:46.557 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67929.log 2023-10-30T14:28:46.557 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66914.log 2023-10-30T14:28:46.557 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69286.log 2023-10-30T14:28:46.558 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67929.log.gz 2023-10-30T14:28:46.558 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69664.log 2023-10-30T14:28:46.558 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66914.log.gz 2023-10-30T14:28:46.558 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44514.log.gz 2023-10-30T14:28:46.561 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54521.log 2023-10-30T14:28:46.561 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69286.log.gz 2023-10-30T14:28:46.561 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54014.log.gz 2023-10-30T14:28:46.562 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44568.log 2023-10-30T14:28:46.562 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69664.log.gz 2023-10-30T14:28:46.562 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60748.log 2023-10-30T14:28:46.563 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54521.log.gz 2023-10-30T14:28:46.563 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58576.log 2023-10-30T14:28:46.563 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54796.log 2023-10-30T14:28:46.563 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60748.log.gz 2023-10-30T14:28:46.565 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58576.log: 88.1% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58576.log.gz 2023-10-30T14:28:46.566 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44568.log.gz 2023-10-30T14:28:46.566 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68559.log 2023-10-30T14:28:46.566 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58547.log 2023-10-30T14:28:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69508.log 2023-10-30T14:28:46.567 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68559.log.gz 2023-10-30T14:28:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43980.log 2023-10-30T14:28:46.568 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58547.log.gz 2023-10-30T14:28:46.568 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54796.log.gz 2023-10-30T14:28:46.570 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65558.log 2023-10-30T14:28:46.570 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69508.log.gz 2023-10-30T14:28:46.570 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50307.log 2023-10-30T14:28:46.571 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56114.log 2023-10-30T14:28:46.571 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.65558.log.gz 2023-10-30T14:28:46.571 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66616.log 2023-10-30T14:28:46.571 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69716.log 2023-10-30T14:28:46.572 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56114.log.gz 2023-10-30T14:28:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66616.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43980.log.gz 2023-10-30T14:28:46.574 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66616.log.gz 2023-10-30T14:28:46.574 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70033.log 2023-10-30T14:28:46.574 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69716.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50307.log.gz 2023-10-30T14:28:46.575 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46712.log 2023-10-30T14:28:46.575 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69716.log.gz 2023-10-30T14:28:46.575 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70033.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70033.log.gz 2023-10-30T14:28:46.576 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51221.log 2023-10-30T14:28:46.576 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60026.log 2023-10-30T14:28:46.577 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46958.log 2023-10-30T14:28:46.578 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60026.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46712.log.gz 2023-10-30T14:28:46.578 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51355.log 2023-10-30T14:28:46.579 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60026.log.gz 2023-10-30T14:28:46.579 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46958.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51221.log.gz 2023-10-30T14:28:46.579 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57079.log 2023-10-30T14:28:46.579 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52098.log 2023-10-30T14:28:46.581 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66180.log 2023-10-30T14:28:46.581 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57079.log.gz 2023-10-30T14:28:46.582 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70467.log 2023-10-30T14:28:46.582 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46958.log.gz 2023-10-30T14:28:46.582 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62826.log 2023-10-30T14:28:46.583 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66180.log.gz 2023-10-30T14:28:46.583 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70467.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51355.log.gz 2023-10-30T14:28:46.584 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70467.log.gz 2023-10-30T14:28:46.585 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64764.log 2023-10-30T14:28:46.585 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48739.log 2023-10-30T14:28:46.585 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62826.log.gz 2023-10-30T14:28:46.585 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64764.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64764.log.gz 2023-10-30T14:28:46.586 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64467.log 2023-10-30T14:28:46.587 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48739.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52098.log.gz 2023-10-30T14:28:46.588 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63318.log 2023-10-30T14:28:46.588 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67019.log 2023-10-30T14:28:46.588 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64467.log.gz 2023-10-30T14:28:46.589 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51087.log 2023-10-30T14:28:46.589 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63318.log.gz 2023-10-30T14:28:46.590 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52335.log 2023-10-30T14:28:46.591 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67019.log.gz 2023-10-30T14:28:46.591 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44724.log 2023-10-30T14:28:46.591 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48739.log.gz 2023-10-30T14:28:46.592 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42685.log 2023-10-30T14:28:46.592 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51169.log 2023-10-30T14:28:46.593 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44620.log 2023-10-30T14:28:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46984.log 2023-10-30T14:28:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64184.log 2023-10-30T14:28:46.595 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46984.log: gzip -5 89.2% --verbose -- /var/log/ceph/ceph-client.admin.56373.log -- replaced with /var/log/ceph/ceph-client.admin.51087.log.gz 2023-10-30T14:28:46.595 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.596 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64184.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64184.log.gz 2023-10-30T14:28:46.596 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52661.log 2023-10-30T14:28:46.597 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67877.log 2023-10-30T14:28:46.598 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56373.log.gz 2023-10-30T14:28:46.599 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52661.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52335.log.gz 2023-10-30T14:28:46.599 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69771.log 2023-10-30T14:28:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48206.log 2023-10-30T14:28:46.601 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67877.log.gz 2023-10-30T14:28:46.602 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69771.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69771.log.gz 2023-10-30T14:28:46.602 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44724.log.gz 2023-10-30T14:28:46.602 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42685.log.gz 2023-10-30T14:28:46.603 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51169.log.gz 2023-10-30T14:28:46.603 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44620.log.gz 2023-10-30T14:28:46.603 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58813.log 2023-10-30T14:28:46.603 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46984.log.gz 2023-10-30T14:28:46.606 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48206.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52661.log.gz 2023-10-30T14:28:46.606 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58142.log 2023-10-30T14:28:46.606 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54358.log 2023-10-30T14:28:46.607 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58813.log.gz 2023-10-30T14:28:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58142.log: gzip -5 --verbose 89.1% -- /var/log/ceph/ceph-client.admin.70311.log -- replaced with /var/log/ceph/ceph-client.admin.58142.log.gz 2023-10-30T14:28:46.607 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62580.log 2023-10-30T14:28:46.609 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52852.log 2023-10-30T14:28:46.610 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70311.log.gz 2023-10-30T14:28:46.610 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70675.log 2023-10-30T14:28:46.610 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62580.log.gz 2023-10-30T14:28:46.611 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48206.log.gz 2023-10-30T14:28:46.611 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52852.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54358.log.gz 2023-10-30T14:28:46.611 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57456.log 2023-10-30T14:28:46.612 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53542.log 2023-10-30T14:28:46.612 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70675.log.gz 2023-10-30T14:28:46.614 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38025.log 2023-10-30T14:28:46.614 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57456.log.gz 2023-10-30T14:28:46.614 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65452.log 2023-10-30T14:28:46.615 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52852.log.gz 2023-10-30T14:28:46.615 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65665.log 2023-10-30T14:28:46.615 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58384.log 2023-10-30T14:28:46.616 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65452.log.gz 2023-10-30T14:28:46.616 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62694.log 2023-10-30T14:28:46.616 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65665.log.gz 2023-10-30T14:28:46.618 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64105.log 2023-10-30T14:28:46.618 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58384.log.gz 2023-10-30T14:28:46.619 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51754.log 2023-10-30T14:28:46.619 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62694.log.gz 2023-10-30T14:28:46.619 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53542.log.gz/var/log/ceph/ceph-client.admin.64105.log: 2023-10-30T14:28:46.619 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64105.log.gz 2023-10-30T14:28:46.620 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53333.log 2023-10-30T14:28:46.620 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53857.log 2023-10-30T14:28:46.622 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56674.log 2023-10-30T14:28:46.622 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42628.log 2023-10-30T14:28:46.622 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56674.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56674.log.gz 2023-10-30T14:28:46.623 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65829.log 2023-10-30T14:28:46.623 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51754.log.gz 2023-10-30T14:28:46.623 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53751.log 2023-10-30T14:28:46.624 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53333.log.gz 2023-10-30T14:28:46.625 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48578.log 2023-10-30T14:28:46.625 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65829.log.gz 2023-10-30T14:28:46.626 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48895.log 2023-10-30T14:28:46.626 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48578.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53857.log.gz 2023-10-30T14:28:46.626 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66538.log 2023-10-30T14:28:46.628 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48895.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42628.log.gz 2023-10-30T14:28:46.628 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54410.log 2023-10-30T14:28:46.628 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42426.log 2023-10-30T14:28:46.629 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66538.log.gz 2023-10-30T14:28:46.629 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57272.log 2023-10-30T14:28:46.630 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53751.log.gz 2023-10-30T14:28:46.630 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63292.log 89.0% 2023-10-30T14:28:46.630 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48578.log.gz 2023-10-30T14:28:46.632 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58436.log 2023-10-30T14:28:46.632 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48895.log.gz 2023-10-30T14:28:46.633 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57272.log.gz 2023-10-30T14:28:46.633 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54410.log.gz 2023-10-30T14:28:46.633 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42426.log.gz 2023-10-30T14:28:46.634 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63292.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.63292.log.gz 2023-10-30T14:28:46.634 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59661.log 2023-10-30T14:28:46.635 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49316.log 2023-10-30T14:28:46.636 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58436.log.gz 2023-10-30T14:28:46.636 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.38025.log.gz 2023-10-30T14:28:46.636 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59661.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59661.log.gz 2023-10-30T14:28:46.637 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68856.log 2023-10-30T14:28:46.637 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44266.log 2023-10-30T14:28:46.637 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68856.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68856.log.gz 2023-10-30T14:28:46.638 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68411.log 2023-10-30T14:28:46.639 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44750.log 2023-10-30T14:28:46.639 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42519.log 2023-10-30T14:28:46.640 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68411.log.gz 2023-10-30T14:28:46.640 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49614.log 2023-10-30T14:28:46.640 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49316.log.gz 2023-10-30T14:28:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57848.log 2023-10-30T14:28:46.642 INFO:teuthology.orchestra.run.smithi046.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.44266.log.gz 2023-10-30T14:28:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50605.log 2023-10-30T14:28:46.643 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55731.log 2023-10-30T14:28:46.643 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57848.log.gz 2023-10-30T14:28:46.644 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63643.log 2023-10-30T14:28:46.645 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44750.log.gz 2023-10-30T14:28:46.645 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54604.log 2023-10-30T14:28:46.645 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55731.log.gz 2023-10-30T14:28:46.646 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42519.log.gz 2023-10-30T14:28:46.646 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43139.log 2023-10-30T14:28:46.646 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63643.log.gz 2023-10-30T14:28:46.648 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68147.log 2023-10-30T14:28:46.648 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42659.log 2023-10-30T14:28:46.649 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68147.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49614.log.gz 2023-10-30T14:28:46.649 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57955.log 2023-10-30T14:28:46.649 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68147.log.gz 2023-10-30T14:28:46.649 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42659.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50605.log.gz 2023-10-30T14:28:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68251.log 2023-10-30T14:28:46.651 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54630.log 2023-10-30T14:28:46.652 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57955.log.gz 2023-10-30T14:28:46.652 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70977.log 2023-10-30T14:28:46.652 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68251.log.gz 2023-10-30T14:28:46.653 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54604.log.gz 2023-10-30T14:28:46.654 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52983.log 89.3% 2023-10-30T14:28:46.654 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43139.log.gz 2023-10-30T14:28:46.655 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54173.log 2023-10-30T14:28:46.655 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70977.log.gz 2023-10-30T14:28:46.655 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43624.log 2023-10-30T14:28:46.656 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50789.log 2023-10-30T14:28:46.657 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54199.log 2023-10-30T14:28:46.657 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52557.log 2023-10-30T14:28:46.658 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42659.log.gz 2023-10-30T14:28:46.658 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54737.log 2023-10-30T14:28:46.658 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54630.log.gz 2023-10-30T14:28:46.659 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53255.log 2023-10-30T14:28:46.660 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52768.log 2023-10-30T14:28:46.661 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64844.log 2023-10-30T14:28:46.661 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45508.log 2023-10-30T14:28:46.662 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49053.log 2023-10-30T14:28:46.663 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64844.log.gz 2023-10-30T14:28:46.663 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45508.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52983.log.gz 2023-10-30T14:28:46.664 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61248.log 2023-10-30T14:28:46.664 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52904.log 2023-10-30T14:28:46.665 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54173.log.gz 2023-10-30T14:28:46.666 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61248.log: gzip -5 88.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43624.log.gz /var/log/ceph/ceph-client.admin.60052.log 2023-10-30T14:28:46.666 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:28:46.667 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50789.log.gz 2023-10-30T14:28:46.667 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54199.log.gz 2023-10-30T14:28:46.667 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61248.log.gz 2023-10-30T14:28:46.668 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52904.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52557.log.gz 2023-10-30T14:28:46.668 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56726.log 2023-10-30T14:28:46.670 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58009.log 2023-10-30T14:28:46.670 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60052.log.gz 2023-10-30T14:28:46.671 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56726.log: gzip -5 --verbose 88.2% -- -- replaced with /var/log/ceph/ceph-client.admin.56726.log.gz 2023-10-30T14:28:46.671 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.53962.log 2023-10-30T14:28:46.671 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58009.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54737.log.gz 2023-10-30T14:28:46.672 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46771.log 88.4% 2023-10-30T14:28:46.672 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58009.log.gz 2023-10-30T14:28:46.672 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53962.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52768.log.gz 2023-10-30T14:28:46.673 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45508.log.gz 2023-10-30T14:28:46.673 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48843.log 2023-10-30T14:28:46.674 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.4.log: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-30T14:28:46.676 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53255.log.gz 2023-10-30T14:28:46.677 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46771.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53962.log.gz 2023-10-30T14:28:46.677 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49053.log.gz 2023-10-30T14:28:46.677 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51140.log 2023-10-30T14:28:46.678 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70808.log 2023-10-30T14:28:46.678 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68040.log 2023-10-30T14:28:46.678 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46771.log.gz 2023-10-30T14:28:46.679 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48843.log.gz 2023-10-30T14:28:46.679 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70808.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70808.log.gz 2023-10-30T14:28:46.679 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56568.log 2023-10-30T14:28:46.679 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68040.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52904.log.gz 2023-10-30T14:28:46.680 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70623.log 2023-10-30T14:28:46.680 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68040.log.gz 2023-10-30T14:28:46.680 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56568.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56568.log.gz 2023-10-30T14:28:46.681 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48393.log 2023-10-30T14:28:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44880.log 2023-10-30T14:28:46.683 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70623.log.gz 2023-10-30T14:28:46.683 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48393.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.52583.log 2023-10-30T14:28:46.684 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51140.log.gz 2023-10-30T14:28:46.684 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46932.log 2023-10-30T14:28:46.684 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53281.log 2023-10-30T14:28:46.684 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48523.log 2023-10-30T14:28:46.685 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48393.log.gz 2023-10-30T14:28:46.685 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44880.log.gz 2023-10-30T14:28:46.685 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53281.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52583.log.gz 2023-10-30T14:28:46.685 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50947.log 2023-10-30T14:28:46.686 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53281.log.gz 2023-10-30T14:28:46.688 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38144.log 2023-10-30T14:28:46.688 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48523.log.gz 2023-10-30T14:28:46.688 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67371.log 2023-10-30T14:28:46.689 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50947.log.gz 2023-10-30T14:28:46.689 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57324.log 2023-10-30T14:28:46.689 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66260.log 2023-10-30T14:28:46.689 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67371.log.gz 2023-10-30T14:28:46.690 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53203.log 2023-10-30T14:28:46.690 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57324.log.gz 2023-10-30T14:28:46.692 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51025.log 2023-10-30T14:28:46.692 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66260.log.gz 2023-10-30T14:28:46.692 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45397.log 2023-10-30T14:28:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58330.log 2023-10-30T14:28:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58204.log 2023-10-30T14:28:46.694 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56510.log 2023-10-30T14:28:46.695 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58330.log.gz 2023-10-30T14:28:46.695 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53203.log.gz 2023-10-30T14:28:46.695 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58204.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58204.log.gz 2023-10-30T14:28:46.696 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70259.log 2023-10-30T14:28:46.696 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51025.log.gz 2023-10-30T14:28:46.696 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56510.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45397.log.gz 2023-10-30T14:28:46.696 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56510.log.gz 2023-10-30T14:28:46.698 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T14:28:46.698 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69260.log 2023-10-30T14:28:46.698 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70259.log.gz 2023-10-30T14:28:46.699 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.audit.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46932.log.gz 2023-10-30T14:28:46.699 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44494.log 2023-10-30T14:28:46.700 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45188.log 2023-10-30T14:28:46.701 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69260.log.gz 2023-10-30T14:28:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.44494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.44494.log.gz 2023-10-30T14:28:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63129.log 2023-10-30T14:28:46.701 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46608.log 2023-10-30T14:28:46.703 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45188.log.gz 2023-10-30T14:28:46.704 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63129.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63129.log.gz 2023-10-30T14:28:46.704 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49804.log 2023-10-30T14:28:46.704 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46608.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-30T14:28:46.704 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46608.log.gz 2023-10-30T14:28:46.705 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67397.log 2023-10-30T14:28:46.706 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55260.log 2023-10-30T14:28:46.707 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52362.log 2023-10-30T14:28:46.707 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67397.log.gz 2023-10-30T14:28:46.707 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53177.log 2023-10-30T14:28:46.709 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58278.log 2023-10-30T14:28:46.709 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49804.log.gz 2023-10-30T14:28:46.709 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61679.log 2023-10-30T14:28:46.710 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55260.log.gz 2023-10-30T14:28:46.711 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58278.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58278.log.gz 2023-10-30T14:28:46.711 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64712.log 2023-10-30T14:28:46.711 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61679.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61679.log.gz 2023-10-30T14:28:46.712 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-30T14:28:46.712 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52362.log.gz 2023-10-30T14:28:46.713 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64572.log 2023-10-30T14:28:46.713 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64712.log.gz 2023-10-30T14:28:46.714 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53177.log.gz 2023-10-30T14:28:46.714 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43427.log 2023-10-30T14:28:46.715 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64572.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64572.log.gz 2023-10-30T14:28:46.715 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43426.log 2023-10-30T14:28:46.716 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65711.log 2023-10-30T14:28:46.716 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70337.log 2023-10-30T14:28:46.717 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65711.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.65711.log.gz 2023-10-30T14:28:46.718 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66128.log 2023-10-30T14:28:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70337.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70337.log.gz 2023-10-30T14:28:46.719 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57737.log 2023-10-30T14:28:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56347.log 2023-10-30T14:28:46.721 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66128.log.gz 2023-10-30T14:28:46.721 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43427.log.gz 2023-10-30T14:28:46.722 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57737.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57737.log.gz 2023-10-30T14:28:46.722 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50489.log 2023-10-30T14:28:46.722 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56347.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43426.log.gz 2023-10-30T14:28:46.723 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56347.log.gz 2023-10-30T14:28:46.724 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43203.log 2023-10-30T14:28:46.724 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38144.log.gz 2023-10-30T14:28:46.725 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63781.log 2023-10-30T14:28:46.725 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60236.log 2023-10-30T14:28:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63781.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50489.log.gz 2023-10-30T14:28:46.727 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63781.log.gz 2023-10-30T14:28:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51806.log 2023-10-30T14:28:46.728 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64262.log 88.6% 2023-10-30T14:28:46.728 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60236.log.gz 2023-10-30T14:28:46.729 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68329.log 2023-10-30T14:28:46.730 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43203.log.gz 2023-10-30T14:28:46.730 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64262.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64262.log.gz 2023-10-30T14:28:46.731 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51806.log.gz 2023-10-30T14:28:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62449.log 2023-10-30T14:28:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44594.log 2023-10-30T14:28:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62449.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68329.log.gz 2023-10-30T14:28:46.733 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62014.log 2023-10-30T14:28:46.733 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62449.log.gz 2023-10-30T14:28:46.734 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42763.log 2023-10-30T14:28:46.735 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42819.log 2023-10-30T14:28:46.735 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62014.log.gz 2023-10-30T14:28:46.735 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44594.log.gz 2023-10-30T14:28:46.736 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60639.log 2023-10-30T14:28:46.737 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63833.log 2023-10-30T14:28:46.737 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62253.log 2023-10-30T14:28:46.738 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60639.log.gz 2023-10-30T14:28:46.739 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63833.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42763.log.gz 2023-10-30T14:28:46.739 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.63833.log.gz 2023-10-30T14:28:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54936.log 2023-10-30T14:28:46.740 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42819.log.gz 2023-10-30T14:28:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66668.log 2023-10-30T14:28:46.741 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62253.log.gz 2023-10-30T14:28:46.742 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65994.log 2023-10-30T14:28:46.743 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65151.log 2023-10-30T14:28:46.743 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66668.log.gz 2023-10-30T14:28:46.744 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54936.log.gz 2023-10-30T14:28:46.744 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65994.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65994.log.gz 2023-10-30T14:28:46.745 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70181.log 2023-10-30T14:28:46.746 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54332.log 2023-10-30T14:28:46.746 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65151.log.gz 2023-10-30T14:28:46.747 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70181.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70181.log.gz 2023-10-30T14:28:46.747 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54040.log 2023-10-30T14:28:46.748 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70732.log 2023-10-30T14:28:46.749 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62309.log 2023-10-30T14:28:46.750 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70732.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70732.log.gz 2023-10-30T14:28:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61116.log 2023-10-30T14:28:46.751 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62309.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62309.log.gz 2023-10-30T14:28:46.752 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60554.log 2023-10-30T14:28:46.753 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54332.log.gz 2023-10-30T14:28:46.753 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55286.log 2023-10-30T14:28:46.754 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61116.log.gz 2023-10-30T14:28:46.754 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54040.log.gz 2023-10-30T14:28:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60554.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60554.log.gz 2023-10-30T14:28:46.755 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61335.log 2023-10-30T14:28:46.756 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44084.log 2023-10-30T14:28:46.757 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61335.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61335.log.gz 2023-10-30T14:28:46.757 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69455.log 2023-10-30T14:28:46.758 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55286.log.gz 2023-10-30T14:28:46.758 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58682.log 2023-10-30T14:28:46.759 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69455.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69455.log.gz 2023-10-30T14:28:46.760 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70285.log 2023-10-30T14:28:46.761 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58682.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58682.log.gz 2023-10-30T14:28:46.761 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51991.log 2023-10-30T14:28:46.761 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44084.log.gz 2023-10-30T14:28:46.762 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65909.log 2023-10-30T14:28:46.762 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70285.log.gz 2023-10-30T14:28:46.763 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67504.log 2023-10-30T14:28:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49212.log 2023-10-30T14:28:46.765 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65909.log.gz 2023-10-30T14:28:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67504.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51991.log.gz 2023-10-30T14:28:46.766 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67504.log.gz 2023-10-30T14:28:46.766 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43219.log 2023-10-30T14:28:46.767 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59205.log 2023-10-30T14:28:46.768 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42602.log 2023-10-30T14:28:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50576.log 2023-10-30T14:28:46.770 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59205.log.gz 2023-10-30T14:28:46.770 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49212.log.gz 2023-10-30T14:28:46.770 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43219.log.gz 2023-10-30T14:28:46.771 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66510.log 2023-10-30T14:28:46.772 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68092.log 2023-10-30T14:28:46.773 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42904.log 2023-10-30T14:28:46.773 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66510.log.gz 2023-10-30T14:28:46.774 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68092.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68092.log.gz 2023-10-30T14:28:46.775 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42602.log.gz 2023-10-30T14:28:46.775 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50576.log.gz 2023-10-30T14:28:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65855.log 2023-10-30T14:28:46.776 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68585.log 2023-10-30T14:28:46.777 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65855.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65855.log.gz 2023-10-30T14:28:46.777 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55208.log 2023-10-30T14:28:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63051.log 2023-10-30T14:28:46.779 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68585.log.gz 2023-10-30T14:28:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55208.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42904.log.gz 2023-10-30T14:28:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60182.log 2023-10-30T14:28:46.781 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63051.log: 88.0% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63051.log.gz 2023-10-30T14:28:46.781 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55208.log.gz 2023-10-30T14:28:46.781 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43218.log 2023-10-30T14:28:46.782 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63077.log 2023-10-30T14:28:46.783 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60182.log.gz 2023-10-30T14:28:46.783 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55477.log 2023-10-30T14:28:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63077.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63077.log.gz 2023-10-30T14:28:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65203.log 2023-10-30T14:28:46.785 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43218.log.gz 2023-10-30T14:28:46.786 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65125.log 2023-10-30T14:28:46.787 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58304.log 2023-10-30T14:28:46.787 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.65203.log.gz 2023-10-30T14:28:46.788 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65125.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65125.log.gz 2023-10-30T14:28:46.788 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55477.log.gz 2023-10-30T14:28:46.789 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57607.log 2023-10-30T14:28:46.790 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43928.log 2023-10-30T14:28:46.790 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58304.log: /var/log/ceph/ceph-client.admin.57607.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58304.log.gz 2023-10-30T14:28:46.790 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52395.log 2023-10-30T14:28:46.791 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57607.log.gz 2023-10-30T14:28:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49588.log 2023-10-30T14:28:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59724.log 2023-10-30T14:28:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49588.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43928.log.gz 2023-10-30T14:28:46.793 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.52395.log.gz -5 2023-10-30T14:28:46.794 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.69234.log 2023-10-30T14:28:46.794 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49588.log.gz 2023-10-30T14:28:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54384.log 2023-10-30T14:28:46.795 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59724.log.gz 2023-10-30T14:28:46.796 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69234.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69234.log.gz 2023-10-30T14:28:46.796 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50333.log 2023-10-30T14:28:46.797 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69482.log 2023-10-30T14:28:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66940.log 2023-10-30T14:28:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69482.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69482.log.gz 2023-10-30T14:28:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61090.log 2023-10-30T14:28:46.801 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60931.log 2023-10-30T14:28:46.801 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66940.log.gz 2023-10-30T14:28:46.803 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61090.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54384.log.gz 2023-10-30T14:28:46.803 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57245.log 2023-10-30T14:28:46.803 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61090.log.gz 2023-10-30T14:28:46.804 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60931.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50333.log.gz 2023-10-30T14:28:46.805 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60931.log.gz 2023-10-30T14:28:46.805 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64598.log 2023-10-30T14:28:46.805 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53307.log 2023-10-30T14:28:46.806 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57245.log.gz 2023-10-30T14:28:46.807 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64598.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64598.log.gz -5 2023-10-30T14:28:46.807 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42711.log 2023-10-30T14:28:46.807 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63531.log 2023-10-30T14:28:46.808 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62774.log 2023-10-30T14:28:46.809 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53594.log 2023-10-30T14:28:46.809 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63531.log.gz 2023-10-30T14:28:46.810 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62774.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62774.log.gz 2023-10-30T14:28:46.811 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54253.log 2023-10-30T14:28:46.811 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53307.log.gz 2023-10-30T14:28:46.812 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62147.log 2023-10-30T14:28:46.813 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59777.log 2023-10-30T14:28:46.814 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52739.log 2023-10-30T14:28:46.814 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62147.log.gz 2023-10-30T14:28:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62720.log 2023-10-30T14:28:46.816 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59777.log.gz 2023-10-30T14:28:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48367.log 2023-10-30T14:28:46.817 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58604.log 2023-10-30T14:28:46.818 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62720.log.gz 2023-10-30T14:28:46.819 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62201.log 2023-10-30T14:28:46.819 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58604.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42711.log.gz 2023-10-30T14:28:46.820 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54682.log 2023-10-30T14:28:46.821 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58604.log.gz 2023-10-30T14:28:46.821 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62201.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54253.log.gz 2023-10-30T14:28:46.821 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52739.log.gz 2023-10-30T14:28:46.822 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42737.log 2023-10-30T14:28:46.823 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62201.log.gz 2023-10-30T14:28:46.823 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53594.log.gz 2023-10-30T14:28:46.823 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54682.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48367.log.gz 2023-10-30T14:28:46.823 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70233.log 2023-10-30T14:28:46.824 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55390.log 2023-10-30T14:28:46.825 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60210.log 2023-10-30T14:28:46.825 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54682.log.gz 2023-10-30T14:28:46.826 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70233.log.gz 2023-10-30T14:28:46.826 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62904.log 2023-10-30T14:28:46.827 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62878.log 2023-10-30T14:28:46.828 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60210.log.gz 2023-10-30T14:28:46.828 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62904.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62904.log.gz 2023-10-30T14:28:46.829 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54147.log 2023-10-30T14:28:46.830 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42737.log.gz 2023-10-30T14:28:46.830 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55390.log.gz 2023-10-30T14:28:46.831 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61444.log 2023-10-30T14:28:46.831 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62878.log.gz 2023-10-30T14:28:46.832 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66154.log 2023-10-30T14:28:46.833 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54066.log 2023-10-30T14:28:46.833 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61444.log.gz 2023-10-30T14:28:46.834 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60722.log 2023-10-30T14:28:46.834 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66154.log.gz 2023-10-30T14:28:46.835 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54147.log.gz 2023-10-30T14:28:46.835 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58891.log 2023-10-30T14:28:46.836 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60722.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60722.log.gz 2023-10-30T14:28:46.836 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57164.log 2023-10-30T14:28:46.837 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53229.log 2023-10-30T14:28:46.838 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58891.log.gz 2023-10-30T14:28:46.839 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64158.log 2023-10-30T14:28:46.839 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54066.log.gz 2023-10-30T14:28:46.840 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57164.log.gz 2023-10-30T14:28:46.840 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70155.log 2023-10-30T14:28:46.841 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64158.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64158.log.gz 2023-10-30T14:28:46.842 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59477.log 2023-10-30T14:28:46.842 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56008.log 2023-10-30T14:28:46.843 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70155.log.gz 2023-10-30T14:28:46.844 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59477.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53229.log.gz 2023-10-30T14:28:46.844 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59477.log.gz 2023-10-30T14:28:46.845 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49481.log 2023-10-30T14:28:46.845 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53385.log 2023-10-30T14:28:46.846 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56008.log.gz 2023-10-30T14:28:46.846 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64131.log 2023-10-30T14:28:46.848 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52531.log 2023-10-30T14:28:46.848 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49481.log.gz 2023-10-30T14:28:46.849 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52477.log 2023-10-30T14:28:46.849 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64131.log.gz 2023-10-30T14:28:46.850 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59020.log 2023-10-30T14:28:46.851 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55444.log 2023-10-30T14:28:46.852 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59020.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59020.log.gz 2023-10-30T14:28:46.853 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53385.log.gz 2023-10-30T14:28:46.853 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52687.log 2023-10-30T14:28:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50522.log 2023-10-30T14:28:46.855 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55153.log 2023-10-30T14:28:46.856 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50522.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52531.log.gz 2023-10-30T14:28:46.856 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52477.log.gz 2023-10-30T14:28:46.857 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63207.log 2023-10-30T14:28:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51885.log 2023-10-30T14:28:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48232.log 2023-10-30T14:28:46.859 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63207.log.gz 2023-10-30T14:28:46.860 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50411.log 2023-10-30T14:28:46.861 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55444.log.gz 2023-10-30T14:28:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60879.log 2023-10-30T14:28:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50411.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52687.log.gz 2023-10-30T14:28:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46878.log 2023-10-30T14:28:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69534.log 2023-10-30T14:28:46.864 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60879.log.gz 2023-10-30T14:28:46.865 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53910.log 2023-10-30T14:28:46.865 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50522.log.gz 2023-10-30T14:28:46.866 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69534.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69534.log.gz 2023-10-30T14:28:46.866 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42982.log 2023-10-30T14:28:46.867 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51885.log.gz 2023-10-30T14:28:46.868 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45581.log 2023-10-30T14:28:46.868 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64027.log 2023-10-30T14:28:46.869 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49027.log 2023-10-30T14:28:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64027.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64027.log.gz 2023-10-30T14:28:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62423.log 2023-10-30T14:28:46.871 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48232.log.gz 2023-10-30T14:28:46.872 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58923.log 2023-10-30T14:28:46.873 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49027.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55153.log.gz 2023-10-30T14:28:46.873 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50411.log.gz 2023-10-30T14:28:46.874 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62423.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46878.log.gz 2023-10-30T14:28:46.875 INFO:teuthology.orchestra.run.smithi046.stderr:gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42982.log.gz -5 2023-10-30T14:28:46.875 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53805.log 2023-10-30T14:28:46.875 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62423.log.gz 2023-10-30T14:28:46.876 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53910.log.gz 2023-10-30T14:28:46.876 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45581.log.gz 2023-10-30T14:28:46.876 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49027.log.gz 2023-10-30T14:28:46.876 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49536.log 2023-10-30T14:28:46.877 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58923.log.gz 2023-10-30T14:28:46.878 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61196.log 2023-10-30T14:28:46.878 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67583.log 2023-10-30T14:28:46.879 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69039.log 2023-10-30T14:28:46.879 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61196.log.gz 2023-10-30T14:28:46.880 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67583.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67583.log.gz 2023-10-30T14:28:46.880 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58521.log 2023-10-30T14:28:46.881 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53805.log.gz 2023-10-30T14:28:46.882 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69039.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69039.log.gz 2023-10-30T14:28:46.882 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63944.log 2023-10-30T14:28:46.882 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53494.log 2023-10-30T14:28:46.883 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58521.log.gz 2023-10-30T14:28:46.883 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63944.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49536.log.gz 2023-10-30T14:28:46.884 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63944.log.gz 2023-10-30T14:28:46.884 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70782.log 2023-10-30T14:28:46.885 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57350.log 2023-10-30T14:28:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54093.log 2023-10-30T14:28:46.886 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70782.log.gz 2023-10-30T14:28:46.887 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57350.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57350.log.gz 2023-10-30T14:28:46.887 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60424.log 2023-10-30T14:28:46.888 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61823.log 2023-10-30T14:28:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60424.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53494.log.gz 2023-10-30T14:28:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63453.log 2023-10-30T14:28:46.890 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54093.log.gz 2023-10-30T14:28:46.890 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60424.log.gz 2023-10-30T14:28:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61823.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61823.log.gz 2023-10-30T14:28:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64683.log 2023-10-30T14:28:46.892 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63453.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63453.log.gz 2023-10-30T14:28:46.892 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48765.log 2023-10-30T14:28:46.893 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64683.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64683.log.gz 2023-10-30T14:28:46.894 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53359.log 2023-10-30T14:28:46.894 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49396.log 2023-10-30T14:28:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53359.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.33173.log 2023-10-30T14:28:46.896 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62554.log 2023-10-30T14:28:46.897 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.tmp-client.admin.33173.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.33173.log.gz 2023-10-30T14:28:46.898 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52257.log 2023-10-30T14:28:46.898 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48765.log.gz 2023-10-30T14:28:46.899 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62554.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62554.log.gz 2023-10-30T14:28:46.900 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46660.log 2023-10-30T14:28:46.900 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58761.log 2023-10-30T14:28:46.901 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53359.log.gz 2023-10-30T14:28:46.902 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51937.log 2023-10-30T14:28:46.902 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49396.log.gz 2023-10-30T14:28:46.903 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58761.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58761.log.gz 2023-10-30T14:28:46.903 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45294.log 2023-10-30T14:28:46.904 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58839.log 2023-10-30T14:28:46.905 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55094.log 2023-10-30T14:28:46.906 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60800.log 2023-10-30T14:28:46.907 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58839.log.gz 2023-10-30T14:28:46.907 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58358.log 2023-10-30T14:28:46.908 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52257.log.gz 2023-10-30T14:28:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60800.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60800.log.gz 2023-10-30T14:28:46.909 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69312.log 2023-10-30T14:28:46.910 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46660.log.gz 2023-10-30T14:28:46.911 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58358.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51937.log.gz 2023-10-30T14:28:46.911 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58358.log.gz 2023-10-30T14:28:46.911 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69817.log 2023-10-30T14:28:46.912 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69312.log.gz 2023-10-30T14:28:46.912 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45294.log.gz 2023-10-30T14:28:46.913 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51195.log 2023-10-30T14:28:46.914 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61875.log 2023-10-30T14:28:46.914 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69817.log.gz 2023-10-30T14:28:46.915 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70597.log 2023-10-30T14:28:46.915 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61875.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61875.log.gz 2023-10-30T14:28:46.916 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68723.log 2023-10-30T14:28:46.916 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55094.log.gz 2023-10-30T14:28:46.917 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49562.log 2023-10-30T14:28:46.918 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51195.log.gz 2023-10-30T14:28:46.918 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70597.log.gz 2023-10-30T14:28:46.919 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68723.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68723.log.gz 2023-10-30T14:28:46.919 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53673.log 2023-10-30T14:28:46.920 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67204.log 2023-10-30T14:28:46.921 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65803.log 2023-10-30T14:28:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55234.log 2023-10-30T14:28:46.922 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67204.log.gz 2023-10-30T14:28:46.923 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65803.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65803.log.gz 2023-10-30T14:28:46.924 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50736.log 2023-10-30T14:28:46.925 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44906.log 2023-10-30T14:28:46.925 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49562.log.gz 2023-10-30T14:28:46.926 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50736.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53673.log.gz 2023-10-30T14:28:46.926 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52231.log 2023-10-30T14:28:46.927 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57983.log 2023-10-30T14:28:46.928 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69690.log 2023-10-30T14:28:46.929 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69371.log 2023-10-30T14:28:46.930 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55234.log.gz 2023-10-30T14:28:46.930 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57983.log.gz 2023-10-30T14:28:46.931 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50736.log.gz 2023-10-30T14:28:46.932 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69690.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69690.log.gz 2023-10-30T14:28:46.932 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56060.log 2023-10-30T14:28:46.932 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44906.log.gz 2023-10-30T14:28:46.933 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62069.log 2023-10-30T14:28:46.934 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69371.log.gz 2023-10-30T14:28:46.934 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50999.log 2023-10-30T14:28:46.934 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56060.log.gz 2023-10-30T14:28:46.935 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62069.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52231.log.gz 2023-10-30T14:28:46.936 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62069.log.gz 2023-10-30T14:28:46.936 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68830.log 2023-10-30T14:28:46.936 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68775.log 2023-10-30T14:28:46.938 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55364.log 2023-10-30T14:28:46.938 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68830.log.gz 2023-10-30T14:28:46.938 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49666.log 2023-10-30T14:28:46.939 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68775.log.gz 2023-10-30T14:28:46.939 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56034.log 2023-10-30T14:28:46.940 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50999.log.gz 2023-10-30T14:28:46.941 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53699.log 2023-10-30T14:28:46.941 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66701.log 2023-10-30T14:28:46.942 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56034.log.gz 2023-10-30T14:28:46.943 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51459.log 2023-10-30T14:28:46.943 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55364.log.gz 2023-10-30T14:28:46.944 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66701.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49666.log.gz 2023-10-30T14:28:46.944 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70493.log 2023-10-30T14:28:46.945 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66701.log.gz 2023-10-30T14:28:46.945 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66312.log 2023-10-30T14:28:46.946 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70493.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70493.log.gz 2023-10-30T14:28:46.947 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45784.log 2023-10-30T14:28:46.948 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66312.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66312.log.gz 2023-10-30T14:28:46.948 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60580.log 2023-10-30T14:28:46.949 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57874.log 2023-10-30T14:28:46.949 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60580.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53699.log.gz 2023-10-30T14:28:46.950 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60580.log.gz 2023-10-30T14:28:46.950 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-10-30T14:28:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57874.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57874.log.gz 2023-10-30T14:28:46.952 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70415.log 2023-10-30T14:28:46.952 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51459.log.gz 2023-10-30T14:28:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mds.e.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45784.log.gz 2023-10-30T14:28:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59503.log 2023-10-30T14:28:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53725.log 2023-10-30T14:28:46.955 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70415.log.gz 2023-10-30T14:28:46.956 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59503.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59503.log.gz 2023-10-30T14:28:46.956 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53779.log 2023-10-30T14:28:46.956 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69069.log 2023-10-30T14:28:46.957 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45369.log 2023-10-30T14:28:46.958 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69069.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69069.log.gz 2023-10-30T14:28:46.958 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46826.log 2023-10-30T14:28:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46852.log 2023-10-30T14:28:46.961 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59556.log 2023-10-30T14:28:46.961 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53725.log.gz 2023-10-30T14:28:46.962 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51544.log 2023-10-30T14:28:46.963 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53779.log.gz 2023-10-30T14:28:46.963 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45369.log.gz 2023-10-30T14:28:46.963 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46826.log.gz 2023-10-30T14:28:46.964 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59556.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46852.log.gz 2023-10-30T14:28:46.964 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51728.log 2023-10-30T14:28:46.965 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59556.log.gz 2023-10-30T14:28:46.966 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43620.log 2023-10-30T14:28:46.966 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70860.log 2023-10-30T14:28:46.967 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53411.log 2023-10-30T14:28:46.967 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52146.log 88.6% 2023-10-30T14:28:46.968 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51544.log.gz 2023-10-30T14:28:46.969 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70860.log.gz 2023-10-30T14:28:46.969 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53411.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43620.log.gz 2023-10-30T14:28:46.969 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66888.log 2023-10-30T14:28:46.970 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67045.log 2023-10-30T14:28:46.971 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53411.log.gz 2023-10-30T14:28:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66888.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66888.log.gz 2023-10-30T14:28:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55757.log 2023-10-30T14:28:46.973 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52146.log.gz 2023-10-30T14:28:46.973 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67045.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67045.log.gz 2023-10-30T14:28:46.974 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61903.log 2023-10-30T14:28:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54910.log 2023-10-30T14:28:46.975 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55757.log.gz 2023-10-30T14:28:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61903.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61903.log.gz 2023-10-30T14:28:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61168.log 2023-10-30T14:28:46.977 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58656.log 2023-10-30T14:28:46.978 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61168.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61168.log.gz 2023-10-30T14:28:46.978 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46797.log 2023-10-30T14:28:46.979 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46904.log 2023-10-30T14:28:46.980 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58656.log.gz 2023-10-30T14:28:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46797.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54910.log.gz 2023-10-30T14:28:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54305.log 2023-10-30T14:28:46.982 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46797.log.gz 2023-10-30T14:28:46.982 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54763.log 2023-10-30T14:28:46.983 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61142.log 2023-10-30T14:28:46.984 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60665.log 2023-10-30T14:28:46.985 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46904.log.gz 2023-10-30T14:28:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61142.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61142.log.gz 2023-10-30T14:28:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43607.log 2023-10-30T14:28:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60665.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54305.log.gz 2023-10-30T14:28:46.988 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54763.log.gz 2023-10-30T14:28:46.988 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60665.log.gz 2023-10-30T14:28:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64236.log 2023-10-30T14:28:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70571.log 2023-10-30T14:28:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64236.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51728.log.gz 2023-10-30T14:28:46.991 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64236.log.gz 2023-10-30T14:28:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47990.log 2023-10-30T14:28:46.991 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43607.log.gz 2023-10-30T14:28:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59613.log 2023-10-30T14:28:46.992 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70571.log.gz 2023-10-30T14:28:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43213.log 2023-10-30T14:28:46.994 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55068.log 2023-10-30T14:28:46.994 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59613.log.gz 2023-10-30T14:28:46.994 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47990.log.gz 2023-10-30T14:28:46.995 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44110.log 2023-10-30T14:28:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54962.log 2023-10-30T14:28:46.997 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70059.log 2023-10-30T14:28:46.997 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43213.log.gz 2023-10-30T14:28:46.998 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54962.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55068.log.gz 2023-10-30T14:28:47.000 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70059.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70059.log.gz 2023-10-30T14:28:47.002 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44110.log.gz 2023-10-30T14:28:47.003 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54962.log.gz 2023-10-30T14:28:47.166 INFO:teuthology.orchestra.run.smithi046.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-30T14:28:47.259 INFO:teuthology.orchestra.run.smithi154.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-30T14:28:47.294 INFO:teuthology.orchestra.run.smithi154.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-30T14:28:47.839 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T14:28:52.775 INFO:teuthology.orchestra.run.smithi154.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-30T14:28:52.863 INFO:teuthology.orchestra.run.smithi154.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-30T14:28:56.637 INFO:teuthology.orchestra.run.smithi046.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-30T14:28:57.015 INFO:teuthology.orchestra.run.smithi154.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-30T14:28:57.199 INFO:teuthology.orchestra.run.smithi154.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-30T14:28:59.287 INFO:teuthology.orchestra.run.smithi154.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-30T14:28:59.289 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-10-30T14:28:59.289 INFO:teuthology.orchestra.run.smithi154.stderr:real 0m13.271s 2023-10-30T14:28:59.289 INFO:teuthology.orchestra.run.smithi154.stderr:user 0m50.321s 2023-10-30T14:28:59.289 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m1.740s 2023-10-30T14:29:07.921 INFO:teuthology.orchestra.run.smithi046.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-30T14:29:14.876 INFO:teuthology.orchestra.run.smithi046.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-30T14:29:25.071 INFO:teuthology.orchestra.run.smithi046.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-30T14:29:29.202 INFO:teuthology.orchestra.run.smithi046.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-30T14:29:42.498 INFO:teuthology.orchestra.run.smithi046.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-30T14:31:10.123 INFO:teuthology.orchestra.run.smithi046.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-10-30T14:31:10.124 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-10-30T14:31:10.125 INFO:teuthology.orchestra.run.smithi046.stderr:real 2m24.135s 2023-10-30T14:31:10.125 INFO:teuthology.orchestra.run.smithi046.stderr:user 2m8.236s 2023-10-30T14:31:10.125 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m14.482s 2023-10-30T14:31:10.127 INFO:tasks.ceph:Archiving logs... 2023-10-30T14:31:10.127 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045/remote/smithi046/log 2023-10-30T14:31:10.128 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T14:31:43.328 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045/remote/smithi154/log 2023-10-30T14:31:43.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T14:31:48.661 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-30T14:31:48.671 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-30T14:31:48.671 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T14:31:48.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T14:31:48.746 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-30T14:31:48.756 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-30T14:31:48.756 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T14:31:48.761 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T14:31:48.777 INFO:teuthology.orchestra.run.smithi154.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T14:31:48.778 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================== 2023-10-30T14:31:48.778 INFO:teuthology.orchestra.run.smithi154.stdout:*hv01.front.sepi 44.4.53.2 2 u 71 128 377 0.148 -5.019 0.927 2023-10-30T14:31:48.778 INFO:teuthology.orchestra.run.smithi154.stdout:+hv02.front.sepi 63.231.80.2 3 u 125 128 377 0.124 -1.213 1.268 2023-10-30T14:31:48.779 INFO:teuthology.orchestra.run.smithi154.stdout:+hv03.front.sepi 74.6.168.72 3 u 14 128 377 0.168 -2.131 1.279 2023-10-30T14:31:48.779 INFO:teuthology.orchestra.run.smithi154.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-30T14:31:48.806 INFO:teuthology.orchestra.run.smithi046.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T14:31:48.806 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================== 2023-10-30T14:31:48.806 INFO:teuthology.orchestra.run.smithi046.stdout:*hv01.front.sepi 44.4.53.2 2 u 83 256 377 0.150 -2.095 0.750 2023-10-30T14:31:48.807 INFO:teuthology.orchestra.run.smithi046.stdout:+hv02.front.sepi 63.231.80.2 3 u 127 256 377 0.121 -0.225 0.229 2023-10-30T14:31:48.807 INFO:teuthology.orchestra.run.smithi046.stdout:+hv03.front.sepi 74.6.168.72 3 u 145 256 377 0.153 -0.637 0.558 2023-10-30T14:31:48.807 INFO:teuthology.orchestra.run.smithi046.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-30T14:31:48.809 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-30T14:31:48.818 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-30T14:31:48.819 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-30T14:31:48.850 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-30T14:31:48.862 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-30T14:31:48.965 INFO:teuthology.task.internal:Duration was 4853.474245 seconds 2023-10-30T14:31:48.966 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-30T14:31:48.977 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-30T14:31:48.977 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T14:31:48.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T14:31:49.251 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-30T14:31:49.251 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2023-10-30T14:31:49.252 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T14:31:49.276 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2023-10-30T14:31:49.276 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T14:31:49.288 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-30T14:31:49.289 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T14:31:49.327 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T14:31:49.404 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-30T14:31:49.404 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T14:31:49.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T14:31:49.625 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-30T14:31:49.636 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-30T14:31:49.637 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T14:31:49.659 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T14:31:49.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-30T14:31:49.685 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T14:31:49.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T14:31:49.729 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2023-10-30T14:31:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2023-10-30T14:31:49.774 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T14:31:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:31:49.823 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T14:31:49.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T14:31:49.828 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-30T14:31:49.839 INFO:teuthology.task.internal:Transferring archived files... 2023-10-30T14:31:49.840 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045/remote/smithi046 2023-10-30T14:31:49.840 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T14:31:49.889 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045/remote/smithi154 2023-10-30T14:31:49.890 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T14:31:49.908 INFO:teuthology.task.internal:Removing archive directory... 2023-10-30T14:31:49.909 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T14:31:49.930 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T14:31:49.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-30T14:31:49.966 INFO:teuthology.task.internal:Not uploading archives. 2023-10-30T14:31:49.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-30T14:31:49.976 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-30T14:31:49.977 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T14:31:49.982 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T14:31:49.988 INFO:teuthology.orchestra.run.smithi046.stdout: 657970 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 14:31 /home/ubuntu/cephtest 2023-10-30T14:31:50.003 INFO:teuthology.orchestra.run.smithi154.stdout: 524292 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 14:31 /home/ubuntu/cephtest 2023-10-30T14:31:50.004 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-30T14:31:50.016 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-30T14:31:50.037 INFO:teuthology.nuke:Checking targets against current locks 2023-10-30T14:31:50.060 DEBUG:teuthology.nuke:shortname: smithi046 2023-10-30T14:31:50.061 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T14:31:50.086 DEBUG:teuthology.nuke:shortname: smithi154 2023-10-30T14:31:50.086 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T14:31:50.089 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 13:01:04.544015', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE8KRBqKi4J4ac52uwYhjG88eMhxcjlX+sg7m3/vyM3wFY/30AJq3vthxKneW/V1m+99tGjKFak5Zb3TtmS+yWc='} 2023-10-30T14:31:50.109 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441045', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 13:01:04.542408', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBQUkOw5PZhB5D+tQQfIqdQCcoWCwOAKnTkiMvcCnO+2w8k4FWZdHEkcrgVnAq8Uovev07ctOSFh0WtmGK+Dgo='} 2023-10-30T14:31:50.136 INFO:teuthology.orchestra.console.smithi046:Power off 2023-10-30T14:31:50.137 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T14:31:50.163 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2023-10-30T14:31:50.163 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:31:50.180 INFO:teuthology.orchestra.console.smithi154:Power off 2023-10-30T14:31:50.181 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T14:31:50.190 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2023-10-30T14:31:50.203 DEBUG:teuthology.orchestra.console.smithi154:power off output: Chassis Power Control: Down/Off 2023-10-30T14:31:50.203 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:31:50.228 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2023-10-30T14:31:54.192 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:31:54.229 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:31:54.303 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2023-10-30T14:31:54.342 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2023-10-30T14:31:58.306 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:31:58.342 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T14:31:58.417 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2023-10-30T14:31:58.452 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off 2023-10-30T14:31:58.519 INFO:teuthology.orchestra.console.smithi046:Power off completed 2023-10-30T14:31:58.554 INFO:teuthology.orchestra.console.smithi154:Power off completed 2023-10-30T14:31:58.660 INFO:teuthology.run:Summary data: client.0-kernel-sha1: eacf00cc51e204f401b4d5d4f84825a68f373c74 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 4853.474244832993 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on smithi046 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-split.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2d97ed70c57841a19ce6f9750f8134ee status: fail success: false 2023-10-30T14:31:58.661 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T14:31:58.740 INFO:teuthology.run:FAIL